2024-03-06T07:07:41.595 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-06T07:07:41.595 DEBUG:teuthology.run:Teuthology command: teuthology --name leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi --description fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} --verbose --owner scheduled_leonidus@teuthology --archive /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583045 -- /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583045/orig.config.yaml 2024-03-06T07:07:41.641 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:07:41.718 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583045 branch: wip-lusov-qdb description: fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} email: null first_in_suite: false job_id: '7583045' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - Replacing daemon mds - \(MDS_INSUFFICIENT_STANDBY\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MDS_DAMAGE\) - \(FS_DEGRADED\) sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: disabled: true install: ceph: flavor: default sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd kclient: syntax: v2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 seed: 5072 sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd sleep_before_teardown: 0 subset: 246/10000 suite: fs suite_branch: wip-lusov-qdb suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi001.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKALq4zEqGpGYLaiiOGwZ7qnJ/XLWKAx+2LZBoOmFGiRSpv8TRDeLuBEvCbSwK2CTmZRnmuIej2HD0aox/FiC68= smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLZZgtAwrVih+44oDkYrNyHiWI3sLRPpRARTCFN6ZaHTMKDPew6Nx769IQH3Zx8HSMODQC5eLh3FE355iWsox7w= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - exec: mon.a: - ceph config set mgr mgr/crash/warn_recent_interval 0 - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_failover teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-06_07:00:27 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-06T07:07:41.718 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa; will attempt to use it 2024-03-06T07:07:41.719 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks 2024-03-06T07:07:41.719 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-06T07:07:41.720 INFO:teuthology.task.internal:Checking packages... 2024-03-06T07:07:41.747 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd' 2024-03-06T07:07:41.747 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-06T07:07:41.747 INFO:teuthology.packaging:ref: None 2024-03-06T07:07:41.747 INFO:teuthology.packaging:tag: None 2024-03-06T07:07:41.747 INFO:teuthology.packaging:branch: wip-lusov-qdb 2024-03-06T07:07:41.747 INFO:teuthology.packaging:sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:07:41.747 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb 2024-03-06T07:07:41.984 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1739.g1bc250c7 2024-03-06T07:07:41.985 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-06T07:07:41.992 INFO:teuthology.task.internal:no buildpackages task found 2024-03-06T07:07:41.992 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-06T07:07:42.010 INFO:teuthology.task.internal:Saving configuration 2024-03-06T07:07:42.027 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-06T07:07:42.033 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:07:42.061 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583045', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:01:54.233623', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKALq4zEqGpGYLaiiOGwZ7qnJ/XLWKAx+2LZBoOmFGiRSpv8TRDeLuBEvCbSwK2CTmZRnmuIej2HD0aox/FiC68='} 2024-03-06T07:07:42.083 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583045', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:01:54.232112', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLZZgtAwrVih+44oDkYrNyHiWI3sLRPpRARTCFN6ZaHTMKDPew6Nx769IQH3Zx8HSMODQC5eLh3FE355iWsox7w='} 2024-03-06T07:07:42.083 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-06T07:07:42.089 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-06T07:07:42.090 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-03-06T07:07:42.090 INFO:teuthology.run_tasks:Running task console_log... 2024-03-06T07:07:42.155 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0d32a230d0>, signals=[15]) 2024-03-06T07:07:42.156 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-06T07:07:42.162 INFO:teuthology.task.internal:Opening connections... 2024-03-06T07:07:42.162 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi001.front.sepia.ceph.com 2024-03-06T07:07:42.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:07:42.270 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-03-06T07:07:42.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:07:42.361 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-06T07:07:42.368 DEBUG:teuthology.orchestra.run.smithi001:> uname -m 2024-03-06T07:07:42.386 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2024-03-06T07:07:42.386 DEBUG:teuthology.orchestra.run.smithi001:> cat /etc/os-release 2024-03-06T07:07:42.443 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="CentOS Stream" 2024-03-06T07:07:42.443 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="9" 2024-03-06T07:07:42.444 INFO:teuthology.orchestra.run.smithi001.stdout:ID="centos" 2024-03-06T07:07:42.444 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:07:42.444 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="9" 2024-03-06T07:07:42.444 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:07:42.444 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:07:42.444 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31" 2024-03-06T07:07:42.444 INFO:teuthology.orchestra.run.smithi001.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:07:42.444 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:07:42.444 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:07:42.444 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:07:42.444 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:07:42.444 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:07:42.445 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2024-03-06T07:07:42.466 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-03-06T07:07:42.483 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-03-06T07:07:42.483 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-03-06T07:07:42.539 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-03-06T07:07:42.540 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-03-06T07:07:42.540 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-03-06T07:07:42.540 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:07:42.540 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-03-06T07:07:42.540 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:07:42.540 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:07:42.540 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-03-06T07:07:42.540 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:07:42.540 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:07:42.540 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:07:42.540 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:07:42.541 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:07:42.541 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:07:42.541 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-03-06T07:07:42.566 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-06T07:07:42.575 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-06T07:07:42.614 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-06T07:07:42.614 DEBUG:teuthology.orchestra.run.smithi001:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:07:42.616 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:07:42.632 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-06T07:07:42.638 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-06T07:07:42.639 DEBUG:teuthology.orchestra.run.smithi001:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:07:42.673 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:07:42.696 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-06T07:07:42.748 INFO:teuthology.run_tasks:Running task kernel... 2024-03-06T07:07:42.763 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-03-06T07:07:42.764 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}} 2024-03-06T07:07:42.764 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-03-06T07:07:42.764 DEBUG:teuthology.orchestra.run.smithi001:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:07:42.765 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:07:42.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:07:42.781 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-03-06T07:07:42.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:07:42.782 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-03-06T07:07:42.838 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:07:42.838 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-361.el9.x86_64 2024-03-06T07:07:42.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-03-06T07:07:42.841 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:07:42.841 INFO:teuthology.task.kernel:Running kernel on smithi001: 5.14.0-361.el9.x86_64 2024-03-06T07:07:42.841 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum install -y kernel 2024-03-06T07:07:45.030 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-06T07:07:45.257 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 9.8 MB/s | 6.7 MB 00:00 2024-03-06T07:07:46.489 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-06T07:07:46.632 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 7.7 MB/s | 2.0 MB 00:00 2024-03-06T07:07:47.329 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-06T07:07:48.564 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.0 MB 00:01 2024-03-06T07:07:50.438 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 9.4 MB/s | 19 MB 00:01 2024-03-06T07:07:50.540 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-06T07:07:54.606 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-06T07:07:54.608 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-06T07:07:56.298 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2024-03-06T07:07:56.310 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 15 kB 00:00 2024-03-06T07:07:57.433 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-06T07:07:59.594 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 6.5 MB/s | 20 MB 00:03 2024-03-06T07:08:02.491 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-06T07:08:04.602 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-06T07:08:04.748 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-06T07:08:04.748 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-06T07:08:04.834 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-06T07:08:04.839 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-06T07:08:04.839 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-03-06T07:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-06T07:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-06T07:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-06T07:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-06T07:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-06T07:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-06T07:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-06T07:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-06T07:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-06T07:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout:Install 4 Packages 2024-03-06T07:08:04.840 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:08:04.841 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 100 M 2024-03-06T07:08:04.841 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 123 M 2024-03-06T07:08:04.841 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-06T07:08:06.259 INFO:teuthology.orchestra.run.smithi001.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 4.6 MB/s | 6.0 MB 00:01 2024-03-06T07:08:06.767 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-06T07:08:06.767 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-06T07:08:06.847 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-06T07:08:06.853 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-06T07:08:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-03-06T07:08:06.854 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-06T07:08:06.854 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-06T07:08:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-06T07:08:06.854 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-06T07:08:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-06T07:08:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-06T07:08:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-06T07:08:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:08:06.854 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-06T07:08:06.854 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-06T07:08:06.854 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-03-06T07:08:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:08:06.855 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 100 M 2024-03-06T07:08:06.855 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 123 M 2024-03-06T07:08:06.855 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-06T07:08:07.212 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 21 MB/s | 6.0 MB 00:00 2024-03-06T07:08:07.554 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 33 MB/s | 21 MB 00:00 2024-03-06T07:08:08.043 INFO:teuthology.orchestra.run.smithi001.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 6.8 MB/s | 21 MB 00:03 2024-03-06T07:08:08.104 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 33 MB/s | 39 MB 00:01 2024-03-06T07:08:08.446 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 27 MB/s | 33 MB 00:01 2024-03-06T07:08:08.446 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:08:08.446 INFO:teuthology.orchestra.run.smithi098.stdout:Total 63 MB/s | 100 MB 00:01 2024-03-06T07:08:09.336 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-06T07:08:09.708 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-06T07:08:09.709 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-06T07:08:10.160 INFO:teuthology.orchestra.run.smithi001.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 7.6 MB/s | 39 MB 00:05 2024-03-06T07:08:10.543 INFO:teuthology.orchestra.run.smithi001.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 7.8 MB/s | 33 MB 00:04 2024-03-06T07:08:10.544 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:08:10.544 INFO:teuthology.orchestra.run.smithi001.stdout:Total 17 MB/s | 100 MB 00:05 2024-03-06T07:08:11.345 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-06T07:08:11.345 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-06T07:08:11.471 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-06T07:08:11.892 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-06T07:08:11.892 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-06T07:08:13.286 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-06T07:08:13.286 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-06T07:08:14.050 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-06T07:08:14.710 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:08:14.832 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:08:16.130 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-06T07:08:16.807 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:08:16.943 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:08:17.608 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:08:17.793 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:08:19.882 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:08:20.070 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:08:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:08:22.729 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:25.172 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:08:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:27.523 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:30.418 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:56.828 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:56.829 INFO:teuthology.orchestra.run.smithi098.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:08:56.829 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:08:56.909 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:57.425 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:57.425 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:08:57.425 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:08:57.425 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:08:57.830 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:57.830 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:08:57.830 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-06T07:08:57.830 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-06T07:08:57.830 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-06T07:08:57.830 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-06T07:08:57.830 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-06T07:08:57.830 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:08:57.830 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-06T07:08:58.372 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:08:58.554 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:08:58.555 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:08:58.556 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-06T07:08:58.556 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-06T07:08:58.556 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-06T07:08:58.556 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2024-03-06T07:08:58.556 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-03-06T07:08:58.556 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:08:58.677 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:08:58.677 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:08:58.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2024-03-06T07:08:58.719 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2024-03-06T07:08:58.719 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2024-03-06T07:08:58.719 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.06 2024-03-06T07:08:58.720 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 68.el9 2024-03-06T07:08:58.720 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2024-03-06T07:08:58.720 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:08:58.720 INFO:teuthology.orchestra.run.smithi098.stdout:Group : Unspecified 2024-03-06T07:08:58.720 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 8264254 2024-03-06T07:08:58.720 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2024-03-06T07:08:58.720 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:08:58.720 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:08:58.720 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:08:58.720 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:08:58.720 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : builder@centos.org 2024-03-06T07:08:58.720 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2024-03-06T07:08:58.721 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:08:58.721 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2024-03-06T07:08:58.721 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2024-03-06T07:08:58.721 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:08:58.721 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:08:58.721 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:08:58.721 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:08:58.721 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2024-03-06T07:08:58.721 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:08:58.721 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:08:58.722 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:08:58.722 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 5.14.0-427.el9.x86_64 2024-03-06T07:08:58.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:08:59.326 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-03-06T07:09:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:09:00.714 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-03-06T07:09:00.717 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-03-06T07:09:00.732 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.yOPpUOX6Sk 2024-03-06T07:09:00.733 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yOPpUOX6Sk 2024-03-06T07:09:00.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.yOPpUOX6Sk 2024-03-06T07:09:00.934 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.yOPpUOX6Sk is 6KB 2024-03-06T07:09:00.966 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:09:00.966 INFO:teuthology.orchestra.run.smithi001.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:09:00.966 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:09:00.983 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.yOPpUOX6Sk 2024-03-06T07:09:00.997 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:09:01.055 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:09:01.069 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:09:01.069 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:09:01.069 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:09:01.069 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-06T07:09:01.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-06T07:09:01.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-03-06T07:09:01.516 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:09:01.517 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:09:01.517 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:09:01.517 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:09:01.994 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:09:01.994 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:09:01.994 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-06T07:09:01.994 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-06T07:09:01.994 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-06T07:09:01.994 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-06T07:09:01.994 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-06T07:09:01.994 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:09:01.994 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-06T07:09:02.469 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:09:02.620 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:09:02.621 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:09:02.621 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-06T07:09:02.621 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-06T07:09:02.621 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-06T07:09:02.621 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-06T07:09:02.621 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi001.front.sepia.ceph.com, path=None, version=distro) 2024-03-06T07:09:02.621 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:09:02.747 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:09:02.747 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:09:02.747 DEBUG:teuthology.orchestra.run.smithi001:> sudo rpm -qi grub2-tools 2024-03-06T07:09:02.785 INFO:teuthology.orchestra.run.smithi001.stdout:Name : grub2-tools 2024-03-06T07:09:02.785 INFO:teuthology.orchestra.run.smithi001.stdout:Epoch : 1 2024-03-06T07:09:02.785 INFO:teuthology.orchestra.run.smithi001.stdout:Version : 2.06 2024-03-06T07:09:02.785 INFO:teuthology.orchestra.run.smithi001.stdout:Release : 68.el9 2024-03-06T07:09:02.785 INFO:teuthology.orchestra.run.smithi001.stdout:Architecture: x86_64 2024-03-06T07:09:02.785 INFO:teuthology.orchestra.run.smithi001.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:09:02.786 INFO:teuthology.orchestra.run.smithi001.stdout:Group : Unspecified 2024-03-06T07:09:02.786 INFO:teuthology.orchestra.run.smithi001.stdout:Size : 8264254 2024-03-06T07:09:02.786 INFO:teuthology.orchestra.run.smithi001.stdout:License : GPLv3+ 2024-03-06T07:09:02.786 INFO:teuthology.orchestra.run.smithi001.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:09:02.786 INFO:teuthology.orchestra.run.smithi001.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:09:02.786 INFO:teuthology.orchestra.run.smithi001.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:09:02.786 INFO:teuthology.orchestra.run.smithi001.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:09:02.786 INFO:teuthology.orchestra.run.smithi001.stdout:Packager : builder@centos.org 2024-03-06T07:09:02.786 INFO:teuthology.orchestra.run.smithi001.stdout:Vendor : CentOS 2024-03-06T07:09:02.786 INFO:teuthology.orchestra.run.smithi001.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:09:02.786 INFO:teuthology.orchestra.run.smithi001.stdout:Summary : Support tools for GRUB. 2024-03-06T07:09:02.786 INFO:teuthology.orchestra.run.smithi001.stdout:Description : 2024-03-06T07:09:02.786 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:09:02.786 INFO:teuthology.orchestra.run.smithi001.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:09:02.787 INFO:teuthology.orchestra.run.smithi001.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:09:02.787 INFO:teuthology.orchestra.run.smithi001.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:09:02.787 INFO:teuthology.orchestra.run.smithi001.stdout:hardware devices. 2024-03-06T07:09:02.787 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:09:02.787 INFO:teuthology.orchestra.run.smithi001.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:09:02.788 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:09:02.788 INFO:teuthology.task.kernel:Updating grub on smithi001 to boot 5.14.0-427.el9.x86_64 2024-03-06T07:09:02.788 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:09:03.462 INFO:teuthology.orchestra.run.smithi001.stderr:Generating grub configuration file ... 2024-03-06T07:09:04.854 INFO:teuthology.orchestra.run.smithi001.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:09:04.887 INFO:teuthology.orchestra.run.smithi001.stderr:done 2024-03-06T07:09:04.890 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-03-06T07:09:04.905 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.PEJWOnEJ3R 2024-03-06T07:09:04.905 DEBUG:teuthology.orchestra.run.smithi001:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.PEJWOnEJ3R 2024-03-06T07:09:04.982 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.PEJWOnEJ3R 2024-03-06T07:09:05.121 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.PEJWOnEJ3R is 6KB 2024-03-06T07:09:05.169 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.PEJWOnEJ3R 2024-03-06T07:09:05.184 DEBUG:teuthology.orchestra.run.smithi001:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:09:05.254 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:09:05.254 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:09:05.254 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:09:05.254 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-06T07:09:05.255 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-06T07:09:05.420 DEBUG:teuthology.orchestra.run.smithi001:> sudo shutdown -r now 2024-03-06T07:09:31.237 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-03-06T07:09:31.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-06T07:09:31.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:09:35.428 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-06T07:09:35.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-06T07:09:35.429 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:09:49.756 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-06T07:09:53.852 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-03-06T07:09:58.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-06T07:09:58.761 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:01.820 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-06T07:10:02.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-06T07:10:02.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:13.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-06T07:10:13.833 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:16.892 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-06T07:10:31.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-06T07:10:31.898 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:34.972 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-06T07:10:52.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-06T07:10:52.990 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:53.365 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-06T07:10:53.744 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-06T07:10:53.744 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "5.14.0-427.el9.x86_64"... 2024-03-06T07:10:53.745 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-03-06T07:10:53.761 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-427.el9.x86_64 2024-03-06T07:10:53.761 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-06T07:10:53.761 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:10:53.761 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:10:54.762 DEBUG:teuthology.parallel:result is None 2024-03-06T07:11:02.857 DEBUG:teuthology.orchestra.remote:timed out 2024-03-06T07:11:14.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-06T07:11:14.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:11:15.150 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-03-06T07:11:15.504 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-06T07:11:15.505 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-427.el9.x86_64"... 2024-03-06T07:11:15.505 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-03-06T07:11:15.520 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-427.el9.x86_64 2024-03-06T07:11:15.520 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-06T07:11:15.520 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:11:15.521 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:11:16.521 DEBUG:teuthology.parallel:result is None 2024-03-06T07:11:16.522 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-06T07:11:16.529 INFO:teuthology.task.internal:Creating test directory... 2024-03-06T07:11:16.530 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:11:16.532 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:11:16.550 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-06T07:11:16.557 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-06T07:11:16.567 INFO:teuthology.task.internal:Creating archive directory... 2024-03-06T07:11:16.567 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:11:16.590 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:11:16.633 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-06T07:11:16.643 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-06T07:11:16.643 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:11:16.654 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:11:16.742 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:11:16.770 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:11:16.780 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:11:16.812 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:11:16.814 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-06T07:11:16.822 INFO:teuthology.task.internal:Configuring sudo... 2024-03-06T07:11:16.822 DEBUG:teuthology.orchestra.run.smithi001:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:11:16.824 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:11:16.855 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-06T07:11:16.864 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-06T07:11:16.864 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:11:16.895 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:11:16.911 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:11:16.994 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:11:17.042 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:11:17.123 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:11:17.169 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:11:17.170 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:11:17.237 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:11:17.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:11:17.336 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:11:17.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:11:17.463 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:11:17.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:11:17.531 DEBUG:teuthology.orchestra.run.smithi001:> sudo service rsyslog restart 2024-03-06T07:11:17.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-03-06T07:11:17.576 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:11:17.622 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:11:17.938 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-06T07:11:17.949 INFO:teuthology.task.internal:Starting timer... 2024-03-06T07:11:17.949 INFO:teuthology.run_tasks:Running task pcp... 2024-03-06T07:11:17.972 INFO:teuthology.run_tasks:Running task selinux... 2024-03-06T07:11:17.980 DEBUG:teuthology.orchestra.run.smithi001:> sudo service auditd rotate 2024-03-06T07:11:18.038 INFO:teuthology.orchestra.run.smithi001.stdout:Rotating logs: 2024-03-06T07:11:18.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-03-06T07:11:18.099 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-03-06T07:11:18.101 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-06T07:11:18.101 DEBUG:teuthology.orchestra.run.smithi001:> /usr/sbin/getenforce 2024-03-06T07:11:18.124 INFO:teuthology.orchestra.run.smithi001.stdout:Permissive 2024-03-06T07:11:18.124 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-03-06T07:11:18.146 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-03-06T07:11:18.146 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi001.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi098.front.sepia.ceph.com': 'permissive'} 2024-03-06T07:11:18.146 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:11:18.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:11:18.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:11:18.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:11:18.222 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-06T07:11:18.222 DEBUG:teuthology.orchestra.run.smithi001:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:11:18.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:11:18.305 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-06T07:11:18.315 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-06T07:11:18.377 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-06T07:11:18.379 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi001.front.sepia.ceph.com,smithi098.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-06T07:15:35.638 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi001.front.sepia.ceph.com'), Remote(name='ubuntu@smithi098.front.sepia.ceph.com')] 2024-03-06T07:15:35.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-06T07:15:35.641 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:15:35.734 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-03-06T07:15:35.815 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-06T07:15:35.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-06T07:15:35.816 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:15:35.906 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-06T07:15:35.984 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-06T07:15:35.984 INFO:teuthology.run_tasks:Running task clock... 2024-03-06T07:15:35.995 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-06T07:15:35.995 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:15:35.996 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:15:35.999 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:15:35.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:15:36.027 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:15:36.044 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:15:36.055 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:15:36.071 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:15:36.073 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: ntpd: command not found 2024-03-06T07:15:36.086 INFO:teuthology.orchestra.run.smithi001.stdout:506 Cannot talk to daemon 2024-03-06T07:15:36.101 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-03-06T07:15:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:15:36.117 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-03-06T07:15:36.119 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:15:36.131 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:15:36.147 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:15:36.170 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:15:36.174 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:15:36.174 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-03-06T07:15:36.174 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:15:36.174 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:15:36.174 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:15:36.174 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:15:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:15:36.201 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:15:36.202 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-06T07:15:36.202 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:15:36.202 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:15:36.202 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:15:36.202 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:15:36.203 INFO:teuthology.run_tasks:Running task install... 2024-03-06T07:15:36.212 DEBUG:teuthology.task.install:project ceph 2024-03-06T07:15:36.213 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}} 2024-03-06T07:15:36.213 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'} 2024-03-06T07:15:36.213 INFO:teuthology.task.install:Using flavor: default 2024-03-06T07:15:36.220 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-06T07:15:36.220 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-06T07:15:36.221 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:15:36.222 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:15:36.224 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:15:36.225 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:15:36.367 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:15:36.367 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:15:36.455 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:15:36.455 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:15:36.687 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:15:36.688 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:15:36.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:15:36.716 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:15:36.717 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:15:36.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:15:36.807 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:15:36.807 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:15:36.807 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:15:36.837 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:15:36.837 DEBUG:teuthology.orchestra.run.smithi001:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:15:36.868 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:15:36.906 DEBUG:teuthology.orchestra.run.smithi001:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:15:36.934 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-03-06T07:15:36.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-03-06T07:15:36.988 DEBUG:teuthology.orchestra.run.smithi001:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:15:37.055 INFO:teuthology.orchestra.run.smithi001.stdout:check_obsoletes = 1 2024-03-06T07:15:37.056 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum clean all 2024-03-06T07:15:37.182 INFO:teuthology.orchestra.run.smithi098.stdout:74 files removed 2024-03-06T07:15:37.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:15:37.310 INFO:teuthology.orchestra.run.smithi001.stdout:74 files removed 2024-03-06T07:15:37.337 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:15:38.331 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 103 kB/s | 83 kB 00:00 2024-03-06T07:15:38.560 INFO:teuthology.orchestra.run.smithi001.stdout:ceph packages for x86_64 92 kB/s | 83 kB 00:00 2024-03-06T07:15:39.198 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-06T07:15:39.349 INFO:teuthology.orchestra.run.smithi001.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-06T07:15:39.898 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 2.9 kB/s | 1.9 kB 00:00 2024-03-06T07:15:40.175 INFO:teuthology.orchestra.run.smithi001.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-06T07:15:40.476 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-06T07:15:40.627 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-06T07:15:41.911 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-06T07:15:42.185 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-06T07:15:42.900 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-06T07:15:45.460 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.0 MB 00:02 2024-03-06T07:15:49.444 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 6.6 MB/s | 19 MB 00:02 2024-03-06T07:15:53.503 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-06T07:15:55.254 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 15 kB 00:00 2024-03-06T07:15:59.858 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 4.6 MB/s | 20 MB 00:04 2024-03-06T07:16:05.080 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-06T07:16:07.698 INFO:teuthology.orchestra.run.smithi001.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:16:07.699 INFO:teuthology.orchestra.run.smithi001.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:16:07.714 INFO:teuthology.orchestra.run.smithi001.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:16:07.715 INFO:teuthology.orchestra.run.smithi001.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:16:07.716 INFO:teuthology.orchestra.run.smithi001.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:16:07.717 INFO:teuthology.orchestra.run.smithi001.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:16:07.720 INFO:teuthology.orchestra.run.smithi001.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:16:07.723 INFO:teuthology.orchestra.run.smithi001.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:16:07.724 INFO:teuthology.orchestra.run.smithi001.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:16:07.812 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-06T07:16:07.831 INFO:teuthology.orchestra.run.smithi001.stdout:============================================================================================ 2024-03-06T07:16:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-03-06T07:16:07.832 INFO:teuthology.orchestra.run.smithi001.stdout:============================================================================================ 2024-03-06T07:16:07.832 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-06T07:16:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:16:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:16:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:16:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:16:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:16:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:16:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:16:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:16:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:16:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:16:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:16:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:16:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:16:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:16:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:16:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:16:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:16:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:16:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:16:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:16:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:16:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:16:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:16:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:16:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:16:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:16:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:16:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:16:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:16:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:16:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:16:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:16:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:16:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:16:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:16:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:16:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:16:07.835 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrading: 2024-03-06T07:16:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:16:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:16:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:16:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:16:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:16:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:16:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:16:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:16:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:16:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:16:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:16:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:16:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:16:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:16:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:16:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:16:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:16:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:16:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:16:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:16:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:16:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:16:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:16:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:16:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:16:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:16:07.837 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-06T07:16:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:16:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:16:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:16:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:16:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:16:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:16:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:16:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:16:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:16:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:16:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:16:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:16:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:16:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:16:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:16:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:16:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:16:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:16:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:16:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:16:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:16:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:16:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:16:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:16:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:16:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:16:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:16:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:16:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:16:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:16:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:16:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:16:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:16:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:16:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:16:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:16:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:16:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:16:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:16:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:16:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:16:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:16:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:16:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:16:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:16:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:16:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:16:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:16:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:16:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:16:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:16:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:16:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:16:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:16:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:16:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:16:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:16:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:16:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:16:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:16:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:16:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:16:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:16:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:16:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:16:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:16:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:16:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:16:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:16:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:16:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:16:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:16:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:16:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:16:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:16:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:16:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:16:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:16:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:16:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:16:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:16:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:16:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:16:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:16:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:16:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:16:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:16:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:16:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:16:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:16:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:16:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:16:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:16:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:16:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:16:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:16:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:16:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:16:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:16:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:16:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:16:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:16:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:16:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:16:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:16:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:16:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:16:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:16:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:16:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:16:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:16:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:16:07.845 INFO:teuthology.orchestra.run.smithi001.stdout:Installing weak dependencies: 2024-03-06T07:16:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:16:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:16:07.845 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-06T07:16:07.845 INFO:teuthology.orchestra.run.smithi001.stdout:============================================================================================ 2024-03-06T07:16:07.845 INFO:teuthology.orchestra.run.smithi001.stdout:Install 150 Packages 2024-03-06T07:16:07.845 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrade 26 Packages 2024-03-06T07:16:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:16:07.846 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 224 M 2024-03-06T07:16:07.846 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-06T07:16:08.815 INFO:teuthology.orchestra.run.smithi001.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-03-06T07:16:10.123 INFO:teuthology.orchestra.run.smithi001.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 679 kB/s | 888 kB 00:01 2024-03-06T07:16:10.257 INFO:teuthology.orchestra.run.smithi001.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 1.1 MB/s | 145 kB 00:00 2024-03-06T07:16:11.440 INFO:teuthology.orchestra.run.smithi001.stdout:(4/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 2.0 MB/s | 2.3 MB 00:01 2024-03-06T07:16:11.824 INFO:teuthology.orchestra.run.smithi001.stdout:(5/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 1.6 MB/s | 5.3 MB 00:03 2024-03-06T07:16:12.057 INFO:teuthology.orchestra.run.smithi001.stdout:(6/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 2.4 MB/s | 1.5 MB 00:00 2024-03-06T07:16:13.291 INFO:teuthology.orchestra.run.smithi001.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 3.2 MB/s | 4.7 MB 00:01 2024-03-06T07:16:14.399 INFO:teuthology.orchestra.run.smithi001.stdout:(8/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x86 7.2 MB/s | 17 MB 00:02 2024-03-06T07:16:14.450 INFO:teuthology.orchestra.run.smithi001.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 508 kB/s | 25 kB 00:00 2024-03-06T07:16:16.025 INFO:teuthology.orchestra.run.smithi001.stdout:(10/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 3.6 MB/s | 9.8 MB 00:02 2024-03-06T07:16:16.108 INFO:teuthology.orchestra.run.smithi001.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 2.6 MB/s | 220 kB 00:00 2024-03-06T07:16:16.325 INFO:teuthology.orchestra.run.smithi001.stdout:(12/176): ceph-common-19.0.0-1739.g1bc250c7.el9 2.6 MB/s | 20 MB 00:07 2024-03-06T07:16:16.350 INFO:teuthology.orchestra.run.smithi001.stdout:(13/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 112 kB/s | 27 kB 00:00 2024-03-06T07:16:16.401 INFO:teuthology.orchestra.run.smithi001.stdout:(14/176): libcephfs-devel-19.0.0-1739.g1bc250c7 418 kB/s | 31 kB 00:00 2024-03-06T07:16:16.493 INFO:teuthology.orchestra.run.smithi001.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 1.7 MB/s | 163 kB 00:00 2024-03-06T07:16:16.576 INFO:teuthology.orchestra.run.smithi001.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 1.5 MB/s | 127 kB 00:00 2024-03-06T07:16:16.627 INFO:teuthology.orchestra.run.smithi001.stdout:(17/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 2.6 MB/s | 747 kB 00:00 2024-03-06T07:16:16.777 INFO:teuthology.orchestra.run.smithi001.stdout:(18/176): libradosstriper1-19.0.0-1739.g1bc250c 2.4 MB/s | 501 kB 00:00 2024-03-06T07:16:16.860 INFO:teuthology.orchestra.run.smithi001.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 544 kB/s | 45 kB 00:00 2024-03-06T07:16:16.944 INFO:teuthology.orchestra.run.smithi001.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 1.5 MB/s | 131 kB 00:00 2024-03-06T07:16:17.007 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 582 kB/s | 19 MB 00:33 2024-03-06T07:16:17.028 INFO:teuthology.orchestra.run.smithi001.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 1.9 MB/s | 162 kB 00:00 2024-03-06T07:16:17.194 INFO:teuthology.orchestra.run.smithi001.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 1.9 MB/s | 321 kB 00:00 2024-03-06T07:16:17.353 INFO:teuthology.orchestra.run.smithi001.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 1.8 MB/s | 298 kB 00:00 2024-03-06T07:16:17.445 INFO:teuthology.orchestra.run.smithi001.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 1.1 MB/s | 99 kB 00:00 2024-03-06T07:16:17.528 INFO:teuthology.orchestra.run.smithi001.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 1.0 MB/s | 85 kB 00:00 2024-03-06T07:16:18.387 INFO:teuthology.orchestra.run.smithi001.stdout:(26/176): ceph-test-19.0.0-1739.g1bc250c7.el9.x 12 MB/s | 47 MB 00:03 2024-03-06T07:16:18.463 INFO:teuthology.orchestra.run.smithi001.stdout:(27/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 2.8 MB/s | 5.1 MB 00:01 2024-03-06T07:16:18.488 INFO:teuthology.orchestra.run.smithi001.stdout:(28/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 1.7 MB/s | 171 kB 00:00 2024-03-06T07:16:18.513 INFO:teuthology.orchestra.run.smithi001.stdout:(29/176): ceph-grafana-dashboards-19.0.0-1739.g 628 kB/s | 31 kB 00:00 2024-03-06T07:16:18.538 INFO:teuthology.orchestra.run.smithi001.stdout:(30/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 2.7 MB/s | 138 kB 00:00 2024-03-06T07:16:18.605 INFO:teuthology.orchestra.run.smithi001.stdout:(31/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 2.9 MB/s | 3.1 MB 00:01 2024-03-06T07:16:18.747 INFO:teuthology.orchestra.run.smithi001.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 1.7 MB/s | 248 kB 00:00 2024-03-06T07:16:18.831 INFO:teuthology.orchestra.run.smithi001.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 594 kB/s | 49 kB 00:00 2024-03-06T07:16:18.915 INFO:teuthology.orchestra.run.smithi001.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 184 kB/s | 15 kB 00:00 2024-03-06T07:16:18.956 INFO:teuthology.orchestra.run.smithi001.stdout:(35/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 4.0 MB/s | 1.8 MB 00:00 2024-03-06T07:16:19.073 INFO:teuthology.orchestra.run.smithi001.stdout:(36/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 1.6 MB/s | 264 kB 00:00 2024-03-06T07:16:19.157 INFO:teuthology.orchestra.run.smithi001.stdout:(37/176): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-03-06T07:16:19.182 INFO:teuthology.orchestra.run.smithi001.stdout:(38/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 168 kB/s | 18 kB 00:00 2024-03-06T07:16:19.224 INFO:teuthology.orchestra.run.smithi001.stdout:(39/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 2.7 MB/s | 748 kB 00:00 2024-03-06T07:16:19.541 INFO:teuthology.orchestra.run.smithi001.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 402 kB/s | 106 kB 00:00 2024-03-06T07:16:19.591 INFO:teuthology.orchestra.run.smithi001.stdout:(41/176): automake-1.16.2-8.el9.noarch.rpm 1.5 MB/s | 662 kB 00:00 2024-03-06T07:16:19.617 INFO:teuthology.orchestra.run.smithi001.stdout:(42/176): bison-3.7.4-5.el9.x86_64.rpm 2.1 MB/s | 920 kB 00:00 2024-03-06T07:16:19.642 INFO:teuthology.orchestra.run.smithi001.stdout:(43/176): elfutils-libelf-devel-0.190-2.el9.x86 204 kB/s | 20 kB 00:00 2024-03-06T07:16:19.667 INFO:teuthology.orchestra.run.smithi001.stdout:(44/176): flex-2.6.4-9.el9.x86_64.rpm 4.0 MB/s | 307 kB 00:00 2024-03-06T07:16:19.692 INFO:teuthology.orchestra.run.smithi001.stdout:(45/176): flexiblas-3.0.4-8.el9.x86_64.rpm 640 kB/s | 32 kB 00:00 2024-03-06T07:16:19.776 INFO:teuthology.orchestra.run.smithi001.stdout:(46/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 22 MB/s | 3.0 MB 00:00 2024-03-06T07:16:19.793 INFO:teuthology.orchestra.run.smithi001.stdout:(47/176): flexiblas-openblas-openmp-3.0.4-8.el9 138 kB/s | 17 kB 00:00 2024-03-06T07:16:19.810 INFO:teuthology.orchestra.run.smithi001.stdout:(48/176): libaio-devel-0.3.111-13.el9.x86_64.rp 88 kB/s | 10 kB 00:00 2024-03-06T07:16:19.843 INFO:teuthology.orchestra.run.smithi001.stdout:(49/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-06T07:16:19.868 INFO:teuthology.orchestra.run.smithi001.stdout:(50/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 759 kB/s | 44 kB 00:00 2024-03-06T07:16:19.894 INFO:teuthology.orchestra.run.smithi001.stdout:(51/176): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-03-06T07:16:19.919 INFO:teuthology.orchestra.run.smithi001.stdout:(52/176): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 578 kB 00:00 2024-03-06T07:16:19.936 INFO:teuthology.orchestra.run.smithi001.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-06T07:16:19.953 INFO:teuthology.orchestra.run.smithi001.stdout:(54/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 422 kB/s | 24 kB 00:00 2024-03-06T07:16:19.978 INFO:teuthology.orchestra.run.smithi001.stdout:(55/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-06T07:16:20.003 INFO:teuthology.orchestra.run.smithi001.stdout:(56/176): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 187 kB 00:00 2024-03-06T07:16:20.020 INFO:teuthology.orchestra.run.smithi001.stdout:(57/176): openblas-0.3.21-2.el9.x86_64.rpm 831 kB/s | 34 kB 00:00 2024-03-06T07:16:20.104 INFO:teuthology.orchestra.run.smithi001.stdout:(58/176): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-06T07:16:20.146 INFO:teuthology.orchestra.run.smithi001.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 12 MB/s | 1.5 MB 00:00 2024-03-06T07:16:20.196 INFO:teuthology.orchestra.run.smithi001.stdout:(60/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 16 MB/s | 3.0 MB 00:00 2024-03-06T07:16:20.221 INFO:teuthology.orchestra.run.smithi001.stdout:(61/176): postgresql-contrib-13.11-1.el9.x86_64 6.9 MB/s | 811 kB 00:00 2024-03-06T07:16:20.246 INFO:teuthology.orchestra.run.smithi001.stdout:(62/176): postgresql-private-libs-13.11-1.el9.x 1.4 MB/s | 134 kB 00:00 2024-03-06T07:16:20.305 INFO:teuthology.orchestra.run.smithi001.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-03-06T07:16:20.438 INFO:teuthology.orchestra.run.smithi001.stdout:(64/176): postgresql-server-13.11-1.el9.x86_64. 26 MB/s | 5.7 MB 00:00 2024-03-06T07:16:20.464 INFO:teuthology.orchestra.run.smithi001.stdout:(65/176): python3-attrs-20.3.0-7.el9.noarch.rpm 357 kB/s | 78 kB 00:00 2024-03-06T07:16:20.581 INFO:teuthology.orchestra.run.smithi001.stdout:(66/176): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-06T07:16:20.606 INFO:teuthology.orchestra.run.smithi001.stdout:(67/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.3 MB/s | 227 kB 00:00 2024-03-06T07:16:20.631 INFO:teuthology.orchestra.run.smithi001.stdout:(68/176): python3-jsonpatch-1.21-16.el9.noarch. 148 kB/s | 24 kB 00:00 2024-03-06T07:16:20.657 INFO:teuthology.orchestra.run.smithi001.stdout:(69/176): python3-jsonpointer-2.0-4.el9.noarch. 241 kB/s | 18 kB 00:00 2024-03-06T07:16:20.682 INFO:teuthology.orchestra.run.smithi001.stdout:(70/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-06T07:16:20.707 INFO:teuthology.orchestra.run.smithi001.stdout:(71/176): python3-markupsafe-1.1.1-12.el9.x86_6 420 kB/s | 32 kB 00:00 2024-03-06T07:16:20.807 INFO:teuthology.orchestra.run.smithi001.stdout:(72/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-06T07:16:20.832 INFO:teuthology.orchestra.run.smithi001.stdout:(73/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-06T07:16:20.858 INFO:teuthology.orchestra.run.smithi001.stdout:(74/176): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-06T07:16:20.883 INFO:teuthology.orchestra.run.smithi001.stdout:(75/176): python3-packaging-20.9-5.el9.noarch.r 922 kB/s | 69 kB 00:00 2024-03-06T07:16:20.908 INFO:teuthology.orchestra.run.smithi001.stdout:(76/176): python3-prettytable-0.7.2-27.el9.noar 825 kB/s | 41 kB 00:00 2024-03-06T07:16:20.933 INFO:teuthology.orchestra.run.smithi001.stdout:(77/176): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-06T07:16:20.959 INFO:teuthology.orchestra.run.smithi001.stdout:(78/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-06T07:16:20.984 INFO:teuthology.orchestra.run.smithi001.stdout:(79/176): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-06T07:16:21.009 INFO:teuthology.orchestra.run.smithi001.stdout:(80/176): python3-pytz-2021.1-5.el9.noarch.rpm 631 kB/s | 47 kB 00:00 2024-03-06T07:16:21.034 INFO:teuthology.orchestra.run.smithi001.stdout:(81/176): python3-requests-oauthlib-1.3.0-12.el 573 kB/s | 43 kB 00:00 2024-03-06T07:16:21.101 INFO:teuthology.orchestra.run.smithi001.stdout:(82/176): python3-toml-0.10.2-6.el9.noarch.rpm 422 kB/s | 39 kB 00:00 2024-03-06T07:16:21.435 INFO:teuthology.orchestra.run.smithi001.stdout:(83/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-03-06T07:16:21.460 INFO:teuthology.orchestra.run.smithi001.stdout:(84/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 97 kB/s | 41 kB 00:00 2024-03-06T07:16:21.485 INFO:teuthology.orchestra.run.smithi001.stdout:(85/176): qatlib-23.11.0-1.el9.x86_64.rpm 567 kB/s | 215 kB 00:00 2024-03-06T07:16:21.510 INFO:teuthology.orchestra.run.smithi001.stdout:(86/176): qatlib-service-23.11.0-1.el9.x86_64.r 683 kB/s | 34 kB 00:00 2024-03-06T07:16:21.535 INFO:teuthology.orchestra.run.smithi001.stdout:(87/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 583 kB/s | 43 kB 00:00 2024-03-06T07:16:21.561 INFO:teuthology.orchestra.run.smithi001.stdout:(88/176): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-06T07:16:21.586 INFO:teuthology.orchestra.run.smithi001.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2024-03-06T07:16:21.611 INFO:teuthology.orchestra.run.smithi001.stdout:(90/176): uuid-1.6.2-55.el9.x86_64.rpm 752 kB/s | 56 kB 00:00 2024-03-06T07:16:21.636 INFO:teuthology.orchestra.run.smithi001.stdout:(91/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 832 kB/s | 63 kB 00:00 2024-03-06T07:16:21.662 INFO:teuthology.orchestra.run.smithi001.stdout:(92/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 586 kB/s | 44 kB 00:00 2024-03-06T07:16:21.687 INFO:teuthology.orchestra.run.smithi001.stdout:(93/176): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-06T07:16:21.712 INFO:teuthology.orchestra.run.smithi001.stdout:(94/176): jq-1.6-16.el9.x86_64.rpm 2.4 MB/s | 186 kB 00:00 2024-03-06T07:16:21.746 INFO:teuthology.orchestra.run.smithi001.stdout:(95/176): libgfortran-11.4.1-3.el9.x86_64.rpm 9.4 MB/s | 802 kB 00:00 2024-03-06T07:16:21.780 INFO:teuthology.orchestra.run.smithi001.stdout:(96/176): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 194 kB 00:00 2024-03-06T07:16:21.805 INFO:teuthology.orchestra.run.smithi001.stdout:(97/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.3 MB/s | 217 kB 00:00 2024-03-06T07:16:21.830 INFO:teuthology.orchestra.run.smithi001.stdout:(98/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.8 MB/s | 241 kB 00:00 2024-03-06T07:16:21.888 INFO:teuthology.orchestra.run.smithi001.stdout:(99/176): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-06T07:16:21.914 INFO:teuthology.orchestra.run.smithi001.stdout:(100/176): python3-ply-3.11-14.el9.noarch.rpm 951 kB/s | 103 kB 00:00 2024-03-06T07:16:21.947 INFO:teuthology.orchestra.run.smithi001.stdout:(101/176): python3-pycparser-2.20-6.el9.noarch. 1.0 MB/s | 124 kB 00:00 2024-03-06T07:16:21.973 INFO:teuthology.orchestra.run.smithi001.stdout:(102/176): python3-requests-2.25.1-8.el9.noarch 1.3 MB/s | 113 kB 00:00 2024-03-06T07:16:22.000 INFO:teuthology.orchestra.run.smithi001.stdout:(103/176): python3-urllib3-1.26.5-5.el9.noarch. 3.7 MB/s | 187 kB 00:00 2024-03-06T07:16:22.123 INFO:teuthology.orchestra.run.smithi001.stdout:(104/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.4 MB/s | 245 kB 00:00 2024-03-06T07:16:22.599 INFO:teuthology.orchestra.run.smithi001.stdout:(105/176): lua-devel-5.4.4-4.el9.x86_64.rpm 36 kB/s | 22 kB 00:00 2024-03-06T07:16:22.674 INFO:teuthology.orchestra.run.smithi001.stdout:(106/176): abseil-cpp-20211102.0-3.el9.x86_64.r 996 kB/s | 548 kB 00:00 2024-03-06T07:16:22.999 INFO:teuthology.orchestra.run.smithi001.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 60 kB/s | 19 kB 00:00 2024-03-06T07:16:23.049 INFO:teuthology.orchestra.run.smithi001.stdout:(108/176): gperftools-libs-2.9.1-2.el9.x86_64.r 686 kB/s | 309 kB 00:00 2024-03-06T07:16:23.233 INFO:teuthology.orchestra.run.smithi001.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 142 kB/s | 26 kB 00:00 2024-03-06T07:16:23.325 INFO:teuthology.orchestra.run.smithi001.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 529 kB/s | 49 kB 00:00 2024-03-06T07:16:23.425 INFO:teuthology.orchestra.run.smithi001.stdout:(111/176): libunwind-1.6.2-1.el9.x86_64.rpm 675 kB/s | 67 kB 00:00 2024-03-06T07:16:23.525 INFO:teuthology.orchestra.run.smithi001.stdout:(112/176): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-03-06T07:16:23.701 INFO:teuthology.orchestra.run.smithi001.stdout:(113/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-03-06T07:16:23.734 INFO:teuthology.orchestra.run.smithi001.stdout:(114/176): protobuf-compiler-3.14.0-13.el9.x86_ 497 kB/s | 863 kB 00:01 2024-03-06T07:16:23.801 INFO:teuthology.orchestra.run.smithi001.stdout:(115/176): python3-asyncssh-2.13.2-4.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-03-06T07:16:23.841 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 1.8 MB/s | 6.1 MB 00:03 2024-03-06T07:16:23.893 INFO:teuthology.orchestra.run.smithi001.stdout:(116/176): python3-cachetools-4.2.4-1.el9.noarc 351 kB/s | 32 kB 00:00 2024-03-06T07:16:23.985 INFO:teuthology.orchestra.run.smithi001.stdout:(117/176): python3-certifi-2021.10.8-2.el9.noar 164 kB/s | 15 kB 00:00 2024-03-06T07:16:24.035 INFO:teuthology.orchestra.run.smithi001.stdout:(118/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 145 kB/s | 43 kB 00:00 2024-03-06T07:16:24.085 INFO:teuthology.orchestra.run.smithi001.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-03-06T07:16:24.211 INFO:teuthology.orchestra.run.smithi001.stdout:(120/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-03-06T07:16:24.336 INFO:teuthology.orchestra.run.smithi001.stdout:(121/176): python3-colorama-0.4.4-12.el9.noarch 283 kB/s | 35 kB 00:00 2024-03-06T07:16:24.470 INFO:teuthology.orchestra.run.smithi001.stdout:(122/176): python3-google-auth-2.28.0-1.el9.noa 1.5 MB/s | 203 kB 00:00 2024-03-06T07:16:24.553 INFO:teuthology.orchestra.run.smithi001.stdout:(123/176): python3-cherrypy-18.6.1-2.el9.noarch 692 kB/s | 358 kB 00:00 2024-03-06T07:16:24.687 INFO:teuthology.orchestra.run.smithi001.stdout:(124/176): python3-grpcio-1.46.7-10.el9.x86_64. 9.4 MB/s | 2.0 MB 00:00 2024-03-06T07:16:24.737 INFO:teuthology.orchestra.run.smithi001.stdout:(125/176): python3-grpcio-tools-1.46.7-10.el9.x 787 kB/s | 144 kB 00:00 2024-03-06T07:16:24.779 INFO:teuthology.orchestra.run.smithi001.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-03-06T07:16:24.829 INFO:teuthology.orchestra.run.smithi001.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-03-06T07:16:24.871 INFO:teuthology.orchestra.run.smithi001.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-03-06T07:16:24.922 INFO:teuthology.orchestra.run.smithi001.stdout:(129/176): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-03-06T07:16:24.963 INFO:teuthology.orchestra.run.smithi001.stdout:(130/176): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-03-06T07:16:25.064 INFO:teuthology.orchestra.run.smithi001.stdout:(131/176): python3-logutils-0.3.5-21.el9.noarch 462 kB/s | 46 kB 00:00 2024-03-06T07:16:25.164 INFO:teuthology.orchestra.run.smithi001.stdout:(132/176): python3-more-itertools-8.12.0-2.el9. 787 kB/s | 79 kB 00:00 2024-03-06T07:16:25.265 INFO:teuthology.orchestra.run.smithi001.stdout:(133/176): python3-natsort-7.1.1-5.el9.noarch.r 578 kB/s | 58 kB 00:00 2024-03-06T07:16:25.365 INFO:teuthology.orchestra.run.smithi001.stdout:(134/176): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-06T07:16:25.367 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 94 kB/s | 15 kB 00:00 2024-03-06T07:16:25.473 INFO:teuthology.orchestra.run.smithi001.stdout:(135/176): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-03-06T07:16:25.574 INFO:teuthology.orchestra.run.smithi001.stdout:(136/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 901 kB/s | 90 kB 00:00 2024-03-06T07:16:25.641 INFO:teuthology.orchestra.run.smithi001.stdout:(137/176): python3-kubernetes-26.1.0-2.el9.noar 1.4 MB/s | 1.0 MB 00:00 2024-03-06T07:16:25.749 INFO:teuthology.orchestra.run.smithi001.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 161 kB/s | 28 kB 00:00 2024-03-06T07:16:25.775 INFO:teuthology.orchestra.run.smithi001.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 230 kB/s | 31 kB 00:00 2024-03-06T07:16:25.850 INFO:teuthology.orchestra.run.smithi001.stdout:(140/176): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-03-06T07:16:25.875 INFO:teuthology.orchestra.run.smithi001.stdout:(141/176): python3-rsa-4.9-2.el9.noarch.rpm 591 kB/s | 59 kB 00:00 2024-03-06T07:16:25.951 INFO:teuthology.orchestra.run.smithi001.stdout:(142/176): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-03-06T07:16:25.976 INFO:teuthology.orchestra.run.smithi001.stdout:(143/176): python3-typing-extensions-4.4.0-2.el 508 kB/s | 51 kB 00:00 2024-03-06T07:16:26.043 INFO:teuthology.orchestra.run.smithi001.stdout:(144/176): libarrow-9.0.0-10.el9.x86_64.rpm 1.5 MB/s | 4.4 MB 00:03 2024-03-06T07:16:26.068 INFO:teuthology.orchestra.run.smithi001.stdout:(145/176): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-06T07:16:26.093 INFO:teuthology.orchestra.run.smithi001.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 765 kB/s | 90 kB 00:00 2024-03-06T07:16:26.160 INFO:teuthology.orchestra.run.smithi001.stdout:(147/176): python3-zc-lockfile-2.0-10.el9.noarc 218 kB/s | 20 kB 00:00 2024-03-06T07:16:26.210 INFO:teuthology.orchestra.run.smithi001.stdout:(148/176): python3-werkzeug-2.0.3-3.el9.1.noarc 2.5 MB/s | 427 kB 00:00 2024-03-06T07:16:26.269 INFO:teuthology.orchestra.run.smithi001.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-06T07:16:26.403 INFO:teuthology.orchestra.run.smithi001.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2024-03-06T07:16:26.436 INFO:teuthology.orchestra.run.smithi001.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-03-06T07:16:26.545 INFO:teuthology.orchestra.run.smithi001.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 21 MB/s | 2.3 MB 00:00 2024-03-06T07:16:26.637 INFO:teuthology.orchestra.run.smithi001.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 20 MB/s | 1.8 MB 00:00 2024-03-06T07:16:26.662 INFO:teuthology.orchestra.run.smithi001.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.6 MB/s | 41 kB 00:00 2024-03-06T07:16:26.695 INFO:teuthology.orchestra.run.smithi001.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 7.2 MB/s | 247 kB 00:00 2024-03-06T07:16:26.737 INFO:teuthology.orchestra.run.smithi001.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 69 kB 00:00 2024-03-06T07:16:26.763 INFO:teuthology.orchestra.run.smithi001.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.4 MB/s | 35 kB 00:00 2024-03-06T07:16:26.805 INFO:teuthology.orchestra.run.smithi001.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 191 kB 00:00 2024-03-06T07:16:26.847 INFO:teuthology.orchestra.run.smithi001.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-03-06T07:16:26.905 INFO:teuthology.orchestra.run.smithi001.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-06T07:16:26.947 INFO:teuthology.orchestra.run.smithi001.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 10 MB/s | 433 kB 00:00 2024-03-06T07:16:26.981 INFO:teuthology.orchestra.run.smithi001.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 660 kB/s | 22 kB 00:00 2024-03-06T07:16:27.014 INFO:teuthology.orchestra.run.smithi001.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 107 kB 00:00 2024-03-06T07:16:27.048 INFO:teuthology.orchestra.run.smithi001.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 4.5 MB/s | 154 kB 00:00 2024-03-06T07:16:27.082 INFO:teuthology.orchestra.run.smithi001.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 4.0 MB/s | 135 kB 00:00 2024-03-06T07:16:27.115 INFO:teuthology.orchestra.run.smithi001.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.9 MB/s | 63 kB 00:00 2024-03-06T07:16:27.149 INFO:teuthology.orchestra.run.smithi001.stdout:(167/176): libuuid-2.37.4-18.el9.x86_64.rpm 831 kB/s | 28 kB 00:00 2024-03-06T07:16:27.241 INFO:teuthology.orchestra.run.smithi001.stdout:(168/176): openssl-3.0.7-27.el9.x86_64.rpm 13 MB/s | 1.2 MB 00:00 2024-03-06T07:16:27.325 INFO:teuthology.orchestra.run.smithi001.stdout:(169/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 26 MB/s | 2.2 MB 00:00 2024-03-06T07:16:27.408 INFO:teuthology.orchestra.run.smithi001.stdout:(170/176): util-linux-2.37.4-18.el9.x86_64.rpm 26 MB/s | 2.2 MB 00:00 2024-03-06T07:16:27.450 INFO:teuthology.orchestra.run.smithi001.stdout:(171/176): util-linux-core-2.37.4-18.el9.x86_64 10 MB/s | 438 kB 00:00 2024-03-06T07:16:27.509 INFO:teuthology.orchestra.run.smithi001.stdout:(172/176): librados2-19.0.0-1739.g1bc250c7.el9. 2.6 MB/s | 3.4 MB 00:01 2024-03-06T07:16:27.534 INFO:teuthology.orchestra.run.smithi001.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 359 kB/s | 30 kB 00:00 2024-03-06T07:16:27.559 INFO:teuthology.orchestra.run.smithi001.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-06T07:16:27.593 INFO:teuthology.orchestra.run.smithi001.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-03-06T07:16:27.993 INFO:teuthology.orchestra.run.smithi001.stdout:(176/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 1.8 MB/s | 3.1 MB 00:01 2024-03-06T07:16:27.998 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:16:27.998 INFO:teuthology.orchestra.run.smithi001.stdout:Total 11 MB/s | 224 MB 00:20 2024-03-06T07:16:28.045 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-06T07:16:28.383 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-06T07:16:28.384 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-06T07:16:30.279 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 21 MB 00:04 2024-03-06T07:16:30.420 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-06T07:16:30.420 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-06T07:16:35.276 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 10 kB/s | 1.7 kB 00:00 2024-03-06T07:16:37.814 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-06T07:16:37.815 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:16:37.816 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:16:37.828 INFO:teuthology.orchestra.run.smithi098.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:16:37.829 INFO:teuthology.orchestra.run.smithi098.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:16:37.830 INFO:teuthology.orchestra.run.smithi098.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:16:37.831 INFO:teuthology.orchestra.run.smithi098.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:16:37.833 INFO:teuthology.orchestra.run.smithi098.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:16:37.835 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:16:37.836 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:16:37.889 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:16:37.916 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:16:37.917 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-06T07:16:37.939 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-06T07:16:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-06T07:16:37.940 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-06T07:16:37.940 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-06T07:16:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:16:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:16:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:16:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:16:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:16:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:16:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:16:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:16:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:16:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:16:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:16:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:16:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:16:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:16:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:16:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:16:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:16:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:16:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:16:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:16:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:16:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:16:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:16:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:16:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:16:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:16:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:16:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:16:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:16:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:16:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:16:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:16:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:16:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:16:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:16:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:16:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:16:37.943 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-03-06T07:16:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:16:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:16:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:16:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:16:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:16:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:16:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:16:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:16:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:16:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:16:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:16:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:16:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:16:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:16:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:16:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:16:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:16:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:16:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:16:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:16:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:16:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:16:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:16:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:16:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:16:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:16:37.945 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-06T07:16:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:16:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:16:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:16:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:16:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:16:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:16:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:16:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:16:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:16:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:16:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:16:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:16:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:16:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:16:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:16:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:16:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:16:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:16:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:16:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:16:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:16:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:16:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:16:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:16:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:16:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:16:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:16:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:16:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:16:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:16:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:16:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:16:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:16:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:16:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:16:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:16:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:16:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:16:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:16:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:16:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:16:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:16:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:16:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:16:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:16:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:16:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:16:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:16:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:16:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:16:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:16:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:16:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:16:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:16:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:16:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:16:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:16:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:16:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:16:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:16:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:16:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:16:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:16:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:16:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:16:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:16:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:16:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:16:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:16:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:16:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:16:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:16:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:16:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:16:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:16:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:16:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:16:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:16:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:16:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:16:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:16:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:16:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:16:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:16:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:16:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:16:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:16:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:16:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:16:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:16:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:16:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:16:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:16:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:16:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:16:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:16:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:16:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:16:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:16:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:16:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:16:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:16:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:16:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:16:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:16:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:16:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:16:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:16:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:16:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:16:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:16:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:16:37.954 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-03-06T07:16:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:16:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:16:37.954 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-06T07:16:37.954 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-06T07:16:37.954 INFO:teuthology.orchestra.run.smithi098.stdout:Install 150 Packages 2024-03-06T07:16:37.954 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 26 Packages 2024-03-06T07:16:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:16:37.955 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:16:37.955 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 224 M 2024-03-06T07:16:37.955 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-06T07:16:37.980 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:16:38.033 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:16:38.072 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:16:38.127 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:16:38.177 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:16:38.209 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:16:38.254 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:16:38.604 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:16:38.652 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:16:38.764 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:16:38.786 INFO:teuthology.orchestra.run.smithi098.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 14 kB/s | 6.6 kB 00:00 2024-03-06T07:16:38.789 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:16:38.865 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:16:38.898 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:16:39.061 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:16:39.116 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:16:39.144 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:16:39.249 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:16:39.278 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:16:39.343 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:16:39.382 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:16:39.404 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:16:39.420 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:16:39.562 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:16:39.594 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:16:39.634 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:16:39.826 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:16:40.086 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:16:40.146 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:16:40.263 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:16:40.323 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:16:40.523 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:16:40.631 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:16:40.727 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:16:40.761 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:16:40.834 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:16:40.885 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:16:40.915 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:16:40.950 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:16:40.991 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:16:41.070 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:16:41.115 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:16:41.158 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:16:41.190 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:16:41.218 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:16:41.291 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:16:41.448 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:16:41.492 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:16:41.521 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:16:41.593 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:16:41.635 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:16:41.695 INFO:teuthology.orchestra.run.smithi098.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 306 kB/s | 888 kB 00:02 2024-03-06T07:16:41.713 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:16:41.858 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:16:41.915 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:16:42.170 INFO:teuthology.orchestra.run.smithi098.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 306 kB/s | 145 kB 00:00 2024-03-06T07:16:42.467 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:16:42.544 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:16:42.570 INFO:teuthology.orchestra.run.smithi098.stdout:(4/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 1.3 MB/s | 5.3 MB 00:04 2024-03-06T07:16:42.642 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:16:42.677 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:16:42.685 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:16:42.725 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:16:42.762 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:16:42.794 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:16:42.904 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:16:42.971 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:16:43.227 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:16:43.388 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:16:43.490 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:16:43.535 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:16:43.620 INFO:teuthology.orchestra.run.smithi098.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 1.4 MB/s | 1.5 MB 00:01 2024-03-06T07:16:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:16:43.838 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:16:43.904 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:16:43.977 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:16:44.013 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:16:44.050 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:16:44.098 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:16:44.141 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:16:44.185 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:16:44.226 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:16:44.377 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:16:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:16:44.455 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:16:44.496 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:16:44.603 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:16:44.637 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:16:44.677 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:16:44.702 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:16:44.727 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:16:44.806 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:16:44.933 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:16:45.040 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:16:45.085 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:16:45.144 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:16:45.200 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:16:45.391 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:16:45.439 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:16:45.584 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:16:45.629 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:16:45.662 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:16:45.694 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:16:45.729 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:16:45.759 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:16:45.788 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:16:45.822 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:16:45.861 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:16:45.893 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:16:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:16:46.002 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:16:46.033 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:16:46.102 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:16:46.107 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:16:46.142 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:16:46.234 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:16:46.324 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:16:46.382 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:16:46.413 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:16:46.461 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:16:46.561 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:16:46.616 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:16:47.403 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:16:47.453 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:16:47.487 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:16:47.536 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:16:47.628 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:16:47.679 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:16:47.865 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:16:47.894 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:16:47.977 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:16:48.554 INFO:teuthology.orchestra.run.smithi098.stdout:(6/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 973 kB/s | 4.7 MB 00:04 2024-03-06T07:16:48.612 INFO:teuthology.orchestra.run.smithi098.stdout:(7/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 367 kB/s | 2.3 MB 00:06 2024-03-06T07:16:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:16:48.770 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:16:50.049 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:16:50.130 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:16:50.157 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:16:50.484 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:16:50.512 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:16:50.586 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:16:50.616 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:16:50.653 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:16:51.129 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:16:51.190 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:16:51.261 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:16:51.292 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:16:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:16:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:16:54.202 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:16:54.241 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:16:54.329 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:16:54.329 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:16:54.329 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:16:54.329 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:16:54.376 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:17:06.211 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:17:06.212 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /sys 2024-03-06T07:17:06.212 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /proc 2024-03-06T07:17:06.212 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /mnt 2024-03-06T07:17:06.212 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /var/tmp 2024-03-06T07:17:06.212 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /home 2024-03-06T07:17:06.213 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /root 2024-03-06T07:17:06.213 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /tmp 2024-03-06T07:17:06.213 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:17:06.380 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:17:06.396 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:17:06.396 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:17:06.396 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:17:06.396 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:17:07.925 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:17:07.950 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:17:07.951 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:17:07.951 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:17:07.951 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:17:07.951 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:17:07.951 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:17:08.176 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:17:08.198 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:17:08.198 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:17:08.198 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:17:08.198 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:17:08.198 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:17:08.198 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:17:08.660 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:17:08.683 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:17:08.683 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:17:08.683 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:17:08.683 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:17:08.683 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:17:08.683 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:17:08.778 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:17:09.213 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:17:09.244 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:17:09.267 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:17:09.321 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:17:09.350 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:17:09.590 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:17:09.617 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:17:10.628 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:17:10.659 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:17:10.768 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:17:10.929 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:17:10.953 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:17:10.976 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:17:10.976 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:17:10.976 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:17:10.976 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:17:10.976 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:17:10.976 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:17:11.020 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:17:11.031 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:17:11.085 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:17:11.990 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:17:12.022 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:17:12.044 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:17:12.045 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:17:12.045 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:17:12.045 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:17:12.045 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:17:12.045 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:17:12.089 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:17:12.111 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:17:12.111 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:17:12.111 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:17:12.111 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:17:12.165 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:17:12.187 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:17:12.187 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:17:12.187 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:17:12.187 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:17:12.187 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:17:12.187 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:17:12.527 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:17:12.549 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:17:12.550 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:17:12.550 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:17:12.550 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:17:12.550 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:17:12.550 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:17:14.960 INFO:teuthology.orchestra.run.smithi098.stdout:(8/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x86 652 kB/s | 17 MB 00:26 2024-03-06T07:17:15.061 INFO:teuthology.orchestra.run.smithi098.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 252 kB/s | 25 kB 00:00 2024-03-06T07:17:16.277 INFO:teuthology.orchestra.run.smithi098.stdout:(10/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 362 kB/s | 9.8 MB 00:27 2024-03-06T07:17:16.911 INFO:teuthology.orchestra.run.smithi098.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 348 kB/s | 220 kB 00:00 2024-03-06T07:17:17.044 INFO:teuthology.orchestra.run.smithi098.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 203 kB/s | 27 kB 00:00 2024-03-06T07:17:17.164 INFO:teuthology.orchestra.run.smithi098.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 268 kB/s | 31 kB 00:00 2024-03-06T07:17:17.221 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:17:17.262 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:17:17.292 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:17:17.319 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:17:17.521 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:17:17.647 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:17:17.684 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:17:17.744 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:17:17.792 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:17:17.867 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:17:17.911 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:17:17.954 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:17:17.985 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:17:18.014 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:17:18.079 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:17:18.128 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:17:18.162 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:17:18.785 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:17:18.819 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:17:18.838 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:17:19.186 INFO:teuthology.orchestra.run.smithi098.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 369 kB/s | 747 kB 00:02 2024-03-06T07:17:19.687 INFO:teuthology.orchestra.run.smithi098.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 326 kB/s | 163 kB 00:00 2024-03-06T07:17:19.720 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:17:19.788 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:17:19.831 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:17:19.927 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:17:19.959 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:17:19.992 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:17:20.018 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:17:20.044 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:17:20.044 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:17:20.087 INFO:teuthology.orchestra.run.smithi098.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 317 kB/s | 127 kB 00:00 2024-03-06T07:17:20.128 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:17:20.156 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:17:20.187 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:17:20.187 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:17:20.272 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:17:20.298 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:17:20.332 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:17:20.356 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:17:20.392 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:17:20.424 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:17:20.458 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:17:20.505 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:17:20.537 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:17:20.562 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:17:20.588 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:17:20.613 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:17:20.638 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:17:20.645 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:17:20.645 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:17:20.824 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:17:20.824 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:17:20.907 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:17:20.939 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:17:20.966 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:17:21.001 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:17:21.027 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:17:21.104 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:17:21.471 INFO:teuthology.orchestra.run.smithi098.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 362 kB/s | 501 kB 00:01 2024-03-06T07:17:25.839 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:17:25.840 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:17:25.840 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:17:25.840 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:17:25.840 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:17:25.840 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:17:25.840 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:17:25.840 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:17:25.840 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:17:25.840 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:17:25.840 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:17:25.840 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:17:25.840 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:17:25.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:17:25.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:17:25.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:17:25.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:17:25.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:17:25.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:17:25.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:17:25.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:17:25.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:17:25.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:17:25.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:17:25.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:17:25.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:17:25.842 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:17:25.842 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:17:25.842 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:17:25.842 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:17:25.842 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:17:25.842 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:17:25.842 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:17:25.842 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:17:25.842 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:17:25.842 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:17:25.842 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:17:25.842 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:17:25.842 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:17:25.843 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:17:25.843 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:17:25.843 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:17:25.843 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:17:25.843 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:17:25.843 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:17:25.843 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:17:25.843 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:17:25.843 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:17:25.843 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:17:25.843 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:17:25.843 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:17:25.843 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:17:25.844 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:17:25.844 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:17:25.844 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:17:25.844 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:17:25.844 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:17:25.844 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:17:25.844 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:17:25.844 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:17:25.844 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:17:25.844 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:17:25.844 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:17:25.844 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:17:25.844 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:17:25.845 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:17:25.845 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:17:25.845 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:17:25.845 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:17:25.845 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:17:25.845 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:17:25.845 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:17:25.845 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:17:25.845 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:17:25.845 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:17:25.845 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:17:25.846 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:17:25.846 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:17:25.846 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:17:25.846 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:17:25.846 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:17:25.846 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:17:25.846 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:17:25.846 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:17:25.846 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:17:25.846 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:17:25.846 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:17:25.847 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:17:25.847 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:17:25.847 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:17:25.847 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:17:25.847 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:17:25.847 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:17:25.847 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:17:25.847 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:17:25.847 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:17:25.847 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:17:25.847 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:17:25.847 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:17:25.847 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:17:25.847 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:17:25.848 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:17:25.848 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:17:25.848 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:17:25.848 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:17:25.848 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:17:25.848 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:17:25.848 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:17:25.848 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:17:25.848 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:17:25.848 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:17:25.848 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:17:25.848 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:17:25.848 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:17:25.849 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:17:25.849 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:17:25.849 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:17:25.849 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:17:25.849 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:17:25.849 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:17:25.849 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:17:25.849 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:17:25.849 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:17:25.849 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:17:25.849 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:17:25.849 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:17:25.849 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:17:25.849 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:17:25.850 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:17:25.850 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:17:25.850 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:17:25.850 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:17:25.850 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:17:25.850 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:17:25.850 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:17:25.850 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:17:25.850 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:17:25.850 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:17:25.850 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:17:25.850 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:17:25.850 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:17:25.850 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:17:25.851 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:17:25.851 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:17:25.851 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:17:25.851 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:17:25.851 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:17:25.851 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:17:25.851 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:17:25.851 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:17:25.851 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:17:25.851 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:17:25.851 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:17:25.851 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:17:25.851 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:17:25.852 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:17:25.852 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:17:25.852 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:17:25.852 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:17:25.852 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:17:25.852 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:17:25.852 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:17:25.852 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:17:25.852 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:17:25.852 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:17:25.852 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:17:25.852 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:17:25.852 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:17:25.853 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:17:25.853 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:17:25.853 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:17:25.853 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:17:25.853 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:17:25.853 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:17:25.853 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:17:25.853 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:17:25.853 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:17:25.853 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:17:25.853 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:17:25.853 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:17:25.853 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:17:25.853 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:17:25.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:17:25.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:17:25.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:17:25.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:17:25.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:17:25.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:17:25.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:17:25.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:17:25.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:17:25.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:17:25.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:17:25.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:17:25.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:17:25.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:17:25.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:17:25.855 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:17:25.855 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:17:25.855 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:17:25.855 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:17:25.855 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:17:26.845 INFO:teuthology.orchestra.run.smithi098.stdout:(18/176): ceph-common-19.0.0-1739.g1bc250c7.el9 431 kB/s | 20 MB 00:48 2024-03-06T07:17:27.045 INFO:teuthology.orchestra.run.smithi098.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 226 kB/s | 45 kB 00:00 2024-03-06T07:17:27.546 INFO:teuthology.orchestra.run.smithi098.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 263 kB/s | 131 kB 00:00 2024-03-06T07:17:28.040 INFO:teuthology.orchestra.run.smithi098.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 329 kB/s | 162 kB 00:00 2024-03-06T07:17:28.654 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:17:28.654 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:17:28.654 INFO:teuthology.orchestra.run.smithi001.stdout:Upgraded: 2024-03-06T07:17:28.654 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:17:28.654 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:17:28.654 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:17:28.654 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:17:28.654 INFO:teuthology.orchestra.run.smithi001.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:17:28.654 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:17:28.655 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:17:28.655 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:17:28.655 INFO:teuthology.orchestra.run.smithi001.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:17:28.655 INFO:teuthology.orchestra.run.smithi001.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:17:28.655 INFO:teuthology.orchestra.run.smithi001.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:17:28.655 INFO:teuthology.orchestra.run.smithi001.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:17:28.655 INFO:teuthology.orchestra.run.smithi001.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:17:28.655 INFO:teuthology.orchestra.run.smithi001.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:17:28.655 INFO:teuthology.orchestra.run.smithi001.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:17:28.655 INFO:teuthology.orchestra.run.smithi001.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.655 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.655 INFO:teuthology.orchestra.run.smithi001.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:17:28.655 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:17:28.655 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:17:28.655 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:17:28.656 INFO:teuthology.orchestra.run.smithi001.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:17:28.656 INFO:teuthology.orchestra.run.smithi001.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:17:28.656 INFO:teuthology.orchestra.run.smithi001.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:17:28.656 INFO:teuthology.orchestra.run.smithi001.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:17:28.656 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:17:28.656 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-06T07:17:28.656 INFO:teuthology.orchestra.run.smithi001.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:17:28.656 INFO:teuthology.orchestra.run.smithi001.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:17:28.656 INFO:teuthology.orchestra.run.smithi001.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:17:28.656 INFO:teuthology.orchestra.run.smithi001.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:17:28.656 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.656 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.656 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.656 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.657 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:28.657 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.657 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.657 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.657 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:28.657 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:28.657 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:28.657 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:28.657 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:28.657 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.657 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.657 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:28.657 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.657 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.658 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.658 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:28.658 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:28.658 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.658 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.658 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:28.658 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:17:28.658 INFO:teuthology.orchestra.run.smithi001.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:17:28.658 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:17:28.658 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:17:28.658 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:17:28.658 INFO:teuthology.orchestra.run.smithi001.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:17:28.658 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:17:28.658 INFO:teuthology.orchestra.run.smithi001.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:17:28.658 INFO:teuthology.orchestra.run.smithi001.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:17:28.659 INFO:teuthology.orchestra.run.smithi001.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:17:28.659 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:17:28.659 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:17:28.659 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.659 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.659 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.659 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:17:28.659 INFO:teuthology.orchestra.run.smithi001.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:17:28.659 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:17:28.659 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:17:28.659 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:17:28.659 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.659 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.659 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:17:28.659 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.660 INFO:teuthology.orchestra.run.smithi001.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:17:28.660 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:17:28.660 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:17:28.660 INFO:teuthology.orchestra.run.smithi001.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:17:28.660 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:17:28.660 INFO:teuthology.orchestra.run.smithi001.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:17:28.660 INFO:teuthology.orchestra.run.smithi001.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:17:28.660 INFO:teuthology.orchestra.run.smithi001.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:17:28.660 INFO:teuthology.orchestra.run.smithi001.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:17:28.660 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:17:28.660 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:17:28.660 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:17:28.660 INFO:teuthology.orchestra.run.smithi001.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:17:28.660 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:17:28.661 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:17:28.661 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:17:28.661 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:17:28.661 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:17:28.661 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:17:28.661 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:17:28.661 INFO:teuthology.orchestra.run.smithi001.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:17:28.661 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:17:28.661 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:17:28.661 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:17:28.661 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.661 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.661 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.661 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:17:28.661 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:17:28.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:17:28.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:17:28.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:17:28.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:17:28.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:17:28.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:17:28.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:17:28.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:17:28.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:17:28.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:17:28.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:17:28.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:17:28.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:17:28.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:17:28.663 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:17:28.663 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:17:28.663 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:17:28.663 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:17:28.663 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:17:28.663 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:17:28.663 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:17:28.663 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:17:28.663 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:17:28.663 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:17:28.663 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:17:28.663 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:17:28.663 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:17:28.663 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:17:28.663 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:17:28.664 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:17:28.664 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:17:28.664 INFO:teuthology.orchestra.run.smithi001.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:17:28.664 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:17:28.664 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:17:28.664 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:17:28.664 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:17:28.664 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:17:28.664 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:17:28.664 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.664 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.664 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:17:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:17:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:17:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:17:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:17:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:17:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:17:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:17:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:17:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:17:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:17:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:17:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:17:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:17:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:17:28.666 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:17:28.666 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:17:28.666 INFO:teuthology.orchestra.run.smithi001.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:17:28.666 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.666 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.666 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:28.666 INFO:teuthology.orchestra.run.smithi001.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:17:28.666 INFO:teuthology.orchestra.run.smithi001.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:17:28.666 INFO:teuthology.orchestra.run.smithi001.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:17:28.666 INFO:teuthology.orchestra.run.smithi001.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:17:28.666 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:17:28.666 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:17:28.667 INFO:teuthology.orchestra.run.smithi001.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:17:28.667 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:17:28.667 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-06T07:17:28.921 INFO:teuthology.orchestra.run.smithi098.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 363 kB/s | 321 kB 00:00 2024-03-06T07:17:29.288 DEBUG:teuthology.parallel:result is None 2024-03-06T07:17:29.688 INFO:teuthology.orchestra.run.smithi098.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 388 kB/s | 298 kB 00:00 2024-03-06T07:17:29.988 INFO:teuthology.orchestra.run.smithi098.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 331 kB/s | 99 kB 00:00 2024-03-06T07:17:30.205 INFO:teuthology.orchestra.run.smithi098.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 393 kB/s | 85 kB 00:00 2024-03-06T07:17:33.872 INFO:teuthology.orchestra.run.smithi098.stdout:(26/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 423 kB/s | 5.1 MB 00:12 2024-03-06T07:17:34.380 INFO:teuthology.orchestra.run.smithi098.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 336 kB/s | 171 kB 00:00 2024-03-06T07:17:34.506 INFO:teuthology.orchestra.run.smithi098.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 251 kB/s | 31 kB 00:00 2024-03-06T07:17:35.223 INFO:teuthology.orchestra.run.smithi098.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 193 kB/s | 138 kB 00:00 2024-03-06T07:17:38.356 INFO:teuthology.orchestra.run.smithi098.stdout:(30/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 388 kB/s | 3.1 MB 00:08 2024-03-06T07:17:39.431 INFO:teuthology.orchestra.run.smithi098.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 433 kB/s | 1.8 MB 00:04 2024-03-06T07:17:40.023 INFO:teuthology.orchestra.run.smithi098.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 420 kB/s | 248 kB 00:00 2024-03-06T07:17:40.223 INFO:teuthology.orchestra.run.smithi098.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 246 kB/s | 49 kB 00:00 2024-03-06T07:17:40.348 INFO:teuthology.orchestra.run.smithi098.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 123 kB/s | 15 kB 00:00 2024-03-06T07:17:40.907 INFO:teuthology.orchestra.run.smithi098.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 472 kB/s | 264 kB 00:00 2024-03-06T07:17:42.516 INFO:teuthology.orchestra.run.smithi098.stdout:(36/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 465 kB/s | 748 kB 00:01 2024-03-06T07:17:42.632 INFO:teuthology.orchestra.run.smithi098.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 156 kB/s | 18 kB 00:00 2024-03-06T07:17:42.891 INFO:teuthology.orchestra.run.smithi098.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 2.5 MB/s | 662 kB 00:00 2024-03-06T07:17:43.016 INFO:teuthology.orchestra.run.smithi098.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 7.2 MB/s | 920 kB 00:00 2024-03-06T07:17:43.050 INFO:teuthology.orchestra.run.smithi098.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-06T07:17:43.075 INFO:teuthology.orchestra.run.smithi098.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 813 kB/s | 20 kB 00:00 2024-03-06T07:17:43.117 INFO:teuthology.orchestra.run.smithi098.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-06T07:17:43.167 INFO:teuthology.orchestra.run.smithi098.stdout:(43/176): flexiblas-3.0.4-8.el9.x86_64.rpm 634 kB/s | 32 kB 00:00 2024-03-06T07:17:43.276 INFO:teuthology.orchestra.run.smithi098.stdout:(44/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-06T07:17:43.327 INFO:teuthology.orchestra.run.smithi098.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 343 kB/s | 17 kB 00:00 2024-03-06T07:17:43.360 INFO:teuthology.orchestra.run.smithi098.stdout:(46/176): libaio-devel-0.3.111-13.el9.x86_64.rp 303 kB/s | 10 kB 00:00 2024-03-06T07:17:43.394 INFO:teuthology.orchestra.run.smithi098.stdout:(47/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-06T07:17:43.427 INFO:teuthology.orchestra.run.smithi098.stdout:(48/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-06T07:17:43.469 INFO:teuthology.orchestra.run.smithi098.stdout:(49/176): librdkafka-1.6.1-102.el9.x86_64.rpm 15 MB/s | 661 kB 00:00 2024-03-06T07:17:43.511 INFO:teuthology.orchestra.run.smithi098.stdout:(50/176): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-03-06T07:17:43.545 INFO:teuthology.orchestra.run.smithi098.stdout:(51/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 730 kB/s | 24 kB 00:00 2024-03-06T07:17:43.578 INFO:teuthology.orchestra.run.smithi098.stdout:(52/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-06T07:17:43.621 INFO:teuthology.orchestra.run.smithi098.stdout:(53/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-06T07:17:43.688 INFO:teuthology.orchestra.run.smithi098.stdout:(54/176): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-06T07:17:43.738 INFO:teuthology.orchestra.run.smithi098.stdout:(55/176): openblas-0.3.21-2.el9.x86_64.rpm 684 kB/s | 34 kB 00:00 2024-03-06T07:17:43.997 INFO:teuthology.orchestra.run.smithi098.stdout:(56/176): openblas-openmp-0.3.21-2.el9.x86_64.r 18 MB/s | 4.7 MB 00:00 2024-03-06T07:17:44.139 INFO:teuthology.orchestra.run.smithi098.stdout:(57/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 21 MB/s | 3.0 MB 00:00 2024-03-06T07:17:44.273 INFO:teuthology.orchestra.run.smithi098.stdout:(58/176): postgresql-13.11-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-03-06T07:17:44.331 INFO:teuthology.orchestra.run.smithi098.stdout:(59/176): postgresql-contrib-13.11-1.el9.x86_64 14 MB/s | 811 kB 00:00 2024-03-06T07:17:44.365 INFO:teuthology.orchestra.run.smithi098.stdout:(60/176): postgresql-private-libs-13.11-1.el9.x 3.9 MB/s | 134 kB 00:00 2024-03-06T07:17:44.557 INFO:teuthology.orchestra.run.smithi098.stdout:(61/176): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2024-03-06T07:17:44.615 INFO:teuthology.orchestra.run.smithi098.stdout:(62/176): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-03-06T07:17:44.649 INFO:teuthology.orchestra.run.smithi098.stdout:(63/176): python3-attrs-20.3.0-7.el9.noarch.rpm 2.3 MB/s | 78 kB 00:00 2024-03-06T07:17:44.816 INFO:teuthology.orchestra.run.smithi098.stdout:(64/176): python3-babel-2.9.1-2.el9.noarch.rpm 35 MB/s | 5.8 MB 00:00 2024-03-06T07:17:44.851 INFO:teuthology.orchestra.run.smithi098.stdout:(65/176): python3-jinja2-2.11.3-5.el9.noarch.rp 6.7 MB/s | 227 kB 00:00 2024-03-06T07:17:44.875 INFO:teuthology.orchestra.run.smithi098.stdout:(66/176): python3-jsonpatch-1.21-16.el9.noarch. 978 kB/s | 24 kB 00:00 2024-03-06T07:17:44.900 INFO:teuthology.orchestra.run.smithi098.stdout:(67/176): python3-jsonpointer-2.0-4.el9.noarch. 727 kB/s | 18 kB 00:00 2024-03-06T07:17:44.934 INFO:teuthology.orchestra.run.smithi098.stdout:(68/176): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-06T07:17:44.959 INFO:teuthology.orchestra.run.smithi098.stdout:(69/176): python3-markupsafe-1.1.1-12.el9.x86_6 1.2 MB/s | 32 kB 00:00 2024-03-06T07:17:45.084 INFO:teuthology.orchestra.run.smithi098.stdout:(70/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-06T07:17:45.118 INFO:teuthology.orchestra.run.smithi098.stdout:(71/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 6.4 MB/s | 217 kB 00:00 2024-03-06T07:17:45.151 INFO:teuthology.orchestra.run.smithi098.stdout:(72/176): python3-oauthlib-3.1.1-5.el9.noarch.r 5.1 MB/s | 174 kB 00:00 2024-03-06T07:17:45.177 INFO:teuthology.orchestra.run.smithi098.stdout:(73/176): python3-packaging-20.9-5.el9.noarch.r 2.7 MB/s | 69 kB 00:00 2024-03-06T07:17:45.210 INFO:teuthology.orchestra.run.smithi098.stdout:(74/176): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-06T07:17:45.253 INFO:teuthology.orchestra.run.smithi098.stdout:(75/176): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-06T07:17:45.286 INFO:teuthology.orchestra.run.smithi098.stdout:(76/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.9 MB/s | 132 kB 00:00 2024-03-06T07:17:45.328 INFO:teuthology.orchestra.run.smithi098.stdout:(77/176): python3-pyasn1-modules-0.4.8-6.el9.no 4.9 MB/s | 211 kB 00:00 2024-03-06T07:17:45.362 INFO:teuthology.orchestra.run.smithi098.stdout:(78/176): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-06T07:17:45.395 INFO:teuthology.orchestra.run.smithi098.stdout:(79/176): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-06T07:17:45.754 INFO:teuthology.orchestra.run.smithi098.stdout:(80/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-06T07:17:45.788 INFO:teuthology.orchestra.run.smithi098.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-06T07:17:45.821 INFO:teuthology.orchestra.run.smithi098.stdout:(82/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.2 MB/s | 41 kB 00:00 2024-03-06T07:17:45.872 INFO:teuthology.orchestra.run.smithi098.stdout:(83/176): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 215 kB 00:00 2024-03-06T07:17:45.914 INFO:teuthology.orchestra.run.smithi098.stdout:(84/176): qatlib-service-23.11.0-1.el9.x86_64.r 809 kB/s | 34 kB 00:00 2024-03-06T07:17:45.948 INFO:teuthology.orchestra.run.smithi098.stdout:(85/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-03-06T07:17:45.990 INFO:teuthology.orchestra.run.smithi098.stdout:(86/176): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-06T07:17:46.023 INFO:teuthology.orchestra.run.smithi098.stdout:(87/176): uuid-1.6.2-55.el9.x86_64.rpm 1.7 MB/s | 56 kB 00:00 2024-03-06T07:17:46.057 INFO:teuthology.orchestra.run.smithi098.stdout:(88/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.5 MB/s | 87 kB 00:00 2024-03-06T07:17:46.091 INFO:teuthology.orchestra.run.smithi098.stdout:(89/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-06T07:17:46.124 INFO:teuthology.orchestra.run.smithi098.stdout:(90/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-06T07:17:46.158 INFO:teuthology.orchestra.run.smithi098.stdout:(91/176): fuse-2.9.9-15.el9.x86_64.rpm 2.3 MB/s | 79 kB 00:00 2024-03-06T07:17:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:(92/176): jq-1.6-16.el9.x86_64.rpm 5.5 MB/s | 186 kB 00:00 2024-03-06T07:17:46.233 INFO:teuthology.orchestra.run.smithi098.stdout:(93/176): libgfortran-11.4.1-3.el9.x86_64.rpm 19 MB/s | 802 kB 00:00 2024-03-06T07:17:46.267 INFO:teuthology.orchestra.run.smithi098.stdout:(94/176): libquadmath-11.4.1-3.el9.x86_64.rpm 5.7 MB/s | 194 kB 00:00 2024-03-06T07:17:46.309 INFO:teuthology.orchestra.run.smithi098.stdout:(95/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-06T07:17:46.343 INFO:teuthology.orchestra.run.smithi098.stdout:(96/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 7.1 MB/s | 241 kB 00:00 2024-03-06T07:17:46.402 INFO:teuthology.orchestra.run.smithi098.stdout:(97/176): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-06T07:17:46.435 INFO:teuthology.orchestra.run.smithi098.stdout:(98/176): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-06T07:17:46.469 INFO:teuthology.orchestra.run.smithi098.stdout:(99/176): python3-pycparser-2.20-6.el9.noarch.r 3.6 MB/s | 124 kB 00:00 2024-03-06T07:17:46.502 INFO:teuthology.orchestra.run.smithi098.stdout:(100/176): python3-requests-2.25.1-8.el9.noarch 3.3 MB/s | 113 kB 00:00 2024-03-06T07:17:46.536 INFO:teuthology.orchestra.run.smithi098.stdout:(101/176): python3-urllib3-1.26.5-5.el9.noarch. 5.5 MB/s | 187 kB 00:00 2024-03-06T07:17:46.678 INFO:teuthology.orchestra.run.smithi098.stdout:(102/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.7 MB/s | 245 kB 00:00 2024-03-06T07:17:46.937 INFO:teuthology.orchestra.run.smithi098.stdout:(103/176): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2024-03-06T07:17:47.537 INFO:teuthology.orchestra.run.smithi098.stdout:(104/176): protobuf-compiler-3.14.0-13.el9.x86_ 1.4 MB/s | 863 kB 00:00 2024-03-06T07:17:47.796 INFO:teuthology.orchestra.run.smithi098.stdout:(105/176): abseil-cpp-20211102.0-3.el9.x86_64.r 2.1 MB/s | 548 kB 00:00 2024-03-06T07:17:47.896 INFO:teuthology.orchestra.run.smithi098.stdout:(106/176): gperftools-libs-2.9.1-2.el9.x86_64.r 3.0 MB/s | 309 kB 00:00 2024-03-06T07:17:47.954 INFO:teuthology.orchestra.run.smithi098.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 333 kB/s | 19 kB 00:00 2024-03-06T07:17:48.338 INFO:teuthology.orchestra.run.smithi098.stdout:(108/176): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-06T07:17:48.405 INFO:teuthology.orchestra.run.smithi098.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 392 kB/s | 26 kB 00:00 2024-03-06T07:17:48.464 INFO:teuthology.orchestra.run.smithi098.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 834 kB/s | 49 kB 00:00 2024-03-06T07:17:48.548 INFO:teuthology.orchestra.run.smithi098.stdout:(111/176): libunwind-1.6.2-1.el9.x86_64.rpm 807 kB/s | 67 kB 00:00 2024-03-06T07:17:48.656 INFO:teuthology.orchestra.run.smithi098.stdout:(112/176): luarocks-3.9.2-1.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-03-06T07:17:48.748 INFO:teuthology.orchestra.run.smithi098.stdout:(113/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2024-03-06T07:17:48.832 INFO:teuthology.orchestra.run.smithi098.stdout:(114/176): python3-asyncssh-2.13.2-4.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-03-06T07:17:48.890 INFO:teuthology.orchestra.run.smithi098.stdout:(115/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 746 kB/s | 43 kB 00:00 2024-03-06T07:17:48.966 INFO:teuthology.orchestra.run.smithi098.stdout:(116/176): python3-cachetools-4.2.4-1.el9.noarc 430 kB/s | 32 kB 00:00 2024-03-06T07:17:49.049 INFO:teuthology.orchestra.run.smithi098.stdout:(117/176): python3-certifi-2021.10.8-2.el9.noar 180 kB/s | 15 kB 00:00 2024-03-06T07:17:49.125 INFO:teuthology.orchestra.run.smithi098.stdout:(118/176): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-06T07:17:49.200 INFO:teuthology.orchestra.run.smithi098.stdout:(119/176): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-03-06T07:17:49.275 INFO:teuthology.orchestra.run.smithi098.stdout:(120/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-03-06T07:17:49.325 INFO:teuthology.orchestra.run.smithi098.stdout:(121/176): python3-colorama-0.4.4-12.el9.noarch 706 kB/s | 35 kB 00:00 2024-03-06T07:17:49.393 INFO:teuthology.orchestra.run.smithi098.stdout:(122/176): python3-google-auth-2.28.0-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-06T07:17:49.534 INFO:teuthology.orchestra.run.smithi098.stdout:(123/176): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-03-06T07:17:49.610 INFO:teuthology.orchestra.run.smithi098.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-03-06T07:17:49.685 INFO:teuthology.orchestra.run.smithi098.stdout:(125/176): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-06T07:17:49.777 INFO:teuthology.orchestra.run.smithi098.stdout:(126/176): python3-jaraco-classes-3.2.1-5.el9.n 193 kB/s | 18 kB 00:00 2024-03-06T07:17:49.917 INFO:teuthology.orchestra.run.smithi098.stdout:(127/176): python3-jaraco-collections-3.0.0-8.e 165 kB/s | 23 kB 00:00 2024-03-06T07:17:49.979 INFO:teuthology.orchestra.run.smithi098.stdout:(128/176): python3-jaraco-functools-3.5.0-2.el9 325 kB/s | 19 kB 00:00 2024-03-06T07:17:50.045 INFO:teuthology.orchestra.run.smithi098.stdout:(129/176): python3-jaraco-text-3.2.0-6.el9.noar 295 kB/s | 20 kB 00:00 2024-03-06T07:17:50.145 INFO:teuthology.orchestra.run.smithi098.stdout:(130/176): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-03-06T07:17:50.204 INFO:teuthology.orchestra.run.smithi098.stdout:(131/176): python3-logutils-0.3.5-21.el9.noarch 794 kB/s | 46 kB 00:00 2024-03-06T07:17:50.262 INFO:teuthology.orchestra.run.smithi098.stdout:(132/176): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-06T07:17:50.337 INFO:teuthology.orchestra.run.smithi098.stdout:(133/176): python3-natsort-7.1.1-5.el9.noarch.r 770 kB/s | 58 kB 00:00 2024-03-06T07:17:50.421 INFO:teuthology.orchestra.run.smithi098.stdout:(134/176): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-06T07:17:50.497 INFO:teuthology.orchestra.run.smithi098.stdout:(135/176): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-03-06T07:17:50.572 INFO:teuthology.orchestra.run.smithi098.stdout:(136/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-06T07:17:50.647 INFO:teuthology.orchestra.run.smithi098.stdout:(137/176): python3-pyperclip-1.8.2-2.el9.noarch 376 kB/s | 28 kB 00:00 2024-03-06T07:17:50.714 INFO:teuthology.orchestra.run.smithi098.stdout:(138/176): python3-repoze-lru-0.7-16.el9.noarch 461 kB/s | 31 kB 00:00 2024-03-06T07:17:50.781 INFO:teuthology.orchestra.run.smithi098.stdout:(139/176): python3-routes-2.5.1-5.el9.noarch.rp 2.8 MB/s | 188 kB 00:00 2024-03-06T07:17:50.840 INFO:teuthology.orchestra.run.smithi098.stdout:(140/176): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-06T07:17:50.899 INFO:teuthology.orchestra.run.smithi098.stdout:(141/176): python3-tempora-5.0.0-2.el9.noarch.r 614 kB/s | 36 kB 00:00 2024-03-06T07:17:50.966 INFO:teuthology.orchestra.run.smithi098.stdout:(142/176): python3-typing-extensions-4.4.0-2.el 763 kB/s | 51 kB 00:00 2024-03-06T07:17:51.049 INFO:teuthology.orchestra.run.smithi098.stdout:(143/176): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-06T07:17:51.133 INFO:teuthology.orchestra.run.smithi098.stdout:(144/176): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-03-06T07:17:51.217 INFO:teuthology.orchestra.run.smithi098.stdout:(145/176): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-06T07:17:51.300 INFO:teuthology.orchestra.run.smithi098.stdout:(146/176): python3-zc-lockfile-2.0-10.el9.noarc 240 kB/s | 20 kB 00:00 2024-03-06T07:17:51.384 INFO:teuthology.orchestra.run.smithi098.stdout:(147/176): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-06T07:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout:(148/176): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-06T07:17:57.534 INFO:teuthology.orchestra.run.smithi098.stdout:(149/176): ceph-mgr-diskprediction-local-19.0.0 396 kB/s | 7.4 MB 00:19 2024-03-06T07:18:00.826 INFO:teuthology.orchestra.run.smithi098.stdout:(150/176): librados2-19.0.0-1739.g1bc250c7.el9. 373 kB/s | 3.4 MB 00:09 2024-03-06T07:18:00.859 INFO:teuthology.orchestra.run.smithi098.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-03-06T07:18:00.968 INFO:teuthology.orchestra.run.smithi098.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 21 MB/s | 2.3 MB 00:00 2024-03-06T07:18:01.060 INFO:teuthology.orchestra.run.smithi098.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 20 MB/s | 1.8 MB 00:00 2024-03-06T07:18:01.102 INFO:teuthology.orchestra.run.smithi098.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 992 kB/s | 41 kB 00:00 2024-03-06T07:18:01.143 INFO:teuthology.orchestra.run.smithi098.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.8 MB/s | 247 kB 00:00 2024-03-06T07:18:01.177 INFO:teuthology.orchestra.run.smithi098.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2024-03-06T07:18:01.219 INFO:teuthology.orchestra.run.smithi098.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 847 kB/s | 35 kB 00:00 2024-03-06T07:18:01.261 INFO:teuthology.orchestra.run.smithi098.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 191 kB 00:00 2024-03-06T07:18:01.303 INFO:teuthology.orchestra.run.smithi098.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-03-06T07:18:01.387 INFO:teuthology.orchestra.run.smithi098.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 9.4 MB/s | 802 kB 00:00 2024-03-06T07:18:01.454 INFO:teuthology.orchestra.run.smithi098.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 6.4 MB/s | 433 kB 00:00 2024-03-06T07:18:01.496 INFO:teuthology.orchestra.run.smithi098.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 526 kB/s | 22 kB 00:00 2024-03-06T07:18:01.546 INFO:teuthology.orchestra.run.smithi098.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 107 kB 00:00 2024-03-06T07:18:01.588 INFO:teuthology.orchestra.run.smithi098.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 154 kB 00:00 2024-03-06T07:18:01.639 INFO:teuthology.orchestra.run.smithi098.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 135 kB 00:00 2024-03-06T07:18:01.681 INFO:teuthology.orchestra.run.smithi098.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 63 kB 00:00 2024-03-06T07:18:01.723 INFO:teuthology.orchestra.run.smithi098.stdout:(167/176): libuuid-2.37.4-18.el9.x86_64.rpm 662 kB/s | 28 kB 00:00 2024-03-06T07:18:01.798 INFO:teuthology.orchestra.run.smithi098.stdout:(168/176): openssl-3.0.7-27.el9.x86_64.rpm 15 MB/s | 1.2 MB 00:00 2024-03-06T07:18:01.915 INFO:teuthology.orchestra.run.smithi098.stdout:(169/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 18 MB/s | 2.2 MB 00:00 2024-03-06T07:18:02.065 INFO:teuthology.orchestra.run.smithi098.stdout:(170/176): util-linux-2.37.4-18.el9.x86_64.rpm 15 MB/s | 2.2 MB 00:00 2024-03-06T07:18:02.132 INFO:teuthology.orchestra.run.smithi098.stdout:(171/176): util-linux-core-2.37.4-18.el9.x86_64 6.4 MB/s | 438 kB 00:00 2024-03-06T07:18:02.183 INFO:teuthology.orchestra.run.smithi098.stdout:(172/176): util-linux-user-2.37.4-18.el9.x86_64 601 kB/s | 30 kB 00:00 2024-03-06T07:18:02.241 INFO:teuthology.orchestra.run.smithi098.stdout:(173/176): xfsdump-3.1.12-4.el9.x86_64.rpm 5.4 MB/s | 321 kB 00:00 2024-03-06T07:18:02.325 INFO:teuthology.orchestra.run.smithi098.stdout:(174/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 13 MB/s | 1.1 MB 00:00 2024-03-06T07:18:05.425 INFO:teuthology.orchestra.run.smithi098.stdout:(175/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 407 kB/s | 3.1 MB 00:07 2024-03-06T07:18:28.708 INFO:teuthology.orchestra.run.smithi098.stdout:(176/176): ceph-test-19.0.0-1739.g1bc250c7.el9. 659 kB/s | 47 MB 01:13 2024-03-06T07:18:28.710 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:18:28.710 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.0 MB/s | 224 MB 01:50 2024-03-06T07:18:28.750 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-06T07:18:29.058 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-06T07:18:29.058 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-06T07:18:31.113 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-06T07:18:31.114 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-06T07:18:38.016 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-06T07:18:38.102 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:18:38.137 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:18:38.167 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:18:38.191 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:18:38.250 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:18:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:18:38.343 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:18:38.384 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:18:38.410 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:18:38.490 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:18:38.821 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:18:38.878 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:18:39.000 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:18:39.125 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:18:39.262 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:18:39.301 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:18:39.451 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:18:39.522 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:18:39.550 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:18:39.653 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:18:39.676 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:18:39.724 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:18:39.755 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:18:39.778 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:18:39.793 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:18:39.961 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:18:39.990 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:18:40.022 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:18:40.202 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:18:40.348 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:18:40.409 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:18:40.535 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:18:40.585 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:18:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:18:40.902 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:18:41.004 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:18:41.040 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:18:41.110 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:18:41.163 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:18:41.196 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:18:41.229 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:18:41.269 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:18:41.334 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:18:41.368 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:18:41.421 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:18:41.453 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:18:41.481 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:18:41.551 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:18:41.586 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:18:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:18:41.658 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:18:41.726 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:18:41.772 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:18:41.846 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:18:41.988 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:18:42.042 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:18:42.559 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:18:42.637 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:18:42.715 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:18:42.766 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:18:42.774 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:18:42.813 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:18:42.850 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:18:42.882 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:18:42.997 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:18:43.084 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:18:43.384 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:18:43.560 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:18:43.627 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:18:43.680 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:18:43.822 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:18:43.883 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:18:43.938 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:18:44.011 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:18:44.042 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:18:44.078 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:18:44.125 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:18:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:18:44.205 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:18:44.246 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:18:44.381 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:18:44.428 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:18:44.458 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:18:44.499 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:18:44.593 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:18:44.647 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:18:44.786 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:18:44.949 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:18:45.098 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:18:45.183 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:18:45.308 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:18:45.407 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:18:45.456 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:18:45.505 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:18:45.561 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:18:45.748 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:18:45.786 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:18:45.915 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:18:45.963 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:18:46.000 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:18:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:18:46.067 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:18:46.098 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:18:46.151 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:18:46.186 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:18:46.225 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:18:46.257 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:18:46.287 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:18:46.363 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:18:46.397 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:18:46.439 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:18:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:18:46.515 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:18:46.604 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:18:46.687 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:18:46.729 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:18:46.761 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:18:46.807 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:18:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:18:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:18:47.676 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:18:47.722 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:18:47.756 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:18:47.804 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:18:47.891 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:18:47.932 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:18:48.108 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:18:48.146 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:18:48.223 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:18:48.864 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:18:48.980 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:18:50.194 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:18:50.267 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:18:50.294 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:18:50.606 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:18:50.633 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:18:50.707 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:18:50.746 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:18:50.782 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:18:51.244 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:18:51.278 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:18:51.349 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:18:51.371 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:18:53.610 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:18:53.643 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:18:54.181 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:18:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:18:54.306 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:18:54.306 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:18:54.306 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:18:54.306 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:18:54.339 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:19:06.073 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:19:06.074 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-03-06T07:19:06.074 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-03-06T07:19:06.074 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-03-06T07:19:06.074 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-03-06T07:19:06.074 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-03-06T07:19:06.074 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-03-06T07:19:06.074 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-03-06T07:19:06.074 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:06.246 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:19:06.261 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:19:06.261 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:19:06.261 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:19:06.261 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:07.784 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:19:07.809 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:19:07.810 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:19:07.810 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:19:07.810 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:19:07.810 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:19:07.810 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:19:08.056 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:19:08.056 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:19:08.057 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:19:08.057 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:19:08.057 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:19:08.057 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:08.510 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:19:08.533 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:19:08.533 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:19:08.533 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:19:08.533 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:19:08.533 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:19:08.533 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:08.620 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:19:09.037 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:19:09.069 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:19:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:19:09.160 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:19:09.183 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:19:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:19:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:19:10.417 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:19:10.445 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:19:10.545 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:19:10.717 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:19:10.761 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:19:10.784 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:19:10.784 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:19:10.784 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:19:10.784 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:19:10.784 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:19:10.784 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:10.831 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:19:10.842 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:19:10.893 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:19:11.747 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:19:11.783 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:19:11.806 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:19:11.806 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:19:11.806 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:19:11.806 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:19:11.807 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:19:11.807 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:11.851 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:19:11.873 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:19:11.873 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:19:11.873 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:19:11.873 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:11.926 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:19:11.948 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:19:11.948 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:19:11.948 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:19:11.948 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:19:11.948 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:19:11.949 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:12.261 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:19:12.282 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:19:12.283 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:19:12.283 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:19:12.283 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:19:12.283 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:19:12.283 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:16.732 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:19:16.773 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:19:16.803 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:19:16.831 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:19:17.018 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:19:17.131 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:19:17.179 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:19:17.226 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:19:17.279 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:19:17.345 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:19:17.381 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:19:17.425 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:19:17.456 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:19:17.485 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:19:17.564 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:19:17.618 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:19:17.659 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:19:18.238 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:19:18.282 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:19:18.303 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:19:19.110 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:19:19.178 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:19:19.219 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:19:19.310 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:19:19.340 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:19:19.365 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:19:19.391 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:19:19.418 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:19:19.418 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:19:19.485 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:19:19.513 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:19:19.543 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:19:19.543 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:19:19.628 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:19:19.655 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:19:19.696 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:19:19.721 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:19:19.751 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:19:19.781 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:19:19.815 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:19:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:19:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:19:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:19:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:19:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:19:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:19:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:19:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:19:20.190 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:19:20.191 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:19:20.273 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:19:20.305 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:19:20.332 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:19:20.357 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:19:20.383 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:19:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:19:25.256 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:19:25.256 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:19:25.256 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:19:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:19:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:19:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:19:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:19:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:19:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:19:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:19:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:19:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:19:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:19:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:19:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:19:25.258 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:19:25.258 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:19:25.258 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:19:25.258 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:19:25.258 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:19:25.258 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:19:25.258 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:19:25.258 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:19:25.258 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:19:25.258 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:19:25.258 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:19:25.258 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:19:25.258 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:19:25.278 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:19:25.278 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:19:25.278 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:19:25.278 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:19:25.278 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:19:25.278 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:19:25.278 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:19:25.278 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:19:25.278 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:19:25.278 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:19:25.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:19:25.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:19:25.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:19:25.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:19:25.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:19:25.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:19:25.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:19:25.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:19:25.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:19:25.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:19:25.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:19:25.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:19:25.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:19:25.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:19:25.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:19:25.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:19:25.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:19:25.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:19:25.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:19:25.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:19:25.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:19:25.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:19:25.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:19:25.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:19:25.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:19:25.281 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:19:25.281 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:19:25.281 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:19:25.281 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:19:25.281 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:19:25.281 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:19:25.281 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:19:25.281 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:19:25.281 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:19:25.281 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:19:25.281 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:19:25.281 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:19:25.281 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:19:25.282 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:19:25.282 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:19:25.282 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:19:25.282 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:19:25.282 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:19:25.282 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:19:25.282 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:19:25.282 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:19:25.282 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:19:25.282 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:19:25.282 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:19:25.282 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:19:25.282 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:19:25.283 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:19:25.283 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:19:25.283 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:19:25.283 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:19:25.283 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:19:25.283 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:19:25.283 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:19:25.283 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:19:25.283 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:19:25.283 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:19:25.283 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:19:25.283 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:19:25.283 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:19:25.283 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:19:25.284 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:19:25.284 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:19:25.284 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:19:25.284 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:19:25.284 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:19:25.284 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:19:25.284 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:19:25.284 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:19:25.284 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:19:25.284 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:19:25.284 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:19:25.284 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:19:25.284 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:19:25.285 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:19:25.285 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:19:25.285 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:19:25.285 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:19:25.285 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:19:25.285 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:19:25.285 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:19:25.285 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:19:25.285 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:19:25.285 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:19:25.285 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:19:25.285 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:19:25.285 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:19:25.285 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:19:25.286 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:19:25.286 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:19:25.286 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:19:25.286 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:19:25.286 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:19:25.286 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:19:25.286 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:19:25.286 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:19:25.286 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:19:25.286 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:19:25.286 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:19:25.286 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:19:25.286 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:19:25.286 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:19:25.286 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:19:25.287 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:19:25.287 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:19:25.287 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:19:25.287 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:19:25.287 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:19:25.287 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:19:25.287 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:19:25.287 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:19:25.287 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:19:25.287 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:19:25.287 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:19:25.287 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:19:25.287 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:19:25.287 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:19:25.288 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:19:25.288 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:19:25.288 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:19:25.288 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:19:25.288 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:19:25.288 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:19:25.288 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:19:25.288 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:19:25.288 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:19:25.288 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:19:25.288 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:19:25.288 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:19:25.288 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:19:25.288 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:19:25.289 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:19:25.289 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:19:25.289 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:19:25.289 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:19:25.289 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:19:25.289 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:19:25.289 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:19:25.289 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:19:25.289 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:19:25.289 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:19:25.289 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:19:25.289 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:19:25.289 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:19:25.290 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:19:25.290 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:19:25.290 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:19:25.290 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:19:25.290 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:19:25.290 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:19:25.290 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:19:25.290 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:19:25.290 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:19:25.290 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:19:25.290 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:19:25.290 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:19:25.290 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:19:25.290 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:19:25.290 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:19:25.291 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:19:28.160 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:19:28.160 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:28.160 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-03-06T07:19:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:19:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:19:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:19:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:19:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:19:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:19:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:19:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:19:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:19:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:19:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:19:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:19:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:19:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:19:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:19:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:19:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:19:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:19:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:19:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:19:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:19:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:19:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:19:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:19:28.162 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-06T07:19:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:19:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:19:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:19:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:19:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:19:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:19:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:19:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:19:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:19:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:19:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:19:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:19:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:19:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:19:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:19:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:19:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:19:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:19:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:19:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:19:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:19:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:19:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:19:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:19:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:19:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:19:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:19:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:19:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:19:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:19:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:19:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:19:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:19:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:19:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:19:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:19:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:19:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:19:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:19:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:19:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:19:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:19:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:19:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:19:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:19:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:19:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:19:28.168 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:19:28.168 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:19:28.168 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:19:28.168 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:19:28.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:19:28.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:19:28.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:19:28.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:19:28.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:19:28.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.168 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:19:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:19:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:19:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:19:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:19:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:19:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:19:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:19:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:19:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:19:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:19:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:19:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:19:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:19:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:19:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:19:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:19:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:19:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:19:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:19:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:19:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:19:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:19:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:19:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:19:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:19:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:19:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:19:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:19:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:19:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:19:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:19:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:19:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:19:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:19:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:19:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:19:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:19:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:19:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:19:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:19:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:19:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:19:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:19:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:19:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:19:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:19:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:19:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:19:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:19:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:19:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:19:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:19:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:19:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:19:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:19:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:19:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:19:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:19:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:19:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:19:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:19:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:19:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:19:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:19:28.174 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:19:28.174 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:28.174 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-06T07:19:28.789 DEBUG:teuthology.parallel:result is None 2024-03-06T07:19:28.789 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:19:28.933 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:19:28.960 INFO:teuthology.orchestra.run.smithi001.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:19:28.960 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:19:28.961 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:19:28.962 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:19:29.181 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:19:29.203 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:19:29.204 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:19:29.204 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:19:29.205 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-06T07:19:29.206 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:19:29.206 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:19:29.241 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:19:29.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:19:29.274 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-06T07:19:29.275 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:19:29.275 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:19:29.311 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:19:29.379 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:19:29.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:19:29.407 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:19:29.475 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-06T07:19:29.475 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:19:29.475 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:19:29.505 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:19:29.573 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:19:29.573 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:19:29.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:19:29.668 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-06T07:19:29.668 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:19:29.668 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:19:29.695 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:19:29.763 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:19:29.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:19:29.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:19:29.858 INFO:teuthology.run_tasks:Running task ceph... 2024-03-06T07:19:29.946 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-06T07:19:29.946 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 777 /var/log/ceph 2024-03-06T07:19:29.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /var/log/ceph 2024-03-06T07:19:29.975 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-06T07:19:29.975 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:19:30.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:19:30.042 INFO:tasks.ceph:Creating extra log directories... 2024-03-06T07:19:30.042 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:19:30.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:19:30.116 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-06T07:19:30.116 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-06T07:19:30.116 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-06T07:19:30.116 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:19:30.116 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:19:30.164 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:19:30.281 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:19:30.307 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:19:30.307 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:19:30.322 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:19:30.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:19:30.472 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-06T07:19:30.473 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-06T07:19:30.473 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583045', 'branch': 'wip-lusov-qdb', 'description': 'fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}', 'email': None, 'first_in_suite': False, 'job_id': '7583045', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'ceph-fuse': {'disabled': True}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'kclient': {'syntax': 'v2'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'seed': 5072, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'sleep_before_teardown': 0, 'subset': '246/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi001.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKALq4zEqGpGYLaiiOGwZ7qnJ/XLWKAx+2LZBoOmFGiRSpv8TRDeLuBEvCbSwK2CTmZRnmuIej2HD0aox/FiC68=', 'smithi098.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLZZgtAwrVih+44oDkYrNyHiWI3sLRPpRARTCFN6ZaHTMKDPew6Nx769IQH3Zx8HSMODQC5eLh3FE355iWsox7w='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'cluster': 'ceph'}}, {'kclient': None}, {'exec': {'mon.a': ['ceph config set mgr mgr/crash/warn_recent_interval 0']}}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_failover']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-06_07:00:27', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-06T07:19:30.473 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:19:30.475 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:19:30.522 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:19:30.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:19:30.589 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:19:30.589 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:19:30.610 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:19:30.611 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_1 2024-03-06T07:19:30.665 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:19:30.665 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:19:30.665 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-06T07:19:30.665 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:19:30.665 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:19:30.665 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-06 07:17:21.455422504 +0000 2024-03-06T07:19:30.665 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-06 07:15:11.732831016 +0000 2024-03-06T07:19:30.665 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-06 07:15:11.732831016 +0000 2024-03-06T07:19:30.665 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-06 07:15:11.732831016 +0000 2024-03-06T07:19:30.666 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:19:30.729 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-06T07:19:30.730 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-06T07:19:30.730 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 8.3313e-05 s, 6.1 MB/s 2024-03-06T07:19:30.731 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:19:30.786 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_2 2024-03-06T07:19:30.840 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:19:30.840 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:19:30.840 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-06T07:19:30.840 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:19:30.840 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:19:30.841 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-06 07:17:21.456422492 +0000 2024-03-06T07:19:30.841 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-06 07:15:12.074827160 +0000 2024-03-06T07:19:30.841 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-06 07:15:12.074827160 +0000 2024-03-06T07:19:30.841 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-06 07:15:12.074827160 +0000 2024-03-06T07:19:30.841 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:19:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-06T07:19:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-06T07:19:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.6087e-05 s, 5.3 MB/s 2024-03-06T07:19:30.907 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:19:30.963 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_3 2024-03-06T07:19:31.017 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:19:31.017 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:19:31.017 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-06T07:19:31.017 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:19:31.017 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:19:31.017 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-06 07:17:21.456422492 +0000 2024-03-06T07:19:31.017 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-06 07:15:12.430823145 +0000 2024-03-06T07:19:31.017 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-06 07:15:12.430823145 +0000 2024-03-06T07:19:31.018 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-06 07:15:12.430823145 +0000 2024-03-06T07:19:31.018 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:19:31.081 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-06T07:19:31.082 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-06T07:19:31.082 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 8.935e-05 s, 5.7 MB/s 2024-03-06T07:19:31.083 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:19:31.139 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_4 2024-03-06T07:19:31.193 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:19:31.193 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:19:31.193 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-06T07:19:31.193 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:19:31.193 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:19:31.194 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-06 07:17:21.456422492 +0000 2024-03-06T07:19:31.194 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-06 07:15:12.755819480 +0000 2024-03-06T07:19:31.194 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-06 07:15:12.755819480 +0000 2024-03-06T07:19:31.194 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-06 07:15:12.755819480 +0000 2024-03-06T07:19:31.194 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:19:31.259 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-06T07:19:31.259 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-06T07:19:31.259 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.6304e-05 s, 5.3 MB/s 2024-03-06T07:19:31.260 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:19:31.317 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:19:31.317 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:19:31.318 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:19:31.334 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:19:31.334 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-03-06T07:19:31.389 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:19:31.389 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:19:31.390 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-06T07:19:31.390 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:19:31.390 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:19:31.390 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-06 07:19:20.780692824 +0000 2024-03-06T07:19:31.390 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-06 07:15:09.903426472 +0000 2024-03-06T07:19:31.390 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-06 07:15:09.903426472 +0000 2024-03-06T07:19:31.390 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-06 07:15:09.903426472 +0000 2024-03-06T07:19:31.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:19:31.456 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-06T07:19:31.456 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-06T07:19:31.456 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.00020695 s, 2.5 MB/s 2024-03-06T07:19:31.457 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:19:31.514 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-03-06T07:19:31.570 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:19:31.570 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:19:31.570 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-06T07:19:31.570 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:19:31.570 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:19:31.571 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-06 07:19:20.781692813 +0000 2024-03-06T07:19:31.571 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-06 07:15:10.278422197 +0000 2024-03-06T07:19:31.571 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-06 07:15:10.278422197 +0000 2024-03-06T07:19:31.571 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-06 07:15:10.278422197 +0000 2024-03-06T07:19:31.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:19:31.637 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-06T07:19:31.638 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-06T07:19:31.638 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000196084 s, 2.6 MB/s 2024-03-06T07:19:31.639 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:19:31.695 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-03-06T07:19:31.750 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:19:31.750 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:19:31.750 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-06T07:19:31.750 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:19:31.750 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:19:31.750 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-06 07:19:20.781692813 +0000 2024-03-06T07:19:31.750 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-06 07:15:10.621418291 +0000 2024-03-06T07:19:31.750 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-06 07:15:10.621418291 +0000 2024-03-06T07:19:31.750 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-06 07:15:10.621418291 +0000 2024-03-06T07:19:31.751 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:19:31.815 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-06T07:19:31.815 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-06T07:19:31.816 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000176755 s, 2.9 MB/s 2024-03-06T07:19:31.817 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:19:31.872 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-03-06T07:19:31.927 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:19:31.927 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:19:31.927 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-06T07:19:31.927 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:19:31.928 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:19:31.928 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-06 07:19:20.782692801 +0000 2024-03-06T07:19:31.928 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-06 07:15:11.037413554 +0000 2024-03-06T07:19:31.928 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-06 07:15:11.037413554 +0000 2024-03-06T07:19:31.928 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-06 07:15:11.037413554 +0000 2024-03-06T07:19:31.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:19:31.992 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-06T07:19:31.993 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-06T07:19:31.993 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000209124 s, 2.4 MB/s 2024-03-06T07:19:31.994 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:19:32.049 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:19:32.049 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:19:32.049 INFO:tasks.ceph:Generating config... 2024-03-06T07:19:32.050 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-06T07:19:32.051 INFO:tasks.ceph:[client] debug client = 20 2024-03-06T07:19:32.051 INFO:tasks.ceph:[client] debug ms = 1 2024-03-06T07:19:32.051 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-06T07:19:32.051 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-06T07:19:32.051 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-03-06T07:19:32.051 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-06T07:19:32.051 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-06T07:19:32.051 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-06T07:19:32.051 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-06T07:19:32.051 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-06T07:19:32.051 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-06T07:19:32.051 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-06T07:19:32.051 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-06T07:19:32.051 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-06T07:19:32.051 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-06T07:19:32.052 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-06T07:19:32.052 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-06T07:19:32.052 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-06T07:19:32.052 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-06T07:19:32.052 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-06T07:19:32.052 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-06T07:19:32.052 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-06T07:19:32.052 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-06T07:19:32.052 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-06T07:19:32.052 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-06T07:19:32.052 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-06T07:19:32.052 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-06T07:19:32.052 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-06T07:19:32.052 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-06T07:19:32.052 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-06T07:19:32.052 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-06T07:19:32.053 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-06T07:19:32.053 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-06T07:19:32.053 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-06T07:19:32.053 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-06T07:19:32.053 INFO:tasks.ceph:Setting up mon.a... 2024-03-06T07:19:32.053 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-06T07:19:32.106 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.keyring 2024-03-06T07:19:32.109 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-06T07:19:32.155 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:19:32.183 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.1'), ('mon.b', '172.21.15.98'), ('mon.c', '[v2:172.21.15.98:3301,v1:172.21.15.98:6790]')] 2024-03-06T07:19:32.183 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.1,172.21.15.98,[v2:172.21.15.98:3301,v1:172.21.15.98:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-06T07:19:32.184 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:19:32.184 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:19:32.240 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.1 --add b 172.21.15.98 --addv c '[v2:172.21.15.98:3301,v1:172.21.15.98:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:19:32.326 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:19:32.326 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: generated fsid 8f3c19ba-4ac6-4a54-88df-9262183d23ef 2024-03-06T07:19:32.327 INFO:teuthology.orchestra.run.smithi001.stdout:setting min_mon_release = quincy 2024-03-06T07:19:32.327 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 0 2024-03-06T07:19:32.327 INFO:teuthology.orchestra.run.smithi001.stdout:fsid 8f3c19ba-4ac6-4a54-88df-9262183d23ef 2024-03-06T07:19:32.327 INFO:teuthology.orchestra.run.smithi001.stdout:last_changed 2024-03-06T07:19:32.325805+0000 2024-03-06T07:19:32.327 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-03-06T07:19:32.325805+0000 2024-03-06T07:19:32.327 INFO:teuthology.orchestra.run.smithi001.stdout:min_mon_release 17 (quincy) 2024-03-06T07:19:32.327 INFO:teuthology.orchestra.run.smithi001.stdout:election_strategy: 1 2024-03-06T07:19:32.327 INFO:teuthology.orchestra.run.smithi001.stdout:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-06T07:19:32.327 INFO:teuthology.orchestra.run.smithi001.stdout:1: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.b 2024-03-06T07:19:32.327 INFO:teuthology.orchestra.run.smithi001.stdout:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.c 2024-03-06T07:19:32.327 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-06T07:19:32.328 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:19:32.344 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8f3c19ba-4ac6-4a54-88df-9262183d23ef... 2024-03-06T07:19:32.345 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:19:32.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:19:32.436 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-03-06T07:19:32.436 INFO:teuthology.orchestra.run.smithi001.stdout: chdir = "" 2024-03-06T07:19:32.436 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:19:32.436 INFO:teuthology.orchestra.run.smithi001.stdout: auth supported = cephx 2024-03-06T07:19:32.436 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.436 INFO:teuthology.orchestra.run.smithi001.stdout: filestore xattr use omap = true 2024-03-06T07:19:32.436 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.436 INFO:teuthology.orchestra.run.smithi001.stdout: mon clock drift allowed = 1.000 2024-03-06T07:19:32.437 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.437 INFO:teuthology.orchestra.run.smithi001.stdout: osd crush chooseleaf type = 0 2024-03-06T07:19:32.437 INFO:teuthology.orchestra.run.smithi001.stdout: auth debug = true 2024-03-06T07:19:32.437 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.437 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on old message = true 2024-03-06T07:19:32.437 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on bug = true 2024-03-06T07:19:32.437 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.437 INFO:teuthology.orchestra.run.smithi001.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:19:32.437 INFO:teuthology.orchestra.run.smithi001.stdout: mon pg warn max object skew = 0 2024-03-06T07:19:32.437 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.437 INFO:teuthology.orchestra.run.smithi001.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:19:32.437 INFO:teuthology.orchestra.run.smithi001.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:19:32.437 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.437 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default size = 2 2024-03-06T07:19:32.437 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.437 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow primary affinity = true 2024-03-06T07:19:32.438 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow pg remap = true 2024-03-06T07:19:32.438 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:19:32.438 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:19:32.438 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on no sortbitwise = false 2024-03-06T07:19:32.438 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:19:32.438 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on too few osds = false 2024-03-06T07:19:32.438 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:19:32.438 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:19:32.438 INFO:teuthology.orchestra.run.smithi001.stdout: mon_allow_pool_size_one = true 2024-03-06T07:19:32.438 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.438 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:19:32.438 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.438 INFO:teuthology.orchestra.run.smithi001.stdout: osd default data pool replay window = 5 2024-03-06T07:19:32.438 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.438 INFO:teuthology.orchestra.run.smithi001.stdout: mon allow pool delete = true 2024-03-06T07:19:32.439 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.439 INFO:teuthology.orchestra.run.smithi001.stdout: mon cluster log file level = debug 2024-03-06T07:19:32.439 INFO:teuthology.orchestra.run.smithi001.stdout: debug asserts on shutdown = true 2024-03-06T07:19:32.439 INFO:teuthology.orchestra.run.smithi001.stdout: mon health detail to clog = false 2024-03-06T07:19:32.439 INFO:teuthology.orchestra.run.smithi001.stdout: mon host = "172.21.15.1,172.21.15.98,[v2:172.21.15.98:3301,v1:172.21.15.98:6790]" 2024-03-06T07:19:32.439 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on skipped message = False 2024-03-06T07:19:32.439 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = 8f3c19ba-4ac6-4a54-88df-9262183d23ef 2024-03-06T07:19:32.439 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.439 INFO:teuthology.orchestra.run.smithi001.stdout:[osd] 2024-03-06T07:19:32.439 INFO:teuthology.orchestra.run.smithi001.stdout: osd journal size = 100 2024-03-06T07:19:32.439 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.439 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub load threshold = 5.0 2024-03-06T07:19:32.439 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub max interval = 600 2024-03-06T07:19:32.439 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:19:32.439 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.440 INFO:teuthology.orchestra.run.smithi001.stdout: osd recover clone overlap = true 2024-03-06T07:19:32.440 INFO:teuthology.orchestra.run.smithi001.stdout: osd recovery max chunk = 1048576 2024-03-06T07:19:32.440 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.440 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug shutdown = true 2024-03-06T07:19:32.440 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug op order = true 2024-03-06T07:19:32.440 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify stray on activate = true 2024-03-06T07:19:32.440 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug trim objects = true 2024-03-06T07:19:32.440 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.440 INFO:teuthology.orchestra.run.smithi001.stdout: osd open classes on start = true 2024-03-06T07:19:32.440 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug pg log writeout = true 2024-03-06T07:19:32.440 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.440 INFO:teuthology.orchestra.run.smithi001.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:19:32.440 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.440 INFO:teuthology.orchestra.run.smithi001.stdout: osd map max advance = 10 2024-03-06T07:19:32.440 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.441 INFO:teuthology.orchestra.run.smithi001.stdout: journal zero on create = true 2024-03-06T07:19:32.441 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.441 INFO:teuthology.orchestra.run.smithi001.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:19:32.441 INFO:teuthology.orchestra.run.smithi001.stdout: filestore apply finisher threads = 3 2024-03-06T07:19:32.441 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.441 INFO:teuthology.orchestra.run.smithi001.stdout: bdev debug aio = true 2024-03-06T07:19:32.441 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug misdirected ops = true 2024-03-06T07:19:32.441 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore block size = 96636764160 2024-03-06T07:19:32.441 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore compression mode = aggressive 2024-03-06T07:19:32.441 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore fsck on mount = True 2024-03-06T07:19:32.441 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluefs = 20 2024-03-06T07:19:32.441 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluestore = 20 2024-03-06T07:19:32.441 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-06T07:19:32.441 INFO:teuthology.orchestra.run.smithi001.stdout: debug osd = 20 2024-03-06T07:19:32.441 INFO:teuthology.orchestra.run.smithi001.stdout: debug rocksdb = 10 2024-03-06T07:19:32.442 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:19:32.442 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd full ratio = 0.9 2024-03-06T07:19:32.442 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:19:32.442 INFO:teuthology.orchestra.run.smithi001.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:19:32.442 INFO:teuthology.orchestra.run.smithi001.stdout: osd objectstore = bluestore 2024-03-06T07:19:32.442 INFO:teuthology.orchestra.run.smithi001.stdout: osd op complaint time = 180 2024-03-06T07:19:32.442 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.442 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr] 2024-03-06T07:19:32.442 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-06T07:19:32.442 INFO:teuthology.orchestra.run.smithi001.stdout: debug mgr = 20 2024-03-06T07:19:32.442 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-03-06T07:19:32.442 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-03-06T07:19:32.442 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:19:32.442 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:19:32.442 INFO:teuthology.orchestra.run.smithi001.stdout: mgr/telemetry/nag = false 2024-03-06T07:19:32.443 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.443 INFO:teuthology.orchestra.run.smithi001.stdout:[mon] 2024-03-06T07:19:32.443 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-06T07:19:32.443 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-03-06T07:19:32.443 INFO:teuthology.orchestra.run.smithi001.stdout: debug paxos = 20 2024-03-06T07:19:32.443 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-03-06T07:19:32.443 INFO:teuthology.orchestra.run.smithi001.stdout: mon data avail warn = 5 2024-03-06T07:19:32.443 INFO:teuthology.orchestra.run.smithi001.stdout: mon mgr mkfs grace = 240 2024-03-06T07:19:32.443 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:19:32.443 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd reporter subtree level = osd 2024-03-06T07:19:32.443 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd prime pg temp = true 2024-03-06T07:19:32.443 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:19:32.443 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.443 INFO:teuthology.orchestra.run.smithi001.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:19:32.443 INFO:teuthology.orchestra.run.smithi001.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:19:32.443 INFO:teuthology.orchestra.run.smithi001.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:19:32.444 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.444 INFO:teuthology.orchestra.run.smithi001.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:19:32.444 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:19:32.444 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:19:32.444 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.444 INFO:teuthology.orchestra.run.smithi001.stdout: # 1m isn't quite enough 2024-03-06T07:19:32.444 INFO:teuthology.orchestra.run.smithi001.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:19:32.444 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.444 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:19:32.444 INFO:teuthology.orchestra.run.smithi001.stdout: mon op complaint time = 120 2024-03-06T07:19:32.444 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:19:32.444 INFO:teuthology.orchestra.run.smithi001.stdout:[client] 2024-03-06T07:19:32.444 INFO:teuthology.orchestra.run.smithi001.stdout: rgw cache enabled = true 2024-03-06T07:19:32.444 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable ops log = true 2024-03-06T07:19:32.444 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable usage log = true 2024-03-06T07:19:32.445 INFO:teuthology.orchestra.run.smithi001.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:19:32.445 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:19:32.445 INFO:teuthology.orchestra.run.smithi001.stdout: client mount timeout = 600 2024-03-06T07:19:32.445 INFO:teuthology.orchestra.run.smithi001.stdout: debug client = 20 2024-03-06T07:19:32.445 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-06T07:19:32.445 INFO:teuthology.orchestra.run.smithi001.stdout: rados mon op timeout = 900 2024-03-06T07:19:32.445 INFO:teuthology.orchestra.run.smithi001.stdout: rados osd op timeout = 900 2024-03-06T07:19:32.445 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.a] 2024-03-06T07:19:32.445 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.b] 2024-03-06T07:19:32.445 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.c] 2024-03-06T07:19:32.445 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.a] 2024-03-06T07:19:32.445 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.c] 2024-03-06T07:19:32.445 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.b] 2024-03-06T07:19:32.445 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.d] 2024-03-06T07:19:32.445 INFO:teuthology.orchestra.run.smithi001.stdout:[mds] 2024-03-06T07:19:32.445 INFO:teuthology.orchestra.run.smithi001.stdout: debug mds = 20 2024-03-06T07:19:32.446 INFO:teuthology.orchestra.run.smithi001.stdout: debug mds balancer = 20 2024-03-06T07:19:32.446 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-06T07:19:32.446 INFO:teuthology.orchestra.run.smithi001.stdout: mds debug frag = True 2024-03-06T07:19:32.446 INFO:teuthology.orchestra.run.smithi001.stdout: mds debug scatterstat = True 2024-03-06T07:19:32.446 INFO:teuthology.orchestra.run.smithi001.stdout: mds op complaint time = 180 2024-03-06T07:19:32.446 INFO:teuthology.orchestra.run.smithi001.stdout: mds verify scatter = True 2024-03-06T07:19:32.446 INFO:teuthology.orchestra.run.smithi001.stdout: osd op complaint time = 180 2024-03-06T07:19:32.446 INFO:teuthology.orchestra.run.smithi001.stdout: rados mon op timeout = 900 2024-03-06T07:19:32.446 INFO:teuthology.orchestra.run.smithi001.stdout: rados osd op timeout = 900 2024-03-06T07:19:32.447 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-03-06T07:19:32.447 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2024-03-06T07:19:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:19:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2024-03-06T07:19:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2024-03-06T07:19:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2024-03-06T07:19:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2024-03-06T07:19:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2024-03-06T07:19:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2024-03-06T07:19:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2024-03-06T07:19:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:19:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2024-03-06T07:19:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:19:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:19:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2024-03-06T07:19:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2024-03-06T07:19:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2024-03-06T07:19:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:19:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:19:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2024-03-06T07:19:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:19:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2024-03-06T07:19:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:19:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:19:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2024-03-06T07:19:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:19:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2024-03-06T07:19:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2024-03-06T07:19:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2024-03-06T07:19:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2024-03-06T07:19:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2024-03-06T07:19:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = "172.21.15.1,172.21.15.98,[v2:172.21.15.98:3301,v1:172.21.15.98:6790]" 2024-03-06T07:19:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on skipped message = False 2024-03-06T07:19:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = 8f3c19ba-4ac6-4a54-88df-9262183d23ef 2024-03-06T07:19:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.451 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2024-03-06T07:19:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2024-03-06T07:19:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2024-03-06T07:19:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2024-03-06T07:19:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:19:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2024-03-06T07:19:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2024-03-06T07:19:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2024-03-06T07:19:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2024-03-06T07:19:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2024-03-06T07:19:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug trim objects = true 2024-03-06T07:19:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2024-03-06T07:19:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2024-03-06T07:19:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:19:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2024-03-06T07:19:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2024-03-06T07:19:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:19:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2024-03-06T07:19:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2024-03-06T07:19:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2024-03-06T07:19:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore block size = 96636764160 2024-03-06T07:19:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore compression mode = aggressive 2024-03-06T07:19:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore fsck on mount = True 2024-03-06T07:19:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluefs = 20 2024-03-06T07:19:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluestore = 20 2024-03-06T07:19:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-06T07:19:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 20 2024-03-06T07:19:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: debug rocksdb = 10 2024-03-06T07:19:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:19:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd full ratio = 0.9 2024-03-06T07:19:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:19:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:19:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: osd objectstore = bluestore 2024-03-06T07:19:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: osd op complaint time = 180 2024-03-06T07:19:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.455 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2024-03-06T07:19:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-06T07:19:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2024-03-06T07:19:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-03-06T07:19:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-03-06T07:19:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:19:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:19:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2024-03-06T07:19:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.455 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2024-03-06T07:19:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-06T07:19:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-03-06T07:19:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2024-03-06T07:19:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-03-06T07:19:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2024-03-06T07:19:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2024-03-06T07:19:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:19:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2024-03-06T07:19:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2024-03-06T07:19:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:19:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:19:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:19:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:19:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:19:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:19:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:19:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: # 1m isn't quite enough 2024-03-06T07:19:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:19:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:19:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: mon op complaint time = 120 2024-03-06T07:19:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-06T07:19:32.457 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2024-03-06T07:19:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2024-03-06T07:19:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2024-03-06T07:19:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2024-03-06T07:19:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:19:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:19:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: client mount timeout = 600 2024-03-06T07:19:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: debug client = 20 2024-03-06T07:19:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-06T07:19:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: rados mon op timeout = 900 2024-03-06T07:19:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: rados osd op timeout = 900 2024-03-06T07:19:32.458 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2024-03-06T07:19:32.458 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.b] 2024-03-06T07:19:32.458 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.c] 2024-03-06T07:19:32.458 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a] 2024-03-06T07:19:32.458 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c] 2024-03-06T07:19:32.458 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b] 2024-03-06T07:19:32.458 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d] 2024-03-06T07:19:32.458 INFO:teuthology.orchestra.run.smithi098.stdout:[mds] 2024-03-06T07:19:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: debug mds = 20 2024-03-06T07:19:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: debug mds balancer = 20 2024-03-06T07:19:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-06T07:19:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: mds debug frag = True 2024-03-06T07:19:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: mds debug scatterstat = True 2024-03-06T07:19:32.459 INFO:teuthology.orchestra.run.smithi098.stdout: mds op complaint time = 180 2024-03-06T07:19:32.459 INFO:teuthology.orchestra.run.smithi098.stdout: mds verify scatter = True 2024-03-06T07:19:32.459 INFO:teuthology.orchestra.run.smithi098.stdout: osd op complaint time = 180 2024-03-06T07:19:32.459 INFO:teuthology.orchestra.run.smithi098.stdout: rados mon op timeout = 900 2024-03-06T07:19:32.459 INFO:teuthology.orchestra.run.smithi098.stdout: rados osd op timeout = 900 2024-03-06T07:19:32.460 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-06T07:19:32.461 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-06T07:19:32.555 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-06T07:19:32.556 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:19:32.556 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-06T07:19:32.570 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:19:32.570 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-06T07:19:32.624 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi001.front.sepia.ceph.com 2024-03-06T07:19:32.625 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:19:32.625 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:19:32.625 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:19:32.703 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:19:32.703 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:19:32.757 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2024-03-06T07:19:32.758 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:19:32.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:19:32.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:19:32.799 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:19:32.799 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:19:32.855 INFO:tasks.ceph:Setting up mon nodes... 2024-03-06T07:19:32.855 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-06T07:19:32.855 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:19:32.909 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:19:32.911 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:19:32.970 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:19:32.973 INFO:tasks.ceph:Setting up mds nodes... 2024-03-06T07:19:32.973 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:19:33.027 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:19:33.029 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-06T07:19:33.056 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:19:33.150 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:19:33.152 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-06T07:19:33.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:19:33.235 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:19:33.237 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-06T07:19:33.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:19:33.358 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:19:33.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-06T07:19:33.387 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-06T07:19:33.387 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-06T07:19:33.429 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-06T07:19:33.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-06T07:19:33.487 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-06T07:19:33.502 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-06T07:19:33.502 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:19:33.502 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-06T07:19:33.529 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:19:33.529 INFO:tasks.ceph:role: osd.0 2024-03-06T07:19:33.529 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-06T07:19:33.529 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:19:33.594 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:19:33.595 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:19:33.595 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:19:33.595 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:19:33.595 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:19:33.595 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:19:33.595 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:19:33.595 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:19:33.595 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:19:33.595 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:19:33.627 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-06T07:19:33.628 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-06T07:19:33.628 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-06T07:19:33.699 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-06T07:19:33.771 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-06T07:19:33.837 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:19:33.837 INFO:tasks.ceph:role: osd.1 2024-03-06T07:19:33.837 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-06T07:19:33.837 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:19:33.904 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:19:33.904 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:19:33.904 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:19:33.904 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:19:33.904 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:19:33.904 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:19:33.904 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:19:33.904 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:19:33.904 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:19:33.905 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:19:33.936 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-06T07:19:33.938 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-06T07:19:33.938 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-06T07:19:34.008 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-06T07:19:34.079 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-06T07:19:34.145 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:19:34.145 INFO:tasks.ceph:role: osd.2 2024-03-06T07:19:34.146 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-06T07:19:34.146 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:19:34.211 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:19:34.211 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:19:34.211 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:19:34.211 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:19:34.211 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:19:34.212 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:19:34.212 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:19:34.212 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:19:34.212 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:19:34.212 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:19:34.243 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-06T07:19:34.245 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-06T07:19:34.245 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-06T07:19:34.326 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-06T07:19:34.397 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-06T07:19:34.464 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:19:34.464 INFO:tasks.ceph:role: osd.3 2024-03-06T07:19:34.464 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-06T07:19:34.465 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:19:34.530 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:19:34.531 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:19:34.531 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:19:34.531 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:19:34.531 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:19:34.531 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:19:34.531 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:19:34.531 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:19:34.531 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:19:34.531 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:19:34.564 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-06T07:19:34.567 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-06T07:19:34.567 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-06T07:19:34.636 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-06T07:19:34.707 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:19:34.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:34.804+0000 7ff406e15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-06T07:19:34.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:34.804+0000 7ff406e15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-06T07:19:34.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:34.804+0000 7ff406e15640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:19:34.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:34.804+0000 7ff406e15640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:19:34.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:34.804+0000 7ff406e15640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:19:34.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:34.804+0000 7ff406e15640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-06T07:19:37.650 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-06T07:19:37.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:19:37.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:37.770+0000 7f24d7eb3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-06T07:19:37.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:37.771+0000 7f24d7eb3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-06T07:19:37.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:37.771+0000 7f24d7eb3640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:19:37.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:37.771+0000 7f24d7eb3640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:19:37.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:37.771+0000 7f24d7eb3640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:19:37.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:37.771+0000 7f24d7eb3640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-06T07:19:40.624 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-06T07:19:40.655 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:19:40.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:40.752+0000 7f049f620640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-06T07:19:40.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:40.752+0000 7f049f620640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-06T07:19:40.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:40.752+0000 7f049f620640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:19:40.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:40.752+0000 7f049f620640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:19:40.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:40.752+0000 7f049f620640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:19:40.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:40.752+0000 7f049f620640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-06T07:19:43.605 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-06T07:19:43.637 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:19:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:43.725+0000 7f7a8428b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-06T07:19:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:43.725+0000 7f7a8428b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-06T07:19:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:43.725+0000 7f7a8428b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:19:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:43.725+0000 7f7a8428b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:19:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:43.725+0000 7f7a8428b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:19:43.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:19:43.725+0000 7f7a8428b640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-06T07:19:46.572 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-06T07:19:46.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-06T07:19:46.637 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:19:46.637 INFO:tasks.ceph:role: osd.4 2024-03-06T07:19:46.637 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-06T07:19:46.637 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:19:46.715 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:19:46.715 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:19:46.715 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:19:46.715 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:19:46.715 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:19:46.715 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:19:46.715 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:19:46.715 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:19:46.715 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:19:46.716 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:19:47.233 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-06T07:19:47.235 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-06T07:19:47.235 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-06T07:19:47.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-06T07:19:47.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-06T07:19:47.445 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:19:47.445 INFO:tasks.ceph:role: osd.5 2024-03-06T07:19:47.445 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-06T07:19:47.446 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:19:47.520 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:19:47.520 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:19:47.520 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:19:47.520 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:19:47.520 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:19:47.520 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:19:47.520 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:19:47.520 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:19:47.521 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:19:47.521 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:19:47.981 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-06T07:19:47.983 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-06T07:19:47.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-06T07:19:48.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-06T07:19:48.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-06T07:19:48.169 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:19:48.169 INFO:tasks.ceph:role: osd.6 2024-03-06T07:19:48.169 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-06T07:19:48.169 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:19:48.243 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:19:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:19:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:19:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:19:48.243 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:19:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:19:48.244 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:19:48.244 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:19:48.244 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:19:48.244 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:19:48.729 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-06T07:19:48.731 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-06T07:19:48.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-06T07:19:48.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-06T07:19:48.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-06T07:19:48.972 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:19:48.972 INFO:tasks.ceph:role: osd.7 2024-03-06T07:19:48.972 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-06T07:19:48.972 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:19:49.046 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:19:49.046 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:19:49.046 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:19:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:19:49.047 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:19:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:19:49.047 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:19:49.047 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:19:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:19:49.047 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:19:49.573 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-06T07:19:49.575 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-06T07:19:49.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-06T07:19:49.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-06T07:19:49.747 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:19:49.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:49.850+0000 7f73ca819640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-06T07:19:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:49.850+0000 7f73ca819640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-06T07:19:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:49.850+0000 7f73ca819640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:19:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:49.850+0000 7f73ca819640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:19:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:49.850+0000 7f73ca819640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:19:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:49.851+0000 7f73ca819640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-06T07:19:52.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-06T07:19:52.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:19:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:52.829+0000 7f4336fbd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-06T07:19:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:52.829+0000 7f4336fbd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-06T07:19:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:52.829+0000 7f4336fbd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:19:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:52.829+0000 7f4336fbd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:19:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:52.829+0000 7f4336fbd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:19:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:52.830+0000 7f4336fbd640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-06T07:19:55.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-06T07:19:55.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:19:55.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:55.818+0000 7fb921219640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-06T07:19:55.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:55.819+0000 7fb921219640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-06T07:19:55.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:55.819+0000 7fb921219640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:19:55.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:55.819+0000 7fb921219640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:19:55.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:55.819+0000 7fb921219640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:19:55.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:55.819+0000 7fb921219640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-06T07:19:58.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-06T07:19:58.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:19:58.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:58.786+0000 7ff53160c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-06T07:19:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:58.787+0000 7ff53160c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-06T07:19:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:58.787+0000 7ff53160c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:19:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:58.787+0000 7ff53160c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:19:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:58.787+0000 7ff53160c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:19:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:19:58.787+0000 7ff53160c640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-06T07:20:00.474 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:00.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:01.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-06T07:20:01.679 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-06T07:20:01.680 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:20:01.680 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-06T07:20:01.707 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:20:01.708 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-06T07:20:01.774 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:20:01.774 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-06T07:20:01.839 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:20:01.840 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-06T07:20:01.905 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:20:01.906 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-06T07:20:01.971 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:20:01.972 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-06T07:20:02.038 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:20:02.038 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-06T07:20:02.103 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:20:02.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-06T07:20:02.135 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:20:02.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-06T07:20:02.208 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:20:02.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-06T07:20:02.277 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:20:02.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-06T07:20:02.346 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:20:02.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-06T07:20:02.415 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:20:02.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-06T07:20:02.483 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:20:02.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-06T07:20:02.551 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:20:02.551 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-06T07:20:02.567 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:20:02.567 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-06T07:20:02.607 INFO:tasks.ceph:Adding keys to all mons... 2024-03-06T07:20:02.607 DEBUG:teuthology.orchestra.run.smithi001:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:20:02.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:20:02.650 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.y] 2024-03-06T07:20:02.651 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAEGehlAl8rNhAAvdU0Qvo/FJYnxc4/H8MIqQ== 2024-03-06T07:20:02.651 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.a] 2024-03-06T07:20:02.651 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehlnxyWARAAKIPXU0IafNQ1RzBPX+3DYw== 2024-03-06T07:20:02.651 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.c] 2024-03-06T07:20:02.651 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehl+IzqCBAAmgLkarFGt9Ihs/8kJYicBQ== 2024-03-06T07:20:02.652 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.0] 2024-03-06T07:20:02.652 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAGGehlQK39LxAAOpPIZvl3FqQwCgyo3yoOGQ== 2024-03-06T07:20:02.652 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.1] 2024-03-06T07:20:02.652 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAJGehlTxwDLhAAMwN9E0+3Jh5XNXnLo+fw3g== 2024-03-06T07:20:02.652 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.2] 2024-03-06T07:20:02.652 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAMGehl4qrlLBAAm0VWQRNvSQIpDx5xf4Fyig== 2024-03-06T07:20:02.652 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.3] 2024-03-06T07:20:02.652 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAPGehllO5HKxAAozm6ycG+fRxatoZTogeJTA== 2024-03-06T07:20:02.652 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.x] 2024-03-06T07:20:02.652 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAEGehlJ/jPORAAhLk5MTGafsGKhZSYhvhBZg== 2024-03-06T07:20:02.652 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.b] 2024-03-06T07:20:02.652 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehl2fXxDRAAuLqH+HpKcN1JTts9ZtDTXA== 2024-03-06T07:20:02.653 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.d] 2024-03-06T07:20:02.653 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehlipdRFRAAA34vrsTEoX3L1CKb1YAV9g== 2024-03-06T07:20:02.653 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.4] 2024-03-06T07:20:02.653 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAVGehlLtLBMhAAUrV6q6ZCQluIcoo/S/NdbA== 2024-03-06T07:20:02.653 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.5] 2024-03-06T07:20:02.653 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAYGehlHp59MRAAxxgpSPTEZkYKjl3fr0xeeQ== 2024-03-06T07:20:02.653 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.6] 2024-03-06T07:20:02.653 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAbGehlTnXcMBAAFOBqcnh5ubAz+o6AGjccWQ== 2024-03-06T07:20:02.653 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.7] 2024-03-06T07:20:02.653 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAeGehl1qH1LhAAaSl7Aj7qlN6d4xt5SKAmcg== 2024-03-06T07:20:02.653 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-06T07:20:02.653 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng== 2024-03-06T07:20:02.653 INFO:teuthology.orchestra.run.smithi001.stdout:[client.1] 2024-03-06T07:20:02.653 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehlMyv7HBAAi7vt7aZWEaZuMpFrB5ofZw== 2024-03-06T07:20:02.672 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.y] 2024-03-06T07:20:02.672 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAEGehlAl8rNhAAvdU0Qvo/FJYnxc4/H8MIqQ== 2024-03-06T07:20:02.672 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a] 2024-03-06T07:20:02.672 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAFGehlnxyWARAAKIPXU0IafNQ1RzBPX+3DYw== 2024-03-06T07:20:02.673 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c] 2024-03-06T07:20:02.673 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAFGehl+IzqCBAAmgLkarFGt9Ihs/8kJYicBQ== 2024-03-06T07:20:02.673 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.0] 2024-03-06T07:20:02.673 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAGGehlQK39LxAAOpPIZvl3FqQwCgyo3yoOGQ== 2024-03-06T07:20:02.673 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.1] 2024-03-06T07:20:02.673 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAJGehlTxwDLhAAMwN9E0+3Jh5XNXnLo+fw3g== 2024-03-06T07:20:02.673 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.2] 2024-03-06T07:20:02.673 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAMGehl4qrlLBAAm0VWQRNvSQIpDx5xf4Fyig== 2024-03-06T07:20:02.674 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.3] 2024-03-06T07:20:02.674 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAPGehllO5HKxAAozm6ycG+fRxatoZTogeJTA== 2024-03-06T07:20:02.674 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.x] 2024-03-06T07:20:02.674 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAEGehlJ/jPORAAhLk5MTGafsGKhZSYhvhBZg== 2024-03-06T07:20:02.674 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b] 2024-03-06T07:20:02.674 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAFGehl2fXxDRAAuLqH+HpKcN1JTts9ZtDTXA== 2024-03-06T07:20:02.674 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d] 2024-03-06T07:20:02.674 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAFGehlipdRFRAAA34vrsTEoX3L1CKb1YAV9g== 2024-03-06T07:20:02.674 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.4] 2024-03-06T07:20:02.675 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAVGehlLtLBMhAAUrV6q6ZCQluIcoo/S/NdbA== 2024-03-06T07:20:02.675 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.5] 2024-03-06T07:20:02.675 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAYGehlHp59MRAAxxgpSPTEZkYKjl3fr0xeeQ== 2024-03-06T07:20:02.675 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.6] 2024-03-06T07:20:02.675 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAbGehlTnXcMBAAFOBqcnh5ubAz+o6AGjccWQ== 2024-03-06T07:20:02.675 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.7] 2024-03-06T07:20:02.675 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAeGehl1qH1LhAAaSl7Aj7qlN6d4xt5SKAmcg== 2024-03-06T07:20:02.675 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-03-06T07:20:02.676 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng== 2024-03-06T07:20:02.676 INFO:teuthology.orchestra.run.smithi098.stdout:[client.1] 2024-03-06T07:20:02.676 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAFGehlMyv7HBAAi7vt7aZWEaZuMpFrB5ofZw== 2024-03-06T07:20:02.677 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:20:02.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:20:02.770 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:20:02.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:20:02.827 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:20:02.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:20:02.884 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:20:02.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:20:02.938 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:20:02.940 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:20:02.992 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:20:02.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:20:03.045 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:20:03.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:20:03.099 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:20:03.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:20:03.154 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:20:03.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:20:03.210 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:20:03.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:20:03.266 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:20:03.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:20:03.320 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:20:03.322 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:20:03.374 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:20:03.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:20:03.429 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:20:03.431 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:20:03.483 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:20:03.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:20:03.539 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:20:03.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:20:03.597 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-06T07:20:03.597 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-06T07:20:03.624 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:20:03.721 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-06T07:20:03.747 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-06T07:20:03.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:20:03.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-06T07:20:03.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-06T07:20:03.966 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:20:04.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-06T07:20:04.091 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:04.093 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:04.146 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-06T07:20:04.147 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-06T07:20:04.147 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-06T07:20:04.149 INFO:tasks.ceph.mon.a:Started 2024-03-06T07:20:04.149 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-06T07:20:04.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-06T07:20:04.189 INFO:tasks.ceph.mon.b:Started 2024-03-06T07:20:04.189 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-06T07:20:04.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-06T07:20:04.191 INFO:tasks.ceph.mon.c:Started 2024-03-06T07:20:04.191 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-06T07:20:04.192 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-06T07:20:04.192 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-06T07:20:04.194 INFO:tasks.ceph.mgr.y:Started 2024-03-06T07:20:04.194 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-06T07:20:04.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-06T07:20:04.196 INFO:tasks.ceph.mgr.x:Started 2024-03-06T07:20:04.196 INFO:tasks.ceph:Setting crush tunables to default 2024-03-06T07:20:04.196 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph osd crush tunables default 2024-03-06T07:20:04.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.296+0000 7f562e25c640 1 Processor -- start 2024-03-06T07:20:04.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.296+0000 7f562e25c640 1 -- start start 2024-03-06T07:20:04.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.297+0000 7f562e25c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5628152820 0x7f56280c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.297+0000 7f562e25c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56280c0670 0x7f56280c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.297+0000 7f562e25c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56280c7180 con 0x7f56280c7930 2024-03-06T07:20:04.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.297+0000 7f562e25c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5628155460 con 0x7f5628152820 2024-03-06T07:20:04.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.297+0000 7f562e25c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56280c6bc0 con 0x7f56280c0670 2024-03-06T07:20:04.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.297+0000 7f56277fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5628152820 0x7f56280c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.297+0000 7f56277fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5628152820 0x7f56280c0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44530/0 (socket says 172.21.15.1:44530) 2024-03-06T07:20:04.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.297+0000 7f56277fe640 1 -- 172.21.15.1:0/1904060917 learned_addr learned my addr 172.21.15.1:0/1904060917 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:04.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.297+0000 7f56277fe640 1 -- 172.21.15.1:0/1904060917 >> v2:172.21.15.1:3300/0 conn(0x7f5628152820 msgr2=0x7f56280c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:20:04.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.297+0000 7f56277fe640 1 -- 172.21.15.1:0/1904060917 >> v2:172.21.15.1:3300/0 conn(0x7f5628152820 msgr2=0x7f56280c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:20:04.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.297+0000 7f56277fe640 1 --2- 172.21.15.1:0/1904060917 >> v2:172.21.15.1:3300/0 conn(0x7f5628152820 0x7f56280c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:20:04.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.297+0000 7f56277fe640 1 --2- 172.21.15.1:0/1904060917 >> v2:172.21.15.1:3300/0 conn(0x7f5628152820 0x7f56280c0130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:20:04.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.297+0000 7f5626ffd640 1 --2- 172.21.15.1:0/1904060917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56280c0670 0x7f56280c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.298+0000 7f5626ffd640 1 -- 172.21.15.1:0/1904060917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56280c0670 msgr2=0x7f56280c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-06T07:20:04.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.298+0000 7f5626ffd640 1 -- 172.21.15.1:0/1904060917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56280c0670 msgr2=0x7f56280c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:20:04.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.298+0000 7f5626ffd640 1 --2- 172.21.15.1:0/1904060917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56280c0670 0x7f56280c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:20:04.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.298+0000 7f5626ffd640 1 --2- 172.21.15.1:0/1904060917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56280c0670 0x7f56280c73f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:20:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.476+0000 7f56267fc640 1 -- 172.21.15.1:0/1904060917 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156364841 0 0) 0x7f56280c7180 con 0x7f56280c7930 2024-03-06T07:20:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.476+0000 7f56267fc640 1 -- 172.21.15.1:0/1904060917 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5610003520 con 0x7f56280c7930 2024-03-06T07:20:04.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56267fc640 1 -- 172.21.15.1:0/1904060917 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3488850744 0 0) 0x7f5610003520 con 0x7f56280c7930 2024-03-06T07:20:04.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56267fc640 1 -- 172.21.15.1:0/1904060917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56280c0670 msgr2=0x7f56280c73f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:20:04.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56267fc640 1 --2- 172.21.15.1:0/1904060917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56280c0670 0x7f56280c73f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56267fc640 1 -- 172.21.15.1:0/1904060917 >> v2:172.21.15.1:3300/0 conn(0x7f5628152820 msgr2=0x7f56280c0130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:20:04.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56267fc640 1 --2- 172.21.15.1:0/1904060917 >> v2:172.21.15.1:3300/0 conn(0x7f5628152820 0x7f56280c0130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56267fc640 1 -- 172.21.15.1:0/1904060917 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56280c6ea0 con 0x7f56280c7930 2024-03-06T07:20:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56267fc640 1 -- 172.21.15.1:0/1904060917 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f56200019f0 con 0x7f56280c7930 2024-03-06T07:20:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56267fc640 1 -- 172.21.15.1:0/1904060917 >> v1:172.21.15.98:6789/0 conn(0x7f56280c7930 legacy=0x7f5628158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56267fc640 1 --2- 172.21.15.1:0/1904060917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56100039b0 0x7f5610003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56267fc640 1 --2- 172.21.15.1:0/1904060917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56280c0670 0x7f5610004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56267fc640 1 --2- 172.21.15.1:0/1904060917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5610004d40 0x7f56100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56267fc640 1 -- 172.21.15.1:0/1904060917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5628155460 con 0x7f56100039b0 2024-03-06T07:20:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56267fc640 1 -- 172.21.15.1:0/1904060917 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56280c6bc0 con 0x7f5610004d40 2024-03-06T07:20:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56267fc640 1 -- 172.21.15.1:0/1904060917 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f561000baf0 con 0x7f56280c0670 2024-03-06T07:20:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56277fe640 1 --2- 172.21.15.1:0/1904060917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56100039b0 0x7f5610003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f5627fff640 1 --2- 172.21.15.1:0/1904060917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5610004d40 0x7f56100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f5626ffd640 1 --2- 172.21.15.1:0/1904060917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56280c0670 0x7f5610004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56277fe640 1 -- 172.21.15.1:0/1904060917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56280c0670 msgr2=0x7f5610004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56277fe640 1 --2- 172.21.15.1:0/1904060917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56280c0670 0x7f5610004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56277fe640 1 -- 172.21.15.1:0/1904060917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5610004d40 msgr2=0x7f56100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56277fe640 1 --2- 172.21.15.1:0/1904060917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5610004d40 0x7f56100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f56277fe640 1 -- 172.21.15.1:0/1904060917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5610003520 con 0x7f56100039b0 2024-03-06T07:20:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.480+0000 7f5627fff640 1 --2- 172.21.15.1:0/1904060917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5610004d40 0x7f56100051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.481+0000 7f56277fe640 1 --2- 172.21.15.1:0/1904060917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56100039b0 0x7f5610003dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5618009890 tx=0x7f5610005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72fd35524f7c5f8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.481+0000 7f56267fc640 1 -- 172.21.15.1:0/1904060917 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5618034070 con 0x7f56100039b0 2024-03-06T07:20:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.481+0000 7f56267fc640 1 -- 172.21.15.1:0/1904060917 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5618030680 con 0x7f56100039b0 2024-03-06T07:20:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.481+0000 7f56267fc640 1 -- 172.21.15.1:0/1904060917 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561800f040 con 0x7f56100039b0 2024-03-06T07:20:04.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.482+0000 7f562e25c640 1 -- 172.21.15.1:0/1904060917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56100039b0 msgr2=0x7f5610003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.482+0000 7f562e25c640 1 --2- 172.21.15.1:0/1904060917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56100039b0 0x7f5610003dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5618009890 tx=0x7f5610005830 comp rx=0 tx=0).stop 2024-03-06T07:20:04.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.482+0000 7f5627fff640 1 -- 172.21.15.1:0/1904060917 reap_dead start 2024-03-06T07:20:04.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.482+0000 7f562e25c640 1 -- 172.21.15.1:0/1904060917 shutdown_connections 2024-03-06T07:20:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.482+0000 7f562e25c640 1 -- 172.21.15.1:0/1904060917 >> 172.21.15.1:0/1904060917 conn(0x7f5628072720 msgr2=0x7f5628072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.482+0000 7f562e25c640 1 -- 172.21.15.1:0/1904060917 shutdown_connections 2024-03-06T07:20:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.482+0000 7f562e25c640 1 -- 172.21.15.1:0/1904060917 wait complete. 2024-03-06T07:20:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.483+0000 7f562e25c640 1 Processor -- start 2024-03-06T07:20:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.484+0000 7f562e25c640 1 -- start start 2024-03-06T07:20:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.484+0000 7f562e25c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56100039b0 0x7f56281566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.484+0000 7f562e25c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5628157620 0x7f5628157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.484+0000 7f562e25c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56281ee0e0 0x7f56281ee560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.484+0000 7f562e25c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5628152d80 con 0x7f56281ee0e0 2024-03-06T07:20:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.484+0000 7f562e25c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5628152c00 con 0x7f5628157620 2024-03-06T07:20:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.484+0000 7f562e25c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56280af6f0 con 0x7f56100039b0 2024-03-06T07:20:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.484+0000 7f5627fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56281ee0e0 0x7f56281ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.484+0000 7f5627fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56281ee0e0 0x7f56281ee560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60110/0 (socket says 172.21.15.1:60110) 2024-03-06T07:20:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.484+0000 7f5627fff640 1 -- 172.21.15.1:0/3485501407 learned_addr learned my addr 172.21.15.1:0/3485501407 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.484+0000 7f56277fe640 1 --2- 172.21.15.1:0/3485501407 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56100039b0 0x7f56281566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.484+0000 7f5626ffd640 1 --2- 172.21.15.1:0/3485501407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5628157620 0x7f5628157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.485+0000 7f5627fff640 1 -- 172.21.15.1:0/3485501407 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56100039b0 msgr2=0x7f56281566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.485+0000 7f5627fff640 1 --2- 172.21.15.1:0/3485501407 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56100039b0 0x7f56281566c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.485+0000 7f5627fff640 1 -- 172.21.15.1:0/3485501407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5628157620 msgr2=0x7f5628157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.485+0000 7f5627fff640 1 --2- 172.21.15.1:0/3485501407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5628157620 0x7f5628157aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.485+0000 7f5627fff640 1 -- 172.21.15.1:0/3485501407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56280aae50 con 0x7f56281ee0e0 2024-03-06T07:20:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.485+0000 7f5627fff640 1 --2- 172.21.15.1:0/3485501407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56281ee0e0 0x7f56281ee560 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f562806c030 tx=0x7f562000f1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.485+0000 7f560ffff640 1 -- 172.21.15.1:0/3485501407 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562001a070 con 0x7f56281ee0e0 2024-03-06T07:20:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.485+0000 7f560ffff640 1 -- 172.21.15.1:0/3485501407 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5620003b80 con 0x7f56281ee0e0 2024-03-06T07:20:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.485+0000 7f562e25c640 1 -- 172.21.15.1:0/3485501407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5628156d70 con 0x7f56281ee0e0 2024-03-06T07:20:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.485+0000 7f560ffff640 1 -- 172.21.15.1:0/3485501407 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562000c0e0 con 0x7f56281ee0e0 2024-03-06T07:20:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.485+0000 7f562e25c640 1 -- 172.21.15.1:0/3485501407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56280c35a0 con 0x7f56281ee0e0 2024-03-06T07:20:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.485+0000 7f560ffff640 1 -- 172.21.15.1:0/3485501407 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f562001d070 con 0x7f56281ee0e0 2024-03-06T07:20:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.485+0000 7f560ffff640 1 -- 172.21.15.1:0/3485501407 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5620005580 con 0x7f56281ee0e0 2024-03-06T07:20:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.486+0000 7f562e25c640 1 -- 172.21.15.1:0/3485501407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56280cb850 con 0x7f56281ee0e0 2024-03-06T07:20:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.488+0000 7f560ffff640 1 -- 172.21.15.1:0/3485501407 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f56280cb850 con 0x7f56281ee0e0 2024-03-06T07:20:04.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.555+0000 7f562e25c640 1 -- 172.21.15.1:0/3485501407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f56280686e0 con 0x7f56281ee0e0 2024-03-06T07:20:04.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.556+0000 7f560ffff640 1 -- 172.21.15.1:0/3485501407 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5620024070 con 0x7f56281ee0e0 2024-03-06T07:20:04.558 INFO:teuthology.orchestra.run.smithi001.stderr:adjusted tunables profile to default 2024-03-06T07:20:04.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.557+0000 7f562e25c640 1 -- 172.21.15.1:0/3485501407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56281ee0e0 msgr2=0x7f56281ee560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.557+0000 7f562e25c640 1 --2- 172.21.15.1:0/3485501407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56281ee0e0 0x7f56281ee560 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f562806c030 tx=0x7f562000f1d0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.557+0000 7f562e25c640 1 -- 172.21.15.1:0/3485501407 shutdown_connections 2024-03-06T07:20:04.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.557+0000 7f562e25c640 1 --2- 172.21.15.1:0/3485501407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56281ee0e0 0x7f56281ee560 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.557+0000 7f562e25c640 1 --2- 172.21.15.1:0/3485501407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5628157620 0x7f5628157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.557+0000 7f562e25c640 1 --2- 172.21.15.1:0/3485501407 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56100039b0 0x7f56281566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.557+0000 7f562e25c640 1 -- 172.21.15.1:0/3485501407 >> 172.21.15.1:0/3485501407 conn(0x7f5628072720 msgr2=0x7f5628072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:04.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.557+0000 7f562e25c640 1 -- 172.21.15.1:0/3485501407 shutdown_connections 2024-03-06T07:20:04.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:04.557+0000 7f562e25c640 1 -- 172.21.15.1:0/3485501407 wait complete. 2024-03-06T07:20:04.569 INFO:tasks.ceph:check_enable_crimson: False 2024-03-06T07:20:04.569 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-06T07:20:04.569 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:20:04.570 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-06T07:20:04.596 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:20:04.596 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-06T07:20:04.663 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:20:04.663 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-06T07:20:04.728 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:20:04.729 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-06T07:20:04.795 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:20:04.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-06T07:20:04.822 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:20:04.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-06T07:20:04.888 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:20:04.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-06T07:20:04.954 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:04.952+0000 7fe965466040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:20:04.955 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:20:04.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-06T07:20:05.009 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:05.008+0000 7fe965466040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:20:05.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 6b306d19-7ed3-4551-b20a-863757aace9e 0 2024-03-06T07:20:05.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.160+0000 7ff14b59e640 1 Processor -- start 2024-03-06T07:20:05.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.161+0000 7ff14b59e640 1 -- start start 2024-03-06T07:20:05.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.161+0000 7ff14b59e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff14c0c3120 0x7ff14c06dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.161+0000 7ff14b59e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff14c155980 0x7ff14c0ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.161+0000 7ff14b59e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c072d00 con 0x7ff14c152480 2024-03-06T07:20:05.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.161+0000 7ff14b59e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14c072ab0 con 0x7ff14c155980 2024-03-06T07:20:05.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.161+0000 7ff14b59e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14c0c1a20 con 0x7ff14c0c3120 2024-03-06T07:20:05.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.162+0000 7ff14a59c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff14c152480 0x7ff14c0c2470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47948/0 (socket says 172.21.15.98:47948) 2024-03-06T07:20:05.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.162+0000 7ff14a59c640 1 -- 172.21.15.98:0/2866141929 learned_addr learned my addr 172.21.15.98:0/2866141929 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-06T07:20:05.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.162+0000 7ff149d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff14c0c3120 0x7ff14c06dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.162+0000 7ff14ad9d640 1 --2- 172.21.15.98:0/2866141929 >> v2:172.21.15.1:3300/0 conn(0x7ff14c155980 0x7ff14c0ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.162+0000 7ff149d9b640 1 -- 172.21.15.98:0/2866141929 >> v2:172.21.15.1:3300/0 conn(0x7ff14c155980 msgr2=0x7ff14c0ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:05.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.162+0000 7ff149d9b640 1 --2- 172.21.15.98:0/2866141929 >> v2:172.21.15.1:3300/0 conn(0x7ff14c155980 0x7ff14c0ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.162+0000 7ff149d9b640 1 -- 172.21.15.98:0/2866141929 >> v1:172.21.15.98:6789/0 conn(0x7ff14c152480 legacy=0x7ff14c0c2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.162+0000 7ff14959a640 1 -- 172.21.15.98:0/2866141929 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212257130 0 0) 0x7ff14c072d00 con 0x7ff14c152480 2024-03-06T07:20:05.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.162+0000 7ff149d9b640 1 -- 172.21.15.98:0/2866141929 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c072fe0 con 0x7ff14c0c3120 2024-03-06T07:20:05.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.163+0000 7ff149d9b640 1 --2- 172.21.15.98:0/2866141929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff14c0c3120 0x7ff14c06dd30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff14c0c38a0 tx=0x7ff13c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d49179a5e364872 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.163+0000 7ff14959a640 1 -- 172.21.15.98:0/2866141929 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c035020 con 0x7ff14c0c3120 2024-03-06T07:20:05.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.163+0000 7ff14959a640 1 -- 172.21.15.98:0/2866141929 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13c031830 con 0x7ff14c0c3120 2024-03-06T07:20:05.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.163+0000 7ff14959a640 1 -- 172.21.15.98:0/2866141929 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c031c00 con 0x7ff14c0c3120 2024-03-06T07:20:05.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.164+0000 7ff14b59e640 1 -- 172.21.15.98:0/2866141929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff14c0c3120 msgr2=0x7ff14c06dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.164+0000 7ff14b59e640 1 --2- 172.21.15.98:0/2866141929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff14c0c3120 0x7ff14c06dd30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff14c0c38a0 tx=0x7ff13c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.164+0000 7ff14b59e640 1 -- 172.21.15.98:0/2866141929 shutdown_connections 2024-03-06T07:20:05.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.164+0000 7ff14b59e640 1 --2- 172.21.15.98:0/2866141929 >> v2:172.21.15.1:3300/0 conn(0x7ff14c155980 0x7ff14c0ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.164+0000 7ff14b59e640 1 --2- 172.21.15.98:0/2866141929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff14c0c3120 0x7ff14c06dd30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.164+0000 7ff14b59e640 1 -- 172.21.15.98:0/2866141929 >> 172.21.15.98:0/2866141929 conn(0x7ff14c071b80 msgr2=0x7ff14c071f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:05.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.164+0000 7ff14b59e640 1 -- 172.21.15.98:0/2866141929 shutdown_connections 2024-03-06T07:20:05.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.164+0000 7ff14b59e640 1 -- 172.21.15.98:0/2866141929 wait complete. 2024-03-06T07:20:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.165+0000 7ff14b59e640 1 Processor -- start 2024-03-06T07:20:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.165+0000 7ff14b59e640 1 -- start start 2024-03-06T07:20:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.166+0000 7ff14b59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff14c0c3120 0x7ff14c14c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.166+0000 7ff14b59e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff14c152480 0x7ff14c1eb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.166+0000 7ff14b59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff14c155980 0x7ff14c156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.166+0000 7ff14b59e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0c1f50 con 0x7ff14c155980 2024-03-06T07:20:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.166+0000 7ff14b59e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0c2b80 con 0x7ff14c0c3120 2024-03-06T07:20:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.166+0000 7ff14b59e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c155db0 con 0x7ff14c152480 2024-03-06T07:20:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.166+0000 7ff14a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff14c0c3120 0x7ff14c14c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.166+0000 7ff14a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff14c0c3120 0x7ff14c14c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43974/0 (socket says 172.21.15.98:43974) 2024-03-06T07:20:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.166+0000 7ff14a59c640 1 -- 172.21.15.98:0/2496848804 learned_addr learned my addr 172.21.15.98:0/2496848804 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-06T07:20:05.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.166+0000 7ff149d9b640 1 --2- 172.21.15.98:0/2496848804 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff14c152480 0x7ff14c1eb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.166+0000 7ff14ad9d640 1 --2- 172.21.15.98:0/2496848804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff14c155980 0x7ff14c156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.166+0000 7ff14a59c640 1 -- 172.21.15.98:0/2496848804 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff14c152480 msgr2=0x7ff14c1eb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.166+0000 7ff14a59c640 1 --2- 172.21.15.98:0/2496848804 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff14c152480 0x7ff14c1eb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.166+0000 7ff14a59c640 1 -- 172.21.15.98:0/2496848804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff14c155980 msgr2=0x7ff14c156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.166+0000 7ff14a59c640 1 --2- 172.21.15.98:0/2496848804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff14c155980 0x7ff14c156f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.166+0000 7ff14a59c640 1 -- 172.21.15.98:0/2496848804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c06e8f0 con 0x7ff14c0c3120 2024-03-06T07:20:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.167+0000 7ff14a59c640 1 --2- 172.21.15.98:0/2496848804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff14c0c3120 0x7ff14c14c720 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff14000c4d0 tx=0x7ff14000c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.167+0000 7ff13affd640 1 -- 172.21.15.98:0/2496848804 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff140018070 con 0x7ff14c0c3120 2024-03-06T07:20:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.167+0000 7ff13affd640 1 -- 172.21.15.98:0/2496848804 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff140007500 con 0x7ff14c0c3120 2024-03-06T07:20:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.167+0000 7ff14b59e640 1 -- 172.21.15.98:0/2496848804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14c14eff0 con 0x7ff14c0c3120 2024-03-06T07:20:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.167+0000 7ff13affd640 1 -- 172.21.15.98:0/2496848804 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff140004a20 con 0x7ff14c0c3120 2024-03-06T07:20:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.167+0000 7ff14b59e640 1 -- 172.21.15.98:0/2496848804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c14ed10 con 0x7ff14c0c3120 2024-03-06T07:20:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.167+0000 7ff13affd640 1 -- 172.21.15.98:0/2496848804 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff14001b070 con 0x7ff14c0c3120 2024-03-06T07:20:05.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.167+0000 7ff13affd640 1 -- 172.21.15.98:0/2496848804 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff140012400 con 0x7ff14c0c3120 2024-03-06T07:20:05.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.168+0000 7ff14b59e640 1 -- 172.21.15.98:0/2496848804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14c072890 con 0x7ff14c0c3120 2024-03-06T07:20:05.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.172+0000 7ff13affd640 1 -- 172.21.15.98:0/2496848804 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff14001d440 con 0x7ff14c0c3120 2024-03-06T07:20:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.244+0000 7ff14b59e640 1 -- 172.21.15.98:0/2496848804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6b306d19-7ed3-4551-b20a-863757aace9e", "id": 0} v 0) v1 -- 0x7ff14c14af70 con 0x7ff14c0c3120 2024-03-06T07:20:05.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.247+0000 7ff13affd640 1 -- 172.21.15.98:0/2496848804 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6b306d19-7ed3-4551-b20a-863757aace9e", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff14001d6f0 con 0x7ff14c0c3120 2024-03-06T07:20:05.249 INFO:teuthology.orchestra.run.smithi098.stdout:0 2024-03-06T07:20:05.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.248+0000 7ff14b59e640 1 -- 172.21.15.98:0/2496848804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff14c0c3120 msgr2=0x7ff14c14c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.248+0000 7ff14b59e640 1 --2- 172.21.15.98:0/2496848804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff14c0c3120 0x7ff14c14c720 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff14000c4d0 tx=0x7ff14000c990 comp rx=0 tx=0).stop 2024-03-06T07:20:05.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.248+0000 7ff14b59e640 1 -- 172.21.15.98:0/2496848804 shutdown_connections 2024-03-06T07:20:05.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.248+0000 7ff14b59e640 1 --2- 172.21.15.98:0/2496848804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff14c155980 0x7ff14c156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.248+0000 7ff14b59e640 1 --2- 172.21.15.98:0/2496848804 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff14c152480 0x7ff14c1eb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.248+0000 7ff14b59e640 1 --2- 172.21.15.98:0/2496848804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff14c0c3120 0x7ff14c14c720 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.248+0000 7ff14b59e640 1 -- 172.21.15.98:0/2496848804 >> 172.21.15.98:0/2496848804 conn(0x7ff14c071b80 msgr2=0x7ff14c154f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:05.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.248+0000 7ff14b59e640 1 -- 172.21.15.98:0/2496848804 shutdown_connections 2024-03-06T07:20:05.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.248+0000 7ff14b59e640 1 -- 172.21.15.98:0/2496848804 wait complete. 2024-03-06T07:20:05.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 7706d997-9bc6-4ed2-bfa0-3f7aec78afdb 1 2024-03-06T07:20:05.344 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:05.341+0000 7f52f5642040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:20:05.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.358+0000 7fec34525640 1 Processor -- start 2024-03-06T07:20:05.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.358+0000 7fec34525640 1 -- start start 2024-03-06T07:20:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.358+0000 7fec34525640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fec2c0c8780 0x7fec2c0c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.358+0000 7fec34525640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c14d550 con 0x7fec2c070710 2024-03-06T07:20:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.358+0000 7fec34525640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c153070 con 0x7fec2c14d170 2024-03-06T07:20:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.358+0000 7fec34525640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec2c153330 con 0x7fec2c0c8780 2024-03-06T07:20:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.359+0000 7fec322c1640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fec2c14d170 0x7fec2c0c32a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47964/0 (socket says 172.21.15.98:47964) 2024-03-06T07:20:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.359+0000 7fec322c1640 1 -- 172.21.15.98:0/462196742 learned_addr learned my addr 172.21.15.98:0/462196742 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-06T07:20:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.359+0000 7fec31ac0640 1 --2- 172.21.15.98:0/462196742 >> v2:172.21.15.1:3300/0 conn(0x7fec2c0c8780 0x7fec2c0c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.359+0000 7fec312bf640 1 -- 172.21.15.98:0/462196742 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285006397 0 0) 0x7fec2c153070 con 0x7fec2c14d170 2024-03-06T07:20:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.359+0000 7fec312bf640 1 -- 172.21.15.98:0/462196742 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec18003520 con 0x7fec2c14d170 2024-03-06T07:20:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.359+0000 7fec312bf640 1 -- 172.21.15.98:0/462196742 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323078014 0 0) 0x7fec24001490 con 0x7fec2c070710 2024-03-06T07:20:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.359+0000 7fec312bf640 1 -- 172.21.15.98:0/462196742 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec2c153070 con 0x7fec2c070710 2024-03-06T07:20:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.359+0000 7fec312bf640 1 -- 172.21.15.98:0/462196742 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2646178419 0 0) 0x7fec18003520 con 0x7fec2c14d170 2024-03-06T07:20:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.359+0000 7fec31ac0640 1 -- 172.21.15.98:0/462196742 >> v1:172.21.15.98:6789/0 conn(0x7fec2c14d170 legacy=0x7fec2c0c32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.359+0000 7fec31ac0640 1 -- 172.21.15.98:0/462196742 >> v1:172.21.15.1:6789/0 conn(0x7fec2c070710 legacy=0x7fec2c070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.359+0000 7fec31ac0640 1 -- 172.21.15.98:0/462196742 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c153610 con 0x7fec2c0c8780 2024-03-06T07:20:05.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.360+0000 7fec31ac0640 1 --2- 172.21.15.98:0/462196742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c0c8780 0x7fec2c0c8c30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fec28009720 tx=0x7fec2802b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a93fe54a9eec98d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.360+0000 7fec312bf640 1 -- 172.21.15.98:0/462196742 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec28037070 con 0x7fec2c0c8780 2024-03-06T07:20:05.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.360+0000 7fec312bf640 1 -- 172.21.15.98:0/462196742 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec28030510 con 0x7fec2c0c8780 2024-03-06T07:20:05.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.360+0000 7fec312bf640 1 -- 172.21.15.98:0/462196742 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c001670 con 0x7fec2c0c8780 2024-03-06T07:20:05.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.361+0000 7fec34525640 1 -- 172.21.15.98:0/462196742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c0c8780 msgr2=0x7fec2c0c8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.361+0000 7fec34525640 1 --2- 172.21.15.98:0/462196742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c0c8780 0x7fec2c0c8c30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fec28009720 tx=0x7fec2802b6a0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.361+0000 7fec34525640 1 -- 172.21.15.98:0/462196742 shutdown_connections 2024-03-06T07:20:05.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.361+0000 7fec34525640 1 --2- 172.21.15.98:0/462196742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c0c8780 0x7fec2c0c8c30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.361+0000 7fec34525640 1 -- 172.21.15.98:0/462196742 >> 172.21.15.98:0/462196742 conn(0x7fec2c0aa5a0 msgr2=0x7fec2c0aa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:05.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.361+0000 7fec34525640 1 -- 172.21.15.98:0/462196742 shutdown_connections 2024-03-06T07:20:05.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.361+0000 7fec34525640 1 -- 172.21.15.98:0/462196742 wait complete. 2024-03-06T07:20:05.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.362+0000 7fec34525640 1 Processor -- start 2024-03-06T07:20:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.362+0000 7fec34525640 1 -- start start 2024-03-06T07:20:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.362+0000 7fec34525640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec2c070710 0x7fec2c14faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.362+0000 7fec34525640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec2c0c8780 0x7fec2c14c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.362+0000 7fec34525640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c14d170 0x7fec2c14aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.362+0000 7fec34525640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c071260 con 0x7fec2c14d170 2024-03-06T07:20:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.362+0000 7fec34525640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c153bd0 con 0x7fec2c0c8780 2024-03-06T07:20:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.362+0000 7fec34525640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c0c39b0 con 0x7fec2c070710 2024-03-06T07:20:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec31ac0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec2c0c8780 0x7fec2c14c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec322c1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec2c070710 0x7fec2c14faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec31ac0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec2c0c8780 0x7fec2c14c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43984/0 (socket says 172.21.15.98:43984) 2024-03-06T07:20:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec322c1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec2c070710 0x7fec2c14faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:47312/0 (socket says 172.21.15.98:47312) 2024-03-06T07:20:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec31ac0640 1 -- 172.21.15.98:0/4117808067 learned_addr learned my addr 172.21.15.98:0/4117808067 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-06T07:20:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec32ac2640 1 --2- 172.21.15.98:0/4117808067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c14d170 0x7fec2c14aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec322c1640 1 -- 172.21.15.98:0/4117808067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec2c0c8780 msgr2=0x7fec2c14c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec322c1640 1 --2- 172.21.15.98:0/4117808067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec2c0c8780 0x7fec2c14c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec322c1640 1 -- 172.21.15.98:0/4117808067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c14d170 msgr2=0x7fec2c14aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec322c1640 1 --2- 172.21.15.98:0/4117808067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c14d170 0x7fec2c14aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec322c1640 1 -- 172.21.15.98:0/4117808067 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c14c9c0 con 0x7fec2c070710 2024-03-06T07:20:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec31ac0640 1 --2- 172.21.15.98:0/4117808067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec2c0c8780 0x7fec2c14c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec322c1640 1 --2- 172.21.15.98:0/4117808067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec2c070710 0x7fec2c14faa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00c520 tx=0x7fec1c00c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec22ffd640 1 -- 172.21.15.98:0/4117808067 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c018070 con 0x7fec2c070710 2024-03-06T07:20:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec22ffd640 1 -- 172.21.15.98:0/4117808067 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1c0074e0 con 0x7fec2c070710 2024-03-06T07:20:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec34525640 1 -- 172.21.15.98:0/4117808067 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2c1506d0 con 0x7fec2c070710 2024-03-06T07:20:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec22ffd640 1 -- 172.21.15.98:0/4117808067 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c0049e0 con 0x7fec2c070710 2024-03-06T07:20:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec34525640 1 -- 172.21.15.98:0/4117808067 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2c1503f0 con 0x7fec2c070710 2024-03-06T07:20:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec22ffd640 1 -- 172.21.15.98:0/4117808067 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fec1c01b070 con 0x7fec2c070710 2024-03-06T07:20:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.363+0000 7fec22ffd640 1 -- 172.21.15.98:0/4117808067 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fec1c0127b0 con 0x7fec2c070710 2024-03-06T07:20:05.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.364+0000 7fec34525640 1 -- 172.21.15.98:0/4117808067 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2c0686e0 con 0x7fec2c070710 2024-03-06T07:20:05.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.366+0000 7fec22ffd640 1 -- 172.21.15.98:0/4117808067 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fec1c022070 con 0x7fec2c070710 2024-03-06T07:20:05.399 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:05.397+0000 7f52f5642040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:20:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.437+0000 7fec34525640 1 -- 172.21.15.98:0/4117808067 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7706d997-9bc6-4ed2-bfa0-3f7aec78afdb", "id": 1} v 0) v1 -- 0x7fec2c0c66f0 con 0x7fec2c070710 2024-03-06T07:20:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.440+0000 7fec22ffd640 1 -- 172.21.15.98:0/4117808067 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7706d997-9bc6-4ed2-bfa0-3f7aec78afdb", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fec1c012af0 con 0x7fec2c070710 2024-03-06T07:20:05.441 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-03-06T07:20:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.440+0000 7fec34525640 1 -- 172.21.15.98:0/4117808067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec2c070710 msgr2=0x7fec2c14faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.440+0000 7fec34525640 1 --2- 172.21.15.98:0/4117808067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec2c070710 0x7fec2c14faa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00c520 tx=0x7fec1c00c9e0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.440+0000 7fec34525640 1 -- 172.21.15.98:0/4117808067 shutdown_connections 2024-03-06T07:20:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.440+0000 7fec34525640 1 --2- 172.21.15.98:0/4117808067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c14d170 0x7fec2c14aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.440+0000 7fec34525640 1 --2- 172.21.15.98:0/4117808067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec2c0c8780 0x7fec2c14c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.440+0000 7fec34525640 1 --2- 172.21.15.98:0/4117808067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec2c070710 0x7fec2c14faa0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.440+0000 7fec34525640 1 -- 172.21.15.98:0/4117808067 >> 172.21.15.98:0/4117808067 conn(0x7fec2c0aa5a0 msgr2=0x7fec2c0c7450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.441+0000 7fec34525640 1 -- 172.21.15.98:0/4117808067 shutdown_connections 2024-03-06T07:20:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.441+0000 7fec34525640 1 -- 172.21.15.98:0/4117808067 wait complete. 2024-03-06T07:20:05.453 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 2fc36263-d860-4ecd-a97d-8bb7810aa070 2 2024-03-06T07:20:05.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.551+0000 7f597c064640 1 Processor -- start 2024-03-06T07:20:05.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.551+0000 7f597c064640 1 -- start start 2024-03-06T07:20:05.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.552+0000 7f597c064640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f59740c6370 0x7f59740c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.552+0000 7f597c064640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f597406f2f0 0x7f5974155800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.552+0000 7f597c064640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59740c3090 con 0x7f59740c2b90 2024-03-06T07:20:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.552+0000 7f597c064640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59740c4520 con 0x7f59740c6370 2024-03-06T07:20:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.552+0000 7f597c064640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59740c4800 con 0x7f597406f2f0 2024-03-06T07:20:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.552+0000 7f5979e00640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f59740c6370 0x7f59740c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.552+0000 7f59795ff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f59740c2b90 0x7f597406ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.98:55024/0 (socket says 172.21.15.98:55024) 2024-03-06T07:20:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.552+0000 7f59795ff640 1 -- 172.21.15.98:0/1294247491 learned_addr learned my addr 172.21.15.98:0/1294247491 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-06T07:20:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.552+0000 7f597a601640 1 --2- 172.21.15.98:0/1294247491 >> v2:172.21.15.98:3300/0 conn(0x7f597406f2f0 0x7f5974155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.553+0000 7f5979e00640 1 -- 172.21.15.98:0/1294247491 >> v2:172.21.15.98:3300/0 conn(0x7f597406f2f0 msgr2=0x7f5974155800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.553+0000 7f5979e00640 1 --2- 172.21.15.98:0/1294247491 >> v2:172.21.15.98:3300/0 conn(0x7f597406f2f0 0x7f5974155800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.553+0000 7f5979e00640 1 -- 172.21.15.98:0/1294247491 >> v1:172.21.15.1:6789/0 conn(0x7f59740c2b90 legacy=0x7f597406ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.553+0000 7f5979e00640 1 -- 172.21.15.98:0/1294247491 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59740c4230 con 0x7f59740c6370 2024-03-06T07:20:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.553+0000 7f5978dfe640 1 -- 172.21.15.98:0/1294247491 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580675659 0 0) 0x7f59740c3090 con 0x7f59740c2b90 2024-03-06T07:20:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.553+0000 7f597a601640 1 --2- 172.21.15.98:0/1294247491 >> v2:172.21.15.98:3300/0 conn(0x7f597406f2f0 0x7f5974155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:05.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.554+0000 7f5979e00640 1 --2- 172.21.15.98:0/1294247491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59740c6370 0x7f59740c58c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f59740c87d0 tx=0x7f596402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e7efa5c8ad3bfe2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.554+0000 7f5978dfe640 1 -- 172.21.15.98:0/1294247491 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59640315a0 con 0x7f59740c6370 2024-03-06T07:20:05.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.554+0000 7f5978dfe640 1 -- 172.21.15.98:0/1294247491 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5964037070 con 0x7f59740c6370 2024-03-06T07:20:05.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.554+0000 7f5978dfe640 1 -- 172.21.15.98:0/1294247491 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5964031960 con 0x7f59740c6370 2024-03-06T07:20:05.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.554+0000 7f597c064640 1 -- 172.21.15.98:0/1294247491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59740c6370 msgr2=0x7f59740c58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.554+0000 7f597c064640 1 --2- 172.21.15.98:0/1294247491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59740c6370 0x7f59740c58c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f59740c87d0 tx=0x7f596402b840 comp rx=0 tx=0).stop 2024-03-06T07:20:05.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.554+0000 7f597c064640 1 -- 172.21.15.98:0/1294247491 shutdown_connections 2024-03-06T07:20:05.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.554+0000 7f597c064640 1 --2- 172.21.15.98:0/1294247491 >> v2:172.21.15.98:3300/0 conn(0x7f597406f2f0 0x7f5974155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.554+0000 7f597c064640 1 --2- 172.21.15.98:0/1294247491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59740c6370 0x7f59740c58c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.554+0000 7f597c064640 1 -- 172.21.15.98:0/1294247491 >> 172.21.15.98:0/1294247491 conn(0x7f597406c730 msgr2=0x7f59740c79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:05.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.555+0000 7f597c064640 1 -- 172.21.15.98:0/1294247491 shutdown_connections 2024-03-06T07:20:05.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.555+0000 7f597c064640 1 -- 172.21.15.98:0/1294247491 wait complete. 2024-03-06T07:20:05.559 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:05.555+0000 7fe965466040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:20:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.556+0000 7f597c064640 1 Processor -- start 2024-03-06T07:20:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.556+0000 7f597c064640 1 -- start start 2024-03-06T07:20:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.556+0000 7f597c064640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f597406f2f0 0x7f59741d6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.556+0000 7f597c064640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740c2b90 0x7f59741ce450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.556+0000 7f597c064640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59740c6370 0x7f59741ce990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.556+0000 7f597c064640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597406f880 con 0x7f59740c6370 2024-03-06T07:20:05.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.556+0000 7f597c064640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597406f700 con 0x7f59740c2b90 2024-03-06T07:20:05.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.556+0000 7f597c064640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59740c6750 con 0x7f597406f2f0 2024-03-06T07:20:05.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.556+0000 7f5979e00640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f597406f2f0 0x7f59741d6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.556+0000 7f59795ff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740c2b90 0x7f59741ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.556+0000 7f5979e00640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f597406f2f0 0x7f59741d6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:47322/0 (socket says 172.21.15.98:47322) 2024-03-06T07:20:05.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.556+0000 7f5979e00640 1 -- 172.21.15.98:0/2417013953 learned_addr learned my addr 172.21.15.98:0/2417013953 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-06T07:20:05.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.557+0000 7f597a601640 1 --2- 172.21.15.98:0/2417013953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59740c6370 0x7f59741ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.557+0000 7f59795ff640 1 -- 172.21.15.98:0/2417013953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f597406f2f0 msgr2=0x7f59741d6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.557+0000 7f59795ff640 1 --2- 172.21.15.98:0/2417013953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f597406f2f0 0x7f59741d6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.557+0000 7f59795ff640 1 -- 172.21.15.98:0/2417013953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59740c6370 msgr2=0x7f59741ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.557+0000 7f59795ff640 1 --2- 172.21.15.98:0/2417013953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59740c6370 0x7f59741ce990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.557+0000 7f59795ff640 1 -- 172.21.15.98:0/2417013953 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59741d4680 con 0x7f59740c2b90 2024-03-06T07:20:05.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.557+0000 7f5979e00640 1 --2- 172.21.15.98:0/2417013953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f597406f2f0 0x7f59741d6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.557+0000 7f59795ff640 1 --2- 172.21.15.98:0/2417013953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740c2b90 0x7f59741ce450 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f597000c450 tx=0x7f597000c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.557+0000 7f596a7fc640 1 -- 172.21.15.98:0/2417013953 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59700074e0 con 0x7f59740c2b90 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.557+0000 7f596a7fc640 1 -- 172.21.15.98:0/2417013953 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59700047e0 con 0x7f59740c2b90 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.557+0000 7f596a7fc640 1 -- 172.21.15.98:0/2417013953 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5970004bd0 con 0x7f59740c2b90 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.557+0000 7f597c064640 1 -- 172.21.15.98:0/2417013953 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59741eb130 con 0x7f59740c2b90 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.557+0000 7f597c064640 1 -- 172.21.15.98:0/2417013953 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59741d31f0 con 0x7f59740c2b90 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.557+0000 7f596a7fc640 1 -- 172.21.15.98:0/2417013953 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5970011070 con 0x7f59740c2b90 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.557+0000 7f596a7fc640 1 -- 172.21.15.98:0/2417013953 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f59700137e0 con 0x7f59740c2b90 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.558+0000 7f597c064640 1 -- 172.21.15.98:0/2417013953 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59740686e0 con 0x7f59740c2b90 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.560+0000 7f596a7fc640 1 -- 172.21.15.98:0/2417013953 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5970004d70 con 0x7f59740c2b90 2024-03-06T07:20:05.613 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:05.611+0000 7fe965466040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:20:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.632+0000 7f597c064640 1 -- 172.21.15.98:0/2417013953 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2fc36263-d860-4ecd-a97d-8bb7810aa070", "id": 2} v 0) v1 -- 0x7f59740c3090 con 0x7f59740c2b90 2024-03-06T07:20:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.635+0000 7f596a7fc640 1 -- 172.21.15.98:0/2417013953 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2fc36263-d860-4ecd-a97d-8bb7810aa070", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f59700133c0 con 0x7f59740c2b90 2024-03-06T07:20:05.637 INFO:teuthology.orchestra.run.smithi098.stdout:2 2024-03-06T07:20:05.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.636+0000 7f597c064640 1 -- 172.21.15.98:0/2417013953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740c2b90 msgr2=0x7f59741ce450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.636+0000 7f597c064640 1 --2- 172.21.15.98:0/2417013953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740c2b90 0x7f59741ce450 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f597000c450 tx=0x7f597000c850 comp rx=0 tx=0).stop 2024-03-06T07:20:05.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.636+0000 7f597c064640 1 -- 172.21.15.98:0/2417013953 shutdown_connections 2024-03-06T07:20:05.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.636+0000 7f597c064640 1 --2- 172.21.15.98:0/2417013953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59740c6370 0x7f59741ce990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.636+0000 7f597c064640 1 --2- 172.21.15.98:0/2417013953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740c2b90 0x7f59741ce450 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.636+0000 7f597c064640 1 --2- 172.21.15.98:0/2417013953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f597406f2f0 0x7f59741d6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.636+0000 7f597c064640 1 -- 172.21.15.98:0/2417013953 >> 172.21.15.98:0/2417013953 conn(0x7f597406c730 msgr2=0x7f597406ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:05.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.636+0000 7f597c064640 1 -- 172.21.15.98:0/2417013953 shutdown_connections 2024-03-06T07:20:05.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.636+0000 7f597c064640 1 -- 172.21.15.98:0/2417013953 wait complete. 2024-03-06T07:20:05.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 68f1b153-7890-4c48-ac45-293337d0e529 3 2024-03-06T07:20:05.661 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:05.659+0000 7fe965466040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:20:05.718 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:05.716+0000 7fe965466040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:20:05.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.748+0000 7f673426f640 1 Processor -- start 2024-03-06T07:20:05.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.748+0000 7f673426f640 1 -- start start 2024-03-06T07:20:05.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.749+0000 7f673426f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f672c0c6370 0x7f672c0c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.749+0000 7f673426f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f672c0c2b90 0x7f672c06ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.749+0000 7f673426f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f672c0c3090 con 0x7f672c06f120 2024-03-06T07:20:05.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.749+0000 7f673426f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f672c0c4520 con 0x7f672c0c2b90 2024-03-06T07:20:05.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.749+0000 7f673426f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672c0c4800 con 0x7f672c0c6370 2024-03-06T07:20:05.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.749+0000 7f673200b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f672c0c6370 0x7f672c0c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.749+0000 7f673200b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f672c0c6370 0x7f672c0c58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:47328/0 (socket says 172.21.15.98:47328) 2024-03-06T07:20:05.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.749+0000 7f673200b640 1 -- 172.21.15.98:0/2988446958 learned_addr learned my addr 172.21.15.98:0/2988446958 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-06T07:20:05.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.749+0000 7f673180a640 1 --2- 172.21.15.98:0/2988446958 >> v2:172.21.15.1:3300/0 conn(0x7f672c0c2b90 0x7f672c06ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.749+0000 7f673200b640 1 -- 172.21.15.98:0/2988446958 >> v2:172.21.15.1:3300/0 conn(0x7f672c0c2b90 msgr2=0x7f672c06ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:05.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.749+0000 7f673200b640 1 --2- 172.21.15.98:0/2988446958 >> v2:172.21.15.1:3300/0 conn(0x7f672c0c2b90 0x7f672c06ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.749+0000 7f673200b640 1 -- 172.21.15.98:0/2988446958 >> v1:172.21.15.1:6789/0 conn(0x7f672c06f120 legacy=0x7f672c072630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.749+0000 7f673200b640 1 -- 172.21.15.98:0/2988446958 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c0c4230 con 0x7f672c0c6370 2024-03-06T07:20:05.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.749+0000 7f673180a640 1 --2- 172.21.15.98:0/2988446958 >> v2:172.21.15.1:3300/0 conn(0x7f672c0c2b90 0x7f672c06ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:05.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.750+0000 7f673200b640 1 --2- 172.21.15.98:0/2988446958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f672c0c6370 0x7f672c0c58c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f671c009870 tx=0x7f671c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12f5b78737f1b8c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.750+0000 7f6731009640 1 -- 172.21.15.98:0/2988446958 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c035020 con 0x7f672c0c6370 2024-03-06T07:20:05.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.750+0000 7f6731009640 1 -- 172.21.15.98:0/2988446958 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671c031640 con 0x7f672c0c6370 2024-03-06T07:20:05.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.750+0000 7f6731009640 1 -- 172.21.15.98:0/2988446958 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c02d040 con 0x7f672c0c6370 2024-03-06T07:20:05.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.751+0000 7f673426f640 1 -- 172.21.15.98:0/2988446958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f672c0c6370 msgr2=0x7f672c0c58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.751+0000 7f673426f640 1 --2- 172.21.15.98:0/2988446958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f672c0c6370 0x7f672c0c58c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f671c009870 tx=0x7f671c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.751+0000 7f673426f640 1 -- 172.21.15.98:0/2988446958 shutdown_connections 2024-03-06T07:20:05.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.751+0000 7f673426f640 1 --2- 172.21.15.98:0/2988446958 >> v2:172.21.15.1:3300/0 conn(0x7f672c0c2b90 0x7f672c06ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.751+0000 7f673426f640 1 --2- 172.21.15.98:0/2988446958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f672c0c6370 0x7f672c0c58c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.751+0000 7f673426f640 1 -- 172.21.15.98:0/2988446958 >> 172.21.15.98:0/2988446958 conn(0x7f672c06c730 msgr2=0x7f672c0c79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:05.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.751+0000 7f673426f640 1 -- 172.21.15.98:0/2988446958 shutdown_connections 2024-03-06T07:20:05.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.751+0000 7f673426f640 1 -- 172.21.15.98:0/2988446958 wait complete. 2024-03-06T07:20:05.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.752+0000 7f673426f640 1 Processor -- start 2024-03-06T07:20:05.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.752+0000 7f673426f640 1 -- start start 2024-03-06T07:20:05.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673426f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f672c06f120 0x7f672c1d7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673426f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f672c0c2b90 0x7f672c1def40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673426f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f672c0c6370 0x7f672c1cd800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673426f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c0c68d0 con 0x7f672c0c6370 2024-03-06T07:20:05.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673426f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c0c6750 con 0x7f672c06f120 2024-03-06T07:20:05.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673426f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672c06f590 con 0x7f672c0c2b90 2024-03-06T07:20:05.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673180a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f672c0c2b90 0x7f672c1def40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673180a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f672c0c2b90 0x7f672c1def40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:47336/0 (socket says 172.21.15.98:47336) 2024-03-06T07:20:05.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673180a640 1 -- 172.21.15.98:0/2161020761 learned_addr learned my addr 172.21.15.98:0/2161020761 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-06T07:20:05.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673200b640 1 --2- 172.21.15.98:0/2161020761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f672c06f120 0x7f672c1d7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673280c640 1 --2- 172.21.15.98:0/2161020761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f672c0c6370 0x7f672c1cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673180a640 1 -- 172.21.15.98:0/2161020761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f672c06f120 msgr2=0x7f672c1d7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673180a640 1 --2- 172.21.15.98:0/2161020761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f672c06f120 0x7f672c1d7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673180a640 1 -- 172.21.15.98:0/2161020761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f672c0c6370 msgr2=0x7f672c1cd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673180a640 1 --2- 172.21.15.98:0/2161020761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f672c0c6370 0x7f672c1cd800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673180a640 1 -- 172.21.15.98:0/2161020761 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c1eb890 con 0x7f672c0c2b90 2024-03-06T07:20:05.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673180a640 1 --2- 172.21.15.98:0/2161020761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f672c0c2b90 0x7f672c1def40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f672800e7e0 tx=0x7f672800eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f6722ffd640 1 -- 172.21.15.98:0/2161020761 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672800f040 con 0x7f672c0c2b90 2024-03-06T07:20:05.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f6722ffd640 1 -- 172.21.15.98:0/2161020761 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6728004030 con 0x7f672c0c2b90 2024-03-06T07:20:05.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f6722ffd640 1 -- 172.21.15.98:0/2161020761 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728004400 con 0x7f672c0c2b90 2024-03-06T07:20:05.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673426f640 1 -- 172.21.15.98:0/2161020761 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f672c1ee260 con 0x7f672c0c2b90 2024-03-06T07:20:05.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.753+0000 7f673426f640 1 -- 172.21.15.98:0/2161020761 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f672c1d7e80 con 0x7f672c0c2b90 2024-03-06T07:20:05.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.754+0000 7f6722ffd640 1 -- 172.21.15.98:0/2161020761 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f672800ca60 con 0x7f672c0c2b90 2024-03-06T07:20:05.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.754+0000 7f6722ffd640 1 -- 172.21.15.98:0/2161020761 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f67280139e0 con 0x7f672c0c2b90 2024-03-06T07:20:05.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.754+0000 7f673426f640 1 -- 172.21.15.98:0/2161020761 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f672c0686e0 con 0x7f672c0c2b90 2024-03-06T07:20:05.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.757+0000 7f6722ffd640 1 -- 172.21.15.98:0/2161020761 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6728017020 con 0x7f672c0c2b90 2024-03-06T07:20:05.763 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:05.761+0000 7fe965466040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:20:05.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.827+0000 7f673426f640 1 -- 172.21.15.98:0/2161020761 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd new", "uuid": "68f1b153-7890-4c48-ac45-293337d0e529", "id": 3} v 0) v1 -- 0x7f672c0acf50 con 0x7f672c0c2b90 2024-03-06T07:20:05.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.830+0000 7f6722ffd640 1 -- 172.21.15.98:0/2161020761 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "68f1b153-7890-4c48-ac45-293337d0e529", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6728013cb0 con 0x7f672c0c2b90 2024-03-06T07:20:05.832 INFO:teuthology.orchestra.run.smithi098.stdout:3 2024-03-06T07:20:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.831+0000 7f673426f640 1 -- 172.21.15.98:0/2161020761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f672c0c2b90 msgr2=0x7f672c1def40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.831+0000 7f673426f640 1 --2- 172.21.15.98:0/2161020761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f672c0c2b90 0x7f672c1def40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f672800e7e0 tx=0x7f672800eca0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.831+0000 7f673426f640 1 -- 172.21.15.98:0/2161020761 shutdown_connections 2024-03-06T07:20:05.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.831+0000 7f673426f640 1 --2- 172.21.15.98:0/2161020761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f672c0c6370 0x7f672c1cd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.831+0000 7f673426f640 1 --2- 172.21.15.98:0/2161020761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f672c0c2b90 0x7f672c1def40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.831+0000 7f673426f640 1 --2- 172.21.15.98:0/2161020761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f672c06f120 0x7f672c1d7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.831+0000 7f673426f640 1 -- 172.21.15.98:0/2161020761 >> 172.21.15.98:0/2161020761 conn(0x7f672c06c730 msgr2=0x7f672c0aafc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:05.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.831+0000 7f673426f640 1 -- 172.21.15.98:0/2161020761 shutdown_connections 2024-03-06T07:20:05.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.831+0000 7f673426f640 1 -- 172.21.15.98:0/2161020761 wait complete. 2024-03-06T07:20:05.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 62134c92-7dbe-4266-915d-c6333d17aeb9 4 2024-03-06T07:20:05.882 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:05.880+0000 7fe965466040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:20:05.954 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:05.952+0000 7f52f5642040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:20:05.955 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:05.954+0000 7fe965466040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:20:05.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.980+0000 7f0dc73bc640 1 Processor -- start 2024-03-06T07:20:05.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.980+0000 7f0dc73bc640 1 -- start start 2024-03-06T07:20:05.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.981+0000 7f0dc73bc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0dc006ccf0 0x7f0dc0072510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.981+0000 7f0dc73bc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc00c25e0 con 0x7f0dc0072a50 2024-03-06T07:20:05.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.981+0000 7f0dc73bc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc00c2d40 con 0x7f0dc00a6fc0 2024-03-06T07:20:05.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.981+0000 7f0dc73bc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc00c73f0 con 0x7f0dc006ccf0 2024-03-06T07:20:05.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.981+0000 7f0dc5158640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0dc006ccf0 0x7f0dc0072510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.981+0000 7f0dc5959640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0dc00a6fc0 0x7f0dc015fb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47978/0 (socket says 172.21.15.98:47978) 2024-03-06T07:20:05.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.981+0000 7f0dc5959640 1 -- 172.21.15.98:0/1155707815 learned_addr learned my addr 172.21.15.98:0/1155707815 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-06T07:20:05.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.981+0000 7f0dc5158640 1 -- 172.21.15.98:0/1155707815 >> v1:172.21.15.98:6789/0 conn(0x7f0dc00a6fc0 legacy=0x7f0dc015fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.981+0000 7f0dc5158640 1 -- 172.21.15.98:0/1155707815 >> v1:172.21.15.1:6789/0 conn(0x7f0dc0072a50 legacy=0x7f0dc0155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.981+0000 7f0dc5158640 1 -- 172.21.15.98:0/1155707815 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc00c2fe0 con 0x7f0dc006ccf0 2024-03-06T07:20:05.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.982+0000 7f0dc5158640 1 --2- 172.21.15.98:0/1155707815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dc006ccf0 0x7f0dc0072510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0db00097a0 tx=0x7f0db002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42003e0233ce3ce0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.982+0000 7f0db7fff640 1 -- 172.21.15.98:0/1155707815 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db0035020 con 0x7f0dc006ccf0 2024-03-06T07:20:05.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.982+0000 7f0db7fff640 1 -- 172.21.15.98:0/1155707815 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db0031830 con 0x7f0dc006ccf0 2024-03-06T07:20:05.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.982+0000 7f0db7fff640 1 -- 172.21.15.98:0/1155707815 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db0031c00 con 0x7f0dc006ccf0 2024-03-06T07:20:05.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.983+0000 7f0dc73bc640 1 -- 172.21.15.98:0/1155707815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dc006ccf0 msgr2=0x7f0dc0072510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.983+0000 7f0dc73bc640 1 --2- 172.21.15.98:0/1155707815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dc006ccf0 0x7f0dc0072510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0db00097a0 tx=0x7f0db002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.983+0000 7f0dc73bc640 1 -- 172.21.15.98:0/1155707815 shutdown_connections 2024-03-06T07:20:05.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.983+0000 7f0dc73bc640 1 --2- 172.21.15.98:0/1155707815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dc006ccf0 0x7f0dc0072510 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.983+0000 7f0dc73bc640 1 -- 172.21.15.98:0/1155707815 >> 172.21.15.98:0/1155707815 conn(0x7f0dc00a9700 msgr2=0x7f0dc00a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:05.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.983+0000 7f0dc73bc640 1 -- 172.21.15.98:0/1155707815 shutdown_connections 2024-03-06T07:20:05.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.983+0000 7f0dc73bc640 1 -- 172.21.15.98:0/1155707815 wait complete. 2024-03-06T07:20:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.984+0000 7f0dc73bc640 1 Processor -- start 2024-03-06T07:20:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.984+0000 7f0dc73bc640 1 -- start start 2024-03-06T07:20:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0dc73bc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dc006ccf0 0x7f0dc01dfe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0dc73bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dc0072a50 0x7f0dc0157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0dc73bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dc00a6fc0 0x7f0dc01eb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0dc73bc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc00c6ea0 con 0x7f0dc006ccf0 2024-03-06T07:20:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0dc73bc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc00c76d0 con 0x7f0dc00a6fc0 2024-03-06T07:20:05.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0dc73bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc0154210 con 0x7f0dc0072a50 2024-03-06T07:20:05.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0dc5959640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dc00a6fc0 0x7f0dc01eb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0dc4957640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dc0072a50 0x7f0dc0157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0dc5959640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dc00a6fc0 0x7f0dc01eb5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44038/0 (socket says 172.21.15.98:44038) 2024-03-06T07:20:05.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0dc5959640 1 -- 172.21.15.98:0/870122173 learned_addr learned my addr 172.21.15.98:0/870122173 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-06T07:20:05.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0dc5158640 1 --2- 172.21.15.98:0/870122173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dc006ccf0 0x7f0dc01dfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0dc4957640 1 -- 172.21.15.98:0/870122173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dc00a6fc0 msgr2=0x7f0dc01eb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0dc4957640 1 --2- 172.21.15.98:0/870122173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dc00a6fc0 0x7f0dc01eb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0dc4957640 1 -- 172.21.15.98:0/870122173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dc006ccf0 msgr2=0x7f0dc01dfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0dc4957640 1 --2- 172.21.15.98:0/870122173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dc006ccf0 0x7f0dc01dfe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0dc4957640 1 -- 172.21.15.98:0/870122173 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc0154f60 con 0x7f0dc0072a50 2024-03-06T07:20:05.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0dc4957640 1 --2- 172.21.15.98:0/870122173 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dc0072a50 0x7f0dc0157740 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc00c340 tx=0x7f0dbc00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0db5ffb640 1 -- 172.21.15.98:0/870122173 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc0052d0 con 0x7f0dc0072a50 2024-03-06T07:20:05.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.985+0000 7f0db5ffb640 1 -- 172.21.15.98:0/870122173 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dbc005490 con 0x7f0dc0072a50 2024-03-06T07:20:05.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.986+0000 7f0dc73bc640 1 -- 172.21.15.98:0/870122173 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc0155240 con 0x7f0dc0072a50 2024-03-06T07:20:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.986+0000 7f0db5ffb640 1 -- 172.21.15.98:0/870122173 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc007260 con 0x7f0dc0072a50 2024-03-06T07:20:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.986+0000 7f0db5ffb640 1 -- 172.21.15.98:0/870122173 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0dbc007400 con 0x7f0dc0072a50 2024-03-06T07:20:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.986+0000 7f0dc73bc640 1 -- 172.21.15.98:0/870122173 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc01eb090 con 0x7f0dc0072a50 2024-03-06T07:20:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.986+0000 7f0db5ffb640 1 -- 172.21.15.98:0/870122173 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f0dbc003650 con 0x7f0dc0072a50 2024-03-06T07:20:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.987+0000 7f0dc73bc640 1 -- 172.21.15.98:0/870122173 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc00c27c0 con 0x7f0dc0072a50 2024-03-06T07:20:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:05.989+0000 7f0db5ffb640 1 -- 172.21.15.98:0/870122173 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0dc00c27c0 con 0x7f0dc0072a50 2024-03-06T07:20:06.005 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:06.003+0000 7fe965466040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:20:06.010 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:06.008+0000 7f52f5642040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:20:06.056 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:06.054+0000 7f52f5642040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:20:06.059 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:06.057+0000 7fe965466040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:20:06.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.061+0000 7f0dc73bc640 1 -- 172.21.15.98:0/870122173 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd new", "uuid": "62134c92-7dbe-4266-915d-c6333d17aeb9", "id": 4} v 0) v1 -- 0x7f0dc00686e0 con 0x7f0dc0072a50 2024-03-06T07:20:06.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.064+0000 7f0db5ffb640 1 -- 172.21.15.98:0/870122173 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "62134c92-7dbe-4266-915d-c6333d17aeb9", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0dc00c25e0 con 0x7f0dc0072a50 2024-03-06T07:20:06.066 INFO:teuthology.orchestra.run.smithi098.stdout:4 2024-03-06T07:20:06.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.065+0000 7f0dc73bc640 1 -- 172.21.15.98:0/870122173 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dc0072a50 msgr2=0x7f0dc0157740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.065+0000 7f0dc73bc640 1 --2- 172.21.15.98:0/870122173 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dc0072a50 0x7f0dc0157740 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc00c340 tx=0x7f0dbc00c800 comp rx=0 tx=0).stop 2024-03-06T07:20:06.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.065+0000 7f0dc73bc640 1 -- 172.21.15.98:0/870122173 shutdown_connections 2024-03-06T07:20:06.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.065+0000 7f0dc73bc640 1 --2- 172.21.15.98:0/870122173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dc00a6fc0 0x7f0dc01eb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.065+0000 7f0dc73bc640 1 --2- 172.21.15.98:0/870122173 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dc0072a50 0x7f0dc0157740 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.065+0000 7f0dc73bc640 1 --2- 172.21.15.98:0/870122173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dc006ccf0 0x7f0dc01dfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.065+0000 7f0dc73bc640 1 -- 172.21.15.98:0/870122173 >> 172.21.15.98:0/870122173 conn(0x7f0dc00a9700 msgr2=0x7f0dc00a7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:06.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.065+0000 7f0dc73bc640 1 -- 172.21.15.98:0/870122173 shutdown_connections 2024-03-06T07:20:06.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.065+0000 7f0dc73bc640 1 -- 172.21.15.98:0/870122173 wait complete. 2024-03-06T07:20:06.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 46eacdc5-0b2a-4977-995e-47240d980c20 5 2024-03-06T07:20:06.109 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:06.108+0000 7fe965466040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:20:06.112 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:06.110+0000 7f52f5642040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:20:06.156 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:06.154+0000 7f52f5642040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:20:06.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.176+0000 7f0271a1c640 1 Processor -- start 2024-03-06T07:20:06.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.176+0000 7f0271a1c640 1 -- start start 2024-03-06T07:20:06.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.176+0000 7f0271a1c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f026c0c6ce0 0x7f026c13e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.176+0000 7f0271a1c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f026c13e6b0 0x7f026c0c1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.176+0000 7f0271a1c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f026c0c18a0 0x7f026c158980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.176+0000 7f0271a1c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f026c1551b0 con 0x7f026c0c6ce0 2024-03-06T07:20:06.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.176+0000 7f0271a1c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f026c155470 con 0x7f026c13e6b0 2024-03-06T07:20:06.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.176+0000 7f0271a1c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026c155750 con 0x7f026c0c18a0 2024-03-06T07:20:06.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.176+0000 7f026b7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f026c0c18a0 0x7f026c158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.176+0000 7f026a7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f026c13e6b0 0x7f026c0c1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.176+0000 7f026b7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f026c0c18a0 0x7f026c158980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:47354/0 (socket says 172.21.15.98:47354) 2024-03-06T07:20:06.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.176+0000 7f026b7fe640 1 -- 172.21.15.98:0/3739271974 learned_addr learned my addr 172.21.15.98:0/3739271974 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-06T07:20:06.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.177+0000 7f026affd640 1 --2- 172.21.15.98:0/3739271974 >> v2:172.21.15.1:3300/0 conn(0x7f026c0c6ce0 0x7f026c13e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.177+0000 7f026b7fe640 1 -- 172.21.15.98:0/3739271974 >> v2:172.21.15.98:3300/0 conn(0x7f026c13e6b0 msgr2=0x7f026c0c1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:06.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.177+0000 7f026b7fe640 1 --2- 172.21.15.98:0/3739271974 >> v2:172.21.15.98:3300/0 conn(0x7f026c13e6b0 0x7f026c0c1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.177+0000 7f026b7fe640 1 -- 172.21.15.98:0/3739271974 >> v2:172.21.15.1:3300/0 conn(0x7f026c0c6ce0 msgr2=0x7f026c13e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:06.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.177+0000 7f026b7fe640 1 --2- 172.21.15.98:0/3739271974 >> v2:172.21.15.1:3300/0 conn(0x7f026c0c6ce0 0x7f026c13e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.177+0000 7f026b7fe640 1 -- 172.21.15.98:0/3739271974 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c155a30 con 0x7f026c0c18a0 2024-03-06T07:20:06.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.177+0000 7f026a7fc640 1 --2- 172.21.15.98:0/3739271974 >> v2:172.21.15.98:3300/0 conn(0x7f026c13e6b0 0x7f026c0c1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:06.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.177+0000 7f026b7fe640 1 --2- 172.21.15.98:0/3739271974 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f026c0c18a0 0x7f026c158980 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f026c155680 tx=0x7f026402be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b73571c34eacb621 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:06.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.178+0000 7f0269ffb640 1 -- 172.21.15.98:0/3739271974 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026402c9c0 con 0x7f026c0c18a0 2024-03-06T07:20:06.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.178+0000 7f0269ffb640 1 -- 172.21.15.98:0/3739271974 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026402cb60 con 0x7f026c0c18a0 2024-03-06T07:20:06.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.178+0000 7f0269ffb640 1 -- 172.21.15.98:0/3739271974 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02640343f0 con 0x7f026c0c18a0 2024-03-06T07:20:06.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.178+0000 7f0271a1c640 1 -- 172.21.15.98:0/3739271974 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f026c0c18a0 msgr2=0x7f026c158980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.178+0000 7f0271a1c640 1 --2- 172.21.15.98:0/3739271974 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f026c0c18a0 0x7f026c158980 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f026c155680 tx=0x7f026402be20 comp rx=0 tx=0).stop 2024-03-06T07:20:06.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.178+0000 7f0271a1c640 1 -- 172.21.15.98:0/3739271974 shutdown_connections 2024-03-06T07:20:06.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.178+0000 7f0271a1c640 1 --2- 172.21.15.98:0/3739271974 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f026c0c18a0 0x7f026c158980 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.178+0000 7f0271a1c640 1 --2- 172.21.15.98:0/3739271974 >> v2:172.21.15.98:3300/0 conn(0x7f026c13e6b0 0x7f026c0c1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.178+0000 7f0271a1c640 1 --2- 172.21.15.98:0/3739271974 >> v2:172.21.15.1:3300/0 conn(0x7f026c0c6ce0 0x7f026c13e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.178+0000 7f0271a1c640 1 -- 172.21.15.98:0/3739271974 >> 172.21.15.98:0/3739271974 conn(0x7f026c06da30 msgr2=0x7f026c066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:06.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.178+0000 7f0271a1c640 1 -- 172.21.15.98:0/3739271974 shutdown_connections 2024-03-06T07:20:06.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.178+0000 7f0271a1c640 1 -- 172.21.15.98:0/3739271974 wait complete. 2024-03-06T07:20:06.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.179+0000 7f0271a1c640 1 Processor -- start 2024-03-06T07:20:06.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.180+0000 7f0271a1c640 1 -- start start 2024-03-06T07:20:06.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.180+0000 7f0271a1c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f026c0c18a0 0x7f026c0ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.180+0000 7f0271a1c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f026c0c6ce0 0x7f026c1e1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.180+0000 7f0271a1c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f026c13e6b0 0x7f026c139d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.180+0000 7f0271a1c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c0c1d30 con 0x7f026c0c18a0 2024-03-06T07:20:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.180+0000 7f0271a1c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c155ff0 con 0x7f026c0c6ce0 2024-03-06T07:20:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.180+0000 7f0271a1c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026c0c70c0 con 0x7f026c13e6b0 2024-03-06T07:20:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.180+0000 7f026b7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f026c13e6b0 0x7f026c139d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.180+0000 7f026b7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f026c13e6b0 0x7f026c139d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:47364/0 (socket says 172.21.15.98:47364) 2024-03-06T07:20:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.180+0000 7f026b7fe640 1 -- 172.21.15.98:0/1383592215 learned_addr learned my addr 172.21.15.98:0/1383592215 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-06T07:20:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.180+0000 7f026a7fc640 1 --2- 172.21.15.98:0/1383592215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f026c0c6ce0 0x7f026c1e1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.180+0000 7f026affd640 1 --2- 172.21.15.98:0/1383592215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f026c0c18a0 0x7f026c0ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.181+0000 7f026b7fe640 1 -- 172.21.15.98:0/1383592215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f026c0c6ce0 msgr2=0x7f026c1e1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.181+0000 7f026b7fe640 1 --2- 172.21.15.98:0/1383592215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f026c0c6ce0 0x7f026c1e1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.181+0000 7f026b7fe640 1 -- 172.21.15.98:0/1383592215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f026c0c18a0 msgr2=0x7f026c0ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.181+0000 7f026b7fe640 1 --2- 172.21.15.98:0/1383592215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f026c0c18a0 0x7f026c0ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.181+0000 7f026b7fe640 1 -- 172.21.15.98:0/1383592215 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c0c96f0 con 0x7f026c13e6b0 2024-03-06T07:20:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.181+0000 7f026a7fc640 1 --2- 172.21.15.98:0/1383592215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f026c0c6ce0 0x7f026c1e1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:06.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.181+0000 7f026b7fe640 1 --2- 172.21.15.98:0/1383592215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f026c13e6b0 0x7f026c139d10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0264009f40 tx=0x7f0264004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:06.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.181+0000 7f0253fff640 1 -- 172.21.15.98:0/1383592215 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264043020 con 0x7f026c13e6b0 2024-03-06T07:20:06.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.181+0000 7f0253fff640 1 -- 172.21.15.98:0/1383592215 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026403e450 con 0x7f026c13e6b0 2024-03-06T07:20:06.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.181+0000 7f0271a1c640 1 -- 172.21.15.98:0/1383592215 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f026c0c89e0 con 0x7f026c13e6b0 2024-03-06T07:20:06.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.181+0000 7f0253fff640 1 -- 172.21.15.98:0/1383592215 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264009510 con 0x7f026c13e6b0 2024-03-06T07:20:06.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.181+0000 7f0271a1c640 1 -- 172.21.15.98:0/1383592215 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f026c0cac90 con 0x7f026c13e6b0 2024-03-06T07:20:06.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.181+0000 7f0253fff640 1 -- 172.21.15.98:0/1383592215 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0264046020 con 0x7f026c13e6b0 2024-03-06T07:20:06.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.181+0000 7f0253fff640 1 -- 172.21.15.98:0/1383592215 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f026402c9d0 con 0x7f026c13e6b0 2024-03-06T07:20:06.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.182+0000 7f0271a1c640 1 -- 172.21.15.98:0/1383592215 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f026c067fd0 con 0x7f026c13e6b0 2024-03-06T07:20:06.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.184+0000 7f0253fff640 1 -- 172.21.15.98:0/1383592215 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f026404d020 con 0x7f026c13e6b0 2024-03-06T07:20:06.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.256+0000 7f0271a1c640 1 -- 172.21.15.98:0/1383592215 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd new", "uuid": "46eacdc5-0b2a-4977-995e-47240d980c20", "id": 5} v 0) v1 -- 0x7f026c0c6850 con 0x7f026c13e6b0 2024-03-06T07:20:06.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.259+0000 7f0253fff640 1 -- 172.21.15.98:0/1383592215 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "46eacdc5-0b2a-4977-995e-47240d980c20", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f026403e840 con 0x7f026c13e6b0 2024-03-06T07:20:06.261 INFO:teuthology.orchestra.run.smithi098.stdout:5 2024-03-06T07:20:06.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.260+0000 7f0271a1c640 1 -- 172.21.15.98:0/1383592215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f026c13e6b0 msgr2=0x7f026c139d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.260+0000 7f0271a1c640 1 --2- 172.21.15.98:0/1383592215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f026c13e6b0 0x7f026c139d10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0264009f40 tx=0x7f0264004a40 comp rx=0 tx=0).stop 2024-03-06T07:20:06.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.260+0000 7f0271a1c640 1 -- 172.21.15.98:0/1383592215 shutdown_connections 2024-03-06T07:20:06.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.260+0000 7f0271a1c640 1 --2- 172.21.15.98:0/1383592215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f026c13e6b0 0x7f026c139d10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.260+0000 7f0271a1c640 1 --2- 172.21.15.98:0/1383592215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f026c0c6ce0 0x7f026c1e1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.260+0000 7f0271a1c640 1 --2- 172.21.15.98:0/1383592215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f026c0c18a0 0x7f026c0ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.260+0000 7f0271a1c640 1 -- 172.21.15.98:0/1383592215 >> 172.21.15.98:0/1383592215 conn(0x7f026c06da30 msgr2=0x7f026c0c52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:06.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.260+0000 7f0271a1c640 1 -- 172.21.15.98:0/1383592215 shutdown_connections 2024-03-06T07:20:06.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.260+0000 7f0271a1c640 1 -- 172.21.15.98:0/1383592215 wait complete. 2024-03-06T07:20:06.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 296b86b0-3cfb-4b09-97ca-4c1c62d547f0 6 2024-03-06T07:20:06.276 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:06.274+0000 7f52f5642040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:20:06.348 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:06.346+0000 7f52f5642040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:20:06.396 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:06.394+0000 7f52f5642040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:20:06.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.413+0000 7efccda55640 1 Processor -- start 2024-03-06T07:20:06.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.413+0000 7efccda55640 1 -- start start 2024-03-06T07:20:06.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.414+0000 7efccda55640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcc80acc50 0x7efcc80c3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.414+0000 7efccda55640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efcc80c11a0 0x7efcc80c3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.414+0000 7efccda55640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc81551b0 con 0x7efcc80c18d0 2024-03-06T07:20:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.414+0000 7efccda55640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc8155dc0 con 0x7efcc80c11a0 2024-03-06T07:20:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.414+0000 7efccda55640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc8155490 con 0x7efcc80acc50 2024-03-06T07:20:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.414+0000 7efcc77fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efcc80c11a0 0x7efcc80c3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.414+0000 7efcc6ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcc80acc50 0x7efcc80c3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.414+0000 7efcc77fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efcc80c11a0 0x7efcc80c3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44064/0 (socket says 172.21.15.98:44064) 2024-03-06T07:20:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.414+0000 7efcc77fe640 1 -- 172.21.15.98:0/1450993372 learned_addr learned my addr 172.21.15.98:0/1450993372 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-06T07:20:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.414+0000 7efcc67fc640 1 --1- 172.21.15.98:0/1450993372 >> v1:172.21.15.98:6789/0 conn(0x7efcc80c18d0 0x7efcc8158790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47984/0 (socket says 172.21.15.98:47984) 2024-03-06T07:20:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.414+0000 7efcc77fe640 1 -- 172.21.15.98:0/1450993372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcc80acc50 msgr2=0x7efcc80c3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.414+0000 7efcc77fe640 1 --2- 172.21.15.98:0/1450993372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcc80acc50 0x7efcc80c3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.414+0000 7efcc77fe640 1 -- 172.21.15.98:0/1450993372 >> v1:172.21.15.98:6789/0 conn(0x7efcc80c18d0 legacy=0x7efcc8158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.414+0000 7efcc5ffb640 1 -- 172.21.15.98:0/1450993372 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275863628 0 0) 0x7efcc81551b0 con 0x7efcc80c18d0 2024-03-06T07:20:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.415+0000 7efcc77fe640 1 -- 172.21.15.98:0/1450993372 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc8155770 con 0x7efcc80c11a0 2024-03-06T07:20:06.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.415+0000 7efcc6ffd640 1 --2- 172.21.15.98:0/1450993372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcc80acc50 0x7efcc80c3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.415+0000 7efcc77fe640 1 --2- 172.21.15.98:0/1450993372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcc80c11a0 0x7efcc80c3a40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efcc0009e50 tx=0x7efcc002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e656d59b0491809 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.415+0000 7efcc5ffb640 1 -- 172.21.15.98:0/1450993372 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc002cb00 con 0x7efcc80c11a0 2024-03-06T07:20:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.415+0000 7efcc5ffb640 1 -- 172.21.15.98:0/1450993372 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc002cca0 con 0x7efcc80c11a0 2024-03-06T07:20:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.415+0000 7efcc5ffb640 1 -- 172.21.15.98:0/1450993372 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc0034610 con 0x7efcc80c11a0 2024-03-06T07:20:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.416+0000 7efccda55640 1 -- 172.21.15.98:0/1450993372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcc80c11a0 msgr2=0x7efcc80c3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.416+0000 7efccda55640 1 --2- 172.21.15.98:0/1450993372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcc80c11a0 0x7efcc80c3a40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efcc0009e50 tx=0x7efcc002bf60 comp rx=0 tx=0).stop 2024-03-06T07:20:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.416+0000 7efccda55640 1 -- 172.21.15.98:0/1450993372 shutdown_connections 2024-03-06T07:20:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.416+0000 7efccda55640 1 --2- 172.21.15.98:0/1450993372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcc80c11a0 0x7efcc80c3a40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.416+0000 7efccda55640 1 --2- 172.21.15.98:0/1450993372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcc80acc50 0x7efcc80c3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.416+0000 7efccda55640 1 -- 172.21.15.98:0/1450993372 >> 172.21.15.98:0/1450993372 conn(0x7efcc806aff0 msgr2=0x7efcc806b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:06.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.416+0000 7efccda55640 1 -- 172.21.15.98:0/1450993372 shutdown_connections 2024-03-06T07:20:06.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.416+0000 7efccda55640 1 -- 172.21.15.98:0/1450993372 wait complete. 2024-03-06T07:20:06.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.417+0000 7efccda55640 1 Processor -- start 2024-03-06T07:20:06.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.417+0000 7efccda55640 1 -- start start 2024-03-06T07:20:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.418+0000 7efccda55640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcc80acc50 0x7efcc80c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.418+0000 7efccda55640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcc80c11a0 0x7efcc8136930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.418+0000 7efccda55640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcc80c18d0 0x7efcc8136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.418+0000 7efccda55640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80c41e0 con 0x7efcc80c18d0 2024-03-06T07:20:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.418+0000 7efccda55640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80c4060 con 0x7efcc80acc50 2024-03-06T07:20:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.418+0000 7efccda55640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc8158ea0 con 0x7efcc80c11a0 2024-03-06T07:20:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.418+0000 7efcc6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcc80acc50 0x7efcc80c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.418+0000 7efcc67fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcc80c11a0 0x7efcc8136930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.418+0000 7efcc67fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcc80c11a0 0x7efcc8136930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:47372/0 (socket says 172.21.15.98:47372) 2024-03-06T07:20:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.418+0000 7efcc67fc640 1 -- 172.21.15.98:0/1663104107 learned_addr learned my addr 172.21.15.98:0/1663104107 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-06T07:20:06.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.418+0000 7efcc77fe640 1 --2- 172.21.15.98:0/1663104107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcc80c18d0 0x7efcc8136e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.418+0000 7efcc6ffd640 1 -- 172.21.15.98:0/1663104107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcc80c11a0 msgr2=0x7efcc8136930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.418+0000 7efcc6ffd640 1 --2- 172.21.15.98:0/1663104107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcc80c11a0 0x7efcc8136930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.418+0000 7efcc6ffd640 1 -- 172.21.15.98:0/1663104107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcc80c18d0 msgr2=0x7efcc8136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.418+0000 7efcc6ffd640 1 --2- 172.21.15.98:0/1663104107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcc80c18d0 0x7efcc8136e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.418+0000 7efcc6ffd640 1 -- 172.21.15.98:0/1663104107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc8158800 con 0x7efcc80acc50 2024-03-06T07:20:06.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.418+0000 7efcc6ffd640 1 --2- 172.21.15.98:0/1663104107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcc80acc50 0x7efcc80c54b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efcbc00e930 tx=0x7efcbc00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:06.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.419+0000 7efcaffff640 1 -- 172.21.15.98:0/1663104107 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc017070 con 0x7efcc80acc50 2024-03-06T07:20:06.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.419+0000 7efcaffff640 1 -- 172.21.15.98:0/1663104107 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcbc00ca00 con 0x7efcc80acc50 2024-03-06T07:20:06.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.419+0000 7efccda55640 1 -- 172.21.15.98:0/1663104107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc81ee720 con 0x7efcc80acc50 2024-03-06T07:20:06.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.419+0000 7efcaffff640 1 -- 172.21.15.98:0/1663104107 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc00cdd0 con 0x7efcc80acc50 2024-03-06T07:20:06.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.419+0000 7efccda55640 1 -- 172.21.15.98:0/1663104107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc81ee440 con 0x7efcc80acc50 2024-03-06T07:20:06.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.419+0000 7efcaffff640 1 -- 172.21.15.98:0/1663104107 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7efcbc01a070 con 0x7efcc80acc50 2024-03-06T07:20:06.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.419+0000 7efcaffff640 1 -- 172.21.15.98:0/1663104107 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7efcbc01ca40 con 0x7efcc80acc50 2024-03-06T07:20:06.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.419+0000 7efccda55640 1 -- 172.21.15.98:0/1663104107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc80686e0 con 0x7efcc80acc50 2024-03-06T07:20:06.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.421+0000 7efcaffff640 1 -- 172.21.15.98:0/1663104107 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7efcbc01c450 con 0x7efcc80acc50 2024-03-06T07:20:06.450 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:06.448+0000 7f52f5642040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:20:06.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.496+0000 7efccda55640 1 -- 172.21.15.98:0/1663104107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd new", "uuid": "296b86b0-3cfb-4b09-97ca-4c1c62d547f0", "id": 6} v 0) v1 -- 0x7efcc80c0580 con 0x7efcc80acc50 2024-03-06T07:20:06.499 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:06.497+0000 7f52f5642040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:20:06.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.498+0000 7efcaffff640 1 -- 172.21.15.98:0/1663104107 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "296b86b0-3cfb-4b09-97ca-4c1c62d547f0", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7efcbc01c630 con 0x7efcc80acc50 2024-03-06T07:20:06.500 INFO:teuthology.orchestra.run.smithi098.stdout:6 2024-03-06T07:20:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.499+0000 7efccda55640 1 -- 172.21.15.98:0/1663104107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcc80acc50 msgr2=0x7efcc80c54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.499+0000 7efccda55640 1 --2- 172.21.15.98:0/1663104107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcc80acc50 0x7efcc80c54b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efcbc00e930 tx=0x7efcbc00edf0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.499+0000 7efccda55640 1 -- 172.21.15.98:0/1663104107 shutdown_connections 2024-03-06T07:20:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.499+0000 7efccda55640 1 --2- 172.21.15.98:0/1663104107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcc80c18d0 0x7efcc8136e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.499+0000 7efccda55640 1 --2- 172.21.15.98:0/1663104107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcc80c11a0 0x7efcc8136930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.499+0000 7efccda55640 1 --2- 172.21.15.98:0/1663104107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcc80acc50 0x7efcc80c54b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.499+0000 7efccda55640 1 -- 172.21.15.98:0/1663104107 >> 172.21.15.98:0/1663104107 conn(0x7efcc806aff0 msgr2=0x7efcc80c27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.500+0000 7efccda55640 1 -- 172.21.15.98:0/1663104107 shutdown_connections 2024-03-06T07:20:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.500+0000 7efccda55640 1 -- 172.21.15.98:0/1663104107 wait complete. 2024-03-06T07:20:06.509 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:06.507+0000 7fe965466040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:20:06.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 69c21780-8b78-49d6-b246-e9d21297f754 7 2024-03-06T07:20:06.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.610+0000 7fe31d525640 1 Processor -- start 2024-03-06T07:20:06.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.610+0000 7fe31d525640 1 -- start start 2024-03-06T07:20:06.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.611+0000 7fe31d525640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe3180c8970 0x7fe3181564d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.611+0000 7fe31d525640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe318156a10 0x7fe31815d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.611+0000 7fe31d525640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3180c8db0 con 0x7fe3180c8590 2024-03-06T07:20:06.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.611+0000 7fe31d525640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe318153a20 con 0x7fe3180c8970 2024-03-06T07:20:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.611+0000 7fe31d525640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3180c9010 con 0x7fe318156a10 2024-03-06T07:20:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.611+0000 7fe3177fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe318156a10 0x7fe31815d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.611+0000 7fe316ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe3180c8590 0x7fe3180c3210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47994/0 (socket says 172.21.15.98:47994) 2024-03-06T07:20:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.611+0000 7fe316ffd640 1 -- 172.21.15.98:0/1872120155 learned_addr learned my addr 172.21.15.98:0/1872120155 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-06T07:20:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.611+0000 7fe30ffff640 1 --2- 172.21.15.98:0/1872120155 >> v2:172.21.15.1:3300/0 conn(0x7fe3180c8970 0x7fe3181564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.611+0000 7fe3177fe640 1 -- 172.21.15.98:0/1872120155 >> v2:172.21.15.1:3300/0 conn(0x7fe3180c8970 msgr2=0x7fe3181564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.611+0000 7fe3177fe640 1 --2- 172.21.15.98:0/1872120155 >> v2:172.21.15.1:3300/0 conn(0x7fe3180c8970 0x7fe3181564d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.611+0000 7fe3167fc640 1 -- 172.21.15.98:0/1872120155 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254137498 0 0) 0x7fe3180c8db0 con 0x7fe3180c8590 2024-03-06T07:20:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.611+0000 7fe3177fe640 1 -- 172.21.15.98:0/1872120155 >> v1:172.21.15.98:6789/0 conn(0x7fe3180c8590 legacy=0x7fe3180c3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.611+0000 7fe3177fe640 1 -- 172.21.15.98:0/1872120155 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3181530f0 con 0x7fe318156a10 2024-03-06T07:20:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.612+0000 7fe3177fe640 1 --2- 172.21.15.98:0/1872120155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe318156a10 0x7fe31815d860 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe310009d90 tx=0x7fe31002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b97733749bcc8a41 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.612+0000 7fe3167fc640 1 -- 172.21.15.98:0/1872120155 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe310039070 con 0x7fe318156a10 2024-03-06T07:20:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.612+0000 7fe3167fc640 1 -- 172.21.15.98:0/1872120155 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31002cc50 con 0x7fe318156a10 2024-03-06T07:20:06.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.612+0000 7fe3167fc640 1 -- 172.21.15.98:0/1872120155 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe310034440 con 0x7fe318156a10 2024-03-06T07:20:06.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.613+0000 7fe31d525640 1 -- 172.21.15.98:0/1872120155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe318156a10 msgr2=0x7fe31815d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.613+0000 7fe31d525640 1 --2- 172.21.15.98:0/1872120155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe318156a10 0x7fe31815d860 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe310009d90 tx=0x7fe31002bea0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.613+0000 7fe31d525640 1 -- 172.21.15.98:0/1872120155 shutdown_connections 2024-03-06T07:20:06.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.613+0000 7fe31d525640 1 --2- 172.21.15.98:0/1872120155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe318156a10 0x7fe31815d860 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.613+0000 7fe31d525640 1 --2- 172.21.15.98:0/1872120155 >> v2:172.21.15.1:3300/0 conn(0x7fe3180c8970 0x7fe3181564d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.613+0000 7fe31d525640 1 -- 172.21.15.98:0/1872120155 >> 172.21.15.98:0/1872120155 conn(0x7fe3180aa7a0 msgr2=0x7fe3180aab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:06.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.613+0000 7fe31d525640 1 -- 172.21.15.98:0/1872120155 shutdown_connections 2024-03-06T07:20:06.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.613+0000 7fe31d525640 1 -- 172.21.15.98:0/1872120155 wait complete. 2024-03-06T07:20:06.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.614+0000 7fe31d525640 1 Processor -- start 2024-03-06T07:20:06.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.614+0000 7fe31d525640 1 -- start start 2024-03-06T07:20:06.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe31d525640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3180c8590 0x7fe318152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe31d525640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3180c8970 0x7fe3181489f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe31d525640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe318156a10 0x7fe318148f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe31d525640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe318070060 con 0x7fe3180c8970 2024-03-06T07:20:06.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe31d525640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3181536b0 con 0x7fe3180c8590 2024-03-06T07:20:06.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe31d525640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3180701e0 con 0x7fe318156a10 2024-03-06T07:20:06.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe3177fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe318156a10 0x7fe318148f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe316ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3180c8590 0x7fe318152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe316ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3180c8590 0x7fe318152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44088/0 (socket says 172.21.15.98:44088) 2024-03-06T07:20:06.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe316ffd640 1 -- 172.21.15.98:0/3376941051 learned_addr learned my addr 172.21.15.98:0/3376941051 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-06T07:20:06.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe3177fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe318156a10 0x7fe318148f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:47386/0 (socket says 172.21.15.98:47386) 2024-03-06T07:20:06.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe30ffff640 1 --2- 172.21.15.98:0/3376941051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3180c8970 0x7fe3181489f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe316ffd640 1 -- 172.21.15.98:0/3376941051 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe318156a10 msgr2=0x7fe318148f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe316ffd640 1 --2- 172.21.15.98:0/3376941051 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe318156a10 0x7fe318148f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe316ffd640 1 -- 172.21.15.98:0/3376941051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3180c8970 msgr2=0x7fe3181489f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe316ffd640 1 --2- 172.21.15.98:0/3376941051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3180c8970 0x7fe3181489f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe316ffd640 1 -- 172.21.15.98:0/3376941051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3181eba10 con 0x7fe3180c8590 2024-03-06T07:20:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe3177fe640 1 --2- 172.21.15.98:0/3376941051 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe318156a10 0x7fe318148f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe316ffd640 1 --2- 172.21.15.98:0/3376941051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3180c8590 0x7fe318152450 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe30800c430 tx=0x7fe30800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe30f7fe640 1 -- 172.21.15.98:0/3376941051 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3080074e0 con 0x7fe3180c8590 2024-03-06T07:20:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe30f7fe640 1 -- 172.21.15.98:0/3376941051 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3080047e0 con 0x7fe3180c8590 2024-03-06T07:20:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe31d525640 1 -- 172.21.15.98:0/3376941051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3181ebcf0 con 0x7fe3180c8590 2024-03-06T07:20:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.615+0000 7fe30f7fe640 1 -- 172.21.15.98:0/3376941051 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308004bd0 con 0x7fe3180c8590 2024-03-06T07:20:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.616+0000 7fe31d525640 1 -- 172.21.15.98:0/3376941051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3181ec2b0 con 0x7fe3180c8590 2024-03-06T07:20:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.616+0000 7fe30f7fe640 1 -- 172.21.15.98:0/3376941051 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe308011070 con 0x7fe3180c8590 2024-03-06T07:20:06.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.616+0000 7fe30f7fe640 1 -- 172.21.15.98:0/3376941051 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fe308013a20 con 0x7fe3180c8590 2024-03-06T07:20:06.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.616+0000 7fe31d525640 1 -- 172.21.15.98:0/3376941051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3180686e0 con 0x7fe3180c8590 2024-03-06T07:20:06.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.618+0000 7fe30f7fe640 1 -- 172.21.15.98:0/3376941051 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe308004d70 con 0x7fe3180c8590 2024-03-06T07:20:06.651 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:06.649+0000 7fe965466040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:20:06.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.691+0000 7fe31d525640 1 -- 172.21.15.98:0/3376941051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd new", "uuid": "69c21780-8b78-49d6-b246-e9d21297f754", "id": 7} v 0) v1 -- 0x7fe3180c8370 con 0x7fe3180c8590 2024-03-06T07:20:06.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.695+0000 7fe30f7fe640 1 -- 172.21.15.98:0/3376941051 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "69c21780-8b78-49d6-b246-e9d21297f754", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe308013c20 con 0x7fe3180c8590 2024-03-06T07:20:06.696 INFO:teuthology.orchestra.run.smithi098.stdout:7 2024-03-06T07:20:06.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.695+0000 7fe31d525640 1 -- 172.21.15.98:0/3376941051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3180c8590 msgr2=0x7fe318152450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.695+0000 7fe31d525640 1 --2- 172.21.15.98:0/3376941051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3180c8590 0x7fe318152450 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe30800c430 tx=0x7fe30800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.696+0000 7fe31d525640 1 -- 172.21.15.98:0/3376941051 shutdown_connections 2024-03-06T07:20:06.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.696+0000 7fe31d525640 1 --2- 172.21.15.98:0/3376941051 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe318156a10 0x7fe318148f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.696+0000 7fe31d525640 1 --2- 172.21.15.98:0/3376941051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3180c8970 0x7fe3181489f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.696+0000 7fe31d525640 1 --2- 172.21.15.98:0/3376941051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3180c8590 0x7fe318152450 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.696+0000 7fe31d525640 1 -- 172.21.15.98:0/3376941051 >> 172.21.15.98:0/3376941051 conn(0x7fe3180aa7a0 msgr2=0x7fe3180c7120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:06.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.696+0000 7fe31d525640 1 -- 172.21.15.98:0/3376941051 shutdown_connections 2024-03-06T07:20:06.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-06T07:20:06.696+0000 7fe31d525640 1 -- 172.21.15.98:0/3376941051 wait complete. 2024-03-06T07:20:06.708 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-06T07:20:06.708 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-06T07:20:06.710 INFO:tasks.ceph.osd.0:Started 2024-03-06T07:20:06.711 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-06T07:20:06.711 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-06T07:20:06.713 INFO:tasks.ceph.osd.1:Started 2024-03-06T07:20:06.713 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-06T07:20:06.714 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-06T07:20:06.716 INFO:tasks.ceph.osd.2:Started 2024-03-06T07:20:06.716 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-06T07:20:06.716 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-06T07:20:06.718 INFO:tasks.ceph.osd.3:Started 2024-03-06T07:20:06.718 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-06T07:20:06.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-06T07:20:06.720 INFO:tasks.ceph.osd.4:Started 2024-03-06T07:20:06.720 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-06T07:20:06.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-06T07:20:06.722 INFO:tasks.ceph.osd.5:Started 2024-03-06T07:20:06.722 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-06T07:20:06.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-06T07:20:06.724 INFO:tasks.ceph.osd.6:Started 2024-03-06T07:20:06.724 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-06T07:20:06.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-06T07:20:06.727 INFO:tasks.ceph.osd.7:Started 2024-03-06T07:20:06.727 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:06.784 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:06.782+0000 7fe965466040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:20:06.843 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:06.841+0000 7fe965466040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:20:06.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.857+0000 7f631777a640 1 Processor -- start 2024-03-06T07:20:06.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.858+0000 7f631777a640 1 -- start start 2024-03-06T07:20:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.858+0000 7f631777a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f63100c4560 0x7f6310155c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.858+0000 7f631777a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63100c39c0 0x7f63101561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.858+0000 7f631777a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63100c7420 con 0x7f63100c4940 2024-03-06T07:20:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.858+0000 7f631777a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63101566f0 con 0x7f63100c39c0 2024-03-06T07:20:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.858+0000 7f631777a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63100ab760 con 0x7f63100c4560 2024-03-06T07:20:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.858+0000 7f6314d15640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63100c39c0 0x7f63101561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.859+0000 7f6314d15640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63100c39c0 0x7f63101561b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60166/0 (socket says 172.21.15.1:60166) 2024-03-06T07:20:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.859+0000 7f6314d15640 1 -- 172.21.15.1:0/3094042222 learned_addr learned my addr 172.21.15.1:0/3094042222 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.859+0000 7f6315516640 1 --2- 172.21.15.1:0/3094042222 >> v2:172.21.15.98:3300/0 conn(0x7f63100c4560 0x7f6310155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.859+0000 7f6314d15640 1 -- 172.21.15.1:0/3094042222 >> v2:172.21.15.98:3300/0 conn(0x7f63100c4560 msgr2=0x7f6310155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.859+0000 7f6314d15640 1 --2- 172.21.15.1:0/3094042222 >> v2:172.21.15.98:3300/0 conn(0x7f63100c4560 0x7f6310155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.859+0000 7f6314d15640 1 -- 172.21.15.1:0/3094042222 >> v1:172.21.15.98:6789/0 conn(0x7f63100c4940 legacy=0x7f63100ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.859+0000 7f6314d15640 1 -- 172.21.15.1:0/3094042222 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63100c76c0 con 0x7f63100c39c0 2024-03-06T07:20:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.860+0000 7f6314d15640 1 --2- 172.21.15.1:0/3094042222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63100c39c0 0x7f63101561b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f63100c5ce0 tx=0x7f630002e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc167f8b21ef6c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.860+0000 7f6307fff640 1 -- 172.21.15.1:0/3094042222 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630003b070 con 0x7f63100c39c0 2024-03-06T07:20:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.860+0000 7f6307fff640 1 -- 172.21.15.1:0/3094042222 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6300036440 con 0x7f63100c39c0 2024-03-06T07:20:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.860+0000 7f6307fff640 1 -- 172.21.15.1:0/3094042222 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300036810 con 0x7f63100c39c0 2024-03-06T07:20:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.860+0000 7f631777a640 1 -- 172.21.15.1:0/3094042222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63100c39c0 msgr2=0x7f63101561b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.860+0000 7f631777a640 1 --2- 172.21.15.1:0/3094042222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63100c39c0 0x7f63101561b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f63100c5ce0 tx=0x7f630002e170 comp rx=0 tx=0).stop 2024-03-06T07:20:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.860+0000 7f631777a640 1 -- 172.21.15.1:0/3094042222 shutdown_connections 2024-03-06T07:20:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.860+0000 7f631777a640 1 --2- 172.21.15.1:0/3094042222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63100c39c0 0x7f63101561b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.860+0000 7f631777a640 1 --2- 172.21.15.1:0/3094042222 >> v2:172.21.15.98:3300/0 conn(0x7f63100c4560 0x7f6310155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.860+0000 7f631777a640 1 -- 172.21.15.1:0/3094042222 >> 172.21.15.1:0/3094042222 conn(0x7f63100c7a50 msgr2=0x7f63100c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.860+0000 7f631777a640 1 -- 172.21.15.1:0/3094042222 shutdown_connections 2024-03-06T07:20:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.860+0000 7f631777a640 1 -- 172.21.15.1:0/3094042222 wait complete. 2024-03-06T07:20:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.862+0000 7f631777a640 1 Processor -- start 2024-03-06T07:20:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.862+0000 7f631777a640 1 -- start start 2024-03-06T07:20:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.862+0000 7f631777a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63100c39c0 0x7f63101eec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.862+0000 7f631777a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63100c4560 0x7f63100a9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.862+0000 7f631777a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63100c4940 0x7f63101eb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.862+0000 7f631777a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631006e6d0 con 0x7f63100c4940 2024-03-06T07:20:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.862+0000 7f631777a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63100abd00 con 0x7f63100c4560 2024-03-06T07:20:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.862+0000 7f631777a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f631006e850 con 0x7f63100c39c0 2024-03-06T07:20:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.862+0000 7f6315d17640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63100c4940 0x7f63101eb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.862+0000 7f6315d17640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63100c4940 0x7f63101eb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60180/0 (socket says 172.21.15.1:60180) 2024-03-06T07:20:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.862+0000 7f6315d17640 1 -- 172.21.15.1:0/1610847402 learned_addr learned my addr 172.21.15.1:0/1610847402 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.862+0000 7f6315516640 1 --2- 172.21.15.1:0/1610847402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63100c39c0 0x7f63101eec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.862+0000 7f6314d15640 1 --2- 172.21.15.1:0/1610847402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63100c4560 0x7f63100a9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.863+0000 7f6315d17640 1 -- 172.21.15.1:0/1610847402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63100c39c0 msgr2=0x7f63101eec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.863+0000 7f6315d17640 1 --2- 172.21.15.1:0/1610847402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63100c39c0 0x7f63101eec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.863+0000 7f6315d17640 1 -- 172.21.15.1:0/1610847402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63100c4560 msgr2=0x7f63100a9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.863+0000 7f6315d17640 1 --2- 172.21.15.1:0/1610847402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63100c4560 0x7f63100a9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.863+0000 7f6315d17640 1 -- 172.21.15.1:0/1610847402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631006fe40 con 0x7f63100c4940 2024-03-06T07:20:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.863+0000 7f6315d17640 1 --2- 172.21.15.1:0/1610847402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63100c4940 0x7f63101eb3d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f630800c700 tx=0x7f630800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.863+0000 7f6305ffb640 1 -- 172.21.15.1:0/1610847402 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6308003b70 con 0x7f63100c4940 2024-03-06T07:20:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.863+0000 7f6305ffb640 1 -- 172.21.15.1:0/1610847402 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6308003d10 con 0x7f63100c4940 2024-03-06T07:20:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.863+0000 7f631777a640 1 -- 172.21.15.1:0/1610847402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6310070120 con 0x7f63100c4940 2024-03-06T07:20:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.863+0000 7f6305ffb640 1 -- 172.21.15.1:0/1610847402 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6308005310 con 0x7f63100c4940 2024-03-06T07:20:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.863+0000 7f631777a640 1 -- 172.21.15.1:0/1610847402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63101eaeb0 con 0x7f63100c4940 2024-03-06T07:20:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.863+0000 7f6305ffb640 1 -- 172.21.15.1:0/1610847402 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f63080054b0 con 0x7f63100c4940 2024-03-06T07:20:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.863+0000 7f6305ffb640 1 -- 172.21.15.1:0/1610847402 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f6308012b70 con 0x7f63100c4940 2024-03-06T07:20:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.864+0000 7f631777a640 1 -- 172.21.15.1:0/1610847402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63100686e0 con 0x7f63100c4940 2024-03-06T07:20:06.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.868+0000 7f6305ffb640 1 -- 172.21.15.1:0/1610847402 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6308012e10 con 0x7f63100c4940 2024-03-06T07:20:06.930 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:06.928+0000 7f52f5642040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:20:06.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.931+0000 7f631777a640 1 -- 172.21.15.1:0/1610847402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63100c2230 con 0x7f63100c4940 2024-03-06T07:20:06.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.931+0000 7f6305ffb640 1 -- 172.21.15.1:0/1610847402 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f630800a640 con 0x7f63100c4940 2024-03-06T07:20:06.934 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:06.934 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":10,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:06.693057+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:06.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.932+0000 7f631777a640 1 -- 172.21.15.1:0/1610847402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63100c4940 msgr2=0x7f63101eb3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.932+0000 7f631777a640 1 --2- 172.21.15.1:0/1610847402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63100c4940 0x7f63101eb3d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f630800c700 tx=0x7f630800cbc0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.932+0000 7f631777a640 1 -- 172.21.15.1:0/1610847402 shutdown_connections 2024-03-06T07:20:06.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.932+0000 7f631777a640 1 --2- 172.21.15.1:0/1610847402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63100c4940 0x7f63101eb3d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.932+0000 7f631777a640 1 --2- 172.21.15.1:0/1610847402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63100c4560 0x7f63100a9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.932+0000 7f631777a640 1 --2- 172.21.15.1:0/1610847402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63100c39c0 0x7f63101eec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.932+0000 7f631777a640 1 -- 172.21.15.1:0/1610847402 >> 172.21.15.1:0/1610847402 conn(0x7f63100c7a50 msgr2=0x7f63100c7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:06.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.932+0000 7f631777a640 1 -- 172.21.15.1:0/1610847402 shutdown_connections 2024-03-06T07:20:06.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:06.932+0000 7f631777a640 1 -- 172.21.15.1:0/1610847402 wait complete. 2024-03-06T07:20:06.945 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-06T07:20:06.945 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-06T07:20:06.955 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:06.953+0000 7fe965466040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:20:07.049 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:07.047+0000 7fe965466040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:20:07.071 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:07.069+0000 7f52f5642040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:20:07.182 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:07.180+0000 7f52f5642040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:20:07.205 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:07.203+0000 7fe965466040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:20:07.229 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:07.227+0000 7f52f5642040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:20:07.246 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:07.244+0000 7fe965466040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:20:07.323 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-06T07:20:07.321+0000 7f35818cc640 -1 Falling back to public interface 2024-03-06T07:20:07.328 INFO:tasks.ceph.osd.6.smithi098.stderr:2024-03-06T07:20:07.326+0000 7f3e97668640 -1 Falling back to public interface 2024-03-06T07:20:07.333 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-06T07:20:07.331+0000 7fa98880f640 -1 Falling back to public interface 2024-03-06T07:20:07.336 INFO:tasks.ceph.osd.4.smithi098.stderr:2024-03-06T07:20:07.334+0000 7f33c260d640 -1 Falling back to public interface 2024-03-06T07:20:07.337 INFO:tasks.ceph.osd.5.smithi098.stderr:2024-03-06T07:20:07.335+0000 7f27b26f9640 -1 Falling back to public interface 2024-03-06T07:20:07.341 INFO:tasks.ceph.osd.7.smithi098.stderr:2024-03-06T07:20:07.339+0000 7f2968413640 -1 Falling back to public interface 2024-03-06T07:20:07.341 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-06T07:20:07.339+0000 7f04cf8ae640 -1 Falling back to public interface 2024-03-06T07:20:07.342 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-06T07:20:07.339+0000 7f3a5dc0e640 -1 Falling back to public interface 2024-03-06T07:20:07.348 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:07.346+0000 7f52f5642040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:20:07.405 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:07.403+0000 7fe965466040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:20:07.447 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:07.446+0000 7f52f5642040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:20:07.608 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:07.606+0000 7f52f5642040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:20:07.652 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:07.650+0000 7f52f5642040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:20:07.749 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:07.814 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:07.812+0000 7f52f5642040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:20:07.839 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.837+0000 7f597dad4640 1 Processor -- start 2024-03-06T07:20:07.839 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.837+0000 7f597dad4640 1 -- start start 2024-03-06T07:20:07.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.838+0000 7f597dad4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f59780c3ce0 0x7f5978158700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:07.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.838+0000 7f597dad4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f59780c36e0 0x7f59780c52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:07.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.838+0000 7f597dad4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59780c7510 con 0x7f597806da20 2024-03-06T07:20:07.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.838+0000 7f597dad4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5978155ce0 con 0x7f59780c3ce0 2024-03-06T07:20:07.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.838+0000 7f597dad4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5978155fc0 con 0x7f59780c36e0 2024-03-06T07:20:07.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.838+0000 7f5976ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f59780c3ce0 0x7f5978158700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:07.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.838+0000 7f59777fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f597806da20 0x7f59780c6e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36942/0 (socket says 172.21.15.1:36942) 2024-03-06T07:20:07.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.838+0000 7f59777fe640 1 -- 172.21.15.1:0/3934747327 learned_addr learned my addr 172.21.15.1:0/3934747327 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:07.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.838+0000 7f5977fff640 1 --2- 172.21.15.1:0/3934747327 >> v2:172.21.15.98:3300/0 conn(0x7f59780c36e0 0x7f59780c52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:07.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.838+0000 7f5976ffd640 1 -- 172.21.15.1:0/3934747327 >> v2:172.21.15.98:3300/0 conn(0x7f59780c36e0 msgr2=0x7f59780c52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:07.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.838+0000 7f5976ffd640 1 --2- 172.21.15.1:0/3934747327 >> v2:172.21.15.98:3300/0 conn(0x7f59780c36e0 0x7f59780c52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.838+0000 7f5976ffd640 1 -- 172.21.15.1:0/3934747327 >> v1:172.21.15.1:6789/0 conn(0x7f597806da20 legacy=0x7f59780c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:07.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.838+0000 7f5976ffd640 1 -- 172.21.15.1:0/3934747327 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59780c8080 con 0x7f59780c3ce0 2024-03-06T07:20:07.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.838+0000 7f5977fff640 1 --2- 172.21.15.1:0/3934747327 >> v2:172.21.15.98:3300/0 conn(0x7f59780c36e0 0x7f59780c52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:07.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.839+0000 7f5976ffd640 1 --2- 172.21.15.1:0/3934747327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59780c3ce0 0x7f5978158700 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f59780a85a0 tx=0x7f596802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8143d45f138486d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:07.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.839+0000 7f59767fc640 1 -- 172.21.15.1:0/3934747327 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968038070 con 0x7f59780c3ce0 2024-03-06T07:20:07.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.839+0000 7f59767fc640 1 -- 172.21.15.1:0/3934747327 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5968031710 con 0x7f59780c3ce0 2024-03-06T07:20:07.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.839+0000 7f59767fc640 1 -- 172.21.15.1:0/3934747327 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968031ae0 con 0x7f59780c3ce0 2024-03-06T07:20:07.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.840+0000 7f597dad4640 1 -- 172.21.15.1:0/3934747327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59780c3ce0 msgr2=0x7f5978158700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:07.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.840+0000 7f597dad4640 1 --2- 172.21.15.1:0/3934747327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59780c3ce0 0x7f5978158700 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f59780a85a0 tx=0x7f596802b7b0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.840+0000 7f597dad4640 1 -- 172.21.15.1:0/3934747327 shutdown_connections 2024-03-06T07:20:07.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.840+0000 7f597dad4640 1 --2- 172.21.15.1:0/3934747327 >> v2:172.21.15.98:3300/0 conn(0x7f59780c36e0 0x7f59780c52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.840+0000 7f597dad4640 1 --2- 172.21.15.1:0/3934747327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59780c3ce0 0x7f5978158700 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.840+0000 7f597dad4640 1 -- 172.21.15.1:0/3934747327 >> 172.21.15.1:0/3934747327 conn(0x7f597806b620 msgr2=0x7f59780c88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:07.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.840+0000 7f597dad4640 1 -- 172.21.15.1:0/3934747327 shutdown_connections 2024-03-06T07:20:07.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.840+0000 7f597dad4640 1 -- 172.21.15.1:0/3934747327 wait complete. 2024-03-06T07:20:07.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.841+0000 7f597dad4640 1 Processor -- start 2024-03-06T07:20:07.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.841+0000 7f597dad4640 1 -- start start 2024-03-06T07:20:07.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.842+0000 7f597dad4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f597806da20 0x7f597814e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:07.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.842+0000 7f597dad4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59780c36e0 0x7f5978154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:07.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.842+0000 7f597dad4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59780c3ce0 0x7f59781edea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:07.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.842+0000 7f597dad4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597806df80 con 0x7f59780c3ce0 2024-03-06T07:20:07.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.842+0000 7f597dad4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597806de00 con 0x7f59780c36e0 2024-03-06T07:20:07.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.842+0000 7f597dad4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59780c67d0 con 0x7f597806da20 2024-03-06T07:20:07.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.842+0000 7f5977fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59780c3ce0 0x7f59781edea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:07.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.842+0000 7f5977fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59780c3ce0 0x7f59781edea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60208/0 (socket says 172.21.15.1:60208) 2024-03-06T07:20:07.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.842+0000 7f5977fff640 1 -- 172.21.15.1:0/3675443351 learned_addr learned my addr 172.21.15.1:0/3675443351 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:07.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.842+0000 7f5976ffd640 1 --2- 172.21.15.1:0/3675443351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59780c36e0 0x7f5978154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:07.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.842+0000 7f59777fe640 1 --2- 172.21.15.1:0/3675443351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f597806da20 0x7f597814e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:07.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.842+0000 7f5977fff640 1 -- 172.21.15.1:0/3675443351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f597806da20 msgr2=0x7f597814e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:07.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.842+0000 7f5977fff640 1 --2- 172.21.15.1:0/3675443351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f597806da20 0x7f597814e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.842+0000 7f5977fff640 1 -- 172.21.15.1:0/3675443351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59780c36e0 msgr2=0x7f5978154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:07.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.842+0000 7f5977fff640 1 --2- 172.21.15.1:0/3675443351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59780c36e0 0x7f5978154370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.842+0000 7f5977fff640 1 -- 172.21.15.1:0/3675443351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5978157750 con 0x7f59780c3ce0 2024-03-06T07:20:07.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.842+0000 7f5977fff640 1 --2- 172.21.15.1:0/3675443351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59780c3ce0 0x7f59781edea0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5970007e60 tx=0x7f597000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:07.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.843+0000 7f596ffff640 1 -- 172.21.15.1:0/3675443351 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597001a070 con 0x7f59780c3ce0 2024-03-06T07:20:07.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.843+0000 7f596ffff640 1 -- 172.21.15.1:0/3675443351 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5970003bb0 con 0x7f59780c3ce0 2024-03-06T07:20:07.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.843+0000 7f597dad4640 1 -- 172.21.15.1:0/3675443351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5978157ce0 con 0x7f59780c3ce0 2024-03-06T07:20:07.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.843+0000 7f596ffff640 1 -- 172.21.15.1:0/3675443351 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597000de20 con 0x7f59780c3ce0 2024-03-06T07:20:07.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.843+0000 7f597dad4640 1 -- 172.21.15.1:0/3675443351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59781582a0 con 0x7f59780c3ce0 2024-03-06T07:20:07.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.843+0000 7f596ffff640 1 -- 172.21.15.1:0/3675443351 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f597001d070 con 0x7f59780c3ce0 2024-03-06T07:20:07.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.843+0000 7f596ffff640 1 -- 172.21.15.1:0/3675443351 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f5970015b10 con 0x7f59780c3ce0 2024-03-06T07:20:07.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.844+0000 7f597dad4640 1 -- 172.21.15.1:0/3675443351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f597804f3a0 con 0x7f59780c3ce0 2024-03-06T07:20:07.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.846+0000 7f596ffff640 1 -- 172.21.15.1:0/3675443351 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5970024020 con 0x7f59780c3ce0 2024-03-06T07:20:07.911 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.909+0000 7f597dad4640 1 -- 172.21.15.1:0/3675443351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59780c97a0 con 0x7f59780c3ce0 2024-03-06T07:20:07.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.909+0000 7f596ffff640 1 -- 172.21.15.1:0/3675443351 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f5970010070 con 0x7f59780c3ce0 2024-03-06T07:20:07.912 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-06T07:20:07.912 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:06.693057+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:07.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.910+0000 7f597dad4640 1 -- 172.21.15.1:0/3675443351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59780c3ce0 msgr2=0x7f59781edea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:07.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.910+0000 7f597dad4640 1 --2- 172.21.15.1:0/3675443351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59780c3ce0 0x7f59781edea0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5970007e60 tx=0x7f597000d4a0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.910+0000 7f597dad4640 1 -- 172.21.15.1:0/3675443351 shutdown_connections 2024-03-06T07:20:07.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.910+0000 7f597dad4640 1 --2- 172.21.15.1:0/3675443351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59780c3ce0 0x7f59781edea0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.910+0000 7f597dad4640 1 --2- 172.21.15.1:0/3675443351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59780c36e0 0x7f5978154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.910+0000 7f597dad4640 1 --2- 172.21.15.1:0/3675443351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f597806da20 0x7f597814e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.910+0000 7f597dad4640 1 -- 172.21.15.1:0/3675443351 >> 172.21.15.1:0/3675443351 conn(0x7f597806b620 msgr2=0x7f59780c6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:07.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.910+0000 7f597dad4640 1 -- 172.21.15.1:0/3675443351 shutdown_connections 2024-03-06T07:20:07.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:07.910+0000 7f597dad4640 1 -- 172.21.15.1:0/3675443351 wait complete. 2024-03-06T07:20:07.921 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:20:08.222 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:08.220+0000 7fe965466040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:20:08.499 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:08.497+0000 7fe965466040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:20:08.550 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:08.548+0000 7fe965466040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:20:08.641 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:08.639+0000 7f52f5642040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:20:08.663 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:08.661+0000 7fe965466040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:20:08.932 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:08.930+0000 7f52f5642040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:20:08.985 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:08.983+0000 7f52f5642040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:20:09.104 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-06T07:20:09.102+0000 7f52f5642040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:20:09.692 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:09.690+0000 7fe9068d1640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-06T07:20:09.692 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-06T07:20:09.690+0000 7fe9068d1640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-06T07:20:10.791 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-06T07:20:10.789+0000 7f35818cc640 -1 osd.2 0 log_to_monitors true 2024-03-06T07:20:10.795 INFO:tasks.ceph.osd.7.smithi098.stderr:2024-03-06T07:20:10.794+0000 7f2968413640 -1 osd.7 0 log_to_monitors true 2024-03-06T07:20:10.849 INFO:tasks.ceph.osd.5.smithi098.stderr:2024-03-06T07:20:10.847+0000 7f27b26f9640 -1 osd.5 0 log_to_monitors true 2024-03-06T07:20:10.856 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-06T07:20:10.854+0000 7f04cf8ae640 -1 osd.0 0 log_to_monitors true 2024-03-06T07:20:10.859 INFO:tasks.ceph.osd.6.smithi098.stderr:2024-03-06T07:20:10.858+0000 7f3e97668640 -1 osd.6 0 log_to_monitors true 2024-03-06T07:20:10.861 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-06T07:20:10.860+0000 7f3a5dc0e640 -1 osd.1 0 log_to_monitors true 2024-03-06T07:20:10.869 INFO:tasks.ceph.osd.4.smithi098.stderr:2024-03-06T07:20:10.868+0000 7f33c260d640 -1 osd.4 0 log_to_monitors true 2024-03-06T07:20:10.871 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-06T07:20:10.869+0000 7fa98880f640 -1 osd.3 0 log_to_monitors true 2024-03-06T07:20:13.662 INFO:tasks.ceph.osd.6.smithi098.stderr:2024-03-06T07:20:13.660+0000 7f3e935fe640 -1 osd.6 0 waiting for initial osdmap 2024-03-06T07:20:13.676 INFO:tasks.ceph.osd.7.smithi098.stderr:2024-03-06T07:20:13.674+0000 7f2963d7d640 -1 osd.7 0 waiting for initial osdmap 2024-03-06T07:20:13.677 INFO:tasks.ceph.osd.5.smithi098.stderr:2024-03-06T07:20:13.675+0000 7f27ae68f640 -1 osd.5 0 waiting for initial osdmap 2024-03-06T07:20:13.678 INFO:tasks.ceph.osd.6.smithi098.stderr:2024-03-06T07:20:13.676+0000 7f3e8b41e640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:20:13.682 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-06T07:20:13.680+0000 7f04cb844640 -1 osd.0 0 waiting for initial osdmap 2024-03-06T07:20:13.688 INFO:tasks.ceph.osd.7.smithi098.stderr:2024-03-06T07:20:13.687+0000 7f295bb9d640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:20:13.691 INFO:tasks.ceph.osd.5.smithi098.stderr:2024-03-06T07:20:13.690+0000 7f27a64af640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:20:13.692 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-06T07:20:13.691+0000 7f04c3664640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:20:13.701 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-06T07:20:13.698+0000 7f357d862640 -1 osd.2 0 waiting for initial osdmap 2024-03-06T07:20:13.703 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-06T07:20:13.702+0000 7fa984295640 -1 osd.3 0 waiting for initial osdmap 2024-03-06T07:20:13.704 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-06T07:20:13.702+0000 7f3575682640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:20:13.713 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-06T07:20:13.710+0000 7f3a59df8640 -1 osd.1 0 waiting for initial osdmap 2024-03-06T07:20:13.716 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-06T07:20:13.715+0000 7fa97c0b5640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:20:13.733 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-06T07:20:13.731+0000 7f3a51404640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:20:13.799 INFO:tasks.ceph.osd.4.smithi098.stderr:2024-03-06T07:20:13.798+0000 7f33bdfc0640 -1 osd.4 0 waiting for initial osdmap 2024-03-06T07:20:13.814 INFO:tasks.ceph.osd.4.smithi098.stderr:2024-03-06T07:20:13.812+0000 7f33b5de0640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:20:14.728 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:14.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.822+0000 7f4d0e793640 1 Processor -- start 2024-03-06T07:20:14.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.822+0000 7f4d0e793640 1 -- start start 2024-03-06T07:20:14.826 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.823+0000 7f4d0e793640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4d081568e0 0x7f4d080c7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:14.826 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.823+0000 7f4d0e793640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d080c39f0 0x7f4d08070020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:14.826 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.823+0000 7f4d0e793640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d080c3590 con 0x7f4d08156cc0 2024-03-06T07:20:14.826 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.823+0000 7f4d0e793640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d080c44c0 con 0x7f4d081568e0 2024-03-06T07:20:14.826 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.823+0000 7f4d0e793640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d080c3340 con 0x7f4d080c39f0 2024-03-06T07:20:14.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.823+0000 7f4d07fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4d081568e0 0x7f4d080c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:14.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.823+0000 7f4d077fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4d08156cc0 0x7f4d080c4c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56230/0 (socket says 172.21.15.1:56230) 2024-03-06T07:20:14.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.823+0000 7f4d077fe640 1 -- 172.21.15.1:0/69819814 learned_addr learned my addr 172.21.15.1:0/69819814 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:14.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.823+0000 7f4d07fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4d081568e0 0x7f4d080c7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35660/0 (socket says 172.21.15.1:35660) 2024-03-06T07:20:14.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.824+0000 7f4d0cd30640 1 --2- 172.21.15.1:0/69819814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d080c39f0 0x7f4d08070020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:14.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.824+0000 7f4d07fff640 1 -- 172.21.15.1:0/69819814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d080c39f0 msgr2=0x7f4d08070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:14.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.824+0000 7f4d07fff640 1 --2- 172.21.15.1:0/69819814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d080c39f0 0x7f4d08070020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:14.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.824+0000 7f4d06ffd640 1 -- 172.21.15.1:0/69819814 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071013113 0 0) 0x7f4d080c3590 con 0x7f4d08156cc0 2024-03-06T07:20:14.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.824+0000 7f4d07fff640 1 -- 172.21.15.1:0/69819814 >> v1:172.21.15.1:6789/0 conn(0x7f4d08156cc0 legacy=0x7f4d080c4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:14.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.824+0000 7f4d07fff640 1 -- 172.21.15.1:0/69819814 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d08157220 con 0x7f4d081568e0 2024-03-06T07:20:14.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.824+0000 7f4d07fff640 1 --2- 172.21.15.1:0/69819814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d081568e0 0x7f4d080c7250 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4cf80097a0 tx=0x7f4cf802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e77302d1c1b3c49a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:14.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.824+0000 7f4d06ffd640 1 -- 172.21.15.1:0/69819814 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf80315a0 con 0x7f4d081568e0 2024-03-06T07:20:14.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.824+0000 7f4d06ffd640 1 -- 172.21.15.1:0/69819814 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf8037070 con 0x7f4d081568e0 2024-03-06T07:20:14.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.825+0000 7f4d06ffd640 1 -- 172.21.15.1:0/69819814 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf8031960 con 0x7f4d081568e0 2024-03-06T07:20:14.831 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.825+0000 7f4d0e793640 1 -- 172.21.15.1:0/69819814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d081568e0 msgr2=0x7f4d080c7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:14.831 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.825+0000 7f4d0e793640 1 --2- 172.21.15.1:0/69819814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d081568e0 0x7f4d080c7250 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4cf80097a0 tx=0x7f4cf802b840 comp rx=0 tx=0).stop 2024-03-06T07:20:14.831 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.825+0000 7f4d0e793640 1 -- 172.21.15.1:0/69819814 shutdown_connections 2024-03-06T07:20:14.831 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.825+0000 7f4d0e793640 1 --2- 172.21.15.1:0/69819814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d080c39f0 0x7f4d08070020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:14.831 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.825+0000 7f4d0e793640 1 --2- 172.21.15.1:0/69819814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d081568e0 0x7f4d080c7250 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:14.831 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.825+0000 7f4d0e793640 1 -- 172.21.15.1:0/69819814 >> 172.21.15.1:0/69819814 conn(0x7f4d080aa6f0 msgr2=0x7f4d080aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:14.831 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.825+0000 7f4d0e793640 1 -- 172.21.15.1:0/69819814 shutdown_connections 2024-03-06T07:20:14.831 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.825+0000 7f4d0e793640 1 -- 172.21.15.1:0/69819814 wait complete. 2024-03-06T07:20:14.831 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.826+0000 7f4d0e793640 1 Processor -- start 2024-03-06T07:20:14.831 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.826+0000 7f4d0e793640 1 -- start start 2024-03-06T07:20:14.831 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.827+0000 7f4d0e793640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d080c39f0 0x7f4d0814a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:14.831 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.827+0000 7f4d0e793640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d081568e0 0x7f4d081483a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:14.831 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.827+0000 7f4d0e793640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d08156cc0 0x7f4d081488e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:14.832 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.827+0000 7f4d0e793640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d08150530 con 0x7f4d08156cc0 2024-03-06T07:20:14.832 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.827+0000 7f4d0e793640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d081503b0 con 0x7f4d080c39f0 2024-03-06T07:20:14.832 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.827+0000 7f4d0e793640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d081506b0 con 0x7f4d081568e0 2024-03-06T07:20:14.832 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.827+0000 7f4d0cd30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d08156cc0 0x7f4d081488e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:14.832 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.827+0000 7f4d0cd30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d08156cc0 0x7f4d081488e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35676/0 (socket says 172.21.15.1:35676) 2024-03-06T07:20:14.832 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.827+0000 7f4d0cd30640 1 -- 172.21.15.1:0/3311111942 learned_addr learned my addr 172.21.15.1:0/3311111942 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:14.832 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.827+0000 7f4d077fe640 1 --2- 172.21.15.1:0/3311111942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d081568e0 0x7f4d081483a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:14.832 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.827+0000 7f4d07fff640 1 --2- 172.21.15.1:0/3311111942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d080c39f0 0x7f4d0814a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:14.832 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.828+0000 7f4d0cd30640 1 -- 172.21.15.1:0/3311111942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d081568e0 msgr2=0x7f4d081483a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:14.832 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.828+0000 7f4d0cd30640 1 --2- 172.21.15.1:0/3311111942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d081568e0 0x7f4d081483a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:14.832 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.828+0000 7f4d0cd30640 1 -- 172.21.15.1:0/3311111942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d080c39f0 msgr2=0x7f4d0814a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:14.832 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.828+0000 7f4d0cd30640 1 --2- 172.21.15.1:0/3311111942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d080c39f0 0x7f4d0814a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:14.832 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.828+0000 7f4d0cd30640 1 -- 172.21.15.1:0/3311111942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d081538b0 con 0x7f4d08156cc0 2024-03-06T07:20:14.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.828+0000 7f4d07fff640 1 --2- 172.21.15.1:0/3311111942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d080c39f0 0x7f4d0814a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:14.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.828+0000 7f4d077fe640 1 --2- 172.21.15.1:0/3311111942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d081568e0 0x7f4d081483a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:14.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.828+0000 7f4d0cd30640 1 --2- 172.21.15.1:0/3311111942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d08156cc0 0x7f4d081488e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000cda0 tx=0x7f4d0000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:14.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.828+0000 7f4d04ff9640 1 -- 172.21.15.1:0/3311111942 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d00003b90 con 0x7f4d08156cc0 2024-03-06T07:20:14.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.828+0000 7f4d04ff9640 1 -- 172.21.15.1:0/3311111942 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d00003d30 con 0x7f4d08156cc0 2024-03-06T07:20:14.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.828+0000 7f4d04ff9640 1 -- 172.21.15.1:0/3311111942 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d00004240 con 0x7f4d08156cc0 2024-03-06T07:20:14.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.828+0000 7f4d0e793640 1 -- 172.21.15.1:0/3311111942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d08153b90 con 0x7f4d08156cc0 2024-03-06T07:20:14.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.828+0000 7f4d0e793640 1 -- 172.21.15.1:0/3311111942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d0814ac20 con 0x7f4d08156cc0 2024-03-06T07:20:14.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.829+0000 7f4d0e793640 1 -- 172.21.15.1:0/3311111942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d0804f3a0 con 0x7f4d08156cc0 2024-03-06T07:20:14.834 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.829+0000 7f4d04ff9640 1 -- 172.21.15.1:0/3311111942 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f4d000043e0 con 0x7f4d08156cc0 2024-03-06T07:20:14.834 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.830+0000 7f4d04ff9640 1 --2- 172.21.15.1:0/3311111942 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cf406b440 0x7f4cf406d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:14.834 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.830+0000 7f4d04ff9640 1 -- 172.21.15.1:0/3311111942 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f4d0008a700 con 0x7f4d08156cc0 2024-03-06T07:20:14.834 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.830+0000 7f4d07fff640 1 --2- 172.21.15.1:0/3311111942 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cf406b440 0x7f4cf406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:14.834 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.831+0000 7f4d07fff640 1 --2- 172.21.15.1:0/3311111942 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cf406b440 0x7f4cf406d900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8036000 tx=0x7f4cf8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:14.836 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:14.835+0000 7f4d04ff9640 1 -- 172.21.15.1:0/3311111942 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d0001d340 con 0x7f4d08156cc0 2024-03-06T07:20:15.021 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:15.019+0000 7f4d0e793640 1 -- 172.21.15.1:0/3311111942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d080c7ac0 con 0x7f4d08156cc0 2024-03-06T07:20:15.022 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:15.020+0000 7f4d04ff9640 1 -- 172.21.15.1:0/3311111942 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10789 (secure 0 0 0) 0x7f4d00059bc0 con 0x7f4d08156cc0 2024-03-06T07:20:15.022 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-06T07:20:15.022 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":13,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:14.686886+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:15.024 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:15.022+0000 7f4d0e793640 1 -- 172.21.15.1:0/3311111942 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cf406b440 msgr2=0x7f4cf406d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.024 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:15.022+0000 7f4d0e793640 1 --2- 172.21.15.1:0/3311111942 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cf406b440 0x7f4cf406d900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8036000 tx=0x7f4cf8008030 comp rx=0 tx=0).stop 2024-03-06T07:20:15.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:15.022+0000 7f4d0e793640 1 -- 172.21.15.1:0/3311111942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d08156cc0 msgr2=0x7f4d081488e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:15.022+0000 7f4d0e793640 1 --2- 172.21.15.1:0/3311111942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d08156cc0 0x7f4d081488e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000cda0 tx=0x7f4d0000a660 comp rx=0 tx=0).stop 2024-03-06T07:20:15.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:15.022+0000 7f4d0e793640 1 -- 172.21.15.1:0/3311111942 shutdown_connections 2024-03-06T07:20:15.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:15.022+0000 7f4d0e793640 1 --2- 172.21.15.1:0/3311111942 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cf406b440 0x7f4cf406d900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:15.022+0000 7f4d0e793640 1 --2- 172.21.15.1:0/3311111942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d08156cc0 0x7f4d081488e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:15.022+0000 7f4d0e793640 1 --2- 172.21.15.1:0/3311111942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d081568e0 0x7f4d081483a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:15.022+0000 7f4d0e793640 1 --2- 172.21.15.1:0/3311111942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d080c39f0 0x7f4d0814a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:15.022+0000 7f4d0e793640 1 -- 172.21.15.1:0/3311111942 >> 172.21.15.1:0/3311111942 conn(0x7f4d080aa6f0 msgr2=0x7f4d0806f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:15.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:15.022+0000 7f4d0e793640 1 -- 172.21.15.1:0/3311111942 shutdown_connections 2024-03-06T07:20:15.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-06T07:20:15.022+0000 7f4d0e793640 1 -- 172.21.15.1:0/3311111942 wait complete. 2024-03-06T07:20:15.034 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-06T07:20:15.034 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-06T07:20:15.034 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:20:15.034 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:20:15.037 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:20:15.037 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:20:15.037 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:20:15.039 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:20:15.039 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:20:15.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:20:15.042 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:20:15.042 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:20:15.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:20:15.044 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:20:15.045 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-06T07:20:15.047 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:15.128 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:20:15.129 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:20:15.129 INFO:tasks.ceph.mds.d.smithi098.stderr:2024-03-06T07:20:15.127+0000 7f09a886f640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-06T07:20:15.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.148+0000 7f78bfac9640 1 Processor -- start 2024-03-06T07:20:15.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.148+0000 7f78bfac9640 1 -- start start 2024-03-06T07:20:15.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.149+0000 7f78bfac9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f78b8165c30 0x7f78b8163830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.149+0000 7f78bfac9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f78b808a970 0x7f78b806d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.149+0000 7f78bfac9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b8161ec0 con 0x7f78b806d550 2024-03-06T07:20:15.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.149+0000 7f78bfac9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b81621a0 con 0x7f78b8165c30 2024-03-06T07:20:15.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.149+0000 7f78bfac9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b8162480 con 0x7f78b808a970 2024-03-06T07:20:15.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.149+0000 7f78bd064640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f78b8165c30 0x7f78b8163830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:15.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.149+0000 7f78bd865640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f78b806d550 0x7f78b8163120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56240/0 (socket says 172.21.15.1:56240) 2024-03-06T07:20:15.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.149+0000 7f78bd865640 1 -- 172.21.15.1:0/731672234 learned_addr learned my addr 172.21.15.1:0/731672234 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:15.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.149+0000 7f78be066640 1 --2- 172.21.15.1:0/731672234 >> v2:172.21.15.98:3300/0 conn(0x7f78b808a970 0x7f78b806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:15.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.149+0000 7f78bc863640 1 -- 172.21.15.1:0/731672234 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2759183395 0 0) 0x7f78b8161ec0 con 0x7f78b806d550 2024-03-06T07:20:15.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.186+0000 7f78bc863640 1 -- 172.21.15.1:0/731672234 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78a4003420 con 0x7f78b806d550 2024-03-06T07:20:15.190 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:20:15.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.186+0000 7f78bd064640 1 -- 172.21.15.1:0/731672234 >> v2:172.21.15.98:3300/0 conn(0x7f78b808a970 msgr2=0x7f78b806d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:15.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.186+0000 7f78bd064640 1 --2- 172.21.15.1:0/731672234 >> v2:172.21.15.98:3300/0 conn(0x7f78b808a970 0x7f78b806d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.186+0000 7f78bd064640 1 -- 172.21.15.1:0/731672234 >> v1:172.21.15.1:6789/0 conn(0x7f78b806d550 legacy=0x7f78b8163120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.186+0000 7f78bd064640 1 -- 172.21.15.1:0/731672234 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b8162760 con 0x7f78b8165c30 2024-03-06T07:20:15.191 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:20:15.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.187+0000 7f78bd064640 1 --2- 172.21.15.1:0/731672234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78b8165c30 0x7f78b8163830 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f78b80c5a00 tx=0x7f78a8034050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16620c0ba481b66f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:15.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.188+0000 7f78bc863640 1 -- 172.21.15.1:0/731672234 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a803a070 con 0x7f78b8165c30 2024-03-06T07:20:15.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.188+0000 7f78bc863640 1 -- 172.21.15.1:0/731672234 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a8034e00 con 0x7f78b8165c30 2024-03-06T07:20:15.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.188+0000 7f78bc863640 1 -- 172.21.15.1:0/731672234 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a803e600 con 0x7f78b8165c30 2024-03-06T07:20:15.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.188+0000 7f78bfac9640 1 -- 172.21.15.1:0/731672234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78b8165c30 msgr2=0x7f78b8163830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.188+0000 7f78bfac9640 1 --2- 172.21.15.1:0/731672234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78b8165c30 0x7f78b8163830 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f78b80c5a00 tx=0x7f78a8034050 comp rx=0 tx=0).stop 2024-03-06T07:20:15.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.188+0000 7f78bfac9640 1 -- 172.21.15.1:0/731672234 shutdown_connections 2024-03-06T07:20:15.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.188+0000 7f78bfac9640 1 --2- 172.21.15.1:0/731672234 >> v2:172.21.15.98:3300/0 conn(0x7f78b808a970 0x7f78b806d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.188+0000 7f78bfac9640 1 --2- 172.21.15.1:0/731672234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78b8165c30 0x7f78b8163830 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.188+0000 7f78bfac9640 1 -- 172.21.15.1:0/731672234 >> 172.21.15.1:0/731672234 conn(0x7f78b80c6860 msgr2=0x7f78b80c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:15.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.189+0000 7f78bfac9640 1 -- 172.21.15.1:0/731672234 shutdown_connections 2024-03-06T07:20:15.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.189+0000 7f78bfac9640 1 -- 172.21.15.1:0/731672234 wait complete. 2024-03-06T07:20:15.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.190+0000 7f78bfac9640 1 Processor -- start 2024-03-06T07:20:15.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.190+0000 7f78bfac9640 1 -- start start 2024-03-06T07:20:15.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78bfac9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78b806d550 0x7f78b81f8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78bfac9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78b808a970 0x7f78b8164760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78bfac9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78b8165c30 0x7f78b81fb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78bfac9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b816c570 con 0x7f78b808a970 2024-03-06T07:20:15.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78bfac9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b80c1630 con 0x7f78b8165c30 2024-03-06T07:20:15.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78bfac9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78b816c6f0 con 0x7f78b806d550 2024-03-06T07:20:15.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78bd064640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78b808a970 0x7f78b8164760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:15.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78bd865640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78b806d550 0x7f78b81f8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:15.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78bd865640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78b806d550 0x7f78b81f8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:59972/0 (socket says 172.21.15.1:59972) 2024-03-06T07:20:15.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78bd064640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78b808a970 0x7f78b8164760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35744/0 (socket says 172.21.15.1:35744) 2024-03-06T07:20:15.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78bd865640 1 -- 172.21.15.1:0/1147304389 learned_addr learned my addr 172.21.15.1:0/1147304389 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:15.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78be066640 1 --2- 172.21.15.1:0/1147304389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78b8165c30 0x7f78b81fb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:15.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78bd064640 1 -- 172.21.15.1:0/1147304389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78b806d550 msgr2=0x7f78b81f8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78bd064640 1 --2- 172.21.15.1:0/1147304389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78b806d550 0x7f78b81f8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78bd064640 1 -- 172.21.15.1:0/1147304389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78b8165c30 msgr2=0x7f78b81fb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78bd064640 1 --2- 172.21.15.1:0/1147304389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78b8165c30 0x7f78b81fb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78bd064640 1 -- 172.21.15.1:0/1147304389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b80c2e90 con 0x7f78b808a970 2024-03-06T07:20:15.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.191+0000 7f78bd064640 1 --2- 172.21.15.1:0/1147304389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78b808a970 0x7f78b8164760 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f78a803c000 tx=0x7f78a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:15.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.192+0000 7f78ae7fc640 1 -- 172.21.15.1:0/1147304389 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a804d070 con 0x7f78b808a970 2024-03-06T07:20:15.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.192+0000 7f78ae7fc640 1 -- 172.21.15.1:0/1147304389 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a8049850 con 0x7f78b808a970 2024-03-06T07:20:15.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.192+0000 7f78bfac9640 1 -- 172.21.15.1:0/1147304389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b8163a00 con 0x7f78b808a970 2024-03-06T07:20:15.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.192+0000 7f78ae7fc640 1 -- 172.21.15.1:0/1147304389 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a8049c00 con 0x7f78b808a970 2024-03-06T07:20:15.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.192+0000 7f78bfac9640 1 -- 172.21.15.1:0/1147304389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b8163c80 con 0x7f78b808a970 2024-03-06T07:20:15.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.193+0000 7f78ae7fc640 1 -- 172.21.15.1:0/1147304389 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f78a8034cc0 con 0x7f78b808a970 2024-03-06T07:20:15.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.193+0000 7f78bfac9640 1 -- 172.21.15.1:0/1147304389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b8067fd0 con 0x7f78b808a970 2024-03-06T07:20:15.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.193+0000 7f78ae7fc640 1 --2- 172.21.15.1:0/1147304389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f789c06b470 0x7f789c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.194+0000 7f78ae7fc640 1 -- 172.21.15.1:0/1147304389 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f78a80b4140 con 0x7f78b808a970 2024-03-06T07:20:15.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.194+0000 7f78bd865640 1 --2- 172.21.15.1:0/1147304389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f789c06b470 0x7f789c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:15.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.194+0000 7f78bd865640 1 --2- 172.21.15.1:0/1147304389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f789c06b470 0x7f789c06d930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f78b4005880 tx=0x7f78b4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:15.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.198+0000 7f78ae7fc640 1 -- 172.21.15.1:0/1147304389 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78a8056070 con 0x7f78b808a970 2024-03-06T07:20:15.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.380+0000 7f78bfac9640 1 -- 172.21.15.1:0/1147304389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78b80acbf0 con 0x7f78b808a970 2024-03-06T07:20:15.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.381+0000 7f78ae7fc640 1 -- 172.21.15.1:0/1147304389 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10789 (secure 0 0 0) 0x7f78a8096e20 con 0x7f78b808a970 2024-03-06T07:20:15.383 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:15.383 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":13,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:14.686886+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:15.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.383+0000 7f78bfac9640 1 -- 172.21.15.1:0/1147304389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f789c06b470 msgr2=0x7f789c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.383+0000 7f78bfac9640 1 --2- 172.21.15.1:0/1147304389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f789c06b470 0x7f789c06d930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f78b4005880 tx=0x7f78b4009040 comp rx=0 tx=0).stop 2024-03-06T07:20:15.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.383+0000 7f78bfac9640 1 -- 172.21.15.1:0/1147304389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78b808a970 msgr2=0x7f78b8164760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.383+0000 7f78bfac9640 1 --2- 172.21.15.1:0/1147304389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78b808a970 0x7f78b8164760 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f78a803c000 tx=0x7f78a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.383+0000 7f78bfac9640 1 -- 172.21.15.1:0/1147304389 shutdown_connections 2024-03-06T07:20:15.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.383+0000 7f78bfac9640 1 --2- 172.21.15.1:0/1147304389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f789c06b470 0x7f789c06d930 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.383+0000 7f78bfac9640 1 --2- 172.21.15.1:0/1147304389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78b8165c30 0x7f78b81fb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.383+0000 7f78bfac9640 1 --2- 172.21.15.1:0/1147304389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78b808a970 0x7f78b8164760 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.383+0000 7f78bfac9640 1 --2- 172.21.15.1:0/1147304389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78b806d550 0x7f78b81f8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.383+0000 7f78bfac9640 1 -- 172.21.15.1:0/1147304389 >> 172.21.15.1:0/1147304389 conn(0x7f78b80c6860 msgr2=0x7f78b81602c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:15.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.383+0000 7f78bfac9640 1 -- 172.21.15.1:0/1147304389 shutdown_connections 2024-03-06T07:20:15.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.383+0000 7f78bfac9640 1 -- 172.21.15.1:0/1147304389 wait complete. 2024-03-06T07:20:15.396 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-03-06T07:20:15.397 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:15.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.496+0000 7fc339c4e640 1 Processor -- start 2024-03-06T07:20:15.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.496+0000 7fc339c4e640 1 -- start start 2024-03-06T07:20:15.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.497+0000 7fc339c4e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc334152730 0x7fc3340c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.497+0000 7fc339c4e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc3340c6fd0 0x7fc3340c1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.497+0000 7fc339c4e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3340c1cf0 con 0x7fc3340c18d0 2024-03-06T07:20:15.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.497+0000 7fc339c4e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3341588e0 con 0x7fc334152730 2024-03-06T07:20:15.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.497+0000 7fc339c4e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc334158bc0 con 0x7fc3340c6fd0 2024-03-06T07:20:15.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.497+0000 7fc3337fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc334152730 0x7fc3340c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:15.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.497+0000 7fc333fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc3340c18d0 0x7fc3341552e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56248/0 (socket says 172.21.15.1:56248) 2024-03-06T07:20:15.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.497+0000 7fc333fff640 1 -- 172.21.15.1:0/1140328894 learned_addr learned my addr 172.21.15.1:0/1140328894 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:15.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.498+0000 7fc332ffd640 1 --2- 172.21.15.1:0/1140328894 >> v2:172.21.15.98:3300/0 conn(0x7fc3340c6fd0 0x7fc3340c1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:15.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.498+0000 7fc3337fe640 1 -- 172.21.15.1:0/1140328894 >> v2:172.21.15.98:3300/0 conn(0x7fc3340c6fd0 msgr2=0x7fc3340c1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:15.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.498+0000 7fc3337fe640 1 --2- 172.21.15.1:0/1140328894 >> v2:172.21.15.98:3300/0 conn(0x7fc3340c6fd0 0x7fc3340c1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.498+0000 7fc3337fe640 1 -- 172.21.15.1:0/1140328894 >> v1:172.21.15.1:6789/0 conn(0x7fc3340c18d0 legacy=0x7fc3341552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.498+0000 7fc3337fe640 1 -- 172.21.15.1:0/1140328894 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc334158ea0 con 0x7fc334152730 2024-03-06T07:20:15.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.498+0000 7fc3337fe640 1 --2- 172.21.15.1:0/1140328894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc334152730 0x7fc3340c6a90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc33408ca60 tx=0x7fc32402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb607482e5847a71 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:15.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.498+0000 7fc3327fc640 1 -- 172.21.15.1:0/1140328894 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc324038070 con 0x7fc334152730 2024-03-06T07:20:15.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.498+0000 7fc3327fc640 1 -- 172.21.15.1:0/1140328894 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc324031710 con 0x7fc334152730 2024-03-06T07:20:15.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.498+0000 7fc3327fc640 1 -- 172.21.15.1:0/1140328894 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc324031ae0 con 0x7fc334152730 2024-03-06T07:20:15.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.499+0000 7fc339c4e640 1 -- 172.21.15.1:0/1140328894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc334152730 msgr2=0x7fc3340c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.499+0000 7fc339c4e640 1 --2- 172.21.15.1:0/1140328894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc334152730 0x7fc3340c6a90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc33408ca60 tx=0x7fc32402b7b0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.499+0000 7fc339c4e640 1 -- 172.21.15.1:0/1140328894 shutdown_connections 2024-03-06T07:20:15.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.499+0000 7fc339c4e640 1 --2- 172.21.15.1:0/1140328894 >> v2:172.21.15.98:3300/0 conn(0x7fc3340c6fd0 0x7fc3340c1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.499+0000 7fc339c4e640 1 --2- 172.21.15.1:0/1140328894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc334152730 0x7fc3340c6a90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.499+0000 7fc339c4e640 1 -- 172.21.15.1:0/1140328894 >> 172.21.15.1:0/1140328894 conn(0x7fc334066d00 msgr2=0x7fc334067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:15.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.499+0000 7fc339c4e640 1 -- 172.21.15.1:0/1140328894 shutdown_connections 2024-03-06T07:20:15.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.499+0000 7fc339c4e640 1 -- 172.21.15.1:0/1140328894 wait complete. 2024-03-06T07:20:15.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.500+0000 7fc339c4e640 1 Processor -- start 2024-03-06T07:20:15.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.500+0000 7fc339c4e640 1 -- start start 2024-03-06T07:20:15.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.501+0000 7fc339c4e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3340c18d0 0x7fc334157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.501+0000 7fc339c4e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3340c6fd0 0x7fc3341def30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.501+0000 7fc339c4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc334152730 0x7fc3341df470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.501+0000 7fc339c4e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc334152b10 con 0x7fc3340c18d0 2024-03-06T07:20:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.501+0000 7fc339c4e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc334159460 con 0x7fc334152730 2024-03-06T07:20:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.501+0000 7fc339c4e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3341559f0 con 0x7fc3340c6fd0 2024-03-06T07:20:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.501+0000 7fc3337fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3340c18d0 0x7fc334157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.501+0000 7fc3337fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3340c18d0 0x7fc334157950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35766/0 (socket says 172.21.15.1:35766) 2024-03-06T07:20:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.501+0000 7fc3337fe640 1 -- 172.21.15.1:0/2201377030 learned_addr learned my addr 172.21.15.1:0/2201377030 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.501+0000 7fc3337fe640 1 -- 172.21.15.1:0/2201377030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3340c6fd0 msgr2=0x7fc3341def30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.501+0000 7fc3337fe640 1 --2- 172.21.15.1:0/2201377030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3340c6fd0 0x7fc3341def30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.501+0000 7fc333fff640 1 --2- 172.21.15.1:0/2201377030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc334152730 0x7fc3341df470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.501+0000 7fc3337fe640 1 -- 172.21.15.1:0/2201377030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc334152730 msgr2=0x7fc3341df470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.501+0000 7fc3337fe640 1 --2- 172.21.15.1:0/2201377030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc334152730 0x7fc3341df470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.501+0000 7fc3337fe640 1 -- 172.21.15.1:0/2201377030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33414e7c0 con 0x7fc3340c18d0 2024-03-06T07:20:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.502+0000 7fc3337fe640 1 --2- 172.21.15.1:0/2201377030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3340c18d0 0x7fc334157950 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc324000c00 tx=0x7fc3240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.502+0000 7fc31bfff640 1 -- 172.21.15.1:0/2201377030 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc324042070 con 0x7fc3340c18d0 2024-03-06T07:20:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.502+0000 7fc31bfff640 1 -- 172.21.15.1:0/2201377030 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc324033a40 con 0x7fc3340c18d0 2024-03-06T07:20:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.502+0000 7fc339c4e640 1 -- 172.21.15.1:0/2201377030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3341eea50 con 0x7fc3340c18d0 2024-03-06T07:20:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.502+0000 7fc31bfff640 1 -- 172.21.15.1:0/2201377030 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc324038040 con 0x7fc3340c18d0 2024-03-06T07:20:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.502+0000 7fc339c4e640 1 -- 172.21.15.1:0/2201377030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc33414ea30 con 0x7fc3340c18d0 2024-03-06T07:20:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.503+0000 7fc339c4e640 1 -- 172.21.15.1:0/2201377030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc334067fd0 con 0x7fc3340c18d0 2024-03-06T07:20:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.503+0000 7fc31bfff640 1 -- 172.21.15.1:0/2201377030 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fc32403b070 con 0x7fc3340c18d0 2024-03-06T07:20:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.504+0000 7fc31bfff640 1 --2- 172.21.15.1:0/2201377030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc32006b470 0x7fc32006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.504+0000 7fc31bfff640 1 -- 172.21.15.1:0/2201377030 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fc3240ab6b0 con 0x7fc3340c18d0 2024-03-06T07:20:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.504+0000 7fc332ffd640 1 --2- 172.21.15.1:0/2201377030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc32006b470 0x7fc32006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:15.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.504+0000 7fc332ffd640 1 --2- 172.21.15.1:0/2201377030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc32006b470 0x7fc32006d930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc328006fd0 tx=0x7fc328008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.508+0000 7fc31bfff640 1 -- 172.21.15.1:0/2201377030 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc32407ac90 con 0x7fc3340c18d0 2024-03-06T07:20:15.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.727+0000 7fc339c4e640 1 -- 172.21.15.1:0/2201377030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3340c6870 con 0x7fc3340c18d0 2024-03-06T07:20:15.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.727+0000 7fc31bfff640 1 -- 172.21.15.1:0/2201377030 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7fc32407a590 con 0x7fc3340c18d0 2024-03-06T07:20:15.730 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:15.730 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2442906233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2442906233},{"type":"v1","addr":"172.21.15.98:6837","nonce":2442906233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-06T07:20:15.730 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 2 2024-03-06T07:20:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.729+0000 7fc339c4e640 1 -- 172.21.15.1:0/2201377030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc32006b470 msgr2=0x7fc32006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.729+0000 7fc339c4e640 1 --2- 172.21.15.1:0/2201377030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc32006b470 0x7fc32006d930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc328006fd0 tx=0x7fc328008040 comp rx=0 tx=0).stop 2024-03-06T07:20:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.729+0000 7fc339c4e640 1 -- 172.21.15.1:0/2201377030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3340c18d0 msgr2=0x7fc334157950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.729+0000 7fc339c4e640 1 --2- 172.21.15.1:0/2201377030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3340c18d0 0x7fc334157950 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc324000c00 tx=0x7fc3240046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.729+0000 7fc339c4e640 1 -- 172.21.15.1:0/2201377030 shutdown_connections 2024-03-06T07:20:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.729+0000 7fc339c4e640 1 --2- 172.21.15.1:0/2201377030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc32006b470 0x7fc32006d930 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.729+0000 7fc339c4e640 1 --2- 172.21.15.1:0/2201377030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc334152730 0x7fc3341df470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.729+0000 7fc339c4e640 1 --2- 172.21.15.1:0/2201377030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3340c6fd0 0x7fc3341def30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.729+0000 7fc339c4e640 1 --2- 172.21.15.1:0/2201377030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3340c18d0 0x7fc334157950 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.729+0000 7fc339c4e640 1 -- 172.21.15.1:0/2201377030 >> 172.21.15.1:0/2201377030 conn(0x7fc334066d00 msgr2=0x7fc3340670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.730+0000 7fc339c4e640 1 -- 172.21.15.1:0/2201377030 shutdown_connections 2024-03-06T07:20:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:15.730+0000 7fc339c4e640 1 -- 172.21.15.1:0/2201377030 wait complete. 2024-03-06T07:20:17.743 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.844+0000 7f3678572640 1 Processor -- start 2024-03-06T07:20:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.844+0000 7f3678572640 1 -- start start 2024-03-06T07:20:17.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.845+0000 7f3678572640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36701543a0 0x7f367015e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:17.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.845+0000 7f3678572640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3670150900 con 0x7f3670150520 2024-03-06T07:20:17.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.845+0000 7f3678572640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3670071790 con 0x7f36700c32f0 2024-03-06T07:20:17.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.845+0000 7f3678572640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3670153ed0 con 0x7f36701543a0 2024-03-06T07:20:17.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.845+0000 7f367630e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3670150520 0x7f36701537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56262/0 (socket says 172.21.15.1:56262) 2024-03-06T07:20:17.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.845+0000 7f367630e640 1 -- 172.21.15.1:0/696923990 learned_addr learned my addr 172.21.15.1:0/696923990 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:17.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.845+0000 7f3676b0f640 1 --2- 172.21.15.1:0/696923990 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36701543a0 0x7f367015e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:17.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.845+0000 7f367530c640 1 -- 172.21.15.1:0/696923990 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667506606 0 0) 0x7f3670150900 con 0x7f3670150520 2024-03-06T07:20:17.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.845+0000 7f367530c640 1 -- 172.21.15.1:0/696923990 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f365c003520 con 0x7f3670150520 2024-03-06T07:20:17.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.846+0000 7f3676b0f640 1 -- 172.21.15.1:0/696923990 >> v1:172.21.15.98:6789/0 conn(0x7f36700c32f0 legacy=0x7f3670070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:17.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.846+0000 7f3676b0f640 1 -- 172.21.15.1:0/696923990 >> v1:172.21.15.1:6789/0 conn(0x7f3670150520 legacy=0x7f36701537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:17.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.846+0000 7f3676b0f640 1 -- 172.21.15.1:0/696923990 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3670157050 con 0x7f36701543a0 2024-03-06T07:20:17.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.846+0000 7f3676b0f640 1 --2- 172.21.15.1:0/696923990 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36701543a0 0x7f367015e840 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3668009dc0 tx=0x7f366802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=750bf9edf65cc91a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:17.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.847+0000 7f367530c640 1 -- 172.21.15.1:0/696923990 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668039070 con 0x7f36701543a0 2024-03-06T07:20:17.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.847+0000 7f367530c640 1 -- 172.21.15.1:0/696923990 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366802ccd0 con 0x7f36701543a0 2024-03-06T07:20:17.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.847+0000 7f367530c640 1 -- 172.21.15.1:0/696923990 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668034660 con 0x7f36701543a0 2024-03-06T07:20:17.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.847+0000 7f3678572640 1 -- 172.21.15.1:0/696923990 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36701543a0 msgr2=0x7f367015e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:17.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.847+0000 7f3678572640 1 --2- 172.21.15.1:0/696923990 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36701543a0 0x7f367015e840 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3668009dc0 tx=0x7f366802bf20 comp rx=0 tx=0).stop 2024-03-06T07:20:17.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.847+0000 7f3678572640 1 -- 172.21.15.1:0/696923990 shutdown_connections 2024-03-06T07:20:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.847+0000 7f3678572640 1 --2- 172.21.15.1:0/696923990 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36701543a0 0x7f367015e840 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.847+0000 7f3678572640 1 -- 172.21.15.1:0/696923990 >> 172.21.15.1:0/696923990 conn(0x7f36700c88f0 msgr2=0x7f36700c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.847+0000 7f3678572640 1 -- 172.21.15.1:0/696923990 shutdown_connections 2024-03-06T07:20:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.847+0000 7f3678572640 1 -- 172.21.15.1:0/696923990 wait complete. 2024-03-06T07:20:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.848+0000 7f3678572640 1 Processor -- start 2024-03-06T07:20:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.848+0000 7f3678572640 1 -- start start 2024-03-06T07:20:17.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.849+0000 7f3678572640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36700c32f0 0x7f36701523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:17.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.849+0000 7f3678572640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3670150520 0x7f36700a8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:17.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.849+0000 7f3678572640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36701543a0 0x7f3670155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:17.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.849+0000 7f3678572640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36701532a0 con 0x7f36700c32f0 2024-03-06T07:20:17.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.849+0000 7f3678572640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36701547e0 con 0x7f36701543a0 2024-03-06T07:20:17.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.849+0000 7f3678572640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3670157330 con 0x7f3670150520 2024-03-06T07:20:17.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.849+0000 7f367630e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36700c32f0 0x7f36701523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:17.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.849+0000 7f367630e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36700c32f0 0x7f36701523a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35776/0 (socket says 172.21.15.1:35776) 2024-03-06T07:20:17.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.849+0000 7f367630e640 1 -- 172.21.15.1:0/2963151466 learned_addr learned my addr 172.21.15.1:0/2963151466 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:17.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.849+0000 7f3675b0d640 1 --2- 172.21.15.1:0/2963151466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3670150520 0x7f36700a8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:17.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.849+0000 7f3676b0f640 1 --2- 172.21.15.1:0/2963151466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36701543a0 0x7f3670155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:17.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.849+0000 7f367630e640 1 -- 172.21.15.1:0/2963151466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3670150520 msgr2=0x7f36700a8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:17.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.849+0000 7f367630e640 1 --2- 172.21.15.1:0/2963151466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3670150520 0x7f36700a8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:17.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.849+0000 7f367630e640 1 -- 172.21.15.1:0/2963151466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36701543a0 msgr2=0x7f3670155830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:17.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.849+0000 7f367630e640 1 --2- 172.21.15.1:0/2963151466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36701543a0 0x7f3670155830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:17.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.849+0000 7f367630e640 1 -- 172.21.15.1:0/2963151466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3670152aa0 con 0x7f36700c32f0 2024-03-06T07:20:17.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.849+0000 7f367630e640 1 --2- 172.21.15.1:0/2963151466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36700c32f0 0x7f36701523a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f366000c530 tx=0x7f366000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:17.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.850+0000 7f3666ffd640 1 -- 172.21.15.1:0/2963151466 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3660017070 con 0x7f36700c32f0 2024-03-06T07:20:17.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.850+0000 7f3666ffd640 1 -- 172.21.15.1:0/2963151466 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3660005ce0 con 0x7f36700c32f0 2024-03-06T07:20:17.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.850+0000 7f3678572640 1 -- 172.21.15.1:0/2963151466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36701515b0 con 0x7f36700c32f0 2024-03-06T07:20:17.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.850+0000 7f3666ffd640 1 -- 172.21.15.1:0/2963151466 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36600072b0 con 0x7f36700c32f0 2024-03-06T07:20:17.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.850+0000 7f3678572640 1 -- 172.21.15.1:0/2963151466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3670151b70 con 0x7f36700c32f0 2024-03-06T07:20:17.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.851+0000 7f3678572640 1 -- 172.21.15.1:0/2963151466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36700686e0 con 0x7f36700c32f0 2024-03-06T07:20:17.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.851+0000 7f3666ffd640 1 -- 172.21.15.1:0/2963151466 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f366001a070 con 0x7f36700c32f0 2024-03-06T07:20:17.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.851+0000 7f3666ffd640 1 --2- 172.21.15.1:0/2963151466 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f365406b470 0x7f365406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:17.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.852+0000 7f3666ffd640 1 -- 172.21.15.1:0/2963151466 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3660089190 con 0x7f36700c32f0 2024-03-06T07:20:17.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.852+0000 7f3675b0d640 1 --2- 172.21.15.1:0/2963151466 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f365406b470 0x7f365406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:17.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.852+0000 7f3675b0d640 1 --2- 172.21.15.1:0/2963151466 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f365406b470 0x7f365406d930 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f367014bb80 tx=0x7f366c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:17.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:17.856+0000 7f3666ffd640 1 -- 172.21.15.1:0/2963151466 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3660057e20 con 0x7f36700c32f0 2024-03-06T07:20:18.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.074+0000 7f3678572640 1 -- 172.21.15.1:0/2963151466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f367014e900 con 0x7f36700c32f0 2024-03-06T07:20:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.074+0000 7f3666ffd640 1 -- 172.21.15.1:0/2963151466 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3685 (secure 0 0 0) 0x7f366005bec0 con 0x7f36700c32f0 2024-03-06T07:20:18.076 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:18.076 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2822958969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2822958969},{"type":"v1","addr":"172.21.15.98:6836","nonce":2822958969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2442906233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2442906233},{"type":"v1","addr":"172.21.15.98:6837","nonce":2442906233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/448746024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":448746024},{"type":"v1","addr":"172.21.15.1:6834","nonce":448746024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1315115982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1315115982},{"type":"v1","addr":"172.21.15.1:6835","nonce":1315115982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-06T07:20:18.077 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 3 2024-03-06T07:20:18.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.076+0000 7f3678572640 1 -- 172.21.15.1:0/2963151466 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f365406b470 msgr2=0x7f365406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:18.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.076+0000 7f3678572640 1 --2- 172.21.15.1:0/2963151466 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f365406b470 0x7f365406d930 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f367014bb80 tx=0x7f366c009040 comp rx=0 tx=0).stop 2024-03-06T07:20:18.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.076+0000 7f3678572640 1 -- 172.21.15.1:0/2963151466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36700c32f0 msgr2=0x7f36701523a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.076+0000 7f3678572640 1 --2- 172.21.15.1:0/2963151466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36700c32f0 0x7f36701523a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f366000c530 tx=0x7f366000c9f0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.076+0000 7f3678572640 1 -- 172.21.15.1:0/2963151466 shutdown_connections 2024-03-06T07:20:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.076+0000 7f3678572640 1 --2- 172.21.15.1:0/2963151466 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f365406b470 0x7f365406d930 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.076+0000 7f3678572640 1 --2- 172.21.15.1:0/2963151466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36701543a0 0x7f3670155830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.076+0000 7f3678572640 1 --2- 172.21.15.1:0/2963151466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3670150520 0x7f36700a8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.076+0000 7f3678572640 1 --2- 172.21.15.1:0/2963151466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36700c32f0 0x7f36701523a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.076+0000 7f3678572640 1 -- 172.21.15.1:0/2963151466 >> 172.21.15.1:0/2963151466 conn(0x7f36700c88f0 msgr2=0x7f367014a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.076+0000 7f3678572640 1 -- 172.21.15.1:0/2963151466 shutdown_connections 2024-03-06T07:20:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.076+0000 7f3678572640 1 -- 172.21.15.1:0/2963151466 wait complete. 2024-03-06T07:20:18.089 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:18.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.187+0000 7f54ad78c640 1 Processor -- start 2024-03-06T07:20:18.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.187+0000 7f54ad78c640 1 -- start start 2024-03-06T07:20:18.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.188+0000 7f54ad78c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54a80c4560 0x7f54a8155c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.188+0000 7f54ad78c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54a80c39c0 0x7f54a81561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.188+0000 7f54ad78c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a80c7420 con 0x7f54a80c4940 2024-03-06T07:20:18.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.188+0000 7f54ad78c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54a81566f0 con 0x7f54a80c39c0 2024-03-06T07:20:18.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.188+0000 7f54ad78c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a80ab760 con 0x7f54a80c4560 2024-03-06T07:20:18.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.188+0000 7f54a67fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54a80c39c0 0x7f54a81561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.188+0000 7f54a77fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f54a80c4940 0x7f54a80ac220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56272/0 (socket says 172.21.15.1:56272) 2024-03-06T07:20:18.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.188+0000 7f54a77fe640 1 -- 172.21.15.1:0/1591380758 learned_addr learned my addr 172.21.15.1:0/1591380758 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:18.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.188+0000 7f54a6ffd640 1 --2- 172.21.15.1:0/1591380758 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54a80c4560 0x7f54a8155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.188+0000 7f54a5ffb640 1 -- 172.21.15.1:0/1591380758 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092287752 0 0) 0x7f54a80c7420 con 0x7f54a80c4940 2024-03-06T07:20:18.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.188+0000 7f54a5ffb640 1 -- 172.21.15.1:0/1591380758 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5494003520 con 0x7f54a80c4940 2024-03-06T07:20:18.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.189+0000 7f54a67fc640 1 -- 172.21.15.1:0/1591380758 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54a80c4560 msgr2=0x7f54a8155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:18.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.189+0000 7f54a67fc640 1 --2- 172.21.15.1:0/1591380758 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54a80c4560 0x7f54a8155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.189+0000 7f54a67fc640 1 -- 172.21.15.1:0/1591380758 >> v1:172.21.15.1:6789/0 conn(0x7f54a80c4940 legacy=0x7f54a80ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:18.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.189+0000 7f54a67fc640 1 -- 172.21.15.1:0/1591380758 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a80c76c0 con 0x7f54a80c39c0 2024-03-06T07:20:18.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.189+0000 7f54a67fc640 1 --2- 172.21.15.1:0/1591380758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54a80c39c0 0x7f54a81561b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f54900097a0 tx=0x7f549002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b63b0c416459d78c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:18.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.189+0000 7f54a5ffb640 1 -- 172.21.15.1:0/1591380758 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5490035020 con 0x7f54a80c39c0 2024-03-06T07:20:18.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.189+0000 7f54a5ffb640 1 -- 172.21.15.1:0/1591380758 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5490031830 con 0x7f54a80c39c0 2024-03-06T07:20:18.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.189+0000 7f54a5ffb640 1 -- 172.21.15.1:0/1591380758 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5490031c00 con 0x7f54a80c39c0 2024-03-06T07:20:18.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.190+0000 7f54ad78c640 1 -- 172.21.15.1:0/1591380758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54a80c39c0 msgr2=0x7f54a81561b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:18.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.190+0000 7f54ad78c640 1 --2- 172.21.15.1:0/1591380758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54a80c39c0 0x7f54a81561b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f54900097a0 tx=0x7f549002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.190+0000 7f54ad78c640 1 -- 172.21.15.1:0/1591380758 shutdown_connections 2024-03-06T07:20:18.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.190+0000 7f54ad78c640 1 --2- 172.21.15.1:0/1591380758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54a80c39c0 0x7f54a81561b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.190+0000 7f54ad78c640 1 --2- 172.21.15.1:0/1591380758 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54a80c4560 0x7f54a8155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.190+0000 7f54ad78c640 1 -- 172.21.15.1:0/1591380758 >> 172.21.15.1:0/1591380758 conn(0x7f54a80c7a50 msgr2=0x7f54a80c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:18.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.190+0000 7f54ad78c640 1 -- 172.21.15.1:0/1591380758 shutdown_connections 2024-03-06T07:20:18.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.190+0000 7f54ad78c640 1 -- 172.21.15.1:0/1591380758 wait complete. 2024-03-06T07:20:18.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.191+0000 7f54ad78c640 1 Processor -- start 2024-03-06T07:20:18.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.191+0000 7f54ad78c640 1 -- start start 2024-03-06T07:20:18.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.192+0000 7f54ad78c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54a80c39c0 0x7f54a806b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.192+0000 7f54ad78c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54a80c4560 0x7f54a8152520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.192+0000 7f54ad78c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54a80c4940 0x7f54a814c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.192+0000 7f54ad78c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a806e6d0 con 0x7f54a80c39c0 2024-03-06T07:20:18.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.192+0000 7f54ad78c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a80abd00 con 0x7f54a80c4940 2024-03-06T07:20:18.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.192+0000 7f54ad78c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a806e850 con 0x7f54a80c4560 2024-03-06T07:20:18.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.192+0000 7f54a6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54a80c39c0 0x7f54a806b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.192+0000 7f54a6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54a80c39c0 0x7f54a806b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35786/0 (socket says 172.21.15.1:35786) 2024-03-06T07:20:18.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.192+0000 7f54a6ffd640 1 -- 172.21.15.1:0/3972703596 learned_addr learned my addr 172.21.15.1:0/3972703596 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:18.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.192+0000 7f54a67fc640 1 --2- 172.21.15.1:0/3972703596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54a80c4560 0x7f54a8152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.192+0000 7f54a77fe640 1 --2- 172.21.15.1:0/3972703596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54a80c4940 0x7f54a814c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.192+0000 7f54a6ffd640 1 -- 172.21.15.1:0/3972703596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54a80c4560 msgr2=0x7f54a8152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:18.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.192+0000 7f54a6ffd640 1 --2- 172.21.15.1:0/3972703596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54a80c4560 0x7f54a8152520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.192+0000 7f54a6ffd640 1 -- 172.21.15.1:0/3972703596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54a80c4940 msgr2=0x7f54a814c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:18.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.192+0000 7f54a6ffd640 1 --2- 172.21.15.1:0/3972703596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54a80c4940 0x7f54a814c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.192+0000 7f54a6ffd640 1 -- 172.21.15.1:0/3972703596 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a8150d70 con 0x7f54a80c39c0 2024-03-06T07:20:18.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.192+0000 7f54a77fe640 1 --2- 172.21.15.1:0/3972703596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54a80c4940 0x7f54a814c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:18.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.193+0000 7f54a6ffd640 1 --2- 172.21.15.1:0/3972703596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54a80c39c0 0x7f54a806b260 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f54a000e9b0 tx=0x7f54a000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:18.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.193+0000 7f549f7fe640 1 -- 172.21.15.1:0/3972703596 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0017070 con 0x7f54a80c39c0 2024-03-06T07:20:18.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.193+0000 7f549f7fe640 1 -- 172.21.15.1:0/3972703596 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a000c9a0 con 0x7f54a80c39c0 2024-03-06T07:20:18.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.193+0000 7f54ad78c640 1 -- 172.21.15.1:0/3972703596 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a8151050 con 0x7f54a80c39c0 2024-03-06T07:20:18.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.193+0000 7f549f7fe640 1 -- 172.21.15.1:0/3972703596 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a000cd90 con 0x7f54a80c39c0 2024-03-06T07:20:18.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.193+0000 7f54ad78c640 1 -- 172.21.15.1:0/3972703596 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a8151610 con 0x7f54a80c39c0 2024-03-06T07:20:18.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.194+0000 7f54ad78c640 1 -- 172.21.15.1:0/3972703596 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a80686e0 con 0x7f54a80c39c0 2024-03-06T07:20:18.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.194+0000 7f549f7fe640 1 -- 172.21.15.1:0/3972703596 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f54a001a070 con 0x7f54a80c39c0 2024-03-06T07:20:18.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.195+0000 7f549f7fe640 1 --2- 172.21.15.1:0/3972703596 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f548806b440 0x7f548806d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.195+0000 7f549f7fe640 1 -- 172.21.15.1:0/3972703596 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f54a008a090 con 0x7f54a80c39c0 2024-03-06T07:20:18.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.195+0000 7f54a67fc640 1 --2- 172.21.15.1:0/3972703596 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f548806b440 0x7f548806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.195+0000 7f54a67fc640 1 --2- 172.21.15.1:0/3972703596 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f548806b440 0x7f548806d900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5490036000 tx=0x7f5490008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:18.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.199+0000 7f549f7fe640 1 -- 172.21.15.1:0/3972703596 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54a00593d0 con 0x7f54a80c39c0 2024-03-06T07:20:18.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.385+0000 7f54ad78c640 1 -- 172.21.15.1:0/3972703596 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54a80c5da0 con 0x7f54a80c39c0 2024-03-06T07:20:18.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.386+0000 7f549f7fe640 1 -- 172.21.15.1:0/3972703596 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12764 (secure 0 0 0) 0x7f54a0058cd0 con 0x7f54a80c39c0 2024-03-06T07:20:18.388 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:18.388 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":16,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:17.811483+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:18.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.388+0000 7f54ad78c640 1 -- 172.21.15.1:0/3972703596 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f548806b440 msgr2=0x7f548806d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:18.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.388+0000 7f54ad78c640 1 --2- 172.21.15.1:0/3972703596 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f548806b440 0x7f548806d900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5490036000 tx=0x7f5490008030 comp rx=0 tx=0).stop 2024-03-06T07:20:18.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.388+0000 7f54ad78c640 1 -- 172.21.15.1:0/3972703596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54a80c39c0 msgr2=0x7f54a806b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:18.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.388+0000 7f54ad78c640 1 --2- 172.21.15.1:0/3972703596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54a80c39c0 0x7f54a806b260 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f54a000e9b0 tx=0x7f54a000ee70 comp rx=0 tx=0).stop 2024-03-06T07:20:18.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.388+0000 7f54ad78c640 1 -- 172.21.15.1:0/3972703596 shutdown_connections 2024-03-06T07:20:18.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.388+0000 7f54ad78c640 1 --2- 172.21.15.1:0/3972703596 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f548806b440 0x7f548806d900 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.388+0000 7f54ad78c640 1 --2- 172.21.15.1:0/3972703596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54a80c4940 0x7f54a814c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.388+0000 7f54ad78c640 1 --2- 172.21.15.1:0/3972703596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54a80c4560 0x7f54a8152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.388+0000 7f54ad78c640 1 --2- 172.21.15.1:0/3972703596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54a80c39c0 0x7f54a806b260 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.388+0000 7f54ad78c640 1 -- 172.21.15.1:0/3972703596 >> 172.21.15.1:0/3972703596 conn(0x7f54a80c7a50 msgr2=0x7f54a814f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:18.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.388+0000 7f54ad78c640 1 -- 172.21.15.1:0/3972703596 shutdown_connections 2024-03-06T07:20:18.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.388+0000 7f54ad78c640 1 -- 172.21.15.1:0/3972703596 wait complete. 2024-03-06T07:20:18.401 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:20:18.401 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:20:18.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.503+0000 7f4a5d64e640 1 Processor -- start 2024-03-06T07:20:18.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.503+0000 7f4a5d64e640 1 -- start start 2024-03-06T07:20:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.504+0000 7f4a5d64e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4a580c4560 0x7f4a58155c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.504+0000 7f4a5d64e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4a580c39c0 0x7f4a581561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.504+0000 7f4a5d64e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a580c4940 0x7f4a580ac220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.504+0000 7f4a5d64e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a581566f0 con 0x7f4a580c4560 2024-03-06T07:20:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.504+0000 7f4a5d64e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a580c7420 con 0x7f4a580c39c0 2024-03-06T07:20:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.504+0000 7f4a5d64e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a580ab760 con 0x7f4a580c4940 2024-03-06T07:20:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.504+0000 7f4a56ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4a580c4560 0x7f4a58155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.504+0000 7f4a56ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4a580c4560 0x7f4a58155c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35790/0 (socket says 172.21.15.1:35790) 2024-03-06T07:20:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.504+0000 7f4a56ffd640 1 -- 172.21.15.1:0/3670803881 learned_addr learned my addr 172.21.15.1:0/3670803881 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.504+0000 7f4a577fe640 1 --2- 172.21.15.1:0/3670803881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a580c4940 0x7f4a580ac220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.504+0000 7f4a567fc640 1 --2- 172.21.15.1:0/3670803881 >> v2:172.21.15.98:3300/0 conn(0x7f4a580c39c0 0x7f4a581561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.505+0000 7f4a56ffd640 1 -- 172.21.15.1:0/3670803881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a580c4940 msgr2=0x7f4a580ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:18.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.505+0000 7f4a56ffd640 1 --2- 172.21.15.1:0/3670803881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a580c4940 0x7f4a580ac220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.505+0000 7f4a56ffd640 1 -- 172.21.15.1:0/3670803881 >> v2:172.21.15.98:3300/0 conn(0x7f4a580c39c0 msgr2=0x7f4a581561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:18.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.505+0000 7f4a56ffd640 1 --2- 172.21.15.1:0/3670803881 >> v2:172.21.15.98:3300/0 conn(0x7f4a580c39c0 0x7f4a581561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.505+0000 7f4a56ffd640 1 -- 172.21.15.1:0/3670803881 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a580c76c0 con 0x7f4a580c4560 2024-03-06T07:20:18.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.505+0000 7f4a56ffd640 1 --2- 172.21.15.1:0/3670803881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a580c4560 0x7f4a58155c70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4a580c5cb0 tx=0x7f4a4802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f5c216475f6a62 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:18.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.505+0000 7f4a55ffb640 1 -- 172.21.15.1:0/3670803881 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4802ca70 con 0x7f4a580c4560 2024-03-06T07:20:18.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.505+0000 7f4a55ffb640 1 -- 172.21.15.1:0/3670803881 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4802cc10 con 0x7f4a580c4560 2024-03-06T07:20:18.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.505+0000 7f4a55ffb640 1 -- 172.21.15.1:0/3670803881 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a480343f0 con 0x7f4a580c4560 2024-03-06T07:20:18.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.506+0000 7f4a5d64e640 1 -- 172.21.15.1:0/3670803881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a580c4560 msgr2=0x7f4a58155c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:18.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.506+0000 7f4a5d64e640 1 --2- 172.21.15.1:0/3670803881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a580c4560 0x7f4a58155c70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4a580c5cb0 tx=0x7f4a4802bed0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.506+0000 7f4a5d64e640 1 -- 172.21.15.1:0/3670803881 shutdown_connections 2024-03-06T07:20:18.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.506+0000 7f4a5d64e640 1 --2- 172.21.15.1:0/3670803881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a580c4940 0x7f4a580ac220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.506+0000 7f4a5d64e640 1 --2- 172.21.15.1:0/3670803881 >> v2:172.21.15.98:3300/0 conn(0x7f4a580c39c0 0x7f4a581561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.506+0000 7f4a5d64e640 1 --2- 172.21.15.1:0/3670803881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a580c4560 0x7f4a58155c70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.506+0000 7f4a5d64e640 1 -- 172.21.15.1:0/3670803881 >> 172.21.15.1:0/3670803881 conn(0x7f4a580c7a50 msgr2=0x7f4a580c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:18.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.506+0000 7f4a5d64e640 1 -- 172.21.15.1:0/3670803881 shutdown_connections 2024-03-06T07:20:18.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.506+0000 7f4a5d64e640 1 -- 172.21.15.1:0/3670803881 wait complete. 2024-03-06T07:20:18.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.507+0000 7f4a5d64e640 1 Processor -- start 2024-03-06T07:20:18.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.507+0000 7f4a5d64e640 1 -- start start 2024-03-06T07:20:18.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.508+0000 7f4a5d64e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a580c39c0 0x7f4a581db9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.508+0000 7f4a5d64e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a580c4560 0x7f4a581dcae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.508+0000 7f4a5d64e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a580c4940 0x7f4a581dd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.508+0000 7f4a5d64e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5806e6d0 con 0x7f4a580c39c0 2024-03-06T07:20:18.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.508+0000 7f4a5d64e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a580abd00 con 0x7f4a580c4560 2024-03-06T07:20:18.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.508+0000 7f4a5d64e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5806e850 con 0x7f4a580c4940 2024-03-06T07:20:18.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.508+0000 7f4a56ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a580c39c0 0x7f4a581db9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.508+0000 7f4a56ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a580c39c0 0x7f4a581db9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35796/0 (socket says 172.21.15.1:35796) 2024-03-06T07:20:18.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.508+0000 7f4a56ffd640 1 -- 172.21.15.1:0/651756674 learned_addr learned my addr 172.21.15.1:0/651756674 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:18.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.508+0000 7f4a567fc640 1 --2- 172.21.15.1:0/651756674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a580c4560 0x7f4a581dcae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.508+0000 7f4a577fe640 1 --2- 172.21.15.1:0/651756674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a580c4940 0x7f4a581dd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.508+0000 7f4a56ffd640 1 -- 172.21.15.1:0/651756674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a580c4940 msgr2=0x7f4a581dd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:18.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.508+0000 7f4a56ffd640 1 --2- 172.21.15.1:0/651756674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a580c4940 0x7f4a581dd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.508+0000 7f4a56ffd640 1 -- 172.21.15.1:0/651756674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a580c4560 msgr2=0x7f4a581dcae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:18.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.508+0000 7f4a56ffd640 1 --2- 172.21.15.1:0/651756674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a580c4560 0x7f4a581dcae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.508+0000 7f4a56ffd640 1 -- 172.21.15.1:0/651756674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a581e95d0 con 0x7f4a580c39c0 2024-03-06T07:20:18.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.509+0000 7f4a56ffd640 1 --2- 172.21.15.1:0/651756674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a580c39c0 0x7f4a581db9d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4a48036040 tx=0x7f4a48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:18.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.509+0000 7f4a4f7fe640 1 -- 172.21.15.1:0/651756674 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48043020 con 0x7f4a580c39c0 2024-03-06T07:20:18.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.509+0000 7f4a4f7fe640 1 -- 172.21.15.1:0/651756674 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4803e450 con 0x7f4a580c39c0 2024-03-06T07:20:18.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.509+0000 7f4a5d64e640 1 -- 172.21.15.1:0/651756674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a581dc000 con 0x7f4a580c39c0 2024-03-06T07:20:18.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.509+0000 7f4a4f7fe640 1 -- 172.21.15.1:0/651756674 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48009510 con 0x7f4a580c39c0 2024-03-06T07:20:18.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.509+0000 7f4a5d64e640 1 -- 172.21.15.1:0/651756674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a581dd9a0 con 0x7f4a580c39c0 2024-03-06T07:20:18.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.510+0000 7f4a5d64e640 1 -- 172.21.15.1:0/651756674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a580c5f50 con 0x7f4a580c39c0 2024-03-06T07:20:18.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.510+0000 7f4a4f7fe640 1 -- 172.21.15.1:0/651756674 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f4a48046020 con 0x7f4a580c39c0 2024-03-06T07:20:18.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.510+0000 7f4a4f7fe640 1 --2- 172.21.15.1:0/651756674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a3806b440 0x7f4a3806d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.511+0000 7f4a4f7fe640 1 -- 172.21.15.1:0/651756674 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4a480ac4a0 con 0x7f4a580c39c0 2024-03-06T07:20:18.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.511+0000 7f4a567fc640 1 --2- 172.21.15.1:0/651756674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a3806b440 0x7f4a3806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.511+0000 7f4a567fc640 1 --2- 172.21.15.1:0/651756674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a3806b440 0x7f4a3806d900 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4a580c8310 tx=0x7f4a40008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:18.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.515+0000 7f4a4f7fe640 1 -- 172.21.15.1:0/651756674 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a580c5f50 con 0x7f4a580c39c0 2024-03-06T07:20:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.694+0000 7f4a5d64e640 1 -- 172.21.15.1:0/651756674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4a580686e0 con 0x7f4a580c39c0 2024-03-06T07:20:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.694+0000 7f4a4f7fe640 1 -- 172.21.15.1:0/651756674 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4a4804d030 con 0x7f4a580c39c0 2024-03-06T07:20:18.696 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:20:18.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.696+0000 7f4a5d64e640 1 -- 172.21.15.1:0/651756674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a3806b440 msgr2=0x7f4a3806d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:18.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.696+0000 7f4a5d64e640 1 --2- 172.21.15.1:0/651756674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a3806b440 0x7f4a3806d900 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4a580c8310 tx=0x7f4a40008040 comp rx=0 tx=0).stop 2024-03-06T07:20:18.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.696+0000 7f4a5d64e640 1 -- 172.21.15.1:0/651756674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a580c39c0 msgr2=0x7f4a581db9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:18.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.696+0000 7f4a5d64e640 1 --2- 172.21.15.1:0/651756674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a580c39c0 0x7f4a581db9d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4a48036040 tx=0x7f4a48004a40 comp rx=0 tx=0).stop 2024-03-06T07:20:18.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.696+0000 7f4a5d64e640 1 -- 172.21.15.1:0/651756674 shutdown_connections 2024-03-06T07:20:18.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.696+0000 7f4a5d64e640 1 --2- 172.21.15.1:0/651756674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a3806b440 0x7f4a3806d900 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.696+0000 7f4a5d64e640 1 --2- 172.21.15.1:0/651756674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a580c4940 0x7f4a581dd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.696+0000 7f4a5d64e640 1 --2- 172.21.15.1:0/651756674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a580c4560 0x7f4a581dcae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.696+0000 7f4a5d64e640 1 --2- 172.21.15.1:0/651756674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a580c39c0 0x7f4a581db9d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.696+0000 7f4a5d64e640 1 -- 172.21.15.1:0/651756674 >> 172.21.15.1:0/651756674 conn(0x7f4a580c7a50 msgr2=0x7f4a580c50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:18.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.696+0000 7f4a5d64e640 1 -- 172.21.15.1:0/651756674 shutdown_connections 2024-03-06T07:20:18.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.696+0000 7f4a5d64e640 1 -- 172.21.15.1:0/651756674 wait complete. 2024-03-06T07:20:18.709 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:20:18.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.807+0000 7f7501033640 1 Processor -- start 2024-03-06T07:20:18.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.808+0000 7f7501033640 1 -- start start 2024-03-06T07:20:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.808+0000 7f7501033640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f74fc0c6b10 0x7f74fc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.808+0000 7f7501033640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74fc0bf9d0 0x7f74fc0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.808+0000 7f7501033640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74fc0c2fc0 con 0x7f74fc152650 2024-03-06T07:20:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.808+0000 7f7501033640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74fc155800 con 0x7f74fc0c6b10 2024-03-06T07:20:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.808+0000 7f7501033640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc158da0 con 0x7f74fc0bf9d0 2024-03-06T07:20:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.808+0000 7f74f3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f74fc0c6b10 0x7f74fc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.808+0000 7f74f3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f74fc0c6b10 0x7f74fc155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35800/0 (socket says 172.21.15.1:35800) 2024-03-06T07:20:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.808+0000 7f74f3fff640 1 -- 172.21.15.1:0/4100618532 learned_addr learned my addr 172.21.15.1:0/4100618532 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.809+0000 7f74fb59e640 1 --2- 172.21.15.1:0/4100618532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74fc0bf9d0 0x7f74fc0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.809+0000 7f74f3fff640 1 -- 172.21.15.1:0/4100618532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74fc0bf9d0 msgr2=0x7f74fc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:18.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.809+0000 7f74f3fff640 1 --2- 172.21.15.1:0/4100618532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74fc0bf9d0 0x7f74fc0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.809+0000 7f74f3fff640 1 -- 172.21.15.1:0/4100618532 >> v1:172.21.15.98:6789/0 conn(0x7f74fc152650 legacy=0x7f74fc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:18.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.809+0000 7f74f3fff640 1 -- 172.21.15.1:0/4100618532 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74fc0c6690 con 0x7f74fc0c6b10 2024-03-06T07:20:18.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.809+0000 7f74f3fff640 1 --2- 172.21.15.1:0/4100618532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc0c6b10 0x7f74fc155c90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f74e4009870 tx=0x7f74e402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f936fbf12cd53a14 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.809+0000 7f74fa59c640 1 -- 172.21.15.1:0/4100618532 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e4035020 con 0x7f74fc0c6b10 2024-03-06T07:20:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.810+0000 7f74fa59c640 1 -- 172.21.15.1:0/4100618532 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e4031640 con 0x7f74fc0c6b10 2024-03-06T07:20:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.810+0000 7f74fa59c640 1 -- 172.21.15.1:0/4100618532 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e402d040 con 0x7f74fc0c6b10 2024-03-06T07:20:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.810+0000 7f7501033640 1 -- 172.21.15.1:0/4100618532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc0c6b10 msgr2=0x7f74fc155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.810+0000 7f7501033640 1 --2- 172.21.15.1:0/4100618532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc0c6b10 0x7f74fc155c90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f74e4009870 tx=0x7f74e402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.810+0000 7f7501033640 1 -- 172.21.15.1:0/4100618532 shutdown_connections 2024-03-06T07:20:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.810+0000 7f7501033640 1 --2- 172.21.15.1:0/4100618532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74fc0bf9d0 0x7f74fc0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.810+0000 7f7501033640 1 --2- 172.21.15.1:0/4100618532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc0c6b10 0x7f74fc155c90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.810+0000 7f7501033640 1 -- 172.21.15.1:0/4100618532 >> 172.21.15.1:0/4100618532 conn(0x7f74fc066ee0 msgr2=0x7f74fc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:18.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.810+0000 7f7501033640 1 -- 172.21.15.1:0/4100618532 shutdown_connections 2024-03-06T07:20:18.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.810+0000 7f7501033640 1 -- 172.21.15.1:0/4100618532 wait complete. 2024-03-06T07:20:18.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.811+0000 7f7501033640 1 Processor -- start 2024-03-06T07:20:18.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.811+0000 7f7501033640 1 -- start start 2024-03-06T07:20:18.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.812+0000 7f7501033640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74fc0bf9d0 0x7f74fc14e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.812+0000 7f7501033640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc152650 0x7f74fc14aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.812+0000 7f7501033640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74fc14b020 0x7f74fc1ee370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.812+0000 7f7501033640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc1551b0 con 0x7f74fc152650 2024-03-06T07:20:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.812+0000 7f7501033640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc0c6fa0 con 0x7f74fc0bf9d0 2024-03-06T07:20:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.812+0000 7f7501033640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc155490 con 0x7f74fc14b020 2024-03-06T07:20:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.812+0000 7f74f3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc152650 0x7f74fc14aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.812+0000 7f74f3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc152650 0x7f74fc14aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35814/0 (socket says 172.21.15.1:35814) 2024-03-06T07:20:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.812+0000 7f74f3fff640 1 -- 172.21.15.1:0/1678359224 learned_addr learned my addr 172.21.15.1:0/1678359224 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.812+0000 7f74fb59e640 1 --2- 172.21.15.1:0/1678359224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74fc14b020 0x7f74fc1ee370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.812+0000 7f74fad9d640 1 --2- 172.21.15.1:0/1678359224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74fc0bf9d0 0x7f74fc14e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.812+0000 7f74f3fff640 1 -- 172.21.15.1:0/1678359224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74fc14b020 msgr2=0x7f74fc1ee370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.812+0000 7f74f3fff640 1 --2- 172.21.15.1:0/1678359224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74fc14b020 0x7f74fc1ee370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.812+0000 7f74f3fff640 1 -- 172.21.15.1:0/1678359224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74fc0bf9d0 msgr2=0x7f74fc14e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.812+0000 7f74f3fff640 1 --2- 172.21.15.1:0/1678359224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74fc0bf9d0 0x7f74fc14e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.812+0000 7f74f3fff640 1 -- 172.21.15.1:0/1678359224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74fc158a00 con 0x7f74fc152650 2024-03-06T07:20:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.812+0000 7f74fb59e640 1 --2- 172.21.15.1:0/1678359224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74fc14b020 0x7f74fc1ee370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.813+0000 7f74f3fff640 1 --2- 172.21.15.1:0/1678359224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc152650 0x7f74fc14aae0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f74e402b930 tx=0x7f74e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.813+0000 7f74f37fe640 1 -- 172.21.15.1:0/1678359224 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e4040070 con 0x7f74fc152650 2024-03-06T07:20:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.813+0000 7f74f37fe640 1 -- 172.21.15.1:0/1678359224 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e403e070 con 0x7f74fc152650 2024-03-06T07:20:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.813+0000 7f74f37fe640 1 -- 172.21.15.1:0/1678359224 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e4035040 con 0x7f74fc152650 2024-03-06T07:20:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.813+0000 7f7501033640 1 -- 172.21.15.1:0/1678359224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74fc1e0e40 con 0x7f74fc152650 2024-03-06T07:20:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.813+0000 7f7501033640 1 -- 172.21.15.1:0/1678359224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74fc14ea30 con 0x7f74fc152650 2024-03-06T07:20:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.814+0000 7f7501033640 1 -- 172.21.15.1:0/1678359224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74fc067fd0 con 0x7f74fc152650 2024-03-06T07:20:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.814+0000 7f74f37fe640 1 -- 172.21.15.1:0/1678359224 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f74e4038020 con 0x7f74fc152650 2024-03-06T07:20:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.814+0000 7f74f37fe640 1 --2- 172.21.15.1:0/1678359224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74dc06b440 0x7f74dc06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.815+0000 7f74f37fe640 1 -- 172.21.15.1:0/1678359224 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f74e40ab750 con 0x7f74fc152650 2024-03-06T07:20:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.815+0000 7f74fad9d640 1 --2- 172.21.15.1:0/1678359224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74dc06b440 0x7f74dc06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.815+0000 7f74fad9d640 1 --2- 172.21.15.1:0/1678359224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74dc06b440 0x7f74dc06d900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f74fc14f690 tx=0x7f74ec009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:18.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.819+0000 7f74f37fe640 1 -- 172.21.15.1:0/1678359224 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74e407aa90 con 0x7f74fc152650 2024-03-06T07:20:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:18.827+0000 7f74f37fe640 1 -- 172.21.15.1:0/1678359224 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74e4076a00 con 0x7f74fc152650 2024-03-06T07:20:19.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.003+0000 7f7501033640 1 -- 172.21.15.1:0/1678359224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f74fc154e80 con 0x7f74fc152650 2024-03-06T07:20:19.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.003+0000 7f74f37fe640 1 -- 172.21.15.1:0/1678359224 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f74e40a5960 con 0x7f74fc152650 2024-03-06T07:20:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:19.006 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-06T07:20:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:20:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-06T07:20:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-06T07:20:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:20:19.006 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-06T07:20:19.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.005+0000 7f7501033640 1 -- 172.21.15.1:0/1678359224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74dc06b440 msgr2=0x7f74dc06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:19.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.005+0000 7f7501033640 1 --2- 172.21.15.1:0/1678359224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74dc06b440 0x7f74dc06d900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f74fc14f690 tx=0x7f74ec009040 comp rx=0 tx=0).stop 2024-03-06T07:20:19.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.005+0000 7f7501033640 1 -- 172.21.15.1:0/1678359224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc152650 msgr2=0x7f74fc14aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:19.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.005+0000 7f7501033640 1 --2- 172.21.15.1:0/1678359224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc152650 0x7f74fc14aae0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f74e402b930 tx=0x7f74e40046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.005+0000 7f7501033640 1 -- 172.21.15.1:0/1678359224 shutdown_connections 2024-03-06T07:20:19.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.005+0000 7f7501033640 1 --2- 172.21.15.1:0/1678359224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74dc06b440 0x7f74dc06d900 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.005+0000 7f7501033640 1 --2- 172.21.15.1:0/1678359224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74fc14b020 0x7f74fc1ee370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.005+0000 7f7501033640 1 --2- 172.21.15.1:0/1678359224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc152650 0x7f74fc14aae0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.005+0000 7f7501033640 1 --2- 172.21.15.1:0/1678359224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74fc0bf9d0 0x7f74fc14e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.005+0000 7f7501033640 1 -- 172.21.15.1:0/1678359224 >> 172.21.15.1:0/1678359224 conn(0x7f74fc066ee0 msgr2=0x7f74fc155ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:19.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.005+0000 7f7501033640 1 -- 172.21.15.1:0/1678359224 shutdown_connections 2024-03-06T07:20:19.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.005+0000 7f7501033640 1 -- 172.21.15.1:0/1678359224 wait complete. 2024-03-06T07:20:19.018 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:20:19.018 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:20:19.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.120+0000 7f1e9b0c3640 1 Processor -- start 2024-03-06T07:20:19.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.120+0000 7f1e9b0c3640 1 -- start start 2024-03-06T07:20:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.121+0000 7f1e9b0c3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1e940c3400 0x7f1e94154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.121+0000 7f1e9b0c3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e941571c0 con 0x7f1e940c79c0 2024-03-06T07:20:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.121+0000 7f1e9b0c3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e94153130 con 0x7f1e94150750 2024-03-06T07:20:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.121+0000 7f1e9b0c3640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e94154040 con 0x7f1e940c3400 2024-03-06T07:20:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.121+0000 7f1e99660640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1e940c79c0 0x7f1e94070d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56286/0 (socket says 172.21.15.1:56286) 2024-03-06T07:20:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.121+0000 7f1e99660640 1 -- 172.21.15.1:0/254379594 learned_addr learned my addr 172.21.15.1:0/254379594 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e937fe640 1 -- 172.21.15.1:0/254379594 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966340070 0 0) 0x7f1e941571c0 con 0x7f1e940c79c0 2024-03-06T07:20:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e937fe640 1 -- 172.21.15.1:0/254379594 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e84003520 con 0x7f1e940c79c0 2024-03-06T07:20:19.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e93fff640 1 --2- 172.21.15.1:0/254379594 >> v2:172.21.15.98:3300/0 conn(0x7f1e940c3400 0x7f1e94154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:19.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e937fe640 1 -- 172.21.15.1:0/254379594 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3892407302 0 0) 0x7f1e84003520 con 0x7f1e940c79c0 2024-03-06T07:20:19.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e937fe640 1 -- 172.21.15.1:0/254379594 >> v2:172.21.15.98:3300/0 conn(0x7f1e940c3400 msgr2=0x7f1e94154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:19.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e937fe640 1 --2- 172.21.15.1:0/254379594 >> v2:172.21.15.98:3300/0 conn(0x7f1e940c3400 0x7f1e94154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e937fe640 1 -- 172.21.15.1:0/254379594 >> v1:172.21.15.98:6789/0 conn(0x7f1e94150750 legacy=0x7f1e94153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:19.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e937fe640 1 -- 172.21.15.1:0/254379594 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e94153410 con 0x7f1e940c79c0 2024-03-06T07:20:19.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e937fe640 1 -- 172.21.15.1:0/254379594 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f1e8c0019f0 con 0x7f1e940c79c0 2024-03-06T07:20:19.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e937fe640 1 -- 172.21.15.1:0/254379594 >> v1:172.21.15.1:6789/0 conn(0x7f1e940c79c0 legacy=0x7f1e94070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:19.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e937fe640 1 --2- 172.21.15.1:0/254379594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e840039b0 0x7f1e84003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:19.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e937fe640 1 --2- 172.21.15.1:0/254379594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e84004800 0x7f1e84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:19.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e937fe640 1 --2- 172.21.15.1:0/254379594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e840051c0 0x7f1e8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:19.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e937fe640 1 -- 172.21.15.1:0/254379594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e94154040 con 0x7f1e84004800 2024-03-06T07:20:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e937fe640 1 -- 172.21.15.1:0/254379594 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8400c060 con 0x7f1e840039b0 2024-03-06T07:20:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e937fe640 1 -- 172.21.15.1:0/254379594 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8400c260 con 0x7f1e840051c0 2024-03-06T07:20:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e93fff640 1 --2- 172.21.15.1:0/254379594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e84004800 0x7f1e84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e98e5f640 1 --2- 172.21.15.1:0/254379594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e840039b0 0x7f1e84003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e99660640 1 --2- 172.21.15.1:0/254379594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e840051c0 0x7f1e8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e93fff640 1 -- 172.21.15.1:0/254379594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e840051c0 msgr2=0x7f1e8400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e93fff640 1 --2- 172.21.15.1:0/254379594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e840051c0 0x7f1e8400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e93fff640 1 -- 172.21.15.1:0/254379594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e840039b0 msgr2=0x7f1e84003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e93fff640 1 --2- 172.21.15.1:0/254379594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e840039b0 0x7f1e84003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e93fff640 1 -- 172.21.15.1:0/254379594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e84003520 con 0x7f1e84004800 2024-03-06T07:20:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.122+0000 7f1e99660640 1 -- 172.21.15.1:0/254379594 reap_dead start 2024-03-06T07:20:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.123+0000 7f1e93fff640 1 --2- 172.21.15.1:0/254379594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e84004800 0x7f1e84004c80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1e940c8990 tx=0x7f1e84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678fdeab8e89d28c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.123+0000 7f1e937fe640 1 -- 172.21.15.1:0/254379594 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e80039070 con 0x7f1e84004800 2024-03-06T07:20:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.123+0000 7f1e937fe640 1 -- 172.21.15.1:0/254379594 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8002c660 con 0x7f1e84004800 2024-03-06T07:20:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.123+0000 7f1e937fe640 1 -- 172.21.15.1:0/254379594 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e80009690 con 0x7f1e84004800 2024-03-06T07:20:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.124+0000 7f1e9b0c3640 1 -- 172.21.15.1:0/254379594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e84004800 msgr2=0x7f1e84004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.124+0000 7f1e9b0c3640 1 --2- 172.21.15.1:0/254379594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e84004800 0x7f1e84004c80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1e940c8990 tx=0x7f1e84005830 comp rx=0 tx=0).stop 2024-03-06T07:20:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.124+0000 7f1e9b0c3640 1 -- 172.21.15.1:0/254379594 shutdown_connections 2024-03-06T07:20:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.124+0000 7f1e9b0c3640 1 --2- 172.21.15.1:0/254379594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e84004800 0x7f1e84004c80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.124+0000 7f1e9b0c3640 1 -- 172.21.15.1:0/254379594 >> 172.21.15.1:0/254379594 conn(0x7f1e9406b620 msgr2=0x7f1e940c8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.124+0000 7f1e9b0c3640 1 -- 172.21.15.1:0/254379594 shutdown_connections 2024-03-06T07:20:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.124+0000 7f1e9b0c3640 1 -- 172.21.15.1:0/254379594 wait complete. 2024-03-06T07:20:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.125+0000 7f1e9b0c3640 1 Processor -- start 2024-03-06T07:20:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.125+0000 7f1e9b0c3640 1 -- start start 2024-03-06T07:20:19.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e9b0c3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e84004800 0x7f1e940ac880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:19.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e9b0c3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e9414d280 0x7f1e940c20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:19.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e9b0c3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e9414c9f0 0x7f1e9414b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:19.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e9b0c3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e940c7e40 con 0x7f1e9414c9f0 2024-03-06T07:20:19.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e9b0c3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e941574a0 con 0x7f1e9414d280 2024-03-06T07:20:19.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e9b0c3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e940c7fc0 con 0x7f1e84004800 2024-03-06T07:20:19.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e99660640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e9414c9f0 0x7f1e9414b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:19.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e99660640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e9414c9f0 0x7f1e9414b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35820/0 (socket says 172.21.15.1:35820) 2024-03-06T07:20:19.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e99660640 1 -- 172.21.15.1:0/1620285176 learned_addr learned my addr 172.21.15.1:0/1620285176 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:19.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e98e5f640 1 --2- 172.21.15.1:0/1620285176 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e84004800 0x7f1e940ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:19.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e93fff640 1 --2- 172.21.15.1:0/1620285176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e9414d280 0x7f1e940c20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e99660640 1 -- 172.21.15.1:0/1620285176 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e84004800 msgr2=0x7f1e940ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e99660640 1 --2- 172.21.15.1:0/1620285176 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e84004800 0x7f1e940ac880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e99660640 1 -- 172.21.15.1:0/1620285176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e9414d280 msgr2=0x7f1e940c20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e99660640 1 --2- 172.21.15.1:0/1620285176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e9414d280 0x7f1e940c20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e99660640 1 -- 172.21.15.1:0/1620285176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9406e0f0 con 0x7f1e9414c9f0 2024-03-06T07:20:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e99660640 1 --2- 172.21.15.1:0/1620285176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e9414c9f0 0x7f1e9414b240 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1e940c6160 tx=0x7f1e8c007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e917fa640 1 -- 172.21.15.1:0/1620285176 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c017020 con 0x7f1e9414c9f0 2024-03-06T07:20:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e917fa640 1 -- 172.21.15.1:0/1620285176 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8c00aac0 con 0x7f1e9414c9f0 2024-03-06T07:20:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e9b0c3640 1 -- 172.21.15.1:0/1620285176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e9414dd60 con 0x7f1e9414c9f0 2024-03-06T07:20:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e917fa640 1 -- 172.21.15.1:0/1620285176 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c0133f0 con 0x7f1e9414c9f0 2024-03-06T07:20:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.126+0000 7f1e9b0c3640 1 -- 172.21.15.1:0/1620285176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e9414c1f0 con 0x7f1e9414c9f0 2024-03-06T07:20:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.127+0000 7f1e9b0c3640 1 -- 172.21.15.1:0/1620285176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e940686e0 con 0x7f1e9414c9f0 2024-03-06T07:20:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.128+0000 7f1e917fa640 1 -- 172.21.15.1:0/1620285176 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e8c01a020 con 0x7f1e9414c9f0 2024-03-06T07:20:19.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.128+0000 7f1e917fa640 1 --2- 172.21.15.1:0/1620285176 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e7806b500 0x7f1e7806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:19.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.128+0000 7f1e917fa640 1 -- 172.21.15.1:0/1620285176 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1e8c08a470 con 0x7f1e9414c9f0 2024-03-06T07:20:19.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.128+0000 7f1e98e5f640 1 --2- 172.21.15.1:0/1620285176 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e7806b500 0x7f1e7806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:19.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.129+0000 7f1e98e5f640 1 --2- 172.21.15.1:0/1620285176 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e7806b500 0x7f1e7806d9c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1e88000c00 tx=0x7f1e88009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:19.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.133+0000 7f1e917fa640 1 -- 172.21.15.1:0/1620285176 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e8c021020 con 0x7f1e9414c9f0 2024-03-06T07:20:19.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.318+0000 7f1e9b0c3640 1 -- 172.21.15.1:0/1620285176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1e9414fca0 con 0x7f1e9414c9f0 2024-03-06T07:20:19.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.828+0000 7f1e917fa640 1 -- 172.21.15.1:0/1620285176 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1e8c0596d0 con 0x7f1e9414c9f0 2024-03-06T07:20:19.830 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-06T07:20:19.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.831+0000 7f1e9b0c3640 1 -- 172.21.15.1:0/1620285176 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e7806b500 msgr2=0x7f1e7806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:19.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.831+0000 7f1e9b0c3640 1 --2- 172.21.15.1:0/1620285176 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e7806b500 0x7f1e7806d9c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1e88000c00 tx=0x7f1e88009040 comp rx=0 tx=0).stop 2024-03-06T07:20:19.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.831+0000 7f1e9b0c3640 1 -- 172.21.15.1:0/1620285176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e9414c9f0 msgr2=0x7f1e9414b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:19.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.831+0000 7f1e9b0c3640 1 --2- 172.21.15.1:0/1620285176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e9414c9f0 0x7f1e9414b240 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1e940c6160 tx=0x7f1e8c007a90 comp rx=0 tx=0).stop 2024-03-06T07:20:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.831+0000 7f1e9b0c3640 1 -- 172.21.15.1:0/1620285176 shutdown_connections 2024-03-06T07:20:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.831+0000 7f1e9b0c3640 1 --2- 172.21.15.1:0/1620285176 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e7806b500 0x7f1e7806d9c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.831+0000 7f1e9b0c3640 1 --2- 172.21.15.1:0/1620285176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e9414c9f0 0x7f1e9414b240 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.831+0000 7f1e9b0c3640 1 --2- 172.21.15.1:0/1620285176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e9414d280 0x7f1e940c20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.831+0000 7f1e9b0c3640 1 --2- 172.21.15.1:0/1620285176 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e84004800 0x7f1e940ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.831+0000 7f1e9b0c3640 1 -- 172.21.15.1:0/1620285176 >> 172.21.15.1:0/1620285176 conn(0x7f1e9406b620 msgr2=0x7f1e940c6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.831+0000 7f1e9b0c3640 1 -- 172.21.15.1:0/1620285176 shutdown_connections 2024-03-06T07:20:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.832+0000 7f1e9b0c3640 1 -- 172.21.15.1:0/1620285176 wait complete. 2024-03-06T07:20:19.849 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-06T07:20:19.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.957+0000 7f6f7d2ed640 1 Processor -- start 2024-03-06T07:20:19.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.957+0000 7f6f7d2ed640 1 -- start start 2024-03-06T07:20:19.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.957+0000 7f6f7d2ed640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6f78156cd0 0x7f6f78157120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:19.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.957+0000 7f6f7d2ed640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f780c3300 con 0x7f6f78152690 2024-03-06T07:20:19.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.957+0000 7f6f7d2ed640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f780c35a0 con 0x7f6f780c6730 2024-03-06T07:20:19.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.957+0000 7f6f7d2ed640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f780c3880 con 0x7f6f78156cd0 2024-03-06T07:20:19.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.958+0000 7f6f76ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6f78152690 0x7f6f78155ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56290/0 (socket says 172.21.15.1:56290) 2024-03-06T07:20:19.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.958+0000 7f6f76ffd640 1 -- 172.21.15.1:0/4162674082 learned_addr learned my addr 172.21.15.1:0/4162674082 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:19.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.958+0000 7f6f777fe640 1 --2- 172.21.15.1:0/4162674082 >> v2:172.21.15.1:3300/0 conn(0x7f6f78156cd0 0x7f6f78157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:19.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.958+0000 7f6f777fe640 1 -- 172.21.15.1:0/4162674082 >> v1:172.21.15.98:6789/0 conn(0x7f6f780c6730 legacy=0x7f6f780c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:19.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.958+0000 7f6f777fe640 1 -- 172.21.15.1:0/4162674082 >> v1:172.21.15.1:6789/0 conn(0x7f6f78152690 legacy=0x7f6f78155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:19.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.958+0000 7f6f777fe640 1 -- 172.21.15.1:0/4162674082 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f780c3b60 con 0x7f6f78156cd0 2024-03-06T07:20:19.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.959+0000 7f6f777fe640 1 --2- 172.21.15.1:0/4162674082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f78156cd0 0x7f6f78157120 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009d90 tx=0x7f6f7002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0aea5cfb16bbeb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:19.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.959+0000 7f6f75ffb640 1 -- 172.21.15.1:0/4162674082 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f70039070 con 0x7f6f78156cd0 2024-03-06T07:20:19.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.959+0000 7f6f75ffb640 1 -- 172.21.15.1:0/4162674082 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7002cbd0 con 0x7f6f78156cd0 2024-03-06T07:20:19.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.959+0000 7f6f75ffb640 1 -- 172.21.15.1:0/4162674082 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f70034440 con 0x7f6f78156cd0 2024-03-06T07:20:19.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.959+0000 7f6f7d2ed640 1 -- 172.21.15.1:0/4162674082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f78156cd0 msgr2=0x7f6f78157120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:19.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.959+0000 7f6f7d2ed640 1 --2- 172.21.15.1:0/4162674082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f78156cd0 0x7f6f78157120 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009d90 tx=0x7f6f7002be20 comp rx=0 tx=0).stop 2024-03-06T07:20:19.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.959+0000 7f6f7d2ed640 1 -- 172.21.15.1:0/4162674082 shutdown_connections 2024-03-06T07:20:19.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.959+0000 7f6f7d2ed640 1 --2- 172.21.15.1:0/4162674082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f78156cd0 0x7f6f78157120 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.959+0000 7f6f7d2ed640 1 -- 172.21.15.1:0/4162674082 >> 172.21.15.1:0/4162674082 conn(0x7f6f7806b410 msgr2=0x7f6f7806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:19.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.959+0000 7f6f7d2ed640 1 -- 172.21.15.1:0/4162674082 shutdown_connections 2024-03-06T07:20:19.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.959+0000 7f6f7d2ed640 1 -- 172.21.15.1:0/4162674082 wait complete. 2024-03-06T07:20:19.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.960+0000 7f6f7d2ed640 1 Processor -- start 2024-03-06T07:20:19.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.961+0000 7f6f7d2ed640 1 -- start start 2024-03-06T07:20:19.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.961+0000 7f6f7d2ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f780c6730 0x7f6f78153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:19.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.961+0000 7f6f7d2ed640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f78152690 0x7f6f7814b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:19.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.961+0000 7f6f7d2ed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f78156cd0 0x7f6f7814bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:19.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.961+0000 7f6f7d2ed640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f780c6060 con 0x7f6f78152690 2024-03-06T07:20:19.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.961+0000 7f6f7d2ed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f780c5ee0 con 0x7f6f780c6730 2024-03-06T07:20:19.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.961+0000 7f6f7d2ed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f78152a70 con 0x7f6f78156cd0 2024-03-06T07:20:19.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.961+0000 7f6f767fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f78152690 0x7f6f7814b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:19.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.961+0000 7f6f767fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f78152690 0x7f6f7814b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35830/0 (socket says 172.21.15.1:35830) 2024-03-06T07:20:19.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.961+0000 7f6f767fc640 1 -- 172.21.15.1:0/678767737 learned_addr learned my addr 172.21.15.1:0/678767737 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:19.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.962+0000 7f6f777fe640 1 --2- 172.21.15.1:0/678767737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f78156cd0 0x7f6f7814bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:19.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.962+0000 7f6f76ffd640 1 --2- 172.21.15.1:0/678767737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f780c6730 0x7f6f78153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:19.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.962+0000 7f6f767fc640 1 -- 172.21.15.1:0/678767737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f78156cd0 msgr2=0x7f6f7814bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:19.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.962+0000 7f6f767fc640 1 --2- 172.21.15.1:0/678767737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f78156cd0 0x7f6f7814bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.962+0000 7f6f767fc640 1 -- 172.21.15.1:0/678767737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f780c6730 msgr2=0x7f6f78153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:19.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.962+0000 7f6f767fc640 1 --2- 172.21.15.1:0/678767737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f780c6730 0x7f6f78153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:19.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.962+0000 7f6f767fc640 1 -- 172.21.15.1:0/678767737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7814c080 con 0x7f6f78152690 2024-03-06T07:20:19.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.962+0000 7f6f767fc640 1 --2- 172.21.15.1:0/678767737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f78152690 0x7f6f7814b600 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00b420 tx=0x7f6f6c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:19.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.962+0000 7f6f5f7fe640 1 -- 172.21.15.1:0/678767737 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c011830 con 0x7f6f78152690 2024-03-06T07:20:19.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.962+0000 7f6f5f7fe640 1 -- 172.21.15.1:0/678767737 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f6c016070 con 0x7f6f78152690 2024-03-06T07:20:19.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.962+0000 7f6f7d2ed640 1 -- 172.21.15.1:0/678767737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f780ab7a0 con 0x7f6f78152690 2024-03-06T07:20:19.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.962+0000 7f6f5f7fe640 1 -- 172.21.15.1:0/678767737 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c011c10 con 0x7f6f78152690 2024-03-06T07:20:19.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.962+0000 7f6f7d2ed640 1 -- 172.21.15.1:0/678767737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f780aca30 con 0x7f6f78152690 2024-03-06T07:20:19.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.963+0000 7f6f7d2ed640 1 -- 172.21.15.1:0/678767737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f780ac540 con 0x7f6f78152690 2024-03-06T07:20:19.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.964+0000 7f6f5f7fe640 1 -- 172.21.15.1:0/678767737 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f6c01b460 con 0x7f6f78152690 2024-03-06T07:20:19.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.964+0000 7f6f5f7fe640 1 --2- 172.21.15.1:0/678767737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f6406b4d0 0x7f6f6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:19.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.964+0000 7f6f5f7fe640 1 -- 172.21.15.1:0/678767737 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f6f6c089b40 con 0x7f6f78152690 2024-03-06T07:20:19.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.964+0000 7f6f76ffd640 1 --2- 172.21.15.1:0/678767737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f6406b4d0 0x7f6f6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:19.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.965+0000 7f6f76ffd640 1 --2- 172.21.15.1:0/678767737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f6406b4d0 0x7f6f6406d990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6f68000f60 tx=0x7f6f68009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:19.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:19.968+0000 7f6f5f7fe640 1 -- 172.21.15.1:0/678767737 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f6c020020 con 0x7f6f78152690 2024-03-06T07:20:20.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.153+0000 7f6f7d2ed640 1 -- 172.21.15.1:0/678767737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f6f78067fd0 con 0x7f6f78152690 2024-03-06T07:20:20.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.833+0000 7f6f5f7fe640 1 -- 172.21.15.1:0/678767737 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) v1 ==== 171+0+0 (secure 0 0 0) 0x7f6f6c058ba0 con 0x7f6f78152690 2024-03-06T07:20:20.835 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-06T07:20:20.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.837+0000 7f6f7d2ed640 1 -- 172.21.15.1:0/678767737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f6406b4d0 msgr2=0x7f6f6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:20.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.837+0000 7f6f7d2ed640 1 --2- 172.21.15.1:0/678767737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f6406b4d0 0x7f6f6406d990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6f68000f60 tx=0x7f6f68009040 comp rx=0 tx=0).stop 2024-03-06T07:20:20.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.837+0000 7f6f7d2ed640 1 -- 172.21.15.1:0/678767737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f78152690 msgr2=0x7f6f7814b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:20.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.837+0000 7f6f7d2ed640 1 --2- 172.21.15.1:0/678767737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f78152690 0x7f6f7814b600 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00b420 tx=0x7f6f6c00b8e0 comp rx=0 tx=0).stop 2024-03-06T07:20:20.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.837+0000 7f6f7d2ed640 1 -- 172.21.15.1:0/678767737 shutdown_connections 2024-03-06T07:20:20.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.837+0000 7f6f7d2ed640 1 --2- 172.21.15.1:0/678767737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f6406b4d0 0x7f6f6406d990 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:20.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.837+0000 7f6f7d2ed640 1 --2- 172.21.15.1:0/678767737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f78156cd0 0x7f6f7814bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:20.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.837+0000 7f6f7d2ed640 1 --2- 172.21.15.1:0/678767737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f78152690 0x7f6f7814b600 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:20.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.837+0000 7f6f7d2ed640 1 --2- 172.21.15.1:0/678767737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f780c6730 0x7f6f78153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:20.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.837+0000 7f6f7d2ed640 1 -- 172.21.15.1:0/678767737 >> 172.21.15.1:0/678767737 conn(0x7f6f7806b410 msgr2=0x7f6f78070600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:20.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.838+0000 7f6f7d2ed640 1 -- 172.21.15.1:0/678767737 shutdown_connections 2024-03-06T07:20:20.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.838+0000 7f6f7d2ed640 1 -- 172.21.15.1:0/678767737 wait complete. 2024-03-06T07:20:20.867 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:20:20.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.975+0000 7f2b2480e640 1 Processor -- start 2024-03-06T07:20:20.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.975+0000 7f2b2480e640 1 -- start start 2024-03-06T07:20:20.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.976+0000 7f2b2480e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2b200c5520 0x7f2b201550d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.976+0000 7f2b2480e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2b200c84c0 0x7f2b20158d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.976+0000 7f2b2480e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b200c7650 con 0x7f2b20159240 2024-03-06T07:20:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.976+0000 7f2b2480e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b200c78f0 con 0x7f2b200c5520 2024-03-06T07:20:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.976+0000 7f2b2480e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b201587e0 con 0x7f2b200c84c0 2024-03-06T07:20:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.976+0000 7f2b1e59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2b200c5520 0x7f2b201550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.976+0000 7f2b1ed9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2b20159240 0x7f2b2006da30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56294/0 (socket says 172.21.15.1:56294) 2024-03-06T07:20:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.976+0000 7f2b1ed9d640 1 -- 172.21.15.1:0/3924181587 learned_addr learned my addr 172.21.15.1:0/3924181587 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.976+0000 7f2b1dd9b640 1 --2- 172.21.15.1:0/3924181587 >> v2:172.21.15.98:3300/0 conn(0x7f2b200c84c0 0x7f2b20158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.977+0000 7f2b1e59c640 1 -- 172.21.15.1:0/3924181587 >> v2:172.21.15.98:3300/0 conn(0x7f2b200c84c0 msgr2=0x7f2b20158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.977+0000 7f2b1e59c640 1 --2- 172.21.15.1:0/3924181587 >> v2:172.21.15.98:3300/0 conn(0x7f2b200c84c0 0x7f2b20158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.977+0000 7f2b1e59c640 1 -- 172.21.15.1:0/3924181587 >> v1:172.21.15.1:6789/0 conn(0x7f2b20159240 legacy=0x7f2b2006da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.977+0000 7f2b1e59c640 1 -- 172.21.15.1:0/3924181587 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b200c3ce0 con 0x7f2b200c5520 2024-03-06T07:20:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.977+0000 7f2b1e59c640 1 --2- 172.21.15.1:0/3924181587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b200c5520 0x7f2b201550d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2b2006be70 tx=0x7f2b1802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22eade70c7b88522 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.977+0000 7f2b1d59a640 1 -- 172.21.15.1:0/3924181587 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b180315a0 con 0x7f2b200c5520 2024-03-06T07:20:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.977+0000 7f2b1d59a640 1 -- 172.21.15.1:0/3924181587 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b18037070 con 0x7f2b200c5520 2024-03-06T07:20:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.977+0000 7f2b1d59a640 1 -- 172.21.15.1:0/3924181587 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b18031960 con 0x7f2b200c5520 2024-03-06T07:20:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.978+0000 7f2b2480e640 1 -- 172.21.15.1:0/3924181587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b200c5520 msgr2=0x7f2b201550d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.978+0000 7f2b2480e640 1 --2- 172.21.15.1:0/3924181587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b200c5520 0x7f2b201550d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2b2006be70 tx=0x7f2b1802b840 comp rx=0 tx=0).stop 2024-03-06T07:20:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.978+0000 7f2b2480e640 1 -- 172.21.15.1:0/3924181587 shutdown_connections 2024-03-06T07:20:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.978+0000 7f2b2480e640 1 --2- 172.21.15.1:0/3924181587 >> v2:172.21.15.98:3300/0 conn(0x7f2b200c84c0 0x7f2b20158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.978+0000 7f2b2480e640 1 --2- 172.21.15.1:0/3924181587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b200c5520 0x7f2b201550d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.978+0000 7f2b2480e640 1 -- 172.21.15.1:0/3924181587 >> 172.21.15.1:0/3924181587 conn(0x7f2b200c88f0 msgr2=0x7f2b200c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.978+0000 7f2b2480e640 1 -- 172.21.15.1:0/3924181587 shutdown_connections 2024-03-06T07:20:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.978+0000 7f2b2480e640 1 -- 172.21.15.1:0/3924181587 wait complete. 2024-03-06T07:20:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.979+0000 7f2b2480e640 1 Processor -- start 2024-03-06T07:20:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.979+0000 7f2b2480e640 1 -- start start 2024-03-06T07:20:20.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.980+0000 7f2b2480e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b200c5520 0x7f2b201ee950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:20.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.980+0000 7f2b2480e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b200c84c0 0x7f2b201eaa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:20.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.980+0000 7f2b2480e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b20159240 0x7f2b2014ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:20.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.980+0000 7f2b2480e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2006e2c0 con 0x7f2b200c5520 2024-03-06T07:20:20.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.980+0000 7f2b2480e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2006e140 con 0x7f2b20159240 2024-03-06T07:20:20.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.980+0000 7f2b2480e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b200c5900 con 0x7f2b200c84c0 2024-03-06T07:20:20.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.980+0000 7f2b1e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b200c5520 0x7f2b201ee950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:20.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.980+0000 7f2b1e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b200c5520 0x7f2b201ee950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35854/0 (socket says 172.21.15.1:35854) 2024-03-06T07:20:20.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.980+0000 7f2b1e59c640 1 -- 172.21.15.1:0/756148827 learned_addr learned my addr 172.21.15.1:0/756148827 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:20.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.980+0000 7f2b1ed9d640 1 --2- 172.21.15.1:0/756148827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b20159240 0x7f2b2014ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.980+0000 7f2b1dd9b640 1 --2- 172.21.15.1:0/756148827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b200c84c0 0x7f2b201eaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.980+0000 7f2b1e59c640 1 -- 172.21.15.1:0/756148827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b200c84c0 msgr2=0x7f2b201eaa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.980+0000 7f2b1e59c640 1 --2- 172.21.15.1:0/756148827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b200c84c0 0x7f2b201eaa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.980+0000 7f2b1e59c640 1 -- 172.21.15.1:0/756148827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b20159240 msgr2=0x7f2b2014ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.980+0000 7f2b1e59c640 1 --2- 172.21.15.1:0/756148827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b20159240 0x7f2b2014ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.980+0000 7f2b1e59c640 1 -- 172.21.15.1:0/756148827 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b200ac220 con 0x7f2b200c5520 2024-03-06T07:20:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.981+0000 7f2b1e59c640 1 --2- 172.21.15.1:0/756148827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b200c5520 0x7f2b201ee950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2b18009890 tx=0x7f2b180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.981+0000 7f2b0effd640 1 -- 172.21.15.1:0/756148827 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b18032630 con 0x7f2b200c5520 2024-03-06T07:20:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.981+0000 7f2b0effd640 1 -- 172.21.15.1:0/756148827 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b180327d0 con 0x7f2b200c5520 2024-03-06T07:20:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.981+0000 7f2b2480e640 1 -- 172.21.15.1:0/756148827 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b201eee90 con 0x7f2b200c5520 2024-03-06T07:20:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.981+0000 7f2b0effd640 1 -- 172.21.15.1:0/756148827 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b18032ba0 con 0x7f2b200c5520 2024-03-06T07:20:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.981+0000 7f2b2480e640 1 -- 172.21.15.1:0/756148827 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b201ee150 con 0x7f2b200c5520 2024-03-06T07:20:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.982+0000 7f2b2480e640 1 -- 172.21.15.1:0/756148827 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b200adb70 con 0x7f2b200c5520 2024-03-06T07:20:20.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.982+0000 7f2b0effd640 1 -- 172.21.15.1:0/756148827 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b1804c020 con 0x7f2b200c5520 2024-03-06T07:20:20.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.983+0000 7f2b0effd640 1 --2- 172.21.15.1:0/756148827 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b0006b4d0 0x7f2b0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:20.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.983+0000 7f2b0effd640 1 -- 172.21.15.1:0/756148827 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f2b1802f020 con 0x7f2b200c5520 2024-03-06T07:20:20.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.983+0000 7f2b1dd9b640 1 --2- 172.21.15.1:0/756148827 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b0006b4d0 0x7f2b0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:20.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.983+0000 7f2b1dd9b640 1 --2- 172.21.15.1:0/756148827 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b0006b4d0 0x7f2b0006d990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b200c9590 tx=0x7f2b14008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:20.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:20.987+0000 7f2b0effd640 1 -- 172.21.15.1:0/756148827 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b18040070 con 0x7f2b200c5520 2024-03-06T07:20:21.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.206+0000 7f2b2480e640 1 -- 172.21.15.1:0/756148827 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2b200686e0 con 0x7f2b200c5520 2024-03-06T07:20:21.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.834+0000 7f2b0effd640 1 -- 172.21.15.1:0/756148827 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f2b1807a4e0 con 0x7f2b200c5520 2024-03-06T07:20:21.837 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 2 and data pool 3 2024-03-06T07:20:21.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.838+0000 7f2b2480e640 1 -- 172.21.15.1:0/756148827 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b0006b4d0 msgr2=0x7f2b0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:21.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.838+0000 7f2b2480e640 1 --2- 172.21.15.1:0/756148827 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b0006b4d0 0x7f2b0006d990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b200c9590 tx=0x7f2b14008040 comp rx=0 tx=0).stop 2024-03-06T07:20:21.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.838+0000 7f2b2480e640 1 -- 172.21.15.1:0/756148827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b200c5520 msgr2=0x7f2b201ee950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:21.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.838+0000 7f2b2480e640 1 --2- 172.21.15.1:0/756148827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b200c5520 0x7f2b201ee950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2b18009890 tx=0x7f2b180046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.839+0000 7f2b2480e640 1 -- 172.21.15.1:0/756148827 shutdown_connections 2024-03-06T07:20:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.839+0000 7f2b2480e640 1 --2- 172.21.15.1:0/756148827 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b0006b4d0 0x7f2b0006d990 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.839+0000 7f2b2480e640 1 --2- 172.21.15.1:0/756148827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b20159240 0x7f2b2014ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.839+0000 7f2b2480e640 1 --2- 172.21.15.1:0/756148827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b200c84c0 0x7f2b201eaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.839+0000 7f2b2480e640 1 --2- 172.21.15.1:0/756148827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b200c5520 0x7f2b201ee950 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.839+0000 7f2b2480e640 1 -- 172.21.15.1:0/756148827 >> 172.21.15.1:0/756148827 conn(0x7f2b200c88f0 msgr2=0x7f2b2006d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.839+0000 7f2b2480e640 1 -- 172.21.15.1:0/756148827 shutdown_connections 2024-03-06T07:20:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.840+0000 7f2b2480e640 1 -- 172.21.15.1:0/756148827 wait complete. 2024-03-06T07:20:21.873 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-06T07:20:21.874 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-06T07:20:21.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.982+0000 7f939042a640 1 Processor -- start 2024-03-06T07:20:21.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.982+0000 7f939042a640 1 -- start start 2024-03-06T07:20:21.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.983+0000 7f939042a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93880c1480 0x7f9388152360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:21.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.983+0000 7f939042a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93880c6b00 0x7f93881528a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:21.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.983+0000 7f939042a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93881554d0 con 0x7f93880c7f00 2024-03-06T07:20:21.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.983+0000 7f939042a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93881557b0 con 0x7f93880c1480 2024-03-06T07:20:21.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.983+0000 7f939042a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9388155a90 con 0x7f93880c6b00 2024-03-06T07:20:21.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.983+0000 7f938d9c5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93880c6b00 0x7f93881528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:21.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.983+0000 7f938e1c6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93880c1480 0x7f9388152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:21.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.983+0000 7f938e1c6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93880c1480 0x7f9388152360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:49188/0 (socket says 172.21.15.1:49188) 2024-03-06T07:20:21.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.983+0000 7f938e1c6640 1 -- 172.21.15.1:0/3420208573 learned_addr learned my addr 172.21.15.1:0/3420208573 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:21.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.983+0000 7f938e9c7640 1 --1- 172.21.15.1:0/3420208573 >> v1:172.21.15.98:6789/0 conn(0x7f93880c7f00 0x7f93880bfff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:35538/0 (socket says 172.21.15.1:35538) 2024-03-06T07:20:21.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.983+0000 7f938e1c6640 1 -- 172.21.15.1:0/3420208573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93880c6b00 msgr2=0x7f93881528a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:21.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.983+0000 7f938e1c6640 1 --2- 172.21.15.1:0/3420208573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93880c6b00 0x7f93881528a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:21.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.984+0000 7f938e1c6640 1 -- 172.21.15.1:0/3420208573 >> v1:172.21.15.98:6789/0 conn(0x7f93880c7f00 legacy=0x7f93880bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:21.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.984+0000 7f938d1c4640 1 -- 172.21.15.1:0/3420208573 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405237056 0 0) 0x7f93881554d0 con 0x7f93880c7f00 2024-03-06T07:20:21.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.984+0000 7f938e1c6640 1 -- 172.21.15.1:0/3420208573 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9388155d70 con 0x7f93880c1480 2024-03-06T07:20:21.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.984+0000 7f938e1c6640 1 --2- 172.21.15.1:0/3420208573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93880c1480 0x7f9388152360 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9384009870 tx=0x7f938402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc220b3d0d9ae8ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:21.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.984+0000 7f938d1c4640 1 -- 172.21.15.1:0/3420208573 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9384035020 con 0x7f93880c1480 2024-03-06T07:20:21.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.985+0000 7f938d1c4640 1 -- 172.21.15.1:0/3420208573 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9384031640 con 0x7f93880c1480 2024-03-06T07:20:21.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.985+0000 7f938d1c4640 1 -- 172.21.15.1:0/3420208573 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938402d040 con 0x7f93880c1480 2024-03-06T07:20:21.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.985+0000 7f939042a640 1 -- 172.21.15.1:0/3420208573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93880c1480 msgr2=0x7f9388152360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:21.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.985+0000 7f939042a640 1 --2- 172.21.15.1:0/3420208573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93880c1480 0x7f9388152360 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9384009870 tx=0x7f938402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:21.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.985+0000 7f939042a640 1 -- 172.21.15.1:0/3420208573 shutdown_connections 2024-03-06T07:20:21.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.985+0000 7f939042a640 1 --2- 172.21.15.1:0/3420208573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93880c6b00 0x7f93881528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:21.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.985+0000 7f939042a640 1 --2- 172.21.15.1:0/3420208573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93880c1480 0x7f9388152360 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:21.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.985+0000 7f939042a640 1 -- 172.21.15.1:0/3420208573 >> 172.21.15.1:0/3420208573 conn(0x7f938808b610 msgr2=0x7f9388066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:21.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.985+0000 7f939042a640 1 -- 172.21.15.1:0/3420208573 shutdown_connections 2024-03-06T07:20:21.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.985+0000 7f939042a640 1 -- 172.21.15.1:0/3420208573 wait complete. 2024-03-06T07:20:21.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.986+0000 7f939042a640 1 Processor -- start 2024-03-06T07:20:21.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.986+0000 7f939042a640 1 -- start start 2024-03-06T07:20:21.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.987+0000 7f939042a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93880c1480 0x7f9388144f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:21.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.987+0000 7f939042a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93880c6b00 0x7f93881454d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:21.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.987+0000 7f939042a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c7f00 0x7f93880ae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:21.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.987+0000 7f939042a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93880c1860 con 0x7f93880c7f00 2024-03-06T07:20:21.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.987+0000 7f939042a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93880bfb90 con 0x7f93880c1480 2024-03-06T07:20:21.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.987+0000 7f939042a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93880c19e0 con 0x7f93880c6b00 2024-03-06T07:20:21.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.987+0000 7f938e9c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c7f00 0x7f93880ae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:21.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.987+0000 7f938e9c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c7f00 0x7f93880ae0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35864/0 (socket says 172.21.15.1:35864) 2024-03-06T07:20:21.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.987+0000 7f938e9c7640 1 -- 172.21.15.1:0/4187996609 learned_addr learned my addr 172.21.15.1:0/4187996609 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:21.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.987+0000 7f938e1c6640 1 --2- 172.21.15.1:0/4187996609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93880c1480 0x7f9388144f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:21.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.987+0000 7f938d9c5640 1 --2- 172.21.15.1:0/4187996609 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93880c6b00 0x7f93881454d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:21.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.987+0000 7f938e9c7640 1 -- 172.21.15.1:0/4187996609 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93880c6b00 msgr2=0x7f93881454d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:21.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.987+0000 7f938e9c7640 1 --2- 172.21.15.1:0/4187996609 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93880c6b00 0x7f93881454d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:21.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.987+0000 7f938e9c7640 1 -- 172.21.15.1:0/4187996609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93880c1480 msgr2=0x7f9388144f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:21.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.987+0000 7f938e9c7640 1 --2- 172.21.15.1:0/4187996609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93880c1480 0x7f9388144f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:21.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.987+0000 7f938e9c7640 1 -- 172.21.15.1:0/4187996609 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9388123cc0 con 0x7f93880c7f00 2024-03-06T07:20:21.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.988+0000 7f938e9c7640 1 --2- 172.21.15.1:0/4187996609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c7f00 0x7f93880ae0c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f938000c730 tx=0x7f938000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:21.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.988+0000 7f937effd640 1 -- 172.21.15.1:0/4187996609 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9380018070 con 0x7f93880c7f00 2024-03-06T07:20:21.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.988+0000 7f937effd640 1 -- 172.21.15.1:0/4187996609 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938000ce40 con 0x7f93880c7f00 2024-03-06T07:20:21.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.988+0000 7f937effd640 1 -- 172.21.15.1:0/4187996609 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938000a860 con 0x7f93880c7f00 2024-03-06T07:20:21.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.988+0000 7f939042a640 1 -- 172.21.15.1:0/4187996609 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93881eb870 con 0x7f93880c7f00 2024-03-06T07:20:21.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.988+0000 7f939042a640 1 -- 172.21.15.1:0/4187996609 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9388124260 con 0x7f93880c7f00 2024-03-06T07:20:21.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.989+0000 7f939042a640 1 -- 172.21.15.1:0/4187996609 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93880680b0 con 0x7f93880c7f00 2024-03-06T07:20:21.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.989+0000 7f937effd640 1 -- 172.21.15.1:0/4187996609 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f938000aa00 con 0x7f93880c7f00 2024-03-06T07:20:21.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.989+0000 7f937effd640 1 --2- 172.21.15.1:0/4187996609 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f936c06b500 0x7f936c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:21.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.990+0000 7f937effd640 1 -- 172.21.15.1:0/4187996609 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9380089660 con 0x7f93880c7f00 2024-03-06T07:20:21.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.990+0000 7f938e1c6640 1 --2- 172.21.15.1:0/4187996609 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f936c06b500 0x7f936c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:21.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.990+0000 7f938e1c6640 1 --2- 172.21.15.1:0/4187996609 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f936c06b500 0x7f936c06d9c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9384036000 tx=0x7f938403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:21.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:21.994+0000 7f937effd640 1 -- 172.21.15.1:0/4187996609 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9380057ea0 con 0x7f93880c7f00 2024-03-06T07:20:22.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.183+0000 7f939042a640 1 -- 172.21.15.1:0/4187996609 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f93880b06b0 con 0x7f93880c7f00 2024-03-06T07:20:22.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.836+0000 7f937effd640 1 -- 172.21.15.1:0/4187996609 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) v1 ==== 157+0+0 (secure 0 0 0) 0x7f938005bf40 con 0x7f93880c7f00 2024-03-06T07:20:22.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.841+0000 7f939042a640 1 -- 172.21.15.1:0/4187996609 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f936c06b500 msgr2=0x7f936c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:22.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.841+0000 7f939042a640 1 --2- 172.21.15.1:0/4187996609 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f936c06b500 0x7f936c06d9c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9384036000 tx=0x7f938403c040 comp rx=0 tx=0).stop 2024-03-06T07:20:22.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.841+0000 7f939042a640 1 -- 172.21.15.1:0/4187996609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c7f00 msgr2=0x7f93880ae0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:22.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.841+0000 7f939042a640 1 --2- 172.21.15.1:0/4187996609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c7f00 0x7f93880ae0c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f938000c730 tx=0x7f938000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.841+0000 7f939042a640 1 -- 172.21.15.1:0/4187996609 shutdown_connections 2024-03-06T07:20:22.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.841+0000 7f939042a640 1 --2- 172.21.15.1:0/4187996609 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f936c06b500 0x7f936c06d9c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.841+0000 7f939042a640 1 --2- 172.21.15.1:0/4187996609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c7f00 0x7f93880ae0c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.841+0000 7f939042a640 1 --2- 172.21.15.1:0/4187996609 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93880c6b00 0x7f93881454d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.841+0000 7f939042a640 1 --2- 172.21.15.1:0/4187996609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93880c1480 0x7f9388144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.841+0000 7f939042a640 1 -- 172.21.15.1:0/4187996609 >> 172.21.15.1:0/4187996609 conn(0x7f938808b610 msgr2=0x7f9388066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:22.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.841+0000 7f939042a640 1 -- 172.21.15.1:0/4187996609 shutdown_connections 2024-03-06T07:20:22.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.841+0000 7f939042a640 1 -- 172.21.15.1:0/4187996609 wait complete. 2024-03-06T07:20:22.857 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:20:22.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.968+0000 7f88748df640 1 Processor -- start 2024-03-06T07:20:22.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.968+0000 7f88748df640 1 -- start start 2024-03-06T07:20:22.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.969+0000 7f88748df640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88701539e0 0x7f8870153040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:22.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.969+0000 7f88748df640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f88700c6820 0x7f88700c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:22.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.969+0000 7f88748df640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88700c6d30 con 0x7f8870153dc0 2024-03-06T07:20:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.969+0000 7f88748df640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88700c7940 con 0x7f88701539e0 2024-03-06T07:20:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.969+0000 7f88748df640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88700c7010 con 0x7f88700c6820 2024-03-06T07:20:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.969+0000 7f886e59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88701539e0 0x7f8870153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.969+0000 7f886e59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88701539e0 0x7f8870153040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35874/0 (socket says 172.21.15.1:35874) 2024-03-06T07:20:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.969+0000 7f886e59c640 1 -- 172.21.15.1:0/580013439 learned_addr learned my addr 172.21.15.1:0/580013439 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.969+0000 7f886dd9b640 1 --1- 172.21.15.1:0/580013439 >> v1:172.21.15.1:6789/0 conn(0x7f8870153dc0 0x7f8870154530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56308/0 (socket says 172.21.15.1:56308) 2024-03-06T07:20:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.969+0000 7f886ed9d640 1 --2- 172.21.15.1:0/580013439 >> v2:172.21.15.98:3300/0 conn(0x7f88700c6820 0x7f88700c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.969+0000 7f886e59c640 1 -- 172.21.15.1:0/580013439 >> v2:172.21.15.98:3300/0 conn(0x7f88700c6820 msgr2=0x7f88700c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.969+0000 7f886e59c640 1 --2- 172.21.15.1:0/580013439 >> v2:172.21.15.98:3300/0 conn(0x7f88700c6820 0x7f88700c7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.969+0000 7f886e59c640 1 -- 172.21.15.1:0/580013439 >> v1:172.21.15.1:6789/0 conn(0x7f8870153dc0 legacy=0x7f8870154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:22.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.969+0000 7f886e59c640 1 -- 172.21.15.1:0/580013439 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88700c72f0 con 0x7f88701539e0 2024-03-06T07:20:22.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.969+0000 7f886ed9d640 1 --2- 172.21.15.1:0/580013439 >> v2:172.21.15.98:3300/0 conn(0x7f88700c6820 0x7f88700c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:22.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.970+0000 7f886e59c640 1 --2- 172.21.15.1:0/580013439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88701539e0 0x7f8870153040 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8870070930 tx=0x7f886802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7d5db445308dd2f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:22.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.970+0000 7f886d59a640 1 -- 172.21.15.1:0/580013439 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886802cb20 con 0x7f88701539e0 2024-03-06T07:20:22.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.970+0000 7f886d59a640 1 -- 172.21.15.1:0/580013439 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886802ccc0 con 0x7f88701539e0 2024-03-06T07:20:22.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.970+0000 7f886d59a640 1 -- 172.21.15.1:0/580013439 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868034610 con 0x7f88701539e0 2024-03-06T07:20:22.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.971+0000 7f88748df640 1 -- 172.21.15.1:0/580013439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88701539e0 msgr2=0x7f8870153040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:22.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.971+0000 7f88748df640 1 --2- 172.21.15.1:0/580013439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88701539e0 0x7f8870153040 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8870070930 tx=0x7f886802bf80 comp rx=0 tx=0).stop 2024-03-06T07:20:22.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.971+0000 7f88748df640 1 -- 172.21.15.1:0/580013439 shutdown_connections 2024-03-06T07:20:22.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.971+0000 7f88748df640 1 --2- 172.21.15.1:0/580013439 >> v2:172.21.15.98:3300/0 conn(0x7f88700c6820 0x7f88700c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.971+0000 7f88748df640 1 --2- 172.21.15.1:0/580013439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88701539e0 0x7f8870153040 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.971+0000 7f88748df640 1 -- 172.21.15.1:0/580013439 >> 172.21.15.1:0/580013439 conn(0x7f887006b620 msgr2=0x7f88700aa530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:22.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.971+0000 7f88748df640 1 -- 172.21.15.1:0/580013439 shutdown_connections 2024-03-06T07:20:22.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.971+0000 7f88748df640 1 -- 172.21.15.1:0/580013439 wait complete. 2024-03-06T07:20:22.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.972+0000 7f88748df640 1 Processor -- start 2024-03-06T07:20:22.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.972+0000 7f88748df640 1 -- start start 2024-03-06T07:20:22.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.973+0000 7f88748df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88700c6820 0x7f88700c1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:22.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.973+0000 7f88748df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88701539e0 0x7f88701eb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:22.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.973+0000 7f88748df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870153dc0 0x7f8870144b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:22.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.973+0000 7f88748df640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88700c2e20 con 0x7f88701539e0 2024-03-06T07:20:22.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.973+0000 7f88748df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88700c8310 con 0x7f8870153dc0 2024-03-06T07:20:22.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.973+0000 7f88748df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88700c2fa0 con 0x7f88700c6820 2024-03-06T07:20:22.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.973+0000 7f886dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88701539e0 0x7f88701eb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:22.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.973+0000 7f886dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88701539e0 0x7f88701eb160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35882/0 (socket says 172.21.15.1:35882) 2024-03-06T07:20:22.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.973+0000 7f886dd9b640 1 -- 172.21.15.1:0/3548923641 learned_addr learned my addr 172.21.15.1:0/3548923641 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:22.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.973+0000 7f886e59c640 1 --2- 172.21.15.1:0/3548923641 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88700c6820 0x7f88700c1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:22.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.973+0000 7f886ed9d640 1 --2- 172.21.15.1:0/3548923641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870153dc0 0x7f8870144b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:22.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.973+0000 7f886dd9b640 1 -- 172.21.15.1:0/3548923641 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88700c6820 msgr2=0x7f88700c1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:22.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.973+0000 7f886dd9b640 1 --2- 172.21.15.1:0/3548923641 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88700c6820 0x7f88700c1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.973+0000 7f886dd9b640 1 -- 172.21.15.1:0/3548923641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870153dc0 msgr2=0x7f8870144b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:22.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.973+0000 7f886dd9b640 1 --2- 172.21.15.1:0/3548923641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870153dc0 0x7f8870144b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.973+0000 7f886dd9b640 1 -- 172.21.15.1:0/3548923641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8870155970 con 0x7f88701539e0 2024-03-06T07:20:22.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.974+0000 7f886dd9b640 1 --2- 172.21.15.1:0/3548923641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88701539e0 0x7f88701eb160 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f885800c3f0 tx=0x7f885800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:22.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.974+0000 7f8866ffd640 1 -- 172.21.15.1:0/3548923641 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88580074e0 con 0x7f88701539e0 2024-03-06T07:20:22.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.974+0000 7f8866ffd640 1 -- 172.21.15.1:0/3548923641 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8858007680 con 0x7f88701539e0 2024-03-06T07:20:22.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.974+0000 7f88748df640 1 -- 172.21.15.1:0/3548923641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88701548b0 con 0x7f88701539e0 2024-03-06T07:20:22.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.974+0000 7f8866ffd640 1 -- 172.21.15.1:0/3548923641 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8858004a00 con 0x7f88701539e0 2024-03-06T07:20:22.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.974+0000 7f88748df640 1 -- 172.21.15.1:0/3548923641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88701545d0 con 0x7f88701539e0 2024-03-06T07:20:22.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.975+0000 7f88748df640 1 -- 172.21.15.1:0/3548923641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8870067fd0 con 0x7f88701539e0 2024-03-06T07:20:22.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.975+0000 7f8866ffd640 1 -- 172.21.15.1:0/3548923641 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f885801a020 con 0x7f88701539e0 2024-03-06T07:20:22.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.975+0000 7f8866ffd640 1 --2- 172.21.15.1:0/3548923641 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f885006b4d0 0x7f885006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:22.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.976+0000 7f8866ffd640 1 -- 172.21.15.1:0/3548923641 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6007+0+0 (secure 0 0 0) 0x7f8858089a20 con 0x7f88701539e0 2024-03-06T07:20:22.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.976+0000 7f886e59c640 1 --2- 172.21.15.1:0/3548923641 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f885006b4d0 0x7f885006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:22.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.976+0000 7f886e59c640 1 --2- 172.21.15.1:0/3548923641 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f885006b4d0 0x7f885006d990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8868035000 tx=0x7f8868030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:22.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:22.980+0000 7f8866ffd640 1 -- 172.21.15.1:0/3548923641 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8858010300 con 0x7f88701539e0 2024-03-06T07:20:23.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:23.251+0000 7f88748df640 1 -- 172.21.15.1:0/3548923641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f887014f1c0 con 0x7f88701539e0 2024-03-06T07:20:24.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:24.867+0000 7f8866ffd640 1 -- 172.21.15.1:0/3548923641 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) v1 ==== 228+0+0 (secure 0 0 0) 0x7f8858058800 con 0x7f88701539e0 2024-03-06T07:20:24.869 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data_ec' created 2024-03-06T07:20:24.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:24.870+0000 7f88748df640 1 -- 172.21.15.1:0/3548923641 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f885006b4d0 msgr2=0x7f885006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:24.870+0000 7f88748df640 1 --2- 172.21.15.1:0/3548923641 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f885006b4d0 0x7f885006d990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8868035000 tx=0x7f8868030000 comp rx=0 tx=0).stop 2024-03-06T07:20:24.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:24.870+0000 7f88748df640 1 -- 172.21.15.1:0/3548923641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88701539e0 msgr2=0x7f88701eb160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:24.870+0000 7f88748df640 1 --2- 172.21.15.1:0/3548923641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88701539e0 0x7f88701eb160 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f885800c3f0 tx=0x7f885800c8b0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:24.870+0000 7f88748df640 1 -- 172.21.15.1:0/3548923641 shutdown_connections 2024-03-06T07:20:24.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:24.870+0000 7f88748df640 1 --2- 172.21.15.1:0/3548923641 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f885006b4d0 0x7f885006d990 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:24.870+0000 7f88748df640 1 --2- 172.21.15.1:0/3548923641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8870153dc0 0x7f8870144b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:24.870+0000 7f88748df640 1 --2- 172.21.15.1:0/3548923641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88701539e0 0x7f88701eb160 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:24.870+0000 7f88748df640 1 --2- 172.21.15.1:0/3548923641 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88700c6820 0x7f88700c1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:24.870+0000 7f88748df640 1 -- 172.21.15.1:0/3548923641 >> 172.21.15.1:0/3548923641 conn(0x7f887006b620 msgr2=0x7f8870152280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:24.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:24.871+0000 7f88748df640 1 -- 172.21.15.1:0/3548923641 shutdown_connections 2024-03-06T07:20:24.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:24.871+0000 7f88748df640 1 -- 172.21.15.1:0/3548923641 wait complete. 2024-03-06T07:20:24.888 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-06T07:20:25.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.029+0000 7faed1281640 1 Processor -- start 2024-03-06T07:20:25.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.029+0000 7faed1281640 1 -- start start 2024-03-06T07:20:25.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.030+0000 7faed1281640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faecc0c5060 0x7faecc1552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.030+0000 7faed1281640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faecc0c5c60 0x7faecc155810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.030+0000 7faed1281640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc0c6a10 con 0x7faecc152a10 2024-03-06T07:20:25.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.030+0000 7faed1281640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faecc0c6cf0 con 0x7faecc0c5060 2024-03-06T07:20:25.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.030+0000 7faed1281640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faecc0c0990 con 0x7faecc0c5c60 2024-03-06T07:20:25.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.030+0000 7faecb7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faecc0c5c60 0x7faecc155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.030+0000 7faec3fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faecc0c5060 0x7faecc1552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.030+0000 7faecb7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faecc0c5c60 0x7faecc155810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:60188/0 (socket says 172.21.15.1:60188) 2024-03-06T07:20:25.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.030+0000 7faecb7fe640 1 -- 172.21.15.1:0/1609973074 learned_addr learned my addr 172.21.15.1:0/1609973074 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:25.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.031+0000 7faecb7fe640 1 -- 172.21.15.1:0/1609973074 >> v2:172.21.15.98:3300/0 conn(0x7faecc0c5060 msgr2=0x7faecc1552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:25.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.031+0000 7faecb7fe640 1 --2- 172.21.15.1:0/1609973074 >> v2:172.21.15.98:3300/0 conn(0x7faecc0c5060 0x7faecc1552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.031+0000 7faecb7fe640 1 -- 172.21.15.1:0/1609973074 >> v1:172.21.15.98:6789/0 conn(0x7faecc152a10 legacy=0x7faecc0cab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.031+0000 7faecb7fe640 1 -- 172.21.15.1:0/1609973074 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc0c0c70 con 0x7faecc0c5c60 2024-03-06T07:20:25.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.031+0000 7faec3fff640 1 --2- 172.21.15.1:0/1609973074 >> v2:172.21.15.98:3300/0 conn(0x7faecc0c5060 0x7faecc1552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:25.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.031+0000 7faecb7fe640 1 --2- 172.21.15.1:0/1609973074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faecc0c5c60 0x7faecc155810 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faec4009e50 tx=0x7faec402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cc76519ba61a1d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:25.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.032+0000 7faeca7fc640 1 -- 172.21.15.1:0/1609973074 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec402ca70 con 0x7faecc0c5c60 2024-03-06T07:20:25.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.032+0000 7faeca7fc640 1 -- 172.21.15.1:0/1609973074 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec402cc10 con 0x7faecc0c5c60 2024-03-06T07:20:25.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.032+0000 7faeca7fc640 1 -- 172.21.15.1:0/1609973074 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec40343f0 con 0x7faecc0c5c60 2024-03-06T07:20:25.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.032+0000 7faed1281640 1 -- 172.21.15.1:0/1609973074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faecc0c5c60 msgr2=0x7faecc155810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.032+0000 7faed1281640 1 --2- 172.21.15.1:0/1609973074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faecc0c5c60 0x7faecc155810 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faec4009e50 tx=0x7faec402bed0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.032+0000 7faed1281640 1 -- 172.21.15.1:0/1609973074 shutdown_connections 2024-03-06T07:20:25.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.032+0000 7faed1281640 1 --2- 172.21.15.1:0/1609973074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faecc0c5c60 0x7faecc155810 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.032+0000 7faed1281640 1 --2- 172.21.15.1:0/1609973074 >> v2:172.21.15.98:3300/0 conn(0x7faecc0c5060 0x7faecc1552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.032+0000 7faed1281640 1 -- 172.21.15.1:0/1609973074 >> 172.21.15.1:0/1609973074 conn(0x7faecc072d50 msgr2=0x7faecc06d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.032+0000 7faed1281640 1 -- 172.21.15.1:0/1609973074 shutdown_connections 2024-03-06T07:20:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.032+0000 7faed1281640 1 -- 172.21.15.1:0/1609973074 wait complete. 2024-03-06T07:20:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.034+0000 7faed1281640 1 Processor -- start 2024-03-06T07:20:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.034+0000 7faed1281640 1 -- start start 2024-03-06T07:20:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.035+0000 7faed1281640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faecc0c5060 0x7faecc14aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.035+0000 7faed1281640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faecc0c5c60 0x7faecc14afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.035+0000 7faed1281640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faecc152a10 0x7faecc1ee2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.035+0000 7faed1281640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc0cb3f0 con 0x7faecc0c5c60 2024-03-06T07:20:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.035+0000 7faed1281640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc0cb270 con 0x7faecc0c5060 2024-03-06T07:20:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.035+0000 7faed1281640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faecc0c6630 con 0x7faecc152a10 2024-03-06T07:20:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.035+0000 7faec3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faecc0c5c60 0x7faecc14afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.035+0000 7faec3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faecc0c5c60 0x7faecc14afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34696/0 (socket says 172.21.15.1:34696) 2024-03-06T07:20:25.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.035+0000 7faec3fff640 1 -- 172.21.15.1:0/2181544499 learned_addr learned my addr 172.21.15.1:0/2181544499 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:25.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.036+0000 7faecb7fe640 1 --2- 172.21.15.1:0/2181544499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faecc152a10 0x7faecc1ee2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.036+0000 7faecaffd640 1 --2- 172.21.15.1:0/2181544499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faecc0c5060 0x7faecc14aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.036+0000 7faec3fff640 1 -- 172.21.15.1:0/2181544499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faecc152a10 msgr2=0x7faecc1ee2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.036+0000 7faec3fff640 1 --2- 172.21.15.1:0/2181544499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faecc152a10 0x7faecc1ee2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.036+0000 7faec3fff640 1 -- 172.21.15.1:0/2181544499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faecc0c5060 msgr2=0x7faecc14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.036+0000 7faec3fff640 1 --2- 172.21.15.1:0/2181544499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faecc0c5060 0x7faecc14aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.036+0000 7faec3fff640 1 -- 172.21.15.1:0/2181544499 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc156210 con 0x7faecc0c5c60 2024-03-06T07:20:25.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.036+0000 7faec3fff640 1 --2- 172.21.15.1:0/2181544499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faecc0c5c60 0x7faecc14afd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7faeb400e8d0 tx=0x7faeb400ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:25.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.036+0000 7faec37fe640 1 -- 172.21.15.1:0/2181544499 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb4017070 con 0x7faecc0c5c60 2024-03-06T07:20:25.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.036+0000 7faec37fe640 1 -- 172.21.15.1:0/2181544499 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb400c940 con 0x7faecc0c5c60 2024-03-06T07:20:25.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.036+0000 7faed1281640 1 -- 172.21.15.1:0/2181544499 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faecc1568a0 con 0x7faecc0c5c60 2024-03-06T07:20:25.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.036+0000 7faec37fe640 1 -- 172.21.15.1:0/2181544499 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb400cd30 con 0x7faecc0c5c60 2024-03-06T07:20:25.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.036+0000 7faed1281640 1 -- 172.21.15.1:0/2181544499 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faecc157880 con 0x7faecc0c5c60 2024-03-06T07:20:25.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.037+0000 7faed1281640 1 -- 172.21.15.1:0/2181544499 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faecc0c3b80 con 0x7faecc0c5c60 2024-03-06T07:20:25.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.038+0000 7faec37fe640 1 -- 172.21.15.1:0/2181544499 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faeb401a070 con 0x7faecc0c5c60 2024-03-06T07:20:25.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.038+0000 7faec37fe640 1 --2- 172.21.15.1:0/2181544499 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faeac06b4d0 0x7faeac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.038+0000 7faec37fe640 1 -- 172.21.15.1:0/2181544499 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6500+0+0 (secure 0 0 0) 0x7faeb4089e50 con 0x7faecc0c5c60 2024-03-06T07:20:25.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.038+0000 7faecaffd640 1 --2- 172.21.15.1:0/2181544499 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faeac06b4d0 0x7faeac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.039+0000 7faecaffd640 1 --2- 172.21.15.1:0/2181544499 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faeac06b4d0 0x7faeac06d990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faebc001780 tx=0x7faebc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:25.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.047+0000 7faec37fe640 1 -- 172.21.15.1:0/2181544499 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faeb40583f0 con 0x7faecc0c5c60 2024-03-06T07:20:25.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.227+0000 7faed1281640 1 -- 172.21.15.1:0/2181544499 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7faecc0680b0 con 0x7faecc0c5c60 2024-03-06T07:20:25.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.868+0000 7faec37fe640 1 -- 172.21.15.1:0/2181544499 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) v1 ==== 169+0+0 (secure 0 0 0) 0x7faeb405c490 con 0x7faecc0c5c60 2024-03-06T07:20:25.870 INFO:teuthology.orchestra.run.smithi001.stderr:set pool 4 allow_ec_overwrites to true 2024-03-06T07:20:25.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.874+0000 7faed1281640 1 -- 172.21.15.1:0/2181544499 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faeac06b4d0 msgr2=0x7faeac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.874+0000 7faed1281640 1 --2- 172.21.15.1:0/2181544499 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faeac06b4d0 0x7faeac06d990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faebc001780 tx=0x7faebc009040 comp rx=0 tx=0).stop 2024-03-06T07:20:25.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.874+0000 7faed1281640 1 -- 172.21.15.1:0/2181544499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faecc0c5c60 msgr2=0x7faecc14afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.874+0000 7faed1281640 1 --2- 172.21.15.1:0/2181544499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faecc0c5c60 0x7faecc14afd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7faeb400e8d0 tx=0x7faeb400ed90 comp rx=0 tx=0).stop 2024-03-06T07:20:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.875+0000 7faed1281640 1 -- 172.21.15.1:0/2181544499 shutdown_connections 2024-03-06T07:20:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.876+0000 7faed1281640 1 --2- 172.21.15.1:0/2181544499 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faeac06b4d0 0x7faeac06d990 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.876+0000 7faed1281640 1 --2- 172.21.15.1:0/2181544499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faecc152a10 0x7faecc1ee2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.876+0000 7faed1281640 1 --2- 172.21.15.1:0/2181544499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faecc0c5c60 0x7faecc14afd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.876+0000 7faed1281640 1 --2- 172.21.15.1:0/2181544499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faecc0c5060 0x7faecc14aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.876+0000 7faed1281640 1 -- 172.21.15.1:0/2181544499 >> 172.21.15.1:0/2181544499 conn(0x7faecc072d50 msgr2=0x7faecc14e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.876+0000 7faed1281640 1 -- 172.21.15.1:0/2181544499 shutdown_connections 2024-03-06T07:20:25.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:25.876+0000 7faed1281640 1 -- 172.21.15.1:0/2181544499 wait complete. 2024-03-06T07:20:25.902 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-06T07:20:26.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.003+0000 7f811e3c9640 1 Processor -- start 2024-03-06T07:20:26.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.003+0000 7f811e3c9640 1 -- start start 2024-03-06T07:20:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f811e3c9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f81180c39c0 0x7f81180ac220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f811e3c9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81180c7420 con 0x7f81180c4940 2024-03-06T07:20:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f811e3c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81180ab760 con 0x7f81180c4560 2024-03-06T07:20:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f811e3c9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81180c76c0 con 0x7f81180c39c0 2024-03-06T07:20:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f811c966640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f81180c4940 0x7f81180ac760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42946/0 (socket says 172.21.15.1:42946) 2024-03-06T07:20:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f811c966640 1 -- 172.21.15.1:0/3815879671 learned_addr learned my addr 172.21.15.1:0/3815879671 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f8116ffd640 1 -- 172.21.15.1:0/3815879671 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668096359 0 0) 0x7f81180c7420 con 0x7f81180c4940 2024-03-06T07:20:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f81177fe640 1 --2- 172.21.15.1:0/3815879671 >> v2:172.21.15.98:3300/0 conn(0x7f81180c39c0 0x7f81180ac220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f8116ffd640 1 -- 172.21.15.1:0/3815879671 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8100003520 con 0x7f81180c4940 2024-03-06T07:20:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f8116ffd640 1 -- 172.21.15.1:0/3815879671 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1604811020 0 0) 0x7f8100003520 con 0x7f81180c4940 2024-03-06T07:20:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f8116ffd640 1 -- 172.21.15.1:0/3815879671 >> v2:172.21.15.98:3300/0 conn(0x7f81180c39c0 msgr2=0x7f81180ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f8116ffd640 1 --2- 172.21.15.1:0/3815879671 >> v2:172.21.15.98:3300/0 conn(0x7f81180c39c0 0x7f81180ac220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f8116ffd640 1 -- 172.21.15.1:0/3815879671 >> v1:172.21.15.98:6789/0 conn(0x7f81180c4560 legacy=0x7f8118155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f8116ffd640 1 -- 172.21.15.1:0/3815879671 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81180aba20 con 0x7f81180c4940 2024-03-06T07:20:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f8116ffd640 1 -- 172.21.15.1:0/3815879671 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f8110001a10 con 0x7f81180c4940 2024-03-06T07:20:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f8116ffd640 1 -- 172.21.15.1:0/3815879671 >> v1:172.21.15.1:6789/0 conn(0x7f81180c4940 legacy=0x7f81180ac760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f8116ffd640 1 --2- 172.21.15.1:0/3815879671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81000039b0 0x7f8100003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f8116ffd640 1 --2- 172.21.15.1:0/3815879671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8100004800 0x7f8100004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f8116ffd640 1 --2- 172.21.15.1:0/3815879671 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81000051c0 0x7f810000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.004+0000 7f8116ffd640 1 -- 172.21.15.1:0/3815879671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81180c76c0 con 0x7f8100004800 2024-03-06T07:20:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.005+0000 7f8116ffd640 1 -- 172.21.15.1:0/3815879671 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810000c060 con 0x7f81000039b0 2024-03-06T07:20:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.005+0000 7f8116ffd640 1 -- 172.21.15.1:0/3815879671 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810000c260 con 0x7f81000051c0 2024-03-06T07:20:26.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.005+0000 7f81177fe640 1 --2- 172.21.15.1:0/3815879671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8100004800 0x7f8100004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.005+0000 7f8117fff640 1 --2- 172.21.15.1:0/3815879671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81000039b0 0x7f8100003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.005+0000 7f811c966640 1 --2- 172.21.15.1:0/3815879671 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81000051c0 0x7f810000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.005+0000 7f81177fe640 1 -- 172.21.15.1:0/3815879671 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81000051c0 msgr2=0x7f810000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:26.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.005+0000 7f81177fe640 1 --2- 172.21.15.1:0/3815879671 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81000051c0 0x7f810000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.005+0000 7f81177fe640 1 -- 172.21.15.1:0/3815879671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81000039b0 msgr2=0x7f8100003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:26.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.005+0000 7f81177fe640 1 --2- 172.21.15.1:0/3815879671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81000039b0 0x7f8100003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.005+0000 7f811c966640 1 -- 172.21.15.1:0/3815879671 reap_dead start 2024-03-06T07:20:26.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.005+0000 7f81177fe640 1 -- 172.21.15.1:0/3815879671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8100003520 con 0x7f8100004800 2024-03-06T07:20:26.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.006+0000 7f81177fe640 1 --2- 172.21.15.1:0/3815879671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8100004800 0x7f8100004c80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8108009ac0 tx=0x7f8100005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f5df19a03e4a16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:26.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.006+0000 7f8116ffd640 1 -- 172.21.15.1:0/3815879671 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108039070 con 0x7f8100004800 2024-03-06T07:20:26.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.006+0000 7f8116ffd640 1 -- 172.21.15.1:0/3815879671 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810802c6b0 con 0x7f8100004800 2024-03-06T07:20:26.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.006+0000 7f8116ffd640 1 -- 172.21.15.1:0/3815879671 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108009690 con 0x7f8100004800 2024-03-06T07:20:26.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.006+0000 7f811e3c9640 1 -- 172.21.15.1:0/3815879671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8100004800 msgr2=0x7f8100004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.006+0000 7f811e3c9640 1 --2- 172.21.15.1:0/3815879671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8100004800 0x7f8100004c80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8108009ac0 tx=0x7f8100005830 comp rx=0 tx=0).stop 2024-03-06T07:20:26.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.006+0000 7f811e3c9640 1 -- 172.21.15.1:0/3815879671 shutdown_connections 2024-03-06T07:20:26.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.006+0000 7f811e3c9640 1 --2- 172.21.15.1:0/3815879671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8100004800 0x7f8100004c80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.006+0000 7f811e3c9640 1 -- 172.21.15.1:0/3815879671 >> 172.21.15.1:0/3815879671 conn(0x7f81180c7a50 msgr2=0x7f81180c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:26.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.006+0000 7f811e3c9640 1 -- 172.21.15.1:0/3815879671 shutdown_connections 2024-03-06T07:20:26.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.006+0000 7f811e3c9640 1 -- 172.21.15.1:0/3815879671 wait complete. 2024-03-06T07:20:26.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.007+0000 7f811e3c9640 1 Processor -- start 2024-03-06T07:20:26.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.008+0000 7f811e3c9640 1 -- start start 2024-03-06T07:20:26.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.008+0000 7f811e3c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8100004800 0x7f81181e6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.008+0000 7f811e3c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81180c15e0 0x7f8118070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.008+0000 7f811e3c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81180a8930 0x7f81181e9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.008+0000 7f811e3c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811806e850 con 0x7f81180c15e0 2024-03-06T07:20:26.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.008+0000 7f811e3c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811806e6d0 con 0x7f8100004800 2024-03-06T07:20:26.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.008+0000 7f811e3c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81180c63a0 con 0x7f81180a8930 2024-03-06T07:20:26.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.008+0000 7f81177fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81180c15e0 0x7f8118070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.008+0000 7f81177fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81180c15e0 0x7f8118070030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34712/0 (socket says 172.21.15.1:34712) 2024-03-06T07:20:26.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.008+0000 7f81177fe640 1 -- 172.21.15.1:0/2140217472 learned_addr learned my addr 172.21.15.1:0/2140217472 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:26.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.009+0000 7f811c966640 1 --2- 172.21.15.1:0/2140217472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81180a8930 0x7f81181e9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.009+0000 7f8117fff640 1 --2- 172.21.15.1:0/2140217472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8100004800 0x7f81181e6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.009+0000 7f81177fe640 1 -- 172.21.15.1:0/2140217472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81180a8930 msgr2=0x7f81181e9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.009+0000 7f81177fe640 1 --2- 172.21.15.1:0/2140217472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81180a8930 0x7f81181e9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.009+0000 7f81177fe640 1 -- 172.21.15.1:0/2140217472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8100004800 msgr2=0x7f81181e6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.009+0000 7f81177fe640 1 --2- 172.21.15.1:0/2140217472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8100004800 0x7f81181e6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.009+0000 7f81177fe640 1 -- 172.21.15.1:0/2140217472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81181e6380 con 0x7f81180c15e0 2024-03-06T07:20:26.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.009+0000 7f81177fe640 1 --2- 172.21.15.1:0/2140217472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81180c15e0 0x7f8118070030 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8108000c00 tx=0x7f8108004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:26.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.009+0000 7f8114ff9640 1 -- 172.21.15.1:0/2140217472 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108042020 con 0x7f81180c15e0 2024-03-06T07:20:26.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.009+0000 7f8114ff9640 1 -- 172.21.15.1:0/2140217472 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810803a040 con 0x7f81180c15e0 2024-03-06T07:20:26.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.009+0000 7f811e3c9640 1 -- 172.21.15.1:0/2140217472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8118070570 con 0x7f81180c15e0 2024-03-06T07:20:26.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.009+0000 7f8114ff9640 1 -- 172.21.15.1:0/2140217472 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108039070 con 0x7f81180c15e0 2024-03-06T07:20:26.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.009+0000 7f811e3c9640 1 -- 172.21.15.1:0/2140217472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81181e6620 con 0x7f81180c15e0 2024-03-06T07:20:26.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.010+0000 7f811e3c9640 1 -- 172.21.15.1:0/2140217472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81180686e0 con 0x7f81180c15e0 2024-03-06T07:20:26.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.011+0000 7f8114ff9640 1 -- 172.21.15.1:0/2140217472 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8108045020 con 0x7f81180c15e0 2024-03-06T07:20:26.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.011+0000 7f8114ff9640 1 --2- 172.21.15.1:0/2140217472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f810406b4d0 0x7f810406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.012+0000 7f8114ff9640 1 -- 172.21.15.1:0/2140217472 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f81080ab8a0 con 0x7f81180c15e0 2024-03-06T07:20:26.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.012+0000 7f8117fff640 1 --2- 172.21.15.1:0/2140217472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f810406b4d0 0x7f810406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.012+0000 7f8117fff640 1 --2- 172.21.15.1:0/2140217472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f810406b4d0 0x7f810406d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f81180505a0 tx=0x7f810c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:26.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.016+0000 7f8114ff9640 1 -- 172.21.15.1:0/2140217472 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f810804c070 con 0x7f81180c15e0 2024-03-06T07:20:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.236+0000 7f811e3c9640 1 -- 172.21.15.1:0/2140217472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f81180c6180 con 0x7f81180c15e0 2024-03-06T07:20:26.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.879+0000 7f8114ff9640 1 -- 172.21.15.1:0/2140217472 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7f8108079db0 con 0x7f81180c15e0 2024-03-06T07:20:26.881 INFO:teuthology.orchestra.run.smithi001.stderr:added data pool 4 to fsmap 2024-03-06T07:20:26.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.881+0000 7f811e3c9640 1 -- 172.21.15.1:0/2140217472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f810406b4d0 msgr2=0x7f810406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.881+0000 7f811e3c9640 1 --2- 172.21.15.1:0/2140217472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f810406b4d0 0x7f810406d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f81180505a0 tx=0x7f810c009040 comp rx=0 tx=0).stop 2024-03-06T07:20:26.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.881+0000 7f811e3c9640 1 -- 172.21.15.1:0/2140217472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81180c15e0 msgr2=0x7f8118070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.881+0000 7f811e3c9640 1 --2- 172.21.15.1:0/2140217472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81180c15e0 0x7f8118070030 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8108000c00 tx=0x7f8108004730 comp rx=0 tx=0).stop 2024-03-06T07:20:26.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.881+0000 7f811e3c9640 1 -- 172.21.15.1:0/2140217472 shutdown_connections 2024-03-06T07:20:26.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.881+0000 7f811e3c9640 1 --2- 172.21.15.1:0/2140217472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f810406b4d0 0x7f810406d990 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.881+0000 7f811e3c9640 1 --2- 172.21.15.1:0/2140217472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81180a8930 0x7f81181e9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.881+0000 7f811e3c9640 1 --2- 172.21.15.1:0/2140217472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81180c15e0 0x7f8118070030 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.881+0000 7f811e3c9640 1 --2- 172.21.15.1:0/2140217472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8100004800 0x7f81181e6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.881+0000 7f811e3c9640 1 -- 172.21.15.1:0/2140217472 >> 172.21.15.1:0/2140217472 conn(0x7f81180c7a50 msgr2=0x7f81180ac120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:26.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.881+0000 7f811e3c9640 1 -- 172.21.15.1:0/2140217472 shutdown_connections 2024-03-06T07:20:26.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.881+0000 7f811e3c9640 1 -- 172.21.15.1:0/2140217472 wait complete. 2024-03-06T07:20:26.894 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:26.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.993+0000 7efd8d1c9640 1 Processor -- start 2024-03-06T07:20:26.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.993+0000 7efd8d1c9640 1 -- start start 2024-03-06T07:20:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.994+0000 7efd8d1c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd8806f6d0 0x7efd88155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.994+0000 7efd8d1c9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd880a84b0 con 0x7efd880c14f0 2024-03-06T07:20:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.994+0000 7efd8d1c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd881550f0 con 0x7efd881528b0 2024-03-06T07:20:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.994+0000 7efd8d1c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd880c0690 con 0x7efd8806f6d0 2024-03-06T07:20:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.994+0000 7efd7ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efd880c14f0 0x7efd881555f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42956/0 (socket says 172.21.15.1:42956) 2024-03-06T07:20:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.994+0000 7efd7ffff640 1 -- 172.21.15.1:0/1736613626 learned_addr learned my addr 172.21.15.1:0/1736613626 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.994+0000 7efd8759e640 1 --2- 172.21.15.1:0/1736613626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd8806f6d0 0x7efd88155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.994+0000 7efd8659c640 1 -- 172.21.15.1:0/1736613626 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714085273 0 0) 0x7efd880a84b0 con 0x7efd880c14f0 2024-03-06T07:20:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.994+0000 7efd8659c640 1 -- 172.21.15.1:0/1736613626 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd74003520 con 0x7efd880c14f0 2024-03-06T07:20:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.994+0000 7efd8659c640 1 -- 172.21.15.1:0/1736613626 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1628639929 0 0) 0x7efd74003520 con 0x7efd880c14f0 2024-03-06T07:20:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd8659c640 1 -- 172.21.15.1:0/1736613626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd8806f6d0 msgr2=0x7efd88155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd8659c640 1 --2- 172.21.15.1:0/1736613626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd8806f6d0 0x7efd88155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd8659c640 1 -- 172.21.15.1:0/1736613626 >> v1:172.21.15.98:6789/0 conn(0x7efd881528b0 legacy=0x7efd880a9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd8659c640 1 -- 172.21.15.1:0/1736613626 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd880c0970 con 0x7efd880c14f0 2024-03-06T07:20:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd8659c640 1 -- 172.21.15.1:0/1736613626 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7efd70001670 con 0x7efd880c14f0 2024-03-06T07:20:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd8659c640 1 -- 172.21.15.1:0/1736613626 >> v1:172.21.15.1:6789/0 conn(0x7efd880c14f0 legacy=0x7efd881555f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd8659c640 1 --2- 172.21.15.1:0/1736613626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd740039e0 0x7efd74003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd8659c640 1 --2- 172.21.15.1:0/1736613626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd74004800 0x7efd74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd8659c640 1 --2- 172.21.15.1:0/1736613626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd8806f6d0 0x7efd74005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd8659c640 1 -- 172.21.15.1:0/1736613626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd880c0690 con 0x7efd74004800 2024-03-06T07:20:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd8659c640 1 -- 172.21.15.1:0/1736613626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7400bb20 con 0x7efd740039e0 2024-03-06T07:20:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd8659c640 1 -- 172.21.15.1:0/1736613626 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7400bd20 con 0x7efd8806f6d0 2024-03-06T07:20:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd7ffff640 1 --2- 172.21.15.1:0/1736613626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd74004800 0x7efd74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd86d9d640 1 --2- 172.21.15.1:0/1736613626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd740039e0 0x7efd74003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd8759e640 1 --2- 172.21.15.1:0/1736613626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd8806f6d0 0x7efd74005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd7ffff640 1 -- 172.21.15.1:0/1736613626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd8806f6d0 msgr2=0x7efd74005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd7ffff640 1 --2- 172.21.15.1:0/1736613626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd8806f6d0 0x7efd74005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd7ffff640 1 -- 172.21.15.1:0/1736613626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd740039e0 msgr2=0x7efd74003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd7ffff640 1 --2- 172.21.15.1:0/1736613626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd740039e0 0x7efd74003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.995+0000 7efd7ffff640 1 -- 172.21.15.1:0/1736613626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd74003520 con 0x7efd74004800 2024-03-06T07:20:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.996+0000 7efd7ffff640 1 --2- 172.21.15.1:0/1736613626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd74004800 0x7efd74004c80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efd7000c210 tx=0x7efd74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9c7d43b7576a54 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.996+0000 7efd8659c640 1 -- 172.21.15.1:0/1736613626 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd70007d10 con 0x7efd74004800 2024-03-06T07:20:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.996+0000 7efd8659c640 1 -- 172.21.15.1:0/1736613626 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd70030070 con 0x7efd74004800 2024-03-06T07:20:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.996+0000 7efd8659c640 1 -- 172.21.15.1:0/1736613626 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd70005d10 con 0x7efd74004800 2024-03-06T07:20:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.996+0000 7efd8d1c9640 1 -- 172.21.15.1:0/1736613626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd74004800 msgr2=0x7efd74004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.996+0000 7efd8d1c9640 1 --2- 172.21.15.1:0/1736613626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd74004800 0x7efd74004c80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efd7000c210 tx=0x7efd74005830 comp rx=0 tx=0).stop 2024-03-06T07:20:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.996+0000 7efd8759e640 1 -- 172.21.15.1:0/1736613626 reap_dead start 2024-03-06T07:20:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.996+0000 7efd8d1c9640 1 -- 172.21.15.1:0/1736613626 shutdown_connections 2024-03-06T07:20:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.996+0000 7efd8d1c9640 1 -- 172.21.15.1:0/1736613626 >> 172.21.15.1:0/1736613626 conn(0x7efd880c3cc0 msgr2=0x7efd880c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.996+0000 7efd8d1c9640 1 -- 172.21.15.1:0/1736613626 shutdown_connections 2024-03-06T07:20:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.996+0000 7efd8d1c9640 1 -- 172.21.15.1:0/1736613626 wait complete. 2024-03-06T07:20:27.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.998+0000 7efd8d1c9640 1 Processor -- start 2024-03-06T07:20:27.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.998+0000 7efd8d1c9640 1 -- start start 2024-03-06T07:20:27.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.998+0000 7efd8d1c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd74004800 0x7efd881475b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.998+0000 7efd8d1c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd8814d610 0x7efd88147af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.998+0000 7efd8d1c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd881ee190 0x7efd881ee610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.998+0000 7efd8d1c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8806fdb0 con 0x7efd74004800 2024-03-06T07:20:27.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.998+0000 7efd8d1c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd880c0f30 con 0x7efd881ee190 2024-03-06T07:20:27.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.998+0000 7efd8d1c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd880a9e70 con 0x7efd8814d610 2024-03-06T07:20:27.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.998+0000 7efd86d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd74004800 0x7efd881475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.998+0000 7efd86d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd74004800 0x7efd881475b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34740/0 (socket says 172.21.15.1:34740) 2024-03-06T07:20:27.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.998+0000 7efd86d9d640 1 -- 172.21.15.1:0/3964172600 learned_addr learned my addr 172.21.15.1:0/3964172600 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:27.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.999+0000 7efd8759e640 1 --2- 172.21.15.1:0/3964172600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd881ee190 0x7efd881ee610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.999+0000 7efd7ffff640 1 --2- 172.21.15.1:0/3964172600 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd8814d610 0x7efd88147af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.999+0000 7efd86d9d640 1 -- 172.21.15.1:0/3964172600 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd8814d610 msgr2=0x7efd88147af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.999+0000 7efd86d9d640 1 --2- 172.21.15.1:0/3964172600 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd8814d610 0x7efd88147af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.999+0000 7efd86d9d640 1 -- 172.21.15.1:0/3964172600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd881ee190 msgr2=0x7efd881ee610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.999+0000 7efd86d9d640 1 --2- 172.21.15.1:0/3964172600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd881ee190 0x7efd881ee610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.999+0000 7efd86d9d640 1 -- 172.21.15.1:0/3964172600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8814cbf0 con 0x7efd74004800 2024-03-06T07:20:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.999+0000 7efd7ffff640 1 --2- 172.21.15.1:0/3964172600 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd8814d610 0x7efd88147af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.999+0000 7efd86d9d640 1 --2- 172.21.15.1:0/3964172600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd74004800 0x7efd881475b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efd7800f620 tx=0x7efd7800fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.999+0000 7efd7f7fe640 1 -- 172.21.15.1:0/3964172600 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd780052e0 con 0x7efd74004800 2024-03-06T07:20:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.999+0000 7efd7f7fe640 1 -- 172.21.15.1:0/3964172600 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd78018070 con 0x7efd74004800 2024-03-06T07:20:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.999+0000 7efd7f7fe640 1 -- 172.21.15.1:0/3964172600 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd78003810 con 0x7efd74004800 2024-03-06T07:20:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.999+0000 7efd8d1c9640 1 -- 172.21.15.1:0/3964172600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd881548b0 con 0x7efd74004800 2024-03-06T07:20:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:26.999+0000 7efd8d1c9640 1 -- 172.21.15.1:0/3964172600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd8814ce90 con 0x7efd74004800 2024-03-06T07:20:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.000+0000 7efd8d1c9640 1 -- 172.21.15.1:0/3964172600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd880ad560 con 0x7efd74004800 2024-03-06T07:20:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.001+0000 7efd7f7fe640 1 -- 172.21.15.1:0/3964172600 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd780039b0 con 0x7efd74004800 2024-03-06T07:20:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.001+0000 7efd7f7fe640 1 --2- 172.21.15.1:0/3964172600 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efd6806b500 0x7efd6806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.001+0000 7efd7f7fe640 1 -- 172.21.15.1:0/3964172600 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7efd7808a220 con 0x7efd74004800 2024-03-06T07:20:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.001+0000 7efd7ffff640 1 --2- 172.21.15.1:0/3964172600 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efd6806b500 0x7efd6806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.002+0000 7efd7ffff640 1 --2- 172.21.15.1:0/3964172600 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efd6806b500 0x7efd6806d9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efd70034000 tx=0x7efd70009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.006+0000 7efd7f7fe640 1 -- 172.21.15.1:0/3964172600 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd880ad560 con 0x7efd74004800 2024-03-06T07:20:27.145 INFO:tasks.ceph.mds.a.smithi001.stderr:2024-03-06T07:20:27.143+0000 7f11b8bbd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:20:27.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.228+0000 7efd8d1c9640 1 -- 172.21.15.1:0/3964172600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd880686e0 con 0x7efd74004800 2024-03-06T07:20:27.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.229+0000 7efd7f7fe640 1 -- 172.21.15.1:0/3964172600 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4964 (secure 0 0 0) 0x7efd78058df0 con 0x7efd74004800 2024-03-06T07:20:27.231 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:27.231 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2822958969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2822958969},{"type":"v1","addr":"172.21.15.98:6836","nonce":2822958969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2442906233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2442906233},{"type":"v1","addr":"172.21.15.98:6837","nonce":2442906233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/448746024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":448746024},{"type":"v1","addr":"172.21.15.1:6834","nonce":448746024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:21.207834+0000","modified":"2024-03-06T07:20:26.238480+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1315115982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1315115982},{"type":"v1","addr":"172.21.15.1:6835","nonce":1315115982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-03-06T07:20:27.231 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 7 2024-03-06T07:20:27.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.231+0000 7efd8d1c9640 1 -- 172.21.15.1:0/3964172600 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efd6806b500 msgr2=0x7efd6806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.231+0000 7efd8d1c9640 1 --2- 172.21.15.1:0/3964172600 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efd6806b500 0x7efd6806d9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efd70034000 tx=0x7efd70009030 comp rx=0 tx=0).stop 2024-03-06T07:20:27.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.231+0000 7efd8d1c9640 1 -- 172.21.15.1:0/3964172600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd74004800 msgr2=0x7efd881475b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.231+0000 7efd8d1c9640 1 --2- 172.21.15.1:0/3964172600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd74004800 0x7efd881475b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efd7800f620 tx=0x7efd7800fae0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.231+0000 7efd8d1c9640 1 -- 172.21.15.1:0/3964172600 shutdown_connections 2024-03-06T07:20:27.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.231+0000 7efd8d1c9640 1 --2- 172.21.15.1:0/3964172600 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efd6806b500 0x7efd6806d9c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.231+0000 7efd8d1c9640 1 --2- 172.21.15.1:0/3964172600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd881ee190 0x7efd881ee610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.231+0000 7efd8d1c9640 1 --2- 172.21.15.1:0/3964172600 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd8814d610 0x7efd88147af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.231+0000 7efd8d1c9640 1 --2- 172.21.15.1:0/3964172600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd74004800 0x7efd881475b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.231+0000 7efd8d1c9640 1 -- 172.21.15.1:0/3964172600 >> 172.21.15.1:0/3964172600 conn(0x7efd880c3cc0 msgr2=0x7efd8814f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.231+0000 7efd8d1c9640 1 -- 172.21.15.1:0/3964172600 shutdown_connections 2024-03-06T07:20:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.231+0000 7efd8d1c9640 1 -- 172.21.15.1:0/3964172600 wait complete. 2024-03-06T07:20:27.243 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:27.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.342+0000 7fb5b8242640 1 Processor -- start 2024-03-06T07:20:27.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.342+0000 7fb5b8242640 1 -- start start 2024-03-06T07:20:27.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.343+0000 7fb5b8242640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb5b00c2e40 0x7fb5b006c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.343+0000 7fb5b8242640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5b00c3220 0x7fb5b00c5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.343+0000 7fb5b8242640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b006eb10 con 0x7fb5b00c5dd0 2024-03-06T07:20:27.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.343+0000 7fb5b8242640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b006f400 con 0x7fb5b00c2e40 2024-03-06T07:20:27.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.343+0000 7fb5b8242640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b006edb0 con 0x7fb5b00c3220 2024-03-06T07:20:27.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.343+0000 7fb5b5fde640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb5b00c2e40 0x7fb5b006c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.343+0000 7fb5b5fde640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb5b00c2e40 0x7fb5b006c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34754/0 (socket says 172.21.15.1:34754) 2024-03-06T07:20:27.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.343+0000 7fb5b5fde640 1 -- 172.21.15.1:0/4071015292 learned_addr learned my addr 172.21.15.1:0/4071015292 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:27.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.343+0000 7fb5b57dd640 1 --2- 172.21.15.1:0/4071015292 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5b00c3220 0x7fb5b00c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.343+0000 7fb5b5fde640 1 -- 172.21.15.1:0/4071015292 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5b00c3220 msgr2=0x7fb5b00c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:27.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.343+0000 7fb5b5fde640 1 --2- 172.21.15.1:0/4071015292 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5b00c3220 0x7fb5b00c5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.343+0000 7fb5b5fde640 1 -- 172.21.15.1:0/4071015292 >> v1:172.21.15.98:6789/0 conn(0x7fb5b00c5dd0 legacy=0x7fb5b0072510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.343+0000 7fb5b5fde640 1 -- 172.21.15.1:0/4071015292 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b006f090 con 0x7fb5b00c2e40 2024-03-06T07:20:27.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.344+0000 7fb5b5fde640 1 --2- 172.21.15.1:0/4071015292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5b00c2e40 0x7fb5b006c100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0098a0 tx=0x7fb5ac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca4bb09b357b21e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.344+0000 7fb5b4fdc640 1 -- 172.21.15.1:0/4071015292 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac035020 con 0x7fb5b00c2e40 2024-03-06T07:20:27.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.344+0000 7fb5b4fdc640 1 -- 172.21.15.1:0/4071015292 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5ac031640 con 0x7fb5b00c2e40 2024-03-06T07:20:27.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.344+0000 7fb5b4fdc640 1 -- 172.21.15.1:0/4071015292 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac02d040 con 0x7fb5b00c2e40 2024-03-06T07:20:27.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.345+0000 7fb5b8242640 1 -- 172.21.15.1:0/4071015292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5b00c2e40 msgr2=0x7fb5b006c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.345+0000 7fb5b8242640 1 --2- 172.21.15.1:0/4071015292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5b00c2e40 0x7fb5b006c100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0098a0 tx=0x7fb5ac02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.345+0000 7fb5b8242640 1 -- 172.21.15.1:0/4071015292 shutdown_connections 2024-03-06T07:20:27.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.345+0000 7fb5b8242640 1 --2- 172.21.15.1:0/4071015292 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5b00c3220 0x7fb5b00c5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.345+0000 7fb5b8242640 1 --2- 172.21.15.1:0/4071015292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5b00c2e40 0x7fb5b006c100 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.345+0000 7fb5b8242640 1 -- 172.21.15.1:0/4071015292 >> 172.21.15.1:0/4071015292 conn(0x7fb5b00a9700 msgr2=0x7fb5b00a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:27.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.345+0000 7fb5b8242640 1 -- 172.21.15.1:0/4071015292 shutdown_connections 2024-03-06T07:20:27.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.345+0000 7fb5b8242640 1 -- 172.21.15.1:0/4071015292 wait complete. 2024-03-06T07:20:27.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.346+0000 7fb5b8242640 1 Processor -- start 2024-03-06T07:20:27.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.347+0000 7fb5b8242640 1 -- start start 2024-03-06T07:20:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b8242640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5b00c2e40 0x7fb5b00bdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b8242640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b00c3220 0x7fb5b01e7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b8242640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5b00c5dd0 0x7fb5b00bef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b8242640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b006c7c0 con 0x7fb5b00c5dd0 2024-03-06T07:20:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b8242640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b006c640 con 0x7fb5b00c3220 2024-03-06T07:20:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b8242640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b00a6e30 con 0x7fb5b00c2e40 2024-03-06T07:20:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b67df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5b00c5dd0 0x7fb5b00bef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b67df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5b00c5dd0 0x7fb5b00bef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34756/0 (socket says 172.21.15.1:34756) 2024-03-06T07:20:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b67df640 1 -- 172.21.15.1:0/2851267089 learned_addr learned my addr 172.21.15.1:0/2851267089 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b5fde640 1 --2- 172.21.15.1:0/2851267089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5b00c2e40 0x7fb5b00bdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b57dd640 1 --2- 172.21.15.1:0/2851267089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b00c3220 0x7fb5b01e7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b67df640 1 -- 172.21.15.1:0/2851267089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5b00c2e40 msgr2=0x7fb5b00bdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b67df640 1 --2- 172.21.15.1:0/2851267089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5b00c2e40 0x7fb5b00bdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b67df640 1 -- 172.21.15.1:0/2851267089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b00c3220 msgr2=0x7fb5b01e7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b67df640 1 --2- 172.21.15.1:0/2851267089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b00c3220 0x7fb5b01e7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b67df640 1 -- 172.21.15.1:0/2851267089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b00bca10 con 0x7fb5b00c5dd0 2024-03-06T07:20:27.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b67df640 1 --2- 172.21.15.1:0/2851267089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5b00c5dd0 0x7fb5b00bef10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8002190 tx=0x7fb5a800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5a67fc640 1 -- 172.21.15.1:0/2851267089 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8003b90 con 0x7fb5b00c5dd0 2024-03-06T07:20:27.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5a67fc640 1 -- 172.21.15.1:0/2851267089 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a8003d30 con 0x7fb5b00c5dd0 2024-03-06T07:20:27.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b8242640 1 -- 172.21.15.1:0/2851267089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b00bf450 con 0x7fb5b00c5dd0 2024-03-06T07:20:27.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5a67fc640 1 -- 172.21.15.1:0/2851267089 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8005310 con 0x7fb5b00c5dd0 2024-03-06T07:20:27.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.348+0000 7fb5b8242640 1 -- 172.21.15.1:0/2851267089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b00bcc80 con 0x7fb5b00c5dd0 2024-03-06T07:20:27.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.349+0000 7fb5b8242640 1 -- 172.21.15.1:0/2851267089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b00686e0 con 0x7fb5b00c5dd0 2024-03-06T07:20:27.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.350+0000 7fb5a67fc640 1 -- 172.21.15.1:0/2851267089 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb5a80054b0 con 0x7fb5b00c5dd0 2024-03-06T07:20:27.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.350+0000 7fb5a67fc640 1 --2- 172.21.15.1:0/2851267089 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb59406b4d0 0x7fb59406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.350+0000 7fb5a67fc640 1 -- 172.21.15.1:0/2851267089 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb5a808c260 con 0x7fb5b00c5dd0 2024-03-06T07:20:27.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.350+0000 7fb5b5fde640 1 --2- 172.21.15.1:0/2851267089 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb59406b4d0 0x7fb59406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.351+0000 7fb5b5fde640 1 --2- 172.21.15.1:0/2851267089 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb59406b4d0 0x7fb59406d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac036000 tx=0x7fb5ac03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.354+0000 7fb5a67fc640 1 -- 172.21.15.1:0/2851267089 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb5a8016020 con 0x7fb5b00c5dd0 2024-03-06T07:20:27.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.535+0000 7fb5b8242640 1 -- 172.21.15.1:0/2851267089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5b01da160 con 0x7fb5b00c5dd0 2024-03-06T07:20:27.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.536+0000 7fb5a67fc640 1 -- 172.21.15.1:0/2851267089 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18407 (secure 0 0 0) 0x7fb5a805a750 con 0x7fb5b00c5dd0 2024-03-06T07:20:27.539 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:27.539 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":24,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:26.238279+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:27.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.538+0000 7fb5b8242640 1 -- 172.21.15.1:0/2851267089 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb59406b4d0 msgr2=0x7fb59406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.538+0000 7fb5b8242640 1 --2- 172.21.15.1:0/2851267089 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb59406b4d0 0x7fb59406d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac036000 tx=0x7fb5ac03c040 comp rx=0 tx=0).stop 2024-03-06T07:20:27.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.538+0000 7fb5b8242640 1 -- 172.21.15.1:0/2851267089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5b00c5dd0 msgr2=0x7fb5b00bef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.538+0000 7fb5b8242640 1 --2- 172.21.15.1:0/2851267089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5b00c5dd0 0x7fb5b00bef10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8002190 tx=0x7fb5a800cbc0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.538+0000 7fb5b8242640 1 -- 172.21.15.1:0/2851267089 shutdown_connections 2024-03-06T07:20:27.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.538+0000 7fb5b8242640 1 --2- 172.21.15.1:0/2851267089 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb59406b4d0 0x7fb59406d990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.538+0000 7fb5b8242640 1 --2- 172.21.15.1:0/2851267089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5b00c5dd0 0x7fb5b00bef10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.538+0000 7fb5b8242640 1 --2- 172.21.15.1:0/2851267089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b00c3220 0x7fb5b01e7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.538+0000 7fb5b8242640 1 --2- 172.21.15.1:0/2851267089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5b00c2e40 0x7fb5b00bdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.538+0000 7fb5b8242640 1 -- 172.21.15.1:0/2851267089 >> 172.21.15.1:0/2851267089 conn(0x7fb5b00a9700 msgr2=0x7fb5b01dd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:27.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.538+0000 7fb5b8242640 1 -- 172.21.15.1:0/2851267089 shutdown_connections 2024-03-06T07:20:27.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.538+0000 7fb5b8242640 1 -- 172.21.15.1:0/2851267089 wait complete. 2024-03-06T07:20:27.551 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:27.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.651+0000 7f06af965640 1 Processor -- start 2024-03-06T07:20:27.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.651+0000 7f06af965640 1 -- start start 2024-03-06T07:20:27.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.652+0000 7f06af965640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06a80c1390 0x7f06a80c17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.652+0000 7f06af965640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f06a81552e0 0x7f06a8155750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.652+0000 7f06af965640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a81588e0 con 0x7f06a8152730 2024-03-06T07:20:27.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.652+0000 7f06af965640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06a80c1d30 con 0x7f06a81552e0 2024-03-06T07:20:27.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.652+0000 7f06af965640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a8158bc0 con 0x7f06a80c1390 2024-03-06T07:20:27.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.652+0000 7f06adf02640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f06a81552e0 0x7f06a8155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.652+0000 7f06ad701640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f06a8152730 0x7f06a80c6a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42964/0 (socket says 172.21.15.1:42964) 2024-03-06T07:20:27.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.652+0000 7f06ad701640 1 -- 172.21.15.1:0/2421007156 learned_addr learned my addr 172.21.15.1:0/2421007156 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:27.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.652+0000 7f06acf00640 1 --2- 172.21.15.1:0/2421007156 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06a80c1390 0x7f06a80c17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.652+0000 7f06adf02640 1 -- 172.21.15.1:0/2421007156 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06a80c1390 msgr2=0x7f06a80c17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:27.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.652+0000 7f06adf02640 1 --2- 172.21.15.1:0/2421007156 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06a80c1390 0x7f06a80c17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.652+0000 7f06adf02640 1 -- 172.21.15.1:0/2421007156 >> v1:172.21.15.1:6789/0 conn(0x7f06a8152730 legacy=0x7f06a80c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.652+0000 7f06adf02640 1 -- 172.21.15.1:0/2421007156 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a8158ea0 con 0x7f06a81552e0 2024-03-06T07:20:27.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.652+0000 7f069ffff640 1 -- 172.21.15.1:0/2421007156 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971293416 0 0) 0x7f06a81588e0 con 0x7f06a8152730 2024-03-06T07:20:27.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.653+0000 7f06adf02640 1 --2- 172.21.15.1:0/2421007156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06a81552e0 0x7f06a8155750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009d90 tx=0x7f06a002bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f24c41ee0fcba378 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.653+0000 7f069ffff640 1 -- 172.21.15.1:0/2421007156 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a0039070 con 0x7f06a81552e0 2024-03-06T07:20:27.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.653+0000 7f069ffff640 1 -- 172.21.15.1:0/2421007156 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a002cb30 con 0x7f06a81552e0 2024-03-06T07:20:27.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.653+0000 7f069ffff640 1 -- 172.21.15.1:0/2421007156 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a0034440 con 0x7f06a81552e0 2024-03-06T07:20:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.654+0000 7f06af965640 1 -- 172.21.15.1:0/2421007156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06a81552e0 msgr2=0x7f06a8155750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.654+0000 7f06af965640 1 --2- 172.21.15.1:0/2421007156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06a81552e0 0x7f06a8155750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009d90 tx=0x7f06a002bd80 comp rx=0 tx=0).stop 2024-03-06T07:20:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.654+0000 7f06af965640 1 -- 172.21.15.1:0/2421007156 shutdown_connections 2024-03-06T07:20:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.654+0000 7f06af965640 1 --2- 172.21.15.1:0/2421007156 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06a80c1390 0x7f06a80c17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.654+0000 7f06af965640 1 --2- 172.21.15.1:0/2421007156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06a81552e0 0x7f06a8155750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.654+0000 7f06af965640 1 -- 172.21.15.1:0/2421007156 >> 172.21.15.1:0/2421007156 conn(0x7f06a8066d00 msgr2=0x7f06a8067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.654+0000 7f06af965640 1 -- 172.21.15.1:0/2421007156 shutdown_connections 2024-03-06T07:20:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.654+0000 7f06af965640 1 -- 172.21.15.1:0/2421007156 wait complete. 2024-03-06T07:20:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.655+0000 7f06af965640 1 Processor -- start 2024-03-06T07:20:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.655+0000 7f06af965640 1 -- start start 2024-03-06T07:20:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.655+0000 7f06af965640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06a80c1390 0x7f06a81539a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.655+0000 7f06af965640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06a8152730 0x7f06a81eb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.655+0000 7f06af965640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a81552e0 0x7f06a81ebab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.655+0000 7f06af965640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a8152b10 con 0x7f06a80c1390 2024-03-06T07:20:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.655+0000 7f06af965640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a8159460 con 0x7f06a81552e0 2024-03-06T07:20:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.655+0000 7f06af965640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a80c71a0 con 0x7f06a8152730 2024-03-06T07:20:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.656+0000 7f06ad701640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06a80c1390 0x7f06a81539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.656+0000 7f06ad701640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06a80c1390 0x7f06a81539a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34762/0 (socket says 172.21.15.1:34762) 2024-03-06T07:20:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.656+0000 7f06ad701640 1 -- 172.21.15.1:0/1887735000 learned_addr learned my addr 172.21.15.1:0/1887735000 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.656+0000 7f06adf02640 1 --2- 172.21.15.1:0/1887735000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a81552e0 0x7f06a81ebab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.656+0000 7f06acf00640 1 --2- 172.21.15.1:0/1887735000 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06a8152730 0x7f06a81eb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.656+0000 7f06ad701640 1 -- 172.21.15.1:0/1887735000 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06a8152730 msgr2=0x7f06a81eb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.656+0000 7f06ad701640 1 --2- 172.21.15.1:0/1887735000 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06a8152730 0x7f06a81eb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.656+0000 7f06ad701640 1 -- 172.21.15.1:0/1887735000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a81552e0 msgr2=0x7f06a81ebab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.656+0000 7f06ad701640 1 --2- 172.21.15.1:0/1887735000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a81552e0 0x7f06a81ebab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.656+0000 7f06ad701640 1 -- 172.21.15.1:0/1887735000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a81ebff0 con 0x7f06a80c1390 2024-03-06T07:20:27.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.656+0000 7f06adf02640 1 --2- 172.21.15.1:0/1887735000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a81552e0 0x7f06a81ebab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:27.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.656+0000 7f06ad701640 1 --2- 172.21.15.1:0/1887735000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06a80c1390 0x7f06a81539a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f06a400c4a0 tx=0x7f06a400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.656+0000 7f069dffb640 1 -- 172.21.15.1:0/1887735000 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a4017070 con 0x7f06a80c1390 2024-03-06T07:20:27.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.656+0000 7f069dffb640 1 -- 172.21.15.1:0/1887735000 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a4005ce0 con 0x7f06a80c1390 2024-03-06T07:20:27.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.656+0000 7f06af965640 1 -- 172.21.15.1:0/1887735000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a81eb070 con 0x7f06a80c1390 2024-03-06T07:20:27.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.656+0000 7f069dffb640 1 -- 172.21.15.1:0/1887735000 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a40072b0 con 0x7f06a80c1390 2024-03-06T07:20:27.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.656+0000 7f06af965640 1 -- 172.21.15.1:0/1887735000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a81ef0a0 con 0x7f06a80c1390 2024-03-06T07:20:27.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.658+0000 7f069dffb640 1 -- 172.21.15.1:0/1887735000 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f06a401a070 con 0x7f06a80c1390 2024-03-06T07:20:27.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.658+0000 7f06af965640 1 -- 172.21.15.1:0/1887735000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a806b620 con 0x7f06a80c1390 2024-03-06T07:20:27.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.658+0000 7f069dffb640 1 --2- 172.21.15.1:0/1887735000 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f068c06b4d0 0x7f068c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.658+0000 7f069dffb640 1 -- 172.21.15.1:0/1887735000 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f06a40898c0 con 0x7f06a80c1390 2024-03-06T07:20:27.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.658+0000 7f06acf00640 1 --2- 172.21.15.1:0/1887735000 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f068c06b4d0 0x7f068c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.659+0000 7f06acf00640 1 --2- 172.21.15.1:0/1887735000 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f068c06b4d0 0x7f068c06d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f06a8158810 tx=0x7f0698008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.662+0000 7f069dffb640 1 -- 172.21.15.1:0/1887735000 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f06a806b620 con 0x7f06a80c1390 2024-03-06T07:20:27.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.843+0000 7f06af965640 1 -- 172.21.15.1:0/1887735000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06a80c4460 con 0x7f06a80c1390 2024-03-06T07:20:27.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.845+0000 7f069dffb640 1 -- 172.21.15.1:0/1887735000 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18407 (secure 0 0 0) 0x7f06a81ebff0 con 0x7f06a80c1390 2024-03-06T07:20:27.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:27.847 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":24,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:26.238279+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.847+0000 7f06af965640 1 -- 172.21.15.1:0/1887735000 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f068c06b4d0 msgr2=0x7f068c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.847+0000 7f06af965640 1 --2- 172.21.15.1:0/1887735000 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f068c06b4d0 0x7f068c06d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f06a8158810 tx=0x7f0698008040 comp rx=0 tx=0).stop 2024-03-06T07:20:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.847+0000 7f06af965640 1 -- 172.21.15.1:0/1887735000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06a80c1390 msgr2=0x7f06a81539a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.847+0000 7f06af965640 1 --2- 172.21.15.1:0/1887735000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06a80c1390 0x7f06a81539a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f06a400c4a0 tx=0x7f06a400c960 comp rx=0 tx=0).stop 2024-03-06T07:20:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.847+0000 7f06af965640 1 -- 172.21.15.1:0/1887735000 shutdown_connections 2024-03-06T07:20:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.847+0000 7f06af965640 1 --2- 172.21.15.1:0/1887735000 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f068c06b4d0 0x7f068c06d990 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.847+0000 7f06af965640 1 --2- 172.21.15.1:0/1887735000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06a81552e0 0x7f06a81ebab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.847+0000 7f06af965640 1 --2- 172.21.15.1:0/1887735000 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06a8152730 0x7f06a81eb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.847+0000 7f06af965640 1 --2- 172.21.15.1:0/1887735000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06a80c1390 0x7f06a81539a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.847+0000 7f06af965640 1 -- 172.21.15.1:0/1887735000 >> 172.21.15.1:0/1887735000 conn(0x7f06a8066d00 msgr2=0x7f06a80c6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.847+0000 7f06af965640 1 -- 172.21.15.1:0/1887735000 shutdown_connections 2024-03-06T07:20:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.847+0000 7f06af965640 1 -- 172.21.15.1:0/1887735000 wait complete. 2024-03-06T07:20:27.875 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:27.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.976+0000 7fcfae5bd640 1 Processor -- start 2024-03-06T07:20:27.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.976+0000 7fcfae5bd640 1 -- start start 2024-03-06T07:20:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.977+0000 7fcfae5bd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcfa8152820 0x7fcfa80c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.977+0000 7fcfae5bd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfa80af6f0 0x7fcfa815fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.977+0000 7fcfae5bd640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa80c7180 con 0x7fcfa80c0670 2024-03-06T07:20:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.977+0000 7fcfae5bd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa8155460 con 0x7fcfa8152820 2024-03-06T07:20:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.977+0000 7fcfae5bd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa80c6bc0 con 0x7fcfa80af6f0 2024-03-06T07:20:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.977+0000 7fcfa77fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcfa80c0670 0x7fcfa80c73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42976/0 (socket says 172.21.15.1:42976) 2024-03-06T07:20:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.977+0000 7fcfa77fe640 1 -- 172.21.15.1:0/885343668 learned_addr learned my addr 172.21.15.1:0/885343668 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.977+0000 7fcfa6ffd640 1 -- 172.21.15.1:0/885343668 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167910591 0 0) 0x7fcfa80c7180 con 0x7fcfa80c0670 2024-03-06T07:20:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.977+0000 7fcfacb5a640 1 --2- 172.21.15.1:0/885343668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfa80af6f0 0x7fcfa815fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.977+0000 7fcfa7fff640 1 --2- 172.21.15.1:0/885343668 >> v2:172.21.15.98:3300/0 conn(0x7fcfa8152820 0x7fcfa80c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.977+0000 7fcfa6ffd640 1 -- 172.21.15.1:0/885343668 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf90003520 con 0x7fcfa80c0670 2024-03-06T07:20:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfa6ffd640 1 -- 172.21.15.1:0/885343668 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1523745776 0 0) 0x7fcf90003520 con 0x7fcfa80c0670 2024-03-06T07:20:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfa6ffd640 1 -- 172.21.15.1:0/885343668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfa80af6f0 msgr2=0x7fcfa815fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfa6ffd640 1 --2- 172.21.15.1:0/885343668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfa80af6f0 0x7fcfa815fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfa6ffd640 1 -- 172.21.15.1:0/885343668 >> v2:172.21.15.98:3300/0 conn(0x7fcfa8152820 msgr2=0x7fcfa80c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfa6ffd640 1 --2- 172.21.15.1:0/885343668 >> v2:172.21.15.98:3300/0 conn(0x7fcfa8152820 0x7fcfa80c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfa6ffd640 1 -- 172.21.15.1:0/885343668 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa80c6ea0 con 0x7fcfa80c0670 2024-03-06T07:20:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfa6ffd640 1 -- 172.21.15.1:0/885343668 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fcf9c001670 con 0x7fcfa80c0670 2024-03-06T07:20:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfa6ffd640 1 -- 172.21.15.1:0/885343668 >> v1:172.21.15.1:6789/0 conn(0x7fcfa80c0670 legacy=0x7fcfa80c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfa6ffd640 1 --2- 172.21.15.1:0/885343668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf900039b0 0x7fcf90003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfa6ffd640 1 --2- 172.21.15.1:0/885343668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfa80af6f0 0x7fcf90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfa6ffd640 1 --2- 172.21.15.1:0/885343668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf90004d40 0x7fcf900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfa6ffd640 1 -- 172.21.15.1:0/885343668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa8155460 con 0x7fcf90004d40 2024-03-06T07:20:27.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfa6ffd640 1 -- 172.21.15.1:0/885343668 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa80c6bc0 con 0x7fcf900039b0 2024-03-06T07:20:27.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfa6ffd640 1 -- 172.21.15.1:0/885343668 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9000b9c0 con 0x7fcfa80af6f0 2024-03-06T07:20:27.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfacb5a640 1 --2- 172.21.15.1:0/885343668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf90004d40 0x7fcf900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfa7fff640 1 --2- 172.21.15.1:0/885343668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf900039b0 0x7fcf90003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfa77fe640 1 --2- 172.21.15.1:0/885343668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfa80af6f0 0x7fcf90004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfacb5a640 1 -- 172.21.15.1:0/885343668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfa80af6f0 msgr2=0x7fcf90004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:27.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfacb5a640 1 --2- 172.21.15.1:0/885343668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfa80af6f0 0x7fcf90004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfacb5a640 1 -- 172.21.15.1:0/885343668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf900039b0 msgr2=0x7fcf90003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:27.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfacb5a640 1 --2- 172.21.15.1:0/885343668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf900039b0 0x7fcf90003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.978+0000 7fcfacb5a640 1 -- 172.21.15.1:0/885343668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf90003520 con 0x7fcf90004d40 2024-03-06T07:20:27.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.979+0000 7fcfacb5a640 1 --2- 172.21.15.1:0/885343668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf90004d40 0x7fcf900051c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009f90 tx=0x7fcf90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b097dce1c1f041a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.979+0000 7fcfa6ffd640 1 -- 172.21.15.1:0/885343668 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0039070 con 0x7fcf90004d40 2024-03-06T07:20:27.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.979+0000 7fcfa6ffd640 1 -- 172.21.15.1:0/885343668 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa002cbd0 con 0x7fcf90004d40 2024-03-06T07:20:27.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.979+0000 7fcfa6ffd640 1 -- 172.21.15.1:0/885343668 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0009c10 con 0x7fcf90004d40 2024-03-06T07:20:27.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.980+0000 7fcfae5bd640 1 -- 172.21.15.1:0/885343668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf90004d40 msgr2=0x7fcf900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.980+0000 7fcfae5bd640 1 --2- 172.21.15.1:0/885343668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf90004d40 0x7fcf900051c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009f90 tx=0x7fcf90005830 comp rx=0 tx=0).stop 2024-03-06T07:20:27.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.980+0000 7fcfacb5a640 1 -- 172.21.15.1:0/885343668 reap_dead start 2024-03-06T07:20:27.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.980+0000 7fcfae5bd640 1 -- 172.21.15.1:0/885343668 shutdown_connections 2024-03-06T07:20:27.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.980+0000 7fcfae5bd640 1 -- 172.21.15.1:0/885343668 >> 172.21.15.1:0/885343668 conn(0x7fcfa8072720 msgr2=0x7fcfa8072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:27.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.980+0000 7fcfae5bd640 1 -- 172.21.15.1:0/885343668 shutdown_connections 2024-03-06T07:20:27.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.980+0000 7fcfae5bd640 1 -- 172.21.15.1:0/885343668 wait complete. 2024-03-06T07:20:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.981+0000 7fcfae5bd640 1 Processor -- start 2024-03-06T07:20:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.981+0000 7fcfae5bd640 1 -- start start 2024-03-06T07:20:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfae5bd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf90004d40 0x7fcfa81ea5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfae5bd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfa80c3ae0 0x7fcfa80c2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfae5bd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfa814ace0 0x7fcfa8158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfae5bd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa80c7c80 con 0x7fcfa814ace0 2024-03-06T07:20:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfae5bd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa80c7b00 con 0x7fcfa80c3ae0 2024-03-06T07:20:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfae5bd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa8152c00 con 0x7fcf90004d40 2024-03-06T07:20:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfacb5a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfa814ace0 0x7fcfa8158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfacb5a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfa814ace0 0x7fcfa8158c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34778/0 (socket says 172.21.15.1:34778) 2024-03-06T07:20:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfacb5a640 1 -- 172.21.15.1:0/2053894836 learned_addr learned my addr 172.21.15.1:0/2053894836 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfa7fff640 1 --2- 172.21.15.1:0/2053894836 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf90004d40 0x7fcfa81ea5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfa77fe640 1 --2- 172.21.15.1:0/2053894836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfa80c3ae0 0x7fcfa80c2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfacb5a640 1 -- 172.21.15.1:0/2053894836 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf90004d40 msgr2=0x7fcfa81ea5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfacb5a640 1 --2- 172.21.15.1:0/2053894836 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf90004d40 0x7fcfa81ea5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfacb5a640 1 -- 172.21.15.1:0/2053894836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfa80c3ae0 msgr2=0x7fcfa80c2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfacb5a640 1 --2- 172.21.15.1:0/2053894836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfa80c3ae0 0x7fcfa80c2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfacb5a640 1 -- 172.21.15.1:0/2053894836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa81eaae0 con 0x7fcfa814ace0 2024-03-06T07:20:27.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfacb5a640 1 --2- 172.21.15.1:0/2053894836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfa814ace0 0x7fcfa8158c60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0037040 tx=0x7fcfa0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfa4ff9640 1 -- 172.21.15.1:0/2053894836 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0039030 con 0x7fcfa814ace0 2024-03-06T07:20:27.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfa4ff9640 1 -- 172.21.15.1:0/2053894836 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa0043070 con 0x7fcfa814ace0 2024-03-06T07:20:27.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfae5bd640 1 -- 172.21.15.1:0/2053894836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa8157ec0 con 0x7fcfa814ace0 2024-03-06T07:20:27.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.982+0000 7fcfa4ff9640 1 -- 172.21.15.1:0/2053894836 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0009c10 con 0x7fcfa814ace0 2024-03-06T07:20:27.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.983+0000 7fcfae5bd640 1 -- 172.21.15.1:0/2053894836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa81eb060 con 0x7fcfa814ace0 2024-03-06T07:20:27.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.984+0000 7fcfae5bd640 1 -- 172.21.15.1:0/2053894836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa80686e0 con 0x7fcfa814ace0 2024-03-06T07:20:27.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.984+0000 7fcfa4ff9640 1 -- 172.21.15.1:0/2053894836 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcfa003d7d0 con 0x7fcfa814ace0 2024-03-06T07:20:27.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.984+0000 7fcfa4ff9640 1 --2- 172.21.15.1:0/2053894836 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf9406b500 0x7fcf9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.984+0000 7fcfa4ff9640 1 -- 172.21.15.1:0/2053894836 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcfa00aca80 con 0x7fcfa814ace0 2024-03-06T07:20:27.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.984+0000 7fcfa7fff640 1 --2- 172.21.15.1:0/2053894836 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf9406b500 0x7fcf9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.985+0000 7fcfa7fff640 1 --2- 172.21.15.1:0/2053894836 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf9406b500 0x7fcf9406d9c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009bc0 tx=0x7fcf98009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:27.988+0000 7fcfa4ff9640 1 -- 172.21.15.1:0/2053894836 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcfa00aee20 con 0x7fcfa814ace0 2024-03-06T07:20:28.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.219+0000 7fcfae5bd640 1 -- 172.21.15.1:0/2053894836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfa80c5b40 con 0x7fcfa814ace0 2024-03-06T07:20:28.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.219+0000 7fcfa4ff9640 1 -- 172.21.15.1:0/2053894836 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4964 (secure 0 0 0) 0x7fcfa007b670 con 0x7fcfa814ace0 2024-03-06T07:20:28.222 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:28.222 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2822958969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2822958969},{"type":"v1","addr":"172.21.15.98:6836","nonce":2822958969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2442906233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2442906233},{"type":"v1","addr":"172.21.15.98:6837","nonce":2442906233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/448746024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":448746024},{"type":"v1","addr":"172.21.15.1:6834","nonce":448746024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:21.207834+0000","modified":"2024-03-06T07:20:26.238480+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1315115982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1315115982},{"type":"v1","addr":"172.21.15.1:6835","nonce":1315115982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-03-06T07:20:28.222 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 7 2024-03-06T07:20:28.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.221+0000 7fcfae5bd640 1 -- 172.21.15.1:0/2053894836 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf9406b500 msgr2=0x7fcf9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:28.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.221+0000 7fcfae5bd640 1 --2- 172.21.15.1:0/2053894836 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf9406b500 0x7fcf9406d9c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009bc0 tx=0x7fcf98009190 comp rx=0 tx=0).stop 2024-03-06T07:20:28.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.221+0000 7fcfae5bd640 1 -- 172.21.15.1:0/2053894836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfa814ace0 msgr2=0x7fcfa8158c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:28.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.221+0000 7fcfae5bd640 1 --2- 172.21.15.1:0/2053894836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfa814ace0 0x7fcfa8158c60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0037040 tx=0x7fcfa0009750 comp rx=0 tx=0).stop 2024-03-06T07:20:28.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.221+0000 7fcfae5bd640 1 -- 172.21.15.1:0/2053894836 shutdown_connections 2024-03-06T07:20:28.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.221+0000 7fcfae5bd640 1 --2- 172.21.15.1:0/2053894836 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf9406b500 0x7fcf9406d9c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.221+0000 7fcfae5bd640 1 --2- 172.21.15.1:0/2053894836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfa814ace0 0x7fcfa8158c60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.221+0000 7fcfae5bd640 1 --2- 172.21.15.1:0/2053894836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfa80c3ae0 0x7fcfa80c2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.221+0000 7fcfae5bd640 1 --2- 172.21.15.1:0/2053894836 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf90004d40 0x7fcfa81ea5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.221+0000 7fcfae5bd640 1 -- 172.21.15.1:0/2053894836 >> 172.21.15.1:0/2053894836 conn(0x7fcfa8072720 msgr2=0x7fcfa8072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:28.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.222+0000 7fcfae5bd640 1 -- 172.21.15.1:0/2053894836 shutdown_connections 2024-03-06T07:20:28.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.222+0000 7fcfae5bd640 1 -- 172.21.15.1:0/2053894836 wait complete. 2024-03-06T07:20:28.234 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.333+0000 7fbbb606f640 1 Processor -- start 2024-03-06T07:20:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.333+0000 7fbbb606f640 1 -- start start 2024-03-06T07:20:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.333+0000 7fbbb606f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbbb00c3970 0x7fbbb00c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.333+0000 7fbbb606f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbbb0152840 0x7fbbb01553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.333+0000 7fbbb606f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb00c5e40 con 0x7fbbb00c60b0 2024-03-06T07:20:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.333+0000 7fbbb606f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbb00c67f0 con 0x7fbbb0152840 2024-03-06T07:20:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.333+0000 7fbbb606f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbb00c6a40 con 0x7fbbb00c3970 2024-03-06T07:20:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.334+0000 7fbbaffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbbb0152840 0x7fbbb01553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.334+0000 7fbbaeffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbbb00c60b0 0x7fbbb0152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42990/0 (socket says 172.21.15.1:42990) 2024-03-06T07:20:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.334+0000 7fbbaeffd640 1 -- 172.21.15.1:0/3959626386 learned_addr learned my addr 172.21.15.1:0/3959626386 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.334+0000 7fbbaf7fe640 1 --2- 172.21.15.1:0/3959626386 >> v2:172.21.15.98:3300/0 conn(0x7fbbb00c3970 0x7fbbb00c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.334+0000 7fbbaffff640 1 -- 172.21.15.1:0/3959626386 >> v2:172.21.15.98:3300/0 conn(0x7fbbb00c3970 msgr2=0x7fbbb00c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.334+0000 7fbbaffff640 1 --2- 172.21.15.1:0/3959626386 >> v2:172.21.15.98:3300/0 conn(0x7fbbb00c3970 0x7fbbb00c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.334+0000 7fbbaffff640 1 -- 172.21.15.1:0/3959626386 >> v1:172.21.15.1:6789/0 conn(0x7fbbb00c60b0 legacy=0x7fbbb0152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.334+0000 7fbbae7fc640 1 -- 172.21.15.1:0/3959626386 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637406990 0 0) 0x7fbbb00c5e40 con 0x7fbbb00c60b0 2024-03-06T07:20:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.334+0000 7fbbaffff640 1 -- 172.21.15.1:0/3959626386 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb00c6d20 con 0x7fbbb0152840 2024-03-06T07:20:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.335+0000 7fbbaffff640 1 --2- 172.21.15.1:0/3959626386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbb0152840 0x7fbbb01553d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00c34c0 tx=0x7fbba802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e6b59f8ba12cdd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.335+0000 7fbbae7fc640 1 -- 172.21.15.1:0/3959626386 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba8039070 con 0x7fbbb0152840 2024-03-06T07:20:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.335+0000 7fbbae7fc640 1 -- 172.21.15.1:0/3959626386 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba802cc80 con 0x7fbbb0152840 2024-03-06T07:20:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.335+0000 7fbbae7fc640 1 -- 172.21.15.1:0/3959626386 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba8034660 con 0x7fbbb0152840 2024-03-06T07:20:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.335+0000 7fbbb606f640 1 -- 172.21.15.1:0/3959626386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbb0152840 msgr2=0x7fbbb01553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.335+0000 7fbbb606f640 1 --2- 172.21.15.1:0/3959626386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbb0152840 0x7fbbb01553d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00c34c0 tx=0x7fbba802bed0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.335+0000 7fbbb606f640 1 -- 172.21.15.1:0/3959626386 shutdown_connections 2024-03-06T07:20:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.335+0000 7fbbb606f640 1 --2- 172.21.15.1:0/3959626386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbb0152840 0x7fbbb01553d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.335+0000 7fbbb606f640 1 --2- 172.21.15.1:0/3959626386 >> v2:172.21.15.98:3300/0 conn(0x7fbbb00c3970 0x7fbbb00c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.335+0000 7fbbb606f640 1 -- 172.21.15.1:0/3959626386 >> 172.21.15.1:0/3959626386 conn(0x7fbbb00c88f0 msgr2=0x7fbbb00c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.335+0000 7fbbb606f640 1 -- 172.21.15.1:0/3959626386 shutdown_connections 2024-03-06T07:20:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.335+0000 7fbbb606f640 1 -- 172.21.15.1:0/3959626386 wait complete. 2024-03-06T07:20:28.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.336+0000 7fbbb606f640 1 Processor -- start 2024-03-06T07:20:28.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.337+0000 7fbbb606f640 1 -- start start 2024-03-06T07:20:28.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.337+0000 7fbbb606f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbb00c3970 0x7fbbb01ed9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.337+0000 7fbbb606f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbb00c60b0 0x7fbbb01ea3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.337+0000 7fbbb606f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbb0152840 0x7fbbb00a9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.337+0000 7fbbb606f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb00c75c0 con 0x7fbbb00c3970 2024-03-06T07:20:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.337+0000 7fbbb606f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb00c72e0 con 0x7fbbb00c60b0 2024-03-06T07:20:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.337+0000 7fbbb606f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb00c3d50 con 0x7fbbb0152840 2024-03-06T07:20:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.337+0000 7fbbaf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbb00c3970 0x7fbbb01ed9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.337+0000 7fbbaf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbb00c3970 0x7fbbb01ed9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34794/0 (socket says 172.21.15.1:34794) 2024-03-06T07:20:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.337+0000 7fbbaf7fe640 1 -- 172.21.15.1:0/2406561292 learned_addr learned my addr 172.21.15.1:0/2406561292 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.338+0000 7fbbaf7fe640 1 -- 172.21.15.1:0/2406561292 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbb0152840 msgr2=0x7fbbb00a9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.338+0000 7fbbaf7fe640 1 --2- 172.21.15.1:0/2406561292 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbb0152840 0x7fbbb00a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.338+0000 7fbbaf7fe640 1 -- 172.21.15.1:0/2406561292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbb00c60b0 msgr2=0x7fbbb01ea3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.338+0000 7fbbaeffd640 1 --2- 172.21.15.1:0/2406561292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbb00c60b0 0x7fbbb01ea3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.338+0000 7fbbaf7fe640 1 --2- 172.21.15.1:0/2406561292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbb00c60b0 0x7fbbb01ea3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.338+0000 7fbbaf7fe640 1 -- 172.21.15.1:0/2406561292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb01edf80 con 0x7fbbb00c3970 2024-03-06T07:20:28.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.338+0000 7fbbaf7fe640 1 --2- 172.21.15.1:0/2406561292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbb00c3970 0x7fbbb01ed9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbba000c7f0 tx=0x7fbba000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:28.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.338+0000 7fbb97fff640 1 -- 172.21.15.1:0/2406561292 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba0015070 con 0x7fbbb00c3970 2024-03-06T07:20:28.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.338+0000 7fbb97fff640 1 -- 172.21.15.1:0/2406561292 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba0004240 con 0x7fbbb00c3970 2024-03-06T07:20:28.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.338+0000 7fbbb606f640 1 -- 172.21.15.1:0/2406561292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb01ee260 con 0x7fbbb00c3970 2024-03-06T07:20:28.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.338+0000 7fbb97fff640 1 -- 172.21.15.1:0/2406561292 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba0007620 con 0x7fbbb00c3970 2024-03-06T07:20:28.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.338+0000 7fbbb606f640 1 -- 172.21.15.1:0/2406561292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb0156530 con 0x7fbbb00c3970 2024-03-06T07:20:28.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.339+0000 7fbbb606f640 1 -- 172.21.15.1:0/2406561292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb00a9ea0 con 0x7fbbb00c3970 2024-03-06T07:20:28.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.340+0000 7fbb97fff640 1 -- 172.21.15.1:0/2406561292 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbba00077c0 con 0x7fbbb00c3970 2024-03-06T07:20:28.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.340+0000 7fbb97fff640 1 --2- 172.21.15.1:0/2406561292 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb9c06b4d0 0x7fbb9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:28.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.340+0000 7fbb97fff640 1 -- 172.21.15.1:0/2406561292 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbba008a7b0 con 0x7fbbb00c3970 2024-03-06T07:20:28.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.340+0000 7fbbaeffd640 1 --2- 172.21.15.1:0/2406561292 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb9c06b4d0 0x7fbb9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:28.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.341+0000 7fbbaeffd640 1 --2- 172.21.15.1:0/2406561292 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb9c06b4d0 0x7fbb9c06d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbbb01519e0 tx=0x7fbba4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:28.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.344+0000 7fbb97fff640 1 -- 172.21.15.1:0/2406561292 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbba008ce20 con 0x7fbbb00c3970 2024-03-06T07:20:28.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.527+0000 7fbbb606f640 1 -- 172.21.15.1:0/2406561292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbb00686e0 con 0x7fbbb00c3970 2024-03-06T07:20:28.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.528+0000 7fbb97fff640 1 -- 172.21.15.1:0/2406561292 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7fbba00592f0 con 0x7fbbb00c3970 2024-03-06T07:20:28.530 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:28.531 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:28.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.530+0000 7fbbb606f640 1 -- 172.21.15.1:0/2406561292 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb9c06b4d0 msgr2=0x7fbb9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:28.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.530+0000 7fbbb606f640 1 --2- 172.21.15.1:0/2406561292 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb9c06b4d0 0x7fbb9c06d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbbb01519e0 tx=0x7fbba4009040 comp rx=0 tx=0).stop 2024-03-06T07:20:28.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.530+0000 7fbbb606f640 1 -- 172.21.15.1:0/2406561292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbb00c3970 msgr2=0x7fbbb01ed9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:28.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.530+0000 7fbbb606f640 1 --2- 172.21.15.1:0/2406561292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbb00c3970 0x7fbbb01ed9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbba000c7f0 tx=0x7fbba000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.530+0000 7fbbb606f640 1 -- 172.21.15.1:0/2406561292 shutdown_connections 2024-03-06T07:20:28.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.530+0000 7fbbb606f640 1 --2- 172.21.15.1:0/2406561292 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb9c06b4d0 0x7fbb9c06d990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.530+0000 7fbbb606f640 1 --2- 172.21.15.1:0/2406561292 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbb0152840 0x7fbbb00a9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.530+0000 7fbbb606f640 1 --2- 172.21.15.1:0/2406561292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbb00c60b0 0x7fbbb01ea3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.530+0000 7fbbb606f640 1 --2- 172.21.15.1:0/2406561292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbb00c3970 0x7fbbb01ed9a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.530+0000 7fbbb606f640 1 -- 172.21.15.1:0/2406561292 >> 172.21.15.1:0/2406561292 conn(0x7fbbb00c88f0 msgr2=0x7fbbb00aaff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:28.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.530+0000 7fbbb606f640 1 -- 172.21.15.1:0/2406561292 shutdown_connections 2024-03-06T07:20:28.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:28.530+0000 7fbbb606f640 1 -- 172.21.15.1:0/2406561292 wait complete. 2024-03-06T07:20:28.947 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:20:21.207834+0000', 'modified': '2024-03-06T07:20:26.238480+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4281}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4281': {'gid': 4281, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1315115982', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1315115982}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1315115982}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4281, 'qdb_cluster': [4281]} 2024-03-06T07:20:28.947 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:20:28.947 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:20:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.048+0000 7f7856d02640 1 Processor -- start 2024-03-06T07:20:29.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.049+0000 7f7856d02640 1 -- start start 2024-03-06T07:20:29.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.049+0000 7f7856d02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78500c32f0 0x7f78501543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.049+0000 7f7856d02640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7850070c80 0x7f78500710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.049+0000 7f7856d02640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7850150900 con 0x7f7850150520 2024-03-06T07:20:29.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.049+0000 7f7856d02640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7850153ed0 con 0x7f7850070c80 2024-03-06T07:20:29.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.049+0000 7f7856d02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7850071790 con 0x7f78500c32f0 2024-03-06T07:20:29.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.049+0000 7f7854a9e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7850150520 0x7f78501537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42994/0 (socket says 172.21.15.1:42994) 2024-03-06T07:20:29.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.049+0000 7f7854a9e640 1 -- 172.21.15.1:0/570662446 learned_addr learned my addr 172.21.15.1:0/570662446 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:29.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784ffff640 1 --2- 172.21.15.1:0/570662446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78500c32f0 0x7f78501543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784f7fe640 1 -- 172.21.15.1:0/570662446 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485439924 0 0) 0x7f7850150900 con 0x7f7850150520 2024-03-06T07:20:29.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f785529f640 1 --2- 172.21.15.1:0/570662446 >> v2:172.21.15.98:3300/0 conn(0x7f7850070c80 0x7f78500710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784f7fe640 1 -- 172.21.15.1:0/570662446 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f783c003520 con 0x7f7850150520 2024-03-06T07:20:29.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784f7fe640 1 -- 172.21.15.1:0/570662446 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2640343684 0 0) 0x7f783c003520 con 0x7f7850150520 2024-03-06T07:20:29.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784f7fe640 1 -- 172.21.15.1:0/570662446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78500c32f0 msgr2=0x7f78501543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:29.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784f7fe640 1 --2- 172.21.15.1:0/570662446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78500c32f0 0x7f78501543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784f7fe640 1 -- 172.21.15.1:0/570662446 >> v2:172.21.15.98:3300/0 conn(0x7f7850070c80 msgr2=0x7f78500710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:29.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784f7fe640 1 --2- 172.21.15.1:0/570662446 >> v2:172.21.15.98:3300/0 conn(0x7f7850070c80 0x7f78500710f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784f7fe640 1 -- 172.21.15.1:0/570662446 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7850157050 con 0x7f7850150520 2024-03-06T07:20:29.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784f7fe640 1 -- 172.21.15.1:0/570662446 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f78480019f0 con 0x7f7850150520 2024-03-06T07:20:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784f7fe640 1 -- 172.21.15.1:0/570662446 >> v1:172.21.15.1:6789/0 conn(0x7f7850150520 legacy=0x7f78501537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784f7fe640 1 --2- 172.21.15.1:0/570662446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783c0039b0 0x7f783c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784f7fe640 1 --2- 172.21.15.1:0/570662446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783c004800 0x7f783c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784f7fe640 1 --2- 172.21.15.1:0/570662446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78500c32f0 0x7f783c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784f7fe640 1 -- 172.21.15.1:0/570662446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7850153ed0 con 0x7f783c004800 2024-03-06T07:20:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784f7fe640 1 -- 172.21.15.1:0/570662446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7850071790 con 0x7f783c0039b0 2024-03-06T07:20:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784f7fe640 1 -- 172.21.15.1:0/570662446 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f783c00b9c0 con 0x7f78500c32f0 2024-03-06T07:20:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784ffff640 1 --2- 172.21.15.1:0/570662446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783c004800 0x7f783c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f7854a9e640 1 --2- 172.21.15.1:0/570662446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783c0039b0 0x7f783c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f785529f640 1 --2- 172.21.15.1:0/570662446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78500c32f0 0x7f783c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784ffff640 1 -- 172.21.15.1:0/570662446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78500c32f0 msgr2=0x7f783c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784ffff640 1 --2- 172.21.15.1:0/570662446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78500c32f0 0x7f783c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784ffff640 1 -- 172.21.15.1:0/570662446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783c0039b0 msgr2=0x7f783c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784ffff640 1 --2- 172.21.15.1:0/570662446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783c0039b0 0x7f783c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f784ffff640 1 -- 172.21.15.1:0/570662446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783c003520 con 0x7f783c004800 2024-03-06T07:20:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.050+0000 7f785529f640 1 --2- 172.21.15.1:0/570662446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78500c32f0 0x7f783c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.051+0000 7f784ffff640 1 --2- 172.21.15.1:0/570662446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783c004800 0x7f783c004c80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7838009ac0 tx=0x7f783c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0f2ff295efb354 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.051+0000 7f784f7fe640 1 -- 172.21.15.1:0/570662446 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7838039070 con 0x7f783c004800 2024-03-06T07:20:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.051+0000 7f784f7fe640 1 -- 172.21.15.1:0/570662446 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783802c6b0 con 0x7f783c004800 2024-03-06T07:20:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.051+0000 7f784f7fe640 1 -- 172.21.15.1:0/570662446 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7838009690 con 0x7f783c004800 2024-03-06T07:20:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.052+0000 7f7856d02640 1 -- 172.21.15.1:0/570662446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783c004800 msgr2=0x7f783c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.052+0000 7f7856d02640 1 --2- 172.21.15.1:0/570662446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783c004800 0x7f783c004c80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7838009ac0 tx=0x7f783c005830 comp rx=0 tx=0).stop 2024-03-06T07:20:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.052+0000 7f785529f640 1 -- 172.21.15.1:0/570662446 reap_dead start 2024-03-06T07:20:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.052+0000 7f7856d02640 1 -- 172.21.15.1:0/570662446 shutdown_connections 2024-03-06T07:20:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.052+0000 7f7856d02640 1 -- 172.21.15.1:0/570662446 >> 172.21.15.1:0/570662446 conn(0x7f78500c88f0 msgr2=0x7f78500c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.052+0000 7f7856d02640 1 -- 172.21.15.1:0/570662446 shutdown_connections 2024-03-06T07:20:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.052+0000 7f7856d02640 1 -- 172.21.15.1:0/570662446 wait complete. 2024-03-06T07:20:29.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.053+0000 7f7856d02640 1 Processor -- start 2024-03-06T07:20:29.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.053+0000 7f7856d02640 1 -- start start 2024-03-06T07:20:29.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f7856d02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783c004800 0x7f785014b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f7856d02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f785014b5f0 0x7f78501550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f7856d02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78501555e0 0x7f78501e8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f7856d02640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7850157330 con 0x7f783c004800 2024-03-06T07:20:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f7856d02640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78501532a0 con 0x7f78501555e0 2024-03-06T07:20:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f7856d02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78500c7ec0 con 0x7f785014b5f0 2024-03-06T07:20:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f7854a9e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783c004800 0x7f785014b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f7854a9e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783c004800 0x7f785014b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34806/0 (socket says 172.21.15.1:34806) 2024-03-06T07:20:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f7854a9e640 1 -- 172.21.15.1:0/231111741 learned_addr learned my addr 172.21.15.1:0/231111741 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f784ffff640 1 --2- 172.21.15.1:0/231111741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f785014b5f0 0x7f78501550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f785529f640 1 --2- 172.21.15.1:0/231111741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78501555e0 0x7f78501e8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f7854a9e640 1 -- 172.21.15.1:0/231111741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f785014b5f0 msgr2=0x7f78501550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f7854a9e640 1 --2- 172.21.15.1:0/231111741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f785014b5f0 0x7f78501550a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f7854a9e640 1 -- 172.21.15.1:0/231111741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78501555e0 msgr2=0x7f78501e8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f7854a9e640 1 --2- 172.21.15.1:0/231111741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78501555e0 0x7f78501e8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f7854a9e640 1 -- 172.21.15.1:0/231111741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785014c640 con 0x7f783c004800 2024-03-06T07:20:29.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f7854a9e640 1 --2- 172.21.15.1:0/231111741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783c004800 0x7f785014b0b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f78480079f0 tx=0x7f784800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:29.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f784d7fa640 1 -- 172.21.15.1:0/231111741 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78480133f0 con 0x7f783c004800 2024-03-06T07:20:29.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f784d7fa640 1 -- 172.21.15.1:0/231111741 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7848018070 con 0x7f783c004800 2024-03-06T07:20:29.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.054+0000 7f7856d02640 1 -- 172.21.15.1:0/231111741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f785014c080 con 0x7f783c004800 2024-03-06T07:20:29.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.055+0000 7f784d7fa640 1 -- 172.21.15.1:0/231111741 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78480135d0 con 0x7f783c004800 2024-03-06T07:20:29.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.055+0000 7f7856d02640 1 -- 172.21.15.1:0/231111741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78501e8ad0 con 0x7f783c004800 2024-03-06T07:20:29.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.056+0000 7f784d7fa640 1 -- 172.21.15.1:0/231111741 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7848013770 con 0x7f783c004800 2024-03-06T07:20:29.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.056+0000 7f784d7fa640 1 --2- 172.21.15.1:0/231111741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f783006b4d0 0x7f783006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.057+0000 7f784d7fa640 1 -- 172.21.15.1:0/231111741 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f784808abc0 con 0x7f783c004800 2024-03-06T07:20:29.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.057+0000 7f784ffff640 1 --2- 172.21.15.1:0/231111741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f783006b4d0 0x7f783006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.057+0000 7f784ffff640 1 --2- 172.21.15.1:0/231111741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f783006b4d0 0x7f783006d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7838036000 tx=0x7f7838030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:29.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.057+0000 7f78599ca740 1 Processor -- start 2024-03-06T07:20:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.057+0000 7f78599ca740 1 -- start start 2024-03-06T07:20:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.058+0000 7f78599ca740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ecf50 0x561fd95eaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.058+0000 7f78599ca740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd95eaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.058+0000 7f78599ca740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd95eb4b0 0x561fd95e7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.058+0000 7f78599ca740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd95516b0 con 0x561fd95ed330 2024-03-06T07:20:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.058+0000 7f78599ca740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd94faa60 con 0x561fd95eb4b0 2024-03-06T07:20:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.058+0000 7f78599ca740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x561fd954d750 con 0x561fd95ecf50 2024-03-06T07:20:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.058+0000 7f784ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd95eaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.058+0000 7f784ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd95eaf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34820/0 (socket says 172.21.15.1:34820) 2024-03-06T07:20:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.058+0000 7f784ffff640 1 -- 172.21.15.1:0/239086677 learned_addr learned my addr 172.21.15.1:0/239086677 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.058+0000 7f7854a9e640 1 --2- 172.21.15.1:0/239086677 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ecf50 0x561fd95eaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.058+0000 7f785529f640 1 --2- 172.21.15.1:0/239086677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd95eb4b0 0x561fd95e7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.058+0000 7f784ffff640 1 -- 172.21.15.1:0/239086677 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ecf50 msgr2=0x561fd95eaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.058+0000 7f784ffff640 1 --2- 172.21.15.1:0/239086677 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ecf50 0x561fd95eaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.058+0000 7f784ffff640 1 -- 172.21.15.1:0/239086677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd95eb4b0 msgr2=0x561fd95e7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.058+0000 7f784ffff640 1 --2- 172.21.15.1:0/239086677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd95eb4b0 0x561fd95e7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.059+0000 7f784ffff640 1 -- 172.21.15.1:0/239086677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561fd9518f30 con 0x561fd95ed330 2024-03-06T07:20:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.059+0000 7f784ffff640 1 --2- 172.21.15.1:0/239086677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd95eaf70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7838037040 tx=0x7f783802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e400345082d80cc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.059+0000 7f78467fc640 1 -- 172.21.15.1:0/239086677 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783804d070 con 0x561fd95ed330 2024-03-06T07:20:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.059+0000 7f78467fc640 1 -- 172.21.15.1:0/239086677 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783803a040 con 0x561fd95ed330 2024-03-06T07:20:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.059+0000 7f78467fc640 1 -- 172.21.15.1:0/239086677 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7838043040 con 0x561fd95ed330 2024-03-06T07:20:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.059+0000 7f78599ca740 1 -- 172.21.15.1:0/239086677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 msgr2=0x561fd95eaf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.059+0000 7f78599ca740 1 --2- 172.21.15.1:0/239086677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd95eaf70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7838037040 tx=0x7f783802e000 comp rx=0 tx=0).stop 2024-03-06T07:20:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.059+0000 7f78599ca740 1 -- 172.21.15.1:0/239086677 shutdown_connections 2024-03-06T07:20:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.059+0000 7f78599ca740 1 --2- 172.21.15.1:0/239086677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd95eb4b0 0x561fd95e7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.059+0000 7f78599ca740 1 --2- 172.21.15.1:0/239086677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd95eaf70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.059+0000 7f78599ca740 1 --2- 172.21.15.1:0/239086677 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ecf50 0x561fd95eaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.059+0000 7f78599ca740 1 -- 172.21.15.1:0/239086677 >> 172.21.15.1:0/239086677 conn(0x561fd95e97d0 msgr2=0x561fd960e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.059+0000 7f78599ca740 1 -- 172.21.15.1:0/239086677 shutdown_connections 2024-03-06T07:20:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.060+0000 7f78599ca740 1 -- 172.21.15.1:0/239086677 wait complete. 2024-03-06T07:20:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.060+0000 7f78599ca740 1 -- start start 2024-03-06T07:20:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.060+0000 7f78599ca740 1 Processor -- start 2024-03-06T07:20:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f78599ca740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ecf50 0x561fd960f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f78599ca740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd960d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f78599ca740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd963a320 0x561fd963cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f78599ca740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd960f230 con 0x561fd95ed330 2024-03-06T07:20:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f78599ca740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd950dea0 con 0x561fd963a320 2024-03-06T07:20:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f78599ca740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x561fd960ec70 con 0x561fd95ecf50 2024-03-06T07:20:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f78599ca740 10 client.0 _ll_register_callbacks cb 0x561fd950d870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:20:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f784ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd960d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f784ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd960d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34822/0 (socket says 172.21.15.1:34822) 2024-03-06T07:20:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f784ffff640 1 -- 172.21.15.1:0/586359693 learned_addr learned my addr 172.21.15.1:0/586359693 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f7854a9e640 1 --2- 172.21.15.1:0/586359693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ecf50 0x561fd960f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f785529f640 1 --2- 172.21.15.1:0/586359693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd963a320 0x561fd963cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f784ffff640 1 -- 172.21.15.1:0/586359693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ecf50 msgr2=0x561fd960f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f784ffff640 1 --2- 172.21.15.1:0/586359693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ecf50 0x561fd960f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f784ffff640 1 -- 172.21.15.1:0/586359693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd963a320 msgr2=0x561fd963cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f784ffff640 1 --2- 172.21.15.1:0/586359693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd963a320 0x561fd963cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f784ffff640 1 -- 172.21.15.1:0/586359693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561fd950e270 con 0x561fd95ed330 2024-03-06T07:20:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f784ffff640 1 --2- 172.21.15.1:0/586359693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd960d3b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7838004650 tx=0x7f78380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20aa6059208e0988 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.061+0000 7f782f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:20:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.062+0000 7f782f7fe640 1 -- 172.21.15.1:0/586359693 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7838057070 con 0x561fd95ed330 2024-03-06T07:20:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.062+0000 7f782f7fe640 1 -- 172.21.15.1:0/586359693 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f783802cb90 con 0x561fd95ed330 2024-03-06T07:20:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.062+0000 7f78599ca740 1 -- 172.21.15.1:0/586359693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f783803a040 con 0x561fd95ed330 2024-03-06T07:20:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.062+0000 7f782f7fe640 1 -- 172.21.15.1:0/586359693 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7838062050 con 0x561fd95ed330 2024-03-06T07:20:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.062+0000 7f78599ca740 10 client.4419 fetch_fsmap learned FSMap version 7 2024-03-06T07:20:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.062+0000 7f78599ca740 1 -- 172.21.15.1:0/586359693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x561fd92662d0 con 0x561fd95ed330 2024-03-06T07:20:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.062+0000 7f782f7fe640 1 -- 172.21.15.1:0/586359693 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783803a040 con 0x561fd95ed330 2024-03-06T07:20:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.062+0000 7f782f7fe640 1 -- 172.21.15.1:0/586359693 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 7) v1 ==== 3322+0+0 (secure 0 0 0) 0x7f7838065020 con 0x561fd95ed330 2024-03-06T07:20:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.062+0000 7f78599ca740 10 client.4419 fetch_fsmap finished waiting for FSMap version 7 2024-03-06T07:20:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.062+0000 7f78599ca740 10 client.4419 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:20:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.062+0000 7f78599ca740 20 client.4419 populate_metadata read hostname 'smithi001' 2024-03-06T07:20:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.062+0000 7f78599ca740 1 --2- 172.21.15.1:0/586359693 >> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] conn(0x561fd963d3c0 0x561fd9648330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.062+0000 7f78599ca740 4 client.4419 mds_command: new command op to 4281 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:20:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.062+0000 7f78599ca740 1 -- 172.21.15.1:0/586359693 --> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561fd955ebf0 con 0x561fd963d3c0 2024-03-06T07:20:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.062+0000 7f7854a9e640 1 --2- 172.21.15.1:0/586359693 >> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] conn(0x561fd963d3c0 0x561fd9648330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.063+0000 7f782f7fe640 10 client.4419 ms_handle_connect on v2:172.21.15.1:6833/1315115982 2024-03-06T07:20:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.063+0000 7f7854a9e640 1 --2- 172.21.15.1:0/586359693 >> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] conn(0x561fd963d3c0 0x561fd9648330 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c16fe9d6af05b269 server_cookie=ee6b04b02e593191 in_seq=0 out_seq=0 2024-03-06T07:20:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.063+0000 7f782f7fe640 1 -- 172.21.15.1:0/586359693 <== mds.0 v2:172.21.15.1:6833/1315115982 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561fd955ebf0 con 0x561fd963d3c0 2024-03-06T07:20:29.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.063+0000 7f782f7fe640 10 client.4419 handle_command_reply: tid=0 2024-03-06T07:20:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.063+0000 7f78599ca740 1 client.4419 shutdown 2024-03-06T07:20:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.063+0000 7f78599ca740 1 -- 172.21.15.1:0/586359693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 msgr2=0x561fd960d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.063+0000 7f78599ca740 1 --2- 172.21.15.1:0/586359693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd960d3b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7838004650 tx=0x7f78380046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.063+0000 7f78599ca740 1 -- 172.21.15.1:0/586359693 shutdown_connections 2024-03-06T07:20:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.063+0000 7f78599ca740 1 --2- 172.21.15.1:0/586359693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd963a320 0x561fd963cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.063+0000 7f78599ca740 1 --2- 172.21.15.1:0/586359693 >> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] conn(0x561fd963d3c0 0x561fd9648330 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.063+0000 7f78599ca740 1 --2- 172.21.15.1:0/586359693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd960d3b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.063+0000 7f78599ca740 1 --2- 172.21.15.1:0/586359693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ecf50 0x561fd960f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.063+0000 7f782f7fe640 0 client.4419 ms_handle_reset on v2:172.21.15.1:6833/1315115982 2024-03-06T07:20:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.063+0000 7f78599ca740 1 -- 172.21.15.1:0/586359693 >> 172.21.15.1:0/586359693 conn(0x561fd95eb4b0 msgr2=0x561fd95eb8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.063+0000 7f78599ca740 1 -- 172.21.15.1:0/586359693 shutdown_connections 2024-03-06T07:20:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.063+0000 7f78599ca740 1 -- 172.21.15.1:0/586359693 wait complete. 2024-03-06T07:20:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.064+0000 7f78599ca740 20 client.4419 trim_cache size 0 max 16384 2024-03-06T07:20:29.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.080+0000 7f78599ca740 1 Processor -- start 2024-03-06T07:20:29.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.080+0000 7f78599ca740 1 -- start start 2024-03-06T07:20:29.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.081+0000 7f78599ca740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ecf50 0x561fd960d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.081+0000 7f78599ca740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd9612a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.081+0000 7f78599ca740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd963a320 0x561fd9612f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.081+0000 7f78599ca740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78501f6040 con 0x561fd95ed330 2024-03-06T07:20:29.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.081+0000 7f78599ca740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78501eccd0 con 0x561fd963a320 2024-03-06T07:20:29.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.081+0000 7f78599ca740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78501f5a40 con 0x561fd95ecf50 2024-03-06T07:20:29.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.081+0000 7f784ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd9612a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.081+0000 7f784ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd9612a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34824/0 (socket says 172.21.15.1:34824) 2024-03-06T07:20:29.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.081+0000 7f784ffff640 1 -- 172.21.15.1:0/226820392 learned_addr learned my addr 172.21.15.1:0/226820392 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:29.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.081+0000 7f784ffff640 1 -- 172.21.15.1:0/226820392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ecf50 msgr2=0x561fd960d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:29.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.081+0000 7f784ffff640 1 --2- 172.21.15.1:0/226820392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ecf50 0x561fd960d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.081+0000 7f784ffff640 1 -- 172.21.15.1:0/226820392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd963a320 msgr2=0x561fd9612f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:29.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.081+0000 7f784ffff640 1 --2- 172.21.15.1:0/226820392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd963a320 0x561fd9612f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.081+0000 7f784ffff640 1 -- 172.21.15.1:0/226820392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561fd93306b0 con 0x561fd95ed330 2024-03-06T07:20:29.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.081+0000 7f784ffff640 1 --2- 172.21.15.1:0/226820392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd9612a30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f78380038b0 tx=0x7f7838050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c6aef33af51824e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:29.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.081+0000 7f7845ffb640 1 -- 172.21.15.1:0/226820392 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7838043070 con 0x561fd95ed330 2024-03-06T07:20:29.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.082+0000 7f7845ffb640 1 -- 172.21.15.1:0/226820392 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783802ca10 con 0x561fd95ed330 2024-03-06T07:20:29.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.082+0000 7f7845ffb640 1 -- 172.21.15.1:0/226820392 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7838057030 con 0x561fd95ed330 2024-03-06T07:20:29.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.082+0000 7f78599ca740 1 -- 172.21.15.1:0/226820392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 msgr2=0x561fd9612a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.082+0000 7f78599ca740 1 --2- 172.21.15.1:0/226820392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd9612a30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f78380038b0 tx=0x7f7838050000 comp rx=0 tx=0).stop 2024-03-06T07:20:29.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.082+0000 7f78599ca740 1 -- 172.21.15.1:0/226820392 shutdown_connections 2024-03-06T07:20:29.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.082+0000 7f78599ca740 1 --2- 172.21.15.1:0/226820392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd963a320 0x561fd9612f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.082+0000 7f78599ca740 1 --2- 172.21.15.1:0/226820392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ed330 0x561fd9612a30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.082+0000 7f78599ca740 1 --2- 172.21.15.1:0/226820392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ecf50 0x561fd960d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.082+0000 7f78599ca740 1 -- 172.21.15.1:0/226820392 >> 172.21.15.1:0/226820392 conn(0x561fd95eb4b0 msgr2=0x561fd95e6510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:29.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.082+0000 7f78599ca740 1 -- 172.21.15.1:0/226820392 shutdown_connections 2024-03-06T07:20:29.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.082+0000 7f78599ca740 1 -- 172.21.15.1:0/226820392 wait complete. 2024-03-06T07:20:29.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.083+0000 7f78599ca740 1 -- start start 2024-03-06T07:20:29.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.083+0000 7f78599ca740 1 Processor -- start 2024-03-06T07:20:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.083+0000 7f78599ca740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ecf50 0x561fd95ebc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.083+0000 7f78599ca740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ed330 0x561fd960c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.083+0000 7f78599ca740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd963a320 0x561fd9695ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.083+0000 7f78599ca740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd960f230 con 0x561fd95ecf50 2024-03-06T07:20:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.083+0000 7f78599ca740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd950dea0 con 0x561fd963a320 2024-03-06T07:20:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.083+0000 7f78599ca740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x561fd9637250 con 0x561fd95ed330 2024-03-06T07:20:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f78599ca740 10 client.0 _ll_register_callbacks cb 0x7f7848055350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:20:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f7854a9e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ecf50 0x561fd95ebc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f7854a9e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ecf50 0x561fd95ebc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34840/0 (socket says 172.21.15.1:34840) 2024-03-06T07:20:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f7854a9e640 1 -- 172.21.15.1:0/1239346838 learned_addr learned my addr 172.21.15.1:0/1239346838 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f784ffff640 1 --2- 172.21.15.1:0/1239346838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ed330 0x561fd960c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f785529f640 1 --2- 172.21.15.1:0/1239346838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd963a320 0x561fd9695ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f7854a9e640 1 -- 172.21.15.1:0/1239346838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ed330 msgr2=0x561fd960c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f7854a9e640 1 --2- 172.21.15.1:0/1239346838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ed330 0x561fd960c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f7854a9e640 1 -- 172.21.15.1:0/1239346838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd963a320 msgr2=0x561fd9695ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f7854a9e640 1 --2- 172.21.15.1:0/1239346838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd963a320 0x561fd9695ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f7854a9e640 1 -- 172.21.15.1:0/1239346838 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561fd960cf20 con 0x561fd95ecf50 2024-03-06T07:20:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f785529f640 1 --2- 172.21.15.1:0/1239346838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd963a320 0x561fd9695ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f7854a9e640 1 --2- 172.21.15.1:0/1239346838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ecf50 0x561fd95ebc90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7848055510 tx=0x7f78480617d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d413757d42c298ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f782f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:20:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f782f7fe640 1 -- 172.21.15.1:0/1239346838 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7848055720 con 0x561fd95ecf50 2024-03-06T07:20:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f782f7fe640 1 -- 172.21.15.1:0/1239346838 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7848061280 con 0x561fd95ecf50 2024-03-06T07:20:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f78599ca740 1 -- 172.21.15.1:0/1239346838 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f783802ca10 con 0x561fd95ecf50 2024-03-06T07:20:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f782f7fe640 1 -- 172.21.15.1:0/1239346838 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78480558c0 con 0x561fd95ecf50 2024-03-06T07:20:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f78599ca740 10 client.4425 fetch_fsmap learned FSMap version 7 2024-03-06T07:20:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f78599ca740 1 -- 172.21.15.1:0/1239346838 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x561fd95e7260 con 0x561fd95ecf50 2024-03-06T07:20:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.084+0000 7f782f7fe640 1 -- 172.21.15.1:0/1239346838 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783802ca10 con 0x561fd95ecf50 2024-03-06T07:20:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f782f7fe640 1 -- 172.21.15.1:0/1239346838 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 7) v1 ==== 3322+0+0 (secure 0 0 0) 0x7f7848058500 con 0x561fd95ecf50 2024-03-06T07:20:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f78599ca740 10 client.4425 fetch_fsmap finished waiting for FSMap version 7 2024-03-06T07:20:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f78599ca740 10 client.4425 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:20:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f78599ca740 20 client.4425 populate_metadata read hostname 'smithi001' 2024-03-06T07:20:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f78599ca740 1 --2- 172.21.15.1:0/1239346838 >> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] conn(0x561fd96252f0 0x561fd96256d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f78599ca740 4 client.4425 mds_command: new command op to 4281 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:20:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f78599ca740 1 -- 172.21.15.1:0/1239346838 --> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561fd9558f40 con 0x561fd96252f0 2024-03-06T07:20:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f784ffff640 1 --2- 172.21.15.1:0/1239346838 >> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] conn(0x561fd96252f0 0x561fd96256d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f784ffff640 1 --2- 172.21.15.1:0/1239346838 >> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] conn(0x561fd96252f0 0x561fd96256d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7b6a3fe10b92b0e server_cookie=61c5d2547dd1b676 in_seq=0 out_seq=0 2024-03-06T07:20:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f782f7fe640 10 client.4425 ms_handle_connect on v2:172.21.15.1:6833/1315115982 2024-03-06T07:20:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f782f7fe640 1 -- 172.21.15.1:0/1239346838 <== mds.0 v2:172.21.15.1:6833/1315115982 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x561fd9558f40 con 0x561fd96252f0 2024-03-06T07:20:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f782f7fe640 10 client.4425 handle_command_reply: tid=0 2024-03-06T07:20:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f78599ca740 1 client.4425 shutdown 2024-03-06T07:20:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f78599ca740 1 -- 172.21.15.1:0/1239346838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ecf50 msgr2=0x561fd95ebc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f78599ca740 1 --2- 172.21.15.1:0/1239346838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ecf50 0x561fd95ebc90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7848055510 tx=0x7f78480617d0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f78599ca740 1 -- 172.21.15.1:0/1239346838 shutdown_connections 2024-03-06T07:20:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f78599ca740 1 --2- 172.21.15.1:0/1239346838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fd963a320 0x561fd9695ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f78599ca740 1 --2- 172.21.15.1:0/1239346838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561fd95ed330 0x561fd960c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f78599ca740 1 --2- 172.21.15.1:0/1239346838 >> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] conn(0x561fd96252f0 0x561fd96256d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.085+0000 7f78599ca740 1 --2- 172.21.15.1:0/1239346838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561fd95ecf50 0x561fd95ebc90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.086+0000 7f782f7fe640 0 client.4425 ms_handle_reset on v2:172.21.15.1:6833/1315115982 2024-03-06T07:20:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.086+0000 7f78599ca740 1 -- 172.21.15.1:0/1239346838 >> 172.21.15.1:0/1239346838 conn(0x561fd95eb4b0 msgr2=0x561fd9626380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.086+0000 7f78599ca740 1 -- 172.21.15.1:0/1239346838 shutdown_connections 2024-03-06T07:20:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.086+0000 7f78599ca740 1 -- 172.21.15.1:0/1239346838 wait complete. 2024-03-06T07:20:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.086+0000 7f78599ca740 20 client.4425 trim_cache size 0 max 16384 2024-03-06T07:20:29.095 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:20:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:20:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:20:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4281, 2024-03-06T07:20:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:20:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:20:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:20:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 7.2500654129999997, 2024-03-06T07:20:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 7, 2024-03-06T07:20:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 25, 2024-03-06T07:20:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 24, 2024-03-06T07:20:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 13.896938835 2024-03-06T07:20:29.096 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:20:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.086+0000 7f7856d02640 1 -- 172.21.15.1:0/231111741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f783006b4d0 msgr2=0x7f783006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.086+0000 7f7856d02640 1 --2- 172.21.15.1:0/231111741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f783006b4d0 0x7f783006d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7838036000 tx=0x7f7838030000 comp rx=0 tx=0).stop 2024-03-06T07:20:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.086+0000 7f7856d02640 1 -- 172.21.15.1:0/231111741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783c004800 msgr2=0x7f785014b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.086+0000 7f7856d02640 1 --2- 172.21.15.1:0/231111741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783c004800 0x7f785014b0b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f78480079f0 tx=0x7f784800a660 comp rx=0 tx=0).stop 2024-03-06T07:20:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.086+0000 7f7856d02640 1 -- 172.21.15.1:0/231111741 shutdown_connections 2024-03-06T07:20:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.086+0000 7f7856d02640 1 --2- 172.21.15.1:0/231111741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f783006b4d0 0x7f783006d990 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.086+0000 7f7856d02640 1 --2- 172.21.15.1:0/231111741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78501555e0 0x7f78501e8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.086+0000 7f7856d02640 1 --2- 172.21.15.1:0/231111741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f785014b5f0 0x7f78501550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.086+0000 7f7856d02640 1 --2- 172.21.15.1:0/231111741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783c004800 0x7f785014b0b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.086+0000 7f7856d02640 1 -- 172.21.15.1:0/231111741 >> 172.21.15.1:0/231111741 conn(0x7f78500c88f0 msgr2=0x7f78500c36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.086+0000 7f7856d02640 1 -- 172.21.15.1:0/231111741 shutdown_connections 2024-03-06T07:20:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.086+0000 7f7856d02640 1 -- 172.21.15.1:0/231111741 wait complete. 2024-03-06T07:20:29.099 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:20:29.099 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-06T07:20:29.099 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:20:29.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.197+0000 7f23ff676640 1 Processor -- start 2024-03-06T07:20:29.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.198+0000 7f23ff676640 1 -- start start 2024-03-06T07:20:29.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.198+0000 7f23ff676640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23f80c8590 0x7f23f80c8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.198+0000 7f23ff676640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f23f80c3210 0x7f23f81564d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.198+0000 7f23ff676640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23f80c9010 con 0x7f23f8156a10 2024-03-06T07:20:29.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.198+0000 7f23ff676640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f8153a20 con 0x7f23f80c8590 2024-03-06T07:20:29.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.198+0000 7f23ff676640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f81530f0 con 0x7f23f80c3210 2024-03-06T07:20:29.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.198+0000 7f23fd412640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23f80c8590 0x7f23f80c8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.198+0000 7f23fd412640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23f80c8590 0x7f23f80c8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34850/0 (socket says 172.21.15.1:34850) 2024-03-06T07:20:29.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.199+0000 7f23fd412640 1 -- 172.21.15.1:0/3392741205 learned_addr learned my addr 172.21.15.1:0/3392741205 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:29.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.199+0000 7f23fcc11640 1 --2- 172.21.15.1:0/3392741205 >> v2:172.21.15.98:3300/0 conn(0x7f23f80c3210 0x7f23f81564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.199+0000 7f23fd412640 1 -- 172.21.15.1:0/3392741205 >> v2:172.21.15.98:3300/0 conn(0x7f23f80c3210 msgr2=0x7f23f81564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:29.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.199+0000 7f23fd412640 1 --2- 172.21.15.1:0/3392741205 >> v2:172.21.15.98:3300/0 conn(0x7f23f80c3210 0x7f23f81564d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.199+0000 7f23fd412640 1 -- 172.21.15.1:0/3392741205 >> v1:172.21.15.98:6789/0 conn(0x7f23f8156a10 legacy=0x7f23f80c6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:29.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.199+0000 7f23fd412640 1 -- 172.21.15.1:0/3392741205 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f81533d0 con 0x7f23f80c8590 2024-03-06T07:20:29.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.199+0000 7f23fd412640 1 --2- 172.21.15.1:0/3392741205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23f80c8590 0x7f23f80c8970 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f23f80c92f0 tx=0x7f23f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7fd7f39c765769 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:29.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.200+0000 7f23effff640 1 -- 172.21.15.1:0/3392741205 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4035020 con 0x7f23f80c8590 2024-03-06T07:20:29.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.200+0000 7f23effff640 1 -- 172.21.15.1:0/3392741205 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f4031870 con 0x7f23f80c8590 2024-03-06T07:20:29.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.200+0000 7f23effff640 1 -- 172.21.15.1:0/3392741205 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4031c40 con 0x7f23f80c8590 2024-03-06T07:20:29.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.200+0000 7f23ff676640 1 -- 172.21.15.1:0/3392741205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23f80c8590 msgr2=0x7f23f80c8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.200+0000 7f23ff676640 1 --2- 172.21.15.1:0/3392741205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23f80c8590 0x7f23f80c8970 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f23f80c92f0 tx=0x7f23f402b910 comp rx=0 tx=0).stop 2024-03-06T07:20:29.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.200+0000 7f23ff676640 1 -- 172.21.15.1:0/3392741205 shutdown_connections 2024-03-06T07:20:29.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.200+0000 7f23ff676640 1 --2- 172.21.15.1:0/3392741205 >> v2:172.21.15.98:3300/0 conn(0x7f23f80c3210 0x7f23f81564d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.200+0000 7f23ff676640 1 --2- 172.21.15.1:0/3392741205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23f80c8590 0x7f23f80c8970 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.200+0000 7f23ff676640 1 -- 172.21.15.1:0/3392741205 >> 172.21.15.1:0/3392741205 conn(0x7f23f80aa7a0 msgr2=0x7f23f80aab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:29.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.200+0000 7f23ff676640 1 -- 172.21.15.1:0/3392741205 shutdown_connections 2024-03-06T07:20:29.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.200+0000 7f23ff676640 1 -- 172.21.15.1:0/3392741205 wait complete. 2024-03-06T07:20:29.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.201+0000 7f23ff676640 1 Processor -- start 2024-03-06T07:20:29.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.201+0000 7f23ff676640 1 -- start start 2024-03-06T07:20:29.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.202+0000 7f23ff676640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23f80c3210 0x7f23f806d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.202+0000 7f23ff676640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23f80c8590 0x7f23f8155d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.202+0000 7f23ff676640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23f8156a10 0x7f23f814e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.202+0000 7f23ff676640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f80701e0 con 0x7f23f80c3210 2024-03-06T07:20:29.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.202+0000 7f23ff676640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f8070060 con 0x7f23f80c8590 2024-03-06T07:20:29.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.202+0000 7f23ff676640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f80c3650 con 0x7f23f8156a10 2024-03-06T07:20:29.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.202+0000 7f23fd412640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23f80c3210 0x7f23f806d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.202+0000 7f23fd412640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23f80c3210 0x7f23f806d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34852/0 (socket says 172.21.15.1:34852) 2024-03-06T07:20:29.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.202+0000 7f23fd412640 1 -- 172.21.15.1:0/2231564130 learned_addr learned my addr 172.21.15.1:0/2231564130 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:29.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.202+0000 7f23fdc13640 1 --2- 172.21.15.1:0/2231564130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23f8156a10 0x7f23f814e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.202+0000 7f23fcc11640 1 --2- 172.21.15.1:0/2231564130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23f80c8590 0x7f23f8155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.202+0000 7f23fd412640 1 -- 172.21.15.1:0/2231564130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23f8156a10 msgr2=0x7f23f814e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.202+0000 7f23fd412640 1 --2- 172.21.15.1:0/2231564130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23f8156a10 0x7f23f814e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.202+0000 7f23fd412640 1 -- 172.21.15.1:0/2231564130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23f80c8590 msgr2=0x7f23f8155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.202+0000 7f23fd412640 1 --2- 172.21.15.1:0/2231564130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23f80c8590 0x7f23f8155d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.202+0000 7f23fd412640 1 -- 172.21.15.1:0/2231564130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f814dd00 con 0x7f23f80c3210 2024-03-06T07:20:29.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.202+0000 7f23fcc11640 1 --2- 172.21.15.1:0/2231564130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23f80c8590 0x7f23f8155d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:29.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.203+0000 7f23fd412640 1 --2- 172.21.15.1:0/2231564130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23f80c3210 0x7f23f806d070 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f23f4000c00 tx=0x7f23f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:29.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.203+0000 7f23edffb640 1 -- 172.21.15.1:0/2231564130 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4040070 con 0x7f23f80c3210 2024-03-06T07:20:29.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.203+0000 7f23edffb640 1 -- 172.21.15.1:0/2231564130 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f4030d00 con 0x7f23f80c3210 2024-03-06T07:20:29.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.203+0000 7f23ff676640 1 -- 172.21.15.1:0/2231564130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f8150580 con 0x7f23f80c3210 2024-03-06T07:20:29.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.203+0000 7f23edffb640 1 -- 172.21.15.1:0/2231564130 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4035040 con 0x7f23f80c3210 2024-03-06T07:20:29.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.203+0000 7f23ff676640 1 -- 172.21.15.1:0/2231564130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f81566c0 con 0x7f23f80c3210 2024-03-06T07:20:29.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.204+0000 7f23ff676640 1 -- 172.21.15.1:0/2231564130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f80686e0 con 0x7f23f80c3210 2024-03-06T07:20:29.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.205+0000 7f23edffb640 1 -- 172.21.15.1:0/2231564130 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f23f404b060 con 0x7f23f80c3210 2024-03-06T07:20:29.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.205+0000 7f23edffb640 1 --2- 172.21.15.1:0/2231564130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f23dc06b4d0 0x7f23dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.205+0000 7f23edffb640 1 -- 172.21.15.1:0/2231564130 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f23f40ac200 con 0x7f23f80c3210 2024-03-06T07:20:29.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.205+0000 7f23fcc11640 1 --2- 172.21.15.1:0/2231564130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f23dc06b4d0 0x7f23dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.206+0000 7f23fcc11640 1 --2- 172.21.15.1:0/2231564130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f23dc06b4d0 0x7f23dc06d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f23e8006fd0 tx=0x7f23e8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:29.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.210+0000 7f23edffb640 1 -- 172.21.15.1:0/2231564130 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23f407add0 con 0x7f23f80c3210 2024-03-06T07:20:29.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.430+0000 7f23ff676640 1 -- 172.21.15.1:0/2231564130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f23f80c8370 con 0x7f23f80c3210 2024-03-06T07:20:29.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.430+0000 7f23edffb640 1 -- 172.21.15.1:0/2231564130 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7f23f407a6d0 con 0x7f23f80c3210 2024-03-06T07:20:29.433 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-06T07:20:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.433+0000 7f23ff676640 1 -- 172.21.15.1:0/2231564130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f23dc06b4d0 msgr2=0x7f23dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.433+0000 7f23ff676640 1 --2- 172.21.15.1:0/2231564130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f23dc06b4d0 0x7f23dc06d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f23e8006fd0 tx=0x7f23e8008040 comp rx=0 tx=0).stop 2024-03-06T07:20:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.433+0000 7f23ff676640 1 -- 172.21.15.1:0/2231564130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23f80c3210 msgr2=0x7f23f806d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.433+0000 7f23ff676640 1 --2- 172.21.15.1:0/2231564130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23f80c3210 0x7f23f806d070 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f23f4000c00 tx=0x7f23f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.433+0000 7f23ff676640 1 -- 172.21.15.1:0/2231564130 shutdown_connections 2024-03-06T07:20:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.433+0000 7f23ff676640 1 --2- 172.21.15.1:0/2231564130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f23dc06b4d0 0x7f23dc06d990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.433+0000 7f23ff676640 1 --2- 172.21.15.1:0/2231564130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23f8156a10 0x7f23f814e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.433+0000 7f23ff676640 1 --2- 172.21.15.1:0/2231564130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23f80c8590 0x7f23f8155d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.433+0000 7f23ff676640 1 --2- 172.21.15.1:0/2231564130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23f80c3210 0x7f23f806d070 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.433+0000 7f23ff676640 1 -- 172.21.15.1:0/2231564130 >> 172.21.15.1:0/2231564130 conn(0x7f23f80aa7a0 msgr2=0x7f23f80c3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.433+0000 7f23ff676640 1 -- 172.21.15.1:0/2231564130 shutdown_connections 2024-03-06T07:20:29.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:29.433+0000 7f23ff676640 1 -- 172.21.15.1:0/2231564130 wait complete. 2024-03-06T07:20:29.449 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:20:29.449 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:20:29.449 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-06T07:20:29.449 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-06T07:20:29.449 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-06T07:20:29.450 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:20:29.450 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:20:29.450 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:20:29.450 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-06T07:20:29.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:29.450 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-06T07:20:29.465 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:20:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:20:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:20:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:20:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-06T07:20:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:20:29.465 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:20:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:20:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-06T07:20:29.465 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:20:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:20:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-06T07:20:29.466 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:20:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:20:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-06T07:20:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:20:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 42611sec preferred_lft 42611sec 2024-03-06T07:20:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-03-06T07:20:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:20:29.466 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:20:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:20:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-06T07:20:29.467 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:20:29.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:29.467 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-06T07:20:29.467 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-06T07:20:29.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:20:29.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr flush dev ceph-brx 2024-03-06T07:20:29.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set ceph-brx up 2024-03-06T07:20:29.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:20:29.467 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-06T07:20:29.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:29.624 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:20:29.650 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-06T07:20:29.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:29.652 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-06T07:20:29.721 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-06T07:20:29.721 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:20:29.721 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:20:29.721 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:20:29.721 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:20:29.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:29.722 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-06T07:20:29.722 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-06T07:20:29.722 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:20:29.722 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:20:29.722 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:20:29.722 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-06T07:20:29.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:29.972 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-06T07:20:29.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:29.987 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list-id 2024-03-06T07:20:30.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:30.042 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-06T07:20:30.042 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-06T07:20:30.042 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-06T07:20:30.042 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-06T07:20:30.042 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-06T07:20:30.124 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-06T07:20:30.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:30.124 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-06T07:20:30.124 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-06T07:20:30.124 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-06T07:20:30.124 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:20:30.124 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-06T07:20:30.125 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-06T07:20:30.125 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-06T07:20:30.125 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-06T07:20:30.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:30.262 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-06T07:20:30.262 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-06T07:20:30.262 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-06T07:20:30.262 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:20:30.262 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-06T07:20:30.305 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-06T07:20:30.306 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:20:30.306 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-06T07:20:30.357 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-06T07:20:30.358 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:20:30.358 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-06T07:20:30.411 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-06T07:20:30.478 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:20:30.533 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-06T07:20:30.534 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-06T07:20:30.534 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-06T07:20:30.534 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-06T07:20:30.534 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-06T07:20:30.534 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-06T07:20:30.534 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-06T07:20:30.534 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-06T07:20:30.534 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-06T07:20:30.534 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-06T07:20:30.534 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-06T07:20:30.534 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-06T07:20:30.534 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:20:30.534 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-06T07:20:30.535 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-06T07:20:30.535 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-06T07:20:30.535 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-06T07:20:30.535 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:20:30.535 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-06T07:20:30.535 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-06T07:20:30.535 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:20:30.535 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:20:30.535 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:20:30.535 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-06T07:20:30.535 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-06T07:20:30.535 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:20:30.535 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:20:30.535 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:20:30.535 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:20:30.536 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-06T07:20:30.536 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-06T07:20:30.536 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-06T07:20:30.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:30.536 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-06T07:20:30.565 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:30.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:30.588 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:20:30.588 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-06T07:20:30.630 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-06T07:20:30.657 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:20:30.715 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-06T07:20:30.715 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-06T07:20:30.715 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-06T07:20:30.715 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-06T07:20:30.715 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-06T07:20:30.715 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-06T07:20:30.715 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-06T07:20:30.715 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-06T07:20:30.715 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-06T07:20:30.716 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-06T07:20:30.716 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-06T07:20:30.716 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-06T07:20:30.716 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:20:30.716 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-06T07:20:30.716 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-06T07:20:30.716 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-06T07:20:30.716 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-06T07:20:30.716 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:20:30.716 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-06T07:20:30.716 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-06T07:20:30.716 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:20:30.716 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:20:30.716 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:20:30.717 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-06T07:20:30.717 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-06T07:20:30.717 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:20:30.717 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:20:30.717 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:20:30.717 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:20:30.717 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-06T07:20:30.717 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-06T07:20:30.717 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-06T07:20:30.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:30.718 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-06T07:20:30.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi001.stderr:2024-03-06T07:20:30.716+0000 7f54e4cfa480 -1 init, newargv = 0x5652f13b87f0 newargc=15 2024-03-06T07:20:30.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi001.stderr:ceph-fuse[43211]: starting ceph client 2024-03-06T07:20:30.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi001.stderr:ceph-fuse[43211]: starting fuse 2024-03-06T07:20:30.733 INFO:teuthology.orchestra.run.smithi001.stdout:43 2024-03-06T07:20:30.734 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-06T07:20:30.734 DEBUG:teuthology.orchestra.run.smithi001:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:20:30.734 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-06T07:20:30.734 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-06T07:20:30.734 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-06T07:20:30.734 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-06T07:20:30.734 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-06T07:20:30.734 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-06T07:20:30.734 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-06T07:20:30.734 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-06T07:20:30.734 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-06T07:20:30.735 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-06T07:20:30.735 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-06T07:20:30.735 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-06T07:20:30.735 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-06T07:20:30.735 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-06T07:20:30.735 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-06T07:20:30.735 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-06T07:20:30.735 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:20:30.735 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:20:30.736 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:20:30.736 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-06T07:20:30.736 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-06T07:20:30.736 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-06T07:20:30.736 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:20:30.736 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-06T07:20:30.736 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.admin")) 2024-03-06T07:20:30.736 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-06T07:20:30.854 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.admin.43211.asok 2024-03-06T07:20:30.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:20:30.864 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.43211.asok 2024-03-06T07:20:30.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:30.864 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.43211.asok status 2024-03-06T07:20:31.001 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:20:31.002 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-06T07:20:31.002 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:20:31.002 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:20:31.002 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "admin", 2024-03-06T07:20:31.002 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-06T07:20:31.002 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-06T07:20:31.002 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "43211", 2024-03-06T07:20:31.002 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-06T07:20:31.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:20:31.002 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-06T07:20:31.002 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-06T07:20:31.002 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4440, 2024-03-06T07:20:31.003 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-06T07:20:31.003 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-06T07:20:31.003 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-06T07:20:31.003 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4440 2024-03-06T07:20:31.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:20:31.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-06T07:20:31.003 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-06T07:20:31.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:20:31.003 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1509479769 2024-03-06T07:20:31.003 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:20:31.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:20:31.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-06T07:20:31.003 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-06T07:20:31.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:20:31.004 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1509479769 2024-03-06T07:20:31.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:20:31.004 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.4440 192.168.0.1:0/1509479769", 2024-03-06T07:20:31.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/1509479769", 2024-03-06T07:20:31.004 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-06T07:20:31.004 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 7, 2024-03-06T07:20:31.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 25, 2024-03-06T07:20:31.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:20:31.004 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-06T07:20:31.004 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-06T07:20:31.004 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:20:31.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:31.010 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-06T07:20:31.010 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-06T07:20:31.024 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-06T07:20:31.025 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-06T07:20:31.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:31.025 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-06T07:20:31.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:31.095 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-06T07:20:31.178 INFO:teuthology.orchestra.run.smithi001.stdout:# file: . 2024-03-06T07:20:31.178 INFO:teuthology.orchestra.run.smithi001.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-06T07:20:31.178 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:31.178 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:20:31.178 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:20:31.232 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:31.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.398+0000 7f96d6a40640 1 Processor -- start 2024-03-06T07:20:31.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.398+0000 7f96d6a40640 1 -- start start 2024-03-06T07:20:31.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.399+0000 7f96d6a40640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f96d0152650 0x7f96d0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.399+0000 7f96d6a40640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d00c2fc0 con 0x7f96d00c6b10 2024-03-06T07:20:31.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.399+0000 7f96d6a40640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d0158da0 con 0x7f96d00bf9d0 2024-03-06T07:20:31.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.399+0000 7f96d6a40640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96d0155800 con 0x7f96d0152650 2024-03-06T07:20:31.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.399+0000 7f96cffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f96d0152650 0x7f96d0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.399+0000 7f96cf7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f96d00c6b10 0x7f96d0158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42996/0 (socket says 172.21.15.1:42996) 2024-03-06T07:20:31.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.399+0000 7f96cf7fe640 1 -- 172.21.15.1:0/1374356201 learned_addr learned my addr 172.21.15.1:0/1374356201 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:31.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.399+0000 7f96ceffd640 1 -- 172.21.15.1:0/1374356201 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560257430 0 0) 0x7f96d00c2fc0 con 0x7f96d00c6b10 2024-03-06T07:20:31.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.399+0000 7f96ceffd640 1 -- 172.21.15.1:0/1374356201 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96b8003520 con 0x7f96d00c6b10 2024-03-06T07:20:31.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.399+0000 7f96cffff640 1 -- 172.21.15.1:0/1374356201 >> v1:172.21.15.98:6789/0 conn(0x7f96d00bf9d0 legacy=0x7f96d00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.399+0000 7f96cffff640 1 -- 172.21.15.1:0/1374356201 >> v1:172.21.15.1:6789/0 conn(0x7f96d00c6b10 legacy=0x7f96d0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.399+0000 7f96cffff640 1 -- 172.21.15.1:0/1374356201 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d00c6690 con 0x7f96d0152650 2024-03-06T07:20:31.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.400+0000 7f96cffff640 1 --2- 172.21.15.1:0/1374356201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96d0152650 0x7f96d0155c90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f96c40097a0 tx=0x7f96c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75ca9f0cbc1e9e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:31.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.400+0000 7f96ceffd640 1 -- 172.21.15.1:0/1374356201 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4035020 con 0x7f96d0152650 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.400+0000 7f96ceffd640 1 -- 172.21.15.1:0/1374356201 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c4031830 con 0x7f96d0152650 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.400+0000 7f96ceffd640 1 -- 172.21.15.1:0/1374356201 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4031be0 con 0x7f96d0152650 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.401+0000 7f96d6a40640 1 -- 172.21.15.1:0/1374356201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96d0152650 msgr2=0x7f96d0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.401+0000 7f96d6a40640 1 --2- 172.21.15.1:0/1374356201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96d0152650 0x7f96d0155c90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f96c40097a0 tx=0x7f96c402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.401+0000 7f96d6a40640 1 -- 172.21.15.1:0/1374356201 shutdown_connections 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.401+0000 7f96d6a40640 1 --2- 172.21.15.1:0/1374356201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96d0152650 0x7f96d0155c90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.401+0000 7f96d6a40640 1 -- 172.21.15.1:0/1374356201 >> 172.21.15.1:0/1374356201 conn(0x7f96d0066ee0 msgr2=0x7f96d00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.401+0000 7f96d6a40640 1 -- 172.21.15.1:0/1374356201 shutdown_connections 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.401+0000 7f96d6a40640 1 -- 172.21.15.1:0/1374356201 wait complete. 2024-03-06T07:20:31.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.402+0000 7f96d6a40640 1 Processor -- start 2024-03-06T07:20:31.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.402+0000 7f96d6a40640 1 -- start start 2024-03-06T07:20:31.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.403+0000 7f96d6a40640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96d00bf9d0 0x7f96d014cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.403+0000 7f96d6a40640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96d0152650 0x7f96d014d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.403+0000 7f96d6a40640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96d0156490 0x7f96d01e1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.403+0000 7f96d6a40640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d01551b0 con 0x7f96d00bf9d0 2024-03-06T07:20:31.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.403+0000 7f96d6a40640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d00c6fa0 con 0x7f96d0152650 2024-03-06T07:20:31.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.403+0000 7f96d6a40640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96d0155490 con 0x7f96d0156490 2024-03-06T07:20:31.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.403+0000 7f96cffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96d00bf9d0 0x7f96d014cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.403+0000 7f96cffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96d00bf9d0 0x7f96d014cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34880/0 (socket says 172.21.15.1:34880) 2024-03-06T07:20:31.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.403+0000 7f96cffff640 1 -- 172.21.15.1:0/1049845354 learned_addr learned my addr 172.21.15.1:0/1049845354 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:31.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.403+0000 7f96cf7fe640 1 --2- 172.21.15.1:0/1049845354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96d0152650 0x7f96d014d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.403+0000 7f96d4fdd640 1 --2- 172.21.15.1:0/1049845354 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96d0156490 0x7f96d01e1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.403+0000 7f96cffff640 1 -- 172.21.15.1:0/1049845354 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96d0156490 msgr2=0x7f96d01e1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.403+0000 7f96cffff640 1 --2- 172.21.15.1:0/1049845354 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96d0156490 0x7f96d01e1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.403+0000 7f96cffff640 1 -- 172.21.15.1:0/1049845354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96d0152650 msgr2=0x7f96d014d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.403+0000 7f96cffff640 1 --2- 172.21.15.1:0/1049845354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96d0152650 0x7f96d014d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.403+0000 7f96cffff640 1 -- 172.21.15.1:0/1049845354 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d014da40 con 0x7f96d00bf9d0 2024-03-06T07:20:31.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.403+0000 7f96cf7fe640 1 --2- 172.21.15.1:0/1049845354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96d0152650 0x7f96d014d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:31.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.404+0000 7f96cffff640 1 --2- 172.21.15.1:0/1049845354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96d00bf9d0 0x7f96d014cfc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f96c40097a0 tx=0x7f96c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:31.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.404+0000 7f96ccff9640 1 -- 172.21.15.1:0/1049845354 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4035070 con 0x7f96d00bf9d0 2024-03-06T07:20:31.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.404+0000 7f96ccff9640 1 -- 172.21.15.1:0/1049845354 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c4004230 con 0x7f96d00bf9d0 2024-03-06T07:20:31.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.404+0000 7f96ccff9640 1 -- 172.21.15.1:0/1049845354 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c403a630 con 0x7f96d00bf9d0 2024-03-06T07:20:31.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.404+0000 7f96d6a40640 1 -- 172.21.15.1:0/1049845354 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d01df480 con 0x7f96d00bf9d0 2024-03-06T07:20:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.404+0000 7f96d6a40640 1 -- 172.21.15.1:0/1049845354 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d0157500 con 0x7f96d00bf9d0 2024-03-06T07:20:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.405+0000 7f96d6a40640 1 -- 172.21.15.1:0/1049845354 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d00c2f50 con 0x7f96d00bf9d0 2024-03-06T07:20:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.405+0000 7f96ccff9640 1 -- 172.21.15.1:0/1049845354 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96c4038020 con 0x7f96d00bf9d0 2024-03-06T07:20:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.406+0000 7f96ccff9640 1 --2- 172.21.15.1:0/1049845354 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f96bc06b4d0 0x7f96bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.406+0000 7f96ccff9640 1 -- 172.21.15.1:0/1049845354 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f96c40ac0b0 con 0x7f96d00bf9d0 2024-03-06T07:20:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.406+0000 7f96cf7fe640 1 --2- 172.21.15.1:0/1049845354 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f96bc06b4d0 0x7f96bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.406+0000 7f96cf7fe640 1 --2- 172.21.15.1:0/1049845354 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f96bc06b4d0 0x7f96bc06d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f96d0068950 tx=0x7f96c0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.410+0000 7f96ccff9640 1 -- 172.21.15.1:0/1049845354 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96d00c2f50 con 0x7f96d00bf9d0 2024-03-06T07:20:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.674+0000 7f96d6a40640 1 -- 172.21.15.1:0/1049845354 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96d0067fd0 con 0x7f96d00bf9d0 2024-03-06T07:20:31.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.675+0000 7f96ccff9640 1 -- 172.21.15.1:0/1049845354 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7f96c4040070 con 0x7f96d00bf9d0 2024-03-06T07:20:31.678 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:31.678 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:31.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.677+0000 7f96d6a40640 1 -- 172.21.15.1:0/1049845354 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f96bc06b4d0 msgr2=0x7f96bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.677+0000 7f96d6a40640 1 --2- 172.21.15.1:0/1049845354 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f96bc06b4d0 0x7f96bc06d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f96d0068950 tx=0x7f96c0009040 comp rx=0 tx=0).stop 2024-03-06T07:20:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.677+0000 7f96d6a40640 1 -- 172.21.15.1:0/1049845354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96d00bf9d0 msgr2=0x7f96d014cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.677+0000 7f96d6a40640 1 --2- 172.21.15.1:0/1049845354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96d00bf9d0 0x7f96d014cfc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f96c40097a0 tx=0x7f96c40046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.677+0000 7f96d6a40640 1 -- 172.21.15.1:0/1049845354 shutdown_connections 2024-03-06T07:20:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.677+0000 7f96d6a40640 1 --2- 172.21.15.1:0/1049845354 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f96bc06b4d0 0x7f96bc06d990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.677+0000 7f96d6a40640 1 --2- 172.21.15.1:0/1049845354 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96d0156490 0x7f96d01e1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.677+0000 7f96d6a40640 1 --2- 172.21.15.1:0/1049845354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96d0152650 0x7f96d014d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.677+0000 7f96d6a40640 1 --2- 172.21.15.1:0/1049845354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96d00bf9d0 0x7f96d014cfc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.677+0000 7f96d6a40640 1 -- 172.21.15.1:0/1049845354 >> 172.21.15.1:0/1049845354 conn(0x7f96d0066ee0 msgr2=0x7f96d00c04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.677+0000 7f96d6a40640 1 -- 172.21.15.1:0/1049845354 shutdown_connections 2024-03-06T07:20:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.677+0000 7f96d6a40640 1 -- 172.21.15.1:0/1049845354 wait complete. 2024-03-06T07:20:31.694 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:20:19.337560+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:20:20.172791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:20:23.884071+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:20:31.695 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:20:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.800+0000 7f197501c640 1 Processor -- start 2024-03-06T07:20:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.800+0000 7f197501c640 1 -- start start 2024-03-06T07:20:31.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.801+0000 7f197501c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19700c6b10 0x7f1970155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.801+0000 7f197501c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f19700bf9d0 0x7f19700c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.801+0000 7f197501c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19700c2fc0 con 0x7f1970152650 2024-03-06T07:20:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.801+0000 7f197501c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1970155800 con 0x7f19700bf9d0 2024-03-06T07:20:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.801+0000 7f197501c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1970158da0 con 0x7f19700c6b10 2024-03-06T07:20:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.801+0000 7f196f59e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f19700bf9d0 0x7f19700c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.801+0000 7f196f59e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f19700bf9d0 0x7f19700c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34894/0 (socket says 172.21.15.1:34894) 2024-03-06T07:20:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.801+0000 7f196f59e640 1 -- 172.21.15.1:0/3722205389 learned_addr learned my addr 172.21.15.1:0/3722205389 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.801+0000 7f196e59c640 1 --2- 172.21.15.1:0/3722205389 >> v2:172.21.15.98:3300/0 conn(0x7f19700c6b10 0x7f1970155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.801+0000 7f196f59e640 1 -- 172.21.15.1:0/3722205389 >> v2:172.21.15.98:3300/0 conn(0x7f19700c6b10 msgr2=0x7f1970155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.801+0000 7f196f59e640 1 --2- 172.21.15.1:0/3722205389 >> v2:172.21.15.98:3300/0 conn(0x7f19700c6b10 0x7f1970155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.801+0000 7f196f59e640 1 -- 172.21.15.1:0/3722205389 >> v1:172.21.15.98:6789/0 conn(0x7f1970152650 legacy=0x7f1970158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.801+0000 7f196f59e640 1 -- 172.21.15.1:0/3722205389 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19700c6690 con 0x7f19700bf9d0 2024-03-06T07:20:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.802+0000 7f196f59e640 1 --2- 172.21.15.1:0/3722205389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19700bf9d0 0x7f19700c1460 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f19700cb8e0 tx=0x7f196802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7901a58f52340e59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.802+0000 7f196dd9b640 1 -- 172.21.15.1:0/3722205389 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196802ca80 con 0x7f19700bf9d0 2024-03-06T07:20:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.802+0000 7f196dd9b640 1 -- 172.21.15.1:0/3722205389 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196802cc20 con 0x7f19700bf9d0 2024-03-06T07:20:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.802+0000 7f196dd9b640 1 -- 172.21.15.1:0/3722205389 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19680343f0 con 0x7f19700bf9d0 2024-03-06T07:20:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.803+0000 7f197501c640 1 -- 172.21.15.1:0/3722205389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19700bf9d0 msgr2=0x7f19700c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.803+0000 7f197501c640 1 --2- 172.21.15.1:0/3722205389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19700bf9d0 0x7f19700c1460 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f19700cb8e0 tx=0x7f196802bee0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.803+0000 7f197501c640 1 -- 172.21.15.1:0/3722205389 shutdown_connections 2024-03-06T07:20:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.803+0000 7f197501c640 1 --2- 172.21.15.1:0/3722205389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19700bf9d0 0x7f19700c1460 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.803+0000 7f197501c640 1 --2- 172.21.15.1:0/3722205389 >> v2:172.21.15.98:3300/0 conn(0x7f19700c6b10 0x7f1970155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.803+0000 7f197501c640 1 -- 172.21.15.1:0/3722205389 >> 172.21.15.1:0/3722205389 conn(0x7f1970066ee0 msgr2=0x7f19700672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:31.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.803+0000 7f197501c640 1 -- 172.21.15.1:0/3722205389 shutdown_connections 2024-03-06T07:20:31.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.803+0000 7f197501c640 1 -- 172.21.15.1:0/3722205389 wait complete. 2024-03-06T07:20:31.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.804+0000 7f197501c640 1 Processor -- start 2024-03-06T07:20:31.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.804+0000 7f197501c640 1 -- start start 2024-03-06T07:20:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.805+0000 7f197501c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19700bf9d0 0x7f197014e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.805+0000 7f197501c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1970152650 0x7f19701eb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.805+0000 7f197501c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1970151250 0x7f19701eb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.805+0000 7f197501c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19701551b0 con 0x7f1970151250 2024-03-06T07:20:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.805+0000 7f197501c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19700c6fa0 con 0x7f19700bf9d0 2024-03-06T07:20:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.805+0000 7f197501c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1970155490 con 0x7f1970152650 2024-03-06T07:20:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.805+0000 7f196f59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1970151250 0x7f19701eb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.805+0000 7f196f59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1970151250 0x7f19701eb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34904/0 (socket says 172.21.15.1:34904) 2024-03-06T07:20:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.805+0000 7f196f59e640 1 -- 172.21.15.1:0/400702011 learned_addr learned my addr 172.21.15.1:0/400702011 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.805+0000 7f196ed9d640 1 --2- 172.21.15.1:0/400702011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19700bf9d0 0x7f197014e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.805+0000 7f196e59c640 1 --2- 172.21.15.1:0/400702011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1970152650 0x7f19701eb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.805+0000 7f196f59e640 1 -- 172.21.15.1:0/400702011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1970152650 msgr2=0x7f19701eb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.805+0000 7f196f59e640 1 --2- 172.21.15.1:0/400702011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1970152650 0x7f19701eb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.805+0000 7f196f59e640 1 -- 172.21.15.1:0/400702011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19700bf9d0 msgr2=0x7f197014e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.805+0000 7f196f59e640 1 --2- 172.21.15.1:0/400702011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19700bf9d0 0x7f197014e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.805+0000 7f196f59e640 1 -- 172.21.15.1:0/400702011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1970157df0 con 0x7f1970151250 2024-03-06T07:20:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.806+0000 7f196f59e640 1 --2- 172.21.15.1:0/400702011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1970151250 0x7f19701eb9d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1968009f40 tx=0x7f1968004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.806+0000 7f19577fe640 1 -- 172.21.15.1:0/400702011 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1968043020 con 0x7f1970151250 2024-03-06T07:20:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.806+0000 7f19577fe640 1 -- 172.21.15.1:0/400702011 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196803e450 con 0x7f1970151250 2024-03-06T07:20:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.806+0000 7f197501c640 1 -- 172.21.15.1:0/400702011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19701580d0 con 0x7f1970151250 2024-03-06T07:20:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.806+0000 7f19577fe640 1 -- 172.21.15.1:0/400702011 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1968009510 con 0x7f1970151250 2024-03-06T07:20:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.806+0000 7f197501c640 1 -- 172.21.15.1:0/400702011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f197014e9a0 con 0x7f1970151250 2024-03-06T07:20:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.807+0000 7f197501c640 1 -- 172.21.15.1:0/400702011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1970067fd0 con 0x7f1970151250 2024-03-06T07:20:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.807+0000 7f19577fe640 1 -- 172.21.15.1:0/400702011 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1968046020 con 0x7f1970151250 2024-03-06T07:20:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.807+0000 7f19577fe640 1 --2- 172.21.15.1:0/400702011 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f195c06b4d0 0x7f195c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.808+0000 7f19577fe640 1 -- 172.21.15.1:0/400702011 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f19680acb80 con 0x7f1970151250 2024-03-06T07:20:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.808+0000 7f196ed9d640 1 --2- 172.21.15.1:0/400702011 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f195c06b4d0 0x7f195c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.808+0000 7f196ed9d640 1 --2- 172.21.15.1:0/400702011 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f195c06b4d0 0x7f195c06d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1960000f60 tx=0x7f1960009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:31.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.812+0000 7f19577fe640 1 -- 172.21.15.1:0/400702011 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f196804d030 con 0x7f1970151250 2024-03-06T07:20:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.991+0000 7f197501c640 1 -- 172.21.15.1:0/400702011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f19700b06f0 con 0x7f1970151250 2024-03-06T07:20:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.991+0000 7f19577fe640 1 -- 172.21.15.1:0/400702011 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f196807b070 con 0x7f1970151250 2024-03-06T07:20:31.993 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:20:31.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.993+0000 7f197501c640 1 -- 172.21.15.1:0/400702011 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f195c06b4d0 msgr2=0x7f195c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.993+0000 7f197501c640 1 --2- 172.21.15.1:0/400702011 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f195c06b4d0 0x7f195c06d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1960000f60 tx=0x7f1960009040 comp rx=0 tx=0).stop 2024-03-06T07:20:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.993+0000 7f197501c640 1 -- 172.21.15.1:0/400702011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1970151250 msgr2=0x7f19701eb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.993+0000 7f197501c640 1 --2- 172.21.15.1:0/400702011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1970151250 0x7f19701eb9d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1968009f40 tx=0x7f1968004a40 comp rx=0 tx=0).stop 2024-03-06T07:20:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.993+0000 7f197501c640 1 -- 172.21.15.1:0/400702011 shutdown_connections 2024-03-06T07:20:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.993+0000 7f197501c640 1 --2- 172.21.15.1:0/400702011 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f195c06b4d0 0x7f195c06d990 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.993+0000 7f197501c640 1 --2- 172.21.15.1:0/400702011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1970151250 0x7f19701eb9d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.993+0000 7f197501c640 1 --2- 172.21.15.1:0/400702011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1970152650 0x7f19701eb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.993+0000 7f197501c640 1 --2- 172.21.15.1:0/400702011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19700bf9d0 0x7f197014e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.993+0000 7f197501c640 1 -- 172.21.15.1:0/400702011 >> 172.21.15.1:0/400702011 conn(0x7f1970066ee0 msgr2=0x7f19700c52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.993+0000 7f197501c640 1 -- 172.21.15.1:0/400702011 shutdown_connections 2024-03-06T07:20:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:31.993+0000 7f197501c640 1 -- 172.21.15.1:0/400702011 wait complete. 2024-03-06T07:20:32.006 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:20:32.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.109+0000 7f17d697a640 1 Processor -- start 2024-03-06T07:20:32.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.109+0000 7f17d697a640 1 -- start start 2024-03-06T07:20:32.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.109+0000 7f17d697a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f17d00a6fc0 0x7f17d0155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.109+0000 7f17d697a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c6510 0x7f17d01556c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.109+0000 7f17d697a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d00c25e0 con 0x7f17d006ccf0 2024-03-06T07:20:32.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.109+0000 7f17d697a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d0155c00 con 0x7f17d00a6fc0 2024-03-06T07:20:32.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.110+0000 7f17d697a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d00c2d40 con 0x7f17d00c6510 2024-03-06T07:20:32.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.110+0000 7f17cf7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f17d00a6fc0 0x7f17d0155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.110+0000 7f17cffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f17d006ccf0 0x7f17d0072510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42998/0 (socket says 172.21.15.1:42998) 2024-03-06T07:20:32.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.110+0000 7f17cffff640 1 -- 172.21.15.1:0/2146970533 learned_addr learned my addr 172.21.15.1:0/2146970533 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:32.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.110+0000 7f17d4f17640 1 --2- 172.21.15.1:0/2146970533 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c6510 0x7f17d01556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.110+0000 7f17cf7fe640 1 -- 172.21.15.1:0/2146970533 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c6510 msgr2=0x7f17d01556c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:32.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.110+0000 7f17cf7fe640 1 --2- 172.21.15.1:0/2146970533 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c6510 0x7f17d01556c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.110+0000 7f17cf7fe640 1 -- 172.21.15.1:0/2146970533 >> v1:172.21.15.1:6789/0 conn(0x7f17d006ccf0 legacy=0x7f17d0072510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.110+0000 7f17ceffd640 1 -- 172.21.15.1:0/2146970533 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138647244 0 0) 0x7f17c0001110 con 0x7f17d006ccf0 2024-03-06T07:20:32.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.110+0000 7f17cf7fe640 1 -- 172.21.15.1:0/2146970533 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d00c73f0 con 0x7f17d00a6fc0 2024-03-06T07:20:32.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.111+0000 7f17cf7fe640 1 --2- 172.21.15.1:0/2146970533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d00a6fc0 0x7f17d0155180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f17c40097a0 tx=0x7f17c402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f089d0140d5365c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.111+0000 7f17ceffd640 1 -- 172.21.15.1:0/2146970533 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c40315a0 con 0x7f17d00a6fc0 2024-03-06T07:20:32.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.111+0000 7f17ceffd640 1 -- 172.21.15.1:0/2146970533 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c4037070 con 0x7f17d00a6fc0 2024-03-06T07:20:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.111+0000 7f17ceffd640 1 -- 172.21.15.1:0/2146970533 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4031960 con 0x7f17d00a6fc0 2024-03-06T07:20:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.111+0000 7f17d697a640 1 -- 172.21.15.1:0/2146970533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d00a6fc0 msgr2=0x7f17d0155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.111+0000 7f17d697a640 1 --2- 172.21.15.1:0/2146970533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d00a6fc0 0x7f17d0155180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f17c40097a0 tx=0x7f17c402b840 comp rx=0 tx=0).stop 2024-03-06T07:20:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.111+0000 7f17d697a640 1 -- 172.21.15.1:0/2146970533 shutdown_connections 2024-03-06T07:20:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.111+0000 7f17d697a640 1 --2- 172.21.15.1:0/2146970533 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c6510 0x7f17d01556c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.111+0000 7f17d697a640 1 --2- 172.21.15.1:0/2146970533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d00a6fc0 0x7f17d0155180 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.111+0000 7f17d697a640 1 -- 172.21.15.1:0/2146970533 >> 172.21.15.1:0/2146970533 conn(0x7f17d00a9700 msgr2=0x7f17d00a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.112+0000 7f17d697a640 1 -- 172.21.15.1:0/2146970533 shutdown_connections 2024-03-06T07:20:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.112+0000 7f17d697a640 1 -- 172.21.15.1:0/2146970533 wait complete. 2024-03-06T07:20:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.113+0000 7f17d697a640 1 Processor -- start 2024-03-06T07:20:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.113+0000 7f17d697a640 1 -- start start 2024-03-06T07:20:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.113+0000 7f17d697a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d006ccf0 0x7f17d00abab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.113+0000 7f17d697a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d00a6fc0 0x7f17d006f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.113+0000 7f17d697a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c6510 0x7f17d006f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.113+0000 7f17d697a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d00c76d0 con 0x7f17d006ccf0 2024-03-06T07:20:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.113+0000 7f17d697a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d006d0d0 con 0x7f17d00a6fc0 2024-03-06T07:20:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.113+0000 7f17d697a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d0072c20 con 0x7f17d00c6510 2024-03-06T07:20:32.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17cffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d006ccf0 0x7f17d00abab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17cffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d006ccf0 0x7f17d00abab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34920/0 (socket says 172.21.15.1:34920) 2024-03-06T07:20:32.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17cffff640 1 -- 172.21.15.1:0/1711706149 learned_addr learned my addr 172.21.15.1:0/1711706149 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:32.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17cf7fe640 1 --2- 172.21.15.1:0/1711706149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d00a6fc0 0x7f17d006f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17d4f17640 1 --2- 172.21.15.1:0/1711706149 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c6510 0x7f17d006f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17cffff640 1 -- 172.21.15.1:0/1711706149 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c6510 msgr2=0x7f17d006f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17cffff640 1 --2- 172.21.15.1:0/1711706149 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c6510 0x7f17d006f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17cffff640 1 -- 172.21.15.1:0/1711706149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d00a6fc0 msgr2=0x7f17d006f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17cffff640 1 --2- 172.21.15.1:0/1711706149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d00a6fc0 0x7f17d006f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17cffff640 1 -- 172.21.15.1:0/1711706149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d006e450 con 0x7f17d006ccf0 2024-03-06T07:20:32.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17d4f17640 1 --2- 172.21.15.1:0/1711706149 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c6510 0x7f17d006f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:32.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17cf7fe640 1 --2- 172.21.15.1:0/1711706149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d00a6fc0 0x7f17d006f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17cffff640 1 --2- 172.21.15.1:0/1711706149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d006ccf0 0x7f17d00abab0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f17c000c430 tx=0x7f17c000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17ccff9640 1 -- 172.21.15.1:0/1711706149 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c00074e0 con 0x7f17d006ccf0 2024-03-06T07:20:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17ccff9640 1 -- 172.21.15.1:0/1711706149 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c00047e0 con 0x7f17d006ccf0 2024-03-06T07:20:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17d697a640 1 -- 172.21.15.1:0/1711706149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d006e730 con 0x7f17d006ccf0 2024-03-06T07:20:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17ccff9640 1 -- 172.21.15.1:0/1711706149 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c0004bd0 con 0x7f17d006ccf0 2024-03-06T07:20:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.114+0000 7f17d697a640 1 -- 172.21.15.1:0/1711706149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d00abff0 con 0x7f17d006ccf0 2024-03-06T07:20:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.115+0000 7f17d697a640 1 -- 172.21.15.1:0/1711706149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d00686e0 con 0x7f17d006ccf0 2024-03-06T07:20:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.116+0000 7f17ccff9640 1 -- 172.21.15.1:0/1711706149 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17c0011070 con 0x7f17d006ccf0 2024-03-06T07:20:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.117+0000 7f17ccff9640 1 --2- 172.21.15.1:0/1711706149 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17bc06b4d0 0x7f17bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.117+0000 7f17ccff9640 1 -- 172.21.15.1:0/1711706149 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f17c008a0c0 con 0x7f17d006ccf0 2024-03-06T07:20:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.117+0000 7f17cf7fe640 1 --2- 172.21.15.1:0/1711706149 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17bc06b4d0 0x7f17bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.117+0000 7f17cf7fe640 1 --2- 172.21.15.1:0/1711706149 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17bc06b4d0 0x7f17bc06d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f17c4036000 tx=0x7f17c4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.125+0000 7f17ccff9640 1 -- 172.21.15.1:0/1711706149 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17d00c25e0 con 0x7f17d006ccf0 2024-03-06T07:20:32.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.307+0000 7f17d697a640 1 -- 172.21.15.1:0/1711706149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f17d00c58e0 con 0x7f17d006ccf0 2024-03-06T07:20:32.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.307+0000 7f17ccff9640 1 -- 172.21.15.1:0/1711706149 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f17c000e280 con 0x7f17d006ccf0 2024-03-06T07:20:32.310 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:20:32.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.310+0000 7f17d697a640 1 -- 172.21.15.1:0/1711706149 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17bc06b4d0 msgr2=0x7f17bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.310+0000 7f17d697a640 1 --2- 172.21.15.1:0/1711706149 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17bc06b4d0 0x7f17bc06d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f17c4036000 tx=0x7f17c4008030 comp rx=0 tx=0).stop 2024-03-06T07:20:32.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.310+0000 7f17d697a640 1 -- 172.21.15.1:0/1711706149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d006ccf0 msgr2=0x7f17d00abab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.310+0000 7f17d697a640 1 --2- 172.21.15.1:0/1711706149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d006ccf0 0x7f17d00abab0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f17c000c430 tx=0x7f17c000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.310+0000 7f17d697a640 1 -- 172.21.15.1:0/1711706149 shutdown_connections 2024-03-06T07:20:32.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.310+0000 7f17d697a640 1 --2- 172.21.15.1:0/1711706149 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17bc06b4d0 0x7f17bc06d990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.310+0000 7f17d697a640 1 --2- 172.21.15.1:0/1711706149 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c6510 0x7f17d006f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.310+0000 7f17d697a640 1 --2- 172.21.15.1:0/1711706149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d00a6fc0 0x7f17d006f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.310+0000 7f17d697a640 1 --2- 172.21.15.1:0/1711706149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d006ccf0 0x7f17d00abab0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.310+0000 7f17d697a640 1 -- 172.21.15.1:0/1711706149 >> 172.21.15.1:0/1711706149 conn(0x7f17d00a9700 msgr2=0x7f17d00c43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:32.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.310+0000 7f17d697a640 1 -- 172.21.15.1:0/1711706149 shutdown_connections 2024-03-06T07:20:32.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.310+0000 7f17d697a640 1 -- 172.21.15.1:0/1711706149 wait complete. 2024-03-06T07:20:32.322 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:20:32.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.424+0000 7f0f8bdd7640 1 Processor -- start 2024-03-06T07:20:32.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.424+0000 7f0f8bdd7640 1 -- start start 2024-03-06T07:20:32.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.425+0000 7f0f8bdd7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f840c7cc0 0x7f0f84156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.425+0000 7f0f8bdd7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f84154a70 con 0x7f0f8414cf90 2024-03-06T07:20:32.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.425+0000 7f0f8bdd7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f84153a90 con 0x7f0f84070be0 2024-03-06T07:20:32.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.425+0000 7f0f8bdd7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8414d370 con 0x7f0f840c7cc0 2024-03-06T07:20:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.425+0000 7f0f89b73640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0f8414cf90 0x7f0f84154360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43010/0 (socket says 172.21.15.1:43010) 2024-03-06T07:20:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.425+0000 7f0f89b73640 1 -- 172.21.15.1:0/3787237859 learned_addr learned my addr 172.21.15.1:0/3787237859 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.425+0000 7f0f89372640 1 --2- 172.21.15.1:0/3787237859 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f840c7cc0 0x7f0f84156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.425+0000 7f0f88b71640 1 -- 172.21.15.1:0/3787237859 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174400845 0 0) 0x7f0f84154a70 con 0x7f0f8414cf90 2024-03-06T07:20:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.425+0000 7f0f88b71640 1 -- 172.21.15.1:0/3787237859 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f70003520 con 0x7f0f8414cf90 2024-03-06T07:20:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f88b71640 1 -- 172.21.15.1:0/3787237859 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2941200624 0 0) 0x7f0f70003520 con 0x7f0f8414cf90 2024-03-06T07:20:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f88b71640 1 -- 172.21.15.1:0/3787237859 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f840c7cc0 msgr2=0x7f0f84156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f88b71640 1 --2- 172.21.15.1:0/3787237859 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f840c7cc0 0x7f0f84156fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f88b71640 1 -- 172.21.15.1:0/3787237859 >> v1:172.21.15.98:6789/0 conn(0x7f0f84070be0 legacy=0x7f0f8415d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f88b71640 1 -- 172.21.15.1:0/3787237859 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f84153d70 con 0x7f0f8414cf90 2024-03-06T07:20:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f88b71640 1 -- 172.21.15.1:0/3787237859 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f0f80001670 con 0x7f0f8414cf90 2024-03-06T07:20:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f88b71640 1 -- 172.21.15.1:0/3787237859 >> v1:172.21.15.1:6789/0 conn(0x7f0f8414cf90 legacy=0x7f0f84154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f88b71640 1 --2- 172.21.15.1:0/3787237859 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f840c7cc0 0x7f0f700039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f88b71640 1 --2- 172.21.15.1:0/3787237859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f70003f60 0x7f0f70004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f88b71640 1 --2- 172.21.15.1:0/3787237859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f70004d40 0x7f0f700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f88b71640 1 -- 172.21.15.1:0/3787237859 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8414d370 con 0x7f0f70004d40 2024-03-06T07:20:32.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f88b71640 1 -- 172.21.15.1:0/3787237859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7000bc00 con 0x7f0f70003f60 2024-03-06T07:20:32.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f88b71640 1 -- 172.21.15.1:0/3787237859 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7000be00 con 0x7f0f840c7cc0 2024-03-06T07:20:32.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f8a374640 1 --2- 172.21.15.1:0/3787237859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f70004d40 0x7f0f700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f89b73640 1 --2- 172.21.15.1:0/3787237859 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f840c7cc0 0x7f0f700039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f89372640 1 --2- 172.21.15.1:0/3787237859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f70003f60 0x7f0f70004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f8a374640 1 -- 172.21.15.1:0/3787237859 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f840c7cc0 msgr2=0x7f0f700039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:32.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f8a374640 1 --2- 172.21.15.1:0/3787237859 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f840c7cc0 0x7f0f700039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f8a374640 1 -- 172.21.15.1:0/3787237859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f70003f60 msgr2=0x7f0f70004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:32.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f8a374640 1 --2- 172.21.15.1:0/3787237859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f70003f60 0x7f0f70004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.426+0000 7f0f8a374640 1 -- 172.21.15.1:0/3787237859 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f70003520 con 0x7f0f70004d40 2024-03-06T07:20:32.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.427+0000 7f0f8a374640 1 --2- 172.21.15.1:0/3787237859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f70004d40 0x7f0f700051c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c001e50 tx=0x7f0f70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=126d78aed866c424 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.427+0000 7f0f88b71640 1 -- 172.21.15.1:0/3787237859 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c001a70 con 0x7f0f70004d40 2024-03-06T07:20:32.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.427+0000 7f0f88b71640 1 -- 172.21.15.1:0/3787237859 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7c003b70 con 0x7f0f70004d40 2024-03-06T07:20:32.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.427+0000 7f0f88b71640 1 -- 172.21.15.1:0/3787237859 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c0050d0 con 0x7f0f70004d40 2024-03-06T07:20:32.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.428+0000 7f0f8bdd7640 1 -- 172.21.15.1:0/3787237859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f70004d40 msgr2=0x7f0f700051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.428+0000 7f0f8bdd7640 1 --2- 172.21.15.1:0/3787237859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f70004d40 0x7f0f700051c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c001e50 tx=0x7f0f70005830 comp rx=0 tx=0).stop 2024-03-06T07:20:32.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.428+0000 7f0f8a374640 1 -- 172.21.15.1:0/3787237859 reap_dead start 2024-03-06T07:20:32.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.428+0000 7f0f8bdd7640 1 -- 172.21.15.1:0/3787237859 shutdown_connections 2024-03-06T07:20:32.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.428+0000 7f0f8bdd7640 1 -- 172.21.15.1:0/3787237859 >> 172.21.15.1:0/3787237859 conn(0x7f0f840c88f0 msgr2=0x7f0f840c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:32.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.428+0000 7f0f8bdd7640 1 -- 172.21.15.1:0/3787237859 shutdown_connections 2024-03-06T07:20:32.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.428+0000 7f0f8bdd7640 1 -- 172.21.15.1:0/3787237859 wait complete. 2024-03-06T07:20:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.430+0000 7f0f8bdd7640 1 Processor -- start 2024-03-06T07:20:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.430+0000 7f0f8bdd7640 1 -- start start 2024-03-06T07:20:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f8bdd7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f70004d40 0x7f0f84147270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f8bdd7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f840ac970 0x7f0f841ec200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f8bdd7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f840ad700 0x7f0f841455b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f8bdd7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f841567e0 con 0x7f0f840ad700 2024-03-06T07:20:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f8bdd7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f840c6190 con 0x7f0f70004d40 2024-03-06T07:20:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f8bdd7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f84156ac0 con 0x7f0f840ac970 2024-03-06T07:20:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f8a374640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f840ad700 0x7f0f841455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f8a374640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f840ad700 0x7f0f841455b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34942/0 (socket says 172.21.15.1:34942) 2024-03-06T07:20:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f8a374640 1 -- 172.21.15.1:0/2888028332 learned_addr learned my addr 172.21.15.1:0/2888028332 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f89b73640 1 --2- 172.21.15.1:0/2888028332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f70004d40 0x7f0f84147270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f89372640 1 --2- 172.21.15.1:0/2888028332 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f840ac970 0x7f0f841ec200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f8a374640 1 -- 172.21.15.1:0/2888028332 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f840ac970 msgr2=0x7f0f841ec200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f8a374640 1 --2- 172.21.15.1:0/2888028332 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f840ac970 0x7f0f841ec200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f8a374640 1 -- 172.21.15.1:0/2888028332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f70004d40 msgr2=0x7f0f84147270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f8a374640 1 --2- 172.21.15.1:0/2888028332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f70004d40 0x7f0f84147270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f8a374640 1 -- 172.21.15.1:0/2888028332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f841eca40 con 0x7f0f840ad700 2024-03-06T07:20:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f8a374640 1 --2- 172.21.15.1:0/2888028332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f840ad700 0x7f0f841455b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c001f70 tx=0x7f0f7c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f7a7fc640 1 -- 172.21.15.1:0/2888028332 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c007db0 con 0x7f0f840ad700 2024-03-06T07:20:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f7a7fc640 1 -- 172.21.15.1:0/2888028332 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7c003970 con 0x7f0f840ad700 2024-03-06T07:20:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f8bdd7640 1 -- 172.21.15.1:0/2888028332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f841ecd20 con 0x7f0f840ad700 2024-03-06T07:20:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.431+0000 7f0f7a7fc640 1 -- 172.21.15.1:0/2888028332 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c02f6a0 con 0x7f0f840ad700 2024-03-06T07:20:32.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.432+0000 7f0f8bdd7640 1 -- 172.21.15.1:0/2888028332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f841ec740 con 0x7f0f840ad700 2024-03-06T07:20:32.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.433+0000 7f0f8bdd7640 1 -- 172.21.15.1:0/2888028332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f84157340 con 0x7f0f840ad700 2024-03-06T07:20:32.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.433+0000 7f0f7a7fc640 1 -- 172.21.15.1:0/2888028332 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f7c02f840 con 0x7f0f840ad700 2024-03-06T07:20:32.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.433+0000 7f0f7a7fc640 1 --2- 172.21.15.1:0/2888028332 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0f6806b500 0x7f0f6806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.433+0000 7f0f7a7fc640 1 -- 172.21.15.1:0/2888028332 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0f7c0abb60 con 0x7f0f840ad700 2024-03-06T07:20:32.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.434+0000 7f0f89b73640 1 --2- 172.21.15.1:0/2888028332 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0f6806b500 0x7f0f6806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.434+0000 7f0f89b73640 1 --2- 172.21.15.1:0/2888028332 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0f6806b500 0x7f0f6806d9c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0f800010e0 tx=0x7f0f800078f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.438+0000 7f0f7a7fc640 1 -- 172.21.15.1:0/2888028332 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f84157340 con 0x7f0f840ad700 2024-03-06T07:20:32.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.626+0000 7f0f8bdd7640 1 -- 172.21.15.1:0/2888028332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0f84070150 con 0x7f0f840ad700 2024-03-06T07:20:32.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.626+0000 7f0f7a7fc640 1 -- 172.21.15.1:0/2888028332 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0f84070150 con 0x7f0f840ad700 2024-03-06T07:20:32.628 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:20:32.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.628+0000 7f0f8bdd7640 1 -- 172.21.15.1:0/2888028332 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0f6806b500 msgr2=0x7f0f6806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.628+0000 7f0f8bdd7640 1 --2- 172.21.15.1:0/2888028332 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0f6806b500 0x7f0f6806d9c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0f800010e0 tx=0x7f0f800078f0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.628+0000 7f0f8bdd7640 1 -- 172.21.15.1:0/2888028332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f840ad700 msgr2=0x7f0f841455b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.628+0000 7f0f8bdd7640 1 --2- 172.21.15.1:0/2888028332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f840ad700 0x7f0f841455b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c001f70 tx=0x7f0f7c0014a0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.628+0000 7f0f8bdd7640 1 -- 172.21.15.1:0/2888028332 shutdown_connections 2024-03-06T07:20:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.628+0000 7f0f8bdd7640 1 --2- 172.21.15.1:0/2888028332 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0f6806b500 0x7f0f6806d9c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.628+0000 7f0f8bdd7640 1 --2- 172.21.15.1:0/2888028332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f840ad700 0x7f0f841455b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.628+0000 7f0f8bdd7640 1 --2- 172.21.15.1:0/2888028332 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f840ac970 0x7f0f841ec200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.628+0000 7f0f8bdd7640 1 --2- 172.21.15.1:0/2888028332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f70004d40 0x7f0f84147270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.628+0000 7f0f8bdd7640 1 -- 172.21.15.1:0/2888028332 >> 172.21.15.1:0/2888028332 conn(0x7f0f840c88f0 msgr2=0x7f0f84151670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.628+0000 7f0f8bdd7640 1 -- 172.21.15.1:0/2888028332 shutdown_connections 2024-03-06T07:20:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.628+0000 7f0f8bdd7640 1 -- 172.21.15.1:0/2888028332 wait complete. 2024-03-06T07:20:32.640 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:20:32.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.744+0000 7f8455fc7640 1 Processor -- start 2024-03-06T07:20:32.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.744+0000 7f8455fc7640 1 -- start start 2024-03-06T07:20:32.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.745+0000 7f8455fc7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8450150640 0x7f845006aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.745+0000 7f8455fc7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8450152eb0 0x7f8450153360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.745+0000 7f8455fc7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84500c1bc0 con 0x7f84500a9cb0 2024-03-06T07:20:32.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.745+0000 7f8455fc7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84500c2690 con 0x7f8450150640 2024-03-06T07:20:32.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.745+0000 7f8455fc7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84500c1ea0 con 0x7f8450152eb0 2024-03-06T07:20:32.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.745+0000 7f844f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8450150640 0x7f845006aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.745+0000 7f8446ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f84500a9cb0 0x7f845006b530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43020/0 (socket says 172.21.15.1:43020) 2024-03-06T07:20:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.745+0000 7f8446ffd640 1 -- 172.21.15.1:0/649407936 learned_addr learned my addr 172.21.15.1:0/649407936 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.746+0000 7f844ffff640 1 --2- 172.21.15.1:0/649407936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8450152eb0 0x7f8450153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.746+0000 7f844f7fe640 1 -- 172.21.15.1:0/649407936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8450152eb0 msgr2=0x7f8450153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.746+0000 7f844f7fe640 1 --2- 172.21.15.1:0/649407936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8450152eb0 0x7f8450153360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.746+0000 7f844f7fe640 1 -- 172.21.15.1:0/649407936 >> v1:172.21.15.1:6789/0 conn(0x7f84500a9cb0 legacy=0x7f845006b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.746+0000 7f844effd640 1 -- 172.21.15.1:0/649407936 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689789260 0 0) 0x7f84500c1bc0 con 0x7f84500a9cb0 2024-03-06T07:20:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.746+0000 7f844f7fe640 1 -- 172.21.15.1:0/649407936 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8450156fd0 con 0x7f8450150640 2024-03-06T07:20:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.746+0000 7f844f7fe640 1 --2- 172.21.15.1:0/649407936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8450150640 0x7f845006aff0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f84400097a0 tx=0x7f844002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d8f0b2e7e186db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.747+0000 7f844effd640 1 -- 172.21.15.1:0/649407936 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8440038070 con 0x7f8450150640 2024-03-06T07:20:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.747+0000 7f844effd640 1 -- 172.21.15.1:0/649407936 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8440031700 con 0x7f8450150640 2024-03-06T07:20:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.747+0000 7f844effd640 1 -- 172.21.15.1:0/649407936 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8440031ad0 con 0x7f8450150640 2024-03-06T07:20:32.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.747+0000 7f8455fc7640 1 -- 172.21.15.1:0/649407936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8450150640 msgr2=0x7f845006aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.747+0000 7f8455fc7640 1 --2- 172.21.15.1:0/649407936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8450150640 0x7f845006aff0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f84400097a0 tx=0x7f844002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.747+0000 7f8455fc7640 1 -- 172.21.15.1:0/649407936 shutdown_connections 2024-03-06T07:20:32.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.747+0000 7f8455fc7640 1 --2- 172.21.15.1:0/649407936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8450152eb0 0x7f8450153360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.747+0000 7f8455fc7640 1 --2- 172.21.15.1:0/649407936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8450150640 0x7f845006aff0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.747+0000 7f8455fc7640 1 -- 172.21.15.1:0/649407936 >> 172.21.15.1:0/649407936 conn(0x7f84500c3cc0 msgr2=0x7f84500c40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.747+0000 7f8455fc7640 1 -- 172.21.15.1:0/649407936 shutdown_connections 2024-03-06T07:20:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.747+0000 7f8455fc7640 1 -- 172.21.15.1:0/649407936 wait complete. 2024-03-06T07:20:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.749+0000 7f8455fc7640 1 Processor -- start 2024-03-06T07:20:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.749+0000 7f8455fc7640 1 -- start start 2024-03-06T07:20:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.750+0000 7f8455fc7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84500a9cb0 0x7f845014a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.750+0000 7f8455fc7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8450150640 0x7f84500bcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.750+0000 7f8455fc7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8450152eb0 0x7f84501555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.750+0000 7f8455fc7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8450150ba0 con 0x7f84500a9cb0 2024-03-06T07:20:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.750+0000 7f8455fc7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8450150a20 con 0x7f8450150640 2024-03-06T07:20:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.750+0000 7f8455fc7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84500c2b20 con 0x7f8450152eb0 2024-03-06T07:20:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.750+0000 7f844f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84500a9cb0 0x7f845014a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.750+0000 7f844f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84500a9cb0 0x7f845014a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34952/0 (socket says 172.21.15.1:34952) 2024-03-06T07:20:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.750+0000 7f844f7fe640 1 -- 172.21.15.1:0/2445590905 learned_addr learned my addr 172.21.15.1:0/2445590905 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.750+0000 7f844f7fe640 1 -- 172.21.15.1:0/2445590905 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8450152eb0 msgr2=0x7f84501555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.750+0000 7f844f7fe640 1 --2- 172.21.15.1:0/2445590905 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8450152eb0 0x7f84501555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.750+0000 7f844f7fe640 1 -- 172.21.15.1:0/2445590905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8450150640 msgr2=0x7f84500bcaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.750+0000 7f844f7fe640 1 --2- 172.21.15.1:0/2445590905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8450150640 0x7f84500bcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.750+0000 7f844f7fe640 1 -- 172.21.15.1:0/2445590905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84500bc280 con 0x7f84500a9cb0 2024-03-06T07:20:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.751+0000 7f844f7fe640 1 --2- 172.21.15.1:0/2445590905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84500a9cb0 0x7f845014a2e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f84400097a0 tx=0x7f84400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.751+0000 7f844cff9640 1 -- 172.21.15.1:0/2445590905 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8440038040 con 0x7f84500a9cb0 2024-03-06T07:20:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.751+0000 7f844cff9640 1 -- 172.21.15.1:0/2445590905 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8440004030 con 0x7f84500a9cb0 2024-03-06T07:20:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.751+0000 7f8455fc7640 1 -- 172.21.15.1:0/2445590905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84501e9790 con 0x7f84500a9cb0 2024-03-06T07:20:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.751+0000 7f844cff9640 1 -- 172.21.15.1:0/2445590905 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8440004420 con 0x7f84500a9cb0 2024-03-06T07:20:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.751+0000 7f8455fc7640 1 -- 172.21.15.1:0/2445590905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84501e9a70 con 0x7f84500a9cb0 2024-03-06T07:20:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.752+0000 7f8455fc7640 1 -- 172.21.15.1:0/2445590905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84500686e0 con 0x7f84500a9cb0 2024-03-06T07:20:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.752+0000 7f844cff9640 1 -- 172.21.15.1:0/2445590905 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8440033800 con 0x7f84500a9cb0 2024-03-06T07:20:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.753+0000 7f844cff9640 1 --2- 172.21.15.1:0/2445590905 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f843006b4d0 0x7f843006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.753+0000 7f844cff9640 1 -- 172.21.15.1:0/2445590905 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f84400ac740 con 0x7f84500a9cb0 2024-03-06T07:20:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.753+0000 7f8446ffd640 1 --2- 172.21.15.1:0/2445590905 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f843006b4d0 0x7f843006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.754+0000 7f8446ffd640 1 --2- 172.21.15.1:0/2445590905 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f843006b4d0 0x7f843006d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8438007fd0 tx=0x7f8438007960 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.757+0000 7f844cff9640 1 -- 172.21.15.1:0/2445590905 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8440030070 con 0x7f84500a9cb0 2024-03-06T07:20:32.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.937+0000 7f8455fc7640 1 -- 172.21.15.1:0/2445590905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f84500bc760 con 0x7f84500a9cb0 2024-03-06T07:20:32.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.938+0000 7f844cff9640 1 -- 172.21.15.1:0/2445590905 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f844007b310 con 0x7f84500a9cb0 2024-03-06T07:20:32.940 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:20:32.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.940+0000 7f8455fc7640 1 -- 172.21.15.1:0/2445590905 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f843006b4d0 msgr2=0x7f843006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.940+0000 7f8455fc7640 1 --2- 172.21.15.1:0/2445590905 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f843006b4d0 0x7f843006d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8438007fd0 tx=0x7f8438007960 comp rx=0 tx=0).stop 2024-03-06T07:20:32.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.940+0000 7f8455fc7640 1 -- 172.21.15.1:0/2445590905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84500a9cb0 msgr2=0x7f845014a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.940+0000 7f8455fc7640 1 --2- 172.21.15.1:0/2445590905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84500a9cb0 0x7f845014a2e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f84400097a0 tx=0x7f84400046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.940+0000 7f8455fc7640 1 -- 172.21.15.1:0/2445590905 shutdown_connections 2024-03-06T07:20:32.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.940+0000 7f8455fc7640 1 --2- 172.21.15.1:0/2445590905 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f843006b4d0 0x7f843006d990 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8438007fd0 tx=0x7f8438007960 comp rx=0 tx=0).stop 2024-03-06T07:20:32.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.940+0000 7f8455fc7640 1 --2- 172.21.15.1:0/2445590905 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8450152eb0 0x7f84501555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.940+0000 7f8455fc7640 1 --2- 172.21.15.1:0/2445590905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8450150640 0x7f84500bcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.940+0000 7f8455fc7640 1 --2- 172.21.15.1:0/2445590905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84500a9cb0 0x7f845014a2e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.940+0000 7f8455fc7640 1 -- 172.21.15.1:0/2445590905 >> 172.21.15.1:0/2445590905 conn(0x7f84500c3cc0 msgr2=0x7f84501521d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:32.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.940+0000 7f8455fc7640 1 -- 172.21.15.1:0/2445590905 shutdown_connections 2024-03-06T07:20:32.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:32.940+0000 7f8455fc7640 1 -- 172.21.15.1:0/2445590905 wait complete. 2024-03-06T07:20:32.952 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:20:33.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.052+0000 7f1642756640 1 Processor -- start 2024-03-06T07:20:33.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.052+0000 7f1642756640 1 -- start start 2024-03-06T07:20:33.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.053+0000 7f1642756640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f163c159140 0x7f163c0c4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.053+0000 7f1642756640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f163c0c7510 0x7f163c15fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.053+0000 7f1642756640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f163c072d00 con 0x7f163c0c69b0 2024-03-06T07:20:33.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.053+0000 7f1642756640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f163c0c0e80 con 0x7f163c159140 2024-03-06T07:20:33.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.053+0000 7f1642756640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163c0c13b0 con 0x7f163c0c7510 2024-03-06T07:20:33.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.053+0000 7f163bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f163c159140 0x7f163c0c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.053+0000 7f163bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f163c159140 0x7f163c0c4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34960/0 (socket says 172.21.15.1:34960) 2024-03-06T07:20:33.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.053+0000 7f163bfff640 1 -- 172.21.15.1:0/4187105865 learned_addr learned my addr 172.21.15.1:0/4187105865 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:33.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.053+0000 7f1640cf3640 1 --2- 172.21.15.1:0/4187105865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f163c0c7510 0x7f163c15fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.053+0000 7f163bfff640 1 -- 172.21.15.1:0/4187105865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f163c0c7510 msgr2=0x7f163c15fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:33.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.053+0000 7f163bfff640 1 --2- 172.21.15.1:0/4187105865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f163c0c7510 0x7f163c15fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.053+0000 7f163bfff640 1 -- 172.21.15.1:0/4187105865 >> v1:172.21.15.98:6789/0 conn(0x7f163c0c69b0 legacy=0x7f163c0c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.053+0000 7f163bfff640 1 -- 172.21.15.1:0/4187105865 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163c073180 con 0x7f163c159140 2024-03-06T07:20:33.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.054+0000 7f163bfff640 1 --2- 172.21.15.1:0/4187105865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c159140 0x7f163c0c4490 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1630009870 tx=0x7f163002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c0ed13ef1826f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.054+0000 7f163affd640 1 -- 172.21.15.1:0/4187105865 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1630035020 con 0x7f163c159140 2024-03-06T07:20:33.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.054+0000 7f163affd640 1 -- 172.21.15.1:0/4187105865 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1630031640 con 0x7f163c159140 2024-03-06T07:20:33.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.054+0000 7f163affd640 1 -- 172.21.15.1:0/4187105865 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163002d040 con 0x7f163c159140 2024-03-06T07:20:33.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.055+0000 7f1642756640 1 -- 172.21.15.1:0/4187105865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c159140 msgr2=0x7f163c0c4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.055+0000 7f1642756640 1 --2- 172.21.15.1:0/4187105865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c159140 0x7f163c0c4490 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1630009870 tx=0x7f163002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.055+0000 7f1642756640 1 -- 172.21.15.1:0/4187105865 shutdown_connections 2024-03-06T07:20:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.055+0000 7f1642756640 1 --2- 172.21.15.1:0/4187105865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f163c0c7510 0x7f163c15fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.055+0000 7f1642756640 1 --2- 172.21.15.1:0/4187105865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c159140 0x7f163c0c4490 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.055+0000 7f1642756640 1 -- 172.21.15.1:0/4187105865 >> 172.21.15.1:0/4187105865 conn(0x7f163c0c7a50 msgr2=0x7f163c0c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.055+0000 7f1642756640 1 -- 172.21.15.1:0/4187105865 shutdown_connections 2024-03-06T07:20:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.055+0000 7f1642756640 1 -- 172.21.15.1:0/4187105865 wait complete. 2024-03-06T07:20:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.056+0000 7f1642756640 1 Processor -- start 2024-03-06T07:20:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.056+0000 7f1642756640 1 -- start start 2024-03-06T07:20:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.057+0000 7f1642756640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f163c0c69b0 0x7f163c153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.057+0000 7f1642756640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0c7510 0x7f163c14e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.057+0000 7f1642756640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f163c159140 0x7f163c1eb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.057+0000 7f1642756640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163c0c10d0 con 0x7f163c0c7510 2024-03-06T07:20:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.057+0000 7f1642756640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163c0c6520 con 0x7f163c0c69b0 2024-03-06T07:20:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.057+0000 7f1642756640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163c159520 con 0x7f163c159140 2024-03-06T07:20:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.057+0000 7f163b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0c7510 0x7f163c14e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.057+0000 7f163b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0c7510 0x7f163c14e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34968/0 (socket says 172.21.15.1:34968) 2024-03-06T07:20:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.057+0000 7f163b7fe640 1 -- 172.21.15.1:0/3128524199 learned_addr learned my addr 172.21.15.1:0/3128524199 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.057+0000 7f163bfff640 1 --2- 172.21.15.1:0/3128524199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f163c0c69b0 0x7f163c153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.057+0000 7f1640cf3640 1 --2- 172.21.15.1:0/3128524199 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f163c159140 0x7f163c1eb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.057+0000 7f163b7fe640 1 -- 172.21.15.1:0/3128524199 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f163c159140 msgr2=0x7f163c1eb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.057+0000 7f163b7fe640 1 --2- 172.21.15.1:0/3128524199 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f163c159140 0x7f163c1eb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.057+0000 7f163b7fe640 1 -- 172.21.15.1:0/3128524199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f163c0c69b0 msgr2=0x7f163c153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.057+0000 7f163b7fe640 1 --2- 172.21.15.1:0/3128524199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f163c0c69b0 0x7f163c153620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.057+0000 7f163b7fe640 1 -- 172.21.15.1:0/3128524199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163c150f80 con 0x7f163c0c7510 2024-03-06T07:20:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.057+0000 7f163b7fe640 1 --2- 172.21.15.1:0/3128524199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0c7510 0x7f163c14e020 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f162c00c3a0 tx=0x7f162c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.058+0000 7f1638ff9640 1 -- 172.21.15.1:0/3128524199 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c0052d0 con 0x7f163c0c7510 2024-03-06T07:20:33.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.058+0000 7f1638ff9640 1 -- 172.21.15.1:0/3128524199 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c005490 con 0x7f163c0c7510 2024-03-06T07:20:33.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.058+0000 7f1642756640 1 -- 172.21.15.1:0/3128524199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f163c0a9eb0 con 0x7f163c0c7510 2024-03-06T07:20:33.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.058+0000 7f1638ff9640 1 -- 172.21.15.1:0/3128524199 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c007260 con 0x7f163c0c7510 2024-03-06T07:20:33.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.058+0000 7f1642756640 1 -- 172.21.15.1:0/3128524199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f163c0a8fa0 con 0x7f163c0c7510 2024-03-06T07:20:33.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.059+0000 7f1642756640 1 -- 172.21.15.1:0/3128524199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f163c1eafc0 con 0x7f163c0c7510 2024-03-06T07:20:33.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.059+0000 7f1638ff9640 1 -- 172.21.15.1:0/3128524199 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f162c007400 con 0x7f163c0c7510 2024-03-06T07:20:33.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.059+0000 7f1638ff9640 1 --2- 172.21.15.1:0/3128524199 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f162806b4d0 0x7f162806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.060+0000 7f1638ff9640 1 -- 172.21.15.1:0/3128524199 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f162c089900 con 0x7f163c0c7510 2024-03-06T07:20:33.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.060+0000 7f163bfff640 1 --2- 172.21.15.1:0/3128524199 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f162806b4d0 0x7f162806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.060+0000 7f163bfff640 1 --2- 172.21.15.1:0/3128524199 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f162806b4d0 0x7f162806d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1630036000 tx=0x7f163003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.064+0000 7f1638ff9640 1 -- 172.21.15.1:0/3128524199 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f163c1eafc0 con 0x7f163c0c7510 2024-03-06T07:20:33.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.257+0000 7f1642756640 1 -- 172.21.15.1:0/3128524199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f163c0686e0 con 0x7f163c0c7510 2024-03-06T07:20:33.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.258+0000 7f1638ff9640 1 -- 172.21.15.1:0/3128524199 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f162c057e80 con 0x7f163c0c7510 2024-03-06T07:20:33.260 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-06T07:20:33.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.260+0000 7f1642756640 1 -- 172.21.15.1:0/3128524199 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f162806b4d0 msgr2=0x7f162806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.260+0000 7f1642756640 1 --2- 172.21.15.1:0/3128524199 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f162806b4d0 0x7f162806d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1630036000 tx=0x7f163003c040 comp rx=0 tx=0).stop 2024-03-06T07:20:33.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.260+0000 7f1642756640 1 -- 172.21.15.1:0/3128524199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0c7510 msgr2=0x7f163c14e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.260+0000 7f1642756640 1 --2- 172.21.15.1:0/3128524199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0c7510 0x7f163c14e020 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f162c00c3a0 tx=0x7f162c00c860 comp rx=0 tx=0).stop 2024-03-06T07:20:33.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.260+0000 7f1642756640 1 -- 172.21.15.1:0/3128524199 shutdown_connections 2024-03-06T07:20:33.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.260+0000 7f1642756640 1 --2- 172.21.15.1:0/3128524199 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f162806b4d0 0x7f162806d990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.260+0000 7f1642756640 1 --2- 172.21.15.1:0/3128524199 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f163c159140 0x7f163c1eb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.260+0000 7f1642756640 1 --2- 172.21.15.1:0/3128524199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0c7510 0x7f163c14e020 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.260+0000 7f1642756640 1 --2- 172.21.15.1:0/3128524199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f163c0c69b0 0x7f163c153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.260+0000 7f1642756640 1 -- 172.21.15.1:0/3128524199 >> 172.21.15.1:0/3128524199 conn(0x7f163c0c7a50 msgr2=0x7f163c156d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:33.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.260+0000 7f1642756640 1 -- 172.21.15.1:0/3128524199 shutdown_connections 2024-03-06T07:20:33.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.260+0000 7f1642756640 1 -- 172.21.15.1:0/3128524199 wait complete. 2024-03-06T07:20:33.272 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:20:33.273 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:20:33.289 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:33.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.427+0000 7fbeff00e640 1 Processor -- start 2024-03-06T07:20:33.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.427+0000 7fbeff00e640 1 -- start start 2024-03-06T07:20:33.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.428+0000 7fbeff00e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbef8159140 0x7fbef80c4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.428+0000 7fbeff00e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbef80c7510 0x7fbef815fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.428+0000 7fbeff00e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef8072d00 con 0x7fbef80c69b0 2024-03-06T07:20:33.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.428+0000 7fbeff00e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef80c0e80 con 0x7fbef8159140 2024-03-06T07:20:33.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.428+0000 7fbeff00e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef80c13b0 con 0x7fbef80c7510 2024-03-06T07:20:33.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.428+0000 7fbeeffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbef80c69b0 0x7fbef80c2b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43028/0 (socket says 172.21.15.1:43028) 2024-03-06T07:20:33.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.428+0000 7fbeeffff640 1 -- 172.21.15.1:0/3319487274 learned_addr learned my addr 172.21.15.1:0/3319487274 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:33.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.428+0000 7fbefcdaa640 1 --2- 172.21.15.1:0/3319487274 >> v2:172.21.15.98:3300/0 conn(0x7fbef8159140 0x7fbef80c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.428+0000 7fbefd5ab640 1 --2- 172.21.15.1:0/3319487274 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbef80c7510 0x7fbef815fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.428+0000 7fbeef7fe640 1 -- 172.21.15.1:0/3319487274 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217891178 0 0) 0x7fbef8072d00 con 0x7fbef80c69b0 2024-03-06T07:20:33.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.428+0000 7fbeef7fe640 1 -- 172.21.15.1:0/3319487274 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbee0003520 con 0x7fbef80c69b0 2024-03-06T07:20:33.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.429+0000 7fbefd5ab640 1 -- 172.21.15.1:0/3319487274 >> v2:172.21.15.98:3300/0 conn(0x7fbef8159140 msgr2=0x7fbef80c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:33.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.429+0000 7fbefd5ab640 1 --2- 172.21.15.1:0/3319487274 >> v2:172.21.15.98:3300/0 conn(0x7fbef8159140 0x7fbef80c4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.429+0000 7fbefd5ab640 1 -- 172.21.15.1:0/3319487274 >> v1:172.21.15.1:6789/0 conn(0x7fbef80c69b0 legacy=0x7fbef80c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.429+0000 7fbeef7fe640 1 -- 172.21.15.1:0/3319487274 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4289760727 0 0) 0x7fbee0003520 con 0x7fbef80c69b0 2024-03-06T07:20:33.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.429+0000 7fbefd5ab640 1 -- 172.21.15.1:0/3319487274 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef8073180 con 0x7fbef80c7510 2024-03-06T07:20:33.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.429+0000 7fbefcdaa640 1 --2- 172.21.15.1:0/3319487274 >> v2:172.21.15.98:3300/0 conn(0x7fbef8159140 0x7fbef80c4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:33.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.429+0000 7fbefd5ab640 1 --2- 172.21.15.1:0/3319487274 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbef80c7510 0x7fbef815fb10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbef0009d90 tx=0x7fbef002bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3fe213052761bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.430+0000 7fbeef7fe640 1 -- 172.21.15.1:0/3319487274 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef0039070 con 0x7fbef80c7510 2024-03-06T07:20:33.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.430+0000 7fbeef7fe640 1 -- 172.21.15.1:0/3319487274 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef002c810 con 0x7fbef80c7510 2024-03-06T07:20:33.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.430+0000 7fbeef7fe640 1 -- 172.21.15.1:0/3319487274 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee8001670 con 0x7fbef80c7510 2024-03-06T07:20:33.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.430+0000 7fbeff00e640 1 -- 172.21.15.1:0/3319487274 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbef80c7510 msgr2=0x7fbef815fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.430+0000 7fbeff00e640 1 --2- 172.21.15.1:0/3319487274 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbef80c7510 0x7fbef815fb10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbef0009d90 tx=0x7fbef002bce0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.430+0000 7fbeff00e640 1 -- 172.21.15.1:0/3319487274 shutdown_connections 2024-03-06T07:20:33.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.430+0000 7fbeff00e640 1 --2- 172.21.15.1:0/3319487274 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbef80c7510 0x7fbef815fb10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.430+0000 7fbeff00e640 1 --2- 172.21.15.1:0/3319487274 >> v2:172.21.15.98:3300/0 conn(0x7fbef8159140 0x7fbef80c4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.430+0000 7fbeff00e640 1 -- 172.21.15.1:0/3319487274 >> 172.21.15.1:0/3319487274 conn(0x7fbef80c7a50 msgr2=0x7fbef80c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.430+0000 7fbeff00e640 1 -- 172.21.15.1:0/3319487274 shutdown_connections 2024-03-06T07:20:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.430+0000 7fbeff00e640 1 -- 172.21.15.1:0/3319487274 wait complete. 2024-03-06T07:20:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.431+0000 7fbeff00e640 1 Processor -- start 2024-03-06T07:20:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.431+0000 7fbeff00e640 1 -- start start 2024-03-06T07:20:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.432+0000 7fbeff00e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbef80c69b0 0x7fbef81ea990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.432+0000 7fbeff00e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbef80c7510 0x7fbef8155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.432+0000 7fbeff00e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbef8159140 0x7fbef80a7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.432+0000 7fbeff00e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef80c10d0 con 0x7fbef80c7510 2024-03-06T07:20:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.432+0000 7fbeff00e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef80c6520 con 0x7fbef80c69b0 2024-03-06T07:20:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.432+0000 7fbeff00e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef8159520 con 0x7fbef8159140 2024-03-06T07:20:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.432+0000 7fbeeffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbef80c7510 0x7fbef8155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.432+0000 7fbeeffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbef80c7510 0x7fbef8155d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34978/0 (socket says 172.21.15.1:34978) 2024-03-06T07:20:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.432+0000 7fbeeffff640 1 -- 172.21.15.1:0/2513975911 learned_addr learned my addr 172.21.15.1:0/2513975911 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.432+0000 7fbefcdaa640 1 --2- 172.21.15.1:0/2513975911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbef80c69b0 0x7fbef81ea990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.432+0000 7fbefd5ab640 1 --2- 172.21.15.1:0/2513975911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbef8159140 0x7fbef80a7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.432+0000 7fbeeffff640 1 -- 172.21.15.1:0/2513975911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbef8159140 msgr2=0x7fbef80a7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.432+0000 7fbeeffff640 1 --2- 172.21.15.1:0/2513975911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbef8159140 0x7fbef80a7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.432+0000 7fbeeffff640 1 -- 172.21.15.1:0/2513975911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbef80c69b0 msgr2=0x7fbef81ea990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.432+0000 7fbeeffff640 1 --2- 172.21.15.1:0/2513975911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbef80c69b0 0x7fbef81ea990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.432+0000 7fbeeffff640 1 -- 172.21.15.1:0/2513975911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef814a7a0 con 0x7fbef80c7510 2024-03-06T07:20:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.433+0000 7fbeeffff640 1 --2- 172.21.15.1:0/2513975911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbef80c7510 0x7fbef8155d50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbee800c7b0 tx=0x7fbee800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.433+0000 7fbeed7fa640 1 -- 172.21.15.1:0/2513975911 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee8005ad0 con 0x7fbef80c7510 2024-03-06T07:20:33.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.433+0000 7fbeed7fa640 1 -- 172.21.15.1:0/2513975911 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee8005c70 con 0x7fbef80c7510 2024-03-06T07:20:33.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.433+0000 7fbeff00e640 1 -- 172.21.15.1:0/2513975911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef81572b0 con 0x7fbef80c7510 2024-03-06T07:20:33.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.433+0000 7fbeed7fa640 1 -- 172.21.15.1:0/2513975911 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee80072b0 con 0x7fbef80c7510 2024-03-06T07:20:33.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.433+0000 7fbeff00e640 1 -- 172.21.15.1:0/2513975911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef8147840 con 0x7fbef80c7510 2024-03-06T07:20:33.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.434+0000 7fbeff00e640 1 -- 172.21.15.1:0/2513975911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef80686e0 con 0x7fbef80c7510 2024-03-06T07:20:33.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.434+0000 7fbeed7fa640 1 -- 172.21.15.1:0/2513975911 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbee8007450 con 0x7fbef80c7510 2024-03-06T07:20:33.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.435+0000 7fbeed7fa640 1 --2- 172.21.15.1:0/2513975911 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbee406b4d0 0x7fbee406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.435+0000 7fbeed7fa640 1 -- 172.21.15.1:0/2513975911 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbee808a280 con 0x7fbef80c7510 2024-03-06T07:20:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.435+0000 7fbefcdaa640 1 --2- 172.21.15.1:0/2513975911 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbee406b4d0 0x7fbee406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.435+0000 7fbefcdaa640 1 --2- 172.21.15.1:0/2513975911 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbee406b4d0 0x7fbee406d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009a30 tx=0x7fbef4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.439+0000 7fbeed7fa640 1 -- 172.21.15.1:0/2513975911 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbee8010300 con 0x7fbef80c7510 2024-03-06T07:20:33.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.619+0000 7fbeff00e640 1 -- 172.21.15.1:0/2513975911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbef80a71a0 con 0x7fbef80c7510 2024-03-06T07:20:33.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.621+0000 7fbeed7fa640 1 -- 172.21.15.1:0/2513975911 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7fbee8058f10 con 0x7fbef80c7510 2024-03-06T07:20:33.623 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:33.623 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:33.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.623+0000 7fbeff00e640 1 -- 172.21.15.1:0/2513975911 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbee406b4d0 msgr2=0x7fbee406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.623+0000 7fbeff00e640 1 --2- 172.21.15.1:0/2513975911 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbee406b4d0 0x7fbee406d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009a30 tx=0x7fbef4008040 comp rx=0 tx=0).stop 2024-03-06T07:20:33.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.623+0000 7fbeff00e640 1 -- 172.21.15.1:0/2513975911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbef80c7510 msgr2=0x7fbef8155d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.623+0000 7fbeff00e640 1 --2- 172.21.15.1:0/2513975911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbef80c7510 0x7fbef8155d50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbee800c7b0 tx=0x7fbee800cc70 comp rx=0 tx=0).stop 2024-03-06T07:20:33.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.623+0000 7fbeff00e640 1 -- 172.21.15.1:0/2513975911 shutdown_connections 2024-03-06T07:20:33.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.623+0000 7fbeff00e640 1 --2- 172.21.15.1:0/2513975911 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbee406b4d0 0x7fbee406d990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.623+0000 7fbeff00e640 1 --2- 172.21.15.1:0/2513975911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbef8159140 0x7fbef80a7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.623+0000 7fbeff00e640 1 --2- 172.21.15.1:0/2513975911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbef80c7510 0x7fbef8155d50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.623+0000 7fbeff00e640 1 --2- 172.21.15.1:0/2513975911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbef80c69b0 0x7fbef81ea990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.623+0000 7fbeff00e640 1 -- 172.21.15.1:0/2513975911 >> 172.21.15.1:0/2513975911 conn(0x7fbef80c7a50 msgr2=0x7fbef80c3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:33.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.623+0000 7fbeff00e640 1 -- 172.21.15.1:0/2513975911 shutdown_connections 2024-03-06T07:20:33.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.623+0000 7fbeff00e640 1 -- 172.21.15.1:0/2513975911 wait complete. 2024-03-06T07:20:33.636 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:20:19.337560+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:20:20.172791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:20:23.884071+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:20:33.637 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:20:33.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.760+0000 7f44ad756640 1 Processor -- start 2024-03-06T07:20:33.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.760+0000 7f44ad756640 1 -- start start 2024-03-06T07:20:33.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.761+0000 7f44ad756640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f44a80c3970 0x7f44a80c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.761+0000 7f44ad756640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a80c60b0 0x7f44a8152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.761+0000 7f44ad756640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f44a8152670 0x7f44a81553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.761+0000 7f44ad756640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44a80c67f0 con 0x7f44a80c3970 2024-03-06T07:20:33.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.761+0000 7f44ad756640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44a80c5e40 con 0x7f44a8152670 2024-03-06T07:20:33.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.761+0000 7f44ad756640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80c6a40 con 0x7f44a80c60b0 2024-03-06T07:20:33.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.761+0000 7f44a6ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f44a80c3970 0x7f44a80c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.761+0000 7f44a6ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f44a80c3970 0x7f44a80c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34988/0 (socket says 172.21.15.1:34988) 2024-03-06T07:20:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.761+0000 7f44a6ffd640 1 -- 172.21.15.1:0/2899939855 learned_addr learned my addr 172.21.15.1:0/2899939855 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.761+0000 7f44a77fe640 1 --2- 172.21.15.1:0/2899939855 >> v2:172.21.15.98:3300/0 conn(0x7f44a8152670 0x7f44a81553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.761+0000 7f449ffff640 1 --2- 172.21.15.1:0/2899939855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a80c60b0 0x7f44a8152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.761+0000 7f44a6ffd640 1 -- 172.21.15.1:0/2899939855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a80c60b0 msgr2=0x7f44a8152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.761+0000 7f44a6ffd640 1 --2- 172.21.15.1:0/2899939855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a80c60b0 0x7f44a8152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.761+0000 7f44a6ffd640 1 -- 172.21.15.1:0/2899939855 >> v2:172.21.15.98:3300/0 conn(0x7f44a8152670 msgr2=0x7f44a81553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.761+0000 7f44a6ffd640 1 --2- 172.21.15.1:0/2899939855 >> v2:172.21.15.98:3300/0 conn(0x7f44a8152670 0x7f44a81553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.761+0000 7f44a6ffd640 1 -- 172.21.15.1:0/2899939855 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a80c6d20 con 0x7f44a80c3970 2024-03-06T07:20:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.761+0000 7f44a77fe640 1 --2- 172.21.15.1:0/2899939855 >> v2:172.21.15.98:3300/0 conn(0x7f44a8152670 0x7f44a81553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.762+0000 7f44a6ffd640 1 --2- 172.21.15.1:0/2899939855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c3970 0x7f44a80c8170 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f44a80ae4a0 tx=0x7f449802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711b448aa5f66bcc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.762+0000 7f44a67fc640 1 -- 172.21.15.1:0/2899939855 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4498035020 con 0x7f44a80c3970 2024-03-06T07:20:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.762+0000 7f44a67fc640 1 -- 172.21.15.1:0/2899939855 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4498031870 con 0x7f44a80c3970 2024-03-06T07:20:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.762+0000 7f44a67fc640 1 -- 172.21.15.1:0/2899939855 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4498031c40 con 0x7f44a80c3970 2024-03-06T07:20:33.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.763+0000 7f44ad756640 1 -- 172.21.15.1:0/2899939855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c3970 msgr2=0x7f44a80c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.763+0000 7f44ad756640 1 --2- 172.21.15.1:0/2899939855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c3970 0x7f44a80c8170 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f44a80ae4a0 tx=0x7f449802b910 comp rx=0 tx=0).stop 2024-03-06T07:20:33.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.763+0000 7f44ad756640 1 -- 172.21.15.1:0/2899939855 shutdown_connections 2024-03-06T07:20:33.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.763+0000 7f44ad756640 1 --2- 172.21.15.1:0/2899939855 >> v2:172.21.15.98:3300/0 conn(0x7f44a8152670 0x7f44a81553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.763+0000 7f44ad756640 1 --2- 172.21.15.1:0/2899939855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a80c60b0 0x7f44a8152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.763+0000 7f44ad756640 1 --2- 172.21.15.1:0/2899939855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c3970 0x7f44a80c8170 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.763+0000 7f44ad756640 1 -- 172.21.15.1:0/2899939855 >> 172.21.15.1:0/2899939855 conn(0x7f44a80c88f0 msgr2=0x7f44a80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:33.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.763+0000 7f44ad756640 1 -- 172.21.15.1:0/2899939855 shutdown_connections 2024-03-06T07:20:33.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.763+0000 7f44ad756640 1 -- 172.21.15.1:0/2899939855 wait complete. 2024-03-06T07:20:33.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.764+0000 7f44ad756640 1 Processor -- start 2024-03-06T07:20:33.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.764+0000 7f44ad756640 1 -- start start 2024-03-06T07:20:33.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f44ad756640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44a80c3970 0x7f44a80707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f44ad756640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c60b0 0x7f44a80ab9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f44ad756640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a8152670 0x7f44a80abee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f44ad756640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80c75c0 con 0x7f44a80c60b0 2024-03-06T07:20:33.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f44ad756640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80c72e0 con 0x7f44a80c3970 2024-03-06T07:20:33.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f44ad756640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80c3d50 con 0x7f44a8152670 2024-03-06T07:20:33.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f449ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c60b0 0x7f44a80ab9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f449ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c60b0 0x7f44a80ab9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35002/0 (socket says 172.21.15.1:35002) 2024-03-06T07:20:33.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f449ffff640 1 -- 172.21.15.1:0/2039810991 learned_addr learned my addr 172.21.15.1:0/2039810991 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:33.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f44a6ffd640 1 --2- 172.21.15.1:0/2039810991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44a80c3970 0x7f44a80707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f44a77fe640 1 --2- 172.21.15.1:0/2039810991 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a8152670 0x7f44a80abee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f449ffff640 1 -- 172.21.15.1:0/2039810991 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a8152670 msgr2=0x7f44a80abee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f449ffff640 1 --2- 172.21.15.1:0/2039810991 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a8152670 0x7f44a80abee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f449ffff640 1 -- 172.21.15.1:0/2039810991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44a80c3970 msgr2=0x7f44a80707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f449ffff640 1 --2- 172.21.15.1:0/2039810991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44a80c3970 0x7f44a80707a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f449ffff640 1 -- 172.21.15.1:0/2039810991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a80ace20 con 0x7f44a80c60b0 2024-03-06T07:20:33.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f449ffff640 1 --2- 172.21.15.1:0/2039810991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c60b0 0x7f44a80ab9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f449000d800 tx=0x7f449000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f449f7fe640 1 -- 172.21.15.1:0/2039810991 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4490004030 con 0x7f44a80c60b0 2024-03-06T07:20:33.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f449f7fe640 1 -- 172.21.15.1:0/2039810991 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44900041f0 con 0x7f44a80c60b0 2024-03-06T07:20:33.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f44ad756640 1 -- 172.21.15.1:0/2039810991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a80adc90 con 0x7f44a80c60b0 2024-03-06T07:20:33.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f449f7fe640 1 -- 172.21.15.1:0/2039810991 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44900096a0 con 0x7f44a80c60b0 2024-03-06T07:20:33.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.765+0000 7f44ad756640 1 -- 172.21.15.1:0/2039810991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a80ad0c0 con 0x7f44a80c60b0 2024-03-06T07:20:33.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.766+0000 7f44ad756640 1 -- 172.21.15.1:0/2039810991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a80c50c0 con 0x7f44a80c60b0 2024-03-06T07:20:33.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.767+0000 7f449f7fe640 1 -- 172.21.15.1:0/2039810991 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4490009840 con 0x7f44a80c60b0 2024-03-06T07:20:33.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.767+0000 7f449f7fe640 1 --2- 172.21.15.1:0/2039810991 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 0x7f448806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.767+0000 7f449f7fe640 1 -- 172.21.15.1:0/2039810991 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f449008a710 con 0x7f44a80c60b0 2024-03-06T07:20:33.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.768+0000 7f44a6ffd640 1 --2- 172.21.15.1:0/2039810991 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 0x7f448806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.768+0000 7f44a6ffd640 1 --2- 172.21.15.1:0/2039810991 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 0x7f448806d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4498036000 tx=0x7f449803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.772+0000 7f449f7fe640 1 -- 172.21.15.1:0/2039810991 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44a80c50c0 con 0x7f44a80c60b0 2024-03-06T07:20:33.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.969+0000 7f44ad756640 1 -- 172.21.15.1:0/2039810991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f44a80686e0 con 0x7f44a80c60b0 2024-03-06T07:20:33.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.969+0000 7f449f7fe640 1 -- 172.21.15.1:0/2039810991 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4490058bf0 con 0x7f44a80c60b0 2024-03-06T07:20:33.971 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:20:33.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.971+0000 7f44ad756640 1 -- 172.21.15.1:0/2039810991 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 msgr2=0x7f448806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.971+0000 7f44ad756640 1 --2- 172.21.15.1:0/2039810991 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 0x7f448806d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4498036000 tx=0x7f449803c040 comp rx=0 tx=0).stop 2024-03-06T07:20:33.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.971+0000 7f44ad756640 1 -- 172.21.15.1:0/2039810991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c60b0 msgr2=0x7f44a80ab9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.971+0000 7f44ad756640 1 --2- 172.21.15.1:0/2039810991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c60b0 0x7f44a80ab9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f449000d800 tx=0x7f449000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.971+0000 7f44ad756640 1 -- 172.21.15.1:0/2039810991 shutdown_connections 2024-03-06T07:20:33.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.971+0000 7f44ad756640 1 --2- 172.21.15.1:0/2039810991 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 0x7f448806d990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.971+0000 7f44ad756640 1 --2- 172.21.15.1:0/2039810991 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a8152670 0x7f44a80abee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.971+0000 7f44ad756640 1 --2- 172.21.15.1:0/2039810991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c60b0 0x7f44a80ab9a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.971+0000 7f44ad756640 1 --2- 172.21.15.1:0/2039810991 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44a80c3970 0x7f44a80707a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.971+0000 7f44ad756640 1 -- 172.21.15.1:0/2039810991 >> 172.21.15.1:0/2039810991 conn(0x7f44a80c88f0 msgr2=0x7f44a80aaf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:33.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.971+0000 7f44ad756640 1 -- 172.21.15.1:0/2039810991 shutdown_connections 2024-03-06T07:20:33.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:33.971+0000 7f44ad756640 1 -- 172.21.15.1:0/2039810991 wait complete. 2024-03-06T07:20:33.983 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:20:34.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.086+0000 7fac98e76640 1 Processor -- start 2024-03-06T07:20:34.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.086+0000 7fac98e76640 1 -- start start 2024-03-06T07:20:34.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.087+0000 7fac98e76640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac94152650 0x7fac94155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.087+0000 7fac98e76640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac940bf9d0 0x7fac940c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.087+0000 7fac98e76640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac940c2fc0 con 0x7fac940c6b10 2024-03-06T07:20:34.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.087+0000 7fac98e76640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac94155800 con 0x7fac940bf9d0 2024-03-06T07:20:34.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.087+0000 7fac98e76640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac94158da0 con 0x7fac94152650 2024-03-06T07:20:34.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.087+0000 7fac92d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac940bf9d0 0x7fac940c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.087+0000 7fac91d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fac940c6b10 0x7fac94158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43044/0 (socket says 172.21.15.1:43044) 2024-03-06T07:20:34.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.087+0000 7fac91d9b640 1 -- 172.21.15.1:0/3499066934 learned_addr learned my addr 172.21.15.1:0/3499066934 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:34.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.088+0000 7fac9259c640 1 --2- 172.21.15.1:0/3499066934 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac94152650 0x7fac94155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.088+0000 7fac92d9d640 1 -- 172.21.15.1:0/3499066934 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac94152650 msgr2=0x7fac94155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:34.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.088+0000 7fac92d9d640 1 --2- 172.21.15.1:0/3499066934 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac94152650 0x7fac94155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.088+0000 7fac92d9d640 1 -- 172.21.15.1:0/3499066934 >> v1:172.21.15.1:6789/0 conn(0x7fac940c6b10 legacy=0x7fac94158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.088+0000 7fac92d9d640 1 -- 172.21.15.1:0/3499066934 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac940c6690 con 0x7fac940bf9d0 2024-03-06T07:20:34.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.088+0000 7fac9159a640 1 -- 172.21.15.1:0/3499066934 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077525013 0 0) 0x7fac940c2fc0 con 0x7fac940c6b10 2024-03-06T07:20:34.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.088+0000 7fac92d9d640 1 --2- 172.21.15.1:0/3499066934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac940bf9d0 0x7fac940c1460 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fac84009d90 tx=0x7fac8402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e4b6fc11c29240 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:34.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.089+0000 7fac9159a640 1 -- 172.21.15.1:0/3499066934 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84039070 con 0x7fac940bf9d0 2024-03-06T07:20:34.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.089+0000 7fac9159a640 1 -- 172.21.15.1:0/3499066934 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8402cbc0 con 0x7fac940bf9d0 2024-03-06T07:20:34.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.089+0000 7fac9159a640 1 -- 172.21.15.1:0/3499066934 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84034440 con 0x7fac940bf9d0 2024-03-06T07:20:34.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.089+0000 7fac98e76640 1 -- 172.21.15.1:0/3499066934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac940bf9d0 msgr2=0x7fac940c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.089+0000 7fac98e76640 1 --2- 172.21.15.1:0/3499066934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac940bf9d0 0x7fac940c1460 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fac84009d90 tx=0x7fac8402be10 comp rx=0 tx=0).stop 2024-03-06T07:20:34.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.089+0000 7fac98e76640 1 -- 172.21.15.1:0/3499066934 shutdown_connections 2024-03-06T07:20:34.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.089+0000 7fac98e76640 1 --2- 172.21.15.1:0/3499066934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac940bf9d0 0x7fac940c1460 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.089+0000 7fac98e76640 1 --2- 172.21.15.1:0/3499066934 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac94152650 0x7fac94155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.089+0000 7fac98e76640 1 -- 172.21.15.1:0/3499066934 >> 172.21.15.1:0/3499066934 conn(0x7fac94066ee0 msgr2=0x7fac940672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:34.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.090+0000 7fac98e76640 1 -- 172.21.15.1:0/3499066934 shutdown_connections 2024-03-06T07:20:34.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.090+0000 7fac98e76640 1 -- 172.21.15.1:0/3499066934 wait complete. 2024-03-06T07:20:34.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.091+0000 7fac98e76640 1 Processor -- start 2024-03-06T07:20:34.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.091+0000 7fac98e76640 1 -- start start 2024-03-06T07:20:34.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.091+0000 7fac98e76640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac940bf9d0 0x7fac941eedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac98e76640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac94152650 0x7fac940c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac98e76640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac941eb460 0x7fac941eb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac98e76640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac941551b0 con 0x7fac941eb460 2024-03-06T07:20:34.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac98e76640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac940c6fa0 con 0x7fac940bf9d0 2024-03-06T07:20:34.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac98e76640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac94155490 con 0x7fac94152650 2024-03-06T07:20:34.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac92d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac941eb460 0x7fac941eb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac92d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac941eb460 0x7fac941eb910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35014/0 (socket says 172.21.15.1:35014) 2024-03-06T07:20:34.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac92d9d640 1 -- 172.21.15.1:0/3774889297 learned_addr learned my addr 172.21.15.1:0/3774889297 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:34.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac9259c640 1 --2- 172.21.15.1:0/3774889297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac940bf9d0 0x7fac941eedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac91d9b640 1 --2- 172.21.15.1:0/3774889297 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac94152650 0x7fac940c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac92d9d640 1 -- 172.21.15.1:0/3774889297 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac94152650 msgr2=0x7fac940c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac92d9d640 1 --2- 172.21.15.1:0/3774889297 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac94152650 0x7fac940c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac92d9d640 1 -- 172.21.15.1:0/3774889297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac940bf9d0 msgr2=0x7fac941eedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac92d9d640 1 --2- 172.21.15.1:0/3774889297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac940bf9d0 0x7fac941eedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac92d9d640 1 -- 172.21.15.1:0/3774889297 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac94154f90 con 0x7fac941eb460 2024-03-06T07:20:34.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac91d9b640 1 --2- 172.21.15.1:0/3774889297 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac94152650 0x7fac940c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:34.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac92d9d640 1 --2- 172.21.15.1:0/3774889297 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac941eb460 0x7fac941eb910 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fac84009d90 tx=0x7fac84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:34.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac8affd640 1 -- 172.21.15.1:0/3774889297 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84039050 con 0x7fac941eb460 2024-03-06T07:20:34.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac8affd640 1 -- 172.21.15.1:0/3774889297 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac84045070 con 0x7fac941eb460 2024-03-06T07:20:34.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac98e76640 1 -- 172.21.15.1:0/3774889297 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac941549d0 con 0x7fac941eb460 2024-03-06T07:20:34.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac8affd640 1 -- 172.21.15.1:0/3774889297 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8403e830 con 0x7fac941eb460 2024-03-06T07:20:34.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.092+0000 7fac98e76640 1 -- 172.21.15.1:0/3774889297 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac940ca500 con 0x7fac941eb460 2024-03-06T07:20:34.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.094+0000 7fac98e76640 1 -- 172.21.15.1:0/3774889297 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac94152200 con 0x7fac941eb460 2024-03-06T07:20:34.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.094+0000 7fac8affd640 1 -- 172.21.15.1:0/3774889297 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac8403e9d0 con 0x7fac941eb460 2024-03-06T07:20:34.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.094+0000 7fac8affd640 1 --2- 172.21.15.1:0/3774889297 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac7406b4d0 0x7fac7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.094+0000 7fac8affd640 1 -- 172.21.15.1:0/3774889297 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fac840ad480 con 0x7fac941eb460 2024-03-06T07:20:34.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.094+0000 7fac9259c640 1 --2- 172.21.15.1:0/3774889297 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac7406b4d0 0x7fac7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.095+0000 7fac9259c640 1 --2- 172.21.15.1:0/3774889297 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac7406b4d0 0x7fac7406d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fac8c006fd0 tx=0x7fac8c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:34.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.099+0000 7fac8affd640 1 -- 172.21.15.1:0/3774889297 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac94152200 con 0x7fac941eb460 2024-03-06T07:20:34.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.279+0000 7fac98e76640 1 -- 172.21.15.1:0/3774889297 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fac94067fd0 con 0x7fac941eb460 2024-03-06T07:20:34.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.279+0000 7fac8affd640 1 -- 172.21.15.1:0/3774889297 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fac8407c170 con 0x7fac941eb460 2024-03-06T07:20:34.281 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:20:34.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.282+0000 7fac98e76640 1 -- 172.21.15.1:0/3774889297 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac7406b4d0 msgr2=0x7fac7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.282+0000 7fac98e76640 1 --2- 172.21.15.1:0/3774889297 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac7406b4d0 0x7fac7406d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fac8c006fd0 tx=0x7fac8c008040 comp rx=0 tx=0).stop 2024-03-06T07:20:34.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.282+0000 7fac98e76640 1 -- 172.21.15.1:0/3774889297 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac941eb460 msgr2=0x7fac941eb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.282+0000 7fac98e76640 1 --2- 172.21.15.1:0/3774889297 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac941eb460 0x7fac941eb910 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fac84009d90 tx=0x7fac84004a40 comp rx=0 tx=0).stop 2024-03-06T07:20:34.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.282+0000 7fac98e76640 1 -- 172.21.15.1:0/3774889297 shutdown_connections 2024-03-06T07:20:34.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.282+0000 7fac98e76640 1 --2- 172.21.15.1:0/3774889297 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac7406b4d0 0x7fac7406d990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.282+0000 7fac98e76640 1 --2- 172.21.15.1:0/3774889297 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac941eb460 0x7fac941eb910 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.282+0000 7fac98e76640 1 --2- 172.21.15.1:0/3774889297 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac94152650 0x7fac940c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.282+0000 7fac98e76640 1 --2- 172.21.15.1:0/3774889297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac940bf9d0 0x7fac941eedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.282+0000 7fac98e76640 1 -- 172.21.15.1:0/3774889297 >> 172.21.15.1:0/3774889297 conn(0x7fac94066ee0 msgr2=0x7fac940672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:34.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.282+0000 7fac98e76640 1 -- 172.21.15.1:0/3774889297 shutdown_connections 2024-03-06T07:20:34.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.282+0000 7fac98e76640 1 -- 172.21.15.1:0/3774889297 wait complete. 2024-03-06T07:20:34.295 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:20:34.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.397+0000 7f89ec99a640 1 Processor -- start 2024-03-06T07:20:34.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.397+0000 7f89ec99a640 1 -- start start 2024-03-06T07:20:34.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.397+0000 7f89ec99a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89e80c2b30 0x7f89e80724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.398+0000 7f89ec99a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e8151dd0 con 0x7f89e8152b70 2024-03-06T07:20:34.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.398+0000 7f89ec99a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e8156190 con 0x7f89e80a5fa0 2024-03-06T07:20:34.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.398+0000 7f89ec99a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e81526e0 con 0x7f89e80c2b30 2024-03-06T07:20:34.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.398+0000 7f89ddd9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f89e8152b70 0x7f89e80c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57348/0 (socket says 172.21.15.1:57348) 2024-03-06T07:20:34.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.398+0000 7f89ddd9b640 1 -- 172.21.15.1:0/4233750890 learned_addr learned my addr 172.21.15.1:0/4233750890 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:34.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.398+0000 7f89e659c640 1 --2- 172.21.15.1:0/4233750890 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89e80c2b30 0x7f89e80724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.398+0000 7f89e5d9b640 1 -- 172.21.15.1:0/4233750890 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135571533 0 0) 0x7f89e8151dd0 con 0x7f89e8152b70 2024-03-06T07:20:34.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.398+0000 7f89e5d9b640 1 -- 172.21.15.1:0/4233750890 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89d4003520 con 0x7f89e8152b70 2024-03-06T07:20:34.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.398+0000 7f89e5d9b640 1 -- 172.21.15.1:0/4233750890 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984762874 0 0) 0x7f89e8156190 con 0x7f89e80a5fa0 2024-03-06T07:20:34.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.398+0000 7f89e5d9b640 1 -- 172.21.15.1:0/4233750890 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e8151dd0 con 0x7f89e80a5fa0 2024-03-06T07:20:34.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.398+0000 7f89e659c640 1 -- 172.21.15.1:0/4233750890 >> v1:172.21.15.98:6789/0 conn(0x7f89e80a5fa0 legacy=0x7f89e815c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.398+0000 7f89e659c640 1 -- 172.21.15.1:0/4233750890 >> v1:172.21.15.1:6789/0 conn(0x7f89e8152b70 legacy=0x7f89e80c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.398+0000 7f89e659c640 1 -- 172.21.15.1:0/4233750890 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e8152090 con 0x7f89e80c2b30 2024-03-06T07:20:34.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.399+0000 7f89e659c640 1 --2- 172.21.15.1:0/4233750890 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89e80c2b30 0x7f89e80724c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f89d80097d0 tx=0x7f89d802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c6390a747cefcb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:34.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.399+0000 7f89e5d9b640 1 -- 172.21.15.1:0/4233750890 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d8035020 con 0x7f89e80c2b30 2024-03-06T07:20:34.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.399+0000 7f89e5d9b640 1 -- 172.21.15.1:0/4233750890 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d8031890 con 0x7f89e80c2b30 2024-03-06T07:20:34.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.399+0000 7f89e5d9b640 1 -- 172.21.15.1:0/4233750890 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d8031c40 con 0x7f89e80c2b30 2024-03-06T07:20:34.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.400+0000 7f89ec99a640 1 -- 172.21.15.1:0/4233750890 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89e80c2b30 msgr2=0x7f89e80724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.400+0000 7f89ec99a640 1 --2- 172.21.15.1:0/4233750890 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89e80c2b30 0x7f89e80724c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f89d80097d0 tx=0x7f89d802b930 comp rx=0 tx=0).stop 2024-03-06T07:20:34.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.400+0000 7f89ec99a640 1 -- 172.21.15.1:0/4233750890 shutdown_connections 2024-03-06T07:20:34.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.400+0000 7f89ec99a640 1 --2- 172.21.15.1:0/4233750890 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89e80c2b30 0x7f89e80724c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.400+0000 7f89ec99a640 1 -- 172.21.15.1:0/4233750890 >> 172.21.15.1:0/4233750890 conn(0x7f89e80c6860 msgr2=0x7f89e80c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:34.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.400+0000 7f89ec99a640 1 -- 172.21.15.1:0/4233750890 shutdown_connections 2024-03-06T07:20:34.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.400+0000 7f89ec99a640 1 -- 172.21.15.1:0/4233750890 wait complete. 2024-03-06T07:20:34.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.401+0000 7f89ec99a640 1 Processor -- start 2024-03-06T07:20:34.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.401+0000 7f89ec99a640 1 -- start start 2024-03-06T07:20:34.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89ec99a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89e80a5fa0 0x7f89e8149a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89ec99a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e80c2b30 0x7f89e8149fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89ec99a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e8152b70 0x7f89e8142ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89ec99a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e8155f00 con 0x7f89e8152b70 2024-03-06T07:20:34.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89ec99a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e8155d80 con 0x7f89e80c2b30 2024-03-06T07:20:34.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89ec99a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e8072a00 con 0x7f89e80a5fa0 2024-03-06T07:20:34.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89e6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e8152b70 0x7f89e8142ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89e6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e8152b70 0x7f89e8142ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40194/0 (socket says 172.21.15.1:40194) 2024-03-06T07:20:34.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89e6d9d640 1 -- 172.21.15.1:0/2239227209 learned_addr learned my addr 172.21.15.1:0/2239227209 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:34.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89e659c640 1 --2- 172.21.15.1:0/2239227209 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89e80a5fa0 0x7f89e8149a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89ddd9b640 1 --2- 172.21.15.1:0/2239227209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e80c2b30 0x7f89e8149fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89e6d9d640 1 -- 172.21.15.1:0/2239227209 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89e80a5fa0 msgr2=0x7f89e8149a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89e6d9d640 1 --2- 172.21.15.1:0/2239227209 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89e80a5fa0 0x7f89e8149a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89e6d9d640 1 -- 172.21.15.1:0/2239227209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e80c2b30 msgr2=0x7f89e8149fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89e6d9d640 1 --2- 172.21.15.1:0/2239227209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e80c2b30 0x7f89e8149fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89e6d9d640 1 -- 172.21.15.1:0/2239227209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e81437b0 con 0x7f89e8152b70 2024-03-06T07:20:34.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89e659c640 1 --2- 172.21.15.1:0/2239227209 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89e80a5fa0 0x7f89e8149a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:34.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89ddd9b640 1 --2- 172.21.15.1:0/2239227209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e80c2b30 0x7f89e8149fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:34.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.402+0000 7f89e6d9d640 1 --2- 172.21.15.1:0/2239227209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e8152b70 0x7f89e8142ad0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f89e000c790 tx=0x7f89e000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:34.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.403+0000 7f89df7fe640 1 -- 172.21.15.1:0/2239227209 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e0003b70 con 0x7f89e8152b70 2024-03-06T07:20:34.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.403+0000 7f89df7fe640 1 -- 172.21.15.1:0/2239227209 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e0003d10 con 0x7f89e8152b70 2024-03-06T07:20:34.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.403+0000 7f89ec99a640 1 -- 172.21.15.1:0/2239227209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e81dee10 con 0x7f89e8152b70 2024-03-06T07:20:34.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.403+0000 7f89df7fe640 1 -- 172.21.15.1:0/2239227209 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e000aa50 con 0x7f89e8152b70 2024-03-06T07:20:34.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.403+0000 7f89ec99a640 1 -- 172.21.15.1:0/2239227209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e81e9da0 con 0x7f89e8152b70 2024-03-06T07:20:34.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.404+0000 7f89ec99a640 1 -- 172.21.15.1:0/2239227209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e8067fd0 con 0x7f89e8152b70 2024-03-06T07:20:34.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.404+0000 7f89df7fe640 1 -- 172.21.15.1:0/2239227209 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89e000abf0 con 0x7f89e8152b70 2024-03-06T07:20:34.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.405+0000 7f89df7fe640 1 --2- 172.21.15.1:0/2239227209 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f89c806b4d0 0x7f89c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.405+0000 7f89df7fe640 1 -- 172.21.15.1:0/2239227209 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f89e008a4a0 con 0x7f89e8152b70 2024-03-06T07:20:34.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.405+0000 7f89e659c640 1 --2- 172.21.15.1:0/2239227209 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f89c806b4d0 0x7f89c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.405+0000 7f89e659c640 1 --2- 172.21.15.1:0/2239227209 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f89c806b4d0 0x7f89c806d990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f89d8036000 tx=0x7f89d8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:34.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.409+0000 7f89df7fe640 1 -- 172.21.15.1:0/2239227209 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89e0059070 con 0x7f89e8152b70 2024-03-06T07:20:34.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.592+0000 7f89ec99a640 1 -- 172.21.15.1:0/2239227209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f89e8071790 con 0x7f89e8152b70 2024-03-06T07:20:34.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.592+0000 7f89df7fe640 1 -- 172.21.15.1:0/2239227209 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f89e0058970 con 0x7f89e8152b70 2024-03-06T07:20:34.594 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:20:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.594+0000 7f89ec99a640 1 -- 172.21.15.1:0/2239227209 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f89c806b4d0 msgr2=0x7f89c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.594+0000 7f89ec99a640 1 --2- 172.21.15.1:0/2239227209 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f89c806b4d0 0x7f89c806d990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f89d8036000 tx=0x7f89d8008030 comp rx=0 tx=0).stop 2024-03-06T07:20:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.594+0000 7f89ec99a640 1 -- 172.21.15.1:0/2239227209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e8152b70 msgr2=0x7f89e8142ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.594+0000 7f89ec99a640 1 --2- 172.21.15.1:0/2239227209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e8152b70 0x7f89e8142ad0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f89e000c790 tx=0x7f89e000cc50 comp rx=0 tx=0).stop 2024-03-06T07:20:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.594+0000 7f89ec99a640 1 -- 172.21.15.1:0/2239227209 shutdown_connections 2024-03-06T07:20:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.594+0000 7f89ec99a640 1 --2- 172.21.15.1:0/2239227209 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f89c806b4d0 0x7f89c806d990 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.594+0000 7f89ec99a640 1 --2- 172.21.15.1:0/2239227209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e8152b70 0x7f89e8142ad0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.594+0000 7f89ec99a640 1 --2- 172.21.15.1:0/2239227209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e80c2b30 0x7f89e8149fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.594+0000 7f89ec99a640 1 --2- 172.21.15.1:0/2239227209 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89e80a5fa0 0x7f89e8149a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.594+0000 7f89ec99a640 1 -- 172.21.15.1:0/2239227209 >> 172.21.15.1:0/2239227209 conn(0x7f89e80c6860 msgr2=0x7f89e806ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.594+0000 7f89ec99a640 1 -- 172.21.15.1:0/2239227209 shutdown_connections 2024-03-06T07:20:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.594+0000 7f89ec99a640 1 -- 172.21.15.1:0/2239227209 wait complete. 2024-03-06T07:20:34.610 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:20:34.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.714+0000 7f14885f7640 1 Processor -- start 2024-03-06T07:20:34.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.714+0000 7f14885f7640 1 -- start start 2024-03-06T07:20:34.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.714+0000 7f14885f7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f14801506d0 0x7f1480153650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.714+0000 7f14885f7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f14800c32f0 0x7f14800712f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.714+0000 7f14885f7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1480152e70 con 0x7f1480153b90 2024-03-06T07:20:34.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.714+0000 7f14885f7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1480071830 con 0x7f14800c32f0 2024-03-06T07:20:34.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.714+0000 7f14885f7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14800c7fa0 con 0x7f14801506d0 2024-03-06T07:20:34.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.715+0000 7f1486b94640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f14800c32f0 0x7f14800712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.715+0000 7f1486b94640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f14800c32f0 0x7f14800712f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40206/0 (socket says 172.21.15.1:40206) 2024-03-06T07:20:34.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.715+0000 7f1486b94640 1 -- 172.21.15.1:0/435864192 learned_addr learned my addr 172.21.15.1:0/435864192 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:34.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.715+0000 7f1486393640 1 --2- 172.21.15.1:0/435864192 >> v2:172.21.15.98:3300/0 conn(0x7f14801506d0 0x7f1480153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.715+0000 7f1486b94640 1 -- 172.21.15.1:0/435864192 >> v2:172.21.15.98:3300/0 conn(0x7f14801506d0 msgr2=0x7f1480153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:34.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.715+0000 7f1486b94640 1 --2- 172.21.15.1:0/435864192 >> v2:172.21.15.98:3300/0 conn(0x7f14801506d0 0x7f1480153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.715+0000 7f1486b94640 1 -- 172.21.15.1:0/435864192 >> v1:172.21.15.98:6789/0 conn(0x7f1480153b90 legacy=0x7f1480070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.715+0000 7f1486b94640 1 -- 172.21.15.1:0/435864192 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14800c7cc0 con 0x7f14800c32f0 2024-03-06T07:20:34.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.716+0000 7f1486b94640 1 --2- 172.21.15.1:0/435864192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800c32f0 0x7f14800712f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f148006e370 tx=0x7f147802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b0708efa9a0c2f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:34.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.716+0000 7f1485391640 1 -- 172.21.15.1:0/435864192 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147802cb20 con 0x7f14800c32f0 2024-03-06T07:20:34.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.716+0000 7f1485391640 1 -- 172.21.15.1:0/435864192 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f147802ccc0 con 0x7f14800c32f0 2024-03-06T07:20:34.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.716+0000 7f1485391640 1 -- 172.21.15.1:0/435864192 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1478034610 con 0x7f14800c32f0 2024-03-06T07:20:34.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.716+0000 7f14885f7640 1 -- 172.21.15.1:0/435864192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800c32f0 msgr2=0x7f14800712f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.716+0000 7f14885f7640 1 --2- 172.21.15.1:0/435864192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800c32f0 0x7f14800712f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f148006e370 tx=0x7f147802bf80 comp rx=0 tx=0).stop 2024-03-06T07:20:34.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.716+0000 7f14885f7640 1 -- 172.21.15.1:0/435864192 shutdown_connections 2024-03-06T07:20:34.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.716+0000 7f14885f7640 1 --2- 172.21.15.1:0/435864192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800c32f0 0x7f14800712f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.716+0000 7f14885f7640 1 --2- 172.21.15.1:0/435864192 >> v2:172.21.15.98:3300/0 conn(0x7f14801506d0 0x7f1480153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.716+0000 7f14885f7640 1 -- 172.21.15.1:0/435864192 >> 172.21.15.1:0/435864192 conn(0x7f14800c88f0 msgr2=0x7f14800c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:34.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.716+0000 7f14885f7640 1 -- 172.21.15.1:0/435864192 shutdown_connections 2024-03-06T07:20:34.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.717+0000 7f14885f7640 1 -- 172.21.15.1:0/435864192 wait complete. 2024-03-06T07:20:34.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.718+0000 7f14885f7640 1 Processor -- start 2024-03-06T07:20:34.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.718+0000 7f14885f7640 1 -- start start 2024-03-06T07:20:34.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.718+0000 7f14885f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14800c32f0 0x7f148014d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.718+0000 7f14885f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14801506d0 0x7f14801dce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.718+0000 7f14885f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800adae0 0x7f14801ee4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.718+0000 7f14885f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14801571f0 con 0x7f14800adae0 2024-03-06T07:20:34.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.718+0000 7f14885f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14801531c0 con 0x7f14800c32f0 2024-03-06T07:20:34.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.718+0000 7f14885f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148006f4b0 con 0x7f14801506d0 2024-03-06T07:20:34.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.718+0000 7f1486b94640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800adae0 0x7f14801ee4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.718+0000 7f1486b94640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800adae0 0x7f14801ee4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40212/0 (socket says 172.21.15.1:40212) 2024-03-06T07:20:34.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.718+0000 7f1486b94640 1 -- 172.21.15.1:0/1970916002 learned_addr learned my addr 172.21.15.1:0/1970916002 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:34.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.719+0000 7f1486393640 1 --2- 172.21.15.1:0/1970916002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14800c32f0 0x7f148014d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.719+0000 7f1485b92640 1 --2- 172.21.15.1:0/1970916002 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14801506d0 0x7f14801dce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.719+0000 7f1486b94640 1 -- 172.21.15.1:0/1970916002 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14801506d0 msgr2=0x7f14801dce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.719+0000 7f1486b94640 1 --2- 172.21.15.1:0/1970916002 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14801506d0 0x7f14801dce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.719+0000 7f1486b94640 1 -- 172.21.15.1:0/1970916002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14800c32f0 msgr2=0x7f148014d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.719+0000 7f1486b94640 1 --2- 172.21.15.1:0/1970916002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14800c32f0 0x7f148014d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.719+0000 7f1486b94640 1 -- 172.21.15.1:0/1970916002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14801dd570 con 0x7f14800adae0 2024-03-06T07:20:34.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.719+0000 7f1485b92640 1 --2- 172.21.15.1:0/1970916002 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14801506d0 0x7f14801dce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:34.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.719+0000 7f1486b94640 1 --2- 172.21.15.1:0/1970916002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800adae0 0x7f14801ee4e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1478036040 tx=0x7f1478004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:34.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.719+0000 7f1476ffd640 1 -- 172.21.15.1:0/1970916002 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1478043020 con 0x7f14800adae0 2024-03-06T07:20:34.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.719+0000 7f1476ffd640 1 -- 172.21.15.1:0/1970916002 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f147803e450 con 0x7f14800adae0 2024-03-06T07:20:34.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.719+0000 7f1476ffd640 1 -- 172.21.15.1:0/1970916002 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1478009510 con 0x7f14800adae0 2024-03-06T07:20:34.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.719+0000 7f14885f7640 1 -- 172.21.15.1:0/1970916002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14801eee20 con 0x7f14800adae0 2024-03-06T07:20:34.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.719+0000 7f14885f7640 1 -- 172.21.15.1:0/1970916002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14801ef100 con 0x7f14800adae0 2024-03-06T07:20:34.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.720+0000 7f14885f7640 1 -- 172.21.15.1:0/1970916002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148006e8d0 con 0x7f14800adae0 2024-03-06T07:20:34.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.721+0000 7f1476ffd640 1 -- 172.21.15.1:0/1970916002 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1478046020 con 0x7f14800adae0 2024-03-06T07:20:34.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.721+0000 7f1476ffd640 1 --2- 172.21.15.1:0/1970916002 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f146406b4d0 0x7f146406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.721+0000 7f1476ffd640 1 -- 172.21.15.1:0/1970916002 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f14780acc80 con 0x7f14800adae0 2024-03-06T07:20:34.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.722+0000 7f1486393640 1 --2- 172.21.15.1:0/1970916002 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f146406b4d0 0x7f146406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.722+0000 7f1486393640 1 --2- 172.21.15.1:0/1970916002 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f146406b4d0 0x7f146406d990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f14800c53e0 tx=0x7f147c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:34.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.726+0000 7f1476ffd640 1 -- 172.21.15.1:0/1970916002 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f148006e8d0 con 0x7f14800adae0 2024-03-06T07:20:34.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.906+0000 7f14885f7640 1 -- 172.21.15.1:0/1970916002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f148014eba0 con 0x7f14800adae0 2024-03-06T07:20:34.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.906+0000 7f1476ffd640 1 -- 172.21.15.1:0/1970916002 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f148014eba0 con 0x7f14800adae0 2024-03-06T07:20:34.912 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:20:34.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.908+0000 7f14885f7640 1 -- 172.21.15.1:0/1970916002 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f146406b4d0 msgr2=0x7f146406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.908+0000 7f14885f7640 1 --2- 172.21.15.1:0/1970916002 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f146406b4d0 0x7f146406d990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f14800c53e0 tx=0x7f147c009190 comp rx=0 tx=0).stop 2024-03-06T07:20:34.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.908+0000 7f14885f7640 1 -- 172.21.15.1:0/1970916002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800adae0 msgr2=0x7f14801ee4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.908+0000 7f14885f7640 1 --2- 172.21.15.1:0/1970916002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800adae0 0x7f14801ee4e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1478036040 tx=0x7f1478004a40 comp rx=0 tx=0).stop 2024-03-06T07:20:34.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.908+0000 7f14885f7640 1 -- 172.21.15.1:0/1970916002 shutdown_connections 2024-03-06T07:20:34.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.908+0000 7f14885f7640 1 --2- 172.21.15.1:0/1970916002 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f146406b4d0 0x7f146406d990 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.908+0000 7f14885f7640 1 --2- 172.21.15.1:0/1970916002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800adae0 0x7f14801ee4e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.908+0000 7f14885f7640 1 --2- 172.21.15.1:0/1970916002 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14801506d0 0x7f14801dce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.908+0000 7f14885f7640 1 --2- 172.21.15.1:0/1970916002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14800c32f0 0x7f148014d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.908+0000 7f14885f7640 1 -- 172.21.15.1:0/1970916002 >> 172.21.15.1:0/1970916002 conn(0x7f14800c88f0 msgr2=0x7f14800c2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:34.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.908+0000 7f14885f7640 1 -- 172.21.15.1:0/1970916002 shutdown_connections 2024-03-06T07:20:34.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:34.908+0000 7f14885f7640 1 -- 172.21.15.1:0/1970916002 wait complete. 2024-03-06T07:20:34.921 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:20:35.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.022+0000 7efc91de9640 1 Processor -- start 2024-03-06T07:20:35.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.022+0000 7efc91de9640 1 -- start start 2024-03-06T07:20:35.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.023+0000 7efc91de9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc8c06d060 0x7efc8c1558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.023+0000 7efc91de9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8c0c71c0 con 0x7efc8c0c3eb0 2024-03-06T07:20:35.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.023+0000 7efc91de9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8c0c4e50 con 0x7efc8c0af640 2024-03-06T07:20:35.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.023+0000 7efc91de9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c0c59b0 con 0x7efc8c06d060 2024-03-06T07:20:35.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.023+0000 7efc8b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efc8c0c3eb0 0x7efc8c0c7400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57358/0 (socket says 172.21.15.1:57358) 2024-03-06T07:20:35.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.023+0000 7efc8b7fe640 1 -- 172.21.15.1:0/1306739575 learned_addr learned my addr 172.21.15.1:0/1306739575 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:35.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.023+0000 7efc8affd640 1 --2- 172.21.15.1:0/1306739575 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc8c06d060 0x7efc8c1558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.024+0000 7efc8a7fc640 1 -- 172.21.15.1:0/1306739575 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743738085 0 0) 0x7efc8c0c71c0 con 0x7efc8c0c3eb0 2024-03-06T07:20:35.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.024+0000 7efc8a7fc640 1 -- 172.21.15.1:0/1306739575 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc78003520 con 0x7efc8c0c3eb0 2024-03-06T07:20:35.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.024+0000 7efc8a7fc640 1 -- 172.21.15.1:0/1306739575 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748423614 0 0) 0x7efc8c0c4e50 con 0x7efc8c0af640 2024-03-06T07:20:35.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.024+0000 7efc8a7fc640 1 -- 172.21.15.1:0/1306739575 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc8c0c71c0 con 0x7efc8c0af640 2024-03-06T07:20:35.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.024+0000 7efc8affd640 1 -- 172.21.15.1:0/1306739575 >> v1:172.21.15.98:6789/0 conn(0x7efc8c0af640 legacy=0x7efc8c15fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.024+0000 7efc8a7fc640 1 -- 172.21.15.1:0/1306739575 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2754835575 0 0) 0x7efc78003520 con 0x7efc8c0c3eb0 2024-03-06T07:20:35.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.024+0000 7efc8affd640 1 -- 172.21.15.1:0/1306739575 >> v1:172.21.15.1:6789/0 conn(0x7efc8c0c3eb0 legacy=0x7efc8c0c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.024+0000 7efc8affd640 1 -- 172.21.15.1:0/1306739575 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c0c6ee0 con 0x7efc8c06d060 2024-03-06T07:20:35.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.025+0000 7efc8affd640 1 --2- 172.21.15.1:0/1306739575 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc8c06d060 0x7efc8c1558b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efc740097d0 tx=0x7efc7402b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b96fa908bdc94301 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:35.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.025+0000 7efc8a7fc640 1 -- 172.21.15.1:0/1306739575 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc74037070 con 0x7efc8c06d060 2024-03-06T07:20:35.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.025+0000 7efc8a7fc640 1 -- 172.21.15.1:0/1306739575 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc74030510 con 0x7efc8c06d060 2024-03-06T07:20:35.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.025+0000 7efc8a7fc640 1 -- 172.21.15.1:0/1306739575 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7c001670 con 0x7efc8c06d060 2024-03-06T07:20:35.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.025+0000 7efc91de9640 1 -- 172.21.15.1:0/1306739575 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc8c06d060 msgr2=0x7efc8c1558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.025+0000 7efc91de9640 1 --2- 172.21.15.1:0/1306739575 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc8c06d060 0x7efc8c1558b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efc740097d0 tx=0x7efc7402b6e0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.025+0000 7efc91de9640 1 -- 172.21.15.1:0/1306739575 shutdown_connections 2024-03-06T07:20:35.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.025+0000 7efc91de9640 1 --2- 172.21.15.1:0/1306739575 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc8c06d060 0x7efc8c1558b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.025+0000 7efc91de9640 1 -- 172.21.15.1:0/1306739575 >> 172.21.15.1:0/1306739575 conn(0x7efc8c066d00 msgr2=0x7efc8c067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:35.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.025+0000 7efc91de9640 1 -- 172.21.15.1:0/1306739575 shutdown_connections 2024-03-06T07:20:35.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.025+0000 7efc91de9640 1 -- 172.21.15.1:0/1306739575 wait complete. 2024-03-06T07:20:35.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.026+0000 7efc91de9640 1 Processor -- start 2024-03-06T07:20:35.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.027+0000 7efc91de9640 1 -- start start 2024-03-06T07:20:35.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.027+0000 7efc91de9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc8c06d060 0x7efc8c1eef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.027+0000 7efc91de9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc8c0af640 0x7efc8c0cac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.027+0000 7efc91de9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc8c0c3eb0 0x7efc8c1eb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.027+0000 7efc91de9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c155420 con 0x7efc8c06d060 2024-03-06T07:20:35.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.027+0000 7efc91de9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c0c4c00 con 0x7efc8c0c3eb0 2024-03-06T07:20:35.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.027+0000 7efc91de9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c0c4290 con 0x7efc8c0af640 2024-03-06T07:20:35.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.027+0000 7efc8b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc8c06d060 0x7efc8c1eef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.027+0000 7efc8b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc8c06d060 0x7efc8c1eef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40214/0 (socket says 172.21.15.1:40214) 2024-03-06T07:20:35.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.027+0000 7efc8b7fe640 1 -- 172.21.15.1:0/175374563 learned_addr learned my addr 172.21.15.1:0/175374563 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:35.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.027+0000 7efc8affd640 1 --2- 172.21.15.1:0/175374563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc8c0af640 0x7efc8c0cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.028+0000 7efc8bfff640 1 --2- 172.21.15.1:0/175374563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc8c0c3eb0 0x7efc8c1eb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.028+0000 7efc8b7fe640 1 -- 172.21.15.1:0/175374563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc8c0af640 msgr2=0x7efc8c0cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.028+0000 7efc8b7fe640 1 --2- 172.21.15.1:0/175374563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc8c0af640 0x7efc8c0cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.028+0000 7efc8b7fe640 1 -- 172.21.15.1:0/175374563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc8c0c3eb0 msgr2=0x7efc8c1eb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.028+0000 7efc8b7fe640 1 --2- 172.21.15.1:0/175374563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc8c0c3eb0 0x7efc8c1eb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.028+0000 7efc8b7fe640 1 -- 172.21.15.1:0/175374563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c1494e0 con 0x7efc8c06d060 2024-03-06T07:20:35.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.028+0000 7efc8b7fe640 1 --2- 172.21.15.1:0/175374563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc8c06d060 0x7efc8c1eef90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7efc7c00c750 tx=0x7efc7c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:35.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.028+0000 7efc83fff640 1 -- 172.21.15.1:0/175374563 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7c005ad0 con 0x7efc8c06d060 2024-03-06T07:20:35.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.028+0000 7efc83fff640 1 -- 172.21.15.1:0/175374563 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc7c005c70 con 0x7efc8c06d060 2024-03-06T07:20:35.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.028+0000 7efc91de9640 1 -- 172.21.15.1:0/175374563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc8c148f20 con 0x7efc8c06d060 2024-03-06T07:20:35.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.028+0000 7efc83fff640 1 -- 172.21.15.1:0/175374563 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7c0072b0 con 0x7efc8c06d060 2024-03-06T07:20:35.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.028+0000 7efc91de9640 1 -- 172.21.15.1:0/175374563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc8c1ee790 con 0x7efc8c06d060 2024-03-06T07:20:35.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.029+0000 7efc91de9640 1 -- 172.21.15.1:0/175374563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc8c067fd0 con 0x7efc8c06d060 2024-03-06T07:20:35.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.029+0000 7efc83fff640 1 -- 172.21.15.1:0/175374563 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc7c007450 con 0x7efc8c06d060 2024-03-06T07:20:35.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.030+0000 7efc83fff640 1 --2- 172.21.15.1:0/175374563 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc6c06b500 0x7efc6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.030+0000 7efc83fff640 1 -- 172.21.15.1:0/175374563 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7efc7c08a130 con 0x7efc8c06d060 2024-03-06T07:20:35.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.030+0000 7efc8affd640 1 --2- 172.21.15.1:0/175374563 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc6c06b500 0x7efc6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.030+0000 7efc8affd640 1 --2- 172.21.15.1:0/175374563 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc6c06b500 0x7efc6c06d9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efc74034000 tx=0x7efc74008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:35.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.034+0000 7efc83fff640 1 -- 172.21.15.1:0/175374563 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc8c1494e0 con 0x7efc8c06d060 2024-03-06T07:20:35.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.228+0000 7efc91de9640 1 -- 172.21.15.1:0/175374563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7efc8c0c5b30 con 0x7efc8c06d060 2024-03-06T07:20:35.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.228+0000 7efc83fff640 1 -- 172.21.15.1:0/175374563 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7efc7c005e20 con 0x7efc8c06d060 2024-03-06T07:20:35.230 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-06T07:20:35.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.230+0000 7efc91de9640 1 -- 172.21.15.1:0/175374563 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc6c06b500 msgr2=0x7efc6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.230+0000 7efc91de9640 1 --2- 172.21.15.1:0/175374563 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc6c06b500 0x7efc6c06d9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efc74034000 tx=0x7efc74008030 comp rx=0 tx=0).stop 2024-03-06T07:20:35.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.230+0000 7efc91de9640 1 -- 172.21.15.1:0/175374563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc8c06d060 msgr2=0x7efc8c1eef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.230+0000 7efc91de9640 1 --2- 172.21.15.1:0/175374563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc8c06d060 0x7efc8c1eef90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7efc7c00c750 tx=0x7efc7c00cc10 comp rx=0 tx=0).stop 2024-03-06T07:20:35.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.230+0000 7efc91de9640 1 -- 172.21.15.1:0/175374563 shutdown_connections 2024-03-06T07:20:35.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.230+0000 7efc91de9640 1 --2- 172.21.15.1:0/175374563 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc6c06b500 0x7efc6c06d9c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.230+0000 7efc91de9640 1 --2- 172.21.15.1:0/175374563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc8c0c3eb0 0x7efc8c1eb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.230+0000 7efc91de9640 1 --2- 172.21.15.1:0/175374563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc8c0af640 0x7efc8c0cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.230+0000 7efc91de9640 1 --2- 172.21.15.1:0/175374563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc8c06d060 0x7efc8c1eef90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.230+0000 7efc91de9640 1 -- 172.21.15.1:0/175374563 >> 172.21.15.1:0/175374563 conn(0x7efc8c066d00 msgr2=0x7efc8c0bf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:35.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.230+0000 7efc91de9640 1 -- 172.21.15.1:0/175374563 shutdown_connections 2024-03-06T07:20:35.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.230+0000 7efc91de9640 1 -- 172.21.15.1:0/175374563 wait complete. 2024-03-06T07:20:35.243 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-06T07:20:35.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:35.243 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-06T07:20:35.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi001.stderr:ceph-fuse[43211]: fuse finished with error 0 and tester_r 0 2024-03-06T07:20:35.270 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:20:35.309 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-06T07:20:35.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:35.310 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-06T07:20:35.374 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:35.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.512+0000 7f5cc249d640 1 Processor -- start 2024-03-06T07:20:35.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.512+0000 7f5cc249d640 1 -- start start 2024-03-06T07:20:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.513+0000 7f5cc249d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cbc152840 0x7f5cbc0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.513+0000 7f5cc249d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5cbc0c60b0 0x7f5cbc1553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.513+0000 7f5cc249d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cbc0c5e40 con 0x7f5cbc0c3970 2024-03-06T07:20:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.513+0000 7f5cc249d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cbc0c67f0 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.513+0000 7f5cc249d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cbc0c6a40 con 0x7f5cbc152840 2024-03-06T07:20:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.513+0000 7f5cc0a3a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5cbc0c60b0 0x7f5cbc1553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.513+0000 7f5cc0a3a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5cbc0c60b0 0x7f5cbc1553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40216/0 (socket says 172.21.15.1:40216) 2024-03-06T07:20:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.513+0000 7f5cc0a3a640 1 -- 172.21.15.1:0/4187211031 learned_addr learned my addr 172.21.15.1:0/4187211031 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.513+0000 7f5cbb7fe640 1 --2- 172.21.15.1:0/4187211031 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cbc152840 0x7f5cbc0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.514+0000 7f5cc0a3a640 1 -- 172.21.15.1:0/4187211031 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cbc152840 msgr2=0x7f5cbc0c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.514+0000 7f5cc0a3a640 1 --2- 172.21.15.1:0/4187211031 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cbc152840 0x7f5cbc0c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.514+0000 7f5cc0a3a640 1 -- 172.21.15.1:0/4187211031 >> v1:172.21.15.98:6789/0 conn(0x7f5cbc0c3970 legacy=0x7f5cbc152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.514+0000 7f5cc0a3a640 1 -- 172.21.15.1:0/4187211031 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cbc0c6d20 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.514+0000 7f5cc0a3a640 1 --2- 172.21.15.1:0/4187211031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cbc0c60b0 0x7f5cbc1553d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009e50 tx=0x7f5cb402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc38387dad66563 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.514+0000 7f5cbaffd640 1 -- 172.21.15.1:0/4187211031 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb402ca70 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.515+0000 7f5cbaffd640 1 -- 172.21.15.1:0/4187211031 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb402cc10 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.515+0000 7f5cbaffd640 1 -- 172.21.15.1:0/4187211031 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb40343f0 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.515+0000 7f5cc249d640 1 -- 172.21.15.1:0/4187211031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cbc0c60b0 msgr2=0x7f5cbc1553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.515+0000 7f5cc249d640 1 --2- 172.21.15.1:0/4187211031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cbc0c60b0 0x7f5cbc1553d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009e50 tx=0x7f5cb402bed0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.515+0000 7f5cc249d640 1 -- 172.21.15.1:0/4187211031 shutdown_connections 2024-03-06T07:20:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.515+0000 7f5cc249d640 1 --2- 172.21.15.1:0/4187211031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cbc0c60b0 0x7f5cbc1553d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.515+0000 7f5cc249d640 1 --2- 172.21.15.1:0/4187211031 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cbc152840 0x7f5cbc0c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.515+0000 7f5cc249d640 1 -- 172.21.15.1:0/4187211031 >> 172.21.15.1:0/4187211031 conn(0x7f5cbc0c88f0 msgr2=0x7f5cbc0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.515+0000 7f5cc249d640 1 -- 172.21.15.1:0/4187211031 shutdown_connections 2024-03-06T07:20:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.515+0000 7f5cc249d640 1 -- 172.21.15.1:0/4187211031 wait complete. 2024-03-06T07:20:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.516+0000 7f5cc249d640 1 Processor -- start 2024-03-06T07:20:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.516+0000 7f5cc249d640 1 -- start start 2024-03-06T07:20:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.517+0000 7f5cc249d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cbc0c3970 0x7f5cbc06fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.517+0000 7f5cc249d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cbc0c60b0 0x7f5cbc1ee1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.517+0000 7f5cc249d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cbc152840 0x7f5cbc156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.517+0000 7f5cc249d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cbc0c75c0 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.517+0000 7f5cc249d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cbc0c72e0 con 0x7f5cbc0c3970 2024-03-06T07:20:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.517+0000 7f5cc249d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cbc0c3d50 con 0x7f5cbc152840 2024-03-06T07:20:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.517+0000 7f5cbb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cbc0c60b0 0x7f5cbc1ee1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.517+0000 7f5cbb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cbc0c60b0 0x7f5cbc1ee1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40218/0 (socket says 172.21.15.1:40218) 2024-03-06T07:20:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.517+0000 7f5cbb7fe640 1 -- 172.21.15.1:0/1810913755 learned_addr learned my addr 172.21.15.1:0/1810913755 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.517+0000 7f5cc0a3a640 1 --2- 172.21.15.1:0/1810913755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cbc152840 0x7f5cbc156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.517+0000 7f5cbbfff640 1 --2- 172.21.15.1:0/1810913755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cbc0c3970 0x7f5cbc06fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.517+0000 7f5cbb7fe640 1 -- 172.21.15.1:0/1810913755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cbc152840 msgr2=0x7f5cbc156f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.517+0000 7f5cbb7fe640 1 --2- 172.21.15.1:0/1810913755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cbc152840 0x7f5cbc156f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.517+0000 7f5cbb7fe640 1 -- 172.21.15.1:0/1810913755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cbc0c3970 msgr2=0x7f5cbc06fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.517+0000 7f5cbb7fe640 1 --2- 172.21.15.1:0/1810913755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cbc0c3970 0x7f5cbc06fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.517+0000 7f5cbb7fe640 1 -- 172.21.15.1:0/1810913755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cbc1eebe0 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.518+0000 7f5cbb7fe640 1 --2- 172.21.15.1:0/1810913755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cbc0c60b0 0x7f5cbc1ee1f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5cb000d5c0 tx=0x7f5cb000da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.518+0000 7f5cb8ff9640 1 -- 172.21.15.1:0/1810913755 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb00125f0 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.518+0000 7f5cb8ff9640 1 -- 172.21.15.1:0/1810913755 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb0019070 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.518+0000 7f5cb8ff9640 1 -- 172.21.15.1:0/1810913755 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb00129d0 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.518+0000 7f5cc249d640 1 -- 172.21.15.1:0/1810913755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cbc157dc0 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.518+0000 7f5cc249d640 1 -- 172.21.15.1:0/1810913755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cbc0aa300 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.519+0000 7f5cc249d640 1 -- 172.21.15.1:0/1810913755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cbc0686e0 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.519+0000 7f5cb8ff9640 1 -- 172.21.15.1:0/1810913755 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5cb0012b70 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.520+0000 7f5cb8ff9640 1 --2- 172.21.15.1:0/1810913755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ca806b4d0 0x7f5ca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.520+0000 7f5cb8ff9640 1 -- 172.21.15.1:0/1810913755 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5cb008a690 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.520+0000 7f5cbbfff640 1 --2- 172.21.15.1:0/1810913755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ca806b4d0 0x7f5ca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.521+0000 7f5cbbfff640 1 --2- 172.21.15.1:0/1810913755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ca806b4d0 0x7f5ca806d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc1e15f0 tx=0x7f5cac007e30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:35.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.524+0000 7f5cb8ff9640 1 -- 172.21.15.1:0/1810913755 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5cb0058c10 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.706+0000 7f5cc249d640 1 -- 172.21.15.1:0/1810913755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5cbc0748b0 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.707+0000 7f5cb8ff9640 1 -- 172.21.15.1:0/1810913755 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7f5cb005ccb0 con 0x7f5cbc0c60b0 2024-03-06T07:20:35.709 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:35.710 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.709+0000 7f5cc249d640 1 -- 172.21.15.1:0/1810913755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ca806b4d0 msgr2=0x7f5ca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.709+0000 7f5cc249d640 1 --2- 172.21.15.1:0/1810913755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ca806b4d0 0x7f5ca806d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc1e15f0 tx=0x7f5cac007e30 comp rx=0 tx=0).stop 2024-03-06T07:20:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.709+0000 7f5cc249d640 1 -- 172.21.15.1:0/1810913755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cbc0c60b0 msgr2=0x7f5cbc1ee1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.709+0000 7f5cc249d640 1 --2- 172.21.15.1:0/1810913755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cbc0c60b0 0x7f5cbc1ee1f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5cb000d5c0 tx=0x7f5cb000da80 comp rx=0 tx=0).stop 2024-03-06T07:20:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.709+0000 7f5cc249d640 1 -- 172.21.15.1:0/1810913755 shutdown_connections 2024-03-06T07:20:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.709+0000 7f5cc249d640 1 --2- 172.21.15.1:0/1810913755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ca806b4d0 0x7f5ca806d990 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.709+0000 7f5cc249d640 1 --2- 172.21.15.1:0/1810913755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5cbc152840 0x7f5cbc156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.709+0000 7f5cc249d640 1 --2- 172.21.15.1:0/1810913755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cbc0c60b0 0x7f5cbc1ee1f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.709+0000 7f5cc249d640 1 --2- 172.21.15.1:0/1810913755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cbc0c3970 0x7f5cbc06fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.709+0000 7f5cc249d640 1 -- 172.21.15.1:0/1810913755 >> 172.21.15.1:0/1810913755 conn(0x7f5cbc0c88f0 msgr2=0x7f5cbc06c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.709+0000 7f5cc249d640 1 -- 172.21.15.1:0/1810913755 shutdown_connections 2024-03-06T07:20:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.709+0000 7f5cc249d640 1 -- 172.21.15.1:0/1810913755 wait complete. 2024-03-06T07:20:35.725 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:35.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.825+0000 7f57bb69a640 1 Processor -- start 2024-03-06T07:20:35.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.825+0000 7f57bb69a640 1 -- start start 2024-03-06T07:20:35.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.826+0000 7f57bb69a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f57b40c3970 0x7f57b40c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.826+0000 7f57bb69a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b40c60b0 0x7f57b4152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.826+0000 7f57bb69a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f57b4152670 0x7f57b41553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.826+0000 7f57bb69a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57b40c67f0 con 0x7f57b4152670 2024-03-06T07:20:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.826+0000 7f57bb69a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57b40c5e40 con 0x7f57b40c3970 2024-03-06T07:20:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.826+0000 7f57bb69a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b40c6a40 con 0x7f57b40c60b0 2024-03-06T07:20:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.826+0000 7f57b9c37640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f57b4152670 0x7f57b41553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.826+0000 7f57b9c37640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f57b4152670 0x7f57b41553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40222/0 (socket says 172.21.15.1:40222) 2024-03-06T07:20:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.826+0000 7f57b9c37640 1 -- 172.21.15.1:0/3784620293 learned_addr learned my addr 172.21.15.1:0/3784620293 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.826+0000 7f57b9436640 1 --2- 172.21.15.1:0/3784620293 >> v2:172.21.15.98:3300/0 conn(0x7f57b40c3970 0x7f57b40c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.826+0000 7f57b8c35640 1 --2- 172.21.15.1:0/3784620293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b40c60b0 0x7f57b4152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.826+0000 7f57b9c37640 1 -- 172.21.15.1:0/3784620293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b40c60b0 msgr2=0x7f57b4152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.826+0000 7f57b9c37640 1 --2- 172.21.15.1:0/3784620293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b40c60b0 0x7f57b4152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.826+0000 7f57b9c37640 1 -- 172.21.15.1:0/3784620293 >> v2:172.21.15.98:3300/0 conn(0x7f57b40c3970 msgr2=0x7f57b40c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.826+0000 7f57b9c37640 1 --2- 172.21.15.1:0/3784620293 >> v2:172.21.15.98:3300/0 conn(0x7f57b40c3970 0x7f57b40c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.826+0000 7f57b9c37640 1 -- 172.21.15.1:0/3784620293 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b40c6d20 con 0x7f57b4152670 2024-03-06T07:20:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.826+0000 7f57b9436640 1 --2- 172.21.15.1:0/3784620293 >> v2:172.21.15.98:3300/0 conn(0x7f57b40c3970 0x7f57b40c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.827+0000 7f57b9c37640 1 --2- 172.21.15.1:0/3784620293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b4152670 0x7f57b41553d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f57b40c34c0 tx=0x7f57ac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97d21aa0668742f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.828+0000 7f57abfff640 1 -- 172.21.15.1:0/3784620293 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac02ca70 con 0x7f57b4152670 2024-03-06T07:20:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.828+0000 7f57abfff640 1 -- 172.21.15.1:0/3784620293 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ac02cc10 con 0x7f57b4152670 2024-03-06T07:20:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.828+0000 7f57abfff640 1 -- 172.21.15.1:0/3784620293 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac0343f0 con 0x7f57b4152670 2024-03-06T07:20:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.828+0000 7f57bb69a640 1 -- 172.21.15.1:0/3784620293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b4152670 msgr2=0x7f57b41553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.828+0000 7f57bb69a640 1 --2- 172.21.15.1:0/3784620293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b4152670 0x7f57b41553d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f57b40c34c0 tx=0x7f57ac02bed0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.828+0000 7f57bb69a640 1 -- 172.21.15.1:0/3784620293 shutdown_connections 2024-03-06T07:20:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.828+0000 7f57bb69a640 1 --2- 172.21.15.1:0/3784620293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b4152670 0x7f57b41553d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.828+0000 7f57bb69a640 1 --2- 172.21.15.1:0/3784620293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b40c60b0 0x7f57b4152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.828+0000 7f57bb69a640 1 --2- 172.21.15.1:0/3784620293 >> v2:172.21.15.98:3300/0 conn(0x7f57b40c3970 0x7f57b40c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.828+0000 7f57bb69a640 1 -- 172.21.15.1:0/3784620293 >> 172.21.15.1:0/3784620293 conn(0x7f57b40c88f0 msgr2=0x7f57b40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.828+0000 7f57bb69a640 1 -- 172.21.15.1:0/3784620293 shutdown_connections 2024-03-06T07:20:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.828+0000 7f57bb69a640 1 -- 172.21.15.1:0/3784620293 wait complete. 2024-03-06T07:20:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.829+0000 7f57bb69a640 1 Processor -- start 2024-03-06T07:20:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.829+0000 7f57bb69a640 1 -- start start 2024-03-06T07:20:35.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57bb69a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b40c3970 0x7f57b4153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57bb69a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b40c60b0 0x7f57b414c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57bb69a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b4152670 0x7f57b414ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57bb69a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b40c75c0 con 0x7f57b40c60b0 2024-03-06T07:20:35.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57bb69a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b40c72e0 con 0x7f57b40c3970 2024-03-06T07:20:35.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57bb69a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b40c3d50 con 0x7f57b4152670 2024-03-06T07:20:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57b8c35640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b40c60b0 0x7f57b414c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57b8c35640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b40c60b0 0x7f57b414c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40224/0 (socket says 172.21.15.1:40224) 2024-03-06T07:20:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57b8c35640 1 -- 172.21.15.1:0/352479123 learned_addr learned my addr 172.21.15.1:0/352479123 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57b9c37640 1 --2- 172.21.15.1:0/352479123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b4152670 0x7f57b414ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57b9436640 1 --2- 172.21.15.1:0/352479123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b40c3970 0x7f57b4153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57b8c35640 1 -- 172.21.15.1:0/352479123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b4152670 msgr2=0x7f57b414ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57b8c35640 1 --2- 172.21.15.1:0/352479123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b4152670 0x7f57b414ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57b8c35640 1 -- 172.21.15.1:0/352479123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b40c3970 msgr2=0x7f57b4153830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57b8c35640 1 --2- 172.21.15.1:0/352479123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b40c3970 0x7f57b4153830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57b8c35640 1 -- 172.21.15.1:0/352479123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b40a9b70 con 0x7f57b40c60b0 2024-03-06T07:20:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57b9c37640 1 --2- 172.21.15.1:0/352479123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b4152670 0x7f57b414ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.830+0000 7f57b9436640 1 --2- 172.21.15.1:0/352479123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b40c3970 0x7f57b4153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.831+0000 7f57b8c35640 1 --2- 172.21.15.1:0/352479123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b40c60b0 0x7f57b414c520 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f57b000d830 tx=0x7f57b000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.831+0000 7f57a9ffb640 1 -- 172.21.15.1:0/352479123 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b0004030 con 0x7f57b40c60b0 2024-03-06T07:20:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.831+0000 7f57a9ffb640 1 -- 172.21.15.1:0/352479123 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b00041d0 con 0x7f57b40c60b0 2024-03-06T07:20:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.831+0000 7f57bb69a640 1 -- 172.21.15.1:0/352479123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b40a9e50 con 0x7f57b40c60b0 2024-03-06T07:20:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.831+0000 7f57a9ffb640 1 -- 172.21.15.1:0/352479123 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b000b6c0 con 0x7f57b40c60b0 2024-03-06T07:20:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.831+0000 7f57bb69a640 1 -- 172.21.15.1:0/352479123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b4156fd0 con 0x7f57b40c60b0 2024-03-06T07:20:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.832+0000 7f57bb69a640 1 -- 172.21.15.1:0/352479123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b40686e0 con 0x7f57b40c60b0 2024-03-06T07:20:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.832+0000 7f57a9ffb640 1 -- 172.21.15.1:0/352479123 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f57b000ba70 con 0x7f57b40c60b0 2024-03-06T07:20:35.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.833+0000 7f57a9ffb640 1 --2- 172.21.15.1:0/352479123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f579806b500 0x7f579806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.833+0000 7f57a9ffb640 1 -- 172.21.15.1:0/352479123 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f57b008a9d0 con 0x7f57b40c60b0 2024-03-06T07:20:35.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.833+0000 7f57b9436640 1 --2- 172.21.15.1:0/352479123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f579806b500 0x7f579806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.834+0000 7f57b9436640 1 --2- 172.21.15.1:0/352479123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f579806b500 0x7f579806d9c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f57a4006fd0 tx=0x7f57a4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:35.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:35.841+0000 7f57a9ffb640 1 -- 172.21.15.1:0/352479123 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f57b00595a0 con 0x7f57b40c60b0 2024-03-06T07:20:36.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.027+0000 7f57bb69a640 1 -- 172.21.15.1:0/352479123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57b4153d70 con 0x7f57b40c60b0 2024-03-06T07:20:36.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.028+0000 7f57a9ffb640 1 -- 172.21.15.1:0/352479123 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7f57b00045c0 con 0x7f57b40c60b0 2024-03-06T07:20:36.030 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:36.030 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:36.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.030+0000 7f57bb69a640 1 -- 172.21.15.1:0/352479123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f579806b500 msgr2=0x7f579806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:36.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.030+0000 7f57bb69a640 1 --2- 172.21.15.1:0/352479123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f579806b500 0x7f579806d9c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f57a4006fd0 tx=0x7f57a4008040 comp rx=0 tx=0).stop 2024-03-06T07:20:36.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.030+0000 7f57bb69a640 1 -- 172.21.15.1:0/352479123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b40c60b0 msgr2=0x7f57b414c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:36.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.030+0000 7f57bb69a640 1 --2- 172.21.15.1:0/352479123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b40c60b0 0x7f57b414c520 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f57b000d830 tx=0x7f57b000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.030+0000 7f57bb69a640 1 -- 172.21.15.1:0/352479123 shutdown_connections 2024-03-06T07:20:36.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.030+0000 7f57bb69a640 1 --2- 172.21.15.1:0/352479123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f579806b500 0x7f579806d9c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.030+0000 7f57bb69a640 1 --2- 172.21.15.1:0/352479123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b4152670 0x7f57b414ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.030+0000 7f57bb69a640 1 --2- 172.21.15.1:0/352479123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b40c60b0 0x7f57b414c520 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.030+0000 7f57bb69a640 1 --2- 172.21.15.1:0/352479123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b40c3970 0x7f57b4153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.030+0000 7f57bb69a640 1 -- 172.21.15.1:0/352479123 >> 172.21.15.1:0/352479123 conn(0x7f57b40c88f0 msgr2=0x7f57b40aa880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:36.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.030+0000 7f57bb69a640 1 -- 172.21.15.1:0/352479123 shutdown_connections 2024-03-06T07:20:36.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.030+0000 7f57bb69a640 1 -- 172.21.15.1:0/352479123 wait complete. 2024-03-06T07:20:36.043 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:20:36.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.147+0000 7f191a5b6640 1 Processor -- start 2024-03-06T07:20:36.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.147+0000 7f191a5b6640 1 -- start start 2024-03-06T07:20:36.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.147+0000 7f191a5b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19140c69b0 0x7f19140c2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:36.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.147+0000 7f191a5b6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f19140c7510 0x7f19140c3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:36.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.147+0000 7f191a5b6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1914072d00 con 0x7f1914159140 2024-03-06T07:20:36.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.147+0000 7f191a5b6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19140c0e80 con 0x7f19140c7510 2024-03-06T07:20:36.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.147+0000 7f191a5b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19140c13b0 con 0x7f19140c69b0 2024-03-06T07:20:36.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.147+0000 7f1918b53640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f19140c7510 0x7f19140c3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:36.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.147+0000 7f1913fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1914159140 0x7f19140c4490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57362/0 (socket says 172.21.15.1:57362) 2024-03-06T07:20:36.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.147+0000 7f1913fff640 1 -- 172.21.15.1:0/1470598416 learned_addr learned my addr 172.21.15.1:0/1470598416 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:36.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.148+0000 7f19137fe640 1 --2- 172.21.15.1:0/1470598416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19140c69b0 0x7f19140c2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:36.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.148+0000 7f1918b53640 1 -- 172.21.15.1:0/1470598416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19140c69b0 msgr2=0x7f19140c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:36.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.148+0000 7f1918b53640 1 --2- 172.21.15.1:0/1470598416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19140c69b0 0x7f19140c2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.148+0000 7f1918b53640 1 -- 172.21.15.1:0/1470598416 >> v1:172.21.15.1:6789/0 conn(0x7f1914159140 legacy=0x7f19140c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:36.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.148+0000 7f1912ffd640 1 -- 172.21.15.1:0/1470598416 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498963454 0 0) 0x7f1914072d00 con 0x7f1914159140 2024-03-06T07:20:36.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.148+0000 7f1918b53640 1 -- 172.21.15.1:0/1470598416 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1914073180 con 0x7f19140c7510 2024-03-06T07:20:36.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.149+0000 7f1918b53640 1 --2- 172.21.15.1:0/1470598416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19140c7510 0x7f19140c3070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f190c009d90 tx=0x7f190c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc37bf558f2e1cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:36.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.149+0000 7f1912ffd640 1 -- 172.21.15.1:0/1470598416 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c039070 con 0x7f19140c7510 2024-03-06T07:20:36.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.149+0000 7f1912ffd640 1 -- 172.21.15.1:0/1470598416 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190c02cbc0 con 0x7f19140c7510 2024-03-06T07:20:36.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.149+0000 7f1912ffd640 1 -- 172.21.15.1:0/1470598416 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c034440 con 0x7f19140c7510 2024-03-06T07:20:36.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.149+0000 7f191a5b6640 1 -- 172.21.15.1:0/1470598416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19140c7510 msgr2=0x7f19140c3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:36.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.149+0000 7f191a5b6640 1 --2- 172.21.15.1:0/1470598416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19140c7510 0x7f19140c3070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f190c009d90 tx=0x7f190c02be10 comp rx=0 tx=0).stop 2024-03-06T07:20:36.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.149+0000 7f191a5b6640 1 -- 172.21.15.1:0/1470598416 shutdown_connections 2024-03-06T07:20:36.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.149+0000 7f191a5b6640 1 --2- 172.21.15.1:0/1470598416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19140c69b0 0x7f19140c2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.149+0000 7f191a5b6640 1 --2- 172.21.15.1:0/1470598416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19140c7510 0x7f19140c3070 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.149+0000 7f191a5b6640 1 -- 172.21.15.1:0/1470598416 >> 172.21.15.1:0/1470598416 conn(0x7f19140c7a50 msgr2=0x7f19140c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:36.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.149+0000 7f191a5b6640 1 -- 172.21.15.1:0/1470598416 shutdown_connections 2024-03-06T07:20:36.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.149+0000 7f191a5b6640 1 -- 172.21.15.1:0/1470598416 wait complete. 2024-03-06T07:20:36.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.151+0000 7f191a5b6640 1 Processor -- start 2024-03-06T07:20:36.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.151+0000 7f191a5b6640 1 -- start start 2024-03-06T07:20:36.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.152+0000 7f191a5b6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19140c69b0 0x7f19141edef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:36.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.152+0000 7f191a5b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19140c7510 0x7f19140a9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:36.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.152+0000 7f191a5b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1914159140 0x7f19140aa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:36.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.152+0000 7f191a5b6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19140c35b0 con 0x7f19140c69b0 2024-03-06T07:20:36.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.152+0000 7f191a5b6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19140c6520 con 0x7f19140c7510 2024-03-06T07:20:36.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.152+0000 7f191a5b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19140c10d0 con 0x7f1914159140 2024-03-06T07:20:36.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.152+0000 7f1913fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19140c69b0 0x7f19141edef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:36.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.152+0000 7f1913fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19140c69b0 0x7f19141edef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40242/0 (socket says 172.21.15.1:40242) 2024-03-06T07:20:36.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.152+0000 7f1913fff640 1 -- 172.21.15.1:0/3451233166 learned_addr learned my addr 172.21.15.1:0/3451233166 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:36.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.153+0000 7f1918b53640 1 --2- 172.21.15.1:0/3451233166 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1914159140 0x7f19140aa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:36.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.153+0000 7f19137fe640 1 --2- 172.21.15.1:0/3451233166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19140c7510 0x7f19140a9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:36.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.153+0000 7f1913fff640 1 -- 172.21.15.1:0/3451233166 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1914159140 msgr2=0x7f19140aa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:36.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.153+0000 7f1913fff640 1 --2- 172.21.15.1:0/3451233166 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1914159140 0x7f19140aa2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.153+0000 7f1913fff640 1 -- 172.21.15.1:0/3451233166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19140c7510 msgr2=0x7f19140a9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:36.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.153+0000 7f1913fff640 1 --2- 172.21.15.1:0/3451233166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19140c7510 0x7f19140a9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.153+0000 7f1913fff640 1 -- 172.21.15.1:0/3451233166 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19141eebc0 con 0x7f19140c69b0 2024-03-06T07:20:36.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.153+0000 7f1918b53640 1 --2- 172.21.15.1:0/3451233166 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1914159140 0x7f19140aa2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:36.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.153+0000 7f1913fff640 1 --2- 172.21.15.1:0/3451233166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19140c69b0 0x7f19141edef0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f190400c400 tx=0x7f190400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:36.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.153+0000 7f1910ff9640 1 -- 172.21.15.1:0/3451233166 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1904017070 con 0x7f19140c69b0 2024-03-06T07:20:36.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.153+0000 7f1910ff9640 1 -- 172.21.15.1:0/3451233166 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1904005ce0 con 0x7f19140c69b0 2024-03-06T07:20:36.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.153+0000 7f191a5b6640 1 -- 172.21.15.1:0/3451233166 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19141ee8e0 con 0x7f19140c69b0 2024-03-06T07:20:36.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.153+0000 7f1910ff9640 1 -- 172.21.15.1:0/3451233166 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1904007310 con 0x7f19140c69b0 2024-03-06T07:20:36.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.153+0000 7f191a5b6640 1 -- 172.21.15.1:0/3451233166 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19140a7fe0 con 0x7f19140c69b0 2024-03-06T07:20:36.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.154+0000 7f191a5b6640 1 -- 172.21.15.1:0/3451233166 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19140addf0 con 0x7f19140c69b0 2024-03-06T07:20:36.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.155+0000 7f1910ff9640 1 -- 172.21.15.1:0/3451233166 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f190401a070 con 0x7f19140c69b0 2024-03-06T07:20:36.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.155+0000 7f1910ff9640 1 --2- 172.21.15.1:0/3451233166 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f190006b4d0 0x7f190006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:36.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.155+0000 7f1910ff9640 1 -- 172.21.15.1:0/3451233166 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f19040898f0 con 0x7f19140c69b0 2024-03-06T07:20:36.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.155+0000 7f19137fe640 1 --2- 172.21.15.1:0/3451233166 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f190006b4d0 0x7f190006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:36.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.156+0000 7f19137fe640 1 --2- 172.21.15.1:0/3451233166 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f190006b4d0 0x7f190006d990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f19141547c0 tx=0x7f1908008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:36.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.159+0000 7f1910ff9640 1 -- 172.21.15.1:0/3451233166 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19140addf0 con 0x7f19140c69b0 2024-03-06T07:20:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.378+0000 7f191a5b6640 1 -- 172.21.15.1:0/3451233166 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f19140686e0 con 0x7f19140c69b0 2024-03-06T07:20:36.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.904+0000 7f1910ff9640 1 -- 172.21.15.1:0/3451233166 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7f19040584c0 con 0x7f19140c69b0 2024-03-06T07:20:36.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.906+0000 7f191a5b6640 1 -- 172.21.15.1:0/3451233166 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f190006b4d0 msgr2=0x7f190006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:36.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.906+0000 7f191a5b6640 1 --2- 172.21.15.1:0/3451233166 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f190006b4d0 0x7f190006d990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f19141547c0 tx=0x7f1908008040 comp rx=0 tx=0).stop 2024-03-06T07:20:36.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.906+0000 7f191a5b6640 1 -- 172.21.15.1:0/3451233166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19140c69b0 msgr2=0x7f19141edef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:36.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.906+0000 7f191a5b6640 1 --2- 172.21.15.1:0/3451233166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19140c69b0 0x7f19141edef0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f190400c400 tx=0x7f190400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.906+0000 7f191a5b6640 1 -- 172.21.15.1:0/3451233166 shutdown_connections 2024-03-06T07:20:36.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.906+0000 7f191a5b6640 1 --2- 172.21.15.1:0/3451233166 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f190006b4d0 0x7f190006d990 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.906+0000 7f191a5b6640 1 --2- 172.21.15.1:0/3451233166 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1914159140 0x7f19140aa2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.906+0000 7f191a5b6640 1 --2- 172.21.15.1:0/3451233166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19140c7510 0x7f19140a9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.906+0000 7f191a5b6640 1 --2- 172.21.15.1:0/3451233166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19140c69b0 0x7f19141edef0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.906+0000 7f191a5b6640 1 -- 172.21.15.1:0/3451233166 >> 172.21.15.1:0/3451233166 conn(0x7f19140c7a50 msgr2=0x7f19140c7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:36.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.907+0000 7f191a5b6640 1 -- 172.21.15.1:0/3451233166 shutdown_connections 2024-03-06T07:20:36.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:36.907+0000 7f191a5b6640 1 -- 172.21.15.1:0/3451233166 wait complete. 2024-03-06T07:20:36.919 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-03-06T07:20:36.920 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:20:37.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.026+0000 7f461bec9640 1 Processor -- start 2024-03-06T07:20:37.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.026+0000 7f461bec9640 1 -- start start 2024-03-06T07:20:37.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.027+0000 7f461bec9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f46140c2e40 0x7f461406c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.027+0000 7f461bec9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f46140a6e30 0x7f4614155790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.027+0000 7f461bec9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f461406eb10 con 0x7f46140c3220 2024-03-06T07:20:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.027+0000 7f461bec9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461406f400 con 0x7f46140a6e30 2024-03-06T07:20:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.027+0000 7f461bec9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461406edb0 con 0x7f46140c2e40 2024-03-06T07:20:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.028+0000 7f461a466640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f46140a6e30 0x7f4614155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.028+0000 7f4619464640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f46140c3220 0x7f46140c5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57376/0 (socket says 172.21.15.1:57376) 2024-03-06T07:20:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.028+0000 7f4619464640 1 -- 172.21.15.1:0/1979427133 learned_addr learned my addr 172.21.15.1:0/1979427133 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.028+0000 7f4619c65640 1 --2- 172.21.15.1:0/1979427133 >> v2:172.21.15.98:3300/0 conn(0x7f46140c2e40 0x7f461406c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.028+0000 7f461a466640 1 -- 172.21.15.1:0/1979427133 >> v2:172.21.15.98:3300/0 conn(0x7f46140c2e40 msgr2=0x7f461406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.028+0000 7f461a466640 1 --2- 172.21.15.1:0/1979427133 >> v2:172.21.15.98:3300/0 conn(0x7f46140c2e40 0x7f461406c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.028+0000 7f461a466640 1 -- 172.21.15.1:0/1979427133 >> v1:172.21.15.1:6789/0 conn(0x7f46140c3220 legacy=0x7f46140c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.028+0000 7f4618c63640 1 -- 172.21.15.1:0/1979427133 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776259164 0 0) 0x7f461406eb10 con 0x7f46140c3220 2024-03-06T07:20:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.028+0000 7f461a466640 1 -- 172.21.15.1:0/1979427133 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461406f090 con 0x7f46140a6e30 2024-03-06T07:20:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.028+0000 7f4619c65640 1 --2- 172.21.15.1:0/1979427133 >> v2:172.21.15.98:3300/0 conn(0x7f46140c2e40 0x7f461406c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.029+0000 7f461a466640 1 --2- 172.21.15.1:0/1979427133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46140a6e30 0x7f4614155790 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4614067d00 tx=0x7f460c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca90250650da1a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.029+0000 7f4618c63640 1 -- 172.21.15.1:0/1979427133 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c039070 con 0x7f46140a6e30 2024-03-06T07:20:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.029+0000 7f4618c63640 1 -- 172.21.15.1:0/1979427133 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460c02cbf0 con 0x7f46140a6e30 2024-03-06T07:20:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.029+0000 7f4618c63640 1 -- 172.21.15.1:0/1979427133 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c034440 con 0x7f46140a6e30 2024-03-06T07:20:37.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.029+0000 7f461bec9640 1 -- 172.21.15.1:0/1979427133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46140a6e30 msgr2=0x7f4614155790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.029+0000 7f461bec9640 1 --2- 172.21.15.1:0/1979427133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46140a6e30 0x7f4614155790 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4614067d00 tx=0x7f460c02be40 comp rx=0 tx=0).stop 2024-03-06T07:20:37.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.029+0000 7f461bec9640 1 -- 172.21.15.1:0/1979427133 shutdown_connections 2024-03-06T07:20:37.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.029+0000 7f461bec9640 1 --2- 172.21.15.1:0/1979427133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46140a6e30 0x7f4614155790 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.029+0000 7f461bec9640 1 --2- 172.21.15.1:0/1979427133 >> v2:172.21.15.98:3300/0 conn(0x7f46140c2e40 0x7f461406c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.029+0000 7f461bec9640 1 -- 172.21.15.1:0/1979427133 >> 172.21.15.1:0/1979427133 conn(0x7f46140a9700 msgr2=0x7f46140a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:37.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.029+0000 7f461bec9640 1 -- 172.21.15.1:0/1979427133 shutdown_connections 2024-03-06T07:20:37.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.030+0000 7f461bec9640 1 -- 172.21.15.1:0/1979427133 wait complete. 2024-03-06T07:20:37.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.031+0000 7f461bec9640 1 Processor -- start 2024-03-06T07:20:37.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.031+0000 7f461bec9640 1 -- start start 2024-03-06T07:20:37.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.031+0000 7f461bec9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46140a6e30 0x7f46140c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.031+0000 7f461bec9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46140c2e40 0x7f46141ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.031+0000 7f461bec9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46140c3220 0x7f46141ce920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.031+0000 7f461bec9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461406c640 con 0x7f46140c3220 2024-03-06T07:20:37.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.031+0000 7f461bec9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461406ca00 con 0x7f46140a6e30 2024-03-06T07:20:37.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.031+0000 7f461bec9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461406c7c0 con 0x7f46140c2e40 2024-03-06T07:20:37.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.031+0000 7f461a466640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46140c3220 0x7f46141ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.031+0000 7f461a466640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46140c3220 0x7f46141ce920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40258/0 (socket says 172.21.15.1:40258) 2024-03-06T07:20:37.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.031+0000 7f461a466640 1 -- 172.21.15.1:0/650298067 learned_addr learned my addr 172.21.15.1:0/650298067 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:37.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.032+0000 7f4619c65640 1 --2- 172.21.15.1:0/650298067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46140a6e30 0x7f46140c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.032+0000 7f4619464640 1 --2- 172.21.15.1:0/650298067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46140c2e40 0x7f46141ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.032+0000 7f461a466640 1 -- 172.21.15.1:0/650298067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46140c2e40 msgr2=0x7f46141ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.032+0000 7f461a466640 1 --2- 172.21.15.1:0/650298067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46140c2e40 0x7f46141ce3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.032+0000 7f461a466640 1 -- 172.21.15.1:0/650298067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46140a6e30 msgr2=0x7f46140c4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.032+0000 7f461a466640 1 --2- 172.21.15.1:0/650298067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46140a6e30 0x7f46140c4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.032+0000 7f461a466640 1 -- 172.21.15.1:0/650298067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46141ead50 con 0x7f46140c3220 2024-03-06T07:20:37.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.032+0000 7f461a466640 1 --2- 172.21.15.1:0/650298067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46140c3220 0x7f46141ce920 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f460c009e80 tx=0x7f460c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:37.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.032+0000 7f460a7fc640 1 -- 172.21.15.1:0/650298067 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c039050 con 0x7f46140c3220 2024-03-06T07:20:37.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.032+0000 7f460a7fc640 1 -- 172.21.15.1:0/650298067 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460c045070 con 0x7f46140c3220 2024-03-06T07:20:37.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.032+0000 7f460a7fc640 1 -- 172.21.15.1:0/650298067 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c03e830 con 0x7f46140c3220 2024-03-06T07:20:37.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.032+0000 7f461bec9640 1 -- 172.21.15.1:0/650298067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46140c16f0 con 0x7f46140c3220 2024-03-06T07:20:37.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.032+0000 7f461bec9640 1 -- 172.21.15.1:0/650298067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46141eafd0 con 0x7f46140c3220 2024-03-06T07:20:37.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.033+0000 7f461bec9640 1 -- 172.21.15.1:0/650298067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46140686e0 con 0x7f46140c3220 2024-03-06T07:20:37.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.034+0000 7f460a7fc640 1 -- 172.21.15.1:0/650298067 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f460c03e9d0 con 0x7f46140c3220 2024-03-06T07:20:37.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.034+0000 7f460a7fc640 1 --2- 172.21.15.1:0/650298067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f45f806b4d0 0x7f45f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.034+0000 7f460a7fc640 1 -- 172.21.15.1:0/650298067 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f460c0ad4a0 con 0x7f46140c3220 2024-03-06T07:20:37.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.034+0000 7f4619c65640 1 --2- 172.21.15.1:0/650298067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f45f806b4d0 0x7f45f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.035+0000 7f4619c65640 1 --2- 172.21.15.1:0/650298067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f45f806b4d0 0x7f45f806d990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f46141ee8c0 tx=0x7f4604008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:37.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.039+0000 7f460a7fc640 1 -- 172.21.15.1:0/650298067 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46141ead50 con 0x7f46140c3220 2024-03-06T07:20:37.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.258+0000 7f461bec9640 1 -- 172.21.15.1:0/650298067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f46140c1db0 con 0x7f46140c3220 2024-03-06T07:20:37.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.905+0000 7f460a7fc640 1 -- 172.21.15.1:0/650298067 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f460c07c170 con 0x7f46140c3220 2024-03-06T07:20:37.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.908+0000 7f461bec9640 1 -- 172.21.15.1:0/650298067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f45f806b4d0 msgr2=0x7f45f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.908+0000 7f461bec9640 1 --2- 172.21.15.1:0/650298067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f45f806b4d0 0x7f45f806d990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f46141ee8c0 tx=0x7f4604008040 comp rx=0 tx=0).stop 2024-03-06T07:20:37.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.908+0000 7f461bec9640 1 -- 172.21.15.1:0/650298067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46140c3220 msgr2=0x7f46141ce920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.908+0000 7f461bec9640 1 --2- 172.21.15.1:0/650298067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46140c3220 0x7f46141ce920 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f460c009e80 tx=0x7f460c004a40 comp rx=0 tx=0).stop 2024-03-06T07:20:37.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.908+0000 7f461bec9640 1 -- 172.21.15.1:0/650298067 shutdown_connections 2024-03-06T07:20:37.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.908+0000 7f461bec9640 1 --2- 172.21.15.1:0/650298067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f45f806b4d0 0x7f45f806d990 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.908+0000 7f461bec9640 1 --2- 172.21.15.1:0/650298067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46140c3220 0x7f46141ce920 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.908+0000 7f461bec9640 1 --2- 172.21.15.1:0/650298067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46140c2e40 0x7f46141ce3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.908+0000 7f461bec9640 1 --2- 172.21.15.1:0/650298067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46140a6e30 0x7f46140c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.908+0000 7f461bec9640 1 -- 172.21.15.1:0/650298067 >> 172.21.15.1:0/650298067 conn(0x7f46140a9700 msgr2=0x7f46140a8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:37.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.908+0000 7f461bec9640 1 -- 172.21.15.1:0/650298067 shutdown_connections 2024-03-06T07:20:37.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:37.908+0000 7f461bec9640 1 -- 172.21.15.1:0/650298067 wait complete. 2024-03-06T07:20:37.920 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:20:37.921 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:38.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.019+0000 7f5be0aaf640 1 Processor -- start 2024-03-06T07:20:38.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.019+0000 7f5be0aaf640 1 -- start start 2024-03-06T07:20:38.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.020+0000 7f5be0aaf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5bdc0c3970 0x7f5bdc0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.020+0000 7f5be0aaf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bdc152840 0x7f5bdc1553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.020+0000 7f5be0aaf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc0c5e40 con 0x7f5bdc0c60b0 2024-03-06T07:20:38.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.020+0000 7f5be0aaf640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bdc0c67f0 con 0x7f5bdc0c3970 2024-03-06T07:20:38.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.020+0000 7f5be0aaf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc0c6a40 con 0x7f5bdc152840 2024-03-06T07:20:38.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.021+0000 7f5bdad9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bdc152840 0x7f5bdc1553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.021+0000 7f5bda59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5bdc0c3970 0x7f5bdc0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.021+0000 7f5bda59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5bdc0c3970 0x7f5bdc0c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:44286/0 (socket says 172.21.15.1:44286) 2024-03-06T07:20:38.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.021+0000 7f5bdad9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bdc152840 0x7f5bdc1553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:41180/0 (socket says 172.21.15.1:41180) 2024-03-06T07:20:38.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.021+0000 7f5bda59c640 1 -- 172.21.15.1:0/424844829 learned_addr learned my addr 172.21.15.1:0/424844829 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:38.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.021+0000 7f5bdad9d640 1 -- 172.21.15.1:0/424844829 >> v2:172.21.15.98:3300/0 conn(0x7f5bdc0c3970 msgr2=0x7f5bdc0c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:38.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.021+0000 7f5bdad9d640 1 --2- 172.21.15.1:0/424844829 >> v2:172.21.15.98:3300/0 conn(0x7f5bdc0c3970 0x7f5bdc0c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.021+0000 7f5bdad9d640 1 -- 172.21.15.1:0/424844829 >> v1:172.21.15.98:6789/0 conn(0x7f5bdc0c60b0 legacy=0x7f5bdc152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.021+0000 7f5bdad9d640 1 -- 172.21.15.1:0/424844829 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc0c6d20 con 0x7f5bdc152840 2024-03-06T07:20:38.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.021+0000 7f5bda59c640 1 --2- 172.21.15.1:0/424844829 >> v2:172.21.15.98:3300/0 conn(0x7f5bdc0c3970 0x7f5bdc0c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:38.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.022+0000 7f5bdad9d640 1 --2- 172.21.15.1:0/424844829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bdc152840 0x7f5bdc1553d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009d90 tx=0x7f5bcc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a0e7c2f71c0ecb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:38.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.022+0000 7f5bd959a640 1 -- 172.21.15.1:0/424844829 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc039070 con 0x7f5bdc152840 2024-03-06T07:20:38.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.022+0000 7f5bd959a640 1 -- 172.21.15.1:0/424844829 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bcc02cc50 con 0x7f5bdc152840 2024-03-06T07:20:38.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.022+0000 7f5bd959a640 1 -- 172.21.15.1:0/424844829 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc034440 con 0x7f5bdc152840 2024-03-06T07:20:38.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.022+0000 7f5be0aaf640 1 -- 172.21.15.1:0/424844829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bdc152840 msgr2=0x7f5bdc1553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.022+0000 7f5be0aaf640 1 --2- 172.21.15.1:0/424844829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bdc152840 0x7f5bdc1553d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009d90 tx=0x7f5bcc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.022+0000 7f5be0aaf640 1 -- 172.21.15.1:0/424844829 shutdown_connections 2024-03-06T07:20:38.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.022+0000 7f5be0aaf640 1 --2- 172.21.15.1:0/424844829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bdc152840 0x7f5bdc1553d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.022+0000 7f5be0aaf640 1 --2- 172.21.15.1:0/424844829 >> v2:172.21.15.98:3300/0 conn(0x7f5bdc0c3970 0x7f5bdc0c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.022+0000 7f5be0aaf640 1 -- 172.21.15.1:0/424844829 >> 172.21.15.1:0/424844829 conn(0x7f5bdc0c88f0 msgr2=0x7f5bdc0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:38.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.023+0000 7f5be0aaf640 1 -- 172.21.15.1:0/424844829 shutdown_connections 2024-03-06T07:20:38.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.023+0000 7f5be0aaf640 1 -- 172.21.15.1:0/424844829 wait complete. 2024-03-06T07:20:38.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.024+0000 7f5be0aaf640 1 Processor -- start 2024-03-06T07:20:38.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.024+0000 7f5be0aaf640 1 -- start start 2024-03-06T07:20:38.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.024+0000 7f5be0aaf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bdc0c3970 0x7f5bdc14a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.024+0000 7f5be0aaf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bdc0c60b0 0x7f5bdc1ee240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.024+0000 7f5be0aaf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bdc152840 0x7f5bdc1ee780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.024+0000 7f5be0aaf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc0c75c0 con 0x7f5bdc0c3970 2024-03-06T07:20:38.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.024+0000 7f5be0aaf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc0c72e0 con 0x7f5bdc0c60b0 2024-03-06T07:20:38.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.024+0000 7f5be0aaf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc0c3d50 con 0x7f5bdc152840 2024-03-06T07:20:38.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.025+0000 7f5bda59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bdc0c3970 0x7f5bdc14a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.025+0000 7f5bda59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bdc0c3970 0x7f5bdc14a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40268/0 (socket says 172.21.15.1:40268) 2024-03-06T07:20:38.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.025+0000 7f5bda59c640 1 -- 172.21.15.1:0/737414916 learned_addr learned my addr 172.21.15.1:0/737414916 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:38.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.025+0000 7f5bdad9d640 1 --2- 172.21.15.1:0/737414916 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bdc152840 0x7f5bdc1ee780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.025+0000 7f5bd9d9b640 1 --2- 172.21.15.1:0/737414916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bdc0c60b0 0x7f5bdc1ee240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.025+0000 7f5bda59c640 1 -- 172.21.15.1:0/737414916 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bdc152840 msgr2=0x7f5bdc1ee780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.025+0000 7f5bda59c640 1 --2- 172.21.15.1:0/737414916 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bdc152840 0x7f5bdc1ee780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.025+0000 7f5bda59c640 1 -- 172.21.15.1:0/737414916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bdc0c60b0 msgr2=0x7f5bdc1ee240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.025+0000 7f5bda59c640 1 --2- 172.21.15.1:0/737414916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bdc0c60b0 0x7f5bdc1ee240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.025+0000 7f5bda59c640 1 -- 172.21.15.1:0/737414916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc1eecc0 con 0x7f5bdc0c3970 2024-03-06T07:20:38.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.025+0000 7f5bdad9d640 1 --2- 172.21.15.1:0/737414916 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bdc152840 0x7f5bdc1ee780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:38.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.025+0000 7f5bda59c640 1 --2- 172.21.15.1:0/737414916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bdc0c3970 0x7f5bdc14a850 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5bd400e9d0 tx=0x7f5bd400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:38.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.025+0000 7f5bd2ffd640 1 -- 172.21.15.1:0/737414916 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4017070 con 0x7f5bdc0c3970 2024-03-06T07:20:38.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.025+0000 7f5bd2ffd640 1 -- 172.21.15.1:0/737414916 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd400ca00 con 0x7f5bdc0c3970 2024-03-06T07:20:38.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.025+0000 7f5be0aaf640 1 -- 172.21.15.1:0/737414916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bdc153ec0 con 0x7f5bdc0c3970 2024-03-06T07:20:38.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.025+0000 7f5bd2ffd640 1 -- 172.21.15.1:0/737414916 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd400cdb0 con 0x7f5bdc0c3970 2024-03-06T07:20:38.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.025+0000 7f5be0aaf640 1 -- 172.21.15.1:0/737414916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bdc071220 con 0x7f5bdc0c3970 2024-03-06T07:20:38.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.026+0000 7f5be0aaf640 1 -- 172.21.15.1:0/737414916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bdc0c64e0 con 0x7f5bdc0c3970 2024-03-06T07:20:38.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.027+0000 7f5bd2ffd640 1 -- 172.21.15.1:0/737414916 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5bd401a070 con 0x7f5bdc0c3970 2024-03-06T07:20:38.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.027+0000 7f5bd2ffd640 1 --2- 172.21.15.1:0/737414916 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bbc06b4d0 0x7f5bbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.028+0000 7f5bd2ffd640 1 -- 172.21.15.1:0/737414916 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5bd408a420 con 0x7f5bdc0c3970 2024-03-06T07:20:38.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.028+0000 7f5bd9d9b640 1 --2- 172.21.15.1:0/737414916 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bbc06b4d0 0x7f5bbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.028+0000 7f5bd9d9b640 1 --2- 172.21.15.1:0/737414916 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bbc06b4d0 0x7f5bbc06d990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc1ef090 tx=0x7f5bc4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:38.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.032+0000 7f5bd2ffd640 1 -- 172.21.15.1:0/737414916 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5bdc0c64e0 con 0x7f5bdc0c3970 2024-03-06T07:20:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.250+0000 7f5be0aaf640 1 -- 172.21.15.1:0/737414916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bdc0686e0 con 0x7f5bdc0c3970 2024-03-06T07:20:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.250+0000 7f5bd2ffd640 1 -- 172.21.15.1:0/737414916 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4964 (secure 0 0 0) 0x7f5bd4058890 con 0x7f5bdc0c3970 2024-03-06T07:20:38.252 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:38.253 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2822958969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2822958969},{"type":"v1","addr":"172.21.15.98:6836","nonce":2822958969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2442906233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2442906233},{"type":"v1","addr":"172.21.15.98:6837","nonce":2442906233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/448746024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":448746024},{"type":"v1","addr":"172.21.15.1:6834","nonce":448746024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:21.207834+0000","modified":"2024-03-06T07:20:37.259661+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1315115982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1315115982},{"type":"v1","addr":"172.21.15.1:6835","nonce":1315115982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-03-06T07:20:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-03-06T07:20:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.252+0000 7f5be0aaf640 1 -- 172.21.15.1:0/737414916 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bbc06b4d0 msgr2=0x7f5bbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.252+0000 7f5be0aaf640 1 --2- 172.21.15.1:0/737414916 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bbc06b4d0 0x7f5bbc06d990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc1ef090 tx=0x7f5bc4009040 comp rx=0 tx=0).stop 2024-03-06T07:20:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.252+0000 7f5be0aaf640 1 -- 172.21.15.1:0/737414916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bdc0c3970 msgr2=0x7f5bdc14a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.252+0000 7f5be0aaf640 1 --2- 172.21.15.1:0/737414916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bdc0c3970 0x7f5bdc14a850 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5bd400e9d0 tx=0x7f5bd400ee90 comp rx=0 tx=0).stop 2024-03-06T07:20:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.252+0000 7f5be0aaf640 1 -- 172.21.15.1:0/737414916 shutdown_connections 2024-03-06T07:20:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.252+0000 7f5be0aaf640 1 --2- 172.21.15.1:0/737414916 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bbc06b4d0 0x7f5bbc06d990 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.252+0000 7f5be0aaf640 1 --2- 172.21.15.1:0/737414916 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bdc152840 0x7f5bdc1ee780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.252+0000 7f5be0aaf640 1 --2- 172.21.15.1:0/737414916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bdc0c60b0 0x7f5bdc1ee240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.252+0000 7f5be0aaf640 1 --2- 172.21.15.1:0/737414916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bdc0c3970 0x7f5bdc14a850 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.252+0000 7f5be0aaf640 1 -- 172.21.15.1:0/737414916 >> 172.21.15.1:0/737414916 conn(0x7f5bdc0c88f0 msgr2=0x7f5bdc0aae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.252+0000 7f5be0aaf640 1 -- 172.21.15.1:0/737414916 shutdown_connections 2024-03-06T07:20:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.252+0000 7f5be0aaf640 1 -- 172.21.15.1:0/737414916 wait complete. 2024-03-06T07:20:38.266 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:38.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.368+0000 7fd8583f7640 1 Processor -- start 2024-03-06T07:20:38.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.368+0000 7fd8583f7640 1 -- start start 2024-03-06T07:20:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.369+0000 7fd8583f7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8500c7510 0x7fd850160b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.369+0000 7fd8583f7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd850072d00 con 0x7fd8500c69b0 2024-03-06T07:20:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.369+0000 7fd8583f7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd850073180 con 0x7fd850159140 2024-03-06T07:20:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.369+0000 7fd8583f7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8500c0e80 con 0x7fd8500c7510 2024-03-06T07:20:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.369+0000 7fd856994640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8500c7510 0x7fd850160b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.369+0000 7fd856994640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8500c7510 0x7fd850160b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40270/0 (socket says 172.21.15.1:40270) 2024-03-06T07:20:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.369+0000 7fd856994640 1 -- 172.21.15.1:0/1582555443 learned_addr learned my addr 172.21.15.1:0/1582555443 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.369+0000 7fd855992640 1 --1- 172.21.15.1:0/1582555443 >> v1:172.21.15.1:6789/0 conn(0x7fd8500c69b0 0x7fd8500c2b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57378/0 (socket says 172.21.15.1:57378) 2024-03-06T07:20:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.370+0000 7fd856994640 1 -- 172.21.15.1:0/1582555443 >> v1:172.21.15.98:6789/0 conn(0x7fd850159140 legacy=0x7fd8500c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.370+0000 7fd856994640 1 -- 172.21.15.1:0/1582555443 >> v1:172.21.15.1:6789/0 conn(0x7fd8500c69b0 legacy=0x7fd8500c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.370+0000 7fd855191640 1 -- 172.21.15.1:0/1582555443 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144439547 0 0) 0x7fd850072d00 con 0x7fd8500c69b0 2024-03-06T07:20:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.370+0000 7fd856994640 1 -- 172.21.15.1:0/1582555443 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd850073460 con 0x7fd8500c7510 2024-03-06T07:20:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.370+0000 7fd856994640 1 --2- 172.21.15.1:0/1582555443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8500c7510 0x7fd850160b20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009e50 tx=0x7fd84c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc7146f2701b759 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.370+0000 7fd855191640 1 -- 172.21.15.1:0/1582555443 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c02cb20 con 0x7fd8500c7510 2024-03-06T07:20:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.371+0000 7fd855191640 1 -- 172.21.15.1:0/1582555443 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd84c02ccc0 con 0x7fd8500c7510 2024-03-06T07:20:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.371+0000 7fd855191640 1 -- 172.21.15.1:0/1582555443 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c034610 con 0x7fd8500c7510 2024-03-06T07:20:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.371+0000 7fd8583f7640 1 -- 172.21.15.1:0/1582555443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8500c7510 msgr2=0x7fd850160b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.371+0000 7fd8583f7640 1 --2- 172.21.15.1:0/1582555443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8500c7510 0x7fd850160b20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009e50 tx=0x7fd84c02bf80 comp rx=0 tx=0).stop 2024-03-06T07:20:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.371+0000 7fd8583f7640 1 -- 172.21.15.1:0/1582555443 shutdown_connections 2024-03-06T07:20:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.371+0000 7fd8583f7640 1 --2- 172.21.15.1:0/1582555443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8500c7510 0x7fd850160b20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.371+0000 7fd8583f7640 1 -- 172.21.15.1:0/1582555443 >> 172.21.15.1:0/1582555443 conn(0x7fd8500c7a50 msgr2=0x7fd8500c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.371+0000 7fd8583f7640 1 -- 172.21.15.1:0/1582555443 shutdown_connections 2024-03-06T07:20:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.371+0000 7fd8583f7640 1 -- 172.21.15.1:0/1582555443 wait complete. 2024-03-06T07:20:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.372+0000 7fd8583f7640 1 Processor -- start 2024-03-06T07:20:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.372+0000 7fd8583f7640 1 -- start start 2024-03-06T07:20:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd8583f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8500c69b0 0x7fd850151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd8583f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8500c7510 0x7fd8500abbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd8583f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd850159140 0x7fd8501532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd8583f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8500c6520 con 0x7fd850159140 2024-03-06T07:20:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd8583f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8500c13b0 con 0x7fd8500c7510 2024-03-06T07:20:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd8583f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8500c4ba0 con 0x7fd8500c69b0 2024-03-06T07:20:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd856994640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd850159140 0x7fd8501532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd856994640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd850159140 0x7fd8501532f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40274/0 (socket says 172.21.15.1:40274) 2024-03-06T07:20:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd856994640 1 -- 172.21.15.1:0/1761795427 learned_addr learned my addr 172.21.15.1:0/1761795427 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd856193640 1 --2- 172.21.15.1:0/1761795427 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8500c69b0 0x7fd850151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd855992640 1 --2- 172.21.15.1:0/1761795427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8500c7510 0x7fd8500abbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd856994640 1 -- 172.21.15.1:0/1761795427 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8500c69b0 msgr2=0x7fd850151560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd856994640 1 --2- 172.21.15.1:0/1761795427 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8500c69b0 0x7fd850151560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd856994640 1 -- 172.21.15.1:0/1761795427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8500c7510 msgr2=0x7fd8500abbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd856994640 1 --2- 172.21.15.1:0/1761795427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8500c7510 0x7fd8500abbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd856994640 1 -- 172.21.15.1:0/1761795427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8501ea990 con 0x7fd850159140 2024-03-06T07:20:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd855992640 1 --2- 172.21.15.1:0/1761795427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8500c7510 0x7fd8500abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd856193640 1 --2- 172.21.15.1:0/1761795427 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8500c69b0 0x7fd850151560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.373+0000 7fd856994640 1 --2- 172.21.15.1:0/1761795427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd850159140 0x7fd8501532f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009e50 tx=0x7fd84c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.374+0000 7fd846ffd640 1 -- 172.21.15.1:0/1761795427 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c043020 con 0x7fd850159140 2024-03-06T07:20:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.374+0000 7fd846ffd640 1 -- 172.21.15.1:0/1761795427 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd84c03e450 con 0x7fd850159140 2024-03-06T07:20:38.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.374+0000 7fd8583f7640 1 -- 172.21.15.1:0/1761795427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd850151bf0 con 0x7fd850159140 2024-03-06T07:20:38.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.374+0000 7fd846ffd640 1 -- 172.21.15.1:0/1761795427 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c009510 con 0x7fd850159140 2024-03-06T07:20:38.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.374+0000 7fd8583f7640 1 -- 172.21.15.1:0/1761795427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8501eaf30 con 0x7fd850159140 2024-03-06T07:20:38.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.375+0000 7fd8583f7640 1 -- 172.21.15.1:0/1761795427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8500686e0 con 0x7fd850159140 2024-03-06T07:20:38.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.375+0000 7fd846ffd640 1 -- 172.21.15.1:0/1761795427 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd84c046020 con 0x7fd850159140 2024-03-06T07:20:38.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.375+0000 7fd846ffd640 1 --2- 172.21.15.1:0/1761795427 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd83406b4d0 0x7fd83406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.375+0000 7fd846ffd640 1 -- 172.21.15.1:0/1761795427 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd84c0acd80 con 0x7fd850159140 2024-03-06T07:20:38.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.376+0000 7fd856193640 1 --2- 172.21.15.1:0/1761795427 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd83406b4d0 0x7fd83406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.376+0000 7fd856193640 1 --2- 172.21.15.1:0/1761795427 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd83406b4d0 0x7fd83406d990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd848000f60 tx=0x7fd848009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:38.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.380+0000 7fd846ffd640 1 -- 172.21.15.1:0/1761795427 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd84c04d030 con 0x7fd850159140 2024-03-06T07:20:38.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.563+0000 7fd8583f7640 1 -- 172.21.15.1:0/1761795427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd850072f60 con 0x7fd850159140 2024-03-06T07:20:38.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.564+0000 7fd846ffd640 1 -- 172.21.15.1:0/1761795427 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7fd84c07b950 con 0x7fd850159140 2024-03-06T07:20:38.566 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:38.566 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:38.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.566+0000 7fd8583f7640 1 -- 172.21.15.1:0/1761795427 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd83406b4d0 msgr2=0x7fd83406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.566+0000 7fd8583f7640 1 --2- 172.21.15.1:0/1761795427 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd83406b4d0 0x7fd83406d990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd848000f60 tx=0x7fd848009040 comp rx=0 tx=0).stop 2024-03-06T07:20:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.566+0000 7fd8583f7640 1 -- 172.21.15.1:0/1761795427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd850159140 msgr2=0x7fd8501532f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.566+0000 7fd8583f7640 1 --2- 172.21.15.1:0/1761795427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd850159140 0x7fd8501532f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009e50 tx=0x7fd84c004a40 comp rx=0 tx=0).stop 2024-03-06T07:20:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.566+0000 7fd8583f7640 1 -- 172.21.15.1:0/1761795427 shutdown_connections 2024-03-06T07:20:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.566+0000 7fd8583f7640 1 --2- 172.21.15.1:0/1761795427 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd83406b4d0 0x7fd83406d990 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.566+0000 7fd8583f7640 1 --2- 172.21.15.1:0/1761795427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd850159140 0x7fd8501532f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.566+0000 7fd8583f7640 1 --2- 172.21.15.1:0/1761795427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8500c7510 0x7fd8500abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.566+0000 7fd8583f7640 1 --2- 172.21.15.1:0/1761795427 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8500c69b0 0x7fd850151560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.566+0000 7fd8583f7640 1 -- 172.21.15.1:0/1761795427 >> 172.21.15.1:0/1761795427 conn(0x7fd8500c7a50 msgr2=0x7fd8501569f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.566+0000 7fd8583f7640 1 -- 172.21.15.1:0/1761795427 shutdown_connections 2024-03-06T07:20:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.566+0000 7fd8583f7640 1 -- 172.21.15.1:0/1761795427 wait complete. 2024-03-06T07:20:38.579 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:20:38.580 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:20:38.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.679+0000 7f8e3b642640 1 Processor -- start 2024-03-06T07:20:38.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.679+0000 7f8e3b642640 1 -- start start 2024-03-06T07:20:38.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.680+0000 7f8e3b642640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8e3414f740 0x7f8e3406dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.680+0000 7f8e3b642640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3406f810 con 0x7f8e340c3620 2024-03-06T07:20:38.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.680+0000 7f8e3b642640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3406fdb0 con 0x7f8e3406c370 2024-03-06T07:20:38.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.680+0000 7f8e3b642640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e340c2010 con 0x7f8e3414f740 2024-03-06T07:20:38.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.680+0000 7f8e38bdd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8e340c3620 0x7f8e3414f030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57382/0 (socket says 172.21.15.1:57382) 2024-03-06T07:20:38.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.680+0000 7f8e38bdd640 1 -- 172.21.15.1:0/1735582598 learned_addr learned my addr 172.21.15.1:0/1735582598 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:38.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.680+0000 7f8e39bdf640 1 --2- 172.21.15.1:0/1735582598 >> v2:172.21.15.98:3300/0 conn(0x7f8e3414f740 0x7f8e3406dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.680+0000 7f8e2bfff640 1 -- 172.21.15.1:0/1735582598 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022869899 0 0) 0x7f8e3406f810 con 0x7f8e340c3620 2024-03-06T07:20:38.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.681+0000 7f8e2bfff640 1 -- 172.21.15.1:0/1735582598 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e20003520 con 0x7f8e340c3620 2024-03-06T07:20:38.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.681+0000 7f8e39bdf640 1 -- 172.21.15.1:0/1735582598 >> v1:172.21.15.98:6789/0 conn(0x7f8e3406c370 legacy=0x7f8e34155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.681+0000 7f8e2bfff640 1 -- 172.21.15.1:0/1735582598 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817847273 0 0) 0x7f8e3406fdb0 con 0x7f8e3406c370 2024-03-06T07:20:38.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.681+0000 7f8e39bdf640 1 -- 172.21.15.1:0/1735582598 >> v1:172.21.15.1:6789/0 conn(0x7f8e340c3620 legacy=0x7f8e3414f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.681+0000 7f8e39bdf640 1 -- 172.21.15.1:0/1735582598 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3406fad0 con 0x7f8e3414f740 2024-03-06T07:20:38.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.681+0000 7f8e39bdf640 1 --2- 172.21.15.1:0/1735582598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e3414f740 0x7f8e3406dd60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c0097a0 tx=0x7f8e2c02b740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34e5d0efe459eeef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:38.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.682+0000 7f8e2bfff640 1 -- 172.21.15.1:0/1735582598 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c037070 con 0x7f8e3414f740 2024-03-06T07:20:38.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.682+0000 7f8e2bfff640 1 -- 172.21.15.1:0/1735582598 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2c030510 con 0x7f8e3414f740 2024-03-06T07:20:38.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.682+0000 7f8e2bfff640 1 -- 172.21.15.1:0/1735582598 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e24001670 con 0x7f8e3414f740 2024-03-06T07:20:38.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.682+0000 7f8e3b642640 1 -- 172.21.15.1:0/1735582598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e3414f740 msgr2=0x7f8e3406dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.682+0000 7f8e3b642640 1 --2- 172.21.15.1:0/1735582598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e3414f740 0x7f8e3406dd60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c0097a0 tx=0x7f8e2c02b740 comp rx=0 tx=0).stop 2024-03-06T07:20:38.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.682+0000 7f8e3b642640 1 -- 172.21.15.1:0/1735582598 shutdown_connections 2024-03-06T07:20:38.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.682+0000 7f8e3b642640 1 --2- 172.21.15.1:0/1735582598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e3414f740 0x7f8e3406dd60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.682+0000 7f8e3b642640 1 -- 172.21.15.1:0/1735582598 >> 172.21.15.1:0/1735582598 conn(0x7f8e340c3cc0 msgr2=0x7f8e340c40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:38.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.682+0000 7f8e3b642640 1 -- 172.21.15.1:0/1735582598 shutdown_connections 2024-03-06T07:20:38.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.682+0000 7f8e3b642640 1 -- 172.21.15.1:0/1735582598 wait complete. 2024-03-06T07:20:38.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.683+0000 7f8e3b642640 1 Processor -- start 2024-03-06T07:20:38.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.684+0000 7f8e3b642640 1 -- start start 2024-03-06T07:20:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.684+0000 7f8e3b642640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e3406c370 0x7f8e341495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.684+0000 7f8e3b642640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e340c3620 0x7f8e341e8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.684+0000 7f8e3b642640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e3414f740 0x7f8e3414d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.684+0000 7f8e3b642640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e34155a60 con 0x7f8e3406c370 2024-03-06T07:20:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.684+0000 7f8e3b642640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e340a9880 con 0x7f8e3414f740 2024-03-06T07:20:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.684+0000 7f8e3b642640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e34155be0 con 0x7f8e340c3620 2024-03-06T07:20:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.684+0000 7f8e393de640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e3406c370 0x7f8e341495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.684+0000 7f8e393de640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e3406c370 0x7f8e341495d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40290/0 (socket says 172.21.15.1:40290) 2024-03-06T07:20:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.684+0000 7f8e393de640 1 -- 172.21.15.1:0/3391753548 learned_addr learned my addr 172.21.15.1:0/3391753548 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.685+0000 7f8e38bdd640 1 --2- 172.21.15.1:0/3391753548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e340c3620 0x7f8e341e8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.685+0000 7f8e393de640 1 -- 172.21.15.1:0/3391753548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e340c3620 msgr2=0x7f8e341e8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.685+0000 7f8e393de640 1 --2- 172.21.15.1:0/3391753548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e340c3620 0x7f8e341e8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.685+0000 7f8e393de640 1 -- 172.21.15.1:0/3391753548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e3414f740 msgr2=0x7f8e3414d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.685+0000 7f8e393de640 1 --2- 172.21.15.1:0/3391753548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e3414f740 0x7f8e3414d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.685+0000 7f8e393de640 1 -- 172.21.15.1:0/3391753548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e341e9580 con 0x7f8e3406c370 2024-03-06T07:20:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.685+0000 7f8e393de640 1 --2- 172.21.15.1:0/3391753548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e3406c370 0x7f8e341495d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000c810 tx=0x7f8e3000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.685+0000 7f8e29ffb640 1 -- 172.21.15.1:0/3391753548 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e30003da0 con 0x7f8e3406c370 2024-03-06T07:20:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.685+0000 7f8e29ffb640 1 -- 172.21.15.1:0/3391753548 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e30005070 con 0x7f8e3406c370 2024-03-06T07:20:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.685+0000 7f8e3b642640 1 -- 172.21.15.1:0/3391753548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e340c67e0 con 0x7f8e3406c370 2024-03-06T07:20:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.685+0000 7f8e29ffb640 1 -- 172.21.15.1:0/3391753548 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e30005420 con 0x7f8e3406c370 2024-03-06T07:20:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.685+0000 7f8e3b642640 1 -- 172.21.15.1:0/3391753548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e3414b590 con 0x7f8e3406c370 2024-03-06T07:20:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.686+0000 7f8e3b642640 1 -- 172.21.15.1:0/3391753548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e340686e0 con 0x7f8e3406c370 2024-03-06T07:20:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.687+0000 7f8e29ffb640 1 -- 172.21.15.1:0/3391753548 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e300055c0 con 0x7f8e3406c370 2024-03-06T07:20:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.687+0000 7f8e29ffb640 1 --2- 172.21.15.1:0/3391753548 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8e1806b4d0 0x7f8e1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.687+0000 7f8e29ffb640 1 -- 172.21.15.1:0/3391753548 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f8e3008a510 con 0x7f8e3406c370 2024-03-06T07:20:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.687+0000 7f8e38bdd640 1 --2- 172.21.15.1:0/3391753548 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8e1806b4d0 0x7f8e1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.688+0000 7f8e38bdd640 1 --2- 172.21.15.1:0/3391753548 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8e1806b4d0 0x7f8e1806d990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8e34152f60 tx=0x7f8e24001990 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:38.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.692+0000 7f8e29ffb640 1 -- 172.21.15.1:0/3391753548 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e30059050 con 0x7f8e3406c370 2024-03-06T07:20:38.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.917+0000 7f8e3b642640 1 -- 172.21.15.1:0/3391753548 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8e340c1df0 con 0x7f8e1806b4d0 2024-03-06T07:20:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.926+0000 7f8e29ffb640 1 -- 172.21.15.1:0/3391753548 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332611 (secure 0 0 0) 0x7f8e340c1df0 con 0x7f8e1806b4d0 2024-03-06T07:20:38.933 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:38.935 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:20:38.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.930+0000 7f8e3b642640 1 -- 172.21.15.1:0/3391753548 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8e1806b4d0 msgr2=0x7f8e1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.930+0000 7f8e3b642640 1 --2- 172.21.15.1:0/3391753548 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8e1806b4d0 0x7f8e1806d990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8e34152f60 tx=0x7f8e24001990 comp rx=0 tx=0).stop 2024-03-06T07:20:38.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.930+0000 7f8e3b642640 1 -- 172.21.15.1:0/3391753548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e3406c370 msgr2=0x7f8e341495d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.930+0000 7f8e3b642640 1 --2- 172.21.15.1:0/3391753548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e3406c370 0x7f8e341495d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000c810 tx=0x7f8e3000ccd0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.930+0000 7f8e3b642640 1 -- 172.21.15.1:0/3391753548 shutdown_connections 2024-03-06T07:20:38.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.930+0000 7f8e3b642640 1 --2- 172.21.15.1:0/3391753548 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8e1806b4d0 0x7f8e1806d990 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.930+0000 7f8e3b642640 1 --2- 172.21.15.1:0/3391753548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e3414f740 0x7f8e3414d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.930+0000 7f8e3b642640 1 --2- 172.21.15.1:0/3391753548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e340c3620 0x7f8e341e8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.930+0000 7f8e3b642640 1 --2- 172.21.15.1:0/3391753548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e3406c370 0x7f8e341495d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.930+0000 7f8e3b642640 1 -- 172.21.15.1:0/3391753548 >> 172.21.15.1:0/3391753548 conn(0x7f8e340c3cc0 msgr2=0x7f8e340c40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:38.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.930+0000 7f8e3b642640 1 -- 172.21.15.1:0/3391753548 shutdown_connections 2024-03-06T07:20:38.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:38.930+0000 7f8e3b642640 1 -- 172.21.15.1:0/3391753548 wait complete. 2024-03-06T07:20:38.944 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-03-06T07:20:38.690346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594867,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":230924,"kb_used_data":2272,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754743796,"statfs":{"total":773094113280,"available":772857647104,"internally_reserved":0,"allocated":2326528,"data_stored":2048878,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2089,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.812155"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701618+0000","last_change":"2024-03-06T07:20:20.860733+0000","last_active":"2024-03-06T07:20:28.701618+0000","last_peered":"2024-03-06T07:20:28.701618+0000","last_clean":"2024-03-06T07:20:28.701618+0000","last_became_active":"2024-03-06T07:20:20.860412+0000","last_became_peered":"2024-03-06T07:20:20.860412+0000","last_unstale":"2024-03-06T07:20:28.701618+0000","last_undegraded":"2024-03-06T07:20:28.701618+0000","last_fullsized":"2024-03-06T07:20:28.701618+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:44.080359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880598+0000","last_change":"2024-03-06T07:20:21.869098+0000","last_active":"2024-03-06T07:20:27.880598+0000","last_peered":"2024-03-06T07:20:27.880598+0000","last_clean":"2024-03-06T07:20:27.880598+0000","last_became_active":"2024-03-06T07:20:21.868808+0000","last_became_peered":"2024-03-06T07:20:21.868808+0000","last_unstale":"2024-03-06T07:20:27.880598+0000","last_undegraded":"2024-03-06T07:20:27.880598+0000","last_fullsized":"2024-03-06T07:20:27.880598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.423089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879651+0000","last_change":"2024-03-06T07:20:26.504312+0000","last_active":"2024-03-06T07:20:27.879651+0000","last_peered":"2024-03-06T07:20:27.879651+0000","last_clean":"2024-03-06T07:20:27.879651+0000","last_became_active":"2024-03-06T07:20:26.503884+0000","last_became_peered":"2024-03-06T07:20:26.503884+0000","last_unstale":"2024-03-06T07:20:27.879651+0000","last_undegraded":"2024-03-06T07:20:27.879651+0000","last_fullsized":"2024-03-06T07:20:27.879651+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:57:56.136234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880779+0000","last_change":"2024-03-06T07:20:20.854641+0000","last_active":"2024-03-06T07:20:27.880779+0000","last_peered":"2024-03-06T07:20:27.880779+0000","last_clean":"2024-03-06T07:20:27.880779+0000","last_became_active":"2024-03-06T07:20:20.854361+0000","last_became_peered":"2024-03-06T07:20:20.854361+0000","last_unstale":"2024-03-06T07:20:27.880779+0000","last_undegraded":"2024-03-06T07:20:27.880779+0000","last_fullsized":"2024-03-06T07:20:27.880779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:57.497682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701411+0000","last_change":"2024-03-06T07:20:21.857450+0000","last_active":"2024-03-06T07:20:28.701411+0000","last_peered":"2024-03-06T07:20:28.701411+0000","last_clean":"2024-03-06T07:20:28.701411+0000","last_became_active":"2024-03-06T07:20:21.855756+0000","last_became_peered":"2024-03-06T07:20:21.855756+0000","last_unstale":"2024-03-06T07:20:28.701411+0000","last_undegraded":"2024-03-06T07:20:28.701411+0000","last_fullsized":"2024-03-06T07:20:28.701411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:17.903391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879763+0000","last_change":"2024-03-06T07:20:25.889839+0000","last_active":"2024-03-06T07:20:27.879763+0000","last_peered":"2024-03-06T07:20:27.879763+0000","last_clean":"2024-03-06T07:20:27.879763+0000","last_became_active":"2024-03-06T07:20:25.889266+0000","last_became_peered":"2024-03-06T07:20:25.889266+0000","last_unstale":"2024-03-06T07:20:27.879763+0000","last_undegraded":"2024-03-06T07:20:27.879763+0000","last_fullsized":"2024-03-06T07:20:27.879763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:51.236302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201405+0000","last_change":"2024-03-06T07:20:20.859562+0000","last_active":"2024-03-06T07:20:28.201405+0000","last_peered":"2024-03-06T07:20:28.201405+0000","last_clean":"2024-03-06T07:20:28.201405+0000","last_became_active":"2024-03-06T07:20:20.859282+0000","last_became_peered":"2024-03-06T07:20:20.859282+0000","last_unstale":"2024-03-06T07:20:28.201405+0000","last_undegraded":"2024-03-06T07:20:28.201405+0000","last_fullsized":"2024-03-06T07:20:28.201405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:52.187127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701578+0000","last_change":"2024-03-06T07:20:21.863171+0000","last_active":"2024-03-06T07:20:28.701578+0000","last_peered":"2024-03-06T07:20:28.701578+0000","last_clean":"2024-03-06T07:20:28.701578+0000","last_became_active":"2024-03-06T07:20:21.862904+0000","last_became_peered":"2024-03-06T07:20:21.862904+0000","last_unstale":"2024-03-06T07:20:28.701578+0000","last_undegraded":"2024-03-06T07:20:28.701578+0000","last_fullsized":"2024-03-06T07:20:28.701578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:07.625355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880420+0000","last_change":"2024-03-06T07:20:25.888576+0000","last_active":"2024-03-06T07:20:27.880420+0000","last_peered":"2024-03-06T07:20:27.880420+0000","last_clean":"2024-03-06T07:20:27.880420+0000","last_became_active":"2024-03-06T07:20:25.886558+0000","last_became_peered":"2024-03-06T07:20:25.886558+0000","last_unstale":"2024-03-06T07:20:27.880420+0000","last_undegraded":"2024-03-06T07:20:27.880420+0000","last_fullsized":"2024-03-06T07:20:27.880420+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:27.800537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201157+0000","last_change":"2024-03-06T07:20:20.861724+0000","last_active":"2024-03-06T07:20:28.201157+0000","last_peered":"2024-03-06T07:20:28.201157+0000","last_clean":"2024-03-06T07:20:28.201157+0000","last_became_active":"2024-03-06T07:20:20.861432+0000","last_became_peered":"2024-03-06T07:20:20.861432+0000","last_unstale":"2024-03-06T07:20:28.201157+0000","last_undegraded":"2024-03-06T07:20:28.201157+0000","last_fullsized":"2024-03-06T07:20:28.201157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:43.621233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702026+0000","last_change":"2024-03-06T07:20:21.862894+0000","last_active":"2024-03-06T07:20:28.702026+0000","last_peered":"2024-03-06T07:20:28.702026+0000","last_clean":"2024-03-06T07:20:28.702026+0000","last_became_active":"2024-03-06T07:20:21.862624+0000","last_became_peered":"2024-03-06T07:20:21.862624+0000","last_unstale":"2024-03-06T07:20:28.702026+0000","last_undegraded":"2024-03-06T07:20:28.702026+0000","last_fullsized":"2024-03-06T07:20:28.702026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:53.701404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879914+0000","last_change":"2024-03-06T07:20:25.891479+0000","last_active":"2024-03-06T07:20:27.879914+0000","last_peered":"2024-03-06T07:20:27.879914+0000","last_clean":"2024-03-06T07:20:27.879914+0000","last_became_active":"2024-03-06T07:20:25.891124+0000","last_became_peered":"2024-03-06T07:20:25.891124+0000","last_unstale":"2024-03-06T07:20:27.879914+0000","last_undegraded":"2024-03-06T07:20:27.879914+0000","last_fullsized":"2024-03-06T07:20:27.879914+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:31.908538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701544+0000","last_change":"2024-03-06T07:20:21.861447+0000","last_active":"2024-03-06T07:20:28.701544+0000","last_peered":"2024-03-06T07:20:28.701544+0000","last_clean":"2024-03-06T07:20:28.701544+0000","last_became_active":"2024-03-06T07:20:21.861169+0000","last_became_peered":"2024-03-06T07:20:21.861169+0000","last_unstale":"2024-03-06T07:20:28.701544+0000","last_undegraded":"2024-03-06T07:20:28.701544+0000","last_fullsized":"2024-03-06T07:20:28.701544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:28.363605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201185+0000","last_change":"2024-03-06T07:20:20.858871+0000","last_active":"2024-03-06T07:20:28.201185+0000","last_peered":"2024-03-06T07:20:28.201185+0000","last_clean":"2024-03-06T07:20:28.201185+0000","last_became_active":"2024-03-06T07:20:20.858481+0000","last_became_peered":"2024-03-06T07:20:20.858481+0000","last_unstale":"2024-03-06T07:20:28.201185+0000","last_undegraded":"2024-03-06T07:20:28.201185+0000","last_fullsized":"2024-03-06T07:20:28.201185+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:58.697205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880550+0000","last_change":"2024-03-06T07:20:25.890028+0000","last_active":"2024-03-06T07:20:27.880550+0000","last_peered":"2024-03-06T07:20:27.880550+0000","last_clean":"2024-03-06T07:20:27.880550+0000","last_became_active":"2024-03-06T07:20:25.889634+0000","last_became_peered":"2024-03-06T07:20:25.889634+0000","last_unstale":"2024-03-06T07:20:27.880550+0000","last_undegraded":"2024-03-06T07:20:27.880550+0000","last_fullsized":"2024-03-06T07:20:27.880550+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:34.847448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880189+0000","last_change":"2024-03-06T07:20:21.863836+0000","last_active":"2024-03-06T07:20:27.880189+0000","last_peered":"2024-03-06T07:20:27.880189+0000","last_clean":"2024-03-06T07:20:27.880189+0000","last_became_active":"2024-03-06T07:20:21.863451+0000","last_became_peered":"2024-03-06T07:20:21.863451+0000","last_unstale":"2024-03-06T07:20:27.880189+0000","last_undegraded":"2024-03-06T07:20:27.880189+0000","last_fullsized":"2024-03-06T07:20:27.880189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:07.625355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365689+0000","last_change":"2024-03-06T07:20:20.847182+0000","last_active":"2024-03-06T07:20:28.365689+0000","last_peered":"2024-03-06T07:20:28.365689+0000","last_clean":"2024-03-06T07:20:28.365689+0000","last_became_active":"2024-03-06T07:20:20.846875+0000","last_became_peered":"2024-03-06T07:20:20.846875+0000","last_unstale":"2024-03-06T07:20:28.365689+0000","last_undegraded":"2024-03-06T07:20:28.365689+0000","last_fullsized":"2024-03-06T07:20:28.365689+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:47.332802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880571+0000","last_change":"2024-03-06T07:20:25.890712+0000","last_active":"2024-03-06T07:20:27.880571+0000","last_peered":"2024-03-06T07:20:27.880571+0000","last_clean":"2024-03-06T07:20:27.880571+0000","last_became_active":"2024-03-06T07:20:25.889840+0000","last_became_peered":"2024-03-06T07:20:25.889840+0000","last_unstale":"2024-03-06T07:20:27.880571+0000","last_undegraded":"2024-03-06T07:20:27.880571+0000","last_fullsized":"2024-03-06T07:20:27.880571+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:46.450423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879843+0000","last_change":"2024-03-06T07:20:21.865384+0000","last_active":"2024-03-06T07:20:27.879843+0000","last_peered":"2024-03-06T07:20:27.879843+0000","last_clean":"2024-03-06T07:20:27.879843+0000","last_became_active":"2024-03-06T07:20:21.865118+0000","last_became_peered":"2024-03-06T07:20:21.865118+0000","last_unstale":"2024-03-06T07:20:27.879843+0000","last_undegraded":"2024-03-06T07:20:27.879843+0000","last_fullsized":"2024-03-06T07:20:27.879843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:36.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365440+0000","last_change":"2024-03-06T07:20:20.856580+0000","last_active":"2024-03-06T07:20:28.365440+0000","last_peered":"2024-03-06T07:20:28.365440+0000","last_clean":"2024-03-06T07:20:28.365440+0000","last_became_active":"2024-03-06T07:20:20.856117+0000","last_became_peered":"2024-03-06T07:20:20.856117+0000","last_unstale":"2024-03-06T07:20:28.365440+0000","last_undegraded":"2024-03-06T07:20:28.365440+0000","last_fullsized":"2024-03-06T07:20:28.365440+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:43.621233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366340+0000","last_change":"2024-03-06T07:20:26.502055+0000","last_active":"2024-03-06T07:20:28.366340+0000","last_peered":"2024-03-06T07:20:28.366340+0000","last_clean":"2024-03-06T07:20:28.366340+0000","last_became_active":"2024-03-06T07:20:26.501689+0000","last_became_peered":"2024-03-06T07:20:26.501689+0000","last_unstale":"2024-03-06T07:20:28.366340+0000","last_undegraded":"2024-03-06T07:20:28.366340+0000","last_fullsized":"2024-03-06T07:20:28.366340+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:34.847448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880426+0000","last_change":"2024-03-06T07:20:21.859723+0000","last_active":"2024-03-06T07:20:27.880426+0000","last_peered":"2024-03-06T07:20:27.880426+0000","last_clean":"2024-03-06T07:20:27.880426+0000","last_became_active":"2024-03-06T07:20:21.859392+0000","last_became_peered":"2024-03-06T07:20:21.859392+0000","last_unstale":"2024-03-06T07:20:27.880426+0000","last_undegraded":"2024-03-06T07:20:27.880426+0000","last_fullsized":"2024-03-06T07:20:27.880426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:59.701834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.200895+0000","last_change":"2024-03-06T07:20:20.860602+0000","last_active":"2024-03-06T07:20:28.200895+0000","last_peered":"2024-03-06T07:20:28.200895+0000","last_clean":"2024-03-06T07:20:28.200895+0000","last_became_active":"2024-03-06T07:20:20.860328+0000","last_became_peered":"2024-03-06T07:20:20.860328+0000","last_unstale":"2024-03-06T07:20:28.200895+0000","last_undegraded":"2024-03-06T07:20:28.200895+0000","last_fullsized":"2024-03-06T07:20:28.200895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:25.763288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880136+0000","last_change":"2024-03-06T07:20:25.889087+0000","last_active":"2024-03-06T07:20:27.880136+0000","last_peered":"2024-03-06T07:20:27.880136+0000","last_clean":"2024-03-06T07:20:27.880136+0000","last_became_active":"2024-03-06T07:20:25.888562+0000","last_became_peered":"2024-03-06T07:20:25.888562+0000","last_unstale":"2024-03-06T07:20:27.880136+0000","last_undegraded":"2024-03-06T07:20:27.880136+0000","last_fullsized":"2024-03-06T07:20:27.880136+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:22.654122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701536+0000","last_change":"2024-03-06T07:20:21.857055+0000","last_active":"2024-03-06T07:20:28.701536+0000","last_peered":"2024-03-06T07:20:28.701536+0000","last_clean":"2024-03-06T07:20:28.701536+0000","last_became_active":"2024-03-06T07:20:21.856743+0000","last_became_peered":"2024-03-06T07:20:21.856743+0000","last_unstale":"2024-03-06T07:20:28.701536+0000","last_undegraded":"2024-03-06T07:20:28.701536+0000","last_fullsized":"2024-03-06T07:20:28.701536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:20.911597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702767+0000","last_change":"2024-03-06T07:20:20.858020+0000","last_active":"2024-03-06T07:20:28.702767+0000","last_peered":"2024-03-06T07:20:28.702767+0000","last_clean":"2024-03-06T07:20:28.702767+0000","last_became_active":"2024-03-06T07:20:20.857748+0000","last_became_peered":"2024-03-06T07:20:20.857748+0000","last_unstale":"2024-03-06T07:20:28.702767+0000","last_undegraded":"2024-03-06T07:20:28.702767+0000","last_fullsized":"2024-03-06T07:20:28.702767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:47.332802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880011+0000","last_change":"2024-03-06T07:20:26.499973+0000","last_active":"2024-03-06T07:20:27.880011+0000","last_peered":"2024-03-06T07:20:27.880011+0000","last_clean":"2024-03-06T07:20:27.880011+0000","last_became_active":"2024-03-06T07:20:26.499480+0000","last_became_peered":"2024-03-06T07:20:26.499480+0000","last_unstale":"2024-03-06T07:20:27.880011+0000","last_undegraded":"2024-03-06T07:20:27.880011+0000","last_fullsized":"2024-03-06T07:20:27.880011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:55.146213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880824+0000","last_change":"2024-03-06T07:20:21.858239+0000","last_active":"2024-03-06T07:20:27.880824+0000","last_peered":"2024-03-06T07:20:27.880824+0000","last_clean":"2024-03-06T07:20:27.880824+0000","last_became_active":"2024-03-06T07:20:21.857896+0000","last_became_peered":"2024-03-06T07:20:21.857896+0000","last_unstale":"2024-03-06T07:20:27.880824+0000","last_undegraded":"2024-03-06T07:20:27.880824+0000","last_fullsized":"2024-03-06T07:20:27.880824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366190+0000","last_change":"2024-03-06T07:20:20.853233+0000","last_active":"2024-03-06T07:20:28.366190+0000","last_peered":"2024-03-06T07:20:28.366190+0000","last_clean":"2024-03-06T07:20:28.366190+0000","last_became_active":"2024-03-06T07:20:20.852941+0000","last_became_peered":"2024-03-06T07:20:20.852941+0000","last_unstale":"2024-03-06T07:20:28.366190+0000","last_undegraded":"2024-03-06T07:20:28.366190+0000","last_fullsized":"2024-03-06T07:20:28.366190+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:25.763288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879421+0000","last_change":"2024-03-06T07:20:26.499314+0000","last_active":"2024-03-06T07:20:27.879421+0000","last_peered":"2024-03-06T07:20:27.879421+0000","last_clean":"2024-03-06T07:20:27.879421+0000","last_became_active":"2024-03-06T07:20:26.498625+0000","last_became_peered":"2024-03-06T07:20:26.498625+0000","last_unstale":"2024-03-06T07:20:27.879421+0000","last_undegraded":"2024-03-06T07:20:27.879421+0000","last_fullsized":"2024-03-06T07:20:27.879421+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:05.727952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365917+0000","last_change":"2024-03-06T07:20:21.861526+0000","last_active":"2024-03-06T07:20:28.365917+0000","last_peered":"2024-03-06T07:20:28.365917+0000","last_clean":"2024-03-06T07:20:28.365917+0000","last_became_active":"2024-03-06T07:20:21.861256+0000","last_became_peered":"2024-03-06T07:20:21.861256+0000","last_unstale":"2024-03-06T07:20:28.365917+0000","last_undegraded":"2024-03-06T07:20:28.365917+0000","last_fullsized":"2024-03-06T07:20:28.365917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.423089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201484+0000","last_change":"2024-03-06T07:20:20.858234+0000","last_active":"2024-03-06T07:20:28.201484+0000","last_peered":"2024-03-06T07:20:28.201484+0000","last_clean":"2024-03-06T07:20:28.201484+0000","last_became_active":"2024-03-06T07:20:20.857910+0000","last_became_peered":"2024-03-06T07:20:20.857910+0000","last_unstale":"2024-03-06T07:20:28.201484+0000","last_undegraded":"2024-03-06T07:20:28.201484+0000","last_fullsized":"2024-03-06T07:20:28.201484+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:04.756600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701326+0000","last_change":"2024-03-06T07:20:26.498727+0000","last_active":"2024-03-06T07:20:28.701326+0000","last_peered":"2024-03-06T07:20:28.701326+0000","last_clean":"2024-03-06T07:20:28.701326+0000","last_became_active":"2024-03-06T07:20:26.498272+0000","last_became_peered":"2024-03-06T07:20:26.498272+0000","last_unstale":"2024-03-06T07:20:28.701326+0000","last_undegraded":"2024-03-06T07:20:28.701326+0000","last_fullsized":"2024-03-06T07:20:28.701326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:28.909072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702156+0000","last_change":"2024-03-06T07:20:21.868324+0000","last_active":"2024-03-06T07:20:28.702156+0000","last_peered":"2024-03-06T07:20:28.702156+0000","last_clean":"2024-03-06T07:20:28.702156+0000","last_became_active":"2024-03-06T07:20:21.868032+0000","last_became_peered":"2024-03-06T07:20:21.868032+0000","last_unstale":"2024-03-06T07:20:28.702156+0000","last_undegraded":"2024-03-06T07:20:28.702156+0000","last_fullsized":"2024-03-06T07:20:28.702156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:17.299208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366009+0000","last_change":"2024-03-06T07:20:20.851073+0000","last_active":"2024-03-06T07:20:28.366009+0000","last_peered":"2024-03-06T07:20:28.366009+0000","last_clean":"2024-03-06T07:20:28.366009+0000","last_became_active":"2024-03-06T07:20:20.850774+0000","last_became_peered":"2024-03-06T07:20:20.850774+0000","last_unstale":"2024-03-06T07:20:28.366009+0000","last_undegraded":"2024-03-06T07:20:28.366009+0000","last_fullsized":"2024-03-06T07:20:28.366009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:15.716891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365536+0000","last_change":"2024-03-06T07:20:25.895118+0000","last_active":"2024-03-06T07:20:28.365536+0000","last_peered":"2024-03-06T07:20:28.365536+0000","last_clean":"2024-03-06T07:20:28.365536+0000","last_became_active":"2024-03-06T07:20:25.894354+0000","last_became_peered":"2024-03-06T07:20:25.894354+0000","last_unstale":"2024-03-06T07:20:28.365536+0000","last_undegraded":"2024-03-06T07:20:28.365536+0000","last_fullsized":"2024-03-06T07:20:28.365536+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:28.909072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880665+0000","last_change":"2024-03-06T07:20:21.861910+0000","last_active":"2024-03-06T07:20:27.880665+0000","last_peered":"2024-03-06T07:20:27.880665+0000","last_clean":"2024-03-06T07:20:27.880665+0000","last_became_active":"2024-03-06T07:20:21.861659+0000","last_became_peered":"2024-03-06T07:20:21.861659+0000","last_unstale":"2024-03-06T07:20:27.880665+0000","last_undegraded":"2024-03-06T07:20:27.880665+0000","last_fullsized":"2024-03-06T07:20:27.880665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:44.625861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701886+0000","last_change":"2024-03-06T07:20:20.849833+0000","last_active":"2024-03-06T07:20:28.701886+0000","last_peered":"2024-03-06T07:20:28.701886+0000","last_clean":"2024-03-06T07:20:28.701886+0000","last_became_active":"2024-03-06T07:20:20.849563+0000","last_became_peered":"2024-03-06T07:20:20.849563+0000","last_unstale":"2024-03-06T07:20:28.701886+0000","last_undegraded":"2024-03-06T07:20:28.701886+0000","last_fullsized":"2024-03-06T07:20:28.701886+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:43.621233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366005+0000","last_change":"2024-03-06T07:20:25.897100+0000","last_active":"2024-03-06T07:20:28.366005+0000","last_peered":"2024-03-06T07:20:28.366005+0000","last_clean":"2024-03-06T07:20:28.366005+0000","last_became_active":"2024-03-06T07:20:25.896658+0000","last_became_peered":"2024-03-06T07:20:25.896658+0000","last_unstale":"2024-03-06T07:20:28.366005+0000","last_undegraded":"2024-03-06T07:20:28.366005+0000","last_fullsized":"2024-03-06T07:20:28.366005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:27.800537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365526+0000","last_change":"2024-03-06T07:20:21.864469+0000","last_active":"2024-03-06T07:20:28.365526+0000","last_peered":"2024-03-06T07:20:28.365526+0000","last_clean":"2024-03-06T07:20:28.365526+0000","last_became_active":"2024-03-06T07:20:21.864141+0000","last_became_peered":"2024-03-06T07:20:21.864141+0000","last_unstale":"2024-03-06T07:20:28.365526+0000","last_undegraded":"2024-03-06T07:20:28.365526+0000","last_fullsized":"2024-03-06T07:20:28.365526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:17.903391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.878975+0000","last_change":"2024-03-06T07:20:20.845652+0000","last_active":"2024-03-06T07:20:27.878975+0000","last_peered":"2024-03-06T07:20:27.878975+0000","last_clean":"2024-03-06T07:20:27.878975+0000","last_became_active":"2024-03-06T07:20:20.845350+0000","last_became_peered":"2024-03-06T07:20:20.845350+0000","last_unstale":"2024-03-06T07:20:27.878975+0000","last_undegraded":"2024-03-06T07:20:27.878975+0000","last_fullsized":"2024-03-06T07:20:27.878975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:53.997041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366358+0000","last_change":"2024-03-06T07:20:25.898164+0000","last_active":"2024-03-06T07:20:28.366358+0000","last_peered":"2024-03-06T07:20:28.366358+0000","last_clean":"2024-03-06T07:20:28.366358+0000","last_became_active":"2024-03-06T07:20:25.897755+0000","last_became_peered":"2024-03-06T07:20:25.897755+0000","last_unstale":"2024-03-06T07:20:28.366358+0000","last_undegraded":"2024-03-06T07:20:28.366358+0000","last_fullsized":"2024-03-06T07:20:28.366358+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:46.450423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701002+0000","last_change":"2024-03-06T07:20:21.850395+0000","last_active":"2024-03-06T07:20:28.701002+0000","last_peered":"2024-03-06T07:20:28.701002+0000","last_clean":"2024-03-06T07:20:28.701002+0000","last_became_active":"2024-03-06T07:20:21.850117+0000","last_became_peered":"2024-03-06T07:20:21.850117+0000","last_unstale":"2024-03-06T07:20:28.701002+0000","last_undegraded":"2024-03-06T07:20:28.701002+0000","last_fullsized":"2024-03-06T07:20:28.701002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:42.612183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.200831+0000","last_change":"2024-03-06T07:20:20.857982+0000","last_active":"2024-03-06T07:20:28.200831+0000","last_peered":"2024-03-06T07:20:28.200831+0000","last_clean":"2024-03-06T07:20:28.200831+0000","last_became_active":"2024-03-06T07:20:20.857726+0000","last_became_peered":"2024-03-06T07:20:20.857726+0000","last_unstale":"2024-03-06T07:20:28.200831+0000","last_undegraded":"2024-03-06T07:20:28.200831+0000","last_fullsized":"2024-03-06T07:20:28.200831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:24.258322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701481+0000","last_change":"2024-03-06T07:20:26.502037+0000","last_active":"2024-03-06T07:20:28.701481+0000","last_peered":"2024-03-06T07:20:28.701481+0000","last_clean":"2024-03-06T07:20:28.701481+0000","last_became_active":"2024-03-06T07:20:26.501666+0000","last_became_peered":"2024-03-06T07:20:26.501666+0000","last_unstale":"2024-03-06T07:20:28.701481+0000","last_undegraded":"2024-03-06T07:20:28.701481+0000","last_fullsized":"2024-03-06T07:20:28.701481+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:27.800537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880611+0000","last_change":"2024-03-06T07:20:21.858533+0000","last_active":"2024-03-06T07:20:27.880611+0000","last_peered":"2024-03-06T07:20:27.880611+0000","last_clean":"2024-03-06T07:20:27.880611+0000","last_became_active":"2024-03-06T07:20:21.857499+0000","last_became_peered":"2024-03-06T07:20:21.857499+0000","last_unstale":"2024-03-06T07:20:27.880611+0000","last_undegraded":"2024-03-06T07:20:27.880611+0000","last_fullsized":"2024-03-06T07:20:27.880611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:27.196530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701468+0000","last_change":"2024-03-06T07:20:20.858802+0000","last_active":"2024-03-06T07:20:28.701468+0000","last_peered":"2024-03-06T07:20:28.701468+0000","last_clean":"2024-03-06T07:20:28.701468+0000","last_became_active":"2024-03-06T07:20:20.857457+0000","last_became_peered":"2024-03-06T07:20:20.857457+0000","last_unstale":"2024-03-06T07:20:28.701468+0000","last_undegraded":"2024-03-06T07:20:28.701468+0000","last_fullsized":"2024-03-06T07:20:28.701468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:57.497682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":179,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201006+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:20:28.201006+0000","last_peered":"2024-03-06T07:20:28.201006+0000","last_clean":"2024-03-06T07:20:28.201006+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:20:28.201006+0000","last_undegraded":"2024-03-06T07:20:28.201006+0000","last_fullsized":"2024-03-06T07:20:28.201006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702367+0000","last_change":"2024-03-06T07:20:26.502317+0000","last_active":"2024-03-06T07:20:28.702367+0000","last_peered":"2024-03-06T07:20:28.702367+0000","last_clean":"2024-03-06T07:20:28.702367+0000","last_became_active":"2024-03-06T07:20:26.501807+0000","last_became_peered":"2024-03-06T07:20:26.501807+0000","last_unstale":"2024-03-06T07:20:28.702367+0000","last_undegraded":"2024-03-06T07:20:28.702367+0000","last_fullsized":"2024-03-06T07:20:28.702367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:30.890766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201288+0000","last_change":"2024-03-06T07:20:21.855669+0000","last_active":"2024-03-06T07:20:28.201288+0000","last_peered":"2024-03-06T07:20:28.201288+0000","last_clean":"2024-03-06T07:20:28.201288+0000","last_became_active":"2024-03-06T07:20:21.855154+0000","last_became_peered":"2024-03-06T07:20:21.855154+0000","last_unstale":"2024-03-06T07:20:28.201288+0000","last_undegraded":"2024-03-06T07:20:28.201288+0000","last_fullsized":"2024-03-06T07:20:28.201288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:26.855742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879323+0000","last_change":"2024-03-06T07:20:20.852702+0000","last_active":"2024-03-06T07:20:27.879323+0000","last_peered":"2024-03-06T07:20:27.879323+0000","last_clean":"2024-03-06T07:20:27.879323+0000","last_became_active":"2024-03-06T07:20:20.852434+0000","last_became_peered":"2024-03-06T07:20:20.852434+0000","last_unstale":"2024-03-06T07:20:27.879323+0000","last_undegraded":"2024-03-06T07:20:27.879323+0000","last_fullsized":"2024-03-06T07:20:27.879323+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:44.080359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702280+0000","last_change":"2024-03-06T07:20:26.499169+0000","last_active":"2024-03-06T07:20:28.702280+0000","last_peered":"2024-03-06T07:20:28.702280+0000","last_clean":"2024-03-06T07:20:28.702280+0000","last_became_active":"2024-03-06T07:20:26.498541+0000","last_became_peered":"2024-03-06T07:20:26.498541+0000","last_unstale":"2024-03-06T07:20:28.702280+0000","last_undegraded":"2024-03-06T07:20:28.702280+0000","last_fullsized":"2024-03-06T07:20:28.702280+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:49.037215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701682+0000","last_change":"2024-03-06T07:20:21.857920+0000","last_active":"2024-03-06T07:20:28.701682+0000","last_peered":"2024-03-06T07:20:28.701682+0000","last_clean":"2024-03-06T07:20:28.701682+0000","last_became_active":"2024-03-06T07:20:21.857645+0000","last_became_peered":"2024-03-06T07:20:21.857645+0000","last_unstale":"2024-03-06T07:20:28.701682+0000","last_undegraded":"2024-03-06T07:20:28.701682+0000","last_fullsized":"2024-03-06T07:20:28.701682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:47.201278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366028+0000","last_change":"2024-03-06T07:20:20.857009+0000","last_active":"2024-03-06T07:20:28.366028+0000","last_peered":"2024-03-06T07:20:28.366028+0000","last_clean":"2024-03-06T07:20:28.366028+0000","last_became_active":"2024-03-06T07:20:20.856701+0000","last_became_peered":"2024-03-06T07:20:20.856701+0000","last_unstale":"2024-03-06T07:20:28.366028+0000","last_undegraded":"2024-03-06T07:20:28.366028+0000","last_fullsized":"2024-03-06T07:20:28.366028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:24.258322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701482+0000","last_change":"2024-03-06T07:20:25.887514+0000","last_active":"2024-03-06T07:20:28.701482+0000","last_peered":"2024-03-06T07:20:28.701482+0000","last_clean":"2024-03-06T07:20:28.701482+0000","last_became_active":"2024-03-06T07:20:25.887125+0000","last_became_peered":"2024-03-06T07:20:25.887125+0000","last_unstale":"2024-03-06T07:20:28.701482+0000","last_undegraded":"2024-03-06T07:20:28.701482+0000","last_fullsized":"2024-03-06T07:20:28.701482+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:34.847448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201678+0000","last_change":"2024-03-06T07:20:21.851028+0000","last_active":"2024-03-06T07:20:28.201678+0000","last_peered":"2024-03-06T07:20:28.201678+0000","last_clean":"2024-03-06T07:20:28.201678+0000","last_became_active":"2024-03-06T07:20:21.850762+0000","last_became_peered":"2024-03-06T07:20:21.850762+0000","last_unstale":"2024-03-06T07:20:28.201678+0000","last_undegraded":"2024-03-06T07:20:28.201678+0000","last_fullsized":"2024-03-06T07:20:28.201678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:27.873513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879685+0000","last_change":"2024-03-06T07:20:20.849572+0000","last_active":"2024-03-06T07:20:27.879685+0000","last_peered":"2024-03-06T07:20:27.879685+0000","last_clean":"2024-03-06T07:20:27.879685+0000","last_became_active":"2024-03-06T07:20:20.849266+0000","last_became_peered":"2024-03-06T07:20:20.849266+0000","last_unstale":"2024-03-06T07:20:27.879685+0000","last_undegraded":"2024-03-06T07:20:27.879685+0000","last_fullsized":"2024-03-06T07:20:27.879685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:47.332802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701273+0000","last_change":"2024-03-06T07:20:26.502258+0000","last_active":"2024-03-06T07:20:28.701273+0000","last_peered":"2024-03-06T07:20:28.701273+0000","last_clean":"2024-03-06T07:20:28.701273+0000","last_became_active":"2024-03-06T07:20:26.501819+0000","last_became_peered":"2024-03-06T07:20:26.501819+0000","last_unstale":"2024-03-06T07:20:28.701273+0000","last_undegraded":"2024-03-06T07:20:28.701273+0000","last_fullsized":"2024-03-06T07:20:28.701273+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:34.847448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201451+0000","last_change":"2024-03-06T07:20:21.844939+0000","last_active":"2024-03-06T07:20:28.201451+0000","last_peered":"2024-03-06T07:20:28.201451+0000","last_clean":"2024-03-06T07:20:28.201451+0000","last_became_active":"2024-03-06T07:20:21.844652+0000","last_became_peered":"2024-03-06T07:20:21.844652+0000","last_unstale":"2024-03-06T07:20:28.201451+0000","last_undegraded":"2024-03-06T07:20:28.201451+0000","last_fullsized":"2024-03-06T07:20:28.201451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:07.625355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880344+0000","last_change":"2024-03-06T07:20:20.850150+0000","last_active":"2024-03-06T07:20:27.880344+0000","last_peered":"2024-03-06T07:20:27.880344+0000","last_clean":"2024-03-06T07:20:27.880344+0000","last_became_active":"2024-03-06T07:20:20.849834+0000","last_became_peered":"2024-03-06T07:20:20.849834+0000","last_unstale":"2024-03-06T07:20:27.880344+0000","last_undegraded":"2024-03-06T07:20:27.880344+0000","last_fullsized":"2024-03-06T07:20:27.880344+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:56.227109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879732+0000","last_change":"2024-03-06T07:20:25.888682+0000","last_active":"2024-03-06T07:20:27.879732+0000","last_peered":"2024-03-06T07:20:27.879732+0000","last_clean":"2024-03-06T07:20:27.879732+0000","last_became_active":"2024-03-06T07:20:25.888259+0000","last_became_peered":"2024-03-06T07:20:25.888259+0000","last_unstale":"2024-03-06T07:20:27.879732+0000","last_undegraded":"2024-03-06T07:20:27.879732+0000","last_fullsized":"2024-03-06T07:20:27.879732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:57.736429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201088+0000","last_change":"2024-03-06T07:20:21.851600+0000","last_active":"2024-03-06T07:20:28.201088+0000","last_peered":"2024-03-06T07:20:28.201088+0000","last_clean":"2024-03-06T07:20:28.201088+0000","last_became_active":"2024-03-06T07:20:21.851347+0000","last_became_peered":"2024-03-06T07:20:21.851347+0000","last_unstale":"2024-03-06T07:20:28.201088+0000","last_undegraded":"2024-03-06T07:20:28.201088+0000","last_fullsized":"2024-03-06T07:20:28.201088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:03.883136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.878780+0000","last_change":"2024-03-06T07:20:20.847128+0000","last_active":"2024-03-06T07:20:27.878780+0000","last_peered":"2024-03-06T07:20:27.878780+0000","last_clean":"2024-03-06T07:20:27.878780+0000","last_became_active":"2024-03-06T07:20:20.846822+0000","last_became_peered":"2024-03-06T07:20:20.846822+0000","last_unstale":"2024-03-06T07:20:27.878780+0000","last_undegraded":"2024-03-06T07:20:27.878780+0000","last_fullsized":"2024-03-06T07:20:27.878780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:57.497682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701347+0000","last_change":"2024-03-06T07:20:25.888397+0000","last_active":"2024-03-06T07:20:28.701347+0000","last_peered":"2024-03-06T07:20:28.701347+0000","last_clean":"2024-03-06T07:20:28.701347+0000","last_became_active":"2024-03-06T07:20:25.888017+0000","last_became_peered":"2024-03-06T07:20:25.888017+0000","last_unstale":"2024-03-06T07:20:28.701347+0000","last_undegraded":"2024-03-06T07:20:28.701347+0000","last_fullsized":"2024-03-06T07:20:28.701347+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:22.654122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702085+0000","last_change":"2024-03-06T07:20:21.863218+0000","last_active":"2024-03-06T07:20:28.702085+0000","last_peered":"2024-03-06T07:20:28.702085+0000","last_clean":"2024-03-06T07:20:28.702085+0000","last_became_active":"2024-03-06T07:20:21.862943+0000","last_became_peered":"2024-03-06T07:20:21.862943+0000","last_unstale":"2024-03-06T07:20:28.702085+0000","last_undegraded":"2024-03-06T07:20:28.702085+0000","last_fullsized":"2024-03-06T07:20:28.702085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:17.903391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701478+0000","last_change":"2024-03-06T07:20:20.855196+0000","last_active":"2024-03-06T07:20:28.701478+0000","last_peered":"2024-03-06T07:20:28.701478+0000","last_clean":"2024-03-06T07:20:28.701478+0000","last_became_active":"2024-03-06T07:20:20.854047+0000","last_became_peered":"2024-03-06T07:20:20.854047+0000","last_unstale":"2024-03-06T07:20:28.701478+0000","last_undegraded":"2024-03-06T07:20:28.701478+0000","last_fullsized":"2024-03-06T07:20:28.701478+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:15.716891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880771+0000","last_change":"2024-03-06T07:20:26.502104+0000","last_active":"2024-03-06T07:20:27.880771+0000","last_peered":"2024-03-06T07:20:27.880771+0000","last_clean":"2024-03-06T07:20:27.880771+0000","last_became_active":"2024-03-06T07:20:26.501701+0000","last_became_peered":"2024-03-06T07:20:26.501701+0000","last_unstale":"2024-03-06T07:20:27.880771+0000","last_undegraded":"2024-03-06T07:20:27.880771+0000","last_fullsized":"2024-03-06T07:20:27.880771+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:13.344053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201065+0000","last_change":"2024-03-06T07:20:21.850971+0000","last_active":"2024-03-06T07:20:28.201065+0000","last_peered":"2024-03-06T07:20:28.201065+0000","last_clean":"2024-03-06T07:20:28.201065+0000","last_became_active":"2024-03-06T07:20:21.850700+0000","last_became_peered":"2024-03-06T07:20:21.850700+0000","last_unstale":"2024-03-06T07:20:28.201065+0000","last_undegraded":"2024-03-06T07:20:28.201065+0000","last_fullsized":"2024-03-06T07:20:28.201065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:17.299208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701777+0000","last_change":"2024-03-06T07:20:20.856747+0000","last_active":"2024-03-06T07:20:28.701777+0000","last_peered":"2024-03-06T07:20:28.701777+0000","last_clean":"2024-03-06T07:20:28.701777+0000","last_became_active":"2024-03-06T07:20:20.856431+0000","last_became_peered":"2024-03-06T07:20:20.856431+0000","last_unstale":"2024-03-06T07:20:28.701777+0000","last_undegraded":"2024-03-06T07:20:28.701777+0000","last_fullsized":"2024-03-06T07:20:28.701777+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:15.716891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880386+0000","last_change":"2024-03-06T07:20:26.498675+0000","last_active":"2024-03-06T07:20:27.880386+0000","last_peered":"2024-03-06T07:20:27.880386+0000","last_clean":"2024-03-06T07:20:27.880386+0000","last_became_active":"2024-03-06T07:20:26.498199+0000","last_became_peered":"2024-03-06T07:20:26.498199+0000","last_unstale":"2024-03-06T07:20:27.880386+0000","last_undegraded":"2024-03-06T07:20:27.880386+0000","last_fullsized":"2024-03-06T07:20:27.880386+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:53.210084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701227+0000","last_change":"2024-03-06T07:20:21.845586+0000","last_active":"2024-03-06T07:20:28.701227+0000","last_peered":"2024-03-06T07:20:28.701227+0000","last_clean":"2024-03-06T07:20:28.701227+0000","last_became_active":"2024-03-06T07:20:21.845311+0000","last_became_peered":"2024-03-06T07:20:21.845311+0000","last_unstale":"2024-03-06T07:20:28.701227+0000","last_undegraded":"2024-03-06T07:20:28.701227+0000","last_fullsized":"2024-03-06T07:20:28.701227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:58.153137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701540+0000","last_change":"2024-03-06T07:20:20.849649+0000","last_active":"2024-03-06T07:20:28.701540+0000","last_peered":"2024-03-06T07:20:28.701540+0000","last_clean":"2024-03-06T07:20:28.701540+0000","last_became_active":"2024-03-06T07:20:20.849199+0000","last_became_peered":"2024-03-06T07:20:20.849199+0000","last_unstale":"2024-03-06T07:20:28.701540+0000","last_undegraded":"2024-03-06T07:20:28.701540+0000","last_fullsized":"2024-03-06T07:20:28.701540+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:22.023999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702184+0000","last_change":"2024-03-06T07:20:26.499070+0000","last_active":"2024-03-06T07:20:28.702184+0000","last_peered":"2024-03-06T07:20:28.702184+0000","last_clean":"2024-03-06T07:20:28.702184+0000","last_became_active":"2024-03-06T07:20:26.498601+0000","last_became_peered":"2024-03-06T07:20:26.498601+0000","last_unstale":"2024-03-06T07:20:28.702184+0000","last_undegraded":"2024-03-06T07:20:28.702184+0000","last_fullsized":"2024-03-06T07:20:28.702184+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:46.450423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201193+0000","last_change":"2024-03-06T07:20:21.857736+0000","last_active":"2024-03-06T07:20:28.201193+0000","last_peered":"2024-03-06T07:20:28.201193+0000","last_clean":"2024-03-06T07:20:28.201193+0000","last_became_active":"2024-03-06T07:20:21.857492+0000","last_became_peered":"2024-03-06T07:20:21.857492+0000","last_unstale":"2024-03-06T07:20:28.201193+0000","last_undegraded":"2024-03-06T07:20:28.201193+0000","last_fullsized":"2024-03-06T07:20:28.201193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:45.002191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:30.723150+0000","last_change":"2024-03-06T07:20:20.856316+0000","last_active":"2024-03-06T07:20:30.723150+0000","last_peered":"2024-03-06T07:20:30.723150+0000","last_clean":"2024-03-06T07:20:30.723150+0000","last_became_active":"2024-03-06T07:20:20.855994+0000","last_became_peered":"2024-03-06T07:20:20.855994+0000","last_unstale":"2024-03-06T07:20:30.723150+0000","last_undegraded":"2024-03-06T07:20:30.723150+0000","last_fullsized":"2024-03-06T07:20:30.723150+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:53.997041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3701,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365632+0000","last_change":"2024-03-06T07:20:25.895379+0000","last_active":"2024-03-06T07:20:28.365632+0000","last_peered":"2024-03-06T07:20:28.365632+0000","last_clean":"2024-03-06T07:20:28.365632+0000","last_became_active":"2024-03-06T07:20:25.894458+0000","last_became_peered":"2024-03-06T07:20:25.894458+0000","last_unstale":"2024-03-06T07:20:28.365632+0000","last_undegraded":"2024-03-06T07:20:28.365632+0000","last_fullsized":"2024-03-06T07:20:28.365632+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:22.654122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880548+0000","last_change":"2024-03-06T07:20:21.844398+0000","last_active":"2024-03-06T07:20:27.880548+0000","last_peered":"2024-03-06T07:20:27.880548+0000","last_clean":"2024-03-06T07:20:27.880548+0000","last_became_active":"2024-03-06T07:20:21.843839+0000","last_became_peered":"2024-03-06T07:20:21.843839+0000","last_unstale":"2024-03-06T07:20:27.880548+0000","last_undegraded":"2024-03-06T07:20:27.880548+0000","last_fullsized":"2024-03-06T07:20:27.880548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:25.262951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365950+0000","last_change":"2024-03-06T07:20:20.850772+0000","last_active":"2024-03-06T07:20:28.365950+0000","last_peered":"2024-03-06T07:20:28.365950+0000","last_clean":"2024-03-06T07:20:28.365950+0000","last_became_active":"2024-03-06T07:20:20.850490+0000","last_became_peered":"2024-03-06T07:20:20.850490+0000","last_unstale":"2024-03-06T07:20:28.365950+0000","last_undegraded":"2024-03-06T07:20:28.365950+0000","last_fullsized":"2024-03-06T07:20:28.365950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:58.697205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879970+0000","last_change":"2024-03-06T07:20:25.887556+0000","last_active":"2024-03-06T07:20:27.879970+0000","last_peered":"2024-03-06T07:20:27.879970+0000","last_clean":"2024-03-06T07:20:27.879970+0000","last_became_active":"2024-03-06T07:20:25.887175+0000","last_became_peered":"2024-03-06T07:20:25.887175+0000","last_unstale":"2024-03-06T07:20:27.879970+0000","last_undegraded":"2024-03-06T07:20:27.879970+0000","last_fullsized":"2024-03-06T07:20:27.879970+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:28.909072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701501+0000","last_change":"2024-03-06T07:20:21.856048+0000","last_active":"2024-03-06T07:20:28.701501+0000","last_peered":"2024-03-06T07:20:28.701501+0000","last_clean":"2024-03-06T07:20:28.701501+0000","last_became_active":"2024-03-06T07:20:21.854721+0000","last_became_peered":"2024-03-06T07:20:21.854721+0000","last_unstale":"2024-03-06T07:20:28.701501+0000","last_undegraded":"2024-03-06T07:20:28.701501+0000","last_fullsized":"2024-03-06T07:20:28.701501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:00.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702082+0000","last_change":"2024-03-06T07:20:20.859239+0000","last_active":"2024-03-06T07:20:28.702082+0000","last_peered":"2024-03-06T07:20:28.702082+0000","last_clean":"2024-03-06T07:20:28.702082+0000","last_became_active":"2024-03-06T07:20:20.858860+0000","last_became_peered":"2024-03-06T07:20:20.858860+0000","last_unstale":"2024-03-06T07:20:28.702082+0000","last_undegraded":"2024-03-06T07:20:28.702082+0000","last_fullsized":"2024-03-06T07:20:28.702082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:47.332802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879712+0000","last_change":"2024-03-06T07:20:25.890675+0000","last_active":"2024-03-06T07:20:27.879712+0000","last_peered":"2024-03-06T07:20:27.879712+0000","last_clean":"2024-03-06T07:20:27.879712+0000","last_became_active":"2024-03-06T07:20:25.890282+0000","last_became_peered":"2024-03-06T07:20:25.890282+0000","last_unstale":"2024-03-06T07:20:27.879712+0000","last_undegraded":"2024-03-06T07:20:27.879712+0000","last_fullsized":"2024-03-06T07:20:27.879712+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:10:13.387120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702028+0000","last_change":"2024-03-06T07:20:21.864960+0000","last_active":"2024-03-06T07:20:28.702028+0000","last_peered":"2024-03-06T07:20:28.702028+0000","last_clean":"2024-03-06T07:20:28.702028+0000","last_became_active":"2024-03-06T07:20:21.864687+0000","last_became_peered":"2024-03-06T07:20:21.864687+0000","last_unstale":"2024-03-06T07:20:28.702028+0000","last_undegraded":"2024-03-06T07:20:28.702028+0000","last_fullsized":"2024-03-06T07:20:28.702028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:27.196530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702647+0000","last_change":"2024-03-06T07:20:20.857187+0000","last_active":"2024-03-06T07:20:28.702647+0000","last_peered":"2024-03-06T07:20:28.702647+0000","last_clean":"2024-03-06T07:20:28.702647+0000","last_became_active":"2024-03-06T07:20:20.856898+0000","last_became_peered":"2024-03-06T07:20:20.856898+0000","last_unstale":"2024-03-06T07:20:28.702647+0000","last_undegraded":"2024-03-06T07:20:28.702647+0000","last_fullsized":"2024-03-06T07:20:28.702647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:57.497682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879802+0000","last_change":"2024-03-06T07:20:26.503331+0000","last_active":"2024-03-06T07:20:27.879802+0000","last_peered":"2024-03-06T07:20:27.879802+0000","last_clean":"2024-03-06T07:20:27.879802+0000","last_became_active":"2024-03-06T07:20:26.502770+0000","last_became_peered":"2024-03-06T07:20:26.502770+0000","last_unstale":"2024-03-06T07:20:27.879802+0000","last_undegraded":"2024-03-06T07:20:27.879802+0000","last_fullsized":"2024-03-06T07:20:27.879802+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:13.262016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701297+0000","last_change":"2024-03-06T07:20:26.502227+0000","last_active":"2024-03-06T07:20:28.701297+0000","last_peered":"2024-03-06T07:20:28.701297+0000","last_clean":"2024-03-06T07:20:28.701297+0000","last_became_active":"2024-03-06T07:20:26.501641+0000","last_became_peered":"2024-03-06T07:20:26.501641+0000","last_unstale":"2024-03-06T07:20:28.701297+0000","last_undegraded":"2024-03-06T07:20:28.701297+0000","last_fullsized":"2024-03-06T07:20:28.701297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:31.908538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365327+0000","last_change":"2024-03-06T07:20:21.866845+0000","last_active":"2024-03-06T07:20:28.365327+0000","last_peered":"2024-03-06T07:20:28.365327+0000","last_clean":"2024-03-06T07:20:28.365327+0000","last_became_active":"2024-03-06T07:20:21.866576+0000","last_became_peered":"2024-03-06T07:20:21.866576+0000","last_unstale":"2024-03-06T07:20:28.365327+0000","last_undegraded":"2024-03-06T07:20:28.365327+0000","last_fullsized":"2024-03-06T07:20:28.365327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:36.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701723+0000","last_change":"2024-03-06T07:20:20.853894+0000","last_active":"2024-03-06T07:20:28.701723+0000","last_peered":"2024-03-06T07:20:28.701723+0000","last_clean":"2024-03-06T07:20:28.701723+0000","last_became_active":"2024-03-06T07:20:20.853538+0000","last_became_peered":"2024-03-06T07:20:20.853538+0000","last_unstale":"2024-03-06T07:20:28.701723+0000","last_undegraded":"2024-03-06T07:20:28.701723+0000","last_fullsized":"2024-03-06T07:20:28.701723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:25.763288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702543+0000","last_change":"2024-03-06T07:20:25.894539+0000","last_active":"2024-03-06T07:20:28.702543+0000","last_peered":"2024-03-06T07:20:28.702543+0000","last_clean":"2024-03-06T07:20:28.702543+0000","last_became_active":"2024-03-06T07:20:25.894137+0000","last_became_peered":"2024-03-06T07:20:25.894137+0000","last_unstale":"2024-03-06T07:20:28.702543+0000","last_undegraded":"2024-03-06T07:20:28.702543+0000","last_fullsized":"2024-03-06T07:20:28.702543+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:57.736429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879760+0000","last_change":"2024-03-06T07:20:21.869266+0000","last_active":"2024-03-06T07:20:27.879760+0000","last_peered":"2024-03-06T07:20:27.879760+0000","last_clean":"2024-03-06T07:20:27.879760+0000","last_became_active":"2024-03-06T07:20:21.861979+0000","last_became_peered":"2024-03-06T07:20:21.861979+0000","last_unstale":"2024-03-06T07:20:27.879760+0000","last_undegraded":"2024-03-06T07:20:27.879760+0000","last_fullsized":"2024-03-06T07:20:27.879760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:17.903391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879911+0000","last_change":"2024-03-06T07:20:20.853365+0000","last_active":"2024-03-06T07:20:27.879911+0000","last_peered":"2024-03-06T07:20:27.879911+0000","last_clean":"2024-03-06T07:20:27.879911+0000","last_became_active":"2024-03-06T07:20:20.853065+0000","last_became_peered":"2024-03-06T07:20:20.853065+0000","last_unstale":"2024-03-06T07:20:27.879911+0000","last_undegraded":"2024-03-06T07:20:27.879911+0000","last_fullsized":"2024-03-06T07:20:27.879911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:53.997041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702584+0000","last_change":"2024-03-06T07:20:20.852681+0000","last_active":"2024-03-06T07:20:28.702584+0000","last_peered":"2024-03-06T07:20:28.702584+0000","last_clean":"2024-03-06T07:20:28.702584+0000","last_became_active":"2024-03-06T07:20:20.852406+0000","last_became_peered":"2024-03-06T07:20:20.852406+0000","last_unstale":"2024-03-06T07:20:28.702584+0000","last_undegraded":"2024-03-06T07:20:28.702584+0000","last_fullsized":"2024-03-06T07:20:28.702584+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:25.763288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701475+0000","last_change":"2024-03-06T07:20:21.854101+0000","last_active":"2024-03-06T07:20:28.701475+0000","last_peered":"2024-03-06T07:20:28.701475+0000","last_clean":"2024-03-06T07:20:28.701475+0000","last_became_active":"2024-03-06T07:20:21.853720+0000","last_became_peered":"2024-03-06T07:20:21.853720+0000","last_unstale":"2024-03-06T07:20:28.701475+0000","last_undegraded":"2024-03-06T07:20:28.701475+0000","last_fullsized":"2024-03-06T07:20:28.701475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.423089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879159+0000","last_change":"2024-03-06T07:20:25.889455+0000","last_active":"2024-03-06T07:20:27.879159+0000","last_peered":"2024-03-06T07:20:27.879159+0000","last_clean":"2024-03-06T07:20:27.879159+0000","last_became_active":"2024-03-06T07:20:25.889021+0000","last_became_peered":"2024-03-06T07:20:25.889021+0000","last_unstale":"2024-03-06T07:20:27.879159+0000","last_undegraded":"2024-03-06T07:20:27.879159+0000","last_fullsized":"2024-03-06T07:20:27.879159+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:46.647208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880275+0000","last_change":"2024-03-06T07:20:20.860219+0000","last_active":"2024-03-06T07:20:27.880275+0000","last_peered":"2024-03-06T07:20:27.880275+0000","last_clean":"2024-03-06T07:20:27.880275+0000","last_became_active":"2024-03-06T07:20:20.859862+0000","last_became_peered":"2024-03-06T07:20:20.859862+0000","last_unstale":"2024-03-06T07:20:27.880275+0000","last_undegraded":"2024-03-06T07:20:27.880275+0000","last_fullsized":"2024-03-06T07:20:27.880275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:44.080359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879980+0000","last_change":"2024-03-06T07:20:21.858660+0000","last_active":"2024-03-06T07:20:27.879980+0000","last_peered":"2024-03-06T07:20:27.879980+0000","last_clean":"2024-03-06T07:20:27.879980+0000","last_became_active":"2024-03-06T07:20:21.858397+0000","last_became_peered":"2024-03-06T07:20:21.858397+0000","last_unstale":"2024-03-06T07:20:27.879980+0000","last_undegraded":"2024-03-06T07:20:27.879980+0000","last_fullsized":"2024-03-06T07:20:27.879980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:16.721519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880388+0000","last_change":"2024-03-06T07:20:25.889183+0000","last_active":"2024-03-06T07:20:27.880388+0000","last_peered":"2024-03-06T07:20:27.880388+0000","last_clean":"2024-03-06T07:20:27.880388+0000","last_became_active":"2024-03-06T07:20:25.888836+0000","last_became_peered":"2024-03-06T07:20:25.888836+0000","last_unstale":"2024-03-06T07:20:27.880388+0000","last_undegraded":"2024-03-06T07:20:27.880388+0000","last_fullsized":"2024-03-06T07:20:27.880388+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:28.743175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701231+0000","last_change":"2024-03-06T07:20:21.863258+0000","last_active":"2024-03-06T07:20:28.701231+0000","last_peered":"2024-03-06T07:20:28.701231+0000","last_clean":"2024-03-06T07:20:28.701231+0000","last_became_active":"2024-03-06T07:20:21.862976+0000","last_became_peered":"2024-03-06T07:20:21.862976+0000","last_unstale":"2024-03-06T07:20:28.701231+0000","last_undegraded":"2024-03-06T07:20:28.701231+0000","last_fullsized":"2024-03-06T07:20:28.701231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.878887+0000","last_change":"2024-03-06T07:20:21.860263+0000","last_active":"2024-03-06T07:20:27.878887+0000","last_peered":"2024-03-06T07:20:27.878887+0000","last_clean":"2024-03-06T07:20:27.878887+0000","last_became_active":"2024-03-06T07:20:21.859978+0000","last_became_peered":"2024-03-06T07:20:21.859978+0000","last_unstale":"2024-03-06T07:20:27.878887+0000","last_undegraded":"2024-03-06T07:20:27.878887+0000","last_fullsized":"2024-03-06T07:20:27.878887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:59.701834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701885+0000","last_change":"2024-03-06T07:20:21.868562+0000","last_active":"2024-03-06T07:20:28.701885+0000","last_peered":"2024-03-06T07:20:28.701885+0000","last_clean":"2024-03-06T07:20:28.701885+0000","last_became_active":"2024-03-06T07:20:21.868239+0000","last_became_peered":"2024-03-06T07:20:21.868239+0000","last_unstale":"2024-03-06T07:20:28.701885+0000","last_undegraded":"2024-03-06T07:20:28.701885+0000","last_fullsized":"2024-03-06T07:20:28.701885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:45.002191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701377+0000","last_change":"2024-03-06T07:20:21.860455+0000","last_active":"2024-03-06T07:20:28.701377+0000","last_peered":"2024-03-06T07:20:28.701377+0000","last_clean":"2024-03-06T07:20:28.701377+0000","last_became_active":"2024-03-06T07:20:21.859855+0000","last_became_peered":"2024-03-06T07:20:21.859855+0000","last_unstale":"2024-03-06T07:20:28.701377+0000","last_undegraded":"2024-03-06T07:20:28.701377+0000","last_fullsized":"2024-03-06T07:20:28.701377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879935+0000","last_change":"2024-03-06T07:20:21.856836+0000","last_active":"2024-03-06T07:20:27.879935+0000","last_peered":"2024-03-06T07:20:27.879935+0000","last_clean":"2024-03-06T07:20:27.879935+0000","last_became_active":"2024-03-06T07:20:21.856421+0000","last_became_peered":"2024-03-06T07:20:21.856421+0000","last_unstale":"2024-03-06T07:20:27.879935+0000","last_undegraded":"2024-03-06T07:20:27.879935+0000","last_fullsized":"2024-03-06T07:20:27.879935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:00.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365852+0000","last_change":"2024-03-06T07:20:21.866911+0000","last_active":"2024-03-06T07:20:28.365852+0000","last_peered":"2024-03-06T07:20:28.365852+0000","last_clean":"2024-03-06T07:20:28.365852+0000","last_became_active":"2024-03-06T07:20:21.866557+0000","last_became_peered":"2024-03-06T07:20:21.866557+0000","last_unstale":"2024-03-06T07:20:28.365852+0000","last_undegraded":"2024-03-06T07:20:28.365852+0000","last_fullsized":"2024-03-06T07:20:28.365852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702389+0000","last_change":"2024-03-06T07:20:21.866315+0000","last_active":"2024-03-06T07:20:28.702389+0000","last_peered":"2024-03-06T07:20:28.702389+0000","last_clean":"2024-03-06T07:20:28.702389+0000","last_became_active":"2024-03-06T07:20:21.866035+0000","last_became_peered":"2024-03-06T07:20:21.866035+0000","last_unstale":"2024-03-06T07:20:28.702389+0000","last_undegraded":"2024-03-06T07:20:28.702389+0000","last_fullsized":"2024-03-06T07:20:28.702389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:20.911597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880753+0000","last_change":"2024-03-06T07:20:21.866364+0000","last_active":"2024-03-06T07:20:27.880753+0000","last_peered":"2024-03-06T07:20:27.880753+0000","last_clean":"2024-03-06T07:20:27.880753+0000","last_became_active":"2024-03-06T07:20:21.866091+0000","last_became_peered":"2024-03-06T07:20:21.866091+0000","last_unstale":"2024-03-06T07:20:27.880753+0000","last_undegraded":"2024-03-06T07:20:27.880753+0000","last_fullsized":"2024-03-06T07:20:27.880753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:01.604000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701743+0000","last_change":"2024-03-06T07:20:21.868488+0000","last_active":"2024-03-06T07:20:28.701743+0000","last_peered":"2024-03-06T07:20:28.701743+0000","last_clean":"2024-03-06T07:20:28.701743+0000","last_became_active":"2024-03-06T07:20:21.868091+0000","last_became_peered":"2024-03-06T07:20:21.868091+0000","last_unstale":"2024-03-06T07:20:28.701743+0000","last_undegraded":"2024-03-06T07:20:28.701743+0000","last_fullsized":"2024-03-06T07:20:28.701743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:00.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880000+0000","last_change":"2024-03-06T07:20:21.853440+0000","last_active":"2024-03-06T07:20:27.880000+0000","last_peered":"2024-03-06T07:20:27.880000+0000","last_clean":"2024-03-06T07:20:27.880000+0000","last_became_active":"2024-03-06T07:20:21.853089+0000","last_became_peered":"2024-03-06T07:20:21.853089+0000","last_unstale":"2024-03-06T07:20:27.880000+0000","last_undegraded":"2024-03-06T07:20:27.880000+0000","last_fullsized":"2024-03-06T07:20:27.880000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:20.911597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701633+0000","last_change":"2024-03-06T07:20:21.865493+0000","last_active":"2024-03-06T07:20:28.701633+0000","last_peered":"2024-03-06T07:20:28.701633+0000","last_clean":"2024-03-06T07:20:28.701633+0000","last_became_active":"2024-03-06T07:20:21.865203+0000","last_became_peered":"2024-03-06T07:20:21.865203+0000","last_unstale":"2024-03-06T07:20:28.701633+0000","last_undegraded":"2024-03-06T07:20:28.701633+0000","last_fullsized":"2024-03-06T07:20:28.701633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:20.882851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701280+0000","last_change":"2024-03-06T07:20:21.847412+0000","last_active":"2024-03-06T07:20:28.701280+0000","last_peered":"2024-03-06T07:20:28.701280+0000","last_clean":"2024-03-06T07:20:28.701280+0000","last_became_active":"2024-03-06T07:20:21.847081+0000","last_became_peered":"2024-03-06T07:20:21.847081+0000","last_unstale":"2024-03-06T07:20:28.701280+0000","last_undegraded":"2024-03-06T07:20:28.701280+0000","last_fullsized":"2024-03-06T07:20:28.701280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:28.363605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702051+0000","last_change":"2024-03-06T07:20:21.861878+0000","last_active":"2024-03-06T07:20:28.702051+0000","last_peered":"2024-03-06T07:20:28.702051+0000","last_clean":"2024-03-06T07:20:28.702051+0000","last_became_active":"2024-03-06T07:20:21.859115+0000","last_became_peered":"2024-03-06T07:20:21.859115+0000","last_unstale":"2024-03-06T07:20:28.702051+0000","last_undegraded":"2024-03-06T07:20:28.702051+0000","last_fullsized":"2024-03-06T07:20:28.702051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:42.612183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201453+0000","last_change":"2024-03-06T07:20:21.860975+0000","last_active":"2024-03-06T07:20:28.201453+0000","last_peered":"2024-03-06T07:20:28.201453+0000","last_clean":"2024-03-06T07:20:28.201453+0000","last_became_active":"2024-03-06T07:20:21.860414+0000","last_became_peered":"2024-03-06T07:20:21.860414+0000","last_unstale":"2024-03-06T07:20:28.201453+0000","last_undegraded":"2024-03-06T07:20:28.201453+0000","last_fullsized":"2024-03-06T07:20:28.201453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:53.701404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879575+0000","last_change":"2024-03-06T07:20:21.867097+0000","last_active":"2024-03-06T07:20:27.879575+0000","last_peered":"2024-03-06T07:20:27.879575+0000","last_clean":"2024-03-06T07:20:27.879575+0000","last_became_active":"2024-03-06T07:20:21.866821+0000","last_became_peered":"2024-03-06T07:20:21.866821+0000","last_unstale":"2024-03-06T07:20:27.879575+0000","last_undegraded":"2024-03-06T07:20:27.879575+0000","last_fullsized":"2024-03-06T07:20:27.879575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:25.262951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702397+0000","last_change":"2024-03-06T07:20:21.866152+0000","last_active":"2024-03-06T07:20:28.702397+0000","last_peered":"2024-03-06T07:20:28.702397+0000","last_clean":"2024-03-06T07:20:28.702397+0000","last_became_active":"2024-03-06T07:20:21.865919+0000","last_became_peered":"2024-03-06T07:20:21.865919+0000","last_unstale":"2024-03-06T07:20:28.702397+0000","last_undegraded":"2024-03-06T07:20:28.702397+0000","last_fullsized":"2024-03-06T07:20:28.702397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:26.855742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701560+0000","last_change":"2024-03-06T07:20:21.860761+0000","last_active":"2024-03-06T07:20:28.701560+0000","last_peered":"2024-03-06T07:20:28.701560+0000","last_clean":"2024-03-06T07:20:28.701560+0000","last_became_active":"2024-03-06T07:20:21.860444+0000","last_became_peered":"2024-03-06T07:20:21.860444+0000","last_unstale":"2024-03-06T07:20:28.701560+0000","last_undegraded":"2024-03-06T07:20:28.701560+0000","last_fullsized":"2024-03-06T07:20:28.701560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365409+0000","last_change":"2024-03-06T07:20:21.855033+0000","last_active":"2024-03-06T07:20:28.365409+0000","last_peered":"2024-03-06T07:20:28.365409+0000","last_clean":"2024-03-06T07:20:28.365409+0000","last_became_active":"2024-03-06T07:20:21.854554+0000","last_became_peered":"2024-03-06T07:20:21.854554+0000","last_unstale":"2024-03-06T07:20:28.365409+0000","last_undegraded":"2024-03-06T07:20:28.365409+0000","last_fullsized":"2024-03-06T07:20:28.365409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:20.911597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701196+0000","last_change":"2024-03-06T07:20:21.856900+0000","last_active":"2024-03-06T07:20:28.701196+0000","last_peered":"2024-03-06T07:20:28.701196+0000","last_clean":"2024-03-06T07:20:28.701196+0000","last_became_active":"2024-03-06T07:20:21.856628+0000","last_became_peered":"2024-03-06T07:20:21.856628+0000","last_unstale":"2024-03-06T07:20:28.701196+0000","last_undegraded":"2024-03-06T07:20:28.701196+0000","last_fullsized":"2024-03-06T07:20:28.701196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:47.276655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880101+0000","last_change":"2024-03-06T07:20:21.868803+0000","last_active":"2024-03-06T07:20:27.880101+0000","last_peered":"2024-03-06T07:20:27.880101+0000","last_clean":"2024-03-06T07:20:27.880101+0000","last_became_active":"2024-03-06T07:20:21.868518+0000","last_became_peered":"2024-03-06T07:20:21.868518+0000","last_unstale":"2024-03-06T07:20:27.880101+0000","last_undegraded":"2024-03-06T07:20:27.880101+0000","last_fullsized":"2024-03-06T07:20:27.880101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:03.883136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880416+0000","last_change":"2024-03-06T07:20:21.863674+0000","last_active":"2024-03-06T07:20:27.880416+0000","last_peered":"2024-03-06T07:20:27.880416+0000","last_clean":"2024-03-06T07:20:27.880416+0000","last_became_active":"2024-03-06T07:20:21.863138+0000","last_became_peered":"2024-03-06T07:20:21.863138+0000","last_unstale":"2024-03-06T07:20:27.880416+0000","last_undegraded":"2024-03-06T07:20:27.880416+0000","last_fullsized":"2024-03-06T07:20:27.880416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:26.767916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702522+0000","last_change":"2024-03-06T07:20:21.858343+0000","last_active":"2024-03-06T07:20:28.702522+0000","last_peered":"2024-03-06T07:20:28.702522+0000","last_clean":"2024-03-06T07:20:28.702522+0000","last_became_active":"2024-03-06T07:20:21.858078+0000","last_became_peered":"2024-03-06T07:20:21.858078+0000","last_unstale":"2024-03-06T07:20:28.702522+0000","last_undegraded":"2024-03-06T07:20:28.702522+0000","last_fullsized":"2024-03-06T07:20:28.702522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.423089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701847+0000","last_change":"2024-03-06T07:20:21.866157+0000","last_active":"2024-03-06T07:20:28.701847+0000","last_peered":"2024-03-06T07:20:28.701847+0000","last_clean":"2024-03-06T07:20:28.701847+0000","last_became_active":"2024-03-06T07:20:21.865306+0000","last_became_peered":"2024-03-06T07:20:21.865306+0000","last_unstale":"2024-03-06T07:20:28.701847+0000","last_undegraded":"2024-03-06T07:20:28.701847+0000","last_fullsized":"2024-03-06T07:20:28.701847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:36.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201624+0000","last_change":"2024-03-06T07:20:21.855346+0000","last_active":"2024-03-06T07:20:28.201624+0000","last_peered":"2024-03-06T07:20:28.201624+0000","last_clean":"2024-03-06T07:20:28.201624+0000","last_became_active":"2024-03-06T07:20:21.855059+0000","last_became_peered":"2024-03-06T07:20:21.855059+0000","last_unstale":"2024-03-06T07:20:28.201624+0000","last_undegraded":"2024-03-06T07:20:28.201624+0000","last_fullsized":"2024-03-06T07:20:28.201624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:51.111188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879524+0000","last_change":"2024-03-06T07:20:21.868545+0000","last_active":"2024-03-06T07:20:27.879524+0000","last_peered":"2024-03-06T07:20:27.879524+0000","last_clean":"2024-03-06T07:20:27.879524+0000","last_became_active":"2024-03-06T07:20:21.868188+0000","last_became_peered":"2024-03-06T07:20:21.868188+0000","last_unstale":"2024-03-06T07:20:27.879524+0000","last_undegraded":"2024-03-06T07:20:27.879524+0000","last_fullsized":"2024-03-06T07:20:27.879524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:28.363605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201003+0000","last_change":"2024-03-06T07:20:21.856718+0000","last_active":"2024-03-06T07:20:28.201003+0000","last_peered":"2024-03-06T07:20:28.201003+0000","last_clean":"2024-03-06T07:20:28.201003+0000","last_became_active":"2024-03-06T07:20:21.856451+0000","last_became_peered":"2024-03-06T07:20:21.856451+0000","last_unstale":"2024-03-06T07:20:28.201003+0000","last_undegraded":"2024-03-06T07:20:28.201003+0000","last_fullsized":"2024-03-06T07:20:28.201003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:01.604000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701625+0000","last_change":"2024-03-06T07:20:21.861834+0000","last_active":"2024-03-06T07:20:28.701625+0000","last_peered":"2024-03-06T07:20:28.701625+0000","last_clean":"2024-03-06T07:20:28.701625+0000","last_became_active":"2024-03-06T07:20:21.860998+0000","last_became_peered":"2024-03-06T07:20:21.860998+0000","last_unstale":"2024-03-06T07:20:28.701625+0000","last_undegraded":"2024-03-06T07:20:28.701625+0000","last_fullsized":"2024-03-06T07:20:28.701625+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:01.604000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.200939+0000","last_change":"2024-03-06T07:20:21.860505+0000","last_active":"2024-03-06T07:20:28.200939+0000","last_peered":"2024-03-06T07:20:28.200939+0000","last_clean":"2024-03-06T07:20:28.200939+0000","last_became_active":"2024-03-06T07:20:21.860241+0000","last_became_peered":"2024-03-06T07:20:21.860241+0000","last_unstale":"2024-03-06T07:20:28.200939+0000","last_undegraded":"2024-03-06T07:20:28.200939+0000","last_fullsized":"2024-03-06T07:20:28.200939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:20.882851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701769+0000","last_change":"2024-03-06T07:20:21.868402+0000","last_active":"2024-03-06T07:20:28.701769+0000","last_peered":"2024-03-06T07:20:28.701769+0000","last_clean":"2024-03-06T07:20:28.701769+0000","last_became_active":"2024-03-06T07:20:21.868068+0000","last_became_peered":"2024-03-06T07:20:21.868068+0000","last_unstale":"2024-03-06T07:20:28.701769+0000","last_undegraded":"2024-03-06T07:20:28.701769+0000","last_fullsized":"2024-03-06T07:20:28.701769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:03.883136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701196+0000","last_change":"2024-03-06T07:20:21.860664+0000","last_active":"2024-03-06T07:20:28.701196+0000","last_peered":"2024-03-06T07:20:28.701196+0000","last_clean":"2024-03-06T07:20:28.701196+0000","last_became_active":"2024-03-06T07:20:21.860393+0000","last_became_peered":"2024-03-06T07:20:21.860393+0000","last_unstale":"2024-03-06T07:20:28.701196+0000","last_undegraded":"2024-03-06T07:20:28.701196+0000","last_fullsized":"2024-03-06T07:20:28.701196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:36.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366176+0000","last_change":"2024-03-06T07:20:21.860087+0000","last_active":"2024-03-06T07:20:28.366176+0000","last_peered":"2024-03-06T07:20:28.366176+0000","last_clean":"2024-03-06T07:20:28.366176+0000","last_became_active":"2024-03-06T07:20:21.859804+0000","last_became_peered":"2024-03-06T07:20:21.859804+0000","last_unstale":"2024-03-06T07:20:28.366176+0000","last_undegraded":"2024-03-06T07:20:28.366176+0000","last_fullsized":"2024-03-06T07:20:28.366176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:00.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201546+0000","last_change":"2024-03-06T07:20:21.859581+0000","last_active":"2024-03-06T07:20:28.201546+0000","last_peered":"2024-03-06T07:20:28.201546+0000","last_clean":"2024-03-06T07:20:28.201546+0000","last_became_active":"2024-03-06T07:20:21.859305+0000","last_became_peered":"2024-03-06T07:20:21.859305+0000","last_unstale":"2024-03-06T07:20:28.201546+0000","last_undegraded":"2024-03-06T07:20:28.201546+0000","last_fullsized":"2024-03-06T07:20:28.201546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:47.201278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4499,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":698343,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":111094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":344064,"data_stored":698399,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:20:38.950 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:20:39.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.084+0000 7f4577c88640 1 Processor -- start 2024-03-06T07:20:39.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.084+0000 7f4577c88640 1 -- start start 2024-03-06T07:20:39.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.086+0000 7f4577c88640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f45700c4ad0 0x7f4570156930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.086+0000 7f4577c88640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4570157580 con 0x7f4570150760 2024-03-06T07:20:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.086+0000 7f4577c88640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45700c78e0 con 0x7f457006f3d0 2024-03-06T07:20:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.086+0000 7f4577c88640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45700c7b80 con 0x7f45700c4ad0 2024-03-06T07:20:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.086+0000 7f4575a24640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4570150760 0x7f45700c52a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57392/0 (socket says 172.21.15.1:57392) 2024-03-06T07:20:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.086+0000 7f4575a24640 1 -- 172.21.15.1:0/2217136545 learned_addr learned my addr 172.21.15.1:0/2217136545 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.086+0000 7f4575223640 1 --2- 172.21.15.1:0/2217136545 >> v2:172.21.15.98:3300/0 conn(0x7f45700c4ad0 0x7f4570156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.086+0000 7f4574a22640 1 -- 172.21.15.1:0/2217136545 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540057732 0 0) 0x7f4570157580 con 0x7f4570150760 2024-03-06T07:20:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.086+0000 7f4574a22640 1 -- 172.21.15.1:0/2217136545 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f455c003520 con 0x7f4570150760 2024-03-06T07:20:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.086+0000 7f4575223640 1 -- 172.21.15.1:0/2217136545 >> v1:172.21.15.98:6789/0 conn(0x7f457006f3d0 legacy=0x7f4570156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.086+0000 7f4575223640 1 -- 172.21.15.1:0/2217136545 >> v1:172.21.15.1:6789/0 conn(0x7f4570150760 legacy=0x7f45700c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.086+0000 7f4575223640 1 -- 172.21.15.1:0/2217136545 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45700c7e60 con 0x7f45700c4ad0 2024-03-06T07:20:39.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.087+0000 7f4575223640 1 --2- 172.21.15.1:0/2217136545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45700c4ad0 0x7f4570156930 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f45600097a0 tx=0x7f456002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc65dd7a95cac4ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.087+0000 7f4574a22640 1 -- 172.21.15.1:0/2217136545 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4560035020 con 0x7f45700c4ad0 2024-03-06T07:20:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.087+0000 7f4574a22640 1 -- 172.21.15.1:0/2217136545 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4560031830 con 0x7f45700c4ad0 2024-03-06T07:20:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.087+0000 7f4574a22640 1 -- 172.21.15.1:0/2217136545 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4560031be0 con 0x7f45700c4ad0 2024-03-06T07:20:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.088+0000 7f4577c88640 1 -- 172.21.15.1:0/2217136545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45700c4ad0 msgr2=0x7f4570156930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.088+0000 7f4577c88640 1 --2- 172.21.15.1:0/2217136545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45700c4ad0 0x7f4570156930 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f45600097a0 tx=0x7f456002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.088+0000 7f4577c88640 1 -- 172.21.15.1:0/2217136545 shutdown_connections 2024-03-06T07:20:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.088+0000 7f4577c88640 1 --2- 172.21.15.1:0/2217136545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45700c4ad0 0x7f4570156930 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.088+0000 7f4577c88640 1 -- 172.21.15.1:0/2217136545 >> 172.21.15.1:0/2217136545 conn(0x7f45700c8810 msgr2=0x7f45700c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.088+0000 7f4577c88640 1 -- 172.21.15.1:0/2217136545 shutdown_connections 2024-03-06T07:20:39.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.088+0000 7f4577c88640 1 -- 172.21.15.1:0/2217136545 wait complete. 2024-03-06T07:20:39.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.090+0000 7f4577c88640 1 Processor -- start 2024-03-06T07:20:39.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.090+0000 7f4577c88640 1 -- start start 2024-03-06T07:20:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f4577c88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f457006f3d0 0x7f457014b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f4577c88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45700c4ad0 0x7f457014f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f4577c88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4570150760 0x7f45700ac6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f4577c88640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45700c59b0 con 0x7f4570150760 2024-03-06T07:20:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f4577c88640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45700c8420 con 0x7f45700c4ad0 2024-03-06T07:20:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f4577c88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45700c5b30 con 0x7f457006f3d0 2024-03-06T07:20:39.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f4576225640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4570150760 0x7f45700ac6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f4576225640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4570150760 0x7f45700ac6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40306/0 (socket says 172.21.15.1:40306) 2024-03-06T07:20:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f4576225640 1 -- 172.21.15.1:0/3987089607 learned_addr learned my addr 172.21.15.1:0/3987089607 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f4575223640 1 --2- 172.21.15.1:0/3987089607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45700c4ad0 0x7f457014f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f4575a24640 1 --2- 172.21.15.1:0/3987089607 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f457006f3d0 0x7f457014b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f4576225640 1 -- 172.21.15.1:0/3987089607 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f457006f3d0 msgr2=0x7f457014b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f4576225640 1 --2- 172.21.15.1:0/3987089607 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f457006f3d0 0x7f457014b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f4576225640 1 -- 172.21.15.1:0/3987089607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45700c4ad0 msgr2=0x7f457014f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f4576225640 1 --2- 172.21.15.1:0/3987089607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45700c4ad0 0x7f457014f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f4576225640 1 -- 172.21.15.1:0/3987089607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4570150ed0 con 0x7f4570150760 2024-03-06T07:20:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f4576225640 1 --2- 172.21.15.1:0/3987089607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4570150760 0x7f45700ac6f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f456800c7a0 tx=0x7f456800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.091+0000 7f45667fc640 1 -- 172.21.15.1:0/3987089607 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568003b70 con 0x7f4570150760 2024-03-06T07:20:39.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.092+0000 7f45667fc640 1 -- 172.21.15.1:0/3987089607 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4568003d30 con 0x7f4570150760 2024-03-06T07:20:39.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.092+0000 7f4577c88640 1 -- 172.21.15.1:0/3987089607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45700ad940 con 0x7f4570150760 2024-03-06T07:20:39.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.092+0000 7f45667fc640 1 -- 172.21.15.1:0/3987089607 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456800aa90 con 0x7f4570150760 2024-03-06T07:20:39.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.092+0000 7f4577c88640 1 -- 172.21.15.1:0/3987089607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4570151190 con 0x7f4570150760 2024-03-06T07:20:39.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.092+0000 7f4577c88640 1 -- 172.21.15.1:0/3987089607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45700c3e40 con 0x7f4570150760 2024-03-06T07:20:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.093+0000 7f45667fc640 1 -- 172.21.15.1:0/3987089607 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f456800ac30 con 0x7f4570150760 2024-03-06T07:20:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.093+0000 7f45667fc640 1 --2- 172.21.15.1:0/3987089607 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f455406b4d0 0x7f455406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.094+0000 7f45667fc640 1 -- 172.21.15.1:0/3987089607 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f456808a490 con 0x7f4570150760 2024-03-06T07:20:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.094+0000 7f4575a24640 1 --2- 172.21.15.1:0/3987089607 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f455406b4d0 0x7f455406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.094+0000 7f4575a24640 1 --2- 172.21.15.1:0/3987089607 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f455406b4d0 0x7f455406d990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f456c0011e0 tx=0x7f456c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.098+0000 7f45667fc640 1 -- 172.21.15.1:0/3987089607 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45700c3e40 con 0x7f4570150760 2024-03-06T07:20:39.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.275+0000 7f4577c88640 1 -- 172.21.15.1:0/3987089607 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f45700686e0 con 0x7f455406b4d0 2024-03-06T07:20:39.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.283+0000 7f45667fc640 1 -- 172.21.15.1:0/3987089607 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332611 (secure 0 0 0) 0x7f45700686e0 con 0x7f455406b4d0 2024-03-06T07:20:39.289 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:39.290 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:20:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.287+0000 7f4577c88640 1 -- 172.21.15.1:0/3987089607 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f455406b4d0 msgr2=0x7f455406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.287+0000 7f4577c88640 1 --2- 172.21.15.1:0/3987089607 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f455406b4d0 0x7f455406d990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f456c0011e0 tx=0x7f456c009040 comp rx=0 tx=0).stop 2024-03-06T07:20:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.287+0000 7f4577c88640 1 -- 172.21.15.1:0/3987089607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4570150760 msgr2=0x7f45700ac6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.287+0000 7f4577c88640 1 --2- 172.21.15.1:0/3987089607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4570150760 0x7f45700ac6f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f456800c7a0 tx=0x7f456800cc60 comp rx=0 tx=0).stop 2024-03-06T07:20:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.287+0000 7f4577c88640 1 -- 172.21.15.1:0/3987089607 shutdown_connections 2024-03-06T07:20:39.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.287+0000 7f4577c88640 1 --2- 172.21.15.1:0/3987089607 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f455406b4d0 0x7f455406d990 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.287+0000 7f4577c88640 1 --2- 172.21.15.1:0/3987089607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4570150760 0x7f45700ac6f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.287+0000 7f4577c88640 1 --2- 172.21.15.1:0/3987089607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45700c4ad0 0x7f457014f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.287+0000 7f4577c88640 1 --2- 172.21.15.1:0/3987089607 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f457006f3d0 0x7f457014b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.287+0000 7f4577c88640 1 -- 172.21.15.1:0/3987089607 >> 172.21.15.1:0/3987089607 conn(0x7f45700c8810 msgr2=0x7f45700c6400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:39.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.287+0000 7f4577c88640 1 -- 172.21.15.1:0/3987089607 shutdown_connections 2024-03-06T07:20:39.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.287+0000 7f4577c88640 1 -- 172.21.15.1:0/3987089607 wait complete. 2024-03-06T07:20:39.299 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-03-06T07:20:38.690346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594867,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":230924,"kb_used_data":2272,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754743796,"statfs":{"total":773094113280,"available":772857647104,"internally_reserved":0,"allocated":2326528,"data_stored":2048878,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2089,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.812155"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701618+0000","last_change":"2024-03-06T07:20:20.860733+0000","last_active":"2024-03-06T07:20:28.701618+0000","last_peered":"2024-03-06T07:20:28.701618+0000","last_clean":"2024-03-06T07:20:28.701618+0000","last_became_active":"2024-03-06T07:20:20.860412+0000","last_became_peered":"2024-03-06T07:20:20.860412+0000","last_unstale":"2024-03-06T07:20:28.701618+0000","last_undegraded":"2024-03-06T07:20:28.701618+0000","last_fullsized":"2024-03-06T07:20:28.701618+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:44.080359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880598+0000","last_change":"2024-03-06T07:20:21.869098+0000","last_active":"2024-03-06T07:20:27.880598+0000","last_peered":"2024-03-06T07:20:27.880598+0000","last_clean":"2024-03-06T07:20:27.880598+0000","last_became_active":"2024-03-06T07:20:21.868808+0000","last_became_peered":"2024-03-06T07:20:21.868808+0000","last_unstale":"2024-03-06T07:20:27.880598+0000","last_undegraded":"2024-03-06T07:20:27.880598+0000","last_fullsized":"2024-03-06T07:20:27.880598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.423089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879651+0000","last_change":"2024-03-06T07:20:26.504312+0000","last_active":"2024-03-06T07:20:27.879651+0000","last_peered":"2024-03-06T07:20:27.879651+0000","last_clean":"2024-03-06T07:20:27.879651+0000","last_became_active":"2024-03-06T07:20:26.503884+0000","last_became_peered":"2024-03-06T07:20:26.503884+0000","last_unstale":"2024-03-06T07:20:27.879651+0000","last_undegraded":"2024-03-06T07:20:27.879651+0000","last_fullsized":"2024-03-06T07:20:27.879651+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:57:56.136234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880779+0000","last_change":"2024-03-06T07:20:20.854641+0000","last_active":"2024-03-06T07:20:27.880779+0000","last_peered":"2024-03-06T07:20:27.880779+0000","last_clean":"2024-03-06T07:20:27.880779+0000","last_became_active":"2024-03-06T07:20:20.854361+0000","last_became_peered":"2024-03-06T07:20:20.854361+0000","last_unstale":"2024-03-06T07:20:27.880779+0000","last_undegraded":"2024-03-06T07:20:27.880779+0000","last_fullsized":"2024-03-06T07:20:27.880779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:57.497682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701411+0000","last_change":"2024-03-06T07:20:21.857450+0000","last_active":"2024-03-06T07:20:28.701411+0000","last_peered":"2024-03-06T07:20:28.701411+0000","last_clean":"2024-03-06T07:20:28.701411+0000","last_became_active":"2024-03-06T07:20:21.855756+0000","last_became_peered":"2024-03-06T07:20:21.855756+0000","last_unstale":"2024-03-06T07:20:28.701411+0000","last_undegraded":"2024-03-06T07:20:28.701411+0000","last_fullsized":"2024-03-06T07:20:28.701411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:17.903391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879763+0000","last_change":"2024-03-06T07:20:25.889839+0000","last_active":"2024-03-06T07:20:27.879763+0000","last_peered":"2024-03-06T07:20:27.879763+0000","last_clean":"2024-03-06T07:20:27.879763+0000","last_became_active":"2024-03-06T07:20:25.889266+0000","last_became_peered":"2024-03-06T07:20:25.889266+0000","last_unstale":"2024-03-06T07:20:27.879763+0000","last_undegraded":"2024-03-06T07:20:27.879763+0000","last_fullsized":"2024-03-06T07:20:27.879763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:51.236302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201405+0000","last_change":"2024-03-06T07:20:20.859562+0000","last_active":"2024-03-06T07:20:28.201405+0000","last_peered":"2024-03-06T07:20:28.201405+0000","last_clean":"2024-03-06T07:20:28.201405+0000","last_became_active":"2024-03-06T07:20:20.859282+0000","last_became_peered":"2024-03-06T07:20:20.859282+0000","last_unstale":"2024-03-06T07:20:28.201405+0000","last_undegraded":"2024-03-06T07:20:28.201405+0000","last_fullsized":"2024-03-06T07:20:28.201405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:52.187127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701578+0000","last_change":"2024-03-06T07:20:21.863171+0000","last_active":"2024-03-06T07:20:28.701578+0000","last_peered":"2024-03-06T07:20:28.701578+0000","last_clean":"2024-03-06T07:20:28.701578+0000","last_became_active":"2024-03-06T07:20:21.862904+0000","last_became_peered":"2024-03-06T07:20:21.862904+0000","last_unstale":"2024-03-06T07:20:28.701578+0000","last_undegraded":"2024-03-06T07:20:28.701578+0000","last_fullsized":"2024-03-06T07:20:28.701578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:07.625355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880420+0000","last_change":"2024-03-06T07:20:25.888576+0000","last_active":"2024-03-06T07:20:27.880420+0000","last_peered":"2024-03-06T07:20:27.880420+0000","last_clean":"2024-03-06T07:20:27.880420+0000","last_became_active":"2024-03-06T07:20:25.886558+0000","last_became_peered":"2024-03-06T07:20:25.886558+0000","last_unstale":"2024-03-06T07:20:27.880420+0000","last_undegraded":"2024-03-06T07:20:27.880420+0000","last_fullsized":"2024-03-06T07:20:27.880420+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:27.800537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201157+0000","last_change":"2024-03-06T07:20:20.861724+0000","last_active":"2024-03-06T07:20:28.201157+0000","last_peered":"2024-03-06T07:20:28.201157+0000","last_clean":"2024-03-06T07:20:28.201157+0000","last_became_active":"2024-03-06T07:20:20.861432+0000","last_became_peered":"2024-03-06T07:20:20.861432+0000","last_unstale":"2024-03-06T07:20:28.201157+0000","last_undegraded":"2024-03-06T07:20:28.201157+0000","last_fullsized":"2024-03-06T07:20:28.201157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:43.621233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702026+0000","last_change":"2024-03-06T07:20:21.862894+0000","last_active":"2024-03-06T07:20:28.702026+0000","last_peered":"2024-03-06T07:20:28.702026+0000","last_clean":"2024-03-06T07:20:28.702026+0000","last_became_active":"2024-03-06T07:20:21.862624+0000","last_became_peered":"2024-03-06T07:20:21.862624+0000","last_unstale":"2024-03-06T07:20:28.702026+0000","last_undegraded":"2024-03-06T07:20:28.702026+0000","last_fullsized":"2024-03-06T07:20:28.702026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:53.701404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879914+0000","last_change":"2024-03-06T07:20:25.891479+0000","last_active":"2024-03-06T07:20:27.879914+0000","last_peered":"2024-03-06T07:20:27.879914+0000","last_clean":"2024-03-06T07:20:27.879914+0000","last_became_active":"2024-03-06T07:20:25.891124+0000","last_became_peered":"2024-03-06T07:20:25.891124+0000","last_unstale":"2024-03-06T07:20:27.879914+0000","last_undegraded":"2024-03-06T07:20:27.879914+0000","last_fullsized":"2024-03-06T07:20:27.879914+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:31.908538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701544+0000","last_change":"2024-03-06T07:20:21.861447+0000","last_active":"2024-03-06T07:20:28.701544+0000","last_peered":"2024-03-06T07:20:28.701544+0000","last_clean":"2024-03-06T07:20:28.701544+0000","last_became_active":"2024-03-06T07:20:21.861169+0000","last_became_peered":"2024-03-06T07:20:21.861169+0000","last_unstale":"2024-03-06T07:20:28.701544+0000","last_undegraded":"2024-03-06T07:20:28.701544+0000","last_fullsized":"2024-03-06T07:20:28.701544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:28.363605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201185+0000","last_change":"2024-03-06T07:20:20.858871+0000","last_active":"2024-03-06T07:20:28.201185+0000","last_peered":"2024-03-06T07:20:28.201185+0000","last_clean":"2024-03-06T07:20:28.201185+0000","last_became_active":"2024-03-06T07:20:20.858481+0000","last_became_peered":"2024-03-06T07:20:20.858481+0000","last_unstale":"2024-03-06T07:20:28.201185+0000","last_undegraded":"2024-03-06T07:20:28.201185+0000","last_fullsized":"2024-03-06T07:20:28.201185+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:58.697205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880550+0000","last_change":"2024-03-06T07:20:25.890028+0000","last_active":"2024-03-06T07:20:27.880550+0000","last_peered":"2024-03-06T07:20:27.880550+0000","last_clean":"2024-03-06T07:20:27.880550+0000","last_became_active":"2024-03-06T07:20:25.889634+0000","last_became_peered":"2024-03-06T07:20:25.889634+0000","last_unstale":"2024-03-06T07:20:27.880550+0000","last_undegraded":"2024-03-06T07:20:27.880550+0000","last_fullsized":"2024-03-06T07:20:27.880550+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:34.847448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880189+0000","last_change":"2024-03-06T07:20:21.863836+0000","last_active":"2024-03-06T07:20:27.880189+0000","last_peered":"2024-03-06T07:20:27.880189+0000","last_clean":"2024-03-06T07:20:27.880189+0000","last_became_active":"2024-03-06T07:20:21.863451+0000","last_became_peered":"2024-03-06T07:20:21.863451+0000","last_unstale":"2024-03-06T07:20:27.880189+0000","last_undegraded":"2024-03-06T07:20:27.880189+0000","last_fullsized":"2024-03-06T07:20:27.880189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:07.625355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365689+0000","last_change":"2024-03-06T07:20:20.847182+0000","last_active":"2024-03-06T07:20:28.365689+0000","last_peered":"2024-03-06T07:20:28.365689+0000","last_clean":"2024-03-06T07:20:28.365689+0000","last_became_active":"2024-03-06T07:20:20.846875+0000","last_became_peered":"2024-03-06T07:20:20.846875+0000","last_unstale":"2024-03-06T07:20:28.365689+0000","last_undegraded":"2024-03-06T07:20:28.365689+0000","last_fullsized":"2024-03-06T07:20:28.365689+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:47.332802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880571+0000","last_change":"2024-03-06T07:20:25.890712+0000","last_active":"2024-03-06T07:20:27.880571+0000","last_peered":"2024-03-06T07:20:27.880571+0000","last_clean":"2024-03-06T07:20:27.880571+0000","last_became_active":"2024-03-06T07:20:25.889840+0000","last_became_peered":"2024-03-06T07:20:25.889840+0000","last_unstale":"2024-03-06T07:20:27.880571+0000","last_undegraded":"2024-03-06T07:20:27.880571+0000","last_fullsized":"2024-03-06T07:20:27.880571+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:46.450423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879843+0000","last_change":"2024-03-06T07:20:21.865384+0000","last_active":"2024-03-06T07:20:27.879843+0000","last_peered":"2024-03-06T07:20:27.879843+0000","last_clean":"2024-03-06T07:20:27.879843+0000","last_became_active":"2024-03-06T07:20:21.865118+0000","last_became_peered":"2024-03-06T07:20:21.865118+0000","last_unstale":"2024-03-06T07:20:27.879843+0000","last_undegraded":"2024-03-06T07:20:27.879843+0000","last_fullsized":"2024-03-06T07:20:27.879843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:36.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365440+0000","last_change":"2024-03-06T07:20:20.856580+0000","last_active":"2024-03-06T07:20:28.365440+0000","last_peered":"2024-03-06T07:20:28.365440+0000","last_clean":"2024-03-06T07:20:28.365440+0000","last_became_active":"2024-03-06T07:20:20.856117+0000","last_became_peered":"2024-03-06T07:20:20.856117+0000","last_unstale":"2024-03-06T07:20:28.365440+0000","last_undegraded":"2024-03-06T07:20:28.365440+0000","last_fullsized":"2024-03-06T07:20:28.365440+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:43.621233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366340+0000","last_change":"2024-03-06T07:20:26.502055+0000","last_active":"2024-03-06T07:20:28.366340+0000","last_peered":"2024-03-06T07:20:28.366340+0000","last_clean":"2024-03-06T07:20:28.366340+0000","last_became_active":"2024-03-06T07:20:26.501689+0000","last_became_peered":"2024-03-06T07:20:26.501689+0000","last_unstale":"2024-03-06T07:20:28.366340+0000","last_undegraded":"2024-03-06T07:20:28.366340+0000","last_fullsized":"2024-03-06T07:20:28.366340+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:34.847448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880426+0000","last_change":"2024-03-06T07:20:21.859723+0000","last_active":"2024-03-06T07:20:27.880426+0000","last_peered":"2024-03-06T07:20:27.880426+0000","last_clean":"2024-03-06T07:20:27.880426+0000","last_became_active":"2024-03-06T07:20:21.859392+0000","last_became_peered":"2024-03-06T07:20:21.859392+0000","last_unstale":"2024-03-06T07:20:27.880426+0000","last_undegraded":"2024-03-06T07:20:27.880426+0000","last_fullsized":"2024-03-06T07:20:27.880426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:59.701834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.200895+0000","last_change":"2024-03-06T07:20:20.860602+0000","last_active":"2024-03-06T07:20:28.200895+0000","last_peered":"2024-03-06T07:20:28.200895+0000","last_clean":"2024-03-06T07:20:28.200895+0000","last_became_active":"2024-03-06T07:20:20.860328+0000","last_became_peered":"2024-03-06T07:20:20.860328+0000","last_unstale":"2024-03-06T07:20:28.200895+0000","last_undegraded":"2024-03-06T07:20:28.200895+0000","last_fullsized":"2024-03-06T07:20:28.200895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:25.763288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880136+0000","last_change":"2024-03-06T07:20:25.889087+0000","last_active":"2024-03-06T07:20:27.880136+0000","last_peered":"2024-03-06T07:20:27.880136+0000","last_clean":"2024-03-06T07:20:27.880136+0000","last_became_active":"2024-03-06T07:20:25.888562+0000","last_became_peered":"2024-03-06T07:20:25.888562+0000","last_unstale":"2024-03-06T07:20:27.880136+0000","last_undegraded":"2024-03-06T07:20:27.880136+0000","last_fullsized":"2024-03-06T07:20:27.880136+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:22.654122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701536+0000","last_change":"2024-03-06T07:20:21.857055+0000","last_active":"2024-03-06T07:20:28.701536+0000","last_peered":"2024-03-06T07:20:28.701536+0000","last_clean":"2024-03-06T07:20:28.701536+0000","last_became_active":"2024-03-06T07:20:21.856743+0000","last_became_peered":"2024-03-06T07:20:21.856743+0000","last_unstale":"2024-03-06T07:20:28.701536+0000","last_undegraded":"2024-03-06T07:20:28.701536+0000","last_fullsized":"2024-03-06T07:20:28.701536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:20.911597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702767+0000","last_change":"2024-03-06T07:20:20.858020+0000","last_active":"2024-03-06T07:20:28.702767+0000","last_peered":"2024-03-06T07:20:28.702767+0000","last_clean":"2024-03-06T07:20:28.702767+0000","last_became_active":"2024-03-06T07:20:20.857748+0000","last_became_peered":"2024-03-06T07:20:20.857748+0000","last_unstale":"2024-03-06T07:20:28.702767+0000","last_undegraded":"2024-03-06T07:20:28.702767+0000","last_fullsized":"2024-03-06T07:20:28.702767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:47.332802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880011+0000","last_change":"2024-03-06T07:20:26.499973+0000","last_active":"2024-03-06T07:20:27.880011+0000","last_peered":"2024-03-06T07:20:27.880011+0000","last_clean":"2024-03-06T07:20:27.880011+0000","last_became_active":"2024-03-06T07:20:26.499480+0000","last_became_peered":"2024-03-06T07:20:26.499480+0000","last_unstale":"2024-03-06T07:20:27.880011+0000","last_undegraded":"2024-03-06T07:20:27.880011+0000","last_fullsized":"2024-03-06T07:20:27.880011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:55.146213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880824+0000","last_change":"2024-03-06T07:20:21.858239+0000","last_active":"2024-03-06T07:20:27.880824+0000","last_peered":"2024-03-06T07:20:27.880824+0000","last_clean":"2024-03-06T07:20:27.880824+0000","last_became_active":"2024-03-06T07:20:21.857896+0000","last_became_peered":"2024-03-06T07:20:21.857896+0000","last_unstale":"2024-03-06T07:20:27.880824+0000","last_undegraded":"2024-03-06T07:20:27.880824+0000","last_fullsized":"2024-03-06T07:20:27.880824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366190+0000","last_change":"2024-03-06T07:20:20.853233+0000","last_active":"2024-03-06T07:20:28.366190+0000","last_peered":"2024-03-06T07:20:28.366190+0000","last_clean":"2024-03-06T07:20:28.366190+0000","last_became_active":"2024-03-06T07:20:20.852941+0000","last_became_peered":"2024-03-06T07:20:20.852941+0000","last_unstale":"2024-03-06T07:20:28.366190+0000","last_undegraded":"2024-03-06T07:20:28.366190+0000","last_fullsized":"2024-03-06T07:20:28.366190+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:25.763288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879421+0000","last_change":"2024-03-06T07:20:26.499314+0000","last_active":"2024-03-06T07:20:27.879421+0000","last_peered":"2024-03-06T07:20:27.879421+0000","last_clean":"2024-03-06T07:20:27.879421+0000","last_became_active":"2024-03-06T07:20:26.498625+0000","last_became_peered":"2024-03-06T07:20:26.498625+0000","last_unstale":"2024-03-06T07:20:27.879421+0000","last_undegraded":"2024-03-06T07:20:27.879421+0000","last_fullsized":"2024-03-06T07:20:27.879421+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:05.727952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365917+0000","last_change":"2024-03-06T07:20:21.861526+0000","last_active":"2024-03-06T07:20:28.365917+0000","last_peered":"2024-03-06T07:20:28.365917+0000","last_clean":"2024-03-06T07:20:28.365917+0000","last_became_active":"2024-03-06T07:20:21.861256+0000","last_became_peered":"2024-03-06T07:20:21.861256+0000","last_unstale":"2024-03-06T07:20:28.365917+0000","last_undegraded":"2024-03-06T07:20:28.365917+0000","last_fullsized":"2024-03-06T07:20:28.365917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.423089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201484+0000","last_change":"2024-03-06T07:20:20.858234+0000","last_active":"2024-03-06T07:20:28.201484+0000","last_peered":"2024-03-06T07:20:28.201484+0000","last_clean":"2024-03-06T07:20:28.201484+0000","last_became_active":"2024-03-06T07:20:20.857910+0000","last_became_peered":"2024-03-06T07:20:20.857910+0000","last_unstale":"2024-03-06T07:20:28.201484+0000","last_undegraded":"2024-03-06T07:20:28.201484+0000","last_fullsized":"2024-03-06T07:20:28.201484+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:04.756600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701326+0000","last_change":"2024-03-06T07:20:26.498727+0000","last_active":"2024-03-06T07:20:28.701326+0000","last_peered":"2024-03-06T07:20:28.701326+0000","last_clean":"2024-03-06T07:20:28.701326+0000","last_became_active":"2024-03-06T07:20:26.498272+0000","last_became_peered":"2024-03-06T07:20:26.498272+0000","last_unstale":"2024-03-06T07:20:28.701326+0000","last_undegraded":"2024-03-06T07:20:28.701326+0000","last_fullsized":"2024-03-06T07:20:28.701326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:28.909072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702156+0000","last_change":"2024-03-06T07:20:21.868324+0000","last_active":"2024-03-06T07:20:28.702156+0000","last_peered":"2024-03-06T07:20:28.702156+0000","last_clean":"2024-03-06T07:20:28.702156+0000","last_became_active":"2024-03-06T07:20:21.868032+0000","last_became_peered":"2024-03-06T07:20:21.868032+0000","last_unstale":"2024-03-06T07:20:28.702156+0000","last_undegraded":"2024-03-06T07:20:28.702156+0000","last_fullsized":"2024-03-06T07:20:28.702156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:17.299208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366009+0000","last_change":"2024-03-06T07:20:20.851073+0000","last_active":"2024-03-06T07:20:28.366009+0000","last_peered":"2024-03-06T07:20:28.366009+0000","last_clean":"2024-03-06T07:20:28.366009+0000","last_became_active":"2024-03-06T07:20:20.850774+0000","last_became_peered":"2024-03-06T07:20:20.850774+0000","last_unstale":"2024-03-06T07:20:28.366009+0000","last_undegraded":"2024-03-06T07:20:28.366009+0000","last_fullsized":"2024-03-06T07:20:28.366009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:15.716891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365536+0000","last_change":"2024-03-06T07:20:25.895118+0000","last_active":"2024-03-06T07:20:28.365536+0000","last_peered":"2024-03-06T07:20:28.365536+0000","last_clean":"2024-03-06T07:20:28.365536+0000","last_became_active":"2024-03-06T07:20:25.894354+0000","last_became_peered":"2024-03-06T07:20:25.894354+0000","last_unstale":"2024-03-06T07:20:28.365536+0000","last_undegraded":"2024-03-06T07:20:28.365536+0000","last_fullsized":"2024-03-06T07:20:28.365536+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:28.909072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880665+0000","last_change":"2024-03-06T07:20:21.861910+0000","last_active":"2024-03-06T07:20:27.880665+0000","last_peered":"2024-03-06T07:20:27.880665+0000","last_clean":"2024-03-06T07:20:27.880665+0000","last_became_active":"2024-03-06T07:20:21.861659+0000","last_became_peered":"2024-03-06T07:20:21.861659+0000","last_unstale":"2024-03-06T07:20:27.880665+0000","last_undegraded":"2024-03-06T07:20:27.880665+0000","last_fullsized":"2024-03-06T07:20:27.880665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:44.625861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701886+0000","last_change":"2024-03-06T07:20:20.849833+0000","last_active":"2024-03-06T07:20:28.701886+0000","last_peered":"2024-03-06T07:20:28.701886+0000","last_clean":"2024-03-06T07:20:28.701886+0000","last_became_active":"2024-03-06T07:20:20.849563+0000","last_became_peered":"2024-03-06T07:20:20.849563+0000","last_unstale":"2024-03-06T07:20:28.701886+0000","last_undegraded":"2024-03-06T07:20:28.701886+0000","last_fullsized":"2024-03-06T07:20:28.701886+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:43.621233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366005+0000","last_change":"2024-03-06T07:20:25.897100+0000","last_active":"2024-03-06T07:20:28.366005+0000","last_peered":"2024-03-06T07:20:28.366005+0000","last_clean":"2024-03-06T07:20:28.366005+0000","last_became_active":"2024-03-06T07:20:25.896658+0000","last_became_peered":"2024-03-06T07:20:25.896658+0000","last_unstale":"2024-03-06T07:20:28.366005+0000","last_undegraded":"2024-03-06T07:20:28.366005+0000","last_fullsized":"2024-03-06T07:20:28.366005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:27.800537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365526+0000","last_change":"2024-03-06T07:20:21.864469+0000","last_active":"2024-03-06T07:20:28.365526+0000","last_peered":"2024-03-06T07:20:28.365526+0000","last_clean":"2024-03-06T07:20:28.365526+0000","last_became_active":"2024-03-06T07:20:21.864141+0000","last_became_peered":"2024-03-06T07:20:21.864141+0000","last_unstale":"2024-03-06T07:20:28.365526+0000","last_undegraded":"2024-03-06T07:20:28.365526+0000","last_fullsized":"2024-03-06T07:20:28.365526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:17.903391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.878975+0000","last_change":"2024-03-06T07:20:20.845652+0000","last_active":"2024-03-06T07:20:27.878975+0000","last_peered":"2024-03-06T07:20:27.878975+0000","last_clean":"2024-03-06T07:20:27.878975+0000","last_became_active":"2024-03-06T07:20:20.845350+0000","last_became_peered":"2024-03-06T07:20:20.845350+0000","last_unstale":"2024-03-06T07:20:27.878975+0000","last_undegraded":"2024-03-06T07:20:27.878975+0000","last_fullsized":"2024-03-06T07:20:27.878975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:53.997041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366358+0000","last_change":"2024-03-06T07:20:25.898164+0000","last_active":"2024-03-06T07:20:28.366358+0000","last_peered":"2024-03-06T07:20:28.366358+0000","last_clean":"2024-03-06T07:20:28.366358+0000","last_became_active":"2024-03-06T07:20:25.897755+0000","last_became_peered":"2024-03-06T07:20:25.897755+0000","last_unstale":"2024-03-06T07:20:28.366358+0000","last_undegraded":"2024-03-06T07:20:28.366358+0000","last_fullsized":"2024-03-06T07:20:28.366358+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:46.450423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701002+0000","last_change":"2024-03-06T07:20:21.850395+0000","last_active":"2024-03-06T07:20:28.701002+0000","last_peered":"2024-03-06T07:20:28.701002+0000","last_clean":"2024-03-06T07:20:28.701002+0000","last_became_active":"2024-03-06T07:20:21.850117+0000","last_became_peered":"2024-03-06T07:20:21.850117+0000","last_unstale":"2024-03-06T07:20:28.701002+0000","last_undegraded":"2024-03-06T07:20:28.701002+0000","last_fullsized":"2024-03-06T07:20:28.701002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:42.612183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.200831+0000","last_change":"2024-03-06T07:20:20.857982+0000","last_active":"2024-03-06T07:20:28.200831+0000","last_peered":"2024-03-06T07:20:28.200831+0000","last_clean":"2024-03-06T07:20:28.200831+0000","last_became_active":"2024-03-06T07:20:20.857726+0000","last_became_peered":"2024-03-06T07:20:20.857726+0000","last_unstale":"2024-03-06T07:20:28.200831+0000","last_undegraded":"2024-03-06T07:20:28.200831+0000","last_fullsized":"2024-03-06T07:20:28.200831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:24.258322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701481+0000","last_change":"2024-03-06T07:20:26.502037+0000","last_active":"2024-03-06T07:20:28.701481+0000","last_peered":"2024-03-06T07:20:28.701481+0000","last_clean":"2024-03-06T07:20:28.701481+0000","last_became_active":"2024-03-06T07:20:26.501666+0000","last_became_peered":"2024-03-06T07:20:26.501666+0000","last_unstale":"2024-03-06T07:20:28.701481+0000","last_undegraded":"2024-03-06T07:20:28.701481+0000","last_fullsized":"2024-03-06T07:20:28.701481+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:27.800537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880611+0000","last_change":"2024-03-06T07:20:21.858533+0000","last_active":"2024-03-06T07:20:27.880611+0000","last_peered":"2024-03-06T07:20:27.880611+0000","last_clean":"2024-03-06T07:20:27.880611+0000","last_became_active":"2024-03-06T07:20:21.857499+0000","last_became_peered":"2024-03-06T07:20:21.857499+0000","last_unstale":"2024-03-06T07:20:27.880611+0000","last_undegraded":"2024-03-06T07:20:27.880611+0000","last_fullsized":"2024-03-06T07:20:27.880611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:27.196530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701468+0000","last_change":"2024-03-06T07:20:20.858802+0000","last_active":"2024-03-06T07:20:28.701468+0000","last_peered":"2024-03-06T07:20:28.701468+0000","last_clean":"2024-03-06T07:20:28.701468+0000","last_became_active":"2024-03-06T07:20:20.857457+0000","last_became_peered":"2024-03-06T07:20:20.857457+0000","last_unstale":"2024-03-06T07:20:28.701468+0000","last_undegraded":"2024-03-06T07:20:28.701468+0000","last_fullsized":"2024-03-06T07:20:28.701468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:57.497682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":179,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201006+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:20:28.201006+0000","last_peered":"2024-03-06T07:20:28.201006+0000","last_clean":"2024-03-06T07:20:28.201006+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:20:28.201006+0000","last_undegraded":"2024-03-06T07:20:28.201006+0000","last_fullsized":"2024-03-06T07:20:28.201006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702367+0000","last_change":"2024-03-06T07:20:26.502317+0000","last_active":"2024-03-06T07:20:28.702367+0000","last_peered":"2024-03-06T07:20:28.702367+0000","last_clean":"2024-03-06T07:20:28.702367+0000","last_became_active":"2024-03-06T07:20:26.501807+0000","last_became_peered":"2024-03-06T07:20:26.501807+0000","last_unstale":"2024-03-06T07:20:28.702367+0000","last_undegraded":"2024-03-06T07:20:28.702367+0000","last_fullsized":"2024-03-06T07:20:28.702367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:30.890766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201288+0000","last_change":"2024-03-06T07:20:21.855669+0000","last_active":"2024-03-06T07:20:28.201288+0000","last_peered":"2024-03-06T07:20:28.201288+0000","last_clean":"2024-03-06T07:20:28.201288+0000","last_became_active":"2024-03-06T07:20:21.855154+0000","last_became_peered":"2024-03-06T07:20:21.855154+0000","last_unstale":"2024-03-06T07:20:28.201288+0000","last_undegraded":"2024-03-06T07:20:28.201288+0000","last_fullsized":"2024-03-06T07:20:28.201288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:26.855742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879323+0000","last_change":"2024-03-06T07:20:20.852702+0000","last_active":"2024-03-06T07:20:27.879323+0000","last_peered":"2024-03-06T07:20:27.879323+0000","last_clean":"2024-03-06T07:20:27.879323+0000","last_became_active":"2024-03-06T07:20:20.852434+0000","last_became_peered":"2024-03-06T07:20:20.852434+0000","last_unstale":"2024-03-06T07:20:27.879323+0000","last_undegraded":"2024-03-06T07:20:27.879323+0000","last_fullsized":"2024-03-06T07:20:27.879323+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:44.080359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702280+0000","last_change":"2024-03-06T07:20:26.499169+0000","last_active":"2024-03-06T07:20:28.702280+0000","last_peered":"2024-03-06T07:20:28.702280+0000","last_clean":"2024-03-06T07:20:28.702280+0000","last_became_active":"2024-03-06T07:20:26.498541+0000","last_became_peered":"2024-03-06T07:20:26.498541+0000","last_unstale":"2024-03-06T07:20:28.702280+0000","last_undegraded":"2024-03-06T07:20:28.702280+0000","last_fullsized":"2024-03-06T07:20:28.702280+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:49.037215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701682+0000","last_change":"2024-03-06T07:20:21.857920+0000","last_active":"2024-03-06T07:20:28.701682+0000","last_peered":"2024-03-06T07:20:28.701682+0000","last_clean":"2024-03-06T07:20:28.701682+0000","last_became_active":"2024-03-06T07:20:21.857645+0000","last_became_peered":"2024-03-06T07:20:21.857645+0000","last_unstale":"2024-03-06T07:20:28.701682+0000","last_undegraded":"2024-03-06T07:20:28.701682+0000","last_fullsized":"2024-03-06T07:20:28.701682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:47.201278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366028+0000","last_change":"2024-03-06T07:20:20.857009+0000","last_active":"2024-03-06T07:20:28.366028+0000","last_peered":"2024-03-06T07:20:28.366028+0000","last_clean":"2024-03-06T07:20:28.366028+0000","last_became_active":"2024-03-06T07:20:20.856701+0000","last_became_peered":"2024-03-06T07:20:20.856701+0000","last_unstale":"2024-03-06T07:20:28.366028+0000","last_undegraded":"2024-03-06T07:20:28.366028+0000","last_fullsized":"2024-03-06T07:20:28.366028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:24.258322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701482+0000","last_change":"2024-03-06T07:20:25.887514+0000","last_active":"2024-03-06T07:20:28.701482+0000","last_peered":"2024-03-06T07:20:28.701482+0000","last_clean":"2024-03-06T07:20:28.701482+0000","last_became_active":"2024-03-06T07:20:25.887125+0000","last_became_peered":"2024-03-06T07:20:25.887125+0000","last_unstale":"2024-03-06T07:20:28.701482+0000","last_undegraded":"2024-03-06T07:20:28.701482+0000","last_fullsized":"2024-03-06T07:20:28.701482+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:34.847448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201678+0000","last_change":"2024-03-06T07:20:21.851028+0000","last_active":"2024-03-06T07:20:28.201678+0000","last_peered":"2024-03-06T07:20:28.201678+0000","last_clean":"2024-03-06T07:20:28.201678+0000","last_became_active":"2024-03-06T07:20:21.850762+0000","last_became_peered":"2024-03-06T07:20:21.850762+0000","last_unstale":"2024-03-06T07:20:28.201678+0000","last_undegraded":"2024-03-06T07:20:28.201678+0000","last_fullsized":"2024-03-06T07:20:28.201678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:27.873513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879685+0000","last_change":"2024-03-06T07:20:20.849572+0000","last_active":"2024-03-06T07:20:27.879685+0000","last_peered":"2024-03-06T07:20:27.879685+0000","last_clean":"2024-03-06T07:20:27.879685+0000","last_became_active":"2024-03-06T07:20:20.849266+0000","last_became_peered":"2024-03-06T07:20:20.849266+0000","last_unstale":"2024-03-06T07:20:27.879685+0000","last_undegraded":"2024-03-06T07:20:27.879685+0000","last_fullsized":"2024-03-06T07:20:27.879685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:47.332802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701273+0000","last_change":"2024-03-06T07:20:26.502258+0000","last_active":"2024-03-06T07:20:28.701273+0000","last_peered":"2024-03-06T07:20:28.701273+0000","last_clean":"2024-03-06T07:20:28.701273+0000","last_became_active":"2024-03-06T07:20:26.501819+0000","last_became_peered":"2024-03-06T07:20:26.501819+0000","last_unstale":"2024-03-06T07:20:28.701273+0000","last_undegraded":"2024-03-06T07:20:28.701273+0000","last_fullsized":"2024-03-06T07:20:28.701273+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:34.847448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201451+0000","last_change":"2024-03-06T07:20:21.844939+0000","last_active":"2024-03-06T07:20:28.201451+0000","last_peered":"2024-03-06T07:20:28.201451+0000","last_clean":"2024-03-06T07:20:28.201451+0000","last_became_active":"2024-03-06T07:20:21.844652+0000","last_became_peered":"2024-03-06T07:20:21.844652+0000","last_unstale":"2024-03-06T07:20:28.201451+0000","last_undegraded":"2024-03-06T07:20:28.201451+0000","last_fullsized":"2024-03-06T07:20:28.201451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:07.625355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880344+0000","last_change":"2024-03-06T07:20:20.850150+0000","last_active":"2024-03-06T07:20:27.880344+0000","last_peered":"2024-03-06T07:20:27.880344+0000","last_clean":"2024-03-06T07:20:27.880344+0000","last_became_active":"2024-03-06T07:20:20.849834+0000","last_became_peered":"2024-03-06T07:20:20.849834+0000","last_unstale":"2024-03-06T07:20:27.880344+0000","last_undegraded":"2024-03-06T07:20:27.880344+0000","last_fullsized":"2024-03-06T07:20:27.880344+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:56.227109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879732+0000","last_change":"2024-03-06T07:20:25.888682+0000","last_active":"2024-03-06T07:20:27.879732+0000","last_peered":"2024-03-06T07:20:27.879732+0000","last_clean":"2024-03-06T07:20:27.879732+0000","last_became_active":"2024-03-06T07:20:25.888259+0000","last_became_peered":"2024-03-06T07:20:25.888259+0000","last_unstale":"2024-03-06T07:20:27.879732+0000","last_undegraded":"2024-03-06T07:20:27.879732+0000","last_fullsized":"2024-03-06T07:20:27.879732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:57.736429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201088+0000","last_change":"2024-03-06T07:20:21.851600+0000","last_active":"2024-03-06T07:20:28.201088+0000","last_peered":"2024-03-06T07:20:28.201088+0000","last_clean":"2024-03-06T07:20:28.201088+0000","last_became_active":"2024-03-06T07:20:21.851347+0000","last_became_peered":"2024-03-06T07:20:21.851347+0000","last_unstale":"2024-03-06T07:20:28.201088+0000","last_undegraded":"2024-03-06T07:20:28.201088+0000","last_fullsized":"2024-03-06T07:20:28.201088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:03.883136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.878780+0000","last_change":"2024-03-06T07:20:20.847128+0000","last_active":"2024-03-06T07:20:27.878780+0000","last_peered":"2024-03-06T07:20:27.878780+0000","last_clean":"2024-03-06T07:20:27.878780+0000","last_became_active":"2024-03-06T07:20:20.846822+0000","last_became_peered":"2024-03-06T07:20:20.846822+0000","last_unstale":"2024-03-06T07:20:27.878780+0000","last_undegraded":"2024-03-06T07:20:27.878780+0000","last_fullsized":"2024-03-06T07:20:27.878780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:57.497682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701347+0000","last_change":"2024-03-06T07:20:25.888397+0000","last_active":"2024-03-06T07:20:28.701347+0000","last_peered":"2024-03-06T07:20:28.701347+0000","last_clean":"2024-03-06T07:20:28.701347+0000","last_became_active":"2024-03-06T07:20:25.888017+0000","last_became_peered":"2024-03-06T07:20:25.888017+0000","last_unstale":"2024-03-06T07:20:28.701347+0000","last_undegraded":"2024-03-06T07:20:28.701347+0000","last_fullsized":"2024-03-06T07:20:28.701347+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:22.654122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702085+0000","last_change":"2024-03-06T07:20:21.863218+0000","last_active":"2024-03-06T07:20:28.702085+0000","last_peered":"2024-03-06T07:20:28.702085+0000","last_clean":"2024-03-06T07:20:28.702085+0000","last_became_active":"2024-03-06T07:20:21.862943+0000","last_became_peered":"2024-03-06T07:20:21.862943+0000","last_unstale":"2024-03-06T07:20:28.702085+0000","last_undegraded":"2024-03-06T07:20:28.702085+0000","last_fullsized":"2024-03-06T07:20:28.702085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:17.903391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701478+0000","last_change":"2024-03-06T07:20:20.855196+0000","last_active":"2024-03-06T07:20:28.701478+0000","last_peered":"2024-03-06T07:20:28.701478+0000","last_clean":"2024-03-06T07:20:28.701478+0000","last_became_active":"2024-03-06T07:20:20.854047+0000","last_became_peered":"2024-03-06T07:20:20.854047+0000","last_unstale":"2024-03-06T07:20:28.701478+0000","last_undegraded":"2024-03-06T07:20:28.701478+0000","last_fullsized":"2024-03-06T07:20:28.701478+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:15.716891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880771+0000","last_change":"2024-03-06T07:20:26.502104+0000","last_active":"2024-03-06T07:20:27.880771+0000","last_peered":"2024-03-06T07:20:27.880771+0000","last_clean":"2024-03-06T07:20:27.880771+0000","last_became_active":"2024-03-06T07:20:26.501701+0000","last_became_peered":"2024-03-06T07:20:26.501701+0000","last_unstale":"2024-03-06T07:20:27.880771+0000","last_undegraded":"2024-03-06T07:20:27.880771+0000","last_fullsized":"2024-03-06T07:20:27.880771+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:13.344053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201065+0000","last_change":"2024-03-06T07:20:21.850971+0000","last_active":"2024-03-06T07:20:28.201065+0000","last_peered":"2024-03-06T07:20:28.201065+0000","last_clean":"2024-03-06T07:20:28.201065+0000","last_became_active":"2024-03-06T07:20:21.850700+0000","last_became_peered":"2024-03-06T07:20:21.850700+0000","last_unstale":"2024-03-06T07:20:28.201065+0000","last_undegraded":"2024-03-06T07:20:28.201065+0000","last_fullsized":"2024-03-06T07:20:28.201065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:17.299208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701777+0000","last_change":"2024-03-06T07:20:20.856747+0000","last_active":"2024-03-06T07:20:28.701777+0000","last_peered":"2024-03-06T07:20:28.701777+0000","last_clean":"2024-03-06T07:20:28.701777+0000","last_became_active":"2024-03-06T07:20:20.856431+0000","last_became_peered":"2024-03-06T07:20:20.856431+0000","last_unstale":"2024-03-06T07:20:28.701777+0000","last_undegraded":"2024-03-06T07:20:28.701777+0000","last_fullsized":"2024-03-06T07:20:28.701777+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:15.716891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880386+0000","last_change":"2024-03-06T07:20:26.498675+0000","last_active":"2024-03-06T07:20:27.880386+0000","last_peered":"2024-03-06T07:20:27.880386+0000","last_clean":"2024-03-06T07:20:27.880386+0000","last_became_active":"2024-03-06T07:20:26.498199+0000","last_became_peered":"2024-03-06T07:20:26.498199+0000","last_unstale":"2024-03-06T07:20:27.880386+0000","last_undegraded":"2024-03-06T07:20:27.880386+0000","last_fullsized":"2024-03-06T07:20:27.880386+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:53.210084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701227+0000","last_change":"2024-03-06T07:20:21.845586+0000","last_active":"2024-03-06T07:20:28.701227+0000","last_peered":"2024-03-06T07:20:28.701227+0000","last_clean":"2024-03-06T07:20:28.701227+0000","last_became_active":"2024-03-06T07:20:21.845311+0000","last_became_peered":"2024-03-06T07:20:21.845311+0000","last_unstale":"2024-03-06T07:20:28.701227+0000","last_undegraded":"2024-03-06T07:20:28.701227+0000","last_fullsized":"2024-03-06T07:20:28.701227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:58.153137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701540+0000","last_change":"2024-03-06T07:20:20.849649+0000","last_active":"2024-03-06T07:20:28.701540+0000","last_peered":"2024-03-06T07:20:28.701540+0000","last_clean":"2024-03-06T07:20:28.701540+0000","last_became_active":"2024-03-06T07:20:20.849199+0000","last_became_peered":"2024-03-06T07:20:20.849199+0000","last_unstale":"2024-03-06T07:20:28.701540+0000","last_undegraded":"2024-03-06T07:20:28.701540+0000","last_fullsized":"2024-03-06T07:20:28.701540+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:22.023999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702184+0000","last_change":"2024-03-06T07:20:26.499070+0000","last_active":"2024-03-06T07:20:28.702184+0000","last_peered":"2024-03-06T07:20:28.702184+0000","last_clean":"2024-03-06T07:20:28.702184+0000","last_became_active":"2024-03-06T07:20:26.498601+0000","last_became_peered":"2024-03-06T07:20:26.498601+0000","last_unstale":"2024-03-06T07:20:28.702184+0000","last_undegraded":"2024-03-06T07:20:28.702184+0000","last_fullsized":"2024-03-06T07:20:28.702184+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:46.450423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201193+0000","last_change":"2024-03-06T07:20:21.857736+0000","last_active":"2024-03-06T07:20:28.201193+0000","last_peered":"2024-03-06T07:20:28.201193+0000","last_clean":"2024-03-06T07:20:28.201193+0000","last_became_active":"2024-03-06T07:20:21.857492+0000","last_became_peered":"2024-03-06T07:20:21.857492+0000","last_unstale":"2024-03-06T07:20:28.201193+0000","last_undegraded":"2024-03-06T07:20:28.201193+0000","last_fullsized":"2024-03-06T07:20:28.201193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:45.002191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:30.723150+0000","last_change":"2024-03-06T07:20:20.856316+0000","last_active":"2024-03-06T07:20:30.723150+0000","last_peered":"2024-03-06T07:20:30.723150+0000","last_clean":"2024-03-06T07:20:30.723150+0000","last_became_active":"2024-03-06T07:20:20.855994+0000","last_became_peered":"2024-03-06T07:20:20.855994+0000","last_unstale":"2024-03-06T07:20:30.723150+0000","last_undegraded":"2024-03-06T07:20:30.723150+0000","last_fullsized":"2024-03-06T07:20:30.723150+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:53.997041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3701,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365632+0000","last_change":"2024-03-06T07:20:25.895379+0000","last_active":"2024-03-06T07:20:28.365632+0000","last_peered":"2024-03-06T07:20:28.365632+0000","last_clean":"2024-03-06T07:20:28.365632+0000","last_became_active":"2024-03-06T07:20:25.894458+0000","last_became_peered":"2024-03-06T07:20:25.894458+0000","last_unstale":"2024-03-06T07:20:28.365632+0000","last_undegraded":"2024-03-06T07:20:28.365632+0000","last_fullsized":"2024-03-06T07:20:28.365632+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:22.654122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880548+0000","last_change":"2024-03-06T07:20:21.844398+0000","last_active":"2024-03-06T07:20:27.880548+0000","last_peered":"2024-03-06T07:20:27.880548+0000","last_clean":"2024-03-06T07:20:27.880548+0000","last_became_active":"2024-03-06T07:20:21.843839+0000","last_became_peered":"2024-03-06T07:20:21.843839+0000","last_unstale":"2024-03-06T07:20:27.880548+0000","last_undegraded":"2024-03-06T07:20:27.880548+0000","last_fullsized":"2024-03-06T07:20:27.880548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:25.262951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365950+0000","last_change":"2024-03-06T07:20:20.850772+0000","last_active":"2024-03-06T07:20:28.365950+0000","last_peered":"2024-03-06T07:20:28.365950+0000","last_clean":"2024-03-06T07:20:28.365950+0000","last_became_active":"2024-03-06T07:20:20.850490+0000","last_became_peered":"2024-03-06T07:20:20.850490+0000","last_unstale":"2024-03-06T07:20:28.365950+0000","last_undegraded":"2024-03-06T07:20:28.365950+0000","last_fullsized":"2024-03-06T07:20:28.365950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:58.697205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879970+0000","last_change":"2024-03-06T07:20:25.887556+0000","last_active":"2024-03-06T07:20:27.879970+0000","last_peered":"2024-03-06T07:20:27.879970+0000","last_clean":"2024-03-06T07:20:27.879970+0000","last_became_active":"2024-03-06T07:20:25.887175+0000","last_became_peered":"2024-03-06T07:20:25.887175+0000","last_unstale":"2024-03-06T07:20:27.879970+0000","last_undegraded":"2024-03-06T07:20:27.879970+0000","last_fullsized":"2024-03-06T07:20:27.879970+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:28.909072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701501+0000","last_change":"2024-03-06T07:20:21.856048+0000","last_active":"2024-03-06T07:20:28.701501+0000","last_peered":"2024-03-06T07:20:28.701501+0000","last_clean":"2024-03-06T07:20:28.701501+0000","last_became_active":"2024-03-06T07:20:21.854721+0000","last_became_peered":"2024-03-06T07:20:21.854721+0000","last_unstale":"2024-03-06T07:20:28.701501+0000","last_undegraded":"2024-03-06T07:20:28.701501+0000","last_fullsized":"2024-03-06T07:20:28.701501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:00.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702082+0000","last_change":"2024-03-06T07:20:20.859239+0000","last_active":"2024-03-06T07:20:28.702082+0000","last_peered":"2024-03-06T07:20:28.702082+0000","last_clean":"2024-03-06T07:20:28.702082+0000","last_became_active":"2024-03-06T07:20:20.858860+0000","last_became_peered":"2024-03-06T07:20:20.858860+0000","last_unstale":"2024-03-06T07:20:28.702082+0000","last_undegraded":"2024-03-06T07:20:28.702082+0000","last_fullsized":"2024-03-06T07:20:28.702082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:47.332802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879712+0000","last_change":"2024-03-06T07:20:25.890675+0000","last_active":"2024-03-06T07:20:27.879712+0000","last_peered":"2024-03-06T07:20:27.879712+0000","last_clean":"2024-03-06T07:20:27.879712+0000","last_became_active":"2024-03-06T07:20:25.890282+0000","last_became_peered":"2024-03-06T07:20:25.890282+0000","last_unstale":"2024-03-06T07:20:27.879712+0000","last_undegraded":"2024-03-06T07:20:27.879712+0000","last_fullsized":"2024-03-06T07:20:27.879712+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:10:13.387120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702028+0000","last_change":"2024-03-06T07:20:21.864960+0000","last_active":"2024-03-06T07:20:28.702028+0000","last_peered":"2024-03-06T07:20:28.702028+0000","last_clean":"2024-03-06T07:20:28.702028+0000","last_became_active":"2024-03-06T07:20:21.864687+0000","last_became_peered":"2024-03-06T07:20:21.864687+0000","last_unstale":"2024-03-06T07:20:28.702028+0000","last_undegraded":"2024-03-06T07:20:28.702028+0000","last_fullsized":"2024-03-06T07:20:28.702028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:27.196530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702647+0000","last_change":"2024-03-06T07:20:20.857187+0000","last_active":"2024-03-06T07:20:28.702647+0000","last_peered":"2024-03-06T07:20:28.702647+0000","last_clean":"2024-03-06T07:20:28.702647+0000","last_became_active":"2024-03-06T07:20:20.856898+0000","last_became_peered":"2024-03-06T07:20:20.856898+0000","last_unstale":"2024-03-06T07:20:28.702647+0000","last_undegraded":"2024-03-06T07:20:28.702647+0000","last_fullsized":"2024-03-06T07:20:28.702647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:57.497682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879802+0000","last_change":"2024-03-06T07:20:26.503331+0000","last_active":"2024-03-06T07:20:27.879802+0000","last_peered":"2024-03-06T07:20:27.879802+0000","last_clean":"2024-03-06T07:20:27.879802+0000","last_became_active":"2024-03-06T07:20:26.502770+0000","last_became_peered":"2024-03-06T07:20:26.502770+0000","last_unstale":"2024-03-06T07:20:27.879802+0000","last_undegraded":"2024-03-06T07:20:27.879802+0000","last_fullsized":"2024-03-06T07:20:27.879802+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:13.262016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701297+0000","last_change":"2024-03-06T07:20:26.502227+0000","last_active":"2024-03-06T07:20:28.701297+0000","last_peered":"2024-03-06T07:20:28.701297+0000","last_clean":"2024-03-06T07:20:28.701297+0000","last_became_active":"2024-03-06T07:20:26.501641+0000","last_became_peered":"2024-03-06T07:20:26.501641+0000","last_unstale":"2024-03-06T07:20:28.701297+0000","last_undegraded":"2024-03-06T07:20:28.701297+0000","last_fullsized":"2024-03-06T07:20:28.701297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:31.908538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365327+0000","last_change":"2024-03-06T07:20:21.866845+0000","last_active":"2024-03-06T07:20:28.365327+0000","last_peered":"2024-03-06T07:20:28.365327+0000","last_clean":"2024-03-06T07:20:28.365327+0000","last_became_active":"2024-03-06T07:20:21.866576+0000","last_became_peered":"2024-03-06T07:20:21.866576+0000","last_unstale":"2024-03-06T07:20:28.365327+0000","last_undegraded":"2024-03-06T07:20:28.365327+0000","last_fullsized":"2024-03-06T07:20:28.365327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:36.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701723+0000","last_change":"2024-03-06T07:20:20.853894+0000","last_active":"2024-03-06T07:20:28.701723+0000","last_peered":"2024-03-06T07:20:28.701723+0000","last_clean":"2024-03-06T07:20:28.701723+0000","last_became_active":"2024-03-06T07:20:20.853538+0000","last_became_peered":"2024-03-06T07:20:20.853538+0000","last_unstale":"2024-03-06T07:20:28.701723+0000","last_undegraded":"2024-03-06T07:20:28.701723+0000","last_fullsized":"2024-03-06T07:20:28.701723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:25.763288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702543+0000","last_change":"2024-03-06T07:20:25.894539+0000","last_active":"2024-03-06T07:20:28.702543+0000","last_peered":"2024-03-06T07:20:28.702543+0000","last_clean":"2024-03-06T07:20:28.702543+0000","last_became_active":"2024-03-06T07:20:25.894137+0000","last_became_peered":"2024-03-06T07:20:25.894137+0000","last_unstale":"2024-03-06T07:20:28.702543+0000","last_undegraded":"2024-03-06T07:20:28.702543+0000","last_fullsized":"2024-03-06T07:20:28.702543+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:57.736429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879760+0000","last_change":"2024-03-06T07:20:21.869266+0000","last_active":"2024-03-06T07:20:27.879760+0000","last_peered":"2024-03-06T07:20:27.879760+0000","last_clean":"2024-03-06T07:20:27.879760+0000","last_became_active":"2024-03-06T07:20:21.861979+0000","last_became_peered":"2024-03-06T07:20:21.861979+0000","last_unstale":"2024-03-06T07:20:27.879760+0000","last_undegraded":"2024-03-06T07:20:27.879760+0000","last_fullsized":"2024-03-06T07:20:27.879760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:17.903391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879911+0000","last_change":"2024-03-06T07:20:20.853365+0000","last_active":"2024-03-06T07:20:27.879911+0000","last_peered":"2024-03-06T07:20:27.879911+0000","last_clean":"2024-03-06T07:20:27.879911+0000","last_became_active":"2024-03-06T07:20:20.853065+0000","last_became_peered":"2024-03-06T07:20:20.853065+0000","last_unstale":"2024-03-06T07:20:27.879911+0000","last_undegraded":"2024-03-06T07:20:27.879911+0000","last_fullsized":"2024-03-06T07:20:27.879911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:53.997041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702584+0000","last_change":"2024-03-06T07:20:20.852681+0000","last_active":"2024-03-06T07:20:28.702584+0000","last_peered":"2024-03-06T07:20:28.702584+0000","last_clean":"2024-03-06T07:20:28.702584+0000","last_became_active":"2024-03-06T07:20:20.852406+0000","last_became_peered":"2024-03-06T07:20:20.852406+0000","last_unstale":"2024-03-06T07:20:28.702584+0000","last_undegraded":"2024-03-06T07:20:28.702584+0000","last_fullsized":"2024-03-06T07:20:28.702584+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:25.763288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701475+0000","last_change":"2024-03-06T07:20:21.854101+0000","last_active":"2024-03-06T07:20:28.701475+0000","last_peered":"2024-03-06T07:20:28.701475+0000","last_clean":"2024-03-06T07:20:28.701475+0000","last_became_active":"2024-03-06T07:20:21.853720+0000","last_became_peered":"2024-03-06T07:20:21.853720+0000","last_unstale":"2024-03-06T07:20:28.701475+0000","last_undegraded":"2024-03-06T07:20:28.701475+0000","last_fullsized":"2024-03-06T07:20:28.701475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.423089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879159+0000","last_change":"2024-03-06T07:20:25.889455+0000","last_active":"2024-03-06T07:20:27.879159+0000","last_peered":"2024-03-06T07:20:27.879159+0000","last_clean":"2024-03-06T07:20:27.879159+0000","last_became_active":"2024-03-06T07:20:25.889021+0000","last_became_peered":"2024-03-06T07:20:25.889021+0000","last_unstale":"2024-03-06T07:20:27.879159+0000","last_undegraded":"2024-03-06T07:20:27.879159+0000","last_fullsized":"2024-03-06T07:20:27.879159+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:46.647208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880275+0000","last_change":"2024-03-06T07:20:20.860219+0000","last_active":"2024-03-06T07:20:27.880275+0000","last_peered":"2024-03-06T07:20:27.880275+0000","last_clean":"2024-03-06T07:20:27.880275+0000","last_became_active":"2024-03-06T07:20:20.859862+0000","last_became_peered":"2024-03-06T07:20:20.859862+0000","last_unstale":"2024-03-06T07:20:27.880275+0000","last_undegraded":"2024-03-06T07:20:27.880275+0000","last_fullsized":"2024-03-06T07:20:27.880275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:44.080359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879980+0000","last_change":"2024-03-06T07:20:21.858660+0000","last_active":"2024-03-06T07:20:27.879980+0000","last_peered":"2024-03-06T07:20:27.879980+0000","last_clean":"2024-03-06T07:20:27.879980+0000","last_became_active":"2024-03-06T07:20:21.858397+0000","last_became_peered":"2024-03-06T07:20:21.858397+0000","last_unstale":"2024-03-06T07:20:27.879980+0000","last_undegraded":"2024-03-06T07:20:27.879980+0000","last_fullsized":"2024-03-06T07:20:27.879980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:16.721519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880388+0000","last_change":"2024-03-06T07:20:25.889183+0000","last_active":"2024-03-06T07:20:27.880388+0000","last_peered":"2024-03-06T07:20:27.880388+0000","last_clean":"2024-03-06T07:20:27.880388+0000","last_became_active":"2024-03-06T07:20:25.888836+0000","last_became_peered":"2024-03-06T07:20:25.888836+0000","last_unstale":"2024-03-06T07:20:27.880388+0000","last_undegraded":"2024-03-06T07:20:27.880388+0000","last_fullsized":"2024-03-06T07:20:27.880388+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:28.743175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701231+0000","last_change":"2024-03-06T07:20:21.863258+0000","last_active":"2024-03-06T07:20:28.701231+0000","last_peered":"2024-03-06T07:20:28.701231+0000","last_clean":"2024-03-06T07:20:28.701231+0000","last_became_active":"2024-03-06T07:20:21.862976+0000","last_became_peered":"2024-03-06T07:20:21.862976+0000","last_unstale":"2024-03-06T07:20:28.701231+0000","last_undegraded":"2024-03-06T07:20:28.701231+0000","last_fullsized":"2024-03-06T07:20:28.701231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.878887+0000","last_change":"2024-03-06T07:20:21.860263+0000","last_active":"2024-03-06T07:20:27.878887+0000","last_peered":"2024-03-06T07:20:27.878887+0000","last_clean":"2024-03-06T07:20:27.878887+0000","last_became_active":"2024-03-06T07:20:21.859978+0000","last_became_peered":"2024-03-06T07:20:21.859978+0000","last_unstale":"2024-03-06T07:20:27.878887+0000","last_undegraded":"2024-03-06T07:20:27.878887+0000","last_fullsized":"2024-03-06T07:20:27.878887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:59.701834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701885+0000","last_change":"2024-03-06T07:20:21.868562+0000","last_active":"2024-03-06T07:20:28.701885+0000","last_peered":"2024-03-06T07:20:28.701885+0000","last_clean":"2024-03-06T07:20:28.701885+0000","last_became_active":"2024-03-06T07:20:21.868239+0000","last_became_peered":"2024-03-06T07:20:21.868239+0000","last_unstale":"2024-03-06T07:20:28.701885+0000","last_undegraded":"2024-03-06T07:20:28.701885+0000","last_fullsized":"2024-03-06T07:20:28.701885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:45.002191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701377+0000","last_change":"2024-03-06T07:20:21.860455+0000","last_active":"2024-03-06T07:20:28.701377+0000","last_peered":"2024-03-06T07:20:28.701377+0000","last_clean":"2024-03-06T07:20:28.701377+0000","last_became_active":"2024-03-06T07:20:21.859855+0000","last_became_peered":"2024-03-06T07:20:21.859855+0000","last_unstale":"2024-03-06T07:20:28.701377+0000","last_undegraded":"2024-03-06T07:20:28.701377+0000","last_fullsized":"2024-03-06T07:20:28.701377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879935+0000","last_change":"2024-03-06T07:20:21.856836+0000","last_active":"2024-03-06T07:20:27.879935+0000","last_peered":"2024-03-06T07:20:27.879935+0000","last_clean":"2024-03-06T07:20:27.879935+0000","last_became_active":"2024-03-06T07:20:21.856421+0000","last_became_peered":"2024-03-06T07:20:21.856421+0000","last_unstale":"2024-03-06T07:20:27.879935+0000","last_undegraded":"2024-03-06T07:20:27.879935+0000","last_fullsized":"2024-03-06T07:20:27.879935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:00.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365852+0000","last_change":"2024-03-06T07:20:21.866911+0000","last_active":"2024-03-06T07:20:28.365852+0000","last_peered":"2024-03-06T07:20:28.365852+0000","last_clean":"2024-03-06T07:20:28.365852+0000","last_became_active":"2024-03-06T07:20:21.866557+0000","last_became_peered":"2024-03-06T07:20:21.866557+0000","last_unstale":"2024-03-06T07:20:28.365852+0000","last_undegraded":"2024-03-06T07:20:28.365852+0000","last_fullsized":"2024-03-06T07:20:28.365852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702389+0000","last_change":"2024-03-06T07:20:21.866315+0000","last_active":"2024-03-06T07:20:28.702389+0000","last_peered":"2024-03-06T07:20:28.702389+0000","last_clean":"2024-03-06T07:20:28.702389+0000","last_became_active":"2024-03-06T07:20:21.866035+0000","last_became_peered":"2024-03-06T07:20:21.866035+0000","last_unstale":"2024-03-06T07:20:28.702389+0000","last_undegraded":"2024-03-06T07:20:28.702389+0000","last_fullsized":"2024-03-06T07:20:28.702389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:20.911597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880753+0000","last_change":"2024-03-06T07:20:21.866364+0000","last_active":"2024-03-06T07:20:27.880753+0000","last_peered":"2024-03-06T07:20:27.880753+0000","last_clean":"2024-03-06T07:20:27.880753+0000","last_became_active":"2024-03-06T07:20:21.866091+0000","last_became_peered":"2024-03-06T07:20:21.866091+0000","last_unstale":"2024-03-06T07:20:27.880753+0000","last_undegraded":"2024-03-06T07:20:27.880753+0000","last_fullsized":"2024-03-06T07:20:27.880753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:01.604000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701743+0000","last_change":"2024-03-06T07:20:21.868488+0000","last_active":"2024-03-06T07:20:28.701743+0000","last_peered":"2024-03-06T07:20:28.701743+0000","last_clean":"2024-03-06T07:20:28.701743+0000","last_became_active":"2024-03-06T07:20:21.868091+0000","last_became_peered":"2024-03-06T07:20:21.868091+0000","last_unstale":"2024-03-06T07:20:28.701743+0000","last_undegraded":"2024-03-06T07:20:28.701743+0000","last_fullsized":"2024-03-06T07:20:28.701743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:00.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880000+0000","last_change":"2024-03-06T07:20:21.853440+0000","last_active":"2024-03-06T07:20:27.880000+0000","last_peered":"2024-03-06T07:20:27.880000+0000","last_clean":"2024-03-06T07:20:27.880000+0000","last_became_active":"2024-03-06T07:20:21.853089+0000","last_became_peered":"2024-03-06T07:20:21.853089+0000","last_unstale":"2024-03-06T07:20:27.880000+0000","last_undegraded":"2024-03-06T07:20:27.880000+0000","last_fullsized":"2024-03-06T07:20:27.880000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:20.911597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701633+0000","last_change":"2024-03-06T07:20:21.865493+0000","last_active":"2024-03-06T07:20:28.701633+0000","last_peered":"2024-03-06T07:20:28.701633+0000","last_clean":"2024-03-06T07:20:28.701633+0000","last_became_active":"2024-03-06T07:20:21.865203+0000","last_became_peered":"2024-03-06T07:20:21.865203+0000","last_unstale":"2024-03-06T07:20:28.701633+0000","last_undegraded":"2024-03-06T07:20:28.701633+0000","last_fullsized":"2024-03-06T07:20:28.701633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:20.882851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701280+0000","last_change":"2024-03-06T07:20:21.847412+0000","last_active":"2024-03-06T07:20:28.701280+0000","last_peered":"2024-03-06T07:20:28.701280+0000","last_clean":"2024-03-06T07:20:28.701280+0000","last_became_active":"2024-03-06T07:20:21.847081+0000","last_became_peered":"2024-03-06T07:20:21.847081+0000","last_unstale":"2024-03-06T07:20:28.701280+0000","last_undegraded":"2024-03-06T07:20:28.701280+0000","last_fullsized":"2024-03-06T07:20:28.701280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:28.363605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702051+0000","last_change":"2024-03-06T07:20:21.861878+0000","last_active":"2024-03-06T07:20:28.702051+0000","last_peered":"2024-03-06T07:20:28.702051+0000","last_clean":"2024-03-06T07:20:28.702051+0000","last_became_active":"2024-03-06T07:20:21.859115+0000","last_became_peered":"2024-03-06T07:20:21.859115+0000","last_unstale":"2024-03-06T07:20:28.702051+0000","last_undegraded":"2024-03-06T07:20:28.702051+0000","last_fullsized":"2024-03-06T07:20:28.702051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:42.612183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201453+0000","last_change":"2024-03-06T07:20:21.860975+0000","last_active":"2024-03-06T07:20:28.201453+0000","last_peered":"2024-03-06T07:20:28.201453+0000","last_clean":"2024-03-06T07:20:28.201453+0000","last_became_active":"2024-03-06T07:20:21.860414+0000","last_became_peered":"2024-03-06T07:20:21.860414+0000","last_unstale":"2024-03-06T07:20:28.201453+0000","last_undegraded":"2024-03-06T07:20:28.201453+0000","last_fullsized":"2024-03-06T07:20:28.201453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:53.701404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879575+0000","last_change":"2024-03-06T07:20:21.867097+0000","last_active":"2024-03-06T07:20:27.879575+0000","last_peered":"2024-03-06T07:20:27.879575+0000","last_clean":"2024-03-06T07:20:27.879575+0000","last_became_active":"2024-03-06T07:20:21.866821+0000","last_became_peered":"2024-03-06T07:20:21.866821+0000","last_unstale":"2024-03-06T07:20:27.879575+0000","last_undegraded":"2024-03-06T07:20:27.879575+0000","last_fullsized":"2024-03-06T07:20:27.879575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:25.262951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702397+0000","last_change":"2024-03-06T07:20:21.866152+0000","last_active":"2024-03-06T07:20:28.702397+0000","last_peered":"2024-03-06T07:20:28.702397+0000","last_clean":"2024-03-06T07:20:28.702397+0000","last_became_active":"2024-03-06T07:20:21.865919+0000","last_became_peered":"2024-03-06T07:20:21.865919+0000","last_unstale":"2024-03-06T07:20:28.702397+0000","last_undegraded":"2024-03-06T07:20:28.702397+0000","last_fullsized":"2024-03-06T07:20:28.702397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:26.855742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701560+0000","last_change":"2024-03-06T07:20:21.860761+0000","last_active":"2024-03-06T07:20:28.701560+0000","last_peered":"2024-03-06T07:20:28.701560+0000","last_clean":"2024-03-06T07:20:28.701560+0000","last_became_active":"2024-03-06T07:20:21.860444+0000","last_became_peered":"2024-03-06T07:20:21.860444+0000","last_unstale":"2024-03-06T07:20:28.701560+0000","last_undegraded":"2024-03-06T07:20:28.701560+0000","last_fullsized":"2024-03-06T07:20:28.701560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365409+0000","last_change":"2024-03-06T07:20:21.855033+0000","last_active":"2024-03-06T07:20:28.365409+0000","last_peered":"2024-03-06T07:20:28.365409+0000","last_clean":"2024-03-06T07:20:28.365409+0000","last_became_active":"2024-03-06T07:20:21.854554+0000","last_became_peered":"2024-03-06T07:20:21.854554+0000","last_unstale":"2024-03-06T07:20:28.365409+0000","last_undegraded":"2024-03-06T07:20:28.365409+0000","last_fullsized":"2024-03-06T07:20:28.365409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:20.911597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701196+0000","last_change":"2024-03-06T07:20:21.856900+0000","last_active":"2024-03-06T07:20:28.701196+0000","last_peered":"2024-03-06T07:20:28.701196+0000","last_clean":"2024-03-06T07:20:28.701196+0000","last_became_active":"2024-03-06T07:20:21.856628+0000","last_became_peered":"2024-03-06T07:20:21.856628+0000","last_unstale":"2024-03-06T07:20:28.701196+0000","last_undegraded":"2024-03-06T07:20:28.701196+0000","last_fullsized":"2024-03-06T07:20:28.701196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:47.276655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880101+0000","last_change":"2024-03-06T07:20:21.868803+0000","last_active":"2024-03-06T07:20:27.880101+0000","last_peered":"2024-03-06T07:20:27.880101+0000","last_clean":"2024-03-06T07:20:27.880101+0000","last_became_active":"2024-03-06T07:20:21.868518+0000","last_became_peered":"2024-03-06T07:20:21.868518+0000","last_unstale":"2024-03-06T07:20:27.880101+0000","last_undegraded":"2024-03-06T07:20:27.880101+0000","last_fullsized":"2024-03-06T07:20:27.880101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:03.883136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880416+0000","last_change":"2024-03-06T07:20:21.863674+0000","last_active":"2024-03-06T07:20:27.880416+0000","last_peered":"2024-03-06T07:20:27.880416+0000","last_clean":"2024-03-06T07:20:27.880416+0000","last_became_active":"2024-03-06T07:20:21.863138+0000","last_became_peered":"2024-03-06T07:20:21.863138+0000","last_unstale":"2024-03-06T07:20:27.880416+0000","last_undegraded":"2024-03-06T07:20:27.880416+0000","last_fullsized":"2024-03-06T07:20:27.880416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:26.767916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702522+0000","last_change":"2024-03-06T07:20:21.858343+0000","last_active":"2024-03-06T07:20:28.702522+0000","last_peered":"2024-03-06T07:20:28.702522+0000","last_clean":"2024-03-06T07:20:28.702522+0000","last_became_active":"2024-03-06T07:20:21.858078+0000","last_became_peered":"2024-03-06T07:20:21.858078+0000","last_unstale":"2024-03-06T07:20:28.702522+0000","last_undegraded":"2024-03-06T07:20:28.702522+0000","last_fullsized":"2024-03-06T07:20:28.702522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.423089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701847+0000","last_change":"2024-03-06T07:20:21.866157+0000","last_active":"2024-03-06T07:20:28.701847+0000","last_peered":"2024-03-06T07:20:28.701847+0000","last_clean":"2024-03-06T07:20:28.701847+0000","last_became_active":"2024-03-06T07:20:21.865306+0000","last_became_peered":"2024-03-06T07:20:21.865306+0000","last_unstale":"2024-03-06T07:20:28.701847+0000","last_undegraded":"2024-03-06T07:20:28.701847+0000","last_fullsized":"2024-03-06T07:20:28.701847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:36.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201624+0000","last_change":"2024-03-06T07:20:21.855346+0000","last_active":"2024-03-06T07:20:28.201624+0000","last_peered":"2024-03-06T07:20:28.201624+0000","last_clean":"2024-03-06T07:20:28.201624+0000","last_became_active":"2024-03-06T07:20:21.855059+0000","last_became_peered":"2024-03-06T07:20:21.855059+0000","last_unstale":"2024-03-06T07:20:28.201624+0000","last_undegraded":"2024-03-06T07:20:28.201624+0000","last_fullsized":"2024-03-06T07:20:28.201624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:51.111188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879524+0000","last_change":"2024-03-06T07:20:21.868545+0000","last_active":"2024-03-06T07:20:27.879524+0000","last_peered":"2024-03-06T07:20:27.879524+0000","last_clean":"2024-03-06T07:20:27.879524+0000","last_became_active":"2024-03-06T07:20:21.868188+0000","last_became_peered":"2024-03-06T07:20:21.868188+0000","last_unstale":"2024-03-06T07:20:27.879524+0000","last_undegraded":"2024-03-06T07:20:27.879524+0000","last_fullsized":"2024-03-06T07:20:27.879524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:28.363605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201003+0000","last_change":"2024-03-06T07:20:21.856718+0000","last_active":"2024-03-06T07:20:28.201003+0000","last_peered":"2024-03-06T07:20:28.201003+0000","last_clean":"2024-03-06T07:20:28.201003+0000","last_became_active":"2024-03-06T07:20:21.856451+0000","last_became_peered":"2024-03-06T07:20:21.856451+0000","last_unstale":"2024-03-06T07:20:28.201003+0000","last_undegraded":"2024-03-06T07:20:28.201003+0000","last_fullsized":"2024-03-06T07:20:28.201003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:01.604000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701625+0000","last_change":"2024-03-06T07:20:21.861834+0000","last_active":"2024-03-06T07:20:28.701625+0000","last_peered":"2024-03-06T07:20:28.701625+0000","last_clean":"2024-03-06T07:20:28.701625+0000","last_became_active":"2024-03-06T07:20:21.860998+0000","last_became_peered":"2024-03-06T07:20:21.860998+0000","last_unstale":"2024-03-06T07:20:28.701625+0000","last_undegraded":"2024-03-06T07:20:28.701625+0000","last_fullsized":"2024-03-06T07:20:28.701625+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:01.604000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.200939+0000","last_change":"2024-03-06T07:20:21.860505+0000","last_active":"2024-03-06T07:20:28.200939+0000","last_peered":"2024-03-06T07:20:28.200939+0000","last_clean":"2024-03-06T07:20:28.200939+0000","last_became_active":"2024-03-06T07:20:21.860241+0000","last_became_peered":"2024-03-06T07:20:21.860241+0000","last_unstale":"2024-03-06T07:20:28.200939+0000","last_undegraded":"2024-03-06T07:20:28.200939+0000","last_fullsized":"2024-03-06T07:20:28.200939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:20.882851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701769+0000","last_change":"2024-03-06T07:20:21.868402+0000","last_active":"2024-03-06T07:20:28.701769+0000","last_peered":"2024-03-06T07:20:28.701769+0000","last_clean":"2024-03-06T07:20:28.701769+0000","last_became_active":"2024-03-06T07:20:21.868068+0000","last_became_peered":"2024-03-06T07:20:21.868068+0000","last_unstale":"2024-03-06T07:20:28.701769+0000","last_undegraded":"2024-03-06T07:20:28.701769+0000","last_fullsized":"2024-03-06T07:20:28.701769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:03.883136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701196+0000","last_change":"2024-03-06T07:20:21.860664+0000","last_active":"2024-03-06T07:20:28.701196+0000","last_peered":"2024-03-06T07:20:28.701196+0000","last_clean":"2024-03-06T07:20:28.701196+0000","last_became_active":"2024-03-06T07:20:21.860393+0000","last_became_peered":"2024-03-06T07:20:21.860393+0000","last_unstale":"2024-03-06T07:20:28.701196+0000","last_undegraded":"2024-03-06T07:20:28.701196+0000","last_fullsized":"2024-03-06T07:20:28.701196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:36.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366176+0000","last_change":"2024-03-06T07:20:21.860087+0000","last_active":"2024-03-06T07:20:28.366176+0000","last_peered":"2024-03-06T07:20:28.366176+0000","last_clean":"2024-03-06T07:20:28.366176+0000","last_became_active":"2024-03-06T07:20:21.859804+0000","last_became_peered":"2024-03-06T07:20:21.859804+0000","last_unstale":"2024-03-06T07:20:28.366176+0000","last_undegraded":"2024-03-06T07:20:28.366176+0000","last_fullsized":"2024-03-06T07:20:28.366176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:00.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201546+0000","last_change":"2024-03-06T07:20:21.859581+0000","last_active":"2024-03-06T07:20:28.201546+0000","last_peered":"2024-03-06T07:20:28.201546+0000","last_clean":"2024-03-06T07:20:28.201546+0000","last_became_active":"2024-03-06T07:20:21.859305+0000","last_became_peered":"2024-03-06T07:20:21.859305+0000","last_unstale":"2024-03-06T07:20:28.201546+0000","last_undegraded":"2024-03-06T07:20:28.201546+0000","last_fullsized":"2024-03-06T07:20:28.201546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:47.201278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4499,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":698343,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":111094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":344064,"data_stored":698399,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:20:39.304 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:20:39.305 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-06T07:20:39.305 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-06T07:20:39.305 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-06T07:20:39.306 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-06T07:20:39.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.438+0000 7f4b37781640 1 Processor -- start 2024-03-06T07:20:39.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.438+0000 7f4b37781640 1 -- start start 2024-03-06T07:20:39.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.440+0000 7f4b37781640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4b300c3970 0x7f4b300c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.440+0000 7f4b37781640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4b300c60b0 0x7f4b30152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.440+0000 7f4b37781640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b300c5e40 con 0x7f4b30152670 2024-03-06T07:20:39.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.440+0000 7f4b37781640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b300c67f0 con 0x7f4b300c60b0 2024-03-06T07:20:39.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.440+0000 7f4b37781640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b300c6a40 con 0x7f4b300c3970 2024-03-06T07:20:39.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.440+0000 7f4b34d1c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4b300c60b0 0x7f4b30152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.440+0000 7f4b34d1c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4b300c60b0 0x7f4b30152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40310/0 (socket says 172.21.15.1:40310) 2024-03-06T07:20:39.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.440+0000 7f4b34d1c640 1 -- 172.21.15.1:0/2341720688 learned_addr learned my addr 172.21.15.1:0/2341720688 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:39.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.440+0000 7f4b3551d640 1 --2- 172.21.15.1:0/2341720688 >> v2:172.21.15.98:3300/0 conn(0x7f4b300c3970 0x7f4b300c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.440+0000 7f4b34d1c640 1 -- 172.21.15.1:0/2341720688 >> v2:172.21.15.98:3300/0 conn(0x7f4b300c3970 msgr2=0x7f4b300c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:39.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.440+0000 7f4b34d1c640 1 --2- 172.21.15.1:0/2341720688 >> v2:172.21.15.98:3300/0 conn(0x7f4b300c3970 0x7f4b300c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.440+0000 7f4b34d1c640 1 -- 172.21.15.1:0/2341720688 >> v1:172.21.15.98:6789/0 conn(0x7f4b30152670 legacy=0x7f4b301553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.440+0000 7f4b34d1c640 1 -- 172.21.15.1:0/2341720688 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b300c6d20 con 0x7f4b300c60b0 2024-03-06T07:20:39.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.441+0000 7f4b34d1c640 1 --2- 172.21.15.1:0/2341720688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b300c60b0 0x7f4b30152130 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4b300c34c0 tx=0x7f4b2c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bebf9fb8b222a754 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.441+0000 7f4b23fff640 1 -- 172.21.15.1:0/2341720688 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c034070 con 0x7f4b300c60b0 2024-03-06T07:20:39.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.441+0000 7f4b23fff640 1 -- 172.21.15.1:0/2341720688 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2c030860 con 0x7f4b300c60b0 2024-03-06T07:20:39.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.441+0000 7f4b23fff640 1 -- 172.21.15.1:0/2341720688 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c030c30 con 0x7f4b300c60b0 2024-03-06T07:20:39.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.442+0000 7f4b37781640 1 -- 172.21.15.1:0/2341720688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b300c60b0 msgr2=0x7f4b30152130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.442+0000 7f4b37781640 1 --2- 172.21.15.1:0/2341720688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b300c60b0 0x7f4b30152130 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4b300c34c0 tx=0x7f4b2c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.442+0000 7f4b37781640 1 -- 172.21.15.1:0/2341720688 shutdown_connections 2024-03-06T07:20:39.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.442+0000 7f4b37781640 1 --2- 172.21.15.1:0/2341720688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b300c60b0 0x7f4b30152130 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.442+0000 7f4b37781640 1 --2- 172.21.15.1:0/2341720688 >> v2:172.21.15.98:3300/0 conn(0x7f4b300c3970 0x7f4b300c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.442+0000 7f4b37781640 1 -- 172.21.15.1:0/2341720688 >> 172.21.15.1:0/2341720688 conn(0x7f4b300c88f0 msgr2=0x7f4b300c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:39.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.442+0000 7f4b37781640 1 -- 172.21.15.1:0/2341720688 shutdown_connections 2024-03-06T07:20:39.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.442+0000 7f4b37781640 1 -- 172.21.15.1:0/2341720688 wait complete. 2024-03-06T07:20:39.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.443+0000 7f4b37781640 1 Processor -- start 2024-03-06T07:20:39.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.443+0000 7f4b37781640 1 -- start start 2024-03-06T07:20:39.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.443+0000 7f4b37781640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b300c3970 0x7f4b300acfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.443+0000 7f4b37781640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b300c60b0 0x7f4b30070a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.443+0000 7f4b37781640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b30152670 0x7f4b301ee240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.443+0000 7f4b37781640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b300c75c0 con 0x7f4b300c60b0 2024-03-06T07:20:39.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.443+0000 7f4b37781640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b300c72e0 con 0x7f4b300c3970 2024-03-06T07:20:39.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.443+0000 7f4b37781640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b300c3d50 con 0x7f4b30152670 2024-03-06T07:20:39.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.444+0000 7f4b34d1c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b300c60b0 0x7f4b30070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.444+0000 7f4b34d1c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b300c60b0 0x7f4b30070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40324/0 (socket says 172.21.15.1:40324) 2024-03-06T07:20:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.444+0000 7f4b34d1c640 1 -- 172.21.15.1:0/3588766823 learned_addr learned my addr 172.21.15.1:0/3588766823 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.444+0000 7f4b3551d640 1 --2- 172.21.15.1:0/3588766823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b300c3970 0x7f4b300acfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.444+0000 7f4b35d1e640 1 --2- 172.21.15.1:0/3588766823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b30152670 0x7f4b301ee240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.444+0000 7f4b34d1c640 1 -- 172.21.15.1:0/3588766823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b30152670 msgr2=0x7f4b301ee240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.444+0000 7f4b34d1c640 1 --2- 172.21.15.1:0/3588766823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b30152670 0x7f4b301ee240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.444+0000 7f4b34d1c640 1 -- 172.21.15.1:0/3588766823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b300c3970 msgr2=0x7f4b300acfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.444+0000 7f4b34d1c640 1 --2- 172.21.15.1:0/3588766823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b300c3970 0x7f4b300acfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.444+0000 7f4b34d1c640 1 -- 172.21.15.1:0/3588766823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b300ac550 con 0x7f4b300c60b0 2024-03-06T07:20:39.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.444+0000 7f4b35d1e640 1 --2- 172.21.15.1:0/3588766823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b30152670 0x7f4b301ee240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:39.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.444+0000 7f4b34d1c640 1 --2- 172.21.15.1:0/3588766823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b300c60b0 0x7f4b30070a30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c000c00 tx=0x7f4b2c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.444+0000 7f4b21ffb640 1 -- 172.21.15.1:0/3588766823 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c048070 con 0x7f4b300c60b0 2024-03-06T07:20:39.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.444+0000 7f4b21ffb640 1 -- 172.21.15.1:0/3588766823 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2c030660 con 0x7f4b300c60b0 2024-03-06T07:20:39.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.444+0000 7f4b37781640 1 -- 172.21.15.1:0/3588766823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b300adb30 con 0x7f4b300c60b0 2024-03-06T07:20:39.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.444+0000 7f4b21ffb640 1 -- 172.21.15.1:0/3588766823 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c034040 con 0x7f4b300c60b0 2024-03-06T07:20:39.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.444+0000 7f4b37781640 1 -- 172.21.15.1:0/3588766823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b300acaa0 con 0x7f4b300c60b0 2024-03-06T07:20:39.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.445+0000 7f4b37781640 1 -- 172.21.15.1:0/3588766823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b300686e0 con 0x7f4b300c60b0 2024-03-06T07:20:39.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.446+0000 7f4b21ffb640 1 -- 172.21.15.1:0/3588766823 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b2c04b070 con 0x7f4b300c60b0 2024-03-06T07:20:39.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.447+0000 7f4b21ffb640 1 --2- 172.21.15.1:0/3588766823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4b1406b4d0 0x7f4b1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.447+0000 7f4b21ffb640 1 -- 172.21.15.1:0/3588766823 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4b2c0ac530 con 0x7f4b300c60b0 2024-03-06T07:20:39.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.447+0000 7f4b3551d640 1 --2- 172.21.15.1:0/3588766823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4b1406b4d0 0x7f4b1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.448+0000 7f4b3551d640 1 --2- 172.21.15.1:0/3588766823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4b1406b4d0 0x7f4b1406d990 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4b24009b40 tx=0x7f4b24009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.455+0000 7f4b21ffb640 1 -- 172.21.15.1:0/3588766823 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b2c07aa10 con 0x7f4b300c60b0 2024-03-06T07:20:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.787+0000 7f4b37781640 1 -- 172.21.15.1:0/3588766823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f4b300c7740 con 0x7f4b300c60b0 2024-03-06T07:20:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.790+0000 7f4b21ffb640 1 -- 172.21.15.1:0/3588766823 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171784 (secure 0 0 0) 0x7f4b2c07eab0 con 0x7f4b300c60b0 2024-03-06T07:20:39.794 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.792+0000 7f4b37781640 1 -- 172.21.15.1:0/3588766823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4b1406b4d0 msgr2=0x7f4b1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.792+0000 7f4b37781640 1 --2- 172.21.15.1:0/3588766823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4b1406b4d0 0x7f4b1406d990 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4b24009b40 tx=0x7f4b24009190 comp rx=0 tx=0).stop 2024-03-06T07:20:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.792+0000 7f4b37781640 1 -- 172.21.15.1:0/3588766823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b300c60b0 msgr2=0x7f4b30070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.792+0000 7f4b37781640 1 --2- 172.21.15.1:0/3588766823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b300c60b0 0x7f4b30070a30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c000c00 tx=0x7f4b2c004730 comp rx=0 tx=0).stop 2024-03-06T07:20:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.792+0000 7f4b37781640 1 -- 172.21.15.1:0/3588766823 shutdown_connections 2024-03-06T07:20:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.792+0000 7f4b37781640 1 --2- 172.21.15.1:0/3588766823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4b1406b4d0 0x7f4b1406d990 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.792+0000 7f4b37781640 1 --2- 172.21.15.1:0/3588766823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4b30152670 0x7f4b301ee240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.792+0000 7f4b37781640 1 --2- 172.21.15.1:0/3588766823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b300c60b0 0x7f4b30070a30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.792+0000 7f4b37781640 1 --2- 172.21.15.1:0/3588766823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b300c3970 0x7f4b300acfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.792+0000 7f4b37781640 1 -- 172.21.15.1:0/3588766823 >> 172.21.15.1:0/3588766823 conn(0x7f4b300c88f0 msgr2=0x7f4b3006c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.792+0000 7f4b37781640 1 -- 172.21.15.1:0/3588766823 shutdown_connections 2024-03-06T07:20:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.792+0000 7f4b37781640 1 -- 172.21.15.1:0/3588766823 wait complete. 2024-03-06T07:20:39.805 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":5,"flags":0,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3776520052},{"type":"v1","addr":"172.21.15.98:6833","nonce":3776520052}]},"active_addr":"172.21.15.98:6833/3776520052","active_change":"2024-03-06T07:20:08.675502+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":3911179184}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":2206321675}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":2662701986}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":212940952}]}]} 2024-03-06T07:20:39.808 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-06T07:20:39.808 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-06T07:20:39.808 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.947+0000 7f2b6f5c9640 1 Processor -- start 2024-03-06T07:20:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.947+0000 7f2b6f5c9640 1 -- start start 2024-03-06T07:20:39.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.948+0000 7f2b6f5c9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2b68152690 0x7f2b68155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.948+0000 7f2b6f5c9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b681561e0 con 0x7f2b68156cd0 2024-03-06T07:20:39.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.948+0000 7f2b6f5c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b680c3300 con 0x7f2b680c6730 2024-03-06T07:20:39.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.948+0000 7f2b6f5c9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b680c35a0 con 0x7f2b68152690 2024-03-06T07:20:39.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.948+0000 7f2b6db66640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2b68156cd0 0x7f2b68157120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57400/0 (socket says 172.21.15.1:57400) 2024-03-06T07:20:39.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.948+0000 7f2b6db66640 1 -- 172.21.15.1:0/3321689106 learned_addr learned my addr 172.21.15.1:0/3321689106 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:39.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.948+0000 7f2b6d365640 1 --2- 172.21.15.1:0/3321689106 >> v2:172.21.15.98:3300/0 conn(0x7f2b68152690 0x7f2b68155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.948+0000 7f2b5ffff640 1 -- 172.21.15.1:0/3321689106 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366555977 0 0) 0x7f2b60001490 con 0x7f2b68156cd0 2024-03-06T07:20:39.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.948+0000 7f2b5ffff640 1 -- 172.21.15.1:0/3321689106 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b54003520 con 0x7f2b68156cd0 2024-03-06T07:20:39.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.948+0000 7f2b6d365640 1 -- 172.21.15.1:0/3321689106 >> v1:172.21.15.98:6789/0 conn(0x7f2b680c6730 legacy=0x7f2b680c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.948+0000 7f2b6d365640 1 -- 172.21.15.1:0/3321689106 >> v1:172.21.15.1:6789/0 conn(0x7f2b68156cd0 legacy=0x7f2b68157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.948+0000 7f2b6d365640 1 -- 172.21.15.1:0/3321689106 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b680c3880 con 0x7f2b68152690 2024-03-06T07:20:39.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.949+0000 7f2b6d365640 1 --2- 172.21.15.1:0/3321689106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b68152690 0x7f2b68155ca0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2b640097a0 tx=0x7f2b6402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f099d03ec20a3825 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.949+0000 7f2b5ffff640 1 -- 172.21.15.1:0/3321689106 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64035020 con 0x7f2b68152690 2024-03-06T07:20:39.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.949+0000 7f2b5ffff640 1 -- 172.21.15.1:0/3321689106 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b64031830 con 0x7f2b68152690 2024-03-06T07:20:39.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.949+0000 7f2b5ffff640 1 -- 172.21.15.1:0/3321689106 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64031be0 con 0x7f2b68152690 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.950+0000 7f2b6f5c9640 1 -- 172.21.15.1:0/3321689106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b68152690 msgr2=0x7f2b68155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.950+0000 7f2b6f5c9640 1 --2- 172.21.15.1:0/3321689106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b68152690 0x7f2b68155ca0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2b640097a0 tx=0x7f2b6402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.950+0000 7f2b6f5c9640 1 -- 172.21.15.1:0/3321689106 shutdown_connections 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.950+0000 7f2b6f5c9640 1 --2- 172.21.15.1:0/3321689106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b68152690 0x7f2b68155ca0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.950+0000 7f2b6f5c9640 1 -- 172.21.15.1:0/3321689106 >> 172.21.15.1:0/3321689106 conn(0x7f2b6806b410 msgr2=0x7f2b6806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.950+0000 7f2b6f5c9640 1 -- 172.21.15.1:0/3321689106 shutdown_connections 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.950+0000 7f2b6f5c9640 1 -- 172.21.15.1:0/3321689106 wait complete. 2024-03-06T07:20:39.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.951+0000 7f2b6f5c9640 1 Processor -- start 2024-03-06T07:20:39.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.951+0000 7f2b6f5c9640 1 -- start start 2024-03-06T07:20:39.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.952+0000 7f2b6f5c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b680c6730 0x7f2b6806d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.952+0000 7f2b6f5c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b68152690 0x7f2b6814a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.952+0000 7f2b6f5c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b68156cd0 0x7f2b6814a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.952+0000 7f2b6f5c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b680c5ee0 con 0x7f2b680c6730 2024-03-06T07:20:39.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.952+0000 7f2b6f5c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b681567b0 con 0x7f2b68156cd0 2024-03-06T07:20:39.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.952+0000 7f2b6f5c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b680c6060 con 0x7f2b68152690 2024-03-06T07:20:39.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.952+0000 7f2b6d365640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b680c6730 0x7f2b6806d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.952+0000 7f2b6d365640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b680c6730 0x7f2b6806d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40338/0 (socket says 172.21.15.1:40338) 2024-03-06T07:20:39.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.952+0000 7f2b6d365640 1 -- 172.21.15.1:0/1471624429 learned_addr learned my addr 172.21.15.1:0/1471624429 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:39.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.952+0000 7f2b6cb64640 1 --2- 172.21.15.1:0/1471624429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b68152690 0x7f2b6814a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.952+0000 7f2b6db66640 1 --2- 172.21.15.1:0/1471624429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b68156cd0 0x7f2b6814a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.952+0000 7f2b6d365640 1 -- 172.21.15.1:0/1471624429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b68152690 msgr2=0x7f2b6814a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.952+0000 7f2b6d365640 1 --2- 172.21.15.1:0/1471624429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b68152690 0x7f2b6814a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.952+0000 7f2b6d365640 1 -- 172.21.15.1:0/1471624429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b68156cd0 msgr2=0x7f2b6814a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.952+0000 7f2b6d365640 1 --2- 172.21.15.1:0/1471624429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b68156cd0 0x7f2b6814a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.952+0000 7f2b6d365640 1 -- 172.21.15.1:0/1471624429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b68155280 con 0x7f2b680c6730 2024-03-06T07:20:39.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.952+0000 7f2b6d365640 1 --2- 172.21.15.1:0/1471624429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b680c6730 0x7f2b6806d4f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2b64000c00 tx=0x7f2b640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.953+0000 7f2b5dffb640 1 -- 172.21.15.1:0/1471624429 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64035070 con 0x7f2b680c6730 2024-03-06T07:20:39.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.953+0000 7f2b5dffb640 1 -- 172.21.15.1:0/1471624429 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b64004230 con 0x7f2b680c6730 2024-03-06T07:20:39.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.953+0000 7f2b6f5c9640 1 -- 172.21.15.1:0/1471624429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b6814af40 con 0x7f2b680c6730 2024-03-06T07:20:39.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.953+0000 7f2b5dffb640 1 -- 172.21.15.1:0/1471624429 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6403a630 con 0x7f2b680c6730 2024-03-06T07:20:39.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.953+0000 7f2b6f5c9640 1 -- 172.21.15.1:0/1471624429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b6806da30 con 0x7f2b680c6730 2024-03-06T07:20:39.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.954+0000 7f2b6f5c9640 1 -- 172.21.15.1:0/1471624429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b68067fd0 con 0x7f2b680c6730 2024-03-06T07:20:39.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.954+0000 7f2b5dffb640 1 -- 172.21.15.1:0/1471624429 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b64038020 con 0x7f2b680c6730 2024-03-06T07:20:39.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.955+0000 7f2b5dffb640 1 --2- 172.21.15.1:0/1471624429 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b4c06b4d0 0x7f2b4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.955+0000 7f2b5dffb640 1 -- 172.21.15.1:0/1471624429 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2b640ac020 con 0x7f2b680c6730 2024-03-06T07:20:39.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.955+0000 7f2b6cb64640 1 --2- 172.21.15.1:0/1471624429 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b4c06b4d0 0x7f2b4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.955+0000 7f2b6cb64640 1 --2- 172.21.15.1:0/1471624429 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b4c06b4d0 0x7f2b4c06d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2b68151c80 tx=0x7f2b58009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:39.963+0000 7f2b5dffb640 1 -- 172.21.15.1:0/1471624429 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b64040070 con 0x7f2b680c6730 2024-03-06T07:20:40.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.150+0000 7f2b6f5c9640 1 -- 172.21.15.1:0/1471624429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b680c24b0 con 0x7f2b680c6730 2024-03-06T07:20:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.151+0000 7f2b5dffb640 1 -- 172.21.15.1:0/1471624429 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7f2b6407a5a0 con 0x7f2b680c6730 2024-03-06T07:20:40.153 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:40.154 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.153+0000 7f2b6f5c9640 1 -- 172.21.15.1:0/1471624429 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b4c06b4d0 msgr2=0x7f2b4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.153+0000 7f2b6f5c9640 1 --2- 172.21.15.1:0/1471624429 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b4c06b4d0 0x7f2b4c06d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2b68151c80 tx=0x7f2b58009040 comp rx=0 tx=0).stop 2024-03-06T07:20:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.153+0000 7f2b6f5c9640 1 -- 172.21.15.1:0/1471624429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b680c6730 msgr2=0x7f2b6806d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.153+0000 7f2b6f5c9640 1 --2- 172.21.15.1:0/1471624429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b680c6730 0x7f2b6806d4f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2b64000c00 tx=0x7f2b640046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.153+0000 7f2b6f5c9640 1 -- 172.21.15.1:0/1471624429 shutdown_connections 2024-03-06T07:20:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.153+0000 7f2b6f5c9640 1 --2- 172.21.15.1:0/1471624429 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b4c06b4d0 0x7f2b4c06d990 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.153+0000 7f2b6f5c9640 1 --2- 172.21.15.1:0/1471624429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b68156cd0 0x7f2b6814a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.153+0000 7f2b6f5c9640 1 --2- 172.21.15.1:0/1471624429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b68152690 0x7f2b6814a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.153+0000 7f2b6f5c9640 1 --2- 172.21.15.1:0/1471624429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b680c6730 0x7f2b6806d4f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.153+0000 7f2b6f5c9640 1 -- 172.21.15.1:0/1471624429 >> 172.21.15.1:0/1471624429 conn(0x7f2b6806b410 msgr2=0x7f2b6806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.153+0000 7f2b6f5c9640 1 -- 172.21.15.1:0/1471624429 shutdown_connections 2024-03-06T07:20:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.153+0000 7f2b6f5c9640 1 -- 172.21.15.1:0/1471624429 wait complete. 2024-03-06T07:20:40.166 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-06T07:20:40.166 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:40.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.265+0000 7f2c5eedf640 1 Processor -- start 2024-03-06T07:20:40.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.265+0000 7f2c5eedf640 1 -- start start 2024-03-06T07:20:40.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.266+0000 7f2c5eedf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2c580c4560 0x7f2c58155c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.267+0000 7f2c5eedf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c580c39c0 0x7f2c581561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.267+0000 7f2c5eedf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c580c7420 con 0x7f2c580c4940 2024-03-06T07:20:40.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.267+0000 7f2c5eedf640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c581566f0 con 0x7f2c580c4560 2024-03-06T07:20:40.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.267+0000 7f2c5eedf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c580ab760 con 0x7f2c580c39c0 2024-03-06T07:20:40.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.267+0000 7f2c57fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c580c39c0 0x7f2c581561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.267+0000 7f2c5cc7b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2c580c4560 0x7f2c58155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.267+0000 7f2c57fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c580c39c0 0x7f2c581561b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:41252/0 (socket says 172.21.15.1:41252) 2024-03-06T07:20:40.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.267+0000 7f2c57fff640 1 -- 172.21.15.1:0/1871481753 learned_addr learned my addr 172.21.15.1:0/1871481753 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:40.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.267+0000 7f2c5cc7b640 1 -- 172.21.15.1:0/1871481753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c580c39c0 msgr2=0x7f2c581561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:40.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.267+0000 7f2c5cc7b640 1 --2- 172.21.15.1:0/1871481753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c580c39c0 0x7f2c581561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.267+0000 7f2c5cc7b640 1 -- 172.21.15.1:0/1871481753 >> v1:172.21.15.98:6789/0 conn(0x7f2c580c4940 legacy=0x7f2c580ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.267+0000 7f2c5cc7b640 1 -- 172.21.15.1:0/1871481753 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c580c76c0 con 0x7f2c580c4560 2024-03-06T07:20:40.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.268+0000 7f2c5cc7b640 1 --2- 172.21.15.1:0/1871481753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c580c4560 0x7f2c58155c70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c0097a0 tx=0x7f2c4c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f54fc3e0ba5475c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.268+0000 7f2c577fe640 1 -- 172.21.15.1:0/1871481753 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c035020 con 0x7f2c580c4560 2024-03-06T07:20:40.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.268+0000 7f2c577fe640 1 -- 172.21.15.1:0/1871481753 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4c031830 con 0x7f2c580c4560 2024-03-06T07:20:40.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.268+0000 7f2c577fe640 1 -- 172.21.15.1:0/1871481753 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c031c00 con 0x7f2c580c4560 2024-03-06T07:20:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.269+0000 7f2c5eedf640 1 -- 172.21.15.1:0/1871481753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c580c4560 msgr2=0x7f2c58155c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.269+0000 7f2c5eedf640 1 --2- 172.21.15.1:0/1871481753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c580c4560 0x7f2c58155c70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c0097a0 tx=0x7f2c4c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.269+0000 7f2c5eedf640 1 -- 172.21.15.1:0/1871481753 shutdown_connections 2024-03-06T07:20:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.269+0000 7f2c5eedf640 1 --2- 172.21.15.1:0/1871481753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c580c39c0 0x7f2c581561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.269+0000 7f2c5eedf640 1 --2- 172.21.15.1:0/1871481753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c580c4560 0x7f2c58155c70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.269+0000 7f2c5eedf640 1 -- 172.21.15.1:0/1871481753 >> 172.21.15.1:0/1871481753 conn(0x7f2c580c7a50 msgr2=0x7f2c580c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.269+0000 7f2c5eedf640 1 -- 172.21.15.1:0/1871481753 shutdown_connections 2024-03-06T07:20:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.269+0000 7f2c5eedf640 1 -- 172.21.15.1:0/1871481753 wait complete. 2024-03-06T07:20:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.270+0000 7f2c5eedf640 1 Processor -- start 2024-03-06T07:20:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.270+0000 7f2c5eedf640 1 -- start start 2024-03-06T07:20:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c5eedf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c580c39c0 0x7f2c581e8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c5eedf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c580c4560 0x7f2c580c0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c5eedf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c580c4940 0x7f2c5806b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c5eedf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5806e6d0 con 0x7f2c580c4940 2024-03-06T07:20:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c5eedf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c580abd00 con 0x7f2c580c39c0 2024-03-06T07:20:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c5eedf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5806e850 con 0x7f2c580c4560 2024-03-06T07:20:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c5d47c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c580c4940 0x7f2c5806b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c5d47c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c580c4940 0x7f2c5806b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40348/0 (socket says 172.21.15.1:40348) 2024-03-06T07:20:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c5d47c640 1 -- 172.21.15.1:0/3009993346 learned_addr learned my addr 172.21.15.1:0/3009993346 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c5cc7b640 1 --2- 172.21.15.1:0/3009993346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c580c39c0 0x7f2c581e8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c57fff640 1 --2- 172.21.15.1:0/3009993346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c580c4560 0x7f2c580c0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c5d47c640 1 -- 172.21.15.1:0/3009993346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c580c4560 msgr2=0x7f2c580c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c5d47c640 1 --2- 172.21.15.1:0/3009993346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c580c4560 0x7f2c580c0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c5d47c640 1 -- 172.21.15.1:0/3009993346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c580c39c0 msgr2=0x7f2c581e8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c5d47c640 1 --2- 172.21.15.1:0/3009993346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c580c39c0 0x7f2c581e8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c5d47c640 1 -- 172.21.15.1:0/3009993346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c580bff20 con 0x7f2c580c4940 2024-03-06T07:20:40.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c57fff640 1 --2- 172.21.15.1:0/3009993346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c580c4560 0x7f2c580c0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:40.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.271+0000 7f2c5d47c640 1 --2- 172.21.15.1:0/3009993346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c580c4940 0x7f2c5806b260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2c5000c6b0 tx=0x7f2c5000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.272+0000 7f2c557fa640 1 -- 172.21.15.1:0/3009993346 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c50003b70 con 0x7f2c580c4940 2024-03-06T07:20:40.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.272+0000 7f2c557fa640 1 -- 172.21.15.1:0/3009993346 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c50003d10 con 0x7f2c580c4940 2024-03-06T07:20:40.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.272+0000 7f2c5eedf640 1 -- 172.21.15.1:0/3009993346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c580c0200 con 0x7f2c580c4940 2024-03-06T07:20:40.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.272+0000 7f2c557fa640 1 -- 172.21.15.1:0/3009993346 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c50005310 con 0x7f2c580c4940 2024-03-06T07:20:40.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.272+0000 7f2c5eedf640 1 -- 172.21.15.1:0/3009993346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c580c07c0 con 0x7f2c580c4940 2024-03-06T07:20:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.273+0000 7f2c5eedf640 1 -- 172.21.15.1:0/3009993346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c580686e0 con 0x7f2c580c4940 2024-03-06T07:20:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.273+0000 7f2c557fa640 1 -- 172.21.15.1:0/3009993346 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c500054b0 con 0x7f2c580c4940 2024-03-06T07:20:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.274+0000 7f2c557fa640 1 --2- 172.21.15.1:0/3009993346 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c3c06b4d0 0x7f2c3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.274+0000 7f2c557fa640 1 -- 172.21.15.1:0/3009993346 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2c5008a150 con 0x7f2c580c4940 2024-03-06T07:20:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.274+0000 7f2c5cc7b640 1 --2- 172.21.15.1:0/3009993346 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c3c06b4d0 0x7f2c3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.275+0000 7f2c5cc7b640 1 --2- 172.21.15.1:0/3009993346 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c3c06b4d0 0x7f2c3c06d990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c036000 tx=0x7f2c4c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.278+0000 7f2c557fa640 1 -- 172.21.15.1:0/3009993346 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c500586d0 con 0x7f2c580c4940 2024-03-06T07:20:40.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.459+0000 7f2c5eedf640 1 -- 172.21.15.1:0/3009993346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c580c7ed0 con 0x7f2c580c4940 2024-03-06T07:20:40.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.460+0000 7f2c557fa640 1 -- 172.21.15.1:0/3009993346 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7f2c5005c770 con 0x7f2c580c4940 2024-03-06T07:20:40.462 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:40.462 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:40.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.461+0000 7f2c5eedf640 1 -- 172.21.15.1:0/3009993346 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c3c06b4d0 msgr2=0x7f2c3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.461+0000 7f2c5eedf640 1 --2- 172.21.15.1:0/3009993346 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c3c06b4d0 0x7f2c3c06d990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c036000 tx=0x7f2c4c008030 comp rx=0 tx=0).stop 2024-03-06T07:20:40.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.461+0000 7f2c5eedf640 1 -- 172.21.15.1:0/3009993346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c580c4940 msgr2=0x7f2c5806b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.461+0000 7f2c5eedf640 1 --2- 172.21.15.1:0/3009993346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c580c4940 0x7f2c5806b260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2c5000c6b0 tx=0x7f2c5000cb70 comp rx=0 tx=0).stop 2024-03-06T07:20:40.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.462+0000 7f2c5eedf640 1 -- 172.21.15.1:0/3009993346 shutdown_connections 2024-03-06T07:20:40.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.462+0000 7f2c5eedf640 1 --2- 172.21.15.1:0/3009993346 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c3c06b4d0 0x7f2c3c06d990 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.462+0000 7f2c5eedf640 1 --2- 172.21.15.1:0/3009993346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c580c4940 0x7f2c5806b260 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.462+0000 7f2c5eedf640 1 --2- 172.21.15.1:0/3009993346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c580c4560 0x7f2c580c0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.462+0000 7f2c5eedf640 1 --2- 172.21.15.1:0/3009993346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c580c39c0 0x7f2c581e8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.462+0000 7f2c5eedf640 1 -- 172.21.15.1:0/3009993346 >> 172.21.15.1:0/3009993346 conn(0x7f2c580c7a50 msgr2=0x7f2c58155a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.462+0000 7f2c5eedf640 1 -- 172.21.15.1:0/3009993346 shutdown_connections 2024-03-06T07:20:40.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.462+0000 7f2c5eedf640 1 -- 172.21.15.1:0/3009993346 wait complete. 2024-03-06T07:20:40.477 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-06T07:20:40.477 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-06T07:20:40.478 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-06T07:20:40.478 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-06T07:20:40.479 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-06T07:20:40.479 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-06T07:20:40.480 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-06T07:20:40.480 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-06T07:20:40.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.651+0000 7f14d865c640 1 Processor -- start 2024-03-06T07:20:40.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.651+0000 7f14d865c640 1 -- start start 2024-03-06T07:20:40.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.653+0000 7f14d865c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f14d014e7d0 0x7f14d006ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.653+0000 7f14d865c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f14d00c0b90 0x7f14d006aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.653+0000 7f14d865c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d0151ff0 con 0x7f14d006b530 2024-03-06T07:20:40.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.653+0000 7f14d865c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d00c1180 con 0x7f14d014e7d0 2024-03-06T07:20:40.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.653+0000 7f14d865c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d00c6350 con 0x7f14d00c0b90 2024-03-06T07:20:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.653+0000 7fc9b23a2640 1 Processor -- start 2024-03-06T07:20:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.653+0000 7fa931099640 1 Processor -- start 2024-03-06T07:20:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.653+0000 7f14d6bf9640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f14d006b530 0x7f14d0154a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57408/0 (socket says 172.21.15.1:57408) 2024-03-06T07:20:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.653+0000 7f14d6bf9640 1 -- 172.21.15.1:0/2688823823 learned_addr learned my addr 172.21.15.1:0/2688823823 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.653+0000 7f14d5bf7640 1 --2- 172.21.15.1:0/2688823823 >> v2:172.21.15.98:3300/0 conn(0x7f14d00c0b90 0x7f14d006aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.653+0000 7f14d63f8640 1 --2- 172.21.15.1:0/2688823823 >> v2:172.21.15.1:3300/0 conn(0x7f14d014e7d0 0x7f14d006ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.654+0000 7fc9b23a2640 1 -- start start 2024-03-06T07:20:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.654+0000 7f909a4a0640 1 Processor -- start 2024-03-06T07:20:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.654+0000 7f909a4a0640 1 -- start start 2024-03-06T07:20:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.654+0000 7f14d63f8640 1 -- 172.21.15.1:0/2688823823 >> v2:172.21.15.98:3300/0 conn(0x7f14d00c0b90 msgr2=0x7f14d006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.654+0000 7f14d63f8640 1 --2- 172.21.15.1:0/2688823823 >> v2:172.21.15.98:3300/0 conn(0x7f14d00c0b90 0x7f14d006aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.654+0000 7f14d63f8640 1 -- 172.21.15.1:0/2688823823 >> v1:172.21.15.1:6789/0 conn(0x7f14d006b530 legacy=0x7f14d0154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.654+0000 7f14d63f8640 1 -- 172.21.15.1:0/2688823823 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d006d4f0 con 0x7f14d014e7d0 2024-03-06T07:20:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.654+0000 7f14d53f6640 1 -- 172.21.15.1:0/2688823823 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125491105 0 0) 0x7f14d0151ff0 con 0x7f14d006b530 2024-03-06T07:20:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.654+0000 7f14d5bf7640 1 --2- 172.21.15.1:0/2688823823 >> v2:172.21.15.98:3300/0 conn(0x7f14d00c0b90 0x7f14d006aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.653+0000 7fa931099640 1 -- start start 2024-03-06T07:20:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.654+0000 7fa931099640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa92c06edd0 0x7fa92c0c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fa931099640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa92c0c0c20 0x7fa92c0c5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fa931099640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92c0c1180 con 0x7fa92c159540 2024-03-06T07:20:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fa931099640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa92c0c5570 con 0x7fa92c0c0c20 2024-03-06T07:20:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fa931099640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa92c0c4680 con 0x7fa92c06edd0 2024-03-06T07:20:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fa92b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa92c0c0c20 0x7fa92c0c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fa930898640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa92c159540 0x7fa92c06b290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57410/0 (socket says 172.21.15.1:57410) 2024-03-06T07:20:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fa92b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa92c0c0c20 0x7fa92c0c5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40376/0 (socket says 172.21.15.1:40376) 2024-03-06T07:20:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fa930898640 1 -- 172.21.15.1:0/2806679151 learned_addr learned my addr 172.21.15.1:0/2806679151 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7f909a4a0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f908c04b110 0x7f908c006ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7f909a4a0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908c0d9c60 con 0x7f908c0d3b40 2024-03-06T07:20:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7f9098c9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f908c04b110 0x7f908c006ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7f909a4a0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908c0d9f40 con 0x7f908c007430 2024-03-06T07:20:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7f909a4a0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908c0da220 con 0x7f908c04b110 2024-03-06T07:20:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7f9098c9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f908c04b110 0x7f908c006ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40402/0 (socket says 172.21.15.1:40402) 2024-03-06T07:20:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7f9098c9d640 1 -- 172.21.15.1:0/802599473 learned_addr learned my addr 172.21.15.1:0/802599473 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fa92bfff640 1 --2- 172.21.15.1:0/2806679151 >> v2:172.21.15.98:3300/0 conn(0x7fa92c06edd0 0x7fa92c0c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fc9b23a2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc9ac0a60b0 0x7fc9ac0c6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fc9b23a2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc9ac0c7310 0x7fc9ac151130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fc9b23a2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9ac154c90 0x7fc9ac14e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fc9b23a2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9ac06f450 con 0x7fc9ac0c7310 2024-03-06T07:20:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fc9b23a2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9ac0c6560 con 0x7fc9ac0a60b0 2024-03-06T07:20:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fc9b23a2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac1554c0 con 0x7fc9ac154c90 2024-03-06T07:20:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fc9b0b9f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc9ac0c7310 0x7fc9ac151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fc9b13a0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc9ac0a60b0 0x7fc9ac0c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fc9b13a0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc9ac0a60b0 0x7fc9ac0c6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:44412/0 (socket says 172.21.15.1:44412) 2024-03-06T07:20:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fc9b13a0640 1 -- 172.21.15.1:0/374655084 learned_addr learned my addr 172.21.15.1:0/374655084 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fc9b1ba1640 1 --2- 172.21.15.1:0/374655084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9ac154c90 0x7fc9ac14e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fa92b7fe640 1 -- 172.21.15.1:0/2806679151 >> v2:172.21.15.98:3300/0 conn(0x7fa92c06edd0 msgr2=0x7fa92c0c4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fa92b7fe640 1 --2- 172.21.15.1:0/2806679151 >> v2:172.21.15.98:3300/0 conn(0x7fa92c06edd0 0x7fa92c0c4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fa92b7fe640 1 -- 172.21.15.1:0/2806679151 >> v1:172.21.15.1:6789/0 conn(0x7fa92c159540 legacy=0x7fa92c06b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.655+0000 7fa92b7fe640 1 -- 172.21.15.1:0/2806679151 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92c159a40 con 0x7fa92c0c0c20 2024-03-06T07:20:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.656+0000 7f9098c9d640 1 -- 172.21.15.1:0/802599473 >> v1:172.21.15.98:6789/0 conn(0x7f908c007430 legacy=0x7f908c0d6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.656+0000 7f9098c9d640 1 -- 172.21.15.1:0/802599473 >> v1:172.21.15.1:6789/0 conn(0x7f908c0d3b40 legacy=0x7f908c046b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.656+0000 7fc9b13a0640 1 -- 172.21.15.1:0/374655084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9ac154c90 msgr2=0x7fc9ac14e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.656+0000 7fc9b13a0640 1 --2- 172.21.15.1:0/374655084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9ac154c90 0x7fc9ac14e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.656+0000 7fc9b13a0640 1 -- 172.21.15.1:0/374655084 >> v2:172.21.15.1:3300/0 conn(0x7fc9ac0c7310 msgr2=0x7fc9ac151130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.656+0000 7fc9b13a0640 1 --2- 172.21.15.1:0/374655084 >> v2:172.21.15.1:3300/0 conn(0x7fc9ac0c7310 0x7fc9ac151130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.656+0000 7fc9b13a0640 1 -- 172.21.15.1:0/374655084 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ac06d460 con 0x7fc9ac0a60b0 2024-03-06T07:20:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.656+0000 7f9098c9d640 1 -- 172.21.15.1:0/802599473 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c0da500 con 0x7f908c04b110 2024-03-06T07:20:40.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.657+0000 7fa92b7fe640 1 --2- 172.21.15.1:0/2806679151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c0c0c20 0x7fa92c0c5030 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa92c06fa80 tx=0x7fa92402bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4f3df4119c0992f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.656+0000 7f14d63f8640 1 --2- 172.21.15.1:0/2688823823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d014e7d0 0x7f14d006ef40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f14d006ecc0 tx=0x7f14c802bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91c58ce03a33fb77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.657+0000 7f14d53f6640 1 -- 172.21.15.1:0/2688823823 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c8039070 con 0x7f14d014e7d0 2024-03-06T07:20:40.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.657+0000 7f14d53f6640 1 -- 172.21.15.1:0/2688823823 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c802cb50 con 0x7f14d014e7d0 2024-03-06T07:20:40.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.657+0000 7fa92affd640 1 -- 172.21.15.1:0/2806679151 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa924039070 con 0x7fa92c0c0c20 2024-03-06T07:20:40.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.657+0000 7fa92affd640 1 -- 172.21.15.1:0/2806679151 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa92402cb50 con 0x7fa92c0c0c20 2024-03-06T07:20:40.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.657+0000 7fa92affd640 1 -- 172.21.15.1:0/2806679151 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa924034440 con 0x7fa92c0c0c20 2024-03-06T07:20:40.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.657+0000 7f9098c9d640 1 --2- 172.21.15.1:0/802599473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f908c04b110 0x7f908c006ef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f90840091e0 tx=0x7f908402cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192fc160b7c2e487 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7fa931099640 1 -- 172.21.15.1:0/2806679151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c0c0c20 msgr2=0x7fa92c0c5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7fa931099640 1 --2- 172.21.15.1:0/2806679151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c0c0c20 0x7fa92c0c5030 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa92c06fa80 tx=0x7fa92402bda0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f908bfff640 1 -- 172.21.15.1:0/802599473 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908403a070 con 0x7f908c04b110 2024-03-06T07:20:40.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f908bfff640 1 -- 172.21.15.1:0/802599473 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f908402d9f0 con 0x7f908c04b110 2024-03-06T07:20:40.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f908bfff640 1 -- 172.21.15.1:0/802599473 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908402ddc0 con 0x7f908c04b110 2024-03-06T07:20:40.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f14d865c640 1 -- 172.21.15.1:0/2688823823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d014e7d0 msgr2=0x7f14d006ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f14d865c640 1 --2- 172.21.15.1:0/2688823823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d014e7d0 0x7f14d006ef40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f14d006ecc0 tx=0x7f14c802bda0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f14d53f6640 1 -- 172.21.15.1:0/2688823823 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c8034440 con 0x7f14d014e7d0 2024-03-06T07:20:40.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7fa931099640 1 -- 172.21.15.1:0/2806679151 shutdown_connections 2024-03-06T07:20:40.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7fa931099640 1 --2- 172.21.15.1:0/2806679151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c0c0c20 0x7fa92c0c5030 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7fa931099640 1 --2- 172.21.15.1:0/2806679151 >> v2:172.21.15.98:3300/0 conn(0x7fa92c06edd0 0x7fa92c0c4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7fa931099640 1 -- 172.21.15.1:0/2806679151 >> 172.21.15.1:0/2806679151 conn(0x7fa92c0c7290 msgr2=0x7fa92c0a8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.659+0000 7fc9b13a0640 1 --2- 172.21.15.1:0/374655084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9ac0a60b0 0x7fc9ac0c6020 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac0c7790 tx=0x7fc99c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddc50c7bf05c009 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.659+0000 7fc9a3fff640 1 -- 172.21.15.1:0/374655084 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c035020 con 0x7fc9ac0a60b0 2024-03-06T07:20:40.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.659+0000 7fc9a3fff640 1 -- 172.21.15.1:0/374655084 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99c031640 con 0x7fc9ac0a60b0 2024-03-06T07:20:40.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.659+0000 7fc9a3fff640 1 -- 172.21.15.1:0/374655084 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c02d040 con 0x7fc9ac0a60b0 2024-03-06T07:20:40.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.659+0000 7f909a4a0640 1 -- 172.21.15.1:0/802599473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f908c04b110 msgr2=0x7f908c006ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.659+0000 7f909a4a0640 1 --2- 172.21.15.1:0/802599473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f908c04b110 0x7f908c006ef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f90840091e0 tx=0x7f908402cc40 comp rx=0 tx=0).stop 2024-03-06T07:20:40.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.659+0000 7f909a4a0640 1 -- 172.21.15.1:0/802599473 shutdown_connections 2024-03-06T07:20:40.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.659+0000 7f909a4a0640 1 --2- 172.21.15.1:0/802599473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f908c04b110 0x7f908c006ef0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.659+0000 7f909a4a0640 1 -- 172.21.15.1:0/802599473 >> 172.21.15.1:0/802599473 conn(0x7f908c048250 msgr2=0x7f908c048660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f14d865c640 1 -- 172.21.15.1:0/2688823823 shutdown_connections 2024-03-06T07:20:40.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f14d865c640 1 --2- 172.21.15.1:0/2688823823 >> v2:172.21.15.98:3300/0 conn(0x7f14d00c0b90 0x7f14d006aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f14d865c640 1 --2- 172.21.15.1:0/2688823823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d014e7d0 0x7f14d006ef40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f14d865c640 1 -- 172.21.15.1:0/2688823823 >> 172.21.15.1:0/2688823823 conn(0x7f14d00c6e60 msgr2=0x7f14d00a8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.660+0000 7f73b2f7a640 1 Processor -- start 2024-03-06T07:20:40.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.660+0000 7f73b2f7a640 1 -- start start 2024-03-06T07:20:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7fa931099640 1 -- 172.21.15.1:0/2806679151 shutdown_connections 2024-03-06T07:20:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7fa931099640 1 -- 172.21.15.1:0/2806679151 wait complete. 2024-03-06T07:20:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.657+0000 7f5154ed0640 1 Processor -- start 2024-03-06T07:20:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.657+0000 7f5154ed0640 1 -- start start 2024-03-06T07:20:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f5154ed0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5148046150 0x7f5148046530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f5154ed0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51480f2260 0x7f51480f89e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f5154ed0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f514800b070 0x7f51480f9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f5154ed0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5148047420 con 0x7f51480f2260 2024-03-06T07:20:40.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f5154ed0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51480f7f90 con 0x7f5148046150 2024-03-06T07:20:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f5154ed0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51480f8550 con 0x7f514800b070 2024-03-06T07:20:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f514effd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51480f2260 0x7f51480f89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.659+0000 7f514effd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51480f2260 0x7f51480f89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40412/0 (socket says 172.21.15.1:40412) 2024-03-06T07:20:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.659+0000 7f514effd640 1 -- 172.21.15.1:0/2840596967 learned_addr learned my addr 172.21.15.1:0/2840596967 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.659+0000 7f514effd640 1 -- 172.21.15.1:0/2840596967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f514800b070 msgr2=0x7f51480f9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.659+0000 7f514f7fe640 1 --2- 172.21.15.1:0/2840596967 >> v2:172.21.15.98:3300/0 conn(0x7f5148046150 0x7f5148046530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.658+0000 7f514ffff640 1 --2- 172.21.15.1:0/2840596967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f514800b070 0x7f51480f9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.659+0000 7f514effd640 1 --2- 172.21.15.1:0/2840596967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f514800b070 0x7f51480f9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.659+0000 7f514effd640 1 -- 172.21.15.1:0/2840596967 >> v2:172.21.15.98:3300/0 conn(0x7f5148046150 msgr2=0x7f5148046530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.659+0000 7f514effd640 1 --2- 172.21.15.1:0/2840596967 >> v2:172.21.15.98:3300/0 conn(0x7f5148046150 0x7f5148046530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.659+0000 7f514effd640 1 -- 172.21.15.1:0/2840596967 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514803f130 con 0x7f51480f2260 2024-03-06T07:20:40.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.660+0000 7f514ffff640 1 --2- 172.21.15.1:0/2840596967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f514800b070 0x7f51480f9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:40.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.660+0000 7fc9b23a2640 1 -- 172.21.15.1:0/374655084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9ac0a60b0 msgr2=0x7fc9ac0c6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.660+0000 7fc9b23a2640 1 --2- 172.21.15.1:0/374655084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9ac0a60b0 0x7fc9ac0c6020 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac0c7790 tx=0x7fc99c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.660+0000 7fa931099640 1 Processor -- start 2024-03-06T07:20:40.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.660+0000 7f514f7fe640 1 --2- 172.21.15.1:0/2840596967 >> v2:172.21.15.98:3300/0 conn(0x7f5148046150 0x7f5148046530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.660+0000 7fc9b23a2640 1 -- 172.21.15.1:0/374655084 shutdown_connections 2024-03-06T07:20:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.660+0000 7fc9b23a2640 1 --2- 172.21.15.1:0/374655084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9ac154c90 0x7fc9ac14e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.660+0000 7fc9b23a2640 1 --2- 172.21.15.1:0/374655084 >> v2:172.21.15.1:3300/0 conn(0x7fc9ac0c7310 0x7fc9ac151130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.660+0000 7fc9b23a2640 1 --2- 172.21.15.1:0/374655084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9ac0a60b0 0x7fc9ac0c6020 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.660+0000 7fc9b23a2640 1 -- 172.21.15.1:0/374655084 >> 172.21.15.1:0/374655084 conn(0x7fc9ac0c6e60 msgr2=0x7fc9ac0a8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.660+0000 7f909a4a0640 1 -- 172.21.15.1:0/802599473 shutdown_connections 2024-03-06T07:20:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.660+0000 7f909a4a0640 1 -- 172.21.15.1:0/802599473 wait complete. 2024-03-06T07:20:40.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.660+0000 7f14d865c640 1 -- 172.21.15.1:0/2688823823 shutdown_connections 2024-03-06T07:20:40.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.660+0000 7f14d865c640 1 -- 172.21.15.1:0/2688823823 wait complete. 2024-03-06T07:20:40.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.660+0000 7f514effd640 1 --2- 172.21.15.1:0/2840596967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51480f2260 0x7f51480f89e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f51480481b0 tx=0x7f513c02fda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e83828a1dd309ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.661+0000 7f514e7fc640 1 -- 172.21.15.1:0/2840596967 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c036070 con 0x7f51480f2260 2024-03-06T07:20:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.661+0000 7f514e7fc640 1 -- 172.21.15.1:0/2840596967 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513c030b50 con 0x7f51480f2260 2024-03-06T07:20:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.661+0000 7f73b2f7a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f73ac0c4560 0x7f73ac155c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.661+0000 7f73b2f7a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f73ac0c4940 0x7f73ac1561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7f909a4a0640 1 Processor -- start 2024-03-06T07:20:40.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7f909a4a0640 1 -- start start 2024-03-06T07:20:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7f73b0d16640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f73ac0c4560 0x7f73ac155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.661+0000 7fa931099640 1 -- start start 2024-03-06T07:20:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7fa931099640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa92c06edd0 0x7fa92c15cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7fa931099640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa92c0c0c20 0x7fa92c15d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7fa931099640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c159540 0x7fa92c156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7f73b2f7a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac0c7420 con 0x7f73ac0c39c0 2024-03-06T07:20:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7f73abfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f73ac0c39c0 0x7f73ac0ac220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57416/0 (socket says 172.21.15.1:57416) 2024-03-06T07:20:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7f73abfff640 1 -- 172.21.15.1:0/1813508386 learned_addr learned my addr 172.21.15.1:0/1813508386 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7f73b1517640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f73ac0c4940 0x7f73ac1561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7f73b0d16640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f73ac0c4560 0x7f73ac155c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:44438/0 (socket says 172.21.15.1:44438) 2024-03-06T07:20:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7f73b2f7a640 1 -- 172.21.15.1:0/1813508386 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ac1566f0 con 0x7f73ac0c4940 2024-03-06T07:20:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7f73b2f7a640 1 -- 172.21.15.1:0/1813508386 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ac0ab760 con 0x7f73ac0c4560 2024-03-06T07:20:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7f14d865c640 1 Processor -- start 2024-03-06T07:20:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7fa92b7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa92c0c0c20 0x7fa92c15d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7fa92b7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa92c0c0c20 0x7fa92c15d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:41298/0 (socket says 172.21.15.1:41298) 2024-03-06T07:20:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7fa930898640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c159540 0x7fa92c156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7fa930898640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c159540 0x7fa92c156090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40428/0 (socket says 172.21.15.1:40428) 2024-03-06T07:20:40.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7fa92bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa92c06edd0 0x7fa92c15cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7f73ab7fe640 1 -- 172.21.15.1:0/1813508386 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881519821 0 0) 0x7f73ac0c7420 con 0x7f73ac0c39c0 2024-03-06T07:20:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7fa92bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa92c06edd0 0x7fa92c15cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:44450/0 (socket says 172.21.15.1:44450) 2024-03-06T07:20:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7f73ab7fe640 1 -- 172.21.15.1:0/1813508386 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f739c003520 con 0x7f73ac0c39c0 2024-03-06T07:20:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7fa92b7fe640 1 -- 172.21.15.1:0/2361699789 learned_addr learned my addr 172.21.15.1:0/2361699789 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:40.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7fa931099640 1 -- 172.21.15.1:0/2361699789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c06f1b0 con 0x7fa92c159540 2024-03-06T07:20:40.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f909a4a0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f908c007430 0x7f908c0c9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f909a4a0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f908c04b110 0x7f908c0d2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f909a4a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f908c0d3b40 0x7f908c1745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7fa931099640 1 -- 172.21.15.1:0/2361699789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c15a2e0 con 0x7fa92c06edd0 2024-03-06T07:20:40.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f909a4a0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0d3f20 con 0x7f908c007430 2024-03-06T07:20:40.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f909a4a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0d9930 con 0x7f908c0d3b40 2024-03-06T07:20:40.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f909a4a0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0304f0 con 0x7f908c04b110 2024-03-06T07:20:40.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7fa931099640 1 -- 172.21.15.1:0/2361699789 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c06f330 con 0x7fa92c0c0c20 2024-03-06T07:20:40.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f9098c9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f908c04b110 0x7f908c0d2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f9098c9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f908c04b110 0x7f908c0d2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:41314/0 (socket says 172.21.15.1:41314) 2024-03-06T07:20:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f9098c9d640 1 -- 172.21.15.1:0/4108433486 learned_addr learned my addr 172.21.15.1:0/4108433486 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f9099c9f640 1 --2- 172.21.15.1:0/4108433486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f908c0d3b40 0x7f908c1745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7f14d865c640 1 -- start start 2024-03-06T07:20:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f14d865c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14d006b530 0x7f14d00a77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f14d865c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14d00c0b90 0x7f14d00a9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f14d865c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d014e7d0 0x7f14d00aa000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f14d865c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d006d790 con 0x7f14d014e7d0 2024-03-06T07:20:40.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f14d865c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d0151ac0 con 0x7f14d006b530 2024-03-06T07:20:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f14d865c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d00c7310 con 0x7f14d00c0b90 2024-03-06T07:20:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f14d63f8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14d006b530 0x7f14d00a77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7fa930898640 1 -- 172.21.15.1:0/2361699789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa92c0c0c20 msgr2=0x7fa92c15d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f14d5bf7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14d00c0b90 0x7f14d00a9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f14d5bf7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14d00c0b90 0x7f14d00a9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:41316/0 (socket says 172.21.15.1:41316) 2024-03-06T07:20:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f14d5bf7640 1 -- 172.21.15.1:0/445168768 learned_addr learned my addr 172.21.15.1:0/445168768 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7f73b1517640 1 -- 172.21.15.1:0/1813508386 >> v2:172.21.15.98:3300/0 conn(0x7f73ac0c4560 msgr2=0x7f73ac155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7f73b1517640 1 --2- 172.21.15.1:0/1813508386 >> v2:172.21.15.98:3300/0 conn(0x7f73ac0c4560 0x7f73ac155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7f73b1517640 1 -- 172.21.15.1:0/1813508386 >> v1:172.21.15.1:6789/0 conn(0x7f73ac0c39c0 legacy=0x7f73ac0ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7f73b1517640 1 -- 172.21.15.1:0/1813508386 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac0c76c0 con 0x7f73ac0c4940 2024-03-06T07:20:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7f14d6bf9640 1 --2- 172.21.15.1:0/445168768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d014e7d0 0x7f14d00aa000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7fa930898640 1 --2- 172.21.15.1:0/2361699789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa92c0c0c20 0x7fa92c15d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7fa930898640 1 -- 172.21.15.1:0/2361699789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa92c06edd0 msgr2=0x7fa92c15cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7f9099c9f640 1 -- 172.21.15.1:0/4108433486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f908c04b110 msgr2=0x7f908c0d2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7f9099c9f640 1 --2- 172.21.15.1:0/4108433486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f908c04b110 0x7f908c0d2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7f9099c9f640 1 -- 172.21.15.1:0/4108433486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f908c007430 msgr2=0x7f908c0c9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7f9099c9f640 1 --2- 172.21.15.1:0/4108433486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f908c007430 0x7f908c0c9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7f9099c9f640 1 -- 172.21.15.1:0/4108433486 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c0c97c0 con 0x7f908c0d3b40 2024-03-06T07:20:40.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7fa930898640 1 --2- 172.21.15.1:0/2361699789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa92c06edd0 0x7fa92c15cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.663+0000 7fa930898640 1 -- 172.21.15.1:0/2361699789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92c0aae40 con 0x7fa92c159540 2024-03-06T07:20:40.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7f73b1517640 1 --2- 172.21.15.1:0/1813508386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73ac0c4940 0x7f73ac1561b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0c5ce0 tx=0x7f73a402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d37f45bd4397f48 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7fc9b23a2640 1 -- 172.21.15.1:0/374655084 shutdown_connections 2024-03-06T07:20:40.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.662+0000 7fc9b23a2640 1 -- 172.21.15.1:0/374655084 wait complete. 2024-03-06T07:20:40.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7f14d5bf7640 1 -- 172.21.15.1:0/445168768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14d006b530 msgr2=0x7f14d00a77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7f14d5bf7640 1 --2- 172.21.15.1:0/445168768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14d006b530 0x7f14d00a77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7f14d5bf7640 1 -- 172.21.15.1:0/445168768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d014e7d0 msgr2=0x7f14d00aa000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7fa930898640 1 --2- 172.21.15.1:0/2361699789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c159540 0x7fa92c156090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa92c159970 tx=0x7fa91400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7fa928ff9640 1 -- 172.21.15.1:0/2361699789 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa914015020 con 0x7fa92c159540 2024-03-06T07:20:40.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7f14d5bf7640 1 --2- 172.21.15.1:0/445168768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d014e7d0 0x7f14d00aa000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7f14d5bf7640 1 -- 172.21.15.1:0/445168768 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d00a83c0 con 0x7f14d00c0b90 2024-03-06T07:20:40.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.665+0000 7f14d63f8640 1 --2- 172.21.15.1:0/445168768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14d006b530 0x7f14d00a77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:40.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.665+0000 7fa928ff9640 1 -- 172.21.15.1:0/2361699789 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa914011830 con 0x7fa92c159540 2024-03-06T07:20:40.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.665+0000 7f14d6bf9640 1 --2- 172.21.15.1:0/445168768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d014e7d0 0x7f14d00aa000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:40.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.665+0000 7fa928ff9640 1 -- 172.21.15.1:0/2361699789 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa914011c20 con 0x7fa92c159540 2024-03-06T07:20:40.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7f514e7fc640 1 -- 172.21.15.1:0/2840596967 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c03a400 con 0x7f51480f2260 2024-03-06T07:20:40.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.665+0000 7f5154ed0640 1 -- 172.21.15.1:0/2840596967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51480f2260 msgr2=0x7f51480f89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.665+0000 7f5154ed0640 1 --2- 172.21.15.1:0/2840596967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51480f2260 0x7f51480f89e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f51480481b0 tx=0x7f513c02fda0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7f73ab7fe640 1 -- 172.21.15.1:0/1813508386 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a4039070 con 0x7f73ac0c4940 2024-03-06T07:20:40.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7f73ab7fe640 1 -- 172.21.15.1:0/1813508386 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a402cc70 con 0x7f73ac0c4940 2024-03-06T07:20:40.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.664+0000 7f73ab7fe640 1 -- 172.21.15.1:0/1813508386 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a4034660 con 0x7f73ac0c4940 2024-03-06T07:20:40.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.665+0000 7fa92bfff640 1 --2- 172.21.15.1:0/2361699789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa92c06edd0 0x7fa92c15cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:40.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.665+0000 7fa931099640 1 -- 172.21.15.1:0/2361699789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa92c15b720 con 0x7fa92c159540 2024-03-06T07:20:40.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.665+0000 7fa931099640 1 -- 172.21.15.1:0/2361699789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa92c15bce0 con 0x7fa92c159540 2024-03-06T07:20:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.665+0000 7f73b2f7a640 1 -- 172.21.15.1:0/1813508386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73ac0c4940 msgr2=0x7f73ac1561b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.665+0000 7f73b2f7a640 1 --2- 172.21.15.1:0/1813508386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73ac0c4940 0x7f73ac1561b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0c5ce0 tx=0x7f73a402bec0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.665+0000 7f14d5bf7640 1 --2- 172.21.15.1:0/445168768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14d00c0b90 0x7f14d00a9ac0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f14cc0066f0 tx=0x7f14cc00b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.665+0000 7f14c6ffd640 1 -- 172.21.15.1:0/445168768 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc004230 con 0x7f14d00c0b90 2024-03-06T07:20:40.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.666+0000 7f9099c9f640 1 --2- 172.21.15.1:0/4108433486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f908c0d3b40 0x7f908c1745d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9094050c30 tx=0x7f9094068ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.666+0000 7f14d865c640 1 -- 172.21.15.1:0/445168768 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d01efc10 con 0x7f14d00c0b90 2024-03-06T07:20:40.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.666+0000 7f14d865c640 1 -- 172.21.15.1:0/445168768 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d0150d40 con 0x7f14d00c0b90 2024-03-06T07:20:40.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.666+0000 7f14c6ffd640 1 -- 172.21.15.1:0/445168768 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc0043d0 con 0x7f14d00c0b90 2024-03-06T07:20:40.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.666+0000 7f14c6ffd640 1 -- 172.21.15.1:0/445168768 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc0102b0 con 0x7f14d00c0b90 2024-03-06T07:20:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.666+0000 7f9089ffb640 1 -- 172.21.15.1:0/4108433486 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909406d070 con 0x7f908c0d3b40 2024-03-06T07:20:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.666+0000 7f909a4a0640 1 -- 172.21.15.1:0/4108433486 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908c02ed70 con 0x7f908c0d3b40 2024-03-06T07:20:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.667+0000 7f909a4a0640 1 -- 172.21.15.1:0/4108433486 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908c030a00 con 0x7f908c0d3b40 2024-03-06T07:20:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.667+0000 7f73b2f7a640 1 -- 172.21.15.1:0/1813508386 shutdown_connections 2024-03-06T07:20:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.667+0000 7f73b2f7a640 1 --2- 172.21.15.1:0/1813508386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73ac0c4940 0x7f73ac1561b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.667+0000 7f73b2f7a640 1 --2- 172.21.15.1:0/1813508386 >> v2:172.21.15.98:3300/0 conn(0x7f73ac0c4560 0x7f73ac155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.667+0000 7f14d865c640 1 -- 172.21.15.1:0/445168768 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f14b8001030 con 0x7f14d00c0b90 2024-03-06T07:20:40.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.667+0000 7f73b2f7a640 1 -- 172.21.15.1:0/1813508386 >> 172.21.15.1:0/1813508386 conn(0x7f73ac0c7a50 msgr2=0x7f73ac0c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.667+0000 7f9089ffb640 1 -- 172.21.15.1:0/4108433486 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909408e030 con 0x7f908c0d3b40 2024-03-06T07:20:40.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.667+0000 7f73b2f7a640 1 -- 172.21.15.1:0/1813508386 shutdown_connections 2024-03-06T07:20:40.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.668+0000 7f9089ffb640 1 -- 172.21.15.1:0/4108433486 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094093420 con 0x7f908c0d3b40 2024-03-06T07:20:40.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.668+0000 7f14c6ffd640 1 -- 172.21.15.1:0/445168768 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14cc003610 con 0x7f14d00c0b90 2024-03-06T07:20:40.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.667+0000 7f73b2f7a640 1 -- 172.21.15.1:0/1813508386 wait complete. 2024-03-06T07:20:40.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.666+0000 7fa931099640 1 -- 172.21.15.1:0/2361699789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa92c067fd0 con 0x7fa92c159540 2024-03-06T07:20:40.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.667+0000 7f5154ed0640 1 -- 172.21.15.1:0/2840596967 shutdown_connections 2024-03-06T07:20:40.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.667+0000 7f5154ed0640 1 --2- 172.21.15.1:0/2840596967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f514800b070 0x7f51480f9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.667+0000 7f5154ed0640 1 --2- 172.21.15.1:0/2840596967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51480f2260 0x7f51480f89e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.667+0000 7f5154ed0640 1 --2- 172.21.15.1:0/2840596967 >> v2:172.21.15.98:3300/0 conn(0x7f5148046150 0x7f5148046530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.668+0000 7f5154ed0640 1 -- 172.21.15.1:0/2840596967 >> 172.21.15.1:0/2840596967 conn(0x7f5148005ee0 msgr2=0x7f51480062f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.668+0000 7f907f7fe640 1 -- 172.21.15.1:0/4108433486 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f908c02f050 con 0x7f908c0d3b40 2024-03-06T07:20:40.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.668+0000 7f5154ed0640 1 -- 172.21.15.1:0/2840596967 shutdown_connections 2024-03-06T07:20:40.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.668+0000 7f14c6ffd640 1 --2- 172.21.15.1:0/445168768 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f14b406b5a0 0x7f14b406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.669+0000 7f14c6ffd640 1 -- 172.21.15.1:0/445168768 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f14cc08a850 con 0x7f14d00c0b90 2024-03-06T07:20:40.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.669+0000 7f73b2f7a640 1 Processor -- start 2024-03-06T07:20:40.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.669+0000 7f14d63f8640 1 --2- 172.21.15.1:0/445168768 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f14b406b5a0 0x7f14b406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.669+0000 7f73b2f7a640 1 -- start start 2024-03-06T07:20:40.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.670+0000 7f9089ffb640 1 -- 172.21.15.1:0/4108433486 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90940936c0 con 0x7f908c0d3b40 2024-03-06T07:20:40.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.670+0000 7f14c6ffd640 1 --2- 172.21.15.1:0/445168768 >> [v2:172.21.15.1:6808/1031158088,v1:172.21.15.1:6809/1031158088] conn(0x7f14b4074d10 0x7f14b4077150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.670+0000 7f14c6ffd640 1 -- 172.21.15.1:0/445168768 --> [v2:172.21.15.1:6808/1031158088,v1:172.21.15.1:6809/1031158088] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f14cc011e50 con 0x7f14b4074d10 2024-03-06T07:20:40.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.670+0000 7f14c6ffd640 1 -- 172.21.15.1:0/445168768 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14cc08f2a0 con 0x7f14d00c0b90 2024-03-06T07:20:40.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.670+0000 7f14d6bf9640 1 --2- 172.21.15.1:0/445168768 >> [v2:172.21.15.1:6808/1031158088,v1:172.21.15.1:6809/1031158088] conn(0x7f14b4074d10 0x7f14b4077150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.670+0000 7f9089ffb640 1 --2- 172.21.15.1:0/4108433486 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f907806b5a0 0x7f907806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.669+0000 7fa928ff9640 1 -- 172.21.15.1:0/2361699789 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa914018020 con 0x7fa92c159540 2024-03-06T07:20:40.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.670+0000 7f9089ffb640 1 -- 172.21.15.1:0/4108433486 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9094105ea0 con 0x7f908c0d3b40 2024-03-06T07:20:40.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.670+0000 7fa928ff9640 1 --2- 172.21.15.1:0/2361699789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa90c06b5a0 0x7fa90c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f14d6bf9640 1 --2- 172.21.15.1:0/445168768 >> [v2:172.21.15.1:6808/1031158088,v1:172.21.15.1:6809/1031158088] conn(0x7f14b4074d10 0x7f14b4077150 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f909949e640 1 --2- 172.21.15.1:0/4108433486 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f907806b5a0 0x7f907806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7fa928ff9640 1 -- 172.21.15.1:0/2361699789 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa91408a350 con 0x7fa92c159540 2024-03-06T07:20:40.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f73b2f7a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73ac0c39c0 0x7f73ac1e9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7fa928ff9640 1 --2- 172.21.15.1:0/2361699789 >> [v2:172.21.15.98:6816/851921598,v1:172.21.15.98:6817/851921598] conn(0x7fa90c074d10 0x7fa90c077150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f9089ffb640 1 --2- 172.21.15.1:0/4108433486 >> [v2:172.21.15.1:6817/682560127,v1:172.21.15.1:6819/682560127] conn(0x7f9078074d10 0x7f9078077150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f73b0d16640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73ac0c39c0 0x7f73ac1e9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7fa928ff9640 1 -- 172.21.15.1:0/2361699789 --> [v2:172.21.15.98:6816/851921598,v1:172.21.15.98:6817/851921598] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa914005640 con 0x7fa90c074d10 2024-03-06T07:20:40.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f73b0d16640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73ac0c39c0 0x7f73ac1e9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40464/0 (socket says 172.21.15.1:40464) 2024-03-06T07:20:40.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f73b2f7a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73ac0c4560 0x7f73ac1ea230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f73b0d16640 1 -- 172.21.15.1:0/2951244894 learned_addr learned my addr 172.21.15.1:0/2951244894 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:40.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7fa928ff9640 1 -- 172.21.15.1:0/2361699789 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa914091050 con 0x7fa92c159540 2024-03-06T07:20:40.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f909949e640 1 --2- 172.21.15.1:0/4108433486 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f907806b5a0 0x7f907806da60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9090000f60 tx=0x7f90900079b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f73b2f7a640 1 --2- 172.21.15.1:0/2951244894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac0c4940 0x7f73ac0bef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.668+0000 7f5154ed0640 1 -- 172.21.15.1:0/2840596967 wait complete. 2024-03-06T07:20:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f73b2f7a640 1 -- 172.21.15.1:0/2951244894 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac06e6d0 con 0x7f73ac0c39c0 2024-03-06T07:20:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f9098c9d640 1 --2- 172.21.15.1:0/4108433486 >> [v2:172.21.15.1:6817/682560127,v1:172.21.15.1:6819/682560127] conn(0x7f9078074d10 0x7f9078077150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f73b2f7a640 1 -- 172.21.15.1:0/2951244894 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac0abd00 con 0x7f73ac0c4940 2024-03-06T07:20:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.670+0000 7f5154ed0640 1 Processor -- start 2024-03-06T07:20:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.670+0000 7f5154ed0640 1 -- start start 2024-03-06T07:20:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f5154ed0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f514800b070 0x7f5148191050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f5154ed0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5148046150 0x7f514802ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f5154ed0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51480428a0 0x7f514802f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f5154ed0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514800b630 con 0x7f514800b070 2024-03-06T07:20:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f5154ed0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51480f99c0 con 0x7f5148046150 2024-03-06T07:20:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f5154ed0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514800b910 con 0x7f51480428a0 2024-03-06T07:20:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f514effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5148046150 0x7f514802ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f514effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5148046150 0x7f514802ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:44482/0 (socket says 172.21.15.1:44482) 2024-03-06T07:20:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.671+0000 7f514effd640 1 -- 172.21.15.1:0/1976148701 learned_addr learned my addr 172.21.15.1:0/1976148701 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f514ffff640 1 --2- 172.21.15.1:0/1976148701 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51480428a0 0x7f514802f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7fa92b7fe640 1 --2- 172.21.15.1:0/2361699789 >> [v2:172.21.15.98:6816/851921598,v1:172.21.15.98:6817/851921598] conn(0x7fa90c074d10 0x7fa90c077150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f14d63f8640 1 --2- 172.21.15.1:0/445168768 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f14b406b5a0 0x7f14b406da60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f14c8036000 tx=0x7f14c8030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f73b2f7a640 1 -- 172.21.15.1:0/2951244894 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac06e850 con 0x7f73ac0c4560 2024-03-06T07:20:40.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f9098c9d640 1 --2- 172.21.15.1:0/4108433486 >> [v2:172.21.15.1:6817/682560127,v1:172.21.15.1:6819/682560127] conn(0x7f9078074d10 0x7f9078077150 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f9089ffb640 1 -- 172.21.15.1:0/4108433486 --> [v2:172.21.15.1:6817/682560127,v1:172.21.15.1:6819/682560127] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f909408ce50 con 0x7f9078074d10 2024-03-06T07:20:40.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f14c6ffd640 1 -- 172.21.15.1:0/445168768 <== osd.3 v2:172.21.15.1:6808/1031158088 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f14cc011e50 con 0x7f14b4074d10 2024-03-06T07:20:40.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f73b1517640 1 --2- 172.21.15.1:0/2951244894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac0c4940 0x7f73ac0bef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f73abfff640 1 --2- 172.21.15.1:0/2951244894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73ac0c4560 0x7f73ac1ea230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7fa92bfff640 1 --2- 172.21.15.1:0/2361699789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa90c06b5a0 0x7fa90c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7fa92b7fe640 1 --2- 172.21.15.1:0/2361699789 >> [v2:172.21.15.98:6816/851921598,v1:172.21.15.98:6817/851921598] conn(0x7fa90c074d10 0x7fa90c077150 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f514effd640 1 -- 172.21.15.1:0/1976148701 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51480428a0 msgr2=0x7f514802f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f514effd640 1 --2- 172.21.15.1:0/1976148701 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51480428a0 0x7f514802f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f514effd640 1 -- 172.21.15.1:0/1976148701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f514800b070 msgr2=0x7f5148191050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f514f7fe640 1 --2- 172.21.15.1:0/1976148701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f514800b070 0x7f5148191050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f514effd640 1 --2- 172.21.15.1:0/1976148701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f514800b070 0x7f5148191050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f514effd640 1 -- 172.21.15.1:0/1976148701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514802ffa0 con 0x7f5148046150 2024-03-06T07:20:40.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f514f7fe640 1 --2- 172.21.15.1:0/1976148701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f514800b070 0x7f5148191050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:40.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f514effd640 1 --2- 172.21.15.1:0/1976148701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5148046150 0x7f514802ebb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f513c03c040 tx=0x7f513c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.673+0000 7fa92bfff640 1 --2- 172.21.15.1:0/2361699789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa90c06b5a0 0x7fa90c06da60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa91c009ac0 tx=0x7fa91c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.673+0000 7f9089ffb640 1 -- 172.21.15.1:0/4108433486 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9094106290 con 0x7f908c0d3b40 2024-03-06T07:20:40.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f73b0d16640 1 -- 172.21.15.1:0/2951244894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73ac0c4560 msgr2=0x7f73ac1ea230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f73b0d16640 1 --2- 172.21.15.1:0/2951244894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73ac0c4560 0x7f73ac1ea230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f73b0d16640 1 -- 172.21.15.1:0/2951244894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac0c4940 msgr2=0x7f73ac0bef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f73b0d16640 1 --2- 172.21.15.1:0/2951244894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac0c4940 0x7f73ac0bef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.672+0000 7f73b0d16640 1 -- 172.21.15.1:0/2951244894 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac0c0a70 con 0x7f73ac0c39c0 2024-03-06T07:20:40.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.673+0000 7f9089ffb640 1 -- 172.21.15.1:0/4108433486 <== osd.1 v2:172.21.15.1:6817/682560127 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f909408ce50 con 0x7f9078074d10 2024-03-06T07:20:40.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.673+0000 7f73b0d16640 1 --2- 172.21.15.1:0/2951244894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73ac0c39c0 0x7f73ac1e9cf0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f73a000fd80 tx=0x7f73a000d6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.673+0000 7f73a97fa640 1 -- 172.21.15.1:0/2951244894 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0011650 con 0x7f73ac0c39c0 2024-03-06T07:20:40.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.673+0000 7f73a97fa640 1 -- 172.21.15.1:0/2951244894 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a00117f0 con 0x7f73ac0c39c0 2024-03-06T07:20:40.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.673+0000 7f73a97fa640 1 -- 172.21.15.1:0/2951244894 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0011650 con 0x7f73ac0c39c0 2024-03-06T07:20:40.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.673+0000 7f73b2f7a640 1 -- 172.21.15.1:0/2951244894 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73ac0bcbf0 con 0x7f73ac0c39c0 2024-03-06T07:20:40.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.674+0000 7f73b2f7a640 1 -- 172.21.15.1:0/2951244894 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ac0c1c90 con 0x7f73ac0c39c0 2024-03-06T07:20:40.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.673+0000 7f513bfff640 1 -- 172.21.15.1:0/1976148701 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c048030 con 0x7f5148046150 2024-03-06T07:20:40.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.673+0000 7f513bfff640 1 -- 172.21.15.1:0/1976148701 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513c044610 con 0x7f5148046150 2024-03-06T07:20:40.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.674+0000 7f513bfff640 1 -- 172.21.15.1:0/1976148701 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c044a00 con 0x7f5148046150 2024-03-06T07:20:40.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.674+0000 7f5154ed0640 1 -- 172.21.15.1:0/1976148701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5148191970 con 0x7f5148046150 2024-03-06T07:20:40.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.674+0000 7f5154ed0640 1 -- 172.21.15.1:0/1976148701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5148030bd0 con 0x7f5148046150 2024-03-06T07:20:40.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.675+0000 7fa928ff9640 1 -- 172.21.15.1:0/2361699789 <== osd.5 v2:172.21.15.98:6816/851921598 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa914005640 con 0x7fa90c074d10 2024-03-06T07:20:40.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.675+0000 7f5139ffb640 1 -- 172.21.15.1:0/1976148701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f51480ef230 con 0x7f5148046150 2024-03-06T07:20:40.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.676+0000 7fc9b23a2640 1 Processor -- start 2024-03-06T07:20:40.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.676+0000 7f73b2f7a640 1 -- 172.21.15.1:0/2951244894 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f73ac0686e0 con 0x7f73ac0c39c0 2024-03-06T07:20:40.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.676+0000 7fc9b23a2640 1 -- start start 2024-03-06T07:20:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.677+0000 7f73a97fa640 1 -- 172.21.15.1:0/2951244894 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73a0019020 con 0x7f73ac0c39c0 2024-03-06T07:20:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.677+0000 7f73a97fa640 1 --2- 172.21.15.1:0/2951244894 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f739006b8d0 0x7f739006dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.677+0000 7f73a97fa640 1 -- 172.21.15.1:0/2951244894 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f73a0011450 con 0x7f73ac0c39c0 2024-03-06T07:20:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.677+0000 7f73a97fa640 1 --2- 172.21.15.1:0/2951244894 >> [v2:172.21.15.98:6808/2308139585,v1:172.21.15.98:6809/2308139585] conn(0x7f7390074f50 0x7f7390077390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.677+0000 7f73a97fa640 1 -- 172.21.15.1:0/2951244894 --> [v2:172.21.15.98:6808/2308139585,v1:172.21.15.98:6809/2308139585] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7390077a60 con 0x7f7390074f50 2024-03-06T07:20:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.677+0000 7f73a97fa640 1 -- 172.21.15.1:0/2951244894 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73a008c5a0 con 0x7f73ac0c39c0 2024-03-06T07:20:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.678+0000 7f73b1517640 1 --2- 172.21.15.1:0/2951244894 >> [v2:172.21.15.98:6808/2308139585,v1:172.21.15.98:6809/2308139585] conn(0x7f7390074f50 0x7f7390077390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.678+0000 7f513bfff640 1 -- 172.21.15.1:0/1976148701 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f513c04b070 con 0x7f5148046150 2024-03-06T07:20:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.678+0000 7f73abfff640 1 --2- 172.21.15.1:0/2951244894 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f739006b8d0 0x7f739006dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.678+0000 7f73b1517640 1 --2- 172.21.15.1:0/2951244894 >> [v2:172.21.15.98:6808/2308139585,v1:172.21.15.98:6809/2308139585] conn(0x7f7390074f50 0x7f7390077390 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.678+0000 7f513bfff640 1 --2- 172.21.15.1:0/1976148701 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f513406b4d0 0x7f513406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.678+0000 7f513bfff640 1 -- 172.21.15.1:0/1976148701 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f513c0acba0 con 0x7f5148046150 2024-03-06T07:20:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.679+0000 7f514f7fe640 1 --2- 172.21.15.1:0/1976148701 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f513406b4d0 0x7f513406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.679+0000 7f513bfff640 1 --2- 172.21.15.1:0/1976148701 >> [v2:172.21.15.98:6824/3249148736,v1:172.21.15.98:6825/3249148736] conn(0x7f5134074c40 0x7f5134077080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.679+0000 7f513bfff640 1 -- 172.21.15.1:0/1976148701 --> [v2:172.21.15.98:6824/3249148736,v1:172.21.15.98:6825/3249148736] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f513c040e50 con 0x7f5134074c40 2024-03-06T07:20:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.679+0000 7f513bfff640 1 -- 172.21.15.1:0/1976148701 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f513c0acf90 con 0x7f5148046150 2024-03-06T07:20:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.679+0000 7f514ffff640 1 --2- 172.21.15.1:0/1976148701 >> [v2:172.21.15.98:6824/3249148736,v1:172.21.15.98:6825/3249148736] conn(0x7f5134074c40 0x7f5134077080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.679+0000 7f73a97fa640 1 -- 172.21.15.1:0/2951244894 <== osd.4 v2:172.21.15.98:6808/2308139585 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7390077a60 con 0x7f7390074f50 2024-03-06T07:20:40.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.680+0000 7f514f7fe640 1 --2- 172.21.15.1:0/1976148701 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f513406b4d0 0x7f513406d990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f514804b730 tx=0x7f5144008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.680+0000 7f514ffff640 1 --2- 172.21.15.1:0/1976148701 >> [v2:172.21.15.98:6824/3249148736,v1:172.21.15.98:6825/3249148736] conn(0x7f5134074c40 0x7f5134077080 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.681+0000 7f513bfff640 1 -- 172.21.15.1:0/1976148701 <== osd.7 v2:172.21.15.98:6824/3249148736 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f513c040e50 con 0x7f5134074c40 2024-03-06T07:20:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.686+0000 7fc9b23a2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9ac0c7310 0x7fc9ac150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.686+0000 7fc9b13a0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9ac0c7310 0x7fc9ac150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.687+0000 7fc9b13a0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9ac0c7310 0x7fc9ac150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40494/0 (socket says 172.21.15.1:40494) 2024-03-06T07:20:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.687+0000 7fc9b13a0640 1 -- 172.21.15.1:0/1411711393 learned_addr learned my addr 172.21.15.1:0/1411711393 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.687+0000 7f73abfff640 1 --2- 172.21.15.1:0/2951244894 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f739006b8d0 0x7f739006dd90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f73ac070e10 tx=0x7f7398007b50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.687+0000 7fc9b23a2640 1 --2- 172.21.15.1:0/1411711393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9ac154c90 0x7fc9ac141f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.688+0000 7fc9b0b9f640 1 --2- 172.21.15.1:0/1411711393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9ac154c90 0x7fc9ac141f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.688+0000 7fc9b23a2640 1 --2- 172.21.15.1:0/1411711393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9ac144690 0x7fc9ac152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.689+0000 7fc9b23a2640 1 -- 172.21.15.1:0/1411711393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac073120 con 0x7fc9ac0c7310 2024-03-06T07:20:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.689+0000 7fc9b23a2640 1 -- 172.21.15.1:0/1411711393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac14eaa0 con 0x7fc9ac154c90 2024-03-06T07:20:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.689+0000 7fc9b23a2640 1 -- 172.21.15.1:0/1411711393 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac0732a0 con 0x7fc9ac144690 2024-03-06T07:20:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.691+0000 7fc9b0b9f640 1 -- 172.21.15.1:0/1411711393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9ac144690 msgr2=0x7fc9ac152e00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:20:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.691+0000 7fc9b0b9f640 1 --2- 172.21.15.1:0/1411711393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9ac144690 0x7fc9ac152e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.692+0000 7fc9b0b9f640 1 -- 172.21.15.1:0/1411711393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9ac0c7310 msgr2=0x7fc9ac150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.692+0000 7fc9b0b9f640 1 --2- 172.21.15.1:0/1411711393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9ac0c7310 0x7fc9ac150000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.692+0000 7fc9b0b9f640 1 -- 172.21.15.1:0/1411711393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ac150540 con 0x7fc9ac154c90 2024-03-06T07:20:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.693+0000 7fc9b13a0640 1 --2- 172.21.15.1:0/1411711393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9ac0c7310 0x7fc9ac150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:40.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.693+0000 7fc9b0b9f640 1 --2- 172.21.15.1:0/1411711393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9ac154c90 0x7fc9ac141f40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4007e90 tx=0x7fc9a400e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.695+0000 7ff9e08e6640 1 Processor -- start 2024-03-06T07:20:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.696+0000 7fc9a1ffb640 1 -- 172.21.15.1:0/1411711393 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4014420 con 0x7fc9ac154c90 2024-03-06T07:20:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.696+0000 7fc9a1ffb640 1 -- 172.21.15.1:0/1411711393 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a4019070 con 0x7fc9ac154c90 2024-03-06T07:20:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.696+0000 7fc9a1ffb640 1 -- 172.21.15.1:0/1411711393 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a40147c0 con 0x7fc9ac154c90 2024-03-06T07:20:40.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.696+0000 7fc9b23a2640 1 -- 172.21.15.1:0/1411711393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9ac14f040 con 0x7fc9ac154c90 2024-03-06T07:20:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.695+0000 7ff9e08e6640 1 -- start start 2024-03-06T07:20:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.696+0000 7ff9e08e6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff9dc14e940 0x7ff9dc151130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.696+0000 7ff9e08e6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9dc154f00 con 0x7ff9dc073120 2024-03-06T07:20:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.696+0000 7ff9e08e6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9dc1551e0 con 0x7ff9dc14e560 2024-03-06T07:20:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.696+0000 7ff9e08e6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9dc06f450 con 0x7ff9dc14e940 2024-03-06T07:20:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.696+0000 7ff9dbfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff9dc073120 0x7ff9dc06aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57428/0 (socket says 172.21.15.1:57428) 2024-03-06T07:20:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.696+0000 7ff9daffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff9dc14e940 0x7ff9dc151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.696+0000 7ff9dbfff640 1 -- 172.21.15.1:0/3505742710 learned_addr learned my addr 172.21.15.1:0/3505742710 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:40.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.697+0000 7fc9b23a2640 1 -- 172.21.15.1:0/1411711393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ac152480 con 0x7fc9ac154c90 2024-03-06T07:20:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.697+0000 7ff9da7fc640 1 -- 172.21.15.1:0/3505742710 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917478874 0 0) 0x7ff9dc154f00 con 0x7ff9dc073120 2024-03-06T07:20:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.697+0000 7ff9daffd640 1 -- 172.21.15.1:0/3505742710 >> v1:172.21.15.98:6789/0 conn(0x7ff9dc14e560 legacy=0x7ff9dc0c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.697+0000 7ff9daffd640 1 -- 172.21.15.1:0/3505742710 >> v1:172.21.15.1:6789/0 conn(0x7ff9dc073120 legacy=0x7ff9dc06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.697+0000 7ff9daffd640 1 -- 172.21.15.1:0/3505742710 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9dc1554c0 con 0x7ff9dc14e940 2024-03-06T07:20:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.698+0000 7ff9daffd640 1 --2- 172.21.15.1:0/3505742710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9dc14e940 0x7ff9dc151130 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009670 tx=0x7ff9d402e250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2a44d767725704f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.700+0000 7ff9da7fc640 1 -- 172.21.15.1:0/3505742710 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d403b070 con 0x7ff9dc14e940 2024-03-06T07:20:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.700+0000 7ff9da7fc640 1 -- 172.21.15.1:0/3505742710 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d402edf0 con 0x7ff9dc14e940 2024-03-06T07:20:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.701+0000 7ff9da7fc640 1 -- 172.21.15.1:0/3505742710 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4036640 con 0x7ff9dc14e940 2024-03-06T07:20:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.700+0000 7ff9e08e6640 1 -- 172.21.15.1:0/3505742710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9dc14e940 msgr2=0x7ff9dc151130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.701+0000 7ff9e08e6640 1 --2- 172.21.15.1:0/3505742710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9dc14e940 0x7ff9dc151130 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009670 tx=0x7ff9d402e250 comp rx=0 tx=0).stop 2024-03-06T07:20:40.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.701+0000 7ff9e08e6640 1 -- 172.21.15.1:0/3505742710 shutdown_connections 2024-03-06T07:20:40.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.702+0000 7fc9b23a2640 1 -- 172.21.15.1:0/1411711393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc9ac067fd0 con 0x7fc9ac154c90 2024-03-06T07:20:40.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.702+0000 7ff9e08e6640 1 --2- 172.21.15.1:0/3505742710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9dc14e940 0x7ff9dc151130 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.702+0000 7ff9e08e6640 1 -- 172.21.15.1:0/3505742710 >> 172.21.15.1:0/3505742710 conn(0x7ff9dc0c6e60 msgr2=0x7ff9dc0a8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.703+0000 7f14d865c640 1 -- 172.21.15.1:0/445168768 --> [v2:172.21.15.1:6808/1031158088,v1:172.21.15.1:6809/1031158088] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f14b8002dd0 con 0x7f14b4074d10 2024-03-06T07:20:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.703+0000 7ff9e08e6640 1 -- 172.21.15.1:0/3505742710 shutdown_connections 2024-03-06T07:20:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.703+0000 7f907f7fe640 1 -- 172.21.15.1:0/4108433486 --> [v2:172.21.15.1:6817/682560127,v1:172.21.15.1:6819/682560127] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f908c0052c0 con 0x7f9078074d10 2024-03-06T07:20:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.703+0000 7fc9a1ffb640 1 -- 172.21.15.1:0/1411711393 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc9a4014960 con 0x7fc9ac154c90 2024-03-06T07:20:40.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.703+0000 7fc9a1ffb640 1 --2- 172.21.15.1:0/1411711393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc99006b5a0 0x7fc99006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.703+0000 7fc9a1ffb640 1 -- 172.21.15.1:0/1411711393 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc9a408ac70 con 0x7fc9ac154c90 2024-03-06T07:20:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.703+0000 7fc9a1ffb640 1 --2- 172.21.15.1:0/1411711393 >> [v2:172.21.15.1:6816/3842251104,v1:172.21.15.1:6818/3842251104] conn(0x7fc990074c90 0x7fc9900770d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.703+0000 7fc9a1ffb640 1 -- 172.21.15.1:0/1411711393 --> [v2:172.21.15.1:6816/3842251104,v1:172.21.15.1:6818/3842251104] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9a401aec0 con 0x7fc990074c90 2024-03-06T07:20:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.703+0000 7fc9a1ffb640 1 -- 172.21.15.1:0/1411711393 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9a4058a80 con 0x7fc9ac154c90 2024-03-06T07:20:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.704+0000 7f14c6ffd640 1 -- 172.21.15.1:0/445168768 <== osd.3 v2:172.21.15.1:6808/1031158088 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f14b8002dd0 con 0x7f14b4074d10 2024-03-06T07:20:40.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.704+0000 7f9089ffb640 1 -- 172.21.15.1:0/4108433486 <== osd.1 v2:172.21.15.1:6817/682560127 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f908c0052c0 con 0x7f9078074d10 2024-03-06T07:20:40.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.704+0000 7f14c4ff9640 1 -- 172.21.15.1:0/445168768 >> [v2:172.21.15.1:6808/1031158088,v1:172.21.15.1:6809/1031158088] conn(0x7f14b4074d10 msgr2=0x7f14b4077150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.704+0000 7f14c4ff9640 1 --2- 172.21.15.1:0/445168768 >> [v2:172.21.15.1:6808/1031158088,v1:172.21.15.1:6809/1031158088] conn(0x7f14b4074d10 0x7f14b4077150 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.704+0000 7fc9b1ba1640 1 --2- 172.21.15.1:0/1411711393 >> [v2:172.21.15.1:6816/3842251104,v1:172.21.15.1:6818/3842251104] conn(0x7fc990074c90 0x7fc9900770d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.704+0000 7f14c4ff9640 1 -- 172.21.15.1:0/445168768 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f14b406b5a0 msgr2=0x7f14b406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.705+0000 7f14c4ff9640 1 --2- 172.21.15.1:0/445168768 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f14b406b5a0 0x7f14b406da60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f14c8036000 tx=0x7f14c8030000 comp rx=0 tx=0).stop 2024-03-06T07:20:40.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.705+0000 7f14c4ff9640 1 -- 172.21.15.1:0/445168768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14d00c0b90 msgr2=0x7f14d00a9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.705+0000 7f14c4ff9640 1 --2- 172.21.15.1:0/445168768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14d00c0b90 0x7f14d00a9ac0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f14cc0066f0 tx=0x7f14cc00b770 comp rx=0 tx=0).stop 2024-03-06T07:20:40.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.705+0000 7f14d6bf9640 1 -- 172.21.15.1:0/445168768 reap_dead start 2024-03-06T07:20:40.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.705+0000 7f14c4ff9640 1 -- 172.21.15.1:0/445168768 shutdown_connections 2024-03-06T07:20:40.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.705+0000 7f14c4ff9640 1 -- 172.21.15.1:0/445168768 >> 172.21.15.1:0/445168768 conn(0x7f14d00c6e60 msgr2=0x7f14d00c4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.703+0000 7ff9e08e6640 1 -- 172.21.15.1:0/3505742710 wait complete. 2024-03-06T07:20:40.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.705+0000 7fc9b1ba1640 1 --2- 172.21.15.1:0/1411711393 >> [v2:172.21.15.1:6816/3842251104,v1:172.21.15.1:6818/3842251104] conn(0x7fc990074c90 0x7fc9900770d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.705+0000 7fc9b13a0640 1 --2- 172.21.15.1:0/1411711393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc99006b5a0 0x7fc99006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.705+0000 7f907f7fe640 1 -- 172.21.15.1:0/4108433486 >> [v2:172.21.15.1:6817/682560127,v1:172.21.15.1:6819/682560127] conn(0x7f9078074d10 msgr2=0x7f9078077150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.705+0000 7f907f7fe640 1 --2- 172.21.15.1:0/4108433486 >> [v2:172.21.15.1:6817/682560127,v1:172.21.15.1:6819/682560127] conn(0x7f9078074d10 0x7f9078077150 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.705+0000 7ff9e08e6640 1 Processor -- start 2024-03-06T07:20:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.705+0000 7f907f7fe640 1 -- 172.21.15.1:0/4108433486 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f907806b5a0 msgr2=0x7f907806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.705+0000 7f907f7fe640 1 --2- 172.21.15.1:0/4108433486 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f907806b5a0 0x7f907806da60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9090000f60 tx=0x7f90900079b0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.705+0000 7f907f7fe640 1 -- 172.21.15.1:0/4108433486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f908c0d3b40 msgr2=0x7f908c1745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.705+0000 7f907f7fe640 1 --2- 172.21.15.1:0/4108433486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f908c0d3b40 0x7f908c1745d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9094050c30 tx=0x7f9094068ab0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.705+0000 7ff9e08e6640 1 -- start start 2024-03-06T07:20:40.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.706+0000 7f14c4ff9640 1 -- 172.21.15.1:0/445168768 shutdown_connections 2024-03-06T07:20:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.706+0000 7fc9a1ffb640 1 -- 172.21.15.1:0/1411711393 <== osd.0 v2:172.21.15.1:6816/3842251104 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc9a401aec0 con 0x7fc990074c90 2024-03-06T07:20:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.706+0000 7f9099c9f640 1 -- 172.21.15.1:0/4108433486 reap_dead start 2024-03-06T07:20:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.706+0000 7f907f7fe640 1 -- 172.21.15.1:0/4108433486 shutdown_connections 2024-03-06T07:20:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.706+0000 7f907f7fe640 1 -- 172.21.15.1:0/4108433486 >> 172.21.15.1:0/4108433486 conn(0x7f908c048250 msgr2=0x7f908c006850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.706+0000 7f14c4ff9640 1 -- 172.21.15.1:0/445168768 wait complete. 2024-03-06T07:20:40.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.706+0000 7f907f7fe640 1 -- 172.21.15.1:0/4108433486 shutdown_connections 2024-03-06T07:20:40.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.707+0000 7f907f7fe640 1 -- 172.21.15.1:0/4108433486 wait complete. 2024-03-06T07:20:40.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.706+0000 7ff9e08e6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9dc073120 0x7ff9dc150aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.706+0000 7ff9e08e6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9dc14e560 0x7ff9dc14cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.706+0000 7ff9e08e6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9dc14e940 0x7ff9dc14d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.706+0000 7ff9e08e6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9dc06b700 con 0x7ff9dc14e940 2024-03-06T07:20:40.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.706+0000 7ff9e08e6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9dc151670 con 0x7ff9dc14e560 2024-03-06T07:20:40.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.706+0000 7ff9e08e6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9dc06b880 con 0x7ff9dc073120 2024-03-06T07:20:40.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.707+0000 7ff9dbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9dc14e940 0x7ff9dc14d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.707+0000 7ff9dbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9dc14e940 0x7ff9dc14d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40496/0 (socket says 172.21.15.1:40496) 2024-03-06T07:20:40.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.707+0000 7ff9dbfff640 1 -- 172.21.15.1:0/834956691 learned_addr learned my addr 172.21.15.1:0/834956691 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:40.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.707+0000 7ff9db7fe640 1 --2- 172.21.15.1:0/834956691 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9dc073120 0x7ff9dc150aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.707+0000 7ff9daffd640 1 --2- 172.21.15.1:0/834956691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9dc14e560 0x7ff9dc14cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.707+0000 7ff9dbfff640 1 -- 172.21.15.1:0/834956691 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9dc073120 msgr2=0x7ff9dc150aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.707+0000 7ff9dbfff640 1 --2- 172.21.15.1:0/834956691 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9dc073120 0x7ff9dc150aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.707+0000 7ff9dbfff640 1 -- 172.21.15.1:0/834956691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9dc14e560 msgr2=0x7ff9dc14cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.707+0000 7ff9dbfff640 1 --2- 172.21.15.1:0/834956691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9dc14e560 0x7ff9dc14cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.707+0000 7ff9dbfff640 1 -- 172.21.15.1:0/834956691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9dc14e100 con 0x7ff9dc14e940 2024-03-06T07:20:40.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.708+0000 7ff9dbfff640 1 --2- 172.21.15.1:0/834956691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9dc14e940 0x7ff9dc14d4e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00c480 tx=0x7ff9cc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.708+0000 7ff9c3fff640 1 -- 172.21.15.1:0/834956691 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc018070 con 0x7ff9dc14e940 2024-03-06T07:20:40.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.708+0000 7ff9c3fff640 1 -- 172.21.15.1:0/834956691 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9cc0074e0 con 0x7ff9dc14e940 2024-03-06T07:20:40.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.708+0000 7ff9c3fff640 1 -- 172.21.15.1:0/834956691 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc004a20 con 0x7ff9dc14e940 2024-03-06T07:20:40.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.708+0000 7ff9e08e6640 1 -- 172.21.15.1:0/834956691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9dc14ef40 con 0x7ff9dc14e940 2024-03-06T07:20:40.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.709+0000 7f5139ffb640 1 -- 172.21.15.1:0/1976148701 --> [v2:172.21.15.98:6824/3249148736,v1:172.21.15.98:6825/3249148736] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f51480ef230 con 0x7f5134074c40 2024-03-06T07:20:40.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.709+0000 7ff9e08e6640 1 -- 172.21.15.1:0/834956691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9dc1fb5a0 con 0x7ff9dc14e940 2024-03-06T07:20:40.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.710+0000 7ff9c3fff640 1 -- 172.21.15.1:0/834956691 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff9cc004bc0 con 0x7ff9dc14e940 2024-03-06T07:20:40.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.711+0000 7f513bfff640 1 -- 172.21.15.1:0/1976148701 <== osd.7 v2:172.21.15.98:6824/3249148736 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f513c040e50 con 0x7f5134074c40 2024-03-06T07:20:40.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.711+0000 7ff9c3fff640 1 --2- 172.21.15.1:0/834956691 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff9c806b4d0 0x7ff9c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.711+0000 7ff9c1ffb640 1 -- 172.21.15.1:0/834956691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff9dc1fb0f0 con 0x7ff9dc14e940 2024-03-06T07:20:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.711+0000 7ff9c3fff640 1 -- 172.21.15.1:0/834956691 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff9cc08e060 con 0x7ff9dc14e940 2024-03-06T07:20:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.711+0000 7ff9c3fff640 1 --2- 172.21.15.1:0/834956691 >> [v2:172.21.15.98:6800/2463344330,v1:172.21.15.98:6801/2463344330] conn(0x7ff9c8074b40 0x7ff9c8076f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.711+0000 7ff9c3fff640 1 -- 172.21.15.1:0/834956691 --> [v2:172.21.15.98:6800/2463344330,v1:172.21.15.98:6801/2463344330] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9c8077650 con 0x7ff9c8074b40 2024-03-06T07:20:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.711+0000 7ff9c3fff640 1 -- 172.21.15.1:0/834956691 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9cc053d80 con 0x7ff9dc14e940 2024-03-06T07:20:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.711+0000 7ff9db7fe640 1 --2- 172.21.15.1:0/834956691 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff9c806b4d0 0x7ff9c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.712+0000 7f5139ffb640 1 -- 172.21.15.1:0/1976148701 >> [v2:172.21.15.98:6824/3249148736,v1:172.21.15.98:6825/3249148736] conn(0x7f5134074c40 msgr2=0x7f5134077080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.712+0000 7f5139ffb640 1 --2- 172.21.15.1:0/1976148701 >> [v2:172.21.15.98:6824/3249148736,v1:172.21.15.98:6825/3249148736] conn(0x7f5134074c40 0x7f5134077080 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.712+0000 7ff9daffd640 1 --2- 172.21.15.1:0/834956691 >> [v2:172.21.15.98:6800/2463344330,v1:172.21.15.98:6801/2463344330] conn(0x7ff9c8074b40 0x7ff9c8076f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.712+0000 7ff9daffd640 1 --2- 172.21.15.1:0/834956691 >> [v2:172.21.15.98:6800/2463344330,v1:172.21.15.98:6801/2463344330] conn(0x7ff9c8074b40 0x7ff9c8076f80 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.713+0000 7fc9b13a0640 1 --2- 172.21.15.1:0/1411711393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc99006b5a0 0x7fc99006da60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc99c009630 tx=0x7fc99c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.713+0000 7f5139ffb640 1 -- 172.21.15.1:0/1976148701 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f513406b4d0 msgr2=0x7f513406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.713+0000 7ff9db7fe640 1 --2- 172.21.15.1:0/834956691 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff9c806b4d0 0x7ff9c806d990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0001780 tx=0x7ff9d0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.713+0000 7f5139ffb640 1 --2- 172.21.15.1:0/1976148701 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f513406b4d0 0x7f513406d990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f514804b730 tx=0x7f5144008040 comp rx=0 tx=0).stop 2024-03-06T07:20:40.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.713+0000 7fa931099640 1 -- 172.21.15.1:0/2361699789 --> [v2:172.21.15.98:6816/851921598,v1:172.21.15.98:6817/851921598] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa92c0c5930 con 0x7fa90c074d10 2024-03-06T07:20:40.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.713+0000 7ff9c3fff640 1 -- 172.21.15.1:0/834956691 <== osd.6 v2:172.21.15.98:6800/2463344330 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff9c8077650 con 0x7ff9c8074b40 2024-03-06T07:20:40.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.713+0000 7f5139ffb640 1 -- 172.21.15.1:0/1976148701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5148046150 msgr2=0x7f514802ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.713+0000 7f5139ffb640 1 --2- 172.21.15.1:0/1976148701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5148046150 0x7f514802ebb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f513c03c040 tx=0x7f513c004730 comp rx=0 tx=0).stop 2024-03-06T07:20:40.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.714+0000 7f514ffff640 1 -- 172.21.15.1:0/1976148701 reap_dead start 2024-03-06T07:20:40.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.714+0000 7f5139ffb640 1 -- 172.21.15.1:0/1976148701 shutdown_connections 2024-03-06T07:20:40.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.714+0000 7f5139ffb640 1 -- 172.21.15.1:0/1976148701 >> 172.21.15.1:0/1976148701 conn(0x7f5148005ee0 msgr2=0x7f514802b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.714+0000 7fa928ff9640 1 -- 172.21.15.1:0/2361699789 <== osd.5 v2:172.21.15.98:6816/851921598 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa92c0c5930 con 0x7fa90c074d10 2024-03-06T07:20:40.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.714+0000 7f73b2f7a640 1 -- 172.21.15.1:0/2951244894 --> [v2:172.21.15.98:6808/2308139585,v1:172.21.15.98:6809/2308139585] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f73ac0bcdc0 con 0x7f7390074f50 2024-03-06T07:20:40.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.715+0000 7f5139ffb640 1 -- 172.21.15.1:0/1976148701 shutdown_connections 2024-03-06T07:20:40.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.715+0000 7f5139ffb640 1 -- 172.21.15.1:0/1976148701 wait complete. 2024-03-06T07:20:40.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.715+0000 7fa931099640 1 -- 172.21.15.1:0/2361699789 >> [v2:172.21.15.98:6816/851921598,v1:172.21.15.98:6817/851921598] conn(0x7fa90c074d10 msgr2=0x7fa90c077150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.715+0000 7fa931099640 1 --2- 172.21.15.1:0/2361699789 >> [v2:172.21.15.98:6816/851921598,v1:172.21.15.98:6817/851921598] conn(0x7fa90c074d10 0x7fa90c077150 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.715+0000 7f73a97fa640 1 -- 172.21.15.1:0/2951244894 <== osd.4 v2:172.21.15.98:6808/2308139585 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f73ac0bcdc0 con 0x7f7390074f50 2024-03-06T07:20:40.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.715+0000 7fa931099640 1 -- 172.21.15.1:0/2361699789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa90c06b5a0 msgr2=0x7fa90c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.715+0000 7fa931099640 1 --2- 172.21.15.1:0/2361699789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa90c06b5a0 0x7fa90c06da60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa91c009ac0 tx=0x7fa91c008040 comp rx=0 tx=0).stop 2024-03-06T07:20:40.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.715+0000 7fa931099640 1 -- 172.21.15.1:0/2361699789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c159540 msgr2=0x7fa92c156090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.715+0000 7fa931099640 1 --2- 172.21.15.1:0/2361699789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c159540 0x7fa92c156090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa92c159970 tx=0x7fa91400b870 comp rx=0 tx=0).stop 2024-03-06T07:20:40.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.715+0000 7fa930898640 1 -- 172.21.15.1:0/2361699789 reap_dead start 2024-03-06T07:20:40.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.715+0000 7fa931099640 1 -- 172.21.15.1:0/2361699789 shutdown_connections 2024-03-06T07:20:40.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.715+0000 7fa931099640 1 -- 172.21.15.1:0/2361699789 >> 172.21.15.1:0/2361699789 conn(0x7fa92c0c7290 msgr2=0x7fa92c0c3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.716+0000 7fa931099640 1 -- 172.21.15.1:0/2361699789 shutdown_connections 2024-03-06T07:20:40.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.716+0000 7fa931099640 1 -- 172.21.15.1:0/2361699789 wait complete. 2024-03-06T07:20:40.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.716+0000 7f73b2f7a640 1 -- 172.21.15.1:0/2951244894 >> [v2:172.21.15.98:6808/2308139585,v1:172.21.15.98:6809/2308139585] conn(0x7f7390074f50 msgr2=0x7f7390077390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.716+0000 7f73b2f7a640 1 --2- 172.21.15.1:0/2951244894 >> [v2:172.21.15.98:6808/2308139585,v1:172.21.15.98:6809/2308139585] conn(0x7f7390074f50 0x7f7390077390 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.716+0000 7f73b2f7a640 1 -- 172.21.15.1:0/2951244894 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f739006b8d0 msgr2=0x7f739006dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.716+0000 7f73b2f7a640 1 --2- 172.21.15.1:0/2951244894 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f739006b8d0 0x7f739006dd90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f73ac070e10 tx=0x7f7398007b50 comp rx=0 tx=0).stop 2024-03-06T07:20:40.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.716+0000 7f73b2f7a640 1 -- 172.21.15.1:0/2951244894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73ac0c39c0 msgr2=0x7f73ac1e9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.716+0000 7f73b2f7a640 1 --2- 172.21.15.1:0/2951244894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73ac0c39c0 0x7f73ac1e9cf0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f73a000fd80 tx=0x7f73a000d6c0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.716+0000 7f73b1517640 1 -- 172.21.15.1:0/2951244894 reap_dead start 2024-03-06T07:20:40.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.716+0000 7f73b2f7a640 1 -- 172.21.15.1:0/2951244894 shutdown_connections 2024-03-06T07:20:40.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.716+0000 7f73b2f7a640 1 -- 172.21.15.1:0/2951244894 >> 172.21.15.1:0/2951244894 conn(0x7f73ac0c7a50 msgr2=0x7f73ac0aa050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.717+0000 7f73b2f7a640 1 -- 172.21.15.1:0/2951244894 shutdown_connections 2024-03-06T07:20:40.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.718+0000 7f73b2f7a640 1 -- 172.21.15.1:0/2951244894 wait complete. 2024-03-06T07:20:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.719+0000 7fea1544e640 1 Processor -- start 2024-03-06T07:20:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.719+0000 7fea1544e640 1 -- start start 2024-03-06T07:20:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.720+0000 7fea1544e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fea10156ae0 0x7fea10159fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.720+0000 7fea1544e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea100c33c0 0x7fea100c3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.720+0000 7fea1544e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fea100c6730 0x7fea100c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.721+0000 7fea1544e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea1015a500 con 0x7fea100c6730 2024-03-06T07:20:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.721+0000 7fea1544e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea1006c100 con 0x7fea10156ae0 2024-03-06T07:20:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.721+0000 7fea1544e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea100c4980 con 0x7fea100c33c0 2024-03-06T07:20:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.725+0000 7fea0e7fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea100c33c0 0x7fea100c3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.725+0000 7fea0e7fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea100c33c0 0x7fea100c3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:41360/0 (socket says 172.21.15.1:41360) 2024-03-06T07:20:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.725+0000 7fea0e7fc640 1 -- 172.21.15.1:0/2429074926 learned_addr learned my addr 172.21.15.1:0/2429074926 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.725+0000 7fea0e7fc640 1 -- 172.21.15.1:0/2429074926 >> v2:172.21.15.98:3300/0 conn(0x7fea10156ae0 msgr2=0x7fea10159fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:20:40.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.725+0000 7fea0e7fc640 1 --2- 172.21.15.1:0/2429074926 >> v2:172.21.15.98:3300/0 conn(0x7fea10156ae0 0x7fea10159fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.725+0000 7fea0e7fc640 1 -- 172.21.15.1:0/2429074926 >> v2:172.21.15.1:3300/0 conn(0x7fea100c6730 msgr2=0x7fea100c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:40.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.725+0000 7fea0e7fc640 1 --2- 172.21.15.1:0/2429074926 >> v2:172.21.15.1:3300/0 conn(0x7fea100c6730 0x7fea100c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.725+0000 7fea0e7fc640 1 -- 172.21.15.1:0/2429074926 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea100c4c60 con 0x7fea100c33c0 2024-03-06T07:20:40.759 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-06T07:20:40.759 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:20:40.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.726+0000 7fea0e7fc640 1 --2- 172.21.15.1:0/2429074926 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea100c33c0 0x7fea100c3820 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fea100aeb10 tx=0x7fea0002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac0589fd0f3917c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.727+0000 7fea0dffb640 1 -- 172.21.15.1:0/2429074926 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea00035020 con 0x7fea100c33c0 2024-03-06T07:20:40.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.727+0000 7fea0dffb640 1 -- 172.21.15.1:0/2429074926 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea00031830 con 0x7fea100c33c0 2024-03-06T07:20:40.761 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-06T07:20:40.761 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:20:40.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.728+0000 7fea1544e640 1 -- 172.21.15.1:0/2429074926 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea100c33c0 msgr2=0x7fea100c3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.728+0000 7fea1544e640 1 --2- 172.21.15.1:0/2429074926 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea100c33c0 0x7fea100c3820 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fea100aeb10 tx=0x7fea0002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.728+0000 7fea1544e640 1 -- 172.21.15.1:0/2429074926 shutdown_connections 2024-03-06T07:20:40.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.728+0000 7fea1544e640 1 --2- 172.21.15.1:0/2429074926 >> v2:172.21.15.1:3300/0 conn(0x7fea100c6730 0x7fea100c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.728+0000 7fea1544e640 1 --2- 172.21.15.1:0/2429074926 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea100c33c0 0x7fea100c3820 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.728+0000 7fea1544e640 1 --2- 172.21.15.1:0/2429074926 >> v2:172.21.15.98:3300/0 conn(0x7fea10156ae0 0x7fea10159fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.728+0000 7fea1544e640 1 -- 172.21.15.1:0/2429074926 >> 172.21.15.1:0/2429074926 conn(0x7fea100c52a0 msgr2=0x7fea100c5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.728+0000 7fea1544e640 1 -- 172.21.15.1:0/2429074926 shutdown_connections 2024-03-06T07:20:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.728+0000 7fea1544e640 1 -- 172.21.15.1:0/2429074926 wait complete. 2024-03-06T07:20:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.730+0000 7fea1544e640 1 Processor -- start 2024-03-06T07:20:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.730+0000 7fea1544e640 1 -- start start 2024-03-06T07:20:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.731+0000 7fea1544e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea100c33c0 0x7fea100a8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.731+0000 7fea1544e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea100c6730 0x7fea100bd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.732+0000 7fea0e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea100c6730 0x7fea100bd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.732+0000 7fea0effd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea100c33c0 0x7fea100a8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.732+0000 7fea0effd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea100c33c0 0x7fea100a8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:41374/0 (socket says 172.21.15.1:41374) 2024-03-06T07:20:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.732+0000 7fea0e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea100c6730 0x7fea100bd9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40518/0 (socket says 172.21.15.1:40518) 2024-03-06T07:20:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.732+0000 7fea0effd640 1 -- 172.21.15.1:0/3783923442 learned_addr learned my addr 172.21.15.1:0/3783923442 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.732+0000 7fea1544e640 1 --2- 172.21.15.1:0/3783923442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea10156ae0 0x7fea101f2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.732+0000 7fea1544e640 1 -- 172.21.15.1:0/3783923442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea10156ec0 con 0x7fea100c6730 2024-03-06T07:20:40.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.732+0000 7fea1544e640 1 -- 172.21.15.1:0/3783923442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1006c350 con 0x7fea10156ae0 2024-03-06T07:20:40.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.732+0000 7fea1544e640 1 -- 172.21.15.1:0/3783923442 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea100c3e40 con 0x7fea100c33c0 2024-03-06T07:20:40.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.732+0000 7fea0f7fe640 1 --2- 172.21.15.1:0/3783923442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea10156ae0 0x7fea101f2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.733+0000 7fea0f7fe640 1 -- 172.21.15.1:0/3783923442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea100c33c0 msgr2=0x7fea100a8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.733+0000 7fea0f7fe640 1 --2- 172.21.15.1:0/3783923442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea100c33c0 0x7fea100a8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.733+0000 7fea0f7fe640 1 -- 172.21.15.1:0/3783923442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea100c6730 msgr2=0x7fea100bd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.733+0000 7fea0f7fe640 1 --2- 172.21.15.1:0/3783923442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea100c6730 0x7fea100bd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.733+0000 7fea0f7fe640 1 -- 172.21.15.1:0/3783923442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea101efa10 con 0x7fea10156ae0 2024-03-06T07:20:40.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.733+0000 7fea0e7fc640 1 --2- 172.21.15.1:0/3783923442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea100c6730 0x7fea100bd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:40.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.733+0000 7fea0f7fe640 1 --2- 172.21.15.1:0/3783923442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea10156ae0 0x7fea101f2020 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fea0800be20 tx=0x7fea0800bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.734+0000 7fe9f77fe640 1 -- 172.21.15.1:0/3783923442 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08019070 con 0x7fea10156ae0 2024-03-06T07:20:40.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.734+0000 7fe9f77fe640 1 -- 172.21.15.1:0/3783923442 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea08014440 con 0x7fea10156ae0 2024-03-06T07:20:40.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.734+0000 7fe9f77fe640 1 -- 172.21.15.1:0/3783923442 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08014830 con 0x7fea10156ae0 2024-03-06T07:20:40.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.734+0000 7fea1544e640 1 -- 172.21.15.1:0/3783923442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea101eec50 con 0x7fea10156ae0 2024-03-06T07:20:40.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.734+0000 7fea1544e640 1 -- 172.21.15.1:0/3783923442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea101ef210 con 0x7fea10156ae0 2024-03-06T07:20:40.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.735+0000 7fe9f57fa640 1 -- 172.21.15.1:0/3783923442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe9e8001030 con 0x7fea10156ae0 2024-03-06T07:20:40.767 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-06T07:20:40.767 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:20:40.768 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-06T07:20:40.769 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:20:40.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.736+0000 7fe9f77fe640 1 -- 172.21.15.1:0/3783923442 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea0801c070 con 0x7fea10156ae0 2024-03-06T07:20:40.770 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-06T07:20:40.770 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:20:40.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.738+0000 7fe9f77fe640 1 --2- 172.21.15.1:0/3783923442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9fc06b5a0 0x7fe9fc06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.738+0000 7fe9f77fe640 1 -- 172.21.15.1:0/3783923442 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fea0808a3b0 con 0x7fea10156ae0 2024-03-06T07:20:40.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.738+0000 7fc9b23a2640 1 -- 172.21.15.1:0/1411711393 --> [v2:172.21.15.1:6816/3842251104,v1:172.21.15.1:6818/3842251104] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc9ac0c5d90 con 0x7fc990074c90 2024-03-06T07:20:40.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.738+0000 7fe9f77fe640 1 --2- 172.21.15.1:0/3783923442 >> [v2:172.21.15.1:6800/113114752,v1:172.21.15.1:6801/113114752] conn(0x7fe9fc074c90 0x7fe9fc0770d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.738+0000 7fe9f77fe640 1 -- 172.21.15.1:0/3783923442 --> [v2:172.21.15.1:6800/113114752,v1:172.21.15.1:6801/113114752] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9fc0777a0 con 0x7fe9fc074c90 2024-03-06T07:20:40.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.738+0000 7fe9f77fe640 1 -- 172.21.15.1:0/3783923442 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea08091050 con 0x7fea10156ae0 2024-03-06T07:20:40.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.738+0000 7fea0effd640 1 --2- 172.21.15.1:0/3783923442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9fc06b5a0 0x7fe9fc06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.738+0000 7fea0e7fc640 1 --2- 172.21.15.1:0/3783923442 >> [v2:172.21.15.1:6800/113114752,v1:172.21.15.1:6801/113114752] conn(0x7fe9fc074c90 0x7fe9fc0770d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.739+0000 7fc9a1ffb640 1 -- 172.21.15.1:0/1411711393 <== osd.0 v2:172.21.15.1:6816/3842251104 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc9ac0c5d90 con 0x7fc990074c90 2024-03-06T07:20:40.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.739+0000 7fea0effd640 1 --2- 172.21.15.1:0/3783923442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9fc06b5a0 0x7fe9fc06da60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fea04006fd0 tx=0x7fea04008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.739+0000 7fc9b23a2640 1 -- 172.21.15.1:0/1411711393 >> [v2:172.21.15.1:6816/3842251104,v1:172.21.15.1:6818/3842251104] conn(0x7fc990074c90 msgr2=0x7fc9900770d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.739+0000 7fc9b23a2640 1 --2- 172.21.15.1:0/1411711393 >> [v2:172.21.15.1:6816/3842251104,v1:172.21.15.1:6818/3842251104] conn(0x7fc990074c90 0x7fc9900770d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.739+0000 7fea0e7fc640 1 --2- 172.21.15.1:0/3783923442 >> [v2:172.21.15.1:6800/113114752,v1:172.21.15.1:6801/113114752] conn(0x7fe9fc074c90 0x7fe9fc0770d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.739+0000 7fc9b23a2640 1 -- 172.21.15.1:0/1411711393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc99006b5a0 msgr2=0x7fc99006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.739+0000 7fc9b23a2640 1 --2- 172.21.15.1:0/1411711393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc99006b5a0 0x7fc99006da60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc99c009630 tx=0x7fc99c03c040 comp rx=0 tx=0).stop 2024-03-06T07:20:40.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.739+0000 7fc9b23a2640 1 -- 172.21.15.1:0/1411711393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9ac154c90 msgr2=0x7fc9ac141f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.739+0000 7fc9b23a2640 1 --2- 172.21.15.1:0/1411711393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9ac154c90 0x7fc9ac141f40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4007e90 tx=0x7fc9a400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.739+0000 7fc9b1ba1640 1 -- 172.21.15.1:0/1411711393 reap_dead start 2024-03-06T07:20:40.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.739+0000 7fc9b23a2640 1 -- 172.21.15.1:0/1411711393 shutdown_connections 2024-03-06T07:20:40.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.739+0000 7fc9b23a2640 1 -- 172.21.15.1:0/1411711393 >> 172.21.15.1:0/1411711393 conn(0x7fc9ac0c6e60 msgr2=0x7fc9ac0713b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.739+0000 7fc9b23a2640 1 -- 172.21.15.1:0/1411711393 shutdown_connections 2024-03-06T07:20:40.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.739+0000 7fc9b23a2640 1 -- 172.21.15.1:0/1411711393 wait complete. 2024-03-06T07:20:40.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.740+0000 7ff9c1ffb640 1 -- 172.21.15.1:0/834956691 --> [v2:172.21.15.98:6800/2463344330,v1:172.21.15.98:6801/2463344330] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff9dc1fb0f0 con 0x7ff9c8074b40 2024-03-06T07:20:40.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.740+0000 7fe9f77fe640 1 -- 172.21.15.1:0/3783923442 <== osd.2 v2:172.21.15.1:6800/113114752 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe9fc0777a0 con 0x7fe9fc074c90 2024-03-06T07:20:40.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.740+0000 7ff9c3fff640 1 -- 172.21.15.1:0/834956691 <== osd.6 v2:172.21.15.98:6800/2463344330 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff9dc1fb0f0 con 0x7ff9c8074b40 2024-03-06T07:20:40.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.741+0000 7ff9c1ffb640 1 -- 172.21.15.1:0/834956691 >> [v2:172.21.15.98:6800/2463344330,v1:172.21.15.98:6801/2463344330] conn(0x7ff9c8074b40 msgr2=0x7ff9c8076f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.741+0000 7ff9c1ffb640 1 --2- 172.21.15.1:0/834956691 >> [v2:172.21.15.98:6800/2463344330,v1:172.21.15.98:6801/2463344330] conn(0x7ff9c8074b40 0x7ff9c8076f80 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.741+0000 7ff9c1ffb640 1 -- 172.21.15.1:0/834956691 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff9c806b4d0 msgr2=0x7ff9c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.741+0000 7ff9c1ffb640 1 --2- 172.21.15.1:0/834956691 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff9c806b4d0 0x7ff9c806d990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0001780 tx=0x7ff9d0009040 comp rx=0 tx=0).stop 2024-03-06T07:20:40.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.741+0000 7ff9c1ffb640 1 -- 172.21.15.1:0/834956691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9dc14e940 msgr2=0x7ff9dc14d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.741+0000 7ff9c1ffb640 1 --2- 172.21.15.1:0/834956691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9dc14e940 0x7ff9dc14d4e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00c480 tx=0x7ff9cc00c940 comp rx=0 tx=0).stop 2024-03-06T07:20:40.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.741+0000 7ff9dbfff640 1 -- 172.21.15.1:0/834956691 reap_dead start 2024-03-06T07:20:40.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.741+0000 7ff9c1ffb640 1 -- 172.21.15.1:0/834956691 shutdown_connections 2024-03-06T07:20:40.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.741+0000 7ff9c1ffb640 1 -- 172.21.15.1:0/834956691 >> 172.21.15.1:0/834956691 conn(0x7ff9dc0c6e60 msgr2=0x7ff9dc072240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.741+0000 7ff9c1ffb640 1 -- 172.21.15.1:0/834956691 shutdown_connections 2024-03-06T07:20:40.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.741+0000 7ff9c1ffb640 1 -- 172.21.15.1:0/834956691 wait complete. 2024-03-06T07:20:40.778 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-06T07:20:40.778 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:20:40.779 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-06T07:20:40.780 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:20:40.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.760+0000 7fe9f57fa640 1 -- 172.21.15.1:0/3783923442 --> [v2:172.21.15.1:6800/113114752,v1:172.21.15.1:6801/113114752] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe9e8002dd0 con 0x7fe9fc074c90 2024-03-06T07:20:40.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.761+0000 7fe9f77fe640 1 -- 172.21.15.1:0/3783923442 <== osd.2 v2:172.21.15.1:6800/113114752 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe9e8002dd0 con 0x7fe9fc074c90 2024-03-06T07:20:40.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.761+0000 7fe9f57fa640 1 -- 172.21.15.1:0/3783923442 >> [v2:172.21.15.1:6800/113114752,v1:172.21.15.1:6801/113114752] conn(0x7fe9fc074c90 msgr2=0x7fe9fc0770d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.761+0000 7fe9f57fa640 1 --2- 172.21.15.1:0/3783923442 >> [v2:172.21.15.1:6800/113114752,v1:172.21.15.1:6801/113114752] conn(0x7fe9fc074c90 0x7fe9fc0770d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.761+0000 7fe9f57fa640 1 -- 172.21.15.1:0/3783923442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9fc06b5a0 msgr2=0x7fe9fc06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.761+0000 7fe9f57fa640 1 --2- 172.21.15.1:0/3783923442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9fc06b5a0 0x7fe9fc06da60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fea04006fd0 tx=0x7fea04008040 comp rx=0 tx=0).stop 2024-03-06T07:20:40.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.761+0000 7fe9f57fa640 1 -- 172.21.15.1:0/3783923442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea10156ae0 msgr2=0x7fea101f2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.761+0000 7fe9f57fa640 1 --2- 172.21.15.1:0/3783923442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea10156ae0 0x7fea101f2020 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fea0800be20 tx=0x7fea0800bf10 comp rx=0 tx=0).stop 2024-03-06T07:20:40.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.761+0000 7fea0f7fe640 1 -- 172.21.15.1:0/3783923442 reap_dead start 2024-03-06T07:20:40.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.761+0000 7fe9f57fa640 1 -- 172.21.15.1:0/3783923442 shutdown_connections 2024-03-06T07:20:40.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.761+0000 7fe9f57fa640 1 -- 172.21.15.1:0/3783923442 >> 172.21.15.1:0/3783923442 conn(0x7fea100c52a0 msgr2=0x7fea10157370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.761+0000 7fe9f57fa640 1 -- 172.21.15.1:0/3783923442 shutdown_connections 2024-03-06T07:20:40.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.761+0000 7fe9f57fa640 1 -- 172.21.15.1:0/3783923442 wait complete. 2024-03-06T07:20:40.783 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-06T07:20:40.783 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:20:40.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.987+0000 7fe4511e0640 1 Processor -- start 2024-03-06T07:20:40.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.987+0000 7fe4511e0640 1 -- start start 2024-03-06T07:20:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.987+0000 7f5ed63f6640 1 Processor -- start 2024-03-06T07:20:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.987+0000 7f5ed63f6640 1 -- start start 2024-03-06T07:20:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.988+0000 7f5ed63f6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5ec800bd40 0x7f5ec8046b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.988+0000 7f5ed63f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ec80d6700 0x7f5ec80d6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.988+0000 7f5ed63f6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec8047040 con 0x7f5ec804b100 2024-03-06T07:20:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.988+0000 7f5ed63f6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ec80d9f30 con 0x7f5ec800bd40 2024-03-06T07:20:40.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.988+0000 7f5ed63f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec80da210 con 0x7f5ec80d6700 2024-03-06T07:20:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.988+0000 7fe4511e0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe4440d6fc0 0x7fe44404c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.988+0000 7fe4511e0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44404b500 con 0x7fe444049df0 2024-03-06T07:20:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.988+0000 7fe4511e0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4440d75a0 con 0x7fe4440d4650 2024-03-06T07:20:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.988+0000 7fe4511e0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4440d4a30 con 0x7fe4440d6fc0 2024-03-06T07:20:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.989+0000 7f5ecf7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5ec804b100 0x7f5ec8006ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57434/0 (socket says 172.21.15.1:57434) 2024-03-06T07:20:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.989+0000 7f5ecf7fe640 1 -- 172.21.15.1:0/3352540515 learned_addr learned my addr 172.21.15.1:0/3352540515 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:40.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.989+0000 7f5ed4993640 1 --2- 172.21.15.1:0/3352540515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ec80d6700 0x7f5ec80d6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.989+0000 7fe44bfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe4440d4650 0x7fe4440471d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:55056/0 (socket says 172.21.15.1:55056) 2024-03-06T07:20:40.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.989+0000 7fe44bfff640 1 -- 172.21.15.1:0/1519965244 learned_addr learned my addr 172.21.15.1:0/1519965244 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:40.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.989+0000 7fe4509df640 1 --2- 172.21.15.1:0/1519965244 >> v2:172.21.15.1:3300/0 conn(0x7fe4440d6fc0 0x7fe44404c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.989+0000 7fe44affd640 1 -- 172.21.15.1:0/1519965244 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064804401 0 0) 0x7fe4440d75a0 con 0x7fe4440d4650 2024-03-06T07:20:40.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.989+0000 7fe44affd640 1 -- 172.21.15.1:0/1519965244 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe43c003520 con 0x7fe4440d4650 2024-03-06T07:20:40.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.989+0000 7f5ed4993640 1 -- 172.21.15.1:0/3352540515 >> v2:172.21.15.98:3300/0 conn(0x7f5ec800bd40 msgr2=0x7f5ec8046b00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:20:40.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.989+0000 7f5ed4993640 1 --2- 172.21.15.1:0/3352540515 >> v2:172.21.15.98:3300/0 conn(0x7f5ec800bd40 0x7f5ec8046b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.989+0000 7f5ed4993640 1 -- 172.21.15.1:0/3352540515 >> v1:172.21.15.1:6789/0 conn(0x7f5ec804b100 legacy=0x7f5ec8006ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.989+0000 7f5ed4993640 1 -- 172.21.15.1:0/3352540515 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec80da4f0 con 0x7f5ec80d6700 2024-03-06T07:20:40.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.989+0000 7fe44affd640 1 -- 172.21.15.1:0/1519965244 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486211522 0 0) 0x7fe44404b500 con 0x7fe444049df0 2024-03-06T07:20:40.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.990+0000 7fe44affd640 1 -- 172.21.15.1:0/1519965244 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4440d75a0 con 0x7fe444049df0 2024-03-06T07:20:40.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.990+0000 7fe44affd640 1 -- 172.21.15.1:0/1519965244 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2177473588 0 0) 0x7fe43c003520 con 0x7fe4440d4650 2024-03-06T07:20:40.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.990+0000 7fe4509df640 1 -- 172.21.15.1:0/1519965244 >> v1:172.21.15.98:6789/0 conn(0x7fe4440d4650 legacy=0x7fe4440471d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.990+0000 7fe4509df640 1 -- 172.21.15.1:0/1519965244 >> v1:172.21.15.1:6789/0 conn(0x7fe444049df0 legacy=0x7fe44404ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.990+0000 7fe4509df640 1 -- 172.21.15.1:0/1519965244 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4440d7880 con 0x7fe4440d6fc0 2024-03-06T07:20:40.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.990+0000 7fab8dafc640 1 Processor -- start 2024-03-06T07:20:40.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.990+0000 7f5ed4993640 1 --2- 172.21.15.1:0/3352540515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ec80d6700 0x7f5ec80d6b90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0000b60 tx=0x7f5ed00a7db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d1dc6c2a7519ce3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.991+0000 7fab8dafc640 1 -- start start 2024-03-06T07:20:40.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.991+0000 7fe4509df640 1 --2- 172.21.15.1:0/1519965244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4440d6fc0 0x7fe44404c110 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe44c000b60 tx=0x7fe44c0a6da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=257dbeddaf052e48 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.991+0000 7fe44affd640 1 -- 172.21.15.1:0/1519965244 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c0b4070 con 0x7fe4440d6fc0 2024-03-06T07:20:40.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.991+0000 7fe44affd640 1 -- 172.21.15.1:0/1519965244 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe44c06e040 con 0x7fe4440d6fc0 2024-03-06T07:20:40.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.990+0000 7f5eceffd640 1 -- 172.21.15.1:0/3352540515 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed00b5070 con 0x7f5ec80d6700 2024-03-06T07:20:40.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.990+0000 7f5eceffd640 1 -- 172.21.15.1:0/3352540515 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed006e040 con 0x7f5ec80d6700 2024-03-06T07:20:40.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.990+0000 7f5eceffd640 1 -- 172.21.15.1:0/3352540515 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed00a8e60 con 0x7f5ec80d6700 2024-03-06T07:20:40.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.991+0000 7f5ed63f6640 1 -- 172.21.15.1:0/3352540515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ec80d6700 msgr2=0x7f5ec80d6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.991+0000 7f5ed63f6640 1 --2- 172.21.15.1:0/3352540515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ec80d6700 0x7f5ec80d6b90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0000b60 tx=0x7f5ed00a7db0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.991+0000 7f5ed63f6640 1 -- 172.21.15.1:0/3352540515 shutdown_connections 2024-03-06T07:20:40.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.991+0000 7f5ed63f6640 1 --2- 172.21.15.1:0/3352540515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ec80d6700 0x7f5ec80d6b90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.991+0000 7f5ed63f6640 1 --2- 172.21.15.1:0/3352540515 >> v2:172.21.15.98:3300/0 conn(0x7f5ec800bd40 0x7f5ec8046b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.991+0000 7f5ed63f6640 1 -- 172.21.15.1:0/3352540515 >> 172.21.15.1:0/3352540515 conn(0x7f5ec8048240 msgr2=0x7f5ec8048650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.992+0000 7fab8dafc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fab8000f890 0x7fab8000e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.992+0000 7f5ed63f6640 1 -- 172.21.15.1:0/3352540515 shutdown_connections 2024-03-06T07:20:40.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.992+0000 7fab8dafc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8000e1d0 con 0x7fab800d7b20 2024-03-06T07:20:40.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.992+0000 7fab8dafc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab800de080 con 0x7fab8004dd20 2024-03-06T07:20:40.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.992+0000 7fab8dafc640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab800de360 con 0x7fab8000f890 2024-03-06T07:20:40.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.992+0000 7fab87fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fab8004dd20 0x7fab80007220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:55060/0 (socket says 172.21.15.1:55060) 2024-03-06T07:20:40.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.992+0000 7fab87fff640 1 -- 172.21.15.1:0/3936096809 learned_addr learned my addr 172.21.15.1:0/3936096809 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:40.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.992+0000 7fab8cafa640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fab800d7b20 0x7fab800dd8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57456/0 (socket says 172.21.15.1:57456) 2024-03-06T07:20:40.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.992+0000 7fab8d2fb640 1 --2- 172.21.15.1:0/3936096809 >> v2:172.21.15.1:3300/0 conn(0x7fab8000f890 0x7fab8000e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.992+0000 7fe44affd640 1 -- 172.21.15.1:0/1519965244 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe440001670 con 0x7fe4440d6fc0 2024-03-06T07:20:40.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.992+0000 7fe4511e0640 1 -- 172.21.15.1:0/1519965244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4440d6fc0 msgr2=0x7fe44404c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.992+0000 7fe4511e0640 1 --2- 172.21.15.1:0/1519965244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4440d6fc0 0x7fe44404c110 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe44c000b60 tx=0x7fe44c0a6da0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.993+0000 7fe4511e0640 1 -- 172.21.15.1:0/1519965244 shutdown_connections 2024-03-06T07:20:40.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.993+0000 7fe4511e0640 1 --2- 172.21.15.1:0/1519965244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4440d6fc0 0x7fe44404c110 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.993+0000 7fe4511e0640 1 -- 172.21.15.1:0/1519965244 >> 172.21.15.1:0/1519965244 conn(0x7fe4440492d0 msgr2=0x7fe4440496e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.993+0000 7fab877fe640 1 -- 172.21.15.1:0/3936096809 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835053437 0 0) 0x7fab800de080 con 0x7fab8004dd20 2024-03-06T07:20:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.993+0000 7fab877fe640 1 -- 172.21.15.1:0/3936096809 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab78003520 con 0x7fab8004dd20 2024-03-06T07:20:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.993+0000 7fab8d2fb640 1 -- 172.21.15.1:0/3936096809 >> v1:172.21.15.98:6789/0 conn(0x7fab8004dd20 legacy=0x7fab80007220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.993+0000 7fab8d2fb640 1 -- 172.21.15.1:0/3936096809 >> v1:172.21.15.1:6789/0 conn(0x7fab800d7b20 legacy=0x7fab800dd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.993+0000 7fab8d2fb640 1 -- 172.21.15.1:0/3936096809 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab800de640 con 0x7fab8000f890 2024-03-06T07:20:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.992+0000 7f5ed63f6640 1 -- 172.21.15.1:0/3352540515 wait complete. 2024-03-06T07:20:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.994+0000 7f5ed63f6640 1 Processor -- start 2024-03-06T07:20:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.994+0000 7f5ed63f6640 1 -- start start 2024-03-06T07:20:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.993+0000 7fe4511e0640 1 -- 172.21.15.1:0/1519965244 shutdown_connections 2024-03-06T07:20:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.994+0000 7fab8d2fb640 1 --2- 172.21.15.1:0/3936096809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab8000f890 0x7fab8000e990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fab8804f850 tx=0x7fab880a6ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d90ff021e6b61a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.994+0000 7fab877fe640 1 -- 172.21.15.1:0/3936096809 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab880b4070 con 0x7fab8000f890 2024-03-06T07:20:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.994+0000 7fab877fe640 1 -- 172.21.15.1:0/3936096809 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab8806e040 con 0x7fab8000f890 2024-03-06T07:20:41.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.993+0000 7fe4511e0640 1 -- 172.21.15.1:0/1519965244 wait complete. 2024-03-06T07:20:41.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.995+0000 7f5ed63f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ec800bd40 0x7f5ec80308e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.995+0000 7fe4511e0640 1 Processor -- start 2024-03-06T07:20:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.995+0000 7f5ed63f6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ec804b100 0x7f5ec81780e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.995+0000 7f5ed63f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ec80d6700 0x7f5ec8174190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.995+0000 7f5ed63f6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec80075f0 con 0x7f5ec804b100 2024-03-06T07:20:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.995+0000 7f5ed63f6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec80d9920 con 0x7f5ec800bd40 2024-03-06T07:20:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.995+0000 7f5ed63f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec8007770 con 0x7f5ec80d6700 2024-03-06T07:20:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.995+0000 7f5ecf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ec804b100 0x7f5ec81780e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.995+0000 7f5ecf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ec804b100 0x7f5ec81780e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40542/0 (socket says 172.21.15.1:40542) 2024-03-06T07:20:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.995+0000 7f5ecf7fe640 1 -- 172.21.15.1:0/3167207022 learned_addr learned my addr 172.21.15.1:0/3167207022 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.995+0000 7f5ecffff640 1 --2- 172.21.15.1:0/3167207022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ec800bd40 0x7f5ec80308e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.995+0000 7fab877fe640 1 -- 172.21.15.1:0/3936096809 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab880af440 con 0x7fab8000f890 2024-03-06T07:20:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7f0a7be88640 1 Processor -- start 2024-03-06T07:20:41.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7fab8dafc640 1 -- 172.21.15.1:0/3936096809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab8000f890 msgr2=0x7fab8000e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7fab8dafc640 1 --2- 172.21.15.1:0/3936096809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab8000f890 0x7fab8000e990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fab8804f850 tx=0x7fab880a6ef0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7f0a7be88640 1 -- start start 2024-03-06T07:20:41.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.995+0000 7fe4511e0640 1 -- start start 2024-03-06T07:20:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7f5ecffff640 1 -- 172.21.15.1:0/3167207022 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ec80d6700 msgr2=0x7f5ec8174190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.995+0000 7f5ed4993640 1 --2- 172.21.15.1:0/3167207022 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ec80d6700 0x7f5ec8174190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7f5ecffff640 1 --2- 172.21.15.1:0/3167207022 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ec80d6700 0x7f5ec8174190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7f5ecffff640 1 -- 172.21.15.1:0/3167207022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ec804b100 msgr2=0x7f5ec81780e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7f5ecffff640 1 --2- 172.21.15.1:0/3167207022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ec804b100 0x7f5ec81780e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7f5ecffff640 1 -- 172.21.15.1:0/3167207022 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec8178620 con 0x7f5ec800bd40 2024-03-06T07:20:41.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7f5ed4993640 1 --2- 172.21.15.1:0/3167207022 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ec80d6700 0x7f5ec8174190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:41.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7f5ecf7fe640 1 --2- 172.21.15.1:0/3167207022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ec804b100 0x7f5ec81780e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:41.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7fe4511e0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe444049df0 0x7fe444045d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7fab8dafc640 1 -- 172.21.15.1:0/3936096809 shutdown_connections 2024-03-06T07:20:41.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7fab8dafc640 1 --2- 172.21.15.1:0/3936096809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab8000f890 0x7fab8000e990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7fab8dafc640 1 -- 172.21.15.1:0/3936096809 >> 172.21.15.1:0/3936096809 conn(0x7fab8004c680 msgr2=0x7fab8004ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:41.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7f5ecffff640 1 --2- 172.21.15.1:0/3167207022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ec800bd40 0x7f5ec80308e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ec400b3e0 tx=0x7f5ec400b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7fab8dafc640 1 -- 172.21.15.1:0/3936096809 shutdown_connections 2024-03-06T07:20:41.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7fab8dafc640 1 -- 172.21.15.1:0/3936096809 wait complete. 2024-03-06T07:20:41.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7fe4511e0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4440d4650 0x7fe4440d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7fe4511e0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4440d6fc0 0x7fe4440d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7fe4511e0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44400ca10 con 0x7fe444049df0 2024-03-06T07:20:41.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7f0a7be88640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0a740c5d90 0x7f0a7406d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7f0a7be88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a7415a0e0 0x7f0a740c48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7f0a7be88640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7406df30 con 0x7f0a740c8100 2024-03-06T07:20:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7f0a7be88640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a74159c50 con 0x7f0a740c5d90 2024-03-06T07:20:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7f0a7be88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7406e210 con 0x7f0a7415a0e0 2024-03-06T07:20:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7f0a79c24640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0a740c8100 0x7f0a7415cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57460/0 (socket says 172.21.15.1:57460) 2024-03-06T07:20:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7f0a79c24640 1 -- 172.21.15.1:0/1430368405 learned_addr learned my addr 172.21.15.1:0/1430368405 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7f0a79423640 1 --2- 172.21.15.1:0/1430368405 >> v2:172.21.15.1:3300/0 conn(0x7f0a740c5d90 0x7f0a7406d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7f0a7a425640 1 --2- 172.21.15.1:0/1430368405 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a7415a0e0 0x7f0a740c48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7f5eccff9640 1 -- 172.21.15.1:0/3167207022 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec4015020 con 0x7f5ec800bd40 2024-03-06T07:20:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7f5eccff9640 1 -- 172.21.15.1:0/3167207022 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec4011850 con 0x7f5ec800bd40 2024-03-06T07:20:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7f5eccff9640 1 -- 172.21.15.1:0/3167207022 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec4011c40 con 0x7f5ec800bd40 2024-03-06T07:20:41.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7f5ed63f6640 1 -- 172.21.15.1:0/3167207022 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec802eb90 con 0x7f5ec800bd40 2024-03-06T07:20:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7f5ed63f6640 1 -- 172.21.15.1:0/3167207022 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec80d5c70 con 0x7f5ec800bd40 2024-03-06T07:20:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7fae757c7640 1 Processor -- start 2024-03-06T07:20:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7fae757c7640 1 -- start start 2024-03-06T07:20:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7fae757c7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae700c3720 0x7fae70159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7fae757c7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae700c1a10 0x7fae700c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7fae757c7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae7015cce0 0x7fae7015d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7fae757c7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7006d760 con 0x7fae700c1a10 2024-03-06T07:20:41.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7fae757c7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7006da40 con 0x7fae7015cce0 2024-03-06T07:20:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7fae757c7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7006dd20 con 0x7fae700c3720 2024-03-06T07:20:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.998+0000 7fae6e7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae700c1a10 0x7fae700c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.998+0000 7fae6e7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae700c1a10 0x7fae700c3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40584/0 (socket says 172.21.15.1:40584) 2024-03-06T07:20:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.998+0000 7fae6e7fc640 1 -- 172.21.15.1:0/832151303 learned_addr learned my addr 172.21.15.1:0/832151303 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.998+0000 7fae6effd640 1 --2- 172.21.15.1:0/832151303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae700c3720 0x7fae70159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.998+0000 7fae6f7fe640 1 --2- 172.21.15.1:0/832151303 >> v2:172.21.15.98:3300/0 conn(0x7fae7015cce0 0x7fae7015d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.998+0000 7f0a79423640 1 -- 172.21.15.1:0/1430368405 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a7415a0e0 msgr2=0x7f0a740c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.998+0000 7f0a79423640 1 --2- 172.21.15.1:0/1430368405 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a7415a0e0 0x7f0a740c48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.998+0000 7f0a79423640 1 -- 172.21.15.1:0/1430368405 >> v1:172.21.15.1:6789/0 conn(0x7f0a740c8100 legacy=0x7f0a7415cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.998+0000 7f0a79423640 1 -- 172.21.15.1:0/1430368405 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a74159600 con 0x7f0a740c5d90 2024-03-06T07:20:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7fe4511e0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4440d7e40 con 0x7fe4440d4650 2024-03-06T07:20:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.996+0000 7fe4511e0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4440478e0 con 0x7fe4440d6fc0 2024-03-06T07:20:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7fe44b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4440d4650 0x7fe4440d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7fe44b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4440d4650 0x7fe4440d0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:44542/0 (socket says 172.21.15.1:44542) 2024-03-06T07:20:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.997+0000 7fe44b7fe640 1 -- 172.21.15.1:0/3020782893 learned_addr learned my addr 172.21.15.1:0/3020782893 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fe44bfff640 1 --2- 172.21.15.1:0/3020782893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe444049df0 0x7fe444045d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.998+0000 7fae6effd640 1 -- 172.21.15.1:0/832151303 >> v2:172.21.15.98:3300/0 conn(0x7fae7015cce0 msgr2=0x7fae7015d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:41.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.998+0000 7fae6effd640 1 --2- 172.21.15.1:0/832151303 >> v2:172.21.15.98:3300/0 conn(0x7fae7015cce0 0x7fae7015d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.998+0000 7fae6effd640 1 -- 172.21.15.1:0/832151303 >> v2:172.21.15.1:3300/0 conn(0x7fae700c1a10 msgr2=0x7fae700c3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:41.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.998+0000 7fae6effd640 1 --2- 172.21.15.1:0/832151303 >> v2:172.21.15.1:3300/0 conn(0x7fae700c1a10 0x7fae700c3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.998+0000 7fae6effd640 1 -- 172.21.15.1:0/832151303 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7006e000 con 0x7fae700c3720 2024-03-06T07:20:41.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.998+0000 7fab8dafc640 1 Processor -- start 2024-03-06T07:20:41.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:40.998+0000 7fab8dafc640 1 -- start start 2024-03-06T07:20:41.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fae6effd640 1 --2- 172.21.15.1:0/832151303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae700c3720 0x7fae70159eb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fae700c5c10 tx=0x7fae6402b7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=facb9d163041bf65 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fab8dafc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab8000f890 0x7fab800d3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fab8dafc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab8004dd20 0x7fab800ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fab8dafc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab800d7b20 0x7fab800ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fab8dafc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab80007ab0 con 0x7fab8000f890 2024-03-06T07:20:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fab8dafc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab80007930 con 0x7fab800d7b20 2024-03-06T07:20:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fab8dafc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8000eed0 con 0x7fab8004dd20 2024-03-06T07:20:41.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7f0a79423640 1 --2- 172.21.15.1:0/1430368405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a740c5d90 0x7f0a7406d8c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0a640097a0 tx=0x7f0a6402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dc5b93b88f20202 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fe4509df640 1 --2- 172.21.15.1:0/3020782893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4440d6fc0 0x7fe4440d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fab8d2fb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab800d7b20 0x7fab800ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fab8d2fb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab800d7b20 0x7fab800ca8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:44560/0 (socket says 172.21.15.1:44560) 2024-03-06T07:20:41.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fab8d2fb640 1 -- 172.21.15.1:0/2622522769 learned_addr learned my addr 172.21.15.1:0/2622522769 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:41.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fab87fff640 1 --2- 172.21.15.1:0/2622522769 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab8004dd20 0x7fab800ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fae6dffb640 1 -- 172.21.15.1:0/832151303 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae64038070 con 0x7fae700c3720 2024-03-06T07:20:41.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fae6dffb640 1 -- 172.21.15.1:0/832151303 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae64033620 con 0x7fae700c3720 2024-03-06T07:20:41.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fab8cafa640 1 --2- 172.21.15.1:0/2622522769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab8000f890 0x7fab800d3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7f0a78c22640 1 -- 172.21.15.1:0/1430368405 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a640317c0 con 0x7f0a740c5d90 2024-03-06T07:20:41.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7f0a78c22640 1 -- 172.21.15.1:0/1430368405 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a64037070 con 0x7f0a740c5d90 2024-03-06T07:20:41.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7f0a78c22640 1 -- 172.21.15.1:0/1430368405 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64031bc0 con 0x7f0a740c5d90 2024-03-06T07:20:41.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fe44b7fe640 1 -- 172.21.15.1:0/3020782893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4440d6fc0 msgr2=0x7fe4440d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fe44b7fe640 1 --2- 172.21.15.1:0/3020782893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4440d6fc0 0x7fe4440d12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fe44b7fe640 1 -- 172.21.15.1:0/3020782893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe444049df0 msgr2=0x7fe444045d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fe44b7fe640 1 --2- 172.21.15.1:0/3020782893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe444049df0 0x7fe444045d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.000+0000 7fe44b7fe640 1 -- 172.21.15.1:0/3020782893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe444175080 con 0x7fe4440d4650 2024-03-06T07:20:41.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.001+0000 7fab8cafa640 1 -- 172.21.15.1:0/2622522769 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab8004dd20 msgr2=0x7fab800ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.001+0000 7fab8cafa640 1 --2- 172.21.15.1:0/2622522769 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab8004dd20 0x7fab800ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.001+0000 7fab8cafa640 1 -- 172.21.15.1:0/2622522769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab800d7b20 msgr2=0x7fab800ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.001+0000 7fab8cafa640 1 --2- 172.21.15.1:0/2622522769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab800d7b20 0x7fab800ca8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.001+0000 7fab8cafa640 1 -- 172.21.15.1:0/2622522769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab800cb810 con 0x7fab8000f890 2024-03-06T07:20:41.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.001+0000 7fae757c7640 1 -- 172.21.15.1:0/832151303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae700c3720 msgr2=0x7fae70159eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.001+0000 7fae757c7640 1 --2- 172.21.15.1:0/832151303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae700c3720 0x7fae70159eb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fae700c5c10 tx=0x7fae6402b7d0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.001+0000 7fab8cafa640 1 --2- 172.21.15.1:0/2622522769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab8000f890 0x7fab800d3080 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fab7400c3c0 tx=0x7fab7400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.001+0000 7fe44bfff640 1 --2- 172.21.15.1:0/3020782893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe444049df0 0x7fe444045d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:41.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.001+0000 7f0a7be88640 1 -- 172.21.15.1:0/1430368405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a740c5d90 msgr2=0x7f0a7406d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.001+0000 7f0a7be88640 1 --2- 172.21.15.1:0/1430368405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a740c5d90 0x7f0a7406d8c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0a640097a0 tx=0x7f0a6402b860 comp rx=0 tx=0).stop 2024-03-06T07:20:41.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.001+0000 7fae757c7640 1 -- 172.21.15.1:0/832151303 shutdown_connections 2024-03-06T07:20:41.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.001+0000 7fae757c7640 1 --2- 172.21.15.1:0/832151303 >> v2:172.21.15.98:3300/0 conn(0x7fae7015cce0 0x7fae7015d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.001+0000 7fae757c7640 1 --2- 172.21.15.1:0/832151303 >> v2:172.21.15.1:3300/0 conn(0x7fae700c1a10 0x7fae700c3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.001+0000 7fae757c7640 1 --2- 172.21.15.1:0/832151303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae700c3720 0x7fae70159eb0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.001+0000 7fae757c7640 1 -- 172.21.15.1:0/832151303 >> 172.21.15.1:0/832151303 conn(0x7fae700c6860 msgr2=0x7fae700c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:41.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.002+0000 7fe44b7fe640 1 --2- 172.21.15.1:0/3020782893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4440d4650 0x7fe4440d0da0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe43800c490 tx=0x7fe43800c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.002+0000 7fae757c7640 1 -- 172.21.15.1:0/832151303 shutdown_connections 2024-03-06T07:20:41.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.002+0000 7fab857fa640 1 -- 172.21.15.1:0/2622522769 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab740074e0 con 0x7fab8000f890 2024-03-06T07:20:41.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.002+0000 7fae757c7640 1 -- 172.21.15.1:0/832151303 wait complete. 2024-03-06T07:20:41.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.002+0000 7fab857fa640 1 -- 172.21.15.1:0/2622522769 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab740047e0 con 0x7fab8000f890 2024-03-06T07:20:41.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.002+0000 7fab8dafc640 1 -- 172.21.15.1:0/2622522769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab800cb080 con 0x7fab8000f890 2024-03-06T07:20:41.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.002+0000 7fab8dafc640 1 -- 172.21.15.1:0/2622522769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab800d35c0 con 0x7fab8000f890 2024-03-06T07:20:41.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.002+0000 7fe448ff9640 1 -- 172.21.15.1:0/3020782893 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4380052d0 con 0x7fe4440d4650 2024-03-06T07:20:41.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.002+0000 7f0a7be88640 1 -- 172.21.15.1:0/1430368405 shutdown_connections 2024-03-06T07:20:41.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.002+0000 7f0a7be88640 1 --2- 172.21.15.1:0/1430368405 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a7415a0e0 0x7f0a740c48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.002+0000 7f0a7be88640 1 --2- 172.21.15.1:0/1430368405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a740c5d90 0x7f0a7406d8c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.002+0000 7f0a7be88640 1 -- 172.21.15.1:0/1430368405 >> 172.21.15.1:0/1430368405 conn(0x7f0a740c88f0 msgr2=0x7f0a740c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:41.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.002+0000 7fe448ff9640 1 -- 172.21.15.1:0/3020782893 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe438005470 con 0x7fe4440d4650 2024-03-06T07:20:41.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.002+0000 7fe448ff9640 1 -- 172.21.15.1:0/3020782893 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4380052d0 con 0x7fe4440d4650 2024-03-06T07:20:41.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.002+0000 7fe4511e0640 1 -- 172.21.15.1:0/3020782893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4441742c0 con 0x7fe4440d4650 2024-03-06T07:20:41.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.002+0000 7fe4511e0640 1 -- 172.21.15.1:0/3020782893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe444174880 con 0x7fe4440d4650 2024-03-06T07:20:41.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.002+0000 7f5ed63f6640 1 -- 172.21.15.1:0/3167207022 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec80052b0 con 0x7f5ec800bd40 2024-03-06T07:20:41.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.004+0000 7f5eccff9640 1 -- 172.21.15.1:0/3167207022 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ec4018020 con 0x7f5ec800bd40 2024-03-06T07:20:41.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.004+0000 7f5eccff9640 1 --2- 172.21.15.1:0/3167207022 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eb406b5a0 0x7f5eb406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.005+0000 7f5eccff9640 1 -- 172.21.15.1:0/3167207022 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5ec408a2b0 con 0x7f5ec800bd40 2024-03-06T07:20:41.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.005+0000 7fae757c7640 1 Processor -- start 2024-03-06T07:20:41.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.004+0000 7fe4367fc640 1 -- 172.21.15.1:0/3020782893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe444005700 con 0x7fe4440d4650 2024-03-06T07:20:41.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.005+0000 7fe448ff9640 1 -- 172.21.15.1:0/3020782893 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe438012630 con 0x7fe4440d4650 2024-03-06T07:20:41.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.004+0000 7f0a7be88640 1 -- 172.21.15.1:0/1430368405 shutdown_connections 2024-03-06T07:20:41.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.005+0000 7f0a7be88640 1 -- 172.21.15.1:0/1430368405 wait complete. 2024-03-06T07:20:41.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.005+0000 7fab857fa640 1 -- 172.21.15.1:0/2622522769 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab74004bd0 con 0x7fab8000f890 2024-03-06T07:20:41.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.005+0000 7f0235672640 1 Processor -- start 2024-03-06T07:20:41.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.005+0000 7f27ff58a640 1 Processor -- start 2024-03-06T07:20:41.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.005+0000 7f0235672640 1 -- start start 2024-03-06T07:20:41.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.005+0000 7f5ecf7fe640 1 --2- 172.21.15.1:0/3167207022 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eb406b5a0 0x7f5eb406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.005+0000 7f27ff58a640 1 -- start start 2024-03-06T07:20:41.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.006+0000 7f5ecf7fe640 1 --2- 172.21.15.1:0/3167207022 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eb406b5a0 0x7f5eb406da60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5ec803f1c0 tx=0x7f5ebc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.006+0000 7fab8dafc640 1 -- 172.21.15.1:0/2622522769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab64005290 con 0x7fab8000f890 2024-03-06T07:20:41.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.006+0000 7fae757c7640 1 -- start start 2024-03-06T07:20:41.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.006+0000 7fe448ff9640 1 --2- 172.21.15.1:0/3020782893 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe43006b6d0 0x7fe43006db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.006+0000 7fe448ff9640 1 -- 172.21.15.1:0/3020782893 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe438089d70 con 0x7fe4440d4650 2024-03-06T07:20:41.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.006+0000 7f0a7be88640 1 Processor -- start 2024-03-06T07:20:41.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7fe44bfff640 1 --2- 172.21.15.1:0/3020782893 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe43006b6d0 0x7fe43006db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.006+0000 7f0a7be88640 1 -- start start 2024-03-06T07:20:41.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7f0a7be88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a740c5d90 0x7f0a740aca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7f0a7be88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a740c8100 0x7f0a741f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7f0a7be88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a7415a0e0 0x7f0a741f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7fae757c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae700c1a10 0x7fae70071f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7fae757c7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae700c3720 0x7fae700ac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7fae757c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae7015cce0 0x7fae701521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7fae757c7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7015d6d0 con 0x7fae700c1a10 2024-03-06T07:20:41.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7fae757c7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae70156dd0 con 0x7fae7015cce0 2024-03-06T07:20:41.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7fae757c7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7015d850 con 0x7fae700c3720 2024-03-06T07:20:41.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7fae6f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae7015cce0 0x7fae701521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7fae6f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae7015cce0 0x7fae701521c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:44566/0 (socket says 172.21.15.1:44566) 2024-03-06T07:20:41.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7fae6f7fe640 1 -- 172.21.15.1:0/3951063277 learned_addr learned my addr 172.21.15.1:0/3951063277 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:41.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7fae6effd640 1 --2- 172.21.15.1:0/3951063277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae700c1a10 0x7fae70071f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7fae6e7fc640 1 --2- 172.21.15.1:0/3951063277 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae700c3720 0x7fae700ac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7fae6effd640 1 -- 172.21.15.1:0/3951063277 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae700c3720 msgr2=0x7fae700ac1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7fae6effd640 1 --2- 172.21.15.1:0/3951063277 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae700c3720 0x7fae700ac1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7fae6effd640 1 -- 172.21.15.1:0/3951063277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae7015cce0 msgr2=0x7fae701521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7fae6effd640 1 --2- 172.21.15.1:0/3951063277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae7015cce0 0x7fae701521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7fae6effd640 1 -- 172.21.15.1:0/3951063277 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae70072a10 con 0x7fae700c1a10 2024-03-06T07:20:41.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7fae6e7fc640 1 --2- 172.21.15.1:0/3951063277 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae700c3720 0x7fae700ac1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:41.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7fae6effd640 1 --2- 172.21.15.1:0/3951063277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae700c1a10 0x7fae70071f10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fae64033000 tx=0x7fae640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.006+0000 7f0235672640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f02280dbcd0 0x7f0228047490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.006+0000 7f0235672640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f02280df170 0x7f0228006ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.006+0000 7f0235672640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f022800bb40 con 0x7f0228007420 2024-03-06T07:20:41.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.006+0000 7f0235672640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f022800a410 con 0x7f02280df170 2024-03-06T07:20:41.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.006+0000 7f0235672640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0228048050 con 0x7f02280dbcd0 2024-03-06T07:20:41.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7f022ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f02280dbcd0 0x7f0228047490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7f0234e71640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0228007420 0x7f022800a8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57466/0 (socket says 172.21.15.1:57466) 2024-03-06T07:20:41.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7f0234e71640 1 -- 172.21.15.1:0/2190441509 learned_addr learned my addr 172.21.15.1:0/2190441509 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:41.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7f022f7fe640 1 --2- 172.21.15.1:0/2190441509 >> v2:172.21.15.1:3300/0 conn(0x7f02280df170 0x7f0228006ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7f022ffff640 1 -- 172.21.15.1:0/2190441509 >> v2:172.21.15.1:3300/0 conn(0x7f02280df170 msgr2=0x7f0228006ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:41.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7f022ffff640 1 --2- 172.21.15.1:0/2190441509 >> v2:172.21.15.1:3300/0 conn(0x7f02280df170 0x7f0228006ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7f022ffff640 1 -- 172.21.15.1:0/2190441509 >> v1:172.21.15.1:6789/0 conn(0x7f0228007420 legacy=0x7f022800a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7fab857fa640 1 -- 172.21.15.1:0/2622522769 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab7401a020 con 0x7fab8000f890 2024-03-06T07:20:41.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7fab857fa640 1 --2- 172.21.15.1:0/2622522769 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fab6c06b530 0x7fab6c06d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7fab857fa640 1 -- 172.21.15.1:0/2622522769 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fab74089b50 con 0x7fab8000f890 2024-03-06T07:20:41.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7fab87fff640 1 --2- 172.21.15.1:0/2622522769 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fab6c06b530 0x7fab6c06d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.006+0000 7f27ff58a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27f8150590 0x7f27f80c3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.006+0000 7f27ff58a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27f80c3dc0 0x7f27f81568b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.006+0000 7f27ff58a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f80c6670 con 0x7f27f80c1260 2024-03-06T07:20:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.006+0000 7f27ff58a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f8150970 con 0x7f27f80c3dc0 2024-03-06T07:20:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.006+0000 7f27ff58a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f80c1e60 con 0x7f27f8150590 2024-03-06T07:20:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7f27fd326640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27f8150590 0x7f27f80c3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7f27fdb27640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f27f80c1260 0x7f27f8156df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:55062/0 (socket says 172.21.15.1:55062) 2024-03-06T07:20:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7f27fdb27640 1 -- 172.21.15.1:0/2541739654 learned_addr learned my addr 172.21.15.1:0/2541739654 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7f27fd326640 1 -- 172.21.15.1:0/2541739654 >> v2:172.21.15.98:3300/0 conn(0x7f27f80c3dc0 msgr2=0x7f27f81568b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:20:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7f27fd326640 1 --2- 172.21.15.1:0/2541739654 >> v2:172.21.15.98:3300/0 conn(0x7f27f80c3dc0 0x7f27f81568b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7f27fd326640 1 -- 172.21.15.1:0/2541739654 >> v1:172.21.15.98:6789/0 conn(0x7f27f80c1260 legacy=0x7f27f8156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.007+0000 7f27fd326640 1 -- 172.21.15.1:0/2541739654 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f8153330 con 0x7f27f8150590 2024-03-06T07:20:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7f022ffff640 1 -- 172.21.15.1:0/2190441509 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022800a0e0 con 0x7f02280dbcd0 2024-03-06T07:20:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7f0a7a425640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a7415a0e0 0x7f0a741f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7f0a79c24640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a740c5d90 0x7f0a740aca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7f0a7a425640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a7415a0e0 0x7f0a741f3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40624/0 (socket says 172.21.15.1:40624) 2024-03-06T07:20:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.008+0000 7f0a79c24640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a740c5d90 0x7f0a740aca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:44578/0 (socket says 172.21.15.1:44578) 2024-03-06T07:20:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7f0a79423640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a740c8100 0x7f0a741f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7f0a79423640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a740c8100 0x7f0a741f2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:41482/0 (socket says 172.21.15.1:41482) 2024-03-06T07:20:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7f27fd326640 1 --2- 172.21.15.1:0/2541739654 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27f8150590 0x7f27f80c3880 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0097a0 tx=0x7f27ec02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=934a96763e908462 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7f0a7a425640 1 -- 172.21.15.1:0/394151180 learned_addr learned my addr 172.21.15.1:0/394151180 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7f0a7be88640 1 -- 172.21.15.1:0/394151180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a740c2f20 con 0x7f0a7415a0e0 2024-03-06T07:20:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7fab87fff640 1 --2- 172.21.15.1:0/2622522769 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fab6c06b530 0x7fab6c06d9f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fab80043200 tx=0x7fab7c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7f0a7be88640 1 -- 172.21.15.1:0/394151180 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7415d1f0 con 0x7f0a740c5d90 2024-03-06T07:20:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7f0a7be88640 1 -- 172.21.15.1:0/394151180 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a740c84e0 con 0x7f0a740c8100 2024-03-06T07:20:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7f27f7fff640 1 -- 172.21.15.1:0/2541739654 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec035020 con 0x7f27f8150590 2024-03-06T07:20:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7f27f7fff640 1 -- 172.21.15.1:0/2541739654 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ec031830 con 0x7f27f8150590 2024-03-06T07:20:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7f27f7fff640 1 -- 172.21.15.1:0/2541739654 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec031c00 con 0x7f27f8150590 2024-03-06T07:20:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7fe44bfff640 1 --2- 172.21.15.1:0/3020782893 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe43006b6d0 0x7fe43006db90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe440000f60 tx=0x7fe440000ed0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7f022ffff640 1 --2- 172.21.15.1:0/2190441509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02280dbcd0 0x7f0228047490 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f02280078f0 tx=0x7f021c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e1f49f1d7e2e890 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7f0a7a425640 1 -- 172.21.15.1:0/394151180 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a740c8100 msgr2=0x7f0a741f2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7f0a7a425640 1 --2- 172.21.15.1:0/394151180 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a740c8100 0x7f0a741f2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7f0a7a425640 1 -- 172.21.15.1:0/394151180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a740c5d90 msgr2=0x7f0a740aca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7f0a7a425640 1 --2- 172.21.15.1:0/394151180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a740c5d90 0x7f0a740aca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.009+0000 7f0a7a425640 1 -- 172.21.15.1:0/394151180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a741ef2f0 con 0x7f0a7415a0e0 2024-03-06T07:20:41.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.010+0000 7f0a7a425640 1 --2- 172.21.15.1:0/394151180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a7415a0e0 0x7f0a741f3180 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00e190 tx=0x7f0a6c00e650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.010+0000 7f27ff58a640 1 -- 172.21.15.1:0/2541739654 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27f8150590 msgr2=0x7f27f80c3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.010+0000 7f27ff58a640 1 --2- 172.21.15.1:0/2541739654 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27f8150590 0x7f27f80c3880 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0097a0 tx=0x7f27ec02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.010+0000 7f27ff58a640 1 -- 172.21.15.1:0/2541739654 shutdown_connections 2024-03-06T07:20:41.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.010+0000 7f27ff58a640 1 --2- 172.21.15.1:0/2541739654 >> v2:172.21.15.98:3300/0 conn(0x7f27f80c3dc0 0x7f27f81568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.010+0000 7f27ff58a640 1 --2- 172.21.15.1:0/2541739654 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27f8150590 0x7f27f80c3880 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.010+0000 7f27ff58a640 1 -- 172.21.15.1:0/2541739654 >> 172.21.15.1:0/2541739654 conn(0x7f27f80c6860 msgr2=0x7f27f80c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:41.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.010+0000 7f5eccff9640 1 -- 172.21.15.1:0/3167207022 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ec401f020 con 0x7f5ec800bd40 2024-03-06T07:20:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.010+0000 7f27ff58a640 1 -- 172.21.15.1:0/2541739654 shutdown_connections 2024-03-06T07:20:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.010+0000 7f27ff58a640 1 -- 172.21.15.1:0/2541739654 wait complete. 2024-03-06T07:20:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.012+0000 7f27ff58a640 1 Processor -- start 2024-03-06T07:20:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.012+0000 7f27ff58a640 1 -- start start 2024-03-06T07:20:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.013+0000 7f27ff58a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27f80c1260 0x7f27f8155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.013+0000 7f27ff58a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27f80c3dc0 0x7f27f80a8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.013+0000 7f27ff58a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f8150590 0x7f27f80a88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.013+0000 7f27fd326640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27f80c1260 0x7f27f8155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.013+0000 7f27fd326640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27f80c1260 0x7f27f8155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40634/0 (socket says 172.21.15.1:40634) 2024-03-06T07:20:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.013+0000 7f27fd326640 1 -- 172.21.15.1:0/1852166948 learned_addr learned my addr 172.21.15.1:0/1852166948 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:41.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.013+0000 7f27ff58a640 1 -- 172.21.15.1:0/1852166948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f8153bd0 con 0x7f27f80c1260 2024-03-06T07:20:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.013+0000 7f27ff58a640 1 -- 172.21.15.1:0/1852166948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f81538f0 con 0x7f27f8150590 2024-03-06T07:20:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.013+0000 7f27ff58a640 1 -- 172.21.15.1:0/1852166948 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f806df60 con 0x7f27f80c3dc0 2024-03-06T07:20:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.013+0000 7f27fcb25640 1 --2- 172.21.15.1:0/1852166948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27f80c3dc0 0x7f27f80a8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.013+0000 7f27fdb27640 1 --2- 172.21.15.1:0/1852166948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f8150590 0x7f27f80a88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.014+0000 7f27fcb25640 1 -- 172.21.15.1:0/1852166948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f8150590 msgr2=0x7f27f80a88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.014+0000 7f27fcb25640 1 --2- 172.21.15.1:0/1852166948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f8150590 0x7f27f80a88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.014+0000 7f27fcb25640 1 -- 172.21.15.1:0/1852166948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27f80c1260 msgr2=0x7f27f8155830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.014+0000 7f27fcb25640 1 --2- 172.21.15.1:0/1852166948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27f80c1260 0x7f27f8155830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.014+0000 7f27fcb25640 1 -- 172.21.15.1:0/1852166948 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f814d5f0 con 0x7f27f80c3dc0 2024-03-06T07:20:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.014+0000 7f27fd326640 1 --2- 172.21.15.1:0/1852166948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27f80c1260 0x7f27f8155830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:41.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.014+0000 7f27fcb25640 1 --2- 172.21.15.1:0/1852166948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27f80c3dc0 0x7f27f80a8390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f27e800be50 tx=0x7f27e8009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.014+0000 7f0a6a7fc640 1 -- 172.21.15.1:0/394151180 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c014620 con 0x7f0a7415a0e0 2024-03-06T07:20:41.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.014+0000 7f0a6a7fc640 1 -- 172.21.15.1:0/394151180 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6c019070 con 0x7f0a7415a0e0 2024-03-06T07:20:41.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.014+0000 7fab857fa640 1 -- 172.21.15.1:0/2622522769 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab74058760 con 0x7fab8000f890 2024-03-06T07:20:41.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.014+0000 7f022effd640 1 -- 172.21.15.1:0/2190441509 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c0315a0 con 0x7f02280dbcd0 2024-03-06T07:20:41.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.014+0000 7f022effd640 1 -- 172.21.15.1:0/2190441509 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021c037070 con 0x7f02280dbcd0 2024-03-06T07:20:41.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.015+0000 7f0235672640 1 -- 172.21.15.1:0/2190441509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02280dbcd0 msgr2=0x7f0228047490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.015+0000 7fae577fe640 1 -- 172.21.15.1:0/3951063277 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae64043020 con 0x7fae700c1a10 2024-03-06T07:20:41.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.015+0000 7fae577fe640 1 -- 172.21.15.1:0/3951063277 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae64032bc0 con 0x7fae700c1a10 2024-03-06T07:20:41.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.015+0000 7fae577fe640 1 -- 172.21.15.1:0/3951063277 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae64038040 con 0x7fae700c1a10 2024-03-06T07:20:41.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.015+0000 7fae757c7640 1 -- 172.21.15.1:0/3951063277 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae70072cf0 con 0x7fae700c1a10 2024-03-06T07:20:41.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.015+0000 7fae757c7640 1 -- 172.21.15.1:0/3951063277 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7014da70 con 0x7fae700c1a10 2024-03-06T07:20:41.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.015+0000 7f0235672640 1 --2- 172.21.15.1:0/2190441509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02280dbcd0 0x7f0228047490 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f02280078f0 tx=0x7f021c02b840 comp rx=0 tx=0).stop 2024-03-06T07:20:41.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.015+0000 7f0a7be88640 1 -- 172.21.15.1:0/394151180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a740a8690 con 0x7f0a7415a0e0 2024-03-06T07:20:41.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.015+0000 7f0a7be88640 1 -- 172.21.15.1:0/394151180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a740a92c0 con 0x7f0a7415a0e0 2024-03-06T07:20:41.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.015+0000 7f0a6a7fc640 1 -- 172.21.15.1:0/394151180 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c014a00 con 0x7f0a7415a0e0 2024-03-06T07:20:41.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.016+0000 7f27f5ffb640 1 -- 172.21.15.1:0/1852166948 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e8019070 con 0x7f27f80c3dc0 2024-03-06T07:20:41.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.016+0000 7f0235672640 1 -- 172.21.15.1:0/2190441509 shutdown_connections 2024-03-06T07:20:41.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.016+0000 7f0235672640 1 --2- 172.21.15.1:0/2190441509 >> v2:172.21.15.1:3300/0 conn(0x7f02280df170 0x7f0228006ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.016+0000 7f0235672640 1 --2- 172.21.15.1:0/2190441509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02280dbcd0 0x7f0228047490 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.016+0000 7f0235672640 1 -- 172.21.15.1:0/2190441509 >> 172.21.15.1:0/2190441509 conn(0x7f0228048240 msgr2=0x7f0228048650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:41.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.016+0000 7f27ff58a640 1 -- 172.21.15.1:0/1852166948 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f814d8b0 con 0x7f27f80c3dc0 2024-03-06T07:20:41.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.016+0000 7f27ff58a640 1 -- 172.21.15.1:0/1852166948 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f80a7e70 con 0x7f27f80c3dc0 2024-03-06T07:20:41.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.016+0000 7f0235672640 1 -- 172.21.15.1:0/2190441509 shutdown_connections 2024-03-06T07:20:41.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.016+0000 7f27f5ffb640 1 -- 172.21.15.1:0/1852166948 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e8014440 con 0x7f27f80c3dc0 2024-03-06T07:20:41.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.016+0000 7f0235672640 1 -- 172.21.15.1:0/2190441509 wait complete. 2024-03-06T07:20:41.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.017+0000 7f0a7be88640 1 -- 172.21.15.1:0/394151180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a740686e0 con 0x7f0a7415a0e0 2024-03-06T07:20:41.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.019+0000 7fe448ff9640 1 -- 172.21.15.1:0/3020782893 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe4380588c0 con 0x7fe4440d4650 2024-03-06T07:20:41.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.017+0000 7f27f5ffb640 1 -- 172.21.15.1:0/1852166948 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e8014830 con 0x7f27f80c3dc0 2024-03-06T07:20:41.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.018+0000 7f27f5ffb640 1 -- 172.21.15.1:0/1852166948 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f27e801c070 con 0x7f27f80c3dc0 2024-03-06T07:20:41.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.018+0000 7f27f5ffb640 1 --2- 172.21.15.1:0/1852166948 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f27e406b520 0x7f27e406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.019+0000 7f27f5ffb640 1 -- 172.21.15.1:0/1852166948 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f27e808a420 con 0x7f27f80c3dc0 2024-03-06T07:20:41.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.017+0000 7fae757c7640 1 -- 172.21.15.1:0/3951063277 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae70067fd0 con 0x7fae700c1a10 2024-03-06T07:20:41.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.019+0000 7f0a6a7fc640 1 -- 172.21.15.1:0/394151180 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a6c014ba0 con 0x7f0a7415a0e0 2024-03-06T07:20:41.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.018+0000 7f0235672640 1 Processor -- start 2024-03-06T07:20:41.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.019+0000 7f27ff58a640 1 -- 172.21.15.1:0/1852166948 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d8005290 con 0x7f27f80c3dc0 2024-03-06T07:20:41.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.018+0000 7f0235672640 1 -- start start 2024-03-06T07:20:41.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.020+0000 7f0a6a7fc640 1 --2- 172.21.15.1:0/394151180 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a5806b5a0 0x7f0a5806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.019+0000 7f0235672640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0228007420 0x7f0228174c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.019+0000 7f0235672640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02280dbcd0 0x7f0228042890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.019+0000 7f0235672640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f02280df170 0x7f02280dd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.019+0000 7f0235672640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02280dc0b0 con 0x7f02280dbcd0 2024-03-06T07:20:41.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.019+0000 7f0235672640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0228047d70 con 0x7f0228007420 2024-03-06T07:20:41.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.019+0000 7f0235672640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02280df5e0 con 0x7f02280df170 2024-03-06T07:20:41.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.020+0000 7f0a79c24640 1 --2- 172.21.15.1:0/394151180 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a5806b5a0 0x7f0a5806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.020+0000 7f0a6a7fc640 1 -- 172.21.15.1:0/394151180 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0a6c08ad40 con 0x7f0a7415a0e0 2024-03-06T07:20:41.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.020+0000 7f0234e71640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f02280df170 0x7f02280dd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.020+0000 7f0234e71640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f02280df170 0x7f02280dd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:41510/0 (socket says 172.21.15.1:41510) 2024-03-06T07:20:41.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.020+0000 7f0234e71640 1 -- 172.21.15.1:0/3915377975 learned_addr learned my addr 172.21.15.1:0/3915377975 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:41.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.020+0000 7f022ffff640 1 --2- 172.21.15.1:0/3915377975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0228007420 0x7f0228174c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.020+0000 7f022f7fe640 1 --2- 172.21.15.1:0/3915377975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02280dbcd0 0x7f0228042890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.020+0000 7f0234e71640 1 -- 172.21.15.1:0/3915377975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0228007420 msgr2=0x7f0228174c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.020+0000 7f0234e71640 1 --2- 172.21.15.1:0/3915377975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0228007420 0x7f0228174c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.020+0000 7f0234e71640 1 -- 172.21.15.1:0/3915377975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02280dbcd0 msgr2=0x7f0228042890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.019+0000 7f27fd326640 1 --2- 172.21.15.1:0/1852166948 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f27e406b520 0x7f27e406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.020+0000 7f0234e71640 1 --2- 172.21.15.1:0/3915377975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02280dbcd0 0x7f0228042890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.020+0000 7f0234e71640 1 -- 172.21.15.1:0/3915377975 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02281741d0 con 0x7f02280df170 2024-03-06T07:20:41.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.020+0000 7f0a79c24640 1 --2- 172.21.15.1:0/394151180 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a5806b5a0 0x7f0a5806da60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0a74153290 tx=0x7f0a70009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.021+0000 7fae577fe640 1 -- 172.21.15.1:0/3951063277 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae64046020 con 0x7fae700c1a10 2024-03-06T07:20:41.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.021+0000 7f27fd326640 1 --2- 172.21.15.1:0/1852166948 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f27e406b520 0x7f27e406d9e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f27ec009560 tx=0x7f27ec008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.022+0000 7fae577fe640 1 --2- 172.21.15.1:0/3951063277 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fae5c06b5a0 0x7fae5c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.022+0000 7fae577fe640 1 -- 172.21.15.1:0/3951063277 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fae640ac690 con 0x7fae700c1a10 2024-03-06T07:20:41.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.022+0000 7f0234e71640 1 --2- 172.21.15.1:0/3915377975 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f02280df170 0x7f02280dd820 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0230068680 tx=0x7f0230073560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.022+0000 7fae6e7fc640 1 --2- 172.21.15.1:0/3951063277 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fae5c06b5a0 0x7fae5c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.023+0000 7fae6e7fc640 1 --2- 172.21.15.1:0/3951063277 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fae5c06b5a0 0x7fae5c06da60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fae68009440 tx=0x7fae6800b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.025+0000 7f27f5ffb640 1 -- 172.21.15.1:0/1852166948 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f27e80589f0 con 0x7f27f80c3dc0 2024-03-06T07:20:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.025+0000 7f022cff9640 1 -- 172.21.15.1:0/3915377975 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0230074c00 con 0x7f02280df170 2024-03-06T07:20:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.025+0000 7f022cff9640 1 -- 172.21.15.1:0/3915377975 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0230092380 con 0x7f02280df170 2024-03-06T07:20:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.025+0000 7f022cff9640 1 -- 172.21.15.1:0/3915377975 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0230092770 con 0x7f02280df170 2024-03-06T07:20:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.025+0000 7f0235672640 1 -- 172.21.15.1:0/3915377975 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0228175180 con 0x7f02280df170 2024-03-06T07:20:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.025+0000 7f0235672640 1 -- 172.21.15.1:0/3915377975 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02280405a0 con 0x7f02280df170 2024-03-06T07:20:41.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.027+0000 7f0235672640 1 -- 172.21.15.1:0/3915377975 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02280052b0 con 0x7f02280df170 2024-03-06T07:20:41.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.028+0000 7fae577fe640 1 -- 172.21.15.1:0/3951063277 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae6404d030 con 0x7fae700c1a10 2024-03-06T07:20:41.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.030+0000 7f022cff9640 1 -- 172.21.15.1:0/3915377975 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0230092910 con 0x7f02280df170 2024-03-06T07:20:41.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.030+0000 7f022cff9640 1 --2- 172.21.15.1:0/3915377975 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f021406b5a0 0x7f021406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.030+0000 7f022cff9640 1 -- 172.21.15.1:0/3915377975 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0230104cf0 con 0x7f02280df170 2024-03-06T07:20:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.031+0000 7f022ffff640 1 --2- 172.21.15.1:0/3915377975 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f021406b5a0 0x7f021406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.031+0000 7f022ffff640 1 --2- 172.21.15.1:0/3915377975 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f021406b5a0 0x7f021406da60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f021c036000 tx=0x7f021c03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.034+0000 7f0a6a7fc640 1 -- 172.21.15.1:0/394151180 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a6c0598f0 con 0x7f0a7415a0e0 2024-03-06T07:20:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.035+0000 7f022cff9640 1 -- 172.21.15.1:0/3915377975 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02300d3250 con 0x7f02280df170 2024-03-06T07:20:41.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.094+0000 7f35ac22a640 1 Processor -- start 2024-03-06T07:20:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.098+0000 7f35ac22a640 1 -- start start 2024-03-06T07:20:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.099+0000 7f35ac22a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35a40c5960 0x7f35a40c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.099+0000 7f35ac22a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35a4153ab0 0x7f35a4070b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.099+0000 7f35ac22a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a40c67c0 con 0x7f35a40c4ec0 2024-03-06T07:20:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.099+0000 7f35ac22a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a40c6a60 con 0x7f35a40c5960 2024-03-06T07:20:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.099+0000 7f35ac22a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a40c6d40 con 0x7f35a4153ab0 2024-03-06T07:20:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.100+0000 7f35a9fc6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35a40c5960 0x7f35a40c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.100+0000 7f35a9fc6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35a40c5960 0x7f35a40c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40640/0 (socket says 172.21.15.1:40640) 2024-03-06T07:20:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.100+0000 7f35a9fc6640 1 -- 172.21.15.1:0/3330523137 learned_addr learned my addr 172.21.15.1:0/3330523137 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.101+0000 7f35a9fc6640 1 -- 172.21.15.1:0/3330523137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35a4153ab0 msgr2=0x7f35a4070b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:20:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.101+0000 7f35a9fc6640 1 --2- 172.21.15.1:0/3330523137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35a4153ab0 0x7f35a4070b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.101+0000 7f35a9fc6640 1 -- 172.21.15.1:0/3330523137 >> v1:172.21.15.1:6789/0 conn(0x7f35a40c4ec0 legacy=0x7f35a4156620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:20:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.101+0000 7f35a9fc6640 1 -- 172.21.15.1:0/3330523137 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a40c7020 con 0x7f35a40c5960 2024-03-06T07:20:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.102+0000 7f35a9fc6640 1 --2- 172.21.15.1:0/3330523137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35a40c5960 0x7f35a40c4980 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f359c00b1a0 tx=0x7f359c02c140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610b9ed87fa65fdf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.103+0000 7f35a8fc4640 1 -- 172.21.15.1:0/3330523137 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c02cae0 con 0x7f35a40c5960 2024-03-06T07:20:41.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.103+0000 7f35a8fc4640 1 -- 172.21.15.1:0/3330523137 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359c02cc80 con 0x7f35a40c5960 2024-03-06T07:20:41.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.104+0000 7f35ac22a640 1 -- 172.21.15.1:0/3330523137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35a40c5960 msgr2=0x7f35a40c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.104+0000 7f35ac22a640 1 --2- 172.21.15.1:0/3330523137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35a40c5960 0x7f35a40c4980 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f359c00b1a0 tx=0x7f359c02c140 comp rx=0 tx=0).stop 2024-03-06T07:20:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.106+0000 7f35ac22a640 1 -- 172.21.15.1:0/3330523137 shutdown_connections 2024-03-06T07:20:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.106+0000 7f35ac22a640 1 --2- 172.21.15.1:0/3330523137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35a4153ab0 0x7f35a4070b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.106+0000 7f35ac22a640 1 --2- 172.21.15.1:0/3330523137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35a40c5960 0x7f35a40c4980 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.106+0000 7f35ac22a640 1 -- 172.21.15.1:0/3330523137 >> 172.21.15.1:0/3330523137 conn(0x7f35a40aa550 msgr2=0x7f35a40aa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.107+0000 7f35ac22a640 1 -- 172.21.15.1:0/3330523137 shutdown_connections 2024-03-06T07:20:41.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.108+0000 7f35ac22a640 1 -- 172.21.15.1:0/3330523137 wait complete. 2024-03-06T07:20:41.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.109+0000 7f35ac22a640 1 Processor -- start 2024-03-06T07:20:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.110+0000 7f35ac22a640 1 -- start start 2024-03-06T07:20:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.111+0000 7f35ac22a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35a40c4ec0 0x7f35a406c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.111+0000 7f35ac22a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35a40c5960 0x7f35a40ac900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.111+0000 7f35ac22a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35a4153ab0 0x7f35a4156e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.111+0000 7f35ac22a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a406f550 con 0x7f35a40c5960 2024-03-06T07:20:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.111+0000 7f35ac22a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a406f3d0 con 0x7f35a40c4ec0 2024-03-06T07:20:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.111+0000 7f35ac22a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a40c5d40 con 0x7f35a4153ab0 2024-03-06T07:20:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.112+0000 7f35a97c5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35a40c5960 0x7f35a40ac900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.112+0000 7f35a97c5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35a40c5960 0x7f35a40ac900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40654/0 (socket says 172.21.15.1:40654) 2024-03-06T07:20:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.112+0000 7f35a97c5640 1 -- 172.21.15.1:0/2122737128 learned_addr learned my addr 172.21.15.1:0/2122737128 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.113+0000 7f35a97c5640 1 -- 172.21.15.1:0/2122737128 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35a4153ab0 msgr2=0x7f35a4156e60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:20:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.113+0000 7f35a97c5640 1 --2- 172.21.15.1:0/2122737128 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35a4153ab0 0x7f35a4156e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.113+0000 7f35a97c5640 1 -- 172.21.15.1:0/2122737128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35a40c4ec0 msgr2=0x7f35a406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.113+0000 7f35a97c5640 1 --2- 172.21.15.1:0/2122737128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35a40c4ec0 0x7f35a406c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.113+0000 7f35a97c5640 1 -- 172.21.15.1:0/2122737128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a41567d0 con 0x7f35a40c5960 2024-03-06T07:20:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.113+0000 7f35a97c5640 1 --2- 172.21.15.1:0/2122737128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35a40c5960 0x7f35a40ac900 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f35a000b450 tx=0x7f35a000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.114+0000 7f359a7fc640 1 -- 172.21.15.1:0/2122737128 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a0015020 con 0x7f35a40c5960 2024-03-06T07:20:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.114+0000 7f35ac22a640 1 -- 172.21.15.1:0/2122737128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a41efe60 con 0x7f35a40c5960 2024-03-06T07:20:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.114+0000 7f35ac22a640 1 -- 172.21.15.1:0/2122737128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a40a7840 con 0x7f35a40c5960 2024-03-06T07:20:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.116+0000 7f35ac22a640 1 -- 172.21.15.1:0/2122737128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a4067fd0 con 0x7f35a40c5960 2024-03-06T07:20:41.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.127+0000 7f359a7fc640 1 -- 172.21.15.1:0/2122737128 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a0011890 con 0x7f35a40c5960 2024-03-06T07:20:41.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.127+0000 7f359a7fc640 1 -- 172.21.15.1:0/2122737128 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a0011c80 con 0x7f35a40c5960 2024-03-06T07:20:41.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.127+0000 7f359a7fc640 1 -- 172.21.15.1:0/2122737128 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f35a0018020 con 0x7f35a40c5960 2024-03-06T07:20:41.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.127+0000 7f359a7fc640 1 --2- 172.21.15.1:0/2122737128 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f358806b5d0 0x7f358806da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:41.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.128+0000 7f359a7fc640 1 -- 172.21.15.1:0/2122737128 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f35a008aef0 con 0x7f35a40c5960 2024-03-06T07:20:41.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.128+0000 7f359a7fc640 1 -- 172.21.15.1:0/2122737128 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f35a001f020 con 0x7f35a40c5960 2024-03-06T07:20:41.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.130+0000 7f35a9fc6640 1 --2- 172.21.15.1:0/2122737128 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f358806b5d0 0x7f358806da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:41.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.147+0000 7f35a9fc6640 1 --2- 172.21.15.1:0/2122737128 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f358806b5d0 0x7f358806da90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f359c035000 tx=0x7f359c02e000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:41.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.419+0000 7f5ed63f6640 1 -- 172.21.15.1:0/3167207022 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f5ec8048020 con 0x7f5ec800bd40 2024-03-06T07:20:41.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.420+0000 7f5eccff9640 1 -- 172.21.15.1:0/3167207022 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5ec4058e60 con 0x7f5ec800bd40 2024-03-06T07:20:41.423 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-06T07:20:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.424+0000 7f5eba7fc640 1 -- 172.21.15.1:0/3167207022 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eb406b5a0 msgr2=0x7f5eb406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.424+0000 7f5eba7fc640 1 --2- 172.21.15.1:0/3167207022 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eb406b5a0 0x7f5eb406da60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5ec803f1c0 tx=0x7f5ebc009040 comp rx=0 tx=0).stop 2024-03-06T07:20:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.424+0000 7f5eba7fc640 1 -- 172.21.15.1:0/3167207022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ec800bd40 msgr2=0x7f5ec80308e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.424+0000 7f5eba7fc640 1 --2- 172.21.15.1:0/3167207022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ec800bd40 0x7f5ec80308e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ec400b3e0 tx=0x7f5ec400b8a0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.424+0000 7f5eba7fc640 1 -- 172.21.15.1:0/3167207022 shutdown_connections 2024-03-06T07:20:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.424+0000 7f5eba7fc640 1 --2- 172.21.15.1:0/3167207022 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eb406b5a0 0x7f5eb406da60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.424+0000 7f5eba7fc640 1 --2- 172.21.15.1:0/3167207022 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ec80d6700 0x7f5ec8174190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.425+0000 7f5eba7fc640 1 --2- 172.21.15.1:0/3167207022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ec804b100 0x7f5ec81780e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.425+0000 7f5eba7fc640 1 --2- 172.21.15.1:0/3167207022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ec800bd40 0x7f5ec80308e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.425+0000 7f5eba7fc640 1 -- 172.21.15.1:0/3167207022 >> 172.21.15.1:0/3167207022 conn(0x7f5ec8048240 msgr2=0x7f5ec802c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.425+0000 7f5eba7fc640 1 -- 172.21.15.1:0/3167207022 shutdown_connections 2024-03-06T07:20:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.425+0000 7f5eba7fc640 1 -- 172.21.15.1:0/3167207022 wait complete. 2024-03-06T07:20:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.440+0000 7f0a7be88640 1 -- 172.21.15.1:0/394151180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f0a740c7620 con 0x7f0a7415a0e0 2024-03-06T07:20:41.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.441+0000 7f0a6a7fc640 1 -- 172.21.15.1:0/394151180 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0a6c0591f0 con 0x7f0a7415a0e0 2024-03-06T07:20:41.443 INFO:teuthology.orchestra.run.smithi001.stdout:55834574854 2024-03-06T07:20:41.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.442+0000 7fab8dafc640 1 -- 172.21.15.1:0/2622522769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fab64005580 con 0x7fab8000f890 2024-03-06T07:20:41.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.443+0000 7fab857fa640 1 -- 172.21.15.1:0/2622522769 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fab74089e20 con 0x7fab8000f890 2024-03-06T07:20:41.445 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-06T07:20:41.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.443+0000 7fe4367fc640 1 -- 172.21.15.1:0/3020782893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe44404b500 con 0x7fe4440d4650 2024-03-06T07:20:41.446 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.0 2024-03-06T07:20:41.446 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-06T07:20:41.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.444+0000 7fe448ff9640 1 -- 172.21.15.1:0/3020782893 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe4380581c0 con 0x7fe4440d4650 2024-03-06T07:20:41.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.444+0000 7f0a7be88640 1 -- 172.21.15.1:0/394151180 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a5806b5a0 msgr2=0x7f0a5806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.444+0000 7f0a7be88640 1 --2- 172.21.15.1:0/394151180 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a5806b5a0 0x7f0a5806da60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0a74153290 tx=0x7f0a70009040 comp rx=0 tx=0).stop 2024-03-06T07:20:41.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.444+0000 7f0a7be88640 1 -- 172.21.15.1:0/394151180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a7415a0e0 msgr2=0x7f0a741f3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.444+0000 7f0a7be88640 1 --2- 172.21.15.1:0/394151180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a7415a0e0 0x7f0a741f3180 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00e190 tx=0x7f0a6c00e650 comp rx=0 tx=0).stop 2024-03-06T07:20:41.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.444+0000 7f0a7be88640 1 -- 172.21.15.1:0/394151180 shutdown_connections 2024-03-06T07:20:41.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.444+0000 7f0a7be88640 1 --2- 172.21.15.1:0/394151180 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a5806b5a0 0x7f0a5806da60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.444+0000 7f0a7be88640 1 --2- 172.21.15.1:0/394151180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a7415a0e0 0x7f0a741f3180 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.444+0000 7f0a7be88640 1 --2- 172.21.15.1:0/394151180 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a740c8100 0x7f0a741f2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.444+0000 7f0a7be88640 1 --2- 172.21.15.1:0/394151180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a740c5d90 0x7f0a740aca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.444+0000 7f0a7be88640 1 -- 172.21.15.1:0/394151180 >> 172.21.15.1:0/394151180 conn(0x7f0a740c88f0 msgr2=0x7f0a74074450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:41.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.445+0000 7f0a7be88640 1 -- 172.21.15.1:0/394151180 shutdown_connections 2024-03-06T07:20:41.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.445+0000 7f0a7be88640 1 -- 172.21.15.1:0/394151180 wait complete. 2024-03-06T07:20:41.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fab8dafc640 1 -- 172.21.15.1:0/2622522769 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fab6c06b530 msgr2=0x7fab6c06d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fab8dafc640 1 --2- 172.21.15.1:0/2622522769 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fab6c06b530 0x7fab6c06d9f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fab80043200 tx=0x7fab7c009040 comp rx=0 tx=0).stop 2024-03-06T07:20:41.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fab8dafc640 1 -- 172.21.15.1:0/2622522769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab8000f890 msgr2=0x7fab800d3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fab8dafc640 1 --2- 172.21.15.1:0/2622522769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab8000f890 0x7fab800d3080 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fab7400c3c0 tx=0x7fab7400c880 comp rx=0 tx=0).stop 2024-03-06T07:20:41.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fab8dafc640 1 -- 172.21.15.1:0/2622522769 shutdown_connections 2024-03-06T07:20:41.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fab8dafc640 1 --2- 172.21.15.1:0/2622522769 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fab6c06b530 0x7fab6c06d9f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fab8dafc640 1 --2- 172.21.15.1:0/2622522769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab800d7b20 0x7fab800ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fab8dafc640 1 --2- 172.21.15.1:0/2622522769 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab8004dd20 0x7fab800ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fab8dafc640 1 --2- 172.21.15.1:0/2622522769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab8000f890 0x7fab800d3080 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fab8dafc640 1 -- 172.21.15.1:0/2622522769 >> 172.21.15.1:0/2622522769 conn(0x7fab8004c680 msgr2=0x7fab80006150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:41.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fab8dafc640 1 -- 172.21.15.1:0/2622522769 shutdown_connections 2024-03-06T07:20:41.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fe4367fc640 1 -- 172.21.15.1:0/3020782893 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe43006b6d0 msgr2=0x7fe43006db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fe4367fc640 1 --2- 172.21.15.1:0/3020782893 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe43006b6d0 0x7fe43006db90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe440000f60 tx=0x7fe440000ed0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fe4367fc640 1 -- 172.21.15.1:0/3020782893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4440d4650 msgr2=0x7fe4440d0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fe4367fc640 1 --2- 172.21.15.1:0/3020782893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4440d4650 0x7fe4440d0da0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe43800c490 tx=0x7fe43800c950 comp rx=0 tx=0).stop 2024-03-06T07:20:41.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fe4367fc640 1 -- 172.21.15.1:0/3020782893 shutdown_connections 2024-03-06T07:20:41.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fe4367fc640 1 --2- 172.21.15.1:0/3020782893 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe43006b6d0 0x7fe43006db90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fe4367fc640 1 --2- 172.21.15.1:0/3020782893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4440d6fc0 0x7fe4440d12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fe4367fc640 1 --2- 172.21.15.1:0/3020782893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4440d4650 0x7fe4440d0da0 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe43800c490 tx=0x7fe43800c950 comp rx=0 tx=0).stop 2024-03-06T07:20:41.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fe4367fc640 1 --2- 172.21.15.1:0/3020782893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe444049df0 0x7fe444045d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.446+0000 7fe4367fc640 1 -- 172.21.15.1:0/3020782893 >> 172.21.15.1:0/3020782893 conn(0x7fe4440492d0 msgr2=0x7fe4440d6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.447+0000 7fab8dafc640 1 -- 172.21.15.1:0/2622522769 wait complete. 2024-03-06T07:20:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.447+0000 7fe4367fc640 1 -- 172.21.15.1:0/3020782893 shutdown_connections 2024-03-06T07:20:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.447+0000 7fe4367fc640 1 -- 172.21.15.1:0/3020782893 wait complete. 2024-03-06T07:20:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.450+0000 7fae757c7640 1 -- 172.21.15.1:0/3951063277 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fae700a6c30 con 0x7fae700c1a10 2024-03-06T07:20:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.451+0000 7fae577fe640 1 -- 172.21.15.1:0/3951063277 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fae6407b240 con 0x7fae700c1a10 2024-03-06T07:20:41.453 INFO:teuthology.orchestra.run.smithi001.stdout:55834574854 2024-03-06T07:20:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.454+0000 7fae757c7640 1 -- 172.21.15.1:0/3951063277 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fae5c06b5a0 msgr2=0x7fae5c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.454+0000 7fae757c7640 1 --2- 172.21.15.1:0/3951063277 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fae5c06b5a0 0x7fae5c06da60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fae68009440 tx=0x7fae6800b040 comp rx=0 tx=0).stop 2024-03-06T07:20:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.454+0000 7fae757c7640 1 -- 172.21.15.1:0/3951063277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae700c1a10 msgr2=0x7fae70071f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.454+0000 7fae757c7640 1 --2- 172.21.15.1:0/3951063277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae700c1a10 0x7fae70071f10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fae64033000 tx=0x7fae640046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.454+0000 7fae757c7640 1 -- 172.21.15.1:0/3951063277 shutdown_connections 2024-03-06T07:20:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.454+0000 7fae757c7640 1 --2- 172.21.15.1:0/3951063277 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fae5c06b5a0 0x7fae5c06da60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.454+0000 7fae757c7640 1 --2- 172.21.15.1:0/3951063277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae7015cce0 0x7fae701521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.454+0000 7fae757c7640 1 --2- 172.21.15.1:0/3951063277 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae700c3720 0x7fae700ac1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.454+0000 7fae757c7640 1 --2- 172.21.15.1:0/3951063277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae700c1a10 0x7fae70071f10 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.454+0000 7fae757c7640 1 -- 172.21.15.1:0/3951063277 >> 172.21.15.1:0/3951063277 conn(0x7fae700c6860 msgr2=0x7fae700c4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.454+0000 7f27ff58a640 1 -- 172.21.15.1:0/1852166948 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f27d8005850 con 0x7f27f80c3dc0 2024-03-06T07:20:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.454+0000 7f27f5ffb640 1 -- 172.21.15.1:0/1852166948 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f27e805ca90 con 0x7f27f80c3dc0 2024-03-06T07:20:41.459 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-06T07:20:41.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.455+0000 7fae757c7640 1 -- 172.21.15.1:0/3951063277 shutdown_connections 2024-03-06T07:20:41.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.455+0000 7fae757c7640 1 -- 172.21.15.1:0/3951063277 wait complete. 2024-03-06T07:20:41.468 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.1 2024-03-06T07:20:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.466+0000 7f27ff58a640 1 -- 172.21.15.1:0/1852166948 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f27e406b520 msgr2=0x7f27e406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.466+0000 7f27ff58a640 1 --2- 172.21.15.1:0/1852166948 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f27e406b520 0x7f27e406d9e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f27ec009560 tx=0x7f27ec008030 comp rx=0 tx=0).stop 2024-03-06T07:20:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.466+0000 7f27ff58a640 1 -- 172.21.15.1:0/1852166948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27f80c3dc0 msgr2=0x7f27f80a8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.466+0000 7f27ff58a640 1 --2- 172.21.15.1:0/1852166948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27f80c3dc0 0x7f27f80a8390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f27e800be50 tx=0x7f27e8009f70 comp rx=0 tx=0).stop 2024-03-06T07:20:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.466+0000 7f27ff58a640 1 -- 172.21.15.1:0/1852166948 shutdown_connections 2024-03-06T07:20:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.466+0000 7f27ff58a640 1 --2- 172.21.15.1:0/1852166948 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f27e406b520 0x7f27e406d9e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.466+0000 7f27ff58a640 1 --2- 172.21.15.1:0/1852166948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f8150590 0x7f27f80a88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.466+0000 7f27ff58a640 1 --2- 172.21.15.1:0/1852166948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27f80c3dc0 0x7f27f80a8390 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.466+0000 7f27ff58a640 1 --2- 172.21.15.1:0/1852166948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27f80c1260 0x7f27f8155830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.466+0000 7f27ff58a640 1 -- 172.21.15.1:0/1852166948 >> 172.21.15.1:0/1852166948 conn(0x7f27f80c6860 msgr2=0x7f27f80c6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:41.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.467+0000 7f27ff58a640 1 -- 172.21.15.1:0/1852166948 shutdown_connections 2024-03-06T07:20:41.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.467+0000 7f27ff58a640 1 -- 172.21.15.1:0/1852166948 wait complete. 2024-03-06T07:20:41.470 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.3 2024-03-06T07:20:41.470 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.7 2024-03-06T07:20:41.476 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.6 2024-03-06T07:20:41.481 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2024-03-06T07:20:41.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.500+0000 7f35ac22a640 1 -- 172.21.15.1:0/2122737128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f35a40c7a70 con 0x7f35a40c5960 2024-03-06T07:20:41.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.500+0000 7f359a7fc640 1 -- 172.21.15.1:0/2122737128 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f35a000d040 con 0x7f35a40c5960 2024-03-06T07:20:41.502 INFO:teuthology.orchestra.run.smithi001.stdout:55834574854 2024-03-06T07:20:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.503+0000 7f35ac22a640 1 -- 172.21.15.1:0/2122737128 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f358806b5d0 msgr2=0x7f358806da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.503+0000 7f35ac22a640 1 --2- 172.21.15.1:0/2122737128 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f358806b5d0 0x7f358806da90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f359c035000 tx=0x7f359c02e000 comp rx=0 tx=0).stop 2024-03-06T07:20:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.503+0000 7f35ac22a640 1 -- 172.21.15.1:0/2122737128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35a40c5960 msgr2=0x7f35a40ac900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.503+0000 7f35ac22a640 1 --2- 172.21.15.1:0/2122737128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35a40c5960 0x7f35a40ac900 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f35a000b450 tx=0x7f35a000b910 comp rx=0 tx=0).stop 2024-03-06T07:20:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.503+0000 7f35ac22a640 1 -- 172.21.15.1:0/2122737128 shutdown_connections 2024-03-06T07:20:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.503+0000 7f35ac22a640 1 --2- 172.21.15.1:0/2122737128 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f358806b5d0 0x7f358806da90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.503+0000 7f35ac22a640 1 --2- 172.21.15.1:0/2122737128 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35a4153ab0 0x7f35a4156e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.503+0000 7f35ac22a640 1 --2- 172.21.15.1:0/2122737128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35a40c5960 0x7f35a40ac900 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.503+0000 7f35ac22a640 1 --2- 172.21.15.1:0/2122737128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35a40c4ec0 0x7f35a406c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.503+0000 7f35ac22a640 1 -- 172.21.15.1:0/2122737128 >> 172.21.15.1:0/2122737128 conn(0x7f35a40aa550 msgr2=0x7f35a40745e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.503+0000 7f35ac22a640 1 -- 172.21.15.1:0/2122737128 shutdown_connections 2024-03-06T07:20:41.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.503+0000 7f35ac22a640 1 -- 172.21.15.1:0/2122737128 wait complete. 2024-03-06T07:20:41.515 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.2 2024-03-06T07:20:41.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.633+0000 7f0235672640 1 -- 172.21.15.1:0/3915377975 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f022800b580 con 0x7f02280df170 2024-03-06T07:20:41.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.634+0000 7f022cff9640 1 -- 172.21.15.1:0/3915377975 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f02300d72f0 con 0x7f02280df170 2024-03-06T07:20:41.636 INFO:teuthology.orchestra.run.smithi001.stdout:55834574854 2024-03-06T07:20:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.636+0000 7f0235672640 1 -- 172.21.15.1:0/3915377975 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f021406b5a0 msgr2=0x7f021406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.636+0000 7f0235672640 1 --2- 172.21.15.1:0/3915377975 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f021406b5a0 0x7f021406da60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f021c036000 tx=0x7f021c03d040 comp rx=0 tx=0).stop 2024-03-06T07:20:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.636+0000 7f0235672640 1 -- 172.21.15.1:0/3915377975 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f02280df170 msgr2=0x7f02280dd820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.636+0000 7f0235672640 1 --2- 172.21.15.1:0/3915377975 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f02280df170 0x7f02280dd820 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0230068680 tx=0x7f0230073560 comp rx=0 tx=0).stop 2024-03-06T07:20:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.636+0000 7f0235672640 1 -- 172.21.15.1:0/3915377975 shutdown_connections 2024-03-06T07:20:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.636+0000 7f0235672640 1 --2- 172.21.15.1:0/3915377975 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f021406b5a0 0x7f021406da60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.636+0000 7f0235672640 1 --2- 172.21.15.1:0/3915377975 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f02280df170 0x7f02280dd820 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.636+0000 7f0235672640 1 --2- 172.21.15.1:0/3915377975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02280dbcd0 0x7f0228042890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.636+0000 7f0235672640 1 --2- 172.21.15.1:0/3915377975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0228007420 0x7f0228174c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.636+0000 7f0235672640 1 -- 172.21.15.1:0/3915377975 >> 172.21.15.1:0/3915377975 conn(0x7f0228048240 msgr2=0x7f022802c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.636+0000 7f0235672640 1 -- 172.21.15.1:0/3915377975 shutdown_connections 2024-03-06T07:20:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:41.636+0000 7f0235672640 1 -- 172.21.15.1:0/3915377975 wait complete. 2024-03-06T07:20:41.648 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.5 2024-03-06T07:20:42.447 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:20:42.469 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:20:42.471 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:20:42.472 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:20:42.477 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:20:42.482 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:20:42.516 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:20:42.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.591+0000 7fdfeb822640 1 Processor -- start 2024-03-06T07:20:42.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.591+0000 7fdfeb822640 1 -- start start 2024-03-06T07:20:42.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.592+0000 7fdfeb822640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfe414e8b0 0x7fdfe406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.592+0000 7fdfeb822640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdfe406e1d0 0x7fdfe4154a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.592+0000 7fdfeb822640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe40c25f0 con 0x7fdfe40c1490 2024-03-06T07:20:42.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.592+0000 7fdfeb822640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfe40c28d0 con 0x7fdfe406e1d0 2024-03-06T07:20:42.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.592+0000 7fdfeb822640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe40c2bb0 con 0x7fdfe414e8b0 2024-03-06T07:20:42.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.593+0000 7fdfe9dbf640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdfe40c1490 0x7fdfe4154f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57480/0 (socket says 172.21.15.1:57480) 2024-03-06T07:20:42.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.593+0000 7fdfe9dbf640 1 -- 172.21.15.1:0/3194521861 learned_addr learned my addr 172.21.15.1:0/3194521861 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:42.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.593+0000 7fdfe8dbd640 1 --2- 172.21.15.1:0/3194521861 >> v2:172.21.15.98:3300/0 conn(0x7fdfe406e1d0 0x7fdfe4154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.593+0000 7fdfe95be640 1 --2- 172.21.15.1:0/3194521861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfe414e8b0 0x7fdfe406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.594+0000 7fdfe8dbd640 1 -- 172.21.15.1:0/3194521861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfe414e8b0 msgr2=0x7fdfe406dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:42.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.594+0000 7fdfe8dbd640 1 --2- 172.21.15.1:0/3194521861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfe414e8b0 0x7fdfe406dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.594+0000 7fdfe8dbd640 1 -- 172.21.15.1:0/3194521861 >> v1:172.21.15.1:6789/0 conn(0x7fdfe40c1490 legacy=0x7fdfe4154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.594+0000 7fdfdbfff640 1 -- 172.21.15.1:0/3194521861 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270913834 0 0) 0x7fdfe40c25f0 con 0x7fdfe40c1490 2024-03-06T07:20:42.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.595+0000 7fdfe8dbd640 1 -- 172.21.15.1:0/3194521861 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe40c2e90 con 0x7fdfe406e1d0 2024-03-06T07:20:42.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.603+0000 7fdfe8dbd640 1 --2- 172.21.15.1:0/3194521861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe406e1d0 0x7fdfe4154a10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdfd40097a0 tx=0x7fdfd402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2eae3294223d0df6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.604+0000 7fdfdbfff640 1 -- 172.21.15.1:0/3194521861 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd40315a0 con 0x7fdfe406e1d0 2024-03-06T07:20:42.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.604+0000 7fdfdbfff640 1 -- 172.21.15.1:0/3194521861 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd4037070 con 0x7fdfe406e1d0 2024-03-06T07:20:42.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.604+0000 7fdfdbfff640 1 -- 172.21.15.1:0/3194521861 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4031960 con 0x7fdfe406e1d0 2024-03-06T07:20:42.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.605+0000 7fdfeb822640 1 -- 172.21.15.1:0/3194521861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe406e1d0 msgr2=0x7fdfe4154a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.605+0000 7fdfeb822640 1 --2- 172.21.15.1:0/3194521861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe406e1d0 0x7fdfe4154a10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdfd40097a0 tx=0x7fdfd402b840 comp rx=0 tx=0).stop 2024-03-06T07:20:42.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.606+0000 7fdfeb822640 1 -- 172.21.15.1:0/3194521861 shutdown_connections 2024-03-06T07:20:42.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.606+0000 7fdfeb822640 1 --2- 172.21.15.1:0/3194521861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe406e1d0 0x7fdfe4154a10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.606+0000 7fdfeb822640 1 --2- 172.21.15.1:0/3194521861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfe414e8b0 0x7fdfe406dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.606+0000 7fdfeb822640 1 -- 172.21.15.1:0/3194521861 >> 172.21.15.1:0/3194521861 conn(0x7fdfe40c6860 msgr2=0x7fdfe40c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:42.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.606+0000 7fdfeb822640 1 -- 172.21.15.1:0/3194521861 shutdown_connections 2024-03-06T07:20:42.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.607+0000 7fdfeb822640 1 -- 172.21.15.1:0/3194521861 wait complete. 2024-03-06T07:20:42.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.608+0000 7fdfeb822640 1 Processor -- start 2024-03-06T07:20:42.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.608+0000 7fdfeb822640 1 -- start start 2024-03-06T07:20:42.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.610+0000 7fdfeb822640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfe406e1d0 0x7fdfe40ac200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.610+0000 7fdfeb822640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe40c1490 0x7fdfe4149240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.610+0000 7fdfeb822640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe414e8b0 0x7fdfe4144d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.610+0000 7fdfeb822640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe414ee10 con 0x7fdfe414e8b0 2024-03-06T07:20:42.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.610+0000 7fdfeb822640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe414ec90 con 0x7fdfe40c1490 2024-03-06T07:20:42.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.610+0000 7fdfeb822640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe40c7380 con 0x7fdfe406e1d0 2024-03-06T07:20:42.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.610+0000 7fdfe9dbf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe414e8b0 0x7fdfe4144d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.610+0000 7fdfe9dbf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe414e8b0 0x7fdfe4144d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40656/0 (socket says 172.21.15.1:40656) 2024-03-06T07:20:42.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.610+0000 7fdfe9dbf640 1 -- 172.21.15.1:0/1660361745 learned_addr learned my addr 172.21.15.1:0/1660361745 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:42.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.611+0000 7fdfe9dbf640 1 -- 172.21.15.1:0/1660361745 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfe406e1d0 msgr2=0x7fdfe40ac200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:20:42.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.611+0000 7fdfe9dbf640 1 --2- 172.21.15.1:0/1660361745 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfe406e1d0 0x7fdfe40ac200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.611+0000 7fdfe9dbf640 1 -- 172.21.15.1:0/1660361745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe40c1490 msgr2=0x7fdfe4149240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:20:42.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.611+0000 7fdfe9dbf640 1 --2- 172.21.15.1:0/1660361745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe40c1490 0x7fdfe4149240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.611+0000 7fdfe9dbf640 1 -- 172.21.15.1:0/1660361745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe4145db0 con 0x7fdfe414e8b0 2024-03-06T07:20:42.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.611+0000 7fdfe9dbf640 1 --2- 172.21.15.1:0/1660361745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe414e8b0 0x7fdfe4144d50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00c7a0 tx=0x7fdfdc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.612+0000 7fdfd9ffb640 1 -- 172.21.15.1:0/1660361745 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc018070 con 0x7fdfe414e8b0 2024-03-06T07:20:42.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.612+0000 7fdfeb822640 1 -- 172.21.15.1:0/1660361745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe4145ad0 con 0x7fdfe414e8b0 2024-03-06T07:20:42.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.612+0000 7fdfeb822640 1 -- 172.21.15.1:0/1660361745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe40a9c90 con 0x7fdfe414e8b0 2024-03-06T07:20:42.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.612+0000 7fdfd9ffb640 1 -- 172.21.15.1:0/1660361745 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfdc00a640 con 0x7fdfe414e8b0 2024-03-06T07:20:42.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.613+0000 7fdfd9ffb640 1 -- 172.21.15.1:0/1660361745 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc00aa50 con 0x7fdfe414e8b0 2024-03-06T07:20:42.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.616+0000 7fdfd9ffb640 1 -- 172.21.15.1:0/1660361745 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdfdc01b070 con 0x7fdfe414e8b0 2024-03-06T07:20:42.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.616+0000 7fdfd9ffb640 1 --2- 172.21.15.1:0/1660361745 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfc806b5d0 0x7fdfc806da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.616+0000 7fdfd9ffb640 1 -- 172.21.15.1:0/1660361745 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fdfdc089aa0 con 0x7fdfe414e8b0 2024-03-06T07:20:42.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.617+0000 7fdfe95be640 1 --2- 172.21.15.1:0/1660361745 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfc806b5d0 0x7fdfc806da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.618+0000 7fdfe95be640 1 --2- 172.21.15.1:0/1660361745 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfc806b5d0 0x7fdfc806da90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdfe000b7f0 tx=0x7fdfe0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.623+0000 7fdfeb822640 1 -- 172.21.15.1:0/1660361745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc4005290 con 0x7fdfe414e8b0 2024-03-06T07:20:42.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.632+0000 7fdfd9ffb640 1 -- 172.21.15.1:0/1660361745 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdfdc058000 con 0x7fdfe414e8b0 2024-03-06T07:20:42.649 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:20:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.654+0000 7f88cee0e640 1 Processor -- start 2024-03-06T07:20:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.654+0000 7f88cee0e640 1 -- start start 2024-03-06T07:20:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.652+0000 7f20c3148640 1 Processor -- start 2024-03-06T07:20:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.653+0000 7f20c3148640 1 -- start start 2024-03-06T07:20:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.654+0000 7f20c3148640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f20bc06d550 0x7f20bc06d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.654+0000 7f20c3148640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20bc165c30 0x7f20bc163120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.654+0000 7f20c3148640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20bc161ec0 con 0x7f20bc08a970 2024-03-06T07:20:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.654+0000 7f20c3148640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20bc1621a0 con 0x7f20bc06d550 2024-03-06T07:20:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.655+0000 7f88cee0e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88c8150760 0x7f88c80c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.655+0000 7f88cee0e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c80c4ad0 0x7f88c8157040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.655+0000 7f88cee0e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c8157580 con 0x7f88c80c57e0 2024-03-06T07:20:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.655+0000 7f88cee0e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c80c78e0 con 0x7f88c8150760 2024-03-06T07:20:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.655+0000 7f88cee0e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80c7b80 con 0x7f88c80c4ad0 2024-03-06T07:20:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.656+0000 7f88ccbaa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88c8150760 0x7f88c80c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.656+0000 7f88ccbaa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88c8150760 0x7f88c80c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40674/0 (socket says 172.21.15.1:40674) 2024-03-06T07:20:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.656+0000 7f88ccbaa640 1 -- 172.21.15.1:0/1483112246 learned_addr learned my addr 172.21.15.1:0/1483112246 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.656+0000 7f88cd3ab640 1 --2- 172.21.15.1:0/1483112246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c80c4ad0 0x7f88c8157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.657+0000 7f88cd3ab640 1 -- 172.21.15.1:0/1483112246 >> v2:172.21.15.1:3300/0 conn(0x7f88c8150760 msgr2=0x7f88c80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.657+0000 7f88cd3ab640 1 --2- 172.21.15.1:0/1483112246 >> v2:172.21.15.1:3300/0 conn(0x7f88c8150760 0x7f88c80c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.654+0000 7f20c3148640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20bc162480 con 0x7f20bc165c30 2024-03-06T07:20:42.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.657+0000 7f88c77fe640 1 -- 172.21.15.1:0/1483112246 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906997155 0 0) 0x7f88c8157580 con 0x7f88c80c57e0 2024-03-06T07:20:42.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.656+0000 7f51423bf640 1 Processor -- start 2024-03-06T07:20:42.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.657+0000 7f51423bf640 1 -- start start 2024-03-06T07:20:42.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.657+0000 7f88cd3ab640 1 -- 172.21.15.1:0/1483112246 >> v1:172.21.15.1:6789/0 conn(0x7f88c80c57e0 legacy=0x7f88c8156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.657+0000 7f88cd3ab640 1 -- 172.21.15.1:0/1483112246 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c80c7e60 con 0x7f88c80c4ad0 2024-03-06T07:20:42.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.657+0000 7f51f6222640 1 Processor -- start 2024-03-06T07:20:42.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.658+0000 7f51423bf640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f513c0c3720 0x7f513c159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.658+0000 7f51423bf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f513c0c1a10 0x7f513c0c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.658+0000 7f51423bf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513c06d760 con 0x7f513c15cce0 2024-03-06T07:20:42.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.658+0000 7f51423bf640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513c06da40 con 0x7f513c0c3720 2024-03-06T07:20:42.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.658+0000 7f51423bf640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513c06dd20 con 0x7f513c0c1a10 2024-03-06T07:20:42.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.658+0000 7f514095c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f513c15cce0 0x7f513c15d190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:55082/0 (socket says 172.21.15.1:55082) 2024-03-06T07:20:42.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.658+0000 7f514095c640 1 -- 172.21.15.1:0/4139257521 learned_addr learned my addr 172.21.15.1:0/4139257521 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:42.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.658+0000 7f51f6222640 1 -- start start 2024-03-06T07:20:42.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.658+0000 7f513b7fe640 1 --2- 172.21.15.1:0/4139257521 >> v2:172.21.15.98:3300/0 conn(0x7f513c0c1a10 0x7f513c0c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f513affd640 1 -- 172.21.15.1:0/4139257521 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266981246 0 0) 0x7f513c06d760 con 0x7f513c15cce0 2024-03-06T07:20:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f51f6222640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51f00abcf0 0x7f51f00acd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f51f6222640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51f00ac0d0 0x7f51f015caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f51f6222640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f00c37e0 con 0x7f51f00ac7f0 2024-03-06T07:20:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f51f6222640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f01595e0 con 0x7f51f00ac0d0 2024-03-06T07:20:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f51f6222640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f0159880 con 0x7f51f00abcf0 2024-03-06T07:20:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.654+0000 7f20c0ee4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f20bc06d550 0x7f20bc06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.654+0000 7f20c0ee4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f20bc06d550 0x7f20bc06d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40670/0 (socket says 172.21.15.1:40670) 2024-03-06T07:20:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.654+0000 7f20c0ee4640 1 -- 172.21.15.1:0/757485601 learned_addr learned my addr 172.21.15.1:0/757485601 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.654+0000 7f20bbfff640 1 --2- 172.21.15.1:0/757485601 >> v2:172.21.15.98:3300/0 conn(0x7f20bc165c30 0x7f20bc163120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.655+0000 7f20bb7fe640 1 -- 172.21.15.1:0/757485601 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116416294 0 0) 0x7f20bc161ec0 con 0x7f20bc08a970 2024-03-06T07:20:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.655+0000 7f20bb7fe640 1 -- 172.21.15.1:0/757485601 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20a4003500 con 0x7f20bc08a970 2024-03-06T07:20:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.655+0000 7f20bbfff640 1 -- 172.21.15.1:0/757485601 >> v2:172.21.15.1:3300/0 conn(0x7f20bc06d550 msgr2=0x7f20bc06d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.655+0000 7f20bbfff640 1 --2- 172.21.15.1:0/757485601 >> v2:172.21.15.1:3300/0 conn(0x7f20bc06d550 0x7f20bc06d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.655+0000 7f20bbfff640 1 -- 172.21.15.1:0/757485601 >> v1:172.21.15.98:6789/0 conn(0x7f20bc08a970 legacy=0x7f20bc163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.655+0000 7f20bbfff640 1 -- 172.21.15.1:0/757485601 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20bc162760 con 0x7f20bc165c30 2024-03-06T07:20:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.656+0000 7f20c0ee4640 1 --2- 172.21.15.1:0/757485601 >> v2:172.21.15.1:3300/0 conn(0x7f20bc06d550 0x7f20bc06d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.657+0000 7f20bbfff640 1 --2- 172.21.15.1:0/757485601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20bc165c30 0x7f20bc163120 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20bc0c3f50 tx=0x7f20ac02bb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98bfa1c0b72ced04 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.657+0000 7f20bb7fe640 1 -- 172.21.15.1:0/757485601 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ac00e4e0 con 0x7f20bc165c30 2024-03-06T07:20:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.658+0000 7f20bb7fe640 1 -- 172.21.15.1:0/757485601 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20ac00c730 con 0x7f20bc165c30 2024-03-06T07:20:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.658+0000 7f20bb7fe640 1 -- 172.21.15.1:0/757485601 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ac00cb00 con 0x7f20bc165c30 2024-03-06T07:20:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f20c3148640 1 -- 172.21.15.1:0/757485601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20bc165c30 msgr2=0x7f20bc163120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f20c3148640 1 --2- 172.21.15.1:0/757485601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20bc165c30 0x7f20bc163120 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20bc0c3f50 tx=0x7f20ac02bb30 comp rx=0 tx=0).stop 2024-03-06T07:20:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f20c3148640 1 -- 172.21.15.1:0/757485601 shutdown_connections 2024-03-06T07:20:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f20c3148640 1 --2- 172.21.15.1:0/757485601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20bc165c30 0x7f20bc163120 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f20c3148640 1 --2- 172.21.15.1:0/757485601 >> v2:172.21.15.1:3300/0 conn(0x7f20bc06d550 0x7f20bc06d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f20c3148640 1 -- 172.21.15.1:0/757485601 >> 172.21.15.1:0/757485601 conn(0x7f20bc0c6860 msgr2=0x7f20bc0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f20c3148640 1 -- 172.21.15.1:0/757485601 shutdown_connections 2024-03-06T07:20:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f20c3148640 1 -- 172.21.15.1:0/757485601 wait complete. 2024-03-06T07:20:42.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f51f5a21640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51f00ac0d0 0x7f51f015caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f51f5a21640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51f00ac0d0 0x7f51f015caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40696/0 (socket says 172.21.15.1:40696) 2024-03-06T07:20:42.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f51f5a21640 1 -- 172.21.15.1:0/3870848469 learned_addr learned my addr 172.21.15.1:0/3870848469 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:42.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.660+0000 7f51f5220640 1 --2- 172.21.15.1:0/3870848469 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51f00abcf0 0x7f51f00acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f513affd640 1 -- 172.21.15.1:0/4139257521 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5124003520 con 0x7f513c15cce0 2024-03-06T07:20:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f513b7fe640 1 -- 172.21.15.1:0/4139257521 >> v2:172.21.15.1:3300/0 conn(0x7f513c0c3720 msgr2=0x7f513c159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f513b7fe640 1 --2- 172.21.15.1:0/4139257521 >> v2:172.21.15.1:3300/0 conn(0x7f513c0c3720 0x7f513c159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f513b7fe640 1 -- 172.21.15.1:0/4139257521 >> v1:172.21.15.98:6789/0 conn(0x7f513c15cce0 legacy=0x7f513c15d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.659+0000 7f513b7fe640 1 -- 172.21.15.1:0/4139257521 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c06e000 con 0x7f513c0c1a10 2024-03-06T07:20:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.660+0000 7f88ccbaa640 1 --2- 172.21.15.1:0/1483112246 >> v2:172.21.15.1:3300/0 conn(0x7f88c8150760 0x7f88c80c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:42.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.660+0000 7f51f5a21640 1 -- 172.21.15.1:0/3870848469 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51f00abcf0 msgr2=0x7f51f00acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:42.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.661+0000 7f51f5a21640 1 --2- 172.21.15.1:0/3870848469 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51f00abcf0 0x7f51f00acd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.661+0000 7f20c3148640 1 Processor -- start 2024-03-06T07:20:42.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.661+0000 7f51f5a21640 1 -- 172.21.15.1:0/3870848469 >> v1:172.21.15.1:6789/0 conn(0x7f51f00ac7f0 legacy=0x7f51f0072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.661+0000 7f51f5a21640 1 -- 172.21.15.1:0/3870848469 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f0159b60 con 0x7f51f00ac0d0 2024-03-06T07:20:42.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.661+0000 7f20c3148640 1 -- start start 2024-03-06T07:20:42.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.661+0000 7f513b7fe640 1 --2- 172.21.15.1:0/4139257521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c0c1a10 0x7f513c0c3b00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f513c0c43d0 tx=0x7f512c02b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7efe0be43587280f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.661+0000 7f513affd640 1 -- 172.21.15.1:0/4139257521 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c02e070 con 0x7f513c0c1a10 2024-03-06T07:20:42.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.661+0000 7f513affd640 1 -- 172.21.15.1:0/4139257521 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512c037070 con 0x7f513c0c1a10 2024-03-06T07:20:42.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.661+0000 7f5bbf59e640 1 Processor -- start 2024-03-06T07:20:42.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f20c3148640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20bc06d550 0x7f20bc15c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f20c3148640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20bc08a970 0x7f20bc164ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f513affd640 1 -- 172.21.15.1:0/4139257521 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c031a40 con 0x7f513c0c1a10 2024-03-06T07:20:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f20c3148640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20bc165c30 0x7f20bc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f20c3148640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc16c570 con 0x7f20bc165c30 2024-03-06T07:20:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f20c3148640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc0c1630 con 0x7f20bc06d550 2024-03-06T07:20:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f20c3148640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc16c6f0 con 0x7f20bc08a970 2024-03-06T07:20:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f51423bf640 1 -- 172.21.15.1:0/4139257521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c0c1a10 msgr2=0x7f513c0c3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f51423bf640 1 --2- 172.21.15.1:0/4139257521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c0c1a10 0x7f513c0c3b00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f513c0c43d0 tx=0x7f512c02b880 comp rx=0 tx=0).stop 2024-03-06T07:20:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f51423bf640 1 -- 172.21.15.1:0/4139257521 shutdown_connections 2024-03-06T07:20:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f51423bf640 1 --2- 172.21.15.1:0/4139257521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c0c1a10 0x7f513c0c3b00 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f513c0c43d0 tx=0x7f512c02b880 comp rx=0 tx=0).stop 2024-03-06T07:20:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f51423bf640 1 --2- 172.21.15.1:0/4139257521 >> v2:172.21.15.1:3300/0 conn(0x7f513c0c3720 0x7f513c159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f51423bf640 1 -- 172.21.15.1:0/4139257521 >> 172.21.15.1:0/4139257521 conn(0x7f513c0c6860 msgr2=0x7f513c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f51423bf640 1 -- 172.21.15.1:0/4139257521 shutdown_connections 2024-03-06T07:20:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f20c0ee4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20bc06d550 0x7f20bc15c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f20c16e5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20bc165c30 0x7f20bc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f20c16e5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20bc165c30 0x7f20bc071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40712/0 (socket says 172.21.15.1:40712) 2024-03-06T07:20:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f20c16e5640 1 -- 172.21.15.1:0/1055487035 learned_addr learned my addr 172.21.15.1:0/1055487035 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.663+0000 7f20bbfff640 1 --2- 172.21.15.1:0/1055487035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20bc08a970 0x7f20bc164ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.661+0000 7f5bbf59e640 1 -- start start 2024-03-06T07:20:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.663+0000 7f51f5a21640 1 --2- 172.21.15.1:0/3870848469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51f00ac0d0 0x7f51f015caa0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009e50 tx=0x7f51e802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a43e3b74526c4131 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.663+0000 7f5bbf59e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5bb80d80a0 0x7f5bb80e1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.663+0000 7f5bbf59e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb80d3f40 con 0x7f5bb80d3b60 2024-03-06T07:20:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.663+0000 7f5bbf59e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb80d71f0 con 0x7f5bb800bac0 2024-03-06T07:20:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.663+0000 7f51e7fff640 1 -- 172.21.15.1:0/3870848469 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e802c9f0 con 0x7f51f00ac0d0 2024-03-06T07:20:42.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.663+0000 7f5bbf59e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb80d6fa0 con 0x7f5bb80d80a0 2024-03-06T07:20:42.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.663+0000 7f51e7fff640 1 -- 172.21.15.1:0/3870848469 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e802cb90 con 0x7f51f00ac0d0 2024-03-06T07:20:42.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.663+0000 7f5bbdd9b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5bb800bac0 0x7f5bb80d7990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:55092/0 (socket says 172.21.15.1:55092) 2024-03-06T07:20:42.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.663+0000 7f5bbdd9b640 1 -- 172.21.15.1:0/1748406043 learned_addr learned my addr 172.21.15.1:0/1748406043 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:42.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.663+0000 7f5bbd59a640 1 -- 172.21.15.1:0/1748406043 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877196657 0 0) 0x7f5bb80d71f0 con 0x7f5bb800bac0 2024-03-06T07:20:42.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.663+0000 7f5bbd59a640 1 -- 172.21.15.1:0/1748406043 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba8003500 con 0x7f5bb800bac0 2024-03-06T07:20:42.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.663+0000 7f5bbd59a640 1 -- 172.21.15.1:0/1748406043 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3449790509 0 0) 0x7f5ba8003500 con 0x7f5bb800bac0 2024-03-06T07:20:42.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.663+0000 7f5bbd59a640 1 -- 172.21.15.1:0/1748406043 >> v2:172.21.15.98:3300/0 conn(0x7f5bb80d80a0 msgr2=0x7f5bb80e1dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:20:42.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.663+0000 7f5bbd59a640 1 --2- 172.21.15.1:0/1748406043 >> v2:172.21.15.98:3300/0 conn(0x7f5bb80d80a0 0x7f5bb80e1dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.664+0000 7f5bbd59a640 1 -- 172.21.15.1:0/1748406043 >> v1:172.21.15.1:6789/0 conn(0x7f5bb80d3b60 legacy=0x7f5bb8006bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.664+0000 7f5bbd59a640 1 -- 172.21.15.1:0/1748406043 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb8046450 con 0x7f5bb800bac0 2024-03-06T07:20:42.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.664+0000 7f5bbd59a640 1 -- 172.21.15.1:0/1748406043 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f5bb0004cd0 con 0x7f5bb800bac0 2024-03-06T07:20:42.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.664+0000 7f51f6222640 1 -- 172.21.15.1:0/3870848469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51f00ac0d0 msgr2=0x7f51f015caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.664+0000 7f5bbd59a640 1 -- 172.21.15.1:0/1748406043 >> v1:172.21.15.98:6789/0 conn(0x7f5bb800bac0 legacy=0x7f5bb80d7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.664+0000 7f5bbd59a640 1 --2- 172.21.15.1:0/1748406043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ba8003b60 0x7f5ba8003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.664+0000 7f5bbd59a640 1 --2- 172.21.15.1:0/1748406043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ba8006870 0x7f5ba8008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.664+0000 7f5bbd59a640 1 --2- 172.21.15.1:0/1748406043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ba80095a0 0x7f5ba8009a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.664+0000 7f5bbd59a640 1 -- 172.21.15.1:0/1748406043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb80d6fa0 con 0x7f5ba8006870 2024-03-06T07:20:42.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.664+0000 7f5bbd59a640 1 -- 172.21.15.1:0/1748406043 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba800a330 con 0x7f5ba80095a0 2024-03-06T07:20:42.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.664+0000 7f5bbd59a640 1 -- 172.21.15.1:0/1748406043 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba800a530 con 0x7f5ba8003b60 2024-03-06T07:20:42.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.664+0000 7f5bbdd9b640 1 --2- 172.21.15.1:0/1748406043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ba8006870 0x7f5ba8008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.664+0000 7f51e7fff640 1 -- 172.21.15.1:0/3870848469 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e802c9f0 con 0x7f51f00ac0d0 2024-03-06T07:20:42.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f51f6222640 1 --2- 172.21.15.1:0/3870848469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51f00ac0d0 0x7f51f015caa0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009e50 tx=0x7f51e802be50 comp rx=0 tx=0).stop 2024-03-06T07:20:42.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f51f6222640 1 -- 172.21.15.1:0/3870848469 shutdown_connections 2024-03-06T07:20:42.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f51f6222640 1 --2- 172.21.15.1:0/3870848469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51f00ac0d0 0x7f51f015caa0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f51f6222640 1 --2- 172.21.15.1:0/3870848469 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51f00abcf0 0x7f51f00acd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f51f6222640 1 -- 172.21.15.1:0/3870848469 >> 172.21.15.1:0/3870848469 conn(0x7f51f0071b80 msgr2=0x7f51f0071f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:42.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f51f6222640 1 -- 172.21.15.1:0/3870848469 shutdown_connections 2024-03-06T07:20:42.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f20c16e5640 1 -- 172.21.15.1:0/1055487035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20bc08a970 msgr2=0x7f20bc164ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f20c16e5640 1 --2- 172.21.15.1:0/1055487035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20bc08a970 0x7f20bc164ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f20c16e5640 1 -- 172.21.15.1:0/1055487035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20bc06d550 msgr2=0x7f20bc15c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f20c16e5640 1 --2- 172.21.15.1:0/1055487035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20bc06d550 0x7f20bc15c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f20c16e5640 1 -- 172.21.15.1:0/1055487035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20bc15ccc0 con 0x7f20bc165c30 2024-03-06T07:20:42.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f20c0ee4640 1 --2- 172.21.15.1:0/1055487035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20bc06d550 0x7f20bc15c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:42.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f88cd3ab640 1 --2- 172.21.15.1:0/1483112246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c80c4ad0 0x7f88c8157040 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f88b80097a0 tx=0x7f88b802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1e444f53b4959b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f5bbdd9b640 1 -- 172.21.15.1:0/1748406043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ba8003b60 msgr2=0x7f5ba8003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:42.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f5bbdd9b640 1 --2- 172.21.15.1:0/1748406043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ba8003b60 0x7f5ba8003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f5bbdd9b640 1 -- 172.21.15.1:0/1748406043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ba80095a0 msgr2=0x7f5ba8009a50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:20:42.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f5bbdd9b640 1 --2- 172.21.15.1:0/1748406043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ba80095a0 0x7f5ba8009a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f88c77fe640 1 -- 172.21.15.1:0/1483112246 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8031780 con 0x7f88c80c4ad0 2024-03-06T07:20:42.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f88c77fe640 1 -- 172.21.15.1:0/1483112246 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b8037070 con 0x7f88c80c4ad0 2024-03-06T07:20:42.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f88c77fe640 1 -- 172.21.15.1:0/1483112246 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8031b40 con 0x7f88c80c4ad0 2024-03-06T07:20:42.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f5bbdd9b640 1 -- 172.21.15.1:0/1748406043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb8046450 con 0x7f5ba8006870 2024-03-06T07:20:42.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f5bc4d8a640 1 -- 172.21.15.1:0/1748406043 reap_dead start 2024-03-06T07:20:42.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.662+0000 7f51423bf640 1 -- 172.21.15.1:0/4139257521 wait complete. 2024-03-06T07:20:42.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f20c16e5640 1 --2- 172.21.15.1:0/1055487035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20bc165c30 0x7f20bc071ce0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f20b400c7a0 tx=0x7f20b400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.664+0000 7f51423bf640 1 Processor -- start 2024-03-06T07:20:42.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.664+0000 7f51423bf640 1 -- start start 2024-03-06T07:20:42.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f20b97fa640 1 -- 172.21.15.1:0/1055487035 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b4003b70 con 0x7f20bc165c30 2024-03-06T07:20:42.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f51423bf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f513c0c3720 0x7f513c0aa540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f51423bf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c15cce0 0x7f513c1435c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f51423bf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f513c151070 0x7f513c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f20c3148640 1 -- 172.21.15.1:0/1055487035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20bc164300 con 0x7f20bc165c30 2024-03-06T07:20:42.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f51423bf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c15d8a0 con 0x7f513c151070 2024-03-06T07:20:42.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f51423bf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c156dd0 con 0x7f513c15cce0 2024-03-06T07:20:42.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f20c3148640 1 -- 172.21.15.1:0/1055487035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20bc1f8ab0 con 0x7f20bc165c30 2024-03-06T07:20:42.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f51423bf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513c163e00 con 0x7f513c0c3720 2024-03-06T07:20:42.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f514095c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f513c151070 0x7f513c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f5bbdd9b640 1 --2- 172.21.15.1:0/1748406043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ba8006870 0x7f5ba8008d00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5bb000d360 tx=0x7f5ba80037f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bea0488d1f9d613 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f5bbd59a640 1 -- 172.21.15.1:0/1748406043 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0038070 con 0x7f5ba8006870 2024-03-06T07:20:42.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f514095c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f513c151070 0x7f513c071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40730/0 (socket says 172.21.15.1:40730) 2024-03-06T07:20:42.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f5bbd59a640 1 -- 172.21.15.1:0/1748406043 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb0009040 con 0x7f5ba8006870 2024-03-06T07:20:42.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f514095c640 1 -- 172.21.15.1:0/1723167330 learned_addr learned my addr 172.21.15.1:0/1723167330 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:42.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f5bbd59a640 1 -- 172.21.15.1:0/1748406043 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0001b80 con 0x7f5ba8006870 2024-03-06T07:20:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f514095c640 1 -- 172.21.15.1:0/1723167330 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f513c0c3720 msgr2=0x7f513c0aa540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:20:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f514095c640 1 --2- 172.21.15.1:0/1723167330 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f513c0c3720 0x7f513c0aa540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f514095c640 1 -- 172.21.15.1:0/1723167330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c15cce0 msgr2=0x7f513c1435c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:20:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f514095c640 1 --2- 172.21.15.1:0/1723167330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c15cce0 0x7f513c1435c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.666+0000 7f514095c640 1 -- 172.21.15.1:0/1723167330 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c142b40 con 0x7f513c151070 2024-03-06T07:20:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.667+0000 7f514095c640 1 --2- 172.21.15.1:0/1723167330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f513c151070 0x7f513c071ce0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f513400cb10 tx=0x7f513400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.667+0000 7f20b97fa640 1 -- 172.21.15.1:0/1055487035 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b4003d30 con 0x7f20bc165c30 2024-03-06T07:20:42.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.665+0000 7f51f6222640 1 -- 172.21.15.1:0/3870848469 wait complete. 2024-03-06T07:20:42.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.667+0000 7f51f6222640 1 Processor -- start 2024-03-06T07:20:42.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.667+0000 7f20b97fa640 1 -- 172.21.15.1:0/1055487035 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b400aa90 con 0x7f20bc165c30 2024-03-06T07:20:42.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.667+0000 7f5bbf59e640 1 -- 172.21.15.1:0/1748406043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ba8006870 msgr2=0x7f5ba8008d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.667+0000 7f5bbf59e640 1 --2- 172.21.15.1:0/1748406043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ba8006870 0x7f5ba8008d00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5bb000d360 tx=0x7f5ba80037f0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.668+0000 7f5138ff9640 1 -- 172.21.15.1:0/1723167330 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5134017070 con 0x7f513c151070 2024-03-06T07:20:42.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.668+0000 7f5138ff9640 1 -- 172.21.15.1:0/1723167330 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513400ac70 con 0x7f513c151070 2024-03-06T07:20:42.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.667+0000 7f51f6222640 1 -- start start 2024-03-06T07:20:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.668+0000 7f51f6222640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51f00abcf0 0x7f51f00c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.668+0000 7f51f6222640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51f00ac0d0 0x7f51f015bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.668+0000 7f51f6222640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51f00ac7f0 0x7f51f015c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.668+0000 7f51f6222640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f0072f00 con 0x7f51f00ac0d0 2024-03-06T07:20:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.668+0000 7f51f6222640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f0072d80 con 0x7f51f00ac7f0 2024-03-06T07:20:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.668+0000 7f51f6222640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f015cfe0 con 0x7f51f00abcf0 2024-03-06T07:20:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.668+0000 7f5138ff9640 1 -- 172.21.15.1:0/1723167330 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5134012600 con 0x7f513c151070 2024-03-06T07:20:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.668+0000 7f51423bf640 1 -- 172.21.15.1:0/1723167330 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513c140530 con 0x7f513c151070 2024-03-06T07:20:42.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.668+0000 7f51423bf640 1 -- 172.21.15.1:0/1723167330 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513c0ac4e0 con 0x7f513c151070 2024-03-06T07:20:42.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f51f4a1f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51f00ac0d0 0x7f51f015bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f51f5a21640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51f00ac7f0 0x7f51f015c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f51f4a1f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51f00ac0d0 0x7f51f015bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40734/0 (socket says 172.21.15.1:40734) 2024-03-06T07:20:42.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f51f4a1f640 1 -- 172.21.15.1:0/542918781 learned_addr learned my addr 172.21.15.1:0/542918781 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:42.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f88cee0e640 1 -- 172.21.15.1:0/1483112246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c80c4ad0 msgr2=0x7f88c8157040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f88cee0e640 1 --2- 172.21.15.1:0/1483112246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c80c4ad0 0x7f88c8157040 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f88b80097a0 tx=0x7f88b802b820 comp rx=0 tx=0).stop 2024-03-06T07:20:42.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f5bbf59e640 1 -- 172.21.15.1:0/1748406043 shutdown_connections 2024-03-06T07:20:42.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f5bbf59e640 1 --2- 172.21.15.1:0/1748406043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ba8006870 0x7f5ba8008d00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f5bbf59e640 1 -- 172.21.15.1:0/1748406043 >> 172.21.15.1:0/1748406043 conn(0x7f5bb80060b0 msgr2=0x7f5bb80064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:42.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f51f4a1f640 1 -- 172.21.15.1:0/542918781 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51f00abcf0 msgr2=0x7f51f00c5fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:20:42.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f51f4a1f640 1 --2- 172.21.15.1:0/542918781 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51f00abcf0 0x7f51f00c5fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f51f4a1f640 1 -- 172.21.15.1:0/542918781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51f00ac7f0 msgr2=0x7f51f015c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f51f4a1f640 1 --2- 172.21.15.1:0/542918781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51f00ac7f0 0x7f51f015c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f51f4a1f640 1 -- 172.21.15.1:0/542918781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f01fa700 con 0x7f51f00ac0d0 2024-03-06T07:20:42.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f88cee0e640 1 -- 172.21.15.1:0/1483112246 shutdown_connections 2024-03-06T07:20:42.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f88cee0e640 1 --2- 172.21.15.1:0/1483112246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c80c4ad0 0x7f88c8157040 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f88cee0e640 1 --2- 172.21.15.1:0/1483112246 >> v2:172.21.15.1:3300/0 conn(0x7f88c8150760 0x7f88c80c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f88cee0e640 1 -- 172.21.15.1:0/1483112246 >> 172.21.15.1:0/1483112246 conn(0x7f88c80c8810 msgr2=0x7f88c80c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:42.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.670+0000 7f88cee0e640 1 -- 172.21.15.1:0/1483112246 shutdown_connections 2024-03-06T07:20:42.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.670+0000 7f20c3148640 1 -- 172.21.15.1:0/1055487035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209c005290 con 0x7f20bc165c30 2024-03-06T07:20:42.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.670+0000 7f51f4a1f640 1 --2- 172.21.15.1:0/542918781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51f00ac0d0 0x7f51f015bc80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00c450 tx=0x7f51ec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.670+0000 7f88cee0e640 1 -- 172.21.15.1:0/1483112246 wait complete. 2024-03-06T07:20:42.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.671+0000 7f88cee0e640 1 Processor -- start 2024-03-06T07:20:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.671+0000 7f5138ff9640 1 -- 172.21.15.1:0/1723167330 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f513401a070 con 0x7f513c151070 2024-03-06T07:20:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.671+0000 7f51423bf640 1 -- 172.21.15.1:0/1723167330 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f511c005290 con 0x7f513c151070 2024-03-06T07:20:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.672+0000 7f5138ff9640 1 --2- 172.21.15.1:0/1723167330 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f512806b520 0x7f512806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.671+0000 7f88cee0e640 1 -- start start 2024-03-06T07:20:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f88cee0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88c80c4ad0 0x7f88c8154540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f88cee0e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88c80c57e0 0x7f88c8155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f88cee0e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c8150760 0x7f88c8155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f88cee0e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80c3280 con 0x7f88c80c57e0 2024-03-06T07:20:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f88cee0e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80c8420 con 0x7f88c80c4ad0 2024-03-06T07:20:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f88cee0e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80c3400 con 0x7f88c8150760 2024-03-06T07:20:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f88cd3ab640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c8150760 0x7f88c8155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.674+0000 7f88cd3ab640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c8150760 0x7f88c8155b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:41580/0 (socket says 172.21.15.1:41580) 2024-03-06T07:20:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.674+0000 7f88cd3ab640 1 -- 172.21.15.1:0/3628778103 learned_addr learned my addr 172.21.15.1:0/3628778103 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f5138ff9640 1 -- 172.21.15.1:0/1723167330 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f513408a2b0 con 0x7f513c151070 2024-03-06T07:20:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.669+0000 7f5bbf59e640 1 -- 172.21.15.1:0/1748406043 shutdown_connections 2024-03-06T07:20:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.670+0000 7f5bbf59e640 1 -- 172.21.15.1:0/1748406043 wait complete. 2024-03-06T07:20:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.671+0000 7f5bbf59e640 1 Processor -- start 2024-03-06T07:20:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.671+0000 7f5bbf59e640 1 -- start start 2024-03-06T07:20:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f5bbf59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ba8006870 0x7f5bb80da400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f5bbf59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bb80d84a0 0x7f5bb80cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f5bbf59e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bb804a070 0x7f5bb818af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f5bbf59e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb802e370 con 0x7f5ba8006870 2024-03-06T07:20:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f5bbf59e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb80d74d0 con 0x7f5bb80d84a0 2024-03-06T07:20:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f5bbf59e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb8046d30 con 0x7f5bb804a070 2024-03-06T07:20:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f5bbdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bb80d84a0 0x7f5bb80cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f5bbdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bb80d84a0 0x7f5bb80cf1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:44674/0 (socket says 172.21.15.1:44674) 2024-03-06T07:20:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f5bbdd9b640 1 -- 172.21.15.1:0/2538201451 learned_addr learned my addr 172.21.15.1:0/2538201451 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f5bbe59c640 1 --2- 172.21.15.1:0/2538201451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ba8006870 0x7f5bb80da400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.674+0000 7f88c7fff640 1 --2- 172.21.15.1:0/3628778103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88c80c57e0 0x7f88c8155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.672+0000 7f20b97fa640 1 -- 172.21.15.1:0/1055487035 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f20b400ac30 con 0x7f20bc165c30 2024-03-06T07:20:42.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.672+0000 7f20b97fa640 1 --2- 172.21.15.1:0/1055487035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20a806b4d0 0x7f20a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f20c0ee4640 1 --2- 172.21.15.1:0/1055487035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20a806b4d0 0x7f20a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.672+0000 7f51e5ffb640 1 -- 172.21.15.1:0/542918781 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0052d0 con 0x7f51f00ac0d0 2024-03-06T07:20:42.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.672+0000 7f51e5ffb640 1 -- 172.21.15.1:0/542918781 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51ec005470 con 0x7f51f00ac0d0 2024-03-06T07:20:42.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.672+0000 7f51e5ffb640 1 -- 172.21.15.1:0/542918781 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec012870 con 0x7f51f00ac0d0 2024-03-06T07:20:42.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f51f6222640 1 -- 172.21.15.1:0/542918781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f01fa9a0 con 0x7f51f00ac0d0 2024-03-06T07:20:42.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.673+0000 7f51f6222640 1 -- 172.21.15.1:0/542918781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f01faf60 con 0x7f51f00ac0d0 2024-03-06T07:20:42.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.674+0000 7f88cd3ab640 1 -- 172.21.15.1:0/3628778103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88c80c4ad0 msgr2=0x7f88c8154540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:20:42.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.674+0000 7f88cd3ab640 1 --2- 172.21.15.1:0/3628778103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88c80c4ad0 0x7f88c8154540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.675+0000 7f51f6222640 1 -- 172.21.15.1:0/542918781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f01f7160 con 0x7f51f00ac0d0 2024-03-06T07:20:42.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.674+0000 7f88cd3ab640 1 -- 172.21.15.1:0/3628778103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88c80c57e0 msgr2=0x7f88c8155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.674+0000 7f88cd3ab640 1 --2- 172.21.15.1:0/3628778103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88c80c57e0 0x7f88c8155640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.674+0000 7f88cd3ab640 1 -- 172.21.15.1:0/3628778103 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c8140170 con 0x7f88c8150760 2024-03-06T07:20:42.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.674+0000 7f5bc4d8a640 1 --2- 172.21.15.1:0/2538201451 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bb804a070 0x7f5bb818af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.675+0000 7f513bfff640 1 --2- 172.21.15.1:0/1723167330 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f512806b520 0x7f512806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.675+0000 7f88cd3ab640 1 --2- 172.21.15.1:0/3628778103 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c8150760 0x7f88c8155b80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f88b8034040 tx=0x7f88b80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.675+0000 7f5bbdd9b640 1 -- 172.21.15.1:0/2538201451 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bb804a070 msgr2=0x7f5bb818af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.675+0000 7f5bbdd9b640 1 --2- 172.21.15.1:0/2538201451 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bb804a070 0x7f5bb818af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.675+0000 7f5bbdd9b640 1 -- 172.21.15.1:0/2538201451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ba8006870 msgr2=0x7f5bb80da400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.675+0000 7f5bbdd9b640 1 --2- 172.21.15.1:0/2538201451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ba8006870 0x7f5bb80da400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.675+0000 7f5bbdd9b640 1 -- 172.21.15.1:0/2538201451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb80d4da0 con 0x7f5bb80d84a0 2024-03-06T07:20:42.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.675+0000 7f5bbe59c640 1 --2- 172.21.15.1:0/2538201451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ba8006870 0x7f5bb80da400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:42.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.675+0000 7f88c57fa640 1 -- 172.21.15.1:0/3628778103 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8032610 con 0x7f88c8150760 2024-03-06T07:20:42.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.675+0000 7f88c57fa640 1 -- 172.21.15.1:0/3628778103 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b80327d0 con 0x7f88c8150760 2024-03-06T07:20:42.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.675+0000 7f88c57fa640 1 -- 172.21.15.1:0/3628778103 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8032bc0 con 0x7f88c8150760 2024-03-06T07:20:42.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.675+0000 7f88cee0e640 1 -- 172.21.15.1:0/3628778103 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c813dbf0 con 0x7f88c8150760 2024-03-06T07:20:42.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.676+0000 7f513bfff640 1 --2- 172.21.15.1:0/1723167330 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f512806b520 0x7f512806d9e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f513c155b30 tx=0x7f5130006c20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.676+0000 7f20c0ee4640 1 --2- 172.21.15.1:0/1055487035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20a806b4d0 0x7f20a806d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f20b0009220 tx=0x7f20b0008050 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.676+0000 7f20b97fa640 1 -- 172.21.15.1:0/1055487035 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f20b408a480 con 0x7f20bc165c30 2024-03-06T07:20:42.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.676+0000 7f5bbdd9b640 1 --2- 172.21.15.1:0/2538201451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bb80d84a0 0x7f5bb80cf1c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5bb000d450 tx=0x7f5bb0004fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.675+0000 7f88cee0e640 1 -- 172.21.15.1:0/3628778103 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c8154a80 con 0x7f88c8150760 2024-03-06T07:20:42.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.676+0000 7f5baeffd640 1 -- 172.21.15.1:0/2538201451 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0038040 con 0x7f5bb80d84a0 2024-03-06T07:20:42.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.676+0000 7f5baeffd640 1 -- 172.21.15.1:0/2538201451 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb0033420 con 0x7f5bb80d84a0 2024-03-06T07:20:42.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.676+0000 7f5baeffd640 1 -- 172.21.15.1:0/2538201451 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0001b80 con 0x7f5bb80d84a0 2024-03-06T07:20:42.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.677+0000 7f5bbf59e640 1 -- 172.21.15.1:0/2538201451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb80d5080 con 0x7f5bb80d84a0 2024-03-06T07:20:42.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.677+0000 7f5bbf59e640 1 -- 172.21.15.1:0/2538201451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb80d5d80 con 0x7f5bb80d84a0 2024-03-06T07:20:42.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.677+0000 7f88cee0e640 1 -- 172.21.15.1:0/3628778103 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c80686e0 con 0x7f88c8150760 2024-03-06T07:20:42.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.678+0000 7f5bbf59e640 1 -- 172.21.15.1:0/2538201451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb8005490 con 0x7f5bb80d84a0 2024-03-06T07:20:42.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.679+0000 7f5baeffd640 1 -- 172.21.15.1:0/2538201451 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5bb0047020 con 0x7f5bb80d84a0 2024-03-06T07:20:42.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.679+0000 7f88c57fa640 1 -- 172.21.15.1:0/3628778103 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88b80453f0 con 0x7f88c8150760 2024-03-06T07:20:42.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.679+0000 7f5baeffd640 1 --2- 172.21.15.1:0/2538201451 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ba006b500 0x7f5ba006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.680+0000 7f5baeffd640 1 -- 172.21.15.1:0/2538201451 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5bb00abe40 con 0x7f5bb80d84a0 2024-03-06T07:20:42.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.680+0000 7f88c57fa640 1 --2- 172.21.15.1:0/3628778103 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88b406b520 0x7f88b406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.680+0000 7f5bbe59c640 1 --2- 172.21.15.1:0/2538201451 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ba006b500 0x7f5ba006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.680+0000 7f88c57fa640 1 -- 172.21.15.1:0/3628778103 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f88b80abf50 con 0x7f88c8150760 2024-03-06T07:20:42.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.680+0000 7f88ccbaa640 1 --2- 172.21.15.1:0/3628778103 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88b406b520 0x7f88b406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.680+0000 7f5bbe59c640 1 --2- 172.21.15.1:0/2538201451 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ba006b500 0x7f5ba006d9c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0050c30 tx=0x7f5bc0068fe0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.681+0000 7f88ccbaa640 1 --2- 172.21.15.1:0/3628778103 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88b406b520 0x7f88b406d9e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009c90 tx=0x7f88c000b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.681+0000 7f5138ff9640 1 -- 172.21.15.1:0/1723167330 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f513c142b40 con 0x7f513c151070 2024-03-06T07:20:42.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.684+0000 7f5baeffd640 1 -- 172.21.15.1:0/2538201451 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5bb007a320 con 0x7f5bb80d84a0 2024-03-06T07:20:42.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.685+0000 7f88c57fa640 1 -- 172.21.15.1:0/3628778103 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88b8040070 con 0x7f88c8150760 2024-03-06T07:20:42.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.686+0000 7f20b97fa640 1 -- 172.21.15.1:0/1055487035 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f20b4058f20 con 0x7f20bc165c30 2024-03-06T07:20:42.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.690+0000 7f51e5ffb640 1 -- 172.21.15.1:0/542918781 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f51ec012a10 con 0x7f51f00ac0d0 2024-03-06T07:20:42.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.690+0000 7f51e5ffb640 1 --2- 172.21.15.1:0/542918781 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51d406b520 0x7f51d406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.695+0000 7f51e5ffb640 1 -- 172.21.15.1:0/542918781 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f51ec057120 con 0x7f51f00ac0d0 2024-03-06T07:20:42.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.696+0000 7f51e5ffb640 1 -- 172.21.15.1:0/542918781 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51ec059780 con 0x7f51f00ac0d0 2024-03-06T07:20:42.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.698+0000 7f1854f32640 1 Processor -- start 2024-03-06T07:20:42.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.698+0000 7f1854f32640 1 -- start start 2024-03-06T07:20:42.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.696+0000 7f51f5220640 1 --2- 172.21.15.1:0/542918781 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51d406b520 0x7f51d406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.699+0000 7f1854f32640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f18500c4760 0x7f18500c0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.699+0000 7f1854f32640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1850159680 con 0x7f185015cc10 2024-03-06T07:20:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.699+0000 7f1854f32640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1850159960 con 0x7f1850150140 2024-03-06T07:20:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.699+0000 7f1854f32640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f185015a290 con 0x7f18500c4760 2024-03-06T07:20:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.699+0000 7f184dd9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f18500c4760 0x7f18500c0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.699+0000 7f184dd9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f18500c4760 0x7f18500c0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:44686/0 (socket says 172.21.15.1:44686) 2024-03-06T07:20:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.700+0000 7f184dd9b640 1 -- 172.21.15.1:0/3547914489 learned_addr learned my addr 172.21.15.1:0/3547914489 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.699+0000 7f184ed9d640 1 --1- 172.21.15.1:0/3547914489 >> v1:172.21.15.1:6789/0 conn(0x7f185015cc10 0x7f185015d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57526/0 (socket says 172.21.15.1:57526) 2024-03-06T07:20:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.700+0000 7f184dd9b640 1 -- 172.21.15.1:0/3547914489 >> v1:172.21.15.98:6789/0 conn(0x7f1850150140 legacy=0x7f18500c1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.700+0000 7f184dd9b640 1 -- 172.21.15.1:0/3547914489 >> v1:172.21.15.1:6789/0 conn(0x7f185015cc10 legacy=0x7f185015d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.700+0000 7f184dd9b640 1 -- 172.21.15.1:0/3547914489 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1850159c40 con 0x7f18500c4760 2024-03-06T07:20:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.701+0000 7f51f5220640 1 --2- 172.21.15.1:0/542918781 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51d406b520 0x7f51d406d9e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f51e0009930 tx=0x7f51e0009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.701+0000 7f184dd9b640 1 --2- 172.21.15.1:0/3547914489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18500c4760 0x7f18500c0ae0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1840009870 tx=0x7f184002ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=215d9f950dba0598 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.701+0000 7f184d59a640 1 -- 172.21.15.1:0/3547914489 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184002f850 con 0x7f18500c4760 2024-03-06T07:20:42.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.702+0000 7f184d59a640 1 -- 172.21.15.1:0/3547914489 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184002f9f0 con 0x7f18500c4760 2024-03-06T07:20:42.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.703+0000 7f1854f32640 1 -- 172.21.15.1:0/3547914489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18500c4760 msgr2=0x7f18500c0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.703+0000 7f1854f32640 1 --2- 172.21.15.1:0/3547914489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18500c4760 0x7f18500c0ae0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1840009870 tx=0x7f184002ba30 comp rx=0 tx=0).stop 2024-03-06T07:20:42.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.709+0000 7f1854f32640 1 -- 172.21.15.1:0/3547914489 shutdown_connections 2024-03-06T07:20:42.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.709+0000 7f1854f32640 1 --2- 172.21.15.1:0/3547914489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18500c4760 0x7f18500c0ae0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.709+0000 7f1854f32640 1 -- 172.21.15.1:0/3547914489 >> 172.21.15.1:0/3547914489 conn(0x7f18500c6860 msgr2=0x7f18500c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:42.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.715+0000 7f1854f32640 1 -- 172.21.15.1:0/3547914489 shutdown_connections 2024-03-06T07:20:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.719+0000 7f1854f32640 1 -- 172.21.15.1:0/3547914489 wait complete. 2024-03-06T07:20:42.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.721+0000 7f1854f32640 1 Processor -- start 2024-03-06T07:20:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.721+0000 7f1854f32640 1 -- start start 2024-03-06T07:20:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.722+0000 7f1854f32640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18500c4760 0x7f18501f7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.722+0000 7f1854f32640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1850150140 0x7f18500ab850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.722+0000 7f1854f32640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f185015cc10 0x7f18500abd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.722+0000 7f1854f32640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18500c1af0 con 0x7f18500c4760 2024-03-06T07:20:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.723+0000 7f184dd9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1850150140 0x7f18500ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.723+0000 7f184dd9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1850150140 0x7f18500ab850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:41594/0 (socket says 172.21.15.1:41594) 2024-03-06T07:20:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.723+0000 7f184dd9b640 1 -- 172.21.15.1:0/675288495 learned_addr learned my addr 172.21.15.1:0/675288495 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.723+0000 7f1854f32640 1 -- 172.21.15.1:0/675288495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18500c1970 con 0x7f185015cc10 2024-03-06T07:20:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.723+0000 7f1854f32640 1 -- 172.21.15.1:0/675288495 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185015d730 con 0x7f1850150140 2024-03-06T07:20:42.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.725+0000 7f184e59c640 1 --2- 172.21.15.1:0/675288495 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18500c4760 0x7f18501f7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.725+0000 7f184e59c640 1 -- 172.21.15.1:0/675288495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1850150140 msgr2=0x7f18500ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.725+0000 7f184e59c640 1 --2- 172.21.15.1:0/675288495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1850150140 0x7f18500ab850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.725+0000 7f184e59c640 1 -- 172.21.15.1:0/675288495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f185015cc10 msgr2=0x7f18500abd90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:20:42.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.725+0000 7f184e59c640 1 --2- 172.21.15.1:0/675288495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f185015cc10 0x7f18500abd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.725+0000 7f184e59c640 1 -- 172.21.15.1:0/675288495 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18501f83e0 con 0x7f18500c4760 2024-03-06T07:20:42.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.726+0000 7f184e59c640 1 --2- 172.21.15.1:0/675288495 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18500c4760 0x7f18501f7b10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f184800c730 tx=0x7f184800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.726+0000 7f183effd640 1 -- 172.21.15.1:0/675288495 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848016020 con 0x7f18500c4760 2024-03-06T07:20:42.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.726+0000 7f183effd640 1 -- 172.21.15.1:0/675288495 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1848006d50 con 0x7f18500c4760 2024-03-06T07:20:42.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.727+0000 7f183effd640 1 -- 172.21.15.1:0/675288495 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848003bb0 con 0x7f18500c4760 2024-03-06T07:20:42.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.731+0000 7f1854f32640 1 -- 172.21.15.1:0/675288495 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18500c08e0 con 0x7f18500c4760 2024-03-06T07:20:42.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.731+0000 7f1854f32640 1 -- 172.21.15.1:0/675288495 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1850073300 con 0x7f18500c4760 2024-03-06T07:20:42.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.733+0000 7f1854f32640 1 -- 172.21.15.1:0/675288495 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1850067fd0 con 0x7f18500c4760 2024-03-06T07:20:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.744+0000 7f183effd640 1 -- 172.21.15.1:0/675288495 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f184800a7f0 con 0x7f18500c4760 2024-03-06T07:20:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.744+0000 7f183effd640 1 --2- 172.21.15.1:0/675288495 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f183006b5d0 0x7f183006da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.744+0000 7f183effd640 1 -- 172.21.15.1:0/675288495 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1848089bb0 con 0x7f18500c4760 2024-03-06T07:20:42.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.745+0000 7f184dd9b640 1 --2- 172.21.15.1:0/675288495 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f183006b5d0 0x7f183006da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.752+0000 7f183effd640 1 -- 172.21.15.1:0/675288495 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f18480586e0 con 0x7f18500c4760 2024-03-06T07:20:42.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.758+0000 7f184dd9b640 1 --2- 172.21.15.1:0/675288495 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f183006b5d0 0x7f183006da90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f18500bc750 tx=0x7f1840008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.835+0000 7f0f1c9c9640 1 Processor -- start 2024-03-06T07:20:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.841+0000 7f0f1c9c9640 1 -- start start 2024-03-06T07:20:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.842+0000 7f0f1c9c9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0f1814f530 0x7f0f180726c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.842+0000 7f0f1c9c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f1806e6d0 0x7f0f180c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.842+0000 7f0f1c9c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f18152b90 con 0x7f0f18072c00 2024-03-06T07:20:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.842+0000 7f0f1c9c9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f180c6f90 con 0x7f0f1814f530 2024-03-06T07:20:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.842+0000 7f0f1c9c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f180c5950 con 0x7f0f1806e6d0 2024-03-06T07:20:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.842+0000 7f0f1659c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0f1814f530 0x7f0f180726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.842+0000 7f0f1659c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0f1814f530 0x7f0f180726c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:44700/0 (socket says 172.21.15.1:44700) 2024-03-06T07:20:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.842+0000 7f0f1659c640 1 -- 172.21.15.1:0/681880112 learned_addr learned my addr 172.21.15.1:0/681880112 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.843+0000 7f0f1659c640 1 -- 172.21.15.1:0/681880112 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f1806e6d0 msgr2=0x7f0f180c5de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:20:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.843+0000 7f0f1659c640 1 --2- 172.21.15.1:0/681880112 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f1806e6d0 0x7f0f180c5de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.843+0000 7f0f1659c640 1 -- 172.21.15.1:0/681880112 >> v1:172.21.15.98:6789/0 conn(0x7f0f18072c00 legacy=0x7f0f180c6320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:20:42.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.843+0000 7f0f1659c640 1 -- 172.21.15.1:0/681880112 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f180c71e0 con 0x7f0f1814f530 2024-03-06T07:20:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.848+0000 7f0f1659c640 1 --2- 172.21.15.1:0/681880112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f1814f530 0x7f0f180726c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009e50 tx=0x7f0f1002bf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7aead86bb4a3998c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.850+0000 7f0f1559a640 1 -- 172.21.15.1:0/681880112 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1002caa0 con 0x7f0f1814f530 2024-03-06T07:20:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.850+0000 7f0f1559a640 1 -- 172.21.15.1:0/681880112 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1002cc40 con 0x7f0f1814f530 2024-03-06T07:20:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.850+0000 7f0f1559a640 1 -- 172.21.15.1:0/681880112 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f100343f0 con 0x7f0f1814f530 2024-03-06T07:20:42.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.853+0000 7f0f1c9c9640 1 -- 172.21.15.1:0/681880112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f1814f530 msgr2=0x7f0f180726c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.853+0000 7f0f1c9c9640 1 --2- 172.21.15.1:0/681880112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f1814f530 0x7f0f180726c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009e50 tx=0x7f0f1002bf00 comp rx=0 tx=0).stop 2024-03-06T07:20:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.861+0000 7f0f1c9c9640 1 -- 172.21.15.1:0/681880112 shutdown_connections 2024-03-06T07:20:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.861+0000 7f0f1c9c9640 1 --2- 172.21.15.1:0/681880112 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f1806e6d0 0x7f0f180c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.861+0000 7f0f1c9c9640 1 --2- 172.21.15.1:0/681880112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f1814f530 0x7f0f180726c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.861+0000 7f0f1c9c9640 1 -- 172.21.15.1:0/681880112 >> 172.21.15.1:0/681880112 conn(0x7f0f1806c100 msgr2=0x7f0f1806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:42.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.873+0000 7f0f1c9c9640 1 -- 172.21.15.1:0/681880112 shutdown_connections 2024-03-06T07:20:42.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.883+0000 7f0f1c9c9640 1 -- 172.21.15.1:0/681880112 wait complete. 2024-03-06T07:20:42.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.885+0000 7f0f1c9c9640 1 Processor -- start 2024-03-06T07:20:42.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.890+0000 7f0f1c9c9640 1 -- start start 2024-03-06T07:20:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.891+0000 7f0f1c9c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1806e6d0 0x7f0f18150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.891+0000 7f0f1c9c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f18072c00 0x7f0f18149860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.891+0000 7f0f1c9c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f1814f530 0x7f0f18149da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.891+0000 7f0f1c9c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f181527f0 con 0x7f0f1806e6d0 2024-03-06T07:20:42.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.891+0000 7f0f1c9c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1806ff20 con 0x7f0f1814f530 2024-03-06T07:20:42.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.892+0000 7f0f1c9c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f18152970 con 0x7f0f18072c00 2024-03-06T07:20:42.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.894+0000 7f0f1659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1806e6d0 0x7f0f18150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.894+0000 7f0f1659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1806e6d0 0x7f0f18150870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40756/0 (socket says 172.21.15.1:40756) 2024-03-06T07:20:42.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.894+0000 7f0f1659c640 1 -- 172.21.15.1:0/1939770781 learned_addr learned my addr 172.21.15.1:0/1939770781 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:42.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.894+0000 7f0f16d9d640 1 --2- 172.21.15.1:0/1939770781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f1814f530 0x7f0f18149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.895+0000 7f0f1659c640 1 -- 172.21.15.1:0/1939770781 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f18072c00 msgr2=0x7f0f18149860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:20:42.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.895+0000 7f0f1659c640 1 --2- 172.21.15.1:0/1939770781 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f18072c00 0x7f0f18149860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.895+0000 7f0f1659c640 1 -- 172.21.15.1:0/1939770781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f1814f530 msgr2=0x7f0f18149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:42.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.895+0000 7f0f1659c640 1 --2- 172.21.15.1:0/1939770781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f1814f530 0x7f0f18149da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:42.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.895+0000 7f0f1659c640 1 -- 172.21.15.1:0/1939770781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f181e7850 con 0x7f0f1806e6d0 2024-03-06T07:20:42.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.895+0000 7f0f1659c640 1 --2- 172.21.15.1:0/1939770781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1806e6d0 0x7f0f18150870 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0f10036040 tx=0x7f0f10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.896+0000 7f0f06ffd640 1 -- 172.21.15.1:0/1939770781 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f10046070 con 0x7f0f1806e6d0 2024-03-06T07:20:42.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.896+0000 7f0f06ffd640 1 -- 172.21.15.1:0/1939770781 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1002c8a0 con 0x7f0f1806e6d0 2024-03-06T07:20:42.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.896+0000 7f0f06ffd640 1 -- 172.21.15.1:0/1939770781 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f10040ac0 con 0x7f0f1806e6d0 2024-03-06T07:20:42.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.902+0000 7f0f1c9c9640 1 -- 172.21.15.1:0/1939770781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f181e7b30 con 0x7f0f1806e6d0 2024-03-06T07:20:42.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.902+0000 7f0f1c9c9640 1 -- 172.21.15.1:0/1939770781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f181e80f0 con 0x7f0f1806e6d0 2024-03-06T07:20:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.905+0000 7f0f1c9c9640 1 -- 172.21.15.1:0/1939770781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f180686e0 con 0x7f0f1806e6d0 2024-03-06T07:20:42.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.907+0000 7f0f06ffd640 1 -- 172.21.15.1:0/1939770781 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f10005d10 con 0x7f0f1806e6d0 2024-03-06T07:20:42.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.908+0000 7f0f06ffd640 1 --2- 172.21.15.1:0/1939770781 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ef806b4d0 0x7f0ef806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:42.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.908+0000 7f0f06ffd640 1 -- 172.21.15.1:0/1939770781 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0f100b0830 con 0x7f0f1806e6d0 2024-03-06T07:20:42.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.916+0000 7f0f15d9b640 1 --2- 172.21.15.1:0/1939770781 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ef806b4d0 0x7f0ef806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:42.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.917+0000 7f0f15d9b640 1 --2- 172.21.15.1:0/1939770781 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ef806b4d0 0x7f0ef806d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0f181527c0 tx=0x7f0f08006bd0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:42.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:42.917+0000 7f0f06ffd640 1 -- 172.21.15.1:0/1939770781 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f10050030 con 0x7f0f1806e6d0 2024-03-06T07:20:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.031+0000 7fdfeb822640 1 -- 172.21.15.1:0/1660361745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fdfc4005580 con 0x7fdfe414e8b0 2024-03-06T07:20:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.047+0000 7f88cee0e640 1 -- 172.21.15.1:0/3628778103 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f88c80c5310 con 0x7f88c8150760 2024-03-06T07:20:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.047+0000 7fdfd9ffb640 1 -- 172.21.15.1:0/1660361745 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdfdc05c0a0 con 0x7fdfe414e8b0 2024-03-06T07:20:43.050 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-06T07:20:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.048+0000 7f88c57fa640 1 -- 172.21.15.1:0/3628778103 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f88b802f050 con 0x7f88c8150760 2024-03-06T07:20:43.051 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-06T07:20:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.050+0000 7fdfeb822640 1 -- 172.21.15.1:0/1660361745 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfc806b5d0 msgr2=0x7fdfc806da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.050+0000 7fdfeb822640 1 --2- 172.21.15.1:0/1660361745 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfc806b5d0 0x7fdfc806da90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdfe000b7f0 tx=0x7fdfe0008040 comp rx=0 tx=0).stop 2024-03-06T07:20:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.050+0000 7fdfeb822640 1 -- 172.21.15.1:0/1660361745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe414e8b0 msgr2=0x7fdfe4144d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.050+0000 7fdfeb822640 1 --2- 172.21.15.1:0/1660361745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe414e8b0 0x7fdfe4144d50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00c7a0 tx=0x7fdfdc00cc60 comp rx=0 tx=0).stop 2024-03-06T07:20:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.050+0000 7fdfeb822640 1 -- 172.21.15.1:0/1660361745 shutdown_connections 2024-03-06T07:20:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.050+0000 7fdfeb822640 1 --2- 172.21.15.1:0/1660361745 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfc806b5d0 0x7fdfc806da90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.050+0000 7fdfeb822640 1 --2- 172.21.15.1:0/1660361745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe414e8b0 0x7fdfe4144d50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.050+0000 7fdfeb822640 1 --2- 172.21.15.1:0/1660361745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe40c1490 0x7fdfe4149240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.050+0000 7fdfeb822640 1 --2- 172.21.15.1:0/1660361745 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfe406e1d0 0x7fdfe40ac200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.050+0000 7fdfeb822640 1 -- 172.21.15.1:0/1660361745 >> 172.21.15.1:0/1660361745 conn(0x7fdfe40c6860 msgr2=0x7fdfe4072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:43.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.052+0000 7fdfeb822640 1 -- 172.21.15.1:0/1660361745 shutdown_connections 2024-03-06T07:20:43.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.053+0000 7fdfeb822640 1 -- 172.21.15.1:0/1660361745 wait complete. 2024-03-06T07:20:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.053+0000 7f88cee0e640 1 -- 172.21.15.1:0/3628778103 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88b406b520 msgr2=0x7f88b406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.053+0000 7f88cee0e640 1 --2- 172.21.15.1:0/3628778103 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88b406b520 0x7f88b406d9e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009c90 tx=0x7f88c000b040 comp rx=0 tx=0).stop 2024-03-06T07:20:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.053+0000 7f88cee0e640 1 -- 172.21.15.1:0/3628778103 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c8150760 msgr2=0x7f88c8155b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.053+0000 7f88cee0e640 1 --2- 172.21.15.1:0/3628778103 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c8150760 0x7f88c8155b80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f88b8034040 tx=0x7f88b80046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.053+0000 7f88cee0e640 1 -- 172.21.15.1:0/3628778103 shutdown_connections 2024-03-06T07:20:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.053+0000 7f88cee0e640 1 --2- 172.21.15.1:0/3628778103 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88b406b520 0x7f88b406d9e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.053+0000 7f88cee0e640 1 --2- 172.21.15.1:0/3628778103 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c8150760 0x7f88c8155b80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.053+0000 7f88cee0e640 1 --2- 172.21.15.1:0/3628778103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88c80c57e0 0x7f88c8155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.053+0000 7f88cee0e640 1 --2- 172.21.15.1:0/3628778103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88c80c4ad0 0x7f88c8154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.053+0000 7f88cee0e640 1 -- 172.21.15.1:0/3628778103 >> 172.21.15.1:0/3628778103 conn(0x7f88c80c8810 msgr2=0x7f88c806f3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.053+0000 7f88cee0e640 1 -- 172.21.15.1:0/3628778103 shutdown_connections 2024-03-06T07:20:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.053+0000 7f88cee0e640 1 -- 172.21.15.1:0/3628778103 wait complete. 2024-03-06T07:20:43.071 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.0 2024-03-06T07:20:43.071 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:43.083 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.7 2024-03-06T07:20:43.083 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:43.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.111+0000 7f51423bf640 1 -- 172.21.15.1:0/1723167330 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f511c005580 con 0x7f513c151070 2024-03-06T07:20:43.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.111+0000 7f5138ff9640 1 -- 172.21.15.1:0/1723167330 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f513400ae20 con 0x7f513c151070 2024-03-06T07:20:43.113 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-06T07:20:43.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.114+0000 7f51423bf640 1 -- 172.21.15.1:0/1723167330 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f512806b520 msgr2=0x7f512806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.114+0000 7f51423bf640 1 --2- 172.21.15.1:0/1723167330 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f512806b520 0x7f512806d9e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f513c155b30 tx=0x7f5130006c20 comp rx=0 tx=0).stop 2024-03-06T07:20:43.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.114+0000 7f51423bf640 1 -- 172.21.15.1:0/1723167330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f513c151070 msgr2=0x7f513c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.114+0000 7f51423bf640 1 --2- 172.21.15.1:0/1723167330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f513c151070 0x7f513c071ce0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f513400cb10 tx=0x7f513400a060 comp rx=0 tx=0).stop 2024-03-06T07:20:43.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.114+0000 7f51423bf640 1 -- 172.21.15.1:0/1723167330 shutdown_connections 2024-03-06T07:20:43.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.114+0000 7f51423bf640 1 --2- 172.21.15.1:0/1723167330 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f512806b520 0x7f512806d9e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.114+0000 7f51423bf640 1 --2- 172.21.15.1:0/1723167330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f513c151070 0x7f513c071ce0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.114+0000 7f51423bf640 1 --2- 172.21.15.1:0/1723167330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c15cce0 0x7f513c1435c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.114+0000 7f51423bf640 1 --2- 172.21.15.1:0/1723167330 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f513c0c3720 0x7f513c0aa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.114+0000 7f51423bf640 1 -- 172.21.15.1:0/1723167330 >> 172.21.15.1:0/1723167330 conn(0x7f513c0c6860 msgr2=0x7f513c163250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:43.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.114+0000 7f51423bf640 1 -- 172.21.15.1:0/1723167330 shutdown_connections 2024-03-06T07:20:43.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.114+0000 7f51423bf640 1 -- 172.21.15.1:0/1723167330 wait complete. 2024-03-06T07:20:43.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.121+0000 7f20c3148640 1 -- 172.21.15.1:0/1055487035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f209c005580 con 0x7f20bc165c30 2024-03-06T07:20:43.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.121+0000 7f20b97fa640 1 -- 172.21.15.1:0/1055487035 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f20b4058820 con 0x7f20bc165c30 2024-03-06T07:20:43.123 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-06T07:20:43.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.124+0000 7f20c3148640 1 -- 172.21.15.1:0/1055487035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20a806b4d0 msgr2=0x7f20a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.124+0000 7f20c3148640 1 --2- 172.21.15.1:0/1055487035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20a806b4d0 0x7f20a806d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f20b0009220 tx=0x7f20b0008050 comp rx=0 tx=0).stop 2024-03-06T07:20:43.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.124+0000 7f20c3148640 1 -- 172.21.15.1:0/1055487035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20bc165c30 msgr2=0x7f20bc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.124+0000 7f20c3148640 1 --2- 172.21.15.1:0/1055487035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20bc165c30 0x7f20bc071ce0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f20b400c7a0 tx=0x7f20b400cc60 comp rx=0 tx=0).stop 2024-03-06T07:20:43.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.124+0000 7f20c3148640 1 -- 172.21.15.1:0/1055487035 shutdown_connections 2024-03-06T07:20:43.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.124+0000 7f20c3148640 1 --2- 172.21.15.1:0/1055487035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20a806b4d0 0x7f20a806d990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.124+0000 7f20c3148640 1 --2- 172.21.15.1:0/1055487035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20bc165c30 0x7f20bc071ce0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.124+0000 7f20c3148640 1 --2- 172.21.15.1:0/1055487035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20bc08a970 0x7f20bc164ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.124+0000 7f20c3148640 1 --2- 172.21.15.1:0/1055487035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20bc06d550 0x7f20bc15c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.124+0000 7f20c3148640 1 -- 172.21.15.1:0/1055487035 >> 172.21.15.1:0/1055487035 conn(0x7f20bc0c6860 msgr2=0x7f20bc1602a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:43.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.125+0000 7f20c3148640 1 -- 172.21.15.1:0/1055487035 shutdown_connections 2024-03-06T07:20:43.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.125+0000 7f20c3148640 1 -- 172.21.15.1:0/1055487035 wait complete. 2024-03-06T07:20:43.136 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2024-03-06T07:20:43.136 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:43.138 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2024-03-06T07:20:43.138 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:43.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.141+0000 7f51f6222640 1 -- 172.21.15.1:0/542918781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f51f00686e0 con 0x7f51f00ac0d0 2024-03-06T07:20:43.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.142+0000 7f51e5ffb640 1 -- 172.21.15.1:0/542918781 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f51ec059780 con 0x7f51f00ac0d0 2024-03-06T07:20:43.143 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-06T07:20:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.145+0000 7f51f6222640 1 -- 172.21.15.1:0/542918781 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51d406b520 msgr2=0x7f51d406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.145+0000 7f51f6222640 1 --2- 172.21.15.1:0/542918781 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51d406b520 0x7f51d406d9e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f51e0009930 tx=0x7f51e0009190 comp rx=0 tx=0).stop 2024-03-06T07:20:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.145+0000 7f51f6222640 1 -- 172.21.15.1:0/542918781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51f00ac0d0 msgr2=0x7f51f015bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.145+0000 7f51f6222640 1 --2- 172.21.15.1:0/542918781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51f00ac0d0 0x7f51f015bc80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00c450 tx=0x7f51ec00c910 comp rx=0 tx=0).stop 2024-03-06T07:20:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.145+0000 7f51f6222640 1 -- 172.21.15.1:0/542918781 shutdown_connections 2024-03-06T07:20:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.145+0000 7f51f6222640 1 --2- 172.21.15.1:0/542918781 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51d406b520 0x7f51d406d9e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.145+0000 7f51f6222640 1 --2- 172.21.15.1:0/542918781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51f00ac7f0 0x7f51f015c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.145+0000 7f51f6222640 1 --2- 172.21.15.1:0/542918781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51f00ac0d0 0x7f51f015bc80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.145+0000 7f51f6222640 1 --2- 172.21.15.1:0/542918781 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51f00abcf0 0x7f51f00c5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.145+0000 7f51f6222640 1 -- 172.21.15.1:0/542918781 >> 172.21.15.1:0/542918781 conn(0x7f51f0071b80 msgr2=0x7f51f00c1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:43.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.145+0000 7f51f6222640 1 -- 172.21.15.1:0/542918781 shutdown_connections 2024-03-06T07:20:43.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.145+0000 7f51f6222640 1 -- 172.21.15.1:0/542918781 wait complete. 2024-03-06T07:20:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.149+0000 7f5bbf59e640 1 -- 172.21.15.1:0/2538201451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f5bb8006ea0 con 0x7f5bb80d84a0 2024-03-06T07:20:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.150+0000 7f5baeffd640 1 -- 172.21.15.1:0/2538201451 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5bb007e3c0 con 0x7f5bb80d84a0 2024-03-06T07:20:43.151 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-06T07:20:43.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.152+0000 7f5bbf59e640 1 -- 172.21.15.1:0/2538201451 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ba006b500 msgr2=0x7f5ba006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.152+0000 7f5bbf59e640 1 --2- 172.21.15.1:0/2538201451 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ba006b500 0x7f5ba006d9c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0050c30 tx=0x7f5bc0068fe0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.152+0000 7f5bbf59e640 1 -- 172.21.15.1:0/2538201451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bb80d84a0 msgr2=0x7f5bb80cf1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.152+0000 7f5bbf59e640 1 --2- 172.21.15.1:0/2538201451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bb80d84a0 0x7f5bb80cf1c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5bb000d450 tx=0x7f5bb0004fd0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.152+0000 7f5bbf59e640 1 -- 172.21.15.1:0/2538201451 shutdown_connections 2024-03-06T07:20:43.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.152+0000 7f5bbf59e640 1 --2- 172.21.15.1:0/2538201451 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ba006b500 0x7f5ba006d9c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.152+0000 7f5bbf59e640 1 --2- 172.21.15.1:0/2538201451 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bb804a070 0x7f5bb818af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.152+0000 7f5bbf59e640 1 --2- 172.21.15.1:0/2538201451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bb80d84a0 0x7f5bb80cf1c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.152+0000 7f5bbf59e640 1 --2- 172.21.15.1:0/2538201451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ba8006870 0x7f5bb80da400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.152+0000 7f5bbf59e640 1 -- 172.21.15.1:0/2538201451 >> 172.21.15.1:0/2538201451 conn(0x7f5bb80060b0 msgr2=0x7f5bb80d1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:43.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.152+0000 7f5bbf59e640 1 -- 172.21.15.1:0/2538201451 shutdown_connections 2024-03-06T07:20:43.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.152+0000 7f5bbf59e640 1 -- 172.21.15.1:0/2538201451 wait complete. 2024-03-06T07:20:43.163 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2024-03-06T07:20:43.163 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:43.167 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2024-03-06T07:20:43.167 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.185+0000 7f1854f32640 1 -- 172.21.15.1:0/675288495 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f18500c2030 con 0x7f18500c4760 2024-03-06T07:20:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.186+0000 7f183effd640 1 -- 172.21.15.1:0/675288495 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1848057fe0 con 0x7f18500c4760 2024-03-06T07:20:43.188 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-06T07:20:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.187+0000 7f1854f32640 1 -- 172.21.15.1:0/675288495 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f183006b5d0 msgr2=0x7f183006da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.187+0000 7f1854f32640 1 --2- 172.21.15.1:0/675288495 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f183006b5d0 0x7f183006da90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f18500bc750 tx=0x7f1840008030 comp rx=0 tx=0).stop 2024-03-06T07:20:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.187+0000 7f1854f32640 1 -- 172.21.15.1:0/675288495 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18500c4760 msgr2=0x7f18501f7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.187+0000 7f1854f32640 1 --2- 172.21.15.1:0/675288495 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18500c4760 0x7f18501f7b10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f184800c730 tx=0x7f184800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.188+0000 7f1854f32640 1 -- 172.21.15.1:0/675288495 shutdown_connections 2024-03-06T07:20:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.188+0000 7f1854f32640 1 --2- 172.21.15.1:0/675288495 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f183006b5d0 0x7f183006da90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.188+0000 7f1854f32640 1 --2- 172.21.15.1:0/675288495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f185015cc10 0x7f18500abd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.188+0000 7f1854f32640 1 --2- 172.21.15.1:0/675288495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1850150140 0x7f18500ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.188+0000 7f1854f32640 1 --2- 172.21.15.1:0/675288495 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18500c4760 0x7f18501f7b10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.188+0000 7f1854f32640 1 -- 172.21.15.1:0/675288495 >> 172.21.15.1:0/675288495 conn(0x7f18500c6860 msgr2=0x7f18500c6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.188+0000 7f1854f32640 1 -- 172.21.15.1:0/675288495 shutdown_connections 2024-03-06T07:20:43.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.188+0000 7f1854f32640 1 -- 172.21.15.1:0/675288495 wait complete. 2024-03-06T07:20:43.200 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.2 2024-03-06T07:20:43.201 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.259+0000 7f0f1c9c9640 1 -- 172.21.15.1:0/1939770781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0f180c6d70 con 0x7f0f1806e6d0 2024-03-06T07:20:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.259+0000 7f0f06ffd640 1 -- 172.21.15.1:0/1939770781 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0f100aee20 con 0x7f0f1806e6d0 2024-03-06T07:20:43.261 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-06T07:20:43.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.261+0000 7f0f1c9c9640 1 -- 172.21.15.1:0/1939770781 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ef806b4d0 msgr2=0x7f0ef806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.261+0000 7f0f1c9c9640 1 --2- 172.21.15.1:0/1939770781 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ef806b4d0 0x7f0ef806d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0f181527c0 tx=0x7f0f08006bd0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.261+0000 7f0f1c9c9640 1 -- 172.21.15.1:0/1939770781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1806e6d0 msgr2=0x7f0f18150870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.261+0000 7f0f1c9c9640 1 --2- 172.21.15.1:0/1939770781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1806e6d0 0x7f0f18150870 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0f10036040 tx=0x7f0f10004a40 comp rx=0 tx=0).stop 2024-03-06T07:20:43.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.261+0000 7f0f1c9c9640 1 -- 172.21.15.1:0/1939770781 shutdown_connections 2024-03-06T07:20:43.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.261+0000 7f0f1c9c9640 1 --2- 172.21.15.1:0/1939770781 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ef806b4d0 0x7f0ef806d990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.261+0000 7f0f1c9c9640 1 --2- 172.21.15.1:0/1939770781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f1814f530 0x7f0f18149da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.261+0000 7f0f1c9c9640 1 --2- 172.21.15.1:0/1939770781 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f18072c00 0x7f0f18149860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.261+0000 7f0f1c9c9640 1 --2- 172.21.15.1:0/1939770781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1806e6d0 0x7f0f18150870 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.261+0000 7f0f1c9c9640 1 -- 172.21.15.1:0/1939770781 >> 172.21.15.1:0/1939770781 conn(0x7f0f1806c100 msgr2=0x7f0f18074450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:43.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.261+0000 7f0f1c9c9640 1 -- 172.21.15.1:0/1939770781 shutdown_connections 2024-03-06T07:20:43.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.261+0000 7f0f1c9c9640 1 -- 172.21.15.1:0/1939770781 wait complete. 2024-03-06T07:20:43.274 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2024-03-06T07:20:43.274 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:43.274 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:20:43.274 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:20:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.381+0000 7f138d28d640 1 Processor -- start 2024-03-06T07:20:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.381+0000 7f138d28d640 1 -- start start 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.382+0000 7f138d28d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f13880c3eb0 0x7f13880c7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.382+0000 7f138d28d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13880c71c0 con 0x7f138806d060 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.382+0000 7f138d28d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13880c4e50 con 0x7f13880c7940 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.382+0000 7f138d28d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13880c59b0 con 0x7f13880c3eb0 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.382+0000 7f1386ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f13880c3eb0 0x7f13880c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.382+0000 7f13867fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f138806d060 0x7f13881558b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57534/0 (socket says 172.21.15.1:57534) 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.382+0000 7f13867fc640 1 -- 172.21.15.1:0/852602262 learned_addr learned my addr 172.21.15.1:0/852602262 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.382+0000 7f1386ffd640 1 -- 172.21.15.1:0/852602262 >> v1:172.21.15.98:6789/0 conn(0x7f13880c7940 legacy=0x7f138815fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.382+0000 7f1386ffd640 1 -- 172.21.15.1:0/852602262 >> v1:172.21.15.1:6789/0 conn(0x7f138806d060 legacy=0x7f13881558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.382+0000 7f1386ffd640 1 -- 172.21.15.1:0/852602262 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13880c6ee0 con 0x7f13880c3eb0 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.382+0000 7f1385ffb640 1 -- 172.21.15.1:0/852602262 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23960406 0 0) 0x7f13880c71c0 con 0x7f138806d060 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.383+0000 7f1386ffd640 1 --2- 172.21.15.1:0/852602262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13880c3eb0 0x7f13880c7400 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f13780097a0 tx=0x7f137802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f72763339929d70a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.383+0000 7f1385ffb640 1 -- 172.21.15.1:0/852602262 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13780315a0 con 0x7f13880c3eb0 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.383+0000 7f1385ffb640 1 -- 172.21.15.1:0/852602262 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1378037070 con 0x7f13880c3eb0 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.383+0000 7f1385ffb640 1 -- 172.21.15.1:0/852602262 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1378031960 con 0x7f13880c3eb0 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.384+0000 7f138d28d640 1 -- 172.21.15.1:0/852602262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13880c3eb0 msgr2=0x7f13880c7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.384+0000 7f138d28d640 1 --2- 172.21.15.1:0/852602262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13880c3eb0 0x7f13880c7400 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f13780097a0 tx=0x7f137802b840 comp rx=0 tx=0).stop 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.384+0000 7f138d28d640 1 -- 172.21.15.1:0/852602262 shutdown_connections 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.384+0000 7f138d28d640 1 --2- 172.21.15.1:0/852602262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13880c3eb0 0x7f13880c7400 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.384+0000 7f138d28d640 1 -- 172.21.15.1:0/852602262 >> 172.21.15.1:0/852602262 conn(0x7f1388066d00 msgr2=0x7f1388067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.384+0000 7f138d28d640 1 -- 172.21.15.1:0/852602262 shutdown_connections 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.384+0000 7f138d28d640 1 -- 172.21.15.1:0/852602262 wait complete. 2024-03-06T07:20:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.385+0000 7f138d28d640 1 Processor -- start 2024-03-06T07:20:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.385+0000 7f138d28d640 1 -- start start 2024-03-06T07:20:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f138d28d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f138806d060 0x7f13880ae660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f138d28d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13880c3eb0 0x7f13880aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f138d28d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13880c7940 0x7f138814d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f138d28d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1388155420 con 0x7f138806d060 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f138d28d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13880c4c00 con 0x7f13880c7940 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f138d28d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1388155fc0 con 0x7f13880c3eb0 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f1386ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f138806d060 0x7f13880ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f1386ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f138806d060 0x7f13880ae660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40774/0 (socket says 172.21.15.1:40774) 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f1386ffd640 1 -- 172.21.15.1:0/3325688317 learned_addr learned my addr 172.21.15.1:0/3325688317 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f13867fc640 1 --2- 172.21.15.1:0/3325688317 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13880c3eb0 0x7f13880aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f13877fe640 1 --2- 172.21.15.1:0/3325688317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13880c7940 0x7f138814d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f1386ffd640 1 -- 172.21.15.1:0/3325688317 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13880c3eb0 msgr2=0x7f13880aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f1386ffd640 1 --2- 172.21.15.1:0/3325688317 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13880c3eb0 0x7f13880aed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f1386ffd640 1 -- 172.21.15.1:0/3325688317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13880c7940 msgr2=0x7f138814d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f1386ffd640 1 --2- 172.21.15.1:0/3325688317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13880c7940 0x7f138814d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f1386ffd640 1 -- 172.21.15.1:0/3325688317 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13881eb500 con 0x7f138806d060 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f13877fe640 1 --2- 172.21.15.1:0/3325688317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13880c7940 0x7f138814d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.386+0000 7f1386ffd640 1 --2- 172.21.15.1:0/3325688317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f138806d060 0x7f13880ae660 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f137802be50 tx=0x7f13780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.387+0000 7f137f7fe640 1 -- 172.21.15.1:0/3325688317 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1378032840 con 0x7f138806d060 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.387+0000 7f137f7fe640 1 -- 172.21.15.1:0/3325688317 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13780329e0 con 0x7f138806d060 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.387+0000 7f137f7fe640 1 -- 172.21.15.1:0/3325688317 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1378032dd0 con 0x7f138806d060 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.387+0000 7f138d28d640 1 -- 172.21.15.1:0/3325688317 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13881eba80 con 0x7f138806d060 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.387+0000 7f138d28d640 1 -- 172.21.15.1:0/3325688317 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13880c8920 con 0x7f138806d060 2024-03-06T07:20:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.388+0000 7f138d28d640 1 -- 172.21.15.1:0/3325688317 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1388067fd0 con 0x7f138806d060 2024-03-06T07:20:43.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.388+0000 7f137f7fe640 1 -- 172.21.15.1:0/3325688317 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f137804c020 con 0x7f138806d060 2024-03-06T07:20:43.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.388+0000 7f137f7fe640 1 --2- 172.21.15.1:0/3325688317 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f136806b4d0 0x7f136806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:43.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.389+0000 7f137f7fe640 1 -- 172.21.15.1:0/3325688317 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f137802f020 con 0x7f138806d060 2024-03-06T07:20:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.389+0000 7f13867fc640 1 --2- 172.21.15.1:0/3325688317 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f136806b4d0 0x7f136806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.389+0000 7f13867fc640 1 --2- 172.21.15.1:0/3325688317 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f136806b4d0 0x7f136806d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1370007f70 tx=0x7f1370007f00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.393+0000 7f137f7fe640 1 -- 172.21.15.1:0/3325688317 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1378040070 con 0x7f138806d060 2024-03-06T07:20:43.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.600+0000 7f138d28d640 1 -- 172.21.15.1:0/3325688317 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f13880c5b30 con 0x7f136806b4d0 2024-03-06T07:20:43.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.608+0000 7f137f7fe640 1 -- 172.21.15.1:0/3325688317 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332611 (secure 0 0 0) 0x7f13880c5b30 con 0x7f136806b4d0 2024-03-06T07:20:43.614 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:43.616 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:20:43.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.611+0000 7f138d28d640 1 -- 172.21.15.1:0/3325688317 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f136806b4d0 msgr2=0x7f136806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.611+0000 7f138d28d640 1 --2- 172.21.15.1:0/3325688317 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f136806b4d0 0x7f136806d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1370007f70 tx=0x7f1370007f00 comp rx=0 tx=0).stop 2024-03-06T07:20:43.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.611+0000 7f138d28d640 1 -- 172.21.15.1:0/3325688317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f138806d060 msgr2=0x7f13880ae660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.611+0000 7f138d28d640 1 --2- 172.21.15.1:0/3325688317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f138806d060 0x7f13880ae660 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f137802be50 tx=0x7f13780046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.611+0000 7f138d28d640 1 -- 172.21.15.1:0/3325688317 shutdown_connections 2024-03-06T07:20:43.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.611+0000 7f138d28d640 1 --2- 172.21.15.1:0/3325688317 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f136806b4d0 0x7f136806d990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.611+0000 7f138d28d640 1 --2- 172.21.15.1:0/3325688317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13880c7940 0x7f138814d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.611+0000 7f138d28d640 1 --2- 172.21.15.1:0/3325688317 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13880c3eb0 0x7f13880aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.611+0000 7f138d28d640 1 --2- 172.21.15.1:0/3325688317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f138806d060 0x7f13880ae660 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.611+0000 7f138d28d640 1 -- 172.21.15.1:0/3325688317 >> 172.21.15.1:0/3325688317 conn(0x7f1388066d00 msgr2=0x7f13880670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:43.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.612+0000 7f138d28d640 1 -- 172.21.15.1:0/3325688317 shutdown_connections 2024-03-06T07:20:43.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.612+0000 7f138d28d640 1 -- 172.21.15.1:0/3325688317 wait complete. 2024-03-06T07:20:43.628 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-03-06T07:20:42.691719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":230920,"kb_used_data":2272,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754743800,"statfs":{"total":773094113280,"available":772857651200,"internally_reserved":0,"allocated":2326528,"data_stored":2049160,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004241"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701618+0000","last_change":"2024-03-06T07:20:20.860733+0000","last_active":"2024-03-06T07:20:28.701618+0000","last_peered":"2024-03-06T07:20:28.701618+0000","last_clean":"2024-03-06T07:20:28.701618+0000","last_became_active":"2024-03-06T07:20:20.860412+0000","last_became_peered":"2024-03-06T07:20:20.860412+0000","last_unstale":"2024-03-06T07:20:28.701618+0000","last_undegraded":"2024-03-06T07:20:28.701618+0000","last_fullsized":"2024-03-06T07:20:28.701618+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:44.080359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880598+0000","last_change":"2024-03-06T07:20:21.869098+0000","last_active":"2024-03-06T07:20:27.880598+0000","last_peered":"2024-03-06T07:20:27.880598+0000","last_clean":"2024-03-06T07:20:27.880598+0000","last_became_active":"2024-03-06T07:20:21.868808+0000","last_became_peered":"2024-03-06T07:20:21.868808+0000","last_unstale":"2024-03-06T07:20:27.880598+0000","last_undegraded":"2024-03-06T07:20:27.880598+0000","last_fullsized":"2024-03-06T07:20:27.880598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.423089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879651+0000","last_change":"2024-03-06T07:20:26.504312+0000","last_active":"2024-03-06T07:20:27.879651+0000","last_peered":"2024-03-06T07:20:27.879651+0000","last_clean":"2024-03-06T07:20:27.879651+0000","last_became_active":"2024-03-06T07:20:26.503884+0000","last_became_peered":"2024-03-06T07:20:26.503884+0000","last_unstale":"2024-03-06T07:20:27.879651+0000","last_undegraded":"2024-03-06T07:20:27.879651+0000","last_fullsized":"2024-03-06T07:20:27.879651+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:57:56.136234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880779+0000","last_change":"2024-03-06T07:20:20.854641+0000","last_active":"2024-03-06T07:20:27.880779+0000","last_peered":"2024-03-06T07:20:27.880779+0000","last_clean":"2024-03-06T07:20:27.880779+0000","last_became_active":"2024-03-06T07:20:20.854361+0000","last_became_peered":"2024-03-06T07:20:20.854361+0000","last_unstale":"2024-03-06T07:20:27.880779+0000","last_undegraded":"2024-03-06T07:20:27.880779+0000","last_fullsized":"2024-03-06T07:20:27.880779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:57.497682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701411+0000","last_change":"2024-03-06T07:20:21.857450+0000","last_active":"2024-03-06T07:20:28.701411+0000","last_peered":"2024-03-06T07:20:28.701411+0000","last_clean":"2024-03-06T07:20:28.701411+0000","last_became_active":"2024-03-06T07:20:21.855756+0000","last_became_peered":"2024-03-06T07:20:21.855756+0000","last_unstale":"2024-03-06T07:20:28.701411+0000","last_undegraded":"2024-03-06T07:20:28.701411+0000","last_fullsized":"2024-03-06T07:20:28.701411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:17.903391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879763+0000","last_change":"2024-03-06T07:20:25.889839+0000","last_active":"2024-03-06T07:20:27.879763+0000","last_peered":"2024-03-06T07:20:27.879763+0000","last_clean":"2024-03-06T07:20:27.879763+0000","last_became_active":"2024-03-06T07:20:25.889266+0000","last_became_peered":"2024-03-06T07:20:25.889266+0000","last_unstale":"2024-03-06T07:20:27.879763+0000","last_undegraded":"2024-03-06T07:20:27.879763+0000","last_fullsized":"2024-03-06T07:20:27.879763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:51.236302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201405+0000","last_change":"2024-03-06T07:20:20.859562+0000","last_active":"2024-03-06T07:20:28.201405+0000","last_peered":"2024-03-06T07:20:28.201405+0000","last_clean":"2024-03-06T07:20:28.201405+0000","last_became_active":"2024-03-06T07:20:20.859282+0000","last_became_peered":"2024-03-06T07:20:20.859282+0000","last_unstale":"2024-03-06T07:20:28.201405+0000","last_undegraded":"2024-03-06T07:20:28.201405+0000","last_fullsized":"2024-03-06T07:20:28.201405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:52.187127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701578+0000","last_change":"2024-03-06T07:20:21.863171+0000","last_active":"2024-03-06T07:20:28.701578+0000","last_peered":"2024-03-06T07:20:28.701578+0000","last_clean":"2024-03-06T07:20:28.701578+0000","last_became_active":"2024-03-06T07:20:21.862904+0000","last_became_peered":"2024-03-06T07:20:21.862904+0000","last_unstale":"2024-03-06T07:20:28.701578+0000","last_undegraded":"2024-03-06T07:20:28.701578+0000","last_fullsized":"2024-03-06T07:20:28.701578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:07.625355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880420+0000","last_change":"2024-03-06T07:20:25.888576+0000","last_active":"2024-03-06T07:20:27.880420+0000","last_peered":"2024-03-06T07:20:27.880420+0000","last_clean":"2024-03-06T07:20:27.880420+0000","last_became_active":"2024-03-06T07:20:25.886558+0000","last_became_peered":"2024-03-06T07:20:25.886558+0000","last_unstale":"2024-03-06T07:20:27.880420+0000","last_undegraded":"2024-03-06T07:20:27.880420+0000","last_fullsized":"2024-03-06T07:20:27.880420+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:27.800537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201157+0000","last_change":"2024-03-06T07:20:20.861724+0000","last_active":"2024-03-06T07:20:28.201157+0000","last_peered":"2024-03-06T07:20:28.201157+0000","last_clean":"2024-03-06T07:20:28.201157+0000","last_became_active":"2024-03-06T07:20:20.861432+0000","last_became_peered":"2024-03-06T07:20:20.861432+0000","last_unstale":"2024-03-06T07:20:28.201157+0000","last_undegraded":"2024-03-06T07:20:28.201157+0000","last_fullsized":"2024-03-06T07:20:28.201157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:43.621233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702026+0000","last_change":"2024-03-06T07:20:21.862894+0000","last_active":"2024-03-06T07:20:28.702026+0000","last_peered":"2024-03-06T07:20:28.702026+0000","last_clean":"2024-03-06T07:20:28.702026+0000","last_became_active":"2024-03-06T07:20:21.862624+0000","last_became_peered":"2024-03-06T07:20:21.862624+0000","last_unstale":"2024-03-06T07:20:28.702026+0000","last_undegraded":"2024-03-06T07:20:28.702026+0000","last_fullsized":"2024-03-06T07:20:28.702026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:53.701404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879914+0000","last_change":"2024-03-06T07:20:25.891479+0000","last_active":"2024-03-06T07:20:27.879914+0000","last_peered":"2024-03-06T07:20:27.879914+0000","last_clean":"2024-03-06T07:20:27.879914+0000","last_became_active":"2024-03-06T07:20:25.891124+0000","last_became_peered":"2024-03-06T07:20:25.891124+0000","last_unstale":"2024-03-06T07:20:27.879914+0000","last_undegraded":"2024-03-06T07:20:27.879914+0000","last_fullsized":"2024-03-06T07:20:27.879914+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:31.908538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701544+0000","last_change":"2024-03-06T07:20:21.861447+0000","last_active":"2024-03-06T07:20:28.701544+0000","last_peered":"2024-03-06T07:20:28.701544+0000","last_clean":"2024-03-06T07:20:28.701544+0000","last_became_active":"2024-03-06T07:20:21.861169+0000","last_became_peered":"2024-03-06T07:20:21.861169+0000","last_unstale":"2024-03-06T07:20:28.701544+0000","last_undegraded":"2024-03-06T07:20:28.701544+0000","last_fullsized":"2024-03-06T07:20:28.701544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:28.363605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201185+0000","last_change":"2024-03-06T07:20:20.858871+0000","last_active":"2024-03-06T07:20:28.201185+0000","last_peered":"2024-03-06T07:20:28.201185+0000","last_clean":"2024-03-06T07:20:28.201185+0000","last_became_active":"2024-03-06T07:20:20.858481+0000","last_became_peered":"2024-03-06T07:20:20.858481+0000","last_unstale":"2024-03-06T07:20:28.201185+0000","last_undegraded":"2024-03-06T07:20:28.201185+0000","last_fullsized":"2024-03-06T07:20:28.201185+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:58.697205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880550+0000","last_change":"2024-03-06T07:20:25.890028+0000","last_active":"2024-03-06T07:20:27.880550+0000","last_peered":"2024-03-06T07:20:27.880550+0000","last_clean":"2024-03-06T07:20:27.880550+0000","last_became_active":"2024-03-06T07:20:25.889634+0000","last_became_peered":"2024-03-06T07:20:25.889634+0000","last_unstale":"2024-03-06T07:20:27.880550+0000","last_undegraded":"2024-03-06T07:20:27.880550+0000","last_fullsized":"2024-03-06T07:20:27.880550+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:34.847448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880189+0000","last_change":"2024-03-06T07:20:21.863836+0000","last_active":"2024-03-06T07:20:27.880189+0000","last_peered":"2024-03-06T07:20:27.880189+0000","last_clean":"2024-03-06T07:20:27.880189+0000","last_became_active":"2024-03-06T07:20:21.863451+0000","last_became_peered":"2024-03-06T07:20:21.863451+0000","last_unstale":"2024-03-06T07:20:27.880189+0000","last_undegraded":"2024-03-06T07:20:27.880189+0000","last_fullsized":"2024-03-06T07:20:27.880189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:07.625355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365689+0000","last_change":"2024-03-06T07:20:20.847182+0000","last_active":"2024-03-06T07:20:28.365689+0000","last_peered":"2024-03-06T07:20:28.365689+0000","last_clean":"2024-03-06T07:20:28.365689+0000","last_became_active":"2024-03-06T07:20:20.846875+0000","last_became_peered":"2024-03-06T07:20:20.846875+0000","last_unstale":"2024-03-06T07:20:28.365689+0000","last_undegraded":"2024-03-06T07:20:28.365689+0000","last_fullsized":"2024-03-06T07:20:28.365689+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:47.332802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880571+0000","last_change":"2024-03-06T07:20:25.890712+0000","last_active":"2024-03-06T07:20:27.880571+0000","last_peered":"2024-03-06T07:20:27.880571+0000","last_clean":"2024-03-06T07:20:27.880571+0000","last_became_active":"2024-03-06T07:20:25.889840+0000","last_became_peered":"2024-03-06T07:20:25.889840+0000","last_unstale":"2024-03-06T07:20:27.880571+0000","last_undegraded":"2024-03-06T07:20:27.880571+0000","last_fullsized":"2024-03-06T07:20:27.880571+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:46.450423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879843+0000","last_change":"2024-03-06T07:20:21.865384+0000","last_active":"2024-03-06T07:20:27.879843+0000","last_peered":"2024-03-06T07:20:27.879843+0000","last_clean":"2024-03-06T07:20:27.879843+0000","last_became_active":"2024-03-06T07:20:21.865118+0000","last_became_peered":"2024-03-06T07:20:21.865118+0000","last_unstale":"2024-03-06T07:20:27.879843+0000","last_undegraded":"2024-03-06T07:20:27.879843+0000","last_fullsized":"2024-03-06T07:20:27.879843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:36.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365440+0000","last_change":"2024-03-06T07:20:20.856580+0000","last_active":"2024-03-06T07:20:28.365440+0000","last_peered":"2024-03-06T07:20:28.365440+0000","last_clean":"2024-03-06T07:20:28.365440+0000","last_became_active":"2024-03-06T07:20:20.856117+0000","last_became_peered":"2024-03-06T07:20:20.856117+0000","last_unstale":"2024-03-06T07:20:28.365440+0000","last_undegraded":"2024-03-06T07:20:28.365440+0000","last_fullsized":"2024-03-06T07:20:28.365440+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:43.621233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366340+0000","last_change":"2024-03-06T07:20:26.502055+0000","last_active":"2024-03-06T07:20:28.366340+0000","last_peered":"2024-03-06T07:20:28.366340+0000","last_clean":"2024-03-06T07:20:28.366340+0000","last_became_active":"2024-03-06T07:20:26.501689+0000","last_became_peered":"2024-03-06T07:20:26.501689+0000","last_unstale":"2024-03-06T07:20:28.366340+0000","last_undegraded":"2024-03-06T07:20:28.366340+0000","last_fullsized":"2024-03-06T07:20:28.366340+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:34.847448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880426+0000","last_change":"2024-03-06T07:20:21.859723+0000","last_active":"2024-03-06T07:20:27.880426+0000","last_peered":"2024-03-06T07:20:27.880426+0000","last_clean":"2024-03-06T07:20:27.880426+0000","last_became_active":"2024-03-06T07:20:21.859392+0000","last_became_peered":"2024-03-06T07:20:21.859392+0000","last_unstale":"2024-03-06T07:20:27.880426+0000","last_undegraded":"2024-03-06T07:20:27.880426+0000","last_fullsized":"2024-03-06T07:20:27.880426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:59.701834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.200895+0000","last_change":"2024-03-06T07:20:20.860602+0000","last_active":"2024-03-06T07:20:28.200895+0000","last_peered":"2024-03-06T07:20:28.200895+0000","last_clean":"2024-03-06T07:20:28.200895+0000","last_became_active":"2024-03-06T07:20:20.860328+0000","last_became_peered":"2024-03-06T07:20:20.860328+0000","last_unstale":"2024-03-06T07:20:28.200895+0000","last_undegraded":"2024-03-06T07:20:28.200895+0000","last_fullsized":"2024-03-06T07:20:28.200895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:25.763288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880136+0000","last_change":"2024-03-06T07:20:25.889087+0000","last_active":"2024-03-06T07:20:27.880136+0000","last_peered":"2024-03-06T07:20:27.880136+0000","last_clean":"2024-03-06T07:20:27.880136+0000","last_became_active":"2024-03-06T07:20:25.888562+0000","last_became_peered":"2024-03-06T07:20:25.888562+0000","last_unstale":"2024-03-06T07:20:27.880136+0000","last_undegraded":"2024-03-06T07:20:27.880136+0000","last_fullsized":"2024-03-06T07:20:27.880136+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:22.654122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701536+0000","last_change":"2024-03-06T07:20:21.857055+0000","last_active":"2024-03-06T07:20:28.701536+0000","last_peered":"2024-03-06T07:20:28.701536+0000","last_clean":"2024-03-06T07:20:28.701536+0000","last_became_active":"2024-03-06T07:20:21.856743+0000","last_became_peered":"2024-03-06T07:20:21.856743+0000","last_unstale":"2024-03-06T07:20:28.701536+0000","last_undegraded":"2024-03-06T07:20:28.701536+0000","last_fullsized":"2024-03-06T07:20:28.701536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:20.911597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702767+0000","last_change":"2024-03-06T07:20:20.858020+0000","last_active":"2024-03-06T07:20:28.702767+0000","last_peered":"2024-03-06T07:20:28.702767+0000","last_clean":"2024-03-06T07:20:28.702767+0000","last_became_active":"2024-03-06T07:20:20.857748+0000","last_became_peered":"2024-03-06T07:20:20.857748+0000","last_unstale":"2024-03-06T07:20:28.702767+0000","last_undegraded":"2024-03-06T07:20:28.702767+0000","last_fullsized":"2024-03-06T07:20:28.702767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:47.332802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880011+0000","last_change":"2024-03-06T07:20:26.499973+0000","last_active":"2024-03-06T07:20:27.880011+0000","last_peered":"2024-03-06T07:20:27.880011+0000","last_clean":"2024-03-06T07:20:27.880011+0000","last_became_active":"2024-03-06T07:20:26.499480+0000","last_became_peered":"2024-03-06T07:20:26.499480+0000","last_unstale":"2024-03-06T07:20:27.880011+0000","last_undegraded":"2024-03-06T07:20:27.880011+0000","last_fullsized":"2024-03-06T07:20:27.880011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:55.146213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880824+0000","last_change":"2024-03-06T07:20:21.858239+0000","last_active":"2024-03-06T07:20:27.880824+0000","last_peered":"2024-03-06T07:20:27.880824+0000","last_clean":"2024-03-06T07:20:27.880824+0000","last_became_active":"2024-03-06T07:20:21.857896+0000","last_became_peered":"2024-03-06T07:20:21.857896+0000","last_unstale":"2024-03-06T07:20:27.880824+0000","last_undegraded":"2024-03-06T07:20:27.880824+0000","last_fullsized":"2024-03-06T07:20:27.880824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366190+0000","last_change":"2024-03-06T07:20:20.853233+0000","last_active":"2024-03-06T07:20:28.366190+0000","last_peered":"2024-03-06T07:20:28.366190+0000","last_clean":"2024-03-06T07:20:28.366190+0000","last_became_active":"2024-03-06T07:20:20.852941+0000","last_became_peered":"2024-03-06T07:20:20.852941+0000","last_unstale":"2024-03-06T07:20:28.366190+0000","last_undegraded":"2024-03-06T07:20:28.366190+0000","last_fullsized":"2024-03-06T07:20:28.366190+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:25.763288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879421+0000","last_change":"2024-03-06T07:20:26.499314+0000","last_active":"2024-03-06T07:20:27.879421+0000","last_peered":"2024-03-06T07:20:27.879421+0000","last_clean":"2024-03-06T07:20:27.879421+0000","last_became_active":"2024-03-06T07:20:26.498625+0000","last_became_peered":"2024-03-06T07:20:26.498625+0000","last_unstale":"2024-03-06T07:20:27.879421+0000","last_undegraded":"2024-03-06T07:20:27.879421+0000","last_fullsized":"2024-03-06T07:20:27.879421+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:05.727952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365917+0000","last_change":"2024-03-06T07:20:21.861526+0000","last_active":"2024-03-06T07:20:28.365917+0000","last_peered":"2024-03-06T07:20:28.365917+0000","last_clean":"2024-03-06T07:20:28.365917+0000","last_became_active":"2024-03-06T07:20:21.861256+0000","last_became_peered":"2024-03-06T07:20:21.861256+0000","last_unstale":"2024-03-06T07:20:28.365917+0000","last_undegraded":"2024-03-06T07:20:28.365917+0000","last_fullsized":"2024-03-06T07:20:28.365917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.423089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201484+0000","last_change":"2024-03-06T07:20:20.858234+0000","last_active":"2024-03-06T07:20:28.201484+0000","last_peered":"2024-03-06T07:20:28.201484+0000","last_clean":"2024-03-06T07:20:28.201484+0000","last_became_active":"2024-03-06T07:20:20.857910+0000","last_became_peered":"2024-03-06T07:20:20.857910+0000","last_unstale":"2024-03-06T07:20:28.201484+0000","last_undegraded":"2024-03-06T07:20:28.201484+0000","last_fullsized":"2024-03-06T07:20:28.201484+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:04.756600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701326+0000","last_change":"2024-03-06T07:20:26.498727+0000","last_active":"2024-03-06T07:20:28.701326+0000","last_peered":"2024-03-06T07:20:28.701326+0000","last_clean":"2024-03-06T07:20:28.701326+0000","last_became_active":"2024-03-06T07:20:26.498272+0000","last_became_peered":"2024-03-06T07:20:26.498272+0000","last_unstale":"2024-03-06T07:20:28.701326+0000","last_undegraded":"2024-03-06T07:20:28.701326+0000","last_fullsized":"2024-03-06T07:20:28.701326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:28.909072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702156+0000","last_change":"2024-03-06T07:20:21.868324+0000","last_active":"2024-03-06T07:20:28.702156+0000","last_peered":"2024-03-06T07:20:28.702156+0000","last_clean":"2024-03-06T07:20:28.702156+0000","last_became_active":"2024-03-06T07:20:21.868032+0000","last_became_peered":"2024-03-06T07:20:21.868032+0000","last_unstale":"2024-03-06T07:20:28.702156+0000","last_undegraded":"2024-03-06T07:20:28.702156+0000","last_fullsized":"2024-03-06T07:20:28.702156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:17.299208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366009+0000","last_change":"2024-03-06T07:20:20.851073+0000","last_active":"2024-03-06T07:20:28.366009+0000","last_peered":"2024-03-06T07:20:28.366009+0000","last_clean":"2024-03-06T07:20:28.366009+0000","last_became_active":"2024-03-06T07:20:20.850774+0000","last_became_peered":"2024-03-06T07:20:20.850774+0000","last_unstale":"2024-03-06T07:20:28.366009+0000","last_undegraded":"2024-03-06T07:20:28.366009+0000","last_fullsized":"2024-03-06T07:20:28.366009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:15.716891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365536+0000","last_change":"2024-03-06T07:20:25.895118+0000","last_active":"2024-03-06T07:20:28.365536+0000","last_peered":"2024-03-06T07:20:28.365536+0000","last_clean":"2024-03-06T07:20:28.365536+0000","last_became_active":"2024-03-06T07:20:25.894354+0000","last_became_peered":"2024-03-06T07:20:25.894354+0000","last_unstale":"2024-03-06T07:20:28.365536+0000","last_undegraded":"2024-03-06T07:20:28.365536+0000","last_fullsized":"2024-03-06T07:20:28.365536+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:28.909072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880665+0000","last_change":"2024-03-06T07:20:21.861910+0000","last_active":"2024-03-06T07:20:27.880665+0000","last_peered":"2024-03-06T07:20:27.880665+0000","last_clean":"2024-03-06T07:20:27.880665+0000","last_became_active":"2024-03-06T07:20:21.861659+0000","last_became_peered":"2024-03-06T07:20:21.861659+0000","last_unstale":"2024-03-06T07:20:27.880665+0000","last_undegraded":"2024-03-06T07:20:27.880665+0000","last_fullsized":"2024-03-06T07:20:27.880665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:44.625861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701886+0000","last_change":"2024-03-06T07:20:20.849833+0000","last_active":"2024-03-06T07:20:28.701886+0000","last_peered":"2024-03-06T07:20:28.701886+0000","last_clean":"2024-03-06T07:20:28.701886+0000","last_became_active":"2024-03-06T07:20:20.849563+0000","last_became_peered":"2024-03-06T07:20:20.849563+0000","last_unstale":"2024-03-06T07:20:28.701886+0000","last_undegraded":"2024-03-06T07:20:28.701886+0000","last_fullsized":"2024-03-06T07:20:28.701886+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:43.621233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366005+0000","last_change":"2024-03-06T07:20:25.897100+0000","last_active":"2024-03-06T07:20:28.366005+0000","last_peered":"2024-03-06T07:20:28.366005+0000","last_clean":"2024-03-06T07:20:28.366005+0000","last_became_active":"2024-03-06T07:20:25.896658+0000","last_became_peered":"2024-03-06T07:20:25.896658+0000","last_unstale":"2024-03-06T07:20:28.366005+0000","last_undegraded":"2024-03-06T07:20:28.366005+0000","last_fullsized":"2024-03-06T07:20:28.366005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:27.800537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365526+0000","last_change":"2024-03-06T07:20:21.864469+0000","last_active":"2024-03-06T07:20:28.365526+0000","last_peered":"2024-03-06T07:20:28.365526+0000","last_clean":"2024-03-06T07:20:28.365526+0000","last_became_active":"2024-03-06T07:20:21.864141+0000","last_became_peered":"2024-03-06T07:20:21.864141+0000","last_unstale":"2024-03-06T07:20:28.365526+0000","last_undegraded":"2024-03-06T07:20:28.365526+0000","last_fullsized":"2024-03-06T07:20:28.365526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:17.903391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.878975+0000","last_change":"2024-03-06T07:20:20.845652+0000","last_active":"2024-03-06T07:20:27.878975+0000","last_peered":"2024-03-06T07:20:27.878975+0000","last_clean":"2024-03-06T07:20:27.878975+0000","last_became_active":"2024-03-06T07:20:20.845350+0000","last_became_peered":"2024-03-06T07:20:20.845350+0000","last_unstale":"2024-03-06T07:20:27.878975+0000","last_undegraded":"2024-03-06T07:20:27.878975+0000","last_fullsized":"2024-03-06T07:20:27.878975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:53.997041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366358+0000","last_change":"2024-03-06T07:20:25.898164+0000","last_active":"2024-03-06T07:20:28.366358+0000","last_peered":"2024-03-06T07:20:28.366358+0000","last_clean":"2024-03-06T07:20:28.366358+0000","last_became_active":"2024-03-06T07:20:25.897755+0000","last_became_peered":"2024-03-06T07:20:25.897755+0000","last_unstale":"2024-03-06T07:20:28.366358+0000","last_undegraded":"2024-03-06T07:20:28.366358+0000","last_fullsized":"2024-03-06T07:20:28.366358+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:46.450423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701002+0000","last_change":"2024-03-06T07:20:21.850395+0000","last_active":"2024-03-06T07:20:28.701002+0000","last_peered":"2024-03-06T07:20:28.701002+0000","last_clean":"2024-03-06T07:20:28.701002+0000","last_became_active":"2024-03-06T07:20:21.850117+0000","last_became_peered":"2024-03-06T07:20:21.850117+0000","last_unstale":"2024-03-06T07:20:28.701002+0000","last_undegraded":"2024-03-06T07:20:28.701002+0000","last_fullsized":"2024-03-06T07:20:28.701002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:42.612183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.200831+0000","last_change":"2024-03-06T07:20:20.857982+0000","last_active":"2024-03-06T07:20:28.200831+0000","last_peered":"2024-03-06T07:20:28.200831+0000","last_clean":"2024-03-06T07:20:28.200831+0000","last_became_active":"2024-03-06T07:20:20.857726+0000","last_became_peered":"2024-03-06T07:20:20.857726+0000","last_unstale":"2024-03-06T07:20:28.200831+0000","last_undegraded":"2024-03-06T07:20:28.200831+0000","last_fullsized":"2024-03-06T07:20:28.200831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:24.258322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701481+0000","last_change":"2024-03-06T07:20:26.502037+0000","last_active":"2024-03-06T07:20:28.701481+0000","last_peered":"2024-03-06T07:20:28.701481+0000","last_clean":"2024-03-06T07:20:28.701481+0000","last_became_active":"2024-03-06T07:20:26.501666+0000","last_became_peered":"2024-03-06T07:20:26.501666+0000","last_unstale":"2024-03-06T07:20:28.701481+0000","last_undegraded":"2024-03-06T07:20:28.701481+0000","last_fullsized":"2024-03-06T07:20:28.701481+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:27.800537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880611+0000","last_change":"2024-03-06T07:20:21.858533+0000","last_active":"2024-03-06T07:20:27.880611+0000","last_peered":"2024-03-06T07:20:27.880611+0000","last_clean":"2024-03-06T07:20:27.880611+0000","last_became_active":"2024-03-06T07:20:21.857499+0000","last_became_peered":"2024-03-06T07:20:21.857499+0000","last_unstale":"2024-03-06T07:20:27.880611+0000","last_undegraded":"2024-03-06T07:20:27.880611+0000","last_fullsized":"2024-03-06T07:20:27.880611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:27.196530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701468+0000","last_change":"2024-03-06T07:20:20.858802+0000","last_active":"2024-03-06T07:20:28.701468+0000","last_peered":"2024-03-06T07:20:28.701468+0000","last_clean":"2024-03-06T07:20:28.701468+0000","last_became_active":"2024-03-06T07:20:20.857457+0000","last_became_peered":"2024-03-06T07:20:20.857457+0000","last_unstale":"2024-03-06T07:20:28.701468+0000","last_undegraded":"2024-03-06T07:20:28.701468+0000","last_fullsized":"2024-03-06T07:20:28.701468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:57.497682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":179,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201006+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:20:28.201006+0000","last_peered":"2024-03-06T07:20:28.201006+0000","last_clean":"2024-03-06T07:20:28.201006+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:20:28.201006+0000","last_undegraded":"2024-03-06T07:20:28.201006+0000","last_fullsized":"2024-03-06T07:20:28.201006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702367+0000","last_change":"2024-03-06T07:20:26.502317+0000","last_active":"2024-03-06T07:20:28.702367+0000","last_peered":"2024-03-06T07:20:28.702367+0000","last_clean":"2024-03-06T07:20:28.702367+0000","last_became_active":"2024-03-06T07:20:26.501807+0000","last_became_peered":"2024-03-06T07:20:26.501807+0000","last_unstale":"2024-03-06T07:20:28.702367+0000","last_undegraded":"2024-03-06T07:20:28.702367+0000","last_fullsized":"2024-03-06T07:20:28.702367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:30.890766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201288+0000","last_change":"2024-03-06T07:20:21.855669+0000","last_active":"2024-03-06T07:20:28.201288+0000","last_peered":"2024-03-06T07:20:28.201288+0000","last_clean":"2024-03-06T07:20:28.201288+0000","last_became_active":"2024-03-06T07:20:21.855154+0000","last_became_peered":"2024-03-06T07:20:21.855154+0000","last_unstale":"2024-03-06T07:20:28.201288+0000","last_undegraded":"2024-03-06T07:20:28.201288+0000","last_fullsized":"2024-03-06T07:20:28.201288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:26.855742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879323+0000","last_change":"2024-03-06T07:20:20.852702+0000","last_active":"2024-03-06T07:20:27.879323+0000","last_peered":"2024-03-06T07:20:27.879323+0000","last_clean":"2024-03-06T07:20:27.879323+0000","last_became_active":"2024-03-06T07:20:20.852434+0000","last_became_peered":"2024-03-06T07:20:20.852434+0000","last_unstale":"2024-03-06T07:20:27.879323+0000","last_undegraded":"2024-03-06T07:20:27.879323+0000","last_fullsized":"2024-03-06T07:20:27.879323+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:44.080359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702280+0000","last_change":"2024-03-06T07:20:26.499169+0000","last_active":"2024-03-06T07:20:28.702280+0000","last_peered":"2024-03-06T07:20:28.702280+0000","last_clean":"2024-03-06T07:20:28.702280+0000","last_became_active":"2024-03-06T07:20:26.498541+0000","last_became_peered":"2024-03-06T07:20:26.498541+0000","last_unstale":"2024-03-06T07:20:28.702280+0000","last_undegraded":"2024-03-06T07:20:28.702280+0000","last_fullsized":"2024-03-06T07:20:28.702280+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:49.037215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701682+0000","last_change":"2024-03-06T07:20:21.857920+0000","last_active":"2024-03-06T07:20:28.701682+0000","last_peered":"2024-03-06T07:20:28.701682+0000","last_clean":"2024-03-06T07:20:28.701682+0000","last_became_active":"2024-03-06T07:20:21.857645+0000","last_became_peered":"2024-03-06T07:20:21.857645+0000","last_unstale":"2024-03-06T07:20:28.701682+0000","last_undegraded":"2024-03-06T07:20:28.701682+0000","last_fullsized":"2024-03-06T07:20:28.701682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:47.201278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366028+0000","last_change":"2024-03-06T07:20:20.857009+0000","last_active":"2024-03-06T07:20:28.366028+0000","last_peered":"2024-03-06T07:20:28.366028+0000","last_clean":"2024-03-06T07:20:28.366028+0000","last_became_active":"2024-03-06T07:20:20.856701+0000","last_became_peered":"2024-03-06T07:20:20.856701+0000","last_unstale":"2024-03-06T07:20:28.366028+0000","last_undegraded":"2024-03-06T07:20:28.366028+0000","last_fullsized":"2024-03-06T07:20:28.366028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:24.258322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701482+0000","last_change":"2024-03-06T07:20:25.887514+0000","last_active":"2024-03-06T07:20:28.701482+0000","last_peered":"2024-03-06T07:20:28.701482+0000","last_clean":"2024-03-06T07:20:28.701482+0000","last_became_active":"2024-03-06T07:20:25.887125+0000","last_became_peered":"2024-03-06T07:20:25.887125+0000","last_unstale":"2024-03-06T07:20:28.701482+0000","last_undegraded":"2024-03-06T07:20:28.701482+0000","last_fullsized":"2024-03-06T07:20:28.701482+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:34.847448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201678+0000","last_change":"2024-03-06T07:20:21.851028+0000","last_active":"2024-03-06T07:20:28.201678+0000","last_peered":"2024-03-06T07:20:28.201678+0000","last_clean":"2024-03-06T07:20:28.201678+0000","last_became_active":"2024-03-06T07:20:21.850762+0000","last_became_peered":"2024-03-06T07:20:21.850762+0000","last_unstale":"2024-03-06T07:20:28.201678+0000","last_undegraded":"2024-03-06T07:20:28.201678+0000","last_fullsized":"2024-03-06T07:20:28.201678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:27.873513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879685+0000","last_change":"2024-03-06T07:20:20.849572+0000","last_active":"2024-03-06T07:20:27.879685+0000","last_peered":"2024-03-06T07:20:27.879685+0000","last_clean":"2024-03-06T07:20:27.879685+0000","last_became_active":"2024-03-06T07:20:20.849266+0000","last_became_peered":"2024-03-06T07:20:20.849266+0000","last_unstale":"2024-03-06T07:20:27.879685+0000","last_undegraded":"2024-03-06T07:20:27.879685+0000","last_fullsized":"2024-03-06T07:20:27.879685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:47.332802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701273+0000","last_change":"2024-03-06T07:20:26.502258+0000","last_active":"2024-03-06T07:20:28.701273+0000","last_peered":"2024-03-06T07:20:28.701273+0000","last_clean":"2024-03-06T07:20:28.701273+0000","last_became_active":"2024-03-06T07:20:26.501819+0000","last_became_peered":"2024-03-06T07:20:26.501819+0000","last_unstale":"2024-03-06T07:20:28.701273+0000","last_undegraded":"2024-03-06T07:20:28.701273+0000","last_fullsized":"2024-03-06T07:20:28.701273+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:34.847448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201451+0000","last_change":"2024-03-06T07:20:21.844939+0000","last_active":"2024-03-06T07:20:28.201451+0000","last_peered":"2024-03-06T07:20:28.201451+0000","last_clean":"2024-03-06T07:20:28.201451+0000","last_became_active":"2024-03-06T07:20:21.844652+0000","last_became_peered":"2024-03-06T07:20:21.844652+0000","last_unstale":"2024-03-06T07:20:28.201451+0000","last_undegraded":"2024-03-06T07:20:28.201451+0000","last_fullsized":"2024-03-06T07:20:28.201451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:07.625355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880344+0000","last_change":"2024-03-06T07:20:20.850150+0000","last_active":"2024-03-06T07:20:27.880344+0000","last_peered":"2024-03-06T07:20:27.880344+0000","last_clean":"2024-03-06T07:20:27.880344+0000","last_became_active":"2024-03-06T07:20:20.849834+0000","last_became_peered":"2024-03-06T07:20:20.849834+0000","last_unstale":"2024-03-06T07:20:27.880344+0000","last_undegraded":"2024-03-06T07:20:27.880344+0000","last_fullsized":"2024-03-06T07:20:27.880344+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:56.227109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879732+0000","last_change":"2024-03-06T07:20:25.888682+0000","last_active":"2024-03-06T07:20:27.879732+0000","last_peered":"2024-03-06T07:20:27.879732+0000","last_clean":"2024-03-06T07:20:27.879732+0000","last_became_active":"2024-03-06T07:20:25.888259+0000","last_became_peered":"2024-03-06T07:20:25.888259+0000","last_unstale":"2024-03-06T07:20:27.879732+0000","last_undegraded":"2024-03-06T07:20:27.879732+0000","last_fullsized":"2024-03-06T07:20:27.879732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:57.736429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201088+0000","last_change":"2024-03-06T07:20:21.851600+0000","last_active":"2024-03-06T07:20:28.201088+0000","last_peered":"2024-03-06T07:20:28.201088+0000","last_clean":"2024-03-06T07:20:28.201088+0000","last_became_active":"2024-03-06T07:20:21.851347+0000","last_became_peered":"2024-03-06T07:20:21.851347+0000","last_unstale":"2024-03-06T07:20:28.201088+0000","last_undegraded":"2024-03-06T07:20:28.201088+0000","last_fullsized":"2024-03-06T07:20:28.201088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:03.883136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.878780+0000","last_change":"2024-03-06T07:20:20.847128+0000","last_active":"2024-03-06T07:20:27.878780+0000","last_peered":"2024-03-06T07:20:27.878780+0000","last_clean":"2024-03-06T07:20:27.878780+0000","last_became_active":"2024-03-06T07:20:20.846822+0000","last_became_peered":"2024-03-06T07:20:20.846822+0000","last_unstale":"2024-03-06T07:20:27.878780+0000","last_undegraded":"2024-03-06T07:20:27.878780+0000","last_fullsized":"2024-03-06T07:20:27.878780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:57.497682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701347+0000","last_change":"2024-03-06T07:20:25.888397+0000","last_active":"2024-03-06T07:20:28.701347+0000","last_peered":"2024-03-06T07:20:28.701347+0000","last_clean":"2024-03-06T07:20:28.701347+0000","last_became_active":"2024-03-06T07:20:25.888017+0000","last_became_peered":"2024-03-06T07:20:25.888017+0000","last_unstale":"2024-03-06T07:20:28.701347+0000","last_undegraded":"2024-03-06T07:20:28.701347+0000","last_fullsized":"2024-03-06T07:20:28.701347+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:22.654122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702085+0000","last_change":"2024-03-06T07:20:21.863218+0000","last_active":"2024-03-06T07:20:28.702085+0000","last_peered":"2024-03-06T07:20:28.702085+0000","last_clean":"2024-03-06T07:20:28.702085+0000","last_became_active":"2024-03-06T07:20:21.862943+0000","last_became_peered":"2024-03-06T07:20:21.862943+0000","last_unstale":"2024-03-06T07:20:28.702085+0000","last_undegraded":"2024-03-06T07:20:28.702085+0000","last_fullsized":"2024-03-06T07:20:28.702085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:17.903391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701478+0000","last_change":"2024-03-06T07:20:20.855196+0000","last_active":"2024-03-06T07:20:28.701478+0000","last_peered":"2024-03-06T07:20:28.701478+0000","last_clean":"2024-03-06T07:20:28.701478+0000","last_became_active":"2024-03-06T07:20:20.854047+0000","last_became_peered":"2024-03-06T07:20:20.854047+0000","last_unstale":"2024-03-06T07:20:28.701478+0000","last_undegraded":"2024-03-06T07:20:28.701478+0000","last_fullsized":"2024-03-06T07:20:28.701478+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:15.716891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880771+0000","last_change":"2024-03-06T07:20:26.502104+0000","last_active":"2024-03-06T07:20:27.880771+0000","last_peered":"2024-03-06T07:20:27.880771+0000","last_clean":"2024-03-06T07:20:27.880771+0000","last_became_active":"2024-03-06T07:20:26.501701+0000","last_became_peered":"2024-03-06T07:20:26.501701+0000","last_unstale":"2024-03-06T07:20:27.880771+0000","last_undegraded":"2024-03-06T07:20:27.880771+0000","last_fullsized":"2024-03-06T07:20:27.880771+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:13.344053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201065+0000","last_change":"2024-03-06T07:20:21.850971+0000","last_active":"2024-03-06T07:20:28.201065+0000","last_peered":"2024-03-06T07:20:28.201065+0000","last_clean":"2024-03-06T07:20:28.201065+0000","last_became_active":"2024-03-06T07:20:21.850700+0000","last_became_peered":"2024-03-06T07:20:21.850700+0000","last_unstale":"2024-03-06T07:20:28.201065+0000","last_undegraded":"2024-03-06T07:20:28.201065+0000","last_fullsized":"2024-03-06T07:20:28.201065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:17.299208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701777+0000","last_change":"2024-03-06T07:20:20.856747+0000","last_active":"2024-03-06T07:20:28.701777+0000","last_peered":"2024-03-06T07:20:28.701777+0000","last_clean":"2024-03-06T07:20:28.701777+0000","last_became_active":"2024-03-06T07:20:20.856431+0000","last_became_peered":"2024-03-06T07:20:20.856431+0000","last_unstale":"2024-03-06T07:20:28.701777+0000","last_undegraded":"2024-03-06T07:20:28.701777+0000","last_fullsized":"2024-03-06T07:20:28.701777+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:15.716891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880386+0000","last_change":"2024-03-06T07:20:26.498675+0000","last_active":"2024-03-06T07:20:27.880386+0000","last_peered":"2024-03-06T07:20:27.880386+0000","last_clean":"2024-03-06T07:20:27.880386+0000","last_became_active":"2024-03-06T07:20:26.498199+0000","last_became_peered":"2024-03-06T07:20:26.498199+0000","last_unstale":"2024-03-06T07:20:27.880386+0000","last_undegraded":"2024-03-06T07:20:27.880386+0000","last_fullsized":"2024-03-06T07:20:27.880386+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:53.210084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701227+0000","last_change":"2024-03-06T07:20:21.845586+0000","last_active":"2024-03-06T07:20:28.701227+0000","last_peered":"2024-03-06T07:20:28.701227+0000","last_clean":"2024-03-06T07:20:28.701227+0000","last_became_active":"2024-03-06T07:20:21.845311+0000","last_became_peered":"2024-03-06T07:20:21.845311+0000","last_unstale":"2024-03-06T07:20:28.701227+0000","last_undegraded":"2024-03-06T07:20:28.701227+0000","last_fullsized":"2024-03-06T07:20:28.701227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:58.153137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701540+0000","last_change":"2024-03-06T07:20:20.849649+0000","last_active":"2024-03-06T07:20:28.701540+0000","last_peered":"2024-03-06T07:20:28.701540+0000","last_clean":"2024-03-06T07:20:28.701540+0000","last_became_active":"2024-03-06T07:20:20.849199+0000","last_became_peered":"2024-03-06T07:20:20.849199+0000","last_unstale":"2024-03-06T07:20:28.701540+0000","last_undegraded":"2024-03-06T07:20:28.701540+0000","last_fullsized":"2024-03-06T07:20:28.701540+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:22.023999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702184+0000","last_change":"2024-03-06T07:20:26.499070+0000","last_active":"2024-03-06T07:20:28.702184+0000","last_peered":"2024-03-06T07:20:28.702184+0000","last_clean":"2024-03-06T07:20:28.702184+0000","last_became_active":"2024-03-06T07:20:26.498601+0000","last_became_peered":"2024-03-06T07:20:26.498601+0000","last_unstale":"2024-03-06T07:20:28.702184+0000","last_undegraded":"2024-03-06T07:20:28.702184+0000","last_fullsized":"2024-03-06T07:20:28.702184+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:46.450423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201193+0000","last_change":"2024-03-06T07:20:21.857736+0000","last_active":"2024-03-06T07:20:28.201193+0000","last_peered":"2024-03-06T07:20:28.201193+0000","last_clean":"2024-03-06T07:20:28.201193+0000","last_became_active":"2024-03-06T07:20:21.857492+0000","last_became_peered":"2024-03-06T07:20:21.857492+0000","last_unstale":"2024-03-06T07:20:28.201193+0000","last_undegraded":"2024-03-06T07:20:28.201193+0000","last_fullsized":"2024-03-06T07:20:28.201193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:45.002191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":33,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:31.174467+0000","last_change":"2024-03-06T07:20:20.856316+0000","last_active":"2024-03-06T07:20:31.174467+0000","last_peered":"2024-03-06T07:20:31.174467+0000","last_clean":"2024-03-06T07:20:31.174467+0000","last_became_active":"2024-03-06T07:20:20.855994+0000","last_became_peered":"2024-03-06T07:20:20.855994+0000","last_unstale":"2024-03-06T07:20:31.174467+0000","last_undegraded":"2024-03-06T07:20:31.174467+0000","last_fullsized":"2024-03-06T07:20:31.174467+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:53.997041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365632+0000","last_change":"2024-03-06T07:20:25.895379+0000","last_active":"2024-03-06T07:20:28.365632+0000","last_peered":"2024-03-06T07:20:28.365632+0000","last_clean":"2024-03-06T07:20:28.365632+0000","last_became_active":"2024-03-06T07:20:25.894458+0000","last_became_peered":"2024-03-06T07:20:25.894458+0000","last_unstale":"2024-03-06T07:20:28.365632+0000","last_undegraded":"2024-03-06T07:20:28.365632+0000","last_fullsized":"2024-03-06T07:20:28.365632+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:22.654122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880548+0000","last_change":"2024-03-06T07:20:21.844398+0000","last_active":"2024-03-06T07:20:27.880548+0000","last_peered":"2024-03-06T07:20:27.880548+0000","last_clean":"2024-03-06T07:20:27.880548+0000","last_became_active":"2024-03-06T07:20:21.843839+0000","last_became_peered":"2024-03-06T07:20:21.843839+0000","last_unstale":"2024-03-06T07:20:27.880548+0000","last_undegraded":"2024-03-06T07:20:27.880548+0000","last_fullsized":"2024-03-06T07:20:27.880548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:25.262951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365950+0000","last_change":"2024-03-06T07:20:20.850772+0000","last_active":"2024-03-06T07:20:28.365950+0000","last_peered":"2024-03-06T07:20:28.365950+0000","last_clean":"2024-03-06T07:20:28.365950+0000","last_became_active":"2024-03-06T07:20:20.850490+0000","last_became_peered":"2024-03-06T07:20:20.850490+0000","last_unstale":"2024-03-06T07:20:28.365950+0000","last_undegraded":"2024-03-06T07:20:28.365950+0000","last_fullsized":"2024-03-06T07:20:28.365950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:58.697205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879970+0000","last_change":"2024-03-06T07:20:25.887556+0000","last_active":"2024-03-06T07:20:27.879970+0000","last_peered":"2024-03-06T07:20:27.879970+0000","last_clean":"2024-03-06T07:20:27.879970+0000","last_became_active":"2024-03-06T07:20:25.887175+0000","last_became_peered":"2024-03-06T07:20:25.887175+0000","last_unstale":"2024-03-06T07:20:27.879970+0000","last_undegraded":"2024-03-06T07:20:27.879970+0000","last_fullsized":"2024-03-06T07:20:27.879970+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:28.909072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701501+0000","last_change":"2024-03-06T07:20:21.856048+0000","last_active":"2024-03-06T07:20:28.701501+0000","last_peered":"2024-03-06T07:20:28.701501+0000","last_clean":"2024-03-06T07:20:28.701501+0000","last_became_active":"2024-03-06T07:20:21.854721+0000","last_became_peered":"2024-03-06T07:20:21.854721+0000","last_unstale":"2024-03-06T07:20:28.701501+0000","last_undegraded":"2024-03-06T07:20:28.701501+0000","last_fullsized":"2024-03-06T07:20:28.701501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:00.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702082+0000","last_change":"2024-03-06T07:20:20.859239+0000","last_active":"2024-03-06T07:20:28.702082+0000","last_peered":"2024-03-06T07:20:28.702082+0000","last_clean":"2024-03-06T07:20:28.702082+0000","last_became_active":"2024-03-06T07:20:20.858860+0000","last_became_peered":"2024-03-06T07:20:20.858860+0000","last_unstale":"2024-03-06T07:20:28.702082+0000","last_undegraded":"2024-03-06T07:20:28.702082+0000","last_fullsized":"2024-03-06T07:20:28.702082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:47.332802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879712+0000","last_change":"2024-03-06T07:20:25.890675+0000","last_active":"2024-03-06T07:20:27.879712+0000","last_peered":"2024-03-06T07:20:27.879712+0000","last_clean":"2024-03-06T07:20:27.879712+0000","last_became_active":"2024-03-06T07:20:25.890282+0000","last_became_peered":"2024-03-06T07:20:25.890282+0000","last_unstale":"2024-03-06T07:20:27.879712+0000","last_undegraded":"2024-03-06T07:20:27.879712+0000","last_fullsized":"2024-03-06T07:20:27.879712+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:10:13.387120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702028+0000","last_change":"2024-03-06T07:20:21.864960+0000","last_active":"2024-03-06T07:20:28.702028+0000","last_peered":"2024-03-06T07:20:28.702028+0000","last_clean":"2024-03-06T07:20:28.702028+0000","last_became_active":"2024-03-06T07:20:21.864687+0000","last_became_peered":"2024-03-06T07:20:21.864687+0000","last_unstale":"2024-03-06T07:20:28.702028+0000","last_undegraded":"2024-03-06T07:20:28.702028+0000","last_fullsized":"2024-03-06T07:20:28.702028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:27.196530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702647+0000","last_change":"2024-03-06T07:20:20.857187+0000","last_active":"2024-03-06T07:20:28.702647+0000","last_peered":"2024-03-06T07:20:28.702647+0000","last_clean":"2024-03-06T07:20:28.702647+0000","last_became_active":"2024-03-06T07:20:20.856898+0000","last_became_peered":"2024-03-06T07:20:20.856898+0000","last_unstale":"2024-03-06T07:20:28.702647+0000","last_undegraded":"2024-03-06T07:20:28.702647+0000","last_fullsized":"2024-03-06T07:20:28.702647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:57.497682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879802+0000","last_change":"2024-03-06T07:20:26.503331+0000","last_active":"2024-03-06T07:20:27.879802+0000","last_peered":"2024-03-06T07:20:27.879802+0000","last_clean":"2024-03-06T07:20:27.879802+0000","last_became_active":"2024-03-06T07:20:26.502770+0000","last_became_peered":"2024-03-06T07:20:26.502770+0000","last_unstale":"2024-03-06T07:20:27.879802+0000","last_undegraded":"2024-03-06T07:20:27.879802+0000","last_fullsized":"2024-03-06T07:20:27.879802+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:13.262016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701297+0000","last_change":"2024-03-06T07:20:26.502227+0000","last_active":"2024-03-06T07:20:28.701297+0000","last_peered":"2024-03-06T07:20:28.701297+0000","last_clean":"2024-03-06T07:20:28.701297+0000","last_became_active":"2024-03-06T07:20:26.501641+0000","last_became_peered":"2024-03-06T07:20:26.501641+0000","last_unstale":"2024-03-06T07:20:28.701297+0000","last_undegraded":"2024-03-06T07:20:28.701297+0000","last_fullsized":"2024-03-06T07:20:28.701297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:31.908538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365327+0000","last_change":"2024-03-06T07:20:21.866845+0000","last_active":"2024-03-06T07:20:28.365327+0000","last_peered":"2024-03-06T07:20:28.365327+0000","last_clean":"2024-03-06T07:20:28.365327+0000","last_became_active":"2024-03-06T07:20:21.866576+0000","last_became_peered":"2024-03-06T07:20:21.866576+0000","last_unstale":"2024-03-06T07:20:28.365327+0000","last_undegraded":"2024-03-06T07:20:28.365327+0000","last_fullsized":"2024-03-06T07:20:28.365327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:36.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701723+0000","last_change":"2024-03-06T07:20:20.853894+0000","last_active":"2024-03-06T07:20:28.701723+0000","last_peered":"2024-03-06T07:20:28.701723+0000","last_clean":"2024-03-06T07:20:28.701723+0000","last_became_active":"2024-03-06T07:20:20.853538+0000","last_became_peered":"2024-03-06T07:20:20.853538+0000","last_unstale":"2024-03-06T07:20:28.701723+0000","last_undegraded":"2024-03-06T07:20:28.701723+0000","last_fullsized":"2024-03-06T07:20:28.701723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:25.763288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702543+0000","last_change":"2024-03-06T07:20:25.894539+0000","last_active":"2024-03-06T07:20:28.702543+0000","last_peered":"2024-03-06T07:20:28.702543+0000","last_clean":"2024-03-06T07:20:28.702543+0000","last_became_active":"2024-03-06T07:20:25.894137+0000","last_became_peered":"2024-03-06T07:20:25.894137+0000","last_unstale":"2024-03-06T07:20:28.702543+0000","last_undegraded":"2024-03-06T07:20:28.702543+0000","last_fullsized":"2024-03-06T07:20:28.702543+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:57.736429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879760+0000","last_change":"2024-03-06T07:20:21.869266+0000","last_active":"2024-03-06T07:20:27.879760+0000","last_peered":"2024-03-06T07:20:27.879760+0000","last_clean":"2024-03-06T07:20:27.879760+0000","last_became_active":"2024-03-06T07:20:21.861979+0000","last_became_peered":"2024-03-06T07:20:21.861979+0000","last_unstale":"2024-03-06T07:20:27.879760+0000","last_undegraded":"2024-03-06T07:20:27.879760+0000","last_fullsized":"2024-03-06T07:20:27.879760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:17.903391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879911+0000","last_change":"2024-03-06T07:20:20.853365+0000","last_active":"2024-03-06T07:20:27.879911+0000","last_peered":"2024-03-06T07:20:27.879911+0000","last_clean":"2024-03-06T07:20:27.879911+0000","last_became_active":"2024-03-06T07:20:20.853065+0000","last_became_peered":"2024-03-06T07:20:20.853065+0000","last_unstale":"2024-03-06T07:20:27.879911+0000","last_undegraded":"2024-03-06T07:20:27.879911+0000","last_fullsized":"2024-03-06T07:20:27.879911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:53.997041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702584+0000","last_change":"2024-03-06T07:20:20.852681+0000","last_active":"2024-03-06T07:20:28.702584+0000","last_peered":"2024-03-06T07:20:28.702584+0000","last_clean":"2024-03-06T07:20:28.702584+0000","last_became_active":"2024-03-06T07:20:20.852406+0000","last_became_peered":"2024-03-06T07:20:20.852406+0000","last_unstale":"2024-03-06T07:20:28.702584+0000","last_undegraded":"2024-03-06T07:20:28.702584+0000","last_fullsized":"2024-03-06T07:20:28.702584+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:25.763288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701475+0000","last_change":"2024-03-06T07:20:21.854101+0000","last_active":"2024-03-06T07:20:28.701475+0000","last_peered":"2024-03-06T07:20:28.701475+0000","last_clean":"2024-03-06T07:20:28.701475+0000","last_became_active":"2024-03-06T07:20:21.853720+0000","last_became_peered":"2024-03-06T07:20:21.853720+0000","last_unstale":"2024-03-06T07:20:28.701475+0000","last_undegraded":"2024-03-06T07:20:28.701475+0000","last_fullsized":"2024-03-06T07:20:28.701475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.423089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879159+0000","last_change":"2024-03-06T07:20:25.889455+0000","last_active":"2024-03-06T07:20:27.879159+0000","last_peered":"2024-03-06T07:20:27.879159+0000","last_clean":"2024-03-06T07:20:27.879159+0000","last_became_active":"2024-03-06T07:20:25.889021+0000","last_became_peered":"2024-03-06T07:20:25.889021+0000","last_unstale":"2024-03-06T07:20:27.879159+0000","last_undegraded":"2024-03-06T07:20:27.879159+0000","last_fullsized":"2024-03-06T07:20:27.879159+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:46.647208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880275+0000","last_change":"2024-03-06T07:20:20.860219+0000","last_active":"2024-03-06T07:20:27.880275+0000","last_peered":"2024-03-06T07:20:27.880275+0000","last_clean":"2024-03-06T07:20:27.880275+0000","last_became_active":"2024-03-06T07:20:20.859862+0000","last_became_peered":"2024-03-06T07:20:20.859862+0000","last_unstale":"2024-03-06T07:20:27.880275+0000","last_undegraded":"2024-03-06T07:20:27.880275+0000","last_fullsized":"2024-03-06T07:20:27.880275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:44.080359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879980+0000","last_change":"2024-03-06T07:20:21.858660+0000","last_active":"2024-03-06T07:20:27.879980+0000","last_peered":"2024-03-06T07:20:27.879980+0000","last_clean":"2024-03-06T07:20:27.879980+0000","last_became_active":"2024-03-06T07:20:21.858397+0000","last_became_peered":"2024-03-06T07:20:21.858397+0000","last_unstale":"2024-03-06T07:20:27.879980+0000","last_undegraded":"2024-03-06T07:20:27.879980+0000","last_fullsized":"2024-03-06T07:20:27.879980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:16.721519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880388+0000","last_change":"2024-03-06T07:20:25.889183+0000","last_active":"2024-03-06T07:20:27.880388+0000","last_peered":"2024-03-06T07:20:27.880388+0000","last_clean":"2024-03-06T07:20:27.880388+0000","last_became_active":"2024-03-06T07:20:25.888836+0000","last_became_peered":"2024-03-06T07:20:25.888836+0000","last_unstale":"2024-03-06T07:20:27.880388+0000","last_undegraded":"2024-03-06T07:20:27.880388+0000","last_fullsized":"2024-03-06T07:20:27.880388+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:28.743175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701231+0000","last_change":"2024-03-06T07:20:21.863258+0000","last_active":"2024-03-06T07:20:28.701231+0000","last_peered":"2024-03-06T07:20:28.701231+0000","last_clean":"2024-03-06T07:20:28.701231+0000","last_became_active":"2024-03-06T07:20:21.862976+0000","last_became_peered":"2024-03-06T07:20:21.862976+0000","last_unstale":"2024-03-06T07:20:28.701231+0000","last_undegraded":"2024-03-06T07:20:28.701231+0000","last_fullsized":"2024-03-06T07:20:28.701231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.878887+0000","last_change":"2024-03-06T07:20:21.860263+0000","last_active":"2024-03-06T07:20:27.878887+0000","last_peered":"2024-03-06T07:20:27.878887+0000","last_clean":"2024-03-06T07:20:27.878887+0000","last_became_active":"2024-03-06T07:20:21.859978+0000","last_became_peered":"2024-03-06T07:20:21.859978+0000","last_unstale":"2024-03-06T07:20:27.878887+0000","last_undegraded":"2024-03-06T07:20:27.878887+0000","last_fullsized":"2024-03-06T07:20:27.878887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:59.701834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701885+0000","last_change":"2024-03-06T07:20:21.868562+0000","last_active":"2024-03-06T07:20:28.701885+0000","last_peered":"2024-03-06T07:20:28.701885+0000","last_clean":"2024-03-06T07:20:28.701885+0000","last_became_active":"2024-03-06T07:20:21.868239+0000","last_became_peered":"2024-03-06T07:20:21.868239+0000","last_unstale":"2024-03-06T07:20:28.701885+0000","last_undegraded":"2024-03-06T07:20:28.701885+0000","last_fullsized":"2024-03-06T07:20:28.701885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:45.002191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701377+0000","last_change":"2024-03-06T07:20:21.860455+0000","last_active":"2024-03-06T07:20:28.701377+0000","last_peered":"2024-03-06T07:20:28.701377+0000","last_clean":"2024-03-06T07:20:28.701377+0000","last_became_active":"2024-03-06T07:20:21.859855+0000","last_became_peered":"2024-03-06T07:20:21.859855+0000","last_unstale":"2024-03-06T07:20:28.701377+0000","last_undegraded":"2024-03-06T07:20:28.701377+0000","last_fullsized":"2024-03-06T07:20:28.701377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879935+0000","last_change":"2024-03-06T07:20:21.856836+0000","last_active":"2024-03-06T07:20:27.879935+0000","last_peered":"2024-03-06T07:20:27.879935+0000","last_clean":"2024-03-06T07:20:27.879935+0000","last_became_active":"2024-03-06T07:20:21.856421+0000","last_became_peered":"2024-03-06T07:20:21.856421+0000","last_unstale":"2024-03-06T07:20:27.879935+0000","last_undegraded":"2024-03-06T07:20:27.879935+0000","last_fullsized":"2024-03-06T07:20:27.879935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:00.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365852+0000","last_change":"2024-03-06T07:20:21.866911+0000","last_active":"2024-03-06T07:20:28.365852+0000","last_peered":"2024-03-06T07:20:28.365852+0000","last_clean":"2024-03-06T07:20:28.365852+0000","last_became_active":"2024-03-06T07:20:21.866557+0000","last_became_peered":"2024-03-06T07:20:21.866557+0000","last_unstale":"2024-03-06T07:20:28.365852+0000","last_undegraded":"2024-03-06T07:20:28.365852+0000","last_fullsized":"2024-03-06T07:20:28.365852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702389+0000","last_change":"2024-03-06T07:20:21.866315+0000","last_active":"2024-03-06T07:20:28.702389+0000","last_peered":"2024-03-06T07:20:28.702389+0000","last_clean":"2024-03-06T07:20:28.702389+0000","last_became_active":"2024-03-06T07:20:21.866035+0000","last_became_peered":"2024-03-06T07:20:21.866035+0000","last_unstale":"2024-03-06T07:20:28.702389+0000","last_undegraded":"2024-03-06T07:20:28.702389+0000","last_fullsized":"2024-03-06T07:20:28.702389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:20.911597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880753+0000","last_change":"2024-03-06T07:20:21.866364+0000","last_active":"2024-03-06T07:20:27.880753+0000","last_peered":"2024-03-06T07:20:27.880753+0000","last_clean":"2024-03-06T07:20:27.880753+0000","last_became_active":"2024-03-06T07:20:21.866091+0000","last_became_peered":"2024-03-06T07:20:21.866091+0000","last_unstale":"2024-03-06T07:20:27.880753+0000","last_undegraded":"2024-03-06T07:20:27.880753+0000","last_fullsized":"2024-03-06T07:20:27.880753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:01.604000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701743+0000","last_change":"2024-03-06T07:20:21.868488+0000","last_active":"2024-03-06T07:20:28.701743+0000","last_peered":"2024-03-06T07:20:28.701743+0000","last_clean":"2024-03-06T07:20:28.701743+0000","last_became_active":"2024-03-06T07:20:21.868091+0000","last_became_peered":"2024-03-06T07:20:21.868091+0000","last_unstale":"2024-03-06T07:20:28.701743+0000","last_undegraded":"2024-03-06T07:20:28.701743+0000","last_fullsized":"2024-03-06T07:20:28.701743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:00.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880000+0000","last_change":"2024-03-06T07:20:21.853440+0000","last_active":"2024-03-06T07:20:27.880000+0000","last_peered":"2024-03-06T07:20:27.880000+0000","last_clean":"2024-03-06T07:20:27.880000+0000","last_became_active":"2024-03-06T07:20:21.853089+0000","last_became_peered":"2024-03-06T07:20:21.853089+0000","last_unstale":"2024-03-06T07:20:27.880000+0000","last_undegraded":"2024-03-06T07:20:27.880000+0000","last_fullsized":"2024-03-06T07:20:27.880000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:20.911597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701633+0000","last_change":"2024-03-06T07:20:21.865493+0000","last_active":"2024-03-06T07:20:28.701633+0000","last_peered":"2024-03-06T07:20:28.701633+0000","last_clean":"2024-03-06T07:20:28.701633+0000","last_became_active":"2024-03-06T07:20:21.865203+0000","last_became_peered":"2024-03-06T07:20:21.865203+0000","last_unstale":"2024-03-06T07:20:28.701633+0000","last_undegraded":"2024-03-06T07:20:28.701633+0000","last_fullsized":"2024-03-06T07:20:28.701633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:20.882851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701280+0000","last_change":"2024-03-06T07:20:21.847412+0000","last_active":"2024-03-06T07:20:28.701280+0000","last_peered":"2024-03-06T07:20:28.701280+0000","last_clean":"2024-03-06T07:20:28.701280+0000","last_became_active":"2024-03-06T07:20:21.847081+0000","last_became_peered":"2024-03-06T07:20:21.847081+0000","last_unstale":"2024-03-06T07:20:28.701280+0000","last_undegraded":"2024-03-06T07:20:28.701280+0000","last_fullsized":"2024-03-06T07:20:28.701280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:28.363605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702051+0000","last_change":"2024-03-06T07:20:21.861878+0000","last_active":"2024-03-06T07:20:28.702051+0000","last_peered":"2024-03-06T07:20:28.702051+0000","last_clean":"2024-03-06T07:20:28.702051+0000","last_became_active":"2024-03-06T07:20:21.859115+0000","last_became_peered":"2024-03-06T07:20:21.859115+0000","last_unstale":"2024-03-06T07:20:28.702051+0000","last_undegraded":"2024-03-06T07:20:28.702051+0000","last_fullsized":"2024-03-06T07:20:28.702051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:42.612183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201453+0000","last_change":"2024-03-06T07:20:21.860975+0000","last_active":"2024-03-06T07:20:28.201453+0000","last_peered":"2024-03-06T07:20:28.201453+0000","last_clean":"2024-03-06T07:20:28.201453+0000","last_became_active":"2024-03-06T07:20:21.860414+0000","last_became_peered":"2024-03-06T07:20:21.860414+0000","last_unstale":"2024-03-06T07:20:28.201453+0000","last_undegraded":"2024-03-06T07:20:28.201453+0000","last_fullsized":"2024-03-06T07:20:28.201453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:53.701404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879575+0000","last_change":"2024-03-06T07:20:21.867097+0000","last_active":"2024-03-06T07:20:27.879575+0000","last_peered":"2024-03-06T07:20:27.879575+0000","last_clean":"2024-03-06T07:20:27.879575+0000","last_became_active":"2024-03-06T07:20:21.866821+0000","last_became_peered":"2024-03-06T07:20:21.866821+0000","last_unstale":"2024-03-06T07:20:27.879575+0000","last_undegraded":"2024-03-06T07:20:27.879575+0000","last_fullsized":"2024-03-06T07:20:27.879575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:25.262951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702397+0000","last_change":"2024-03-06T07:20:21.866152+0000","last_active":"2024-03-06T07:20:28.702397+0000","last_peered":"2024-03-06T07:20:28.702397+0000","last_clean":"2024-03-06T07:20:28.702397+0000","last_became_active":"2024-03-06T07:20:21.865919+0000","last_became_peered":"2024-03-06T07:20:21.865919+0000","last_unstale":"2024-03-06T07:20:28.702397+0000","last_undegraded":"2024-03-06T07:20:28.702397+0000","last_fullsized":"2024-03-06T07:20:28.702397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:26.855742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701560+0000","last_change":"2024-03-06T07:20:21.860761+0000","last_active":"2024-03-06T07:20:28.701560+0000","last_peered":"2024-03-06T07:20:28.701560+0000","last_clean":"2024-03-06T07:20:28.701560+0000","last_became_active":"2024-03-06T07:20:21.860444+0000","last_became_peered":"2024-03-06T07:20:21.860444+0000","last_unstale":"2024-03-06T07:20:28.701560+0000","last_undegraded":"2024-03-06T07:20:28.701560+0000","last_fullsized":"2024-03-06T07:20:28.701560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365409+0000","last_change":"2024-03-06T07:20:21.855033+0000","last_active":"2024-03-06T07:20:28.365409+0000","last_peered":"2024-03-06T07:20:28.365409+0000","last_clean":"2024-03-06T07:20:28.365409+0000","last_became_active":"2024-03-06T07:20:21.854554+0000","last_became_peered":"2024-03-06T07:20:21.854554+0000","last_unstale":"2024-03-06T07:20:28.365409+0000","last_undegraded":"2024-03-06T07:20:28.365409+0000","last_fullsized":"2024-03-06T07:20:28.365409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:20.911597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701196+0000","last_change":"2024-03-06T07:20:21.856900+0000","last_active":"2024-03-06T07:20:28.701196+0000","last_peered":"2024-03-06T07:20:28.701196+0000","last_clean":"2024-03-06T07:20:28.701196+0000","last_became_active":"2024-03-06T07:20:21.856628+0000","last_became_peered":"2024-03-06T07:20:21.856628+0000","last_unstale":"2024-03-06T07:20:28.701196+0000","last_undegraded":"2024-03-06T07:20:28.701196+0000","last_fullsized":"2024-03-06T07:20:28.701196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:47.276655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880101+0000","last_change":"2024-03-06T07:20:21.868803+0000","last_active":"2024-03-06T07:20:27.880101+0000","last_peered":"2024-03-06T07:20:27.880101+0000","last_clean":"2024-03-06T07:20:27.880101+0000","last_became_active":"2024-03-06T07:20:21.868518+0000","last_became_peered":"2024-03-06T07:20:21.868518+0000","last_unstale":"2024-03-06T07:20:27.880101+0000","last_undegraded":"2024-03-06T07:20:27.880101+0000","last_fullsized":"2024-03-06T07:20:27.880101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:03.883136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880416+0000","last_change":"2024-03-06T07:20:21.863674+0000","last_active":"2024-03-06T07:20:27.880416+0000","last_peered":"2024-03-06T07:20:27.880416+0000","last_clean":"2024-03-06T07:20:27.880416+0000","last_became_active":"2024-03-06T07:20:21.863138+0000","last_became_peered":"2024-03-06T07:20:21.863138+0000","last_unstale":"2024-03-06T07:20:27.880416+0000","last_undegraded":"2024-03-06T07:20:27.880416+0000","last_fullsized":"2024-03-06T07:20:27.880416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:26.767916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702522+0000","last_change":"2024-03-06T07:20:21.858343+0000","last_active":"2024-03-06T07:20:28.702522+0000","last_peered":"2024-03-06T07:20:28.702522+0000","last_clean":"2024-03-06T07:20:28.702522+0000","last_became_active":"2024-03-06T07:20:21.858078+0000","last_became_peered":"2024-03-06T07:20:21.858078+0000","last_unstale":"2024-03-06T07:20:28.702522+0000","last_undegraded":"2024-03-06T07:20:28.702522+0000","last_fullsized":"2024-03-06T07:20:28.702522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.423089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701847+0000","last_change":"2024-03-06T07:20:21.866157+0000","last_active":"2024-03-06T07:20:28.701847+0000","last_peered":"2024-03-06T07:20:28.701847+0000","last_clean":"2024-03-06T07:20:28.701847+0000","last_became_active":"2024-03-06T07:20:21.865306+0000","last_became_peered":"2024-03-06T07:20:21.865306+0000","last_unstale":"2024-03-06T07:20:28.701847+0000","last_undegraded":"2024-03-06T07:20:28.701847+0000","last_fullsized":"2024-03-06T07:20:28.701847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:36.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201624+0000","last_change":"2024-03-06T07:20:21.855346+0000","last_active":"2024-03-06T07:20:28.201624+0000","last_peered":"2024-03-06T07:20:28.201624+0000","last_clean":"2024-03-06T07:20:28.201624+0000","last_became_active":"2024-03-06T07:20:21.855059+0000","last_became_peered":"2024-03-06T07:20:21.855059+0000","last_unstale":"2024-03-06T07:20:28.201624+0000","last_undegraded":"2024-03-06T07:20:28.201624+0000","last_fullsized":"2024-03-06T07:20:28.201624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:51.111188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879524+0000","last_change":"2024-03-06T07:20:21.868545+0000","last_active":"2024-03-06T07:20:27.879524+0000","last_peered":"2024-03-06T07:20:27.879524+0000","last_clean":"2024-03-06T07:20:27.879524+0000","last_became_active":"2024-03-06T07:20:21.868188+0000","last_became_peered":"2024-03-06T07:20:21.868188+0000","last_unstale":"2024-03-06T07:20:27.879524+0000","last_undegraded":"2024-03-06T07:20:27.879524+0000","last_fullsized":"2024-03-06T07:20:27.879524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:28.363605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201003+0000","last_change":"2024-03-06T07:20:21.856718+0000","last_active":"2024-03-06T07:20:28.201003+0000","last_peered":"2024-03-06T07:20:28.201003+0000","last_clean":"2024-03-06T07:20:28.201003+0000","last_became_active":"2024-03-06T07:20:21.856451+0000","last_became_peered":"2024-03-06T07:20:21.856451+0000","last_unstale":"2024-03-06T07:20:28.201003+0000","last_undegraded":"2024-03-06T07:20:28.201003+0000","last_fullsized":"2024-03-06T07:20:28.201003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:01.604000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701625+0000","last_change":"2024-03-06T07:20:21.861834+0000","last_active":"2024-03-06T07:20:28.701625+0000","last_peered":"2024-03-06T07:20:28.701625+0000","last_clean":"2024-03-06T07:20:28.701625+0000","last_became_active":"2024-03-06T07:20:21.860998+0000","last_became_peered":"2024-03-06T07:20:21.860998+0000","last_unstale":"2024-03-06T07:20:28.701625+0000","last_undegraded":"2024-03-06T07:20:28.701625+0000","last_fullsized":"2024-03-06T07:20:28.701625+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:01.604000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.200939+0000","last_change":"2024-03-06T07:20:21.860505+0000","last_active":"2024-03-06T07:20:28.200939+0000","last_peered":"2024-03-06T07:20:28.200939+0000","last_clean":"2024-03-06T07:20:28.200939+0000","last_became_active":"2024-03-06T07:20:21.860241+0000","last_became_peered":"2024-03-06T07:20:21.860241+0000","last_unstale":"2024-03-06T07:20:28.200939+0000","last_undegraded":"2024-03-06T07:20:28.200939+0000","last_fullsized":"2024-03-06T07:20:28.200939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:20.882851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701769+0000","last_change":"2024-03-06T07:20:21.868402+0000","last_active":"2024-03-06T07:20:28.701769+0000","last_peered":"2024-03-06T07:20:28.701769+0000","last_clean":"2024-03-06T07:20:28.701769+0000","last_became_active":"2024-03-06T07:20:21.868068+0000","last_became_peered":"2024-03-06T07:20:21.868068+0000","last_unstale":"2024-03-06T07:20:28.701769+0000","last_undegraded":"2024-03-06T07:20:28.701769+0000","last_fullsized":"2024-03-06T07:20:28.701769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:03.883136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701196+0000","last_change":"2024-03-06T07:20:21.860664+0000","last_active":"2024-03-06T07:20:28.701196+0000","last_peered":"2024-03-06T07:20:28.701196+0000","last_clean":"2024-03-06T07:20:28.701196+0000","last_became_active":"2024-03-06T07:20:21.860393+0000","last_became_peered":"2024-03-06T07:20:21.860393+0000","last_unstale":"2024-03-06T07:20:28.701196+0000","last_undegraded":"2024-03-06T07:20:28.701196+0000","last_fullsized":"2024-03-06T07:20:28.701196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:36.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366176+0000","last_change":"2024-03-06T07:20:21.860087+0000","last_active":"2024-03-06T07:20:28.366176+0000","last_peered":"2024-03-06T07:20:28.366176+0000","last_clean":"2024-03-06T07:20:28.366176+0000","last_became_active":"2024-03-06T07:20:21.859804+0000","last_became_peered":"2024-03-06T07:20:21.859804+0000","last_unstale":"2024-03-06T07:20:28.366176+0000","last_undegraded":"2024-03-06T07:20:28.366176+0000","last_fullsized":"2024-03-06T07:20:28.366176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:00.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201546+0000","last_change":"2024-03-06T07:20:21.859581+0000","last_active":"2024-03-06T07:20:28.201546+0000","last_peered":"2024-03-06T07:20:28.201546+0000","last_clean":"2024-03-06T07:20:28.201546+0000","last_became_active":"2024-03-06T07:20:21.859305+0000","last_became_peered":"2024-03-06T07:20:21.859305+0000","last_unstale":"2024-03-06T07:20:28.201546+0000","last_undegraded":"2024-03-06T07:20:28.201546+0000","last_fullsized":"2024-03-06T07:20:28.201546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:47.201278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":698343,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":344064,"data_stored":698399,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:20:43.634 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:20:43.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.768+0000 7f9986f7a640 1 Processor -- start 2024-03-06T07:20:43.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.768+0000 7f9986f7a640 1 -- start start 2024-03-06T07:20:43.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.769+0000 7f9986f7a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9980152820 0x7f99800c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:43.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.769+0000 7f9986f7a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99800c0670 0x7f99800c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:43.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.769+0000 7f9986f7a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99800c7180 con 0x7f99800c7930 2024-03-06T07:20:43.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.769+0000 7f9986f7a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9980155460 con 0x7f9980152820 2024-03-06T07:20:43.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.769+0000 7f9986f7a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99800c6bc0 con 0x7f99800c0670 2024-03-06T07:20:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f9985517640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f99800c7930 0x7f9980158830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57536/0 (socket says 172.21.15.1:57536) 2024-03-06T07:20:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f9985517640 1 -- 172.21.15.1:0/1620514800 learned_addr learned my addr 172.21.15.1:0/1620514800 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997ffff640 1 --2- 172.21.15.1:0/1620514800 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99800c0670 0x7f99800c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f9984d16640 1 --2- 172.21.15.1:0/1620514800 >> v2:172.21.15.98:3300/0 conn(0x7f9980152820 0x7f99800c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997f7fe640 1 -- 172.21.15.1:0/1620514800 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798830153 0 0) 0x7f99800c7180 con 0x7f99800c7930 2024-03-06T07:20:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997f7fe640 1 -- 172.21.15.1:0/1620514800 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9968003520 con 0x7f99800c7930 2024-03-06T07:20:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997f7fe640 1 -- 172.21.15.1:0/1620514800 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 932898994 0 0) 0x7f9968003520 con 0x7f99800c7930 2024-03-06T07:20:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997f7fe640 1 -- 172.21.15.1:0/1620514800 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99800c0670 msgr2=0x7f99800c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997f7fe640 1 --2- 172.21.15.1:0/1620514800 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99800c0670 0x7f99800c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997f7fe640 1 -- 172.21.15.1:0/1620514800 >> v2:172.21.15.98:3300/0 conn(0x7f9980152820 msgr2=0x7f99800c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997f7fe640 1 --2- 172.21.15.1:0/1620514800 >> v2:172.21.15.98:3300/0 conn(0x7f9980152820 0x7f99800c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997f7fe640 1 -- 172.21.15.1:0/1620514800 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99800c6ea0 con 0x7f99800c7930 2024-03-06T07:20:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997f7fe640 1 -- 172.21.15.1:0/1620514800 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f99780019f0 con 0x7f99800c7930 2024-03-06T07:20:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997f7fe640 1 -- 172.21.15.1:0/1620514800 >> v1:172.21.15.1:6789/0 conn(0x7f99800c7930 legacy=0x7f9980158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997f7fe640 1 --2- 172.21.15.1:0/1620514800 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99800c0670 0x7f99680039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997f7fe640 1 --2- 172.21.15.1:0/1620514800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9968003f30 0x7f9968004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997f7fe640 1 --2- 172.21.15.1:0/1620514800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9968004d40 0x7f99680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997f7fe640 1 -- 172.21.15.1:0/1620514800 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9980155460 con 0x7f9968003f30 2024-03-06T07:20:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997f7fe640 1 -- 172.21.15.1:0/1620514800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99800c6bc0 con 0x7f9968004d40 2024-03-06T07:20:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997f7fe640 1 -- 172.21.15.1:0/1620514800 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996800b9c0 con 0x7f99800c0670 2024-03-06T07:20:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f997ffff640 1 --2- 172.21.15.1:0/1620514800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9968003f30 0x7f9968004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f9984d16640 1 --2- 172.21.15.1:0/1620514800 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99800c0670 0x7f99680039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.770+0000 7f9985517640 1 --2- 172.21.15.1:0/1620514800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9968004d40 0x7f99680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.771+0000 7f997ffff640 1 -- 172.21.15.1:0/1620514800 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99800c0670 msgr2=0x7f99680039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.771+0000 7f997ffff640 1 --2- 172.21.15.1:0/1620514800 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99800c0670 0x7f99680039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.771+0000 7f997ffff640 1 -- 172.21.15.1:0/1620514800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9968004d40 msgr2=0x7f99680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.771+0000 7f997ffff640 1 --2- 172.21.15.1:0/1620514800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9968004d40 0x7f99680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.771+0000 7f997ffff640 1 -- 172.21.15.1:0/1620514800 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9968003520 con 0x7f9968003f30 2024-03-06T07:20:43.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.771+0000 7f997ffff640 1 --2- 172.21.15.1:0/1620514800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9968003f30 0x7f9968004800 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9974009a90 tx=0x7f9968005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed0cd205153109bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:43.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.771+0000 7f997f7fe640 1 -- 172.21.15.1:0/1620514800 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9974039070 con 0x7f9968003f30 2024-03-06T07:20:43.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.771+0000 7f997f7fe640 1 -- 172.21.15.1:0/1620514800 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997402c6b0 con 0x7f9968003f30 2024-03-06T07:20:43.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.771+0000 7f997f7fe640 1 -- 172.21.15.1:0/1620514800 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9974009690 con 0x7f9968003f30 2024-03-06T07:20:43.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.772+0000 7f9986f7a640 1 -- 172.21.15.1:0/1620514800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9968003f30 msgr2=0x7f9968004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.772+0000 7f9986f7a640 1 --2- 172.21.15.1:0/1620514800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9968003f30 0x7f9968004800 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9974009a90 tx=0x7f9968005830 comp rx=0 tx=0).stop 2024-03-06T07:20:43.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.772+0000 7f9985517640 1 -- 172.21.15.1:0/1620514800 reap_dead start 2024-03-06T07:20:43.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.772+0000 7f9986f7a640 1 -- 172.21.15.1:0/1620514800 shutdown_connections 2024-03-06T07:20:43.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.772+0000 7f9986f7a640 1 -- 172.21.15.1:0/1620514800 >> 172.21.15.1:0/1620514800 conn(0x7f9980072720 msgr2=0x7f9980072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:43.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.772+0000 7f9986f7a640 1 -- 172.21.15.1:0/1620514800 shutdown_connections 2024-03-06T07:20:43.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.772+0000 7f9986f7a640 1 -- 172.21.15.1:0/1620514800 wait complete. 2024-03-06T07:20:43.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.774+0000 7f9986f7a640 1 Processor -- start 2024-03-06T07:20:43.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.774+0000 7f9986f7a640 1 -- start start 2024-03-06T07:20:43.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f9986f7a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9968003f30 0x7f9980156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:43.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f9986f7a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99800c43a0 0x7f99801ebd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:43.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f9986f7a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9980149700 0x7f99801ee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:43.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f9986f7a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9980152d80 con 0x7f99800c43a0 2024-03-06T07:20:43.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f9986f7a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9980152c00 con 0x7f9980149700 2024-03-06T07:20:43.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f9986f7a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99800af6f0 con 0x7f9968003f30 2024-03-06T07:20:43.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f997ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99800c43a0 0x7f99801ebd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:43.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f997ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99800c43a0 0x7f99801ebd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40798/0 (socket says 172.21.15.1:40798) 2024-03-06T07:20:43.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f997ffff640 1 -- 172.21.15.1:0/1616949202 learned_addr learned my addr 172.21.15.1:0/1616949202 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:43.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f9984d16640 1 --2- 172.21.15.1:0/1616949202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9968003f30 0x7f9980156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:43.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f9985517640 1 --2- 172.21.15.1:0/1616949202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9980149700 0x7f99801ee7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:43.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f997ffff640 1 -- 172.21.15.1:0/1616949202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9968003f30 msgr2=0x7f9980156630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f997ffff640 1 --2- 172.21.15.1:0/1616949202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9968003f30 0x7f9980156630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f997ffff640 1 -- 172.21.15.1:0/1616949202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9980149700 msgr2=0x7f99801ee7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f997ffff640 1 --2- 172.21.15.1:0/1616949202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9980149700 0x7f99801ee7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f997ffff640 1 -- 172.21.15.1:0/1616949202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99801ec240 con 0x7f99800c43a0 2024-03-06T07:20:43.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f9985517640 1 --2- 172.21.15.1:0/1616949202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9980149700 0x7f99801ee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:43.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.775+0000 7f9984d16640 1 --2- 172.21.15.1:0/1616949202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9968003f30 0x7f9980156630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:43.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.776+0000 7f997ffff640 1 --2- 172.21.15.1:0/1616949202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99800c43a0 0x7f99801ebd00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9974000c00 tx=0x7f9974004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:43.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.776+0000 7f997d7fa640 1 -- 172.21.15.1:0/1616949202 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9974042020 con 0x7f99800c43a0 2024-03-06T07:20:43.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.776+0000 7f997d7fa640 1 -- 172.21.15.1:0/1616949202 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997403a040 con 0x7f99800c43a0 2024-03-06T07:20:43.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.776+0000 7f9986f7a640 1 -- 172.21.15.1:0/1616949202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99801eb330 con 0x7f99800c43a0 2024-03-06T07:20:43.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.776+0000 7f997d7fa640 1 -- 172.21.15.1:0/1616949202 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9974039070 con 0x7f99800c43a0 2024-03-06T07:20:43.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.776+0000 7f9986f7a640 1 -- 172.21.15.1:0/1616949202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99801edfa0 con 0x7f99800c43a0 2024-03-06T07:20:43.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.777+0000 7f9986f7a640 1 -- 172.21.15.1:0/1616949202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99800686e0 con 0x7f99800c43a0 2024-03-06T07:20:43.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.777+0000 7f997d7fa640 1 -- 172.21.15.1:0/1616949202 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9974045020 con 0x7f99800c43a0 2024-03-06T07:20:43.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.778+0000 7f997d7fa640 1 --2- 172.21.15.1:0/1616949202 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f996c06b4d0 0x7f996c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:43.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.778+0000 7f997d7fa640 1 -- 172.21.15.1:0/1616949202 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f99740ab350 con 0x7f99800c43a0 2024-03-06T07:20:43.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.778+0000 7f9984d16640 1 --2- 172.21.15.1:0/1616949202 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f996c06b4d0 0x7f996c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:43.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.779+0000 7f9984d16640 1 --2- 172.21.15.1:0/1616949202 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f996c06b4d0 0x7f996c06d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9970009c40 tx=0x7f9970008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:43.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.782+0000 7f997d7fa640 1 -- 172.21.15.1:0/1616949202 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9974079f20 con 0x7f99800c43a0 2024-03-06T07:20:43.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.958+0000 7f9986f7a640 1 -- 172.21.15.1:0/1616949202 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f99800c3850 con 0x7f996c06b4d0 2024-03-06T07:20:43.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.967+0000 7f997d7fa640 1 -- 172.21.15.1:0/1616949202 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332611 (secure 0 0 0) 0x7f9970009890 con 0x7f996c06b4d0 2024-03-06T07:20:43.972 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:43.974 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:20:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.970+0000 7f9986f7a640 1 -- 172.21.15.1:0/1616949202 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f996c06b4d0 msgr2=0x7f996c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.970+0000 7f9986f7a640 1 --2- 172.21.15.1:0/1616949202 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f996c06b4d0 0x7f996c06d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9970009c40 tx=0x7f9970008040 comp rx=0 tx=0).stop 2024-03-06T07:20:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.970+0000 7f9986f7a640 1 -- 172.21.15.1:0/1616949202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99800c43a0 msgr2=0x7f99801ebd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.970+0000 7f9986f7a640 1 --2- 172.21.15.1:0/1616949202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99800c43a0 0x7f99801ebd00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9974000c00 tx=0x7f9974004730 comp rx=0 tx=0).stop 2024-03-06T07:20:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.970+0000 7f9986f7a640 1 -- 172.21.15.1:0/1616949202 shutdown_connections 2024-03-06T07:20:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.970+0000 7f9986f7a640 1 --2- 172.21.15.1:0/1616949202 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f996c06b4d0 0x7f996c06d990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.970+0000 7f9986f7a640 1 --2- 172.21.15.1:0/1616949202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9980149700 0x7f99801ee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.970+0000 7f9986f7a640 1 --2- 172.21.15.1:0/1616949202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99800c43a0 0x7f99801ebd00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.970+0000 7f9986f7a640 1 --2- 172.21.15.1:0/1616949202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9968003f30 0x7f9980156630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:43.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.970+0000 7f9986f7a640 1 -- 172.21.15.1:0/1616949202 >> 172.21.15.1:0/1616949202 conn(0x7f9980072720 msgr2=0x7f9980151a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:43.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.970+0000 7f9986f7a640 1 -- 172.21.15.1:0/1616949202 shutdown_connections 2024-03-06T07:20:43.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:43.970+0000 7f9986f7a640 1 -- 172.21.15.1:0/1616949202 wait complete. 2024-03-06T07:20:43.983 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-03-06T07:20:42.691719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":230920,"kb_used_data":2272,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754743800,"statfs":{"total":773094113280,"available":772857651200,"internally_reserved":0,"allocated":2326528,"data_stored":2049160,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004241"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701618+0000","last_change":"2024-03-06T07:20:20.860733+0000","last_active":"2024-03-06T07:20:28.701618+0000","last_peered":"2024-03-06T07:20:28.701618+0000","last_clean":"2024-03-06T07:20:28.701618+0000","last_became_active":"2024-03-06T07:20:20.860412+0000","last_became_peered":"2024-03-06T07:20:20.860412+0000","last_unstale":"2024-03-06T07:20:28.701618+0000","last_undegraded":"2024-03-06T07:20:28.701618+0000","last_fullsized":"2024-03-06T07:20:28.701618+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:44.080359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880598+0000","last_change":"2024-03-06T07:20:21.869098+0000","last_active":"2024-03-06T07:20:27.880598+0000","last_peered":"2024-03-06T07:20:27.880598+0000","last_clean":"2024-03-06T07:20:27.880598+0000","last_became_active":"2024-03-06T07:20:21.868808+0000","last_became_peered":"2024-03-06T07:20:21.868808+0000","last_unstale":"2024-03-06T07:20:27.880598+0000","last_undegraded":"2024-03-06T07:20:27.880598+0000","last_fullsized":"2024-03-06T07:20:27.880598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.423089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879651+0000","last_change":"2024-03-06T07:20:26.504312+0000","last_active":"2024-03-06T07:20:27.879651+0000","last_peered":"2024-03-06T07:20:27.879651+0000","last_clean":"2024-03-06T07:20:27.879651+0000","last_became_active":"2024-03-06T07:20:26.503884+0000","last_became_peered":"2024-03-06T07:20:26.503884+0000","last_unstale":"2024-03-06T07:20:27.879651+0000","last_undegraded":"2024-03-06T07:20:27.879651+0000","last_fullsized":"2024-03-06T07:20:27.879651+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:57:56.136234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880779+0000","last_change":"2024-03-06T07:20:20.854641+0000","last_active":"2024-03-06T07:20:27.880779+0000","last_peered":"2024-03-06T07:20:27.880779+0000","last_clean":"2024-03-06T07:20:27.880779+0000","last_became_active":"2024-03-06T07:20:20.854361+0000","last_became_peered":"2024-03-06T07:20:20.854361+0000","last_unstale":"2024-03-06T07:20:27.880779+0000","last_undegraded":"2024-03-06T07:20:27.880779+0000","last_fullsized":"2024-03-06T07:20:27.880779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:57.497682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701411+0000","last_change":"2024-03-06T07:20:21.857450+0000","last_active":"2024-03-06T07:20:28.701411+0000","last_peered":"2024-03-06T07:20:28.701411+0000","last_clean":"2024-03-06T07:20:28.701411+0000","last_became_active":"2024-03-06T07:20:21.855756+0000","last_became_peered":"2024-03-06T07:20:21.855756+0000","last_unstale":"2024-03-06T07:20:28.701411+0000","last_undegraded":"2024-03-06T07:20:28.701411+0000","last_fullsized":"2024-03-06T07:20:28.701411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:17.903391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879763+0000","last_change":"2024-03-06T07:20:25.889839+0000","last_active":"2024-03-06T07:20:27.879763+0000","last_peered":"2024-03-06T07:20:27.879763+0000","last_clean":"2024-03-06T07:20:27.879763+0000","last_became_active":"2024-03-06T07:20:25.889266+0000","last_became_peered":"2024-03-06T07:20:25.889266+0000","last_unstale":"2024-03-06T07:20:27.879763+0000","last_undegraded":"2024-03-06T07:20:27.879763+0000","last_fullsized":"2024-03-06T07:20:27.879763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:51.236302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201405+0000","last_change":"2024-03-06T07:20:20.859562+0000","last_active":"2024-03-06T07:20:28.201405+0000","last_peered":"2024-03-06T07:20:28.201405+0000","last_clean":"2024-03-06T07:20:28.201405+0000","last_became_active":"2024-03-06T07:20:20.859282+0000","last_became_peered":"2024-03-06T07:20:20.859282+0000","last_unstale":"2024-03-06T07:20:28.201405+0000","last_undegraded":"2024-03-06T07:20:28.201405+0000","last_fullsized":"2024-03-06T07:20:28.201405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:52.187127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701578+0000","last_change":"2024-03-06T07:20:21.863171+0000","last_active":"2024-03-06T07:20:28.701578+0000","last_peered":"2024-03-06T07:20:28.701578+0000","last_clean":"2024-03-06T07:20:28.701578+0000","last_became_active":"2024-03-06T07:20:21.862904+0000","last_became_peered":"2024-03-06T07:20:21.862904+0000","last_unstale":"2024-03-06T07:20:28.701578+0000","last_undegraded":"2024-03-06T07:20:28.701578+0000","last_fullsized":"2024-03-06T07:20:28.701578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:07.625355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880420+0000","last_change":"2024-03-06T07:20:25.888576+0000","last_active":"2024-03-06T07:20:27.880420+0000","last_peered":"2024-03-06T07:20:27.880420+0000","last_clean":"2024-03-06T07:20:27.880420+0000","last_became_active":"2024-03-06T07:20:25.886558+0000","last_became_peered":"2024-03-06T07:20:25.886558+0000","last_unstale":"2024-03-06T07:20:27.880420+0000","last_undegraded":"2024-03-06T07:20:27.880420+0000","last_fullsized":"2024-03-06T07:20:27.880420+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:27.800537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201157+0000","last_change":"2024-03-06T07:20:20.861724+0000","last_active":"2024-03-06T07:20:28.201157+0000","last_peered":"2024-03-06T07:20:28.201157+0000","last_clean":"2024-03-06T07:20:28.201157+0000","last_became_active":"2024-03-06T07:20:20.861432+0000","last_became_peered":"2024-03-06T07:20:20.861432+0000","last_unstale":"2024-03-06T07:20:28.201157+0000","last_undegraded":"2024-03-06T07:20:28.201157+0000","last_fullsized":"2024-03-06T07:20:28.201157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:43.621233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702026+0000","last_change":"2024-03-06T07:20:21.862894+0000","last_active":"2024-03-06T07:20:28.702026+0000","last_peered":"2024-03-06T07:20:28.702026+0000","last_clean":"2024-03-06T07:20:28.702026+0000","last_became_active":"2024-03-06T07:20:21.862624+0000","last_became_peered":"2024-03-06T07:20:21.862624+0000","last_unstale":"2024-03-06T07:20:28.702026+0000","last_undegraded":"2024-03-06T07:20:28.702026+0000","last_fullsized":"2024-03-06T07:20:28.702026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:53.701404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879914+0000","last_change":"2024-03-06T07:20:25.891479+0000","last_active":"2024-03-06T07:20:27.879914+0000","last_peered":"2024-03-06T07:20:27.879914+0000","last_clean":"2024-03-06T07:20:27.879914+0000","last_became_active":"2024-03-06T07:20:25.891124+0000","last_became_peered":"2024-03-06T07:20:25.891124+0000","last_unstale":"2024-03-06T07:20:27.879914+0000","last_undegraded":"2024-03-06T07:20:27.879914+0000","last_fullsized":"2024-03-06T07:20:27.879914+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:31.908538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701544+0000","last_change":"2024-03-06T07:20:21.861447+0000","last_active":"2024-03-06T07:20:28.701544+0000","last_peered":"2024-03-06T07:20:28.701544+0000","last_clean":"2024-03-06T07:20:28.701544+0000","last_became_active":"2024-03-06T07:20:21.861169+0000","last_became_peered":"2024-03-06T07:20:21.861169+0000","last_unstale":"2024-03-06T07:20:28.701544+0000","last_undegraded":"2024-03-06T07:20:28.701544+0000","last_fullsized":"2024-03-06T07:20:28.701544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:28.363605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201185+0000","last_change":"2024-03-06T07:20:20.858871+0000","last_active":"2024-03-06T07:20:28.201185+0000","last_peered":"2024-03-06T07:20:28.201185+0000","last_clean":"2024-03-06T07:20:28.201185+0000","last_became_active":"2024-03-06T07:20:20.858481+0000","last_became_peered":"2024-03-06T07:20:20.858481+0000","last_unstale":"2024-03-06T07:20:28.201185+0000","last_undegraded":"2024-03-06T07:20:28.201185+0000","last_fullsized":"2024-03-06T07:20:28.201185+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:58.697205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880550+0000","last_change":"2024-03-06T07:20:25.890028+0000","last_active":"2024-03-06T07:20:27.880550+0000","last_peered":"2024-03-06T07:20:27.880550+0000","last_clean":"2024-03-06T07:20:27.880550+0000","last_became_active":"2024-03-06T07:20:25.889634+0000","last_became_peered":"2024-03-06T07:20:25.889634+0000","last_unstale":"2024-03-06T07:20:27.880550+0000","last_undegraded":"2024-03-06T07:20:27.880550+0000","last_fullsized":"2024-03-06T07:20:27.880550+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:34.847448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880189+0000","last_change":"2024-03-06T07:20:21.863836+0000","last_active":"2024-03-06T07:20:27.880189+0000","last_peered":"2024-03-06T07:20:27.880189+0000","last_clean":"2024-03-06T07:20:27.880189+0000","last_became_active":"2024-03-06T07:20:21.863451+0000","last_became_peered":"2024-03-06T07:20:21.863451+0000","last_unstale":"2024-03-06T07:20:27.880189+0000","last_undegraded":"2024-03-06T07:20:27.880189+0000","last_fullsized":"2024-03-06T07:20:27.880189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:07.625355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365689+0000","last_change":"2024-03-06T07:20:20.847182+0000","last_active":"2024-03-06T07:20:28.365689+0000","last_peered":"2024-03-06T07:20:28.365689+0000","last_clean":"2024-03-06T07:20:28.365689+0000","last_became_active":"2024-03-06T07:20:20.846875+0000","last_became_peered":"2024-03-06T07:20:20.846875+0000","last_unstale":"2024-03-06T07:20:28.365689+0000","last_undegraded":"2024-03-06T07:20:28.365689+0000","last_fullsized":"2024-03-06T07:20:28.365689+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:47.332802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880571+0000","last_change":"2024-03-06T07:20:25.890712+0000","last_active":"2024-03-06T07:20:27.880571+0000","last_peered":"2024-03-06T07:20:27.880571+0000","last_clean":"2024-03-06T07:20:27.880571+0000","last_became_active":"2024-03-06T07:20:25.889840+0000","last_became_peered":"2024-03-06T07:20:25.889840+0000","last_unstale":"2024-03-06T07:20:27.880571+0000","last_undegraded":"2024-03-06T07:20:27.880571+0000","last_fullsized":"2024-03-06T07:20:27.880571+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:46.450423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879843+0000","last_change":"2024-03-06T07:20:21.865384+0000","last_active":"2024-03-06T07:20:27.879843+0000","last_peered":"2024-03-06T07:20:27.879843+0000","last_clean":"2024-03-06T07:20:27.879843+0000","last_became_active":"2024-03-06T07:20:21.865118+0000","last_became_peered":"2024-03-06T07:20:21.865118+0000","last_unstale":"2024-03-06T07:20:27.879843+0000","last_undegraded":"2024-03-06T07:20:27.879843+0000","last_fullsized":"2024-03-06T07:20:27.879843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:36.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365440+0000","last_change":"2024-03-06T07:20:20.856580+0000","last_active":"2024-03-06T07:20:28.365440+0000","last_peered":"2024-03-06T07:20:28.365440+0000","last_clean":"2024-03-06T07:20:28.365440+0000","last_became_active":"2024-03-06T07:20:20.856117+0000","last_became_peered":"2024-03-06T07:20:20.856117+0000","last_unstale":"2024-03-06T07:20:28.365440+0000","last_undegraded":"2024-03-06T07:20:28.365440+0000","last_fullsized":"2024-03-06T07:20:28.365440+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:43.621233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366340+0000","last_change":"2024-03-06T07:20:26.502055+0000","last_active":"2024-03-06T07:20:28.366340+0000","last_peered":"2024-03-06T07:20:28.366340+0000","last_clean":"2024-03-06T07:20:28.366340+0000","last_became_active":"2024-03-06T07:20:26.501689+0000","last_became_peered":"2024-03-06T07:20:26.501689+0000","last_unstale":"2024-03-06T07:20:28.366340+0000","last_undegraded":"2024-03-06T07:20:28.366340+0000","last_fullsized":"2024-03-06T07:20:28.366340+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:34.847448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880426+0000","last_change":"2024-03-06T07:20:21.859723+0000","last_active":"2024-03-06T07:20:27.880426+0000","last_peered":"2024-03-06T07:20:27.880426+0000","last_clean":"2024-03-06T07:20:27.880426+0000","last_became_active":"2024-03-06T07:20:21.859392+0000","last_became_peered":"2024-03-06T07:20:21.859392+0000","last_unstale":"2024-03-06T07:20:27.880426+0000","last_undegraded":"2024-03-06T07:20:27.880426+0000","last_fullsized":"2024-03-06T07:20:27.880426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:59.701834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.200895+0000","last_change":"2024-03-06T07:20:20.860602+0000","last_active":"2024-03-06T07:20:28.200895+0000","last_peered":"2024-03-06T07:20:28.200895+0000","last_clean":"2024-03-06T07:20:28.200895+0000","last_became_active":"2024-03-06T07:20:20.860328+0000","last_became_peered":"2024-03-06T07:20:20.860328+0000","last_unstale":"2024-03-06T07:20:28.200895+0000","last_undegraded":"2024-03-06T07:20:28.200895+0000","last_fullsized":"2024-03-06T07:20:28.200895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:25.763288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880136+0000","last_change":"2024-03-06T07:20:25.889087+0000","last_active":"2024-03-06T07:20:27.880136+0000","last_peered":"2024-03-06T07:20:27.880136+0000","last_clean":"2024-03-06T07:20:27.880136+0000","last_became_active":"2024-03-06T07:20:25.888562+0000","last_became_peered":"2024-03-06T07:20:25.888562+0000","last_unstale":"2024-03-06T07:20:27.880136+0000","last_undegraded":"2024-03-06T07:20:27.880136+0000","last_fullsized":"2024-03-06T07:20:27.880136+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:22.654122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701536+0000","last_change":"2024-03-06T07:20:21.857055+0000","last_active":"2024-03-06T07:20:28.701536+0000","last_peered":"2024-03-06T07:20:28.701536+0000","last_clean":"2024-03-06T07:20:28.701536+0000","last_became_active":"2024-03-06T07:20:21.856743+0000","last_became_peered":"2024-03-06T07:20:21.856743+0000","last_unstale":"2024-03-06T07:20:28.701536+0000","last_undegraded":"2024-03-06T07:20:28.701536+0000","last_fullsized":"2024-03-06T07:20:28.701536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:20.911597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702767+0000","last_change":"2024-03-06T07:20:20.858020+0000","last_active":"2024-03-06T07:20:28.702767+0000","last_peered":"2024-03-06T07:20:28.702767+0000","last_clean":"2024-03-06T07:20:28.702767+0000","last_became_active":"2024-03-06T07:20:20.857748+0000","last_became_peered":"2024-03-06T07:20:20.857748+0000","last_unstale":"2024-03-06T07:20:28.702767+0000","last_undegraded":"2024-03-06T07:20:28.702767+0000","last_fullsized":"2024-03-06T07:20:28.702767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:47.332802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880011+0000","last_change":"2024-03-06T07:20:26.499973+0000","last_active":"2024-03-06T07:20:27.880011+0000","last_peered":"2024-03-06T07:20:27.880011+0000","last_clean":"2024-03-06T07:20:27.880011+0000","last_became_active":"2024-03-06T07:20:26.499480+0000","last_became_peered":"2024-03-06T07:20:26.499480+0000","last_unstale":"2024-03-06T07:20:27.880011+0000","last_undegraded":"2024-03-06T07:20:27.880011+0000","last_fullsized":"2024-03-06T07:20:27.880011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:55.146213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880824+0000","last_change":"2024-03-06T07:20:21.858239+0000","last_active":"2024-03-06T07:20:27.880824+0000","last_peered":"2024-03-06T07:20:27.880824+0000","last_clean":"2024-03-06T07:20:27.880824+0000","last_became_active":"2024-03-06T07:20:21.857896+0000","last_became_peered":"2024-03-06T07:20:21.857896+0000","last_unstale":"2024-03-06T07:20:27.880824+0000","last_undegraded":"2024-03-06T07:20:27.880824+0000","last_fullsized":"2024-03-06T07:20:27.880824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366190+0000","last_change":"2024-03-06T07:20:20.853233+0000","last_active":"2024-03-06T07:20:28.366190+0000","last_peered":"2024-03-06T07:20:28.366190+0000","last_clean":"2024-03-06T07:20:28.366190+0000","last_became_active":"2024-03-06T07:20:20.852941+0000","last_became_peered":"2024-03-06T07:20:20.852941+0000","last_unstale":"2024-03-06T07:20:28.366190+0000","last_undegraded":"2024-03-06T07:20:28.366190+0000","last_fullsized":"2024-03-06T07:20:28.366190+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:25.763288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879421+0000","last_change":"2024-03-06T07:20:26.499314+0000","last_active":"2024-03-06T07:20:27.879421+0000","last_peered":"2024-03-06T07:20:27.879421+0000","last_clean":"2024-03-06T07:20:27.879421+0000","last_became_active":"2024-03-06T07:20:26.498625+0000","last_became_peered":"2024-03-06T07:20:26.498625+0000","last_unstale":"2024-03-06T07:20:27.879421+0000","last_undegraded":"2024-03-06T07:20:27.879421+0000","last_fullsized":"2024-03-06T07:20:27.879421+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:05.727952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365917+0000","last_change":"2024-03-06T07:20:21.861526+0000","last_active":"2024-03-06T07:20:28.365917+0000","last_peered":"2024-03-06T07:20:28.365917+0000","last_clean":"2024-03-06T07:20:28.365917+0000","last_became_active":"2024-03-06T07:20:21.861256+0000","last_became_peered":"2024-03-06T07:20:21.861256+0000","last_unstale":"2024-03-06T07:20:28.365917+0000","last_undegraded":"2024-03-06T07:20:28.365917+0000","last_fullsized":"2024-03-06T07:20:28.365917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.423089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201484+0000","last_change":"2024-03-06T07:20:20.858234+0000","last_active":"2024-03-06T07:20:28.201484+0000","last_peered":"2024-03-06T07:20:28.201484+0000","last_clean":"2024-03-06T07:20:28.201484+0000","last_became_active":"2024-03-06T07:20:20.857910+0000","last_became_peered":"2024-03-06T07:20:20.857910+0000","last_unstale":"2024-03-06T07:20:28.201484+0000","last_undegraded":"2024-03-06T07:20:28.201484+0000","last_fullsized":"2024-03-06T07:20:28.201484+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:04.756600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701326+0000","last_change":"2024-03-06T07:20:26.498727+0000","last_active":"2024-03-06T07:20:28.701326+0000","last_peered":"2024-03-06T07:20:28.701326+0000","last_clean":"2024-03-06T07:20:28.701326+0000","last_became_active":"2024-03-06T07:20:26.498272+0000","last_became_peered":"2024-03-06T07:20:26.498272+0000","last_unstale":"2024-03-06T07:20:28.701326+0000","last_undegraded":"2024-03-06T07:20:28.701326+0000","last_fullsized":"2024-03-06T07:20:28.701326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:28.909072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702156+0000","last_change":"2024-03-06T07:20:21.868324+0000","last_active":"2024-03-06T07:20:28.702156+0000","last_peered":"2024-03-06T07:20:28.702156+0000","last_clean":"2024-03-06T07:20:28.702156+0000","last_became_active":"2024-03-06T07:20:21.868032+0000","last_became_peered":"2024-03-06T07:20:21.868032+0000","last_unstale":"2024-03-06T07:20:28.702156+0000","last_undegraded":"2024-03-06T07:20:28.702156+0000","last_fullsized":"2024-03-06T07:20:28.702156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:17.299208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366009+0000","last_change":"2024-03-06T07:20:20.851073+0000","last_active":"2024-03-06T07:20:28.366009+0000","last_peered":"2024-03-06T07:20:28.366009+0000","last_clean":"2024-03-06T07:20:28.366009+0000","last_became_active":"2024-03-06T07:20:20.850774+0000","last_became_peered":"2024-03-06T07:20:20.850774+0000","last_unstale":"2024-03-06T07:20:28.366009+0000","last_undegraded":"2024-03-06T07:20:28.366009+0000","last_fullsized":"2024-03-06T07:20:28.366009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:15.716891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365536+0000","last_change":"2024-03-06T07:20:25.895118+0000","last_active":"2024-03-06T07:20:28.365536+0000","last_peered":"2024-03-06T07:20:28.365536+0000","last_clean":"2024-03-06T07:20:28.365536+0000","last_became_active":"2024-03-06T07:20:25.894354+0000","last_became_peered":"2024-03-06T07:20:25.894354+0000","last_unstale":"2024-03-06T07:20:28.365536+0000","last_undegraded":"2024-03-06T07:20:28.365536+0000","last_fullsized":"2024-03-06T07:20:28.365536+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:28.909072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880665+0000","last_change":"2024-03-06T07:20:21.861910+0000","last_active":"2024-03-06T07:20:27.880665+0000","last_peered":"2024-03-06T07:20:27.880665+0000","last_clean":"2024-03-06T07:20:27.880665+0000","last_became_active":"2024-03-06T07:20:21.861659+0000","last_became_peered":"2024-03-06T07:20:21.861659+0000","last_unstale":"2024-03-06T07:20:27.880665+0000","last_undegraded":"2024-03-06T07:20:27.880665+0000","last_fullsized":"2024-03-06T07:20:27.880665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:44.625861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701886+0000","last_change":"2024-03-06T07:20:20.849833+0000","last_active":"2024-03-06T07:20:28.701886+0000","last_peered":"2024-03-06T07:20:28.701886+0000","last_clean":"2024-03-06T07:20:28.701886+0000","last_became_active":"2024-03-06T07:20:20.849563+0000","last_became_peered":"2024-03-06T07:20:20.849563+0000","last_unstale":"2024-03-06T07:20:28.701886+0000","last_undegraded":"2024-03-06T07:20:28.701886+0000","last_fullsized":"2024-03-06T07:20:28.701886+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:43.621233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366005+0000","last_change":"2024-03-06T07:20:25.897100+0000","last_active":"2024-03-06T07:20:28.366005+0000","last_peered":"2024-03-06T07:20:28.366005+0000","last_clean":"2024-03-06T07:20:28.366005+0000","last_became_active":"2024-03-06T07:20:25.896658+0000","last_became_peered":"2024-03-06T07:20:25.896658+0000","last_unstale":"2024-03-06T07:20:28.366005+0000","last_undegraded":"2024-03-06T07:20:28.366005+0000","last_fullsized":"2024-03-06T07:20:28.366005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:27.800537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365526+0000","last_change":"2024-03-06T07:20:21.864469+0000","last_active":"2024-03-06T07:20:28.365526+0000","last_peered":"2024-03-06T07:20:28.365526+0000","last_clean":"2024-03-06T07:20:28.365526+0000","last_became_active":"2024-03-06T07:20:21.864141+0000","last_became_peered":"2024-03-06T07:20:21.864141+0000","last_unstale":"2024-03-06T07:20:28.365526+0000","last_undegraded":"2024-03-06T07:20:28.365526+0000","last_fullsized":"2024-03-06T07:20:28.365526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:17.903391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.878975+0000","last_change":"2024-03-06T07:20:20.845652+0000","last_active":"2024-03-06T07:20:27.878975+0000","last_peered":"2024-03-06T07:20:27.878975+0000","last_clean":"2024-03-06T07:20:27.878975+0000","last_became_active":"2024-03-06T07:20:20.845350+0000","last_became_peered":"2024-03-06T07:20:20.845350+0000","last_unstale":"2024-03-06T07:20:27.878975+0000","last_undegraded":"2024-03-06T07:20:27.878975+0000","last_fullsized":"2024-03-06T07:20:27.878975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:53.997041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366358+0000","last_change":"2024-03-06T07:20:25.898164+0000","last_active":"2024-03-06T07:20:28.366358+0000","last_peered":"2024-03-06T07:20:28.366358+0000","last_clean":"2024-03-06T07:20:28.366358+0000","last_became_active":"2024-03-06T07:20:25.897755+0000","last_became_peered":"2024-03-06T07:20:25.897755+0000","last_unstale":"2024-03-06T07:20:28.366358+0000","last_undegraded":"2024-03-06T07:20:28.366358+0000","last_fullsized":"2024-03-06T07:20:28.366358+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:46.450423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701002+0000","last_change":"2024-03-06T07:20:21.850395+0000","last_active":"2024-03-06T07:20:28.701002+0000","last_peered":"2024-03-06T07:20:28.701002+0000","last_clean":"2024-03-06T07:20:28.701002+0000","last_became_active":"2024-03-06T07:20:21.850117+0000","last_became_peered":"2024-03-06T07:20:21.850117+0000","last_unstale":"2024-03-06T07:20:28.701002+0000","last_undegraded":"2024-03-06T07:20:28.701002+0000","last_fullsized":"2024-03-06T07:20:28.701002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:42.612183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.200831+0000","last_change":"2024-03-06T07:20:20.857982+0000","last_active":"2024-03-06T07:20:28.200831+0000","last_peered":"2024-03-06T07:20:28.200831+0000","last_clean":"2024-03-06T07:20:28.200831+0000","last_became_active":"2024-03-06T07:20:20.857726+0000","last_became_peered":"2024-03-06T07:20:20.857726+0000","last_unstale":"2024-03-06T07:20:28.200831+0000","last_undegraded":"2024-03-06T07:20:28.200831+0000","last_fullsized":"2024-03-06T07:20:28.200831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:24.258322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701481+0000","last_change":"2024-03-06T07:20:26.502037+0000","last_active":"2024-03-06T07:20:28.701481+0000","last_peered":"2024-03-06T07:20:28.701481+0000","last_clean":"2024-03-06T07:20:28.701481+0000","last_became_active":"2024-03-06T07:20:26.501666+0000","last_became_peered":"2024-03-06T07:20:26.501666+0000","last_unstale":"2024-03-06T07:20:28.701481+0000","last_undegraded":"2024-03-06T07:20:28.701481+0000","last_fullsized":"2024-03-06T07:20:28.701481+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:27.800537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880611+0000","last_change":"2024-03-06T07:20:21.858533+0000","last_active":"2024-03-06T07:20:27.880611+0000","last_peered":"2024-03-06T07:20:27.880611+0000","last_clean":"2024-03-06T07:20:27.880611+0000","last_became_active":"2024-03-06T07:20:21.857499+0000","last_became_peered":"2024-03-06T07:20:21.857499+0000","last_unstale":"2024-03-06T07:20:27.880611+0000","last_undegraded":"2024-03-06T07:20:27.880611+0000","last_fullsized":"2024-03-06T07:20:27.880611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:27.196530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701468+0000","last_change":"2024-03-06T07:20:20.858802+0000","last_active":"2024-03-06T07:20:28.701468+0000","last_peered":"2024-03-06T07:20:28.701468+0000","last_clean":"2024-03-06T07:20:28.701468+0000","last_became_active":"2024-03-06T07:20:20.857457+0000","last_became_peered":"2024-03-06T07:20:20.857457+0000","last_unstale":"2024-03-06T07:20:28.701468+0000","last_undegraded":"2024-03-06T07:20:28.701468+0000","last_fullsized":"2024-03-06T07:20:28.701468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:57.497682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":179,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201006+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:20:28.201006+0000","last_peered":"2024-03-06T07:20:28.201006+0000","last_clean":"2024-03-06T07:20:28.201006+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:20:28.201006+0000","last_undegraded":"2024-03-06T07:20:28.201006+0000","last_fullsized":"2024-03-06T07:20:28.201006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702367+0000","last_change":"2024-03-06T07:20:26.502317+0000","last_active":"2024-03-06T07:20:28.702367+0000","last_peered":"2024-03-06T07:20:28.702367+0000","last_clean":"2024-03-06T07:20:28.702367+0000","last_became_active":"2024-03-06T07:20:26.501807+0000","last_became_peered":"2024-03-06T07:20:26.501807+0000","last_unstale":"2024-03-06T07:20:28.702367+0000","last_undegraded":"2024-03-06T07:20:28.702367+0000","last_fullsized":"2024-03-06T07:20:28.702367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:30.890766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201288+0000","last_change":"2024-03-06T07:20:21.855669+0000","last_active":"2024-03-06T07:20:28.201288+0000","last_peered":"2024-03-06T07:20:28.201288+0000","last_clean":"2024-03-06T07:20:28.201288+0000","last_became_active":"2024-03-06T07:20:21.855154+0000","last_became_peered":"2024-03-06T07:20:21.855154+0000","last_unstale":"2024-03-06T07:20:28.201288+0000","last_undegraded":"2024-03-06T07:20:28.201288+0000","last_fullsized":"2024-03-06T07:20:28.201288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:26.855742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879323+0000","last_change":"2024-03-06T07:20:20.852702+0000","last_active":"2024-03-06T07:20:27.879323+0000","last_peered":"2024-03-06T07:20:27.879323+0000","last_clean":"2024-03-06T07:20:27.879323+0000","last_became_active":"2024-03-06T07:20:20.852434+0000","last_became_peered":"2024-03-06T07:20:20.852434+0000","last_unstale":"2024-03-06T07:20:27.879323+0000","last_undegraded":"2024-03-06T07:20:27.879323+0000","last_fullsized":"2024-03-06T07:20:27.879323+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:44.080359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702280+0000","last_change":"2024-03-06T07:20:26.499169+0000","last_active":"2024-03-06T07:20:28.702280+0000","last_peered":"2024-03-06T07:20:28.702280+0000","last_clean":"2024-03-06T07:20:28.702280+0000","last_became_active":"2024-03-06T07:20:26.498541+0000","last_became_peered":"2024-03-06T07:20:26.498541+0000","last_unstale":"2024-03-06T07:20:28.702280+0000","last_undegraded":"2024-03-06T07:20:28.702280+0000","last_fullsized":"2024-03-06T07:20:28.702280+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:49.037215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701682+0000","last_change":"2024-03-06T07:20:21.857920+0000","last_active":"2024-03-06T07:20:28.701682+0000","last_peered":"2024-03-06T07:20:28.701682+0000","last_clean":"2024-03-06T07:20:28.701682+0000","last_became_active":"2024-03-06T07:20:21.857645+0000","last_became_peered":"2024-03-06T07:20:21.857645+0000","last_unstale":"2024-03-06T07:20:28.701682+0000","last_undegraded":"2024-03-06T07:20:28.701682+0000","last_fullsized":"2024-03-06T07:20:28.701682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:47.201278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366028+0000","last_change":"2024-03-06T07:20:20.857009+0000","last_active":"2024-03-06T07:20:28.366028+0000","last_peered":"2024-03-06T07:20:28.366028+0000","last_clean":"2024-03-06T07:20:28.366028+0000","last_became_active":"2024-03-06T07:20:20.856701+0000","last_became_peered":"2024-03-06T07:20:20.856701+0000","last_unstale":"2024-03-06T07:20:28.366028+0000","last_undegraded":"2024-03-06T07:20:28.366028+0000","last_fullsized":"2024-03-06T07:20:28.366028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:24.258322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701482+0000","last_change":"2024-03-06T07:20:25.887514+0000","last_active":"2024-03-06T07:20:28.701482+0000","last_peered":"2024-03-06T07:20:28.701482+0000","last_clean":"2024-03-06T07:20:28.701482+0000","last_became_active":"2024-03-06T07:20:25.887125+0000","last_became_peered":"2024-03-06T07:20:25.887125+0000","last_unstale":"2024-03-06T07:20:28.701482+0000","last_undegraded":"2024-03-06T07:20:28.701482+0000","last_fullsized":"2024-03-06T07:20:28.701482+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:34.847448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201678+0000","last_change":"2024-03-06T07:20:21.851028+0000","last_active":"2024-03-06T07:20:28.201678+0000","last_peered":"2024-03-06T07:20:28.201678+0000","last_clean":"2024-03-06T07:20:28.201678+0000","last_became_active":"2024-03-06T07:20:21.850762+0000","last_became_peered":"2024-03-06T07:20:21.850762+0000","last_unstale":"2024-03-06T07:20:28.201678+0000","last_undegraded":"2024-03-06T07:20:28.201678+0000","last_fullsized":"2024-03-06T07:20:28.201678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:27.873513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879685+0000","last_change":"2024-03-06T07:20:20.849572+0000","last_active":"2024-03-06T07:20:27.879685+0000","last_peered":"2024-03-06T07:20:27.879685+0000","last_clean":"2024-03-06T07:20:27.879685+0000","last_became_active":"2024-03-06T07:20:20.849266+0000","last_became_peered":"2024-03-06T07:20:20.849266+0000","last_unstale":"2024-03-06T07:20:27.879685+0000","last_undegraded":"2024-03-06T07:20:27.879685+0000","last_fullsized":"2024-03-06T07:20:27.879685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:47.332802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701273+0000","last_change":"2024-03-06T07:20:26.502258+0000","last_active":"2024-03-06T07:20:28.701273+0000","last_peered":"2024-03-06T07:20:28.701273+0000","last_clean":"2024-03-06T07:20:28.701273+0000","last_became_active":"2024-03-06T07:20:26.501819+0000","last_became_peered":"2024-03-06T07:20:26.501819+0000","last_unstale":"2024-03-06T07:20:28.701273+0000","last_undegraded":"2024-03-06T07:20:28.701273+0000","last_fullsized":"2024-03-06T07:20:28.701273+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:34.847448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201451+0000","last_change":"2024-03-06T07:20:21.844939+0000","last_active":"2024-03-06T07:20:28.201451+0000","last_peered":"2024-03-06T07:20:28.201451+0000","last_clean":"2024-03-06T07:20:28.201451+0000","last_became_active":"2024-03-06T07:20:21.844652+0000","last_became_peered":"2024-03-06T07:20:21.844652+0000","last_unstale":"2024-03-06T07:20:28.201451+0000","last_undegraded":"2024-03-06T07:20:28.201451+0000","last_fullsized":"2024-03-06T07:20:28.201451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:07.625355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880344+0000","last_change":"2024-03-06T07:20:20.850150+0000","last_active":"2024-03-06T07:20:27.880344+0000","last_peered":"2024-03-06T07:20:27.880344+0000","last_clean":"2024-03-06T07:20:27.880344+0000","last_became_active":"2024-03-06T07:20:20.849834+0000","last_became_peered":"2024-03-06T07:20:20.849834+0000","last_unstale":"2024-03-06T07:20:27.880344+0000","last_undegraded":"2024-03-06T07:20:27.880344+0000","last_fullsized":"2024-03-06T07:20:27.880344+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:56.227109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879732+0000","last_change":"2024-03-06T07:20:25.888682+0000","last_active":"2024-03-06T07:20:27.879732+0000","last_peered":"2024-03-06T07:20:27.879732+0000","last_clean":"2024-03-06T07:20:27.879732+0000","last_became_active":"2024-03-06T07:20:25.888259+0000","last_became_peered":"2024-03-06T07:20:25.888259+0000","last_unstale":"2024-03-06T07:20:27.879732+0000","last_undegraded":"2024-03-06T07:20:27.879732+0000","last_fullsized":"2024-03-06T07:20:27.879732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:57.736429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201088+0000","last_change":"2024-03-06T07:20:21.851600+0000","last_active":"2024-03-06T07:20:28.201088+0000","last_peered":"2024-03-06T07:20:28.201088+0000","last_clean":"2024-03-06T07:20:28.201088+0000","last_became_active":"2024-03-06T07:20:21.851347+0000","last_became_peered":"2024-03-06T07:20:21.851347+0000","last_unstale":"2024-03-06T07:20:28.201088+0000","last_undegraded":"2024-03-06T07:20:28.201088+0000","last_fullsized":"2024-03-06T07:20:28.201088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:03.883136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.878780+0000","last_change":"2024-03-06T07:20:20.847128+0000","last_active":"2024-03-06T07:20:27.878780+0000","last_peered":"2024-03-06T07:20:27.878780+0000","last_clean":"2024-03-06T07:20:27.878780+0000","last_became_active":"2024-03-06T07:20:20.846822+0000","last_became_peered":"2024-03-06T07:20:20.846822+0000","last_unstale":"2024-03-06T07:20:27.878780+0000","last_undegraded":"2024-03-06T07:20:27.878780+0000","last_fullsized":"2024-03-06T07:20:27.878780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:57.497682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701347+0000","last_change":"2024-03-06T07:20:25.888397+0000","last_active":"2024-03-06T07:20:28.701347+0000","last_peered":"2024-03-06T07:20:28.701347+0000","last_clean":"2024-03-06T07:20:28.701347+0000","last_became_active":"2024-03-06T07:20:25.888017+0000","last_became_peered":"2024-03-06T07:20:25.888017+0000","last_unstale":"2024-03-06T07:20:28.701347+0000","last_undegraded":"2024-03-06T07:20:28.701347+0000","last_fullsized":"2024-03-06T07:20:28.701347+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:22.654122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702085+0000","last_change":"2024-03-06T07:20:21.863218+0000","last_active":"2024-03-06T07:20:28.702085+0000","last_peered":"2024-03-06T07:20:28.702085+0000","last_clean":"2024-03-06T07:20:28.702085+0000","last_became_active":"2024-03-06T07:20:21.862943+0000","last_became_peered":"2024-03-06T07:20:21.862943+0000","last_unstale":"2024-03-06T07:20:28.702085+0000","last_undegraded":"2024-03-06T07:20:28.702085+0000","last_fullsized":"2024-03-06T07:20:28.702085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:17.903391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701478+0000","last_change":"2024-03-06T07:20:20.855196+0000","last_active":"2024-03-06T07:20:28.701478+0000","last_peered":"2024-03-06T07:20:28.701478+0000","last_clean":"2024-03-06T07:20:28.701478+0000","last_became_active":"2024-03-06T07:20:20.854047+0000","last_became_peered":"2024-03-06T07:20:20.854047+0000","last_unstale":"2024-03-06T07:20:28.701478+0000","last_undegraded":"2024-03-06T07:20:28.701478+0000","last_fullsized":"2024-03-06T07:20:28.701478+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:15.716891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880771+0000","last_change":"2024-03-06T07:20:26.502104+0000","last_active":"2024-03-06T07:20:27.880771+0000","last_peered":"2024-03-06T07:20:27.880771+0000","last_clean":"2024-03-06T07:20:27.880771+0000","last_became_active":"2024-03-06T07:20:26.501701+0000","last_became_peered":"2024-03-06T07:20:26.501701+0000","last_unstale":"2024-03-06T07:20:27.880771+0000","last_undegraded":"2024-03-06T07:20:27.880771+0000","last_fullsized":"2024-03-06T07:20:27.880771+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:13.344053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201065+0000","last_change":"2024-03-06T07:20:21.850971+0000","last_active":"2024-03-06T07:20:28.201065+0000","last_peered":"2024-03-06T07:20:28.201065+0000","last_clean":"2024-03-06T07:20:28.201065+0000","last_became_active":"2024-03-06T07:20:21.850700+0000","last_became_peered":"2024-03-06T07:20:21.850700+0000","last_unstale":"2024-03-06T07:20:28.201065+0000","last_undegraded":"2024-03-06T07:20:28.201065+0000","last_fullsized":"2024-03-06T07:20:28.201065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:17.299208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701777+0000","last_change":"2024-03-06T07:20:20.856747+0000","last_active":"2024-03-06T07:20:28.701777+0000","last_peered":"2024-03-06T07:20:28.701777+0000","last_clean":"2024-03-06T07:20:28.701777+0000","last_became_active":"2024-03-06T07:20:20.856431+0000","last_became_peered":"2024-03-06T07:20:20.856431+0000","last_unstale":"2024-03-06T07:20:28.701777+0000","last_undegraded":"2024-03-06T07:20:28.701777+0000","last_fullsized":"2024-03-06T07:20:28.701777+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:15.716891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880386+0000","last_change":"2024-03-06T07:20:26.498675+0000","last_active":"2024-03-06T07:20:27.880386+0000","last_peered":"2024-03-06T07:20:27.880386+0000","last_clean":"2024-03-06T07:20:27.880386+0000","last_became_active":"2024-03-06T07:20:26.498199+0000","last_became_peered":"2024-03-06T07:20:26.498199+0000","last_unstale":"2024-03-06T07:20:27.880386+0000","last_undegraded":"2024-03-06T07:20:27.880386+0000","last_fullsized":"2024-03-06T07:20:27.880386+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:53.210084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701227+0000","last_change":"2024-03-06T07:20:21.845586+0000","last_active":"2024-03-06T07:20:28.701227+0000","last_peered":"2024-03-06T07:20:28.701227+0000","last_clean":"2024-03-06T07:20:28.701227+0000","last_became_active":"2024-03-06T07:20:21.845311+0000","last_became_peered":"2024-03-06T07:20:21.845311+0000","last_unstale":"2024-03-06T07:20:28.701227+0000","last_undegraded":"2024-03-06T07:20:28.701227+0000","last_fullsized":"2024-03-06T07:20:28.701227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:58.153137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701540+0000","last_change":"2024-03-06T07:20:20.849649+0000","last_active":"2024-03-06T07:20:28.701540+0000","last_peered":"2024-03-06T07:20:28.701540+0000","last_clean":"2024-03-06T07:20:28.701540+0000","last_became_active":"2024-03-06T07:20:20.849199+0000","last_became_peered":"2024-03-06T07:20:20.849199+0000","last_unstale":"2024-03-06T07:20:28.701540+0000","last_undegraded":"2024-03-06T07:20:28.701540+0000","last_fullsized":"2024-03-06T07:20:28.701540+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:22.023999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702184+0000","last_change":"2024-03-06T07:20:26.499070+0000","last_active":"2024-03-06T07:20:28.702184+0000","last_peered":"2024-03-06T07:20:28.702184+0000","last_clean":"2024-03-06T07:20:28.702184+0000","last_became_active":"2024-03-06T07:20:26.498601+0000","last_became_peered":"2024-03-06T07:20:26.498601+0000","last_unstale":"2024-03-06T07:20:28.702184+0000","last_undegraded":"2024-03-06T07:20:28.702184+0000","last_fullsized":"2024-03-06T07:20:28.702184+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:46.450423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201193+0000","last_change":"2024-03-06T07:20:21.857736+0000","last_active":"2024-03-06T07:20:28.201193+0000","last_peered":"2024-03-06T07:20:28.201193+0000","last_clean":"2024-03-06T07:20:28.201193+0000","last_became_active":"2024-03-06T07:20:21.857492+0000","last_became_peered":"2024-03-06T07:20:21.857492+0000","last_unstale":"2024-03-06T07:20:28.201193+0000","last_undegraded":"2024-03-06T07:20:28.201193+0000","last_fullsized":"2024-03-06T07:20:28.201193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:45.002191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":33,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:31.174467+0000","last_change":"2024-03-06T07:20:20.856316+0000","last_active":"2024-03-06T07:20:31.174467+0000","last_peered":"2024-03-06T07:20:31.174467+0000","last_clean":"2024-03-06T07:20:31.174467+0000","last_became_active":"2024-03-06T07:20:20.855994+0000","last_became_peered":"2024-03-06T07:20:20.855994+0000","last_unstale":"2024-03-06T07:20:31.174467+0000","last_undegraded":"2024-03-06T07:20:31.174467+0000","last_fullsized":"2024-03-06T07:20:31.174467+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:53.997041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365632+0000","last_change":"2024-03-06T07:20:25.895379+0000","last_active":"2024-03-06T07:20:28.365632+0000","last_peered":"2024-03-06T07:20:28.365632+0000","last_clean":"2024-03-06T07:20:28.365632+0000","last_became_active":"2024-03-06T07:20:25.894458+0000","last_became_peered":"2024-03-06T07:20:25.894458+0000","last_unstale":"2024-03-06T07:20:28.365632+0000","last_undegraded":"2024-03-06T07:20:28.365632+0000","last_fullsized":"2024-03-06T07:20:28.365632+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:22.654122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880548+0000","last_change":"2024-03-06T07:20:21.844398+0000","last_active":"2024-03-06T07:20:27.880548+0000","last_peered":"2024-03-06T07:20:27.880548+0000","last_clean":"2024-03-06T07:20:27.880548+0000","last_became_active":"2024-03-06T07:20:21.843839+0000","last_became_peered":"2024-03-06T07:20:21.843839+0000","last_unstale":"2024-03-06T07:20:27.880548+0000","last_undegraded":"2024-03-06T07:20:27.880548+0000","last_fullsized":"2024-03-06T07:20:27.880548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:25.262951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365950+0000","last_change":"2024-03-06T07:20:20.850772+0000","last_active":"2024-03-06T07:20:28.365950+0000","last_peered":"2024-03-06T07:20:28.365950+0000","last_clean":"2024-03-06T07:20:28.365950+0000","last_became_active":"2024-03-06T07:20:20.850490+0000","last_became_peered":"2024-03-06T07:20:20.850490+0000","last_unstale":"2024-03-06T07:20:28.365950+0000","last_undegraded":"2024-03-06T07:20:28.365950+0000","last_fullsized":"2024-03-06T07:20:28.365950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:58.697205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879970+0000","last_change":"2024-03-06T07:20:25.887556+0000","last_active":"2024-03-06T07:20:27.879970+0000","last_peered":"2024-03-06T07:20:27.879970+0000","last_clean":"2024-03-06T07:20:27.879970+0000","last_became_active":"2024-03-06T07:20:25.887175+0000","last_became_peered":"2024-03-06T07:20:25.887175+0000","last_unstale":"2024-03-06T07:20:27.879970+0000","last_undegraded":"2024-03-06T07:20:27.879970+0000","last_fullsized":"2024-03-06T07:20:27.879970+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:28.909072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701501+0000","last_change":"2024-03-06T07:20:21.856048+0000","last_active":"2024-03-06T07:20:28.701501+0000","last_peered":"2024-03-06T07:20:28.701501+0000","last_clean":"2024-03-06T07:20:28.701501+0000","last_became_active":"2024-03-06T07:20:21.854721+0000","last_became_peered":"2024-03-06T07:20:21.854721+0000","last_unstale":"2024-03-06T07:20:28.701501+0000","last_undegraded":"2024-03-06T07:20:28.701501+0000","last_fullsized":"2024-03-06T07:20:28.701501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:00.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702082+0000","last_change":"2024-03-06T07:20:20.859239+0000","last_active":"2024-03-06T07:20:28.702082+0000","last_peered":"2024-03-06T07:20:28.702082+0000","last_clean":"2024-03-06T07:20:28.702082+0000","last_became_active":"2024-03-06T07:20:20.858860+0000","last_became_peered":"2024-03-06T07:20:20.858860+0000","last_unstale":"2024-03-06T07:20:28.702082+0000","last_undegraded":"2024-03-06T07:20:28.702082+0000","last_fullsized":"2024-03-06T07:20:28.702082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:47.332802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879712+0000","last_change":"2024-03-06T07:20:25.890675+0000","last_active":"2024-03-06T07:20:27.879712+0000","last_peered":"2024-03-06T07:20:27.879712+0000","last_clean":"2024-03-06T07:20:27.879712+0000","last_became_active":"2024-03-06T07:20:25.890282+0000","last_became_peered":"2024-03-06T07:20:25.890282+0000","last_unstale":"2024-03-06T07:20:27.879712+0000","last_undegraded":"2024-03-06T07:20:27.879712+0000","last_fullsized":"2024-03-06T07:20:27.879712+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:10:13.387120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702028+0000","last_change":"2024-03-06T07:20:21.864960+0000","last_active":"2024-03-06T07:20:28.702028+0000","last_peered":"2024-03-06T07:20:28.702028+0000","last_clean":"2024-03-06T07:20:28.702028+0000","last_became_active":"2024-03-06T07:20:21.864687+0000","last_became_peered":"2024-03-06T07:20:21.864687+0000","last_unstale":"2024-03-06T07:20:28.702028+0000","last_undegraded":"2024-03-06T07:20:28.702028+0000","last_fullsized":"2024-03-06T07:20:28.702028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:27.196530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702647+0000","last_change":"2024-03-06T07:20:20.857187+0000","last_active":"2024-03-06T07:20:28.702647+0000","last_peered":"2024-03-06T07:20:28.702647+0000","last_clean":"2024-03-06T07:20:28.702647+0000","last_became_active":"2024-03-06T07:20:20.856898+0000","last_became_peered":"2024-03-06T07:20:20.856898+0000","last_unstale":"2024-03-06T07:20:28.702647+0000","last_undegraded":"2024-03-06T07:20:28.702647+0000","last_fullsized":"2024-03-06T07:20:28.702647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:57.497682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879802+0000","last_change":"2024-03-06T07:20:26.503331+0000","last_active":"2024-03-06T07:20:27.879802+0000","last_peered":"2024-03-06T07:20:27.879802+0000","last_clean":"2024-03-06T07:20:27.879802+0000","last_became_active":"2024-03-06T07:20:26.502770+0000","last_became_peered":"2024-03-06T07:20:26.502770+0000","last_unstale":"2024-03-06T07:20:27.879802+0000","last_undegraded":"2024-03-06T07:20:27.879802+0000","last_fullsized":"2024-03-06T07:20:27.879802+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:13.262016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701297+0000","last_change":"2024-03-06T07:20:26.502227+0000","last_active":"2024-03-06T07:20:28.701297+0000","last_peered":"2024-03-06T07:20:28.701297+0000","last_clean":"2024-03-06T07:20:28.701297+0000","last_became_active":"2024-03-06T07:20:26.501641+0000","last_became_peered":"2024-03-06T07:20:26.501641+0000","last_unstale":"2024-03-06T07:20:28.701297+0000","last_undegraded":"2024-03-06T07:20:28.701297+0000","last_fullsized":"2024-03-06T07:20:28.701297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:31.908538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365327+0000","last_change":"2024-03-06T07:20:21.866845+0000","last_active":"2024-03-06T07:20:28.365327+0000","last_peered":"2024-03-06T07:20:28.365327+0000","last_clean":"2024-03-06T07:20:28.365327+0000","last_became_active":"2024-03-06T07:20:21.866576+0000","last_became_peered":"2024-03-06T07:20:21.866576+0000","last_unstale":"2024-03-06T07:20:28.365327+0000","last_undegraded":"2024-03-06T07:20:28.365327+0000","last_fullsized":"2024-03-06T07:20:28.365327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:36.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701723+0000","last_change":"2024-03-06T07:20:20.853894+0000","last_active":"2024-03-06T07:20:28.701723+0000","last_peered":"2024-03-06T07:20:28.701723+0000","last_clean":"2024-03-06T07:20:28.701723+0000","last_became_active":"2024-03-06T07:20:20.853538+0000","last_became_peered":"2024-03-06T07:20:20.853538+0000","last_unstale":"2024-03-06T07:20:28.701723+0000","last_undegraded":"2024-03-06T07:20:28.701723+0000","last_fullsized":"2024-03-06T07:20:28.701723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:25.763288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702543+0000","last_change":"2024-03-06T07:20:25.894539+0000","last_active":"2024-03-06T07:20:28.702543+0000","last_peered":"2024-03-06T07:20:28.702543+0000","last_clean":"2024-03-06T07:20:28.702543+0000","last_became_active":"2024-03-06T07:20:25.894137+0000","last_became_peered":"2024-03-06T07:20:25.894137+0000","last_unstale":"2024-03-06T07:20:28.702543+0000","last_undegraded":"2024-03-06T07:20:28.702543+0000","last_fullsized":"2024-03-06T07:20:28.702543+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:57.736429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879760+0000","last_change":"2024-03-06T07:20:21.869266+0000","last_active":"2024-03-06T07:20:27.879760+0000","last_peered":"2024-03-06T07:20:27.879760+0000","last_clean":"2024-03-06T07:20:27.879760+0000","last_became_active":"2024-03-06T07:20:21.861979+0000","last_became_peered":"2024-03-06T07:20:21.861979+0000","last_unstale":"2024-03-06T07:20:27.879760+0000","last_undegraded":"2024-03-06T07:20:27.879760+0000","last_fullsized":"2024-03-06T07:20:27.879760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:17.903391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879911+0000","last_change":"2024-03-06T07:20:20.853365+0000","last_active":"2024-03-06T07:20:27.879911+0000","last_peered":"2024-03-06T07:20:27.879911+0000","last_clean":"2024-03-06T07:20:27.879911+0000","last_became_active":"2024-03-06T07:20:20.853065+0000","last_became_peered":"2024-03-06T07:20:20.853065+0000","last_unstale":"2024-03-06T07:20:27.879911+0000","last_undegraded":"2024-03-06T07:20:27.879911+0000","last_fullsized":"2024-03-06T07:20:27.879911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:53.997041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702584+0000","last_change":"2024-03-06T07:20:20.852681+0000","last_active":"2024-03-06T07:20:28.702584+0000","last_peered":"2024-03-06T07:20:28.702584+0000","last_clean":"2024-03-06T07:20:28.702584+0000","last_became_active":"2024-03-06T07:20:20.852406+0000","last_became_peered":"2024-03-06T07:20:20.852406+0000","last_unstale":"2024-03-06T07:20:28.702584+0000","last_undegraded":"2024-03-06T07:20:28.702584+0000","last_fullsized":"2024-03-06T07:20:28.702584+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:25.763288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701475+0000","last_change":"2024-03-06T07:20:21.854101+0000","last_active":"2024-03-06T07:20:28.701475+0000","last_peered":"2024-03-06T07:20:28.701475+0000","last_clean":"2024-03-06T07:20:28.701475+0000","last_became_active":"2024-03-06T07:20:21.853720+0000","last_became_peered":"2024-03-06T07:20:21.853720+0000","last_unstale":"2024-03-06T07:20:28.701475+0000","last_undegraded":"2024-03-06T07:20:28.701475+0000","last_fullsized":"2024-03-06T07:20:28.701475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.423089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879159+0000","last_change":"2024-03-06T07:20:25.889455+0000","last_active":"2024-03-06T07:20:27.879159+0000","last_peered":"2024-03-06T07:20:27.879159+0000","last_clean":"2024-03-06T07:20:27.879159+0000","last_became_active":"2024-03-06T07:20:25.889021+0000","last_became_peered":"2024-03-06T07:20:25.889021+0000","last_unstale":"2024-03-06T07:20:27.879159+0000","last_undegraded":"2024-03-06T07:20:27.879159+0000","last_fullsized":"2024-03-06T07:20:27.879159+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:46.647208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880275+0000","last_change":"2024-03-06T07:20:20.860219+0000","last_active":"2024-03-06T07:20:27.880275+0000","last_peered":"2024-03-06T07:20:27.880275+0000","last_clean":"2024-03-06T07:20:27.880275+0000","last_became_active":"2024-03-06T07:20:20.859862+0000","last_became_peered":"2024-03-06T07:20:20.859862+0000","last_unstale":"2024-03-06T07:20:27.880275+0000","last_undegraded":"2024-03-06T07:20:27.880275+0000","last_fullsized":"2024-03-06T07:20:27.880275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:19.824912+0000","last_clean_scrub_stamp":"2024-03-06T07:20:19.824912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:44.080359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879980+0000","last_change":"2024-03-06T07:20:21.858660+0000","last_active":"2024-03-06T07:20:27.879980+0000","last_peered":"2024-03-06T07:20:27.879980+0000","last_clean":"2024-03-06T07:20:27.879980+0000","last_became_active":"2024-03-06T07:20:21.858397+0000","last_became_peered":"2024-03-06T07:20:21.858397+0000","last_unstale":"2024-03-06T07:20:27.879980+0000","last_undegraded":"2024-03-06T07:20:27.879980+0000","last_fullsized":"2024-03-06T07:20:27.879980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:16.721519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880388+0000","last_change":"2024-03-06T07:20:25.889183+0000","last_active":"2024-03-06T07:20:27.880388+0000","last_peered":"2024-03-06T07:20:27.880388+0000","last_clean":"2024-03-06T07:20:27.880388+0000","last_became_active":"2024-03-06T07:20:25.888836+0000","last_became_peered":"2024-03-06T07:20:25.888836+0000","last_unstale":"2024-03-06T07:20:27.880388+0000","last_undegraded":"2024-03-06T07:20:27.880388+0000","last_fullsized":"2024-03-06T07:20:27.880388+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:24.864565+0000","last_clean_scrub_stamp":"2024-03-06T07:20:24.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:28.743175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701231+0000","last_change":"2024-03-06T07:20:21.863258+0000","last_active":"2024-03-06T07:20:28.701231+0000","last_peered":"2024-03-06T07:20:28.701231+0000","last_clean":"2024-03-06T07:20:28.701231+0000","last_became_active":"2024-03-06T07:20:21.862976+0000","last_became_peered":"2024-03-06T07:20:21.862976+0000","last_unstale":"2024-03-06T07:20:28.701231+0000","last_undegraded":"2024-03-06T07:20:28.701231+0000","last_fullsized":"2024-03-06T07:20:28.701231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.878887+0000","last_change":"2024-03-06T07:20:21.860263+0000","last_active":"2024-03-06T07:20:27.878887+0000","last_peered":"2024-03-06T07:20:27.878887+0000","last_clean":"2024-03-06T07:20:27.878887+0000","last_became_active":"2024-03-06T07:20:21.859978+0000","last_became_peered":"2024-03-06T07:20:21.859978+0000","last_unstale":"2024-03-06T07:20:27.878887+0000","last_undegraded":"2024-03-06T07:20:27.878887+0000","last_fullsized":"2024-03-06T07:20:27.878887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:59.701834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701885+0000","last_change":"2024-03-06T07:20:21.868562+0000","last_active":"2024-03-06T07:20:28.701885+0000","last_peered":"2024-03-06T07:20:28.701885+0000","last_clean":"2024-03-06T07:20:28.701885+0000","last_became_active":"2024-03-06T07:20:21.868239+0000","last_became_peered":"2024-03-06T07:20:21.868239+0000","last_unstale":"2024-03-06T07:20:28.701885+0000","last_undegraded":"2024-03-06T07:20:28.701885+0000","last_fullsized":"2024-03-06T07:20:28.701885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:45.002191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701377+0000","last_change":"2024-03-06T07:20:21.860455+0000","last_active":"2024-03-06T07:20:28.701377+0000","last_peered":"2024-03-06T07:20:28.701377+0000","last_clean":"2024-03-06T07:20:28.701377+0000","last_became_active":"2024-03-06T07:20:21.859855+0000","last_became_peered":"2024-03-06T07:20:21.859855+0000","last_unstale":"2024-03-06T07:20:28.701377+0000","last_undegraded":"2024-03-06T07:20:28.701377+0000","last_fullsized":"2024-03-06T07:20:28.701377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879935+0000","last_change":"2024-03-06T07:20:21.856836+0000","last_active":"2024-03-06T07:20:27.879935+0000","last_peered":"2024-03-06T07:20:27.879935+0000","last_clean":"2024-03-06T07:20:27.879935+0000","last_became_active":"2024-03-06T07:20:21.856421+0000","last_became_peered":"2024-03-06T07:20:21.856421+0000","last_unstale":"2024-03-06T07:20:27.879935+0000","last_undegraded":"2024-03-06T07:20:27.879935+0000","last_fullsized":"2024-03-06T07:20:27.879935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:00.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365852+0000","last_change":"2024-03-06T07:20:21.866911+0000","last_active":"2024-03-06T07:20:28.365852+0000","last_peered":"2024-03-06T07:20:28.365852+0000","last_clean":"2024-03-06T07:20:28.365852+0000","last_became_active":"2024-03-06T07:20:21.866557+0000","last_became_peered":"2024-03-06T07:20:21.866557+0000","last_unstale":"2024-03-06T07:20:28.365852+0000","last_undegraded":"2024-03-06T07:20:28.365852+0000","last_fullsized":"2024-03-06T07:20:28.365852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702389+0000","last_change":"2024-03-06T07:20:21.866315+0000","last_active":"2024-03-06T07:20:28.702389+0000","last_peered":"2024-03-06T07:20:28.702389+0000","last_clean":"2024-03-06T07:20:28.702389+0000","last_became_active":"2024-03-06T07:20:21.866035+0000","last_became_peered":"2024-03-06T07:20:21.866035+0000","last_unstale":"2024-03-06T07:20:28.702389+0000","last_undegraded":"2024-03-06T07:20:28.702389+0000","last_fullsized":"2024-03-06T07:20:28.702389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:20.911597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880753+0000","last_change":"2024-03-06T07:20:21.866364+0000","last_active":"2024-03-06T07:20:27.880753+0000","last_peered":"2024-03-06T07:20:27.880753+0000","last_clean":"2024-03-06T07:20:27.880753+0000","last_became_active":"2024-03-06T07:20:21.866091+0000","last_became_peered":"2024-03-06T07:20:21.866091+0000","last_unstale":"2024-03-06T07:20:27.880753+0000","last_undegraded":"2024-03-06T07:20:27.880753+0000","last_fullsized":"2024-03-06T07:20:27.880753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:01.604000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701743+0000","last_change":"2024-03-06T07:20:21.868488+0000","last_active":"2024-03-06T07:20:28.701743+0000","last_peered":"2024-03-06T07:20:28.701743+0000","last_clean":"2024-03-06T07:20:28.701743+0000","last_became_active":"2024-03-06T07:20:21.868091+0000","last_became_peered":"2024-03-06T07:20:21.868091+0000","last_unstale":"2024-03-06T07:20:28.701743+0000","last_undegraded":"2024-03-06T07:20:28.701743+0000","last_fullsized":"2024-03-06T07:20:28.701743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:00.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880000+0000","last_change":"2024-03-06T07:20:21.853440+0000","last_active":"2024-03-06T07:20:27.880000+0000","last_peered":"2024-03-06T07:20:27.880000+0000","last_clean":"2024-03-06T07:20:27.880000+0000","last_became_active":"2024-03-06T07:20:21.853089+0000","last_became_peered":"2024-03-06T07:20:21.853089+0000","last_unstale":"2024-03-06T07:20:27.880000+0000","last_undegraded":"2024-03-06T07:20:27.880000+0000","last_fullsized":"2024-03-06T07:20:27.880000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:20.911597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701633+0000","last_change":"2024-03-06T07:20:21.865493+0000","last_active":"2024-03-06T07:20:28.701633+0000","last_peered":"2024-03-06T07:20:28.701633+0000","last_clean":"2024-03-06T07:20:28.701633+0000","last_became_active":"2024-03-06T07:20:21.865203+0000","last_became_peered":"2024-03-06T07:20:21.865203+0000","last_unstale":"2024-03-06T07:20:28.701633+0000","last_undegraded":"2024-03-06T07:20:28.701633+0000","last_fullsized":"2024-03-06T07:20:28.701633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:20.882851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701280+0000","last_change":"2024-03-06T07:20:21.847412+0000","last_active":"2024-03-06T07:20:28.701280+0000","last_peered":"2024-03-06T07:20:28.701280+0000","last_clean":"2024-03-06T07:20:28.701280+0000","last_became_active":"2024-03-06T07:20:21.847081+0000","last_became_peered":"2024-03-06T07:20:21.847081+0000","last_unstale":"2024-03-06T07:20:28.701280+0000","last_undegraded":"2024-03-06T07:20:28.701280+0000","last_fullsized":"2024-03-06T07:20:28.701280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:28.363605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702051+0000","last_change":"2024-03-06T07:20:21.861878+0000","last_active":"2024-03-06T07:20:28.702051+0000","last_peered":"2024-03-06T07:20:28.702051+0000","last_clean":"2024-03-06T07:20:28.702051+0000","last_became_active":"2024-03-06T07:20:21.859115+0000","last_became_peered":"2024-03-06T07:20:21.859115+0000","last_unstale":"2024-03-06T07:20:28.702051+0000","last_undegraded":"2024-03-06T07:20:28.702051+0000","last_fullsized":"2024-03-06T07:20:28.702051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:42.612183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201453+0000","last_change":"2024-03-06T07:20:21.860975+0000","last_active":"2024-03-06T07:20:28.201453+0000","last_peered":"2024-03-06T07:20:28.201453+0000","last_clean":"2024-03-06T07:20:28.201453+0000","last_became_active":"2024-03-06T07:20:21.860414+0000","last_became_peered":"2024-03-06T07:20:21.860414+0000","last_unstale":"2024-03-06T07:20:28.201453+0000","last_undegraded":"2024-03-06T07:20:28.201453+0000","last_fullsized":"2024-03-06T07:20:28.201453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:53.701404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879575+0000","last_change":"2024-03-06T07:20:21.867097+0000","last_active":"2024-03-06T07:20:27.879575+0000","last_peered":"2024-03-06T07:20:27.879575+0000","last_clean":"2024-03-06T07:20:27.879575+0000","last_became_active":"2024-03-06T07:20:21.866821+0000","last_became_peered":"2024-03-06T07:20:21.866821+0000","last_unstale":"2024-03-06T07:20:27.879575+0000","last_undegraded":"2024-03-06T07:20:27.879575+0000","last_fullsized":"2024-03-06T07:20:27.879575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:25.262951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702397+0000","last_change":"2024-03-06T07:20:21.866152+0000","last_active":"2024-03-06T07:20:28.702397+0000","last_peered":"2024-03-06T07:20:28.702397+0000","last_clean":"2024-03-06T07:20:28.702397+0000","last_became_active":"2024-03-06T07:20:21.865919+0000","last_became_peered":"2024-03-06T07:20:21.865919+0000","last_unstale":"2024-03-06T07:20:28.702397+0000","last_undegraded":"2024-03-06T07:20:28.702397+0000","last_fullsized":"2024-03-06T07:20:28.702397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:26.855742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701560+0000","last_change":"2024-03-06T07:20:21.860761+0000","last_active":"2024-03-06T07:20:28.701560+0000","last_peered":"2024-03-06T07:20:28.701560+0000","last_clean":"2024-03-06T07:20:28.701560+0000","last_became_active":"2024-03-06T07:20:21.860444+0000","last_became_peered":"2024-03-06T07:20:21.860444+0000","last_unstale":"2024-03-06T07:20:28.701560+0000","last_undegraded":"2024-03-06T07:20:28.701560+0000","last_fullsized":"2024-03-06T07:20:28.701560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:54.020809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.365409+0000","last_change":"2024-03-06T07:20:21.855033+0000","last_active":"2024-03-06T07:20:28.365409+0000","last_peered":"2024-03-06T07:20:28.365409+0000","last_clean":"2024-03-06T07:20:28.365409+0000","last_became_active":"2024-03-06T07:20:21.854554+0000","last_became_peered":"2024-03-06T07:20:21.854554+0000","last_unstale":"2024-03-06T07:20:28.365409+0000","last_undegraded":"2024-03-06T07:20:28.365409+0000","last_fullsized":"2024-03-06T07:20:28.365409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:20.911597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701196+0000","last_change":"2024-03-06T07:20:21.856900+0000","last_active":"2024-03-06T07:20:28.701196+0000","last_peered":"2024-03-06T07:20:28.701196+0000","last_clean":"2024-03-06T07:20:28.701196+0000","last_became_active":"2024-03-06T07:20:21.856628+0000","last_became_peered":"2024-03-06T07:20:21.856628+0000","last_unstale":"2024-03-06T07:20:28.701196+0000","last_undegraded":"2024-03-06T07:20:28.701196+0000","last_fullsized":"2024-03-06T07:20:28.701196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:47.276655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880101+0000","last_change":"2024-03-06T07:20:21.868803+0000","last_active":"2024-03-06T07:20:27.880101+0000","last_peered":"2024-03-06T07:20:27.880101+0000","last_clean":"2024-03-06T07:20:27.880101+0000","last_became_active":"2024-03-06T07:20:21.868518+0000","last_became_peered":"2024-03-06T07:20:21.868518+0000","last_unstale":"2024-03-06T07:20:27.880101+0000","last_undegraded":"2024-03-06T07:20:27.880101+0000","last_fullsized":"2024-03-06T07:20:27.880101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:03.883136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.880416+0000","last_change":"2024-03-06T07:20:21.863674+0000","last_active":"2024-03-06T07:20:27.880416+0000","last_peered":"2024-03-06T07:20:27.880416+0000","last_clean":"2024-03-06T07:20:27.880416+0000","last_became_active":"2024-03-06T07:20:21.863138+0000","last_became_peered":"2024-03-06T07:20:21.863138+0000","last_unstale":"2024-03-06T07:20:27.880416+0000","last_undegraded":"2024-03-06T07:20:27.880416+0000","last_fullsized":"2024-03-06T07:20:27.880416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:26.767916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.702522+0000","last_change":"2024-03-06T07:20:21.858343+0000","last_active":"2024-03-06T07:20:28.702522+0000","last_peered":"2024-03-06T07:20:28.702522+0000","last_clean":"2024-03-06T07:20:28.702522+0000","last_became_active":"2024-03-06T07:20:21.858078+0000","last_became_peered":"2024-03-06T07:20:21.858078+0000","last_unstale":"2024-03-06T07:20:28.702522+0000","last_undegraded":"2024-03-06T07:20:28.702522+0000","last_fullsized":"2024-03-06T07:20:28.702522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.423089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701847+0000","last_change":"2024-03-06T07:20:21.866157+0000","last_active":"2024-03-06T07:20:28.701847+0000","last_peered":"2024-03-06T07:20:28.701847+0000","last_clean":"2024-03-06T07:20:28.701847+0000","last_became_active":"2024-03-06T07:20:21.865306+0000","last_became_peered":"2024-03-06T07:20:21.865306+0000","last_unstale":"2024-03-06T07:20:28.701847+0000","last_undegraded":"2024-03-06T07:20:28.701847+0000","last_fullsized":"2024-03-06T07:20:28.701847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:36.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201624+0000","last_change":"2024-03-06T07:20:21.855346+0000","last_active":"2024-03-06T07:20:28.201624+0000","last_peered":"2024-03-06T07:20:28.201624+0000","last_clean":"2024-03-06T07:20:28.201624+0000","last_became_active":"2024-03-06T07:20:21.855059+0000","last_became_peered":"2024-03-06T07:20:21.855059+0000","last_unstale":"2024-03-06T07:20:28.201624+0000","last_undegraded":"2024-03-06T07:20:28.201624+0000","last_fullsized":"2024-03-06T07:20:28.201624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:51.111188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:27.879524+0000","last_change":"2024-03-06T07:20:21.868545+0000","last_active":"2024-03-06T07:20:27.879524+0000","last_peered":"2024-03-06T07:20:27.879524+0000","last_clean":"2024-03-06T07:20:27.879524+0000","last_became_active":"2024-03-06T07:20:21.868188+0000","last_became_peered":"2024-03-06T07:20:21.868188+0000","last_unstale":"2024-03-06T07:20:27.879524+0000","last_undegraded":"2024-03-06T07:20:27.879524+0000","last_fullsized":"2024-03-06T07:20:27.879524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:28.363605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201003+0000","last_change":"2024-03-06T07:20:21.856718+0000","last_active":"2024-03-06T07:20:28.201003+0000","last_peered":"2024-03-06T07:20:28.201003+0000","last_clean":"2024-03-06T07:20:28.201003+0000","last_became_active":"2024-03-06T07:20:21.856451+0000","last_became_peered":"2024-03-06T07:20:21.856451+0000","last_unstale":"2024-03-06T07:20:28.201003+0000","last_undegraded":"2024-03-06T07:20:28.201003+0000","last_fullsized":"2024-03-06T07:20:28.201003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:01.604000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701625+0000","last_change":"2024-03-06T07:20:21.861834+0000","last_active":"2024-03-06T07:20:28.701625+0000","last_peered":"2024-03-06T07:20:28.701625+0000","last_clean":"2024-03-06T07:20:28.701625+0000","last_became_active":"2024-03-06T07:20:21.860998+0000","last_became_peered":"2024-03-06T07:20:21.860998+0000","last_unstale":"2024-03-06T07:20:28.701625+0000","last_undegraded":"2024-03-06T07:20:28.701625+0000","last_fullsized":"2024-03-06T07:20:28.701625+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:01.604000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.200939+0000","last_change":"2024-03-06T07:20:21.860505+0000","last_active":"2024-03-06T07:20:28.200939+0000","last_peered":"2024-03-06T07:20:28.200939+0000","last_clean":"2024-03-06T07:20:28.200939+0000","last_became_active":"2024-03-06T07:20:21.860241+0000","last_became_peered":"2024-03-06T07:20:21.860241+0000","last_unstale":"2024-03-06T07:20:28.200939+0000","last_undegraded":"2024-03-06T07:20:28.200939+0000","last_fullsized":"2024-03-06T07:20:28.200939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:20.882851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701769+0000","last_change":"2024-03-06T07:20:21.868402+0000","last_active":"2024-03-06T07:20:28.701769+0000","last_peered":"2024-03-06T07:20:28.701769+0000","last_clean":"2024-03-06T07:20:28.701769+0000","last_became_active":"2024-03-06T07:20:21.868068+0000","last_became_peered":"2024-03-06T07:20:21.868068+0000","last_unstale":"2024-03-06T07:20:28.701769+0000","last_undegraded":"2024-03-06T07:20:28.701769+0000","last_fullsized":"2024-03-06T07:20:28.701769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:03.883136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.701196+0000","last_change":"2024-03-06T07:20:21.860664+0000","last_active":"2024-03-06T07:20:28.701196+0000","last_peered":"2024-03-06T07:20:28.701196+0000","last_clean":"2024-03-06T07:20:28.701196+0000","last_became_active":"2024-03-06T07:20:21.860393+0000","last_became_peered":"2024-03-06T07:20:21.860393+0000","last_unstale":"2024-03-06T07:20:28.701196+0000","last_undegraded":"2024-03-06T07:20:28.701196+0000","last_fullsized":"2024-03-06T07:20:28.701196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:36.185596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.366176+0000","last_change":"2024-03-06T07:20:21.860087+0000","last_active":"2024-03-06T07:20:28.366176+0000","last_peered":"2024-03-06T07:20:28.366176+0000","last_clean":"2024-03-06T07:20:28.366176+0000","last_became_active":"2024-03-06T07:20:21.859804+0000","last_became_peered":"2024-03-06T07:20:21.859804+0000","last_unstale":"2024-03-06T07:20:28.366176+0000","last_undegraded":"2024-03-06T07:20:28.366176+0000","last_fullsized":"2024-03-06T07:20:28.366176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:00.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-06T07:20:28.201546+0000","last_change":"2024-03-06T07:20:21.859581+0000","last_active":"2024-03-06T07:20:28.201546+0000","last_peered":"2024-03-06T07:20:28.201546+0000","last_clean":"2024-03-06T07:20:28.201546+0000","last_became_active":"2024-03-06T07:20:21.859305+0000","last_became_peered":"2024-03-06T07:20:21.859305+0000","last_unstale":"2024-03-06T07:20:28.201546+0000","last_undegraded":"2024-03-06T07:20:28.201546+0000","last_fullsized":"2024-03-06T07:20:28.201546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:20.829541+0000","last_clean_scrub_stamp":"2024-03-06T07:20:20.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:47.201278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":698343,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":344064,"data_stored":698399,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:20:43.988 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:20:43.988 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-06T07:20:43.988 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-06T07:20:43.988 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:20:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.121+0000 7fc32152c640 1 Processor -- start 2024-03-06T07:20:44.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.121+0000 7fc32152c640 1 -- start start 2024-03-06T07:20:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.122+0000 7fc32152c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc31c152850 0x7fc31c0c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.122+0000 7fc32152c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc31c0bf9e0 0x7fc31c0c5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.122+0000 7fc32152c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc31c156150 con 0x7fc31c0c4aa0 2024-03-06T07:20:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.122+0000 7fc32152c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc31c152c30 con 0x7fc31c0bf9e0 2024-03-06T07:20:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.122+0000 7fc32152c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc31c067810 con 0x7fc31c152850 2024-03-06T07:20:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.122+0000 7fc313fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc31c0bf9e0 0x7fc31c0c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.122+0000 7fc313fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc31c0bf9e0 0x7fc31c0c5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40814/0 (socket says 172.21.15.1:40814) 2024-03-06T07:20:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.122+0000 7fc313fff640 1 -- 172.21.15.1:0/13649643 learned_addr learned my addr 172.21.15.1:0/13649643 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.122+0000 7fc313fff640 1 -- 172.21.15.1:0/13649643 >> v2:172.21.15.98:3300/0 conn(0x7fc31c152850 msgr2=0x7fc31c0c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.122+0000 7fc313fff640 1 --2- 172.21.15.1:0/13649643 >> v2:172.21.15.98:3300/0 conn(0x7fc31c152850 0x7fc31c0c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.122+0000 7fc313fff640 1 -- 172.21.15.1:0/13649643 >> v1:172.21.15.98:6789/0 conn(0x7fc31c0c4aa0 legacy=0x7fc31c158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.122+0000 7fc313fff640 1 -- 172.21.15.1:0/13649643 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc31c155390 con 0x7fc31c0bf9e0 2024-03-06T07:20:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.123+0000 7fc313fff640 1 --2- 172.21.15.1:0/13649643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc31c0bf9e0 0x7fc31c0c5320 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc31c08c400 tx=0x7fc30402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99ffb1bcdbdf8124 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.123+0000 7fc31a7fc640 1 -- 172.21.15.1:0/13649643 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc304038070 con 0x7fc31c0bf9e0 2024-03-06T07:20:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.123+0000 7fc31a7fc640 1 -- 172.21.15.1:0/13649643 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30402f050 con 0x7fc31c0bf9e0 2024-03-06T07:20:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.123+0000 7fc31a7fc640 1 -- 172.21.15.1:0/13649643 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc304031970 con 0x7fc31c0bf9e0 2024-03-06T07:20:44.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.124+0000 7fc32152c640 1 -- 172.21.15.1:0/13649643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc31c0bf9e0 msgr2=0x7fc31c0c5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.124+0000 7fc32152c640 1 --2- 172.21.15.1:0/13649643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc31c0bf9e0 0x7fc31c0c5320 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc31c08c400 tx=0x7fc30402b7f0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.124+0000 7fc32152c640 1 -- 172.21.15.1:0/13649643 shutdown_connections 2024-03-06T07:20:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.124+0000 7fc32152c640 1 --2- 172.21.15.1:0/13649643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc31c0bf9e0 0x7fc31c0c5320 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.124+0000 7fc32152c640 1 --2- 172.21.15.1:0/13649643 >> v2:172.21.15.98:3300/0 conn(0x7fc31c152850 0x7fc31c0c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.124+0000 7fc32152c640 1 -- 172.21.15.1:0/13649643 >> 172.21.15.1:0/13649643 conn(0x7fc31c08b400 msgr2=0x7fc31c066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.124+0000 7fc32152c640 1 -- 172.21.15.1:0/13649643 shutdown_connections 2024-03-06T07:20:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.124+0000 7fc32152c640 1 -- 172.21.15.1:0/13649643 wait complete. 2024-03-06T07:20:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.125+0000 7fc32152c640 1 Processor -- start 2024-03-06T07:20:44.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.125+0000 7fc32152c640 1 -- start start 2024-03-06T07:20:44.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.126+0000 7fc32152c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc31c0bf9e0 0x7fc31c1eb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.126+0000 7fc32152c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc31c0c4aa0 0x7fc31c0c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.126+0000 7fc32152c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc31c152850 0x7fc31c153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.126+0000 7fc32152c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc31c155950 con 0x7fc31c152850 2024-03-06T07:20:44.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.126+0000 7fc32152c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc31c155cc0 con 0x7fc31c0bf9e0 2024-03-06T07:20:44.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.126+0000 7fc32152c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc31c0c4550 con 0x7fc31c0c4aa0 2024-03-06T07:20:44.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.127+0000 7fc31b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc31c152850 0x7fc31c153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.127+0000 7fc31b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc31c152850 0x7fc31c153010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40822/0 (socket says 172.21.15.1:40822) 2024-03-06T07:20:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.127+0000 7fc31b7fe640 1 -- 172.21.15.1:0/3579422026 learned_addr learned my addr 172.21.15.1:0/3579422026 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.127+0000 7fc313fff640 1 --2- 172.21.15.1:0/3579422026 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc31c0c4aa0 0x7fc31c0c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.127+0000 7fc31affd640 1 --2- 172.21.15.1:0/3579422026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc31c0bf9e0 0x7fc31c1eb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.127+0000 7fc31b7fe640 1 -- 172.21.15.1:0/3579422026 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc31c0c4aa0 msgr2=0x7fc31c0c9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.127+0000 7fc31b7fe640 1 --2- 172.21.15.1:0/3579422026 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc31c0c4aa0 0x7fc31c0c9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.127+0000 7fc31b7fe640 1 -- 172.21.15.1:0/3579422026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc31c0bf9e0 msgr2=0x7fc31c1eb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.127+0000 7fc31b7fe640 1 --2- 172.21.15.1:0/3579422026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc31c0bf9e0 0x7fc31c1eb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.127+0000 7fc31b7fe640 1 -- 172.21.15.1:0/3579422026 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc31c154b60 con 0x7fc31c152850 2024-03-06T07:20:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.127+0000 7fc31affd640 1 --2- 172.21.15.1:0/3579422026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc31c0bf9e0 0x7fc31c1eb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.127+0000 7fc31b7fe640 1 --2- 172.21.15.1:0/3579422026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc31c152850 0x7fc31c153010 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc31400c780 tx=0x7fc31400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.127+0000 7fc3137fe640 1 -- 172.21.15.1:0/3579422026 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc314017070 con 0x7fc31c152850 2024-03-06T07:20:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.127+0000 7fc3137fe640 1 -- 172.21.15.1:0/3579422026 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc314003da0 con 0x7fc31c152850 2024-03-06T07:20:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.127+0000 7fc3137fe640 1 -- 172.21.15.1:0/3579422026 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3140052b0 con 0x7fc31c152850 2024-03-06T07:20:44.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.127+0000 7fc32152c640 1 -- 172.21.15.1:0/3579422026 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc31c14b240 con 0x7fc31c152850 2024-03-06T07:20:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.127+0000 7fc32152c640 1 -- 172.21.15.1:0/3579422026 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc31c156850 con 0x7fc31c152850 2024-03-06T07:20:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.129+0000 7fc32152c640 1 -- 172.21.15.1:0/3579422026 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc31c067fd0 con 0x7fc31c152850 2024-03-06T07:20:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.129+0000 7fc3137fe640 1 -- 172.21.15.1:0/3579422026 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc31401a070 con 0x7fc31c152850 2024-03-06T07:20:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.129+0000 7fc3137fe640 1 --2- 172.21.15.1:0/3579422026 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc2fc06b4d0 0x7fc2fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.129+0000 7fc3137fe640 1 -- 172.21.15.1:0/3579422026 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc314089a70 con 0x7fc31c152850 2024-03-06T07:20:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.130+0000 7fc31affd640 1 --2- 172.21.15.1:0/3579422026 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc2fc06b4d0 0x7fc2fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.130+0000 7fc31affd640 1 --2- 172.21.15.1:0/3579422026 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc2fc06b4d0 0x7fc2fc06d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc30c009890 tx=0x7fc30c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:44.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.133+0000 7fc3137fe640 1 -- 172.21.15.1:0/3579422026 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc314057ff0 con 0x7fc31c152850 2024-03-06T07:20:44.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.389+0000 7fc32152c640 1 -- 172.21.15.1:0/3579422026 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc31c0c6590 con 0x7fc31c152850 2024-03-06T07:20:44.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.389+0000 7fc3137fe640 1 -- 172.21.15.1:0/3579422026 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fc31405c090 con 0x7fc31c152850 2024-03-06T07:20:44.391 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:44.392 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:20:44.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.391+0000 7fc32152c640 1 -- 172.21.15.1:0/3579422026 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc2fc06b4d0 msgr2=0x7fc2fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:44.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.391+0000 7fc32152c640 1 --2- 172.21.15.1:0/3579422026 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc2fc06b4d0 0x7fc2fc06d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc30c009890 tx=0x7fc30c008040 comp rx=0 tx=0).stop 2024-03-06T07:20:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.391+0000 7fc32152c640 1 -- 172.21.15.1:0/3579422026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc31c152850 msgr2=0x7fc31c153010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.391+0000 7fc32152c640 1 --2- 172.21.15.1:0/3579422026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc31c152850 0x7fc31c153010 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc31400c780 tx=0x7fc31400cc40 comp rx=0 tx=0).stop 2024-03-06T07:20:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.391+0000 7fc32152c640 1 -- 172.21.15.1:0/3579422026 shutdown_connections 2024-03-06T07:20:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.391+0000 7fc32152c640 1 --2- 172.21.15.1:0/3579422026 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc2fc06b4d0 0x7fc2fc06d990 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.391+0000 7fc32152c640 1 --2- 172.21.15.1:0/3579422026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc31c152850 0x7fc31c153010 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.391+0000 7fc32152c640 1 --2- 172.21.15.1:0/3579422026 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc31c0c4aa0 0x7fc31c0c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.391+0000 7fc32152c640 1 --2- 172.21.15.1:0/3579422026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc31c0bf9e0 0x7fc31c1eb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.391+0000 7fc32152c640 1 -- 172.21.15.1:0/3579422026 >> 172.21.15.1:0/3579422026 conn(0x7fc31c08b400 msgr2=0x7fc31c0c0390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.391+0000 7fc32152c640 1 -- 172.21.15.1:0/3579422026 shutdown_connections 2024-03-06T07:20:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.391+0000 7fc32152c640 1 -- 172.21.15.1:0/3579422026 wait complete. 2024-03-06T07:20:44.404 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-06T07:20:44.404 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:44.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.507+0000 7f30dd078640 1 Processor -- start 2024-03-06T07:20:44.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.507+0000 7f30dd078640 1 -- start start 2024-03-06T07:20:44.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.508+0000 7f30dd078640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f30d8152650 0x7f30d8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.508+0000 7f30dd078640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d80c6b10 0x7f30d8158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.508+0000 7f30dd078640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d80c2fc0 con 0x7f30d80bf9d0 2024-03-06T07:20:44.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.508+0000 7f30dd078640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30d8155800 con 0x7f30d8152650 2024-03-06T07:20:44.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.508+0000 7f30dd078640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d80c6690 con 0x7f30d80c6b10 2024-03-06T07:20:44.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.508+0000 7f30d6d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f30d8152650 0x7f30d8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:44.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.508+0000 7f30d6d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f30d8152650 0x7f30d8155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52770/0 (socket says 172.21.15.1:52770) 2024-03-06T07:20:44.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.508+0000 7f30d6d9d640 1 -- 172.21.15.1:0/4246113919 learned_addr learned my addr 172.21.15.1:0/4246113919 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:44.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.508+0000 7f30cffff640 1 --2- 172.21.15.1:0/4246113919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d80c6b10 0x7f30d8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:44.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.508+0000 7f30d6d9d640 1 -- 172.21.15.1:0/4246113919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d80c6b10 msgr2=0x7f30d8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:44.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.508+0000 7f30d6d9d640 1 --2- 172.21.15.1:0/4246113919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d80c6b10 0x7f30d8158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.508+0000 7f30d6d9d640 1 -- 172.21.15.1:0/4246113919 >> v1:172.21.15.98:6789/0 conn(0x7f30d80bf9d0 legacy=0x7f30d80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:44.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.508+0000 7f30d6d9d640 1 -- 172.21.15.1:0/4246113919 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d80c7280 con 0x7f30d8152650 2024-03-06T07:20:44.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.509+0000 7f30d6d9d640 1 --2- 172.21.15.1:0/4246113919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d8152650 0x7f30d8155c90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f30c8009870 tx=0x7f30c802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999b49235d52b7de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:44.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.509+0000 7f30d659c640 1 -- 172.21.15.1:0/4246113919 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c8035020 con 0x7f30d8152650 2024-03-06T07:20:44.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.509+0000 7f30d659c640 1 -- 172.21.15.1:0/4246113919 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c8031640 con 0x7f30d8152650 2024-03-06T07:20:44.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.509+0000 7f30d659c640 1 -- 172.21.15.1:0/4246113919 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c802d040 con 0x7f30d8152650 2024-03-06T07:20:44.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.509+0000 7f30dd078640 1 -- 172.21.15.1:0/4246113919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d8152650 msgr2=0x7f30d8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:44.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.510+0000 7f30dd078640 1 --2- 172.21.15.1:0/4246113919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d8152650 0x7f30d8155c90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f30c8009870 tx=0x7f30c802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.510+0000 7f30dd078640 1 -- 172.21.15.1:0/4246113919 shutdown_connections 2024-03-06T07:20:44.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.510+0000 7f30dd078640 1 --2- 172.21.15.1:0/4246113919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d80c6b10 0x7f30d8158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.510+0000 7f30dd078640 1 --2- 172.21.15.1:0/4246113919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d8152650 0x7f30d8155c90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.510+0000 7f30dd078640 1 -- 172.21.15.1:0/4246113919 >> 172.21.15.1:0/4246113919 conn(0x7f30d8066ee0 msgr2=0x7f30d80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:44.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.510+0000 7f30dd078640 1 -- 172.21.15.1:0/4246113919 shutdown_connections 2024-03-06T07:20:44.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.510+0000 7f30dd078640 1 -- 172.21.15.1:0/4246113919 wait complete. 2024-03-06T07:20:44.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.511+0000 7f30dd078640 1 Processor -- start 2024-03-06T07:20:44.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.511+0000 7f30dd078640 1 -- start start 2024-03-06T07:20:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30dd078640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d80bf9d0 0x7f30d81ef0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30dd078640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d8152650 0x7f30d814c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30dd078640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30d8143d70 0x7f30d81de970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30dd078640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d8155490 con 0x7f30d80bf9d0 2024-03-06T07:20:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30dd078640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d81551b0 con 0x7f30d8143d70 2024-03-06T07:20:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30dd078640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d8152a30 con 0x7f30d8152650 2024-03-06T07:20:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30d6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d80bf9d0 0x7f30d81ef0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30d6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d80bf9d0 0x7f30d81ef0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52780/0 (socket says 172.21.15.1:52780) 2024-03-06T07:20:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30d6d9d640 1 -- 172.21.15.1:0/3288123194 learned_addr learned my addr 172.21.15.1:0/3288123194 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30d6d9d640 1 -- 172.21.15.1:0/3288123194 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d8152650 msgr2=0x7f30d814c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30d759e640 1 --2- 172.21.15.1:0/3288123194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30d8143d70 0x7f30d81de970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30cffff640 1 --2- 172.21.15.1:0/3288123194 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d8152650 0x7f30d814c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30d6d9d640 1 --2- 172.21.15.1:0/3288123194 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d8152650 0x7f30d814c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30d6d9d640 1 -- 172.21.15.1:0/3288123194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30d8143d70 msgr2=0x7f30d81de970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30d6d9d640 1 --2- 172.21.15.1:0/3288123194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30d8143d70 0x7f30d81de970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30d6d9d640 1 -- 172.21.15.1:0/3288123194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d81ef5e0 con 0x7f30d80bf9d0 2024-03-06T07:20:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30cffff640 1 --2- 172.21.15.1:0/3288123194 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d8152650 0x7f30d814c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.512+0000 7f30d759e640 1 --2- 172.21.15.1:0/3288123194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30d8143d70 0x7f30d81de970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.513+0000 7f30d6d9d640 1 --2- 172.21.15.1:0/3288123194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d80bf9d0 0x7f30d81ef0a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f30c8000c00 tx=0x7f30c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.513+0000 7f30cf7fe640 1 -- 172.21.15.1:0/3288123194 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c8040070 con 0x7f30d80bf9d0 2024-03-06T07:20:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.513+0000 7f30cf7fe640 1 -- 172.21.15.1:0/3288123194 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c80399f0 con 0x7f30d80bf9d0 2024-03-06T07:20:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.513+0000 7f30cf7fe640 1 -- 172.21.15.1:0/3288123194 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c8035040 con 0x7f30d80bf9d0 2024-03-06T07:20:44.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.513+0000 7f30dd078640 1 -- 172.21.15.1:0/3288123194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d814c7a0 con 0x7f30d80bf9d0 2024-03-06T07:20:44.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.513+0000 7f30dd078640 1 -- 172.21.15.1:0/3288123194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d80c9020 con 0x7f30d80bf9d0 2024-03-06T07:20:44.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.514+0000 7f30dd078640 1 -- 172.21.15.1:0/3288123194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d8067fd0 con 0x7f30d80bf9d0 2024-03-06T07:20:44.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.514+0000 7f30cf7fe640 1 -- 172.21.15.1:0/3288123194 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f30c804b060 con 0x7f30d80bf9d0 2024-03-06T07:20:44.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.515+0000 7f30cf7fe640 1 --2- 172.21.15.1:0/3288123194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30b806b500 0x7f30b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.515+0000 7f30cf7fe640 1 -- 172.21.15.1:0/3288123194 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f30c80abcc0 con 0x7f30d80bf9d0 2024-03-06T07:20:44.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.515+0000 7f30cffff640 1 --2- 172.21.15.1:0/3288123194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30b806b500 0x7f30b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:44.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.515+0000 7f30cffff640 1 --2- 172.21.15.1:0/3288123194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30b806b500 0x7f30b806d9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f30d80679a0 tx=0x7f30c0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:44.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.519+0000 7f30cf7fe640 1 -- 172.21.15.1:0/3288123194 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30c807a890 con 0x7f30d80bf9d0 2024-03-06T07:20:44.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.773+0000 7f30dd078640 1 -- 172.21.15.1:0/3288123194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30d81502f0 con 0x7f30d80bf9d0 2024-03-06T07:20:44.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.774+0000 7f30cf7fe640 1 -- 172.21.15.1:0/3288123194 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7f30c807a190 con 0x7f30d80bf9d0 2024-03-06T07:20:44.776 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:44.776 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:44.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.777+0000 7f30dd078640 1 -- 172.21.15.1:0/3288123194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30b806b500 msgr2=0x7f30b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:44.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.777+0000 7f30dd078640 1 --2- 172.21.15.1:0/3288123194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30b806b500 0x7f30b806d9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f30d80679a0 tx=0x7f30c0008040 comp rx=0 tx=0).stop 2024-03-06T07:20:44.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.777+0000 7f30dd078640 1 -- 172.21.15.1:0/3288123194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d80bf9d0 msgr2=0x7f30d81ef0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:44.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.777+0000 7f30dd078640 1 --2- 172.21.15.1:0/3288123194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d80bf9d0 0x7f30d81ef0a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f30c8000c00 tx=0x7f30c80046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.777+0000 7f30dd078640 1 -- 172.21.15.1:0/3288123194 shutdown_connections 2024-03-06T07:20:44.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.777+0000 7f30dd078640 1 --2- 172.21.15.1:0/3288123194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30b806b500 0x7f30b806d9c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.777+0000 7f30dd078640 1 --2- 172.21.15.1:0/3288123194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30d8143d70 0x7f30d81de970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.777+0000 7f30dd078640 1 --2- 172.21.15.1:0/3288123194 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d8152650 0x7f30d814c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.777+0000 7f30dd078640 1 --2- 172.21.15.1:0/3288123194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d80bf9d0 0x7f30d81ef0a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.777+0000 7f30dd078640 1 -- 172.21.15.1:0/3288123194 >> 172.21.15.1:0/3288123194 conn(0x7f30d8066ee0 msgr2=0x7f30d80c5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:44.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.777+0000 7f30dd078640 1 -- 172.21.15.1:0/3288123194 shutdown_connections 2024-03-06T07:20:44.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.777+0000 7f30dd078640 1 -- 172.21.15.1:0/3288123194 wait complete. 2024-03-06T07:20:44.790 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:20:19.337560+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:20:20.172791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:20:23.884071+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:20:44.790 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:20:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.924+0000 7fdfa6d56640 1 Processor -- start 2024-03-06T07:20:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.924+0000 7fdfa6d56640 1 -- start start 2024-03-06T07:20:44.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.924+0000 7fdfa6d56640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfa00c4420 0x7fdfa00c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.924+0000 7fdfa6d56640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdfa00a7270 0x7fdfa00c6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.925+0000 7fdfa6d56640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa006eb90 con 0x7fdfa00c4800 2024-03-06T07:20:44.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.925+0000 7fdfa6d56640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa00c6560 con 0x7fdfa00a7270 2024-03-06T07:20:44.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.925+0000 7fdfa6d56640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa006ee00 con 0x7fdfa00c4420 2024-03-06T07:20:44.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.925+0000 7fdf97fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdfa00a7270 0x7fdfa00c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:44.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.925+0000 7fdf97fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdfa00a7270 0x7fdfa00c6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52790/0 (socket says 172.21.15.1:52790) 2024-03-06T07:20:44.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.925+0000 7fdf97fff640 1 -- 172.21.15.1:0/442975439 learned_addr learned my addr 172.21.15.1:0/442975439 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:44.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.925+0000 7fdfa4af2640 1 --2- 172.21.15.1:0/442975439 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfa00c4420 0x7fdfa00c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:44.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.925+0000 7fdf97fff640 1 -- 172.21.15.1:0/442975439 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfa00c4420 msgr2=0x7fdfa00c5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:44.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.925+0000 7fdf97fff640 1 --2- 172.21.15.1:0/442975439 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfa00c4420 0x7fdfa00c5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.925+0000 7fdf97fff640 1 -- 172.21.15.1:0/442975439 >> v1:172.21.15.98:6789/0 conn(0x7fdfa00c4800 legacy=0x7fdfa0072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:44.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.925+0000 7fdf97fff640 1 -- 172.21.15.1:0/442975439 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa006f0e0 con 0x7fdfa00a7270 2024-03-06T07:20:44.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.926+0000 7fdf97fff640 1 --2- 172.21.15.1:0/442975439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa00a7270 0x7fdfa00c6020 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdf9000bb10 tx=0x7fdf9002e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db3ad3f6f97c1bc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:44.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.927+0000 7fdf977fe640 1 -- 172.21.15.1:0/442975439 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9003b070 con 0x7fdfa00a7270 2024-03-06T07:20:44.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.927+0000 7fdf977fe640 1 -- 172.21.15.1:0/442975439 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9002ede0 con 0x7fdfa00a7270 2024-03-06T07:20:44.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.927+0000 7fdf977fe640 1 -- 172.21.15.1:0/442975439 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf90036660 con 0x7fdfa00a7270 2024-03-06T07:20:44.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.928+0000 7fdfa6d56640 1 -- 172.21.15.1:0/442975439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa00a7270 msgr2=0x7fdfa00c6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:44.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.928+0000 7fdfa6d56640 1 --2- 172.21.15.1:0/442975439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa00a7270 0x7fdfa00c6020 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdf9000bb10 tx=0x7fdf9002e030 comp rx=0 tx=0).stop 2024-03-06T07:20:44.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.928+0000 7fdfa6d56640 1 -- 172.21.15.1:0/442975439 shutdown_connections 2024-03-06T07:20:44.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.928+0000 7fdfa6d56640 1 --2- 172.21.15.1:0/442975439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa00a7270 0x7fdfa00c6020 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.928+0000 7fdfa6d56640 1 --2- 172.21.15.1:0/442975439 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfa00c4420 0x7fdfa00c5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.928+0000 7fdfa6d56640 1 -- 172.21.15.1:0/442975439 >> 172.21.15.1:0/442975439 conn(0x7fdfa006c730 msgr2=0x7fdfa00c79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:44.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.928+0000 7fdfa6d56640 1 -- 172.21.15.1:0/442975439 shutdown_connections 2024-03-06T07:20:44.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.928+0000 7fdfa6d56640 1 -- 172.21.15.1:0/442975439 wait complete. 2024-03-06T07:20:44.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.929+0000 7fdfa6d56640 1 Processor -- start 2024-03-06T07:20:44.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.929+0000 7fdfa6d56640 1 -- start start 2024-03-06T07:20:44.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.930+0000 7fdfa6d56640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfa00a7270 0x7fdfa01eb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.930+0000 7fdfa6d56640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa00c4420 0x7fdfa00a6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.930+0000 7fdfa6d56640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfa00c4800 0x7fdfa0154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.930+0000 7fdfa6d56640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa006f8e0 con 0x7fdfa00c4420 2024-03-06T07:20:44.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.930+0000 7fdfa6d56640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa00c2dd0 con 0x7fdfa00c4800 2024-03-06T07:20:44.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.930+0000 7fdfa6d56640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa00c3100 con 0x7fdfa00a7270 2024-03-06T07:20:44.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.930+0000 7fdf97fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa00c4420 0x7fdfa00a6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:44.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.930+0000 7fdf97fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa00c4420 0x7fdfa00a6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52804/0 (socket says 172.21.15.1:52804) 2024-03-06T07:20:44.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.930+0000 7fdf97fff640 1 -- 172.21.15.1:0/1190728082 learned_addr learned my addr 172.21.15.1:0/1190728082 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:44.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.930+0000 7fdfa4af2640 1 --2- 172.21.15.1:0/1190728082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfa00a7270 0x7fdfa01eb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:44.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.930+0000 7fdfa52f3640 1 --2- 172.21.15.1:0/1190728082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfa00c4800 0x7fdfa0154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:44.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.931+0000 7fdf97fff640 1 -- 172.21.15.1:0/1190728082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfa00a7270 msgr2=0x7fdfa01eb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:44.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.931+0000 7fdf97fff640 1 --2- 172.21.15.1:0/1190728082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfa00a7270 0x7fdfa01eb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.931+0000 7fdf97fff640 1 -- 172.21.15.1:0/1190728082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfa00c4800 msgr2=0x7fdfa0154bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:44.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.931+0000 7fdf97fff640 1 --2- 172.21.15.1:0/1190728082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfa00c4800 0x7fdfa0154bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:44.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.931+0000 7fdf97fff640 1 -- 172.21.15.1:0/1190728082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa01ebd10 con 0x7fdfa00c4420 2024-03-06T07:20:44.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.931+0000 7fdf97fff640 1 --2- 172.21.15.1:0/1190728082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa00c4420 0x7fdfa00a6970 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdf9000b2c0 tx=0x7fdf90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:44.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.931+0000 7fdf957fa640 1 -- 172.21.15.1:0/1190728082 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf90046070 con 0x7fdfa00c4420 2024-03-06T07:20:44.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.931+0000 7fdf957fa640 1 -- 172.21.15.1:0/1190728082 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9003d050 con 0x7fdfa00c4420 2024-03-06T07:20:44.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.931+0000 7fdfa6d56640 1 -- 172.21.15.1:0/1190728082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa01dd710 con 0x7fdfa00c4420 2024-03-06T07:20:44.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.931+0000 7fdf957fa640 1 -- 172.21.15.1:0/1190728082 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9003b030 con 0x7fdfa00c4420 2024-03-06T07:20:44.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.931+0000 7fdfa6d56640 1 -- 172.21.15.1:0/1190728082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa00ac320 con 0x7fdfa00c4420 2024-03-06T07:20:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.932+0000 7fdfa6d56640 1 -- 172.21.15.1:0/1190728082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa00686e0 con 0x7fdfa00c4420 2024-03-06T07:20:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.933+0000 7fdf957fa640 1 -- 172.21.15.1:0/1190728082 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf90049070 con 0x7fdfa00c4420 2024-03-06T07:20:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.933+0000 7fdf957fa640 1 --2- 172.21.15.1:0/1190728082 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf8c06b4d0 0x7fdf8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.933+0000 7fdf957fa640 1 -- 172.21.15.1:0/1190728082 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fdf900abab0 con 0x7fdfa00c4420 2024-03-06T07:20:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.933+0000 7fdfa4af2640 1 --2- 172.21.15.1:0/1190728082 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf8c06b4d0 0x7fdf8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.934+0000 7fdfa4af2640 1 --2- 172.21.15.1:0/1190728082 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf8c06b4d0 0x7fdf8c06d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c006fd0 tx=0x7fdf9c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:44.937+0000 7fdf957fa640 1 -- 172.21.15.1:0/1190728082 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdfa01ebd10 con 0x7fdfa00c4420 2024-03-06T07:20:45.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.119+0000 7fdfa6d56640 1 -- 172.21.15.1:0/1190728082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdfa00c6ef0 con 0x7fdfa00c4420 2024-03-06T07:20:45.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.120+0000 7fdf957fa640 1 -- 172.21.15.1:0/1190728082 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdf9007a030 con 0x7fdfa00c4420 2024-03-06T07:20:45.122 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:20:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.122+0000 7fdfa6d56640 1 -- 172.21.15.1:0/1190728082 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf8c06b4d0 msgr2=0x7fdf8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.122+0000 7fdfa6d56640 1 --2- 172.21.15.1:0/1190728082 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf8c06b4d0 0x7fdf8c06d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c006fd0 tx=0x7fdf9c008040 comp rx=0 tx=0).stop 2024-03-06T07:20:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.122+0000 7fdfa6d56640 1 -- 172.21.15.1:0/1190728082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa00c4420 msgr2=0x7fdfa00a6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.122+0000 7fdfa6d56640 1 --2- 172.21.15.1:0/1190728082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa00c4420 0x7fdfa00a6970 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdf9000b2c0 tx=0x7fdf90004730 comp rx=0 tx=0).stop 2024-03-06T07:20:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.122+0000 7fdfa6d56640 1 -- 172.21.15.1:0/1190728082 shutdown_connections 2024-03-06T07:20:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.122+0000 7fdfa6d56640 1 --2- 172.21.15.1:0/1190728082 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf8c06b4d0 0x7fdf8c06d990 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.122+0000 7fdfa6d56640 1 --2- 172.21.15.1:0/1190728082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfa00c4800 0x7fdfa0154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.122+0000 7fdfa6d56640 1 --2- 172.21.15.1:0/1190728082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa00c4420 0x7fdfa00a6970 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.122+0000 7fdfa6d56640 1 --2- 172.21.15.1:0/1190728082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfa00a7270 0x7fdfa01eb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.122+0000 7fdfa6d56640 1 -- 172.21.15.1:0/1190728082 >> 172.21.15.1:0/1190728082 conn(0x7fdfa006c730 msgr2=0x7fdfa01e0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:45.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.122+0000 7fdfa6d56640 1 -- 172.21.15.1:0/1190728082 shutdown_connections 2024-03-06T07:20:45.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.122+0000 7fdfa6d56640 1 -- 172.21.15.1:0/1190728082 wait complete. 2024-03-06T07:20:45.135 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:20:45.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.248+0000 7fcc234aa640 1 Processor -- start 2024-03-06T07:20:45.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.248+0000 7fcc234aa640 1 -- start start 2024-03-06T07:20:45.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.249+0000 7fcc234aa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc1c159140 0x7fcc1c0c4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.249+0000 7fcc234aa640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcc1c0c7510 0x7fcc1c15fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.249+0000 7fcc234aa640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1c072d00 con 0x7fcc1c0c69b0 2024-03-06T07:20:45.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.249+0000 7fcc234aa640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc1c0c0e80 con 0x7fcc1c0c7510 2024-03-06T07:20:45.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.249+0000 7fcc234aa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c0c13b0 con 0x7fcc1c159140 2024-03-06T07:20:45.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.249+0000 7fcc21246640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc1c159140 0x7fcc1c0c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.249+0000 7fcc21246640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc1c159140 0x7fcc1c0c4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:41640/0 (socket says 172.21.15.1:41640) 2024-03-06T07:20:45.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.249+0000 7fcc21246640 1 -- 172.21.15.1:0/2574137148 learned_addr learned my addr 172.21.15.1:0/2574137148 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:45.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.249+0000 7fcc21a47640 1 --2- 172.21.15.1:0/2574137148 >> v2:172.21.15.98:3300/0 conn(0x7fcc1c0c7510 0x7fcc1c15fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.250+0000 7fcc21246640 1 -- 172.21.15.1:0/2574137148 >> v2:172.21.15.98:3300/0 conn(0x7fcc1c0c7510 msgr2=0x7fcc1c15fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:45.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.250+0000 7fcc21246640 1 --2- 172.21.15.1:0/2574137148 >> v2:172.21.15.98:3300/0 conn(0x7fcc1c0c7510 0x7fcc1c15fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.250+0000 7fcc21246640 1 -- 172.21.15.1:0/2574137148 >> v1:172.21.15.98:6789/0 conn(0x7fcc1c0c69b0 legacy=0x7fcc1c0c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:45.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.250+0000 7fcc21246640 1 -- 172.21.15.1:0/2574137148 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c073180 con 0x7fcc1c159140 2024-03-06T07:20:45.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.251+0000 7fcc21246640 1 --2- 172.21.15.1:0/2574137148 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc1c159140 0x7fcc1c0c4490 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009870 tx=0x7fcc1002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c608b847b4d7026 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:45.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.251+0000 7fcc0ffff640 1 -- 172.21.15.1:0/2574137148 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10035020 con 0x7fcc1c159140 2024-03-06T07:20:45.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.251+0000 7fcc0ffff640 1 -- 172.21.15.1:0/2574137148 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc10031640 con 0x7fcc1c159140 2024-03-06T07:20:45.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.251+0000 7fcc0ffff640 1 -- 172.21.15.1:0/2574137148 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1002d040 con 0x7fcc1c159140 2024-03-06T07:20:45.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.251+0000 7fcc234aa640 1 -- 172.21.15.1:0/2574137148 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc1c159140 msgr2=0x7fcc1c0c4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:45.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.251+0000 7fcc234aa640 1 --2- 172.21.15.1:0/2574137148 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc1c159140 0x7fcc1c0c4490 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009870 tx=0x7fcc1002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.251+0000 7fcc234aa640 1 -- 172.21.15.1:0/2574137148 shutdown_connections 2024-03-06T07:20:45.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.251+0000 7fcc234aa640 1 --2- 172.21.15.1:0/2574137148 >> v2:172.21.15.98:3300/0 conn(0x7fcc1c0c7510 0x7fcc1c15fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.251+0000 7fcc234aa640 1 --2- 172.21.15.1:0/2574137148 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc1c159140 0x7fcc1c0c4490 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.251+0000 7fcc234aa640 1 -- 172.21.15.1:0/2574137148 >> 172.21.15.1:0/2574137148 conn(0x7fcc1c0c7a50 msgr2=0x7fcc1c0c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:45.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.251+0000 7fcc234aa640 1 -- 172.21.15.1:0/2574137148 shutdown_connections 2024-03-06T07:20:45.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.251+0000 7fcc234aa640 1 -- 172.21.15.1:0/2574137148 wait complete. 2024-03-06T07:20:45.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.252+0000 7fcc234aa640 1 Processor -- start 2024-03-06T07:20:45.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.252+0000 7fcc234aa640 1 -- start start 2024-03-06T07:20:45.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.253+0000 7fcc234aa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc1c0c69b0 0x7fcc1c14e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.253+0000 7fcc234aa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc1c0c7510 0x7fcc1c153520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.253+0000 7fcc234aa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc1c159140 0x7fcc1c14ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.253+0000 7fcc234aa640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c0c10d0 con 0x7fcc1c159140 2024-03-06T07:20:45.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.253+0000 7fcc234aa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c0c6520 con 0x7fcc1c0c69b0 2024-03-06T07:20:45.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.253+0000 7fcc234aa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c159520 con 0x7fcc1c0c7510 2024-03-06T07:20:45.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.253+0000 7fcc21a47640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc1c159140 0x7fcc1c14ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.253+0000 7fcc21a47640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc1c159140 0x7fcc1c14ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52818/0 (socket says 172.21.15.1:52818) 2024-03-06T07:20:45.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.253+0000 7fcc21a47640 1 -- 172.21.15.1:0/3243278890 learned_addr learned my addr 172.21.15.1:0/3243278890 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:45.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.253+0000 7fcc20a45640 1 --2- 172.21.15.1:0/3243278890 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc1c0c7510 0x7fcc1c153520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.253+0000 7fcc21246640 1 --2- 172.21.15.1:0/3243278890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc1c0c69b0 0x7fcc1c14e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.254+0000 7fcc21a47640 1 -- 172.21.15.1:0/3243278890 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc1c0c7510 msgr2=0x7fcc1c153520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:45.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.254+0000 7fcc21a47640 1 --2- 172.21.15.1:0/3243278890 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc1c0c7510 0x7fcc1c153520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.254+0000 7fcc21a47640 1 -- 172.21.15.1:0/3243278890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc1c0c69b0 msgr2=0x7fcc1c14e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:45.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.254+0000 7fcc21a47640 1 --2- 172.21.15.1:0/3243278890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc1c0c69b0 0x7fcc1c14e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.254+0000 7fcc21a47640 1 -- 172.21.15.1:0/3243278890 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c0aa5f0 con 0x7fcc1c159140 2024-03-06T07:20:45.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.254+0000 7fcc21a47640 1 --2- 172.21.15.1:0/3243278890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc1c159140 0x7fcc1c14ed50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400efc0 tx=0x7fcc1400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:45.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.254+0000 7fcc0dffb640 1 -- 172.21.15.1:0/3243278890 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc14018020 con 0x7fcc1c159140 2024-03-06T07:20:45.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.254+0000 7fcc0dffb640 1 -- 172.21.15.1:0/3243278890 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc14014480 con 0x7fcc1c159140 2024-03-06T07:20:45.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.254+0000 7fcc0dffb640 1 -- 172.21.15.1:0/3243278890 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc14014870 con 0x7fcc1c159140 2024-03-06T07:20:45.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.254+0000 7fcc234aa640 1 -- 172.21.15.1:0/3243278890 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1c1ebcb0 con 0x7fcc1c159140 2024-03-06T07:20:45.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.254+0000 7fcc234aa640 1 -- 172.21.15.1:0/3243278890 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc1c14df90 con 0x7fcc1c159140 2024-03-06T07:20:45.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.255+0000 7fcc234aa640 1 -- 172.21.15.1:0/3243278890 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc1c0686e0 con 0x7fcc1c159140 2024-03-06T07:20:45.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.256+0000 7fcc0dffb640 1 -- 172.21.15.1:0/3243278890 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc14005d10 con 0x7fcc1c159140 2024-03-06T07:20:45.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.256+0000 7fcc0dffb640 1 --2- 172.21.15.1:0/3243278890 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc0006b4d0 0x7fcc0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.256+0000 7fcc0dffb640 1 -- 172.21.15.1:0/3243278890 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcc1408b1c0 con 0x7fcc1c159140 2024-03-06T07:20:45.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.256+0000 7fcc21246640 1 --2- 172.21.15.1:0/3243278890 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc0006b4d0 0x7fcc0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.257+0000 7fcc21246640 1 --2- 172.21.15.1:0/3243278890 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc0006b4d0 0x7fcc0006d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcc10036000 tx=0x7fcc1003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:45.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.260+0000 7fcc0dffb640 1 -- 172.21.15.1:0/3243278890 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc14022030 con 0x7fcc1c159140 2024-03-06T07:20:45.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.480+0000 7fcc234aa640 1 -- 172.21.15.1:0/3243278890 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcc1c070080 con 0x7fcc1c159140 2024-03-06T07:20:45.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.481+0000 7fcc0dffb640 1 -- 172.21.15.1:0/3243278890 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcc14010040 con 0x7fcc1c159140 2024-03-06T07:20:45.483 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:20:45.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.483+0000 7fcc234aa640 1 -- 172.21.15.1:0/3243278890 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc0006b4d0 msgr2=0x7fcc0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:45.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.483+0000 7fcc234aa640 1 --2- 172.21.15.1:0/3243278890 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc0006b4d0 0x7fcc0006d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcc10036000 tx=0x7fcc1003c040 comp rx=0 tx=0).stop 2024-03-06T07:20:45.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.483+0000 7fcc234aa640 1 -- 172.21.15.1:0/3243278890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc1c159140 msgr2=0x7fcc1c14ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:45.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.483+0000 7fcc234aa640 1 --2- 172.21.15.1:0/3243278890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc1c159140 0x7fcc1c14ed50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400efc0 tx=0x7fcc1400c490 comp rx=0 tx=0).stop 2024-03-06T07:20:45.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.483+0000 7fcc234aa640 1 -- 172.21.15.1:0/3243278890 shutdown_connections 2024-03-06T07:20:45.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.483+0000 7fcc234aa640 1 --2- 172.21.15.1:0/3243278890 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc0006b4d0 0x7fcc0006d990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.483+0000 7fcc234aa640 1 --2- 172.21.15.1:0/3243278890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc1c159140 0x7fcc1c14ed50 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.483+0000 7fcc234aa640 1 --2- 172.21.15.1:0/3243278890 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc1c0c7510 0x7fcc1c153520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.483+0000 7fcc234aa640 1 --2- 172.21.15.1:0/3243278890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc1c0c69b0 0x7fcc1c14e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.483+0000 7fcc234aa640 1 -- 172.21.15.1:0/3243278890 >> 172.21.15.1:0/3243278890 conn(0x7fcc1c0c7a50 msgr2=0x7fcc1c0aba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:45.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.483+0000 7fcc234aa640 1 -- 172.21.15.1:0/3243278890 shutdown_connections 2024-03-06T07:20:45.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.483+0000 7fcc234aa640 1 -- 172.21.15.1:0/3243278890 wait complete. 2024-03-06T07:20:45.499 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:20:45.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.601+0000 7f6ac02df640 1 Processor -- start 2024-03-06T07:20:45.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.601+0000 7f6ac02df640 1 -- start start 2024-03-06T07:20:45.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.602+0000 7f6ac02df640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6ab8152650 0x7f6ab8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.602+0000 7f6ac02df640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6ab80c6b10 0x7f6ab8158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.602+0000 7f6ac02df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ab80bf9d0 0x7f6ab80c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.602+0000 7f6ac02df640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab8155800 con 0x7f6ab80c6b10 2024-03-06T07:20:45.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.602+0000 7f6ac02df640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab80c2fc0 con 0x7f6ab8152650 2024-03-06T07:20:45.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.602+0000 7f6ac02df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab80c6690 con 0x7f6ab80bf9d0 2024-03-06T07:20:45.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.603+0000 7f6abd87a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6ab80c6b10 0x7f6ab8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.603+0000 7f6abd87a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6ab80c6b10 0x7f6ab8158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52820/0 (socket says 172.21.15.1:52820) 2024-03-06T07:20:45.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.603+0000 7f6abd87a640 1 -- 172.21.15.1:0/798945613 learned_addr learned my addr 172.21.15.1:0/798945613 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:45.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.603+0000 7f6abe87c640 1 --2- 172.21.15.1:0/798945613 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ab80bf9d0 0x7f6ab80c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.603+0000 7f6abe07b640 1 --2- 172.21.15.1:0/798945613 >> v2:172.21.15.98:3300/0 conn(0x7f6ab8152650 0x7f6ab8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.603+0000 7f6abd87a640 1 -- 172.21.15.1:0/798945613 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ab80bf9d0 msgr2=0x7f6ab80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:45.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.603+0000 7f6abd87a640 1 --2- 172.21.15.1:0/798945613 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ab80bf9d0 0x7f6ab80c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.603+0000 7f6abd87a640 1 -- 172.21.15.1:0/798945613 >> v2:172.21.15.98:3300/0 conn(0x7f6ab8152650 msgr2=0x7f6ab8155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:45.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.603+0000 7f6abd87a640 1 --2- 172.21.15.1:0/798945613 >> v2:172.21.15.98:3300/0 conn(0x7f6ab8152650 0x7f6ab8155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.603+0000 7f6abd87a640 1 -- 172.21.15.1:0/798945613 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab80c7280 con 0x7f6ab80c6b10 2024-03-06T07:20:45.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.603+0000 7f6abe07b640 1 --2- 172.21.15.1:0/798945613 >> v2:172.21.15.98:3300/0 conn(0x7f6ab8152650 0x7f6ab8155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:45.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.604+0000 7f6abd87a640 1 --2- 172.21.15.1:0/798945613 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab80c6b10 0x7f6ab8158690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80cac90 tx=0x7f6aac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9bf7fce0fc416db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.604+0000 7f6abd079640 1 -- 172.21.15.1:0/798945613 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac02e070 con 0x7f6ab80c6b10 2024-03-06T07:20:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.604+0000 7f6abd079640 1 -- 172.21.15.1:0/798945613 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aac037070 con 0x7f6ab80c6b10 2024-03-06T07:20:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.604+0000 7f6abd079640 1 -- 172.21.15.1:0/798945613 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac031a00 con 0x7f6ab80c6b10 2024-03-06T07:20:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.605+0000 7f6ac02df640 1 -- 172.21.15.1:0/798945613 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab80c6b10 msgr2=0x7f6ab8158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.605+0000 7f6ac02df640 1 --2- 172.21.15.1:0/798945613 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab80c6b10 0x7f6ab8158690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80cac90 tx=0x7f6aac02b880 comp rx=0 tx=0).stop 2024-03-06T07:20:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.605+0000 7f6ac02df640 1 -- 172.21.15.1:0/798945613 shutdown_connections 2024-03-06T07:20:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.605+0000 7f6ac02df640 1 --2- 172.21.15.1:0/798945613 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ab80bf9d0 0x7f6ab80c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.605+0000 7f6ac02df640 1 --2- 172.21.15.1:0/798945613 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab80c6b10 0x7f6ab8158690 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.605+0000 7f6ac02df640 1 --2- 172.21.15.1:0/798945613 >> v2:172.21.15.98:3300/0 conn(0x7f6ab8152650 0x7f6ab8155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.605+0000 7f6ac02df640 1 -- 172.21.15.1:0/798945613 >> 172.21.15.1:0/798945613 conn(0x7f6ab8066ee0 msgr2=0x7f6ab80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.605+0000 7f6ac02df640 1 -- 172.21.15.1:0/798945613 shutdown_connections 2024-03-06T07:20:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.605+0000 7f6ac02df640 1 -- 172.21.15.1:0/798945613 wait complete. 2024-03-06T07:20:45.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.607+0000 7f6ac02df640 1 Processor -- start 2024-03-06T07:20:45.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.607+0000 7f6ac02df640 1 -- start start 2024-03-06T07:20:45.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6ac02df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ab80bf9d0 0x7f6ab814c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6ac02df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab8152650 0x7f6ab81eb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6ac02df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ab8142680 0x7f6ab814e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6ac02df640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab8155490 con 0x7f6ab8152650 2024-03-06T07:20:45.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6ac02df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab81551b0 con 0x7f6ab80bf9d0 2024-03-06T07:20:45.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6ac02df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab8152a30 con 0x7f6ab8142680 2024-03-06T07:20:45.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6abd87a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab8152650 0x7f6ab81eb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6abd87a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab8152650 0x7f6ab81eb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52824/0 (socket says 172.21.15.1:52824) 2024-03-06T07:20:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6abd87a640 1 -- 172.21.15.1:0/1704117181 learned_addr learned my addr 172.21.15.1:0/1704117181 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6abe87c640 1 --2- 172.21.15.1:0/1704117181 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ab8142680 0x7f6ab814e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6abe07b640 1 --2- 172.21.15.1:0/1704117181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ab80bf9d0 0x7f6ab814c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6abd87a640 1 -- 172.21.15.1:0/1704117181 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ab8142680 msgr2=0x7f6ab814e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6abd87a640 1 --2- 172.21.15.1:0/1704117181 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ab8142680 0x7f6ab814e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6abd87a640 1 -- 172.21.15.1:0/1704117181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ab80bf9d0 msgr2=0x7f6ab814c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6abd87a640 1 --2- 172.21.15.1:0/1704117181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ab80bf9d0 0x7f6ab814c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6abd87a640 1 -- 172.21.15.1:0/1704117181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab814c810 con 0x7f6ab8152650 2024-03-06T07:20:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6abe07b640 1 --2- 172.21.15.1:0/1704117181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ab80bf9d0 0x7f6ab814c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.608+0000 7f6abd87a640 1 --2- 172.21.15.1:0/1704117181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab8152650 0x7f6ab81eb6f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6aac02b8c0 tx=0x7f6aac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.609+0000 7f6aaaffd640 1 -- 172.21.15.1:0/1704117181 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac032670 con 0x7f6ab8152650 2024-03-06T07:20:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.609+0000 7f6aaaffd640 1 -- 172.21.15.1:0/1704117181 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aac032830 con 0x7f6ab8152650 2024-03-06T07:20:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.609+0000 7f6ac02df640 1 -- 172.21.15.1:0/1704117181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab81ebc30 con 0x7f6ab8152650 2024-03-06T07:20:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.609+0000 7f6aaaffd640 1 -- 172.21.15.1:0/1704117181 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac032c20 con 0x7f6ab8152650 2024-03-06T07:20:45.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.609+0000 7f6ac02df640 1 -- 172.21.15.1:0/1704117181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab81eaef0 con 0x7f6ab8152650 2024-03-06T07:20:45.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.610+0000 7f6ac02df640 1 -- 172.21.15.1:0/1704117181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab8067fd0 con 0x7f6ab8152650 2024-03-06T07:20:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.611+0000 7f6aaaffd640 1 -- 172.21.15.1:0/1704117181 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6aac04c060 con 0x7f6ab8152650 2024-03-06T07:20:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.611+0000 7f6aaaffd640 1 --2- 172.21.15.1:0/1704117181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6a9c06b4d0 0x7f6a9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.611+0000 7f6aaaffd640 1 -- 172.21.15.1:0/1704117181 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6aac0abe40 con 0x7f6ab8152650 2024-03-06T07:20:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.611+0000 7f6abe07b640 1 --2- 172.21.15.1:0/1704117181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6a9c06b4d0 0x7f6a9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.612+0000 7f6abe07b640 1 --2- 172.21.15.1:0/1704117181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6a9c06b4d0 0x7f6a9c06d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8152220 tx=0x7f6ab4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:45.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.615+0000 7f6aaaffd640 1 -- 172.21.15.1:0/1704117181 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6aac040070 con 0x7f6ab8152650 2024-03-06T07:20:45.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.819+0000 7f6ac02df640 1 -- 172.21.15.1:0/1704117181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6ab80b06f0 con 0x7f6ab8152650 2024-03-06T07:20:45.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.819+0000 7f6aaaffd640 1 -- 172.21.15.1:0/1704117181 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6aac07aa10 con 0x7f6ab8152650 2024-03-06T07:20:45.821 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:20:45.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.821+0000 7f6ac02df640 1 -- 172.21.15.1:0/1704117181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6a9c06b4d0 msgr2=0x7f6a9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:45.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.821+0000 7f6ac02df640 1 --2- 172.21.15.1:0/1704117181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6a9c06b4d0 0x7f6a9c06d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8152220 tx=0x7f6ab4008040 comp rx=0 tx=0).stop 2024-03-06T07:20:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.821+0000 7f6ac02df640 1 -- 172.21.15.1:0/1704117181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab8152650 msgr2=0x7f6ab81eb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.821+0000 7f6ac02df640 1 --2- 172.21.15.1:0/1704117181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab8152650 0x7f6ab81eb6f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6aac02b8c0 tx=0x7f6aac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.821+0000 7f6ac02df640 1 -- 172.21.15.1:0/1704117181 shutdown_connections 2024-03-06T07:20:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.821+0000 7f6ac02df640 1 --2- 172.21.15.1:0/1704117181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6a9c06b4d0 0x7f6a9c06d990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.821+0000 7f6ac02df640 1 --2- 172.21.15.1:0/1704117181 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ab8142680 0x7f6ab814e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.821+0000 7f6ac02df640 1 --2- 172.21.15.1:0/1704117181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab8152650 0x7f6ab81eb6f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.821+0000 7f6ac02df640 1 --2- 172.21.15.1:0/1704117181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ab80bf9d0 0x7f6ab814c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.821+0000 7f6ac02df640 1 -- 172.21.15.1:0/1704117181 >> 172.21.15.1:0/1704117181 conn(0x7f6ab8066ee0 msgr2=0x7f6ab80c4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.821+0000 7f6ac02df640 1 -- 172.21.15.1:0/1704117181 shutdown_connections 2024-03-06T07:20:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.821+0000 7f6ac02df640 1 -- 172.21.15.1:0/1704117181 wait complete. 2024-03-06T07:20:45.834 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:20:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.943+0000 7f666def4640 1 Processor -- start 2024-03-06T07:20:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.943+0000 7f666def4640 1 -- start start 2024-03-06T07:20:45.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.945+0000 7f666def4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f66680af6f0 0x7f66680c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.945+0000 7f666def4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f66680c7930 0x7f666815fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.945+0000 7f666def4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66680c7180 con 0x7f6668152820 2024-03-06T07:20:45.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.945+0000 7f666def4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6668155460 con 0x7f66680c7930 2024-03-06T07:20:45.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.945+0000 7f666def4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66680c6bc0 con 0x7f66680af6f0 2024-03-06T07:20:45.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.945+0000 7f6667fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f66680c7930 0x7f666815fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.945+0000 7f6667fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f66680c7930 0x7f666815fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52828/0 (socket says 172.21.15.1:52828) 2024-03-06T07:20:45.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.945+0000 7f6667fff640 1 -- 172.21.15.1:0/1864488281 learned_addr learned my addr 172.21.15.1:0/1864488281 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:45.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.945+0000 7f6666ffd640 1 --2- 172.21.15.1:0/1864488281 >> v2:172.21.15.98:3300/0 conn(0x7f66680af6f0 0x7f66680c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.945+0000 7f6667fff640 1 -- 172.21.15.1:0/1864488281 >> v2:172.21.15.98:3300/0 conn(0x7f66680af6f0 msgr2=0x7f66680c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:45.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.945+0000 7f6667fff640 1 --2- 172.21.15.1:0/1864488281 >> v2:172.21.15.98:3300/0 conn(0x7f66680af6f0 0x7f66680c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.945+0000 7f6667fff640 1 -- 172.21.15.1:0/1864488281 >> v1:172.21.15.98:6789/0 conn(0x7f6668152820 legacy=0x7f66680c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:45.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.945+0000 7f6667fff640 1 -- 172.21.15.1:0/1864488281 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66680c6ea0 con 0x7f66680c7930 2024-03-06T07:20:45.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.946+0000 7f6667fff640 1 --2- 172.21.15.1:0/1864488281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66680c7930 0x7f666815fb60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f666806cca0 tx=0x7f666002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74ca83e3e4098bf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:45.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.946+0000 7f66667fc640 1 -- 172.21.15.1:0/1864488281 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666002ca80 con 0x7f66680c7930 2024-03-06T07:20:45.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.946+0000 7f66667fc640 1 -- 172.21.15.1:0/1864488281 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666002cc20 con 0x7f66680c7930 2024-03-06T07:20:45.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.946+0000 7f66667fc640 1 -- 172.21.15.1:0/1864488281 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66600343f0 con 0x7f66680c7930 2024-03-06T07:20:45.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.947+0000 7f666def4640 1 -- 172.21.15.1:0/1864488281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66680c7930 msgr2=0x7f666815fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:45.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.947+0000 7f666def4640 1 --2- 172.21.15.1:0/1864488281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66680c7930 0x7f666815fb60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f666806cca0 tx=0x7f666002bee0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.947+0000 7f666def4640 1 -- 172.21.15.1:0/1864488281 shutdown_connections 2024-03-06T07:20:45.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.947+0000 7f666def4640 1 --2- 172.21.15.1:0/1864488281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66680c7930 0x7f666815fb60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.947+0000 7f666def4640 1 --2- 172.21.15.1:0/1864488281 >> v2:172.21.15.98:3300/0 conn(0x7f66680af6f0 0x7f66680c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.947+0000 7f666def4640 1 -- 172.21.15.1:0/1864488281 >> 172.21.15.1:0/1864488281 conn(0x7f6668072720 msgr2=0x7f6668072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:45.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.947+0000 7f666def4640 1 -- 172.21.15.1:0/1864488281 shutdown_connections 2024-03-06T07:20:45.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.947+0000 7f666def4640 1 -- 172.21.15.1:0/1864488281 wait complete. 2024-03-06T07:20:45.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.949+0000 7f666def4640 1 Processor -- start 2024-03-06T07:20:45.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.949+0000 7f666def4640 1 -- start start 2024-03-06T07:20:45.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f666def4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66680af6f0 0x7f6668156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f666def4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66680c7930 0x7f66680c31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f666def4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6668152820 0x7f66680c3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f666def4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6668152c00 con 0x7f6668152820 2024-03-06T07:20:45.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f666def4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66680c0840 con 0x7f66680af6f0 2024-03-06T07:20:45.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f666def4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6668152d80 con 0x7f66680c7930 2024-03-06T07:20:45.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f6667fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6668152820 0x7f66680c3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f6667fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6668152820 0x7f66680c3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52830/0 (socket says 172.21.15.1:52830) 2024-03-06T07:20:45.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f6667fff640 1 -- 172.21.15.1:0/2109280151 learned_addr learned my addr 172.21.15.1:0/2109280151 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:45.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f6666ffd640 1 --2- 172.21.15.1:0/2109280151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66680c7930 0x7f66680c31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f66677fe640 1 --2- 172.21.15.1:0/2109280151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66680af6f0 0x7f6668156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f6667fff640 1 -- 172.21.15.1:0/2109280151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66680c7930 msgr2=0x7f66680c31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:45.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f6667fff640 1 --2- 172.21.15.1:0/2109280151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66680c7930 0x7f66680c31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f6667fff640 1 -- 172.21.15.1:0/2109280151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66680af6f0 msgr2=0x7f6668156560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:45.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f6667fff640 1 --2- 172.21.15.1:0/2109280151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66680af6f0 0x7f6668156560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:45.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f6667fff640 1 -- 172.21.15.1:0/2109280151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66681eb980 con 0x7f6668152820 2024-03-06T07:20:45.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f6666ffd640 1 --2- 172.21.15.1:0/2109280151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66680c7930 0x7f66680c31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:45.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.950+0000 7f6667fff640 1 --2- 172.21.15.1:0/2109280151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6668152820 0x7f66680c3720 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6660009f40 tx=0x7f6660004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:45.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.951+0000 7f665ffff640 1 -- 172.21.15.1:0/2109280151 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6660043020 con 0x7f6668152820 2024-03-06T07:20:45.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.951+0000 7f665ffff640 1 -- 172.21.15.1:0/2109280151 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666003e450 con 0x7f6668152820 2024-03-06T07:20:45.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.951+0000 7f666def4640 1 -- 172.21.15.1:0/2109280151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66681ebc20 con 0x7f6668152820 2024-03-06T07:20:45.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.951+0000 7f665ffff640 1 -- 172.21.15.1:0/2109280151 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666003e860 con 0x7f6668152820 2024-03-06T07:20:45.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.951+0000 7f666def4640 1 -- 172.21.15.1:0/2109280151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66680c29e0 con 0x7f6668152820 2024-03-06T07:20:45.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.952+0000 7f666def4640 1 -- 172.21.15.1:0/2109280151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66680686e0 con 0x7f6668152820 2024-03-06T07:20:45.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.952+0000 7f665ffff640 1 -- 172.21.15.1:0/2109280151 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6660046020 con 0x7f6668152820 2024-03-06T07:20:45.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.953+0000 7f665ffff640 1 --2- 172.21.15.1:0/2109280151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f664806b500 0x7f664806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:45.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.953+0000 7f665ffff640 1 -- 172.21.15.1:0/2109280151 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f66600acd30 con 0x7f6668152820 2024-03-06T07:20:45.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.953+0000 7f66677fe640 1 --2- 172.21.15.1:0/2109280151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f664806b500 0x7f664806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:45.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.954+0000 7f66677fe640 1 --2- 172.21.15.1:0/2109280151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f664806b500 0x7f664806d9c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6650006fd0 tx=0x7f6650008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:45.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:45.957+0000 7f665ffff640 1 -- 172.21.15.1:0/2109280151 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f666004d070 con 0x7f6668152820 2024-03-06T07:20:46.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.138+0000 7f666def4640 1 -- 172.21.15.1:0/2109280151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f66680c5b40 con 0x7f6668152820 2024-03-06T07:20:46.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.139+0000 7f665ffff640 1 -- 172.21.15.1:0/2109280151 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6660033e40 con 0x7f6668152820 2024-03-06T07:20:46.141 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:20:46.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.141+0000 7f666def4640 1 -- 172.21.15.1:0/2109280151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f664806b500 msgr2=0x7f664806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:46.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.141+0000 7f666def4640 1 --2- 172.21.15.1:0/2109280151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f664806b500 0x7f664806d9c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6650006fd0 tx=0x7f6650008040 comp rx=0 tx=0).stop 2024-03-06T07:20:46.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.141+0000 7f666def4640 1 -- 172.21.15.1:0/2109280151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6668152820 msgr2=0x7f66680c3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:46.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.141+0000 7f666def4640 1 --2- 172.21.15.1:0/2109280151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6668152820 0x7f66680c3720 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6660009f40 tx=0x7f6660004a40 comp rx=0 tx=0).stop 2024-03-06T07:20:46.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.141+0000 7f666def4640 1 -- 172.21.15.1:0/2109280151 shutdown_connections 2024-03-06T07:20:46.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.141+0000 7f666def4640 1 --2- 172.21.15.1:0/2109280151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f664806b500 0x7f664806d9c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.141+0000 7f666def4640 1 --2- 172.21.15.1:0/2109280151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6668152820 0x7f66680c3720 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.141+0000 7f666def4640 1 --2- 172.21.15.1:0/2109280151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66680c7930 0x7f66680c31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.141+0000 7f666def4640 1 --2- 172.21.15.1:0/2109280151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66680af6f0 0x7f6668156560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.141+0000 7f666def4640 1 -- 172.21.15.1:0/2109280151 >> 172.21.15.1:0/2109280151 conn(0x7f6668072720 msgr2=0x7f66680bff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:46.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.141+0000 7f666def4640 1 -- 172.21.15.1:0/2109280151 shutdown_connections 2024-03-06T07:20:46.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.141+0000 7f666def4640 1 -- 172.21.15.1:0/2109280151 wait complete. 2024-03-06T07:20:46.154 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:46.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.253+0000 7fef0835e640 1 Processor -- start 2024-03-06T07:20:46.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.253+0000 7fef0835e640 1 -- start start 2024-03-06T07:20:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.254+0000 7fef0835e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef00152650 0x7fef00155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.254+0000 7fef0835e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fef000c6b10 0x7fef00158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.254+0000 7fef0835e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef000c2fc0 con 0x7fef000bf9d0 2024-03-06T07:20:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.254+0000 7fef0835e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef00155800 con 0x7fef000c6b10 2024-03-06T07:20:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.254+0000 7fef0835e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef000c6690 con 0x7fef00152650 2024-03-06T07:20:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.254+0000 7fef058f9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fef000c6b10 0x7fef00158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.254+0000 7fef058f9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fef000c6b10 0x7fef00158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52834/0 (socket says 172.21.15.1:52834) 2024-03-06T07:20:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.254+0000 7fef058f9640 1 -- 172.21.15.1:0/3681992153 learned_addr learned my addr 172.21.15.1:0/3681992153 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.254+0000 7fef060fa640 1 --2- 172.21.15.1:0/3681992153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef00152650 0x7fef00155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.255+0000 7fef058f9640 1 -- 172.21.15.1:0/3681992153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef00152650 msgr2=0x7fef00155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.255+0000 7fef058f9640 1 --2- 172.21.15.1:0/3681992153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef00152650 0x7fef00155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.255+0000 7fef058f9640 1 -- 172.21.15.1:0/3681992153 >> v1:172.21.15.98:6789/0 conn(0x7fef000bf9d0 legacy=0x7fef000c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:46.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.255+0000 7fef058f9640 1 -- 172.21.15.1:0/3681992153 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef000c7280 con 0x7fef000c6b10 2024-03-06T07:20:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.255+0000 7fef058f9640 1 --2- 172.21.15.1:0/3681992153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef000c6b10 0x7fef00158690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7feef0009870 tx=0x7feef002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a08b7cffc9ff56 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.255+0000 7fef050f8640 1 -- 172.21.15.1:0/3681992153 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef00317c0 con 0x7fef000c6b10 2024-03-06T07:20:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.255+0000 7fef050f8640 1 -- 172.21.15.1:0/3681992153 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef0037070 con 0x7fef000c6b10 2024-03-06T07:20:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.255+0000 7fef050f8640 1 -- 172.21.15.1:0/3681992153 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef0031b80 con 0x7fef000c6b10 2024-03-06T07:20:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.256+0000 7fef0835e640 1 -- 172.21.15.1:0/3681992153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef000c6b10 msgr2=0x7fef00158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.256+0000 7fef0835e640 1 --2- 172.21.15.1:0/3681992153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef000c6b10 0x7fef00158690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7feef0009870 tx=0x7feef002b860 comp rx=0 tx=0).stop 2024-03-06T07:20:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.256+0000 7fef0835e640 1 -- 172.21.15.1:0/3681992153 shutdown_connections 2024-03-06T07:20:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.256+0000 7fef0835e640 1 --2- 172.21.15.1:0/3681992153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef000c6b10 0x7fef00158690 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.256+0000 7fef0835e640 1 --2- 172.21.15.1:0/3681992153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef00152650 0x7fef00155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.256+0000 7fef0835e640 1 -- 172.21.15.1:0/3681992153 >> 172.21.15.1:0/3681992153 conn(0x7fef00066ee0 msgr2=0x7fef000672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:46.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.256+0000 7fef0835e640 1 -- 172.21.15.1:0/3681992153 shutdown_connections 2024-03-06T07:20:46.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.256+0000 7fef0835e640 1 -- 172.21.15.1:0/3681992153 wait complete. 2024-03-06T07:20:46.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.257+0000 7fef0835e640 1 Processor -- start 2024-03-06T07:20:46.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.257+0000 7fef0835e640 1 -- start start 2024-03-06T07:20:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef0835e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef000bf9d0 0x7fef001549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef0835e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef00152650 0x7fef001ee520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef0835e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef0014cab0 0x7fef000c5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef0835e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef00155490 con 0x7fef000bf9d0 2024-03-06T07:20:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef0835e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef001551b0 con 0x7fef00152650 2024-03-06T07:20:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef0835e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef00152a30 con 0x7fef0014cab0 2024-03-06T07:20:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef060fa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef000bf9d0 0x7fef001549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef060fa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef000bf9d0 0x7fef001549b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52850/0 (socket says 172.21.15.1:52850) 2024-03-06T07:20:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef060fa640 1 -- 172.21.15.1:0/4289521114 learned_addr learned my addr 172.21.15.1:0/4289521114 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef058f9640 1 --2- 172.21.15.1:0/4289521114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef00152650 0x7fef001ee520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef068fb640 1 --2- 172.21.15.1:0/4289521114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef0014cab0 0x7fef000c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef060fa640 1 -- 172.21.15.1:0/4289521114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef0014cab0 msgr2=0x7fef000c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef060fa640 1 --2- 172.21.15.1:0/4289521114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef0014cab0 0x7fef000c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef060fa640 1 -- 172.21.15.1:0/4289521114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef00152650 msgr2=0x7fef001ee520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef060fa640 1 --2- 172.21.15.1:0/4289521114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef00152650 0x7fef001ee520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef060fa640 1 -- 172.21.15.1:0/4289521114 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef001eea60 con 0x7fef000bf9d0 2024-03-06T07:20:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef068fb640 1 --2- 172.21.15.1:0/4289521114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef0014cab0 0x7fef000c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef058f9640 1 --2- 172.21.15.1:0/4289521114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef00152650 0x7fef001ee520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.258+0000 7fef060fa640 1 --2- 172.21.15.1:0/4289521114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef000bf9d0 0x7fef001549b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7feefc00d800 tx=0x7feefc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.259+0000 7feef6ffd640 1 -- 172.21.15.1:0/4289521114 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feefc00b6c0 con 0x7fef000bf9d0 2024-03-06T07:20:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.259+0000 7feef6ffd640 1 -- 172.21.15.1:0/4289521114 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feefc019070 con 0x7fef000bf9d0 2024-03-06T07:20:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.259+0000 7feef6ffd640 1 -- 172.21.15.1:0/4289521114 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feefc00ba80 con 0x7fef000bf9d0 2024-03-06T07:20:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.259+0000 7fef0835e640 1 -- 172.21.15.1:0/4289521114 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef000ca5b0 con 0x7fef000bf9d0 2024-03-06T07:20:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.259+0000 7fef0835e640 1 -- 172.21.15.1:0/4289521114 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef0014dd40 con 0x7fef000bf9d0 2024-03-06T07:20:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.260+0000 7fef0835e640 1 -- 172.21.15.1:0/4289521114 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef00067fd0 con 0x7fef000bf9d0 2024-03-06T07:20:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.260+0000 7feef6ffd640 1 -- 172.21.15.1:0/4289521114 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feefc00bc20 con 0x7fef000bf9d0 2024-03-06T07:20:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.261+0000 7feef6ffd640 1 --2- 172.21.15.1:0/4289521114 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feee406b500 0x7feee406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.261+0000 7feef6ffd640 1 -- 172.21.15.1:0/4289521114 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7feefc08a910 con 0x7fef000bf9d0 2024-03-06T07:20:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.261+0000 7fef058f9640 1 --2- 172.21.15.1:0/4289521114 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feee406b500 0x7feee406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.261+0000 7fef058f9640 1 --2- 172.21.15.1:0/4289521114 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feee406b500 0x7feee406d9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7feef0036000 tx=0x7feef0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.265+0000 7feef6ffd640 1 -- 172.21.15.1:0/4289521114 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feefc01c360 con 0x7fef000bf9d0 2024-03-06T07:20:46.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.666+0000 7fef0835e640 1 -- 172.21.15.1:0/4289521114 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef000b06f0 con 0x7fef000bf9d0 2024-03-06T07:20:46.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.667+0000 7feef6ffd640 1 -- 172.21.15.1:0/4289521114 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4964 (secure 0 0 0) 0x7feefc0594e0 con 0x7fef000bf9d0 2024-03-06T07:20:46.669 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:46.669 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2822958969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2822958969},{"type":"v1","addr":"172.21.15.98:6836","nonce":2822958969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2442906233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2442906233},{"type":"v1","addr":"172.21.15.98:6837","nonce":2442906233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/448746024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":448746024},{"type":"v1","addr":"172.21.15.1:6834","nonce":448746024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:21.207834+0000","modified":"2024-03-06T07:20:37.259661+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1315115982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1315115982},{"type":"v1","addr":"172.21.15.1:6835","nonce":1315115982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-03-06T07:20:46.669 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-03-06T07:20:46.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.669+0000 7fef0835e640 1 -- 172.21.15.1:0/4289521114 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feee406b500 msgr2=0x7feee406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:46.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.669+0000 7fef0835e640 1 --2- 172.21.15.1:0/4289521114 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feee406b500 0x7feee406d9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7feef0036000 tx=0x7feef0008030 comp rx=0 tx=0).stop 2024-03-06T07:20:46.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.669+0000 7fef0835e640 1 -- 172.21.15.1:0/4289521114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef000bf9d0 msgr2=0x7fef001549b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:46.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.669+0000 7fef0835e640 1 --2- 172.21.15.1:0/4289521114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef000bf9d0 0x7fef001549b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7feefc00d800 tx=0x7feefc00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.669+0000 7fef0835e640 1 -- 172.21.15.1:0/4289521114 shutdown_connections 2024-03-06T07:20:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.669+0000 7fef0835e640 1 --2- 172.21.15.1:0/4289521114 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feee406b500 0x7feee406d9c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.669+0000 7fef0835e640 1 --2- 172.21.15.1:0/4289521114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef0014cab0 0x7fef000c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.669+0000 7fef0835e640 1 --2- 172.21.15.1:0/4289521114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef00152650 0x7fef001ee520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.669+0000 7fef0835e640 1 --2- 172.21.15.1:0/4289521114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef000bf9d0 0x7fef001549b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.669+0000 7fef0835e640 1 -- 172.21.15.1:0/4289521114 >> 172.21.15.1:0/4289521114 conn(0x7fef00066ee0 msgr2=0x7fef000672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.669+0000 7fef0835e640 1 -- 172.21.15.1:0/4289521114 shutdown_connections 2024-03-06T07:20:46.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.669+0000 7fef0835e640 1 -- 172.21.15.1:0/4289521114 wait complete. 2024-03-06T07:20:46.682 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:46.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.782+0000 7f9c1bf13640 1 Processor -- start 2024-03-06T07:20:46.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.783+0000 7f9c1bf13640 1 -- start start 2024-03-06T07:20:46.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.783+0000 7f9c1bf13640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9c140c69b0 0x7f9c140c2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:46.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.783+0000 7f9c1bf13640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c140c7510 0x7f9c140c3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:46.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.783+0000 7f9c1bf13640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c14072d00 con 0x7f9c14159140 2024-03-06T07:20:46.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.783+0000 7f9c1bf13640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c140c0e80 con 0x7f9c140c69b0 2024-03-06T07:20:46.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.783+0000 7f9c1bf13640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140c13b0 con 0x7f9c140c7510 2024-03-06T07:20:46.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.784+0000 7f9c194ae640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9c140c69b0 0x7f9c140c2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:46.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.784+0000 7f9c194ae640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9c140c69b0 0x7f9c140c2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52862/0 (socket says 172.21.15.1:52862) 2024-03-06T07:20:46.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.784+0000 7f9c194ae640 1 -- 172.21.15.1:0/3275678142 learned_addr learned my addr 172.21.15.1:0/3275678142 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:46.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.784+0000 7f9c1a4b0640 1 --2- 172.21.15.1:0/3275678142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c140c7510 0x7f9c140c3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:46.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.784+0000 7f9c194ae640 1 -- 172.21.15.1:0/3275678142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c140c7510 msgr2=0x7f9c140c3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:46.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.784+0000 7f9c194ae640 1 --2- 172.21.15.1:0/3275678142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c140c7510 0x7f9c140c3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.784+0000 7f9c194ae640 1 -- 172.21.15.1:0/3275678142 >> v1:172.21.15.98:6789/0 conn(0x7f9c14159140 legacy=0x7f9c140c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:46.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.784+0000 7f9c194ae640 1 -- 172.21.15.1:0/3275678142 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c14073180 con 0x7f9c140c69b0 2024-03-06T07:20:46.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.785+0000 7f9c194ae640 1 --2- 172.21.15.1:0/3275678142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c140c69b0 0x7f9c140c2b30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009870 tx=0x7f9c0402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e13b944f18210501 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:46.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.785+0000 7f9c18cad640 1 -- 172.21.15.1:0/3275678142 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c04035020 con 0x7f9c140c69b0 2024-03-06T07:20:46.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.785+0000 7f9c18cad640 1 -- 172.21.15.1:0/3275678142 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c04031640 con 0x7f9c140c69b0 2024-03-06T07:20:46.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.785+0000 7f9c18cad640 1 -- 172.21.15.1:0/3275678142 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0402d040 con 0x7f9c140c69b0 2024-03-06T07:20:46.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.786+0000 7f9c1bf13640 1 -- 172.21.15.1:0/3275678142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c140c69b0 msgr2=0x7f9c140c2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:46.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.786+0000 7f9c1bf13640 1 --2- 172.21.15.1:0/3275678142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c140c69b0 0x7f9c140c2b30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009870 tx=0x7f9c0402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.786+0000 7f9c1bf13640 1 -- 172.21.15.1:0/3275678142 shutdown_connections 2024-03-06T07:20:46.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.786+0000 7f9c1bf13640 1 --2- 172.21.15.1:0/3275678142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c140c7510 0x7f9c140c3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.786+0000 7f9c1bf13640 1 --2- 172.21.15.1:0/3275678142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c140c69b0 0x7f9c140c2b30 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.786+0000 7f9c1bf13640 1 -- 172.21.15.1:0/3275678142 >> 172.21.15.1:0/3275678142 conn(0x7f9c140c7a50 msgr2=0x7f9c140c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:46.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.786+0000 7f9c1bf13640 1 -- 172.21.15.1:0/3275678142 shutdown_connections 2024-03-06T07:20:46.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.786+0000 7f9c1bf13640 1 -- 172.21.15.1:0/3275678142 wait complete. 2024-03-06T07:20:46.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.787+0000 7f9c1bf13640 1 Processor -- start 2024-03-06T07:20:46.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.787+0000 7f9c1bf13640 1 -- start start 2024-03-06T07:20:46.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c1bf13640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c140c69b0 0x7f9c141ebb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:46.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c1bf13640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c140c7510 0x7f9c141eeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:46.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c1bf13640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c14159140 0x7f9c14151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:46.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c1bf13640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140c35b0 con 0x7f9c14159140 2024-03-06T07:20:46.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c1bf13640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140c6520 con 0x7f9c140c69b0 2024-03-06T07:20:46.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c1bf13640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140c10d0 con 0x7f9c140c7510 2024-03-06T07:20:46.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c1a4b0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c14159140 0x7f9c14151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:46.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c1a4b0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c14159140 0x7f9c14151680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52870/0 (socket says 172.21.15.1:52870) 2024-03-06T07:20:46.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c1a4b0640 1 -- 172.21.15.1:0/3492499557 learned_addr learned my addr 172.21.15.1:0/3492499557 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:46.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c194ae640 1 --2- 172.21.15.1:0/3492499557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c140c7510 0x7f9c141eeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:46.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c19caf640 1 --2- 172.21.15.1:0/3492499557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c140c69b0 0x7f9c141ebb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:46.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c1a4b0640 1 -- 172.21.15.1:0/3492499557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c140c7510 msgr2=0x7f9c141eeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:46.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c1a4b0640 1 --2- 172.21.15.1:0/3492499557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c140c7510 0x7f9c141eeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c1a4b0640 1 -- 172.21.15.1:0/3492499557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c140c69b0 msgr2=0x7f9c141ebb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:46.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c1a4b0640 1 --2- 172.21.15.1:0/3492499557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c140c69b0 0x7f9c141ebb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c1a4b0640 1 -- 172.21.15.1:0/3492499557 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140accb0 con 0x7f9c14159140 2024-03-06T07:20:46.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c194ae640 1 --2- 172.21.15.1:0/3492499557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c140c7510 0x7f9c141eeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:46.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c1a4b0640 1 --2- 172.21.15.1:0/3492499557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c14159140 0x7f9c14151680 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c009e30 tx=0x7f9c0c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:46.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c0a7fc640 1 -- 172.21.15.1:0/3492499557 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c018020 con 0x7f9c14159140 2024-03-06T07:20:46.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.788+0000 7f9c0a7fc640 1 -- 172.21.15.1:0/3492499557 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0c014480 con 0x7f9c14159140 2024-03-06T07:20:46.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.789+0000 7f9c1bf13640 1 -- 172.21.15.1:0/3492499557 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c141eb030 con 0x7f9c14159140 2024-03-06T07:20:46.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.789+0000 7f9c0a7fc640 1 -- 172.21.15.1:0/3492499557 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c014870 con 0x7f9c14159140 2024-03-06T07:20:46.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.789+0000 7f9c1bf13640 1 -- 172.21.15.1:0/3492499557 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c141eb5f0 con 0x7f9c14159140 2024-03-06T07:20:46.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.789+0000 7f9c1bf13640 1 -- 172.21.15.1:0/3492499557 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c140a7be0 con 0x7f9c14159140 2024-03-06T07:20:46.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.790+0000 7f9c0a7fc640 1 -- 172.21.15.1:0/3492499557 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c0c005d10 con 0x7f9c14159140 2024-03-06T07:20:46.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.790+0000 7f9c0a7fc640 1 --2- 172.21.15.1:0/3492499557 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bf806b500 0x7f9bf806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:46.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.790+0000 7f9c0a7fc640 1 -- 172.21.15.1:0/3492499557 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9c0c08b210 con 0x7f9c14159140 2024-03-06T07:20:46.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.791+0000 7f9c19caf640 1 --2- 172.21.15.1:0/3492499557 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bf806b500 0x7f9bf806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:46.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.791+0000 7f9c19caf640 1 --2- 172.21.15.1:0/3492499557 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bf806b500 0x7f9bf806d9c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9c100016e0 tx=0x7f9c10009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:46.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:46.795+0000 7f9c0a7fc640 1 -- 172.21.15.1:0/3492499557 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c140a7be0 con 0x7f9c14159140 2024-03-06T07:20:47.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.025+0000 7f9c1bf13640 1 -- 172.21.15.1:0/3492499557 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c140686e0 con 0x7f9c14159140 2024-03-06T07:20:47.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.026+0000 7f9c0a7fc640 1 -- 172.21.15.1:0/3492499557 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7f9c0c022030 con 0x7f9c14159140 2024-03-06T07:20:47.028 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:47.028 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:47.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.027+0000 7f9c1bf13640 1 -- 172.21.15.1:0/3492499557 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bf806b500 msgr2=0x7f9bf806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.027+0000 7f9c1bf13640 1 --2- 172.21.15.1:0/3492499557 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bf806b500 0x7f9bf806d9c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9c100016e0 tx=0x7f9c10009040 comp rx=0 tx=0).stop 2024-03-06T07:20:47.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.027+0000 7f9c1bf13640 1 -- 172.21.15.1:0/3492499557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c14159140 msgr2=0x7f9c14151680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.027+0000 7f9c1bf13640 1 --2- 172.21.15.1:0/3492499557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c14159140 0x7f9c14151680 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c009e30 tx=0x7f9c0c00c490 comp rx=0 tx=0).stop 2024-03-06T07:20:47.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.028+0000 7f9c1bf13640 1 -- 172.21.15.1:0/3492499557 shutdown_connections 2024-03-06T07:20:47.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.028+0000 7f9c1bf13640 1 --2- 172.21.15.1:0/3492499557 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bf806b500 0x7f9bf806d9c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.028+0000 7f9c1bf13640 1 --2- 172.21.15.1:0/3492499557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c14159140 0x7f9c14151680 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.028+0000 7f9c1bf13640 1 --2- 172.21.15.1:0/3492499557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c140c7510 0x7f9c141eeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.028+0000 7f9c1bf13640 1 --2- 172.21.15.1:0/3492499557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c140c69b0 0x7f9c141ebb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.028+0000 7f9c1bf13640 1 -- 172.21.15.1:0/3492499557 >> 172.21.15.1:0/3492499557 conn(0x7f9c140c7a50 msgr2=0x7f9c140a7450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:47.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.028+0000 7f9c1bf13640 1 -- 172.21.15.1:0/3492499557 shutdown_connections 2024-03-06T07:20:47.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.028+0000 7f9c1bf13640 1 -- 172.21.15.1:0/3492499557 wait complete. 2024-03-06T07:20:47.041 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:20:19.337560+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:20:20.172791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:20:23.884071+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:20:47.041 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:20:47.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.143+0000 7f526a271640 1 Processor -- start 2024-03-06T07:20:47.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.143+0000 7f526a271640 1 -- start start 2024-03-06T07:20:47.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.144+0000 7f526a271640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f52640c5520 0x7f52641550d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.144+0000 7f526a271640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52640c3700 0x7f526406da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.144+0000 7f526a271640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52640c7650 con 0x7f52640c84c0 2024-03-06T07:20:47.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.144+0000 7f526a271640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52640c78f0 con 0x7f52640c5520 2024-03-06T07:20:47.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.144+0000 7f526a271640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52641587e0 con 0x7f52640c3700 2024-03-06T07:20:47.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.144+0000 7f5263fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f52640c5520 0x7f52641550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.144+0000 7f5263fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f52640c5520 0x7f52641550d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52884/0 (socket says 172.21.15.1:52884) 2024-03-06T07:20:47.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.144+0000 7f5263fff640 1 -- 172.21.15.1:0/1217440633 learned_addr learned my addr 172.21.15.1:0/1217440633 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:47.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.144+0000 7f526880e640 1 --2- 172.21.15.1:0/1217440633 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52640c3700 0x7f526406da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.145+0000 7f5263fff640 1 -- 172.21.15.1:0/1217440633 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52640c3700 msgr2=0x7f526406da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:47.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.145+0000 7f5263fff640 1 --2- 172.21.15.1:0/1217440633 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52640c3700 0x7f526406da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.145+0000 7f5263fff640 1 -- 172.21.15.1:0/1217440633 >> v1:172.21.15.98:6789/0 conn(0x7f52640c84c0 legacy=0x7f5264158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.145+0000 7f5263fff640 1 -- 172.21.15.1:0/1217440633 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5264159410 con 0x7f52640c5520 2024-03-06T07:20:47.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.145+0000 7f5263fff640 1 --2- 172.21.15.1:0/1217440633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52640c5520 0x7f52641550d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5258009870 tx=0x7f525802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b64df14e9518f01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:47.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.146+0000 7f5262ffd640 1 -- 172.21.15.1:0/1217440633 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258035020 con 0x7f52640c5520 2024-03-06T07:20:47.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.146+0000 7f5262ffd640 1 -- 172.21.15.1:0/1217440633 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5258031640 con 0x7f52640c5520 2024-03-06T07:20:47.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.146+0000 7f5262ffd640 1 -- 172.21.15.1:0/1217440633 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525802d040 con 0x7f52640c5520 2024-03-06T07:20:47.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.146+0000 7f526a271640 1 -- 172.21.15.1:0/1217440633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52640c5520 msgr2=0x7f52641550d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.146+0000 7f526a271640 1 --2- 172.21.15.1:0/1217440633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52640c5520 0x7f52641550d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5258009870 tx=0x7f525802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.146+0000 7f526a271640 1 -- 172.21.15.1:0/1217440633 shutdown_connections 2024-03-06T07:20:47.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.146+0000 7f526a271640 1 --2- 172.21.15.1:0/1217440633 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52640c3700 0x7f526406da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.146+0000 7f526a271640 1 --2- 172.21.15.1:0/1217440633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52640c5520 0x7f52641550d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.146+0000 7f526a271640 1 -- 172.21.15.1:0/1217440633 >> 172.21.15.1:0/1217440633 conn(0x7f52640c88f0 msgr2=0x7f52640c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:47.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.146+0000 7f526a271640 1 -- 172.21.15.1:0/1217440633 shutdown_connections 2024-03-06T07:20:47.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.146+0000 7f526a271640 1 -- 172.21.15.1:0/1217440633 wait complete. 2024-03-06T07:20:47.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.147+0000 7f526a271640 1 Processor -- start 2024-03-06T07:20:47.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.147+0000 7f526a271640 1 -- start start 2024-03-06T07:20:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.148+0000 7f526a271640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52640c3700 0x7f52640ac3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.148+0000 7f526a271640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52640c5520 0x7f52640a9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.148+0000 7f526a271640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52640c84c0 0x7f52641ee930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.148+0000 7f526a271640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52640c4220 con 0x7f52640c84c0 2024-03-06T07:20:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.148+0000 7f526a271640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52640c40a0 con 0x7f52640c3700 2024-03-06T07:20:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.148+0000 7f526a271640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52640c5900 con 0x7f52640c5520 2024-03-06T07:20:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.148+0000 7f526880e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52640c84c0 0x7f52641ee930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.148+0000 7f526880e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52640c84c0 0x7f52641ee930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52890/0 (socket says 172.21.15.1:52890) 2024-03-06T07:20:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.148+0000 7f526880e640 1 -- 172.21.15.1:0/992251160 learned_addr learned my addr 172.21.15.1:0/992251160 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.148+0000 7f52637fe640 1 --2- 172.21.15.1:0/992251160 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52640c5520 0x7f52640a9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.149+0000 7f5263fff640 1 --2- 172.21.15.1:0/992251160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52640c3700 0x7f52640ac3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.149+0000 7f526880e640 1 -- 172.21.15.1:0/992251160 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52640c5520 msgr2=0x7f52640a9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.149+0000 7f526880e640 1 --2- 172.21.15.1:0/992251160 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52640c5520 0x7f52640a9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.149+0000 7f526880e640 1 -- 172.21.15.1:0/992251160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52640c3700 msgr2=0x7f52640ac3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.149+0000 7f526880e640 1 --2- 172.21.15.1:0/992251160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52640c3700 0x7f52640ac3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.149+0000 7f526880e640 1 -- 172.21.15.1:0/992251160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526414faf0 con 0x7f52640c84c0 2024-03-06T07:20:47.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.149+0000 7f5263fff640 1 --2- 172.21.15.1:0/992251160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52640c3700 0x7f52640ac3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:47.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.149+0000 7f52637fe640 1 --2- 172.21.15.1:0/992251160 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52640c5520 0x7f52640a9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:47.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.149+0000 7f526880e640 1 --2- 172.21.15.1:0/992251160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52640c84c0 0x7f52641ee930 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f525c009dd0 tx=0x7f525c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:47.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.149+0000 7f5260ff9640 1 -- 172.21.15.1:0/992251160 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c018020 con 0x7f52640c84c0 2024-03-06T07:20:47.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.149+0000 7f5260ff9640 1 -- 172.21.15.1:0/992251160 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525c014480 con 0x7f52640c84c0 2024-03-06T07:20:47.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.149+0000 7f526a271640 1 -- 172.21.15.1:0/992251160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5264150800 con 0x7f52640c84c0 2024-03-06T07:20:47.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.149+0000 7f5260ff9640 1 -- 172.21.15.1:0/992251160 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c014870 con 0x7f52640c84c0 2024-03-06T07:20:47.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.149+0000 7f526a271640 1 -- 172.21.15.1:0/992251160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52641e0fa0 con 0x7f52640c84c0 2024-03-06T07:20:47.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.150+0000 7f526a271640 1 -- 172.21.15.1:0/992251160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52640686e0 con 0x7f52640c84c0 2024-03-06T07:20:47.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.151+0000 7f5260ff9640 1 -- 172.21.15.1:0/992251160 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f525c005d10 con 0x7f52640c84c0 2024-03-06T07:20:47.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.151+0000 7f5260ff9640 1 --2- 172.21.15.1:0/992251160 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f525006b4d0 0x7f525006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.151+0000 7f5260ff9640 1 -- 172.21.15.1:0/992251160 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f525c08b420 con 0x7f52640c84c0 2024-03-06T07:20:47.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.151+0000 7f5263fff640 1 --2- 172.21.15.1:0/992251160 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f525006b4d0 0x7f525006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.152+0000 7f5263fff640 1 --2- 172.21.15.1:0/992251160 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f525006b4d0 0x7f525006d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5258036000 tx=0x7f5258008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:47.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.159+0000 7f5260ff9640 1 -- 172.21.15.1:0/992251160 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f525c022030 con 0x7f52640c84c0 2024-03-06T07:20:47.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.346+0000 7f526a271640 1 -- 172.21.15.1:0/992251160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f52640adea0 con 0x7f52640c84c0 2024-03-06T07:20:47.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.347+0000 7f5260ff9640 1 -- 172.21.15.1:0/992251160 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f525c010040 con 0x7f52640c84c0 2024-03-06T07:20:47.349 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:20:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.348+0000 7f526a271640 1 -- 172.21.15.1:0/992251160 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f525006b4d0 msgr2=0x7f525006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.348+0000 7f526a271640 1 --2- 172.21.15.1:0/992251160 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f525006b4d0 0x7f525006d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5258036000 tx=0x7f5258008030 comp rx=0 tx=0).stop 2024-03-06T07:20:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.348+0000 7f526a271640 1 -- 172.21.15.1:0/992251160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52640c84c0 msgr2=0x7f52641ee930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.348+0000 7f526a271640 1 --2- 172.21.15.1:0/992251160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52640c84c0 0x7f52641ee930 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f525c009dd0 tx=0x7f525c00c490 comp rx=0 tx=0).stop 2024-03-06T07:20:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.349+0000 7f526a271640 1 -- 172.21.15.1:0/992251160 shutdown_connections 2024-03-06T07:20:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.349+0000 7f526a271640 1 --2- 172.21.15.1:0/992251160 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f525006b4d0 0x7f525006d990 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.349+0000 7f526a271640 1 --2- 172.21.15.1:0/992251160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52640c84c0 0x7f52641ee930 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.349+0000 7f526a271640 1 --2- 172.21.15.1:0/992251160 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52640c5520 0x7f52640a9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.349+0000 7f526a271640 1 --2- 172.21.15.1:0/992251160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52640c3700 0x7f52640ac3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.349+0000 7f526a271640 1 -- 172.21.15.1:0/992251160 >> 172.21.15.1:0/992251160 conn(0x7f52640c88f0 msgr2=0x7f5264153120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.349+0000 7f526a271640 1 -- 172.21.15.1:0/992251160 shutdown_connections 2024-03-06T07:20:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.349+0000 7f526a271640 1 -- 172.21.15.1:0/992251160 wait complete. 2024-03-06T07:20:47.362 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:20:47.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.465+0000 7f90a1a88640 1 Processor -- start 2024-03-06T07:20:47.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.465+0000 7f90a1a88640 1 -- start start 2024-03-06T07:20:47.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.466+0000 7f90a1a88640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f909c0c8590 0x7f909c0c8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.466+0000 7f90a1a88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f909c0c3210 0x7f909c1564d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.466+0000 7f90a1a88640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909c0c9010 con 0x7f909c156a10 2024-03-06T07:20:47.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.466+0000 7f90a1a88640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f909c153a20 con 0x7f909c0c8590 2024-03-06T07:20:47.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.466+0000 7f90a1a88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909c1530f0 con 0x7f909c0c3210 2024-03-06T07:20:47.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.466+0000 7f909b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f909c0c8590 0x7f909c0c8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.466+0000 7f909b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f909c0c8590 0x7f909c0c8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52894/0 (socket says 172.21.15.1:52894) 2024-03-06T07:20:47.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.466+0000 7f909b7fe640 1 -- 172.21.15.1:0/181750520 learned_addr learned my addr 172.21.15.1:0/181750520 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:47.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.466+0000 7f909affd640 1 --2- 172.21.15.1:0/181750520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f909c0c3210 0x7f909c1564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.466+0000 7f909b7fe640 1 -- 172.21.15.1:0/181750520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f909c0c3210 msgr2=0x7f909c1564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:47.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.466+0000 7f909b7fe640 1 --2- 172.21.15.1:0/181750520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f909c0c3210 0x7f909c1564d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.466+0000 7f909b7fe640 1 -- 172.21.15.1:0/181750520 >> v1:172.21.15.98:6789/0 conn(0x7f909c156a10 legacy=0x7f909c0c6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.467+0000 7f909b7fe640 1 -- 172.21.15.1:0/181750520 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c1533d0 con 0x7f909c0c8590 2024-03-06T07:20:47.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.467+0000 7f909b7fe640 1 --2- 172.21.15.1:0/181750520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f909c0c8590 0x7f909c0c8970 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9090009870 tx=0x7f909002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80cc9c7e6f3c0614 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:47.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.467+0000 7f909a7fc640 1 -- 172.21.15.1:0/181750520 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090034070 con 0x7f909c0c8590 2024-03-06T07:20:47.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.467+0000 7f909a7fc640 1 -- 172.21.15.1:0/181750520 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9090030840 con 0x7f909c0c8590 2024-03-06T07:20:47.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.467+0000 7f909a7fc640 1 -- 172.21.15.1:0/181750520 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090030c10 con 0x7f909c0c8590 2024-03-06T07:20:47.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.468+0000 7f90a1a88640 1 -- 172.21.15.1:0/181750520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f909c0c8590 msgr2=0x7f909c0c8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.468+0000 7f90a1a88640 1 --2- 172.21.15.1:0/181750520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f909c0c8590 0x7f909c0c8970 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9090009870 tx=0x7f909002b980 comp rx=0 tx=0).stop 2024-03-06T07:20:47.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.468+0000 7f90a1a88640 1 -- 172.21.15.1:0/181750520 shutdown_connections 2024-03-06T07:20:47.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.468+0000 7f90a1a88640 1 --2- 172.21.15.1:0/181750520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f909c0c3210 0x7f909c1564d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.468+0000 7f90a1a88640 1 --2- 172.21.15.1:0/181750520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f909c0c8590 0x7f909c0c8970 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.468+0000 7f90a1a88640 1 -- 172.21.15.1:0/181750520 >> 172.21.15.1:0/181750520 conn(0x7f909c0aa7a0 msgr2=0x7f909c0aab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:47.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.468+0000 7f90a1a88640 1 -- 172.21.15.1:0/181750520 shutdown_connections 2024-03-06T07:20:47.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.468+0000 7f90a1a88640 1 -- 172.21.15.1:0/181750520 wait complete. 2024-03-06T07:20:47.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.469+0000 7f90a1a88640 1 Processor -- start 2024-03-06T07:20:47.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.469+0000 7f90a1a88640 1 -- start start 2024-03-06T07:20:47.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.470+0000 7f90a1a88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f909c0c3210 0x7f909c0ac990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.470+0000 7f90a1a88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f909c0c8590 0x7f909c1527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.470+0000 7f90a1a88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f909c156a10 0x7f909c1480a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.470+0000 7f90a1a88640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c0701e0 con 0x7f909c0c3210 2024-03-06T07:20:47.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.470+0000 7f90a1a88640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c070060 con 0x7f909c0c8590 2024-03-06T07:20:47.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.470+0000 7f90a1a88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909c0c3650 con 0x7f909c156a10 2024-03-06T07:20:47.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.470+0000 7f909b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f909c0c3210 0x7f909c0ac990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.471+0000 7f909b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f909c0c3210 0x7f909c0ac990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52906/0 (socket says 172.21.15.1:52906) 2024-03-06T07:20:47.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.471+0000 7f909b7fe640 1 -- 172.21.15.1:0/908958819 learned_addr learned my addr 172.21.15.1:0/908958819 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:47.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.471+0000 7f909bfff640 1 --2- 172.21.15.1:0/908958819 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f909c156a10 0x7f909c1480a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.471+0000 7f909affd640 1 --2- 172.21.15.1:0/908958819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f909c0c8590 0x7f909c1527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.471+0000 7f909b7fe640 1 -- 172.21.15.1:0/908958819 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f909c156a10 msgr2=0x7f909c1480a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.471+0000 7f909b7fe640 1 --2- 172.21.15.1:0/908958819 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f909c156a10 0x7f909c1480a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.471+0000 7f909b7fe640 1 -- 172.21.15.1:0/908958819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f909c0c8590 msgr2=0x7f909c1527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.471+0000 7f909b7fe640 1 --2- 172.21.15.1:0/908958819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f909c0c8590 0x7f909c1527f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.471+0000 7f909b7fe640 1 -- 172.21.15.1:0/908958819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c1ecb90 con 0x7f909c0c3210 2024-03-06T07:20:47.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.471+0000 7f909bfff640 1 --2- 172.21.15.1:0/908958819 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f909c156a10 0x7f909c1480a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:47.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.471+0000 7f909affd640 1 --2- 172.21.15.1:0/908958819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f909c0c8590 0x7f909c1527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:47.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.471+0000 7f909b7fe640 1 --2- 172.21.15.1:0/908958819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f909c0c3210 0x7f909c0ac990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9090009870 tx=0x7f9090004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:47.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.471+0000 7f9083fff640 1 -- 172.21.15.1:0/908958819 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090048070 con 0x7f909c0c3210 2024-03-06T07:20:47.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.471+0000 7f9083fff640 1 -- 172.21.15.1:0/908958819 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9090030640 con 0x7f909c0c3210 2024-03-06T07:20:47.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.471+0000 7f90a1a88640 1 -- 172.21.15.1:0/908958819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909c1ebdd0 con 0x7f909c0c3210 2024-03-06T07:20:47.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.471+0000 7f9083fff640 1 -- 172.21.15.1:0/908958819 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090034040 con 0x7f909c0c3210 2024-03-06T07:20:47.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.471+0000 7f90a1a88640 1 -- 172.21.15.1:0/908958819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909c1ec390 con 0x7f909c0c3210 2024-03-06T07:20:47.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.472+0000 7f90a1a88640 1 -- 172.21.15.1:0/908958819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f909c0686e0 con 0x7f909c0c3210 2024-03-06T07:20:47.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.473+0000 7f9083fff640 1 -- 172.21.15.1:0/908958819 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f909004b070 con 0x7f909c0c3210 2024-03-06T07:20:47.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.473+0000 7f9083fff640 1 --2- 172.21.15.1:0/908958819 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f908806b4d0 0x7f908806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.473+0000 7f9083fff640 1 -- 172.21.15.1:0/908958819 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f90900ac3f0 con 0x7f909c0c3210 2024-03-06T07:20:47.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.473+0000 7f909affd640 1 --2- 172.21.15.1:0/908958819 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f908806b4d0 0x7f908806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.474+0000 7f909affd640 1 --2- 172.21.15.1:0/908958819 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f908806b4d0 0x7f908806d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f909c0c4f90 tx=0x7f908c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:47.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.478+0000 7f9083fff640 1 -- 172.21.15.1:0/908958819 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f909007afc0 con 0x7f909c0c3210 2024-03-06T07:20:47.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.661+0000 7f90a1a88640 1 -- 172.21.15.1:0/908958819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f909c0c8370 con 0x7f909c0c3210 2024-03-06T07:20:47.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.661+0000 7f9083fff640 1 -- 172.21.15.1:0/908958819 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f909007a8c0 con 0x7f909c0c3210 2024-03-06T07:20:47.665 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:20:47.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.663+0000 7f90a1a88640 1 -- 172.21.15.1:0/908958819 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f908806b4d0 msgr2=0x7f908806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.663+0000 7f90a1a88640 1 --2- 172.21.15.1:0/908958819 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f908806b4d0 0x7f908806d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f909c0c4f90 tx=0x7f908c008040 comp rx=0 tx=0).stop 2024-03-06T07:20:47.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.663+0000 7f90a1a88640 1 -- 172.21.15.1:0/908958819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f909c0c3210 msgr2=0x7f909c0ac990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.663+0000 7f90a1a88640 1 --2- 172.21.15.1:0/908958819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f909c0c3210 0x7f909c0ac990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9090009870 tx=0x7f9090004730 comp rx=0 tx=0).stop 2024-03-06T07:20:47.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.664+0000 7f90a1a88640 1 -- 172.21.15.1:0/908958819 shutdown_connections 2024-03-06T07:20:47.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.664+0000 7f90a1a88640 1 --2- 172.21.15.1:0/908958819 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f908806b4d0 0x7f908806d990 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.664+0000 7f90a1a88640 1 --2- 172.21.15.1:0/908958819 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f909c156a10 0x7f909c1480a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.664+0000 7f90a1a88640 1 --2- 172.21.15.1:0/908958819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f909c0c8590 0x7f909c1527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.664+0000 7f90a1a88640 1 --2- 172.21.15.1:0/908958819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f909c0c3210 0x7f909c0ac990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.664+0000 7f90a1a88640 1 -- 172.21.15.1:0/908958819 >> 172.21.15.1:0/908958819 conn(0x7f909c0aa7a0 msgr2=0x7f909c0c6c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:47.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.664+0000 7f90a1a88640 1 -- 172.21.15.1:0/908958819 shutdown_connections 2024-03-06T07:20:47.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.664+0000 7f90a1a88640 1 -- 172.21.15.1:0/908958819 wait complete. 2024-03-06T07:20:47.676 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:20:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.782+0000 7f91b400e640 1 Processor -- start 2024-03-06T07:20:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.782+0000 7f91b400e640 1 -- start start 2024-03-06T07:20:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.782+0000 7f91b400e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f91ac159140 0x7f91ac0c4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.782+0000 7f91b400e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91ac0c69b0 0x7f91ac0c2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.782+0000 7f91b400e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac072d00 con 0x7f91ac0c7510 2024-03-06T07:20:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.782+0000 7f91b400e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91ac0c0e80 con 0x7f91ac159140 2024-03-06T07:20:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.782+0000 7f91b400e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91ac0c13b0 con 0x7f91ac0c69b0 2024-03-06T07:20:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.783+0000 7f91b1daa640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f91ac159140 0x7f91ac0c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.783+0000 7f91b15a9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91ac0c69b0 0x7f91ac0c2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.783+0000 7f91b25ab640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f91ac0c7510 0x7f91ac06aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:33820/0 (socket says 172.21.15.1:33820) 2024-03-06T07:20:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.783+0000 7f91b25ab640 1 -- 172.21.15.1:0/368891990 learned_addr learned my addr 172.21.15.1:0/368891990 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.783+0000 7f91b1daa640 1 -- 172.21.15.1:0/368891990 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91ac0c69b0 msgr2=0x7f91ac0c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.783+0000 7f91b1daa640 1 --2- 172.21.15.1:0/368891990 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91ac0c69b0 0x7f91ac0c2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.783+0000 7f91b1daa640 1 -- 172.21.15.1:0/368891990 >> v1:172.21.15.98:6789/0 conn(0x7f91ac0c7510 legacy=0x7f91ac06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.783+0000 7f91b1daa640 1 -- 172.21.15.1:0/368891990 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ac073180 con 0x7f91ac159140 2024-03-06T07:20:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.783+0000 7f91b0da8640 1 -- 172.21.15.1:0/368891990 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516878143 0 0) 0x7f91ac072d00 con 0x7f91ac0c7510 2024-03-06T07:20:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.784+0000 7f91b1daa640 1 --2- 172.21.15.1:0/368891990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ac159140 0x7f91ac0c4490 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f91a80097a0 tx=0x7f91a802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3975e5eff0c29cc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.784+0000 7f91b0da8640 1 -- 172.21.15.1:0/368891990 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a80315a0 con 0x7f91ac159140 2024-03-06T07:20:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.784+0000 7f91b0da8640 1 -- 172.21.15.1:0/368891990 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a8037070 con 0x7f91ac159140 2024-03-06T07:20:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.784+0000 7f91b0da8640 1 -- 172.21.15.1:0/368891990 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a8031960 con 0x7f91ac159140 2024-03-06T07:20:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.785+0000 7f91b400e640 1 -- 172.21.15.1:0/368891990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ac159140 msgr2=0x7f91ac0c4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.785+0000 7f91b400e640 1 --2- 172.21.15.1:0/368891990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ac159140 0x7f91ac0c4490 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f91a80097a0 tx=0x7f91a802b840 comp rx=0 tx=0).stop 2024-03-06T07:20:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.785+0000 7f91b400e640 1 -- 172.21.15.1:0/368891990 shutdown_connections 2024-03-06T07:20:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.785+0000 7f91b400e640 1 --2- 172.21.15.1:0/368891990 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91ac0c69b0 0x7f91ac0c2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.785+0000 7f91b400e640 1 --2- 172.21.15.1:0/368891990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ac159140 0x7f91ac0c4490 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.785+0000 7f91b400e640 1 -- 172.21.15.1:0/368891990 >> 172.21.15.1:0/368891990 conn(0x7f91ac0c7a50 msgr2=0x7f91ac0c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.785+0000 7f91b400e640 1 -- 172.21.15.1:0/368891990 shutdown_connections 2024-03-06T07:20:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.785+0000 7f91b400e640 1 -- 172.21.15.1:0/368891990 wait complete. 2024-03-06T07:20:47.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.787+0000 7f91b400e640 1 Processor -- start 2024-03-06T07:20:47.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.787+0000 7f91b400e640 1 -- start start 2024-03-06T07:20:47.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.788+0000 7f91b400e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ac0c69b0 0x7f91ac1ee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.788+0000 7f91b400e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91ac0c7510 0x7f91ac0ac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.788+0000 7f91b400e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91ac159140 0x7f91ac14cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.788+0000 7f91b400e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ac0c10d0 con 0x7f91ac159140 2024-03-06T07:20:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.788+0000 7f91b400e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ac0c6520 con 0x7f91ac0c69b0 2024-03-06T07:20:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.788+0000 7f91b400e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91ac159520 con 0x7f91ac0c7510 2024-03-06T07:20:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.788+0000 7f91b25ab640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91ac159140 0x7f91ac14cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.788+0000 7f91b25ab640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91ac159140 0x7f91ac14cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52910/0 (socket says 172.21.15.1:52910) 2024-03-06T07:20:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.788+0000 7f91b25ab640 1 -- 172.21.15.1:0/2584521604 learned_addr learned my addr 172.21.15.1:0/2584521604 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.788+0000 7f91b1daa640 1 --2- 172.21.15.1:0/2584521604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ac0c69b0 0x7f91ac1ee7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.788+0000 7f91b15a9640 1 --2- 172.21.15.1:0/2584521604 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91ac0c7510 0x7f91ac0ac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.788+0000 7f91b25ab640 1 -- 172.21.15.1:0/2584521604 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91ac0c7510 msgr2=0x7f91ac0ac1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.788+0000 7f91b25ab640 1 --2- 172.21.15.1:0/2584521604 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91ac0c7510 0x7f91ac0ac1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.788+0000 7f91b25ab640 1 -- 172.21.15.1:0/2584521604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ac0c69b0 msgr2=0x7f91ac1ee7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.788+0000 7f91b25ab640 1 --2- 172.21.15.1:0/2584521604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ac0c69b0 0x7f91ac1ee7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.788+0000 7f91b25ab640 1 -- 172.21.15.1:0/2584521604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ac14de70 con 0x7f91ac159140 2024-03-06T07:20:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.788+0000 7f91b25ab640 1 --2- 172.21.15.1:0/2584521604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91ac159140 0x7f91ac14cb50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f91a400caf0 tx=0x7f91a400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:47.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.789+0000 7f91a27fc640 1 -- 172.21.15.1:0/2584521604 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a400aca0 con 0x7f91ac159140 2024-03-06T07:20:47.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.789+0000 7f91a27fc640 1 -- 172.21.15.1:0/2584521604 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a4012400 con 0x7f91ac159140 2024-03-06T07:20:47.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.789+0000 7f91a27fc640 1 -- 172.21.15.1:0/2584521604 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a4012830 con 0x7f91ac159140 2024-03-06T07:20:47.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.789+0000 7f91b400e640 1 -- 172.21.15.1:0/2584521604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ac14e150 con 0x7f91ac159140 2024-03-06T07:20:47.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.789+0000 7f91b400e640 1 -- 172.21.15.1:0/2584521604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ac0ac710 con 0x7f91ac159140 2024-03-06T07:20:47.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.790+0000 7f91b400e640 1 -- 172.21.15.1:0/2584521604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91ac0686e0 con 0x7f91ac159140 2024-03-06T07:20:47.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.790+0000 7f91a27fc640 1 -- 172.21.15.1:0/2584521604 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91a40129d0 con 0x7f91ac159140 2024-03-06T07:20:47.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.791+0000 7f91a27fc640 1 --2- 172.21.15.1:0/2584521604 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f919006b4d0 0x7f919006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.791+0000 7f91a27fc640 1 -- 172.21.15.1:0/2584521604 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f91a408ab50 con 0x7f91ac159140 2024-03-06T07:20:47.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.791+0000 7f91b1daa640 1 --2- 172.21.15.1:0/2584521604 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f919006b4d0 0x7f919006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.791+0000 7f91b1daa640 1 --2- 172.21.15.1:0/2584521604 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f919006b4d0 0x7f919006d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f91a8036000 tx=0x7f91a8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:47.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.795+0000 7f91a27fc640 1 -- 172.21.15.1:0/2584521604 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91a4059720 con 0x7f91ac159140 2024-03-06T07:20:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.978+0000 7f91b400e640 1 -- 172.21.15.1:0/2584521604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f91ac072f60 con 0x7f91ac159140 2024-03-06T07:20:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.978+0000 7f91a27fc640 1 -- 172.21.15.1:0/2584521604 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f91a4059020 con 0x7f91ac159140 2024-03-06T07:20:47.980 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:20:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.980+0000 7f91b400e640 1 -- 172.21.15.1:0/2584521604 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f919006b4d0 msgr2=0x7f919006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.980+0000 7f91b400e640 1 --2- 172.21.15.1:0/2584521604 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f919006b4d0 0x7f919006d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f91a8036000 tx=0x7f91a8008030 comp rx=0 tx=0).stop 2024-03-06T07:20:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.980+0000 7f91b400e640 1 -- 172.21.15.1:0/2584521604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91ac159140 msgr2=0x7f91ac14cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.980+0000 7f91b400e640 1 --2- 172.21.15.1:0/2584521604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91ac159140 0x7f91ac14cb50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f91a400caf0 tx=0x7f91a400a060 comp rx=0 tx=0).stop 2024-03-06T07:20:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.980+0000 7f91b400e640 1 -- 172.21.15.1:0/2584521604 shutdown_connections 2024-03-06T07:20:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.980+0000 7f91b400e640 1 --2- 172.21.15.1:0/2584521604 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f919006b4d0 0x7f919006d990 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.980+0000 7f91b400e640 1 --2- 172.21.15.1:0/2584521604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91ac159140 0x7f91ac14cb50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.980+0000 7f91b400e640 1 --2- 172.21.15.1:0/2584521604 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91ac0c7510 0x7f91ac0ac1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.980+0000 7f91b400e640 1 --2- 172.21.15.1:0/2584521604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ac0c69b0 0x7f91ac1ee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.980+0000 7f91b400e640 1 -- 172.21.15.1:0/2584521604 >> 172.21.15.1:0/2584521604 conn(0x7f91ac0c7a50 msgr2=0x7f91ac0ab340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.980+0000 7f91b400e640 1 -- 172.21.15.1:0/2584521604 shutdown_connections 2024-03-06T07:20:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:47.980+0000 7f91b400e640 1 -- 172.21.15.1:0/2584521604 wait complete. 2024-03-06T07:20:47.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:20:48.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.095+0000 7f87ddbbc640 1 Processor -- start 2024-03-06T07:20:48.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.095+0000 7f87ddbbc640 1 -- start start 2024-03-06T07:20:48.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.096+0000 7f87ddbbc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87d814cf90 0x7f87d8156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.096+0000 7f87ddbbc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87d8070be0 0x7f87d8071090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.096+0000 7f87ddbbc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d8154a70 con 0x7f87d80c7cc0 2024-03-06T07:20:48.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.096+0000 7f87ddbbc640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d814d370 con 0x7f87d8070be0 2024-03-06T07:20:48.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.096+0000 7f87ddbbc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d80c6190 con 0x7f87d814cf90 2024-03-06T07:20:48.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.096+0000 7f87d7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87d8070be0 0x7f87d8071090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.096+0000 7f87d6ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f87d80c7cc0 0x7f87d8154360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46880/0 (socket says 172.21.15.1:46880) 2024-03-06T07:20:48.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.096+0000 7f87d6ffd640 1 -- 172.21.15.1:0/340724172 learned_addr learned my addr 172.21.15.1:0/340724172 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:48.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.096+0000 7f87d77fe640 1 --2- 172.21.15.1:0/340724172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87d814cf90 0x7f87d8156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.096+0000 7f87d7fff640 1 -- 172.21.15.1:0/340724172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87d814cf90 msgr2=0x7f87d8156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:48.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.096+0000 7f87d7fff640 1 --2- 172.21.15.1:0/340724172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87d814cf90 0x7f87d8156fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.096+0000 7f87d7fff640 1 -- 172.21.15.1:0/340724172 >> v1:172.21.15.1:6789/0 conn(0x7f87d80c7cc0 legacy=0x7f87d8154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.096+0000 7f87d7fff640 1 -- 172.21.15.1:0/340724172 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d8153a90 con 0x7f87d8070be0 2024-03-06T07:20:48.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.096+0000 7f87d67fc640 1 -- 172.21.15.1:0/340724172 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784897280 0 0) 0x7f87d8154a70 con 0x7f87d80c7cc0 2024-03-06T07:20:48.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.097+0000 7f87d7fff640 1 --2- 172.21.15.1:0/340724172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87d8070be0 0x7f87d8071090 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f87d0009800 tx=0x7f87d002b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97910ce509895035 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.097+0000 7f87d67fc640 1 -- 172.21.15.1:0/340724172 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d0039070 con 0x7f87d8070be0 2024-03-06T07:20:48.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.098+0000 7f87d67fc640 1 -- 172.21.15.1:0/340724172 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d0030070 con 0x7f87d8070be0 2024-03-06T07:20:48.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.098+0000 7f87d67fc640 1 -- 172.21.15.1:0/340724172 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d00318f0 con 0x7f87d8070be0 2024-03-06T07:20:48.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.098+0000 7f87ddbbc640 1 -- 172.21.15.1:0/340724172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87d8070be0 msgr2=0x7f87d8071090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.098+0000 7f87ddbbc640 1 --2- 172.21.15.1:0/340724172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87d8070be0 0x7f87d8071090 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f87d0009800 tx=0x7f87d002b770 comp rx=0 tx=0).stop 2024-03-06T07:20:48.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.098+0000 7f87ddbbc640 1 -- 172.21.15.1:0/340724172 shutdown_connections 2024-03-06T07:20:48.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.098+0000 7f87ddbbc640 1 --2- 172.21.15.1:0/340724172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87d8070be0 0x7f87d8071090 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.098+0000 7f87ddbbc640 1 --2- 172.21.15.1:0/340724172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87d814cf90 0x7f87d8156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.098+0000 7f87ddbbc640 1 -- 172.21.15.1:0/340724172 >> 172.21.15.1:0/340724172 conn(0x7f87d80c88f0 msgr2=0x7f87d80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:48.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.098+0000 7f87ddbbc640 1 -- 172.21.15.1:0/340724172 shutdown_connections 2024-03-06T07:20:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.098+0000 7f87ddbbc640 1 -- 172.21.15.1:0/340724172 wait complete. 2024-03-06T07:20:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.099+0000 7f87ddbbc640 1 Processor -- start 2024-03-06T07:20:48.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.099+0000 7f87ddbbc640 1 -- start start 2024-03-06T07:20:48.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.100+0000 7f87ddbbc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87d8070be0 0x7f87d81e9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.100+0000 7f87ddbbc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87d80c7cc0 0x7f87d80aae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.100+0000 7f87ddbbc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87d814cf90 0x7f87d81ebdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.100+0000 7f87ddbbc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d81567e0 con 0x7f87d8070be0 2024-03-06T07:20:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.100+0000 7f87ddbbc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d8156500 con 0x7f87d80c7cc0 2024-03-06T07:20:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.100+0000 7f87ddbbc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d8156ac0 con 0x7f87d814cf90 2024-03-06T07:20:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.100+0000 7f87d77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87d8070be0 0x7f87d81e9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.100+0000 7f87d77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87d8070be0 0x7f87d81e9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52922/0 (socket says 172.21.15.1:52922) 2024-03-06T07:20:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.100+0000 7f87d77fe640 1 -- 172.21.15.1:0/542208008 learned_addr learned my addr 172.21.15.1:0/542208008 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.100+0000 7f87d77fe640 1 -- 172.21.15.1:0/542208008 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87d814cf90 msgr2=0x7f87d81ebdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.100+0000 7f87d77fe640 1 --2- 172.21.15.1:0/542208008 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87d814cf90 0x7f87d81ebdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.100+0000 7f87d77fe640 1 -- 172.21.15.1:0/542208008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87d80c7cc0 msgr2=0x7f87d80aae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.100+0000 7f87d77fe640 1 --2- 172.21.15.1:0/542208008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87d80c7cc0 0x7f87d80aae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.100+0000 7f87d77fe640 1 -- 172.21.15.1:0/542208008 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d80ab380 con 0x7f87d8070be0 2024-03-06T07:20:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.101+0000 7f87d77fe640 1 --2- 172.21.15.1:0/542208008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87d8070be0 0x7f87d81e9ac0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f87c8009d10 tx=0x7f87c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.101+0000 7f87bffff640 1 -- 172.21.15.1:0/542208008 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8018020 con 0x7f87d8070be0 2024-03-06T07:20:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.101+0000 7f87bffff640 1 -- 172.21.15.1:0/542208008 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c8014480 con 0x7f87d8070be0 2024-03-06T07:20:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.101+0000 7f87ddbbc640 1 -- 172.21.15.1:0/542208008 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d814d6d0 con 0x7f87d8070be0 2024-03-06T07:20:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.101+0000 7f87bffff640 1 -- 172.21.15.1:0/542208008 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8014830 con 0x7f87d8070be0 2024-03-06T07:20:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.101+0000 7f87ddbbc640 1 -- 172.21.15.1:0/542208008 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d81e02d0 con 0x7f87d8070be0 2024-03-06T07:20:48.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.102+0000 7f87ddbbc640 1 -- 172.21.15.1:0/542208008 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d80686e0 con 0x7f87d8070be0 2024-03-06T07:20:48.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.102+0000 7f87bffff640 1 -- 172.21.15.1:0/542208008 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87c8005d10 con 0x7f87d8070be0 2024-03-06T07:20:48.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.103+0000 7f87bffff640 1 --2- 172.21.15.1:0/542208008 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f87c406b4d0 0x7f87c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.103+0000 7f87bffff640 1 -- 172.21.15.1:0/542208008 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f87c808b1d0 con 0x7f87d8070be0 2024-03-06T07:20:48.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.103+0000 7f87d6ffd640 1 --2- 172.21.15.1:0/542208008 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f87c406b4d0 0x7f87c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.103+0000 7f87d6ffd640 1 --2- 172.21.15.1:0/542208008 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f87c406b4d0 0x7f87c406d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f87cc007fd0 tx=0x7f87cc007920 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.107+0000 7f87bffff640 1 -- 172.21.15.1:0/542208008 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87c8022030 con 0x7f87d8070be0 2024-03-06T07:20:48.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.300+0000 7f87ddbbc640 1 -- 172.21.15.1:0/542208008 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f87d80c7aa0 con 0x7f87d8070be0 2024-03-06T07:20:48.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.300+0000 7f87bffff640 1 -- 172.21.15.1:0/542208008 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f87c8010040 con 0x7f87d8070be0 2024-03-06T07:20:48.302 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:20:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.304+0000 7f87ddbbc640 1 -- 172.21.15.1:0/542208008 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f87c406b4d0 msgr2=0x7f87c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.304+0000 7f87ddbbc640 1 --2- 172.21.15.1:0/542208008 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f87c406b4d0 0x7f87c406d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f87cc007fd0 tx=0x7f87cc007920 comp rx=0 tx=0).stop 2024-03-06T07:20:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.304+0000 7f87ddbbc640 1 -- 172.21.15.1:0/542208008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87d8070be0 msgr2=0x7f87d81e9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.304+0000 7f87ddbbc640 1 --2- 172.21.15.1:0/542208008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87d8070be0 0x7f87d81e9ac0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f87c8009d10 tx=0x7f87c800c490 comp rx=0 tx=0).stop 2024-03-06T07:20:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.304+0000 7f87ddbbc640 1 -- 172.21.15.1:0/542208008 shutdown_connections 2024-03-06T07:20:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.304+0000 7f87ddbbc640 1 --2- 172.21.15.1:0/542208008 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f87c406b4d0 0x7f87c406d990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.304+0000 7f87ddbbc640 1 --2- 172.21.15.1:0/542208008 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87d814cf90 0x7f87d81ebdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.304+0000 7f87ddbbc640 1 --2- 172.21.15.1:0/542208008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87d80c7cc0 0x7f87d80aae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.304+0000 7f87ddbbc640 1 --2- 172.21.15.1:0/542208008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87d8070be0 0x7f87d81e9ac0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.304+0000 7f87ddbbc640 1 -- 172.21.15.1:0/542208008 >> 172.21.15.1:0/542208008 conn(0x7f87d80c88f0 msgr2=0x7f87d814afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.304+0000 7f87ddbbc640 1 -- 172.21.15.1:0/542208008 shutdown_connections 2024-03-06T07:20:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.304+0000 7f87ddbbc640 1 -- 172.21.15.1:0/542208008 wait complete. 2024-03-06T07:20:48.316 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:48.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.417+0000 7ff20b64e640 1 Processor -- start 2024-03-06T07:20:48.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.417+0000 7ff20b64e640 1 -- start start 2024-03-06T07:20:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.417+0000 7ff20b64e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2040c32f0 0x7ff2041543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.417+0000 7ff20b64e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff204070c80 0x7ff2040710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.417+0000 7ff20b64e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff204150900 con 0x7ff204150520 2024-03-06T07:20:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.417+0000 7ff20b64e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff204153ed0 con 0x7ff204070c80 2024-03-06T07:20:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.417+0000 7ff20b64e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff204071790 con 0x7ff2040c32f0 2024-03-06T07:20:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.417+0000 7ff209beb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff204070c80 0x7ff2040710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.417+0000 7ff209beb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff204070c80 0x7ff2040710f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52930/0 (socket says 172.21.15.1:52930) 2024-03-06T07:20:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.417+0000 7ff209beb640 1 -- 172.21.15.1:0/1745958275 learned_addr learned my addr 172.21.15.1:0/1745958275 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.417+0000 7ff2093ea640 1 --1- 172.21.15.1:0/1745958275 >> v1:172.21.15.1:6789/0 conn(0x7ff204150520 0x7ff2041537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46886/0 (socket says 172.21.15.1:46886) 2024-03-06T07:20:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.417+0000 7ff208be9640 1 --2- 172.21.15.1:0/1745958275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2040c32f0 0x7ff2041543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.418+0000 7ff209beb640 1 -- 172.21.15.1:0/1745958275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2040c32f0 msgr2=0x7ff2041543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.418+0000 7ff209beb640 1 --2- 172.21.15.1:0/1745958275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2040c32f0 0x7ff2041543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.418+0000 7ff209beb640 1 -- 172.21.15.1:0/1745958275 >> v1:172.21.15.1:6789/0 conn(0x7ff204150520 legacy=0x7ff2041537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.418+0000 7ff209beb640 1 -- 172.21.15.1:0/1745958275 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff204157050 con 0x7ff204070c80 2024-03-06T07:20:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.419+0000 7ff209beb640 1 --2- 172.21.15.1:0/1745958275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff204070c80 0x7ff2040710f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc009e50 tx=0x7ff1fc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5298c3d40c5f729a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.419+0000 7ff1fbfff640 1 -- 172.21.15.1:0/1745958275 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc02cb00 con 0x7ff204070c80 2024-03-06T07:20:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.419+0000 7ff1fbfff640 1 -- 172.21.15.1:0/1745958275 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1fc02cca0 con 0x7ff204070c80 2024-03-06T07:20:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.419+0000 7ff1fbfff640 1 -- 172.21.15.1:0/1745958275 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc034610 con 0x7ff204070c80 2024-03-06T07:20:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.419+0000 7ff20b64e640 1 -- 172.21.15.1:0/1745958275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff204070c80 msgr2=0x7ff2040710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.419+0000 7ff20b64e640 1 --2- 172.21.15.1:0/1745958275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff204070c80 0x7ff2040710f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc009e50 tx=0x7ff1fc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:20:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.419+0000 7ff20b64e640 1 -- 172.21.15.1:0/1745958275 shutdown_connections 2024-03-06T07:20:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.419+0000 7ff20b64e640 1 --2- 172.21.15.1:0/1745958275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2040c32f0 0x7ff2041543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.419+0000 7ff20b64e640 1 --2- 172.21.15.1:0/1745958275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff204070c80 0x7ff2040710f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.419+0000 7ff20b64e640 1 -- 172.21.15.1:0/1745958275 >> 172.21.15.1:0/1745958275 conn(0x7ff2040c88f0 msgr2=0x7ff2040c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.419+0000 7ff20b64e640 1 -- 172.21.15.1:0/1745958275 shutdown_connections 2024-03-06T07:20:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.419+0000 7ff20b64e640 1 -- 172.21.15.1:0/1745958275 wait complete. 2024-03-06T07:20:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.421+0000 7ff20b64e640 1 Processor -- start 2024-03-06T07:20:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.421+0000 7ff20b64e640 1 -- start start 2024-03-06T07:20:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff20b64e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff204070c80 0x7ff204147ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff20b64e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2040c32f0 0x7ff204143690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff20b64e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff204150520 0x7ff204143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff20b64e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff204157330 con 0x7ff2040c32f0 2024-03-06T07:20:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff20b64e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2041532a0 con 0x7ff204150520 2024-03-06T07:20:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff20b64e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040c7ec0 con 0x7ff204070c80 2024-03-06T07:20:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff208be9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2040c32f0 0x7ff204143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff208be9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2040c32f0 0x7ff204143690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52938/0 (socket says 172.21.15.1:52938) 2024-03-06T07:20:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff208be9640 1 -- 172.21.15.1:0/3503022945 learned_addr learned my addr 172.21.15.1:0/3503022945 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff2093ea640 1 --2- 172.21.15.1:0/3503022945 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff204070c80 0x7ff204147ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff209beb640 1 --2- 172.21.15.1:0/3503022945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff204150520 0x7ff204143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff208be9640 1 -- 172.21.15.1:0/3503022945 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff204070c80 msgr2=0x7ff204147ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff208be9640 1 --2- 172.21.15.1:0/3503022945 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff204070c80 0x7ff204147ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff208be9640 1 -- 172.21.15.1:0/3503022945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff204150520 msgr2=0x7ff204143bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff208be9640 1 --2- 172.21.15.1:0/3503022945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff204150520 0x7ff204143bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff208be9640 1 -- 172.21.15.1:0/3503022945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2041dffe0 con 0x7ff2040c32f0 2024-03-06T07:20:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff209beb640 1 --2- 172.21.15.1:0/3503022945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff204150520 0x7ff204143bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff208be9640 1 --2- 172.21.15.1:0/3503022945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2040c32f0 0x7ff204143690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400e9b0 tx=0x7ff1f400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff1f9ffb640 1 -- 172.21.15.1:0/3503022945 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4017070 con 0x7ff2040c32f0 2024-03-06T07:20:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff1f9ffb640 1 -- 172.21.15.1:0/3503022945 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f400c9a0 con 0x7ff2040c32f0 2024-03-06T07:20:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff20b64e640 1 -- 172.21.15.1:0/3503022945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff204149bc0 con 0x7ff2040c32f0 2024-03-06T07:20:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff1f9ffb640 1 -- 172.21.15.1:0/3503022945 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f400cd90 con 0x7ff2040c32f0 2024-03-06T07:20:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.423+0000 7ff20b64e640 1 -- 172.21.15.1:0/3503022945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff204148420 con 0x7ff2040c32f0 2024-03-06T07:20:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.424+0000 7ff20b64e640 1 -- 172.21.15.1:0/3503022945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2040686e0 con 0x7ff2040c32f0 2024-03-06T07:20:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.425+0000 7ff1f9ffb640 1 -- 172.21.15.1:0/3503022945 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1f401a070 con 0x7ff2040c32f0 2024-03-06T07:20:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.425+0000 7ff1f9ffb640 1 --2- 172.21.15.1:0/3503022945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1e806b4d0 0x7ff1e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.425+0000 7ff1f9ffb640 1 -- 172.21.15.1:0/3503022945 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff1f408a8f0 con 0x7ff2040c32f0 2024-03-06T07:20:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.425+0000 7ff2093ea640 1 --2- 172.21.15.1:0/3503022945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1e806b4d0 0x7ff1e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.426+0000 7ff2093ea640 1 --2- 172.21.15.1:0/3503022945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1e806b4d0 0x7ff1e806d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff200000f60 tx=0x7ff200007ce0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.430+0000 7ff1f9ffb640 1 -- 172.21.15.1:0/3503022945 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1f4059440 con 0x7ff2040c32f0 2024-03-06T07:20:48.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.647+0000 7ff20b64e640 1 -- 172.21.15.1:0/3503022945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2040c7b10 con 0x7ff2040c32f0 2024-03-06T07:20:48.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.648+0000 7ff1f9ffb640 1 -- 172.21.15.1:0/3503022945 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4964 (secure 0 0 0) 0x7ff1f4058d40 con 0x7ff2040c32f0 2024-03-06T07:20:48.650 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:48.650 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2822958969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2822958969},{"type":"v1","addr":"172.21.15.98:6836","nonce":2822958969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2442906233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2442906233},{"type":"v1","addr":"172.21.15.98:6837","nonce":2442906233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/448746024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":448746024},{"type":"v1","addr":"172.21.15.1:6834","nonce":448746024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:21.207834+0000","modified":"2024-03-06T07:20:37.259661+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1315115982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1315115982},{"type":"v1","addr":"172.21.15.1:6835","nonce":1315115982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-03-06T07:20:48.650 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-03-06T07:20:48.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.651+0000 7ff20b64e640 1 -- 172.21.15.1:0/3503022945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1e806b4d0 msgr2=0x7ff1e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.651+0000 7ff20b64e640 1 --2- 172.21.15.1:0/3503022945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1e806b4d0 0x7ff1e806d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff200000f60 tx=0x7ff200007ce0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.651+0000 7ff20b64e640 1 -- 172.21.15.1:0/3503022945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2040c32f0 msgr2=0x7ff204143690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.651+0000 7ff20b64e640 1 --2- 172.21.15.1:0/3503022945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2040c32f0 0x7ff204143690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400e9b0 tx=0x7ff1f400ee70 comp rx=0 tx=0).stop 2024-03-06T07:20:48.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.651+0000 7ff20b64e640 1 -- 172.21.15.1:0/3503022945 shutdown_connections 2024-03-06T07:20:48.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.651+0000 7ff20b64e640 1 --2- 172.21.15.1:0/3503022945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1e806b4d0 0x7ff1e806d990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.651+0000 7ff20b64e640 1 --2- 172.21.15.1:0/3503022945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff204150520 0x7ff204143bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.651+0000 7ff20b64e640 1 --2- 172.21.15.1:0/3503022945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2040c32f0 0x7ff204143690 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.651+0000 7ff20b64e640 1 --2- 172.21.15.1:0/3503022945 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff204070c80 0x7ff204147ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.651+0000 7ff20b64e640 1 -- 172.21.15.1:0/3503022945 >> 172.21.15.1:0/3503022945 conn(0x7ff2040c88f0 msgr2=0x7ff2040c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:48.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.651+0000 7ff20b64e640 1 -- 172.21.15.1:0/3503022945 shutdown_connections 2024-03-06T07:20:48.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.651+0000 7ff20b64e640 1 -- 172.21.15.1:0/3503022945 wait complete. 2024-03-06T07:20:48.663 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:48.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.765+0000 7f96eae81640 1 Processor -- start 2024-03-06T07:20:48.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.765+0000 7f96eae81640 1 -- start start 2024-03-06T07:20:48.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.765+0000 7f96eae81640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f96e40c3970 0x7f96e40c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.766+0000 7f96eae81640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e40c5e40 con 0x7f96e40c60b0 2024-03-06T07:20:48.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.766+0000 7f96eae81640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e40c6a40 con 0x7f96e4152840 2024-03-06T07:20:48.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.766+0000 7f96eae81640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96e40c67f0 con 0x7f96e40c3970 2024-03-06T07:20:48.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.766+0000 7f96e8c1d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f96e40c3970 0x7f96e40c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.766+0000 7f96dbfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f96e40c60b0 0x7f96e4152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46890/0 (socket says 172.21.15.1:46890) 2024-03-06T07:20:48.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.766+0000 7f96dbfff640 1 -- 172.21.15.1:0/3882260526 learned_addr learned my addr 172.21.15.1:0/3882260526 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:48.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.766+0000 7f96e8c1d640 1 -- 172.21.15.1:0/3882260526 >> v1:172.21.15.98:6789/0 conn(0x7f96e4152840 legacy=0x7f96e41553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.766+0000 7f96e8c1d640 1 -- 172.21.15.1:0/3882260526 >> v1:172.21.15.1:6789/0 conn(0x7f96e40c60b0 legacy=0x7f96e4152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.766+0000 7f96db7fe640 1 -- 172.21.15.1:0/3882260526 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653006940 0 0) 0x7f96e40c5e40 con 0x7f96e40c60b0 2024-03-06T07:20:48.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.766+0000 7f96e8c1d640 1 -- 172.21.15.1:0/3882260526 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e40c6d20 con 0x7f96e40c3970 2024-03-06T07:20:48.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.767+0000 7f96e8c1d640 1 --2- 172.21.15.1:0/3882260526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96e40c3970 0x7f96e40c8170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f96e00097a0 tx=0x7f96e002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b5225a7ef319d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.767+0000 7f96db7fe640 1 -- 172.21.15.1:0/3882260526 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e0034070 con 0x7f96e40c3970 2024-03-06T07:20:48.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.767+0000 7f96db7fe640 1 -- 172.21.15.1:0/3882260526 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e0030610 con 0x7f96e40c3970 2024-03-06T07:20:48.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.767+0000 7f96db7fe640 1 -- 172.21.15.1:0/3882260526 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e00309e0 con 0x7f96e40c3970 2024-03-06T07:20:48.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.767+0000 7f96eae81640 1 -- 172.21.15.1:0/3882260526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96e40c3970 msgr2=0x7f96e40c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.767+0000 7f96eae81640 1 --2- 172.21.15.1:0/3882260526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96e40c3970 0x7f96e40c8170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f96e00097a0 tx=0x7f96e002b960 comp rx=0 tx=0).stop 2024-03-06T07:20:48.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.767+0000 7f96eae81640 1 -- 172.21.15.1:0/3882260526 shutdown_connections 2024-03-06T07:20:48.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.767+0000 7f96eae81640 1 --2- 172.21.15.1:0/3882260526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96e40c3970 0x7f96e40c8170 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.767+0000 7f96eae81640 1 -- 172.21.15.1:0/3882260526 >> 172.21.15.1:0/3882260526 conn(0x7f96e40c88f0 msgr2=0x7f96e40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:48.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.768+0000 7f96eae81640 1 -- 172.21.15.1:0/3882260526 shutdown_connections 2024-03-06T07:20:48.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.768+0000 7f96eae81640 1 -- 172.21.15.1:0/3882260526 wait complete. 2024-03-06T07:20:48.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.769+0000 7f96eae81640 1 Processor -- start 2024-03-06T07:20:48.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.769+0000 7f96eae81640 1 -- start start 2024-03-06T07:20:48.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.770+0000 7f96eae81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e40c3970 0x7f96e4153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.770+0000 7f96eae81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96e40c60b0 0x7f96e41546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.770+0000 7f96eae81640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96e4152840 0x7f96e414c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.770+0000 7f96eae81640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e40c75c0 con 0x7f96e4152840 2024-03-06T07:20:48.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.770+0000 7f96eae81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e40c72e0 con 0x7f96e40c3970 2024-03-06T07:20:48.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.770+0000 7f96eae81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e40c3d50 con 0x7f96e40c60b0 2024-03-06T07:20:48.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.771+0000 7f96e941e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96e4152840 0x7f96e414c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.771+0000 7f96e941e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96e4152840 0x7f96e414c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52956/0 (socket says 172.21.15.1:52956) 2024-03-06T07:20:48.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.771+0000 7f96e941e640 1 -- 172.21.15.1:0/3325263741 learned_addr learned my addr 172.21.15.1:0/3325263741 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:48.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.771+0000 7f96dbfff640 1 --2- 172.21.15.1:0/3325263741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96e40c60b0 0x7f96e41546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.771+0000 7f96e8c1d640 1 --2- 172.21.15.1:0/3325263741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e40c3970 0x7f96e4153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.771+0000 7f96e941e640 1 -- 172.21.15.1:0/3325263741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96e40c60b0 msgr2=0x7f96e41546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.771+0000 7f96e941e640 1 --2- 172.21.15.1:0/3325263741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96e40c60b0 0x7f96e41546c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.771+0000 7f96e941e640 1 -- 172.21.15.1:0/3325263741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e40c3970 msgr2=0x7f96e4153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.771+0000 7f96e941e640 1 --2- 172.21.15.1:0/3325263741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e40c3970 0x7f96e4153750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.771+0000 7f96e941e640 1 -- 172.21.15.1:0/3325263741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e41ea570 con 0x7f96e4152840 2024-03-06T07:20:48.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.771+0000 7f96e941e640 1 --2- 172.21.15.1:0/3325263741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96e4152840 0x7f96e414c850 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00c680 tx=0x7f96dc00cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.771+0000 7f96d97fa640 1 -- 172.21.15.1:0/3325263741 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96dc003b70 con 0x7f96e4152840 2024-03-06T07:20:48.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.771+0000 7f96d97fa640 1 -- 172.21.15.1:0/3325263741 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96dc003d10 con 0x7f96e4152840 2024-03-06T07:20:48.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.771+0000 7f96d97fa640 1 -- 172.21.15.1:0/3325263741 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96dc005310 con 0x7f96e4152840 2024-03-06T07:20:48.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.771+0000 7f96eae81640 1 -- 172.21.15.1:0/3325263741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e41ea7f0 con 0x7f96e4152840 2024-03-06T07:20:48.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.771+0000 7f96eae81640 1 -- 172.21.15.1:0/3325263741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e41eadb0 con 0x7f96e4152840 2024-03-06T07:20:48.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.772+0000 7f96eae81640 1 -- 172.21.15.1:0/3325263741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e40686e0 con 0x7f96e4152840 2024-03-06T07:20:48.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.773+0000 7f96d97fa640 1 -- 172.21.15.1:0/3325263741 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96dc0054b0 con 0x7f96e4152840 2024-03-06T07:20:48.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.773+0000 7f96d97fa640 1 --2- 172.21.15.1:0/3325263741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f96d006b4d0 0x7f96d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.773+0000 7f96d97fa640 1 -- 172.21.15.1:0/3325263741 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f96dc08a240 con 0x7f96e4152840 2024-03-06T07:20:48.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.773+0000 7f96e8c1d640 1 --2- 172.21.15.1:0/3325263741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f96d006b4d0 0x7f96d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.774+0000 7f96e8c1d640 1 --2- 172.21.15.1:0/3325263741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f96d006b4d0 0x7f96d006d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f96e0036000 tx=0x7f96e0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.778+0000 7f96d97fa640 1 -- 172.21.15.1:0/3325263741 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96e41ea570 con 0x7f96e4152840 2024-03-06T07:20:48.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.961+0000 7f96eae81640 1 -- 172.21.15.1:0/3325263741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96e4155cb0 con 0x7f96e4152840 2024-03-06T07:20:48.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.962+0000 7f96d97fa640 1 -- 172.21.15.1:0/3325263741 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7f96dc058720 con 0x7f96e4152840 2024-03-06T07:20:48.965 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:48.965 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.964+0000 7f96eae81640 1 -- 172.21.15.1:0/3325263741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f96d006b4d0 msgr2=0x7f96d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.964+0000 7f96eae81640 1 --2- 172.21.15.1:0/3325263741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f96d006b4d0 0x7f96d006d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f96e0036000 tx=0x7f96e0008030 comp rx=0 tx=0).stop 2024-03-06T07:20:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.964+0000 7f96eae81640 1 -- 172.21.15.1:0/3325263741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96e4152840 msgr2=0x7f96e414c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.964+0000 7f96eae81640 1 --2- 172.21.15.1:0/3325263741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96e4152840 0x7f96e414c850 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00c680 tx=0x7f96dc00cb40 comp rx=0 tx=0).stop 2024-03-06T07:20:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.964+0000 7f96eae81640 1 -- 172.21.15.1:0/3325263741 shutdown_connections 2024-03-06T07:20:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.964+0000 7f96eae81640 1 --2- 172.21.15.1:0/3325263741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f96d006b4d0 0x7f96d006d990 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.964+0000 7f96eae81640 1 --2- 172.21.15.1:0/3325263741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96e4152840 0x7f96e414c850 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.964+0000 7f96eae81640 1 --2- 172.21.15.1:0/3325263741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96e40c60b0 0x7f96e41546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.964+0000 7f96eae81640 1 --2- 172.21.15.1:0/3325263741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e40c3970 0x7f96e4153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.964+0000 7f96eae81640 1 -- 172.21.15.1:0/3325263741 >> 172.21.15.1:0/3325263741 conn(0x7f96e40c88f0 msgr2=0x7f96e4074450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.964+0000 7f96eae81640 1 -- 172.21.15.1:0/3325263741 shutdown_connections 2024-03-06T07:20:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:48.964+0000 7f96eae81640 1 -- 172.21.15.1:0/3325263741 wait complete. 2024-03-06T07:20:48.978 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:49.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.077+0000 7f3cf6083640 1 Processor -- start 2024-03-06T07:20:49.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.077+0000 7f3cf6083640 1 -- start start 2024-03-06T07:20:49.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.078+0000 7f3cf6083640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3cf0150520 0x7f3cf01543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.078+0000 7f3cf6083640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf0150900 con 0x7f3cf0070c80 2024-03-06T07:20:49.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.078+0000 7f3cf6083640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf0071790 con 0x7f3cf00c32f0 2024-03-06T07:20:49.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.078+0000 7f3cf6083640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cf0153ed0 con 0x7f3cf0150520 2024-03-06T07:20:49.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.079+0000 7f3cef7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3cf0150520 0x7f3cf01543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.079+0000 7f3ceffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3cf0070c80 0x7f3cf015d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46896/0 (socket says 172.21.15.1:46896) 2024-03-06T07:20:49.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.079+0000 7f3cef7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3cf0150520 0x7f3cf01543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52958/0 (socket says 172.21.15.1:52958) 2024-03-06T07:20:49.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.079+0000 7f3ceffff640 1 -- 172.21.15.1:0/1017284619 learned_addr learned my addr 172.21.15.1:0/1017284619 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:49.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.079+0000 7f3cef7fe640 1 -- 172.21.15.1:0/1017284619 >> v1:172.21.15.98:6789/0 conn(0x7f3cf00c32f0 legacy=0x7f3cf01537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.079+0000 7f3cef7fe640 1 -- 172.21.15.1:0/1017284619 >> v1:172.21.15.1:6789/0 conn(0x7f3cf0070c80 legacy=0x7f3cf015d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.079+0000 7f3ceeffd640 1 -- 172.21.15.1:0/1017284619 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709791212 0 0) 0x7f3cf0150900 con 0x7f3cf0070c80 2024-03-06T07:20:49.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.079+0000 7f3cef7fe640 1 -- 172.21.15.1:0/1017284619 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf0157050 con 0x7f3cf0150520 2024-03-06T07:20:49.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.080+0000 7f3cef7fe640 1 --2- 172.21.15.1:0/1017284619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cf0150520 0x7f3cf01543a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3ce00097a0 tx=0x7f3ce002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d96d22f3b27703c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:49.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.080+0000 7f3ceeffd640 1 -- 172.21.15.1:0/1017284619 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0034070 con 0x7f3cf0150520 2024-03-06T07:20:49.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.080+0000 7f3ceeffd640 1 -- 172.21.15.1:0/1017284619 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce0030610 con 0x7f3cf0150520 2024-03-06T07:20:49.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.080+0000 7f3ceeffd640 1 -- 172.21.15.1:0/1017284619 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce00309e0 con 0x7f3cf0150520 2024-03-06T07:20:49.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.081+0000 7f3cf6083640 1 -- 172.21.15.1:0/1017284619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cf0150520 msgr2=0x7f3cf01543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.081+0000 7f3cf6083640 1 --2- 172.21.15.1:0/1017284619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cf0150520 0x7f3cf01543a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3ce00097a0 tx=0x7f3ce002b960 comp rx=0 tx=0).stop 2024-03-06T07:20:49.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.081+0000 7f3cf6083640 1 -- 172.21.15.1:0/1017284619 shutdown_connections 2024-03-06T07:20:49.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.081+0000 7f3cf6083640 1 --2- 172.21.15.1:0/1017284619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cf0150520 0x7f3cf01543a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.081+0000 7f3cf6083640 1 -- 172.21.15.1:0/1017284619 >> 172.21.15.1:0/1017284619 conn(0x7f3cf00c88f0 msgr2=0x7f3cf00c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:49.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.081+0000 7f3cf6083640 1 -- 172.21.15.1:0/1017284619 shutdown_connections 2024-03-06T07:20:49.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.081+0000 7f3cf6083640 1 -- 172.21.15.1:0/1017284619 wait complete. 2024-03-06T07:20:49.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.082+0000 7f3cf6083640 1 Processor -- start 2024-03-06T07:20:49.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.083+0000 7f3cf6083640 1 -- start start 2024-03-06T07:20:49.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cf6083640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cf0070c80 0x7f3cf014f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cf6083640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cf00c32f0 0x7f3cf0143720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cf6083640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cf0150520 0x7f3cf0143c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cf6083640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf0157330 con 0x7f3cf0070c80 2024-03-06T07:20:49.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cf6083640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf01532a0 con 0x7f3cf0150520 2024-03-06T07:20:49.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cf6083640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf00c7ec0 con 0x7f3cf00c32f0 2024-03-06T07:20:49.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cef7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cf0070c80 0x7f3cf014f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cef7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cf0070c80 0x7f3cf014f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52968/0 (socket says 172.21.15.1:52968) 2024-03-06T07:20:49.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cef7fe640 1 -- 172.21.15.1:0/3849012204 learned_addr learned my addr 172.21.15.1:0/3849012204 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:49.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3ceffff640 1 --2- 172.21.15.1:0/3849012204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cf0150520 0x7f3cf0143c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3ce6ffd640 1 --2- 172.21.15.1:0/3849012204 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cf00c32f0 0x7f3cf0143720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cef7fe640 1 -- 172.21.15.1:0/3849012204 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cf00c32f0 msgr2=0x7f3cf0143720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cef7fe640 1 --2- 172.21.15.1:0/3849012204 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cf00c32f0 0x7f3cf0143720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cef7fe640 1 -- 172.21.15.1:0/3849012204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cf0150520 msgr2=0x7f3cf0143c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cef7fe640 1 --2- 172.21.15.1:0/3849012204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cf0150520 0x7f3cf0143c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cef7fe640 1 -- 172.21.15.1:0/3849012204 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf014d7d0 con 0x7f3cf0070c80 2024-03-06T07:20:49.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cef7fe640 1 --2- 172.21.15.1:0/3849012204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cf0070c80 0x7f3cf014f7c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3ce00098c0 tx=0x7f3ce00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:49.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cecff9640 1 -- 172.21.15.1:0/3849012204 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0034040 con 0x7f3cf0070c80 2024-03-06T07:20:49.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cecff9640 1 -- 172.21.15.1:0/3849012204 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce003acc0 con 0x7f3cf0070c80 2024-03-06T07:20:49.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cf6083640 1 -- 172.21.15.1:0/3849012204 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf01495a0 con 0x7f3cf0070c80 2024-03-06T07:20:49.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cecff9640 1 -- 172.21.15.1:0/3849012204 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0044630 con 0x7f3cf0070c80 2024-03-06T07:20:49.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.085+0000 7f3cf6083640 1 -- 172.21.15.1:0/3849012204 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf014dd50 con 0x7f3cf0070c80 2024-03-06T07:20:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.086+0000 7f3cf6083640 1 -- 172.21.15.1:0/3849012204 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf014df90 con 0x7f3cf0070c80 2024-03-06T07:20:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.086+0000 7f3cecff9640 1 -- 172.21.15.1:0/3849012204 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ce004b060 con 0x7f3cf0070c80 2024-03-06T07:20:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.086+0000 7f3cecff9640 1 --2- 172.21.15.1:0/3849012204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cd006b4d0 0x7f3cd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.086+0000 7f3cecff9640 1 -- 172.21.15.1:0/3849012204 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3ce00ac400 con 0x7f3cf0070c80 2024-03-06T07:20:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.087+0000 7f3ce6ffd640 1 --2- 172.21.15.1:0/3849012204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cd006b4d0 0x7f3cd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.087+0000 7f3ce6ffd640 1 --2- 172.21.15.1:0/3849012204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cd006b4d0 0x7f3cd006d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3cf01dffc0 tx=0x7f3cd8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:49.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.091+0000 7f3cecff9640 1 -- 172.21.15.1:0/3849012204 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ce003f070 con 0x7f3cf0070c80 2024-03-06T07:20:49.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.349+0000 7f3cf6083640 1 -- 172.21.15.1:0/3849012204 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cf00c8660 con 0x7f3cf0070c80 2024-03-06T07:20:49.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.350+0000 7f3cecff9640 1 -- 172.21.15.1:0/3849012204 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4964 (secure 0 0 0) 0x7f3ce0037e40 con 0x7f3cf0070c80 2024-03-06T07:20:49.352 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:49.352 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2822958969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2822958969},{"type":"v1","addr":"172.21.15.98:6836","nonce":2822958969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2442906233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2442906233},{"type":"v1","addr":"172.21.15.98:6837","nonce":2442906233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/448746024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":448746024},{"type":"v1","addr":"172.21.15.1:6834","nonce":448746024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:21.207834+0000","modified":"2024-03-06T07:20:37.259661+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1315115982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1315115982},{"type":"v1","addr":"172.21.15.1:6835","nonce":1315115982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-03-06T07:20:49.352 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-03-06T07:20:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.353+0000 7f3cf6083640 1 -- 172.21.15.1:0/3849012204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cd006b4d0 msgr2=0x7f3cd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.353+0000 7f3cf6083640 1 --2- 172.21.15.1:0/3849012204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cd006b4d0 0x7f3cd006d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3cf01dffc0 tx=0x7f3cd8009040 comp rx=0 tx=0).stop 2024-03-06T07:20:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.353+0000 7f3cf6083640 1 -- 172.21.15.1:0/3849012204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cf0070c80 msgr2=0x7f3cf014f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.353+0000 7f3cf6083640 1 --2- 172.21.15.1:0/3849012204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cf0070c80 0x7f3cf014f7c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3ce00098c0 tx=0x7f3ce00046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.353+0000 7f3cf6083640 1 -- 172.21.15.1:0/3849012204 shutdown_connections 2024-03-06T07:20:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.353+0000 7f3cf6083640 1 --2- 172.21.15.1:0/3849012204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cd006b4d0 0x7f3cd006d990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.353+0000 7f3cf6083640 1 --2- 172.21.15.1:0/3849012204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cf0150520 0x7f3cf0143c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.353+0000 7f3cf6083640 1 --2- 172.21.15.1:0/3849012204 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cf00c32f0 0x7f3cf0143720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.353+0000 7f3cf6083640 1 --2- 172.21.15.1:0/3849012204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cf0070c80 0x7f3cf014f7c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.353+0000 7f3cf6083640 1 -- 172.21.15.1:0/3849012204 >> 172.21.15.1:0/3849012204 conn(0x7f3cf00c88f0 msgr2=0x7f3cf0150ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:49.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.353+0000 7f3cf6083640 1 -- 172.21.15.1:0/3849012204 shutdown_connections 2024-03-06T07:20:49.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.353+0000 7f3cf6083640 1 -- 172.21.15.1:0/3849012204 wait complete. 2024-03-06T07:20:49.767 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:20:21.207834+0000', 'modified': '2024-03-06T07:20:37.259661+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4281}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4281': {'gid': 4281, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1315115982', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1315115982}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1315115982}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4281, 'qdb_cluster': [4281]} 2024-03-06T07:20:49.767 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:20:49.767 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:20:49.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.869+0000 7fe7ba9f6640 1 Processor -- start 2024-03-06T07:20:49.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.869+0000 7fe7ba9f6640 1 -- start start 2024-03-06T07:20:49.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.870+0000 7fe7ba9f6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe7b4152730 0x7fe7b40c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.870+0000 7fe7ba9f6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe7b40c1aa0 0x7fe7b41552e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.870+0000 7fe7ba9f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b41588e0 con 0x7fe7b40c6fd0 2024-03-06T07:20:49.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.870+0000 7fe7ba9f6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b4158bc0 con 0x7fe7b4152730 2024-03-06T07:20:49.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.870+0000 7fe7ba9f6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b4158ea0 con 0x7fe7b40c1aa0 2024-03-06T07:20:49.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.870+0000 7fe7b3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe7b4152730 0x7fe7b40c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.870+0000 7fe7b3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe7b4152730 0x7fe7b40c6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52982/0 (socket says 172.21.15.1:52982) 2024-03-06T07:20:49.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.870+0000 7fe7b3fff640 1 -- 172.21.15.1:0/584885284 learned_addr learned my addr 172.21.15.1:0/584885284 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:49.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.870+0000 7fe7b8f93640 1 --2- 172.21.15.1:0/584885284 >> v2:172.21.15.98:3300/0 conn(0x7fe7b40c1aa0 0x7fe7b41552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.870+0000 7fe7b3fff640 1 -- 172.21.15.1:0/584885284 >> v2:172.21.15.98:3300/0 conn(0x7fe7b40c1aa0 msgr2=0x7fe7b41552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:49.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.870+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/584885284 >> v2:172.21.15.98:3300/0 conn(0x7fe7b40c1aa0 0x7fe7b41552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.870+0000 7fe7b3fff640 1 -- 172.21.15.1:0/584885284 >> v1:172.21.15.98:6789/0 conn(0x7fe7b40c6fd0 legacy=0x7fe7b40c1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:49.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.870+0000 7fe7b3fff640 1 -- 172.21.15.1:0/584885284 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b4159180 con 0x7fe7b4152730 2024-03-06T07:20:49.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.870+0000 7fe7b8f93640 1 --2- 172.21.15.1:0/584885284 >> v2:172.21.15.98:3300/0 conn(0x7fe7b40c1aa0 0x7fe7b41552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:49.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.871+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/584885284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7b4152730 0x7fe7b40c6a90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe7b40b0440 tx=0x7fe7a402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5c1415e8e539c27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:49.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.871+0000 7fe7b2ffd640 1 -- 172.21.15.1:0/584885284 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4038070 con 0x7fe7b4152730 2024-03-06T07:20:49.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.871+0000 7fe7b2ffd640 1 -- 172.21.15.1:0/584885284 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a402f050 con 0x7fe7b4152730 2024-03-06T07:20:49.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.871+0000 7fe7b2ffd640 1 -- 172.21.15.1:0/584885284 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4031970 con 0x7fe7b4152730 2024-03-06T07:20:49.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.872+0000 7fe7ba9f6640 1 -- 172.21.15.1:0/584885284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7b4152730 msgr2=0x7fe7b40c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.872+0000 7fe7ba9f6640 1 --2- 172.21.15.1:0/584885284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7b4152730 0x7fe7b40c6a90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe7b40b0440 tx=0x7fe7a402b7f0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.872+0000 7fe7ba9f6640 1 -- 172.21.15.1:0/584885284 shutdown_connections 2024-03-06T07:20:49.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.872+0000 7fe7ba9f6640 1 --2- 172.21.15.1:0/584885284 >> v2:172.21.15.98:3300/0 conn(0x7fe7b40c1aa0 0x7fe7b41552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.872+0000 7fe7ba9f6640 1 --2- 172.21.15.1:0/584885284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7b4152730 0x7fe7b40c6a90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.872+0000 7fe7ba9f6640 1 -- 172.21.15.1:0/584885284 >> 172.21.15.1:0/584885284 conn(0x7fe7b4066d00 msgr2=0x7fe7b4067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:49.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.872+0000 7fe7ba9f6640 1 -- 172.21.15.1:0/584885284 shutdown_connections 2024-03-06T07:20:49.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.872+0000 7fe7ba9f6640 1 -- 172.21.15.1:0/584885284 wait complete. 2024-03-06T07:20:49.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.873+0000 7fe7ba9f6640 1 Processor -- start 2024-03-06T07:20:49.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.873+0000 7fe7ba9f6640 1 -- start start 2024-03-06T07:20:49.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.873+0000 7fe7ba9f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7b40c1aa0 0x7fe7b4157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.873+0000 7fe7ba9f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7b40c6fd0 0x7fe7b41df250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.873+0000 7fe7ba9f6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7b4152730 0x7fe7b41df790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7ba9f6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b4155820 con 0x7fe7b4152730 2024-03-06T07:20:49.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7ba9f6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b4152b10 con 0x7fe7b40c1aa0 2024-03-06T07:20:49.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7ba9f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b41559a0 con 0x7fe7b40c6fd0 2024-03-06T07:20:49.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7b8f93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7b4152730 0x7fe7b41df790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7b8f93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7b4152730 0x7fe7b41df790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52996/0 (socket says 172.21.15.1:52996) 2024-03-06T07:20:49.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7b8f93640 1 -- 172.21.15.1:0/4145334040 learned_addr learned my addr 172.21.15.1:0/4145334040 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:49.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/4145334040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7b40c1aa0 0x7fe7b4157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7b37fe640 1 --2- 172.21.15.1:0/4145334040 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7b40c6fd0 0x7fe7b41df250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7b8f93640 1 -- 172.21.15.1:0/4145334040 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7b40c6fd0 msgr2=0x7fe7b41df250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7b8f93640 1 --2- 172.21.15.1:0/4145334040 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7b40c6fd0 0x7fe7b41df250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7b8f93640 1 -- 172.21.15.1:0/4145334040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7b40c1aa0 msgr2=0x7fe7b4157d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7b8f93640 1 --2- 172.21.15.1:0/4145334040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7b40c1aa0 0x7fe7b4157d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7b8f93640 1 -- 172.21.15.1:0/4145334040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b40c9ab0 con 0x7fe7b4152730 2024-03-06T07:20:49.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/4145334040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7b40c1aa0 0x7fe7b4157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:49.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7b8f93640 1 --2- 172.21.15.1:0/4145334040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7b4152730 0x7fe7b41df790 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac00cdd0 tx=0x7fe7ac00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:49.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7b0ff9640 1 -- 172.21.15.1:0/4145334040 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac01a070 con 0x7fe7b4152730 2024-03-06T07:20:49.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7b0ff9640 1 -- 172.21.15.1:0/4145334040 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ac00d840 con 0x7fe7b4152730 2024-03-06T07:20:49.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7ba9f6640 1 -- 172.21.15.1:0/4145334040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b41eec20 con 0x7fe7b4152730 2024-03-06T07:20:49.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7b0ff9640 1 -- 172.21.15.1:0/4145334040 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac00dc70 con 0x7fe7b4152730 2024-03-06T07:20:49.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.874+0000 7fe7ba9f6640 1 -- 172.21.15.1:0/4145334040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b414ede0 con 0x7fe7b4152730 2024-03-06T07:20:49.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.876+0000 7fe7b0ff9640 1 -- 172.21.15.1:0/4145334040 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7ac01d070 con 0x7fe7b4152730 2024-03-06T07:20:49.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.876+0000 7fe7b0ff9640 1 --2- 172.21.15.1:0/4145334040 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe79406b500 0x7fe79406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.876+0000 7fe7b0ff9640 1 -- 172.21.15.1:0/4145334040 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe7ac08acf0 con 0x7fe7b4152730 2024-03-06T07:20:49.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.876+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/4145334040 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe79406b500 0x7fe79406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.877+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/4145334040 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe79406b500 0x7fe79406d9c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4035000 tx=0x7fe7a4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:49.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.878+0000 7fe7bd5d6740 1 Processor -- start 2024-03-06T07:20:49.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.878+0000 7fe7bd5d6740 1 -- start start 2024-03-06T07:20:49.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.879+0000 7fe7bd5d6740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bb0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.879+0000 7fe7bd5d6740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bb3330 0x5564e8bb0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.879+0000 7fe7bd5d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bb14b0 0x5564e8bad680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.879+0000 7fe7bd5d6740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e8b176b0 con 0x5564e8bb2f50 2024-03-06T07:20:49.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.879+0000 7fe7bd5d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e8ac0a60 con 0x5564e8bb14b0 2024-03-06T07:20:49.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.879+0000 7fe7bd5d6740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5564e8b13750 con 0x5564e8bb3330 2024-03-06T07:20:49.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.879+0000 7fe7b3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bb0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.879+0000 7fe7b3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bb0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53006/0 (socket says 172.21.15.1:53006) 2024-03-06T07:20:49.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.879+0000 7fe7b3fff640 1 -- 172.21.15.1:0/2682303350 learned_addr learned my addr 172.21.15.1:0/2682303350 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:49.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.880+0000 7fe7b3fff640 1 -- 172.21.15.1:0/2682303350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bb3330 msgr2=0x5564e8bb0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:49.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.880+0000 7fe7b37fe640 1 --2- 172.21.15.1:0/2682303350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bb3330 0x5564e8bb0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.880+0000 7fe7b8f93640 1 --2- 172.21.15.1:0/2682303350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bb14b0 0x5564e8bad680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.880+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/2682303350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bb3330 0x5564e8bb0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.880+0000 7fe7b3fff640 1 -- 172.21.15.1:0/2682303350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bb14b0 msgr2=0x5564e8bad680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:49.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.880+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/2682303350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bb14b0 0x5564e8bad680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.880+0000 7fe7b3fff640 1 -- 172.21.15.1:0/2682303350 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564e8adef30 con 0x5564e8bb2f50 2024-03-06T07:20:49.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.880+0000 7fe7b8f93640 1 --2- 172.21.15.1:0/2682303350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bb14b0 0x5564e8bad680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:49.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.880+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/2682303350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bb0a30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4036040 tx=0x7fe7a4031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e64bae144bf84c24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:49.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.880+0000 7fe7a9ffb640 1 -- 172.21.15.1:0/2682303350 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a404e070 con 0x5564e8bb2f50 2024-03-06T07:20:49.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.880+0000 7fe7a9ffb640 1 -- 172.21.15.1:0/2682303350 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a4031d70 con 0x5564e8bb2f50 2024-03-06T07:20:49.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.880+0000 7fe7a9ffb640 1 -- 172.21.15.1:0/2682303350 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4038040 con 0x5564e8bb2f50 2024-03-06T07:20:49.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.881+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/2682303350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 msgr2=0x5564e8bb0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.881+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/2682303350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bb0a30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4036040 tx=0x7fe7a4031000 comp rx=0 tx=0).stop 2024-03-06T07:20:49.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.881+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/2682303350 shutdown_connections 2024-03-06T07:20:49.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.881+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/2682303350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bb14b0 0x5564e8bad680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.881+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/2682303350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bb3330 0x5564e8bb0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.881+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/2682303350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bb0a30 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.881+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/2682303350 >> 172.21.15.1:0/2682303350 conn(0x5564e8baf7d0 msgr2=0x5564e8bd43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:49.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.881+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/2682303350 shutdown_connections 2024-03-06T07:20:49.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.881+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/2682303350 wait complete. 2024-03-06T07:20:49.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.881+0000 7fe7bd5d6740 1 -- start start 2024-03-06T07:20:49.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.881+0000 7fe7bd5d6740 1 Processor -- start 2024-03-06T07:20:49.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.882+0000 7fe7bd5d6740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bef710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.882+0000 7fe7bd5d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bb3330 0x5564e8bd5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.882+0000 7fe7bd5d6740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bdb290 0x5564e8bcffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.882+0000 7fe7bd5d6740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e8bd5230 con 0x5564e8bb2f50 2024-03-06T07:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.882+0000 7fe7bd5d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e8ad3ea0 con 0x5564e8bb3330 2024-03-06T07:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.882+0000 7fe7bd5d6740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5564e8bd4c70 con 0x5564e8bdb290 2024-03-06T07:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.882+0000 7fe7b3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bef710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.882+0000 7fe7bd5d6740 10 client.0 _ll_register_callbacks cb 0x5564e8ad3870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.882+0000 7fe7b3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bef710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53022/0 (socket says 172.21.15.1:53022) 2024-03-06T07:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.882+0000 7fe7b3fff640 1 -- 172.21.15.1:0/3725349441 learned_addr learned my addr 172.21.15.1:0/3725349441 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.882+0000 7fe7b8f93640 1 --2- 172.21.15.1:0/3725349441 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bdb290 0x5564e8bcffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.882+0000 7fe7b37fe640 1 --2- 172.21.15.1:0/3725349441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bb3330 0x5564e8bd5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe7b3fff640 1 -- 172.21.15.1:0/3725349441 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bdb290 msgr2=0x5564e8bcffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/3725349441 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bdb290 0x5564e8bcffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe7b3fff640 1 -- 172.21.15.1:0/3725349441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bb3330 msgr2=0x5564e8bd5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/3725349441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bb3330 0x5564e8bd5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe7b3fff640 1 -- 172.21.15.1:0/3725349441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5564e8ad4270 con 0x5564e8bb2f50 2024-03-06T07:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe7b8f93640 1 --2- 172.21.15.1:0/3725349441 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bdb290 0x5564e8bcffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe7b37fe640 1 --2- 172.21.15.1:0/3725349441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bb3330 0x5564e8bd5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/3725349441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bef710 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009960 tx=0x7fe7a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae679e296fe5b495 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe792ffd640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe792ffd640 1 -- 172.21.15.1:0/3725349441 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a404e070 con 0x5564e8bb2f50 2024-03-06T07:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe792ffd640 1 -- 172.21.15.1:0/3725349441 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe7a4031990 con 0x5564e8bb2f50 2024-03-06T07:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/3725349441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe7a4031d70 con 0x5564e8bb2f50 2024-03-06T07:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe792ffd640 1 -- 172.21.15.1:0/3725349441 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7a4038040 con 0x5564e8bb2f50 2024-03-06T07:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe7bd5d6740 10 client.4860 fetch_fsmap learned FSMap version 9 2024-03-06T07:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/3725349441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x5564e882c2d0 con 0x5564e8bb2f50 2024-03-06T07:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe792ffd640 1 -- 172.21.15.1:0/3725349441 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a4031d70 con 0x5564e8bb2f50 2024-03-06T07:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe792ffd640 1 -- 172.21.15.1:0/3725349441 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 9) v1 ==== 3322+0+0 (secure 0 0 0) 0x7fe7a4033440 con 0x5564e8bb2f50 2024-03-06T07:20:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe7bd5d6740 10 client.4860 fetch_fsmap finished waiting for FSMap version 9 2024-03-06T07:20:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe7bd5d6740 10 client.4860 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:20:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe7bd5d6740 20 client.4860 populate_metadata read hostname 'smithi001' 2024-03-06T07:20:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/3725349441 >> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] conn(0x5564e8c039e0 0x5564e8c0e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe7bd5d6740 4 client.4860 mds_command: new command op to 4281 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:20:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.883+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/3725349441 --> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5564e8b24bf0 con 0x5564e8c039e0 2024-03-06T07:20:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.884+0000 7fe7b37fe640 1 --2- 172.21.15.1:0/3725349441 >> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] conn(0x5564e8c039e0 0x5564e8c0e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.884+0000 7fe7b37fe640 1 --2- 172.21.15.1:0/3725349441 >> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] conn(0x5564e8c039e0 0x5564e8c0e870 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49aab7d9276e81e3 server_cookie=9f1cec59cb50cec5 in_seq=0 out_seq=0 2024-03-06T07:20:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.884+0000 7fe792ffd640 10 client.4860 ms_handle_connect on v2:172.21.15.1:6833/1315115982 2024-03-06T07:20:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.885+0000 7fe792ffd640 1 -- 172.21.15.1:0/3725349441 <== mds.0 v2:172.21.15.1:6833/1315115982 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5564e8b24bf0 con 0x5564e8c039e0 2024-03-06T07:20:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.885+0000 7fe792ffd640 10 client.4860 handle_command_reply: tid=0 2024-03-06T07:20:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.885+0000 7fe7bd5d6740 1 client.4860 shutdown 2024-03-06T07:20:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.885+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/3725349441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 msgr2=0x5564e8bef710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.885+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/3725349441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bef710 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009960 tx=0x7fe7a4004730 comp rx=0 tx=0).stop 2024-03-06T07:20:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.885+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/3725349441 shutdown_connections 2024-03-06T07:20:49.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.885+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/3725349441 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bdb290 0x5564e8bcffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.885+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/3725349441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bb3330 0x5564e8bd5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.885+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/3725349441 >> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] conn(0x5564e8c039e0 0x5564e8c0e870 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.885+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/3725349441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bef710 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.885+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/3725349441 >> 172.21.15.1:0/3725349441 conn(0x5564e8bb14b0 msgr2=0x5564e8bac510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:49.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.885+0000 7fe792ffd640 0 client.4860 ms_handle_reset on v2:172.21.15.1:6833/1315115982 2024-03-06T07:20:49.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.885+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/3725349441 shutdown_connections 2024-03-06T07:20:49.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.885+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/3725349441 wait complete. 2024-03-06T07:20:49.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.885+0000 7fe7bd5d6740 20 client.4860 trim_cache size 0 max 16384 2024-03-06T07:20:49.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.905+0000 7fe7bd5d6740 1 Processor -- start 2024-03-06T07:20:49.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.905+0000 7fe7bd5d6740 1 -- start start 2024-03-06T07:20:49.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.906+0000 7fe7bd5d6740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bffc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.906+0000 7fe7bd5d6740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bb3330 0x5564e8bf2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.906+0000 7fe7bd5d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bdb290 0x5564e8bf3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.906+0000 7fe7bd5d6740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b41f7be0 con 0x5564e8bb2f50 2024-03-06T07:20:49.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.906+0000 7fe7bd5d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b41f6fe0 con 0x5564e8bdb290 2024-03-06T07:20:49.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.906+0000 7fe7bd5d6740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b41f75e0 con 0x5564e8bb3330 2024-03-06T07:20:49.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.906+0000 7fe7b3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bffc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.906+0000 7fe7b3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bffc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53032/0 (socket says 172.21.15.1:53032) 2024-03-06T07:20:49.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.906+0000 7fe7b3fff640 1 -- 172.21.15.1:0/739235564 learned_addr learned my addr 172.21.15.1:0/739235564 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:49.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.906+0000 7fe7b3fff640 1 -- 172.21.15.1:0/739235564 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bb3330 msgr2=0x5564e8bf2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:49.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.906+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/739235564 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bb3330 0x5564e8bf2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.906+0000 7fe7b3fff640 1 -- 172.21.15.1:0/739235564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bdb290 msgr2=0x5564e8bf3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:49.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.906+0000 7fe7b8f93640 1 --2- 172.21.15.1:0/739235564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bdb290 0x5564e8bf3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.906+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/739235564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bdb290 0x5564e8bf3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.906+0000 7fe7b3fff640 1 -- 172.21.15.1:0/739235564 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564e88f66b0 con 0x5564e8bb2f50 2024-03-06T07:20:49.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.906+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/739235564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bffc00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe7a402be00 tx=0x7fe7a404b070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e75415d6e7f297 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:49.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.906+0000 7fe7a97fa640 1 -- 172.21.15.1:0/739235564 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4063070 con 0x5564e8bb2f50 2024-03-06T07:20:49.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.907+0000 7fe7a97fa640 1 -- 172.21.15.1:0/739235564 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a4031e30 con 0x5564e8bb2f50 2024-03-06T07:20:49.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.907+0000 7fe7a97fa640 1 -- 172.21.15.1:0/739235564 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a404e040 con 0x5564e8bb2f50 2024-03-06T07:20:49.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.907+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/739235564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 msgr2=0x5564e8bffc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.907+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/739235564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bffc00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe7a402be00 tx=0x7fe7a404b070 comp rx=0 tx=0).stop 2024-03-06T07:20:49.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.907+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/739235564 shutdown_connections 2024-03-06T07:20:49.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.907+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/739235564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bdb290 0x5564e8bf3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.907+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/739235564 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bb3330 0x5564e8bf2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.907+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/739235564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bb2f50 0x5564e8bffc00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.907+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/739235564 >> 172.21.15.1:0/739235564 conn(0x5564e8bb14b0 msgr2=0x5564e8bf0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:49.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.907+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/739235564 shutdown_connections 2024-03-06T07:20:49.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.907+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/739235564 wait complete. 2024-03-06T07:20:49.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.908+0000 7fe7bd5d6740 1 -- start start 2024-03-06T07:20:49.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.908+0000 7fe7bd5d6740 1 Processor -- start 2024-03-06T07:20:49.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7bd5d6740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bb2f50 0x5564e8bd68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7bd5d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bb3330 0x5564e8bcd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7bd5d6740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bdb290 0x5564e8bee1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7bd5d6740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e8bd7430 con 0x5564e8bdb290 2024-03-06T07:20:49.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7bd5d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e8bd7950 con 0x5564e8bb3330 2024-03-06T07:20:49.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7bd5d6740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5564e8bd0740 con 0x5564e8bb2f50 2024-03-06T07:20:49.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7bd5d6740 10 client.0 _ll_register_callbacks cb 0x7fe7a404c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:20:49.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7b8f93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bdb290 0x5564e8bee1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7b8f93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bdb290 0x5564e8bee1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53038/0 (socket says 172.21.15.1:53038) 2024-03-06T07:20:49.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7b8f93640 1 -- 172.21.15.1:0/2217284147 learned_addr learned my addr 172.21.15.1:0/2217284147 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:49.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/2217284147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bb2f50 0x5564e8bd68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7b37fe640 1 --2- 172.21.15.1:0/2217284147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bb3330 0x5564e8bcd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7b8f93640 1 -- 172.21.15.1:0/2217284147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bb2f50 msgr2=0x5564e8bd68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:49.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7b8f93640 1 --2- 172.21.15.1:0/2217284147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bb2f50 0x5564e8bd68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7b8f93640 1 -- 172.21.15.1:0/2217284147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bb3330 msgr2=0x5564e8bcd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:49.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7b8f93640 1 --2- 172.21.15.1:0/2217284147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bb3330 0x5564e8bcd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7b8f93640 1 -- 172.21.15.1:0/2217284147 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5564e8bcd770 con 0x5564e8bdb290 2024-03-06T07:20:49.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7b8f93640 1 --2- 172.21.15.1:0/2217284147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bdb290 0x5564e8bee1b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac088e70 tx=0x7fe7ac0591e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269887c2b590709c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:49.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe792ffd640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:20:49.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe792ffd640 1 -- 172.21.15.1:0/2217284147 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac00de10 con 0x5564e8bdb290 2024-03-06T07:20:49.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe792ffd640 1 -- 172.21.15.1:0/2217284147 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe7ac0613a0 con 0x5564e8bdb290 2024-03-06T07:20:49.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/2217284147 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe7a4031e30 con 0x5564e8bdb290 2024-03-06T07:20:49.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe792ffd640 1 -- 172.21.15.1:0/2217284147 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7ac0615a0 con 0x5564e8bdb290 2024-03-06T07:20:49.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7bd5d6740 10 client.4866 fetch_fsmap learned FSMap version 9 2024-03-06T07:20:49.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.909+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/2217284147 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x5564e8bae110 con 0x5564e8bdb290 2024-03-06T07:20:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.910+0000 7fe792ffd640 1 -- 172.21.15.1:0/2217284147 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a4031e30 con 0x5564e8bdb290 2024-03-06T07:20:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.910+0000 7fe792ffd640 1 -- 172.21.15.1:0/2217284147 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 9) v1 ==== 3322+0+0 (secure 0 0 0) 0x7fe7ac058570 con 0x5564e8bdb290 2024-03-06T07:20:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.910+0000 7fe7bd5d6740 10 client.4866 fetch_fsmap finished waiting for FSMap version 9 2024-03-06T07:20:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.910+0000 7fe7bd5d6740 10 client.4866 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:20:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.910+0000 7fe7bd5d6740 20 client.4866 populate_metadata read hostname 'smithi001' 2024-03-06T07:20:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.910+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/2217284147 >> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] conn(0x5564e8bacec0 0x5564e8bad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.910+0000 7fe7bd5d6740 4 client.4866 mds_command: new command op to 4281 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:20:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.910+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/2217284147 --> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5564e8b1ef40 con 0x5564e8bacec0 2024-03-06T07:20:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.910+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/2217284147 >> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] conn(0x5564e8bacec0 0x5564e8bad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.910+0000 7fe792ffd640 10 client.4866 ms_handle_connect on v2:172.21.15.1:6833/1315115982 2024-03-06T07:20:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.910+0000 7fe7b3fff640 1 --2- 172.21.15.1:0/2217284147 >> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] conn(0x5564e8bacec0 0x5564e8bad2a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfd237c52551539b server_cookie=8d63b0ba6242e1fc in_seq=0 out_seq=0 2024-03-06T07:20:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.910+0000 7fe792ffd640 1 -- 172.21.15.1:0/2217284147 <== mds.0 v2:172.21.15.1:6833/1315115982 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x5564e8b1ef40 con 0x5564e8bacec0 2024-03-06T07:20:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.910+0000 7fe792ffd640 10 client.4866 handle_command_reply: tid=0 2024-03-06T07:20:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.910+0000 7fe7bd5d6740 1 client.4866 shutdown 2024-03-06T07:20:49.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/2217284147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bdb290 msgr2=0x5564e8bee1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/2217284147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bdb290 0x5564e8bee1b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac088e70 tx=0x7fe7ac0591e0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/2217284147 shutdown_connections 2024-03-06T07:20:49.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/2217284147 >> [v2:172.21.15.1:6833/1315115982,v1:172.21.15.1:6835/1315115982] conn(0x5564e8bacec0 0x5564e8bad2a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/2217284147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5564e8bdb290 0x5564e8bee1b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/2217284147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564e8bb3330 0x5564e8bcd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe792ffd640 0 client.4866 ms_handle_reset on v2:172.21.15.1:6833/1315115982 2024-03-06T07:20:49.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7bd5d6740 1 --2- 172.21.15.1:0/2217284147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5564e8bb2f50 0x5564e8bd68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/2217284147 >> 172.21.15.1:0/2217284147 conn(0x5564e8bb14b0 msgr2=0x5564e8bffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:49.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/2217284147 shutdown_connections 2024-03-06T07:20:49.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7bd5d6740 1 -- 172.21.15.1:0/2217284147 wait complete. 2024-03-06T07:20:49.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7bd5d6740 20 client.4866 trim_cache size 0 max 16384 2024-03-06T07:20:49.918 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:20:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:20:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:20:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4281, 2024-03-06T07:20:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:20:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:20:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:20:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 28.074264426999999, 2024-03-06T07:20:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 9, 2024-03-06T07:20:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 25, 2024-03-06T07:20:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 25, 2024-03-06T07:20:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 34.721142065999999 2024-03-06T07:20:49.918 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:20:49.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7ba9f6640 1 -- 172.21.15.1:0/4145334040 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe79406b500 msgr2=0x7fe79406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7ba9f6640 1 --2- 172.21.15.1:0/4145334040 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe79406b500 0x7fe79406d9c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4035000 tx=0x7fe7a4008030 comp rx=0 tx=0).stop 2024-03-06T07:20:49.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7ba9f6640 1 -- 172.21.15.1:0/4145334040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7b4152730 msgr2=0x7fe7b41df790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7ba9f6640 1 --2- 172.21.15.1:0/4145334040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7b4152730 0x7fe7b41df790 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac00cdd0 tx=0x7fe7ac00a710 comp rx=0 tx=0).stop 2024-03-06T07:20:49.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7ba9f6640 1 -- 172.21.15.1:0/4145334040 shutdown_connections 2024-03-06T07:20:49.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7ba9f6640 1 --2- 172.21.15.1:0/4145334040 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe79406b500 0x7fe79406d9c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7ba9f6640 1 --2- 172.21.15.1:0/4145334040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7b4152730 0x7fe7b41df790 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7ba9f6640 1 --2- 172.21.15.1:0/4145334040 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7b40c6fd0 0x7fe7b41df250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7ba9f6640 1 --2- 172.21.15.1:0/4145334040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7b40c1aa0 0x7fe7b4157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.911+0000 7fe7ba9f6640 1 -- 172.21.15.1:0/4145334040 >> 172.21.15.1:0/4145334040 conn(0x7fe7b4066d00 msgr2=0x7fe7b40c3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.912+0000 7fe7ba9f6640 1 -- 172.21.15.1:0/4145334040 shutdown_connections 2024-03-06T07:20:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:49.912+0000 7fe7ba9f6640 1 -- 172.21.15.1:0/4145334040 wait complete. 2024-03-06T07:20:49.924 INFO:teuthology.run_tasks:Running task kclient... 2024-03-06T07:20:49.941 INFO:tasks.kclient:Mounting kernel clients... 2024-03-06T07:20:49.941 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-03-06T07:20:49.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:49.941 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-06T07:20:49.956 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-06T07:20:49.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:49.957 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-06T07:20:50.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:50.023 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link delete ceph-brx 2024-03-06T07:20:50.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:50.122 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-03-06T07:20:50.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:50.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-03-06T07:20:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:Cannot find device "ceph-brx" 2024-03-06T07:20:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:20:50.216 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-03-06T07:20:50.216 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:20:50.217 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:20:50.217 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:20:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.319+0000 7f1c2f3ac640 1 Processor -- start 2024-03-06T07:20:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.320+0000 7f1c2f3ac640 1 -- start start 2024-03-06T07:20:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.320+0000 7f1c2f3ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c280c7cc0 0x7f1c28156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.320+0000 7f1c2f3ac640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1c28070be0 0x7f1c28071090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.320+0000 7f1c2f3ac640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c28154a70 con 0x7f1c2814cf90 2024-03-06T07:20:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.320+0000 7f1c2f3ac640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c2814d370 con 0x7f1c28070be0 2024-03-06T07:20:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.320+0000 7f1c2f3ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280c6190 con 0x7f1c280c7cc0 2024-03-06T07:20:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.321+0000 7f1c2c947640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c280c7cc0 0x7f1c28156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.321+0000 7f1c2d949640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1c28070be0 0x7f1c28071090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.321+0000 7f1c2c947640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c280c7cc0 0x7f1c28156fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:41842/0 (socket says 172.21.15.1:41842) 2024-03-06T07:20:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.321+0000 7f1c2c947640 1 -- 172.21.15.1:0/1511221629 learned_addr learned my addr 172.21.15.1:0/1511221629 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.321+0000 7f1c1ffff640 1 -- 172.21.15.1:0/1511221629 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598929036 0 0) 0x7f1c28154a70 con 0x7f1c2814cf90 2024-03-06T07:20:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.321+0000 7f1c1ffff640 1 -- 172.21.15.1:0/1511221629 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c14003520 con 0x7f1c2814cf90 2024-03-06T07:20:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.321+0000 7f1c2c947640 1 -- 172.21.15.1:0/1511221629 >> v2:172.21.15.98:3300/0 conn(0x7f1c28070be0 msgr2=0x7f1c28071090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.321+0000 7f1c2c947640 1 --2- 172.21.15.1:0/1511221629 >> v2:172.21.15.98:3300/0 conn(0x7f1c28070be0 0x7f1c28071090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.321+0000 7f1c2c947640 1 -- 172.21.15.1:0/1511221629 >> v1:172.21.15.98:6789/0 conn(0x7f1c2814cf90 legacy=0x7f1c28154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.321+0000 7f1c2c947640 1 -- 172.21.15.1:0/1511221629 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c28153a90 con 0x7f1c280c7cc0 2024-03-06T07:20:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.321+0000 7f1c2d949640 1 --2- 172.21.15.1:0/1511221629 >> v2:172.21.15.98:3300/0 conn(0x7f1c28070be0 0x7f1c28071090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.322+0000 7f1c2c947640 1 --2- 172.21.15.1:0/1511221629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c280c7cc0 0x7f1c28156fe0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1c180098a0 tx=0x7f1c1802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=589948ab0c7fe3e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:50.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.323+0000 7f1c1ffff640 1 -- 172.21.15.1:0/1511221629 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c180317c0 con 0x7f1c280c7cc0 2024-03-06T07:20:50.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.323+0000 7f1c1ffff640 1 -- 172.21.15.1:0/1511221629 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c18037070 con 0x7f1c280c7cc0 2024-03-06T07:20:50.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.323+0000 7f1c1ffff640 1 -- 172.21.15.1:0/1511221629 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c18031b60 con 0x7f1c280c7cc0 2024-03-06T07:20:50.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.323+0000 7f1c2f3ac640 1 -- 172.21.15.1:0/1511221629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c280c7cc0 msgr2=0x7f1c28156fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:50.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.323+0000 7f1c2f3ac640 1 --2- 172.21.15.1:0/1511221629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c280c7cc0 0x7f1c28156fe0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1c180098a0 tx=0x7f1c1802b860 comp rx=0 tx=0).stop 2024-03-06T07:20:50.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.323+0000 7f1c2f3ac640 1 -- 172.21.15.1:0/1511221629 shutdown_connections 2024-03-06T07:20:50.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.323+0000 7f1c2f3ac640 1 --2- 172.21.15.1:0/1511221629 >> v2:172.21.15.98:3300/0 conn(0x7f1c28070be0 0x7f1c28071090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:50.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.323+0000 7f1c2f3ac640 1 --2- 172.21.15.1:0/1511221629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c280c7cc0 0x7f1c28156fe0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:50.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.323+0000 7f1c2f3ac640 1 -- 172.21.15.1:0/1511221629 >> 172.21.15.1:0/1511221629 conn(0x7f1c280c88f0 msgr2=0x7f1c280c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:50.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.323+0000 7f1c2f3ac640 1 -- 172.21.15.1:0/1511221629 shutdown_connections 2024-03-06T07:20:50.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.323+0000 7f1c2f3ac640 1 -- 172.21.15.1:0/1511221629 wait complete. 2024-03-06T07:20:50.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.324+0000 7f1c2f3ac640 1 Processor -- start 2024-03-06T07:20:50.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.324+0000 7f1c2f3ac640 1 -- start start 2024-03-06T07:20:50.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.325+0000 7f1c2f3ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c28070be0 0x7f1c2814d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:50.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.325+0000 7f1c2f3ac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c280c7cc0 0x7f1c280ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:50.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.325+0000 7f1c2f3ac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2814cf90 0x7f1c28155650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:50.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.325+0000 7f1c2f3ac640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c281567e0 con 0x7f1c2814cf90 2024-03-06T07:20:50.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.325+0000 7f1c2f3ac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c28156500 con 0x7f1c280c7cc0 2024-03-06T07:20:50.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.325+0000 7f1c2f3ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c28156ac0 con 0x7f1c28070be0 2024-03-06T07:20:50.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.325+0000 7f1c2d949640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2814cf90 0x7f1c28155650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:50.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.325+0000 7f1c2d949640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2814cf90 0x7f1c28155650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53054/0 (socket says 172.21.15.1:53054) 2024-03-06T07:20:50.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.325+0000 7f1c2d949640 1 -- 172.21.15.1:0/2861725443 learned_addr learned my addr 172.21.15.1:0/2861725443 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:50.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.325+0000 7f1c2d148640 1 --2- 172.21.15.1:0/2861725443 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c28070be0 0x7f1c2814d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:50.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.325+0000 7f1c2c947640 1 --2- 172.21.15.1:0/2861725443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c280c7cc0 0x7f1c280ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:50.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.326+0000 7f1c2d949640 1 -- 172.21.15.1:0/2861725443 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c28070be0 msgr2=0x7f1c2814d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:50.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.326+0000 7f1c2d949640 1 --2- 172.21.15.1:0/2861725443 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c28070be0 0x7f1c2814d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:50.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.326+0000 7f1c2d949640 1 -- 172.21.15.1:0/2861725443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c280c7cc0 msgr2=0x7f1c280ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:50.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.326+0000 7f1c2d949640 1 --2- 172.21.15.1:0/2861725443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c280c7cc0 0x7f1c280ac7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:50.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.326+0000 7f1c2d949640 1 -- 172.21.15.1:0/2861725443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c280a89c0 con 0x7f1c2814cf90 2024-03-06T07:20:50.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.326+0000 7f1c2c947640 1 --2- 172.21.15.1:0/2861725443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c280c7cc0 0x7f1c280ac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:50.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.326+0000 7f1c2d949640 1 --2- 172.21.15.1:0/2861725443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2814cf90 0x7f1c28155650 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000efc0 tx=0x7f1c20009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:50.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.326+0000 7f1c1dffb640 1 -- 172.21.15.1:0/2861725443 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c20018030 con 0x7f1c2814cf90 2024-03-06T07:20:50.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.326+0000 7f1c1dffb640 1 -- 172.21.15.1:0/2861725443 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c2000cde0 con 0x7f1c2814cf90 2024-03-06T07:20:50.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.326+0000 7f1c2f3ac640 1 -- 172.21.15.1:0/2861725443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c280a8c60 con 0x7f1c2814cf90 2024-03-06T07:20:50.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.326+0000 7f1c1dffb640 1 -- 172.21.15.1:0/2861725443 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c20003bb0 con 0x7f1c2814cf90 2024-03-06T07:20:50.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.326+0000 7f1c2f3ac640 1 -- 172.21.15.1:0/2861725443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c28155b90 con 0x7f1c2814cf90 2024-03-06T07:20:50.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.327+0000 7f1c2f3ac640 1 -- 172.21.15.1:0/2861725443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c280686e0 con 0x7f1c2814cf90 2024-03-06T07:20:50.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.328+0000 7f1c1dffb640 1 -- 172.21.15.1:0/2861725443 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c2001b070 con 0x7f1c2814cf90 2024-03-06T07:20:50.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.328+0000 7f1c1dffb640 1 --2- 172.21.15.1:0/2861725443 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c0c06b4d0 0x7f1c0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:50.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.328+0000 7f1c1dffb640 1 -- 172.21.15.1:0/2861725443 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1c2008ae60 con 0x7f1c2814cf90 2024-03-06T07:20:50.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.328+0000 7f1c2d148640 1 --2- 172.21.15.1:0/2861725443 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c0c06b4d0 0x7f1c0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:50.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.329+0000 7f1c2d148640 1 --2- 172.21.15.1:0/2861725443 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c0c06b4d0 0x7f1c0c06d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1c24007fd0 tx=0x7f1c24009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:50.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.332+0000 7f1c1dffb640 1 -- 172.21.15.1:0/2861725443 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c20010040 con 0x7f1c2814cf90 2024-03-06T07:20:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.554+0000 7f1c2f3ac640 1 -- 172.21.15.1:0/2861725443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1c2814edb0 con 0x7f1c2814cf90 2024-03-06T07:20:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.554+0000 7f1c1dffb640 1 -- 172.21.15.1:0/2861725443 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f1c20059a30 con 0x7f1c2814cf90 2024-03-06T07:20:50.556 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-06T07:20:50.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.556+0000 7f1c2f3ac640 1 -- 172.21.15.1:0/2861725443 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c0c06b4d0 msgr2=0x7f1c0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:50.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.556+0000 7f1c2f3ac640 1 --2- 172.21.15.1:0/2861725443 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c0c06b4d0 0x7f1c0c06d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1c24007fd0 tx=0x7f1c24009040 comp rx=0 tx=0).stop 2024-03-06T07:20:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.556+0000 7f1c2f3ac640 1 -- 172.21.15.1:0/2861725443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2814cf90 msgr2=0x7f1c28155650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.556+0000 7f1c2f3ac640 1 --2- 172.21.15.1:0/2861725443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2814cf90 0x7f1c28155650 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000efc0 tx=0x7f1c20009f90 comp rx=0 tx=0).stop 2024-03-06T07:20:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.556+0000 7f1c2f3ac640 1 -- 172.21.15.1:0/2861725443 shutdown_connections 2024-03-06T07:20:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.556+0000 7f1c2f3ac640 1 --2- 172.21.15.1:0/2861725443 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c0c06b4d0 0x7f1c0c06d990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.556+0000 7f1c2f3ac640 1 --2- 172.21.15.1:0/2861725443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2814cf90 0x7f1c28155650 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.556+0000 7f1c2f3ac640 1 --2- 172.21.15.1:0/2861725443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c280c7cc0 0x7f1c280ac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.556+0000 7f1c2f3ac640 1 --2- 172.21.15.1:0/2861725443 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c28070be0 0x7f1c2814d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.556+0000 7f1c2f3ac640 1 -- 172.21.15.1:0/2861725443 >> 172.21.15.1:0/2861725443 conn(0x7f1c280c88f0 msgr2=0x7f1c281526f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.556+0000 7f1c2f3ac640 1 -- 172.21.15.1:0/2861725443 shutdown_connections 2024-03-06T07:20:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:50.556+0000 7f1c2f3ac640 1 -- 172.21.15.1:0/2861725443 wait complete. 2024-03-06T07:20:50.569 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:20:50.569 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:20:50.569 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-06T07:20:50.569 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:20:50.569 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:20:50.569 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:20:50.569 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:20:50.570 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:20:50.570 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:20:50.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:50.570 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-06T07:20:50.585 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:20:50.586 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:20:50.586 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:20:50.586 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:20:50.586 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-06T07:20:50.586 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:20:50.586 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:20:50.586 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:20:50.586 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-06T07:20:50.586 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:20:50.586 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:20:50.586 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-06T07:20:50.586 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:20:50.586 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:20:50.586 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-06T07:20:50.587 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:20:50.587 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 42590sec preferred_lft 42590sec 2024-03-06T07:20:50.587 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-03-06T07:20:50.587 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:20:50.587 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:20:50.587 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:20:50.587 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-06T07:20:50.587 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:20:50.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:50.587 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-06T07:20:50.588 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-06T07:20:50.588 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:20:50.588 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr flush dev ceph-brx 2024-03-06T07:20:50.588 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set ceph-brx up 2024-03-06T07:20:50.588 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:20:50.588 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-06T07:20:50.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:50.694 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:20:50.720 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-06T07:20:50.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:50.722 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-06T07:20:50.777 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-06T07:20:50.778 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:20:50.778 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:20:50.778 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:20:50.778 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:20:50.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:50.778 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-06T07:20:50.778 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-06T07:20:50.778 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:20:50.778 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:20:50.779 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:20:50.779 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-06T07:20:50.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:50.873 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-06T07:20:50.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:50.887 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list-id 2024-03-06T07:20:50.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:50.941 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-06T07:20:50.941 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-06T07:20:50.941 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:20:50.941 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-06T07:20:50.941 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-06T07:20:51.023 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-06T07:20:51.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:51.024 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-06T07:20:51.024 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-06T07:20:51.024 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-06T07:20:51.025 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:20:51.025 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-06T07:20:51.025 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-06T07:20:51.025 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-06T07:20:51.025 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-06T07:20:51.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:51.150 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-06T07:20:51.150 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-06T07:20:51.150 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-06T07:20:51.150 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:20:51.150 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-06T07:20:51.191 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:20:51.191 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:20:51.246 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:20:51.247 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:20:51.247 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:20:51.300 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:20:51.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:51.301 DEBUG:teuthology.orchestra.run.smithi001:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:20:51.386 INFO:teuthology.orchestra.run.smithi001.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:20:51.386 INFO:teuthology.orchestra.run.smithi001.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:20:51.498 INFO:teuthology.orchestra.run.smithi001.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:20:51.499 INFO:teuthology.orchestra.run.smithi001.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:20:51.499 INFO:teuthology.orchestra.run.smithi001.stdout:mount.ceph: resolved to: "172.21.15.1:3300,172.21.15.98:3300,172.21.15.98:3301" 2024-03-06T07:20:51.499 INFO:teuthology.orchestra.run.smithi001.stdout:mount.ceph: trying mount with new device syntax: 0@8f3c19ba-4ac6-4a54-88df-9262183d23ef.cephfs=/ 2024-03-06T07:20:51.499 INFO:teuthology.orchestra.run.smithi001.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.1:3300/172.21.15.98:3300/172.21.15.98:3301" will pass to kernel 2024-03-06T07:20:51.500 INFO:teuthology.orchestra.run.smithi001.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:20:51.500 INFO:teuthology.orchestra.run.smithi001.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:20:51.500 INFO:teuthology.orchestra.run.smithi001.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:20:51.500 INFO:teuthology.orchestra.run.smithi001.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:20:51.500 INFO:teuthology.orchestra.run.smithi001.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:20:51.502 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:20:51.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:51.502 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:20:51.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:51.529 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:20:51.585 INFO:teuthology.orchestra.run.smithi001.stdout:client4589 2024-03-06T07:20:51.585 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:20:51.585 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:20:51.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:51.641 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:20:51.696 INFO:teuthology.orchestra.run.smithi001.stdout:client4589 2024-03-06T07:20:51.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:51.697 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:20:51.752 INFO:teuthology.orchestra.run.smithi001.stdout:client4589 2024-03-06T07:20:51.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:51.753 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/8f3c19ba-4ac6-4a54-88df-9262183d23ef.client4589/status') 2024-03-06T07:20:51.818 INFO:teuthology.orchestra.run.smithi001.stdout:instance: client.4589 (3)172.21.15.1:0/615794759 2024-03-06T07:20:51.818 INFO:teuthology.orchestra.run.smithi001.stdout:blocklisted: false 2024-03-06T07:20:51.818 INFO:teuthology.orchestra.run.smithi001.stderr:0+1 records in 2024-03-06T07:20:51.819 INFO:teuthology.orchestra.run.smithi001.stderr:0+1 records out 2024-03-06T07:20:51.819 INFO:teuthology.orchestra.run.smithi001.stderr:68 bytes copied, 1.9968e-05 s, 3.4 MB/s 2024-03-06T07:20:51.820 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-03-06T07:20:51.820 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:20:51.820 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:20:51.820 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:20:51.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.958+0000 7f9ca173e640 1 Processor -- start 2024-03-06T07:20:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.958+0000 7f9ca173e640 1 -- start start 2024-03-06T07:20:51.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.959+0000 7f9ca173e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9c9c072bd0 0x7f9c9c0c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:51.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.959+0000 7f9ca173e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9c9c152b70 0x7f9c9c0c3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:51.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.959+0000 7f9ca173e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9c151dd0 con 0x7f9c9c0c2b30 2024-03-06T07:20:51.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.959+0000 7f9ca173e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c9c1526e0 con 0x7f9c9c072bd0 2024-03-06T07:20:51.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.959+0000 7f9ca173e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c9c156190 con 0x7f9c9c152b70 2024-03-06T07:20:51.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.959+0000 7f9c9a7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9c9c072bd0 0x7f9c9c0c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.959+0000 7f9c9a7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9c9c072bd0 0x7f9c9c0c32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53066/0 (socket says 172.21.15.1:53066) 2024-03-06T07:20:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.959+0000 7f9c9a7fc640 1 -- 172.21.15.1:0/4085220210 learned_addr learned my addr 172.21.15.1:0/4085220210 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.960+0000 7f9c9b7fe640 1 --2- 172.21.15.1:0/4085220210 >> v2:172.21.15.98:3300/0 conn(0x7f9c9c152b70 0x7f9c9c0c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.960+0000 7f9c9a7fc640 1 -- 172.21.15.1:0/4085220210 >> v2:172.21.15.98:3300/0 conn(0x7f9c9c152b70 msgr2=0x7f9c9c0c3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.960+0000 7f9c9a7fc640 1 --2- 172.21.15.1:0/4085220210 >> v2:172.21.15.98:3300/0 conn(0x7f9c9c152b70 0x7f9c9c0c3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.960+0000 7f9c9a7fc640 1 -- 172.21.15.1:0/4085220210 >> v1:172.21.15.98:6789/0 conn(0x7f9c9c0c2b30 legacy=0x7f9c9c0724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.960+0000 7f9c9a7fc640 1 -- 172.21.15.1:0/4085220210 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c152090 con 0x7f9c9c072bd0 2024-03-06T07:20:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.960+0000 7f9c9a7fc640 1 --2- 172.21.15.1:0/4085220210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c9c072bd0 0x7f9c9c0c32f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c06f710 tx=0x7f9c9002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec9e6641f42119ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.961+0000 7f9c99ffb640 1 -- 172.21.15.1:0/4085220210 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c90034070 con 0x7f9c9c072bd0 2024-03-06T07:20:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.961+0000 7f9c99ffb640 1 -- 172.21.15.1:0/4085220210 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c90030860 con 0x7f9c9c072bd0 2024-03-06T07:20:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.961+0000 7f9c99ffb640 1 -- 172.21.15.1:0/4085220210 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c90030c30 con 0x7f9c9c072bd0 2024-03-06T07:20:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.961+0000 7f9ca173e640 1 -- 172.21.15.1:0/4085220210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c9c072bd0 msgr2=0x7f9c9c0c32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.961+0000 7f9ca173e640 1 --2- 172.21.15.1:0/4085220210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c9c072bd0 0x7f9c9c0c32f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c06f710 tx=0x7f9c9002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:20:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.961+0000 7f9ca173e640 1 -- 172.21.15.1:0/4085220210 shutdown_connections 2024-03-06T07:20:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.961+0000 7f9ca173e640 1 --2- 172.21.15.1:0/4085220210 >> v2:172.21.15.98:3300/0 conn(0x7f9c9c152b70 0x7f9c9c0c3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.961+0000 7f9ca173e640 1 --2- 172.21.15.1:0/4085220210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c9c072bd0 0x7f9c9c0c32f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.961+0000 7f9ca173e640 1 -- 172.21.15.1:0/4085220210 >> 172.21.15.1:0/4085220210 conn(0x7f9c9c0c6860 msgr2=0x7f9c9c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.961+0000 7f9ca173e640 1 -- 172.21.15.1:0/4085220210 shutdown_connections 2024-03-06T07:20:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.961+0000 7f9ca173e640 1 -- 172.21.15.1:0/4085220210 wait complete. 2024-03-06T07:20:51.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.963+0000 7f9ca173e640 1 Processor -- start 2024-03-06T07:20:51.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.963+0000 7f9ca173e640 1 -- start start 2024-03-06T07:20:51.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.963+0000 7f9ca173e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c9c072bd0 0x7f9c9c1eb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:51.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.963+0000 7f9ca173e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c9c0c2b30 0x7f9c9c0c3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.963+0000 7f9ca173e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c9c152b70 0x7f9c9c149820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.963+0000 7f9ca173e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c0c3ef0 con 0x7f9c9c0c2b30 2024-03-06T07:20:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.963+0000 7f9ca173e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c0c3d70 con 0x7f9c9c152b70 2024-03-06T07:20:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.963+0000 7f9ca173e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c155d80 con 0x7f9c9c072bd0 2024-03-06T07:20:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.963+0000 7f9c9a7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c9c0c2b30 0x7f9c9c0c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.963+0000 7f9c9a7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c9c0c2b30 0x7f9c9c0c3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53068/0 (socket says 172.21.15.1:53068) 2024-03-06T07:20:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.964+0000 7f9c9a7fc640 1 -- 172.21.15.1:0/1557943665 learned_addr learned my addr 172.21.15.1:0/1557943665 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.964+0000 7f9c9affd640 1 --2- 172.21.15.1:0/1557943665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c9c072bd0 0x7f9c9c1eb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.964+0000 7f9c9b7fe640 1 --2- 172.21.15.1:0/1557943665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c9c152b70 0x7f9c9c149820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.964+0000 7f9c9a7fc640 1 -- 172.21.15.1:0/1557943665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c9c072bd0 msgr2=0x7f9c9c1eb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.964+0000 7f9c9a7fc640 1 --2- 172.21.15.1:0/1557943665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c9c072bd0 0x7f9c9c1eb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.964+0000 7f9c9a7fc640 1 -- 172.21.15.1:0/1557943665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c9c152b70 msgr2=0x7f9c9c149820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.964+0000 7f9c9a7fc640 1 --2- 172.21.15.1:0/1557943665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c9c152b70 0x7f9c9c149820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.964+0000 7f9c9a7fc640 1 -- 172.21.15.1:0/1557943665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c1eb5a0 con 0x7f9c9c0c2b30 2024-03-06T07:20:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.964+0000 7f9c9b7fe640 1 --2- 172.21.15.1:0/1557943665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c9c152b70 0x7f9c9c149820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.964+0000 7f9c9a7fc640 1 --2- 172.21.15.1:0/1557943665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c9c0c2b30 0x7f9c9c0c3180 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9c9002b9e0 tx=0x7f9c90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.964+0000 7f9c837fe640 1 -- 172.21.15.1:0/1557943665 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c90048070 con 0x7f9c9c0c2b30 2024-03-06T07:20:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.964+0000 7f9c837fe640 1 -- 172.21.15.1:0/1557943665 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c90030660 con 0x7f9c9c0c2b30 2024-03-06T07:20:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.964+0000 7f9ca173e640 1 -- 172.21.15.1:0/1557943665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9c0bdc60 con 0x7f9c9c0c2b30 2024-03-06T07:20:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.964+0000 7f9c837fe640 1 -- 172.21.15.1:0/1557943665 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c90034040 con 0x7f9c9c0c2b30 2024-03-06T07:20:51.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.964+0000 7f9ca173e640 1 -- 172.21.15.1:0/1557943665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9c1e8660 con 0x7f9c9c0c2b30 2024-03-06T07:20:51.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.965+0000 7f9ca173e640 1 -- 172.21.15.1:0/1557943665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9c067fd0 con 0x7f9c9c0c2b30 2024-03-06T07:20:51.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.966+0000 7f9c837fe640 1 -- 172.21.15.1:0/1557943665 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c9004b070 con 0x7f9c9c0c2b30 2024-03-06T07:20:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.966+0000 7f9c837fe640 1 --2- 172.21.15.1:0/1557943665 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c8806b4d0 0x7f9c8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.966+0000 7f9c837fe640 1 -- 172.21.15.1:0/1557943665 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9c900ac5a0 con 0x7f9c9c0c2b30 2024-03-06T07:20:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.966+0000 7f9c9affd640 1 --2- 172.21.15.1:0/1557943665 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c8806b4d0 0x7f9c8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.967+0000 7f9c9affd640 1 --2- 172.21.15.1:0/1557943665 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c8806b4d0 0x7f9c8806d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c0015a0 tx=0x7f9c8c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:51.970+0000 7f9c837fe640 1 -- 172.21.15.1:0/1557943665 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c9007b170 con 0x7f9c9c0c2b30 2024-03-06T07:20:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:52.188+0000 7f9ca173e640 1 -- 172.21.15.1:0/1557943665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9c9c071790 con 0x7f9c9c0c2b30 2024-03-06T07:20:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:52.189+0000 7f9c837fe640 1 -- 172.21.15.1:0/1557943665 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f9c90030830 con 0x7f9c9c0c2b30 2024-03-06T07:20:52.191 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-06T07:20:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:52.190+0000 7f9ca173e640 1 -- 172.21.15.1:0/1557943665 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c8806b4d0 msgr2=0x7f9c8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:52.190+0000 7f9ca173e640 1 --2- 172.21.15.1:0/1557943665 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c8806b4d0 0x7f9c8806d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c0015a0 tx=0x7f9c8c009040 comp rx=0 tx=0).stop 2024-03-06T07:20:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:52.190+0000 7f9ca173e640 1 -- 172.21.15.1:0/1557943665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c9c0c2b30 msgr2=0x7f9c9c0c3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:52.190+0000 7f9ca173e640 1 --2- 172.21.15.1:0/1557943665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c9c0c2b30 0x7f9c9c0c3180 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9c9002b9e0 tx=0x7f9c90004730 comp rx=0 tx=0).stop 2024-03-06T07:20:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:52.191+0000 7f9ca173e640 1 -- 172.21.15.1:0/1557943665 shutdown_connections 2024-03-06T07:20:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:52.191+0000 7f9ca173e640 1 --2- 172.21.15.1:0/1557943665 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c8806b4d0 0x7f9c8806d990 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:52.191+0000 7f9ca173e640 1 --2- 172.21.15.1:0/1557943665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c9c152b70 0x7f9c9c149820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:52.191+0000 7f9ca173e640 1 --2- 172.21.15.1:0/1557943665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c9c0c2b30 0x7f9c9c0c3180 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:52.191+0000 7f9ca173e640 1 --2- 172.21.15.1:0/1557943665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c9c072bd0 0x7f9c9c1eb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:52.191+0000 7f9ca173e640 1 -- 172.21.15.1:0/1557943665 >> 172.21.15.1:0/1557943665 conn(0x7f9c9c0c6860 msgr2=0x7f9c9c0c4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:52.191+0000 7f9ca173e640 1 -- 172.21.15.1:0/1557943665 shutdown_connections 2024-03-06T07:20:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:52.191+0000 7f9ca173e640 1 -- 172.21.15.1:0/1557943665 wait complete. 2024-03-06T07:20:52.203 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:20:52.203 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:20:52.203 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-03-06T07:20:52.204 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-06T07:20:52.204 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:20:52.204 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:20:52.204 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:20:52.204 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:20:52.204 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:20:52.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:52.204 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-03-06T07:20:52.223 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:20:52.223 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:20:52.223 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:20:52.224 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:20:52.224 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-03-06T07:20:52.224 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:20:52.224 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:20:52.224 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:20:52.224 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-03-06T07:20:52.224 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:20:52.224 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:20:52.224 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-03-06T07:20:52.224 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:20:52.224 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:20:52.224 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:20:52.225 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:20:52.225 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:20:52.225 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42583sec preferred_lft 42583sec 2024-03-06T07:20:52.225 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-03-06T07:20:52.225 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:20:52.225 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:20:52.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:52.225 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2024-03-06T07:20:52.225 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-06T07:20:52.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:20:52.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2024-03-06T07:20:52.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2024-03-06T07:20:52.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:20:52.226 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-06T07:20:52.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:52.394 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:20:52.421 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-03-06T07:20:52.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:52.423 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-03-06T07:20:52.497 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-03-06T07:20:52.497 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:20:52.497 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:20:52.497 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:20:52.497 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:20:52.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:52.497 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2024-03-06T07:20:52.497 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-06T07:20:52.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:20:52.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:20:52.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:20:52.498 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-06T07:20:52.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:52.803 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-03-06T07:20:52.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:52.818 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2024-03-06T07:20:52.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:52.873 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2024-03-06T07:20:52.873 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-06T07:20:52.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:20:52.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-06T07:20:52.873 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-06T07:20:52.958 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-06T07:20:52.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:52.959 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2024-03-06T07:20:52.959 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-06T07:20:52.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-06T07:20:52.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:20:52.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-06T07:20:52.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-06T07:20:52.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-06T07:20:52.959 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-06T07:20:53.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:53.059 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2024-03-06T07:20:53.059 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-06T07:20:53.059 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-03-06T07:20:53.059 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:20:53.059 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-06T07:20:53.101 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:20:53.101 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-06T07:20:53.159 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:20:53.159 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:20:53.159 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:20:53.214 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-03-06T07:20:53.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:53.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:20:53.299 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:20:53.299 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:20:53.408 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:20:53.408 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:20:53.408 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.1:3300,172.21.15.98:3300,172.21.15.98:3301" 2024-03-06T07:20:53.408 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with new device syntax: 1@8f3c19ba-4ac6-4a54-88df-9262183d23ef.cephfs=/ 2024-03-06T07:20:53.408 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.1:3300/172.21.15.98:3300/172.21.15.98:3301" will pass to kernel 2024-03-06T07:20:53.409 INFO:teuthology.orchestra.run.smithi098.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-06T07:20:53.409 INFO:teuthology.orchestra.run.smithi098.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:20:53.409 INFO:teuthology.orchestra.run.smithi098.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:20:53.409 INFO:teuthology.orchestra.run.smithi098.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:20:53.409 INFO:teuthology.orchestra.run.smithi098.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:20:53.411 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:20:53.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:53.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:20:53.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:53.438 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:20:53.493 INFO:teuthology.orchestra.run.smithi098.stdout:client4592 2024-03-06T07:20:53.493 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:20:53.493 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:20:53.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:53.549 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:20:53.604 INFO:teuthology.orchestra.run.smithi098.stdout:client4592 2024-03-06T07:20:53.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:53.604 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:20:53.660 INFO:teuthology.orchestra.run.smithi098.stdout:client4592 2024-03-06T07:20:53.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:20:53.661 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/8f3c19ba-4ac6-4a54-88df-9262183d23ef.client4592/status') 2024-03-06T07:20:53.727 INFO:teuthology.orchestra.run.smithi098.stdout:instance: client.4592 (3)192.168.0.1:0/1082482593 2024-03-06T07:20:53.727 INFO:teuthology.orchestra.run.smithi098.stdout:blocklisted: false 2024-03-06T07:20:53.728 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records in 2024-03-06T07:20:53.728 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records out 2024-03-06T07:20:53.728 INFO:teuthology.orchestra.run.smithi098.stderr:69 bytes copied, 2.331e-05 s, 3.0 MB/s 2024-03-06T07:20:53.729 INFO:teuthology.run_tasks:Running task exec... 2024-03-06T07:20:53.739 INFO:teuthology.task.exec:Executing custom commands... 2024-03-06T07:20:53.739 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi001.front.sepia.ceph.com 2024-03-06T07:20:53.739 DEBUG:teuthology.orchestra.run.smithi001:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/crash/warn_recent_interval 0' 2024-03-06T07:20:53.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.843+0000 7f9436dc7640 1 Processor -- start 2024-03-06T07:20:53.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.843+0000 7f9436dc7640 1 -- start start 2024-03-06T07:20:53.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.844+0000 7f9436dc7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94300caf00 0x7f94300cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:53.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.844+0000 7f9436dc7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f94300c71f0 0x7f94300c7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:53.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.844+0000 7f9436dc7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9430155500 con 0x7f9430152920 2024-03-06T07:20:53.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.844+0000 7f9436dc7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9430155e30 con 0x7f94300caf00 2024-03-06T07:20:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.844+0000 7f9436dc7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94301557e0 con 0x7f94300c71f0 2024-03-06T07:20:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.844+0000 7f9434b63640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9430152920 0x7f94300c53a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46904/0 (socket says 172.21.15.1:46904) 2024-03-06T07:20:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.844+0000 7f9434b63640 1 -- 172.21.15.1:0/2265289088 learned_addr learned my addr 172.21.15.1:0/2265289088 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.844+0000 7f942ffff640 1 --2- 172.21.15.1:0/2265289088 >> v2:172.21.15.1:3300/0 conn(0x7f94300caf00 0x7f94300cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.844+0000 7f9435364640 1 --2- 172.21.15.1:0/2265289088 >> v2:172.21.15.98:3300/0 conn(0x7f94300c71f0 0x7f94300c7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.844+0000 7f942f7fe640 1 -- 172.21.15.1:0/2265289088 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307437138 0 0) 0x7f9430155500 con 0x7f9430152920 2024-03-06T07:20:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.844+0000 7f942f7fe640 1 -- 172.21.15.1:0/2265289088 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9418003520 con 0x7f9430152920 2024-03-06T07:20:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.844+0000 7f942ffff640 1 -- 172.21.15.1:0/2265289088 >> v2:172.21.15.98:3300/0 conn(0x7f94300c71f0 msgr2=0x7f94300c7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.844+0000 7f942ffff640 1 --2- 172.21.15.1:0/2265289088 >> v2:172.21.15.98:3300/0 conn(0x7f94300c71f0 0x7f94300c7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.844+0000 7f942ffff640 1 -- 172.21.15.1:0/2265289088 >> v1:172.21.15.1:6789/0 conn(0x7f9430152920 legacy=0x7f94300c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.844+0000 7f942ffff640 1 -- 172.21.15.1:0/2265289088 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9430155ac0 con 0x7f94300caf00 2024-03-06T07:20:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.844+0000 7f9435364640 1 --2- 172.21.15.1:0/2265289088 >> v2:172.21.15.98:3300/0 conn(0x7f94300c71f0 0x7f94300c7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:53.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.845+0000 7f942ffff640 1 --2- 172.21.15.1:0/2265289088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94300caf00 0x7f94300cb390 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f94300ca9f0 tx=0x7f942402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a505831405add63e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:53.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.845+0000 7f942f7fe640 1 -- 172.21.15.1:0/2265289088 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94240315a0 con 0x7f94300caf00 2024-03-06T07:20:53.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.845+0000 7f942f7fe640 1 -- 172.21.15.1:0/2265289088 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9424037070 con 0x7f94300caf00 2024-03-06T07:20:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.845+0000 7f942f7fe640 1 -- 172.21.15.1:0/2265289088 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9424031960 con 0x7f94300caf00 2024-03-06T07:20:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.846+0000 7f9436dc7640 1 -- 172.21.15.1:0/2265289088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94300caf00 msgr2=0x7f94300cb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.846+0000 7f9436dc7640 1 --2- 172.21.15.1:0/2265289088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94300caf00 0x7f94300cb390 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f94300ca9f0 tx=0x7f942402b840 comp rx=0 tx=0).stop 2024-03-06T07:20:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.846+0000 7f9436dc7640 1 -- 172.21.15.1:0/2265289088 shutdown_connections 2024-03-06T07:20:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.846+0000 7f9436dc7640 1 --2- 172.21.15.1:0/2265289088 >> v2:172.21.15.98:3300/0 conn(0x7f94300c71f0 0x7f94300c7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.846+0000 7f9436dc7640 1 --2- 172.21.15.1:0/2265289088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94300caf00 0x7f94300cb390 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.846+0000 7f9436dc7640 1 -- 172.21.15.1:0/2265289088 >> 172.21.15.1:0/2265289088 conn(0x7f9430067130 msgr2=0x7f9430067540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.846+0000 7f9436dc7640 1 -- 172.21.15.1:0/2265289088 shutdown_connections 2024-03-06T07:20:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.846+0000 7f9436dc7640 1 -- 172.21.15.1:0/2265289088 wait complete. 2024-03-06T07:20:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.847+0000 7f9436dc7640 1 Processor -- start 2024-03-06T07:20:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.847+0000 7f9436dc7640 1 -- start start 2024-03-06T07:20:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.848+0000 7f9436dc7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94300c71f0 0x7f94300c4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.848+0000 7f9436dc7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94300caf00 0x7f94300c2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.848+0000 7f9436dc7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9430152920 0x7f94300ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.848+0000 7f9436dc7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94300c5ab0 con 0x7f94300c71f0 2024-03-06T07:20:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.848+0000 7f9436dc7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943006d010 con 0x7f9430152920 2024-03-06T07:20:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.848+0000 7f9436dc7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94300c5c30 con 0x7f94300caf00 2024-03-06T07:20:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.848+0000 7f9434b63640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94300c71f0 0x7f94300c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.848+0000 7f9434b63640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94300c71f0 0x7f94300c4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53090/0 (socket says 172.21.15.1:53090) 2024-03-06T07:20:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.848+0000 7f9434b63640 1 -- 172.21.15.1:0/2010725480 learned_addr learned my addr 172.21.15.1:0/2010725480 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.848+0000 7f9435364640 1 --2- 172.21.15.1:0/2010725480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9430152920 0x7f94300ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.848+0000 7f942ffff640 1 --2- 172.21.15.1:0/2010725480 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94300caf00 0x7f94300c2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.848+0000 7f9434b63640 1 -- 172.21.15.1:0/2010725480 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94300caf00 msgr2=0x7f94300c2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.848+0000 7f9434b63640 1 --2- 172.21.15.1:0/2010725480 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94300caf00 0x7f94300c2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.848+0000 7f9434b63640 1 -- 172.21.15.1:0/2010725480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9430152920 msgr2=0x7f94300ae2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.848+0000 7f9434b63640 1 --2- 172.21.15.1:0/2010725480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9430152920 0x7f94300ae2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.848+0000 7f9434b63640 1 -- 172.21.15.1:0/2010725480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94300c26a0 con 0x7f94300c71f0 2024-03-06T07:20:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.848+0000 7f942ffff640 1 --2- 172.21.15.1:0/2010725480 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94300caf00 0x7f94300c2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.849+0000 7f9434b63640 1 --2- 172.21.15.1:0/2010725480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94300c71f0 0x7f94300c4750 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f942800c3c0 tx=0x7f942800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.849+0000 7f942d7fa640 1 -- 172.21.15.1:0/2010725480 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9428017070 con 0x7f94300c71f0 2024-03-06T07:20:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.849+0000 7f942d7fa640 1 -- 172.21.15.1:0/2010725480 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9428005ce0 con 0x7f94300c71f0 2024-03-06T07:20:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.849+0000 7f942d7fa640 1 -- 172.21.15.1:0/2010725480 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94280072f0 con 0x7f94300c71f0 2024-03-06T07:20:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.849+0000 7f9436dc7640 1 -- 172.21.15.1:0/2010725480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94301eeda0 con 0x7f94300c71f0 2024-03-06T07:20:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.849+0000 7f9436dc7640 1 -- 172.21.15.1:0/2010725480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9430150140 con 0x7f94300c71f0 2024-03-06T07:20:53.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.850+0000 7f9436dc7640 1 -- 172.21.15.1:0/2010725480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94300c33a0 con 0x7f94300c71f0 2024-03-06T07:20:53.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.850+0000 7f942d7fa640 1 -- 172.21.15.1:0/2010725480 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f942801a070 con 0x7f94300c71f0 2024-03-06T07:20:53.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.851+0000 7f942d7fa640 1 --2- 172.21.15.1:0/2010725480 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f941c06b500 0x7f941c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:53.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.851+0000 7f942d7fa640 1 -- 172.21.15.1:0/2010725480 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9428089890 con 0x7f94300c71f0 2024-03-06T07:20:53.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.851+0000 7f942ffff640 1 --2- 172.21.15.1:0/2010725480 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f941c06b500 0x7f941c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:53.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.851+0000 7f942ffff640 1 --2- 172.21.15.1:0/2010725480 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f941c06b500 0x7f941c06d9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9424036000 tx=0x7f9424008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:53.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:53.855+0000 7f942d7fa640 1 -- 172.21.15.1:0/2010725480 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94280583d0 con 0x7f94300c71f0 2024-03-06T07:20:54.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.040+0000 7f9436dc7640 1 -- 172.21.15.1:0/2010725480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/crash/warn_recent_interval}] v 0) v1 -- 0x7f94300c2b80 con 0x7f94300c71f0 2024-03-06T07:20:54.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.042+0000 7f942d7fa640 1 -- 172.21.15.1:0/2010725480 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/crash/warn_recent_interval}]=0 v10)=0 v10) v1 ==== 128+0+0 (secure 0 0 0) 0x7f94300c2b80 con 0x7f94300c71f0 2024-03-06T07:20:54.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.044+0000 7f9436dc7640 1 -- 172.21.15.1:0/2010725480 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f941c06b500 msgr2=0x7f941c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.044+0000 7f9436dc7640 1 --2- 172.21.15.1:0/2010725480 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f941c06b500 0x7f941c06d9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9424036000 tx=0x7f9424008030 comp rx=0 tx=0).stop 2024-03-06T07:20:54.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.044+0000 7f9436dc7640 1 -- 172.21.15.1:0/2010725480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94300c71f0 msgr2=0x7f94300c4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.044+0000 7f9436dc7640 1 --2- 172.21.15.1:0/2010725480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94300c71f0 0x7f94300c4750 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f942800c3c0 tx=0x7f942800c880 comp rx=0 tx=0).stop 2024-03-06T07:20:54.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.044+0000 7f9436dc7640 1 -- 172.21.15.1:0/2010725480 shutdown_connections 2024-03-06T07:20:54.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.044+0000 7f9436dc7640 1 --2- 172.21.15.1:0/2010725480 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f941c06b500 0x7f941c06d9c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.044+0000 7f9436dc7640 1 --2- 172.21.15.1:0/2010725480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9430152920 0x7f94300ae2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.044+0000 7f9436dc7640 1 --2- 172.21.15.1:0/2010725480 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94300caf00 0x7f94300c2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.044+0000 7f9436dc7640 1 --2- 172.21.15.1:0/2010725480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94300c71f0 0x7f94300c4750 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.044+0000 7f9436dc7640 1 -- 172.21.15.1:0/2010725480 >> 172.21.15.1:0/2010725480 conn(0x7f9430067130 msgr2=0x7f9430153830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:54.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.045+0000 7f9436dc7640 1 -- 172.21.15.1:0/2010725480 shutdown_connections 2024-03-06T07:20:54.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.045+0000 7f9436dc7640 1 -- 172.21.15.1:0/2010725480 wait complete. 2024-03-06T07:20:54.060 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-06T07:20:54.076 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:54.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.178+0000 7f825ff81640 1 Processor -- start 2024-03-06T07:20:54.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.179+0000 7f825ff81640 1 -- start start 2024-03-06T07:20:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.180+0000 7f825ff81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8258152840 0x7f82580c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.180+0000 7f825ff81640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82580c5e40 con 0x7f82580c3970 2024-03-06T07:20:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.180+0000 7f825ff81640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82580c6a40 con 0x7f82580c60b0 2024-03-06T07:20:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.180+0000 7f825ff81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82580c67f0 con 0x7f8258152840 2024-03-06T07:20:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.180+0000 7f825dd1d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f82580c3970 0x7f8258152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46908/0 (socket says 172.21.15.1:46908) 2024-03-06T07:20:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.180+0000 7f825dd1d640 1 -- 172.21.15.1:0/4128797645 learned_addr learned my addr 172.21.15.1:0/4128797645 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.180+0000 7f825d51c640 1 --2- 172.21.15.1:0/4128797645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8258152840 0x7f82580c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.180+0000 7f825cd1b640 1 -- 172.21.15.1:0/4128797645 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797357318 0 0) 0x7f82580c5e40 con 0x7f82580c3970 2024-03-06T07:20:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.180+0000 7f825cd1b640 1 -- 172.21.15.1:0/4128797645 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8244003520 con 0x7f82580c3970 2024-03-06T07:20:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825cd1b640 1 -- 172.21.15.1:0/4128797645 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48955010 0 0) 0x7f82580c6a40 con 0x7f82580c60b0 2024-03-06T07:20:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825cd1b640 1 -- 172.21.15.1:0/4128797645 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82580c5e40 con 0x7f82580c60b0 2024-03-06T07:20:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825cd1b640 1 -- 172.21.15.1:0/4128797645 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3462186186 0 0) 0x7f8244003520 con 0x7f82580c3970 2024-03-06T07:20:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825cd1b640 1 -- 172.21.15.1:0/4128797645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8258152840 msgr2=0x7f82580c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825cd1b640 1 --2- 172.21.15.1:0/4128797645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8258152840 0x7f82580c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825cd1b640 1 -- 172.21.15.1:0/4128797645 >> v1:172.21.15.98:6789/0 conn(0x7f82580c60b0 legacy=0x7f82581553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825cd1b640 1 -- 172.21.15.1:0/4128797645 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82580c6d20 con 0x7f82580c3970 2024-03-06T07:20:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825cd1b640 1 -- 172.21.15.1:0/4128797645 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f824c001670 con 0x7f82580c3970 2024-03-06T07:20:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825d51c640 1 --2- 172.21.15.1:0/4128797645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8258152840 0x7f82580c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825cd1b640 1 -- 172.21.15.1:0/4128797645 >> v1:172.21.15.1:6789/0 conn(0x7f82580c3970 legacy=0x7f8258152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825cd1b640 1 --2- 172.21.15.1:0/4128797645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8244003b70 0x7f8244003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825cd1b640 1 --2- 172.21.15.1:0/4128797645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8258152840 0x7f8244004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825cd1b640 1 --2- 172.21.15.1:0/4128797645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8244004d40 0x7f82440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825cd1b640 1 -- 172.21.15.1:0/4128797645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82580c67f0 con 0x7f8244003b70 2024-03-06T07:20:54.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825cd1b640 1 -- 172.21.15.1:0/4128797645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824400ba70 con 0x7f8244004d40 2024-03-06T07:20:54.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825cd1b640 1 -- 172.21.15.1:0/4128797645 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824400bc70 con 0x7f8258152840 2024-03-06T07:20:54.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825dd1d640 1 --2- 172.21.15.1:0/4128797645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8244003b70 0x7f8244003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825d51c640 1 --2- 172.21.15.1:0/4128797645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8258152840 0x7f8244004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825e51e640 1 --2- 172.21.15.1:0/4128797645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8244004d40 0x7f82440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825dd1d640 1 -- 172.21.15.1:0/4128797645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8258152840 msgr2=0x7f8244004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:54.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825dd1d640 1 --2- 172.21.15.1:0/4128797645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8258152840 0x7f8244004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825dd1d640 1 -- 172.21.15.1:0/4128797645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8244004d40 msgr2=0x7f82440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:54.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825dd1d640 1 --2- 172.21.15.1:0/4128797645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8244004d40 0x7f82440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.181+0000 7f825dd1d640 1 -- 172.21.15.1:0/4128797645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8244003520 con 0x7f8244003b70 2024-03-06T07:20:54.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.182+0000 7f825dd1d640 1 --2- 172.21.15.1:0/4128797645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8244003b70 0x7f8244003f70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f824c007fa0 tx=0x7f8244005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7534085f818f827 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.182+0000 7f825cd1b640 1 -- 172.21.15.1:0/4128797645 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824c007bc0 con 0x7f8244003b70 2024-03-06T07:20:54.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.182+0000 7f825cd1b640 1 -- 172.21.15.1:0/4128797645 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f824c030070 con 0x7f8244003b70 2024-03-06T07:20:54.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.182+0000 7f825cd1b640 1 -- 172.21.15.1:0/4128797645 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824c005d30 con 0x7f8244003b70 2024-03-06T07:20:54.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.183+0000 7f825ff81640 1 -- 172.21.15.1:0/4128797645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8244003b70 msgr2=0x7f8244003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.183+0000 7f825ff81640 1 --2- 172.21.15.1:0/4128797645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8244003b70 0x7f8244003f70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f824c007fa0 tx=0x7f8244005830 comp rx=0 tx=0).stop 2024-03-06T07:20:54.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.183+0000 7f825e51e640 1 -- 172.21.15.1:0/4128797645 reap_dead start 2024-03-06T07:20:54.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.183+0000 7f825ff81640 1 -- 172.21.15.1:0/4128797645 shutdown_connections 2024-03-06T07:20:54.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.183+0000 7f825ff81640 1 -- 172.21.15.1:0/4128797645 >> 172.21.15.1:0/4128797645 conn(0x7f82580c88f0 msgr2=0x7f82580c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:54.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.183+0000 7f825ff81640 1 -- 172.21.15.1:0/4128797645 shutdown_connections 2024-03-06T07:20:54.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.183+0000 7f825ff81640 1 -- 172.21.15.1:0/4128797645 wait complete. 2024-03-06T07:20:54.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.184+0000 7f825ff81640 1 Processor -- start 2024-03-06T07:20:54.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.184+0000 7f825ff81640 1 -- start start 2024-03-06T07:20:54.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.184+0000 7f825ff81640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8244003b70 0x7f825814f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.184+0000 7f825ff81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82580ad7c0 0x7f82580ac640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.184+0000 7f825ff81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82580acb80 0x7f82581edd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.184+0000 7f825ff81640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82580c75c0 con 0x7f8244003b70 2024-03-06T07:20:54.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.184+0000 7f825ff81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82580c72e0 con 0x7f82580ad7c0 2024-03-06T07:20:54.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.184+0000 7f825ff81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82580c3d50 con 0x7f82580acb80 2024-03-06T07:20:54.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.185+0000 7f825dd1d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8244003b70 0x7f825814f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.185+0000 7f825dd1d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8244003b70 0x7f825814f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53102/0 (socket says 172.21.15.1:53102) 2024-03-06T07:20:54.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.185+0000 7f825dd1d640 1 -- 172.21.15.1:0/1205819492 learned_addr learned my addr 172.21.15.1:0/1205819492 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:54.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.185+0000 7f825d51c640 1 --2- 172.21.15.1:0/1205819492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82580ad7c0 0x7f82580ac640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.185+0000 7f825e51e640 1 --2- 172.21.15.1:0/1205819492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82580acb80 0x7f82581edd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.185+0000 7f825dd1d640 1 -- 172.21.15.1:0/1205819492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82580acb80 msgr2=0x7f82581edd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.185+0000 7f825dd1d640 1 --2- 172.21.15.1:0/1205819492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82580acb80 0x7f82581edd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.185+0000 7f825dd1d640 1 -- 172.21.15.1:0/1205819492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82580ad7c0 msgr2=0x7f82580ac640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.185+0000 7f825dd1d640 1 --2- 172.21.15.1:0/1205819492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82580ad7c0 0x7f82580ac640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.185+0000 7f825dd1d640 1 -- 172.21.15.1:0/1205819492 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82581eb290 con 0x7f8244003b70 2024-03-06T07:20:54.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.185+0000 7f825dd1d640 1 --2- 172.21.15.1:0/1205819492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8244003b70 0x7f825814f420 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f824c001200 tx=0x7f82581550c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.185+0000 7f824a7fc640 1 -- 172.21.15.1:0/1205819492 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824c007bc0 con 0x7f8244003b70 2024-03-06T07:20:54.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.185+0000 7f824a7fc640 1 -- 172.21.15.1:0/1205819492 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f824c02cba0 con 0x7f8244003b70 2024-03-06T07:20:54.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.185+0000 7f825ff81640 1 -- 172.21.15.1:0/1205819492 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82581eb570 con 0x7f8244003b70 2024-03-06T07:20:54.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.185+0000 7f824a7fc640 1 -- 172.21.15.1:0/1205819492 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824c001b20 con 0x7f8244003b70 2024-03-06T07:20:54.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.185+0000 7f825ff81640 1 -- 172.21.15.1:0/1205819492 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82580abe40 con 0x7f8244003b70 2024-03-06T07:20:54.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.186+0000 7f825ff81640 1 -- 172.21.15.1:0/1205819492 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82580686e0 con 0x7f8244003b70 2024-03-06T07:20:54.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.187+0000 7f824a7fc640 1 -- 172.21.15.1:0/1205819492 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f824c007070 con 0x7f8244003b70 2024-03-06T07:20:54.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.187+0000 7f824a7fc640 1 --2- 172.21.15.1:0/1205819492 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f823c06b500 0x7f823c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.187+0000 7f824a7fc640 1 -- 172.21.15.1:0/1205819492 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f824c0ab6c0 con 0x7f8244003b70 2024-03-06T07:20:54.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.187+0000 7f825d51c640 1 --2- 172.21.15.1:0/1205819492 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f823c06b500 0x7f823c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.188+0000 7f825d51c640 1 --2- 172.21.15.1:0/1205819492 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f823c06b500 0x7f823c06d9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8258074b70 tx=0x7f8254009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.191+0000 7f824a7fc640 1 -- 172.21.15.1:0/1205819492 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f824c079c40 con 0x7f8244003b70 2024-03-06T07:20:54.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.371+0000 7f825ff81640 1 -- 172.21.15.1:0/1205819492 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82580c86d0 con 0x7f8244003b70 2024-03-06T07:20:54.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.373+0000 7f824a7fc640 1 -- 172.21.15.1:0/1205819492 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7f824c07dce0 con 0x7f8244003b70 2024-03-06T07:20:54.375 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:54.375 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:54.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.374+0000 7f825ff81640 1 -- 172.21.15.1:0/1205819492 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f823c06b500 msgr2=0x7f823c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.375+0000 7f825ff81640 1 --2- 172.21.15.1:0/1205819492 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f823c06b500 0x7f823c06d9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8258074b70 tx=0x7f8254009190 comp rx=0 tx=0).stop 2024-03-06T07:20:54.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.375+0000 7f825ff81640 1 -- 172.21.15.1:0/1205819492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8244003b70 msgr2=0x7f825814f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.375+0000 7f825ff81640 1 --2- 172.21.15.1:0/1205819492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8244003b70 0x7f825814f420 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f824c001200 tx=0x7f82581550c0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.375+0000 7f825ff81640 1 -- 172.21.15.1:0/1205819492 shutdown_connections 2024-03-06T07:20:54.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.375+0000 7f825ff81640 1 --2- 172.21.15.1:0/1205819492 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f823c06b500 0x7f823c06d9c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.375+0000 7f825ff81640 1 --2- 172.21.15.1:0/1205819492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82580acb80 0x7f82581edd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.375+0000 7f825ff81640 1 --2- 172.21.15.1:0/1205819492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82580ad7c0 0x7f82580ac640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.375+0000 7f825ff81640 1 --2- 172.21.15.1:0/1205819492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8244003b70 0x7f825814f420 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.375+0000 7f825ff81640 1 -- 172.21.15.1:0/1205819492 >> 172.21.15.1:0/1205819492 conn(0x7f82580c88f0 msgr2=0x7f825806d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:54.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.375+0000 7f825ff81640 1 -- 172.21.15.1:0/1205819492 shutdown_connections 2024-03-06T07:20:54.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.375+0000 7f825ff81640 1 -- 172.21.15.1:0/1205819492 wait complete. 2024-03-06T07:20:54.388 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:20:19.337560+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:20:20.172791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:20:23.884071+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:20:54.388 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:20:54.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.491+0000 7f17d560e640 1 Processor -- start 2024-03-06T07:20:54.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.491+0000 7f17d560e640 1 -- start start 2024-03-06T07:20:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.492+0000 7f17d560e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f17d0150520 0x7f17d01543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.492+0000 7f17d560e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c32f0 0x7f17d01537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.492+0000 7f17d560e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d0150900 con 0x7f17d0070c80 2024-03-06T07:20:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.492+0000 7f17d560e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d0071790 con 0x7f17d0150520 2024-03-06T07:20:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.492+0000 7f17d560e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d0157050 con 0x7f17d00c32f0 2024-03-06T07:20:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.493+0000 7f17cf7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f17d0070c80 0x7f17d00c67f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36286/0 (socket says 172.21.15.1:36286) 2024-03-06T07:20:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.493+0000 7f17cf7fe640 1 -- 172.21.15.1:0/2454175193 learned_addr learned my addr 172.21.15.1:0/2454175193 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.493+0000 7f17ce7fc640 1 --2- 172.21.15.1:0/2454175193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c32f0 0x7f17d01537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.493+0000 7f17ceffd640 1 --2- 172.21.15.1:0/2454175193 >> v2:172.21.15.98:3300/0 conn(0x7f17d0150520 0x7f17d01543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.493+0000 7f17cdffb640 1 -- 172.21.15.1:0/2454175193 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535947849 0 0) 0x7f17d0150900 con 0x7f17d0070c80 2024-03-06T07:20:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.493+0000 7f17cdffb640 1 -- 172.21.15.1:0/2454175193 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17bc003520 con 0x7f17d0070c80 2024-03-06T07:20:54.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.493+0000 7f17ceffd640 1 -- 172.21.15.1:0/2454175193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c32f0 msgr2=0x7f17d01537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:54.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.493+0000 7f17ceffd640 1 --2- 172.21.15.1:0/2454175193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c32f0 0x7f17d01537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.493+0000 7f17ceffd640 1 -- 172.21.15.1:0/2454175193 >> v1:172.21.15.1:6789/0 conn(0x7f17d0070c80 legacy=0x7f17d00c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.493+0000 7f17ceffd640 1 -- 172.21.15.1:0/2454175193 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d0152fc0 con 0x7f17d0150520 2024-03-06T07:20:54.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.493+0000 7f17ce7fc640 1 --2- 172.21.15.1:0/2454175193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c32f0 0x7f17d01537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:54.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.494+0000 7f17ceffd640 1 --2- 172.21.15.1:0/2454175193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d0150520 0x7f17d01543a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f17c00097a0 tx=0x7f17c002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d073b81b78b296b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.494+0000 7f17cdffb640 1 -- 172.21.15.1:0/2454175193 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c0035020 con 0x7f17d0150520 2024-03-06T07:20:54.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.494+0000 7f17cdffb640 1 -- 172.21.15.1:0/2454175193 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c0031830 con 0x7f17d0150520 2024-03-06T07:20:54.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.494+0000 7f17cdffb640 1 -- 172.21.15.1:0/2454175193 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c0031c00 con 0x7f17d0150520 2024-03-06T07:20:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.494+0000 7f17d560e640 1 -- 172.21.15.1:0/2454175193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d0150520 msgr2=0x7f17d01543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.494+0000 7f17d560e640 1 --2- 172.21.15.1:0/2454175193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d0150520 0x7f17d01543a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f17c00097a0 tx=0x7f17c002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.494+0000 7f17d560e640 1 -- 172.21.15.1:0/2454175193 shutdown_connections 2024-03-06T07:20:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.494+0000 7f17d560e640 1 --2- 172.21.15.1:0/2454175193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c32f0 0x7f17d01537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.494+0000 7f17d560e640 1 --2- 172.21.15.1:0/2454175193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d0150520 0x7f17d01543a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.495+0000 7f17d560e640 1 -- 172.21.15.1:0/2454175193 >> 172.21.15.1:0/2454175193 conn(0x7f17d00c88f0 msgr2=0x7f17d00c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.495+0000 7f17d560e640 1 -- 172.21.15.1:0/2454175193 shutdown_connections 2024-03-06T07:20:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.495+0000 7f17d560e640 1 -- 172.21.15.1:0/2454175193 wait complete. 2024-03-06T07:20:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.496+0000 7f17d560e640 1 Processor -- start 2024-03-06T07:20:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.496+0000 7f17d560e640 1 -- start start 2024-03-06T07:20:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.496+0000 7f17d560e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d0070c80 0x7f17d014f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.496+0000 7f17d560e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c32f0 0x7f17d0143270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.496+0000 7f17d560e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d0150520 0x7f17d01437b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17d560e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d00c7ec0 con 0x7f17d0070c80 2024-03-06T07:20:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17d560e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d0157330 con 0x7f17d0150520 2024-03-06T07:20:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17d560e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d0153d00 con 0x7f17d00c32f0 2024-03-06T07:20:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17ceffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d0070c80 0x7f17d014f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17ceffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d0070c80 0x7f17d014f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36764/0 (socket says 172.21.15.1:36764) 2024-03-06T07:20:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17ceffd640 1 -- 172.21.15.1:0/205606574 learned_addr learned my addr 172.21.15.1:0/205606574 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17ce7fc640 1 --2- 172.21.15.1:0/205606574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c32f0 0x7f17d0143270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17cf7fe640 1 --2- 172.21.15.1:0/205606574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d0150520 0x7f17d01437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17ceffd640 1 -- 172.21.15.1:0/205606574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c32f0 msgr2=0x7f17d0143270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17ceffd640 1 --2- 172.21.15.1:0/205606574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c32f0 0x7f17d0143270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17ceffd640 1 -- 172.21.15.1:0/205606574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d0150520 msgr2=0x7f17d01437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17ceffd640 1 --2- 172.21.15.1:0/205606574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d0150520 0x7f17d01437b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17ceffd640 1 -- 172.21.15.1:0/205606574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d01ec830 con 0x7f17d0070c80 2024-03-06T07:20:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17ceffd640 1 --2- 172.21.15.1:0/205606574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d0070c80 0x7f17d014f740 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f17c00098c0 tx=0x7f17c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17c77fe640 1 -- 172.21.15.1:0/205606574 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c0040070 con 0x7f17d0070c80 2024-03-06T07:20:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17c77fe640 1 -- 172.21.15.1:0/205606574 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c003e070 con 0x7f17d0070c80 2024-03-06T07:20:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17d560e640 1 -- 172.21.15.1:0/205606574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d00c3f40 con 0x7f17d0070c80 2024-03-06T07:20:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17c77fe640 1 -- 172.21.15.1:0/205606574 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c0035040 con 0x7f17d0070c80 2024-03-06T07:20:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.497+0000 7f17d560e640 1 -- 172.21.15.1:0/205606574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d014fc80 con 0x7f17d0070c80 2024-03-06T07:20:54.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.498+0000 7f17d560e640 1 -- 172.21.15.1:0/205606574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d00686e0 con 0x7f17d0070c80 2024-03-06T07:20:54.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.499+0000 7f17c77fe640 1 -- 172.21.15.1:0/205606574 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17c0038020 con 0x7f17d0070c80 2024-03-06T07:20:54.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.499+0000 7f17c77fe640 1 --2- 172.21.15.1:0/205606574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17b006b500 0x7f17b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.499+0000 7f17c77fe640 1 -- 172.21.15.1:0/205606574 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f17c00ab850 con 0x7f17d0070c80 2024-03-06T07:20:54.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.499+0000 7f17ce7fc640 1 --2- 172.21.15.1:0/205606574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17b006b500 0x7f17b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.500+0000 7f17ce7fc640 1 --2- 172.21.15.1:0/205606574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17b006b500 0x7f17b006d9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f17d014ddf0 tx=0x7f17b8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.504+0000 7f17c77fe640 1 -- 172.21.15.1:0/205606574 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17c0079d50 con 0x7f17d0070c80 2024-03-06T07:20:54.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.684+0000 7f17d560e640 1 -- 172.21.15.1:0/205606574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f17d00c7b10 con 0x7f17d0070c80 2024-03-06T07:20:54.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.685+0000 7f17c77fe640 1 -- 172.21.15.1:0/205606574 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f17c007ddf0 con 0x7f17d0070c80 2024-03-06T07:20:54.687 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:20:54.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.686+0000 7f17d560e640 1 -- 172.21.15.1:0/205606574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17b006b500 msgr2=0x7f17b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.686+0000 7f17d560e640 1 --2- 172.21.15.1:0/205606574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17b006b500 0x7f17b006d9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f17d014ddf0 tx=0x7f17b8009190 comp rx=0 tx=0).stop 2024-03-06T07:20:54.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.686+0000 7f17d560e640 1 -- 172.21.15.1:0/205606574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d0070c80 msgr2=0x7f17d014f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.686+0000 7f17d560e640 1 --2- 172.21.15.1:0/205606574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d0070c80 0x7f17d014f740 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f17c00098c0 tx=0x7f17c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.687+0000 7f17d560e640 1 -- 172.21.15.1:0/205606574 shutdown_connections 2024-03-06T07:20:54.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.687+0000 7f17d560e640 1 --2- 172.21.15.1:0/205606574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17b006b500 0x7f17b006d9c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.687+0000 7f17d560e640 1 --2- 172.21.15.1:0/205606574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17d0150520 0x7f17d01437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.687+0000 7f17d560e640 1 --2- 172.21.15.1:0/205606574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17d00c32f0 0x7f17d0143270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.687+0000 7f17d560e640 1 --2- 172.21.15.1:0/205606574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17d0070c80 0x7f17d014f740 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.687+0000 7f17d560e640 1 -- 172.21.15.1:0/205606574 >> 172.21.15.1:0/205606574 conn(0x7f17d00c88f0 msgr2=0x7f17d014ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:54.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.687+0000 7f17d560e640 1 -- 172.21.15.1:0/205606574 shutdown_connections 2024-03-06T07:20:54.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.687+0000 7f17d560e640 1 -- 172.21.15.1:0/205606574 wait complete. 2024-03-06T07:20:54.700 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:20:54.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.802+0000 7f4ae8d3a640 1 Processor -- start 2024-03-06T07:20:54.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.802+0000 7f4ae8d3a640 1 -- start start 2024-03-06T07:20:54.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.803+0000 7f4ae8d3a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4ae406b260 0x7f4ae406b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.803+0000 7f4ae8d3a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4ae40c1e60 0x7f4ae4158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.803+0000 7f4ae8d3a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae4072d90 con 0x7f4ae40c1910 2024-03-06T07:20:54.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.803+0000 7f4ae8d3a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ae40c10b0 con 0x7f4ae40c1e60 2024-03-06T07:20:54.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.803+0000 7f4ae8d3a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ae40c1540 con 0x7f4ae406b260 2024-03-06T07:20:54.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.803+0000 7f4ae2d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4ae40c1e60 0x7f4ae4158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.803+0000 7f4ad9d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4ae40c1910 0x7f4ae40c45b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36292/0 (socket says 172.21.15.1:36292) 2024-03-06T07:20:54.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.803+0000 7f4ad9d9b640 1 -- 172.21.15.1:0/1905551429 learned_addr learned my addr 172.21.15.1:0/1905551429 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.803+0000 7f4ae259c640 1 --2- 172.21.15.1:0/1905551429 >> v2:172.21.15.98:3300/0 conn(0x7f4ae406b260 0x7f4ae406b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.803+0000 7f4ae2d9d640 1 -- 172.21.15.1:0/1905551429 >> v2:172.21.15.98:3300/0 conn(0x7f4ae406b260 msgr2=0x7f4ae406b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.803+0000 7f4ae2d9d640 1 --2- 172.21.15.1:0/1905551429 >> v2:172.21.15.98:3300/0 conn(0x7f4ae406b260 0x7f4ae406b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.803+0000 7f4ae2d9d640 1 -- 172.21.15.1:0/1905551429 >> v1:172.21.15.1:6789/0 conn(0x7f4ae40c1910 legacy=0x7f4ae40c45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.803+0000 7f4ae2d9d640 1 -- 172.21.15.1:0/1905551429 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae406bc50 con 0x7f4ae40c1e60 2024-03-06T07:20:54.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.803+0000 7f4ae1d9b640 1 -- 172.21.15.1:0/1905551429 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781664309 0 0) 0x7f4ae4072d90 con 0x7f4ae40c1910 2024-03-06T07:20:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.804+0000 7f4ae259c640 1 --2- 172.21.15.1:0/1905551429 >> v2:172.21.15.98:3300/0 conn(0x7f4ae406b260 0x7f4ae406b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.804+0000 7f4ae2d9d640 1 --2- 172.21.15.1:0/1905551429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae40c1e60 0x7f4ae4158e10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4ae40c5610 tx=0x7f4adc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e28520fa1a5f30c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.804+0000 7f4ae1d9b640 1 -- 172.21.15.1:0/1905551429 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4adc039070 con 0x7f4ae40c1e60 2024-03-06T07:20:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.804+0000 7f4ae1d9b640 1 -- 172.21.15.1:0/1905551429 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4adc02cbf0 con 0x7f4ae40c1e60 2024-03-06T07:20:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.804+0000 7f4ae1d9b640 1 -- 172.21.15.1:0/1905551429 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4adc034440 con 0x7f4ae40c1e60 2024-03-06T07:20:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.805+0000 7f4ae8d3a640 1 -- 172.21.15.1:0/1905551429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae40c1e60 msgr2=0x7f4ae4158e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.805+0000 7f4ae8d3a640 1 --2- 172.21.15.1:0/1905551429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae40c1e60 0x7f4ae4158e10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4ae40c5610 tx=0x7f4adc02be40 comp rx=0 tx=0).stop 2024-03-06T07:20:54.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.805+0000 7f4ae8d3a640 1 -- 172.21.15.1:0/1905551429 shutdown_connections 2024-03-06T07:20:54.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.805+0000 7f4ae8d3a640 1 --2- 172.21.15.1:0/1905551429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae40c1e60 0x7f4ae4158e10 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.805+0000 7f4ae8d3a640 1 --2- 172.21.15.1:0/1905551429 >> v2:172.21.15.98:3300/0 conn(0x7f4ae406b260 0x7f4ae406b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.805+0000 7f4ae8d3a640 1 -- 172.21.15.1:0/1905551429 >> 172.21.15.1:0/1905551429 conn(0x7f4ae40c7a50 msgr2=0x7f4ae40c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:54.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.805+0000 7f4ae8d3a640 1 -- 172.21.15.1:0/1905551429 shutdown_connections 2024-03-06T07:20:54.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.805+0000 7f4ae8d3a640 1 -- 172.21.15.1:0/1905551429 wait complete. 2024-03-06T07:20:54.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.806+0000 7f4ae8d3a640 1 Processor -- start 2024-03-06T07:20:54.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.806+0000 7f4ae8d3a640 1 -- start start 2024-03-06T07:20:54.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.807+0000 7f4ae8d3a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae406b260 0x7f4ae40a9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.807+0000 7f4ae8d3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4ae40c1910 0x7f4ae40a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.807+0000 7f4ae8d3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae40c1e60 0x7f4ae40a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.807+0000 7f4ae8d3a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae40c4cc0 con 0x7f4ae406b260 2024-03-06T07:20:54.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.807+0000 7f4ae8d3a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae40c6320 con 0x7f4ae40c1e60 2024-03-06T07:20:54.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.807+0000 7f4ae8d3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae4159350 con 0x7f4ae40c1910 2024-03-06T07:20:54.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.807+0000 7f4ae259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae406b260 0x7f4ae40a9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.807+0000 7f4ae259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae406b260 0x7f4ae40a9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36786/0 (socket says 172.21.15.1:36786) 2024-03-06T07:20:54.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.807+0000 7f4ae259c640 1 -- 172.21.15.1:0/2008152869 learned_addr learned my addr 172.21.15.1:0/2008152869 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:54.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.807+0000 7f4ad9d9b640 1 --2- 172.21.15.1:0/2008152869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4ae40c1910 0x7f4ae40a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.807+0000 7f4ae2d9d640 1 --2- 172.21.15.1:0/2008152869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae40c1e60 0x7f4ae40a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.807+0000 7f4ae259c640 1 -- 172.21.15.1:0/2008152869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4ae40c1910 msgr2=0x7f4ae40a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.807+0000 7f4ae259c640 1 --2- 172.21.15.1:0/2008152869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4ae40c1910 0x7f4ae40a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.807+0000 7f4ae259c640 1 -- 172.21.15.1:0/2008152869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae40c1e60 msgr2=0x7f4ae40a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.807+0000 7f4ae259c640 1 --2- 172.21.15.1:0/2008152869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae40c1e60 0x7f4ae40a5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.808+0000 7f4ae259c640 1 -- 172.21.15.1:0/2008152869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae409a950 con 0x7f4ae406b260 2024-03-06T07:20:54.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.808+0000 7f4ae259c640 1 --2- 172.21.15.1:0/2008152869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae406b260 0x7f4ae40a9a50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400c820 tx=0x7f4ad400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.808+0000 7f4adb7fe640 1 -- 172.21.15.1:0/2008152869 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad4015070 con 0x7f4ae406b260 2024-03-06T07:20:54.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.808+0000 7f4adb7fe640 1 -- 172.21.15.1:0/2008152869 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad4007a30 con 0x7f4ae406b260 2024-03-06T07:20:54.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.808+0000 7f4ae8d3a640 1 -- 172.21.15.1:0/2008152869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ae4070bf0 con 0x7f4ae406b260 2024-03-06T07:20:54.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.808+0000 7f4adb7fe640 1 -- 172.21.15.1:0/2008152869 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad4007e00 con 0x7f4ae406b260 2024-03-06T07:20:54.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.808+0000 7f4ae8d3a640 1 -- 172.21.15.1:0/2008152869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ae4156200 con 0x7f4ae406b260 2024-03-06T07:20:54.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.809+0000 7f4ae8d3a640 1 -- 172.21.15.1:0/2008152869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ae40686e0 con 0x7f4ae406b260 2024-03-06T07:20:54.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.810+0000 7f4adb7fe640 1 -- 172.21.15.1:0/2008152869 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ad4018070 con 0x7f4ae406b260 2024-03-06T07:20:54.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.810+0000 7f4adb7fe640 1 --2- 172.21.15.1:0/2008152869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4ac406b500 0x7f4ac406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.811+0000 7f4adb7fe640 1 -- 172.21.15.1:0/2008152869 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4ad408a410 con 0x7f4ae406b260 2024-03-06T07:20:54.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.811+0000 7f4ad9d9b640 1 --2- 172.21.15.1:0/2008152869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4ac406b500 0x7f4ac406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.811+0000 7f4ad9d9b640 1 --2- 172.21.15.1:0/2008152869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4ac406b500 0x7f4ac406d9c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4ae406fe60 tx=0x7f4acc007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.815+0000 7f4adb7fe640 1 -- 172.21.15.1:0/2008152869 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ad4058910 con 0x7f4ae406b260 2024-03-06T07:20:54.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.996+0000 7f4ae8d3a640 1 -- 172.21.15.1:0/2008152869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4ae40a9f90 con 0x7f4ae406b260 2024-03-06T07:20:54.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.997+0000 7f4adb7fe640 1 -- 172.21.15.1:0/2008152869 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4ad405c9b0 con 0x7f4ae406b260 2024-03-06T07:20:54.999 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:20:55.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.998+0000 7f4ae8d3a640 1 -- 172.21.15.1:0/2008152869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4ac406b500 msgr2=0x7f4ac406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.998+0000 7f4ae8d3a640 1 --2- 172.21.15.1:0/2008152869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4ac406b500 0x7f4ac406d9c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4ae406fe60 tx=0x7f4acc007d50 comp rx=0 tx=0).stop 2024-03-06T07:20:55.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.998+0000 7f4ae8d3a640 1 -- 172.21.15.1:0/2008152869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae406b260 msgr2=0x7f4ae40a9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.998+0000 7f4ae8d3a640 1 --2- 172.21.15.1:0/2008152869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae406b260 0x7f4ae40a9a50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400c820 tx=0x7f4ad400cc20 comp rx=0 tx=0).stop 2024-03-06T07:20:55.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.998+0000 7f4ae8d3a640 1 -- 172.21.15.1:0/2008152869 shutdown_connections 2024-03-06T07:20:55.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.998+0000 7f4ae8d3a640 1 --2- 172.21.15.1:0/2008152869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4ac406b500 0x7f4ac406d9c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.998+0000 7f4ae8d3a640 1 --2- 172.21.15.1:0/2008152869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae40c1e60 0x7f4ae40a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.998+0000 7f4ae8d3a640 1 --2- 172.21.15.1:0/2008152869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4ae40c1910 0x7f4ae40a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.998+0000 7f4ae8d3a640 1 --2- 172.21.15.1:0/2008152869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae406b260 0x7f4ae40a9a50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.999+0000 7f4ae8d3a640 1 -- 172.21.15.1:0/2008152869 >> 172.21.15.1:0/2008152869 conn(0x7f4ae40c7a50 msgr2=0x7f4ae4074450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:55.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.999+0000 7f4ae8d3a640 1 -- 172.21.15.1:0/2008152869 shutdown_connections 2024-03-06T07:20:55.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:54.999+0000 7f4ae8d3a640 1 -- 172.21.15.1:0/2008152869 wait complete. 2024-03-06T07:20:55.011 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:20:55.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.116+0000 7fbe9c2af640 1 Processor -- start 2024-03-06T07:20:55.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.116+0000 7fbe9c2af640 1 -- start start 2024-03-06T07:20:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.117+0000 7fbe9c2af640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbe940c3970 0x7fbe940c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.117+0000 7fbe9c2af640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe940c60b0 0x7fbe94152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.117+0000 7fbe9c2af640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe940c5e40 con 0x7fbe94152670 2024-03-06T07:20:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.117+0000 7fbe9c2af640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe940c67f0 con 0x7fbe940c3970 2024-03-06T07:20:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.117+0000 7fbe9c2af640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe940c6a40 con 0x7fbe940c60b0 2024-03-06T07:20:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.118+0000 7fbe9a04b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbe940c3970 0x7fbe940c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.118+0000 7fbe9a04b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbe940c3970 0x7fbe940c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36800/0 (socket says 172.21.15.1:36800) 2024-03-06T07:20:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.118+0000 7fbe9a04b640 1 -- 172.21.15.1:0/3435118742 learned_addr learned my addr 172.21.15.1:0/3435118742 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:55.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.118+0000 7fbe9984a640 1 --2- 172.21.15.1:0/3435118742 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe940c60b0 0x7fbe94152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.118+0000 7fbe9a04b640 1 -- 172.21.15.1:0/3435118742 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe940c60b0 msgr2=0x7fbe94152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.118+0000 7fbe9a04b640 1 --2- 172.21.15.1:0/3435118742 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe940c60b0 0x7fbe94152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.118+0000 7fbe9a04b640 1 -- 172.21.15.1:0/3435118742 >> v1:172.21.15.98:6789/0 conn(0x7fbe94152670 legacy=0x7fbe941553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.118+0000 7fbe9a04b640 1 -- 172.21.15.1:0/3435118742 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe940c6d20 con 0x7fbe940c3970 2024-03-06T07:20:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.119+0000 7fbe9a04b640 1 --2- 172.21.15.1:0/3435118742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe940c3970 0x7fbe940c8170 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c009870 tx=0x7fbe8c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e9fa86c2d57acef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.119+0000 7fbe99049640 1 -- 172.21.15.1:0/3435118742 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c034070 con 0x7fbe940c3970 2024-03-06T07:20:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.119+0000 7fbe99049640 1 -- 172.21.15.1:0/3435118742 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8c030840 con 0x7fbe940c3970 2024-03-06T07:20:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.119+0000 7fbe99049640 1 -- 172.21.15.1:0/3435118742 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c030c10 con 0x7fbe940c3970 2024-03-06T07:20:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.119+0000 7fbe9c2af640 1 -- 172.21.15.1:0/3435118742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe940c3970 msgr2=0x7fbe940c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.119+0000 7fbe9c2af640 1 --2- 172.21.15.1:0/3435118742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe940c3970 0x7fbe940c8170 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c009870 tx=0x7fbe8c02b980 comp rx=0 tx=0).stop 2024-03-06T07:20:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.119+0000 7fbe9c2af640 1 -- 172.21.15.1:0/3435118742 shutdown_connections 2024-03-06T07:20:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.119+0000 7fbe9c2af640 1 --2- 172.21.15.1:0/3435118742 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe940c60b0 0x7fbe94152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.119+0000 7fbe9c2af640 1 --2- 172.21.15.1:0/3435118742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe940c3970 0x7fbe940c8170 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.119+0000 7fbe9c2af640 1 -- 172.21.15.1:0/3435118742 >> 172.21.15.1:0/3435118742 conn(0x7fbe940c88f0 msgr2=0x7fbe940c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.119+0000 7fbe9c2af640 1 -- 172.21.15.1:0/3435118742 shutdown_connections 2024-03-06T07:20:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.120+0000 7fbe9c2af640 1 -- 172.21.15.1:0/3435118742 wait complete. 2024-03-06T07:20:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.121+0000 7fbe9c2af640 1 Processor -- start 2024-03-06T07:20:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.121+0000 7fbe9c2af640 1 -- start start 2024-03-06T07:20:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.121+0000 7fbe9c2af640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe940c3970 0x7fbe9414a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.121+0000 7fbe9c2af640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe940c60b0 0x7fbe94070940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.121+0000 7fbe9c2af640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe94152670 0x7fbe9414be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.121+0000 7fbe9c2af640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe940c75c0 con 0x7fbe940c60b0 2024-03-06T07:20:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.121+0000 7fbe9c2af640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe940c72e0 con 0x7fbe940c3970 2024-03-06T07:20:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.121+0000 7fbe9c2af640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe940c3d50 con 0x7fbe94152670 2024-03-06T07:20:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.121+0000 7fbe9984a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe940c60b0 0x7fbe94070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.121+0000 7fbe9984a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe940c60b0 0x7fbe94070940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36814/0 (socket says 172.21.15.1:36814) 2024-03-06T07:20:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.121+0000 7fbe9984a640 1 -- 172.21.15.1:0/3731085478 learned_addr learned my addr 172.21.15.1:0/3731085478 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.122+0000 7fbe9a84c640 1 --2- 172.21.15.1:0/3731085478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe94152670 0x7fbe9414be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.122+0000 7fbe9a04b640 1 --2- 172.21.15.1:0/3731085478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe940c3970 0x7fbe9414a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.122+0000 7fbe9984a640 1 -- 172.21.15.1:0/3731085478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe94152670 msgr2=0x7fbe9414be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.122+0000 7fbe9984a640 1 --2- 172.21.15.1:0/3731085478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe94152670 0x7fbe9414be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.122+0000 7fbe9984a640 1 -- 172.21.15.1:0/3731085478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe940c3970 msgr2=0x7fbe9414a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.122+0000 7fbe9984a640 1 --2- 172.21.15.1:0/3731085478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe940c3970 0x7fbe9414a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.122+0000 7fbe9984a640 1 -- 172.21.15.1:0/3731085478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9414ac70 con 0x7fbe940c60b0 2024-03-06T07:20:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.122+0000 7fbe9984a640 1 --2- 172.21.15.1:0/3731085478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe940c60b0 0x7fbe94070940 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400d830 tx=0x7fbe8400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.122+0000 7fbe8affd640 1 -- 172.21.15.1:0/3731085478 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe84007e10 con 0x7fbe940c60b0 2024-03-06T07:20:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.122+0000 7fbe8affd640 1 -- 172.21.15.1:0/3731085478 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe84019070 con 0x7fbe940c60b0 2024-03-06T07:20:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.122+0000 7fbe9c2af640 1 -- 172.21.15.1:0/3731085478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9414a490 con 0x7fbe940c60b0 2024-03-06T07:20:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.122+0000 7fbe8affd640 1 -- 172.21.15.1:0/3731085478 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8400bb10 con 0x7fbe940c60b0 2024-03-06T07:20:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.122+0000 7fbe9c2af640 1 -- 172.21.15.1:0/3731085478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe941eeca0 con 0x7fbe940c60b0 2024-03-06T07:20:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.123+0000 7fbe9c2af640 1 -- 172.21.15.1:0/3731085478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe940686e0 con 0x7fbe940c60b0 2024-03-06T07:20:55.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.124+0000 7fbe8affd640 1 -- 172.21.15.1:0/3731085478 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe8400bcb0 con 0x7fbe940c60b0 2024-03-06T07:20:55.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.124+0000 7fbe8affd640 1 --2- 172.21.15.1:0/3731085478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe7806b4d0 0x7fbe7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.124+0000 7fbe8affd640 1 -- 172.21.15.1:0/3731085478 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbe8408a740 con 0x7fbe940c60b0 2024-03-06T07:20:55.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.124+0000 7fbe9a04b640 1 --2- 172.21.15.1:0/3731085478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe7806b4d0 0x7fbe7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.125+0000 7fbe9a04b640 1 --2- 172.21.15.1:0/3731085478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe7806b4d0 0x7fbe7806d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c036000 tx=0x7fbe8c03b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:55.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.128+0000 7fbe8affd640 1 -- 172.21.15.1:0/3731085478 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe84023020 con 0x7fbe940c60b0 2024-03-06T07:20:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.312+0000 7fbe9c2af640 1 -- 172.21.15.1:0/3731085478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbe940c31f0 con 0x7fbe940c60b0 2024-03-06T07:20:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.312+0000 7fbe8affd640 1 -- 172.21.15.1:0/3731085478 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbe9414ac70 con 0x7fbe940c60b0 2024-03-06T07:20:55.314 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:20:55.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.314+0000 7fbe9c2af640 1 -- 172.21.15.1:0/3731085478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe7806b4d0 msgr2=0x7fbe7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.314+0000 7fbe9c2af640 1 --2- 172.21.15.1:0/3731085478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe7806b4d0 0x7fbe7806d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c036000 tx=0x7fbe8c03b040 comp rx=0 tx=0).stop 2024-03-06T07:20:55.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.314+0000 7fbe9c2af640 1 -- 172.21.15.1:0/3731085478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe940c60b0 msgr2=0x7fbe94070940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.314+0000 7fbe9c2af640 1 --2- 172.21.15.1:0/3731085478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe940c60b0 0x7fbe94070940 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400d830 tx=0x7fbe8400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.314+0000 7fbe9c2af640 1 -- 172.21.15.1:0/3731085478 shutdown_connections 2024-03-06T07:20:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.314+0000 7fbe9c2af640 1 --2- 172.21.15.1:0/3731085478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe7806b4d0 0x7fbe7806d990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.314+0000 7fbe9c2af640 1 --2- 172.21.15.1:0/3731085478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe94152670 0x7fbe9414be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.314+0000 7fbe9c2af640 1 --2- 172.21.15.1:0/3731085478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe940c60b0 0x7fbe94070940 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.314+0000 7fbe9c2af640 1 --2- 172.21.15.1:0/3731085478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe940c3970 0x7fbe9414a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.314+0000 7fbe9c2af640 1 -- 172.21.15.1:0/3731085478 >> 172.21.15.1:0/3731085478 conn(0x7fbe940c88f0 msgr2=0x7fbe940aa680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.314+0000 7fbe9c2af640 1 -- 172.21.15.1:0/3731085478 shutdown_connections 2024-03-06T07:20:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.314+0000 7fbe9c2af640 1 -- 172.21.15.1:0/3731085478 wait complete. 2024-03-06T07:20:55.327 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:20:55.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.430+0000 7fed1764e640 1 Processor -- start 2024-03-06T07:20:55.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.430+0000 7fed1764e640 1 -- start start 2024-03-06T07:20:55.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.431+0000 7fed1764e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fed100c4bf0 0x7fed10156460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.431+0000 7fed1764e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fed101569a0 0x7fed1015d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.432+0000 7fed1764e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed10153a00 con 0x7fed100a7050 2024-03-06T07:20:55.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.432+0000 7fed1764e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed10153ce0 con 0x7fed100c4bf0 2024-03-06T07:20:55.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.432+0000 7fed1764e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed100c3cb0 con 0x7fed101569a0 2024-03-06T07:20:55.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.432+0000 7fed14be9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fed100c4bf0 0x7fed10156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.432+0000 7fed14be9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fed100c4bf0 0x7fed10156460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36820/0 (socket says 172.21.15.1:36820) 2024-03-06T07:20:55.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.432+0000 7fed14be9640 1 -- 172.21.15.1:0/2404559680 learned_addr learned my addr 172.21.15.1:0/2404559680 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:55.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.432+0000 7fed15beb640 1 --2- 172.21.15.1:0/2404559680 >> v2:172.21.15.98:3300/0 conn(0x7fed101569a0 0x7fed1015d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.432+0000 7fed14be9640 1 -- 172.21.15.1:0/2404559680 >> v2:172.21.15.98:3300/0 conn(0x7fed101569a0 msgr2=0x7fed1015d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:55.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.432+0000 7fed14be9640 1 --2- 172.21.15.1:0/2404559680 >> v2:172.21.15.98:3300/0 conn(0x7fed101569a0 0x7fed1015d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.432+0000 7fed14be9640 1 -- 172.21.15.1:0/2404559680 >> v1:172.21.15.98:6789/0 conn(0x7fed100a7050 legacy=0x7fed100c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.432+0000 7fed14be9640 1 -- 172.21.15.1:0/2404559680 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed100c3f90 con 0x7fed100c4bf0 2024-03-06T07:20:55.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.432+0000 7fed15beb640 1 --2- 172.21.15.1:0/2404559680 >> v2:172.21.15.98:3300/0 conn(0x7fed101569a0 0x7fed1015d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:55.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.433+0000 7fed14be9640 1 --2- 172.21.15.1:0/2404559680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed100c4bf0 0x7fed10156460 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fed10072850 tx=0x7fed0002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad8c5378f949cb2f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:55.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.433+0000 7fed07fff640 1 -- 172.21.15.1:0/2404559680 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0002e070 con 0x7fed100c4bf0 2024-03-06T07:20:55.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.433+0000 7fed07fff640 1 -- 172.21.15.1:0/2404559680 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed00037070 con 0x7fed100c4bf0 2024-03-06T07:20:55.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.433+0000 7fed07fff640 1 -- 172.21.15.1:0/2404559680 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00031a00 con 0x7fed100c4bf0 2024-03-06T07:20:55.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.433+0000 7fed1764e640 1 -- 172.21.15.1:0/2404559680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed100c4bf0 msgr2=0x7fed10156460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.433+0000 7fed1764e640 1 --2- 172.21.15.1:0/2404559680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed100c4bf0 0x7fed10156460 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fed10072850 tx=0x7fed0002b880 comp rx=0 tx=0).stop 2024-03-06T07:20:55.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.434+0000 7fed1764e640 1 -- 172.21.15.1:0/2404559680 shutdown_connections 2024-03-06T07:20:55.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.434+0000 7fed1764e640 1 --2- 172.21.15.1:0/2404559680 >> v2:172.21.15.98:3300/0 conn(0x7fed101569a0 0x7fed1015d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.434+0000 7fed1764e640 1 --2- 172.21.15.1:0/2404559680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed100c4bf0 0x7fed10156460 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.434+0000 7fed1764e640 1 -- 172.21.15.1:0/2404559680 >> 172.21.15.1:0/2404559680 conn(0x7fed100c7a50 msgr2=0x7fed100c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:55.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.434+0000 7fed1764e640 1 -- 172.21.15.1:0/2404559680 shutdown_connections 2024-03-06T07:20:55.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.434+0000 7fed1764e640 1 -- 172.21.15.1:0/2404559680 wait complete. 2024-03-06T07:20:55.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.435+0000 7fed1764e640 1 Processor -- start 2024-03-06T07:20:55.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.435+0000 7fed1764e640 1 -- start start 2024-03-06T07:20:55.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.436+0000 7fed1764e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed100a7050 0x7fed1014dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.436+0000 7fed1764e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed100c4bf0 0x7fed101ebad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.436+0000 7fed1764e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed101569a0 0x7fed101ec010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.436+0000 7fed1764e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed100c2450 con 0x7fed100a7050 2024-03-06T07:20:55.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.436+0000 7fed1764e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed10072f60 con 0x7fed101569a0 2024-03-06T07:20:55.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.436+0000 7fed1764e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed100c25d0 con 0x7fed100c4bf0 2024-03-06T07:20:55.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed153ea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed100a7050 0x7fed1014dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed153ea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed100a7050 0x7fed1014dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36836/0 (socket says 172.21.15.1:36836) 2024-03-06T07:20:55.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed153ea640 1 -- 172.21.15.1:0/2598356574 learned_addr learned my addr 172.21.15.1:0/2598356574 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:55.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed14be9640 1 --2- 172.21.15.1:0/2598356574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed100c4bf0 0x7fed101ebad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed15beb640 1 --2- 172.21.15.1:0/2598356574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed101569a0 0x7fed101ec010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed153ea640 1 -- 172.21.15.1:0/2598356574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed100c4bf0 msgr2=0x7fed101ebad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed153ea640 1 --2- 172.21.15.1:0/2598356574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed100c4bf0 0x7fed101ebad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed153ea640 1 -- 172.21.15.1:0/2598356574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed101569a0 msgr2=0x7fed101ec010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed153ea640 1 --2- 172.21.15.1:0/2598356574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed101569a0 0x7fed101ec010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed153ea640 1 -- 172.21.15.1:0/2598356574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed10151580 con 0x7fed100a7050 2024-03-06T07:20:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed15beb640 1 --2- 172.21.15.1:0/2598356574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed101569a0 0x7fed101ec010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed14be9640 1 --2- 172.21.15.1:0/2598356574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed100c4bf0 0x7fed101ebad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed153ea640 1 --2- 172.21.15.1:0/2598356574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed100a7050 0x7fed1014dc70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00c400 tx=0x7fed0c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed05ffb640 1 -- 172.21.15.1:0/2598356574 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c0052d0 con 0x7fed100a7050 2024-03-06T07:20:55.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed05ffb640 1 -- 172.21.15.1:0/2598356574 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0c005490 con 0x7fed100a7050 2024-03-06T07:20:55.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed05ffb640 1 -- 172.21.15.1:0/2598356574 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c007260 con 0x7fed100a7050 2024-03-06T07:20:55.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed1764e640 1 -- 172.21.15.1:0/2598356574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed101ec850 con 0x7fed100a7050 2024-03-06T07:20:55.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.437+0000 7fed1764e640 1 -- 172.21.15.1:0/2598356574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed10151b10 con 0x7fed100a7050 2024-03-06T07:20:55.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.438+0000 7fed1764e640 1 -- 172.21.15.1:0/2598356574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed100686e0 con 0x7fed100a7050 2024-03-06T07:20:55.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.439+0000 7fed05ffb640 1 -- 172.21.15.1:0/2598356574 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed0c007400 con 0x7fed100a7050 2024-03-06T07:20:55.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.439+0000 7fed05ffb640 1 --2- 172.21.15.1:0/2598356574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fecf406b4d0 0x7fecf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.440+0000 7fed05ffb640 1 -- 172.21.15.1:0/2598356574 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fed0c089a30 con 0x7fed100a7050 2024-03-06T07:20:55.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.440+0000 7fed14be9640 1 --2- 172.21.15.1:0/2598356574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fecf406b4d0 0x7fecf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.440+0000 7fed14be9640 1 --2- 172.21.15.1:0/2598356574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fecf406b4d0 0x7fecf406d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fed00036000 tx=0x7fed0003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:55.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.443+0000 7fed05ffb640 1 -- 172.21.15.1:0/2598356574 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed0c058600 con 0x7fed100a7050 2024-03-06T07:20:55.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.622+0000 7fed1764e640 1 -- 172.21.15.1:0/2598356574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fed10072b40 con 0x7fed100a7050 2024-03-06T07:20:55.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.623+0000 7fed05ffb640 1 -- 172.21.15.1:0/2598356574 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fed0c057f00 con 0x7fed100a7050 2024-03-06T07:20:55.625 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:20:55.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.625+0000 7fed1764e640 1 -- 172.21.15.1:0/2598356574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fecf406b4d0 msgr2=0x7fecf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.625+0000 7fed1764e640 1 --2- 172.21.15.1:0/2598356574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fecf406b4d0 0x7fecf406d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fed00036000 tx=0x7fed0003d040 comp rx=0 tx=0).stop 2024-03-06T07:20:55.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.625+0000 7fed1764e640 1 -- 172.21.15.1:0/2598356574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed100a7050 msgr2=0x7fed1014dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.625+0000 7fed1764e640 1 --2- 172.21.15.1:0/2598356574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed100a7050 0x7fed1014dc70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00c400 tx=0x7fed0c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.625+0000 7fed1764e640 1 -- 172.21.15.1:0/2598356574 shutdown_connections 2024-03-06T07:20:55.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.625+0000 7fed1764e640 1 --2- 172.21.15.1:0/2598356574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fecf406b4d0 0x7fecf406d990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.625+0000 7fed1764e640 1 --2- 172.21.15.1:0/2598356574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed101569a0 0x7fed101ec010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.625+0000 7fed1764e640 1 --2- 172.21.15.1:0/2598356574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed100c4bf0 0x7fed101ebad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.625+0000 7fed1764e640 1 --2- 172.21.15.1:0/2598356574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed100a7050 0x7fed1014dc70 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.625+0000 7fed1764e640 1 -- 172.21.15.1:0/2598356574 >> 172.21.15.1:0/2598356574 conn(0x7fed100c7a50 msgr2=0x7fed100c44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:55.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.625+0000 7fed1764e640 1 -- 172.21.15.1:0/2598356574 shutdown_connections 2024-03-06T07:20:55.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.625+0000 7fed1764e640 1 -- 172.21.15.1:0/2598356574 wait complete. 2024-03-06T07:20:55.638 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:55.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.737+0000 7f84f6563640 1 Processor -- start 2024-03-06T07:20:55.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.737+0000 7f84f6563640 1 -- start start 2024-03-06T07:20:55.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84f6563640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f84f0152840 0x7f84f00c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84f6563640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84f00c60b0 0x7f84f01553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84f6563640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f00c5e40 con 0x7f84f00c3970 2024-03-06T07:20:55.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84f6563640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84f00c67f0 con 0x7f84f0152840 2024-03-06T07:20:55.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84f6563640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f00c6a40 con 0x7f84f00c60b0 2024-03-06T07:20:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84effff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f84f00c3970 0x7f84f0152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36298/0 (socket says 172.21.15.1:36298) 2024-03-06T07:20:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84effff640 1 -- 172.21.15.1:0/3706396417 learned_addr learned my addr 172.21.15.1:0/3706396417 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84f4b00640 1 --2- 172.21.15.1:0/3706396417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84f00c60b0 0x7f84f01553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84e7fff640 1 --2- 172.21.15.1:0/3706396417 >> v2:172.21.15.98:3300/0 conn(0x7f84f0152840 0x7f84f00c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84ef7fe640 1 -- 172.21.15.1:0/3706396417 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317258452 0 0) 0x7f84f00c5e40 con 0x7f84f00c3970 2024-03-06T07:20:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84ef7fe640 1 -- 172.21.15.1:0/3706396417 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84dc003520 con 0x7f84f00c3970 2024-03-06T07:20:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84ef7fe640 1 -- 172.21.15.1:0/3706396417 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1836758809 0 0) 0x7f84dc003520 con 0x7f84f00c3970 2024-03-06T07:20:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84ef7fe640 1 -- 172.21.15.1:0/3706396417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84f00c60b0 msgr2=0x7f84f01553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84ef7fe640 1 --2- 172.21.15.1:0/3706396417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84f00c60b0 0x7f84f01553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84ef7fe640 1 -- 172.21.15.1:0/3706396417 >> v2:172.21.15.98:3300/0 conn(0x7f84f0152840 msgr2=0x7f84f00c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84ef7fe640 1 --2- 172.21.15.1:0/3706396417 >> v2:172.21.15.98:3300/0 conn(0x7f84f0152840 0x7f84f00c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84ef7fe640 1 -- 172.21.15.1:0/3706396417 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f00c6d20 con 0x7f84f00c3970 2024-03-06T07:20:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.738+0000 7f84ef7fe640 1 -- 172.21.15.1:0/3706396417 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f84e0001670 con 0x7f84f00c3970 2024-03-06T07:20:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.739+0000 7f84ef7fe640 1 -- 172.21.15.1:0/3706396417 >> v1:172.21.15.1:6789/0 conn(0x7f84f00c3970 legacy=0x7f84f0152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.739+0000 7f84ef7fe640 1 --2- 172.21.15.1:0/3706396417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84dc0039b0 0x7f84dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.739+0000 7f84ef7fe640 1 --2- 172.21.15.1:0/3706396417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84f00c60b0 0x7f84dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.739+0000 7f84ef7fe640 1 --2- 172.21.15.1:0/3706396417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84dc004d40 0x7f84dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.739+0000 7f84ef7fe640 1 -- 172.21.15.1:0/3706396417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f00c67f0 con 0x7f84dc004d40 2024-03-06T07:20:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.739+0000 7f84ef7fe640 1 -- 172.21.15.1:0/3706396417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f00c6a40 con 0x7f84dc0039b0 2024-03-06T07:20:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.739+0000 7f84ef7fe640 1 -- 172.21.15.1:0/3706396417 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc00b9c0 con 0x7f84f00c60b0 2024-03-06T07:20:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.739+0000 7f84f4b00640 1 --2- 172.21.15.1:0/3706396417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84dc004d40 0x7f84dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.739+0000 7f84effff640 1 --2- 172.21.15.1:0/3706396417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84dc0039b0 0x7f84dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.739+0000 7f84e7fff640 1 --2- 172.21.15.1:0/3706396417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84f00c60b0 0x7f84dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.739+0000 7f84f4b00640 1 -- 172.21.15.1:0/3706396417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84f00c60b0 msgr2=0x7f84dc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.739+0000 7f84f4b00640 1 --2- 172.21.15.1:0/3706396417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84f00c60b0 0x7f84dc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.739+0000 7f84f4b00640 1 -- 172.21.15.1:0/3706396417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84dc0039b0 msgr2=0x7f84dc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.739+0000 7f84f4b00640 1 --2- 172.21.15.1:0/3706396417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84dc0039b0 0x7f84dc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.739+0000 7f84f4b00640 1 -- 172.21.15.1:0/3706396417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc003520 con 0x7f84dc004d40 2024-03-06T07:20:55.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.740+0000 7f84f4b00640 1 --2- 172.21.15.1:0/3706396417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84dc004d40 0x7f84dc0051c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f84f006e2e0 tx=0x7f84dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=617d4e2f4d3c52d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:55.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.740+0000 7f84ef7fe640 1 -- 172.21.15.1:0/3706396417 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e8039070 con 0x7f84dc004d40 2024-03-06T07:20:55.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.740+0000 7f84ef7fe640 1 -- 172.21.15.1:0/3706396417 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e802cb90 con 0x7f84dc004d40 2024-03-06T07:20:55.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.740+0000 7f84ef7fe640 1 -- 172.21.15.1:0/3706396417 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e8009c10 con 0x7f84dc004d40 2024-03-06T07:20:55.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.741+0000 7f84f6563640 1 -- 172.21.15.1:0/3706396417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84dc004d40 msgr2=0x7f84dc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.741+0000 7f84f6563640 1 --2- 172.21.15.1:0/3706396417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84dc004d40 0x7f84dc0051c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f84f006e2e0 tx=0x7f84dc005830 comp rx=0 tx=0).stop 2024-03-06T07:20:55.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.741+0000 7f84f4b00640 1 -- 172.21.15.1:0/3706396417 reap_dead start 2024-03-06T07:20:55.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.741+0000 7f84f6563640 1 -- 172.21.15.1:0/3706396417 shutdown_connections 2024-03-06T07:20:55.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.741+0000 7f84f6563640 1 -- 172.21.15.1:0/3706396417 >> 172.21.15.1:0/3706396417 conn(0x7f84f00c88f0 msgr2=0x7f84f00c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:55.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.741+0000 7f84f6563640 1 -- 172.21.15.1:0/3706396417 shutdown_connections 2024-03-06T07:20:55.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.741+0000 7f84f6563640 1 -- 172.21.15.1:0/3706396417 wait complete. 2024-03-06T07:20:55.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.743+0000 7f84f6563640 1 Processor -- start 2024-03-06T07:20:55.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.743+0000 7f84f6563640 1 -- start start 2024-03-06T07:20:55.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.744+0000 7f84f6563640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84f01573f0 0x7f84f01567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.744+0000 7f84f6563640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84f01eba10 0x7f84f00a9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.744+0000 7f84f6563640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84f00ace10 0x7f84f013ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.744+0000 7f84f6563640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f00c75c0 con 0x7f84f01573f0 2024-03-06T07:20:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.744+0000 7f84f6563640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f00c72e0 con 0x7f84f00ace10 2024-03-06T07:20:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.744+0000 7f84f6563640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f00c3d50 con 0x7f84f01eba10 2024-03-06T07:20:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.744+0000 7f84effff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84f01573f0 0x7f84f01567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.744+0000 7f84effff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84f01573f0 0x7f84f01567a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36856/0 (socket says 172.21.15.1:36856) 2024-03-06T07:20:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.744+0000 7f84effff640 1 -- 172.21.15.1:0/1251952395 learned_addr learned my addr 172.21.15.1:0/1251952395 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.744+0000 7f84effff640 1 -- 172.21.15.1:0/1251952395 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84f01eba10 msgr2=0x7f84f00a9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.744+0000 7f84effff640 1 --2- 172.21.15.1:0/1251952395 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84f01eba10 0x7f84f00a9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.744+0000 7f84effff640 1 -- 172.21.15.1:0/1251952395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84f00ace10 msgr2=0x7f84f013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.744+0000 7f84effff640 1 --2- 172.21.15.1:0/1251952395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84f00ace10 0x7f84f013ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.744+0000 7f84effff640 1 -- 172.21.15.1:0/1251952395 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f00a9e60 con 0x7f84f01573f0 2024-03-06T07:20:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.744+0000 7f84effff640 1 --2- 172.21.15.1:0/1251952395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84f01573f0 0x7f84f01567a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f84e0007370 tx=0x7f84e0007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.744+0000 7f84ed7fa640 1 -- 172.21.15.1:0/1251952395 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e001a070 con 0x7f84f01573f0 2024-03-06T07:20:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.745+0000 7f84ed7fa640 1 -- 172.21.15.1:0/1251952395 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e000f940 con 0x7f84f01573f0 2024-03-06T07:20:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.745+0000 7f84f6563640 1 -- 172.21.15.1:0/1251952395 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f0155cc0 con 0x7f84f01573f0 2024-03-06T07:20:55.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.745+0000 7f84ed7fa640 1 -- 172.21.15.1:0/1251952395 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e000fd50 con 0x7f84f01573f0 2024-03-06T07:20:55.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.745+0000 7f84f6563640 1 -- 172.21.15.1:0/1251952395 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f0156280 con 0x7f84f01573f0 2024-03-06T07:20:55.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.746+0000 7f84f6563640 1 -- 172.21.15.1:0/1251952395 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f00686e0 con 0x7f84f01573f0 2024-03-06T07:20:55.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.746+0000 7f84ed7fa640 1 -- 172.21.15.1:0/1251952395 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84e001d070 con 0x7f84f01573f0 2024-03-06T07:20:55.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.746+0000 7f84ed7fa640 1 --2- 172.21.15.1:0/1251952395 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f84d006b500 0x7f84d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.747+0000 7f84ed7fa640 1 -- 172.21.15.1:0/1251952395 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f84e0089e80 con 0x7f84f01573f0 2024-03-06T07:20:55.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.747+0000 7f84e7fff640 1 --2- 172.21.15.1:0/1251952395 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f84d006b500 0x7f84d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.747+0000 7f84e7fff640 1 --2- 172.21.15.1:0/1251952395 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f84d006b500 0x7f84d006d9c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f84d8006e80 tx=0x7f84d8006df0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:55.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:55.751+0000 7f84ed7fa640 1 -- 172.21.15.1:0/1251952395 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84e0058a50 con 0x7f84f01573f0 2024-03-06T07:20:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.058+0000 7f84f6563640 1 -- 172.21.15.1:0/1251952395 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84f0156450 con 0x7f84f01573f0 2024-03-06T07:20:56.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.059+0000 7f84ed7fa640 1 -- 172.21.15.1:0/1251952395 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7f84e0058350 con 0x7f84f01573f0 2024-03-06T07:20:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:56.061 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:56.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.061+0000 7f84f6563640 1 -- 172.21.15.1:0/1251952395 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f84d006b500 msgr2=0x7f84d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.061+0000 7f84f6563640 1 --2- 172.21.15.1:0/1251952395 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f84d006b500 0x7f84d006d9c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f84d8006e80 tx=0x7f84d8006df0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.061+0000 7f84f6563640 1 -- 172.21.15.1:0/1251952395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84f01573f0 msgr2=0x7f84f01567a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.061+0000 7f84f6563640 1 --2- 172.21.15.1:0/1251952395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84f01573f0 0x7f84f01567a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f84e0007370 tx=0x7f84e0007830 comp rx=0 tx=0).stop 2024-03-06T07:20:56.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.061+0000 7f84f6563640 1 -- 172.21.15.1:0/1251952395 shutdown_connections 2024-03-06T07:20:56.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.061+0000 7f84f6563640 1 --2- 172.21.15.1:0/1251952395 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f84d006b500 0x7f84d006d9c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.061+0000 7f84f6563640 1 --2- 172.21.15.1:0/1251952395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84f00ace10 0x7f84f013ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.061+0000 7f84f6563640 1 --2- 172.21.15.1:0/1251952395 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84f01eba10 0x7f84f00a9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.061+0000 7f84f6563640 1 --2- 172.21.15.1:0/1251952395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84f01573f0 0x7f84f01567a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.061+0000 7f84f6563640 1 -- 172.21.15.1:0/1251952395 >> 172.21.15.1:0/1251952395 conn(0x7f84f00c88f0 msgr2=0x7f84f01546b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:56.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.061+0000 7f84f6563640 1 -- 172.21.15.1:0/1251952395 shutdown_connections 2024-03-06T07:20:56.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.061+0000 7f84f6563640 1 -- 172.21.15.1:0/1251952395 wait complete. 2024-03-06T07:20:56.074 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:20:19.337560+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:20:20.172791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:20:23.884071+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:20:56.074 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:20:56.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.176+0000 7f2377a78640 1 Processor -- start 2024-03-06T07:20:56.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.176+0000 7f2377a78640 1 -- start start 2024-03-06T07:20:56.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.177+0000 7f2377a78640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23701485c0 0x7f237006f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.177+0000 7f2377a78640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f23701489a0 0x7f237006c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.177+0000 7f2377a78640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f237006c640 0x7f23700aa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.177+0000 7f2377a78640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23700ac760 con 0x7f23701485c0 2024-03-06T07:20:56.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.177+0000 7f2377a78640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2370151f30 con 0x7f23701489a0 2024-03-06T07:20:56.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.177+0000 7f2377a78640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2370152210 con 0x7f237006c640 2024-03-06T07:20:56.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.177+0000 7f2375814640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23701485c0 0x7f237006f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.177+0000 7f2375814640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23701485c0 0x7f237006f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36864/0 (socket says 172.21.15.1:36864) 2024-03-06T07:20:56.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.177+0000 7f2375814640 1 -- 172.21.15.1:0/923779375 learned_addr learned my addr 172.21.15.1:0/923779375 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:56.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.177+0000 7f2376015640 1 --2- 172.21.15.1:0/923779375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f237006c640 0x7f23700aa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.177+0000 7f2375013640 1 --2- 172.21.15.1:0/923779375 >> v2:172.21.15.98:3300/0 conn(0x7f23701489a0 0x7f237006c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.178+0000 7f2375814640 1 -- 172.21.15.1:0/923779375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f237006c640 msgr2=0x7f23700aa3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:56.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.178+0000 7f2375814640 1 --2- 172.21.15.1:0/923779375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f237006c640 0x7f23700aa3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.178+0000 7f2375814640 1 -- 172.21.15.1:0/923779375 >> v2:172.21.15.98:3300/0 conn(0x7f23701489a0 msgr2=0x7f237006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:56.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.178+0000 7f2375814640 1 --2- 172.21.15.1:0/923779375 >> v2:172.21.15.98:3300/0 conn(0x7f23701489a0 0x7f237006c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.178+0000 7f2375814640 1 -- 172.21.15.1:0/923779375 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23701524f0 con 0x7f23701485c0 2024-03-06T07:20:56.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.178+0000 7f2375013640 1 --2- 172.21.15.1:0/923779375 >> v2:172.21.15.98:3300/0 conn(0x7f23701489a0 0x7f237006c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:56.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.179+0000 7f2375814640 1 --2- 172.21.15.1:0/923779375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701485c0 0x7f237006f780 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f237006eec0 tx=0x7f236402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c0e308c7ef9498 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.179+0000 7f2374812640 1 -- 172.21.15.1:0/923779375 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236402ca70 con 0x7f23701485c0 2024-03-06T07:20:56.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.179+0000 7f2374812640 1 -- 172.21.15.1:0/923779375 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236402cc10 con 0x7f23701485c0 2024-03-06T07:20:56.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.179+0000 7f2374812640 1 -- 172.21.15.1:0/923779375 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23640343f0 con 0x7f23701485c0 2024-03-06T07:20:56.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.179+0000 7f2377a78640 1 -- 172.21.15.1:0/923779375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701485c0 msgr2=0x7f237006f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.179+0000 7f2377a78640 1 --2- 172.21.15.1:0/923779375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701485c0 0x7f237006f780 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f237006eec0 tx=0x7f236402bed0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.180+0000 7f2377a78640 1 -- 172.21.15.1:0/923779375 shutdown_connections 2024-03-06T07:20:56.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.180+0000 7f2377a78640 1 --2- 172.21.15.1:0/923779375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f237006c640 0x7f23700aa3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.180+0000 7f2377a78640 1 --2- 172.21.15.1:0/923779375 >> v2:172.21.15.98:3300/0 conn(0x7f23701489a0 0x7f237006c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.180+0000 7f2377a78640 1 --2- 172.21.15.1:0/923779375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701485c0 0x7f237006f780 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.180+0000 7f2377a78640 1 -- 172.21.15.1:0/923779375 >> 172.21.15.1:0/923779375 conn(0x7f23700c3cc0 msgr2=0x7f23700c40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.180+0000 7f2377a78640 1 -- 172.21.15.1:0/923779375 shutdown_connections 2024-03-06T07:20:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.180+0000 7f2377a78640 1 -- 172.21.15.1:0/923779375 wait complete. 2024-03-06T07:20:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.181+0000 7f2377a78640 1 Processor -- start 2024-03-06T07:20:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.181+0000 7f2377a78640 1 -- start start 2024-03-06T07:20:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.181+0000 7f2377a78640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f237006c640 0x7f23700c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.181+0000 7f2377a78640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23701485c0 0x7f237014fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.181+0000 7f2377a78640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701489a0 0x7f23701500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.181+0000 7f2377a78640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237006dec0 con 0x7f23701489a0 2024-03-06T07:20:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.181+0000 7f2377a78640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237006dd40 con 0x7f23701485c0 2024-03-06T07:20:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.181+0000 7f2377a78640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237006e040 con 0x7f237006c640 2024-03-06T07:20:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f2376015640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701489a0 0x7f23701500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f2376015640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701489a0 0x7f23701500d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36868/0 (socket says 172.21.15.1:36868) 2024-03-06T07:20:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f2376015640 1 -- 172.21.15.1:0/3447960531 learned_addr learned my addr 172.21.15.1:0/3447960531 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f2375013640 1 --2- 172.21.15.1:0/3447960531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23701485c0 0x7f237014fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f2376015640 1 -- 172.21.15.1:0/3447960531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f237006c640 msgr2=0x7f23700c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f2375814640 1 --2- 172.21.15.1:0/3447960531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f237006c640 0x7f23700c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f2376015640 1 --2- 172.21.15.1:0/3447960531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f237006c640 0x7f23700c6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f2376015640 1 -- 172.21.15.1:0/3447960531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23701485c0 msgr2=0x7f237014fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f2376015640 1 --2- 172.21.15.1:0/3447960531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23701485c0 0x7f237014fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f2376015640 1 -- 172.21.15.1:0/3447960531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23701e6cd0 con 0x7f23701489a0 2024-03-06T07:20:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f2375013640 1 --2- 172.21.15.1:0/3447960531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23701485c0 0x7f237014fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f2375814640 1 --2- 172.21.15.1:0/3447960531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f237006c640 0x7f23700c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f2376015640 1 --2- 172.21.15.1:0/3447960531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701489a0 0x7f23701500d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f236800e820 tx=0x7f236800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f23627fc640 1 -- 172.21.15.1:0/3447960531 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2368017070 con 0x7f23701489a0 2024-03-06T07:20:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f23627fc640 1 -- 172.21.15.1:0/3447960531 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236800c730 con 0x7f23701489a0 2024-03-06T07:20:56.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f2377a78640 1 -- 172.21.15.1:0/3447960531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23701e6fb0 con 0x7f23701489a0 2024-03-06T07:20:56.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f23627fc640 1 -- 172.21.15.1:0/3447960531 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236800cb20 con 0x7f23701489a0 2024-03-06T07:20:56.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.182+0000 7f2377a78640 1 -- 172.21.15.1:0/3447960531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23701e7570 con 0x7f23701489a0 2024-03-06T07:20:56.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.183+0000 7f2377a78640 1 -- 172.21.15.1:0/3447960531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23700686e0 con 0x7f23701489a0 2024-03-06T07:20:56.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.184+0000 7f23627fc640 1 -- 172.21.15.1:0/3447960531 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f236801a070 con 0x7f23701489a0 2024-03-06T07:20:56.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.185+0000 7f23627fc640 1 --2- 172.21.15.1:0/3447960531 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f235406b500 0x7f235406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.185+0000 7f23627fc640 1 -- 172.21.15.1:0/3447960531 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f236808a6b0 con 0x7f23701489a0 2024-03-06T07:20:56.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.185+0000 7f2375814640 1 --2- 172.21.15.1:0/3447960531 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f235406b500 0x7f235406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.185+0000 7f2375814640 1 --2- 172.21.15.1:0/3447960531 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f235406b500 0x7f235406d9c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2364035000 tx=0x7f2364030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.189+0000 7f23627fc640 1 -- 172.21.15.1:0/3447960531 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23680591f0 con 0x7f23701489a0 2024-03-06T07:20:56.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.369+0000 7f2377a78640 1 -- 172.21.15.1:0/3447960531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f23701e7740 con 0x7f23701489a0 2024-03-06T07:20:56.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.370+0000 7f23627fc640 1 -- 172.21.15.1:0/3447960531 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2368058af0 con 0x7f23701489a0 2024-03-06T07:20:56.372 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:20:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.371+0000 7f2377a78640 1 -- 172.21.15.1:0/3447960531 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f235406b500 msgr2=0x7f235406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.371+0000 7f2377a78640 1 --2- 172.21.15.1:0/3447960531 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f235406b500 0x7f235406d9c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2364035000 tx=0x7f2364030000 comp rx=0 tx=0).stop 2024-03-06T07:20:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.371+0000 7f2377a78640 1 -- 172.21.15.1:0/3447960531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701489a0 msgr2=0x7f23701500d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.371+0000 7f2377a78640 1 --2- 172.21.15.1:0/3447960531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701489a0 0x7f23701500d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f236800e820 tx=0x7f236800ece0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.371+0000 7f2377a78640 1 -- 172.21.15.1:0/3447960531 shutdown_connections 2024-03-06T07:20:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.371+0000 7f2377a78640 1 --2- 172.21.15.1:0/3447960531 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f235406b500 0x7f235406d9c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.371+0000 7f2377a78640 1 --2- 172.21.15.1:0/3447960531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701489a0 0x7f23701500d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.371+0000 7f2377a78640 1 --2- 172.21.15.1:0/3447960531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23701485c0 0x7f237014fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.371+0000 7f2377a78640 1 --2- 172.21.15.1:0/3447960531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f237006c640 0x7f23700c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.371+0000 7f2377a78640 1 -- 172.21.15.1:0/3447960531 >> 172.21.15.1:0/3447960531 conn(0x7f23700c3cc0 msgr2=0x7f2370071d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:56.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.372+0000 7f2377a78640 1 -- 172.21.15.1:0/3447960531 shutdown_connections 2024-03-06T07:20:56.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.372+0000 7f2377a78640 1 -- 172.21.15.1:0/3447960531 wait complete. 2024-03-06T07:20:56.384 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:20:56.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.488+0000 7f9c761b5640 1 Processor -- start 2024-03-06T07:20:56.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.488+0000 7f9c761b5640 1 -- start start 2024-03-06T07:20:56.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.488+0000 7f9c761b5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c70152650 0x7f9c70155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.488+0000 7f9c761b5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c700c2fc0 con 0x7f9c700c6b10 2024-03-06T07:20:56.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.488+0000 7f9c761b5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c70158da0 con 0x7f9c700bf9d0 2024-03-06T07:20:56.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.488+0000 7f9c761b5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c70155800 con 0x7f9c70152650 2024-03-06T07:20:56.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.489+0000 7f9c6effd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9c700c6b10 0x7f9c70158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36310/0 (socket says 172.21.15.1:36310) 2024-03-06T07:20:56.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.489+0000 7f9c6effd640 1 -- 172.21.15.1:0/1993616860 learned_addr learned my addr 172.21.15.1:0/1993616860 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:56.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.489+0000 7f9c6f7fe640 1 --2- 172.21.15.1:0/1993616860 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c70152650 0x7f9c70155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.489+0000 7f9c6e7fc640 1 -- 172.21.15.1:0/1993616860 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089378102 0 0) 0x7f9c700c2fc0 con 0x7f9c700c6b10 2024-03-06T07:20:56.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.489+0000 7f9c6e7fc640 1 -- 172.21.15.1:0/1993616860 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c58003520 con 0x7f9c700c6b10 2024-03-06T07:20:56.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.489+0000 7f9c6f7fe640 1 -- 172.21.15.1:0/1993616860 >> v1:172.21.15.98:6789/0 conn(0x7f9c700bf9d0 legacy=0x7f9c700c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.489+0000 7f9c6f7fe640 1 -- 172.21.15.1:0/1993616860 >> v1:172.21.15.1:6789/0 conn(0x7f9c700c6b10 legacy=0x7f9c70158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.489+0000 7f9c6f7fe640 1 -- 172.21.15.1:0/1993616860 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c700c6690 con 0x7f9c70152650 2024-03-06T07:20:56.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.490+0000 7f9c6f7fe640 1 --2- 172.21.15.1:0/1993616860 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c70152650 0x7f9c70155c90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9c600097d0 tx=0x7f9c6002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9e7bf1fd3cd6b22 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.490+0000 7f9c6e7fc640 1 -- 172.21.15.1:0/1993616860 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c60035020 con 0x7f9c70152650 2024-03-06T07:20:56.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.490+0000 7f9c6e7fc640 1 -- 172.21.15.1:0/1993616860 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c60031800 con 0x7f9c70152650 2024-03-06T07:20:56.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.490+0000 7f9c6e7fc640 1 -- 172.21.15.1:0/1993616860 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c60031bb0 con 0x7f9c70152650 2024-03-06T07:20:56.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.491+0000 7f9c761b5640 1 -- 172.21.15.1:0/1993616860 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c70152650 msgr2=0x7f9c70155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.491+0000 7f9c761b5640 1 --2- 172.21.15.1:0/1993616860 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c70152650 0x7f9c70155c90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9c600097d0 tx=0x7f9c6002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.491+0000 7f9c761b5640 1 -- 172.21.15.1:0/1993616860 shutdown_connections 2024-03-06T07:20:56.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.491+0000 7f9c761b5640 1 --2- 172.21.15.1:0/1993616860 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c70152650 0x7f9c70155c90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.491+0000 7f9c761b5640 1 -- 172.21.15.1:0/1993616860 >> 172.21.15.1:0/1993616860 conn(0x7f9c70066ee0 msgr2=0x7f9c700672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:56.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.491+0000 7f9c761b5640 1 -- 172.21.15.1:0/1993616860 shutdown_connections 2024-03-06T07:20:56.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.491+0000 7f9c761b5640 1 -- 172.21.15.1:0/1993616860 wait complete. 2024-03-06T07:20:56.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.492+0000 7f9c761b5640 1 Processor -- start 2024-03-06T07:20:56.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.492+0000 7f9c761b5640 1 -- start start 2024-03-06T07:20:56.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.492+0000 7f9c761b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c700bf9d0 0x7f9c70154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.492+0000 7f9c761b5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c70152650 0x7f9c7014d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.492+0000 7f9c761b5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c70158750 0x7f9c7014fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.492+0000 7f9c761b5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c701551b0 con 0x7f9c70152650 2024-03-06T07:20:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.492+0000 7f9c761b5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c700c6fa0 con 0x7f9c700bf9d0 2024-03-06T07:20:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.492+0000 7f9c761b5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c70155490 con 0x7f9c70158750 2024-03-06T07:20:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.493+0000 7f9c6effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c70152650 0x7f9c7014d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.493+0000 7f9c6effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c70152650 0x7f9c7014d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36874/0 (socket says 172.21.15.1:36874) 2024-03-06T07:20:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.493+0000 7f9c6effd640 1 -- 172.21.15.1:0/1169333190 learned_addr learned my addr 172.21.15.1:0/1169333190 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.493+0000 7f9c6f7fe640 1 --2- 172.21.15.1:0/1169333190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c700bf9d0 0x7f9c70154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.493+0000 7f9c6ffff640 1 --2- 172.21.15.1:0/1169333190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c70158750 0x7f9c7014fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.493+0000 7f9c6effd640 1 -- 172.21.15.1:0/1169333190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c70158750 msgr2=0x7f9c7014fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.493+0000 7f9c6effd640 1 --2- 172.21.15.1:0/1169333190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c70158750 0x7f9c7014fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.493+0000 7f9c6effd640 1 -- 172.21.15.1:0/1169333190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c700bf9d0 msgr2=0x7f9c70154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.493+0000 7f9c6effd640 1 --2- 172.21.15.1:0/1169333190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c700bf9d0 0x7f9c70154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.493+0000 7f9c6effd640 1 -- 172.21.15.1:0/1169333190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7014c280 con 0x7f9c70152650 2024-03-06T07:20:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.493+0000 7f9c6effd640 1 --2- 172.21.15.1:0/1169333190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c70152650 0x7f9c7014d600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9c6400c470 tx=0x7f9c6400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.493+0000 7f9c57fff640 1 -- 172.21.15.1:0/1169333190 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c64017070 con 0x7f9c70152650 2024-03-06T07:20:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.493+0000 7f9c761b5640 1 -- 172.21.15.1:0/1169333190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c7014c560 con 0x7f9c70152650 2024-03-06T07:20:56.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.493+0000 7f9c57fff640 1 -- 172.21.15.1:0/1169333190 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c64005ce0 con 0x7f9c70152650 2024-03-06T07:20:56.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.493+0000 7f9c57fff640 1 -- 172.21.15.1:0/1169333190 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c640072f0 con 0x7f9c70152650 2024-03-06T07:20:56.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.493+0000 7f9c761b5640 1 -- 172.21.15.1:0/1169333190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7014cb20 con 0x7f9c70152650 2024-03-06T07:20:56.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.495+0000 7f9c761b5640 1 -- 172.21.15.1:0/1169333190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c70067fd0 con 0x7f9c70152650 2024-03-06T07:20:56.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.495+0000 7f9c57fff640 1 -- 172.21.15.1:0/1169333190 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c6401a070 con 0x7f9c70152650 2024-03-06T07:20:56.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.496+0000 7f9c57fff640 1 --2- 172.21.15.1:0/1169333190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c5c06b500 0x7f9c5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.496+0000 7f9c57fff640 1 -- 172.21.15.1:0/1169333190 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9c64089880 con 0x7f9c70152650 2024-03-06T07:20:56.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.496+0000 7f9c6f7fe640 1 --2- 172.21.15.1:0/1169333190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c5c06b500 0x7f9c5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.497+0000 7f9c6f7fe640 1 --2- 172.21.15.1:0/1169333190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c5c06b500 0x7f9c5c06d9c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9c60036000 tx=0x7f9c6003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.500+0000 7f9c57fff640 1 -- 172.21.15.1:0/1169333190 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c64057e40 con 0x7f9c70152650 2024-03-06T07:20:56.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.680+0000 7f9c761b5640 1 -- 172.21.15.1:0/1169333190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9c700b06f0 con 0x7f9c70152650 2024-03-06T07:20:56.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.681+0000 7f9c57fff640 1 -- 172.21.15.1:0/1169333190 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9c6405bee0 con 0x7f9c70152650 2024-03-06T07:20:56.683 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:20:56.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.683+0000 7f9c761b5640 1 -- 172.21.15.1:0/1169333190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c5c06b500 msgr2=0x7f9c5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.683+0000 7f9c761b5640 1 --2- 172.21.15.1:0/1169333190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c5c06b500 0x7f9c5c06d9c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9c60036000 tx=0x7f9c6003d040 comp rx=0 tx=0).stop 2024-03-06T07:20:56.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.683+0000 7f9c761b5640 1 -- 172.21.15.1:0/1169333190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c70152650 msgr2=0x7f9c7014d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.683+0000 7f9c761b5640 1 --2- 172.21.15.1:0/1169333190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c70152650 0x7f9c7014d600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9c6400c470 tx=0x7f9c6400c930 comp rx=0 tx=0).stop 2024-03-06T07:20:56.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.683+0000 7f9c761b5640 1 -- 172.21.15.1:0/1169333190 shutdown_connections 2024-03-06T07:20:56.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.683+0000 7f9c761b5640 1 --2- 172.21.15.1:0/1169333190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c5c06b500 0x7f9c5c06d9c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.683+0000 7f9c761b5640 1 --2- 172.21.15.1:0/1169333190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c70158750 0x7f9c7014fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.683+0000 7f9c761b5640 1 --2- 172.21.15.1:0/1169333190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c70152650 0x7f9c7014d600 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.683+0000 7f9c761b5640 1 --2- 172.21.15.1:0/1169333190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c700bf9d0 0x7f9c70154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.683+0000 7f9c761b5640 1 -- 172.21.15.1:0/1169333190 >> 172.21.15.1:0/1169333190 conn(0x7f9c70066ee0 msgr2=0x7f9c700cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:56.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.683+0000 7f9c761b5640 1 -- 172.21.15.1:0/1169333190 shutdown_connections 2024-03-06T07:20:56.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.683+0000 7f9c761b5640 1 -- 172.21.15.1:0/1169333190 wait complete. 2024-03-06T07:20:56.696 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:20:56.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.801+0000 7fd88f229640 1 Processor -- start 2024-03-06T07:20:56.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.801+0000 7fd88f229640 1 -- start start 2024-03-06T07:20:56.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.803+0000 7fd88f229640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd888152820 0x7fd8880c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.803+0000 7fd88f229640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8880c7180 con 0x7fd8880af6f0 2024-03-06T07:20:56.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.803+0000 7fd88f229640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd888155460 con 0x7fd8880c0670 2024-03-06T07:20:56.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.803+0000 7fd88f229640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8880c6bc0 con 0x7fd888152820 2024-03-06T07:20:56.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.803+0000 7fd88cfc5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd888152820 0x7fd8880c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.803+0000 7fd88d7c6640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd8880af6f0 0x7fd88815fb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36320/0 (socket says 172.21.15.1:36320) 2024-03-06T07:20:56.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.803+0000 7fd88d7c6640 1 -- 172.21.15.1:0/1258237067 learned_addr learned my addr 172.21.15.1:0/1258237067 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:56.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.803+0000 7fd88cfc5640 1 -- 172.21.15.1:0/1258237067 >> v1:172.21.15.98:6789/0 conn(0x7fd8880c0670 legacy=0x7fd8880c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:56.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.803+0000 7fd88cfc5640 1 -- 172.21.15.1:0/1258237067 >> v1:172.21.15.1:6789/0 conn(0x7fd8880af6f0 legacy=0x7fd88815fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.803+0000 7fd8877fe640 1 -- 172.21.15.1:0/1258237067 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681683952 0 0) 0x7fd8880c7180 con 0x7fd8880af6f0 2024-03-06T07:20:56.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.803+0000 7fd88cfc5640 1 -- 172.21.15.1:0/1258237067 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8880c6ea0 con 0x7fd888152820 2024-03-06T07:20:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.804+0000 7fd88cfc5640 1 --2- 172.21.15.1:0/1258237067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd888152820 0x7fd8880c0130 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0097a0 tx=0x7fd87c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9abc871f33e08c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.804+0000 7fd8877fe640 1 -- 172.21.15.1:0/1258237067 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c035020 con 0x7fd888152820 2024-03-06T07:20:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.804+0000 7fd8877fe640 1 -- 172.21.15.1:0/1258237067 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87c031830 con 0x7fd888152820 2024-03-06T07:20:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.804+0000 7fd8877fe640 1 -- 172.21.15.1:0/1258237067 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c031c00 con 0x7fd888152820 2024-03-06T07:20:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.805+0000 7fd88f229640 1 -- 172.21.15.1:0/1258237067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd888152820 msgr2=0x7fd8880c0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.805+0000 7fd88f229640 1 --2- 172.21.15.1:0/1258237067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd888152820 0x7fd8880c0130 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0097a0 tx=0x7fd87c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.805+0000 7fd88f229640 1 -- 172.21.15.1:0/1258237067 shutdown_connections 2024-03-06T07:20:56.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.805+0000 7fd88f229640 1 --2- 172.21.15.1:0/1258237067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd888152820 0x7fd8880c0130 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.805+0000 7fd88f229640 1 -- 172.21.15.1:0/1258237067 >> 172.21.15.1:0/1258237067 conn(0x7fd888072720 msgr2=0x7fd888072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:56.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.805+0000 7fd88f229640 1 -- 172.21.15.1:0/1258237067 shutdown_connections 2024-03-06T07:20:56.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.805+0000 7fd88f229640 1 -- 172.21.15.1:0/1258237067 wait complete. 2024-03-06T07:20:56.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.806+0000 7fd88f229640 1 Processor -- start 2024-03-06T07:20:56.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.806+0000 7fd88f229640 1 -- start start 2024-03-06T07:20:56.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd88f229640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8880af6f0 0x7fd88814ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd88f229640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8880c0670 0x7fd88814a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd88f229640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd888152820 0x7fd88814ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd88f229640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8880c7c80 con 0x7fd8880af6f0 2024-03-06T07:20:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd88f229640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8880c7b00 con 0x7fd888152820 2024-03-06T07:20:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd88f229640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd888152c00 con 0x7fd8880c0670 2024-03-06T07:20:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd88cfc5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8880af6f0 0x7fd88814ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd88cfc5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8880af6f0 0x7fd88814ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36898/0 (socket says 172.21.15.1:36898) 2024-03-06T07:20:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd88cfc5640 1 -- 172.21.15.1:0/3929452917 learned_addr learned my addr 172.21.15.1:0/3929452917 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd88cfc5640 1 -- 172.21.15.1:0/3929452917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8880c0670 msgr2=0x7fd88814a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd88cfc5640 1 --2- 172.21.15.1:0/3929452917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8880c0670 0x7fd88814a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd88cfc5640 1 -- 172.21.15.1:0/3929452917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd888152820 msgr2=0x7fd88814ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd88cfc5640 1 --2- 172.21.15.1:0/3929452917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd888152820 0x7fd88814ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd88cfc5640 1 -- 172.21.15.1:0/3929452917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8880c2e10 con 0x7fd8880af6f0 2024-03-06T07:20:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd88cfc5640 1 --2- 172.21.15.1:0/3929452917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8880af6f0 0x7fd88814ca10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd87c034040 tx=0x7fd87c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd8857fa640 1 -- 172.21.15.1:0/3929452917 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c004230 con 0x7fd8880af6f0 2024-03-06T07:20:56.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd8857fa640 1 -- 172.21.15.1:0/3929452917 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87c0043f0 con 0x7fd8880af6f0 2024-03-06T07:20:56.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd88f229640 1 -- 172.21.15.1:0/3929452917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8881edf00 con 0x7fd8880af6f0 2024-03-06T07:20:56.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd8857fa640 1 -- 172.21.15.1:0/3929452917 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c003850 con 0x7fd8880af6f0 2024-03-06T07:20:56.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.808+0000 7fd88f229640 1 -- 172.21.15.1:0/3929452917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8881ee4c0 con 0x7fd8880af6f0 2024-03-06T07:20:56.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.809+0000 7fd88f229640 1 -- 172.21.15.1:0/3929452917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8880686e0 con 0x7fd8880af6f0 2024-03-06T07:20:56.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.810+0000 7fd8857fa640 1 -- 172.21.15.1:0/3929452917 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd87c038020 con 0x7fd8880af6f0 2024-03-06T07:20:56.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.811+0000 7fd8857fa640 1 --2- 172.21.15.1:0/3929452917 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd86c06b4d0 0x7fd86c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.811+0000 7fd8857fa640 1 -- 172.21.15.1:0/3929452917 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd87c0ac000 con 0x7fd8880af6f0 2024-03-06T07:20:56.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.811+0000 7fd887fff640 1 --2- 172.21.15.1:0/3929452917 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd86c06b4d0 0x7fd86c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.811+0000 7fd887fff640 1 --2- 172.21.15.1:0/3929452917 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd86c06b4d0 0x7fd86c06d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd888151470 tx=0x7fd8740079b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.815+0000 7fd8857fa640 1 -- 172.21.15.1:0/3929452917 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd87c040070 con 0x7fd8880af6f0 2024-03-06T07:20:56.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.997+0000 7fd88f229640 1 -- 172.21.15.1:0/3929452917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd8880c5b40 con 0x7fd8880af6f0 2024-03-06T07:20:56.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.997+0000 7fd8857fa640 1 -- 172.21.15.1:0/3929452917 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd87c07acf0 con 0x7fd8880af6f0 2024-03-06T07:20:56.999 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:20:57.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.999+0000 7fd88f229640 1 -- 172.21.15.1:0/3929452917 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd86c06b4d0 msgr2=0x7fd86c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.999+0000 7fd88f229640 1 --2- 172.21.15.1:0/3929452917 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd86c06b4d0 0x7fd86c06d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd888151470 tx=0x7fd8740079b0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.999+0000 7fd88f229640 1 -- 172.21.15.1:0/3929452917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8880af6f0 msgr2=0x7fd88814ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.999+0000 7fd88f229640 1 --2- 172.21.15.1:0/3929452917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8880af6f0 0x7fd88814ca10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd87c034040 tx=0x7fd87c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.999+0000 7fd88f229640 1 -- 172.21.15.1:0/3929452917 shutdown_connections 2024-03-06T07:20:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.999+0000 7fd88f229640 1 --2- 172.21.15.1:0/3929452917 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd86c06b4d0 0x7fd86c06d990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.999+0000 7fd88f229640 1 --2- 172.21.15.1:0/3929452917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd888152820 0x7fd88814ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.999+0000 7fd88f229640 1 --2- 172.21.15.1:0/3929452917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8880c0670 0x7fd88814a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.999+0000 7fd88f229640 1 --2- 172.21.15.1:0/3929452917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8880af6f0 0x7fd88814ca10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.999+0000 7fd88f229640 1 -- 172.21.15.1:0/3929452917 >> 172.21.15.1:0/3929452917 conn(0x7fd888072720 msgr2=0x7fd888074450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.999+0000 7fd88f229640 1 -- 172.21.15.1:0/3929452917 shutdown_connections 2024-03-06T07:20:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:56.999+0000 7fd88f229640 1 -- 172.21.15.1:0/3929452917 wait complete. 2024-03-06T07:20:57.012 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:20:57.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.116+0000 7f6367dbd640 1 Processor -- start 2024-03-06T07:20:57.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.116+0000 7f6367dbd640 1 -- start start 2024-03-06T07:20:57.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.116+0000 7f6367dbd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63600c3970 0x7f63600c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.117+0000 7f6367dbd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f63600c60b0 0x7f6360152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.117+0000 7f6367dbd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6360152670 0x7f63601553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.117+0000 7f6367dbd640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63600c67f0 con 0x7f63600c3970 2024-03-06T07:20:57.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.117+0000 7f6367dbd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63600c5e40 con 0x7f63600c60b0 2024-03-06T07:20:57.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.117+0000 7f6367dbd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63600c6a40 con 0x7f6360152670 2024-03-06T07:20:57.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.117+0000 7f6365b59640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63600c3970 0x7f63600c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.117+0000 7f6365b59640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63600c3970 0x7f63600c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36904/0 (socket says 172.21.15.1:36904) 2024-03-06T07:20:57.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.117+0000 7f6365b59640 1 -- 172.21.15.1:0/2043843587 learned_addr learned my addr 172.21.15.1:0/2043843587 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:57.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.117+0000 7f6365b59640 1 -- 172.21.15.1:0/2043843587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6360152670 msgr2=0x7f63601553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:57.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.117+0000 7f6365b59640 1 --2- 172.21.15.1:0/2043843587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6360152670 0x7f63601553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.117+0000 7f6365b59640 1 -- 172.21.15.1:0/2043843587 >> v2:172.21.15.98:3300/0 conn(0x7f63600c60b0 msgr2=0x7f6360152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:57.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.117+0000 7f6365b59640 1 --2- 172.21.15.1:0/2043843587 >> v2:172.21.15.98:3300/0 conn(0x7f63600c60b0 0x7f6360152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.117+0000 7f6365b59640 1 -- 172.21.15.1:0/2043843587 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63600c6d20 con 0x7f63600c3970 2024-03-06T07:20:57.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.118+0000 7f6365b59640 1 --2- 172.21.15.1:0/2043843587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63600c3970 0x7f63600c8170 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f63600719c0 tx=0x7f635802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae4f5392fad156d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.118+0000 7f6364b57640 1 -- 172.21.15.1:0/2043843587 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6358035020 con 0x7f63600c3970 2024-03-06T07:20:57.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.118+0000 7f6364b57640 1 -- 172.21.15.1:0/2043843587 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6358031890 con 0x7f63600c3970 2024-03-06T07:20:57.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.118+0000 7f6364b57640 1 -- 172.21.15.1:0/2043843587 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6358031c60 con 0x7f63600c3970 2024-03-06T07:20:57.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.118+0000 7f6367dbd640 1 -- 172.21.15.1:0/2043843587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63600c3970 msgr2=0x7f63600c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.118+0000 7f6367dbd640 1 --2- 172.21.15.1:0/2043843587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63600c3970 0x7f63600c8170 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f63600719c0 tx=0x7f635802b930 comp rx=0 tx=0).stop 2024-03-06T07:20:57.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.118+0000 7f6367dbd640 1 -- 172.21.15.1:0/2043843587 shutdown_connections 2024-03-06T07:20:57.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.118+0000 7f6367dbd640 1 --2- 172.21.15.1:0/2043843587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6360152670 0x7f63601553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.118+0000 7f6367dbd640 1 --2- 172.21.15.1:0/2043843587 >> v2:172.21.15.98:3300/0 conn(0x7f63600c60b0 0x7f6360152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.118+0000 7f6367dbd640 1 --2- 172.21.15.1:0/2043843587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63600c3970 0x7f63600c8170 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.118+0000 7f6367dbd640 1 -- 172.21.15.1:0/2043843587 >> 172.21.15.1:0/2043843587 conn(0x7f63600c88f0 msgr2=0x7f63600c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:57.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.119+0000 7f6367dbd640 1 -- 172.21.15.1:0/2043843587 shutdown_connections 2024-03-06T07:20:57.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.119+0000 7f6367dbd640 1 -- 172.21.15.1:0/2043843587 wait complete. 2024-03-06T07:20:57.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.120+0000 7f6367dbd640 1 Processor -- start 2024-03-06T07:20:57.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.120+0000 7f6367dbd640 1 -- start start 2024-03-06T07:20:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.120+0000 7f6367dbd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63600c3970 0x7f63600707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.120+0000 7f6367dbd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63600c60b0 0x7f6360157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.120+0000 7f6367dbd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6360152670 0x7f63601edfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.120+0000 7f6367dbd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63600c75c0 con 0x7f6360152670 2024-03-06T07:20:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.120+0000 7f6367dbd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63600c72e0 con 0x7f63600c60b0 2024-03-06T07:20:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.120+0000 7f6367dbd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63600c3d50 con 0x7f63600c3970 2024-03-06T07:20:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.120+0000 7f636635a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6360152670 0x7f63601edfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.121+0000 7f636635a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6360152670 0x7f63601edfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36920/0 (socket says 172.21.15.1:36920) 2024-03-06T07:20:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.121+0000 7f636635a640 1 -- 172.21.15.1:0/3827138347 learned_addr learned my addr 172.21.15.1:0/3827138347 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.121+0000 7f6365b59640 1 --2- 172.21.15.1:0/3827138347 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63600c3970 0x7f63600707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.121+0000 7f6365358640 1 --2- 172.21.15.1:0/3827138347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63600c60b0 0x7f6360157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.121+0000 7f636635a640 1 -- 172.21.15.1:0/3827138347 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63600c3970 msgr2=0x7f63600707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.121+0000 7f636635a640 1 --2- 172.21.15.1:0/3827138347 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63600c3970 0x7f63600707a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.121+0000 7f636635a640 1 -- 172.21.15.1:0/3827138347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63600c60b0 msgr2=0x7f6360157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.121+0000 7f636635a640 1 --2- 172.21.15.1:0/3827138347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63600c60b0 0x7f6360157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.121+0000 7f636635a640 1 -- 172.21.15.1:0/3827138347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63601561f0 con 0x7f6360152670 2024-03-06T07:20:57.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.121+0000 7f636635a640 1 --2- 172.21.15.1:0/3827138347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6360152670 0x7f63601edfa0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f635c00b970 tx=0x7f635c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.121+0000 7f63567fc640 1 -- 172.21.15.1:0/3827138347 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c019070 con 0x7f6360152670 2024-03-06T07:20:57.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.121+0000 7f63567fc640 1 -- 172.21.15.1:0/3827138347 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635c00cac0 con 0x7f6360152670 2024-03-06T07:20:57.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.121+0000 7f63567fc640 1 -- 172.21.15.1:0/3827138347 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c014440 con 0x7f6360152670 2024-03-06T07:20:57.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.121+0000 7f6367dbd640 1 -- 172.21.15.1:0/3827138347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63601564d0 con 0x7f6360152670 2024-03-06T07:20:57.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.121+0000 7f6367dbd640 1 -- 172.21.15.1:0/3827138347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6360156a90 con 0x7f6360152670 2024-03-06T07:20:57.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.122+0000 7f6367dbd640 1 -- 172.21.15.1:0/3827138347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63600686e0 con 0x7f6360152670 2024-03-06T07:20:57.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.123+0000 7f63567fc640 1 -- 172.21.15.1:0/3827138347 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f635c01c070 con 0x7f6360152670 2024-03-06T07:20:57.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.123+0000 7f63567fc640 1 --2- 172.21.15.1:0/3827138347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f634406b4d0 0x7f634406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.123+0000 7f63567fc640 1 -- 172.21.15.1:0/3827138347 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f635c08a080 con 0x7f6360152670 2024-03-06T07:20:57.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.123+0000 7f6365b59640 1 --2- 172.21.15.1:0/3827138347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f634406b4d0 0x7f634406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.124+0000 7f6365b59640 1 --2- 172.21.15.1:0/3827138347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f634406b4d0 0x7f634406d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6358036000 tx=0x7f635803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.127+0000 7f63567fc640 1 -- 172.21.15.1:0/3827138347 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f635c058600 con 0x7f6360152670 2024-03-06T07:20:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.310+0000 7f6367dbd640 1 -- 172.21.15.1:0/3827138347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6360155570 con 0x7f6360152670 2024-03-06T07:20:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.310+0000 7f63567fc640 1 -- 172.21.15.1:0/3827138347 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f635c05c6a0 con 0x7f6360152670 2024-03-06T07:20:57.312 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:20:57.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.312+0000 7f6367dbd640 1 -- 172.21.15.1:0/3827138347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f634406b4d0 msgr2=0x7f634406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.312+0000 7f6367dbd640 1 --2- 172.21.15.1:0/3827138347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f634406b4d0 0x7f634406d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6358036000 tx=0x7f635803c040 comp rx=0 tx=0).stop 2024-03-06T07:20:57.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.312+0000 7f6367dbd640 1 -- 172.21.15.1:0/3827138347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6360152670 msgr2=0x7f63601edfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.312+0000 7f6367dbd640 1 --2- 172.21.15.1:0/3827138347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6360152670 0x7f63601edfa0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f635c00b970 tx=0x7f635c00bd70 comp rx=0 tx=0).stop 2024-03-06T07:20:57.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.312+0000 7f6367dbd640 1 -- 172.21.15.1:0/3827138347 shutdown_connections 2024-03-06T07:20:57.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.312+0000 7f6367dbd640 1 --2- 172.21.15.1:0/3827138347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f634406b4d0 0x7f634406d990 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.312+0000 7f6367dbd640 1 --2- 172.21.15.1:0/3827138347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6360152670 0x7f63601edfa0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.312+0000 7f6367dbd640 1 --2- 172.21.15.1:0/3827138347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63600c60b0 0x7f6360157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.312+0000 7f6367dbd640 1 --2- 172.21.15.1:0/3827138347 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63600c3970 0x7f63600707a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.312+0000 7f6367dbd640 1 -- 172.21.15.1:0/3827138347 >> 172.21.15.1:0/3827138347 conn(0x7f63600c88f0 msgr2=0x7f636006c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:57.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.312+0000 7f6367dbd640 1 -- 172.21.15.1:0/3827138347 shutdown_connections 2024-03-06T07:20:57.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.312+0000 7f6367dbd640 1 -- 172.21.15.1:0/3827138347 wait complete. 2024-03-06T07:20:57.328 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:57.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.429+0000 7f944618a640 1 Processor -- start 2024-03-06T07:20:57.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.429+0000 7f944618a640 1 -- start start 2024-03-06T07:20:57.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.429+0000 7f944618a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94400c2b30 0x7f94400724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.429+0000 7f944618a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9440151dd0 con 0x7f94400a5fa0 2024-03-06T07:20:57.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.429+0000 7f944618a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9440156190 con 0x7f9440152b70 2024-03-06T07:20:57.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.429+0000 7f944618a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94401526e0 con 0x7f94400c2b30 2024-03-06T07:20:57.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.430+0000 7f943f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94400c2b30 0x7f94400724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.430+0000 7f943ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f94400a5fa0 0x7f944015c750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36322/0 (socket says 172.21.15.1:36322) 2024-03-06T07:20:57.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.430+0000 7f943ffff640 1 -- 172.21.15.1:0/945202802 learned_addr learned my addr 172.21.15.1:0/945202802 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:57.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.430+0000 7f943f7fe640 1 -- 172.21.15.1:0/945202802 >> v1:172.21.15.98:6789/0 conn(0x7f9440152b70 legacy=0x7f94400c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:57.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.430+0000 7f943f7fe640 1 -- 172.21.15.1:0/945202802 >> v1:172.21.15.1:6789/0 conn(0x7f94400a5fa0 legacy=0x7f944015c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.430+0000 7f943f7fe640 1 -- 172.21.15.1:0/945202802 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9440152090 con 0x7f94400c2b30 2024-03-06T07:20:57.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.430+0000 7f943e7fc640 1 -- 172.21.15.1:0/945202802 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857370280 0 0) 0x7f9440151dd0 con 0x7f94400a5fa0 2024-03-06T07:20:57.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.431+0000 7f943f7fe640 1 --2- 172.21.15.1:0/945202802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94400c2b30 0x7f94400724c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f94300097a0 tx=0x7f943002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7275b9c2d4183905 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.431+0000 7f943e7fc640 1 -- 172.21.15.1:0/945202802 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430035020 con 0x7f94400c2b30 2024-03-06T07:20:57.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.431+0000 7f943e7fc640 1 -- 172.21.15.1:0/945202802 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9430031830 con 0x7f94400c2b30 2024-03-06T07:20:57.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.431+0000 7f943e7fc640 1 -- 172.21.15.1:0/945202802 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430031c00 con 0x7f94400c2b30 2024-03-06T07:20:57.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.431+0000 7f944618a640 1 -- 172.21.15.1:0/945202802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94400c2b30 msgr2=0x7f94400724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.431+0000 7f944618a640 1 --2- 172.21.15.1:0/945202802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94400c2b30 0x7f94400724c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f94300097a0 tx=0x7f943002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.431+0000 7f944618a640 1 -- 172.21.15.1:0/945202802 shutdown_connections 2024-03-06T07:20:57.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.431+0000 7f944618a640 1 --2- 172.21.15.1:0/945202802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94400c2b30 0x7f94400724c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.431+0000 7f944618a640 1 -- 172.21.15.1:0/945202802 >> 172.21.15.1:0/945202802 conn(0x7f94400c6860 msgr2=0x7f94400c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:57.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.431+0000 7f944618a640 1 -- 172.21.15.1:0/945202802 shutdown_connections 2024-03-06T07:20:57.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.431+0000 7f944618a640 1 -- 172.21.15.1:0/945202802 wait complete. 2024-03-06T07:20:57.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.432+0000 7f944618a640 1 Processor -- start 2024-03-06T07:20:57.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.433+0000 7f944618a640 1 -- start start 2024-03-06T07:20:57.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f944618a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94400a5fa0 0x7f944014c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f944618a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94400c2b30 0x7f944014a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f944618a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9440152b70 0x7f944014af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f944618a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9440155f00 con 0x7f94400c2b30 2024-03-06T07:20:57.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f944618a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9440155d80 con 0x7f94400a5fa0 2024-03-06T07:20:57.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f944618a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9440072a00 con 0x7f9440152b70 2024-03-06T07:20:57.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f943effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94400c2b30 0x7f944014a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f943effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94400c2b30 0x7f944014a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36940/0 (socket says 172.21.15.1:36940) 2024-03-06T07:20:57.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f943effd640 1 -- 172.21.15.1:0/1181646653 learned_addr learned my addr 172.21.15.1:0/1181646653 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:57.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f943f7fe640 1 --2- 172.21.15.1:0/1181646653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94400a5fa0 0x7f944014c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f943ffff640 1 --2- 172.21.15.1:0/1181646653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9440152b70 0x7f944014af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f943effd640 1 -- 172.21.15.1:0/1181646653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9440152b70 msgr2=0x7f944014af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f943effd640 1 --2- 172.21.15.1:0/1181646653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9440152b70 0x7f944014af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f943effd640 1 -- 172.21.15.1:0/1181646653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94400a5fa0 msgr2=0x7f944014c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f943effd640 1 --2- 172.21.15.1:0/1181646653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94400a5fa0 0x7f944014c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f943effd640 1 -- 172.21.15.1:0/1181646653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9440145c90 con 0x7f94400c2b30 2024-03-06T07:20:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f943f7fe640 1 --2- 172.21.15.1:0/1181646653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94400a5fa0 0x7f944014c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f943effd640 1 --2- 172.21.15.1:0/1181646653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94400c2b30 0x7f944014a9f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f943400b3f0 tx=0x7f943400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f9427fff640 1 -- 172.21.15.1:0/1181646653 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9434015020 con 0x7f94400c2b30 2024-03-06T07:20:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.434+0000 7f9427fff640 1 -- 172.21.15.1:0/1181646653 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9434011830 con 0x7f94400c2b30 2024-03-06T07:20:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.435+0000 7f944618a640 1 -- 172.21.15.1:0/1181646653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9440149c50 con 0x7f94400c2b30 2024-03-06T07:20:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.435+0000 7f9427fff640 1 -- 172.21.15.1:0/1181646653 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9434011c20 con 0x7f94400c2b30 2024-03-06T07:20:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.435+0000 7f944618a640 1 -- 172.21.15.1:0/1181646653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f944014a210 con 0x7f94400c2b30 2024-03-06T07:20:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.435+0000 7f944618a640 1 -- 172.21.15.1:0/1181646653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9440067fd0 con 0x7f94400c2b30 2024-03-06T07:20:57.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.436+0000 7f9427fff640 1 -- 172.21.15.1:0/1181646653 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9434018020 con 0x7f94400c2b30 2024-03-06T07:20:57.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.436+0000 7f9427fff640 1 --2- 172.21.15.1:0/1181646653 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f942c06b500 0x7f942c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.436+0000 7f9427fff640 1 -- 172.21.15.1:0/1181646653 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f943408a100 con 0x7f94400c2b30 2024-03-06T07:20:57.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.437+0000 7f943f7fe640 1 --2- 172.21.15.1:0/1181646653 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f942c06b500 0x7f942c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.437+0000 7f943f7fe640 1 --2- 172.21.15.1:0/1181646653 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f942c06b500 0x7f942c06d9c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9430036000 tx=0x7f943003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.440+0000 7f9427fff640 1 -- 172.21.15.1:0/1181646653 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f943401f030 con 0x7f94400c2b30 2024-03-06T07:20:57.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.623+0000 7f944618a640 1 -- 172.21.15.1:0/1181646653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94400a5b10 con 0x7f94400c2b30 2024-03-06T07:20:57.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.624+0000 7f9427fff640 1 -- 172.21.15.1:0/1181646653 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7f94340172c0 con 0x7f94400c2b30 2024-03-06T07:20:57.626 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:57.626 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:57.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.626+0000 7f944618a640 1 -- 172.21.15.1:0/1181646653 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f942c06b500 msgr2=0x7f942c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.626+0000 7f944618a640 1 --2- 172.21.15.1:0/1181646653 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f942c06b500 0x7f942c06d9c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9430036000 tx=0x7f943003c040 comp rx=0 tx=0).stop 2024-03-06T07:20:57.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.626+0000 7f944618a640 1 -- 172.21.15.1:0/1181646653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94400c2b30 msgr2=0x7f944014a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.626+0000 7f944618a640 1 --2- 172.21.15.1:0/1181646653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94400c2b30 0x7f944014a9f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f943400b3f0 tx=0x7f943400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.626+0000 7f944618a640 1 -- 172.21.15.1:0/1181646653 shutdown_connections 2024-03-06T07:20:57.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.626+0000 7f944618a640 1 --2- 172.21.15.1:0/1181646653 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f942c06b500 0x7f942c06d9c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.626+0000 7f944618a640 1 --2- 172.21.15.1:0/1181646653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9440152b70 0x7f944014af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.626+0000 7f944618a640 1 --2- 172.21.15.1:0/1181646653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94400c2b30 0x7f944014a9f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.626+0000 7f944618a640 1 --2- 172.21.15.1:0/1181646653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94400a5fa0 0x7f944014c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.626+0000 7f944618a640 1 -- 172.21.15.1:0/1181646653 >> 172.21.15.1:0/1181646653 conn(0x7f94400c6860 msgr2=0x7f944014f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:57.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.626+0000 7f944618a640 1 -- 172.21.15.1:0/1181646653 shutdown_connections 2024-03-06T07:20:57.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.626+0000 7f944618a640 1 -- 172.21.15.1:0/1181646653 wait complete. 2024-03-06T07:20:57.639 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:20:19.337560+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:20:20.172791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:20:23.884071+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:20:57.639 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:20:57.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.744+0000 7fd9ab3d0640 1 Processor -- start 2024-03-06T07:20:57.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.744+0000 7fd9ab3d0640 1 -- start start 2024-03-06T07:20:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.745+0000 7fd9ab3d0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd9a40c69b0 0x7fd9a40c2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.745+0000 7fd9ab3d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a40c7510 0x7fd9a40c3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.745+0000 7fd9ab3d0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a4072d00 con 0x7fd9a4159140 2024-03-06T07:20:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.745+0000 7fd9ab3d0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a40c0e80 con 0x7fd9a40c69b0 2024-03-06T07:20:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.745+0000 7fd9ab3d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a40c13b0 con 0x7fd9a40c7510 2024-03-06T07:20:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.745+0000 7fd9a896b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd9a40c69b0 0x7fd9a40c2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.745+0000 7fd9a896b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd9a40c69b0 0x7fd9a40c2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36950/0 (socket says 172.21.15.1:36950) 2024-03-06T07:20:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.745+0000 7fd9a896b640 1 -- 172.21.15.1:0/1016588253 learned_addr learned my addr 172.21.15.1:0/1016588253 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.745+0000 7fd9a996d640 1 --2- 172.21.15.1:0/1016588253 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a40c7510 0x7fd9a40c3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.745+0000 7fd9a896b640 1 -- 172.21.15.1:0/1016588253 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a40c7510 msgr2=0x7fd9a40c3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:57.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.745+0000 7fd9a896b640 1 --2- 172.21.15.1:0/1016588253 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a40c7510 0x7fd9a40c3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.745+0000 7fd9a896b640 1 -- 172.21.15.1:0/1016588253 >> v1:172.21.15.98:6789/0 conn(0x7fd9a4159140 legacy=0x7fd9a40c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:57.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.745+0000 7fd9a896b640 1 -- 172.21.15.1:0/1016588253 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a4073180 con 0x7fd9a40c69b0 2024-03-06T07:20:57.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.745+0000 7fd9a996d640 1 --2- 172.21.15.1:0/1016588253 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a40c7510 0x7fd9a40c3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:57.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.746+0000 7fd9a896b640 1 --2- 172.21.15.1:0/1016588253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a40c69b0 0x7fd9a40c2b30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd994009870 tx=0x7fd99402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c085f7d036b1b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.746+0000 7fd99bfff640 1 -- 172.21.15.1:0/1016588253 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9940317c0 con 0x7fd9a40c69b0 2024-03-06T07:20:57.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.746+0000 7fd99bfff640 1 -- 172.21.15.1:0/1016588253 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd994037070 con 0x7fd9a40c69b0 2024-03-06T07:20:57.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.746+0000 7fd99bfff640 1 -- 172.21.15.1:0/1016588253 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd994031b80 con 0x7fd9a40c69b0 2024-03-06T07:20:57.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.747+0000 7fd9ab3d0640 1 -- 172.21.15.1:0/1016588253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a40c69b0 msgr2=0x7fd9a40c2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.747+0000 7fd9ab3d0640 1 --2- 172.21.15.1:0/1016588253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a40c69b0 0x7fd9a40c2b30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd994009870 tx=0x7fd99402b860 comp rx=0 tx=0).stop 2024-03-06T07:20:57.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.747+0000 7fd9ab3d0640 1 -- 172.21.15.1:0/1016588253 shutdown_connections 2024-03-06T07:20:57.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.747+0000 7fd9ab3d0640 1 --2- 172.21.15.1:0/1016588253 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a40c7510 0x7fd9a40c3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.747+0000 7fd9ab3d0640 1 --2- 172.21.15.1:0/1016588253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a40c69b0 0x7fd9a40c2b30 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.747+0000 7fd9ab3d0640 1 -- 172.21.15.1:0/1016588253 >> 172.21.15.1:0/1016588253 conn(0x7fd9a40c7a50 msgr2=0x7fd9a40c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:57.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.747+0000 7fd9ab3d0640 1 -- 172.21.15.1:0/1016588253 shutdown_connections 2024-03-06T07:20:57.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.747+0000 7fd9ab3d0640 1 -- 172.21.15.1:0/1016588253 wait complete. 2024-03-06T07:20:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.748+0000 7fd9ab3d0640 1 Processor -- start 2024-03-06T07:20:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.748+0000 7fd9ab3d0640 1 -- start start 2024-03-06T07:20:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9ab3d0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a40c69b0 0x7fd9a40a98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9ab3d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a40c7510 0x7fd9a40aa610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9ab3d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9a4159140 0x7fd9a4155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9ab3d0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a40c35b0 con 0x7fd9a40c69b0 2024-03-06T07:20:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9ab3d0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a40c6520 con 0x7fd9a4159140 2024-03-06T07:20:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9ab3d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a40c10d0 con 0x7fd9a40c7510 2024-03-06T07:20:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9a916c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a40c69b0 0x7fd9a40a98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9a916c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a40c69b0 0x7fd9a40a98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36958/0 (socket says 172.21.15.1:36958) 2024-03-06T07:20:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9a916c640 1 -- 172.21.15.1:0/388554935 learned_addr learned my addr 172.21.15.1:0/388554935 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9a996d640 1 --2- 172.21.15.1:0/388554935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9a4159140 0x7fd9a4155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9a896b640 1 --2- 172.21.15.1:0/388554935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a40c7510 0x7fd9a40aa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9a916c640 1 -- 172.21.15.1:0/388554935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a40c7510 msgr2=0x7fd9a40aa610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9a916c640 1 --2- 172.21.15.1:0/388554935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a40c7510 0x7fd9a40aa610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9a916c640 1 -- 172.21.15.1:0/388554935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9a4159140 msgr2=0x7fd9a4155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9a916c640 1 --2- 172.21.15.1:0/388554935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9a4159140 0x7fd9a4155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9a916c640 1 -- 172.21.15.1:0/388554935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a4099e40 con 0x7fd9a40c69b0 2024-03-06T07:20:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9a996d640 1 --2- 172.21.15.1:0/388554935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9a4159140 0x7fd9a4155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9a896b640 1 --2- 172.21.15.1:0/388554935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a40c7510 0x7fd9a40aa610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.749+0000 7fd9a916c640 1 --2- 172.21.15.1:0/388554935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a40c69b0 0x7fd9a40a98d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000b3f0 tx=0x7fd9a000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.750+0000 7fd999ffb640 1 -- 172.21.15.1:0/388554935 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0015020 con 0x7fd9a40c69b0 2024-03-06T07:20:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.750+0000 7fd999ffb640 1 -- 172.21.15.1:0/388554935 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a0011830 con 0x7fd9a40c69b0 2024-03-06T07:20:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.750+0000 7fd9ab3d0640 1 -- 172.21.15.1:0/388554935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a40a9e10 con 0x7fd9a40c69b0 2024-03-06T07:20:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.750+0000 7fd999ffb640 1 -- 172.21.15.1:0/388554935 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0011c20 con 0x7fd9a40c69b0 2024-03-06T07:20:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.750+0000 7fd9ab3d0640 1 -- 172.21.15.1:0/388554935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a40a6bc0 con 0x7fd9a40c69b0 2024-03-06T07:20:57.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.751+0000 7fd9ab3d0640 1 -- 172.21.15.1:0/388554935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a40686e0 con 0x7fd9a40c69b0 2024-03-06T07:20:57.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.751+0000 7fd999ffb640 1 -- 172.21.15.1:0/388554935 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9a0018020 con 0x7fd9a40c69b0 2024-03-06T07:20:57.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.752+0000 7fd999ffb640 1 --2- 172.21.15.1:0/388554935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd98806b500 0x7fd98806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.752+0000 7fd999ffb640 1 -- 172.21.15.1:0/388554935 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd9a008a070 con 0x7fd9a40c69b0 2024-03-06T07:20:57.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.752+0000 7fd9a896b640 1 --2- 172.21.15.1:0/388554935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd98806b500 0x7fd98806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.752+0000 7fd9a896b640 1 --2- 172.21.15.1:0/388554935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd98806b500 0x7fd98806d9c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd994036000 tx=0x7fd994008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.756+0000 7fd999ffb640 1 -- 172.21.15.1:0/388554935 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9a001f030 con 0x7fd9a40c69b0 2024-03-06T07:20:57.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.940+0000 7fd9ab3d0640 1 -- 172.21.15.1:0/388554935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd9a4157cc0 con 0x7fd9a40c69b0 2024-03-06T07:20:57.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.940+0000 7fd999ffb640 1 -- 172.21.15.1:0/388554935 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd9a00172c0 con 0x7fd9a40c69b0 2024-03-06T07:20:57.942 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:20:57.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.943+0000 7fd9ab3d0640 1 -- 172.21.15.1:0/388554935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd98806b500 msgr2=0x7fd98806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.943+0000 7fd9ab3d0640 1 --2- 172.21.15.1:0/388554935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd98806b500 0x7fd98806d9c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd994036000 tx=0x7fd994008030 comp rx=0 tx=0).stop 2024-03-06T07:20:57.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.943+0000 7fd9ab3d0640 1 -- 172.21.15.1:0/388554935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a40c69b0 msgr2=0x7fd9a40a98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.943+0000 7fd9ab3d0640 1 --2- 172.21.15.1:0/388554935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a40c69b0 0x7fd9a40a98d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000b3f0 tx=0x7fd9a000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.943+0000 7fd9ab3d0640 1 -- 172.21.15.1:0/388554935 shutdown_connections 2024-03-06T07:20:57.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.943+0000 7fd9ab3d0640 1 --2- 172.21.15.1:0/388554935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd98806b500 0x7fd98806d9c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.943+0000 7fd9ab3d0640 1 --2- 172.21.15.1:0/388554935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9a4159140 0x7fd9a4155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.943+0000 7fd9ab3d0640 1 --2- 172.21.15.1:0/388554935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a40c7510 0x7fd9a40aa610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.943+0000 7fd9ab3d0640 1 --2- 172.21.15.1:0/388554935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a40c69b0 0x7fd9a40a98d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.943+0000 7fd9ab3d0640 1 -- 172.21.15.1:0/388554935 >> 172.21.15.1:0/388554935 conn(0x7fd9a40c7a50 msgr2=0x7fd9a4070060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:57.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.943+0000 7fd9ab3d0640 1 -- 172.21.15.1:0/388554935 shutdown_connections 2024-03-06T07:20:57.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:57.943+0000 7fd9ab3d0640 1 -- 172.21.15.1:0/388554935 wait complete. 2024-03-06T07:20:57.956 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:20:58.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.059+0000 7fa930f81640 1 Processor -- start 2024-03-06T07:20:58.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.059+0000 7fa930f81640 1 -- start start 2024-03-06T07:20:58.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.059+0000 7fa930f81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa92c0c3220 0x7fa92c0c5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.059+0000 7fa930f81640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa92c0c5dd0 0x7fa92c155790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.059+0000 7fa930f81640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92c06eb10 con 0x7fa92c0c2e40 2024-03-06T07:20:58.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.059+0000 7fa930f81640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa92c06f400 con 0x7fa92c0c5dd0 2024-03-06T07:20:58.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.059+0000 7fa930f81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c06edb0 con 0x7fa92c0c3220 2024-03-06T07:20:58.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.060+0000 7fa92ad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa92c0c5dd0 0x7fa92c155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.060+0000 7fa92ad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa92c0c5dd0 0x7fa92c155790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36970/0 (socket says 172.21.15.1:36970) 2024-03-06T07:20:58.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.060+0000 7fa92ad9d640 1 -- 172.21.15.1:0/3677702626 learned_addr learned my addr 172.21.15.1:0/3677702626 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:58.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.060+0000 7fa92a59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa92c0c2e40 0x7fa92c06c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36326/0 (socket says 172.21.15.1:36326) 2024-03-06T07:20:58.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.060+0000 7fa921d9b640 1 --2- 172.21.15.1:0/3677702626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa92c0c3220 0x7fa92c0c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.060+0000 7fa92ad9d640 1 -- 172.21.15.1:0/3677702626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa92c0c3220 msgr2=0x7fa92c0c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:58.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.060+0000 7fa92ad9d640 1 --2- 172.21.15.1:0/3677702626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa92c0c3220 0x7fa92c0c5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.060+0000 7fa92ad9d640 1 -- 172.21.15.1:0/3677702626 >> v1:172.21.15.1:6789/0 conn(0x7fa92c0c2e40 legacy=0x7fa92c06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.060+0000 7fa92ad9d640 1 -- 172.21.15.1:0/3677702626 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92c06f090 con 0x7fa92c0c5dd0 2024-03-06T07:20:58.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.060+0000 7fa929d9b640 1 -- 172.21.15.1:0/3677702626 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239606538 0 0) 0x7fa92c06eb10 con 0x7fa92c0c2e40 2024-03-06T07:20:58.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.061+0000 7fa92ad9d640 1 --2- 172.21.15.1:0/3677702626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c0c5dd0 0x7fa92c155790 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa92400d180 tx=0x7fa92402e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e0bcf565a83e509 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.061+0000 7fa929d9b640 1 -- 172.21.15.1:0/3677702626 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92402ea60 con 0x7fa92c0c5dd0 2024-03-06T07:20:58.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.061+0000 7fa929d9b640 1 -- 172.21.15.1:0/3677702626 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa92402ec00 con 0x7fa92c0c5dd0 2024-03-06T07:20:58.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.061+0000 7fa929d9b640 1 -- 172.21.15.1:0/3677702626 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa924009600 con 0x7fa92c0c5dd0 2024-03-06T07:20:58.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.062+0000 7fa930f81640 1 -- 172.21.15.1:0/3677702626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c0c5dd0 msgr2=0x7fa92c155790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.062+0000 7fa930f81640 1 --2- 172.21.15.1:0/3677702626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c0c5dd0 0x7fa92c155790 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa92400d180 tx=0x7fa92402e0c0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.062+0000 7fa930f81640 1 -- 172.21.15.1:0/3677702626 shutdown_connections 2024-03-06T07:20:58.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.062+0000 7fa930f81640 1 --2- 172.21.15.1:0/3677702626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa92c0c3220 0x7fa92c0c5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.062+0000 7fa930f81640 1 --2- 172.21.15.1:0/3677702626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c0c5dd0 0x7fa92c155790 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.062+0000 7fa930f81640 1 -- 172.21.15.1:0/3677702626 >> 172.21.15.1:0/3677702626 conn(0x7fa92c0a9700 msgr2=0x7fa92c0a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:58.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.062+0000 7fa930f81640 1 -- 172.21.15.1:0/3677702626 shutdown_connections 2024-03-06T07:20:58.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.062+0000 7fa930f81640 1 -- 172.21.15.1:0/3677702626 wait complete. 2024-03-06T07:20:58.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.063+0000 7fa930f81640 1 Processor -- start 2024-03-06T07:20:58.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.063+0000 7fa930f81640 1 -- start start 2024-03-06T07:20:58.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa930f81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa92c0c2e40 0x7fa92c0c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa930f81640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c0c3220 0x7fa92c1d81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa930f81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa92c0c5dd0 0x7fa92c1d8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa930f81640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c0a6fb0 con 0x7fa92c0c3220 2024-03-06T07:20:58.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa930f81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c0a6e30 con 0x7fa92c0c2e40 2024-03-06T07:20:58.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa930f81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c06f890 con 0x7fa92c0c5dd0 2024-03-06T07:20:58.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa921d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c0c3220 0x7fa92c1d81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa921d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c0c3220 0x7fa92c1d81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36984/0 (socket says 172.21.15.1:36984) 2024-03-06T07:20:58.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa921d9b640 1 -- 172.21.15.1:0/1264837640 learned_addr learned my addr 172.21.15.1:0/1264837640 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:58.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa92a59c640 1 --2- 172.21.15.1:0/1264837640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa92c0c2e40 0x7fa92c0c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa92ad9d640 1 --2- 172.21.15.1:0/1264837640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa92c0c5dd0 0x7fa92c1d8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa921d9b640 1 -- 172.21.15.1:0/1264837640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa92c0c5dd0 msgr2=0x7fa92c1d8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa921d9b640 1 --2- 172.21.15.1:0/1264837640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa92c0c5dd0 0x7fa92c1d8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa921d9b640 1 -- 172.21.15.1:0/1264837640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa92c0c2e40 msgr2=0x7fa92c0c0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa921d9b640 1 --2- 172.21.15.1:0/1264837640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa92c0c2e40 0x7fa92c0c0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa921d9b640 1 -- 172.21.15.1:0/1264837640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92c1edc50 con 0x7fa92c0c3220 2024-03-06T07:20:58.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa92ad9d640 1 --2- 172.21.15.1:0/1264837640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa92c0c5dd0 0x7fa92c1d8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:58.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.064+0000 7fa92a59c640 1 --2- 172.21.15.1:0/1264837640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa92c0c2e40 0x7fa92c0c0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:58.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.065+0000 7fa921d9b640 1 --2- 172.21.15.1:0/1264837640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c0c3220 0x7fa92c1d81c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa91400c820 tx=0x7fa91400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.065+0000 7fa9237fe640 1 -- 172.21.15.1:0/1264837640 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa914015070 con 0x7fa92c0c3220 2024-03-06T07:20:58.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.065+0000 7fa9237fe640 1 -- 172.21.15.1:0/1264837640 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa914004240 con 0x7fa92c0c3220 2024-03-06T07:20:58.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.065+0000 7fa930f81640 1 -- 172.21.15.1:0/1264837640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa92c1edf30 con 0x7fa92c0c3220 2024-03-06T07:20:58.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.065+0000 7fa9237fe640 1 -- 172.21.15.1:0/1264837640 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91400ce10 con 0x7fa92c0c3220 2024-03-06T07:20:58.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.065+0000 7fa930f81640 1 -- 172.21.15.1:0/1264837640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa92c1ee4f0 con 0x7fa92c0c3220 2024-03-06T07:20:58.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.066+0000 7fa930f81640 1 -- 172.21.15.1:0/1264837640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa92c0686e0 con 0x7fa92c0c3220 2024-03-06T07:20:58.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.066+0000 7fa9237fe640 1 -- 172.21.15.1:0/1264837640 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa914007830 con 0x7fa92c0c3220 2024-03-06T07:20:58.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.067+0000 7fa9237fe640 1 --2- 172.21.15.1:0/1264837640 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa90c06b4d0 0x7fa90c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.067+0000 7fa9237fe640 1 -- 172.21.15.1:0/1264837640 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa91408a410 con 0x7fa92c0c3220 2024-03-06T07:20:58.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.067+0000 7fa92a59c640 1 --2- 172.21.15.1:0/1264837640 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa90c06b4d0 0x7fa90c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.068+0000 7fa92a59c640 1 --2- 172.21.15.1:0/1264837640 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa90c06b4d0 0x7fa90c06d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa91c000f60 tx=0x7fa91c007c30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.072+0000 7fa9237fe640 1 -- 172.21.15.1:0/1264837640 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa914058f50 con 0x7fa92c0c3220 2024-03-06T07:20:58.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.252+0000 7fa930f81640 1 -- 172.21.15.1:0/1264837640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa92c0c5320 con 0x7fa92c0c3220 2024-03-06T07:20:58.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.252+0000 7fa9237fe640 1 -- 172.21.15.1:0/1264837640 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa914058850 con 0x7fa92c0c3220 2024-03-06T07:20:58.255 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:20:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.255+0000 7fa930f81640 1 -- 172.21.15.1:0/1264837640 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa90c06b4d0 msgr2=0x7fa90c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.255+0000 7fa930f81640 1 --2- 172.21.15.1:0/1264837640 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa90c06b4d0 0x7fa90c06d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa91c000f60 tx=0x7fa91c007c30 comp rx=0 tx=0).stop 2024-03-06T07:20:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.255+0000 7fa930f81640 1 -- 172.21.15.1:0/1264837640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c0c3220 msgr2=0x7fa92c1d81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.255+0000 7fa930f81640 1 --2- 172.21.15.1:0/1264837640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c0c3220 0x7fa92c1d81c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa91400c820 tx=0x7fa91400cce0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.255+0000 7fa930f81640 1 -- 172.21.15.1:0/1264837640 shutdown_connections 2024-03-06T07:20:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.255+0000 7fa930f81640 1 --2- 172.21.15.1:0/1264837640 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa90c06b4d0 0x7fa90c06d990 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.255+0000 7fa930f81640 1 --2- 172.21.15.1:0/1264837640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa92c0c5dd0 0x7fa92c1d8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.255+0000 7fa930f81640 1 --2- 172.21.15.1:0/1264837640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c0c3220 0x7fa92c1d81c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.255+0000 7fa930f81640 1 --2- 172.21.15.1:0/1264837640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa92c0c2e40 0x7fa92c0c0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.255+0000 7fa930f81640 1 -- 172.21.15.1:0/1264837640 >> 172.21.15.1:0/1264837640 conn(0x7fa92c0a9700 msgr2=0x7fa92c0a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:58.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.255+0000 7fa930f81640 1 -- 172.21.15.1:0/1264837640 shutdown_connections 2024-03-06T07:20:58.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.255+0000 7fa930f81640 1 -- 172.21.15.1:0/1264837640 wait complete. 2024-03-06T07:20:58.268 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:20:58.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.370+0000 7f5d900d4640 1 Processor -- start 2024-03-06T07:20:58.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.370+0000 7f5d900d4640 1 -- start start 2024-03-06T07:20:58.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.371+0000 7f5d900d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d880c3970 0x7f5d880c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.371+0000 7f5d900d4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5d880c60b0 0x7f5d88152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.371+0000 7f5d900d4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d880c5e40 con 0x7f5d88152670 2024-03-06T07:20:58.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.371+0000 7f5d900d4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d88152ad0 con 0x7f5d880c60b0 2024-03-06T07:20:58.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.371+0000 7f5d900d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880c67f0 con 0x7f5d880c3970 2024-03-06T07:20:58.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.371+0000 7f5d8d66f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5d880c60b0 0x7f5d88152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.371+0000 7f5d8d66f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5d880c60b0 0x7f5d88152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36990/0 (socket says 172.21.15.1:36990) 2024-03-06T07:20:58.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.371+0000 7f5d8d66f640 1 -- 172.21.15.1:0/969394573 learned_addr learned my addr 172.21.15.1:0/969394573 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:58.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.371+0000 7f5d8de70640 1 --2- 172.21.15.1:0/969394573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d880c3970 0x7f5d880c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.371+0000 7f5d8d66f640 1 -- 172.21.15.1:0/969394573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d880c3970 msgr2=0x7f5d880c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:58.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.371+0000 7f5d8d66f640 1 --2- 172.21.15.1:0/969394573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d880c3970 0x7f5d880c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.371+0000 7f5d8d66f640 1 -- 172.21.15.1:0/969394573 >> v1:172.21.15.98:6789/0 conn(0x7f5d88152670 legacy=0x7f5d881553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:58.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.371+0000 7f5d8d66f640 1 -- 172.21.15.1:0/969394573 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d880c6a40 con 0x7f5d880c60b0 2024-03-06T07:20:58.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.372+0000 7f5d8d66f640 1 --2- 172.21.15.1:0/969394573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d880c60b0 0x7f5d88152130 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009870 tx=0x7f5d7802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a066828ad2c008 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.372+0000 7f5d8ce6e640 1 -- 172.21.15.1:0/969394573 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78035020 con 0x7f5d880c60b0 2024-03-06T07:20:58.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.372+0000 7f5d8ce6e640 1 -- 172.21.15.1:0/969394573 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d78031640 con 0x7f5d880c60b0 2024-03-06T07:20:58.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.372+0000 7f5d8ce6e640 1 -- 172.21.15.1:0/969394573 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7802d040 con 0x7f5d880c60b0 2024-03-06T07:20:58.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.373+0000 7f5d900d4640 1 -- 172.21.15.1:0/969394573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d880c60b0 msgr2=0x7f5d88152130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.373+0000 7f5d900d4640 1 --2- 172.21.15.1:0/969394573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d880c60b0 0x7f5d88152130 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009870 tx=0x7f5d7802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.373+0000 7f5d900d4640 1 -- 172.21.15.1:0/969394573 shutdown_connections 2024-03-06T07:20:58.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.373+0000 7f5d900d4640 1 --2- 172.21.15.1:0/969394573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d880c60b0 0x7f5d88152130 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.373+0000 7f5d900d4640 1 --2- 172.21.15.1:0/969394573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d880c3970 0x7f5d880c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.373+0000 7f5d900d4640 1 -- 172.21.15.1:0/969394573 >> 172.21.15.1:0/969394573 conn(0x7f5d880c88f0 msgr2=0x7f5d880c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:58.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.373+0000 7f5d900d4640 1 -- 172.21.15.1:0/969394573 shutdown_connections 2024-03-06T07:20:58.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.373+0000 7f5d900d4640 1 -- 172.21.15.1:0/969394573 wait complete. 2024-03-06T07:20:58.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.374+0000 7f5d900d4640 1 Processor -- start 2024-03-06T07:20:58.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.374+0000 7f5d900d4640 1 -- start start 2024-03-06T07:20:58.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.375+0000 7f5d900d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d880c3970 0x7f5d880c2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.375+0000 7f5d900d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d880c60b0 0x7f5d8814a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.375+0000 7f5d900d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d88152670 0x7f5d8814ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.375+0000 7f5d900d4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880c72e0 con 0x7f5d880c3970 2024-03-06T07:20:58.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.375+0000 7f5d900d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880c7000 con 0x7f5d880c60b0 2024-03-06T07:20:58.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.375+0000 7f5d900d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880c75c0 con 0x7f5d88152670 2024-03-06T07:20:58.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.375+0000 7f5d8de70640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d880c3970 0x7f5d880c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.375+0000 7f5d8de70640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d880c3970 0x7f5d880c2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36994/0 (socket says 172.21.15.1:36994) 2024-03-06T07:20:58.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.375+0000 7f5d8de70640 1 -- 172.21.15.1:0/3918415210 learned_addr learned my addr 172.21.15.1:0/3918415210 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:58.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.375+0000 7f5d8d66f640 1 --2- 172.21.15.1:0/3918415210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d880c60b0 0x7f5d8814a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.375+0000 7f5d8e671640 1 --2- 172.21.15.1:0/3918415210 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d88152670 0x7f5d8814ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.375+0000 7f5d8de70640 1 -- 172.21.15.1:0/3918415210 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d88152670 msgr2=0x7f5d8814ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.375+0000 7f5d8de70640 1 --2- 172.21.15.1:0/3918415210 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d88152670 0x7f5d8814ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.375+0000 7f5d8de70640 1 -- 172.21.15.1:0/3918415210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d880c60b0 msgr2=0x7f5d8814a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.375+0000 7f5d8de70640 1 --2- 172.21.15.1:0/3918415210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d880c60b0 0x7f5d8814a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.375+0000 7f5d8de70640 1 -- 172.21.15.1:0/3918415210 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d881577a0 con 0x7f5d880c3970 2024-03-06T07:20:58.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.375+0000 7f5d8de70640 1 --2- 172.21.15.1:0/3918415210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d880c3970 0x7f5d880c2230 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5d8400d830 tx=0x7f5d8400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.376+0000 7f5d7e7fc640 1 -- 172.21.15.1:0/3918415210 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8400b6c0 con 0x7f5d880c3970 2024-03-06T07:20:58.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.376+0000 7f5d7e7fc640 1 -- 172.21.15.1:0/3918415210 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d84019070 con 0x7f5d880c3970 2024-03-06T07:20:58.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.376+0000 7f5d900d4640 1 -- 172.21.15.1:0/3918415210 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d88157a80 con 0x7f5d880c3970 2024-03-06T07:20:58.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.376+0000 7f5d7e7fc640 1 -- 172.21.15.1:0/3918415210 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8400ba80 con 0x7f5d880c3970 2024-03-06T07:20:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.376+0000 7f5d900d4640 1 -- 172.21.15.1:0/3918415210 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d88158040 con 0x7f5d880c3970 2024-03-06T07:20:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.377+0000 7f5d900d4640 1 -- 172.21.15.1:0/3918415210 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d880686e0 con 0x7f5d880c3970 2024-03-06T07:20:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.377+0000 7f5d7e7fc640 1 -- 172.21.15.1:0/3918415210 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d8401e460 con 0x7f5d880c3970 2024-03-06T07:20:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.377+0000 7f5d7e7fc640 1 --2- 172.21.15.1:0/3918415210 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d6c06b4d0 0x7f5d6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.378+0000 7f5d7e7fc640 1 -- 172.21.15.1:0/3918415210 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5d8408a9f0 con 0x7f5d880c3970 2024-03-06T07:20:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.378+0000 7f5d8d66f640 1 --2- 172.21.15.1:0/3918415210 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d6c06b4d0 0x7f5d6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.378+0000 7f5d8d66f640 1 --2- 172.21.15.1:0/3918415210 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d6c06b4d0 0x7f5d6c06d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5d78036000 tx=0x7f5d7803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.382+0000 7f5d7e7fc640 1 -- 172.21.15.1:0/3918415210 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d84058f70 con 0x7f5d880c3970 2024-03-06T07:20:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.563+0000 7f5d900d4640 1 -- 172.21.15.1:0/3918415210 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5d880c2770 con 0x7f5d880c3970 2024-03-06T07:20:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.563+0000 7f5d7e7fc640 1 -- 172.21.15.1:0/3918415210 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5d8405d010 con 0x7f5d880c3970 2024-03-06T07:20:58.566 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:20:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.565+0000 7f5d900d4640 1 -- 172.21.15.1:0/3918415210 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d6c06b4d0 msgr2=0x7f5d6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.565+0000 7f5d900d4640 1 --2- 172.21.15.1:0/3918415210 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d6c06b4d0 0x7f5d6c06d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5d78036000 tx=0x7f5d7803c040 comp rx=0 tx=0).stop 2024-03-06T07:20:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.565+0000 7f5d900d4640 1 -- 172.21.15.1:0/3918415210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d880c3970 msgr2=0x7f5d880c2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.565+0000 7f5d900d4640 1 --2- 172.21.15.1:0/3918415210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d880c3970 0x7f5d880c2230 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5d8400d830 tx=0x7f5d8400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.565+0000 7f5d900d4640 1 -- 172.21.15.1:0/3918415210 shutdown_connections 2024-03-06T07:20:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.565+0000 7f5d900d4640 1 --2- 172.21.15.1:0/3918415210 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d6c06b4d0 0x7f5d6c06d990 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.565+0000 7f5d900d4640 1 --2- 172.21.15.1:0/3918415210 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d88152670 0x7f5d8814ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.565+0000 7f5d900d4640 1 --2- 172.21.15.1:0/3918415210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d880c60b0 0x7f5d8814a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.565+0000 7f5d900d4640 1 --2- 172.21.15.1:0/3918415210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d880c3970 0x7f5d880c2230 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.565+0000 7f5d900d4640 1 -- 172.21.15.1:0/3918415210 >> 172.21.15.1:0/3918415210 conn(0x7f5d880c88f0 msgr2=0x7f5d88074600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.565+0000 7f5d900d4640 1 -- 172.21.15.1:0/3918415210 shutdown_connections 2024-03-06T07:20:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.565+0000 7f5d900d4640 1 -- 172.21.15.1:0/3918415210 wait complete. 2024-03-06T07:20:58.578 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:20:58.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.681+0000 7fa12218a640 1 Processor -- start 2024-03-06T07:20:58.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.681+0000 7fa12218a640 1 -- start start 2024-03-06T07:20:58.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.681+0000 7fa12218a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa11c0c6b10 0x7fa11c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.681+0000 7fa12218a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa11c0bf9d0 0x7fa11c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.681+0000 7fa12218a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11c0c2fc0 con 0x7fa11c152650 2024-03-06T07:20:58.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.681+0000 7fa12218a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11c155800 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.681+0000 7fa12218a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11c158da0 con 0x7fa11c0c6b10 2024-03-06T07:20:58.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.682+0000 7fa11bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa11c0bf9d0 0x7fa11c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.682+0000 7fa11bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa11c0bf9d0 0x7fa11c0c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37010/0 (socket says 172.21.15.1:37010) 2024-03-06T07:20:58.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.682+0000 7fa11b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa11c152650 0x7fa11c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36332/0 (socket says 172.21.15.1:36332) 2024-03-06T07:20:58.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.682+0000 7fa11bfff640 1 -- 172.21.15.1:0/1929555231 learned_addr learned my addr 172.21.15.1:0/1929555231 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:58.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.682+0000 7fa11affd640 1 --2- 172.21.15.1:0/1929555231 >> v2:172.21.15.98:3300/0 conn(0x7fa11c0c6b10 0x7fa11c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.682+0000 7fa11bfff640 1 -- 172.21.15.1:0/1929555231 >> v2:172.21.15.98:3300/0 conn(0x7fa11c0c6b10 msgr2=0x7fa11c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:58.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.682+0000 7fa11bfff640 1 --2- 172.21.15.1:0/1929555231 >> v2:172.21.15.98:3300/0 conn(0x7fa11c0c6b10 0x7fa11c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.682+0000 7fa11bfff640 1 -- 172.21.15.1:0/1929555231 >> v1:172.21.15.1:6789/0 conn(0x7fa11c152650 legacy=0x7fa11c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.682+0000 7fa11bfff640 1 -- 172.21.15.1:0/1929555231 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c0c6690 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.682+0000 7fa11a7fc640 1 -- 172.21.15.1:0/1929555231 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820836521 0 0) 0x7fa11c0c2fc0 con 0x7fa11c152650 2024-03-06T07:20:58.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.683+0000 7fa11bfff640 1 --2- 172.21.15.1:0/1929555231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11c0bf9d0 0x7fa11c0c1460 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0cb8e0 tx=0x7fa11402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7db7a1774ef21ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.683+0000 7fa11a7fc640 1 -- 172.21.15.1:0/1929555231 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11402ca80 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.683+0000 7fa11a7fc640 1 -- 172.21.15.1:0/1929555231 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11402cc20 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.683+0000 7fa11a7fc640 1 -- 172.21.15.1:0/1929555231 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1140343f0 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.683+0000 7fa12218a640 1 -- 172.21.15.1:0/1929555231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11c0bf9d0 msgr2=0x7fa11c0c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.683+0000 7fa12218a640 1 --2- 172.21.15.1:0/1929555231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11c0bf9d0 0x7fa11c0c1460 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0cb8e0 tx=0x7fa11402bee0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.683+0000 7fa12218a640 1 -- 172.21.15.1:0/1929555231 shutdown_connections 2024-03-06T07:20:58.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.683+0000 7fa12218a640 1 --2- 172.21.15.1:0/1929555231 >> v2:172.21.15.98:3300/0 conn(0x7fa11c0c6b10 0x7fa11c155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.683+0000 7fa12218a640 1 --2- 172.21.15.1:0/1929555231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11c0bf9d0 0x7fa11c0c1460 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.683+0000 7fa12218a640 1 -- 172.21.15.1:0/1929555231 >> 172.21.15.1:0/1929555231 conn(0x7fa11c066ee0 msgr2=0x7fa11c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:58.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.683+0000 7fa12218a640 1 -- 172.21.15.1:0/1929555231 shutdown_connections 2024-03-06T07:20:58.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.683+0000 7fa12218a640 1 -- 172.21.15.1:0/1929555231 wait complete. 2024-03-06T07:20:58.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.684+0000 7fa12218a640 1 Processor -- start 2024-03-06T07:20:58.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.685+0000 7fa12218a640 1 -- start start 2024-03-06T07:20:58.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.685+0000 7fa12218a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11c0bf9d0 0x7fa11c1579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.685+0000 7fa12218a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa11c152650 0x7fa11c1ef1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.685+0000 7fa12218a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa11c14b8c0 0x7fa11c1eb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.685+0000 7fa12218a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c1551b0 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.685+0000 7fa12218a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c0c6fa0 con 0x7fa11c152650 2024-03-06T07:20:58.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.685+0000 7fa12218a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c155490 con 0x7fa11c14b8c0 2024-03-06T07:20:58.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.685+0000 7fa11b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11c0bf9d0 0x7fa11c1579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.685+0000 7fa11b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11c0bf9d0 0x7fa11c1579b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37020/0 (socket says 172.21.15.1:37020) 2024-03-06T07:20:58.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.685+0000 7fa11b7fe640 1 -- 172.21.15.1:0/3605167501 learned_addr learned my addr 172.21.15.1:0/3605167501 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:58.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.686+0000 7fa11bfff640 1 --2- 172.21.15.1:0/3605167501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa11c14b8c0 0x7fa11c1eb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.686+0000 7fa11affd640 1 --2- 172.21.15.1:0/3605167501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa11c152650 0x7fa11c1ef1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.686+0000 7fa11b7fe640 1 -- 172.21.15.1:0/3605167501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa11c14b8c0 msgr2=0x7fa11c1eb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.686+0000 7fa11b7fe640 1 --2- 172.21.15.1:0/3605167501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa11c14b8c0 0x7fa11c1eb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.686+0000 7fa11b7fe640 1 -- 172.21.15.1:0/3605167501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa11c152650 msgr2=0x7fa11c1ef1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.686+0000 7fa11b7fe640 1 --2- 172.21.15.1:0/3605167501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa11c152650 0x7fa11c1ef1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.686+0000 7fa11b7fe640 1 -- 172.21.15.1:0/3605167501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c157f90 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.686+0000 7fa11affd640 1 --2- 172.21.15.1:0/3605167501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa11c152650 0x7fa11c1ef1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:58.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.686+0000 7fa11b7fe640 1 --2- 172.21.15.1:0/3605167501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11c0bf9d0 0x7fa11c1579b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00c4d0 tx=0x7fa10c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.686+0000 7fa103fff640 1 -- 172.21.15.1:0/3605167501 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c018070 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.686+0000 7fa103fff640 1 -- 172.21.15.1:0/3605167501 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10c007500 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.686+0000 7fa12218a640 1 -- 172.21.15.1:0/3605167501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa11c158270 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.686+0000 7fa103fff640 1 -- 172.21.15.1:0/3605167501 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c004a20 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.686+0000 7fa12218a640 1 -- 172.21.15.1:0/3605167501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11c0ca1b0 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.687+0000 7fa12218a640 1 -- 172.21.15.1:0/3605167501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa11c067fd0 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.688+0000 7fa103fff640 1 -- 172.21.15.1:0/3605167501 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa10c01b070 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.688+0000 7fa103fff640 1 --2- 172.21.15.1:0/3605167501 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa10806b4d0 0x7fa10806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.688+0000 7fa103fff640 1 -- 172.21.15.1:0/3605167501 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa10c0897f0 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.689+0000 7fa11affd640 1 --2- 172.21.15.1:0/3605167501 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa10806b4d0 0x7fa10806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.689+0000 7fa11affd640 1 --2- 172.21.15.1:0/3605167501 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa10806b4d0 0x7fa10806d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0cb250 tx=0x7fa110008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.692+0000 7fa103fff640 1 -- 172.21.15.1:0/3605167501 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa10c0583c0 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.929+0000 7fa12218a640 1 -- 172.21.15.1:0/3605167501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa11c0b06f0 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.929+0000 7fa103fff640 1 -- 172.21.15.1:0/3605167501 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa10c057cc0 con 0x7fa11c0bf9d0 2024-03-06T07:20:58.931 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:20:58.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.932+0000 7fa12218a640 1 -- 172.21.15.1:0/3605167501 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa10806b4d0 msgr2=0x7fa10806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.932+0000 7fa12218a640 1 --2- 172.21.15.1:0/3605167501 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa10806b4d0 0x7fa10806d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0cb250 tx=0x7fa110008040 comp rx=0 tx=0).stop 2024-03-06T07:20:58.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.932+0000 7fa12218a640 1 -- 172.21.15.1:0/3605167501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11c0bf9d0 msgr2=0x7fa11c1579b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.932+0000 7fa12218a640 1 --2- 172.21.15.1:0/3605167501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11c0bf9d0 0x7fa11c1579b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00c4d0 tx=0x7fa10c00c990 comp rx=0 tx=0).stop 2024-03-06T07:20:58.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.932+0000 7fa12218a640 1 -- 172.21.15.1:0/3605167501 shutdown_connections 2024-03-06T07:20:58.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.932+0000 7fa12218a640 1 --2- 172.21.15.1:0/3605167501 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa10806b4d0 0x7fa10806d990 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.932+0000 7fa12218a640 1 --2- 172.21.15.1:0/3605167501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa11c14b8c0 0x7fa11c1eb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.932+0000 7fa12218a640 1 --2- 172.21.15.1:0/3605167501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa11c152650 0x7fa11c1ef1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.932+0000 7fa12218a640 1 --2- 172.21.15.1:0/3605167501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11c0bf9d0 0x7fa11c1579b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.932+0000 7fa12218a640 1 -- 172.21.15.1:0/3605167501 >> 172.21.15.1:0/3605167501 conn(0x7fa11c066ee0 msgr2=0x7fa11c0c3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:58.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.932+0000 7fa12218a640 1 -- 172.21.15.1:0/3605167501 shutdown_connections 2024-03-06T07:20:58.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:58.932+0000 7fa12218a640 1 -- 172.21.15.1:0/3605167501 wait complete. 2024-03-06T07:20:58.945 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:59.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.086+0000 7f49e63ac640 1 Processor -- start 2024-03-06T07:20:59.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.086+0000 7f49e63ac640 1 -- start start 2024-03-06T07:20:59.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.087+0000 7f49e63ac640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f49e0152650 0x7f49e0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.087+0000 7f49e63ac640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f49e00bf9d0 0x7f49e00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.087+0000 7f49e63ac640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e00c2fc0 con 0x7f49e00c6b10 2024-03-06T07:20:59.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.087+0000 7f49e63ac640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e0155800 con 0x7f49e0152650 2024-03-06T07:20:59.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.087+0000 7f49e63ac640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e0158da0 con 0x7f49e00bf9d0 2024-03-06T07:20:59.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.087+0000 7f49dffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f49e0152650 0x7f49e0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.087+0000 7f49dffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f49e0152650 0x7f49e0155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37034/0 (socket says 172.21.15.1:37034) 2024-03-06T07:20:59.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.087+0000 7f49dffff640 1 -- 172.21.15.1:0/4208971304 learned_addr learned my addr 172.21.15.1:0/4208971304 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:59.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.088+0000 7f49e4949640 1 --2- 172.21.15.1:0/4208971304 >> v2:172.21.15.98:3300/0 conn(0x7f49e00bf9d0 0x7f49e00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.088+0000 7f49dffff640 1 -- 172.21.15.1:0/4208971304 >> v2:172.21.15.98:3300/0 conn(0x7f49e00bf9d0 msgr2=0x7f49e00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:59.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.088+0000 7f49dffff640 1 --2- 172.21.15.1:0/4208971304 >> v2:172.21.15.98:3300/0 conn(0x7f49e00bf9d0 0x7f49e00c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.088+0000 7f49dffff640 1 -- 172.21.15.1:0/4208971304 >> v1:172.21.15.98:6789/0 conn(0x7f49e00c6b10 legacy=0x7f49e0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:59.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.088+0000 7f49dffff640 1 -- 172.21.15.1:0/4208971304 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e00c6690 con 0x7f49e0152650 2024-03-06T07:20:59.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.088+0000 7f49dffff640 1 --2- 172.21.15.1:0/4208971304 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e0152650 0x7f49e0155c90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f49e01597b0 tx=0x7f49d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee0a6bd8245ebd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.089+0000 7f49deffd640 1 -- 172.21.15.1:0/4208971304 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d4035020 con 0x7f49e0152650 2024-03-06T07:20:59.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.089+0000 7f49deffd640 1 -- 172.21.15.1:0/4208971304 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d4031870 con 0x7f49e0152650 2024-03-06T07:20:59.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.089+0000 7f49deffd640 1 -- 172.21.15.1:0/4208971304 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d4031c40 con 0x7f49e0152650 2024-03-06T07:20:59.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.089+0000 7f49e63ac640 1 -- 172.21.15.1:0/4208971304 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e0152650 msgr2=0x7f49e0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.089+0000 7f49e63ac640 1 --2- 172.21.15.1:0/4208971304 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e0152650 0x7f49e0155c90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f49e01597b0 tx=0x7f49d402b910 comp rx=0 tx=0).stop 2024-03-06T07:20:59.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.089+0000 7f49e63ac640 1 -- 172.21.15.1:0/4208971304 shutdown_connections 2024-03-06T07:20:59.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.089+0000 7f49e63ac640 1 --2- 172.21.15.1:0/4208971304 >> v2:172.21.15.98:3300/0 conn(0x7f49e00bf9d0 0x7f49e00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.089+0000 7f49e63ac640 1 --2- 172.21.15.1:0/4208971304 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e0152650 0x7f49e0155c90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.089+0000 7f49e63ac640 1 -- 172.21.15.1:0/4208971304 >> 172.21.15.1:0/4208971304 conn(0x7f49e0066ee0 msgr2=0x7f49e00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:59.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.089+0000 7f49e63ac640 1 -- 172.21.15.1:0/4208971304 shutdown_connections 2024-03-06T07:20:59.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.089+0000 7f49e63ac640 1 -- 172.21.15.1:0/4208971304 wait complete. 2024-03-06T07:20:59.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.090+0000 7f49e63ac640 1 Processor -- start 2024-03-06T07:20:59.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.090+0000 7f49e63ac640 1 -- start start 2024-03-06T07:20:59.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.091+0000 7f49e63ac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49e00bf9d0 0x7f49e0158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.091+0000 7f49e63ac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e0152650 0x7f49e014ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.091+0000 7f49e63ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f49e014b360 0x7f49e01ee1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.091+0000 7f49e63ac640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e01551b0 con 0x7f49e0152650 2024-03-06T07:20:59.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.091+0000 7f49e63ac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e00c6fa0 con 0x7f49e00bf9d0 2024-03-06T07:20:59.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.091+0000 7f49e63ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e0155490 con 0x7f49e014b360 2024-03-06T07:20:59.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.091+0000 7f49df7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e0152650 0x7f49e014ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.091+0000 7f49df7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e0152650 0x7f49e014ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37036/0 (socket says 172.21.15.1:37036) 2024-03-06T07:20:59.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.091+0000 7f49df7fe640 1 -- 172.21.15.1:0/2006800526 learned_addr learned my addr 172.21.15.1:0/2006800526 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:59.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.091+0000 7f49e4949640 1 --2- 172.21.15.1:0/2006800526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f49e014b360 0x7f49e01ee1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.091+0000 7f49dffff640 1 --2- 172.21.15.1:0/2006800526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49e00bf9d0 0x7f49e0158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.092+0000 7f49df7fe640 1 -- 172.21.15.1:0/2006800526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f49e014b360 msgr2=0x7f49e01ee1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.092+0000 7f49df7fe640 1 --2- 172.21.15.1:0/2006800526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f49e014b360 0x7f49e01ee1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.092+0000 7f49df7fe640 1 -- 172.21.15.1:0/2006800526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49e00bf9d0 msgr2=0x7f49e0158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.092+0000 7f49df7fe640 1 --2- 172.21.15.1:0/2006800526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49e00bf9d0 0x7f49e0158790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.092+0000 7f49df7fe640 1 -- 172.21.15.1:0/2006800526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e0156e90 con 0x7f49e0152650 2024-03-06T07:20:59.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.092+0000 7f49df7fe640 1 --2- 172.21.15.1:0/2006800526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e0152650 0x7f49e014ae20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f49d000c400 tx=0x7f49d000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.092+0000 7f49dcff9640 1 -- 172.21.15.1:0/2006800526 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d00052d0 con 0x7f49e0152650 2024-03-06T07:20:59.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.092+0000 7f49dcff9640 1 -- 172.21.15.1:0/2006800526 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d0005490 con 0x7f49e0152650 2024-03-06T07:20:59.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.092+0000 7f49e63ac640 1 -- 172.21.15.1:0/2006800526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e0157170 con 0x7f49e0152650 2024-03-06T07:20:59.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.092+0000 7f49dcff9640 1 -- 172.21.15.1:0/2006800526 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0007260 con 0x7f49e0152650 2024-03-06T07:20:59.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.092+0000 7f49e63ac640 1 -- 172.21.15.1:0/2006800526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e0151d80 con 0x7f49e0152650 2024-03-06T07:20:59.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.093+0000 7f49e63ac640 1 -- 172.21.15.1:0/2006800526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e0067fd0 con 0x7f49e0152650 2024-03-06T07:20:59.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.093+0000 7f49dcff9640 1 -- 172.21.15.1:0/2006800526 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f49d0007400 con 0x7f49e0152650 2024-03-06T07:20:59.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.094+0000 7f49dcff9640 1 --2- 172.21.15.1:0/2006800526 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f49cc06b500 0x7f49cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.094+0000 7f49dcff9640 1 -- 172.21.15.1:0/2006800526 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f49d0089a60 con 0x7f49e0152650 2024-03-06T07:20:59.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.094+0000 7f49dffff640 1 --2- 172.21.15.1:0/2006800526 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f49cc06b500 0x7f49cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.094+0000 7f49dffff640 1 --2- 172.21.15.1:0/2006800526 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f49cc06b500 0x7f49cc06d9c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f49d4036000 tx=0x7f49d403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.098+0000 7f49dcff9640 1 -- 172.21.15.1:0/2006800526 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49d0057e20 con 0x7f49e0152650 2024-03-06T07:20:59.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.284+0000 7f49e63ac640 1 -- 172.21.15.1:0/2006800526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49e00b06f0 con 0x7f49e0152650 2024-03-06T07:20:59.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.286+0000 7f49dcff9640 1 -- 172.21.15.1:0/2006800526 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7f49d005bec0 con 0x7f49e0152650 2024-03-06T07:20:59.288 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:20:59.288 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:59.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.288+0000 7f49e63ac640 1 -- 172.21.15.1:0/2006800526 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f49cc06b500 msgr2=0x7f49cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.288+0000 7f49e63ac640 1 --2- 172.21.15.1:0/2006800526 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f49cc06b500 0x7f49cc06d9c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f49d4036000 tx=0x7f49d403c040 comp rx=0 tx=0).stop 2024-03-06T07:20:59.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.288+0000 7f49e63ac640 1 -- 172.21.15.1:0/2006800526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e0152650 msgr2=0x7f49e014ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.288+0000 7f49e63ac640 1 --2- 172.21.15.1:0/2006800526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e0152650 0x7f49e014ae20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f49d000c400 tx=0x7f49d000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.288+0000 7f49e63ac640 1 -- 172.21.15.1:0/2006800526 shutdown_connections 2024-03-06T07:20:59.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.288+0000 7f49e63ac640 1 --2- 172.21.15.1:0/2006800526 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f49cc06b500 0x7f49cc06d9c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.288+0000 7f49e63ac640 1 --2- 172.21.15.1:0/2006800526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f49e014b360 0x7f49e01ee1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.288+0000 7f49e63ac640 1 --2- 172.21.15.1:0/2006800526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e0152650 0x7f49e014ae20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.288+0000 7f49e63ac640 1 --2- 172.21.15.1:0/2006800526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49e00bf9d0 0x7f49e0158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.288+0000 7f49e63ac640 1 -- 172.21.15.1:0/2006800526 >> 172.21.15.1:0/2006800526 conn(0x7f49e0066ee0 msgr2=0x7f49e00c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:59.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.288+0000 7f49e63ac640 1 -- 172.21.15.1:0/2006800526 shutdown_connections 2024-03-06T07:20:59.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.288+0000 7f49e63ac640 1 -- 172.21.15.1:0/2006800526 wait complete. 2024-03-06T07:20:59.301 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:20:19.337560+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:20:20.172791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:20:23.884071+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:20:59.302 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:20:59.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.404+0000 7f9bd3f02640 1 Processor -- start 2024-03-06T07:20:59.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.404+0000 7f9bd3f02640 1 -- start start 2024-03-06T07:20:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.404+0000 7f9bd3f02640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9bcc152840 0x7f9bcc0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.404+0000 7f9bd3f02640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9bcc0c60b0 0x7f9bcc1553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.404+0000 7f9bd3f02640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc0c5e40 con 0x7f9bcc0c3970 2024-03-06T07:20:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.404+0000 7f9bd3f02640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bcc0c67f0 con 0x7f9bcc152840 2024-03-06T07:20:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.404+0000 7f9bd3f02640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bcc0c6a40 con 0x7f9bcc0c60b0 2024-03-06T07:20:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.405+0000 7f9bd149d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9bcc152840 0x7f9bcc0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.405+0000 7f9bd1c9e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9bcc0c3970 0x7f9bcc152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36348/0 (socket says 172.21.15.1:36348) 2024-03-06T07:20:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.405+0000 7f9bd1c9e640 1 -- 172.21.15.1:0/3881412113 learned_addr learned my addr 172.21.15.1:0/3881412113 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.405+0000 7f9bd149d640 1 -- 172.21.15.1:0/3881412113 >> v2:172.21.15.98:3300/0 conn(0x7f9bcc0c60b0 msgr2=0x7f9bcc1553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.405+0000 7f9bd249f640 1 --2- 172.21.15.1:0/3881412113 >> v2:172.21.15.98:3300/0 conn(0x7f9bcc0c60b0 0x7f9bcc1553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.405+0000 7f9bd149d640 1 --2- 172.21.15.1:0/3881412113 >> v2:172.21.15.98:3300/0 conn(0x7f9bcc0c60b0 0x7f9bcc1553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.405+0000 7f9bd149d640 1 -- 172.21.15.1:0/3881412113 >> v1:172.21.15.1:6789/0 conn(0x7f9bcc0c3970 legacy=0x7f9bcc152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.405+0000 7f9bd0c9c640 1 -- 172.21.15.1:0/3881412113 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663970707 0 0) 0x7f9bcc0c5e40 con 0x7f9bcc0c3970 2024-03-06T07:20:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.405+0000 7f9bd149d640 1 -- 172.21.15.1:0/3881412113 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc0c6d20 con 0x7f9bcc152840 2024-03-06T07:20:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.405+0000 7f9bd249f640 1 --2- 172.21.15.1:0/3881412113 >> v2:172.21.15.98:3300/0 conn(0x7f9bcc0c60b0 0x7f9bcc1553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:59.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.406+0000 7f9bd149d640 1 --2- 172.21.15.1:0/3881412113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc152840 0x7f9bcc0c8170 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc0097a0 tx=0x7f9bbc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966339fb3f5cdb01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.406+0000 7f9bd0c9c640 1 -- 172.21.15.1:0/3881412113 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc035020 con 0x7f9bcc152840 2024-03-06T07:20:59.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.406+0000 7f9bd0c9c640 1 -- 172.21.15.1:0/3881412113 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bbc031830 con 0x7f9bcc152840 2024-03-06T07:20:59.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.406+0000 7f9bd0c9c640 1 -- 172.21.15.1:0/3881412113 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc031c00 con 0x7f9bcc152840 2024-03-06T07:20:59.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.406+0000 7f9bd3f02640 1 -- 172.21.15.1:0/3881412113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc152840 msgr2=0x7f9bcc0c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.406+0000 7f9bd3f02640 1 --2- 172.21.15.1:0/3881412113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc152840 0x7f9bcc0c8170 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc0097a0 tx=0x7f9bbc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.406+0000 7f9bd3f02640 1 -- 172.21.15.1:0/3881412113 shutdown_connections 2024-03-06T07:20:59.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.406+0000 7f9bd3f02640 1 --2- 172.21.15.1:0/3881412113 >> v2:172.21.15.98:3300/0 conn(0x7f9bcc0c60b0 0x7f9bcc1553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.406+0000 7f9bd3f02640 1 --2- 172.21.15.1:0/3881412113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc152840 0x7f9bcc0c8170 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.406+0000 7f9bd3f02640 1 -- 172.21.15.1:0/3881412113 >> 172.21.15.1:0/3881412113 conn(0x7f9bcc0c88f0 msgr2=0x7f9bcc0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:59.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.407+0000 7f9bd3f02640 1 -- 172.21.15.1:0/3881412113 shutdown_connections 2024-03-06T07:20:59.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.407+0000 7f9bd3f02640 1 -- 172.21.15.1:0/3881412113 wait complete. 2024-03-06T07:20:59.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.408+0000 7f9bd3f02640 1 Processor -- start 2024-03-06T07:20:59.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.408+0000 7f9bd3f02640 1 -- start start 2024-03-06T07:20:59.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.408+0000 7f9bd3f02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc0c3970 0x7f9bcc121a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.408+0000 7f9bd3f02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bcc0c60b0 0x7f9bcc0c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.408+0000 7f9bd3f02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9bcc152840 0x7f9bcc1eab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.408+0000 7f9bd3f02640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0c75c0 con 0x7f9bcc0c3970 2024-03-06T07:20:59.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.408+0000 7f9bd3f02640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0c72e0 con 0x7f9bcc0c60b0 2024-03-06T07:20:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.408+0000 7f9bd3f02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0c3d50 con 0x7f9bcc152840 2024-03-06T07:20:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.408+0000 7f9bd1c9e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc0c3970 0x7f9bcc121a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.408+0000 7f9bd1c9e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc0c3970 0x7f9bcc121a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37048/0 (socket says 172.21.15.1:37048) 2024-03-06T07:20:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.408+0000 7f9bd1c9e640 1 -- 172.21.15.1:0/1891911370 learned_addr learned my addr 172.21.15.1:0/1891911370 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.409+0000 7f9bd149d640 1 --2- 172.21.15.1:0/1891911370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bcc0c60b0 0x7f9bcc0c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.409+0000 7f9bd249f640 1 --2- 172.21.15.1:0/1891911370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9bcc152840 0x7f9bcc1eab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.409+0000 7f9bd1c9e640 1 -- 172.21.15.1:0/1891911370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9bcc152840 msgr2=0x7f9bcc1eab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.409+0000 7f9bd1c9e640 1 --2- 172.21.15.1:0/1891911370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9bcc152840 0x7f9bcc1eab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.409+0000 7f9bd1c9e640 1 -- 172.21.15.1:0/1891911370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bcc0c60b0 msgr2=0x7f9bcc0c1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.409+0000 7f9bd1c9e640 1 --2- 172.21.15.1:0/1891911370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bcc0c60b0 0x7f9bcc0c1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.409+0000 7f9bd1c9e640 1 -- 172.21.15.1:0/1891911370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc0ad8b0 con 0x7f9bcc0c3970 2024-03-06T07:20:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.409+0000 7f9bd1c9e640 1 --2- 172.21.15.1:0/1891911370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc0c3970 0x7f9bcc121a70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800c4d0 tx=0x7f9bc800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.409+0000 7f9bc27fc640 1 -- 172.21.15.1:0/1891911370 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8018070 con 0x7f9bcc0c3970 2024-03-06T07:20:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.409+0000 7f9bc27fc640 1 -- 172.21.15.1:0/1891911370 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc8007500 con 0x7f9bcc0c3970 2024-03-06T07:20:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.409+0000 7f9bc27fc640 1 -- 172.21.15.1:0/1891911370 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8004a20 con 0x7f9bcc0c3970 2024-03-06T07:20:59.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.409+0000 7f9bd3f02640 1 -- 172.21.15.1:0/1891911370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bcc156370 con 0x7f9bcc0c3970 2024-03-06T07:20:59.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.409+0000 7f9bd3f02640 1 -- 172.21.15.1:0/1891911370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bcc1eda40 con 0x7f9bcc0c3970 2024-03-06T07:20:59.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.410+0000 7f9bd3f02640 1 -- 172.21.15.1:0/1891911370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bcc0686e0 con 0x7f9bcc0c3970 2024-03-06T07:20:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.410+0000 7f9bc27fc640 1 -- 172.21.15.1:0/1891911370 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bc8004bc0 con 0x7f9bcc0c3970 2024-03-06T07:20:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.411+0000 7f9bc27fc640 1 --2- 172.21.15.1:0/1891911370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bb006b4d0 0x7f9bb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.411+0000 7f9bc27fc640 1 -- 172.21.15.1:0/1891911370 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9bc8089830 con 0x7f9bcc0c3970 2024-03-06T07:20:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.411+0000 7f9bd149d640 1 --2- 172.21.15.1:0/1891911370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bb006b4d0 0x7f9bb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.412+0000 7f9bd149d640 1 --2- 172.21.15.1:0/1891911370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bb006b4d0 0x7f9bb006d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc036000 tx=0x7f9bbc008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.415+0000 7f9bc27fc640 1 -- 172.21.15.1:0/1891911370 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9bc8058400 con 0x7f9bcc0c3970 2024-03-06T07:20:59.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.669+0000 7f9bd3f02640 1 -- 172.21.15.1:0/1891911370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9bcc0c5840 con 0x7f9bcc0c3970 2024-03-06T07:20:59.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.669+0000 7f9bc27fc640 1 -- 172.21.15.1:0/1891911370 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9bc8057d00 con 0x7f9bcc0c3970 2024-03-06T07:20:59.671 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:20:59.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.671+0000 7f9bd3f02640 1 -- 172.21.15.1:0/1891911370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bb006b4d0 msgr2=0x7f9bb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.671+0000 7f9bd3f02640 1 --2- 172.21.15.1:0/1891911370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bb006b4d0 0x7f9bb006d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc036000 tx=0x7f9bbc008030 comp rx=0 tx=0).stop 2024-03-06T07:20:59.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.671+0000 7f9bd3f02640 1 -- 172.21.15.1:0/1891911370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc0c3970 msgr2=0x7f9bcc121a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.671+0000 7f9bd3f02640 1 --2- 172.21.15.1:0/1891911370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc0c3970 0x7f9bcc121a70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800c4d0 tx=0x7f9bc800c990 comp rx=0 tx=0).stop 2024-03-06T07:20:59.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.671+0000 7f9bd3f02640 1 -- 172.21.15.1:0/1891911370 shutdown_connections 2024-03-06T07:20:59.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.671+0000 7f9bd3f02640 1 --2- 172.21.15.1:0/1891911370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bb006b4d0 0x7f9bb006d990 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.671+0000 7f9bd3f02640 1 --2- 172.21.15.1:0/1891911370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9bcc152840 0x7f9bcc1eab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.671+0000 7f9bd3f02640 1 --2- 172.21.15.1:0/1891911370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bcc0c60b0 0x7f9bcc0c1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.671+0000 7f9bd3f02640 1 --2- 172.21.15.1:0/1891911370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc0c3970 0x7f9bcc121a70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.671+0000 7f9bd3f02640 1 -- 172.21.15.1:0/1891911370 >> 172.21.15.1:0/1891911370 conn(0x7f9bcc0c88f0 msgr2=0x7f9bcc152070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:59.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.671+0000 7f9bd3f02640 1 -- 172.21.15.1:0/1891911370 shutdown_connections 2024-03-06T07:20:59.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.671+0000 7f9bd3f02640 1 -- 172.21.15.1:0/1891911370 wait complete. 2024-03-06T07:20:59.684 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:20:59.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.787+0000 7ff8e975e640 1 Processor -- start 2024-03-06T07:20:59.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.787+0000 7ff8e975e640 1 -- start start 2024-03-06T07:20:59.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.788+0000 7ff8e975e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff8e4072510 0x7ff8e40729a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.788+0000 7ff8e975e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff8e41568e0 0x7ff8e4156d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.788+0000 7ff8e975e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e41533f0 con 0x7ff8e4150500 2024-03-06T07:20:59.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.788+0000 7ff8e975e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e4154000 con 0x7ff8e41568e0 2024-03-06T07:20:59.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.788+0000 7ff8e975e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e406b110 con 0x7ff8e4072510 2024-03-06T07:20:59.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.788+0000 7ff8e2ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff8e4150500 0x7ff8e40c6d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36354/0 (socket says 172.21.15.1:36354) 2024-03-06T07:20:59.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.788+0000 7ff8e2ffd640 1 -- 172.21.15.1:0/492151569 learned_addr learned my addr 172.21.15.1:0/492151569 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:20:59.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.788+0000 7ff8e37fe640 1 --2- 172.21.15.1:0/492151569 >> v2:172.21.15.1:3300/0 conn(0x7ff8e41568e0 0x7ff8e4156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.788+0000 7ff8e27fc640 1 -- 172.21.15.1:0/492151569 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323392322 0 0) 0x7ff8e41533f0 con 0x7ff8e4150500 2024-03-06T07:20:59.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.788+0000 7ff8e27fc640 1 -- 172.21.15.1:0/492151569 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8d0003520 con 0x7ff8e4150500 2024-03-06T07:20:59.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.788+0000 7ff8da7fc640 1 --2- 172.21.15.1:0/492151569 >> v2:172.21.15.98:3300/0 conn(0x7ff8e4072510 0x7ff8e40729a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.788+0000 7ff8e37fe640 1 -- 172.21.15.1:0/492151569 >> v2:172.21.15.98:3300/0 conn(0x7ff8e4072510 msgr2=0x7ff8e40729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:59.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.788+0000 7ff8e37fe640 1 --2- 172.21.15.1:0/492151569 >> v2:172.21.15.98:3300/0 conn(0x7ff8e4072510 0x7ff8e40729a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.788+0000 7ff8e37fe640 1 -- 172.21.15.1:0/492151569 >> v1:172.21.15.1:6789/0 conn(0x7ff8e4150500 legacy=0x7ff8e40c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.789+0000 7ff8e37fe640 1 -- 172.21.15.1:0/492151569 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e41536d0 con 0x7ff8e41568e0 2024-03-06T07:20:59.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.789+0000 7ff8e37fe640 1 --2- 172.21.15.1:0/492151569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8e41568e0 0x7ff8e4156d10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff8e406e3b0 tx=0x7ff8dc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341f852ab2423832 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.789+0000 7ff8e27fc640 1 -- 172.21.15.1:0/492151569 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc039070 con 0x7ff8e41568e0 2024-03-06T07:20:59.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.789+0000 7ff8e27fc640 1 -- 172.21.15.1:0/492151569 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8dc02cbf0 con 0x7ff8e41568e0 2024-03-06T07:20:59.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.790+0000 7ff8e27fc640 1 -- 172.21.15.1:0/492151569 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc034440 con 0x7ff8e41568e0 2024-03-06T07:20:59.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.790+0000 7ff8e975e640 1 -- 172.21.15.1:0/492151569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8e41568e0 msgr2=0x7ff8e4156d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.790+0000 7ff8e975e640 1 --2- 172.21.15.1:0/492151569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8e41568e0 0x7ff8e4156d10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff8e406e3b0 tx=0x7ff8dc02be40 comp rx=0 tx=0).stop 2024-03-06T07:20:59.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.790+0000 7ff8e975e640 1 -- 172.21.15.1:0/492151569 shutdown_connections 2024-03-06T07:20:59.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.790+0000 7ff8e975e640 1 --2- 172.21.15.1:0/492151569 >> v2:172.21.15.98:3300/0 conn(0x7ff8e4072510 0x7ff8e40729a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.790+0000 7ff8e975e640 1 --2- 172.21.15.1:0/492151569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8e41568e0 0x7ff8e4156d10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.790+0000 7ff8e975e640 1 -- 172.21.15.1:0/492151569 >> 172.21.15.1:0/492151569 conn(0x7ff8e40a9700 msgr2=0x7ff8e40a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:59.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.790+0000 7ff8e975e640 1 -- 172.21.15.1:0/492151569 shutdown_connections 2024-03-06T07:20:59.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.790+0000 7ff8e975e640 1 -- 172.21.15.1:0/492151569 wait complete. 2024-03-06T07:20:59.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.791+0000 7ff8e975e640 1 Processor -- start 2024-03-06T07:20:59.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.791+0000 7ff8e975e640 1 -- start start 2024-03-06T07:20:59.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.792+0000 7ff8e975e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8e4072510 0x7ff8e40c5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.792+0000 7ff8e975e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8e4150500 0x7ff8e414aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.792+0000 7ff8e975e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8e41568e0 0x7ff8e41ede70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.792+0000 7ff8e975e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e4153010 con 0x7ff8e4150500 2024-03-06T07:20:59.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.792+0000 7ff8e975e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e4153c90 con 0x7ff8e4072510 2024-03-06T07:20:59.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.792+0000 7ff8e975e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e4153190 con 0x7ff8e41568e0 2024-03-06T07:20:59.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.792+0000 7ff8da7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8e4150500 0x7ff8e414aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.792+0000 7ff8da7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8e4150500 0x7ff8e414aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37074/0 (socket says 172.21.15.1:37074) 2024-03-06T07:20:59.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.792+0000 7ff8da7fc640 1 -- 172.21.15.1:0/707390130 learned_addr learned my addr 172.21.15.1:0/707390130 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:20:59.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.792+0000 7ff8da7fc640 1 -- 172.21.15.1:0/707390130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8e41568e0 msgr2=0x7ff8e41ede70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.792+0000 7ff8da7fc640 1 --2- 172.21.15.1:0/707390130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8e41568e0 0x7ff8e41ede70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.792+0000 7ff8da7fc640 1 -- 172.21.15.1:0/707390130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8e4072510 msgr2=0x7ff8e40c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.792+0000 7ff8da7fc640 1 --2- 172.21.15.1:0/707390130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8e4072510 0x7ff8e40c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.792+0000 7ff8da7fc640 1 -- 172.21.15.1:0/707390130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e40c48e0 con 0x7ff8e4150500 2024-03-06T07:20:59.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.792+0000 7ff8da7fc640 1 --2- 172.21.15.1:0/707390130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8e4150500 0x7ff8e414aec0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc00c820 tx=0x7ff8cc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.793+0000 7ff8dbfff640 1 -- 172.21.15.1:0/707390130 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc015070 con 0x7ff8e4150500 2024-03-06T07:20:59.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.793+0000 7ff8dbfff640 1 -- 172.21.15.1:0/707390130 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8cc004240 con 0x7ff8e4150500 2024-03-06T07:20:59.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.793+0000 7ff8e975e640 1 -- 172.21.15.1:0/707390130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e41eb4a0 con 0x7ff8e4150500 2024-03-06T07:20:59.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.793+0000 7ff8dbfff640 1 -- 172.21.15.1:0/707390130 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc00ce10 con 0x7ff8e4150500 2024-03-06T07:20:59.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.793+0000 7ff8e975e640 1 -- 172.21.15.1:0/707390130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e406fe50 con 0x7ff8e4150500 2024-03-06T07:20:59.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.794+0000 7ff8e975e640 1 -- 172.21.15.1:0/707390130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e40686e0 con 0x7ff8e4150500 2024-03-06T07:20:59.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.794+0000 7ff8dbfff640 1 -- 172.21.15.1:0/707390130 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8cc007830 con 0x7ff8e4150500 2024-03-06T07:20:59.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.794+0000 7ff8dbfff640 1 --2- 172.21.15.1:0/707390130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff8c406b4d0 0x7ff8c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.794+0000 7ff8dbfff640 1 -- 172.21.15.1:0/707390130 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff8cc08a420 con 0x7ff8e4150500 2024-03-06T07:20:59.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.795+0000 7ff8e2ffd640 1 --2- 172.21.15.1:0/707390130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff8c406b4d0 0x7ff8c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.795+0000 7ff8e2ffd640 1 --2- 172.21.15.1:0/707390130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff8c406b4d0 0x7ff8c406d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4000f60 tx=0x7ff8d40078f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:20:59.799+0000 7ff8dbfff640 1 -- 172.21.15.1:0/707390130 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8cc058f60 con 0x7ff8e4150500 2024-03-06T07:21:00.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.008+0000 7ff8e975e640 1 -- 172.21.15.1:0/707390130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff8e40c6b70 con 0x7ff8e4150500 2024-03-06T07:21:00.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.008+0000 7ff8dbfff640 1 -- 172.21.15.1:0/707390130 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff8cc058860 con 0x7ff8e4150500 2024-03-06T07:21:00.010 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:21:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.011+0000 7ff8e975e640 1 -- 172.21.15.1:0/707390130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff8c406b4d0 msgr2=0x7ff8c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.011+0000 7ff8e975e640 1 --2- 172.21.15.1:0/707390130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff8c406b4d0 0x7ff8c406d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4000f60 tx=0x7ff8d40078f0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.011+0000 7ff8e975e640 1 -- 172.21.15.1:0/707390130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8e4150500 msgr2=0x7ff8e414aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.011+0000 7ff8e975e640 1 --2- 172.21.15.1:0/707390130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8e4150500 0x7ff8e414aec0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc00c820 tx=0x7ff8cc00cce0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.011+0000 7ff8e975e640 1 -- 172.21.15.1:0/707390130 shutdown_connections 2024-03-06T07:21:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.011+0000 7ff8e975e640 1 --2- 172.21.15.1:0/707390130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff8c406b4d0 0x7ff8c406d990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.011+0000 7ff8e975e640 1 --2- 172.21.15.1:0/707390130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8e41568e0 0x7ff8e41ede70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.011+0000 7ff8e975e640 1 --2- 172.21.15.1:0/707390130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8e4150500 0x7ff8e414aec0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.011+0000 7ff8e975e640 1 --2- 172.21.15.1:0/707390130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8e4072510 0x7ff8e40c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.011+0000 7ff8e975e640 1 -- 172.21.15.1:0/707390130 >> 172.21.15.1:0/707390130 conn(0x7ff8e40a9700 msgr2=0x7ff8e40c6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:00.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.011+0000 7ff8e975e640 1 -- 172.21.15.1:0/707390130 shutdown_connections 2024-03-06T07:21:00.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.011+0000 7ff8e975e640 1 -- 172.21.15.1:0/707390130 wait complete. 2024-03-06T07:21:00.024 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:21:00.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.129+0000 7f667f765640 1 Processor -- start 2024-03-06T07:21:00.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.129+0000 7f667f765640 1 -- start start 2024-03-06T07:21:00.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.130+0000 7f667f765640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f66780c3600 0x7f6678158d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.130+0000 7f667f765640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f66781592d0 0x7f667806aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.130+0000 7f667f765640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66780c3b50 con 0x7f66780a7050 2024-03-06T07:21:00.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.130+0000 7f667f765640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66780c11d0 con 0x7f66780c3600 2024-03-06T07:21:00.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.130+0000 7f667f765640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6678158620 con 0x7f66781592d0 2024-03-06T07:21:00.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.130+0000 7f667cd00640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f66780c3600 0x7f6678158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.130+0000 7f667cd00640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f66780c3600 0x7f6678158d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37084/0 (socket says 172.21.15.1:37084) 2024-03-06T07:21:00.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.130+0000 7f667cd00640 1 -- 172.21.15.1:0/2464512957 learned_addr learned my addr 172.21.15.1:0/2464512957 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:00.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.131+0000 7f667dd02640 1 --2- 172.21.15.1:0/2464512957 >> v2:172.21.15.98:3300/0 conn(0x7f66781592d0 0x7f667806aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.131+0000 7f667cd00640 1 -- 172.21.15.1:0/2464512957 >> v2:172.21.15.98:3300/0 conn(0x7f66781592d0 msgr2=0x7f667806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:00.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.131+0000 7f667cd00640 1 --2- 172.21.15.1:0/2464512957 >> v2:172.21.15.98:3300/0 conn(0x7f66781592d0 0x7f667806aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.131+0000 7f667cd00640 1 -- 172.21.15.1:0/2464512957 >> v1:172.21.15.98:6789/0 conn(0x7f66780a7050 legacy=0x7f6678155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:00.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.131+0000 7f667cd00640 1 -- 172.21.15.1:0/2464512957 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66780c3e30 con 0x7f66780c3600 2024-03-06T07:21:00.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.131+0000 7f667cd00640 1 --2- 172.21.15.1:0/2464512957 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66780c3600 0x7f6678158d90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f66780c2260 tx=0x7f666802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b80f7b1920bf68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:00.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.132+0000 7f666ffff640 1 -- 172.21.15.1:0/2464512957 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6668035020 con 0x7f66780c3600 2024-03-06T07:21:00.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.132+0000 7f666ffff640 1 -- 172.21.15.1:0/2464512957 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6668031870 con 0x7f66780c3600 2024-03-06T07:21:00.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.132+0000 7f666ffff640 1 -- 172.21.15.1:0/2464512957 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6668031c40 con 0x7f66780c3600 2024-03-06T07:21:00.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.132+0000 7f667f765640 1 -- 172.21.15.1:0/2464512957 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66780c3600 msgr2=0x7f6678158d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.132+0000 7f667f765640 1 --2- 172.21.15.1:0/2464512957 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66780c3600 0x7f6678158d90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f66780c2260 tx=0x7f666802b910 comp rx=0 tx=0).stop 2024-03-06T07:21:00.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.132+0000 7f667f765640 1 -- 172.21.15.1:0/2464512957 shutdown_connections 2024-03-06T07:21:00.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.132+0000 7f667f765640 1 --2- 172.21.15.1:0/2464512957 >> v2:172.21.15.98:3300/0 conn(0x7f66781592d0 0x7f667806aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.132+0000 7f667f765640 1 --2- 172.21.15.1:0/2464512957 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66780c3600 0x7f6678158d90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.132+0000 7f667f765640 1 -- 172.21.15.1:0/2464512957 >> 172.21.15.1:0/2464512957 conn(0x7f66780a9fe0 msgr2=0x7f667806c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:00.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.132+0000 7f667f765640 1 -- 172.21.15.1:0/2464512957 shutdown_connections 2024-03-06T07:21:00.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.132+0000 7f667f765640 1 -- 172.21.15.1:0/2464512957 wait complete. 2024-03-06T07:21:00.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.133+0000 7f667f765640 1 Processor -- start 2024-03-06T07:21:00.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.133+0000 7f667f765640 1 -- start start 2024-03-06T07:21:00.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.134+0000 7f667f765640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66780a7050 0x7f66780aac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.134+0000 7f667f765640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66780c3600 0x7f6678157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.134+0000 7f667f765640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678147d90 0x7f66780acfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.134+0000 7f667f765640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66780a7430 con 0x7f6678147d90 2024-03-06T07:21:00.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.134+0000 7f667f765640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6678158900 con 0x7f66780c3600 2024-03-06T07:21:00.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.134+0000 7f667f765640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66780a75b0 con 0x7f66780a7050 2024-03-06T07:21:00.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.134+0000 7f667dd02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678147d90 0x7f66780acfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.134+0000 7f667dd02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678147d90 0x7f66780acfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37090/0 (socket says 172.21.15.1:37090) 2024-03-06T07:21:00.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.134+0000 7f667dd02640 1 -- 172.21.15.1:0/2412921472 learned_addr learned my addr 172.21.15.1:0/2412921472 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:00.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.134+0000 7f667d501640 1 --2- 172.21.15.1:0/2412921472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66780a7050 0x7f66780aac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.134+0000 7f667cd00640 1 --2- 172.21.15.1:0/2412921472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66780c3600 0x7f6678157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.134+0000 7f667dd02640 1 -- 172.21.15.1:0/2412921472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66780a7050 msgr2=0x7f66780aac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.134+0000 7f667dd02640 1 --2- 172.21.15.1:0/2412921472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66780a7050 0x7f66780aac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.135+0000 7f667dd02640 1 -- 172.21.15.1:0/2412921472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66780c3600 msgr2=0x7f6678157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.135+0000 7f667dd02640 1 --2- 172.21.15.1:0/2412921472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66780c3600 0x7f6678157880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.135+0000 7f667dd02640 1 -- 172.21.15.1:0/2412921472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66780ab380 con 0x7f6678147d90 2024-03-06T07:21:00.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.135+0000 7f667d501640 1 --2- 172.21.15.1:0/2412921472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66780a7050 0x7f66780aac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:00.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.135+0000 7f667cd00640 1 --2- 172.21.15.1:0/2412921472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66780c3600 0x7f6678157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:00.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.135+0000 7f667dd02640 1 --2- 172.21.15.1:0/2412921472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678147d90 0x7f66780acfe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6670007e60 tx=0x7f667000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:00.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.135+0000 7f666dffb640 1 -- 172.21.15.1:0/2412921472 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670014420 con 0x7f6678147d90 2024-03-06T07:21:00.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.135+0000 7f666dffb640 1 -- 172.21.15.1:0/2412921472 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6670019070 con 0x7f6678147d90 2024-03-06T07:21:00.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.135+0000 7f667f765640 1 -- 172.21.15.1:0/2412921472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66781ebb50 con 0x7f6678147d90 2024-03-06T07:21:00.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.135+0000 7f666dffb640 1 -- 172.21.15.1:0/2412921472 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670014820 con 0x7f6678147d90 2024-03-06T07:21:00.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.135+0000 7f667f765640 1 -- 172.21.15.1:0/2412921472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66781ebe30 con 0x7f6678147d90 2024-03-06T07:21:00.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.136+0000 7f667f765640 1 -- 172.21.15.1:0/2412921472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66780686e0 con 0x7f6678147d90 2024-03-06T07:21:00.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.137+0000 7f666dffb640 1 -- 172.21.15.1:0/2412921472 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f66700149c0 con 0x7f6678147d90 2024-03-06T07:21:00.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.137+0000 7f666dffb640 1 --2- 172.21.15.1:0/2412921472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f665c06b4d0 0x7f665c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.137+0000 7f666dffb640 1 -- 172.21.15.1:0/2412921472 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f667008ada0 con 0x7f6678147d90 2024-03-06T07:21:00.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.137+0000 7f667d501640 1 --2- 172.21.15.1:0/2412921472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f665c06b4d0 0x7f665c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.138+0000 7f667d501640 1 --2- 172.21.15.1:0/2412921472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f665c06b4d0 0x7f665c06d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6674003eb0 tx=0x7f6674009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:00.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.141+0000 7f666dffb640 1 -- 172.21.15.1:0/2412921472 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f66700598e0 con 0x7f6678147d90 2024-03-06T07:21:00.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.323+0000 7f667f765640 1 -- 172.21.15.1:0/2412921472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f66780c13b0 con 0x7f6678147d90 2024-03-06T07:21:00.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.324+0000 7f666dffb640 1 -- 172.21.15.1:0/2412921472 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f66700591e0 con 0x7f6678147d90 2024-03-06T07:21:00.326 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:21:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.326+0000 7f667f765640 1 -- 172.21.15.1:0/2412921472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f665c06b4d0 msgr2=0x7f665c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.326+0000 7f667f765640 1 --2- 172.21.15.1:0/2412921472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f665c06b4d0 0x7f665c06d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6674003eb0 tx=0x7f6674009040 comp rx=0 tx=0).stop 2024-03-06T07:21:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.326+0000 7f667f765640 1 -- 172.21.15.1:0/2412921472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678147d90 msgr2=0x7f66780acfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.326+0000 7f667f765640 1 --2- 172.21.15.1:0/2412921472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678147d90 0x7f66780acfe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6670007e60 tx=0x7f667000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.326+0000 7f667f765640 1 -- 172.21.15.1:0/2412921472 shutdown_connections 2024-03-06T07:21:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.326+0000 7f667f765640 1 --2- 172.21.15.1:0/2412921472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f665c06b4d0 0x7f665c06d990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.326+0000 7f667f765640 1 --2- 172.21.15.1:0/2412921472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678147d90 0x7f66780acfe0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.326+0000 7f667f765640 1 --2- 172.21.15.1:0/2412921472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66780c3600 0x7f6678157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.326+0000 7f667f765640 1 --2- 172.21.15.1:0/2412921472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66780a7050 0x7f66780aac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.326+0000 7f667f765640 1 -- 172.21.15.1:0/2412921472 >> 172.21.15.1:0/2412921472 conn(0x7f66780a9fe0 msgr2=0x7f6678155080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:00.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.326+0000 7f667f765640 1 -- 172.21.15.1:0/2412921472 shutdown_connections 2024-03-06T07:21:00.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.326+0000 7f667f765640 1 -- 172.21.15.1:0/2412921472 wait complete. 2024-03-06T07:21:00.339 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:21:00.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.443+0000 7f2a67f13640 1 Processor -- start 2024-03-06T07:21:00.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.443+0000 7f2a67f13640 1 -- start start 2024-03-06T07:21:00.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.444+0000 7f2a67f13640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a600c2b30 0x7f2a600724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.444+0000 7f2a67f13640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2a600a5fa0 0x7f2a60072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.444+0000 7f2a67f13640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a60151dd0 con 0x7f2a60152b70 2024-03-06T07:21:00.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.444+0000 7f2a67f13640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a601526e0 con 0x7f2a600a5fa0 2024-03-06T07:21:00.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.444+0000 7f2a67f13640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a60156190 con 0x7f2a600c2b30 2024-03-06T07:21:00.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.444+0000 7f2a664b0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2a600a5fa0 0x7f2a60072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.444+0000 7f2a664b0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2a600a5fa0 0x7f2a60072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37092/0 (socket says 172.21.15.1:37092) 2024-03-06T07:21:00.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.444+0000 7f2a664b0640 1 -- 172.21.15.1:0/1135067953 learned_addr learned my addr 172.21.15.1:0/1135067953 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:00.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.444+0000 7f2a65caf640 1 --2- 172.21.15.1:0/1135067953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a600c2b30 0x7f2a600724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.444+0000 7f2a664b0640 1 -- 172.21.15.1:0/1135067953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a600c2b30 msgr2=0x7f2a600724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:00.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.444+0000 7f2a664b0640 1 --2- 172.21.15.1:0/1135067953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a600c2b30 0x7f2a600724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.444+0000 7f2a664b0640 1 -- 172.21.15.1:0/1135067953 >> v1:172.21.15.98:6789/0 conn(0x7f2a60152b70 legacy=0x7f2a600c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.444+0000 7f2a664b0640 1 -- 172.21.15.1:0/1135067953 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a60152090 con 0x7f2a600a5fa0 2024-03-06T07:21:00.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.445+0000 7f2a664b0640 1 --2- 172.21.15.1:0/1135067953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a600a5fa0 0x7f2a60072a00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2a58009e50 tx=0x7f2a5802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b421f47bf1eee5ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:00.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.445+0000 7f2a64cad640 1 -- 172.21.15.1:0/1135067953 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5802cb00 con 0x7f2a600a5fa0 2024-03-06T07:21:00.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.445+0000 7f2a64cad640 1 -- 172.21.15.1:0/1135067953 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a5802cca0 con 0x7f2a600a5fa0 2024-03-06T07:21:00.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.445+0000 7f2a64cad640 1 -- 172.21.15.1:0/1135067953 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a58034610 con 0x7f2a600a5fa0 2024-03-06T07:21:00.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.446+0000 7f2a67f13640 1 -- 172.21.15.1:0/1135067953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a600a5fa0 msgr2=0x7f2a60072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.446+0000 7f2a67f13640 1 --2- 172.21.15.1:0/1135067953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a600a5fa0 0x7f2a60072a00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2a58009e50 tx=0x7f2a5802bf60 comp rx=0 tx=0).stop 2024-03-06T07:21:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.446+0000 7f2a67f13640 1 -- 172.21.15.1:0/1135067953 shutdown_connections 2024-03-06T07:21:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.446+0000 7f2a67f13640 1 --2- 172.21.15.1:0/1135067953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a600a5fa0 0x7f2a60072a00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.446+0000 7f2a67f13640 1 --2- 172.21.15.1:0/1135067953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a600c2b30 0x7f2a600724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.446+0000 7f2a67f13640 1 -- 172.21.15.1:0/1135067953 >> 172.21.15.1:0/1135067953 conn(0x7f2a600c6860 msgr2=0x7f2a600c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.446+0000 7f2a67f13640 1 -- 172.21.15.1:0/1135067953 shutdown_connections 2024-03-06T07:21:00.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.446+0000 7f2a67f13640 1 -- 172.21.15.1:0/1135067953 wait complete. 2024-03-06T07:21:00.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.448+0000 7f2a67f13640 1 Processor -- start 2024-03-06T07:21:00.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.448+0000 7f2a67f13640 1 -- start start 2024-03-06T07:21:00.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.449+0000 7f2a67f13640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a600a5fa0 0x7f2a6006e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.449+0000 7f2a67f13640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a600c2b30 0x7f2a600c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.449+0000 7f2a67f13640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a60152b70 0x7f2a60147de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.449+0000 7f2a67f13640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a60155f00 con 0x7f2a60152b70 2024-03-06T07:21:00.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.449+0000 7f2a67f13640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a60155d80 con 0x7f2a600a5fa0 2024-03-06T07:21:00.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.449+0000 7f2a67f13640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a600c3a00 con 0x7f2a600c2b30 2024-03-06T07:21:00.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.449+0000 7f2a664b0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a60152b70 0x7f2a60147de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.449+0000 7f2a664b0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a60152b70 0x7f2a60147de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37094/0 (socket says 172.21.15.1:37094) 2024-03-06T07:21:00.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.449+0000 7f2a664b0640 1 -- 172.21.15.1:0/3926912629 learned_addr learned my addr 172.21.15.1:0/3926912629 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:00.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.450+0000 7f2a65caf640 1 --2- 172.21.15.1:0/3926912629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a600a5fa0 0x7f2a6006e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.450+0000 7f2a654ae640 1 --2- 172.21.15.1:0/3926912629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a600c2b30 0x7f2a600c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.450+0000 7f2a664b0640 1 -- 172.21.15.1:0/3926912629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a600c2b30 msgr2=0x7f2a600c4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.450+0000 7f2a664b0640 1 --2- 172.21.15.1:0/3926912629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a600c2b30 0x7f2a600c4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.450+0000 7f2a664b0640 1 -- 172.21.15.1:0/3926912629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a600a5fa0 msgr2=0x7f2a6006e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.450+0000 7f2a664b0640 1 --2- 172.21.15.1:0/3926912629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a600a5fa0 0x7f2a6006e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.450+0000 7f2a664b0640 1 -- 172.21.15.1:0/3926912629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a601dc650 con 0x7f2a60152b70 2024-03-06T07:21:00.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.450+0000 7f2a654ae640 1 --2- 172.21.15.1:0/3926912629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a600c2b30 0x7f2a600c4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:00.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.450+0000 7f2a65caf640 1 --2- 172.21.15.1:0/3926912629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a600a5fa0 0x7f2a6006e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:00.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.450+0000 7f2a664b0640 1 --2- 172.21.15.1:0/3926912629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a60152b70 0x7f2a60147de0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2a58009e50 tx=0x7f2a58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:00.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.450+0000 7f2a567fc640 1 -- 172.21.15.1:0/3926912629 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a58043020 con 0x7f2a60152b70 2024-03-06T07:21:00.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.450+0000 7f2a567fc640 1 -- 172.21.15.1:0/3926912629 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a5802ce50 con 0x7f2a60152b70 2024-03-06T07:21:00.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.450+0000 7f2a67f13640 1 -- 172.21.15.1:0/3926912629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a601dd160 con 0x7f2a60152b70 2024-03-06T07:21:00.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.450+0000 7f2a567fc640 1 -- 172.21.15.1:0/3926912629 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a58009510 con 0x7f2a60152b70 2024-03-06T07:21:00.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.450+0000 7f2a67f13640 1 -- 172.21.15.1:0/3926912629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a6006e710 con 0x7f2a60152b70 2024-03-06T07:21:00.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.451+0000 7f2a67f13640 1 -- 172.21.15.1:0/3926912629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a60067fd0 con 0x7f2a60152b70 2024-03-06T07:21:00.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.452+0000 7f2a567fc640 1 -- 172.21.15.1:0/3926912629 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a58046020 con 0x7f2a60152b70 2024-03-06T07:21:00.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.452+0000 7f2a567fc640 1 --2- 172.21.15.1:0/3926912629 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a4406b4d0 0x7f2a4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.452+0000 7f2a567fc640 1 -- 172.21.15.1:0/3926912629 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2a580acb30 con 0x7f2a60152b70 2024-03-06T07:21:00.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.452+0000 7f2a65caf640 1 --2- 172.21.15.1:0/3926912629 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a4406b4d0 0x7f2a4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.453+0000 7f2a65caf640 1 --2- 172.21.15.1:0/3926912629 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a4406b4d0 0x7f2a4406d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c006fd0 tx=0x7f2a5c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:00.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.457+0000 7f2a567fc640 1 -- 172.21.15.1:0/3926912629 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a5804d030 con 0x7f2a60152b70 2024-03-06T07:21:00.639 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:00.642 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:00.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.668+0000 7f2a67f13640 1 -- 172.21.15.1:0/3926912629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2a600c49a0 con 0x7f2a60152b70 2024-03-06T07:21:00.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.669+0000 7f2a567fc640 1 -- 172.21.15.1:0/3926912629 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2a5807b700 con 0x7f2a60152b70 2024-03-06T07:21:00.671 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:21:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.671+0000 7f2a67f13640 1 -- 172.21.15.1:0/3926912629 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a4406b4d0 msgr2=0x7f2a4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.671+0000 7f2a67f13640 1 --2- 172.21.15.1:0/3926912629 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a4406b4d0 0x7f2a4406d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c006fd0 tx=0x7f2a5c008040 comp rx=0 tx=0).stop 2024-03-06T07:21:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.671+0000 7f2a67f13640 1 -- 172.21.15.1:0/3926912629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a60152b70 msgr2=0x7f2a60147de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.671+0000 7f2a67f13640 1 --2- 172.21.15.1:0/3926912629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a60152b70 0x7f2a60147de0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2a58009e50 tx=0x7f2a58004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.671+0000 7f2a67f13640 1 -- 172.21.15.1:0/3926912629 shutdown_connections 2024-03-06T07:21:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.671+0000 7f2a67f13640 1 --2- 172.21.15.1:0/3926912629 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a4406b4d0 0x7f2a4406d990 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.671+0000 7f2a67f13640 1 --2- 172.21.15.1:0/3926912629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a60152b70 0x7f2a60147de0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.671+0000 7f2a67f13640 1 --2- 172.21.15.1:0/3926912629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a600c2b30 0x7f2a600c4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.671+0000 7f2a67f13640 1 --2- 172.21.15.1:0/3926912629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a600a5fa0 0x7f2a6006e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.671+0000 7f2a67f13640 1 -- 172.21.15.1:0/3926912629 >> 172.21.15.1:0/3926912629 conn(0x7f2a600c6860 msgr2=0x7f2a60150d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.671+0000 7f2a67f13640 1 -- 172.21.15.1:0/3926912629 shutdown_connections 2024-03-06T07:21:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.671+0000 7f2a67f13640 1 -- 172.21.15.1:0/3926912629 wait complete. 2024-03-06T07:21:00.698 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity) That unsetting mds_join_fs will not cause failover if no high-affinity standby exists. 2024-03-06T07:21:00.698 INFO:tasks.cephfs_test_runner:test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-06T07:21:00.698 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:00.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.851+0000 7f882ae15640 1 Processor -- start 2024-03-06T07:21:00.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.851+0000 7f882ae15640 1 -- start start 2024-03-06T07:21:00.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.852+0000 7f882ae15640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8824152650 0x7f8824155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.852+0000 7f882ae15640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f88240c6b10 0x7f8824158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.852+0000 7f882ae15640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88240c2fc0 con 0x7f88240bf9d0 2024-03-06T07:21:00.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.852+0000 7f882ae15640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8824155800 con 0x7f8824152650 2024-03-06T07:21:00.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.852+0000 7f882ae15640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88240c6690 con 0x7f88240c6b10 2024-03-06T07:21:00.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.852+0000 7f8828bb1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8824152650 0x7f8824155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.852+0000 7f8828bb1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8824152650 0x7f8824155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37100/0 (socket says 172.21.15.1:37100) 2024-03-06T07:21:00.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.852+0000 7f8828bb1640 1 -- 172.21.15.1:0/2717939313 learned_addr learned my addr 172.21.15.1:0/2717939313 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:00.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.852+0000 7f88293b2640 1 --1- 172.21.15.1:0/2717939313 >> v1:172.21.15.1:6789/0 conn(0x7f88240bf9d0 0x7f88240c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36362/0 (socket says 172.21.15.1:36362) 2024-03-06T07:21:00.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.852+0000 7f8823fff640 1 --2- 172.21.15.1:0/2717939313 >> v2:172.21.15.98:3300/0 conn(0x7f88240c6b10 0x7f8824158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.852+0000 7f8828bb1640 1 -- 172.21.15.1:0/2717939313 >> v2:172.21.15.98:3300/0 conn(0x7f88240c6b10 msgr2=0x7f8824158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:00.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.852+0000 7f8828bb1640 1 --2- 172.21.15.1:0/2717939313 >> v2:172.21.15.98:3300/0 conn(0x7f88240c6b10 0x7f8824158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.852+0000 7f8828bb1640 1 -- 172.21.15.1:0/2717939313 >> v1:172.21.15.1:6789/0 conn(0x7f88240bf9d0 legacy=0x7f88240c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.852+0000 7f88237fe640 1 -- 172.21.15.1:0/2717939313 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935018348 0 0) 0x7f88240c2fc0 con 0x7f88240bf9d0 2024-03-06T07:21:00.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.852+0000 7f8828bb1640 1 -- 172.21.15.1:0/2717939313 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88240c7280 con 0x7f8824152650 2024-03-06T07:21:00.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.853+0000 7f8828bb1640 1 --2- 172.21.15.1:0/2717939313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8824152650 0x7f8824155c90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f88240cb8e0 tx=0x7f881802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7284fd0327eb6994 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.853+0000 7f88237fe640 1 -- 172.21.15.1:0/2717939313 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8818035020 con 0x7f8824152650 2024-03-06T07:21:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.853+0000 7f88237fe640 1 -- 172.21.15.1:0/2717939313 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8818031870 con 0x7f8824152650 2024-03-06T07:21:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.853+0000 7f88237fe640 1 -- 172.21.15.1:0/2717939313 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8818031c40 con 0x7f8824152650 2024-03-06T07:21:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.854+0000 7f882ae15640 1 -- 172.21.15.1:0/2717939313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8824152650 msgr2=0x7f8824155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.854+0000 7f882ae15640 1 --2- 172.21.15.1:0/2717939313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8824152650 0x7f8824155c90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f88240cb8e0 tx=0x7f881802b910 comp rx=0 tx=0).stop 2024-03-06T07:21:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.854+0000 7f882ae15640 1 -- 172.21.15.1:0/2717939313 shutdown_connections 2024-03-06T07:21:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.854+0000 7f882ae15640 1 --2- 172.21.15.1:0/2717939313 >> v2:172.21.15.98:3300/0 conn(0x7f88240c6b10 0x7f8824158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.854+0000 7f882ae15640 1 --2- 172.21.15.1:0/2717939313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8824152650 0x7f8824155c90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.854+0000 7f882ae15640 1 -- 172.21.15.1:0/2717939313 >> 172.21.15.1:0/2717939313 conn(0x7f8824066ee0 msgr2=0x7f88240672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.854+0000 7f882ae15640 1 -- 172.21.15.1:0/2717939313 shutdown_connections 2024-03-06T07:21:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.854+0000 7f882ae15640 1 -- 172.21.15.1:0/2717939313 wait complete. 2024-03-06T07:21:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.855+0000 7f882ae15640 1 Processor -- start 2024-03-06T07:21:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.855+0000 7f882ae15640 1 -- start start 2024-03-06T07:21:00.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.856+0000 7f882ae15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88240bf9d0 0x7f8824148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.856+0000 7f882ae15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8824152650 0x7f8824149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.856+0000 7f882ae15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f882414a010 0x7f88241edf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.856+0000 7f882ae15640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8824155490 con 0x7f8824152650 2024-03-06T07:21:00.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.856+0000 7f882ae15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88241551b0 con 0x7f882414a010 2024-03-06T07:21:00.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.856+0000 7f882ae15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8824152a30 con 0x7f88240bf9d0 2024-03-06T07:21:00.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.856+0000 7f8823fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8824152650 0x7f8824149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.857+0000 7f8823fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8824152650 0x7f8824149ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37110/0 (socket says 172.21.15.1:37110) 2024-03-06T07:21:00.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.857+0000 7f8823fff640 1 -- 172.21.15.1:0/95210172 learned_addr learned my addr 172.21.15.1:0/95210172 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.857+0000 7f88293b2640 1 --2- 172.21.15.1:0/95210172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f882414a010 0x7f88241edf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.857+0000 7f8828bb1640 1 --2- 172.21.15.1:0/95210172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88240bf9d0 0x7f8824148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.857+0000 7f8823fff640 1 -- 172.21.15.1:0/95210172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88240bf9d0 msgr2=0x7f8824148e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.857+0000 7f8823fff640 1 --2- 172.21.15.1:0/95210172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88240bf9d0 0x7f8824148e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.857+0000 7f8823fff640 1 -- 172.21.15.1:0/95210172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f882414a010 msgr2=0x7f88241edf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.857+0000 7f8823fff640 1 --2- 172.21.15.1:0/95210172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f882414a010 0x7f88241edf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.857+0000 7f8823fff640 1 -- 172.21.15.1:0/95210172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882414a4a0 con 0x7f8824152650 2024-03-06T07:21:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.857+0000 7f8823fff640 1 --2- 172.21.15.1:0/95210172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8824152650 0x7f8824149ad0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f881000d5f0 tx=0x7f881000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.857+0000 7f88217fa640 1 -- 172.21.15.1:0/95210172 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88100125f0 con 0x7f8824152650 2024-03-06T07:21:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.857+0000 7f88217fa640 1 -- 172.21.15.1:0/95210172 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8810019070 con 0x7f8824152650 2024-03-06T07:21:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.857+0000 7f882ae15640 1 -- 172.21.15.1:0/95210172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8824154090 con 0x7f8824152650 2024-03-06T07:21:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.857+0000 7f88217fa640 1 -- 172.21.15.1:0/95210172 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88100129d0 con 0x7f8824152650 2024-03-06T07:21:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.857+0000 7f882ae15640 1 -- 172.21.15.1:0/95210172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88240c9b90 con 0x7f8824152650 2024-03-06T07:21:00.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.859+0000 7f882ae15640 1 -- 172.21.15.1:0/95210172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88241545b0 con 0x7f8824152650 2024-03-06T07:21:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.859+0000 7f88217fa640 1 -- 172.21.15.1:0/95210172 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8810012b70 con 0x7f8824152650 2024-03-06T07:21:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.859+0000 7f88217fa640 1 --2- 172.21.15.1:0/95210172 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f880806b500 0x7f880806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.860+0000 7f88217fa640 1 -- 172.21.15.1:0/95210172 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f881008a730 con 0x7f8824152650 2024-03-06T07:21:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.860+0000 7f8828bb1640 1 --2- 172.21.15.1:0/95210172 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f880806b500 0x7f880806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.860+0000 7f8828bb1640 1 --2- 172.21.15.1:0/95210172 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f880806b500 0x7f880806d9c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8818036000 tx=0x7f8818008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:00.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:00.864+0000 7f88217fa640 1 -- 172.21.15.1:0/95210172 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8810058cb0 con 0x7f8824152650 2024-03-06T07:21:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.094+0000 7f882ae15640 1 -- 172.21.15.1:0/95210172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8824067fd0 con 0x7f8824152650 2024-03-06T07:21:01.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.095+0000 7f88217fa640 1 -- 172.21.15.1:0/95210172 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7f881005cd50 con 0x7f8824152650 2024-03-06T07:21:01.097 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:01.097 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.097+0000 7f882ae15640 1 -- 172.21.15.1:0/95210172 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f880806b500 msgr2=0x7f880806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.097+0000 7f882ae15640 1 --2- 172.21.15.1:0/95210172 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f880806b500 0x7f880806d9c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8818036000 tx=0x7f8818008030 comp rx=0 tx=0).stop 2024-03-06T07:21:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.097+0000 7f882ae15640 1 -- 172.21.15.1:0/95210172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8824152650 msgr2=0x7f8824149ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.097+0000 7f882ae15640 1 --2- 172.21.15.1:0/95210172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8824152650 0x7f8824149ad0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f881000d5f0 tx=0x7f881000dab0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.097+0000 7f882ae15640 1 -- 172.21.15.1:0/95210172 shutdown_connections 2024-03-06T07:21:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.097+0000 7f882ae15640 1 --2- 172.21.15.1:0/95210172 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f880806b500 0x7f880806d9c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.097+0000 7f882ae15640 1 --2- 172.21.15.1:0/95210172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f882414a010 0x7f88241edf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.097+0000 7f882ae15640 1 --2- 172.21.15.1:0/95210172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8824152650 0x7f8824149ad0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.097+0000 7f882ae15640 1 --2- 172.21.15.1:0/95210172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88240bf9d0 0x7f8824148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.097+0000 7f882ae15640 1 -- 172.21.15.1:0/95210172 >> 172.21.15.1:0/95210172 conn(0x7f8824066ee0 msgr2=0x7f88240cabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.097+0000 7f882ae15640 1 -- 172.21.15.1:0/95210172 shutdown_connections 2024-03-06T07:21:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.097+0000 7f882ae15640 1 -- 172.21.15.1:0/95210172 wait complete. 2024-03-06T07:21:01.110 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:20:19.337560+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:20:20.172791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:20:23.884071+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:21:01.111 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.214+0000 7fc6f2a81640 1 Processor -- start 2024-03-06T07:21:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.214+0000 7fc6f2a81640 1 -- start start 2024-03-06T07:21:01.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.215+0000 7fc6f2a81640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc6ec152650 0x7fc6ec155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.215+0000 7fc6f2a81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6ec0bf9d0 0x7fc6ec0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.215+0000 7fc6f2a81640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6ec0c2fc0 con 0x7fc6ec0c6b10 2024-03-06T07:21:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.215+0000 7fc6f2a81640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6ec155800 con 0x7fc6ec152650 2024-03-06T07:21:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.215+0000 7fc6f2a81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec158da0 con 0x7fc6ec0bf9d0 2024-03-06T07:21:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.215+0000 7fc6f081d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc6ec152650 0x7fc6ec155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.215+0000 7fc6f101e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6ec0bf9d0 0x7fc6ec0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.215+0000 7fc6f081d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc6ec152650 0x7fc6ec155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:34292/0 (socket says 172.21.15.1:34292) 2024-03-06T07:21:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.215+0000 7fc6f081d640 1 -- 172.21.15.1:0/458388968 learned_addr learned my addr 172.21.15.1:0/458388968 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.215+0000 7fc6ebfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc6ec0c6b10 0x7fc6ec158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:55364/0 (socket says 172.21.15.1:55364) 2024-03-06T07:21:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.215+0000 7fc6f101e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6ec0bf9d0 0x7fc6ec0c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:34974/0 (socket says 172.21.15.1:34974) 2024-03-06T07:21:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.216+0000 7fc6f081d640 1 -- 172.21.15.1:0/458388968 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6ec0bf9d0 msgr2=0x7fc6ec0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.216+0000 7fc6f081d640 1 --2- 172.21.15.1:0/458388968 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6ec0bf9d0 0x7fc6ec0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.216+0000 7fc6f081d640 1 -- 172.21.15.1:0/458388968 >> v1:172.21.15.98:6789/0 conn(0x7fc6ec0c6b10 legacy=0x7fc6ec158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.216+0000 7fc6f081d640 1 -- 172.21.15.1:0/458388968 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ec0c6690 con 0x7fc6ec152650 2024-03-06T07:21:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.216+0000 7fc6eb7fe640 1 -- 172.21.15.1:0/458388968 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930942116 0 0) 0x7fc6ec0c2fc0 con 0x7fc6ec0c6b10 2024-03-06T07:21:01.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.216+0000 7fc6f101e640 1 --2- 172.21.15.1:0/458388968 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6ec0bf9d0 0x7fc6ec0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.216+0000 7fc6f081d640 1 --2- 172.21.15.1:0/458388968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6ec152650 0x7fc6ec155c90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009870 tx=0x7fc6e002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9db79a780b20bda8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.217+0000 7fc6eb7fe640 1 -- 172.21.15.1:0/458388968 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0035020 con 0x7fc6ec152650 2024-03-06T07:21:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.217+0000 7fc6eb7fe640 1 -- 172.21.15.1:0/458388968 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e0031640 con 0x7fc6ec152650 2024-03-06T07:21:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.217+0000 7fc6eb7fe640 1 -- 172.21.15.1:0/458388968 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e002d040 con 0x7fc6ec152650 2024-03-06T07:21:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.217+0000 7fc6f2a81640 1 -- 172.21.15.1:0/458388968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6ec152650 msgr2=0x7fc6ec155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.217+0000 7fc6f2a81640 1 --2- 172.21.15.1:0/458388968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6ec152650 0x7fc6ec155c90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009870 tx=0x7fc6e002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.217+0000 7fc6f2a81640 1 -- 172.21.15.1:0/458388968 shutdown_connections 2024-03-06T07:21:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.217+0000 7fc6f2a81640 1 --2- 172.21.15.1:0/458388968 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6ec0bf9d0 0x7fc6ec0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.217+0000 7fc6f2a81640 1 --2- 172.21.15.1:0/458388968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6ec152650 0x7fc6ec155c90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.217+0000 7fc6f2a81640 1 -- 172.21.15.1:0/458388968 >> 172.21.15.1:0/458388968 conn(0x7fc6ec066ee0 msgr2=0x7fc6ec0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.217+0000 7fc6f2a81640 1 -- 172.21.15.1:0/458388968 shutdown_connections 2024-03-06T07:21:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.217+0000 7fc6f2a81640 1 -- 172.21.15.1:0/458388968 wait complete. 2024-03-06T07:21:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.219+0000 7fc6f2a81640 1 Processor -- start 2024-03-06T07:21:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.219+0000 7fc6f2a81640 1 -- start start 2024-03-06T07:21:01.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.220+0000 7fc6f2a81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6ec0bf9d0 0x7fc6ec14b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.220+0000 7fc6f2a81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6ec152650 0x7fc6ec150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.220+0000 7fc6f2a81640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec14b020 0x7fc6ec0c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.220+0000 7fc6f2a81640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec1551b0 con 0x7fc6ec14b020 2024-03-06T07:21:01.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.220+0000 7fc6f2a81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec0c6fa0 con 0x7fc6ec152650 2024-03-06T07:21:01.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.220+0000 7fc6f2a81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec155490 con 0x7fc6ec0bf9d0 2024-03-06T07:21:01.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.220+0000 7fc6f101e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec14b020 0x7fc6ec0c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.220+0000 7fc6f101e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec14b020 0x7fc6ec0c9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37112/0 (socket says 172.21.15.1:37112) 2024-03-06T07:21:01.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.220+0000 7fc6f101e640 1 -- 172.21.15.1:0/2219251679 learned_addr learned my addr 172.21.15.1:0/2219251679 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:01.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.220+0000 7fc6ebfff640 1 --2- 172.21.15.1:0/2219251679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6ec152650 0x7fc6ec150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.220+0000 7fc6f081d640 1 --2- 172.21.15.1:0/2219251679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6ec0bf9d0 0x7fc6ec14b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.221+0000 7fc6f101e640 1 -- 172.21.15.1:0/2219251679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6ec0bf9d0 msgr2=0x7fc6ec14b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.221+0000 7fc6f101e640 1 --2- 172.21.15.1:0/2219251679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6ec0bf9d0 0x7fc6ec14b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.221+0000 7fc6f101e640 1 -- 172.21.15.1:0/2219251679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6ec152650 msgr2=0x7fc6ec150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.221+0000 7fc6f101e640 1 --2- 172.21.15.1:0/2219251679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6ec152650 0x7fc6ec150df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.221+0000 7fc6f101e640 1 -- 172.21.15.1:0/2219251679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ec151a20 con 0x7fc6ec14b020 2024-03-06T07:21:01.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.221+0000 7fc6f081d640 1 --2- 172.21.15.1:0/2219251679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6ec0bf9d0 0x7fc6ec14b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:01.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.221+0000 7fc6f101e640 1 --2- 172.21.15.1:0/2219251679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec14b020 0x7fc6ec0c9620 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400efc0 tx=0x7fc6e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:01.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.221+0000 7fc6e97fa640 1 -- 172.21.15.1:0/2219251679 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4018030 con 0x7fc6ec14b020 2024-03-06T07:21:01.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.221+0000 7fc6e97fa640 1 -- 172.21.15.1:0/2219251679 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e400cdf0 con 0x7fc6ec14b020 2024-03-06T07:21:01.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.221+0000 7fc6f2a81640 1 -- 172.21.15.1:0/2219251679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ec151d00 con 0x7fc6ec14b020 2024-03-06T07:21:01.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.221+0000 7fc6e97fa640 1 -- 172.21.15.1:0/2219251679 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4003bb0 con 0x7fc6ec14b020 2024-03-06T07:21:01.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.221+0000 7fc6f2a81640 1 -- 172.21.15.1:0/2219251679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ec0c9b60 con 0x7fc6ec14b020 2024-03-06T07:21:01.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.222+0000 7fc6f2a81640 1 -- 172.21.15.1:0/2219251679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6ec067fd0 con 0x7fc6ec14b020 2024-03-06T07:21:01.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.223+0000 7fc6e97fa640 1 -- 172.21.15.1:0/2219251679 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6e401b070 con 0x7fc6ec14b020 2024-03-06T07:21:01.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.223+0000 7fc6e97fa640 1 --2- 172.21.15.1:0/2219251679 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc6d006b4d0 0x7fc6d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.223+0000 7fc6e97fa640 1 -- 172.21.15.1:0/2219251679 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc6e408ae80 con 0x7fc6ec14b020 2024-03-06T07:21:01.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.224+0000 7fc6f081d640 1 --2- 172.21.15.1:0/2219251679 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc6d006b4d0 0x7fc6d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.224+0000 7fc6f081d640 1 --2- 172.21.15.1:0/2219251679 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc6d006b4d0 0x7fc6d006d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0036000 tx=0x7fc6e003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:01.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.227+0000 7fc6e97fa640 1 -- 172.21.15.1:0/2219251679 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6e4010040 con 0x7fc6ec14b020 2024-03-06T07:21:01.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.408+0000 7fc6f2a81640 1 -- 172.21.15.1:0/2219251679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc6ec158b60 con 0x7fc6ec14b020 2024-03-06T07:21:01.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.408+0000 7fc6e97fa640 1 -- 172.21.15.1:0/2219251679 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc6ec158b60 con 0x7fc6ec14b020 2024-03-06T07:21:01.411 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:21:01.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.410+0000 7fc6f2a81640 1 -- 172.21.15.1:0/2219251679 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc6d006b4d0 msgr2=0x7fc6d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.410+0000 7fc6f2a81640 1 --2- 172.21.15.1:0/2219251679 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc6d006b4d0 0x7fc6d006d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0036000 tx=0x7fc6e003c040 comp rx=0 tx=0).stop 2024-03-06T07:21:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.410+0000 7fc6f2a81640 1 -- 172.21.15.1:0/2219251679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec14b020 msgr2=0x7fc6ec0c9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.410+0000 7fc6f2a81640 1 --2- 172.21.15.1:0/2219251679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec14b020 0x7fc6ec0c9620 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400efc0 tx=0x7fc6e400c490 comp rx=0 tx=0).stop 2024-03-06T07:21:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.410+0000 7fc6f2a81640 1 -- 172.21.15.1:0/2219251679 shutdown_connections 2024-03-06T07:21:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.410+0000 7fc6f2a81640 1 --2- 172.21.15.1:0/2219251679 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc6d006b4d0 0x7fc6d006d990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.410+0000 7fc6f2a81640 1 --2- 172.21.15.1:0/2219251679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec14b020 0x7fc6ec0c9620 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.410+0000 7fc6f2a81640 1 --2- 172.21.15.1:0/2219251679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6ec152650 0x7fc6ec150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.410+0000 7fc6f2a81640 1 --2- 172.21.15.1:0/2219251679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6ec0bf9d0 0x7fc6ec14b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.410+0000 7fc6f2a81640 1 -- 172.21.15.1:0/2219251679 >> 172.21.15.1:0/2219251679 conn(0x7fc6ec066ee0 msgr2=0x7fc6ec0c0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.411+0000 7fc6f2a81640 1 -- 172.21.15.1:0/2219251679 shutdown_connections 2024-03-06T07:21:01.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.411+0000 7fc6f2a81640 1 -- 172.21.15.1:0/2219251679 wait complete. 2024-03-06T07:21:01.423 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:21:01.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.526+0000 7fb74a68d640 1 Processor -- start 2024-03-06T07:21:01.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.526+0000 7fb74a68d640 1 -- start start 2024-03-06T07:21:01.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.527+0000 7fb74a68d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb744152650 0x7fb744155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.527+0000 7fb74a68d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7440bf9d0 0x7fb7440c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.527+0000 7fb74a68d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7440c2fc0 con 0x7fb7440c6b10 2024-03-06T07:21:01.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.527+0000 7fb74a68d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb744155800 con 0x7fb744152650 2024-03-06T07:21:01.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.527+0000 7fb74a68d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb744158da0 con 0x7fb7440bf9d0 2024-03-06T07:21:01.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.527+0000 7fb743fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb744152650 0x7fb744155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.527+0000 7fb743fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb744152650 0x7fb744155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37116/0 (socket says 172.21.15.1:37116) 2024-03-06T07:21:01.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.527+0000 7fb743fff640 1 -- 172.21.15.1:0/1268491143 learned_addr learned my addr 172.21.15.1:0/1268491143 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:01.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.527+0000 7fb743fff640 1 -- 172.21.15.1:0/1268491143 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7440bf9d0 msgr2=0x7fb7440c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:01.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.527+0000 7fb743fff640 1 --2- 172.21.15.1:0/1268491143 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7440bf9d0 0x7fb7440c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.527+0000 7fb743fff640 1 -- 172.21.15.1:0/1268491143 >> v1:172.21.15.98:6789/0 conn(0x7fb7440c6b10 legacy=0x7fb744158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:01.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.527+0000 7fb743fff640 1 -- 172.21.15.1:0/1268491143 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7440c6690 con 0x7fb744152650 2024-03-06T07:21:01.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.528+0000 7fb743fff640 1 --2- 172.21.15.1:0/1268491143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb744152650 0x7fb744155c90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb7440cac90 tx=0x7fb73402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=446477132dfbfb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:01.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.528+0000 7fb742ffd640 1 -- 172.21.15.1:0/1268491143 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7340315a0 con 0x7fb744152650 2024-03-06T07:21:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.528+0000 7fb742ffd640 1 -- 172.21.15.1:0/1268491143 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb734037070 con 0x7fb744152650 2024-03-06T07:21:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.528+0000 7fb742ffd640 1 -- 172.21.15.1:0/1268491143 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb734031960 con 0x7fb744152650 2024-03-06T07:21:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.528+0000 7fb74a68d640 1 -- 172.21.15.1:0/1268491143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb744152650 msgr2=0x7fb744155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.528+0000 7fb74a68d640 1 --2- 172.21.15.1:0/1268491143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb744152650 0x7fb744155c90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb7440cac90 tx=0x7fb73402b840 comp rx=0 tx=0).stop 2024-03-06T07:21:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.528+0000 7fb74a68d640 1 -- 172.21.15.1:0/1268491143 shutdown_connections 2024-03-06T07:21:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.528+0000 7fb74a68d640 1 --2- 172.21.15.1:0/1268491143 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7440bf9d0 0x7fb7440c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.528+0000 7fb74a68d640 1 --2- 172.21.15.1:0/1268491143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb744152650 0x7fb744155c90 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.528+0000 7fb74a68d640 1 -- 172.21.15.1:0/1268491143 >> 172.21.15.1:0/1268491143 conn(0x7fb744066ee0 msgr2=0x7fb7440672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.528+0000 7fb74a68d640 1 -- 172.21.15.1:0/1268491143 shutdown_connections 2024-03-06T07:21:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.529+0000 7fb74a68d640 1 -- 172.21.15.1:0/1268491143 wait complete. 2024-03-06T07:21:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.530+0000 7fb74a68d640 1 Processor -- start 2024-03-06T07:21:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.530+0000 7fb74a68d640 1 -- start start 2024-03-06T07:21:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.530+0000 7fb74a68d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7440bf9d0 0x7fb744158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.530+0000 7fb74a68d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb744152650 0x7fb7441ee3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.530+0000 7fb74a68d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb74414fe70 0x7fb74414af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.530+0000 7fb74a68d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7441551b0 con 0x7fb74414fe70 2024-03-06T07:21:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.530+0000 7fb74a68d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7440c6fa0 con 0x7fb744152650 2024-03-06T07:21:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.530+0000 7fb74a68d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb744155490 con 0x7fb7440bf9d0 2024-03-06T07:21:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.530+0000 7fb748c2a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb74414fe70 0x7fb74414af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.531+0000 7fb748c2a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb74414fe70 0x7fb74414af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37132/0 (socket says 172.21.15.1:37132) 2024-03-06T07:21:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.531+0000 7fb748c2a640 1 -- 172.21.15.1:0/589331873 learned_addr learned my addr 172.21.15.1:0/589331873 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.531+0000 7fb743fff640 1 --2- 172.21.15.1:0/589331873 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7440bf9d0 0x7fb744158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.531+0000 7fb7437fe640 1 --2- 172.21.15.1:0/589331873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb744152650 0x7fb7441ee3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.531+0000 7fb748c2a640 1 -- 172.21.15.1:0/589331873 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7440bf9d0 msgr2=0x7fb744158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.531+0000 7fb748c2a640 1 --2- 172.21.15.1:0/589331873 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7440bf9d0 0x7fb744158790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.531+0000 7fb748c2a640 1 -- 172.21.15.1:0/589331873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb744152650 msgr2=0x7fb7441ee3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.531+0000 7fb748c2a640 1 --2- 172.21.15.1:0/589331873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb744152650 0x7fb7441ee3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.531+0000 7fb748c2a640 1 -- 172.21.15.1:0/589331873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb744156eb0 con 0x7fb74414fe70 2024-03-06T07:21:01.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.531+0000 7fb7437fe640 1 --2- 172.21.15.1:0/589331873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb744152650 0x7fb7441ee3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:01.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.531+0000 7fb748c2a640 1 --2- 172.21.15.1:0/589331873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb74414fe70 0x7fb74414af30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb73c00b970 tx=0x7fb73c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:01.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.531+0000 7fb740ff9640 1 -- 172.21.15.1:0/589331873 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c019070 con 0x7fb74414fe70 2024-03-06T07:21:01.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.531+0000 7fb740ff9640 1 -- 172.21.15.1:0/589331873 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c00cb60 con 0x7fb74414fe70 2024-03-06T07:21:01.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.531+0000 7fb74a68d640 1 -- 172.21.15.1:0/589331873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb744157190 con 0x7fb74414fe70 2024-03-06T07:21:01.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.531+0000 7fb740ff9640 1 -- 172.21.15.1:0/589331873 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c014440 con 0x7fb74414fe70 2024-03-06T07:21:01.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.531+0000 7fb74a68d640 1 -- 172.21.15.1:0/589331873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7441eeee0 con 0x7fb74414fe70 2024-03-06T07:21:01.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.532+0000 7fb74a68d640 1 -- 172.21.15.1:0/589331873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb744067fd0 con 0x7fb74414fe70 2024-03-06T07:21:01.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.533+0000 7fb740ff9640 1 -- 172.21.15.1:0/589331873 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb73c01c070 con 0x7fb74414fe70 2024-03-06T07:21:01.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.533+0000 7fb740ff9640 1 --2- 172.21.15.1:0/589331873 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb73006b4d0 0x7fb73006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.533+0000 7fb740ff9640 1 -- 172.21.15.1:0/589331873 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb73c08a1d0 con 0x7fb74414fe70 2024-03-06T07:21:01.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.533+0000 7fb743fff640 1 --2- 172.21.15.1:0/589331873 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb73006b4d0 0x7fb73006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.534+0000 7fb743fff640 1 --2- 172.21.15.1:0/589331873 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb73006b4d0 0x7fb73006d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb734036000 tx=0x7fb73403d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.537+0000 7fb740ff9640 1 -- 172.21.15.1:0/589331873 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb73c058da0 con 0x7fb74414fe70 2024-03-06T07:21:01.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.717+0000 7fb74a68d640 1 -- 172.21.15.1:0/589331873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb7440b06f0 con 0x7fb74414fe70 2024-03-06T07:21:01.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.718+0000 7fb740ff9640 1 -- 172.21.15.1:0/589331873 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb73c0586a0 con 0x7fb74414fe70 2024-03-06T07:21:01.720 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:21:01.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.719+0000 7fb74a68d640 1 -- 172.21.15.1:0/589331873 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb73006b4d0 msgr2=0x7fb73006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.719+0000 7fb74a68d640 1 --2- 172.21.15.1:0/589331873 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb73006b4d0 0x7fb73006d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb734036000 tx=0x7fb73403d040 comp rx=0 tx=0).stop 2024-03-06T07:21:01.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.719+0000 7fb74a68d640 1 -- 172.21.15.1:0/589331873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb74414fe70 msgr2=0x7fb74414af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.719+0000 7fb74a68d640 1 --2- 172.21.15.1:0/589331873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb74414fe70 0x7fb74414af30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb73c00b970 tx=0x7fb73c00be30 comp rx=0 tx=0).stop 2024-03-06T07:21:01.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.719+0000 7fb74a68d640 1 -- 172.21.15.1:0/589331873 shutdown_connections 2024-03-06T07:21:01.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.719+0000 7fb74a68d640 1 --2- 172.21.15.1:0/589331873 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb73006b4d0 0x7fb73006d990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.719+0000 7fb74a68d640 1 --2- 172.21.15.1:0/589331873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb74414fe70 0x7fb74414af30 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.719+0000 7fb74a68d640 1 --2- 172.21.15.1:0/589331873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb744152650 0x7fb7441ee3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.719+0000 7fb74a68d640 1 --2- 172.21.15.1:0/589331873 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7440bf9d0 0x7fb744158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.719+0000 7fb74a68d640 1 -- 172.21.15.1:0/589331873 >> 172.21.15.1:0/589331873 conn(0x7fb744066ee0 msgr2=0x7fb7440c3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:01.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.720+0000 7fb74a68d640 1 -- 172.21.15.1:0/589331873 shutdown_connections 2024-03-06T07:21:01.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.720+0000 7fb74a68d640 1 -- 172.21.15.1:0/589331873 wait complete. 2024-03-06T07:21:01.735 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:21:01.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.841+0000 7f030a49a640 1 Processor -- start 2024-03-06T07:21:01.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.841+0000 7f030a49a640 1 -- start start 2024-03-06T07:21:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.842+0000 7f030a49a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03040c6b10 0x7f0304155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.842+0000 7f030a49a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f03040bf9d0 0x7f03040c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.842+0000 7f030a49a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03040c2fc0 con 0x7f0304152650 2024-03-06T07:21:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.842+0000 7f030a49a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0304155800 con 0x7f03040bf9d0 2024-03-06T07:21:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.842+0000 7f030a49a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0304158da0 con 0x7f03040c6b10 2024-03-06T07:21:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.842+0000 7f0308a37640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f03040bf9d0 0x7f03040c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.842+0000 7f03037fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03040c6b10 0x7f0304155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.842+0000 7f0308a37640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f03040bf9d0 0x7f03040c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:34318/0 (socket says 172.21.15.1:34318) 2024-03-06T07:21:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.842+0000 7f0308a37640 1 -- 172.21.15.1:0/3209301687 learned_addr learned my addr 172.21.15.1:0/3209301687 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.842+0000 7f0308a37640 1 -- 172.21.15.1:0/3209301687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03040c6b10 msgr2=0x7f0304155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.842+0000 7f0308a37640 1 --2- 172.21.15.1:0/3209301687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03040c6b10 0x7f0304155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.843+0000 7f0308a37640 1 -- 172.21.15.1:0/3209301687 >> v1:172.21.15.98:6789/0 conn(0x7f0304152650 legacy=0x7f0304158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.843+0000 7f0302ffd640 1 -- 172.21.15.1:0/3209301687 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12323814 0 0) 0x7f03040c2fc0 con 0x7f0304152650 2024-03-06T07:21:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.843+0000 7f0308a37640 1 -- 172.21.15.1:0/3209301687 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03040c6690 con 0x7f03040bf9d0 2024-03-06T07:21:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.844+0000 7f0308a37640 1 --2- 172.21.15.1:0/3209301687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03040bf9d0 0x7f03040c1460 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009e50 tx=0x7f02fc02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccdeb8beb0ee4533 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.844+0000 7f0302ffd640 1 -- 172.21.15.1:0/3209301687 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc02ca70 con 0x7f03040bf9d0 2024-03-06T07:21:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.844+0000 7f0302ffd640 1 -- 172.21.15.1:0/3209301687 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc02cc10 con 0x7f03040bf9d0 2024-03-06T07:21:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.844+0000 7f0302ffd640 1 -- 172.21.15.1:0/3209301687 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc0343f0 con 0x7f03040bf9d0 2024-03-06T07:21:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.844+0000 7f030a49a640 1 -- 172.21.15.1:0/3209301687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03040bf9d0 msgr2=0x7f03040c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.844+0000 7f030a49a640 1 --2- 172.21.15.1:0/3209301687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03040bf9d0 0x7f03040c1460 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009e50 tx=0x7f02fc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.844+0000 7f030a49a640 1 -- 172.21.15.1:0/3209301687 shutdown_connections 2024-03-06T07:21:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.844+0000 7f030a49a640 1 --2- 172.21.15.1:0/3209301687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03040bf9d0 0x7f03040c1460 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.844+0000 7f030a49a640 1 --2- 172.21.15.1:0/3209301687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03040c6b10 0x7f0304155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.844+0000 7f030a49a640 1 -- 172.21.15.1:0/3209301687 >> 172.21.15.1:0/3209301687 conn(0x7f0304066ee0 msgr2=0x7f03040672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.845+0000 7f030a49a640 1 -- 172.21.15.1:0/3209301687 shutdown_connections 2024-03-06T07:21:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.845+0000 7f030a49a640 1 -- 172.21.15.1:0/3209301687 wait complete. 2024-03-06T07:21:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.846+0000 7f030a49a640 1 Processor -- start 2024-03-06T07:21:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.846+0000 7f030a49a640 1 -- start start 2024-03-06T07:21:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f030a49a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03040bf9d0 0x7f03041ee0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f030a49a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0304152650 0x7f03041ee630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f030a49a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0304154c30 0x7f030414ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f030a49a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03041551b0 con 0x7f03040bf9d0 2024-03-06T07:21:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f030a49a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03040c6fa0 con 0x7f0304152650 2024-03-06T07:21:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f030a49a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0304155490 con 0x7f0304154c30 2024-03-06T07:21:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f0303fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03040bf9d0 0x7f03041ee0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f03037fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0304152650 0x7f03041ee630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f0303fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03040bf9d0 0x7f03041ee0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37142/0 (socket says 172.21.15.1:37142) 2024-03-06T07:21:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f0308a37640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0304154c30 0x7f030414ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f0303fff640 1 -- 172.21.15.1:0/1916884385 learned_addr learned my addr 172.21.15.1:0/1916884385 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f0303fff640 1 -- 172.21.15.1:0/1916884385 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0304154c30 msgr2=0x7f030414ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f0303fff640 1 --2- 172.21.15.1:0/1916884385 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0304154c30 0x7f030414ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f0303fff640 1 -- 172.21.15.1:0/1916884385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0304152650 msgr2=0x7f03041ee630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f0303fff640 1 --2- 172.21.15.1:0/1916884385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0304152650 0x7f03041ee630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f0303fff640 1 -- 172.21.15.1:0/1916884385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03041e24f0 con 0x7f03040bf9d0 2024-03-06T07:21:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f0303fff640 1 --2- 172.21.15.1:0/1916884385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03040bf9d0 0x7f03041ee0f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f02f400c4e0 tx=0x7f02f400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.847+0000 7f0300ff9640 1 -- 172.21.15.1:0/1916884385 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4017070 con 0x7f03040bf9d0 2024-03-06T07:21:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.848+0000 7f0300ff9640 1 -- 172.21.15.1:0/1916884385 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f40035f0 con 0x7f03040bf9d0 2024-03-06T07:21:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.848+0000 7f030a49a640 1 -- 172.21.15.1:0/1916884385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0304154340 con 0x7f03040bf9d0 2024-03-06T07:21:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.848+0000 7f0300ff9640 1 -- 172.21.15.1:0/1916884385 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f40039e0 con 0x7f03040bf9d0 2024-03-06T07:21:01.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.848+0000 7f030a49a640 1 -- 172.21.15.1:0/1916884385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0304154600 con 0x7f03040bf9d0 2024-03-06T07:21:01.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.849+0000 7f030a49a640 1 -- 172.21.15.1:0/1916884385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0304067fd0 con 0x7f03040bf9d0 2024-03-06T07:21:01.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.849+0000 7f0300ff9640 1 -- 172.21.15.1:0/1916884385 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02f401a070 con 0x7f03040bf9d0 2024-03-06T07:21:01.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.850+0000 7f0300ff9640 1 --2- 172.21.15.1:0/1916884385 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02f006b500 0x7f02f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.850+0000 7f0300ff9640 1 -- 172.21.15.1:0/1916884385 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f02f4089770 con 0x7f03040bf9d0 2024-03-06T07:21:01.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.850+0000 7f03037fe640 1 --2- 172.21.15.1:0/1916884385 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02f006b500 0x7f02f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.850+0000 7f03037fe640 1 --2- 172.21.15.1:0/1916884385 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02f006b500 0x7f02f006d9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0304158d70 tx=0x7f02f8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:01.854+0000 7f0300ff9640 1 -- 172.21.15.1:0/1916884385 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02f4057c70 con 0x7f03040bf9d0 2024-03-06T07:21:02.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.063+0000 7f030a49a640 1 -- 172.21.15.1:0/1916884385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f030414fb30 con 0x7f03040bf9d0 2024-03-06T07:21:02.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.063+0000 7f0300ff9640 1 -- 172.21.15.1:0/1916884385 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f02f405bd10 con 0x7f03040bf9d0 2024-03-06T07:21:02.066 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:21:02.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.065+0000 7f030a49a640 1 -- 172.21.15.1:0/1916884385 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02f006b500 msgr2=0x7f02f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.065+0000 7f030a49a640 1 --2- 172.21.15.1:0/1916884385 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02f006b500 0x7f02f006d9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0304158d70 tx=0x7f02f8009190 comp rx=0 tx=0).stop 2024-03-06T07:21:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.065+0000 7f030a49a640 1 -- 172.21.15.1:0/1916884385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03040bf9d0 msgr2=0x7f03041ee0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.065+0000 7f030a49a640 1 --2- 172.21.15.1:0/1916884385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03040bf9d0 0x7f03041ee0f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f02f400c4e0 tx=0x7f02f400c9a0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.065+0000 7f030a49a640 1 -- 172.21.15.1:0/1916884385 shutdown_connections 2024-03-06T07:21:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.065+0000 7f030a49a640 1 --2- 172.21.15.1:0/1916884385 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02f006b500 0x7f02f006d9c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.065+0000 7f030a49a640 1 --2- 172.21.15.1:0/1916884385 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0304154c30 0x7f030414ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.065+0000 7f030a49a640 1 --2- 172.21.15.1:0/1916884385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0304152650 0x7f03041ee630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.065+0000 7f030a49a640 1 --2- 172.21.15.1:0/1916884385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03040bf9d0 0x7f03041ee0f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.065+0000 7f030a49a640 1 -- 172.21.15.1:0/1916884385 >> 172.21.15.1:0/1916884385 conn(0x7f0304066ee0 msgr2=0x7f03040672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.065+0000 7f030a49a640 1 -- 172.21.15.1:0/1916884385 shutdown_connections 2024-03-06T07:21:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.066+0000 7f030a49a640 1 -- 172.21.15.1:0/1916884385 wait complete. 2024-03-06T07:21:02.078 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:21:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.181+0000 7feba010c640 1 Processor -- start 2024-03-06T07:21:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.181+0000 7feba010c640 1 -- start start 2024-03-06T07:21:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.182+0000 7feba010c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feb980c6b10 0x7feb98155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.182+0000 7feba010c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb980bf9d0 0x7feb980c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.182+0000 7feba010c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb980c2fc0 con 0x7feb98152650 2024-03-06T07:21:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.182+0000 7feba010c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb98155800 con 0x7feb980c6b10 2024-03-06T07:21:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.182+0000 7feba010c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb98158da0 con 0x7feb980bf9d0 2024-03-06T07:21:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.182+0000 7feb9d6a7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feb980c6b10 0x7feb98155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.182+0000 7feb9dea8640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7feb98152650 0x7feb98158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36372/0 (socket says 172.21.15.1:36372) 2024-03-06T07:21:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.182+0000 7feb9dea8640 1 -- 172.21.15.1:0/3302592033 learned_addr learned my addr 172.21.15.1:0/3302592033 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.183+0000 7feb9e6a9640 1 --2- 172.21.15.1:0/3302592033 >> v2:172.21.15.98:3300/0 conn(0x7feb980bf9d0 0x7feb980c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.183+0000 7feb9d6a7640 1 -- 172.21.15.1:0/3302592033 >> v2:172.21.15.98:3300/0 conn(0x7feb980bf9d0 msgr2=0x7feb980c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.183+0000 7feb9d6a7640 1 --2- 172.21.15.1:0/3302592033 >> v2:172.21.15.98:3300/0 conn(0x7feb980bf9d0 0x7feb980c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.183+0000 7feb9d6a7640 1 -- 172.21.15.1:0/3302592033 >> v1:172.21.15.1:6789/0 conn(0x7feb98152650 legacy=0x7feb98158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.183+0000 7feb9cea6640 1 -- 172.21.15.1:0/3302592033 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993873635 0 0) 0x7feb980c2fc0 con 0x7feb98152650 2024-03-06T07:21:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.183+0000 7feb9d6a7640 1 -- 172.21.15.1:0/3302592033 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb980c6690 con 0x7feb980c6b10 2024-03-06T07:21:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.183+0000 7feb9d6a7640 1 --2- 172.21.15.1:0/3302592033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb980c6b10 0x7feb98155c90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7feb981597b0 tx=0x7feb9402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a9492c3764bc8a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.184+0000 7feb9cea6640 1 -- 172.21.15.1:0/3302592033 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb940315a0 con 0x7feb980c6b10 2024-03-06T07:21:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.184+0000 7feb9cea6640 1 -- 172.21.15.1:0/3302592033 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb94037070 con 0x7feb980c6b10 2024-03-06T07:21:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.184+0000 7feb9cea6640 1 -- 172.21.15.1:0/3302592033 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94031960 con 0x7feb980c6b10 2024-03-06T07:21:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.184+0000 7feba010c640 1 -- 172.21.15.1:0/3302592033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb980c6b10 msgr2=0x7feb98155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.184+0000 7feba010c640 1 --2- 172.21.15.1:0/3302592033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb980c6b10 0x7feb98155c90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7feb981597b0 tx=0x7feb9402b840 comp rx=0 tx=0).stop 2024-03-06T07:21:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.184+0000 7feba010c640 1 -- 172.21.15.1:0/3302592033 shutdown_connections 2024-03-06T07:21:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.184+0000 7feba010c640 1 --2- 172.21.15.1:0/3302592033 >> v2:172.21.15.98:3300/0 conn(0x7feb980bf9d0 0x7feb980c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.184+0000 7feba010c640 1 --2- 172.21.15.1:0/3302592033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb980c6b10 0x7feb98155c90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.184+0000 7feba010c640 1 -- 172.21.15.1:0/3302592033 >> 172.21.15.1:0/3302592033 conn(0x7feb98066ee0 msgr2=0x7feb980672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:02.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.184+0000 7feba010c640 1 -- 172.21.15.1:0/3302592033 shutdown_connections 2024-03-06T07:21:02.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.184+0000 7feba010c640 1 -- 172.21.15.1:0/3302592033 wait complete. 2024-03-06T07:21:02.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.185+0000 7feba010c640 1 Processor -- start 2024-03-06T07:21:02.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.185+0000 7feba010c640 1 -- start start 2024-03-06T07:21:02.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.186+0000 7feba010c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb980bf9d0 0x7feb98154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.186+0000 7feba010c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb98152650 0x7feb980c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.186+0000 7feba010c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb98156490 0x7feb98152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.186+0000 7feba010c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb981551b0 con 0x7feb98156490 2024-03-06T07:21:02.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.186+0000 7feba010c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb980c6fa0 con 0x7feb980bf9d0 2024-03-06T07:21:02.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.186+0000 7feba010c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb98155490 con 0x7feb98152650 2024-03-06T07:21:02.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.186+0000 7feb9e6a9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb98156490 0x7feb98152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.186+0000 7feb9e6a9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb98156490 0x7feb98152a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37158/0 (socket says 172.21.15.1:37158) 2024-03-06T07:21:02.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.186+0000 7feb9e6a9640 1 -- 172.21.15.1:0/1947550446 learned_addr learned my addr 172.21.15.1:0/1947550446 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:02.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.186+0000 7feb9d6a7640 1 --2- 172.21.15.1:0/1947550446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb98152650 0x7feb980c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.186+0000 7feb9e6a9640 1 -- 172.21.15.1:0/1947550446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb98152650 msgr2=0x7feb980c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.186+0000 7feb9dea8640 1 --2- 172.21.15.1:0/1947550446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb980bf9d0 0x7feb98154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.186+0000 7feb9e6a9640 1 --2- 172.21.15.1:0/1947550446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb98152650 0x7feb980c8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.186+0000 7feb9e6a9640 1 -- 172.21.15.1:0/1947550446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb980bf9d0 msgr2=0x7feb98154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.186+0000 7feb9e6a9640 1 --2- 172.21.15.1:0/1947550446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb980bf9d0 0x7feb98154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.186+0000 7feb9e6a9640 1 -- 172.21.15.1:0/1947550446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb981de910 con 0x7feb98156490 2024-03-06T07:21:02.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.186+0000 7feb9dea8640 1 --2- 172.21.15.1:0/1947550446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb980bf9d0 0x7feb98154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:02.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.187+0000 7feb9e6a9640 1 --2- 172.21.15.1:0/1947550446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb98156490 0x7feb98152a30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7feb90007e60 tx=0x7feb9000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:02.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.187+0000 7feb8a7fc640 1 -- 172.21.15.1:0/1947550446 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb90014420 con 0x7feb98156490 2024-03-06T07:21:02.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.187+0000 7feb8a7fc640 1 -- 172.21.15.1:0/1947550446 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb90019070 con 0x7feb98156490 2024-03-06T07:21:02.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.187+0000 7feba010c640 1 -- 172.21.15.1:0/1947550446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb981ee810 con 0x7feb98156490 2024-03-06T07:21:02.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.187+0000 7feb8a7fc640 1 -- 172.21.15.1:0/1947550446 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb90014820 con 0x7feb98156490 2024-03-06T07:21:02.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.187+0000 7feba010c640 1 -- 172.21.15.1:0/1947550446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb981547b0 con 0x7feb98156490 2024-03-06T07:21:02.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.188+0000 7feba010c640 1 -- 172.21.15.1:0/1947550446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb98067fd0 con 0x7feb98156490 2024-03-06T07:21:02.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.188+0000 7feb8a7fc640 1 -- 172.21.15.1:0/1947550446 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb900149c0 con 0x7feb98156490 2024-03-06T07:21:02.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.189+0000 7feb8a7fc640 1 --2- 172.21.15.1:0/1947550446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb7c06b4d0 0x7feb7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.189+0000 7feb8a7fc640 1 -- 172.21.15.1:0/1947550446 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7feb9008ad10 con 0x7feb98156490 2024-03-06T07:21:02.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.189+0000 7feb9dea8640 1 --2- 172.21.15.1:0/1947550446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb7c06b4d0 0x7feb7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.189+0000 7feb9dea8640 1 --2- 172.21.15.1:0/1947550446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb7c06b4d0 0x7feb7c06d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7feb8c000f60 tx=0x7feb8c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:02.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.193+0000 7feb8a7fc640 1 -- 172.21.15.1:0/1947550446 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb900598e0 con 0x7feb98156490 2024-03-06T07:21:02.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.461+0000 7feba010c640 1 -- 172.21.15.1:0/1947550446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7feb980b06f0 con 0x7feb98156490 2024-03-06T07:21:02.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.461+0000 7feb8a7fc640 1 -- 172.21.15.1:0/1947550446 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7feb900591e0 con 0x7feb98156490 2024-03-06T07:21:02.463 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:21:02.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.463+0000 7feba010c640 1 -- 172.21.15.1:0/1947550446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb7c06b4d0 msgr2=0x7feb7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.463+0000 7feba010c640 1 --2- 172.21.15.1:0/1947550446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb7c06b4d0 0x7feb7c06d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7feb8c000f60 tx=0x7feb8c009040 comp rx=0 tx=0).stop 2024-03-06T07:21:02.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.463+0000 7feba010c640 1 -- 172.21.15.1:0/1947550446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb98156490 msgr2=0x7feb98152a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.463+0000 7feba010c640 1 --2- 172.21.15.1:0/1947550446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb98156490 0x7feb98152a30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7feb90007e60 tx=0x7feb9000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.463+0000 7feba010c640 1 -- 172.21.15.1:0/1947550446 shutdown_connections 2024-03-06T07:21:02.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.463+0000 7feba010c640 1 --2- 172.21.15.1:0/1947550446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb7c06b4d0 0x7feb7c06d990 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.463+0000 7feba010c640 1 --2- 172.21.15.1:0/1947550446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb98156490 0x7feb98152a30 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.463+0000 7feba010c640 1 --2- 172.21.15.1:0/1947550446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb98152650 0x7feb980c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.463+0000 7feba010c640 1 --2- 172.21.15.1:0/1947550446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb980bf9d0 0x7feb98154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.463+0000 7feba010c640 1 -- 172.21.15.1:0/1947550446 >> 172.21.15.1:0/1947550446 conn(0x7feb98066ee0 msgr2=0x7feb980bfff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:02.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.463+0000 7feba010c640 1 -- 172.21.15.1:0/1947550446 shutdown_connections 2024-03-06T07:21:02.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.463+0000 7feba010c640 1 -- 172.21.15.1:0/1947550446 wait complete. 2024-03-06T07:21:02.476 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop' 2024-03-06T07:21:02.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.575+0000 7ff08cbd0640 1 Processor -- start 2024-03-06T07:21:02.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.575+0000 7ff08cbd0640 1 -- start start 2024-03-06T07:21:02.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.576+0000 7ff08cbd0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff088152650 0x7ff088155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.576+0000 7ff08cbd0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0880c6b10 0x7ff088158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.576+0000 7ff08cbd0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff0880bf9d0 0x7ff0880c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.576+0000 7ff08cbd0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff088155800 con 0x7ff0880bf9d0 2024-03-06T07:21:02.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.576+0000 7ff08cbd0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0880c2fc0 con 0x7ff088152650 2024-03-06T07:21:02.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.576+0000 7ff08cbd0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0880c6690 con 0x7ff0880c6b10 2024-03-06T07:21:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.576+0000 7ff086d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff0880bf9d0 0x7ff0880c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.576+0000 7ff086d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff0880bf9d0 0x7ff0880c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37162/0 (socket says 172.21.15.1:37162) 2024-03-06T07:21:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.576+0000 7ff086d9d640 1 -- 172.21.15.1:0/480958112 learned_addr learned my addr 172.21.15.1:0/480958112 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.576+0000 7ff08659c640 1 --2- 172.21.15.1:0/480958112 >> v2:172.21.15.98:3300/0 conn(0x7ff088152650 0x7ff088155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.576+0000 7ff07dd9b640 1 --2- 172.21.15.1:0/480958112 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0880c6b10 0x7ff088158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.576+0000 7ff086d9d640 1 -- 172.21.15.1:0/480958112 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0880c6b10 msgr2=0x7ff088158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.576+0000 7ff086d9d640 1 --2- 172.21.15.1:0/480958112 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0880c6b10 0x7ff088158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.576+0000 7ff086d9d640 1 -- 172.21.15.1:0/480958112 >> v2:172.21.15.98:3300/0 conn(0x7ff088152650 msgr2=0x7ff088155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.576+0000 7ff086d9d640 1 --2- 172.21.15.1:0/480958112 >> v2:172.21.15.98:3300/0 conn(0x7ff088152650 0x7ff088155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.576+0000 7ff086d9d640 1 -- 172.21.15.1:0/480958112 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0880c7280 con 0x7ff0880bf9d0 2024-03-06T07:21:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.576+0000 7ff08659c640 1 --2- 172.21.15.1:0/480958112 >> v2:172.21.15.98:3300/0 conn(0x7ff088152650 0x7ff088155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.577+0000 7ff086d9d640 1 --2- 172.21.15.1:0/480958112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0880bf9d0 0x7ff0880c1460 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff0880cac90 tx=0x7ff08002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ef2541c534f1e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.577+0000 7ff085d9b640 1 -- 172.21.15.1:0/480958112 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08002c9e0 con 0x7ff0880bf9d0 2024-03-06T07:21:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.577+0000 7ff085d9b640 1 -- 172.21.15.1:0/480958112 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08002cb80 con 0x7ff0880bf9d0 2024-03-06T07:21:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.577+0000 7ff085d9b640 1 -- 172.21.15.1:0/480958112 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0800343f0 con 0x7ff0880bf9d0 2024-03-06T07:21:02.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.578+0000 7ff08cbd0640 1 -- 172.21.15.1:0/480958112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0880bf9d0 msgr2=0x7ff0880c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.578+0000 7ff08cbd0640 1 --2- 172.21.15.1:0/480958112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0880bf9d0 0x7ff0880c1460 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff0880cac90 tx=0x7ff08002be40 comp rx=0 tx=0).stop 2024-03-06T07:21:02.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.578+0000 7ff08cbd0640 1 -- 172.21.15.1:0/480958112 shutdown_connections 2024-03-06T07:21:02.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.578+0000 7ff08cbd0640 1 --2- 172.21.15.1:0/480958112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0880bf9d0 0x7ff0880c1460 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.578+0000 7ff08cbd0640 1 --2- 172.21.15.1:0/480958112 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0880c6b10 0x7ff088158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.578+0000 7ff08cbd0640 1 --2- 172.21.15.1:0/480958112 >> v2:172.21.15.98:3300/0 conn(0x7ff088152650 0x7ff088155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.578+0000 7ff08cbd0640 1 -- 172.21.15.1:0/480958112 >> 172.21.15.1:0/480958112 conn(0x7ff088066ee0 msgr2=0x7ff0880672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:02.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.578+0000 7ff08cbd0640 1 -- 172.21.15.1:0/480958112 shutdown_connections 2024-03-06T07:21:02.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.578+0000 7ff08cbd0640 1 -- 172.21.15.1:0/480958112 wait complete. 2024-03-06T07:21:02.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.579+0000 7ff08cbd0640 1 Processor -- start 2024-03-06T07:21:02.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.579+0000 7ff08cbd0640 1 -- start start 2024-03-06T07:21:02.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.580+0000 7ff08cbd0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0880bf9d0 0x7ff08814c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.580+0000 7ff08cbd0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff088152650 0x7ff0880c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.580+0000 7ff08cbd0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0880c9df0 0x7ff088153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.580+0000 7ff08cbd0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff088155490 con 0x7ff0880c9df0 2024-03-06T07:21:02.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.580+0000 7ff08cbd0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0881551b0 con 0x7ff0880bf9d0 2024-03-06T07:21:02.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.580+0000 7ff08cbd0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff088152a30 con 0x7ff088152650 2024-03-06T07:21:02.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.580+0000 7ff086d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0880c9df0 0x7ff088153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.580+0000 7ff086d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0880c9df0 0x7ff088153860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37166/0 (socket says 172.21.15.1:37166) 2024-03-06T07:21:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.580+0000 7ff086d9d640 1 -- 172.21.15.1:0/911541438 learned_addr learned my addr 172.21.15.1:0/911541438 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.580+0000 7ff086d9d640 1 -- 172.21.15.1:0/911541438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff088152650 msgr2=0x7ff0880c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.580+0000 7ff086d9d640 1 --2- 172.21.15.1:0/911541438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff088152650 0x7ff0880c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.580+0000 7ff086d9d640 1 -- 172.21.15.1:0/911541438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0880bf9d0 msgr2=0x7ff08814c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.580+0000 7ff086d9d640 1 --2- 172.21.15.1:0/911541438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0880bf9d0 0x7ff08814c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.580+0000 7ff086d9d640 1 -- 172.21.15.1:0/911541438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0881ebba0 con 0x7ff0880c9df0 2024-03-06T07:21:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.580+0000 7ff086d9d640 1 --2- 172.21.15.1:0/911541438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0880c9df0 0x7ff088153860 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff080036040 tx=0x7ff080004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.581+0000 7ff07f7fe640 1 -- 172.21.15.1:0/911541438 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff080043020 con 0x7ff0880c9df0 2024-03-06T07:21:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.581+0000 7ff07f7fe640 1 -- 172.21.15.1:0/911541438 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08003e450 con 0x7ff0880c9df0 2024-03-06T07:21:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.581+0000 7ff08cbd0640 1 -- 172.21.15.1:0/911541438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0880c8dd0 con 0x7ff0880c9df0 2024-03-06T07:21:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.581+0000 7ff07f7fe640 1 -- 172.21.15.1:0/911541438 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff080009510 con 0x7ff0880c9df0 2024-03-06T07:21:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.581+0000 7ff08cbd0640 1 -- 172.21.15.1:0/911541438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0880c9390 con 0x7ff0880c9df0 2024-03-06T07:21:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.582+0000 7ff08cbd0640 1 -- 172.21.15.1:0/911541438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff088067fd0 con 0x7ff0880c9df0 2024-03-06T07:21:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.582+0000 7ff07f7fe640 1 -- 172.21.15.1:0/911541438 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff080046020 con 0x7ff0880c9df0 2024-03-06T07:21:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.582+0000 7ff07f7fe640 1 --2- 172.21.15.1:0/911541438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff06806b4d0 0x7ff06806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.583+0000 7ff07f7fe640 1 -- 172.21.15.1:0/911541438 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff0800acd80 con 0x7ff0880c9df0 2024-03-06T07:21:02.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.583+0000 7ff08659c640 1 --2- 172.21.15.1:0/911541438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff06806b4d0 0x7ff06806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.583+0000 7ff08659c640 1 --2- 172.21.15.1:0/911541438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff06806b4d0 0x7ff06806d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff078006fd0 tx=0x7ff078008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:02.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.587+0000 7ff07f7fe640 1 -- 172.21.15.1:0/911541438 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff08004d030 con 0x7ff0880c9df0 2024-03-06T07:21:02.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:02.880+0000 7ff08cbd0640 1 -- 172.21.15.1:0/911541438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]} v 0) v1 -- 0x7ff0880b06f0 con 0x7ff0880c9df0 2024-03-06T07:21:03.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.069+0000 7ff07f7fe640 1 -- 172.21.15.1:0/911541438 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]}]=0 v58) v1 ==== 146+0+0 (secure 0 0 0) 0x7ff08007b950 con 0x7ff0880c9df0 2024-03-06T07:21:03.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.072+0000 7ff08cbd0640 1 -- 172.21.15.1:0/911541438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff06806b4d0 msgr2=0x7ff06806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.072+0000 7ff08cbd0640 1 --2- 172.21.15.1:0/911541438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff06806b4d0 0x7ff06806d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff078006fd0 tx=0x7ff078008040 comp rx=0 tx=0).stop 2024-03-06T07:21:03.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.072+0000 7ff08cbd0640 1 -- 172.21.15.1:0/911541438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0880c9df0 msgr2=0x7ff088153860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.072+0000 7ff08cbd0640 1 --2- 172.21.15.1:0/911541438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0880c9df0 0x7ff088153860 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff080036040 tx=0x7ff080004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:03.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.072+0000 7ff08cbd0640 1 -- 172.21.15.1:0/911541438 shutdown_connections 2024-03-06T07:21:03.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.072+0000 7ff08cbd0640 1 --2- 172.21.15.1:0/911541438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff06806b4d0 0x7ff06806d990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.072+0000 7ff08cbd0640 1 --2- 172.21.15.1:0/911541438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0880c9df0 0x7ff088153860 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.072+0000 7ff08cbd0640 1 --2- 172.21.15.1:0/911541438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff088152650 0x7ff0880c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.072+0000 7ff08cbd0640 1 --2- 172.21.15.1:0/911541438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0880bf9d0 0x7ff08814c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.072+0000 7ff08cbd0640 1 -- 172.21.15.1:0/911541438 >> 172.21.15.1:0/911541438 conn(0x7ff088066ee0 msgr2=0x7ff0880672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:03.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.072+0000 7ff08cbd0640 1 -- 172.21.15.1:0/911541438 shutdown_connections 2024-03-06T07:21:03.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.072+0000 7ff08cbd0640 1 -- 172.21.15.1:0/911541438 wait complete. 2024-03-06T07:21:03.085 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:21:03.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.187+0000 7ff8af3e9640 1 Processor -- start 2024-03-06T07:21:03.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.188+0000 7ff8af3e9640 1 -- start start 2024-03-06T07:21:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.188+0000 7ff8af3e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a8150520 0x7ff8a81543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.188+0000 7ff8af3e9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff8a8070c80 0x7ff8a80710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.188+0000 7ff8af3e9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a8150900 con 0x7ff8a80c32f0 2024-03-06T07:21:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.188+0000 7ff8af3e9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a8153ed0 con 0x7ff8a8070c80 2024-03-06T07:21:03.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.188+0000 7ff8af3e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a8071790 con 0x7ff8a8150520 2024-03-06T07:21:03.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.188+0000 7ff8ad986640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff8a8070c80 0x7ff8a80710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.188+0000 7ff8ac984640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff8a80c32f0 0x7ff8a81537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36384/0 (socket says 172.21.15.1:36384) 2024-03-06T07:21:03.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.188+0000 7ff8ac984640 1 -- 172.21.15.1:0/3374619333 learned_addr learned my addr 172.21.15.1:0/3374619333 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:03.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.189+0000 7ff8ad185640 1 --2- 172.21.15.1:0/3374619333 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a8150520 0x7ff8a81543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.189+0000 7ff8ad986640 1 -- 172.21.15.1:0/3374619333 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a8150520 msgr2=0x7ff8a81543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:03.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.189+0000 7ff8ad986640 1 --2- 172.21.15.1:0/3374619333 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a8150520 0x7ff8a81543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.189+0000 7ff8ad986640 1 -- 172.21.15.1:0/3374619333 >> v1:172.21.15.1:6789/0 conn(0x7ff8a80c32f0 legacy=0x7ff8a81537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.189+0000 7ff89ffff640 1 -- 172.21.15.1:0/3374619333 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089530667 0 0) 0x7ff8a8150900 con 0x7ff8a80c32f0 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.189+0000 7ff8ad986640 1 -- 172.21.15.1:0/3374619333 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a8157050 con 0x7ff8a8070c80 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.189+0000 7ff8ad185640 1 --2- 172.21.15.1:0/3374619333 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a8150520 0x7ff8a81543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.190+0000 7ff8ad986640 1 --2- 172.21.15.1:0/3374619333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a8070c80 0x7ff8a80710f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009d90 tx=0x7ff8a402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=962c44d690080f16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.190+0000 7ff89ffff640 1 -- 172.21.15.1:0/3374619333 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4039070 con 0x7ff8a8070c80 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.190+0000 7ff89ffff640 1 -- 172.21.15.1:0/3374619333 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a402cc50 con 0x7ff8a8070c80 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.190+0000 7ff89ffff640 1 -- 172.21.15.1:0/3374619333 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4034440 con 0x7ff8a8070c80 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.191+0000 7ff8af3e9640 1 -- 172.21.15.1:0/3374619333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a8070c80 msgr2=0x7ff8a80710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.191+0000 7ff8af3e9640 1 --2- 172.21.15.1:0/3374619333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a8070c80 0x7ff8a80710f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009d90 tx=0x7ff8a402bea0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.191+0000 7ff8af3e9640 1 -- 172.21.15.1:0/3374619333 shutdown_connections 2024-03-06T07:21:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.191+0000 7ff8af3e9640 1 --2- 172.21.15.1:0/3374619333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a8070c80 0x7ff8a80710f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.191+0000 7ff8af3e9640 1 --2- 172.21.15.1:0/3374619333 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a8150520 0x7ff8a81543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.191+0000 7ff8af3e9640 1 -- 172.21.15.1:0/3374619333 >> 172.21.15.1:0/3374619333 conn(0x7ff8a80c88f0 msgr2=0x7ff8a80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.191+0000 7ff8af3e9640 1 -- 172.21.15.1:0/3374619333 shutdown_connections 2024-03-06T07:21:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.191+0000 7ff8af3e9640 1 -- 172.21.15.1:0/3374619333 wait complete. 2024-03-06T07:21:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.192+0000 7ff8af3e9640 1 Processor -- start 2024-03-06T07:21:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.192+0000 7ff8af3e9640 1 -- start start 2024-03-06T07:21:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.192+0000 7ff8af3e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a8070c80 0x7ff8a80ac780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.192+0000 7ff8af3e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a80c32f0 0x7ff8a8154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8af3e9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a8150520 0x7ff8a8155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8af3e9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a8157330 con 0x7ff8a8150520 2024-03-06T07:21:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8af3e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a81532a0 con 0x7ff8a8070c80 2024-03-06T07:21:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8af3e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a80c7ec0 con 0x7ff8a80c32f0 2024-03-06T07:21:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8ad986640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a8150520 0x7ff8a8155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8ad986640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a8150520 0x7ff8a8155490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37196/0 (socket says 172.21.15.1:37196) 2024-03-06T07:21:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8ad986640 1 -- 172.21.15.1:0/708217046 learned_addr learned my addr 172.21.15.1:0/708217046 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8ac984640 1 --2- 172.21.15.1:0/708217046 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a80c32f0 0x7ff8a8154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8ad185640 1 --2- 172.21.15.1:0/708217046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a8070c80 0x7ff8a80ac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8ad986640 1 -- 172.21.15.1:0/708217046 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a80c32f0 msgr2=0x7ff8a8154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8ad986640 1 --2- 172.21.15.1:0/708217046 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a80c32f0 0x7ff8a8154f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8ad986640 1 -- 172.21.15.1:0/708217046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a8070c80 msgr2=0x7ff8a80ac780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8ad986640 1 --2- 172.21.15.1:0/708217046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a8070c80 0x7ff8a80ac780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8ad986640 1 -- 172.21.15.1:0/708217046 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a81470b0 con 0x7ff8a8150520 2024-03-06T07:21:03.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8ac984640 1 --2- 172.21.15.1:0/708217046 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a80c32f0 0x7ff8a8154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:03.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8ad185640 1 --2- 172.21.15.1:0/708217046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a8070c80 0x7ff8a80ac780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:03.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8ad986640 1 --2- 172.21.15.1:0/708217046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a8150520 0x7ff8a8155490 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009eb0 tx=0x7ff8a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:03.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff89dffb640 1 -- 172.21.15.1:0/708217046 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4039050 con 0x7ff8a8150520 2024-03-06T07:21:03.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff89dffb640 1 -- 172.21.15.1:0/708217046 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a4045070 con 0x7ff8a8150520 2024-03-06T07:21:03.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff89dffb640 1 -- 172.21.15.1:0/708217046 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a403e610 con 0x7ff8a8150520 2024-03-06T07:21:03.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.193+0000 7ff8af3e9640 1 -- 172.21.15.1:0/708217046 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a8155fa0 con 0x7ff8a8150520 2024-03-06T07:21:03.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.194+0000 7ff8af3e9640 1 -- 172.21.15.1:0/708217046 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a8156560 con 0x7ff8a8150520 2024-03-06T07:21:03.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.195+0000 7ff8af3e9640 1 -- 172.21.15.1:0/708217046 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a80686e0 con 0x7ff8a8150520 2024-03-06T07:21:03.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.196+0000 7ff89dffb640 1 -- 172.21.15.1:0/708217046 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8a403e7b0 con 0x7ff8a8150520 2024-03-06T07:21:03.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.196+0000 7ff89dffb640 1 --2- 172.21.15.1:0/708217046 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff88c06b500 0x7ff88c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.196+0000 7ff89dffb640 1 -- 172.21.15.1:0/708217046 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff8a40ad040 con 0x7ff8a8150520 2024-03-06T07:21:03.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.196+0000 7ff8ad185640 1 --2- 172.21.15.1:0/708217046 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff88c06b500 0x7ff88c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.197+0000 7ff8ad185640 1 --2- 172.21.15.1:0/708217046 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff88c06b500 0x7ff88c06d9c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0006fd0 tx=0x7ff8a0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:03.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.200+0000 7ff89dffb640 1 -- 172.21.15.1:0/708217046 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8a4032e20 con 0x7ff8a8150520 2024-03-06T07:21:03.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.384+0000 7ff8af3e9640 1 -- 172.21.15.1:0/708217046 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff8a814d8d0 con 0x7ff8a8150520 2024-03-06T07:21:03.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.387+0000 7ff89dffb640 1 -- 172.21.15.1:0/708217046 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v11)=0 v11) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff8a407bc10 con 0x7ff8a8150520 2024-03-06T07:21:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.389+0000 7ff8af3e9640 1 -- 172.21.15.1:0/708217046 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff88c06b500 msgr2=0x7ff88c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.389+0000 7ff8af3e9640 1 --2- 172.21.15.1:0/708217046 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff88c06b500 0x7ff88c06d9c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0006fd0 tx=0x7ff8a0008040 comp rx=0 tx=0).stop 2024-03-06T07:21:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.389+0000 7ff8af3e9640 1 -- 172.21.15.1:0/708217046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a8150520 msgr2=0x7ff8a8155490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.389+0000 7ff8af3e9640 1 --2- 172.21.15.1:0/708217046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a8150520 0x7ff8a8155490 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009eb0 tx=0x7ff8a4004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.389+0000 7ff8af3e9640 1 -- 172.21.15.1:0/708217046 shutdown_connections 2024-03-06T07:21:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.389+0000 7ff8af3e9640 1 --2- 172.21.15.1:0/708217046 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff88c06b500 0x7ff88c06d9c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.389+0000 7ff8af3e9640 1 --2- 172.21.15.1:0/708217046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a8150520 0x7ff8a8155490 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.389+0000 7ff8af3e9640 1 --2- 172.21.15.1:0/708217046 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a80c32f0 0x7ff8a8154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.389+0000 7ff8af3e9640 1 --2- 172.21.15.1:0/708217046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a8070c80 0x7ff8a80ac780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.389+0000 7ff8af3e9640 1 -- 172.21.15.1:0/708217046 >> 172.21.15.1:0/708217046 conn(0x7ff8a80c88f0 msgr2=0x7ff8a814aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.389+0000 7ff8af3e9640 1 -- 172.21.15.1:0/708217046 shutdown_connections 2024-03-06T07:21:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.390+0000 7ff8af3e9640 1 -- 172.21.15.1:0/708217046 wait complete. 2024-03-06T07:21:03.402 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:21:03.402 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:21:03.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:21:03.498 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:21:03.498 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:21:03.498 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:21:03.513 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:21:03.513 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:21:03.567 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:21:03.568 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:21:03.622 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:03.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.761+0000 7fad2e08f640 1 Processor -- start 2024-03-06T07:21:03.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.761+0000 7fad2e08f640 1 -- start start 2024-03-06T07:21:03.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.762+0000 7fad2e08f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fad280a7280 0x7fad28072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.762+0000 7fad2e08f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad280732b0 0x7fad28156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.762+0000 7fad2e08f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad28157130 con 0x7fad280c4ef0 2024-03-06T07:21:03.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.762+0000 7fad2e08f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad28152fb0 con 0x7fad280a7280 2024-03-06T07:21:03.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.762+0000 7fad2e08f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad28153290 con 0x7fad280732b0 2024-03-06T07:21:03.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.762+0000 7fad26ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fad280c4ef0 0x7fad281564e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36394/0 (socket says 172.21.15.1:36394) 2024-03-06T07:21:03.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.762+0000 7fad26ffd640 1 -- 172.21.15.1:0/1065323855 learned_addr learned my addr 172.21.15.1:0/1065323855 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:03.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.762+0000 7fad277fe640 1 --2- 172.21.15.1:0/1065323855 >> v2:172.21.15.98:3300/0 conn(0x7fad280a7280 0x7fad28072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.763+0000 7fad27fff640 1 --2- 172.21.15.1:0/1065323855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad280732b0 0x7fad28156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.763+0000 7fad267fc640 1 -- 172.21.15.1:0/1065323855 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194960700 0 0) 0x7fad28157130 con 0x7fad280c4ef0 2024-03-06T07:21:03.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.763+0000 7fad267fc640 1 -- 172.21.15.1:0/1065323855 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad10003520 con 0x7fad280c4ef0 2024-03-06T07:21:03.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.763+0000 7fad267fc640 1 -- 172.21.15.1:0/1065323855 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1400271635 0 0) 0x7fad10003520 con 0x7fad280c4ef0 2024-03-06T07:21:03.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.763+0000 7fad277fe640 1 -- 172.21.15.1:0/1065323855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad280732b0 msgr2=0x7fad28156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:03.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.763+0000 7fad277fe640 1 --2- 172.21.15.1:0/1065323855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad280732b0 0x7fad28156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.763+0000 7fad277fe640 1 -- 172.21.15.1:0/1065323855 >> v1:172.21.15.1:6789/0 conn(0x7fad280c4ef0 legacy=0x7fad281564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.763+0000 7fad277fe640 1 -- 172.21.15.1:0/1065323855 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad28153570 con 0x7fad280a7280 2024-03-06T07:21:03.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.763+0000 7fad27fff640 1 --2- 172.21.15.1:0/1065323855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad280732b0 0x7fad28156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:03.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.764+0000 7fad277fe640 1 --2- 172.21.15.1:0/1065323855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad280a7280 0x7fad28072d70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fad180097a0 tx=0x7fad1802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aecc62e11db07887 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:03.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.764+0000 7fad267fc640 1 -- 172.21.15.1:0/1065323855 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad18037070 con 0x7fad280a7280 2024-03-06T07:21:03.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.764+0000 7fad267fc640 1 -- 172.21.15.1:0/1065323855 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad18030510 con 0x7fad280a7280 2024-03-06T07:21:03.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.764+0000 7fad267fc640 1 -- 172.21.15.1:0/1065323855 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c001670 con 0x7fad280a7280 2024-03-06T07:21:03.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.764+0000 7fad2e08f640 1 -- 172.21.15.1:0/1065323855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad280a7280 msgr2=0x7fad28072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.764+0000 7fad2e08f640 1 --2- 172.21.15.1:0/1065323855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad280a7280 0x7fad28072d70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fad180097a0 tx=0x7fad1802b790 comp rx=0 tx=0).stop 2024-03-06T07:21:03.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.764+0000 7fad2e08f640 1 -- 172.21.15.1:0/1065323855 shutdown_connections 2024-03-06T07:21:03.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.764+0000 7fad2e08f640 1 --2- 172.21.15.1:0/1065323855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad280732b0 0x7fad28156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.764+0000 7fad2e08f640 1 --2- 172.21.15.1:0/1065323855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad280a7280 0x7fad28072d70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.764+0000 7fad2e08f640 1 -- 172.21.15.1:0/1065323855 >> 172.21.15.1:0/1065323855 conn(0x7fad280c7a50 msgr2=0x7fad280c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:03.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.764+0000 7fad2e08f640 1 -- 172.21.15.1:0/1065323855 shutdown_connections 2024-03-06T07:21:03.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.765+0000 7fad2e08f640 1 -- 172.21.15.1:0/1065323855 wait complete. 2024-03-06T07:21:03.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.766+0000 7fad2e08f640 1 Processor -- start 2024-03-06T07:21:03.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.766+0000 7fad2e08f640 1 -- start start 2024-03-06T07:21:03.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.766+0000 7fad2e08f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad280732b0 0x7fad281dfe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.766+0000 7fad2e08f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad280a7280 0x7fad281ebfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.766+0000 7fad2e08f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad280c4ef0 0x7fad2814cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.766+0000 7fad2e08f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad280c24e0 con 0x7fad280a7280 2024-03-06T07:21:03.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.766+0000 7fad2e08f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad28153b30 con 0x7fad280732b0 2024-03-06T07:21:03.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.766+0000 7fad2e08f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad280c2660 con 0x7fad280c4ef0 2024-03-06T07:21:03.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.767+0000 7fad26ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad280a7280 0x7fad281ebfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.767+0000 7fad26ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad280a7280 0x7fad281ebfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37210/0 (socket says 172.21.15.1:37210) 2024-03-06T07:21:03.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.767+0000 7fad26ffd640 1 -- 172.21.15.1:0/2503317237 learned_addr learned my addr 172.21.15.1:0/2503317237 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:03.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.767+0000 7fad277fe640 1 --2- 172.21.15.1:0/2503317237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad280732b0 0x7fad281dfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.767+0000 7fad27fff640 1 --2- 172.21.15.1:0/2503317237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad280c4ef0 0x7fad2814cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.767+0000 7fad26ffd640 1 -- 172.21.15.1:0/2503317237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad280c4ef0 msgr2=0x7fad2814cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.767+0000 7fad26ffd640 1 --2- 172.21.15.1:0/2503317237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad280c4ef0 0x7fad2814cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.767+0000 7fad26ffd640 1 -- 172.21.15.1:0/2503317237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad280732b0 msgr2=0x7fad281dfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.767+0000 7fad26ffd640 1 --2- 172.21.15.1:0/2503317237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad280732b0 0x7fad281dfe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.767+0000 7fad26ffd640 1 -- 172.21.15.1:0/2503317237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2814e100 con 0x7fad280a7280 2024-03-06T07:21:03.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.767+0000 7fad26ffd640 1 --2- 172.21.15.1:0/2503317237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad280a7280 0x7fad281ebfe0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00c750 tx=0x7fad1c00cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:03.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.767+0000 7fad0ffff640 1 -- 172.21.15.1:0/2503317237 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c005ad0 con 0x7fad280a7280 2024-03-06T07:21:03.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.767+0000 7fad0ffff640 1 -- 172.21.15.1:0/2503317237 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1c005c70 con 0x7fad280a7280 2024-03-06T07:21:03.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.767+0000 7fad0ffff640 1 -- 172.21.15.1:0/2503317237 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c005e60 con 0x7fad280a7280 2024-03-06T07:21:03.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.767+0000 7fad2e08f640 1 -- 172.21.15.1:0/2503317237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2814dc60 con 0x7fad280a7280 2024-03-06T07:21:03.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.767+0000 7fad2e08f640 1 -- 172.21.15.1:0/2503317237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad281dc7e0 con 0x7fad280a7280 2024-03-06T07:21:03.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.768+0000 7fad2e08f640 1 -- 172.21.15.1:0/2503317237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad280686e0 con 0x7fad280a7280 2024-03-06T07:21:03.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.769+0000 7fad0ffff640 1 -- 172.21.15.1:0/2503317237 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad1c007290 con 0x7fad280a7280 2024-03-06T07:21:03.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.769+0000 7fad0ffff640 1 --2- 172.21.15.1:0/2503317237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad1406b500 0x7fad1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.769+0000 7fad0ffff640 1 -- 172.21.15.1:0/2503317237 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fad1c08a180 con 0x7fad280a7280 2024-03-06T07:21:03.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.769+0000 7fad277fe640 1 --2- 172.21.15.1:0/2503317237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad1406b500 0x7fad1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.770+0000 7fad277fe640 1 --2- 172.21.15.1:0/2503317237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad1406b500 0x7fad1406d9c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fad18034000 tx=0x7fad1802f000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:03.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.773+0000 7fad0ffff640 1 -- 172.21.15.1:0/2503317237 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad1c058d50 con 0x7fad280a7280 2024-03-06T07:21:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.955+0000 7fad2e08f640 1 -- 172.21.15.1:0/2503317237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad28072b50 con 0x7fad280a7280 2024-03-06T07:21:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.956+0000 7fad0ffff640 1 -- 172.21.15.1:0/2503317237 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7fad1c058650 con 0x7fad280a7280 2024-03-06T07:21:03.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:03.959 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:03.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.959+0000 7fad2e08f640 1 -- 172.21.15.1:0/2503317237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad1406b500 msgr2=0x7fad1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.959+0000 7fad2e08f640 1 --2- 172.21.15.1:0/2503317237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad1406b500 0x7fad1406d9c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fad18034000 tx=0x7fad1802f000 comp rx=0 tx=0).stop 2024-03-06T07:21:03.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.959+0000 7fad2e08f640 1 -- 172.21.15.1:0/2503317237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad280a7280 msgr2=0x7fad281ebfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.959+0000 7fad2e08f640 1 --2- 172.21.15.1:0/2503317237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad280a7280 0x7fad281ebfe0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00c750 tx=0x7fad1c00cb50 comp rx=0 tx=0).stop 2024-03-06T07:21:03.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.959+0000 7fad2e08f640 1 -- 172.21.15.1:0/2503317237 shutdown_connections 2024-03-06T07:21:03.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.959+0000 7fad2e08f640 1 --2- 172.21.15.1:0/2503317237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad1406b500 0x7fad1406d9c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.959+0000 7fad2e08f640 1 --2- 172.21.15.1:0/2503317237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad280c4ef0 0x7fad2814cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.959+0000 7fad2e08f640 1 --2- 172.21.15.1:0/2503317237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad280a7280 0x7fad281ebfe0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.959+0000 7fad2e08f640 1 --2- 172.21.15.1:0/2503317237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad280732b0 0x7fad281dfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.959+0000 7fad2e08f640 1 -- 172.21.15.1:0/2503317237 >> 172.21.15.1:0/2503317237 conn(0x7fad280c7a50 msgr2=0x7fad2814f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:03.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.959+0000 7fad2e08f640 1 -- 172.21.15.1:0/2503317237 shutdown_connections 2024-03-06T07:21:03.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:03.959+0000 7fad2e08f640 1 -- 172.21.15.1:0/2503317237 wait complete. 2024-03-06T07:21:03.974 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:20:19.337560+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:20:20.172791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:20:23.884071+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:21:03.974 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:04.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.082+0000 7f9df8102640 1 Processor -- start 2024-03-06T07:21:04.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.082+0000 7f9df8102640 1 -- start start 2024-03-06T07:21:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.083+0000 7f9df8102640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9df0152650 0x7f9df0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.083+0000 7f9df8102640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9df00c6b10 0x7f9df0158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.083+0000 7f9df8102640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df00c2fc0 con 0x7f9df00bf9d0 2024-03-06T07:21:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.083+0000 7f9df8102640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df0155800 con 0x7f9df0152650 2024-03-06T07:21:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.083+0000 7f9df8102640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df00c6690 con 0x7f9df00c6b10 2024-03-06T07:21:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.083+0000 7f9df5e9e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9df0152650 0x7f9df0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.083+0000 7f9df669f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9df00bf9d0 0x7f9df00c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36404/0 (socket says 172.21.15.1:36404) 2024-03-06T07:21:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.083+0000 7f9df669f640 1 -- 172.21.15.1:0/1042305655 learned_addr learned my addr 172.21.15.1:0/1042305655 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.083+0000 7f9df569d640 1 --2- 172.21.15.1:0/1042305655 >> v2:172.21.15.98:3300/0 conn(0x7f9df00c6b10 0x7f9df0158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.083+0000 7f9df5e9e640 1 -- 172.21.15.1:0/1042305655 >> v2:172.21.15.98:3300/0 conn(0x7f9df00c6b10 msgr2=0x7f9df0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.083+0000 7f9df5e9e640 1 --2- 172.21.15.1:0/1042305655 >> v2:172.21.15.98:3300/0 conn(0x7f9df00c6b10 0x7f9df0158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.083+0000 7f9df5e9e640 1 -- 172.21.15.1:0/1042305655 >> v1:172.21.15.1:6789/0 conn(0x7f9df00bf9d0 legacy=0x7f9df00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.083+0000 7f9df5e9e640 1 -- 172.21.15.1:0/1042305655 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df00c7280 con 0x7f9df0152650 2024-03-06T07:21:04.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.084+0000 7f9df5e9e640 1 --2- 172.21.15.1:0/1042305655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df0152650 0x7f9df0155c90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9df00cb8e0 tx=0x7f9de802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e7a021abe5adae8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:04.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.084+0000 7f9df4e9c640 1 -- 172.21.15.1:0/1042305655 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de80315a0 con 0x7f9df0152650 2024-03-06T07:21:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.084+0000 7f9df4e9c640 1 -- 172.21.15.1:0/1042305655 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de8037070 con 0x7f9df0152650 2024-03-06T07:21:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.084+0000 7f9df4e9c640 1 -- 172.21.15.1:0/1042305655 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de8031960 con 0x7f9df0152650 2024-03-06T07:21:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.085+0000 7f9df8102640 1 -- 172.21.15.1:0/1042305655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df0152650 msgr2=0x7f9df0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.085+0000 7f9df8102640 1 --2- 172.21.15.1:0/1042305655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df0152650 0x7f9df0155c90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9df00cb8e0 tx=0x7f9de802b840 comp rx=0 tx=0).stop 2024-03-06T07:21:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.085+0000 7f9df8102640 1 -- 172.21.15.1:0/1042305655 shutdown_connections 2024-03-06T07:21:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.085+0000 7f9df8102640 1 --2- 172.21.15.1:0/1042305655 >> v2:172.21.15.98:3300/0 conn(0x7f9df00c6b10 0x7f9df0158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.085+0000 7f9df8102640 1 --2- 172.21.15.1:0/1042305655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df0152650 0x7f9df0155c90 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.085+0000 7f9df8102640 1 -- 172.21.15.1:0/1042305655 >> 172.21.15.1:0/1042305655 conn(0x7f9df0066ee0 msgr2=0x7f9df00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.085+0000 7f9df8102640 1 -- 172.21.15.1:0/1042305655 shutdown_connections 2024-03-06T07:21:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.085+0000 7f9df8102640 1 -- 172.21.15.1:0/1042305655 wait complete. 2024-03-06T07:21:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.086+0000 7f9df8102640 1 Processor -- start 2024-03-06T07:21:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.086+0000 7f9df8102640 1 -- start start 2024-03-06T07:21:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df8102640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9df00bf9d0 0x7f9df014ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df8102640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df0152650 0x7f9df014e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df8102640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9df0149b60 0x7f9df01456f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df8102640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df0155490 con 0x7f9df0152650 2024-03-06T07:21:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df8102640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df01551b0 con 0x7f9df00bf9d0 2024-03-06T07:21:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df8102640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df0152a30 con 0x7f9df0149b60 2024-03-06T07:21:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df569d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df0152650 0x7f9df014e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df569d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df0152650 0x7f9df014e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37238/0 (socket says 172.21.15.1:37238) 2024-03-06T07:21:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df569d640 1 -- 172.21.15.1:0/3160091437 learned_addr learned my addr 172.21.15.1:0/3160091437 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df5e9e640 1 --2- 172.21.15.1:0/3160091437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9df00bf9d0 0x7f9df014ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df569d640 1 -- 172.21.15.1:0/3160091437 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9df0149b60 msgr2=0x7f9df01456f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df669f640 1 --2- 172.21.15.1:0/3160091437 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9df0149b60 0x7f9df01456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df569d640 1 --2- 172.21.15.1:0/3160091437 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9df0149b60 0x7f9df01456f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df569d640 1 -- 172.21.15.1:0/3160091437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9df00bf9d0 msgr2=0x7f9df014ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df569d640 1 --2- 172.21.15.1:0/3160091437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9df00bf9d0 0x7f9df014ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df569d640 1 -- 172.21.15.1:0/3160091437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df00c8f40 con 0x7f9df0152650 2024-03-06T07:21:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df5e9e640 1 --2- 172.21.15.1:0/3160091437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9df00bf9d0 0x7f9df014ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.087+0000 7f9df569d640 1 --2- 172.21.15.1:0/3160091437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df0152650 0x7f9df014e7a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9de4007e40 tx=0x7f9de400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:04.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.088+0000 7f9dee7fc640 1 -- 172.21.15.1:0/3160091437 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de4014620 con 0x7f9df0152650 2024-03-06T07:21:04.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.088+0000 7f9dee7fc640 1 -- 172.21.15.1:0/3160091437 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de4019070 con 0x7f9df0152650 2024-03-06T07:21:04.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.088+0000 7f9df8102640 1 -- 172.21.15.1:0/3160091437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df014de20 con 0x7f9df0152650 2024-03-06T07:21:04.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.088+0000 7f9dee7fc640 1 -- 172.21.15.1:0/3160091437 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de4014a00 con 0x7f9df0152650 2024-03-06T07:21:04.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.088+0000 7f9df8102640 1 -- 172.21.15.1:0/3160091437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df00c8a20 con 0x7f9df0152650 2024-03-06T07:21:04.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.089+0000 7f9df8102640 1 -- 172.21.15.1:0/3160091437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df0067fd0 con 0x7f9df0152650 2024-03-06T07:21:04.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.089+0000 7f9dee7fc640 1 -- 172.21.15.1:0/3160091437 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9de4014ba0 con 0x7f9df0152650 2024-03-06T07:21:04.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.090+0000 7f9dee7fc640 1 --2- 172.21.15.1:0/3160091437 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ddc06b500 0x7f9ddc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.090+0000 7f9dee7fc640 1 -- 172.21.15.1:0/3160091437 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9de408ad50 con 0x7f9df0152650 2024-03-06T07:21:04.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.090+0000 7f9df5e9e640 1 --2- 172.21.15.1:0/3160091437 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ddc06b500 0x7f9ddc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.090+0000 7f9df5e9e640 1 --2- 172.21.15.1:0/3160091437 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ddc06b500 0x7f9ddc06d9c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9de8036000 tx=0x7f9de8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:04.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.094+0000 7f9dee7fc640 1 -- 172.21.15.1:0/3160091437 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9de408e030 con 0x7f9df0152650 2024-03-06T07:21:04.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.277+0000 7f9df8102640 1 -- 172.21.15.1:0/3160091437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9df00c1170 con 0x7f9df0152650 2024-03-06T07:21:04.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.278+0000 7f9dee7fc640 1 -- 172.21.15.1:0/3160091437 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9de4059920 con 0x7f9df0152650 2024-03-06T07:21:04.280 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:21:04.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.280+0000 7f9df8102640 1 -- 172.21.15.1:0/3160091437 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ddc06b500 msgr2=0x7f9ddc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.280+0000 7f9df8102640 1 --2- 172.21.15.1:0/3160091437 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ddc06b500 0x7f9ddc06d9c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9de8036000 tx=0x7f9de8008030 comp rx=0 tx=0).stop 2024-03-06T07:21:04.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.280+0000 7f9df8102640 1 -- 172.21.15.1:0/3160091437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df0152650 msgr2=0x7f9df014e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.280+0000 7f9df8102640 1 --2- 172.21.15.1:0/3160091437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df0152650 0x7f9df014e7a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9de4007e40 tx=0x7f9de400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.280+0000 7f9df8102640 1 -- 172.21.15.1:0/3160091437 shutdown_connections 2024-03-06T07:21:04.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.280+0000 7f9df8102640 1 --2- 172.21.15.1:0/3160091437 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ddc06b500 0x7f9ddc06d9c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.280+0000 7f9df8102640 1 --2- 172.21.15.1:0/3160091437 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9df0149b60 0x7f9df01456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.280+0000 7f9df8102640 1 --2- 172.21.15.1:0/3160091437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df0152650 0x7f9df014e7a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.280+0000 7f9df8102640 1 --2- 172.21.15.1:0/3160091437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9df00bf9d0 0x7f9df014ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.280+0000 7f9df8102640 1 -- 172.21.15.1:0/3160091437 >> 172.21.15.1:0/3160091437 conn(0x7f9df0066ee0 msgr2=0x7f9df00c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:04.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.280+0000 7f9df8102640 1 -- 172.21.15.1:0/3160091437 shutdown_connections 2024-03-06T07:21:04.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.280+0000 7f9df8102640 1 -- 172.21.15.1:0/3160091437 wait complete. 2024-03-06T07:21:04.293 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:21:04.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.395+0000 7f31afe15640 1 Processor -- start 2024-03-06T07:21:04.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.395+0000 7f31afe15640 1 -- start start 2024-03-06T07:21:04.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.395+0000 7f31afe15640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f31a8152650 0x7f31a8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.395+0000 7f31afe15640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a80c2fc0 con 0x7f31a80c6b10 2024-03-06T07:21:04.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.395+0000 7f31afe15640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a8158da0 con 0x7f31a80bf9d0 2024-03-06T07:21:04.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.395+0000 7f31afe15640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a8155800 con 0x7f31a8152650 2024-03-06T07:21:04.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.396+0000 7f31adbb1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f31a8152650 0x7f31a8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.396+0000 7f31ad3b0640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f31a80c6b10 0x7f31a8158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59172/0 (socket says 172.21.15.1:59172) 2024-03-06T07:21:04.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.396+0000 7f31ad3b0640 1 -- 172.21.15.1:0/1995423290 learned_addr learned my addr 172.21.15.1:0/1995423290 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:04.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.396+0000 7f31adbb1640 1 -- 172.21.15.1:0/1995423290 >> v1:172.21.15.98:6789/0 conn(0x7f31a80bf9d0 legacy=0x7f31a80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:04.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.396+0000 7f31adbb1640 1 -- 172.21.15.1:0/1995423290 >> v1:172.21.15.1:6789/0 conn(0x7f31a80c6b10 legacy=0x7f31a8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.396+0000 7f31adbb1640 1 -- 172.21.15.1:0/1995423290 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a80c6690 con 0x7f31a8152650 2024-03-06T07:21:04.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.397+0000 7f31adbb1640 1 --2- 172.21.15.1:0/1995423290 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a8152650 0x7f31a8155c90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f31a40097a0 tx=0x7f31a402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef6c6a05afc4c8e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:04.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.397+0000 7f31acbaf640 1 -- 172.21.15.1:0/1995423290 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a4035020 con 0x7f31a8152650 2024-03-06T07:21:04.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.397+0000 7f31acbaf640 1 -- 172.21.15.1:0/1995423290 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a4031640 con 0x7f31a8152650 2024-03-06T07:21:04.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.397+0000 7f31acbaf640 1 -- 172.21.15.1:0/1995423290 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a402d040 con 0x7f31a8152650 2024-03-06T07:21:04.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.397+0000 7f31afe15640 1 -- 172.21.15.1:0/1995423290 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a8152650 msgr2=0x7f31a8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.397+0000 7f31afe15640 1 --2- 172.21.15.1:0/1995423290 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a8152650 0x7f31a8155c90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f31a40097a0 tx=0x7f31a402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.397+0000 7f31afe15640 1 -- 172.21.15.1:0/1995423290 shutdown_connections 2024-03-06T07:21:04.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.397+0000 7f31afe15640 1 --2- 172.21.15.1:0/1995423290 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a8152650 0x7f31a8155c90 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.397+0000 7f31afe15640 1 -- 172.21.15.1:0/1995423290 >> 172.21.15.1:0/1995423290 conn(0x7f31a8066ee0 msgr2=0x7f31a80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:04.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.398+0000 7f31afe15640 1 -- 172.21.15.1:0/1995423290 shutdown_connections 2024-03-06T07:21:04.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.398+0000 7f31afe15640 1 -- 172.21.15.1:0/1995423290 wait complete. 2024-03-06T07:21:04.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.399+0000 7f31afe15640 1 Processor -- start 2024-03-06T07:21:04.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.399+0000 7f31afe15640 1 -- start start 2024-03-06T07:21:04.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.399+0000 7f31afe15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a80bf9d0 0x7f31a814b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.399+0000 7f31afe15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a8152650 0x7f31a814e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.399+0000 7f31afe15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a814fe70 0x7f31a81502f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.399+0000 7f31afe15640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a81551b0 con 0x7f31a8152650 2024-03-06T07:21:04.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.399+0000 7f31afe15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a80c6fa0 con 0x7f31a80bf9d0 2024-03-06T07:21:04.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.399+0000 7f31afe15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a8155490 con 0x7f31a814fe70 2024-03-06T07:21:04.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.399+0000 7f31ad3b0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a8152650 0x7f31a814e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.399+0000 7f31ad3b0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a8152650 0x7f31a814e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52194/0 (socket says 172.21.15.1:52194) 2024-03-06T07:21:04.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.399+0000 7f31ad3b0640 1 -- 172.21.15.1:0/1903307581 learned_addr learned my addr 172.21.15.1:0/1903307581 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:04.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.400+0000 7f31adbb1640 1 --2- 172.21.15.1:0/1903307581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a80bf9d0 0x7f31a814b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.400+0000 7f31ae3b2640 1 --2- 172.21.15.1:0/1903307581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a814fe70 0x7f31a81502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.400+0000 7f31ad3b0640 1 -- 172.21.15.1:0/1903307581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a814fe70 msgr2=0x7f31a81502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.400+0000 7f31ad3b0640 1 --2- 172.21.15.1:0/1903307581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a814fe70 0x7f31a81502f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.400+0000 7f31ad3b0640 1 -- 172.21.15.1:0/1903307581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a80bf9d0 msgr2=0x7f31a814b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.400+0000 7f31ad3b0640 1 --2- 172.21.15.1:0/1903307581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a80bf9d0 0x7f31a814b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.400+0000 7f31ad3b0640 1 -- 172.21.15.1:0/1903307581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a8152e10 con 0x7f31a8152650 2024-03-06T07:21:04.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.400+0000 7f31ae3b2640 1 --2- 172.21.15.1:0/1903307581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a814fe70 0x7f31a81502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:04.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.400+0000 7f31ad3b0640 1 --2- 172.21.15.1:0/1903307581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a8152650 0x7f31a814e680 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f319800c4d0 tx=0x7f319800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:04.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.400+0000 7f319e7fc640 1 -- 172.21.15.1:0/1903307581 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198017070 con 0x7f31a8152650 2024-03-06T07:21:04.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.400+0000 7f319e7fc640 1 -- 172.21.15.1:0/1903307581 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3198005ce0 con 0x7f31a8152650 2024-03-06T07:21:04.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.400+0000 7f31afe15640 1 -- 172.21.15.1:0/1903307581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a8154500 con 0x7f31a8152650 2024-03-06T07:21:04.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.400+0000 7f319e7fc640 1 -- 172.21.15.1:0/1903307581 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198007270 con 0x7f31a8152650 2024-03-06T07:21:04.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.400+0000 7f31afe15640 1 -- 172.21.15.1:0/1903307581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a81547c0 con 0x7f31a8152650 2024-03-06T07:21:04.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.401+0000 7f31afe15640 1 -- 172.21.15.1:0/1903307581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a81542f0 con 0x7f31a8152650 2024-03-06T07:21:04.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.402+0000 7f319e7fc640 1 -- 172.21.15.1:0/1903307581 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f319801a070 con 0x7f31a8152650 2024-03-06T07:21:04.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.402+0000 7f319e7fc640 1 --2- 172.21.15.1:0/1903307581 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f318c06b4d0 0x7f318c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.402+0000 7f319e7fc640 1 -- 172.21.15.1:0/1903307581 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3198089760 con 0x7f31a8152650 2024-03-06T07:21:04.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.402+0000 7f31adbb1640 1 --2- 172.21.15.1:0/1903307581 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f318c06b4d0 0x7f318c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.403+0000 7f31adbb1640 1 --2- 172.21.15.1:0/1903307581 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f318c06b4d0 0x7f318c06d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f31a4036000 tx=0x7f31a403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:04.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.406+0000 7f319e7fc640 1 -- 172.21.15.1:0/1903307581 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3198057c50 con 0x7f31a8152650 2024-03-06T07:21:04.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.610+0000 7f31afe15640 1 -- 172.21.15.1:0/1903307581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f31a8067fd0 con 0x7f31a8152650 2024-03-06T07:21:04.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.610+0000 7f319e7fc640 1 -- 172.21.15.1:0/1903307581 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f319805bcf0 con 0x7f31a8152650 2024-03-06T07:21:04.612 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:21:04.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.613+0000 7f31afe15640 1 -- 172.21.15.1:0/1903307581 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f318c06b4d0 msgr2=0x7f318c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.613+0000 7f31afe15640 1 --2- 172.21.15.1:0/1903307581 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f318c06b4d0 0x7f318c06d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f31a4036000 tx=0x7f31a403c040 comp rx=0 tx=0).stop 2024-03-06T07:21:04.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.613+0000 7f31afe15640 1 -- 172.21.15.1:0/1903307581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a8152650 msgr2=0x7f31a814e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.613+0000 7f31afe15640 1 --2- 172.21.15.1:0/1903307581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a8152650 0x7f31a814e680 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f319800c4d0 tx=0x7f319800c990 comp rx=0 tx=0).stop 2024-03-06T07:21:04.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.613+0000 7f31afe15640 1 -- 172.21.15.1:0/1903307581 shutdown_connections 2024-03-06T07:21:04.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.613+0000 7f31afe15640 1 --2- 172.21.15.1:0/1903307581 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f318c06b4d0 0x7f318c06d990 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.613+0000 7f31afe15640 1 --2- 172.21.15.1:0/1903307581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a814fe70 0x7f31a81502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.613+0000 7f31afe15640 1 --2- 172.21.15.1:0/1903307581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a8152650 0x7f31a814e680 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.613+0000 7f31afe15640 1 --2- 172.21.15.1:0/1903307581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a80bf9d0 0x7f31a814b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.613+0000 7f31afe15640 1 -- 172.21.15.1:0/1903307581 >> 172.21.15.1:0/1903307581 conn(0x7f31a8066ee0 msgr2=0x7f31a80672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:04.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.613+0000 7f31afe15640 1 -- 172.21.15.1:0/1903307581 shutdown_connections 2024-03-06T07:21:04.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.613+0000 7f31afe15640 1 -- 172.21.15.1:0/1903307581 wait complete. 2024-03-06T07:21:04.626 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:21:04.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.728+0000 7f12bc365640 1 Processor -- start 2024-03-06T07:21:04.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.728+0000 7f12bc365640 1 -- start start 2024-03-06T07:21:04.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.729+0000 7f12bc365640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f12b4152820 0x7f12b40c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.729+0000 7f12bc365640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12b40af6f0 0x7f12b415fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.729+0000 7f12bc365640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b40c7180 con 0x7f12b40c0670 2024-03-06T07:21:04.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.729+0000 7f12bc365640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b4155460 con 0x7f12b4152820 2024-03-06T07:21:04.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.729+0000 7f12bc365640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b40c6bc0 con 0x7f12b40af6f0 2024-03-06T07:21:04.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.729+0000 7f12b9900640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f12b40c0670 0x7f12b40c73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59188/0 (socket says 172.21.15.1:59188) 2024-03-06T07:21:04.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.729+0000 7f12b9900640 1 -- 172.21.15.1:0/2628732590 learned_addr learned my addr 172.21.15.1:0/2628732590 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.729+0000 7f12ba101640 1 --2- 172.21.15.1:0/2628732590 >> v2:172.21.15.98:3300/0 conn(0x7f12b4152820 0x7f12b40c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.729+0000 7f12b90ff640 1 -- 172.21.15.1:0/2628732590 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886279371 0 0) 0x7f12b40c7180 con 0x7f12b40c0670 2024-03-06T07:21:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.729+0000 7f12ba902640 1 --2- 172.21.15.1:0/2628732590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12b40af6f0 0x7f12b415fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.729+0000 7f12b90ff640 1 -- 172.21.15.1:0/2628732590 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a0003520 con 0x7f12b40c0670 2024-03-06T07:21:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12b90ff640 1 -- 172.21.15.1:0/2628732590 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3535564052 0 0) 0x7f12a0003520 con 0x7f12b40c0670 2024-03-06T07:21:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12b90ff640 1 -- 172.21.15.1:0/2628732590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12b40af6f0 msgr2=0x7f12b415fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12b90ff640 1 --2- 172.21.15.1:0/2628732590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12b40af6f0 0x7f12b415fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12b90ff640 1 -- 172.21.15.1:0/2628732590 >> v2:172.21.15.98:3300/0 conn(0x7f12b4152820 msgr2=0x7f12b40c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12b90ff640 1 --2- 172.21.15.1:0/2628732590 >> v2:172.21.15.98:3300/0 conn(0x7f12b4152820 0x7f12b40c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12b90ff640 1 -- 172.21.15.1:0/2628732590 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b40c6ea0 con 0x7f12b40c0670 2024-03-06T07:21:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12b90ff640 1 -- 172.21.15.1:0/2628732590 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f12a4001670 con 0x7f12b40c0670 2024-03-06T07:21:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12b90ff640 1 -- 172.21.15.1:0/2628732590 >> v1:172.21.15.1:6789/0 conn(0x7f12b40c0670 legacy=0x7f12b40c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12b90ff640 1 --2- 172.21.15.1:0/2628732590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12a00039b0 0x7f12a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12b90ff640 1 --2- 172.21.15.1:0/2628732590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12b40af6f0 0x7f12a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12b90ff640 1 --2- 172.21.15.1:0/2628732590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a0004d40 0x7f12a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12b90ff640 1 -- 172.21.15.1:0/2628732590 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b4155460 con 0x7f12a0004d40 2024-03-06T07:21:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12b90ff640 1 -- 172.21.15.1:0/2628732590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b40c6bc0 con 0x7f12a00039b0 2024-03-06T07:21:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12b90ff640 1 -- 172.21.15.1:0/2628732590 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a000b9c0 con 0x7f12b40af6f0 2024-03-06T07:21:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12ba902640 1 --2- 172.21.15.1:0/2628732590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a0004d40 0x7f12a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12ba101640 1 --2- 172.21.15.1:0/2628732590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12a00039b0 0x7f12a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12b9900640 1 --2- 172.21.15.1:0/2628732590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12b40af6f0 0x7f12a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12ba902640 1 -- 172.21.15.1:0/2628732590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12b40af6f0 msgr2=0x7f12a0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12ba902640 1 --2- 172.21.15.1:0/2628732590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12b40af6f0 0x7f12a0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12ba902640 1 -- 172.21.15.1:0/2628732590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12a00039b0 msgr2=0x7f12a0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12ba902640 1 --2- 172.21.15.1:0/2628732590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12a00039b0 0x7f12a0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.730+0000 7f12ba902640 1 -- 172.21.15.1:0/2628732590 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a0003520 con 0x7f12a0004d40 2024-03-06T07:21:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.731+0000 7f12ba902640 1 --2- 172.21.15.1:0/2628732590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a0004d40 0x7f12a00051c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009f90 tx=0x7f12a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13dbb01328c43df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.731+0000 7f12b90ff640 1 -- 172.21.15.1:0/2628732590 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac039070 con 0x7f12a0004d40 2024-03-06T07:21:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.731+0000 7f12b90ff640 1 -- 172.21.15.1:0/2628732590 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ac02cbd0 con 0x7f12a0004d40 2024-03-06T07:21:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.731+0000 7f12b90ff640 1 -- 172.21.15.1:0/2628732590 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac009c10 con 0x7f12a0004d40 2024-03-06T07:21:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.732+0000 7f12bc365640 1 -- 172.21.15.1:0/2628732590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a0004d40 msgr2=0x7f12a00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.732+0000 7f12bc365640 1 --2- 172.21.15.1:0/2628732590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a0004d40 0x7f12a00051c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009f90 tx=0x7f12a0005830 comp rx=0 tx=0).stop 2024-03-06T07:21:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.732+0000 7f12ba902640 1 -- 172.21.15.1:0/2628732590 reap_dead start 2024-03-06T07:21:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.732+0000 7f12bc365640 1 -- 172.21.15.1:0/2628732590 shutdown_connections 2024-03-06T07:21:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.732+0000 7f12bc365640 1 -- 172.21.15.1:0/2628732590 >> 172.21.15.1:0/2628732590 conn(0x7f12b4072720 msgr2=0x7f12b4072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.732+0000 7f12bc365640 1 -- 172.21.15.1:0/2628732590 shutdown_connections 2024-03-06T07:21:04.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.732+0000 7f12bc365640 1 -- 172.21.15.1:0/2628732590 wait complete. 2024-03-06T07:21:04.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.733+0000 7f12bc365640 1 Processor -- start 2024-03-06T07:21:04.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.733+0000 7f12bc365640 1 -- start start 2024-03-06T07:21:04.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12bc365640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a0004d40 0x7f12b41ea4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12bc365640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12b4156630 0x7f12b40c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12bc365640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12b41ee180 0x7f12b4154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12bc365640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b40c7c80 con 0x7f12a0004d40 2024-03-06T07:21:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12bc365640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b40c7b00 con 0x7f12b41ee180 2024-03-06T07:21:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12bc365640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b4152c00 con 0x7f12b4156630 2024-03-06T07:21:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12ba101640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a0004d40 0x7f12b41ea4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12ba101640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a0004d40 0x7f12b41ea4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52206/0 (socket says 172.21.15.1:52206) 2024-03-06T07:21:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12ba101640 1 -- 172.21.15.1:0/783226401 learned_addr learned my addr 172.21.15.1:0/783226401 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12ba902640 1 --2- 172.21.15.1:0/783226401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12b41ee180 0x7f12b4154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12b9900640 1 --2- 172.21.15.1:0/783226401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12b4156630 0x7f12b40c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12ba101640 1 -- 172.21.15.1:0/783226401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12b4156630 msgr2=0x7f12b40c9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12ba101640 1 --2- 172.21.15.1:0/783226401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12b4156630 0x7f12b40c9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12ba101640 1 -- 172.21.15.1:0/783226401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12b41ee180 msgr2=0x7f12b4154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12ba101640 1 --2- 172.21.15.1:0/783226401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12b41ee180 0x7f12b4154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12ba101640 1 -- 172.21.15.1:0/783226401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b4156fe0 con 0x7f12a0004d40 2024-03-06T07:21:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12ba101640 1 --2- 172.21.15.1:0/783226401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a0004d40 0x7f12b41ea4a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f12b00119c0 tx=0x7f12b0011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12aaffd640 1 -- 172.21.15.1:0/783226401 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b000fb50 con 0x7f12a0004d40 2024-03-06T07:21:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12aaffd640 1 -- 172.21.15.1:0/783226401 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b000fcf0 con 0x7f12a0004d40 2024-03-06T07:21:04.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12bc365640 1 -- 172.21.15.1:0/783226401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b41572c0 con 0x7f12a0004d40 2024-03-06T07:21:04.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.734+0000 7f12aaffd640 1 -- 172.21.15.1:0/783226401 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b000fea0 con 0x7f12a0004d40 2024-03-06T07:21:04.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.735+0000 7f12bc365640 1 -- 172.21.15.1:0/783226401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b4074790 con 0x7f12a0004d40 2024-03-06T07:21:04.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.736+0000 7f12bc365640 1 -- 172.21.15.1:0/783226401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b40686e0 con 0x7f12a0004d40 2024-03-06T07:21:04.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.736+0000 7f12aaffd640 1 -- 172.21.15.1:0/783226401 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f12b0012040 con 0x7f12a0004d40 2024-03-06T07:21:04.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.737+0000 7f12aaffd640 1 --2- 172.21.15.1:0/783226401 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f129806b500 0x7f129806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.737+0000 7f12aaffd640 1 -- 172.21.15.1:0/783226401 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f12b008ab90 con 0x7f12a0004d40 2024-03-06T07:21:04.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.737+0000 7f12b9900640 1 --2- 172.21.15.1:0/783226401 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f129806b500 0x7f129806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.738+0000 7f12b9900640 1 --2- 172.21.15.1:0/783226401 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f129806b500 0x7f129806d9c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f12b4155390 tx=0x7f12a4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:04.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.741+0000 7f12aaffd640 1 -- 172.21.15.1:0/783226401 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f12b0019030 con 0x7f12a0004d40 2024-03-06T07:21:04.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.922+0000 7f12bc365640 1 -- 172.21.15.1:0/783226401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f12b41eda10 con 0x7f12a0004d40 2024-03-06T07:21:04.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.922+0000 7f12aaffd640 1 -- 172.21.15.1:0/783226401 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f12b0059760 con 0x7f12a0004d40 2024-03-06T07:21:04.925 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:21:04.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.925+0000 7f12bc365640 1 -- 172.21.15.1:0/783226401 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f129806b500 msgr2=0x7f129806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.925+0000 7f12bc365640 1 --2- 172.21.15.1:0/783226401 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f129806b500 0x7f129806d9c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f12b4155390 tx=0x7f12a4009040 comp rx=0 tx=0).stop 2024-03-06T07:21:04.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.925+0000 7f12bc365640 1 -- 172.21.15.1:0/783226401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a0004d40 msgr2=0x7f12b41ea4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.925+0000 7f12bc365640 1 --2- 172.21.15.1:0/783226401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a0004d40 0x7f12b41ea4a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f12b00119c0 tx=0x7f12b0011e80 comp rx=0 tx=0).stop 2024-03-06T07:21:04.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.925+0000 7f12bc365640 1 -- 172.21.15.1:0/783226401 shutdown_connections 2024-03-06T07:21:04.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.925+0000 7f12bc365640 1 --2- 172.21.15.1:0/783226401 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f129806b500 0x7f129806d9c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.925+0000 7f12bc365640 1 --2- 172.21.15.1:0/783226401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12b41ee180 0x7f12b4154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.925+0000 7f12bc365640 1 --2- 172.21.15.1:0/783226401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12b4156630 0x7f12b40c9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.925+0000 7f12bc365640 1 --2- 172.21.15.1:0/783226401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a0004d40 0x7f12b41ea4a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.925+0000 7f12bc365640 1 -- 172.21.15.1:0/783226401 >> 172.21.15.1:0/783226401 conn(0x7f12b4072720 msgr2=0x7f12b406cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:04.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.925+0000 7f12bc365640 1 -- 172.21.15.1:0/783226401 shutdown_connections 2024-03-06T07:21:04.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:04.925+0000 7f12bc365640 1 -- 172.21.15.1:0/783226401 wait complete. 2024-03-06T07:21:04.938 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:21:05.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.044+0000 7f98a617a640 1 Processor -- start 2024-03-06T07:21:05.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.044+0000 7f98a617a640 1 -- start start 2024-03-06T07:21:05.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.045+0000 7f98a617a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a0149a90 0x7f98a00c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.045+0000 7f98a617a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98a00c16c0 0x7f98a0156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.045+0000 7f98a617a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a01530a0 con 0x7f98a00c1260 2024-03-06T07:21:05.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.045+0000 7f98a617a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98a0153380 con 0x7f98a00c16c0 2024-03-06T07:21:05.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.045+0000 7f98a617a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a0153660 con 0x7f98a0149a90 2024-03-06T07:21:05.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.045+0000 7f989ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98a00c16c0 0x7f98a0156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.045+0000 7f989effd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f98a00c1260 0x7f98a01565a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59192/0 (socket says 172.21.15.1:59192) 2024-03-06T07:21:05.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.045+0000 7f989effd640 1 -- 172.21.15.1:0/853928033 learned_addr learned my addr 172.21.15.1:0/853928033 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:05.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.046+0000 7f989f7fe640 1 --2- 172.21.15.1:0/853928033 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a0149a90 0x7f98a00c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.046+0000 7f989ffff640 1 -- 172.21.15.1:0/853928033 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a0149a90 msgr2=0x7f98a00c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:05.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.046+0000 7f989ffff640 1 --2- 172.21.15.1:0/853928033 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a0149a90 0x7f98a00c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.046+0000 7f989ffff640 1 -- 172.21.15.1:0/853928033 >> v1:172.21.15.1:6789/0 conn(0x7f98a00c1260 legacy=0x7f98a01565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.046+0000 7f989ffff640 1 -- 172.21.15.1:0/853928033 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a0153940 con 0x7f98a00c16c0 2024-03-06T07:21:05.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.046+0000 7f989f7fe640 1 --2- 172.21.15.1:0/853928033 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a0149a90 0x7f98a00c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:05.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.046+0000 7f989ffff640 1 --2- 172.21.15.1:0/853928033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a00c16c0 0x7f98a0156cb0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9898009d90 tx=0x7f989802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25094874f4c37a1c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.047+0000 7f989e7fc640 1 -- 172.21.15.1:0/853928033 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9898039070 con 0x7f98a00c16c0 2024-03-06T07:21:05.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.047+0000 7f989e7fc640 1 -- 172.21.15.1:0/853928033 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989802cbc0 con 0x7f98a00c16c0 2024-03-06T07:21:05.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.047+0000 7f989e7fc640 1 -- 172.21.15.1:0/853928033 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9898034440 con 0x7f98a00c16c0 2024-03-06T07:21:05.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.047+0000 7f98a617a640 1 -- 172.21.15.1:0/853928033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a00c16c0 msgr2=0x7f98a0156cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.047+0000 7f98a617a640 1 --2- 172.21.15.1:0/853928033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a00c16c0 0x7f98a0156cb0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9898009d90 tx=0x7f989802be10 comp rx=0 tx=0).stop 2024-03-06T07:21:05.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.047+0000 7f98a617a640 1 -- 172.21.15.1:0/853928033 shutdown_connections 2024-03-06T07:21:05.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.047+0000 7f98a617a640 1 --2- 172.21.15.1:0/853928033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a00c16c0 0x7f98a0156cb0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.047+0000 7f98a617a640 1 --2- 172.21.15.1:0/853928033 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a0149a90 0x7f98a00c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.047+0000 7f98a617a640 1 -- 172.21.15.1:0/853928033 >> 172.21.15.1:0/853928033 conn(0x7f98a00c6860 msgr2=0x7f98a00c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:05.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.047+0000 7f98a617a640 1 -- 172.21.15.1:0/853928033 shutdown_connections 2024-03-06T07:21:05.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.047+0000 7f98a617a640 1 -- 172.21.15.1:0/853928033 wait complete. 2024-03-06T07:21:05.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.048+0000 7f98a617a640 1 Processor -- start 2024-03-06T07:21:05.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.048+0000 7f98a617a640 1 -- start start 2024-03-06T07:21:05.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.049+0000 7f98a617a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a00c1260 0x7f98a0148a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.049+0000 7f98a617a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a00c16c0 0x7f98a014f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.049+0000 7f98a617a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a0149a90 0x7f98a014fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.049+0000 7f98a617a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00c4ca0 con 0x7f98a00c1260 2024-03-06T07:21:05.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.049+0000 7f98a617a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a01571f0 con 0x7f98a0149a90 2024-03-06T07:21:05.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.049+0000 7f98a617a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a006f4a0 con 0x7f98a00c16c0 2024-03-06T07:21:05.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.049+0000 7f989f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a00c1260 0x7f98a0148a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.049+0000 7f989f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a00c1260 0x7f98a0148a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52226/0 (socket says 172.21.15.1:52226) 2024-03-06T07:21:05.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.049+0000 7f989f7fe640 1 -- 172.21.15.1:0/1515883082 learned_addr learned my addr 172.21.15.1:0/1515883082 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:05.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.049+0000 7f989effd640 1 --2- 172.21.15.1:0/1515883082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a00c16c0 0x7f98a014f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.049+0000 7f989ffff640 1 --2- 172.21.15.1:0/1515883082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a0149a90 0x7f98a014fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.049+0000 7f989f7fe640 1 -- 172.21.15.1:0/1515883082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a00c16c0 msgr2=0x7f98a014f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.049+0000 7f989f7fe640 1 --2- 172.21.15.1:0/1515883082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a00c16c0 0x7f98a014f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.049+0000 7f989f7fe640 1 -- 172.21.15.1:0/1515883082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a0149a90 msgr2=0x7f98a014fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.049+0000 7f989f7fe640 1 --2- 172.21.15.1:0/1515883082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a0149a90 0x7f98a014fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.049+0000 7f989f7fe640 1 -- 172.21.15.1:0/1515883082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a014d2e0 con 0x7f98a00c1260 2024-03-06T07:21:05.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.050+0000 7f989effd640 1 --2- 172.21.15.1:0/1515883082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a00c16c0 0x7f98a014f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:05.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.050+0000 7f989ffff640 1 --2- 172.21.15.1:0/1515883082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a0149a90 0x7f98a014fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:05.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.050+0000 7f989f7fe640 1 --2- 172.21.15.1:0/1515883082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a00c1260 0x7f98a0148a80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f988800c7f0 tx=0x7f988800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.050+0000 7f9897fff640 1 -- 172.21.15.1:0/1515883082 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9888015070 con 0x7f98a00c1260 2024-03-06T07:21:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.050+0000 7f9897fff640 1 -- 172.21.15.1:0/1515883082 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9888007820 con 0x7f98a00c1260 2024-03-06T07:21:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.050+0000 7f9897fff640 1 -- 172.21.15.1:0/1515883082 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9888007bd0 con 0x7f98a00c1260 2024-03-06T07:21:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.050+0000 7f98a617a640 1 -- 172.21.15.1:0/1515883082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a01487a0 con 0x7f98a00c1260 2024-03-06T07:21:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.050+0000 7f98a617a640 1 -- 172.21.15.1:0/1515883082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a014ace0 con 0x7f98a00c1260 2024-03-06T07:21:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.051+0000 7f98a617a640 1 -- 172.21.15.1:0/1515883082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a0067fd0 con 0x7f98a00c1260 2024-03-06T07:21:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.052+0000 7f9897fff640 1 -- 172.21.15.1:0/1515883082 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9888018070 con 0x7f98a00c1260 2024-03-06T07:21:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.052+0000 7f9897fff640 1 --2- 172.21.15.1:0/1515883082 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f988006b4d0 0x7f988006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.052+0000 7f9897fff640 1 -- 172.21.15.1:0/1515883082 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f988808a5e0 con 0x7f98a00c1260 2024-03-06T07:21:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.052+0000 7f989effd640 1 --2- 172.21.15.1:0/1515883082 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f988006b4d0 0x7f988006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.053+0000 7f989effd640 1 --2- 172.21.15.1:0/1515883082 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f988006b4d0 0x7f988006d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f98a00ab650 tx=0x7f9890009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.056+0000 7f9897fff640 1 -- 172.21.15.1:0/1515883082 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98880591b0 con 0x7f98a00c1260 2024-03-06T07:21:05.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.238+0000 7f98a617a640 1 -- 172.21.15.1:0/1515883082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f98a006f280 con 0x7f98a00c1260 2024-03-06T07:21:05.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.238+0000 7f9897fff640 1 -- 172.21.15.1:0/1515883082 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9888058ab0 con 0x7f98a00c1260 2024-03-06T07:21:05.240 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:21:05.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.240+0000 7f98a617a640 1 -- 172.21.15.1:0/1515883082 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f988006b4d0 msgr2=0x7f988006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.240+0000 7f98a617a640 1 --2- 172.21.15.1:0/1515883082 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f988006b4d0 0x7f988006d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f98a00ab650 tx=0x7f9890009040 comp rx=0 tx=0).stop 2024-03-06T07:21:05.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.240+0000 7f98a617a640 1 -- 172.21.15.1:0/1515883082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a00c1260 msgr2=0x7f98a0148a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.240+0000 7f98a617a640 1 --2- 172.21.15.1:0/1515883082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a00c1260 0x7f98a0148a80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f988800c7f0 tx=0x7f988800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.240+0000 7f98a617a640 1 -- 172.21.15.1:0/1515883082 shutdown_connections 2024-03-06T07:21:05.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.240+0000 7f98a617a640 1 --2- 172.21.15.1:0/1515883082 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f988006b4d0 0x7f988006d990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.240+0000 7f98a617a640 1 --2- 172.21.15.1:0/1515883082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a0149a90 0x7f98a014fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.240+0000 7f98a617a640 1 --2- 172.21.15.1:0/1515883082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a00c16c0 0x7f98a014f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.240+0000 7f98a617a640 1 --2- 172.21.15.1:0/1515883082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a00c1260 0x7f98a0148a80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.240+0000 7f98a617a640 1 -- 172.21.15.1:0/1515883082 >> 172.21.15.1:0/1515883082 conn(0x7f98a00c6860 msgr2=0x7f98a00a8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:05.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.240+0000 7f98a617a640 1 -- 172.21.15.1:0/1515883082 shutdown_connections 2024-03-06T07:21:05.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.240+0000 7f98a617a640 1 -- 172.21.15.1:0/1515883082 wait complete. 2024-03-06T07:21:05.256 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:21:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.357+0000 7fb22b8c9640 1 Processor -- start 2024-03-06T07:21:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.357+0000 7fb22b8c9640 1 -- start start 2024-03-06T07:21:05.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.358+0000 7fb22b8c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2240c32f0 0x7fb2241543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.358+0000 7fb22b8c9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb224070c80 0x7fb2240710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.358+0000 7fb22b8c9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb224150900 con 0x7fb224150520 2024-03-06T07:21:05.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.358+0000 7fb22b8c9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb224153ed0 con 0x7fb224070c80 2024-03-06T07:21:05.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.358+0000 7fb22b8c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb224071790 con 0x7fb2240c32f0 2024-03-06T07:21:05.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.358+0000 7fb229e66640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb224070c80 0x7fb2240710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.358+0000 7fb229665640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb224150520 0x7fb2241537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59200/0 (socket says 172.21.15.1:59200) 2024-03-06T07:21:05.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.358+0000 7fb229665640 1 -- 172.21.15.1:0/1967143265 learned_addr learned my addr 172.21.15.1:0/1967143265 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:05.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.358+0000 7fb228e64640 1 --2- 172.21.15.1:0/1967143265 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2240c32f0 0x7fb2241543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.358+0000 7fb229e66640 1 -- 172.21.15.1:0/1967143265 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2240c32f0 msgr2=0x7fb2241543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:05.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.358+0000 7fb229e66640 1 --2- 172.21.15.1:0/1967143265 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2240c32f0 0x7fb2241543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.358+0000 7fb229e66640 1 -- 172.21.15.1:0/1967143265 >> v1:172.21.15.1:6789/0 conn(0x7fb224150520 legacy=0x7fb2241537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.358+0000 7fb21bfff640 1 -- 172.21.15.1:0/1967143265 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971494030 0 0) 0x7fb224150900 con 0x7fb224150520 2024-03-06T07:21:05.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.358+0000 7fb229e66640 1 -- 172.21.15.1:0/1967143265 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb224157050 con 0x7fb224070c80 2024-03-06T07:21:05.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.359+0000 7fb229e66640 1 --2- 172.21.15.1:0/1967143265 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb224070c80 0x7fb2240710f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb21c009d90 tx=0x7fb21c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e3fb4b257e400f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.359+0000 7fb21bfff640 1 -- 172.21.15.1:0/1967143265 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21c039070 con 0x7fb224070c80 2024-03-06T07:21:05.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.359+0000 7fb21bfff640 1 -- 172.21.15.1:0/1967143265 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21c02cc50 con 0x7fb224070c80 2024-03-06T07:21:05.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.359+0000 7fb21bfff640 1 -- 172.21.15.1:0/1967143265 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21c034440 con 0x7fb224070c80 2024-03-06T07:21:05.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.359+0000 7fb22b8c9640 1 -- 172.21.15.1:0/1967143265 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb224070c80 msgr2=0x7fb2240710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.360+0000 7fb22b8c9640 1 --2- 172.21.15.1:0/1967143265 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb224070c80 0x7fb2240710f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb21c009d90 tx=0x7fb21c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.360+0000 7fb22b8c9640 1 -- 172.21.15.1:0/1967143265 shutdown_connections 2024-03-06T07:21:05.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.360+0000 7fb22b8c9640 1 --2- 172.21.15.1:0/1967143265 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2240c32f0 0x7fb2241543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.360+0000 7fb22b8c9640 1 --2- 172.21.15.1:0/1967143265 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb224070c80 0x7fb2240710f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.360+0000 7fb22b8c9640 1 -- 172.21.15.1:0/1967143265 >> 172.21.15.1:0/1967143265 conn(0x7fb2240c88f0 msgr2=0x7fb2240c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:05.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.360+0000 7fb22b8c9640 1 -- 172.21.15.1:0/1967143265 shutdown_connections 2024-03-06T07:21:05.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.360+0000 7fb22b8c9640 1 -- 172.21.15.1:0/1967143265 wait complete. 2024-03-06T07:21:05.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.361+0000 7fb22b8c9640 1 Processor -- start 2024-03-06T07:21:05.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.361+0000 7fb22b8c9640 1 -- start start 2024-03-06T07:21:05.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.361+0000 7fb22b8c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb224070c80 0x7fb224155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.361+0000 7fb22b8c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2240c32f0 0x7fb2241e5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.361+0000 7fb22b8c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb224150520 0x7fb2241e5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.361+0000 7fb22b8c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb224157330 con 0x7fb224070c80 2024-03-06T07:21:05.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.361+0000 7fb22b8c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2241532a0 con 0x7fb224150520 2024-03-06T07:21:05.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.361+0000 7fb22b8c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2240c7ec0 con 0x7fb2240c32f0 2024-03-06T07:21:05.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.362+0000 7fb229665640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb224070c80 0x7fb224155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.362+0000 7fb229665640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb224070c80 0x7fb224155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52242/0 (socket says 172.21.15.1:52242) 2024-03-06T07:21:05.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.362+0000 7fb229665640 1 -- 172.21.15.1:0/2620004786 learned_addr learned my addr 172.21.15.1:0/2620004786 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:05.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.362+0000 7fb229e66640 1 --2- 172.21.15.1:0/2620004786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb224150520 0x7fb2241e5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.362+0000 7fb228e64640 1 --2- 172.21.15.1:0/2620004786 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2240c32f0 0x7fb2241e5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.362+0000 7fb229665640 1 -- 172.21.15.1:0/2620004786 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2240c32f0 msgr2=0x7fb2241e5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.362+0000 7fb229665640 1 --2- 172.21.15.1:0/2620004786 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2240c32f0 0x7fb2241e5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.362+0000 7fb229665640 1 -- 172.21.15.1:0/2620004786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb224150520 msgr2=0x7fb2241e5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.362+0000 7fb229665640 1 --2- 172.21.15.1:0/2620004786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb224150520 0x7fb2241e5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.362+0000 7fb229665640 1 -- 172.21.15.1:0/2620004786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2240ab890 con 0x7fb224070c80 2024-03-06T07:21:05.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.362+0000 7fb229665640 1 --2- 172.21.15.1:0/2620004786 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb224070c80 0x7fb224155c30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb22000c460 tx=0x7fb22000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.362+0000 7fb219ffb640 1 -- 172.21.15.1:0/2620004786 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb220018070 con 0x7fb224070c80 2024-03-06T07:21:05.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.362+0000 7fb219ffb640 1 -- 172.21.15.1:0/2620004786 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb220007500 con 0x7fb224070c80 2024-03-06T07:21:05.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.362+0000 7fb22b8c9640 1 -- 172.21.15.1:0/2620004786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2241ec600 con 0x7fb224070c80 2024-03-06T07:21:05.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.362+0000 7fb219ffb640 1 -- 172.21.15.1:0/2620004786 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb220004a20 con 0x7fb224070c80 2024-03-06T07:21:05.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.362+0000 7fb22b8c9640 1 -- 172.21.15.1:0/2620004786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2240abb30 con 0x7fb224070c80 2024-03-06T07:21:05.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.363+0000 7fb22b8c9640 1 -- 172.21.15.1:0/2620004786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2240686e0 con 0x7fb224070c80 2024-03-06T07:21:05.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.364+0000 7fb219ffb640 1 -- 172.21.15.1:0/2620004786 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb22001b070 con 0x7fb224070c80 2024-03-06T07:21:05.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.365+0000 7fb219ffb640 1 --2- 172.21.15.1:0/2620004786 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb20806b4d0 0x7fb20806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.365+0000 7fb219ffb640 1 -- 172.21.15.1:0/2620004786 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb220089710 con 0x7fb224070c80 2024-03-06T07:21:05.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.365+0000 7fb228e64640 1 --2- 172.21.15.1:0/2620004786 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb20806b4d0 0x7fb20806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.365+0000 7fb228e64640 1 --2- 172.21.15.1:0/2620004786 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb20806b4d0 0x7fb20806d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb22414c5d0 tx=0x7fb214008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.369+0000 7fb219ffb640 1 -- 172.21.15.1:0/2620004786 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2200582e0 con 0x7fb224070c80 2024-03-06T07:21:05.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.562+0000 7fb22b8c9640 1 -- 172.21.15.1:0/2620004786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb2240c7b10 con 0x7fb224070c80 2024-03-06T07:21:05.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.562+0000 7fb219ffb640 1 -- 172.21.15.1:0/2620004786 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb220057be0 con 0x7fb224070c80 2024-03-06T07:21:05.564 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-06T07:21:05.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.564+0000 7fb22b8c9640 1 -- 172.21.15.1:0/2620004786 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb20806b4d0 msgr2=0x7fb20806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.564+0000 7fb22b8c9640 1 --2- 172.21.15.1:0/2620004786 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb20806b4d0 0x7fb20806d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb22414c5d0 tx=0x7fb214008040 comp rx=0 tx=0).stop 2024-03-06T07:21:05.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.564+0000 7fb22b8c9640 1 -- 172.21.15.1:0/2620004786 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb224070c80 msgr2=0x7fb224155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.564+0000 7fb22b8c9640 1 --2- 172.21.15.1:0/2620004786 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb224070c80 0x7fb224155c30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb22000c460 tx=0x7fb22000c920 comp rx=0 tx=0).stop 2024-03-06T07:21:05.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.564+0000 7fb22b8c9640 1 -- 172.21.15.1:0/2620004786 shutdown_connections 2024-03-06T07:21:05.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.564+0000 7fb22b8c9640 1 --2- 172.21.15.1:0/2620004786 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb20806b4d0 0x7fb20806d990 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.564+0000 7fb22b8c9640 1 --2- 172.21.15.1:0/2620004786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb224150520 0x7fb2241e5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.564+0000 7fb22b8c9640 1 --2- 172.21.15.1:0/2620004786 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2240c32f0 0x7fb2241e5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.564+0000 7fb22b8c9640 1 --2- 172.21.15.1:0/2620004786 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb224070c80 0x7fb224155c30 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.564+0000 7fb22b8c9640 1 -- 172.21.15.1:0/2620004786 >> 172.21.15.1:0/2620004786 conn(0x7fb2240c88f0 msgr2=0x7fb2240c6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:05.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.564+0000 7fb22b8c9640 1 -- 172.21.15.1:0/2620004786 shutdown_connections 2024-03-06T07:21:05.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.564+0000 7fb22b8c9640 1 -- 172.21.15.1:0/2620004786 wait complete. 2024-03-06T07:21:05.577 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:21:05.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:21:05.577 DEBUG:teuthology.orchestra.run.smithi001:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-03-06T07:21:05.604 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-06T07:21:05.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:21:05.605 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:21:05.660 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:21:05.663 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:21:05.684 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:21:05.684 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:21:05.743 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:21:05.743 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:21:05.801 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:05.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.902+0000 7fd5136cf640 1 Processor -- start 2024-03-06T07:21:05.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.902+0000 7fd5136cf640 1 -- start start 2024-03-06T07:21:05.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.903+0000 7fd5136cf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd50c152650 0x7fd50c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.903+0000 7fd5136cf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd50c0c6b10 0x7fd50c158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.903+0000 7fd5136cf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c0c2fc0 con 0x7fd50c0bf9d0 2024-03-06T07:21:05.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.903+0000 7fd5136cf640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd50c155800 con 0x7fd50c152650 2024-03-06T07:21:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.903+0000 7fd5136cf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c0c6690 con 0x7fd50c0c6b10 2024-03-06T07:21:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.903+0000 7fd511c6c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd50c0bf9d0 0x7fd50c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59216/0 (socket says 172.21.15.1:59216) 2024-03-06T07:21:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.903+0000 7fd511c6c640 1 -- 172.21.15.1:0/390842027 learned_addr learned my addr 172.21.15.1:0/390842027 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.903+0000 7fd510c6a640 1 --2- 172.21.15.1:0/390842027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd50c0c6b10 0x7fd50c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.903+0000 7fd51146b640 1 --2- 172.21.15.1:0/390842027 >> v2:172.21.15.98:3300/0 conn(0x7fd50c152650 0x7fd50c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.903+0000 7fd503fff640 1 -- 172.21.15.1:0/390842027 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893016847 0 0) 0x7fd50c0c2fc0 con 0x7fd50c0bf9d0 2024-03-06T07:21:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.903+0000 7fd503fff640 1 -- 172.21.15.1:0/390842027 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f8003520 con 0x7fd50c0bf9d0 2024-03-06T07:21:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.903+0000 7fd51146b640 1 -- 172.21.15.1:0/390842027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd50c0c6b10 msgr2=0x7fd50c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.903+0000 7fd51146b640 1 --2- 172.21.15.1:0/390842027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd50c0c6b10 0x7fd50c158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.903+0000 7fd51146b640 1 -- 172.21.15.1:0/390842027 >> v1:172.21.15.1:6789/0 conn(0x7fd50c0bf9d0 legacy=0x7fd50c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.903+0000 7fd503fff640 1 -- 172.21.15.1:0/390842027 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2315446168 0 0) 0x7fd4f8003520 con 0x7fd50c0bf9d0 2024-03-06T07:21:05.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.903+0000 7fd51146b640 1 -- 172.21.15.1:0/390842027 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c0c7280 con 0x7fd50c152650 2024-03-06T07:21:05.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.904+0000 7fd51146b640 1 --2- 172.21.15.1:0/390842027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd50c152650 0x7fd50c155c90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd5080097a0 tx=0x7fd50802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85675683f126e28a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.904+0000 7fd503fff640 1 -- 172.21.15.1:0/390842027 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd508037070 con 0x7fd50c152650 2024-03-06T07:21:05.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.904+0000 7fd503fff640 1 -- 172.21.15.1:0/390842027 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd508030510 con 0x7fd50c152650 2024-03-06T07:21:05.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.904+0000 7fd503fff640 1 -- 172.21.15.1:0/390842027 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5040019f0 con 0x7fd50c152650 2024-03-06T07:21:05.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.905+0000 7fd5136cf640 1 -- 172.21.15.1:0/390842027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd50c152650 msgr2=0x7fd50c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.905+0000 7fd5136cf640 1 --2- 172.21.15.1:0/390842027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd50c152650 0x7fd50c155c90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd5080097a0 tx=0x7fd50802b710 comp rx=0 tx=0).stop 2024-03-06T07:21:05.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.905+0000 7fd5136cf640 1 -- 172.21.15.1:0/390842027 shutdown_connections 2024-03-06T07:21:05.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.905+0000 7fd5136cf640 1 --2- 172.21.15.1:0/390842027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd50c0c6b10 0x7fd50c158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.905+0000 7fd5136cf640 1 --2- 172.21.15.1:0/390842027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd50c152650 0x7fd50c155c90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.905+0000 7fd5136cf640 1 -- 172.21.15.1:0/390842027 >> 172.21.15.1:0/390842027 conn(0x7fd50c066ee0 msgr2=0x7fd50c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:05.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.905+0000 7fd5136cf640 1 -- 172.21.15.1:0/390842027 shutdown_connections 2024-03-06T07:21:05.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.905+0000 7fd5136cf640 1 -- 172.21.15.1:0/390842027 wait complete. 2024-03-06T07:21:05.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.906+0000 7fd5136cf640 1 Processor -- start 2024-03-06T07:21:05.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.906+0000 7fd5136cf640 1 -- start start 2024-03-06T07:21:05.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd5136cf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd50c0bf9d0 0x7fd50c1538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd5136cf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd50c152650 0x7fd50c153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd5136cf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd50c1e12d0 0x7fd50c145660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd5136cf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c155490 con 0x7fd50c0bf9d0 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd5136cf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c1551b0 con 0x7fd50c1e12d0 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd5136cf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c152a30 con 0x7fd50c152650 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd51146b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd50c0bf9d0 0x7fd50c1538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd51146b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd50c0bf9d0 0x7fd50c1538c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52246/0 (socket says 172.21.15.1:52246) 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd51146b640 1 -- 172.21.15.1:0/2491554268 learned_addr learned my addr 172.21.15.1:0/2491554268 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd51146b640 1 -- 172.21.15.1:0/2491554268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd50c152650 msgr2=0x7fd50c153e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd510c6a640 1 --2- 172.21.15.1:0/2491554268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd50c152650 0x7fd50c153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd511c6c640 1 --2- 172.21.15.1:0/2491554268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd50c1e12d0 0x7fd50c145660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd51146b640 1 --2- 172.21.15.1:0/2491554268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd50c152650 0x7fd50c153e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd51146b640 1 -- 172.21.15.1:0/2491554268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd50c1e12d0 msgr2=0x7fd50c145660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd51146b640 1 --2- 172.21.15.1:0/2491554268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd50c1e12d0 0x7fd50c145660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd51146b640 1 -- 172.21.15.1:0/2491554268 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c0c9cd0 con 0x7fd50c0bf9d0 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd511c6c640 1 --2- 172.21.15.1:0/2491554268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd50c1e12d0 0x7fd50c145660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.907+0000 7fd51146b640 1 --2- 172.21.15.1:0/2491554268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd50c0bf9d0 0x7fd50c1538c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd50802b750 tx=0x7fd5080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.908+0000 7fd501ffb640 1 -- 172.21.15.1:0/2491554268 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd508040020 con 0x7fd50c0bf9d0 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.908+0000 7fd501ffb640 1 -- 172.21.15.1:0/2491554268 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd508038040 con 0x7fd50c0bf9d0 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.908+0000 7fd501ffb640 1 -- 172.21.15.1:0/2491554268 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd508037070 con 0x7fd50c0bf9d0 2024-03-06T07:21:05.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.908+0000 7fd5136cf640 1 -- 172.21.15.1:0/2491554268 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd50c0c9fb0 con 0x7fd50c0bf9d0 2024-03-06T07:21:05.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.908+0000 7fd5136cf640 1 -- 172.21.15.1:0/2491554268 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd50c0c9350 con 0x7fd50c0bf9d0 2024-03-06T07:21:05.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.909+0000 7fd5136cf640 1 -- 172.21.15.1:0/2491554268 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd50c067fd0 con 0x7fd50c0bf9d0 2024-03-06T07:21:05.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.909+0000 7fd501ffb640 1 -- 172.21.15.1:0/2491554268 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd508043020 con 0x7fd50c0bf9d0 2024-03-06T07:21:05.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.910+0000 7fd501ffb640 1 --2- 172.21.15.1:0/2491554268 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd4f006b4d0 0x7fd4f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.910+0000 7fd501ffb640 1 -- 172.21.15.1:0/2491554268 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd5080ab9d0 con 0x7fd50c0bf9d0 2024-03-06T07:21:05.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.910+0000 7fd510c6a640 1 --2- 172.21.15.1:0/2491554268 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd4f006b4d0 0x7fd4f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.910+0000 7fd510c6a640 1 --2- 172.21.15.1:0/2491554268 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd4f006b4d0 0x7fd4f006d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd50c1deab0 tx=0x7fd4fc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:05.914+0000 7fd501ffb640 1 -- 172.21.15.1:0/2491554268 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd50804a030 con 0x7fd50c0bf9d0 2024-03-06T07:21:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.096+0000 7fd5136cf640 1 -- 172.21.15.1:0/2491554268 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd50c0ca180 con 0x7fd50c0bf9d0 2024-03-06T07:21:06.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.097+0000 7fd501ffb640 1 -- 172.21.15.1:0/2491554268 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7fd50807a5a0 con 0x7fd50c0bf9d0 2024-03-06T07:21:06.099 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:06.100 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:06.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.099+0000 7fd5136cf640 1 -- 172.21.15.1:0/2491554268 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd4f006b4d0 msgr2=0x7fd4f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.099+0000 7fd5136cf640 1 --2- 172.21.15.1:0/2491554268 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd4f006b4d0 0x7fd4f006d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd50c1deab0 tx=0x7fd4fc008040 comp rx=0 tx=0).stop 2024-03-06T07:21:06.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.099+0000 7fd5136cf640 1 -- 172.21.15.1:0/2491554268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd50c0bf9d0 msgr2=0x7fd50c1538c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.099+0000 7fd5136cf640 1 --2- 172.21.15.1:0/2491554268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd50c0bf9d0 0x7fd50c1538c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd50802b750 tx=0x7fd5080046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.099+0000 7fd5136cf640 1 -- 172.21.15.1:0/2491554268 shutdown_connections 2024-03-06T07:21:06.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.099+0000 7fd5136cf640 1 --2- 172.21.15.1:0/2491554268 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd4f006b4d0 0x7fd4f006d990 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.099+0000 7fd5136cf640 1 --2- 172.21.15.1:0/2491554268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd50c1e12d0 0x7fd50c145660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.099+0000 7fd5136cf640 1 --2- 172.21.15.1:0/2491554268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd50c152650 0x7fd50c153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.099+0000 7fd5136cf640 1 --2- 172.21.15.1:0/2491554268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd50c0bf9d0 0x7fd50c1538c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.099+0000 7fd5136cf640 1 -- 172.21.15.1:0/2491554268 >> 172.21.15.1:0/2491554268 conn(0x7fd50c066ee0 msgr2=0x7fd50c0c5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:06.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.099+0000 7fd5136cf640 1 -- 172.21.15.1:0/2491554268 shutdown_connections 2024-03-06T07:21:06.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.099+0000 7fd5136cf640 1 -- 172.21.15.1:0/2491554268 wait complete. 2024-03-06T07:21:06.112 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:20:19.337560+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:20:20.172791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:20:23.884071+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:21:06.112 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:06.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.215+0000 7faea9ad4640 1 Processor -- start 2024-03-06T07:21:06.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.215+0000 7faea9ad4640 1 -- start start 2024-03-06T07:21:06.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.216+0000 7faea9ad4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea40a7280 0x7faea4072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.216+0000 7faea9ad4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faea40732b0 0x7faea4156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.216+0000 7faea9ad4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea4157130 con 0x7faea40c4ef0 2024-03-06T07:21:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.216+0000 7faea9ad4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea4152fb0 con 0x7faea40732b0 2024-03-06T07:21:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.216+0000 7faea9ad4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea4153290 con 0x7faea40a7280 2024-03-06T07:21:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.216+0000 7faea3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faea40732b0 0x7faea4156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.216+0000 7faea3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faea40732b0 0x7faea4156bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52258/0 (socket says 172.21.15.1:52258) 2024-03-06T07:21:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.216+0000 7faea3fff640 1 -- 172.21.15.1:0/3553417983 learned_addr learned my addr 172.21.15.1:0/3553417983 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.216+0000 7faea37fe640 1 --2- 172.21.15.1:0/3553417983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea40a7280 0x7faea4072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.216+0000 7faea3fff640 1 -- 172.21.15.1:0/3553417983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea40a7280 msgr2=0x7faea4072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.216+0000 7faea3fff640 1 --2- 172.21.15.1:0/3553417983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea40a7280 0x7faea4072d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.216+0000 7faea3fff640 1 -- 172.21.15.1:0/3553417983 >> v1:172.21.15.98:6789/0 conn(0x7faea40c4ef0 legacy=0x7faea41564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.216+0000 7faea3fff640 1 -- 172.21.15.1:0/3553417983 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea4153570 con 0x7faea40732b0 2024-03-06T07:21:06.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.217+0000 7faea3fff640 1 --2- 172.21.15.1:0/3553417983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faea40732b0 0x7faea4156bf0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fae9c009e50 tx=0x7fae9c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6314c894475aa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:06.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.218+0000 7faea27fc640 1 -- 172.21.15.1:0/3553417983 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9c02ca70 con 0x7faea40732b0 2024-03-06T07:21:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.218+0000 7faea27fc640 1 -- 172.21.15.1:0/3553417983 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9c02cc10 con 0x7faea40732b0 2024-03-06T07:21:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.218+0000 7faea27fc640 1 -- 172.21.15.1:0/3553417983 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9c0343f0 con 0x7faea40732b0 2024-03-06T07:21:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.218+0000 7faea9ad4640 1 -- 172.21.15.1:0/3553417983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faea40732b0 msgr2=0x7faea4156bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.218+0000 7faea9ad4640 1 --2- 172.21.15.1:0/3553417983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faea40732b0 0x7faea4156bf0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fae9c009e50 tx=0x7fae9c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.218+0000 7faea9ad4640 1 -- 172.21.15.1:0/3553417983 shutdown_connections 2024-03-06T07:21:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.218+0000 7faea9ad4640 1 --2- 172.21.15.1:0/3553417983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faea40732b0 0x7faea4156bf0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.218+0000 7faea9ad4640 1 --2- 172.21.15.1:0/3553417983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea40a7280 0x7faea4072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.218+0000 7faea9ad4640 1 -- 172.21.15.1:0/3553417983 >> 172.21.15.1:0/3553417983 conn(0x7faea40c7a50 msgr2=0x7faea40c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.218+0000 7faea9ad4640 1 -- 172.21.15.1:0/3553417983 shutdown_connections 2024-03-06T07:21:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.218+0000 7faea9ad4640 1 -- 172.21.15.1:0/3553417983 wait complete. 2024-03-06T07:21:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.219+0000 7faea9ad4640 1 Processor -- start 2024-03-06T07:21:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.219+0000 7faea9ad4640 1 -- start start 2024-03-06T07:21:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.220+0000 7faea9ad4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faea40732b0 0x7faea414c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea9ad4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faea40a7280 0x7faea41eca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea9ad4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea40c4ef0 0x7faea40a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea9ad4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea40c24e0 con 0x7faea40a7280 2024-03-06T07:21:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea9ad4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea4153b30 con 0x7faea40732b0 2024-03-06T07:21:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea9ad4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea40c2660 con 0x7faea40c4ef0 2024-03-06T07:21:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faea40a7280 0x7faea41eca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faea40a7280 0x7faea41eca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52266/0 (socket says 172.21.15.1:52266) 2024-03-06T07:21:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea2ffd640 1 -- 172.21.15.1:0/3589570764 learned_addr learned my addr 172.21.15.1:0/3589570764 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea37fe640 1 --2- 172.21.15.1:0/3589570764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faea40732b0 0x7faea414c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea3fff640 1 --2- 172.21.15.1:0/3589570764 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea40c4ef0 0x7faea40a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea2ffd640 1 -- 172.21.15.1:0/3589570764 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea40c4ef0 msgr2=0x7faea40a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea2ffd640 1 --2- 172.21.15.1:0/3589570764 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea40c4ef0 0x7faea40a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea2ffd640 1 -- 172.21.15.1:0/3589570764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faea40732b0 msgr2=0x7faea414c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea2ffd640 1 --2- 172.21.15.1:0/3589570764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faea40732b0 0x7faea414c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea2ffd640 1 -- 172.21.15.1:0/3589570764 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea414c690 con 0x7faea40a7280 2024-03-06T07:21:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea37fe640 1 --2- 172.21.15.1:0/3589570764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faea40732b0 0x7faea414c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea2ffd640 1 --2- 172.21.15.1:0/3589570764 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faea40a7280 0x7faea41eca40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fae9800c410 tx=0x7fae9800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7fae8bfff640 1 -- 172.21.15.1:0/3589570764 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae98017070 con 0x7faea40a7280 2024-03-06T07:21:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7fae8bfff640 1 -- 172.21.15.1:0/3589570764 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae980035f0 con 0x7faea40a7280 2024-03-06T07:21:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea9ad4640 1 -- 172.21.15.1:0/3589570764 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea41ecf80 con 0x7faea40a7280 2024-03-06T07:21:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7fae8bfff640 1 -- 172.21.15.1:0/3589570764 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae980039e0 con 0x7faea40a7280 2024-03-06T07:21:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.221+0000 7faea9ad4640 1 -- 172.21.15.1:0/3589570764 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea41dcd50 con 0x7faea40a7280 2024-03-06T07:21:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.223+0000 7faea9ad4640 1 -- 172.21.15.1:0/3589570764 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea40686e0 con 0x7faea40a7280 2024-03-06T07:21:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.223+0000 7fae8bfff640 1 -- 172.21.15.1:0/3589570764 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae9801a070 con 0x7faea40a7280 2024-03-06T07:21:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.224+0000 7fae8bfff640 1 --2- 172.21.15.1:0/3589570764 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fae9006b500 0x7fae9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.224+0000 7fae8bfff640 1 -- 172.21.15.1:0/3589570764 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fae980898c0 con 0x7faea40a7280 2024-03-06T07:21:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.224+0000 7faea37fe640 1 --2- 172.21.15.1:0/3589570764 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fae9006b500 0x7fae9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.225+0000 7faea37fe640 1 --2- 172.21.15.1:0/3589570764 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fae9006b500 0x7fae9006d9c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fae9400a9d0 tx=0x7fae94008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.228+0000 7fae8bfff640 1 -- 172.21.15.1:0/3589570764 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faea414c690 con 0x7faea40a7280 2024-03-06T07:21:06.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.410+0000 7faea9ad4640 1 -- 172.21.15.1:0/3589570764 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faea4150af0 con 0x7faea40a7280 2024-03-06T07:21:06.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.410+0000 7fae8bfff640 1 -- 172.21.15.1:0/3589570764 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7fae98058490 con 0x7faea40a7280 2024-03-06T07:21:06.412 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:21:06.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.413+0000 7faea9ad4640 1 -- 172.21.15.1:0/3589570764 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fae9006b500 msgr2=0x7fae9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.413+0000 7faea9ad4640 1 --2- 172.21.15.1:0/3589570764 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fae9006b500 0x7fae9006d9c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fae9400a9d0 tx=0x7fae94008040 comp rx=0 tx=0).stop 2024-03-06T07:21:06.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.413+0000 7faea9ad4640 1 -- 172.21.15.1:0/3589570764 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faea40a7280 msgr2=0x7faea41eca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.413+0000 7faea9ad4640 1 --2- 172.21.15.1:0/3589570764 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faea40a7280 0x7faea41eca40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fae9800c410 tx=0x7fae9800c8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.413+0000 7faea9ad4640 1 -- 172.21.15.1:0/3589570764 shutdown_connections 2024-03-06T07:21:06.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.413+0000 7faea9ad4640 1 --2- 172.21.15.1:0/3589570764 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fae9006b500 0x7fae9006d9c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.413+0000 7faea9ad4640 1 --2- 172.21.15.1:0/3589570764 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faea40c4ef0 0x7faea40a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.413+0000 7faea9ad4640 1 --2- 172.21.15.1:0/3589570764 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faea40a7280 0x7faea41eca40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.413+0000 7faea9ad4640 1 --2- 172.21.15.1:0/3589570764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faea40732b0 0x7faea414c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.413+0000 7faea9ad4640 1 -- 172.21.15.1:0/3589570764 >> 172.21.15.1:0/3589570764 conn(0x7faea40c7a50 msgr2=0x7faea40c7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:06.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.413+0000 7faea9ad4640 1 -- 172.21.15.1:0/3589570764 shutdown_connections 2024-03-06T07:21:06.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.413+0000 7faea9ad4640 1 -- 172.21.15.1:0/3589570764 wait complete. 2024-03-06T07:21:06.426 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:21:06.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.530+0000 7f35bd77a640 1 Processor -- start 2024-03-06T07:21:06.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.530+0000 7f35bd77a640 1 -- start start 2024-03-06T07:21:06.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.531+0000 7f35bd77a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35b8152920 0x7f35b806aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.531+0000 7f35bd77a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b8155dc0 con 0x7f35b8155220 2024-03-06T07:21:06.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.531+0000 7f35bd77a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b8158910 con 0x7f35b806b530 2024-03-06T07:21:06.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.531+0000 7f35bd77a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b8152d00 con 0x7f35b8152920 2024-03-06T07:21:06.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.531+0000 7f35b6ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35b8152920 0x7f35b806aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.531+0000 7f35b77fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f35b8155220 0x7f35b81556b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59220/0 (socket says 172.21.15.1:59220) 2024-03-06T07:21:06.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.531+0000 7f35b6ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35b8152920 0x7f35b806aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52268/0 (socket says 172.21.15.1:52268) 2024-03-06T07:21:06.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.531+0000 7f35b77fe640 1 -- 172.21.15.1:0/3404417453 learned_addr learned my addr 172.21.15.1:0/3404417453 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:06.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.531+0000 7f35b6ffd640 1 -- 172.21.15.1:0/3404417453 >> v1:172.21.15.98:6789/0 conn(0x7f35b806b530 legacy=0x7f35b80c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.531+0000 7f35b6ffd640 1 -- 172.21.15.1:0/3404417453 >> v1:172.21.15.1:6789/0 conn(0x7f35b8155220 legacy=0x7f35b81556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.531+0000 7f35b6ffd640 1 -- 172.21.15.1:0/3404417453 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b8158bf0 con 0x7f35b8152920 2024-03-06T07:21:06.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.531+0000 7f35b5ffb640 1 -- 172.21.15.1:0/3404417453 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120121287 0 0) 0x7f35b8155dc0 con 0x7f35b8155220 2024-03-06T07:21:06.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.532+0000 7f35b6ffd640 1 --2- 172.21.15.1:0/3404417453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8152920 0x7f35b806aff0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f35a80097a0 tx=0x7f35a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd06c278597b538a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:06.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.532+0000 7f35b5ffb640 1 -- 172.21.15.1:0/3404417453 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a8035020 con 0x7f35b8152920 2024-03-06T07:21:06.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.532+0000 7f35b5ffb640 1 -- 172.21.15.1:0/3404417453 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a8031830 con 0x7f35b8152920 2024-03-06T07:21:06.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.532+0000 7f35b5ffb640 1 -- 172.21.15.1:0/3404417453 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a8031c00 con 0x7f35b8152920 2024-03-06T07:21:06.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.533+0000 7f35bd77a640 1 -- 172.21.15.1:0/3404417453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8152920 msgr2=0x7f35b806aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.533+0000 7f35bd77a640 1 --2- 172.21.15.1:0/3404417453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8152920 0x7f35b806aff0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f35a80097a0 tx=0x7f35a802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.533+0000 7f35bd77a640 1 -- 172.21.15.1:0/3404417453 shutdown_connections 2024-03-06T07:21:06.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.533+0000 7f35bd77a640 1 --2- 172.21.15.1:0/3404417453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8152920 0x7f35b806aff0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.533+0000 7f35bd77a640 1 -- 172.21.15.1:0/3404417453 >> 172.21.15.1:0/3404417453 conn(0x7f35b80a9700 msgr2=0x7f35b80a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:06.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.533+0000 7f35bd77a640 1 -- 172.21.15.1:0/3404417453 shutdown_connections 2024-03-06T07:21:06.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.533+0000 7f35bd77a640 1 -- 172.21.15.1:0/3404417453 wait complete. 2024-03-06T07:21:06.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.534+0000 7f35bd77a640 1 Processor -- start 2024-03-06T07:21:06.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.534+0000 7f35bd77a640 1 -- start start 2024-03-06T07:21:06.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35bd77a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b806b530 0x7f35b806f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35bd77a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b8152920 0x7f35b8157850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35bd77a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8155220 0x7f35b806e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35bd77a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b8159240 con 0x7f35b8155220 2024-03-06T07:21:06.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35bd77a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b81596d0 con 0x7f35b8152920 2024-03-06T07:21:06.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35bd77a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b80c6380 con 0x7f35b806b530 2024-03-06T07:21:06.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35b77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8155220 0x7f35b806e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35b77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8155220 0x7f35b806e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52280/0 (socket says 172.21.15.1:52280) 2024-03-06T07:21:06.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35b77fe640 1 -- 172.21.15.1:0/947201116 learned_addr learned my addr 172.21.15.1:0/947201116 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:06.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35b6ffd640 1 --2- 172.21.15.1:0/947201116 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b806b530 0x7f35b806f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35b67fc640 1 --2- 172.21.15.1:0/947201116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b8152920 0x7f35b8157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35b77fe640 1 -- 172.21.15.1:0/947201116 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b806b530 msgr2=0x7f35b806f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35b77fe640 1 --2- 172.21.15.1:0/947201116 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b806b530 0x7f35b806f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35b77fe640 1 -- 172.21.15.1:0/947201116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b8152920 msgr2=0x7f35b8157850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35b77fe640 1 --2- 172.21.15.1:0/947201116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b8152920 0x7f35b8157850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35b77fe640 1 -- 172.21.15.1:0/947201116 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b81eba50 con 0x7f35b8155220 2024-03-06T07:21:06.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35b67fc640 1 --2- 172.21.15.1:0/947201116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b8152920 0x7f35b8157850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:06.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f35b77fe640 1 --2- 172.21.15.1:0/947201116 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8155220 0x7f35b806e3b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f35b000c730 tx=0x7f35b000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:06.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f359f7fe640 1 -- 172.21.15.1:0/947201116 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b0018070 con 0x7f35b8155220 2024-03-06T07:21:06.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.535+0000 7f359f7fe640 1 -- 172.21.15.1:0/947201116 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b000ce40 con 0x7f35b8155220 2024-03-06T07:21:06.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.536+0000 7f359f7fe640 1 -- 172.21.15.1:0/947201116 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b000a860 con 0x7f35b8155220 2024-03-06T07:21:06.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.536+0000 7f35bd77a640 1 -- 172.21.15.1:0/947201116 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b8145ac0 con 0x7f35b8155220 2024-03-06T07:21:06.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.536+0000 7f35bd77a640 1 -- 172.21.15.1:0/947201116 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b8157d90 con 0x7f35b8155220 2024-03-06T07:21:06.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.536+0000 7f35bd77a640 1 -- 172.21.15.1:0/947201116 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b81eb560 con 0x7f35b8155220 2024-03-06T07:21:06.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.537+0000 7f359f7fe640 1 -- 172.21.15.1:0/947201116 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f35b001b070 con 0x7f35b8155220 2024-03-06T07:21:06.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.538+0000 7f359f7fe640 1 --2- 172.21.15.1:0/947201116 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f35a406b4d0 0x7f35a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.538+0000 7f359f7fe640 1 -- 172.21.15.1:0/947201116 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f35b0089960 con 0x7f35b8155220 2024-03-06T07:21:06.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.538+0000 7f35b6ffd640 1 --2- 172.21.15.1:0/947201116 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f35a406b4d0 0x7f35a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.538+0000 7f35b6ffd640 1 --2- 172.21.15.1:0/947201116 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f35a406b4d0 0x7f35a406d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f35a8036000 tx=0x7f35a8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:06.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.542+0000 7f359f7fe640 1 -- 172.21.15.1:0/947201116 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f35b0057ee0 con 0x7f35b8155220 2024-03-06T07:21:06.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.723+0000 7f35bd77a640 1 -- 172.21.15.1:0/947201116 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f35b8067fd0 con 0x7f35b8155220 2024-03-06T07:21:06.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.723+0000 7f359f7fe640 1 -- 172.21.15.1:0/947201116 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f35b005bf80 con 0x7f35b8155220 2024-03-06T07:21:06.725 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:21:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.725+0000 7f35bd77a640 1 -- 172.21.15.1:0/947201116 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f35a406b4d0 msgr2=0x7f35a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.725+0000 7f35bd77a640 1 --2- 172.21.15.1:0/947201116 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f35a406b4d0 0x7f35a406d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f35a8036000 tx=0x7f35a8008030 comp rx=0 tx=0).stop 2024-03-06T07:21:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.725+0000 7f35bd77a640 1 -- 172.21.15.1:0/947201116 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8155220 msgr2=0x7f35b806e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.725+0000 7f35bd77a640 1 --2- 172.21.15.1:0/947201116 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8155220 0x7f35b806e3b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f35b000c730 tx=0x7f35b000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.725+0000 7f35bd77a640 1 -- 172.21.15.1:0/947201116 shutdown_connections 2024-03-06T07:21:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.725+0000 7f35bd77a640 1 --2- 172.21.15.1:0/947201116 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f35a406b4d0 0x7f35a406d990 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.725+0000 7f35bd77a640 1 --2- 172.21.15.1:0/947201116 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8155220 0x7f35b806e3b0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.725+0000 7f35bd77a640 1 --2- 172.21.15.1:0/947201116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b8152920 0x7f35b8157850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.725+0000 7f35bd77a640 1 --2- 172.21.15.1:0/947201116 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b806b530 0x7f35b806f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.725+0000 7f35bd77a640 1 -- 172.21.15.1:0/947201116 >> 172.21.15.1:0/947201116 conn(0x7f35b80a9700 msgr2=0x7f35b80a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.725+0000 7f35bd77a640 1 -- 172.21.15.1:0/947201116 shutdown_connections 2024-03-06T07:21:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.725+0000 7f35bd77a640 1 -- 172.21.15.1:0/947201116 wait complete. 2024-03-06T07:21:06.738 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:21:06.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.840+0000 7f5b7671e640 1 Processor -- start 2024-03-06T07:21:06.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.840+0000 7f5b7671e640 1 -- start start 2024-03-06T07:21:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.841+0000 7f5b7671e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5b7014cf90 0x7f5b70156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.841+0000 7f5b7671e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b700c7cc0 0x7f5b70154360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.841+0000 7f5b7671e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b70153a90 con 0x7f5b70070be0 2024-03-06T07:21:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.841+0000 7f5b7671e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b7014d370 con 0x7f5b7014cf90 2024-03-06T07:21:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.841+0000 7f5b7671e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b700c6190 con 0x7f5b700c7cc0 2024-03-06T07:21:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.841+0000 7f5b74cbb640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5b70070be0 0x7f5b700c67f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59224/0 (socket says 172.21.15.1:59224) 2024-03-06T07:21:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.841+0000 7f5b74cbb640 1 -- 172.21.15.1:0/790123365 learned_addr learned my addr 172.21.15.1:0/790123365 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6ffff640 1 --2- 172.21.15.1:0/790123365 >> v2:172.21.15.98:3300/0 conn(0x7f5b7014cf90 0x7f5b70156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6f7fe640 1 --2- 172.21.15.1:0/790123365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b700c7cc0 0x7f5b70154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6effd640 1 -- 172.21.15.1:0/790123365 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491571838 0 0) 0x7f5b70153a90 con 0x7f5b70070be0 2024-03-06T07:21:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6effd640 1 -- 172.21.15.1:0/790123365 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b58003520 con 0x7f5b70070be0 2024-03-06T07:21:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6effd640 1 -- 172.21.15.1:0/790123365 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 931752994 0 0) 0x7f5b58003520 con 0x7f5b70070be0 2024-03-06T07:21:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6effd640 1 -- 172.21.15.1:0/790123365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b700c7cc0 msgr2=0x7f5b70154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6effd640 1 --2- 172.21.15.1:0/790123365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b700c7cc0 0x7f5b70154360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6effd640 1 -- 172.21.15.1:0/790123365 >> v2:172.21.15.98:3300/0 conn(0x7f5b7014cf90 msgr2=0x7f5b70156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6effd640 1 --2- 172.21.15.1:0/790123365 >> v2:172.21.15.98:3300/0 conn(0x7f5b7014cf90 0x7f5b70156fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6effd640 1 -- 172.21.15.1:0/790123365 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b70153d70 con 0x7f5b70070be0 2024-03-06T07:21:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6effd640 1 -- 172.21.15.1:0/790123365 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f5b680019f0 con 0x7f5b70070be0 2024-03-06T07:21:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6effd640 1 -- 172.21.15.1:0/790123365 >> v1:172.21.15.1:6789/0 conn(0x7f5b70070be0 legacy=0x7f5b700c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6effd640 1 --2- 172.21.15.1:0/790123365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b700c7cc0 0x7f5b580039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6effd640 1 --2- 172.21.15.1:0/790123365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b58003f60 0x7f5b58004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6effd640 1 --2- 172.21.15.1:0/790123365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b58004d40 0x7f5b580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6effd640 1 -- 172.21.15.1:0/790123365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7014d370 con 0x7f5b58004d40 2024-03-06T07:21:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6effd640 1 -- 172.21.15.1:0/790123365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b700c6190 con 0x7f5b58003f60 2024-03-06T07:21:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6effd640 1 -- 172.21.15.1:0/790123365 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5800baf0 con 0x7f5b700c7cc0 2024-03-06T07:21:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b74cbb640 1 --2- 172.21.15.1:0/790123365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b58004d40 0x7f5b580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6ffff640 1 --2- 172.21.15.1:0/790123365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b700c7cc0 0x7f5b580039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b6f7fe640 1 --2- 172.21.15.1:0/790123365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b58003f60 0x7f5b58004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b74cbb640 1 -- 172.21.15.1:0/790123365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b700c7cc0 msgr2=0x7f5b580039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b74cbb640 1 --2- 172.21.15.1:0/790123365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b700c7cc0 0x7f5b580039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b74cbb640 1 -- 172.21.15.1:0/790123365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b58003f60 msgr2=0x7f5b58004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b74cbb640 1 --2- 172.21.15.1:0/790123365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b58003f60 0x7f5b58004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.842+0000 7f5b74cbb640 1 -- 172.21.15.1:0/790123365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b58003520 con 0x7f5b58004d40 2024-03-06T07:21:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.843+0000 7f5b74cbb640 1 --2- 172.21.15.1:0/790123365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b58004d40 0x7f5b580051c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5b6800a840 tx=0x7f5b58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d11b1667511b97 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:06.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.843+0000 7f5b6effd640 1 -- 172.21.15.1:0/790123365 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6800a460 con 0x7f5b58004d40 2024-03-06T07:21:06.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.843+0000 7f5b6effd640 1 -- 172.21.15.1:0/790123365 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b68034650 con 0x7f5b58004d40 2024-03-06T07:21:06.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.843+0000 7f5b6effd640 1 -- 172.21.15.1:0/790123365 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b68034a80 con 0x7f5b58004d40 2024-03-06T07:21:06.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.844+0000 7f5b7671e640 1 -- 172.21.15.1:0/790123365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b58004d40 msgr2=0x7f5b580051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.844+0000 7f5b7671e640 1 --2- 172.21.15.1:0/790123365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b58004d40 0x7f5b580051c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5b6800a840 tx=0x7f5b58005830 comp rx=0 tx=0).stop 2024-03-06T07:21:06.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.844+0000 7f5b74cbb640 1 -- 172.21.15.1:0/790123365 reap_dead start 2024-03-06T07:21:06.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.844+0000 7f5b7671e640 1 -- 172.21.15.1:0/790123365 shutdown_connections 2024-03-06T07:21:06.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.844+0000 7f5b7671e640 1 -- 172.21.15.1:0/790123365 >> 172.21.15.1:0/790123365 conn(0x7f5b700c88f0 msgr2=0x7f5b700c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:06.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.844+0000 7f5b7671e640 1 -- 172.21.15.1:0/790123365 shutdown_connections 2024-03-06T07:21:06.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.844+0000 7f5b7671e640 1 -- 172.21.15.1:0/790123365 wait complete. 2024-03-06T07:21:06.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.845+0000 7f5b7671e640 1 Processor -- start 2024-03-06T07:21:06.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.845+0000 7f5b7671e640 1 -- start start 2024-03-06T07:21:06.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.846+0000 7f5b7671e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b58004d40 0x7f5b7006cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.846+0000 7f5b7671e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b7014ea70 0x7f5b701eb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.846+0000 7f5b7671e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b7014fbd0 0x7f5b701ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.846+0000 7f5b7671e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b70156ac0 con 0x7f5b58004d40 2024-03-06T07:21:06.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.846+0000 7f5b7671e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b701567e0 con 0x7f5b7014ea70 2024-03-06T07:21:06.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.846+0000 7f5b7671e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b700c2e00 con 0x7f5b7014fbd0 2024-03-06T07:21:06.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.846+0000 7f5b6ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b58004d40 0x7f5b7006cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.846+0000 7f5b6ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b58004d40 0x7f5b7006cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52312/0 (socket says 172.21.15.1:52312) 2024-03-06T07:21:06.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.846+0000 7f5b6ffff640 1 -- 172.21.15.1:0/490905365 learned_addr learned my addr 172.21.15.1:0/490905365 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:06.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.846+0000 7f5b6f7fe640 1 --2- 172.21.15.1:0/490905365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b7014ea70 0x7f5b701eb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.846+0000 7f5b74cbb640 1 --2- 172.21.15.1:0/490905365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b7014fbd0 0x7f5b701ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.846+0000 7f5b6ffff640 1 -- 172.21.15.1:0/490905365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b7014fbd0 msgr2=0x7f5b701ebe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.846+0000 7f5b6ffff640 1 --2- 172.21.15.1:0/490905365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b7014fbd0 0x7f5b701ebe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.846+0000 7f5b6ffff640 1 -- 172.21.15.1:0/490905365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b7014ea70 msgr2=0x7f5b701eb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.846+0000 7f5b6ffff640 1 --2- 172.21.15.1:0/490905365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b7014ea70 0x7f5b701eb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.846+0000 7f5b6ffff640 1 -- 172.21.15.1:0/490905365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b700ada40 con 0x7f5b58004d40 2024-03-06T07:21:06.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.846+0000 7f5b6f7fe640 1 --2- 172.21.15.1:0/490905365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b7014ea70 0x7f5b701eb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:06.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.847+0000 7f5b6ffff640 1 --2- 172.21.15.1:0/490905365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b58004d40 0x7f5b7006cb30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5b60011a70 tx=0x7f5b60011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:06.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.847+0000 7f5b6cff9640 1 -- 172.21.15.1:0/490905365 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6000fb50 con 0x7f5b58004d40 2024-03-06T07:21:06.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.847+0000 7f5b6cff9640 1 -- 172.21.15.1:0/490905365 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6000fcf0 con 0x7f5b58004d40 2024-03-06T07:21:06.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.847+0000 7f5b6cff9640 1 -- 172.21.15.1:0/490905365 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6000fea0 con 0x7f5b58004d40 2024-03-06T07:21:06.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.847+0000 7f5b7671e640 1 -- 172.21.15.1:0/490905365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b700add20 con 0x7f5b58004d40 2024-03-06T07:21:06.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.847+0000 7f5b7671e640 1 -- 172.21.15.1:0/490905365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b701ec3c0 con 0x7f5b58004d40 2024-03-06T07:21:06.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.848+0000 7f5b7671e640 1 -- 172.21.15.1:0/490905365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b700686e0 con 0x7f5b58004d40 2024-03-06T07:21:06.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.848+0000 7f5b6cff9640 1 -- 172.21.15.1:0/490905365 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b60012040 con 0x7f5b58004d40 2024-03-06T07:21:06.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.849+0000 7f5b6cff9640 1 --2- 172.21.15.1:0/490905365 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b5c06b4d0 0x7f5b5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.849+0000 7f5b6cff9640 1 -- 172.21.15.1:0/490905365 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5b6008abb0 con 0x7f5b58004d40 2024-03-06T07:21:06.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.849+0000 7f5b6f7fe640 1 --2- 172.21.15.1:0/490905365 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b5c06b4d0 0x7f5b5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.850+0000 7f5b6f7fe640 1 --2- 172.21.15.1:0/490905365 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b5c06b4d0 0x7f5b5c06d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5b7014eeb0 tx=0x7f5b64008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:06.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:06.853+0000 7f5b6cff9640 1 -- 172.21.15.1:0/490905365 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b60019030 con 0x7f5b58004d40 2024-03-06T07:21:07.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.096+0000 7f5b7671e640 1 -- 172.21.15.1:0/490905365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5b7014d530 con 0x7f5b58004d40 2024-03-06T07:21:07.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.096+0000 7f5b6cff9640 1 -- 172.21.15.1:0/490905365 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5b60059780 con 0x7f5b58004d40 2024-03-06T07:21:07.098 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:21:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.099+0000 7f5b7671e640 1 -- 172.21.15.1:0/490905365 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b5c06b4d0 msgr2=0x7f5b5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.099+0000 7f5b7671e640 1 --2- 172.21.15.1:0/490905365 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b5c06b4d0 0x7f5b5c06d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5b7014eeb0 tx=0x7f5b64008040 comp rx=0 tx=0).stop 2024-03-06T07:21:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.099+0000 7f5b7671e640 1 -- 172.21.15.1:0/490905365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b58004d40 msgr2=0x7f5b7006cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.099+0000 7f5b7671e640 1 --2- 172.21.15.1:0/490905365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b58004d40 0x7f5b7006cb30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5b60011a70 tx=0x7f5b60011f30 comp rx=0 tx=0).stop 2024-03-06T07:21:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.099+0000 7f5b7671e640 1 -- 172.21.15.1:0/490905365 shutdown_connections 2024-03-06T07:21:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.099+0000 7f5b7671e640 1 --2- 172.21.15.1:0/490905365 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b5c06b4d0 0x7f5b5c06d990 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.099+0000 7f5b7671e640 1 --2- 172.21.15.1:0/490905365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b7014fbd0 0x7f5b701ebe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.099+0000 7f5b7671e640 1 --2- 172.21.15.1:0/490905365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b7014ea70 0x7f5b701eb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.099+0000 7f5b7671e640 1 --2- 172.21.15.1:0/490905365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b58004d40 0x7f5b7006cb30 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.099+0000 7f5b7671e640 1 -- 172.21.15.1:0/490905365 >> 172.21.15.1:0/490905365 conn(0x7f5b700c88f0 msgr2=0x7f5b70152ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:07.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.099+0000 7f5b7671e640 1 -- 172.21.15.1:0/490905365 shutdown_connections 2024-03-06T07:21:07.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.099+0000 7f5b7671e640 1 -- 172.21.15.1:0/490905365 wait complete. 2024-03-06T07:21:07.111 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:21:07.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.214+0000 7fd8638eb640 1 Processor -- start 2024-03-06T07:21:07.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.214+0000 7fd8638eb640 1 -- start start 2024-03-06T07:21:07.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.215+0000 7fd8638eb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd85c152690 0x7fd85c155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.215+0000 7fd8638eb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd85c1561e0 con 0x7fd85c156cd0 2024-03-06T07:21:07.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.215+0000 7fd8638eb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd85c0c3300 con 0x7fd85c0c6730 2024-03-06T07:21:07.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.215+0000 7fd8638eb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd85c0c35a0 con 0x7fd85c152690 2024-03-06T07:21:07.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.215+0000 7fd861687640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd85c152690 0x7fd85c155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.215+0000 7fd861e88640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd85c156cd0 0x7fd85c157120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59236/0 (socket says 172.21.15.1:59236) 2024-03-06T07:21:07.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.215+0000 7fd861e88640 1 -- 172.21.15.1:0/2051099942 learned_addr learned my addr 172.21.15.1:0/2051099942 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:07.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.215+0000 7fd861687640 1 -- 172.21.15.1:0/2051099942 >> v1:172.21.15.98:6789/0 conn(0x7fd85c0c6730 legacy=0x7fd85c0c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:07.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.215+0000 7fd861687640 1 -- 172.21.15.1:0/2051099942 >> v1:172.21.15.1:6789/0 conn(0x7fd85c156cd0 legacy=0x7fd85c157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.215+0000 7fd861687640 1 -- 172.21.15.1:0/2051099942 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85c0c3880 con 0x7fd85c152690 2024-03-06T07:21:07.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.215+0000 7fd85bfff640 1 -- 172.21.15.1:0/2051099942 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427457789 0 0) 0x7fd84c001490 con 0x7fd85c156cd0 2024-03-06T07:21:07.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.216+0000 7fd861687640 1 --2- 172.21.15.1:0/2051099942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c152690 0x7fd85c155ca0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd8540097a0 tx=0x7fd85402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bed7b6ab8cee33c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:07.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.217+0000 7fd85bfff640 1 -- 172.21.15.1:0/2051099942 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8540315a0 con 0x7fd85c152690 2024-03-06T07:21:07.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.217+0000 7fd85bfff640 1 -- 172.21.15.1:0/2051099942 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd854037070 con 0x7fd85c152690 2024-03-06T07:21:07.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.217+0000 7fd85bfff640 1 -- 172.21.15.1:0/2051099942 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd854031960 con 0x7fd85c152690 2024-03-06T07:21:07.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.217+0000 7fd8638eb640 1 -- 172.21.15.1:0/2051099942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c152690 msgr2=0x7fd85c155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.217+0000 7fd8638eb640 1 --2- 172.21.15.1:0/2051099942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c152690 0x7fd85c155ca0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd8540097a0 tx=0x7fd85402b840 comp rx=0 tx=0).stop 2024-03-06T07:21:07.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.217+0000 7fd8638eb640 1 -- 172.21.15.1:0/2051099942 shutdown_connections 2024-03-06T07:21:07.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.217+0000 7fd8638eb640 1 --2- 172.21.15.1:0/2051099942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c152690 0x7fd85c155ca0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.217+0000 7fd8638eb640 1 -- 172.21.15.1:0/2051099942 >> 172.21.15.1:0/2051099942 conn(0x7fd85c06b410 msgr2=0x7fd85c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:07.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.217+0000 7fd8638eb640 1 -- 172.21.15.1:0/2051099942 shutdown_connections 2024-03-06T07:21:07.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.217+0000 7fd8638eb640 1 -- 172.21.15.1:0/2051099942 wait complete. 2024-03-06T07:21:07.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.218+0000 7fd8638eb640 1 Processor -- start 2024-03-06T07:21:07.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.218+0000 7fd8638eb640 1 -- start start 2024-03-06T07:21:07.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.219+0000 7fd8638eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd85c0c6730 0x7fd85c0c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.219+0000 7fd8638eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c152690 0x7fd85c06cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.219+0000 7fd8638eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c156cd0 0x7fd85c1ee8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.219+0000 7fd8638eb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c0c5ee0 con 0x7fd85c152690 2024-03-06T07:21:07.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.219+0000 7fd8638eb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c1567b0 con 0x7fd85c156cd0 2024-03-06T07:21:07.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.219+0000 7fd8638eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c0c6060 con 0x7fd85c0c6730 2024-03-06T07:21:07.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.220+0000 7fd860e86640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c152690 0x7fd85c06cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.220+0000 7fd860e86640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c152690 0x7fd85c06cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52330/0 (socket says 172.21.15.1:52330) 2024-03-06T07:21:07.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.220+0000 7fd860e86640 1 -- 172.21.15.1:0/2806067622 learned_addr learned my addr 172.21.15.1:0/2806067622 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:07.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.220+0000 7fd861687640 1 --2- 172.21.15.1:0/2806067622 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd85c0c6730 0x7fd85c0c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.220+0000 7fd861e88640 1 --2- 172.21.15.1:0/2806067622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c156cd0 0x7fd85c1ee8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.220+0000 7fd860e86640 1 -- 172.21.15.1:0/2806067622 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd85c0c6730 msgr2=0x7fd85c0c7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.220+0000 7fd860e86640 1 --2- 172.21.15.1:0/2806067622 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd85c0c6730 0x7fd85c0c7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.220+0000 7fd860e86640 1 -- 172.21.15.1:0/2806067622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c156cd0 msgr2=0x7fd85c1ee8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.220+0000 7fd860e86640 1 --2- 172.21.15.1:0/2806067622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c156cd0 0x7fd85c1ee8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.220+0000 7fd860e86640 1 -- 172.21.15.1:0/2806067622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85c0c8520 con 0x7fd85c152690 2024-03-06T07:21:07.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.220+0000 7fd861e88640 1 --2- 172.21.15.1:0/2806067622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c156cd0 0x7fd85c1ee8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:07.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.220+0000 7fd860e86640 1 --2- 172.21.15.1:0/2806067622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c152690 0x7fd85c06cf50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd85000b3f0 tx=0x7fd85000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:07.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.220+0000 7fd859ffb640 1 -- 172.21.15.1:0/2806067622 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd850015020 con 0x7fd85c152690 2024-03-06T07:21:07.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.220+0000 7fd859ffb640 1 -- 172.21.15.1:0/2806067622 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd850011830 con 0x7fd85c152690 2024-03-06T07:21:07.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.220+0000 7fd859ffb640 1 -- 172.21.15.1:0/2806067622 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd850011c20 con 0x7fd85c152690 2024-03-06T07:21:07.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.220+0000 7fd8638eb640 1 -- 172.21.15.1:0/2806067622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd85c1ef690 con 0x7fd85c152690 2024-03-06T07:21:07.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.221+0000 7fd8638eb640 1 -- 172.21.15.1:0/2806067622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85c0c8ac0 con 0x7fd85c152690 2024-03-06T07:21:07.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.221+0000 7fd8638eb640 1 -- 172.21.15.1:0/2806067622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd85c067fd0 con 0x7fd85c152690 2024-03-06T07:21:07.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.222+0000 7fd859ffb640 1 -- 172.21.15.1:0/2806067622 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd850018020 con 0x7fd85c152690 2024-03-06T07:21:07.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.222+0000 7fd859ffb640 1 --2- 172.21.15.1:0/2806067622 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd84806b500 0x7fd84806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.222+0000 7fd859ffb640 1 -- 172.21.15.1:0/2806067622 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd85008a0b0 con 0x7fd85c152690 2024-03-06T07:21:07.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.222+0000 7fd861687640 1 --2- 172.21.15.1:0/2806067622 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd84806b500 0x7fd84806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.223+0000 7fd861687640 1 --2- 172.21.15.1:0/2806067622 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd84806b500 0x7fd84806d9c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd854036000 tx=0x7fd854008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:07.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.227+0000 7fd859ffb640 1 -- 172.21.15.1:0/2806067622 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd85001f030 con 0x7fd85c152690 2024-03-06T07:21:07.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.406+0000 7fd8638eb640 1 -- 172.21.15.1:0/2806067622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd85c0c2c40 con 0x7fd85c152690 2024-03-06T07:21:07.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.406+0000 7fd859ffb640 1 -- 172.21.15.1:0/2806067622 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd8500172c0 con 0x7fd85c152690 2024-03-06T07:21:07.409 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:21:07.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.409+0000 7fd8638eb640 1 -- 172.21.15.1:0/2806067622 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd84806b500 msgr2=0x7fd84806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.409+0000 7fd8638eb640 1 --2- 172.21.15.1:0/2806067622 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd84806b500 0x7fd84806d9c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd854036000 tx=0x7fd854008030 comp rx=0 tx=0).stop 2024-03-06T07:21:07.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.409+0000 7fd8638eb640 1 -- 172.21.15.1:0/2806067622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c152690 msgr2=0x7fd85c06cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.409+0000 7fd8638eb640 1 --2- 172.21.15.1:0/2806067622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c152690 0x7fd85c06cf50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd85000b3f0 tx=0x7fd85000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.409+0000 7fd8638eb640 1 -- 172.21.15.1:0/2806067622 shutdown_connections 2024-03-06T07:21:07.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.409+0000 7fd8638eb640 1 --2- 172.21.15.1:0/2806067622 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd84806b500 0x7fd84806d9c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.409+0000 7fd8638eb640 1 --2- 172.21.15.1:0/2806067622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c156cd0 0x7fd85c1ee8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.409+0000 7fd8638eb640 1 --2- 172.21.15.1:0/2806067622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c152690 0x7fd85c06cf50 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.409+0000 7fd8638eb640 1 --2- 172.21.15.1:0/2806067622 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd85c0c6730 0x7fd85c0c7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.409+0000 7fd8638eb640 1 -- 172.21.15.1:0/2806067622 >> 172.21.15.1:0/2806067622 conn(0x7fd85c06b410 msgr2=0x7fd85c06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:07.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.409+0000 7fd8638eb640 1 -- 172.21.15.1:0/2806067622 shutdown_connections 2024-03-06T07:21:07.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.409+0000 7fd8638eb640 1 -- 172.21.15.1:0/2806067622 wait complete. 2024-03-06T07:21:07.422 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:21:07.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.522+0000 7f654ff79640 1 Processor -- start 2024-03-06T07:21:07.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.522+0000 7f654ff79640 1 -- start start 2024-03-06T07:21:07.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.523+0000 7f654ff79640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f65480c6b10 0x7f6548155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.523+0000 7f654ff79640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480bf9d0 0x7f65480c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.523+0000 7f654ff79640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65480c2fc0 con 0x7f6548152650 2024-03-06T07:21:07.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.523+0000 7f654ff79640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6548155800 con 0x7f65480c6b10 2024-03-06T07:21:07.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.523+0000 7f654ff79640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6548158da0 con 0x7f65480bf9d0 2024-03-06T07:21:07.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.523+0000 7f654dd15640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6548152650 0x7f6548158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59240/0 (socket says 172.21.15.1:59240) 2024-03-06T07:21:07.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.523+0000 7f654dd15640 1 -- 172.21.15.1:0/1149074370 learned_addr learned my addr 172.21.15.1:0/1149074370 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:07.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654cd13640 1 -- 172.21.15.1:0/1149074370 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202322959 0 0) 0x7f65480c2fc0 con 0x7f6548152650 2024-03-06T07:21:07.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654cd13640 1 -- 172.21.15.1:0/1149074370 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6534003520 con 0x7f6548152650 2024-03-06T07:21:07.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654e516640 1 --2- 172.21.15.1:0/1149074370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480bf9d0 0x7f65480c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654d514640 1 --2- 172.21.15.1:0/1149074370 >> v2:172.21.15.98:3300/0 conn(0x7f65480c6b10 0x7f6548155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654cd13640 1 -- 172.21.15.1:0/1149074370 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3746741556 0 0) 0x7f6534003520 con 0x7f6548152650 2024-03-06T07:21:07.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654cd13640 1 -- 172.21.15.1:0/1149074370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480bf9d0 msgr2=0x7f65480c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:07.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654cd13640 1 --2- 172.21.15.1:0/1149074370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480bf9d0 0x7f65480c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654cd13640 1 -- 172.21.15.1:0/1149074370 >> v2:172.21.15.98:3300/0 conn(0x7f65480c6b10 msgr2=0x7f6548155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:07.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654cd13640 1 --2- 172.21.15.1:0/1149074370 >> v2:172.21.15.98:3300/0 conn(0x7f65480c6b10 0x7f6548155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654cd13640 1 -- 172.21.15.1:0/1149074370 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65480c6690 con 0x7f6548152650 2024-03-06T07:21:07.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654cd13640 1 -- 172.21.15.1:0/1149074370 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f6544001690 con 0x7f6548152650 2024-03-06T07:21:07.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654cd13640 1 -- 172.21.15.1:0/1149074370 >> v1:172.21.15.1:6789/0 conn(0x7f6548152650 legacy=0x7f6548158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654cd13640 1 --2- 172.21.15.1:0/1149074370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65340039b0 0x7f6534003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654cd13640 1 --2- 172.21.15.1:0/1149074370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480bf9d0 0x7f6534004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654cd13640 1 --2- 172.21.15.1:0/1149074370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6534004d40 0x7f65340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654cd13640 1 -- 172.21.15.1:0/1149074370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6548155800 con 0x7f65340039b0 2024-03-06T07:21:07.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654cd13640 1 -- 172.21.15.1:0/1149074370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6548158da0 con 0x7f6534004d40 2024-03-06T07:21:07.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654cd13640 1 -- 172.21.15.1:0/1149074370 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f653400b9f0 con 0x7f65480bf9d0 2024-03-06T07:21:07.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654dd15640 1 --2- 172.21.15.1:0/1149074370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65340039b0 0x7f6534003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654d514640 1 --2- 172.21.15.1:0/1149074370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480bf9d0 0x7f6534004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654e516640 1 --2- 172.21.15.1:0/1149074370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6534004d40 0x7f65340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654dd15640 1 -- 172.21.15.1:0/1149074370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480bf9d0 msgr2=0x7f6534004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654dd15640 1 --2- 172.21.15.1:0/1149074370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480bf9d0 0x7f6534004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654dd15640 1 -- 172.21.15.1:0/1149074370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6534004d40 msgr2=0x7f65340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654dd15640 1 --2- 172.21.15.1:0/1149074370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6534004d40 0x7f65340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.524+0000 7f654dd15640 1 -- 172.21.15.1:0/1149074370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6534003520 con 0x7f65340039b0 2024-03-06T07:21:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.525+0000 7f654dd15640 1 --2- 172.21.15.1:0/1149074370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65340039b0 0x7f6534003dd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6544007fd0 tx=0x7f6534005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92eab831941cf50d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.525+0000 7f654cd13640 1 -- 172.21.15.1:0/1149074370 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6544007d50 con 0x7f65340039b0 2024-03-06T07:21:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.525+0000 7f654cd13640 1 -- 172.21.15.1:0/1149074370 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6544030070 con 0x7f65340039b0 2024-03-06T07:21:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.525+0000 7f654cd13640 1 -- 172.21.15.1:0/1149074370 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6544005d10 con 0x7f65340039b0 2024-03-06T07:21:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.526+0000 7f654ff79640 1 -- 172.21.15.1:0/1149074370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65340039b0 msgr2=0x7f6534003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.526+0000 7f654ff79640 1 --2- 172.21.15.1:0/1149074370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65340039b0 0x7f6534003dd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6544007fd0 tx=0x7f6534005830 comp rx=0 tx=0).stop 2024-03-06T07:21:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.526+0000 7f654e516640 1 -- 172.21.15.1:0/1149074370 reap_dead start 2024-03-06T07:21:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.526+0000 7f654ff79640 1 -- 172.21.15.1:0/1149074370 shutdown_connections 2024-03-06T07:21:07.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.526+0000 7f654ff79640 1 -- 172.21.15.1:0/1149074370 >> 172.21.15.1:0/1149074370 conn(0x7f6548066ee0 msgr2=0x7f65480672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:07.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.526+0000 7f654ff79640 1 -- 172.21.15.1:0/1149074370 shutdown_connections 2024-03-06T07:21:07.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.526+0000 7f654ff79640 1 -- 172.21.15.1:0/1149074370 wait complete. 2024-03-06T07:21:07.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.527+0000 7f654ff79640 1 Processor -- start 2024-03-06T07:21:07.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.527+0000 7f654ff79640 1 -- start start 2024-03-06T07:21:07.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f654ff79640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65340039b0 0x7f65481de790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f654ff79640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65481567d0 0x7f65480c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f654ff79640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6548150fd0 0x7f654814a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f654ff79640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65481551b0 con 0x7f65340039b0 2024-03-06T07:21:07.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f654ff79640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65480c6fa0 con 0x7f65481567d0 2024-03-06T07:21:07.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f654ff79640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6548155490 con 0x7f6548150fd0 2024-03-06T07:21:07.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f654dd15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65340039b0 0x7f65481de790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f654dd15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65340039b0 0x7f65481de790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52348/0 (socket says 172.21.15.1:52348) 2024-03-06T07:21:07.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f654dd15640 1 -- 172.21.15.1:0/650878372 learned_addr learned my addr 172.21.15.1:0/650878372 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:07.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f654e516640 1 --2- 172.21.15.1:0/650878372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6548150fd0 0x7f654814a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f654d514640 1 --2- 172.21.15.1:0/650878372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65481567d0 0x7f65480c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f654dd15640 1 -- 172.21.15.1:0/650878372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6548150fd0 msgr2=0x7f654814a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f654dd15640 1 --2- 172.21.15.1:0/650878372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6548150fd0 0x7f654814a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f654dd15640 1 -- 172.21.15.1:0/650878372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65481567d0 msgr2=0x7f65480c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f654dd15640 1 --2- 172.21.15.1:0/650878372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65481567d0 0x7f65480c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f654dd15640 1 -- 172.21.15.1:0/650878372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6548156c30 con 0x7f65340039b0 2024-03-06T07:21:07.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f654dd15640 1 --2- 172.21.15.1:0/650878372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65340039b0 0x7f65481de790 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f654400c300 tx=0x7f65481561d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:07.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.528+0000 7f653e7fc640 1 -- 172.21.15.1:0/650878372 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6544001c90 con 0x7f65340039b0 2024-03-06T07:21:07.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.529+0000 7f653e7fc640 1 -- 172.21.15.1:0/650878372 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65440049f0 con 0x7f65340039b0 2024-03-06T07:21:07.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.529+0000 7f654ff79640 1 -- 172.21.15.1:0/650878372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6548151b20 con 0x7f65340039b0 2024-03-06T07:21:07.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.529+0000 7f653e7fc640 1 -- 172.21.15.1:0/650878372 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6544004dc0 con 0x7f65340039b0 2024-03-06T07:21:07.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.529+0000 7f654ff79640 1 -- 172.21.15.1:0/650878372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65481ee830 con 0x7f65340039b0 2024-03-06T07:21:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.529+0000 7f654ff79640 1 -- 172.21.15.1:0/650878372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6548067fd0 con 0x7f65340039b0 2024-03-06T07:21:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.530+0000 7f653e7fc640 1 -- 172.21.15.1:0/650878372 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6544007070 con 0x7f65340039b0 2024-03-06T07:21:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.530+0000 7f653e7fc640 1 --2- 172.21.15.1:0/650878372 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f652c06b4d0 0x7f652c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.531+0000 7f653e7fc640 1 -- 172.21.15.1:0/650878372 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f65440abc40 con 0x7f65340039b0 2024-03-06T07:21:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.531+0000 7f654d514640 1 --2- 172.21.15.1:0/650878372 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f652c06b4d0 0x7f652c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.531+0000 7f654d514640 1 --2- 172.21.15.1:0/650878372 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f652c06b4d0 0x7f652c06d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6548151840 tx=0x7f6538009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.538+0000 7f653e7fc640 1 -- 172.21.15.1:0/650878372 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6544046060 con 0x7f65340039b0 2024-03-06T07:21:07.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.747+0000 7f654ff79640 1 -- 172.21.15.1:0/650878372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f65480b06f0 con 0x7f65340039b0 2024-03-06T07:21:07.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.748+0000 7f653e7fc640 1 -- 172.21.15.1:0/650878372 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f654407a8f0 con 0x7f65340039b0 2024-03-06T07:21:07.750 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-06T07:21:07.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.750+0000 7f654ff79640 1 -- 172.21.15.1:0/650878372 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f652c06b4d0 msgr2=0x7f652c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.750+0000 7f654ff79640 1 --2- 172.21.15.1:0/650878372 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f652c06b4d0 0x7f652c06d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6548151840 tx=0x7f6538009190 comp rx=0 tx=0).stop 2024-03-06T07:21:07.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.750+0000 7f654ff79640 1 -- 172.21.15.1:0/650878372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65340039b0 msgr2=0x7f65481de790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.750+0000 7f654ff79640 1 --2- 172.21.15.1:0/650878372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65340039b0 0x7f65481de790 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f654400c300 tx=0x7f65481561d0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.750+0000 7f654ff79640 1 -- 172.21.15.1:0/650878372 shutdown_connections 2024-03-06T07:21:07.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.750+0000 7f654ff79640 1 --2- 172.21.15.1:0/650878372 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f652c06b4d0 0x7f652c06d990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.750+0000 7f654ff79640 1 --2- 172.21.15.1:0/650878372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6548150fd0 0x7f654814a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.750+0000 7f654ff79640 1 --2- 172.21.15.1:0/650878372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65481567d0 0x7f65480c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.750+0000 7f654ff79640 1 --2- 172.21.15.1:0/650878372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65340039b0 0x7f65481de790 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.750+0000 7f654ff79640 1 -- 172.21.15.1:0/650878372 >> 172.21.15.1:0/650878372 conn(0x7f6548066ee0 msgr2=0x7f65480c0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:07.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.750+0000 7f654ff79640 1 -- 172.21.15.1:0/650878372 shutdown_connections 2024-03-06T07:21:07.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.750+0000 7f654ff79640 1 -- 172.21.15.1:0/650878372 wait complete. 2024-03-06T07:21:07.766 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-06T07:21:07.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:21:07.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-03-06T07:21:07.801 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-03-06T07:21:07.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:21:07.801 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:21:07.875 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:21:07.875 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:07.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.975+0000 7f296a89a640 1 Processor -- start 2024-03-06T07:21:07.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.975+0000 7f296a89a640 1 -- start start 2024-03-06T07:21:07.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.975+0000 7f296a89a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f29640c6b10 0x7f2964155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.975+0000 7f296a89a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29640bf9d0 0x7f29640c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.975+0000 7f296a89a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29640c2fc0 con 0x7f2964152650 2024-03-06T07:21:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.975+0000 7f296a89a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2964155800 con 0x7f29640c6b10 2024-03-06T07:21:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.975+0000 7f296a89a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2964158da0 con 0x7f29640bf9d0 2024-03-06T07:21:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2963fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2964152650 0x7f2964158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59250/0 (socket says 172.21.15.1:59250) 2024-03-06T07:21:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2963fff640 1 -- 172.21.15.1:0/818688594 learned_addr learned my addr 172.21.15.1:0/818688594 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f29637fe640 1 --2- 172.21.15.1:0/818688594 >> v2:172.21.15.98:3300/0 conn(0x7f29640c6b10 0x7f2964155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2968e37640 1 --2- 172.21.15.1:0/818688594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29640bf9d0 0x7f29640c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2962ffd640 1 -- 172.21.15.1:0/818688594 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541234545 0 0) 0x7f29640c2fc0 con 0x7f2964152650 2024-03-06T07:21:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2962ffd640 1 -- 172.21.15.1:0/818688594 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f294c003520 con 0x7f2964152650 2024-03-06T07:21:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2962ffd640 1 -- 172.21.15.1:0/818688594 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2107699898 0 0) 0x7f294c003520 con 0x7f2964152650 2024-03-06T07:21:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2962ffd640 1 -- 172.21.15.1:0/818688594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29640bf9d0 msgr2=0x7f29640c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2962ffd640 1 --2- 172.21.15.1:0/818688594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29640bf9d0 0x7f29640c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2962ffd640 1 -- 172.21.15.1:0/818688594 >> v2:172.21.15.98:3300/0 conn(0x7f29640c6b10 msgr2=0x7f2964155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2962ffd640 1 --2- 172.21.15.1:0/818688594 >> v2:172.21.15.98:3300/0 conn(0x7f29640c6b10 0x7f2964155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2962ffd640 1 -- 172.21.15.1:0/818688594 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29640c6690 con 0x7f2964152650 2024-03-06T07:21:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2962ffd640 1 -- 172.21.15.1:0/818688594 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f2954001690 con 0x7f2964152650 2024-03-06T07:21:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2962ffd640 1 -- 172.21.15.1:0/818688594 >> v1:172.21.15.1:6789/0 conn(0x7f2964152650 legacy=0x7f2964158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2962ffd640 1 --2- 172.21.15.1:0/818688594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f294c0039b0 0x7f294c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2962ffd640 1 --2- 172.21.15.1:0/818688594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29640bf9d0 0x7f294c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2962ffd640 1 --2- 172.21.15.1:0/818688594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f294c004d40 0x7f294c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2962ffd640 1 -- 172.21.15.1:0/818688594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2964155800 con 0x7f294c0039b0 2024-03-06T07:21:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2962ffd640 1 -- 172.21.15.1:0/818688594 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2964158da0 con 0x7f294c004d40 2024-03-06T07:21:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2962ffd640 1 -- 172.21.15.1:0/818688594 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294c00b9f0 con 0x7f29640bf9d0 2024-03-06T07:21:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2963fff640 1 --2- 172.21.15.1:0/818688594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f294c0039b0 0x7f294c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f29637fe640 1 --2- 172.21.15.1:0/818688594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29640bf9d0 0x7f294c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.976+0000 7f2968e37640 1 --2- 172.21.15.1:0/818688594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f294c004d40 0x7f294c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.977+0000 7f2963fff640 1 -- 172.21.15.1:0/818688594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29640bf9d0 msgr2=0x7f294c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.977+0000 7f2963fff640 1 --2- 172.21.15.1:0/818688594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29640bf9d0 0x7f294c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.977+0000 7f2963fff640 1 -- 172.21.15.1:0/818688594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f294c004d40 msgr2=0x7f294c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.977+0000 7f2963fff640 1 --2- 172.21.15.1:0/818688594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f294c004d40 0x7f294c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.977+0000 7f2963fff640 1 -- 172.21.15.1:0/818688594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c003520 con 0x7f294c0039b0 2024-03-06T07:21:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.977+0000 7f29637fe640 1 --2- 172.21.15.1:0/818688594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29640bf9d0 0x7f294c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.977+0000 7f2968e37640 1 --2- 172.21.15.1:0/818688594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f294c004d40 0x7f294c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.977+0000 7f2963fff640 1 --2- 172.21.15.1:0/818688594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f294c0039b0 0x7f294c003dd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2954007fd0 tx=0x7f294c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8228cd8c00fc5f86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.977+0000 7f2962ffd640 1 -- 172.21.15.1:0/818688594 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2954007d50 con 0x7f294c0039b0 2024-03-06T07:21:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.978+0000 7f2962ffd640 1 -- 172.21.15.1:0/818688594 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2954030070 con 0x7f294c0039b0 2024-03-06T07:21:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.978+0000 7f2962ffd640 1 -- 172.21.15.1:0/818688594 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2954005d30 con 0x7f294c0039b0 2024-03-06T07:21:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.978+0000 7f296a89a640 1 -- 172.21.15.1:0/818688594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f294c0039b0 msgr2=0x7f294c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.978+0000 7f296a89a640 1 --2- 172.21.15.1:0/818688594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f294c0039b0 0x7f294c003dd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2954007fd0 tx=0x7f294c005830 comp rx=0 tx=0).stop 2024-03-06T07:21:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.978+0000 7f2968e37640 1 -- 172.21.15.1:0/818688594 reap_dead start 2024-03-06T07:21:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.978+0000 7f296a89a640 1 -- 172.21.15.1:0/818688594 shutdown_connections 2024-03-06T07:21:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.978+0000 7f296a89a640 1 -- 172.21.15.1:0/818688594 >> 172.21.15.1:0/818688594 conn(0x7f2964066ee0 msgr2=0x7f29640672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.978+0000 7f296a89a640 1 -- 172.21.15.1:0/818688594 shutdown_connections 2024-03-06T07:21:07.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.978+0000 7f296a89a640 1 -- 172.21.15.1:0/818688594 wait complete. 2024-03-06T07:21:07.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.979+0000 7f296a89a640 1 Processor -- start 2024-03-06T07:21:07.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.980+0000 7f296a89a640 1 -- start start 2024-03-06T07:21:07.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f296a89a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f294c0039b0 0x7f29641ebb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f296a89a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29641578c0 0x7f2964156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f296a89a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2964156f90 0x7f296413cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f296a89a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29641551b0 con 0x7f2964156f90 2024-03-06T07:21:07.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f296a89a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29640c6fa0 con 0x7f29641578c0 2024-03-06T07:21:07.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f296a89a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2964155490 con 0x7f294c0039b0 2024-03-06T07:21:07.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f2968e37640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2964156f90 0x7f296413cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f2968e37640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2964156f90 0x7f296413cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52358/0 (socket says 172.21.15.1:52358) 2024-03-06T07:21:07.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f2968e37640 1 -- 172.21.15.1:0/2868089195 learned_addr learned my addr 172.21.15.1:0/2868089195 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:07.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f2963fff640 1 --2- 172.21.15.1:0/2868089195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f294c0039b0 0x7f29641ebb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f2968e37640 1 -- 172.21.15.1:0/2868089195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f294c0039b0 msgr2=0x7f29641ebb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f2968e37640 1 --2- 172.21.15.1:0/2868089195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f294c0039b0 0x7f29641ebb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f29637fe640 1 --2- 172.21.15.1:0/2868089195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29641578c0 0x7f2964156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f2968e37640 1 -- 172.21.15.1:0/2868089195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29641578c0 msgr2=0x7f2964156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:07.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f2968e37640 1 --2- 172.21.15.1:0/2868089195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29641578c0 0x7f2964156a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:07.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f2968e37640 1 -- 172.21.15.1:0/2868089195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2964157410 con 0x7f2964156f90 2024-03-06T07:21:07.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f2968e37640 1 --2- 172.21.15.1:0/2868089195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2964156f90 0x7f296413cf70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f296408c760 tx=0x7f295c010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:07.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.981+0000 7f2960ff9640 1 -- 172.21.15.1:0/2868089195 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c01a030 con 0x7f2964156f90 2024-03-06T07:21:07.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.982+0000 7f2960ff9640 1 -- 172.21.15.1:0/2868089195 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295c010d20 con 0x7f2964156f90 2024-03-06T07:21:07.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.982+0000 7f296a89a640 1 -- 172.21.15.1:0/2868089195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f296414daf0 con 0x7f2964156f90 2024-03-06T07:21:07.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.982+0000 7f2960ff9640 1 -- 172.21.15.1:0/2868089195 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c003bd0 con 0x7f2964156f90 2024-03-06T07:21:07.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.982+0000 7f296a89a640 1 -- 172.21.15.1:0/2868089195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296414d530 con 0x7f2964156f90 2024-03-06T07:21:07.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.982+0000 7f296a89a640 1 -- 172.21.15.1:0/2868089195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2964067fd0 con 0x7f2964156f90 2024-03-06T07:21:07.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.983+0000 7f2960ff9640 1 -- 172.21.15.1:0/2868089195 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f295c005d10 con 0x7f2964156f90 2024-03-06T07:21:07.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.983+0000 7f2960ff9640 1 --2- 172.21.15.1:0/2868089195 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f295006b4d0 0x7f295006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:07.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.983+0000 7f2960ff9640 1 -- 172.21.15.1:0/2868089195 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f295c08abd0 con 0x7f2964156f90 2024-03-06T07:21:07.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.984+0000 7f2963fff640 1 --2- 172.21.15.1:0/2868089195 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f295006b4d0 0x7f295006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:07.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.984+0000 7f2963fff640 1 --2- 172.21.15.1:0/2868089195 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f295006b4d0 0x7f295006d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2954034000 tx=0x7f2954039040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:07.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:07.988+0000 7f2960ff9640 1 -- 172.21.15.1:0/2868089195 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f295c025020 con 0x7f2964156f90 2024-03-06T07:21:08.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.224+0000 7f296a89a640 1 -- 172.21.15.1:0/2868089195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29640c48b0 con 0x7f2964156f90 2024-03-06T07:21:08.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.224+0000 7f2960ff9640 1 -- 172.21.15.1:0/2868089195 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4964 (secure 0 0 0) 0x7f295c059700 con 0x7f2964156f90 2024-03-06T07:21:08.227 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:08.227 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2822958969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2822958969},{"type":"v1","addr":"172.21.15.98:6836","nonce":2822958969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2442906233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2442906233},{"type":"v1","addr":"172.21.15.98:6837","nonce":2442906233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/448746024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":448746024},{"type":"v1","addr":"172.21.15.1:6834","nonce":448746024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:21.207834+0000","modified":"2024-03-06T07:20:37.259661+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1315115982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1315115982},{"type":"v1","addr":"172.21.15.1:6835","nonce":1315115982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-03-06T07:21:08.227 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-03-06T07:21:08.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.226+0000 7f296a89a640 1 -- 172.21.15.1:0/2868089195 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f295006b4d0 msgr2=0x7f295006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:08.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.226+0000 7f296a89a640 1 --2- 172.21.15.1:0/2868089195 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f295006b4d0 0x7f295006d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2954034000 tx=0x7f2954039040 comp rx=0 tx=0).stop 2024-03-06T07:21:08.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.226+0000 7f296a89a640 1 -- 172.21.15.1:0/2868089195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2964156f90 msgr2=0x7f296413cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:08.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.226+0000 7f296a89a640 1 --2- 172.21.15.1:0/2868089195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2964156f90 0x7f296413cf70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f296408c760 tx=0x7f295c010590 comp rx=0 tx=0).stop 2024-03-06T07:21:08.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.226+0000 7f296a89a640 1 -- 172.21.15.1:0/2868089195 shutdown_connections 2024-03-06T07:21:08.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.226+0000 7f296a89a640 1 --2- 172.21.15.1:0/2868089195 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f295006b4d0 0x7f295006d990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.226+0000 7f296a89a640 1 --2- 172.21.15.1:0/2868089195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2964156f90 0x7f296413cf70 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.226+0000 7f296a89a640 1 --2- 172.21.15.1:0/2868089195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29641578c0 0x7f2964156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.226+0000 7f296a89a640 1 --2- 172.21.15.1:0/2868089195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f294c0039b0 0x7f29641ebb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.226+0000 7f296a89a640 1 -- 172.21.15.1:0/2868089195 >> 172.21.15.1:0/2868089195 conn(0x7f2964066ee0 msgr2=0x7f29640cac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:08.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.226+0000 7f296a89a640 1 -- 172.21.15.1:0/2868089195 shutdown_connections 2024-03-06T07:21:08.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.226+0000 7f296a89a640 1 -- 172.21.15.1:0/2868089195 wait complete. 2024-03-06T07:21:08.240 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:08.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.340+0000 7f4bc977a640 1 Processor -- start 2024-03-06T07:21:08.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.340+0000 7f4bc977a640 1 -- start start 2024-03-06T07:21:08.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.341+0000 7f4bc977a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bc4150520 0x7f4bc41543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:08.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.341+0000 7f4bc977a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4bc40c32f0 0x7f4bc41537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:08.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.341+0000 7f4bc977a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc4150900 con 0x7f4bc4070c80 2024-03-06T07:21:08.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.341+0000 7f4bc977a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc4071790 con 0x7f4bc40c32f0 2024-03-06T07:21:08.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.341+0000 7f4bc977a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc4157050 con 0x7f4bc4150520 2024-03-06T07:21:08.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.341+0000 7f4bba7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4bc40c32f0 0x7f4bc41537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:08.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.341+0000 7f4bc2ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bc4150520 0x7f4bc41543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:08.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.341+0000 7f4bc37fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4bc4070c80 0x7f4bc40c67f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:47714/0 (socket says 172.21.15.1:47714) 2024-03-06T07:21:08.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.341+0000 7f4bc37fe640 1 -- 172.21.15.1:0/54462815 learned_addr learned my addr 172.21.15.1:0/54462815 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:08.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.342+0000 7f4bba7fc640 1 -- 172.21.15.1:0/54462815 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bc4150520 msgr2=0x7f4bc41543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:08.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.342+0000 7f4bba7fc640 1 --2- 172.21.15.1:0/54462815 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bc4150520 0x7f4bc41543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.342+0000 7f4bba7fc640 1 -- 172.21.15.1:0/54462815 >> v1:172.21.15.98:6789/0 conn(0x7f4bc4070c80 legacy=0x7f4bc40c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:08.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.342+0000 7f4bba7fc640 1 -- 172.21.15.1:0/54462815 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc4152fc0 con 0x7f4bc40c32f0 2024-03-06T07:21:08.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.343+0000 7f4bba7fc640 1 --2- 172.21.15.1:0/54462815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bc40c32f0 0x7f4bc41537c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0097a0 tx=0x7f4bac02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86ee5eb87b6220c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:08.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.343+0000 7f4bc27fc640 1 -- 172.21.15.1:0/54462815 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac035020 con 0x7f4bc40c32f0 2024-03-06T07:21:08.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.343+0000 7f4bc27fc640 1 -- 172.21.15.1:0/54462815 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bac031830 con 0x7f4bc40c32f0 2024-03-06T07:21:08.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.343+0000 7f4bc27fc640 1 -- 172.21.15.1:0/54462815 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac031c00 con 0x7f4bc40c32f0 2024-03-06T07:21:08.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.343+0000 7f4bc977a640 1 -- 172.21.15.1:0/54462815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bc40c32f0 msgr2=0x7f4bc41537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:08.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.343+0000 7f4bc977a640 1 --2- 172.21.15.1:0/54462815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bc40c32f0 0x7f4bc41537c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0097a0 tx=0x7f4bac02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.343+0000 7f4bc977a640 1 -- 172.21.15.1:0/54462815 shutdown_connections 2024-03-06T07:21:08.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.343+0000 7f4bc977a640 1 --2- 172.21.15.1:0/54462815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bc40c32f0 0x7f4bc41537c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.343+0000 7f4bc977a640 1 --2- 172.21.15.1:0/54462815 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bc4150520 0x7f4bc41543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.343+0000 7f4bc977a640 1 -- 172.21.15.1:0/54462815 >> 172.21.15.1:0/54462815 conn(0x7f4bc40c88f0 msgr2=0x7f4bc40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:08.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.343+0000 7f4bc977a640 1 -- 172.21.15.1:0/54462815 shutdown_connections 2024-03-06T07:21:08.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.343+0000 7f4bc977a640 1 -- 172.21.15.1:0/54462815 wait complete. 2024-03-06T07:21:08.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.344+0000 7f4bc977a640 1 Processor -- start 2024-03-06T07:21:08.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.345+0000 7f4bc977a640 1 -- start start 2024-03-06T07:21:08.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.345+0000 7f4bc977a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bc4070c80 0x7f4bc41ebee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:08.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.345+0000 7f4bc977a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bc40c32f0 0x7f4bc4155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:08.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.345+0000 7f4bc977a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc4150520 0x7f4bc4145c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:08.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.345+0000 7f4bc977a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc40c7ec0 con 0x7f4bc4150520 2024-03-06T07:21:08.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.345+0000 7f4bc977a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc4157330 con 0x7f4bc4070c80 2024-03-06T07:21:08.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.345+0000 7f4bc977a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc4153d00 con 0x7f4bc40c32f0 2024-03-06T07:21:08.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.345+0000 7f4bc37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc4150520 0x7f4bc4145c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:08.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.345+0000 7f4bc37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc4150520 0x7f4bc4145c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52372/0 (socket says 172.21.15.1:52372) 2024-03-06T07:21:08.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.345+0000 7f4bc37fe640 1 -- 172.21.15.1:0/161653097 learned_addr learned my addr 172.21.15.1:0/161653097 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:08.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.345+0000 7f4bba7fc640 1 --2- 172.21.15.1:0/161653097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bc40c32f0 0x7f4bc4155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:08.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.345+0000 7f4bc2ffd640 1 --2- 172.21.15.1:0/161653097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bc4070c80 0x7f4bc41ebee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:08.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.346+0000 7f4bc37fe640 1 -- 172.21.15.1:0/161653097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bc40c32f0 msgr2=0x7f4bc4155140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:08.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.346+0000 7f4bc37fe640 1 --2- 172.21.15.1:0/161653097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bc40c32f0 0x7f4bc4155140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.346+0000 7f4bc37fe640 1 -- 172.21.15.1:0/161653097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bc4070c80 msgr2=0x7f4bc41ebee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:08.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.346+0000 7f4bc37fe640 1 --2- 172.21.15.1:0/161653097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bc4070c80 0x7f4bc41ebee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.346+0000 7f4bc37fe640 1 -- 172.21.15.1:0/161653097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc40ad730 con 0x7f4bc4150520 2024-03-06T07:21:08.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.346+0000 7f4bc37fe640 1 --2- 172.21.15.1:0/161653097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc4150520 0x7f4bc4145c00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00cbf0 tx=0x7f4bbc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:08.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.346+0000 7f4bbbfff640 1 -- 172.21.15.1:0/161653097 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc017070 con 0x7f4bc4150520 2024-03-06T07:21:08.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.346+0000 7f4bbbfff640 1 -- 172.21.15.1:0/161653097 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bbc00ad40 con 0x7f4bc4150520 2024-03-06T07:21:08.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.346+0000 7f4bc977a640 1 -- 172.21.15.1:0/161653097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc41ec7b0 con 0x7f4bc4150520 2024-03-06T07:21:08.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.346+0000 7f4bbbfff640 1 -- 172.21.15.1:0/161653097 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc012600 con 0x7f4bc4150520 2024-03-06T07:21:08.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.346+0000 7f4bc977a640 1 -- 172.21.15.1:0/161653097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc4155f60 con 0x7f4bc4150520 2024-03-06T07:21:08.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.347+0000 7f4bc977a640 1 -- 172.21.15.1:0/161653097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc414a5e0 con 0x7f4bc4150520 2024-03-06T07:21:08.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.348+0000 7f4bbbfff640 1 -- 172.21.15.1:0/161653097 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4bbc01a070 con 0x7f4bc4150520 2024-03-06T07:21:08.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.348+0000 7f4bbbfff640 1 --2- 172.21.15.1:0/161653097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4ba406b4d0 0x7f4ba406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:08.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.348+0000 7f4bbbfff640 1 -- 172.21.15.1:0/161653097 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4bbc08a1b0 con 0x7f4bc4150520 2024-03-06T07:21:08.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.349+0000 7f4bc2ffd640 1 --2- 172.21.15.1:0/161653097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4ba406b4d0 0x7f4ba406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:08.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.349+0000 7f4bc2ffd640 1 --2- 172.21.15.1:0/161653097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4ba406b4d0 0x7f4ba406d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4009aa0 tx=0x7f4bb4009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:08.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.357+0000 7f4bbbfff640 1 -- 172.21.15.1:0/161653097 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4bc414a5e0 con 0x7f4bc4150520 2024-03-06T07:21:08.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.551+0000 7f4bc977a640 1 -- 172.21.15.1:0/161653097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bc40686e0 con 0x7f4bc4150520 2024-03-06T07:21:08.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.552+0000 7f4bbbfff640 1 -- 172.21.15.1:0/161653097 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18394 (secure 0 0 0) 0x7f4bbc058690 con 0x7f4bc4150520 2024-03-06T07:21:08.554 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:08.554 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:20:27.875975+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.554+0000 7f4bc977a640 1 -- 172.21.15.1:0/161653097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4ba406b4d0 msgr2=0x7f4ba406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.554+0000 7f4bc977a640 1 --2- 172.21.15.1:0/161653097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4ba406b4d0 0x7f4ba406d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4009aa0 tx=0x7f4bb4009190 comp rx=0 tx=0).stop 2024-03-06T07:21:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.554+0000 7f4bc977a640 1 -- 172.21.15.1:0/161653097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc4150520 msgr2=0x7f4bc4145c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.554+0000 7f4bc977a640 1 --2- 172.21.15.1:0/161653097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc4150520 0x7f4bc4145c00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00cbf0 tx=0x7f4bbc00a430 comp rx=0 tx=0).stop 2024-03-06T07:21:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.554+0000 7f4bc977a640 1 -- 172.21.15.1:0/161653097 shutdown_connections 2024-03-06T07:21:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.554+0000 7f4bc977a640 1 --2- 172.21.15.1:0/161653097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4ba406b4d0 0x7f4ba406d990 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.554+0000 7f4bc977a640 1 --2- 172.21.15.1:0/161653097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc4150520 0x7f4bc4145c00 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.554+0000 7f4bc977a640 1 --2- 172.21.15.1:0/161653097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bc40c32f0 0x7f4bc4155140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.554+0000 7f4bc977a640 1 --2- 172.21.15.1:0/161653097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bc4070c80 0x7f4bc41ebee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.554+0000 7f4bc977a640 1 -- 172.21.15.1:0/161653097 >> 172.21.15.1:0/161653097 conn(0x7f4bc40c88f0 msgr2=0x7f4bc414c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:08.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.554+0000 7f4bc977a640 1 -- 172.21.15.1:0/161653097 shutdown_connections 2024-03-06T07:21:08.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.555+0000 7f4bc977a640 1 -- 172.21.15.1:0/161653097 wait complete. 2024-03-06T07:21:08.568 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:20:19.337560+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:20:20.172791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:20:23.884071+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:21:08.568 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:08.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.711+0000 7fb5ae2a4640 1 Processor -- start 2024-03-06T07:21:08.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.711+0000 7fb5ae2a4640 1 -- start start 2024-03-06T07:21:08.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.712+0000 7fb5ae2a4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb5a8152650 0x7fb5a8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:08.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.712+0000 7fb5ae2a4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb5a80c6b10 0x7fb5a8158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:08.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.712+0000 7fb5ae2a4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a80c2fc0 con 0x7fb5a80bf9d0 2024-03-06T07:21:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.712+0000 7fb5ae2a4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a8155800 con 0x7fb5a8152650 2024-03-06T07:21:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.712+0000 7fb5ae2a4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a80c6690 con 0x7fb5a80c6b10 2024-03-06T07:21:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.712+0000 7fb5a7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb5a8152650 0x7fb5a8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.712+0000 7fb5a7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb5a8152650 0x7fb5a8155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52380/0 (socket says 172.21.15.1:52380) 2024-03-06T07:21:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.712+0000 7fb5a7fff640 1 -- 172.21.15.1:0/3875797891 learned_addr learned my addr 172.21.15.1:0/3875797891 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.712+0000 7fb5a77fe640 1 --2- 172.21.15.1:0/3875797891 >> v2:172.21.15.98:3300/0 conn(0x7fb5a80c6b10 0x7fb5a8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.712+0000 7fb5a7fff640 1 -- 172.21.15.1:0/3875797891 >> v2:172.21.15.98:3300/0 conn(0x7fb5a80c6b10 msgr2=0x7fb5a8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.712+0000 7fb5a7fff640 1 --2- 172.21.15.1:0/3875797891 >> v2:172.21.15.98:3300/0 conn(0x7fb5a80c6b10 0x7fb5a8158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.712+0000 7fb5a7fff640 1 -- 172.21.15.1:0/3875797891 >> v1:172.21.15.98:6789/0 conn(0x7fb5a80bf9d0 legacy=0x7fb5a80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.712+0000 7fb5a7fff640 1 -- 172.21.15.1:0/3875797891 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a80c7280 con 0x7fb5a8152650 2024-03-06T07:21:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.713+0000 7fb5a7fff640 1 --2- 172.21.15.1:0/3875797891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5a8152650 0x7fb5a8155c90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb5a80cb8e0 tx=0x7fb59802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beefeb5b76ccabfe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.713+0000 7fb5a6ffd640 1 -- 172.21.15.1:0/3875797891 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59802e070 con 0x7fb5a8152650 2024-03-06T07:21:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.713+0000 7fb5a6ffd640 1 -- 172.21.15.1:0/3875797891 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb598037070 con 0x7fb5a8152650 2024-03-06T07:21:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.713+0000 7fb5a6ffd640 1 -- 172.21.15.1:0/3875797891 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb598031a00 con 0x7fb5a8152650 2024-03-06T07:21:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.714+0000 7fb5ae2a4640 1 -- 172.21.15.1:0/3875797891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5a8152650 msgr2=0x7fb5a8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.714+0000 7fb5ae2a4640 1 --2- 172.21.15.1:0/3875797891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5a8152650 0x7fb5a8155c90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb5a80cb8e0 tx=0x7fb59802b880 comp rx=0 tx=0).stop 2024-03-06T07:21:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.714+0000 7fb5ae2a4640 1 -- 172.21.15.1:0/3875797891 shutdown_connections 2024-03-06T07:21:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.714+0000 7fb5ae2a4640 1 --2- 172.21.15.1:0/3875797891 >> v2:172.21.15.98:3300/0 conn(0x7fb5a80c6b10 0x7fb5a8158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.714+0000 7fb5ae2a4640 1 --2- 172.21.15.1:0/3875797891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5a8152650 0x7fb5a8155c90 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.714+0000 7fb5ae2a4640 1 -- 172.21.15.1:0/3875797891 >> 172.21.15.1:0/3875797891 conn(0x7fb5a8066ee0 msgr2=0x7fb5a80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.714+0000 7fb5ae2a4640 1 -- 172.21.15.1:0/3875797891 shutdown_connections 2024-03-06T07:21:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.714+0000 7fb5ae2a4640 1 -- 172.21.15.1:0/3875797891 wait complete. 2024-03-06T07:21:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.715+0000 7fb5ae2a4640 1 Processor -- start 2024-03-06T07:21:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.715+0000 7fb5ae2a4640 1 -- start start 2024-03-06T07:21:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5ae2a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5a80bf9d0 0x7fb5a81533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5ae2a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5a8152650 0x7fb5a8154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5ae2a4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5a814c540 0x7fb5a8150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5ae2a4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a8155490 con 0x7fb5a814c540 2024-03-06T07:21:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5ae2a4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a81551b0 con 0x7fb5a8152650 2024-03-06T07:21:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5ae2a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a8152a30 con 0x7fb5a80bf9d0 2024-03-06T07:21:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5ac841640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5a814c540 0x7fb5a8150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5ac841640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5a814c540 0x7fb5a8150a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52390/0 (socket says 172.21.15.1:52390) 2024-03-06T07:21:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5ac841640 1 -- 172.21.15.1:0/1938421863 learned_addr learned my addr 172.21.15.1:0/1938421863 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5a77fe640 1 --2- 172.21.15.1:0/1938421863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5a8152650 0x7fb5a8154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5a7fff640 1 --2- 172.21.15.1:0/1938421863 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5a80bf9d0 0x7fb5a81533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5ac841640 1 -- 172.21.15.1:0/1938421863 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5a80bf9d0 msgr2=0x7fb5a81533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5ac841640 1 --2- 172.21.15.1:0/1938421863 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5a80bf9d0 0x7fb5a81533a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5ac841640 1 -- 172.21.15.1:0/1938421863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5a8152650 msgr2=0x7fb5a8154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5ac841640 1 --2- 172.21.15.1:0/1938421863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5a8152650 0x7fb5a8154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5ac841640 1 -- 172.21.15.1:0/1938421863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a8151ca0 con 0x7fb5a814c540 2024-03-06T07:21:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5ac841640 1 --2- 172.21.15.1:0/1938421863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5a814c540 0x7fb5a8150a40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb5a000b970 tx=0x7fb5a000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5a4ff9640 1 -- 172.21.15.1:0/1938421863 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0019070 con 0x7fb5a814c540 2024-03-06T07:21:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5a4ff9640 1 -- 172.21.15.1:0/1938421863 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a000cb40 con 0x7fb5a814c540 2024-03-06T07:21:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5ae2a4640 1 -- 172.21.15.1:0/1938421863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a81ee6d0 con 0x7fb5a814c540 2024-03-06T07:21:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.716+0000 7fb5a4ff9640 1 -- 172.21.15.1:0/1938421863 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0014440 con 0x7fb5a814c540 2024-03-06T07:21:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.717+0000 7fb5ae2a4640 1 -- 172.21.15.1:0/1938421863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a8154170 con 0x7fb5a814c540 2024-03-06T07:21:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.717+0000 7fb5ae2a4640 1 -- 172.21.15.1:0/1938421863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a80c1bb0 con 0x7fb5a814c540 2024-03-06T07:21:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.718+0000 7fb5a4ff9640 1 -- 172.21.15.1:0/1938421863 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb5a001c070 con 0x7fb5a814c540 2024-03-06T07:21:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.719+0000 7fb5a4ff9640 1 --2- 172.21.15.1:0/1938421863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb59406b4d0 0x7fb59406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.719+0000 7fb5a4ff9640 1 -- 172.21.15.1:0/1938421863 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb5a008a100 con 0x7fb5a814c540 2024-03-06T07:21:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.719+0000 7fb5a7fff640 1 --2- 172.21.15.1:0/1938421863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb59406b4d0 0x7fb59406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:08.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.720+0000 7fb5a7fff640 1 --2- 172.21.15.1:0/1938421863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb59406b4d0 0x7fb59406d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb598036000 tx=0x7fb59803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.724+0000 7fb5a4ff9640 1 -- 172.21.15.1:0/1938421863 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb5a80c1bb0 con 0x7fb5a814c540 2024-03-06T07:21:08.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.907+0000 7fb5ae2a4640 1 -- 172.21.15.1:0/1938421863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb5a8067fd0 con 0x7fb5a814c540 2024-03-06T07:21:08.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.907+0000 7fb5a4ff9640 1 -- 172.21.15.1:0/1938421863 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb5a0058680 con 0x7fb5a814c540 2024-03-06T07:21:08.909 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:21:08.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.909+0000 7fb5ae2a4640 1 -- 172.21.15.1:0/1938421863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb59406b4d0 msgr2=0x7fb59406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:08.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.909+0000 7fb5ae2a4640 1 --2- 172.21.15.1:0/1938421863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb59406b4d0 0x7fb59406d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb598036000 tx=0x7fb59803d040 comp rx=0 tx=0).stop 2024-03-06T07:21:08.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.909+0000 7fb5ae2a4640 1 -- 172.21.15.1:0/1938421863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5a814c540 msgr2=0x7fb5a8150a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:08.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.909+0000 7fb5ae2a4640 1 --2- 172.21.15.1:0/1938421863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5a814c540 0x7fb5a8150a40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb5a000b970 tx=0x7fb5a000be30 comp rx=0 tx=0).stop 2024-03-06T07:21:08.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.909+0000 7fb5ae2a4640 1 -- 172.21.15.1:0/1938421863 shutdown_connections 2024-03-06T07:21:08.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.909+0000 7fb5ae2a4640 1 --2- 172.21.15.1:0/1938421863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb59406b4d0 0x7fb59406d990 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.909+0000 7fb5ae2a4640 1 --2- 172.21.15.1:0/1938421863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5a814c540 0x7fb5a8150a40 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.909+0000 7fb5ae2a4640 1 --2- 172.21.15.1:0/1938421863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5a8152650 0x7fb5a8154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.909+0000 7fb5ae2a4640 1 --2- 172.21.15.1:0/1938421863 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5a80bf9d0 0x7fb5a81533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:08.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.909+0000 7fb5ae2a4640 1 -- 172.21.15.1:0/1938421863 >> 172.21.15.1:0/1938421863 conn(0x7fb5a8066ee0 msgr2=0x7fb5a814e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:08.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.909+0000 7fb5ae2a4640 1 -- 172.21.15.1:0/1938421863 shutdown_connections 2024-03-06T07:21:08.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:08.909+0000 7fb5ae2a4640 1 -- 172.21.15.1:0/1938421863 wait complete. 2024-03-06T07:21:08.922 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:21:09.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.025+0000 7ff52a8aa640 1 Processor -- start 2024-03-06T07:21:09.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.025+0000 7ff52a8aa640 1 -- start start 2024-03-06T07:21:09.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.026+0000 7ff52a8aa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff5240c32f0 0x7ff5241543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.026+0000 7ff52a8aa640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff524070c80 0x7ff5240710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.026+0000 7ff52a8aa640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff524150900 con 0x7ff524150520 2024-03-06T07:21:09.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.026+0000 7ff52a8aa640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff524153ed0 con 0x7ff5240c32f0 2024-03-06T07:21:09.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.026+0000 7ff52a8aa640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff524071790 con 0x7ff524070c80 2024-03-06T07:21:09.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.026+0000 7ff5237fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff5240c32f0 0x7ff5241543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.026+0000 7ff523fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff524150520 0x7ff5241537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59256/0 (socket says 172.21.15.1:59256) 2024-03-06T07:21:09.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.026+0000 7ff523fff640 1 -- 172.21.15.1:0/4066821618 learned_addr learned my addr 172.21.15.1:0/4066821618 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:09.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.027+0000 7ff528e47640 1 --2- 172.21.15.1:0/4066821618 >> v2:172.21.15.98:3300/0 conn(0x7ff524070c80 0x7ff5240710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.027+0000 7ff5237fe640 1 -- 172.21.15.1:0/4066821618 >> v2:172.21.15.98:3300/0 conn(0x7ff524070c80 msgr2=0x7ff5240710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:09.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.027+0000 7ff5237fe640 1 --2- 172.21.15.1:0/4066821618 >> v2:172.21.15.98:3300/0 conn(0x7ff524070c80 0x7ff5240710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.027+0000 7ff5237fe640 1 -- 172.21.15.1:0/4066821618 >> v1:172.21.15.1:6789/0 conn(0x7ff524150520 legacy=0x7ff5241537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.027+0000 7ff5237fe640 1 -- 172.21.15.1:0/4066821618 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff524157050 con 0x7ff5240c32f0 2024-03-06T07:21:09.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.027+0000 7ff522ffd640 1 -- 172.21.15.1:0/4066821618 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057999417 0 0) 0x7ff524150900 con 0x7ff524150520 2024-03-06T07:21:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.028+0000 7ff5237fe640 1 --2- 172.21.15.1:0/4066821618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5240c32f0 0x7ff5241543a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff5240c3190 tx=0x7ff51402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=381dcd2b789b4def server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.028+0000 7ff522ffd640 1 -- 172.21.15.1:0/4066821618 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff514035020 con 0x7ff5240c32f0 2024-03-06T07:21:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.028+0000 7ff522ffd640 1 -- 172.21.15.1:0/4066821618 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff514031830 con 0x7ff5240c32f0 2024-03-06T07:21:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.028+0000 7ff522ffd640 1 -- 172.21.15.1:0/4066821618 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff514031c00 con 0x7ff5240c32f0 2024-03-06T07:21:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.028+0000 7ff52a8aa640 1 -- 172.21.15.1:0/4066821618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5240c32f0 msgr2=0x7ff5241543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.028+0000 7ff52a8aa640 1 --2- 172.21.15.1:0/4066821618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5240c32f0 0x7ff5241543a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff5240c3190 tx=0x7ff51402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.028+0000 7ff52a8aa640 1 -- 172.21.15.1:0/4066821618 shutdown_connections 2024-03-06T07:21:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.028+0000 7ff52a8aa640 1 --2- 172.21.15.1:0/4066821618 >> v2:172.21.15.98:3300/0 conn(0x7ff524070c80 0x7ff5240710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.028+0000 7ff52a8aa640 1 --2- 172.21.15.1:0/4066821618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5240c32f0 0x7ff5241543a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.028+0000 7ff52a8aa640 1 -- 172.21.15.1:0/4066821618 >> 172.21.15.1:0/4066821618 conn(0x7ff5240c88f0 msgr2=0x7ff5240c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.028+0000 7ff52a8aa640 1 -- 172.21.15.1:0/4066821618 shutdown_connections 2024-03-06T07:21:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.028+0000 7ff52a8aa640 1 -- 172.21.15.1:0/4066821618 wait complete. 2024-03-06T07:21:09.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.029+0000 7ff52a8aa640 1 Processor -- start 2024-03-06T07:21:09.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.030+0000 7ff52a8aa640 1 -- start start 2024-03-06T07:21:09.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.030+0000 7ff52a8aa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff524070c80 0x7ff524146030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.030+0000 7ff52a8aa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5240c32f0 0x7ff5240ac920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.030+0000 7ff52a8aa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff524150520 0x7ff5241ea970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.030+0000 7ff52a8aa640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff524157330 con 0x7ff524070c80 2024-03-06T07:21:09.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.030+0000 7ff52a8aa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5241532a0 con 0x7ff5240c32f0 2024-03-06T07:21:09.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.030+0000 7ff52a8aa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240c7ec0 con 0x7ff524150520 2024-03-06T07:21:09.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.030+0000 7ff523fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff524070c80 0x7ff524146030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.030+0000 7ff523fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff524070c80 0x7ff524146030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52412/0 (socket says 172.21.15.1:52412) 2024-03-06T07:21:09.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.030+0000 7ff523fff640 1 -- 172.21.15.1:0/1480430327 learned_addr learned my addr 172.21.15.1:0/1480430327 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:09.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.031+0000 7ff5237fe640 1 --2- 172.21.15.1:0/1480430327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5240c32f0 0x7ff5240ac920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.031+0000 7ff528e47640 1 --2- 172.21.15.1:0/1480430327 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff524150520 0x7ff5241ea970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.031+0000 7ff523fff640 1 -- 172.21.15.1:0/1480430327 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff524150520 msgr2=0x7ff5241ea970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.031+0000 7ff523fff640 1 --2- 172.21.15.1:0/1480430327 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff524150520 0x7ff5241ea970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.031+0000 7ff523fff640 1 -- 172.21.15.1:0/1480430327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5240c32f0 msgr2=0x7ff5240ac920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.031+0000 7ff523fff640 1 --2- 172.21.15.1:0/1480430327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5240c32f0 0x7ff5240ac920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.031+0000 7ff523fff640 1 -- 172.21.15.1:0/1480430327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5241eede0 con 0x7ff524070c80 2024-03-06T07:21:09.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.031+0000 7ff5237fe640 1 --2- 172.21.15.1:0/1480430327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5240c32f0 0x7ff5240ac920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:09.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.031+0000 7ff523fff640 1 --2- 172.21.15.1:0/1480430327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff524070c80 0x7ff524146030 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff51800c430 tx=0x7ff51800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:09.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.031+0000 7ff520ff9640 1 -- 172.21.15.1:0/1480430327 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5180074e0 con 0x7ff524070c80 2024-03-06T07:21:09.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.031+0000 7ff520ff9640 1 -- 172.21.15.1:0/1480430327 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5180047e0 con 0x7ff524070c80 2024-03-06T07:21:09.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.031+0000 7ff52a8aa640 1 -- 172.21.15.1:0/1480430327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff524156ad0 con 0x7ff524070c80 2024-03-06T07:21:09.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.031+0000 7ff520ff9640 1 -- 172.21.15.1:0/1480430327 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff518004bd0 con 0x7ff524070c80 2024-03-06T07:21:09.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.031+0000 7ff52a8aa640 1 -- 172.21.15.1:0/1480430327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff524147470 con 0x7ff524070c80 2024-03-06T07:21:09.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.032+0000 7ff52a8aa640 1 -- 172.21.15.1:0/1480430327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5240686e0 con 0x7ff524070c80 2024-03-06T07:21:09.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.033+0000 7ff520ff9640 1 -- 172.21.15.1:0/1480430327 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff518011070 con 0x7ff524070c80 2024-03-06T07:21:09.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.033+0000 7ff520ff9640 1 --2- 172.21.15.1:0/1480430327 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff51006b4d0 0x7ff51006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.033+0000 7ff520ff9640 1 -- 172.21.15.1:0/1480430327 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff51808a120 con 0x7ff524070c80 2024-03-06T07:21:09.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.033+0000 7ff5237fe640 1 --2- 172.21.15.1:0/1480430327 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff51006b4d0 0x7ff51006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.034+0000 7ff5237fe640 1 --2- 172.21.15.1:0/1480430327 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff51006b4d0 0x7ff51006d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff514036000 tx=0x7ff514008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:09.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.038+0000 7ff520ff9640 1 -- 172.21.15.1:0/1480430327 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff518058cf0 con 0x7ff524070c80 2024-03-06T07:21:09.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.225+0000 7ff52a8aa640 1 -- 172.21.15.1:0/1480430327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff5240c7b10 con 0x7ff524070c80 2024-03-06T07:21:09.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.226+0000 7ff520ff9640 1 -- 172.21.15.1:0/1480430327 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff5180585f0 con 0x7ff524070c80 2024-03-06T07:21:09.228 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:21:09.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.228+0000 7ff52a8aa640 1 -- 172.21.15.1:0/1480430327 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff51006b4d0 msgr2=0x7ff51006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.228+0000 7ff52a8aa640 1 --2- 172.21.15.1:0/1480430327 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff51006b4d0 0x7ff51006d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff514036000 tx=0x7ff514008030 comp rx=0 tx=0).stop 2024-03-06T07:21:09.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.228+0000 7ff52a8aa640 1 -- 172.21.15.1:0/1480430327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff524070c80 msgr2=0x7ff524146030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.228+0000 7ff52a8aa640 1 --2- 172.21.15.1:0/1480430327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff524070c80 0x7ff524146030 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff51800c430 tx=0x7ff51800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.228+0000 7ff52a8aa640 1 -- 172.21.15.1:0/1480430327 shutdown_connections 2024-03-06T07:21:09.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.228+0000 7ff52a8aa640 1 --2- 172.21.15.1:0/1480430327 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff51006b4d0 0x7ff51006d990 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.228+0000 7ff52a8aa640 1 --2- 172.21.15.1:0/1480430327 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff524150520 0x7ff5241ea970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.228+0000 7ff52a8aa640 1 --2- 172.21.15.1:0/1480430327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5240c32f0 0x7ff5240ac920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.228+0000 7ff52a8aa640 1 --2- 172.21.15.1:0/1480430327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff524070c80 0x7ff524146030 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.228+0000 7ff52a8aa640 1 -- 172.21.15.1:0/1480430327 >> 172.21.15.1:0/1480430327 conn(0x7ff5240c88f0 msgr2=0x7ff524152750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:09.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.228+0000 7ff52a8aa640 1 -- 172.21.15.1:0/1480430327 shutdown_connections 2024-03-06T07:21:09.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.228+0000 7ff52a8aa640 1 -- 172.21.15.1:0/1480430327 wait complete. 2024-03-06T07:21:09.241 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:21:09.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.343+0000 7f1ad06fb640 1 Processor -- start 2024-03-06T07:21:09.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.343+0000 7f1ad06fb640 1 -- start start 2024-03-06T07:21:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.344+0000 7f1ad06fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ac8152840 0x7f1ac80c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.344+0000 7f1ad06fb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac80c5e40 con 0x7f1ac80c60b0 2024-03-06T07:21:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.344+0000 7f1ad06fb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac80c6a40 con 0x7f1ac80c3970 2024-03-06T07:21:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.344+0000 7f1ad06fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac80c67f0 con 0x7f1ac8152840 2024-03-06T07:21:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.345+0000 7f1acec98640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1ac80c60b0 0x7f1ac81553d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59270/0 (socket says 172.21.15.1:59270) 2024-03-06T07:21:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.345+0000 7f1acec98640 1 -- 172.21.15.1:0/164634458 learned_addr learned my addr 172.21.15.1:0/164634458 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.345+0000 7f1acdc96640 1 --2- 172.21.15.1:0/164634458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ac8152840 0x7f1ac80c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.345+0000 7f1acd495640 1 -- 172.21.15.1:0/164634458 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559190904 0 0) 0x7f1ac80c5e40 con 0x7f1ac80c60b0 2024-03-06T07:21:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.345+0000 7f1acd495640 1 -- 172.21.15.1:0/164634458 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ab4003520 con 0x7f1ac80c60b0 2024-03-06T07:21:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.345+0000 7f1acd495640 1 -- 172.21.15.1:0/164634458 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364312933 0 0) 0x7f1ac80c6a40 con 0x7f1ac80c3970 2024-03-06T07:21:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.345+0000 7f1acd495640 1 -- 172.21.15.1:0/164634458 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac80c5e40 con 0x7f1ac80c3970 2024-03-06T07:21:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.345+0000 7f1acdc96640 1 -- 172.21.15.1:0/164634458 >> v1:172.21.15.98:6789/0 conn(0x7f1ac80c3970 legacy=0x7f1ac8152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.345+0000 7f1acdc96640 1 -- 172.21.15.1:0/164634458 >> v1:172.21.15.1:6789/0 conn(0x7f1ac80c60b0 legacy=0x7f1ac81553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.345+0000 7f1acdc96640 1 -- 172.21.15.1:0/164634458 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac80c6d20 con 0x7f1ac8152840 2024-03-06T07:21:09.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.346+0000 7f1acdc96640 1 --2- 172.21.15.1:0/164634458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ac8152840 0x7f1ac80c8170 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1ac40097d0 tx=0x7f1ac402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=274b3ab8dacc0e6f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:09.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.346+0000 7f1acd495640 1 -- 172.21.15.1:0/164634458 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4035020 con 0x7f1ac8152840 2024-03-06T07:21:09.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.346+0000 7f1acd495640 1 -- 172.21.15.1:0/164634458 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac4031890 con 0x7f1ac8152840 2024-03-06T07:21:09.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.346+0000 7f1acd495640 1 -- 172.21.15.1:0/164634458 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4031c40 con 0x7f1ac8152840 2024-03-06T07:21:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.347+0000 7f1ad06fb640 1 -- 172.21.15.1:0/164634458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ac8152840 msgr2=0x7f1ac80c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.347+0000 7f1ad06fb640 1 --2- 172.21.15.1:0/164634458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ac8152840 0x7f1ac80c8170 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1ac40097d0 tx=0x7f1ac402b930 comp rx=0 tx=0).stop 2024-03-06T07:21:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.347+0000 7f1ad06fb640 1 -- 172.21.15.1:0/164634458 shutdown_connections 2024-03-06T07:21:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.347+0000 7f1ad06fb640 1 --2- 172.21.15.1:0/164634458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ac8152840 0x7f1ac80c8170 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.347+0000 7f1ad06fb640 1 -- 172.21.15.1:0/164634458 >> 172.21.15.1:0/164634458 conn(0x7f1ac80c88f0 msgr2=0x7f1ac80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.347+0000 7f1ad06fb640 1 -- 172.21.15.1:0/164634458 shutdown_connections 2024-03-06T07:21:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.347+0000 7f1ad06fb640 1 -- 172.21.15.1:0/164634458 wait complete. 2024-03-06T07:21:09.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.348+0000 7f1ad06fb640 1 Processor -- start 2024-03-06T07:21:09.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.348+0000 7f1ad06fb640 1 -- start start 2024-03-06T07:21:09.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1ad06fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ac80c3970 0x7f1ac814a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1ad06fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ac80c60b0 0x7f1ac814a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1ad06fb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ac8152840 0x7f1ac81503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1ad06fb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac80c75c0 con 0x7f1ac80c3970 2024-03-06T07:21:09.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1ad06fb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac80c72e0 con 0x7f1ac8152840 2024-03-06T07:21:09.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1ad06fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac80c3d50 con 0x7f1ac80c60b0 2024-03-06T07:21:09.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1ace497640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ac80c3970 0x7f1ac814a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1ace497640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ac80c3970 0x7f1ac814a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52426/0 (socket says 172.21.15.1:52426) 2024-03-06T07:21:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1ace497640 1 -- 172.21.15.1:0/4087666524 learned_addr learned my addr 172.21.15.1:0/4087666524 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1acdc96640 1 --2- 172.21.15.1:0/4087666524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ac80c60b0 0x7f1ac814a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1acec98640 1 --2- 172.21.15.1:0/4087666524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ac8152840 0x7f1ac81503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1ace497640 1 -- 172.21.15.1:0/4087666524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ac80c60b0 msgr2=0x7f1ac814a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1ace497640 1 --2- 172.21.15.1:0/4087666524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ac80c60b0 0x7f1ac814a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1ace497640 1 -- 172.21.15.1:0/4087666524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ac8152840 msgr2=0x7f1ac81503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1ace497640 1 --2- 172.21.15.1:0/4087666524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ac8152840 0x7f1ac81503c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1ace497640 1 -- 172.21.15.1:0/4087666524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac81eb620 con 0x7f1ac80c3970 2024-03-06T07:21:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1acec98640 1 --2- 172.21.15.1:0/4087666524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ac8152840 0x7f1ac81503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1ace497640 1 --2- 172.21.15.1:0/4087666524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ac80c3970 0x7f1ac814a0b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00c490 tx=0x7f1abc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.349+0000 7f1abaffd640 1 -- 172.21.15.1:0/4087666524 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc0052d0 con 0x7f1ac80c3970 2024-03-06T07:21:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.350+0000 7f1abaffd640 1 -- 172.21.15.1:0/4087666524 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1abc005470 con 0x7f1ac80c3970 2024-03-06T07:21:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.350+0000 7f1ad06fb640 1 -- 172.21.15.1:0/4087666524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac81eb8a0 con 0x7f1ac80c3970 2024-03-06T07:21:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.350+0000 7f1abaffd640 1 -- 172.21.15.1:0/4087666524 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc012850 con 0x7f1ac80c3970 2024-03-06T07:21:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.350+0000 7f1ad06fb640 1 -- 172.21.15.1:0/4087666524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac80716a0 con 0x7f1ac80c3970 2024-03-06T07:21:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.350+0000 7f1ad06fb640 1 -- 172.21.15.1:0/4087666524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac80c64e0 con 0x7f1ac80c3970 2024-03-06T07:21:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.351+0000 7f1abaffd640 1 -- 172.21.15.1:0/4087666524 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1abc0129f0 con 0x7f1ac80c3970 2024-03-06T07:21:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.351+0000 7f1abaffd640 1 --2- 172.21.15.1:0/4087666524 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1aac06b4d0 0x7f1aac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.351+0000 7f1abaffd640 1 -- 172.21.15.1:0/4087666524 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f1abc08a2f0 con 0x7f1ac80c3970 2024-03-06T07:21:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.351+0000 7f1acdc96640 1 --2- 172.21.15.1:0/4087666524 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1aac06b4d0 0x7f1aac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.352+0000 7f1acdc96640 1 --2- 172.21.15.1:0/4087666524 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1aac06b4d0 0x7f1aac06d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4036000 tx=0x7f1ac403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:09.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.360+0000 7f1abaffd640 1 -- 172.21.15.1:0/4087666524 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ac80c64e0 con 0x7f1ac80c3970 2024-03-06T07:21:09.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.541+0000 7f1ad06fb640 1 -- 172.21.15.1:0/4087666524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1ac80686e0 con 0x7f1ac80c3970 2024-03-06T07:21:09.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.541+0000 7f1abaffd640 1 -- 172.21.15.1:0/4087666524 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1ac81eb620 con 0x7f1ac80c3970 2024-03-06T07:21:09.543 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:21:09.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.543+0000 7f1ad06fb640 1 -- 172.21.15.1:0/4087666524 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1aac06b4d0 msgr2=0x7f1aac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.543+0000 7f1ad06fb640 1 --2- 172.21.15.1:0/4087666524 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1aac06b4d0 0x7f1aac06d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4036000 tx=0x7f1ac403c040 comp rx=0 tx=0).stop 2024-03-06T07:21:09.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.543+0000 7f1ad06fb640 1 -- 172.21.15.1:0/4087666524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ac80c3970 msgr2=0x7f1ac814a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.543+0000 7f1ad06fb640 1 --2- 172.21.15.1:0/4087666524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ac80c3970 0x7f1ac814a0b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00c490 tx=0x7f1abc00c950 comp rx=0 tx=0).stop 2024-03-06T07:21:09.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.543+0000 7f1ad06fb640 1 -- 172.21.15.1:0/4087666524 shutdown_connections 2024-03-06T07:21:09.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.543+0000 7f1ad06fb640 1 --2- 172.21.15.1:0/4087666524 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1aac06b4d0 0x7f1aac06d990 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.543+0000 7f1ad06fb640 1 --2- 172.21.15.1:0/4087666524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ac8152840 0x7f1ac81503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.543+0000 7f1ad06fb640 1 --2- 172.21.15.1:0/4087666524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ac80c60b0 0x7f1ac814a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.543+0000 7f1ad06fb640 1 --2- 172.21.15.1:0/4087666524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ac80c3970 0x7f1ac814a0b0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.543+0000 7f1ad06fb640 1 -- 172.21.15.1:0/4087666524 >> 172.21.15.1:0/4087666524 conn(0x7f1ac80c88f0 msgr2=0x7f1ac80c1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:09.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.543+0000 7f1ad06fb640 1 -- 172.21.15.1:0/4087666524 shutdown_connections 2024-03-06T07:21:09.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.543+0000 7f1ad06fb640 1 -- 172.21.15.1:0/4087666524 wait complete. 2024-03-06T07:21:09.556 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:21:09.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.659+0000 7ff2e986b640 1 Processor -- start 2024-03-06T07:21:09.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.659+0000 7ff2e986b640 1 -- start start 2024-03-06T07:21:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.660+0000 7ff2e986b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2e4152a30 0x7ff2e40c7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.660+0000 7ff2e986b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff2e40afcb0 0x7ff2e406d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.660+0000 7ff2e986b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff2e40c7940 0x7ff2e406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.660+0000 7ff2e986b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e4155260 con 0x7ff2e40afcb0 2024-03-06T07:21:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.660+0000 7ff2e986b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e40c6fb0 con 0x7ff2e40c7940 2024-03-06T07:21:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.660+0000 7ff2e986b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e4155520 con 0x7ff2e4152a30 2024-03-06T07:21:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.660+0000 7ff2dbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff2e40afcb0 0x7ff2e406d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.660+0000 7ff2dbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff2e40afcb0 0x7ff2e406d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52428/0 (socket says 172.21.15.1:52428) 2024-03-06T07:21:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.660+0000 7ff2dbfff640 1 -- 172.21.15.1:0/3618120146 learned_addr learned my addr 172.21.15.1:0/3618120146 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.660+0000 7ff2e2ffd640 1 --2- 172.21.15.1:0/3618120146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2e4152a30 0x7ff2e40c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.660+0000 7ff2e37fe640 1 --2- 172.21.15.1:0/3618120146 >> v2:172.21.15.98:3300/0 conn(0x7ff2e40c7940 0x7ff2e406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.660+0000 7ff2dbfff640 1 -- 172.21.15.1:0/3618120146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2e4152a30 msgr2=0x7ff2e40c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:09.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.660+0000 7ff2dbfff640 1 --2- 172.21.15.1:0/3618120146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2e4152a30 0x7ff2e40c7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.660+0000 7ff2dbfff640 1 -- 172.21.15.1:0/3618120146 >> v2:172.21.15.98:3300/0 conn(0x7ff2e40c7940 msgr2=0x7ff2e406d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:09.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.660+0000 7ff2dbfff640 1 --2- 172.21.15.1:0/3618120146 >> v2:172.21.15.98:3300/0 conn(0x7ff2e40c7940 0x7ff2e406d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.660+0000 7ff2dbfff640 1 -- 172.21.15.1:0/3618120146 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e4155800 con 0x7ff2e40afcb0 2024-03-06T07:21:09.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.660+0000 7ff2e37fe640 1 --2- 172.21.15.1:0/3618120146 >> v2:172.21.15.98:3300/0 conn(0x7ff2e40c7940 0x7ff2e406d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:09.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.661+0000 7ff2dbfff640 1 --2- 172.21.15.1:0/3618120146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2e40afcb0 0x7ff2e406d060 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40c5610 tx=0x7ff2cc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df74f3bdf08c249f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:09.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.661+0000 7ff2e27fc640 1 -- 172.21.15.1:0/3618120146 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc02e070 con 0x7ff2e40afcb0 2024-03-06T07:21:09.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.661+0000 7ff2e27fc640 1 -- 172.21.15.1:0/3618120146 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2cc037070 con 0x7ff2e40afcb0 2024-03-06T07:21:09.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.661+0000 7ff2e27fc640 1 -- 172.21.15.1:0/3618120146 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc031a00 con 0x7ff2e40afcb0 2024-03-06T07:21:09.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.662+0000 7ff2e986b640 1 -- 172.21.15.1:0/3618120146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2e40afcb0 msgr2=0x7ff2e406d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.662+0000 7ff2e986b640 1 --2- 172.21.15.1:0/3618120146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2e40afcb0 0x7ff2e406d060 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40c5610 tx=0x7ff2cc02b880 comp rx=0 tx=0).stop 2024-03-06T07:21:09.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.662+0000 7ff2e986b640 1 -- 172.21.15.1:0/3618120146 shutdown_connections 2024-03-06T07:21:09.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.662+0000 7ff2e986b640 1 --2- 172.21.15.1:0/3618120146 >> v2:172.21.15.98:3300/0 conn(0x7ff2e40c7940 0x7ff2e406d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.662+0000 7ff2e986b640 1 --2- 172.21.15.1:0/3618120146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2e40afcb0 0x7ff2e406d060 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.662+0000 7ff2e986b640 1 --2- 172.21.15.1:0/3618120146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2e4152a30 0x7ff2e40c7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.662+0000 7ff2e986b640 1 -- 172.21.15.1:0/3618120146 >> 172.21.15.1:0/3618120146 conn(0x7ff2e408b8e0 msgr2=0x7ff2e4066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:09.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.662+0000 7ff2e986b640 1 -- 172.21.15.1:0/3618120146 shutdown_connections 2024-03-06T07:21:09.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.662+0000 7ff2e986b640 1 -- 172.21.15.1:0/3618120146 wait complete. 2024-03-06T07:21:09.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.663+0000 7ff2e986b640 1 Processor -- start 2024-03-06T07:21:09.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.663+0000 7ff2e986b640 1 -- start start 2024-03-06T07:21:09.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.664+0000 7ff2e986b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e40afcb0 0x7ff2e41eb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.664+0000 7ff2e986b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2e40c7940 0x7ff2e40c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.664+0000 7ff2e986b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2e4152a30 0x7ff2e40c2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.664+0000 7ff2e986b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e40c6cc0 con 0x7ff2e40c7940 2024-03-06T07:21:09.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.664+0000 7ff2e986b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e40c6b40 con 0x7ff2e40afcb0 2024-03-06T07:21:09.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.664+0000 7ff2e986b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e40af6d0 con 0x7ff2e4152a30 2024-03-06T07:21:09.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.664+0000 7ff2dbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2e40c7940 0x7ff2e40c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.664+0000 7ff2dbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2e40c7940 0x7ff2e40c9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52442/0 (socket says 172.21.15.1:52442) 2024-03-06T07:21:09.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.664+0000 7ff2dbfff640 1 -- 172.21.15.1:0/2988985206 learned_addr learned my addr 172.21.15.1:0/2988985206 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:09.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.664+0000 7ff2e37fe640 1 --2- 172.21.15.1:0/2988985206 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2e4152a30 0x7ff2e40c2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.664+0000 7ff2e2ffd640 1 --2- 172.21.15.1:0/2988985206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e40afcb0 0x7ff2e41eb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.664+0000 7ff2dbfff640 1 -- 172.21.15.1:0/2988985206 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2e4152a30 msgr2=0x7ff2e40c2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.664+0000 7ff2dbfff640 1 --2- 172.21.15.1:0/2988985206 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2e4152a30 0x7ff2e40c2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.664+0000 7ff2dbfff640 1 -- 172.21.15.1:0/2988985206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e40afcb0 msgr2=0x7ff2e41eb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.664+0000 7ff2dbfff640 1 --2- 172.21.15.1:0/2988985206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e40afcb0 0x7ff2e41eb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.664+0000 7ff2dbfff640 1 -- 172.21.15.1:0/2988985206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e41eac70 con 0x7ff2e40c7940 2024-03-06T07:21:09.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.664+0000 7ff2dbfff640 1 --2- 172.21.15.1:0/2988985206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2e40c7940 0x7ff2e40c9ab0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc02b8c0 tx=0x7ff2cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:09.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.665+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/2988985206 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc032670 con 0x7ff2e40c7940 2024-03-06T07:21:09.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.665+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/2988985206 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2cc032830 con 0x7ff2e40c7940 2024-03-06T07:21:09.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.665+0000 7ff2e986b640 1 -- 172.21.15.1:0/2988985206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e40c4770 con 0x7ff2e40c7940 2024-03-06T07:21:09.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.665+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/2988985206 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc032c20 con 0x7ff2e40c7940 2024-03-06T07:21:09.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.665+0000 7ff2e986b640 1 -- 172.21.15.1:0/2988985206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e41eaf30 con 0x7ff2e40c7940 2024-03-06T07:21:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.666+0000 7ff2e986b640 1 -- 172.21.15.1:0/2988985206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e4067fd0 con 0x7ff2e40c7940 2024-03-06T07:21:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.666+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/2988985206 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2cc0453f0 con 0x7ff2e40c7940 2024-03-06T07:21:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.667+0000 7ff2e0ff9640 1 --2- 172.21.15.1:0/2988985206 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff2c406b4d0 0x7ff2c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.667+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/2988985206 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7ff2cc0ac340 con 0x7ff2e40c7940 2024-03-06T07:21:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.667+0000 7ff2e2ffd640 1 --2- 172.21.15.1:0/2988985206 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff2c406b4d0 0x7ff2c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.668+0000 7ff2e2ffd640 1 --2- 172.21.15.1:0/2988985206 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff2c406b4d0 0x7ff2c406d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4009a30 tx=0x7ff2d4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:09.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.671+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/2988985206 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2cc040070 con 0x7ff2e40c7940 2024-03-06T07:21:09.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.851+0000 7ff2e986b640 1 -- 172.21.15.1:0/2988985206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff2e40c5b50 con 0x7ff2e40c7940 2024-03-06T07:21:09.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.852+0000 7ff2e0ff9640 1 -- 172.21.15.1:0/2988985206 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff2cc07ae50 con 0x7ff2e40c7940 2024-03-06T07:21:09.853 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:21:09.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.854+0000 7ff2e986b640 1 -- 172.21.15.1:0/2988985206 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff2c406b4d0 msgr2=0x7ff2c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.854+0000 7ff2e986b640 1 --2- 172.21.15.1:0/2988985206 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff2c406b4d0 0x7ff2c406d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4009a30 tx=0x7ff2d4008040 comp rx=0 tx=0).stop 2024-03-06T07:21:09.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.854+0000 7ff2e986b640 1 -- 172.21.15.1:0/2988985206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2e40c7940 msgr2=0x7ff2e40c9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.854+0000 7ff2e986b640 1 --2- 172.21.15.1:0/2988985206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2e40c7940 0x7ff2e40c9ab0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc02b8c0 tx=0x7ff2cc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.854+0000 7ff2e986b640 1 -- 172.21.15.1:0/2988985206 shutdown_connections 2024-03-06T07:21:09.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.854+0000 7ff2e986b640 1 --2- 172.21.15.1:0/2988985206 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff2c406b4d0 0x7ff2c406d990 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.854+0000 7ff2e986b640 1 --2- 172.21.15.1:0/2988985206 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2e4152a30 0x7ff2e40c2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.854+0000 7ff2e986b640 1 --2- 172.21.15.1:0/2988985206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2e40c7940 0x7ff2e40c9ab0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.854+0000 7ff2e986b640 1 --2- 172.21.15.1:0/2988985206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e40afcb0 0x7ff2e41eb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.854+0000 7ff2e986b640 1 -- 172.21.15.1:0/2988985206 >> 172.21.15.1:0/2988985206 conn(0x7ff2e408b8e0 msgr2=0x7ff2e4154770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:09.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.854+0000 7ff2e986b640 1 -- 172.21.15.1:0/2988985206 shutdown_connections 2024-03-06T07:21:09.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.854+0000 7ff2e986b640 1 -- 172.21.15.1:0/2988985206 wait complete. 2024-03-06T07:21:09.868 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:09.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.967+0000 7fe1e6488640 1 Processor -- start 2024-03-06T07:21:09.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.967+0000 7fe1e6488640 1 -- start start 2024-03-06T07:21:09.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.968+0000 7fe1e6488640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1e00bf9d0 0x7fe1e0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.968+0000 7fe1e6488640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e00c2fc0 con 0x7fe1e00c6b10 2024-03-06T07:21:09.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.968+0000 7fe1e6488640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e0158da0 con 0x7fe1e0152650 2024-03-06T07:21:09.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.968+0000 7fe1e6488640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1e0155800 con 0x7fe1e00bf9d0 2024-03-06T07:21:09.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.968+0000 7fe1df7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe1e00c6b10 0x7fe1e00c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59282/0 (socket says 172.21.15.1:59282) 2024-03-06T07:21:09.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.968+0000 7fe1df7fe640 1 -- 172.21.15.1:0/4109055448 learned_addr learned my addr 172.21.15.1:0/4109055448 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:09.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.968+0000 7fe1e4a25640 1 --2- 172.21.15.1:0/4109055448 >> v2:172.21.15.98:3300/0 conn(0x7fe1e00bf9d0 0x7fe1e0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1deffd640 1 -- 172.21.15.1:0/4109055448 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205691778 0 0) 0x7fe1e00c2fc0 con 0x7fe1e00c6b10 2024-03-06T07:21:09.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1deffd640 1 -- 172.21.15.1:0/4109055448 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1cc003520 con 0x7fe1e00c6b10 2024-03-06T07:21:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1deffd640 1 -- 172.21.15.1:0/4109055448 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2732129543 0 0) 0x7fe1cc003520 con 0x7fe1e00c6b10 2024-03-06T07:21:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1deffd640 1 -- 172.21.15.1:0/4109055448 >> v2:172.21.15.98:3300/0 conn(0x7fe1e00bf9d0 msgr2=0x7fe1e0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1deffd640 1 --2- 172.21.15.1:0/4109055448 >> v2:172.21.15.98:3300/0 conn(0x7fe1e00bf9d0 0x7fe1e0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1deffd640 1 -- 172.21.15.1:0/4109055448 >> v1:172.21.15.98:6789/0 conn(0x7fe1e0152650 legacy=0x7fe1e0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1deffd640 1 -- 172.21.15.1:0/4109055448 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e00c6690 con 0x7fe1e00c6b10 2024-03-06T07:21:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1deffd640 1 -- 172.21.15.1:0/4109055448 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fe1c8001670 con 0x7fe1e00c6b10 2024-03-06T07:21:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1deffd640 1 -- 172.21.15.1:0/4109055448 >> v1:172.21.15.1:6789/0 conn(0x7fe1e00c6b10 legacy=0x7fe1e00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1deffd640 1 --2- 172.21.15.1:0/4109055448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1cc0039b0 0x7fe1cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1deffd640 1 --2- 172.21.15.1:0/4109055448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1cc004800 0x7fe1cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1deffd640 1 --2- 172.21.15.1:0/4109055448 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1cc0051c0 0x7fe1cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1deffd640 1 -- 172.21.15.1:0/4109055448 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e0155800 con 0x7fe1cc0039b0 2024-03-06T07:21:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1deffd640 1 -- 172.21.15.1:0/4109055448 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc00c160 con 0x7fe1cc004800 2024-03-06T07:21:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1deffd640 1 -- 172.21.15.1:0/4109055448 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc00c360 con 0x7fe1cc0051c0 2024-03-06T07:21:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1dffff640 1 --2- 172.21.15.1:0/4109055448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1cc0039b0 0x7fe1cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1df7fe640 1 --2- 172.21.15.1:0/4109055448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1cc004800 0x7fe1cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1e4a25640 1 --2- 172.21.15.1:0/4109055448 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1cc0051c0 0x7fe1cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1dffff640 1 -- 172.21.15.1:0/4109055448 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1cc0051c0 msgr2=0x7fe1cc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1dffff640 1 --2- 172.21.15.1:0/4109055448 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1cc0051c0 0x7fe1cc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1dffff640 1 -- 172.21.15.1:0/4109055448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1cc004800 msgr2=0x7fe1cc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1dffff640 1 --2- 172.21.15.1:0/4109055448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1cc004800 0x7fe1cc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1dffff640 1 -- 172.21.15.1:0/4109055448 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1cc003520 con 0x7fe1cc0039b0 2024-03-06T07:21:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.969+0000 7fe1e4a25640 1 -- 172.21.15.1:0/4109055448 reap_dead start 2024-03-06T07:21:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.970+0000 7fe1dffff640 1 --2- 172.21.15.1:0/4109055448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1cc0039b0 0x7fe1cc003dd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0001750 tx=0x7fe1cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20fbf08274e6b0b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.970+0000 7fe1deffd640 1 -- 172.21.15.1:0/4109055448 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0001370 con 0x7fe1cc0039b0 2024-03-06T07:21:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.970+0000 7fe1deffd640 1 -- 172.21.15.1:0/4109055448 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d00052d0 con 0x7fe1cc0039b0 2024-03-06T07:21:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.970+0000 7fe1deffd640 1 -- 172.21.15.1:0/4109055448 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0003830 con 0x7fe1cc0039b0 2024-03-06T07:21:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.971+0000 7fe1e6488640 1 -- 172.21.15.1:0/4109055448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1cc0039b0 msgr2=0x7fe1cc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.971+0000 7fe1e6488640 1 --2- 172.21.15.1:0/4109055448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1cc0039b0 0x7fe1cc003dd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0001750 tx=0x7fe1cc005830 comp rx=0 tx=0).stop 2024-03-06T07:21:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.971+0000 7fe1e6488640 1 -- 172.21.15.1:0/4109055448 shutdown_connections 2024-03-06T07:21:09.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.971+0000 7fe1e6488640 1 --2- 172.21.15.1:0/4109055448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1cc0039b0 0x7fe1cc003dd0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.971+0000 7fe1e6488640 1 -- 172.21.15.1:0/4109055448 >> 172.21.15.1:0/4109055448 conn(0x7fe1e0066ee0 msgr2=0x7fe1e00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:09.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.971+0000 7fe1e6488640 1 -- 172.21.15.1:0/4109055448 shutdown_connections 2024-03-06T07:21:09.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.971+0000 7fe1e6488640 1 -- 172.21.15.1:0/4109055448 wait complete. 2024-03-06T07:21:09.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.972+0000 7fe1e6488640 1 Processor -- start 2024-03-06T07:21:09.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.972+0000 7fe1e6488640 1 -- start start 2024-03-06T07:21:09.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.973+0000 7fe1e6488640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1cc0039b0 0x7fe1e0158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.973+0000 7fe1e6488640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1e014b370 0x7fe1e01ee290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.973+0000 7fe1e6488640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1e01ee7d0 0x7fe1e0150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.973+0000 7fe1e6488640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e01551b0 con 0x7fe1cc0039b0 2024-03-06T07:21:09.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.973+0000 7fe1e6488640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e00c6fa0 con 0x7fe1e014b370 2024-03-06T07:21:09.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.973+0000 7fe1e6488640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e0155490 con 0x7fe1e01ee7d0 2024-03-06T07:21:09.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.973+0000 7fe1dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1cc0039b0 0x7fe1e0158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.973+0000 7fe1dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1cc0039b0 0x7fe1e0158090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52464/0 (socket says 172.21.15.1:52464) 2024-03-06T07:21:09.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.973+0000 7fe1dffff640 1 -- 172.21.15.1:0/3561502909 learned_addr learned my addr 172.21.15.1:0/3561502909 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:09.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.973+0000 7fe1e4a25640 1 --2- 172.21.15.1:0/3561502909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1e01ee7d0 0x7fe1e0150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.973+0000 7fe1df7fe640 1 --2- 172.21.15.1:0/3561502909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1e014b370 0x7fe1e01ee290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.973+0000 7fe1dffff640 1 -- 172.21.15.1:0/3561502909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1e01ee7d0 msgr2=0x7fe1e0150320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.973+0000 7fe1dffff640 1 --2- 172.21.15.1:0/3561502909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1e01ee7d0 0x7fe1e0150320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.973+0000 7fe1dffff640 1 -- 172.21.15.1:0/3561502909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1e014b370 msgr2=0x7fe1e01ee290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.973+0000 7fe1dffff640 1 --2- 172.21.15.1:0/3561502909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1e014b370 0x7fe1e01ee290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.973+0000 7fe1dffff640 1 -- 172.21.15.1:0/3561502909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e01585d0 con 0x7fe1cc0039b0 2024-03-06T07:21:09.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.973+0000 7fe1dffff640 1 --2- 172.21.15.1:0/3561502909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1cc0039b0 0x7fe1e0158090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0000c00 tx=0x7fe1d0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:09.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.974+0000 7fe1dcff9640 1 -- 172.21.15.1:0/3561502909 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0041070 con 0x7fe1cc0039b0 2024-03-06T07:21:09.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.974+0000 7fe1dcff9640 1 -- 172.21.15.1:0/3561502909 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d0005480 con 0x7fe1cc0039b0 2024-03-06T07:21:09.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.974+0000 7fe1e6488640 1 -- 172.21.15.1:0/3561502909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e01eec80 con 0x7fe1cc0039b0 2024-03-06T07:21:09.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.974+0000 7fe1dcff9640 1 -- 172.21.15.1:0/3561502909 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0001370 con 0x7fe1cc0039b0 2024-03-06T07:21:09.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.974+0000 7fe1e6488640 1 -- 172.21.15.1:0/3561502909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e0158a00 con 0x7fe1cc0039b0 2024-03-06T07:21:09.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.975+0000 7fe1e6488640 1 -- 172.21.15.1:0/3561502909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e01eee50 con 0x7fe1cc0039b0 2024-03-06T07:21:09.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.975+0000 7fe1dcff9640 1 -- 172.21.15.1:0/3561502909 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1d0044070 con 0x7fe1cc0039b0 2024-03-06T07:21:09.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.976+0000 7fe1dcff9640 1 --2- 172.21.15.1:0/3561502909 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe1c006b4d0 0x7fe1c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.976+0000 7fe1dcff9640 1 -- 172.21.15.1:0/3561502909 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fe1d00abf10 con 0x7fe1cc0039b0 2024-03-06T07:21:09.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.976+0000 7fe1df7fe640 1 --2- 172.21.15.1:0/3561502909 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe1c006b4d0 0x7fe1c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.976+0000 7fe1df7fe640 1 --2- 172.21.15.1:0/3561502909 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe1c006b4d0 0x7fe1c006d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe1e014e610 tx=0x7fe1c80078f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:09.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:09.980+0000 7fe1dcff9640 1 -- 172.21.15.1:0/3561502909 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1e01585d0 con 0x7fe1cc0039b0 2024-03-06T07:21:10.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.206+0000 7fe1e6488640 1 -- 172.21.15.1:0/3561502909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1e00b06f0 con 0x7fe1cc0039b0 2024-03-06T07:21:10.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.207+0000 7fe1dcff9640 1 -- 172.21.15.1:0/3561502909 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4964 (secure 0 0 0) 0x7fe1d007a3d0 con 0x7fe1cc0039b0 2024-03-06T07:21:10.209 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:10.209 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2822958969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2822958969},{"type":"v1","addr":"172.21.15.98:6836","nonce":2822958969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2442906233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2442906233},{"type":"v1","addr":"172.21.15.98:6837","nonce":2442906233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/448746024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":448746024},{"type":"v1","addr":"172.21.15.1:6834","nonce":448746024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:21.207834+0000","modified":"2024-03-06T07:20:37.259661+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1315115982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1315115982},{"type":"v1","addr":"172.21.15.1:6835","nonce":1315115982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-03-06T07:21:10.209 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-03-06T07:21:10.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.209+0000 7fe1e6488640 1 -- 172.21.15.1:0/3561502909 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe1c006b4d0 msgr2=0x7fe1c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.209+0000 7fe1e6488640 1 --2- 172.21.15.1:0/3561502909 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe1c006b4d0 0x7fe1c006d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe1e014e610 tx=0x7fe1c80078f0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.209+0000 7fe1e6488640 1 -- 172.21.15.1:0/3561502909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1cc0039b0 msgr2=0x7fe1e0158090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.209+0000 7fe1e6488640 1 --2- 172.21.15.1:0/3561502909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1cc0039b0 0x7fe1e0158090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0000c00 tx=0x7fe1d0001120 comp rx=0 tx=0).stop 2024-03-06T07:21:10.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.209+0000 7fe1e6488640 1 -- 172.21.15.1:0/3561502909 shutdown_connections 2024-03-06T07:21:10.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.209+0000 7fe1e6488640 1 --2- 172.21.15.1:0/3561502909 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe1c006b4d0 0x7fe1c006d990 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.209+0000 7fe1e6488640 1 --2- 172.21.15.1:0/3561502909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1e01ee7d0 0x7fe1e0150320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.209+0000 7fe1e6488640 1 --2- 172.21.15.1:0/3561502909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1e014b370 0x7fe1e01ee290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.209+0000 7fe1e6488640 1 --2- 172.21.15.1:0/3561502909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1cc0039b0 0x7fe1e0158090 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.209+0000 7fe1e6488640 1 -- 172.21.15.1:0/3561502909 >> 172.21.15.1:0/3561502909 conn(0x7fe1e0066ee0 msgr2=0x7fe1e00c0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:10.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.209+0000 7fe1e6488640 1 -- 172.21.15.1:0/3561502909 shutdown_connections 2024-03-06T07:21:10.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.209+0000 7fe1e6488640 1 -- 172.21.15.1:0/3561502909 wait complete. 2024-03-06T07:21:10.222 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:10.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.324+0000 7f1160db5640 1 Processor -- start 2024-03-06T07:21:10.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.324+0000 7f1160db5640 1 -- start start 2024-03-06T07:21:10.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.325+0000 7f1160db5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115c0bf9d0 0x7f115c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.325+0000 7f1160db5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115c0c2fc0 con 0x7f115c0c6b10 2024-03-06T07:21:10.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.325+0000 7f1160db5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115c158da0 con 0x7f115c152650 2024-03-06T07:21:10.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.325+0000 7f1160db5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115c155800 con 0x7f115c0bf9d0 2024-03-06T07:21:10.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.325+0000 7f1159d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f115c0c6b10 0x7f115c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59288/0 (socket says 172.21.15.1:59288) 2024-03-06T07:21:10.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.325+0000 7f1159d9b640 1 -- 172.21.15.1:0/1689959206 learned_addr learned my addr 172.21.15.1:0/1689959206 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:10.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.325+0000 7f115ad9d640 1 --2- 172.21.15.1:0/1689959206 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115c0bf9d0 0x7f115c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115959a640 1 -- 172.21.15.1:0/1689959206 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847222594 0 0) 0x7f115c0c2fc0 con 0x7f115c0c6b10 2024-03-06T07:21:10.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115959a640 1 -- 172.21.15.1:0/1689959206 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1144003520 con 0x7f115c0c6b10 2024-03-06T07:21:10.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115959a640 1 -- 172.21.15.1:0/1689959206 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567607695 0 0) 0x7f115c158da0 con 0x7f115c152650 2024-03-06T07:21:10.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115959a640 1 -- 172.21.15.1:0/1689959206 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f115c0c2fc0 con 0x7f115c152650 2024-03-06T07:21:10.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115959a640 1 -- 172.21.15.1:0/1689959206 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 229480892 0 0) 0x7f1144003520 con 0x7f115c0c6b10 2024-03-06T07:21:10.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115959a640 1 -- 172.21.15.1:0/1689959206 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115c0bf9d0 msgr2=0x7f115c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:10.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115959a640 1 --2- 172.21.15.1:0/1689959206 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115c0bf9d0 0x7f115c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115959a640 1 -- 172.21.15.1:0/1689959206 >> v1:172.21.15.98:6789/0 conn(0x7f115c152650 legacy=0x7f115c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115959a640 1 -- 172.21.15.1:0/1689959206 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115c0c6690 con 0x7f115c0c6b10 2024-03-06T07:21:10.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115959a640 1 -- 172.21.15.1:0/1689959206 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f1150001690 con 0x7f115c0c6b10 2024-03-06T07:21:10.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115959a640 1 -- 172.21.15.1:0/1689959206 >> v1:172.21.15.1:6789/0 conn(0x7f115c0c6b10 legacy=0x7f115c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115959a640 1 --2- 172.21.15.1:0/1689959206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1144003b70 0x7f1144003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115959a640 1 --2- 172.21.15.1:0/1689959206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1144004800 0x7f1144004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115959a640 1 --2- 172.21.15.1:0/1689959206 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115c0bf9d0 0x7f11440051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115959a640 1 -- 172.21.15.1:0/1689959206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115c155800 con 0x7f1144003b70 2024-03-06T07:21:10.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115959a640 1 -- 172.21.15.1:0/1689959206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114400bb00 con 0x7f1144004800 2024-03-06T07:21:10.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115959a640 1 -- 172.21.15.1:0/1689959206 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f114400bd00 con 0x7f115c0bf9d0 2024-03-06T07:21:10.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115a59c640 1 --2- 172.21.15.1:0/1689959206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1144003b70 0x7f1144003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f1159d9b640 1 --2- 172.21.15.1:0/1689959206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1144004800 0x7f1144004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115ad9d640 1 --2- 172.21.15.1:0/1689959206 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115c0bf9d0 0x7f11440051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115a59c640 1 -- 172.21.15.1:0/1689959206 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115c0bf9d0 msgr2=0x7f11440051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:10.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115a59c640 1 --2- 172.21.15.1:0/1689959206 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115c0bf9d0 0x7f11440051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115a59c640 1 -- 172.21.15.1:0/1689959206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1144004800 msgr2=0x7f1144004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:10.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115a59c640 1 --2- 172.21.15.1:0/1689959206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1144004800 0x7f1144004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f115a59c640 1 -- 172.21.15.1:0/1689959206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1144003520 con 0x7f1144003b70 2024-03-06T07:21:10.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.326+0000 7f1159d9b640 1 --2- 172.21.15.1:0/1689959206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1144004800 0x7f1144004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:10.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.327+0000 7f115a59c640 1 --2- 172.21.15.1:0/1689959206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1144003b70 0x7f1144003f70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f114c0018d0 tx=0x7f1144005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d438ea7a787a4a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:10.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.327+0000 7f115959a640 1 -- 172.21.15.1:0/1689959206 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c037070 con 0x7f1144003b70 2024-03-06T07:21:10.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.327+0000 7f115959a640 1 -- 172.21.15.1:0/1689959206 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114c0052d0 con 0x7f1144003b70 2024-03-06T07:21:10.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.327+0000 7f115959a640 1 -- 172.21.15.1:0/1689959206 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c0014f0 con 0x7f1144003b70 2024-03-06T07:21:10.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.328+0000 7f1160db5640 1 -- 172.21.15.1:0/1689959206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1144003b70 msgr2=0x7f1144003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.328+0000 7f1160db5640 1 --2- 172.21.15.1:0/1689959206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1144003b70 0x7f1144003f70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f114c0018d0 tx=0x7f1144005830 comp rx=0 tx=0).stop 2024-03-06T07:21:10.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.328+0000 7f115ad9d640 1 -- 172.21.15.1:0/1689959206 reap_dead start 2024-03-06T07:21:10.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.328+0000 7f1160db5640 1 -- 172.21.15.1:0/1689959206 shutdown_connections 2024-03-06T07:21:10.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.328+0000 7f1160db5640 1 -- 172.21.15.1:0/1689959206 >> 172.21.15.1:0/1689959206 conn(0x7f115c066ee0 msgr2=0x7f115c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:10.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.328+0000 7f1160db5640 1 -- 172.21.15.1:0/1689959206 shutdown_connections 2024-03-06T07:21:10.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.328+0000 7f1160db5640 1 -- 172.21.15.1:0/1689959206 wait complete. 2024-03-06T07:21:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.329+0000 7f1160db5640 1 Processor -- start 2024-03-06T07:21:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.329+0000 7f1160db5640 1 -- start start 2024-03-06T07:21:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.330+0000 7f1160db5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1144003b70 0x7f115c0c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.330+0000 7f1160db5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115c153420 0x7f115c14b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.330+0000 7f1160db5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f115c154190 0x7f115c14b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.330+0000 7f1160db5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115c1551b0 con 0x7f1144003b70 2024-03-06T07:21:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.330+0000 7f1160db5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115c0c6fa0 con 0x7f115c154190 2024-03-06T07:21:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.330+0000 7f1160db5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115c155490 con 0x7f115c153420 2024-03-06T07:21:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.330+0000 7f115a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1144003b70 0x7f115c0c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.330+0000 7f115a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1144003b70 0x7f115c0c9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52482/0 (socket says 172.21.15.1:52482) 2024-03-06T07:21:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.330+0000 7f115a59c640 1 -- 172.21.15.1:0/1696596505 learned_addr learned my addr 172.21.15.1:0/1696596505 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.330+0000 7f1159d9b640 1 --2- 172.21.15.1:0/1696596505 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115c153420 0x7f115c14b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.330+0000 7f115ad9d640 1 --2- 172.21.15.1:0/1696596505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f115c154190 0x7f115c14b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.330+0000 7f115a59c640 1 -- 172.21.15.1:0/1696596505 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115c153420 msgr2=0x7f115c14b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.330+0000 7f115a59c640 1 --2- 172.21.15.1:0/1696596505 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115c153420 0x7f115c14b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.330+0000 7f115a59c640 1 -- 172.21.15.1:0/1696596505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f115c154190 msgr2=0x7f115c14b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.330+0000 7f115a59c640 1 --2- 172.21.15.1:0/1696596505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f115c154190 0x7f115c14b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.330+0000 7f115a59c640 1 -- 172.21.15.1:0/1696596505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115c1eb950 con 0x7f1144003b70 2024-03-06T07:21:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.330+0000 7f115a59c640 1 --2- 172.21.15.1:0/1696596505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1144003b70 0x7f115c0c9c10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f114c035040 tx=0x7f114c0012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.331+0000 7f114affd640 1 -- 172.21.15.1:0/1696596505 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c037070 con 0x7f1144003b70 2024-03-06T07:21:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.331+0000 7f114affd640 1 -- 172.21.15.1:0/1696596505 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114c038040 con 0x7f1144003b70 2024-03-06T07:21:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.331+0000 7f114affd640 1 -- 172.21.15.1:0/1696596505 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c0014f0 con 0x7f1144003b70 2024-03-06T07:21:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.331+0000 7f1160db5640 1 -- 172.21.15.1:0/1696596505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f115c1ebc30 con 0x7f1144003b70 2024-03-06T07:21:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.331+0000 7f1160db5640 1 -- 172.21.15.1:0/1696596505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f115c14bbf0 con 0x7f1144003b70 2024-03-06T07:21:10.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.332+0000 7f1160db5640 1 -- 172.21.15.1:0/1696596505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f115c1ee3e0 con 0x7f1144003b70 2024-03-06T07:21:10.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.332+0000 7f114affd640 1 -- 172.21.15.1:0/1696596505 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f114c0052d0 con 0x7f1144003b70 2024-03-06T07:21:10.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.332+0000 7f114affd640 1 --2- 172.21.15.1:0/1696596505 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f113c06b500 0x7f113c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.333+0000 7f114affd640 1 -- 172.21.15.1:0/1696596505 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f114c0abb80 con 0x7f1144003b70 2024-03-06T07:21:10.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.333+0000 7f1159d9b640 1 --2- 172.21.15.1:0/1696596505 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f113c06b500 0x7f113c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.333+0000 7f1159d9b640 1 --2- 172.21.15.1:0/1696596505 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f113c06b500 0x7f113c06d9c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f115c14e930 tx=0x7f1150007040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:10.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.337+0000 7f114affd640 1 -- 172.21.15.1:0/1696596505 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f114c07a160 con 0x7f1144003b70 2024-03-06T07:21:10.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.518+0000 7f1160db5640 1 -- 172.21.15.1:0/1696596505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f115c067fd0 con 0x7f1144003b70 2024-03-06T07:21:10.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.519+0000 7f114affd640 1 -- 172.21.15.1:0/1696596505 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18728 (secure 0 0 0) 0x7f114c07e200 con 0x7f1144003b70 2024-03-06T07:21:10.521 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:10.521 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":27,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:21:10.090355+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:10.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.520+0000 7f1160db5640 1 -- 172.21.15.1:0/1696596505 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f113c06b500 msgr2=0x7f113c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.520+0000 7f1160db5640 1 --2- 172.21.15.1:0/1696596505 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f113c06b500 0x7f113c06d9c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f115c14e930 tx=0x7f1150007040 comp rx=0 tx=0).stop 2024-03-06T07:21:10.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.520+0000 7f1160db5640 1 -- 172.21.15.1:0/1696596505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1144003b70 msgr2=0x7f115c0c9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.520+0000 7f1160db5640 1 --2- 172.21.15.1:0/1696596505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1144003b70 0x7f115c0c9c10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f114c035040 tx=0x7f114c0012d0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.521+0000 7f1160db5640 1 -- 172.21.15.1:0/1696596505 shutdown_connections 2024-03-06T07:21:10.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.521+0000 7f1160db5640 1 --2- 172.21.15.1:0/1696596505 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f113c06b500 0x7f113c06d9c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.521+0000 7f1160db5640 1 --2- 172.21.15.1:0/1696596505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f115c154190 0x7f115c14b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.521+0000 7f1160db5640 1 --2- 172.21.15.1:0/1696596505 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115c153420 0x7f115c14b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.521+0000 7f1160db5640 1 --2- 172.21.15.1:0/1696596505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1144003b70 0x7f115c0c9c10 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.521+0000 7f1160db5640 1 -- 172.21.15.1:0/1696596505 >> 172.21.15.1:0/1696596505 conn(0x7f115c066ee0 msgr2=0x7f115c0c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:10.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.521+0000 7f1160db5640 1 -- 172.21.15.1:0/1696596505 shutdown_connections 2024-03-06T07:21:10.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.521+0000 7f1160db5640 1 -- 172.21.15.1:0/1696596505 wait complete. 2024-03-06T07:21:10.534 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:21:10.534 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:10.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.633+0000 7fe5b9c81640 1 Processor -- start 2024-03-06T07:21:10.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.633+0000 7fe5b9c81640 1 -- start start 2024-03-06T07:21:10.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.634+0000 7fe5b9c81640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe5b40c3970 0x7fe5b40c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.634+0000 7fe5b9c81640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe5b4152840 0x7fe5b41553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.634+0000 7fe5b9c81640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b40c5e40 con 0x7fe5b40c60b0 2024-03-06T07:21:10.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.634+0000 7fe5b9c81640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5b40c67f0 con 0x7fe5b40c3970 2024-03-06T07:21:10.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.634+0000 7fe5b9c81640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5b40c6a40 con 0x7fe5b4152840 2024-03-06T07:21:10.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.635+0000 7fe5b37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe5b40c3970 0x7fe5b40c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.635+0000 7fe5b37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe5b40c3970 0x7fe5b40c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52496/0 (socket says 172.21.15.1:52496) 2024-03-06T07:21:10.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.635+0000 7fe5b37fe640 1 -- 172.21.15.1:0/1958622828 learned_addr learned my addr 172.21.15.1:0/1958622828 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:10.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.635+0000 7fe5b3fff640 1 --2- 172.21.15.1:0/1958622828 >> v2:172.21.15.98:3300/0 conn(0x7fe5b4152840 0x7fe5b41553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.635+0000 7fe5b37fe640 1 -- 172.21.15.1:0/1958622828 >> v2:172.21.15.98:3300/0 conn(0x7fe5b4152840 msgr2=0x7fe5b41553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.635+0000 7fe5b37fe640 1 --2- 172.21.15.1:0/1958622828 >> v2:172.21.15.98:3300/0 conn(0x7fe5b4152840 0x7fe5b41553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.635+0000 7fe5b37fe640 1 -- 172.21.15.1:0/1958622828 >> v1:172.21.15.98:6789/0 conn(0x7fe5b40c60b0 legacy=0x7fe5b4152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.635+0000 7fe5b37fe640 1 -- 172.21.15.1:0/1958622828 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b40c6d20 con 0x7fe5b40c3970 2024-03-06T07:21:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.635+0000 7fe5b3fff640 1 --2- 172.21.15.1:0/1958622828 >> v2:172.21.15.98:3300/0 conn(0x7fe5b4152840 0x7fe5b41553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.636+0000 7fe5b37fe640 1 --2- 172.21.15.1:0/1958622828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b40c3970 0x7fe5b40c8170 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40c86b0 tx=0x7fe5a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f88e90a2cd41e89 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.636+0000 7fe5b27fc640 1 -- 172.21.15.1:0/1958622828 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4035020 con 0x7fe5b40c3970 2024-03-06T07:21:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.636+0000 7fe5b27fc640 1 -- 172.21.15.1:0/1958622828 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a4031870 con 0x7fe5b40c3970 2024-03-06T07:21:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.636+0000 7fe5b27fc640 1 -- 172.21.15.1:0/1958622828 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4031c40 con 0x7fe5b40c3970 2024-03-06T07:21:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.636+0000 7fe5b9c81640 1 -- 172.21.15.1:0/1958622828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b40c3970 msgr2=0x7fe5b40c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.636+0000 7fe5b9c81640 1 --2- 172.21.15.1:0/1958622828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b40c3970 0x7fe5b40c8170 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40c86b0 tx=0x7fe5a402b910 comp rx=0 tx=0).stop 2024-03-06T07:21:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.637+0000 7fe5b9c81640 1 -- 172.21.15.1:0/1958622828 shutdown_connections 2024-03-06T07:21:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.637+0000 7fe5b9c81640 1 --2- 172.21.15.1:0/1958622828 >> v2:172.21.15.98:3300/0 conn(0x7fe5b4152840 0x7fe5b41553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.637+0000 7fe5b9c81640 1 --2- 172.21.15.1:0/1958622828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b40c3970 0x7fe5b40c8170 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.637+0000 7fe5b9c81640 1 -- 172.21.15.1:0/1958622828 >> 172.21.15.1:0/1958622828 conn(0x7fe5b40c88f0 msgr2=0x7fe5b40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.637+0000 7fe5b9c81640 1 -- 172.21.15.1:0/1958622828 shutdown_connections 2024-03-06T07:21:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.637+0000 7fe5b9c81640 1 -- 172.21.15.1:0/1958622828 wait complete. 2024-03-06T07:21:10.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.638+0000 7fe5b9c81640 1 Processor -- start 2024-03-06T07:21:10.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.638+0000 7fe5b9c81640 1 -- start start 2024-03-06T07:21:10.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.638+0000 7fe5b9c81640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b40c3970 0x7fe5b41580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.638+0000 7fe5b9c81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe5b40c60b0 0x7fe5b414c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.638+0000 7fe5b9c81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5b4152840 0x7fe5b41ea7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.638+0000 7fe5b9c81640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b40c75c0 con 0x7fe5b40c3970 2024-03-06T07:21:10.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.638+0000 7fe5b9c81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b40c72e0 con 0x7fe5b4152840 2024-03-06T07:21:10.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.638+0000 7fe5b9c81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b40c3d50 con 0x7fe5b40c60b0 2024-03-06T07:21:10.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5b37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b40c3970 0x7fe5b41580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5b37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b40c3970 0x7fe5b41580d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52508/0 (socket says 172.21.15.1:52508) 2024-03-06T07:21:10.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5b37fe640 1 -- 172.21.15.1:0/1498575515 learned_addr learned my addr 172.21.15.1:0/1498575515 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5b3fff640 1 --2- 172.21.15.1:0/1498575515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5b4152840 0x7fe5b41ea7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5b2ffd640 1 --2- 172.21.15.1:0/1498575515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe5b40c60b0 0x7fe5b414c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5b37fe640 1 -- 172.21.15.1:0/1498575515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe5b40c60b0 msgr2=0x7fe5b414c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5b37fe640 1 --2- 172.21.15.1:0/1498575515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe5b40c60b0 0x7fe5b414c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5b37fe640 1 -- 172.21.15.1:0/1498575515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5b4152840 msgr2=0x7fe5b41ea7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5b37fe640 1 --2- 172.21.15.1:0/1498575515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5b4152840 0x7fe5b41ea7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5b37fe640 1 -- 172.21.15.1:0/1498575515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b4155940 con 0x7fe5b40c3970 2024-03-06T07:21:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5b2ffd640 1 --2- 172.21.15.1:0/1498575515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe5b40c60b0 0x7fe5b414c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5b3fff640 1 --2- 172.21.15.1:0/1498575515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5b4152840 0x7fe5b41ea7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5b37fe640 1 --2- 172.21.15.1:0/1498575515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b40c3970 0x7fe5b41580d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4000c00 tx=0x7fe5a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5abfff640 1 -- 172.21.15.1:0/1498575515 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a40443f0 con 0x7fe5b40c3970 2024-03-06T07:21:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5abfff640 1 -- 172.21.15.1:0/1498575515 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a4044590 con 0x7fe5b40c3970 2024-03-06T07:21:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5b9c81640 1 -- 172.21.15.1:0/1498575515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b40aca80 con 0x7fe5b40c3970 2024-03-06T07:21:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5abfff640 1 -- 172.21.15.1:0/1498575515 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4044980 con 0x7fe5b40c3970 2024-03-06T07:21:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.639+0000 7fe5b9c81640 1 -- 172.21.15.1:0/1498575515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b41eded0 con 0x7fe5b40c3970 2024-03-06T07:21:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.640+0000 7fe5b9c81640 1 -- 172.21.15.1:0/1498575515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5b40686e0 con 0x7fe5b40c3970 2024-03-06T07:21:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.641+0000 7fe5abfff640 1 -- 172.21.15.1:0/1498575515 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5a404b060 con 0x7fe5b40c3970 2024-03-06T07:21:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.642+0000 7fe5abfff640 1 --2- 172.21.15.1:0/1498575515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe59406b4d0 0x7fe59406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.642+0000 7fe5abfff640 1 -- 172.21.15.1:0/1498575515 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fe5a40ac880 con 0x7fe5b40c3970 2024-03-06T07:21:10.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.642+0000 7fe5b2ffd640 1 --2- 172.21.15.1:0/1498575515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe59406b4d0 0x7fe59406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.642+0000 7fe5b2ffd640 1 --2- 172.21.15.1:0/1498575515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe59406b4d0 0x7fe59406d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe59c007fd0 tx=0x7fe59c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:10.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.646+0000 7fe5abfff640 1 -- 172.21.15.1:0/1498575515 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5a403f070 con 0x7fe5b40c3970 2024-03-06T07:21:10.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.884+0000 7fe5b9c81640 1 -- 172.21.15.1:0/1498575515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5b41ee590 con 0x7fe5b40c3970 2024-03-06T07:21:10.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.884+0000 7fe5abfff640 1 -- 172.21.15.1:0/1498575515 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4964 (secure 0 0 0) 0x7fe5a407b390 con 0x7fe5b40c3970 2024-03-06T07:21:10.886 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:10.887 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2822958969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2822958969},{"type":"v1","addr":"172.21.15.98:6836","nonce":2822958969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2442906233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2442906233},{"type":"v1","addr":"172.21.15.98:6837","nonce":2442906233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/448746024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":448746024},{"type":"v1","addr":"172.21.15.1:6834","nonce":448746024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:21.207834+0000","modified":"2024-03-06T07:20:37.259661+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1315115982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1315115982},{"type":"v1","addr":"172.21.15.1:6835","nonce":1315115982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-03-06T07:21:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-03-06T07:21:10.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.887+0000 7fe5b9c81640 1 -- 172.21.15.1:0/1498575515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe59406b4d0 msgr2=0x7fe59406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.887+0000 7fe5b9c81640 1 --2- 172.21.15.1:0/1498575515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe59406b4d0 0x7fe59406d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe59c007fd0 tx=0x7fe59c009040 comp rx=0 tx=0).stop 2024-03-06T07:21:10.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.887+0000 7fe5b9c81640 1 -- 172.21.15.1:0/1498575515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b40c3970 msgr2=0x7fe5b41580d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.887+0000 7fe5b9c81640 1 --2- 172.21.15.1:0/1498575515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b40c3970 0x7fe5b41580d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4000c00 tx=0x7fe5a40046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.887+0000 7fe5b9c81640 1 -- 172.21.15.1:0/1498575515 shutdown_connections 2024-03-06T07:21:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.887+0000 7fe5b9c81640 1 --2- 172.21.15.1:0/1498575515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe59406b4d0 0x7fe59406d990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.887+0000 7fe5b9c81640 1 --2- 172.21.15.1:0/1498575515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5b4152840 0x7fe5b41ea7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.887+0000 7fe5b9c81640 1 --2- 172.21.15.1:0/1498575515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe5b40c60b0 0x7fe5b414c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.887+0000 7fe5b9c81640 1 --2- 172.21.15.1:0/1498575515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b40c3970 0x7fe5b41580d0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.887+0000 7fe5b9c81640 1 -- 172.21.15.1:0/1498575515 >> 172.21.15.1:0/1498575515 conn(0x7fe5b40c88f0 msgr2=0x7fe5b40c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.887+0000 7fe5b9c81640 1 -- 172.21.15.1:0/1498575515 shutdown_connections 2024-03-06T07:21:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:10.887+0000 7fe5b9c81640 1 -- 172.21.15.1:0/1498575515 wait complete. 2024-03-06T07:21:10.901 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.003+0000 7f7659c55640 1 Processor -- start 2024-03-06T07:21:11.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.004+0000 7f7659c55640 1 -- start start 2024-03-06T07:21:11.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.005+0000 7f7659c55640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7654152650 0x7f7654155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.005+0000 7f7659c55640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540bf9d0 0x7f76540c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.005+0000 7f7659c55640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76540c2fc0 con 0x7f76540c6b10 2024-03-06T07:21:11.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.005+0000 7f7659c55640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7654155800 con 0x7f7654152650 2024-03-06T07:21:11.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.005+0000 7f7659c55640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7654158da0 con 0x7f76540bf9d0 2024-03-06T07:21:11.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.005+0000 7f76537fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7654152650 0x7f7654155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.005+0000 7f7652ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f76540c6b10 0x7f7654158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59290/0 (socket says 172.21.15.1:59290) 2024-03-06T07:21:11.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.005+0000 7f7652ffd640 1 -- 172.21.15.1:0/90448992 learned_addr learned my addr 172.21.15.1:0/90448992 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:11.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.005+0000 7f7653fff640 1 --2- 172.21.15.1:0/90448992 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540bf9d0 0x7f76540c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.005+0000 7f76537fe640 1 -- 172.21.15.1:0/90448992 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540bf9d0 msgr2=0x7f76540c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.005+0000 7f76537fe640 1 --2- 172.21.15.1:0/90448992 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540bf9d0 0x7f76540c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.005+0000 7f76537fe640 1 -- 172.21.15.1:0/90448992 >> v1:172.21.15.1:6789/0 conn(0x7f76540c6b10 legacy=0x7f7654158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.005+0000 7f76537fe640 1 -- 172.21.15.1:0/90448992 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76540c6690 con 0x7f7654152650 2024-03-06T07:21:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.005+0000 7f76527fc640 1 -- 172.21.15.1:0/90448992 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937002276 0 0) 0x7f76540c2fc0 con 0x7f76540c6b10 2024-03-06T07:21:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.005+0000 7f7653fff640 1 --2- 172.21.15.1:0/90448992 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540bf9d0 0x7f76540c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.006+0000 7f76537fe640 1 --2- 172.21.15.1:0/90448992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7654152650 0x7f7654155c90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f76440097a0 tx=0x7f764402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33799dc29ce2d9d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.006+0000 7f76527fc640 1 -- 172.21.15.1:0/90448992 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76440315a0 con 0x7f7654152650 2024-03-06T07:21:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.006+0000 7f76527fc640 1 -- 172.21.15.1:0/90448992 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7644037070 con 0x7f7654152650 2024-03-06T07:21:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.006+0000 7f76527fc640 1 -- 172.21.15.1:0/90448992 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644031960 con 0x7f7654152650 2024-03-06T07:21:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.007+0000 7f7659c55640 1 -- 172.21.15.1:0/90448992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7654152650 msgr2=0x7f7654155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.007+0000 7f7659c55640 1 --2- 172.21.15.1:0/90448992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7654152650 0x7f7654155c90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f76440097a0 tx=0x7f764402b840 comp rx=0 tx=0).stop 2024-03-06T07:21:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.007+0000 7f7659c55640 1 -- 172.21.15.1:0/90448992 shutdown_connections 2024-03-06T07:21:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.007+0000 7f7659c55640 1 --2- 172.21.15.1:0/90448992 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540bf9d0 0x7f76540c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.007+0000 7f7659c55640 1 --2- 172.21.15.1:0/90448992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7654152650 0x7f7654155c90 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.007+0000 7f7659c55640 1 -- 172.21.15.1:0/90448992 >> 172.21.15.1:0/90448992 conn(0x7f7654066ee0 msgr2=0x7f76540672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.007+0000 7f7659c55640 1 -- 172.21.15.1:0/90448992 shutdown_connections 2024-03-06T07:21:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.007+0000 7f7659c55640 1 -- 172.21.15.1:0/90448992 wait complete. 2024-03-06T07:21:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.008+0000 7f7659c55640 1 Processor -- start 2024-03-06T07:21:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.008+0000 7f7659c55640 1 -- start start 2024-03-06T07:21:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f7659c55640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540bf9d0 0x7f765414fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f7659c55640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7654152650 0x7f76541e21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f7659c55640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76540ca580 0x7f76540c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f7659c55640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76541551b0 con 0x7f7654152650 2024-03-06T07:21:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f7659c55640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76540c6fa0 con 0x7f76540ca580 2024-03-06T07:21:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f7659c55640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7654155490 con 0x7f76540bf9d0 2024-03-06T07:21:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f7652ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7654152650 0x7f76541e21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f7652ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7654152650 0x7f76541e21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52538/0 (socket says 172.21.15.1:52538) 2024-03-06T07:21:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f7652ffd640 1 -- 172.21.15.1:0/2070924520 learned_addr learned my addr 172.21.15.1:0/2070924520 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f76537fe640 1 --2- 172.21.15.1:0/2070924520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540bf9d0 0x7f765414fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f7653fff640 1 --2- 172.21.15.1:0/2070924520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76540ca580 0x7f76540c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f7652ffd640 1 -- 172.21.15.1:0/2070924520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540bf9d0 msgr2=0x7f765414fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f7652ffd640 1 --2- 172.21.15.1:0/2070924520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540bf9d0 0x7f765414fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f7652ffd640 1 -- 172.21.15.1:0/2070924520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76540ca580 msgr2=0x7f76540c5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f7652ffd640 1 --2- 172.21.15.1:0/2070924520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76540ca580 0x7f76540c5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f7652ffd640 1 -- 172.21.15.1:0/2070924520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765414bc40 con 0x7f7654152650 2024-03-06T07:21:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f7652ffd640 1 --2- 172.21.15.1:0/2070924520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7654152650 0x7f76541e21c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f764800c4a0 tx=0x7f764800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.009+0000 7f763bfff640 1 -- 172.21.15.1:0/2070924520 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7648017070 con 0x7f7654152650 2024-03-06T07:21:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.010+0000 7f763bfff640 1 -- 172.21.15.1:0/2070924520 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7648005ce0 con 0x7f7654152650 2024-03-06T07:21:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.010+0000 7f7659c55640 1 -- 172.21.15.1:0/2070924520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7654151af0 con 0x7f7654152650 2024-03-06T07:21:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.010+0000 7f763bfff640 1 -- 172.21.15.1:0/2070924520 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76480072b0 con 0x7f7654152650 2024-03-06T07:21:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.010+0000 7f7659c55640 1 -- 172.21.15.1:0/2070924520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7654156810 con 0x7f7654152650 2024-03-06T07:21:11.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.010+0000 7f7659c55640 1 -- 172.21.15.1:0/2070924520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7654067fd0 con 0x7f7654152650 2024-03-06T07:21:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.011+0000 7f763bfff640 1 -- 172.21.15.1:0/2070924520 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f764801a070 con 0x7f7654152650 2024-03-06T07:21:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.012+0000 7f763bfff640 1 --2- 172.21.15.1:0/2070924520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764006b4d0 0x7f764006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.012+0000 7f763bfff640 1 -- 172.21.15.1:0/2070924520 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f7648089840 con 0x7f7654152650 2024-03-06T07:21:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.012+0000 7f76537fe640 1 --2- 172.21.15.1:0/2070924520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764006b4d0 0x7f764006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.013+0000 7f76537fe640 1 --2- 172.21.15.1:0/2070924520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764006b4d0 0x7f764006d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7644036000 tx=0x7f7644008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:11.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.016+0000 7f763bfff640 1 -- 172.21.15.1:0/2070924520 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7648057d00 con 0x7f7654152650 2024-03-06T07:21:11.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.312+0000 7f7659c55640 1 -- 172.21.15.1:0/2070924520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76540b06f0 con 0x7f7654152650 2024-03-06T07:21:11.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.312+0000 7f763bfff640 1 -- 172.21.15.1:0/2070924520 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4964 (secure 0 0 0) 0x7f764805bda0 con 0x7f7654152650 2024-03-06T07:21:11.314 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:11.315 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2822958969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2822958969},{"type":"v1","addr":"172.21.15.98:6836","nonce":2822958969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2442906233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2442906233},{"type":"v1","addr":"172.21.15.98:6837","nonce":2442906233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/448746024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":448746024},{"type":"v1","addr":"172.21.15.1:6834","nonce":448746024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:21.207834+0000","modified":"2024-03-06T07:20:37.259661+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1315115982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1315115982},{"type":"v1","addr":"172.21.15.1:6835","nonce":1315115982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-03-06T07:21:11.315 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-03-06T07:21:11.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.315+0000 7f7659c55640 1 -- 172.21.15.1:0/2070924520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764006b4d0 msgr2=0x7f764006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.315+0000 7f7659c55640 1 --2- 172.21.15.1:0/2070924520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764006b4d0 0x7f764006d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7644036000 tx=0x7f7644008030 comp rx=0 tx=0).stop 2024-03-06T07:21:11.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.315+0000 7f7659c55640 1 -- 172.21.15.1:0/2070924520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7654152650 msgr2=0x7f76541e21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.315+0000 7f7659c55640 1 --2- 172.21.15.1:0/2070924520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7654152650 0x7f76541e21c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f764800c4a0 tx=0x7f764800c960 comp rx=0 tx=0).stop 2024-03-06T07:21:11.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.315+0000 7f7659c55640 1 -- 172.21.15.1:0/2070924520 shutdown_connections 2024-03-06T07:21:11.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.315+0000 7f7659c55640 1 --2- 172.21.15.1:0/2070924520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764006b4d0 0x7f764006d990 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.315+0000 7f7659c55640 1 --2- 172.21.15.1:0/2070924520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76540ca580 0x7f76540c5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.315+0000 7f7659c55640 1 --2- 172.21.15.1:0/2070924520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7654152650 0x7f76541e21c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.315+0000 7f7659c55640 1 --2- 172.21.15.1:0/2070924520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540bf9d0 0x7f765414fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.315+0000 7f7659c55640 1 -- 172.21.15.1:0/2070924520 >> 172.21.15.1:0/2070924520 conn(0x7f7654066ee0 msgr2=0x7f76540c3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:11.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.315+0000 7f7659c55640 1 -- 172.21.15.1:0/2070924520 shutdown_connections 2024-03-06T07:21:11.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.315+0000 7f7659c55640 1 -- 172.21.15.1:0/2070924520 wait complete. 2024-03-06T07:21:11.329 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:11.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.431+0000 7fb612281640 1 Processor -- start 2024-03-06T07:21:11.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.431+0000 7fb612281640 1 -- start start 2024-03-06T07:21:11.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.432+0000 7fb612281640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb60c072c20 0x7fb60c0a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.432+0000 7fb612281640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb60c06c100 0x7fb60c06c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.432+0000 7fb612281640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb60c14fa40 con 0x7fb60c14f660 2024-03-06T07:21:11.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.432+0000 7fb612281640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60c06f6f0 con 0x7fb60c06c100 2024-03-06T07:21:11.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.432+0000 7fb612281640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c0a89f0 con 0x7fb60c072c20 2024-03-06T07:21:11.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.432+0000 7fb61081e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb60c06c100 0x7fb60c06c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.432+0000 7fb60bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb60c14f660 0x7fb60c152a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59292/0 (socket says 172.21.15.1:59292) 2024-03-06T07:21:11.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.432+0000 7fb60bfff640 1 -- 172.21.15.1:0/953701320 learned_addr learned my addr 172.21.15.1:0/953701320 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:11.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.432+0000 7fb60b7fe640 1 --2- 172.21.15.1:0/953701320 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb60c072c20 0x7fb60c0a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.432+0000 7fb61081e640 1 -- 172.21.15.1:0/953701320 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb60c072c20 msgr2=0x7fb60c0a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:11.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.432+0000 7fb61081e640 1 --2- 172.21.15.1:0/953701320 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb60c072c20 0x7fb60c0a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.433+0000 7fb61081e640 1 -- 172.21.15.1:0/953701320 >> v1:172.21.15.1:6789/0 conn(0x7fb60c14f660 legacy=0x7fb60c152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.433+0000 7fb60affd640 1 -- 172.21.15.1:0/953701320 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625835771 0 0) 0x7fb60c14fa40 con 0x7fb60c14f660 2024-03-06T07:21:11.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.433+0000 7fb61081e640 1 -- 172.21.15.1:0/953701320 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c06f940 con 0x7fb60c06c100 2024-03-06T07:21:11.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.433+0000 7fb61081e640 1 --2- 172.21.15.1:0/953701320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb60c06c100 0x7fb60c06c590 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb60400bfa0 tx=0x7fb60402e010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=920eedd0dc0512c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.433+0000 7fb60affd640 1 -- 172.21.15.1:0/953701320 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60402e9b0 con 0x7fb60c06c100 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.433+0000 7fb60affd640 1 -- 172.21.15.1:0/953701320 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb60402eb50 con 0x7fb60c06c100 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.434+0000 7fb60affd640 1 -- 172.21.15.1:0/953701320 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb604009540 con 0x7fb60c06c100 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.434+0000 7fb612281640 1 -- 172.21.15.1:0/953701320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb60c06c100 msgr2=0x7fb60c06c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.434+0000 7fb612281640 1 --2- 172.21.15.1:0/953701320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb60c06c100 0x7fb60c06c590 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb60400bfa0 tx=0x7fb60402e010 comp rx=0 tx=0).stop 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.434+0000 7fb612281640 1 -- 172.21.15.1:0/953701320 shutdown_connections 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.434+0000 7fb612281640 1 --2- 172.21.15.1:0/953701320 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb60c072c20 0x7fb60c0a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.434+0000 7fb612281640 1 --2- 172.21.15.1:0/953701320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb60c06c100 0x7fb60c06c590 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.434+0000 7fb612281640 1 -- 172.21.15.1:0/953701320 >> 172.21.15.1:0/953701320 conn(0x7fb60c071cf0 msgr2=0x7fb60c0720d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.434+0000 7fb612281640 1 -- 172.21.15.1:0/953701320 shutdown_connections 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.434+0000 7fb612281640 1 -- 172.21.15.1:0/953701320 wait complete. 2024-03-06T07:21:11.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.435+0000 7fb612281640 1 Processor -- start 2024-03-06T07:21:11.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.436+0000 7fb612281640 1 -- start start 2024-03-06T07:21:11.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.436+0000 7fb612281640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb60c06c100 0x7fb60c0c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.436+0000 7fb612281640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb60c072c20 0x7fb60c0c05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.436+0000 7fb612281640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb60c14f660 0x7fb60c1eac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.436+0000 7fb612281640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c1524e0 con 0x7fb60c06c100 2024-03-06T07:21:11.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.436+0000 7fb612281640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c152200 con 0x7fb60c072c20 2024-03-06T07:21:11.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.436+0000 7fb612281640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c067050 con 0x7fb60c14f660 2024-03-06T07:21:11.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.436+0000 7fb60bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb60c06c100 0x7fb60c0c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.437+0000 7fb60bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb60c06c100 0x7fb60c0c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52566/0 (socket says 172.21.15.1:52566) 2024-03-06T07:21:11.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.437+0000 7fb60bfff640 1 -- 172.21.15.1:0/807412167 learned_addr learned my addr 172.21.15.1:0/807412167 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:11.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.437+0000 7fb61081e640 1 --2- 172.21.15.1:0/807412167 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb60c14f660 0x7fb60c1eac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.437+0000 7fb60b7fe640 1 --2- 172.21.15.1:0/807412167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb60c072c20 0x7fb60c0c05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.437+0000 7fb60bfff640 1 -- 172.21.15.1:0/807412167 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb60c14f660 msgr2=0x7fb60c1eac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.437+0000 7fb60bfff640 1 --2- 172.21.15.1:0/807412167 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb60c14f660 0x7fb60c1eac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.437+0000 7fb60bfff640 1 -- 172.21.15.1:0/807412167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb60c072c20 msgr2=0x7fb60c0c05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.437+0000 7fb60bfff640 1 --2- 172.21.15.1:0/807412167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb60c072c20 0x7fb60c0c05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.437+0000 7fb60bfff640 1 -- 172.21.15.1:0/807412167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c0c5d10 con 0x7fb60c06c100 2024-03-06T07:21:11.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.437+0000 7fb60b7fe640 1 --2- 172.21.15.1:0/807412167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb60c072c20 0x7fb60c0c05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:11.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.437+0000 7fb60bfff640 1 --2- 172.21.15.1:0/807412167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb60c06c100 0x7fb60c0c57d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00c4d0 tx=0x7fb5fc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:11.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.437+0000 7fb608ff9640 1 -- 172.21.15.1:0/807412167 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc018070 con 0x7fb60c06c100 2024-03-06T07:21:11.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.437+0000 7fb608ff9640 1 -- 172.21.15.1:0/807412167 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5fc007500 con 0x7fb60c06c100 2024-03-06T07:21:11.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.437+0000 7fb612281640 1 -- 172.21.15.1:0/807412167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60c0ab370 con 0x7fb60c06c100 2024-03-06T07:21:11.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.437+0000 7fb608ff9640 1 -- 172.21.15.1:0/807412167 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc004a20 con 0x7fb60c06c100 2024-03-06T07:21:11.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.437+0000 7fb612281640 1 -- 172.21.15.1:0/807412167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60c0ab930 con 0x7fb60c06c100 2024-03-06T07:21:11.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.438+0000 7fb612281640 1 -- 172.21.15.1:0/807412167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb60c0686e0 con 0x7fb60c06c100 2024-03-06T07:21:11.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.439+0000 7fb608ff9640 1 -- 172.21.15.1:0/807412167 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb5fc004bc0 con 0x7fb60c06c100 2024-03-06T07:21:11.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.440+0000 7fb608ff9640 1 --2- 172.21.15.1:0/807412167 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb5f806b4d0 0x7fb5f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.440+0000 7fb608ff9640 1 -- 172.21.15.1:0/807412167 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fb5fc0897f0 con 0x7fb60c06c100 2024-03-06T07:21:11.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.440+0000 7fb60b7fe640 1 --2- 172.21.15.1:0/807412167 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb5f806b4d0 0x7fb5f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.441+0000 7fb60b7fe640 1 --2- 172.21.15.1:0/807412167 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb5f806b4d0 0x7fb5f806d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb60c1e4e70 tx=0x7fb600008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:11.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.448+0000 7fb608ff9640 1 -- 172.21.15.1:0/807412167 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb5fc057c30 con 0x7fb60c06c100 2024-03-06T07:21:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.629+0000 7fb612281640 1 -- 172.21.15.1:0/807412167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb60c0672b0 con 0x7fb60c06c100 2024-03-06T07:21:11.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.630+0000 7fb608ff9640 1 -- 172.21.15.1:0/807412167 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18728 (secure 0 0 0) 0x7fb5fc05bcd0 con 0x7fb60c06c100 2024-03-06T07:21:11.633 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:11.633 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":27,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:21:10.090355+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:19.337560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:20.172791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:20:23.884071+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:11.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.632+0000 7fb612281640 1 -- 172.21.15.1:0/807412167 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb5f806b4d0 msgr2=0x7fb5f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.632+0000 7fb612281640 1 --2- 172.21.15.1:0/807412167 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb5f806b4d0 0x7fb5f806d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb60c1e4e70 tx=0x7fb600008040 comp rx=0 tx=0).stop 2024-03-06T07:21:11.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.632+0000 7fb612281640 1 -- 172.21.15.1:0/807412167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb60c06c100 msgr2=0x7fb60c0c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.632+0000 7fb612281640 1 --2- 172.21.15.1:0/807412167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb60c06c100 0x7fb60c0c57d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00c4d0 tx=0x7fb5fc00c990 comp rx=0 tx=0).stop 2024-03-06T07:21:11.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.632+0000 7fb612281640 1 -- 172.21.15.1:0/807412167 shutdown_connections 2024-03-06T07:21:11.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.632+0000 7fb612281640 1 --2- 172.21.15.1:0/807412167 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb5f806b4d0 0x7fb5f806d990 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.632+0000 7fb612281640 1 --2- 172.21.15.1:0/807412167 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb60c14f660 0x7fb60c1eac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.632+0000 7fb612281640 1 --2- 172.21.15.1:0/807412167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb60c072c20 0x7fb60c0c05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.632+0000 7fb612281640 1 --2- 172.21.15.1:0/807412167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb60c06c100 0x7fb60c0c57d0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.632+0000 7fb612281640 1 -- 172.21.15.1:0/807412167 >> 172.21.15.1:0/807412167 conn(0x7fb60c071cf0 msgr2=0x7fb60c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:11.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.632+0000 7fb612281640 1 -- 172.21.15.1:0/807412167 shutdown_connections 2024-03-06T07:21:11.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.633+0000 7fb612281640 1 -- 172.21.15.1:0/807412167 wait complete. 2024-03-06T07:21:11.646 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:21:11.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.747+0000 7ff76e071640 1 Processor -- start 2024-03-06T07:21:11.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.747+0000 7ff76e071640 1 -- start start 2024-03-06T07:21:11.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.748+0000 7ff76e071640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff7680c3970 0x7ff7680c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.748+0000 7ff76e071640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff7680c60b0 0x7ff768152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.748+0000 7ff76e071640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff768152670 0x7ff7681553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.748+0000 7ff76e071640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7680c67f0 con 0x7ff7680c3970 2024-03-06T07:21:11.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.748+0000 7ff76e071640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7680c5e40 con 0x7ff7680c60b0 2024-03-06T07:21:11.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.748+0000 7ff76e071640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7680c6a40 con 0x7ff768152670 2024-03-06T07:21:11.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.748+0000 7ff7677fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff7680c3970 0x7ff7680c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.748+0000 7ff7677fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff7680c3970 0x7ff7680c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52580/0 (socket says 172.21.15.1:52580) 2024-03-06T07:21:11.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.748+0000 7ff7677fe640 1 -- 172.21.15.1:0/3623642637 learned_addr learned my addr 172.21.15.1:0/3623642637 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:11.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.748+0000 7ff767fff640 1 --2- 172.21.15.1:0/3623642637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff768152670 0x7ff7681553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.748+0000 7ff7677fe640 1 -- 172.21.15.1:0/3623642637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff768152670 msgr2=0x7ff7681553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:11.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.748+0000 7ff766ffd640 1 --2- 172.21.15.1:0/3623642637 >> v2:172.21.15.98:3300/0 conn(0x7ff7680c60b0 0x7ff768152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.748+0000 7ff7677fe640 1 --2- 172.21.15.1:0/3623642637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff768152670 0x7ff7681553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.748+0000 7ff7677fe640 1 -- 172.21.15.1:0/3623642637 >> v2:172.21.15.98:3300/0 conn(0x7ff7680c60b0 msgr2=0x7ff768152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:11.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.748+0000 7ff7677fe640 1 --2- 172.21.15.1:0/3623642637 >> v2:172.21.15.98:3300/0 conn(0x7ff7680c60b0 0x7ff768152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.748+0000 7ff7677fe640 1 -- 172.21.15.1:0/3623642637 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7680c6d20 con 0x7ff7680c3970 2024-03-06T07:21:11.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.749+0000 7ff7677fe640 1 --2- 172.21.15.1:0/3623642637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7680c3970 0x7ff7680c8170 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff7680719c0 tx=0x7ff75802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f15e0de0e66f87c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:11.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.749+0000 7ff7667fc640 1 -- 172.21.15.1:0/3623642637 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff758035020 con 0x7ff7680c3970 2024-03-06T07:21:11.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.749+0000 7ff7667fc640 1 -- 172.21.15.1:0/3623642637 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff758031870 con 0x7ff7680c3970 2024-03-06T07:21:11.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.749+0000 7ff7667fc640 1 -- 172.21.15.1:0/3623642637 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff758031c40 con 0x7ff7680c3970 2024-03-06T07:21:11.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.749+0000 7ff76e071640 1 -- 172.21.15.1:0/3623642637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7680c3970 msgr2=0x7ff7680c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.749+0000 7ff76e071640 1 --2- 172.21.15.1:0/3623642637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7680c3970 0x7ff7680c8170 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff7680719c0 tx=0x7ff75802b910 comp rx=0 tx=0).stop 2024-03-06T07:21:11.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.749+0000 7ff76e071640 1 -- 172.21.15.1:0/3623642637 shutdown_connections 2024-03-06T07:21:11.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.749+0000 7ff76e071640 1 --2- 172.21.15.1:0/3623642637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff768152670 0x7ff7681553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.749+0000 7ff76e071640 1 --2- 172.21.15.1:0/3623642637 >> v2:172.21.15.98:3300/0 conn(0x7ff7680c60b0 0x7ff768152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.749+0000 7ff76e071640 1 --2- 172.21.15.1:0/3623642637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7680c3970 0x7ff7680c8170 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.749+0000 7ff76e071640 1 -- 172.21.15.1:0/3623642637 >> 172.21.15.1:0/3623642637 conn(0x7ff7680c88f0 msgr2=0x7ff7680c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:11.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.750+0000 7ff76e071640 1 -- 172.21.15.1:0/3623642637 shutdown_connections 2024-03-06T07:21:11.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.750+0000 7ff76e071640 1 -- 172.21.15.1:0/3623642637 wait complete. 2024-03-06T07:21:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.751+0000 7ff76e071640 1 Processor -- start 2024-03-06T07:21:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.751+0000 7ff76e071640 1 -- start start 2024-03-06T07:21:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.751+0000 7ff76e071640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7680c3970 0x7ff7680c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.751+0000 7ff76e071640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7680c60b0 0x7ff7681eabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.751+0000 7ff76e071640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff768152670 0x7ff7681eb100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.751+0000 7ff76e071640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7680c75c0 con 0x7ff7680c3970 2024-03-06T07:21:11.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.751+0000 7ff76e071640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7680c72e0 con 0x7ff7680c60b0 2024-03-06T07:21:11.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.751+0000 7ff76e071640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7680c3d50 con 0x7ff768152670 2024-03-06T07:21:11.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.752+0000 7ff7677fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7680c3970 0x7ff7680c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.752+0000 7ff7677fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7680c3970 0x7ff7680c1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52582/0 (socket says 172.21.15.1:52582) 2024-03-06T07:21:11.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.752+0000 7ff7677fe640 1 -- 172.21.15.1:0/1967019862 learned_addr learned my addr 172.21.15.1:0/1967019862 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:11.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.752+0000 7ff767fff640 1 --2- 172.21.15.1:0/1967019862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff768152670 0x7ff7681eb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.752+0000 7ff766ffd640 1 --2- 172.21.15.1:0/1967019862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7680c60b0 0x7ff7681eabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.752+0000 7ff7677fe640 1 -- 172.21.15.1:0/1967019862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff768152670 msgr2=0x7ff7681eb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.752+0000 7ff7677fe640 1 --2- 172.21.15.1:0/1967019862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff768152670 0x7ff7681eb100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.752+0000 7ff7677fe640 1 -- 172.21.15.1:0/1967019862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7680c60b0 msgr2=0x7ff7681eabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.752+0000 7ff7677fe640 1 --2- 172.21.15.1:0/1967019862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7680c60b0 0x7ff7681eabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.752+0000 7ff7677fe640 1 -- 172.21.15.1:0/1967019862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7681eb640 con 0x7ff7680c3970 2024-03-06T07:21:11.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.752+0000 7ff767fff640 1 --2- 172.21.15.1:0/1967019862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff768152670 0x7ff7681eb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:11.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.752+0000 7ff7677fe640 1 --2- 172.21.15.1:0/1967019862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7680c3970 0x7ff7680c1f90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff75802b950 tx=0x7ff7580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:11.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.752+0000 7ff74ffff640 1 -- 172.21.15.1:0/1967019862 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff758040070 con 0x7ff7680c3970 2024-03-06T07:21:11.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.752+0000 7ff74ffff640 1 -- 172.21.15.1:0/1967019862 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75803a420 con 0x7ff7680c3970 2024-03-06T07:21:11.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.752+0000 7ff76e071640 1 -- 172.21.15.1:0/1967019862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7681582f0 con 0x7ff7680c3970 2024-03-06T07:21:11.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.752+0000 7ff74ffff640 1 -- 172.21.15.1:0/1967019862 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff758035040 con 0x7ff7680c3970 2024-03-06T07:21:11.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.752+0000 7ff76e071640 1 -- 172.21.15.1:0/1967019862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7681ee190 con 0x7ff7680c3970 2024-03-06T07:21:11.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.753+0000 7ff76e071640 1 -- 172.21.15.1:0/1967019862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7680686e0 con 0x7ff7680c3970 2024-03-06T07:21:11.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.754+0000 7ff74ffff640 1 -- 172.21.15.1:0/1967019862 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff758038020 con 0x7ff7680c3970 2024-03-06T07:21:11.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.754+0000 7ff74ffff640 1 --2- 172.21.15.1:0/1967019862 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff75406b4d0 0x7ff75406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.754+0000 7ff74ffff640 1 -- 172.21.15.1:0/1967019862 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7ff7580ac5a0 con 0x7ff7680c3970 2024-03-06T07:21:11.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.754+0000 7ff766ffd640 1 --2- 172.21.15.1:0/1967019862 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff75406b4d0 0x7ff75406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.755+0000 7ff766ffd640 1 --2- 172.21.15.1:0/1967019862 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff75406b4d0 0x7ff75406d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff768068ad0 tx=0x7ff75c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:11.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.758+0000 7ff74ffff640 1 -- 172.21.15.1:0/1967019862 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff75807b020 con 0x7ff7680c3970 2024-03-06T07:21:11.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.978+0000 7ff76e071640 1 -- 172.21.15.1:0/1967019862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff768074b30 con 0x7ff7680c3970 2024-03-06T07:21:11.980 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:21:11.979+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:21:11.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.980+0000 7ff74ffff640 1 -- 172.21.15.1:0/1967019862 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff75807a920 con 0x7ff7680c3970 2024-03-06T07:21:11.982 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:21:11.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.983+0000 7ff76e071640 1 -- 172.21.15.1:0/1967019862 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff75406b4d0 msgr2=0x7ff75406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.983+0000 7ff76e071640 1 --2- 172.21.15.1:0/1967019862 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff75406b4d0 0x7ff75406d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff768068ad0 tx=0x7ff75c008040 comp rx=0 tx=0).stop 2024-03-06T07:21:11.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.983+0000 7ff76e071640 1 -- 172.21.15.1:0/1967019862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7680c3970 msgr2=0x7ff7680c1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.983+0000 7ff76e071640 1 --2- 172.21.15.1:0/1967019862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7680c3970 0x7ff7680c1f90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff75802b950 tx=0x7ff7580046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.983+0000 7ff76e071640 1 -- 172.21.15.1:0/1967019862 shutdown_connections 2024-03-06T07:21:11.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.983+0000 7ff76e071640 1 --2- 172.21.15.1:0/1967019862 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff75406b4d0 0x7ff75406d990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.983+0000 7ff76e071640 1 --2- 172.21.15.1:0/1967019862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff768152670 0x7ff7681eb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.983+0000 7ff76e071640 1 --2- 172.21.15.1:0/1967019862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7680c60b0 0x7ff7681eabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.983+0000 7ff76e071640 1 --2- 172.21.15.1:0/1967019862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7680c3970 0x7ff7680c1f90 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.983+0000 7ff76e071640 1 -- 172.21.15.1:0/1967019862 >> 172.21.15.1:0/1967019862 conn(0x7ff7680c88f0 msgr2=0x7ff768158a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:11.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.983+0000 7ff76e071640 1 -- 172.21.15.1:0/1967019862 shutdown_connections 2024-03-06T07:21:11.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:11.983+0000 7ff76e071640 1 -- 172.21.15.1:0/1967019862 wait complete. 2024-03-06T07:21:11.986 INFO:tasks.ceph.mds.a.smithi001.stderr: -3480> 2024-03-06T07:20:27.143+0000 7f11b8bbd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:11.996 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:21:12.015 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:21:12.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.098+0000 7f654cd0c640 1 Processor -- start 2024-03-06T07:21:12.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.098+0000 7f654cd0c640 1 -- start start 2024-03-06T07:21:12.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.098+0000 7f654cd0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480c6b10 0x7f6548155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:12.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.099+0000 7f654cd0c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65480c2fc0 con 0x7f6548152650 2024-03-06T07:21:12.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.099+0000 7f654cd0c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6548158da0 con 0x7f65480bf9d0 2024-03-06T07:21:12.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.099+0000 7f654cd0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6548155800 con 0x7f65480c6b10 2024-03-06T07:21:12.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.099+0000 7f654659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6548152650 0x7f6548158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59320/0 (socket says 172.21.15.1:59320) 2024-03-06T07:21:12.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.099+0000 7f654659c640 1 -- 172.21.15.1:0/4172749532 learned_addr learned my addr 172.21.15.1:0/4172749532 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:12.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.099+0000 7f653dd9b640 1 --2- 172.21.15.1:0/4172749532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480c6b10 0x7f6548155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:12.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.099+0000 7f6545d9b640 1 -- 172.21.15.1:0/4172749532 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023147972 0 0) 0x7f65480c2fc0 con 0x7f6548152650 2024-03-06T07:21:12.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.099+0000 7f6545d9b640 1 -- 172.21.15.1:0/4172749532 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6534003520 con 0x7f6548152650 2024-03-06T07:21:12.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.099+0000 7f6545d9b640 1 -- 172.21.15.1:0/4172749532 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115230672 0 0) 0x7f6548158da0 con 0x7f65480bf9d0 2024-03-06T07:21:12.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.099+0000 7f653dd9b640 1 -- 172.21.15.1:0/4172749532 >> v1:172.21.15.98:6789/0 conn(0x7f65480bf9d0 legacy=0x7f65480c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:12.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.099+0000 7f653dd9b640 1 -- 172.21.15.1:0/4172749532 >> v1:172.21.15.1:6789/0 conn(0x7f6548152650 legacy=0x7f6548158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:12.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.099+0000 7f653dd9b640 1 -- 172.21.15.1:0/4172749532 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65480c6690 con 0x7f65480c6b10 2024-03-06T07:21:12.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.100+0000 7f653dd9b640 1 --2- 172.21.15.1:0/4172749532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480c6b10 0x7f6548155c90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f65300097d0 tx=0x7f653002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcc671ca9d9b9b1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:12.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.100+0000 7f6545d9b640 1 -- 172.21.15.1:0/4172749532 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530035020 con 0x7f65480c6b10 2024-03-06T07:21:12.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.100+0000 7f6545d9b640 1 -- 172.21.15.1:0/4172749532 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6530031800 con 0x7f65480c6b10 2024-03-06T07:21:12.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.100+0000 7f6545d9b640 1 -- 172.21.15.1:0/4172749532 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530031bd0 con 0x7f65480c6b10 2024-03-06T07:21:12.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.101+0000 7f654cd0c640 1 -- 172.21.15.1:0/4172749532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480c6b10 msgr2=0x7f6548155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:12.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.101+0000 7f654cd0c640 1 --2- 172.21.15.1:0/4172749532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480c6b10 0x7f6548155c90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f65300097d0 tx=0x7f653002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:21:12.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.101+0000 7f654cd0c640 1 -- 172.21.15.1:0/4172749532 shutdown_connections 2024-03-06T07:21:12.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.101+0000 7f654cd0c640 1 --2- 172.21.15.1:0/4172749532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480c6b10 0x7f6548155c90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:12.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.101+0000 7f654cd0c640 1 -- 172.21.15.1:0/4172749532 >> 172.21.15.1:0/4172749532 conn(0x7f6548066ee0 msgr2=0x7f65480672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:12.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.101+0000 7f654cd0c640 1 -- 172.21.15.1:0/4172749532 shutdown_connections 2024-03-06T07:21:12.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.101+0000 7f654cd0c640 1 -- 172.21.15.1:0/4172749532 wait complete. 2024-03-06T07:21:12.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.102+0000 7f654cd0c640 1 Processor -- start 2024-03-06T07:21:12.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.102+0000 7f654cd0c640 1 -- start start 2024-03-06T07:21:12.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.103+0000 7f654cd0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480bf9d0 0x7f654814e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:12.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.103+0000 7f654cd0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6548152650 0x7f6548154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:12.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.103+0000 7f654cd0c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6548158880 0x7f65481edcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:12.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.103+0000 7f654cd0c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65481551b0 con 0x7f6548158880 2024-03-06T07:21:12.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.103+0000 7f654cd0c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65480c6fa0 con 0x7f6548152650 2024-03-06T07:21:12.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.103+0000 7f654cd0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6548155490 con 0x7f65480bf9d0 2024-03-06T07:21:12.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.103+0000 7f6546d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6548158880 0x7f65481edcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:12.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.103+0000 7f654659c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480bf9d0 0x7f654814e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:12.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.103+0000 7f6546d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6548158880 0x7f65481edcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52598/0 (socket says 172.21.15.1:52598) 2024-03-06T07:21:12.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.103+0000 7f6546d9d640 1 -- 172.21.15.1:0/2645601681 learned_addr learned my addr 172.21.15.1:0/2645601681 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:12.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.103+0000 7f653dd9b640 1 --2- 172.21.15.1:0/2645601681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6548152650 0x7f6548154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:12.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.103+0000 7f6546d9d640 1 -- 172.21.15.1:0/2645601681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480bf9d0 msgr2=0x7f654814e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:12.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.103+0000 7f6546d9d640 1 --2- 172.21.15.1:0/2645601681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480bf9d0 0x7f654814e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:12.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.103+0000 7f6546d9d640 1 -- 172.21.15.1:0/2645601681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6548152650 msgr2=0x7f6548154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:12.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.103+0000 7f6546d9d640 1 --2- 172.21.15.1:0/2645601681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6548152650 0x7f6548154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:12.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.103+0000 7f6546d9d640 1 -- 172.21.15.1:0/2645601681 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65480afbc0 con 0x7f6548158880 2024-03-06T07:21:12.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.103+0000 7f654659c640 1 --2- 172.21.15.1:0/2645601681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480bf9d0 0x7f654814e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:12.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.104+0000 7f6546d9d640 1 --2- 172.21.15.1:0/2645601681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6548158880 0x7f65481edcf0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f654000c780 tx=0x7f654000cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:12.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.104+0000 7f653f7fe640 1 -- 172.21.15.1:0/2645601681 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540003b70 con 0x7f6548158880 2024-03-06T07:21:12.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.104+0000 7f653f7fe640 1 -- 172.21.15.1:0/2645601681 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6540003d10 con 0x7f6548158880 2024-03-06T07:21:12.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.104+0000 7f654cd0c640 1 -- 172.21.15.1:0/2645601681 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65481e1f60 con 0x7f6548158880 2024-03-06T07:21:12.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.104+0000 7f653f7fe640 1 -- 172.21.15.1:0/2645601681 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540005270 con 0x7f6548158880 2024-03-06T07:21:12.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.104+0000 7f654cd0c640 1 -- 172.21.15.1:0/2645601681 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65480b0140 con 0x7f6548158880 2024-03-06T07:21:12.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.105+0000 7f654cd0c640 1 -- 172.21.15.1:0/2645601681 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65481e2400 con 0x7f6548158880 2024-03-06T07:21:12.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.105+0000 7f653f7fe640 1 -- 172.21.15.1:0/2645601681 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6540005410 con 0x7f6548158880 2024-03-06T07:21:12.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.106+0000 7f653f7fe640 1 --2- 172.21.15.1:0/2645601681 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f652806b500 0x7f652806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:12.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.106+0000 7f653f7fe640 1 -- 172.21.15.1:0/2645601681 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f654008a3b0 con 0x7f6548158880 2024-03-06T07:21:12.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.106+0000 7f654659c640 1 --2- 172.21.15.1:0/2645601681 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f652806b500 0x7f652806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:12.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.106+0000 7f654659c640 1 --2- 172.21.15.1:0/2645601681 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f652806b500 0x7f652806d9c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6538005880 tx=0x7f6538009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:12.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.110+0000 7f653f7fe640 1 -- 172.21.15.1:0/2645601681 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65481e2400 con 0x7f6548158880 2024-03-06T07:21:12.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.328+0000 7f654cd0c640 1 -- 172.21.15.1:0/2645601681 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f65480b06f0 con 0x7f6548158880 2024-03-06T07:21:12.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.982+0000 7f653f7fe640 1 -- 172.21.15.1:0/2645601681 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6540058770 con 0x7f6548158880 2024-03-06T07:21:12.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.984+0000 7f654cd0c640 1 -- 172.21.15.1:0/2645601681 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f652806b500 msgr2=0x7f652806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:12.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.984+0000 7f654cd0c640 1 --2- 172.21.15.1:0/2645601681 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f652806b500 0x7f652806d9c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6538005880 tx=0x7f6538009040 comp rx=0 tx=0).stop 2024-03-06T07:21:12.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.984+0000 7f654cd0c640 1 -- 172.21.15.1:0/2645601681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6548158880 msgr2=0x7f65481edcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:12.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.984+0000 7f654cd0c640 1 --2- 172.21.15.1:0/2645601681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6548158880 0x7f65481edcf0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f654000c780 tx=0x7f654000cb80 comp rx=0 tx=0).stop 2024-03-06T07:21:12.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.984+0000 7f654cd0c640 1 -- 172.21.15.1:0/2645601681 shutdown_connections 2024-03-06T07:21:12.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.984+0000 7f654cd0c640 1 --2- 172.21.15.1:0/2645601681 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f652806b500 0x7f652806d9c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:12.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.984+0000 7f654cd0c640 1 --2- 172.21.15.1:0/2645601681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6548158880 0x7f65481edcf0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:12.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.984+0000 7f654cd0c640 1 --2- 172.21.15.1:0/2645601681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6548152650 0x7f6548154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:12.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.984+0000 7f654cd0c640 1 --2- 172.21.15.1:0/2645601681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65480bf9d0 0x7f654814e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:12.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.984+0000 7f654cd0c640 1 -- 172.21.15.1:0/2645601681 >> 172.21.15.1:0/2645601681 conn(0x7f6548066ee0 msgr2=0x7f65480c3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:12.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.984+0000 7f654cd0c640 1 -- 172.21.15.1:0/2645601681 shutdown_connections 2024-03-06T07:21:12.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:12.984+0000 7f654cd0c640 1 -- 172.21.15.1:0/2645601681 wait complete. 2024-03-06T07:21:13.000 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:21:13.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.151+0000 7fccbfaaf640 1 Processor -- start 2024-03-06T07:21:13.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.152+0000 7fccbfaaf640 1 -- start start 2024-03-06T07:21:13.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.152+0000 7fccbfaaf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fccb80c16e0 0x7fccb8152470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:13.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.152+0000 7fccbfaaf640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fccb81529b0 0x7fccb8155c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:13.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.152+0000 7fccbfaaf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb8158890 con 0x7fccb80c70c0 2024-03-06T07:21:13.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.152+0000 7fccbfaaf640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb8158b70 con 0x7fccb81529b0 2024-03-06T07:21:13.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.152+0000 7fccbfaaf640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb8158e50 con 0x7fccb80c16e0 2024-03-06T07:21:13.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.152+0000 7fccbe04c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fccb81529b0 0x7fccb8155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:13.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.153+0000 7fccbe04c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fccb81529b0 0x7fccb8155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52600/0 (socket says 172.21.15.1:52600) 2024-03-06T07:21:13.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.153+0000 7fccbe04c640 1 -- 172.21.15.1:0/1876242151 learned_addr learned my addr 172.21.15.1:0/1876242151 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:13.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.153+0000 7fccbd84b640 1 --2- 172.21.15.1:0/1876242151 >> v2:172.21.15.98:3300/0 conn(0x7fccb80c16e0 0x7fccb8152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:13.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.153+0000 7fccbe04c640 1 -- 172.21.15.1:0/1876242151 >> v2:172.21.15.98:3300/0 conn(0x7fccb80c16e0 msgr2=0x7fccb8152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:13.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.153+0000 7fccbe04c640 1 --2- 172.21.15.1:0/1876242151 >> v2:172.21.15.98:3300/0 conn(0x7fccb80c16e0 0x7fccb8152470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:13.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.153+0000 7fccbe04c640 1 -- 172.21.15.1:0/1876242151 >> v1:172.21.15.98:6789/0 conn(0x7fccb80c70c0 legacy=0x7fccb8155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:13.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.153+0000 7fccbe04c640 1 -- 172.21.15.1:0/1876242151 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb8159130 con 0x7fccb81529b0 2024-03-06T07:21:13.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.153+0000 7fccbe04c640 1 --2- 172.21.15.1:0/1876242151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb81529b0 0x7fccb8155c60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fccb808bbd0 tx=0x7fccb002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e6215cb1982e726 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:13.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.154+0000 7fccbc849640 1 -- 172.21.15.1:0/1876242151 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb002c9f0 con 0x7fccb81529b0 2024-03-06T07:21:13.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.154+0000 7fccbc849640 1 -- 172.21.15.1:0/1876242151 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb002cb90 con 0x7fccb81529b0 2024-03-06T07:21:13.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.154+0000 7fccbc849640 1 -- 172.21.15.1:0/1876242151 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb00343f0 con 0x7fccb81529b0 2024-03-06T07:21:13.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.154+0000 7fccbfaaf640 1 -- 172.21.15.1:0/1876242151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb81529b0 msgr2=0x7fccb8155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:13.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.154+0000 7fccbfaaf640 1 --2- 172.21.15.1:0/1876242151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb81529b0 0x7fccb8155c60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fccb808bbd0 tx=0x7fccb002be50 comp rx=0 tx=0).stop 2024-03-06T07:21:13.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.154+0000 7fccbfaaf640 1 -- 172.21.15.1:0/1876242151 shutdown_connections 2024-03-06T07:21:13.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.154+0000 7fccbfaaf640 1 --2- 172.21.15.1:0/1876242151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb81529b0 0x7fccb8155c60 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:13.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.154+0000 7fccbfaaf640 1 --2- 172.21.15.1:0/1876242151 >> v2:172.21.15.98:3300/0 conn(0x7fccb80c16e0 0x7fccb8152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:13.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.154+0000 7fccbfaaf640 1 -- 172.21.15.1:0/1876242151 >> 172.21.15.1:0/1876242151 conn(0x7fccb8066d00 msgr2=0x7fccb8067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:13.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.154+0000 7fccbfaaf640 1 -- 172.21.15.1:0/1876242151 shutdown_connections 2024-03-06T07:21:13.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.154+0000 7fccbfaaf640 1 -- 172.21.15.1:0/1876242151 wait complete. 2024-03-06T07:21:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.155+0000 7fccbfaaf640 1 Processor -- start 2024-03-06T07:21:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.155+0000 7fccbfaaf640 1 -- start start 2024-03-06T07:21:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.156+0000 7fccbfaaf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccb80c16e0 0x7fccb814cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.156+0000 7fccbfaaf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb80c70c0 0x7fccb8157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.156+0000 7fccbfaaf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccb81529b0 0x7fccb8147f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.156+0000 7fccbfaaf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb81561a0 con 0x7fccb80c70c0 2024-03-06T07:21:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.156+0000 7fccbfaaf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb81596f0 con 0x7fccb80c16e0 2024-03-06T07:21:13.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.156+0000 7fccbfaaf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb80c68c0 con 0x7fccb81529b0 2024-03-06T07:21:13.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.156+0000 7fccbd04a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb80c70c0 0x7fccb8157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:13.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.156+0000 7fccbd04a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb80c70c0 0x7fccb8157950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52616/0 (socket says 172.21.15.1:52616) 2024-03-06T07:21:13.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.156+0000 7fccbd04a640 1 -- 172.21.15.1:0/1049314746 learned_addr learned my addr 172.21.15.1:0/1049314746 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:13.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.156+0000 7fccbe04c640 1 --2- 172.21.15.1:0/1049314746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccb81529b0 0x7fccb8147f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:13.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.156+0000 7fccbd84b640 1 --2- 172.21.15.1:0/1049314746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccb80c16e0 0x7fccb814cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:13.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.157+0000 7fccbd04a640 1 -- 172.21.15.1:0/1049314746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccb81529b0 msgr2=0x7fccb8147f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:13.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.157+0000 7fccbd04a640 1 --2- 172.21.15.1:0/1049314746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccb81529b0 0x7fccb8147f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:13.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.157+0000 7fccbd04a640 1 -- 172.21.15.1:0/1049314746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccb80c16e0 msgr2=0x7fccb814cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:13.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.157+0000 7fccbd04a640 1 --2- 172.21.15.1:0/1049314746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccb80c16e0 0x7fccb814cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:13.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.157+0000 7fccbd04a640 1 -- 172.21.15.1:0/1049314746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb8149a70 con 0x7fccb80c70c0 2024-03-06T07:21:13.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.157+0000 7fccbe04c640 1 --2- 172.21.15.1:0/1049314746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccb81529b0 0x7fccb8147f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:13.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.157+0000 7fccbd84b640 1 --2- 172.21.15.1:0/1049314746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccb80c16e0 0x7fccb814cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:13.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.157+0000 7fccbd04a640 1 --2- 172.21.15.1:0/1049314746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb80c70c0 0x7fccb8157950 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcca800c440 tx=0x7fcca800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:13.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.157+0000 7fccae7fc640 1 -- 172.21.15.1:0/1049314746 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8017070 con 0x7fccb80c70c0 2024-03-06T07:21:13.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.157+0000 7fccae7fc640 1 -- 172.21.15.1:0/1049314746 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca80035f0 con 0x7fccb80c70c0 2024-03-06T07:21:13.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.157+0000 7fccbfaaf640 1 -- 172.21.15.1:0/1049314746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb8149d50 con 0x7fccb80c70c0 2024-03-06T07:21:13.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.157+0000 7fccae7fc640 1 -- 172.21.15.1:0/1049314746 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca80039e0 con 0x7fccb80c70c0 2024-03-06T07:21:13.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.157+0000 7fccbfaaf640 1 -- 172.21.15.1:0/1049314746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb814a310 con 0x7fccb80c70c0 2024-03-06T07:21:13.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.158+0000 7fccbfaaf640 1 -- 172.21.15.1:0/1049314746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb80680b0 con 0x7fccb80c70c0 2024-03-06T07:21:13.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.158+0000 7fccae7fc640 1 -- 172.21.15.1:0/1049314746 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcca801a070 con 0x7fccb80c70c0 2024-03-06T07:21:13.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.159+0000 7fccae7fc640 1 --2- 172.21.15.1:0/1049314746 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc9c06b4d0 0x7fcc9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:13.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.159+0000 7fccae7fc640 1 -- 172.21.15.1:0/1049314746 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fcca8089930 con 0x7fccb80c70c0 2024-03-06T07:21:13.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.159+0000 7fccbd84b640 1 --2- 172.21.15.1:0/1049314746 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc9c06b4d0 0x7fcc9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:13.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.160+0000 7fccbd84b640 1 --2- 172.21.15.1:0/1049314746 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc9c06b4d0 0x7fcc9c06d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fccb4008890 tx=0x7fccb4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:13.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.163+0000 7fccae7fc640 1 -- 172.21.15.1:0/1049314746 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcca80583e0 con 0x7fccb80c70c0 2024-03-06T07:21:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:13.344+0000 7fccbfaaf640 1 -- 172.21.15.1:0/1049314746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fccb80b0880 con 0x7fccb80c70c0 2024-03-06T07:21:14.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.115+0000 7fccae7fc640 1 -- 172.21.15.1:0/1049314746 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) v1 ==== 181+0+0 (secure 0 0 0) 0x7fcca8087e20 con 0x7fccb80c70c0 2024-03-06T07:21:14.117 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:21:14.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.119+0000 7fccbfaaf640 1 -- 172.21.15.1:0/1049314746 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc9c06b4d0 msgr2=0x7fcc9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:14.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.119+0000 7fccbfaaf640 1 --2- 172.21.15.1:0/1049314746 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc9c06b4d0 0x7fcc9c06d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fccb4008890 tx=0x7fccb4008040 comp rx=0 tx=0).stop 2024-03-06T07:21:14.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.119+0000 7fccbfaaf640 1 -- 172.21.15.1:0/1049314746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb80c70c0 msgr2=0x7fccb8157950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:14.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.119+0000 7fccbfaaf640 1 --2- 172.21.15.1:0/1049314746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb80c70c0 0x7fccb8157950 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcca800c440 tx=0x7fcca800c900 comp rx=0 tx=0).stop 2024-03-06T07:21:14.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.119+0000 7fccbfaaf640 1 -- 172.21.15.1:0/1049314746 shutdown_connections 2024-03-06T07:21:14.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.119+0000 7fccbfaaf640 1 --2- 172.21.15.1:0/1049314746 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc9c06b4d0 0x7fcc9c06d990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:14.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.119+0000 7fccbfaaf640 1 --2- 172.21.15.1:0/1049314746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccb81529b0 0x7fccb8147f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:14.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.119+0000 7fccbfaaf640 1 --2- 172.21.15.1:0/1049314746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb80c70c0 0x7fccb8157950 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:14.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.119+0000 7fccbfaaf640 1 --2- 172.21.15.1:0/1049314746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccb80c16e0 0x7fccb814cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:14.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.119+0000 7fccbfaaf640 1 -- 172.21.15.1:0/1049314746 >> 172.21.15.1:0/1049314746 conn(0x7fccb8066d00 msgr2=0x7fccb80c4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:14.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.123+0000 7fccbfaaf640 1 -- 172.21.15.1:0/1049314746 shutdown_connections 2024-03-06T07:21:14.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.123+0000 7fccbfaaf640 1 -- 172.21.15.1:0/1049314746 wait complete. 2024-03-06T07:21:14.142 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:21:14.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.251+0000 7ff3fe35e640 1 Processor -- start 2024-03-06T07:21:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.251+0000 7ff3fe35e640 1 -- start start 2024-03-06T07:21:14.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.252+0000 7ff3fe35e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff3f8155e50 0x7ff3f80cab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:14.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.252+0000 7ff3fe35e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f80c6710 con 0x7ff3f8152780 2024-03-06T07:21:14.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.252+0000 7ff3fe35e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f80c69b0 con 0x7ff3f80c1010 2024-03-06T07:21:14.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.252+0000 7ff3fe35e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f80c6c90 con 0x7ff3f8155e50 2024-03-06T07:21:14.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.252+0000 7ff3f7fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff3f8152780 0x7ff3f80c6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59326/0 (socket says 172.21.15.1:59326) 2024-03-06T07:21:14.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.252+0000 7ff3f7fff640 1 -- 172.21.15.1:0/3818626690 learned_addr learned my addr 172.21.15.1:0/3818626690 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:14.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.252+0000 7ff3f6ffd640 1 -- 172.21.15.1:0/3818626690 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836243221 0 0) 0x7ff3f80c6710 con 0x7ff3f8152780 2024-03-06T07:21:14.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.252+0000 7ff3fc8fb640 1 --2- 172.21.15.1:0/3818626690 >> v2:172.21.15.98:3300/0 conn(0x7ff3f8155e50 0x7ff3f80cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:14.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.252+0000 7ff3f6ffd640 1 -- 172.21.15.1:0/3818626690 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e4003520 con 0x7ff3f8152780 2024-03-06T07:21:14.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f6ffd640 1 -- 172.21.15.1:0/3818626690 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3163112030 0 0) 0x7ff3e4003520 con 0x7ff3f8152780 2024-03-06T07:21:14.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f6ffd640 1 -- 172.21.15.1:0/3818626690 >> v2:172.21.15.98:3300/0 conn(0x7ff3f8155e50 msgr2=0x7ff3f80cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:14.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f6ffd640 1 --2- 172.21.15.1:0/3818626690 >> v2:172.21.15.98:3300/0 conn(0x7ff3f8155e50 0x7ff3f80cab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:14.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f6ffd640 1 -- 172.21.15.1:0/3818626690 >> v1:172.21.15.98:6789/0 conn(0x7ff3f80c1010 legacy=0x7ff3f80c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:14.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f6ffd640 1 -- 172.21.15.1:0/3818626690 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f8155370 con 0x7ff3f8152780 2024-03-06T07:21:14.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f6ffd640 1 -- 172.21.15.1:0/3818626690 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7ff3e0001670 con 0x7ff3f8152780 2024-03-06T07:21:14.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f6ffd640 1 -- 172.21.15.1:0/3818626690 >> v1:172.21.15.1:6789/0 conn(0x7ff3f8152780 legacy=0x7ff3f80c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:14.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f6ffd640 1 --2- 172.21.15.1:0/3818626690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3e40039b0 0x7ff3e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:14.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f6ffd640 1 --2- 172.21.15.1:0/3818626690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3e4004800 0x7ff3e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:14.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f6ffd640 1 --2- 172.21.15.1:0/3818626690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3e40051c0 0x7ff3e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:14.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f6ffd640 1 -- 172.21.15.1:0/3818626690 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f80c6c90 con 0x7ff3e40039b0 2024-03-06T07:21:14.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f6ffd640 1 -- 172.21.15.1:0/3818626690 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e400c160 con 0x7ff3e4004800 2024-03-06T07:21:14.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f6ffd640 1 -- 172.21.15.1:0/3818626690 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e400c360 con 0x7ff3e40051c0 2024-03-06T07:21:14.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f7fff640 1 --2- 172.21.15.1:0/3818626690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3e40039b0 0x7ff3e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:14.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f77fe640 1 --2- 172.21.15.1:0/3818626690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3e4004800 0x7ff3e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:14.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3fc8fb640 1 --2- 172.21.15.1:0/3818626690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3e40051c0 0x7ff3e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:14.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f7fff640 1 -- 172.21.15.1:0/3818626690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3e40051c0 msgr2=0x7ff3e400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:14.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f7fff640 1 --2- 172.21.15.1:0/3818626690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3e40051c0 0x7ff3e400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:14.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f7fff640 1 -- 172.21.15.1:0/3818626690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3e4004800 msgr2=0x7ff3e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:14.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f7fff640 1 --2- 172.21.15.1:0/3818626690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3e4004800 0x7ff3e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:14.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3f7fff640 1 -- 172.21.15.1:0/3818626690 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e4003520 con 0x7ff3e40039b0 2024-03-06T07:21:14.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.253+0000 7ff3fc8fb640 1 -- 172.21.15.1:0/3818626690 reap_dead start 2024-03-06T07:21:14.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.254+0000 7ff3f7fff640 1 --2- 172.21.15.1:0/3818626690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3e40039b0 0x7ff3e4003dd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000c290 tx=0x7ff3e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=391af19c4b6daa9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:14.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.254+0000 7ff3f6ffd640 1 -- 172.21.15.1:0/3818626690 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0007d30 con 0x7ff3e40039b0 2024-03-06T07:21:14.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.254+0000 7ff3f6ffd640 1 -- 172.21.15.1:0/3818626690 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e0030070 con 0x7ff3e40039b0 2024-03-06T07:21:14.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.254+0000 7ff3f6ffd640 1 -- 172.21.15.1:0/3818626690 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0005d30 con 0x7ff3e40039b0 2024-03-06T07:21:14.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.254+0000 7ff3fe35e640 1 -- 172.21.15.1:0/3818626690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3e40039b0 msgr2=0x7ff3e4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:14.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.254+0000 7ff3fe35e640 1 --2- 172.21.15.1:0/3818626690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3e40039b0 0x7ff3e4003dd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000c290 tx=0x7ff3e4005830 comp rx=0 tx=0).stop 2024-03-06T07:21:14.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.254+0000 7ff3fe35e640 1 -- 172.21.15.1:0/3818626690 shutdown_connections 2024-03-06T07:21:14.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.254+0000 7ff3fe35e640 1 --2- 172.21.15.1:0/3818626690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3e40039b0 0x7ff3e4003dd0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:14.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.254+0000 7ff3fe35e640 1 -- 172.21.15.1:0/3818626690 >> 172.21.15.1:0/3818626690 conn(0x7ff3f808b720 msgr2=0x7ff3f806d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:14.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.255+0000 7ff3fe35e640 1 -- 172.21.15.1:0/3818626690 shutdown_connections 2024-03-06T07:21:14.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.255+0000 7ff3fe35e640 1 -- 172.21.15.1:0/3818626690 wait complete. 2024-03-06T07:21:14.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.256+0000 7ff3fe35e640 1 Processor -- start 2024-03-06T07:21:14.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.256+0000 7ff3fe35e640 1 -- start start 2024-03-06T07:21:14.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.256+0000 7ff3fe35e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3e40039b0 0x7ff3f81eedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:14.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.256+0000 7ff3fe35e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3f814ae00 0x7ff3f80c3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:14.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.256+0000 7ff3fe35e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3f81eae10 0x7ff3f80c8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:14.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.256+0000 7ff3fe35e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f8152b60 con 0x7ff3f81eae10 2024-03-06T07:21:14.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.256+0000 7ff3fe35e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f8155930 con 0x7ff3f814ae00 2024-03-06T07:21:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.256+0000 7ff3fe35e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f80c6000 con 0x7ff3e40039b0 2024-03-06T07:21:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.257+0000 7ff3fc8fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3f81eae10 0x7ff3f80c8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.257+0000 7ff3fc8fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3f81eae10 0x7ff3f80c8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52622/0 (socket says 172.21.15.1:52622) 2024-03-06T07:21:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.257+0000 7ff3fc8fb640 1 -- 172.21.15.1:0/747683711 learned_addr learned my addr 172.21.15.1:0/747683711 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.257+0000 7ff3fc8fb640 1 -- 172.21.15.1:0/747683711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3e40039b0 msgr2=0x7ff3f81eedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.257+0000 7ff3fc8fb640 1 --2- 172.21.15.1:0/747683711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3e40039b0 0x7ff3f81eedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.257+0000 7ff3f77fe640 1 --2- 172.21.15.1:0/747683711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3f814ae00 0x7ff3f80c3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.257+0000 7ff3fc8fb640 1 -- 172.21.15.1:0/747683711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3f814ae00 msgr2=0x7ff3f80c3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.257+0000 7ff3fc8fb640 1 --2- 172.21.15.1:0/747683711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3f814ae00 0x7ff3f80c3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.257+0000 7ff3fc8fb640 1 -- 172.21.15.1:0/747683711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f80c7720 con 0x7ff3f81eae10 2024-03-06T07:21:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.257+0000 7ff3fc8fb640 1 --2- 172.21.15.1:0/747683711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3f81eae10 0x7ff3f80c8970 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff3f80c2430 tx=0x7ff3f00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.257+0000 7ff3f4ff9640 1 -- 172.21.15.1:0/747683711 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0011db0 con 0x7ff3f81eae10 2024-03-06T07:21:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.257+0000 7ff3f4ff9640 1 -- 172.21.15.1:0/747683711 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f001b070 con 0x7ff3f81eae10 2024-03-06T07:21:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.257+0000 7ff3fe35e640 1 -- 172.21.15.1:0/747683711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f8157c70 con 0x7ff3f81eae10 2024-03-06T07:21:14.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.257+0000 7ff3f4ff9640 1 -- 172.21.15.1:0/747683711 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0003990 con 0x7ff3f81eae10 2024-03-06T07:21:14.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.257+0000 7ff3fe35e640 1 -- 172.21.15.1:0/747683711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f81ee5d0 con 0x7ff3f81eae10 2024-03-06T07:21:14.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.258+0000 7ff3fe35e640 1 -- 172.21.15.1:0/747683711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f8158190 con 0x7ff3f81eae10 2024-03-06T07:21:14.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.259+0000 7ff3f4ff9640 1 -- 172.21.15.1:0/747683711 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3f0003b30 con 0x7ff3f81eae10 2024-03-06T07:21:14.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.259+0000 7ff3f4ff9640 1 --2- 172.21.15.1:0/747683711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff3d806b500 0x7ff3d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:14.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.259+0000 7ff3f4ff9640 1 -- 172.21.15.1:0/747683711 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6383+0+0 (secure 0 0 0) 0x7ff3f008abc0 con 0x7ff3f81eae10 2024-03-06T07:21:14.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.260+0000 7ff3f7fff640 1 --2- 172.21.15.1:0/747683711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff3d806b500 0x7ff3d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:14.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.260+0000 7ff3f7fff640 1 --2- 172.21.15.1:0/747683711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff3d806b500 0x7ff3d806d9c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0034000 tx=0x7ff3e002e000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:14.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.263+0000 7ff3f4ff9640 1 -- 172.21.15.1:0/747683711 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3f0003e20 con 0x7ff3f81eae10 2024-03-06T07:21:14.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:14.445+0000 7ff3fe35e640 1 -- 172.21.15.1:0/747683711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff3f80c8740 con 0x7ff3f81eae10 2024-03-06T07:21:15.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.119+0000 7ff3f4ff9640 1 -- 172.21.15.1:0/747683711 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff3f80c8740 con 0x7ff3f81eae10 2024-03-06T07:21:15.122 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-06T07:21:15.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.123+0000 7ff3fe35e640 1 -- 172.21.15.1:0/747683711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff3d806b500 msgr2=0x7ff3d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:15.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.123+0000 7ff3fe35e640 1 --2- 172.21.15.1:0/747683711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff3d806b500 0x7ff3d806d9c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0034000 tx=0x7ff3e002e000 comp rx=0 tx=0).stop 2024-03-06T07:21:15.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.123+0000 7ff3fe35e640 1 -- 172.21.15.1:0/747683711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3f81eae10 msgr2=0x7ff3f80c8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:15.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.123+0000 7ff3fe35e640 1 --2- 172.21.15.1:0/747683711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3f81eae10 0x7ff3f80c8970 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff3f80c2430 tx=0x7ff3f00115a0 comp rx=0 tx=0).stop 2024-03-06T07:21:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.123+0000 7ff3fe35e640 1 -- 172.21.15.1:0/747683711 shutdown_connections 2024-03-06T07:21:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.123+0000 7ff3fe35e640 1 --2- 172.21.15.1:0/747683711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff3d806b500 0x7ff3d806d9c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.123+0000 7ff3fe35e640 1 --2- 172.21.15.1:0/747683711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3f81eae10 0x7ff3f80c8970 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.123+0000 7ff3fe35e640 1 --2- 172.21.15.1:0/747683711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3f814ae00 0x7ff3f80c3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.123+0000 7ff3fe35e640 1 --2- 172.21.15.1:0/747683711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3e40039b0 0x7ff3f81eedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.123+0000 7ff3fe35e640 1 -- 172.21.15.1:0/747683711 >> 172.21.15.1:0/747683711 conn(0x7ff3f808b720 msgr2=0x7ff3f806d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.124+0000 7ff3fe35e640 1 -- 172.21.15.1:0/747683711 shutdown_connections 2024-03-06T07:21:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.124+0000 7ff3fe35e640 1 -- 172.21.15.1:0/747683711 wait complete. 2024-03-06T07:21:15.140 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-06T07:21:15.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.250+0000 7fcd38f56640 1 Processor -- start 2024-03-06T07:21:15.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.250+0000 7fcd38f56640 1 -- start start 2024-03-06T07:21:15.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.251+0000 7fcd38f56640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcd340c70c0 0x7fcd34155550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:15.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.251+0000 7fcd38f56640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd34155a90 0x7fcd34155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:15.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.251+0000 7fcd38f56640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd34158890 con 0x7fcd340c16e0 2024-03-06T07:21:15.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.251+0000 7fcd38f56640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd34158b70 con 0x7fcd34155a90 2024-03-06T07:21:15.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.251+0000 7fcd38f56640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd34158e50 con 0x7fcd340c70c0 2024-03-06T07:21:15.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.251+0000 7fcd32d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd34155a90 0x7fcd34155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:15.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.251+0000 7fcd32d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd34155a90 0x7fcd34155ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37220/0 (socket says 172.21.15.1:37220) 2024-03-06T07:21:15.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.251+0000 7fcd32d9d640 1 -- 172.21.15.1:0/657856741 learned_addr learned my addr 172.21.15.1:0/657856741 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:15.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.251+0000 7fcd31d9b640 1 --2- 172.21.15.1:0/657856741 >> v2:172.21.15.98:3300/0 conn(0x7fcd340c70c0 0x7fcd34155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:15.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.251+0000 7fcd32d9d640 1 -- 172.21.15.1:0/657856741 >> v2:172.21.15.98:3300/0 conn(0x7fcd340c70c0 msgr2=0x7fcd34155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:15.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.251+0000 7fcd32d9d640 1 --2- 172.21.15.1:0/657856741 >> v2:172.21.15.98:3300/0 conn(0x7fcd340c70c0 0x7fcd34155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:15.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.251+0000 7fcd32d9d640 1 -- 172.21.15.1:0/657856741 >> v1:172.21.15.98:6789/0 conn(0x7fcd340c16e0 legacy=0x7fcd34152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:15.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.251+0000 7fcd32d9d640 1 -- 172.21.15.1:0/657856741 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd34159130 con 0x7fcd34155a90 2024-03-06T07:21:15.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.252+0000 7fcd32d9d640 1 --2- 172.21.15.1:0/657856741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd34155a90 0x7fcd34155ec0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fcd340b0e40 tx=0x7fcd2c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15e99d16e12124f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:15.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.252+0000 7fcd3159a640 1 -- 172.21.15.1:0/657856741 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c02c9f0 con 0x7fcd34155a90 2024-03-06T07:21:15.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.252+0000 7fcd3159a640 1 -- 172.21.15.1:0/657856741 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2c02cb90 con 0x7fcd34155a90 2024-03-06T07:21:15.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.252+0000 7fcd3159a640 1 -- 172.21.15.1:0/657856741 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c0343f0 con 0x7fcd34155a90 2024-03-06T07:21:15.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.253+0000 7fcd38f56640 1 -- 172.21.15.1:0/657856741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd34155a90 msgr2=0x7fcd34155ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:15.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.253+0000 7fcd38f56640 1 --2- 172.21.15.1:0/657856741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd34155a90 0x7fcd34155ec0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fcd340b0e40 tx=0x7fcd2c02be50 comp rx=0 tx=0).stop 2024-03-06T07:21:15.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.253+0000 7fcd38f56640 1 -- 172.21.15.1:0/657856741 shutdown_connections 2024-03-06T07:21:15.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.253+0000 7fcd38f56640 1 --2- 172.21.15.1:0/657856741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd34155a90 0x7fcd34155ec0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:15.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.253+0000 7fcd38f56640 1 --2- 172.21.15.1:0/657856741 >> v2:172.21.15.98:3300/0 conn(0x7fcd340c70c0 0x7fcd34155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:15.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.253+0000 7fcd38f56640 1 -- 172.21.15.1:0/657856741 >> 172.21.15.1:0/657856741 conn(0x7fcd34066d00 msgr2=0x7fcd34067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:15.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.253+0000 7fcd38f56640 1 -- 172.21.15.1:0/657856741 shutdown_connections 2024-03-06T07:21:15.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.253+0000 7fcd38f56640 1 -- 172.21.15.1:0/657856741 wait complete. 2024-03-06T07:21:15.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.254+0000 7fcd38f56640 1 Processor -- start 2024-03-06T07:21:15.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.254+0000 7fcd38f56640 1 -- start start 2024-03-06T07:21:15.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.255+0000 7fcd38f56640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd340c16e0 0x7fcd341517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:15.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.255+0000 7fcd38f56640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd340c70c0 0x7fcd34153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:15.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.255+0000 7fcd38f56640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd34155a90 0x7fcd340c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:15.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.255+0000 7fcd38f56640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd340c68c0 con 0x7fcd340c70c0 2024-03-06T07:21:15.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.255+0000 7fcd38f56640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd341596f0 con 0x7fcd340c16e0 2024-03-06T07:21:15.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.255+0000 7fcd38f56640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd340cb470 con 0x7fcd34155a90 2024-03-06T07:21:15.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.255+0000 7fcd31d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd340c70c0 0x7fcd34153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:15.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.255+0000 7fcd31d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd340c70c0 0x7fcd34153380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37232/0 (socket says 172.21.15.1:37232) 2024-03-06T07:21:15.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.255+0000 7fcd31d9b640 1 -- 172.21.15.1:0/2020850033 learned_addr learned my addr 172.21.15.1:0/2020850033 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:15.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.255+0000 7fcd3259c640 1 --2- 172.21.15.1:0/2020850033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd340c16e0 0x7fcd341517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:15.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.255+0000 7fcd32d9d640 1 --2- 172.21.15.1:0/2020850033 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd34155a90 0x7fcd340c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:15.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.255+0000 7fcd31d9b640 1 -- 172.21.15.1:0/2020850033 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd34155a90 msgr2=0x7fcd340c8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:15.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.255+0000 7fcd31d9b640 1 --2- 172.21.15.1:0/2020850033 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd34155a90 0x7fcd340c8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:15.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.255+0000 7fcd31d9b640 1 -- 172.21.15.1:0/2020850033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd340c16e0 msgr2=0x7fcd341517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.255+0000 7fcd31d9b640 1 --2- 172.21.15.1:0/2020850033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd340c16e0 0x7fcd341517a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.255+0000 7fcd31d9b640 1 -- 172.21.15.1:0/2020850033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd34150f70 con 0x7fcd340c70c0 2024-03-06T07:21:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.255+0000 7fcd31d9b640 1 --2- 172.21.15.1:0/2020850033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd340c70c0 0x7fcd34153380 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800d800 tx=0x7fcd2800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.256+0000 7fcd22ffd640 1 -- 172.21.15.1:0/2020850033 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2800b6c0 con 0x7fcd340c70c0 2024-03-06T07:21:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.256+0000 7fcd22ffd640 1 -- 172.21.15.1:0/2020850033 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd28019070 con 0x7fcd340c70c0 2024-03-06T07:21:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.256+0000 7fcd38f56640 1 -- 172.21.15.1:0/2020850033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd34151ce0 con 0x7fcd340c70c0 2024-03-06T07:21:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.256+0000 7fcd22ffd640 1 -- 172.21.15.1:0/2020850033 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2800ba80 con 0x7fcd340c70c0 2024-03-06T07:21:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.256+0000 7fcd38f56640 1 -- 172.21.15.1:0/2020850033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd34153e70 con 0x7fcd340c70c0 2024-03-06T07:21:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.257+0000 7fcd38f56640 1 -- 172.21.15.1:0/2020850033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd340680b0 con 0x7fcd340c70c0 2024-03-06T07:21:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.257+0000 7fcd22ffd640 1 -- 172.21.15.1:0/2020850033 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd2800bc20 con 0x7fcd340c70c0 2024-03-06T07:21:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.258+0000 7fcd22ffd640 1 --2- 172.21.15.1:0/2020850033 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd1406b4d0 0x7fcd1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.258+0000 7fcd22ffd640 1 -- 172.21.15.1:0/2020850033 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fcd2808a560 con 0x7fcd340c70c0 2024-03-06T07:21:15.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.258+0000 7fcd3259c640 1 --2- 172.21.15.1:0/2020850033 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd1406b4d0 0x7fcd1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:15.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.258+0000 7fcd3259c640 1 --2- 172.21.15.1:0/2020850033 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd1406b4d0 0x7fcd1406d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcd24004f60 tx=0x7fcd24007e30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:15.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.262+0000 7fcd22ffd640 1 -- 172.21.15.1:0/2020850033 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd28058d90 con 0x7fcd340c70c0 2024-03-06T07:21:15.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:15.442+0000 7fcd38f56640 1 -- 172.21.15.1:0/2020850033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcd340b0880 con 0x7fcd340c70c0 2024-03-06T07:21:16.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.125+0000 7fcd22ffd640 1 -- 172.21.15.1:0/2020850033 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) v1 ==== 178+0+0 (secure 0 0 0) 0x7fcd28088e20 con 0x7fcd340c70c0 2024-03-06T07:21:16.127 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data_ec' removed 2024-03-06T07:21:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.129+0000 7fcd38f56640 1 -- 172.21.15.1:0/2020850033 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd1406b4d0 msgr2=0x7fcd1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.129+0000 7fcd38f56640 1 --2- 172.21.15.1:0/2020850033 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd1406b4d0 0x7fcd1406d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcd24004f60 tx=0x7fcd24007e30 comp rx=0 tx=0).stop 2024-03-06T07:21:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.129+0000 7fcd38f56640 1 -- 172.21.15.1:0/2020850033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd340c70c0 msgr2=0x7fcd34153380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.129+0000 7fcd38f56640 1 --2- 172.21.15.1:0/2020850033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd340c70c0 0x7fcd34153380 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800d800 tx=0x7fcd2800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:21:16.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.131+0000 7fcd38f56640 1 -- 172.21.15.1:0/2020850033 shutdown_connections 2024-03-06T07:21:16.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.131+0000 7fcd38f56640 1 --2- 172.21.15.1:0/2020850033 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd1406b4d0 0x7fcd1406d990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:16.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.131+0000 7fcd38f56640 1 --2- 172.21.15.1:0/2020850033 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd34155a90 0x7fcd340c8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:16.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.131+0000 7fcd38f56640 1 --2- 172.21.15.1:0/2020850033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd340c70c0 0x7fcd34153380 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:16.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.131+0000 7fcd38f56640 1 --2- 172.21.15.1:0/2020850033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd340c16e0 0x7fcd341517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:16.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.131+0000 7fcd38f56640 1 -- 172.21.15.1:0/2020850033 >> 172.21.15.1:0/2020850033 conn(0x7fcd34066d00 msgr2=0x7fcd340c0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:16.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.131+0000 7fcd38f56640 1 -- 172.21.15.1:0/2020850033 shutdown_connections 2024-03-06T07:21:16.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.131+0000 7fcd38f56640 1 -- 172.21.15.1:0/2020850033 wait complete. 2024-03-06T07:21:16.152 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:21:16.152 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:21:16.153 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:21:16.153 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:21:16.153 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:21:16.153 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:21:16.153 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:21:16.153 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:21:16.153 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:21:16.154 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:21:16.154 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:21:16.154 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:21:16.154 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:21:16.154 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:21:16.154 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:21:16.154 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:21:16.163 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:21:16.163 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:21:16.164 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:21:16.164 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:21:16.165 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:21:16.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:21:16.166 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:21:16.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:21:16.206 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:21:16.206 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:16.208 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:21:16.208 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:16.209 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:21:16.209 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:16.210 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:21:16.210 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:16.210 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:21:16.287 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:21:16.287 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:21:16.293 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:21:16.293 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:21:16.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.317+0000 7f5f090e1640 1 Processor -- start 2024-03-06T07:21:16.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.317+0000 7f5f090e1640 1 -- start start 2024-03-06T07:21:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.318+0000 7f5f090e1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5f04152820 0x7f5f040c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.318+0000 7f5f090e1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f040af6f0 0x7f5f0415fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.318+0000 7f5f090e1640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f040c7180 con 0x7f5f040c0670 2024-03-06T07:21:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.318+0000 7f5f090e1640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f04155460 con 0x7f5f04152820 2024-03-06T07:21:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.318+0000 7f5f090e1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f040c6bc0 con 0x7f5f040af6f0 2024-03-06T07:21:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.318+0000 7f5f0259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5f040c0670 0x7f5f040c73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38542/0 (socket says 172.21.15.1:38542) 2024-03-06T07:21:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.318+0000 7f5f0259c640 1 -- 172.21.15.1:0/3512252294 learned_addr learned my addr 172.21.15.1:0/3512252294 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.318+0000 7f5f02d9d640 1 --2- 172.21.15.1:0/3512252294 >> v2:172.21.15.1:3300/0 conn(0x7f5f04152820 0x7f5f040c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.318+0000 7f5f0359e640 1 --2- 172.21.15.1:0/3512252294 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f040af6f0 0x7f5f0415fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.318+0000 7f5f01d9b640 1 -- 172.21.15.1:0/3512252294 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351899585 0 0) 0x7f5f040c7180 con 0x7f5f040c0670 2024-03-06T07:21:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.318+0000 7f5f01d9b640 1 -- 172.21.15.1:0/3512252294 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ef0003520 con 0x7f5f040c0670 2024-03-06T07:21:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.319+0000 7f5f02d9d640 1 -- 172.21.15.1:0/3512252294 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f040af6f0 msgr2=0x7f5f0415fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.319+0000 7f5f02d9d640 1 --2- 172.21.15.1:0/3512252294 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f040af6f0 0x7f5f0415fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.319+0000 7f5f02d9d640 1 -- 172.21.15.1:0/3512252294 >> v1:172.21.15.1:6789/0 conn(0x7f5f040c0670 legacy=0x7f5f040c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.319+0000 7f5f02d9d640 1 -- 172.21.15.1:0/3512252294 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f040c6ea0 con 0x7f5f04152820 2024-03-06T07:21:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.319+0000 7f5f0359e640 1 --2- 172.21.15.1:0/3512252294 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f040af6f0 0x7f5f0415fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.319+0000 7f5f02d9d640 1 --2- 172.21.15.1:0/3512252294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f04152820 0x7f5f040c0130 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5efc0097a0 tx=0x7f5efc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b0d7d68d913aad0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.319+0000 7f5f01d9b640 1 -- 172.21.15.1:0/3512252294 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc035020 con 0x7f5f04152820 2024-03-06T07:21:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.320+0000 7f5f01d9b640 1 -- 172.21.15.1:0/3512252294 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5efc031810 con 0x7f5f04152820 2024-03-06T07:21:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.320+0000 7f5f01d9b640 1 -- 172.21.15.1:0/3512252294 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc031bc0 con 0x7f5f04152820 2024-03-06T07:21:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.320+0000 7f5f090e1640 1 -- 172.21.15.1:0/3512252294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f04152820 msgr2=0x7f5f040c0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.320+0000 7f5f090e1640 1 --2- 172.21.15.1:0/3512252294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f04152820 0x7f5f040c0130 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5efc0097a0 tx=0x7f5efc02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:21:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.320+0000 7f5f090e1640 1 -- 172.21.15.1:0/3512252294 shutdown_connections 2024-03-06T07:21:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.320+0000 7f5f090e1640 1 --2- 172.21.15.1:0/3512252294 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f040af6f0 0x7f5f0415fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.320+0000 7f5f090e1640 1 --2- 172.21.15.1:0/3512252294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f04152820 0x7f5f040c0130 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.320+0000 7f5f090e1640 1 -- 172.21.15.1:0/3512252294 >> 172.21.15.1:0/3512252294 conn(0x7f5f04072720 msgr2=0x7f5f04072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.320+0000 7f5f090e1640 1 -- 172.21.15.1:0/3512252294 shutdown_connections 2024-03-06T07:21:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.320+0000 7f5f090e1640 1 -- 172.21.15.1:0/3512252294 wait complete. 2024-03-06T07:21:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.321+0000 7f5f090e1640 1 Processor -- start 2024-03-06T07:21:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.321+0000 7f5f090e1640 1 -- start start 2024-03-06T07:21:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.322+0000 7f5f090e1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f040af6f0 0x7f5f041e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.322+0000 7f5f090e1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f040c0670 0x7f5f04158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:16.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.322+0000 7f5f090e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f04152820 0x7f5f04158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:16.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.322+0000 7f5f090e1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f040c7c80 con 0x7f5f040af6f0 2024-03-06T07:21:16.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.322+0000 7f5f090e1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f040c7b00 con 0x7f5f04152820 2024-03-06T07:21:16.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.322+0000 7f5f090e1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f04152c00 con 0x7f5f040c0670 2024-03-06T07:21:16.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.322+0000 7f5f02d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f040af6f0 0x7f5f041e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:16.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.322+0000 7f5f02d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f040af6f0 0x7f5f041e7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37256/0 (socket says 172.21.15.1:37256) 2024-03-06T07:21:16.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.322+0000 7f5f02d9d640 1 -- 172.21.15.1:0/318749742 learned_addr learned my addr 172.21.15.1:0/318749742 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:16.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.322+0000 7f5f0259c640 1 --2- 172.21.15.1:0/318749742 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f040c0670 0x7f5f04158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:16.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.322+0000 7f5f0359e640 1 --2- 172.21.15.1:0/318749742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f04152820 0x7f5f04158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:16.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.323+0000 7f5f02d9d640 1 -- 172.21.15.1:0/318749742 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f040c0670 msgr2=0x7f5f04158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:16.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.323+0000 7f5f02d9d640 1 --2- 172.21.15.1:0/318749742 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f040c0670 0x7f5f04158930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:16.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.323+0000 7f5f02d9d640 1 -- 172.21.15.1:0/318749742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f04152820 msgr2=0x7f5f04158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:16.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.323+0000 7f5f02d9d640 1 --2- 172.21.15.1:0/318749742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f04152820 0x7f5f04158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:16.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.323+0000 7f5f02d9d640 1 -- 172.21.15.1:0/318749742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f04155170 con 0x7f5f040af6f0 2024-03-06T07:21:16.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.323+0000 7f5f02d9d640 1 --2- 172.21.15.1:0/318749742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f040af6f0 0x7f5f041e7c90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5efc0098c0 tx=0x7f5efc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:16.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.323+0000 7f5efb7fe640 1 -- 172.21.15.1:0/318749742 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc035070 con 0x7f5f040af6f0 2024-03-06T07:21:16.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.323+0000 7f5efb7fe640 1 -- 172.21.15.1:0/318749742 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5efc003610 con 0x7f5f040af6f0 2024-03-06T07:21:16.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.323+0000 7f5f090e1640 1 -- 172.21.15.1:0/318749742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f040aa260 con 0x7f5f040af6f0 2024-03-06T07:21:16.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.323+0000 7f5efb7fe640 1 -- 172.21.15.1:0/318749742 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc0039c0 con 0x7f5f040af6f0 2024-03-06T07:21:16.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.323+0000 7f5f090e1640 1 -- 172.21.15.1:0/318749742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f04157150 con 0x7f5f040af6f0 2024-03-06T07:21:16.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.324+0000 7f5f090e1640 1 -- 172.21.15.1:0/318749742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f040686e0 con 0x7f5f040af6f0 2024-03-06T07:21:16.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.325+0000 7f5efb7fe640 1 -- 172.21.15.1:0/318749742 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5efc038020 con 0x7f5f040af6f0 2024-03-06T07:21:16.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.325+0000 7f5efb7fe640 1 --2- 172.21.15.1:0/318749742 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ee406b500 0x7f5ee406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:16.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.325+0000 7f5efb7fe640 1 -- 172.21.15.1:0/318749742 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f5efc0abaf0 con 0x7f5f040af6f0 2024-03-06T07:21:16.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.325+0000 7f5f0259c640 1 --2- 172.21.15.1:0/318749742 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ee406b500 0x7f5ee406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:16.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.326+0000 7f5f0259c640 1 --2- 172.21.15.1:0/318749742 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ee406b500 0x7f5ee406d9c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5f04151890 tx=0x7f5eec007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.329+0000 7f5efb7fe640 1 -- 172.21.15.1:0/318749742 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5efc07a4a0 con 0x7f5f040af6f0 2024-03-06T07:21:16.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:16.523+0000 7f5f090e1640 1 -- 172.21.15.1:0/318749742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5f040c4c70 con 0x7f5f040af6f0 2024-03-06T07:21:17.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.134+0000 7f5efb7fe640 1 -- 172.21.15.1:0/318749742 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5efc07e540 con 0x7f5f040af6f0 2024-03-06T07:21:17.136 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-06T07:21:17.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.137+0000 7f5f090e1640 1 -- 172.21.15.1:0/318749742 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ee406b500 msgr2=0x7f5ee406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.137+0000 7f5f090e1640 1 --2- 172.21.15.1:0/318749742 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ee406b500 0x7f5ee406d9c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5f04151890 tx=0x7f5eec007d30 comp rx=0 tx=0).stop 2024-03-06T07:21:17.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.137+0000 7f5f090e1640 1 -- 172.21.15.1:0/318749742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f040af6f0 msgr2=0x7f5f041e7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.137+0000 7f5f090e1640 1 --2- 172.21.15.1:0/318749742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f040af6f0 0x7f5f041e7c90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5efc0098c0 tx=0x7f5efc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.138+0000 7f5f090e1640 1 -- 172.21.15.1:0/318749742 shutdown_connections 2024-03-06T07:21:17.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.138+0000 7f5f090e1640 1 --2- 172.21.15.1:0/318749742 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ee406b500 0x7f5ee406d9c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.138+0000 7f5f090e1640 1 --2- 172.21.15.1:0/318749742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f04152820 0x7f5f04158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.138+0000 7f5f090e1640 1 --2- 172.21.15.1:0/318749742 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f040c0670 0x7f5f04158930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.138+0000 7f5f090e1640 1 --2- 172.21.15.1:0/318749742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f040af6f0 0x7f5f041e7c90 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.138+0000 7f5f090e1640 1 -- 172.21.15.1:0/318749742 >> 172.21.15.1:0/318749742 conn(0x7f5f04072720 msgr2=0x7f5f040c73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:17.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.138+0000 7f5f090e1640 1 -- 172.21.15.1:0/318749742 shutdown_connections 2024-03-06T07:21:17.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.138+0000 7f5f090e1640 1 -- 172.21.15.1:0/318749742 wait complete. 2024-03-06T07:21:17.154 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:21:17.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.258+0000 7f658fa4e640 1 Processor -- start 2024-03-06T07:21:17.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.258+0000 7f658fa4e640 1 -- start start 2024-03-06T07:21:17.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.259+0000 7f658fa4e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6588149a90 0x7f65880c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.259+0000 7f658fa4e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f65880c16f0 0x7f6588156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.259+0000 7f658fa4e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65881530a0 con 0x7f65880c1260 2024-03-06T07:21:17.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.259+0000 7f658fa4e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6588153380 con 0x7f6588149a90 2024-03-06T07:21:17.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.259+0000 7f658fa4e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6588153660 con 0x7f65880c16f0 2024-03-06T07:21:17.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.259+0000 7f658d7ea640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6588149a90 0x7f65880c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.259+0000 7f658d7ea640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6588149a90 0x7f65880c4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37262/0 (socket says 172.21.15.1:37262) 2024-03-06T07:21:17.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.259+0000 7f658d7ea640 1 -- 172.21.15.1:0/2517460886 learned_addr learned my addr 172.21.15.1:0/2517460886 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:17.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.259+0000 7f658dfeb640 1 --2- 172.21.15.1:0/2517460886 >> v2:172.21.15.98:3300/0 conn(0x7f65880c16f0 0x7f6588156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.259+0000 7f658d7ea640 1 -- 172.21.15.1:0/2517460886 >> v2:172.21.15.98:3300/0 conn(0x7f65880c16f0 msgr2=0x7f6588156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:17.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.259+0000 7f658d7ea640 1 --2- 172.21.15.1:0/2517460886 >> v2:172.21.15.98:3300/0 conn(0x7f65880c16f0 0x7f6588156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.259+0000 7f658d7ea640 1 -- 172.21.15.1:0/2517460886 >> v1:172.21.15.98:6789/0 conn(0x7f65880c1260 legacy=0x7f65881565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:17.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.259+0000 7f658d7ea640 1 -- 172.21.15.1:0/2517460886 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6588153940 con 0x7f6588149a90 2024-03-06T07:21:17.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.259+0000 7f658dfeb640 1 --2- 172.21.15.1:0/2517460886 >> v2:172.21.15.98:3300/0 conn(0x7f65880c16f0 0x7f6588156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:17.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.260+0000 7f658d7ea640 1 --2- 172.21.15.1:0/2517460886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6588149a90 0x7f65880c4760 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f65880c3280 tx=0x7f658002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd69353436dd410 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:17.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.260+0000 7f657ffff640 1 -- 172.21.15.1:0/2517460886 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658002e070 con 0x7f6588149a90 2024-03-06T07:21:17.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.260+0000 7f657ffff640 1 -- 172.21.15.1:0/2517460886 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6580037070 con 0x7f6588149a90 2024-03-06T07:21:17.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.260+0000 7f657ffff640 1 -- 172.21.15.1:0/2517460886 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6580031a00 con 0x7f6588149a90 2024-03-06T07:21:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.261+0000 7f658fa4e640 1 -- 172.21.15.1:0/2517460886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6588149a90 msgr2=0x7f65880c4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.261+0000 7f658fa4e640 1 --2- 172.21.15.1:0/2517460886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6588149a90 0x7f65880c4760 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f65880c3280 tx=0x7f658002b880 comp rx=0 tx=0).stop 2024-03-06T07:21:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.261+0000 7f658fa4e640 1 -- 172.21.15.1:0/2517460886 shutdown_connections 2024-03-06T07:21:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.261+0000 7f658fa4e640 1 --2- 172.21.15.1:0/2517460886 >> v2:172.21.15.98:3300/0 conn(0x7f65880c16f0 0x7f6588156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.261+0000 7f658fa4e640 1 --2- 172.21.15.1:0/2517460886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6588149a90 0x7f65880c4760 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.261+0000 7f658fa4e640 1 -- 172.21.15.1:0/2517460886 >> 172.21.15.1:0/2517460886 conn(0x7f65880c6860 msgr2=0x7f65880c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.261+0000 7f658fa4e640 1 -- 172.21.15.1:0/2517460886 shutdown_connections 2024-03-06T07:21:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.261+0000 7f658fa4e640 1 -- 172.21.15.1:0/2517460886 wait complete. 2024-03-06T07:21:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.262+0000 7f658fa4e640 1 Processor -- start 2024-03-06T07:21:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.262+0000 7f658fa4e640 1 -- start start 2024-03-06T07:21:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f658fa4e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65880c1260 0x7f658814f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f658fa4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65880c16f0 0x7f6588145870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f658fa4e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6588149a90 0x7f6588145db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f658fa4e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65880c4ca0 con 0x7f6588149a90 2024-03-06T07:21:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f658fa4e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65881571f0 con 0x7f65880c16f0 2024-03-06T07:21:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f658fa4e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658806f4a0 con 0x7f65880c1260 2024-03-06T07:21:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f658dfeb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6588149a90 0x7f6588145db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f658dfeb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6588149a90 0x7f6588145db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37274/0 (socket says 172.21.15.1:37274) 2024-03-06T07:21:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f658dfeb640 1 -- 172.21.15.1:0/742582413 learned_addr learned my addr 172.21.15.1:0/742582413 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f658dfeb640 1 -- 172.21.15.1:0/742582413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65880c1260 msgr2=0x7f658814f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f658cfe9640 1 --2- 172.21.15.1:0/742582413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65880c16f0 0x7f6588145870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f658d7ea640 1 --2- 172.21.15.1:0/742582413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65880c1260 0x7f658814f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f658dfeb640 1 --2- 172.21.15.1:0/742582413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65880c1260 0x7f658814f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f658dfeb640 1 -- 172.21.15.1:0/742582413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65880c16f0 msgr2=0x7f6588145870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f658dfeb640 1 --2- 172.21.15.1:0/742582413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65880c16f0 0x7f6588145870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f658dfeb640 1 -- 172.21.15.1:0/742582413 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65881502c0 con 0x7f6588149a90 2024-03-06T07:21:17.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f658dfeb640 1 --2- 172.21.15.1:0/742582413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6588149a90 0x7f6588145db0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f658400cef0 tx=0x7f658400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:17.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f657dffb640 1 -- 172.21.15.1:0/742582413 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658401a070 con 0x7f6588149a90 2024-03-06T07:21:17.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.263+0000 7f657dffb640 1 -- 172.21.15.1:0/742582413 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658400d860 con 0x7f6588149a90 2024-03-06T07:21:17.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.264+0000 7f657dffb640 1 -- 172.21.15.1:0/742582413 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658400dc50 con 0x7f6588149a90 2024-03-06T07:21:17.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.264+0000 7f658fa4e640 1 -- 172.21.15.1:0/742582413 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6588155520 con 0x7f6588149a90 2024-03-06T07:21:17.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.264+0000 7f658fa4e640 1 -- 172.21.15.1:0/742582413 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6588155ae0 con 0x7f6588149a90 2024-03-06T07:21:17.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.265+0000 7f658fa4e640 1 -- 172.21.15.1:0/742582413 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6588067fd0 con 0x7f6588149a90 2024-03-06T07:21:17.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.265+0000 7f657dffb640 1 -- 172.21.15.1:0/742582413 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f658401d070 con 0x7f6588149a90 2024-03-06T07:21:17.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.265+0000 7f657dffb640 1 --2- 172.21.15.1:0/742582413 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f656c06b4d0 0x7f656c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.265+0000 7f657dffb640 1 -- 172.21.15.1:0/742582413 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f658408a850 con 0x7f6588149a90 2024-03-06T07:21:17.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.266+0000 7f658d7ea640 1 --2- 172.21.15.1:0/742582413 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f656c06b4d0 0x7f656c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.266+0000 7f658d7ea640 1 --2- 172.21.15.1:0/742582413 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f656c06b4d0 0x7f656c06d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6580036000 tx=0x7f6580008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.270+0000 7f657dffb640 1 -- 172.21.15.1:0/742582413 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6584059900 con 0x7f6588149a90 2024-03-06T07:21:17.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.524+0000 7f658fa4e640 1 -- 172.21.15.1:0/742582413 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f658806f280 con 0x7f6588149a90 2024-03-06T07:21:17.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.524+0000 7f657dffb640 1 -- 172.21.15.1:0/742582413 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6065 (secure 0 0 0) 0x7f6584059200 con 0x7f6588149a90 2024-03-06T07:21:17.526 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:17.526 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:21:17.526 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-06T07:21:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-06T07:21:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlnxyWARAAKIPXU0IafNQ1RzBPX+3DYw==", 2024-03-06T07:21:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:21:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:21:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:21:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:21:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-06T07:21:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl2fXxDRAAuLqH+HpKcN1JTts9ZtDTXA==", 2024-03-06T07:21:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:21:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:21:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:21:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:21:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-06T07:21:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl+IzqCBAAmgLkarFGt9Ihs/8kJYicBQ==", 2024-03-06T07:21:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:21:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:21:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:21:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:21:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-06T07:21:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlipdRFRAAA34vrsTEoX3L1CKb1YAV9g==", 2024-03-06T07:21:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:21:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:21:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:21:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:21:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-06T07:21:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAGGehlQK39LxAAOpPIZvl3FqQwCgyo3yoOGQ==", 2024-03-06T07:21:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:21:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:21:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:21:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-06T07:21:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAJGehlTxwDLhAAMwN9E0+3Jh5XNXnLo+fw3g==", 2024-03-06T07:21:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:21:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:21:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:21:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-06T07:21:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAMGehl4qrlLBAAm0VWQRNvSQIpDx5xf4Fyig==", 2024-03-06T07:21:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:21:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:21:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:21:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-06T07:21:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAPGehllO5HKxAAozm6ycG+fRxatoZTogeJTA==", 2024-03-06T07:21:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:21:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:21:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:21:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-06T07:21:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAVGehlLtLBMhAAUrV6q6ZCQluIcoo/S/NdbA==", 2024-03-06T07:21:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:21:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:21:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:21:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-06T07:21:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAYGehlHp59MRAAxxgpSPTEZkYKjl3fr0xeeQ==", 2024-03-06T07:21:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:21:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:21:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:21:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-06T07:21:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAbGehlTnXcMBAAFOBqcnh5ubAz+o6AGjccWQ==", 2024-03-06T07:21:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:21:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:21:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:21:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-06T07:21:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAeGehl1qH1LhAAaSl7Aj7qlN6d4xt5SKAmcg==", 2024-03-06T07:21:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:21:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:21:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:21:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-06T07:21:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng==", 2024-03-06T07:21:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:21:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-06T07:21:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-06T07:21:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-06T07:21:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-06T07:21:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlMyv7HBAAi7vt7aZWEaZuMpFrB5ofZw==", 2024-03-06T07:21:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:21:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-06T07:21:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-06T07:21:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-06T07:21:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-06T07:21:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlBS/vIBAAN86N2K4ze5PJB8kR869Lfg==", 2024-03-06T07:21:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:21:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:21:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:21:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:21:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-mds", 2024-03-06T07:21:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAkGehlyVDRGxAA/KOyY3sGHvMdwugBBmTh2g==", 2024-03-06T07:21:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-mds" 2024-03-06T07:21:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-mgr", 2024-03-06T07:21:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAkGehlv1nRGxAA7KFhgwo9LpDZ1vlD6R/WDQ==", 2024-03-06T07:21:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-06T07:21:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-osd", 2024-03-06T07:21:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAkGehlh2DRGxAAEU8oWsQXoCytasnyicHWEQ==", 2024-03-06T07:21:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-osd" 2024-03-06T07:21:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-rbd", 2024-03-06T07:21:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAkGehl3WbRGxAA2XcRxpSL3aq6KZZwFLeDzw==", 2024-03-06T07:21:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-06T07:21:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-06T07:21:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAkGehlVG3RGxAALe9NWMTrmzECsYPdHFUahw==", 2024-03-06T07:21:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-06T07:21:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-rgw", 2024-03-06T07:21:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAkGehlJXTRGxAAL91zwYX5n3ocmTiI8LbdfA==", 2024-03-06T07:21:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-06T07:21:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-06T07:21:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlJ/jPORAAhLk5MTGafsGKhZSYhvhBZg==", 2024-03-06T07:21:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:21:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:21:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:21:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:21:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-06T07:21:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlAl8rNhAAvdU0Qvo/FJYnxc4/H8MIqQ==", 2024-03-06T07:21:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:21:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:21:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:21:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:21:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-06T07:21:17.543 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:21:17.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.526+0000 7f658fa4e640 1 -- 172.21.15.1:0/742582413 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f656c06b4d0 msgr2=0x7f656c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.526+0000 7f658fa4e640 1 --2- 172.21.15.1:0/742582413 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f656c06b4d0 0x7f656c06d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6580036000 tx=0x7f6580008030 comp rx=0 tx=0).stop 2024-03-06T07:21:17.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.526+0000 7f658fa4e640 1 -- 172.21.15.1:0/742582413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6588149a90 msgr2=0x7f6588145db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.526+0000 7f658fa4e640 1 --2- 172.21.15.1:0/742582413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6588149a90 0x7f6588145db0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f658400cef0 tx=0x7f658400a710 comp rx=0 tx=0).stop 2024-03-06T07:21:17.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.526+0000 7f658fa4e640 1 -- 172.21.15.1:0/742582413 shutdown_connections 2024-03-06T07:21:17.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.526+0000 7f658fa4e640 1 --2- 172.21.15.1:0/742582413 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f656c06b4d0 0x7f656c06d990 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.526+0000 7f658fa4e640 1 --2- 172.21.15.1:0/742582413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6588149a90 0x7f6588145db0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.526+0000 7f658fa4e640 1 --2- 172.21.15.1:0/742582413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65880c16f0 0x7f6588145870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.526+0000 7f658fa4e640 1 --2- 172.21.15.1:0/742582413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f65880c1260 0x7f658814f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.526+0000 7f658fa4e640 1 -- 172.21.15.1:0/742582413 >> 172.21.15.1:0/742582413 conn(0x7f65880c6860 msgr2=0x7f65880c6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:17.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.526+0000 7f658fa4e640 1 -- 172.21.15.1:0/742582413 shutdown_connections 2024-03-06T07:21:17.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.526+0000 7f658fa4e640 1 -- 172.21.15.1:0/742582413 wait complete. 2024-03-06T07:21:17.545 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-06T07:21:17.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.646+0000 7f2a78fa0640 1 Processor -- start 2024-03-06T07:21:17.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.646+0000 7f2a78fa0640 1 -- start start 2024-03-06T07:21:17.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.646+0000 7f2a78fa0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2a74152840 0x7f2a740ca890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.646+0000 7f2a78fa0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740c6ee0 0x7f2a740c5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.646+0000 7f2a78fa0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a74155e90 con 0x7f2a740cadd0 2024-03-06T07:21:17.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.646+0000 7f2a78fa0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a740c1b20 con 0x7f2a74152840 2024-03-06T07:21:17.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.646+0000 7f2a78fa0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a740c14d0 con 0x7f2a740c6ee0 2024-03-06T07:21:17.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.646+0000 7f2a737fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2a74152840 0x7f2a740ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.647+0000 7f2a73fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2a740cadd0 0x7f2a740c59d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38554/0 (socket says 172.21.15.1:38554) 2024-03-06T07:21:17.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.647+0000 7f2a737fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2a74152840 0x7f2a740ca890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37290/0 (socket says 172.21.15.1:37290) 2024-03-06T07:21:17.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.647+0000 7f2a73fff640 1 -- 172.21.15.1:0/3864569491 learned_addr learned my addr 172.21.15.1:0/3864569491 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:17.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.647+0000 7f2a6affd640 1 --2- 172.21.15.1:0/3864569491 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740c6ee0 0x7f2a740c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.647+0000 7f2a737fe640 1 -- 172.21.15.1:0/3864569491 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740c6ee0 msgr2=0x7f2a740c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:17.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.647+0000 7f2a737fe640 1 --2- 172.21.15.1:0/3864569491 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740c6ee0 0x7f2a740c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.647+0000 7f2a737fe640 1 -- 172.21.15.1:0/3864569491 >> v1:172.21.15.1:6789/0 conn(0x7f2a740cadd0 legacy=0x7f2a740c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.647+0000 7f2a737fe640 1 -- 172.21.15.1:0/3864569491 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a740c17b0 con 0x7f2a74152840 2024-03-06T07:21:17.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.647+0000 7f2a72ffd640 1 -- 172.21.15.1:0/3864569491 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013695855 0 0) 0x7f2a74155e90 con 0x7f2a740cadd0 2024-03-06T07:21:17.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.648+0000 7f2a737fe640 1 --2- 172.21.15.1:0/3864569491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a74152840 0x7f2a740ca890 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2a640097a0 tx=0x7f2a6402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b3350119ed0c5f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:17.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.648+0000 7f2a72ffd640 1 -- 172.21.15.1:0/3864569491 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a640315a0 con 0x7f2a74152840 2024-03-06T07:21:17.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.648+0000 7f2a72ffd640 1 -- 172.21.15.1:0/3864569491 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a64037070 con 0x7f2a74152840 2024-03-06T07:21:17.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.648+0000 7f2a72ffd640 1 -- 172.21.15.1:0/3864569491 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64031960 con 0x7f2a74152840 2024-03-06T07:21:17.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.648+0000 7f2a78fa0640 1 -- 172.21.15.1:0/3864569491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a74152840 msgr2=0x7f2a740ca890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.648+0000 7f2a78fa0640 1 --2- 172.21.15.1:0/3864569491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a74152840 0x7f2a740ca890 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2a640097a0 tx=0x7f2a6402b840 comp rx=0 tx=0).stop 2024-03-06T07:21:17.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.648+0000 7f2a78fa0640 1 -- 172.21.15.1:0/3864569491 shutdown_connections 2024-03-06T07:21:17.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.648+0000 7f2a78fa0640 1 --2- 172.21.15.1:0/3864569491 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740c6ee0 0x7f2a740c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.648+0000 7f2a78fa0640 1 --2- 172.21.15.1:0/3864569491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a74152840 0x7f2a740ca890 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.648+0000 7f2a78fa0640 1 -- 172.21.15.1:0/3864569491 >> 172.21.15.1:0/3864569491 conn(0x7f2a74067520 msgr2=0x7f2a7406d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:17.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.648+0000 7f2a78fa0640 1 -- 172.21.15.1:0/3864569491 shutdown_connections 2024-03-06T07:21:17.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.648+0000 7f2a78fa0640 1 -- 172.21.15.1:0/3864569491 wait complete. 2024-03-06T07:21:17.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.649+0000 7f2a78fa0640 1 Processor -- start 2024-03-06T07:21:17.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.650+0000 7f2a78fa0640 1 -- start start 2024-03-06T07:21:17.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.650+0000 7f2a78fa0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a740c6ee0 0x7f2a741ee750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.650+0000 7f2a78fa0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740cadd0 0x7f2a740c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.650+0000 7f2a78fa0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a74152840 0x7f2a7414a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.650+0000 7f2a78fa0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a740bf900 con 0x7f2a740c6ee0 2024-03-06T07:21:17.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.650+0000 7f2a78fa0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a74155b60 con 0x7f2a74152840 2024-03-06T07:21:17.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.650+0000 7f2a78fa0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a740bfa80 con 0x7f2a740cadd0 2024-03-06T07:21:17.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.650+0000 7f2a737fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a740c6ee0 0x7f2a741ee750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.650+0000 7f2a737fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a740c6ee0 0x7f2a741ee750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37304/0 (socket says 172.21.15.1:37304) 2024-03-06T07:21:17.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.650+0000 7f2a737fe640 1 -- 172.21.15.1:0/2893321873 learned_addr learned my addr 172.21.15.1:0/2893321873 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:17.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.651+0000 7f2a737fe640 1 -- 172.21.15.1:0/2893321873 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740cadd0 msgr2=0x7f2a740c99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.651+0000 7f2a737fe640 1 --2- 172.21.15.1:0/2893321873 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740cadd0 0x7f2a740c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.651+0000 7f2a737fe640 1 -- 172.21.15.1:0/2893321873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a74152840 msgr2=0x7f2a7414a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.651+0000 7f2a737fe640 1 --2- 172.21.15.1:0/2893321873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a74152840 0x7f2a7414a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.651+0000 7f2a737fe640 1 -- 172.21.15.1:0/2893321873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a74153fc0 con 0x7f2a740c6ee0 2024-03-06T07:21:17.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.651+0000 7f2a737fe640 1 --2- 172.21.15.1:0/2893321873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a740c6ee0 0x7f2a741ee750 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2a640098c0 tx=0x7f2a640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:17.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.651+0000 7f2a70ff9640 1 -- 172.21.15.1:0/2893321873 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64032840 con 0x7f2a740c6ee0 2024-03-06T07:21:17.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.651+0000 7f2a70ff9640 1 -- 172.21.15.1:0/2893321873 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a640329e0 con 0x7f2a740c6ee0 2024-03-06T07:21:17.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.651+0000 7f2a78fa0640 1 -- 172.21.15.1:0/2893321873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a741542a0 con 0x7f2a740c6ee0 2024-03-06T07:21:17.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.651+0000 7f2a70ff9640 1 -- 172.21.15.1:0/2893321873 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64032dd0 con 0x7f2a740c6ee0 2024-03-06T07:21:17.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.651+0000 7f2a78fa0640 1 -- 172.21.15.1:0/2893321873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a741568d0 con 0x7f2a740c6ee0 2024-03-06T07:21:17.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.652+0000 7f2a78fa0640 1 -- 172.21.15.1:0/2893321873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a74067fd0 con 0x7f2a740c6ee0 2024-03-06T07:21:17.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.652+0000 7f2a70ff9640 1 -- 172.21.15.1:0/2893321873 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a6404c020 con 0x7f2a740c6ee0 2024-03-06T07:21:17.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.653+0000 7f2a70ff9640 1 --2- 172.21.15.1:0/2893321873 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a5406b4d0 0x7f2a5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.653+0000 7f2a70ff9640 1 -- 172.21.15.1:0/2893321873 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f2a6402f020 con 0x7f2a740c6ee0 2024-03-06T07:21:17.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.653+0000 7f2a6affd640 1 --2- 172.21.15.1:0/2893321873 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a5406b4d0 0x7f2a5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.654+0000 7f2a6affd640 1 --2- 172.21.15.1:0/2893321873 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a5406b4d0 0x7f2a5406d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2a740c06d0 tx=0x7f2a5c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:17.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.657+0000 7f2a70ff9640 1 -- 172.21.15.1:0/2893321873 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a64040070 con 0x7f2a740c6ee0 2024-03-06T07:21:17.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.905+0000 7f2a78fa0640 1 -- 172.21.15.1:0/2893321873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f2a740c51e0 con 0x7f2a740c6ee0 2024-03-06T07:21:17.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.908+0000 7f2a70ff9640 1 -- 172.21.15.1:0/2893321873 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f2a6407acd0 con 0x7f2a740c6ee0 2024-03-06T07:21:17.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.910+0000 7f2a78fa0640 1 -- 172.21.15.1:0/2893321873 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a5406b4d0 msgr2=0x7f2a5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.910+0000 7f2a78fa0640 1 --2- 172.21.15.1:0/2893321873 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a5406b4d0 0x7f2a5406d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2a740c06d0 tx=0x7f2a5c008040 comp rx=0 tx=0).stop 2024-03-06T07:21:17.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.910+0000 7f2a78fa0640 1 -- 172.21.15.1:0/2893321873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a740c6ee0 msgr2=0x7f2a741ee750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.910+0000 7f2a78fa0640 1 --2- 172.21.15.1:0/2893321873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a740c6ee0 0x7f2a741ee750 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2a640098c0 tx=0x7f2a640046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.910+0000 7f2a78fa0640 1 -- 172.21.15.1:0/2893321873 shutdown_connections 2024-03-06T07:21:17.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.910+0000 7f2a78fa0640 1 --2- 172.21.15.1:0/2893321873 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a5406b4d0 0x7f2a5406d990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.910+0000 7f2a78fa0640 1 --2- 172.21.15.1:0/2893321873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a74152840 0x7f2a7414a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.910+0000 7f2a78fa0640 1 --2- 172.21.15.1:0/2893321873 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740cadd0 0x7f2a740c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.910+0000 7f2a78fa0640 1 --2- 172.21.15.1:0/2893321873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a740c6ee0 0x7f2a741ee750 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.910+0000 7f2a78fa0640 1 -- 172.21.15.1:0/2893321873 >> 172.21.15.1:0/2893321873 conn(0x7f2a74067520 msgr2=0x7f2a74151b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:17.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.910+0000 7f2a78fa0640 1 -- 172.21.15.1:0/2893321873 shutdown_connections 2024-03-06T07:21:17.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:17.910+0000 7f2a78fa0640 1 -- 172.21.15.1:0/2893321873 wait complete. 2024-03-06T07:21:17.923 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-06T07:21:18.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.023+0000 7fc10701c640 1 Processor -- start 2024-03-06T07:21:18.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.023+0000 7fc10701c640 1 -- start start 2024-03-06T07:21:18.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.023+0000 7fc10701c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc1000c8cd0 0x7fc1000c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.023+0000 7fc10701c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc1000c3260 0x7fc100155b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.023+0000 7fc10701c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1000c2b90 con 0x7fc1000c7560 2024-03-06T07:21:18.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.023+0000 7fc10701c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1000c37b0 con 0x7fc1000c3260 2024-03-06T07:21:18.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.023+0000 7fc10701c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1000c3a90 con 0x7fc1000c8cd0 2024-03-06T07:21:18.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.024+0000 7fc0fffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc1000c3260 0x7fc100155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.024+0000 7fc1055b9640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc1000c7560 0x7fc100159010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38570/0 (socket says 172.21.15.1:38570) 2024-03-06T07:21:18.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.024+0000 7fc0fffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc1000c3260 0x7fc100155b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37318/0 (socket says 172.21.15.1:37318) 2024-03-06T07:21:18.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.024+0000 7fc1055b9640 1 -- 172.21.15.1:0/2464592606 learned_addr learned my addr 172.21.15.1:0/2464592606 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:18.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.024+0000 7fc104db8640 1 --2- 172.21.15.1:0/2464592606 >> v2:172.21.15.98:3300/0 conn(0x7fc1000c8cd0 0x7fc1000c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.024+0000 7fc0fffff640 1 -- 172.21.15.1:0/2464592606 >> v2:172.21.15.98:3300/0 conn(0x7fc1000c8cd0 msgr2=0x7fc1000c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:18.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.024+0000 7fc0fffff640 1 --2- 172.21.15.1:0/2464592606 >> v2:172.21.15.98:3300/0 conn(0x7fc1000c8cd0 0x7fc1000c90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.024+0000 7fc0fffff640 1 -- 172.21.15.1:0/2464592606 >> v1:172.21.15.1:6789/0 conn(0x7fc1000c7560 legacy=0x7fc100159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.024+0000 7fc0ff7fe640 1 -- 172.21.15.1:0/2464592606 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129678037 0 0) 0x7fc1000c2b90 con 0x7fc1000c7560 2024-03-06T07:21:18.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.024+0000 7fc0fffff640 1 -- 172.21.15.1:0/2464592606 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc100158810 con 0x7fc1000c3260 2024-03-06T07:21:18.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.025+0000 7fc0fffff640 1 --2- 172.21.15.1:0/2464592606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1000c3260 0x7fc100155b70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc1000c6400 tx=0x7fc0ec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c4df362122615 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.025+0000 7fc0ff7fe640 1 -- 172.21.15.1:0/2464592606 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec0315a0 con 0x7fc1000c3260 2024-03-06T07:21:18.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.025+0000 7fc0ff7fe640 1 -- 172.21.15.1:0/2464592606 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ec037070 con 0x7fc1000c3260 2024-03-06T07:21:18.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.025+0000 7fc0ff7fe640 1 -- 172.21.15.1:0/2464592606 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec031960 con 0x7fc1000c3260 2024-03-06T07:21:18.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.025+0000 7fc10701c640 1 -- 172.21.15.1:0/2464592606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1000c3260 msgr2=0x7fc100155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.025+0000 7fc10701c640 1 --2- 172.21.15.1:0/2464592606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1000c3260 0x7fc100155b70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc1000c6400 tx=0x7fc0ec02b840 comp rx=0 tx=0).stop 2024-03-06T07:21:18.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.025+0000 7fc10701c640 1 -- 172.21.15.1:0/2464592606 shutdown_connections 2024-03-06T07:21:18.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.025+0000 7fc10701c640 1 --2- 172.21.15.1:0/2464592606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1000c3260 0x7fc100155b70 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.025+0000 7fc10701c640 1 --2- 172.21.15.1:0/2464592606 >> v2:172.21.15.98:3300/0 conn(0x7fc1000c8cd0 0x7fc1000c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.025+0000 7fc10701c640 1 -- 172.21.15.1:0/2464592606 >> 172.21.15.1:0/2464592606 conn(0x7fc10006b3e0 msgr2=0x7fc10006b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:18.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.025+0000 7fc10701c640 1 -- 172.21.15.1:0/2464592606 shutdown_connections 2024-03-06T07:21:18.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.025+0000 7fc10701c640 1 -- 172.21.15.1:0/2464592606 wait complete. 2024-03-06T07:21:18.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.026+0000 7fc10701c640 1 Processor -- start 2024-03-06T07:21:18.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.027+0000 7fc10701c640 1 -- start start 2024-03-06T07:21:18.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.027+0000 7fc10701c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1000c3260 0x7fc1000ac9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.027+0000 7fc10701c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1000c7560 0x7fc1000aabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.027+0000 7fc10701c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1000c8cd0 0x7fc100154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.027+0000 7fc10701c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1000c86e0 con 0x7fc1000c8cd0 2024-03-06T07:21:18.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.027+0000 7fc10701c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1000c8560 con 0x7fc1000c7560 2024-03-06T07:21:18.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.027+0000 7fc10701c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1000c79c0 con 0x7fc1000c3260 2024-03-06T07:21:18.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.027+0000 7fc1055b9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1000c8cd0 0x7fc100154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.027+0000 7fc1055b9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1000c8cd0 0x7fc100154640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37322/0 (socket says 172.21.15.1:37322) 2024-03-06T07:21:18.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.027+0000 7fc1055b9640 1 -- 172.21.15.1:0/3918015281 learned_addr learned my addr 172.21.15.1:0/3918015281 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:18.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.028+0000 7fc0fffff640 1 --2- 172.21.15.1:0/3918015281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1000c7560 0x7fc1000aabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.028+0000 7fc104db8640 1 --2- 172.21.15.1:0/3918015281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1000c3260 0x7fc1000ac9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.028+0000 7fc1055b9640 1 -- 172.21.15.1:0/3918015281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1000c3260 msgr2=0x7fc1000ac9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.028+0000 7fc1055b9640 1 --2- 172.21.15.1:0/3918015281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1000c3260 0x7fc1000ac9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.028+0000 7fc1055b9640 1 -- 172.21.15.1:0/3918015281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1000c7560 msgr2=0x7fc1000aabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.028+0000 7fc1055b9640 1 --2- 172.21.15.1:0/3918015281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1000c7560 0x7fc1000aabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.028+0000 7fc1055b9640 1 -- 172.21.15.1:0/3918015281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1001ea9c0 con 0x7fc1000c8cd0 2024-03-06T07:21:18.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.028+0000 7fc1055b9640 1 --2- 172.21.15.1:0/3918015281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1000c8cd0 0x7fc100154640 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800c7a0 tx=0x7fc0f800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.028+0000 7fc0fd7fa640 1 -- 172.21.15.1:0/3918015281 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8018070 con 0x7fc1000c8cd0 2024-03-06T07:21:18.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.028+0000 7fc0fd7fa640 1 -- 172.21.15.1:0/3918015281 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f800a640 con 0x7fc1000c8cd0 2024-03-06T07:21:18.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.028+0000 7fc0fd7fa640 1 -- 172.21.15.1:0/3918015281 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f800aab0 con 0x7fc1000c8cd0 2024-03-06T07:21:18.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.028+0000 7fc10701c640 1 -- 172.21.15.1:0/3918015281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1001eac60 con 0x7fc1000c8cd0 2024-03-06T07:21:18.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.028+0000 7fc10701c640 1 -- 172.21.15.1:0/3918015281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1001eb220 con 0x7fc1000c8cd0 2024-03-06T07:21:18.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.029+0000 7fc10701c640 1 -- 172.21.15.1:0/3918015281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1000c6760 con 0x7fc1000c8cd0 2024-03-06T07:21:18.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.030+0000 7fc0fd7fa640 1 -- 172.21.15.1:0/3918015281 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0f800ac50 con 0x7fc1000c8cd0 2024-03-06T07:21:18.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.030+0000 7fc0fd7fa640 1 --2- 172.21.15.1:0/3918015281 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc0e406b4d0 0x7fc0e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.030+0000 7fc0fd7fa640 1 -- 172.21.15.1:0/3918015281 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc0f80896d0 con 0x7fc1000c8cd0 2024-03-06T07:21:18.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.030+0000 7fc104db8640 1 --2- 172.21.15.1:0/3918015281 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc0e406b4d0 0x7fc0e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.031+0000 7fc104db8640 1 --2- 172.21.15.1:0/3918015281 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc0e406b4d0 0x7fc0e406d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc0f400a970 tx=0x7fc0f4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.034+0000 7fc0fd7fa640 1 -- 172.21.15.1:0/3918015281 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc0f8058700 con 0x7fc1000c8cd0 2024-03-06T07:21:18.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.286+0000 7fc10701c640 1 -- 172.21.15.1:0/3918015281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fc1000c2d60 con 0x7fc1000c8cd0 2024-03-06T07:21:18.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.287+0000 7fc0fd7fa640 1 -- 172.21.15.1:0/3918015281 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7fc0f8058000 con 0x7fc1000c8cd0 2024-03-06T07:21:18.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.289+0000 7fc10701c640 1 -- 172.21.15.1:0/3918015281 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc0e406b4d0 msgr2=0x7fc0e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.289+0000 7fc10701c640 1 --2- 172.21.15.1:0/3918015281 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc0e406b4d0 0x7fc0e406d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc0f400a970 tx=0x7fc0f4008040 comp rx=0 tx=0).stop 2024-03-06T07:21:18.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.289+0000 7fc10701c640 1 -- 172.21.15.1:0/3918015281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1000c8cd0 msgr2=0x7fc100154640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.289+0000 7fc10701c640 1 --2- 172.21.15.1:0/3918015281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1000c8cd0 0x7fc100154640 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800c7a0 tx=0x7fc0f800cc60 comp rx=0 tx=0).stop 2024-03-06T07:21:18.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.289+0000 7fc10701c640 1 -- 172.21.15.1:0/3918015281 shutdown_connections 2024-03-06T07:21:18.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.289+0000 7fc10701c640 1 --2- 172.21.15.1:0/3918015281 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc0e406b4d0 0x7fc0e406d990 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.289+0000 7fc10701c640 1 --2- 172.21.15.1:0/3918015281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1000c8cd0 0x7fc100154640 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.289+0000 7fc10701c640 1 --2- 172.21.15.1:0/3918015281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1000c7560 0x7fc1000aabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.289+0000 7fc10701c640 1 --2- 172.21.15.1:0/3918015281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1000c3260 0x7fc1000ac9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.289+0000 7fc10701c640 1 -- 172.21.15.1:0/3918015281 >> 172.21.15.1:0/3918015281 conn(0x7fc10006b3e0 msgr2=0x7fc10006b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:18.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.289+0000 7fc10701c640 1 -- 172.21.15.1:0/3918015281 shutdown_connections 2024-03-06T07:21:18.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.290+0000 7fc10701c640 1 -- 172.21.15.1:0/3918015281 wait complete. 2024-03-06T07:21:18.302 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-06T07:21:18.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.402+0000 7fc5cc35e640 1 Processor -- start 2024-03-06T07:21:18.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.402+0000 7fc5cc35e640 1 -- start start 2024-03-06T07:21:18.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5cc35e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5c4156cd0 0x7fc5c4157160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5cc35e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c40c3300 con 0x7fc5c40c6730 2024-03-06T07:21:18.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5cc35e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c40c35a0 con 0x7fc5c4152690 2024-03-06T07:21:18.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5cc35e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c40c3880 con 0x7fc5c4156cd0 2024-03-06T07:21:18.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c98f9640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc5c40c6730 0x7fc5c40c57d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38574/0 (socket says 172.21.15.1:38574) 2024-03-06T07:21:18.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c98f9640 1 -- 172.21.15.1:0/3548123063 learned_addr learned my addr 172.21.15.1:0/3548123063 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:18.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5ca8fb640 1 --2- 172.21.15.1:0/3548123063 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5c4156cd0 0x7fc5c4157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c90f8640 1 -- 172.21.15.1:0/3548123063 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818657226 0 0) 0x7fc5c40c3300 con 0x7fc5c40c6730 2024-03-06T07:21:18.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c90f8640 1 -- 172.21.15.1:0/3548123063 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b0003520 con 0x7fc5c40c6730 2024-03-06T07:21:18.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c90f8640 1 -- 172.21.15.1:0/3548123063 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336562734 0 0) 0x7fc5c40c35a0 con 0x7fc5c4152690 2024-03-06T07:21:18.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c90f8640 1 -- 172.21.15.1:0/3548123063 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c40c3300 con 0x7fc5c4152690 2024-03-06T07:21:18.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c90f8640 1 -- 172.21.15.1:0/3548123063 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3604717359 0 0) 0x7fc5b0003520 con 0x7fc5c40c6730 2024-03-06T07:21:18.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c90f8640 1 -- 172.21.15.1:0/3548123063 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5c4156cd0 msgr2=0x7fc5c4157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:18.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c90f8640 1 --2- 172.21.15.1:0/3548123063 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5c4156cd0 0x7fc5c4157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c90f8640 1 -- 172.21.15.1:0/3548123063 >> v1:172.21.15.98:6789/0 conn(0x7fc5c4152690 legacy=0x7fc5c4155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c90f8640 1 -- 172.21.15.1:0/3548123063 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c40c3b60 con 0x7fc5c40c6730 2024-03-06T07:21:18.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c90f8640 1 -- 172.21.15.1:0/3548123063 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fc5b4001670 con 0x7fc5c40c6730 2024-03-06T07:21:18.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c90f8640 1 -- 172.21.15.1:0/3548123063 >> v1:172.21.15.1:6789/0 conn(0x7fc5c40c6730 legacy=0x7fc5c40c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c90f8640 1 --2- 172.21.15.1:0/3548123063 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b0003b70 0x7fc5b0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c90f8640 1 --2- 172.21.15.1:0/3548123063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5b0004800 0x7fc5b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c90f8640 1 --2- 172.21.15.1:0/3548123063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b00051c0 0x7fc5b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c90f8640 1 -- 172.21.15.1:0/3548123063 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c40c3880 con 0x7fc5b0004800 2024-03-06T07:21:18.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c90f8640 1 -- 172.21.15.1:0/3548123063 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b000c090 con 0x7fc5b00051c0 2024-03-06T07:21:18.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5c90f8640 1 -- 172.21.15.1:0/3548123063 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b000c290 con 0x7fc5b0003b70 2024-03-06T07:21:18.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.403+0000 7fc5ca8fb640 1 --2- 172.21.15.1:0/3548123063 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5c4156cd0 0x7fc5c4157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:18.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.404+0000 7fc5c98f9640 1 --2- 172.21.15.1:0/3548123063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5b0004800 0x7fc5b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.404+0000 7fc5ca0fa640 1 --2- 172.21.15.1:0/3548123063 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b0003b70 0x7fc5b0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.404+0000 7fc5ca8fb640 1 --2- 172.21.15.1:0/3548123063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b00051c0 0x7fc5b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.404+0000 7fc5c98f9640 1 -- 172.21.15.1:0/3548123063 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b0003b70 msgr2=0x7fc5b0003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:18.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.404+0000 7fc5c98f9640 1 --2- 172.21.15.1:0/3548123063 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b0003b70 0x7fc5b0003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.404+0000 7fc5c98f9640 1 -- 172.21.15.1:0/3548123063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b00051c0 msgr2=0x7fc5b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:18.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.404+0000 7fc5c98f9640 1 --2- 172.21.15.1:0/3548123063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b00051c0 0x7fc5b000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.404+0000 7fc5c98f9640 1 -- 172.21.15.1:0/3548123063 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b0003520 con 0x7fc5b0004800 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.404+0000 7fc5ca8fb640 1 --2- 172.21.15.1:0/3548123063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b00051c0 0x7fc5b000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.405+0000 7fc5c98f9640 1 --2- 172.21.15.1:0/3548123063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5b0004800 0x7fc5b0004c80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4007fb0 tx=0x7fc5b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef8924486b3a64f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.405+0000 7fc5c90f8640 1 -- 172.21.15.1:0/3548123063 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4007c00 con 0x7fc5b0004800 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.405+0000 7fc5c90f8640 1 -- 172.21.15.1:0/3548123063 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b4030070 con 0x7fc5b0004800 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.405+0000 7fc5c90f8640 1 -- 172.21.15.1:0/3548123063 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4005d30 con 0x7fc5b0004800 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.405+0000 7fc5cc35e640 1 -- 172.21.15.1:0/3548123063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5b0004800 msgr2=0x7fc5b0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.405+0000 7fc5cc35e640 1 --2- 172.21.15.1:0/3548123063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5b0004800 0x7fc5b0004c80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4007fb0 tx=0x7fc5b0005830 comp rx=0 tx=0).stop 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.405+0000 7fc5ca8fb640 1 -- 172.21.15.1:0/3548123063 reap_dead start 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.405+0000 7fc5cc35e640 1 -- 172.21.15.1:0/3548123063 shutdown_connections 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.405+0000 7fc5cc35e640 1 -- 172.21.15.1:0/3548123063 >> 172.21.15.1:0/3548123063 conn(0x7fc5c406b410 msgr2=0x7fc5c406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.405+0000 7fc5cc35e640 1 -- 172.21.15.1:0/3548123063 shutdown_connections 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.405+0000 7fc5cc35e640 1 -- 172.21.15.1:0/3548123063 wait complete. 2024-03-06T07:21:18.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.406+0000 7fc5cc35e640 1 Processor -- start 2024-03-06T07:21:18.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.406+0000 7fc5cc35e640 1 -- start start 2024-03-06T07:21:18.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.407+0000 7fc5cc35e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b0004800 0x7fc5c40ad1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.407+0000 7fc5cc35e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5c406d400 0x7fc5c40709b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.407+0000 7fc5cc35e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c41ea990 0x7fc5c40aba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.407+0000 7fc5cc35e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c40c6060 con 0x7fc5c41ea990 2024-03-06T07:21:18.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.407+0000 7fc5cc35e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c40c5ee0 con 0x7fc5b0004800 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.407+0000 7fc5cc35e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c4152a70 con 0x7fc5c406d400 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.407+0000 7fc5ca8fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c41ea990 0x7fc5c40aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.407+0000 7fc5ca8fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c41ea990 0x7fc5c40aba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37338/0 (socket says 172.21.15.1:37338) 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.407+0000 7fc5ca8fb640 1 -- 172.21.15.1:0/1871882784 learned_addr learned my addr 172.21.15.1:0/1871882784 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.407+0000 7fc5ca0fa640 1 --2- 172.21.15.1:0/1871882784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b0004800 0x7fc5c40ad1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.407+0000 7fc5ca8fb640 1 -- 172.21.15.1:0/1871882784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5c406d400 msgr2=0x7fc5c40709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.407+0000 7fc5ca8fb640 1 --2- 172.21.15.1:0/1871882784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5c406d400 0x7fc5c40709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.407+0000 7fc5ca8fb640 1 -- 172.21.15.1:0/1871882784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b0004800 msgr2=0x7fc5c40ad1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.407+0000 7fc5ca8fb640 1 --2- 172.21.15.1:0/1871882784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b0004800 0x7fc5c40ad1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.407+0000 7fc5ca8fb640 1 -- 172.21.15.1:0/1871882784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c40ac730 con 0x7fc5c41ea990 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.408+0000 7fc5ca8fb640 1 --2- 172.21.15.1:0/1871882784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c41ea990 0x7fc5c40aba70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40c4f30 tx=0x7fc5bc00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.408+0000 7fc5baffd640 1 -- 172.21.15.1:0/1871882784 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc01c070 con 0x7fc5c41ea990 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.408+0000 7fc5baffd640 1 -- 172.21.15.1:0/1871882784 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5bc017600 con 0x7fc5c41ea990 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.408+0000 7fc5cc35e640 1 -- 172.21.15.1:0/1871882784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c41eec70 con 0x7fc5c41ea990 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.408+0000 7fc5baffd640 1 -- 172.21.15.1:0/1871882784 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc0179d0 con 0x7fc5c41ea990 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.408+0000 7fc5cc35e640 1 -- 172.21.15.1:0/1871882784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c4070ef0 con 0x7fc5c41ea990 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.409+0000 7fc5cc35e640 1 -- 172.21.15.1:0/1871882784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c40a8e70 con 0x7fc5c41ea990 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.409+0000 7fc5baffd640 1 -- 172.21.15.1:0/1871882784 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5bc01f070 con 0x7fc5c41ea990 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.410+0000 7fc5baffd640 1 --2- 172.21.15.1:0/1871882784 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5a806b4d0 0x7fc5a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.410+0000 7fc5baffd640 1 -- 172.21.15.1:0/1871882784 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc5bc08a1d0 con 0x7fc5c41ea990 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.410+0000 7fc5ca0fa640 1 --2- 172.21.15.1:0/1871882784 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5a806b4d0 0x7fc5a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.410+0000 7fc5ca0fa640 1 --2- 172.21.15.1:0/1871882784 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5a806b4d0 0x7fc5a806d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40c6d10 tx=0x7fc5c0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.414+0000 7fc5baffd640 1 -- 172.21.15.1:0/1871882784 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5c40a8e70 con 0x7fc5c41ea990 2024-03-06T07:21:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.666+0000 7fc5cc35e640 1 -- 172.21.15.1:0/1871882784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fc5c41eee40 con 0x7fc5c41ea990 2024-03-06T07:21:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.669+0000 7fc5baffd640 1 -- 172.21.15.1:0/1871882784 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7fc5c41eee40 con 0x7fc5c41ea990 2024-03-06T07:21:18.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.671+0000 7fc5cc35e640 1 -- 172.21.15.1:0/1871882784 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5a806b4d0 msgr2=0x7fc5a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.671+0000 7fc5cc35e640 1 --2- 172.21.15.1:0/1871882784 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5a806b4d0 0x7fc5a806d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40c6d10 tx=0x7fc5c0009040 comp rx=0 tx=0).stop 2024-03-06T07:21:18.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.671+0000 7fc5cc35e640 1 -- 172.21.15.1:0/1871882784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c41ea990 msgr2=0x7fc5c40aba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.671+0000 7fc5cc35e640 1 --2- 172.21.15.1:0/1871882784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c41ea990 0x7fc5c40aba70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40c4f30 tx=0x7fc5bc00f790 comp rx=0 tx=0).stop 2024-03-06T07:21:18.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.671+0000 7fc5cc35e640 1 -- 172.21.15.1:0/1871882784 shutdown_connections 2024-03-06T07:21:18.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.671+0000 7fc5cc35e640 1 --2- 172.21.15.1:0/1871882784 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5a806b4d0 0x7fc5a806d990 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.671+0000 7fc5cc35e640 1 --2- 172.21.15.1:0/1871882784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c41ea990 0x7fc5c40aba70 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.671+0000 7fc5cc35e640 1 --2- 172.21.15.1:0/1871882784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5c406d400 0x7fc5c40709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.671+0000 7fc5cc35e640 1 --2- 172.21.15.1:0/1871882784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b0004800 0x7fc5c40ad1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.671+0000 7fc5cc35e640 1 -- 172.21.15.1:0/1871882784 >> 172.21.15.1:0/1871882784 conn(0x7fc5c406b410 msgr2=0x7fc5c40c57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:18.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.671+0000 7fc5cc35e640 1 -- 172.21.15.1:0/1871882784 shutdown_connections 2024-03-06T07:21:18.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.671+0000 7fc5cc35e640 1 -- 172.21.15.1:0/1871882784 wait complete. 2024-03-06T07:21:18.684 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-06T07:21:18.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.784+0000 7f24d3f5e640 1 Processor -- start 2024-03-06T07:21:18.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.784+0000 7f24d3f5e640 1 -- start start 2024-03-06T07:21:18.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d3f5e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f24cc152920 0x7f24cc06aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d3f5e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24cc06b530 0x7f24cc0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d3f5e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24cc158910 con 0x7f24cc0c5d90 2024-03-06T07:21:18.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d3f5e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24cc152d00 con 0x7f24cc152920 2024-03-06T07:21:18.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d3f5e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc1596d0 con 0x7f24cc06b530 2024-03-06T07:21:18.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d24fb640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f24cc0c5d90 0x7f24cc155220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38578/0 (socket says 172.21.15.1:38578) 2024-03-06T07:21:18.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d24fb640 1 -- 172.21.15.1:0/534420406 learned_addr learned my addr 172.21.15.1:0/534420406 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:18.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d0cf8640 1 -- 172.21.15.1:0/534420406 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535507070 0 0) 0x7f24cc158910 con 0x7f24cc0c5d90 2024-03-06T07:21:18.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d0cf8640 1 -- 172.21.15.1:0/534420406 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24b4003520 con 0x7f24cc0c5d90 2024-03-06T07:21:18.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d1cfa640 1 --2- 172.21.15.1:0/534420406 >> v2:172.21.15.98:3300/0 conn(0x7f24cc152920 0x7f24cc06aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d14f9640 1 --2- 172.21.15.1:0/534420406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24cc06b530 0x7f24cc0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d0cf8640 1 -- 172.21.15.1:0/534420406 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2475208440 0 0) 0x7f24b4003520 con 0x7f24cc0c5d90 2024-03-06T07:21:18.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d0cf8640 1 -- 172.21.15.1:0/534420406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24cc06b530 msgr2=0x7f24cc0c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:18.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d0cf8640 1 --2- 172.21.15.1:0/534420406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24cc06b530 0x7f24cc0c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d0cf8640 1 -- 172.21.15.1:0/534420406 >> v2:172.21.15.98:3300/0 conn(0x7f24cc152920 msgr2=0x7f24cc06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:18.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d0cf8640 1 --2- 172.21.15.1:0/534420406 >> v2:172.21.15.98:3300/0 conn(0x7f24cc152920 0x7f24cc06aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d0cf8640 1 -- 172.21.15.1:0/534420406 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24cc158bf0 con 0x7f24cc0c5d90 2024-03-06T07:21:18.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.785+0000 7f24d0cf8640 1 -- 172.21.15.1:0/534420406 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f24bc0019f0 con 0x7f24cc0c5d90 2024-03-06T07:21:18.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.786+0000 7f24d0cf8640 1 -- 172.21.15.1:0/534420406 >> v1:172.21.15.1:6789/0 conn(0x7f24cc0c5d90 legacy=0x7f24cc155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.786+0000 7f24d0cf8640 1 --2- 172.21.15.1:0/534420406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24cc06b530 0x7f24b40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.786+0000 7f24d0cf8640 1 --2- 172.21.15.1:0/534420406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24b4003f30 0x7f24b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.786+0000 7f24d0cf8640 1 --2- 172.21.15.1:0/534420406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24b4004d40 0x7f24b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.786+0000 7f24d0cf8640 1 -- 172.21.15.1:0/534420406 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc152d00 con 0x7f24b4004d40 2024-03-06T07:21:18.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.786+0000 7f24d0cf8640 1 -- 172.21.15.1:0/534420406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc1596d0 con 0x7f24b4003f30 2024-03-06T07:21:18.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.786+0000 7f24d0cf8640 1 -- 172.21.15.1:0/534420406 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b400ba10 con 0x7f24cc06b530 2024-03-06T07:21:18.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.786+0000 7f24d24fb640 1 --2- 172.21.15.1:0/534420406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24b4004d40 0x7f24b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.786+0000 7f24d14f9640 1 --2- 172.21.15.1:0/534420406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24b4003f30 0x7f24b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.786+0000 7f24d1cfa640 1 --2- 172.21.15.1:0/534420406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24cc06b530 0x7f24b40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.786+0000 7f24d24fb640 1 -- 172.21.15.1:0/534420406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24cc06b530 msgr2=0x7f24b40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:18.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.786+0000 7f24d24fb640 1 --2- 172.21.15.1:0/534420406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24cc06b530 0x7f24b40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.786+0000 7f24d24fb640 1 -- 172.21.15.1:0/534420406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24b4003f30 msgr2=0x7f24b4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:18.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.786+0000 7f24d24fb640 1 --2- 172.21.15.1:0/534420406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24b4003f30 0x7f24b4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.786+0000 7f24d24fb640 1 -- 172.21.15.1:0/534420406 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b4003520 con 0x7f24b4004d40 2024-03-06T07:21:18.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.787+0000 7f24d24fb640 1 --2- 172.21.15.1:0/534420406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24b4004d40 0x7f24b40051c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00a880 tx=0x7f24b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ed033ae12b8c20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.787+0000 7f24d0cf8640 1 -- 172.21.15.1:0/534420406 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc00a4a0 con 0x7f24b4004d40 2024-03-06T07:21:18.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.787+0000 7f24d0cf8640 1 -- 172.21.15.1:0/534420406 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24bc034650 con 0x7f24b4004d40 2024-03-06T07:21:18.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.787+0000 7f24d0cf8640 1 -- 172.21.15.1:0/534420406 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc034a80 con 0x7f24b4004d40 2024-03-06T07:21:18.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.787+0000 7f24d3f5e640 1 -- 172.21.15.1:0/534420406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24b4004d40 msgr2=0x7f24b40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.787+0000 7f24d3f5e640 1 --2- 172.21.15.1:0/534420406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24b4004d40 0x7f24b40051c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00a880 tx=0x7f24b4005830 comp rx=0 tx=0).stop 2024-03-06T07:21:18.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.787+0000 7f24d24fb640 1 -- 172.21.15.1:0/534420406 reap_dead start 2024-03-06T07:21:18.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.787+0000 7f24d3f5e640 1 -- 172.21.15.1:0/534420406 shutdown_connections 2024-03-06T07:21:18.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.787+0000 7f24d3f5e640 1 -- 172.21.15.1:0/534420406 >> 172.21.15.1:0/534420406 conn(0x7f24cc0a9700 msgr2=0x7f24cc0a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:18.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.788+0000 7f24d3f5e640 1 -- 172.21.15.1:0/534420406 shutdown_connections 2024-03-06T07:21:18.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.788+0000 7f24d3f5e640 1 -- 172.21.15.1:0/534420406 wait complete. 2024-03-06T07:21:18.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.789+0000 7f24d3f5e640 1 Processor -- start 2024-03-06T07:21:18.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.789+0000 7f24d3f5e640 1 -- start start 2024-03-06T07:21:18.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.789+0000 7f24d3f5e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24cc0c4300 0x7f24cc1538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.789+0000 7f24d3f5e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc1eb970 0x7f24cc0a7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.789+0000 7f24d3f5e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24cc14bab0 0x7f24cc14bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.789+0000 7f24d3f5e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc0c6380 con 0x7f24cc14bab0 2024-03-06T07:21:18.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.789+0000 7f24d3f5e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc159240 con 0x7f24cc1eb970 2024-03-06T07:21:18.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.789+0000 7f24d3f5e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc0c5850 con 0x7f24cc0c4300 2024-03-06T07:21:18.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.789+0000 7f24d24fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24cc14bab0 0x7f24cc14bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.789+0000 7f24d24fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24cc14bab0 0x7f24cc14bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37356/0 (socket says 172.21.15.1:37356) 2024-03-06T07:21:18.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.789+0000 7f24d24fb640 1 -- 172.21.15.1:0/1137304204 learned_addr learned my addr 172.21.15.1:0/1137304204 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:18.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.790+0000 7f24d14f9640 1 --2- 172.21.15.1:0/1137304204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc1eb970 0x7f24cc0a7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.790+0000 7f24d1cfa640 1 --2- 172.21.15.1:0/1137304204 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24cc0c4300 0x7f24cc1538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.790+0000 7f24d24fb640 1 -- 172.21.15.1:0/1137304204 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24cc0c4300 msgr2=0x7f24cc1538c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.790+0000 7f24d24fb640 1 --2- 172.21.15.1:0/1137304204 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24cc0c4300 0x7f24cc1538c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.790+0000 7f24d24fb640 1 -- 172.21.15.1:0/1137304204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc1eb970 msgr2=0x7f24cc0a7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.790+0000 7f24d24fb640 1 --2- 172.21.15.1:0/1137304204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc1eb970 0x7f24cc0a7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.790+0000 7f24d24fb640 1 -- 172.21.15.1:0/1137304204 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24cc0a8e50 con 0x7f24cc14bab0 2024-03-06T07:21:18.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.790+0000 7f24d14f9640 1 --2- 172.21.15.1:0/1137304204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc1eb970 0x7f24cc0a7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:18.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.790+0000 7f24d24fb640 1 --2- 172.21.15.1:0/1137304204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24cc14bab0 0x7f24cc14bf30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f24bc02cc70 tx=0x7f24bc001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.790+0000 7f24ca7fc640 1 -- 172.21.15.1:0/1137304204 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc045070 con 0x7f24cc14bab0 2024-03-06T07:21:18.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.790+0000 7f24ca7fc640 1 -- 172.21.15.1:0/1137304204 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24bc03a040 con 0x7f24cc14bab0 2024-03-06T07:21:18.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.790+0000 7f24d3f5e640 1 -- 172.21.15.1:0/1137304204 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24cc0a9130 con 0x7f24cc14bab0 2024-03-06T07:21:18.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.790+0000 7f24ca7fc640 1 -- 172.21.15.1:0/1137304204 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc034c40 con 0x7f24cc14bab0 2024-03-06T07:21:18.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.790+0000 7f24d3f5e640 1 -- 172.21.15.1:0/1137304204 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24cc0a7580 con 0x7f24cc14bab0 2024-03-06T07:21:18.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.791+0000 7f24d3f5e640 1 -- 172.21.15.1:0/1137304204 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24cc067fd0 con 0x7f24cc14bab0 2024-03-06T07:21:18.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.792+0000 7f24ca7fc640 1 -- 172.21.15.1:0/1137304204 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24bc048070 con 0x7f24cc14bab0 2024-03-06T07:21:18.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.792+0000 7f24ca7fc640 1 --2- 172.21.15.1:0/1137304204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f24b806b4d0 0x7f24b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.792+0000 7f24ca7fc640 1 -- 172.21.15.1:0/1137304204 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f24bc0ab9f0 con 0x7f24cc14bab0 2024-03-06T07:21:18.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.792+0000 7f24d1cfa640 1 --2- 172.21.15.1:0/1137304204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f24b806b4d0 0x7f24b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.793+0000 7f24d1cfa640 1 --2- 172.21.15.1:0/1137304204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f24b806b4d0 0x7f24b806d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009b80 tx=0x7f24c4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:18.796+0000 7f24ca7fc640 1 -- 172.21.15.1:0/1137304204 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24bc07aac0 con 0x7f24cc14bab0 2024-03-06T07:21:19.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.049+0000 7f24d3f5e640 1 -- 172.21.15.1:0/1137304204 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f24cc0701b0 con 0x7f24cc14bab0 2024-03-06T07:21:19.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.050+0000 7f24ca7fc640 1 -- 172.21.15.1:0/1137304204 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f24bc07a3c0 con 0x7f24cc14bab0 2024-03-06T07:21:19.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.053+0000 7f24d3f5e640 1 -- 172.21.15.1:0/1137304204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f24b806b4d0 msgr2=0x7f24b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.053+0000 7f24d3f5e640 1 --2- 172.21.15.1:0/1137304204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f24b806b4d0 0x7f24b806d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009b80 tx=0x7f24c4008040 comp rx=0 tx=0).stop 2024-03-06T07:21:19.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.053+0000 7f24d3f5e640 1 -- 172.21.15.1:0/1137304204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24cc14bab0 msgr2=0x7f24cc14bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.053+0000 7f24d3f5e640 1 --2- 172.21.15.1:0/1137304204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24cc14bab0 0x7f24cc14bf30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f24bc02cc70 tx=0x7f24bc001dc0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.053+0000 7f24d3f5e640 1 -- 172.21.15.1:0/1137304204 shutdown_connections 2024-03-06T07:21:19.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.053+0000 7f24d3f5e640 1 --2- 172.21.15.1:0/1137304204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f24b806b4d0 0x7f24b806d990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.053+0000 7f24d3f5e640 1 --2- 172.21.15.1:0/1137304204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24cc14bab0 0x7f24cc14bf30 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.053+0000 7f24d3f5e640 1 --2- 172.21.15.1:0/1137304204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc1eb970 0x7f24cc0a7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.053+0000 7f24d3f5e640 1 --2- 172.21.15.1:0/1137304204 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24cc0c4300 0x7f24cc1538c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.053+0000 7f24d3f5e640 1 -- 172.21.15.1:0/1137304204 >> 172.21.15.1:0/1137304204 conn(0x7f24cc0a9700 msgr2=0x7f24cc0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:19.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.053+0000 7f24d3f5e640 1 -- 172.21.15.1:0/1137304204 shutdown_connections 2024-03-06T07:21:19.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.053+0000 7f24d3f5e640 1 -- 172.21.15.1:0/1137304204 wait complete. 2024-03-06T07:21:19.066 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-06T07:21:19.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.167+0000 7f779dca4640 1 Processor -- start 2024-03-06T07:21:19.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.167+0000 7f779dca4640 1 -- start start 2024-03-06T07:21:19.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.168+0000 7f779dca4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77980c8d10 0x7f77980c90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.168+0000 7f779dca4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7798155dc0 0x7f779815fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.168+0000 7f779dca4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7798155200 con 0x7f77981590f0 2024-03-06T07:21:19.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.168+0000 7f779dca4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77980c33e0 con 0x7f7798155dc0 2024-03-06T07:21:19.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.168+0000 7f779dca4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77980c36c0 con 0x7f77980c8d10 2024-03-06T07:21:19.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.168+0000 7f77977fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77980c8d10 0x7f77980c90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.168+0000 7f7797fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7798155dc0 0x7f779815fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.168+0000 7f7796ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f77981590f0 0x7f77981556b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:51410/0 (socket says 172.21.15.1:51410) 2024-03-06T07:21:19.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.168+0000 7f7797fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7798155dc0 0x7f779815fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:46646/0 (socket says 172.21.15.1:46646) 2024-03-06T07:21:19.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.168+0000 7f7796ffd640 1 -- 172.21.15.1:0/1154375271 learned_addr learned my addr 172.21.15.1:0/1154375271 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:19.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.168+0000 7f77977fe640 1 -- 172.21.15.1:0/1154375271 >> v2:172.21.15.98:3300/0 conn(0x7f7798155dc0 msgr2=0x7f779815fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:19.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.168+0000 7f77977fe640 1 --2- 172.21.15.1:0/1154375271 >> v2:172.21.15.98:3300/0 conn(0x7f7798155dc0 0x7f779815fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.168+0000 7f77977fe640 1 -- 172.21.15.1:0/1154375271 >> v1:172.21.15.98:6789/0 conn(0x7f77981590f0 legacy=0x7f77981556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.168+0000 7f77977fe640 1 -- 172.21.15.1:0/1154375271 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77980c39a0 con 0x7f77980c8d10 2024-03-06T07:21:19.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.168+0000 7f7797fff640 1 --2- 172.21.15.1:0/1154375271 >> v2:172.21.15.98:3300/0 conn(0x7f7798155dc0 0x7f779815fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:19.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.169+0000 7f77977fe640 1 --2- 172.21.15.1:0/1154375271 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77980c8d10 0x7f77980c90f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f778c0097a0 tx=0x7f778c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91609f42f76fb308 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.169+0000 7f77967fc640 1 -- 172.21.15.1:0/1154375271 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c035020 con 0x7f77980c8d10 2024-03-06T07:21:19.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.169+0000 7f77967fc640 1 -- 172.21.15.1:0/1154375271 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778c031830 con 0x7f77980c8d10 2024-03-06T07:21:19.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.169+0000 7f77967fc640 1 -- 172.21.15.1:0/1154375271 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c031c00 con 0x7f77980c8d10 2024-03-06T07:21:19.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.170+0000 7f779dca4640 1 -- 172.21.15.1:0/1154375271 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77980c8d10 msgr2=0x7f77980c90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.170+0000 7f779dca4640 1 --2- 172.21.15.1:0/1154375271 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77980c8d10 0x7f77980c90f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f778c0097a0 tx=0x7f778c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.170+0000 7f779dca4640 1 -- 172.21.15.1:0/1154375271 shutdown_connections 2024-03-06T07:21:19.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.170+0000 7f779dca4640 1 --2- 172.21.15.1:0/1154375271 >> v2:172.21.15.98:3300/0 conn(0x7f7798155dc0 0x7f779815fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.170+0000 7f779dca4640 1 --2- 172.21.15.1:0/1154375271 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77980c8d10 0x7f77980c90f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.170+0000 7f779dca4640 1 -- 172.21.15.1:0/1154375271 >> 172.21.15.1:0/1154375271 conn(0x7f779806b3e0 msgr2=0x7f779806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:19.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.170+0000 7f779dca4640 1 -- 172.21.15.1:0/1154375271 shutdown_connections 2024-03-06T07:21:19.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.170+0000 7f779dca4640 1 -- 172.21.15.1:0/1154375271 wait complete. 2024-03-06T07:21:19.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.171+0000 7f779dca4640 1 Processor -- start 2024-03-06T07:21:19.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.171+0000 7f779dca4640 1 -- start start 2024-03-06T07:21:19.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.172+0000 7f779dca4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77980c8d10 0x7f77981eed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.172+0000 7f779dca4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7798155dc0 0x7f77981577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.172+0000 7f779dca4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77981590f0 0x7f77980ab040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.172+0000 7f779dca4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77981529e0 con 0x7f7798155dc0 2024-03-06T07:21:19.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.172+0000 7f779dca4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7798159550 con 0x7f77980c8d10 2024-03-06T07:21:19.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.172+0000 7f779dca4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7798152b60 con 0x7f77981590f0 2024-03-06T07:21:19.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.172+0000 7f7796ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7798155dc0 0x7f77981577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.172+0000 7f7796ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7798155dc0 0x7f77981577a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37364/0 (socket says 172.21.15.1:37364) 2024-03-06T07:21:19.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.172+0000 7f7796ffd640 1 -- 172.21.15.1:0/523672408 learned_addr learned my addr 172.21.15.1:0/523672408 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:19.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.172+0000 7f77977fe640 1 --2- 172.21.15.1:0/523672408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77980c8d10 0x7f77981eed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.172+0000 7f7797fff640 1 --2- 172.21.15.1:0/523672408 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77981590f0 0x7f77980ab040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.172+0000 7f7796ffd640 1 -- 172.21.15.1:0/523672408 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77981590f0 msgr2=0x7f77980ab040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.172+0000 7f7796ffd640 1 --2- 172.21.15.1:0/523672408 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77981590f0 0x7f77980ab040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.172+0000 7f7796ffd640 1 -- 172.21.15.1:0/523672408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77980c8d10 msgr2=0x7f77981eed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.172+0000 7f7796ffd640 1 --2- 172.21.15.1:0/523672408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77980c8d10 0x7f77981eed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.172+0000 7f7796ffd640 1 -- 172.21.15.1:0/523672408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77980aab40 con 0x7f7798155dc0 2024-03-06T07:21:19.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.173+0000 7f7796ffd640 1 --2- 172.21.15.1:0/523672408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7798155dc0 0x7f77981577a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f778800c400 tx=0x7f778800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.173+0000 7f777ffff640 1 -- 172.21.15.1:0/523672408 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7788017070 con 0x7f7798155dc0 2024-03-06T07:21:19.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.173+0000 7f777ffff640 1 -- 172.21.15.1:0/523672408 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7788005ce0 con 0x7f7798155dc0 2024-03-06T07:21:19.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.173+0000 7f777ffff640 1 -- 172.21.15.1:0/523672408 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77880072f0 con 0x7f7798155dc0 2024-03-06T07:21:19.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.173+0000 7f779dca4640 1 -- 172.21.15.1:0/523672408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7798158150 con 0x7f7798155dc0 2024-03-06T07:21:19.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.173+0000 7f779dca4640 1 -- 172.21.15.1:0/523672408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7798154940 con 0x7f7798155dc0 2024-03-06T07:21:19.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.174+0000 7f779dca4640 1 -- 172.21.15.1:0/523672408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7798062c00 con 0x7f7798155dc0 2024-03-06T07:21:19.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.174+0000 7f777ffff640 1 -- 172.21.15.1:0/523672408 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f778801a070 con 0x7f7798155dc0 2024-03-06T07:21:19.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.174+0000 7f777ffff640 1 --2- 172.21.15.1:0/523672408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f778406b4d0 0x7f778406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.175+0000 7f777ffff640 1 -- 172.21.15.1:0/523672408 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f7788089390 con 0x7f7798155dc0 2024-03-06T07:21:19.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.175+0000 7f77977fe640 1 --2- 172.21.15.1:0/523672408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f778406b4d0 0x7f778406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.175+0000 7f77977fe640 1 --2- 172.21.15.1:0/523672408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f778406b4d0 0x7f778406d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f778c036000 tx=0x7f778c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.179+0000 7f777ffff640 1 -- 172.21.15.1:0/523672408 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7798062c00 con 0x7f7798155dc0 2024-03-06T07:21:19.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.429+0000 7f779dca4640 1 -- 172.21.15.1:0/523672408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f7798067fd0 con 0x7f7798155dc0 2024-03-06T07:21:19.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.431+0000 7f777ffff640 1 -- 172.21.15.1:0/523672408 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7788057d00 con 0x7f7798155dc0 2024-03-06T07:21:19.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.433+0000 7f779dca4640 1 -- 172.21.15.1:0/523672408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f778406b4d0 msgr2=0x7f778406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.433+0000 7f779dca4640 1 --2- 172.21.15.1:0/523672408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f778406b4d0 0x7f778406d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f778c036000 tx=0x7f778c03c040 comp rx=0 tx=0).stop 2024-03-06T07:21:19.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.433+0000 7f779dca4640 1 -- 172.21.15.1:0/523672408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7798155dc0 msgr2=0x7f77981577a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.433+0000 7f779dca4640 1 --2- 172.21.15.1:0/523672408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7798155dc0 0x7f77981577a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f778800c400 tx=0x7f778800c800 comp rx=0 tx=0).stop 2024-03-06T07:21:19.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.433+0000 7f779dca4640 1 -- 172.21.15.1:0/523672408 shutdown_connections 2024-03-06T07:21:19.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.433+0000 7f779dca4640 1 --2- 172.21.15.1:0/523672408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f778406b4d0 0x7f778406d990 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.433+0000 7f779dca4640 1 --2- 172.21.15.1:0/523672408 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77981590f0 0x7f77980ab040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.433+0000 7f779dca4640 1 --2- 172.21.15.1:0/523672408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7798155dc0 0x7f77981577a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.433+0000 7f779dca4640 1 --2- 172.21.15.1:0/523672408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77980c8d10 0x7f77981eed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.433+0000 7f779dca4640 1 -- 172.21.15.1:0/523672408 >> 172.21.15.1:0/523672408 conn(0x7f779806b3e0 msgr2=0x7f7798150ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.433+0000 7f779dca4640 1 -- 172.21.15.1:0/523672408 shutdown_connections 2024-03-06T07:21:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.433+0000 7f779dca4640 1 -- 172.21.15.1:0/523672408 wait complete. 2024-03-06T07:21:19.446 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-06T07:21:19.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.547+0000 7fd8edc84640 1 Processor -- start 2024-03-06T07:21:19.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.547+0000 7fd8edc84640 1 -- start start 2024-03-06T07:21:19.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.547+0000 7fd8edc84640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8e80c3eb0 0x7fd8e80c7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.547+0000 7fd8edc84640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e80c71c0 con 0x7fd8e80c7940 2024-03-06T07:21:19.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.547+0000 7fd8edc84640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e80c4e50 con 0x7fd8e806d060 2024-03-06T07:21:19.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.547+0000 7fd8edc84640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e80c59b0 con 0x7fd8e80c3eb0 2024-03-06T07:21:19.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.548+0000 7fd8e77fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8e80c3eb0 0x7fd8e80c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.548+0000 7fd8e7fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd8e80c7940 0x7fd8e815fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38580/0 (socket says 172.21.15.1:38580) 2024-03-06T07:21:19.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.548+0000 7fd8e7fff640 1 -- 172.21.15.1:0/3715151410 learned_addr learned my addr 172.21.15.1:0/3715151410 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:19.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.548+0000 7fd8e77fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8e80c3eb0 0x7fd8e80c7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37370/0 (socket says 172.21.15.1:37370) 2024-03-06T07:21:19.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.548+0000 7fd8e77fe640 1 -- 172.21.15.1:0/3715151410 >> v1:172.21.15.98:6789/0 conn(0x7fd8e806d060 legacy=0x7fd8e81558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:19.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.548+0000 7fd8e77fe640 1 -- 172.21.15.1:0/3715151410 >> v1:172.21.15.1:6789/0 conn(0x7fd8e80c7940 legacy=0x7fd8e815fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.548+0000 7fd8e77fe640 1 -- 172.21.15.1:0/3715151410 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e80c6ee0 con 0x7fd8e80c3eb0 2024-03-06T07:21:19.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.548+0000 7fd8e67fc640 1 -- 172.21.15.1:0/3715151410 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270520938 0 0) 0x7fd8e80c71c0 con 0x7fd8e80c7940 2024-03-06T07:21:19.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.549+0000 7fd8e77fe640 1 --2- 172.21.15.1:0/3715151410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e80c3eb0 0x7fd8e80c7400 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd8d80097a0 tx=0x7fd8d802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bede8d208e8f7f84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.549+0000 7fd8e67fc640 1 -- 172.21.15.1:0/3715151410 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8038070 con 0x7fd8e80c3eb0 2024-03-06T07:21:19.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.549+0000 7fd8e67fc640 1 -- 172.21.15.1:0/3715151410 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d8031700 con 0x7fd8e80c3eb0 2024-03-06T07:21:19.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.549+0000 7fd8e67fc640 1 -- 172.21.15.1:0/3715151410 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8031ad0 con 0x7fd8e80c3eb0 2024-03-06T07:21:19.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.549+0000 7fd8edc84640 1 -- 172.21.15.1:0/3715151410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e80c3eb0 msgr2=0x7fd8e80c7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.549+0000 7fd8edc84640 1 --2- 172.21.15.1:0/3715151410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e80c3eb0 0x7fd8e80c7400 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd8d80097a0 tx=0x7fd8d802b7a0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.549+0000 7fd8edc84640 1 -- 172.21.15.1:0/3715151410 shutdown_connections 2024-03-06T07:21:19.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.549+0000 7fd8edc84640 1 --2- 172.21.15.1:0/3715151410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e80c3eb0 0x7fd8e80c7400 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.549+0000 7fd8edc84640 1 -- 172.21.15.1:0/3715151410 >> 172.21.15.1:0/3715151410 conn(0x7fd8e8066d00 msgr2=0x7fd8e8067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:19.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.549+0000 7fd8edc84640 1 -- 172.21.15.1:0/3715151410 shutdown_connections 2024-03-06T07:21:19.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.549+0000 7fd8edc84640 1 -- 172.21.15.1:0/3715151410 wait complete. 2024-03-06T07:21:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.550+0000 7fd8edc84640 1 Processor -- start 2024-03-06T07:21:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.551+0000 7fd8edc84640 1 -- start start 2024-03-06T07:21:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.551+0000 7fd8edc84640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e806d060 0x7fd8e80c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.551+0000 7fd8edc84640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e80c3eb0 0x7fd8e8158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.551+0000 7fd8edc84640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e80c7940 0x7fd8e8156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.551+0000 7fd8edc84640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e8155420 con 0x7fd8e80c7940 2024-03-06T07:21:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.551+0000 7fd8edc84640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e80c4c00 con 0x7fd8e80c3eb0 2024-03-06T07:21:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.551+0000 7fd8edc84640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e8155fc0 con 0x7fd8e806d060 2024-03-06T07:21:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.551+0000 7fd8e7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e80c7940 0x7fd8e8156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.551+0000 7fd8e7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e80c7940 0x7fd8e8156560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37380/0 (socket says 172.21.15.1:37380) 2024-03-06T07:21:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.551+0000 7fd8e7fff640 1 -- 172.21.15.1:0/287678415 learned_addr learned my addr 172.21.15.1:0/287678415 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.552+0000 7fd8e7fff640 1 -- 172.21.15.1:0/287678415 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e806d060 msgr2=0x7fd8e80c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.552+0000 7fd8e6ffd640 1 --2- 172.21.15.1:0/287678415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e80c3eb0 0x7fd8e8158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.552+0000 7fd8e7fff640 1 --2- 172.21.15.1:0/287678415 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e806d060 0x7fd8e80c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.552+0000 7fd8e7fff640 1 -- 172.21.15.1:0/287678415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e80c3eb0 msgr2=0x7fd8e8158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.552+0000 7fd8e7fff640 1 --2- 172.21.15.1:0/287678415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e80c3eb0 0x7fd8e8158870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.552+0000 7fd8e7fff640 1 -- 172.21.15.1:0/287678415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e814bc00 con 0x7fd8e80c7940 2024-03-06T07:21:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.552+0000 7fd8e6ffd640 1 --2- 172.21.15.1:0/287678415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e80c3eb0 0x7fd8e8158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.552+0000 7fd8e7fff640 1 --2- 172.21.15.1:0/287678415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e80c7940 0x7fd8e8156560 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000c730 tx=0x7fd8e000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.552+0000 7fd8cffff640 1 -- 172.21.15.1:0/287678415 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0018070 con 0x7fd8e80c7940 2024-03-06T07:21:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.552+0000 7fd8cffff640 1 -- 172.21.15.1:0/287678415 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e000ce40 con 0x7fd8e80c7940 2024-03-06T07:21:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.552+0000 7fd8edc84640 1 -- 172.21.15.1:0/287678415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e814d3c0 con 0x7fd8e80c7940 2024-03-06T07:21:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.552+0000 7fd8cffff640 1 -- 172.21.15.1:0/287678415 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e000a860 con 0x7fd8e80c7940 2024-03-06T07:21:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.552+0000 7fd8edc84640 1 -- 172.21.15.1:0/287678415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e80c9ff0 con 0x7fd8e80c7940 2024-03-06T07:21:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.553+0000 7fd8edc84640 1 -- 172.21.15.1:0/287678415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e8067fd0 con 0x7fd8e80c7940 2024-03-06T07:21:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.553+0000 7fd8cffff640 1 -- 172.21.15.1:0/287678415 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8e001b070 con 0x7fd8e80c7940 2024-03-06T07:21:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.554+0000 7fd8cffff640 1 --2- 172.21.15.1:0/287678415 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8d406b500 0x7fd8d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.554+0000 7fd8cffff640 1 -- 172.21.15.1:0/287678415 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd8e0089560 con 0x7fd8e80c7940 2024-03-06T07:21:19.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.554+0000 7fd8e77fe640 1 --2- 172.21.15.1:0/287678415 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8d406b500 0x7fd8d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.555+0000 7fd8e77fe640 1 --2- 172.21.15.1:0/287678415 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8d406b500 0x7fd8d406d9c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8035000 tx=0x7fd8d8030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.558+0000 7fd8cffff640 1 -- 172.21.15.1:0/287678415 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8e0058630 con 0x7fd8e80c7940 2024-03-06T07:21:19.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.812+0000 7fd8edc84640 1 -- 172.21.15.1:0/287678415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7fd8e814d170 con 0x7fd8e80c7940 2024-03-06T07:21:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.814+0000 7fd8cffff640 1 -- 172.21.15.1:0/287678415 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7fd8e0057f30 con 0x7fd8e80c7940 2024-03-06T07:21:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.816+0000 7fd8edc84640 1 -- 172.21.15.1:0/287678415 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8d406b500 msgr2=0x7fd8d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.816+0000 7fd8edc84640 1 --2- 172.21.15.1:0/287678415 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8d406b500 0x7fd8d406d9c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8035000 tx=0x7fd8d8030000 comp rx=0 tx=0).stop 2024-03-06T07:21:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.816+0000 7fd8edc84640 1 -- 172.21.15.1:0/287678415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e80c7940 msgr2=0x7fd8e8156560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.816+0000 7fd8edc84640 1 --2- 172.21.15.1:0/287678415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e80c7940 0x7fd8e8156560 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000c730 tx=0x7fd8e000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.816+0000 7fd8edc84640 1 -- 172.21.15.1:0/287678415 shutdown_connections 2024-03-06T07:21:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.816+0000 7fd8edc84640 1 --2- 172.21.15.1:0/287678415 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8d406b500 0x7fd8d406d9c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.816+0000 7fd8edc84640 1 --2- 172.21.15.1:0/287678415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e80c7940 0x7fd8e8156560 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.816+0000 7fd8edc84640 1 --2- 172.21.15.1:0/287678415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e80c3eb0 0x7fd8e8158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.816+0000 7fd8edc84640 1 --2- 172.21.15.1:0/287678415 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e806d060 0x7fd8e80c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.816+0000 7fd8edc84640 1 -- 172.21.15.1:0/287678415 >> 172.21.15.1:0/287678415 conn(0x7fd8e8066d00 msgr2=0x7fd8e8074450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.816+0000 7fd8edc84640 1 -- 172.21.15.1:0/287678415 shutdown_connections 2024-03-06T07:21:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.816+0000 7fd8edc84640 1 -- 172.21.15.1:0/287678415 wait complete. 2024-03-06T07:21:19.829 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:19.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.928+0000 7f19ebf65640 1 Processor -- start 2024-03-06T07:21:19.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.928+0000 7f19ebf65640 1 -- start start 2024-03-06T07:21:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19ebf65640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19e4152920 0x7f19e406aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19ebf65640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19e4155220 0x7f19e41556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19ebf65640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e4158910 con 0x7f19e406b530 2024-03-06T07:21:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19ebf65640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e4152d00 con 0x7f19e4155220 2024-03-06T07:21:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19ebf65640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e41596d0 con 0x7f19e4152920 2024-03-06T07:21:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19e9500640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f19e406b530 0x7f19e40c5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38594/0 (socket says 172.21.15.1:38594) 2024-03-06T07:21:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19e9500640 1 -- 172.21.15.1:0/2871407717 learned_addr learned my addr 172.21.15.1:0/2871407717 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19ea502640 1 --2- 172.21.15.1:0/2871407717 >> v2:172.21.15.98:3300/0 conn(0x7f19e4155220 0x7f19e41556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19e9d01640 1 --2- 172.21.15.1:0/2871407717 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19e4152920 0x7f19e406aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19e8cff640 1 -- 172.21.15.1:0/2871407717 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4096388822 0 0) 0x7f19e4158910 con 0x7f19e406b530 2024-03-06T07:21:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19e8cff640 1 -- 172.21.15.1:0/2871407717 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d0003520 con 0x7f19e406b530 2024-03-06T07:21:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19e8cff640 1 -- 172.21.15.1:0/2871407717 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2917488227 0 0) 0x7f19d0003520 con 0x7f19e406b530 2024-03-06T07:21:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19e8cff640 1 -- 172.21.15.1:0/2871407717 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19e4152920 msgr2=0x7f19e406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:19.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19e8cff640 1 --2- 172.21.15.1:0/2871407717 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19e4152920 0x7f19e406aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19e8cff640 1 -- 172.21.15.1:0/2871407717 >> v2:172.21.15.98:3300/0 conn(0x7f19e4155220 msgr2=0x7f19e41556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:19.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19e8cff640 1 --2- 172.21.15.1:0/2871407717 >> v2:172.21.15.98:3300/0 conn(0x7f19e4155220 0x7f19e41556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19e8cff640 1 -- 172.21.15.1:0/2871407717 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e4158bf0 con 0x7f19e406b530 2024-03-06T07:21:19.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.929+0000 7f19e8cff640 1 -- 172.21.15.1:0/2871407717 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f19d4001670 con 0x7f19e406b530 2024-03-06T07:21:19.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.930+0000 7f19e8cff640 1 -- 172.21.15.1:0/2871407717 >> v1:172.21.15.1:6789/0 conn(0x7f19e406b530 legacy=0x7f19e40c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.930+0000 7f19e8cff640 1 --2- 172.21.15.1:0/2871407717 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19d00039b0 0x7f19d0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.930+0000 7f19e8cff640 1 --2- 172.21.15.1:0/2871407717 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19e4152920 0x7f19d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.930+0000 7f19e8cff640 1 --2- 172.21.15.1:0/2871407717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19d0004d40 0x7f19d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.930+0000 7f19e8cff640 1 -- 172.21.15.1:0/2871407717 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e4152d00 con 0x7f19d00039b0 2024-03-06T07:21:19.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.930+0000 7f19e8cff640 1 -- 172.21.15.1:0/2871407717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e41596d0 con 0x7f19d0004d40 2024-03-06T07:21:19.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.930+0000 7f19e8cff640 1 -- 172.21.15.1:0/2871407717 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d000b9c0 con 0x7f19e4152920 2024-03-06T07:21:19.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.930+0000 7f19e9d01640 1 --2- 172.21.15.1:0/2871407717 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19d00039b0 0x7f19d0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.930+0000 7f19e9500640 1 --2- 172.21.15.1:0/2871407717 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19e4152920 0x7f19d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.930+0000 7f19ea502640 1 --2- 172.21.15.1:0/2871407717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19d0004d40 0x7f19d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.930+0000 7f19e9d01640 1 -- 172.21.15.1:0/2871407717 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19e4152920 msgr2=0x7f19d0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:19.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.930+0000 7f19e9d01640 1 --2- 172.21.15.1:0/2871407717 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19e4152920 0x7f19d0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.930+0000 7f19e9d01640 1 -- 172.21.15.1:0/2871407717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19d0004d40 msgr2=0x7f19d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:19.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.930+0000 7f19e9d01640 1 --2- 172.21.15.1:0/2871407717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19d0004d40 0x7f19d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.930+0000 7f19e9d01640 1 -- 172.21.15.1:0/2871407717 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d0003520 con 0x7f19d00039b0 2024-03-06T07:21:19.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.930+0000 7f19ea502640 1 --2- 172.21.15.1:0/2871407717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19d0004d40 0x7f19d00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:19.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.931+0000 7f19e9d01640 1 --2- 172.21.15.1:0/2871407717 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19d00039b0 0x7f19d0003dd0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009ac0 tx=0x7f19d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe1e7a068d7ea66 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.931+0000 7f19e8cff640 1 -- 172.21.15.1:0/2871407717 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0039070 con 0x7f19d00039b0 2024-03-06T07:21:19.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.931+0000 7f19e8cff640 1 -- 172.21.15.1:0/2871407717 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e002c6b0 con 0x7f19d00039b0 2024-03-06T07:21:19.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.931+0000 7f19e8cff640 1 -- 172.21.15.1:0/2871407717 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0009690 con 0x7f19d00039b0 2024-03-06T07:21:19.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.931+0000 7f19ebf65640 1 -- 172.21.15.1:0/2871407717 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19d00039b0 msgr2=0x7f19d0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.931+0000 7f19ebf65640 1 --2- 172.21.15.1:0/2871407717 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19d00039b0 0x7f19d0003dd0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009ac0 tx=0x7f19d0005830 comp rx=0 tx=0).stop 2024-03-06T07:21:19.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.931+0000 7f19ea502640 1 -- 172.21.15.1:0/2871407717 reap_dead start 2024-03-06T07:21:19.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.931+0000 7f19ebf65640 1 -- 172.21.15.1:0/2871407717 shutdown_connections 2024-03-06T07:21:19.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.931+0000 7f19ebf65640 1 -- 172.21.15.1:0/2871407717 >> 172.21.15.1:0/2871407717 conn(0x7f19e40a9700 msgr2=0x7f19e40a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:19.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.931+0000 7f19ebf65640 1 -- 172.21.15.1:0/2871407717 shutdown_connections 2024-03-06T07:21:19.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.931+0000 7f19ebf65640 1 -- 172.21.15.1:0/2871407717 wait complete. 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.933+0000 7f19ebf65640 1 Processor -- start 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.933+0000 7f19ebf65640 1 -- start start 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.933+0000 7f19ebf65640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19d00039b0 0x7f19e40a8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.933+0000 7f19ebf65640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19e40ad260 0x7f19e4148dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.933+0000 7f19ebf65640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19e4149310 0x7f19e41eae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.933+0000 7f19ebf65640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e40c6380 con 0x7f19d00039b0 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.933+0000 7f19ebf65640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e4159240 con 0x7f19e4149310 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.933+0000 7f19ebf65640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e40c5f60 con 0x7f19e40ad260 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.933+0000 7f19e9d01640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19d00039b0 0x7f19e40a8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.933+0000 7f19e9d01640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19d00039b0 0x7f19e40a8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37410/0 (socket says 172.21.15.1:37410) 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.933+0000 7f19e9d01640 1 -- 172.21.15.1:0/1911848479 learned_addr learned my addr 172.21.15.1:0/1911848479 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.934+0000 7f19ea502640 1 --2- 172.21.15.1:0/1911848479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19e4149310 0x7f19e41eae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.934+0000 7f19e9d01640 1 -- 172.21.15.1:0/1911848479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19e40ad260 msgr2=0x7f19e4148dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.934+0000 7f19e9500640 1 --2- 172.21.15.1:0/1911848479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19e40ad260 0x7f19e4148dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.934+0000 7f19e9d01640 1 --2- 172.21.15.1:0/1911848479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19e40ad260 0x7f19e4148dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.934+0000 7f19e9d01640 1 -- 172.21.15.1:0/1911848479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19e4149310 msgr2=0x7f19e41eae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.934+0000 7f19e9d01640 1 --2- 172.21.15.1:0/1911848479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19e4149310 0x7f19e41eae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.934+0000 7f19e9d01640 1 -- 172.21.15.1:0/1911848479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e40a9240 con 0x7f19d00039b0 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.934+0000 7f19e9d01640 1 --2- 172.21.15.1:0/1911848479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19d00039b0 0x7f19e40a8960 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f19e0037040 tx=0x7f19e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.934+0000 7f19da7fc640 1 -- 172.21.15.1:0/1911848479 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0042020 con 0x7f19d00039b0 2024-03-06T07:21:19.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.934+0000 7f19da7fc640 1 -- 172.21.15.1:0/1911848479 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e003a040 con 0x7f19d00039b0 2024-03-06T07:21:19.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.934+0000 7f19ebf65640 1 -- 172.21.15.1:0/1911848479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e4158300 con 0x7f19d00039b0 2024-03-06T07:21:19.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.934+0000 7f19da7fc640 1 -- 172.21.15.1:0/1911848479 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0039070 con 0x7f19d00039b0 2024-03-06T07:21:19.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.934+0000 7f19ebf65640 1 -- 172.21.15.1:0/1911848479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e41570e0 con 0x7f19d00039b0 2024-03-06T07:21:19.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.935+0000 7f19ebf65640 1 -- 172.21.15.1:0/1911848479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e406bdf0 con 0x7f19d00039b0 2024-03-06T07:21:19.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.935+0000 7f19da7fc640 1 -- 172.21.15.1:0/1911848479 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19e0045020 con 0x7f19d00039b0 2024-03-06T07:21:19.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.936+0000 7f19da7fc640 1 --2- 172.21.15.1:0/1911848479 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19c806b500 0x7f19c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.936+0000 7f19da7fc640 1 -- 172.21.15.1:0/1911848479 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f19e00ab440 con 0x7f19d00039b0 2024-03-06T07:21:19.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.936+0000 7f19e9500640 1 --2- 172.21.15.1:0/1911848479 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19c806b500 0x7f19c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.936+0000 7f19e9500640 1 --2- 172.21.15.1:0/1911848479 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19c806b500 0x7f19c806d9c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f19e41e8c70 tx=0x7f19d4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:19.940+0000 7f19da7fc640 1 -- 172.21.15.1:0/1911848479 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19e004c070 con 0x7f19d00039b0 2024-03-06T07:21:20.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.120+0000 7f19ebf65640 1 -- 172.21.15.1:0/1911848479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19e40c68f0 con 0x7f19d00039b0 2024-03-06T07:21:20.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.121+0000 7f19da7fc640 1 -- 172.21.15.1:0/1911848479 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+13044 (secure 0 0 0) 0x7f19e007a510 con 0x7f19d00039b0 2024-03-06T07:21:20.123 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:20.123 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":35,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:21:17.141581+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.123+0000 7f19ebf65640 1 -- 172.21.15.1:0/1911848479 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19c806b500 msgr2=0x7f19c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.123+0000 7f19ebf65640 1 --2- 172.21.15.1:0/1911848479 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19c806b500 0x7f19c806d9c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f19e41e8c70 tx=0x7f19d4009040 comp rx=0 tx=0).stop 2024-03-06T07:21:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.123+0000 7f19ebf65640 1 -- 172.21.15.1:0/1911848479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19d00039b0 msgr2=0x7f19e40a8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.123+0000 7f19ebf65640 1 --2- 172.21.15.1:0/1911848479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19d00039b0 0x7f19e40a8960 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f19e0037040 tx=0x7f19e0004730 comp rx=0 tx=0).stop 2024-03-06T07:21:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.123+0000 7f19ebf65640 1 -- 172.21.15.1:0/1911848479 shutdown_connections 2024-03-06T07:21:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.123+0000 7f19ebf65640 1 --2- 172.21.15.1:0/1911848479 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19c806b500 0x7f19c806d9c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.123+0000 7f19ebf65640 1 --2- 172.21.15.1:0/1911848479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19e4149310 0x7f19e41eae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.123+0000 7f19ebf65640 1 --2- 172.21.15.1:0/1911848479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19e40ad260 0x7f19e4148dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.123+0000 7f19ebf65640 1 --2- 172.21.15.1:0/1911848479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19d00039b0 0x7f19e40a8960 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.123+0000 7f19ebf65640 1 -- 172.21.15.1:0/1911848479 >> 172.21.15.1:0/1911848479 conn(0x7f19e40a9700 msgr2=0x7f19e4153850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:20.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.123+0000 7f19ebf65640 1 -- 172.21.15.1:0/1911848479 shutdown_connections 2024-03-06T07:21:20.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.123+0000 7f19ebf65640 1 -- 172.21.15.1:0/1911848479 wait complete. 2024-03-06T07:21:20.136 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:20.136 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:20.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.239+0000 7f0f211b5640 1 Processor -- start 2024-03-06T07:21:20.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.239+0000 7f0f211b5640 1 -- start start 2024-03-06T07:21:20.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.239+0000 7f0f211b5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0f1c06e670 0x7f0f1c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.239+0000 7f0f211b5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0f1c0a8520 0x7f0f1c0c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.240+0000 7f0f211b5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1c06f060 con 0x7f0f1c150290 2024-03-06T07:21:20.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.240+0000 7f0f211b5640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1c0c2ca0 con 0x7f0f1c06e670 2024-03-06T07:21:20.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.240+0000 7f0f211b5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1c0c26e0 con 0x7f0f1c0a8520 2024-03-06T07:21:20.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.240+0000 7f0f1ad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0f1c06e670 0x7f0f1c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.240+0000 7f0f1b59e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0f1c150290 0x7f0f1c153290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38602/0 (socket says 172.21.15.1:38602) 2024-03-06T07:21:20.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.240+0000 7f0f1ad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0f1c06e670 0x7f0f1c06ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37418/0 (socket says 172.21.15.1:37418) 2024-03-06T07:21:20.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.240+0000 7f0f1b59e640 1 -- 172.21.15.1:0/329477344 learned_addr learned my addr 172.21.15.1:0/329477344 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:20.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.240+0000 7f0f13fff640 1 --2- 172.21.15.1:0/329477344 >> v2:172.21.15.98:3300/0 conn(0x7f0f1c0a8520 0x7f0f1c0c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.240+0000 7f0f1ad9d640 1 -- 172.21.15.1:0/329477344 >> v2:172.21.15.98:3300/0 conn(0x7f0f1c0a8520 msgr2=0x7f0f1c0c1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:20.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.240+0000 7f0f1ad9d640 1 --2- 172.21.15.1:0/329477344 >> v2:172.21.15.98:3300/0 conn(0x7f0f1c0a8520 0x7f0f1c0c1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.240+0000 7f0f1ad9d640 1 -- 172.21.15.1:0/329477344 >> v1:172.21.15.1:6789/0 conn(0x7f0f1c150290 legacy=0x7f0f1c153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.240+0000 7f0f1ad9d640 1 -- 172.21.15.1:0/329477344 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c0c29c0 con 0x7f0f1c06e670 2024-03-06T07:21:20.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.240+0000 7f0f1a59c640 1 -- 172.21.15.1:0/329477344 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591568616 0 0) 0x7f0f1c06f060 con 0x7f0f1c150290 2024-03-06T07:21:20.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.240+0000 7f0f13fff640 1 --2- 172.21.15.1:0/329477344 >> v2:172.21.15.98:3300/0 conn(0x7f0f1c0a8520 0x7f0f1c0c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:20.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.241+0000 7f0f1ad9d640 1 --2- 172.21.15.1:0/329477344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c06e670 0x7f0f1c06ea50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c072e60 tx=0x7f0f0c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=658090b5a1386553 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.241+0000 7f0f1a59c640 1 -- 172.21.15.1:0/329477344 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c0315a0 con 0x7f0f1c06e670 2024-03-06T07:21:20.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.241+0000 7f0f1a59c640 1 -- 172.21.15.1:0/329477344 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0c037070 con 0x7f0f1c06e670 2024-03-06T07:21:20.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.241+0000 7f0f1a59c640 1 -- 172.21.15.1:0/329477344 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c031960 con 0x7f0f1c06e670 2024-03-06T07:21:20.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.241+0000 7f0f211b5640 1 -- 172.21.15.1:0/329477344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c06e670 msgr2=0x7f0f1c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.241+0000 7f0f211b5640 1 --2- 172.21.15.1:0/329477344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c06e670 0x7f0f1c06ea50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c072e60 tx=0x7f0f0c02b840 comp rx=0 tx=0).stop 2024-03-06T07:21:20.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.241+0000 7f0f211b5640 1 -- 172.21.15.1:0/329477344 shutdown_connections 2024-03-06T07:21:20.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.241+0000 7f0f211b5640 1 --2- 172.21.15.1:0/329477344 >> v2:172.21.15.98:3300/0 conn(0x7f0f1c0a8520 0x7f0f1c0c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.241+0000 7f0f211b5640 1 --2- 172.21.15.1:0/329477344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c06e670 0x7f0f1c06ea50 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.241+0000 7f0f211b5640 1 -- 172.21.15.1:0/329477344 >> 172.21.15.1:0/329477344 conn(0x7f0f1c0c3cc0 msgr2=0x7f0f1c0c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:20.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.241+0000 7f0f211b5640 1 -- 172.21.15.1:0/329477344 shutdown_connections 2024-03-06T07:21:20.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.241+0000 7f0f211b5640 1 -- 172.21.15.1:0/329477344 wait complete. 2024-03-06T07:21:20.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.243+0000 7f0f211b5640 1 Processor -- start 2024-03-06T07:21:20.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.243+0000 7f0f211b5640 1 -- start start 2024-03-06T07:21:20.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.243+0000 7f0f211b5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f1c06e670 0x7f0f1c154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.243+0000 7f0f211b5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c0a8520 0x7f0f1c1e5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.243+0000 7f0f211b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f1c150290 0x7f0f1c1e5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.243+0000 7f0f211b5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c150ad0 con 0x7f0f1c0a8520 2024-03-06T07:21:20.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.243+0000 7f0f211b5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c150950 con 0x7f0f1c150290 2024-03-06T07:21:20.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.243+0000 7f0f211b5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c1539a0 con 0x7f0f1c06e670 2024-03-06T07:21:20.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.244+0000 7f0f13fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c0a8520 0x7f0f1c1e5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.244+0000 7f0f13fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c0a8520 0x7f0f1c1e5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37422/0 (socket says 172.21.15.1:37422) 2024-03-06T07:21:20.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.244+0000 7f0f13fff640 1 -- 172.21.15.1:0/2342065612 learned_addr learned my addr 172.21.15.1:0/2342065612 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:20.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.244+0000 7f0f1b59e640 1 --2- 172.21.15.1:0/2342065612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f1c150290 0x7f0f1c1e5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.244+0000 7f0f1ad9d640 1 --2- 172.21.15.1:0/2342065612 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f1c06e670 0x7f0f1c154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.244+0000 7f0f13fff640 1 -- 172.21.15.1:0/2342065612 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f1c06e670 msgr2=0x7f0f1c154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.244+0000 7f0f13fff640 1 --2- 172.21.15.1:0/2342065612 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f1c06e670 0x7f0f1c154960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.244+0000 7f0f13fff640 1 -- 172.21.15.1:0/2342065612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f1c150290 msgr2=0x7f0f1c1e5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.244+0000 7f0f13fff640 1 --2- 172.21.15.1:0/2342065612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f1c150290 0x7f0f1c1e5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.244+0000 7f0f13fff640 1 -- 172.21.15.1:0/2342065612 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c1e9460 con 0x7f0f1c0a8520 2024-03-06T07:21:20.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.244+0000 7f0f13fff640 1 --2- 172.21.15.1:0/2342065612 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c0a8520 0x7f0f1c1e5a30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400c7f0 tx=0x7f0f0400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.244+0000 7f0f137fe640 1 -- 172.21.15.1:0/2342065612 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04004230 con 0x7f0f1c0a8520 2024-03-06T07:21:20.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.244+0000 7f0f137fe640 1 -- 172.21.15.1:0/2342065612 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f040043d0 con 0x7f0f1c0a8520 2024-03-06T07:21:20.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.244+0000 7f0f211b5640 1 -- 172.21.15.1:0/2342065612 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1c155350 con 0x7f0f1c0a8520 2024-03-06T07:21:20.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.244+0000 7f0f137fe640 1 -- 172.21.15.1:0/2342065612 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04007a70 con 0x7f0f1c0a8520 2024-03-06T07:21:20.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.244+0000 7f0f211b5640 1 -- 172.21.15.1:0/2342065612 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1c148160 con 0x7f0f1c0a8520 2024-03-06T07:21:20.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.245+0000 7f0f211b5640 1 -- 172.21.15.1:0/2342065612 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1c067fd0 con 0x7f0f1c0a8520 2024-03-06T07:21:20.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.246+0000 7f0f137fe640 1 -- 172.21.15.1:0/2342065612 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f04007c10 con 0x7f0f1c0a8520 2024-03-06T07:21:20.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.246+0000 7f0f137fe640 1 --2- 172.21.15.1:0/2342065612 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0efc06b4d0 0x7f0efc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.246+0000 7f0f137fe640 1 -- 172.21.15.1:0/2342065612 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f0f0408a060 con 0x7f0f1c0a8520 2024-03-06T07:21:20.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.246+0000 7f0f1ad9d640 1 --2- 172.21.15.1:0/2342065612 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0efc06b4d0 0x7f0efc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.247+0000 7f0f1ad9d640 1 --2- 172.21.15.1:0/2342065612 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0efc06b4d0 0x7f0efc06d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c036000 tx=0x7f0f0c03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.250+0000 7f0f137fe640 1 -- 172.21.15.1:0/2342065612 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f04015020 con 0x7f0f1c0a8520 2024-03-06T07:21:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.430+0000 7f0f211b5640 1 -- 172.21.15.1:0/2342065612 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0f1c157280 con 0x7f0f1c0a8520 2024-03-06T07:21:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.431+0000 7f0f137fe640 1 -- 172.21.15.1:0/2342065612 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0f04059130 con 0x7f0f1c0a8520 2024-03-06T07:21:20.433 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:21:20.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.432+0000 7f0f211b5640 1 -- 172.21.15.1:0/2342065612 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0efc06b4d0 msgr2=0x7f0efc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.432+0000 7f0f211b5640 1 --2- 172.21.15.1:0/2342065612 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0efc06b4d0 0x7f0efc06d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c036000 tx=0x7f0f0c03d040 comp rx=0 tx=0).stop 2024-03-06T07:21:20.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.432+0000 7f0f211b5640 1 -- 172.21.15.1:0/2342065612 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c0a8520 msgr2=0x7f0f1c1e5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.432+0000 7f0f211b5640 1 --2- 172.21.15.1:0/2342065612 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c0a8520 0x7f0f1c1e5a30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400c7f0 tx=0x7f0f0400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.432+0000 7f0f211b5640 1 -- 172.21.15.1:0/2342065612 shutdown_connections 2024-03-06T07:21:20.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.432+0000 7f0f211b5640 1 --2- 172.21.15.1:0/2342065612 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0efc06b4d0 0x7f0efc06d990 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.432+0000 7f0f211b5640 1 --2- 172.21.15.1:0/2342065612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f1c150290 0x7f0f1c1e5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.432+0000 7f0f211b5640 1 --2- 172.21.15.1:0/2342065612 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c0a8520 0x7f0f1c1e5a30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.432+0000 7f0f211b5640 1 --2- 172.21.15.1:0/2342065612 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f1c06e670 0x7f0f1c154960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.432+0000 7f0f211b5640 1 -- 172.21.15.1:0/2342065612 >> 172.21.15.1:0/2342065612 conn(0x7f0f1c0c3cc0 msgr2=0x7f0f1c0a8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:20.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.433+0000 7f0f211b5640 1 -- 172.21.15.1:0/2342065612 shutdown_connections 2024-03-06T07:21:20.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.433+0000 7f0f211b5640 1 -- 172.21.15.1:0/2342065612 wait complete. 2024-03-06T07:21:20.445 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:21:20.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.546+0000 7fa314965640 1 Processor -- start 2024-03-06T07:21:20.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.546+0000 7fa314965640 1 -- start start 2024-03-06T07:21:20.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.547+0000 7fa314965640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3100c3eb0 0x7fa3100c7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.547+0000 7fa314965640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3100c71c0 con 0x7fa3100c7940 2024-03-06T07:21:20.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.547+0000 7fa314965640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3100c4e50 con 0x7fa31006d060 2024-03-06T07:21:20.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.547+0000 7fa314965640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100c59b0 con 0x7fa3100c3eb0 2024-03-06T07:21:20.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.547+0000 7fa30ed9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa3100c7940 0x7fa31015fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38614/0 (socket says 172.21.15.1:38614) 2024-03-06T07:21:20.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.547+0000 7fa30ed9d640 1 -- 172.21.15.1:0/3803500646 learned_addr learned my addr 172.21.15.1:0/3803500646 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:20.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.547+0000 7fa30e59c640 1 --2- 172.21.15.1:0/3803500646 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3100c3eb0 0x7fa3100c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.547+0000 7fa30d59a640 1 -- 172.21.15.1:0/3803500646 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116402492 0 0) 0x7fa3100c71c0 con 0x7fa3100c7940 2024-03-06T07:21:20.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.547+0000 7fa30d59a640 1 -- 172.21.15.1:0/3803500646 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2fc003520 con 0x7fa3100c7940 2024-03-06T07:21:20.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30d59a640 1 -- 172.21.15.1:0/3803500646 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 703815328 0 0) 0x7fa2fc003520 con 0x7fa3100c7940 2024-03-06T07:21:20.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30d59a640 1 -- 172.21.15.1:0/3803500646 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3100c3eb0 msgr2=0x7fa3100c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:20.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30d59a640 1 --2- 172.21.15.1:0/3803500646 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3100c3eb0 0x7fa3100c7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30d59a640 1 -- 172.21.15.1:0/3803500646 >> v1:172.21.15.98:6789/0 conn(0x7fa31006d060 legacy=0x7fa3101558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30d59a640 1 -- 172.21.15.1:0/3803500646 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3100c6ee0 con 0x7fa3100c7940 2024-03-06T07:21:20.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30d59a640 1 -- 172.21.15.1:0/3803500646 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fa3000019f0 con 0x7fa3100c7940 2024-03-06T07:21:20.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30d59a640 1 -- 172.21.15.1:0/3803500646 >> v1:172.21.15.1:6789/0 conn(0x7fa3100c7940 legacy=0x7fa31015fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30d59a640 1 --2- 172.21.15.1:0/3803500646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2fc0039e0 0x7fa2fc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30d59a640 1 --2- 172.21.15.1:0/3803500646 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3100c3eb0 0x7fa2fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30d59a640 1 --2- 172.21.15.1:0/3803500646 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2fc004d40 0x7fa2fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30d59a640 1 -- 172.21.15.1:0/3803500646 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100c59b0 con 0x7fa2fc004d40 2024-03-06T07:21:20.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30d59a640 1 -- 172.21.15.1:0/3803500646 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc00bb80 con 0x7fa2fc0039e0 2024-03-06T07:21:20.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30d59a640 1 -- 172.21.15.1:0/3803500646 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc00bd80 con 0x7fa3100c3eb0 2024-03-06T07:21:20.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30ed9d640 1 --2- 172.21.15.1:0/3803500646 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2fc004d40 0x7fa2fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30e59c640 1 --2- 172.21.15.1:0/3803500646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2fc0039e0 0x7fa2fc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30dd9b640 1 --2- 172.21.15.1:0/3803500646 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3100c3eb0 0x7fa2fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30ed9d640 1 -- 172.21.15.1:0/3803500646 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3100c3eb0 msgr2=0x7fa2fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:20.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30ed9d640 1 --2- 172.21.15.1:0/3803500646 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3100c3eb0 0x7fa2fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30ed9d640 1 -- 172.21.15.1:0/3803500646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2fc0039e0 msgr2=0x7fa2fc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:20.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30ed9d640 1 --2- 172.21.15.1:0/3803500646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2fc0039e0 0x7fa2fc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.548+0000 7fa30ed9d640 1 -- 172.21.15.1:0/3803500646 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2fc003520 con 0x7fa2fc004d40 2024-03-06T07:21:20.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.549+0000 7fa30ed9d640 1 --2- 172.21.15.1:0/3803500646 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2fc004d40 0x7fa2fc0051c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa30000a840 tx=0x7fa2fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518785d2eaa90238 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.549+0000 7fa30d59a640 1 -- 172.21.15.1:0/3803500646 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30000a460 con 0x7fa2fc004d40 2024-03-06T07:21:20.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.549+0000 7fa30d59a640 1 -- 172.21.15.1:0/3803500646 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa300034650 con 0x7fa2fc004d40 2024-03-06T07:21:20.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.549+0000 7fa30d59a640 1 -- 172.21.15.1:0/3803500646 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa300034a40 con 0x7fa2fc004d40 2024-03-06T07:21:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.550+0000 7fa314965640 1 -- 172.21.15.1:0/3803500646 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2fc004d40 msgr2=0x7fa2fc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.550+0000 7fa314965640 1 --2- 172.21.15.1:0/3803500646 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2fc004d40 0x7fa2fc0051c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa30000a840 tx=0x7fa2fc005830 comp rx=0 tx=0).stop 2024-03-06T07:21:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.550+0000 7fa30ed9d640 1 -- 172.21.15.1:0/3803500646 reap_dead start 2024-03-06T07:21:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.550+0000 7fa314965640 1 -- 172.21.15.1:0/3803500646 shutdown_connections 2024-03-06T07:21:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.550+0000 7fa314965640 1 -- 172.21.15.1:0/3803500646 >> 172.21.15.1:0/3803500646 conn(0x7fa310066d00 msgr2=0x7fa310067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.550+0000 7fa314965640 1 -- 172.21.15.1:0/3803500646 shutdown_connections 2024-03-06T07:21:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.550+0000 7fa314965640 1 -- 172.21.15.1:0/3803500646 wait complete. 2024-03-06T07:21:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.551+0000 7fa314965640 1 Processor -- start 2024-03-06T07:21:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.551+0000 7fa314965640 1 -- start start 2024-03-06T07:21:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa314965640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2fc004d40 0x7fa3100cac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa314965640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3101591f0 0x7fa3101472c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa314965640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa310148360 0x7fa310147800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa314965640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa310155420 con 0x7fa3101591f0 2024-03-06T07:21:20.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa314965640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100c4c00 con 0x7fa2fc004d40 2024-03-06T07:21:20.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa314965640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa310155fc0 con 0x7fa310148360 2024-03-06T07:21:20.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa30dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3101591f0 0x7fa3101472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa30dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3101591f0 0x7fa3101472c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37442/0 (socket says 172.21.15.1:37442) 2024-03-06T07:21:20.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa30dd9b640 1 -- 172.21.15.1:0/347610 learned_addr learned my addr 172.21.15.1:0/347610 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:20.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa30ed9d640 1 --2- 172.21.15.1:0/347610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa310148360 0x7fa310147800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa30e59c640 1 --2- 172.21.15.1:0/347610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2fc004d40 0x7fa3100cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa30dd9b640 1 -- 172.21.15.1:0/347610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa310148360 msgr2=0x7fa310147800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa30dd9b640 1 --2- 172.21.15.1:0/347610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa310148360 0x7fa310147800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa30dd9b640 1 -- 172.21.15.1:0/347610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2fc004d40 msgr2=0x7fa3100cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa30dd9b640 1 --2- 172.21.15.1:0/347610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2fc004d40 0x7fa3100cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa30dd9b640 1 -- 172.21.15.1:0/347610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa310147d40 con 0x7fa3101591f0 2024-03-06T07:21:20.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa30dd9b640 1 --2- 172.21.15.1:0/347610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3101591f0 0x7fa3101472c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa2f800f7f0 tx=0x7fa2f800fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa306ffd640 1 -- 172.21.15.1:0/347610 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f80050d0 con 0x7fa3101591f0 2024-03-06T07:21:20.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa306ffd640 1 -- 172.21.15.1:0/347610 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f8018070 con 0x7fa3101591f0 2024-03-06T07:21:20.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa314965640 1 -- 172.21.15.1:0/347610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31014b350 con 0x7fa3101591f0 2024-03-06T07:21:20.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa306ffd640 1 -- 172.21.15.1:0/347610 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f800d730 con 0x7fa3101591f0 2024-03-06T07:21:20.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.552+0000 7fa314965640 1 -- 172.21.15.1:0/347610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa310148ff0 con 0x7fa3101591f0 2024-03-06T07:21:20.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.553+0000 7fa314965640 1 -- 172.21.15.1:0/347610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa310067fd0 con 0x7fa3101591f0 2024-03-06T07:21:20.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.554+0000 7fa306ffd640 1 -- 172.21.15.1:0/347610 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2f800d8d0 con 0x7fa3101591f0 2024-03-06T07:21:20.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.554+0000 7fa306ffd640 1 --2- 172.21.15.1:0/347610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2f006b4d0 0x7fa2f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.554+0000 7fa306ffd640 1 -- 172.21.15.1:0/347610 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fa2f8089e70 con 0x7fa3101591f0 2024-03-06T07:21:20.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.554+0000 7fa30e59c640 1 --2- 172.21.15.1:0/347610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2f006b4d0 0x7fa2f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.555+0000 7fa30e59c640 1 --2- 172.21.15.1:0/347610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2f006b4d0 0x7fa2f006d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa308009b00 tx=0x7fa308008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.558+0000 7fa306ffd640 1 -- 172.21.15.1:0/347610 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2f8058eb0 con 0x7fa3101591f0 2024-03-06T07:21:20.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.741+0000 7fa314965640 1 -- 172.21.15.1:0/347610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa3100c5b30 con 0x7fa3101591f0 2024-03-06T07:21:20.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.741+0000 7fa306ffd640 1 -- 172.21.15.1:0/347610 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa2f80587b0 con 0x7fa3101591f0 2024-03-06T07:21:20.743 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:20.744 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-06T07:21:20.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:21:20.744 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-06T07:21:20.744 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-06T07:21:20.744 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:21:20.744 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-06T07:21:20.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.743+0000 7fa314965640 1 -- 172.21.15.1:0/347610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2f006b4d0 msgr2=0x7fa2f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.743+0000 7fa314965640 1 --2- 172.21.15.1:0/347610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2f006b4d0 0x7fa2f006d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa308009b00 tx=0x7fa308008040 comp rx=0 tx=0).stop 2024-03-06T07:21:20.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.743+0000 7fa314965640 1 -- 172.21.15.1:0/347610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3101591f0 msgr2=0x7fa3101472c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.743+0000 7fa314965640 1 --2- 172.21.15.1:0/347610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3101591f0 0x7fa3101472c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa2f800f7f0 tx=0x7fa2f800fcb0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.743+0000 7fa314965640 1 -- 172.21.15.1:0/347610 shutdown_connections 2024-03-06T07:21:20.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.743+0000 7fa314965640 1 --2- 172.21.15.1:0/347610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2f006b4d0 0x7fa2f006d990 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.743+0000 7fa314965640 1 --2- 172.21.15.1:0/347610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa310148360 0x7fa310147800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.743+0000 7fa314965640 1 --2- 172.21.15.1:0/347610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3101591f0 0x7fa3101472c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.743+0000 7fa314965640 1 --2- 172.21.15.1:0/347610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2fc004d40 0x7fa3100cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.743+0000 7fa314965640 1 -- 172.21.15.1:0/347610 >> 172.21.15.1:0/347610 conn(0x7fa310066d00 msgr2=0x7fa310153030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:20.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.743+0000 7fa314965640 1 -- 172.21.15.1:0/347610 shutdown_connections 2024-03-06T07:21:20.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.743+0000 7fa314965640 1 -- 172.21.15.1:0/347610 wait complete. 2024-03-06T07:21:20.757 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:21:20.757 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:21:20.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.860+0000 7f963d2eb640 1 Processor -- start 2024-03-06T07:21:20.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.860+0000 7f963d2eb640 1 -- start start 2024-03-06T07:21:20.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.861+0000 7f963d2eb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f96380a7380 0x7f963806c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.861+0000 7f963d2eb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9638152d50 con 0x7f96380c75f0 2024-03-06T07:21:20.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.861+0000 7f963d2eb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9638151f90 con 0x7f963806c640 2024-03-06T07:21:20.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.861+0000 7f963d2eb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9638152270 con 0x7f96380a7380 2024-03-06T07:21:20.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.861+0000 7f96377fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f96380c75f0 0x7f963815c700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38618/0 (socket says 172.21.15.1:38618) 2024-03-06T07:21:20.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.861+0000 7f96377fe640 1 -- 172.21.15.1:0/2546581094 learned_addr learned my addr 172.21.15.1:0/2546581094 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:20.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.861+0000 7f9636ffd640 1 --2- 172.21.15.1:0/2546581094 >> v2:172.21.15.98:3300/0 conn(0x7f96380a7380 0x7f963806c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.861+0000 7f9635ffb640 1 -- 172.21.15.1:0/2546581094 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137063215 0 0) 0x7f9638152d50 con 0x7f96380c75f0 2024-03-06T07:21:20.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.861+0000 7f9635ffb640 1 -- 172.21.15.1:0/2546581094 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9624003520 con 0x7f96380c75f0 2024-03-06T07:21:20.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.861+0000 7f9635ffb640 1 -- 172.21.15.1:0/2546581094 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933390189 0 0) 0x7f9638151f90 con 0x7f963806c640 2024-03-06T07:21:20.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.861+0000 7f9635ffb640 1 -- 172.21.15.1:0/2546581094 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9638152d50 con 0x7f963806c640 2024-03-06T07:21:20.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.862+0000 7f9636ffd640 1 -- 172.21.15.1:0/2546581094 >> v1:172.21.15.98:6789/0 conn(0x7f963806c640 legacy=0x7f963806ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.862+0000 7f9635ffb640 1 -- 172.21.15.1:0/2546581094 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1604847585 0 0) 0x7f9624003520 con 0x7f96380c75f0 2024-03-06T07:21:20.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.862+0000 7f9636ffd640 1 -- 172.21.15.1:0/2546581094 >> v1:172.21.15.1:6789/0 conn(0x7f96380c75f0 legacy=0x7f963815c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.862+0000 7f9636ffd640 1 -- 172.21.15.1:0/2546581094 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9638152550 con 0x7f96380a7380 2024-03-06T07:21:20.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.862+0000 7f9636ffd640 1 --2- 172.21.15.1:0/2546581094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96380a7380 0x7f963806c100 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f96300097a0 tx=0x7f963002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79e8c88f5bed2d46 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.863+0000 7f9635ffb640 1 -- 172.21.15.1:0/2546581094 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9630038070 con 0x7f96380a7380 2024-03-06T07:21:20.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.863+0000 7f9635ffb640 1 -- 172.21.15.1:0/2546581094 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9630031500 con 0x7f96380a7380 2024-03-06T07:21:20.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.863+0000 7f9635ffb640 1 -- 172.21.15.1:0/2546581094 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96280019f0 con 0x7f96380a7380 2024-03-06T07:21:20.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.863+0000 7f963d2eb640 1 -- 172.21.15.1:0/2546581094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96380a7380 msgr2=0x7f963806c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.863+0000 7f963d2eb640 1 --2- 172.21.15.1:0/2546581094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96380a7380 0x7f963806c100 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f96300097a0 tx=0x7f963002b820 comp rx=0 tx=0).stop 2024-03-06T07:21:20.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.863+0000 7f963d2eb640 1 -- 172.21.15.1:0/2546581094 shutdown_connections 2024-03-06T07:21:20.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.863+0000 7f963d2eb640 1 --2- 172.21.15.1:0/2546581094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96380a7380 0x7f963806c100 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.863+0000 7f963d2eb640 1 -- 172.21.15.1:0/2546581094 >> 172.21.15.1:0/2546581094 conn(0x7f96380c45c0 msgr2=0x7f96380c49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:20.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.863+0000 7f963d2eb640 1 -- 172.21.15.1:0/2546581094 shutdown_connections 2024-03-06T07:21:20.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.863+0000 7f963d2eb640 1 -- 172.21.15.1:0/2546581094 wait complete. 2024-03-06T07:21:20.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.864+0000 7f963d2eb640 1 Processor -- start 2024-03-06T07:21:20.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.864+0000 7f963d2eb640 1 -- start start 2024-03-06T07:21:20.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.865+0000 7f963d2eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f963806c640 0x7f96381e1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.865+0000 7f963d2eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96380a7380 0x7f96381ee700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.865+0000 7f963d2eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96380c75f0 0x7f96381eec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.865+0000 7f963d2eb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963806f7a0 con 0x7f963806c640 2024-03-06T07:21:20.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.865+0000 7f963d2eb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96380c6bf0 con 0x7f96380c75f0 2024-03-06T07:21:20.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.865+0000 7f963d2eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9638155370 con 0x7f96380a7380 2024-03-06T07:21:20.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.865+0000 7f9636ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f963806c640 0x7f96381e1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.865+0000 7f9636ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f963806c640 0x7f96381e1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37448/0 (socket says 172.21.15.1:37448) 2024-03-06T07:21:20.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.865+0000 7f9636ffd640 1 -- 172.21.15.1:0/3160230449 learned_addr learned my addr 172.21.15.1:0/3160230449 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:20.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.865+0000 7f96377fe640 1 --2- 172.21.15.1:0/3160230449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96380c75f0 0x7f96381eec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.865+0000 7f96367fc640 1 --2- 172.21.15.1:0/3160230449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96380a7380 0x7f96381ee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.865+0000 7f9636ffd640 1 -- 172.21.15.1:0/3160230449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96380a7380 msgr2=0x7f96381ee700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.865+0000 7f9636ffd640 1 --2- 172.21.15.1:0/3160230449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96380a7380 0x7f96381ee700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.865+0000 7f9636ffd640 1 -- 172.21.15.1:0/3160230449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96380c75f0 msgr2=0x7f96381eec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.865+0000 7f9636ffd640 1 --2- 172.21.15.1:0/3160230449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96380c75f0 0x7f96381eec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.865+0000 7f9636ffd640 1 -- 172.21.15.1:0/3160230449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96380ac840 con 0x7f963806c640 2024-03-06T07:21:20.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.866+0000 7f9636ffd640 1 --2- 172.21.15.1:0/3160230449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f963806c640 0x7f96381e1ea0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9630036040 tx=0x7f96300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.866+0000 7f962f7fe640 1 -- 172.21.15.1:0/3160230449 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9630042070 con 0x7f963806c640 2024-03-06T07:21:20.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.866+0000 7f962f7fe640 1 -- 172.21.15.1:0/3160230449 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f963003d870 con 0x7f963806c640 2024-03-06T07:21:20.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.866+0000 7f963d2eb640 1 -- 172.21.15.1:0/3160230449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963814fd50 con 0x7f963806c640 2024-03-06T07:21:20.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.866+0000 7f962f7fe640 1 -- 172.21.15.1:0/3160230449 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9630038040 con 0x7f963806c640 2024-03-06T07:21:20.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.866+0000 7f963d2eb640 1 -- 172.21.15.1:0/3160230449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9638150640 con 0x7f963806c640 2024-03-06T07:21:20.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.867+0000 7f963d2eb640 1 -- 172.21.15.1:0/3160230449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96380c5480 con 0x7f963806c640 2024-03-06T07:21:20.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.867+0000 7f962f7fe640 1 -- 172.21.15.1:0/3160230449 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9630045070 con 0x7f963806c640 2024-03-06T07:21:20.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.868+0000 7f962f7fe640 1 --2- 172.21.15.1:0/3160230449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f961806b4d0 0x7f961806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.868+0000 7f962f7fe640 1 -- 172.21.15.1:0/3160230449 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f96300ab990 con 0x7f963806c640 2024-03-06T07:21:20.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.868+0000 7f96367fc640 1 --2- 172.21.15.1:0/3160230449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f961806b4d0 0x7f961806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.868+0000 7f96367fc640 1 --2- 172.21.15.1:0/3160230449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f961806b4d0 0x7f961806d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9638151490 tx=0x7f9620009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:20.872+0000 7f962f7fe640 1 -- 172.21.15.1:0/3160230449 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9630030070 con 0x7f963806c640 2024-03-06T07:21:21.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.053+0000 7f963d2eb640 1 -- 172.21.15.1:0/3160230449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9638074ba0 con 0x7f963806c640 2024-03-06T07:21:21.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.144+0000 7f962f7fe640 1 -- 172.21.15.1:0/3160230449 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) v1 ==== 138+0+0 (secure 0 0 0) 0x7f963007aa60 con 0x7f963806c640 2024-03-06T07:21:21.146 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-06T07:21:21.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.146+0000 7f963d2eb640 1 -- 172.21.15.1:0/3160230449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f961806b4d0 msgr2=0x7f961806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.146+0000 7f963d2eb640 1 --2- 172.21.15.1:0/3160230449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f961806b4d0 0x7f961806d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9638151490 tx=0x7f9620009040 comp rx=0 tx=0).stop 2024-03-06T07:21:21.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.146+0000 7f963d2eb640 1 -- 172.21.15.1:0/3160230449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f963806c640 msgr2=0x7f96381e1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.146+0000 7f963d2eb640 1 --2- 172.21.15.1:0/3160230449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f963806c640 0x7f96381e1ea0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9630036040 tx=0x7f96300046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.146+0000 7f963d2eb640 1 -- 172.21.15.1:0/3160230449 shutdown_connections 2024-03-06T07:21:21.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.146+0000 7f963d2eb640 1 --2- 172.21.15.1:0/3160230449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f961806b4d0 0x7f961806d990 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.146+0000 7f963d2eb640 1 --2- 172.21.15.1:0/3160230449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96380c75f0 0x7f96381eec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.146+0000 7f963d2eb640 1 --2- 172.21.15.1:0/3160230449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96380a7380 0x7f96381ee700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.146+0000 7f963d2eb640 1 --2- 172.21.15.1:0/3160230449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f963806c640 0x7f96381e1ea0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.146+0000 7f963d2eb640 1 -- 172.21.15.1:0/3160230449 >> 172.21.15.1:0/3160230449 conn(0x7f96380c45c0 msgr2=0x7f96380c56b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:21.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.146+0000 7f963d2eb640 1 -- 172.21.15.1:0/3160230449 shutdown_connections 2024-03-06T07:21:21.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.146+0000 7f963d2eb640 1 -- 172.21.15.1:0/3160230449 wait complete. 2024-03-06T07:21:21.161 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:21:21.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.268+0000 7f01d4fe2640 1 Processor -- start 2024-03-06T07:21:21.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.268+0000 7f01d4fe2640 1 -- start start 2024-03-06T07:21:21.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.268+0000 7f01d4fe2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01d01505b0 0x7f01d00c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.268+0000 7f01d4fe2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d006f5c0 con 0x7f01d00c1260 2024-03-06T07:21:21.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.268+0000 7f01d4fe2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d0152f80 con 0x7f01d00c16e0 2024-03-06T07:21:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.268+0000 7f01d4fe2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0150990 con 0x7f01d01505b0 2024-03-06T07:21:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01c7fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f01d00c1260 0x7f01d01566e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38620/0 (socket says 172.21.15.1:38620) 2024-03-06T07:21:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01c7fff640 1 -- 172.21.15.1:0/97951855 learned_addr learned my addr 172.21.15.1:0/97951855 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01ce59c640 1 --2- 172.21.15.1:0/97951855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01d01505b0 0x7f01d00c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01cdd9b640 1 -- 172.21.15.1:0/97951855 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946554300 0 0) 0x7f01d006f5c0 con 0x7f01d00c1260 2024-03-06T07:21:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01cdd9b640 1 -- 172.21.15.1:0/97951855 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01bc003520 con 0x7f01d00c1260 2024-03-06T07:21:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01cdd9b640 1 -- 172.21.15.1:0/97951855 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 810869655 0 0) 0x7f01bc003520 con 0x7f01d00c1260 2024-03-06T07:21:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01cdd9b640 1 -- 172.21.15.1:0/97951855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01d01505b0 msgr2=0x7f01d00c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01cdd9b640 1 --2- 172.21.15.1:0/97951855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01d01505b0 0x7f01d00c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01cdd9b640 1 -- 172.21.15.1:0/97951855 >> v1:172.21.15.98:6789/0 conn(0x7f01d00c16e0 legacy=0x7f01d0156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01cdd9b640 1 -- 172.21.15.1:0/97951855 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d0153240 con 0x7f01d00c1260 2024-03-06T07:21:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01cdd9b640 1 -- 172.21.15.1:0/97951855 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f01b8001670 con 0x7f01d00c1260 2024-03-06T07:21:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01cdd9b640 1 -- 172.21.15.1:0/97951855 >> v1:172.21.15.1:6789/0 conn(0x7f01d00c1260 legacy=0x7f01d01566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01cdd9b640 1 --2- 172.21.15.1:0/97951855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01bc0039e0 0x7f01bc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01cdd9b640 1 --2- 172.21.15.1:0/97951855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01bc004800 0x7f01bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01cdd9b640 1 --2- 172.21.15.1:0/97951855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01d01505b0 0x7f01bc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01cdd9b640 1 -- 172.21.15.1:0/97951855 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0150990 con 0x7f01bc004800 2024-03-06T07:21:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01cdd9b640 1 -- 172.21.15.1:0/97951855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc00baf0 con 0x7f01bc0039e0 2024-03-06T07:21:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01cdd9b640 1 -- 172.21.15.1:0/97951855 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc00bcf0 con 0x7f01d01505b0 2024-03-06T07:21:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01c7fff640 1 --2- 172.21.15.1:0/97951855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01bc004800 0x7f01bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01ce59c640 1 --2- 172.21.15.1:0/97951855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01bc0039e0 0x7f01bc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.269+0000 7f01ced9d640 1 --2- 172.21.15.1:0/97951855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01d01505b0 0x7f01bc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.270+0000 7f01c7fff640 1 -- 172.21.15.1:0/97951855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01d01505b0 msgr2=0x7f01bc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.270+0000 7f01c7fff640 1 --2- 172.21.15.1:0/97951855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01d01505b0 0x7f01bc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.270+0000 7f01c7fff640 1 -- 172.21.15.1:0/97951855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01bc0039e0 msgr2=0x7f01bc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.270+0000 7f01c7fff640 1 --2- 172.21.15.1:0/97951855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01bc0039e0 0x7f01bc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.270+0000 7f01c7fff640 1 -- 172.21.15.1:0/97951855 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01bc003520 con 0x7f01bc004800 2024-03-06T07:21:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.270+0000 7f01c7fff640 1 --2- 172.21.15.1:0/97951855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01bc004800 0x7f01bc004c80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f01b800c240 tx=0x7f01bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae5e219092656551 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:21.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.270+0000 7f01cdd9b640 1 -- 172.21.15.1:0/97951855 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b8007d30 con 0x7f01bc004800 2024-03-06T07:21:21.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.271+0000 7f01cdd9b640 1 -- 172.21.15.1:0/97951855 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b8030070 con 0x7f01bc004800 2024-03-06T07:21:21.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.271+0000 7f01cdd9b640 1 -- 172.21.15.1:0/97951855 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b802cd40 con 0x7f01bc004800 2024-03-06T07:21:21.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.271+0000 7f01d4fe2640 1 -- 172.21.15.1:0/97951855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01bc004800 msgr2=0x7f01bc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.271+0000 7f01d4fe2640 1 --2- 172.21.15.1:0/97951855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01bc004800 0x7f01bc004c80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f01b800c240 tx=0x7f01bc005830 comp rx=0 tx=0).stop 2024-03-06T07:21:21.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.271+0000 7f01ced9d640 1 -- 172.21.15.1:0/97951855 reap_dead start 2024-03-06T07:21:21.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.271+0000 7f01d4fe2640 1 -- 172.21.15.1:0/97951855 shutdown_connections 2024-03-06T07:21:21.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.271+0000 7f01d4fe2640 1 -- 172.21.15.1:0/97951855 >> 172.21.15.1:0/97951855 conn(0x7f01d00c6860 msgr2=0x7f01d00c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:21.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.271+0000 7f01d4fe2640 1 -- 172.21.15.1:0/97951855 shutdown_connections 2024-03-06T07:21:21.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.271+0000 7f01d4fe2640 1 -- 172.21.15.1:0/97951855 wait complete. 2024-03-06T07:21:21.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.272+0000 7f01d4fe2640 1 Processor -- start 2024-03-06T07:21:21.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.272+0000 7f01d4fe2640 1 -- start start 2024-03-06T07:21:21.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.273+0000 7f01d4fe2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01bc004800 0x7f01d0145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.273+0000 7f01d4fe2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01d0071f00 0x7f01d01479d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.273+0000 7f01d4fe2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01d0147f10 0x7f01d014c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.273+0000 7f01d4fe2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0153ae0 con 0x7f01d0147f10 2024-03-06T07:21:21.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.273+0000 7f01d4fe2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0153800 con 0x7f01d0071f00 2024-03-06T07:21:21.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.273+0000 7f01d4fe2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d00c4760 con 0x7f01bc004800 2024-03-06T07:21:21.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.273+0000 7f01ced9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01d0147f10 0x7f01d014c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.273+0000 7f01ced9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01d0147f10 0x7f01d014c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37462/0 (socket says 172.21.15.1:37462) 2024-03-06T07:21:21.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.273+0000 7f01ced9d640 1 -- 172.21.15.1:0/2483410823 learned_addr learned my addr 172.21.15.1:0/2483410823 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:21.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.273+0000 7f01c7fff640 1 --2- 172.21.15.1:0/2483410823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01d0071f00 0x7f01d01479d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.273+0000 7f01ce59c640 1 --2- 172.21.15.1:0/2483410823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01bc004800 0x7f01d0145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.273+0000 7f01ced9d640 1 -- 172.21.15.1:0/2483410823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01bc004800 msgr2=0x7f01d0145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.273+0000 7f01ced9d640 1 --2- 172.21.15.1:0/2483410823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01bc004800 0x7f01d0145f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.273+0000 7f01ced9d640 1 -- 172.21.15.1:0/2483410823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01d0071f00 msgr2=0x7f01d01479d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.273+0000 7f01ced9d640 1 --2- 172.21.15.1:0/2483410823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01d0071f00 0x7f01d01479d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.273+0000 7f01ced9d640 1 -- 172.21.15.1:0/2483410823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d0072a30 con 0x7f01d0147f10 2024-03-06T07:21:21.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.273+0000 7f01c7fff640 1 --2- 172.21.15.1:0/2483410823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01d0071f00 0x7f01d01479d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:21.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.274+0000 7f01ced9d640 1 --2- 172.21.15.1:0/2483410823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01d0147f10 0x7f01d014c760 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f01c800fae0 tx=0x7f01c8002120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:21.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.274+0000 7f01c6ffd640 1 -- 172.21.15.1:0/2483410823 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c8003b80 con 0x7f01d0147f10 2024-03-06T07:21:21.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.274+0000 7f01c6ffd640 1 -- 172.21.15.1:0/2483410823 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c8018070 con 0x7f01d0147f10 2024-03-06T07:21:21.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.274+0000 7f01d4fe2640 1 -- 172.21.15.1:0/2483410823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d0072d10 con 0x7f01d0147f10 2024-03-06T07:21:21.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.274+0000 7f01c6ffd640 1 -- 172.21.15.1:0/2483410823 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c80050d0 con 0x7f01d0147f10 2024-03-06T07:21:21.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.274+0000 7f01d4fe2640 1 -- 172.21.15.1:0/2483410823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d014c240 con 0x7f01d0147f10 2024-03-06T07:21:21.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.275+0000 7f01d4fe2640 1 -- 172.21.15.1:0/2483410823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d014c440 con 0x7f01d0147f10 2024-03-06T07:21:21.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.275+0000 7f01c6ffd640 1 -- 172.21.15.1:0/2483410823 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01c8005270 con 0x7f01d0147f10 2024-03-06T07:21:21.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.276+0000 7f01c6ffd640 1 --2- 172.21.15.1:0/2483410823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01b006b4d0 0x7f01b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.276+0000 7f01c6ffd640 1 -- 172.21.15.1:0/2483410823 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f01c808a170 con 0x7f01d0147f10 2024-03-06T07:21:21.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.276+0000 7f01ce59c640 1 --2- 172.21.15.1:0/2483410823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01b006b4d0 0x7f01b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.276+0000 7f01ce59c640 1 --2- 172.21.15.1:0/2483410823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01b006b4d0 0x7f01b006d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f01c0009980 tx=0x7f01c0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:21.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.280+0000 7f01c6ffd640 1 -- 172.21.15.1:0/2483410823 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01d014c440 con 0x7f01d0147f10 2024-03-06T07:21:21.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:21.465+0000 7f01d4fe2640 1 -- 172.21.15.1:0/2483410823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f01d0067fd0 con 0x7f01d0147f10 2024-03-06T07:21:22.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.151+0000 7f01c6ffd640 1 -- 172.21.15.1:0/2483410823 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) v1 ==== 170+0+0 (secure 0 0 0) 0x7f01c801a310 con 0x7f01d0147f10 2024-03-06T07:21:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-06T07:21:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.156+0000 7f01d4fe2640 1 -- 172.21.15.1:0/2483410823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01b006b4d0 msgr2=0x7f01b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.156+0000 7f01d4fe2640 1 --2- 172.21.15.1:0/2483410823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01b006b4d0 0x7f01b006d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f01c0009980 tx=0x7f01c0008040 comp rx=0 tx=0).stop 2024-03-06T07:21:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.156+0000 7f01d4fe2640 1 -- 172.21.15.1:0/2483410823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01d0147f10 msgr2=0x7f01d014c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.156+0000 7f01d4fe2640 1 --2- 172.21.15.1:0/2483410823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01d0147f10 0x7f01d014c760 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f01c800fae0 tx=0x7f01c8002120 comp rx=0 tx=0).stop 2024-03-06T07:21:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.157+0000 7f01d4fe2640 1 -- 172.21.15.1:0/2483410823 shutdown_connections 2024-03-06T07:21:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.157+0000 7f01d4fe2640 1 --2- 172.21.15.1:0/2483410823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01b006b4d0 0x7f01b006d990 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.157+0000 7f01d4fe2640 1 --2- 172.21.15.1:0/2483410823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01d0147f10 0x7f01d014c760 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.157+0000 7f01d4fe2640 1 --2- 172.21.15.1:0/2483410823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01d0071f00 0x7f01d01479d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.157+0000 7f01d4fe2640 1 --2- 172.21.15.1:0/2483410823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01bc004800 0x7f01d0145f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.157+0000 7f01d4fe2640 1 -- 172.21.15.1:0/2483410823 >> 172.21.15.1:0/2483410823 conn(0x7f01d00c6860 msgr2=0x7f01d00c6010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.158+0000 7f01d4fe2640 1 -- 172.21.15.1:0/2483410823 shutdown_connections 2024-03-06T07:21:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.158+0000 7f01d4fe2640 1 -- 172.21.15.1:0/2483410823 wait complete. 2024-03-06T07:21:22.185 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:21:22.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.292+0000 7f1623e4e640 1 Processor -- start 2024-03-06T07:21:22.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.293+0000 7f1623e4e640 1 -- start start 2024-03-06T07:21:22.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.293+0000 7f1623e4e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f161c072f20 0x7f161c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:22.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.293+0000 7f1623e4e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f161c0a6140 con 0x7f161c0c3360 2024-03-06T07:21:22.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.293+0000 7f1623e4e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f161c151aa0 con 0x7f161c06f2f0 2024-03-06T07:21:22.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.293+0000 7f1623e4e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f161c1526b0 con 0x7f161c072f20 2024-03-06T07:21:22.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.293+0000 7f16223eb640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f161c0c3360 0x7f161c0c37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38624/0 (socket says 172.21.15.1:38624) 2024-03-06T07:21:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.294+0000 7f16223eb640 1 -- 172.21.15.1:0/2794967783 learned_addr learned my addr 172.21.15.1:0/2794967783 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.294+0000 7f1621bea640 1 --2- 172.21.15.1:0/2794967783 >> v2:172.21.15.98:3300/0 conn(0x7f161c072f20 0x7f161c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.294+0000 7f1620be8640 1 -- 172.21.15.1:0/2794967783 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938684156 0 0) 0x7f161c0a6140 con 0x7f161c0c3360 2024-03-06T07:21:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.294+0000 7f1620be8640 1 -- 172.21.15.1:0/2794967783 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1604005730 con 0x7f161c0c3360 2024-03-06T07:21:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.294+0000 7f1621bea640 1 -- 172.21.15.1:0/2794967783 >> v1:172.21.15.98:6789/0 conn(0x7f161c06f2f0 legacy=0x7f161c150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.294+0000 7f1621bea640 1 -- 172.21.15.1:0/2794967783 >> v1:172.21.15.1:6789/0 conn(0x7f161c0c3360 legacy=0x7f161c0c37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.294+0000 7f1621bea640 1 -- 172.21.15.1:0/2794967783 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f161c151d80 con 0x7f161c072f20 2024-03-06T07:21:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.295+0000 7f1621bea640 1 --2- 172.21.15.1:0/2794967783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f161c072f20 0x7f161c06edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1614009d90 tx=0x7f161402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4d7bf030d1f7cbd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.295+0000 7f1620be8640 1 -- 172.21.15.1:0/2794967783 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1614039070 con 0x7f161c072f20 2024-03-06T07:21:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.295+0000 7f1620be8640 1 -- 172.21.15.1:0/2794967783 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f161402cc60 con 0x7f161c072f20 2024-03-06T07:21:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.295+0000 7f1620be8640 1 -- 172.21.15.1:0/2794967783 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1614034640 con 0x7f161c072f20 2024-03-06T07:21:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.295+0000 7f1623e4e640 1 -- 172.21.15.1:0/2794967783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f161c072f20 msgr2=0x7f161c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.295+0000 7f1623e4e640 1 --2- 172.21.15.1:0/2794967783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f161c072f20 0x7f161c06edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1614009d90 tx=0x7f161402beb0 comp rx=0 tx=0).stop 2024-03-06T07:21:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.296+0000 7f1623e4e640 1 -- 172.21.15.1:0/2794967783 shutdown_connections 2024-03-06T07:21:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.296+0000 7f1623e4e640 1 --2- 172.21.15.1:0/2794967783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f161c072f20 0x7f161c06edb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.296+0000 7f1623e4e640 1 -- 172.21.15.1:0/2794967783 >> 172.21.15.1:0/2794967783 conn(0x7f161c0c6860 msgr2=0x7f161c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.296+0000 7f1623e4e640 1 -- 172.21.15.1:0/2794967783 shutdown_connections 2024-03-06T07:21:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.296+0000 7f1623e4e640 1 -- 172.21.15.1:0/2794967783 wait complete. 2024-03-06T07:21:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.297+0000 7f1623e4e640 1 Processor -- start 2024-03-06T07:21:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.297+0000 7f1623e4e640 1 -- start start 2024-03-06T07:21:22.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.297+0000 7f1623e4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f161c06f2f0 0x7f161c154970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:22.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.297+0000 7f1623e4e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f161c072f20 0x7f161c154eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:22.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.297+0000 7f1623e4e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f161c0c3360 0x7f161c1e6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.297+0000 7f1623e4e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c1516f0 con 0x7f161c0c3360 2024-03-06T07:21:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.297+0000 7f1623e4e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c152340 con 0x7f161c06f2f0 2024-03-06T07:21:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.297+0000 7f1623e4e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f161c153550 con 0x7f161c072f20 2024-03-06T07:21:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.298+0000 7f16223eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f161c0c3360 0x7f161c1e6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.298+0000 7f16223eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f161c0c3360 0x7f161c1e6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37466/0 (socket says 172.21.15.1:37466) 2024-03-06T07:21:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.298+0000 7f16223eb640 1 -- 172.21.15.1:0/3478876700 learned_addr learned my addr 172.21.15.1:0/3478876700 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.298+0000 7f1621bea640 1 --2- 172.21.15.1:0/3478876700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f161c06f2f0 0x7f161c154970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.298+0000 7f16213e9640 1 --2- 172.21.15.1:0/3478876700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f161c072f20 0x7f161c154eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.298+0000 7f16223eb640 1 -- 172.21.15.1:0/3478876700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f161c072f20 msgr2=0x7f161c154eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.298+0000 7f16223eb640 1 --2- 172.21.15.1:0/3478876700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f161c072f20 0x7f161c154eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.298+0000 7f16223eb640 1 -- 172.21.15.1:0/3478876700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f161c06f2f0 msgr2=0x7f161c154970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.298+0000 7f16223eb640 1 --2- 172.21.15.1:0/3478876700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f161c06f2f0 0x7f161c154970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.298+0000 7f16223eb640 1 -- 172.21.15.1:0/3478876700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f161c145fb0 con 0x7f161c0c3360 2024-03-06T07:21:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.298+0000 7f16223eb640 1 --2- 172.21.15.1:0/3478876700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f161c0c3360 0x7f161c1e6920 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f160c00c390 tx=0x7f160c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.298+0000 7f161a7fc640 1 -- 172.21.15.1:0/3478876700 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160c017070 con 0x7f161c0c3360 2024-03-06T07:21:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.298+0000 7f161a7fc640 1 -- 172.21.15.1:0/3478876700 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160c005ce0 con 0x7f161c0c3360 2024-03-06T07:21:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.298+0000 7f1623e4e640 1 -- 172.21.15.1:0/3478876700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f161c146600 con 0x7f161c0c3360 2024-03-06T07:21:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.298+0000 7f161a7fc640 1 -- 172.21.15.1:0/3478876700 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160c007310 con 0x7f161c0c3360 2024-03-06T07:21:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.298+0000 7f1623e4e640 1 -- 172.21.15.1:0/3478876700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f161c152a50 con 0x7f161c0c3360 2024-03-06T07:21:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.299+0000 7f1623e4e640 1 -- 172.21.15.1:0/3478876700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f161c067fd0 con 0x7f161c0c3360 2024-03-06T07:21:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.300+0000 7f161a7fc640 1 -- 172.21.15.1:0/3478876700 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f160c01a070 con 0x7f161c0c3360 2024-03-06T07:21:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.300+0000 7f161a7fc640 1 --2- 172.21.15.1:0/3478876700 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f160806b4d0 0x7f160806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.300+0000 7f161a7fc640 1 -- 172.21.15.1:0/3478876700 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f160c0895d0 con 0x7f161c0c3360 2024-03-06T07:21:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.300+0000 7f1621bea640 1 --2- 172.21.15.1:0/3478876700 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f160806b4d0 0x7f160806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.301+0000 7f1621bea640 1 --2- 172.21.15.1:0/3478876700 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f160806b4d0 0x7f160806d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1614036000 tx=0x7f1614030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.305+0000 7f161a7fc640 1 -- 172.21.15.1:0/3478876700 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f160c057d60 con 0x7f161c0c3360 2024-03-06T07:21:22.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:22.526+0000 7f1623e4e640 1 -- 172.21.15.1:0/3478876700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f161c06eb90 con 0x7f161c0c3360 2024-03-06T07:21:23.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.161+0000 7f161a7fc640 1 -- 172.21.15.1:0/3478876700 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f160c05be00 con 0x7f161c0c3360 2024-03-06T07:21:23.183 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 5 and data pool 6 2024-03-06T07:21:23.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.185+0000 7f1623e4e640 1 -- 172.21.15.1:0/3478876700 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f160806b4d0 msgr2=0x7f160806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.185+0000 7f1623e4e640 1 --2- 172.21.15.1:0/3478876700 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f160806b4d0 0x7f160806d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1614036000 tx=0x7f1614030000 comp rx=0 tx=0).stop 2024-03-06T07:21:23.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.186+0000 7f1623e4e640 1 -- 172.21.15.1:0/3478876700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f161c0c3360 msgr2=0x7f161c1e6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.186+0000 7f1623e4e640 1 --2- 172.21.15.1:0/3478876700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f161c0c3360 0x7f161c1e6920 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f160c00c390 tx=0x7f160c00c850 comp rx=0 tx=0).stop 2024-03-06T07:21:23.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.186+0000 7f1623e4e640 1 -- 172.21.15.1:0/3478876700 shutdown_connections 2024-03-06T07:21:23.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.186+0000 7f1623e4e640 1 --2- 172.21.15.1:0/3478876700 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f160806b4d0 0x7f160806d990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.186+0000 7f1623e4e640 1 --2- 172.21.15.1:0/3478876700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f161c0c3360 0x7f161c1e6920 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.186+0000 7f1623e4e640 1 --2- 172.21.15.1:0/3478876700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f161c072f20 0x7f161c154eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.186+0000 7f1623e4e640 1 --2- 172.21.15.1:0/3478876700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f161c06f2f0 0x7f161c154970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.186+0000 7f1623e4e640 1 -- 172.21.15.1:0/3478876700 >> 172.21.15.1:0/3478876700 conn(0x7f161c0c6860 msgr2=0x7f161c0c6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:23.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.186+0000 7f1623e4e640 1 -- 172.21.15.1:0/3478876700 shutdown_connections 2024-03-06T07:21:23.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.186+0000 7f1623e4e640 1 -- 172.21.15.1:0/3478876700 wait complete. 2024-03-06T07:21:23.202 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:23.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.306+0000 7fc5d10ed640 1 Processor -- start 2024-03-06T07:21:23.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.306+0000 7fc5d10ed640 1 -- start start 2024-03-06T07:21:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.306+0000 7fc5d10ed640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc5cc152690 0x7fc5cc155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.306+0000 7fc5d10ed640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5cc0c6730 0x7fc5cc0c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.306+0000 7fc5d10ed640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc1561e0 con 0x7fc5cc0c5d10 2024-03-06T07:21:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.306+0000 7fc5d10ed640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5cc0c3300 con 0x7fc5cc0c6730 2024-03-06T07:21:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.306+0000 7fc5d10ed640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5cc0c35a0 con 0x7fc5cc152690 2024-03-06T07:21:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.306+0000 7fc5c3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5cc0c6730 0x7fc5cc0c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.307+0000 7fc5c3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5cc0c6730 0x7fc5cc0c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37480/0 (socket says 172.21.15.1:37480) 2024-03-06T07:21:23.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.307+0000 7fc5c3fff640 1 -- 172.21.15.1:0/1223189320 learned_addr learned my addr 172.21.15.1:0/1223189320 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:23.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.307+0000 7fc5cad9d640 1 --2- 172.21.15.1:0/1223189320 >> v2:172.21.15.98:3300/0 conn(0x7fc5cc152690 0x7fc5cc155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.307+0000 7fc5c3fff640 1 -- 172.21.15.1:0/1223189320 >> v2:172.21.15.98:3300/0 conn(0x7fc5cc152690 msgr2=0x7fc5cc155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:23.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.307+0000 7fc5c3fff640 1 --2- 172.21.15.1:0/1223189320 >> v2:172.21.15.98:3300/0 conn(0x7fc5cc152690 0x7fc5cc155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.307+0000 7fc5c3fff640 1 -- 172.21.15.1:0/1223189320 >> v1:172.21.15.98:6789/0 conn(0x7fc5cc0c5d10 legacy=0x7fc5cc156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.307+0000 7fc5c3fff640 1 -- 172.21.15.1:0/1223189320 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc0c3880 con 0x7fc5cc0c6730 2024-03-06T07:21:23.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.308+0000 7fc5c3fff640 1 --2- 172.21.15.1:0/1223189320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5cc0c6730 0x7fc5cc0c57d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc0ab5f0 tx=0x7fc5b402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41db6d1dd244ed2d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:23.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.308+0000 7fc5ca59c640 1 -- 172.21.15.1:0/1223189320 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4035020 con 0x7fc5cc0c6730 2024-03-06T07:21:23.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.308+0000 7fc5ca59c640 1 -- 172.21.15.1:0/1223189320 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b4031870 con 0x7fc5cc0c6730 2024-03-06T07:21:23.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.308+0000 7fc5ca59c640 1 -- 172.21.15.1:0/1223189320 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4031c40 con 0x7fc5cc0c6730 2024-03-06T07:21:23.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.308+0000 7fc5d10ed640 1 -- 172.21.15.1:0/1223189320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5cc0c6730 msgr2=0x7fc5cc0c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.308+0000 7fc5d10ed640 1 --2- 172.21.15.1:0/1223189320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5cc0c6730 0x7fc5cc0c57d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc0ab5f0 tx=0x7fc5b402b910 comp rx=0 tx=0).stop 2024-03-06T07:21:23.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.308+0000 7fc5d10ed640 1 -- 172.21.15.1:0/1223189320 shutdown_connections 2024-03-06T07:21:23.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.308+0000 7fc5d10ed640 1 --2- 172.21.15.1:0/1223189320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5cc0c6730 0x7fc5cc0c57d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.308+0000 7fc5d10ed640 1 --2- 172.21.15.1:0/1223189320 >> v2:172.21.15.98:3300/0 conn(0x7fc5cc152690 0x7fc5cc155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.308+0000 7fc5d10ed640 1 -- 172.21.15.1:0/1223189320 >> 172.21.15.1:0/1223189320 conn(0x7fc5cc06b410 msgr2=0x7fc5cc06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:23.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.308+0000 7fc5d10ed640 1 -- 172.21.15.1:0/1223189320 shutdown_connections 2024-03-06T07:21:23.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.308+0000 7fc5d10ed640 1 -- 172.21.15.1:0/1223189320 wait complete. 2024-03-06T07:21:23.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.310+0000 7fc5d10ed640 1 Processor -- start 2024-03-06T07:21:23.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.310+0000 7fc5d10ed640 1 -- start start 2024-03-06T07:21:23.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.310+0000 7fc5d10ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5cc0c5d10 0x7fc5cc153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.310+0000 7fc5d10ed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5cc0c6730 0x7fc5cc0c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.310+0000 7fc5d10ed640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5cc152690 0x7fc5cc0aa920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.310+0000 7fc5d10ed640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc152a70 con 0x7fc5cc152690 2024-03-06T07:21:23.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.310+0000 7fc5d10ed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc1567b0 con 0x7fc5cc0c5d10 2024-03-06T07:21:23.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.310+0000 7fc5d10ed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc152bf0 con 0x7fc5cc0c6730 2024-03-06T07:21:23.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.310+0000 7fc5cb59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5cc152690 0x7fc5cc0aa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.310+0000 7fc5cb59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5cc152690 0x7fc5cc0aa920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37492/0 (socket says 172.21.15.1:37492) 2024-03-06T07:21:23.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.310+0000 7fc5cb59e640 1 -- 172.21.15.1:0/836935606 learned_addr learned my addr 172.21.15.1:0/836935606 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:23.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.310+0000 7fc5cad9d640 1 --2- 172.21.15.1:0/836935606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5cc0c5d10 0x7fc5cc153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.310+0000 7fc5c3fff640 1 --2- 172.21.15.1:0/836935606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5cc0c6730 0x7fc5cc0c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.311+0000 7fc5cb59e640 1 -- 172.21.15.1:0/836935606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5cc0c6730 msgr2=0x7fc5cc0c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.311+0000 7fc5cb59e640 1 --2- 172.21.15.1:0/836935606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5cc0c6730 0x7fc5cc0c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.311+0000 7fc5cb59e640 1 -- 172.21.15.1:0/836935606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5cc0c5d10 msgr2=0x7fc5cc153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.311+0000 7fc5cb59e640 1 --2- 172.21.15.1:0/836935606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5cc0c5d10 0x7fc5cc153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.311+0000 7fc5cb59e640 1 -- 172.21.15.1:0/836935606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc074c10 con 0x7fc5cc152690 2024-03-06T07:21:23.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.311+0000 7fc5cb59e640 1 --2- 172.21.15.1:0/836935606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5cc152690 0x7fc5cc0aa920 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400c6b0 tx=0x7fc5c400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:23.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.311+0000 7fc5c37fe640 1 -- 172.21.15.1:0/836935606 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4003b70 con 0x7fc5cc152690 2024-03-06T07:21:23.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.311+0000 7fc5c37fe640 1 -- 172.21.15.1:0/836935606 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c4003d30 con 0x7fc5cc152690 2024-03-06T07:21:23.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.311+0000 7fc5c37fe640 1 -- 172.21.15.1:0/836935606 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4005310 con 0x7fc5cc152690 2024-03-06T07:21:23.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.311+0000 7fc5d10ed640 1 -- 172.21.15.1:0/836935606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5cc14f040 con 0x7fc5cc152690 2024-03-06T07:21:23.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.311+0000 7fc5d10ed640 1 -- 172.21.15.1:0/836935606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5cc1535c0 con 0x7fc5cc152690 2024-03-06T07:21:23.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.312+0000 7fc5d10ed640 1 -- 172.21.15.1:0/836935606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5cc067fd0 con 0x7fc5cc152690 2024-03-06T07:21:23.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.312+0000 7fc5c37fe640 1 -- 172.21.15.1:0/836935606 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5c40054b0 con 0x7fc5cc152690 2024-03-06T07:21:23.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.313+0000 7fc5c37fe640 1 --2- 172.21.15.1:0/836935606 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5ac06b4d0 0x7fc5ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.313+0000 7fc5c37fe640 1 -- 172.21.15.1:0/836935606 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc5c408a020 con 0x7fc5cc152690 2024-03-06T07:21:23.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.313+0000 7fc5cad9d640 1 --2- 172.21.15.1:0/836935606 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5ac06b4d0 0x7fc5ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.314+0000 7fc5cad9d640 1 --2- 172.21.15.1:0/836935606 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5ac06b4d0 0x7fc5ac06d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc14f5c0 tx=0x7fc5bc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:23.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.317+0000 7fc5c37fe640 1 -- 172.21.15.1:0/836935606 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5cc074c10 con 0x7fc5cc152690 2024-03-06T07:21:23.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.498+0000 7fc5d10ed640 1 -- 172.21.15.1:0/836935606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5cc0c2c40 con 0x7fc5cc152690 2024-03-06T07:21:23.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.499+0000 7fc5c37fe640 1 -- 172.21.15.1:0/836935606 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16835 (secure 0 0 0) 0x7fc5cc1561e0 con 0x7fc5cc152690 2024-03-06T07:21:23.501 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:23.502 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":38,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:21:22.528229+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:21.073384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:21.484387+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.501+0000 7fc5d10ed640 1 -- 172.21.15.1:0/836935606 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5ac06b4d0 msgr2=0x7fc5ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.501+0000 7fc5d10ed640 1 --2- 172.21.15.1:0/836935606 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5ac06b4d0 0x7fc5ac06d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc14f5c0 tx=0x7fc5bc008040 comp rx=0 tx=0).stop 2024-03-06T07:21:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.501+0000 7fc5d10ed640 1 -- 172.21.15.1:0/836935606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5cc152690 msgr2=0x7fc5cc0aa920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.501+0000 7fc5d10ed640 1 --2- 172.21.15.1:0/836935606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5cc152690 0x7fc5cc0aa920 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400c6b0 tx=0x7fc5c400cb70 comp rx=0 tx=0).stop 2024-03-06T07:21:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.501+0000 7fc5d10ed640 1 -- 172.21.15.1:0/836935606 shutdown_connections 2024-03-06T07:21:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.501+0000 7fc5d10ed640 1 --2- 172.21.15.1:0/836935606 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5ac06b4d0 0x7fc5ac06d990 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.501+0000 7fc5d10ed640 1 --2- 172.21.15.1:0/836935606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5cc152690 0x7fc5cc0aa920 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.501+0000 7fc5d10ed640 1 --2- 172.21.15.1:0/836935606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5cc0c6730 0x7fc5cc0c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.501+0000 7fc5d10ed640 1 --2- 172.21.15.1:0/836935606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5cc0c5d10 0x7fc5cc153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.501+0000 7fc5d10ed640 1 -- 172.21.15.1:0/836935606 >> 172.21.15.1:0/836935606 conn(0x7fc5cc06b410 msgr2=0x7fc5cc0a84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.501+0000 7fc5d10ed640 1 -- 172.21.15.1:0/836935606 shutdown_connections 2024-03-06T07:21:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.501+0000 7fc5d10ed640 1 -- 172.21.15.1:0/836935606 wait complete. 2024-03-06T07:21:23.515 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:23.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.614+0000 7f25d596b640 1 Processor -- start 2024-03-06T07:21:23.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.614+0000 7f25d596b640 1 -- start start 2024-03-06T07:21:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.615+0000 7f25d596b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25d0152820 0x7f25d00c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.615+0000 7f25d596b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d00c7180 con 0x7f25d00af6f0 2024-03-06T07:21:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.615+0000 7f25d596b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d0155460 con 0x7f25d00c0670 2024-03-06T07:21:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.615+0000 7f25d596b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d00c6bc0 con 0x7f25d0152820 2024-03-06T07:21:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.615+0000 7f25cf7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f25d00af6f0 0x7f25d015fb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38628/0 (socket says 172.21.15.1:38628) 2024-03-06T07:21:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.615+0000 7f25cf7fe640 1 -- 172.21.15.1:0/2599143758 learned_addr learned my addr 172.21.15.1:0/2599143758 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.615+0000 7f25ceffd640 1 --2- 172.21.15.1:0/2599143758 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25d0152820 0x7f25d00c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.615+0000 7f25cdffb640 1 -- 172.21.15.1:0/2599143758 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751825872 0 0) 0x7f25d00c7180 con 0x7f25d00af6f0 2024-03-06T07:21:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.615+0000 7f25cdffb640 1 -- 172.21.15.1:0/2599143758 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25bc003520 con 0x7f25d00af6f0 2024-03-06T07:21:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.616+0000 7f25cdffb640 1 -- 172.21.15.1:0/2599143758 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381218421 0 0) 0x7f25d0155460 con 0x7f25d00c0670 2024-03-06T07:21:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.616+0000 7f25ceffd640 1 -- 172.21.15.1:0/2599143758 >> v1:172.21.15.98:6789/0 conn(0x7f25d00c0670 legacy=0x7f25d00c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.616+0000 7f25ceffd640 1 -- 172.21.15.1:0/2599143758 >> v1:172.21.15.1:6789/0 conn(0x7f25d00af6f0 legacy=0x7f25d015fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.616+0000 7f25ceffd640 1 -- 172.21.15.1:0/2599143758 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d00c6ea0 con 0x7f25d0152820 2024-03-06T07:21:23.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.616+0000 7f25ceffd640 1 --2- 172.21.15.1:0/2599143758 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25d0152820 0x7f25d00c0130 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f25c80097d0 tx=0x7f25c802b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6186d64830cc41d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:23.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.617+0000 7f25cdffb640 1 -- 172.21.15.1:0/2599143758 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c8037070 con 0x7f25d0152820 2024-03-06T07:21:23.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.617+0000 7f25cdffb640 1 -- 172.21.15.1:0/2599143758 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c8030510 con 0x7f25d0152820 2024-03-06T07:21:23.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.617+0000 7f25cdffb640 1 -- 172.21.15.1:0/2599143758 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c00019f0 con 0x7f25d0152820 2024-03-06T07:21:23.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.617+0000 7f25d596b640 1 -- 172.21.15.1:0/2599143758 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25d0152820 msgr2=0x7f25d00c0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.617+0000 7f25d596b640 1 --2- 172.21.15.1:0/2599143758 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25d0152820 0x7f25d00c0130 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f25c80097d0 tx=0x7f25c802b730 comp rx=0 tx=0).stop 2024-03-06T07:21:23.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.617+0000 7f25d596b640 1 -- 172.21.15.1:0/2599143758 shutdown_connections 2024-03-06T07:21:23.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.617+0000 7f25d596b640 1 --2- 172.21.15.1:0/2599143758 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25d0152820 0x7f25d00c0130 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.617+0000 7f25d596b640 1 -- 172.21.15.1:0/2599143758 >> 172.21.15.1:0/2599143758 conn(0x7f25d0072720 msgr2=0x7f25d0072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:23.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.617+0000 7f25d596b640 1 -- 172.21.15.1:0/2599143758 shutdown_connections 2024-03-06T07:21:23.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.617+0000 7f25d596b640 1 -- 172.21.15.1:0/2599143758 wait complete. 2024-03-06T07:21:23.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.618+0000 7f25d596b640 1 Processor -- start 2024-03-06T07:21:23.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.618+0000 7f25d596b640 1 -- start start 2024-03-06T07:21:23.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.619+0000 7f25d596b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25d00af6f0 0x7f25d00c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.619+0000 7f25d596b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25d00c0670 0x7f25d00c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.619+0000 7f25d596b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25d0152820 0x7f25d01e7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.619+0000 7f25d596b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d00c7c80 con 0x7f25d00c0670 2024-03-06T07:21:23.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.619+0000 7f25d596b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d00c7b00 con 0x7f25d0152820 2024-03-06T07:21:23.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.619+0000 7f25d596b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d0152c00 con 0x7f25d00af6f0 2024-03-06T07:21:23.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.619+0000 7f25ce7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25d00c0670 0x7f25d00c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.619+0000 7f25ce7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25d00c0670 0x7f25d00c9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37506/0 (socket says 172.21.15.1:37506) 2024-03-06T07:21:23.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.619+0000 7f25ce7fc640 1 -- 172.21.15.1:0/297955314 learned_addr learned my addr 172.21.15.1:0/297955314 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:23.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.619+0000 7f25ceffd640 1 --2- 172.21.15.1:0/297955314 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25d00af6f0 0x7f25d00c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.619+0000 7f25cf7fe640 1 --2- 172.21.15.1:0/297955314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25d0152820 0x7f25d01e7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.619+0000 7f25ce7fc640 1 -- 172.21.15.1:0/297955314 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25d00af6f0 msgr2=0x7f25d00c9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.619+0000 7f25ce7fc640 1 --2- 172.21.15.1:0/297955314 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25d00af6f0 0x7f25d00c9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.620+0000 7f25ce7fc640 1 -- 172.21.15.1:0/297955314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25d0152820 msgr2=0x7f25d01e7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.620+0000 7f25ce7fc640 1 --2- 172.21.15.1:0/297955314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25d0152820 0x7f25d01e7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.620+0000 7f25ce7fc640 1 -- 172.21.15.1:0/297955314 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d00c91c0 con 0x7f25d00c0670 2024-03-06T07:21:23.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.620+0000 7f25ce7fc640 1 --2- 172.21.15.1:0/297955314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25d00c0670 0x7f25d00c9ea0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f25b800c520 tx=0x7f25b800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:23.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.620+0000 7f25c7fff640 1 -- 172.21.15.1:0/297955314 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b8017070 con 0x7f25d00c0670 2024-03-06T07:21:23.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.620+0000 7f25c7fff640 1 -- 172.21.15.1:0/297955314 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b80035f0 con 0x7f25d00c0670 2024-03-06T07:21:23.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.620+0000 7f25c7fff640 1 -- 172.21.15.1:0/297955314 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b80039e0 con 0x7f25d00c0670 2024-03-06T07:21:23.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.620+0000 7f25d596b640 1 -- 172.21.15.1:0/297955314 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d00ca5f0 con 0x7f25d00c0670 2024-03-06T07:21:23.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.620+0000 7f25d596b640 1 -- 172.21.15.1:0/297955314 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d00cabb0 con 0x7f25d00c0670 2024-03-06T07:21:23.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.621+0000 7f25d596b640 1 -- 172.21.15.1:0/297955314 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d00686e0 con 0x7f25d00c0670 2024-03-06T07:21:23.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.621+0000 7f25c7fff640 1 -- 172.21.15.1:0/297955314 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25b801a070 con 0x7f25d00c0670 2024-03-06T07:21:23.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.622+0000 7f25c7fff640 1 --2- 172.21.15.1:0/297955314 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f25b006b4d0 0x7f25b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.622+0000 7f25c7fff640 1 -- 172.21.15.1:0/297955314 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f25b8089640 con 0x7f25d00c0670 2024-03-06T07:21:23.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.622+0000 7f25ceffd640 1 --2- 172.21.15.1:0/297955314 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f25b006b4d0 0x7f25b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.622+0000 7f25ceffd640 1 --2- 172.21.15.1:0/297955314 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f25b006b4d0 0x7f25b006d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f25c8034000 tx=0x7f25c8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:23.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.626+0000 7f25c7fff640 1 -- 172.21.15.1:0/297955314 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25d00c91c0 con 0x7f25d00c0670 2024-03-06T07:21:23.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.808+0000 7f25d596b640 1 -- 172.21.15.1:0/297955314 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25d004f410 con 0x7f25d00c0670 2024-03-06T07:21:23.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.809+0000 7f25c7fff640 1 -- 172.21.15.1:0/297955314 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16835 (secure 0 0 0) 0x7f25b805e0c0 con 0x7f25d00c0670 2024-03-06T07:21:23.811 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:23.812 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":38,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:21:22.528229+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:21.073384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:21.484387+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:23.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.811+0000 7f25d596b640 1 -- 172.21.15.1:0/297955314 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f25b006b4d0 msgr2=0x7f25b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.811+0000 7f25d596b640 1 --2- 172.21.15.1:0/297955314 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f25b006b4d0 0x7f25b006d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f25c8034000 tx=0x7f25c8008030 comp rx=0 tx=0).stop 2024-03-06T07:21:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.811+0000 7f25d596b640 1 -- 172.21.15.1:0/297955314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25d00c0670 msgr2=0x7f25d00c9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.811+0000 7f25d596b640 1 --2- 172.21.15.1:0/297955314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25d00c0670 0x7f25d00c9ea0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f25b800c520 tx=0x7f25b800c9e0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.811+0000 7f25d596b640 1 -- 172.21.15.1:0/297955314 shutdown_connections 2024-03-06T07:21:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.812+0000 7f25d596b640 1 --2- 172.21.15.1:0/297955314 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f25b006b4d0 0x7f25b006d990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.812+0000 7f25d596b640 1 --2- 172.21.15.1:0/297955314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25d0152820 0x7f25d01e7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.812+0000 7f25d596b640 1 --2- 172.21.15.1:0/297955314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25d00c0670 0x7f25d00c9ea0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.812+0000 7f25d596b640 1 --2- 172.21.15.1:0/297955314 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25d00af6f0 0x7f25d00c9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.812+0000 7f25d596b640 1 -- 172.21.15.1:0/297955314 >> 172.21.15.1:0/297955314 conn(0x7f25d0072720 msgr2=0x7f25d0153130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.812+0000 7f25d596b640 1 -- 172.21.15.1:0/297955314 shutdown_connections 2024-03-06T07:21:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.812+0000 7f25d596b640 1 -- 172.21.15.1:0/297955314 wait complete. 2024-03-06T07:21:23.825 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:21:23.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.927+0000 7f070f181640 1 Processor -- start 2024-03-06T07:21:23.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.927+0000 7f070f181640 1 -- start start 2024-03-06T07:21:23.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.928+0000 7f070f181640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0708152690 0x7f0708155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.928+0000 7f070f181640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07080c6730 0x7f07080c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.928+0000 7f070f181640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07081561e0 con 0x7f07080c5d10 2024-03-06T07:21:23.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.928+0000 7f070f181640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07080c3300 con 0x7f0708152690 2024-03-06T07:21:23.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.928+0000 7f070f181640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07080c35a0 con 0x7f07080c6730 2024-03-06T07:21:23.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.928+0000 7f0707fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07080c6730 0x7f07080c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.928+0000 7f070cf1d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0708152690 0x7f0708155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.928+0000 7f0707fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07080c6730 0x7f07080c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:55730/0 (socket says 172.21.15.1:55730) 2024-03-06T07:21:23.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.928+0000 7f0707fff640 1 -- 172.21.15.1:0/952847051 learned_addr learned my addr 172.21.15.1:0/952847051 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:23.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.928+0000 7f0707fff640 1 -- 172.21.15.1:0/952847051 >> v2:172.21.15.98:3300/0 conn(0x7f0708152690 msgr2=0x7f0708155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:23.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.928+0000 7f0707fff640 1 --2- 172.21.15.1:0/952847051 >> v2:172.21.15.98:3300/0 conn(0x7f0708152690 0x7f0708155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.928+0000 7f0707fff640 1 -- 172.21.15.1:0/952847051 >> v1:172.21.15.98:6789/0 conn(0x7f07080c5d10 legacy=0x7f0708156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.928+0000 7f0707fff640 1 -- 172.21.15.1:0/952847051 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07080c3880 con 0x7f07080c6730 2024-03-06T07:21:23.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.929+0000 7f0707fff640 1 --2- 172.21.15.1:0/952847051 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07080c6730 0x7f07080c57d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f06f4009870 tx=0x7f06f402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7b80522c5d52ec2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:23.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.929+0000 7f07077fe640 1 -- 172.21.15.1:0/952847051 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f4035020 con 0x7f07080c6730 2024-03-06T07:21:23.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.929+0000 7f07077fe640 1 -- 172.21.15.1:0/952847051 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f4031640 con 0x7f07080c6730 2024-03-06T07:21:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.929+0000 7f07077fe640 1 -- 172.21.15.1:0/952847051 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f402d040 con 0x7f07080c6730 2024-03-06T07:21:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.930+0000 7f070f181640 1 -- 172.21.15.1:0/952847051 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07080c6730 msgr2=0x7f07080c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.930+0000 7f070f181640 1 --2- 172.21.15.1:0/952847051 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07080c6730 0x7f07080c57d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f06f4009870 tx=0x7f06f402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.930+0000 7f070f181640 1 -- 172.21.15.1:0/952847051 shutdown_connections 2024-03-06T07:21:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.930+0000 7f070f181640 1 --2- 172.21.15.1:0/952847051 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07080c6730 0x7f07080c57d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.930+0000 7f070f181640 1 --2- 172.21.15.1:0/952847051 >> v2:172.21.15.98:3300/0 conn(0x7f0708152690 0x7f0708155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.930+0000 7f070f181640 1 -- 172.21.15.1:0/952847051 >> 172.21.15.1:0/952847051 conn(0x7f070806b410 msgr2=0x7f070806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.930+0000 7f070f181640 1 -- 172.21.15.1:0/952847051 shutdown_connections 2024-03-06T07:21:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.930+0000 7f070f181640 1 -- 172.21.15.1:0/952847051 wait complete. 2024-03-06T07:21:23.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.931+0000 7f070f181640 1 Processor -- start 2024-03-06T07:21:23.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.931+0000 7f070f181640 1 -- start start 2024-03-06T07:21:23.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.932+0000 7f070f181640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07080c5d10 0x7f0708070ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.932+0000 7f070f181640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07080c6730 0x7f0708146fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.932+0000 7f070f181640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0708152690 0x7f0708147520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.932+0000 7f070f181640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0708152a70 con 0x7f07080c6730 2024-03-06T07:21:23.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.932+0000 7f070f181640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07081567b0 con 0x7f07080c5d10 2024-03-06T07:21:23.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.932+0000 7f070f181640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0708152bf0 con 0x7f0708152690 2024-03-06T07:21:23.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.932+0000 7f0707fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07080c6730 0x7f0708146fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.932+0000 7f0707fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07080c6730 0x7f0708146fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37520/0 (socket says 172.21.15.1:37520) 2024-03-06T07:21:23.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.932+0000 7f0707fff640 1 -- 172.21.15.1:0/1421169711 learned_addr learned my addr 172.21.15.1:0/1421169711 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:23.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.932+0000 7f070d71e640 1 --2- 172.21.15.1:0/1421169711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0708152690 0x7f0708147520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.932+0000 7f070cf1d640 1 --2- 172.21.15.1:0/1421169711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07080c5d10 0x7f0708070ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.932+0000 7f0707fff640 1 -- 172.21.15.1:0/1421169711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0708152690 msgr2=0x7f0708147520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.932+0000 7f0707fff640 1 --2- 172.21.15.1:0/1421169711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0708152690 0x7f0708147520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.932+0000 7f0707fff640 1 -- 172.21.15.1:0/1421169711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07080c5d10 msgr2=0x7f0708070ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.932+0000 7f0707fff640 1 --2- 172.21.15.1:0/1421169711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07080c5d10 0x7f0708070ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.932+0000 7f0707fff640 1 -- 172.21.15.1:0/1421169711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07080c8d50 con 0x7f07080c6730 2024-03-06T07:21:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.933+0000 7f0707fff640 1 --2- 172.21.15.1:0/1421169711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07080c6730 0x7f0708146fe0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f06f4000c00 tx=0x7f06f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.933+0000 7f07057fa640 1 -- 172.21.15.1:0/1421169711 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f4040070 con 0x7f07080c6730 2024-03-06T07:21:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.933+0000 7f07057fa640 1 -- 172.21.15.1:0/1421169711 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f40399f0 con 0x7f07080c6730 2024-03-06T07:21:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.933+0000 7f07057fa640 1 -- 172.21.15.1:0/1421169711 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f4035040 con 0x7f07080c6730 2024-03-06T07:21:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.933+0000 7f070f181640 1 -- 172.21.15.1:0/1421169711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0708147d40 con 0x7f07080c6730 2024-03-06T07:21:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.933+0000 7f070f181640 1 -- 172.21.15.1:0/1421169711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f070814d860 con 0x7f07080c6730 2024-03-06T07:21:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.934+0000 7f070f181640 1 -- 172.21.15.1:0/1421169711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0708067fd0 con 0x7f07080c6730 2024-03-06T07:21:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.934+0000 7f07057fa640 1 -- 172.21.15.1:0/1421169711 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f06f404b060 con 0x7f07080c6730 2024-03-06T07:21:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.934+0000 7f07057fa640 1 --2- 172.21.15.1:0/1421169711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f06ec06b4d0 0x7f06ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.935+0000 7f07057fa640 1 -- 172.21.15.1:0/1421169711 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f06f40abb10 con 0x7f07080c6730 2024-03-06T07:21:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.935+0000 7f070cf1d640 1 --2- 172.21.15.1:0/1421169711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f06ec06b4d0 0x7f06ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.935+0000 7f070cf1d640 1 --2- 172.21.15.1:0/1421169711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f06ec06b4d0 0x7f06ec06d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f07081482f0 tx=0x7f06fc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:23.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:23.939+0000 7f07057fa640 1 -- 172.21.15.1:0/1421169711 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f06f407a890 con 0x7f07080c6730 2024-03-06T07:21:24.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.157+0000 7f070f181640 1 -- 172.21.15.1:0/1421169711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0708147a60 con 0x7f07080c6730 2024-03-06T07:21:24.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.165+0000 7f07057fa640 1 -- 172.21.15.1:0/1421169711 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f06f407a190 con 0x7f07080c6730 2024-03-06T07:21:24.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.167+0000 7f070f181640 1 -- 172.21.15.1:0/1421169711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f06ec06b4d0 msgr2=0x7f06ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.167+0000 7f070f181640 1 --2- 172.21.15.1:0/1421169711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f06ec06b4d0 0x7f06ec06d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f07081482f0 tx=0x7f06fc008040 comp rx=0 tx=0).stop 2024-03-06T07:21:24.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.167+0000 7f070f181640 1 -- 172.21.15.1:0/1421169711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07080c6730 msgr2=0x7f0708146fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.167+0000 7f070f181640 1 --2- 172.21.15.1:0/1421169711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07080c6730 0x7f0708146fe0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f06f4000c00 tx=0x7f06f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.167+0000 7f070f181640 1 -- 172.21.15.1:0/1421169711 shutdown_connections 2024-03-06T07:21:24.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.167+0000 7f070f181640 1 --2- 172.21.15.1:0/1421169711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f06ec06b4d0 0x7f06ec06d990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.167+0000 7f070f181640 1 --2- 172.21.15.1:0/1421169711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0708152690 0x7f0708147520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.167+0000 7f070f181640 1 --2- 172.21.15.1:0/1421169711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07080c6730 0x7f0708146fe0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.167+0000 7f070f181640 1 --2- 172.21.15.1:0/1421169711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07080c5d10 0x7f0708070ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.167+0000 7f070f181640 1 -- 172.21.15.1:0/1421169711 >> 172.21.15.1:0/1421169711 conn(0x7f070806b410 msgr2=0x7f070806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:24.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.167+0000 7f070f181640 1 -- 172.21.15.1:0/1421169711 shutdown_connections 2024-03-06T07:21:24.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.167+0000 7f070f181640 1 -- 172.21.15.1:0/1421169711 wait complete. 2024-03-06T07:21:24.183 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:21:24.183 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:21:24.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.285+0000 7f5d1640e640 1 Processor -- start 2024-03-06T07:21:24.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.285+0000 7f5d1640e640 1 -- start start 2024-03-06T07:21:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.286+0000 7f5d1640e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5d10152820 0x7f5d100c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.286+0000 7f5d1640e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d100af6f0 0x7f5d1015fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.286+0000 7f5d1640e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d100c7180 con 0x7f5d100c0670 2024-03-06T07:21:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.286+0000 7f5d1640e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d10155460 con 0x7f5d10152820 2024-03-06T07:21:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.286+0000 7f5d1640e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d100c6bc0 con 0x7f5d100af6f0 2024-03-06T07:21:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.287+0000 7f5d149ab640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d100af6f0 0x7f5d1015fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.287+0000 7f5d0ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5d10152820 0x7f5d100c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.287+0000 7f5d0ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5d10152820 0x7f5d100c0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:46826/0 (socket says 172.21.15.1:46826) 2024-03-06T07:21:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.287+0000 7f5d149ab640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d100af6f0 0x7f5d1015fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:55736/0 (socket says 172.21.15.1:55736) 2024-03-06T07:21:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.287+0000 7f5d0ffff640 1 -- 172.21.15.1:0/2108477839 learned_addr learned my addr 172.21.15.1:0/2108477839 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.287+0000 7f5d0f7fe640 1 --1- 172.21.15.1:0/2108477839 >> v1:172.21.15.98:6789/0 conn(0x7f5d100c0670 0x7f5d100c73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:51510/0 (socket says 172.21.15.1:51510) 2024-03-06T07:21:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.287+0000 7f5d149ab640 1 -- 172.21.15.1:0/2108477839 >> v2:172.21.15.98:3300/0 conn(0x7f5d10152820 msgr2=0x7f5d100c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.287+0000 7f5d149ab640 1 --2- 172.21.15.1:0/2108477839 >> v2:172.21.15.98:3300/0 conn(0x7f5d10152820 0x7f5d100c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.287+0000 7f5d149ab640 1 -- 172.21.15.1:0/2108477839 >> v1:172.21.15.98:6789/0 conn(0x7f5d100c0670 legacy=0x7f5d100c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.287+0000 7f5d149ab640 1 -- 172.21.15.1:0/2108477839 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d100c6ea0 con 0x7f5d100af6f0 2024-03-06T07:21:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.287+0000 7f5d0effd640 1 -- 172.21.15.1:0/2108477839 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710341642 0 0) 0x7f5d100c7180 con 0x7f5d100c0670 2024-03-06T07:21:24.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.287+0000 7f5d0ffff640 1 --2- 172.21.15.1:0/2108477839 >> v2:172.21.15.98:3300/0 conn(0x7f5d10152820 0x7f5d100c0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:24.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.288+0000 7f5d149ab640 1 --2- 172.21.15.1:0/2108477839 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d100af6f0 0x7f5d1015fb60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5d08009d90 tx=0x7f5d0802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e21a9593763d2f8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.288+0000 7f5d0effd640 1 -- 172.21.15.1:0/2108477839 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08039070 con 0x7f5d100af6f0 2024-03-06T07:21:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.288+0000 7f5d0effd640 1 -- 172.21.15.1:0/2108477839 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0802cc50 con 0x7f5d100af6f0 2024-03-06T07:21:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.288+0000 7f5d0effd640 1 -- 172.21.15.1:0/2108477839 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08034440 con 0x7f5d100af6f0 2024-03-06T07:21:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.288+0000 7f5d1640e640 1 -- 172.21.15.1:0/2108477839 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d100af6f0 msgr2=0x7f5d1015fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.288+0000 7f5d1640e640 1 --2- 172.21.15.1:0/2108477839 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d100af6f0 0x7f5d1015fb60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5d08009d90 tx=0x7f5d0802bea0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.288+0000 7f5d1640e640 1 -- 172.21.15.1:0/2108477839 shutdown_connections 2024-03-06T07:21:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.288+0000 7f5d1640e640 1 --2- 172.21.15.1:0/2108477839 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d100af6f0 0x7f5d1015fb60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.288+0000 7f5d1640e640 1 --2- 172.21.15.1:0/2108477839 >> v2:172.21.15.98:3300/0 conn(0x7f5d10152820 0x7f5d100c0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.288+0000 7f5d1640e640 1 -- 172.21.15.1:0/2108477839 >> 172.21.15.1:0/2108477839 conn(0x7f5d10072720 msgr2=0x7f5d10072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.289+0000 7f5d1640e640 1 -- 172.21.15.1:0/2108477839 shutdown_connections 2024-03-06T07:21:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.289+0000 7f5d1640e640 1 -- 172.21.15.1:0/2108477839 wait complete. 2024-03-06T07:21:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.290+0000 7f5d1640e640 1 Processor -- start 2024-03-06T07:21:24.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.290+0000 7f5d1640e640 1 -- start start 2024-03-06T07:21:24.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.290+0000 7f5d1640e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d100af6f0 0x7f5d10154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.290+0000 7f5d1640e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d100c0670 0x7f5d10155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.290+0000 7f5d1640e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d10152820 0x7f5d10156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.290+0000 7f5d1640e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d100c7c80 con 0x7f5d100af6f0 2024-03-06T07:21:24.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.290+0000 7f5d1640e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d100c7b00 con 0x7f5d100c0670 2024-03-06T07:21:24.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.290+0000 7f5d1640e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d10152c00 con 0x7f5d10152820 2024-03-06T07:21:24.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.291+0000 7f5d0ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d100af6f0 0x7f5d10154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.291+0000 7f5d0ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d100af6f0 0x7f5d10154180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37528/0 (socket says 172.21.15.1:37528) 2024-03-06T07:21:24.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.291+0000 7f5d0ffff640 1 -- 172.21.15.1:0/3086582447 learned_addr learned my addr 172.21.15.1:0/3086582447 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:24.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.291+0000 7f5d0f7fe640 1 --2- 172.21.15.1:0/3086582447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d100c0670 0x7f5d10155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.291+0000 7f5d149ab640 1 --2- 172.21.15.1:0/3086582447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d10152820 0x7f5d10156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.291+0000 7f5d0ffff640 1 -- 172.21.15.1:0/3086582447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d10152820 msgr2=0x7f5d10156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.291+0000 7f5d0ffff640 1 --2- 172.21.15.1:0/3086582447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d10152820 0x7f5d10156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.291+0000 7f5d0ffff640 1 -- 172.21.15.1:0/3086582447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d100c0670 msgr2=0x7f5d10155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.291+0000 7f5d0ffff640 1 --2- 172.21.15.1:0/3086582447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d100c0670 0x7f5d10155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.291+0000 7f5d0ffff640 1 -- 172.21.15.1:0/3086582447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d100aa4f0 con 0x7f5d100af6f0 2024-03-06T07:21:24.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.291+0000 7f5d0ffff640 1 --2- 172.21.15.1:0/3086582447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d100af6f0 0x7f5d10154180 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5d0000e9e0 tx=0x7f5d0000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:24.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.291+0000 7f5d0cff9640 1 -- 172.21.15.1:0/3086582447 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00017070 con 0x7f5d100af6f0 2024-03-06T07:21:24.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.291+0000 7f5d0cff9640 1 -- 172.21.15.1:0/3086582447 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0000c9d0 con 0x7f5d100af6f0 2024-03-06T07:21:24.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.291+0000 7f5d1640e640 1 -- 172.21.15.1:0/3086582447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d101ea900 con 0x7f5d100af6f0 2024-03-06T07:21:24.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.291+0000 7f5d0cff9640 1 -- 172.21.15.1:0/3086582447 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0000cd80 con 0x7f5d100af6f0 2024-03-06T07:21:24.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.291+0000 7f5d1640e640 1 -- 172.21.15.1:0/3086582447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d101eaec0 con 0x7f5d100af6f0 2024-03-06T07:21:24.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.293+0000 7f5d1640e640 1 -- 172.21.15.1:0/3086582447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d100686e0 con 0x7f5d100af6f0 2024-03-06T07:21:24.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.293+0000 7f5d0cff9640 1 -- 172.21.15.1:0/3086582447 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d0001a070 con 0x7f5d100af6f0 2024-03-06T07:21:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.293+0000 7f5d0cff9640 1 --2- 172.21.15.1:0/3086582447 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5cfc06b500 0x7f5cfc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.293+0000 7f5d0cff9640 1 -- 172.21.15.1:0/3086582447 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5d0008a580 con 0x7f5d100af6f0 2024-03-06T07:21:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.293+0000 7f5d0f7fe640 1 --2- 172.21.15.1:0/3086582447 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5cfc06b500 0x7f5cfc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.294+0000 7f5d0f7fe640 1 --2- 172.21.15.1:0/3086582447 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5cfc06b500 0x7f5cfc06d9c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5d10151520 tx=0x7f5d04007b50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:24.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.297+0000 7f5d0cff9640 1 -- 172.21.15.1:0/3086582447 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d00059280 con 0x7f5d100af6f0 2024-03-06T07:21:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:24.516+0000 7f5d1640e640 1 -- 172.21.15.1:0/3086582447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5d100c5b40 con 0x7f5d100af6f0 2024-03-06T07:21:25.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.170+0000 7f5d0cff9640 1 -- 172.21.15.1:0/3086582447 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5d00058b80 con 0x7f5d100af6f0 2024-03-06T07:21:25.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.172+0000 7f5d1640e640 1 -- 172.21.15.1:0/3086582447 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5cfc06b500 msgr2=0x7f5cfc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.172+0000 7f5d1640e640 1 --2- 172.21.15.1:0/3086582447 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5cfc06b500 0x7f5cfc06d9c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5d10151520 tx=0x7f5d04007b50 comp rx=0 tx=0).stop 2024-03-06T07:21:25.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.172+0000 7f5d1640e640 1 -- 172.21.15.1:0/3086582447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d100af6f0 msgr2=0x7f5d10154180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.172+0000 7f5d1640e640 1 --2- 172.21.15.1:0/3086582447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d100af6f0 0x7f5d10154180 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5d0000e9e0 tx=0x7f5d0000eea0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.172+0000 7f5d1640e640 1 -- 172.21.15.1:0/3086582447 shutdown_connections 2024-03-06T07:21:25.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.172+0000 7f5d1640e640 1 --2- 172.21.15.1:0/3086582447 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5cfc06b500 0x7f5cfc06d9c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.172+0000 7f5d1640e640 1 --2- 172.21.15.1:0/3086582447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d10152820 0x7f5d10156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.172+0000 7f5d1640e640 1 --2- 172.21.15.1:0/3086582447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d100c0670 0x7f5d10155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.172+0000 7f5d1640e640 1 --2- 172.21.15.1:0/3086582447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d100af6f0 0x7f5d10154180 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.172+0000 7f5d1640e640 1 -- 172.21.15.1:0/3086582447 >> 172.21.15.1:0/3086582447 conn(0x7f5d10072720 msgr2=0x7f5d1006cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:25.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.172+0000 7f5d1640e640 1 -- 172.21.15.1:0/3086582447 shutdown_connections 2024-03-06T07:21:25.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.172+0000 7f5d1640e640 1 -- 172.21.15.1:0/3086582447 wait complete. 2024-03-06T07:21:25.185 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:21:25.185 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.284+0000 7fb0dd37a640 1 Processor -- start 2024-03-06T07:21:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.284+0000 7fb0dd37a640 1 -- start start 2024-03-06T07:21:25.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.285+0000 7fb0dd37a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb0d80a7050 0x7fb0d80c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.285+0000 7fb0dd37a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d8153a00 con 0x7fb0d80c4a20 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.285+0000 7fb0dd37a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d8153ce0 con 0x7fb0d8156b70 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.285+0000 7fb0dd37a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d80c3cb0 con 0x7fb0d80a7050 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.285+0000 7fb0ce7fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb0d80c4a20 0x7fb0d8156460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60756/0 (socket says 172.21.15.1:60756) 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.285+0000 7fb0ce7fc640 1 -- 172.21.15.1:0/792285193 learned_addr learned my addr 172.21.15.1:0/792285193 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.285+0000 7fb0d6ffd640 1 --2- 172.21.15.1:0/792285193 >> v2:172.21.15.98:3300/0 conn(0x7fb0d80a7050 0x7fb0d80c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.285+0000 7fb0d67fc640 1 -- 172.21.15.1:0/792285193 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882085607 0 0) 0x7fb0d8153a00 con 0x7fb0d80c4a20 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.286+0000 7fb0d67fc640 1 -- 172.21.15.1:0/792285193 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0c4003520 con 0x7fb0d80c4a20 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.286+0000 7fb0d67fc640 1 -- 172.21.15.1:0/792285193 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080928641 0 0) 0x7fb0d8153ce0 con 0x7fb0d8156b70 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.286+0000 7fb0d67fc640 1 -- 172.21.15.1:0/792285193 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d8153a00 con 0x7fb0d8156b70 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.286+0000 7fb0d67fc640 1 -- 172.21.15.1:0/792285193 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2099386739 0 0) 0x7fb0c4003520 con 0x7fb0d80c4a20 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.286+0000 7fb0d6ffd640 1 -- 172.21.15.1:0/792285193 >> v1:172.21.15.98:6789/0 conn(0x7fb0d8156b70 legacy=0x7fb0d815d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.286+0000 7fb0d6ffd640 1 -- 172.21.15.1:0/792285193 >> v1:172.21.15.1:6789/0 conn(0x7fb0d80c4a20 legacy=0x7fb0d8156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.286+0000 7fb0d6ffd640 1 -- 172.21.15.1:0/792285193 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d80c3f90 con 0x7fb0d80a7050 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.287+0000 7fb0d6ffd640 1 --2- 172.21.15.1:0/792285193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d80a7050 0x7fb0d80c44e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb0c80097a0 tx=0x7fb0c802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6901d1c0490868c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.287+0000 7fb0d67fc640 1 -- 172.21.15.1:0/792285193 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8037070 con 0x7fb0d80a7050 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.287+0000 7fb0d67fc640 1 -- 172.21.15.1:0/792285193 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c8030510 con 0x7fb0d80a7050 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.287+0000 7fb0d67fc640 1 -- 172.21.15.1:0/792285193 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0001670 con 0x7fb0d80a7050 2024-03-06T07:21:25.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.287+0000 7fb0dd37a640 1 -- 172.21.15.1:0/792285193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d80a7050 msgr2=0x7fb0d80c44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.287+0000 7fb0dd37a640 1 --2- 172.21.15.1:0/792285193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d80a7050 0x7fb0d80c44e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb0c80097a0 tx=0x7fb0c802b710 comp rx=0 tx=0).stop 2024-03-06T07:21:25.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.287+0000 7fb0dd37a640 1 -- 172.21.15.1:0/792285193 shutdown_connections 2024-03-06T07:21:25.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.287+0000 7fb0dd37a640 1 --2- 172.21.15.1:0/792285193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d80a7050 0x7fb0d80c44e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.287+0000 7fb0dd37a640 1 -- 172.21.15.1:0/792285193 >> 172.21.15.1:0/792285193 conn(0x7fb0d80c7a50 msgr2=0x7fb0d80c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:25.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.287+0000 7fb0dd37a640 1 -- 172.21.15.1:0/792285193 shutdown_connections 2024-03-06T07:21:25.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.287+0000 7fb0dd37a640 1 -- 172.21.15.1:0/792285193 wait complete. 2024-03-06T07:21:25.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.288+0000 7fb0dd37a640 1 Processor -- start 2024-03-06T07:21:25.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.289+0000 7fb0dd37a640 1 -- start start 2024-03-06T07:21:25.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.289+0000 7fb0dd37a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d80a7050 0x7fb0d80a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.289+0000 7fb0dd37a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d80c4a20 0x7fb0d81df490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.289+0000 7fb0dd37a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d8156b70 0x7fb0d80a8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.289+0000 7fb0dd37a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d8072f60 con 0x7fb0d8156b70 2024-03-06T07:21:25.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.289+0000 7fb0dd37a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d80c4ea0 con 0x7fb0d80c4a20 2024-03-06T07:21:25.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.289+0000 7fb0dd37a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d8156fc0 con 0x7fb0d80a7050 2024-03-06T07:21:25.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.289+0000 7fb0d77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d8156b70 0x7fb0d80a8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.289+0000 7fb0d77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d8156b70 0x7fb0d80a8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36982/0 (socket says 172.21.15.1:36982) 2024-03-06T07:21:25.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.289+0000 7fb0d77fe640 1 -- 172.21.15.1:0/4280981065 learned_addr learned my addr 172.21.15.1:0/4280981065 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:25.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.290+0000 7fb0ce7fc640 1 --2- 172.21.15.1:0/4280981065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d80c4a20 0x7fb0d81df490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.290+0000 7fb0d6ffd640 1 --2- 172.21.15.1:0/4280981065 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d80a7050 0x7fb0d80a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.290+0000 7fb0d77fe640 1 -- 172.21.15.1:0/4280981065 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d80a7050 msgr2=0x7fb0d80a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.290+0000 7fb0d77fe640 1 --2- 172.21.15.1:0/4280981065 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d80a7050 0x7fb0d80a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.290+0000 7fb0d77fe640 1 -- 172.21.15.1:0/4280981065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d80c4a20 msgr2=0x7fb0d81df490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.290+0000 7fb0d77fe640 1 --2- 172.21.15.1:0/4280981065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d80c4a20 0x7fb0d81df490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.290+0000 7fb0d77fe640 1 -- 172.21.15.1:0/4280981065 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d81df9d0 con 0x7fb0d8156b70 2024-03-06T07:21:25.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.290+0000 7fb0d77fe640 1 --2- 172.21.15.1:0/4280981065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d8156b70 0x7fb0d80a8fb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb0d000c6e0 tx=0x7fb0d000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.290+0000 7fb0cffff640 1 -- 172.21.15.1:0/4280981065 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0003b70 con 0x7fb0d8156b70 2024-03-06T07:21:25.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.290+0000 7fb0cffff640 1 -- 172.21.15.1:0/4280981065 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d0003d10 con 0x7fb0d8156b70 2024-03-06T07:21:25.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.290+0000 7fb0dd37a640 1 -- 172.21.15.1:0/4280981065 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d81ee6d0 con 0x7fb0d8156b70 2024-03-06T07:21:25.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.290+0000 7fb0cffff640 1 -- 172.21.15.1:0/4280981065 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d00052d0 con 0x7fb0d8156b70 2024-03-06T07:21:25.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.290+0000 7fb0dd37a640 1 -- 172.21.15.1:0/4280981065 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d81eec90 con 0x7fb0d8156b70 2024-03-06T07:21:25.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.291+0000 7fb0dd37a640 1 -- 172.21.15.1:0/4280981065 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d80686e0 con 0x7fb0d8156b70 2024-03-06T07:21:25.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.292+0000 7fb0cffff640 1 -- 172.21.15.1:0/4280981065 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0d0005470 con 0x7fb0d8156b70 2024-03-06T07:21:25.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.292+0000 7fb0cffff640 1 --2- 172.21.15.1:0/4280981065 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0b806b4d0 0x7fb0b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.292+0000 7fb0cffff640 1 -- 172.21.15.1:0/4280981065 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb0d008a1d0 con 0x7fb0d8156b70 2024-03-06T07:21:25.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.292+0000 7fb0d6ffd640 1 --2- 172.21.15.1:0/4280981065 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0b806b4d0 0x7fb0b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.293+0000 7fb0d6ffd640 1 --2- 172.21.15.1:0/4280981065 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0b806b4d0 0x7fb0b806d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8034000 tx=0x7fb0c8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.296+0000 7fb0cffff640 1 -- 172.21.15.1:0/4280981065 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0d0058f50 con 0x7fb0d8156b70 2024-03-06T07:21:25.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.516+0000 7fb0dd37a640 1 -- 172.21.15.1:0/4280981065 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0d81eaf10 con 0x7fb0d8156b70 2024-03-06T07:21:25.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.516+0000 7fb0cffff640 1 -- 172.21.15.1:0/4280981065 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4971 (secure 0 0 0) 0x7fb0d81eaf10 con 0x7fb0d8156b70 2024-03-06T07:21:25.518 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:25.518 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4781,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2438105345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2438105345},{"type":"v1","addr":"172.21.15.1:6835","nonce":2438105345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3354292028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3354292028},{"type":"v1","addr":"172.21.15.98:6836","nonce":3354292028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4975,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3948803645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3948803645},{"type":"v1","addr":"172.21.15.98:6837","nonce":3948803645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:22.528539+0000","modified":"2024-03-06T07:21:24.518239+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1818141892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1818141892},{"type":"v1","addr":"172.21.15.1:6834","nonce":1818141892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":2}]} 2024-03-06T07:21:25.518 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 16 2024-03-06T07:21:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.518+0000 7fb0dd37a640 1 -- 172.21.15.1:0/4280981065 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0b806b4d0 msgr2=0x7fb0b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.518+0000 7fb0dd37a640 1 --2- 172.21.15.1:0/4280981065 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0b806b4d0 0x7fb0b806d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8034000 tx=0x7fb0c8008030 comp rx=0 tx=0).stop 2024-03-06T07:21:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.518+0000 7fb0dd37a640 1 -- 172.21.15.1:0/4280981065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d8156b70 msgr2=0x7fb0d80a8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.518+0000 7fb0dd37a640 1 --2- 172.21.15.1:0/4280981065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d8156b70 0x7fb0d80a8fb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb0d000c6e0 tx=0x7fb0d000cba0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.519+0000 7fb0dd37a640 1 -- 172.21.15.1:0/4280981065 shutdown_connections 2024-03-06T07:21:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.519+0000 7fb0dd37a640 1 --2- 172.21.15.1:0/4280981065 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0b806b4d0 0x7fb0b806d990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.519+0000 7fb0dd37a640 1 --2- 172.21.15.1:0/4280981065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d8156b70 0x7fb0d80a8fb0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.519+0000 7fb0dd37a640 1 --2- 172.21.15.1:0/4280981065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d80c4a20 0x7fb0d81df490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.519+0000 7fb0dd37a640 1 --2- 172.21.15.1:0/4280981065 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d80a7050 0x7fb0d80a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.519+0000 7fb0dd37a640 1 -- 172.21.15.1:0/4280981065 >> 172.21.15.1:0/4280981065 conn(0x7fb0d80c7a50 msgr2=0x7fb0d80c7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.519+0000 7fb0dd37a640 1 -- 172.21.15.1:0/4280981065 shutdown_connections 2024-03-06T07:21:25.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.519+0000 7fb0dd37a640 1 -- 172.21.15.1:0/4280981065 wait complete. 2024-03-06T07:21:25.533 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:25.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.632+0000 7f5105623640 1 Processor -- start 2024-03-06T07:21:25.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.632+0000 7f5105623640 1 -- start start 2024-03-06T07:21:25.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.633+0000 7f5105623640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51000c3eb0 0x7f51000c7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.633+0000 7f5105623640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f510006d060 0x7f51001558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.633+0000 7f5105623640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f51000c7940 0x7f5100155df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.633+0000 7f5105623640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51000c59b0 con 0x7f510006d060 2024-03-06T07:21:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.633+0000 7f5105623640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51000c4c00 con 0x7f51000c7940 2024-03-06T07:21:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.633+0000 7f5105623640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5100155420 con 0x7f51000c3eb0 2024-03-06T07:21:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.633+0000 7f50f7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f510006d060 0x7f51001558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.633+0000 7f50f7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f510006d060 0x7f51001558b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36998/0 (socket says 172.21.15.1:36998) 2024-03-06T07:21:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.633+0000 7f50f7fff640 1 -- 172.21.15.1:0/1610320940 learned_addr learned my addr 172.21.15.1:0/1610320940 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.633+0000 7f50feffd640 1 --2- 172.21.15.1:0/1610320940 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51000c3eb0 0x7f51000c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.633+0000 7f50ff7fe640 1 --2- 172.21.15.1:0/1610320940 >> v2:172.21.15.98:3300/0 conn(0x7f51000c7940 0x7f5100155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.633+0000 7f50f7fff640 1 -- 172.21.15.1:0/1610320940 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51000c3eb0 msgr2=0x7f51000c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.633+0000 7f50f7fff640 1 --2- 172.21.15.1:0/1610320940 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51000c3eb0 0x7f51000c7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.633+0000 7f50f7fff640 1 -- 172.21.15.1:0/1610320940 >> v2:172.21.15.98:3300/0 conn(0x7f51000c7940 msgr2=0x7f5100155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.633+0000 7f50f7fff640 1 --2- 172.21.15.1:0/1610320940 >> v2:172.21.15.98:3300/0 conn(0x7f51000c7940 0x7f5100155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.633+0000 7f50f7fff640 1 -- 172.21.15.1:0/1610320940 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51000c71c0 con 0x7f510006d060 2024-03-06T07:21:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.633+0000 7f50feffd640 1 --2- 172.21.15.1:0/1610320940 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51000c3eb0 0x7f51000c7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:25.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.634+0000 7f50f7fff640 1 --2- 172.21.15.1:0/1610320940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f510006d060 0x7f51001558b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f51000c5820 tx=0x7f50e802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4674b35a95847687 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.634+0000 7f50fe7fc640 1 -- 172.21.15.1:0/1610320940 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e8038070 con 0x7f510006d060 2024-03-06T07:21:25.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.634+0000 7f50fe7fc640 1 -- 172.21.15.1:0/1610320940 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e802f050 con 0x7f510006d060 2024-03-06T07:21:25.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.634+0000 7f50fe7fc640 1 -- 172.21.15.1:0/1610320940 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e8031970 con 0x7f510006d060 2024-03-06T07:21:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.635+0000 7f5105623640 1 -- 172.21.15.1:0/1610320940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f510006d060 msgr2=0x7f51001558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.635+0000 7f5105623640 1 --2- 172.21.15.1:0/1610320940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f510006d060 0x7f51001558b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f51000c5820 tx=0x7f50e802b7f0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.635+0000 7f5105623640 1 -- 172.21.15.1:0/1610320940 shutdown_connections 2024-03-06T07:21:25.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.635+0000 7f5105623640 1 --2- 172.21.15.1:0/1610320940 >> v2:172.21.15.98:3300/0 conn(0x7f51000c7940 0x7f5100155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.635+0000 7f5105623640 1 --2- 172.21.15.1:0/1610320940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f510006d060 0x7f51001558b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.635+0000 7f5105623640 1 --2- 172.21.15.1:0/1610320940 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51000c3eb0 0x7f51000c7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.635+0000 7f5105623640 1 -- 172.21.15.1:0/1610320940 >> 172.21.15.1:0/1610320940 conn(0x7f5100066d00 msgr2=0x7f5100067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:25.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.635+0000 7f5105623640 1 -- 172.21.15.1:0/1610320940 shutdown_connections 2024-03-06T07:21:25.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.635+0000 7f5105623640 1 -- 172.21.15.1:0/1610320940 wait complete. 2024-03-06T07:21:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.636+0000 7f5105623640 1 Processor -- start 2024-03-06T07:21:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.636+0000 7f5105623640 1 -- start start 2024-03-06T07:21:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.637+0000 7f5105623640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f510006d060 0x7f5100146900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.637+0000 7f5105623640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51000c3eb0 0x7f510014a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.637+0000 7f5105623640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51000c7940 0x7f51000c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.637+0000 7f5105623640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51000c4290 con 0x7f51000c7940 2024-03-06T07:21:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.637+0000 7f5105623640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51000c6ee0 con 0x7f510006d060 2024-03-06T07:21:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.637+0000 7f5105623640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51001ea910 con 0x7f51000c3eb0 2024-03-06T07:21:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.637+0000 7f50ff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51000c7940 0x7f51000c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.637+0000 7f50ff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51000c7940 0x7f51000c87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37000/0 (socket says 172.21.15.1:37000) 2024-03-06T07:21:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.637+0000 7f50ff7fe640 1 -- 172.21.15.1:0/370391196 learned_addr learned my addr 172.21.15.1:0/370391196 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.637+0000 7f50f7fff640 1 --2- 172.21.15.1:0/370391196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51000c3eb0 0x7f510014a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.637+0000 7f50feffd640 1 --2- 172.21.15.1:0/370391196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f510006d060 0x7f5100146900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.637+0000 7f50ff7fe640 1 -- 172.21.15.1:0/370391196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51000c3eb0 msgr2=0x7f510014a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.637+0000 7f50ff7fe640 1 --2- 172.21.15.1:0/370391196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51000c3eb0 0x7f510014a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.637+0000 7f50ff7fe640 1 -- 172.21.15.1:0/370391196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f510006d060 msgr2=0x7f5100146900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.637+0000 7f50ff7fe640 1 --2- 172.21.15.1:0/370391196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f510006d060 0x7f5100146900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.637+0000 7f50ff7fe640 1 -- 172.21.15.1:0/370391196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51000ca450 con 0x7f51000c7940 2024-03-06T07:21:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.638+0000 7f50ff7fe640 1 --2- 172.21.15.1:0/370391196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51000c7940 0x7f51000c87f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f50f8007e40 tx=0x7f50f800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.638+0000 7f50f77fe640 1 -- 172.21.15.1:0/370391196 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f8014620 con 0x7f51000c7940 2024-03-06T07:21:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.638+0000 7f50f77fe640 1 -- 172.21.15.1:0/370391196 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f8019070 con 0x7f51000c7940 2024-03-06T07:21:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.638+0000 7f5105623640 1 -- 172.21.15.1:0/370391196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51001eab90 con 0x7f51000c7940 2024-03-06T07:21:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.638+0000 7f50f77fe640 1 -- 172.21.15.1:0/370391196 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f8014a20 con 0x7f51000c7940 2024-03-06T07:21:25.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.638+0000 7f5105623640 1 -- 172.21.15.1:0/370391196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51001eb150 con 0x7f51000c7940 2024-03-06T07:21:25.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.639+0000 7f5105623640 1 -- 172.21.15.1:0/370391196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51000c21c0 con 0x7f51000c7940 2024-03-06T07:21:25.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.639+0000 7f50f77fe640 1 -- 172.21.15.1:0/370391196 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50f8014bc0 con 0x7f51000c7940 2024-03-06T07:21:25.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.640+0000 7f50f77fe640 1 --2- 172.21.15.1:0/370391196 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50e006b4d0 0x7f50e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.640+0000 7f50f77fe640 1 -- 172.21.15.1:0/370391196 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f50f808ab10 con 0x7f51000c7940 2024-03-06T07:21:25.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.640+0000 7f50feffd640 1 --2- 172.21.15.1:0/370391196 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50e006b4d0 0x7f50e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.641+0000 7f50feffd640 1 --2- 172.21.15.1:0/370391196 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50e006b4d0 0x7f50e006d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f50f000a990 tx=0x7f50f0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.644+0000 7f50f77fe640 1 -- 172.21.15.1:0/370391196 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51000c21c0 con 0x7f51000c7940 2024-03-06T07:21:25.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.825+0000 7f5105623640 1 -- 172.21.15.1:0/370391196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5100067fd0 con 0x7f51000c7940 2024-03-06T07:21:25.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.826+0000 7f50f77fe640 1 -- 172.21.15.1:0/370391196 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16822 (secure 0 0 0) 0x7f50f8059890 con 0x7f51000c7940 2024-03-06T07:21:25.829 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:25.829 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":39,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:21:24.161267+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:21.073384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:21.484387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:25.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.828+0000 7f5105623640 1 -- 172.21.15.1:0/370391196 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50e006b4d0 msgr2=0x7f50e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.828+0000 7f5105623640 1 --2- 172.21.15.1:0/370391196 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50e006b4d0 0x7f50e006d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f50f000a990 tx=0x7f50f0008040 comp rx=0 tx=0).stop 2024-03-06T07:21:25.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.828+0000 7f5105623640 1 -- 172.21.15.1:0/370391196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51000c7940 msgr2=0x7f51000c87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.828+0000 7f5105623640 1 --2- 172.21.15.1:0/370391196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51000c7940 0x7f51000c87f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f50f8007e40 tx=0x7f50f800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.828+0000 7f5105623640 1 -- 172.21.15.1:0/370391196 shutdown_connections 2024-03-06T07:21:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.828+0000 7f5105623640 1 --2- 172.21.15.1:0/370391196 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50e006b4d0 0x7f50e006d990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.829+0000 7f5105623640 1 --2- 172.21.15.1:0/370391196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51000c7940 0x7f51000c87f0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.829+0000 7f5105623640 1 --2- 172.21.15.1:0/370391196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51000c3eb0 0x7f510014a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.829+0000 7f5105623640 1 --2- 172.21.15.1:0/370391196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f510006d060 0x7f5100146900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.829+0000 7f5105623640 1 -- 172.21.15.1:0/370391196 >> 172.21.15.1:0/370391196 conn(0x7f5100066d00 msgr2=0x7f5100152dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.829+0000 7f5105623640 1 -- 172.21.15.1:0/370391196 shutdown_connections 2024-03-06T07:21:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.829+0000 7f5105623640 1 -- 172.21.15.1:0/370391196 wait complete. 2024-03-06T07:21:25.842 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:21:25.842 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:25.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.941+0000 7fc84c4e1640 1 Processor -- start 2024-03-06T07:21:25.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.941+0000 7fc84c4e1640 1 -- start start 2024-03-06T07:21:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.942+0000 7fc84c4e1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc8440c3970 0x7fc8440c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.942+0000 7fc84c4e1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc844152840 0x7fc8441553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.942+0000 7fc84c4e1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8440c5e40 con 0x7fc8440c60b0 2024-03-06T07:21:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.942+0000 7fc84c4e1640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8440c67f0 con 0x7fc844152840 2024-03-06T07:21:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.942+0000 7fc84c4e1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8440c6a40 con 0x7fc8440c3970 2024-03-06T07:21:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.942+0000 7fc84aa7e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc844152840 0x7fc8441553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.943+0000 7fc84aa7e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc844152840 0x7fc8441553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37002/0 (socket says 172.21.15.1:37002) 2024-03-06T07:21:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.943+0000 7fc84aa7e640 1 -- 172.21.15.1:0/2018224767 learned_addr learned my addr 172.21.15.1:0/2018224767 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.943+0000 7fc84a27d640 1 --2- 172.21.15.1:0/2018224767 >> v2:172.21.15.98:3300/0 conn(0x7fc8440c3970 0x7fc8440c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.943+0000 7fc84aa7e640 1 -- 172.21.15.1:0/2018224767 >> v2:172.21.15.98:3300/0 conn(0x7fc8440c3970 msgr2=0x7fc8440c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.943+0000 7fc84aa7e640 1 --2- 172.21.15.1:0/2018224767 >> v2:172.21.15.98:3300/0 conn(0x7fc8440c3970 0x7fc8440c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.943+0000 7fc84aa7e640 1 -- 172.21.15.1:0/2018224767 >> v1:172.21.15.98:6789/0 conn(0x7fc8440c60b0 legacy=0x7fc844152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.943+0000 7fc84aa7e640 1 -- 172.21.15.1:0/2018224767 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8440c6d20 con 0x7fc844152840 2024-03-06T07:21:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.944+0000 7fc84aa7e640 1 --2- 172.21.15.1:0/2018224767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc844152840 0x7fc8441553d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc8440c34c0 tx=0x7fc83c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4893970100f07a55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.944+0000 7fc84927b640 1 -- 172.21.15.1:0/2018224767 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c02ca80 con 0x7fc844152840 2024-03-06T07:21:25.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.944+0000 7fc84927b640 1 -- 172.21.15.1:0/2018224767 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83c02cc20 con 0x7fc844152840 2024-03-06T07:21:25.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.944+0000 7fc84927b640 1 -- 172.21.15.1:0/2018224767 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c0343f0 con 0x7fc844152840 2024-03-06T07:21:25.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.944+0000 7fc84c4e1640 1 -- 172.21.15.1:0/2018224767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc844152840 msgr2=0x7fc8441553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.944+0000 7fc84c4e1640 1 --2- 172.21.15.1:0/2018224767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc844152840 0x7fc8441553d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc8440c34c0 tx=0x7fc83c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.944+0000 7fc84c4e1640 1 -- 172.21.15.1:0/2018224767 shutdown_connections 2024-03-06T07:21:25.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.944+0000 7fc84c4e1640 1 --2- 172.21.15.1:0/2018224767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc844152840 0x7fc8441553d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.944+0000 7fc84c4e1640 1 --2- 172.21.15.1:0/2018224767 >> v2:172.21.15.98:3300/0 conn(0x7fc8440c3970 0x7fc8440c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.944+0000 7fc84c4e1640 1 -- 172.21.15.1:0/2018224767 >> 172.21.15.1:0/2018224767 conn(0x7fc8440c88f0 msgr2=0x7fc8440c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:25.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.944+0000 7fc84c4e1640 1 -- 172.21.15.1:0/2018224767 shutdown_connections 2024-03-06T07:21:25.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.944+0000 7fc84c4e1640 1 -- 172.21.15.1:0/2018224767 wait complete. 2024-03-06T07:21:25.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.946+0000 7fc84c4e1640 1 Processor -- start 2024-03-06T07:21:25.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.946+0000 7fc84c4e1640 1 -- start start 2024-03-06T07:21:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.946+0000 7fc84c4e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8440c3970 0x7fc844157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.946+0000 7fc84c4e1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8440c60b0 0x7fc84414c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.946+0000 7fc84c4e1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc844152840 0x7fc84414cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.946+0000 7fc84c4e1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8440c75c0 con 0x7fc8440c60b0 2024-03-06T07:21:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.946+0000 7fc84c4e1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8440c72e0 con 0x7fc8440c3970 2024-03-06T07:21:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.946+0000 7fc84c4e1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8440c3d50 con 0x7fc844152840 2024-03-06T07:21:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.946+0000 7fc849a7c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8440c60b0 0x7fc84414c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.946+0000 7fc849a7c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8440c60b0 0x7fc84414c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37014/0 (socket says 172.21.15.1:37014) 2024-03-06T07:21:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.946+0000 7fc849a7c640 1 -- 172.21.15.1:0/566369908 learned_addr learned my addr 172.21.15.1:0/566369908 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.947+0000 7fc849a7c640 1 -- 172.21.15.1:0/566369908 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc844152840 msgr2=0x7fc84414cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.947+0000 7fc849a7c640 1 --2- 172.21.15.1:0/566369908 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc844152840 0x7fc84414cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.947+0000 7fc849a7c640 1 -- 172.21.15.1:0/566369908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8440c3970 msgr2=0x7fc844157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.947+0000 7fc849a7c640 1 --2- 172.21.15.1:0/566369908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8440c3970 0x7fc844157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.947+0000 7fc849a7c640 1 -- 172.21.15.1:0/566369908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84414d440 con 0x7fc8440c60b0 2024-03-06T07:21:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.947+0000 7fc849a7c640 1 --2- 172.21.15.1:0/566369908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8440c60b0 0x7fc84414c9c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc83400c3f0 tx=0x7fc83400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.947+0000 7fc83affd640 1 -- 172.21.15.1:0/566369908 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc834007020 con 0x7fc8440c60b0 2024-03-06T07:21:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.947+0000 7fc83affd640 1 -- 172.21.15.1:0/566369908 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8340071c0 con 0x7fc8440c60b0 2024-03-06T07:21:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.947+0000 7fc84c4e1640 1 -- 172.21.15.1:0/566369908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc84414e270 con 0x7fc8440c60b0 2024-03-06T07:21:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.947+0000 7fc83affd640 1 -- 172.21.15.1:0/566369908 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc834007590 con 0x7fc8440c60b0 2024-03-06T07:21:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.947+0000 7fc84c4e1640 1 -- 172.21.15.1:0/566369908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8441ee970 con 0x7fc8440c60b0 2024-03-06T07:21:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.948+0000 7fc84c4e1640 1 -- 172.21.15.1:0/566369908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc84414e440 con 0x7fc8440c60b0 2024-03-06T07:21:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.948+0000 7fc83affd640 1 -- 172.21.15.1:0/566369908 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc834019020 con 0x7fc8440c60b0 2024-03-06T07:21:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.949+0000 7fc83affd640 1 --2- 172.21.15.1:0/566369908 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc82806b500 0x7fc82806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.949+0000 7fc83affd640 1 -- 172.21.15.1:0/566369908 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc834089a90 con 0x7fc8440c60b0 2024-03-06T07:21:25.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.949+0000 7fc84a27d640 1 --2- 172.21.15.1:0/566369908 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc82806b500 0x7fc82806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.950+0000 7fc84a27d640 1 --2- 172.21.15.1:0/566369908 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc82806b500 0x7fc82806d9c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc8400098f0 tx=0x7fc840008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:25.953+0000 7fc83affd640 1 -- 172.21.15.1:0/566369908 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc834058810 con 0x7fc8440c60b0 2024-03-06T07:21:26.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.131+0000 7fc84c4e1640 1 -- 172.21.15.1:0/566369908 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc8440686e0 con 0x7fc82806b500 2024-03-06T07:21:26.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.137+0000 7fc83affd640 1 -- 172.21.15.1:0/566369908 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237274 (secure 0 0 0) 0x7fc8440686e0 con 0x7fc82806b500 2024-03-06T07:21:26.142 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:26.143 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:21:26.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.140+0000 7fc84c4e1640 1 -- 172.21.15.1:0/566369908 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc82806b500 msgr2=0x7fc82806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.140+0000 7fc84c4e1640 1 --2- 172.21.15.1:0/566369908 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc82806b500 0x7fc82806d9c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc8400098f0 tx=0x7fc840008040 comp rx=0 tx=0).stop 2024-03-06T07:21:26.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.140+0000 7fc84c4e1640 1 -- 172.21.15.1:0/566369908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8440c60b0 msgr2=0x7fc84414c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.140+0000 7fc84c4e1640 1 --2- 172.21.15.1:0/566369908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8440c60b0 0x7fc84414c9c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc83400c3f0 tx=0x7fc83400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.140+0000 7fc84c4e1640 1 -- 172.21.15.1:0/566369908 shutdown_connections 2024-03-06T07:21:26.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.140+0000 7fc84c4e1640 1 --2- 172.21.15.1:0/566369908 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc82806b500 0x7fc82806d9c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.140+0000 7fc84c4e1640 1 --2- 172.21.15.1:0/566369908 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc844152840 0x7fc84414cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.140+0000 7fc84c4e1640 1 --2- 172.21.15.1:0/566369908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8440c60b0 0x7fc84414c9c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.140+0000 7fc84c4e1640 1 --2- 172.21.15.1:0/566369908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8440c3970 0x7fc844157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.140+0000 7fc84c4e1640 1 -- 172.21.15.1:0/566369908 >> 172.21.15.1:0/566369908 conn(0x7fc8440c88f0 msgr2=0x7fc8440aa680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:26.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.140+0000 7fc84c4e1640 1 -- 172.21.15.1:0/566369908 shutdown_connections 2024-03-06T07:21:26.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.140+0000 7fc84c4e1640 1 -- 172.21.15.1:0/566369908 wait complete. 2024-03-06T07:21:26.153 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-03-06T07:21:24.703091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":236132,"kb_used_data":3264,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738588,"statfs":{"total":773094113280,"available":772852314112,"internally_reserved":0,"allocated":3342336,"data_stored":2544176,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000864"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":197,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:21:17.146365+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:21:17.146365+0000","last_peered":"2024-03-06T07:21:17.146365+0000","last_clean":"2024-03-06T07:21:17.146365+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:21:17.146365+0000","last_undegraded":"2024-03-06T07:21:17.146365+0000","last_fullsized":"2024-03-06T07:21:17.146365+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":466944,"data_stored":760798,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":170430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":170430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":170430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31584,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":401408,"data_stored":170430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31580,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":401408,"data_stored":170430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31580,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":401408,"data_stored":170430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31644,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340196,"statfs":{"total":96636764160,"available":96604360704,"internally_reserved":0,"allocated":466944,"data_stored":760798,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:26.157 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.293+0000 7fc009f6c640 1 Processor -- start 2024-03-06T07:21:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.293+0000 7fc009f6c640 1 -- start start 2024-03-06T07:21:26.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.294+0000 7fc009f6c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc004152650 0x7fc004155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.294+0000 7fc009f6c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc0040bf9d0 0x7fc0040c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.294+0000 7fc009f6c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0040c2fc0 con 0x7fc0040c6b10 2024-03-06T07:21:26.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.294+0000 7fc009f6c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc004155800 con 0x7fc004152650 2024-03-06T07:21:26.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.294+0000 7fc009f6c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc004158da0 con 0x7fc0040bf9d0 2024-03-06T07:21:26.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.294+0000 7fc0037fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc004152650 0x7fc004155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.294+0000 7fc0037fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc004152650 0x7fc004155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37018/0 (socket says 172.21.15.1:37018) 2024-03-06T07:21:26.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.294+0000 7fc0037fe640 1 -- 172.21.15.1:0/965291044 learned_addr learned my addr 172.21.15.1:0/965291044 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:26.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.294+0000 7fc003fff640 1 --2- 172.21.15.1:0/965291044 >> v2:172.21.15.98:3300/0 conn(0x7fc0040bf9d0 0x7fc0040c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.294+0000 7fc0037fe640 1 -- 172.21.15.1:0/965291044 >> v2:172.21.15.98:3300/0 conn(0x7fc0040bf9d0 msgr2=0x7fc0040c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:26.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.294+0000 7fc0037fe640 1 --2- 172.21.15.1:0/965291044 >> v2:172.21.15.98:3300/0 conn(0x7fc0040bf9d0 0x7fc0040c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.294+0000 7fc0037fe640 1 -- 172.21.15.1:0/965291044 >> v1:172.21.15.98:6789/0 conn(0x7fc0040c6b10 legacy=0x7fc004158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.294+0000 7fc0037fe640 1 -- 172.21.15.1:0/965291044 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0040c6690 con 0x7fc004152650 2024-03-06T07:21:26.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.295+0000 7fc0037fe640 1 --2- 172.21.15.1:0/965291044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc004152650 0x7fc004155c90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc0041597b0 tx=0x7fbff402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8900fb461adfcc4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.295+0000 7fc002ffd640 1 -- 172.21.15.1:0/965291044 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff4035020 con 0x7fc004152650 2024-03-06T07:21:26.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.295+0000 7fc002ffd640 1 -- 172.21.15.1:0/965291044 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff4031870 con 0x7fc004152650 2024-03-06T07:21:26.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.295+0000 7fc002ffd640 1 -- 172.21.15.1:0/965291044 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff4031c40 con 0x7fc004152650 2024-03-06T07:21:26.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.296+0000 7fc009f6c640 1 -- 172.21.15.1:0/965291044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc004152650 msgr2=0x7fc004155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.296+0000 7fc009f6c640 1 --2- 172.21.15.1:0/965291044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc004152650 0x7fc004155c90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc0041597b0 tx=0x7fbff402b910 comp rx=0 tx=0).stop 2024-03-06T07:21:26.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.296+0000 7fc009f6c640 1 -- 172.21.15.1:0/965291044 shutdown_connections 2024-03-06T07:21:26.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.296+0000 7fc009f6c640 1 --2- 172.21.15.1:0/965291044 >> v2:172.21.15.98:3300/0 conn(0x7fc0040bf9d0 0x7fc0040c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.296+0000 7fc009f6c640 1 --2- 172.21.15.1:0/965291044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc004152650 0x7fc004155c90 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.296+0000 7fc009f6c640 1 -- 172.21.15.1:0/965291044 >> 172.21.15.1:0/965291044 conn(0x7fc004066ee0 msgr2=0x7fc0040672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:26.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.296+0000 7fc009f6c640 1 -- 172.21.15.1:0/965291044 shutdown_connections 2024-03-06T07:21:26.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.296+0000 7fc009f6c640 1 -- 172.21.15.1:0/965291044 wait complete. 2024-03-06T07:21:26.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.297+0000 7fc009f6c640 1 Processor -- start 2024-03-06T07:21:26.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.297+0000 7fc009f6c640 1 -- start start 2024-03-06T07:21:26.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.298+0000 7fc009f6c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0040bf9d0 0x7fc004158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.298+0000 7fc009f6c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc004152650 0x7fc00414ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.298+0000 7fc009f6c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc00414b360 0x7fc0041ee1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.298+0000 7fc009f6c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0041551b0 con 0x7fc00414b360 2024-03-06T07:21:26.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.298+0000 7fc009f6c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0040c6fa0 con 0x7fc004152650 2024-03-06T07:21:26.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.298+0000 7fc009f6c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc004155490 con 0x7fc0040bf9d0 2024-03-06T07:21:26.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.298+0000 7fc003fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc00414b360 0x7fc0041ee1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.298+0000 7fc003fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc00414b360 0x7fc0041ee1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37026/0 (socket says 172.21.15.1:37026) 2024-03-06T07:21:26.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.298+0000 7fc003fff640 1 -- 172.21.15.1:0/1101985749 learned_addr learned my addr 172.21.15.1:0/1101985749 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:26.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.298+0000 7fbffbfff640 1 --2- 172.21.15.1:0/1101985749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc004152650 0x7fc00414ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.298+0000 7fc0037fe640 1 --2- 172.21.15.1:0/1101985749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0040bf9d0 0x7fc004158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.298+0000 7fc003fff640 1 -- 172.21.15.1:0/1101985749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0040bf9d0 msgr2=0x7fc004158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.298+0000 7fc003fff640 1 --2- 172.21.15.1:0/1101985749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0040bf9d0 0x7fc004158790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.298+0000 7fc003fff640 1 -- 172.21.15.1:0/1101985749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc004152650 msgr2=0x7fc00414ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.298+0000 7fc003fff640 1 --2- 172.21.15.1:0/1101985749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc004152650 0x7fc00414ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.298+0000 7fc003fff640 1 -- 172.21.15.1:0/1101985749 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc004156e90 con 0x7fc00414b360 2024-03-06T07:21:26.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.298+0000 7fc003fff640 1 --2- 172.21.15.1:0/1101985749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc00414b360 0x7fc0041ee1b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbffc00efc0 tx=0x7fbffc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.299+0000 7fc000ff9640 1 -- 172.21.15.1:0/1101985749 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc018020 con 0x7fc00414b360 2024-03-06T07:21:26.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.299+0000 7fc000ff9640 1 -- 172.21.15.1:0/1101985749 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbffc014480 con 0x7fc00414b360 2024-03-06T07:21:26.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.299+0000 7fc009f6c640 1 -- 172.21.15.1:0/1101985749 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc004157170 con 0x7fc00414b360 2024-03-06T07:21:26.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.299+0000 7fc000ff9640 1 -- 172.21.15.1:0/1101985749 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc014870 con 0x7fc00414b360 2024-03-06T07:21:26.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.299+0000 7fc009f6c640 1 -- 172.21.15.1:0/1101985749 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc004151d80 con 0x7fc00414b360 2024-03-06T07:21:26.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.300+0000 7fc009f6c640 1 -- 172.21.15.1:0/1101985749 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc004067fd0 con 0x7fc00414b360 2024-03-06T07:21:26.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.300+0000 7fc000ff9640 1 -- 172.21.15.1:0/1101985749 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbffc005d10 con 0x7fc00414b360 2024-03-06T07:21:26.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.300+0000 7fc000ff9640 1 --2- 172.21.15.1:0/1101985749 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbfe406b500 0x7fbfe406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.301+0000 7fc000ff9640 1 -- 172.21.15.1:0/1101985749 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbffc08b1b0 con 0x7fc00414b360 2024-03-06T07:21:26.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.301+0000 7fc0037fe640 1 --2- 172.21.15.1:0/1101985749 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbfe406b500 0x7fbfe406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.301+0000 7fc0037fe640 1 --2- 172.21.15.1:0/1101985749 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbfe406b500 0x7fbfe406d9c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbff4036000 tx=0x7fbff403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.305+0000 7fc000ff9640 1 -- 172.21.15.1:0/1101985749 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbffc022030 con 0x7fc00414b360 2024-03-06T07:21:26.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.481+0000 7fc009f6c640 1 -- 172.21.15.1:0/1101985749 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc0040b06f0 con 0x7fbfe406b500 2024-03-06T07:21:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.488+0000 7fc000ff9640 1 -- 172.21.15.1:0/1101985749 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237274 (secure 0 0 0) 0x7fc0040b06f0 con 0x7fbfe406b500 2024-03-06T07:21:26.492 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:21:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.490+0000 7fc009f6c640 1 -- 172.21.15.1:0/1101985749 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbfe406b500 msgr2=0x7fbfe406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.490+0000 7fc009f6c640 1 --2- 172.21.15.1:0/1101985749 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbfe406b500 0x7fbfe406d9c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbff4036000 tx=0x7fbff403c040 comp rx=0 tx=0).stop 2024-03-06T07:21:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.490+0000 7fc009f6c640 1 -- 172.21.15.1:0/1101985749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc00414b360 msgr2=0x7fc0041ee1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.490+0000 7fc009f6c640 1 --2- 172.21.15.1:0/1101985749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc00414b360 0x7fc0041ee1b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbffc00efc0 tx=0x7fbffc00c490 comp rx=0 tx=0).stop 2024-03-06T07:21:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.490+0000 7fc009f6c640 1 -- 172.21.15.1:0/1101985749 shutdown_connections 2024-03-06T07:21:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.490+0000 7fc009f6c640 1 --2- 172.21.15.1:0/1101985749 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbfe406b500 0x7fbfe406d9c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.490+0000 7fc009f6c640 1 --2- 172.21.15.1:0/1101985749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc00414b360 0x7fc0041ee1b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.490+0000 7fc009f6c640 1 --2- 172.21.15.1:0/1101985749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc004152650 0x7fc00414ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.490+0000 7fc009f6c640 1 --2- 172.21.15.1:0/1101985749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0040bf9d0 0x7fc004158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.490+0000 7fc009f6c640 1 -- 172.21.15.1:0/1101985749 >> 172.21.15.1:0/1101985749 conn(0x7fc004066ee0 msgr2=0x7fc0040c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.490+0000 7fc009f6c640 1 -- 172.21.15.1:0/1101985749 shutdown_connections 2024-03-06T07:21:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.490+0000 7fc009f6c640 1 -- 172.21.15.1:0/1101985749 wait complete. 2024-03-06T07:21:26.504 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-03-06T07:21:24.703091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":236132,"kb_used_data":3264,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738588,"statfs":{"total":773094113280,"available":772852314112,"internally_reserved":0,"allocated":3342336,"data_stored":2544176,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000864"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:22.145657+0000","last_change":"2024-03-06T07:21:22.145657+0000","last_active":"2024-03-06T07:21:22.145657+0000","last_peered":"2024-03-06T07:21:22.145657+0000","last_clean":"2024-03-06T07:21:22.145657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:22.145657+0000","last_undegraded":"2024-03-06T07:21:22.145657+0000","last_fullsized":"2024-03-06T07:21:22.145657+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:21.140798+0000","last_change":"2024-03-06T07:21:21.140798+0000","last_active":"2024-03-06T07:21:21.140798+0000","last_peered":"2024-03-06T07:21:21.140798+0000","last_clean":"2024-03-06T07:21:21.140798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:21.140798+0000","last_undegraded":"2024-03-06T07:21:21.140798+0000","last_fullsized":"2024-03-06T07:21:21.140798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":197,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:21:17.146365+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:21:17.146365+0000","last_peered":"2024-03-06T07:21:17.146365+0000","last_clean":"2024-03-06T07:21:17.146365+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:21:17.146365+0000","last_undegraded":"2024-03-06T07:21:17.146365+0000","last_fullsized":"2024-03-06T07:21:17.146365+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":466944,"data_stored":760798,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":170430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":170430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":170430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31584,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":401408,"data_stored":170430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31580,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":401408,"data_stored":170430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31580,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":401408,"data_stored":170430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31644,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340196,"statfs":{"total":96636764160,"available":96604360704,"internally_reserved":0,"allocated":466944,"data_stored":760798,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:26.510 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:26.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.644+0000 7fec04ab5640 1 Processor -- start 2024-03-06T07:21:26.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.644+0000 7fec04ab5640 1 -- start start 2024-03-06T07:21:26.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.644+0000 7fec04ab5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fec000c3970 0x7fec000c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.644+0000 7fec04ab5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fec00152840 0x7fec001553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.644+0000 7fec04ab5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec000c5e40 con 0x7fec000c60b0 2024-03-06T07:21:26.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.644+0000 7fec04ab5640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec000c67f0 con 0x7fec000c3970 2024-03-06T07:21:26.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.644+0000 7fec04ab5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec000c6a40 con 0x7fec00152840 2024-03-06T07:21:26.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.645+0000 7febfe59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fec000c3970 0x7fec000c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.645+0000 7febfe59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fec000c3970 0x7fec000c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37038/0 (socket says 172.21.15.1:37038) 2024-03-06T07:21:26.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.645+0000 7febf7fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fec000c60b0 0x7fec00152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60766/0 (socket says 172.21.15.1:60766) 2024-03-06T07:21:26.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.645+0000 7febfe59c640 1 -- 172.21.15.1:0/1295557970 learned_addr learned my addr 172.21.15.1:0/1295557970 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:26.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.645+0000 7febfed9d640 1 --2- 172.21.15.1:0/1295557970 >> v2:172.21.15.98:3300/0 conn(0x7fec00152840 0x7fec001553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.645+0000 7febfe59c640 1 -- 172.21.15.1:0/1295557970 >> v2:172.21.15.98:3300/0 conn(0x7fec00152840 msgr2=0x7fec001553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:26.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.645+0000 7febfe59c640 1 --2- 172.21.15.1:0/1295557970 >> v2:172.21.15.98:3300/0 conn(0x7fec00152840 0x7fec001553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.645+0000 7febfe59c640 1 -- 172.21.15.1:0/1295557970 >> v1:172.21.15.1:6789/0 conn(0x7fec000c60b0 legacy=0x7fec00152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.645+0000 7febfe59c640 1 -- 172.21.15.1:0/1295557970 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec000c6d20 con 0x7fec000c3970 2024-03-06T07:21:26.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.645+0000 7febfed9d640 1 --2- 172.21.15.1:0/1295557970 >> v2:172.21.15.98:3300/0 conn(0x7fec00152840 0x7fec001553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:26.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.646+0000 7febfe59c640 1 --2- 172.21.15.1:0/1295557970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec000c3970 0x7fec000c8170 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fec000c86b0 tx=0x7febf002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2991f0afb296529 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.646+0000 7febfdd9b640 1 -- 172.21.15.1:0/1295557970 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf0034070 con 0x7fec000c3970 2024-03-06T07:21:26.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.646+0000 7febfdd9b640 1 -- 172.21.15.1:0/1295557970 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf0030860 con 0x7fec000c3970 2024-03-06T07:21:26.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.646+0000 7febfdd9b640 1 -- 172.21.15.1:0/1295557970 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf0030c30 con 0x7fec000c3970 2024-03-06T07:21:26.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.646+0000 7fec04ab5640 1 -- 172.21.15.1:0/1295557970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec000c3970 msgr2=0x7fec000c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.646+0000 7fec04ab5640 1 --2- 172.21.15.1:0/1295557970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec000c3970 0x7fec000c8170 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fec000c86b0 tx=0x7febf002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.646+0000 7fec04ab5640 1 -- 172.21.15.1:0/1295557970 shutdown_connections 2024-03-06T07:21:26.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.646+0000 7fec04ab5640 1 --2- 172.21.15.1:0/1295557970 >> v2:172.21.15.98:3300/0 conn(0x7fec00152840 0x7fec001553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.646+0000 7fec04ab5640 1 --2- 172.21.15.1:0/1295557970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec000c3970 0x7fec000c8170 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.646+0000 7fec04ab5640 1 -- 172.21.15.1:0/1295557970 >> 172.21.15.1:0/1295557970 conn(0x7fec000c88f0 msgr2=0x7fec000c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:26.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.646+0000 7fec04ab5640 1 -- 172.21.15.1:0/1295557970 shutdown_connections 2024-03-06T07:21:26.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.646+0000 7fec04ab5640 1 -- 172.21.15.1:0/1295557970 wait complete. 2024-03-06T07:21:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.648+0000 7fec04ab5640 1 Processor -- start 2024-03-06T07:21:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.648+0000 7fec04ab5640 1 -- start start 2024-03-06T07:21:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.648+0000 7fec04ab5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec000c3970 0x7fec000a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.648+0000 7fec04ab5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec000c60b0 0x7fec0006fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.648+0000 7fec04ab5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec00152840 0x7fec000700e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.648+0000 7fec04ab5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec000c75c0 con 0x7fec000c3970 2024-03-06T07:21:26.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.648+0000 7fec04ab5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec000c72e0 con 0x7fec00152840 2024-03-06T07:21:26.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.648+0000 7fec04ab5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec000c3d50 con 0x7fec000c60b0 2024-03-06T07:21:26.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.648+0000 7febfe59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec000c3970 0x7fec000a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.648+0000 7febfe59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec000c3970 0x7fec000a97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37040/0 (socket says 172.21.15.1:37040) 2024-03-06T07:21:26.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.648+0000 7febfe59c640 1 -- 172.21.15.1:0/166166328 learned_addr learned my addr 172.21.15.1:0/166166328 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:26.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.649+0000 7febfed9d640 1 --2- 172.21.15.1:0/166166328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec00152840 0x7fec000700e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.649+0000 7febf7fff640 1 --2- 172.21.15.1:0/166166328 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec000c60b0 0x7fec0006fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.649+0000 7febfe59c640 1 -- 172.21.15.1:0/166166328 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec000c60b0 msgr2=0x7fec0006fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.649+0000 7febfe59c640 1 --2- 172.21.15.1:0/166166328 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec000c60b0 0x7fec0006fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.649+0000 7febfe59c640 1 -- 172.21.15.1:0/166166328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec00152840 msgr2=0x7fec000700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.649+0000 7febfe59c640 1 --2- 172.21.15.1:0/166166328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec00152840 0x7fec000700e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.649+0000 7febfe59c640 1 -- 172.21.15.1:0/166166328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec001eb9b0 con 0x7fec000c3970 2024-03-06T07:21:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.649+0000 7febfe59c640 1 --2- 172.21.15.1:0/166166328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec000c3970 0x7fec000a97d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7febf0000c00 tx=0x7febf0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.649+0000 7febf6ffd640 1 -- 172.21.15.1:0/166166328 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf0048070 con 0x7fec000c3970 2024-03-06T07:21:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.649+0000 7febf6ffd640 1 -- 172.21.15.1:0/166166328 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf0039d90 con 0x7fec000c3970 2024-03-06T07:21:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.649+0000 7febf6ffd640 1 -- 172.21.15.1:0/166166328 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf0034040 con 0x7fec000c3970 2024-03-06T07:21:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.649+0000 7fec04ab5640 1 -- 172.21.15.1:0/166166328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec00070950 con 0x7fec000c3970 2024-03-06T07:21:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.649+0000 7fec04ab5640 1 -- 172.21.15.1:0/166166328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec000a9d10 con 0x7fec000c3970 2024-03-06T07:21:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.650+0000 7febf6ffd640 1 -- 172.21.15.1:0/166166328 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7febf0037070 con 0x7fec000c3970 2024-03-06T07:21:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.650+0000 7fec04ab5640 1 -- 172.21.15.1:0/166166328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec00070e70 con 0x7fec000c3970 2024-03-06T07:21:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.651+0000 7febf6ffd640 1 --2- 172.21.15.1:0/166166328 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febe006b500 0x7febe006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.651+0000 7febf6ffd640 1 -- 172.21.15.1:0/166166328 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7febf00ac320 con 0x7fec000c3970 2024-03-06T07:21:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.651+0000 7febf7fff640 1 --2- 172.21.15.1:0/166166328 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febe006b500 0x7febe006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.651+0000 7febf7fff640 1 --2- 172.21.15.1:0/166166328 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febe006b500 0x7febe006d9c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fec001e88e0 tx=0x7febe8007cc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.655+0000 7febf6ffd640 1 -- 172.21.15.1:0/166166328 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7febf007b0b0 con 0x7fec000c3970 2024-03-06T07:21:26.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.833+0000 7fec04ab5640 1 -- 172.21.15.1:0/166166328 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fec000686e0 con 0x7febe006b500 2024-03-06T07:21:26.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.840+0000 7febf6ffd640 1 -- 172.21.15.1:0/166166328 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252005 (secure 0 0 0) 0x7fec000686e0 con 0x7febe006b500 2024-03-06T07:21:26.845 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:26.846 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:21:26.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.842+0000 7fec04ab5640 1 -- 172.21.15.1:0/166166328 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febe006b500 msgr2=0x7febe006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.842+0000 7fec04ab5640 1 --2- 172.21.15.1:0/166166328 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febe006b500 0x7febe006d9c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fec001e88e0 tx=0x7febe8007cc0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.842+0000 7fec04ab5640 1 -- 172.21.15.1:0/166166328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec000c3970 msgr2=0x7fec000a97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.842+0000 7fec04ab5640 1 --2- 172.21.15.1:0/166166328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec000c3970 0x7fec000a97d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7febf0000c00 tx=0x7febf0004730 comp rx=0 tx=0).stop 2024-03-06T07:21:26.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.842+0000 7fec04ab5640 1 -- 172.21.15.1:0/166166328 shutdown_connections 2024-03-06T07:21:26.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.842+0000 7fec04ab5640 1 --2- 172.21.15.1:0/166166328 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febe006b500 0x7febe006d9c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.842+0000 7fec04ab5640 1 --2- 172.21.15.1:0/166166328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec00152840 0x7fec000700e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.842+0000 7fec04ab5640 1 --2- 172.21.15.1:0/166166328 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec000c60b0 0x7fec0006fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.842+0000 7fec04ab5640 1 --2- 172.21.15.1:0/166166328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec000c3970 0x7fec000a97d0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.842+0000 7fec04ab5640 1 -- 172.21.15.1:0/166166328 >> 172.21.15.1:0/166166328 conn(0x7fec000c88f0 msgr2=0x7fec00158790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.842+0000 7fec04ab5640 1 -- 172.21.15.1:0/166166328 shutdown_connections 2024-03-06T07:21:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.842+0000 7fec04ab5640 1 -- 172.21.15.1:0/166166328 wait complete. 2024-03-06T07:21:26.855 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-03-06T07:21:26.703426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":134,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236476,"kb_used_data":3640,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738244,"statfs":{"total":773094113280,"available":772851961856,"internally_reserved":0,"allocated":3727360,"data_stored":2722724,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.552756"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165116+0000","last_change":"2024-03-06T07:21:23.182706+0000","last_active":"2024-03-06T07:21:24.165116+0000","last_peered":"2024-03-06T07:21:24.165116+0000","last_clean":"2024-03-06T07:21:24.165116+0000","last_became_active":"2024-03-06T07:21:23.182444+0000","last_became_peered":"2024-03-06T07:21:23.182444+0000","last_unstale":"2024-03-06T07:21:24.165116+0000","last_undegraded":"2024-03-06T07:21:24.165116+0000","last_fullsized":"2024-03-06T07:21:24.165116+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:54.354235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164707+0000","last_change":"2024-03-06T07:21:23.167573+0000","last_active":"2024-03-06T07:21:24.164707+0000","last_peered":"2024-03-06T07:21:24.164707+0000","last_clean":"2024-03-06T07:21:24.164707+0000","last_became_active":"2024-03-06T07:21:23.166786+0000","last_became_peered":"2024-03-06T07:21:23.166786+0000","last_unstale":"2024-03-06T07:21:24.164707+0000","last_undegraded":"2024-03-06T07:21:24.164707+0000","last_fullsized":"2024-03-06T07:21:24.164707+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:22:31.654787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.184371+0000","last_change":"2024-03-06T07:21:23.180108+0000","last_active":"2024-03-06T07:21:23.184371+0000","last_peered":"2024-03-06T07:21:23.184371+0000","last_clean":"2024-03-06T07:21:23.184371+0000","last_became_active":"2024-03-06T07:21:23.179812+0000","last_became_peered":"2024-03-06T07:21:23.179812+0000","last_unstale":"2024-03-06T07:21:23.184371+0000","last_undegraded":"2024-03-06T07:21:23.184371+0000","last_fullsized":"2024-03-06T07:21:23.184371+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:12.094182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164235+0000","last_change":"2024-03-06T07:21:23.175989+0000","last_active":"2024-03-06T07:21:24.164235+0000","last_peered":"2024-03-06T07:21:24.164235+0000","last_clean":"2024-03-06T07:21:24.164235+0000","last_became_active":"2024-03-06T07:21:23.175694+0000","last_became_peered":"2024-03-06T07:21:23.175694+0000","last_unstale":"2024-03-06T07:21:24.164235+0000","last_undegraded":"2024-03-06T07:21:24.164235+0000","last_fullsized":"2024-03-06T07:21:24.164235+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:10.625144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164055+0000","last_change":"2024-03-06T07:21:23.176818+0000","last_active":"2024-03-06T07:21:24.164055+0000","last_peered":"2024-03-06T07:21:24.164055+0000","last_clean":"2024-03-06T07:21:24.164055+0000","last_became_active":"2024-03-06T07:21:23.176528+0000","last_became_peered":"2024-03-06T07:21:23.176528+0000","last_unstale":"2024-03-06T07:21:24.164055+0000","last_undegraded":"2024-03-06T07:21:24.164055+0000","last_fullsized":"2024-03-06T07:21:24.164055+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:26.024267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.185276+0000","last_change":"2024-03-06T07:21:23.181991+0000","last_active":"2024-03-06T07:21:23.185276+0000","last_peered":"2024-03-06T07:21:23.185276+0000","last_clean":"2024-03-06T07:21:23.185276+0000","last_became_active":"2024-03-06T07:21:23.181716+0000","last_became_peered":"2024-03-06T07:21:23.181716+0000","last_unstale":"2024-03-06T07:21:23.185276+0000","last_undegraded":"2024-03-06T07:21:23.185276+0000","last_fullsized":"2024-03-06T07:21:23.185276+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:41.747922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.178997+0000","last_change":"2024-03-06T07:21:23.169886+0000","last_active":"2024-03-06T07:21:24.178997+0000","last_peered":"2024-03-06T07:21:24.178997+0000","last_clean":"2024-03-06T07:21:24.178997+0000","last_became_active":"2024-03-06T07:21:23.169504+0000","last_became_peered":"2024-03-06T07:21:23.169504+0000","last_unstale":"2024-03-06T07:21:24.178997+0000","last_undegraded":"2024-03-06T07:21:24.178997+0000","last_fullsized":"2024-03-06T07:21:24.178997+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:28:35.313141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165175+0000","last_change":"2024-03-06T07:21:23.180974+0000","last_active":"2024-03-06T07:21:24.165175+0000","last_peered":"2024-03-06T07:21:24.165175+0000","last_clean":"2024-03-06T07:21:24.165175+0000","last_became_active":"2024-03-06T07:21:23.180703+0000","last_became_peered":"2024-03-06T07:21:23.180703+0000","last_unstale":"2024-03-06T07:21:24.165175+0000","last_undegraded":"2024-03-06T07:21:24.165175+0000","last_fullsized":"2024-03-06T07:21:24.165175+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:06.540792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.181773+0000","last_change":"2024-03-06T07:21:23.179818+0000","last_active":"2024-03-06T07:21:23.181773+0000","last_peered":"2024-03-06T07:21:23.181773+0000","last_clean":"2024-03-06T07:21:23.181773+0000","last_became_active":"2024-03-06T07:21:23.172741+0000","last_became_peered":"2024-03-06T07:21:23.172741+0000","last_unstale":"2024-03-06T07:21:23.181773+0000","last_undegraded":"2024-03-06T07:21:23.181773+0000","last_fullsized":"2024-03-06T07:21:23.181773+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:07.628952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179312+0000","last_change":"2024-03-06T07:21:23.173744+0000","last_active":"2024-03-06T07:21:24.179312+0000","last_peered":"2024-03-06T07:21:24.179312+0000","last_clean":"2024-03-06T07:21:24.179312+0000","last_became_active":"2024-03-06T07:21:23.173427+0000","last_became_peered":"2024-03-06T07:21:23.173427+0000","last_unstale":"2024-03-06T07:21:24.179312+0000","last_undegraded":"2024-03-06T07:21:24.179312+0000","last_fullsized":"2024-03-06T07:21:24.179312+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:12.449775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179447+0000","last_change":"2024-03-06T07:21:23.178961+0000","last_active":"2024-03-06T07:21:24.179447+0000","last_peered":"2024-03-06T07:21:24.179447+0000","last_clean":"2024-03-06T07:21:24.179447+0000","last_became_active":"2024-03-06T07:21:23.178096+0000","last_became_peered":"2024-03-06T07:21:23.178096+0000","last_unstale":"2024-03-06T07:21:24.179447+0000","last_undegraded":"2024-03-06T07:21:24.179447+0000","last_fullsized":"2024-03-06T07:21:24.179447+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:22:31.654787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.170783+0000","last_change":"2024-03-06T07:21:23.169027+0000","last_active":"2024-03-06T07:21:23.170783+0000","last_peered":"2024-03-06T07:21:23.170783+0000","last_clean":"2024-03-06T07:21:23.170783+0000","last_became_active":"2024-03-06T07:21:23.168716+0000","last_became_peered":"2024-03-06T07:21:23.168716+0000","last_unstale":"2024-03-06T07:21:23.170783+0000","last_undegraded":"2024-03-06T07:21:23.170783+0000","last_fullsized":"2024-03-06T07:21:23.170783+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:08.049695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179083+0000","last_change":"2024-03-06T07:21:23.178092+0000","last_active":"2024-03-06T07:21:24.179083+0000","last_peered":"2024-03-06T07:21:24.179083+0000","last_clean":"2024-03-06T07:21:24.179083+0000","last_became_active":"2024-03-06T07:21:23.176982+0000","last_became_peered":"2024-03-06T07:21:23.176982+0000","last_unstale":"2024-03-06T07:21:24.179083+0000","last_undegraded":"2024-03-06T07:21:24.179083+0000","last_fullsized":"2024-03-06T07:21:24.179083+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:41.252208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.282915+0000","last_change":"2024-03-06T07:21:23.172743+0000","last_active":"2024-03-06T07:21:24.282915+0000","last_peered":"2024-03-06T07:21:24.282915+0000","last_clean":"2024-03-06T07:21:24.282915+0000","last_became_active":"2024-03-06T07:21:23.172486+0000","last_became_peered":"2024-03-06T07:21:23.172486+0000","last_unstale":"2024-03-06T07:21:24.282915+0000","last_undegraded":"2024-03-06T07:21:24.282915+0000","last_fullsized":"2024-03-06T07:21:24.282915+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:36.493357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179008+0000","last_change":"2024-03-06T07:21:23.170530+0000","last_active":"2024-03-06T07:21:24.179008+0000","last_peered":"2024-03-06T07:21:24.179008+0000","last_clean":"2024-03-06T07:21:24.179008+0000","last_became_active":"2024-03-06T07:21:23.170269+0000","last_became_peered":"2024-03-06T07:21:23.170269+0000","last_unstale":"2024-03-06T07:21:24.179008+0000","last_undegraded":"2024-03-06T07:21:24.179008+0000","last_fullsized":"2024-03-06T07:21:24.179008+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:11.514224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.179717+0000","last_change":"2024-03-06T07:21:23.175412+0000","last_active":"2024-03-06T07:21:23.179717+0000","last_peered":"2024-03-06T07:21:23.179717+0000","last_clean":"2024-03-06T07:21:23.179717+0000","last_became_active":"2024-03-06T07:21:23.175113+0000","last_became_peered":"2024-03-06T07:21:23.175113+0000","last_unstale":"2024-03-06T07:21:23.179717+0000","last_undegraded":"2024-03-06T07:21:23.179717+0000","last_fullsized":"2024-03-06T07:21:23.179717+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:41.747922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.175474+0000","last_change":"2024-03-06T07:21:23.174714+0000","last_active":"2024-03-06T07:21:23.175474+0000","last_peered":"2024-03-06T07:21:23.175474+0000","last_clean":"2024-03-06T07:21:23.175474+0000","last_became_active":"2024-03-06T07:21:23.174372+0000","last_became_peered":"2024-03-06T07:21:23.174372+0000","last_unstale":"2024-03-06T07:21:23.175474+0000","last_undegraded":"2024-03-06T07:21:23.175474+0000","last_fullsized":"2024-03-06T07:21:23.175474+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:22.077065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165483+0000","last_change":"2024-03-06T07:21:23.173359+0000","last_active":"2024-03-06T07:21:24.165483+0000","last_peered":"2024-03-06T07:21:24.165483+0000","last_clean":"2024-03-06T07:21:24.165483+0000","last_became_active":"2024-03-06T07:21:23.173040+0000","last_became_peered":"2024-03-06T07:21:23.173040+0000","last_unstale":"2024-03-06T07:21:24.165483+0000","last_undegraded":"2024-03-06T07:21:24.165483+0000","last_fullsized":"2024-03-06T07:21:24.165483+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:20.162045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164458+0000","last_change":"2024-03-06T07:21:23.177492+0000","last_active":"2024-03-06T07:21:24.164458+0000","last_peered":"2024-03-06T07:21:24.164458+0000","last_clean":"2024-03-06T07:21:24.164458+0000","last_became_active":"2024-03-06T07:21:23.177109+0000","last_became_peered":"2024-03-06T07:21:23.177109+0000","last_unstale":"2024-03-06T07:21:24.164458+0000","last_undegraded":"2024-03-06T07:21:24.164458+0000","last_fullsized":"2024-03-06T07:21:24.164458+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:50.469010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165246+0000","last_change":"2024-03-06T07:21:23.173277+0000","last_active":"2024-03-06T07:21:24.165246+0000","last_peered":"2024-03-06T07:21:24.165246+0000","last_clean":"2024-03-06T07:21:24.165246+0000","last_became_active":"2024-03-06T07:21:23.172974+0000","last_became_peered":"2024-03-06T07:21:23.172974+0000","last_unstale":"2024-03-06T07:21:24.165246+0000","last_undegraded":"2024-03-06T07:21:24.165246+0000","last_fullsized":"2024-03-06T07:21:24.165246+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:08.049695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179011+0000","last_change":"2024-03-06T07:21:23.176636+0000","last_active":"2024-03-06T07:21:24.179011+0000","last_peered":"2024-03-06T07:21:24.179011+0000","last_clean":"2024-03-06T07:21:24.179011+0000","last_became_active":"2024-03-06T07:21:23.176347+0000","last_became_peered":"2024-03-06T07:21:23.176347+0000","last_unstale":"2024-03-06T07:21:24.179011+0000","last_undegraded":"2024-03-06T07:21:24.179011+0000","last_fullsized":"2024-03-06T07:21:24.179011+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:54.233662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283249+0000","last_change":"2024-03-06T07:21:23.177707+0000","last_active":"2024-03-06T07:21:24.283249+0000","last_peered":"2024-03-06T07:21:24.283249+0000","last_clean":"2024-03-06T07:21:24.283249+0000","last_became_active":"2024-03-06T07:21:23.172182+0000","last_became_peered":"2024-03-06T07:21:23.172182+0000","last_unstale":"2024-03-06T07:21:24.283249+0000","last_undegraded":"2024-03-06T07:21:24.283249+0000","last_fullsized":"2024-03-06T07:21:24.283249+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:19.113059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.163484+0000","last_change":"2024-03-06T07:21:23.159844+0000","last_active":"2024-03-06T07:21:23.163484+0000","last_peered":"2024-03-06T07:21:23.163484+0000","last_clean":"2024-03-06T07:21:23.163484+0000","last_became_active":"2024-03-06T07:21:23.159578+0000","last_became_peered":"2024-03-06T07:21:23.159578+0000","last_unstale":"2024-03-06T07:21:23.163484+0000","last_undegraded":"2024-03-06T07:21:23.163484+0000","last_fullsized":"2024-03-06T07:21:23.163484+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:00.584960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164259+0000","last_change":"2024-03-06T07:21:23.167040+0000","last_active":"2024-03-06T07:21:24.164259+0000","last_peered":"2024-03-06T07:21:24.164259+0000","last_clean":"2024-03-06T07:21:24.164259+0000","last_became_active":"2024-03-06T07:21:23.166340+0000","last_became_peered":"2024-03-06T07:21:23.166340+0000","last_unstale":"2024-03-06T07:21:24.164259+0000","last_undegraded":"2024-03-06T07:21:24.164259+0000","last_fullsized":"2024-03-06T07:21:24.164259+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:59.215536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165465+0000","last_change":"2024-03-06T07:21:23.182761+0000","last_active":"2024-03-06T07:21:24.165465+0000","last_peered":"2024-03-06T07:21:24.165465+0000","last_clean":"2024-03-06T07:21:24.165465+0000","last_became_active":"2024-03-06T07:21:23.182467+0000","last_became_peered":"2024-03-06T07:21:23.182467+0000","last_unstale":"2024-03-06T07:21:24.165465+0000","last_undegraded":"2024-03-06T07:21:24.165465+0000","last_fullsized":"2024-03-06T07:21:24.165465+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:12.094182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.182386+0000","last_change":"2024-03-06T07:21:23.179741+0000","last_active":"2024-03-06T07:21:23.182386+0000","last_peered":"2024-03-06T07:21:23.182386+0000","last_clean":"2024-03-06T07:21:23.182386+0000","last_became_active":"2024-03-06T07:21:23.179444+0000","last_became_peered":"2024-03-06T07:21:23.179444+0000","last_unstale":"2024-03-06T07:21:23.182386+0000","last_undegraded":"2024-03-06T07:21:23.182386+0000","last_fullsized":"2024-03-06T07:21:23.182386+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:10.564903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165441+0000","last_change":"2024-03-06T07:21:23.173847+0000","last_active":"2024-03-06T07:21:24.165441+0000","last_peered":"2024-03-06T07:21:24.165441+0000","last_clean":"2024-03-06T07:21:24.165441+0000","last_became_active":"2024-03-06T07:21:23.173587+0000","last_became_peered":"2024-03-06T07:21:23.173587+0000","last_unstale":"2024-03-06T07:21:24.165441+0000","last_undegraded":"2024-03-06T07:21:24.165441+0000","last_fullsized":"2024-03-06T07:21:24.165441+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:07.628952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179265+0000","last_change":"2024-03-06T07:21:23.170610+0000","last_active":"2024-03-06T07:21:24.179265+0000","last_peered":"2024-03-06T07:21:24.179265+0000","last_clean":"2024-03-06T07:21:24.179265+0000","last_became_active":"2024-03-06T07:21:23.169962+0000","last_became_peered":"2024-03-06T07:21:23.169962+0000","last_unstale":"2024-03-06T07:21:24.179265+0000","last_undegraded":"2024-03-06T07:21:24.179265+0000","last_fullsized":"2024-03-06T07:21:24.179265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:22.556995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165379+0000","last_change":"2024-03-06T07:21:23.175388+0000","last_active":"2024-03-06T07:21:24.165379+0000","last_peered":"2024-03-06T07:21:24.165379+0000","last_clean":"2024-03-06T07:21:24.165379+0000","last_became_active":"2024-03-06T07:21:23.175097+0000","last_became_peered":"2024-03-06T07:21:23.175097+0000","last_unstale":"2024-03-06T07:21:24.165379+0000","last_undegraded":"2024-03-06T07:21:24.165379+0000","last_fullsized":"2024-03-06T07:21:24.165379+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:22.372508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165127+0000","last_change":"2024-03-06T07:21:23.178513+0000","last_active":"2024-03-06T07:21:24.165127+0000","last_peered":"2024-03-06T07:21:24.165127+0000","last_clean":"2024-03-06T07:21:24.165127+0000","last_became_active":"2024-03-06T07:21:23.177896+0000","last_became_peered":"2024-03-06T07:21:23.177896+0000","last_unstale":"2024-03-06T07:21:24.165127+0000","last_undegraded":"2024-03-06T07:21:24.165127+0000","last_fullsized":"2024-03-06T07:21:24.165127+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:00.584960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.169176+0000","last_change":"2024-03-06T07:21:23.162516+0000","last_active":"2024-03-06T07:21:23.169176+0000","last_peered":"2024-03-06T07:21:23.169176+0000","last_clean":"2024-03-06T07:21:23.169176+0000","last_became_active":"2024-03-06T07:21:23.162258+0000","last_became_peered":"2024-03-06T07:21:23.162258+0000","last_unstale":"2024-03-06T07:21:23.169176+0000","last_undegraded":"2024-03-06T07:21:23.169176+0000","last_fullsized":"2024-03-06T07:21:23.169176+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:08.049695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.174382+0000","last_change":"2024-03-06T07:21:23.172221+0000","last_active":"2024-03-06T07:21:23.174382+0000","last_peered":"2024-03-06T07:21:23.174382+0000","last_clean":"2024-03-06T07:21:23.174382+0000","last_became_active":"2024-03-06T07:21:23.171940+0000","last_became_peered":"2024-03-06T07:21:23.171940+0000","last_unstale":"2024-03-06T07:21:23.174382+0000","last_undegraded":"2024-03-06T07:21:23.174382+0000","last_fullsized":"2024-03-06T07:21:23.174382+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:20.162045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.177313+0000","last_change":"2024-03-06T07:21:23.175504+0000","last_active":"2024-03-06T07:21:23.177313+0000","last_peered":"2024-03-06T07:21:23.177313+0000","last_clean":"2024-03-06T07:21:23.177313+0000","last_became_active":"2024-03-06T07:21:23.175180+0000","last_became_peered":"2024-03-06T07:21:23.175180+0000","last_unstale":"2024-03-06T07:21:23.177313+0000","last_undegraded":"2024-03-06T07:21:23.177313+0000","last_fullsized":"2024-03-06T07:21:23.177313+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:52.373147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165580+0000","last_change":"2024-03-06T07:21:22.171884+0000","last_active":"2024-03-06T07:21:24.165580+0000","last_peered":"2024-03-06T07:21:24.165580+0000","last_clean":"2024-03-06T07:21:24.165580+0000","last_became_active":"2024-03-06T07:21:22.171573+0000","last_became_peered":"2024-03-06T07:21:22.171573+0000","last_unstale":"2024-03-06T07:21:24.165580+0000","last_undegraded":"2024-03-06T07:21:24.165580+0000","last_fullsized":"2024-03-06T07:21:24.165580+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:45.006491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.183431+0000","last_change":"2024-03-06T07:21:23.181934+0000","last_active":"2024-03-06T07:21:23.183431+0000","last_peered":"2024-03-06T07:21:23.183431+0000","last_clean":"2024-03-06T07:21:23.183431+0000","last_became_active":"2024-03-06T07:21:23.181645+0000","last_became_peered":"2024-03-06T07:21:23.181645+0000","last_unstale":"2024-03-06T07:21:23.183431+0000","last_undegraded":"2024-03-06T07:21:23.183431+0000","last_fullsized":"2024-03-06T07:21:23.183431+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:22.077065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.191649+0000","last_change":"2024-03-06T07:21:22.170291+0000","last_active":"2024-03-06T07:21:23.191649+0000","last_peered":"2024-03-06T07:21:23.191649+0000","last_clean":"2024-03-06T07:21:23.191649+0000","last_became_active":"2024-03-06T07:21:22.169996+0000","last_became_peered":"2024-03-06T07:21:22.169996+0000","last_unstale":"2024-03-06T07:21:23.191649+0000","last_undegraded":"2024-03-06T07:21:23.191649+0000","last_fullsized":"2024-03-06T07:21:23.191649+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:59.580100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283289+0000","last_change":"2024-03-06T07:21:23.171182+0000","last_active":"2024-03-06T07:21:24.283289+0000","last_peered":"2024-03-06T07:21:24.283289+0000","last_clean":"2024-03-06T07:21:24.283289+0000","last_became_active":"2024-03-06T07:21:23.170864+0000","last_became_peered":"2024-03-06T07:21:23.170864+0000","last_unstale":"2024-03-06T07:21:24.283289+0000","last_undegraded":"2024-03-06T07:21:24.283289+0000","last_fullsized":"2024-03-06T07:21:24.283289+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:07.635594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.191378+0000","last_change":"2024-03-06T07:21:22.174770+0000","last_active":"2024-03-06T07:21:23.191378+0000","last_peered":"2024-03-06T07:21:23.191378+0000","last_clean":"2024-03-06T07:21:23.191378+0000","last_became_active":"2024-03-06T07:21:22.174494+0000","last_became_peered":"2024-03-06T07:21:22.174494+0000","last_unstale":"2024-03-06T07:21:23.191378+0000","last_undegraded":"2024-03-06T07:21:23.191378+0000","last_fullsized":"2024-03-06T07:21:23.191378+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:06.624093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165090+0000","last_change":"2024-03-06T07:21:23.173050+0000","last_active":"2024-03-06T07:21:24.165090+0000","last_peered":"2024-03-06T07:21:24.165090+0000","last_clean":"2024-03-06T07:21:24.165090+0000","last_became_active":"2024-03-06T07:21:23.172741+0000","last_became_peered":"2024-03-06T07:21:23.172741+0000","last_unstale":"2024-03-06T07:21:24.165090+0000","last_undegraded":"2024-03-06T07:21:24.165090+0000","last_fullsized":"2024-03-06T07:21:24.165090+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:15.661528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283240+0000","last_change":"2024-03-06T07:21:22.166297+0000","last_active":"2024-03-06T07:21:24.283240+0000","last_peered":"2024-03-06T07:21:24.283240+0000","last_clean":"2024-03-06T07:21:24.283240+0000","last_became_active":"2024-03-06T07:21:22.166020+0000","last_became_peered":"2024-03-06T07:21:22.166020+0000","last_unstale":"2024-03-06T07:21:24.283240+0000","last_undegraded":"2024-03-06T07:21:24.283240+0000","last_fullsized":"2024-03-06T07:21:24.283240+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:59.580100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.184409+0000","last_change":"2024-03-06T07:21:23.180253+0000","last_active":"2024-03-06T07:21:23.184409+0000","last_peered":"2024-03-06T07:21:23.184409+0000","last_clean":"2024-03-06T07:21:23.184409+0000","last_became_active":"2024-03-06T07:21:23.174841+0000","last_became_peered":"2024-03-06T07:21:23.174841+0000","last_unstale":"2024-03-06T07:21:23.184409+0000","last_undegraded":"2024-03-06T07:21:23.184409+0000","last_fullsized":"2024-03-06T07:21:23.184409+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:20.162045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179164+0000","last_change":"2024-03-06T07:21:22.170216+0000","last_active":"2024-03-06T07:21:24.179164+0000","last_peered":"2024-03-06T07:21:24.179164+0000","last_clean":"2024-03-06T07:21:24.179164+0000","last_became_active":"2024-03-06T07:21:22.169909+0000","last_became_peered":"2024-03-06T07:21:22.169909+0000","last_unstale":"2024-03-06T07:21:24.179164+0000","last_undegraded":"2024-03-06T07:21:24.179164+0000","last_fullsized":"2024-03-06T07:21:24.179164+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:07.044836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179273+0000","last_change":"2024-03-06T07:21:23.176300+0000","last_active":"2024-03-06T07:21:24.179273+0000","last_peered":"2024-03-06T07:21:24.179273+0000","last_clean":"2024-03-06T07:21:24.179273+0000","last_became_active":"2024-03-06T07:21:23.175986+0000","last_became_peered":"2024-03-06T07:21:23.175986+0000","last_unstale":"2024-03-06T07:21:24.179273+0000","last_undegraded":"2024-03-06T07:21:24.179273+0000","last_fullsized":"2024-03-06T07:21:24.179273+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:53.558264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.186822+0000","last_change":"2024-03-06T07:21:22.174498+0000","last_active":"2024-03-06T07:21:23.186822+0000","last_peered":"2024-03-06T07:21:23.186822+0000","last_clean":"2024-03-06T07:21:23.186822+0000","last_became_active":"2024-03-06T07:21:22.174189+0000","last_became_peered":"2024-03-06T07:21:22.174189+0000","last_unstale":"2024-03-06T07:21:23.186822+0000","last_undegraded":"2024-03-06T07:21:23.186822+0000","last_fullsized":"2024-03-06T07:21:23.186822+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:40.673099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283270+0000","last_change":"2024-03-06T07:21:23.180855+0000","last_active":"2024-03-06T07:21:24.283270+0000","last_peered":"2024-03-06T07:21:24.283270+0000","last_clean":"2024-03-06T07:21:24.283270+0000","last_became_active":"2024-03-06T07:21:23.180588+0000","last_became_peered":"2024-03-06T07:21:23.180588+0000","last_unstale":"2024-03-06T07:21:24.283270+0000","last_undegraded":"2024-03-06T07:21:24.283270+0000","last_fullsized":"2024-03-06T07:21:24.283270+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:06.967643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.166457+0000","last_change":"2024-03-06T07:21:22.172223+0000","last_active":"2024-03-06T07:21:23.166457+0000","last_peered":"2024-03-06T07:21:23.166457+0000","last_clean":"2024-03-06T07:21:23.166457+0000","last_became_active":"2024-03-06T07:21:22.171912+0000","last_became_peered":"2024-03-06T07:21:22.171912+0000","last_unstale":"2024-03-06T07:21:23.166457+0000","last_undegraded":"2024-03-06T07:21:23.166457+0000","last_fullsized":"2024-03-06T07:21:23.166457+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:59.580100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164902+0000","last_change":"2024-03-06T07:21:23.174371+0000","last_active":"2024-03-06T07:21:24.164902+0000","last_peered":"2024-03-06T07:21:24.164902+0000","last_clean":"2024-03-06T07:21:24.164902+0000","last_became_active":"2024-03-06T07:21:23.173893+0000","last_became_peered":"2024-03-06T07:21:23.173893+0000","last_unstale":"2024-03-06T07:21:24.164902+0000","last_undegraded":"2024-03-06T07:21:24.164902+0000","last_fullsized":"2024-03-06T07:21:24.164902+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:37.222226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.282797+0000","last_change":"2024-03-06T07:21:22.174586+0000","last_active":"2024-03-06T07:21:24.282797+0000","last_peered":"2024-03-06T07:21:24.282797+0000","last_clean":"2024-03-06T07:21:24.282797+0000","last_became_active":"2024-03-06T07:21:22.174294+0000","last_became_peered":"2024-03-06T07:21:22.174294+0000","last_unstale":"2024-03-06T07:21:24.282797+0000","last_undegraded":"2024-03-06T07:21:24.282797+0000","last_fullsized":"2024-03-06T07:21:24.282797+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:40.673099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164248+0000","last_change":"2024-03-06T07:21:23.175308+0000","last_active":"2024-03-06T07:21:24.164248+0000","last_peered":"2024-03-06T07:21:24.164248+0000","last_clean":"2024-03-06T07:21:24.164248+0000","last_became_active":"2024-03-06T07:21:23.175045+0000","last_became_peered":"2024-03-06T07:21:23.175045+0000","last_unstale":"2024-03-06T07:21:24.164248+0000","last_undegraded":"2024-03-06T07:21:24.164248+0000","last_fullsized":"2024-03-06T07:21:24.164248+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:28.084724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165189+0000","last_change":"2024-03-06T07:21:22.173698+0000","last_active":"2024-03-06T07:21:24.165189+0000","last_peered":"2024-03-06T07:21:24.165189+0000","last_clean":"2024-03-06T07:21:24.165189+0000","last_became_active":"2024-03-06T07:21:22.173426+0000","last_became_peered":"2024-03-06T07:21:22.173426+0000","last_unstale":"2024-03-06T07:21:24.165189+0000","last_undegraded":"2024-03-06T07:21:24.165189+0000","last_fullsized":"2024-03-06T07:21:24.165189+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:16.351884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.174766+0000","last_change":"2024-03-06T07:21:23.171603+0000","last_active":"2024-03-06T07:21:23.174766+0000","last_peered":"2024-03-06T07:21:23.174766+0000","last_clean":"2024-03-06T07:21:23.174766+0000","last_became_active":"2024-03-06T07:21:23.171109+0000","last_became_peered":"2024-03-06T07:21:23.171109+0000","last_unstale":"2024-03-06T07:21:23.174766+0000","last_undegraded":"2024-03-06T07:21:23.174766+0000","last_fullsized":"2024-03-06T07:21:23.174766+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:10.564903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179439+0000","last_change":"2024-03-06T07:21:22.166081+0000","last_active":"2024-03-06T07:21:24.179439+0000","last_peered":"2024-03-06T07:21:24.179439+0000","last_clean":"2024-03-06T07:21:24.179439+0000","last_became_active":"2024-03-06T07:21:22.165820+0000","last_became_peered":"2024-03-06T07:21:22.165820+0000","last_unstale":"2024-03-06T07:21:24.179439+0000","last_undegraded":"2024-03-06T07:21:24.179439+0000","last_fullsized":"2024-03-06T07:21:24.179439+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:59.580100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.185507+0000","last_change":"2024-03-06T07:21:23.180325+0000","last_active":"2024-03-06T07:21:23.185507+0000","last_peered":"2024-03-06T07:21:23.185507+0000","last_clean":"2024-03-06T07:21:23.185507+0000","last_became_active":"2024-03-06T07:21:23.175546+0000","last_became_peered":"2024-03-06T07:21:23.175546+0000","last_unstale":"2024-03-06T07:21:23.185507+0000","last_undegraded":"2024-03-06T07:21:23.185507+0000","last_fullsized":"2024-03-06T07:21:23.185507+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:10.564903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179551+0000","last_change":"2024-03-06T07:21:22.173265+0000","last_active":"2024-03-06T07:21:24.179551+0000","last_peered":"2024-03-06T07:21:24.179551+0000","last_clean":"2024-03-06T07:21:24.179551+0000","last_became_active":"2024-03-06T07:21:22.172945+0000","last_became_peered":"2024-03-06T07:21:22.172945+0000","last_unstale":"2024-03-06T07:21:24.179551+0000","last_undegraded":"2024-03-06T07:21:24.179551+0000","last_fullsized":"2024-03-06T07:21:24.179551+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:21.367648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283250+0000","last_change":"2024-03-06T07:21:23.173659+0000","last_active":"2024-03-06T07:21:24.283250+0000","last_peered":"2024-03-06T07:21:24.283250+0000","last_clean":"2024-03-06T07:21:24.283250+0000","last_became_active":"2024-03-06T07:21:23.173401+0000","last_became_peered":"2024-03-06T07:21:23.173401+0000","last_unstale":"2024-03-06T07:21:24.283250+0000","last_undegraded":"2024-03-06T07:21:24.283250+0000","last_fullsized":"2024-03-06T07:21:24.283250+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:17.356744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164883+0000","last_change":"2024-03-06T07:21:22.170375+0000","last_active":"2024-03-06T07:21:24.164883+0000","last_peered":"2024-03-06T07:21:24.164883+0000","last_clean":"2024-03-06T07:21:24.164883+0000","last_became_active":"2024-03-06T07:21:22.170096+0000","last_became_peered":"2024-03-06T07:21:22.170096+0000","last_unstale":"2024-03-06T07:21:24.164883+0000","last_undegraded":"2024-03-06T07:21:24.164883+0000","last_fullsized":"2024-03-06T07:21:24.164883+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:35.488498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.184430+0000","last_change":"2024-03-06T07:21:23.180941+0000","last_active":"2024-03-06T07:21:23.184430+0000","last_peered":"2024-03-06T07:21:23.184430+0000","last_clean":"2024-03-06T07:21:23.184430+0000","last_became_active":"2024-03-06T07:21:23.180662+0000","last_became_peered":"2024-03-06T07:21:23.180662+0000","last_unstale":"2024-03-06T07:21:23.184430+0000","last_undegraded":"2024-03-06T07:21:23.184430+0000","last_fullsized":"2024-03-06T07:21:23.184430+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:41.747922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165167+0000","last_change":"2024-03-06T07:21:22.168669+0000","last_active":"2024-03-06T07:21:24.165167+0000","last_peered":"2024-03-06T07:21:24.165167+0000","last_clean":"2024-03-06T07:21:24.165167+0000","last_became_active":"2024-03-06T07:21:22.168382+0000","last_became_peered":"2024-03-06T07:21:22.168382+0000","last_unstale":"2024-03-06T07:21:24.165167+0000","last_undegraded":"2024-03-06T07:21:24.165167+0000","last_fullsized":"2024-03-06T07:21:24.165167+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:34.691115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165029+0000","last_change":"2024-03-06T07:21:23.170835+0000","last_active":"2024-03-06T07:21:24.165029+0000","last_peered":"2024-03-06T07:21:24.165029+0000","last_clean":"2024-03-06T07:21:24.165029+0000","last_became_active":"2024-03-06T07:21:23.170538+0000","last_became_peered":"2024-03-06T07:21:23.170538+0000","last_unstale":"2024-03-06T07:21:24.165029+0000","last_undegraded":"2024-03-06T07:21:24.165029+0000","last_fullsized":"2024-03-06T07:21:24.165029+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:08.450197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.166546+0000","last_change":"2024-03-06T07:21:22.173295+0000","last_active":"2024-03-06T07:21:23.166546+0000","last_peered":"2024-03-06T07:21:23.166546+0000","last_clean":"2024-03-06T07:21:23.166546+0000","last_became_active":"2024-03-06T07:21:22.172903+0000","last_became_peered":"2024-03-06T07:21:22.172903+0000","last_unstale":"2024-03-06T07:21:23.166546+0000","last_undegraded":"2024-03-06T07:21:23.166546+0000","last_fullsized":"2024-03-06T07:21:23.166546+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:07.044836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.172290+0000","last_change":"2024-03-06T07:21:23.163139+0000","last_active":"2024-03-06T07:21:23.172290+0000","last_peered":"2024-03-06T07:21:23.172290+0000","last_clean":"2024-03-06T07:21:23.172290+0000","last_became_active":"2024-03-06T07:21:23.162884+0000","last_became_peered":"2024-03-06T07:21:23.162884+0000","last_unstale":"2024-03-06T07:21:23.172290+0000","last_undegraded":"2024-03-06T07:21:23.172290+0000","last_fullsized":"2024-03-06T07:21:23.172290+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:22.372508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164924+0000","last_change":"2024-03-06T07:21:22.171888+0000","last_active":"2024-03-06T07:21:24.164924+0000","last_peered":"2024-03-06T07:21:24.164924+0000","last_clean":"2024-03-06T07:21:24.164924+0000","last_became_active":"2024-03-06T07:21:22.171588+0000","last_became_peered":"2024-03-06T07:21:22.171588+0000","last_unstale":"2024-03-06T07:21:24.164924+0000","last_undegraded":"2024-03-06T07:21:24.164924+0000","last_fullsized":"2024-03-06T07:21:24.164924+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:33.446854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165200+0000","last_change":"2024-03-06T07:21:23.168200+0000","last_active":"2024-03-06T07:21:24.165200+0000","last_peered":"2024-03-06T07:21:24.165200+0000","last_clean":"2024-03-06T07:21:24.165200+0000","last_became_active":"2024-03-06T07:21:23.167903+0000","last_became_peered":"2024-03-06T07:21:23.167903+0000","last_unstale":"2024-03-06T07:21:24.165200+0000","last_undegraded":"2024-03-06T07:21:24.165200+0000","last_fullsized":"2024-03-06T07:21:24.165200+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:01.144803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.188870+0000","last_change":"2024-03-06T07:21:22.175264+0000","last_active":"2024-03-06T07:21:23.188870+0000","last_peered":"2024-03-06T07:21:23.188870+0000","last_clean":"2024-03-06T07:21:23.188870+0000","last_became_active":"2024-03-06T07:21:22.174941+0000","last_became_peered":"2024-03-06T07:21:22.174941+0000","last_unstale":"2024-03-06T07:21:23.188870+0000","last_undegraded":"2024-03-06T07:21:23.188870+0000","last_fullsized":"2024-03-06T07:21:23.188870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:51.086001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164775+0000","last_change":"2024-03-06T07:21:23.171904+0000","last_active":"2024-03-06T07:21:24.164775+0000","last_peered":"2024-03-06T07:21:24.164775+0000","last_clean":"2024-03-06T07:21:24.164775+0000","last_became_active":"2024-03-06T07:21:23.171618+0000","last_became_peered":"2024-03-06T07:21:23.171618+0000","last_unstale":"2024-03-06T07:21:24.164775+0000","last_undegraded":"2024-03-06T07:21:24.164775+0000","last_fullsized":"2024-03-06T07:21:24.164775+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:48.917896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179498+0000","last_change":"2024-03-06T07:21:22.174985+0000","last_active":"2024-03-06T07:21:24.179498+0000","last_peered":"2024-03-06T07:21:24.179498+0000","last_clean":"2024-03-06T07:21:24.179498+0000","last_became_active":"2024-03-06T07:21:22.174284+0000","last_became_peered":"2024-03-06T07:21:22.174284+0000","last_unstale":"2024-03-06T07:21:24.179498+0000","last_undegraded":"2024-03-06T07:21:24.179498+0000","last_fullsized":"2024-03-06T07:21:24.179498+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:11.089323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179490+0000","last_change":"2024-03-06T07:21:23.179740+0000","last_active":"2024-03-06T07:21:24.179490+0000","last_peered":"2024-03-06T07:21:24.179490+0000","last_clean":"2024-03-06T07:21:24.179490+0000","last_became_active":"2024-03-06T07:21:23.177114+0000","last_became_peered":"2024-03-06T07:21:23.177114+0000","last_unstale":"2024-03-06T07:21:24.179490+0000","last_undegraded":"2024-03-06T07:21:24.179490+0000","last_fullsized":"2024-03-06T07:21:24.179490+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:36.368758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283361+0000","last_change":"2024-03-06T07:21:22.171126+0000","last_active":"2024-03-06T07:21:24.283361+0000","last_peered":"2024-03-06T07:21:24.283361+0000","last_clean":"2024-03-06T07:21:24.283361+0000","last_became_active":"2024-03-06T07:21:22.170803+0000","last_became_peered":"2024-03-06T07:21:22.170803+0000","last_unstale":"2024-03-06T07:21:24.283361+0000","last_undegraded":"2024-03-06T07:21:24.283361+0000","last_fullsized":"2024-03-06T07:21:24.283361+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:07.044836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283308+0000","last_change":"2024-03-06T07:21:23.174451+0000","last_active":"2024-03-06T07:21:24.283308+0000","last_peered":"2024-03-06T07:21:24.283308+0000","last_clean":"2024-03-06T07:21:24.283308+0000","last_became_active":"2024-03-06T07:21:23.174072+0000","last_became_peered":"2024-03-06T07:21:23.174072+0000","last_unstale":"2024-03-06T07:21:24.283308+0000","last_undegraded":"2024-03-06T07:21:24.283308+0000","last_fullsized":"2024-03-06T07:21:24.283308+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:05.754195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164461+0000","last_change":"2024-03-06T07:21:22.169060+0000","last_active":"2024-03-06T07:21:24.164461+0000","last_peered":"2024-03-06T07:21:24.164461+0000","last_clean":"2024-03-06T07:21:24.164461+0000","last_became_active":"2024-03-06T07:21:22.168716+0000","last_became_peered":"2024-03-06T07:21:22.168716+0000","last_unstale":"2024-03-06T07:21:24.164461+0000","last_undegraded":"2024-03-06T07:21:24.164461+0000","last_fullsized":"2024-03-06T07:21:24.164461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:15.088870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164509+0000","last_change":"2024-03-06T07:21:23.169921+0000","last_active":"2024-03-06T07:21:24.164509+0000","last_peered":"2024-03-06T07:21:24.164509+0000","last_clean":"2024-03-06T07:21:24.164509+0000","last_became_active":"2024-03-06T07:21:23.169610+0000","last_became_peered":"2024-03-06T07:21:23.169610+0000","last_unstale":"2024-03-06T07:21:24.164509+0000","last_undegraded":"2024-03-06T07:21:24.164509+0000","last_fullsized":"2024-03-06T07:21:24.164509+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:34.451714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.166561+0000","last_change":"2024-03-06T07:21:22.174120+0000","last_active":"2024-03-06T07:21:23.166561+0000","last_peered":"2024-03-06T07:21:23.166561+0000","last_clean":"2024-03-06T07:21:23.166561+0000","last_became_active":"2024-03-06T07:21:22.173826+0000","last_became_peered":"2024-03-06T07:21:22.173826+0000","last_unstale":"2024-03-06T07:21:23.166561+0000","last_undegraded":"2024-03-06T07:21:23.166561+0000","last_fullsized":"2024-03-06T07:21:23.166561+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:21.367648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.175577+0000","last_change":"2024-03-06T07:21:23.171548+0000","last_active":"2024-03-06T07:21:23.175577+0000","last_peered":"2024-03-06T07:21:23.175577+0000","last_clean":"2024-03-06T07:21:23.175577+0000","last_became_active":"2024-03-06T07:21:23.171272+0000","last_became_peered":"2024-03-06T07:21:23.171272+0000","last_unstale":"2024-03-06T07:21:23.175577+0000","last_undegraded":"2024-03-06T07:21:23.175577+0000","last_fullsized":"2024-03-06T07:21:23.175577+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:20.162045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164151+0000","last_change":"2024-03-06T07:21:22.165324+0000","last_active":"2024-03-06T07:21:24.164151+0000","last_peered":"2024-03-06T07:21:24.164151+0000","last_clean":"2024-03-06T07:21:24.164151+0000","last_became_active":"2024-03-06T07:21:22.165064+0000","last_became_peered":"2024-03-06T07:21:22.165064+0000","last_unstale":"2024-03-06T07:21:24.164151+0000","last_undegraded":"2024-03-06T07:21:24.164151+0000","last_fullsized":"2024-03-06T07:21:24.164151+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:42.726655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.184787+0000","last_change":"2024-03-06T07:21:23.180694+0000","last_active":"2024-03-06T07:21:23.184787+0000","last_peered":"2024-03-06T07:21:23.184787+0000","last_clean":"2024-03-06T07:21:23.184787+0000","last_became_active":"2024-03-06T07:21:23.180413+0000","last_became_peered":"2024-03-06T07:21:23.180413+0000","last_unstale":"2024-03-06T07:21:23.184787+0000","last_undegraded":"2024-03-06T07:21:23.184787+0000","last_fullsized":"2024-03-06T07:21:23.184787+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:22.077065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164033+0000","last_change":"2024-03-06T07:21:22.174287+0000","last_active":"2024-03-06T07:21:24.164033+0000","last_peered":"2024-03-06T07:21:24.164033+0000","last_clean":"2024-03-06T07:21:24.164033+0000","last_became_active":"2024-03-06T07:21:22.173947+0000","last_became_peered":"2024-03-06T07:21:22.173947+0000","last_unstale":"2024-03-06T07:21:24.164033+0000","last_undegraded":"2024-03-06T07:21:24.164033+0000","last_fullsized":"2024-03-06T07:21:24.164033+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:02.949041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179184+0000","last_change":"2024-03-06T07:21:23.173219+0000","last_active":"2024-03-06T07:21:24.179184+0000","last_peered":"2024-03-06T07:21:24.179184+0000","last_clean":"2024-03-06T07:21:24.179184+0000","last_became_active":"2024-03-06T07:21:23.170375+0000","last_became_peered":"2024-03-06T07:21:23.170375+0000","last_unstale":"2024-03-06T07:21:24.179184+0000","last_undegraded":"2024-03-06T07:21:24.179184+0000","last_fullsized":"2024-03-06T07:21:24.179184+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:48.584102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.157734+0000","last_change":"2024-03-06T07:21:22.174188+0000","last_active":"2024-03-06T07:21:23.157734+0000","last_peered":"2024-03-06T07:21:23.157734+0000","last_clean":"2024-03-06T07:21:23.157734+0000","last_became_active":"2024-03-06T07:21:22.173899+0000","last_became_peered":"2024-03-06T07:21:22.173899+0000","last_unstale":"2024-03-06T07:21:23.157734+0000","last_undegraded":"2024-03-06T07:21:23.157734+0000","last_fullsized":"2024-03-06T07:21:23.157734+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:26.708237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165047+0000","last_change":"2024-03-06T07:21:23.169931+0000","last_active":"2024-03-06T07:21:24.165047+0000","last_peered":"2024-03-06T07:21:24.165047+0000","last_clean":"2024-03-06T07:21:24.165047+0000","last_became_active":"2024-03-06T07:21:23.169572+0000","last_became_peered":"2024-03-06T07:21:23.169572+0000","last_unstale":"2024-03-06T07:21:24.165047+0000","last_undegraded":"2024-03-06T07:21:24.165047+0000","last_fullsized":"2024-03-06T07:21:24.165047+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:06.313962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.163980+0000","last_change":"2024-03-06T07:21:22.164640+0000","last_active":"2024-03-06T07:21:24.163980+0000","last_peered":"2024-03-06T07:21:24.163980+0000","last_clean":"2024-03-06T07:21:24.163980+0000","last_became_active":"2024-03-06T07:21:22.164106+0000","last_became_peered":"2024-03-06T07:21:22.164106+0000","last_unstale":"2024-03-06T07:21:24.163980+0000","last_undegraded":"2024-03-06T07:21:24.163980+0000","last_fullsized":"2024-03-06T07:21:24.163980+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:31.123680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.184450+0000","last_change":"2024-03-06T07:21:23.182404+0000","last_active":"2024-03-06T07:21:23.184450+0000","last_peered":"2024-03-06T07:21:23.184450+0000","last_clean":"2024-03-06T07:21:23.184450+0000","last_became_active":"2024-03-06T07:21:23.181761+0000","last_became_peered":"2024-03-06T07:21:23.181761+0000","last_unstale":"2024-03-06T07:21:23.184450+0000","last_undegraded":"2024-03-06T07:21:23.184450+0000","last_fullsized":"2024-03-06T07:21:23.184450+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:52.373147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179579+0000","last_change":"2024-03-06T07:21:22.175003+0000","last_active":"2024-03-06T07:21:24.179579+0000","last_peered":"2024-03-06T07:21:24.179579+0000","last_clean":"2024-03-06T07:21:24.179579+0000","last_became_active":"2024-03-06T07:21:22.172966+0000","last_became_peered":"2024-03-06T07:21:22.172966+0000","last_unstale":"2024-03-06T07:21:24.179579+0000","last_undegraded":"2024-03-06T07:21:24.179579+0000","last_fullsized":"2024-03-06T07:21:24.179579+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:06.624093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.174745+0000","last_change":"2024-03-06T07:21:23.171386+0000","last_active":"2024-03-06T07:21:23.174745+0000","last_peered":"2024-03-06T07:21:23.174745+0000","last_clean":"2024-03-06T07:21:23.174745+0000","last_became_active":"2024-03-06T07:21:23.170760+0000","last_became_peered":"2024-03-06T07:21:23.170760+0000","last_unstale":"2024-03-06T07:21:23.174745+0000","last_undegraded":"2024-03-06T07:21:23.174745+0000","last_fullsized":"2024-03-06T07:21:23.174745+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:12.094182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.178209+0000","last_change":"2024-03-06T07:21:22.161701+0000","last_active":"2024-03-06T07:21:24.178209+0000","last_peered":"2024-03-06T07:21:24.178209+0000","last_clean":"2024-03-06T07:21:24.178209+0000","last_became_active":"2024-03-06T07:21:22.161365+0000","last_became_peered":"2024-03-06T07:21:22.161365+0000","last_unstale":"2024-03-06T07:21:24.178209+0000","last_undegraded":"2024-03-06T07:21:24.178209+0000","last_fullsized":"2024-03-06T07:21:24.178209+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:35.557090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164786+0000","last_change":"2024-03-06T07:21:23.173494+0000","last_active":"2024-03-06T07:21:24.164786+0000","last_peered":"2024-03-06T07:21:24.164786+0000","last_clean":"2024-03-06T07:21:24.164786+0000","last_became_active":"2024-03-06T07:21:23.173209+0000","last_became_peered":"2024-03-06T07:21:23.173209+0000","last_unstale":"2024-03-06T07:21:24.164786+0000","last_undegraded":"2024-03-06T07:21:24.164786+0000","last_fullsized":"2024-03-06T07:21:24.164786+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:36.368758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.160431+0000","last_change":"2024-03-06T07:21:22.171248+0000","last_active":"2024-03-06T07:21:23.160431+0000","last_peered":"2024-03-06T07:21:23.160431+0000","last_clean":"2024-03-06T07:21:23.160431+0000","last_became_active":"2024-03-06T07:21:22.170909+0000","last_became_peered":"2024-03-06T07:21:22.170909+0000","last_unstale":"2024-03-06T07:21:23.160431+0000","last_undegraded":"2024-03-06T07:21:23.160431+0000","last_fullsized":"2024-03-06T07:21:23.160431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:40.673099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.169503+0000","last_change":"2024-03-06T07:21:23.168323+0000","last_active":"2024-03-06T07:21:23.169503+0000","last_peered":"2024-03-06T07:21:23.169503+0000","last_clean":"2024-03-06T07:21:23.169503+0000","last_became_active":"2024-03-06T07:21:23.168024+0000","last_became_peered":"2024-03-06T07:21:23.168024+0000","last_unstale":"2024-03-06T07:21:23.169503+0000","last_undegraded":"2024-03-06T07:21:23.169503+0000","last_fullsized":"2024-03-06T07:21:23.169503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:52.090861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.159776+0000","last_change":"2024-03-06T07:21:22.174503+0000","last_active":"2024-03-06T07:21:23.159776+0000","last_peered":"2024-03-06T07:21:23.159776+0000","last_clean":"2024-03-06T07:21:23.159776+0000","last_became_active":"2024-03-06T07:21:22.174138+0000","last_became_peered":"2024-03-06T07:21:22.174138+0000","last_unstale":"2024-03-06T07:21:23.159776+0000","last_undegraded":"2024-03-06T07:21:23.159776+0000","last_fullsized":"2024-03-06T07:21:23.159776+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:26.708237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.173809+0000","last_change":"2024-03-06T07:21:23.170871+0000","last_active":"2024-03-06T07:21:23.173809+0000","last_peered":"2024-03-06T07:21:23.173809+0000","last_clean":"2024-03-06T07:21:23.173809+0000","last_became_active":"2024-03-06T07:21:23.170384+0000","last_became_peered":"2024-03-06T07:21:23.170384+0000","last_unstale":"2024-03-06T07:21:23.173809+0000","last_undegraded":"2024-03-06T07:21:23.173809+0000","last_fullsized":"2024-03-06T07:21:23.173809+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:12.094182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165057+0000","last_change":"2024-03-06T07:21:22.169536+0000","last_active":"2024-03-06T07:21:24.165057+0000","last_peered":"2024-03-06T07:21:24.165057+0000","last_clean":"2024-03-06T07:21:24.165057+0000","last_became_active":"2024-03-06T07:21:22.169265+0000","last_became_peered":"2024-03-06T07:21:22.169265+0000","last_unstale":"2024-03-06T07:21:24.165057+0000","last_undegraded":"2024-03-06T07:21:24.165057+0000","last_fullsized":"2024-03-06T07:21:24.165057+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:49.464151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.282950+0000","last_change":"2024-03-06T07:21:23.170620+0000","last_active":"2024-03-06T07:21:24.282950+0000","last_peered":"2024-03-06T07:21:24.282950+0000","last_clean":"2024-03-06T07:21:24.282950+0000","last_became_active":"2024-03-06T07:21:23.170063+0000","last_became_peered":"2024-03-06T07:21:23.170063+0000","last_unstale":"2024-03-06T07:21:24.282950+0000","last_undegraded":"2024-03-06T07:21:24.282950+0000","last_fullsized":"2024-03-06T07:21:24.282950+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:35.695974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.282861+0000","last_change":"2024-03-06T07:21:22.172986+0000","last_active":"2024-03-06T07:21:24.282861+0000","last_peered":"2024-03-06T07:21:24.282861+0000","last_clean":"2024-03-06T07:21:24.282861+0000","last_became_active":"2024-03-06T07:21:22.172582+0000","last_became_peered":"2024-03-06T07:21:22.172582+0000","last_unstale":"2024-03-06T07:21:24.282861+0000","last_undegraded":"2024-03-06T07:21:24.282861+0000","last_fullsized":"2024-03-06T07:21:24.282861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:51.086001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.174929+0000","last_change":"2024-03-06T07:21:23.170353+0000","last_active":"2024-03-06T07:21:23.174929+0000","last_peered":"2024-03-06T07:21:23.174929+0000","last_clean":"2024-03-06T07:21:23.174929+0000","last_became_active":"2024-03-06T07:21:23.170078+0000","last_became_peered":"2024-03-06T07:21:23.170078+0000","last_unstale":"2024-03-06T07:21:23.174929+0000","last_undegraded":"2024-03-06T07:21:23.174929+0000","last_fullsized":"2024-03-06T07:21:23.174929+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:07.628952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164817+0000","last_change":"2024-03-06T07:21:22.173498+0000","last_active":"2024-03-06T07:21:24.164817+0000","last_peered":"2024-03-06T07:21:24.164817+0000","last_clean":"2024-03-06T07:21:24.164817+0000","last_became_active":"2024-03-06T07:21:22.173198+0000","last_became_peered":"2024-03-06T07:21:22.173198+0000","last_unstale":"2024-03-06T07:21:24.164817+0000","last_undegraded":"2024-03-06T07:21:24.164817+0000","last_fullsized":"2024-03-06T07:21:24.164817+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:18.108199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179450+0000","last_change":"2024-03-06T07:21:23.178114+0000","last_active":"2024-03-06T07:21:24.179450+0000","last_peered":"2024-03-06T07:21:24.179450+0000","last_clean":"2024-03-06T07:21:24.179450+0000","last_became_active":"2024-03-06T07:21:23.177700+0000","last_became_peered":"2024-03-06T07:21:23.177700+0000","last_unstale":"2024-03-06T07:21:24.179450+0000","last_undegraded":"2024-03-06T07:21:24.179450+0000","last_fullsized":"2024-03-06T07:21:24.179450+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:48.917896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.157712+0000","last_change":"2024-03-06T07:21:22.172115+0000","last_active":"2024-03-06T07:21:23.157712+0000","last_peered":"2024-03-06T07:21:23.157712+0000","last_clean":"2024-03-06T07:21:23.157712+0000","last_became_active":"2024-03-06T07:21:22.171844+0000","last_became_peered":"2024-03-06T07:21:22.171844+0000","last_unstale":"2024-03-06T07:21:23.157712+0000","last_undegraded":"2024-03-06T07:21:23.157712+0000","last_fullsized":"2024-03-06T07:21:23.157712+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:45.059780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":203,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.158976+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:21:23.158976+0000","last_peered":"2024-03-06T07:21:23.158976+0000","last_clean":"2024-03-06T07:21:23.158976+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:21:23.158976+0000","last_undegraded":"2024-03-06T07:21:23.158976+0000","last_fullsized":"2024-03-06T07:21:23.158976+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":520192,"data_stored":782523,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":197747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574864,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":446464,"data_stored":190585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":446464,"data_stored":191077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31644,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340196,"statfs":{"total":96636764160,"available":96604360704,"internally_reserved":0,"allocated":466944,"data_stored":198811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":442368,"data_stored":190543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31616,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340224,"statfs":{"total":96636764160,"available":96604389376,"internally_reserved":0,"allocated":442368,"data_stored":190563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":507904,"data_stored":780875,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:26.860 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:21:26.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.995+0000 7fc5a6c39640 1 Processor -- start 2024-03-06T07:21:26.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.995+0000 7fc5a6c39640 1 -- start start 2024-03-06T07:21:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.996+0000 7fc5a6c39640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5a00c3970 0x7fc5a00c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.996+0000 7fc5a6c39640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc5a0152840 0x7fc5a01553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.996+0000 7fc5a6c39640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a00c5e40 con 0x7fc5a00c60b0 2024-03-06T07:21:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.996+0000 7fc5a6c39640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5a00c67f0 con 0x7fc5a00c3970 2024-03-06T07:21:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.996+0000 7fc5a6c39640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5a00c6a40 con 0x7fc5a0152840 2024-03-06T07:21:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.996+0000 7fc5a49d5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5a00c3970 0x7fc5a00c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.996+0000 7fc5a49d5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5a00c3970 0x7fc5a00c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37046/0 (socket says 172.21.15.1:37046) 2024-03-06T07:21:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.996+0000 7fc5a49d5640 1 -- 172.21.15.1:0/3488352892 learned_addr learned my addr 172.21.15.1:0/3488352892 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.996+0000 7fc59ffff640 1 --1- 172.21.15.1:0/3488352892 >> v1:172.21.15.1:6789/0 conn(0x7fc5a00c60b0 0x7fc5a0152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60782/0 (socket says 172.21.15.1:60782) 2024-03-06T07:21:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.996+0000 7fc5a51d6640 1 --2- 172.21.15.1:0/3488352892 >> v2:172.21.15.98:3300/0 conn(0x7fc5a0152840 0x7fc5a01553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.996+0000 7fc5a49d5640 1 -- 172.21.15.1:0/3488352892 >> v2:172.21.15.98:3300/0 conn(0x7fc5a0152840 msgr2=0x7fc5a01553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.996+0000 7fc5a49d5640 1 --2- 172.21.15.1:0/3488352892 >> v2:172.21.15.98:3300/0 conn(0x7fc5a0152840 0x7fc5a01553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.996+0000 7fc5a49d5640 1 -- 172.21.15.1:0/3488352892 >> v1:172.21.15.1:6789/0 conn(0x7fc5a00c60b0 legacy=0x7fc5a0152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.996+0000 7fc5a49d5640 1 -- 172.21.15.1:0/3488352892 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a00c6d20 con 0x7fc5a00c3970 2024-03-06T07:21:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.996+0000 7fc59f7fe640 1 -- 172.21.15.1:0/3488352892 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 666115481 0 0) 0x7fc5a00c5e40 con 0x7fc5a00c60b0 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.997+0000 7fc5a49d5640 1 --2- 172.21.15.1:0/3488352892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a00c3970 0x7fc5a00c8170 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc5a00c86b0 tx=0x7fc59402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9792f73c0522a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.997+0000 7fc59f7fe640 1 -- 172.21.15.1:0/3488352892 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc594034070 con 0x7fc5a00c3970 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.997+0000 7fc59f7fe640 1 -- 172.21.15.1:0/3488352892 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc594030860 con 0x7fc5a00c3970 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.997+0000 7fc59f7fe640 1 -- 172.21.15.1:0/3488352892 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc594030c30 con 0x7fc5a00c3970 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.998+0000 7fc5a6c39640 1 -- 172.21.15.1:0/3488352892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a00c3970 msgr2=0x7fc5a00c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.998+0000 7fc5a6c39640 1 --2- 172.21.15.1:0/3488352892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a00c3970 0x7fc5a00c8170 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc5a00c86b0 tx=0x7fc59402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.998+0000 7fc5a6c39640 1 -- 172.21.15.1:0/3488352892 shutdown_connections 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.998+0000 7fc5a6c39640 1 --2- 172.21.15.1:0/3488352892 >> v2:172.21.15.98:3300/0 conn(0x7fc5a0152840 0x7fc5a01553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.998+0000 7fc5a6c39640 1 --2- 172.21.15.1:0/3488352892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a00c3970 0x7fc5a00c8170 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.998+0000 7fc5a6c39640 1 -- 172.21.15.1:0/3488352892 >> 172.21.15.1:0/3488352892 conn(0x7fc5a00c88f0 msgr2=0x7fc5a00c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.998+0000 7fc5a6c39640 1 -- 172.21.15.1:0/3488352892 shutdown_connections 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.998+0000 7fc5a6c39640 1 -- 172.21.15.1:0/3488352892 wait complete. 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.999+0000 7fc5a6c39640 1 Processor -- start 2024-03-06T07:21:27.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:26.999+0000 7fc5a6c39640 1 -- start start 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc5a6c39640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5a00c3970 0x7fc5a0070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc5a6c39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a00c60b0 0x7fc5a00707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc5a6c39640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a0152840 0x7fc5a014f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc5a6c39640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a00c75c0 con 0x7fc5a0152840 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc5a6c39640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a00c72e0 con 0x7fc5a00c60b0 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc5a6c39640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a00c3d50 con 0x7fc5a00c3970 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc5a51d6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a0152840 0x7fc5a014f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc5a51d6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a0152840 0x7fc5a014f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37060/0 (socket says 172.21.15.1:37060) 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc5a51d6640 1 -- 172.21.15.1:0/1417812601 learned_addr learned my addr 172.21.15.1:0/1417812601 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc5a49d5640 1 --2- 172.21.15.1:0/1417812601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5a00c3970 0x7fc5a0070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc59ffff640 1 --2- 172.21.15.1:0/1417812601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a00c60b0 0x7fc5a00707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc5a51d6640 1 -- 172.21.15.1:0/1417812601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5a00c3970 msgr2=0x7fc5a0070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc5a51d6640 1 --2- 172.21.15.1:0/1417812601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5a00c3970 0x7fc5a0070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc5a51d6640 1 -- 172.21.15.1:0/1417812601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a00c60b0 msgr2=0x7fc5a00707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc5a51d6640 1 --2- 172.21.15.1:0/1417812601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a00c60b0 0x7fc5a00707a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc5a51d6640 1 -- 172.21.15.1:0/1417812601 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a00ad700 con 0x7fc5a0152840 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc59ffff640 1 --2- 172.21.15.1:0/1417812601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a00c60b0 0x7fc5a00707a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc5a51d6640 1 --2- 172.21.15.1:0/1417812601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a0152840 0x7fc5a014f160 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc598009cb0 tx=0x7fc59800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc59d7fa640 1 -- 172.21.15.1:0/1417812601 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc598018020 con 0x7fc5a0152840 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc59d7fa640 1 -- 172.21.15.1:0/1417812601 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc598014480 con 0x7fc5a0152840 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc5a6c39640 1 -- 172.21.15.1:0/1417812601 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a01ee570 con 0x7fc5a0152840 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.001+0000 7fc59d7fa640 1 -- 172.21.15.1:0/1417812601 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc598014870 con 0x7fc5a0152840 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.002+0000 7fc5a6c39640 1 -- 172.21.15.1:0/1417812601 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a01566c0 con 0x7fc5a0152840 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.003+0000 7fc5a6c39640 1 -- 172.21.15.1:0/1417812601 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a00686e0 con 0x7fc5a0152840 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.003+0000 7fc59d7fa640 1 -- 172.21.15.1:0/1417812601 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc598005d10 con 0x7fc5a0152840 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.003+0000 7fc59d7fa640 1 --2- 172.21.15.1:0/1417812601 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc58406b500 0x7fc58406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.003+0000 7fc59d7fa640 1 -- 172.21.15.1:0/1417812601 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc59808b010 con 0x7fc5a0152840 2024-03-06T07:21:27.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.003+0000 7fc5a49d5640 1 --2- 172.21.15.1:0/1417812601 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc58406b500 0x7fc58406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.004+0000 7fc5a49d5640 1 --2- 172.21.15.1:0/1417812601 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc58406b500 0x7fc58406d9c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc594036000 tx=0x7fc594008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.008+0000 7fc59d7fa640 1 -- 172.21.15.1:0/1417812601 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc598022030 con 0x7fc5a0152840 2024-03-06T07:21:27.135 INFO:tasks.ceph.mds.a.smithi001.stderr:2024-03-06T07:21:27.133+0000 7efd774c5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:27.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.261+0000 7fc5a6c39640 1 -- 172.21.15.1:0/1417812601 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fc5a0151f00 con 0x7fc5a0152840 2024-03-06T07:21:27.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.262+0000 7fc59d7fa640 1 -- 172.21.15.1:0/1417812601 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1173 (secure 0 0 0) 0x7fc598010040 con 0x7fc5a0152840 2024-03-06T07:21:27.264 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:27.264 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":82,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709614,"num_in_osds":8,"osd_in_since":1709709606,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":242151424,"bytes_avail":772851961856,"bytes_total":773094113280,"write_bytes_sec":2581,"read_op_per_sec":0,"write_op_per_sec":8},"fsmap":{"epoch":16,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":5280}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-03-06T07:21:16.701529+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:21:27.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.263+0000 7fc5a6c39640 1 -- 172.21.15.1:0/1417812601 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc58406b500 msgr2=0x7fc58406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.263+0000 7fc5a6c39640 1 --2- 172.21.15.1:0/1417812601 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc58406b500 0x7fc58406d9c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc594036000 tx=0x7fc594008030 comp rx=0 tx=0).stop 2024-03-06T07:21:27.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.263+0000 7fc5a6c39640 1 -- 172.21.15.1:0/1417812601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a0152840 msgr2=0x7fc5a014f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.263+0000 7fc5a6c39640 1 --2- 172.21.15.1:0/1417812601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a0152840 0x7fc5a014f160 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc598009cb0 tx=0x7fc59800c490 comp rx=0 tx=0).stop 2024-03-06T07:21:27.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.264+0000 7fc5a6c39640 1 -- 172.21.15.1:0/1417812601 shutdown_connections 2024-03-06T07:21:27.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.264+0000 7fc5a6c39640 1 --2- 172.21.15.1:0/1417812601 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc58406b500 0x7fc58406d9c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.264+0000 7fc5a6c39640 1 --2- 172.21.15.1:0/1417812601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a0152840 0x7fc5a014f160 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.264+0000 7fc5a6c39640 1 --2- 172.21.15.1:0/1417812601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5a00c60b0 0x7fc5a00707a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.264+0000 7fc5a6c39640 1 --2- 172.21.15.1:0/1417812601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5a00c3970 0x7fc5a0070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.264+0000 7fc5a6c39640 1 -- 172.21.15.1:0/1417812601 >> 172.21.15.1:0/1417812601 conn(0x7fc5a00c88f0 msgr2=0x7fc5a00c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.264+0000 7fc5a6c39640 1 -- 172.21.15.1:0/1417812601 shutdown_connections 2024-03-06T07:21:27.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.264+0000 7fc5a6c39640 1 -- 172.21.15.1:0/1417812601 wait complete. 2024-03-06T07:21:27.276 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:21:27.276 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:27.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.375+0000 7f6399df6640 1 Processor -- start 2024-03-06T07:21:27.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.375+0000 7f6399df6640 1 -- start start 2024-03-06T07:21:27.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.376+0000 7f6399df6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6394150520 0x7f63941543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.376+0000 7f6399df6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63940c32f0 0x7f63941537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.376+0000 7f6399df6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6394150900 con 0x7f6394070c80 2024-03-06T07:21:27.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.376+0000 7f6399df6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6394071790 con 0x7f63940c32f0 2024-03-06T07:21:27.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.376+0000 7f6399df6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6394157050 con 0x7f6394150520 2024-03-06T07:21:27.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.376+0000 7f6392ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63940c32f0 0x7f63941537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.376+0000 7f6392ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63940c32f0 0x7f63941537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37064/0 (socket says 172.21.15.1:37064) 2024-03-06T07:21:27.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.376+0000 7f6392ffd640 1 -- 172.21.15.1:0/558099005 learned_addr learned my addr 172.21.15.1:0/558099005 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:27.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.376+0000 7f6392ffd640 1 -- 172.21.15.1:0/558099005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6394150520 msgr2=0x7f63941543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:27.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.376+0000 7f6392ffd640 1 --2- 172.21.15.1:0/558099005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6394150520 0x7f63941543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.376+0000 7f6392ffd640 1 -- 172.21.15.1:0/558099005 >> v1:172.21.15.98:6789/0 conn(0x7f6394070c80 legacy=0x7f63940c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:27.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.376+0000 7f6392ffd640 1 -- 172.21.15.1:0/558099005 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6394152fc0 con 0x7f63940c32f0 2024-03-06T07:21:27.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.377+0000 7f6392ffd640 1 --2- 172.21.15.1:0/558099005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63940c32f0 0x7f63941537c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6384009870 tx=0x7f638402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9282a5d734dd8d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.377+0000 7f63927fc640 1 -- 172.21.15.1:0/558099005 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6384035020 con 0x7f63940c32f0 2024-03-06T07:21:27.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.377+0000 7f63927fc640 1 -- 172.21.15.1:0/558099005 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6384031640 con 0x7f63940c32f0 2024-03-06T07:21:27.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.377+0000 7f63927fc640 1 -- 172.21.15.1:0/558099005 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638402d040 con 0x7f63940c32f0 2024-03-06T07:21:27.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.378+0000 7f6399df6640 1 -- 172.21.15.1:0/558099005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63940c32f0 msgr2=0x7f63941537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.378+0000 7f6399df6640 1 --2- 172.21.15.1:0/558099005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63940c32f0 0x7f63941537c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6384009870 tx=0x7f638402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.378+0000 7f6399df6640 1 -- 172.21.15.1:0/558099005 shutdown_connections 2024-03-06T07:21:27.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.378+0000 7f6399df6640 1 --2- 172.21.15.1:0/558099005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63940c32f0 0x7f63941537c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.378+0000 7f6399df6640 1 --2- 172.21.15.1:0/558099005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6394150520 0x7f63941543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.378+0000 7f6399df6640 1 -- 172.21.15.1:0/558099005 >> 172.21.15.1:0/558099005 conn(0x7f63940c88f0 msgr2=0x7f63940c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.378+0000 7f6399df6640 1 -- 172.21.15.1:0/558099005 shutdown_connections 2024-03-06T07:21:27.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.378+0000 7f6399df6640 1 -- 172.21.15.1:0/558099005 wait complete. 2024-03-06T07:21:27.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.379+0000 7f6399df6640 1 Processor -- start 2024-03-06T07:21:27.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.379+0000 7f6399df6640 1 -- start start 2024-03-06T07:21:27.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.380+0000 7f6399df6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394070c80 0x7f63941eed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.380+0000 7f6399df6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63940c32f0 0x7f6394155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.380+0000 7f6399df6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6394150520 0x7f63940c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.380+0000 7f6399df6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63940c7ec0 con 0x7f6394150520 2024-03-06T07:21:27.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.380+0000 7f6399df6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6394157330 con 0x7f6394070c80 2024-03-06T07:21:27.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.380+0000 7f6399df6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6394153d00 con 0x7f63940c32f0 2024-03-06T07:21:27.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.380+0000 7f6393fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6394150520 0x7f63940c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.380+0000 7f6393fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6394150520 0x7f63940c4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37072/0 (socket says 172.21.15.1:37072) 2024-03-06T07:21:27.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.380+0000 7f6393fff640 1 -- 172.21.15.1:0/2101402554 learned_addr learned my addr 172.21.15.1:0/2101402554 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:27.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.380+0000 7f6392ffd640 1 --2- 172.21.15.1:0/2101402554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63940c32f0 0x7f6394155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.380+0000 7f63937fe640 1 --2- 172.21.15.1:0/2101402554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394070c80 0x7f63941eed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.380+0000 7f6393fff640 1 -- 172.21.15.1:0/2101402554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63940c32f0 msgr2=0x7f6394155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.380+0000 7f6393fff640 1 --2- 172.21.15.1:0/2101402554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63940c32f0 0x7f6394155e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.380+0000 7f6393fff640 1 -- 172.21.15.1:0/2101402554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394070c80 msgr2=0x7f63941eed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.380+0000 7f6393fff640 1 --2- 172.21.15.1:0/2101402554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394070c80 0x7f63941eed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.380+0000 7f6393fff640 1 -- 172.21.15.1:0/2101402554 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63940ace80 con 0x7f6394150520 2024-03-06T07:21:27.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.380+0000 7f6393fff640 1 --2- 172.21.15.1:0/2101402554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6394150520 0x7f63940c4a50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f638c00c780 tx=0x7f638c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.381+0000 7f638bfff640 1 -- 172.21.15.1:0/2101402554 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c017070 con 0x7f6394150520 2024-03-06T07:21:27.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.381+0000 7f638bfff640 1 -- 172.21.15.1:0/2101402554 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638c003da0 con 0x7f6394150520 2024-03-06T07:21:27.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.381+0000 7f638bfff640 1 -- 172.21.15.1:0/2101402554 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c0052b0 con 0x7f6394150520 2024-03-06T07:21:27.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.381+0000 7f6399df6640 1 -- 172.21.15.1:0/2101402554 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63940ad160 con 0x7f6394150520 2024-03-06T07:21:27.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.381+0000 7f6399df6640 1 -- 172.21.15.1:0/2101402554 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63941eaf30 con 0x7f6394150520 2024-03-06T07:21:27.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.382+0000 7f6399df6640 1 -- 172.21.15.1:0/2101402554 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63940686e0 con 0x7f6394150520 2024-03-06T07:21:27.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.382+0000 7f638bfff640 1 -- 172.21.15.1:0/2101402554 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f638c01a070 con 0x7f6394150520 2024-03-06T07:21:27.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.382+0000 7f638bfff640 1 --2- 172.21.15.1:0/2101402554 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f637406b500 0x7f637406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.383+0000 7f638bfff640 1 -- 172.21.15.1:0/2101402554 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f638c0898f0 con 0x7f6394150520 2024-03-06T07:21:27.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.383+0000 7f63937fe640 1 --2- 172.21.15.1:0/2101402554 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f637406b500 0x7f637406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.383+0000 7f63937fe640 1 --2- 172.21.15.1:0/2101402554 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f637406b500 0x7f637406d9c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f637c0098e0 tx=0x7f637c009060 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.387+0000 7f638bfff640 1 -- 172.21.15.1:0/2101402554 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f638c057f80 con 0x7f6394150520 2024-03-06T07:21:27.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.604+0000 7f6399df6640 1 -- 172.21.15.1:0/2101402554 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6394156360 con 0x7f637406b500 2024-03-06T07:21:27.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.610+0000 7f638bfff640 1 -- 172.21.15.1:0/2101402554 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252005 (secure 0 0 0) 0x7f6394156360 con 0x7f637406b500 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:27.617 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:21:27.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.613+0000 7f6399df6640 1 -- 172.21.15.1:0/2101402554 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f637406b500 msgr2=0x7f637406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.613+0000 7f6399df6640 1 --2- 172.21.15.1:0/2101402554 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f637406b500 0x7f637406d9c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f637c0098e0 tx=0x7f637c009060 comp rx=0 tx=0).stop 2024-03-06T07:21:27.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.613+0000 7f6399df6640 1 -- 172.21.15.1:0/2101402554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6394150520 msgr2=0x7f63940c4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.613+0000 7f6399df6640 1 --2- 172.21.15.1:0/2101402554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6394150520 0x7f63940c4a50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f638c00c780 tx=0x7f638c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.614+0000 7f6399df6640 1 -- 172.21.15.1:0/2101402554 shutdown_connections 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.614+0000 7f6399df6640 1 --2- 172.21.15.1:0/2101402554 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f637406b500 0x7f637406d9c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.614+0000 7f6399df6640 1 --2- 172.21.15.1:0/2101402554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6394150520 0x7f63940c4a50 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.614+0000 7f6399df6640 1 --2- 172.21.15.1:0/2101402554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63940c32f0 0x7f6394155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.614+0000 7f6399df6640 1 --2- 172.21.15.1:0/2101402554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6394070c80 0x7f63941eed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.614+0000 7f6399df6640 1 -- 172.21.15.1:0/2101402554 >> 172.21.15.1:0/2101402554 conn(0x7f63940c88f0 msgr2=0x7f63941df620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.614+0000 7f6399df6640 1 -- 172.21.15.1:0/2101402554 shutdown_connections 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:27.614+0000 7f6399df6640 1 -- 172.21.15.1:0/2101402554 wait complete. 2024-03-06T07:21:27.630 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-03-06T07:21:26.703426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":134,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236476,"kb_used_data":3640,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738244,"statfs":{"total":773094113280,"available":772851961856,"internally_reserved":0,"allocated":3727360,"data_stored":2722724,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.552756"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165116+0000","last_change":"2024-03-06T07:21:23.182706+0000","last_active":"2024-03-06T07:21:24.165116+0000","last_peered":"2024-03-06T07:21:24.165116+0000","last_clean":"2024-03-06T07:21:24.165116+0000","last_became_active":"2024-03-06T07:21:23.182444+0000","last_became_peered":"2024-03-06T07:21:23.182444+0000","last_unstale":"2024-03-06T07:21:24.165116+0000","last_undegraded":"2024-03-06T07:21:24.165116+0000","last_fullsized":"2024-03-06T07:21:24.165116+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:54.354235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164707+0000","last_change":"2024-03-06T07:21:23.167573+0000","last_active":"2024-03-06T07:21:24.164707+0000","last_peered":"2024-03-06T07:21:24.164707+0000","last_clean":"2024-03-06T07:21:24.164707+0000","last_became_active":"2024-03-06T07:21:23.166786+0000","last_became_peered":"2024-03-06T07:21:23.166786+0000","last_unstale":"2024-03-06T07:21:24.164707+0000","last_undegraded":"2024-03-06T07:21:24.164707+0000","last_fullsized":"2024-03-06T07:21:24.164707+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:22:31.654787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.184371+0000","last_change":"2024-03-06T07:21:23.180108+0000","last_active":"2024-03-06T07:21:23.184371+0000","last_peered":"2024-03-06T07:21:23.184371+0000","last_clean":"2024-03-06T07:21:23.184371+0000","last_became_active":"2024-03-06T07:21:23.179812+0000","last_became_peered":"2024-03-06T07:21:23.179812+0000","last_unstale":"2024-03-06T07:21:23.184371+0000","last_undegraded":"2024-03-06T07:21:23.184371+0000","last_fullsized":"2024-03-06T07:21:23.184371+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:12.094182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164235+0000","last_change":"2024-03-06T07:21:23.175989+0000","last_active":"2024-03-06T07:21:24.164235+0000","last_peered":"2024-03-06T07:21:24.164235+0000","last_clean":"2024-03-06T07:21:24.164235+0000","last_became_active":"2024-03-06T07:21:23.175694+0000","last_became_peered":"2024-03-06T07:21:23.175694+0000","last_unstale":"2024-03-06T07:21:24.164235+0000","last_undegraded":"2024-03-06T07:21:24.164235+0000","last_fullsized":"2024-03-06T07:21:24.164235+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:10.625144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164055+0000","last_change":"2024-03-06T07:21:23.176818+0000","last_active":"2024-03-06T07:21:24.164055+0000","last_peered":"2024-03-06T07:21:24.164055+0000","last_clean":"2024-03-06T07:21:24.164055+0000","last_became_active":"2024-03-06T07:21:23.176528+0000","last_became_peered":"2024-03-06T07:21:23.176528+0000","last_unstale":"2024-03-06T07:21:24.164055+0000","last_undegraded":"2024-03-06T07:21:24.164055+0000","last_fullsized":"2024-03-06T07:21:24.164055+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:26.024267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.185276+0000","last_change":"2024-03-06T07:21:23.181991+0000","last_active":"2024-03-06T07:21:23.185276+0000","last_peered":"2024-03-06T07:21:23.185276+0000","last_clean":"2024-03-06T07:21:23.185276+0000","last_became_active":"2024-03-06T07:21:23.181716+0000","last_became_peered":"2024-03-06T07:21:23.181716+0000","last_unstale":"2024-03-06T07:21:23.185276+0000","last_undegraded":"2024-03-06T07:21:23.185276+0000","last_fullsized":"2024-03-06T07:21:23.185276+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:41.747922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.178997+0000","last_change":"2024-03-06T07:21:23.169886+0000","last_active":"2024-03-06T07:21:24.178997+0000","last_peered":"2024-03-06T07:21:24.178997+0000","last_clean":"2024-03-06T07:21:24.178997+0000","last_became_active":"2024-03-06T07:21:23.169504+0000","last_became_peered":"2024-03-06T07:21:23.169504+0000","last_unstale":"2024-03-06T07:21:24.178997+0000","last_undegraded":"2024-03-06T07:21:24.178997+0000","last_fullsized":"2024-03-06T07:21:24.178997+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:28:35.313141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165175+0000","last_change":"2024-03-06T07:21:23.180974+0000","last_active":"2024-03-06T07:21:24.165175+0000","last_peered":"2024-03-06T07:21:24.165175+0000","last_clean":"2024-03-06T07:21:24.165175+0000","last_became_active":"2024-03-06T07:21:23.180703+0000","last_became_peered":"2024-03-06T07:21:23.180703+0000","last_unstale":"2024-03-06T07:21:24.165175+0000","last_undegraded":"2024-03-06T07:21:24.165175+0000","last_fullsized":"2024-03-06T07:21:24.165175+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:06.540792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.181773+0000","last_change":"2024-03-06T07:21:23.179818+0000","last_active":"2024-03-06T07:21:23.181773+0000","last_peered":"2024-03-06T07:21:23.181773+0000","last_clean":"2024-03-06T07:21:23.181773+0000","last_became_active":"2024-03-06T07:21:23.172741+0000","last_became_peered":"2024-03-06T07:21:23.172741+0000","last_unstale":"2024-03-06T07:21:23.181773+0000","last_undegraded":"2024-03-06T07:21:23.181773+0000","last_fullsized":"2024-03-06T07:21:23.181773+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:07.628952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179312+0000","last_change":"2024-03-06T07:21:23.173744+0000","last_active":"2024-03-06T07:21:24.179312+0000","last_peered":"2024-03-06T07:21:24.179312+0000","last_clean":"2024-03-06T07:21:24.179312+0000","last_became_active":"2024-03-06T07:21:23.173427+0000","last_became_peered":"2024-03-06T07:21:23.173427+0000","last_unstale":"2024-03-06T07:21:24.179312+0000","last_undegraded":"2024-03-06T07:21:24.179312+0000","last_fullsized":"2024-03-06T07:21:24.179312+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:12.449775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179447+0000","last_change":"2024-03-06T07:21:23.178961+0000","last_active":"2024-03-06T07:21:24.179447+0000","last_peered":"2024-03-06T07:21:24.179447+0000","last_clean":"2024-03-06T07:21:24.179447+0000","last_became_active":"2024-03-06T07:21:23.178096+0000","last_became_peered":"2024-03-06T07:21:23.178096+0000","last_unstale":"2024-03-06T07:21:24.179447+0000","last_undegraded":"2024-03-06T07:21:24.179447+0000","last_fullsized":"2024-03-06T07:21:24.179447+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:22:31.654787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.170783+0000","last_change":"2024-03-06T07:21:23.169027+0000","last_active":"2024-03-06T07:21:23.170783+0000","last_peered":"2024-03-06T07:21:23.170783+0000","last_clean":"2024-03-06T07:21:23.170783+0000","last_became_active":"2024-03-06T07:21:23.168716+0000","last_became_peered":"2024-03-06T07:21:23.168716+0000","last_unstale":"2024-03-06T07:21:23.170783+0000","last_undegraded":"2024-03-06T07:21:23.170783+0000","last_fullsized":"2024-03-06T07:21:23.170783+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:08.049695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179083+0000","last_change":"2024-03-06T07:21:23.178092+0000","last_active":"2024-03-06T07:21:24.179083+0000","last_peered":"2024-03-06T07:21:24.179083+0000","last_clean":"2024-03-06T07:21:24.179083+0000","last_became_active":"2024-03-06T07:21:23.176982+0000","last_became_peered":"2024-03-06T07:21:23.176982+0000","last_unstale":"2024-03-06T07:21:24.179083+0000","last_undegraded":"2024-03-06T07:21:24.179083+0000","last_fullsized":"2024-03-06T07:21:24.179083+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:41.252208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.282915+0000","last_change":"2024-03-06T07:21:23.172743+0000","last_active":"2024-03-06T07:21:24.282915+0000","last_peered":"2024-03-06T07:21:24.282915+0000","last_clean":"2024-03-06T07:21:24.282915+0000","last_became_active":"2024-03-06T07:21:23.172486+0000","last_became_peered":"2024-03-06T07:21:23.172486+0000","last_unstale":"2024-03-06T07:21:24.282915+0000","last_undegraded":"2024-03-06T07:21:24.282915+0000","last_fullsized":"2024-03-06T07:21:24.282915+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:36.493357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179008+0000","last_change":"2024-03-06T07:21:23.170530+0000","last_active":"2024-03-06T07:21:24.179008+0000","last_peered":"2024-03-06T07:21:24.179008+0000","last_clean":"2024-03-06T07:21:24.179008+0000","last_became_active":"2024-03-06T07:21:23.170269+0000","last_became_peered":"2024-03-06T07:21:23.170269+0000","last_unstale":"2024-03-06T07:21:24.179008+0000","last_undegraded":"2024-03-06T07:21:24.179008+0000","last_fullsized":"2024-03-06T07:21:24.179008+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:11.514224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.179717+0000","last_change":"2024-03-06T07:21:23.175412+0000","last_active":"2024-03-06T07:21:23.179717+0000","last_peered":"2024-03-06T07:21:23.179717+0000","last_clean":"2024-03-06T07:21:23.179717+0000","last_became_active":"2024-03-06T07:21:23.175113+0000","last_became_peered":"2024-03-06T07:21:23.175113+0000","last_unstale":"2024-03-06T07:21:23.179717+0000","last_undegraded":"2024-03-06T07:21:23.179717+0000","last_fullsized":"2024-03-06T07:21:23.179717+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:41.747922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.175474+0000","last_change":"2024-03-06T07:21:23.174714+0000","last_active":"2024-03-06T07:21:23.175474+0000","last_peered":"2024-03-06T07:21:23.175474+0000","last_clean":"2024-03-06T07:21:23.175474+0000","last_became_active":"2024-03-06T07:21:23.174372+0000","last_became_peered":"2024-03-06T07:21:23.174372+0000","last_unstale":"2024-03-06T07:21:23.175474+0000","last_undegraded":"2024-03-06T07:21:23.175474+0000","last_fullsized":"2024-03-06T07:21:23.175474+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:22.077065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165483+0000","last_change":"2024-03-06T07:21:23.173359+0000","last_active":"2024-03-06T07:21:24.165483+0000","last_peered":"2024-03-06T07:21:24.165483+0000","last_clean":"2024-03-06T07:21:24.165483+0000","last_became_active":"2024-03-06T07:21:23.173040+0000","last_became_peered":"2024-03-06T07:21:23.173040+0000","last_unstale":"2024-03-06T07:21:24.165483+0000","last_undegraded":"2024-03-06T07:21:24.165483+0000","last_fullsized":"2024-03-06T07:21:24.165483+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:20.162045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164458+0000","last_change":"2024-03-06T07:21:23.177492+0000","last_active":"2024-03-06T07:21:24.164458+0000","last_peered":"2024-03-06T07:21:24.164458+0000","last_clean":"2024-03-06T07:21:24.164458+0000","last_became_active":"2024-03-06T07:21:23.177109+0000","last_became_peered":"2024-03-06T07:21:23.177109+0000","last_unstale":"2024-03-06T07:21:24.164458+0000","last_undegraded":"2024-03-06T07:21:24.164458+0000","last_fullsized":"2024-03-06T07:21:24.164458+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:50.469010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165246+0000","last_change":"2024-03-06T07:21:23.173277+0000","last_active":"2024-03-06T07:21:24.165246+0000","last_peered":"2024-03-06T07:21:24.165246+0000","last_clean":"2024-03-06T07:21:24.165246+0000","last_became_active":"2024-03-06T07:21:23.172974+0000","last_became_peered":"2024-03-06T07:21:23.172974+0000","last_unstale":"2024-03-06T07:21:24.165246+0000","last_undegraded":"2024-03-06T07:21:24.165246+0000","last_fullsized":"2024-03-06T07:21:24.165246+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:08.049695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179011+0000","last_change":"2024-03-06T07:21:23.176636+0000","last_active":"2024-03-06T07:21:24.179011+0000","last_peered":"2024-03-06T07:21:24.179011+0000","last_clean":"2024-03-06T07:21:24.179011+0000","last_became_active":"2024-03-06T07:21:23.176347+0000","last_became_peered":"2024-03-06T07:21:23.176347+0000","last_unstale":"2024-03-06T07:21:24.179011+0000","last_undegraded":"2024-03-06T07:21:24.179011+0000","last_fullsized":"2024-03-06T07:21:24.179011+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:54.233662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283249+0000","last_change":"2024-03-06T07:21:23.177707+0000","last_active":"2024-03-06T07:21:24.283249+0000","last_peered":"2024-03-06T07:21:24.283249+0000","last_clean":"2024-03-06T07:21:24.283249+0000","last_became_active":"2024-03-06T07:21:23.172182+0000","last_became_peered":"2024-03-06T07:21:23.172182+0000","last_unstale":"2024-03-06T07:21:24.283249+0000","last_undegraded":"2024-03-06T07:21:24.283249+0000","last_fullsized":"2024-03-06T07:21:24.283249+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:19.113059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.163484+0000","last_change":"2024-03-06T07:21:23.159844+0000","last_active":"2024-03-06T07:21:23.163484+0000","last_peered":"2024-03-06T07:21:23.163484+0000","last_clean":"2024-03-06T07:21:23.163484+0000","last_became_active":"2024-03-06T07:21:23.159578+0000","last_became_peered":"2024-03-06T07:21:23.159578+0000","last_unstale":"2024-03-06T07:21:23.163484+0000","last_undegraded":"2024-03-06T07:21:23.163484+0000","last_fullsized":"2024-03-06T07:21:23.163484+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:00.584960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164259+0000","last_change":"2024-03-06T07:21:23.167040+0000","last_active":"2024-03-06T07:21:24.164259+0000","last_peered":"2024-03-06T07:21:24.164259+0000","last_clean":"2024-03-06T07:21:24.164259+0000","last_became_active":"2024-03-06T07:21:23.166340+0000","last_became_peered":"2024-03-06T07:21:23.166340+0000","last_unstale":"2024-03-06T07:21:24.164259+0000","last_undegraded":"2024-03-06T07:21:24.164259+0000","last_fullsized":"2024-03-06T07:21:24.164259+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:59.215536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165465+0000","last_change":"2024-03-06T07:21:23.182761+0000","last_active":"2024-03-06T07:21:24.165465+0000","last_peered":"2024-03-06T07:21:24.165465+0000","last_clean":"2024-03-06T07:21:24.165465+0000","last_became_active":"2024-03-06T07:21:23.182467+0000","last_became_peered":"2024-03-06T07:21:23.182467+0000","last_unstale":"2024-03-06T07:21:24.165465+0000","last_undegraded":"2024-03-06T07:21:24.165465+0000","last_fullsized":"2024-03-06T07:21:24.165465+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:12.094182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.182386+0000","last_change":"2024-03-06T07:21:23.179741+0000","last_active":"2024-03-06T07:21:23.182386+0000","last_peered":"2024-03-06T07:21:23.182386+0000","last_clean":"2024-03-06T07:21:23.182386+0000","last_became_active":"2024-03-06T07:21:23.179444+0000","last_became_peered":"2024-03-06T07:21:23.179444+0000","last_unstale":"2024-03-06T07:21:23.182386+0000","last_undegraded":"2024-03-06T07:21:23.182386+0000","last_fullsized":"2024-03-06T07:21:23.182386+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:10.564903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165441+0000","last_change":"2024-03-06T07:21:23.173847+0000","last_active":"2024-03-06T07:21:24.165441+0000","last_peered":"2024-03-06T07:21:24.165441+0000","last_clean":"2024-03-06T07:21:24.165441+0000","last_became_active":"2024-03-06T07:21:23.173587+0000","last_became_peered":"2024-03-06T07:21:23.173587+0000","last_unstale":"2024-03-06T07:21:24.165441+0000","last_undegraded":"2024-03-06T07:21:24.165441+0000","last_fullsized":"2024-03-06T07:21:24.165441+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:07.628952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179265+0000","last_change":"2024-03-06T07:21:23.170610+0000","last_active":"2024-03-06T07:21:24.179265+0000","last_peered":"2024-03-06T07:21:24.179265+0000","last_clean":"2024-03-06T07:21:24.179265+0000","last_became_active":"2024-03-06T07:21:23.169962+0000","last_became_peered":"2024-03-06T07:21:23.169962+0000","last_unstale":"2024-03-06T07:21:24.179265+0000","last_undegraded":"2024-03-06T07:21:24.179265+0000","last_fullsized":"2024-03-06T07:21:24.179265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:22.556995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165379+0000","last_change":"2024-03-06T07:21:23.175388+0000","last_active":"2024-03-06T07:21:24.165379+0000","last_peered":"2024-03-06T07:21:24.165379+0000","last_clean":"2024-03-06T07:21:24.165379+0000","last_became_active":"2024-03-06T07:21:23.175097+0000","last_became_peered":"2024-03-06T07:21:23.175097+0000","last_unstale":"2024-03-06T07:21:24.165379+0000","last_undegraded":"2024-03-06T07:21:24.165379+0000","last_fullsized":"2024-03-06T07:21:24.165379+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:22.372508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165127+0000","last_change":"2024-03-06T07:21:23.178513+0000","last_active":"2024-03-06T07:21:24.165127+0000","last_peered":"2024-03-06T07:21:24.165127+0000","last_clean":"2024-03-06T07:21:24.165127+0000","last_became_active":"2024-03-06T07:21:23.177896+0000","last_became_peered":"2024-03-06T07:21:23.177896+0000","last_unstale":"2024-03-06T07:21:24.165127+0000","last_undegraded":"2024-03-06T07:21:24.165127+0000","last_fullsized":"2024-03-06T07:21:24.165127+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:00.584960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.169176+0000","last_change":"2024-03-06T07:21:23.162516+0000","last_active":"2024-03-06T07:21:23.169176+0000","last_peered":"2024-03-06T07:21:23.169176+0000","last_clean":"2024-03-06T07:21:23.169176+0000","last_became_active":"2024-03-06T07:21:23.162258+0000","last_became_peered":"2024-03-06T07:21:23.162258+0000","last_unstale":"2024-03-06T07:21:23.169176+0000","last_undegraded":"2024-03-06T07:21:23.169176+0000","last_fullsized":"2024-03-06T07:21:23.169176+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:08.049695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.174382+0000","last_change":"2024-03-06T07:21:23.172221+0000","last_active":"2024-03-06T07:21:23.174382+0000","last_peered":"2024-03-06T07:21:23.174382+0000","last_clean":"2024-03-06T07:21:23.174382+0000","last_became_active":"2024-03-06T07:21:23.171940+0000","last_became_peered":"2024-03-06T07:21:23.171940+0000","last_unstale":"2024-03-06T07:21:23.174382+0000","last_undegraded":"2024-03-06T07:21:23.174382+0000","last_fullsized":"2024-03-06T07:21:23.174382+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:20.162045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.177313+0000","last_change":"2024-03-06T07:21:23.175504+0000","last_active":"2024-03-06T07:21:23.177313+0000","last_peered":"2024-03-06T07:21:23.177313+0000","last_clean":"2024-03-06T07:21:23.177313+0000","last_became_active":"2024-03-06T07:21:23.175180+0000","last_became_peered":"2024-03-06T07:21:23.175180+0000","last_unstale":"2024-03-06T07:21:23.177313+0000","last_undegraded":"2024-03-06T07:21:23.177313+0000","last_fullsized":"2024-03-06T07:21:23.177313+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:52.373147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165580+0000","last_change":"2024-03-06T07:21:22.171884+0000","last_active":"2024-03-06T07:21:24.165580+0000","last_peered":"2024-03-06T07:21:24.165580+0000","last_clean":"2024-03-06T07:21:24.165580+0000","last_became_active":"2024-03-06T07:21:22.171573+0000","last_became_peered":"2024-03-06T07:21:22.171573+0000","last_unstale":"2024-03-06T07:21:24.165580+0000","last_undegraded":"2024-03-06T07:21:24.165580+0000","last_fullsized":"2024-03-06T07:21:24.165580+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:45.006491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.183431+0000","last_change":"2024-03-06T07:21:23.181934+0000","last_active":"2024-03-06T07:21:23.183431+0000","last_peered":"2024-03-06T07:21:23.183431+0000","last_clean":"2024-03-06T07:21:23.183431+0000","last_became_active":"2024-03-06T07:21:23.181645+0000","last_became_peered":"2024-03-06T07:21:23.181645+0000","last_unstale":"2024-03-06T07:21:23.183431+0000","last_undegraded":"2024-03-06T07:21:23.183431+0000","last_fullsized":"2024-03-06T07:21:23.183431+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:22.077065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.191649+0000","last_change":"2024-03-06T07:21:22.170291+0000","last_active":"2024-03-06T07:21:23.191649+0000","last_peered":"2024-03-06T07:21:23.191649+0000","last_clean":"2024-03-06T07:21:23.191649+0000","last_became_active":"2024-03-06T07:21:22.169996+0000","last_became_peered":"2024-03-06T07:21:22.169996+0000","last_unstale":"2024-03-06T07:21:23.191649+0000","last_undegraded":"2024-03-06T07:21:23.191649+0000","last_fullsized":"2024-03-06T07:21:23.191649+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:59.580100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283289+0000","last_change":"2024-03-06T07:21:23.171182+0000","last_active":"2024-03-06T07:21:24.283289+0000","last_peered":"2024-03-06T07:21:24.283289+0000","last_clean":"2024-03-06T07:21:24.283289+0000","last_became_active":"2024-03-06T07:21:23.170864+0000","last_became_peered":"2024-03-06T07:21:23.170864+0000","last_unstale":"2024-03-06T07:21:24.283289+0000","last_undegraded":"2024-03-06T07:21:24.283289+0000","last_fullsized":"2024-03-06T07:21:24.283289+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:07.635594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.191378+0000","last_change":"2024-03-06T07:21:22.174770+0000","last_active":"2024-03-06T07:21:23.191378+0000","last_peered":"2024-03-06T07:21:23.191378+0000","last_clean":"2024-03-06T07:21:23.191378+0000","last_became_active":"2024-03-06T07:21:22.174494+0000","last_became_peered":"2024-03-06T07:21:22.174494+0000","last_unstale":"2024-03-06T07:21:23.191378+0000","last_undegraded":"2024-03-06T07:21:23.191378+0000","last_fullsized":"2024-03-06T07:21:23.191378+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:06.624093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165090+0000","last_change":"2024-03-06T07:21:23.173050+0000","last_active":"2024-03-06T07:21:24.165090+0000","last_peered":"2024-03-06T07:21:24.165090+0000","last_clean":"2024-03-06T07:21:24.165090+0000","last_became_active":"2024-03-06T07:21:23.172741+0000","last_became_peered":"2024-03-06T07:21:23.172741+0000","last_unstale":"2024-03-06T07:21:24.165090+0000","last_undegraded":"2024-03-06T07:21:24.165090+0000","last_fullsized":"2024-03-06T07:21:24.165090+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:15.661528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283240+0000","last_change":"2024-03-06T07:21:22.166297+0000","last_active":"2024-03-06T07:21:24.283240+0000","last_peered":"2024-03-06T07:21:24.283240+0000","last_clean":"2024-03-06T07:21:24.283240+0000","last_became_active":"2024-03-06T07:21:22.166020+0000","last_became_peered":"2024-03-06T07:21:22.166020+0000","last_unstale":"2024-03-06T07:21:24.283240+0000","last_undegraded":"2024-03-06T07:21:24.283240+0000","last_fullsized":"2024-03-06T07:21:24.283240+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:59.580100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.184409+0000","last_change":"2024-03-06T07:21:23.180253+0000","last_active":"2024-03-06T07:21:23.184409+0000","last_peered":"2024-03-06T07:21:23.184409+0000","last_clean":"2024-03-06T07:21:23.184409+0000","last_became_active":"2024-03-06T07:21:23.174841+0000","last_became_peered":"2024-03-06T07:21:23.174841+0000","last_unstale":"2024-03-06T07:21:23.184409+0000","last_undegraded":"2024-03-06T07:21:23.184409+0000","last_fullsized":"2024-03-06T07:21:23.184409+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:20.162045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179164+0000","last_change":"2024-03-06T07:21:22.170216+0000","last_active":"2024-03-06T07:21:24.179164+0000","last_peered":"2024-03-06T07:21:24.179164+0000","last_clean":"2024-03-06T07:21:24.179164+0000","last_became_active":"2024-03-06T07:21:22.169909+0000","last_became_peered":"2024-03-06T07:21:22.169909+0000","last_unstale":"2024-03-06T07:21:24.179164+0000","last_undegraded":"2024-03-06T07:21:24.179164+0000","last_fullsized":"2024-03-06T07:21:24.179164+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:07.044836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179273+0000","last_change":"2024-03-06T07:21:23.176300+0000","last_active":"2024-03-06T07:21:24.179273+0000","last_peered":"2024-03-06T07:21:24.179273+0000","last_clean":"2024-03-06T07:21:24.179273+0000","last_became_active":"2024-03-06T07:21:23.175986+0000","last_became_peered":"2024-03-06T07:21:23.175986+0000","last_unstale":"2024-03-06T07:21:24.179273+0000","last_undegraded":"2024-03-06T07:21:24.179273+0000","last_fullsized":"2024-03-06T07:21:24.179273+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:53.558264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.186822+0000","last_change":"2024-03-06T07:21:22.174498+0000","last_active":"2024-03-06T07:21:23.186822+0000","last_peered":"2024-03-06T07:21:23.186822+0000","last_clean":"2024-03-06T07:21:23.186822+0000","last_became_active":"2024-03-06T07:21:22.174189+0000","last_became_peered":"2024-03-06T07:21:22.174189+0000","last_unstale":"2024-03-06T07:21:23.186822+0000","last_undegraded":"2024-03-06T07:21:23.186822+0000","last_fullsized":"2024-03-06T07:21:23.186822+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:40.673099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283270+0000","last_change":"2024-03-06T07:21:23.180855+0000","last_active":"2024-03-06T07:21:24.283270+0000","last_peered":"2024-03-06T07:21:24.283270+0000","last_clean":"2024-03-06T07:21:24.283270+0000","last_became_active":"2024-03-06T07:21:23.180588+0000","last_became_peered":"2024-03-06T07:21:23.180588+0000","last_unstale":"2024-03-06T07:21:24.283270+0000","last_undegraded":"2024-03-06T07:21:24.283270+0000","last_fullsized":"2024-03-06T07:21:24.283270+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:06.967643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.166457+0000","last_change":"2024-03-06T07:21:22.172223+0000","last_active":"2024-03-06T07:21:23.166457+0000","last_peered":"2024-03-06T07:21:23.166457+0000","last_clean":"2024-03-06T07:21:23.166457+0000","last_became_active":"2024-03-06T07:21:22.171912+0000","last_became_peered":"2024-03-06T07:21:22.171912+0000","last_unstale":"2024-03-06T07:21:23.166457+0000","last_undegraded":"2024-03-06T07:21:23.166457+0000","last_fullsized":"2024-03-06T07:21:23.166457+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:59.580100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164902+0000","last_change":"2024-03-06T07:21:23.174371+0000","last_active":"2024-03-06T07:21:24.164902+0000","last_peered":"2024-03-06T07:21:24.164902+0000","last_clean":"2024-03-06T07:21:24.164902+0000","last_became_active":"2024-03-06T07:21:23.173893+0000","last_became_peered":"2024-03-06T07:21:23.173893+0000","last_unstale":"2024-03-06T07:21:24.164902+0000","last_undegraded":"2024-03-06T07:21:24.164902+0000","last_fullsized":"2024-03-06T07:21:24.164902+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:37.222226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.282797+0000","last_change":"2024-03-06T07:21:22.174586+0000","last_active":"2024-03-06T07:21:24.282797+0000","last_peered":"2024-03-06T07:21:24.282797+0000","last_clean":"2024-03-06T07:21:24.282797+0000","last_became_active":"2024-03-06T07:21:22.174294+0000","last_became_peered":"2024-03-06T07:21:22.174294+0000","last_unstale":"2024-03-06T07:21:24.282797+0000","last_undegraded":"2024-03-06T07:21:24.282797+0000","last_fullsized":"2024-03-06T07:21:24.282797+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:40.673099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164248+0000","last_change":"2024-03-06T07:21:23.175308+0000","last_active":"2024-03-06T07:21:24.164248+0000","last_peered":"2024-03-06T07:21:24.164248+0000","last_clean":"2024-03-06T07:21:24.164248+0000","last_became_active":"2024-03-06T07:21:23.175045+0000","last_became_peered":"2024-03-06T07:21:23.175045+0000","last_unstale":"2024-03-06T07:21:24.164248+0000","last_undegraded":"2024-03-06T07:21:24.164248+0000","last_fullsized":"2024-03-06T07:21:24.164248+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:28.084724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165189+0000","last_change":"2024-03-06T07:21:22.173698+0000","last_active":"2024-03-06T07:21:24.165189+0000","last_peered":"2024-03-06T07:21:24.165189+0000","last_clean":"2024-03-06T07:21:24.165189+0000","last_became_active":"2024-03-06T07:21:22.173426+0000","last_became_peered":"2024-03-06T07:21:22.173426+0000","last_unstale":"2024-03-06T07:21:24.165189+0000","last_undegraded":"2024-03-06T07:21:24.165189+0000","last_fullsized":"2024-03-06T07:21:24.165189+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:16.351884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.174766+0000","last_change":"2024-03-06T07:21:23.171603+0000","last_active":"2024-03-06T07:21:23.174766+0000","last_peered":"2024-03-06T07:21:23.174766+0000","last_clean":"2024-03-06T07:21:23.174766+0000","last_became_active":"2024-03-06T07:21:23.171109+0000","last_became_peered":"2024-03-06T07:21:23.171109+0000","last_unstale":"2024-03-06T07:21:23.174766+0000","last_undegraded":"2024-03-06T07:21:23.174766+0000","last_fullsized":"2024-03-06T07:21:23.174766+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:10.564903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179439+0000","last_change":"2024-03-06T07:21:22.166081+0000","last_active":"2024-03-06T07:21:24.179439+0000","last_peered":"2024-03-06T07:21:24.179439+0000","last_clean":"2024-03-06T07:21:24.179439+0000","last_became_active":"2024-03-06T07:21:22.165820+0000","last_became_peered":"2024-03-06T07:21:22.165820+0000","last_unstale":"2024-03-06T07:21:24.179439+0000","last_undegraded":"2024-03-06T07:21:24.179439+0000","last_fullsized":"2024-03-06T07:21:24.179439+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:59.580100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.185507+0000","last_change":"2024-03-06T07:21:23.180325+0000","last_active":"2024-03-06T07:21:23.185507+0000","last_peered":"2024-03-06T07:21:23.185507+0000","last_clean":"2024-03-06T07:21:23.185507+0000","last_became_active":"2024-03-06T07:21:23.175546+0000","last_became_peered":"2024-03-06T07:21:23.175546+0000","last_unstale":"2024-03-06T07:21:23.185507+0000","last_undegraded":"2024-03-06T07:21:23.185507+0000","last_fullsized":"2024-03-06T07:21:23.185507+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:10.564903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179551+0000","last_change":"2024-03-06T07:21:22.173265+0000","last_active":"2024-03-06T07:21:24.179551+0000","last_peered":"2024-03-06T07:21:24.179551+0000","last_clean":"2024-03-06T07:21:24.179551+0000","last_became_active":"2024-03-06T07:21:22.172945+0000","last_became_peered":"2024-03-06T07:21:22.172945+0000","last_unstale":"2024-03-06T07:21:24.179551+0000","last_undegraded":"2024-03-06T07:21:24.179551+0000","last_fullsized":"2024-03-06T07:21:24.179551+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:21.367648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283250+0000","last_change":"2024-03-06T07:21:23.173659+0000","last_active":"2024-03-06T07:21:24.283250+0000","last_peered":"2024-03-06T07:21:24.283250+0000","last_clean":"2024-03-06T07:21:24.283250+0000","last_became_active":"2024-03-06T07:21:23.173401+0000","last_became_peered":"2024-03-06T07:21:23.173401+0000","last_unstale":"2024-03-06T07:21:24.283250+0000","last_undegraded":"2024-03-06T07:21:24.283250+0000","last_fullsized":"2024-03-06T07:21:24.283250+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:17.356744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164883+0000","last_change":"2024-03-06T07:21:22.170375+0000","last_active":"2024-03-06T07:21:24.164883+0000","last_peered":"2024-03-06T07:21:24.164883+0000","last_clean":"2024-03-06T07:21:24.164883+0000","last_became_active":"2024-03-06T07:21:22.170096+0000","last_became_peered":"2024-03-06T07:21:22.170096+0000","last_unstale":"2024-03-06T07:21:24.164883+0000","last_undegraded":"2024-03-06T07:21:24.164883+0000","last_fullsized":"2024-03-06T07:21:24.164883+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:35.488498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.184430+0000","last_change":"2024-03-06T07:21:23.180941+0000","last_active":"2024-03-06T07:21:23.184430+0000","last_peered":"2024-03-06T07:21:23.184430+0000","last_clean":"2024-03-06T07:21:23.184430+0000","last_became_active":"2024-03-06T07:21:23.180662+0000","last_became_peered":"2024-03-06T07:21:23.180662+0000","last_unstale":"2024-03-06T07:21:23.184430+0000","last_undegraded":"2024-03-06T07:21:23.184430+0000","last_fullsized":"2024-03-06T07:21:23.184430+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:41.747922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165167+0000","last_change":"2024-03-06T07:21:22.168669+0000","last_active":"2024-03-06T07:21:24.165167+0000","last_peered":"2024-03-06T07:21:24.165167+0000","last_clean":"2024-03-06T07:21:24.165167+0000","last_became_active":"2024-03-06T07:21:22.168382+0000","last_became_peered":"2024-03-06T07:21:22.168382+0000","last_unstale":"2024-03-06T07:21:24.165167+0000","last_undegraded":"2024-03-06T07:21:24.165167+0000","last_fullsized":"2024-03-06T07:21:24.165167+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:34.691115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165029+0000","last_change":"2024-03-06T07:21:23.170835+0000","last_active":"2024-03-06T07:21:24.165029+0000","last_peered":"2024-03-06T07:21:24.165029+0000","last_clean":"2024-03-06T07:21:24.165029+0000","last_became_active":"2024-03-06T07:21:23.170538+0000","last_became_peered":"2024-03-06T07:21:23.170538+0000","last_unstale":"2024-03-06T07:21:24.165029+0000","last_undegraded":"2024-03-06T07:21:24.165029+0000","last_fullsized":"2024-03-06T07:21:24.165029+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:08.450197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.166546+0000","last_change":"2024-03-06T07:21:22.173295+0000","last_active":"2024-03-06T07:21:23.166546+0000","last_peered":"2024-03-06T07:21:23.166546+0000","last_clean":"2024-03-06T07:21:23.166546+0000","last_became_active":"2024-03-06T07:21:22.172903+0000","last_became_peered":"2024-03-06T07:21:22.172903+0000","last_unstale":"2024-03-06T07:21:23.166546+0000","last_undegraded":"2024-03-06T07:21:23.166546+0000","last_fullsized":"2024-03-06T07:21:23.166546+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:07.044836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.172290+0000","last_change":"2024-03-06T07:21:23.163139+0000","last_active":"2024-03-06T07:21:23.172290+0000","last_peered":"2024-03-06T07:21:23.172290+0000","last_clean":"2024-03-06T07:21:23.172290+0000","last_became_active":"2024-03-06T07:21:23.162884+0000","last_became_peered":"2024-03-06T07:21:23.162884+0000","last_unstale":"2024-03-06T07:21:23.172290+0000","last_undegraded":"2024-03-06T07:21:23.172290+0000","last_fullsized":"2024-03-06T07:21:23.172290+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:22.372508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164924+0000","last_change":"2024-03-06T07:21:22.171888+0000","last_active":"2024-03-06T07:21:24.164924+0000","last_peered":"2024-03-06T07:21:24.164924+0000","last_clean":"2024-03-06T07:21:24.164924+0000","last_became_active":"2024-03-06T07:21:22.171588+0000","last_became_peered":"2024-03-06T07:21:22.171588+0000","last_unstale":"2024-03-06T07:21:24.164924+0000","last_undegraded":"2024-03-06T07:21:24.164924+0000","last_fullsized":"2024-03-06T07:21:24.164924+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:33.446854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165200+0000","last_change":"2024-03-06T07:21:23.168200+0000","last_active":"2024-03-06T07:21:24.165200+0000","last_peered":"2024-03-06T07:21:24.165200+0000","last_clean":"2024-03-06T07:21:24.165200+0000","last_became_active":"2024-03-06T07:21:23.167903+0000","last_became_peered":"2024-03-06T07:21:23.167903+0000","last_unstale":"2024-03-06T07:21:24.165200+0000","last_undegraded":"2024-03-06T07:21:24.165200+0000","last_fullsized":"2024-03-06T07:21:24.165200+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:01.144803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.188870+0000","last_change":"2024-03-06T07:21:22.175264+0000","last_active":"2024-03-06T07:21:23.188870+0000","last_peered":"2024-03-06T07:21:23.188870+0000","last_clean":"2024-03-06T07:21:23.188870+0000","last_became_active":"2024-03-06T07:21:22.174941+0000","last_became_peered":"2024-03-06T07:21:22.174941+0000","last_unstale":"2024-03-06T07:21:23.188870+0000","last_undegraded":"2024-03-06T07:21:23.188870+0000","last_fullsized":"2024-03-06T07:21:23.188870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:51.086001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164775+0000","last_change":"2024-03-06T07:21:23.171904+0000","last_active":"2024-03-06T07:21:24.164775+0000","last_peered":"2024-03-06T07:21:24.164775+0000","last_clean":"2024-03-06T07:21:24.164775+0000","last_became_active":"2024-03-06T07:21:23.171618+0000","last_became_peered":"2024-03-06T07:21:23.171618+0000","last_unstale":"2024-03-06T07:21:24.164775+0000","last_undegraded":"2024-03-06T07:21:24.164775+0000","last_fullsized":"2024-03-06T07:21:24.164775+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:48.917896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179498+0000","last_change":"2024-03-06T07:21:22.174985+0000","last_active":"2024-03-06T07:21:24.179498+0000","last_peered":"2024-03-06T07:21:24.179498+0000","last_clean":"2024-03-06T07:21:24.179498+0000","last_became_active":"2024-03-06T07:21:22.174284+0000","last_became_peered":"2024-03-06T07:21:22.174284+0000","last_unstale":"2024-03-06T07:21:24.179498+0000","last_undegraded":"2024-03-06T07:21:24.179498+0000","last_fullsized":"2024-03-06T07:21:24.179498+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:11.089323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179490+0000","last_change":"2024-03-06T07:21:23.179740+0000","last_active":"2024-03-06T07:21:24.179490+0000","last_peered":"2024-03-06T07:21:24.179490+0000","last_clean":"2024-03-06T07:21:24.179490+0000","last_became_active":"2024-03-06T07:21:23.177114+0000","last_became_peered":"2024-03-06T07:21:23.177114+0000","last_unstale":"2024-03-06T07:21:24.179490+0000","last_undegraded":"2024-03-06T07:21:24.179490+0000","last_fullsized":"2024-03-06T07:21:24.179490+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:36.368758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283361+0000","last_change":"2024-03-06T07:21:22.171126+0000","last_active":"2024-03-06T07:21:24.283361+0000","last_peered":"2024-03-06T07:21:24.283361+0000","last_clean":"2024-03-06T07:21:24.283361+0000","last_became_active":"2024-03-06T07:21:22.170803+0000","last_became_peered":"2024-03-06T07:21:22.170803+0000","last_unstale":"2024-03-06T07:21:24.283361+0000","last_undegraded":"2024-03-06T07:21:24.283361+0000","last_fullsized":"2024-03-06T07:21:24.283361+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:07.044836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283308+0000","last_change":"2024-03-06T07:21:23.174451+0000","last_active":"2024-03-06T07:21:24.283308+0000","last_peered":"2024-03-06T07:21:24.283308+0000","last_clean":"2024-03-06T07:21:24.283308+0000","last_became_active":"2024-03-06T07:21:23.174072+0000","last_became_peered":"2024-03-06T07:21:23.174072+0000","last_unstale":"2024-03-06T07:21:24.283308+0000","last_undegraded":"2024-03-06T07:21:24.283308+0000","last_fullsized":"2024-03-06T07:21:24.283308+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:05.754195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164461+0000","last_change":"2024-03-06T07:21:22.169060+0000","last_active":"2024-03-06T07:21:24.164461+0000","last_peered":"2024-03-06T07:21:24.164461+0000","last_clean":"2024-03-06T07:21:24.164461+0000","last_became_active":"2024-03-06T07:21:22.168716+0000","last_became_peered":"2024-03-06T07:21:22.168716+0000","last_unstale":"2024-03-06T07:21:24.164461+0000","last_undegraded":"2024-03-06T07:21:24.164461+0000","last_fullsized":"2024-03-06T07:21:24.164461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:15.088870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164509+0000","last_change":"2024-03-06T07:21:23.169921+0000","last_active":"2024-03-06T07:21:24.164509+0000","last_peered":"2024-03-06T07:21:24.164509+0000","last_clean":"2024-03-06T07:21:24.164509+0000","last_became_active":"2024-03-06T07:21:23.169610+0000","last_became_peered":"2024-03-06T07:21:23.169610+0000","last_unstale":"2024-03-06T07:21:24.164509+0000","last_undegraded":"2024-03-06T07:21:24.164509+0000","last_fullsized":"2024-03-06T07:21:24.164509+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:34.451714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.166561+0000","last_change":"2024-03-06T07:21:22.174120+0000","last_active":"2024-03-06T07:21:23.166561+0000","last_peered":"2024-03-06T07:21:23.166561+0000","last_clean":"2024-03-06T07:21:23.166561+0000","last_became_active":"2024-03-06T07:21:22.173826+0000","last_became_peered":"2024-03-06T07:21:22.173826+0000","last_unstale":"2024-03-06T07:21:23.166561+0000","last_undegraded":"2024-03-06T07:21:23.166561+0000","last_fullsized":"2024-03-06T07:21:23.166561+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:21.367648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.175577+0000","last_change":"2024-03-06T07:21:23.171548+0000","last_active":"2024-03-06T07:21:23.175577+0000","last_peered":"2024-03-06T07:21:23.175577+0000","last_clean":"2024-03-06T07:21:23.175577+0000","last_became_active":"2024-03-06T07:21:23.171272+0000","last_became_peered":"2024-03-06T07:21:23.171272+0000","last_unstale":"2024-03-06T07:21:23.175577+0000","last_undegraded":"2024-03-06T07:21:23.175577+0000","last_fullsized":"2024-03-06T07:21:23.175577+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:20.162045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164151+0000","last_change":"2024-03-06T07:21:22.165324+0000","last_active":"2024-03-06T07:21:24.164151+0000","last_peered":"2024-03-06T07:21:24.164151+0000","last_clean":"2024-03-06T07:21:24.164151+0000","last_became_active":"2024-03-06T07:21:22.165064+0000","last_became_peered":"2024-03-06T07:21:22.165064+0000","last_unstale":"2024-03-06T07:21:24.164151+0000","last_undegraded":"2024-03-06T07:21:24.164151+0000","last_fullsized":"2024-03-06T07:21:24.164151+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:42.726655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.184787+0000","last_change":"2024-03-06T07:21:23.180694+0000","last_active":"2024-03-06T07:21:23.184787+0000","last_peered":"2024-03-06T07:21:23.184787+0000","last_clean":"2024-03-06T07:21:23.184787+0000","last_became_active":"2024-03-06T07:21:23.180413+0000","last_became_peered":"2024-03-06T07:21:23.180413+0000","last_unstale":"2024-03-06T07:21:23.184787+0000","last_undegraded":"2024-03-06T07:21:23.184787+0000","last_fullsized":"2024-03-06T07:21:23.184787+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:22.077065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164033+0000","last_change":"2024-03-06T07:21:22.174287+0000","last_active":"2024-03-06T07:21:24.164033+0000","last_peered":"2024-03-06T07:21:24.164033+0000","last_clean":"2024-03-06T07:21:24.164033+0000","last_became_active":"2024-03-06T07:21:22.173947+0000","last_became_peered":"2024-03-06T07:21:22.173947+0000","last_unstale":"2024-03-06T07:21:24.164033+0000","last_undegraded":"2024-03-06T07:21:24.164033+0000","last_fullsized":"2024-03-06T07:21:24.164033+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:02.949041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179184+0000","last_change":"2024-03-06T07:21:23.173219+0000","last_active":"2024-03-06T07:21:24.179184+0000","last_peered":"2024-03-06T07:21:24.179184+0000","last_clean":"2024-03-06T07:21:24.179184+0000","last_became_active":"2024-03-06T07:21:23.170375+0000","last_became_peered":"2024-03-06T07:21:23.170375+0000","last_unstale":"2024-03-06T07:21:24.179184+0000","last_undegraded":"2024-03-06T07:21:24.179184+0000","last_fullsized":"2024-03-06T07:21:24.179184+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:48.584102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.157734+0000","last_change":"2024-03-06T07:21:22.174188+0000","last_active":"2024-03-06T07:21:23.157734+0000","last_peered":"2024-03-06T07:21:23.157734+0000","last_clean":"2024-03-06T07:21:23.157734+0000","last_became_active":"2024-03-06T07:21:22.173899+0000","last_became_peered":"2024-03-06T07:21:22.173899+0000","last_unstale":"2024-03-06T07:21:23.157734+0000","last_undegraded":"2024-03-06T07:21:23.157734+0000","last_fullsized":"2024-03-06T07:21:23.157734+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:26.708237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165047+0000","last_change":"2024-03-06T07:21:23.169931+0000","last_active":"2024-03-06T07:21:24.165047+0000","last_peered":"2024-03-06T07:21:24.165047+0000","last_clean":"2024-03-06T07:21:24.165047+0000","last_became_active":"2024-03-06T07:21:23.169572+0000","last_became_peered":"2024-03-06T07:21:23.169572+0000","last_unstale":"2024-03-06T07:21:24.165047+0000","last_undegraded":"2024-03-06T07:21:24.165047+0000","last_fullsized":"2024-03-06T07:21:24.165047+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:06.313962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.163980+0000","last_change":"2024-03-06T07:21:22.164640+0000","last_active":"2024-03-06T07:21:24.163980+0000","last_peered":"2024-03-06T07:21:24.163980+0000","last_clean":"2024-03-06T07:21:24.163980+0000","last_became_active":"2024-03-06T07:21:22.164106+0000","last_became_peered":"2024-03-06T07:21:22.164106+0000","last_unstale":"2024-03-06T07:21:24.163980+0000","last_undegraded":"2024-03-06T07:21:24.163980+0000","last_fullsized":"2024-03-06T07:21:24.163980+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:31.123680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.184450+0000","last_change":"2024-03-06T07:21:23.182404+0000","last_active":"2024-03-06T07:21:23.184450+0000","last_peered":"2024-03-06T07:21:23.184450+0000","last_clean":"2024-03-06T07:21:23.184450+0000","last_became_active":"2024-03-06T07:21:23.181761+0000","last_became_peered":"2024-03-06T07:21:23.181761+0000","last_unstale":"2024-03-06T07:21:23.184450+0000","last_undegraded":"2024-03-06T07:21:23.184450+0000","last_fullsized":"2024-03-06T07:21:23.184450+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:52.373147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179579+0000","last_change":"2024-03-06T07:21:22.175003+0000","last_active":"2024-03-06T07:21:24.179579+0000","last_peered":"2024-03-06T07:21:24.179579+0000","last_clean":"2024-03-06T07:21:24.179579+0000","last_became_active":"2024-03-06T07:21:22.172966+0000","last_became_peered":"2024-03-06T07:21:22.172966+0000","last_unstale":"2024-03-06T07:21:24.179579+0000","last_undegraded":"2024-03-06T07:21:24.179579+0000","last_fullsized":"2024-03-06T07:21:24.179579+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:06.624093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.174745+0000","last_change":"2024-03-06T07:21:23.171386+0000","last_active":"2024-03-06T07:21:23.174745+0000","last_peered":"2024-03-06T07:21:23.174745+0000","last_clean":"2024-03-06T07:21:23.174745+0000","last_became_active":"2024-03-06T07:21:23.170760+0000","last_became_peered":"2024-03-06T07:21:23.170760+0000","last_unstale":"2024-03-06T07:21:23.174745+0000","last_undegraded":"2024-03-06T07:21:23.174745+0000","last_fullsized":"2024-03-06T07:21:23.174745+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:12.094182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.178209+0000","last_change":"2024-03-06T07:21:22.161701+0000","last_active":"2024-03-06T07:21:24.178209+0000","last_peered":"2024-03-06T07:21:24.178209+0000","last_clean":"2024-03-06T07:21:24.178209+0000","last_became_active":"2024-03-06T07:21:22.161365+0000","last_became_peered":"2024-03-06T07:21:22.161365+0000","last_unstale":"2024-03-06T07:21:24.178209+0000","last_undegraded":"2024-03-06T07:21:24.178209+0000","last_fullsized":"2024-03-06T07:21:24.178209+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:35.557090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164786+0000","last_change":"2024-03-06T07:21:23.173494+0000","last_active":"2024-03-06T07:21:24.164786+0000","last_peered":"2024-03-06T07:21:24.164786+0000","last_clean":"2024-03-06T07:21:24.164786+0000","last_became_active":"2024-03-06T07:21:23.173209+0000","last_became_peered":"2024-03-06T07:21:23.173209+0000","last_unstale":"2024-03-06T07:21:24.164786+0000","last_undegraded":"2024-03-06T07:21:24.164786+0000","last_fullsized":"2024-03-06T07:21:24.164786+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:36.368758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.160431+0000","last_change":"2024-03-06T07:21:22.171248+0000","last_active":"2024-03-06T07:21:23.160431+0000","last_peered":"2024-03-06T07:21:23.160431+0000","last_clean":"2024-03-06T07:21:23.160431+0000","last_became_active":"2024-03-06T07:21:22.170909+0000","last_became_peered":"2024-03-06T07:21:22.170909+0000","last_unstale":"2024-03-06T07:21:23.160431+0000","last_undegraded":"2024-03-06T07:21:23.160431+0000","last_fullsized":"2024-03-06T07:21:23.160431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:40.673099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.169503+0000","last_change":"2024-03-06T07:21:23.168323+0000","last_active":"2024-03-06T07:21:23.169503+0000","last_peered":"2024-03-06T07:21:23.169503+0000","last_clean":"2024-03-06T07:21:23.169503+0000","last_became_active":"2024-03-06T07:21:23.168024+0000","last_became_peered":"2024-03-06T07:21:23.168024+0000","last_unstale":"2024-03-06T07:21:23.169503+0000","last_undegraded":"2024-03-06T07:21:23.169503+0000","last_fullsized":"2024-03-06T07:21:23.169503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:52.090861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.159776+0000","last_change":"2024-03-06T07:21:22.174503+0000","last_active":"2024-03-06T07:21:23.159776+0000","last_peered":"2024-03-06T07:21:23.159776+0000","last_clean":"2024-03-06T07:21:23.159776+0000","last_became_active":"2024-03-06T07:21:22.174138+0000","last_became_peered":"2024-03-06T07:21:22.174138+0000","last_unstale":"2024-03-06T07:21:23.159776+0000","last_undegraded":"2024-03-06T07:21:23.159776+0000","last_fullsized":"2024-03-06T07:21:23.159776+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:26.708237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.173809+0000","last_change":"2024-03-06T07:21:23.170871+0000","last_active":"2024-03-06T07:21:23.173809+0000","last_peered":"2024-03-06T07:21:23.173809+0000","last_clean":"2024-03-06T07:21:23.173809+0000","last_became_active":"2024-03-06T07:21:23.170384+0000","last_became_peered":"2024-03-06T07:21:23.170384+0000","last_unstale":"2024-03-06T07:21:23.173809+0000","last_undegraded":"2024-03-06T07:21:23.173809+0000","last_fullsized":"2024-03-06T07:21:23.173809+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:12.094182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165057+0000","last_change":"2024-03-06T07:21:22.169536+0000","last_active":"2024-03-06T07:21:24.165057+0000","last_peered":"2024-03-06T07:21:24.165057+0000","last_clean":"2024-03-06T07:21:24.165057+0000","last_became_active":"2024-03-06T07:21:22.169265+0000","last_became_peered":"2024-03-06T07:21:22.169265+0000","last_unstale":"2024-03-06T07:21:24.165057+0000","last_undegraded":"2024-03-06T07:21:24.165057+0000","last_fullsized":"2024-03-06T07:21:24.165057+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:49.464151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.282950+0000","last_change":"2024-03-06T07:21:23.170620+0000","last_active":"2024-03-06T07:21:24.282950+0000","last_peered":"2024-03-06T07:21:24.282950+0000","last_clean":"2024-03-06T07:21:24.282950+0000","last_became_active":"2024-03-06T07:21:23.170063+0000","last_became_peered":"2024-03-06T07:21:23.170063+0000","last_unstale":"2024-03-06T07:21:24.282950+0000","last_undegraded":"2024-03-06T07:21:24.282950+0000","last_fullsized":"2024-03-06T07:21:24.282950+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:35.695974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.282861+0000","last_change":"2024-03-06T07:21:22.172986+0000","last_active":"2024-03-06T07:21:24.282861+0000","last_peered":"2024-03-06T07:21:24.282861+0000","last_clean":"2024-03-06T07:21:24.282861+0000","last_became_active":"2024-03-06T07:21:22.172582+0000","last_became_peered":"2024-03-06T07:21:22.172582+0000","last_unstale":"2024-03-06T07:21:24.282861+0000","last_undegraded":"2024-03-06T07:21:24.282861+0000","last_fullsized":"2024-03-06T07:21:24.282861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:51.086001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.174929+0000","last_change":"2024-03-06T07:21:23.170353+0000","last_active":"2024-03-06T07:21:23.174929+0000","last_peered":"2024-03-06T07:21:23.174929+0000","last_clean":"2024-03-06T07:21:23.174929+0000","last_became_active":"2024-03-06T07:21:23.170078+0000","last_became_peered":"2024-03-06T07:21:23.170078+0000","last_unstale":"2024-03-06T07:21:23.174929+0000","last_undegraded":"2024-03-06T07:21:23.174929+0000","last_fullsized":"2024-03-06T07:21:23.174929+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:07.628952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164817+0000","last_change":"2024-03-06T07:21:22.173498+0000","last_active":"2024-03-06T07:21:24.164817+0000","last_peered":"2024-03-06T07:21:24.164817+0000","last_clean":"2024-03-06T07:21:24.164817+0000","last_became_active":"2024-03-06T07:21:22.173198+0000","last_became_peered":"2024-03-06T07:21:22.173198+0000","last_unstale":"2024-03-06T07:21:24.164817+0000","last_undegraded":"2024-03-06T07:21:24.164817+0000","last_fullsized":"2024-03-06T07:21:24.164817+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:18.108199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179450+0000","last_change":"2024-03-06T07:21:23.178114+0000","last_active":"2024-03-06T07:21:24.179450+0000","last_peered":"2024-03-06T07:21:24.179450+0000","last_clean":"2024-03-06T07:21:24.179450+0000","last_became_active":"2024-03-06T07:21:23.177700+0000","last_became_peered":"2024-03-06T07:21:23.177700+0000","last_unstale":"2024-03-06T07:21:24.179450+0000","last_undegraded":"2024-03-06T07:21:24.179450+0000","last_fullsized":"2024-03-06T07:21:24.179450+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:48.917896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.157712+0000","last_change":"2024-03-06T07:21:22.172115+0000","last_active":"2024-03-06T07:21:23.157712+0000","last_peered":"2024-03-06T07:21:23.157712+0000","last_clean":"2024-03-06T07:21:23.157712+0000","last_became_active":"2024-03-06T07:21:22.171844+0000","last_became_peered":"2024-03-06T07:21:22.171844+0000","last_unstale":"2024-03-06T07:21:23.157712+0000","last_undegraded":"2024-03-06T07:21:23.157712+0000","last_fullsized":"2024-03-06T07:21:23.157712+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:45.059780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":203,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-06T07:21:23.158976+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:21:23.158976+0000","last_peered":"2024-03-06T07:21:23.158976+0000","last_clean":"2024-03-06T07:21:23.158976+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:21:23.158976+0000","last_undegraded":"2024-03-06T07:21:23.158976+0000","last_fullsized":"2024-03-06T07:21:23.158976+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":520192,"data_stored":782523,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":197747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574864,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":446464,"data_stored":190585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":446464,"data_stored":191077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31644,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340196,"statfs":{"total":96636764160,"available":96604360704,"internally_reserved":0,"allocated":466944,"data_stored":198811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":442368,"data_stored":190543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31616,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340224,"statfs":{"total":96636764160,"available":96604389376,"internally_reserved":0,"allocated":442368,"data_stored":190563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":507904,"data_stored":780875,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:30.636 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:30.709 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:30.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:30.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.737+0000 7f2070078640 1 Processor -- start 2024-03-06T07:21:30.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.737+0000 7f2070078640 1 -- start start 2024-03-06T07:21:30.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.738+0000 7f2070078640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f20680a9760 0x7f20680a9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.738+0000 7f2070078640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2068152b00 con 0x7f206806f6d0 2024-03-06T07:21:30.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.738+0000 7f2070078640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206806fdb0 con 0x7f20680c14f0 2024-03-06T07:21:30.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.738+0000 7f2070078640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20681551f0 con 0x7f20680a9760 2024-03-06T07:21:30.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.738+0000 7f206de14640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f20680a9760 0x7f20680a9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.738+0000 7f206e615640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f206806f6d0 0x7f206815fa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60792/0 (socket says 172.21.15.1:60792) 2024-03-06T07:21:30.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.738+0000 7f206e615640 1 -- 172.21.15.1:0/480324157 learned_addr learned my addr 172.21.15.1:0/480324157 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:30.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.738+0000 7f206de14640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f20680a9760 0x7f20680a9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37078/0 (socket says 172.21.15.1:37078) 2024-03-06T07:21:30.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.738+0000 7f206de14640 1 -- 172.21.15.1:0/480324157 >> v1:172.21.15.98:6789/0 conn(0x7f20680c14f0 legacy=0x7f2068155680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.738+0000 7f206de14640 1 -- 172.21.15.1:0/480324157 >> v1:172.21.15.1:6789/0 conn(0x7f206806f6d0 legacy=0x7f206815fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.738+0000 7f206de14640 1 -- 172.21.15.1:0/480324157 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20680c0720 con 0x7f20680a9760 2024-03-06T07:21:30.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.738+0000 7f206ce12640 1 -- 172.21.15.1:0/480324157 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214642120 0 0) 0x7f2068152b00 con 0x7f206806f6d0 2024-03-06T07:21:30.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.739+0000 7f206de14640 1 --2- 172.21.15.1:0/480324157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20680a9760 0x7f20680a9b40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f20580097a0 tx=0x7f205802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e9c565fb96633ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.739+0000 7f206ce12640 1 -- 172.21.15.1:0/480324157 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20580315a0 con 0x7f20680a9760 2024-03-06T07:21:30.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.739+0000 7f206ce12640 1 -- 172.21.15.1:0/480324157 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2058037070 con 0x7f20680a9760 2024-03-06T07:21:30.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.739+0000 7f206ce12640 1 -- 172.21.15.1:0/480324157 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2058031960 con 0x7f20680a9760 2024-03-06T07:21:30.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.740+0000 7f2070078640 1 -- 172.21.15.1:0/480324157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20680a9760 msgr2=0x7f20680a9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.740+0000 7f2070078640 1 --2- 172.21.15.1:0/480324157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20680a9760 0x7f20680a9b40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f20580097a0 tx=0x7f205802b840 comp rx=0 tx=0).stop 2024-03-06T07:21:30.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.740+0000 7f2070078640 1 -- 172.21.15.1:0/480324157 shutdown_connections 2024-03-06T07:21:30.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.740+0000 7f2070078640 1 --2- 172.21.15.1:0/480324157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20680a9760 0x7f20680a9b40 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.740+0000 7f2070078640 1 -- 172.21.15.1:0/480324157 >> 172.21.15.1:0/480324157 conn(0x7f20680c3cc0 msgr2=0x7f20680c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.740+0000 7f2070078640 1 -- 172.21.15.1:0/480324157 shutdown_connections 2024-03-06T07:21:30.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.740+0000 7f2070078640 1 -- 172.21.15.1:0/480324157 wait complete. 2024-03-06T07:21:30.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.741+0000 7f2070078640 1 Processor -- start 2024-03-06T07:21:30.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.741+0000 7f2070078640 1 -- start start 2024-03-06T07:21:30.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f2070078640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f206806f6d0 0x7f2068154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f2070078640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20680a9760 0x7f206814dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f2070078640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20680c14f0 0x7f206814d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f2070078640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20680c29e0 con 0x7f20680a9760 2024-03-06T07:21:30.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f2070078640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20680a9520 con 0x7f206806f6d0 2024-03-06T07:21:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f2070078640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20680c0ce0 con 0x7f20680c14f0 2024-03-06T07:21:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f206d613640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20680a9760 0x7f206814dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f206d613640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20680a9760 0x7f206814dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37086/0 (socket says 172.21.15.1:37086) 2024-03-06T07:21:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f206d613640 1 -- 172.21.15.1:0/3500432962 learned_addr learned my addr 172.21.15.1:0/3500432962 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f206de14640 1 --2- 172.21.15.1:0/3500432962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f206806f6d0 0x7f2068154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f206e615640 1 --2- 172.21.15.1:0/3500432962 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20680c14f0 0x7f206814d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f206d613640 1 -- 172.21.15.1:0/3500432962 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20680c14f0 msgr2=0x7f206814d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f206d613640 1 --2- 172.21.15.1:0/3500432962 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20680c14f0 0x7f206814d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f206d613640 1 -- 172.21.15.1:0/3500432962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f206806f6d0 msgr2=0x7f2068154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f206d613640 1 --2- 172.21.15.1:0/3500432962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f206806f6d0 0x7f2068154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f206d613640 1 -- 172.21.15.1:0/3500432962 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20681ef170 con 0x7f20680a9760 2024-03-06T07:21:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f206d613640 1 --2- 172.21.15.1:0/3500432962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20680a9760 0x7f206814dc70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f206400c3c0 tx=0x7f206400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f205e7fc640 1 -- 172.21.15.1:0/3500432962 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2064017070 con 0x7f20680a9760 2024-03-06T07:21:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f205e7fc640 1 -- 172.21.15.1:0/3500432962 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20640052f0 con 0x7f20680a9760 2024-03-06T07:21:30.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f2070078640 1 -- 172.21.15.1:0/3500432962 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2068155ca0 con 0x7f20680a9760 2024-03-06T07:21:30.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.742+0000 7f205e7fc640 1 -- 172.21.15.1:0/3500432962 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2064003830 con 0x7f20680a9760 2024-03-06T07:21:30.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.743+0000 7f2070078640 1 -- 172.21.15.1:0/3500432962 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2068156c80 con 0x7f20680a9760 2024-03-06T07:21:30.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.744+0000 7f2070078640 1 -- 172.21.15.1:0/3500432962 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f206806c580 con 0x7f20680a9760 2024-03-06T07:21:30.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.744+0000 7f205e7fc640 1 -- 172.21.15.1:0/3500432962 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f206401a070 con 0x7f20680a9760 2024-03-06T07:21:30.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.744+0000 7f205e7fc640 1 --2- 172.21.15.1:0/3500432962 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f204c06b4d0 0x7f204c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.745+0000 7f205e7fc640 1 -- 172.21.15.1:0/3500432962 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2064089540 con 0x7f20680a9760 2024-03-06T07:21:30.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.745+0000 7f206de14640 1 --2- 172.21.15.1:0/3500432962 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f204c06b4d0 0x7f204c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.745+0000 7f206de14640 1 --2- 172.21.15.1:0/3500432962 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f204c06b4d0 0x7f204c06d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2058036000 tx=0x7f205803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.749+0000 7f205e7fc640 1 -- 172.21.15.1:0/3500432962 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f206806c580 con 0x7f20680a9760 2024-03-06T07:21:30.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.925+0000 7f2070078640 1 -- 172.21.15.1:0/3500432962 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f20680686e0 con 0x7f204c06b4d0 2024-03-06T07:21:30.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.932+0000 7f205e7fc640 1 -- 172.21.15.1:0/3500432962 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251920 (secure 0 0 0) 0x7f20680686e0 con 0x7f204c06b4d0 2024-03-06T07:21:30.936 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:30.938 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:21:30.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.935+0000 7f2070078640 1 -- 172.21.15.1:0/3500432962 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f204c06b4d0 msgr2=0x7f204c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.935+0000 7f2070078640 1 --2- 172.21.15.1:0/3500432962 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f204c06b4d0 0x7f204c06d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2058036000 tx=0x7f205803d040 comp rx=0 tx=0).stop 2024-03-06T07:21:30.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.935+0000 7f2070078640 1 -- 172.21.15.1:0/3500432962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20680a9760 msgr2=0x7f206814dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.935+0000 7f2070078640 1 --2- 172.21.15.1:0/3500432962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20680a9760 0x7f206814dc70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f206400c3c0 tx=0x7f206400c880 comp rx=0 tx=0).stop 2024-03-06T07:21:30.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.935+0000 7f2070078640 1 -- 172.21.15.1:0/3500432962 shutdown_connections 2024-03-06T07:21:30.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.935+0000 7f2070078640 1 --2- 172.21.15.1:0/3500432962 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f204c06b4d0 0x7f204c06d990 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.935+0000 7f2070078640 1 --2- 172.21.15.1:0/3500432962 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20680c14f0 0x7f206814d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.935+0000 7f2070078640 1 --2- 172.21.15.1:0/3500432962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20680a9760 0x7f206814dc70 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.935+0000 7f2070078640 1 --2- 172.21.15.1:0/3500432962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f206806f6d0 0x7f2068154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.935+0000 7f2070078640 1 -- 172.21.15.1:0/3500432962 >> 172.21.15.1:0/3500432962 conn(0x7f20680c3cc0 msgr2=0x7f20680c30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.935+0000 7f2070078640 1 -- 172.21.15.1:0/3500432962 shutdown_connections 2024-03-06T07:21:30.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:30.935+0000 7f2070078640 1 -- 172.21.15.1:0/3500432962 wait complete. 2024-03-06T07:21:30.949 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2024-03-06T07:21:30.704554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236548,"kb_used_data":3712,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738172,"statfs":{"total":773094113280,"available":772851888128,"internally_reserved":0,"allocated":3801088,"data_stored":2762876,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001888"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165116+0000","last_change":"2024-03-06T07:21:23.182706+0000","last_active":"2024-03-06T07:21:24.165116+0000","last_peered":"2024-03-06T07:21:24.165116+0000","last_clean":"2024-03-06T07:21:24.165116+0000","last_became_active":"2024-03-06T07:21:23.182444+0000","last_became_peered":"2024-03-06T07:21:23.182444+0000","last_unstale":"2024-03-06T07:21:24.165116+0000","last_undegraded":"2024-03-06T07:21:24.165116+0000","last_fullsized":"2024-03-06T07:21:24.165116+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:54.354235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164707+0000","last_change":"2024-03-06T07:21:23.167573+0000","last_active":"2024-03-06T07:21:24.164707+0000","last_peered":"2024-03-06T07:21:24.164707+0000","last_clean":"2024-03-06T07:21:24.164707+0000","last_became_active":"2024-03-06T07:21:23.166786+0000","last_became_peered":"2024-03-06T07:21:23.166786+0000","last_unstale":"2024-03-06T07:21:24.164707+0000","last_undegraded":"2024-03-06T07:21:24.164707+0000","last_fullsized":"2024-03-06T07:21:24.164707+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:22:31.654787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373289+0000","last_change":"2024-03-06T07:21:23.180108+0000","last_active":"2024-03-06T07:21:25.373289+0000","last_peered":"2024-03-06T07:21:25.373289+0000","last_clean":"2024-03-06T07:21:25.373289+0000","last_became_active":"2024-03-06T07:21:23.179812+0000","last_became_peered":"2024-03-06T07:21:23.179812+0000","last_unstale":"2024-03-06T07:21:25.373289+0000","last_undegraded":"2024-03-06T07:21:25.373289+0000","last_fullsized":"2024-03-06T07:21:25.373289+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:17.356744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164235+0000","last_change":"2024-03-06T07:21:23.175989+0000","last_active":"2024-03-06T07:21:24.164235+0000","last_peered":"2024-03-06T07:21:24.164235+0000","last_clean":"2024-03-06T07:21:24.164235+0000","last_became_active":"2024-03-06T07:21:23.175694+0000","last_became_peered":"2024-03-06T07:21:23.175694+0000","last_unstale":"2024-03-06T07:21:24.164235+0000","last_undegraded":"2024-03-06T07:21:24.164235+0000","last_fullsized":"2024-03-06T07:21:24.164235+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:10.625144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164055+0000","last_change":"2024-03-06T07:21:23.176818+0000","last_active":"2024-03-06T07:21:24.164055+0000","last_peered":"2024-03-06T07:21:24.164055+0000","last_clean":"2024-03-06T07:21:24.164055+0000","last_became_active":"2024-03-06T07:21:23.176528+0000","last_became_peered":"2024-03-06T07:21:23.176528+0000","last_unstale":"2024-03-06T07:21:24.164055+0000","last_undegraded":"2024-03-06T07:21:24.164055+0000","last_fullsized":"2024-03-06T07:21:24.164055+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:26.024267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373032+0000","last_change":"2024-03-06T07:21:23.181991+0000","last_active":"2024-03-06T07:21:25.373032+0000","last_peered":"2024-03-06T07:21:25.373032+0000","last_clean":"2024-03-06T07:21:25.373032+0000","last_became_active":"2024-03-06T07:21:23.181716+0000","last_became_peered":"2024-03-06T07:21:23.181716+0000","last_unstale":"2024-03-06T07:21:25.373032+0000","last_undegraded":"2024-03-06T07:21:25.373032+0000","last_fullsized":"2024-03-06T07:21:25.373032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:35.695974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.178997+0000","last_change":"2024-03-06T07:21:23.169886+0000","last_active":"2024-03-06T07:21:24.178997+0000","last_peered":"2024-03-06T07:21:24.178997+0000","last_clean":"2024-03-06T07:21:24.178997+0000","last_became_active":"2024-03-06T07:21:23.169504+0000","last_became_peered":"2024-03-06T07:21:23.169504+0000","last_unstale":"2024-03-06T07:21:24.178997+0000","last_undegraded":"2024-03-06T07:21:24.178997+0000","last_fullsized":"2024-03-06T07:21:24.178997+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:28:35.313141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165175+0000","last_change":"2024-03-06T07:21:23.180974+0000","last_active":"2024-03-06T07:21:24.165175+0000","last_peered":"2024-03-06T07:21:24.165175+0000","last_clean":"2024-03-06T07:21:24.165175+0000","last_became_active":"2024-03-06T07:21:23.180703+0000","last_became_peered":"2024-03-06T07:21:23.180703+0000","last_unstale":"2024-03-06T07:21:24.165175+0000","last_undegraded":"2024-03-06T07:21:24.165175+0000","last_fullsized":"2024-03-06T07:21:24.165175+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:06.540792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373655+0000","last_change":"2024-03-06T07:21:23.179818+0000","last_active":"2024-03-06T07:21:25.373655+0000","last_peered":"2024-03-06T07:21:25.373655+0000","last_clean":"2024-03-06T07:21:25.373655+0000","last_became_active":"2024-03-06T07:21:23.172741+0000","last_became_peered":"2024-03-06T07:21:23.172741+0000","last_unstale":"2024-03-06T07:21:25.373655+0000","last_undegraded":"2024-03-06T07:21:25.373655+0000","last_fullsized":"2024-03-06T07:21:25.373655+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:06.967643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179312+0000","last_change":"2024-03-06T07:21:23.173744+0000","last_active":"2024-03-06T07:21:24.179312+0000","last_peered":"2024-03-06T07:21:24.179312+0000","last_clean":"2024-03-06T07:21:24.179312+0000","last_became_active":"2024-03-06T07:21:23.173427+0000","last_became_peered":"2024-03-06T07:21:23.173427+0000","last_unstale":"2024-03-06T07:21:24.179312+0000","last_undegraded":"2024-03-06T07:21:24.179312+0000","last_fullsized":"2024-03-06T07:21:24.179312+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:12.449775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179447+0000","last_change":"2024-03-06T07:21:23.178961+0000","last_active":"2024-03-06T07:21:24.179447+0000","last_peered":"2024-03-06T07:21:24.179447+0000","last_clean":"2024-03-06T07:21:24.179447+0000","last_became_active":"2024-03-06T07:21:23.178096+0000","last_became_peered":"2024-03-06T07:21:23.178096+0000","last_unstale":"2024-03-06T07:21:24.179447+0000","last_undegraded":"2024-03-06T07:21:24.179447+0000","last_fullsized":"2024-03-06T07:21:24.179447+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:22:31.654787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373133+0000","last_change":"2024-03-06T07:21:23.169027+0000","last_active":"2024-03-06T07:21:25.373133+0000","last_peered":"2024-03-06T07:21:25.373133+0000","last_clean":"2024-03-06T07:21:25.373133+0000","last_became_active":"2024-03-06T07:21:23.168716+0000","last_became_peered":"2024-03-06T07:21:23.168716+0000","last_unstale":"2024-03-06T07:21:25.373133+0000","last_undegraded":"2024-03-06T07:21:25.373133+0000","last_fullsized":"2024-03-06T07:21:25.373133+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:19.113059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179083+0000","last_change":"2024-03-06T07:21:23.178092+0000","last_active":"2024-03-06T07:21:24.179083+0000","last_peered":"2024-03-06T07:21:24.179083+0000","last_clean":"2024-03-06T07:21:24.179083+0000","last_became_active":"2024-03-06T07:21:23.176982+0000","last_became_peered":"2024-03-06T07:21:23.176982+0000","last_unstale":"2024-03-06T07:21:24.179083+0000","last_undegraded":"2024-03-06T07:21:24.179083+0000","last_fullsized":"2024-03-06T07:21:24.179083+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:41.252208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.282915+0000","last_change":"2024-03-06T07:21:23.172743+0000","last_active":"2024-03-06T07:21:24.282915+0000","last_peered":"2024-03-06T07:21:24.282915+0000","last_clean":"2024-03-06T07:21:24.282915+0000","last_became_active":"2024-03-06T07:21:23.172486+0000","last_became_peered":"2024-03-06T07:21:23.172486+0000","last_unstale":"2024-03-06T07:21:24.282915+0000","last_undegraded":"2024-03-06T07:21:24.282915+0000","last_fullsized":"2024-03-06T07:21:24.282915+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:36.493357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179008+0000","last_change":"2024-03-06T07:21:23.170530+0000","last_active":"2024-03-06T07:21:24.179008+0000","last_peered":"2024-03-06T07:21:24.179008+0000","last_clean":"2024-03-06T07:21:24.179008+0000","last_became_active":"2024-03-06T07:21:23.170269+0000","last_became_peered":"2024-03-06T07:21:23.170269+0000","last_unstale":"2024-03-06T07:21:24.179008+0000","last_undegraded":"2024-03-06T07:21:24.179008+0000","last_fullsized":"2024-03-06T07:21:24.179008+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:11.514224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.209091+0000","last_change":"2024-03-06T07:21:23.175412+0000","last_active":"2024-03-06T07:21:25.209091+0000","last_peered":"2024-03-06T07:21:25.209091+0000","last_clean":"2024-03-06T07:21:25.209091+0000","last_became_active":"2024-03-06T07:21:23.175113+0000","last_became_peered":"2024-03-06T07:21:23.175113+0000","last_unstale":"2024-03-06T07:21:25.209091+0000","last_undegraded":"2024-03-06T07:21:25.209091+0000","last_fullsized":"2024-03-06T07:21:25.209091+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:35.695974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.209635+0000","last_change":"2024-03-06T07:21:23.174714+0000","last_active":"2024-03-06T07:21:25.209635+0000","last_peered":"2024-03-06T07:21:25.209635+0000","last_clean":"2024-03-06T07:21:25.209635+0000","last_became_active":"2024-03-06T07:21:23.174372+0000","last_became_peered":"2024-03-06T07:21:23.174372+0000","last_unstale":"2024-03-06T07:21:25.209635+0000","last_undegraded":"2024-03-06T07:21:25.209635+0000","last_fullsized":"2024-03-06T07:21:25.209635+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:11.514224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165483+0000","last_change":"2024-03-06T07:21:23.173359+0000","last_active":"2024-03-06T07:21:24.165483+0000","last_peered":"2024-03-06T07:21:24.165483+0000","last_clean":"2024-03-06T07:21:24.165483+0000","last_became_active":"2024-03-06T07:21:23.173040+0000","last_became_peered":"2024-03-06T07:21:23.173040+0000","last_unstale":"2024-03-06T07:21:24.165483+0000","last_undegraded":"2024-03-06T07:21:24.165483+0000","last_fullsized":"2024-03-06T07:21:24.165483+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:20.162045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164458+0000","last_change":"2024-03-06T07:21:23.177492+0000","last_active":"2024-03-06T07:21:24.164458+0000","last_peered":"2024-03-06T07:21:24.164458+0000","last_clean":"2024-03-06T07:21:24.164458+0000","last_became_active":"2024-03-06T07:21:23.177109+0000","last_became_peered":"2024-03-06T07:21:23.177109+0000","last_unstale":"2024-03-06T07:21:24.164458+0000","last_undegraded":"2024-03-06T07:21:24.164458+0000","last_fullsized":"2024-03-06T07:21:24.164458+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:50.469010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165246+0000","last_change":"2024-03-06T07:21:23.173277+0000","last_active":"2024-03-06T07:21:24.165246+0000","last_peered":"2024-03-06T07:21:24.165246+0000","last_clean":"2024-03-06T07:21:24.165246+0000","last_became_active":"2024-03-06T07:21:23.172974+0000","last_became_peered":"2024-03-06T07:21:23.172974+0000","last_unstale":"2024-03-06T07:21:24.165246+0000","last_undegraded":"2024-03-06T07:21:24.165246+0000","last_fullsized":"2024-03-06T07:21:24.165246+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:08.049695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179011+0000","last_change":"2024-03-06T07:21:23.176636+0000","last_active":"2024-03-06T07:21:24.179011+0000","last_peered":"2024-03-06T07:21:24.179011+0000","last_clean":"2024-03-06T07:21:24.179011+0000","last_became_active":"2024-03-06T07:21:23.176347+0000","last_became_peered":"2024-03-06T07:21:23.176347+0000","last_unstale":"2024-03-06T07:21:24.179011+0000","last_undegraded":"2024-03-06T07:21:24.179011+0000","last_fullsized":"2024-03-06T07:21:24.179011+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:54.233662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283249+0000","last_change":"2024-03-06T07:21:23.177707+0000","last_active":"2024-03-06T07:21:24.283249+0000","last_peered":"2024-03-06T07:21:24.283249+0000","last_clean":"2024-03-06T07:21:24.283249+0000","last_became_active":"2024-03-06T07:21:23.172182+0000","last_became_peered":"2024-03-06T07:21:23.172182+0000","last_unstale":"2024-03-06T07:21:24.283249+0000","last_undegraded":"2024-03-06T07:21:24.283249+0000","last_fullsized":"2024-03-06T07:21:24.283249+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:19.113059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.209200+0000","last_change":"2024-03-06T07:21:23.159844+0000","last_active":"2024-03-06T07:21:25.209200+0000","last_peered":"2024-03-06T07:21:25.209200+0000","last_clean":"2024-03-06T07:21:25.209200+0000","last_became_active":"2024-03-06T07:21:23.159578+0000","last_became_peered":"2024-03-06T07:21:23.159578+0000","last_unstale":"2024-03-06T07:21:25.209200+0000","last_undegraded":"2024-03-06T07:21:25.209200+0000","last_fullsized":"2024-03-06T07:21:25.209200+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:19.113059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164259+0000","last_change":"2024-03-06T07:21:23.167040+0000","last_active":"2024-03-06T07:21:24.164259+0000","last_peered":"2024-03-06T07:21:24.164259+0000","last_clean":"2024-03-06T07:21:24.164259+0000","last_became_active":"2024-03-06T07:21:23.166340+0000","last_became_peered":"2024-03-06T07:21:23.166340+0000","last_unstale":"2024-03-06T07:21:24.164259+0000","last_undegraded":"2024-03-06T07:21:24.164259+0000","last_fullsized":"2024-03-06T07:21:24.164259+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:59.215536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165465+0000","last_change":"2024-03-06T07:21:23.182761+0000","last_active":"2024-03-06T07:21:24.165465+0000","last_peered":"2024-03-06T07:21:24.165465+0000","last_clean":"2024-03-06T07:21:24.165465+0000","last_became_active":"2024-03-06T07:21:23.182467+0000","last_became_peered":"2024-03-06T07:21:23.182467+0000","last_unstale":"2024-03-06T07:21:24.165465+0000","last_undegraded":"2024-03-06T07:21:24.165465+0000","last_fullsized":"2024-03-06T07:21:24.165465+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:12.094182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373542+0000","last_change":"2024-03-06T07:21:23.179741+0000","last_active":"2024-03-06T07:21:25.373542+0000","last_peered":"2024-03-06T07:21:25.373542+0000","last_clean":"2024-03-06T07:21:25.373542+0000","last_became_active":"2024-03-06T07:21:23.179444+0000","last_became_peered":"2024-03-06T07:21:23.179444+0000","last_unstale":"2024-03-06T07:21:25.373542+0000","last_undegraded":"2024-03-06T07:21:25.373542+0000","last_fullsized":"2024-03-06T07:21:25.373542+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:07.635594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165441+0000","last_change":"2024-03-06T07:21:23.173847+0000","last_active":"2024-03-06T07:21:24.165441+0000","last_peered":"2024-03-06T07:21:24.165441+0000","last_clean":"2024-03-06T07:21:24.165441+0000","last_became_active":"2024-03-06T07:21:23.173587+0000","last_became_peered":"2024-03-06T07:21:23.173587+0000","last_unstale":"2024-03-06T07:21:24.165441+0000","last_undegraded":"2024-03-06T07:21:24.165441+0000","last_fullsized":"2024-03-06T07:21:24.165441+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:07.628952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179265+0000","last_change":"2024-03-06T07:21:23.170610+0000","last_active":"2024-03-06T07:21:24.179265+0000","last_peered":"2024-03-06T07:21:24.179265+0000","last_clean":"2024-03-06T07:21:24.179265+0000","last_became_active":"2024-03-06T07:21:23.169962+0000","last_became_peered":"2024-03-06T07:21:23.169962+0000","last_unstale":"2024-03-06T07:21:24.179265+0000","last_undegraded":"2024-03-06T07:21:24.179265+0000","last_fullsized":"2024-03-06T07:21:24.179265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:22.556995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165379+0000","last_change":"2024-03-06T07:21:23.175388+0000","last_active":"2024-03-06T07:21:24.165379+0000","last_peered":"2024-03-06T07:21:24.165379+0000","last_clean":"2024-03-06T07:21:24.165379+0000","last_became_active":"2024-03-06T07:21:23.175097+0000","last_became_peered":"2024-03-06T07:21:23.175097+0000","last_unstale":"2024-03-06T07:21:24.165379+0000","last_undegraded":"2024-03-06T07:21:24.165379+0000","last_fullsized":"2024-03-06T07:21:24.165379+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:22.372508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165127+0000","last_change":"2024-03-06T07:21:23.178513+0000","last_active":"2024-03-06T07:21:24.165127+0000","last_peered":"2024-03-06T07:21:24.165127+0000","last_clean":"2024-03-06T07:21:24.165127+0000","last_became_active":"2024-03-06T07:21:23.177896+0000","last_became_peered":"2024-03-06T07:21:23.177896+0000","last_unstale":"2024-03-06T07:21:24.165127+0000","last_undegraded":"2024-03-06T07:21:24.165127+0000","last_fullsized":"2024-03-06T07:21:24.165127+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:00.584960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.209260+0000","last_change":"2024-03-06T07:21:23.162516+0000","last_active":"2024-03-06T07:21:25.209260+0000","last_peered":"2024-03-06T07:21:25.209260+0000","last_clean":"2024-03-06T07:21:25.209260+0000","last_became_active":"2024-03-06T07:21:23.162258+0000","last_became_peered":"2024-03-06T07:21:23.162258+0000","last_unstale":"2024-03-06T07:21:25.209260+0000","last_undegraded":"2024-03-06T07:21:25.209260+0000","last_fullsized":"2024-03-06T07:21:25.209260+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:17.356744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373437+0000","last_change":"2024-03-06T07:21:23.172221+0000","last_active":"2024-03-06T07:21:25.373437+0000","last_peered":"2024-03-06T07:21:25.373437+0000","last_clean":"2024-03-06T07:21:25.373437+0000","last_became_active":"2024-03-06T07:21:23.171940+0000","last_became_peered":"2024-03-06T07:21:23.171940+0000","last_unstale":"2024-03-06T07:21:25.373437+0000","last_undegraded":"2024-03-06T07:21:25.373437+0000","last_fullsized":"2024-03-06T07:21:25.373437+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:06.967643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.209363+0000","last_change":"2024-03-06T07:21:23.175504+0000","last_active":"2024-03-06T07:21:25.209363+0000","last_peered":"2024-03-06T07:21:25.209363+0000","last_clean":"2024-03-06T07:21:25.209363+0000","last_became_active":"2024-03-06T07:21:23.175180+0000","last_became_peered":"2024-03-06T07:21:23.175180+0000","last_unstale":"2024-03-06T07:21:25.209363+0000","last_undegraded":"2024-03-06T07:21:25.209363+0000","last_fullsized":"2024-03-06T07:21:25.209363+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:07.635594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165580+0000","last_change":"2024-03-06T07:21:22.171884+0000","last_active":"2024-03-06T07:21:24.165580+0000","last_peered":"2024-03-06T07:21:24.165580+0000","last_clean":"2024-03-06T07:21:24.165580+0000","last_became_active":"2024-03-06T07:21:22.171573+0000","last_became_peered":"2024-03-06T07:21:22.171573+0000","last_unstale":"2024-03-06T07:21:24.165580+0000","last_undegraded":"2024-03-06T07:21:24.165580+0000","last_fullsized":"2024-03-06T07:21:24.165580+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:45.006491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373574+0000","last_change":"2024-03-06T07:21:23.181934+0000","last_active":"2024-03-06T07:21:25.373574+0000","last_peered":"2024-03-06T07:21:25.373574+0000","last_clean":"2024-03-06T07:21:25.373574+0000","last_became_active":"2024-03-06T07:21:23.181645+0000","last_became_peered":"2024-03-06T07:21:23.181645+0000","last_unstale":"2024-03-06T07:21:25.373574+0000","last_undegraded":"2024-03-06T07:21:25.373574+0000","last_fullsized":"2024-03-06T07:21:25.373574+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:05.754195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373878+0000","last_change":"2024-03-06T07:21:22.170291+0000","last_active":"2024-03-06T07:21:25.373878+0000","last_peered":"2024-03-06T07:21:25.373878+0000","last_clean":"2024-03-06T07:21:25.373878+0000","last_became_active":"2024-03-06T07:21:22.169996+0000","last_became_peered":"2024-03-06T07:21:22.169996+0000","last_unstale":"2024-03-06T07:21:25.373878+0000","last_undegraded":"2024-03-06T07:21:25.373878+0000","last_fullsized":"2024-03-06T07:21:25.373878+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:59.580100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283289+0000","last_change":"2024-03-06T07:21:23.171182+0000","last_active":"2024-03-06T07:21:24.283289+0000","last_peered":"2024-03-06T07:21:24.283289+0000","last_clean":"2024-03-06T07:21:24.283289+0000","last_became_active":"2024-03-06T07:21:23.170864+0000","last_became_peered":"2024-03-06T07:21:23.170864+0000","last_unstale":"2024-03-06T07:21:24.283289+0000","last_undegraded":"2024-03-06T07:21:24.283289+0000","last_fullsized":"2024-03-06T07:21:24.283289+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:07.635594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373697+0000","last_change":"2024-03-06T07:21:22.174770+0000","last_active":"2024-03-06T07:21:25.373697+0000","last_peered":"2024-03-06T07:21:25.373697+0000","last_clean":"2024-03-06T07:21:25.373697+0000","last_became_active":"2024-03-06T07:21:22.174494+0000","last_became_peered":"2024-03-06T07:21:22.174494+0000","last_unstale":"2024-03-06T07:21:25.373697+0000","last_undegraded":"2024-03-06T07:21:25.373697+0000","last_fullsized":"2024-03-06T07:21:25.373697+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:06.624093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165090+0000","last_change":"2024-03-06T07:21:23.173050+0000","last_active":"2024-03-06T07:21:24.165090+0000","last_peered":"2024-03-06T07:21:24.165090+0000","last_clean":"2024-03-06T07:21:24.165090+0000","last_became_active":"2024-03-06T07:21:23.172741+0000","last_became_peered":"2024-03-06T07:21:23.172741+0000","last_unstale":"2024-03-06T07:21:24.165090+0000","last_undegraded":"2024-03-06T07:21:24.165090+0000","last_fullsized":"2024-03-06T07:21:24.165090+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:15.661528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283240+0000","last_change":"2024-03-06T07:21:22.166297+0000","last_active":"2024-03-06T07:21:24.283240+0000","last_peered":"2024-03-06T07:21:24.283240+0000","last_clean":"2024-03-06T07:21:24.283240+0000","last_became_active":"2024-03-06T07:21:22.166020+0000","last_became_peered":"2024-03-06T07:21:22.166020+0000","last_unstale":"2024-03-06T07:21:24.283240+0000","last_undegraded":"2024-03-06T07:21:24.283240+0000","last_fullsized":"2024-03-06T07:21:24.283240+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:59.580100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.372913+0000","last_change":"2024-03-06T07:21:23.180253+0000","last_active":"2024-03-06T07:21:25.372913+0000","last_peered":"2024-03-06T07:21:25.372913+0000","last_clean":"2024-03-06T07:21:25.372913+0000","last_became_active":"2024-03-06T07:21:23.174841+0000","last_became_peered":"2024-03-06T07:21:23.174841+0000","last_unstale":"2024-03-06T07:21:25.372913+0000","last_undegraded":"2024-03-06T07:21:25.372913+0000","last_fullsized":"2024-03-06T07:21:25.372913+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:52.373147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179164+0000","last_change":"2024-03-06T07:21:22.170216+0000","last_active":"2024-03-06T07:21:24.179164+0000","last_peered":"2024-03-06T07:21:24.179164+0000","last_clean":"2024-03-06T07:21:24.179164+0000","last_became_active":"2024-03-06T07:21:22.169909+0000","last_became_peered":"2024-03-06T07:21:22.169909+0000","last_unstale":"2024-03-06T07:21:24.179164+0000","last_undegraded":"2024-03-06T07:21:24.179164+0000","last_fullsized":"2024-03-06T07:21:24.179164+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:07.044836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179273+0000","last_change":"2024-03-06T07:21:23.176300+0000","last_active":"2024-03-06T07:21:24.179273+0000","last_peered":"2024-03-06T07:21:24.179273+0000","last_clean":"2024-03-06T07:21:24.179273+0000","last_became_active":"2024-03-06T07:21:23.175986+0000","last_became_peered":"2024-03-06T07:21:23.175986+0000","last_unstale":"2024-03-06T07:21:24.179273+0000","last_undegraded":"2024-03-06T07:21:24.179273+0000","last_fullsized":"2024-03-06T07:21:24.179273+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:53.558264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.209256+0000","last_change":"2024-03-06T07:21:22.174498+0000","last_active":"2024-03-06T07:21:25.209256+0000","last_peered":"2024-03-06T07:21:25.209256+0000","last_clean":"2024-03-06T07:21:25.209256+0000","last_became_active":"2024-03-06T07:21:22.174189+0000","last_became_peered":"2024-03-06T07:21:22.174189+0000","last_unstale":"2024-03-06T07:21:25.209256+0000","last_undegraded":"2024-03-06T07:21:25.209256+0000","last_fullsized":"2024-03-06T07:21:25.209256+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:40.673099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283270+0000","last_change":"2024-03-06T07:21:23.180855+0000","last_active":"2024-03-06T07:21:24.283270+0000","last_peered":"2024-03-06T07:21:24.283270+0000","last_clean":"2024-03-06T07:21:24.283270+0000","last_became_active":"2024-03-06T07:21:23.180588+0000","last_became_peered":"2024-03-06T07:21:23.180588+0000","last_unstale":"2024-03-06T07:21:24.283270+0000","last_undegraded":"2024-03-06T07:21:24.283270+0000","last_fullsized":"2024-03-06T07:21:24.283270+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:06.967643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373289+0000","last_change":"2024-03-06T07:21:22.172223+0000","last_active":"2024-03-06T07:21:25.373289+0000","last_peered":"2024-03-06T07:21:25.373289+0000","last_clean":"2024-03-06T07:21:25.373289+0000","last_became_active":"2024-03-06T07:21:22.171912+0000","last_became_peered":"2024-03-06T07:21:22.171912+0000","last_unstale":"2024-03-06T07:21:25.373289+0000","last_undegraded":"2024-03-06T07:21:25.373289+0000","last_fullsized":"2024-03-06T07:21:25.373289+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:59.580100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164902+0000","last_change":"2024-03-06T07:21:23.174371+0000","last_active":"2024-03-06T07:21:24.164902+0000","last_peered":"2024-03-06T07:21:24.164902+0000","last_clean":"2024-03-06T07:21:24.164902+0000","last_became_active":"2024-03-06T07:21:23.173893+0000","last_became_peered":"2024-03-06T07:21:23.173893+0000","last_unstale":"2024-03-06T07:21:24.164902+0000","last_undegraded":"2024-03-06T07:21:24.164902+0000","last_fullsized":"2024-03-06T07:21:24.164902+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:37.222226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.282797+0000","last_change":"2024-03-06T07:21:22.174586+0000","last_active":"2024-03-06T07:21:24.282797+0000","last_peered":"2024-03-06T07:21:24.282797+0000","last_clean":"2024-03-06T07:21:24.282797+0000","last_became_active":"2024-03-06T07:21:22.174294+0000","last_became_peered":"2024-03-06T07:21:22.174294+0000","last_unstale":"2024-03-06T07:21:24.282797+0000","last_undegraded":"2024-03-06T07:21:24.282797+0000","last_fullsized":"2024-03-06T07:21:24.282797+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:40.673099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164248+0000","last_change":"2024-03-06T07:21:23.175308+0000","last_active":"2024-03-06T07:21:24.164248+0000","last_peered":"2024-03-06T07:21:24.164248+0000","last_clean":"2024-03-06T07:21:24.164248+0000","last_became_active":"2024-03-06T07:21:23.175045+0000","last_became_peered":"2024-03-06T07:21:23.175045+0000","last_unstale":"2024-03-06T07:21:24.164248+0000","last_undegraded":"2024-03-06T07:21:24.164248+0000","last_fullsized":"2024-03-06T07:21:24.164248+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:28.084724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165189+0000","last_change":"2024-03-06T07:21:22.173698+0000","last_active":"2024-03-06T07:21:24.165189+0000","last_peered":"2024-03-06T07:21:24.165189+0000","last_clean":"2024-03-06T07:21:24.165189+0000","last_became_active":"2024-03-06T07:21:22.173426+0000","last_became_peered":"2024-03-06T07:21:22.173426+0000","last_unstale":"2024-03-06T07:21:24.165189+0000","last_undegraded":"2024-03-06T07:21:24.165189+0000","last_fullsized":"2024-03-06T07:21:24.165189+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:16.351884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.209637+0000","last_change":"2024-03-06T07:21:23.171603+0000","last_active":"2024-03-06T07:21:25.209637+0000","last_peered":"2024-03-06T07:21:25.209637+0000","last_clean":"2024-03-06T07:21:25.209637+0000","last_became_active":"2024-03-06T07:21:23.171109+0000","last_became_peered":"2024-03-06T07:21:23.171109+0000","last_unstale":"2024-03-06T07:21:25.209637+0000","last_undegraded":"2024-03-06T07:21:25.209637+0000","last_fullsized":"2024-03-06T07:21:25.209637+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:54.233662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179439+0000","last_change":"2024-03-06T07:21:22.166081+0000","last_active":"2024-03-06T07:21:24.179439+0000","last_peered":"2024-03-06T07:21:24.179439+0000","last_clean":"2024-03-06T07:21:24.179439+0000","last_became_active":"2024-03-06T07:21:22.165820+0000","last_became_peered":"2024-03-06T07:21:22.165820+0000","last_unstale":"2024-03-06T07:21:24.179439+0000","last_undegraded":"2024-03-06T07:21:24.179439+0000","last_fullsized":"2024-03-06T07:21:24.179439+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:59.580100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373794+0000","last_change":"2024-03-06T07:21:23.180325+0000","last_active":"2024-03-06T07:21:25.373794+0000","last_peered":"2024-03-06T07:21:25.373794+0000","last_clean":"2024-03-06T07:21:25.373794+0000","last_became_active":"2024-03-06T07:21:23.175546+0000","last_became_peered":"2024-03-06T07:21:23.175546+0000","last_unstale":"2024-03-06T07:21:25.373794+0000","last_undegraded":"2024-03-06T07:21:25.373794+0000","last_fullsized":"2024-03-06T07:21:25.373794+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:05.754195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179551+0000","last_change":"2024-03-06T07:21:22.173265+0000","last_active":"2024-03-06T07:21:24.179551+0000","last_peered":"2024-03-06T07:21:24.179551+0000","last_clean":"2024-03-06T07:21:24.179551+0000","last_became_active":"2024-03-06T07:21:22.172945+0000","last_became_peered":"2024-03-06T07:21:22.172945+0000","last_unstale":"2024-03-06T07:21:24.179551+0000","last_undegraded":"2024-03-06T07:21:24.179551+0000","last_fullsized":"2024-03-06T07:21:24.179551+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:21.367648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283250+0000","last_change":"2024-03-06T07:21:23.173659+0000","last_active":"2024-03-06T07:21:24.283250+0000","last_peered":"2024-03-06T07:21:24.283250+0000","last_clean":"2024-03-06T07:21:24.283250+0000","last_became_active":"2024-03-06T07:21:23.173401+0000","last_became_peered":"2024-03-06T07:21:23.173401+0000","last_unstale":"2024-03-06T07:21:24.283250+0000","last_undegraded":"2024-03-06T07:21:24.283250+0000","last_fullsized":"2024-03-06T07:21:24.283250+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:17.356744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164883+0000","last_change":"2024-03-06T07:21:22.170375+0000","last_active":"2024-03-06T07:21:24.164883+0000","last_peered":"2024-03-06T07:21:24.164883+0000","last_clean":"2024-03-06T07:21:24.164883+0000","last_became_active":"2024-03-06T07:21:22.170096+0000","last_became_peered":"2024-03-06T07:21:22.170096+0000","last_unstale":"2024-03-06T07:21:24.164883+0000","last_undegraded":"2024-03-06T07:21:24.164883+0000","last_fullsized":"2024-03-06T07:21:24.164883+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:35.488498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.372913+0000","last_change":"2024-03-06T07:21:23.180941+0000","last_active":"2024-03-06T07:21:25.372913+0000","last_peered":"2024-03-06T07:21:25.372913+0000","last_clean":"2024-03-06T07:21:25.372913+0000","last_became_active":"2024-03-06T07:21:23.180662+0000","last_became_peered":"2024-03-06T07:21:23.180662+0000","last_unstale":"2024-03-06T07:21:25.372913+0000","last_undegraded":"2024-03-06T07:21:25.372913+0000","last_fullsized":"2024-03-06T07:21:25.372913+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:07.635594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165167+0000","last_change":"2024-03-06T07:21:22.168669+0000","last_active":"2024-03-06T07:21:24.165167+0000","last_peered":"2024-03-06T07:21:24.165167+0000","last_clean":"2024-03-06T07:21:24.165167+0000","last_became_active":"2024-03-06T07:21:22.168382+0000","last_became_peered":"2024-03-06T07:21:22.168382+0000","last_unstale":"2024-03-06T07:21:24.165167+0000","last_undegraded":"2024-03-06T07:21:24.165167+0000","last_fullsized":"2024-03-06T07:21:24.165167+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:34.691115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165029+0000","last_change":"2024-03-06T07:21:23.170835+0000","last_active":"2024-03-06T07:21:24.165029+0000","last_peered":"2024-03-06T07:21:24.165029+0000","last_clean":"2024-03-06T07:21:24.165029+0000","last_became_active":"2024-03-06T07:21:23.170538+0000","last_became_peered":"2024-03-06T07:21:23.170538+0000","last_unstale":"2024-03-06T07:21:24.165029+0000","last_undegraded":"2024-03-06T07:21:24.165029+0000","last_fullsized":"2024-03-06T07:21:24.165029+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:08.450197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373131+0000","last_change":"2024-03-06T07:21:22.173295+0000","last_active":"2024-03-06T07:21:25.373131+0000","last_peered":"2024-03-06T07:21:25.373131+0000","last_clean":"2024-03-06T07:21:25.373131+0000","last_became_active":"2024-03-06T07:21:22.172903+0000","last_became_peered":"2024-03-06T07:21:22.172903+0000","last_unstale":"2024-03-06T07:21:25.373131+0000","last_undegraded":"2024-03-06T07:21:25.373131+0000","last_fullsized":"2024-03-06T07:21:25.373131+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:07.044836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.209227+0000","last_change":"2024-03-06T07:21:23.163139+0000","last_active":"2024-03-06T07:21:25.209227+0000","last_peered":"2024-03-06T07:21:25.209227+0000","last_clean":"2024-03-06T07:21:25.209227+0000","last_became_active":"2024-03-06T07:21:23.162884+0000","last_became_peered":"2024-03-06T07:21:23.162884+0000","last_unstale":"2024-03-06T07:21:25.209227+0000","last_undegraded":"2024-03-06T07:21:25.209227+0000","last_fullsized":"2024-03-06T07:21:25.209227+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:06.967643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164924+0000","last_change":"2024-03-06T07:21:22.171888+0000","last_active":"2024-03-06T07:21:24.164924+0000","last_peered":"2024-03-06T07:21:24.164924+0000","last_clean":"2024-03-06T07:21:24.164924+0000","last_became_active":"2024-03-06T07:21:22.171588+0000","last_became_peered":"2024-03-06T07:21:22.171588+0000","last_unstale":"2024-03-06T07:21:24.164924+0000","last_undegraded":"2024-03-06T07:21:24.164924+0000","last_fullsized":"2024-03-06T07:21:24.164924+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:33.446854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165200+0000","last_change":"2024-03-06T07:21:23.168200+0000","last_active":"2024-03-06T07:21:24.165200+0000","last_peered":"2024-03-06T07:21:24.165200+0000","last_clean":"2024-03-06T07:21:24.165200+0000","last_became_active":"2024-03-06T07:21:23.167903+0000","last_became_peered":"2024-03-06T07:21:23.167903+0000","last_unstale":"2024-03-06T07:21:24.165200+0000","last_undegraded":"2024-03-06T07:21:24.165200+0000","last_fullsized":"2024-03-06T07:21:24.165200+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:01.144803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.209119+0000","last_change":"2024-03-06T07:21:22.175264+0000","last_active":"2024-03-06T07:21:25.209119+0000","last_peered":"2024-03-06T07:21:25.209119+0000","last_clean":"2024-03-06T07:21:25.209119+0000","last_became_active":"2024-03-06T07:21:22.174941+0000","last_became_peered":"2024-03-06T07:21:22.174941+0000","last_unstale":"2024-03-06T07:21:25.209119+0000","last_undegraded":"2024-03-06T07:21:25.209119+0000","last_fullsized":"2024-03-06T07:21:25.209119+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:51.086001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164775+0000","last_change":"2024-03-06T07:21:23.171904+0000","last_active":"2024-03-06T07:21:24.164775+0000","last_peered":"2024-03-06T07:21:24.164775+0000","last_clean":"2024-03-06T07:21:24.164775+0000","last_became_active":"2024-03-06T07:21:23.171618+0000","last_became_peered":"2024-03-06T07:21:23.171618+0000","last_unstale":"2024-03-06T07:21:24.164775+0000","last_undegraded":"2024-03-06T07:21:24.164775+0000","last_fullsized":"2024-03-06T07:21:24.164775+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:48.917896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179498+0000","last_change":"2024-03-06T07:21:22.174985+0000","last_active":"2024-03-06T07:21:24.179498+0000","last_peered":"2024-03-06T07:21:24.179498+0000","last_clean":"2024-03-06T07:21:24.179498+0000","last_became_active":"2024-03-06T07:21:22.174284+0000","last_became_peered":"2024-03-06T07:21:22.174284+0000","last_unstale":"2024-03-06T07:21:24.179498+0000","last_undegraded":"2024-03-06T07:21:24.179498+0000","last_fullsized":"2024-03-06T07:21:24.179498+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:11.089323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179490+0000","last_change":"2024-03-06T07:21:23.179740+0000","last_active":"2024-03-06T07:21:24.179490+0000","last_peered":"2024-03-06T07:21:24.179490+0000","last_clean":"2024-03-06T07:21:24.179490+0000","last_became_active":"2024-03-06T07:21:23.177114+0000","last_became_peered":"2024-03-06T07:21:23.177114+0000","last_unstale":"2024-03-06T07:21:24.179490+0000","last_undegraded":"2024-03-06T07:21:24.179490+0000","last_fullsized":"2024-03-06T07:21:24.179490+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:36.368758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283361+0000","last_change":"2024-03-06T07:21:22.171126+0000","last_active":"2024-03-06T07:21:24.283361+0000","last_peered":"2024-03-06T07:21:24.283361+0000","last_clean":"2024-03-06T07:21:24.283361+0000","last_became_active":"2024-03-06T07:21:22.170803+0000","last_became_peered":"2024-03-06T07:21:22.170803+0000","last_unstale":"2024-03-06T07:21:24.283361+0000","last_undegraded":"2024-03-06T07:21:24.283361+0000","last_fullsized":"2024-03-06T07:21:24.283361+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:07.044836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.283308+0000","last_change":"2024-03-06T07:21:23.174451+0000","last_active":"2024-03-06T07:21:24.283308+0000","last_peered":"2024-03-06T07:21:24.283308+0000","last_clean":"2024-03-06T07:21:24.283308+0000","last_became_active":"2024-03-06T07:21:23.174072+0000","last_became_peered":"2024-03-06T07:21:23.174072+0000","last_unstale":"2024-03-06T07:21:24.283308+0000","last_undegraded":"2024-03-06T07:21:24.283308+0000","last_fullsized":"2024-03-06T07:21:24.283308+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:05.754195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164461+0000","last_change":"2024-03-06T07:21:22.169060+0000","last_active":"2024-03-06T07:21:24.164461+0000","last_peered":"2024-03-06T07:21:24.164461+0000","last_clean":"2024-03-06T07:21:24.164461+0000","last_became_active":"2024-03-06T07:21:22.168716+0000","last_became_peered":"2024-03-06T07:21:22.168716+0000","last_unstale":"2024-03-06T07:21:24.164461+0000","last_undegraded":"2024-03-06T07:21:24.164461+0000","last_fullsized":"2024-03-06T07:21:24.164461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:15.088870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164509+0000","last_change":"2024-03-06T07:21:23.169921+0000","last_active":"2024-03-06T07:21:24.164509+0000","last_peered":"2024-03-06T07:21:24.164509+0000","last_clean":"2024-03-06T07:21:24.164509+0000","last_became_active":"2024-03-06T07:21:23.169610+0000","last_became_peered":"2024-03-06T07:21:23.169610+0000","last_unstale":"2024-03-06T07:21:24.164509+0000","last_undegraded":"2024-03-06T07:21:24.164509+0000","last_fullsized":"2024-03-06T07:21:24.164509+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:34.451714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373174+0000","last_change":"2024-03-06T07:21:22.174120+0000","last_active":"2024-03-06T07:21:25.373174+0000","last_peered":"2024-03-06T07:21:25.373174+0000","last_clean":"2024-03-06T07:21:25.373174+0000","last_became_active":"2024-03-06T07:21:22.173826+0000","last_became_peered":"2024-03-06T07:21:22.173826+0000","last_unstale":"2024-03-06T07:21:25.373174+0000","last_undegraded":"2024-03-06T07:21:25.373174+0000","last_fullsized":"2024-03-06T07:21:25.373174+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:21.367648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.209597+0000","last_change":"2024-03-06T07:21:23.171548+0000","last_active":"2024-03-06T07:21:25.209597+0000","last_peered":"2024-03-06T07:21:25.209597+0000","last_clean":"2024-03-06T07:21:25.209597+0000","last_became_active":"2024-03-06T07:21:23.171272+0000","last_became_peered":"2024-03-06T07:21:23.171272+0000","last_unstale":"2024-03-06T07:21:25.209597+0000","last_undegraded":"2024-03-06T07:21:25.209597+0000","last_fullsized":"2024-03-06T07:21:25.209597+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:28:35.313141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164151+0000","last_change":"2024-03-06T07:21:22.165324+0000","last_active":"2024-03-06T07:21:24.164151+0000","last_peered":"2024-03-06T07:21:24.164151+0000","last_clean":"2024-03-06T07:21:24.164151+0000","last_became_active":"2024-03-06T07:21:22.165064+0000","last_became_peered":"2024-03-06T07:21:22.165064+0000","last_unstale":"2024-03-06T07:21:24.164151+0000","last_undegraded":"2024-03-06T07:21:24.164151+0000","last_fullsized":"2024-03-06T07:21:24.164151+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:42.726655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373055+0000","last_change":"2024-03-06T07:21:23.180694+0000","last_active":"2024-03-06T07:21:25.373055+0000","last_peered":"2024-03-06T07:21:25.373055+0000","last_clean":"2024-03-06T07:21:25.373055+0000","last_became_active":"2024-03-06T07:21:23.180413+0000","last_became_peered":"2024-03-06T07:21:23.180413+0000","last_unstale":"2024-03-06T07:21:25.373055+0000","last_undegraded":"2024-03-06T07:21:25.373055+0000","last_fullsized":"2024-03-06T07:21:25.373055+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:36.493357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164033+0000","last_change":"2024-03-06T07:21:22.174287+0000","last_active":"2024-03-06T07:21:24.164033+0000","last_peered":"2024-03-06T07:21:24.164033+0000","last_clean":"2024-03-06T07:21:24.164033+0000","last_became_active":"2024-03-06T07:21:22.173947+0000","last_became_peered":"2024-03-06T07:21:22.173947+0000","last_unstale":"2024-03-06T07:21:24.164033+0000","last_undegraded":"2024-03-06T07:21:24.164033+0000","last_fullsized":"2024-03-06T07:21:24.164033+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:02.949041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179184+0000","last_change":"2024-03-06T07:21:23.173219+0000","last_active":"2024-03-06T07:21:24.179184+0000","last_peered":"2024-03-06T07:21:24.179184+0000","last_clean":"2024-03-06T07:21:24.179184+0000","last_became_active":"2024-03-06T07:21:23.170375+0000","last_became_peered":"2024-03-06T07:21:23.170375+0000","last_unstale":"2024-03-06T07:21:24.179184+0000","last_undegraded":"2024-03-06T07:21:24.179184+0000","last_fullsized":"2024-03-06T07:21:24.179184+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:48.584102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.209380+0000","last_change":"2024-03-06T07:21:22.174188+0000","last_active":"2024-03-06T07:21:25.209380+0000","last_peered":"2024-03-06T07:21:25.209380+0000","last_clean":"2024-03-06T07:21:25.209380+0000","last_became_active":"2024-03-06T07:21:22.173899+0000","last_became_peered":"2024-03-06T07:21:22.173899+0000","last_unstale":"2024-03-06T07:21:25.209380+0000","last_undegraded":"2024-03-06T07:21:25.209380+0000","last_fullsized":"2024-03-06T07:21:25.209380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:26.708237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165047+0000","last_change":"2024-03-06T07:21:23.169931+0000","last_active":"2024-03-06T07:21:24.165047+0000","last_peered":"2024-03-06T07:21:24.165047+0000","last_clean":"2024-03-06T07:21:24.165047+0000","last_became_active":"2024-03-06T07:21:23.169572+0000","last_became_peered":"2024-03-06T07:21:23.169572+0000","last_unstale":"2024-03-06T07:21:24.165047+0000","last_undegraded":"2024-03-06T07:21:24.165047+0000","last_fullsized":"2024-03-06T07:21:24.165047+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:06.313962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.163980+0000","last_change":"2024-03-06T07:21:22.164640+0000","last_active":"2024-03-06T07:21:24.163980+0000","last_peered":"2024-03-06T07:21:24.163980+0000","last_clean":"2024-03-06T07:21:24.163980+0000","last_became_active":"2024-03-06T07:21:22.164106+0000","last_became_peered":"2024-03-06T07:21:22.164106+0000","last_unstale":"2024-03-06T07:21:24.163980+0000","last_undegraded":"2024-03-06T07:21:24.163980+0000","last_fullsized":"2024-03-06T07:21:24.163980+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:31.123680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373206+0000","last_change":"2024-03-06T07:21:23.182404+0000","last_active":"2024-03-06T07:21:25.373206+0000","last_peered":"2024-03-06T07:21:25.373206+0000","last_clean":"2024-03-06T07:21:25.373206+0000","last_became_active":"2024-03-06T07:21:23.181761+0000","last_became_peered":"2024-03-06T07:21:23.181761+0000","last_unstale":"2024-03-06T07:21:25.373206+0000","last_undegraded":"2024-03-06T07:21:25.373206+0000","last_fullsized":"2024-03-06T07:21:25.373206+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:17.356744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179579+0000","last_change":"2024-03-06T07:21:22.175003+0000","last_active":"2024-03-06T07:21:24.179579+0000","last_peered":"2024-03-06T07:21:24.179579+0000","last_clean":"2024-03-06T07:21:24.179579+0000","last_became_active":"2024-03-06T07:21:22.172966+0000","last_became_peered":"2024-03-06T07:21:22.172966+0000","last_unstale":"2024-03-06T07:21:24.179579+0000","last_undegraded":"2024-03-06T07:21:24.179579+0000","last_fullsized":"2024-03-06T07:21:24.179579+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:06.624093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.209490+0000","last_change":"2024-03-06T07:21:23.171386+0000","last_active":"2024-03-06T07:21:25.209490+0000","last_peered":"2024-03-06T07:21:25.209490+0000","last_clean":"2024-03-06T07:21:25.209490+0000","last_became_active":"2024-03-06T07:21:23.170760+0000","last_became_peered":"2024-03-06T07:21:23.170760+0000","last_unstale":"2024-03-06T07:21:25.209490+0000","last_undegraded":"2024-03-06T07:21:25.209490+0000","last_fullsized":"2024-03-06T07:21:25.209490+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:05.754195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.178209+0000","last_change":"2024-03-06T07:21:22.161701+0000","last_active":"2024-03-06T07:21:24.178209+0000","last_peered":"2024-03-06T07:21:24.178209+0000","last_clean":"2024-03-06T07:21:24.178209+0000","last_became_active":"2024-03-06T07:21:22.161365+0000","last_became_peered":"2024-03-06T07:21:22.161365+0000","last_unstale":"2024-03-06T07:21:24.178209+0000","last_undegraded":"2024-03-06T07:21:24.178209+0000","last_fullsized":"2024-03-06T07:21:24.178209+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:35.557090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164786+0000","last_change":"2024-03-06T07:21:23.173494+0000","last_active":"2024-03-06T07:21:24.164786+0000","last_peered":"2024-03-06T07:21:24.164786+0000","last_clean":"2024-03-06T07:21:24.164786+0000","last_became_active":"2024-03-06T07:21:23.173209+0000","last_became_peered":"2024-03-06T07:21:23.173209+0000","last_unstale":"2024-03-06T07:21:24.164786+0000","last_undegraded":"2024-03-06T07:21:24.164786+0000","last_fullsized":"2024-03-06T07:21:24.164786+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:36.368758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373372+0000","last_change":"2024-03-06T07:21:22.171248+0000","last_active":"2024-03-06T07:21:25.373372+0000","last_peered":"2024-03-06T07:21:25.373372+0000","last_clean":"2024-03-06T07:21:25.373372+0000","last_became_active":"2024-03-06T07:21:22.170909+0000","last_became_peered":"2024-03-06T07:21:22.170909+0000","last_unstale":"2024-03-06T07:21:25.373372+0000","last_undegraded":"2024-03-06T07:21:25.373372+0000","last_fullsized":"2024-03-06T07:21:25.373372+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:40.673099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373086+0000","last_change":"2024-03-06T07:21:23.168323+0000","last_active":"2024-03-06T07:21:25.373086+0000","last_peered":"2024-03-06T07:21:25.373086+0000","last_clean":"2024-03-06T07:21:25.373086+0000","last_became_active":"2024-03-06T07:21:23.168024+0000","last_became_peered":"2024-03-06T07:21:23.168024+0000","last_unstale":"2024-03-06T07:21:25.373086+0000","last_undegraded":"2024-03-06T07:21:25.373086+0000","last_fullsized":"2024-03-06T07:21:25.373086+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:35.695974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373321+0000","last_change":"2024-03-06T07:21:22.174503+0000","last_active":"2024-03-06T07:21:25.373321+0000","last_peered":"2024-03-06T07:21:25.373321+0000","last_clean":"2024-03-06T07:21:25.373321+0000","last_became_active":"2024-03-06T07:21:22.174138+0000","last_became_peered":"2024-03-06T07:21:22.174138+0000","last_unstale":"2024-03-06T07:21:25.373321+0000","last_undegraded":"2024-03-06T07:21:25.373321+0000","last_fullsized":"2024-03-06T07:21:25.373321+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:26.708237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.373199+0000","last_change":"2024-03-06T07:21:23.170871+0000","last_active":"2024-03-06T07:21:25.373199+0000","last_peered":"2024-03-06T07:21:25.373199+0000","last_clean":"2024-03-06T07:21:25.373199+0000","last_became_active":"2024-03-06T07:21:23.170384+0000","last_became_peered":"2024-03-06T07:21:23.170384+0000","last_unstale":"2024-03-06T07:21:25.373199+0000","last_undegraded":"2024-03-06T07:21:25.373199+0000","last_fullsized":"2024-03-06T07:21:25.373199+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:19.113059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.165057+0000","last_change":"2024-03-06T07:21:22.169536+0000","last_active":"2024-03-06T07:21:24.165057+0000","last_peered":"2024-03-06T07:21:24.165057+0000","last_clean":"2024-03-06T07:21:24.165057+0000","last_became_active":"2024-03-06T07:21:22.169265+0000","last_became_peered":"2024-03-06T07:21:22.169265+0000","last_unstale":"2024-03-06T07:21:24.165057+0000","last_undegraded":"2024-03-06T07:21:24.165057+0000","last_fullsized":"2024-03-06T07:21:24.165057+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:49.464151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.282950+0000","last_change":"2024-03-06T07:21:23.170620+0000","last_active":"2024-03-06T07:21:24.282950+0000","last_peered":"2024-03-06T07:21:24.282950+0000","last_clean":"2024-03-06T07:21:24.282950+0000","last_became_active":"2024-03-06T07:21:23.170063+0000","last_became_peered":"2024-03-06T07:21:23.170063+0000","last_unstale":"2024-03-06T07:21:24.282950+0000","last_undegraded":"2024-03-06T07:21:24.282950+0000","last_fullsized":"2024-03-06T07:21:24.282950+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:35.695974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.282861+0000","last_change":"2024-03-06T07:21:22.172986+0000","last_active":"2024-03-06T07:21:24.282861+0000","last_peered":"2024-03-06T07:21:24.282861+0000","last_clean":"2024-03-06T07:21:24.282861+0000","last_became_active":"2024-03-06T07:21:22.172582+0000","last_became_peered":"2024-03-06T07:21:22.172582+0000","last_unstale":"2024-03-06T07:21:24.282861+0000","last_undegraded":"2024-03-06T07:21:24.282861+0000","last_fullsized":"2024-03-06T07:21:24.282861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:51.086001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.209675+0000","last_change":"2024-03-06T07:21:23.170353+0000","last_active":"2024-03-06T07:21:25.209675+0000","last_peered":"2024-03-06T07:21:25.209675+0000","last_clean":"2024-03-06T07:21:25.209675+0000","last_became_active":"2024-03-06T07:21:23.170078+0000","last_became_peered":"2024-03-06T07:21:23.170078+0000","last_unstale":"2024-03-06T07:21:25.209675+0000","last_undegraded":"2024-03-06T07:21:25.209675+0000","last_fullsized":"2024-03-06T07:21:25.209675+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:41.252208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.164817+0000","last_change":"2024-03-06T07:21:22.173498+0000","last_active":"2024-03-06T07:21:24.164817+0000","last_peered":"2024-03-06T07:21:24.164817+0000","last_clean":"2024-03-06T07:21:24.164817+0000","last_became_active":"2024-03-06T07:21:22.173198+0000","last_became_peered":"2024-03-06T07:21:22.173198+0000","last_unstale":"2024-03-06T07:21:24.164817+0000","last_undegraded":"2024-03-06T07:21:24.164817+0000","last_fullsized":"2024-03-06T07:21:24.164817+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:18.108199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:24.179450+0000","last_change":"2024-03-06T07:21:23.178114+0000","last_active":"2024-03-06T07:21:24.179450+0000","last_peered":"2024-03-06T07:21:24.179450+0000","last_clean":"2024-03-06T07:21:24.179450+0000","last_became_active":"2024-03-06T07:21:23.177700+0000","last_became_peered":"2024-03-06T07:21:23.177700+0000","last_unstale":"2024-03-06T07:21:24.179450+0000","last_undegraded":"2024-03-06T07:21:24.179450+0000","last_fullsized":"2024-03-06T07:21:24.179450+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:22.145657+0000","last_clean_scrub_stamp":"2024-03-06T07:21:22.145657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:48.917896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.209454+0000","last_change":"2024-03-06T07:21:22.172115+0000","last_active":"2024-03-06T07:21:25.209454+0000","last_peered":"2024-03-06T07:21:25.209454+0000","last_clean":"2024-03-06T07:21:25.209454+0000","last_became_active":"2024-03-06T07:21:22.171844+0000","last_became_peered":"2024-03-06T07:21:22.171844+0000","last_unstale":"2024-03-06T07:21:25.209454+0000","last_undegraded":"2024-03-06T07:21:25.209454+0000","last_fullsized":"2024-03-06T07:21:25.209454+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:21.140798+0000","last_clean_scrub_stamp":"2024-03-06T07:21:21.140798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:45.059780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":205,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:21:25.209288+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:21:25.209288+0000","last_peered":"2024-03-06T07:21:25.209288+0000","last_clean":"2024-03-06T07:21:25.209288+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:21:25.209288+0000","last_undegraded":"2024-03-06T07:21:25.209288+0000","last_fullsized":"2024-03-06T07:21:25.209288+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":532480,"data_stored":789215,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574865,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":197747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":458752,"data_stored":197277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":458752,"data_stored":197769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31644,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340196,"statfs":{"total":96636764160,"available":96604360704,"internally_reserved":0,"allocated":466944,"data_stored":198811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":454656,"data_stored":197235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":197255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31688,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340152,"statfs":{"total":96636764160,"available":96604315648,"internally_reserved":0,"allocated":520192,"data_stored":787567,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:30.953 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:21:30.953 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:21:31.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.102+0000 7f320c439640 1 Processor -- start 2024-03-06T07:21:31.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.102+0000 7f320c439640 1 -- start start 2024-03-06T07:21:31.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.103+0000 7f320c439640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3204152650 0x7f3204155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.103+0000 7f320c439640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32040c6b10 0x7f3204158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.103+0000 7f320c439640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32040c2fc0 con 0x7f32040bf9d0 2024-03-06T07:21:31.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.103+0000 7f320c439640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3204155800 con 0x7f32040c6b10 2024-03-06T07:21:31.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.103+0000 7f320c439640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32040c6690 con 0x7f3204152650 2024-03-06T07:21:31.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.103+0000 7f32099d4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32040c6b10 0x7f3204158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.103+0000 7f32099d4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32040c6b10 0x7f3204158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37096/0 (socket says 172.21.15.1:37096) 2024-03-06T07:21:31.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.103+0000 7f32099d4640 1 -- 172.21.15.1:0/902020560 learned_addr learned my addr 172.21.15.1:0/902020560 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:31.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.104+0000 7f320a1d5640 1 --2- 172.21.15.1:0/902020560 >> v2:172.21.15.98:3300/0 conn(0x7f3204152650 0x7f3204155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.104+0000 7f32099d4640 1 -- 172.21.15.1:0/902020560 >> v2:172.21.15.98:3300/0 conn(0x7f3204152650 msgr2=0x7f3204155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:31.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.104+0000 7f32099d4640 1 --2- 172.21.15.1:0/902020560 >> v2:172.21.15.98:3300/0 conn(0x7f3204152650 0x7f3204155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.104+0000 7f32099d4640 1 -- 172.21.15.1:0/902020560 >> v1:172.21.15.98:6789/0 conn(0x7f32040bf9d0 legacy=0x7f32040c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.104+0000 7f32099d4640 1 -- 172.21.15.1:0/902020560 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32040c7280 con 0x7f32040c6b10 2024-03-06T07:21:31.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.105+0000 7f32099d4640 1 --2- 172.21.15.1:0/902020560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32040c6b10 0x7f3204158690 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f32040cac90 tx=0x7f31f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1984d5ec59de37b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.105+0000 7f32091d3640 1 -- 172.21.15.1:0/902020560 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4035020 con 0x7f32040c6b10 2024-03-06T07:21:31.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.105+0000 7f32091d3640 1 -- 172.21.15.1:0/902020560 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f4031870 con 0x7f32040c6b10 2024-03-06T07:21:31.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.105+0000 7f32091d3640 1 -- 172.21.15.1:0/902020560 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4031c40 con 0x7f32040c6b10 2024-03-06T07:21:31.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.106+0000 7f320c439640 1 -- 172.21.15.1:0/902020560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32040c6b10 msgr2=0x7f3204158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.106+0000 7f320c439640 1 --2- 172.21.15.1:0/902020560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32040c6b10 0x7f3204158690 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f32040cac90 tx=0x7f31f402b910 comp rx=0 tx=0).stop 2024-03-06T07:21:31.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.106+0000 7f320c439640 1 -- 172.21.15.1:0/902020560 shutdown_connections 2024-03-06T07:21:31.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.106+0000 7f320c439640 1 --2- 172.21.15.1:0/902020560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32040c6b10 0x7f3204158690 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.106+0000 7f320c439640 1 --2- 172.21.15.1:0/902020560 >> v2:172.21.15.98:3300/0 conn(0x7f3204152650 0x7f3204155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.106+0000 7f320c439640 1 -- 172.21.15.1:0/902020560 >> 172.21.15.1:0/902020560 conn(0x7f3204066ee0 msgr2=0x7f32040672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.106+0000 7f320c439640 1 -- 172.21.15.1:0/902020560 shutdown_connections 2024-03-06T07:21:31.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.106+0000 7f320c439640 1 -- 172.21.15.1:0/902020560 wait complete. 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.107+0000 7f320c439640 1 Processor -- start 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.107+0000 7f320c439640 1 -- start start 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f320c439640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32040bf9d0 0x7f32041eada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f320c439640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3204152650 0x7f32041431f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f320c439640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3204144df0 0x7f32041e1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f320c439640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3204155490 con 0x7f32040bf9d0 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f320c439640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32041551b0 con 0x7f3204152650 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f320c439640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3204152a30 con 0x7f3204144df0 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f320a1d5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32040bf9d0 0x7f32041eada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f320a1d5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32040bf9d0 0x7f32041eada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37106/0 (socket says 172.21.15.1:37106) 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f320a1d5640 1 -- 172.21.15.1:0/2238698271 learned_addr learned my addr 172.21.15.1:0/2238698271 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f32099d4640 1 --2- 172.21.15.1:0/2238698271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3204152650 0x7f32041431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f320a9d6640 1 --2- 172.21.15.1:0/2238698271 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3204144df0 0x7f32041e1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f320a1d5640 1 -- 172.21.15.1:0/2238698271 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3204144df0 msgr2=0x7f32041e1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f320a1d5640 1 --2- 172.21.15.1:0/2238698271 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3204144df0 0x7f32041e1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f320a1d5640 1 -- 172.21.15.1:0/2238698271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3204152650 msgr2=0x7f32041431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f320a1d5640 1 --2- 172.21.15.1:0/2238698271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3204152650 0x7f32041431f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f320a1d5640 1 -- 172.21.15.1:0/2238698271 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3204142960 con 0x7f32040bf9d0 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f320a1d5640 1 --2- 172.21.15.1:0/2238698271 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32040bf9d0 0x7f32041eada0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f320000d830 tx=0x7f320000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f31faffd640 1 -- 172.21.15.1:0/2238698271 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320000b6c0 con 0x7f32040bf9d0 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f31faffd640 1 -- 172.21.15.1:0/2238698271 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3200019070 con 0x7f32040bf9d0 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f320c439640 1 -- 172.21.15.1:0/2238698271 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32040c9fe0 con 0x7f32040bf9d0 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.108+0000 7f31faffd640 1 -- 172.21.15.1:0/2238698271 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320000ba80 con 0x7f32040bf9d0 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.109+0000 7f320c439640 1 -- 172.21.15.1:0/2238698271 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32041ef1a0 con 0x7f32040bf9d0 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.110+0000 7f320c439640 1 -- 172.21.15.1:0/2238698271 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3204067fd0 con 0x7f32040bf9d0 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.110+0000 7f31faffd640 1 -- 172.21.15.1:0/2238698271 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f320001e460 con 0x7f32040bf9d0 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.110+0000 7f31faffd640 1 --2- 172.21.15.1:0/2238698271 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f31e806b4d0 0x7f31e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.110+0000 7f31faffd640 1 -- 172.21.15.1:0/2238698271 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f320008a920 con 0x7f32040bf9d0 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.111+0000 7f32099d4640 1 --2- 172.21.15.1:0/2238698271 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f31e806b4d0 0x7f31e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.111+0000 7f32099d4640 1 --2- 172.21.15.1:0/2238698271 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f31e806b4d0 0x7f31e806d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f31f4036000 tx=0x7f31f403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.115+0000 7f31faffd640 1 -- 172.21.15.1:0/2238698271 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32000596a0 con 0x7f32040bf9d0 2024-03-06T07:21:31.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.372+0000 7f320c439640 1 -- 172.21.15.1:0/2238698271 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f32040b0950 con 0x7f32040bf9d0 2024-03-06T07:21:31.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.374+0000 7f31faffd640 1 -- 172.21.15.1:0/2238698271 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3200058fa0 con 0x7f32040bf9d0 2024-03-06T07:21:31.376 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-06T07:21:31.377 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng== 2024-03-06T07:21:31.377 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-06T07:21:31.377 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-06T07:21:31.377 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:21:31.377 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-06T07:21:31.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.376+0000 7f320c439640 1 -- 172.21.15.1:0/2238698271 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f31e806b4d0 msgr2=0x7f31e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.376+0000 7f320c439640 1 --2- 172.21.15.1:0/2238698271 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f31e806b4d0 0x7f31e806d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f31f4036000 tx=0x7f31f403c040 comp rx=0 tx=0).stop 2024-03-06T07:21:31.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.377+0000 7f320c439640 1 -- 172.21.15.1:0/2238698271 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32040bf9d0 msgr2=0x7f32041eada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.377+0000 7f320c439640 1 --2- 172.21.15.1:0/2238698271 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32040bf9d0 0x7f32041eada0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f320000d830 tx=0x7f320000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.377+0000 7f320c439640 1 -- 172.21.15.1:0/2238698271 shutdown_connections 2024-03-06T07:21:31.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.377+0000 7f320c439640 1 --2- 172.21.15.1:0/2238698271 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f31e806b4d0 0x7f31e806d990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.377+0000 7f320c439640 1 --2- 172.21.15.1:0/2238698271 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3204144df0 0x7f32041e1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.377+0000 7f320c439640 1 --2- 172.21.15.1:0/2238698271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3204152650 0x7f32041431f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.377+0000 7f320c439640 1 --2- 172.21.15.1:0/2238698271 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32040bf9d0 0x7f32041eada0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.377+0000 7f320c439640 1 -- 172.21.15.1:0/2238698271 >> 172.21.15.1:0/2238698271 conn(0x7f3204066ee0 msgr2=0x7f32040c3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.377+0000 7f320c439640 1 -- 172.21.15.1:0/2238698271 shutdown_connections 2024-03-06T07:21:31.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.377+0000 7f320c439640 1 -- 172.21.15.1:0/2238698271 wait complete. 2024-03-06T07:21:31.390 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:31.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.522+0000 7f6e82b65640 1 Processor -- start 2024-03-06T07:21:31.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.522+0000 7f6e82b65640 1 -- start start 2024-03-06T07:21:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e82b65640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6e7c152840 0x7f6e7c0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e82b65640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e7c0c60b0 0x7f6e7c1553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e82b65640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e7c0c5e40 con 0x7f6e7c0c3970 2024-03-06T07:21:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e82b65640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e7c0c67f0 con 0x7f6e7c152840 2024-03-06T07:21:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e82b65640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c0c6a40 con 0x7f6e7c0c60b0 2024-03-06T07:21:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e80901640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6e7c0c3970 0x7f6e7c152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60798/0 (socket says 172.21.15.1:60798) 2024-03-06T07:21:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e80901640 1 -- 172.21.15.1:0/3306706222 learned_addr learned my addr 172.21.15.1:0/3306706222 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e81102640 1 --2- 172.21.15.1:0/3306706222 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e7c0c60b0 0x7f6e7c1553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e7b7fe640 1 -- 172.21.15.1:0/3306706222 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948793186 0 0) 0x7f6e7c0c5e40 con 0x7f6e7c0c3970 2024-03-06T07:21:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e7bfff640 1 --2- 172.21.15.1:0/3306706222 >> v2:172.21.15.98:3300/0 conn(0x7f6e7c152840 0x7f6e7c0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e7b7fe640 1 -- 172.21.15.1:0/3306706222 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e6c003520 con 0x7f6e7c0c3970 2024-03-06T07:21:31.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e7b7fe640 1 -- 172.21.15.1:0/3306706222 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4146069127 0 0) 0x7f6e6c003520 con 0x7f6e7c0c3970 2024-03-06T07:21:31.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e7b7fe640 1 -- 172.21.15.1:0/3306706222 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e7c0c60b0 msgr2=0x7f6e7c1553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:31.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e7b7fe640 1 --2- 172.21.15.1:0/3306706222 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e7c0c60b0 0x7f6e7c1553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e7b7fe640 1 -- 172.21.15.1:0/3306706222 >> v2:172.21.15.98:3300/0 conn(0x7f6e7c152840 msgr2=0x7f6e7c0c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:31.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e7b7fe640 1 --2- 172.21.15.1:0/3306706222 >> v2:172.21.15.98:3300/0 conn(0x7f6e7c152840 0x7f6e7c0c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e7b7fe640 1 -- 172.21.15.1:0/3306706222 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7c0c6d20 con 0x7f6e7c0c3970 2024-03-06T07:21:31.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e7b7fe640 1 -- 172.21.15.1:0/3306706222 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f6e70001670 con 0x7f6e7c0c3970 2024-03-06T07:21:31.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e7b7fe640 1 -- 172.21.15.1:0/3306706222 >> v1:172.21.15.1:6789/0 conn(0x7f6e7c0c3970 legacy=0x7f6e7c152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e7b7fe640 1 --2- 172.21.15.1:0/3306706222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e6c0039b0 0x7f6e6c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e7b7fe640 1 --2- 172.21.15.1:0/3306706222 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e7c0c60b0 0x7f6e6c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e7b7fe640 1 --2- 172.21.15.1:0/3306706222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e6c004d40 0x7f6e6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e7b7fe640 1 -- 172.21.15.1:0/3306706222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c0c67f0 con 0x7f6e6c004d40 2024-03-06T07:21:31.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e7b7fe640 1 -- 172.21.15.1:0/3306706222 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c0c6a40 con 0x7f6e6c0039b0 2024-03-06T07:21:31.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.523+0000 7f6e7b7fe640 1 -- 172.21.15.1:0/3306706222 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c00b9c0 con 0x7f6e7c0c60b0 2024-03-06T07:21:31.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.524+0000 7f6e81102640 1 --2- 172.21.15.1:0/3306706222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e6c004d40 0x7f6e6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.524+0000 7f6e80901640 1 --2- 172.21.15.1:0/3306706222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e6c0039b0 0x7f6e6c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.524+0000 7f6e7bfff640 1 --2- 172.21.15.1:0/3306706222 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e7c0c60b0 0x7f6e6c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.524+0000 7f6e81102640 1 -- 172.21.15.1:0/3306706222 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e7c0c60b0 msgr2=0x7f6e6c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.524+0000 7f6e81102640 1 --2- 172.21.15.1:0/3306706222 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e7c0c60b0 0x7f6e6c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.524+0000 7f6e81102640 1 -- 172.21.15.1:0/3306706222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e6c0039b0 msgr2=0x7f6e6c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.524+0000 7f6e81102640 1 --2- 172.21.15.1:0/3306706222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e6c0039b0 0x7f6e6c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.524+0000 7f6e81102640 1 -- 172.21.15.1:0/3306706222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c003520 con 0x7f6e6c004d40 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.525+0000 7f6e81102640 1 --2- 172.21.15.1:0/3306706222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e6c004d40 0x7f6e6c0051c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c06e2e0 tx=0x7f6e6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=877a17c32375fde0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.525+0000 7f6e7b7fe640 1 -- 172.21.15.1:0/3306706222 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74039070 con 0x7f6e6c004d40 2024-03-06T07:21:31.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.525+0000 7f6e7b7fe640 1 -- 172.21.15.1:0/3306706222 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7402cb90 con 0x7f6e6c004d40 2024-03-06T07:21:31.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.525+0000 7f6e7b7fe640 1 -- 172.21.15.1:0/3306706222 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74009c10 con 0x7f6e6c004d40 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.526+0000 7f6e82b65640 1 -- 172.21.15.1:0/3306706222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e6c004d40 msgr2=0x7f6e6c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.526+0000 7f6e82b65640 1 --2- 172.21.15.1:0/3306706222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e6c004d40 0x7f6e6c0051c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c06e2e0 tx=0x7f6e6c005830 comp rx=0 tx=0).stop 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.526+0000 7f6e81102640 1 -- 172.21.15.1:0/3306706222 reap_dead start 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.526+0000 7f6e82b65640 1 -- 172.21.15.1:0/3306706222 shutdown_connections 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.526+0000 7f6e82b65640 1 -- 172.21.15.1:0/3306706222 >> 172.21.15.1:0/3306706222 conn(0x7f6e7c0c88f0 msgr2=0x7f6e7c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.526+0000 7f6e82b65640 1 -- 172.21.15.1:0/3306706222 shutdown_connections 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.526+0000 7f6e82b65640 1 -- 172.21.15.1:0/3306706222 wait complete. 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.527+0000 7f6e82b65640 1 Processor -- start 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.527+0000 7f6e82b65640 1 -- start start 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e82b65640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e7c1573f0 0x7f6e7c1567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e82b65640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e7c1eba10 0x7f6e7c0a9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e82b65640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e7c0ace10 0x7f6e7c13ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e82b65640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c0c75c0 con 0x7f6e7c0ace10 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e82b65640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c0c72e0 con 0x7f6e7c1573f0 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e82b65640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c0c3d50 con 0x7f6e7c1eba10 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e81102640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e7c0ace10 0x7f6e7c13ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e81102640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e7c0ace10 0x7f6e7c13ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37122/0 (socket says 172.21.15.1:37122) 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e81102640 1 -- 172.21.15.1:0/1638608770 learned_addr learned my addr 172.21.15.1:0/1638608770 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e7bfff640 1 --2- 172.21.15.1:0/1638608770 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e7c1eba10 0x7f6e7c0a9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e80901640 1 --2- 172.21.15.1:0/1638608770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e7c1573f0 0x7f6e7c1567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e81102640 1 -- 172.21.15.1:0/1638608770 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e7c1eba10 msgr2=0x7f6e7c0a9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e81102640 1 --2- 172.21.15.1:0/1638608770 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e7c1eba10 0x7f6e7c0a9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e81102640 1 -- 172.21.15.1:0/1638608770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e7c1573f0 msgr2=0x7f6e7c1567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e81102640 1 --2- 172.21.15.1:0/1638608770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e7c1573f0 0x7f6e7c1567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e81102640 1 -- 172.21.15.1:0/1638608770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7c0a9e60 con 0x7f6e7c0ace10 2024-03-06T07:21:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e7bfff640 1 --2- 172.21.15.1:0/1638608770 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e7c1eba10 0x7f6e7c0a9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.528+0000 7f6e81102640 1 --2- 172.21.15.1:0/1638608770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e7c0ace10 0x7f6e7c13ef90 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6e7402c600 tx=0x7f6e74009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.529+0000 7f6e797fa640 1 -- 172.21.15.1:0/1638608770 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74039070 con 0x7f6e7c0ace10 2024-03-06T07:21:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.529+0000 7f6e797fa640 1 -- 172.21.15.1:0/1638608770 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e74044070 con 0x7f6e7c0ace10 2024-03-06T07:21:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.529+0000 7f6e82b65640 1 -- 172.21.15.1:0/1638608770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e7c155cc0 con 0x7f6e7c0ace10 2024-03-06T07:21:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.529+0000 7f6e797fa640 1 -- 172.21.15.1:0/1638608770 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74009c10 con 0x7f6e7c0ace10 2024-03-06T07:21:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.529+0000 7f6e82b65640 1 -- 172.21.15.1:0/1638608770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7c156280 con 0x7f6e7c0ace10 2024-03-06T07:21:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.530+0000 7f6e82b65640 1 -- 172.21.15.1:0/1638608770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e7c0686e0 con 0x7f6e7c0ace10 2024-03-06T07:21:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.530+0000 7f6e797fa640 1 -- 172.21.15.1:0/1638608770 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e74005d10 con 0x7f6e7c0ace10 2024-03-06T07:21:31.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.531+0000 7f6e797fa640 1 --2- 172.21.15.1:0/1638608770 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e6006b500 0x7f6e6006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.531+0000 7f6e797fa640 1 -- 172.21.15.1:0/1638608770 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6e740ac2b0 con 0x7f6e7c0ace10 2024-03-06T07:21:31.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.531+0000 7f6e80901640 1 --2- 172.21.15.1:0/1638608770 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e6006b500 0x7f6e6006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.532+0000 7f6e80901640 1 --2- 172.21.15.1:0/1638608770 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e6006b500 0x7f6e6006d9c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6e70007fd0 tx=0x7f6e700078f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.535+0000 7f6e797fa640 1 -- 172.21.15.1:0/1638608770 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e7404d020 con 0x7f6e7c0ace10 2024-03-06T07:21:31.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.784+0000 7f6e82b65640 1 -- 172.21.15.1:0/1638608770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e7c156450 con 0x7f6e7c0ace10 2024-03-06T07:21:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.785+0000 7f6e797fa640 1 -- 172.21.15.1:0/1638608770 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f6e7407b0f0 con 0x7f6e7c0ace10 2024-03-06T07:21:31.787 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:31.787 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4781,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2438105345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2438105345},{"type":"v1","addr":"172.21.15.1:6835","nonce":2438105345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3354292028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3354292028},{"type":"v1","addr":"172.21.15.98:6836","nonce":3354292028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4975,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3948803645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3948803645},{"type":"v1","addr":"172.21.15.98:6837","nonce":3948803645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:22.528539+0000","modified":"2024-03-06T07:21:24.518239+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1818141892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1818141892},{"type":"v1","addr":"172.21.15.1:6834","nonce":1818141892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":2}]} 2024-03-06T07:21:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 16 2024-03-06T07:21:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.787+0000 7f6e82b65640 1 -- 172.21.15.1:0/1638608770 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e6006b500 msgr2=0x7f6e6006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.787+0000 7f6e82b65640 1 --2- 172.21.15.1:0/1638608770 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e6006b500 0x7f6e6006d9c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6e70007fd0 tx=0x7f6e700078f0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.787+0000 7f6e82b65640 1 -- 172.21.15.1:0/1638608770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e7c0ace10 msgr2=0x7f6e7c13ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.788+0000 7f6e82b65640 1 --2- 172.21.15.1:0/1638608770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e7c0ace10 0x7f6e7c13ef90 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6e7402c600 tx=0x7f6e74009750 comp rx=0 tx=0).stop 2024-03-06T07:21:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.788+0000 7f6e82b65640 1 -- 172.21.15.1:0/1638608770 shutdown_connections 2024-03-06T07:21:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.788+0000 7f6e82b65640 1 --2- 172.21.15.1:0/1638608770 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e6006b500 0x7f6e6006d9c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.788+0000 7f6e82b65640 1 --2- 172.21.15.1:0/1638608770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e7c0ace10 0x7f6e7c13ef90 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.788+0000 7f6e82b65640 1 --2- 172.21.15.1:0/1638608770 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e7c1eba10 0x7f6e7c0a9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.788+0000 7f6e82b65640 1 --2- 172.21.15.1:0/1638608770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e7c1573f0 0x7f6e7c1567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.788+0000 7f6e82b65640 1 -- 172.21.15.1:0/1638608770 >> 172.21.15.1:0/1638608770 conn(0x7f6e7c0c88f0 msgr2=0x7f6e7c1546b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.788+0000 7f6e82b65640 1 -- 172.21.15.1:0/1638608770 shutdown_connections 2024-03-06T07:21:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:31.788+0000 7f6e82b65640 1 -- 172.21.15.1:0/1638608770 wait complete. 2024-03-06T07:21:32.203 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:21:22.528539+0000', 'modified': '2024-03-06T07:21:24.518239+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5280}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5280': {'gid': 5280, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6834/1818141892', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1818141892}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1818141892}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5280, 'qdb_cluster': [5280]} 2024-03-06T07:21:32.204 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:21:32.204 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:21:32.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.317+0000 7f635124e640 1 Processor -- start 2024-03-06T07:21:32.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.317+0000 7f635124e640 1 -- start start 2024-03-06T07:21:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.318+0000 7f635124e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f634c0c3970 0x7f634c0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.318+0000 7f635124e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f634c0c5e40 con 0x7f634c152840 2024-03-06T07:21:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.318+0000 7f635124e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f634c0c6a40 con 0x7f634c0c60b0 2024-03-06T07:21:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.318+0000 7f635124e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f634c0c67f0 con 0x7f634c0c3970 2024-03-06T07:21:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.318+0000 7f634b59e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f634c152840 0x7f634c1553d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60814/0 (socket says 172.21.15.1:60814) 2024-03-06T07:21:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.318+0000 7f634b59e640 1 -- 172.21.15.1:0/3154917847 learned_addr learned my addr 172.21.15.1:0/3154917847 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.318+0000 7f634ad9d640 1 --2- 172.21.15.1:0/3154917847 >> v2:172.21.15.98:3300/0 conn(0x7f634c0c3970 0x7f634c0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.318+0000 7f6349d9b640 1 -- 172.21.15.1:0/3154917847 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584264804 0 0) 0x7f634c0c5e40 con 0x7f634c152840 2024-03-06T07:21:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.318+0000 7f6349d9b640 1 -- 172.21.15.1:0/3154917847 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6334003520 con 0x7f634c152840 2024-03-06T07:21:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.319+0000 7f6349d9b640 1 -- 172.21.15.1:0/3154917847 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3518260834 0 0) 0x7f6334003520 con 0x7f634c152840 2024-03-06T07:21:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.319+0000 7f634ad9d640 1 -- 172.21.15.1:0/3154917847 >> v1:172.21.15.98:6789/0 conn(0x7f634c0c60b0 legacy=0x7f634c152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.319+0000 7f634ad9d640 1 -- 172.21.15.1:0/3154917847 >> v1:172.21.15.1:6789/0 conn(0x7f634c152840 legacy=0x7f634c1553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.319+0000 7f634ad9d640 1 -- 172.21.15.1:0/3154917847 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634c0c6d20 con 0x7f634c0c3970 2024-03-06T07:21:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.319+0000 7f634ad9d640 1 --2- 172.21.15.1:0/3154917847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f634c0c3970 0x7f634c0c8170 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f633c0097a0 tx=0x7f633c02b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b4df18fe7290c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.320+0000 7f6349d9b640 1 -- 172.21.15.1:0/3154917847 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c037070 con 0x7f634c0c3970 2024-03-06T07:21:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.320+0000 7f6349d9b640 1 -- 172.21.15.1:0/3154917847 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f633c030510 con 0x7f634c0c3970 2024-03-06T07:21:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.320+0000 7f6349d9b640 1 -- 172.21.15.1:0/3154917847 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63440019f0 con 0x7f634c0c3970 2024-03-06T07:21:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.320+0000 7f635124e640 1 -- 172.21.15.1:0/3154917847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f634c0c3970 msgr2=0x7f634c0c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.320+0000 7f635124e640 1 --2- 172.21.15.1:0/3154917847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f634c0c3970 0x7f634c0c8170 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f633c0097a0 tx=0x7f633c02b680 comp rx=0 tx=0).stop 2024-03-06T07:21:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.320+0000 7f635124e640 1 -- 172.21.15.1:0/3154917847 shutdown_connections 2024-03-06T07:21:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.320+0000 7f635124e640 1 --2- 172.21.15.1:0/3154917847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f634c0c3970 0x7f634c0c8170 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.320+0000 7f635124e640 1 -- 172.21.15.1:0/3154917847 >> 172.21.15.1:0/3154917847 conn(0x7f634c0c88f0 msgr2=0x7f634c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.320+0000 7f635124e640 1 -- 172.21.15.1:0/3154917847 shutdown_connections 2024-03-06T07:21:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.320+0000 7f635124e640 1 -- 172.21.15.1:0/3154917847 wait complete. 2024-03-06T07:21:32.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.322+0000 7f635124e640 1 Processor -- start 2024-03-06T07:21:32.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.322+0000 7f635124e640 1 -- start start 2024-03-06T07:21:32.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.322+0000 7f635124e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f634c0c3970 0x7f634c0ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.322+0000 7f635124e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f634c0c60b0 0x7f634c1ed9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.322+0000 7f635124e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f634c152840 0x7f634c150640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.322+0000 7f635124e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c0c75c0 con 0x7f634c152840 2024-03-06T07:21:32.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.322+0000 7f635124e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c0c72e0 con 0x7f634c0c60b0 2024-03-06T07:21:32.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.322+0000 7f635124e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634c0c3d50 con 0x7f634c0c3970 2024-03-06T07:21:32.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.322+0000 7f634b59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f634c152840 0x7f634c150640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.322+0000 7f634b59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f634c152840 0x7f634c150640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37136/0 (socket says 172.21.15.1:37136) 2024-03-06T07:21:32.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.322+0000 7f634b59e640 1 -- 172.21.15.1:0/507778878 learned_addr learned my addr 172.21.15.1:0/507778878 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:32.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.323+0000 7f634a59c640 1 --2- 172.21.15.1:0/507778878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f634c0c60b0 0x7f634c1ed9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.323+0000 7f634ad9d640 1 --2- 172.21.15.1:0/507778878 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f634c0c3970 0x7f634c0ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.323+0000 7f634b59e640 1 -- 172.21.15.1:0/507778878 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f634c0c3970 msgr2=0x7f634c0ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.323+0000 7f634b59e640 1 --2- 172.21.15.1:0/507778878 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f634c0c3970 0x7f634c0ace80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.323+0000 7f634b59e640 1 -- 172.21.15.1:0/507778878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f634c0c60b0 msgr2=0x7f634c1ed9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.323+0000 7f634b59e640 1 --2- 172.21.15.1:0/507778878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f634c0c60b0 0x7f634c1ed9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.323+0000 7f634b59e640 1 -- 172.21.15.1:0/507778878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634c0ac0e0 con 0x7f634c152840 2024-03-06T07:21:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.323+0000 7f634a59c640 1 --2- 172.21.15.1:0/507778878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f634c0c60b0 0x7f634c1ed9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.323+0000 7f634b59e640 1 --2- 172.21.15.1:0/507778878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f634c152840 0x7f634c150640 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f634400c840 tx=0x7f634400cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.323+0000 7f63337fe640 1 -- 172.21.15.1:0/507778878 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6344018070 con 0x7f634c152840 2024-03-06T07:21:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.323+0000 7f63337fe640 1 -- 172.21.15.1:0/507778878 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f634400a640 con 0x7f634c152840 2024-03-06T07:21:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.323+0000 7f635124e640 1 -- 172.21.15.1:0/507778878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f634c0aded0 con 0x7f634c152840 2024-03-06T07:21:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.323+0000 7f63337fe640 1 -- 172.21.15.1:0/507778878 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634400aa30 con 0x7f634c152840 2024-03-06T07:21:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.323+0000 7f635124e640 1 -- 172.21.15.1:0/507778878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f634c0ad3c0 con 0x7f634c152840 2024-03-06T07:21:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.325+0000 7f63337fe640 1 -- 172.21.15.1:0/507778878 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f634400abd0 con 0x7f634c152840 2024-03-06T07:21:32.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.325+0000 7f63337fe640 1 --2- 172.21.15.1:0/507778878 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f633806b4d0 0x7f633806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.325+0000 7f63337fe640 1 -- 172.21.15.1:0/507778878 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f63440899b0 con 0x7f634c152840 2024-03-06T07:21:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.326+0000 7f634ad9d640 1 --2- 172.21.15.1:0/507778878 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f633806b4d0 0x7f633806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.326+0000 7f6353e37740 1 Processor -- start 2024-03-06T07:21:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.326+0000 7f634ad9d640 1 --2- 172.21.15.1:0/507778878 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f633806b4d0 0x7f633806d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f633c034000 tx=0x7f633c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.326+0000 7f6353e37740 1 -- start start 2024-03-06T07:21:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.327+0000 7f6353e37740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd8f50 0x557148bd6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.327+0000 7f6353e37740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd9330 0x557148bd6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.327+0000 7f6353e37740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bd74b0 0x557148bd3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.327+0000 7f6353e37740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x557148b3d6b0 con 0x557148bd8f50 2024-03-06T07:21:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.327+0000 7f6353e37740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x557148ae6a60 con 0x557148bd74b0 2024-03-06T07:21:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.327+0000 7f6353e37740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x557148b39750 con 0x557148bd9330 2024-03-06T07:21:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.327+0000 7f634ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd8f50 0x557148bd6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.327+0000 7f634ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd8f50 0x557148bd6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37144/0 (socket says 172.21.15.1:37144) 2024-03-06T07:21:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.327+0000 7f634ad9d640 1 -- 172.21.15.1:0/1690575010 learned_addr learned my addr 172.21.15.1:0/1690575010 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.327+0000 7f634a59c640 1 --2- 172.21.15.1:0/1690575010 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd9330 0x557148bd6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.327+0000 7f634b59e640 1 --2- 172.21.15.1:0/1690575010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bd74b0 0x557148bd3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.327+0000 7f634ad9d640 1 -- 172.21.15.1:0/1690575010 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd9330 msgr2=0x557148bd6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.327+0000 7f634ad9d640 1 --2- 172.21.15.1:0/1690575010 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd9330 0x557148bd6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.327+0000 7f634ad9d640 1 -- 172.21.15.1:0/1690575010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bd74b0 msgr2=0x557148bd3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.327+0000 7f634ad9d640 1 --2- 172.21.15.1:0/1690575010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bd74b0 0x557148bd3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.327+0000 7f634ad9d640 1 -- 172.21.15.1:0/1690575010 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557148b04f30 con 0x557148bd8f50 2024-03-06T07:21:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.328+0000 7f634ad9d640 1 --2- 172.21.15.1:0/1690575010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd8f50 0x557148bd6a30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f633c0098c0 tx=0x7f633c02f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c95aefa43ced906f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.328+0000 7f6330ff9640 1 -- 172.21.15.1:0/1690575010 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c04c070 con 0x557148bd8f50 2024-03-06T07:21:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.328+0000 7f6330ff9640 1 -- 172.21.15.1:0/1690575010 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f633c051460 con 0x557148bd8f50 2024-03-06T07:21:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.328+0000 7f6330ff9640 1 -- 172.21.15.1:0/1690575010 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c037040 con 0x557148bd8f50 2024-03-06T07:21:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.328+0000 7f6353e37740 1 -- 172.21.15.1:0/1690575010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd8f50 msgr2=0x557148bd6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.328+0000 7f6353e37740 1 --2- 172.21.15.1:0/1690575010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd8f50 0x557148bd6a30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f633c0098c0 tx=0x7f633c02f000 comp rx=0 tx=0).stop 2024-03-06T07:21:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.328+0000 7f6353e37740 1 -- 172.21.15.1:0/1690575010 shutdown_connections 2024-03-06T07:21:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.328+0000 7f6353e37740 1 --2- 172.21.15.1:0/1690575010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bd74b0 0x557148bd3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.329+0000 7f6353e37740 1 --2- 172.21.15.1:0/1690575010 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd9330 0x557148bd6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.329+0000 7f6353e37740 1 --2- 172.21.15.1:0/1690575010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd8f50 0x557148bd6a30 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.329+0000 7f6353e37740 1 -- 172.21.15.1:0/1690575010 >> 172.21.15.1:0/1690575010 conn(0x557148bd57d0 msgr2=0x557148bfa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.329+0000 7f6353e37740 1 -- 172.21.15.1:0/1690575010 shutdown_connections 2024-03-06T07:21:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.329+0000 7f6353e37740 1 -- 172.21.15.1:0/1690575010 wait complete. 2024-03-06T07:21:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.329+0000 7f6353e37740 1 -- start start 2024-03-06T07:21:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.329+0000 7f6353e37740 1 Processor -- start 2024-03-06T07:21:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.330+0000 7f6353e37740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd8f50 0x557148bfc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.330+0000 7f6353e37740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bd9330 0x557148bf91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.330+0000 7f6353e37740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bffe00 0x557148bf4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.330+0000 7f6353e37740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x557148bfb230 con 0x557148bffe00 2024-03-06T07:21:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.330+0000 7f6353e37740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x557148af9ea0 con 0x557148bd9330 2024-03-06T07:21:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.330+0000 7f6353e37740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x557148bfac70 con 0x557148bd8f50 2024-03-06T07:21:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.330+0000 7f6353e37740 10 client.0 _ll_register_callbacks cb 0x557148af9870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:21:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.330+0000 7f634b59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bffe00 0x557148bf4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.330+0000 7f634b59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bffe00 0x557148bf4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37146/0 (socket says 172.21.15.1:37146) 2024-03-06T07:21:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.330+0000 7f634b59e640 1 -- 172.21.15.1:0/1223782866 learned_addr learned my addr 172.21.15.1:0/1223782866 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.330+0000 7f634a59c640 1 --2- 172.21.15.1:0/1223782866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bd9330 0x557148bf91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.330+0000 7f634ad9d640 1 --2- 172.21.15.1:0/1223782866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd8f50 0x557148bfc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.330+0000 7f634b59e640 1 -- 172.21.15.1:0/1223782866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd8f50 msgr2=0x557148bfc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.330+0000 7f634b59e640 1 --2- 172.21.15.1:0/1223782866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd8f50 0x557148bfc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.330+0000 7f634b59e640 1 -- 172.21.15.1:0/1223782866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bd9330 msgr2=0x557148bf91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.330+0000 7f634b59e640 1 --2- 172.21.15.1:0/1223782866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bd9330 0x557148bf91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.330+0000 7f634b59e640 1 -- 172.21.15.1:0/1223782866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557148afa270 con 0x557148bffe00 2024-03-06T07:21:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.331+0000 7f634b59e640 1 --2- 172.21.15.1:0/1223782866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bffe00 0x557148bf4d20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f6344060190 tx=0x7f6344057a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0d8099cab609f53 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.331+0000 7f6329ffb640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:21:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.331+0000 7f6329ffb640 1 -- 172.21.15.1:0/1223782866 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6344089050 con 0x557148bffe00 2024-03-06T07:21:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.331+0000 7f6329ffb640 1 -- 172.21.15.1:0/1223782866 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6344057400 con 0x557148bffe00 2024-03-06T07:21:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.331+0000 7f6353e37740 1 -- 172.21.15.1:0/1223782866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f633c051460 con 0x557148bffe00 2024-03-06T07:21:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.331+0000 7f6329ffb640 1 -- 172.21.15.1:0/1223782866 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f634405c100 con 0x557148bffe00 2024-03-06T07:21:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.331+0000 7f6353e37740 10 client.5487 fetch_fsmap learned FSMap version 16 2024-03-06T07:21:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.331+0000 7f6353e37740 1 -- 172.21.15.1:0/1223782866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x5571488522d0 con 0x557148bffe00 2024-03-06T07:21:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.331+0000 7f6329ffb640 1 -- 172.21.15.1:0/1223782866 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f633c051460 con 0x557148bffe00 2024-03-06T07:21:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.331+0000 7f6329ffb640 1 -- 172.21.15.1:0/1223782866 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 16) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f634405e090 con 0x557148bffe00 2024-03-06T07:21:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.331+0000 7f6353e37740 10 client.5487 fetch_fsmap finished waiting for FSMap version 16 2024-03-06T07:21:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.331+0000 7f6353e37740 10 client.5487 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:21:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.331+0000 7f6353e37740 20 client.5487 populate_metadata read hostname 'smithi001' 2024-03-06T07:21:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.331+0000 7f6353e37740 1 --2- 172.21.15.1:0/1223782866 >> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] conn(0x557148bf53e0 0x557148c262f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.331+0000 7f6353e37740 4 client.5487 mds_command: new command op to 5280 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:21:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.331+0000 7f6353e37740 1 -- 172.21.15.1:0/1223782866 --> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557148b4abf0 con 0x557148bf53e0 2024-03-06T07:21:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.331+0000 7f634ad9d640 1 --2- 172.21.15.1:0/1223782866 >> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] conn(0x557148bf53e0 0x557148c262f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.332+0000 7f6329ffb640 10 client.5487 ms_handle_connect on v2:172.21.15.1:6832/1818141892 2024-03-06T07:21:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.332+0000 7f634ad9d640 1 --2- 172.21.15.1:0/1223782866 >> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] conn(0x557148bf53e0 0x557148c262f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c52dd2db7228a0b2 server_cookie=2fcf3bfd5d9ac41e in_seq=0 out_seq=0 2024-03-06T07:21:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.332+0000 7f6329ffb640 1 -- 172.21.15.1:0/1223782866 <== mds.0 v2:172.21.15.1:6832/1818141892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557148b4abf0 con 0x557148bf53e0 2024-03-06T07:21:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.332+0000 7f6329ffb640 10 client.5487 handle_command_reply: tid=0 2024-03-06T07:21:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.332+0000 7f6353e37740 1 client.5487 shutdown 2024-03-06T07:21:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.332+0000 7f6353e37740 1 -- 172.21.15.1:0/1223782866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bffe00 msgr2=0x557148bf4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.332+0000 7f6353e37740 1 --2- 172.21.15.1:0/1223782866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bffe00 0x557148bf4d20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f6344060190 tx=0x7f6344057a00 comp rx=0 tx=0).stop 2024-03-06T07:21:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.333+0000 7f6353e37740 1 -- 172.21.15.1:0/1223782866 shutdown_connections 2024-03-06T07:21:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.333+0000 7f6353e37740 1 --2- 172.21.15.1:0/1223782866 >> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] conn(0x557148bf53e0 0x557148c262f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.333+0000 7f6353e37740 1 --2- 172.21.15.1:0/1223782866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bffe00 0x557148bf4d20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.333+0000 7f6329ffb640 0 client.5487 ms_handle_reset on v2:172.21.15.1:6832/1818141892 2024-03-06T07:21:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.333+0000 7f6353e37740 1 --2- 172.21.15.1:0/1223782866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bd9330 0x557148bf91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.333+0000 7f6353e37740 1 --2- 172.21.15.1:0/1223782866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd8f50 0x557148bfc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.333+0000 7f6353e37740 1 -- 172.21.15.1:0/1223782866 >> 172.21.15.1:0/1223782866 conn(0x557148bd74b0 msgr2=0x557148c15b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.333+0000 7f6353e37740 1 -- 172.21.15.1:0/1223782866 shutdown_connections 2024-03-06T07:21:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.333+0000 7f6353e37740 1 -- 172.21.15.1:0/1223782866 wait complete. 2024-03-06T07:21:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.333+0000 7f6353e37740 20 client.5487 trim_cache size 0 max 16384 2024-03-06T07:21:32.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.349+0000 7f6353e37740 1 Processor -- start 2024-03-06T07:21:32.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.349+0000 7f6353e37740 1 -- start start 2024-03-06T07:21:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.350+0000 7f6353e37740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd8f50 0x557148c28da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.350+0000 7f6353e37740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd9330 0x557148bd69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.350+0000 7f6353e37740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bffe00 0x557148bd6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.350+0000 7f6353e37740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c1f7b80 con 0x557148bd9330 2024-03-06T07:21:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.350+0000 7f6353e37740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c1f6d20 con 0x557148bffe00 2024-03-06T07:21:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.350+0000 7f6353e37740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634c1f6120 con 0x557148bd8f50 2024-03-06T07:21:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.350+0000 7f634a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd9330 0x557148bd69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.350+0000 7f634a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd9330 0x557148bd69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37160/0 (socket says 172.21.15.1:37160) 2024-03-06T07:21:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.350+0000 7f634a59c640 1 -- 172.21.15.1:0/916140880 learned_addr learned my addr 172.21.15.1:0/916140880 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.350+0000 7f634a59c640 1 -- 172.21.15.1:0/916140880 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd8f50 msgr2=0x557148c28da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.350+0000 7f634a59c640 1 --2- 172.21.15.1:0/916140880 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd8f50 0x557148c28da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.350+0000 7f634a59c640 1 -- 172.21.15.1:0/916140880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bffe00 msgr2=0x557148bd6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.350+0000 7f634a59c640 1 --2- 172.21.15.1:0/916140880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bffe00 0x557148bd6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.350+0000 7f634a59c640 1 -- 172.21.15.1:0/916140880 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55714891c6b0 con 0x557148bd9330 2024-03-06T07:21:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.351+0000 7f634a59c640 1 --2- 172.21.15.1:0/916140880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd9330 0x557148bd69a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f634c0732d0 tx=0x7f6340009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13acfdd9b5e844da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.351+0000 7f632bfff640 1 -- 172.21.15.1:0/916140880 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634001a070 con 0x557148bd9330 2024-03-06T07:21:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.351+0000 7f632bfff640 1 -- 172.21.15.1:0/916140880 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63400053a0 con 0x557148bd9330 2024-03-06T07:21:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.351+0000 7f632bfff640 1 -- 172.21.15.1:0/916140880 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6340014b40 con 0x557148bd9330 2024-03-06T07:21:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.351+0000 7f6353e37740 1 -- 172.21.15.1:0/916140880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd9330 msgr2=0x557148bd69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.351+0000 7f6353e37740 1 --2- 172.21.15.1:0/916140880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd9330 0x557148bd69a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f634c0732d0 tx=0x7f6340009040 comp rx=0 tx=0).stop 2024-03-06T07:21:32.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.351+0000 7f6353e37740 1 -- 172.21.15.1:0/916140880 shutdown_connections 2024-03-06T07:21:32.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.351+0000 7f6353e37740 1 --2- 172.21.15.1:0/916140880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bffe00 0x557148bd6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.351+0000 7f6353e37740 1 --2- 172.21.15.1:0/916140880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd9330 0x557148bd69a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.351+0000 7f6353e37740 1 --2- 172.21.15.1:0/916140880 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd8f50 0x557148c28da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.351+0000 7f6353e37740 1 -- 172.21.15.1:0/916140880 >> 172.21.15.1:0/916140880 conn(0x557148bd74b0 msgr2=0x557148bf4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.351+0000 7f6353e37740 1 -- 172.21.15.1:0/916140880 shutdown_connections 2024-03-06T07:21:32.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.351+0000 7f6353e37740 1 -- 172.21.15.1:0/916140880 wait complete. 2024-03-06T07:21:32.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.352+0000 7f6353e37740 1 -- start start 2024-03-06T07:21:32.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.352+0000 7f6353e37740 1 Processor -- start 2024-03-06T07:21:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.353+0000 7f6353e37740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd8f50 0x557148c39280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.353+0000 7f6353e37740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd9330 0x557148bfcde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.353+0000 7f6353e37740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bffe00 0x557148c82b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.353+0000 7f6353e37740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x557148bf9c00 con 0x557148bd8f50 2024-03-06T07:21:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.353+0000 7f6353e37740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c1f6720 con 0x557148bffe00 2024-03-06T07:21:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.353+0000 7f6353e37740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x557148bf5260 con 0x557148bd9330 2024-03-06T07:21:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.353+0000 7f6353e37740 10 client.0 _ll_register_callbacks cb 0x7f6344057b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:21:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.353+0000 7f634ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd8f50 0x557148c39280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.353+0000 7f634ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd8f50 0x557148c39280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37164/0 (socket says 172.21.15.1:37164) 2024-03-06T07:21:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.353+0000 7f634ad9d640 1 -- 172.21.15.1:0/2785781571 learned_addr learned my addr 172.21.15.1:0/2785781571 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.353+0000 7f634ad9d640 1 -- 172.21.15.1:0/2785781571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd9330 msgr2=0x557148bfcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.353+0000 7f634ad9d640 1 --2- 172.21.15.1:0/2785781571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd9330 0x557148bfcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.353+0000 7f634ad9d640 1 -- 172.21.15.1:0/2785781571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bffe00 msgr2=0x557148c82b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.353+0000 7f634ad9d640 1 --2- 172.21.15.1:0/2785781571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bffe00 0x557148c82b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.353+0000 7f634ad9d640 1 -- 172.21.15.1:0/2785781571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557148c397c0 con 0x557148bd8f50 2024-03-06T07:21:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.354+0000 7f634ad9d640 1 --2- 172.21.15.1:0/2785781571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd8f50 0x557148c39280 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f633c0098c0 tx=0x7f633c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bcf81aff9594d86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.354+0000 7f6329ffb640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:21:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.354+0000 7f6329ffb640 1 -- 172.21.15.1:0/2785781571 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c040040 con 0x557148bd8f50 2024-03-06T07:21:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.354+0000 7f6329ffb640 1 -- 172.21.15.1:0/2785781571 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f633c05e080 con 0x557148bd8f50 2024-03-06T07:21:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.354+0000 7f6353e37740 1 -- 172.21.15.1:0/2785781571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f63400053a0 con 0x557148bd8f50 2024-03-06T07:21:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.354+0000 7f6329ffb640 1 -- 172.21.15.1:0/2785781571 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f633c066050 con 0x557148bd8f50 2024-03-06T07:21:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.354+0000 7f6353e37740 10 client.5493 fetch_fsmap learned FSMap version 16 2024-03-06T07:21:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.354+0000 7f6353e37740 1 -- 172.21.15.1:0/2785781571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x557148c38e50 con 0x557148bd8f50 2024-03-06T07:21:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.354+0000 7f6329ffb640 1 -- 172.21.15.1:0/2785781571 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63400053a0 con 0x557148bd8f50 2024-03-06T07:21:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.354+0000 7f6329ffb640 1 -- 172.21.15.1:0/2785781571 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 16) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f633c069020 con 0x557148bd8f50 2024-03-06T07:21:32.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.354+0000 7f6353e37740 10 client.5493 fetch_fsmap finished waiting for FSMap version 16 2024-03-06T07:21:32.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.354+0000 7f6353e37740 10 client.5493 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:21:32.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.354+0000 7f6353e37740 20 client.5493 populate_metadata read hostname 'smithi001' 2024-03-06T07:21:32.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.354+0000 7f6353e37740 1 --2- 172.21.15.1:0/2785781571 >> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] conn(0x557148bf57c0 0x557148bf5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.354+0000 7f6353e37740 4 client.5493 mds_command: new command op to 5280 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:21:32.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.354+0000 7f6353e37740 1 -- 172.21.15.1:0/2785781571 --> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557148b44f40 con 0x557148bf57c0 2024-03-06T07:21:32.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.354+0000 7f634a59c640 1 --2- 172.21.15.1:0/2785781571 >> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] conn(0x557148bf57c0 0x557148bf5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.355+0000 7f6329ffb640 10 client.5493 ms_handle_connect on v2:172.21.15.1:6832/1818141892 2024-03-06T07:21:32.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.355+0000 7f634a59c640 1 --2- 172.21.15.1:0/2785781571 >> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] conn(0x557148bf57c0 0x557148bf5ba0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e41767c3e8f5cc5 server_cookie=3a06608f894216cc in_seq=0 out_seq=0 2024-03-06T07:21:32.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.355+0000 7f6329ffb640 1 -- 172.21.15.1:0/2785781571 <== mds.0 v2:172.21.15.1:6832/1818141892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x557148b44f40 con 0x557148bf57c0 2024-03-06T07:21:32.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.355+0000 7f6329ffb640 10 client.5493 handle_command_reply: tid=0 2024-03-06T07:21:32.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.355+0000 7f6353e37740 1 client.5493 shutdown 2024-03-06T07:21:32.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.355+0000 7f6353e37740 1 -- 172.21.15.1:0/2785781571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd8f50 msgr2=0x557148c39280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.355+0000 7f6353e37740 1 --2- 172.21.15.1:0/2785781571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd8f50 0x557148c39280 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f633c0098c0 tx=0x7f633c004730 comp rx=0 tx=0).stop 2024-03-06T07:21:32.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.355+0000 7f6353e37740 1 -- 172.21.15.1:0/2785781571 shutdown_connections 2024-03-06T07:21:32.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.355+0000 7f6353e37740 1 --2- 172.21.15.1:0/2785781571 >> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] conn(0x557148bf57c0 0x557148bf5ba0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.355+0000 7f6353e37740 1 --2- 172.21.15.1:0/2785781571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557148bffe00 0x557148c82b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.355+0000 7f6353e37740 1 --2- 172.21.15.1:0/2785781571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x557148bd9330 0x557148bfcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.355+0000 7f6329ffb640 0 client.5493 ms_handle_reset on v2:172.21.15.1:6832/1818141892 2024-03-06T07:21:32.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.355+0000 7f6353e37740 1 --2- 172.21.15.1:0/2785781571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557148bd8f50 0x557148c39280 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.355+0000 7f6353e37740 1 -- 172.21.15.1:0/2785781571 >> 172.21.15.1:0/2785781571 conn(0x557148bd74b0 msgr2=0x557148c28da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.355+0000 7f6353e37740 1 -- 172.21.15.1:0/2785781571 shutdown_connections 2024-03-06T07:21:32.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.355+0000 7f6353e37740 1 -- 172.21.15.1:0/2785781571 wait complete. 2024-03-06T07:21:32.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.355+0000 7f6353e37740 20 client.5493 trim_cache size 0 max 16384 2024-03-06T07:21:32.364 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:21:32.364 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:21:32.365 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:21:32.365 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 5280, 2024-03-06T07:21:32.365 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:21:32.365 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:21:32.365 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:21:32.365 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 9.1941128370000005, 2024-03-06T07:21:32.365 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 16, 2024-03-06T07:21:32.365 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 39, 2024-03-06T07:21:32.366 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 39, 2024-03-06T07:21:32.366 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 16.068433635000002 2024-03-06T07:21:32.366 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:21:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.356+0000 7f635124e640 1 -- 172.21.15.1:0/507778878 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f633806b4d0 msgr2=0x7f633806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.356+0000 7f635124e640 1 --2- 172.21.15.1:0/507778878 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f633806b4d0 0x7f633806d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f633c034000 tx=0x7f633c008030 comp rx=0 tx=0).stop 2024-03-06T07:21:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.356+0000 7f635124e640 1 -- 172.21.15.1:0/507778878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f634c152840 msgr2=0x7f634c150640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.356+0000 7f635124e640 1 --2- 172.21.15.1:0/507778878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f634c152840 0x7f634c150640 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f634400c840 tx=0x7f634400cd00 comp rx=0 tx=0).stop 2024-03-06T07:21:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.356+0000 7f635124e640 1 -- 172.21.15.1:0/507778878 shutdown_connections 2024-03-06T07:21:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.356+0000 7f635124e640 1 --2- 172.21.15.1:0/507778878 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f633806b4d0 0x7f633806d990 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.356+0000 7f635124e640 1 --2- 172.21.15.1:0/507778878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f634c152840 0x7f634c150640 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.356+0000 7f635124e640 1 --2- 172.21.15.1:0/507778878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f634c0c60b0 0x7f634c1ed9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.356+0000 7f635124e640 1 --2- 172.21.15.1:0/507778878 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f634c0c3970 0x7f634c0ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.356+0000 7f635124e640 1 -- 172.21.15.1:0/507778878 >> 172.21.15.1:0/507778878 conn(0x7f634c0c88f0 msgr2=0x7f634c06c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.356+0000 7f635124e640 1 -- 172.21.15.1:0/507778878 shutdown_connections 2024-03-06T07:21:32.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.356+0000 7f635124e640 1 -- 172.21.15.1:0/507778878 wait complete. 2024-03-06T07:21:32.370 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:32.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.470+0000 7fb02bfff640 1 Processor -- start 2024-03-06T07:21:32.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.470+0000 7fb02bfff640 1 -- start start 2024-03-06T07:21:32.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.470+0000 7fb02bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb02c1526f0 0x7fb02c0c3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.470+0000 7fb02bfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb02c155010 0x7fb02c1589e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.470+0000 7fb02bfff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb02c0c7e70 con 0x7fb02c0c72e0 2024-03-06T07:21:32.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.470+0000 7fb02bfff640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02c1554a0 con 0x7fb02c155010 2024-03-06T07:21:32.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.470+0000 7fb02bfff640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c155780 con 0x7fb02c1526f0 2024-03-06T07:21:32.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.471+0000 7fb02affd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb02c1526f0 0x7fb02c0c3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.471+0000 7fb0227fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb02c155010 0x7fb02c1589e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.471+0000 7fb02affd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb02c1526f0 0x7fb02c0c3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:36648/0 (socket says 172.21.15.1:36648) 2024-03-06T07:21:32.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.471+0000 7fb02affd640 1 -- 172.21.15.1:0/1630039062 learned_addr learned my addr 172.21.15.1:0/1630039062 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:32.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.471+0000 7fb02affd640 1 -- 172.21.15.1:0/1630039062 >> v2:172.21.15.98:3300/0 conn(0x7fb02c155010 msgr2=0x7fb02c1589e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:32.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.471+0000 7fb02affd640 1 --2- 172.21.15.1:0/1630039062 >> v2:172.21.15.98:3300/0 conn(0x7fb02c155010 0x7fb02c1589e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.471+0000 7fb02affd640 1 -- 172.21.15.1:0/1630039062 >> v1:172.21.15.98:6789/0 conn(0x7fb02c0c72e0 legacy=0x7fb02c158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.471+0000 7fb02affd640 1 -- 172.21.15.1:0/1630039062 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02c155a60 con 0x7fb02c1526f0 2024-03-06T07:21:32.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.472+0000 7fb02affd640 1 --2- 172.21.15.1:0/1630039062 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb02c1526f0 0x7fb02c0c3100 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb01c009870 tx=0x7fb01c02ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e80e62de70c42ffc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.472+0000 7fb02a7fc640 1 -- 172.21.15.1:0/1630039062 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c02f840 con 0x7fb02c1526f0 2024-03-06T07:21:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.472+0000 7fb02a7fc640 1 -- 172.21.15.1:0/1630039062 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01c02f9e0 con 0x7fb02c1526f0 2024-03-06T07:21:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.472+0000 7fb02a7fc640 1 -- 172.21.15.1:0/1630039062 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c02fdb0 con 0x7fb02c1526f0 2024-03-06T07:21:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.472+0000 7fb02bfff640 1 -- 172.21.15.1:0/1630039062 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb02c1526f0 msgr2=0x7fb02c0c3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.472+0000 7fb02bfff640 1 --2- 172.21.15.1:0/1630039062 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb02c1526f0 0x7fb02c0c3100 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb01c009870 tx=0x7fb01c02ba20 comp rx=0 tx=0).stop 2024-03-06T07:21:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.472+0000 7fb02bfff640 1 -- 172.21.15.1:0/1630039062 shutdown_connections 2024-03-06T07:21:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.472+0000 7fb02bfff640 1 --2- 172.21.15.1:0/1630039062 >> v2:172.21.15.98:3300/0 conn(0x7fb02c155010 0x7fb02c1589e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.472+0000 7fb02bfff640 1 --2- 172.21.15.1:0/1630039062 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb02c1526f0 0x7fb02c0c3100 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.472+0000 7fb02bfff640 1 -- 172.21.15.1:0/1630039062 >> 172.21.15.1:0/1630039062 conn(0x7fb02c06b620 msgr2=0x7fb02c0aa4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.473+0000 7fb02bfff640 1 -- 172.21.15.1:0/1630039062 shutdown_connections 2024-03-06T07:21:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.473+0000 7fb02bfff640 1 -- 172.21.15.1:0/1630039062 wait complete. 2024-03-06T07:21:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.474+0000 7fb02bfff640 1 Processor -- start 2024-03-06T07:21:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.474+0000 7fb02bfff640 1 -- start start 2024-03-06T07:21:32.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.474+0000 7fb02bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb02c0c72e0 0x7fb02c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb02bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb02c1526f0 0x7fb02c06fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb02bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb02c14bbf0 0x7fb02c1ee200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb02bfff640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c0c37c0 con 0x7fb02c14bbf0 2024-03-06T07:21:32.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb02bfff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c0c3640 con 0x7fb02c1526f0 2024-03-06T07:21:32.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb02bfff640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c0c3940 con 0x7fb02c0c72e0 2024-03-06T07:21:32.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb02b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb02c14bbf0 0x7fb02c1ee200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb02affd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb02c0c72e0 0x7fb02c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb0227fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb02c1526f0 0x7fb02c06fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb02affd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb02c0c72e0 0x7fb02c06d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:36660/0 (socket says 172.21.15.1:36660) 2024-03-06T07:21:32.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb02affd640 1 -- 172.21.15.1:0/117092045 learned_addr learned my addr 172.21.15.1:0/117092045 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:32.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb0227fc640 1 -- 172.21.15.1:0/117092045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb02c0c72e0 msgr2=0x7fb02c06d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb0227fc640 1 --2- 172.21.15.1:0/117092045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb02c0c72e0 0x7fb02c06d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb0227fc640 1 -- 172.21.15.1:0/117092045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb02c14bbf0 msgr2=0x7fb02c1ee200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb0227fc640 1 --2- 172.21.15.1:0/117092045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb02c14bbf0 0x7fb02c1ee200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb0227fc640 1 -- 172.21.15.1:0/117092045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02c0708e0 con 0x7fb02c1526f0 2024-03-06T07:21:32.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb02b7fe640 1 --2- 172.21.15.1:0/117092045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb02c14bbf0 0x7fb02c1ee200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:32.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb02affd640 1 --2- 172.21.15.1:0/117092045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb02c0c72e0 0x7fb02c06d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:32.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb0227fc640 1 --2- 172.21.15.1:0/117092045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb02c1526f0 0x7fb02c06fc20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb01400e8a0 tx=0x7fb01400ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.475+0000 7fb028ff9640 1 -- 172.21.15.1:0/117092045 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb014017070 con 0x7fb02c1526f0 2024-03-06T07:21:32.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.476+0000 7fb028ff9640 1 -- 172.21.15.1:0/117092045 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01400c940 con 0x7fb02c1526f0 2024-03-06T07:21:32.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.476+0000 7fb02bfff640 1 -- 172.21.15.1:0/117092045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb02c070bc0 con 0x7fb02c1526f0 2024-03-06T07:21:32.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.476+0000 7fb028ff9640 1 -- 172.21.15.1:0/117092045 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb014009e60 con 0x7fb02c1526f0 2024-03-06T07:21:32.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.476+0000 7fb02bfff640 1 -- 172.21.15.1:0/117092045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb02c071180 con 0x7fb02c1526f0 2024-03-06T07:21:32.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.476+0000 7fb02bfff640 1 -- 172.21.15.1:0/117092045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02c0686e0 con 0x7fb02c1526f0 2024-03-06T07:21:32.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.477+0000 7fb028ff9640 1 -- 172.21.15.1:0/117092045 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb01401a070 con 0x7fb02c1526f0 2024-03-06T07:21:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.477+0000 7fb028ff9640 1 --2- 172.21.15.1:0/117092045 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb00c06b4d0 0x7fb00c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.477+0000 7fb028ff9640 1 -- 172.21.15.1:0/117092045 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb01408a040 con 0x7fb02c1526f0 2024-03-06T07:21:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.478+0000 7fb02affd640 1 --2- 172.21.15.1:0/117092045 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb00c06b4d0 0x7fb00c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.478+0000 7fb02affd640 1 --2- 172.21.15.1:0/117092045 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb00c06b4d0 0x7fb00c06d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb01c009870 tx=0x7fb01c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.483+0000 7fb028ff9640 1 -- 172.21.15.1:0/117092045 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb014058770 con 0x7fb02c1526f0 2024-03-06T07:21:32.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.702+0000 7fb02bfff640 1 -- 172.21.15.1:0/117092045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb02c0c6640 con 0x7fb02c1526f0 2024-03-06T07:21:32.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.703+0000 7fb028ff9640 1 -- 172.21.15.1:0/117092045 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4971 (secure 0 0 0) 0x7fb01405c810 con 0x7fb02c1526f0 2024-03-06T07:21:32.705 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:32.705 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4781,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2438105345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2438105345},{"type":"v1","addr":"172.21.15.1:6835","nonce":2438105345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3354292028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3354292028},{"type":"v1","addr":"172.21.15.98:6836","nonce":3354292028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4975,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3948803645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3948803645},{"type":"v1","addr":"172.21.15.98:6837","nonce":3948803645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:22.528539+0000","modified":"2024-03-06T07:21:24.518239+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1818141892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1818141892},{"type":"v1","addr":"172.21.15.1:6834","nonce":1818141892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":2}]} 2024-03-06T07:21:32.705 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 16 2024-03-06T07:21:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.706+0000 7fb02bfff640 1 -- 172.21.15.1:0/117092045 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb00c06b4d0 msgr2=0x7fb00c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.706+0000 7fb02bfff640 1 --2- 172.21.15.1:0/117092045 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb00c06b4d0 0x7fb00c06d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb01c009870 tx=0x7fb01c008030 comp rx=0 tx=0).stop 2024-03-06T07:21:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.706+0000 7fb02bfff640 1 -- 172.21.15.1:0/117092045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb02c1526f0 msgr2=0x7fb02c06fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.706+0000 7fb02bfff640 1 --2- 172.21.15.1:0/117092045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb02c1526f0 0x7fb02c06fc20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb01400e8a0 tx=0x7fb01400ed60 comp rx=0 tx=0).stop 2024-03-06T07:21:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.706+0000 7fb02bfff640 1 -- 172.21.15.1:0/117092045 shutdown_connections 2024-03-06T07:21:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.706+0000 7fb02bfff640 1 --2- 172.21.15.1:0/117092045 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb00c06b4d0 0x7fb00c06d990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.706+0000 7fb02bfff640 1 --2- 172.21.15.1:0/117092045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb02c14bbf0 0x7fb02c1ee200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.706+0000 7fb02bfff640 1 --2- 172.21.15.1:0/117092045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb02c1526f0 0x7fb02c06fc20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.706+0000 7fb02bfff640 1 --2- 172.21.15.1:0/117092045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb02c0c72e0 0x7fb02c06d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.706+0000 7fb02bfff640 1 -- 172.21.15.1:0/117092045 >> 172.21.15.1:0/117092045 conn(0x7fb02c06b620 msgr2=0x7fb02c0c6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.706+0000 7fb02bfff640 1 -- 172.21.15.1:0/117092045 shutdown_connections 2024-03-06T07:21:32.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.706+0000 7fb02bfff640 1 -- 172.21.15.1:0/117092045 wait complete. 2024-03-06T07:21:32.719 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 16, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:21:22.528539+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 16, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5280": { "addr": "172.21.15.1:6834/1818141892", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 1818141892, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 1818141892, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5280, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-03-06T07:21:24.518239+0000", "qdb_cluster": [ 5280 ], "qdb_leader": 5280, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5280 } } } ], "standbys": [ { "addr": "172.21.15.1:6835/2438105345", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 2438105345, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 2438105345, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4781, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6836/3354292028", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 3354292028, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 3354292028, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4972, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6837/3948803645", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 3948803645, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 3948803645, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4975, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:21:32.719 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:21:32.720 INFO:tasks.cephfs.test_failover:current = [{'gid': 5280, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6834/1818141892', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1818141892}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1818141892}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 4975, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/3948803645', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3948803645}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3948803645}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 4781, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/2438105345', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 2438105345}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2438105345}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 4972, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/3354292028', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 3354292028}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 3354292028}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}] 2024-03-06T07:21:32.720 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs 2024-03-06T07:21:32.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.822+0000 7f97e553a640 1 Processor -- start 2024-03-06T07:21:32.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.823+0000 7f97e553a640 1 -- start start 2024-03-06T07:21:32.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.824+0000 7f97e553a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97e00c60b0 0x7f97e00c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.824+0000 7f97e553a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e00c5e40 con 0x7f97e0152840 2024-03-06T07:21:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.824+0000 7f97e553a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e00c6a40 con 0x7f97e00c3970 2024-03-06T07:21:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.824+0000 7f97e553a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e00c67f0 con 0x7f97e00c60b0 2024-03-06T07:21:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.824+0000 7f97d7fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f97e0152840 0x7f97e01553d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60816/0 (socket says 172.21.15.1:60816) 2024-03-06T07:21:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.824+0000 7f97d7fff640 1 -- 172.21.15.1:0/3862872073 learned_addr learned my addr 172.21.15.1:0/3862872073 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.824+0000 7f97de7fc640 1 -- 172.21.15.1:0/3862872073 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19887003 0 0) 0x7f97e00c5e40 con 0x7f97e0152840 2024-03-06T07:21:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.824+0000 7f97de7fc640 1 -- 172.21.15.1:0/3862872073 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97cc003520 con 0x7f97e0152840 2024-03-06T07:21:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.824+0000 7f97df7fe640 1 --2- 172.21.15.1:0/3862872073 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97e00c60b0 0x7f97e00c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.824+0000 7f97de7fc640 1 -- 172.21.15.1:0/3862872073 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3392970533 0 0) 0x7f97cc003520 con 0x7f97e0152840 2024-03-06T07:21:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.824+0000 7f97de7fc640 1 -- 172.21.15.1:0/3862872073 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97e00c60b0 msgr2=0x7f97e00c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.824+0000 7f97de7fc640 1 --2- 172.21.15.1:0/3862872073 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97e00c60b0 0x7f97e00c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.824+0000 7f97de7fc640 1 -- 172.21.15.1:0/3862872073 >> v1:172.21.15.98:6789/0 conn(0x7f97e00c3970 legacy=0x7f97e0152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.824+0000 7f97de7fc640 1 -- 172.21.15.1:0/3862872073 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e00c6d20 con 0x7f97e0152840 2024-03-06T07:21:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.824+0000 7f97de7fc640 1 -- 172.21.15.1:0/3862872073 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f97c8001670 con 0x7f97e0152840 2024-03-06T07:21:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.825+0000 7f97de7fc640 1 -- 172.21.15.1:0/3862872073 >> v1:172.21.15.1:6789/0 conn(0x7f97e0152840 legacy=0x7f97e01553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.825+0000 7f97de7fc640 1 --2- 172.21.15.1:0/3862872073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc0039e0 0x7f97cc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.825+0000 7f97de7fc640 1 --2- 172.21.15.1:0/3862872073 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97e00c60b0 0x7f97cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.825+0000 7f97de7fc640 1 --2- 172.21.15.1:0/3862872073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97cc004d40 0x7f97cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.825+0000 7f97de7fc640 1 -- 172.21.15.1:0/3862872073 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e00c67f0 con 0x7f97cc004d40 2024-03-06T07:21:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.825+0000 7f97de7fc640 1 -- 172.21.15.1:0/3862872073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc00ba70 con 0x7f97cc0039e0 2024-03-06T07:21:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.825+0000 7f97de7fc640 1 -- 172.21.15.1:0/3862872073 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc00bc70 con 0x7f97e00c60b0 2024-03-06T07:21:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.825+0000 7f97df7fe640 1 --2- 172.21.15.1:0/3862872073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97cc004d40 0x7f97cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.825+0000 7f97d7fff640 1 --2- 172.21.15.1:0/3862872073 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97e00c60b0 0x7f97cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.825+0000 7f97deffd640 1 --2- 172.21.15.1:0/3862872073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc0039e0 0x7f97cc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.825+0000 7f97df7fe640 1 -- 172.21.15.1:0/3862872073 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97e00c60b0 msgr2=0x7f97cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.825+0000 7f97df7fe640 1 --2- 172.21.15.1:0/3862872073 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97e00c60b0 0x7f97cc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.825+0000 7f97df7fe640 1 -- 172.21.15.1:0/3862872073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc0039e0 msgr2=0x7f97cc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:32.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.825+0000 7f97df7fe640 1 --2- 172.21.15.1:0/3862872073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc0039e0 0x7f97cc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.825+0000 7f97df7fe640 1 -- 172.21.15.1:0/3862872073 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97cc003520 con 0x7f97cc004d40 2024-03-06T07:21:32.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.826+0000 7f97df7fe640 1 --2- 172.21.15.1:0/3862872073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97cc004d40 0x7f97cc0051c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f97e00c64b0 tx=0x7f97cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d075a98877c74146 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.826+0000 7f97de7fc640 1 -- 172.21.15.1:0/3862872073 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8039070 con 0x7f97cc004d40 2024-03-06T07:21:32.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.826+0000 7f97de7fc640 1 -- 172.21.15.1:0/3862872073 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d802cb90 con 0x7f97cc004d40 2024-03-06T07:21:32.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.826+0000 7f97de7fc640 1 -- 172.21.15.1:0/3862872073 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8009c10 con 0x7f97cc004d40 2024-03-06T07:21:32.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.826+0000 7f97e553a640 1 -- 172.21.15.1:0/3862872073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97cc004d40 msgr2=0x7f97cc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.826+0000 7f97e553a640 1 --2- 172.21.15.1:0/3862872073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97cc004d40 0x7f97cc0051c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f97e00c64b0 tx=0x7f97cc005830 comp rx=0 tx=0).stop 2024-03-06T07:21:32.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.826+0000 7f97df7fe640 1 -- 172.21.15.1:0/3862872073 reap_dead start 2024-03-06T07:21:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.826+0000 7f97e553a640 1 -- 172.21.15.1:0/3862872073 shutdown_connections 2024-03-06T07:21:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.826+0000 7f97e553a640 1 -- 172.21.15.1:0/3862872073 >> 172.21.15.1:0/3862872073 conn(0x7f97e00c88f0 msgr2=0x7f97e00c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.827+0000 7f97e553a640 1 -- 172.21.15.1:0/3862872073 shutdown_connections 2024-03-06T07:21:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.827+0000 7f97e553a640 1 -- 172.21.15.1:0/3862872073 wait complete. 2024-03-06T07:21:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.828+0000 7f97e553a640 1 Processor -- start 2024-03-06T07:21:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.828+0000 7f97e553a640 1 -- start start 2024-03-06T07:21:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.829+0000 7f97e553a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97cc004d40 0x7f97e0150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.829+0000 7f97e553a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e01eb160 0x7f97e01edcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.829+0000 7f97e553a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97e00ab9a0 0x7f97e01ee210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.829+0000 7f97e553a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e00c75c0 con 0x7f97e00ab9a0 2024-03-06T07:21:32.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.829+0000 7f97e553a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e00c72e0 con 0x7f97e01eb160 2024-03-06T07:21:32.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.829+0000 7f97e553a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e00c3d50 con 0x7f97cc004d40 2024-03-06T07:21:32.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.829+0000 7f97df7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97e00ab9a0 0x7f97e01ee210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.830+0000 7f97df7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97e00ab9a0 0x7f97e01ee210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37194/0 (socket says 172.21.15.1:37194) 2024-03-06T07:21:32.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.830+0000 7f97df7fe640 1 -- 172.21.15.1:0/2780861680 learned_addr learned my addr 172.21.15.1:0/2780861680 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:32.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.830+0000 7f97deffd640 1 --2- 172.21.15.1:0/2780861680 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97cc004d40 0x7f97e0150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.830+0000 7f97d7fff640 1 --2- 172.21.15.1:0/2780861680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e01eb160 0x7f97e01edcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.830+0000 7f97df7fe640 1 -- 172.21.15.1:0/2780861680 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97cc004d40 msgr2=0x7f97e0150130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.830+0000 7f97df7fe640 1 --2- 172.21.15.1:0/2780861680 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97cc004d40 0x7f97e0150130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.830+0000 7f97df7fe640 1 -- 172.21.15.1:0/2780861680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e01eb160 msgr2=0x7f97e01edcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.830+0000 7f97df7fe640 1 --2- 172.21.15.1:0/2780861680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e01eb160 0x7f97e01edcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.830+0000 7f97df7fe640 1 -- 172.21.15.1:0/2780861680 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e01eb6b0 con 0x7f97e00ab9a0 2024-03-06T07:21:32.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.830+0000 7f97df7fe640 1 --2- 172.21.15.1:0/2780861680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97e00ab9a0 0x7f97e01ee210 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f97d8037040 tx=0x7f97d8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.830+0000 7f97d77fe640 1 -- 172.21.15.1:0/2780861680 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8042020 con 0x7f97e00ab9a0 2024-03-06T07:21:32.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.830+0000 7f97d77fe640 1 -- 172.21.15.1:0/2780861680 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d803a040 con 0x7f97e00ab9a0 2024-03-06T07:21:32.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.830+0000 7f97e553a640 1 -- 172.21.15.1:0/2780861680 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e0070f80 con 0x7f97e00ab9a0 2024-03-06T07:21:32.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.830+0000 7f97d77fe640 1 -- 172.21.15.1:0/2780861680 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8039070 con 0x7f97e00ab9a0 2024-03-06T07:21:32.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.830+0000 7f97e553a640 1 -- 172.21.15.1:0/2780861680 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e0150670 con 0x7f97e00ab9a0 2024-03-06T07:21:32.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.831+0000 7f97e553a640 1 -- 172.21.15.1:0/2780861680 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e00c64e0 con 0x7f97e00ab9a0 2024-03-06T07:21:32.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.832+0000 7f97d77fe640 1 -- 172.21.15.1:0/2780861680 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97d8005d10 con 0x7f97e00ab9a0 2024-03-06T07:21:32.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.832+0000 7f97d77fe640 1 --2- 172.21.15.1:0/2780861680 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f97c006b500 0x7f97c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.833+0000 7f97d77fe640 1 -- 172.21.15.1:0/2780861680 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f97d80ac660 con 0x7f97e00ab9a0 2024-03-06T07:21:32.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.833+0000 7f97deffd640 1 --2- 172.21.15.1:0/2780861680 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f97c006b500 0x7f97c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.833+0000 7f97deffd640 1 --2- 172.21.15.1:0/2780861680 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f97c006b500 0x7f97c006d9c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f97e014d300 tx=0x7f97d0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:32.837+0000 7f97d77fe640 1 -- 172.21.15.1:0/2780861680 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97e00c64e0 con 0x7f97e00ab9a0 2024-03-06T07:21:33.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.019+0000 7f97e553a640 1 -- 172.21.15.1:0/2780861680 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f97e00686e0 con 0x7f97e00ab9a0 2024-03-06T07:21:33.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.022+0000 7f97d77fe640 1 -- 172.21.15.1:0/2780861680 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v12)=0 v12) v1 ==== 116+0+0 (secure 0 0 0) 0x7f97d804c070 con 0x7f97e00ab9a0 2024-03-06T07:21:33.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.024+0000 7f97e553a640 1 -- 172.21.15.1:0/2780861680 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f97c006b500 msgr2=0x7f97c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.024+0000 7f97e553a640 1 --2- 172.21.15.1:0/2780861680 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f97c006b500 0x7f97c006d9c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f97e014d300 tx=0x7f97d0009040 comp rx=0 tx=0).stop 2024-03-06T07:21:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.024+0000 7f97e553a640 1 -- 172.21.15.1:0/2780861680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97e00ab9a0 msgr2=0x7f97e01ee210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.024+0000 7f97e553a640 1 --2- 172.21.15.1:0/2780861680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97e00ab9a0 0x7f97e01ee210 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f97d8037040 tx=0x7f97d8009750 comp rx=0 tx=0).stop 2024-03-06T07:21:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.024+0000 7f97e553a640 1 -- 172.21.15.1:0/2780861680 shutdown_connections 2024-03-06T07:21:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.024+0000 7f97e553a640 1 --2- 172.21.15.1:0/2780861680 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f97c006b500 0x7f97c006d9c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.024+0000 7f97e553a640 1 --2- 172.21.15.1:0/2780861680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97e00ab9a0 0x7f97e01ee210 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.024+0000 7f97e553a640 1 --2- 172.21.15.1:0/2780861680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e01eb160 0x7f97e01edcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.024+0000 7f97e553a640 1 --2- 172.21.15.1:0/2780861680 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97cc004d40 0x7f97e0150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.024+0000 7f97e553a640 1 -- 172.21.15.1:0/2780861680 >> 172.21.15.1:0/2780861680 conn(0x7f97e00c88f0 msgr2=0x7f97e00c8170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.024+0000 7f97e553a640 1 -- 172.21.15.1:0/2780861680 shutdown_connections 2024-03-06T07:21:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.024+0000 7f97e553a640 1 -- 172.21.15.1:0/2780861680 wait complete. 2024-03-06T07:21:33.037 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:33.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.137+0000 7f75cec4e640 1 Processor -- start 2024-03-06T07:21:33.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.137+0000 7f75cec4e640 1 -- start start 2024-03-06T07:21:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75cec4e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f75c80c3970 0x7f75c80c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75cec4e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f75c80c60b0 0x7f75c8152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75cec4e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f75c8152670 0x7f75c81553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75cec4e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c80c67f0 con 0x7f75c80c3970 2024-03-06T07:21:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75cec4e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c80c5e40 con 0x7f75c8152670 2024-03-06T07:21:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75cec4e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80c6a40 con 0x7f75c80c60b0 2024-03-06T07:21:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75cc9ea640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f75c80c3970 0x7f75c80c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75cc9ea640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f75c80c3970 0x7f75c80c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37212/0 (socket says 172.21.15.1:37212) 2024-03-06T07:21:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75cc9ea640 1 -- 172.21.15.1:0/3370450894 learned_addr learned my addr 172.21.15.1:0/3370450894 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75c7fff640 1 --2- 172.21.15.1:0/3370450894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f75c80c60b0 0x7f75c8152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75cc9ea640 1 -- 172.21.15.1:0/3370450894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f75c80c60b0 msgr2=0x7f75c8152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75cc9ea640 1 --2- 172.21.15.1:0/3370450894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f75c80c60b0 0x7f75c8152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75cd1eb640 1 --2- 172.21.15.1:0/3370450894 >> v2:172.21.15.98:3300/0 conn(0x7f75c8152670 0x7f75c81553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75cc9ea640 1 -- 172.21.15.1:0/3370450894 >> v2:172.21.15.98:3300/0 conn(0x7f75c8152670 msgr2=0x7f75c81553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75cc9ea640 1 --2- 172.21.15.1:0/3370450894 >> v2:172.21.15.98:3300/0 conn(0x7f75c8152670 0x7f75c81553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75cc9ea640 1 -- 172.21.15.1:0/3370450894 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c80c6d20 con 0x7f75c80c3970 2024-03-06T07:21:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75c7fff640 1 --2- 172.21.15.1:0/3370450894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f75c80c60b0 0x7f75c8152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.138+0000 7f75cd1eb640 1 --2- 172.21.15.1:0/3370450894 >> v2:172.21.15.98:3300/0 conn(0x7f75c8152670 0x7f75c81553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.139+0000 7f75cc9ea640 1 --2- 172.21.15.1:0/3370450894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80c3970 0x7f75c80c8170 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f75c80ae4a0 tx=0x7f75bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c03493fbf6efced7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.139+0000 7f75c77fe640 1 -- 172.21.15.1:0/3370450894 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc035020 con 0x7f75c80c3970 2024-03-06T07:21:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.139+0000 7f75c77fe640 1 -- 172.21.15.1:0/3370450894 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75bc031870 con 0x7f75c80c3970 2024-03-06T07:21:33.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.139+0000 7f75c77fe640 1 -- 172.21.15.1:0/3370450894 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc031c40 con 0x7f75c80c3970 2024-03-06T07:21:33.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.140+0000 7f75cec4e640 1 -- 172.21.15.1:0/3370450894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80c3970 msgr2=0x7f75c80c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.140+0000 7f75cec4e640 1 --2- 172.21.15.1:0/3370450894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80c3970 0x7f75c80c8170 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f75c80ae4a0 tx=0x7f75bc02b910 comp rx=0 tx=0).stop 2024-03-06T07:21:33.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.140+0000 7f75cec4e640 1 -- 172.21.15.1:0/3370450894 shutdown_connections 2024-03-06T07:21:33.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.140+0000 7f75cec4e640 1 --2- 172.21.15.1:0/3370450894 >> v2:172.21.15.98:3300/0 conn(0x7f75c8152670 0x7f75c81553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.140+0000 7f75cec4e640 1 --2- 172.21.15.1:0/3370450894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f75c80c60b0 0x7f75c8152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.140+0000 7f75cec4e640 1 --2- 172.21.15.1:0/3370450894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80c3970 0x7f75c80c8170 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.140+0000 7f75cec4e640 1 -- 172.21.15.1:0/3370450894 >> 172.21.15.1:0/3370450894 conn(0x7f75c80c88f0 msgr2=0x7f75c80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:33.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.140+0000 7f75cec4e640 1 -- 172.21.15.1:0/3370450894 shutdown_connections 2024-03-06T07:21:33.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.140+0000 7f75cec4e640 1 -- 172.21.15.1:0/3370450894 wait complete. 2024-03-06T07:21:33.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.141+0000 7f75cec4e640 1 Processor -- start 2024-03-06T07:21:33.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.141+0000 7f75cec4e640 1 -- start start 2024-03-06T07:21:33.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.142+0000 7f75cec4e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80c3970 0x7f75c80acb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.142+0000 7f75cec4e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f75c80c60b0 0x7f75c8155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.142+0000 7f75cec4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75c8152670 0x7f75c81ee480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.142+0000 7f75cec4e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80c75c0 con 0x7f75c80c3970 2024-03-06T07:21:33.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.142+0000 7f75cec4e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80c72e0 con 0x7f75c8152670 2024-03-06T07:21:33.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.142+0000 7f75cec4e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80c3d50 con 0x7f75c80c60b0 2024-03-06T07:21:33.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.142+0000 7f75cc9ea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80c3970 0x7f75c80acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.142+0000 7f75cc9ea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80c3970 0x7f75c80acb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37218/0 (socket says 172.21.15.1:37218) 2024-03-06T07:21:33.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.142+0000 7f75cc9ea640 1 -- 172.21.15.1:0/963553035 learned_addr learned my addr 172.21.15.1:0/963553035 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:33.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.143+0000 7f75cc9ea640 1 -- 172.21.15.1:0/963553035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f75c80c60b0 msgr2=0x7f75c8155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.143+0000 7f75cc9ea640 1 --2- 172.21.15.1:0/963553035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f75c80c60b0 0x7f75c8155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.143+0000 7f75cc9ea640 1 -- 172.21.15.1:0/963553035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75c8152670 msgr2=0x7f75c81ee480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.143+0000 7f75cc9ea640 1 --2- 172.21.15.1:0/963553035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75c8152670 0x7f75c81ee480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.143+0000 7f75cc9ea640 1 -- 172.21.15.1:0/963553035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c80ad820 con 0x7f75c80c3970 2024-03-06T07:21:33.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.143+0000 7f75cc9ea640 1 --2- 172.21.15.1:0/963553035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80c3970 0x7f75c80acb60 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f75bc02b950 tx=0x7f75bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.143+0000 7f75c57fa640 1 -- 172.21.15.1:0/963553035 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc040070 con 0x7f75c80c3970 2024-03-06T07:21:33.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.143+0000 7f75c57fa640 1 -- 172.21.15.1:0/963553035 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75bc03a420 con 0x7f75c80c3970 2024-03-06T07:21:33.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.143+0000 7f75cec4e640 1 -- 172.21.15.1:0/963553035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c80ad260 con 0x7f75c80c3970 2024-03-06T07:21:33.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.143+0000 7f75c57fa640 1 -- 172.21.15.1:0/963553035 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc035040 con 0x7f75c80c3970 2024-03-06T07:21:33.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.143+0000 7f75cec4e640 1 -- 172.21.15.1:0/963553035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c80abad0 con 0x7f75c80c3970 2024-03-06T07:21:33.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.144+0000 7f75cec4e640 1 -- 172.21.15.1:0/963553035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c80686e0 con 0x7f75c80c3970 2024-03-06T07:21:33.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.145+0000 7f75c57fa640 1 -- 172.21.15.1:0/963553035 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75bc04b060 con 0x7f75c80c3970 2024-03-06T07:21:33.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.145+0000 7f75c57fa640 1 --2- 172.21.15.1:0/963553035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75ac06b4d0 0x7f75ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.145+0000 7f75c57fa640 1 -- 172.21.15.1:0/963553035 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f75bc0ac270 con 0x7f75c80c3970 2024-03-06T07:21:33.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.145+0000 7f75c7fff640 1 --2- 172.21.15.1:0/963553035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75ac06b4d0 0x7f75ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.146+0000 7f75c7fff640 1 --2- 172.21.15.1:0/963553035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75ac06b4d0 0x7f75ac06d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f75c814fd90 tx=0x7f75b4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.149+0000 7f75c57fa640 1 -- 172.21.15.1:0/963553035 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f75bc07aff0 con 0x7f75c80c3970 2024-03-06T07:21:33.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.372+0000 7f75cec4e640 1 -- 172.21.15.1:0/963553035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75c8150c30 con 0x7f75c80c3970 2024-03-06T07:21:33.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.373+0000 7f75c57fa640 1 -- 172.21.15.1:0/963553035 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f75bc07a8f0 con 0x7f75c80c3970 2024-03-06T07:21:33.375 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:33.375 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4781,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2438105345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2438105345},{"type":"v1","addr":"172.21.15.1:6835","nonce":2438105345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3354292028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3354292028},{"type":"v1","addr":"172.21.15.98:6836","nonce":3354292028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4975,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3948803645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3948803645},{"type":"v1","addr":"172.21.15.98:6837","nonce":3948803645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:22.528539+0000","modified":"2024-03-06T07:21:24.518239+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1818141892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1818141892},{"type":"v1","addr":"172.21.15.1:6834","nonce":1818141892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":2}]} 2024-03-06T07:21:33.375 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 16 2024-03-06T07:21:33.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.375+0000 7f75cec4e640 1 -- 172.21.15.1:0/963553035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75ac06b4d0 msgr2=0x7f75ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.375+0000 7f75cec4e640 1 --2- 172.21.15.1:0/963553035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75ac06b4d0 0x7f75ac06d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f75c814fd90 tx=0x7f75b4008040 comp rx=0 tx=0).stop 2024-03-06T07:21:33.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.375+0000 7f75cec4e640 1 -- 172.21.15.1:0/963553035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80c3970 msgr2=0x7f75c80acb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.375+0000 7f75cec4e640 1 --2- 172.21.15.1:0/963553035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80c3970 0x7f75c80acb60 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f75bc02b950 tx=0x7f75bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.375+0000 7f75cec4e640 1 -- 172.21.15.1:0/963553035 shutdown_connections 2024-03-06T07:21:33.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.375+0000 7f75cec4e640 1 --2- 172.21.15.1:0/963553035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75ac06b4d0 0x7f75ac06d990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.375+0000 7f75cec4e640 1 --2- 172.21.15.1:0/963553035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75c8152670 0x7f75c81ee480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.375+0000 7f75cec4e640 1 --2- 172.21.15.1:0/963553035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f75c80c60b0 0x7f75c8155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.375+0000 7f75cec4e640 1 --2- 172.21.15.1:0/963553035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80c3970 0x7f75c80acb60 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.375+0000 7f75cec4e640 1 -- 172.21.15.1:0/963553035 >> 172.21.15.1:0/963553035 conn(0x7f75c80c88f0 msgr2=0x7f75c80c8d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:33.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.375+0000 7f75cec4e640 1 -- 172.21.15.1:0/963553035 shutdown_connections 2024-03-06T07:21:33.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.375+0000 7f75cec4e640 1 -- 172.21.15.1:0/963553035 wait complete. 2024-03-06T07:21:33.795 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:21:22.528539+0000', 'modified': '2024-03-06T07:21:24.518239+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5280}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5280': {'gid': 5280, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6834/1818141892', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1818141892}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1818141892}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5280, 'qdb_cluster': [5280]} 2024-03-06T07:21:33.796 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:21:33.796 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:21:33.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.899+0000 7ff4f9d65640 1 Processor -- start 2024-03-06T07:21:33.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.899+0000 7ff4f9d65640 1 -- start start 2024-03-06T07:21:33.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.899+0000 7ff4f9d65640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff4f40bf9d0 0x7ff4f4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.899+0000 7ff4f9d65640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f40c2fc0 con 0x7ff4f4152650 2024-03-06T07:21:33.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.899+0000 7ff4f9d65640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f4158da0 con 0x7ff4f40c6b10 2024-03-06T07:21:33.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.899+0000 7ff4f9d65640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4f4155800 con 0x7ff4f40bf9d0 2024-03-06T07:21:33.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.900+0000 7ff4f3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff4f40bf9d0 0x7ff4f4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.900+0000 7ff4f3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff4f40bf9d0 0x7ff4f4155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37228/0 (socket says 172.21.15.1:37228) 2024-03-06T07:21:33.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.900+0000 7ff4f37fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff4f4152650 0x7ff4f4158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60828/0 (socket says 172.21.15.1:60828) 2024-03-06T07:21:33.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.900+0000 7ff4f3fff640 1 -- 172.21.15.1:0/861678719 learned_addr learned my addr 172.21.15.1:0/861678719 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:33.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.900+0000 7ff4f3fff640 1 -- 172.21.15.1:0/861678719 >> v1:172.21.15.98:6789/0 conn(0x7ff4f40c6b10 legacy=0x7ff4f40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.900+0000 7ff4f3fff640 1 -- 172.21.15.1:0/861678719 >> v1:172.21.15.1:6789/0 conn(0x7ff4f4152650 legacy=0x7ff4f4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.900+0000 7ff4f27fc640 1 -- 172.21.15.1:0/861678719 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931574701 0 0) 0x7ff4f40c2fc0 con 0x7ff4f4152650 2024-03-06T07:21:33.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.900+0000 7ff4f3fff640 1 -- 172.21.15.1:0/861678719 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f40c6690 con 0x7ff4f40bf9d0 2024-03-06T07:21:33.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.901+0000 7ff4f3fff640 1 --2- 172.21.15.1:0/861678719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4f40bf9d0 0x7ff4f4155c90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec009e50 tx=0x7ff4ec02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83544a23984feb04 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.901+0000 7ff4f27fc640 1 -- 172.21.15.1:0/861678719 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec02cb00 con 0x7ff4f40bf9d0 2024-03-06T07:21:33.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.901+0000 7ff4f27fc640 1 -- 172.21.15.1:0/861678719 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4ec02cca0 con 0x7ff4f40bf9d0 2024-03-06T07:21:33.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.901+0000 7ff4f27fc640 1 -- 172.21.15.1:0/861678719 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec034610 con 0x7ff4f40bf9d0 2024-03-06T07:21:33.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.902+0000 7ff4f9d65640 1 -- 172.21.15.1:0/861678719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4f40bf9d0 msgr2=0x7ff4f4155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.902+0000 7ff4f9d65640 1 --2- 172.21.15.1:0/861678719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4f40bf9d0 0x7ff4f4155c90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec009e50 tx=0x7ff4ec02bf60 comp rx=0 tx=0).stop 2024-03-06T07:21:33.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.902+0000 7ff4f9d65640 1 -- 172.21.15.1:0/861678719 shutdown_connections 2024-03-06T07:21:33.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.902+0000 7ff4f9d65640 1 --2- 172.21.15.1:0/861678719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4f40bf9d0 0x7ff4f4155c90 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.902+0000 7ff4f9d65640 1 -- 172.21.15.1:0/861678719 >> 172.21.15.1:0/861678719 conn(0x7ff4f4066ee0 msgr2=0x7ff4f40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:33.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.902+0000 7ff4f9d65640 1 -- 172.21.15.1:0/861678719 shutdown_connections 2024-03-06T07:21:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.902+0000 7ff4f9d65640 1 -- 172.21.15.1:0/861678719 wait complete. 2024-03-06T07:21:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.903+0000 7ff4f9d65640 1 Processor -- start 2024-03-06T07:21:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.904+0000 7ff4f9d65640 1 -- start start 2024-03-06T07:21:33.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f9d65640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4f40bf9d0 0x7ff4f4157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f9d65640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4f4152650 0x7ff4f414c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f9d65640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4f4158920 0x7ff4f414cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f9d65640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f41551b0 con 0x7ff4f40bf9d0 2024-03-06T07:21:33.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f9d65640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f40c6fa0 con 0x7ff4f4158920 2024-03-06T07:21:33.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f9d65640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f4155490 con 0x7ff4f4152650 2024-03-06T07:21:33.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4f40bf9d0 0x7ff4f4157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4f40bf9d0 0x7ff4f4157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37234/0 (socket says 172.21.15.1:37234) 2024-03-06T07:21:33.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f37fe640 1 -- 172.21.15.1:0/1213256476 learned_addr learned my addr 172.21.15.1:0/1213256476 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:33.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f3fff640 1 --2- 172.21.15.1:0/1213256476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4f4158920 0x7ff4f414cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f2ffd640 1 --2- 172.21.15.1:0/1213256476 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4f4152650 0x7ff4f414c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f37fe640 1 -- 172.21.15.1:0/1213256476 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4f4152650 msgr2=0x7ff4f414c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f37fe640 1 --2- 172.21.15.1:0/1213256476 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4f4152650 0x7ff4f414c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f37fe640 1 -- 172.21.15.1:0/1213256476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4f4158920 msgr2=0x7ff4f414cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f37fe640 1 --2- 172.21.15.1:0/1213256476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4f4158920 0x7ff4f414cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f37fe640 1 -- 172.21.15.1:0/1213256476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f414d010 con 0x7ff4f40bf9d0 2024-03-06T07:21:33.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f2ffd640 1 --2- 172.21.15.1:0/1213256476 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4f4152650 0x7ff4f414c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:33.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f37fe640 1 --2- 172.21.15.1:0/1213256476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4f40bf9d0 0x7ff4f4157a20 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff4e400c4a0 tx=0x7ff4e400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4dbfff640 1 -- 172.21.15.1:0/1213256476 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e4018070 con 0x7ff4f40bf9d0 2024-03-06T07:21:33.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4dbfff640 1 -- 172.21.15.1:0/1213256476 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e40074e0 con 0x7ff4f40bf9d0 2024-03-06T07:21:33.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4f9d65640 1 -- 172.21.15.1:0/1213256476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f414d490 con 0x7ff4f40bf9d0 2024-03-06T07:21:33.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.905+0000 7ff4dbfff640 1 -- 172.21.15.1:0/1213256476 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e4004a20 con 0x7ff4f40bf9d0 2024-03-06T07:21:33.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.906+0000 7ff4f9d65640 1 -- 172.21.15.1:0/1213256476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f41541f0 con 0x7ff4f40bf9d0 2024-03-06T07:21:33.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.907+0000 7ff4dbfff640 1 -- 172.21.15.1:0/1213256476 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4e401b070 con 0x7ff4f40bf9d0 2024-03-06T07:21:33.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.908+0000 7ff4dbfff640 1 --2- 172.21.15.1:0/1213256476 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4e006b500 0x7ff4e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.908+0000 7ff4dbfff640 1 -- 172.21.15.1:0/1213256476 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff4e40895b0 con 0x7ff4f40bf9d0 2024-03-06T07:21:33.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.908+0000 7ff4f2ffd640 1 --2- 172.21.15.1:0/1213256476 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4e006b500 0x7ff4e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.908+0000 7ff4fca2c740 1 Processor -- start 2024-03-06T07:21:33.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.908+0000 7ff4fca2c740 1 -- start start 2024-03-06T07:21:33.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.908+0000 7ff4f2ffd640 1 --2- 172.21.15.1:0/1213256476 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4e006b500 0x7ff4e006d9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40b04c0 tx=0x7ff4e8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.909+0000 7ff4fca2c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254317f50 0x55a254315a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.909+0000 7ff4fca2c740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254318330 0x55a254315f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.909+0000 7ff4fca2c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2543164b0 0x55a254312680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.909+0000 7ff4fca2c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a25427c6b0 con 0x55a254317f50 2024-03-06T07:21:33.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.909+0000 7ff4fca2c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a254225a60 con 0x55a2543164b0 2024-03-06T07:21:33.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.909+0000 7ff4fca2c740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a254278750 con 0x55a254318330 2024-03-06T07:21:33.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.909+0000 7ff4f37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254317f50 0x55a254315a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.909+0000 7ff4f37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254317f50 0x55a254315a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37242/0 (socket says 172.21.15.1:37242) 2024-03-06T07:21:33.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.909+0000 7ff4f37fe640 1 -- 172.21.15.1:0/2383779409 learned_addr learned my addr 172.21.15.1:0/2383779409 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:33.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.909+0000 7ff4f3fff640 1 --2- 172.21.15.1:0/2383779409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2543164b0 0x55a254312680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.909+0000 7ff4f2ffd640 1 --2- 172.21.15.1:0/2383779409 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254318330 0x55a254315f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.910+0000 7ff4f37fe640 1 -- 172.21.15.1:0/2383779409 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254318330 msgr2=0x55a254315f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:33.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.910+0000 7ff4f37fe640 1 --2- 172.21.15.1:0/2383779409 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254318330 0x55a254315f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.910+0000 7ff4f37fe640 1 -- 172.21.15.1:0/2383779409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2543164b0 msgr2=0x55a254312680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:33.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.910+0000 7ff4f37fe640 1 --2- 172.21.15.1:0/2383779409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2543164b0 0x55a254312680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.910+0000 7ff4f37fe640 1 -- 172.21.15.1:0/2383779409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a254243f30 con 0x55a254317f50 2024-03-06T07:21:33.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.910+0000 7ff4f2ffd640 1 --2- 172.21.15.1:0/2383779409 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254318330 0x55a254315f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:33.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.910+0000 7ff4f37fe640 1 --2- 172.21.15.1:0/2383779409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254317f50 0x55a254315a30 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ff4e40879e0 tx=0x7ff4e40583f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f419b9bcb97f3428 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.910+0000 7ff4d97fa640 1 -- 172.21.15.1:0/2383779409 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e4060340 con 0x55a254317f50 2024-03-06T07:21:33.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.910+0000 7ff4d97fa640 1 -- 172.21.15.1:0/2383779409 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e4055f30 con 0x55a254317f50 2024-03-06T07:21:33.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.910+0000 7ff4d97fa640 1 -- 172.21.15.1:0/2383779409 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e4004bc0 con 0x55a254317f50 2024-03-06T07:21:33.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.911+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2383779409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254317f50 msgr2=0x55a254315a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.911+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/2383779409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254317f50 0x55a254315a30 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ff4e40879e0 tx=0x7ff4e40583f0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.911+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2383779409 shutdown_connections 2024-03-06T07:21:33.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.911+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/2383779409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2543164b0 0x55a254312680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.911+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/2383779409 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254318330 0x55a254315f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.911+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/2383779409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254317f50 0x55a254315a30 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.911+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2383779409 >> 172.21.15.1:0/2383779409 conn(0x55a2543147d0 msgr2=0x55a2543393c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:33.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.911+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2383779409 shutdown_connections 2024-03-06T07:21:33.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.911+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2383779409 wait complete. 2024-03-06T07:21:33.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.912+0000 7ff4fca2c740 1 -- start start 2024-03-06T07:21:33.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.912+0000 7ff4fca2c740 1 Processor -- start 2024-03-06T07:21:33.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.913+0000 7ff4fca2c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a254317f50 0x55a2543129d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.913+0000 7ff4fca2c740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254318330 0x55a254368580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.913+0000 7ff4fca2c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254368ac0 0x55a254333a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.913+0000 7ff4fca2c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a25433a230 con 0x55a254368ac0 2024-03-06T07:21:33.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.913+0000 7ff4fca2c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a254238ea0 con 0x55a254317f50 2024-03-06T07:21:33.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.913+0000 7ff4fca2c740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a254339c70 con 0x55a254318330 2024-03-06T07:21:33.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.913+0000 7ff4fca2c740 10 client.0 _ll_register_callbacks cb 0x55a254238870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:21:33.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.913+0000 7ff4f3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254368ac0 0x55a254333a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.913+0000 7ff4f3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254368ac0 0x55a254333a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37254/0 (socket says 172.21.15.1:37254) 2024-03-06T07:21:33.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.913+0000 7ff4f3fff640 1 -- 172.21.15.1:0/2340958254 learned_addr learned my addr 172.21.15.1:0/2340958254 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:33.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.913+0000 7ff4f2ffd640 1 --2- 172.21.15.1:0/2340958254 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254318330 0x55a254368580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.913+0000 7ff4f37fe640 1 --2- 172.21.15.1:0/2340958254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a254317f50 0x55a2543129d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.913+0000 7ff4f3fff640 1 -- 172.21.15.1:0/2340958254 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254318330 msgr2=0x55a254368580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:33.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.913+0000 7ff4f3fff640 1 --2- 172.21.15.1:0/2340958254 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254318330 0x55a254368580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.913+0000 7ff4f3fff640 1 -- 172.21.15.1:0/2340958254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a254317f50 msgr2=0x55a2543129d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:33.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.913+0000 7ff4f3fff640 1 --2- 172.21.15.1:0/2340958254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a254317f50 0x55a2543129d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.913+0000 7ff4f3fff640 1 -- 172.21.15.1:0/2340958254 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a254239270 con 0x55a254368ac0 2024-03-06T07:21:33.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.914+0000 7ff4f3fff640 1 --2- 172.21.15.1:0/2340958254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254368ac0 0x55a254333a60 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec009e50 tx=0x7ff4ec004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f9195147f60f43c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.914+0000 7ff4d27fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:21:33.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.914+0000 7ff4d27fc640 1 -- 172.21.15.1:0/2340958254 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec034b30 con 0x55a254368ac0 2024-03-06T07:21:33.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.914+0000 7ff4d27fc640 1 -- 172.21.15.1:0/2340958254 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff4ec034cd0 con 0x55a254368ac0 2024-03-06T07:21:33.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.914+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2340958254 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff4e4055f30 con 0x55a254368ac0 2024-03-06T07:21:33.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.914+0000 7ff4d27fc640 1 -- 172.21.15.1:0/2340958254 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4ec003990 con 0x55a254368ac0 2024-03-06T07:21:33.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.914+0000 7ff4fca2c740 10 client.5523 fetch_fsmap learned FSMap version 16 2024-03-06T07:21:33.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.914+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2340958254 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55a253f912d0 con 0x55a254368ac0 2024-03-06T07:21:33.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.914+0000 7ff4d27fc640 1 -- 172.21.15.1:0/2340958254 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e4055f30 con 0x55a254368ac0 2024-03-06T07:21:33.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.914+0000 7ff4d27fc640 1 -- 172.21.15.1:0/2340958254 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 16) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff4ec003b30 con 0x55a254368ac0 2024-03-06T07:21:33.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.914+0000 7ff4fca2c740 10 client.5523 fetch_fsmap finished waiting for FSMap version 16 2024-03-06T07:21:33.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.914+0000 7ff4fca2c740 10 client.5523 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:21:33.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.914+0000 7ff4fca2c740 20 client.5523 populate_metadata read hostname 'smithi001' 2024-03-06T07:21:33.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.914+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/2340958254 >> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] conn(0x55a254338c90 0x55a254373400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.914+0000 7ff4fca2c740 4 client.5523 mds_command: new command op to 5280 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:21:33.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.914+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2340958254 --> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a254289bf0 con 0x55a254338c90 2024-03-06T07:21:33.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.914+0000 7ff4f37fe640 1 --2- 172.21.15.1:0/2340958254 >> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] conn(0x55a254338c90 0x55a254373400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.915+0000 7ff4f37fe640 1 --2- 172.21.15.1:0/2340958254 >> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] conn(0x55a254338c90 0x55a254373400 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e51835929c6d30fa server_cookie=9b6c7c5d7102367f in_seq=0 out_seq=0 2024-03-06T07:21:33.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.915+0000 7ff4d27fc640 10 client.5523 ms_handle_connect on v2:172.21.15.1:6832/1818141892 2024-03-06T07:21:33.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.915+0000 7ff4d27fc640 1 -- 172.21.15.1:0/2340958254 <== mds.0 v2:172.21.15.1:6832/1818141892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a254289bf0 con 0x55a254338c90 2024-03-06T07:21:33.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.915+0000 7ff4d27fc640 10 client.5523 handle_command_reply: tid=0 2024-03-06T07:21:33.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.915+0000 7ff4fca2c740 1 client.5523 shutdown 2024-03-06T07:21:33.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.915+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2340958254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254368ac0 msgr2=0x55a254333a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.915+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/2340958254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254368ac0 0x55a254333a60 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec009e50 tx=0x7ff4ec004ab0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.915+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2340958254 shutdown_connections 2024-03-06T07:21:33.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.915+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/2340958254 >> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] conn(0x55a254338c90 0x55a254373400 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.915+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/2340958254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254368ac0 0x55a254333a60 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.915+0000 7ff4d27fc640 0 client.5523 ms_handle_reset on v2:172.21.15.1:6832/1818141892 2024-03-06T07:21:33.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.915+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/2340958254 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254318330 0x55a254368580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.915+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/2340958254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a254317f50 0x55a2543129d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.915+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2340958254 >> 172.21.15.1:0/2340958254 conn(0x55a2543164b0 msgr2=0x55a254354b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:33.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.916+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2340958254 shutdown_connections 2024-03-06T07:21:33.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.916+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2340958254 wait complete. 2024-03-06T07:21:33.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.916+0000 7ff4fca2c740 20 client.5523 trim_cache size 0 max 16384 2024-03-06T07:21:33.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.932+0000 7ff4fca2c740 1 Processor -- start 2024-03-06T07:21:33.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.932+0000 7ff4fca2c740 1 -- start start 2024-03-06T07:21:33.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.933+0000 7ff4fca2c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a254317f50 0x55a25433d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.933+0000 7ff4fca2c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254318330 0x55a2543136f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.933+0000 7ff4fca2c740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254313c30 0x55a254357650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.933+0000 7ff4fca2c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f41f6db0 con 0x55a254318330 2024-03-06T07:21:33.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.933+0000 7ff4fca2c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f41e2450 con 0x55a254317f50 2024-03-06T07:21:33.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.933+0000 7ff4fca2c740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f41f67b0 con 0x55a254313c30 2024-03-06T07:21:33.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.933+0000 7ff4f2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254318330 0x55a2543136f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.933+0000 7ff4f2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254318330 0x55a2543136f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37264/0 (socket says 172.21.15.1:37264) 2024-03-06T07:21:33.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.933+0000 7ff4f2ffd640 1 -- 172.21.15.1:0/2679161499 learned_addr learned my addr 172.21.15.1:0/2679161499 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:33.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.934+0000 7ff4f37fe640 1 --2- 172.21.15.1:0/2679161499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a254317f50 0x55a25433d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.934+0000 7ff4f3fff640 1 --2- 172.21.15.1:0/2679161499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254313c30 0x55a254357650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.934+0000 7ff4f2ffd640 1 -- 172.21.15.1:0/2679161499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254313c30 msgr2=0x55a254357650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:33.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.934+0000 7ff4f2ffd640 1 --2- 172.21.15.1:0/2679161499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254313c30 0x55a254357650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.934+0000 7ff4f2ffd640 1 -- 172.21.15.1:0/2679161499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a254317f50 msgr2=0x55a25433d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:33.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.934+0000 7ff4f2ffd640 1 --2- 172.21.15.1:0/2679161499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a254317f50 0x55a25433d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.934+0000 7ff4f2ffd640 1 -- 172.21.15.1:0/2679161499 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a25405b6b0 con 0x55a254318330 2024-03-06T07:21:33.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.934+0000 7ff4f37fe640 1 --2- 172.21.15.1:0/2679161499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a254317f50 0x55a25433d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:33.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.934+0000 7ff4f3fff640 1 --2- 172.21.15.1:0/2679161499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254313c30 0x55a254357650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:33.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.934+0000 7ff4f2ffd640 1 --2- 172.21.15.1:0/2679161499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254318330 0x55a2543136f0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800d950 tx=0x7ff4e8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c97435c0b394b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.934+0000 7ff4d8ff9640 1 -- 172.21.15.1:0/2679161499 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e801d040 con 0x55a254318330 2024-03-06T07:21:33.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.934+0000 7ff4d8ff9640 1 -- 172.21.15.1:0/2679161499 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e8026070 con 0x55a254318330 2024-03-06T07:21:33.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.934+0000 7ff4d8ff9640 1 -- 172.21.15.1:0/2679161499 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8014da0 con 0x55a254318330 2024-03-06T07:21:33.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.935+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2679161499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254318330 msgr2=0x55a2543136f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.935+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/2679161499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254318330 0x55a2543136f0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800d950 tx=0x7ff4e8013000 comp rx=0 tx=0).stop 2024-03-06T07:21:33.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.935+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2679161499 shutdown_connections 2024-03-06T07:21:33.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.935+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/2679161499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254313c30 0x55a254357650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.935+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/2679161499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254318330 0x55a2543136f0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.935+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/2679161499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a254317f50 0x55a25433d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.935+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2679161499 >> 172.21.15.1:0/2679161499 conn(0x55a2543164b0 msgr2=0x55a2543156d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:33.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.935+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2679161499 shutdown_connections 2024-03-06T07:21:33.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.935+0000 7ff4fca2c740 1 -- 172.21.15.1:0/2679161499 wait complete. 2024-03-06T07:21:33.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.935+0000 7ff4fca2c740 1 -- start start 2024-03-06T07:21:33.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.935+0000 7ff4fca2c740 1 Processor -- start 2024-03-06T07:21:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4fca2c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254313c30 0x55a254315770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4fca2c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a254317f50 0x55a254357650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4fca2c740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254318330 0x55a2543c11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4fca2c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2543649f0 con 0x55a254313c30 2024-03-06T07:21:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4fca2c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a254351c20 con 0x55a254317f50 2024-03-06T07:21:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4fca2c740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a25433b610 con 0x55a254318330 2024-03-06T07:21:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4fca2c740 10 client.0 _ll_register_callbacks cb 0x7ff4ec02cca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:21:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4f37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254313c30 0x55a254315770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4f37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254313c30 0x55a254315770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37272/0 (socket says 172.21.15.1:37272) 2024-03-06T07:21:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4f37fe640 1 -- 172.21.15.1:0/4175856794 learned_addr learned my addr 172.21.15.1:0/4175856794 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4f2ffd640 1 --2- 172.21.15.1:0/4175856794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a254317f50 0x55a254357650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4f3fff640 1 --2- 172.21.15.1:0/4175856794 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254318330 0x55a2543c11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4f37fe640 1 -- 172.21.15.1:0/4175856794 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254318330 msgr2=0x55a2543c11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4f37fe640 1 --2- 172.21.15.1:0/4175856794 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254318330 0x55a2543c11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4f37fe640 1 -- 172.21.15.1:0/4175856794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a254317f50 msgr2=0x55a254357650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4f37fe640 1 --2- 172.21.15.1:0/4175856794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a254317f50 0x55a254357650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4f37fe640 1 -- 172.21.15.1:0/4175856794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a2543c3f50 con 0x55a254313c30 2024-03-06T07:21:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4f3fff640 1 --2- 172.21.15.1:0/4175856794 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254318330 0x55a2543c11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4f37fe640 1 --2- 172.21.15.1:0/4175856794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254313c30 0x55a254315770 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4057b70 tx=0x7ff4e4089050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c46df2e10b1be03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.936+0000 7ff4d27fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:21:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.937+0000 7ff4d27fc640 1 -- 172.21.15.1:0/4175856794 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e4060340 con 0x55a254313c30 2024-03-06T07:21:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.937+0000 7ff4d27fc640 1 -- 172.21.15.1:0/4175856794 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff4e40542a0 con 0x55a254313c30 2024-03-06T07:21:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.937+0000 7ff4fca2c740 1 -- 172.21.15.1:0/4175856794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff4e8026070 con 0x55a254313c30 2024-03-06T07:21:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.937+0000 7ff4d27fc640 1 -- 172.21.15.1:0/4175856794 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4e4004bc0 con 0x55a254313c30 2024-03-06T07:21:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.937+0000 7ff4fca2c740 10 client.5529 fetch_fsmap learned FSMap version 16 2024-03-06T07:21:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.937+0000 7ff4fca2c740 1 -- 172.21.15.1:0/4175856794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55a254377b90 con 0x55a254313c30 2024-03-06T07:21:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.937+0000 7ff4d27fc640 1 -- 172.21.15.1:0/4175856794 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e8026070 con 0x55a254313c30 2024-03-06T07:21:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.937+0000 7ff4d27fc640 1 -- 172.21.15.1:0/4175856794 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 16) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff4e4057080 con 0x55a254313c30 2024-03-06T07:21:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.937+0000 7ff4fca2c740 10 client.5529 fetch_fsmap finished waiting for FSMap version 16 2024-03-06T07:21:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.937+0000 7ff4fca2c740 10 client.5529 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:21:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.937+0000 7ff4fca2c740 20 client.5529 populate_metadata read hostname 'smithi001' 2024-03-06T07:21:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.937+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/4175856794 >> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] conn(0x55a2543c1910 0x55a254360a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.937+0000 7ff4fca2c740 4 client.5529 mds_command: new command op to 5280 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:21:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.937+0000 7ff4fca2c740 1 -- 172.21.15.1:0/4175856794 --> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a254283f40 con 0x55a2543c1910 2024-03-06T07:21:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.937+0000 7ff4f2ffd640 1 --2- 172.21.15.1:0/4175856794 >> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] conn(0x55a2543c1910 0x55a254360a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.938+0000 7ff4d27fc640 10 client.5529 ms_handle_connect on v2:172.21.15.1:6832/1818141892 2024-03-06T07:21:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.938+0000 7ff4f2ffd640 1 --2- 172.21.15.1:0/4175856794 >> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] conn(0x55a2543c1910 0x55a254360a20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f0d98ca6f2038ba server_cookie=214dfe7c89c5eaf3 in_seq=0 out_seq=0 2024-03-06T07:21:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.938+0000 7ff4d27fc640 1 -- 172.21.15.1:0/4175856794 <== mds.0 v2:172.21.15.1:6832/1818141892 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55a254283f40 con 0x55a2543c1910 2024-03-06T07:21:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.938+0000 7ff4d27fc640 10 client.5529 handle_command_reply: tid=0 2024-03-06T07:21:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.938+0000 7ff4fca2c740 1 client.5529 shutdown 2024-03-06T07:21:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.938+0000 7ff4fca2c740 1 -- 172.21.15.1:0/4175856794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254313c30 msgr2=0x55a254315770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.938+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/4175856794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254313c30 0x55a254315770 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4057b70 tx=0x7ff4e4089050 comp rx=0 tx=0).stop 2024-03-06T07:21:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.938+0000 7ff4fca2c740 1 -- 172.21.15.1:0/4175856794 shutdown_connections 2024-03-06T07:21:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.938+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/4175856794 >> [v2:172.21.15.1:6832/1818141892,v1:172.21.15.1:6834/1818141892] conn(0x55a2543c1910 0x55a254360a20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.938+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/4175856794 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a254318330 0x55a2543c11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.938+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/4175856794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a254317f50 0x55a254357650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.938+0000 7ff4fca2c740 1 --2- 172.21.15.1:0/4175856794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a254313c30 0x55a254315770 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.938+0000 7ff4fca2c740 1 -- 172.21.15.1:0/4175856794 >> 172.21.15.1:0/4175856794 conn(0x55a2543164b0 msgr2=0x55a254313520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.938+0000 7ff4d27fc640 0 client.5529 ms_handle_reset on v2:172.21.15.1:6832/1818141892 2024-03-06T07:21:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.938+0000 7ff4fca2c740 1 -- 172.21.15.1:0/4175856794 shutdown_connections 2024-03-06T07:21:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.938+0000 7ff4fca2c740 1 -- 172.21.15.1:0/4175856794 wait complete. 2024-03-06T07:21:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.938+0000 7ff4fca2c740 20 client.5529 trim_cache size 0 max 16384 2024-03-06T07:21:33.947 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:21:33.947 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:21:33.947 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:21:33.947 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 5280, 2024-03-06T07:21:33.947 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:21:33.947 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:21:33.947 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:21:33.947 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 10.777023466999999, 2024-03-06T07:21:33.948 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 16, 2024-03-06T07:21:33.948 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 39, 2024-03-06T07:21:33.948 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 39, 2024-03-06T07:21:33.948 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 17.651344601000002 2024-03-06T07:21:33.948 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:21:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.939+0000 7ff4f9d65640 1 -- 172.21.15.1:0/1213256476 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4e006b500 msgr2=0x7ff4e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.939+0000 7ff4f9d65640 1 --2- 172.21.15.1:0/1213256476 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4e006b500 0x7ff4e006d9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40b04c0 tx=0x7ff4e8009040 comp rx=0 tx=0).stop 2024-03-06T07:21:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.939+0000 7ff4f9d65640 1 -- 172.21.15.1:0/1213256476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4f40bf9d0 msgr2=0x7ff4f4157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.939+0000 7ff4f9d65640 1 --2- 172.21.15.1:0/1213256476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4f40bf9d0 0x7ff4f4157a20 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff4e400c4a0 tx=0x7ff4e400c960 comp rx=0 tx=0).stop 2024-03-06T07:21:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.939+0000 7ff4f9d65640 1 -- 172.21.15.1:0/1213256476 shutdown_connections 2024-03-06T07:21:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.939+0000 7ff4f9d65640 1 --2- 172.21.15.1:0/1213256476 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4e006b500 0x7ff4e006d9c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.939+0000 7ff4f9d65640 1 --2- 172.21.15.1:0/1213256476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4f4158920 0x7ff4f414cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.939+0000 7ff4f9d65640 1 --2- 172.21.15.1:0/1213256476 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4f4152650 0x7ff4f414c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.939+0000 7ff4f9d65640 1 --2- 172.21.15.1:0/1213256476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4f40bf9d0 0x7ff4f4157a20 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.939+0000 7ff4f9d65640 1 -- 172.21.15.1:0/1213256476 >> 172.21.15.1:0/1213256476 conn(0x7ff4f4066ee0 msgr2=0x7ff4f4151870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.939+0000 7ff4f9d65640 1 -- 172.21.15.1:0/1213256476 shutdown_connections 2024-03-06T07:21:33.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:33.939+0000 7ff4f9d65640 1 -- 172.21.15.1:0/1213256476 wait complete. 2024-03-06T07:21:33.951 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 16, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:21:22.528539+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 16, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5280": { "addr": "172.21.15.1:6834/1818141892", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 1818141892, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 1818141892, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5280, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-03-06T07:21:24.518239+0000", "qdb_cluster": [ 5280 ], "qdb_leader": 5280, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5280 } } } ], "standbys": [ { "addr": "172.21.15.1:6835/2438105345", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 2438105345, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 2438105345, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4781, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6836/3354292028", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 3354292028, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 3354292028, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4972, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6837/3948803645", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 3948803645, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 3948803645, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4975, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:21:33.953 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 2, 'name': 'c', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:21:33.953 INFO:tasks.cephfs.test_failover:current = [{'gid': 5280, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6834/1818141892', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1818141892}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1818141892}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 4975, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/3948803645', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3948803645}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3948803645}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 4781, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/2438105345', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 2438105345}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2438105345}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 4972, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/3354292028', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 3354292028}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 3354292028}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}] 2024-03-06T07:21:33.953 DEBUG:tasks.cephfs.test_failover:2 != -1 2024-03-06T07:21:33.953 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:21:37.202 INFO:tasks.ceph.mds.a.smithi001.stderr: -404> 2024-03-06T07:21:27.133+0000 7efd774c5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:37.226 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:21:38.954 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:39.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.065+0000 7fad78dc9640 1 Processor -- start 2024-03-06T07:21:39.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.066+0000 7fad78dc9640 1 -- start start 2024-03-06T07:21:39.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.066+0000 7fad78dc9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad74149a90 0x7fad740c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.066+0000 7fad78dc9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad740c1260 0x7fad740c16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.067+0000 7fad78dc9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad741530a0 con 0x7fad741565a0 2024-03-06T07:21:39.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.067+0000 7fad78dc9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad74153380 con 0x7fad740c1260 2024-03-06T07:21:39.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.067+0000 7fad78dc9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad74153660 con 0x7fad74149a90 2024-03-06T07:21:39.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.067+0000 7fad71d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad740c1260 0x7fad740c16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.067+0000 7fad71d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad740c1260 0x7fad740c16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45064/0 (socket says 172.21.15.1:45064) 2024-03-06T07:21:39.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.067+0000 7fad71d9b640 1 -- 172.21.15.1:0/2589632829 learned_addr learned my addr 172.21.15.1:0/2589632829 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:39.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.067+0000 7fad7259c640 1 --2- 172.21.15.1:0/2589632829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad74149a90 0x7fad740c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.067+0000 7fad71d9b640 1 -- 172.21.15.1:0/2589632829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad74149a90 msgr2=0x7fad740c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:39.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.067+0000 7fad71d9b640 1 --2- 172.21.15.1:0/2589632829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad74149a90 0x7fad740c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.067+0000 7fad71d9b640 1 -- 172.21.15.1:0/2589632829 >> v1:172.21.15.98:6789/0 conn(0x7fad741565a0 legacy=0x7fad741569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.067+0000 7fad71d9b640 1 -- 172.21.15.1:0/2589632829 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad74153940 con 0x7fad740c1260 2024-03-06T07:21:39.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.068+0000 7fad71d9b640 1 --2- 172.21.15.1:0/2589632829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad740c1260 0x7fad740c16c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fad5c009870 tx=0x7fad5c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f3f71d2dfd0678 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:39.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.068+0000 7fad7159a640 1 -- 172.21.15.1:0/2589632829 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5c035020 con 0x7fad740c1260 2024-03-06T07:21:39.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.068+0000 7fad7159a640 1 -- 172.21.15.1:0/2589632829 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5c031640 con 0x7fad740c1260 2024-03-06T07:21:39.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.068+0000 7fad7159a640 1 -- 172.21.15.1:0/2589632829 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5c02d040 con 0x7fad740c1260 2024-03-06T07:21:39.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.068+0000 7fad78dc9640 1 -- 172.21.15.1:0/2589632829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad740c1260 msgr2=0x7fad740c16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.068+0000 7fad78dc9640 1 --2- 172.21.15.1:0/2589632829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad740c1260 0x7fad740c16c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fad5c009870 tx=0x7fad5c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.069+0000 7fad78dc9640 1 -- 172.21.15.1:0/2589632829 shutdown_connections 2024-03-06T07:21:39.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.069+0000 7fad78dc9640 1 --2- 172.21.15.1:0/2589632829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad740c1260 0x7fad740c16c0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.069+0000 7fad78dc9640 1 --2- 172.21.15.1:0/2589632829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad74149a90 0x7fad740c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.069+0000 7fad78dc9640 1 -- 172.21.15.1:0/2589632829 >> 172.21.15.1:0/2589632829 conn(0x7fad740c6860 msgr2=0x7fad740c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:39.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.069+0000 7fad78dc9640 1 -- 172.21.15.1:0/2589632829 shutdown_connections 2024-03-06T07:21:39.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.069+0000 7fad78dc9640 1 -- 172.21.15.1:0/2589632829 wait complete. 2024-03-06T07:21:39.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.070+0000 7fad78dc9640 1 Processor -- start 2024-03-06T07:21:39.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.070+0000 7fad78dc9640 1 -- start start 2024-03-06T07:21:39.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad78dc9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad740c1260 0x7fad741e9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad78dc9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad74149a90 0x7fad740abb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad78dc9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad741565a0 0x7fad740ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad78dc9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad741570f0 con 0x7fad740c1260 2024-03-06T07:21:39.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad78dc9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad740c4ca0 con 0x7fad741565a0 2024-03-06T07:21:39.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad78dc9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad74157270 con 0x7fad74149a90 2024-03-06T07:21:39.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad7259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad740c1260 0x7fad741e9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad7259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad740c1260 0x7fad741e9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45070/0 (socket says 172.21.15.1:45070) 2024-03-06T07:21:39.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad7259c640 1 -- 172.21.15.1:0/3462108159 learned_addr learned my addr 172.21.15.1:0/3462108159 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:39.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad72d9d640 1 --2- 172.21.15.1:0/3462108159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad741565a0 0x7fad740ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad71d9b640 1 --2- 172.21.15.1:0/3462108159 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad74149a90 0x7fad740abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad7259c640 1 -- 172.21.15.1:0/3462108159 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad74149a90 msgr2=0x7fad740abb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad7259c640 1 --2- 172.21.15.1:0/3462108159 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad74149a90 0x7fad740abb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad7259c640 1 -- 172.21.15.1:0/3462108159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad741565a0 msgr2=0x7fad740ab130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad7259c640 1 --2- 172.21.15.1:0/3462108159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad741565a0 0x7fad740ab130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad7259c640 1 -- 172.21.15.1:0/3462108159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad740ac570 con 0x7fad740c1260 2024-03-06T07:21:39.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad7259c640 1 --2- 172.21.15.1:0/3462108159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad740c1260 0x7fad741e9310 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fad6c007f20 tx=0x7fad6c00e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:39.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad6affd640 1 -- 172.21.15.1:0/3462108159 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c014620 con 0x7fad740c1260 2024-03-06T07:21:39.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad6affd640 1 -- 172.21.15.1:0/3462108159 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6c019070 con 0x7fad740c1260 2024-03-06T07:21:39.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad78dc9640 1 -- 172.21.15.1:0/3462108159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad740ac850 con 0x7fad740c1260 2024-03-06T07:21:39.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad6affd640 1 -- 172.21.15.1:0/3462108159 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c014a00 con 0x7fad740c1260 2024-03-06T07:21:39.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.071+0000 7fad78dc9640 1 -- 172.21.15.1:0/3462108159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad741e8df0 con 0x7fad740c1260 2024-03-06T07:21:39.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.072+0000 7fad78dc9640 1 -- 172.21.15.1:0/3462108159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad74067fd0 con 0x7fad740c1260 2024-03-06T07:21:39.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.073+0000 7fad6affd640 1 -- 172.21.15.1:0/3462108159 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad6c014ba0 con 0x7fad740c1260 2024-03-06T07:21:39.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.073+0000 7fad6affd640 1 --2- 172.21.15.1:0/3462108159 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad5406b4d0 0x7fad5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.073+0000 7fad6affd640 1 -- 172.21.15.1:0/3462108159 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fad6c08ad30 con 0x7fad740c1260 2024-03-06T07:21:39.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.074+0000 7fad71d9b640 1 --2- 172.21.15.1:0/3462108159 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad5406b4d0 0x7fad5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.074+0000 7fad71d9b640 1 --2- 172.21.15.1:0/3462108159 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad5406b4d0 0x7fad5406d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fad5c036000 tx=0x7fad5c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:39.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.078+0000 7fad6affd640 1 -- 172.21.15.1:0/3462108159 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad6c0599d0 con 0x7fad740c1260 2024-03-06T07:21:39.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.302+0000 7fad78dc9640 1 -- 172.21.15.1:0/3462108159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad7406f280 con 0x7fad740c1260 2024-03-06T07:21:39.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.302+0000 7fad6affd640 1 -- 172.21.15.1:0/3462108159 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4964 (secure 0 0 0) 0x7fad6c0592d0 con 0x7fad740c1260 2024-03-06T07:21:39.304 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:39.305 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3354292028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3354292028},{"type":"v1","addr":"172.21.15.98:6836","nonce":3354292028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4975,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3948803645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3948803645},{"type":"v1","addr":"172.21.15.98:6837","nonce":3948803645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5538,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3390157192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3390157192},{"type":"v1","addr":"172.21.15.1:6834","nonce":3390157192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:22.528539+0000","modified":"2024-03-06T07:21:38.203665+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"c","rank":0,"incarnation":18,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.1:6835/2438105345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2438105345},{"type":"v1","addr":"172.21.15.1:6835","nonce":2438105345}]},"join_fscid":2,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-06T07:21:39.305 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 20 2024-03-06T07:21:39.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.304+0000 7fad78dc9640 1 -- 172.21.15.1:0/3462108159 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad5406b4d0 msgr2=0x7fad5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.304+0000 7fad78dc9640 1 --2- 172.21.15.1:0/3462108159 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad5406b4d0 0x7fad5406d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fad5c036000 tx=0x7fad5c03c040 comp rx=0 tx=0).stop 2024-03-06T07:21:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.304+0000 7fad78dc9640 1 -- 172.21.15.1:0/3462108159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad740c1260 msgr2=0x7fad741e9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.304+0000 7fad78dc9640 1 --2- 172.21.15.1:0/3462108159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad740c1260 0x7fad741e9310 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fad6c007f20 tx=0x7fad6c00e570 comp rx=0 tx=0).stop 2024-03-06T07:21:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.305+0000 7fad78dc9640 1 -- 172.21.15.1:0/3462108159 shutdown_connections 2024-03-06T07:21:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.305+0000 7fad78dc9640 1 --2- 172.21.15.1:0/3462108159 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad5406b4d0 0x7fad5406d990 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.305+0000 7fad78dc9640 1 --2- 172.21.15.1:0/3462108159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad741565a0 0x7fad740ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.305+0000 7fad78dc9640 1 --2- 172.21.15.1:0/3462108159 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad74149a90 0x7fad740abb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.305+0000 7fad78dc9640 1 --2- 172.21.15.1:0/3462108159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad740c1260 0x7fad741e9310 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.305+0000 7fad78dc9640 1 -- 172.21.15.1:0/3462108159 >> 172.21.15.1:0/3462108159 conn(0x7fad740c6860 msgr2=0x7fad740c5250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.305+0000 7fad78dc9640 1 -- 172.21.15.1:0/3462108159 shutdown_connections 2024-03-06T07:21:39.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:39.305+0000 7fad78dc9640 1 -- 172.21.15.1:0/3462108159 wait complete. 2024-03-06T07:21:39.720 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:21:22.528539+0000', 'modified': '2024-03-06T07:21:38.203665+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 40, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4781}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4781': {'gid': 4781, 'name': 'c', 'rank': 0, 'incarnation': 18, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.1:6835/2438105345', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 2438105345}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2438105345}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:21:39.720 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4781:up:rejoin 2024-03-06T07:21:40.721 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:40.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.861+0000 7f5d9bcf4640 1 Processor -- start 2024-03-06T07:21:40.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.861+0000 7f5d9bcf4640 1 -- start start 2024-03-06T07:21:40.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.861+0000 7f5d9bcf4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d940c4bf0 0x7f5d94156460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d9bcf4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d94153a00 con 0x7f5d941569a0 2024-03-06T07:21:40.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d9bcf4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d94153ce0 con 0x7f5d940a7050 2024-03-06T07:21:40.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d9bcf4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940c3cb0 con 0x7f5d940c4bf0 2024-03-06T07:21:40.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d9a291640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5d941569a0 0x7f5d9415d760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41462/0 (socket says 172.21.15.1:41462) 2024-03-06T07:21:40.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d9a291640 1 -- 172.21.15.1:0/3325761727 learned_addr learned my addr 172.21.15.1:0/3325761727 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:40.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d9928f640 1 --2- 172.21.15.1:0/3325761727 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d940c4bf0 0x7f5d94156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d98a8e640 1 -- 172.21.15.1:0/3325761727 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185733755 0 0) 0x7f5d94153a00 con 0x7f5d941569a0 2024-03-06T07:21:40.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d98a8e640 1 -- 172.21.15.1:0/3325761727 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d80003520 con 0x7f5d941569a0 2024-03-06T07:21:40.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d98a8e640 1 -- 172.21.15.1:0/3325761727 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1914499951 0 0) 0x7f5d80003520 con 0x7f5d941569a0 2024-03-06T07:21:40.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d98a8e640 1 -- 172.21.15.1:0/3325761727 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d940c4bf0 msgr2=0x7f5d94156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:40.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d98a8e640 1 --2- 172.21.15.1:0/3325761727 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d940c4bf0 0x7f5d94156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d98a8e640 1 -- 172.21.15.1:0/3325761727 >> v1:172.21.15.98:6789/0 conn(0x7f5d940a7050 legacy=0x7f5d940c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d98a8e640 1 -- 172.21.15.1:0/3325761727 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d940c3f90 con 0x7f5d941569a0 2024-03-06T07:21:40.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d98a8e640 1 -- 172.21.15.1:0/3325761727 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f5d8c0019f0 con 0x7f5d941569a0 2024-03-06T07:21:40.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d98a8e640 1 -- 172.21.15.1:0/3325761727 >> v1:172.21.15.1:6789/0 conn(0x7f5d941569a0 legacy=0x7f5d9415d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d98a8e640 1 --2- 172.21.15.1:0/3325761727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d800039e0 0x7f5d80003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d98a8e640 1 --2- 172.21.15.1:0/3325761727 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d940c4bf0 0x7f5d80004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d98a8e640 1 --2- 172.21.15.1:0/3325761727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d80004d40 0x7f5d800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d98a8e640 1 -- 172.21.15.1:0/3325761727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940c3cb0 con 0x7f5d800039e0 2024-03-06T07:21:40.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d98a8e640 1 -- 172.21.15.1:0/3325761727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8000bb50 con 0x7f5d80004d40 2024-03-06T07:21:40.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.862+0000 7f5d98a8e640 1 -- 172.21.15.1:0/3325761727 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8000bd50 con 0x7f5d940c4bf0 2024-03-06T07:21:40.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.863+0000 7f5d99a90640 1 --2- 172.21.15.1:0/3325761727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d800039e0 0x7f5d80003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.863+0000 7f5d9a291640 1 --2- 172.21.15.1:0/3325761727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d80004d40 0x7f5d800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.863+0000 7f5d9928f640 1 --2- 172.21.15.1:0/3325761727 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d940c4bf0 0x7f5d80004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.863+0000 7f5d99a90640 1 -- 172.21.15.1:0/3325761727 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d940c4bf0 msgr2=0x7f5d80004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:40.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.863+0000 7f5d99a90640 1 --2- 172.21.15.1:0/3325761727 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d940c4bf0 0x7f5d80004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.863+0000 7f5d99a90640 1 -- 172.21.15.1:0/3325761727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d80004d40 msgr2=0x7f5d800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:40.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.863+0000 7f5d99a90640 1 --2- 172.21.15.1:0/3325761727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d80004d40 0x7f5d800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.863+0000 7f5d99a90640 1 -- 172.21.15.1:0/3325761727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d80003520 con 0x7f5d800039e0 2024-03-06T07:21:40.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.864+0000 7f5d99a90640 1 --2- 172.21.15.1:0/3325761727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d800039e0 0x7f5d80003dc0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5d90001860 tx=0x7f5d80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d03a2f61f7143e68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:40.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.864+0000 7f5d98a8e640 1 -- 172.21.15.1:0/3325761727 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9002f050 con 0x7f5d800039e0 2024-03-06T07:21:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.864+0000 7f5d98a8e640 1 -- 172.21.15.1:0/3325761727 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d90033650 con 0x7f5d800039e0 2024-03-06T07:21:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.864+0000 7f5d98a8e640 1 -- 172.21.15.1:0/3325761727 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d90001430 con 0x7f5d800039e0 2024-03-06T07:21:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.864+0000 7f5d9bcf4640 1 -- 172.21.15.1:0/3325761727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d800039e0 msgr2=0x7f5d80003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.864+0000 7f5d9bcf4640 1 --2- 172.21.15.1:0/3325761727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d800039e0 0x7f5d80003dc0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5d90001860 tx=0x7f5d80005830 comp rx=0 tx=0).stop 2024-03-06T07:21:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.864+0000 7f5d9a291640 1 -- 172.21.15.1:0/3325761727 reap_dead start 2024-03-06T07:21:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.864+0000 7f5d9bcf4640 1 -- 172.21.15.1:0/3325761727 shutdown_connections 2024-03-06T07:21:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.864+0000 7f5d9bcf4640 1 -- 172.21.15.1:0/3325761727 >> 172.21.15.1:0/3325761727 conn(0x7f5d940c7a50 msgr2=0x7f5d940c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.864+0000 7f5d9bcf4640 1 -- 172.21.15.1:0/3325761727 shutdown_connections 2024-03-06T07:21:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.864+0000 7f5d9bcf4640 1 -- 172.21.15.1:0/3325761727 wait complete. 2024-03-06T07:21:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.865+0000 7f5d9bcf4640 1 Processor -- start 2024-03-06T07:21:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.866+0000 7f5d9bcf4640 1 -- start start 2024-03-06T07:21:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.866+0000 7f5d9bcf4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d800039e0 0x7f5d941df9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.866+0000 7f5d9bcf4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d940aa6e0 0x7f5d940c0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.866+0000 7f5d9bcf4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d9414eff0 0x7f5d940c1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.866+0000 7f5d9bcf4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940c2450 con 0x7f5d9414eff0 2024-03-06T07:21:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.866+0000 7f5d9bcf4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d94072f60 con 0x7f5d800039e0 2024-03-06T07:21:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.866+0000 7f5d9bcf4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940c25d0 con 0x7f5d940aa6e0 2024-03-06T07:21:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.866+0000 7f5d9a291640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d9414eff0 0x7f5d940c1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.866+0000 7f5d9a291640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d9414eff0 0x7f5d940c1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45092/0 (socket says 172.21.15.1:45092) 2024-03-06T07:21:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.866+0000 7f5d9a291640 1 -- 172.21.15.1:0/2590898812 learned_addr learned my addr 172.21.15.1:0/2590898812 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.866+0000 7f5d99a90640 1 --2- 172.21.15.1:0/2590898812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d800039e0 0x7f5d941df9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.866+0000 7f5d9928f640 1 --2- 172.21.15.1:0/2590898812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d940aa6e0 0x7f5d940c0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.867+0000 7f5d9a291640 1 -- 172.21.15.1:0/2590898812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d940aa6e0 msgr2=0x7f5d940c0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.867+0000 7f5d9a291640 1 --2- 172.21.15.1:0/2590898812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d940aa6e0 0x7f5d940c0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.867+0000 7f5d9a291640 1 -- 172.21.15.1:0/2590898812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d800039e0 msgr2=0x7f5d941df9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.867+0000 7f5d9a291640 1 --2- 172.21.15.1:0/2590898812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d800039e0 0x7f5d941df9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.867+0000 7f5d9a291640 1 -- 172.21.15.1:0/2590898812 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d941e8df0 con 0x7f5d9414eff0 2024-03-06T07:21:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.867+0000 7f5d9928f640 1 --2- 172.21.15.1:0/2590898812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d940aa6e0 0x7f5d940c0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.867+0000 7f5d9a291640 1 --2- 172.21.15.1:0/2590898812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d9414eff0 0x7f5d940c1350 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5d940c5c70 tx=0x7f5d8c0079d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:40.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.867+0000 7f5d8a7fc640 1 -- 172.21.15.1:0/2590898812 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c017020 con 0x7f5d9414eff0 2024-03-06T07:21:40.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.867+0000 7f5d8a7fc640 1 -- 172.21.15.1:0/2590898812 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8c00a9b0 con 0x7f5d9414eff0 2024-03-06T07:21:40.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.867+0000 7f5d9bcf4640 1 -- 172.21.15.1:0/2590898812 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d941e9090 con 0x7f5d9414eff0 2024-03-06T07:21:40.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.867+0000 7f5d8a7fc640 1 -- 172.21.15.1:0/2590898812 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c00adc0 con 0x7f5d9414eff0 2024-03-06T07:21:40.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.867+0000 7f5d9bcf4640 1 -- 172.21.15.1:0/2590898812 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d941e9650 con 0x7f5d9414eff0 2024-03-06T07:21:40.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.868+0000 7f5d9bcf4640 1 -- 172.21.15.1:0/2590898812 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d940686e0 con 0x7f5d9414eff0 2024-03-06T07:21:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.869+0000 7f5d8a7fc640 1 -- 172.21.15.1:0/2590898812 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d8c01a020 con 0x7f5d9414eff0 2024-03-06T07:21:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.869+0000 7f5d8a7fc640 1 --2- 172.21.15.1:0/2590898812 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d7806b500 0x7f5d7806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.870+0000 7f5d8a7fc640 1 -- 172.21.15.1:0/2590898812 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5d8c08a7e0 con 0x7f5d9414eff0 2024-03-06T07:21:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.870+0000 7f5d99a90640 1 --2- 172.21.15.1:0/2590898812 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d7806b500 0x7f5d7806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.870+0000 7f5d99a90640 1 --2- 172.21.15.1:0/2590898812 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d7806b500 0x7f5d7806d9c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5d90036000 tx=0x7f5d90031000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:40.874+0000 7f5d8a7fc640 1 -- 172.21.15.1:0/2590898812 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d8c021020 con 0x7f5d9414eff0 2024-03-06T07:21:41.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.093+0000 7f5d9bcf4640 1 -- 172.21.15.1:0/2590898812 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d94072b40 con 0x7f5d9414eff0 2024-03-06T07:21:41.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.093+0000 7f5d8a7fc640 1 -- 172.21.15.1:0/2590898812 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f5d8c059510 con 0x7f5d9414eff0 2024-03-06T07:21:41.096 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:41.096 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3354292028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3354292028},{"type":"v1","addr":"172.21.15.98:6836","nonce":3354292028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4975,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3948803645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3948803645},{"type":"v1","addr":"172.21.15.98:6837","nonce":3948803645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5538,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3390157192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3390157192},{"type":"v1","addr":"172.21.15.1:6834","nonce":3390157192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:22.528539+0000","modified":"2024-03-06T07:21:40.204549+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":9,"addr":"172.21.15.1:6835/2438105345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2438105345},{"type":"v1","addr":"172.21.15.1:6835","nonce":2438105345}]},"join_fscid":2,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4781,"qdb_cluster":[4781]},"id":2}]} 2024-03-06T07:21:41.097 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 21 2024-03-06T07:21:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.095+0000 7f5d9bcf4640 1 -- 172.21.15.1:0/2590898812 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d7806b500 msgr2=0x7f5d7806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.095+0000 7f5d9bcf4640 1 --2- 172.21.15.1:0/2590898812 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d7806b500 0x7f5d7806d9c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5d90036000 tx=0x7f5d90031000 comp rx=0 tx=0).stop 2024-03-06T07:21:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.095+0000 7f5d9bcf4640 1 -- 172.21.15.1:0/2590898812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d9414eff0 msgr2=0x7f5d940c1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.095+0000 7f5d9bcf4640 1 --2- 172.21.15.1:0/2590898812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d9414eff0 0x7f5d940c1350 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5d940c5c70 tx=0x7f5d8c0079d0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.095+0000 7f5d9bcf4640 1 -- 172.21.15.1:0/2590898812 shutdown_connections 2024-03-06T07:21:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.095+0000 7f5d9bcf4640 1 --2- 172.21.15.1:0/2590898812 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d7806b500 0x7f5d7806d9c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.095+0000 7f5d9bcf4640 1 --2- 172.21.15.1:0/2590898812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d9414eff0 0x7f5d940c1350 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.095+0000 7f5d9bcf4640 1 --2- 172.21.15.1:0/2590898812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d940aa6e0 0x7f5d940c0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.095+0000 7f5d9bcf4640 1 --2- 172.21.15.1:0/2590898812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d800039e0 0x7f5d941df9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.095+0000 7f5d9bcf4640 1 -- 172.21.15.1:0/2590898812 >> 172.21.15.1:0/2590898812 conn(0x7f5d940c7a50 msgr2=0x7f5d940c44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.095+0000 7f5d9bcf4640 1 -- 172.21.15.1:0/2590898812 shutdown_connections 2024-03-06T07:21:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.095+0000 7f5d9bcf4640 1 -- 172.21.15.1:0/2590898812 wait complete. 2024-03-06T07:21:41.515 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:21:22.528539+0000', 'modified': '2024-03-06T07:21:40.204549+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 40, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4781}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4781': {'gid': 4781, 'name': 'c', 'rank': 0, 'incarnation': 18, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.1:6835/2438105345', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 2438105345}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2438105345}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4781, 'qdb_cluster': [4781]} 2024-03-06T07:21:41.516 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:21:41.516 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:21:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.617+0000 7f74e5c39640 1 Processor -- start 2024-03-06T07:21:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.617+0000 7f74e5c39640 1 -- start start 2024-03-06T07:21:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.618+0000 7f74e5c39640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74e0152650 0x7f74e0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.618+0000 7f74e5c39640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f74e00c6b10 0x7f74e0158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.618+0000 7f74e5c39640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f74e00bf9d0 0x7f74e00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.618+0000 7f74e5c39640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74e0155800 con 0x7f74e00c6b10 2024-03-06T07:21:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.618+0000 7f74e5c39640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74e00c2fc0 con 0x7f74e00bf9d0 2024-03-06T07:21:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.618+0000 7f74e5c39640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e00c6690 con 0x7f74e0152650 2024-03-06T07:21:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.618+0000 7f74deffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f74e00c6b10 0x7f74e0158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.618+0000 7f74deffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f74e00c6b10 0x7f74e0158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45100/0 (socket says 172.21.15.1:45100) 2024-03-06T07:21:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.618+0000 7f74deffd640 1 -- 172.21.15.1:0/215028102 learned_addr learned my addr 172.21.15.1:0/215028102 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.618+0000 7f74dffff640 1 --2- 172.21.15.1:0/215028102 >> v2:172.21.15.98:3300/0 conn(0x7f74e00bf9d0 0x7f74e00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.618+0000 7f74df7fe640 1 --2- 172.21.15.1:0/215028102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74e0152650 0x7f74e0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.618+0000 7f74deffd640 1 -- 172.21.15.1:0/215028102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74e0152650 msgr2=0x7f74e0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.618+0000 7f74deffd640 1 --2- 172.21.15.1:0/215028102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74e0152650 0x7f74e0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.618+0000 7f74deffd640 1 -- 172.21.15.1:0/215028102 >> v2:172.21.15.98:3300/0 conn(0x7f74e00bf9d0 msgr2=0x7f74e00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.618+0000 7f74deffd640 1 --2- 172.21.15.1:0/215028102 >> v2:172.21.15.98:3300/0 conn(0x7f74e00bf9d0 0x7f74e00c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.618+0000 7f74deffd640 1 -- 172.21.15.1:0/215028102 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e00c7280 con 0x7f74e00c6b10 2024-03-06T07:21:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.619+0000 7f74deffd640 1 --2- 172.21.15.1:0/215028102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74e00c6b10 0x7f74e0158690 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f74e01597b0 tx=0x7f74d402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c08e5cc084cbd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.619+0000 7f74de7fc640 1 -- 172.21.15.1:0/215028102 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d402e070 con 0x7f74e00c6b10 2024-03-06T07:21:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.619+0000 7f74de7fc640 1 -- 172.21.15.1:0/215028102 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d4037070 con 0x7f74e00c6b10 2024-03-06T07:21:41.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.619+0000 7f74de7fc640 1 -- 172.21.15.1:0/215028102 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d4031a00 con 0x7f74e00c6b10 2024-03-06T07:21:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.620+0000 7f74e5c39640 1 -- 172.21.15.1:0/215028102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74e00c6b10 msgr2=0x7f74e0158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.620+0000 7f74e5c39640 1 --2- 172.21.15.1:0/215028102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74e00c6b10 0x7f74e0158690 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f74e01597b0 tx=0x7f74d402b880 comp rx=0 tx=0).stop 2024-03-06T07:21:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.620+0000 7f74e5c39640 1 -- 172.21.15.1:0/215028102 shutdown_connections 2024-03-06T07:21:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.620+0000 7f74e5c39640 1 --2- 172.21.15.1:0/215028102 >> v2:172.21.15.98:3300/0 conn(0x7f74e00bf9d0 0x7f74e00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.620+0000 7f74e5c39640 1 --2- 172.21.15.1:0/215028102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74e00c6b10 0x7f74e0158690 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.620+0000 7f74e5c39640 1 --2- 172.21.15.1:0/215028102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74e0152650 0x7f74e0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.620+0000 7f74e5c39640 1 -- 172.21.15.1:0/215028102 >> 172.21.15.1:0/215028102 conn(0x7f74e0066ee0 msgr2=0x7f74e00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.620+0000 7f74e5c39640 1 -- 172.21.15.1:0/215028102 shutdown_connections 2024-03-06T07:21:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.620+0000 7f74e5c39640 1 -- 172.21.15.1:0/215028102 wait complete. 2024-03-06T07:21:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.621+0000 7f74e5c39640 1 Processor -- start 2024-03-06T07:21:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.621+0000 7f74e5c39640 1 -- start start 2024-03-06T07:21:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74e5c39640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74e00bf9d0 0x7f74e014e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74e5c39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74e0152650 0x7f74e014c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74e5c39640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74e014c950 0x7f74e00c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74e5c39640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e0155490 con 0x7f74e014c950 2024-03-06T07:21:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74e5c39640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e01551b0 con 0x7f74e0152650 2024-03-06T07:21:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74e5c39640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e0152a30 con 0x7f74e00bf9d0 2024-03-06T07:21:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74e014c950 0x7f74e00c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74e014c950 0x7f74e00c87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45110/0 (socket says 172.21.15.1:45110) 2024-03-06T07:21:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74dffff640 1 -- 172.21.15.1:0/3825711686 learned_addr learned my addr 172.21.15.1:0/3825711686 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74df7fe640 1 --2- 172.21.15.1:0/3825711686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74e00bf9d0 0x7f74e014e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74deffd640 1 --2- 172.21.15.1:0/3825711686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74e0152650 0x7f74e014c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74dffff640 1 -- 172.21.15.1:0/3825711686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74e00bf9d0 msgr2=0x7f74e014e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74dffff640 1 --2- 172.21.15.1:0/3825711686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74e00bf9d0 0x7f74e014e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74dffff640 1 -- 172.21.15.1:0/3825711686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74e0152650 msgr2=0x7f74e014c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74dffff640 1 --2- 172.21.15.1:0/3825711686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74e0152650 0x7f74e014c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74dffff640 1 -- 172.21.15.1:0/3825711686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e0149d00 con 0x7f74e014c950 2024-03-06T07:21:41.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74dffff640 1 --2- 172.21.15.1:0/3825711686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74e014c950 0x7f74e00c87d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f74d800d5d0 tx=0x7f74d800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:41.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74c7fff640 1 -- 172.21.15.1:0/3825711686 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d8018020 con 0x7f74e014c950 2024-03-06T07:21:41.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74c7fff640 1 -- 172.21.15.1:0/3825711686 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d8014620 con 0x7f74e014c950 2024-03-06T07:21:41.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74e5c39640 1 -- 172.21.15.1:0/3825711686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e01eef50 con 0x7f74e014c950 2024-03-06T07:21:41.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74c7fff640 1 -- 172.21.15.1:0/3825711686 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d80149f0 con 0x7f74e014c950 2024-03-06T07:21:41.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.622+0000 7f74e5c39640 1 -- 172.21.15.1:0/3825711686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e00c0c80 con 0x7f74e014c950 2024-03-06T07:21:41.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.624+0000 7f74c7fff640 1 -- 172.21.15.1:0/3825711686 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74d801b020 con 0x7f74e014c950 2024-03-06T07:21:41.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.624+0000 7f74c7fff640 1 --2- 172.21.15.1:0/3825711686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74cc06b4d0 0x7f74cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.624+0000 7f74c7fff640 1 -- 172.21.15.1:0/3825711686 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f74d808a3c0 con 0x7f74e014c950 2024-03-06T07:21:41.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.624+0000 7f74df7fe640 1 --2- 172.21.15.1:0/3825711686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74cc06b4d0 0x7f74cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.625+0000 7f74df7fe640 1 --2- 172.21.15.1:0/3825711686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74cc06b4d0 0x7f74cc06d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f74d0003b80 tx=0x7f74d00098d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:41.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.626+0000 7f74e8823740 1 Processor -- start 2024-03-06T07:21:41.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.626+0000 7f74e8823740 1 -- start start 2024-03-06T07:21:41.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.627+0000 7f74e8823740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b760ef50 0x5570b760ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.627+0000 7f74e8823740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b760f330 0x5570b760cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74e8823740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760d4b0 0x5570b7609680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74e8823740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570b75736b0 con 0x5570b760d4b0 2024-03-06T07:21:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74e8823740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570b751ca60 con 0x5570b760ef50 2024-03-06T07:21:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74e8823740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5570b756f750 con 0x5570b760f330 2024-03-06T07:21:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760d4b0 0x5570b7609680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760d4b0 0x5570b7609680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45126/0 (socket says 172.21.15.1:45126) 2024-03-06T07:21:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74df7fe640 1 --2- 172.21.15.1:0/1811539774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b760ef50 0x5570b760ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74dffff640 1 -- 172.21.15.1:0/1811539774 learned_addr learned my addr 172.21.15.1:0/1811539774 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74deffd640 1 --2- 172.21.15.1:0/1811539774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b760f330 0x5570b760cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74dffff640 1 -- 172.21.15.1:0/1811539774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b760f330 msgr2=0x5570b760cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74dffff640 1 --2- 172.21.15.1:0/1811539774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b760f330 0x5570b760cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74dffff640 1 -- 172.21.15.1:0/1811539774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b760ef50 msgr2=0x5570b760ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74dffff640 1 --2- 172.21.15.1:0/1811539774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b760ef50 0x5570b760ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74dffff640 1 -- 172.21.15.1:0/1811539774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570b753af30 con 0x5570b760d4b0 2024-03-06T07:21:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74dffff640 1 --2- 172.21.15.1:0/1811539774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760d4b0 0x5570b7609680 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f74d8054f00 tx=0x7f74d805cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aaf954dd029d8a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:41.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74c57fa640 1 -- 172.21.15.1:0/1811539774 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d8056da0 con 0x5570b760d4b0 2024-03-06T07:21:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74c57fa640 1 -- 172.21.15.1:0/1811539774 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d808a0e0 con 0x5570b760d4b0 2024-03-06T07:21:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.628+0000 7f74c57fa640 1 -- 172.21.15.1:0/1811539774 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d8014b90 con 0x5570b760d4b0 2024-03-06T07:21:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.629+0000 7f74e8823740 1 -- 172.21.15.1:0/1811539774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760d4b0 msgr2=0x5570b7609680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.629+0000 7f74e8823740 1 --2- 172.21.15.1:0/1811539774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760d4b0 0x5570b7609680 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f74d8054f00 tx=0x7f74d805cb40 comp rx=0 tx=0).stop 2024-03-06T07:21:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.629+0000 7f74e8823740 1 -- 172.21.15.1:0/1811539774 shutdown_connections 2024-03-06T07:21:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.629+0000 7f74e8823740 1 --2- 172.21.15.1:0/1811539774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760d4b0 0x5570b7609680 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.629+0000 7f74e8823740 1 --2- 172.21.15.1:0/1811539774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b760f330 0x5570b760cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.629+0000 7f74e8823740 1 --2- 172.21.15.1:0/1811539774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b760ef50 0x5570b760ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.629+0000 7f74e8823740 1 -- 172.21.15.1:0/1811539774 >> 172.21.15.1:0/1811539774 conn(0x5570b760b7d0 msgr2=0x5570b76303c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.629+0000 7f74e8823740 1 -- 172.21.15.1:0/1811539774 shutdown_connections 2024-03-06T07:21:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.629+0000 7f74e8823740 1 -- 172.21.15.1:0/1811539774 wait complete. 2024-03-06T07:21:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.630+0000 7f74e8823740 1 -- start start 2024-03-06T07:21:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.630+0000 7f74e8823740 1 Processor -- start 2024-03-06T07:21:41.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.630+0000 7f74e8823740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760ef50 0x5570b764ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.630+0000 7f74e8823740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b760f330 0x5570b765f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.630+0000 7f74e8823740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b765fb70 0x5570b762fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.630+0000 7f74e8823740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570b7631230 con 0x5570b760ef50 2024-03-06T07:21:41.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.630+0000 7f74e8823740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570b752fea0 con 0x5570b765fb70 2024-03-06T07:21:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.630+0000 7f74e8823740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5570b7630c70 con 0x5570b760f330 2024-03-06T07:21:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74e8823740 10 client.0 _ll_register_callbacks cb 0x5570b752f870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:21:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74df7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760ef50 0x5570b764ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74df7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760ef50 0x5570b764ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45130/0 (socket says 172.21.15.1:45130) 2024-03-06T07:21:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74df7fe640 1 -- 172.21.15.1:0/3965934920 learned_addr learned my addr 172.21.15.1:0/3965934920 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74deffd640 1 --2- 172.21.15.1:0/3965934920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b760f330 0x5570b765f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74dffff640 1 --2- 172.21.15.1:0/3965934920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b765fb70 0x5570b762fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74df7fe640 1 -- 172.21.15.1:0/3965934920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b760f330 msgr2=0x5570b765f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74df7fe640 1 --2- 172.21.15.1:0/3965934920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b760f330 0x5570b765f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74df7fe640 1 -- 172.21.15.1:0/3965934920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b765fb70 msgr2=0x5570b762fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74df7fe640 1 --2- 172.21.15.1:0/3965934920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b765fb70 0x5570b762fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74df7fe640 1 -- 172.21.15.1:0/3965934920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5570b7530270 con 0x5570b760ef50 2024-03-06T07:21:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74dffff640 1 --2- 172.21.15.1:0/3965934920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b765fb70 0x5570b762fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74df7fe640 1 --2- 172.21.15.1:0/3965934920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760ef50 0x5570b764ea70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f74d00179c0 tx=0x7f74d0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae3b4d7735cb094e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74be7fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:21:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74be7fc640 1 -- 172.21.15.1:0/3965934920 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d0015b70 con 0x5570b760ef50 2024-03-06T07:21:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74be7fc640 1 -- 172.21.15.1:0/3965934920 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f74d0012040 con 0x5570b760ef50 2024-03-06T07:21:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74e8823740 1 -- 172.21.15.1:0/3965934920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f74d808a0e0 con 0x5570b760ef50 2024-03-06T07:21:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74be7fc640 1 -- 172.21.15.1:0/3965934920 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74d0015d10 con 0x5570b760ef50 2024-03-06T07:21:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74e8823740 10 client.5562 fetch_fsmap learned FSMap version 21 2024-03-06T07:21:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.631+0000 7f74e8823740 1 -- 172.21.15.1:0/3965934920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x5570b72882d0 con 0x5570b760ef50 2024-03-06T07:21:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.632+0000 7f74be7fc640 1 -- 172.21.15.1:0/3965934920 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d808a0e0 con 0x5570b760ef50 2024-03-06T07:21:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.632+0000 7f74be7fc640 1 -- 172.21.15.1:0/3965934920 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 21) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f74d0016070 con 0x5570b760ef50 2024-03-06T07:21:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.632+0000 7f74e8823740 10 client.5562 fetch_fsmap finished waiting for FSMap version 21 2024-03-06T07:21:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.632+0000 7f74e8823740 10 client.5562 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:21:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.632+0000 7f74e8823740 20 client.5562 populate_metadata read hostname 'smithi001' 2024-03-06T07:21:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.632+0000 7f74e8823740 1 --2- 172.21.15.1:0/3965934920 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x5570b762e640 0x5570b762ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.632+0000 7f74e8823740 4 client.5562 mds_command: new command op to 4781 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:21:41.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.632+0000 7f74e8823740 1 -- 172.21.15.1:0/3965934920 --> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5570b7580bf0 con 0x5570b762e640 2024-03-06T07:21:41.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.632+0000 7f74deffd640 1 --2- 172.21.15.1:0/3965934920 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x5570b762e640 0x5570b762ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.632+0000 7f74be7fc640 10 client.5562 ms_handle_connect on v2:172.21.15.1:6833/2438105345 2024-03-06T07:21:41.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.632+0000 7f74deffd640 1 --2- 172.21.15.1:0/3965934920 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x5570b762e640 0x5570b762ea80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78bd54d449250705 server_cookie=71ce67d204cc35af in_seq=0 out_seq=0 2024-03-06T07:21:41.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.633+0000 7f74be7fc640 1 -- 172.21.15.1:0/3965934920 <== mds.0 v2:172.21.15.1:6833/2438105345 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5570b7580bf0 con 0x5570b762e640 2024-03-06T07:21:41.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.633+0000 7f74be7fc640 10 client.5562 handle_command_reply: tid=0 2024-03-06T07:21:41.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.633+0000 7f74e8823740 1 client.5562 shutdown 2024-03-06T07:21:41.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.633+0000 7f74e8823740 1 -- 172.21.15.1:0/3965934920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760ef50 msgr2=0x5570b764ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.633+0000 7f74e8823740 1 --2- 172.21.15.1:0/3965934920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760ef50 0x5570b764ea70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f74d00179c0 tx=0x7f74d0004ab0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.633+0000 7f74e8823740 1 -- 172.21.15.1:0/3965934920 shutdown_connections 2024-03-06T07:21:41.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.633+0000 7f74e8823740 1 --2- 172.21.15.1:0/3965934920 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x5570b762e640 0x5570b762ea80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.633+0000 7f74e8823740 1 --2- 172.21.15.1:0/3965934920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b765fb70 0x5570b762fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.633+0000 7f74e8823740 1 --2- 172.21.15.1:0/3965934920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b760f330 0x5570b765f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.633+0000 7f74be7fc640 0 client.5562 ms_handle_reset on v2:172.21.15.1:6833/2438105345 2024-03-06T07:21:41.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.633+0000 7f74e8823740 1 --2- 172.21.15.1:0/3965934920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760ef50 0x5570b764ea70 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.633+0000 7f74e8823740 1 -- 172.21.15.1:0/3965934920 >> 172.21.15.1:0/3965934920 conn(0x5570b760d4b0 msgr2=0x5570b760d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:41.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.633+0000 7f74e8823740 1 -- 172.21.15.1:0/3965934920 shutdown_connections 2024-03-06T07:21:41.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.633+0000 7f74e8823740 1 -- 172.21.15.1:0/3965934920 wait complete. 2024-03-06T07:21:41.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.633+0000 7f74e8823740 20 client.5562 trim_cache size 0 max 16384 2024-03-06T07:21:41.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.650+0000 7f74e8823740 1 Processor -- start 2024-03-06T07:21:41.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.650+0000 7f74e8823740 1 -- start start 2024-03-06T07:21:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.651+0000 7f74e8823740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b760ef50 0x5570b76093b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.651+0000 7f74e8823740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b760f330 0x5570b7608510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.651+0000 7f74e8823740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b76098f0 0x5570b7608a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.651+0000 7f74e8823740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e01f7000 con 0x5570b76098f0 2024-03-06T07:21:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.651+0000 7f74e8823740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e0145a20 con 0x5570b760ef50 2024-03-06T07:21:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.651+0000 7f74e8823740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e01f6b60 con 0x5570b760f330 2024-03-06T07:21:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.651+0000 7f74dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b76098f0 0x5570b7608a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.651+0000 7f74dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b76098f0 0x5570b7608a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45138/0 (socket says 172.21.15.1:45138) 2024-03-06T07:21:41.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.651+0000 7f74dffff640 1 -- 172.21.15.1:0/1153970731 learned_addr learned my addr 172.21.15.1:0/1153970731 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.651+0000 7f74df7fe640 1 --2- 172.21.15.1:0/1153970731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b760ef50 0x5570b76093b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.651+0000 7f74dffff640 1 -- 172.21.15.1:0/1153970731 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b760f330 msgr2=0x5570b7608510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.651+0000 7f74dffff640 1 --2- 172.21.15.1:0/1153970731 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b760f330 0x5570b7608510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.651+0000 7f74dffff640 1 -- 172.21.15.1:0/1153970731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b760ef50 msgr2=0x5570b76093b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.651+0000 7f74dffff640 1 --2- 172.21.15.1:0/1153970731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b760ef50 0x5570b76093b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.651+0000 7f74dffff640 1 -- 172.21.15.1:0/1153970731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570b73526b0 con 0x5570b76098f0 2024-03-06T07:21:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.651+0000 7f74df7fe640 1 --2- 172.21.15.1:0/1153970731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b760ef50 0x5570b76093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.651+0000 7f74dffff640 1 --2- 172.21.15.1:0/1153970731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b76098f0 0x5570b7608a50 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f74d80587a0 tx=0x7f74d805b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7132e7b5c54a976 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.652+0000 7f74c4ff9640 1 -- 172.21.15.1:0/1153970731 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d80585e0 con 0x5570b76098f0 2024-03-06T07:21:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.652+0000 7f74c4ff9640 1 -- 172.21.15.1:0/1153970731 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d8056da0 con 0x5570b76098f0 2024-03-06T07:21:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.652+0000 7f74c4ff9640 1 -- 172.21.15.1:0/1153970731 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d8055110 con 0x5570b76098f0 2024-03-06T07:21:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.652+0000 7f74e8823740 1 -- 172.21.15.1:0/1153970731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b76098f0 msgr2=0x5570b7608a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.652+0000 7f74e8823740 1 --2- 172.21.15.1:0/1153970731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b76098f0 0x5570b7608a50 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f74d80587a0 tx=0x7f74d805b000 comp rx=0 tx=0).stop 2024-03-06T07:21:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.652+0000 7f74e8823740 1 -- 172.21.15.1:0/1153970731 shutdown_connections 2024-03-06T07:21:41.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.652+0000 7f74e8823740 1 --2- 172.21.15.1:0/1153970731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b76098f0 0x5570b7608a50 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.652+0000 7f74e8823740 1 --2- 172.21.15.1:0/1153970731 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b760f330 0x5570b7608510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.652+0000 7f74e8823740 1 --2- 172.21.15.1:0/1153970731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b760ef50 0x5570b76093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.652+0000 7f74e8823740 1 -- 172.21.15.1:0/1153970731 >> 172.21.15.1:0/1153970731 conn(0x5570b760d4b0 msgr2=0x5570b765f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:41.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.652+0000 7f74e8823740 1 -- 172.21.15.1:0/1153970731 shutdown_connections 2024-03-06T07:21:41.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.652+0000 7f74e8823740 1 -- 172.21.15.1:0/1153970731 wait complete. 2024-03-06T07:21:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.653+0000 7f74e8823740 1 -- start start 2024-03-06T07:21:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.653+0000 7f74e8823740 1 Processor -- start 2024-03-06T07:21:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.653+0000 7f74e8823740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b76098f0 0x5570b76484c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.653+0000 7f74e8823740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b760ef50 0x5570b76bb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.653+0000 7f74e8823740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760f330 0x5570b76b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.653+0000 7f74e8823740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570b765c3f0 con 0x5570b760f330 2024-03-06T07:21:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.653+0000 7f74e8823740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570b762e4c0 con 0x5570b760ef50 2024-03-06T07:21:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.653+0000 7f74e8823740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5570b765f110 con 0x5570b76098f0 2024-03-06T07:21:41.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.653+0000 7f74e8823740 10 client.0 _ll_register_callbacks cb 0x7f74d0004720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:21:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760f330 0x5570b76b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760f330 0x5570b76b8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45154/0 (socket says 172.21.15.1:45154) 2024-03-06T07:21:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74dffff640 1 -- 172.21.15.1:0/1003435541 learned_addr learned my addr 172.21.15.1:0/1003435541 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74df7fe640 1 --2- 172.21.15.1:0/1003435541 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b76098f0 0x5570b76484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74deffd640 1 --2- 172.21.15.1:0/1003435541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b760ef50 0x5570b76bb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74dffff640 1 -- 172.21.15.1:0/1003435541 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b76098f0 msgr2=0x5570b76484c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74dffff640 1 --2- 172.21.15.1:0/1003435541 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b76098f0 0x5570b76484c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74dffff640 1 -- 172.21.15.1:0/1003435541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b760ef50 msgr2=0x5570b76bb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74dffff640 1 --2- 172.21.15.1:0/1003435541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b760ef50 0x5570b76bb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74dffff640 1 -- 172.21.15.1:0/1003435541 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5570b764e670 con 0x5570b760f330 2024-03-06T07:21:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74dffff640 1 --2- 172.21.15.1:0/1003435541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760f330 0x5570b76b8bc0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f74d80587a0 tx=0x7f74d808a070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37030ea37760a17d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74be7fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:21:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74be7fc640 1 -- 172.21.15.1:0/1003435541 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d8077340 con 0x5570b760f330 2024-03-06T07:21:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74be7fc640 1 -- 172.21.15.1:0/1003435541 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f74d80774e0 con 0x5570b760f330 2024-03-06T07:21:41.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74e8823740 1 -- 172.21.15.1:0/1003435541 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f74d8056da0 con 0x5570b760f330 2024-03-06T07:21:41.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74be7fc640 1 -- 172.21.15.1:0/1003435541 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74d8014b90 con 0x5570b760f330 2024-03-06T07:21:41.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74e8823740 10 client.5568 fetch_fsmap learned FSMap version 21 2024-03-06T07:21:41.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74e8823740 1 -- 172.21.15.1:0/1003435541 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x5570b7632330 con 0x5570b760f330 2024-03-06T07:21:41.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.654+0000 7f74be7fc640 1 -- 172.21.15.1:0/1003435541 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d8056da0 con 0x5570b760f330 2024-03-06T07:21:41.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74be7fc640 1 -- 172.21.15.1:0/1003435541 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 21) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f74d8013050 con 0x5570b760f330 2024-03-06T07:21:41.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74e8823740 10 client.5568 fetch_fsmap finished waiting for FSMap version 21 2024-03-06T07:21:41.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74e8823740 10 client.5568 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:21:41.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74e8823740 20 client.5568 populate_metadata read hostname 'smithi001' 2024-03-06T07:21:41.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74e8823740 1 --2- 172.21.15.1:0/1003435541 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x5570b76b9100 0x5570b76becd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74e8823740 4 client.5568 mds_command: new command op to 4781 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:21:41.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74e8823740 1 -- 172.21.15.1:0/1003435541 --> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5570b757af40 con 0x5570b76b9100 2024-03-06T07:21:41.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74df7fe640 1 --2- 172.21.15.1:0/1003435541 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x5570b76b9100 0x5570b76becd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74be7fc640 10 client.5568 ms_handle_connect on v2:172.21.15.1:6833/2438105345 2024-03-06T07:21:41.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74df7fe640 1 --2- 172.21.15.1:0/1003435541 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x5570b76b9100 0x5570b76becd0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bdecec9723a372b server_cookie=249e65c623de6618 in_seq=0 out_seq=0 2024-03-06T07:21:41.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74be7fc640 1 -- 172.21.15.1:0/1003435541 <== mds.0 v2:172.21.15.1:6833/2438105345 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5570b757af40 con 0x5570b76b9100 2024-03-06T07:21:41.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74be7fc640 10 client.5568 handle_command_reply: tid=0 2024-03-06T07:21:41.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74e8823740 1 client.5568 shutdown 2024-03-06T07:21:41.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74e8823740 1 -- 172.21.15.1:0/1003435541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760f330 msgr2=0x5570b76b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74e8823740 1 --2- 172.21.15.1:0/1003435541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760f330 0x5570b76b8bc0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f74d80587a0 tx=0x7f74d808a070 comp rx=0 tx=0).stop 2024-03-06T07:21:41.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74e8823740 1 -- 172.21.15.1:0/1003435541 shutdown_connections 2024-03-06T07:21:41.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74e8823740 1 --2- 172.21.15.1:0/1003435541 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x5570b76b9100 0x5570b76becd0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74be7fc640 0 client.5568 ms_handle_reset on v2:172.21.15.1:6833/2438105345 2024-03-06T07:21:41.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74e8823740 1 --2- 172.21.15.1:0/1003435541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b760f330 0x5570b76b8bc0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74e8823740 1 --2- 172.21.15.1:0/1003435541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5570b760ef50 0x5570b76bb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.655+0000 7f74e8823740 1 --2- 172.21.15.1:0/1003435541 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5570b76098f0 0x5570b76484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.656+0000 7f74e8823740 1 -- 172.21.15.1:0/1003435541 >> 172.21.15.1:0/1003435541 conn(0x5570b760d4b0 msgr2=0x5570b765f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:41.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.656+0000 7f74e8823740 1 -- 172.21.15.1:0/1003435541 shutdown_connections 2024-03-06T07:21:41.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.656+0000 7f74e8823740 1 -- 172.21.15.1:0/1003435541 wait complete. 2024-03-06T07:21:41.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.656+0000 7f74e8823740 20 client.5568 trim_cache size 0 max 16384 2024-03-06T07:21:41.664 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:21:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:21:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:21:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4781, 2024-03-06T07:21:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:21:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:21:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:21:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 4.4590751959999997, 2024-03-06T07:21:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 21, 2024-03-06T07:21:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 40, 2024-03-06T07:21:41.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 40, 2024-03-06T07:21:41.665 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 25.368121384999998 2024-03-06T07:21:41.665 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:21:41.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.656+0000 7f74e5c39640 1 -- 172.21.15.1:0/3825711686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74cc06b4d0 msgr2=0x7f74cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.656+0000 7f74e5c39640 1 --2- 172.21.15.1:0/3825711686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74cc06b4d0 0x7f74cc06d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f74d0003b80 tx=0x7f74d00098d0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.656+0000 7f74e5c39640 1 -- 172.21.15.1:0/3825711686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74e014c950 msgr2=0x7f74e00c87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.656+0000 7f74e5c39640 1 --2- 172.21.15.1:0/3825711686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74e014c950 0x7f74e00c87d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f74d800d5d0 tx=0x7f74d800da90 comp rx=0 tx=0).stop 2024-03-06T07:21:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.656+0000 7f74e5c39640 1 -- 172.21.15.1:0/3825711686 shutdown_connections 2024-03-06T07:21:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.656+0000 7f74e5c39640 1 --2- 172.21.15.1:0/3825711686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74cc06b4d0 0x7f74cc06d990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.656+0000 7f74e5c39640 1 --2- 172.21.15.1:0/3825711686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74e014c950 0x7f74e00c87d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.656+0000 7f74e5c39640 1 --2- 172.21.15.1:0/3825711686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74e0152650 0x7f74e014c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.656+0000 7f74e5c39640 1 --2- 172.21.15.1:0/3825711686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74e00bf9d0 0x7f74e014e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.656+0000 7f74e5c39640 1 -- 172.21.15.1:0/3825711686 >> 172.21.15.1:0/3825711686 conn(0x7f74e0066ee0 msgr2=0x7f74e00c3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.656+0000 7f74e5c39640 1 -- 172.21.15.1:0/3825711686 shutdown_connections 2024-03-06T07:21:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.656+0000 7f74e5c39640 1 -- 172.21.15.1:0/3825711686 wait complete. 2024-03-06T07:21:41.669 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 21, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:21:22.528539+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 21, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4781": { "addr": "172.21.15.1:6835/2438105345", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 2438105345, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 2438105345, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4781, "incarnation": 18, "join_fscid": 2, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-03-06T07:21:40.204549+0000", "qdb_cluster": [ 4781 ], "qdb_leader": 4781, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4781 } } } ], "standbys": [ { "addr": "172.21.15.98:6836/3354292028", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 3354292028, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 3354292028, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4972, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6837/3948803645", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 3948803645, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 3948803645, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4975, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6834/3390157192", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 3390157192, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 3390157192, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 19, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5538, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:21:41.670 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 2, 'name': 'c', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:21:41.670 INFO:tasks.cephfs.test_failover:current = [{'gid': 5538, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6834/3390157192', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 3390157192}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 3390157192}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 19}, {'gid': 4975, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/3948803645', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3948803645}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3948803645}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 4781, 'name': 'c', 'rank': 0, 'incarnation': 18, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.1:6835/2438105345', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 2438105345}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2438105345}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 4972, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/3354292028', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 3354292028}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 3354292028}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}] 2024-03-06T07:21:41.670 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:21:41.670 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-03-06T07:21:41.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.804+0000 7ff9d5993640 1 Processor -- start 2024-03-06T07:21:41.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.804+0000 7ff9d5993640 1 -- start start 2024-03-06T07:21:41.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.805+0000 7ff9d5993640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9d00c6b10 0x7ff9d0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.805+0000 7ff9d5993640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff9d00bf9d0 0x7ff9d00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.805+0000 7ff9d5993640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d00c2fc0 con 0x7ff9d0152650 2024-03-06T07:21:41.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.805+0000 7ff9d5993640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d0155800 con 0x7ff9d00bf9d0 2024-03-06T07:21:41.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.805+0000 7ff9d5993640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d0158da0 con 0x7ff9d00c6b10 2024-03-06T07:21:41.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.805+0000 7ff9ceffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff9d0152650 0x7ff9d0158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41474/0 (socket says 172.21.15.1:41474) 2024-03-06T07:21:41.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.805+0000 7ff9ceffd640 1 -- 172.21.15.1:0/4086759490 learned_addr learned my addr 172.21.15.1:0/4086759490 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:41.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.806+0000 7ff9cf7fe640 1 --2- 172.21.15.1:0/4086759490 >> v2:172.21.15.98:3300/0 conn(0x7ff9d00bf9d0 0x7ff9d00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.806+0000 7ff9ce7fc640 1 --2- 172.21.15.1:0/4086759490 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9d00c6b10 0x7ff9d0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.806+0000 7ff9cdffb640 1 -- 172.21.15.1:0/4086759490 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315064550 0 0) 0x7ff9d00c2fc0 con 0x7ff9d0152650 2024-03-06T07:21:41.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.806+0000 7ff9cdffb640 1 -- 172.21.15.1:0/4086759490 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9b8003520 con 0x7ff9d0152650 2024-03-06T07:21:41.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.806+0000 7ff9cf7fe640 1 -- 172.21.15.1:0/4086759490 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9d00c6b10 msgr2=0x7ff9d0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:41.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.806+0000 7ff9cf7fe640 1 --2- 172.21.15.1:0/4086759490 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9d00c6b10 0x7ff9d0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.806+0000 7ff9cf7fe640 1 -- 172.21.15.1:0/4086759490 >> v1:172.21.15.1:6789/0 conn(0x7ff9d0152650 legacy=0x7ff9d0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.806+0000 7ff9cf7fe640 1 -- 172.21.15.1:0/4086759490 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d00c6690 con 0x7ff9d00bf9d0 2024-03-06T07:21:41.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.806+0000 7ff9ce7fc640 1 --2- 172.21.15.1:0/4086759490 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9d00c6b10 0x7ff9d0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:41.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.807+0000 7ff9cf7fe640 1 --2- 172.21.15.1:0/4086759490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d00bf9d0 0x7ff9d00c1460 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8009d90 tx=0x7ff9c802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2787792040250f18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:41.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.807+0000 7ff9cdffb640 1 -- 172.21.15.1:0/4086759490 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8039070 con 0x7ff9d00bf9d0 2024-03-06T07:21:41.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.807+0000 7ff9cdffb640 1 -- 172.21.15.1:0/4086759490 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c802cbc0 con 0x7ff9d00bf9d0 2024-03-06T07:21:41.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.807+0000 7ff9cdffb640 1 -- 172.21.15.1:0/4086759490 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8034440 con 0x7ff9d00bf9d0 2024-03-06T07:21:41.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.808+0000 7ff9d5993640 1 -- 172.21.15.1:0/4086759490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d00bf9d0 msgr2=0x7ff9d00c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.808+0000 7ff9d5993640 1 --2- 172.21.15.1:0/4086759490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d00bf9d0 0x7ff9d00c1460 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8009d90 tx=0x7ff9c802be10 comp rx=0 tx=0).stop 2024-03-06T07:21:41.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.808+0000 7ff9d5993640 1 -- 172.21.15.1:0/4086759490 shutdown_connections 2024-03-06T07:21:41.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.808+0000 7ff9d5993640 1 --2- 172.21.15.1:0/4086759490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d00bf9d0 0x7ff9d00c1460 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.808+0000 7ff9d5993640 1 --2- 172.21.15.1:0/4086759490 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9d00c6b10 0x7ff9d0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.808+0000 7ff9d5993640 1 -- 172.21.15.1:0/4086759490 >> 172.21.15.1:0/4086759490 conn(0x7ff9d0066ee0 msgr2=0x7ff9d00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:41.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.808+0000 7ff9d5993640 1 -- 172.21.15.1:0/4086759490 shutdown_connections 2024-03-06T07:21:41.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.808+0000 7ff9d5993640 1 -- 172.21.15.1:0/4086759490 wait complete. 2024-03-06T07:21:41.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.809+0000 7ff9d5993640 1 Processor -- start 2024-03-06T07:21:41.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.809+0000 7ff9d5993640 1 -- start start 2024-03-06T07:21:41.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.810+0000 7ff9d5993640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d00bf9d0 0x7ff9d0152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.810+0000 7ff9d5993640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9d0152650 0x7ff9d014fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.810+0000 7ff9d5993640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9d0150f10 0x7ff9d00c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.810+0000 7ff9d5993640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d01551b0 con 0x7ff9d0152650 2024-03-06T07:21:41.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.810+0000 7ff9d5993640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d00c6fa0 con 0x7ff9d00bf9d0 2024-03-06T07:21:41.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.810+0000 7ff9d5993640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d0155490 con 0x7ff9d0150f10 2024-03-06T07:21:41.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.810+0000 7ff9ce7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9d0152650 0x7ff9d014fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.810+0000 7ff9ce7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9d0152650 0x7ff9d014fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45158/0 (socket says 172.21.15.1:45158) 2024-03-06T07:21:41.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.810+0000 7ff9ce7fc640 1 -- 172.21.15.1:0/231489732 learned_addr learned my addr 172.21.15.1:0/231489732 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:41.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.810+0000 7ff9ceffd640 1 --2- 172.21.15.1:0/231489732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d00bf9d0 0x7ff9d0152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.810+0000 7ff9cf7fe640 1 --2- 172.21.15.1:0/231489732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9d0150f10 0x7ff9d00c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.810+0000 7ff9ce7fc640 1 -- 172.21.15.1:0/231489732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9d0150f10 msgr2=0x7ff9d00c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.810+0000 7ff9ce7fc640 1 --2- 172.21.15.1:0/231489732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9d0150f10 0x7ff9d00c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.810+0000 7ff9ce7fc640 1 -- 172.21.15.1:0/231489732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d00bf9d0 msgr2=0x7ff9d0152fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.810+0000 7ff9ce7fc640 1 --2- 172.21.15.1:0/231489732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d00bf9d0 0x7ff9d0152fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.810+0000 7ff9ce7fc640 1 -- 172.21.15.1:0/231489732 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d00ca230 con 0x7ff9d0152650 2024-03-06T07:21:41.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.810+0000 7ff9cf7fe640 1 --2- 172.21.15.1:0/231489732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9d0150f10 0x7ff9d00c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:41.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.811+0000 7ff9ce7fc640 1 --2- 172.21.15.1:0/231489732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9d0152650 0x7ff9d014fcc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400ec10 tx=0x7ff9c400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:41.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.811+0000 7ff9b7fff640 1 -- 172.21.15.1:0/231489732 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4017070 con 0x7ff9d0152650 2024-03-06T07:21:41.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.811+0000 7ff9d5993640 1 -- 172.21.15.1:0/231489732 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d01eed50 con 0x7ff9d0152650 2024-03-06T07:21:41.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.811+0000 7ff9b7fff640 1 -- 172.21.15.1:0/231489732 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c400cc30 con 0x7ff9d0152650 2024-03-06T07:21:41.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.811+0000 7ff9b7fff640 1 -- 172.21.15.1:0/231489732 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4013470 con 0x7ff9d0152650 2024-03-06T07:21:41.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.811+0000 7ff9d5993640 1 -- 172.21.15.1:0/231489732 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d01ef030 con 0x7ff9d0152650 2024-03-06T07:21:41.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.812+0000 7ff9d5993640 1 -- 172.21.15.1:0/231489732 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d0067fd0 con 0x7ff9d0152650 2024-03-06T07:21:41.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.812+0000 7ff9b7fff640 1 -- 172.21.15.1:0/231489732 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff9c401a070 con 0x7ff9d0152650 2024-03-06T07:21:41.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.813+0000 7ff9b7fff640 1 --2- 172.21.15.1:0/231489732 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff9bc06b4d0 0x7ff9bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.813+0000 7ff9b7fff640 1 -- 172.21.15.1:0/231489732 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff9c408a2d0 con 0x7ff9d0152650 2024-03-06T07:21:41.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.813+0000 7ff9ceffd640 1 --2- 172.21.15.1:0/231489732 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff9bc06b4d0 0x7ff9bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.814+0000 7ff9ceffd640 1 --2- 172.21.15.1:0/231489732 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff9bc06b4d0 0x7ff9bc06d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0005880 tx=0x7ff9c0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:41.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:41.817+0000 7ff9b7fff640 1 -- 172.21.15.1:0/231489732 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff9c4058930 con 0x7ff9d0152650 2024-03-06T07:21:42.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.151+0000 7ff9d5993640 1 -- 172.21.15.1:0/231489732 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7ff9d00c3bf0 con 0x7ff9d0152650 2024-03-06T07:21:42.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.154+0000 7ff9b7fff640 1 -- 172.21.15.1:0/231489732 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v13) v1 ==== 96+0+0 (secure 0 0 0) 0x7ff9c405c9d0 con 0x7ff9d0152650 2024-03-06T07:21:42.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.156+0000 7ff9d5993640 1 -- 172.21.15.1:0/231489732 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff9bc06b4d0 msgr2=0x7ff9bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.156+0000 7ff9d5993640 1 --2- 172.21.15.1:0/231489732 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff9bc06b4d0 0x7ff9bc06d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0005880 tx=0x7ff9c0009040 comp rx=0 tx=0).stop 2024-03-06T07:21:42.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.156+0000 7ff9d5993640 1 -- 172.21.15.1:0/231489732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9d0152650 msgr2=0x7ff9d014fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.156+0000 7ff9d5993640 1 --2- 172.21.15.1:0/231489732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9d0152650 0x7ff9d014fcc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400ec10 tx=0x7ff9c400c520 comp rx=0 tx=0).stop 2024-03-06T07:21:42.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.156+0000 7ff9d5993640 1 -- 172.21.15.1:0/231489732 shutdown_connections 2024-03-06T07:21:42.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.156+0000 7ff9d5993640 1 --2- 172.21.15.1:0/231489732 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff9bc06b4d0 0x7ff9bc06d990 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.156+0000 7ff9d5993640 1 --2- 172.21.15.1:0/231489732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9d0150f10 0x7ff9d00c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.156+0000 7ff9d5993640 1 --2- 172.21.15.1:0/231489732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9d0152650 0x7ff9d014fcc0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.156+0000 7ff9d5993640 1 --2- 172.21.15.1:0/231489732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d00bf9d0 0x7ff9d0152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.156+0000 7ff9d5993640 1 -- 172.21.15.1:0/231489732 >> 172.21.15.1:0/231489732 conn(0x7ff9d0066ee0 msgr2=0x7ff9d00672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.156+0000 7ff9d5993640 1 -- 172.21.15.1:0/231489732 shutdown_connections 2024-03-06T07:21:42.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.156+0000 7ff9d5993640 1 -- 172.21.15.1:0/231489732 wait complete. 2024-03-06T07:21:42.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:42.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.285+0000 7fb2fdeed640 1 Processor -- start 2024-03-06T07:21:42.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.285+0000 7fb2fdeed640 1 -- start start 2024-03-06T07:21:42.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.286+0000 7fb2fdeed640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb2f8152650 0x7fb2f8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.286+0000 7fb2fdeed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2f80c6b10 0x7fb2f8158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.286+0000 7fb2fdeed640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f80c2fc0 con 0x7fb2f80bf9d0 2024-03-06T07:21:42.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.286+0000 7fb2fdeed640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2f8155800 con 0x7fb2f8152650 2024-03-06T07:21:42.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.286+0000 7fb2fdeed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f80c6690 con 0x7fb2f80c6b10 2024-03-06T07:21:42.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.286+0000 7fb2f7fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb2f80bf9d0 0x7fb2f80c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41490/0 (socket says 172.21.15.1:41490) 2024-03-06T07:21:42.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.286+0000 7fb2f7fff640 1 -- 172.21.15.1:0/1795816270 learned_addr learned my addr 172.21.15.1:0/1795816270 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:42.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.286+0000 7fb2f6ffd640 1 --2- 172.21.15.1:0/1795816270 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2f80c6b10 0x7fb2f8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.286+0000 7fb2f77fe640 1 --2- 172.21.15.1:0/1795816270 >> v2:172.21.15.98:3300/0 conn(0x7fb2f8152650 0x7fb2f8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.286+0000 7fb2f67fc640 1 -- 172.21.15.1:0/1795816270 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911748226 0 0) 0x7fb2f80c2fc0 con 0x7fb2f80bf9d0 2024-03-06T07:21:42.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.286+0000 7fb2f67fc640 1 -- 172.21.15.1:0/1795816270 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e0003520 con 0x7fb2f80bf9d0 2024-03-06T07:21:42.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.287+0000 7fb2f6ffd640 1 -- 172.21.15.1:0/1795816270 >> v2:172.21.15.98:3300/0 conn(0x7fb2f8152650 msgr2=0x7fb2f8155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:42.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.287+0000 7fb2f6ffd640 1 --2- 172.21.15.1:0/1795816270 >> v2:172.21.15.98:3300/0 conn(0x7fb2f8152650 0x7fb2f8155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.287+0000 7fb2f67fc640 1 -- 172.21.15.1:0/1795816270 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1594568943 0 0) 0x7fb2e0003520 con 0x7fb2f80bf9d0 2024-03-06T07:21:42.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.287+0000 7fb2f6ffd640 1 -- 172.21.15.1:0/1795816270 >> v1:172.21.15.1:6789/0 conn(0x7fb2f80bf9d0 legacy=0x7fb2f80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.287+0000 7fb2f6ffd640 1 -- 172.21.15.1:0/1795816270 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f80c7280 con 0x7fb2f80c6b10 2024-03-06T07:21:42.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.287+0000 7fb2f77fe640 1 --2- 172.21.15.1:0/1795816270 >> v2:172.21.15.98:3300/0 conn(0x7fb2f8152650 0x7fb2f8155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:42.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.288+0000 7fb2f6ffd640 1 --2- 172.21.15.1:0/1795816270 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2f80c6b10 0x7fb2f8158690 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb2e80097a0 tx=0x7fb2e802b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fd6c3563a0b33d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.288+0000 7fb2f67fc640 1 -- 172.21.15.1:0/1795816270 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8037070 con 0x7fb2f80c6b10 2024-03-06T07:21:42.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.288+0000 7fb2f67fc640 1 -- 172.21.15.1:0/1795816270 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e8030510 con 0x7fb2f80c6b10 2024-03-06T07:21:42.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.288+0000 7fb2f67fc640 1 -- 172.21.15.1:0/1795816270 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f00019f0 con 0x7fb2f80c6b10 2024-03-06T07:21:42.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.289+0000 7fb2fdeed640 1 -- 172.21.15.1:0/1795816270 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2f80c6b10 msgr2=0x7fb2f8158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.289+0000 7fb2fdeed640 1 --2- 172.21.15.1:0/1795816270 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2f80c6b10 0x7fb2f8158690 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb2e80097a0 tx=0x7fb2e802b710 comp rx=0 tx=0).stop 2024-03-06T07:21:42.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.289+0000 7fb2fdeed640 1 -- 172.21.15.1:0/1795816270 shutdown_connections 2024-03-06T07:21:42.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.289+0000 7fb2fdeed640 1 --2- 172.21.15.1:0/1795816270 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2f80c6b10 0x7fb2f8158690 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.289+0000 7fb2fdeed640 1 --2- 172.21.15.1:0/1795816270 >> v2:172.21.15.98:3300/0 conn(0x7fb2f8152650 0x7fb2f8155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.289+0000 7fb2fdeed640 1 -- 172.21.15.1:0/1795816270 >> 172.21.15.1:0/1795816270 conn(0x7fb2f8066ee0 msgr2=0x7fb2f80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.289+0000 7fb2fdeed640 1 -- 172.21.15.1:0/1795816270 shutdown_connections 2024-03-06T07:21:42.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.289+0000 7fb2fdeed640 1 -- 172.21.15.1:0/1795816270 wait complete. 2024-03-06T07:21:42.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.290+0000 7fb2fdeed640 1 Processor -- start 2024-03-06T07:21:42.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.290+0000 7fb2fdeed640 1 -- start start 2024-03-06T07:21:42.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.291+0000 7fb2fdeed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f80bf9d0 0x7fb2f80c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.291+0000 7fb2fdeed640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2f8152650 0x7fb2f81ee360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.291+0000 7fb2fdeed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2f80ca410 0x7fb2f81ee8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.291+0000 7fb2fdeed640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f8155490 con 0x7fb2f8152650 2024-03-06T07:21:42.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.291+0000 7fb2fdeed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f81551b0 con 0x7fb2f80bf9d0 2024-03-06T07:21:42.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.291+0000 7fb2fdeed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f8152a30 con 0x7fb2f80ca410 2024-03-06T07:21:42.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.291+0000 7fb2f6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2f8152650 0x7fb2f81ee360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.291+0000 7fb2f6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2f8152650 0x7fb2f81ee360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45164/0 (socket says 172.21.15.1:45164) 2024-03-06T07:21:42.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.291+0000 7fb2f6ffd640 1 -- 172.21.15.1:0/1541803663 learned_addr learned my addr 172.21.15.1:0/1541803663 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:42.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.291+0000 7fb2f7fff640 1 --2- 172.21.15.1:0/1541803663 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2f80ca410 0x7fb2f81ee8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.291+0000 7fb2f77fe640 1 --2- 172.21.15.1:0/1541803663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f80bf9d0 0x7fb2f80c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.291+0000 7fb2f6ffd640 1 -- 172.21.15.1:0/1541803663 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2f80ca410 msgr2=0x7fb2f81ee8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.291+0000 7fb2f6ffd640 1 --2- 172.21.15.1:0/1541803663 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2f80ca410 0x7fb2f81ee8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.291+0000 7fb2f6ffd640 1 -- 172.21.15.1:0/1541803663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f80bf9d0 msgr2=0x7fb2f80c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.291+0000 7fb2f6ffd640 1 --2- 172.21.15.1:0/1541803663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f80bf9d0 0x7fb2f80c36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.291+0000 7fb2f6ffd640 1 -- 172.21.15.1:0/1541803663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f81eede0 con 0x7fb2f8152650 2024-03-06T07:21:42.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.292+0000 7fb2f6ffd640 1 --2- 172.21.15.1:0/1541803663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2f8152650 0x7fb2f81ee360 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8000c00 tx=0x7fb2e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.292+0000 7fb2dffff640 1 -- 172.21.15.1:0/1541803663 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8041020 con 0x7fb2f8152650 2024-03-06T07:21:42.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.292+0000 7fb2dffff640 1 -- 172.21.15.1:0/1541803663 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e803c870 con 0x7fb2f8152650 2024-03-06T07:21:42.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.292+0000 7fb2fdeed640 1 -- 172.21.15.1:0/1541803663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f80c9430 con 0x7fb2f8152650 2024-03-06T07:21:42.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.292+0000 7fb2dffff640 1 -- 172.21.15.1:0/1541803663 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8037040 con 0x7fb2f8152650 2024-03-06T07:21:42.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.292+0000 7fb2fdeed640 1 -- 172.21.15.1:0/1541803663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f814dcd0 con 0x7fb2f8152650 2024-03-06T07:21:42.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.293+0000 7fb2fdeed640 1 -- 172.21.15.1:0/1541803663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f8067fd0 con 0x7fb2f8152650 2024-03-06T07:21:42.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.293+0000 7fb2dffff640 1 -- 172.21.15.1:0/1541803663 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2e8044020 con 0x7fb2f8152650 2024-03-06T07:21:42.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.294+0000 7fb2dffff640 1 --2- 172.21.15.1:0/1541803663 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb2e406b4d0 0x7fb2e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.294+0000 7fb2dffff640 1 -- 172.21.15.1:0/1541803663 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb2e80abd50 con 0x7fb2f8152650 2024-03-06T07:21:42.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.294+0000 7fb2f77fe640 1 --2- 172.21.15.1:0/1541803663 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb2e406b4d0 0x7fb2e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.295+0000 7fb2f77fe640 1 --2- 172.21.15.1:0/1541803663 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb2e406b4d0 0x7fb2e406d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80c11d0 tx=0x7fb2ec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.298+0000 7fb2dffff640 1 -- 172.21.15.1:0/1541803663 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2e80af030 con 0x7fb2f8152650 2024-03-06T07:21:42.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.516+0000 7fb2fdeed640 1 -- 172.21.15.1:0/1541803663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2f80b06f0 con 0x7fb2f8152650 2024-03-06T07:21:42.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.517+0000 7fb2dffff640 1 -- 172.21.15.1:0/1541803663 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4971 (secure 0 0 0) 0x7fb2e804b030 con 0x7fb2f8152650 2024-03-06T07:21:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:42.519 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3354292028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3354292028},{"type":"v1","addr":"172.21.15.98:6836","nonce":3354292028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4975,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3948803645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3948803645},{"type":"v1","addr":"172.21.15.98:6837","nonce":3948803645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5538,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3390157192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3390157192},{"type":"v1","addr":"172.21.15.1:6834","nonce":3390157192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:22.528539+0000","modified":"2024-03-06T07:21:40.204549+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":9,"addr":"172.21.15.1:6835/2438105345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2438105345},{"type":"v1","addr":"172.21.15.1:6835","nonce":2438105345}]},"join_fscid":2,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4781,"qdb_cluster":[4781]},"id":2}]} 2024-03-06T07:21:42.520 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 21 2024-03-06T07:21:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.519+0000 7fb2fdeed640 1 -- 172.21.15.1:0/1541803663 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb2e406b4d0 msgr2=0x7fb2e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.519+0000 7fb2fdeed640 1 --2- 172.21.15.1:0/1541803663 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb2e406b4d0 0x7fb2e406d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80c11d0 tx=0x7fb2ec008040 comp rx=0 tx=0).stop 2024-03-06T07:21:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.519+0000 7fb2fdeed640 1 -- 172.21.15.1:0/1541803663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2f8152650 msgr2=0x7fb2f81ee360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.519+0000 7fb2fdeed640 1 --2- 172.21.15.1:0/1541803663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2f8152650 0x7fb2f81ee360 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8000c00 tx=0x7fb2e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.519+0000 7fb2fdeed640 1 -- 172.21.15.1:0/1541803663 shutdown_connections 2024-03-06T07:21:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.519+0000 7fb2fdeed640 1 --2- 172.21.15.1:0/1541803663 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb2e406b4d0 0x7fb2e406d990 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.519+0000 7fb2fdeed640 1 --2- 172.21.15.1:0/1541803663 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2f80ca410 0x7fb2f81ee8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.519+0000 7fb2fdeed640 1 --2- 172.21.15.1:0/1541803663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2f8152650 0x7fb2f81ee360 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.519+0000 7fb2fdeed640 1 --2- 172.21.15.1:0/1541803663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2f80bf9d0 0x7fb2f80c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.519+0000 7fb2fdeed640 1 -- 172.21.15.1:0/1541803663 >> 172.21.15.1:0/1541803663 conn(0x7fb2f8066ee0 msgr2=0x7fb2f80672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.519+0000 7fb2fdeed640 1 -- 172.21.15.1:0/1541803663 shutdown_connections 2024-03-06T07:21:42.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:42.519+0000 7fb2fdeed640 1 -- 172.21.15.1:0/1541803663 wait complete. 2024-03-06T07:21:42.934 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:21:22.528539+0000', 'modified': '2024-03-06T07:21:40.204549+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 40, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4781}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4781': {'gid': 4781, 'name': 'c', 'rank': 0, 'incarnation': 18, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.1:6835/2438105345', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 2438105345}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2438105345}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4781, 'qdb_cluster': [4781]} 2024-03-06T07:21:42.935 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:21:42.935 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:21:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.035+0000 7fe031fa2640 1 Processor -- start 2024-03-06T07:21:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.035+0000 7fe031fa2640 1 -- start start 2024-03-06T07:21:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.036+0000 7fe031fa2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe02c152530 0x7fe02c0c2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.036+0000 7fe031fa2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe02c158d80 0x7fe02c159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.036+0000 7fe031fa2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c1558b0 con 0x7fe02c06f9d0 2024-03-06T07:21:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.036+0000 7fe031fa2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe02c155b90 con 0x7fe02c152530 2024-03-06T07:21:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.036+0000 7fe031fa2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c155e70 con 0x7fe02c158d80 2024-03-06T07:21:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.036+0000 7fe030fa0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe02c152530 0x7fe02c0c2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.036+0000 7fe030fa0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe02c152530 0x7fe02c0c2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45166/0 (socket says 172.21.15.1:45166) 2024-03-06T07:21:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.036+0000 7fe030fa0640 1 -- 172.21.15.1:0/2801970249 learned_addr learned my addr 172.21.15.1:0/2801970249 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.036+0000 7fe0317a1640 1 --2- 172.21.15.1:0/2801970249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe02c158d80 0x7fe02c159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.036+0000 7fe030fa0640 1 -- 172.21.15.1:0/2801970249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe02c158d80 msgr2=0x7fe02c159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.036+0000 7fe030fa0640 1 --2- 172.21.15.1:0/2801970249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe02c158d80 0x7fe02c159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.036+0000 7fe030fa0640 1 -- 172.21.15.1:0/2801970249 >> v1:172.21.15.98:6789/0 conn(0x7fe02c06f9d0 legacy=0x7fe02c155060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.037+0000 7fe030fa0640 1 -- 172.21.15.1:0/2801970249 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c158860 con 0x7fe02c152530 2024-03-06T07:21:43.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.037+0000 7fe0317a1640 1 --2- 172.21.15.1:0/2801970249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe02c158d80 0x7fe02c159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:43.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.038+0000 7fe030fa0640 1 --2- 172.21.15.1:0/2801970249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe02c152530 0x7fe02c0c2dc0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe020009870 tx=0x7fe02002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3078ba794a52c8a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.038+0000 7fe02b7fe640 1 -- 172.21.15.1:0/2801970249 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe020034070 con 0x7fe02c152530 2024-03-06T07:21:43.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.038+0000 7fe02b7fe640 1 -- 172.21.15.1:0/2801970249 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe020030840 con 0x7fe02c152530 2024-03-06T07:21:43.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.038+0000 7fe02b7fe640 1 -- 172.21.15.1:0/2801970249 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe020030c10 con 0x7fe02c152530 2024-03-06T07:21:43.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.038+0000 7fe031fa2640 1 -- 172.21.15.1:0/2801970249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe02c152530 msgr2=0x7fe02c0c2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.038+0000 7fe031fa2640 1 --2- 172.21.15.1:0/2801970249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe02c152530 0x7fe02c0c2dc0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe020009870 tx=0x7fe02002b980 comp rx=0 tx=0).stop 2024-03-06T07:21:43.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.038+0000 7fe031fa2640 1 -- 172.21.15.1:0/2801970249 shutdown_connections 2024-03-06T07:21:43.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.038+0000 7fe031fa2640 1 --2- 172.21.15.1:0/2801970249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe02c158d80 0x7fe02c159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.038+0000 7fe031fa2640 1 --2- 172.21.15.1:0/2801970249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe02c152530 0x7fe02c0c2dc0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.038+0000 7fe031fa2640 1 -- 172.21.15.1:0/2801970249 >> 172.21.15.1:0/2801970249 conn(0x7fe02c06eb50 msgr2=0x7fe02c06c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.038+0000 7fe031fa2640 1 -- 172.21.15.1:0/2801970249 shutdown_connections 2024-03-06T07:21:43.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.038+0000 7fe031fa2640 1 -- 172.21.15.1:0/2801970249 wait complete. 2024-03-06T07:21:43.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.039+0000 7fe031fa2640 1 Processor -- start 2024-03-06T07:21:43.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.040+0000 7fe031fa2640 1 -- start start 2024-03-06T07:21:43.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.040+0000 7fe031fa2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe02c06f9d0 0x7fe02c0ab880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.040+0000 7fe031fa2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe02c152530 0x7fe02c152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.040+0000 7fe031fa2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe02c158d80 0x7fe02c153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.040+0000 7fe031fa2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0c41c0 con 0x7fe02c152530 2024-03-06T07:21:43.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.040+0000 7fe031fa2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0c4040 con 0x7fe02c158d80 2024-03-06T07:21:43.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.040+0000 7fe031fa2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0c4340 con 0x7fe02c06f9d0 2024-03-06T07:21:43.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.040+0000 7fe02bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe02c152530 0x7fe02c152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.040+0000 7fe02bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe02c152530 0x7fe02c152b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45168/0 (socket says 172.21.15.1:45168) 2024-03-06T07:21:43.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.040+0000 7fe02bfff640 1 -- 172.21.15.1:0/1571313585 learned_addr learned my addr 172.21.15.1:0/1571313585 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:43.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.041+0000 7fe02bfff640 1 -- 172.21.15.1:0/1571313585 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe02c06f9d0 msgr2=0x7fe02c0ab880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.041+0000 7fe030fa0640 1 --2- 172.21.15.1:0/1571313585 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe02c06f9d0 0x7fe02c0ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.041+0000 7fe0317a1640 1 --2- 172.21.15.1:0/1571313585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe02c158d80 0x7fe02c153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.041+0000 7fe02bfff640 1 --2- 172.21.15.1:0/1571313585 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe02c06f9d0 0x7fe02c0ab880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.041+0000 7fe02bfff640 1 -- 172.21.15.1:0/1571313585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe02c158d80 msgr2=0x7fe02c153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.041+0000 7fe02bfff640 1 --2- 172.21.15.1:0/1571313585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe02c158d80 0x7fe02c153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.041+0000 7fe02bfff640 1 -- 172.21.15.1:0/1571313585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c1eeae0 con 0x7fe02c152530 2024-03-06T07:21:43.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.041+0000 7fe030fa0640 1 --2- 172.21.15.1:0/1571313585 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe02c06f9d0 0x7fe02c0ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:43.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.041+0000 7fe0317a1640 1 --2- 172.21.15.1:0/1571313585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe02c158d80 0x7fe02c153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:43.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.041+0000 7fe02bfff640 1 --2- 172.21.15.1:0/1571313585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe02c152530 0x7fe02c152b20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe01800c3c0 tx=0x7fe01800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.041+0000 7fe0297fa640 1 -- 172.21.15.1:0/1571313585 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe018017070 con 0x7fe02c152530 2024-03-06T07:21:43.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.041+0000 7fe0297fa640 1 -- 172.21.15.1:0/1571313585 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0180052f0 con 0x7fe02c152530 2024-03-06T07:21:43.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.041+0000 7fe031fa2640 1 -- 172.21.15.1:0/1571313585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02c154fc0 con 0x7fe02c152530 2024-03-06T07:21:43.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.041+0000 7fe0297fa640 1 -- 172.21.15.1:0/1571313585 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe018003850 con 0x7fe02c152530 2024-03-06T07:21:43.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.041+0000 7fe031fa2640 1 -- 172.21.15.1:0/1571313585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02c0abdc0 con 0x7fe02c152530 2024-03-06T07:21:43.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.043+0000 7fe0297fa640 1 -- 172.21.15.1:0/1571313585 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe01801a070 con 0x7fe02c152530 2024-03-06T07:21:43.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.043+0000 7fe0297fa640 1 --2- 172.21.15.1:0/1571313585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe01006b500 0x7fe01006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.043+0000 7fe0297fa640 1 -- 172.21.15.1:0/1571313585 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe0180896d0 con 0x7fe02c152530 2024-03-06T07:21:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.044+0000 7fe030fa0640 1 --2- 172.21.15.1:0/1571313585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe01006b500 0x7fe01006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.044+0000 7fe035ddf740 1 Processor -- start 2024-03-06T07:21:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.044+0000 7fe035ddf740 1 -- start start 2024-03-06T07:21:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.044+0000 7fe030fa0640 1 --2- 172.21.15.1:0/1571313585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe01006b500 0x7fe01006d9c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe020036000 tx=0x7fe02003b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.045+0000 7fe035ddf740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c77100 0x55b512c74be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.045+0000 7fe035ddf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c774e0 0x55b512c75120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.045+0000 7fe035ddf740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c75660 0x55b512c71830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.045+0000 7fe035ddf740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b512bdb6b0 con 0x55b512c75660 2024-03-06T07:21:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.045+0000 7fe035ddf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b512b84a60 con 0x55b512c774e0 2024-03-06T07:21:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.045+0000 7fe035ddf740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b512bd7750 con 0x55b512c77100 2024-03-06T07:21:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.045+0000 7fe0317a1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c75660 0x55b512c71830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.045+0000 7fe0317a1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c75660 0x55b512c71830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45170/0 (socket says 172.21.15.1:45170) 2024-03-06T07:21:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.045+0000 7fe0317a1640 1 -- 172.21.15.1:0/2337589785 learned_addr learned my addr 172.21.15.1:0/2337589785 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.045+0000 7fe02bfff640 1 --2- 172.21.15.1:0/2337589785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c774e0 0x55b512c75120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.045+0000 7fe030fa0640 1 --2- 172.21.15.1:0/2337589785 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c77100 0x55b512c74be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.045+0000 7fe0317a1640 1 -- 172.21.15.1:0/2337589785 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c77100 msgr2=0x55b512c74be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.045+0000 7fe0317a1640 1 --2- 172.21.15.1:0/2337589785 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c77100 0x55b512c74be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.045+0000 7fe0317a1640 1 -- 172.21.15.1:0/2337589785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c774e0 msgr2=0x55b512c75120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.045+0000 7fe0317a1640 1 --2- 172.21.15.1:0/2337589785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c774e0 0x55b512c75120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.045+0000 7fe0317a1640 1 -- 172.21.15.1:0/2337589785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b512ba2f30 con 0x55b512c75660 2024-03-06T07:21:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.045+0000 7fe0317a1640 1 --2- 172.21.15.1:0/2337589785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c75660 0x55b512c71830 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe024009fd0 tx=0x7fe0240077c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50aa98aa9da9e9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.046+0000 7fe0167fc640 1 -- 172.21.15.1:0/2337589785 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0240137d0 con 0x55b512c75660 2024-03-06T07:21:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.046+0000 7fe0167fc640 1 -- 172.21.15.1:0/2337589785 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe024019070 con 0x55b512c75660 2024-03-06T07:21:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.046+0000 7fe0167fc640 1 -- 172.21.15.1:0/2337589785 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024013b70 con 0x55b512c75660 2024-03-06T07:21:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.046+0000 7fe035ddf740 1 -- 172.21.15.1:0/2337589785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c75660 msgr2=0x55b512c71830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.046+0000 7fe035ddf740 1 --2- 172.21.15.1:0/2337589785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c75660 0x55b512c71830 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe024009fd0 tx=0x7fe0240077c0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.046+0000 7fe035ddf740 1 -- 172.21.15.1:0/2337589785 shutdown_connections 2024-03-06T07:21:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.046+0000 7fe035ddf740 1 --2- 172.21.15.1:0/2337589785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c75660 0x55b512c71830 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.046+0000 7fe035ddf740 1 --2- 172.21.15.1:0/2337589785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c774e0 0x55b512c75120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.046+0000 7fe035ddf740 1 --2- 172.21.15.1:0/2337589785 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c77100 0x55b512c74be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.046+0000 7fe035ddf740 1 -- 172.21.15.1:0/2337589785 >> 172.21.15.1:0/2337589785 conn(0x55b512c73980 msgr2=0x55b512c98570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.046+0000 7fe035ddf740 1 -- 172.21.15.1:0/2337589785 shutdown_connections 2024-03-06T07:21:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.046+0000 7fe035ddf740 1 -- 172.21.15.1:0/2337589785 wait complete. 2024-03-06T07:21:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.047+0000 7fe035ddf740 1 -- start start 2024-03-06T07:21:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.047+0000 7fe035ddf740 1 Processor -- start 2024-03-06T07:21:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.048+0000 7fe035ddf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c77100 0x55b512c9ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.048+0000 7fe035ddf740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c774e0 0x55b512c9aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.048+0000 7fe035ddf740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c9bf20 0x55b512c9c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.048+0000 7fe035ddf740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b512c993e0 con 0x55b512c774e0 2024-03-06T07:21:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe035ddf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b512b97ea0 con 0x55b512c77100 2024-03-06T07:21:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe035ddf740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b512c98e20 con 0x55b512c9bf20 2024-03-06T07:21:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe035ddf740 10 client.0 _ll_register_callbacks cb 0x55b512b97870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:21:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe02bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c774e0 0x55b512c9aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe02bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c774e0 0x55b512c9aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45176/0 (socket says 172.21.15.1:45176) 2024-03-06T07:21:43.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe02bfff640 1 -- 172.21.15.1:0/833067939 learned_addr learned my addr 172.21.15.1:0/833067939 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe030fa0640 1 --2- 172.21.15.1:0/833067939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c77100 0x55b512c9ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe0317a1640 1 --2- 172.21.15.1:0/833067939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c9bf20 0x55b512c9c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe02bfff640 1 -- 172.21.15.1:0/833067939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c9bf20 msgr2=0x55b512c9c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe02bfff640 1 --2- 172.21.15.1:0/833067939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c9bf20 0x55b512c9c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe02bfff640 1 -- 172.21.15.1:0/833067939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c77100 msgr2=0x55b512c9ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe02bfff640 1 --2- 172.21.15.1:0/833067939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c77100 0x55b512c9ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe02bfff640 1 -- 172.21.15.1:0/833067939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b512b98270 con 0x55b512c774e0 2024-03-06T07:21:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe02bfff640 1 --2- 172.21.15.1:0/833067939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c774e0 0x55b512c9aeb0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe018057c10 tx=0x7fe01805bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6ec1a23b1da275 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe00b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:21:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe00b7fe640 1 -- 172.21.15.1:0/833067939 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0180570d0 con 0x55b512c774e0 2024-03-06T07:21:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe00b7fe640 1 -- 172.21.15.1:0/833067939 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe018057270 con 0x55b512c774e0 2024-03-06T07:21:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe035ddf740 1 -- 172.21.15.1:0/833067939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe024019070 con 0x55b512c774e0 2024-03-06T07:21:43.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe00b7fe640 1 -- 172.21.15.1:0/833067939 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe018075170 con 0x55b512c774e0 2024-03-06T07:21:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe035ddf740 10 client.5592 fetch_fsmap learned FSMap version 21 2024-03-06T07:21:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.049+0000 7fe035ddf740 1 -- 172.21.15.1:0/833067939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x55b5128f02d0 con 0x55b512c774e0 2024-03-06T07:21:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.050+0000 7fe00b7fe640 1 -- 172.21.15.1:0/833067939 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe024019070 con 0x55b512c774e0 2024-03-06T07:21:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.050+0000 7fe00b7fe640 1 -- 172.21.15.1:0/833067939 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 21) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe0180539d0 con 0x55b512c774e0 2024-03-06T07:21:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.050+0000 7fe035ddf740 10 client.5592 fetch_fsmap finished waiting for FSMap version 21 2024-03-06T07:21:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.050+0000 7fe035ddf740 10 client.5592 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:21:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.050+0000 7fe035ddf740 20 client.5592 populate_metadata read hostname 'smithi001' 2024-03-06T07:21:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.050+0000 7fe035ddf740 1 --2- 172.21.15.1:0/833067939 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x55b512c99e80 0x55b512c9a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.050+0000 7fe035ddf740 4 client.5592 mds_command: new command op to 4781 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:21:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.050+0000 7fe035ddf740 1 -- 172.21.15.1:0/833067939 --> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b512be8bf0 con 0x55b512c99e80 2024-03-06T07:21:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.050+0000 7fe030fa0640 1 --2- 172.21.15.1:0/833067939 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x55b512c99e80 0x55b512c9a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.050+0000 7fe00b7fe640 10 client.5592 ms_handle_connect on v2:172.21.15.1:6833/2438105345 2024-03-06T07:21:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.050+0000 7fe030fa0640 1 --2- 172.21.15.1:0/833067939 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x55b512c99e80 0x55b512c9a260 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2053aa7c286eea4 server_cookie=b39251e74837e8ce in_seq=0 out_seq=0 2024-03-06T07:21:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.051+0000 7fe00b7fe640 1 -- 172.21.15.1:0/833067939 <== mds.0 v2:172.21.15.1:6833/2438105345 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b512be8bf0 con 0x55b512c99e80 2024-03-06T07:21:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.051+0000 7fe00b7fe640 10 client.5592 handle_command_reply: tid=0 2024-03-06T07:21:43.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.051+0000 7fe035ddf740 1 client.5592 shutdown 2024-03-06T07:21:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.051+0000 7fe035ddf740 1 -- 172.21.15.1:0/833067939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c774e0 msgr2=0x55b512c9aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.051+0000 7fe035ddf740 1 --2- 172.21.15.1:0/833067939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c774e0 0x55b512c9aeb0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe018057c10 tx=0x7fe01805bdd0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.051+0000 7fe035ddf740 1 -- 172.21.15.1:0/833067939 shutdown_connections 2024-03-06T07:21:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.051+0000 7fe035ddf740 1 --2- 172.21.15.1:0/833067939 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x55b512c99e80 0x55b512c9a260 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.051+0000 7fe035ddf740 1 --2- 172.21.15.1:0/833067939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c9bf20 0x55b512c9c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.051+0000 7fe035ddf740 1 --2- 172.21.15.1:0/833067939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c774e0 0x55b512c9aeb0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.051+0000 7fe00b7fe640 0 client.5592 ms_handle_reset on v2:172.21.15.1:6833/2438105345 2024-03-06T07:21:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.051+0000 7fe035ddf740 1 --2- 172.21.15.1:0/833067939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c77100 0x55b512c9ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.051+0000 7fe035ddf740 1 -- 172.21.15.1:0/833067939 >> 172.21.15.1:0/833067939 conn(0x55b512c75660 msgr2=0x55b512c70840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.051+0000 7fe035ddf740 1 -- 172.21.15.1:0/833067939 shutdown_connections 2024-03-06T07:21:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.051+0000 7fe035ddf740 1 -- 172.21.15.1:0/833067939 wait complete. 2024-03-06T07:21:43.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.051+0000 7fe035ddf740 20 client.5592 trim_cache size 0 max 16384 2024-03-06T07:21:43.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.068+0000 7fe035ddf740 1 Processor -- start 2024-03-06T07:21:43.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.068+0000 7fe035ddf740 1 -- start start 2024-03-06T07:21:43.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe035ddf740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c77100 0x55b512cc0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe035ddf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c774e0 0x55b512cc3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe035ddf740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c9bf20 0x55b512cc40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe035ddf740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c1f6da0 con 0x55b512c9bf20 2024-03-06T07:21:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe035ddf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c14f200 con 0x55b512c774e0 2024-03-06T07:21:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe035ddf740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c1f67a0 con 0x55b512c77100 2024-03-06T07:21:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe0317a1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c9bf20 0x55b512cc40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe0317a1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c9bf20 0x55b512cc40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45182/0 (socket says 172.21.15.1:45182) 2024-03-06T07:21:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe0317a1640 1 -- 172.21.15.1:0/3992747474 learned_addr learned my addr 172.21.15.1:0/3992747474 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe02bfff640 1 --2- 172.21.15.1:0/3992747474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c774e0 0x55b512cc3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe030fa0640 1 --2- 172.21.15.1:0/3992747474 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c77100 0x55b512cc0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe0317a1640 1 -- 172.21.15.1:0/3992747474 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c77100 msgr2=0x55b512cc0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe0317a1640 1 --2- 172.21.15.1:0/3992747474 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c77100 0x55b512cc0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe0317a1640 1 -- 172.21.15.1:0/3992747474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c774e0 msgr2=0x55b512cc3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe0317a1640 1 --2- 172.21.15.1:0/3992747474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c774e0 0x55b512cc3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe0317a1640 1 -- 172.21.15.1:0/3992747474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5129ba6b0 con 0x55b512c9bf20 2024-03-06T07:21:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe030fa0640 1 --2- 172.21.15.1:0/3992747474 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c77100 0x55b512cc0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe02bfff640 1 --2- 172.21.15.1:0/3992747474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c774e0 0x55b512cc3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe0317a1640 1 --2- 172.21.15.1:0/3992747474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c9bf20 0x55b512cc40c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe024016040 tx=0x7fe024013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1632216d1198c8ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe015ffb640 1 -- 172.21.15.1:0/3992747474 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024003990 con 0x55b512c9bf20 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe015ffb640 1 -- 172.21.15.1:0/3992747474 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe024019040 con 0x55b512c9bf20 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.069+0000 7fe015ffb640 1 -- 172.21.15.1:0/3992747474 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024003d90 con 0x55b512c9bf20 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.070+0000 7fe035ddf740 1 -- 172.21.15.1:0/3992747474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c9bf20 msgr2=0x55b512cc40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.070+0000 7fe035ddf740 1 --2- 172.21.15.1:0/3992747474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c9bf20 0x55b512cc40c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe024016040 tx=0x7fe024013000 comp rx=0 tx=0).stop 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.070+0000 7fe035ddf740 1 -- 172.21.15.1:0/3992747474 shutdown_connections 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.070+0000 7fe035ddf740 1 --2- 172.21.15.1:0/3992747474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c9bf20 0x55b512cc40c0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.070+0000 7fe035ddf740 1 --2- 172.21.15.1:0/3992747474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c774e0 0x55b512cc3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.070+0000 7fe035ddf740 1 --2- 172.21.15.1:0/3992747474 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c77100 0x55b512cc0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.070+0000 7fe035ddf740 1 -- 172.21.15.1:0/3992747474 >> 172.21.15.1:0/3992747474 conn(0x55b512c75660 msgr2=0x55b512c9dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.070+0000 7fe035ddf740 1 -- 172.21.15.1:0/3992747474 shutdown_connections 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.070+0000 7fe035ddf740 1 -- 172.21.15.1:0/3992747474 wait complete. 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.070+0000 7fe035ddf740 1 -- start start 2024-03-06T07:21:43.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.071+0000 7fe035ddf740 1 Processor -- start 2024-03-06T07:21:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.071+0000 7fe035ddf740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c77100 0x55b512cccf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.071+0000 7fe035ddf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c774e0 0x55b512d249f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.071+0000 7fe035ddf740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c9bf20 0x55b512d0c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.071+0000 7fe035ddf740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b512c9b6e0 con 0x55b512c9bf20 2024-03-06T07:21:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.071+0000 7fe035ddf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b512c99d00 con 0x55b512c774e0 2024-03-06T07:21:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.071+0000 7fe035ddf740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b512c9d2e0 con 0x55b512c77100 2024-03-06T07:21:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.071+0000 7fe035ddf740 10 client.0 _ll_register_callbacks cb 0x7fe018058470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:21:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.071+0000 7fe0317a1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c9bf20 0x55b512d0c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe0317a1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c9bf20 0x55b512d0c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45184/0 (socket says 172.21.15.1:45184) 2024-03-06T07:21:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe0317a1640 1 -- 172.21.15.1:0/2723780412 learned_addr learned my addr 172.21.15.1:0/2723780412 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe030fa0640 1 --2- 172.21.15.1:0/2723780412 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c77100 0x55b512cccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe02bfff640 1 --2- 172.21.15.1:0/2723780412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c774e0 0x55b512d249f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe0317a1640 1 -- 172.21.15.1:0/2723780412 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c77100 msgr2=0x55b512cccf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe0317a1640 1 --2- 172.21.15.1:0/2723780412 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c77100 0x55b512cccf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe0317a1640 1 -- 172.21.15.1:0/2723780412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c774e0 msgr2=0x55b512d249f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe0317a1640 1 --2- 172.21.15.1:0/2723780412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c774e0 0x55b512d249f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe0317a1640 1 -- 172.21.15.1:0/2723780412 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b512cce020 con 0x55b512c9bf20 2024-03-06T07:21:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe0317a1640 1 --2- 172.21.15.1:0/2723780412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c9bf20 0x55b512d0c050 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe024015000 tx=0x7fe024004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0e8420c7d172d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe00b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:21:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe00b7fe640 1 -- 172.21.15.1:0/2723780412 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024037060 con 0x55b512c9bf20 2024-03-06T07:21:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe00b7fe640 1 -- 172.21.15.1:0/2723780412 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe024004430 con 0x55b512c9bf20 2024-03-06T07:21:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe035ddf740 1 -- 172.21.15.1:0/2723780412 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe024019040 con 0x55b512c9bf20 2024-03-06T07:21:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe00b7fe640 1 -- 172.21.15.1:0/2723780412 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe02403d050 con 0x55b512c9bf20 2024-03-06T07:21:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe035ddf740 10 client.5598 fetch_fsmap learned FSMap version 21 2024-03-06T07:21:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe035ddf740 1 -- 172.21.15.1:0/2723780412 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x55b512d23e40 con 0x55b512c9bf20 2024-03-06T07:21:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.072+0000 7fe00b7fe640 1 -- 172.21.15.1:0/2723780412 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe024019040 con 0x55b512c9bf20 2024-03-06T07:21:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe00b7fe640 1 -- 172.21.15.1:0/2723780412 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 21) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe024040020 con 0x55b512c9bf20 2024-03-06T07:21:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe035ddf740 10 client.5598 fetch_fsmap finished waiting for FSMap version 21 2024-03-06T07:21:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe035ddf740 10 client.5598 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:21:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe035ddf740 20 client.5598 populate_metadata read hostname 'smithi001' 2024-03-06T07:21:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe035ddf740 1 --2- 172.21.15.1:0/2723780412 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x55b512d0c590 0x55b512d26810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe035ddf740 4 client.5598 mds_command: new command op to 4781 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:21:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe035ddf740 1 -- 172.21.15.1:0/2723780412 --> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b512be2f40 con 0x55b512d0c590 2024-03-06T07:21:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe030fa0640 1 --2- 172.21.15.1:0/2723780412 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x55b512d0c590 0x55b512d26810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe030fa0640 1 --2- 172.21.15.1:0/2723780412 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x55b512d0c590 0x55b512d26810 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9766a9514c7c916 server_cookie=e49bed0adf62a12a in_seq=0 out_seq=0 2024-03-06T07:21:43.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe00b7fe640 10 client.5598 ms_handle_connect on v2:172.21.15.1:6833/2438105345 2024-03-06T07:21:43.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe00b7fe640 1 -- 172.21.15.1:0/2723780412 <== mds.0 v2:172.21.15.1:6833/2438105345 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55b512be2f40 con 0x55b512d0c590 2024-03-06T07:21:43.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe00b7fe640 10 client.5598 handle_command_reply: tid=0 2024-03-06T07:21:43.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe035ddf740 1 client.5598 shutdown 2024-03-06T07:21:43.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe035ddf740 1 -- 172.21.15.1:0/2723780412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c9bf20 msgr2=0x55b512d0c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe035ddf740 1 --2- 172.21.15.1:0/2723780412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c9bf20 0x55b512d0c050 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe024015000 tx=0x7fe024004ab0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe035ddf740 1 -- 172.21.15.1:0/2723780412 shutdown_connections 2024-03-06T07:21:43.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe035ddf740 1 --2- 172.21.15.1:0/2723780412 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x55b512d0c590 0x55b512d26810 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe035ddf740 1 --2- 172.21.15.1:0/2723780412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b512c9bf20 0x55b512d0c050 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe035ddf740 1 --2- 172.21.15.1:0/2723780412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b512c774e0 0x55b512d249f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe00b7fe640 0 client.5598 ms_handle_reset on v2:172.21.15.1:6833/2438105345 2024-03-06T07:21:43.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe035ddf740 1 --2- 172.21.15.1:0/2723780412 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b512c77100 0x55b512cccf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.073+0000 7fe035ddf740 1 -- 172.21.15.1:0/2723780412 >> 172.21.15.1:0/2723780412 conn(0x55b512c75660 msgr2=0x55b512c9dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.074+0000 7fe035ddf740 1 -- 172.21.15.1:0/2723780412 shutdown_connections 2024-03-06T07:21:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.074+0000 7fe035ddf740 1 -- 172.21.15.1:0/2723780412 wait complete. 2024-03-06T07:21:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.074+0000 7fe035ddf740 20 client.5598 trim_cache size 0 max 16384 2024-03-06T07:21:43.082 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:21:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:21:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:21:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4781, 2024-03-06T07:21:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:21:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:21:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:21:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 5.8770140419999999, 2024-03-06T07:21:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 21, 2024-03-06T07:21:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 40, 2024-03-06T07:21:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 40, 2024-03-06T07:21:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 26.786060075999998 2024-03-06T07:21:43.083 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:21:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.074+0000 7fe031fa2640 1 -- 172.21.15.1:0/1571313585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe01006b500 msgr2=0x7fe01006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.074+0000 7fe031fa2640 1 --2- 172.21.15.1:0/1571313585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe01006b500 0x7fe01006d9c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe020036000 tx=0x7fe02003b040 comp rx=0 tx=0).stop 2024-03-06T07:21:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.074+0000 7fe031fa2640 1 -- 172.21.15.1:0/1571313585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe02c152530 msgr2=0x7fe02c152b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.074+0000 7fe031fa2640 1 --2- 172.21.15.1:0/1571313585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe02c152530 0x7fe02c152b20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe01800c3c0 tx=0x7fe01800c880 comp rx=0 tx=0).stop 2024-03-06T07:21:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.074+0000 7fe031fa2640 1 -- 172.21.15.1:0/1571313585 shutdown_connections 2024-03-06T07:21:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.074+0000 7fe031fa2640 1 --2- 172.21.15.1:0/1571313585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe01006b500 0x7fe01006d9c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.074+0000 7fe031fa2640 1 --2- 172.21.15.1:0/1571313585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe02c158d80 0x7fe02c153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.074+0000 7fe031fa2640 1 --2- 172.21.15.1:0/1571313585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe02c152530 0x7fe02c152b20 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.074+0000 7fe031fa2640 1 --2- 172.21.15.1:0/1571313585 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe02c06f9d0 0x7fe02c0ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.074+0000 7fe031fa2640 1 -- 172.21.15.1:0/1571313585 >> 172.21.15.1:0/1571313585 conn(0x7fe02c06eb50 msgr2=0x7fe02c0c4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.074+0000 7fe031fa2640 1 -- 172.21.15.1:0/1571313585 shutdown_connections 2024-03-06T07:21:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:43.074+0000 7fe031fa2640 1 -- 172.21.15.1:0/1571313585 wait complete. 2024-03-06T07:21:43.087 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 21, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:21:22.528539+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 21, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4781": { "addr": "172.21.15.1:6835/2438105345", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 2438105345, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 2438105345, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4781, "incarnation": 18, "join_fscid": 2, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-03-06T07:21:40.204549+0000", "qdb_cluster": [ 4781 ], "qdb_leader": 4781, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4781 } } } ], "standbys": [ { "addr": "172.21.15.98:6836/3354292028", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 3354292028, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 3354292028, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4972, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6837/3948803645", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 3948803645, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 3948803645, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4975, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6834/3390157192", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 3390157192, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 3390157192, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 19, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5538, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:21:43.088 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:21:43.088 INFO:tasks.cephfs.test_failover:current = [{'gid': 5538, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6834/3390157192', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 3390157192}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 3390157192}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 19}, {'gid': 4975, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/3948803645', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3948803645}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3948803645}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 4781, 'name': 'c', 'rank': 0, 'incarnation': 18, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.1:6835/2438105345', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 2438105345}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2438105345}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 4972, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/3354292028', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 3354292028}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 3354292028}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}] 2024-03-06T07:21:43.089 DEBUG:tasks.cephfs.test_failover:-1 != 2 2024-03-06T07:21:43.089 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:21:43.145 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-06T07:21:43.144+0000 7f49d74a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:48.090 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.190+0000 7f1815081640 1 Processor -- start 2024-03-06T07:21:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.190+0000 7f1815081640 1 -- start start 2024-03-06T07:21:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.190+0000 7f1815081640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1810152650 0x7f1810155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.190+0000 7f1815081640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18100bf9d0 0x7f18100c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.190+0000 7f1815081640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18100c2fc0 con 0x7f18100c6b10 2024-03-06T07:21:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.190+0000 7f1815081640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1810155800 con 0x7f1810152650 2024-03-06T07:21:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.190+0000 7f1815081640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1810158da0 con 0x7f18100bf9d0 2024-03-06T07:21:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.190+0000 7f180ed9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1810152650 0x7f1810155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.191+0000 7f180ed9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1810152650 0x7f1810155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32866/0 (socket says 172.21.15.1:32866) 2024-03-06T07:21:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.191+0000 7f180ed9d640 1 -- 172.21.15.1:0/2495059745 learned_addr learned my addr 172.21.15.1:0/2495059745 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.191+0000 7f180f59e640 1 --2- 172.21.15.1:0/2495059745 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18100bf9d0 0x7f18100c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.191+0000 7f180ed9d640 1 -- 172.21.15.1:0/2495059745 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18100bf9d0 msgr2=0x7f18100c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.191+0000 7f180ed9d640 1 --2- 172.21.15.1:0/2495059745 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18100bf9d0 0x7f18100c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.191+0000 7f180ed9d640 1 -- 172.21.15.1:0/2495059745 >> v1:172.21.15.98:6789/0 conn(0x7f18100c6b10 legacy=0x7f1810158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.191+0000 7f180ed9d640 1 -- 172.21.15.1:0/2495059745 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18100c6690 con 0x7f1810152650 2024-03-06T07:21:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.192+0000 7f180ed9d640 1 --2- 172.21.15.1:0/2495059745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1810152650 0x7f1810155c90 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f1800009870 tx=0x7f180002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2a8b0a8d72bfe3e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.192+0000 7f180dd9b640 1 -- 172.21.15.1:0/2495059745 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1800035020 con 0x7f1810152650 2024-03-06T07:21:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.192+0000 7f180dd9b640 1 -- 172.21.15.1:0/2495059745 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1800031640 con 0x7f1810152650 2024-03-06T07:21:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.192+0000 7f180dd9b640 1 -- 172.21.15.1:0/2495059745 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180002d040 con 0x7f1810152650 2024-03-06T07:21:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.193+0000 7f1815081640 1 -- 172.21.15.1:0/2495059745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1810152650 msgr2=0x7f1810155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.193+0000 7f1815081640 1 --2- 172.21.15.1:0/2495059745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1810152650 0x7f1810155c90 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f1800009870 tx=0x7f180002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.193+0000 7f1815081640 1 -- 172.21.15.1:0/2495059745 shutdown_connections 2024-03-06T07:21:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.193+0000 7f1815081640 1 --2- 172.21.15.1:0/2495059745 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18100bf9d0 0x7f18100c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.193+0000 7f1815081640 1 --2- 172.21.15.1:0/2495059745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1810152650 0x7f1810155c90 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.193+0000 7f1815081640 1 -- 172.21.15.1:0/2495059745 >> 172.21.15.1:0/2495059745 conn(0x7f1810066ee0 msgr2=0x7f18100672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.193+0000 7f1815081640 1 -- 172.21.15.1:0/2495059745 shutdown_connections 2024-03-06T07:21:48.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.193+0000 7f1815081640 1 -- 172.21.15.1:0/2495059745 wait complete. 2024-03-06T07:21:48.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.194+0000 7f1815081640 1 Processor -- start 2024-03-06T07:21:48.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.194+0000 7f1815081640 1 -- start start 2024-03-06T07:21:48.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f1815081640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18100bf9d0 0x7f18101eb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f1815081640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1810152650 0x7f1810150300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f1815081640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18100c9a10 0x7f1810156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f1815081640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18101551b0 con 0x7f18100bf9d0 2024-03-06T07:21:48.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f1815081640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18100c6fa0 con 0x7f1810152650 2024-03-06T07:21:48.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f1815081640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1810155490 con 0x7f18100c9a10 2024-03-06T07:21:48.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f180ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18100bf9d0 0x7f18101eb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f180ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18100bf9d0 0x7f18101eb270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32880/0 (socket says 172.21.15.1:32880) 2024-03-06T07:21:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f180ed9d640 1 -- 172.21.15.1:0/2635703725 learned_addr learned my addr 172.21.15.1:0/2635703725 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f180e59c640 1 --2- 172.21.15.1:0/2635703725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1810152650 0x7f1810150300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f180f59e640 1 --2- 172.21.15.1:0/2635703725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18100c9a10 0x7f1810156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f180ed9d640 1 -- 172.21.15.1:0/2635703725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18100c9a10 msgr2=0x7f1810156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f180ed9d640 1 --2- 172.21.15.1:0/2635703725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18100c9a10 0x7f1810156df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f180ed9d640 1 -- 172.21.15.1:0/2635703725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1810152650 msgr2=0x7f1810150300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f180ed9d640 1 --2- 172.21.15.1:0/2635703725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1810152650 0x7f1810150300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f180ed9d640 1 -- 172.21.15.1:0/2635703725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18100c8970 con 0x7f18100bf9d0 2024-03-06T07:21:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f180f59e640 1 --2- 172.21.15.1:0/2635703725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18100c9a10 0x7f1810156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f180e59c640 1 --2- 172.21.15.1:0/2635703725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1810152650 0x7f1810150300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.195+0000 7f180ed9d640 1 --2- 172.21.15.1:0/2635703725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18100bf9d0 0x7f18101eb270 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1800000c00 tx=0x7f18000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.196+0000 7f17f77fe640 1 -- 172.21.15.1:0/2635703725 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1800040070 con 0x7f18100bf9d0 2024-03-06T07:21:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.196+0000 7f17f77fe640 1 -- 172.21.15.1:0/2635703725 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18000399f0 con 0x7f18100bf9d0 2024-03-06T07:21:48.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.196+0000 7f17f77fe640 1 -- 172.21.15.1:0/2635703725 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1800035040 con 0x7f18100bf9d0 2024-03-06T07:21:48.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.196+0000 7f1815081640 1 -- 172.21.15.1:0/2635703725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18100c8c50 con 0x7f18100bf9d0 2024-03-06T07:21:48.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.196+0000 7f1815081640 1 -- 172.21.15.1:0/2635703725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18100c9210 con 0x7f18100bf9d0 2024-03-06T07:21:48.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.197+0000 7f1815081640 1 -- 172.21.15.1:0/2635703725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1810067fd0 con 0x7f18100bf9d0 2024-03-06T07:21:48.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.197+0000 7f17f77fe640 1 -- 172.21.15.1:0/2635703725 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f180004b060 con 0x7f18100bf9d0 2024-03-06T07:21:48.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.198+0000 7f17f77fe640 1 --2- 172.21.15.1:0/2635703725 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17fc06b4d0 0x7f17fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.198+0000 7f17f77fe640 1 -- 172.21.15.1:0/2635703725 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f18000abd70 con 0x7f18100bf9d0 2024-03-06T07:21:48.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.198+0000 7f180e59c640 1 --2- 172.21.15.1:0/2635703725 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17fc06b4d0 0x7f17fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.199+0000 7f180e59c640 1 --2- 172.21.15.1:0/2635703725 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17fc06b4d0 0x7f17fc06d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1804007fd0 tx=0x7f1804009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.206+0000 7f17f77fe640 1 -- 172.21.15.1:0/2635703725 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f180007aa00 con 0x7f18100bf9d0 2024-03-06T07:21:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.426+0000 7f1815081640 1 -- 172.21.15.1:0/2635703725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18100b01a0 con 0x7f18100bf9d0 2024-03-06T07:21:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.427+0000 7f17f77fe640 1 -- 172.21.15.1:0/2635703725 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f180007a300 con 0x7f18100bf9d0 2024-03-06T07:21:48.429 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:48.429 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3354292028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3354292028},{"type":"v1","addr":"172.21.15.98:6836","nonce":3354292028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4975,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3948803645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3948803645},{"type":"v1","addr":"172.21.15.98:6837","nonce":3948803645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5538,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3390157192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3390157192},{"type":"v1","addr":"172.21.15.1:6834","nonce":3390157192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:22.528539+0000","modified":"2024-03-06T07:21:43.207337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":10,"addr":"172.21.15.1:6835/2438105345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2438105345},{"type":"v1","addr":"172.21.15.1:6835","nonce":2438105345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4781,"qdb_cluster":[4781]},"id":2}]} 2024-03-06T07:21:48.429 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 22 2024-03-06T07:21:48.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.428+0000 7f1815081640 1 -- 172.21.15.1:0/2635703725 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17fc06b4d0 msgr2=0x7f17fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.428+0000 7f1815081640 1 --2- 172.21.15.1:0/2635703725 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17fc06b4d0 0x7f17fc06d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1804007fd0 tx=0x7f1804009040 comp rx=0 tx=0).stop 2024-03-06T07:21:48.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.428+0000 7f1815081640 1 -- 172.21.15.1:0/2635703725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18100bf9d0 msgr2=0x7f18101eb270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.428+0000 7f1815081640 1 --2- 172.21.15.1:0/2635703725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18100bf9d0 0x7f18101eb270 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1800000c00 tx=0x7f18000046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.428+0000 7f1815081640 1 -- 172.21.15.1:0/2635703725 shutdown_connections 2024-03-06T07:21:48.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.428+0000 7f1815081640 1 --2- 172.21.15.1:0/2635703725 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17fc06b4d0 0x7f17fc06d990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.428+0000 7f1815081640 1 --2- 172.21.15.1:0/2635703725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18100c9a10 0x7f1810156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.428+0000 7f1815081640 1 --2- 172.21.15.1:0/2635703725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1810152650 0x7f1810150300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.428+0000 7f1815081640 1 --2- 172.21.15.1:0/2635703725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18100bf9d0 0x7f18101eb270 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.429+0000 7f1815081640 1 -- 172.21.15.1:0/2635703725 >> 172.21.15.1:0/2635703725 conn(0x7f1810066ee0 msgr2=0x7f18100c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:48.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.429+0000 7f1815081640 1 -- 172.21.15.1:0/2635703725 shutdown_connections 2024-03-06T07:21:48.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.429+0000 7f1815081640 1 -- 172.21.15.1:0/2635703725 wait complete. 2024-03-06T07:21:48.844 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:21:22.528539+0000', 'modified': '2024-03-06T07:21:43.207337+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 40, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4781}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4781': {'gid': 4781, 'name': 'c', 'rank': 0, 'incarnation': 18, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.1:6835/2438105345', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 2438105345}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2438105345}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4781, 'qdb_cluster': [4781]} 2024-03-06T07:21:48.844 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:21:48.844 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:21:48.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.944+0000 7f56142df640 1 Processor -- start 2024-03-06T07:21:48.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.944+0000 7f56142df640 1 -- start start 2024-03-06T07:21:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.945+0000 7f56142df640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f560c0c2b30 0x7f560c0724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.945+0000 7f56142df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f560c152b70 0x7f560c072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.945+0000 7f56142df640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c151dd0 con 0x7f560c0a5fa0 2024-03-06T07:21:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.945+0000 7f56142df640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f560c1526e0 con 0x7f560c0c2b30 2024-03-06T07:21:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.945+0000 7f56142df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560c156190 con 0x7f560c152b70 2024-03-06T07:21:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.945+0000 7f561287c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f560c0a5fa0 0x7f560c0c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55460/0 (socket says 172.21.15.1:55460) 2024-03-06T07:21:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.945+0000 7f561287c640 1 -- 172.21.15.1:0/258214681 learned_addr learned my addr 172.21.15.1:0/258214681 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f561187a640 1 --2- 172.21.15.1:0/258214681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f560c152b70 0x7f560c072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f561207b640 1 --2- 172.21.15.1:0/258214681 >> v2:172.21.15.98:3300/0 conn(0x7f560c0c2b30 0x7f560c0724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f5611079640 1 -- 172.21.15.1:0/258214681 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541262548 0 0) 0x7f560c151dd0 con 0x7f560c0a5fa0 2024-03-06T07:21:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f5611079640 1 -- 172.21.15.1:0/258214681 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55f8003520 con 0x7f560c0a5fa0 2024-03-06T07:21:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f5611079640 1 -- 172.21.15.1:0/258214681 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2285495015 0 0) 0x7f55f8003520 con 0x7f560c0a5fa0 2024-03-06T07:21:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f5611079640 1 -- 172.21.15.1:0/258214681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f560c152b70 msgr2=0x7f560c072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f5611079640 1 --2- 172.21.15.1:0/258214681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f560c152b70 0x7f560c072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f5611079640 1 -- 172.21.15.1:0/258214681 >> v2:172.21.15.98:3300/0 conn(0x7f560c0c2b30 msgr2=0x7f560c0724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f5611079640 1 --2- 172.21.15.1:0/258214681 >> v2:172.21.15.98:3300/0 conn(0x7f560c0c2b30 0x7f560c0724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f5611079640 1 -- 172.21.15.1:0/258214681 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560c152090 con 0x7f560c0a5fa0 2024-03-06T07:21:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f5611079640 1 -- 172.21.15.1:0/258214681 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f5604001670 con 0x7f560c0a5fa0 2024-03-06T07:21:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f5611079640 1 -- 172.21.15.1:0/258214681 >> v1:172.21.15.1:6789/0 conn(0x7f560c0a5fa0 legacy=0x7f560c0c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f5611079640 1 --2- 172.21.15.1:0/258214681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55f80039b0 0x7f55f8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f5611079640 1 --2- 172.21.15.1:0/258214681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55f8004800 0x7f55f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f5611079640 1 --2- 172.21.15.1:0/258214681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f80051c0 0x7f55f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f5611079640 1 -- 172.21.15.1:0/258214681 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c1526e0 con 0x7f55f80039b0 2024-03-06T07:21:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f5611079640 1 -- 172.21.15.1:0/258214681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c156190 con 0x7f55f80051c0 2024-03-06T07:21:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f5611079640 1 -- 172.21.15.1:0/258214681 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f800bf00 con 0x7f55f8004800 2024-03-06T07:21:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f561187a640 1 --2- 172.21.15.1:0/258214681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f560c152b70 0x7f560c072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f561207b640 1 --2- 172.21.15.1:0/258214681 >> v2:172.21.15.98:3300/0 conn(0x7f560c0c2b30 0x7f560c0724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f561207b640 1 --2- 172.21.15.1:0/258214681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55f80039b0 0x7f55f8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f561287c640 1 --2- 172.21.15.1:0/258214681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f80051c0 0x7f55f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.946+0000 7f561187a640 1 --2- 172.21.15.1:0/258214681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55f8004800 0x7f55f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.947+0000 7f561207b640 1 -- 172.21.15.1:0/258214681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55f8004800 msgr2=0x7f55f8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.947+0000 7f561207b640 1 --2- 172.21.15.1:0/258214681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55f8004800 0x7f55f8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.947+0000 7f561207b640 1 -- 172.21.15.1:0/258214681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f80051c0 msgr2=0x7f55f800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.947+0000 7f561207b640 1 --2- 172.21.15.1:0/258214681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f80051c0 0x7f55f800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.947+0000 7f561207b640 1 -- 172.21.15.1:0/258214681 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f8003520 con 0x7f55f80039b0 2024-03-06T07:21:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.947+0000 7f561207b640 1 --2- 172.21.15.1:0/258214681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55f80039b0 0x7f55f8003dd0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5600009ca0 tx=0x7f55f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5137e41b93e35c5b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.947+0000 7f5611079640 1 -- 172.21.15.1:0/258214681 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600009840 con 0x7f55f80039b0 2024-03-06T07:21:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.947+0000 7f5611079640 1 -- 172.21.15.1:0/258214681 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5600004020 con 0x7f55f80039b0 2024-03-06T07:21:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.947+0000 7f5611079640 1 -- 172.21.15.1:0/258214681 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600004410 con 0x7f55f80039b0 2024-03-06T07:21:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.948+0000 7f56142df640 1 -- 172.21.15.1:0/258214681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55f80039b0 msgr2=0x7f55f8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.948+0000 7f56142df640 1 --2- 172.21.15.1:0/258214681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55f80039b0 0x7f55f8003dd0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5600009ca0 tx=0x7f55f8005830 comp rx=0 tx=0).stop 2024-03-06T07:21:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.948+0000 7f561287c640 1 -- 172.21.15.1:0/258214681 reap_dead start 2024-03-06T07:21:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.948+0000 7f56142df640 1 -- 172.21.15.1:0/258214681 shutdown_connections 2024-03-06T07:21:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.948+0000 7f56142df640 1 -- 172.21.15.1:0/258214681 >> 172.21.15.1:0/258214681 conn(0x7f560c0c6860 msgr2=0x7f560c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.948+0000 7f56142df640 1 -- 172.21.15.1:0/258214681 shutdown_connections 2024-03-06T07:21:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.948+0000 7f56142df640 1 -- 172.21.15.1:0/258214681 wait complete. 2024-03-06T07:21:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.949+0000 7f56142df640 1 Processor -- start 2024-03-06T07:21:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.949+0000 7f56142df640 1 -- start start 2024-03-06T07:21:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f56142df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f80039b0 0x7f560c0a6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f56142df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f560c1eed80 0x7f560c0bff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f56142df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f560c0ac160 0x7f560c0ac5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f56142df640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c155f00 con 0x7f560c0ac160 2024-03-06T07:21:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f56142df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c155d80 con 0x7f55f80039b0 2024-03-06T07:21:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f56142df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560c072a00 con 0x7f560c1eed80 2024-03-06T07:21:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f561287c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f560c0ac160 0x7f560c0ac5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f561287c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f560c0ac160 0x7f560c0ac5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32900/0 (socket says 172.21.15.1:32900) 2024-03-06T07:21:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f561287c640 1 -- 172.21.15.1:0/2863385731 learned_addr learned my addr 172.21.15.1:0/2863385731 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f561187a640 1 --2- 172.21.15.1:0/2863385731 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f560c1eed80 0x7f560c0bff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f561207b640 1 --2- 172.21.15.1:0/2863385731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f80039b0 0x7f560c0a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f561287c640 1 -- 172.21.15.1:0/2863385731 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f560c1eed80 msgr2=0x7f560c0bff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f561287c640 1 --2- 172.21.15.1:0/2863385731 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f560c1eed80 0x7f560c0bff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f561287c640 1 -- 172.21.15.1:0/2863385731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f80039b0 msgr2=0x7f560c0a6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f561287c640 1 --2- 172.21.15.1:0/2863385731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f80039b0 0x7f560c0a6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f561287c640 1 -- 172.21.15.1:0/2863385731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560c1dec10 con 0x7f560c0ac160 2024-03-06T07:21:48.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f561207b640 1 --2- 172.21.15.1:0/2863385731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f80039b0 0x7f560c0a6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:48.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.950+0000 7f561287c640 1 --2- 172.21.15.1:0/2863385731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f560c0ac160 0x7f560c0ac5e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f560c0a5f30 tx=0x7f5604007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.951+0000 7f55feffd640 1 -- 172.21.15.1:0/2863385731 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560401a070 con 0x7f560c0ac160 2024-03-06T07:21:48.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.951+0000 7f55feffd640 1 -- 172.21.15.1:0/2863385731 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560400f6b0 con 0x7f560c0ac160 2024-03-06T07:21:48.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.951+0000 7f56142df640 1 -- 172.21.15.1:0/2863385731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f560c1deef0 con 0x7f560c0ac160 2024-03-06T07:21:48.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.951+0000 7f55feffd640 1 -- 172.21.15.1:0/2863385731 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560400fa80 con 0x7f560c0ac160 2024-03-06T07:21:48.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.951+0000 7f56142df640 1 -- 172.21.15.1:0/2863385731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f560c0c0440 con 0x7f560c0ac160 2024-03-06T07:21:48.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.952+0000 7f55feffd640 1 -- 172.21.15.1:0/2863385731 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f560401d070 con 0x7f560c0ac160 2024-03-06T07:21:48.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.953+0000 7f55feffd640 1 --2- 172.21.15.1:0/2863385731 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55f006b500 0x7f55f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.953+0000 7f55feffd640 1 -- 172.21.15.1:0/2863385731 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f56040898a0 con 0x7f560c0ac160 2024-03-06T07:21:48.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.953+0000 7f561207b640 1 --2- 172.21.15.1:0/2863385731 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55f006b500 0x7f55f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.954+0000 7f561207b640 1 --2- 172.21.15.1:0/2863385731 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55f006b500 0x7f55f006d9c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5600035000 tx=0x7f5600030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.954+0000 7f5616f9f740 1 Processor -- start 2024-03-06T07:21:48.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.954+0000 7f5616f9f740 1 -- start start 2024-03-06T07:21:48.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f5616f9f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36243da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f5616f9f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 0x55f36243df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f5616f9f740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f36243e4b0 0x55f36243a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f5616f9f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3623a46b0 con 0x55f36243ff50 2024-03-06T07:21:48.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f5616f9f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f36234da60 con 0x55f362440330 2024-03-06T07:21:48.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f5616f9f740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f3623a0750 con 0x55f36243e4b0 2024-03-06T07:21:48.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f561207b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36243da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f561207b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36243da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32906/0 (socket says 172.21.15.1:32906) 2024-03-06T07:21:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f561207b640 1 -- 172.21.15.1:0/2340056084 learned_addr learned my addr 172.21.15.1:0/2340056084 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f561187a640 1 --2- 172.21.15.1:0/2340056084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 0x55f36243df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f561287c640 1 --2- 172.21.15.1:0/2340056084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f36243e4b0 0x55f36243a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f561207b640 1 -- 172.21.15.1:0/2340056084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f36243e4b0 msgr2=0x55f36243a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f561207b640 1 --2- 172.21.15.1:0/2340056084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f36243e4b0 0x55f36243a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f561207b640 1 -- 172.21.15.1:0/2340056084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 msgr2=0x55f36243df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f561207b640 1 --2- 172.21.15.1:0/2340056084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 0x55f36243df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f561207b640 1 -- 172.21.15.1:0/2340056084 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f36236bf30 con 0x55f36243ff50 2024-03-06T07:21:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f561187a640 1 --2- 172.21.15.1:0/2340056084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 0x55f36243df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f561207b640 1 --2- 172.21.15.1:0/2340056084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36243da30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f560002c510 tx=0x7f560002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b064e95e2099514c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f55effff640 1 -- 172.21.15.1:0/2340056084 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560004d070 con 0x55f36243ff50 2024-03-06T07:21:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f55effff640 1 -- 172.21.15.1:0/2340056084 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5600005da0 con 0x55f36243ff50 2024-03-06T07:21:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.955+0000 7f55effff640 1 -- 172.21.15.1:0/2340056084 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600043040 con 0x55f36243ff50 2024-03-06T07:21:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.956+0000 7f5616f9f740 1 -- 172.21.15.1:0/2340056084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 msgr2=0x55f36243da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.956+0000 7f5616f9f740 1 --2- 172.21.15.1:0/2340056084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36243da30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f560002c510 tx=0x7f560002e000 comp rx=0 tx=0).stop 2024-03-06T07:21:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.956+0000 7f5616f9f740 1 -- 172.21.15.1:0/2340056084 shutdown_connections 2024-03-06T07:21:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.956+0000 7f5616f9f740 1 --2- 172.21.15.1:0/2340056084 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f36243e4b0 0x55f36243a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.956+0000 7f5616f9f740 1 --2- 172.21.15.1:0/2340056084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 0x55f36243df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.956+0000 7f5616f9f740 1 --2- 172.21.15.1:0/2340056084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36243da30 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.956+0000 7f5616f9f740 1 -- 172.21.15.1:0/2340056084 >> 172.21.15.1:0/2340056084 conn(0x55f36243c7d0 msgr2=0x55f3624613c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.956+0000 7f5616f9f740 1 -- 172.21.15.1:0/2340056084 shutdown_connections 2024-03-06T07:21:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.956+0000 7f5616f9f740 1 -- 172.21.15.1:0/2340056084 wait complete. 2024-03-06T07:21:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.957+0000 7f5616f9f740 1 -- start start 2024-03-06T07:21:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.957+0000 7f5616f9f740 1 Processor -- start 2024-03-06T07:21:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f5616f9f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36247c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f5616f9f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 0x55f36248ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f5616f9f740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f3624660a0 0x55f3624789e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f5616f9f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f362462230 con 0x55f36243ff50 2024-03-06T07:21:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f5616f9f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f362360ea0 con 0x55f362440330 2024-03-06T07:21:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f5616f9f740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f362461c70 con 0x55f3624660a0 2024-03-06T07:21:48.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f5616f9f740 10 client.0 _ll_register_callbacks cb 0x55f362360870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:21:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f561207b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36247c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f561207b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36247c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32922/0 (socket says 172.21.15.1:32922) 2024-03-06T07:21:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f561207b640 1 -- 172.21.15.1:0/2728829902 learned_addr learned my addr 172.21.15.1:0/2728829902 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f561287c640 1 --2- 172.21.15.1:0/2728829902 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f3624660a0 0x55f3624789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f561187a640 1 --2- 172.21.15.1:0/2728829902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 0x55f36248ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f561207b640 1 -- 172.21.15.1:0/2728829902 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f3624660a0 msgr2=0x55f3624789e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f561207b640 1 --2- 172.21.15.1:0/2728829902 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f3624660a0 0x55f3624789e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f561207b640 1 -- 172.21.15.1:0/2728829902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 msgr2=0x55f36248ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f561207b640 1 --2- 172.21.15.1:0/2728829902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 0x55f36248ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f561207b640 1 -- 172.21.15.1:0/2728829902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f362361270 con 0x55f36243ff50 2024-03-06T07:21:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f561187a640 1 --2- 172.21.15.1:0/2728829902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 0x55f36248ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f561207b640 1 --2- 172.21.15.1:0/2728829902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36247c040 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5600009dc0 tx=0x7f56000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8bd7051d7403efc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f55ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:21:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.958+0000 7f55ed7fa640 1 -- 172.21.15.1:0/2728829902 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600056060 con 0x55f36243ff50 2024-03-06T07:21:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.959+0000 7f55ed7fa640 1 -- 172.21.15.1:0/2728829902 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5600006450 con 0x55f36243ff50 2024-03-06T07:21:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.959+0000 7f5616f9f740 1 -- 172.21.15.1:0/2728829902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5600005da0 con 0x55f36243ff50 2024-03-06T07:21:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.959+0000 7f55ed7fa640 1 -- 172.21.15.1:0/2728829902 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f560005c050 con 0x55f36243ff50 2024-03-06T07:21:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.959+0000 7f5616f9f740 10 client.5616 fetch_fsmap learned FSMap version 22 2024-03-06T07:21:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.959+0000 7f5616f9f740 1 -- 172.21.15.1:0/2728829902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) v3 -- 0x55f3620b92d0 con 0x55f36243ff50 2024-03-06T07:21:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.959+0000 7f55ed7fa640 1 -- 172.21.15.1:0/2728829902 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5600005da0 con 0x55f36243ff50 2024-03-06T07:21:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.959+0000 7f55ed7fa640 1 -- 172.21.15.1:0/2728829902 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 22) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f560005f020 con 0x55f36243ff50 2024-03-06T07:21:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.959+0000 7f5616f9f740 10 client.5616 fetch_fsmap finished waiting for FSMap version 22 2024-03-06T07:21:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.959+0000 7f5616f9f740 10 client.5616 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:21:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.959+0000 7f5616f9f740 20 client.5616 populate_metadata read hostname 'smithi001' 2024-03-06T07:21:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.959+0000 7f5616f9f740 1 --2- 172.21.15.1:0/2728829902 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x55f362462600 0x55f3624629e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.959+0000 7f5616f9f740 4 client.5616 mds_command: new command op to 4781 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:21:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.959+0000 7f5616f9f740 1 -- 172.21.15.1:0/2728829902 --> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f3623b1bf0 con 0x55f362462600 2024-03-06T07:21:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.959+0000 7f561187a640 1 --2- 172.21.15.1:0/2728829902 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x55f362462600 0x55f3624629e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.960+0000 7f561187a640 1 --2- 172.21.15.1:0/2728829902 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x55f362462600 0x55f3624629e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4dfde2697675579 server_cookie=6e9faffc31388c8f in_seq=0 out_seq=0 2024-03-06T07:21:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.960+0000 7f55ed7fa640 10 client.5616 ms_handle_connect on v2:172.21.15.1:6833/2438105345 2024-03-06T07:21:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.960+0000 7f55ed7fa640 1 -- 172.21.15.1:0/2728829902 <== mds.0 v2:172.21.15.1:6833/2438105345 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f3623b1bf0 con 0x55f362462600 2024-03-06T07:21:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.960+0000 7f55ed7fa640 10 client.5616 handle_command_reply: tid=0 2024-03-06T07:21:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.960+0000 7f5616f9f740 1 client.5616 shutdown 2024-03-06T07:21:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.960+0000 7f5616f9f740 1 -- 172.21.15.1:0/2728829902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 msgr2=0x55f36247c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.960+0000 7f5616f9f740 1 --2- 172.21.15.1:0/2728829902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36247c040 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5600009dc0 tx=0x7f56000046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.960+0000 7f5616f9f740 1 -- 172.21.15.1:0/2728829902 shutdown_connections 2024-03-06T07:21:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.960+0000 7f5616f9f740 1 --2- 172.21.15.1:0/2728829902 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x55f362462600 0x55f3624629e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.960+0000 7f5616f9f740 1 --2- 172.21.15.1:0/2728829902 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f3624660a0 0x55f3624789e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.960+0000 7f5616f9f740 1 --2- 172.21.15.1:0/2728829902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 0x55f36248ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.960+0000 7f55ed7fa640 0 client.5616 ms_handle_reset on v2:172.21.15.1:6833/2438105345 2024-03-06T07:21:48.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.960+0000 7f5616f9f740 1 --2- 172.21.15.1:0/2728829902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36247c040 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.960+0000 7f5616f9f740 1 -- 172.21.15.1:0/2728829902 >> 172.21.15.1:0/2728829902 conn(0x55f36243e4b0 msgr2=0x55f3624610f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:48.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.960+0000 7f5616f9f740 1 -- 172.21.15.1:0/2728829902 shutdown_connections 2024-03-06T07:21:48.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.960+0000 7f5616f9f740 1 -- 172.21.15.1:0/2728829902 wait complete. 2024-03-06T07:21:48.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.960+0000 7f5616f9f740 20 client.5616 trim_cache size 0 max 16384 2024-03-06T07:21:48.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.977+0000 7f5616f9f740 1 Processor -- start 2024-03-06T07:21:48.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.977+0000 7f5616f9f740 1 -- start start 2024-03-06T07:21:48.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.978+0000 7f5616f9f740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f36243ff50 0x55f36247ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.978+0000 7f5616f9f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 0x55f362464b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.978+0000 7f5616f9f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3624660a0 0x55f362465090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.978+0000 7f5616f9f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c1f75c0 con 0x55f3624660a0 2024-03-06T07:21:48.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.978+0000 7f5616f9f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c1f69f0 con 0x55f362440330 2024-03-06T07:21:48.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.978+0000 7f5616f9f740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560c1f6ff0 con 0x55f36243ff50 2024-03-06T07:21:48.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.978+0000 7f561287c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3624660a0 0x55f362465090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.978+0000 7f561287c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3624660a0 0x55f362465090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32926/0 (socket says 172.21.15.1:32926) 2024-03-06T07:21:48.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.978+0000 7f561287c640 1 -- 172.21.15.1:0/3731631741 learned_addr learned my addr 172.21.15.1:0/3731631741 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:48.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.978+0000 7f561287c640 1 -- 172.21.15.1:0/3731631741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f36243ff50 msgr2=0x55f36247ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:48.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.978+0000 7f561287c640 1 --2- 172.21.15.1:0/3731631741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f36243ff50 0x55f36247ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.978+0000 7f561287c640 1 -- 172.21.15.1:0/3731631741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 msgr2=0x55f362464b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:48.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.978+0000 7f561287c640 1 --2- 172.21.15.1:0/3731631741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 0x55f362464b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.978+0000 7f561287c640 1 -- 172.21.15.1:0/3731631741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3621836b0 con 0x55f3624660a0 2024-03-06T07:21:48.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.979+0000 7f561287c640 1 --2- 172.21.15.1:0/3731631741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3624660a0 0x55f362465090 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f56040600b0 tx=0x7f5604057e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f54341464ff5647 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.979+0000 7f55ef7fe640 1 -- 172.21.15.1:0/3731631741 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56040605e0 con 0x55f3624660a0 2024-03-06T07:21:48.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.979+0000 7f55ef7fe640 1 -- 172.21.15.1:0/3731631741 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5604085030 con 0x55f3624660a0 2024-03-06T07:21:48.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.979+0000 7f55ef7fe640 1 -- 172.21.15.1:0/3731631741 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56040545f0 con 0x55f3624660a0 2024-03-06T07:21:48.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.979+0000 7f5616f9f740 1 -- 172.21.15.1:0/3731631741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3624660a0 msgr2=0x55f362465090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.979+0000 7f5616f9f740 1 --2- 172.21.15.1:0/3731631741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3624660a0 0x55f362465090 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f56040600b0 tx=0x7f5604057e00 comp rx=0 tx=0).stop 2024-03-06T07:21:48.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.979+0000 7f5616f9f740 1 -- 172.21.15.1:0/3731631741 shutdown_connections 2024-03-06T07:21:48.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.979+0000 7f5616f9f740 1 --2- 172.21.15.1:0/3731631741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3624660a0 0x55f362465090 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.979+0000 7f5616f9f740 1 --2- 172.21.15.1:0/3731631741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 0x55f362464b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.979+0000 7f5616f9f740 1 --2- 172.21.15.1:0/3731631741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f36243ff50 0x55f36247ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.979+0000 7f5616f9f740 1 -- 172.21.15.1:0/3731631741 >> 172.21.15.1:0/3731631741 conn(0x55f36243e4b0 msgr2=0x55f36247e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:48.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.979+0000 7f5616f9f740 1 -- 172.21.15.1:0/3731631741 shutdown_connections 2024-03-06T07:21:48.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.979+0000 7f5616f9f740 1 -- 172.21.15.1:0/3731631741 wait complete. 2024-03-06T07:21:48.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.980+0000 7f5616f9f740 1 -- start start 2024-03-06T07:21:48.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.980+0000 7f5616f9f740 1 Processor -- start 2024-03-06T07:21:48.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f5616f9f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36243dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f5616f9f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 0x55f3624ef190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f5616f9f740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f3624660a0 0x55f3624e8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f5616f9f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f362360ea0 con 0x55f36243ff50 2024-03-06T07:21:48.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f5616f9f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f36248d880 con 0x55f362440330 2024-03-06T07:21:48.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f5616f9f740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f362462230 con 0x55f3624660a0 2024-03-06T07:21:48.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f5616f9f740 10 client.0 _ll_register_callbacks cb 0x7f55f80045b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:21:48.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f561207b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36243dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f561207b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36243dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32940/0 (socket says 172.21.15.1:32940) 2024-03-06T07:21:48.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f561207b640 1 -- 172.21.15.1:0/1941870513 learned_addr learned my addr 172.21.15.1:0/1941870513 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:48.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f561187a640 1 --2- 172.21.15.1:0/1941870513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 0x55f3624ef190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f561287c640 1 --2- 172.21.15.1:0/1941870513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f3624660a0 0x55f3624e8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f561207b640 1 -- 172.21.15.1:0/1941870513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f3624660a0 msgr2=0x55f3624e8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:48.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f561207b640 1 --2- 172.21.15.1:0/1941870513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f3624660a0 0x55f3624e8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f561207b640 1 -- 172.21.15.1:0/1941870513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 msgr2=0x55f3624ef190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:48.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f561207b640 1 --2- 172.21.15.1:0/1941870513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 0x55f3624ef190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f561207b640 1 -- 172.21.15.1:0/1941870513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f3624894f0 con 0x55f36243ff50 2024-03-06T07:21:48.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f561287c640 1 --2- 172.21.15.1:0/1941870513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f3624660a0 0x55f3624e8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:48.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f561207b640 1 --2- 172.21.15.1:0/1941870513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36243dc80 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5600006f80 tx=0x7f5600004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c901d5940c24c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f55ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:21:48.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f55ed7fa640 1 -- 172.21.15.1:0/1941870513 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600065060 con 0x55f36243ff50 2024-03-06T07:21:48.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f55ed7fa640 1 -- 172.21.15.1:0/1941870513 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5600006250 con 0x55f36243ff50 2024-03-06T07:21:48.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f5616f9f740 1 -- 172.21.15.1:0/1941870513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5604085030 con 0x55f36243ff50 2024-03-06T07:21:48.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f55ed7fa640 1 -- 172.21.15.1:0/1941870513 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f560006b050 con 0x55f36243ff50 2024-03-06T07:21:48.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.981+0000 7f5616f9f740 10 client.5622 fetch_fsmap learned FSMap version 22 2024-03-06T07:21:48.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.982+0000 7f5616f9f740 1 -- 172.21.15.1:0/1941870513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) v3 -- 0x55f3624ef6d0 con 0x55f36243ff50 2024-03-06T07:21:48.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.982+0000 7f55ed7fa640 1 -- 172.21.15.1:0/1941870513 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5604085030 con 0x55f36243ff50 2024-03-06T07:21:48.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.982+0000 7f55ed7fa640 1 -- 172.21.15.1:0/1941870513 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 22) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f560006e020 con 0x55f36243ff50 2024-03-06T07:21:48.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.982+0000 7f5616f9f740 10 client.5622 fetch_fsmap finished waiting for FSMap version 22 2024-03-06T07:21:48.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.982+0000 7f5616f9f740 10 client.5622 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:21:48.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.982+0000 7f5616f9f740 20 client.5622 populate_metadata read hostname 'smithi001' 2024-03-06T07:21:48.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.982+0000 7f5616f9f740 1 --2- 172.21.15.1:0/1941870513 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x55f36249f5c0 0x55f36249f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.982+0000 7f5616f9f740 4 client.5622 mds_command: new command op to 4781 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:21:48.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.982+0000 7f5616f9f740 1 -- 172.21.15.1:0/1941870513 --> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f3623abf40 con 0x55f36249f5c0 2024-03-06T07:21:48.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.982+0000 7f561187a640 1 --2- 172.21.15.1:0/1941870513 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x55f36249f5c0 0x55f36249f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.982+0000 7f55ed7fa640 10 client.5622 ms_handle_connect on v2:172.21.15.1:6833/2438105345 2024-03-06T07:21:48.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.982+0000 7f561187a640 1 --2- 172.21.15.1:0/1941870513 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x55f36249f5c0 0x55f36249f9a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7b7fbe78119b52e server_cookie=e21f2e62d1c38a0d in_seq=0 out_seq=0 2024-03-06T07:21:48.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.982+0000 7f55ed7fa640 1 -- 172.21.15.1:0/1941870513 <== mds.0 v2:172.21.15.1:6833/2438105345 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55f3623abf40 con 0x55f36249f5c0 2024-03-06T07:21:48.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.982+0000 7f55ed7fa640 10 client.5622 handle_command_reply: tid=0 2024-03-06T07:21:48.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.982+0000 7f5616f9f740 1 client.5622 shutdown 2024-03-06T07:21:48.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f5616f9f740 1 -- 172.21.15.1:0/1941870513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 msgr2=0x55f36243dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f5616f9f740 1 --2- 172.21.15.1:0/1941870513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36243dc80 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5600006f80 tx=0x7f5600004730 comp rx=0 tx=0).stop 2024-03-06T07:21:48.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f5616f9f740 1 -- 172.21.15.1:0/1941870513 shutdown_connections 2024-03-06T07:21:48.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f5616f9f740 1 --2- 172.21.15.1:0/1941870513 >> [v2:172.21.15.1:6833/2438105345,v1:172.21.15.1:6835/2438105345] conn(0x55f36249f5c0 0x55f36249f9a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f5616f9f740 1 --2- 172.21.15.1:0/1941870513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55f3624660a0 0x55f3624e8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f5616f9f740 1 --2- 172.21.15.1:0/1941870513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f362440330 0x55f3624ef190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f55ed7fa640 0 client.5622 ms_handle_reset on v2:172.21.15.1:6833/2438105345 2024-03-06T07:21:48.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f5616f9f740 1 --2- 172.21.15.1:0/1941870513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f36243ff50 0x55f36243dc80 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f5616f9f740 1 -- 172.21.15.1:0/1941870513 >> 172.21.15.1:0/1941870513 conn(0x55f36243e4b0 msgr2=0x55f36247ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:48.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f5616f9f740 1 -- 172.21.15.1:0/1941870513 shutdown_connections 2024-03-06T07:21:48.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f5616f9f740 1 -- 172.21.15.1:0/1941870513 wait complete. 2024-03-06T07:21:48.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f5616f9f740 20 client.5622 trim_cache size 0 max 16384 2024-03-06T07:21:48.992 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:21:48.992 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:21:48.992 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:21:48.992 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4781, 2024-03-06T07:21:48.992 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:21:48.992 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:21:48.992 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:21:48.992 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 11.786234704, 2024-03-06T07:21:48.992 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 22, 2024-03-06T07:21:48.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 40, 2024-03-06T07:21:48.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 40, 2024-03-06T07:21:48.993 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 32.695281514999998 2024-03-06T07:21:48.993 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:21:48.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f56142df640 1 -- 172.21.15.1:0/2863385731 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55f006b500 msgr2=0x7f55f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f56142df640 1 --2- 172.21.15.1:0/2863385731 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55f006b500 0x7f55f006d9c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5600035000 tx=0x7f5600030000 comp rx=0 tx=0).stop 2024-03-06T07:21:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f56142df640 1 -- 172.21.15.1:0/2863385731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f560c0ac160 msgr2=0x7f560c0ac5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f56142df640 1 --2- 172.21.15.1:0/2863385731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f560c0ac160 0x7f560c0ac5e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f560c0a5f30 tx=0x7f5604007430 comp rx=0 tx=0).stop 2024-03-06T07:21:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f56142df640 1 -- 172.21.15.1:0/2863385731 shutdown_connections 2024-03-06T07:21:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f56142df640 1 --2- 172.21.15.1:0/2863385731 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55f006b500 0x7f55f006d9c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f56142df640 1 --2- 172.21.15.1:0/2863385731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f560c0ac160 0x7f560c0ac5e0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f56142df640 1 --2- 172.21.15.1:0/2863385731 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f560c1eed80 0x7f560c0bff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f56142df640 1 --2- 172.21.15.1:0/2863385731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55f80039b0 0x7f560c0a6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f56142df640 1 -- 172.21.15.1:0/2863385731 >> 172.21.15.1:0/2863385731 conn(0x7f560c0c6860 msgr2=0x7f560c0c2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f56142df640 1 -- 172.21.15.1:0/2863385731 shutdown_connections 2024-03-06T07:21:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:48.983+0000 7f56142df640 1 -- 172.21.15.1:0/2863385731 wait complete. 2024-03-06T07:21:48.996 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 22, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:21:22.528539+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 22, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4781": { "addr": "172.21.15.1:6835/2438105345", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 2438105345, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 2438105345, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4781, "incarnation": 18, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-03-06T07:21:43.207337+0000", "qdb_cluster": [ 4781 ], "qdb_leader": 4781, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4781 } } } ], "standbys": [ { "addr": "172.21.15.98:6836/3354292028", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 3354292028, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 3354292028, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4972, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6837/3948803645", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 3948803645, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 3948803645, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4975, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6834/3390157192", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 3390157192, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 3390157192, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 19, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5538, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:21:48.997 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:21:48.997 INFO:tasks.cephfs.test_failover:current = [{'gid': 5538, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6834/3390157192', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 3390157192}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 3390157192}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 19}, {'gid': 4975, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/3948803645', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3948803645}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3948803645}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 4781, 'name': 'c', 'rank': 0, 'incarnation': 18, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.1:6835/2438105345', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 2438105345}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2438105345}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 4972, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/3354292028', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 3354292028}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 3354292028}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}] 2024-03-06T07:21:48.997 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:21:48.997 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:21:48.998 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:21:49.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:21:49.096 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:21:49.096 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:21:49.096 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:21:49.112 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:21:49.112 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:21:49.112 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:21:49.129 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:21:49.129 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:49.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.252+0000 7fefa8b25640 1 Processor -- start 2024-03-06T07:21:49.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.252+0000 7fefa8b25640 1 -- start start 2024-03-06T07:21:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.253+0000 7fefa8b25640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fefa40c3970 0x7fefa40c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.253+0000 7fefa8b25640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa40c5e40 con 0x7fefa40c60b0 2024-03-06T07:21:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.253+0000 7fefa8b25640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa40c6a40 con 0x7fefa4152840 2024-03-06T07:21:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.253+0000 7fefa8b25640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefa40c67f0 con 0x7fefa40c3970 2024-03-06T07:21:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.253+0000 7fef9bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fefa40c60b0 0x7fefa4152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55466/0 (socket says 172.21.15.1:55466) 2024-03-06T07:21:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.253+0000 7fef9bfff640 1 -- 172.21.15.1:0/312880357 learned_addr learned my addr 172.21.15.1:0/312880357 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.253+0000 7fefa1d9b640 1 -- 172.21.15.1:0/312880357 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700291201 0 0) 0x7fefa40c5e40 con 0x7fefa40c60b0 2024-03-06T07:21:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.253+0000 7fefa1d9b640 1 -- 172.21.15.1:0/312880357 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef90003520 con 0x7fefa40c60b0 2024-03-06T07:21:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.253+0000 7fefa259c640 1 --2- 172.21.15.1:0/312880357 >> v2:172.21.15.98:3300/0 conn(0x7fefa40c3970 0x7fefa40c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fefa1d9b640 1 -- 172.21.15.1:0/312880357 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4071116594 0 0) 0x7fef90003520 con 0x7fefa40c60b0 2024-03-06T07:21:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fefa1d9b640 1 -- 172.21.15.1:0/312880357 >> v2:172.21.15.98:3300/0 conn(0x7fefa40c3970 msgr2=0x7fefa40c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fefa1d9b640 1 --2- 172.21.15.1:0/312880357 >> v2:172.21.15.98:3300/0 conn(0x7fefa40c3970 0x7fefa40c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fefa1d9b640 1 -- 172.21.15.1:0/312880357 >> v1:172.21.15.98:6789/0 conn(0x7fefa4152840 legacy=0x7fefa41553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fefa1d9b640 1 -- 172.21.15.1:0/312880357 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa40c6d20 con 0x7fefa40c60b0 2024-03-06T07:21:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fefa1d9b640 1 -- 172.21.15.1:0/312880357 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fef8c001670 con 0x7fefa40c60b0 2024-03-06T07:21:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fefa1d9b640 1 -- 172.21.15.1:0/312880357 >> v1:172.21.15.1:6789/0 conn(0x7fefa40c60b0 legacy=0x7fefa4152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fefa1d9b640 1 --2- 172.21.15.1:0/312880357 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef900039b0 0x7fef90003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fefa1d9b640 1 --2- 172.21.15.1:0/312880357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef90004800 0x7fef90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fefa1d9b640 1 --2- 172.21.15.1:0/312880357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef900051c0 0x7fef9000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fefa1d9b640 1 -- 172.21.15.1:0/312880357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa40c67f0 con 0x7fef90004800 2024-03-06T07:21:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fefa1d9b640 1 -- 172.21.15.1:0/312880357 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9000c060 con 0x7fef900051c0 2024-03-06T07:21:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fefa1d9b640 1 -- 172.21.15.1:0/312880357 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9000c260 con 0x7fef900039b0 2024-03-06T07:21:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fef9bfff640 1 --2- 172.21.15.1:0/312880357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef90004800 0x7fef90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fefa259c640 1 --2- 172.21.15.1:0/312880357 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef900039b0 0x7fef90003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fefa2d9d640 1 --2- 172.21.15.1:0/312880357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef900051c0 0x7fef9000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fef9bfff640 1 -- 172.21.15.1:0/312880357 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef900039b0 msgr2=0x7fef90003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:49.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fef9bfff640 1 --2- 172.21.15.1:0/312880357 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef900039b0 0x7fef90003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fef9bfff640 1 -- 172.21.15.1:0/312880357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef900051c0 msgr2=0x7fef9000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:49.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fef9bfff640 1 --2- 172.21.15.1:0/312880357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef900051c0 0x7fef9000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fef9bfff640 1 -- 172.21.15.1:0/312880357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef90003520 con 0x7fef90004800 2024-03-06T07:21:49.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fefa2d9d640 1 --2- 172.21.15.1:0/312880357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef900051c0 0x7fef9000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:49.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.254+0000 7fefa2d9d640 1 -- 172.21.15.1:0/312880357 reap_dead start 2024-03-06T07:21:49.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.255+0000 7fef9bfff640 1 --2- 172.21.15.1:0/312880357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef90004800 0x7fef90004c80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fefa40732d0 tx=0x7fef90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0753bf17fc570f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:49.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.255+0000 7fefa1d9b640 1 -- 172.21.15.1:0/312880357 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c007cf0 con 0x7fef90004800 2024-03-06T07:21:49.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.255+0000 7fefa1d9b640 1 -- 172.21.15.1:0/312880357 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8c030070 con 0x7fef90004800 2024-03-06T07:21:49.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.255+0000 7fefa1d9b640 1 -- 172.21.15.1:0/312880357 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c005d30 con 0x7fef90004800 2024-03-06T07:21:49.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.256+0000 7fefa8b25640 1 -- 172.21.15.1:0/312880357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef90004800 msgr2=0x7fef90004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.256+0000 7fefa8b25640 1 --2- 172.21.15.1:0/312880357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef90004800 0x7fef90004c80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fefa40732d0 tx=0x7fef90005830 comp rx=0 tx=0).stop 2024-03-06T07:21:49.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.256+0000 7fefa8b25640 1 -- 172.21.15.1:0/312880357 shutdown_connections 2024-03-06T07:21:49.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.256+0000 7fefa8b25640 1 --2- 172.21.15.1:0/312880357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef90004800 0x7fef90004c80 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.256+0000 7fefa8b25640 1 -- 172.21.15.1:0/312880357 >> 172.21.15.1:0/312880357 conn(0x7fefa40c88f0 msgr2=0x7fefa40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:49.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.256+0000 7fefa8b25640 1 -- 172.21.15.1:0/312880357 shutdown_connections 2024-03-06T07:21:49.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.256+0000 7fefa8b25640 1 -- 172.21.15.1:0/312880357 wait complete. 2024-03-06T07:21:49.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.257+0000 7fefa8b25640 1 Processor -- start 2024-03-06T07:21:49.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.258+0000 7fefa8b25640 1 -- start start 2024-03-06T07:21:49.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa8b25640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef90004800 0x7fefa4070ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa8b25640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefa414da50 0x7fefa40acc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa8b25640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa40adc90 0x7fefa40a8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa8b25640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa40c75c0 con 0x7fef90004800 2024-03-06T07:21:49.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa8b25640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa40c72e0 con 0x7fefa40adc90 2024-03-06T07:21:49.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa8b25640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa40c3d50 con 0x7fefa414da50 2024-03-06T07:21:49.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef90004800 0x7fefa4070ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef90004800 0x7fefa4070ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32956/0 (socket says 172.21.15.1:32956) 2024-03-06T07:21:49.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa259c640 1 -- 172.21.15.1:0/866356756 learned_addr learned my addr 172.21.15.1:0/866356756 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:49.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fef9bfff640 1 --2- 172.21.15.1:0/866356756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefa414da50 0x7fefa40acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa2d9d640 1 --2- 172.21.15.1:0/866356756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa40adc90 0x7fefa40a8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa259c640 1 -- 172.21.15.1:0/866356756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefa414da50 msgr2=0x7fefa40acc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa259c640 1 --2- 172.21.15.1:0/866356756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefa414da50 0x7fefa40acc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa259c640 1 -- 172.21.15.1:0/866356756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa40adc90 msgr2=0x7fefa40a8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa259c640 1 --2- 172.21.15.1:0/866356756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa40adc90 0x7fefa40a8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa259c640 1 -- 172.21.15.1:0/866356756 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa40ad460 con 0x7fef90004800 2024-03-06T07:21:49.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa2d9d640 1 --2- 172.21.15.1:0/866356756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa40adc90 0x7fefa40a8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:49.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa259c640 1 --2- 172.21.15.1:0/866356756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef90004800 0x7fefa4070ab0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fef94011a70 tx=0x7fef94011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:49.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fef9affd640 1 -- 172.21.15.1:0/866356756 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9400fb50 con 0x7fef90004800 2024-03-06T07:21:49.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fef9affd640 1 -- 172.21.15.1:0/866356756 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9400fcf0 con 0x7fef90004800 2024-03-06T07:21:49.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa8b25640 1 -- 172.21.15.1:0/866356756 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa40ad700 con 0x7fef90004800 2024-03-06T07:21:49.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fef9affd640 1 -- 172.21.15.1:0/866356756 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9400fea0 con 0x7fef90004800 2024-03-06T07:21:49.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.259+0000 7fefa8b25640 1 -- 172.21.15.1:0/866356756 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa40ad160 con 0x7fef90004800 2024-03-06T07:21:49.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.261+0000 7fefa8b25640 1 -- 172.21.15.1:0/866356756 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa40686e0 con 0x7fef90004800 2024-03-06T07:21:49.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.261+0000 7fef9affd640 1 -- 172.21.15.1:0/866356756 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef94012040 con 0x7fef90004800 2024-03-06T07:21:49.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.262+0000 7fef9affd640 1 --2- 172.21.15.1:0/866356756 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fef8406b4d0 0x7fef8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.262+0000 7fef9affd640 1 -- 172.21.15.1:0/866356756 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fef9408aaf0 con 0x7fef90004800 2024-03-06T07:21:49.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.262+0000 7fef9bfff640 1 --2- 172.21.15.1:0/866356756 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fef8406b4d0 0x7fef8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.263+0000 7fef9bfff640 1 --2- 172.21.15.1:0/866356756 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fef8406b4d0 0x7fef8406d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fef8c034000 tx=0x7fef8c007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:49.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.266+0000 7fef9affd640 1 -- 172.21.15.1:0/866356756 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef94019030 con 0x7fef90004800 2024-03-06T07:21:49.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.485+0000 7fefa8b25640 1 -- 172.21.15.1:0/866356756 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefa40c8210 con 0x7fef90004800 2024-03-06T07:21:49.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.485+0000 7fef9affd640 1 -- 172.21.15.1:0/866356756 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fef94059780 con 0x7fef90004800 2024-03-06T07:21:49.487 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:49.488 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3354292028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3354292028},{"type":"v1","addr":"172.21.15.98:6836","nonce":3354292028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4975,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3948803645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3948803645},{"type":"v1","addr":"172.21.15.98:6837","nonce":3948803645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5538,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3390157192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3390157192},{"type":"v1","addr":"172.21.15.1:6834","nonce":3390157192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:22.528539+0000","modified":"2024-03-06T07:21:43.207337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":10,"addr":"172.21.15.1:6835/2438105345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2438105345},{"type":"v1","addr":"172.21.15.1:6835","nonce":2438105345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4781,"qdb_cluster":[4781]},"id":2}]} 2024-03-06T07:21:49.488 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 22 2024-03-06T07:21:49.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.487+0000 7fefa8b25640 1 -- 172.21.15.1:0/866356756 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fef8406b4d0 msgr2=0x7fef8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.487+0000 7fefa8b25640 1 --2- 172.21.15.1:0/866356756 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fef8406b4d0 0x7fef8406d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fef8c034000 tx=0x7fef8c007000 comp rx=0 tx=0).stop 2024-03-06T07:21:49.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.487+0000 7fefa8b25640 1 -- 172.21.15.1:0/866356756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef90004800 msgr2=0x7fefa4070ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.487+0000 7fefa8b25640 1 --2- 172.21.15.1:0/866356756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef90004800 0x7fefa4070ab0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fef94011a70 tx=0x7fef94011f30 comp rx=0 tx=0).stop 2024-03-06T07:21:49.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.487+0000 7fefa8b25640 1 -- 172.21.15.1:0/866356756 shutdown_connections 2024-03-06T07:21:49.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.487+0000 7fefa8b25640 1 --2- 172.21.15.1:0/866356756 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fef8406b4d0 0x7fef8406d990 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.487+0000 7fefa8b25640 1 --2- 172.21.15.1:0/866356756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa40adc90 0x7fefa40a8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.487+0000 7fefa8b25640 1 --2- 172.21.15.1:0/866356756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fefa414da50 0x7fefa40acc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.487+0000 7fefa8b25640 1 --2- 172.21.15.1:0/866356756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef90004800 0x7fefa4070ab0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.487+0000 7fefa8b25640 1 -- 172.21.15.1:0/866356756 >> 172.21.15.1:0/866356756 conn(0x7fefa40c88f0 msgr2=0x7fefa406c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:49.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.487+0000 7fefa8b25640 1 -- 172.21.15.1:0/866356756 shutdown_connections 2024-03-06T07:21:49.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.487+0000 7fefa8b25640 1 -- 172.21.15.1:0/866356756 wait complete. 2024-03-06T07:21:49.500 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:49.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.600+0000 7fa0e957a640 1 Processor -- start 2024-03-06T07:21:49.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.600+0000 7fa0e957a640 1 -- start start 2024-03-06T07:21:49.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.601+0000 7fa0e957a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0e40c3970 0x7fa0e40c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.601+0000 7fa0e957a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e40c5e40 con 0x7fa0e40c60b0 2024-03-06T07:21:49.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.601+0000 7fa0e957a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e40c6a40 con 0x7fa0e4152840 2024-03-06T07:21:49.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.601+0000 7fa0e957a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e40c67f0 con 0x7fa0e40c3970 2024-03-06T07:21:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.601+0000 7fa0dbfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa0e40c60b0 0x7fa0e4152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55482/0 (socket says 172.21.15.1:55482) 2024-03-06T07:21:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.601+0000 7fa0dbfff640 1 -- 172.21.15.1:0/2533967451 learned_addr learned my addr 172.21.15.1:0/2533967451 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.601+0000 7fa0e27fc640 1 -- 172.21.15.1:0/2533967451 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196445889 0 0) 0x7fa0e40c5e40 con 0x7fa0e40c60b0 2024-03-06T07:21:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0e2ffd640 1 --2- 172.21.15.1:0/2533967451 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0e40c3970 0x7fa0e40c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0e27fc640 1 -- 172.21.15.1:0/2533967451 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0d0003520 con 0x7fa0e40c60b0 2024-03-06T07:21:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0e27fc640 1 -- 172.21.15.1:0/2533967451 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3451638027 0 0) 0x7fa0d0003520 con 0x7fa0e40c60b0 2024-03-06T07:21:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0e27fc640 1 -- 172.21.15.1:0/2533967451 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0e40c3970 msgr2=0x7fa0e40c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0e27fc640 1 --2- 172.21.15.1:0/2533967451 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0e40c3970 0x7fa0e40c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0e27fc640 1 -- 172.21.15.1:0/2533967451 >> v1:172.21.15.98:6789/0 conn(0x7fa0e4152840 legacy=0x7fa0e41553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0e27fc640 1 -- 172.21.15.1:0/2533967451 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e40c6d20 con 0x7fa0e40c60b0 2024-03-06T07:21:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0e27fc640 1 -- 172.21.15.1:0/2533967451 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fa0cc001670 con 0x7fa0e40c60b0 2024-03-06T07:21:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0e27fc640 1 -- 172.21.15.1:0/2533967451 >> v1:172.21.15.1:6789/0 conn(0x7fa0e40c60b0 legacy=0x7fa0e4152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0e27fc640 1 --2- 172.21.15.1:0/2533967451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d00039e0 0x7fa0d0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0e27fc640 1 --2- 172.21.15.1:0/2533967451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d0004800 0x7fa0d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0e27fc640 1 --2- 172.21.15.1:0/2533967451 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0e40c3970 0x7fa0d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0e27fc640 1 -- 172.21.15.1:0/2533967451 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e40c67f0 con 0x7fa0d0004800 2024-03-06T07:21:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0e27fc640 1 -- 172.21.15.1:0/2533967451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d000ba70 con 0x7fa0d00039e0 2024-03-06T07:21:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0e27fc640 1 -- 172.21.15.1:0/2533967451 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d000bc70 con 0x7fa0e40c3970 2024-03-06T07:21:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0dbfff640 1 --2- 172.21.15.1:0/2533967451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d0004800 0x7fa0d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0e2ffd640 1 --2- 172.21.15.1:0/2533967451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d00039e0 0x7fa0d0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0e37fe640 1 --2- 172.21.15.1:0/2533967451 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0e40c3970 0x7fa0d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0dbfff640 1 -- 172.21.15.1:0/2533967451 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0e40c3970 msgr2=0x7fa0d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0dbfff640 1 --2- 172.21.15.1:0/2533967451 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0e40c3970 0x7fa0d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0dbfff640 1 -- 172.21.15.1:0/2533967451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d00039e0 msgr2=0x7fa0d0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0dbfff640 1 --2- 172.21.15.1:0/2533967451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d00039e0 0x7fa0d0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.602+0000 7fa0dbfff640 1 -- 172.21.15.1:0/2533967451 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d0003520 con 0x7fa0d0004800 2024-03-06T07:21:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.603+0000 7fa0dbfff640 1 --2- 172.21.15.1:0/2533967451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d0004800 0x7fa0d0004c80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40c64b0 tx=0x7fa0d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671d228837194553 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.603+0000 7fa0e27fc640 1 -- 172.21.15.1:0/2533967451 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc007cf0 con 0x7fa0d0004800 2024-03-06T07:21:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.603+0000 7fa0e27fc640 1 -- 172.21.15.1:0/2533967451 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0cc030070 con 0x7fa0d0004800 2024-03-06T07:21:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.603+0000 7fa0e27fc640 1 -- 172.21.15.1:0/2533967451 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc005d30 con 0x7fa0d0004800 2024-03-06T07:21:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.604+0000 7fa0e957a640 1 -- 172.21.15.1:0/2533967451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d0004800 msgr2=0x7fa0d0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.604+0000 7fa0e957a640 1 --2- 172.21.15.1:0/2533967451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d0004800 0x7fa0d0004c80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40c64b0 tx=0x7fa0d0005830 comp rx=0 tx=0).stop 2024-03-06T07:21:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.604+0000 7fa0e37fe640 1 -- 172.21.15.1:0/2533967451 reap_dead start 2024-03-06T07:21:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.604+0000 7fa0e957a640 1 -- 172.21.15.1:0/2533967451 shutdown_connections 2024-03-06T07:21:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.604+0000 7fa0e957a640 1 -- 172.21.15.1:0/2533967451 >> 172.21.15.1:0/2533967451 conn(0x7fa0e40c88f0 msgr2=0x7fa0e40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.604+0000 7fa0e957a640 1 -- 172.21.15.1:0/2533967451 shutdown_connections 2024-03-06T07:21:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.604+0000 7fa0e957a640 1 -- 172.21.15.1:0/2533967451 wait complete. 2024-03-06T07:21:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.605+0000 7fa0e957a640 1 Processor -- start 2024-03-06T07:21:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.605+0000 7fa0e957a640 1 -- start start 2024-03-06T07:21:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.606+0000 7fa0e957a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0e40707a0 0x7fa0e41ee3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.606+0000 7fa0e957a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0e41564b0 0x7fa0e41ee920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.606+0000 7fa0e957a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0e414b8a0 0x7fa0e414bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.606+0000 7fa0e957a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e40c75c0 con 0x7fa0e40707a0 2024-03-06T07:21:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.606+0000 7fa0e957a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e40c72e0 con 0x7fa0e414b8a0 2024-03-06T07:21:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.606+0000 7fa0e957a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e40c3d50 con 0x7fa0e41564b0 2024-03-06T07:21:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.606+0000 7fa0e2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0e40707a0 0x7fa0e41ee3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.606+0000 7fa0e2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0e40707a0 0x7fa0e41ee3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32976/0 (socket says 172.21.15.1:32976) 2024-03-06T07:21:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.606+0000 7fa0e2ffd640 1 -- 172.21.15.1:0/3577938748 learned_addr learned my addr 172.21.15.1:0/3577938748 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.606+0000 7fa0e2ffd640 1 -- 172.21.15.1:0/3577938748 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0e41564b0 msgr2=0x7fa0e41ee920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.606+0000 7fa0e2ffd640 1 --2- 172.21.15.1:0/3577938748 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0e41564b0 0x7fa0e41ee920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.606+0000 7fa0e2ffd640 1 -- 172.21.15.1:0/3577938748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0e414b8a0 msgr2=0x7fa0e414bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.606+0000 7fa0e2ffd640 1 --2- 172.21.15.1:0/3577938748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0e414b8a0 0x7fa0e414bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.607+0000 7fa0e2ffd640 1 -- 172.21.15.1:0/3577938748 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e41eaa60 con 0x7fa0e40707a0 2024-03-06T07:21:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.607+0000 7fa0e2ffd640 1 --2- 172.21.15.1:0/3577938748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0e40707a0 0x7fa0e41ee3e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa0d40119d0 tx=0x7fa0d4011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.607+0000 7fa0db7fe640 1 -- 172.21.15.1:0/3577938748 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d400fad0 con 0x7fa0e40707a0 2024-03-06T07:21:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.607+0000 7fa0db7fe640 1 -- 172.21.15.1:0/3577938748 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d400fc70 con 0x7fa0e40707a0 2024-03-06T07:21:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.607+0000 7fa0e957a640 1 -- 172.21.15.1:0/3577938748 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e40ab9a0 con 0x7fa0e40707a0 2024-03-06T07:21:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.607+0000 7fa0db7fe640 1 -- 172.21.15.1:0/3577938748 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4014440 con 0x7fa0e40707a0 2024-03-06T07:21:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.607+0000 7fa0e957a640 1 -- 172.21.15.1:0/3577938748 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e414d2f0 con 0x7fa0e40707a0 2024-03-06T07:21:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.608+0000 7fa0e957a640 1 -- 172.21.15.1:0/3577938748 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e406e900 con 0x7fa0e40707a0 2024-03-06T07:21:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.609+0000 7fa0db7fe640 1 -- 172.21.15.1:0/3577938748 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0d4012040 con 0x7fa0e40707a0 2024-03-06T07:21:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.609+0000 7fa0db7fe640 1 --2- 172.21.15.1:0/3577938748 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0c406b500 0x7fa0c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.610+0000 7fa0db7fe640 1 -- 172.21.15.1:0/3577938748 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa0d408a970 con 0x7fa0e40707a0 2024-03-06T07:21:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.610+0000 7fa0dbfff640 1 --2- 172.21.15.1:0/3577938748 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0c406b500 0x7fa0c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.610+0000 7fa0dbfff640 1 --2- 172.21.15.1:0/3577938748 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0c406b500 0x7fa0c406d9c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc034000 tx=0x7fa0cc02e000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:49.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.614+0000 7fa0db7fe640 1 -- 172.21.15.1:0/3577938748 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0e406e900 con 0x7fa0e40707a0 2024-03-06T07:21:49.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.796+0000 7fa0e957a640 1 -- 172.21.15.1:0/3577938748 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0e40686e0 con 0x7fa0e40707a0 2024-03-06T07:21:49.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.797+0000 7fa0db7fe640 1 -- 172.21.15.1:0/3577938748 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+16949 (secure 0 0 0) 0x7fa0d4019030 con 0x7fa0e40707a0 2024-03-06T07:21:49.799 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:49.799 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":40,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:21:37.195560+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:21.073384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:21.484387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6834/1818141892":"2024-03-07T07:21:37.195540+0000","172.21.15.1:6832/1818141892":"2024-03-07T07:21:37.195540+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:49.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.799+0000 7fa0e957a640 1 -- 172.21.15.1:0/3577938748 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0c406b500 msgr2=0x7fa0c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.799+0000 7fa0e957a640 1 --2- 172.21.15.1:0/3577938748 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0c406b500 0x7fa0c406d9c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc034000 tx=0x7fa0cc02e000 comp rx=0 tx=0).stop 2024-03-06T07:21:49.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.799+0000 7fa0e957a640 1 -- 172.21.15.1:0/3577938748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0e40707a0 msgr2=0x7fa0e41ee3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.799+0000 7fa0e957a640 1 --2- 172.21.15.1:0/3577938748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0e40707a0 0x7fa0e41ee3e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa0d40119d0 tx=0x7fa0d4011e90 comp rx=0 tx=0).stop 2024-03-06T07:21:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.799+0000 7fa0e957a640 1 -- 172.21.15.1:0/3577938748 shutdown_connections 2024-03-06T07:21:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.799+0000 7fa0e957a640 1 --2- 172.21.15.1:0/3577938748 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0c406b500 0x7fa0c406d9c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.799+0000 7fa0e957a640 1 --2- 172.21.15.1:0/3577938748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0e414b8a0 0x7fa0e414bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.799+0000 7fa0e957a640 1 --2- 172.21.15.1:0/3577938748 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0e41564b0 0x7fa0e41ee920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.799+0000 7fa0e957a640 1 --2- 172.21.15.1:0/3577938748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0e40707a0 0x7fa0e41ee3e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.799+0000 7fa0e957a640 1 -- 172.21.15.1:0/3577938748 >> 172.21.15.1:0/3577938748 conn(0x7fa0e40c88f0 msgr2=0x7fa0e41500f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.799+0000 7fa0e957a640 1 -- 172.21.15.1:0/3577938748 shutdown_connections 2024-03-06T07:21:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.799+0000 7fa0e957a640 1 -- 172.21.15.1:0/3577938748 wait complete. 2024-03-06T07:21:49.812 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:21.073384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:21.484387+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:49.812 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:49.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.916+0000 7fa532fb5640 1 Processor -- start 2024-03-06T07:21:49.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.916+0000 7fa532fb5640 1 -- start start 2024-03-06T07:21:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa532fb5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa52c0c05f0 0x7fa52c0c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa532fb5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa52c0c14c0 0x7fa52c1558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa532fb5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa52c0c31f0 con 0x7fa52c06f640 2024-03-06T07:21:49.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa532fb5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa52c06fe90 con 0x7fa52c0c05f0 2024-03-06T07:21:49.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa532fb5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c0c2fc0 con 0x7fa52c0c14c0 2024-03-06T07:21:49.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa52bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa52c06f640 0x7fa52c1551b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55498/0 (socket says 172.21.15.1:55498) 2024-03-06T07:21:49.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa52bfff640 1 -- 172.21.15.1:0/2595827819 learned_addr learned my addr 172.21.15.1:0/2595827819 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:49.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa531552640 1 --2- 172.21.15.1:0/2595827819 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa52c0c14c0 0x7fa52c1558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa530d51640 1 --2- 172.21.15.1:0/2595827819 >> v2:172.21.15.98:3300/0 conn(0x7fa52c0c05f0 0x7fa52c0c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa52b7fe640 1 -- 172.21.15.1:0/2595827819 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792717998 0 0) 0x7fa52c0c31f0 con 0x7fa52c06f640 2024-03-06T07:21:49.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa52b7fe640 1 -- 172.21.15.1:0/2595827819 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa51c003520 con 0x7fa52c06f640 2024-03-06T07:21:49.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa52b7fe640 1 -- 172.21.15.1:0/2595827819 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3728320816 0 0) 0x7fa51c003520 con 0x7fa52c06f640 2024-03-06T07:21:49.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa52b7fe640 1 -- 172.21.15.1:0/2595827819 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa52c0c14c0 msgr2=0x7fa52c1558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:49.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa52b7fe640 1 --2- 172.21.15.1:0/2595827819 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa52c0c14c0 0x7fa52c1558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa52b7fe640 1 -- 172.21.15.1:0/2595827819 >> v2:172.21.15.98:3300/0 conn(0x7fa52c0c05f0 msgr2=0x7fa52c0c09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:49.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa52b7fe640 1 --2- 172.21.15.1:0/2595827819 >> v2:172.21.15.98:3300/0 conn(0x7fa52c0c05f0 0x7fa52c0c09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa52b7fe640 1 -- 172.21.15.1:0/2595827819 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c155e00 con 0x7fa52c06f640 2024-03-06T07:21:49.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.917+0000 7fa52b7fe640 1 -- 172.21.15.1:0/2595827819 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fa518001670 con 0x7fa52c06f640 2024-03-06T07:21:49.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.918+0000 7fa52b7fe640 1 -- 172.21.15.1:0/2595827819 >> v1:172.21.15.1:6789/0 conn(0x7fa52c06f640 legacy=0x7fa52c1551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.918+0000 7fa52b7fe640 1 --2- 172.21.15.1:0/2595827819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa51c0039b0 0x7fa51c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.918+0000 7fa52b7fe640 1 --2- 172.21.15.1:0/2595827819 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa52c0c14c0 0x7fa51c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.918+0000 7fa52b7fe640 1 --2- 172.21.15.1:0/2595827819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa51c004d40 0x7fa51c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.918+0000 7fa52b7fe640 1 -- 172.21.15.1:0/2595827819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c06fe90 con 0x7fa51c0039b0 2024-03-06T07:21:49.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.918+0000 7fa52b7fe640 1 -- 172.21.15.1:0/2595827819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c00b9c0 con 0x7fa51c004d40 2024-03-06T07:21:49.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.918+0000 7fa52b7fe640 1 -- 172.21.15.1:0/2595827819 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c00bb40 con 0x7fa52c0c14c0 2024-03-06T07:21:49.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.918+0000 7fa530d51640 1 --2- 172.21.15.1:0/2595827819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa51c0039b0 0x7fa51c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.918+0000 7fa52bfff640 1 --2- 172.21.15.1:0/2595827819 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa52c0c14c0 0x7fa51c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.918+0000 7fa531552640 1 --2- 172.21.15.1:0/2595827819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa51c004d40 0x7fa51c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.918+0000 7fa530d51640 1 -- 172.21.15.1:0/2595827819 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa52c0c14c0 msgr2=0x7fa51c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:49.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.918+0000 7fa530d51640 1 --2- 172.21.15.1:0/2595827819 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa52c0c14c0 0x7fa51c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.918+0000 7fa530d51640 1 -- 172.21.15.1:0/2595827819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa51c004d40 msgr2=0x7fa51c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:49.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.918+0000 7fa530d51640 1 --2- 172.21.15.1:0/2595827819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa51c004d40 0x7fa51c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.918+0000 7fa530d51640 1 -- 172.21.15.1:0/2595827819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c003520 con 0x7fa51c0039b0 2024-03-06T07:21:49.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.919+0000 7fa530d51640 1 --2- 172.21.15.1:0/2595827819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa51c0039b0 0x7fa51c003dd0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa520009ac0 tx=0x7fa51c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4da48685a28dc59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:49.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.919+0000 7fa52b7fe640 1 -- 172.21.15.1:0/2595827819 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa520039070 con 0x7fa51c0039b0 2024-03-06T07:21:49.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.919+0000 7fa52b7fe640 1 -- 172.21.15.1:0/2595827819 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52002c710 con 0x7fa51c0039b0 2024-03-06T07:21:49.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.919+0000 7fa52b7fe640 1 -- 172.21.15.1:0/2595827819 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa520009690 con 0x7fa51c0039b0 2024-03-06T07:21:49.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.920+0000 7fa532fb5640 1 -- 172.21.15.1:0/2595827819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa51c0039b0 msgr2=0x7fa51c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.920+0000 7fa532fb5640 1 --2- 172.21.15.1:0/2595827819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa51c0039b0 0x7fa51c003dd0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa520009ac0 tx=0x7fa51c005830 comp rx=0 tx=0).stop 2024-03-06T07:21:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.920+0000 7fa531552640 1 -- 172.21.15.1:0/2595827819 reap_dead start 2024-03-06T07:21:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.920+0000 7fa532fb5640 1 -- 172.21.15.1:0/2595827819 shutdown_connections 2024-03-06T07:21:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.920+0000 7fa532fb5640 1 -- 172.21.15.1:0/2595827819 >> 172.21.15.1:0/2595827819 conn(0x7fa52c0c3cc0 msgr2=0x7fa52c0c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.920+0000 7fa532fb5640 1 -- 172.21.15.1:0/2595827819 shutdown_connections 2024-03-06T07:21:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.920+0000 7fa532fb5640 1 -- 172.21.15.1:0/2595827819 wait complete. 2024-03-06T07:21:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.921+0000 7fa532fb5640 1 Processor -- start 2024-03-06T07:21:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.921+0000 7fa532fb5640 1 -- start start 2024-03-06T07:21:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.922+0000 7fa532fb5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa51c0039b0 0x7fa52c0c6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.922+0000 7fa532fb5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa52c154b20 0x7fa52c14d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.922+0000 7fa532fb5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa52c1eb2b0 0x7fa52c1edc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.922+0000 7fa532fb5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c0c3770 con 0x7fa51c0039b0 2024-03-06T07:21:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.922+0000 7fa532fb5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c0c3490 con 0x7fa52c1eb2b0 2024-03-06T07:21:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.922+0000 7fa532fb5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c0c1040 con 0x7fa52c154b20 2024-03-06T07:21:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.922+0000 7fa530d51640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa51c0039b0 0x7fa52c0c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.922+0000 7fa530d51640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa51c0039b0 0x7fa52c0c6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33002/0 (socket says 172.21.15.1:33002) 2024-03-06T07:21:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.922+0000 7fa530d51640 1 -- 172.21.15.1:0/2851825477 learned_addr learned my addr 172.21.15.1:0/2851825477 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.922+0000 7fa530d51640 1 -- 172.21.15.1:0/2851825477 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa52c154b20 msgr2=0x7fa52c14d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.922+0000 7fa530d51640 1 --2- 172.21.15.1:0/2851825477 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa52c154b20 0x7fa52c14d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.922+0000 7fa530d51640 1 -- 172.21.15.1:0/2851825477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa52c1eb2b0 msgr2=0x7fa52c1edc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.922+0000 7fa530d51640 1 --2- 172.21.15.1:0/2851825477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa52c1eb2b0 0x7fa52c1edc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.922+0000 7fa530d51640 1 -- 172.21.15.1:0/2851825477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c0c67a0 con 0x7fa51c0039b0 2024-03-06T07:21:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.922+0000 7fa530d51640 1 --2- 172.21.15.1:0/2851825477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa51c0039b0 0x7fa52c0c6260 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fa520009be0 tx=0x7fa520004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.923+0000 7fa5297fa640 1 -- 172.21.15.1:0/2851825477 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa520042020 con 0x7fa51c0039b0 2024-03-06T07:21:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.923+0000 7fa5297fa640 1 -- 172.21.15.1:0/2851825477 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52003a040 con 0x7fa51c0039b0 2024-03-06T07:21:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.923+0000 7fa532fb5640 1 -- 172.21.15.1:0/2851825477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52c0c6a50 con 0x7fa51c0039b0 2024-03-06T07:21:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.923+0000 7fa5297fa640 1 -- 172.21.15.1:0/2851825477 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa520039070 con 0x7fa51c0039b0 2024-03-06T07:21:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.923+0000 7fa532fb5640 1 -- 172.21.15.1:0/2851825477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52c1eaab0 con 0x7fa51c0039b0 2024-03-06T07:21:49.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.924+0000 7fa532fb5640 1 -- 172.21.15.1:0/2851825477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52c067fd0 con 0x7fa51c0039b0 2024-03-06T07:21:49.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.924+0000 7fa5297fa640 1 -- 172.21.15.1:0/2851825477 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa520045020 con 0x7fa51c0039b0 2024-03-06T07:21:49.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.924+0000 7fa5297fa640 1 --2- 172.21.15.1:0/2851825477 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa51006b500 0x7fa51006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.924+0000 7fa5297fa640 1 -- 172.21.15.1:0/2851825477 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa5200ab0a0 con 0x7fa51c0039b0 2024-03-06T07:21:49.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.925+0000 7fa52bfff640 1 --2- 172.21.15.1:0/2851825477 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa51006b500 0x7fa51006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.925+0000 7fa52bfff640 1 --2- 172.21.15.1:0/2851825477 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa51006b500 0x7fa51006d9c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa52c14e2c0 tx=0x7fa518000bc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:49.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:49.928+0000 7fa5297fa640 1 -- 172.21.15.1:0/2851825477 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa520079dd0 con 0x7fa51c0039b0 2024-03-06T07:21:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.108+0000 7fa532fb5640 1 -- 172.21.15.1:0/2851825477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa52c0ac750 con 0x7fa51c0039b0 2024-03-06T07:21:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.108+0000 7fa5297fa640 1 -- 172.21.15.1:0/2851825477 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa5200796d0 con 0x7fa51c0039b0 2024-03-06T07:21:50.110 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:21:50.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.110+0000 7fa532fb5640 1 -- 172.21.15.1:0/2851825477 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa51006b500 msgr2=0x7fa51006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.110+0000 7fa532fb5640 1 --2- 172.21.15.1:0/2851825477 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa51006b500 0x7fa51006d9c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa52c14e2c0 tx=0x7fa518000bc0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.110+0000 7fa532fb5640 1 -- 172.21.15.1:0/2851825477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa51c0039b0 msgr2=0x7fa52c0c6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.110+0000 7fa532fb5640 1 --2- 172.21.15.1:0/2851825477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa51c0039b0 0x7fa52c0c6260 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fa520009be0 tx=0x7fa520004730 comp rx=0 tx=0).stop 2024-03-06T07:21:50.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.110+0000 7fa532fb5640 1 -- 172.21.15.1:0/2851825477 shutdown_connections 2024-03-06T07:21:50.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.110+0000 7fa532fb5640 1 --2- 172.21.15.1:0/2851825477 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa51006b500 0x7fa51006d9c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.110+0000 7fa532fb5640 1 --2- 172.21.15.1:0/2851825477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa52c1eb2b0 0x7fa52c1edc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.110+0000 7fa532fb5640 1 --2- 172.21.15.1:0/2851825477 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa52c154b20 0x7fa52c14d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.110+0000 7fa532fb5640 1 --2- 172.21.15.1:0/2851825477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa51c0039b0 0x7fa52c0c6260 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.110+0000 7fa532fb5640 1 -- 172.21.15.1:0/2851825477 >> 172.21.15.1:0/2851825477 conn(0x7fa52c0c3cc0 msgr2=0x7fa52c14a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.110+0000 7fa532fb5640 1 -- 172.21.15.1:0/2851825477 shutdown_connections 2024-03-06T07:21:50.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.110+0000 7fa532fb5640 1 -- 172.21.15.1:0/2851825477 wait complete. 2024-03-06T07:21:50.123 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:21:50.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.226+0000 7f8c4ca34640 1 Processor -- start 2024-03-06T07:21:50.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.226+0000 7f8c4ca34640 1 -- start start 2024-03-06T07:21:50.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.227+0000 7f8c4ca34640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8c48152840 0x7f8c480c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.227+0000 7f8c4ca34640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c480c60b0 0x7f8c481553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.227+0000 7f8c4ca34640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c480c5e40 con 0x7f8c480c3970 2024-03-06T07:21:50.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.227+0000 7f8c4ca34640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c480c67f0 con 0x7f8c48152840 2024-03-06T07:21:50.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.227+0000 7f8c4ca34640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c480c6a40 con 0x7f8c480c60b0 2024-03-06T07:21:50.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.227+0000 7f8c4659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8c480c3970 0x7f8c48152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55514/0 (socket says 172.21.15.1:55514) 2024-03-06T07:21:50.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.227+0000 7f8c4659c640 1 -- 172.21.15.1:0/3755041934 learned_addr learned my addr 172.21.15.1:0/3755041934 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:50.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.227+0000 7f8c45d9b640 1 --2- 172.21.15.1:0/3755041934 >> v2:172.21.15.98:3300/0 conn(0x7f8c48152840 0x7f8c480c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.227+0000 7f8c46d9d640 1 --2- 172.21.15.1:0/3755041934 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c480c60b0 0x7f8c481553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.227+0000 7f8c4559a640 1 -- 172.21.15.1:0/3755041934 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072472602 0 0) 0x7f8c480c5e40 con 0x7f8c480c3970 2024-03-06T07:21:50.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.227+0000 7f8c4559a640 1 -- 172.21.15.1:0/3755041934 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c30003520 con 0x7f8c480c3970 2024-03-06T07:21:50.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.228+0000 7f8c45d9b640 1 -- 172.21.15.1:0/3755041934 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c480c60b0 msgr2=0x7f8c481553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:50.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.228+0000 7f8c45d9b640 1 --2- 172.21.15.1:0/3755041934 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c480c60b0 0x7f8c481553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.228+0000 7f8c45d9b640 1 -- 172.21.15.1:0/3755041934 >> v1:172.21.15.1:6789/0 conn(0x7f8c480c3970 legacy=0x7f8c48152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.228+0000 7f8c45d9b640 1 -- 172.21.15.1:0/3755041934 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c480c6d20 con 0x7f8c48152840 2024-03-06T07:21:50.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.228+0000 7f8c45d9b640 1 --2- 172.21.15.1:0/3755041934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c48152840 0x7f8c480c8170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c0097a0 tx=0x7f8c3c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=436c01a9c2138340 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.230+0000 7f8c4559a640 1 -- 172.21.15.1:0/3755041934 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c035020 con 0x7f8c48152840 2024-03-06T07:21:50.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.230+0000 7f8c4559a640 1 -- 172.21.15.1:0/3755041934 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c3c031830 con 0x7f8c48152840 2024-03-06T07:21:50.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.230+0000 7f8c4559a640 1 -- 172.21.15.1:0/3755041934 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c031c00 con 0x7f8c48152840 2024-03-06T07:21:50.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.231+0000 7f8c4ca34640 1 -- 172.21.15.1:0/3755041934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c48152840 msgr2=0x7f8c480c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.231+0000 7f8c4ca34640 1 --2- 172.21.15.1:0/3755041934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c48152840 0x7f8c480c8170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c0097a0 tx=0x7f8c3c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.231+0000 7f8c4ca34640 1 -- 172.21.15.1:0/3755041934 shutdown_connections 2024-03-06T07:21:50.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.231+0000 7f8c4ca34640 1 --2- 172.21.15.1:0/3755041934 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c480c60b0 0x7f8c481553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.231+0000 7f8c4ca34640 1 --2- 172.21.15.1:0/3755041934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c48152840 0x7f8c480c8170 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.231+0000 7f8c4ca34640 1 -- 172.21.15.1:0/3755041934 >> 172.21.15.1:0/3755041934 conn(0x7f8c480c88f0 msgr2=0x7f8c480c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.231+0000 7f8c4ca34640 1 -- 172.21.15.1:0/3755041934 shutdown_connections 2024-03-06T07:21:50.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.231+0000 7f8c4ca34640 1 -- 172.21.15.1:0/3755041934 wait complete. 2024-03-06T07:21:50.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.232+0000 7f8c4ca34640 1 Processor -- start 2024-03-06T07:21:50.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.232+0000 7f8c4ca34640 1 -- start start 2024-03-06T07:21:50.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c4ca34640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c480c3970 0x7f8c4814d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c4ca34640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c480c60b0 0x7f8c480a8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c4ca34640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c48152840 0x7f8c481491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c4ca34640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c480c75c0 con 0x7f8c480c3970 2024-03-06T07:21:50.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c4ca34640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c480c72e0 con 0x7f8c48152840 2024-03-06T07:21:50.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c4ca34640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c480c3d50 con 0x7f8c480c60b0 2024-03-06T07:21:50.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c4659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c480c3970 0x7f8c4814d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c4659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c480c3970 0x7f8c4814d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33014/0 (socket says 172.21.15.1:33014) 2024-03-06T07:21:50.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c4659c640 1 -- 172.21.15.1:0/277271768 learned_addr learned my addr 172.21.15.1:0/277271768 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:50.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c45d9b640 1 --2- 172.21.15.1:0/277271768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c480c60b0 0x7f8c480a8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c46d9d640 1 --2- 172.21.15.1:0/277271768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c48152840 0x7f8c481491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c4659c640 1 -- 172.21.15.1:0/277271768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c480c60b0 msgr2=0x7f8c480a8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c4659c640 1 --2- 172.21.15.1:0/277271768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c480c60b0 0x7f8c480a8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c4659c640 1 -- 172.21.15.1:0/277271768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c48152840 msgr2=0x7f8c481491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c4659c640 1 --2- 172.21.15.1:0/277271768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c48152840 0x7f8c481491a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c4659c640 1 -- 172.21.15.1:0/277271768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c481ee150 con 0x7f8c480c3970 2024-03-06T07:21:50.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c4659c640 1 --2- 172.21.15.1:0/277271768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c480c3970 0x7f8c4814d7d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800c460 tx=0x7f8c3800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.233+0000 7f8c36ffd640 1 -- 172.21.15.1:0/277271768 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c38017070 con 0x7f8c480c3970 2024-03-06T07:21:50.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.234+0000 7f8c36ffd640 1 -- 172.21.15.1:0/277271768 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c38005ce0 con 0x7f8c480c3970 2024-03-06T07:21:50.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.234+0000 7f8c4ca34640 1 -- 172.21.15.1:0/277271768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c481ee3f0 con 0x7f8c480c3970 2024-03-06T07:21:50.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.234+0000 7f8c36ffd640 1 -- 172.21.15.1:0/277271768 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c38007290 con 0x7f8c480c3970 2024-03-06T07:21:50.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.234+0000 7f8c4ca34640 1 -- 172.21.15.1:0/277271768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c481ee9b0 con 0x7f8c480c3970 2024-03-06T07:21:50.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.234+0000 7f8c4ca34640 1 -- 172.21.15.1:0/277271768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c480686e0 con 0x7f8c480c3970 2024-03-06T07:21:50.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.235+0000 7f8c36ffd640 1 -- 172.21.15.1:0/277271768 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c3801a070 con 0x7f8c480c3970 2024-03-06T07:21:50.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.235+0000 7f8c36ffd640 1 --2- 172.21.15.1:0/277271768 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c2806b500 0x7f8c2806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.236+0000 7f8c36ffd640 1 -- 172.21.15.1:0/277271768 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8c380895d0 con 0x7f8c480c3970 2024-03-06T07:21:50.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.236+0000 7f8c45d9b640 1 --2- 172.21.15.1:0/277271768 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c2806b500 0x7f8c2806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.236+0000 7f8c45d9b640 1 --2- 172.21.15.1:0/277271768 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c2806b500 0x7f8c2806d9c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c036000 tx=0x7f8c3c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.244+0000 7f8c36ffd640 1 -- 172.21.15.1:0/277271768 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c38057cb0 con 0x7f8c480c3970 2024-03-06T07:21:50.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.423+0000 7f8c4ca34640 1 -- 172.21.15.1:0/277271768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8c481519a0 con 0x7f8c480c3970 2024-03-06T07:21:50.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.424+0000 7f8c36ffd640 1 -- 172.21.15.1:0/277271768 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v40) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8c3805bd50 con 0x7f8c480c3970 2024-03-06T07:21:50.426 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:21:50.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.425+0000 7f8c4ca34640 1 -- 172.21.15.1:0/277271768 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c2806b500 msgr2=0x7f8c2806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.425+0000 7f8c4ca34640 1 --2- 172.21.15.1:0/277271768 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c2806b500 0x7f8c2806d9c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c036000 tx=0x7f8c3c03c040 comp rx=0 tx=0).stop 2024-03-06T07:21:50.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.425+0000 7f8c4ca34640 1 -- 172.21.15.1:0/277271768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c480c3970 msgr2=0x7f8c4814d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.425+0000 7f8c4ca34640 1 --2- 172.21.15.1:0/277271768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c480c3970 0x7f8c4814d7d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800c460 tx=0x7f8c3800c920 comp rx=0 tx=0).stop 2024-03-06T07:21:50.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.425+0000 7f8c4ca34640 1 -- 172.21.15.1:0/277271768 shutdown_connections 2024-03-06T07:21:50.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.426+0000 7f8c4ca34640 1 --2- 172.21.15.1:0/277271768 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c2806b500 0x7f8c2806d9c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.426+0000 7f8c4ca34640 1 --2- 172.21.15.1:0/277271768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c48152840 0x7f8c481491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.426+0000 7f8c4ca34640 1 --2- 172.21.15.1:0/277271768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c480c60b0 0x7f8c480a8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.426+0000 7f8c4ca34640 1 --2- 172.21.15.1:0/277271768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c480c3970 0x7f8c4814d7d0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.426+0000 7f8c4ca34640 1 -- 172.21.15.1:0/277271768 >> 172.21.15.1:0/277271768 conn(0x7f8c480c88f0 msgr2=0x7f8c48158c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.426+0000 7f8c4ca34640 1 -- 172.21.15.1:0/277271768 shutdown_connections 2024-03-06T07:21:50.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.426+0000 7f8c4ca34640 1 -- 172.21.15.1:0/277271768 wait complete. 2024-03-06T07:21:50.438 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:21:50.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.541+0000 7fca289f6640 1 Processor -- start 2024-03-06T07:21:50.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.541+0000 7fca289f6640 1 -- start start 2024-03-06T07:21:50.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca289f6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fca24152820 0x7fca240c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca289f6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fca240c0670 0x7fca240c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca289f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca240c7930 0x7fca24158830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca289f6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca240c7180 con 0x7fca24152820 2024-03-06T07:21:50.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca289f6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca24155460 con 0x7fca240c0670 2024-03-06T07:21:50.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca289f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca240c6bc0 con 0x7fca240c7930 2024-03-06T07:21:50.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca2259c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fca24152820 0x7fca240c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca2259c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fca24152820 0x7fca240c0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33030/0 (socket says 172.21.15.1:33030) 2024-03-06T07:21:50.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca2259c640 1 -- 172.21.15.1:0/2830586741 learned_addr learned my addr 172.21.15.1:0/2830586741 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:50.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca1bfff640 1 --2- 172.21.15.1:0/2830586741 >> v2:172.21.15.98:3300/0 conn(0x7fca240c0670 0x7fca240c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca22d9d640 1 --2- 172.21.15.1:0/2830586741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca240c7930 0x7fca24158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca2259c640 1 -- 172.21.15.1:0/2830586741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca240c7930 msgr2=0x7fca24158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:50.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca2259c640 1 --2- 172.21.15.1:0/2830586741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca240c7930 0x7fca24158830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca2259c640 1 -- 172.21.15.1:0/2830586741 >> v2:172.21.15.98:3300/0 conn(0x7fca240c0670 msgr2=0x7fca240c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca2259c640 1 --2- 172.21.15.1:0/2830586741 >> v2:172.21.15.98:3300/0 conn(0x7fca240c0670 0x7fca240c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca2259c640 1 -- 172.21.15.1:0/2830586741 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca240c6ea0 con 0x7fca24152820 2024-03-06T07:21:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca1bfff640 1 --2- 172.21.15.1:0/2830586741 >> v2:172.21.15.98:3300/0 conn(0x7fca240c0670 0x7fca240c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.542+0000 7fca22d9d640 1 --2- 172.21.15.1:0/2830586741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca240c7930 0x7fca24158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.543+0000 7fca2259c640 1 --2- 172.21.15.1:0/2830586741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca24152820 0x7fca240c0130 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fca2406cca0 tx=0x7fca1402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88aaa669547e2d14 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.543+0000 7fca21d9b640 1 -- 172.21.15.1:0/2830586741 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca14035020 con 0x7fca24152820 2024-03-06T07:21:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.543+0000 7fca21d9b640 1 -- 172.21.15.1:0/2830586741 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1402d040 con 0x7fca24152820 2024-03-06T07:21:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.543+0000 7fca21d9b640 1 -- 172.21.15.1:0/2830586741 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca14031a80 con 0x7fca24152820 2024-03-06T07:21:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.544+0000 7fca289f6640 1 -- 172.21.15.1:0/2830586741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca24152820 msgr2=0x7fca240c0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.544+0000 7fca289f6640 1 --2- 172.21.15.1:0/2830586741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca24152820 0x7fca240c0130 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fca2406cca0 tx=0x7fca1402b900 comp rx=0 tx=0).stop 2024-03-06T07:21:50.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.544+0000 7fca289f6640 1 -- 172.21.15.1:0/2830586741 shutdown_connections 2024-03-06T07:21:50.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.544+0000 7fca289f6640 1 --2- 172.21.15.1:0/2830586741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca240c7930 0x7fca24158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.544+0000 7fca289f6640 1 --2- 172.21.15.1:0/2830586741 >> v2:172.21.15.98:3300/0 conn(0x7fca240c0670 0x7fca240c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.544+0000 7fca289f6640 1 --2- 172.21.15.1:0/2830586741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca24152820 0x7fca240c0130 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.544+0000 7fca289f6640 1 -- 172.21.15.1:0/2830586741 >> 172.21.15.1:0/2830586741 conn(0x7fca24072720 msgr2=0x7fca24072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.544+0000 7fca289f6640 1 -- 172.21.15.1:0/2830586741 shutdown_connections 2024-03-06T07:21:50.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.544+0000 7fca289f6640 1 -- 172.21.15.1:0/2830586741 wait complete. 2024-03-06T07:21:50.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.546+0000 7fca289f6640 1 Processor -- start 2024-03-06T07:21:50.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.546+0000 7fca289f6640 1 -- start start 2024-03-06T07:21:50.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.547+0000 7fca289f6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca240c0670 0x7fca241eea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.547+0000 7fca289f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca240c7930 0x7fca241eb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.547+0000 7fca289f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca24152820 0x7fca24154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.547+0000 7fca289f6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca24152d80 con 0x7fca240c0670 2024-03-06T07:21:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.547+0000 7fca289f6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca24152c00 con 0x7fca24152820 2024-03-06T07:21:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.547+0000 7fca289f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca240af6f0 con 0x7fca240c7930 2024-03-06T07:21:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.547+0000 7fca2259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca240c0670 0x7fca241eea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.547+0000 7fca2259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca240c0670 0x7fca241eea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33044/0 (socket says 172.21.15.1:33044) 2024-03-06T07:21:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.547+0000 7fca2259c640 1 -- 172.21.15.1:0/3031973723 learned_addr learned my addr 172.21.15.1:0/3031973723 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.547+0000 7fca1bfff640 1 --2- 172.21.15.1:0/3031973723 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca240c7930 0x7fca241eb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.547+0000 7fca22d9d640 1 --2- 172.21.15.1:0/3031973723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca24152820 0x7fca24154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.547+0000 7fca2259c640 1 -- 172.21.15.1:0/3031973723 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca240c7930 msgr2=0x7fca241eb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.547+0000 7fca2259c640 1 --2- 172.21.15.1:0/3031973723 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca240c7930 0x7fca241eb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.547+0000 7fca2259c640 1 -- 172.21.15.1:0/3031973723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca24152820 msgr2=0x7fca24154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.547+0000 7fca2259c640 1 --2- 172.21.15.1:0/3031973723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca24152820 0x7fca24154520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.547+0000 7fca2259c640 1 -- 172.21.15.1:0/3031973723 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca241566b0 con 0x7fca240c0670 2024-03-06T07:21:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.548+0000 7fca2259c640 1 --2- 172.21.15.1:0/3031973723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca240c0670 0x7fca241eea30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fca1402b940 tx=0x7fca140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.548+0000 7fca1affd640 1 -- 172.21.15.1:0/3031973723 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca14040070 con 0x7fca240c0670 2024-03-06T07:21:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.548+0000 7fca1affd640 1 -- 172.21.15.1:0/3031973723 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca14039c30 con 0x7fca240c0670 2024-03-06T07:21:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.548+0000 7fca289f6640 1 -- 172.21.15.1:0/3031973723 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca241edf50 con 0x7fca240c0670 2024-03-06T07:21:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.548+0000 7fca1affd640 1 -- 172.21.15.1:0/3031973723 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca14035040 con 0x7fca240c0670 2024-03-06T07:21:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.548+0000 7fca289f6640 1 -- 172.21.15.1:0/3031973723 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca24154a60 con 0x7fca240c0670 2024-03-06T07:21:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.549+0000 7fca289f6640 1 -- 172.21.15.1:0/3031973723 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca240686e0 con 0x7fca240c0670 2024-03-06T07:21:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.549+0000 7fca1affd640 1 -- 172.21.15.1:0/3031973723 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca1404b060 con 0x7fca240c0670 2024-03-06T07:21:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.550+0000 7fca1affd640 1 --2- 172.21.15.1:0/3031973723 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fca0406b500 0x7fca0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.550+0000 7fca1affd640 1 -- 172.21.15.1:0/3031973723 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fca140abf50 con 0x7fca240c0670 2024-03-06T07:21:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.550+0000 7fca1bfff640 1 --2- 172.21.15.1:0/3031973723 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fca0406b500 0x7fca0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.550+0000 7fca1bfff640 1 --2- 172.21.15.1:0/3031973723 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fca0406b500 0x7fca0406d9c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fca24151ed0 tx=0x7fca0c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.554+0000 7fca1affd640 1 -- 172.21.15.1:0/3031973723 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca1407a630 con 0x7fca240c0670 2024-03-06T07:21:50.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.736+0000 7fca289f6640 1 -- 172.21.15.1:0/3031973723 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fca240c5b40 con 0x7fca240c0670 2024-03-06T07:21:50.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.736+0000 7fca1affd640 1 -- 172.21.15.1:0/3031973723 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v40) v1 ==== 100+0+11 (secure 0 0 0) 0x7fca1407e6d0 con 0x7fca240c0670 2024-03-06T07:21:50.738 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:21:50.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.738+0000 7fca289f6640 1 -- 172.21.15.1:0/3031973723 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fca0406b500 msgr2=0x7fca0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.738+0000 7fca289f6640 1 --2- 172.21.15.1:0/3031973723 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fca0406b500 0x7fca0406d9c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fca24151ed0 tx=0x7fca0c008040 comp rx=0 tx=0).stop 2024-03-06T07:21:50.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.738+0000 7fca289f6640 1 -- 172.21.15.1:0/3031973723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca240c0670 msgr2=0x7fca241eea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.738+0000 7fca289f6640 1 --2- 172.21.15.1:0/3031973723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca240c0670 0x7fca241eea30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fca1402b940 tx=0x7fca140046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.738+0000 7fca289f6640 1 -- 172.21.15.1:0/3031973723 shutdown_connections 2024-03-06T07:21:50.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.738+0000 7fca289f6640 1 --2- 172.21.15.1:0/3031973723 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fca0406b500 0x7fca0406d9c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.738+0000 7fca289f6640 1 --2- 172.21.15.1:0/3031973723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca24152820 0x7fca24154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.738+0000 7fca289f6640 1 --2- 172.21.15.1:0/3031973723 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca240c7930 0x7fca241eb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.738+0000 7fca289f6640 1 --2- 172.21.15.1:0/3031973723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca240c0670 0x7fca241eea30 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.738+0000 7fca289f6640 1 -- 172.21.15.1:0/3031973723 >> 172.21.15.1:0/3031973723 conn(0x7fca24072720 msgr2=0x7fca2406cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.738+0000 7fca289f6640 1 -- 172.21.15.1:0/3031973723 shutdown_connections 2024-03-06T07:21:50.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.738+0000 7fca289f6640 1 -- 172.21.15.1:0/3031973723 wait complete. 2024-03-06T07:21:50.751 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:50.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.851+0000 7f4cc0bb5640 1 Processor -- start 2024-03-06T07:21:50.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.851+0000 7f4cc0bb5640 1 -- start start 2024-03-06T07:21:50.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.852+0000 7f4cc0bb5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4cbc152650 0x7f4cbc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.852+0000 7f4cc0bb5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4cbc0c6b10 0x7f4cbc158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.852+0000 7f4cc0bb5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cbc0c2fc0 con 0x7f4cbc0bf9d0 2024-03-06T07:21:50.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.852+0000 7f4cc0bb5640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cbc155800 con 0x7f4cbc0c6b10 2024-03-06T07:21:50.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.852+0000 7f4cc0bb5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cbc0c6690 con 0x7f4cbc152650 2024-03-06T07:21:50.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.852+0000 7f4cb3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4cbc0c6b10 0x7f4cbc158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.852+0000 7f4cbad9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4cbc0bf9d0 0x7f4cbc0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55518/0 (socket says 172.21.15.1:55518) 2024-03-06T07:21:50.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.852+0000 7f4cbad9d640 1 -- 172.21.15.1:0/464175848 learned_addr learned my addr 172.21.15.1:0/464175848 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:50.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.852+0000 7f4cba59c640 1 --2- 172.21.15.1:0/464175848 >> v2:172.21.15.98:3300/0 conn(0x7f4cbc152650 0x7f4cbc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.853+0000 7f4cb3fff640 1 -- 172.21.15.1:0/464175848 >> v2:172.21.15.98:3300/0 conn(0x7f4cbc152650 msgr2=0x7f4cbc155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:50.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.853+0000 7f4cb3fff640 1 --2- 172.21.15.1:0/464175848 >> v2:172.21.15.98:3300/0 conn(0x7f4cbc152650 0x7f4cbc155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.853+0000 7f4cb3fff640 1 -- 172.21.15.1:0/464175848 >> v1:172.21.15.1:6789/0 conn(0x7f4cbc0bf9d0 legacy=0x7f4cbc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.853+0000 7f4cb9d9b640 1 -- 172.21.15.1:0/464175848 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579568824 0 0) 0x7f4cbc0c2fc0 con 0x7f4cbc0bf9d0 2024-03-06T07:21:50.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.853+0000 7f4cb3fff640 1 -- 172.21.15.1:0/464175848 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cbc0c7280 con 0x7f4cbc0c6b10 2024-03-06T07:21:50.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.853+0000 7f4cb3fff640 1 --2- 172.21.15.1:0/464175848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cbc0c6b10 0x7f4cbc158690 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0cac90 tx=0x7f4ca402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=402a78ce48199df6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.854+0000 7f4cb9d9b640 1 -- 172.21.15.1:0/464175848 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca40315a0 con 0x7f4cbc0c6b10 2024-03-06T07:21:50.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.854+0000 7f4cb9d9b640 1 -- 172.21.15.1:0/464175848 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca4037070 con 0x7f4cbc0c6b10 2024-03-06T07:21:50.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.854+0000 7f4cb9d9b640 1 -- 172.21.15.1:0/464175848 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4031960 con 0x7f4cbc0c6b10 2024-03-06T07:21:50.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.854+0000 7f4cc0bb5640 1 -- 172.21.15.1:0/464175848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cbc0c6b10 msgr2=0x7f4cbc158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.854+0000 7f4cc0bb5640 1 --2- 172.21.15.1:0/464175848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cbc0c6b10 0x7f4cbc158690 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0cac90 tx=0x7f4ca402b840 comp rx=0 tx=0).stop 2024-03-06T07:21:50.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.854+0000 7f4cc0bb5640 1 -- 172.21.15.1:0/464175848 shutdown_connections 2024-03-06T07:21:50.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.854+0000 7f4cc0bb5640 1 --2- 172.21.15.1:0/464175848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cbc0c6b10 0x7f4cbc158690 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.854+0000 7f4cc0bb5640 1 --2- 172.21.15.1:0/464175848 >> v2:172.21.15.98:3300/0 conn(0x7f4cbc152650 0x7f4cbc155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.854+0000 7f4cc0bb5640 1 -- 172.21.15.1:0/464175848 >> 172.21.15.1:0/464175848 conn(0x7f4cbc066ee0 msgr2=0x7f4cbc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.854+0000 7f4cc0bb5640 1 -- 172.21.15.1:0/464175848 shutdown_connections 2024-03-06T07:21:50.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.854+0000 7f4cc0bb5640 1 -- 172.21.15.1:0/464175848 wait complete. 2024-03-06T07:21:50.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.855+0000 7f4cc0bb5640 1 Processor -- start 2024-03-06T07:21:50.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.855+0000 7f4cc0bb5640 1 -- start start 2024-03-06T07:21:50.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.856+0000 7f4cc0bb5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cbc0bf9d0 0x7f4cbc158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.856+0000 7f4cc0bb5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cbc152650 0x7f4cbc154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.856+0000 7f4cc0bb5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cbc1443c0 0x7f4cbc144850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.856+0000 7f4cc0bb5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc155490 con 0x7f4cbc1443c0 2024-03-06T07:21:50.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.856+0000 7f4cc0bb5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc1551b0 con 0x7f4cbc0bf9d0 2024-03-06T07:21:50.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.856+0000 7f4cc0bb5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc152a30 con 0x7f4cbc152650 2024-03-06T07:21:50.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.856+0000 7f4cbad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cbc1443c0 0x7f4cbc144850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.856+0000 7f4cbad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cbc1443c0 0x7f4cbc144850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33064/0 (socket says 172.21.15.1:33064) 2024-03-06T07:21:50.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.856+0000 7f4cbad9d640 1 -- 172.21.15.1:0/3184804828 learned_addr learned my addr 172.21.15.1:0/3184804828 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:50.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.856+0000 7f4cba59c640 1 --2- 172.21.15.1:0/3184804828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cbc0bf9d0 0x7f4cbc158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.856+0000 7f4cb3fff640 1 --2- 172.21.15.1:0/3184804828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cbc152650 0x7f4cbc154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.857+0000 7f4cbad9d640 1 -- 172.21.15.1:0/3184804828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cbc152650 msgr2=0x7f4cbc154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.857+0000 7f4cbad9d640 1 --2- 172.21.15.1:0/3184804828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cbc152650 0x7f4cbc154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.857+0000 7f4cbad9d640 1 -- 172.21.15.1:0/3184804828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cbc0bf9d0 msgr2=0x7f4cbc158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.857+0000 7f4cbad9d640 1 --2- 172.21.15.1:0/3184804828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cbc0bf9d0 0x7f4cbc158c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.857+0000 7f4cbad9d640 1 -- 172.21.15.1:0/3184804828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cbc14edc0 con 0x7f4cbc1443c0 2024-03-06T07:21:50.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.857+0000 7f4cb3fff640 1 --2- 172.21.15.1:0/3184804828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cbc152650 0x7f4cbc154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:50.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.857+0000 7f4cba59c640 1 --2- 172.21.15.1:0/3184804828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cbc0bf9d0 0x7f4cbc158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:50.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.857+0000 7f4cbad9d640 1 --2- 172.21.15.1:0/3184804828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cbc1443c0 0x7f4cbc144850 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400c700 tx=0x7f4cb400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.857+0000 7f4cb2ffd640 1 -- 172.21.15.1:0/3184804828 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4018070 con 0x7f4cbc1443c0 2024-03-06T07:21:50.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.857+0000 7f4cb2ffd640 1 -- 172.21.15.1:0/3184804828 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb400ce10 con 0x7f4cbc1443c0 2024-03-06T07:21:50.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.857+0000 7f4cc0bb5640 1 -- 172.21.15.1:0/3184804828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cbc1eeaf0 con 0x7f4cbc1443c0 2024-03-06T07:21:50.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.857+0000 7f4cb2ffd640 1 -- 172.21.15.1:0/3184804828 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb400a880 con 0x7f4cbc1443c0 2024-03-06T07:21:50.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.857+0000 7f4cc0bb5640 1 -- 172.21.15.1:0/3184804828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cbc1eedd0 con 0x7f4cbc1443c0 2024-03-06T07:21:50.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.858+0000 7f4cc0bb5640 1 -- 172.21.15.1:0/3184804828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cbc067fd0 con 0x7f4cbc1443c0 2024-03-06T07:21:50.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.858+0000 7f4cb2ffd640 1 -- 172.21.15.1:0/3184804828 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4cb401b070 con 0x7f4cbc1443c0 2024-03-06T07:21:50.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.859+0000 7f4cb2ffd640 1 --2- 172.21.15.1:0/3184804828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c9c06b500 0x7f4c9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.859+0000 7f4cb2ffd640 1 -- 172.21.15.1:0/3184804828 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f4cb40898d0 con 0x7f4cbc1443c0 2024-03-06T07:21:50.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.859+0000 7f4cba59c640 1 --2- 172.21.15.1:0/3184804828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c9c06b500 0x7f4c9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.860+0000 7f4cba59c640 1 --2- 172.21.15.1:0/3184804828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c9c06b500 0x7f4c9c06d9c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0b0430 tx=0x7f4cac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:50.863+0000 7f4cb2ffd640 1 -- 172.21.15.1:0/3184804828 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4cb4058600 con 0x7f4cbc1443c0 2024-03-06T07:21:51.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.107+0000 7f4cc0bb5640 1 -- 172.21.15.1:0/3184804828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cbc0b06f0 con 0x7f4cbc1443c0 2024-03-06T07:21:51.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.108+0000 7f4cb2ffd640 1 -- 172.21.15.1:0/3184804828 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f4cb4057f00 con 0x7f4cbc1443c0 2024-03-06T07:21:51.110 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:51.110 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3354292028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3354292028},{"type":"v1","addr":"172.21.15.98:6836","nonce":3354292028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4975,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3948803645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3948803645},{"type":"v1","addr":"172.21.15.98:6837","nonce":3948803645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5538,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3390157192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3390157192},{"type":"v1","addr":"172.21.15.1:6834","nonce":3390157192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:22.528539+0000","modified":"2024-03-06T07:21:43.207337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":10,"addr":"172.21.15.1:6835/2438105345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2438105345},{"type":"v1","addr":"172.21.15.1:6835","nonce":2438105345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4781,"qdb_cluster":[4781]},"id":2}]} 2024-03-06T07:21:51.110 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 22 2024-03-06T07:21:51.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.110+0000 7f4cc0bb5640 1 -- 172.21.15.1:0/3184804828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c9c06b500 msgr2=0x7f4c9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.110+0000 7f4cc0bb5640 1 --2- 172.21.15.1:0/3184804828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c9c06b500 0x7f4c9c06d9c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0b0430 tx=0x7f4cac008040 comp rx=0 tx=0).stop 2024-03-06T07:21:51.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.110+0000 7f4cc0bb5640 1 -- 172.21.15.1:0/3184804828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cbc1443c0 msgr2=0x7f4cbc144850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.110+0000 7f4cc0bb5640 1 --2- 172.21.15.1:0/3184804828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cbc1443c0 0x7f4cbc144850 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400c700 tx=0x7f4cb400cbc0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.110+0000 7f4cc0bb5640 1 -- 172.21.15.1:0/3184804828 shutdown_connections 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.110+0000 7f4cc0bb5640 1 --2- 172.21.15.1:0/3184804828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c9c06b500 0x7f4c9c06d9c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.110+0000 7f4cc0bb5640 1 --2- 172.21.15.1:0/3184804828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cbc1443c0 0x7f4cbc144850 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.110+0000 7f4cc0bb5640 1 --2- 172.21.15.1:0/3184804828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cbc152650 0x7f4cbc154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.110+0000 7f4cc0bb5640 1 --2- 172.21.15.1:0/3184804828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cbc0bf9d0 0x7f4cbc158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.110+0000 7f4cc0bb5640 1 -- 172.21.15.1:0/3184804828 >> 172.21.15.1:0/3184804828 conn(0x7f4cbc066ee0 msgr2=0x7f4cbc1506a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.110+0000 7f4cc0bb5640 1 -- 172.21.15.1:0/3184804828 shutdown_connections 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.110+0000 7f4cc0bb5640 1 -- 172.21.15.1:0/3184804828 wait complete. 2024-03-06T07:21:51.123 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:51.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.223+0000 7fd6c2e88640 1 Processor -- start 2024-03-06T07:21:51.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.223+0000 7fd6c2e88640 1 -- start start 2024-03-06T07:21:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.224+0000 7fd6c2e88640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd6bc0c6b10 0x7fd6bc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.224+0000 7fd6c2e88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6bc0bf9d0 0x7fd6bc0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.224+0000 7fd6c2e88640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc0c2fc0 con 0x7fd6bc152650 2024-03-06T07:21:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.224+0000 7fd6c2e88640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6bc155800 con 0x7fd6bc0c6b10 2024-03-06T07:21:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.224+0000 7fd6c2e88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc158da0 con 0x7fd6bc0bf9d0 2024-03-06T07:21:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.224+0000 7fd6c0c24640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd6bc152650 0x7fd6bc158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55520/0 (socket says 172.21.15.1:55520) 2024-03-06T07:21:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.224+0000 7fd6c0c24640 1 -- 172.21.15.1:0/3902829133 learned_addr learned my addr 172.21.15.1:0/3902829133 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.224+0000 7fd6c1425640 1 --2- 172.21.15.1:0/3902829133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6bc0bf9d0 0x7fd6bc0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.224+0000 7fd6bbfff640 1 --2- 172.21.15.1:0/3902829133 >> v2:172.21.15.98:3300/0 conn(0x7fd6bc0c6b10 0x7fd6bc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.224+0000 7fd6bb7fe640 1 -- 172.21.15.1:0/3902829133 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548546842 0 0) 0x7fd6bc0c2fc0 con 0x7fd6bc152650 2024-03-06T07:21:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.224+0000 7fd6bb7fe640 1 -- 172.21.15.1:0/3902829133 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6a8003520 con 0x7fd6bc152650 2024-03-06T07:21:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bb7fe640 1 -- 172.21.15.1:0/3902829133 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 840531911 0 0) 0x7fd6a8003520 con 0x7fd6bc152650 2024-03-06T07:21:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bb7fe640 1 -- 172.21.15.1:0/3902829133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6bc0bf9d0 msgr2=0x7fd6bc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bb7fe640 1 --2- 172.21.15.1:0/3902829133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6bc0bf9d0 0x7fd6bc0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bb7fe640 1 -- 172.21.15.1:0/3902829133 >> v2:172.21.15.98:3300/0 conn(0x7fd6bc0c6b10 msgr2=0x7fd6bc155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bb7fe640 1 --2- 172.21.15.1:0/3902829133 >> v2:172.21.15.98:3300/0 conn(0x7fd6bc0c6b10 0x7fd6bc155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bb7fe640 1 -- 172.21.15.1:0/3902829133 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc0c6690 con 0x7fd6bc152650 2024-03-06T07:21:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bb7fe640 1 -- 172.21.15.1:0/3902829133 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fd6ac001690 con 0x7fd6bc152650 2024-03-06T07:21:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bbfff640 1 --2- 172.21.15.1:0/3902829133 >> v2:172.21.15.98:3300/0 conn(0x7fd6bc0c6b10 0x7fd6bc155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6c1425640 1 --2- 172.21.15.1:0/3902829133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6bc0bf9d0 0x7fd6bc0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bb7fe640 1 -- 172.21.15.1:0/3902829133 >> v1:172.21.15.1:6789/0 conn(0x7fd6bc152650 legacy=0x7fd6bc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bb7fe640 1 --2- 172.21.15.1:0/3902829133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6a80039b0 0x7fd6a8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bb7fe640 1 --2- 172.21.15.1:0/3902829133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6a8004800 0x7fd6a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bb7fe640 1 --2- 172.21.15.1:0/3902829133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6bc0bf9d0 0x7fd6a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bb7fe640 1 -- 172.21.15.1:0/3902829133 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc155800 con 0x7fd6a8004800 2024-03-06T07:21:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bb7fe640 1 -- 172.21.15.1:0/3902829133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc158da0 con 0x7fd6a80039b0 2024-03-06T07:21:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bb7fe640 1 -- 172.21.15.1:0/3902829133 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a800b9c0 con 0x7fd6bc0bf9d0 2024-03-06T07:21:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bbfff640 1 --2- 172.21.15.1:0/3902829133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6a8004800 0x7fd6a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6c0c24640 1 --2- 172.21.15.1:0/3902829133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6a80039b0 0x7fd6a8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6c1425640 1 --2- 172.21.15.1:0/3902829133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6bc0bf9d0 0x7fd6a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bbfff640 1 -- 172.21.15.1:0/3902829133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6bc0bf9d0 msgr2=0x7fd6a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bbfff640 1 --2- 172.21.15.1:0/3902829133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6bc0bf9d0 0x7fd6a80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bbfff640 1 -- 172.21.15.1:0/3902829133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6a80039b0 msgr2=0x7fd6a8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bbfff640 1 --2- 172.21.15.1:0/3902829133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6a80039b0 0x7fd6a8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.225+0000 7fd6bbfff640 1 -- 172.21.15.1:0/3902829133 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a8003520 con 0x7fd6a8004800 2024-03-06T07:21:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.226+0000 7fd6bbfff640 1 --2- 172.21.15.1:0/3902829133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6a8004800 0x7fd6a8004c80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4009c90 tx=0x7fd6a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=757822ec6e9ade9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.228+0000 7fd6bb7fe640 1 -- 172.21.15.1:0/3902829133 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4039070 con 0x7fd6a8004800 2024-03-06T07:21:51.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.228+0000 7fd6bb7fe640 1 -- 172.21.15.1:0/3902829133 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a402c980 con 0x7fd6a8004800 2024-03-06T07:21:51.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.228+0000 7fd6bb7fe640 1 -- 172.21.15.1:0/3902829133 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4009860 con 0x7fd6a8004800 2024-03-06T07:21:51.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.228+0000 7fd6c2e88640 1 -- 172.21.15.1:0/3902829133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6a8004800 msgr2=0x7fd6a8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.228+0000 7fd6c2e88640 1 --2- 172.21.15.1:0/3902829133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6a8004800 0x7fd6a8004c80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4009c90 tx=0x7fd6a8005830 comp rx=0 tx=0).stop 2024-03-06T07:21:51.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.228+0000 7fd6c1425640 1 -- 172.21.15.1:0/3902829133 reap_dead start 2024-03-06T07:21:51.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.229+0000 7fd6c2e88640 1 -- 172.21.15.1:0/3902829133 shutdown_connections 2024-03-06T07:21:51.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.229+0000 7fd6c2e88640 1 -- 172.21.15.1:0/3902829133 >> 172.21.15.1:0/3902829133 conn(0x7fd6bc066ee0 msgr2=0x7fd6bc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.229+0000 7fd6c2e88640 1 -- 172.21.15.1:0/3902829133 shutdown_connections 2024-03-06T07:21:51.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.229+0000 7fd6c2e88640 1 -- 172.21.15.1:0/3902829133 wait complete. 2024-03-06T07:21:51.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.230+0000 7fd6c2e88640 1 Processor -- start 2024-03-06T07:21:51.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.230+0000 7fd6c2e88640 1 -- start start 2024-03-06T07:21:51.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.231+0000 7fd6c2e88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6a8004800 0x7fd6bc14d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.231+0000 7fd6c2e88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6bc14b4f0 0x7fd6bc14b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.231+0000 7fd6c2e88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6bc14a9e0 0x7fd6bc14ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.231+0000 7fd6c2e88640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc1551b0 con 0x7fd6a8004800 2024-03-06T07:21:51.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.231+0000 7fd6c2e88640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc0c6fa0 con 0x7fd6bc14b4f0 2024-03-06T07:21:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.231+0000 7fd6c2e88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc155490 con 0x7fd6bc14a9e0 2024-03-06T07:21:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.231+0000 7fd6c0c24640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6a8004800 0x7fd6bc14d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.231+0000 7fd6c0c24640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6a8004800 0x7fd6bc14d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33082/0 (socket says 172.21.15.1:33082) 2024-03-06T07:21:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.231+0000 7fd6c0c24640 1 -- 172.21.15.1:0/1004559802 learned_addr learned my addr 172.21.15.1:0/1004559802 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.231+0000 7fd6bbfff640 1 --2- 172.21.15.1:0/1004559802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6bc14b4f0 0x7fd6bc14b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.231+0000 7fd6c1425640 1 --2- 172.21.15.1:0/1004559802 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6bc14a9e0 0x7fd6bc14ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.232+0000 7fd6c0c24640 1 -- 172.21.15.1:0/1004559802 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6bc14a9e0 msgr2=0x7fd6bc14ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.232+0000 7fd6c0c24640 1 --2- 172.21.15.1:0/1004559802 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6bc14a9e0 0x7fd6bc14ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.232+0000 7fd6c0c24640 1 -- 172.21.15.1:0/1004559802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6bc14b4f0 msgr2=0x7fd6bc14b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.232+0000 7fd6c0c24640 1 --2- 172.21.15.1:0/1004559802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6bc14b4f0 0x7fd6bc14b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.232+0000 7fd6c0c24640 1 -- 172.21.15.1:0/1004559802 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc14de10 con 0x7fd6a8004800 2024-03-06T07:21:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.232+0000 7fd6c0c24640 1 --2- 172.21.15.1:0/1004559802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6a8004800 0x7fd6bc14d420 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac007340 tx=0x7fd6ac007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.232+0000 7fd6b97fa640 1 -- 172.21.15.1:0/1004559802 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac00f940 con 0x7fd6a8004800 2024-03-06T07:21:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.232+0000 7fd6b97fa640 1 -- 172.21.15.1:0/1004559802 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ac018070 con 0x7fd6a8004800 2024-03-06T07:21:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.232+0000 7fd6b97fa640 1 -- 172.21.15.1:0/1004559802 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac00fd20 con 0x7fd6a8004800 2024-03-06T07:21:51.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.232+0000 7fd6c2e88640 1 -- 172.21.15.1:0/1004559802 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6bc1eaf60 con 0x7fd6a8004800 2024-03-06T07:21:51.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.232+0000 7fd6c2e88640 1 -- 172.21.15.1:0/1004559802 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6bc1eb4e0 con 0x7fd6a8004800 2024-03-06T07:21:51.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.233+0000 7fd6c2e88640 1 -- 172.21.15.1:0/1004559802 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6bc067fd0 con 0x7fd6a8004800 2024-03-06T07:21:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.233+0000 7fd6b97fa640 1 -- 172.21.15.1:0/1004559802 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6ac01d460 con 0x7fd6a8004800 2024-03-06T07:21:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.234+0000 7fd6b97fa640 1 --2- 172.21.15.1:0/1004559802 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd69c06b500 0x7fd69c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.234+0000 7fd6b97fa640 1 -- 172.21.15.1:0/1004559802 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd6ac08a300 con 0x7fd6a8004800 2024-03-06T07:21:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.234+0000 7fd6bbfff640 1 --2- 172.21.15.1:0/1004559802 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd69c06b500 0x7fd69c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.234+0000 7fd6bbfff640 1 --2- 172.21.15.1:0/1004559802 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd69c06b500 0x7fd69c06d9c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4036000 tx=0x7fd6a4030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.238+0000 7fd6b97fa640 1 -- 172.21.15.1:0/1004559802 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6ac058950 con 0x7fd6a8004800 2024-03-06T07:21:51.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.420+0000 7fd6c2e88640 1 -- 172.21.15.1:0/1004559802 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6bc1569e0 con 0x7fd6a8004800 2024-03-06T07:21:51.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.422+0000 7fd6b97fa640 1 -- 172.21.15.1:0/1004559802 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+16949 (secure 0 0 0) 0x7fd6ac05c9f0 con 0x7fd6a8004800 2024-03-06T07:21:51.424 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:51.425 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":40,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:21:37.195560+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:21.073384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:21.484387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6834/1818141892":"2024-03-07T07:21:37.195540+0000","172.21.15.1:6832/1818141892":"2024-03-07T07:21:37.195540+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:51.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.424+0000 7fd6c2e88640 1 -- 172.21.15.1:0/1004559802 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd69c06b500 msgr2=0x7fd69c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.424+0000 7fd6c2e88640 1 --2- 172.21.15.1:0/1004559802 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd69c06b500 0x7fd69c06d9c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4036000 tx=0x7fd6a4030000 comp rx=0 tx=0).stop 2024-03-06T07:21:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.424+0000 7fd6c2e88640 1 -- 172.21.15.1:0/1004559802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6a8004800 msgr2=0x7fd6bc14d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.424+0000 7fd6c2e88640 1 --2- 172.21.15.1:0/1004559802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6a8004800 0x7fd6bc14d420 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac007340 tx=0x7fd6ac007800 comp rx=0 tx=0).stop 2024-03-06T07:21:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.424+0000 7fd6c2e88640 1 -- 172.21.15.1:0/1004559802 shutdown_connections 2024-03-06T07:21:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.424+0000 7fd6c2e88640 1 --2- 172.21.15.1:0/1004559802 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd69c06b500 0x7fd69c06d9c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.424+0000 7fd6c2e88640 1 --2- 172.21.15.1:0/1004559802 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6bc14a9e0 0x7fd6bc14ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.424+0000 7fd6c2e88640 1 --2- 172.21.15.1:0/1004559802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6bc14b4f0 0x7fd6bc14b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.424+0000 7fd6c2e88640 1 --2- 172.21.15.1:0/1004559802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6a8004800 0x7fd6bc14d420 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.424+0000 7fd6c2e88640 1 -- 172.21.15.1:0/1004559802 >> 172.21.15.1:0/1004559802 conn(0x7fd6bc066ee0 msgr2=0x7fd6bc0c3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.425+0000 7fd6c2e88640 1 -- 172.21.15.1:0/1004559802 shutdown_connections 2024-03-06T07:21:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.425+0000 7fd6c2e88640 1 -- 172.21.15.1:0/1004559802 wait complete. 2024-03-06T07:21:51.441 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:21:51.442 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:51.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.541+0000 7f709ca37640 1 Processor -- start 2024-03-06T07:21:51.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.541+0000 7f709ca37640 1 -- start start 2024-03-06T07:21:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.542+0000 7f709ca37640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7098150640 0x7f709806e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.542+0000 7f709ca37640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70980a8590 0x7f70980c25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.542+0000 7f709ca37640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f709806fb20 0x7f7098152f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.542+0000 7f709ca37640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70980c1fd0 con 0x7f7098150640 2024-03-06T07:21:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.542+0000 7f709ca37640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7098157380 con 0x7f70980a8590 2024-03-06T07:21:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.542+0000 7f709ca37640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70981570a0 con 0x7f709806fb20 2024-03-06T07:21:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.542+0000 7f709659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7098150640 0x7f709806e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.542+0000 7f709659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7098150640 0x7f709806e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33098/0 (socket says 172.21.15.1:33098) 2024-03-06T07:21:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.542+0000 7f709659c640 1 -- 172.21.15.1:0/3852708423 learned_addr learned my addr 172.21.15.1:0/3852708423 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.543+0000 7f7096d9d640 1 --2- 172.21.15.1:0/3852708423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f709806fb20 0x7f7098152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.543+0000 7f7095d9b640 1 --2- 172.21.15.1:0/3852708423 >> v2:172.21.15.98:3300/0 conn(0x7f70980a8590 0x7f70980c25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.543+0000 7f709659c640 1 -- 172.21.15.1:0/3852708423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f709806fb20 msgr2=0x7f7098152f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.543+0000 7f709659c640 1 --2- 172.21.15.1:0/3852708423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f709806fb20 0x7f7098152f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.543+0000 7f709659c640 1 -- 172.21.15.1:0/3852708423 >> v2:172.21.15.98:3300/0 conn(0x7f70980a8590 msgr2=0x7f70980c25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.543+0000 7f709659c640 1 --2- 172.21.15.1:0/3852708423 >> v2:172.21.15.98:3300/0 conn(0x7f70980a8590 0x7f70980c25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.543+0000 7f709659c640 1 -- 172.21.15.1:0/3852708423 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70980c1c60 con 0x7f7098150640 2024-03-06T07:21:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.544+0000 7f709659c640 1 --2- 172.21.15.1:0/3852708423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7098150640 0x7f709806e390 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f70980732f0 tx=0x7f708002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155b7a417c0132fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.544+0000 7f709559a640 1 -- 172.21.15.1:0/3852708423 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7080035020 con 0x7f7098150640 2024-03-06T07:21:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.544+0000 7f709559a640 1 -- 172.21.15.1:0/3852708423 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7080031870 con 0x7f7098150640 2024-03-06T07:21:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.544+0000 7f709559a640 1 -- 172.21.15.1:0/3852708423 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7080031c40 con 0x7f7098150640 2024-03-06T07:21:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.545+0000 7f709ca37640 1 -- 172.21.15.1:0/3852708423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7098150640 msgr2=0x7f709806e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.545+0000 7f709ca37640 1 --2- 172.21.15.1:0/3852708423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7098150640 0x7f709806e390 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f70980732f0 tx=0x7f708002b910 comp rx=0 tx=0).stop 2024-03-06T07:21:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.545+0000 7f709ca37640 1 -- 172.21.15.1:0/3852708423 shutdown_connections 2024-03-06T07:21:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.545+0000 7f709ca37640 1 --2- 172.21.15.1:0/3852708423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f709806fb20 0x7f7098152f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.545+0000 7f709ca37640 1 --2- 172.21.15.1:0/3852708423 >> v2:172.21.15.98:3300/0 conn(0x7f70980a8590 0x7f70980c25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.545+0000 7f709ca37640 1 --2- 172.21.15.1:0/3852708423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7098150640 0x7f709806e390 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.545+0000 7f709ca37640 1 -- 172.21.15.1:0/3852708423 >> 172.21.15.1:0/3852708423 conn(0x7f70980c3cc0 msgr2=0x7f70980c40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.545+0000 7f709ca37640 1 -- 172.21.15.1:0/3852708423 shutdown_connections 2024-03-06T07:21:51.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.545+0000 7f709ca37640 1 -- 172.21.15.1:0/3852708423 wait complete. 2024-03-06T07:21:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.546+0000 7f709ca37640 1 Processor -- start 2024-03-06T07:21:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.546+0000 7f709ca37640 1 -- start start 2024-03-06T07:21:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f709ca37640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709806fb20 0x7f709814d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f709ca37640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70980a8590 0x7f7098152050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f709ca37640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7098150640 0x7f70980bcbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f709ca37640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7098150a20 con 0x7f709806fb20 2024-03-06T07:21:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f709ca37640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70980c2af0 con 0x7f7098150640 2024-03-06T07:21:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f709ca37640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7098150ba0 con 0x7f70980a8590 2024-03-06T07:21:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f709659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709806fb20 0x7f709814d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f709659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709806fb20 0x7f709814d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33112/0 (socket says 172.21.15.1:33112) 2024-03-06T07:21:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f709659c640 1 -- 172.21.15.1:0/2137167377 learned_addr learned my addr 172.21.15.1:0/2137167377 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f7096d9d640 1 --2- 172.21.15.1:0/2137167377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7098150640 0x7f70980bcbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f7095d9b640 1 --2- 172.21.15.1:0/2137167377 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70980a8590 0x7f7098152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f709659c640 1 -- 172.21.15.1:0/2137167377 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70980a8590 msgr2=0x7f7098152050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f709659c640 1 --2- 172.21.15.1:0/2137167377 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70980a8590 0x7f7098152050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f709659c640 1 -- 172.21.15.1:0/2137167377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7098150640 msgr2=0x7f70980bcbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f709659c640 1 --2- 172.21.15.1:0/2137167377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7098150640 0x7f70980bcbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f709659c640 1 -- 172.21.15.1:0/2137167377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709814ce70 con 0x7f709806fb20 2024-03-06T07:21:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f7095d9b640 1 --2- 172.21.15.1:0/2137167377 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70980a8590 0x7f7098152050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f709659c640 1 --2- 172.21.15.1:0/2137167377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709806fb20 0x7f709814d150 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f708002b950 tx=0x7f70800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f708effd640 1 -- 172.21.15.1:0/2137167377 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7080040070 con 0x7f709806fb20 2024-03-06T07:21:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.547+0000 7f708effd640 1 -- 172.21.15.1:0/2137167377 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7080030d00 con 0x7f709806fb20 2024-03-06T07:21:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.548+0000 7f709ca37640 1 -- 172.21.15.1:0/2137167377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70981ebb80 con 0x7f709806fb20 2024-03-06T07:21:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.548+0000 7f708effd640 1 -- 172.21.15.1:0/2137167377 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7080035040 con 0x7f709806fb20 2024-03-06T07:21:51.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.548+0000 7f709ca37640 1 -- 172.21.15.1:0/2137167377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70981ec140 con 0x7f709806fb20 2024-03-06T07:21:51.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.548+0000 7f709ca37640 1 -- 172.21.15.1:0/2137167377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70980686e0 con 0x7f709806fb20 2024-03-06T07:21:51.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.549+0000 7f708effd640 1 -- 172.21.15.1:0/2137167377 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7080038020 con 0x7f709806fb20 2024-03-06T07:21:51.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.549+0000 7f708effd640 1 --2- 172.21.15.1:0/2137167377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f707806b4d0 0x7f707806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.549+0000 7f708effd640 1 -- 172.21.15.1:0/2137167377 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f70800ac150 con 0x7f709806fb20 2024-03-06T07:21:51.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.550+0000 7f7095d9b640 1 --2- 172.21.15.1:0/2137167377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f707806b4d0 0x7f707806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.550+0000 7f7095d9b640 1 --2- 172.21.15.1:0/2137167377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f707806b4d0 0x7f707806d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7088006fd0 tx=0x7f7088008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.554+0000 7f708effd640 1 -- 172.21.15.1:0/2137167377 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f708007ae80 con 0x7f709806fb20 2024-03-06T07:21:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.776+0000 7f709ca37640 1 -- 172.21.15.1:0/2137167377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70981ec380 con 0x7f709806fb20 2024-03-06T07:21:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.777+0000 7f708effd640 1 -- 172.21.15.1:0/2137167377 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f708007a780 con 0x7f709806fb20 2024-03-06T07:21:51.779 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:51.779 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3354292028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3354292028},{"type":"v1","addr":"172.21.15.98:6836","nonce":3354292028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4975,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3948803645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3948803645},{"type":"v1","addr":"172.21.15.98:6837","nonce":3948803645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5538,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3390157192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3390157192},{"type":"v1","addr":"172.21.15.1:6834","nonce":3390157192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:22.528539+0000","modified":"2024-03-06T07:21:43.207337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":10,"addr":"172.21.15.1:6835/2438105345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2438105345},{"type":"v1","addr":"172.21.15.1:6835","nonce":2438105345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4781,"qdb_cluster":[4781]},"id":2}]} 2024-03-06T07:21:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 22 2024-03-06T07:21:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.779+0000 7f709ca37640 1 -- 172.21.15.1:0/2137167377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f707806b4d0 msgr2=0x7f707806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.779+0000 7f709ca37640 1 --2- 172.21.15.1:0/2137167377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f707806b4d0 0x7f707806d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7088006fd0 tx=0x7f7088008040 comp rx=0 tx=0).stop 2024-03-06T07:21:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.779+0000 7f709ca37640 1 -- 172.21.15.1:0/2137167377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709806fb20 msgr2=0x7f709814d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.779+0000 7f709ca37640 1 --2- 172.21.15.1:0/2137167377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709806fb20 0x7f709814d150 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f708002b950 tx=0x7f70800046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.779+0000 7f709ca37640 1 -- 172.21.15.1:0/2137167377 shutdown_connections 2024-03-06T07:21:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.779+0000 7f709ca37640 1 --2- 172.21.15.1:0/2137167377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f707806b4d0 0x7f707806d990 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.779+0000 7f709ca37640 1 --2- 172.21.15.1:0/2137167377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7098150640 0x7f70980bcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.779+0000 7f709ca37640 1 --2- 172.21.15.1:0/2137167377 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70980a8590 0x7f7098152050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.779+0000 7f709ca37640 1 --2- 172.21.15.1:0/2137167377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709806fb20 0x7f709814d150 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.779+0000 7f709ca37640 1 -- 172.21.15.1:0/2137167377 >> 172.21.15.1:0/2137167377 conn(0x7f70980c3cc0 msgr2=0x7f70980a8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.779+0000 7f709ca37640 1 -- 172.21.15.1:0/2137167377 shutdown_connections 2024-03-06T07:21:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.779+0000 7f709ca37640 1 -- 172.21.15.1:0/2137167377 wait complete. 2024-03-06T07:21:51.794 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:51.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.893+0000 7f11a6629640 1 Processor -- start 2024-03-06T07:21:51.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.893+0000 7f11a6629640 1 -- start start 2024-03-06T07:21:51.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.894+0000 7f11a6629640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f11a00c6b10 0x7f11a0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.894+0000 7f11a6629640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11a00bf9d0 0x7f11a00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.894+0000 7f11a6629640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a00c2fc0 con 0x7f11a0152650 2024-03-06T07:21:51.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.894+0000 7f11a6629640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a0155800 con 0x7f11a00c6b10 2024-03-06T07:21:51.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.894+0000 7f11a6629640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a0158da0 con 0x7f11a00bf9d0 2024-03-06T07:21:51.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.894+0000 7f11a4bc6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11a00bf9d0 0x7f11a00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.894+0000 7f119f7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f11a00c6b10 0x7f11a0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.894+0000 7f11a4bc6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11a00bf9d0 0x7f11a00c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:33738/0 (socket says 172.21.15.1:33738) 2024-03-06T07:21:51.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.894+0000 7f11a4bc6640 1 -- 172.21.15.1:0/272559441 learned_addr learned my addr 172.21.15.1:0/272559441 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:51.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.894+0000 7f11a4bc6640 1 -- 172.21.15.1:0/272559441 >> v2:172.21.15.98:3300/0 conn(0x7f11a00c6b10 msgr2=0x7f11a0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.894+0000 7f11a4bc6640 1 --2- 172.21.15.1:0/272559441 >> v2:172.21.15.98:3300/0 conn(0x7f11a00c6b10 0x7f11a0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.894+0000 7f11a4bc6640 1 -- 172.21.15.1:0/272559441 >> v1:172.21.15.98:6789/0 conn(0x7f11a0152650 legacy=0x7f11a0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.894+0000 7f11a4bc6640 1 -- 172.21.15.1:0/272559441 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a00c6690 con 0x7f11a00bf9d0 2024-03-06T07:21:51.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.895+0000 7f11a4bc6640 1 --2- 172.21.15.1:0/272559441 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11a00bf9d0 0x7f11a00c1460 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1198009e50 tx=0x7f119802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36a7bcb2220c5cd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.895+0000 7f119effd640 1 -- 172.21.15.1:0/272559441 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119802ca70 con 0x7f11a00bf9d0 2024-03-06T07:21:51.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.895+0000 7f119effd640 1 -- 172.21.15.1:0/272559441 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119802cc10 con 0x7f11a00bf9d0 2024-03-06T07:21:51.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.895+0000 7f119effd640 1 -- 172.21.15.1:0/272559441 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11980343f0 con 0x7f11a00bf9d0 2024-03-06T07:21:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.896+0000 7f11a6629640 1 -- 172.21.15.1:0/272559441 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11a00bf9d0 msgr2=0x7f11a00c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.896+0000 7f11a6629640 1 --2- 172.21.15.1:0/272559441 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11a00bf9d0 0x7f11a00c1460 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1198009e50 tx=0x7f119802bed0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.896+0000 7f11a6629640 1 -- 172.21.15.1:0/272559441 shutdown_connections 2024-03-06T07:21:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.896+0000 7f11a6629640 1 --2- 172.21.15.1:0/272559441 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11a00bf9d0 0x7f11a00c1460 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.896+0000 7f11a6629640 1 --2- 172.21.15.1:0/272559441 >> v2:172.21.15.98:3300/0 conn(0x7f11a00c6b10 0x7f11a0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.896+0000 7f11a6629640 1 -- 172.21.15.1:0/272559441 >> 172.21.15.1:0/272559441 conn(0x7f11a0066ee0 msgr2=0x7f11a00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.896+0000 7f11a6629640 1 -- 172.21.15.1:0/272559441 shutdown_connections 2024-03-06T07:21:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.896+0000 7f11a6629640 1 -- 172.21.15.1:0/272559441 wait complete. 2024-03-06T07:21:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.897+0000 7f11a6629640 1 Processor -- start 2024-03-06T07:21:51.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.897+0000 7f11a6629640 1 -- start start 2024-03-06T07:21:51.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.898+0000 7f11a6629640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11a00bf9d0 0x7f11a014b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.898+0000 7f11a6629640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0152650 0x7f11a01517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.898+0000 7f11a6629640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a01edf40 0x7f11a014e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.898+0000 7f11a6629640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a01551b0 con 0x7f11a0152650 2024-03-06T07:21:51.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.898+0000 7f11a6629640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a00c6fa0 con 0x7f11a01edf40 2024-03-06T07:21:51.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.898+0000 7f11a6629640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a0155490 con 0x7f11a00bf9d0 2024-03-06T07:21:51.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.898+0000 7f119f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0152650 0x7f11a01517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.898+0000 7f119f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0152650 0x7f11a01517f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33126/0 (socket says 172.21.15.1:33126) 2024-03-06T07:21:51.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.898+0000 7f119f7fe640 1 -- 172.21.15.1:0/3699883494 learned_addr learned my addr 172.21.15.1:0/3699883494 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.898+0000 7f119ffff640 1 --2- 172.21.15.1:0/3699883494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11a00bf9d0 0x7f11a014b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.898+0000 7f11a4bc6640 1 --2- 172.21.15.1:0/3699883494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a01edf40 0x7f11a014e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.898+0000 7f119f7fe640 1 -- 172.21.15.1:0/3699883494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11a00bf9d0 msgr2=0x7f11a014b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.898+0000 7f119f7fe640 1 --2- 172.21.15.1:0/3699883494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11a00bf9d0 0x7f11a014b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.898+0000 7f119f7fe640 1 -- 172.21.15.1:0/3699883494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a01edf40 msgr2=0x7f11a014e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.898+0000 7f119f7fe640 1 --2- 172.21.15.1:0/3699883494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a01edf40 0x7f11a014e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.898+0000 7f119f7fe640 1 -- 172.21.15.1:0/3699883494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a014b690 con 0x7f11a0152650 2024-03-06T07:21:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.898+0000 7f119f7fe640 1 --2- 172.21.15.1:0/3699883494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0152650 0x7f11a01517f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f119400e9e0 tx=0x7f119400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.899+0000 7f119cff9640 1 -- 172.21.15.1:0/3699883494 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1194017070 con 0x7f11a0152650 2024-03-06T07:21:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.899+0000 7f119cff9640 1 -- 172.21.15.1:0/3699883494 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119400c9d0 con 0x7f11a0152650 2024-03-06T07:21:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.899+0000 7f119cff9640 1 -- 172.21.15.1:0/3699883494 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119400cdc0 con 0x7f11a0152650 2024-03-06T07:21:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.899+0000 7f11a6629640 1 -- 172.21.15.1:0/3699883494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a01583e0 con 0x7f11a0152650 2024-03-06T07:21:51.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.899+0000 7f11a6629640 1 -- 172.21.15.1:0/3699883494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a01e11d0 con 0x7f11a0152650 2024-03-06T07:21:51.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.900+0000 7f11a6629640 1 -- 172.21.15.1:0/3699883494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a0067fd0 con 0x7f11a0152650 2024-03-06T07:21:51.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.900+0000 7f119cff9640 1 -- 172.21.15.1:0/3699883494 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f119401a070 con 0x7f11a0152650 2024-03-06T07:21:51.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.901+0000 7f119cff9640 1 --2- 172.21.15.1:0/3699883494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f118c06b4d0 0x7f118c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.901+0000 7f119cff9640 1 -- 172.21.15.1:0/3699883494 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f119408a290 con 0x7f11a0152650 2024-03-06T07:21:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.901+0000 7f119ffff640 1 --2- 172.21.15.1:0/3699883494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f118c06b4d0 0x7f118c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.901+0000 7f119ffff640 1 --2- 172.21.15.1:0/3699883494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f118c06b4d0 0x7f118c06d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1190000f60 tx=0x7f1190009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:51.905+0000 7f119cff9640 1 -- 172.21.15.1:0/3699883494 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1194058970 con 0x7f11a0152650 2024-03-06T07:21:52.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.126+0000 7f11a6629640 1 -- 172.21.15.1:0/3699883494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11a01e1540 con 0x7f11a0152650 2024-03-06T07:21:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.127+0000 7f119cff9640 1 -- 172.21.15.1:0/3699883494 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f119405ca10 con 0x7f11a0152650 2024-03-06T07:21:52.129 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:52.129 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3354292028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3354292028},{"type":"v1","addr":"172.21.15.98:6836","nonce":3354292028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4975,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3948803645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3948803645},{"type":"v1","addr":"172.21.15.98:6837","nonce":3948803645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5538,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3390157192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3390157192},{"type":"v1","addr":"172.21.15.1:6834","nonce":3390157192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:22.528539+0000","modified":"2024-03-06T07:21:43.207337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":10,"addr":"172.21.15.1:6835/2438105345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2438105345},{"type":"v1","addr":"172.21.15.1:6835","nonce":2438105345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4781,"qdb_cluster":[4781]},"id":2}]} 2024-03-06T07:21:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 22 2024-03-06T07:21:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.128+0000 7f11a6629640 1 -- 172.21.15.1:0/3699883494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f118c06b4d0 msgr2=0x7f118c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.128+0000 7f11a6629640 1 --2- 172.21.15.1:0/3699883494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f118c06b4d0 0x7f118c06d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1190000f60 tx=0x7f1190009040 comp rx=0 tx=0).stop 2024-03-06T07:21:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.129+0000 7f11a6629640 1 -- 172.21.15.1:0/3699883494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0152650 msgr2=0x7f11a01517f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.129+0000 7f11a6629640 1 --2- 172.21.15.1:0/3699883494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0152650 0x7f11a01517f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f119400e9e0 tx=0x7f119400eea0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.129+0000 7f11a6629640 1 -- 172.21.15.1:0/3699883494 shutdown_connections 2024-03-06T07:21:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.129+0000 7f11a6629640 1 --2- 172.21.15.1:0/3699883494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f118c06b4d0 0x7f118c06d990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.129+0000 7f11a6629640 1 --2- 172.21.15.1:0/3699883494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a01edf40 0x7f11a014e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.129+0000 7f11a6629640 1 --2- 172.21.15.1:0/3699883494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0152650 0x7f11a01517f0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.129+0000 7f11a6629640 1 --2- 172.21.15.1:0/3699883494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11a00bf9d0 0x7f11a014b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.129+0000 7f11a6629640 1 -- 172.21.15.1:0/3699883494 >> 172.21.15.1:0/3699883494 conn(0x7f11a0066ee0 msgr2=0x7f11a00bfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.129+0000 7f11a6629640 1 -- 172.21.15.1:0/3699883494 shutdown_connections 2024-03-06T07:21:52.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.129+0000 7f11a6629640 1 -- 172.21.15.1:0/3699883494 wait complete. 2024-03-06T07:21:52.142 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:52.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.242+0000 7f555ab0f640 1 Processor -- start 2024-03-06T07:21:52.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.242+0000 7f555ab0f640 1 -- start start 2024-03-06T07:21:52.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.242+0000 7f555ab0f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f55540c5520 0x7f55541550d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.242+0000 7f555ab0f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55540c84c0 0x7f5554158d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.242+0000 7f555ab0f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5554159240 0x7f555406da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.242+0000 7f555ab0f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55540c7650 con 0x7f55540c5520 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.242+0000 7f555ab0f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55540c78f0 con 0x7f5554159240 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.242+0000 7f555ab0f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55541587e0 con 0x7f55540c84c0 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.243+0000 7f55588ab640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f55540c5520 0x7f55541550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.243+0000 7f55588ab640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f55540c5520 0x7f55541550d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33140/0 (socket says 172.21.15.1:33140) 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.243+0000 7f55588ab640 1 -- 172.21.15.1:0/925528153 learned_addr learned my addr 172.21.15.1:0/925528153 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.243+0000 7f5553fff640 1 --2- 172.21.15.1:0/925528153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55540c84c0 0x7f5554158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.243+0000 7f55590ac640 1 --2- 172.21.15.1:0/925528153 >> v2:172.21.15.98:3300/0 conn(0x7f5554159240 0x7f555406da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.243+0000 7f55588ab640 1 -- 172.21.15.1:0/925528153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55540c84c0 msgr2=0x7f5554158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.243+0000 7f55588ab640 1 --2- 172.21.15.1:0/925528153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55540c84c0 0x7f5554158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.243+0000 7f55588ab640 1 -- 172.21.15.1:0/925528153 >> v2:172.21.15.98:3300/0 conn(0x7f5554159240 msgr2=0x7f555406da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.243+0000 7f55588ab640 1 --2- 172.21.15.1:0/925528153 >> v2:172.21.15.98:3300/0 conn(0x7f5554159240 0x7f555406da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.243+0000 7f55588ab640 1 -- 172.21.15.1:0/925528153 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55540c3ce0 con 0x7f55540c5520 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.243+0000 7f5553fff640 1 --2- 172.21.15.1:0/925528153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55540c84c0 0x7f5554158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.243+0000 7f55590ac640 1 --2- 172.21.15.1:0/925528153 >> v2:172.21.15.98:3300/0 conn(0x7f5554159240 0x7f555406da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.244+0000 7f55588ab640 1 --2- 172.21.15.1:0/925528153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55540c5520 0x7f55541550d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f55540c2d80 tx=0x7f554802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91094537ab1bc8db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.244+0000 7f55537fe640 1 -- 172.21.15.1:0/925528153 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554802e070 con 0x7f55540c5520 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.244+0000 7f55537fe640 1 -- 172.21.15.1:0/925528153 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5548037070 con 0x7f55540c5520 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.244+0000 7f55537fe640 1 -- 172.21.15.1:0/925528153 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5548031a00 con 0x7f55540c5520 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.244+0000 7f555ab0f640 1 -- 172.21.15.1:0/925528153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55540c5520 msgr2=0x7f55541550d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.244+0000 7f555ab0f640 1 --2- 172.21.15.1:0/925528153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55540c5520 0x7f55541550d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f55540c2d80 tx=0x7f554802b880 comp rx=0 tx=0).stop 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.244+0000 7f555ab0f640 1 -- 172.21.15.1:0/925528153 shutdown_connections 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.244+0000 7f555ab0f640 1 --2- 172.21.15.1:0/925528153 >> v2:172.21.15.98:3300/0 conn(0x7f5554159240 0x7f555406da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.244+0000 7f555ab0f640 1 --2- 172.21.15.1:0/925528153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55540c84c0 0x7f5554158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.244+0000 7f555ab0f640 1 --2- 172.21.15.1:0/925528153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55540c5520 0x7f55541550d0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.244+0000 7f555ab0f640 1 -- 172.21.15.1:0/925528153 >> 172.21.15.1:0/925528153 conn(0x7f55540c88f0 msgr2=0x7f55540c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.244+0000 7f555ab0f640 1 -- 172.21.15.1:0/925528153 shutdown_connections 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.244+0000 7f555ab0f640 1 -- 172.21.15.1:0/925528153 wait complete. 2024-03-06T07:21:52.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.246+0000 7f555ab0f640 1 Processor -- start 2024-03-06T07:21:52.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.246+0000 7f555ab0f640 1 -- start start 2024-03-06T07:21:52.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.247+0000 7f555ab0f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55540c5520 0x7f55541e14f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.247+0000 7f555ab0f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55540c84c0 0x7f55541e1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.247+0000 7f555ab0f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5554159240 0x7f55540ac860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.247+0000 7f555ab0f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55540c5a80 con 0x7f55540c84c0 2024-03-06T07:21:52.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.247+0000 7f555ab0f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55540c5900 con 0x7f55540c5520 2024-03-06T07:21:52.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.247+0000 7f555ab0f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55540c3700 con 0x7f5554159240 2024-03-06T07:21:52.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.247+0000 7f5553fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55540c84c0 0x7f55541e1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.247+0000 7f5553fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55540c84c0 0x7f55541e1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33156/0 (socket says 172.21.15.1:33156) 2024-03-06T07:21:52.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.247+0000 7f5553fff640 1 -- 172.21.15.1:0/3205191686 learned_addr learned my addr 172.21.15.1:0/3205191686 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:52.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.248+0000 7f55588ab640 1 --2- 172.21.15.1:0/3205191686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55540c5520 0x7f55541e14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.248+0000 7f55590ac640 1 --2- 172.21.15.1:0/3205191686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5554159240 0x7f55540ac860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.248+0000 7f5553fff640 1 -- 172.21.15.1:0/3205191686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5554159240 msgr2=0x7f55540ac860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.248+0000 7f5553fff640 1 --2- 172.21.15.1:0/3205191686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5554159240 0x7f55540ac860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.248+0000 7f5553fff640 1 -- 172.21.15.1:0/3205191686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55540c5520 msgr2=0x7f55541e14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.248+0000 7f5553fff640 1 --2- 172.21.15.1:0/3205191686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55540c5520 0x7f55541e14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.248+0000 7f5553fff640 1 -- 172.21.15.1:0/3205191686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55541e0b60 con 0x7f55540c84c0 2024-03-06T07:21:52.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.248+0000 7f55588ab640 1 --2- 172.21.15.1:0/3205191686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55540c5520 0x7f55541e14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:52.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.248+0000 7f5553fff640 1 --2- 172.21.15.1:0/3205191686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55540c84c0 0x7f55541e1a30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f554000e840 tx=0x7f554000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.248+0000 7f55517fa640 1 -- 172.21.15.1:0/3205191686 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554000f040 con 0x7f55540c84c0 2024-03-06T07:21:52.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.248+0000 7f55517fa640 1 -- 172.21.15.1:0/3205191686 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5540004230 con 0x7f55540c84c0 2024-03-06T07:21:52.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.248+0000 7f555ab0f640 1 -- 172.21.15.1:0/3205191686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55541e2120 con 0x7f55540c84c0 2024-03-06T07:21:52.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.248+0000 7f55517fa640 1 -- 172.21.15.1:0/3205191686 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554000c6d0 con 0x7f55540c84c0 2024-03-06T07:21:52.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.248+0000 7f555ab0f640 1 -- 172.21.15.1:0/3205191686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55541eb890 con 0x7f55540c84c0 2024-03-06T07:21:52.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.249+0000 7f555ab0f640 1 -- 172.21.15.1:0/3205191686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5554158600 con 0x7f55540c84c0 2024-03-06T07:21:52.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.250+0000 7f55517fa640 1 -- 172.21.15.1:0/3205191686 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f554000c870 con 0x7f55540c84c0 2024-03-06T07:21:52.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.250+0000 7f55517fa640 1 --2- 172.21.15.1:0/3205191686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f553806b4d0 0x7f553806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.250+0000 7f55517fa640 1 -- 172.21.15.1:0/3205191686 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5540089ff0 con 0x7f55540c84c0 2024-03-06T07:21:52.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.250+0000 7f55588ab640 1 --2- 172.21.15.1:0/3205191686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f553806b4d0 0x7f553806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.251+0000 7f55588ab640 1 --2- 172.21.15.1:0/3205191686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f553806b4d0 0x7f553806d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5548036000 tx=0x7f5548008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.254+0000 7f55517fa640 1 -- 172.21.15.1:0/3205191686 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5554158600 con 0x7f55540c84c0 2024-03-06T07:21:52.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.608+0000 7f555ab0f640 1 -- 172.21.15.1:0/3205191686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55540686e0 con 0x7f55540c84c0 2024-03-06T07:21:52.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.609+0000 7f55517fa640 1 -- 172.21.15.1:0/3205191686 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+16949 (secure 0 0 0) 0x7f5540017020 con 0x7f55540c84c0 2024-03-06T07:21:52.611 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:52.611 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":40,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:21:37.195560+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:21.073384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:21.484387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6834/1818141892":"2024-03-07T07:21:37.195540+0000","172.21.15.1:6832/1818141892":"2024-03-07T07:21:37.195540+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.611+0000 7f555ab0f640 1 -- 172.21.15.1:0/3205191686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f553806b4d0 msgr2=0x7f553806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.611+0000 7f555ab0f640 1 --2- 172.21.15.1:0/3205191686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f553806b4d0 0x7f553806d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5548036000 tx=0x7f5548008030 comp rx=0 tx=0).stop 2024-03-06T07:21:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.611+0000 7f555ab0f640 1 -- 172.21.15.1:0/3205191686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55540c84c0 msgr2=0x7f55541e1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.611+0000 7f555ab0f640 1 --2- 172.21.15.1:0/3205191686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55540c84c0 0x7f55541e1a30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f554000e840 tx=0x7f554000ed00 comp rx=0 tx=0).stop 2024-03-06T07:21:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.611+0000 7f555ab0f640 1 -- 172.21.15.1:0/3205191686 shutdown_connections 2024-03-06T07:21:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.611+0000 7f555ab0f640 1 --2- 172.21.15.1:0/3205191686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f553806b4d0 0x7f553806d990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.611+0000 7f555ab0f640 1 --2- 172.21.15.1:0/3205191686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5554159240 0x7f55540ac860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.611+0000 7f555ab0f640 1 --2- 172.21.15.1:0/3205191686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55540c84c0 0x7f55541e1a30 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.611+0000 7f555ab0f640 1 --2- 172.21.15.1:0/3205191686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55540c5520 0x7f55541e14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.611+0000 7f555ab0f640 1 -- 172.21.15.1:0/3205191686 >> 172.21.15.1:0/3205191686 conn(0x7f55540c88f0 msgr2=0x7f55540a9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.611+0000 7f555ab0f640 1 -- 172.21.15.1:0/3205191686 shutdown_connections 2024-03-06T07:21:52.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.611+0000 7f555ab0f640 1 -- 172.21.15.1:0/3205191686 wait complete. 2024-03-06T07:21:52.624 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:21:52.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.725+0000 7f7b9821c640 1 Processor -- start 2024-03-06T07:21:52.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.725+0000 7f7b9821c640 1 -- start start 2024-03-06T07:21:52.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.726+0000 7f7b9821c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7b90152650 0x7f7b90155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.726+0000 7f7b9821c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7b900c6b10 0x7f7b90158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.726+0000 7f7b9821c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b900c2fc0 con 0x7f7b900bf9d0 2024-03-06T07:21:52.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.726+0000 7f7b9821c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b90155800 con 0x7f7b90152650 2024-03-06T07:21:52.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.726+0000 7f7b9821c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b900c6690 con 0x7f7b900c6b10 2024-03-06T07:21:52.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.726+0000 7f7b95fb8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7b90152650 0x7f7b90155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.726+0000 7f7b967b9640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7b900bf9d0 0x7f7b900c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55528/0 (socket says 172.21.15.1:55528) 2024-03-06T07:21:52.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.726+0000 7f7b95fb8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7b90152650 0x7f7b90155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33172/0 (socket says 172.21.15.1:33172) 2024-03-06T07:21:52.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.726+0000 7f7b967b9640 1 -- 172.21.15.1:0/3648810835 learned_addr learned my addr 172.21.15.1:0/3648810835 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:52.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.726+0000 7f7b957b7640 1 --2- 172.21.15.1:0/3648810835 >> v2:172.21.15.98:3300/0 conn(0x7f7b900c6b10 0x7f7b90158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.726+0000 7f7b95fb8640 1 -- 172.21.15.1:0/3648810835 >> v2:172.21.15.98:3300/0 conn(0x7f7b900c6b10 msgr2=0x7f7b90158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:52.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.726+0000 7f7b95fb8640 1 --2- 172.21.15.1:0/3648810835 >> v2:172.21.15.98:3300/0 conn(0x7f7b900c6b10 0x7f7b90158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.726+0000 7f7b95fb8640 1 -- 172.21.15.1:0/3648810835 >> v1:172.21.15.1:6789/0 conn(0x7f7b900bf9d0 legacy=0x7f7b900c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.726+0000 7f7b95fb8640 1 -- 172.21.15.1:0/3648810835 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b900c7280 con 0x7f7b90152650 2024-03-06T07:21:52.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.726+0000 7f7b94fb6640 1 -- 172.21.15.1:0/3648810835 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476625019 0 0) 0x7f7b900c2fc0 con 0x7f7b900bf9d0 2024-03-06T07:21:52.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.726+0000 7f7b957b7640 1 --2- 172.21.15.1:0/3648810835 >> v2:172.21.15.98:3300/0 conn(0x7f7b900c6b10 0x7f7b90158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:52.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.727+0000 7f7b95fb8640 1 --2- 172.21.15.1:0/3648810835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b90152650 0x7f7b90155c90 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7b900cb8e0 tx=0x7f7b8c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7409a8fa7c778da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.727+0000 7f7b94fb6640 1 -- 172.21.15.1:0/3648810835 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c0315a0 con 0x7f7b90152650 2024-03-06T07:21:52.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.727+0000 7f7b94fb6640 1 -- 172.21.15.1:0/3648810835 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8c037070 con 0x7f7b90152650 2024-03-06T07:21:52.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.727+0000 7f7b94fb6640 1 -- 172.21.15.1:0/3648810835 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c031960 con 0x7f7b90152650 2024-03-06T07:21:52.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.728+0000 7f7b9821c640 1 -- 172.21.15.1:0/3648810835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b90152650 msgr2=0x7f7b90155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.728+0000 7f7b9821c640 1 --2- 172.21.15.1:0/3648810835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b90152650 0x7f7b90155c90 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7b900cb8e0 tx=0x7f7b8c02b840 comp rx=0 tx=0).stop 2024-03-06T07:21:52.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.728+0000 7f7b9821c640 1 -- 172.21.15.1:0/3648810835 shutdown_connections 2024-03-06T07:21:52.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.728+0000 7f7b9821c640 1 --2- 172.21.15.1:0/3648810835 >> v2:172.21.15.98:3300/0 conn(0x7f7b900c6b10 0x7f7b90158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.728+0000 7f7b9821c640 1 --2- 172.21.15.1:0/3648810835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b90152650 0x7f7b90155c90 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.728+0000 7f7b9821c640 1 -- 172.21.15.1:0/3648810835 >> 172.21.15.1:0/3648810835 conn(0x7f7b90066ee0 msgr2=0x7f7b900672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.728+0000 7f7b9821c640 1 -- 172.21.15.1:0/3648810835 shutdown_connections 2024-03-06T07:21:52.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.728+0000 7f7b9821c640 1 -- 172.21.15.1:0/3648810835 wait complete. 2024-03-06T07:21:52.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.729+0000 7f7b9821c640 1 Processor -- start 2024-03-06T07:21:52.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.729+0000 7f7b9821c640 1 -- start start 2024-03-06T07:21:52.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b9821c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7b900bf9d0 0x7f7b901eed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b9821c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b90152650 0x7f7b900c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b9821c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b90156ea0 0x7f7b90143fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b9821c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b90155490 con 0x7f7b90156ea0 2024-03-06T07:21:52.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b9821c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b901551b0 con 0x7f7b90152650 2024-03-06T07:21:52.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b9821c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b90152a30 con 0x7f7b900bf9d0 2024-03-06T07:21:52.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b967b9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b90156ea0 0x7f7b90143fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b967b9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b90156ea0 0x7f7b90143fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33176/0 (socket says 172.21.15.1:33176) 2024-03-06T07:21:52.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b967b9640 1 -- 172.21.15.1:0/2730464189 learned_addr learned my addr 172.21.15.1:0/2730464189 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:52.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b957b7640 1 --2- 172.21.15.1:0/2730464189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b90152650 0x7f7b900c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b95fb8640 1 --2- 172.21.15.1:0/2730464189 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7b900bf9d0 0x7f7b901eed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b967b9640 1 -- 172.21.15.1:0/2730464189 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7b900bf9d0 msgr2=0x7f7b901eed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b967b9640 1 --2- 172.21.15.1:0/2730464189 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7b900bf9d0 0x7f7b901eed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b967b9640 1 -- 172.21.15.1:0/2730464189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b90152650 msgr2=0x7f7b900c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b967b9640 1 --2- 172.21.15.1:0/2730464189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b90152650 0x7f7b900c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b967b9640 1 -- 172.21.15.1:0/2730464189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b90154020 con 0x7f7b90156ea0 2024-03-06T07:21:52.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b967b9640 1 --2- 172.21.15.1:0/2730464189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b90156ea0 0x7f7b90143fe0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7b8800c700 tx=0x7f7b8800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b867fc640 1 -- 172.21.15.1:0/2730464189 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88018070 con 0x7f7b90156ea0 2024-03-06T07:21:52.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b867fc640 1 -- 172.21.15.1:0/2730464189 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8800ce10 con 0x7f7b90156ea0 2024-03-06T07:21:52.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b9821c640 1 -- 172.21.15.1:0/2730464189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b901578c0 con 0x7f7b90156ea0 2024-03-06T07:21:52.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.730+0000 7f7b867fc640 1 -- 172.21.15.1:0/2730464189 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8800a880 con 0x7f7b90156ea0 2024-03-06T07:21:52.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.731+0000 7f7b9821c640 1 -- 172.21.15.1:0/2730464189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b901ee580 con 0x7f7b90156ea0 2024-03-06T07:21:52.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.732+0000 7f7b9821c640 1 -- 172.21.15.1:0/2730464189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b90067fd0 con 0x7f7b90156ea0 2024-03-06T07:21:52.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.732+0000 7f7b867fc640 1 -- 172.21.15.1:0/2730464189 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b8801b070 con 0x7f7b90156ea0 2024-03-06T07:21:52.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.732+0000 7f7b867fc640 1 --2- 172.21.15.1:0/2730464189 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7b7406b4d0 0x7f7b7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.733+0000 7f7b867fc640 1 -- 172.21.15.1:0/2730464189 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f7b880897d0 con 0x7f7b90156ea0 2024-03-06T07:21:52.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.733+0000 7f7b95fb8640 1 --2- 172.21.15.1:0/2730464189 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7b7406b4d0 0x7f7b7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.733+0000 7f7b95fb8640 1 --2- 172.21.15.1:0/2730464189 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7b7406b4d0 0x7f7b7406d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c036000 tx=0x7f7b8c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:52.737+0000 7f7b867fc640 1 -- 172.21.15.1:0/2730464189 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b88057eb0 con 0x7f7b90156ea0 2024-03-06T07:21:53.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.049+0000 7f7b9821c640 1 -- 172.21.15.1:0/2730464189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7b90151e70 con 0x7f7b90156ea0 2024-03-06T07:21:53.051 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:21:53.049+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:21:53.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.051+0000 7f7b867fc640 1 -- 172.21.15.1:0/2730464189 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v23) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7b8805bf50 con 0x7f7b90156ea0 2024-03-06T07:21:53.052 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:21:53.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.053+0000 7f7b9821c640 1 -- 172.21.15.1:0/2730464189 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7b7406b4d0 msgr2=0x7f7b7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.053+0000 7f7b9821c640 1 --2- 172.21.15.1:0/2730464189 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7b7406b4d0 0x7f7b7406d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c036000 tx=0x7f7b8c008030 comp rx=0 tx=0).stop 2024-03-06T07:21:53.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.053+0000 7f7b9821c640 1 -- 172.21.15.1:0/2730464189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b90156ea0 msgr2=0x7f7b90143fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.053+0000 7f7b9821c640 1 --2- 172.21.15.1:0/2730464189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b90156ea0 0x7f7b90143fe0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7b8800c700 tx=0x7f7b8800cbc0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.053+0000 7f7b9821c640 1 -- 172.21.15.1:0/2730464189 shutdown_connections 2024-03-06T07:21:53.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.053+0000 7f7b9821c640 1 --2- 172.21.15.1:0/2730464189 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7b7406b4d0 0x7f7b7406d990 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.053+0000 7f7b9821c640 1 --2- 172.21.15.1:0/2730464189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b90156ea0 0x7f7b90143fe0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.053+0000 7f7b9821c640 1 --2- 172.21.15.1:0/2730464189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b90152650 0x7f7b900c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.053+0000 7f7b9821c640 1 --2- 172.21.15.1:0/2730464189 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7b900bf9d0 0x7f7b901eed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.053+0000 7f7b9821c640 1 -- 172.21.15.1:0/2730464189 >> 172.21.15.1:0/2730464189 conn(0x7f7b90066ee0 msgr2=0x7f7b900672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.053+0000 7f7b9821c640 1 -- 172.21.15.1:0/2730464189 shutdown_connections 2024-03-06T07:21:53.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.053+0000 7f7b9821c640 1 -- 172.21.15.1:0/2730464189 wait complete. 2024-03-06T07:21:53.057 INFO:tasks.ceph.mds.c.smithi001.stderr: -371> 2024-03-06T07:21:43.144+0000 7f49d74a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:53.067 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:21:53.081 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:21:53.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.168+0000 7f14f0bb5640 1 Processor -- start 2024-03-06T07:21:53.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.168+0000 7f14f0bb5640 1 -- start start 2024-03-06T07:21:53.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.169+0000 7f14f0bb5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f14ec150520 0x7f14ec1543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.169+0000 7f14f0bb5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14ec0c32f0 0x7f14ec1537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.169+0000 7f14f0bb5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ec150900 con 0x7f14ec070c80 2024-03-06T07:21:53.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.169+0000 7f14f0bb5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14ec071790 con 0x7f14ec150520 2024-03-06T07:21:53.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.169+0000 7f14f0bb5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec157050 con 0x7f14ec0c32f0 2024-03-06T07:21:53.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.169+0000 7f14ead9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f14ec070c80 0x7f14ec0c67f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55534/0 (socket says 172.21.15.1:55534) 2024-03-06T07:21:53.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.169+0000 7f14ead9d640 1 -- 172.21.15.1:0/2200385608 learned_addr learned my addr 172.21.15.1:0/2200385608 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:53.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.169+0000 7f14e9d9b640 1 --2- 172.21.15.1:0/2200385608 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14ec0c32f0 0x7f14ec1537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.169+0000 7f14ea59c640 1 --2- 172.21.15.1:0/2200385608 >> v2:172.21.15.98:3300/0 conn(0x7f14ec150520 0x7f14ec1543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14e959a640 1 -- 172.21.15.1:0/2200385608 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892173176 0 0) 0x7f14ec150900 con 0x7f14ec070c80 2024-03-06T07:21:53.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14e959a640 1 -- 172.21.15.1:0/2200385608 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d8003520 con 0x7f14ec070c80 2024-03-06T07:21:53.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14e959a640 1 -- 172.21.15.1:0/2200385608 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 10378412 0 0) 0x7f14d8003520 con 0x7f14ec070c80 2024-03-06T07:21:53.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14e959a640 1 -- 172.21.15.1:0/2200385608 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14ec0c32f0 msgr2=0x7f14ec1537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:53.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14e959a640 1 --2- 172.21.15.1:0/2200385608 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14ec0c32f0 0x7f14ec1537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14e959a640 1 -- 172.21.15.1:0/2200385608 >> v2:172.21.15.98:3300/0 conn(0x7f14ec150520 msgr2=0x7f14ec1543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:53.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14e959a640 1 --2- 172.21.15.1:0/2200385608 >> v2:172.21.15.98:3300/0 conn(0x7f14ec150520 0x7f14ec1543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14e959a640 1 -- 172.21.15.1:0/2200385608 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec152fc0 con 0x7f14ec070c80 2024-03-06T07:21:53.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14e959a640 1 -- 172.21.15.1:0/2200385608 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f14dc0019f0 con 0x7f14ec070c80 2024-03-06T07:21:53.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14e959a640 1 -- 172.21.15.1:0/2200385608 >> v1:172.21.15.1:6789/0 conn(0x7f14ec070c80 legacy=0x7f14ec0c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14e959a640 1 --2- 172.21.15.1:0/2200385608 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14ec0c32f0 0x7f14d80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14e959a640 1 --2- 172.21.15.1:0/2200385608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14d8003f30 0x7f14d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14e959a640 1 --2- 172.21.15.1:0/2200385608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d8004d40 0x7f14d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14e959a640 1 -- 172.21.15.1:0/2200385608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec071790 con 0x7f14d8004d40 2024-03-06T07:21:53.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14e959a640 1 -- 172.21.15.1:0/2200385608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec157050 con 0x7f14d8003f30 2024-03-06T07:21:53.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14e959a640 1 -- 172.21.15.1:0/2200385608 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d800b9c0 con 0x7f14ec0c32f0 2024-03-06T07:21:53.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14ead9d640 1 --2- 172.21.15.1:0/2200385608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d8004d40 0x7f14d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14ea59c640 1 --2- 172.21.15.1:0/2200385608 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14ec0c32f0 0x7f14d80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14e9d9b640 1 --2- 172.21.15.1:0/2200385608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14d8003f30 0x7f14d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14ead9d640 1 -- 172.21.15.1:0/2200385608 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14ec0c32f0 msgr2=0x7f14d80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:53.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14ead9d640 1 --2- 172.21.15.1:0/2200385608 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14ec0c32f0 0x7f14d80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14ead9d640 1 -- 172.21.15.1:0/2200385608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14d8003f30 msgr2=0x7f14d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:53.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14ead9d640 1 --2- 172.21.15.1:0/2200385608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14d8003f30 0x7f14d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.170+0000 7f14ead9d640 1 -- 172.21.15.1:0/2200385608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d8003520 con 0x7f14d8004d40 2024-03-06T07:21:53.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.171+0000 7f14ead9d640 1 --2- 172.21.15.1:0/2200385608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d8004d40 0x7f14d80051c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f14dc00a880 tx=0x7f14d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368e6b95263954fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.171+0000 7f14e959a640 1 -- 172.21.15.1:0/2200385608 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc00a4a0 con 0x7f14d8004d40 2024-03-06T07:21:53.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.171+0000 7f14e959a640 1 -- 172.21.15.1:0/2200385608 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14dc034650 con 0x7f14d8004d40 2024-03-06T07:21:53.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.171+0000 7f14e959a640 1 -- 172.21.15.1:0/2200385608 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc034a80 con 0x7f14d8004d40 2024-03-06T07:21:53.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.172+0000 7f14f0bb5640 1 -- 172.21.15.1:0/2200385608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d8004d40 msgr2=0x7f14d80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.172+0000 7f14f0bb5640 1 --2- 172.21.15.1:0/2200385608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d8004d40 0x7f14d80051c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f14dc00a880 tx=0x7f14d8005830 comp rx=0 tx=0).stop 2024-03-06T07:21:53.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.172+0000 7f14ead9d640 1 -- 172.21.15.1:0/2200385608 reap_dead start 2024-03-06T07:21:53.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.172+0000 7f14f0bb5640 1 -- 172.21.15.1:0/2200385608 shutdown_connections 2024-03-06T07:21:53.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.172+0000 7f14f0bb5640 1 -- 172.21.15.1:0/2200385608 >> 172.21.15.1:0/2200385608 conn(0x7f14ec0c88f0 msgr2=0x7f14ec0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.172+0000 7f14f0bb5640 1 -- 172.21.15.1:0/2200385608 shutdown_connections 2024-03-06T07:21:53.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.172+0000 7f14f0bb5640 1 -- 172.21.15.1:0/2200385608 wait complete. 2024-03-06T07:21:53.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.173+0000 7f14f0bb5640 1 Processor -- start 2024-03-06T07:21:53.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.173+0000 7f14f0bb5640 1 -- start start 2024-03-06T07:21:53.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14f0bb5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d8004d40 0x7f14ec156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14f0bb5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14ec06d110 0x7f14ec0c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14f0bb5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14ec1edc10 0x7f14ec06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14f0bb5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec0c7ec0 con 0x7f14d8004d40 2024-03-06T07:21:53.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14f0bb5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec157330 con 0x7f14ec06d110 2024-03-06T07:21:53.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14f0bb5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec06f4e0 con 0x7f14ec1edc10 2024-03-06T07:21:53.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14ea59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d8004d40 0x7f14ec156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14ea59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d8004d40 0x7f14ec156a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33204/0 (socket says 172.21.15.1:33204) 2024-03-06T07:21:53.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14ea59c640 1 -- 172.21.15.1:0/4288346245 learned_addr learned my addr 172.21.15.1:0/4288346245 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:53.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14ead9d640 1 --2- 172.21.15.1:0/4288346245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14ec1edc10 0x7f14ec06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14e9d9b640 1 --2- 172.21.15.1:0/4288346245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14ec06d110 0x7f14ec0c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14ea59c640 1 -- 172.21.15.1:0/4288346245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14ec1edc10 msgr2=0x7f14ec06c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14ea59c640 1 --2- 172.21.15.1:0/4288346245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14ec1edc10 0x7f14ec06c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14ea59c640 1 -- 172.21.15.1:0/4288346245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14ec06d110 msgr2=0x7f14ec0c4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14ea59c640 1 --2- 172.21.15.1:0/4288346245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14ec06d110 0x7f14ec0c4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14ea59c640 1 -- 172.21.15.1:0/4288346245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec0c5800 con 0x7f14d8004d40 2024-03-06T07:21:53.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14e9d9b640 1 --2- 172.21.15.1:0/4288346245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14ec06d110 0x7f14ec0c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:53.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14ea59c640 1 --2- 172.21.15.1:0/4288346245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d8004d40 0x7f14ec156a40 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f14e4011ad0 tx=0x7f14e4011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14e2ffd640 1 -- 172.21.15.1:0/4288346245 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e400fbb0 con 0x7f14d8004d40 2024-03-06T07:21:53.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.174+0000 7f14e2ffd640 1 -- 172.21.15.1:0/4288346245 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e400fd50 con 0x7f14d8004d40 2024-03-06T07:21:53.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.175+0000 7f14e2ffd640 1 -- 172.21.15.1:0/4288346245 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e4014660 con 0x7f14d8004d40 2024-03-06T07:21:53.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.175+0000 7f14f0bb5640 1 -- 172.21.15.1:0/4288346245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ec0c5060 con 0x7f14d8004d40 2024-03-06T07:21:53.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.175+0000 7f14f0bb5640 1 -- 172.21.15.1:0/4288346245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ec06cdb0 con 0x7f14d8004d40 2024-03-06T07:21:53.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.175+0000 7f14f0bb5640 1 -- 172.21.15.1:0/4288346245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ec0686e0 con 0x7f14d8004d40 2024-03-06T07:21:53.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.176+0000 7f14e2ffd640 1 -- 172.21.15.1:0/4288346245 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14e4012040 con 0x7f14d8004d40 2024-03-06T07:21:53.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.176+0000 7f14e2ffd640 1 --2- 172.21.15.1:0/4288346245 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f14cc06b500 0x7f14cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.177+0000 7f14e2ffd640 1 -- 172.21.15.1:0/4288346245 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f14e408ab80 con 0x7f14d8004d40 2024-03-06T07:21:53.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.177+0000 7f14e9d9b640 1 --2- 172.21.15.1:0/4288346245 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f14cc06b500 0x7f14cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.177+0000 7f14e9d9b640 1 --2- 172.21.15.1:0/4288346245 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f14cc06b500 0x7f14cc06d9c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0c31a0 tx=0x7f14d4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.181+0000 7f14e2ffd640 1 -- 172.21.15.1:0/4288346245 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14e4019030 con 0x7f14d8004d40 2024-03-06T07:21:53.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:53.400+0000 7f14f0bb5640 1 -- 172.21.15.1:0/4288346245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f14ec0c7b10 con 0x7f14d8004d40 2024-03-06T07:21:54.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.052+0000 7f14e2ffd640 1 -- 172.21.15.1:0/4288346245 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v24) v1 ==== 104+0+0 (secure 0 0 0) 0x7f14e40597c0 con 0x7f14d8004d40 2024-03-06T07:21:54.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.055+0000 7f14f0bb5640 1 -- 172.21.15.1:0/4288346245 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f14cc06b500 msgr2=0x7f14cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.055+0000 7f14f0bb5640 1 --2- 172.21.15.1:0/4288346245 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f14cc06b500 0x7f14cc06d9c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0c31a0 tx=0x7f14d4008040 comp rx=0 tx=0).stop 2024-03-06T07:21:54.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.055+0000 7f14f0bb5640 1 -- 172.21.15.1:0/4288346245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d8004d40 msgr2=0x7f14ec156a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.055+0000 7f14f0bb5640 1 --2- 172.21.15.1:0/4288346245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d8004d40 0x7f14ec156a40 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f14e4011ad0 tx=0x7f14e4011f90 comp rx=0 tx=0).stop 2024-03-06T07:21:54.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.055+0000 7f14f0bb5640 1 -- 172.21.15.1:0/4288346245 shutdown_connections 2024-03-06T07:21:54.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.055+0000 7f14f0bb5640 1 --2- 172.21.15.1:0/4288346245 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f14cc06b500 0x7f14cc06d9c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.055+0000 7f14f0bb5640 1 --2- 172.21.15.1:0/4288346245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14ec1edc10 0x7f14ec06c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.055+0000 7f14f0bb5640 1 --2- 172.21.15.1:0/4288346245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14ec06d110 0x7f14ec0c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.055+0000 7f14f0bb5640 1 --2- 172.21.15.1:0/4288346245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14d8004d40 0x7f14ec156a40 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.055+0000 7f14f0bb5640 1 -- 172.21.15.1:0/4288346245 >> 172.21.15.1:0/4288346245 conn(0x7f14ec0c88f0 msgr2=0x7f14ec153680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.055+0000 7f14f0bb5640 1 -- 172.21.15.1:0/4288346245 shutdown_connections 2024-03-06T07:21:54.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.055+0000 7f14f0bb5640 1 -- 172.21.15.1:0/4288346245 wait complete. 2024-03-06T07:21:54.071 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:21:54.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.221+0000 7f2c2e3c7640 1 Processor -- start 2024-03-06T07:21:54.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.221+0000 7f2c2e3c7640 1 -- start start 2024-03-06T07:21:54.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.222+0000 7f2c2e3c7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2c28158180 0x7f2c280c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.222+0000 7f2c2e3c7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2c280c5e20 0x7f2c28156a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.222+0000 7f2c2e3c7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c280c6800 0x7f2c28156f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.222+0000 7f2c2e3c7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c28156210 con 0x7f2c28158180 2024-03-06T07:21:54.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.222+0000 7f2c2e3c7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c280c3ea0 con 0x7f2c280c5e20 2024-03-06T07:21:54.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.222+0000 7f2c2e3c7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c28158560 con 0x7f2c280c6800 2024-03-06T07:21:54.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.222+0000 7f2c27fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2c28158180 0x7f2c280c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.222+0000 7f2c27fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2c28158180 0x7f2c280c58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33218/0 (socket says 172.21.15.1:33218) 2024-03-06T07:21:54.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.222+0000 7f2c27fff640 1 -- 172.21.15.1:0/1710022910 learned_addr learned my addr 172.21.15.1:0/1710022910 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:54.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.223+0000 7f2c2c964640 1 --2- 172.21.15.1:0/1710022910 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c280c6800 0x7f2c28156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.223+0000 7f2c277fe640 1 --2- 172.21.15.1:0/1710022910 >> v2:172.21.15.98:3300/0 conn(0x7f2c280c5e20 0x7f2c28156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.223+0000 7f2c27fff640 1 -- 172.21.15.1:0/1710022910 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c280c6800 msgr2=0x7f2c28156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.223+0000 7f2c27fff640 1 --2- 172.21.15.1:0/1710022910 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c280c6800 0x7f2c28156f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.223+0000 7f2c27fff640 1 -- 172.21.15.1:0/1710022910 >> v2:172.21.15.98:3300/0 conn(0x7f2c280c5e20 msgr2=0x7f2c28156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.223+0000 7f2c27fff640 1 --2- 172.21.15.1:0/1710022910 >> v2:172.21.15.98:3300/0 conn(0x7f2c280c5e20 0x7f2c28156a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.223+0000 7f2c27fff640 1 -- 172.21.15.1:0/1710022910 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c28155600 con 0x7f2c28158180 2024-03-06T07:21:54.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.223+0000 7f2c27fff640 1 --2- 172.21.15.1:0/1710022910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c28158180 0x7f2c280c58e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009870 tx=0x7f2c1c02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9996b29782f58f61 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.224+0000 7f2c26ffd640 1 -- 172.21.15.1:0/1710022910 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c038070 con 0x7f2c28158180 2024-03-06T07:21:54.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.224+0000 7f2c26ffd640 1 -- 172.21.15.1:0/1710022910 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1c031740 con 0x7f2c28158180 2024-03-06T07:21:54.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.224+0000 7f2c26ffd640 1 -- 172.21.15.1:0/1710022910 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c02f050 con 0x7f2c28158180 2024-03-06T07:21:54.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.224+0000 7f2c2e3c7640 1 -- 172.21.15.1:0/1710022910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c28158180 msgr2=0x7f2c280c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.224+0000 7f2c2e3c7640 1 --2- 172.21.15.1:0/1710022910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c28158180 0x7f2c280c58e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009870 tx=0x7f2c1c02b7e0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.224+0000 7f2c2e3c7640 1 -- 172.21.15.1:0/1710022910 shutdown_connections 2024-03-06T07:21:54.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.224+0000 7f2c2e3c7640 1 --2- 172.21.15.1:0/1710022910 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c280c6800 0x7f2c28156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.224+0000 7f2c2e3c7640 1 --2- 172.21.15.1:0/1710022910 >> v2:172.21.15.98:3300/0 conn(0x7f2c280c5e20 0x7f2c28156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.224+0000 7f2c2e3c7640 1 --2- 172.21.15.1:0/1710022910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c28158180 0x7f2c280c58e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.224+0000 7f2c2e3c7640 1 -- 172.21.15.1:0/1710022910 >> 172.21.15.1:0/1710022910 conn(0x7f2c2806b410 msgr2=0x7f2c2806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.224+0000 7f2c2e3c7640 1 -- 172.21.15.1:0/1710022910 shutdown_connections 2024-03-06T07:21:54.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.224+0000 7f2c2e3c7640 1 -- 172.21.15.1:0/1710022910 wait complete. 2024-03-06T07:21:54.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.225+0000 7f2c2e3c7640 1 Processor -- start 2024-03-06T07:21:54.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.226+0000 7f2c2e3c7640 1 -- start start 2024-03-06T07:21:54.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.226+0000 7f2c2e3c7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c280c5e20 0x7f2c2806d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.226+0000 7f2c2e3c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c280c6800 0x7f2c2814a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.226+0000 7f2c2e3c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c28158180 0x7f2c2814a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.226+0000 7f2c2e3c7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c28155ea0 con 0x7f2c280c6800 2024-03-06T07:21:54.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.226+0000 7f2c2e3c7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c28155bc0 con 0x7f2c28158180 2024-03-06T07:21:54.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.226+0000 7f2c2e3c7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c281551f0 con 0x7f2c280c5e20 2024-03-06T07:21:54.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.226+0000 7f2c277fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c280c6800 0x7f2c2814a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.226+0000 7f2c277fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c280c6800 0x7f2c2814a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33220/0 (socket says 172.21.15.1:33220) 2024-03-06T07:21:54.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.226+0000 7f2c277fe640 1 -- 172.21.15.1:0/3450327635 learned_addr learned my addr 172.21.15.1:0/3450327635 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:54.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.226+0000 7f2c27fff640 1 --2- 172.21.15.1:0/3450327635 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c280c5e20 0x7f2c2806d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.227+0000 7f2c2c964640 1 --2- 172.21.15.1:0/3450327635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c28158180 0x7f2c2814a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.227+0000 7f2c277fe640 1 -- 172.21.15.1:0/3450327635 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c280c5e20 msgr2=0x7f2c2806d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.227+0000 7f2c277fe640 1 --2- 172.21.15.1:0/3450327635 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c280c5e20 0x7f2c2806d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.227+0000 7f2c277fe640 1 -- 172.21.15.1:0/3450327635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c28158180 msgr2=0x7f2c2814a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.227+0000 7f2c277fe640 1 --2- 172.21.15.1:0/3450327635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c28158180 0x7f2c2814a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.227+0000 7f2c277fe640 1 -- 172.21.15.1:0/3450327635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c28071580 con 0x7f2c280c6800 2024-03-06T07:21:54.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.227+0000 7f2c27fff640 1 --2- 172.21.15.1:0/3450327635 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c280c5e20 0x7f2c2806d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:54.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.227+0000 7f2c2c964640 1 --2- 172.21.15.1:0/3450327635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c28158180 0x7f2c2814a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:54.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.227+0000 7f2c277fe640 1 --2- 172.21.15.1:0/3450327635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c280c6800 0x7f2c2814a340 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800c570 tx=0x7f2c1800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.227+0000 7f2c24ff9640 1 -- 172.21.15.1:0/3450327635 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c18015070 con 0x7f2c280c6800 2024-03-06T07:21:54.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.227+0000 7f2c24ff9640 1 -- 172.21.15.1:0/3450327635 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c18011800 con 0x7f2c280c6800 2024-03-06T07:21:54.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.227+0000 7f2c2e3c7640 1 -- 172.21.15.1:0/3450327635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c281488e0 con 0x7f2c280c6800 2024-03-06T07:21:54.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.227+0000 7f2c24ff9640 1 -- 172.21.15.1:0/3450327635 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c18011bf0 con 0x7f2c280c6800 2024-03-06T07:21:54.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.227+0000 7f2c2e3c7640 1 -- 172.21.15.1:0/3450327635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c28149b40 con 0x7f2c280c6800 2024-03-06T07:21:54.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.228+0000 7f2c2e3c7640 1 -- 172.21.15.1:0/3450327635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c280680b0 con 0x7f2c280c6800 2024-03-06T07:21:54.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.228+0000 7f2c24ff9640 1 -- 172.21.15.1:0/3450327635 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c18018070 con 0x7f2c280c6800 2024-03-06T07:21:54.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.229+0000 7f2c24ff9640 1 --2- 172.21.15.1:0/3450327635 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c1406b4d0 0x7f2c1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.229+0000 7f2c24ff9640 1 -- 172.21.15.1:0/3450327635 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f2c1808a120 con 0x7f2c280c6800 2024-03-06T07:21:54.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.229+0000 7f2c27fff640 1 --2- 172.21.15.1:0/3450327635 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c1406b4d0 0x7f2c1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.230+0000 7f2c27fff640 1 --2- 172.21.15.1:0/3450327635 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c1406b4d0 0x7f2c1406d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c035000 tx=0x7f2c1c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.233+0000 7f2c24ff9640 1 -- 172.21.15.1:0/3450327635 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c18058df0 con 0x7f2c280c6800 2024-03-06T07:21:54.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:54.417+0000 7f2c2e3c7640 1 -- 172.21.15.1:0/3450327635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2c280c2df0 con 0x7f2c280c6800 2024-03-06T07:21:55.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.245+0000 7f2c24ff9640 1 -- 172.21.15.1:0/3450327635 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v42) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2c180586f0 con 0x7f2c280c6800 2024-03-06T07:21:55.247 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:21:55.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.249+0000 7f2c2e3c7640 1 -- 172.21.15.1:0/3450327635 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c1406b4d0 msgr2=0x7f2c1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.249+0000 7f2c2e3c7640 1 --2- 172.21.15.1:0/3450327635 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c1406b4d0 0x7f2c1406d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c035000 tx=0x7f2c1c008030 comp rx=0 tx=0).stop 2024-03-06T07:21:55.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.249+0000 7f2c2e3c7640 1 -- 172.21.15.1:0/3450327635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c280c6800 msgr2=0x7f2c2814a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.249+0000 7f2c2e3c7640 1 --2- 172.21.15.1:0/3450327635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c280c6800 0x7f2c2814a340 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800c570 tx=0x7f2c1800ca30 comp rx=0 tx=0).stop 2024-03-06T07:21:55.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.249+0000 7f2c2e3c7640 1 -- 172.21.15.1:0/3450327635 shutdown_connections 2024-03-06T07:21:55.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.249+0000 7f2c2e3c7640 1 --2- 172.21.15.1:0/3450327635 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c1406b4d0 0x7f2c1406d990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.249+0000 7f2c2e3c7640 1 --2- 172.21.15.1:0/3450327635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c28158180 0x7f2c2814a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.249+0000 7f2c2e3c7640 1 --2- 172.21.15.1:0/3450327635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c280c6800 0x7f2c2814a340 secure :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800c570 tx=0x7f2c1800ca30 comp rx=0 tx=0).stop 2024-03-06T07:21:55.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.249+0000 7f2c2e3c7640 1 --2- 172.21.15.1:0/3450327635 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c280c5e20 0x7f2c2806d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.249+0000 7f2c2e3c7640 1 -- 172.21.15.1:0/3450327635 >> 172.21.15.1:0/3450327635 conn(0x7f2c2806b410 msgr2=0x7f2c28152e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.249+0000 7f2c2e3c7640 1 -- 172.21.15.1:0/3450327635 shutdown_connections 2024-03-06T07:21:55.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.249+0000 7f2c2e3c7640 1 -- 172.21.15.1:0/3450327635 wait complete. 2024-03-06T07:21:55.267 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:21:55.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.375+0000 7f0dbced4640 1 Processor -- start 2024-03-06T07:21:55.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.376+0000 7f0dbced4640 1 -- start start 2024-03-06T07:21:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.376+0000 7f0dbced4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db8155030 0x7f0db8155410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.376+0000 7f0dbced4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db80c1ce0 con 0x7f0db8155950 2024-03-06T07:21:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.376+0000 7f0dbced4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db80c1fc0 con 0x7f0db8071df0 2024-03-06T07:21:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.376+0000 7f0dbced4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80c2b20 con 0x7f0db8155030 2024-03-06T07:21:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.376+0000 7f0db6d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0db8155950 0x7f0db806e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57648/0 (socket says 172.21.15.1:57648) 2024-03-06T07:21:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.376+0000 7f0db6d9d640 1 -- 172.21.15.1:0/3983110581 learned_addr learned my addr 172.21.15.1:0/3983110581 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db659c640 1 --2- 172.21.15.1:0/3983110581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db8155030 0x7f0db8155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db5d9b640 1 -- 172.21.15.1:0/3983110581 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788611303 0 0) 0x7f0db80c1ce0 con 0x7f0db8155950 2024-03-06T07:21:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db5d9b640 1 -- 172.21.15.1:0/3983110581 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0da4005730 con 0x7f0db8155950 2024-03-06T07:21:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db5d9b640 1 -- 172.21.15.1:0/3983110581 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54167438 0 0) 0x7f0db80c1fc0 con 0x7f0db8071df0 2024-03-06T07:21:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db5d9b640 1 -- 172.21.15.1:0/3983110581 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db80c1ce0 con 0x7f0db8071df0 2024-03-06T07:21:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db5d9b640 1 -- 172.21.15.1:0/3983110581 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 95056333 0 0) 0x7f0da4005730 con 0x7f0db8155950 2024-03-06T07:21:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db5d9b640 1 -- 172.21.15.1:0/3983110581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db8155030 msgr2=0x7f0db8155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db5d9b640 1 --2- 172.21.15.1:0/3983110581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db8155030 0x7f0db8155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db5d9b640 1 -- 172.21.15.1:0/3983110581 >> v1:172.21.15.98:6789/0 conn(0x7f0db8071df0 legacy=0x7f0db814eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db5d9b640 1 -- 172.21.15.1:0/3983110581 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db80c2e00 con 0x7f0db8155950 2024-03-06T07:21:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db5d9b640 1 -- 172.21.15.1:0/3983110581 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f0db00019f0 con 0x7f0db8155950 2024-03-06T07:21:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db5d9b640 1 -- 172.21.15.1:0/3983110581 >> v1:172.21.15.1:6789/0 conn(0x7f0db8155950 legacy=0x7f0db806e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db5d9b640 1 --2- 172.21.15.1:0/3983110581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0da4005d80 0x7f0da4006180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db5d9b640 1 --2- 172.21.15.1:0/3983110581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0da4006a10 0x7f0da4006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db5d9b640 1 --2- 172.21.15.1:0/3983110581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0da40073d0 0x7f0da400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db5d9b640 1 -- 172.21.15.1:0/3983110581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80c2b20 con 0x7f0da40073d0 2024-03-06T07:21:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db5d9b640 1 -- 172.21.15.1:0/3983110581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da400e240 con 0x7f0da4006a10 2024-03-06T07:21:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db5d9b640 1 -- 172.21.15.1:0/3983110581 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0da400e440 con 0x7f0da4005d80 2024-03-06T07:21:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db659c640 1 --2- 172.21.15.1:0/3983110581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db8155030 0x7f0db8155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db6d9d640 1 --2- 172.21.15.1:0/3983110581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0da40073d0 0x7f0da400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0daffff640 1 --2- 172.21.15.1:0/3983110581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0da4006a10 0x7f0da4006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.377+0000 7f0db659c640 1 --2- 172.21.15.1:0/3983110581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0da4005d80 0x7f0da4006180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.378+0000 7f0db6d9d640 1 -- 172.21.15.1:0/3983110581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0da4005d80 msgr2=0x7f0da4006180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.378+0000 7f0db6d9d640 1 --2- 172.21.15.1:0/3983110581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0da4005d80 0x7f0da4006180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.378+0000 7f0db6d9d640 1 -- 172.21.15.1:0/3983110581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0da4006a10 msgr2=0x7f0da4006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.378+0000 7f0db6d9d640 1 --2- 172.21.15.1:0/3983110581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0da4006a10 0x7f0da4006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.378+0000 7f0db6d9d640 1 -- 172.21.15.1:0/3983110581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da4005730 con 0x7f0da40073d0 2024-03-06T07:21:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.378+0000 7f0db6d9d640 1 --2- 172.21.15.1:0/3983110581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0da40073d0 0x7f0da400dbd0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0db000a7c0 tx=0x7f0da4007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5662d79c0ce8864d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.378+0000 7f0db5d9b640 1 -- 172.21.15.1:0/3983110581 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db000a410 con 0x7f0da40073d0 2024-03-06T07:21:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.378+0000 7f0db5d9b640 1 -- 172.21.15.1:0/3983110581 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db0034650 con 0x7f0da40073d0 2024-03-06T07:21:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.378+0000 7f0db5d9b640 1 -- 172.21.15.1:0/3983110581 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db0034a80 con 0x7f0da40073d0 2024-03-06T07:21:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.379+0000 7f0dbced4640 1 -- 172.21.15.1:0/3983110581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0da40073d0 msgr2=0x7f0da400dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.379+0000 7f0dbced4640 1 --2- 172.21.15.1:0/3983110581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0da40073d0 0x7f0da400dbd0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0db000a7c0 tx=0x7f0da4007a40 comp rx=0 tx=0).stop 2024-03-06T07:21:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.379+0000 7f0db6d9d640 1 -- 172.21.15.1:0/3983110581 reap_dead start 2024-03-06T07:21:55.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.379+0000 7f0dbced4640 1 -- 172.21.15.1:0/3983110581 shutdown_connections 2024-03-06T07:21:55.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.379+0000 7f0dbced4640 1 -- 172.21.15.1:0/3983110581 >> 172.21.15.1:0/3983110581 conn(0x7f0db80a8e10 msgr2=0x7f0db80c3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.379+0000 7f0dbced4640 1 -- 172.21.15.1:0/3983110581 shutdown_connections 2024-03-06T07:21:55.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.379+0000 7f0dbced4640 1 -- 172.21.15.1:0/3983110581 wait complete. 2024-03-06T07:21:55.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.380+0000 7f0dbced4640 1 Processor -- start 2024-03-06T07:21:55.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.380+0000 7f0dbced4640 1 -- start start 2024-03-06T07:21:55.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.381+0000 7f0dbced4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db814cb80 0x7f0db81ea840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.381+0000 7f0dbced4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0db8147bc0 0x7f0db814cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.381+0000 7f0dbced4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db8141650 0x7f0db8141ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.381+0000 7f0dbced4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db814f530 con 0x7f0db8141650 2024-03-06T07:21:55.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.381+0000 7f0dbced4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db814f3b0 con 0x7f0db8147bc0 2024-03-06T07:21:55.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.381+0000 7f0dbced4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db806e9f0 con 0x7f0db814cb80 2024-03-06T07:21:55.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.381+0000 7f0db6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db8141650 0x7f0db8141ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.381+0000 7f0db6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db8141650 0x7f0db8141ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34862/0 (socket says 172.21.15.1:34862) 2024-03-06T07:21:55.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.381+0000 7f0db6d9d640 1 -- 172.21.15.1:0/3496393347 learned_addr learned my addr 172.21.15.1:0/3496393347 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:55.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.381+0000 7f0daffff640 1 --2- 172.21.15.1:0/3496393347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0db8147bc0 0x7f0db814cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.381+0000 7f0db659c640 1 --2- 172.21.15.1:0/3496393347 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db814cb80 0x7f0db81ea840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.381+0000 7f0db6d9d640 1 -- 172.21.15.1:0/3496393347 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db814cb80 msgr2=0x7f0db81ea840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.381+0000 7f0db6d9d640 1 --2- 172.21.15.1:0/3496393347 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db814cb80 0x7f0db81ea840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.381+0000 7f0db6d9d640 1 -- 172.21.15.1:0/3496393347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0db8147bc0 msgr2=0x7f0db814cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.381+0000 7f0db6d9d640 1 --2- 172.21.15.1:0/3496393347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0db8147bc0 0x7f0db814cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.381+0000 7f0db6d9d640 1 -- 172.21.15.1:0/3496393347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db8149240 con 0x7f0db8141650 2024-03-06T07:21:55.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.382+0000 7f0db6d9d640 1 --2- 172.21.15.1:0/3496393347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db8141650 0x7f0db8141ad0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f0db000a8e0 tx=0x7f0db80c1eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.382+0000 7f0daeffd640 1 -- 172.21.15.1:0/3496393347 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db00019f0 con 0x7f0db8141650 2024-03-06T07:21:55.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.382+0000 7f0daeffd640 1 -- 172.21.15.1:0/3496393347 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db0045070 con 0x7f0db8141650 2024-03-06T07:21:55.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.382+0000 7f0dbced4640 1 -- 172.21.15.1:0/3496393347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db8149520 con 0x7f0db8141650 2024-03-06T07:21:55.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.382+0000 7f0daeffd640 1 -- 172.21.15.1:0/3496393347 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db0034a40 con 0x7f0db8141650 2024-03-06T07:21:55.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.382+0000 7f0dbced4640 1 -- 172.21.15.1:0/3496393347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db80ac710 con 0x7f0db8141650 2024-03-06T07:21:55.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.383+0000 7f0dbced4640 1 -- 172.21.15.1:0/3496393347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db80680b0 con 0x7f0db8141650 2024-03-06T07:21:55.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.383+0000 7f0daeffd640 1 -- 172.21.15.1:0/3496393347 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0db0034be0 con 0x7f0db8141650 2024-03-06T07:21:55.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.384+0000 7f0daeffd640 1 --2- 172.21.15.1:0/3496393347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0d9806b500 0x7f0d9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.384+0000 7f0daeffd640 1 -- 172.21.15.1:0/3496393347 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5835+0+0 (secure 0 0 0) 0x7f0db00ac8b0 con 0x7f0db8141650 2024-03-06T07:21:55.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.384+0000 7f0db659c640 1 --2- 172.21.15.1:0/3496393347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0d9806b500 0x7f0d9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.384+0000 7f0db659c640 1 --2- 172.21.15.1:0/3496393347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0d9806b500 0x7f0d9806d9c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0db80c1460 tx=0x7f0da8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.388+0000 7f0daeffd640 1 -- 172.21.15.1:0/3496393347 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0db007b7c0 con 0x7f0db8141650 2024-03-06T07:21:55.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:55.567+0000 7f0dbced4640 1 -- 172.21.15.1:0/3496393347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0db80c1920 con 0x7f0db8141650 2024-03-06T07:21:56.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.253+0000 7f0daeffd640 1 -- 172.21.15.1:0/3496393347 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v43) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0db00ace20 con 0x7f0db8141650 2024-03-06T07:21:56.255 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-06T07:21:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.257+0000 7f0dbced4640 1 -- 172.21.15.1:0/3496393347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0d9806b500 msgr2=0x7f0d9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.257+0000 7f0dbced4640 1 --2- 172.21.15.1:0/3496393347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0d9806b500 0x7f0d9806d9c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0db80c1460 tx=0x7f0da8008040 comp rx=0 tx=0).stop 2024-03-06T07:21:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.257+0000 7f0dbced4640 1 -- 172.21.15.1:0/3496393347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db8141650 msgr2=0x7f0db8141ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.257+0000 7f0dbced4640 1 --2- 172.21.15.1:0/3496393347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db8141650 0x7f0db8141ad0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f0db000a8e0 tx=0x7f0db80c1eb0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.257+0000 7f0dbced4640 1 -- 172.21.15.1:0/3496393347 shutdown_connections 2024-03-06T07:21:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.257+0000 7f0dbced4640 1 --2- 172.21.15.1:0/3496393347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0d9806b500 0x7f0d9806d9c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.257+0000 7f0dbced4640 1 --2- 172.21.15.1:0/3496393347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db8141650 0x7f0db8141ad0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.257+0000 7f0dbced4640 1 --2- 172.21.15.1:0/3496393347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0db8147bc0 0x7f0db814cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.257+0000 7f0dbced4640 1 --2- 172.21.15.1:0/3496393347 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db814cb80 0x7f0db81ea840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.257+0000 7f0dbced4640 1 -- 172.21.15.1:0/3496393347 >> 172.21.15.1:0/3496393347 conn(0x7f0db80a8e10 msgr2=0x7f0db80c5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.257+0000 7f0dbced4640 1 -- 172.21.15.1:0/3496393347 shutdown_connections 2024-03-06T07:21:56.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.257+0000 7f0dbced4640 1 -- 172.21.15.1:0/3496393347 wait complete. 2024-03-06T07:21:56.277 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:21:56.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.384+0000 7ff834f3e640 1 Processor -- start 2024-03-06T07:21:56.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.384+0000 7ff834f3e640 1 -- start start 2024-03-06T07:21:56.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.385+0000 7ff834f3e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff8300c4dd0 0x7ff8300c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.385+0000 7ff834f3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8300c79c0 0x7ff830152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.385+0000 7ff834f3e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8301566e0 con 0x7ff8300c5d80 2024-03-06T07:21:56.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.385+0000 7ff834f3e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff830156490 con 0x7ff8300c4dd0 2024-03-06T07:21:56.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.385+0000 7ff834f3e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff830157010 con 0x7ff8300c79c0 2024-03-06T07:21:56.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.385+0000 7ff82e59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff8300c4dd0 0x7ff8300c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.385+0000 7ff82e59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff8300c4dd0 0x7ff8300c5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34864/0 (socket says 172.21.15.1:34864) 2024-03-06T07:21:56.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.385+0000 7ff82e59c640 1 -- 172.21.15.1:0/458931323 learned_addr learned my addr 172.21.15.1:0/458931323 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:56.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.385+0000 7ff82ed9d640 1 --1- 172.21.15.1:0/458931323 >> v1:172.21.15.1:6789/0 conn(0x7ff8300c5d80 0x7ff8301533e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57658/0 (socket says 172.21.15.1:57658) 2024-03-06T07:21:56.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.385+0000 7ff82dd9b640 1 --2- 172.21.15.1:0/458931323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8300c79c0 0x7ff830152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.385+0000 7ff82e59c640 1 -- 172.21.15.1:0/458931323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8300c79c0 msgr2=0x7ff830152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.385+0000 7ff82e59c640 1 --2- 172.21.15.1:0/458931323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8300c79c0 0x7ff830152ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.385+0000 7ff82e59c640 1 -- 172.21.15.1:0/458931323 >> v1:172.21.15.1:6789/0 conn(0x7ff8300c5d80 legacy=0x7ff8301533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.385+0000 7ff82e59c640 1 -- 172.21.15.1:0/458931323 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8301569c0 con 0x7ff8300c4dd0 2024-03-06T07:21:56.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.385+0000 7ff82d59a640 1 -- 172.21.15.1:0/458931323 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211819051 0 0) 0x7ff8301566e0 con 0x7ff8300c5d80 2024-03-06T07:21:56.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.385+0000 7ff82dd9b640 1 --2- 172.21.15.1:0/458931323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8300c79c0 0x7ff830152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.386+0000 7ff82e59c640 1 --2- 172.21.15.1:0/458931323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8300c4dd0 0x7ff8300c5840 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff820009870 tx=0x7ff82002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5be0ed54a8a51d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.386+0000 7ff82d59a640 1 -- 172.21.15.1:0/458931323 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8200317c0 con 0x7ff8300c4dd0 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.386+0000 7ff82d59a640 1 -- 172.21.15.1:0/458931323 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff820037070 con 0x7ff8300c4dd0 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.386+0000 7ff82d59a640 1 -- 172.21.15.1:0/458931323 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff820031b80 con 0x7ff8300c4dd0 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.387+0000 7ff834f3e640 1 -- 172.21.15.1:0/458931323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8300c4dd0 msgr2=0x7ff8300c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.387+0000 7ff834f3e640 1 --2- 172.21.15.1:0/458931323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8300c4dd0 0x7ff8300c5840 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff820009870 tx=0x7ff82002b860 comp rx=0 tx=0).stop 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.387+0000 7ff834f3e640 1 -- 172.21.15.1:0/458931323 shutdown_connections 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.387+0000 7ff834f3e640 1 --2- 172.21.15.1:0/458931323 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8300c79c0 0x7ff830152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.387+0000 7ff834f3e640 1 --2- 172.21.15.1:0/458931323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8300c4dd0 0x7ff8300c5840 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.387+0000 7ff834f3e640 1 -- 172.21.15.1:0/458931323 >> 172.21.15.1:0/458931323 conn(0x7ff83006b410 msgr2=0x7ff83006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.387+0000 7ff834f3e640 1 -- 172.21.15.1:0/458931323 shutdown_connections 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.387+0000 7ff834f3e640 1 -- 172.21.15.1:0/458931323 wait complete. 2024-03-06T07:21:56.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.388+0000 7ff834f3e640 1 Processor -- start 2024-03-06T07:21:56.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.388+0000 7ff834f3e640 1 -- start start 2024-03-06T07:21:56.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff834f3e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8300c4dd0 0x7ff8300ad3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff834f3e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8300c5d80 0x7ff83014f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff834f3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8300bf620 0x7ff83014fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff834f3e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8300715f0 con 0x7ff8300c4dd0 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff834f3e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff830153af0 con 0x7ff8300c5d80 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff834f3e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff830071770 con 0x7ff8300bf620 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff82e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8300c4dd0 0x7ff8300ad3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff82e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8300c4dd0 0x7ff8300ad3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34874/0 (socket says 172.21.15.1:34874) 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff82e59c640 1 -- 172.21.15.1:0/1389799733 learned_addr learned my addr 172.21.15.1:0/1389799733 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff82e59c640 1 -- 172.21.15.1:0/1389799733 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8300bf620 msgr2=0x7ff83014fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff82ed9d640 1 --2- 172.21.15.1:0/1389799733 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8300bf620 0x7ff83014fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff82e59c640 1 --2- 172.21.15.1:0/1389799733 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8300bf620 0x7ff83014fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff82e59c640 1 -- 172.21.15.1:0/1389799733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8300c5d80 msgr2=0x7ff83014f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff82e59c640 1 --2- 172.21.15.1:0/1389799733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8300c5d80 0x7ff83014f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff82e59c640 1 -- 172.21.15.1:0/1389799733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8301e8210 con 0x7ff8300c4dd0 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff82e59c640 1 --2- 172.21.15.1:0/1389799733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8300c4dd0 0x7ff8300ad3c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff82002b8a0 tx=0x7ff8200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff81effd640 1 -- 172.21.15.1:0/1389799733 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff820032860 con 0x7ff8300c4dd0 2024-03-06T07:21:56.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff81effd640 1 -- 172.21.15.1:0/1389799733 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff820032a00 con 0x7ff8300c4dd0 2024-03-06T07:21:56.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff834f3e640 1 -- 172.21.15.1:0/1389799733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff830150430 con 0x7ff8300c4dd0 2024-03-06T07:21:56.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.389+0000 7ff81effd640 1 -- 172.21.15.1:0/1389799733 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff820032df0 con 0x7ff8300c4dd0 2024-03-06T07:21:56.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.390+0000 7ff834f3e640 1 -- 172.21.15.1:0/1389799733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8301e87b0 con 0x7ff8300c4dd0 2024-03-06T07:21:56.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.390+0000 7ff834f3e640 1 -- 172.21.15.1:0/1389799733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8301e9150 con 0x7ff8300c4dd0 2024-03-06T07:21:56.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.391+0000 7ff81effd640 1 -- 172.21.15.1:0/1389799733 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff82004c060 con 0x7ff8300c4dd0 2024-03-06T07:21:56.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.391+0000 7ff81effd640 1 --2- 172.21.15.1:0/1389799733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff81006b4d0 0x7ff81006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.391+0000 7ff81effd640 1 -- 172.21.15.1:0/1389799733 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5421+0+0 (secure 0 0 0) 0x7ff8200abe70 con 0x7ff8300c4dd0 2024-03-06T07:21:56.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.392+0000 7ff82dd9b640 1 --2- 172.21.15.1:0/1389799733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff81006b4d0 0x7ff81006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.392+0000 7ff82dd9b640 1 --2- 172.21.15.1:0/1389799733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff81006b4d0 0x7ff81006d990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff824006fd0 tx=0x7ff824008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.396+0000 7ff81effd640 1 -- 172.21.15.1:0/1389799733 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8301e9150 con 0x7ff8300c4dd0 2024-03-06T07:21:56.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.579+0000 7ff834f3e640 1 -- 172.21.15.1:0/1389799733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff830067fd0 con 0x7ff8300c4dd0 2024-03-06T07:21:56.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.583+0000 7ff81effd640 1 -- 172.21.15.1:0/1389799733 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff820040070 con 0x7ff8300c4dd0 2024-03-06T07:21:56.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.584+0000 7ff834f3e640 1 -- 172.21.15.1:0/1389799733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff81006b4d0 msgr2=0x7ff81006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.584+0000 7ff834f3e640 1 --2- 172.21.15.1:0/1389799733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff81006b4d0 0x7ff81006d990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff824006fd0 tx=0x7ff824008040 comp rx=0 tx=0).stop 2024-03-06T07:21:56.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.584+0000 7ff834f3e640 1 -- 172.21.15.1:0/1389799733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8300c4dd0 msgr2=0x7ff8300ad3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.584+0000 7ff834f3e640 1 --2- 172.21.15.1:0/1389799733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8300c4dd0 0x7ff8300ad3c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff82002b8a0 tx=0x7ff8200046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.585+0000 7ff834f3e640 1 -- 172.21.15.1:0/1389799733 shutdown_connections 2024-03-06T07:21:56.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.585+0000 7ff834f3e640 1 --2- 172.21.15.1:0/1389799733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff81006b4d0 0x7ff81006d990 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.585+0000 7ff834f3e640 1 --2- 172.21.15.1:0/1389799733 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8300bf620 0x7ff83014fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.585+0000 7ff834f3e640 1 --2- 172.21.15.1:0/1389799733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8300c5d80 0x7ff83014f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.585+0000 7ff834f3e640 1 --2- 172.21.15.1:0/1389799733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8300c4dd0 0x7ff8300ad3c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.585+0000 7ff834f3e640 1 -- 172.21.15.1:0/1389799733 >> 172.21.15.1:0/1389799733 conn(0x7ff83006b410 msgr2=0x7ff8300a9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.585+0000 7ff834f3e640 1 -- 172.21.15.1:0/1389799733 shutdown_connections 2024-03-06T07:21:56.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.585+0000 7ff834f3e640 1 -- 172.21.15.1:0/1389799733 wait complete. 2024-03-06T07:21:56.597 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-03-06T07:21:56.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.698+0000 7f0ffe9bc640 1 Processor -- start 2024-03-06T07:21:56.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.698+0000 7f0ffe9bc640 1 -- start start 2024-03-06T07:21:56.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.699+0000 7f0ffe9bc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ff8152890 0x7f0ff8159030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.699+0000 7f0ffe9bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ff80c8cf0 0x7f0ff80c9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.699+0000 7f0ffe9bc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff8155080 con 0x7f0ff80c7560 2024-03-06T07:21:56.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.699+0000 7f0ffe9bc640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff8155f70 con 0x7f0ff8152890 2024-03-06T07:21:56.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.699+0000 7f0ffe9bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff80c35b0 con 0x7f0ff80c8cf0 2024-03-06T07:21:56.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.699+0000 7f0ff7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ff8152890 0x7f0ff8159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.699+0000 7f0ff77fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0ff80c7560 0x7f0ff8155860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57664/0 (socket says 172.21.15.1:57664) 2024-03-06T07:21:56.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.699+0000 7f0ff77fe640 1 -- 172.21.15.1:0/548543220 learned_addr learned my addr 172.21.15.1:0/548543220 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:56.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.699+0000 7f0ffcf59640 1 --2- 172.21.15.1:0/548543220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ff80c8cf0 0x7f0ff80c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.699+0000 7f0ff7fff640 1 -- 172.21.15.1:0/548543220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ff80c8cf0 msgr2=0x7f0ff80c9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.699+0000 7f0ff7fff640 1 --2- 172.21.15.1:0/548543220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ff80c8cf0 0x7f0ff80c9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.699+0000 7f0ff7fff640 1 -- 172.21.15.1:0/548543220 >> v1:172.21.15.1:6789/0 conn(0x7f0ff80c7560 legacy=0x7f0ff8155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.699+0000 7f0ff7fff640 1 -- 172.21.15.1:0/548543220 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff80c3890 con 0x7f0ff8152890 2024-03-06T07:21:56.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.700+0000 7f0ff7fff640 1 --2- 172.21.15.1:0/548543220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff8152890 0x7f0ff8159030 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0fec0097d0 tx=0x7f0fec02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c6322ec4fbf53a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.700+0000 7f0ff6ffd640 1 -- 172.21.15.1:0/548543220 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec039070 con 0x7f0ff8152890 2024-03-06T07:21:56.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.700+0000 7f0ff6ffd640 1 -- 172.21.15.1:0/548543220 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fec0316f0 con 0x7f0ff8152890 2024-03-06T07:21:56.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.700+0000 7f0ff6ffd640 1 -- 172.21.15.1:0/548543220 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec031ac0 con 0x7f0ff8152890 2024-03-06T07:21:56.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.701+0000 7f0ffe9bc640 1 -- 172.21.15.1:0/548543220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff8152890 msgr2=0x7f0ff8159030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.701+0000 7f0ffe9bc640 1 --2- 172.21.15.1:0/548543220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff8152890 0x7f0ff8159030 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0fec0097d0 tx=0x7f0fec02b790 comp rx=0 tx=0).stop 2024-03-06T07:21:56.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.701+0000 7f0ffe9bc640 1 -- 172.21.15.1:0/548543220 shutdown_connections 2024-03-06T07:21:56.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.701+0000 7f0ffe9bc640 1 --2- 172.21.15.1:0/548543220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ff80c8cf0 0x7f0ff80c9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.701+0000 7f0ffe9bc640 1 --2- 172.21.15.1:0/548543220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff8152890 0x7f0ff8159030 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.701+0000 7f0ffe9bc640 1 -- 172.21.15.1:0/548543220 >> 172.21.15.1:0/548543220 conn(0x7f0ff806b3e0 msgr2=0x7f0ff806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.701+0000 7f0ffe9bc640 1 -- 172.21.15.1:0/548543220 shutdown_connections 2024-03-06T07:21:56.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.701+0000 7f0ffe9bc640 1 -- 172.21.15.1:0/548543220 wait complete. 2024-03-06T07:21:56.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.702+0000 7f0ffe9bc640 1 Processor -- start 2024-03-06T07:21:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.702+0000 7f0ffe9bc640 1 -- start start 2024-03-06T07:21:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.703+0000 7f0ffe9bc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff80c7560 0x7f0ff80a80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.703+0000 7f0ffe9bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ff80c8cf0 0x7f0ff8156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.703+0000 7f0ffe9bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ff8152890 0x7f0ff81572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.703+0000 7f0ffe9bc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff8159570 con 0x7f0ff80c7560 2024-03-06T07:21:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.703+0000 7f0ffe9bc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff8155340 con 0x7f0ff80c8cf0 2024-03-06T07:21:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.703+0000 7f0ffe9bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff80c3260 con 0x7f0ff8152890 2024-03-06T07:21:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.703+0000 7f0ff7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff80c7560 0x7f0ff80a80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.703+0000 7f0ff7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff80c7560 0x7f0ff80a80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34892/0 (socket says 172.21.15.1:34892) 2024-03-06T07:21:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.703+0000 7f0ff7fff640 1 -- 172.21.15.1:0/4144531285 learned_addr learned my addr 172.21.15.1:0/4144531285 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.703+0000 7f0ffcf59640 1 --2- 172.21.15.1:0/4144531285 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ff8152890 0x7f0ff81572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.703+0000 7f0ff7fff640 1 -- 172.21.15.1:0/4144531285 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ff8152890 msgr2=0x7f0ff81572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.703+0000 7f0ff7fff640 1 --2- 172.21.15.1:0/4144531285 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ff8152890 0x7f0ff81572f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.703+0000 7f0ff7fff640 1 -- 172.21.15.1:0/4144531285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ff80c8cf0 msgr2=0x7f0ff8156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.703+0000 7f0ff7fff640 1 --2- 172.21.15.1:0/4144531285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ff80c8cf0 0x7f0ff8156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.703+0000 7f0ff7fff640 1 -- 172.21.15.1:0/4144531285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff8151f20 con 0x7f0ff80c7560 2024-03-06T07:21:56.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.703+0000 7f0ff7fff640 1 --2- 172.21.15.1:0/4144531285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff80c7560 0x7f0ff80a80a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0fec000c00 tx=0x7f0fec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.704+0000 7f0ff4ff9640 1 -- 172.21.15.1:0/4144531285 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec043070 con 0x7f0ff80c7560 2024-03-06T07:21:56.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.704+0000 7f0ff4ff9640 1 -- 172.21.15.1:0/4144531285 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fec034680 con 0x7f0ff80c7560 2024-03-06T07:21:56.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.704+0000 7f0ffe9bc640 1 -- 172.21.15.1:0/4144531285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff8156990 con 0x7f0ff80c7560 2024-03-06T07:21:56.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.704+0000 7f0ff4ff9640 1 -- 172.21.15.1:0/4144531285 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec039040 con 0x7f0ff80c7560 2024-03-06T07:21:56.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.704+0000 7f0ffe9bc640 1 -- 172.21.15.1:0/4144531285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff81ebd80 con 0x7f0ff80c7560 2024-03-06T07:21:56.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.705+0000 7f0ffe9bc640 1 -- 172.21.15.1:0/4144531285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff8067fd0 con 0x7f0ff80c7560 2024-03-06T07:21:56.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.705+0000 7f0ff4ff9640 1 -- 172.21.15.1:0/4144531285 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0fec03c070 con 0x7f0ff80c7560 2024-03-06T07:21:56.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.705+0000 7f0ff4ff9640 1 --2- 172.21.15.1:0/4144531285 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fe406b4d0 0x7f0fe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.705+0000 7f0ff4ff9640 1 -- 172.21.15.1:0/4144531285 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f0fec0ab9d0 con 0x7f0ff80c7560 2024-03-06T07:21:56.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.706+0000 7f0ff77fe640 1 --2- 172.21.15.1:0/4144531285 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fe406b4d0 0x7f0fe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.706+0000 7f0ff77fe640 1 --2- 172.21.15.1:0/4144531285 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fe406b4d0 0x7f0fe406d990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8000f60 tx=0x7f0fe8007a80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.710+0000 7f0ff4ff9640 1 -- 172.21.15.1:0/4144531285 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ff8151f20 con 0x7f0ff80c7560 2024-03-06T07:21:56.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.892+0000 7f0ffe9bc640 1 -- 172.21.15.1:0/4144531285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7f0ff80c2d80 con 0x7f0ff80c7560 2024-03-06T07:21:56.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.893+0000 7f0ff4ff9640 1 -- 172.21.15.1:0/4144531285 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v14) v1 ==== 96+0+0 (secure 0 0 0) 0x7f0fec07ab20 con 0x7f0ff80c7560 2024-03-06T07:21:56.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.895+0000 7f0ffe9bc640 1 -- 172.21.15.1:0/4144531285 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fe406b4d0 msgr2=0x7f0fe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.895+0000 7f0ffe9bc640 1 --2- 172.21.15.1:0/4144531285 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fe406b4d0 0x7f0fe406d990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8000f60 tx=0x7f0fe8007a80 comp rx=0 tx=0).stop 2024-03-06T07:21:56.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.895+0000 7f0ffe9bc640 1 -- 172.21.15.1:0/4144531285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff80c7560 msgr2=0x7f0ff80a80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.895+0000 7f0ffe9bc640 1 --2- 172.21.15.1:0/4144531285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff80c7560 0x7f0ff80a80a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0fec000c00 tx=0x7f0fec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.895+0000 7f0ffe9bc640 1 -- 172.21.15.1:0/4144531285 shutdown_connections 2024-03-06T07:21:56.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.895+0000 7f0ffe9bc640 1 --2- 172.21.15.1:0/4144531285 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fe406b4d0 0x7f0fe406d990 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.895+0000 7f0ffe9bc640 1 --2- 172.21.15.1:0/4144531285 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ff8152890 0x7f0ff81572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.895+0000 7f0ffe9bc640 1 --2- 172.21.15.1:0/4144531285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ff80c8cf0 0x7f0ff8156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.895+0000 7f0ffe9bc640 1 --2- 172.21.15.1:0/4144531285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff80c7560 0x7f0ff80a80a0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.895+0000 7f0ffe9bc640 1 -- 172.21.15.1:0/4144531285 >> 172.21.15.1:0/4144531285 conn(0x7f0ff806b3e0 msgr2=0x7f0ff81e17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.895+0000 7f0ffe9bc640 1 -- 172.21.15.1:0/4144531285 shutdown_connections 2024-03-06T07:21:56.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:56.895+0000 7f0ffe9bc640 1 -- 172.21.15.1:0/4144531285 wait complete. 2024-03-06T07:21:56.908 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop' 2024-03-06T07:21:57.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.007+0000 7f3c98e15640 1 Processor -- start 2024-03-06T07:21:57.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.007+0000 7f3c98e15640 1 -- start start 2024-03-06T07:21:57.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.008+0000 7f3c98e15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c940c2b30 0x7f3c940724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.008+0000 7f3c98e15640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3c94152b70 0x7f3c94072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.008+0000 7f3c98e15640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c94151dd0 con 0x7f3c940a5fa0 2024-03-06T07:21:57.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.008+0000 7f3c98e15640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c941526e0 con 0x7f3c94152b70 2024-03-06T07:21:57.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.008+0000 7f3c98e15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c94156190 con 0x7f3c940c2b30 2024-03-06T07:21:57.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.008+0000 7f3c92d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3c940a5fa0 0x7f3c940c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57680/0 (socket says 172.21.15.1:57680) 2024-03-06T07:21:57.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.008+0000 7f3c92d9d640 1 -- 172.21.15.1:0/957845492 learned_addr learned my addr 172.21.15.1:0/957845492 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:57.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.008+0000 7f3c9259c640 1 --2- 172.21.15.1:0/957845492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c940c2b30 0x7f3c940724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.008+0000 7f3c91d9b640 1 --2- 172.21.15.1:0/957845492 >> v2:172.21.15.98:3300/0 conn(0x7f3c94152b70 0x7f3c94072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.008+0000 7f3c9159a640 1 -- 172.21.15.1:0/957845492 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039205910 0 0) 0x7f3c94151dd0 con 0x7f3c940a5fa0 2024-03-06T07:21:57.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.008+0000 7f3c9159a640 1 -- 172.21.15.1:0/957845492 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c80003520 con 0x7f3c940a5fa0 2024-03-06T07:21:57.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.008+0000 7f3c9159a640 1 -- 172.21.15.1:0/957845492 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1236963344 0 0) 0x7f3c80003520 con 0x7f3c940a5fa0 2024-03-06T07:21:57.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c9159a640 1 -- 172.21.15.1:0/957845492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c940c2b30 msgr2=0x7f3c940724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:57.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c9159a640 1 --2- 172.21.15.1:0/957845492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c940c2b30 0x7f3c940724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c9159a640 1 -- 172.21.15.1:0/957845492 >> v2:172.21.15.98:3300/0 conn(0x7f3c94152b70 msgr2=0x7f3c94072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:57.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c9159a640 1 --2- 172.21.15.1:0/957845492 >> v2:172.21.15.98:3300/0 conn(0x7f3c94152b70 0x7f3c94072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c9159a640 1 -- 172.21.15.1:0/957845492 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c94152090 con 0x7f3c940a5fa0 2024-03-06T07:21:57.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c9159a640 1 -- 172.21.15.1:0/957845492 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f3c840019f0 con 0x7f3c940a5fa0 2024-03-06T07:21:57.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c9159a640 1 -- 172.21.15.1:0/957845492 >> v1:172.21.15.1:6789/0 conn(0x7f3c940a5fa0 legacy=0x7f3c940c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c9159a640 1 --2- 172.21.15.1:0/957845492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c800039b0 0x7f3c80003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c9159a640 1 --2- 172.21.15.1:0/957845492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c80004800 0x7f3c80004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c9159a640 1 --2- 172.21.15.1:0/957845492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c940c2b30 0x7f3c800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c9159a640 1 -- 172.21.15.1:0/957845492 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c941526e0 con 0x7f3c80004800 2024-03-06T07:21:57.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c9159a640 1 -- 172.21.15.1:0/957845492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c94156190 con 0x7f3c800039b0 2024-03-06T07:21:57.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c9159a640 1 -- 172.21.15.1:0/957845492 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8000b9c0 con 0x7f3c940c2b30 2024-03-06T07:21:57.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c91d9b640 1 --2- 172.21.15.1:0/957845492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c80004800 0x7f3c80004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c9259c640 1 --2- 172.21.15.1:0/957845492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c800039b0 0x7f3c80003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c92d9d640 1 --2- 172.21.15.1:0/957845492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c940c2b30 0x7f3c800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c91d9b640 1 -- 172.21.15.1:0/957845492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c940c2b30 msgr2=0x7f3c800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:57.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c91d9b640 1 --2- 172.21.15.1:0/957845492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c940c2b30 0x7f3c800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c91d9b640 1 -- 172.21.15.1:0/957845492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c800039b0 msgr2=0x7f3c80003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:57.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c91d9b640 1 --2- 172.21.15.1:0/957845492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c800039b0 0x7f3c80003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.009+0000 7f3c91d9b640 1 -- 172.21.15.1:0/957845492 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c80003520 con 0x7f3c80004800 2024-03-06T07:21:57.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.010+0000 7f3c91d9b640 1 --2- 172.21.15.1:0/957845492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c80004800 0x7f3c80004c80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009ac0 tx=0x7f3c80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a513c5fed625976 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.010+0000 7f3c9159a640 1 -- 172.21.15.1:0/957845492 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c039070 con 0x7f3c80004800 2024-03-06T07:21:57.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.010+0000 7f3c9159a640 1 -- 172.21.15.1:0/957845492 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7c02c710 con 0x7f3c80004800 2024-03-06T07:21:57.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.010+0000 7f3c9159a640 1 -- 172.21.15.1:0/957845492 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c009690 con 0x7f3c80004800 2024-03-06T07:21:57.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.010+0000 7f3c98e15640 1 -- 172.21.15.1:0/957845492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c80004800 msgr2=0x7f3c80004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.010+0000 7f3c98e15640 1 --2- 172.21.15.1:0/957845492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c80004800 0x7f3c80004c80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009ac0 tx=0x7f3c80005830 comp rx=0 tx=0).stop 2024-03-06T07:21:57.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.010+0000 7f3c92d9d640 1 -- 172.21.15.1:0/957845492 reap_dead start 2024-03-06T07:21:57.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.011+0000 7f3c98e15640 1 -- 172.21.15.1:0/957845492 shutdown_connections 2024-03-06T07:21:57.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.011+0000 7f3c98e15640 1 -- 172.21.15.1:0/957845492 >> 172.21.15.1:0/957845492 conn(0x7f3c940c6860 msgr2=0x7f3c940c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.011+0000 7f3c98e15640 1 -- 172.21.15.1:0/957845492 shutdown_connections 2024-03-06T07:21:57.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.011+0000 7f3c98e15640 1 -- 172.21.15.1:0/957845492 wait complete. 2024-03-06T07:21:57.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.012+0000 7f3c98e15640 1 Processor -- start 2024-03-06T07:21:57.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.012+0000 7f3c98e15640 1 -- start start 2024-03-06T07:21:57.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.012+0000 7f3c98e15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c80004800 0x7f3c941ee430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.012+0000 7f3c98e15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c9414df10 0x7f3c94151520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.012+0000 7f3c98e15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c94151a60 0x7f3c9414a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.012+0000 7f3c98e15640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c94155f00 con 0x7f3c94151a60 2024-03-06T07:21:57.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.012+0000 7f3c98e15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c94155d80 con 0x7f3c9414df10 2024-03-06T07:21:57.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.012+0000 7f3c98e15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c940724c0 con 0x7f3c80004800 2024-03-06T07:21:57.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.013+0000 7f3c92d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c94151a60 0x7f3c9414a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.013+0000 7f3c92d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c94151a60 0x7f3c9414a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34924/0 (socket says 172.21.15.1:34924) 2024-03-06T07:21:57.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.013+0000 7f3c92d9d640 1 -- 172.21.15.1:0/2328966377 learned_addr learned my addr 172.21.15.1:0/2328966377 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:57.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.013+0000 7f3c9259c640 1 --2- 172.21.15.1:0/2328966377 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c80004800 0x7f3c941ee430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.013+0000 7f3c91d9b640 1 --2- 172.21.15.1:0/2328966377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c9414df10 0x7f3c94151520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.013+0000 7f3c92d9d640 1 -- 172.21.15.1:0/2328966377 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c80004800 msgr2=0x7f3c941ee430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.013+0000 7f3c92d9d640 1 --2- 172.21.15.1:0/2328966377 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c80004800 0x7f3c941ee430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.013+0000 7f3c92d9d640 1 -- 172.21.15.1:0/2328966377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c9414df10 msgr2=0x7f3c94151520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.013+0000 7f3c92d9d640 1 --2- 172.21.15.1:0/2328966377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c9414df10 0x7f3c94151520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.013+0000 7f3c92d9d640 1 -- 172.21.15.1:0/2328966377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c94150450 con 0x7f3c94151a60 2024-03-06T07:21:57.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.013+0000 7f3c92d9d640 1 --2- 172.21.15.1:0/2328966377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c94151a60 0x7f3c9414a150 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3c940a5f30 tx=0x7f3c84007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.013+0000 7f3c8affd640 1 -- 172.21.15.1:0/2328966377 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8400ac40 con 0x7f3c94151a60 2024-03-06T07:21:57.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.013+0000 7f3c8affd640 1 -- 172.21.15.1:0/2328966377 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c84018070 con 0x7f3c94151a60 2024-03-06T07:21:57.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.013+0000 7f3c98e15640 1 -- 172.21.15.1:0/2328966377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c94150730 con 0x7f3c94151a60 2024-03-06T07:21:57.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.013+0000 7f3c8affd640 1 -- 172.21.15.1:0/2328966377 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c840133f0 con 0x7f3c94151a60 2024-03-06T07:21:57.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.013+0000 7f3c98e15640 1 -- 172.21.15.1:0/2328966377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c94150cf0 con 0x7f3c94151a60 2024-03-06T07:21:57.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.014+0000 7f3c98e15640 1 -- 172.21.15.1:0/2328966377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c94067fd0 con 0x7f3c94151a60 2024-03-06T07:21:57.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.015+0000 7f3c8affd640 1 -- 172.21.15.1:0/2328966377 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c84013590 con 0x7f3c94151a60 2024-03-06T07:21:57.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.015+0000 7f3c8affd640 1 --2- 172.21.15.1:0/2328966377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c7406b500 0x7f3c7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.015+0000 7f3c8affd640 1 -- 172.21.15.1:0/2328966377 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f3c8408a640 con 0x7f3c94151a60 2024-03-06T07:21:57.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.015+0000 7f3c9259c640 1 --2- 172.21.15.1:0/2328966377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c7406b500 0x7f3c7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.016+0000 7f3c9259c640 1 --2- 172.21.15.1:0/2328966377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c7406b500 0x7f3c7406d9c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c009a60 tx=0x7f3c8c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.019+0000 7f3c8affd640 1 -- 172.21.15.1:0/2328966377 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c84059670 con 0x7f3c94151a60 2024-03-06T07:21:57.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:57.273+0000 7f3c98e15640 1 -- 172.21.15.1:0/2328966377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]} v 0) v1 -- 0x7f3c940c5770 con 0x7f3c94151a60 2024-03-06T07:21:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.256+0000 7f3c8affd640 1 -- 172.21.15.1:0/2328966377 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]}]=0 v108) v1 ==== 143+0+0 (secure 0 0 0) 0x7f3c84058f70 con 0x7f3c94151a60 2024-03-06T07:21:58.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.258+0000 7f3c98e15640 1 -- 172.21.15.1:0/2328966377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c7406b500 msgr2=0x7f3c7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.258+0000 7f3c98e15640 1 --2- 172.21.15.1:0/2328966377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c7406b500 0x7f3c7406d9c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c009a60 tx=0x7f3c8c009190 comp rx=0 tx=0).stop 2024-03-06T07:21:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.258+0000 7f3c98e15640 1 -- 172.21.15.1:0/2328966377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c94151a60 msgr2=0x7f3c9414a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.258+0000 7f3c98e15640 1 --2- 172.21.15.1:0/2328966377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c94151a60 0x7f3c9414a150 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3c940a5f30 tx=0x7f3c84007b10 comp rx=0 tx=0).stop 2024-03-06T07:21:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.258+0000 7f3c98e15640 1 -- 172.21.15.1:0/2328966377 shutdown_connections 2024-03-06T07:21:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.258+0000 7f3c98e15640 1 --2- 172.21.15.1:0/2328966377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c7406b500 0x7f3c7406d9c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.258+0000 7f3c98e15640 1 --2- 172.21.15.1:0/2328966377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c94151a60 0x7f3c9414a150 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.258+0000 7f3c98e15640 1 --2- 172.21.15.1:0/2328966377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c9414df10 0x7f3c94151520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.258+0000 7f3c98e15640 1 --2- 172.21.15.1:0/2328966377 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c80004800 0x7f3c941ee430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.258+0000 7f3c98e15640 1 -- 172.21.15.1:0/2328966377 >> 172.21.15.1:0/2328966377 conn(0x7f3c940c6860 msgr2=0x7f3c940c4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.259+0000 7f3c98e15640 1 -- 172.21.15.1:0/2328966377 shutdown_connections 2024-03-06T07:21:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.259+0000 7f3c98e15640 1 -- 172.21.15.1:0/2328966377 wait complete. 2024-03-06T07:21:58.274 INFO:tasks.cephfs_test_runner:That unsetting mds_join_fs will not cause failover if no high-affinity standby exists. ... ok 2024-03-06T07:21:58.274 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_last_resort (tasks.cephfs.test_failover.TestClusterAffinity) That a standby with mds_join_fs set to another fs is still used if necessary. 2024-03-06T07:21:58.274 INFO:tasks.cephfs_test_runner:test_join_fs_last_resort (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-06T07:21:58.274 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:58.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.380+0000 7f62fa965640 1 Processor -- start 2024-03-06T07:21:58.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.380+0000 7f62fa965640 1 -- start start 2024-03-06T07:21:58.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.381+0000 7f62fa965640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f62f4150520 0x7f62f41543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.381+0000 7f62fa965640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62f40c32f0 0x7f62f41537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.381+0000 7f62fa965640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f4150900 con 0x7f62f4070c80 2024-03-06T07:21:58.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.381+0000 7f62fa965640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62f4071790 con 0x7f62f4150520 2024-03-06T07:21:58.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.381+0000 7f62fa965640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f4157050 con 0x7f62f40c32f0 2024-03-06T07:21:58.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.381+0000 7f62f3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f62f4150520 0x7f62f41543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.381+0000 7f62f3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f62f4150520 0x7f62f41543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34926/0 (socket says 172.21.15.1:34926) 2024-03-06T07:21:58.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.381+0000 7f62f3fff640 1 -- 172.21.15.1:0/1808909473 learned_addr learned my addr 172.21.15.1:0/1808909473 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:58.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.381+0000 7f62f37fe640 1 --2- 172.21.15.1:0/1808909473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62f40c32f0 0x7f62f41537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.381+0000 7f62f3fff640 1 -- 172.21.15.1:0/1808909473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62f40c32f0 msgr2=0x7f62f41537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.381+0000 7f62f3fff640 1 --2- 172.21.15.1:0/1808909473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62f40c32f0 0x7f62f41537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.381+0000 7f62f3fff640 1 -- 172.21.15.1:0/1808909473 >> v1:172.21.15.98:6789/0 conn(0x7f62f4070c80 legacy=0x7f62f40c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.381+0000 7f62f3fff640 1 -- 172.21.15.1:0/1808909473 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f4152fc0 con 0x7f62f4150520 2024-03-06T07:21:58.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.381+0000 7f62f37fe640 1 --2- 172.21.15.1:0/1808909473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62f40c32f0 0x7f62f41537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.382+0000 7f62f3fff640 1 --2- 172.21.15.1:0/1808909473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62f4150520 0x7f62f41543a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009870 tx=0x7f62e402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbcfd8e9e617e39a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.382+0000 7f62f2ffd640 1 -- 172.21.15.1:0/1808909473 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e4035020 con 0x7f62f4150520 2024-03-06T07:21:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.382+0000 7f62f2ffd640 1 -- 172.21.15.1:0/1808909473 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e4031640 con 0x7f62f4150520 2024-03-06T07:21:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.382+0000 7f62f2ffd640 1 -- 172.21.15.1:0/1808909473 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e402d040 con 0x7f62f4150520 2024-03-06T07:21:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.383+0000 7f62fa965640 1 -- 172.21.15.1:0/1808909473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62f4150520 msgr2=0x7f62f41543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.383+0000 7f62fa965640 1 --2- 172.21.15.1:0/1808909473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62f4150520 0x7f62f41543a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009870 tx=0x7f62e402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.383+0000 7f62fa965640 1 -- 172.21.15.1:0/1808909473 shutdown_connections 2024-03-06T07:21:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.383+0000 7f62fa965640 1 --2- 172.21.15.1:0/1808909473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62f40c32f0 0x7f62f41537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.383+0000 7f62fa965640 1 --2- 172.21.15.1:0/1808909473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62f4150520 0x7f62f41543a0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.383+0000 7f62fa965640 1 -- 172.21.15.1:0/1808909473 >> 172.21.15.1:0/1808909473 conn(0x7f62f40c88f0 msgr2=0x7f62f40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.383+0000 7f62fa965640 1 -- 172.21.15.1:0/1808909473 shutdown_connections 2024-03-06T07:21:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.383+0000 7f62fa965640 1 -- 172.21.15.1:0/1808909473 wait complete. 2024-03-06T07:21:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.384+0000 7f62fa965640 1 Processor -- start 2024-03-06T07:21:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.384+0000 7f62fa965640 1 -- start start 2024-03-06T07:21:58.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62fa965640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62f4070c80 0x7f62f41ec870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62fa965640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62f40c32f0 0x7f62f40c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62fa965640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62f4150520 0x7f62f4142c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62fa965640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f40c7ec0 con 0x7f62f40c32f0 2024-03-06T07:21:58.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62fa965640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f4157330 con 0x7f62f4150520 2024-03-06T07:21:58.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62fa965640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f4153d00 con 0x7f62f4070c80 2024-03-06T07:21:58.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62f37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62f40c32f0 0x7f62f40c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62f37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62f40c32f0 0x7f62f40c4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34942/0 (socket says 172.21.15.1:34942) 2024-03-06T07:21:58.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62f37fe640 1 -- 172.21.15.1:0/1410042308 learned_addr learned my addr 172.21.15.1:0/1410042308 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:58.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62f3fff640 1 --2- 172.21.15.1:0/1410042308 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62f4070c80 0x7f62f41ec870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62f8f02640 1 --2- 172.21.15.1:0/1410042308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62f4150520 0x7f62f4142c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62f37fe640 1 -- 172.21.15.1:0/1410042308 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62f4070c80 msgr2=0x7f62f41ec870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62f37fe640 1 --2- 172.21.15.1:0/1410042308 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62f4070c80 0x7f62f41ec870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62f37fe640 1 -- 172.21.15.1:0/1410042308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62f4150520 msgr2=0x7f62f4142c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62f37fe640 1 --2- 172.21.15.1:0/1410042308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62f4150520 0x7f62f4142c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62f37fe640 1 -- 172.21.15.1:0/1410042308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f41e8f10 con 0x7f62f40c32f0 2024-03-06T07:21:58.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62f37fe640 1 --2- 172.21.15.1:0/1410042308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62f40c32f0 0x7f62f40c4cd0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f62e800c7f0 tx=0x7f62e800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.385+0000 7f62f0ff9640 1 -- 172.21.15.1:0/1410042308 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e8015070 con 0x7f62f40c32f0 2024-03-06T07:21:58.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.386+0000 7f62f0ff9640 1 -- 172.21.15.1:0/1410042308 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e8007a30 con 0x7f62f40c32f0 2024-03-06T07:21:58.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.386+0000 7f62f0ff9640 1 -- 172.21.15.1:0/1410042308 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e8007e00 con 0x7f62f40c32f0 2024-03-06T07:21:58.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.386+0000 7f62fa965640 1 -- 172.21.15.1:0/1410042308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f4143970 con 0x7f62f40c32f0 2024-03-06T07:21:58.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.386+0000 7f62fa965640 1 -- 172.21.15.1:0/1410042308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f41ec350 con 0x7f62f40c32f0 2024-03-06T07:21:58.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.387+0000 7f62fa965640 1 -- 172.21.15.1:0/1410042308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f40686e0 con 0x7f62f40c32f0 2024-03-06T07:21:58.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.387+0000 7f62f0ff9640 1 -- 172.21.15.1:0/1410042308 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f62e8018070 con 0x7f62f40c32f0 2024-03-06T07:21:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.387+0000 7f62f0ff9640 1 --2- 172.21.15.1:0/1410042308 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f62e006b4d0 0x7f62e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.387+0000 7f62f0ff9640 1 -- 172.21.15.1:0/1410042308 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f62e808a080 con 0x7f62f40c32f0 2024-03-06T07:21:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.388+0000 7f62f3fff640 1 --2- 172.21.15.1:0/1410042308 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f62e006b4d0 0x7f62e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.388+0000 7f62f3fff640 1 --2- 172.21.15.1:0/1410042308 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f62e006b4d0 0x7f62e006d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f62e4036000 tx=0x7f62e403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.392+0000 7f62f0ff9640 1 -- 172.21.15.1:0/1410042308 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f62e8059030 con 0x7f62f40c32f0 2024-03-06T07:21:58.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.573+0000 7f62fa965640 1 -- 172.21.15.1:0/1410042308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62f414bf60 con 0x7f62f40c32f0 2024-03-06T07:21:58.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.574+0000 7f62f0ff9640 1 -- 172.21.15.1:0/1410042308 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+13299 (secure 0 0 0) 0x7f62e8058930 con 0x7f62f40c32f0 2024-03-06T07:21:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:21:58.576 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":43,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:21:56.246962+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/2438105345":"2024-03-07T07:21:53.050556+0000","172.21.15.1:6835/2438105345":"2024-03-07T07:21:53.050556+0000","172.21.15.1:6834/1818141892":"2024-03-07T07:21:37.195540+0000","172.21.15.1:6832/1818141892":"2024-03-07T07:21:37.195540+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.576+0000 7f62fa965640 1 -- 172.21.15.1:0/1410042308 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f62e006b4d0 msgr2=0x7f62e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.576+0000 7f62fa965640 1 --2- 172.21.15.1:0/1410042308 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f62e006b4d0 0x7f62e006d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f62e4036000 tx=0x7f62e403c040 comp rx=0 tx=0).stop 2024-03-06T07:21:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.576+0000 7f62fa965640 1 -- 172.21.15.1:0/1410042308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62f40c32f0 msgr2=0x7f62f40c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.576+0000 7f62fa965640 1 --2- 172.21.15.1:0/1410042308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62f40c32f0 0x7f62f40c4cd0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f62e800c7f0 tx=0x7f62e800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.576+0000 7f62fa965640 1 -- 172.21.15.1:0/1410042308 shutdown_connections 2024-03-06T07:21:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.576+0000 7f62fa965640 1 --2- 172.21.15.1:0/1410042308 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f62e006b4d0 0x7f62e006d990 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.576+0000 7f62fa965640 1 --2- 172.21.15.1:0/1410042308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62f4150520 0x7f62f4142c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.576+0000 7f62fa965640 1 --2- 172.21.15.1:0/1410042308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62f40c32f0 0x7f62f40c4cd0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.576+0000 7f62fa965640 1 --2- 172.21.15.1:0/1410042308 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62f4070c80 0x7f62f41ec870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.576+0000 7f62fa965640 1 -- 172.21.15.1:0/1410042308 >> 172.21.15.1:0/1410042308 conn(0x7f62f40c88f0 msgr2=0x7f62f414a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.576+0000 7f62fa965640 1 -- 172.21.15.1:0/1410042308 shutdown_connections 2024-03-06T07:21:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.576+0000 7f62fa965640 1 -- 172.21.15.1:0/1410042308 wait complete. 2024-03-06T07:21:58.589 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:58.589 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:58.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.691+0000 7f92ceff7640 1 Processor -- start 2024-03-06T07:21:58.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.691+0000 7f92ceff7640 1 -- start start 2024-03-06T07:21:58.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.692+0000 7f92ceff7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f92c80c6730 0x7f92c80c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.692+0000 7f92ceff7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f92c80c5d10 0x7f92c8156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.692+0000 7f92ceff7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c80c3300 con 0x7f92c8152690 2024-03-06T07:21:58.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.692+0000 7f92ceff7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c80c35a0 con 0x7f92c80c5d10 2024-03-06T07:21:58.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.692+0000 7f92ceff7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c80c3880 con 0x7f92c80c6730 2024-03-06T07:21:58.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.692+0000 7f92cd594640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f92c80c5d10 0x7f92c8156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.692+0000 7f92cd594640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f92c80c5d10 0x7f92c8156cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34956/0 (socket says 172.21.15.1:34956) 2024-03-06T07:21:58.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.692+0000 7f92cd594640 1 -- 172.21.15.1:0/604672629 learned_addr learned my addr 172.21.15.1:0/604672629 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:58.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.692+0000 7f92c7fff640 1 --2- 172.21.15.1:0/604672629 >> v2:172.21.15.98:3300/0 conn(0x7f92c80c6730 0x7f92c80c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.692+0000 7f92cd594640 1 -- 172.21.15.1:0/604672629 >> v2:172.21.15.98:3300/0 conn(0x7f92c80c6730 msgr2=0x7f92c80c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.692+0000 7f92cd594640 1 --2- 172.21.15.1:0/604672629 >> v2:172.21.15.98:3300/0 conn(0x7f92c80c6730 0x7f92c80c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.692+0000 7f92cd594640 1 -- 172.21.15.1:0/604672629 >> v1:172.21.15.98:6789/0 conn(0x7f92c8152690 legacy=0x7f92c8155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.692+0000 7f92cd594640 1 -- 172.21.15.1:0/604672629 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c80c3b60 con 0x7f92c80c5d10 2024-03-06T07:21:58.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.693+0000 7f92cd594640 1 --2- 172.21.15.1:0/604672629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92c80c5d10 0x7f92c8156cd0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f92c80c6350 tx=0x7f92c002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4616811fc6c116b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.693+0000 7f92c77fe640 1 -- 172.21.15.1:0/604672629 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c002c9c0 con 0x7f92c80c5d10 2024-03-06T07:21:58.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.693+0000 7f92c77fe640 1 -- 172.21.15.1:0/604672629 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c002cb60 con 0x7f92c80c5d10 2024-03-06T07:21:58.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.693+0000 7f92c77fe640 1 -- 172.21.15.1:0/604672629 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c00343f0 con 0x7f92c80c5d10 2024-03-06T07:21:58.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.694+0000 7f92ceff7640 1 -- 172.21.15.1:0/604672629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92c80c5d10 msgr2=0x7f92c8156cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.694+0000 7f92ceff7640 1 --2- 172.21.15.1:0/604672629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92c80c5d10 0x7f92c8156cd0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f92c80c6350 tx=0x7f92c002be20 comp rx=0 tx=0).stop 2024-03-06T07:21:58.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.694+0000 7f92ceff7640 1 -- 172.21.15.1:0/604672629 shutdown_connections 2024-03-06T07:21:58.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.694+0000 7f92ceff7640 1 --2- 172.21.15.1:0/604672629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92c80c5d10 0x7f92c8156cd0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.694+0000 7f92ceff7640 1 --2- 172.21.15.1:0/604672629 >> v2:172.21.15.98:3300/0 conn(0x7f92c80c6730 0x7f92c80c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.694+0000 7f92ceff7640 1 -- 172.21.15.1:0/604672629 >> 172.21.15.1:0/604672629 conn(0x7f92c806b410 msgr2=0x7f92c806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.694+0000 7f92ceff7640 1 -- 172.21.15.1:0/604672629 shutdown_connections 2024-03-06T07:21:58.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.694+0000 7f92ceff7640 1 -- 172.21.15.1:0/604672629 wait complete. 2024-03-06T07:21:58.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.695+0000 7f92ceff7640 1 Processor -- start 2024-03-06T07:21:58.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.695+0000 7f92ceff7640 1 -- start start 2024-03-06T07:21:58.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.695+0000 7f92ceff7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92c80c5d10 0x7f92c8151160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.695+0000 7f92ceff7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92c80c6730 0x7f92c81edf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.695+0000 7f92ceff7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92c8152690 0x7f92c81ee450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.695+0000 7f92ceff7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c8152bf0 con 0x7f92c80c6730 2024-03-06T07:21:58.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.695+0000 7f92ceff7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c8152a70 con 0x7f92c80c5d10 2024-03-06T07:21:58.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.695+0000 7f92ceff7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c8157210 con 0x7f92c8152690 2024-03-06T07:21:58.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.696+0000 7f92c7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92c80c6730 0x7f92c81edf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.696+0000 7f92c7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92c80c6730 0x7f92c81edf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34970/0 (socket says 172.21.15.1:34970) 2024-03-06T07:21:58.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.696+0000 7f92c7fff640 1 -- 172.21.15.1:0/315362591 learned_addr learned my addr 172.21.15.1:0/315362591 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:58.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.696+0000 7f92ccd93640 1 --2- 172.21.15.1:0/315362591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92c80c5d10 0x7f92c8151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.696+0000 7f92cd594640 1 --2- 172.21.15.1:0/315362591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92c8152690 0x7f92c81ee450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.696+0000 7f92c7fff640 1 -- 172.21.15.1:0/315362591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92c8152690 msgr2=0x7f92c81ee450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.696+0000 7f92c7fff640 1 --2- 172.21.15.1:0/315362591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92c8152690 0x7f92c81ee450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.696+0000 7f92c7fff640 1 -- 172.21.15.1:0/315362591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92c80c5d10 msgr2=0x7f92c8151160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.696+0000 7f92c7fff640 1 --2- 172.21.15.1:0/315362591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92c80c5d10 0x7f92c8151160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.696+0000 7f92c7fff640 1 -- 172.21.15.1:0/315362591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c814ef60 con 0x7f92c80c6730 2024-03-06T07:21:58.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.696+0000 7f92ccd93640 1 --2- 172.21.15.1:0/315362591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92c80c5d10 0x7f92c8151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:58.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.696+0000 7f92c7fff640 1 --2- 172.21.15.1:0/315362591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92c80c6730 0x7f92c81edf10 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f92b400c820 tx=0x7f92b400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.696+0000 7f92c57fa640 1 -- 172.21.15.1:0/315362591 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b4015070 con 0x7f92c80c6730 2024-03-06T07:21:58.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.696+0000 7f92c57fa640 1 -- 172.21.15.1:0/315362591 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b4004240 con 0x7f92c80c6730 2024-03-06T07:21:58.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.696+0000 7f92ceff7640 1 -- 172.21.15.1:0/315362591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c81553c0 con 0x7f92c80c6730 2024-03-06T07:21:58.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.696+0000 7f92c57fa640 1 -- 172.21.15.1:0/315362591 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b400ce10 con 0x7f92c80c6730 2024-03-06T07:21:58.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.696+0000 7f92ceff7640 1 -- 172.21.15.1:0/315362591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c80ad600 con 0x7f92c80c6730 2024-03-06T07:21:58.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.697+0000 7f92ceff7640 1 -- 172.21.15.1:0/315362591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c8067fd0 con 0x7f92c80c6730 2024-03-06T07:21:58.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.698+0000 7f92c57fa640 1 -- 172.21.15.1:0/315362591 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92b4007830 con 0x7f92c80c6730 2024-03-06T07:21:58.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.698+0000 7f92c57fa640 1 --2- 172.21.15.1:0/315362591 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f92ac06b500 0x7f92ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.698+0000 7f92c57fa640 1 -- 172.21.15.1:0/315362591 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f92b4089e60 con 0x7f92c80c6730 2024-03-06T07:21:58.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.698+0000 7f92ccd93640 1 --2- 172.21.15.1:0/315362591 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f92ac06b500 0x7f92ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.699+0000 7f92ccd93640 1 --2- 172.21.15.1:0/315362591 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f92ac06b500 0x7f92ac06d9c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f92bc003eb0 tx=0x7f92bc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.702+0000 7f92c57fa640 1 -- 172.21.15.1:0/315362591 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92b4058e90 con 0x7f92c80c6730 2024-03-06T07:21:58.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.886+0000 7f92ceff7640 1 -- 172.21.15.1:0/315362591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f92c80c2c40 con 0x7f92c80c6730 2024-03-06T07:21:58.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.886+0000 7f92c57fa640 1 -- 172.21.15.1:0/315362591 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) v1 ==== 93+0+10 (secure 0 0 0) 0x7f92b4058790 con 0x7f92c80c6730 2024-03-06T07:21:58.888 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:21:58.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.888+0000 7f92ceff7640 1 -- 172.21.15.1:0/315362591 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f92ac06b500 msgr2=0x7f92ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.888+0000 7f92ceff7640 1 --2- 172.21.15.1:0/315362591 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f92ac06b500 0x7f92ac06d9c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f92bc003eb0 tx=0x7f92bc009040 comp rx=0 tx=0).stop 2024-03-06T07:21:58.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.888+0000 7f92ceff7640 1 -- 172.21.15.1:0/315362591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92c80c6730 msgr2=0x7f92c81edf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.888+0000 7f92ceff7640 1 --2- 172.21.15.1:0/315362591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92c80c6730 0x7f92c81edf10 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f92b400c820 tx=0x7f92b400cce0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.888+0000 7f92ceff7640 1 -- 172.21.15.1:0/315362591 shutdown_connections 2024-03-06T07:21:58.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.888+0000 7f92ceff7640 1 --2- 172.21.15.1:0/315362591 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f92ac06b500 0x7f92ac06d9c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.888+0000 7f92ceff7640 1 --2- 172.21.15.1:0/315362591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92c8152690 0x7f92c81ee450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.888+0000 7f92ceff7640 1 --2- 172.21.15.1:0/315362591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92c80c6730 0x7f92c81edf10 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.888+0000 7f92ceff7640 1 --2- 172.21.15.1:0/315362591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92c80c5d10 0x7f92c8151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.888+0000 7f92ceff7640 1 -- 172.21.15.1:0/315362591 >> 172.21.15.1:0/315362591 conn(0x7f92c806b410 msgr2=0x7f92c8156050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.889+0000 7f92ceff7640 1 -- 172.21.15.1:0/315362591 shutdown_connections 2024-03-06T07:21:58.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:58.889+0000 7f92ceff7640 1 -- 172.21.15.1:0/315362591 wait complete. 2024-03-06T07:21:58.901 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort' 2024-03-06T07:21:59.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.001+0000 7f6f57c6d640 1 Processor -- start 2024-03-06T07:21:59.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.001+0000 7f6f57c6d640 1 -- start start 2024-03-06T07:21:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.002+0000 7f6f57c6d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6f50152730 0x7f6f500c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.002+0000 7f6f57c6d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f500c1aa0 0x7f6f501552e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.002+0000 7f6f57c6d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f501588e0 con 0x7f6f500c6fd0 2024-03-06T07:21:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.002+0000 7f6f57c6d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f50158bc0 con 0x7f6f50152730 2024-03-06T07:21:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.002+0000 7f6f57c6d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f50158ea0 con 0x7f6f500c1aa0 2024-03-06T07:21:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.002+0000 7f6f55a09640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6f50152730 0x7f6f500c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.002+0000 7f6f55208640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6f500c6fd0 0x7f6f500c1390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57692/0 (socket says 172.21.15.1:57692) 2024-03-06T07:21:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.002+0000 7f6f55208640 1 -- 172.21.15.1:0/774925262 learned_addr learned my addr 172.21.15.1:0/774925262 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:21:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.002+0000 7f6f5620a640 1 --2- 172.21.15.1:0/774925262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f500c1aa0 0x7f6f501552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.002+0000 7f6f55a09640 1 -- 172.21.15.1:0/774925262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f500c1aa0 msgr2=0x7f6f501552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.002+0000 7f6f55a09640 1 --2- 172.21.15.1:0/774925262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f500c1aa0 0x7f6f501552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.002+0000 7f6f55a09640 1 -- 172.21.15.1:0/774925262 >> v1:172.21.15.1:6789/0 conn(0x7f6f500c6fd0 legacy=0x7f6f500c1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.002+0000 7f6f54a07640 1 -- 172.21.15.1:0/774925262 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277217986 0 0) 0x7f6f501588e0 con 0x7f6f500c6fd0 2024-03-06T07:21:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.002+0000 7f6f55a09640 1 -- 172.21.15.1:0/774925262 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f50159180 con 0x7f6f50152730 2024-03-06T07:21:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.002+0000 7f6f5620a640 1 --2- 172.21.15.1:0/774925262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f500c1aa0 0x7f6f501552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.003+0000 7f6f55a09640 1 --2- 172.21.15.1:0/774925262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f50152730 0x7f6f500c6a90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c0097a0 tx=0x7f6f4c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4b17dc64ec2188 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.003+0000 7f6f54a07640 1 -- 172.21.15.1:0/774925262 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c038070 con 0x7f6f50152730 2024-03-06T07:21:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.003+0000 7f6f54a07640 1 -- 172.21.15.1:0/774925262 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4c031700 con 0x7f6f50152730 2024-03-06T07:21:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.003+0000 7f6f54a07640 1 -- 172.21.15.1:0/774925262 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c031ad0 con 0x7f6f50152730 2024-03-06T07:21:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.004+0000 7f6f57c6d640 1 -- 172.21.15.1:0/774925262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f50152730 msgr2=0x7f6f500c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.004+0000 7f6f57c6d640 1 --2- 172.21.15.1:0/774925262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f50152730 0x7f6f500c6a90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c0097a0 tx=0x7f6f4c02b7a0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.004+0000 7f6f57c6d640 1 -- 172.21.15.1:0/774925262 shutdown_connections 2024-03-06T07:21:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.004+0000 7f6f57c6d640 1 --2- 172.21.15.1:0/774925262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f500c1aa0 0x7f6f501552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.004+0000 7f6f57c6d640 1 --2- 172.21.15.1:0/774925262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f50152730 0x7f6f500c6a90 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.004+0000 7f6f57c6d640 1 -- 172.21.15.1:0/774925262 >> 172.21.15.1:0/774925262 conn(0x7f6f50066d00 msgr2=0x7f6f50067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.004+0000 7f6f57c6d640 1 -- 172.21.15.1:0/774925262 shutdown_connections 2024-03-06T07:21:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.004+0000 7f6f57c6d640 1 -- 172.21.15.1:0/774925262 wait complete. 2024-03-06T07:21:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.005+0000 7f6f57c6d640 1 Processor -- start 2024-03-06T07:21:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.005+0000 7f6f57c6d640 1 -- start start 2024-03-06T07:21:59.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.005+0000 7f6f57c6d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f500c1aa0 0x7f6f5014cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.005+0000 7f6f57c6d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f500c6fd0 0x7f6f5014ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.005+0000 7f6f57c6d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f50152730 0x7f6f50156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.005+0000 7f6f57c6d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f50155820 con 0x7f6f50152730 2024-03-06T07:21:59.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.005+0000 7f6f57c6d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f50152b10 con 0x7f6f500c6fd0 2024-03-06T07:21:59.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.005+0000 7f6f57c6d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f501559a0 con 0x7f6f500c1aa0 2024-03-06T07:21:59.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.006+0000 7f6f5620a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f50152730 0x7f6f50156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.006+0000 7f6f5620a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f50152730 0x7f6f50156f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34998/0 (socket says 172.21.15.1:34998) 2024-03-06T07:21:59.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.006+0000 7f6f5620a640 1 -- 172.21.15.1:0/1105900862 learned_addr learned my addr 172.21.15.1:0/1105900862 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:21:59.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.006+0000 7f6f55a09640 1 --2- 172.21.15.1:0/1105900862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f500c1aa0 0x7f6f5014cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.006+0000 7f6f55208640 1 --2- 172.21.15.1:0/1105900862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f500c6fd0 0x7f6f5014ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.006+0000 7f6f5620a640 1 -- 172.21.15.1:0/1105900862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f500c1aa0 msgr2=0x7f6f5014cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.006+0000 7f6f5620a640 1 --2- 172.21.15.1:0/1105900862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f500c1aa0 0x7f6f5014cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.006+0000 7f6f5620a640 1 -- 172.21.15.1:0/1105900862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f500c6fd0 msgr2=0x7f6f5014ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.006+0000 7f6f5620a640 1 --2- 172.21.15.1:0/1105900862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f500c6fd0 0x7f6f5014ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.006+0000 7f6f5620a640 1 -- 172.21.15.1:0/1105900862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f501ebde0 con 0x7f6f50152730 2024-03-06T07:21:59.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.006+0000 7f6f5620a640 1 --2- 172.21.15.1:0/1105900862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f50152730 0x7f6f50156f00 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800ce00 tx=0x7f6f4800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.006+0000 7f6f467fc640 1 -- 172.21.15.1:0/1105900862 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48003b90 con 0x7f6f50152730 2024-03-06T07:21:59.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.006+0000 7f6f467fc640 1 -- 172.21.15.1:0/1105900862 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f48003d50 con 0x7f6f50152730 2024-03-06T07:21:59.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.006+0000 7f6f57c6d640 1 -- 172.21.15.1:0/1105900862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f501565c0 con 0x7f6f50152730 2024-03-06T07:21:59.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.006+0000 7f6f467fc640 1 -- 172.21.15.1:0/1105900862 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48004260 con 0x7f6f50152730 2024-03-06T07:21:59.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.006+0000 7f6f57c6d640 1 -- 172.21.15.1:0/1105900862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f501ec0c0 con 0x7f6f50152730 2024-03-06T07:21:59.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.007+0000 7f6f57c6d640 1 -- 172.21.15.1:0/1105900862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f50067fd0 con 0x7f6f50152730 2024-03-06T07:21:59.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.008+0000 7f6f467fc640 1 -- 172.21.15.1:0/1105900862 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f48004400 con 0x7f6f50152730 2024-03-06T07:21:59.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.008+0000 7f6f467fc640 1 --2- 172.21.15.1:0/1105900862 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f3406b4d0 0x7f6f3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.008+0000 7f6f467fc640 1 -- 172.21.15.1:0/1105900862 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f6f4808a9c0 con 0x7f6f50152730 2024-03-06T07:21:59.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.008+0000 7f6f55a09640 1 --2- 172.21.15.1:0/1105900862 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f3406b4d0 0x7f6f3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.009+0000 7f6f55a09640 1 --2- 172.21.15.1:0/1105900862 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f3406b4d0 0x7f6f3406d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c035000 tx=0x7f6f4c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.013+0000 7f6f467fc640 1 -- 172.21.15.1:0/1105900862 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f4808ce20 con 0x7f6f50152730 2024-03-06T07:21:59.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:21:59.262+0000 7f6f57c6d640 1 -- 172.21.15.1:0/1105900862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]} v 0) v1 -- 0x7f6f5014ca50 con 0x7f6f50152730 2024-03-06T07:22:00.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.261+0000 7f6f467fc640 1 -- 172.21.15.1:0/1105900862 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]}]=0 v110) v1 ==== 153+0+0 (secure 0 0 0) 0x7f6f48059960 con 0x7f6f50152730 2024-03-06T07:22:00.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.264+0000 7f6f57c6d640 1 -- 172.21.15.1:0/1105900862 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f3406b4d0 msgr2=0x7f6f3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.264+0000 7f6f57c6d640 1 --2- 172.21.15.1:0/1105900862 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f3406b4d0 0x7f6f3406d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c035000 tx=0x7f6f4c008030 comp rx=0 tx=0).stop 2024-03-06T07:22:00.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.264+0000 7f6f57c6d640 1 -- 172.21.15.1:0/1105900862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f50152730 msgr2=0x7f6f50156f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.264+0000 7f6f57c6d640 1 --2- 172.21.15.1:0/1105900862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f50152730 0x7f6f50156f00 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800ce00 tx=0x7f6f4800a710 comp rx=0 tx=0).stop 2024-03-06T07:22:00.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.264+0000 7f6f57c6d640 1 -- 172.21.15.1:0/1105900862 shutdown_connections 2024-03-06T07:22:00.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.264+0000 7f6f57c6d640 1 --2- 172.21.15.1:0/1105900862 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f3406b4d0 0x7f6f3406d990 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.264+0000 7f6f57c6d640 1 --2- 172.21.15.1:0/1105900862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f50152730 0x7f6f50156f00 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.264+0000 7f6f57c6d640 1 --2- 172.21.15.1:0/1105900862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f500c6fd0 0x7f6f5014ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.264+0000 7f6f57c6d640 1 --2- 172.21.15.1:0/1105900862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f500c1aa0 0x7f6f5014cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.264+0000 7f6f57c6d640 1 -- 172.21.15.1:0/1105900862 >> 172.21.15.1:0/1105900862 conn(0x7f6f50066d00 msgr2=0x7f6f500670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.264+0000 7f6f57c6d640 1 -- 172.21.15.1:0/1105900862 shutdown_connections 2024-03-06T07:22:00.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.264+0000 7f6f57c6d640 1 -- 172.21.15.1:0/1105900862 wait complete. 2024-03-06T07:22:00.279 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:22:00.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.388+0000 7f33a2b02640 1 Processor -- start 2024-03-06T07:22:00.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.388+0000 7f33a2b02640 1 -- start start 2024-03-06T07:22:00.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.389+0000 7f33a2b02640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f339c152690 0x7f339c155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.389+0000 7f33a2b02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f339c0c6730 0x7f339c0c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.389+0000 7f33a2b02640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339c1561e0 con 0x7f339c0c5d10 2024-03-06T07:22:00.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.389+0000 7f33a2b02640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339c0c3300 con 0x7f339c152690 2024-03-06T07:22:00.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.389+0000 7f33a2b02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339c0c35a0 con 0x7f339c0c6730 2024-03-06T07:22:00.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.389+0000 7f33a089e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f339c152690 0x7f339c155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.389+0000 7f33a089e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f339c152690 0x7f339c155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35012/0 (socket says 172.21.15.1:35012) 2024-03-06T07:22:00.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.389+0000 7f33a089e640 1 -- 172.21.15.1:0/986625369 learned_addr learned my addr 172.21.15.1:0/986625369 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:00.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.389+0000 7f339bfff640 1 --2- 172.21.15.1:0/986625369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f339c0c6730 0x7f339c0c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.389+0000 7f33a089e640 1 -- 172.21.15.1:0/986625369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f339c0c6730 msgr2=0x7f339c0c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.389+0000 7f33a089e640 1 --2- 172.21.15.1:0/986625369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f339c0c6730 0x7f339c0c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.389+0000 7f33a089e640 1 -- 172.21.15.1:0/986625369 >> v1:172.21.15.98:6789/0 conn(0x7f339c0c5d10 legacy=0x7f339c156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.389+0000 7f33a089e640 1 -- 172.21.15.1:0/986625369 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339c0c3880 con 0x7f339c152690 2024-03-06T07:22:00.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.390+0000 7f33a089e640 1 --2- 172.21.15.1:0/986625369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339c152690 0x7f339c155ca0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3394009e50 tx=0x7f339402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecbcb941015e59c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.390+0000 7f339b7fe640 1 -- 172.21.15.1:0/986625369 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339402c9e0 con 0x7f339c152690 2024-03-06T07:22:00.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.390+0000 7f339b7fe640 1 -- 172.21.15.1:0/986625369 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339402cb80 con 0x7f339c152690 2024-03-06T07:22:00.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.390+0000 7f339b7fe640 1 -- 172.21.15.1:0/986625369 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33940343f0 con 0x7f339c152690 2024-03-06T07:22:00.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.391+0000 7f33a2b02640 1 -- 172.21.15.1:0/986625369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339c152690 msgr2=0x7f339c155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.391+0000 7f33a2b02640 1 --2- 172.21.15.1:0/986625369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339c152690 0x7f339c155ca0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3394009e50 tx=0x7f339402be40 comp rx=0 tx=0).stop 2024-03-06T07:22:00.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.391+0000 7f33a2b02640 1 -- 172.21.15.1:0/986625369 shutdown_connections 2024-03-06T07:22:00.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.391+0000 7f33a2b02640 1 --2- 172.21.15.1:0/986625369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f339c0c6730 0x7f339c0c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.391+0000 7f33a2b02640 1 --2- 172.21.15.1:0/986625369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339c152690 0x7f339c155ca0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.391+0000 7f33a2b02640 1 -- 172.21.15.1:0/986625369 >> 172.21.15.1:0/986625369 conn(0x7f339c06b410 msgr2=0x7f339c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.391+0000 7f33a2b02640 1 -- 172.21.15.1:0/986625369 shutdown_connections 2024-03-06T07:22:00.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.391+0000 7f33a2b02640 1 -- 172.21.15.1:0/986625369 wait complete. 2024-03-06T07:22:00.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.392+0000 7f33a2b02640 1 Processor -- start 2024-03-06T07:22:00.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.392+0000 7f33a2b02640 1 -- start start 2024-03-06T07:22:00.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f33a2b02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339c0c5d10 0x7f339c0709b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f33a2b02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f339c0c6730 0x7f339c1e7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f33a2b02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f339c152690 0x7f339c1e81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f33a2b02640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339c152a70 con 0x7f339c0c5d10 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f33a2b02640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339c1567b0 con 0x7f339c152690 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f33a2b02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339c152bf0 con 0x7f339c0c6730 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f33a089e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339c0c5d10 0x7f339c0709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f33a089e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339c0c5d10 0x7f339c0709b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35026/0 (socket says 172.21.15.1:35026) 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f33a089e640 1 -- 172.21.15.1:0/2733715181 learned_addr learned my addr 172.21.15.1:0/2733715181 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f339bfff640 1 --2- 172.21.15.1:0/2733715181 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f339c0c6730 0x7f339c1e7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f33a109f640 1 --2- 172.21.15.1:0/2733715181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f339c152690 0x7f339c1e81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f33a089e640 1 -- 172.21.15.1:0/2733715181 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f339c0c6730 msgr2=0x7f339c1e7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f33a089e640 1 --2- 172.21.15.1:0/2733715181 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f339c0c6730 0x7f339c1e7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f33a089e640 1 -- 172.21.15.1:0/2733715181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f339c152690 msgr2=0x7f339c1e81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f33a089e640 1 --2- 172.21.15.1:0/2733715181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f339c152690 0x7f339c1e81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f33a089e640 1 -- 172.21.15.1:0/2733715181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339c1eaab0 con 0x7f339c0c5d10 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f33a089e640 1 --2- 172.21.15.1:0/2733715181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339c0c5d10 0x7f339c0709b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f3394000f80 tx=0x7f3394004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.393+0000 7f33997fa640 1 -- 172.21.15.1:0/2733715181 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3394043020 con 0x7f339c0c5d10 2024-03-06T07:22:00.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.394+0000 7f33997fa640 1 -- 172.21.15.1:0/2733715181 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339403e450 con 0x7f339c0c5d10 2024-03-06T07:22:00.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.394+0000 7f33a2b02640 1 -- 172.21.15.1:0/2733715181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f339c1ead90 con 0x7f339c0c5d10 2024-03-06T07:22:00.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.394+0000 7f33997fa640 1 -- 172.21.15.1:0/2733715181 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3394009510 con 0x7f339c0c5d10 2024-03-06T07:22:00.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.394+0000 7f33a2b02640 1 -- 172.21.15.1:0/2733715181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f339c070ef0 con 0x7f339c0c5d10 2024-03-06T07:22:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.394+0000 7f33a2b02640 1 -- 172.21.15.1:0/2733715181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f339c067fd0 con 0x7f339c0c5d10 2024-03-06T07:22:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.395+0000 7f33997fa640 1 -- 172.21.15.1:0/2733715181 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3394005d10 con 0x7f339c0c5d10 2024-03-06T07:22:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.395+0000 7f33997fa640 1 --2- 172.21.15.1:0/2733715181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f338806b4d0 0x7f338806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.396+0000 7f33997fa640 1 -- 172.21.15.1:0/2733715181 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f33940aca30 con 0x7f339c0c5d10 2024-03-06T07:22:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.396+0000 7f339bfff640 1 --2- 172.21.15.1:0/2733715181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f338806b4d0 0x7f338806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.396+0000 7f339bfff640 1 --2- 172.21.15.1:0/2733715181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f338806b4d0 0x7f338806d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f339c071770 tx=0x7f3390008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.400+0000 7f33997fa640 1 -- 172.21.15.1:0/2733715181 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f339c1561e0 con 0x7f339c0c5d10 2024-03-06T07:22:00.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.581+0000 7f33a2b02640 1 -- 172.21.15.1:0/2733715181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f339c0c2c40 con 0x7f339c0c5d10 2024-03-06T07:22:00.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.584+0000 7f33997fa640 1 -- 172.21.15.1:0/2733715181 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v15)=0 v15) v1 ==== 122+0+0 (secure 0 0 0) 0x7f339407b410 con 0x7f339c0c5d10 2024-03-06T07:22:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.586+0000 7f33a2b02640 1 -- 172.21.15.1:0/2733715181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f338806b4d0 msgr2=0x7f338806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.586+0000 7f33a2b02640 1 --2- 172.21.15.1:0/2733715181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f338806b4d0 0x7f338806d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f339c071770 tx=0x7f3390008040 comp rx=0 tx=0).stop 2024-03-06T07:22:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.586+0000 7f33a2b02640 1 -- 172.21.15.1:0/2733715181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339c0c5d10 msgr2=0x7f339c0709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.586+0000 7f33a2b02640 1 --2- 172.21.15.1:0/2733715181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339c0c5d10 0x7f339c0709b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f3394000f80 tx=0x7f3394004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.586+0000 7f33a2b02640 1 -- 172.21.15.1:0/2733715181 shutdown_connections 2024-03-06T07:22:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.586+0000 7f33a2b02640 1 --2- 172.21.15.1:0/2733715181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f338806b4d0 0x7f338806d990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.586+0000 7f33a2b02640 1 --2- 172.21.15.1:0/2733715181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f339c152690 0x7f339c1e81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.586+0000 7f33a2b02640 1 --2- 172.21.15.1:0/2733715181 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f339c0c6730 0x7f339c1e7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.586+0000 7f33a2b02640 1 --2- 172.21.15.1:0/2733715181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339c0c5d10 0x7f339c0709b0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.586+0000 7f33a2b02640 1 -- 172.21.15.1:0/2733715181 >> 172.21.15.1:0/2733715181 conn(0x7f339c06b410 msgr2=0x7f339c1538f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.586+0000 7f33a2b02640 1 -- 172.21.15.1:0/2733715181 shutdown_connections 2024-03-06T07:22:00.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.586+0000 7f33a2b02640 1 -- 172.21.15.1:0/2733715181 wait complete. 2024-03-06T07:22:00.598 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:22:00.599 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:00.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:00.700 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:22:00.700 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:22:00.700 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:00.715 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:22:00.715 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:22:00.715 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:00.732 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:22:00.732 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:22:00.732 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:00.845 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:00.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.855+0000 7f8890b3e640 1 Processor -- start 2024-03-06T07:22:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.855+0000 7f8890b3e640 1 -- start start 2024-03-06T07:22:00.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.856+0000 7f8890b3e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f888c152850 0x7f888c0c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.856+0000 7f8890b3e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f888c0bf9e0 0x7f888c0c5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.856+0000 7f8890b3e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888c156150 con 0x7f888c0c4aa0 2024-03-06T07:22:00.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.856+0000 7f8890b3e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f888c152c30 con 0x7f888c0bf9e0 2024-03-06T07:22:00.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.856+0000 7f8890b3e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f888c067810 con 0x7f888c152850 2024-03-06T07:22:00.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.856+0000 7f8883fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f888c0bf9e0 0x7f888c0c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.856+0000 7f8883fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f888c0bf9e0 0x7f888c0c5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35042/0 (socket says 172.21.15.1:35042) 2024-03-06T07:22:00.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.856+0000 7f8883fff640 1 -- 172.21.15.1:0/3671099771 learned_addr learned my addr 172.21.15.1:0/3671099771 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:00.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.856+0000 7f888a59c640 1 --2- 172.21.15.1:0/3671099771 >> v2:172.21.15.98:3300/0 conn(0x7f888c152850 0x7f888c0c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.856+0000 7f8883fff640 1 -- 172.21.15.1:0/3671099771 >> v2:172.21.15.98:3300/0 conn(0x7f888c152850 msgr2=0x7f888c0c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.856+0000 7f8883fff640 1 --2- 172.21.15.1:0/3671099771 >> v2:172.21.15.98:3300/0 conn(0x7f888c152850 0x7f888c0c67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.856+0000 7f8883fff640 1 -- 172.21.15.1:0/3671099771 >> v1:172.21.15.98:6789/0 conn(0x7f888c0c4aa0 legacy=0x7f888c158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.856+0000 7f8883fff640 1 -- 172.21.15.1:0/3671099771 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888c155390 con 0x7f888c0bf9e0 2024-03-06T07:22:00.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.856+0000 7f8889d9b640 1 -- 172.21.15.1:0/3671099771 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525754649 0 0) 0x7f888c156150 con 0x7f888c0c4aa0 2024-03-06T07:22:00.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.857+0000 7f8883fff640 1 --2- 172.21.15.1:0/3671099771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c0bf9e0 0x7f888c0c5320 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f888c08c400 tx=0x7f887402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0e200054763536 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.857+0000 7f8889d9b640 1 -- 172.21.15.1:0/3671099771 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887402e070 con 0x7f888c0bf9e0 2024-03-06T07:22:00.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.857+0000 7f8889d9b640 1 -- 172.21.15.1:0/3671099771 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8874037070 con 0x7f888c0bf9e0 2024-03-06T07:22:00.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.857+0000 7f8889d9b640 1 -- 172.21.15.1:0/3671099771 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874031a00 con 0x7f888c0bf9e0 2024-03-06T07:22:00.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.858+0000 7f8890b3e640 1 -- 172.21.15.1:0/3671099771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c0bf9e0 msgr2=0x7f888c0c5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.858+0000 7f8890b3e640 1 --2- 172.21.15.1:0/3671099771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c0bf9e0 0x7f888c0c5320 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f888c08c400 tx=0x7f887402b880 comp rx=0 tx=0).stop 2024-03-06T07:22:00.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.858+0000 7f8890b3e640 1 -- 172.21.15.1:0/3671099771 shutdown_connections 2024-03-06T07:22:00.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.858+0000 7f8890b3e640 1 --2- 172.21.15.1:0/3671099771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c0bf9e0 0x7f888c0c5320 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.858+0000 7f8890b3e640 1 --2- 172.21.15.1:0/3671099771 >> v2:172.21.15.98:3300/0 conn(0x7f888c152850 0x7f888c0c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.858+0000 7f8890b3e640 1 -- 172.21.15.1:0/3671099771 >> 172.21.15.1:0/3671099771 conn(0x7f888c08b400 msgr2=0x7f888c066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.858+0000 7f8890b3e640 1 -- 172.21.15.1:0/3671099771 shutdown_connections 2024-03-06T07:22:00.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.858+0000 7f8890b3e640 1 -- 172.21.15.1:0/3671099771 wait complete. 2024-03-06T07:22:00.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.859+0000 7f8890b3e640 1 Processor -- start 2024-03-06T07:22:00.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.859+0000 7f8890b3e640 1 -- start start 2024-03-06T07:22:00.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f8890b3e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f888c0bf9e0 0x7f888c1ee980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f8890b3e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c0c4aa0 0x7f888c1405c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f8890b3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f888c152850 0x7f888c154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f8890b3e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c155950 con 0x7f888c0c4aa0 2024-03-06T07:22:00.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f8890b3e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c155cc0 con 0x7f888c0bf9e0 2024-03-06T07:22:00.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f8890b3e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888c0c4550 con 0x7f888c152850 2024-03-06T07:22:00.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f8883fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c0c4aa0 0x7f888c1405c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f8883fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c0c4aa0 0x7f888c1405c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35054/0 (socket says 172.21.15.1:35054) 2024-03-06T07:22:00.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f8883fff640 1 -- 172.21.15.1:0/2120275986 learned_addr learned my addr 172.21.15.1:0/2120275986 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:00.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f888ad9d640 1 --2- 172.21.15.1:0/2120275986 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f888c152850 0x7f888c154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f888a59c640 1 --2- 172.21.15.1:0/2120275986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f888c0bf9e0 0x7f888c1ee980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f8883fff640 1 -- 172.21.15.1:0/2120275986 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f888c152850 msgr2=0x7f888c154250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f8883fff640 1 --2- 172.21.15.1:0/2120275986 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f888c152850 0x7f888c154250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f8883fff640 1 -- 172.21.15.1:0/2120275986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f888c0bf9e0 msgr2=0x7f888c1ee980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f8883fff640 1 --2- 172.21.15.1:0/2120275986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f888c0bf9e0 0x7f888c1ee980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f8883fff640 1 -- 172.21.15.1:0/2120275986 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888c0c9cb0 con 0x7f888c0c4aa0 2024-03-06T07:22:00.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f888a59c640 1 --2- 172.21.15.1:0/2120275986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f888c0bf9e0 0x7f888c1ee980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:00.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.860+0000 7f8883fff640 1 --2- 172.21.15.1:0/2120275986 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c0c4aa0 0x7f888c1405c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f887402b8c0 tx=0x7f88740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.861+0000 7f8882ffd640 1 -- 172.21.15.1:0/2120275986 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874032670 con 0x7f888c0c4aa0 2024-03-06T07:22:00.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.861+0000 7f8882ffd640 1 -- 172.21.15.1:0/2120275986 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8874032830 con 0x7f888c0c4aa0 2024-03-06T07:22:00.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.861+0000 7f8890b3e640 1 -- 172.21.15.1:0/2120275986 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888c0c9f90 con 0x7f888c0c4aa0 2024-03-06T07:22:00.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.861+0000 7f8882ffd640 1 -- 172.21.15.1:0/2120275986 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874032c20 con 0x7f888c0c4aa0 2024-03-06T07:22:00.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.861+0000 7f8890b3e640 1 -- 172.21.15.1:0/2120275986 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888c1eab90 con 0x7f888c0c4aa0 2024-03-06T07:22:00.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.862+0000 7f8890b3e640 1 -- 172.21.15.1:0/2120275986 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888c0c2250 con 0x7f888c0c4aa0 2024-03-06T07:22:00.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.862+0000 7f8882ffd640 1 -- 172.21.15.1:0/2120275986 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88740453f0 con 0x7f888c0c4aa0 2024-03-06T07:22:00.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.863+0000 7f8882ffd640 1 --2- 172.21.15.1:0/2120275986 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f886c06b4d0 0x7f886c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.863+0000 7f8882ffd640 1 -- 172.21.15.1:0/2120275986 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f88740abe70 con 0x7f888c0c4aa0 2024-03-06T07:22:00.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.863+0000 7f888a59c640 1 --2- 172.21.15.1:0/2120275986 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f886c06b4d0 0x7f886c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.863+0000 7f888a59c640 1 --2- 172.21.15.1:0/2120275986 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f886c06b4d0 0x7f886c06d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f887c009af0 tx=0x7f887c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:00.867+0000 7f8882ffd640 1 -- 172.21.15.1:0/2120275986 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f888c0c2250 con 0x7f888c0c4aa0 2024-03-06T07:22:01.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.088+0000 7f8890b3e640 1 -- 172.21.15.1:0/2120275986 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f888c1ea980 con 0x7f888c0c4aa0 2024-03-06T07:22:01.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.089+0000 7f8882ffd640 1 -- 172.21.15.1:0/2120275986 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+3693 (secure 0 0 0) 0x7f8874040070 con 0x7f888c0c4aa0 2024-03-06T07:22:01.091 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:01.092 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":24,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3354292028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3354292028},{"type":"v1","addr":"172.21.15.98:6836","nonce":3354292028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4975,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3948803645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3948803645},{"type":"v1","addr":"172.21.15.98:6837","nonce":3948803645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5538,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3390157192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3390157192},{"type":"v1","addr":"172.21.15.1:6834","nonce":3390157192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2216478089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2216478089},{"type":"v1","addr":"172.21.15.1:6835","nonce":2216478089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24}],"filesystems":[]} 2024-03-06T07:22:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-03-06T07:22:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.091+0000 7f8890b3e640 1 -- 172.21.15.1:0/2120275986 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f886c06b4d0 msgr2=0x7f886c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.091+0000 7f8890b3e640 1 --2- 172.21.15.1:0/2120275986 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f886c06b4d0 0x7f886c06d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f887c009af0 tx=0x7f887c008040 comp rx=0 tx=0).stop 2024-03-06T07:22:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.091+0000 7f8890b3e640 1 -- 172.21.15.1:0/2120275986 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c0c4aa0 msgr2=0x7f888c1405c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.091+0000 7f8890b3e640 1 --2- 172.21.15.1:0/2120275986 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c0c4aa0 0x7f888c1405c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f887402b8c0 tx=0x7f88740046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.091+0000 7f8890b3e640 1 -- 172.21.15.1:0/2120275986 shutdown_connections 2024-03-06T07:22:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.091+0000 7f8890b3e640 1 --2- 172.21.15.1:0/2120275986 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f886c06b4d0 0x7f886c06d990 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.091+0000 7f8890b3e640 1 --2- 172.21.15.1:0/2120275986 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f888c152850 0x7f888c154250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.091+0000 7f8890b3e640 1 --2- 172.21.15.1:0/2120275986 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c0c4aa0 0x7f888c1405c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.091+0000 7f8890b3e640 1 --2- 172.21.15.1:0/2120275986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f888c0bf9e0 0x7f888c1ee980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.091+0000 7f8890b3e640 1 -- 172.21.15.1:0/2120275986 >> 172.21.15.1:0/2120275986 conn(0x7f888c08b400 msgr2=0x7f888c0c5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.091+0000 7f8890b3e640 1 -- 172.21.15.1:0/2120275986 shutdown_connections 2024-03-06T07:22:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.091+0000 7f8890b3e640 1 -- 172.21.15.1:0/2120275986 wait complete. 2024-03-06T07:22:01.105 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:22:01.105 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:22:01.105 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:22:01.105 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:01.106 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:22:01.106 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:22:01.106 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:22:01.106 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:01.107 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:22:01.107 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:22:01.107 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:22:01.107 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:01.108 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:22:01.108 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:22:01.108 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:22:01.108 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:01.115 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:22:01.115 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:22:01.116 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:22:01.117 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:22:01.119 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:22:01.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:22:01.119 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:22:01.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:22:01.158 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:22:01.158 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:01.159 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:22:01.159 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:01.161 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:22:01.161 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:01.162 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:22:01.162 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:01.162 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:22:01.235 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:22:01.235 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:22:01.243 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:22:01.243 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:22:01.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.266+0000 7f11a4ec9640 1 Processor -- start 2024-03-06T07:22:01.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.266+0000 7f11a4ec9640 1 -- start start 2024-03-06T07:22:01.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.267+0000 7f11a4ec9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f11a00c3970 0x7f11a00c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.267+0000 7f11a4ec9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11a0152840 0x7f11a01553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.267+0000 7f119ed9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11a0152840 0x7f11a01553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.267+0000 7f11a4ec9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a00c5e40 con 0x7f11a00c60b0 2024-03-06T07:22:01.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.267+0000 7f119ed9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11a0152840 0x7f11a01553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35096/0 (socket says 172.21.15.1:35096) 2024-03-06T07:22:01.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.267+0000 7f11a4ec9640 1 -- 172.21.15.1:0/1830554004 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a00c67f0 con 0x7f11a0152840 2024-03-06T07:22:01.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.267+0000 7f119ed9d640 1 -- 172.21.15.1:0/1830554004 learned_addr learned my addr 172.21.15.1:0/1830554004 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:01.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.267+0000 7f11a4ec9640 1 -- 172.21.15.1:0/1830554004 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a00c6a40 con 0x7f11a00c3970 2024-03-06T07:22:01.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.267+0000 7f119e59c640 1 --2- 172.21.15.1:0/1830554004 >> v2:172.21.15.98:3300/0 conn(0x7f11a00c3970 0x7f11a00c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.267+0000 7f119ed9d640 1 -- 172.21.15.1:0/1830554004 >> v2:172.21.15.98:3300/0 conn(0x7f11a00c3970 msgr2=0x7f11a00c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.267+0000 7f119ed9d640 1 --2- 172.21.15.1:0/1830554004 >> v2:172.21.15.98:3300/0 conn(0x7f11a00c3970 0x7f11a00c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.268+0000 7f119ed9d640 1 -- 172.21.15.1:0/1830554004 >> v1:172.21.15.98:6789/0 conn(0x7f11a00c60b0 legacy=0x7f11a0152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.268+0000 7f119ed9d640 1 -- 172.21.15.1:0/1830554004 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a00c6d20 con 0x7f11a0152840 2024-03-06T07:22:01.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.268+0000 7f119ed9d640 1 --2- 172.21.15.1:0/1830554004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0152840 0x7f11a01553d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f11a00c34c0 tx=0x7f119802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79289b774d4cdb91 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.270+0000 7f119d59a640 1 -- 172.21.15.1:0/1830554004 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119802cb10 con 0x7f11a0152840 2024-03-06T07:22:01.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.270+0000 7f119d59a640 1 -- 172.21.15.1:0/1830554004 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119802ccb0 con 0x7f11a0152840 2024-03-06T07:22:01.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.270+0000 7f119d59a640 1 -- 172.21.15.1:0/1830554004 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119802ce60 con 0x7f11a0152840 2024-03-06T07:22:01.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.270+0000 7f11a4ec9640 1 -- 172.21.15.1:0/1830554004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0152840 msgr2=0x7f11a01553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.270+0000 7f11a4ec9640 1 --2- 172.21.15.1:0/1830554004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0152840 0x7f11a01553d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f11a00c34c0 tx=0x7f119802bf70 comp rx=0 tx=0).stop 2024-03-06T07:22:01.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.270+0000 7f11a4ec9640 1 -- 172.21.15.1:0/1830554004 shutdown_connections 2024-03-06T07:22:01.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.270+0000 7f11a4ec9640 1 --2- 172.21.15.1:0/1830554004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0152840 0x7f11a01553d0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.270+0000 7f11a4ec9640 1 --2- 172.21.15.1:0/1830554004 >> v2:172.21.15.98:3300/0 conn(0x7f11a00c3970 0x7f11a00c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.270+0000 7f11a4ec9640 1 -- 172.21.15.1:0/1830554004 >> 172.21.15.1:0/1830554004 conn(0x7f11a00c88f0 msgr2=0x7f11a00c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.271+0000 7f11a4ec9640 1 -- 172.21.15.1:0/1830554004 shutdown_connections 2024-03-06T07:22:01.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.271+0000 7f11a4ec9640 1 -- 172.21.15.1:0/1830554004 wait complete. 2024-03-06T07:22:01.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.272+0000 7f11a4ec9640 1 Processor -- start 2024-03-06T07:22:01.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.272+0000 7f11a4ec9640 1 -- start start 2024-03-06T07:22:01.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.273+0000 7f11a4ec9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a00c3970 0x7f11a01564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.273+0000 7f11a4ec9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a00c60b0 0x7f11a014c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.273+0000 7f11a4ec9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11a0152840 0x7f11a01ee8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.273+0000 7f11a4ec9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a00c75c0 con 0x7f11a00c3970 2024-03-06T07:22:01.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.273+0000 7f11a4ec9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a00c72e0 con 0x7f11a00c60b0 2024-03-06T07:22:01.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.273+0000 7f11a4ec9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a00c3d50 con 0x7f11a0152840 2024-03-06T07:22:01.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.273+0000 7f119e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a00c3970 0x7f11a01564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.273+0000 7f119e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a00c3970 0x7f11a01564b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35102/0 (socket says 172.21.15.1:35102) 2024-03-06T07:22:01.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.273+0000 7f119e59c640 1 -- 172.21.15.1:0/1452063158 learned_addr learned my addr 172.21.15.1:0/1452063158 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:01.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.273+0000 7f119ed9d640 1 --2- 172.21.15.1:0/1452063158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11a0152840 0x7f11a01ee8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.273+0000 7f119dd9b640 1 --2- 172.21.15.1:0/1452063158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a00c60b0 0x7f11a014c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.273+0000 7f119e59c640 1 -- 172.21.15.1:0/1452063158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11a0152840 msgr2=0x7f11a01ee8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.273+0000 7f119e59c640 1 --2- 172.21.15.1:0/1452063158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11a0152840 0x7f11a01ee8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.273+0000 7f119e59c640 1 -- 172.21.15.1:0/1452063158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a00c60b0 msgr2=0x7f11a014c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.273+0000 7f119e59c640 1 --2- 172.21.15.1:0/1452063158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a00c60b0 0x7f11a014c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.273+0000 7f119e59c640 1 -- 172.21.15.1:0/1452063158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a014ad60 con 0x7f11a00c3970 2024-03-06T07:22:01.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.273+0000 7f119e59c640 1 --2- 172.21.15.1:0/1452063158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a00c3970 0x7f11a01564b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f118800e9b0 tx=0x7f118800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.275+0000 7f1196ffd640 1 -- 172.21.15.1:0/1452063158 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1188017070 con 0x7f11a00c3970 2024-03-06T07:22:01.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.276+0000 7f11a4ec9640 1 -- 172.21.15.1:0/1452063158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a014a8c0 con 0x7f11a00c3970 2024-03-06T07:22:01.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.276+0000 7f1196ffd640 1 -- 172.21.15.1:0/1452063158 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118800c9a0 con 0x7f11a00c3970 2024-03-06T07:22:01.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.276+0000 7f1196ffd640 1 -- 172.21.15.1:0/1452063158 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1188021060 con 0x7f11a00c3970 2024-03-06T07:22:01.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.276+0000 7f11a4ec9640 1 -- 172.21.15.1:0/1452063158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a01ee380 con 0x7f11a00c3970 2024-03-06T07:22:01.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.277+0000 7f11a4ec9640 1 -- 172.21.15.1:0/1452063158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a014aae0 con 0x7f11a00c3970 2024-03-06T07:22:01.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.277+0000 7f1196ffd640 1 -- 172.21.15.1:0/1452063158 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f118801a070 con 0x7f11a00c3970 2024-03-06T07:22:01.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.278+0000 7f1196ffd640 1 --2- 172.21.15.1:0/1452063158 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f118006b4d0 0x7f118006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.278+0000 7f1196ffd640 1 -- 172.21.15.1:0/1452063158 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f118808a310 con 0x7f11a00c3970 2024-03-06T07:22:01.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.278+0000 7f119dd9b640 1 --2- 172.21.15.1:0/1452063158 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f118006b4d0 0x7f118006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.278+0000 7f119dd9b640 1 --2- 172.21.15.1:0/1452063158 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f118006b4d0 0x7f118006d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f11a00c64b0 tx=0x7f1190009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.282+0000 7f1196ffd640 1 -- 172.21.15.1:0/1452063158 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11a014aae0 con 0x7f11a00c3970 2024-03-06T07:22:01.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-03-06T07:22:01.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-06T07:22:01.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-06T07:22:01.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:01.477+0000 7f11a4ec9640 1 -- 172.21.15.1:0/1452063158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f11a00686e0 con 0x7f11a00c3970 2024-03-06T07:22:02.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.284+0000 7f1196ffd640 1 -- 172.21.15.1:0/1452063158 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v48) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1188058c70 con 0x7f11a00c3970 2024-03-06T07:22:02.286 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-06T07:22:02.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.287+0000 7f11a4ec9640 1 -- 172.21.15.1:0/1452063158 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f118006b4d0 msgr2=0x7f118006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.287+0000 7f11a4ec9640 1 --2- 172.21.15.1:0/1452063158 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f118006b4d0 0x7f118006d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f11a00c64b0 tx=0x7f1190009040 comp rx=0 tx=0).stop 2024-03-06T07:22:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.287+0000 7f11a4ec9640 1 -- 172.21.15.1:0/1452063158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a00c3970 msgr2=0x7f11a01564b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.287+0000 7f11a4ec9640 1 --2- 172.21.15.1:0/1452063158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a00c3970 0x7f11a01564b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f118800e9b0 tx=0x7f118800ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.287+0000 7f11a4ec9640 1 -- 172.21.15.1:0/1452063158 shutdown_connections 2024-03-06T07:22:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.287+0000 7f11a4ec9640 1 --2- 172.21.15.1:0/1452063158 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f118006b4d0 0x7f118006d990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.287+0000 7f11a4ec9640 1 --2- 172.21.15.1:0/1452063158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11a0152840 0x7f11a01ee8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.287+0000 7f11a4ec9640 1 --2- 172.21.15.1:0/1452063158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a00c60b0 0x7f11a014c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.287+0000 7f11a4ec9640 1 --2- 172.21.15.1:0/1452063158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a00c3970 0x7f11a01564b0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.287+0000 7f11a4ec9640 1 -- 172.21.15.1:0/1452063158 >> 172.21.15.1:0/1452063158 conn(0x7f11a00c88f0 msgr2=0x7f11a00c1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.287+0000 7f11a4ec9640 1 -- 172.21.15.1:0/1452063158 shutdown_connections 2024-03-06T07:22:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.287+0000 7f11a4ec9640 1 -- 172.21.15.1:0/1452063158 wait complete. 2024-03-06T07:22:02.302 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:22:02.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.407+0000 7f8c83deb640 1 Processor -- start 2024-03-06T07:22:02.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.408+0000 7f8c83deb640 1 -- start start 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.408+0000 7f8c83deb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8c7c152650 0x7f8c7c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.408+0000 7f8c83deb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c7c0c6b10 0x7f8c7c158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.408+0000 7f8c83deb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c0c2fc0 con 0x7f8c7c0bf9d0 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.408+0000 7f8c83deb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c7c155800 con 0x7f8c7c152650 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.408+0000 7f8c83deb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0c6690 con 0x7f8c7c0c6b10 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.408+0000 7f8c81b87640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8c7c152650 0x7f8c7c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.408+0000 7f8c82388640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8c7c0bf9d0 0x7f8c7c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57716/0 (socket says 172.21.15.1:57716) 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.408+0000 7f8c82388640 1 -- 172.21.15.1:0/2422672900 learned_addr learned my addr 172.21.15.1:0/2422672900 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.409+0000 7f8c81386640 1 --2- 172.21.15.1:0/2422672900 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c7c0c6b10 0x7f8c7c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.409+0000 7f8c80b85640 1 -- 172.21.15.1:0/2422672900 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291635635 0 0) 0x7f8c7c0c2fc0 con 0x7f8c7c0bf9d0 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.409+0000 7f8c80b85640 1 -- 172.21.15.1:0/2422672900 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c68003520 con 0x7f8c7c0bf9d0 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.409+0000 7f8c81b87640 1 -- 172.21.15.1:0/2422672900 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c7c0c6b10 msgr2=0x7f8c7c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.409+0000 7f8c81b87640 1 --2- 172.21.15.1:0/2422672900 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c7c0c6b10 0x7f8c7c158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.409+0000 7f8c81b87640 1 -- 172.21.15.1:0/2422672900 >> v1:172.21.15.1:6789/0 conn(0x7f8c7c0bf9d0 legacy=0x7f8c7c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.409+0000 7f8c81b87640 1 -- 172.21.15.1:0/2422672900 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c0c7280 con 0x7f8c7c152650 2024-03-06T07:22:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.409+0000 7f8c81386640 1 --2- 172.21.15.1:0/2422672900 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c7c0c6b10 0x7f8c7c158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.410+0000 7f8c81b87640 1 --2- 172.21.15.1:0/2422672900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c7c152650 0x7f8c7c155c90 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8c74009d90 tx=0x7f8c7402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=601ee3f49795ad04 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.410+0000 7f8c80b85640 1 -- 172.21.15.1:0/2422672900 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c74039070 con 0x7f8c7c152650 2024-03-06T07:22:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.410+0000 7f8c80b85640 1 -- 172.21.15.1:0/2422672900 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7402cbc0 con 0x7f8c7c152650 2024-03-06T07:22:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.410+0000 7f8c80b85640 1 -- 172.21.15.1:0/2422672900 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c74034440 con 0x7f8c7c152650 2024-03-06T07:22:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.410+0000 7f8c83deb640 1 -- 172.21.15.1:0/2422672900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c7c152650 msgr2=0x7f8c7c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.410+0000 7f8c83deb640 1 --2- 172.21.15.1:0/2422672900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c7c152650 0x7f8c7c155c90 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8c74009d90 tx=0x7f8c7402be10 comp rx=0 tx=0).stop 2024-03-06T07:22:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.410+0000 7f8c83deb640 1 -- 172.21.15.1:0/2422672900 shutdown_connections 2024-03-06T07:22:02.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.410+0000 7f8c83deb640 1 --2- 172.21.15.1:0/2422672900 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c7c0c6b10 0x7f8c7c158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.410+0000 7f8c83deb640 1 --2- 172.21.15.1:0/2422672900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c7c152650 0x7f8c7c155c90 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.410+0000 7f8c83deb640 1 -- 172.21.15.1:0/2422672900 >> 172.21.15.1:0/2422672900 conn(0x7f8c7c066ee0 msgr2=0x7f8c7c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.410+0000 7f8c83deb640 1 -- 172.21.15.1:0/2422672900 shutdown_connections 2024-03-06T07:22:02.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.410+0000 7f8c83deb640 1 -- 172.21.15.1:0/2422672900 wait complete. 2024-03-06T07:22:02.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.411+0000 7f8c83deb640 1 Processor -- start 2024-03-06T07:22:02.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.412+0000 7f8c83deb640 1 -- start start 2024-03-06T07:22:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.412+0000 7f8c83deb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c7c0bf9d0 0x7f8c7c1eb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.412+0000 7f8c83deb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c7c152650 0x7f8c7c1578e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.412+0000 7f8c83deb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c7c14abf0 0x7f8c7c145740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.412+0000 7f8c83deb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c155490 con 0x7f8c7c0bf9d0 2024-03-06T07:22:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.412+0000 7f8c83deb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c1551b0 con 0x7f8c7c14abf0 2024-03-06T07:22:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.412+0000 7f8c83deb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c152a30 con 0x7f8c7c152650 2024-03-06T07:22:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.412+0000 7f8c81b87640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c7c0bf9d0 0x7f8c7c1eb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.412+0000 7f8c81b87640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c7c0bf9d0 0x7f8c7c1eb6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35132/0 (socket says 172.21.15.1:35132) 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.412+0000 7f8c81b87640 1 -- 172.21.15.1:0/3482407370 learned_addr learned my addr 172.21.15.1:0/3482407370 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.412+0000 7f8c81386640 1 --2- 172.21.15.1:0/3482407370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c7c152650 0x7f8c7c1578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.412+0000 7f8c82388640 1 --2- 172.21.15.1:0/3482407370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c7c14abf0 0x7f8c7c145740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.413+0000 7f8c81b87640 1 -- 172.21.15.1:0/3482407370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c7c152650 msgr2=0x7f8c7c1578e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.413+0000 7f8c81b87640 1 --2- 172.21.15.1:0/3482407370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c7c152650 0x7f8c7c1578e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.413+0000 7f8c81b87640 1 -- 172.21.15.1:0/3482407370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c7c14abf0 msgr2=0x7f8c7c145740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.413+0000 7f8c81b87640 1 --2- 172.21.15.1:0/3482407370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c7c14abf0 0x7f8c7c145740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.413+0000 7f8c81b87640 1 -- 172.21.15.1:0/3482407370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c157e20 con 0x7f8c7c0bf9d0 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.413+0000 7f8c82388640 1 --2- 172.21.15.1:0/3482407370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c7c14abf0 0x7f8c7c145740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.413+0000 7f8c81b87640 1 --2- 172.21.15.1:0/3482407370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c7c0bf9d0 0x7f8c7c1eb6d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8c74009d90 tx=0x7f8c74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.413+0000 7f8c727fc640 1 -- 172.21.15.1:0/3482407370 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c74039050 con 0x7f8c7c0bf9d0 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.413+0000 7f8c727fc640 1 -- 172.21.15.1:0/3482407370 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c74045070 con 0x7f8c7c0bf9d0 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.413+0000 7f8c727fc640 1 -- 172.21.15.1:0/3482407370 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7403e850 con 0x7f8c7c0bf9d0 2024-03-06T07:22:02.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.413+0000 7f8c83deb640 1 -- 172.21.15.1:0/3482407370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7c14b040 con 0x7f8c7c0bf9d0 2024-03-06T07:22:02.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.413+0000 7f8c83deb640 1 -- 172.21.15.1:0/3482407370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7c14b600 con 0x7f8c7c0bf9d0 2024-03-06T07:22:02.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.414+0000 7f8c83deb640 1 -- 172.21.15.1:0/3482407370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7c14b7d0 con 0x7f8c7c0bf9d0 2024-03-06T07:22:02.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.414+0000 7f8c727fc640 1 -- 172.21.15.1:0/3482407370 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c7403e9f0 con 0x7f8c7c0bf9d0 2024-03-06T07:22:02.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.415+0000 7f8c727fc640 1 --2- 172.21.15.1:0/3482407370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c6006b4d0 0x7f8c6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.415+0000 7f8c727fc640 1 -- 172.21.15.1:0/3482407370 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8c740acd50 con 0x7f8c7c0bf9d0 2024-03-06T07:22:02.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.415+0000 7f8c81386640 1 --2- 172.21.15.1:0/3482407370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c6006b4d0 0x7f8c6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.415+0000 7f8c81386640 1 --2- 172.21.15.1:0/3482407370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c6006b4d0 0x7f8c6006d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c151a60 tx=0x7f8c6c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.419+0000 7f8c727fc640 1 -- 172.21.15.1:0/3482407370 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c740b0030 con 0x7f8c7c0bf9d0 2024-03-06T07:22:02.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.668+0000 7f8c83deb640 1 -- 172.21.15.1:0/3482407370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8c7c067fd0 con 0x7f8c7c0bf9d0 2024-03-06T07:22:02.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.668+0000 7f8c727fc640 1 -- 172.21.15.1:0/3482407370 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f8c7407bf40 con 0x7f8c7c0bf9d0 2024-03-06T07:22:02.670 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:02.670 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:22:02.670 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-06T07:22:02.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:02.670 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-06T07:22:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlnxyWARAAKIPXU0IafNQ1RzBPX+3DYw==", 2024-03-06T07:22:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:22:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:22:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:22:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-06T07:22:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl2fXxDRAAuLqH+HpKcN1JTts9ZtDTXA==", 2024-03-06T07:22:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:22:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:22:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:22:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-06T07:22:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl+IzqCBAAmgLkarFGt9Ihs/8kJYicBQ==", 2024-03-06T07:22:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:22:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:22:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:22:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-06T07:22:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlipdRFRAAA34vrsTEoX3L1CKb1YAV9g==", 2024-03-06T07:22:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:22:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:22:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:22:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-06T07:22:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAGGehlQK39LxAAOpPIZvl3FqQwCgyo3yoOGQ==", 2024-03-06T07:22:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:22:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-06T07:22:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAJGehlTxwDLhAAMwN9E0+3Jh5XNXnLo+fw3g==", 2024-03-06T07:22:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:22:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-06T07:22:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAMGehl4qrlLBAAm0VWQRNvSQIpDx5xf4Fyig==", 2024-03-06T07:22:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:22:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-06T07:22:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAPGehllO5HKxAAozm6ycG+fRxatoZTogeJTA==", 2024-03-06T07:22:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:22:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-06T07:22:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAVGehlLtLBMhAAUrV6q6ZCQluIcoo/S/NdbA==", 2024-03-06T07:22:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:22:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-06T07:22:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAYGehlHp59MRAAxxgpSPTEZkYKjl3fr0xeeQ==", 2024-03-06T07:22:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:22:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-06T07:22:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAbGehlTnXcMBAAFOBqcnh5ubAz+o6AGjccWQ==", 2024-03-06T07:22:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:22:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-06T07:22:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAeGehl1qH1LhAAaSl7Aj7qlN6d4xt5SKAmcg==", 2024-03-06T07:22:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:22:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-06T07:22:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng==", 2024-03-06T07:22:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:22:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-06T07:22:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:22:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-06T07:22:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlMyv7HBAAi7vt7aZWEaZuMpFrB5ofZw==", 2024-03-06T07:22:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:22:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-06T07:22:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-06T07:22:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-06T07:22:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-06T07:22:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlBS/vIBAAN86N2K4ze5PJB8kR869Lfg==", 2024-03-06T07:22:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:22:02.681 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:22:02.681 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:22:02.681 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:02.681 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:02.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:02.681 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-06T07:22:02.681 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlJ/jPORAAhLk5MTGafsGKhZSYhvhBZg==", 2024-03-06T07:22:02.681 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:02.681 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:22:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-06T07:22:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlAl8rNhAAvdU0Qvo/FJYnxc4/H8MIqQ==", 2024-03-06T07:22:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:22:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-06T07:22:02.682 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:22:02.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.670+0000 7f8c83deb640 1 -- 172.21.15.1:0/3482407370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c6006b4d0 msgr2=0x7f8c6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.670+0000 7f8c83deb640 1 --2- 172.21.15.1:0/3482407370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c6006b4d0 0x7f8c6006d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c151a60 tx=0x7f8c6c008040 comp rx=0 tx=0).stop 2024-03-06T07:22:02.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.670+0000 7f8c83deb640 1 -- 172.21.15.1:0/3482407370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c7c0bf9d0 msgr2=0x7f8c7c1eb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.670+0000 7f8c83deb640 1 --2- 172.21.15.1:0/3482407370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c7c0bf9d0 0x7f8c7c1eb6d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8c74009d90 tx=0x7f8c74004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:02.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.670+0000 7f8c83deb640 1 -- 172.21.15.1:0/3482407370 shutdown_connections 2024-03-06T07:22:02.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.670+0000 7f8c83deb640 1 --2- 172.21.15.1:0/3482407370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c6006b4d0 0x7f8c6006d990 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.670+0000 7f8c83deb640 1 --2- 172.21.15.1:0/3482407370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c7c14abf0 0x7f8c7c145740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.670+0000 7f8c83deb640 1 --2- 172.21.15.1:0/3482407370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8c7c152650 0x7f8c7c1578e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.670+0000 7f8c83deb640 1 --2- 172.21.15.1:0/3482407370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c7c0bf9d0 0x7f8c7c1eb6d0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.670+0000 7f8c83deb640 1 -- 172.21.15.1:0/3482407370 >> 172.21.15.1:0/3482407370 conn(0x7f8c7c066ee0 msgr2=0x7f8c7c0c5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.670+0000 7f8c83deb640 1 -- 172.21.15.1:0/3482407370 shutdown_connections 2024-03-06T07:22:02.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.670+0000 7f8c83deb640 1 -- 172.21.15.1:0/3482407370 wait complete. 2024-03-06T07:22:02.685 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:02.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.784+0000 7fe064439640 1 Processor -- start 2024-03-06T07:22:02.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.784+0000 7fe064439640 1 -- start start 2024-03-06T07:22:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.785+0000 7fe064439640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe05c159140 0x7fe05c0c4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.785+0000 7fe064439640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05c072d00 con 0x7fe05c0c7510 2024-03-06T07:22:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.785+0000 7fe064439640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05c073180 con 0x7fe05c0c69b0 2024-03-06T07:22:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.785+0000 7fe064439640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe05c0c0e80 con 0x7fe05c159140 2024-03-06T07:22:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.785+0000 7fe0629d6640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe05c0c7510 0x7fe05c15fb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57724/0 (socket says 172.21.15.1:57724) 2024-03-06T07:22:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.785+0000 7fe0629d6640 1 -- 172.21.15.1:0/170842672 learned_addr learned my addr 172.21.15.1:0/170842672 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.785+0000 7fe0621d5640 1 --2- 172.21.15.1:0/170842672 >> v2:172.21.15.98:3300/0 conn(0x7fe05c159140 0x7fe05c0c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.785+0000 7fe0611d3640 1 -- 172.21.15.1:0/170842672 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847948842 0 0) 0x7fe05c072d00 con 0x7fe05c0c7510 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0611d3640 1 -- 172.21.15.1:0/170842672 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe048003520 con 0x7fe05c0c7510 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0611d3640 1 -- 172.21.15.1:0/170842672 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2866542191 0 0) 0x7fe048003520 con 0x7fe05c0c7510 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0611d3640 1 -- 172.21.15.1:0/170842672 >> v2:172.21.15.98:3300/0 conn(0x7fe05c159140 msgr2=0x7fe05c0c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0611d3640 1 --2- 172.21.15.1:0/170842672 >> v2:172.21.15.98:3300/0 conn(0x7fe05c159140 0x7fe05c0c4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0611d3640 1 -- 172.21.15.1:0/170842672 >> v1:172.21.15.98:6789/0 conn(0x7fe05c0c69b0 legacy=0x7fe05c0c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0611d3640 1 -- 172.21.15.1:0/170842672 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05c073460 con 0x7fe05c0c7510 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0611d3640 1 -- 172.21.15.1:0/170842672 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fe0580019f0 con 0x7fe05c0c7510 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0611d3640 1 -- 172.21.15.1:0/170842672 >> v1:172.21.15.1:6789/0 conn(0x7fe05c0c7510 legacy=0x7fe05c15fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0611d3640 1 --2- 172.21.15.1:0/170842672 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0480039b0 0x7fe048003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0611d3640 1 --2- 172.21.15.1:0/170842672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0480043b0 0x7fe048004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0611d3640 1 --2- 172.21.15.1:0/170842672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe048005120 0x7fe04800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0611d3640 1 -- 172.21.15.1:0/170842672 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c0c0e80 con 0x7fe0480043b0 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0611d3640 1 -- 172.21.15.1:0/170842672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04800c060 con 0x7fe048005120 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0611d3640 1 -- 172.21.15.1:0/170842672 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe04800c260 con 0x7fe0480039b0 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0621d5640 1 --2- 172.21.15.1:0/170842672 >> v2:172.21.15.98:3300/0 conn(0x7fe05c159140 0x7fe05c0c4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0619d4640 1 --2- 172.21.15.1:0/170842672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0480043b0 0x7fe048004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0629d6640 1 --2- 172.21.15.1:0/170842672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe048005120 0x7fe04800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0621d5640 1 --2- 172.21.15.1:0/170842672 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0480039b0 0x7fe048003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0619d4640 1 -- 172.21.15.1:0/170842672 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0480039b0 msgr2=0x7fe048003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0619d4640 1 --2- 172.21.15.1:0/170842672 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0480039b0 0x7fe048003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0619d4640 1 -- 172.21.15.1:0/170842672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe048005120 msgr2=0x7fe04800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0619d4640 1 --2- 172.21.15.1:0/170842672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe048005120 0x7fe04800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0629d6640 1 -- 172.21.15.1:0/170842672 reap_dead start 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0619d4640 1 -- 172.21.15.1:0/170842672 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe048003520 con 0x7fe0480043b0 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.786+0000 7fe0621d5640 1 --2- 172.21.15.1:0/170842672 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0480039b0 0x7fe048003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.787+0000 7fe0619d4640 1 --2- 172.21.15.1:0/170842672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0480043b0 0x7fe048004880 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe05c0aa8c0 tx=0x7fe048005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5feefc3486256f53 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.787+0000 7fe0611d3640 1 -- 172.21.15.1:0/170842672 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04c041070 con 0x7fe0480043b0 2024-03-06T07:22:02.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.787+0000 7fe0611d3640 1 -- 172.21.15.1:0/170842672 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe04c034b20 con 0x7fe0480043b0 2024-03-06T07:22:02.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.787+0000 7fe0611d3640 1 -- 172.21.15.1:0/170842672 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04c001430 con 0x7fe0480043b0 2024-03-06T07:22:02.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.788+0000 7fe064439640 1 -- 172.21.15.1:0/170842672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0480043b0 msgr2=0x7fe048004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.788+0000 7fe064439640 1 --2- 172.21.15.1:0/170842672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0480043b0 0x7fe048004880 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe05c0aa8c0 tx=0x7fe048005830 comp rx=0 tx=0).stop 2024-03-06T07:22:02.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.788+0000 7fe064439640 1 -- 172.21.15.1:0/170842672 shutdown_connections 2024-03-06T07:22:02.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.788+0000 7fe064439640 1 --2- 172.21.15.1:0/170842672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0480043b0 0x7fe048004880 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.788+0000 7fe064439640 1 -- 172.21.15.1:0/170842672 >> 172.21.15.1:0/170842672 conn(0x7fe05c0c7a50 msgr2=0x7fe05c0c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.788+0000 7fe064439640 1 -- 172.21.15.1:0/170842672 shutdown_connections 2024-03-06T07:22:02.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.788+0000 7fe064439640 1 -- 172.21.15.1:0/170842672 wait complete. 2024-03-06T07:22:02.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.789+0000 7fe064439640 1 Processor -- start 2024-03-06T07:22:02.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.789+0000 7fe064439640 1 -- start start 2024-03-06T07:22:02.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe064439640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0480043b0 0x7fe05c14d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe064439640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe05c154750 0x7fe05c1e7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe064439640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe05c1e8460 0x7fe05c1e9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe064439640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c0c6520 con 0x7fe05c154750 2024-03-06T07:22:02.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe064439640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c0c13b0 con 0x7fe05c1e8460 2024-03-06T07:22:02.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe064439640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c159520 con 0x7fe0480043b0 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe0619d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe05c154750 0x7fe05c1e7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe0619d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe05c154750 0x7fe05c1e7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35152/0 (socket says 172.21.15.1:35152) 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe0619d4640 1 -- 172.21.15.1:0/699472630 learned_addr learned my addr 172.21.15.1:0/699472630 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe0621d5640 1 --2- 172.21.15.1:0/699472630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0480043b0 0x7fe05c14d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe0629d6640 1 --2- 172.21.15.1:0/699472630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe05c1e8460 0x7fe05c1e9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe0619d4640 1 -- 172.21.15.1:0/699472630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0480043b0 msgr2=0x7fe05c14d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe0619d4640 1 --2- 172.21.15.1:0/699472630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0480043b0 0x7fe05c14d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe0619d4640 1 -- 172.21.15.1:0/699472630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe05c1e8460 msgr2=0x7fe05c1e9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe0619d4640 1 --2- 172.21.15.1:0/699472630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe05c1e8460 0x7fe05c1e9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe0619d4640 1 -- 172.21.15.1:0/699472630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05c154c80 con 0x7fe05c154750 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe0619d4640 1 --2- 172.21.15.1:0/699472630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe05c154750 0x7fe05c1e7f20 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe04c0018f0 tx=0x7fe04c001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe052ffd640 1 -- 172.21.15.1:0/699472630 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04c041050 con 0x7fe05c154750 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe052ffd640 1 -- 172.21.15.1:0/699472630 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe04c04d070 con 0x7fe05c154750 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe064439640 1 -- 172.21.15.1:0/699472630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe05c154f60 con 0x7fe05c154750 2024-03-06T07:22:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe052ffd640 1 -- 172.21.15.1:0/699472630 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04c001430 con 0x7fe05c154750 2024-03-06T07:22:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.790+0000 7fe064439640 1 -- 172.21.15.1:0/699472630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05c155520 con 0x7fe05c154750 2024-03-06T07:22:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.791+0000 7fe064439640 1 -- 172.21.15.1:0/699472630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe05c0addf0 con 0x7fe05c154750 2024-03-06T07:22:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.792+0000 7fe052ffd640 1 -- 172.21.15.1:0/699472630 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe04c048890 con 0x7fe05c154750 2024-03-06T07:22:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.792+0000 7fe052ffd640 1 --2- 172.21.15.1:0/699472630 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe04006b4d0 0x7fe04006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.792+0000 7fe052ffd640 1 -- 172.21.15.1:0/699472630 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fe04c0b4610 con 0x7fe05c154750 2024-03-06T07:22:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.792+0000 7fe0621d5640 1 --2- 172.21.15.1:0/699472630 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe04006b4d0 0x7fe04006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.793+0000 7fe0621d5640 1 --2- 172.21.15.1:0/699472630 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe04006b4d0 0x7fe04006d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe05c0acac0 tx=0x7fe054008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.796+0000 7fe052ffd640 1 -- 172.21.15.1:0/699472630 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe05c0addf0 con 0x7fe05c154750 2024-03-06T07:22:02.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.978+0000 7fe064439640 1 -- 172.21.15.1:0/699472630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe05c1556f0 con 0x7fe05c154750 2024-03-06T07:22:02.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.979+0000 7fe052ffd640 1 -- 172.21.15.1:0/699472630 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+13044 (secure 0 0 0) 0x7fe04c057020 con 0x7fe05c154750 2024-03-06T07:22:02.981 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:02.981 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":48,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:02.283501+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:02.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.981+0000 7fe064439640 1 -- 172.21.15.1:0/699472630 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe04006b4d0 msgr2=0x7fe04006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.981+0000 7fe064439640 1 --2- 172.21.15.1:0/699472630 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe04006b4d0 0x7fe04006d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe05c0acac0 tx=0x7fe054008040 comp rx=0 tx=0).stop 2024-03-06T07:22:02.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.981+0000 7fe064439640 1 -- 172.21.15.1:0/699472630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe05c154750 msgr2=0x7fe05c1e7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.981+0000 7fe064439640 1 --2- 172.21.15.1:0/699472630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe05c154750 0x7fe05c1e7f20 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe04c0018f0 tx=0x7fe04c001230 comp rx=0 tx=0).stop 2024-03-06T07:22:02.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.981+0000 7fe064439640 1 -- 172.21.15.1:0/699472630 shutdown_connections 2024-03-06T07:22:02.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.981+0000 7fe064439640 1 --2- 172.21.15.1:0/699472630 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe04006b4d0 0x7fe04006d990 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.981+0000 7fe064439640 1 --2- 172.21.15.1:0/699472630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe05c1e8460 0x7fe05c1e9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.981+0000 7fe064439640 1 --2- 172.21.15.1:0/699472630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe05c154750 0x7fe05c1e7f20 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.981+0000 7fe064439640 1 --2- 172.21.15.1:0/699472630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0480043b0 0x7fe05c14d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.981+0000 7fe064439640 1 -- 172.21.15.1:0/699472630 >> 172.21.15.1:0/699472630 conn(0x7fe05c0c7a50 msgr2=0x7fe05c0c7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.981+0000 7fe064439640 1 -- 172.21.15.1:0/699472630 shutdown_connections 2024-03-06T07:22:02.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:02.981+0000 7fe064439640 1 -- 172.21.15.1:0/699472630 wait complete. 2024-03-06T07:22:02.994 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:02.994 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:03.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.097+0000 7fe74673e640 1 Processor -- start 2024-03-06T07:22:03.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.097+0000 7fe74673e640 1 -- start start 2024-03-06T07:22:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe74673e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe740072f20 0x7fe7401513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe74673e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe740071f90 0x7fe740072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe74673e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7400a6140 con 0x7fe740150fe0 2024-03-06T07:22:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe74673e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7401526b0 con 0x7fe740072f20 2024-03-06T07:22:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe74673e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe740151aa0 con 0x7fe740071f90 2024-03-06T07:22:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe740150fe0 0x7fe74006edb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57740/0 (socket says 172.21.15.1:57740) 2024-03-06T07:22:03.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73ffff640 1 -- 172.21.15.1:0/908436822 learned_addr learned my addr 172.21.15.1:0/908436822 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:03.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73f7fe640 1 --2- 172.21.15.1:0/908436822 >> v2:172.21.15.98:3300/0 conn(0x7fe740072f20 0x7fe7401513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73effd640 1 -- 172.21.15.1:0/908436822 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654442256 0 0) 0x7fe7400a6140 con 0x7fe740150fe0 2024-03-06T07:22:03.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe744cdb640 1 --2- 172.21.15.1:0/908436822 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe740071f90 0x7fe740072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73effd640 1 -- 172.21.15.1:0/908436822 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe728003520 con 0x7fe740150fe0 2024-03-06T07:22:03.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73effd640 1 -- 172.21.15.1:0/908436822 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2734330143 0 0) 0x7fe728003520 con 0x7fe740150fe0 2024-03-06T07:22:03.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73effd640 1 -- 172.21.15.1:0/908436822 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe740071f90 msgr2=0x7fe740072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73effd640 1 --2- 172.21.15.1:0/908436822 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe740071f90 0x7fe740072420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73effd640 1 -- 172.21.15.1:0/908436822 >> v2:172.21.15.98:3300/0 conn(0x7fe740072f20 msgr2=0x7fe7401513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73effd640 1 --2- 172.21.15.1:0/908436822 >> v2:172.21.15.98:3300/0 conn(0x7fe740072f20 0x7fe7401513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73effd640 1 -- 172.21.15.1:0/908436822 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe740151d80 con 0x7fe740150fe0 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73effd640 1 -- 172.21.15.1:0/908436822 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fe730001670 con 0x7fe740150fe0 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73effd640 1 -- 172.21.15.1:0/908436822 >> v1:172.21.15.1:6789/0 conn(0x7fe740150fe0 legacy=0x7fe74006edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73effd640 1 --2- 172.21.15.1:0/908436822 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe740071f90 0x7fe7280039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73effd640 1 --2- 172.21.15.1:0/908436822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe728003f30 0x7fe728004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73effd640 1 --2- 172.21.15.1:0/908436822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe728004d40 0x7fe7280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73effd640 1 -- 172.21.15.1:0/908436822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe740151aa0 con 0x7fe728003f30 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73effd640 1 -- 172.21.15.1:0/908436822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72800b9c0 con 0x7fe728004d40 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.098+0000 7fe73effd640 1 -- 172.21.15.1:0/908436822 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe72800bb90 con 0x7fe740071f90 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.099+0000 7fe73f7fe640 1 --2- 172.21.15.1:0/908436822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe728003f30 0x7fe728004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.099+0000 7fe73ffff640 1 --2- 172.21.15.1:0/908436822 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe740071f90 0x7fe7280039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.099+0000 7fe744cdb640 1 --2- 172.21.15.1:0/908436822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe728004d40 0x7fe7280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.099+0000 7fe73f7fe640 1 -- 172.21.15.1:0/908436822 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe740071f90 msgr2=0x7fe7280039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.099+0000 7fe73f7fe640 1 --2- 172.21.15.1:0/908436822 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe740071f90 0x7fe7280039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.099+0000 7fe73f7fe640 1 -- 172.21.15.1:0/908436822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe728004d40 msgr2=0x7fe7280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.099+0000 7fe73f7fe640 1 --2- 172.21.15.1:0/908436822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe728004d40 0x7fe7280051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.099+0000 7fe73f7fe640 1 -- 172.21.15.1:0/908436822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe728003520 con 0x7fe728003f30 2024-03-06T07:22:03.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.099+0000 7fe73ffff640 1 --2- 172.21.15.1:0/908436822 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe740071f90 0x7fe7280039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:03.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.100+0000 7fe73f7fe640 1 --2- 172.21.15.1:0/908436822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe728003f30 0x7fe728004800 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe734009af0 tx=0x7fe728005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9b2220b3513b15 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.100+0000 7fe73effd640 1 -- 172.21.15.1:0/908436822 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734039070 con 0x7fe728003f30 2024-03-06T07:22:03.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.100+0000 7fe73effd640 1 -- 172.21.15.1:0/908436822 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73402c710 con 0x7fe728003f30 2024-03-06T07:22:03.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.100+0000 7fe73effd640 1 -- 172.21.15.1:0/908436822 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734009690 con 0x7fe728003f30 2024-03-06T07:22:03.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.100+0000 7fe74673e640 1 -- 172.21.15.1:0/908436822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe728003f30 msgr2=0x7fe728004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.100+0000 7fe74673e640 1 --2- 172.21.15.1:0/908436822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe728003f30 0x7fe728004800 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe734009af0 tx=0x7fe728005830 comp rx=0 tx=0).stop 2024-03-06T07:22:03.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.100+0000 7fe744cdb640 1 -- 172.21.15.1:0/908436822 reap_dead start 2024-03-06T07:22:03.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.100+0000 7fe74673e640 1 -- 172.21.15.1:0/908436822 shutdown_connections 2024-03-06T07:22:03.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.100+0000 7fe74673e640 1 -- 172.21.15.1:0/908436822 >> 172.21.15.1:0/908436822 conn(0x7fe7400c6860 msgr2=0x7fe7400c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.100+0000 7fe74673e640 1 -- 172.21.15.1:0/908436822 shutdown_connections 2024-03-06T07:22:03.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.100+0000 7fe74673e640 1 -- 172.21.15.1:0/908436822 wait complete. 2024-03-06T07:22:03.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.101+0000 7fe74673e640 1 Processor -- start 2024-03-06T07:22:03.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.101+0000 7fe74673e640 1 -- start start 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.102+0000 7fe74673e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe728003f30 0x7fe7400c46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.102+0000 7fe74673e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe740149820 0x7fe740149ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.102+0000 7fe74673e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7400bbf40 0x7fe7400bc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.102+0000 7fe74673e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74006f4c0 con 0x7fe728003f30 2024-03-06T07:22:03.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.102+0000 7fe74673e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe740152340 con 0x7fe740149820 2024-03-06T07:22:03.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.102+0000 7fe74673e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74006f640 con 0x7fe7400bbf40 2024-03-06T07:22:03.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.102+0000 7fe73ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe728003f30 0x7fe7400c46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.102+0000 7fe73ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe728003f30 0x7fe7400c46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35166/0 (socket says 172.21.15.1:35166) 2024-03-06T07:22:03.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.102+0000 7fe73ffff640 1 -- 172.21.15.1:0/5728504 learned_addr learned my addr 172.21.15.1:0/5728504 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:03.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.102+0000 7fe744cdb640 1 --2- 172.21.15.1:0/5728504 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7400bbf40 0x7fe7400bc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.102+0000 7fe73ffff640 1 -- 172.21.15.1:0/5728504 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7400bbf40 msgr2=0x7fe7400bc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.103+0000 7fe73ffff640 1 --2- 172.21.15.1:0/5728504 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7400bbf40 0x7fe7400bc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.103+0000 7fe73ffff640 1 -- 172.21.15.1:0/5728504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe740149820 msgr2=0x7fe740149ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.102+0000 7fe73f7fe640 1 --2- 172.21.15.1:0/5728504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe740149820 0x7fe740149ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.103+0000 7fe73ffff640 1 --2- 172.21.15.1:0/5728504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe740149820 0x7fe740149ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.103+0000 7fe73ffff640 1 -- 172.21.15.1:0/5728504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7400c06c0 con 0x7fe728003f30 2024-03-06T07:22:03.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.103+0000 7fe73f7fe640 1 --2- 172.21.15.1:0/5728504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe740149820 0x7fe740149ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:03.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.103+0000 7fe73ffff640 1 --2- 172.21.15.1:0/5728504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe728003f30 0x7fe7400c46c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe730001d30 tx=0x7fe730004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.103+0000 7fe73cff9640 1 -- 172.21.15.1:0/5728504 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73000fa80 con 0x7fe728003f30 2024-03-06T07:22:03.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.103+0000 7fe73cff9640 1 -- 172.21.15.1:0/5728504 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe730018070 con 0x7fe728003f30 2024-03-06T07:22:03.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.103+0000 7fe74673e640 1 -- 172.21.15.1:0/5728504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe74014a5c0 con 0x7fe728003f30 2024-03-06T07:22:03.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.103+0000 7fe73cff9640 1 -- 172.21.15.1:0/5728504 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73000fe40 con 0x7fe728003f30 2024-03-06T07:22:03.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.103+0000 7fe74673e640 1 -- 172.21.15.1:0/5728504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7400bba20 con 0x7fe728003f30 2024-03-06T07:22:03.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.104+0000 7fe74673e640 1 -- 172.21.15.1:0/5728504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7401eab30 con 0x7fe728003f30 2024-03-06T07:22:03.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.104+0000 7fe73cff9640 1 -- 172.21.15.1:0/5728504 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe73001d460 con 0x7fe728003f30 2024-03-06T07:22:03.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.105+0000 7fe73cff9640 1 --2- 172.21.15.1:0/5728504 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe72c06b4d0 0x7fe72c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.105+0000 7fe73cff9640 1 -- 172.21.15.1:0/5728504 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fe73008a2f0 con 0x7fe728003f30 2024-03-06T07:22:03.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.105+0000 7fe73f7fe640 1 --2- 172.21.15.1:0/5728504 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe72c06b4d0 0x7fe72c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.106+0000 7fe73f7fe640 1 --2- 172.21.15.1:0/5728504 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe72c06b4d0 0x7fe72c06d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe734036000 tx=0x7fe734030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.109+0000 7fe73cff9640 1 -- 172.21.15.1:0/5728504 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7401eab30 con 0x7fe728003f30 2024-03-06T07:22:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.289+0000 7fe74673e640 1 -- 172.21.15.1:0/5728504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe740067fd0 con 0x7fe728003f30 2024-03-06T07:22:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.289+0000 7fe73cff9640 1 -- 172.21.15.1:0/5728504 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe7300593c0 con 0x7fe728003f30 2024-03-06T07:22:03.291 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.291+0000 7fe74673e640 1 -- 172.21.15.1:0/5728504 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe72c06b4d0 msgr2=0x7fe72c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.291+0000 7fe74673e640 1 --2- 172.21.15.1:0/5728504 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe72c06b4d0 0x7fe72c06d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe734036000 tx=0x7fe734030000 comp rx=0 tx=0).stop 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.291+0000 7fe74673e640 1 -- 172.21.15.1:0/5728504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe728003f30 msgr2=0x7fe7400c46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.291+0000 7fe74673e640 1 --2- 172.21.15.1:0/5728504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe728003f30 0x7fe7400c46c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe730001d30 tx=0x7fe730004f90 comp rx=0 tx=0).stop 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.291+0000 7fe74673e640 1 -- 172.21.15.1:0/5728504 shutdown_connections 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.291+0000 7fe74673e640 1 --2- 172.21.15.1:0/5728504 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe72c06b4d0 0x7fe72c06d990 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.291+0000 7fe74673e640 1 --2- 172.21.15.1:0/5728504 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7400bbf40 0x7fe7400bc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.291+0000 7fe74673e640 1 --2- 172.21.15.1:0/5728504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe740149820 0x7fe740149ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.291+0000 7fe74673e640 1 --2- 172.21.15.1:0/5728504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe728003f30 0x7fe7400c46c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.291+0000 7fe74673e640 1 -- 172.21.15.1:0/5728504 >> 172.21.15.1:0/5728504 conn(0x7fe7400c6860 msgr2=0x7fe74006edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.292+0000 7fe74673e640 1 -- 172.21.15.1:0/5728504 shutdown_connections 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.292+0000 7fe74673e640 1 -- 172.21.15.1:0/5728504 wait complete. 2024-03-06T07:22:03.304 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:22:03.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.404+0000 7fd2a0247640 1 Processor -- start 2024-03-06T07:22:03.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.404+0000 7fd2a0247640 1 -- start start 2024-03-06T07:22:03.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.405+0000 7fd2a0247640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd298152690 0x7fd298155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.405+0000 7fd2a0247640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2980c6730 0x7fd2980c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.405+0000 7fd2a0247640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2981561e0 con 0x7fd2980c5d10 2024-03-06T07:22:03.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.405+0000 7fd2a0247640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2980c3300 con 0x7fd298152690 2024-03-06T07:22:03.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.405+0000 7fd2a0247640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2980c35a0 con 0x7fd2980c6730 2024-03-06T07:22:03.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.405+0000 7fd29dfe3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd298152690 0x7fd298155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.405+0000 7fd29e7e4640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd2980c5d10 0x7fd298156cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57742/0 (socket says 172.21.15.1:57742) 2024-03-06T07:22:03.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.405+0000 7fd29e7e4640 1 -- 172.21.15.1:0/364961071 learned_addr learned my addr 172.21.15.1:0/364961071 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:03.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.405+0000 7fd29d7e2640 1 --2- 172.21.15.1:0/364961071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2980c6730 0x7fd2980c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.405+0000 7fd29dfe3640 1 -- 172.21.15.1:0/364961071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2980c6730 msgr2=0x7fd2980c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.405+0000 7fd29dfe3640 1 --2- 172.21.15.1:0/364961071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2980c6730 0x7fd2980c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.405+0000 7fd29dfe3640 1 -- 172.21.15.1:0/364961071 >> v1:172.21.15.1:6789/0 conn(0x7fd2980c5d10 legacy=0x7fd298156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.405+0000 7fd29dfe3640 1 -- 172.21.15.1:0/364961071 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2980c3880 con 0x7fd298152690 2024-03-06T07:22:03.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.405+0000 7fd29cfe1640 1 -- 172.21.15.1:0/364961071 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706353550 0 0) 0x7fd290001490 con 0x7fd2980c5d10 2024-03-06T07:22:03.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.406+0000 7fd29dfe3640 1 --2- 172.21.15.1:0/364961071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd298152690 0x7fd298155ca0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd2940097a0 tx=0x7fd29402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16163a164226a650 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.406+0000 7fd29cfe1640 1 -- 172.21.15.1:0/364961071 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2940315a0 con 0x7fd298152690 2024-03-06T07:22:03.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.406+0000 7fd29cfe1640 1 -- 172.21.15.1:0/364961071 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd294037070 con 0x7fd298152690 2024-03-06T07:22:03.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.406+0000 7fd29cfe1640 1 -- 172.21.15.1:0/364961071 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd294031960 con 0x7fd298152690 2024-03-06T07:22:03.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.407+0000 7fd2a0247640 1 -- 172.21.15.1:0/364961071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd298152690 msgr2=0x7fd298155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.407+0000 7fd2a0247640 1 --2- 172.21.15.1:0/364961071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd298152690 0x7fd298155ca0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd2940097a0 tx=0x7fd29402b840 comp rx=0 tx=0).stop 2024-03-06T07:22:03.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.407+0000 7fd2a0247640 1 -- 172.21.15.1:0/364961071 shutdown_connections 2024-03-06T07:22:03.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.407+0000 7fd2a0247640 1 --2- 172.21.15.1:0/364961071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2980c6730 0x7fd2980c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.407+0000 7fd2a0247640 1 --2- 172.21.15.1:0/364961071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd298152690 0x7fd298155ca0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.407+0000 7fd2a0247640 1 -- 172.21.15.1:0/364961071 >> 172.21.15.1:0/364961071 conn(0x7fd29806b410 msgr2=0x7fd29806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.407+0000 7fd2a0247640 1 -- 172.21.15.1:0/364961071 shutdown_connections 2024-03-06T07:22:03.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.407+0000 7fd2a0247640 1 -- 172.21.15.1:0/364961071 wait complete. 2024-03-06T07:22:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.408+0000 7fd2a0247640 1 Processor -- start 2024-03-06T07:22:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.408+0000 7fd2a0247640 1 -- start start 2024-03-06T07:22:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd2a0247640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2980c5d10 0x7fd2980a9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd2a0247640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2980c6730 0x7fd2981ee910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd2a0247640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd298152690 0x7fd2981e0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd2a0247640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd298152a70 con 0x7fd2980c6730 2024-03-06T07:22:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd2a0247640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2981567b0 con 0x7fd2980c5d10 2024-03-06T07:22:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd2a0247640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd298152bf0 con 0x7fd298152690 2024-03-06T07:22:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd29d7e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2980c6730 0x7fd2981ee910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd29d7e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2980c6730 0x7fd2981ee910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35180/0 (socket says 172.21.15.1:35180) 2024-03-06T07:22:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd29d7e2640 1 -- 172.21.15.1:0/4050141450 learned_addr learned my addr 172.21.15.1:0/4050141450 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd29dfe3640 1 --2- 172.21.15.1:0/4050141450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2980c5d10 0x7fd2980a9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd29e7e4640 1 --2- 172.21.15.1:0/4050141450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd298152690 0x7fd2981e0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd29d7e2640 1 -- 172.21.15.1:0/4050141450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd298152690 msgr2=0x7fd2981e0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd29d7e2640 1 --2- 172.21.15.1:0/4050141450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd298152690 0x7fd2981e0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd29d7e2640 1 -- 172.21.15.1:0/4050141450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2980c5d10 msgr2=0x7fd2980a9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd29d7e2640 1 --2- 172.21.15.1:0/4050141450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2980c5d10 0x7fd2980a9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd29d7e2640 1 -- 172.21.15.1:0/4050141450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2980a8eb0 con 0x7fd2980c6730 2024-03-06T07:22:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd29d7e2640 1 --2- 172.21.15.1:0/4050141450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2980c6730 0x7fd2981ee910 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd28800d5f0 tx=0x7fd28800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd28e7fc640 1 -- 172.21.15.1:0/4050141450 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2880125f0 con 0x7fd2980c6730 2024-03-06T07:22:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd28e7fc640 1 -- 172.21.15.1:0/4050141450 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd288019070 con 0x7fd2980c6730 2024-03-06T07:22:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd2a0247640 1 -- 172.21.15.1:0/4050141450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2981eee50 con 0x7fd2980c6730 2024-03-06T07:22:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.409+0000 7fd28e7fc640 1 -- 172.21.15.1:0/4050141450 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2880129d0 con 0x7fd2980c6730 2024-03-06T07:22:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.410+0000 7fd2a0247640 1 -- 172.21.15.1:0/4050141450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2980a88f0 con 0x7fd2980c6730 2024-03-06T07:22:03.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.410+0000 7fd2a0247640 1 -- 172.21.15.1:0/4050141450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd298067fd0 con 0x7fd2980c6730 2024-03-06T07:22:03.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.411+0000 7fd28e7fc640 1 -- 172.21.15.1:0/4050141450 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd288012b70 con 0x7fd2980c6730 2024-03-06T07:22:03.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.411+0000 7fd28e7fc640 1 --2- 172.21.15.1:0/4050141450 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd27c06b4d0 0x7fd27c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.411+0000 7fd28e7fc640 1 -- 172.21.15.1:0/4050141450 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd28808a220 con 0x7fd2980c6730 2024-03-06T07:22:03.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.412+0000 7fd29dfe3640 1 --2- 172.21.15.1:0/4050141450 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd27c06b4d0 0x7fd27c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.412+0000 7fd29dfe3640 1 --2- 172.21.15.1:0/4050141450 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd27c06b4d0 0x7fd27c06d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd294036000 tx=0x7fd294008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.416+0000 7fd28e7fc640 1 -- 172.21.15.1:0/4050141450 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2980a8eb0 con 0x7fd2980c6730 2024-03-06T07:22:03.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.598+0000 7fd2a0247640 1 -- 172.21.15.1:0/4050141450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd29806df20 con 0x7fd2980c6730 2024-03-06T07:22:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.599+0000 7fd28e7fc640 1 -- 172.21.15.1:0/4050141450 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v48) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd288058ca0 con 0x7fd2980c6730 2024-03-06T07:22:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:03.601 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-06T07:22:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-06T07:22:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-06T07:22:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:03.601 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-06T07:22:03.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.600+0000 7fd2a0247640 1 -- 172.21.15.1:0/4050141450 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd27c06b4d0 msgr2=0x7fd27c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.601+0000 7fd2a0247640 1 --2- 172.21.15.1:0/4050141450 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd27c06b4d0 0x7fd27c06d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd294036000 tx=0x7fd294008030 comp rx=0 tx=0).stop 2024-03-06T07:22:03.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.601+0000 7fd2a0247640 1 -- 172.21.15.1:0/4050141450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2980c6730 msgr2=0x7fd2981ee910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.601+0000 7fd2a0247640 1 --2- 172.21.15.1:0/4050141450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2980c6730 0x7fd2981ee910 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd28800d5f0 tx=0x7fd28800dab0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.601+0000 7fd2a0247640 1 -- 172.21.15.1:0/4050141450 shutdown_connections 2024-03-06T07:22:03.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.601+0000 7fd2a0247640 1 --2- 172.21.15.1:0/4050141450 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd27c06b4d0 0x7fd27c06d990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.601+0000 7fd2a0247640 1 --2- 172.21.15.1:0/4050141450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd298152690 0x7fd2981e0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.601+0000 7fd2a0247640 1 --2- 172.21.15.1:0/4050141450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2980c6730 0x7fd2981ee910 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.601+0000 7fd2a0247640 1 --2- 172.21.15.1:0/4050141450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2980c5d10 0x7fd2980a9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.601+0000 7fd2a0247640 1 -- 172.21.15.1:0/4050141450 >> 172.21.15.1:0/4050141450 conn(0x7fd29806b410 msgr2=0x7fd2980c87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.601+0000 7fd2a0247640 1 -- 172.21.15.1:0/4050141450 shutdown_connections 2024-03-06T07:22:03.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.601+0000 7fd2a0247640 1 -- 172.21.15.1:0/4050141450 wait complete. 2024-03-06T07:22:03.613 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:22:03.613 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:22:03.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.717+0000 7fbe0545c640 1 Processor -- start 2024-03-06T07:22:03.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.717+0000 7fbe0545c640 1 -- start start 2024-03-06T07:22:03.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbe0545c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe000c1e90 0x7fbe00153b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbe0545c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe0006dab0 con 0x7fbe000c5dd0 2024-03-06T07:22:03.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbe0545c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe000c2db0 con 0x7fbe000c4940 2024-03-06T07:22:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbe0545c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe001540c0 con 0x7fbe000c1e90 2024-03-06T07:22:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdfeffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbe000c5dd0 0x7fbe000c1340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57752/0 (socket says 172.21.15.1:57752) 2024-03-06T07:22:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdfeffd640 1 -- 172.21.15.1:0/1845783331 learned_addr learned my addr 172.21.15.1:0/1845783331 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdfe7fc640 1 -- 172.21.15.1:0/1845783331 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052917082 0 0) 0x7fbe0006dab0 con 0x7fbe000c5dd0 2024-03-06T07:22:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdf7fff640 1 --2- 172.21.15.1:0/1845783331 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe000c1e90 0x7fbe00153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdfe7fc640 1 -- 172.21.15.1:0/1845783331 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdec003520 con 0x7fbe000c5dd0 2024-03-06T07:22:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdfe7fc640 1 -- 172.21.15.1:0/1845783331 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 615829912 0 0) 0x7fbdec003520 con 0x7fbe000c5dd0 2024-03-06T07:22:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdfe7fc640 1 -- 172.21.15.1:0/1845783331 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe000c1e90 msgr2=0x7fbe00153b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdfe7fc640 1 --2- 172.21.15.1:0/1845783331 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe000c1e90 0x7fbe00153b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdfe7fc640 1 -- 172.21.15.1:0/1845783331 >> v1:172.21.15.98:6789/0 conn(0x7fbe000c4940 legacy=0x7fbe000c4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdfe7fc640 1 -- 172.21.15.1:0/1845783331 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe000c3090 con 0x7fbe000c5dd0 2024-03-06T07:22:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdfe7fc640 1 -- 172.21.15.1:0/1845783331 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fbdf0001670 con 0x7fbe000c5dd0 2024-03-06T07:22:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdfe7fc640 1 -- 172.21.15.1:0/1845783331 >> v1:172.21.15.1:6789/0 conn(0x7fbe000c5dd0 legacy=0x7fbe000c1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdfe7fc640 1 --2- 172.21.15.1:0/1845783331 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe000c1e90 0x7fbdec0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdfe7fc640 1 --2- 172.21.15.1:0/1845783331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbdec003f60 0x7fbdec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdfe7fc640 1 --2- 172.21.15.1:0/1845783331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdec004d40 0x7fbdec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdfe7fc640 1 -- 172.21.15.1:0/1845783331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe001540c0 con 0x7fbdec004d40 2024-03-06T07:22:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdfe7fc640 1 -- 172.21.15.1:0/1845783331 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec00bb50 con 0x7fbdec003f60 2024-03-06T07:22:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdfe7fc640 1 -- 172.21.15.1:0/1845783331 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec00bd50 con 0x7fbe000c1e90 2024-03-06T07:22:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.718+0000 7fbdff7fe640 1 --2- 172.21.15.1:0/1845783331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdec004d40 0x7fbdec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.719+0000 7fbdfeffd640 1 --2- 172.21.15.1:0/1845783331 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe000c1e90 0x7fbdec0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.719+0000 7fbdf7fff640 1 --2- 172.21.15.1:0/1845783331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbdec003f60 0x7fbdec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.719+0000 7fbdff7fe640 1 -- 172.21.15.1:0/1845783331 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe000c1e90 msgr2=0x7fbdec0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.719+0000 7fbdff7fe640 1 --2- 172.21.15.1:0/1845783331 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe000c1e90 0x7fbdec0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.719+0000 7fbdff7fe640 1 -- 172.21.15.1:0/1845783331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbdec003f60 msgr2=0x7fbdec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.719+0000 7fbdff7fe640 1 --2- 172.21.15.1:0/1845783331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbdec003f60 0x7fbdec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.719+0000 7fbdff7fe640 1 -- 172.21.15.1:0/1845783331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec003520 con 0x7fbdec004d40 2024-03-06T07:22:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.720+0000 7fbdff7fe640 1 --2- 172.21.15.1:0/1845783331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdec004d40 0x7fbdec0051c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8001e50 tx=0x7fbdec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a71296ccce97fc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.720+0000 7fbdfe7fc640 1 -- 172.21.15.1:0/1845783331 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf8001a70 con 0x7fbdec004d40 2024-03-06T07:22:03.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.720+0000 7fbdfe7fc640 1 -- 172.21.15.1:0/1845783331 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf8003b70 con 0x7fbdec004d40 2024-03-06T07:22:03.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.720+0000 7fbdfe7fc640 1 -- 172.21.15.1:0/1845783331 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf80050d0 con 0x7fbdec004d40 2024-03-06T07:22:03.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.720+0000 7fbe0545c640 1 -- 172.21.15.1:0/1845783331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdec004d40 msgr2=0x7fbdec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.720+0000 7fbe0545c640 1 --2- 172.21.15.1:0/1845783331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdec004d40 0x7fbdec0051c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8001e50 tx=0x7fbdec005830 comp rx=0 tx=0).stop 2024-03-06T07:22:03.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.720+0000 7fbdff7fe640 1 -- 172.21.15.1:0/1845783331 reap_dead start 2024-03-06T07:22:03.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.720+0000 7fbe0545c640 1 -- 172.21.15.1:0/1845783331 shutdown_connections 2024-03-06T07:22:03.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.720+0000 7fbe0545c640 1 -- 172.21.15.1:0/1845783331 >> 172.21.15.1:0/1845783331 conn(0x7fbe000a8e10 msgr2=0x7fbe000c3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.720+0000 7fbe0545c640 1 -- 172.21.15.1:0/1845783331 shutdown_connections 2024-03-06T07:22:03.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.720+0000 7fbe0545c640 1 -- 172.21.15.1:0/1845783331 wait complete. 2024-03-06T07:22:03.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.721+0000 7fbe0545c640 1 Processor -- start 2024-03-06T07:22:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.721+0000 7fbe0545c640 1 -- start start 2024-03-06T07:22:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.722+0000 7fbe0545c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe001dfa70 0x7fbe001dfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.722+0000 7fbe0545c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe001572f0 0x7fbe001496d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.722+0000 7fbe0545c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe00149c10 0x7fbe000a9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.722+0000 7fbe0545c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe000c61b0 con 0x7fbe00149c10 2024-03-06T07:22:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.722+0000 7fbe0545c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0006e050 con 0x7fbe001dfa70 2024-03-06T07:22:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.722+0000 7fbe0545c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe000c1a50 con 0x7fbe001572f0 2024-03-06T07:22:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.722+0000 7fbdff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe00149c10 0x7fbe000a9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.722+0000 7fbdff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe00149c10 0x7fbe000a9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35204/0 (socket says 172.21.15.1:35204) 2024-03-06T07:22:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.722+0000 7fbdff7fe640 1 -- 172.21.15.1:0/404579111 learned_addr learned my addr 172.21.15.1:0/404579111 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.722+0000 7fbdf7fff640 1 --2- 172.21.15.1:0/404579111 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe001572f0 0x7fbe001496d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.722+0000 7fbdfeffd640 1 --2- 172.21.15.1:0/404579111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe001dfa70 0x7fbe001dfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.722+0000 7fbdff7fe640 1 -- 172.21.15.1:0/404579111 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe001572f0 msgr2=0x7fbe001496d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.722+0000 7fbdff7fe640 1 --2- 172.21.15.1:0/404579111 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe001572f0 0x7fbe001496d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.722+0000 7fbdff7fe640 1 -- 172.21.15.1:0/404579111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe001dfa70 msgr2=0x7fbe001dfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.722+0000 7fbdff7fe640 1 --2- 172.21.15.1:0/404579111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe001dfa70 0x7fbe001dfe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.723+0000 7fbdff7fe640 1 -- 172.21.15.1:0/404579111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0014a650 con 0x7fbe00149c10 2024-03-06T07:22:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.723+0000 7fbdff7fe640 1 --2- 172.21.15.1:0/404579111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe00149c10 0x7fbe000a9a90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8001390 tx=0x7fbdf80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.723+0000 7fbdf77fe640 1 -- 172.21.15.1:0/404579111 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf8007db0 con 0x7fbe00149c10 2024-03-06T07:22:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.723+0000 7fbdf77fe640 1 -- 172.21.15.1:0/404579111 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf8003970 con 0x7fbe00149c10 2024-03-06T07:22:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.723+0000 7fbe0545c640 1 -- 172.21.15.1:0/404579111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe001e9de0 con 0x7fbe00149c10 2024-03-06T07:22:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.723+0000 7fbdf77fe640 1 -- 172.21.15.1:0/404579111 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf802f660 con 0x7fbe00149c10 2024-03-06T07:22:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.723+0000 7fbe0545c640 1 -- 172.21.15.1:0/404579111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0014a1a0 con 0x7fbe00149c10 2024-03-06T07:22:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.724+0000 7fbe0545c640 1 -- 172.21.15.1:0/404579111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe000680b0 con 0x7fbe00149c10 2024-03-06T07:22:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.724+0000 7fbdf77fe640 1 -- 172.21.15.1:0/404579111 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbdf8041030 con 0x7fbe00149c10 2024-03-06T07:22:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.725+0000 7fbdf77fe640 1 --2- 172.21.15.1:0/404579111 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbde006b500 0x7fbde006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.725+0000 7fbdf77fe640 1 -- 172.21.15.1:0/404579111 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbdf80aaf90 con 0x7fbe00149c10 2024-03-06T07:22:03.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.725+0000 7fbdfeffd640 1 --2- 172.21.15.1:0/404579111 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbde006b500 0x7fbde006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.725+0000 7fbdfeffd640 1 --2- 172.21.15.1:0/404579111 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbde006b500 0x7fbde006d9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0001050 tx=0x7fbdf0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.729+0000 7fbdf77fe640 1 -- 172.21.15.1:0/404579111 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbdf8033020 con 0x7fbe00149c10 2024-03-06T07:22:03.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:03.910+0000 7fbe0545c640 1 -- 172.21.15.1:0/404579111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fbe0006ed80 con 0x7fbe00149c10 2024-03-06T07:22:04.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.293+0000 7fbdf77fe640 1 -- 172.21.15.1:0/404579111 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v49) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbdf8079fd0 con 0x7fbe00149c10 2024-03-06T07:22:04.295 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-06T07:22:04.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.296+0000 7fbe0545c640 1 -- 172.21.15.1:0/404579111 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbde006b500 msgr2=0x7fbde006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.296+0000 7fbe0545c640 1 --2- 172.21.15.1:0/404579111 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbde006b500 0x7fbde006d9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0001050 tx=0x7fbdf0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:04.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.297+0000 7fbe0545c640 1 -- 172.21.15.1:0/404579111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe00149c10 msgr2=0x7fbe000a9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.297+0000 7fbe0545c640 1 --2- 172.21.15.1:0/404579111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe00149c10 0x7fbe000a9a90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8001390 tx=0x7fbdf80014a0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.297+0000 7fbe0545c640 1 -- 172.21.15.1:0/404579111 shutdown_connections 2024-03-06T07:22:04.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.297+0000 7fbe0545c640 1 --2- 172.21.15.1:0/404579111 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbde006b500 0x7fbde006d9c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.297+0000 7fbe0545c640 1 --2- 172.21.15.1:0/404579111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe00149c10 0x7fbe000a9a90 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.297+0000 7fbe0545c640 1 --2- 172.21.15.1:0/404579111 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe001572f0 0x7fbe001496d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.297+0000 7fbe0545c640 1 --2- 172.21.15.1:0/404579111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe001dfa70 0x7fbe001dfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.297+0000 7fbe0545c640 1 -- 172.21.15.1:0/404579111 >> 172.21.15.1:0/404579111 conn(0x7fbe000a8e10 msgr2=0x7fbe000c1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.297+0000 7fbe0545c640 1 -- 172.21.15.1:0/404579111 shutdown_connections 2024-03-06T07:22:04.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.297+0000 7fbe0545c640 1 -- 172.21.15.1:0/404579111 wait complete. 2024-03-06T07:22:04.313 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:22:04.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.422+0000 7fc68481c640 1 Processor -- start 2024-03-06T07:22:04.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.422+0000 7fc68481c640 1 -- start start 2024-03-06T07:22:04.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.422+0000 7fc68481c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc68014cdc0 0x7fc680152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.422+0000 7fc68481c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc680156500 0x7fc6800c2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.422+0000 7fc68481c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6801569e0 con 0x7fc6800c2890 2024-03-06T07:22:04.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.422+0000 7fc68481c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6800c2dd0 con 0x7fc68014cdc0 2024-03-06T07:22:04.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.422+0000 7fc68481c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc680156cc0 con 0x7fc680156500 2024-03-06T07:22:04.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.423+0000 7fc67e59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc68014cdc0 0x7fc680152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.423+0000 7fc67ed9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc6800c2890 0x7fc68006b510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45302/0 (socket says 172.21.15.1:45302) 2024-03-06T07:22:04.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.423+0000 7fc67ed9d640 1 -- 172.21.15.1:0/1945533707 learned_addr learned my addr 172.21.15.1:0/1945533707 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:04.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.423+0000 7fc67e59c640 1 -- 172.21.15.1:0/1945533707 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc680156500 msgr2=0x7fc6800c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.423+0000 7fc67dd9b640 1 --2- 172.21.15.1:0/1945533707 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc680156500 0x7fc6800c2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.423+0000 7fc67e59c640 1 --2- 172.21.15.1:0/1945533707 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc680156500 0x7fc6800c2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.423+0000 7fc67e59c640 1 -- 172.21.15.1:0/1945533707 >> v1:172.21.15.1:6789/0 conn(0x7fc6800c2890 legacy=0x7fc68006b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.423+0000 7fc67d59a640 1 -- 172.21.15.1:0/1945533707 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808917376 0 0) 0x7fc6801569e0 con 0x7fc6800c2890 2024-03-06T07:22:04.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.423+0000 7fc67e59c640 1 -- 172.21.15.1:0/1945533707 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc680156fa0 con 0x7fc68014cdc0 2024-03-06T07:22:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.424+0000 7fc67e59c640 1 --2- 172.21.15.1:0/1945533707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68014cdc0 0x7fc680152f10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc670009d90 tx=0x7fc67002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c48eaefa54cbdc5f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.424+0000 7fc67d59a640 1 -- 172.21.15.1:0/1945533707 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670039070 con 0x7fc68014cdc0 2024-03-06T07:22:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.424+0000 7fc67d59a640 1 -- 172.21.15.1:0/1945533707 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67002cc50 con 0x7fc68014cdc0 2024-03-06T07:22:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.424+0000 7fc67d59a640 1 -- 172.21.15.1:0/1945533707 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670034440 con 0x7fc68014cdc0 2024-03-06T07:22:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.424+0000 7fc68481c640 1 -- 172.21.15.1:0/1945533707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68014cdc0 msgr2=0x7fc680152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.424+0000 7fc68481c640 1 --2- 172.21.15.1:0/1945533707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68014cdc0 0x7fc680152f10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc670009d90 tx=0x7fc67002bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.424+0000 7fc68481c640 1 -- 172.21.15.1:0/1945533707 shutdown_connections 2024-03-06T07:22:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.424+0000 7fc68481c640 1 --2- 172.21.15.1:0/1945533707 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc680156500 0x7fc6800c2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.424+0000 7fc68481c640 1 --2- 172.21.15.1:0/1945533707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68014cdc0 0x7fc680152f10 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.424+0000 7fc68481c640 1 -- 172.21.15.1:0/1945533707 >> 172.21.15.1:0/1945533707 conn(0x7fc6800a9700 msgr2=0x7fc6800a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.424+0000 7fc68481c640 1 -- 172.21.15.1:0/1945533707 shutdown_connections 2024-03-06T07:22:04.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.425+0000 7fc68481c640 1 -- 172.21.15.1:0/1945533707 wait complete. 2024-03-06T07:22:04.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.426+0000 7fc68481c640 1 Processor -- start 2024-03-06T07:22:04.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.426+0000 7fc68481c640 1 -- start start 2024-03-06T07:22:04.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.426+0000 7fc68481c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6800c2890 0x7fc680145fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.426+0000 7fc68481c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68014cdc0 0x7fc6801494b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.426+0000 7fc68481c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc680156500 0x7fc6801ec220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.426+0000 7fc68481c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6800c65e0 con 0x7fc68014cdc0 2024-03-06T07:22:04.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.426+0000 7fc68481c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6800c5a70 con 0x7fc6800c2890 2024-03-06T07:22:04.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.426+0000 7fc68481c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc68014d1a0 con 0x7fc680156500 2024-03-06T07:22:04.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.426+0000 7fc67dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68014cdc0 0x7fc6801494b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.427+0000 7fc67dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68014cdc0 0x7fc6801494b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59366/0 (socket says 172.21.15.1:59366) 2024-03-06T07:22:04.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.427+0000 7fc67dd9b640 1 -- 172.21.15.1:0/1095661325 learned_addr learned my addr 172.21.15.1:0/1095661325 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:04.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.427+0000 7fc67ed9d640 1 --2- 172.21.15.1:0/1095661325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc680156500 0x7fc6801ec220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.427+0000 7fc67e59c640 1 --2- 172.21.15.1:0/1095661325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6800c2890 0x7fc680145fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.427+0000 7fc67dd9b640 1 -- 172.21.15.1:0/1095661325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc680156500 msgr2=0x7fc6801ec220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.427+0000 7fc67dd9b640 1 --2- 172.21.15.1:0/1095661325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc680156500 0x7fc6801ec220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.427+0000 7fc67dd9b640 1 -- 172.21.15.1:0/1095661325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6800c2890 msgr2=0x7fc680145fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.427+0000 7fc67dd9b640 1 --2- 172.21.15.1:0/1095661325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6800c2890 0x7fc680145fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.427+0000 7fc67dd9b640 1 -- 172.21.15.1:0/1095661325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6800acee0 con 0x7fc68014cdc0 2024-03-06T07:22:04.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.427+0000 7fc67e59c640 1 --2- 172.21.15.1:0/1095661325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6800c2890 0x7fc680145fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:04.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.427+0000 7fc67dd9b640 1 --2- 172.21.15.1:0/1095661325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68014cdc0 0x7fc6801494b0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc66800c820 tx=0x7fc66800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.427+0000 7fc676ffd640 1 -- 172.21.15.1:0/1095661325 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc668015070 con 0x7fc68014cdc0 2024-03-06T07:22:04.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.427+0000 7fc676ffd640 1 -- 172.21.15.1:0/1095661325 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc668004240 con 0x7fc68014cdc0 2024-03-06T07:22:04.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.427+0000 7fc68481c640 1 -- 172.21.15.1:0/1095661325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc680151e70 con 0x7fc68014cdc0 2024-03-06T07:22:04.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.427+0000 7fc676ffd640 1 -- 172.21.15.1:0/1095661325 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66800ce10 con 0x7fc68014cdc0 2024-03-06T07:22:04.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.427+0000 7fc68481c640 1 -- 172.21.15.1:0/1095661325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc680152430 con 0x7fc68014cdc0 2024-03-06T07:22:04.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.428+0000 7fc68481c640 1 -- 172.21.15.1:0/1095661325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc680067fd0 con 0x7fc68014cdc0 2024-03-06T07:22:04.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.429+0000 7fc676ffd640 1 -- 172.21.15.1:0/1095661325 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc668007830 con 0x7fc68014cdc0 2024-03-06T07:22:04.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.429+0000 7fc676ffd640 1 --2- 172.21.15.1:0/1095661325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc66006b4d0 0x7fc66006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.429+0000 7fc676ffd640 1 -- 172.21.15.1:0/1095661325 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fc668089ee0 con 0x7fc68014cdc0 2024-03-06T07:22:04.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.429+0000 7fc67e59c640 1 --2- 172.21.15.1:0/1095661325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc66006b4d0 0x7fc66006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.430+0000 7fc67e59c640 1 --2- 172.21.15.1:0/1095661325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc66006b4d0 0x7fc66006d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc670036000 tx=0x7fc670030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.434+0000 7fc676ffd640 1 -- 172.21.15.1:0/1095661325 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6680587f0 con 0x7fc68014cdc0 2024-03-06T07:22:04.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:04.615+0000 7fc68481c640 1 -- 172.21.15.1:0/1095661325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc6800a8670 con 0x7fc68014cdc0 2024-03-06T07:22:05.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.298+0000 7fc676ffd640 1 -- 172.21.15.1:0/1095661325 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v50) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc66805c890 con 0x7fc68014cdc0 2024-03-06T07:22:05.300 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-06T07:22:05.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.301+0000 7fc68481c640 1 -- 172.21.15.1:0/1095661325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc66006b4d0 msgr2=0x7fc66006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.301+0000 7fc68481c640 1 --2- 172.21.15.1:0/1095661325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc66006b4d0 0x7fc66006d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc670036000 tx=0x7fc670030000 comp rx=0 tx=0).stop 2024-03-06T07:22:05.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.301+0000 7fc68481c640 1 -- 172.21.15.1:0/1095661325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68014cdc0 msgr2=0x7fc6801494b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.301+0000 7fc68481c640 1 --2- 172.21.15.1:0/1095661325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68014cdc0 0x7fc6801494b0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc66800c820 tx=0x7fc66800cce0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.301+0000 7fc68481c640 1 -- 172.21.15.1:0/1095661325 shutdown_connections 2024-03-06T07:22:05.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.301+0000 7fc68481c640 1 --2- 172.21.15.1:0/1095661325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc66006b4d0 0x7fc66006d990 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.301+0000 7fc68481c640 1 --2- 172.21.15.1:0/1095661325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc680156500 0x7fc6801ec220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.301+0000 7fc68481c640 1 --2- 172.21.15.1:0/1095661325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68014cdc0 0x7fc6801494b0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.301+0000 7fc68481c640 1 --2- 172.21.15.1:0/1095661325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6800c2890 0x7fc680145fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.301+0000 7fc68481c640 1 -- 172.21.15.1:0/1095661325 >> 172.21.15.1:0/1095661325 conn(0x7fc6800a9700 msgr2=0x7fc6800a8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.301+0000 7fc68481c640 1 -- 172.21.15.1:0/1095661325 shutdown_connections 2024-03-06T07:22:05.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.301+0000 7fc68481c640 1 -- 172.21.15.1:0/1095661325 wait complete. 2024-03-06T07:22:05.331 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:22:05.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.438+0000 7f4110a4e640 1 Processor -- start 2024-03-06T07:22:05.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.438+0000 7f4110a4e640 1 -- start start 2024-03-06T07:22:05.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.439+0000 7f4110a4e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f410c0a7f50 0x7f410c154300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.439+0000 7f4110a4e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c0c8120 con 0x7f410c0c75a0 2024-03-06T07:22:05.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.439+0000 7f4110a4e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c0c7b60 con 0x7f410c150640 2024-03-06T07:22:05.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.439+0000 7f4110a4e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f410c153e30 con 0x7f410c0a7f50 2024-03-06T07:22:05.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.439+0000 7f4109d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f410c0a7f50 0x7f410c154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.439+0000 7f410ad9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f410c0c75a0 0x7f410c0c8d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45316/0 (socket says 172.21.15.1:45316) 2024-03-06T07:22:05.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.439+0000 7f410ad9d640 1 -- 172.21.15.1:0/2558589347 learned_addr learned my addr 172.21.15.1:0/2558589347 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:05.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.439+0000 7f4109d9b640 1 -- 172.21.15.1:0/2558589347 >> v1:172.21.15.98:6789/0 conn(0x7f410c150640 legacy=0x7f410c153720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.440+0000 7f4109d9b640 1 -- 172.21.15.1:0/2558589347 >> v1:172.21.15.1:6789/0 conn(0x7f410c0c75a0 legacy=0x7f410c0c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.440+0000 7f4109d9b640 1 -- 172.21.15.1:0/2558589347 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c0c7e40 con 0x7f410c0a7f50 2024-03-06T07:22:05.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.440+0000 7f410959a640 1 -- 172.21.15.1:0/2558589347 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854327948 0 0) 0x7f410c0c8120 con 0x7f410c0c75a0 2024-03-06T07:22:05.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.440+0000 7f4109d9b640 1 --2- 172.21.15.1:0/2558589347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f410c0a7f50 0x7f410c154300 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f40f40097a0 tx=0x7f40f402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1b16cd508396b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.440+0000 7f410959a640 1 -- 172.21.15.1:0/2558589347 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f40315a0 con 0x7f410c0a7f50 2024-03-06T07:22:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.440+0000 7f410959a640 1 -- 172.21.15.1:0/2558589347 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f4037070 con 0x7f410c0a7f50 2024-03-06T07:22:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.440+0000 7f410959a640 1 -- 172.21.15.1:0/2558589347 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f4031960 con 0x7f410c0a7f50 2024-03-06T07:22:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.441+0000 7f4110a4e640 1 -- 172.21.15.1:0/2558589347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f410c0a7f50 msgr2=0x7f410c154300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.441+0000 7f4110a4e640 1 --2- 172.21.15.1:0/2558589347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f410c0a7f50 0x7f410c154300 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f40f40097a0 tx=0x7f40f402b840 comp rx=0 tx=0).stop 2024-03-06T07:22:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.441+0000 7f4110a4e640 1 -- 172.21.15.1:0/2558589347 shutdown_connections 2024-03-06T07:22:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.441+0000 7f4110a4e640 1 --2- 172.21.15.1:0/2558589347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f410c0a7f50 0x7f410c154300 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.441+0000 7f4110a4e640 1 -- 172.21.15.1:0/2558589347 >> 172.21.15.1:0/2558589347 conn(0x7f410c06b3e0 msgr2=0x7f410c06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.441+0000 7f4110a4e640 1 -- 172.21.15.1:0/2558589347 shutdown_connections 2024-03-06T07:22:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.441+0000 7f4110a4e640 1 -- 172.21.15.1:0/2558589347 wait complete. 2024-03-06T07:22:05.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.442+0000 7f4110a4e640 1 Processor -- start 2024-03-06T07:22:05.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.442+0000 7f4110a4e640 1 -- start start 2024-03-06T07:22:05.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.443+0000 7f4110a4e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f410c0a7f50 0x7f410c0c1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.443+0000 7f4110a4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f410c0c75a0 0x7f410c1ede90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.443+0000 7f4110a4e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f410c150640 0x7f410c1ee3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.443+0000 7f4110a4e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410c150a20 con 0x7f410c0a7f50 2024-03-06T07:22:05.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.443+0000 7f4110a4e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410c153200 con 0x7f410c0c75a0 2024-03-06T07:22:05.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.443+0000 7f4110a4e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f410c071790 con 0x7f410c150640 2024-03-06T07:22:05.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.443+0000 7f410a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f410c0a7f50 0x7f410c0c1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.443+0000 7f410a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f410c0a7f50 0x7f410c0c1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59386/0 (socket says 172.21.15.1:59386) 2024-03-06T07:22:05.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.443+0000 7f410a59c640 1 -- 172.21.15.1:0/3469211347 learned_addr learned my addr 172.21.15.1:0/3469211347 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:05.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.443+0000 7f410a59c640 1 -- 172.21.15.1:0/3469211347 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f410c150640 msgr2=0x7f410c1ee3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.443+0000 7f410a59c640 1 --2- 172.21.15.1:0/3469211347 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f410c150640 0x7f410c1ee3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.443+0000 7f410a59c640 1 -- 172.21.15.1:0/3469211347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f410c0c75a0 msgr2=0x7f410c1ede90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.443+0000 7f410a59c640 1 --2- 172.21.15.1:0/3469211347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f410c0c75a0 0x7f410c1ede90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.443+0000 7f410a59c640 1 -- 172.21.15.1:0/3469211347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c14bdb0 con 0x7f410c0a7f50 2024-03-06T07:22:05.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.444+0000 7f410a59c640 1 --2- 172.21.15.1:0/3469211347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f410c0a7f50 0x7f410c0c1c90 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f410400c480 tx=0x7f410400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.444+0000 7f4102ffd640 1 -- 172.21.15.1:0/3469211347 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41040052d0 con 0x7f410c0a7f50 2024-03-06T07:22:05.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.444+0000 7f4102ffd640 1 -- 172.21.15.1:0/3469211347 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4104005470 con 0x7f410c0a7f50 2024-03-06T07:22:05.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.444+0000 7f4110a4e640 1 -- 172.21.15.1:0/3469211347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410c14c070 con 0x7f410c0a7f50 2024-03-06T07:22:05.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.444+0000 7f4102ffd640 1 -- 172.21.15.1:0/3469211347 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104012870 con 0x7f410c0a7f50 2024-03-06T07:22:05.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.444+0000 7f4110a4e640 1 -- 172.21.15.1:0/3469211347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f410c0aa120 con 0x7f410c0a7f50 2024-03-06T07:22:05.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.445+0000 7f4110a4e640 1 -- 172.21.15.1:0/3469211347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f410c067fd0 con 0x7f410c0a7f50 2024-03-06T07:22:05.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.445+0000 7f4102ffd640 1 -- 172.21.15.1:0/3469211347 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4104012a10 con 0x7f410c0a7f50 2024-03-06T07:22:05.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.446+0000 7f4102ffd640 1 --2- 172.21.15.1:0/3469211347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f40ec06b4d0 0x7f40ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.446+0000 7f4102ffd640 1 -- 172.21.15.1:0/3469211347 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f4104089f30 con 0x7f410c0a7f50 2024-03-06T07:22:05.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.446+0000 7f4109d9b640 1 --2- 172.21.15.1:0/3469211347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f40ec06b4d0 0x7f40ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.446+0000 7f4109d9b640 1 --2- 172.21.15.1:0/3469211347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f40ec06b4d0 0x7f40ec06d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f40f4036000 tx=0x7f40f4030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.450+0000 7f4102ffd640 1 -- 172.21.15.1:0/3469211347 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4104058d10 con 0x7f410c0a7f50 2024-03-06T07:22:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:05.669+0000 7f4110a4e640 1 -- 172.21.15.1:0/3469211347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f410c0c2cf0 con 0x7f410c0a7f50 2024-03-06T07:22:06.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.300+0000 7f4102ffd640 1 -- 172.21.15.1:0/3469211347 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v26) v1 ==== 172+0+0 (secure 0 0 0) 0x7f4104058610 con 0x7f410c0a7f50 2024-03-06T07:22:06.302 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 7 and data pool 8 2024-03-06T07:22:06.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.304+0000 7f4110a4e640 1 -- 172.21.15.1:0/3469211347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f40ec06b4d0 msgr2=0x7f40ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.304+0000 7f4110a4e640 1 --2- 172.21.15.1:0/3469211347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f40ec06b4d0 0x7f40ec06d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f40f4036000 tx=0x7f40f4030000 comp rx=0 tx=0).stop 2024-03-06T07:22:06.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.304+0000 7f4110a4e640 1 -- 172.21.15.1:0/3469211347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f410c0a7f50 msgr2=0x7f410c0c1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.304+0000 7f4110a4e640 1 --2- 172.21.15.1:0/3469211347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f410c0a7f50 0x7f410c0c1c90 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f410400c480 tx=0x7f410400c940 comp rx=0 tx=0).stop 2024-03-06T07:22:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.304+0000 7f4110a4e640 1 -- 172.21.15.1:0/3469211347 shutdown_connections 2024-03-06T07:22:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.304+0000 7f4110a4e640 1 --2- 172.21.15.1:0/3469211347 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f40ec06b4d0 0x7f40ec06d990 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.304+0000 7f4110a4e640 1 --2- 172.21.15.1:0/3469211347 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f410c150640 0x7f410c1ee3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.304+0000 7f4110a4e640 1 --2- 172.21.15.1:0/3469211347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f410c0c75a0 0x7f410c1ede90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.304+0000 7f4110a4e640 1 --2- 172.21.15.1:0/3469211347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f410c0a7f50 0x7f410c0c1c90 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.304+0000 7f4110a4e640 1 -- 172.21.15.1:0/3469211347 >> 172.21.15.1:0/3469211347 conn(0x7f410c06b3e0 msgr2=0x7f410c155b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.304+0000 7f4110a4e640 1 -- 172.21.15.1:0/3469211347 shutdown_connections 2024-03-06T07:22:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.304+0000 7f4110a4e640 1 -- 172.21.15.1:0/3469211347 wait complete. 2024-03-06T07:22:06.338 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:06.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.439+0000 7fd392e1c640 1 Processor -- start 2024-03-06T07:22:06.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.439+0000 7fd392e1c640 1 -- start start 2024-03-06T07:22:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.440+0000 7fd392e1c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd38c0c2b30 0x7fd38c0724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.440+0000 7fd392e1c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd38c152b70 0x7fd38c072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.440+0000 7fd392e1c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c151dd0 con 0x7fd38c0a5fa0 2024-03-06T07:22:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.440+0000 7fd392e1c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd38c1526e0 con 0x7fd38c152b70 2024-03-06T07:22:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.440+0000 7fd392e1c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c156190 con 0x7fd38c0c2b30 2024-03-06T07:22:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.440+0000 7fd38bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd38c152b70 0x7fd38c072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.440+0000 7fd3913b9640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd38c0a5fa0 0x7fd38c0c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45322/0 (socket says 172.21.15.1:45322) 2024-03-06T07:22:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.440+0000 7fd3913b9640 1 -- 172.21.15.1:0/304088702 learned_addr learned my addr 172.21.15.1:0/304088702 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.440+0000 7fd390bb8640 1 --2- 172.21.15.1:0/304088702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd38c0c2b30 0x7fd38c0724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.440+0000 7fd38bfff640 1 -- 172.21.15.1:0/304088702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd38c0c2b30 msgr2=0x7fd38c0724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.440+0000 7fd38bfff640 1 --2- 172.21.15.1:0/304088702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd38c0c2b30 0x7fd38c0724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.440+0000 7fd38b7fe640 1 -- 172.21.15.1:0/304088702 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935640452 0 0) 0x7fd38c151dd0 con 0x7fd38c0a5fa0 2024-03-06T07:22:06.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.440+0000 7fd38bfff640 1 -- 172.21.15.1:0/304088702 >> v1:172.21.15.1:6789/0 conn(0x7fd38c0a5fa0 legacy=0x7fd38c0c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.440+0000 7fd38bfff640 1 -- 172.21.15.1:0/304088702 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd38c152090 con 0x7fd38c152b70 2024-03-06T07:22:06.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.440+0000 7fd390bb8640 1 --2- 172.21.15.1:0/304088702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd38c0c2b30 0x7fd38c0724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:06.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.441+0000 7fd38bfff640 1 --2- 172.21.15.1:0/304088702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd38c152b70 0x7fd38c072a00 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fd3800097a0 tx=0x7fd38002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d51923a9772426 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.441+0000 7fd38b7fe640 1 -- 172.21.15.1:0/304088702 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd380035020 con 0x7fd38c152b70 2024-03-06T07:22:06.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.441+0000 7fd38b7fe640 1 -- 172.21.15.1:0/304088702 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd380031830 con 0x7fd38c152b70 2024-03-06T07:22:06.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.441+0000 7fd38b7fe640 1 -- 172.21.15.1:0/304088702 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd380031c00 con 0x7fd38c152b70 2024-03-06T07:22:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.442+0000 7fd392e1c640 1 -- 172.21.15.1:0/304088702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd38c152b70 msgr2=0x7fd38c072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.442+0000 7fd392e1c640 1 --2- 172.21.15.1:0/304088702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd38c152b70 0x7fd38c072a00 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fd3800097a0 tx=0x7fd38002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.442+0000 7fd392e1c640 1 -- 172.21.15.1:0/304088702 shutdown_connections 2024-03-06T07:22:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.442+0000 7fd392e1c640 1 --2- 172.21.15.1:0/304088702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd38c152b70 0x7fd38c072a00 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.442+0000 7fd392e1c640 1 --2- 172.21.15.1:0/304088702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd38c0c2b30 0x7fd38c0724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.442+0000 7fd392e1c640 1 -- 172.21.15.1:0/304088702 >> 172.21.15.1:0/304088702 conn(0x7fd38c0c6860 msgr2=0x7fd38c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.442+0000 7fd392e1c640 1 -- 172.21.15.1:0/304088702 shutdown_connections 2024-03-06T07:22:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.442+0000 7fd392e1c640 1 -- 172.21.15.1:0/304088702 wait complete. 2024-03-06T07:22:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.443+0000 7fd392e1c640 1 Processor -- start 2024-03-06T07:22:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.443+0000 7fd392e1c640 1 -- start start 2024-03-06T07:22:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd392e1c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd38c0a5fa0 0x7fd38c1deb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd392e1c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd38c0c2b30 0x7fd38c0bf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd392e1c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd38c152b70 0x7fd38c0bfa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd392e1c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c155f00 con 0x7fd38c0a5fa0 2024-03-06T07:22:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd392e1c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c155d80 con 0x7fd38c152b70 2024-03-06T07:22:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd392e1c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c0c3a00 con 0x7fd38c0c2b30 2024-03-06T07:22:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd390bb8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd38c0a5fa0 0x7fd38c1deb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd390bb8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd38c0a5fa0 0x7fd38c1deb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59412/0 (socket says 172.21.15.1:59412) 2024-03-06T07:22:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd390bb8640 1 -- 172.21.15.1:0/784089241 learned_addr learned my addr 172.21.15.1:0/784089241 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd38bfff640 1 --2- 172.21.15.1:0/784089241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd38c0c2b30 0x7fd38c0bf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd3913b9640 1 --2- 172.21.15.1:0/784089241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd38c152b70 0x7fd38c0bfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd390bb8640 1 -- 172.21.15.1:0/784089241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd38c0c2b30 msgr2=0x7fd38c0bf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd390bb8640 1 --2- 172.21.15.1:0/784089241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd38c0c2b30 0x7fd38c0bf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd390bb8640 1 -- 172.21.15.1:0/784089241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd38c152b70 msgr2=0x7fd38c0bfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd390bb8640 1 --2- 172.21.15.1:0/784089241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd38c152b70 0x7fd38c0bfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd390bb8640 1 -- 172.21.15.1:0/784089241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd38c0a7480 con 0x7fd38c0a5fa0 2024-03-06T07:22:06.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd3913b9640 1 --2- 172.21.15.1:0/784089241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd38c152b70 0x7fd38c0bfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:06.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd390bb8640 1 --2- 172.21.15.1:0/784089241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd38c0a5fa0 0x7fd38c1deb50 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd37c00e7e0 tx=0x7fd37c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd3897fa640 1 -- 172.21.15.1:0/784089241 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37c00f040 con 0x7fd38c0a5fa0 2024-03-06T07:22:06.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd3897fa640 1 -- 172.21.15.1:0/784089241 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd37c004030 con 0x7fd38c0a5fa0 2024-03-06T07:22:06.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd392e1c640 1 -- 172.21.15.1:0/784089241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd38c1e9120 con 0x7fd38c0a5fa0 2024-03-06T07:22:06.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd3897fa640 1 -- 172.21.15.1:0/784089241 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37c004400 con 0x7fd38c0a5fa0 2024-03-06T07:22:06.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.444+0000 7fd392e1c640 1 -- 172.21.15.1:0/784089241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd38c1df090 con 0x7fd38c0a5fa0 2024-03-06T07:22:06.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.445+0000 7fd392e1c640 1 -- 172.21.15.1:0/784089241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd38c067fd0 con 0x7fd38c0a5fa0 2024-03-06T07:22:06.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.446+0000 7fd3897fa640 1 -- 172.21.15.1:0/784089241 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd37c00c6d0 con 0x7fd38c0a5fa0 2024-03-06T07:22:06.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.446+0000 7fd3897fa640 1 --2- 172.21.15.1:0/784089241 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd37806b4d0 0x7fd37806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.446+0000 7fd3897fa640 1 -- 172.21.15.1:0/784089241 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd37c08a160 con 0x7fd38c0a5fa0 2024-03-06T07:22:06.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.446+0000 7fd38bfff640 1 --2- 172.21.15.1:0/784089241 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd37806b4d0 0x7fd37806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.447+0000 7fd38bfff640 1 --2- 172.21.15.1:0/784089241 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd37806b4d0 0x7fd37806d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd380036000 tx=0x7fd380008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.450+0000 7fd3897fa640 1 -- 172.21.15.1:0/784089241 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd37c017020 con 0x7fd38c0a5fa0 2024-03-06T07:22:06.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.633+0000 7fd392e1c640 1 -- 172.21.15.1:0/784089241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd38c071790 con 0x7fd38c0a5fa0 2024-03-06T07:22:06.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.634+0000 7fd3897fa640 1 -- 172.21.15.1:0/784089241 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16835 (secure 0 0 0) 0x7fd37c058ee0 con 0x7fd38c0a5fa0 2024-03-06T07:22:06.637 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:06.637 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":51,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:05.671308+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:03.930333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:04.634569+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:06.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.636+0000 7fd392e1c640 1 -- 172.21.15.1:0/784089241 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd37806b4d0 msgr2=0x7fd37806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.636+0000 7fd392e1c640 1 --2- 172.21.15.1:0/784089241 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd37806b4d0 0x7fd37806d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd380036000 tx=0x7fd380008030 comp rx=0 tx=0).stop 2024-03-06T07:22:06.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.636+0000 7fd392e1c640 1 -- 172.21.15.1:0/784089241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd38c0a5fa0 msgr2=0x7fd38c1deb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.636+0000 7fd392e1c640 1 --2- 172.21.15.1:0/784089241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd38c0a5fa0 0x7fd38c1deb50 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd37c00e7e0 tx=0x7fd37c00eca0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.636+0000 7fd392e1c640 1 -- 172.21.15.1:0/784089241 shutdown_connections 2024-03-06T07:22:06.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.636+0000 7fd392e1c640 1 --2- 172.21.15.1:0/784089241 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd37806b4d0 0x7fd37806d990 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.636+0000 7fd392e1c640 1 --2- 172.21.15.1:0/784089241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd38c152b70 0x7fd38c0bfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.636+0000 7fd392e1c640 1 --2- 172.21.15.1:0/784089241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd38c0c2b30 0x7fd38c0bf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.636+0000 7fd392e1c640 1 --2- 172.21.15.1:0/784089241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd38c0a5fa0 0x7fd38c1deb50 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.637+0000 7fd392e1c640 1 -- 172.21.15.1:0/784089241 >> 172.21.15.1:0/784089241 conn(0x7fd38c0c6860 msgr2=0x7fd38c14f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.637+0000 7fd392e1c640 1 -- 172.21.15.1:0/784089241 shutdown_connections 2024-03-06T07:22:06.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.637+0000 7fd392e1c640 1 -- 172.21.15.1:0/784089241 wait complete. 2024-03-06T07:22:06.649 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:06.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.749+0000 7f6962577640 1 Processor -- start 2024-03-06T07:22:06.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.749+0000 7f6962577640 1 -- start start 2024-03-06T07:22:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.749+0000 7f6962577640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f695c072bd0 0x7f695c0c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.749+0000 7f6962577640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f695c152b70 0x7f695c0c3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.749+0000 7f6962577640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f695c151dd0 con 0x7f695c0c2b30 2024-03-06T07:22:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.749+0000 7f6962577640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695c1526e0 con 0x7f695c072bd0 2024-03-06T07:22:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.749+0000 7f6962577640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695c156190 con 0x7f695c152b70 2024-03-06T07:22:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.749+0000 7f695b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f695c072bd0 0x7f695c0c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.749+0000 7f695b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f695c072bd0 0x7f695c0c32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59418/0 (socket says 172.21.15.1:59418) 2024-03-06T07:22:06.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.749+0000 7f695b7fe640 1 -- 172.21.15.1:0/3066523548 learned_addr learned my addr 172.21.15.1:0/3066523548 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:06.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.750+0000 7f6960b14640 1 --2- 172.21.15.1:0/3066523548 >> v2:172.21.15.98:3300/0 conn(0x7f695c152b70 0x7f695c0c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.750+0000 7f695b7fe640 1 -- 172.21.15.1:0/3066523548 >> v2:172.21.15.98:3300/0 conn(0x7f695c152b70 msgr2=0x7f695c0c3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.750+0000 7f695b7fe640 1 --2- 172.21.15.1:0/3066523548 >> v2:172.21.15.98:3300/0 conn(0x7f695c152b70 0x7f695c0c3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.750+0000 7f695b7fe640 1 -- 172.21.15.1:0/3066523548 >> v1:172.21.15.98:6789/0 conn(0x7f695c0c2b30 legacy=0x7f695c0724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.750+0000 7f695b7fe640 1 -- 172.21.15.1:0/3066523548 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c152090 con 0x7f695c072bd0 2024-03-06T07:22:06.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.750+0000 7f695b7fe640 1 --2- 172.21.15.1:0/3066523548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695c072bd0 0x7f695c0c32f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f695c06f710 tx=0x7f694c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c60f8273b0402b89 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.751+0000 7f695affd640 1 -- 172.21.15.1:0/3066523548 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c035020 con 0x7f695c072bd0 2024-03-06T07:22:06.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.751+0000 7f695affd640 1 -- 172.21.15.1:0/3066523548 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694c031870 con 0x7f695c072bd0 2024-03-06T07:22:06.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.751+0000 7f695affd640 1 -- 172.21.15.1:0/3066523548 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c031c40 con 0x7f695c072bd0 2024-03-06T07:22:06.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.751+0000 7f6962577640 1 -- 172.21.15.1:0/3066523548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695c072bd0 msgr2=0x7f695c0c32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.751+0000 7f6962577640 1 --2- 172.21.15.1:0/3066523548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695c072bd0 0x7f695c0c32f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f695c06f710 tx=0x7f694c02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:06.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.751+0000 7f6962577640 1 -- 172.21.15.1:0/3066523548 shutdown_connections 2024-03-06T07:22:06.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.751+0000 7f6962577640 1 --2- 172.21.15.1:0/3066523548 >> v2:172.21.15.98:3300/0 conn(0x7f695c152b70 0x7f695c0c3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.751+0000 7f6962577640 1 --2- 172.21.15.1:0/3066523548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695c072bd0 0x7f695c0c32f0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.751+0000 7f6962577640 1 -- 172.21.15.1:0/3066523548 >> 172.21.15.1:0/3066523548 conn(0x7f695c0c6860 msgr2=0x7f695c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.751+0000 7f6962577640 1 -- 172.21.15.1:0/3066523548 shutdown_connections 2024-03-06T07:22:06.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.751+0000 7f6962577640 1 -- 172.21.15.1:0/3066523548 wait complete. 2024-03-06T07:22:06.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.752+0000 7f6962577640 1 Processor -- start 2024-03-06T07:22:06.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.753+0000 7f6962577640 1 -- start start 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.753+0000 7f6962577640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f695c072bd0 0x7f695c1eb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.753+0000 7f6962577640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695c0c2b30 0x7f695c06e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.753+0000 7f6962577640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f695c152b70 0x7f695c1eeee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.753+0000 7f6962577640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0c3ef0 con 0x7f695c0c2b30 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.753+0000 7f6962577640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0c3d70 con 0x7f695c072bd0 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.753+0000 7f6962577640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695c155d80 con 0x7f695c152b70 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.753+0000 7f695b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695c0c2b30 0x7f695c06e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.753+0000 7f695b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695c0c2b30 0x7f695c06e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59434/0 (socket says 172.21.15.1:59434) 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.753+0000 7f695b7fe640 1 -- 172.21.15.1:0/1435769293 learned_addr learned my addr 172.21.15.1:0/1435769293 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.753+0000 7f695b7fe640 1 -- 172.21.15.1:0/1435769293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f695c152b70 msgr2=0x7f695c1eeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.753+0000 7f695b7fe640 1 --2- 172.21.15.1:0/1435769293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f695c152b70 0x7f695c1eeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.753+0000 7f695b7fe640 1 -- 172.21.15.1:0/1435769293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f695c072bd0 msgr2=0x7f695c1eb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.753+0000 7f695b7fe640 1 --2- 172.21.15.1:0/1435769293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f695c072bd0 0x7f695c1eb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.753+0000 7f695b7fe640 1 -- 172.21.15.1:0/1435769293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c14ba40 con 0x7f695c0c2b30 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.754+0000 7f695b7fe640 1 --2- 172.21.15.1:0/1435769293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695c0c2b30 0x7f695c06e1d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f694c000c00 tx=0x7f694c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.754+0000 7f6958ff9640 1 -- 172.21.15.1:0/1435769293 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c0443f0 con 0x7f695c0c2b30 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.754+0000 7f6958ff9640 1 -- 172.21.15.1:0/1435769293 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694c044590 con 0x7f695c0c2b30 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.754+0000 7f6962577640 1 -- 172.21.15.1:0/1435769293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695c14b480 con 0x7f695c0c2b30 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.754+0000 7f6958ff9640 1 -- 172.21.15.1:0/1435769293 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c044980 con 0x7f695c0c2b30 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.754+0000 7f6962577640 1 -- 172.21.15.1:0/1435769293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695c1ee6e0 con 0x7f695c0c2b30 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.755+0000 7f6962577640 1 -- 172.21.15.1:0/1435769293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695c067fd0 con 0x7f695c0c2b30 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.755+0000 7f6958ff9640 1 -- 172.21.15.1:0/1435769293 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f694c04b060 con 0x7f695c0c2b30 2024-03-06T07:22:06.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.756+0000 7f6958ff9640 1 --2- 172.21.15.1:0/1435769293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f693c06b4d0 0x7f693c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.756+0000 7f6958ff9640 1 -- 172.21.15.1:0/1435769293 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f694c0ac760 con 0x7f695c0c2b30 2024-03-06T07:22:06.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.756+0000 7f695bfff640 1 --2- 172.21.15.1:0/1435769293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f693c06b4d0 0x7f693c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.757+0000 7f695bfff640 1 --2- 172.21.15.1:0/1435769293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f693c06b4d0 0x7f693c06d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6944003cc0 tx=0x7f69440079b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.760+0000 7f6958ff9640 1 -- 172.21.15.1:0/1435769293 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f694c03f070 con 0x7f695c0c2b30 2024-03-06T07:22:06.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.944+0000 7f6962577640 1 -- 172.21.15.1:0/1435769293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f695c0c5e90 con 0x7f695c0c2b30 2024-03-06T07:22:06.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.945+0000 7f6958ff9640 1 -- 172.21.15.1:0/1435769293 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16835 (secure 0 0 0) 0x7f694c07b3b0 con 0x7f695c0c2b30 2024-03-06T07:22:06.947 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:06.947 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":51,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:05.671308+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:03.930333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:04.634569+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.947+0000 7f6962577640 1 -- 172.21.15.1:0/1435769293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f693c06b4d0 msgr2=0x7f693c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.947+0000 7f6962577640 1 --2- 172.21.15.1:0/1435769293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f693c06b4d0 0x7f693c06d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6944003cc0 tx=0x7f69440079b0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.947+0000 7f6962577640 1 -- 172.21.15.1:0/1435769293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695c0c2b30 msgr2=0x7f695c06e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.947+0000 7f6962577640 1 --2- 172.21.15.1:0/1435769293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695c0c2b30 0x7f695c06e1d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f694c000c00 tx=0x7f694c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.947+0000 7f6962577640 1 -- 172.21.15.1:0/1435769293 shutdown_connections 2024-03-06T07:22:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.947+0000 7f6962577640 1 --2- 172.21.15.1:0/1435769293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f693c06b4d0 0x7f693c06d990 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.947+0000 7f6962577640 1 --2- 172.21.15.1:0/1435769293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f695c152b70 0x7f695c1eeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.947+0000 7f6962577640 1 --2- 172.21.15.1:0/1435769293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695c0c2b30 0x7f695c06e1d0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.947+0000 7f6962577640 1 --2- 172.21.15.1:0/1435769293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f695c072bd0 0x7f695c1eb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.947+0000 7f6962577640 1 -- 172.21.15.1:0/1435769293 >> 172.21.15.1:0/1435769293 conn(0x7f695c0c6860 msgr2=0x7f695c154c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.947+0000 7f6962577640 1 -- 172.21.15.1:0/1435769293 shutdown_connections 2024-03-06T07:22:06.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:06.947+0000 7f6962577640 1 -- 172.21.15.1:0/1435769293 wait complete. 2024-03-06T07:22:06.960 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:22:07.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.063+0000 7f9698dd6640 1 Processor -- start 2024-03-06T07:22:07.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.063+0000 7f9698dd6640 1 -- start start 2024-03-06T07:22:07.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.063+0000 7f9698dd6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9694158e20 0x7f9694159280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.063+0000 7f9698dd6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9694071b10 con 0x7f96940c4a50 2024-03-06T07:22:07.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.063+0000 7f9698dd6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96941597c0 con 0x7f96940c3b00 2024-03-06T07:22:07.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.063+0000 7f9698dd6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96940c29b0 con 0x7f9694158e20 2024-03-06T07:22:07.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.064+0000 7f9692d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f96940c4a50 0x7f96940c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45334/0 (socket says 172.21.15.1:45334) 2024-03-06T07:22:07.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.064+0000 7f9692d9d640 1 -- 172.21.15.1:0/1040468549 learned_addr learned my addr 172.21.15.1:0/1040468549 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:07.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.064+0000 7f968bfff640 1 --2- 172.21.15.1:0/1040468549 >> v2:172.21.15.98:3300/0 conn(0x7f9694158e20 0x7f9694159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.064+0000 7f9691d9b640 1 -- 172.21.15.1:0/1040468549 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840267586 0 0) 0x7f9694071b10 con 0x7f96940c4a50 2024-03-06T07:22:07.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.064+0000 7f9691d9b640 1 -- 172.21.15.1:0/1040468549 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9680003520 con 0x7f96940c4a50 2024-03-06T07:22:07.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.064+0000 7f9691d9b640 1 -- 172.21.15.1:0/1040468549 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592283559 0 0) 0x7f96941597c0 con 0x7f96940c3b00 2024-03-06T07:22:07.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.064+0000 7f9691d9b640 1 -- 172.21.15.1:0/1040468549 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9694071b10 con 0x7f96940c3b00 2024-03-06T07:22:07.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.064+0000 7f9691d9b640 1 -- 172.21.15.1:0/1040468549 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2490689810 0 0) 0x7f9680003520 con 0x7f96940c4a50 2024-03-06T07:22:07.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.064+0000 7f968bfff640 1 -- 172.21.15.1:0/1040468549 >> v1:172.21.15.98:6789/0 conn(0x7f96940c3b00 legacy=0x7f9694072780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.064+0000 7f968bfff640 1 -- 172.21.15.1:0/1040468549 >> v1:172.21.15.1:6789/0 conn(0x7f96940c4a50 legacy=0x7f96940c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.064+0000 7f968bfff640 1 -- 172.21.15.1:0/1040468549 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9694071dd0 con 0x7f9694158e20 2024-03-06T07:22:07.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.065+0000 7f968bfff640 1 --2- 172.21.15.1:0/1040468549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694158e20 0x7f9694159280 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f967c0097a0 tx=0x7f967c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be7c5008ace68e2f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.065+0000 7f9691d9b640 1 -- 172.21.15.1:0/1040468549 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c037070 con 0x7f9694158e20 2024-03-06T07:22:07.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.065+0000 7f9691d9b640 1 -- 172.21.15.1:0/1040468549 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967c030510 con 0x7f9694158e20 2024-03-06T07:22:07.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.065+0000 7f9691d9b640 1 -- 172.21.15.1:0/1040468549 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c0019f0 con 0x7f9694158e20 2024-03-06T07:22:07.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.066+0000 7f9698dd6640 1 -- 172.21.15.1:0/1040468549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694158e20 msgr2=0x7f9694159280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.066+0000 7f9698dd6640 1 --2- 172.21.15.1:0/1040468549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694158e20 0x7f9694159280 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f967c0097a0 tx=0x7f967c02b790 comp rx=0 tx=0).stop 2024-03-06T07:22:07.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.066+0000 7f9698dd6640 1 -- 172.21.15.1:0/1040468549 shutdown_connections 2024-03-06T07:22:07.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.066+0000 7f9698dd6640 1 --2- 172.21.15.1:0/1040468549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694158e20 0x7f9694159280 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.066+0000 7f9698dd6640 1 -- 172.21.15.1:0/1040468549 >> 172.21.15.1:0/1040468549 conn(0x7f96940c6f40 msgr2=0x7f96940c7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.066+0000 7f9698dd6640 1 -- 172.21.15.1:0/1040468549 shutdown_connections 2024-03-06T07:22:07.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.066+0000 7f9698dd6640 1 -- 172.21.15.1:0/1040468549 wait complete. 2024-03-06T07:22:07.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.067+0000 7f9698dd6640 1 Processor -- start 2024-03-06T07:22:07.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.067+0000 7f9698dd6640 1 -- start start 2024-03-06T07:22:07.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f9698dd6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96940c3b00 0x7f9694150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f9698dd6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96940c4a50 0x7f96941e7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f9698dd6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694158e20 0x7f96941ebd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f9698dd6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96940c2c00 con 0x7f96940c3b00 2024-03-06T07:22:07.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f9698dd6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96940c2f70 con 0x7f9694158e20 2024-03-06T07:22:07.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f9698dd6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96940c55b0 con 0x7f96940c4a50 2024-03-06T07:22:07.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f969259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96940c3b00 0x7f9694150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f969259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96940c3b00 0x7f9694150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59448/0 (socket says 172.21.15.1:59448) 2024-03-06T07:22:07.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f969259c640 1 -- 172.21.15.1:0/3580524792 learned_addr learned my addr 172.21.15.1:0/3580524792 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:07.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f9692d9d640 1 --2- 172.21.15.1:0/3580524792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694158e20 0x7f96941ebd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f968bfff640 1 --2- 172.21.15.1:0/3580524792 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96940c4a50 0x7f96941e7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f969259c640 1 -- 172.21.15.1:0/3580524792 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96940c4a50 msgr2=0x7f96941e7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f969259c640 1 --2- 172.21.15.1:0/3580524792 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96940c4a50 0x7f96941e7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f969259c640 1 -- 172.21.15.1:0/3580524792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694158e20 msgr2=0x7f96941ebd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f969259c640 1 --2- 172.21.15.1:0/3580524792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694158e20 0x7f96941ebd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f969259c640 1 -- 172.21.15.1:0/3580524792 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969414d830 con 0x7f96940c3b00 2024-03-06T07:22:07.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f969259c640 1 --2- 172.21.15.1:0/3580524792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96940c3b00 0x7f9694150000 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f968400c430 tx=0x7f968400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f968affd640 1 -- 172.21.15.1:0/3580524792 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684017070 con 0x7f96940c3b00 2024-03-06T07:22:07.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f968affd640 1 -- 172.21.15.1:0/3580524792 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96840052f0 con 0x7f96940c3b00 2024-03-06T07:22:07.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f9698dd6640 1 -- 172.21.15.1:0/3580524792 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96940a6670 con 0x7f96940c3b00 2024-03-06T07:22:07.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.068+0000 7f968affd640 1 -- 172.21.15.1:0/3580524792 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684003830 con 0x7f96940c3b00 2024-03-06T07:22:07.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.069+0000 7f9698dd6640 1 -- 172.21.15.1:0/3580524792 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96940a6c30 con 0x7f96940c3b00 2024-03-06T07:22:07.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.069+0000 7f9698dd6640 1 -- 172.21.15.1:0/3580524792 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9694067fd0 con 0x7f96940c3b00 2024-03-06T07:22:07.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.070+0000 7f968affd640 1 -- 172.21.15.1:0/3580524792 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f968401a070 con 0x7f96940c3b00 2024-03-06T07:22:07.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.070+0000 7f968affd640 1 --2- 172.21.15.1:0/3580524792 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f967406b500 0x7f967406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.070+0000 7f968affd640 1 -- 172.21.15.1:0/3580524792 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9684089670 con 0x7f96940c3b00 2024-03-06T07:22:07.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.071+0000 7f968bfff640 1 --2- 172.21.15.1:0/3580524792 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f967406b500 0x7f967406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.071+0000 7f968bfff640 1 --2- 172.21.15.1:0/3580524792 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f967406b500 0x7f967406d9c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f967c034000 tx=0x7f967c02f000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.075+0000 7f968affd640 1 -- 172.21.15.1:0/3580524792 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9684057da0 con 0x7f96940c3b00 2024-03-06T07:22:07.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.300+0000 7f9698dd6640 1 -- 172.21.15.1:0/3580524792 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f96940c74c0 con 0x7f96940c3b00 2024-03-06T07:22:07.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.312+0000 7f968affd640 1 -- 172.21.15.1:0/3580524792 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v27) v1 ==== 118+0+0 (secure 0 0 0) 0x7f968405be40 con 0x7f96940c3b00 2024-03-06T07:22:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.315+0000 7f9698dd6640 1 -- 172.21.15.1:0/3580524792 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f967406b500 msgr2=0x7f967406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.315+0000 7f9698dd6640 1 --2- 172.21.15.1:0/3580524792 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f967406b500 0x7f967406d9c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f967c034000 tx=0x7f967c02f000 comp rx=0 tx=0).stop 2024-03-06T07:22:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.315+0000 7f9698dd6640 1 -- 172.21.15.1:0/3580524792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96940c3b00 msgr2=0x7f9694150000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.315+0000 7f9698dd6640 1 --2- 172.21.15.1:0/3580524792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96940c3b00 0x7f9694150000 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f968400c430 tx=0x7f968400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.315+0000 7f9698dd6640 1 -- 172.21.15.1:0/3580524792 shutdown_connections 2024-03-06T07:22:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.315+0000 7f9698dd6640 1 --2- 172.21.15.1:0/3580524792 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f967406b500 0x7f967406d9c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.315+0000 7f9698dd6640 1 --2- 172.21.15.1:0/3580524792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9694158e20 0x7f96941ebd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.315+0000 7f9698dd6640 1 --2- 172.21.15.1:0/3580524792 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96940c4a50 0x7f96941e7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.315+0000 7f9698dd6640 1 --2- 172.21.15.1:0/3580524792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96940c3b00 0x7f9694150000 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.315+0000 7f9698dd6640 1 -- 172.21.15.1:0/3580524792 >> 172.21.15.1:0/3580524792 conn(0x7f96940c6f40 msgr2=0x7f969406d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.315+0000 7f9698dd6640 1 -- 172.21.15.1:0/3580524792 shutdown_connections 2024-03-06T07:22:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.316+0000 7f9698dd6640 1 -- 172.21.15.1:0/3580524792 wait complete. 2024-03-06T07:22:07.328 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:22:07.329 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:22:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.430+0000 7f24f0415640 1 Processor -- start 2024-03-06T07:22:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.430+0000 7f24f0415640 1 -- start start 2024-03-06T07:22:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.431+0000 7f24f0415640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f24e80c6b10 0x7f24e8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.431+0000 7f24f0415640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e80c2fc0 con 0x7f24e8152650 2024-03-06T07:22:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.431+0000 7f24f0415640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e8158da0 con 0x7f24e80bf9d0 2024-03-06T07:22:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.431+0000 7f24f0415640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24e8155800 con 0x7f24e80c6b10 2024-03-06T07:22:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.431+0000 7f24ed9b0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f24e80c6b10 0x7f24e8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.431+0000 7f24ee1b1640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f24e8152650 0x7f24e8158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45342/0 (socket says 172.21.15.1:45342) 2024-03-06T07:22:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.431+0000 7f24ee1b1640 1 -- 172.21.15.1:0/1730705250 learned_addr learned my addr 172.21.15.1:0/1730705250 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.432+0000 7f24ed9b0640 1 -- 172.21.15.1:0/1730705250 >> v1:172.21.15.98:6789/0 conn(0x7f24e80bf9d0 legacy=0x7f24e80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.432+0000 7f24ed9b0640 1 -- 172.21.15.1:0/1730705250 >> v1:172.21.15.1:6789/0 conn(0x7f24e8152650 legacy=0x7f24e8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.432+0000 7f24ed9b0640 1 -- 172.21.15.1:0/1730705250 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e80c6690 con 0x7f24e80c6b10 2024-03-06T07:22:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.433+0000 7f24ed9b0640 1 --2- 172.21.15.1:0/1730705250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e80c6b10 0x7f24e8155c90 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f24d80097a0 tx=0x7f24d802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd10bab4d688e79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.433+0000 7f24ed1af640 1 -- 172.21.15.1:0/1730705250 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d8037070 con 0x7f24e80c6b10 2024-03-06T07:22:07.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.433+0000 7f24ed1af640 1 -- 172.21.15.1:0/1730705250 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d8030710 con 0x7f24e80c6b10 2024-03-06T07:22:07.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.433+0000 7f24ed1af640 1 -- 172.21.15.1:0/1730705250 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d8030ae0 con 0x7f24e80c6b10 2024-03-06T07:22:07.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.433+0000 7f24f0415640 1 -- 172.21.15.1:0/1730705250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e80c6b10 msgr2=0x7f24e8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.433+0000 7f24f0415640 1 --2- 172.21.15.1:0/1730705250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e80c6b10 0x7f24e8155c90 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f24d80097a0 tx=0x7f24d802b710 comp rx=0 tx=0).stop 2024-03-06T07:22:07.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.433+0000 7f24f0415640 1 -- 172.21.15.1:0/1730705250 shutdown_connections 2024-03-06T07:22:07.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.433+0000 7f24f0415640 1 --2- 172.21.15.1:0/1730705250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e80c6b10 0x7f24e8155c90 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.433+0000 7f24f0415640 1 -- 172.21.15.1:0/1730705250 >> 172.21.15.1:0/1730705250 conn(0x7f24e8066ee0 msgr2=0x7f24e80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.433+0000 7f24f0415640 1 -- 172.21.15.1:0/1730705250 shutdown_connections 2024-03-06T07:22:07.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.434+0000 7f24f0415640 1 -- 172.21.15.1:0/1730705250 wait complete. 2024-03-06T07:22:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.435+0000 7f24f0415640 1 Processor -- start 2024-03-06T07:22:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.435+0000 7f24f0415640 1 -- start start 2024-03-06T07:22:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.435+0000 7f24f0415640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24e80bf9d0 0x7f24e81e1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.435+0000 7f24f0415640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e8152650 0x7f24e8152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.435+0000 7f24f0415640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24e814fd30 0x7f24e814aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.435+0000 7f24f0415640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e81551b0 con 0x7f24e8152650 2024-03-06T07:22:07.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.435+0000 7f24f0415640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e80c6fa0 con 0x7f24e80bf9d0 2024-03-06T07:22:07.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.435+0000 7f24f0415640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e8155490 con 0x7f24e814fd30 2024-03-06T07:22:07.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.435+0000 7f24ed9b0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e8152650 0x7f24e8152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.435+0000 7f24ed9b0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e8152650 0x7f24e8152a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59474/0 (socket says 172.21.15.1:59474) 2024-03-06T07:22:07.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.435+0000 7f24ed9b0640 1 -- 172.21.15.1:0/1110793651 learned_addr learned my addr 172.21.15.1:0/1110793651 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:07.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.436+0000 7f24ed9b0640 1 -- 172.21.15.1:0/1110793651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24e814fd30 msgr2=0x7f24e814aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.436+0000 7f24ed9b0640 1 --2- 172.21.15.1:0/1110793651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24e814fd30 0x7f24e814aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.436+0000 7f24ee1b1640 1 --2- 172.21.15.1:0/1110793651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24e80bf9d0 0x7f24e81e1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.436+0000 7f24ed9b0640 1 -- 172.21.15.1:0/1110793651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24e80bf9d0 msgr2=0x7f24e81e1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.436+0000 7f24ed9b0640 1 --2- 172.21.15.1:0/1110793651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24e80bf9d0 0x7f24e81e1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.436+0000 7f24ed9b0640 1 -- 172.21.15.1:0/1110793651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e8154ed0 con 0x7f24e8152650 2024-03-06T07:22:07.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.436+0000 7f24ee1b1640 1 --2- 172.21.15.1:0/1110793651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24e80bf9d0 0x7f24e81e1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:07.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.436+0000 7f24ed9b0640 1 --2- 172.21.15.1:0/1110793651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e8152650 0x7f24e8152a30 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f24d802b750 tx=0x7f24d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.436+0000 7f24deffd640 1 -- 172.21.15.1:0/1110793651 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d8041020 con 0x7f24e8152650 2024-03-06T07:22:07.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.436+0000 7f24deffd640 1 -- 172.21.15.1:0/1110793651 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d8032a40 con 0x7f24e8152650 2024-03-06T07:22:07.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.436+0000 7f24f0415640 1 -- 172.21.15.1:0/1110793651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e8154710 con 0x7f24e8152650 2024-03-06T07:22:07.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.436+0000 7f24deffd640 1 -- 172.21.15.1:0/1110793651 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d8037040 con 0x7f24e8152650 2024-03-06T07:22:07.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.436+0000 7f24f0415640 1 -- 172.21.15.1:0/1110793651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e80c95f0 con 0x7f24e8152650 2024-03-06T07:22:07.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.437+0000 7f24f0415640 1 -- 172.21.15.1:0/1110793651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e8067fd0 con 0x7f24e8152650 2024-03-06T07:22:07.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.437+0000 7f24deffd640 1 -- 172.21.15.1:0/1110793651 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24d8044020 con 0x7f24e8152650 2024-03-06T07:22:07.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.438+0000 7f24deffd640 1 --2- 172.21.15.1:0/1110793651 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f24cc06b4d0 0x7f24cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.438+0000 7f24deffd640 1 -- 172.21.15.1:0/1110793651 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f24d80abfb0 con 0x7f24e8152650 2024-03-06T07:22:07.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.438+0000 7f24ee1b1640 1 --2- 172.21.15.1:0/1110793651 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f24cc06b4d0 0x7f24cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.439+0000 7f24ee1b1640 1 --2- 172.21.15.1:0/1110793651 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f24cc06b4d0 0x7f24cc06d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f24e4000f60 tx=0x7f24e4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.442+0000 7f24deffd640 1 -- 172.21.15.1:0/1110793651 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24d804b030 con 0x7f24e8152650 2024-03-06T07:22:07.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:07.662+0000 7f24f0415640 1 -- 172.21.15.1:0/1110793651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f24e80c34e0 con 0x7f24e8152650 2024-03-06T07:22:08.033 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-03-06T07:22:08.034 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-06T07:22:08.034 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-03-06T07:22:08.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.316+0000 7f24deffd640 1 -- 172.21.15.1:0/1110793651 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f24d807ad30 con 0x7f24e8152650 2024-03-06T07:22:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.319+0000 7f24f0415640 1 -- 172.21.15.1:0/1110793651 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f24cc06b4d0 msgr2=0x7f24cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.319+0000 7f24f0415640 1 --2- 172.21.15.1:0/1110793651 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f24cc06b4d0 0x7f24cc06d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f24e4000f60 tx=0x7f24e4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.319+0000 7f24f0415640 1 -- 172.21.15.1:0/1110793651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e8152650 msgr2=0x7f24e8152a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.319+0000 7f24f0415640 1 --2- 172.21.15.1:0/1110793651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e8152650 0x7f24e8152a30 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f24d802b750 tx=0x7f24d80046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.319+0000 7f24f0415640 1 -- 172.21.15.1:0/1110793651 shutdown_connections 2024-03-06T07:22:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.319+0000 7f24f0415640 1 --2- 172.21.15.1:0/1110793651 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f24cc06b4d0 0x7f24cc06d990 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.319+0000 7f24f0415640 1 --2- 172.21.15.1:0/1110793651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24e814fd30 0x7f24e814aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.319+0000 7f24f0415640 1 --2- 172.21.15.1:0/1110793651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e8152650 0x7f24e8152a30 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.319+0000 7f24f0415640 1 --2- 172.21.15.1:0/1110793651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24e80bf9d0 0x7f24e81e1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.319+0000 7f24f0415640 1 -- 172.21.15.1:0/1110793651 >> 172.21.15.1:0/1110793651 conn(0x7f24e8066ee0 msgr2=0x7f24e80c3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.319+0000 7f24f0415640 1 -- 172.21.15.1:0/1110793651 shutdown_connections 2024-03-06T07:22:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.319+0000 7f24f0415640 1 -- 172.21.15.1:0/1110793651 wait complete. 2024-03-06T07:22:08.332 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:22:08.332 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:08.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.431+0000 7f888abb6640 1 Processor -- start 2024-03-06T07:22:08.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.431+0000 7f888abb6640 1 -- start start 2024-03-06T07:22:08.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.432+0000 7f888abb6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8884152650 0x7f8884155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.432+0000 7f888abb6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88840bf9d0 0x7f88840c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.432+0000 7f888abb6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88840c2fc0 con 0x7f88840c6b10 2024-03-06T07:22:08.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.432+0000 7f888abb6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8884155800 con 0x7f88840bf9d0 2024-03-06T07:22:08.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.432+0000 7f888abb6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8884158da0 con 0x7f8884152650 2024-03-06T07:22:08.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.432+0000 7f8889153640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88840bf9d0 0x7f88840c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.432+0000 7f8883fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f88840c6b10 0x7f8884158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45348/0 (socket says 172.21.15.1:45348) 2024-03-06T07:22:08.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.432+0000 7f8883fff640 1 -- 172.21.15.1:0/2542429837 learned_addr learned my addr 172.21.15.1:0/2542429837 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:08.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.433+0000 7f8889153640 1 -- 172.21.15.1:0/2542429837 >> v2:172.21.15.98:3300/0 conn(0x7f8884152650 msgr2=0x7f8884155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.433+0000 7f8889153640 1 --2- 172.21.15.1:0/2542429837 >> v2:172.21.15.98:3300/0 conn(0x7f8884152650 0x7f8884155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.433+0000 7f8889153640 1 -- 172.21.15.1:0/2542429837 >> v1:172.21.15.1:6789/0 conn(0x7f88840c6b10 legacy=0x7f8884158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.433+0000 7f8889153640 1 -- 172.21.15.1:0/2542429837 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88840c6690 con 0x7f88840bf9d0 2024-03-06T07:22:08.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.433+0000 7f88837fe640 1 -- 172.21.15.1:0/2542429837 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490260530 0 0) 0x7f88840c2fc0 con 0x7f88840c6b10 2024-03-06T07:22:08.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.433+0000 7f8889153640 1 --2- 172.21.15.1:0/2542429837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88840bf9d0 0x7f88840c1460 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f88840cac90 tx=0x7f887402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8abd8dad1a4cd4dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.434+0000 7f88837fe640 1 -- 172.21.15.1:0/2542429837 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88740315a0 con 0x7f88840bf9d0 2024-03-06T07:22:08.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.434+0000 7f88837fe640 1 -- 172.21.15.1:0/2542429837 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8874037070 con 0x7f88840bf9d0 2024-03-06T07:22:08.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.434+0000 7f88837fe640 1 -- 172.21.15.1:0/2542429837 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874031960 con 0x7f88840bf9d0 2024-03-06T07:22:08.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.434+0000 7f888abb6640 1 -- 172.21.15.1:0/2542429837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88840bf9d0 msgr2=0x7f88840c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.434+0000 7f888abb6640 1 --2- 172.21.15.1:0/2542429837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88840bf9d0 0x7f88840c1460 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f88840cac90 tx=0x7f887402b840 comp rx=0 tx=0).stop 2024-03-06T07:22:08.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.434+0000 7f888abb6640 1 -- 172.21.15.1:0/2542429837 shutdown_connections 2024-03-06T07:22:08.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.434+0000 7f888abb6640 1 --2- 172.21.15.1:0/2542429837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88840bf9d0 0x7f88840c1460 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.434+0000 7f888abb6640 1 --2- 172.21.15.1:0/2542429837 >> v2:172.21.15.98:3300/0 conn(0x7f8884152650 0x7f8884155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.434+0000 7f888abb6640 1 -- 172.21.15.1:0/2542429837 >> 172.21.15.1:0/2542429837 conn(0x7f8884066ee0 msgr2=0x7f88840672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.434+0000 7f888abb6640 1 -- 172.21.15.1:0/2542429837 shutdown_connections 2024-03-06T07:22:08.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.434+0000 7f888abb6640 1 -- 172.21.15.1:0/2542429837 wait complete. 2024-03-06T07:22:08.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.435+0000 7f888abb6640 1 Processor -- start 2024-03-06T07:22:08.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.435+0000 7f888abb6640 1 -- start start 2024-03-06T07:22:08.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.436+0000 7f888abb6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88840bf9d0 0x7f8884153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.436+0000 7f888abb6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8884152650 0x7f88841ee510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.436+0000 7f888abb6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88841df310 0x7f888413f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.436+0000 7f888abb6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88841551b0 con 0x7f88840bf9d0 2024-03-06T07:22:08.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.436+0000 7f888abb6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88840c6fa0 con 0x7f8884152650 2024-03-06T07:22:08.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.436+0000 7f888abb6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8884155490 con 0x7f88841df310 2024-03-06T07:22:08.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.436+0000 7f8888952640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88840bf9d0 0x7f8884153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.436+0000 7f8888952640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88840bf9d0 0x7f8884153cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59488/0 (socket says 172.21.15.1:59488) 2024-03-06T07:22:08.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.436+0000 7f8888952640 1 -- 172.21.15.1:0/3363337995 learned_addr learned my addr 172.21.15.1:0/3363337995 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:08.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.437+0000 7f8888952640 1 -- 172.21.15.1:0/3363337995 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88841df310 msgr2=0x7f888413f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.437+0000 7f8889153640 1 --2- 172.21.15.1:0/3363337995 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88841df310 0x7f888413f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.437+0000 7f8883fff640 1 --2- 172.21.15.1:0/3363337995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8884152650 0x7f88841ee510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.437+0000 7f8888952640 1 --2- 172.21.15.1:0/3363337995 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88841df310 0x7f888413f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.437+0000 7f8888952640 1 -- 172.21.15.1:0/3363337995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8884152650 msgr2=0x7f88841ee510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.437+0000 7f8888952640 1 --2- 172.21.15.1:0/3363337995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8884152650 0x7f88841ee510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.437+0000 7f8888952640 1 -- 172.21.15.1:0/3363337995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88841dfd60 con 0x7f88840bf9d0 2024-03-06T07:22:08.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.437+0000 7f8889153640 1 --2- 172.21.15.1:0/3363337995 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88841df310 0x7f888413f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:08.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.437+0000 7f8883fff640 1 --2- 172.21.15.1:0/3363337995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8884152650 0x7f88841ee510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:08.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.437+0000 7f8888952640 1 --2- 172.21.15.1:0/3363337995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88840bf9d0 0x7f8884153cf0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f887c00ba50 tx=0x7f887c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.437+0000 7f88817fa640 1 -- 172.21.15.1:0/3363337995 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c019070 con 0x7f88840bf9d0 2024-03-06T07:22:08.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.437+0000 7f88817fa640 1 -- 172.21.15.1:0/3363337995 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887c00cba0 con 0x7f88840bf9d0 2024-03-06T07:22:08.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.437+0000 7f88817fa640 1 -- 172.21.15.1:0/3363337995 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c014440 con 0x7f88840bf9d0 2024-03-06T07:22:08.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.437+0000 7f888abb6640 1 -- 172.21.15.1:0/3363337995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88841de830 con 0x7f88840bf9d0 2024-03-06T07:22:08.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.437+0000 7f888abb6640 1 -- 172.21.15.1:0/3363337995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88841dedf0 con 0x7f88840bf9d0 2024-03-06T07:22:08.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.438+0000 7f888abb6640 1 -- 172.21.15.1:0/3363337995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88841521b0 con 0x7f88840bf9d0 2024-03-06T07:22:08.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.438+0000 7f88817fa640 1 -- 172.21.15.1:0/3363337995 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f887c01c070 con 0x7f88840bf9d0 2024-03-06T07:22:08.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.439+0000 7f88817fa640 1 --2- 172.21.15.1:0/3363337995 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f887006b4d0 0x7f887006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.439+0000 7f88817fa640 1 -- 172.21.15.1:0/3363337995 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f887c08a040 con 0x7f88840bf9d0 2024-03-06T07:22:08.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.439+0000 7f8883fff640 1 --2- 172.21.15.1:0/3363337995 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f887006b4d0 0x7f887006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.439+0000 7f8883fff640 1 --2- 172.21.15.1:0/3363337995 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f887006b4d0 0x7f887006d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f88840c6040 tx=0x7f8878009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.443+0000 7f88817fa640 1 -- 172.21.15.1:0/3363337995 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88841521b0 con 0x7f88840bf9d0 2024-03-06T07:22:08.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.692+0000 7f888abb6640 1 -- 172.21.15.1:0/3363337995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f888406b540 con 0x7f88840bf9d0 2024-03-06T07:22:08.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.692+0000 7f88817fa640 1 -- 172.21.15.1:0/3363337995 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4968 (secure 0 0 0) 0x7f888406b540 con 0x7f88840bf9d0 2024-03-06T07:22:08.694 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:08.695 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5084,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2937068915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2937068915},{"type":"v1","addr":"172.21.15.98:6836","nonce":2937068915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5087,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2828783606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2828783606},{"type":"v1","addr":"172.21.15.98:6837","nonce":2828783606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/3845300501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":3845300501},{"type":"v1","addr":"172.21.15.1:6835","nonce":3845300501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:05.671602+0000","modified":"2024-03-06T07:22:07.663851+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5778":{"gid":5778,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/131367647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":131367647},{"type":"v1","addr":"172.21.15.1:6834","nonce":131367647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5778,"qdb_cluster":[5778]},"id":3}]} 2024-03-06T07:22:08.695 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-03-06T07:22:08.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.694+0000 7f888abb6640 1 -- 172.21.15.1:0/3363337995 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f887006b4d0 msgr2=0x7f887006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.694+0000 7f888abb6640 1 --2- 172.21.15.1:0/3363337995 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f887006b4d0 0x7f887006d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f88840c6040 tx=0x7f8878009040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.694+0000 7f888abb6640 1 -- 172.21.15.1:0/3363337995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88840bf9d0 msgr2=0x7f8884153cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.694+0000 7f888abb6640 1 --2- 172.21.15.1:0/3363337995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88840bf9d0 0x7f8884153cf0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f887c00ba50 tx=0x7f887c00bf10 comp rx=0 tx=0).stop 2024-03-06T07:22:08.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.694+0000 7f888abb6640 1 -- 172.21.15.1:0/3363337995 shutdown_connections 2024-03-06T07:22:08.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.694+0000 7f888abb6640 1 --2- 172.21.15.1:0/3363337995 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f887006b4d0 0x7f887006d990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.694+0000 7f888abb6640 1 --2- 172.21.15.1:0/3363337995 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88841df310 0x7f888413f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.694+0000 7f888abb6640 1 --2- 172.21.15.1:0/3363337995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8884152650 0x7f88841ee510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.694+0000 7f888abb6640 1 --2- 172.21.15.1:0/3363337995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88840bf9d0 0x7f8884153cf0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.694+0000 7f888abb6640 1 -- 172.21.15.1:0/3363337995 >> 172.21.15.1:0/3363337995 conn(0x7f8884066ee0 msgr2=0x7f88840c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.695+0000 7f888abb6640 1 -- 172.21.15.1:0/3363337995 shutdown_connections 2024-03-06T07:22:08.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.695+0000 7f888abb6640 1 -- 172.21.15.1:0/3363337995 wait complete. 2024-03-06T07:22:08.707 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:08.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.806+0000 7f4c3eaed640 1 Processor -- start 2024-03-06T07:22:08.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.807+0000 7f4c3eaed640 1 -- start start 2024-03-06T07:22:08.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.808+0000 7f4c3eaed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c380c6b10 0x7f4c38155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.808+0000 7f4c3eaed640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c380bf9d0 0x7f4c380c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.808+0000 7f4c3eaed640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c380c2fc0 con 0x7f4c38152650 2024-03-06T07:22:08.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.808+0000 7f4c3eaed640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c38155800 con 0x7f4c380bf9d0 2024-03-06T07:22:08.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.808+0000 7f4c3eaed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c38158da0 con 0x7f4c380c6b10 2024-03-06T07:22:08.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.808+0000 7f4c3d08a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c380bf9d0 0x7f4c380c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.808+0000 7f4c3d08a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c380bf9d0 0x7f4c380c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59498/0 (socket says 172.21.15.1:59498) 2024-03-06T07:22:08.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.808+0000 7f4c3d08a640 1 -- 172.21.15.1:0/4226341852 learned_addr learned my addr 172.21.15.1:0/4226341852 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:08.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.808+0000 7f4c37fff640 1 --2- 172.21.15.1:0/4226341852 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c380c6b10 0x7f4c38155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.808+0000 7f4c3d08a640 1 -- 172.21.15.1:0/4226341852 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c380c6b10 msgr2=0x7f4c38155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.808+0000 7f4c3d08a640 1 --2- 172.21.15.1:0/4226341852 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c380c6b10 0x7f4c38155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.808+0000 7f4c3d08a640 1 -- 172.21.15.1:0/4226341852 >> v1:172.21.15.98:6789/0 conn(0x7f4c38152650 legacy=0x7f4c38158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.808+0000 7f4c3d08a640 1 -- 172.21.15.1:0/4226341852 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c380c6690 con 0x7f4c380bf9d0 2024-03-06T07:22:08.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.809+0000 7f4c3d08a640 1 --2- 172.21.15.1:0/4226341852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c380bf9d0 0x7f4c380c1460 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4c30009e50 tx=0x7f4c3002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30ead28e4dd1fc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.809+0000 7f4c377fe640 1 -- 172.21.15.1:0/4226341852 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3002c9e0 con 0x7f4c380bf9d0 2024-03-06T07:22:08.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.809+0000 7f4c377fe640 1 -- 172.21.15.1:0/4226341852 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3002cb80 con 0x7f4c380bf9d0 2024-03-06T07:22:08.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.809+0000 7f4c377fe640 1 -- 172.21.15.1:0/4226341852 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c300343f0 con 0x7f4c380bf9d0 2024-03-06T07:22:08.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.810+0000 7f4c3eaed640 1 -- 172.21.15.1:0/4226341852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c380bf9d0 msgr2=0x7f4c380c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.810+0000 7f4c3eaed640 1 --2- 172.21.15.1:0/4226341852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c380bf9d0 0x7f4c380c1460 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4c30009e50 tx=0x7f4c3002be40 comp rx=0 tx=0).stop 2024-03-06T07:22:08.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.810+0000 7f4c3eaed640 1 -- 172.21.15.1:0/4226341852 shutdown_connections 2024-03-06T07:22:08.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.810+0000 7f4c3eaed640 1 --2- 172.21.15.1:0/4226341852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c380bf9d0 0x7f4c380c1460 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.810+0000 7f4c3eaed640 1 --2- 172.21.15.1:0/4226341852 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c380c6b10 0x7f4c38155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.810+0000 7f4c3eaed640 1 -- 172.21.15.1:0/4226341852 >> 172.21.15.1:0/4226341852 conn(0x7f4c38066ee0 msgr2=0x7f4c380672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.810+0000 7f4c3eaed640 1 -- 172.21.15.1:0/4226341852 shutdown_connections 2024-03-06T07:22:08.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.810+0000 7f4c3eaed640 1 -- 172.21.15.1:0/4226341852 wait complete. 2024-03-06T07:22:08.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.811+0000 7f4c3eaed640 1 Processor -- start 2024-03-06T07:22:08.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.811+0000 7f4c3eaed640 1 -- start start 2024-03-06T07:22:08.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c3eaed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c380bf9d0 0x7f4c38152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c3eaed640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c38152650 0x7f4c381eb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c3eaed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c380c9a90 0x7f4c381e13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c3eaed640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c381551b0 con 0x7f4c38152650 2024-03-06T07:22:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c3eaed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c380c6fa0 con 0x7f4c380c9a90 2024-03-06T07:22:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c3eaed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c38155490 con 0x7f4c380bf9d0 2024-03-06T07:22:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c37fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c38152650 0x7f4c381eb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c37fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c38152650 0x7f4c381eb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59508/0 (socket says 172.21.15.1:59508) 2024-03-06T07:22:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c37fff640 1 -- 172.21.15.1:0/3644555237 learned_addr learned my addr 172.21.15.1:0/3644555237 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c3d08a640 1 --2- 172.21.15.1:0/3644555237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c380c9a90 0x7f4c381e13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c37fff640 1 -- 172.21.15.1:0/3644555237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c380bf9d0 msgr2=0x7f4c38152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c3c889640 1 --2- 172.21.15.1:0/3644555237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c380bf9d0 0x7f4c38152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c37fff640 1 --2- 172.21.15.1:0/3644555237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c380bf9d0 0x7f4c38152ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c37fff640 1 -- 172.21.15.1:0/3644555237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c380c9a90 msgr2=0x7f4c381e13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c37fff640 1 --2- 172.21.15.1:0/3644555237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c380c9a90 0x7f4c381e13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c37fff640 1 -- 172.21.15.1:0/3644555237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c38153400 con 0x7f4c38152650 2024-03-06T07:22:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c3d08a640 1 --2- 172.21.15.1:0/3644555237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c380c9a90 0x7f4c381e13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c37fff640 1 --2- 172.21.15.1:0/3644555237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c38152650 0x7f4c381eb580 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4c2800d620 tx=0x7f4c2800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.813+0000 7f4c357fa640 1 -- 172.21.15.1:0/3644555237 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2800b620 con 0x7f4c38152650 2024-03-06T07:22:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.814+0000 7f4c357fa640 1 -- 172.21.15.1:0/3644555237 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c28019070 con 0x7f4c38152650 2024-03-06T07:22:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.814+0000 7f4c357fa640 1 -- 172.21.15.1:0/3644555237 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2800ba00 con 0x7f4c38152650 2024-03-06T07:22:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.814+0000 7f4c3eaed640 1 -- 172.21.15.1:0/3644555237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c3814b250 con 0x7f4c38152650 2024-03-06T07:22:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.814+0000 7f4c3eaed640 1 -- 172.21.15.1:0/3644555237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c381eb080 con 0x7f4c38152650 2024-03-06T07:22:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.815+0000 7f4c3eaed640 1 -- 172.21.15.1:0/3644555237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c3814b420 con 0x7f4c38152650 2024-03-06T07:22:08.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.815+0000 7f4c357fa640 1 -- 172.21.15.1:0/3644555237 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c2800bba0 con 0x7f4c38152650 2024-03-06T07:22:08.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.815+0000 7f4c357fa640 1 --2- 172.21.15.1:0/3644555237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c1806b4d0 0x7f4c1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.816+0000 7f4c357fa640 1 -- 172.21.15.1:0/3644555237 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4c2808a740 con 0x7f4c38152650 2024-03-06T07:22:08.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.816+0000 7f4c3c889640 1 --2- 172.21.15.1:0/3644555237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c1806b4d0 0x7f4c1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.816+0000 7f4c3c889640 1 --2- 172.21.15.1:0/3644555237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c1806b4d0 0x7f4c1806d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4c20003d90 tx=0x7f4c20009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:08.820+0000 7f4c357fa640 1 -- 172.21.15.1:0/3644555237 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c280594c0 con 0x7f4c38152650 2024-03-06T07:22:09.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.125+0000 7f4c3eaed640 1 -- 172.21.15.1:0/3644555237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c38067fd0 con 0x7f4c38152650 2024-03-06T07:22:09.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.126+0000 7f4c357fa640 1 -- 172.21.15.1:0/3644555237 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16822 (secure 0 0 0) 0x7f4c28058dc0 con 0x7f4c38152650 2024-03-06T07:22:09.128 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:09.129 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":52,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:07.309375+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:03.930333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:04.634569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:09.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.129+0000 7f4c3eaed640 1 -- 172.21.15.1:0/3644555237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c1806b4d0 msgr2=0x7f4c1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.129+0000 7f4c3eaed640 1 --2- 172.21.15.1:0/3644555237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c1806b4d0 0x7f4c1806d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4c20003d90 tx=0x7f4c20009040 comp rx=0 tx=0).stop 2024-03-06T07:22:09.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.129+0000 7f4c3eaed640 1 -- 172.21.15.1:0/3644555237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c38152650 msgr2=0x7f4c381eb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.129+0000 7f4c3eaed640 1 --2- 172.21.15.1:0/3644555237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c38152650 0x7f4c381eb580 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4c2800d620 tx=0x7f4c2800dae0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.129+0000 7f4c3eaed640 1 -- 172.21.15.1:0/3644555237 shutdown_connections 2024-03-06T07:22:09.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.129+0000 7f4c3eaed640 1 --2- 172.21.15.1:0/3644555237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c1806b4d0 0x7f4c1806d990 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.129+0000 7f4c3eaed640 1 --2- 172.21.15.1:0/3644555237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c380c9a90 0x7f4c381e13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.129+0000 7f4c3eaed640 1 --2- 172.21.15.1:0/3644555237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c38152650 0x7f4c381eb580 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.129+0000 7f4c3eaed640 1 --2- 172.21.15.1:0/3644555237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c380bf9d0 0x7f4c38152ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.129+0000 7f4c3eaed640 1 -- 172.21.15.1:0/3644555237 >> 172.21.15.1:0/3644555237 conn(0x7f4c38066ee0 msgr2=0x7f4c380cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.129+0000 7f4c3eaed640 1 -- 172.21.15.1:0/3644555237 shutdown_connections 2024-03-06T07:22:09.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.129+0000 7f4c3eaed640 1 -- 172.21.15.1:0/3644555237 wait complete. 2024-03-06T07:22:09.142 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:22:09.142 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:09.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.271+0000 7f3fe2615640 1 Processor -- start 2024-03-06T07:22:09.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.271+0000 7f3fe2615640 1 -- start start 2024-03-06T07:22:09.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.272+0000 7f3fe2615640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fdc0c6b10 0x7f3fdc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.272+0000 7f3fe2615640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fdc0c2fc0 con 0x7f3fdc0bf9d0 2024-03-06T07:22:09.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.272+0000 7f3fe2615640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fdc158da0 con 0x7f3fdc152650 2024-03-06T07:22:09.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.272+0000 7f3fe2615640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc155800 con 0x7f3fdc0c6b10 2024-03-06T07:22:09.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.272+0000 7f3fe0bb2640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3fdc0bf9d0 0x7f3fdc0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45350/0 (socket says 172.21.15.1:45350) 2024-03-06T07:22:09.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.272+0000 7f3fe0bb2640 1 -- 172.21.15.1:0/3531285771 learned_addr learned my addr 172.21.15.1:0/3531285771 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:09.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.272+0000 7f3fdb7fe640 1 --2- 172.21.15.1:0/3531285771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fdc0c6b10 0x7f3fdc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.272+0000 7f3fdaffd640 1 -- 172.21.15.1:0/3531285771 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918961451 0 0) 0x7f3fdc0c2fc0 con 0x7f3fdc0bf9d0 2024-03-06T07:22:09.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.272+0000 7f3fdaffd640 1 -- 172.21.15.1:0/3531285771 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fc8003520 con 0x7f3fdc0bf9d0 2024-03-06T07:22:09.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdaffd640 1 -- 172.21.15.1:0/3531285771 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 875390667 0 0) 0x7f3fc8003520 con 0x7f3fdc0bf9d0 2024-03-06T07:22:09.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdaffd640 1 -- 172.21.15.1:0/3531285771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fdc0c6b10 msgr2=0x7f3fdc155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdaffd640 1 --2- 172.21.15.1:0/3531285771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fdc0c6b10 0x7f3fdc155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdaffd640 1 -- 172.21.15.1:0/3531285771 >> v1:172.21.15.98:6789/0 conn(0x7f3fdc152650 legacy=0x7f3fdc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdaffd640 1 -- 172.21.15.1:0/3531285771 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fdc0c6690 con 0x7f3fdc0bf9d0 2024-03-06T07:22:09.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdaffd640 1 -- 172.21.15.1:0/3531285771 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f3fd40019f0 con 0x7f3fdc0bf9d0 2024-03-06T07:22:09.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdaffd640 1 -- 172.21.15.1:0/3531285771 >> v1:172.21.15.1:6789/0 conn(0x7f3fdc0bf9d0 legacy=0x7f3fdc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdaffd640 1 --2- 172.21.15.1:0/3531285771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc80039e0 0x7f3fc8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdaffd640 1 --2- 172.21.15.1:0/3531285771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fc8004300 0x7f3fc8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdaffd640 1 --2- 172.21.15.1:0/3531285771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc8004fe0 0x7f3fc800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdaffd640 1 -- 172.21.15.1:0/3531285771 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc155800 con 0x7f3fc8004300 2024-03-06T07:22:09.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdaffd640 1 -- 172.21.15.1:0/3531285771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc800bf00 con 0x7f3fc8004fe0 2024-03-06T07:22:09.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdaffd640 1 -- 172.21.15.1:0/3531285771 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc800c100 con 0x7f3fc80039e0 2024-03-06T07:22:09.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdb7fe640 1 --2- 172.21.15.1:0/3531285771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fc8004300 0x7f3fc8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fe0bb2640 1 --2- 172.21.15.1:0/3531285771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc8004fe0 0x7f3fc800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdbfff640 1 --2- 172.21.15.1:0/3531285771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc80039e0 0x7f3fc8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdb7fe640 1 -- 172.21.15.1:0/3531285771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc80039e0 msgr2=0x7f3fc8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdb7fe640 1 --2- 172.21.15.1:0/3531285771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc80039e0 0x7f3fc8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdb7fe640 1 -- 172.21.15.1:0/3531285771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc8004fe0 msgr2=0x7f3fc800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdb7fe640 1 --2- 172.21.15.1:0/3531285771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc8004fe0 0x7f3fc800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdb7fe640 1 -- 172.21.15.1:0/3531285771 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc8003520 con 0x7f3fc8004300 2024-03-06T07:22:09.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fe0bb2640 1 --2- 172.21.15.1:0/3531285771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc8004fe0 0x7f3fc800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:09.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.273+0000 7f3fdbfff640 1 --2- 172.21.15.1:0/3531285771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc80039e0 0x7f3fc8003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:09.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.274+0000 7f3fdb7fe640 1 --2- 172.21.15.1:0/3531285771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fc8004300 0x7f3fc8004800 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4009ac0 tx=0x7f3fc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7805c4ec2571b539 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.274+0000 7f3fdaffd640 1 -- 172.21.15.1:0/3531285771 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4039070 con 0x7f3fc8004300 2024-03-06T07:22:09.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.274+0000 7f3fdaffd640 1 -- 172.21.15.1:0/3531285771 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc402c6b0 con 0x7f3fc8004300 2024-03-06T07:22:09.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.274+0000 7f3fdaffd640 1 -- 172.21.15.1:0/3531285771 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4009690 con 0x7f3fc8004300 2024-03-06T07:22:09.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.275+0000 7f3fe2615640 1 -- 172.21.15.1:0/3531285771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fc8004300 msgr2=0x7f3fc8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.275+0000 7f3fe2615640 1 --2- 172.21.15.1:0/3531285771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fc8004300 0x7f3fc8004800 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4009ac0 tx=0x7f3fc8005830 comp rx=0 tx=0).stop 2024-03-06T07:22:09.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.275+0000 7f3fe0bb2640 1 -- 172.21.15.1:0/3531285771 reap_dead start 2024-03-06T07:22:09.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.275+0000 7f3fe2615640 1 -- 172.21.15.1:0/3531285771 shutdown_connections 2024-03-06T07:22:09.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.275+0000 7f3fe2615640 1 -- 172.21.15.1:0/3531285771 >> 172.21.15.1:0/3531285771 conn(0x7f3fdc066ee0 msgr2=0x7f3fdc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.275+0000 7f3fe2615640 1 -- 172.21.15.1:0/3531285771 shutdown_connections 2024-03-06T07:22:09.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.275+0000 7f3fe2615640 1 -- 172.21.15.1:0/3531285771 wait complete. 2024-03-06T07:22:09.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.276+0000 7f3fe2615640 1 Processor -- start 2024-03-06T07:22:09.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.276+0000 7f3fe2615640 1 -- start start 2024-03-06T07:22:09.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fe2615640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fc8004300 0x7f3fdc157550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fe2615640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fdc158290 0x7f3fdc1eafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fe2615640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fdc154190 0x7f3fdc1ed9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fe2615640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc1551b0 con 0x7f3fc8004300 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fe2615640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc0c6fa0 con 0x7f3fdc158290 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fe2615640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc155490 con 0x7f3fdc154190 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fdbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fc8004300 0x7f3fdc157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fdbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fc8004300 0x7f3fdc157550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59540/0 (socket says 172.21.15.1:59540) 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fdbfff640 1 -- 172.21.15.1:0/3729685260 learned_addr learned my addr 172.21.15.1:0/3729685260 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fdb7fe640 1 --2- 172.21.15.1:0/3729685260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fdc158290 0x7f3fdc1eafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fe0bb2640 1 --2- 172.21.15.1:0/3729685260 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fdc154190 0x7f3fdc1ed9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fdbfff640 1 -- 172.21.15.1:0/3729685260 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fdc154190 msgr2=0x7f3fdc1ed9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fdbfff640 1 --2- 172.21.15.1:0/3729685260 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fdc154190 0x7f3fdc1ed9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fdbfff640 1 -- 172.21.15.1:0/3729685260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fdc158290 msgr2=0x7f3fdc1eafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fdbfff640 1 --2- 172.21.15.1:0/3729685260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fdc158290 0x7f3fdc1eafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fdbfff640 1 -- 172.21.15.1:0/3729685260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fdc157b30 con 0x7f3fc8004300 2024-03-06T07:22:09.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fdbfff640 1 --2- 172.21.15.1:0/3729685260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fc8004300 0x7f3fdc157550 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc00e830 tx=0x7f3fcc00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.277+0000 7f3fd8ff9640 1 -- 172.21.15.1:0/3729685260 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc0050d0 con 0x7f3fc8004300 2024-03-06T07:22:09.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.278+0000 7f3fd8ff9640 1 -- 172.21.15.1:0/3729685260 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fcc018070 con 0x7f3fc8004300 2024-03-06T07:22:09.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.278+0000 7f3fe2615640 1 -- 172.21.15.1:0/3729685260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fdc157e10 con 0x7f3fc8004300 2024-03-06T07:22:09.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.278+0000 7f3fd8ff9640 1 -- 172.21.15.1:0/3729685260 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc005470 con 0x7f3fc8004300 2024-03-06T07:22:09.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.278+0000 7f3fe2615640 1 -- 172.21.15.1:0/3729685260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fdc1ea7e0 con 0x7f3fc8004300 2024-03-06T07:22:09.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.278+0000 7f3fe2615640 1 -- 172.21.15.1:0/3729685260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fdc067fd0 con 0x7f3fc8004300 2024-03-06T07:22:09.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.279+0000 7f3fd8ff9640 1 -- 172.21.15.1:0/3729685260 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fcc00c660 con 0x7f3fc8004300 2024-03-06T07:22:09.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.279+0000 7f3fd8ff9640 1 --2- 172.21.15.1:0/3729685260 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3fbc06b500 0x7f3fbc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.279+0000 7f3fd8ff9640 1 -- 172.21.15.1:0/3729685260 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3fcc089f60 con 0x7f3fc8004300 2024-03-06T07:22:09.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.280+0000 7f3fdb7fe640 1 --2- 172.21.15.1:0/3729685260 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3fbc06b500 0x7f3fbc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.280+0000 7f3fdb7fe640 1 --2- 172.21.15.1:0/3729685260 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3fbc06b500 0x7f3fbc06d9c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8004780 tx=0x7f3fc4030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.284+0000 7f3fd8ff9640 1 -- 172.21.15.1:0/3729685260 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fcc058690 con 0x7f3fc8004300 2024-03-06T07:22:09.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.465+0000 7f3fe2615640 1 -- 172.21.15.1:0/3729685260 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3fdc0b06f0 con 0x7f3fbc06b500 2024-03-06T07:22:09.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.471+0000 7f3fd8ff9640 1 -- 172.21.15.1:0/3729685260 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240099 (secure 0 0 0) 0x7f3fdc0b06f0 con 0x7f3fbc06b500 2024-03-06T07:22:09.476 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:09.477 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:22:09.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.474+0000 7f3fe2615640 1 -- 172.21.15.1:0/3729685260 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3fbc06b500 msgr2=0x7f3fbc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.474+0000 7f3fe2615640 1 --2- 172.21.15.1:0/3729685260 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3fbc06b500 0x7f3fbc06d9c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8004780 tx=0x7f3fc4030000 comp rx=0 tx=0).stop 2024-03-06T07:22:09.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.474+0000 7f3fe2615640 1 -- 172.21.15.1:0/3729685260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fc8004300 msgr2=0x7f3fdc157550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.474+0000 7f3fe2615640 1 --2- 172.21.15.1:0/3729685260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fc8004300 0x7f3fdc157550 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc00e830 tx=0x7f3fcc00ecf0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.474+0000 7f3fe2615640 1 -- 172.21.15.1:0/3729685260 shutdown_connections 2024-03-06T07:22:09.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.474+0000 7f3fe2615640 1 --2- 172.21.15.1:0/3729685260 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3fbc06b500 0x7f3fbc06d9c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.474+0000 7f3fe2615640 1 --2- 172.21.15.1:0/3729685260 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fdc154190 0x7f3fdc1ed9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.474+0000 7f3fe2615640 1 --2- 172.21.15.1:0/3729685260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fdc158290 0x7f3fdc1eafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.474+0000 7f3fe2615640 1 --2- 172.21.15.1:0/3729685260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fc8004300 0x7f3fdc157550 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.474+0000 7f3fe2615640 1 -- 172.21.15.1:0/3729685260 >> 172.21.15.1:0/3729685260 conn(0x7f3fdc066ee0 msgr2=0x7f3fdc0c3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.474+0000 7f3fe2615640 1 -- 172.21.15.1:0/3729685260 shutdown_connections 2024-03-06T07:22:09.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.474+0000 7f3fe2615640 1 -- 172.21.15.1:0/3729685260 wait complete. 2024-03-06T07:22:09.487 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-03-06T07:22:08.713858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237452,"kb_used_data":4600,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737268,"statfs":{"total":773094113280,"available":772850962432,"internally_reserved":0,"allocated":4710400,"data_stored":3199506,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.419004"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.301253+0000","last_change":"2024-03-06T07:22:04.301253+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.301253+0000","last_undegraded":"2024-03-06T07:22:04.301253+0000","last_fullsized":"2024-03-06T07:22:04.301253+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296454+0000","last_change":"2024-03-06T07:22:04.296454+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296454+0000","last_undegraded":"2024-03-06T07:22:04.296454+0000","last_fullsized":"2024-03-06T07:22:04.296454+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296482+0000","last_change":"2024-03-06T07:22:04.296482+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296482+0000","last_undegraded":"2024-03-06T07:22:04.296482+0000","last_fullsized":"2024-03-06T07:22:04.296482+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302512+0000","last_change":"2024-03-06T07:22:04.302512+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302512+0000","last_undegraded":"2024-03-06T07:22:04.302512+0000","last_fullsized":"2024-03-06T07:22:04.302512+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302956+0000","last_change":"2024-03-06T07:22:04.302956+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302956+0000","last_undegraded":"2024-03-06T07:22:04.302956+0000","last_fullsized":"2024-03-06T07:22:04.302956+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.305118+0000","last_change":"2024-03-06T07:22:04.305118+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.305118+0000","last_undegraded":"2024-03-06T07:22:04.305118+0000","last_fullsized":"2024-03-06T07:22:04.305118+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.300365+0000","last_change":"2024-03-06T07:22:04.300365+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.300365+0000","last_undegraded":"2024-03-06T07:22:04.300365+0000","last_fullsized":"2024-03-06T07:22:04.300365+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302506+0000","last_change":"2024-03-06T07:22:04.302506+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302506+0000","last_undegraded":"2024-03-06T07:22:04.302506+0000","last_fullsized":"2024-03-06T07:22:04.302506+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299764+0000","last_change":"2024-03-06T07:22:04.299764+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299764+0000","last_undegraded":"2024-03-06T07:22:04.299764+0000","last_fullsized":"2024-03-06T07:22:04.299764+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.305086+0000","last_change":"2024-03-06T07:22:04.305086+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.305086+0000","last_undegraded":"2024-03-06T07:22:04.305086+0000","last_fullsized":"2024-03-06T07:22:04.305086+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.300440+0000","last_change":"2024-03-06T07:22:04.300440+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.300440+0000","last_undegraded":"2024-03-06T07:22:04.300440+0000","last_fullsized":"2024-03-06T07:22:04.300440+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296462+0000","last_change":"2024-03-06T07:22:04.296462+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296462+0000","last_undegraded":"2024-03-06T07:22:04.296462+0000","last_fullsized":"2024-03-06T07:22:04.296462+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302506+0000","last_change":"2024-03-06T07:22:04.302506+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302506+0000","last_undegraded":"2024-03-06T07:22:04.302506+0000","last_fullsized":"2024-03-06T07:22:04.302506+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299827+0000","last_change":"2024-03-06T07:22:04.299827+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299827+0000","last_undegraded":"2024-03-06T07:22:04.299827+0000","last_fullsized":"2024-03-06T07:22:04.299827+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299759+0000","last_change":"2024-03-06T07:22:04.299759+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299759+0000","last_undegraded":"2024-03-06T07:22:04.299759+0000","last_fullsized":"2024-03-06T07:22:04.299759+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.297059+0000","last_change":"2024-03-06T07:22:04.297059+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.297059+0000","last_undegraded":"2024-03-06T07:22:04.297059+0000","last_fullsized":"2024-03-06T07:22:04.297059+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.303050+0000","last_change":"2024-03-06T07:22:04.303050+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.303050+0000","last_undegraded":"2024-03-06T07:22:04.303050+0000","last_fullsized":"2024-03-06T07:22:04.303050+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.297784+0000","last_change":"2024-03-06T07:22:04.297784+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.297784+0000","last_undegraded":"2024-03-06T07:22:04.297784+0000","last_fullsized":"2024-03-06T07:22:04.297784+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.300752+0000","last_change":"2024-03-06T07:22:04.300752+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.300752+0000","last_undegraded":"2024-03-06T07:22:04.300752+0000","last_fullsized":"2024-03-06T07:22:04.300752+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299956+0000","last_change":"2024-03-06T07:22:04.299956+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299956+0000","last_undegraded":"2024-03-06T07:22:04.299956+0000","last_fullsized":"2024-03-06T07:22:04.299956+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296314+0000","last_change":"2024-03-06T07:22:04.296314+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296314+0000","last_undegraded":"2024-03-06T07:22:04.296314+0000","last_fullsized":"2024-03-06T07:22:04.296314+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.297197+0000","last_change":"2024-03-06T07:22:04.297197+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.297197+0000","last_undegraded":"2024-03-06T07:22:04.297197+0000","last_fullsized":"2024-03-06T07:22:04.297197+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302534+0000","last_change":"2024-03-06T07:22:04.302534+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302534+0000","last_undegraded":"2024-03-06T07:22:04.302534+0000","last_fullsized":"2024-03-06T07:22:04.302534+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.297668+0000","last_change":"2024-03-06T07:22:04.297668+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.297668+0000","last_undegraded":"2024-03-06T07:22:04.297668+0000","last_fullsized":"2024-03-06T07:22:04.297668+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.300822+0000","last_change":"2024-03-06T07:22:04.300822+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.300822+0000","last_undegraded":"2024-03-06T07:22:04.300822+0000","last_fullsized":"2024-03-06T07:22:04.300822+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299878+0000","last_change":"2024-03-06T07:22:04.299878+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299878+0000","last_undegraded":"2024-03-06T07:22:04.299878+0000","last_fullsized":"2024-03-06T07:22:04.299878+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302664+0000","last_change":"2024-03-06T07:22:04.302664+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302664+0000","last_undegraded":"2024-03-06T07:22:04.302664+0000","last_fullsized":"2024-03-06T07:22:04.302664+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296474+0000","last_change":"2024-03-06T07:22:04.296474+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296474+0000","last_undegraded":"2024-03-06T07:22:04.296474+0000","last_fullsized":"2024-03-06T07:22:04.296474+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296739+0000","last_change":"2024-03-06T07:22:04.296739+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296739+0000","last_undegraded":"2024-03-06T07:22:04.296739+0000","last_fullsized":"2024-03-06T07:22:04.296739+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.305047+0000","last_change":"2024-03-06T07:22:04.305047+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.305047+0000","last_undegraded":"2024-03-06T07:22:04.305047+0000","last_fullsized":"2024-03-06T07:22:04.305047+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299860+0000","last_change":"2024-03-06T07:22:04.299860+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299860+0000","last_undegraded":"2024-03-06T07:22:04.299860+0000","last_fullsized":"2024-03-06T07:22:04.299860+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302620+0000","last_change":"2024-03-06T07:22:04.302620+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302620+0000","last_undegraded":"2024-03-06T07:22:04.302620+0000","last_fullsized":"2024-03-06T07:22:04.302620+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":224,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-06T07:22:04.301339+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:22:04.301339+0000","last_peered":"2024-03-06T07:22:04.301339+0000","last_clean":"2024-03-06T07:22:04.301339+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:22:04.301339+0000","last_undegraded":"2024-03-06T07:22:04.301339+0000","last_fullsized":"2024-03-06T07:22:04.301339+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":634880,"data_stored":841162,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27584,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344256,"statfs":{"total":96636764160,"available":96608518144,"internally_reserved":0,"allocated":569344,"data_stored":250794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27584,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344256,"statfs":{"total":96636764160,"available":96608518144,"internally_reserved":0,"allocated":569344,"data_stored":250794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":569344,"data_stored":250794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31748,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340092,"statfs":{"total":96636764160,"available":96604254208,"internally_reserved":0,"allocated":581632,"data_stored":257003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31736,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340104,"statfs":{"total":96636764160,"available":96604266496,"internally_reserved":0,"allocated":569344,"data_stored":250794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31748,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340092,"statfs":{"total":96636764160,"available":96604254208,"internally_reserved":0,"allocated":581632,"data_stored":257003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":634880,"data_stored":841162,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:09.491 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:09.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.627+0000 7ff93d5ac640 1 Processor -- start 2024-03-06T07:22:09.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.627+0000 7ff93d5ac640 1 -- start start 2024-03-06T07:22:09.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.628+0000 7ff93d5ac640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff9380c6b10 0x7ff938155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.628+0000 7ff93d5ac640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9380c2fc0 con 0x7ff9380bf9d0 2024-03-06T07:22:09.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.628+0000 7ff93d5ac640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff938158da0 con 0x7ff938152650 2024-03-06T07:22:09.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.628+0000 7ff93d5ac640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff938155800 con 0x7ff9380c6b10 2024-03-06T07:22:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.628+0000 7ff9377fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff9380bf9d0 0x7ff9380c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45354/0 (socket says 172.21.15.1:45354) 2024-03-06T07:22:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.628+0000 7ff9377fe640 1 -- 172.21.15.1:0/1338059732 learned_addr learned my addr 172.21.15.1:0/1338059732 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.628+0000 7ff9367fc640 1 --2- 172.21.15.1:0/1338059732 >> v2:172.21.15.98:3300/0 conn(0x7ff9380c6b10 0x7ff938155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.628+0000 7ff935ffb640 1 -- 172.21.15.1:0/1338059732 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478633172 0 0) 0x7ff9380c2fc0 con 0x7ff9380bf9d0 2024-03-06T07:22:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.628+0000 7ff935ffb640 1 -- 172.21.15.1:0/1338059732 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff920003520 con 0x7ff9380bf9d0 2024-03-06T07:22:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.628+0000 7ff935ffb640 1 -- 172.21.15.1:0/1338059732 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 351151370 0 0) 0x7ff920003520 con 0x7ff9380bf9d0 2024-03-06T07:22:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.628+0000 7ff935ffb640 1 -- 172.21.15.1:0/1338059732 >> v2:172.21.15.98:3300/0 conn(0x7ff9380c6b10 msgr2=0x7ff938155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.628+0000 7ff935ffb640 1 --2- 172.21.15.1:0/1338059732 >> v2:172.21.15.98:3300/0 conn(0x7ff9380c6b10 0x7ff938155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.628+0000 7ff935ffb640 1 -- 172.21.15.1:0/1338059732 >> v1:172.21.15.98:6789/0 conn(0x7ff938152650 legacy=0x7ff938158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.628+0000 7ff935ffb640 1 -- 172.21.15.1:0/1338059732 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9380c6690 con 0x7ff9380bf9d0 2024-03-06T07:22:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.628+0000 7ff935ffb640 1 -- 172.21.15.1:0/1338059732 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7ff9300019f0 con 0x7ff9380bf9d0 2024-03-06T07:22:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.629+0000 7ff935ffb640 1 -- 172.21.15.1:0/1338059732 >> v1:172.21.15.1:6789/0 conn(0x7ff9380bf9d0 legacy=0x7ff9380c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.629+0000 7ff935ffb640 1 --2- 172.21.15.1:0/1338059732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9200039b0 0x7ff920003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.629+0000 7ff935ffb640 1 --2- 172.21.15.1:0/1338059732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff920004800 0x7ff920004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.629+0000 7ff935ffb640 1 --2- 172.21.15.1:0/1338059732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9200051c0 0x7ff92000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.629+0000 7ff935ffb640 1 -- 172.21.15.1:0/1338059732 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff938155800 con 0x7ff9200051c0 2024-03-06T07:22:09.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.629+0000 7ff935ffb640 1 -- 172.21.15.1:0/1338059732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92000c160 con 0x7ff920004800 2024-03-06T07:22:09.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.629+0000 7ff935ffb640 1 -- 172.21.15.1:0/1338059732 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92000c360 con 0x7ff9200039b0 2024-03-06T07:22:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.629+0000 7ff9377fe640 1 --2- 172.21.15.1:0/1338059732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9200051c0 0x7ff92000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.629+0000 7ff9367fc640 1 --2- 172.21.15.1:0/1338059732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff920004800 0x7ff920004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.629+0000 7ff936ffd640 1 --2- 172.21.15.1:0/1338059732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9200039b0 0x7ff920003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.629+0000 7ff9377fe640 1 -- 172.21.15.1:0/1338059732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9200039b0 msgr2=0x7ff920003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.629+0000 7ff9377fe640 1 --2- 172.21.15.1:0/1338059732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9200039b0 0x7ff920003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.629+0000 7ff9377fe640 1 -- 172.21.15.1:0/1338059732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff920004800 msgr2=0x7ff920004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.629+0000 7ff9377fe640 1 --2- 172.21.15.1:0/1338059732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff920004800 0x7ff920004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.629+0000 7ff9377fe640 1 -- 172.21.15.1:0/1338059732 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff920003520 con 0x7ff9200051c0 2024-03-06T07:22:09.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.629+0000 7ff9367fc640 1 --2- 172.21.15.1:0/1338059732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff920004800 0x7ff920004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:09.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.630+0000 7ff9377fe640 1 -- 172.21.15.1:0/1338059732 reap_dead start 2024-03-06T07:22:09.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.630+0000 7ff9377fe640 1 --2- 172.21.15.1:0/1338059732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9200051c0 0x7ff92000b9c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff93000a840 tx=0x7ff920005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439877e70f6f29b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.630+0000 7ff935ffb640 1 -- 172.21.15.1:0/1338059732 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c009690 con 0x7ff9200051c0 2024-03-06T07:22:09.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.630+0000 7ff935ffb640 1 -- 172.21.15.1:0/1338059732 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff930034650 con 0x7ff9200051c0 2024-03-06T07:22:09.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.630+0000 7ff935ffb640 1 -- 172.21.15.1:0/1338059732 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93000a460 con 0x7ff9200051c0 2024-03-06T07:22:09.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.631+0000 7ff93d5ac640 1 -- 172.21.15.1:0/1338059732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9200051c0 msgr2=0x7ff92000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.631+0000 7ff93d5ac640 1 --2- 172.21.15.1:0/1338059732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9200051c0 0x7ff92000b9c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff93000a840 tx=0x7ff920005830 comp rx=0 tx=0).stop 2024-03-06T07:22:09.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.631+0000 7ff93d5ac640 1 -- 172.21.15.1:0/1338059732 shutdown_connections 2024-03-06T07:22:09.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.631+0000 7ff93d5ac640 1 --2- 172.21.15.1:0/1338059732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9200051c0 0x7ff92000b9c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.631+0000 7ff93d5ac640 1 -- 172.21.15.1:0/1338059732 >> 172.21.15.1:0/1338059732 conn(0x7ff938066ee0 msgr2=0x7ff9380672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.631+0000 7ff93d5ac640 1 -- 172.21.15.1:0/1338059732 shutdown_connections 2024-03-06T07:22:09.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.631+0000 7ff93d5ac640 1 -- 172.21.15.1:0/1338059732 wait complete. 2024-03-06T07:22:09.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.633+0000 7ff93d5ac640 1 Processor -- start 2024-03-06T07:22:09.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.633+0000 7ff93d5ac640 1 -- start start 2024-03-06T07:22:09.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.634+0000 7ff93d5ac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9200051c0 0x7ff938158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.634+0000 7ff93d5ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff938151850 0x7ff9381eb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.634+0000 7ff93d5ac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9381eb800 0x7ff93814afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.634+0000 7ff93d5ac640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9381551b0 con 0x7ff9381eb800 2024-03-06T07:22:09.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.634+0000 7ff93d5ac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9380c6fa0 con 0x7ff9200051c0 2024-03-06T07:22:09.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.634+0000 7ff93d5ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff938155490 con 0x7ff938151850 2024-03-06T07:22:09.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.634+0000 7ff9377fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9381eb800 0x7ff93814afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.634+0000 7ff9377fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9381eb800 0x7ff93814afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59556/0 (socket says 172.21.15.1:59556) 2024-03-06T07:22:09.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.634+0000 7ff9377fe640 1 -- 172.21.15.1:0/816962659 learned_addr learned my addr 172.21.15.1:0/816962659 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:09.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.634+0000 7ff936ffd640 1 --2- 172.21.15.1:0/816962659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9200051c0 0x7ff938158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.634+0000 7ff9367fc640 1 --2- 172.21.15.1:0/816962659 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff938151850 0x7ff9381eb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.634+0000 7ff9377fe640 1 -- 172.21.15.1:0/816962659 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff938151850 msgr2=0x7ff9381eb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.634+0000 7ff9377fe640 1 --2- 172.21.15.1:0/816962659 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff938151850 0x7ff9381eb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.634+0000 7ff9377fe640 1 -- 172.21.15.1:0/816962659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9200051c0 msgr2=0x7ff938158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.634+0000 7ff9377fe640 1 --2- 172.21.15.1:0/816962659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9200051c0 0x7ff938158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.634+0000 7ff9377fe640 1 -- 172.21.15.1:0/816962659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff938156c10 con 0x7ff9381eb800 2024-03-06T07:22:09.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.634+0000 7ff936ffd640 1 --2- 172.21.15.1:0/816962659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9200051c0 0x7ff938158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:09.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.635+0000 7ff9377fe640 1 --2- 172.21.15.1:0/816962659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9381eb800 0x7ff93814afa0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ff93000a930 tx=0x7ff9380bfe40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.635+0000 7ff91f7fe640 1 -- 172.21.15.1:0/816962659 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff930044070 con 0x7ff9381eb800 2024-03-06T07:22:09.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.635+0000 7ff91f7fe640 1 -- 172.21.15.1:0/816962659 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93003a040 con 0x7ff9381eb800 2024-03-06T07:22:09.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.635+0000 7ff93d5ac640 1 -- 172.21.15.1:0/816962659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9381ef070 con 0x7ff9381eb800 2024-03-06T07:22:09.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.635+0000 7ff91f7fe640 1 -- 172.21.15.1:0/816962659 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9300019c0 con 0x7ff9381eb800 2024-03-06T07:22:09.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.635+0000 7ff93d5ac640 1 -- 172.21.15.1:0/816962659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff938157190 con 0x7ff9381eb800 2024-03-06T07:22:09.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.636+0000 7ff93d5ac640 1 -- 172.21.15.1:0/816962659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff938067fd0 con 0x7ff9381eb800 2024-03-06T07:22:09.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.636+0000 7ff91f7fe640 1 -- 172.21.15.1:0/816962659 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff930047070 con 0x7ff9381eb800 2024-03-06T07:22:09.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.637+0000 7ff91f7fe640 1 --2- 172.21.15.1:0/816962659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff92406b4d0 0x7ff92406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.637+0000 7ff91f7fe640 1 -- 172.21.15.1:0/816962659 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff9300abbc0 con 0x7ff9381eb800 2024-03-06T07:22:09.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.637+0000 7ff936ffd640 1 --2- 172.21.15.1:0/816962659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff92406b4d0 0x7ff92406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.638+0000 7ff936ffd640 1 --2- 172.21.15.1:0/816962659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff92406b4d0 0x7ff92406d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff928000c00 tx=0x7ff928009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.641+0000 7ff91f7fe640 1 -- 172.21.15.1:0/816962659 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff93007a260 con 0x7ff9381eb800 2024-03-06T07:22:09.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.820+0000 7ff93d5ac640 1 -- 172.21.15.1:0/816962659 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff9380b06f0 con 0x7ff92406b4d0 2024-03-06T07:22:09.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.826+0000 7ff91f7fe640 1 -- 172.21.15.1:0/816962659 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240099 (secure 0 0 0) 0x7ff9380b06f0 con 0x7ff92406b4d0 2024-03-06T07:22:09.831 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:09.832 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:22:09.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.830+0000 7ff93d5ac640 1 -- 172.21.15.1:0/816962659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff92406b4d0 msgr2=0x7ff92406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.830+0000 7ff93d5ac640 1 --2- 172.21.15.1:0/816962659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff92406b4d0 0x7ff92406d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff928000c00 tx=0x7ff928009040 comp rx=0 tx=0).stop 2024-03-06T07:22:09.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.830+0000 7ff93d5ac640 1 -- 172.21.15.1:0/816962659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9381eb800 msgr2=0x7ff93814afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.830+0000 7ff93d5ac640 1 --2- 172.21.15.1:0/816962659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9381eb800 0x7ff93814afa0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ff93000a930 tx=0x7ff9380bfe40 comp rx=0 tx=0).stop 2024-03-06T07:22:09.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.830+0000 7ff93d5ac640 1 -- 172.21.15.1:0/816962659 shutdown_connections 2024-03-06T07:22:09.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.830+0000 7ff93d5ac640 1 --2- 172.21.15.1:0/816962659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff92406b4d0 0x7ff92406d990 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.830+0000 7ff93d5ac640 1 --2- 172.21.15.1:0/816962659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9381eb800 0x7ff93814afa0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.830+0000 7ff93d5ac640 1 --2- 172.21.15.1:0/816962659 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff938151850 0x7ff9381eb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.830+0000 7ff93d5ac640 1 --2- 172.21.15.1:0/816962659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9200051c0 0x7ff938158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.830+0000 7ff93d5ac640 1 -- 172.21.15.1:0/816962659 >> 172.21.15.1:0/816962659 conn(0x7ff938066ee0 msgr2=0x7ff9380c3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.830+0000 7ff93d5ac640 1 -- 172.21.15.1:0/816962659 shutdown_connections 2024-03-06T07:22:09.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.830+0000 7ff93d5ac640 1 -- 172.21.15.1:0/816962659 wait complete. 2024-03-06T07:22:09.843 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-03-06T07:22:08.713858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237452,"kb_used_data":4600,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737268,"statfs":{"total":773094113280,"available":772850962432,"internally_reserved":0,"allocated":4710400,"data_stored":3199506,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.419004"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.301253+0000","last_change":"2024-03-06T07:22:04.301253+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.301253+0000","last_undegraded":"2024-03-06T07:22:04.301253+0000","last_fullsized":"2024-03-06T07:22:04.301253+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296454+0000","last_change":"2024-03-06T07:22:04.296454+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296454+0000","last_undegraded":"2024-03-06T07:22:04.296454+0000","last_fullsized":"2024-03-06T07:22:04.296454+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296482+0000","last_change":"2024-03-06T07:22:04.296482+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296482+0000","last_undegraded":"2024-03-06T07:22:04.296482+0000","last_fullsized":"2024-03-06T07:22:04.296482+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302512+0000","last_change":"2024-03-06T07:22:04.302512+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302512+0000","last_undegraded":"2024-03-06T07:22:04.302512+0000","last_fullsized":"2024-03-06T07:22:04.302512+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302956+0000","last_change":"2024-03-06T07:22:04.302956+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302956+0000","last_undegraded":"2024-03-06T07:22:04.302956+0000","last_fullsized":"2024-03-06T07:22:04.302956+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.305118+0000","last_change":"2024-03-06T07:22:04.305118+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.305118+0000","last_undegraded":"2024-03-06T07:22:04.305118+0000","last_fullsized":"2024-03-06T07:22:04.305118+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.300365+0000","last_change":"2024-03-06T07:22:04.300365+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.300365+0000","last_undegraded":"2024-03-06T07:22:04.300365+0000","last_fullsized":"2024-03-06T07:22:04.300365+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302506+0000","last_change":"2024-03-06T07:22:04.302506+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302506+0000","last_undegraded":"2024-03-06T07:22:04.302506+0000","last_fullsized":"2024-03-06T07:22:04.302506+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299764+0000","last_change":"2024-03-06T07:22:04.299764+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299764+0000","last_undegraded":"2024-03-06T07:22:04.299764+0000","last_fullsized":"2024-03-06T07:22:04.299764+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.305086+0000","last_change":"2024-03-06T07:22:04.305086+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.305086+0000","last_undegraded":"2024-03-06T07:22:04.305086+0000","last_fullsized":"2024-03-06T07:22:04.305086+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.300440+0000","last_change":"2024-03-06T07:22:04.300440+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.300440+0000","last_undegraded":"2024-03-06T07:22:04.300440+0000","last_fullsized":"2024-03-06T07:22:04.300440+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296462+0000","last_change":"2024-03-06T07:22:04.296462+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296462+0000","last_undegraded":"2024-03-06T07:22:04.296462+0000","last_fullsized":"2024-03-06T07:22:04.296462+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302506+0000","last_change":"2024-03-06T07:22:04.302506+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302506+0000","last_undegraded":"2024-03-06T07:22:04.302506+0000","last_fullsized":"2024-03-06T07:22:04.302506+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299827+0000","last_change":"2024-03-06T07:22:04.299827+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299827+0000","last_undegraded":"2024-03-06T07:22:04.299827+0000","last_fullsized":"2024-03-06T07:22:04.299827+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299759+0000","last_change":"2024-03-06T07:22:04.299759+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299759+0000","last_undegraded":"2024-03-06T07:22:04.299759+0000","last_fullsized":"2024-03-06T07:22:04.299759+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.297059+0000","last_change":"2024-03-06T07:22:04.297059+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.297059+0000","last_undegraded":"2024-03-06T07:22:04.297059+0000","last_fullsized":"2024-03-06T07:22:04.297059+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.303050+0000","last_change":"2024-03-06T07:22:04.303050+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.303050+0000","last_undegraded":"2024-03-06T07:22:04.303050+0000","last_fullsized":"2024-03-06T07:22:04.303050+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.297784+0000","last_change":"2024-03-06T07:22:04.297784+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.297784+0000","last_undegraded":"2024-03-06T07:22:04.297784+0000","last_fullsized":"2024-03-06T07:22:04.297784+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.300752+0000","last_change":"2024-03-06T07:22:04.300752+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.300752+0000","last_undegraded":"2024-03-06T07:22:04.300752+0000","last_fullsized":"2024-03-06T07:22:04.300752+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299956+0000","last_change":"2024-03-06T07:22:04.299956+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299956+0000","last_undegraded":"2024-03-06T07:22:04.299956+0000","last_fullsized":"2024-03-06T07:22:04.299956+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296314+0000","last_change":"2024-03-06T07:22:04.296314+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296314+0000","last_undegraded":"2024-03-06T07:22:04.296314+0000","last_fullsized":"2024-03-06T07:22:04.296314+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.297197+0000","last_change":"2024-03-06T07:22:04.297197+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.297197+0000","last_undegraded":"2024-03-06T07:22:04.297197+0000","last_fullsized":"2024-03-06T07:22:04.297197+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302534+0000","last_change":"2024-03-06T07:22:04.302534+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302534+0000","last_undegraded":"2024-03-06T07:22:04.302534+0000","last_fullsized":"2024-03-06T07:22:04.302534+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.297668+0000","last_change":"2024-03-06T07:22:04.297668+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.297668+0000","last_undegraded":"2024-03-06T07:22:04.297668+0000","last_fullsized":"2024-03-06T07:22:04.297668+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.300822+0000","last_change":"2024-03-06T07:22:04.300822+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.300822+0000","last_undegraded":"2024-03-06T07:22:04.300822+0000","last_fullsized":"2024-03-06T07:22:04.300822+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299878+0000","last_change":"2024-03-06T07:22:04.299878+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299878+0000","last_undegraded":"2024-03-06T07:22:04.299878+0000","last_fullsized":"2024-03-06T07:22:04.299878+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302664+0000","last_change":"2024-03-06T07:22:04.302664+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302664+0000","last_undegraded":"2024-03-06T07:22:04.302664+0000","last_fullsized":"2024-03-06T07:22:04.302664+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296474+0000","last_change":"2024-03-06T07:22:04.296474+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296474+0000","last_undegraded":"2024-03-06T07:22:04.296474+0000","last_fullsized":"2024-03-06T07:22:04.296474+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296739+0000","last_change":"2024-03-06T07:22:04.296739+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296739+0000","last_undegraded":"2024-03-06T07:22:04.296739+0000","last_fullsized":"2024-03-06T07:22:04.296739+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.305047+0000","last_change":"2024-03-06T07:22:04.305047+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.305047+0000","last_undegraded":"2024-03-06T07:22:04.305047+0000","last_fullsized":"2024-03-06T07:22:04.305047+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299860+0000","last_change":"2024-03-06T07:22:04.299860+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299860+0000","last_undegraded":"2024-03-06T07:22:04.299860+0000","last_fullsized":"2024-03-06T07:22:04.299860+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302620+0000","last_change":"2024-03-06T07:22:04.302620+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302620+0000","last_undegraded":"2024-03-06T07:22:04.302620+0000","last_fullsized":"2024-03-06T07:22:04.302620+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":224,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-06T07:22:04.301339+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:22:04.301339+0000","last_peered":"2024-03-06T07:22:04.301339+0000","last_clean":"2024-03-06T07:22:04.301339+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:22:04.301339+0000","last_undegraded":"2024-03-06T07:22:04.301339+0000","last_fullsized":"2024-03-06T07:22:04.301339+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":634880,"data_stored":841162,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27584,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344256,"statfs":{"total":96636764160,"available":96608518144,"internally_reserved":0,"allocated":569344,"data_stored":250794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27584,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344256,"statfs":{"total":96636764160,"available":96608518144,"internally_reserved":0,"allocated":569344,"data_stored":250794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":569344,"data_stored":250794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31748,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340092,"statfs":{"total":96636764160,"available":96604254208,"internally_reserved":0,"allocated":581632,"data_stored":257003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31736,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340104,"statfs":{"total":96636764160,"available":96604266496,"internally_reserved":0,"allocated":569344,"data_stored":250794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31748,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340092,"statfs":{"total":96636764160,"available":96604254208,"internally_reserved":0,"allocated":581632,"data_stored":257003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":634880,"data_stored":841162,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:09.847 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.983+0000 7f8d50965640 1 Processor -- start 2024-03-06T07:22:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.983+0000 7f8d50965640 1 -- start start 2024-03-06T07:22:09.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.985+0000 7f8d50965640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d4c14f660 0x7f8d4c152a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.985+0000 7f8d50965640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8d4c072c20 0x7f8d4c0a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.985+0000 7f8d50965640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d4c14fa40 con 0x7f8d4c06c100 2024-03-06T07:22:09.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.985+0000 7f8d50965640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d4c06f6f0 con 0x7f8d4c072c20 2024-03-06T07:22:09.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.985+0000 7f8d50965640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c152f40 con 0x7f8d4c14f660 2024-03-06T07:22:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.985+0000 7f8d49d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8d4c072c20 0x7f8d4c0a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.985+0000 7f8d4a59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d4c14f660 0x7f8d4c152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.985+0000 7f8d4ad9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8d4c06c100 0x7f8d4c155560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:49002/0 (socket says 172.21.15.1:49002) 2024-03-06T07:22:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.985+0000 7f8d4ad9d640 1 -- 172.21.15.1:0/1684577021 learned_addr learned my addr 172.21.15.1:0/1684577021 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.985+0000 7f8d49d9b640 1 -- 172.21.15.1:0/1684577021 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d4c14f660 msgr2=0x7f8d4c152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.985+0000 7f8d49d9b640 1 --2- 172.21.15.1:0/1684577021 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d4c14f660 0x7f8d4c152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.985+0000 7f8d49d9b640 1 -- 172.21.15.1:0/1684577021 >> v1:172.21.15.98:6789/0 conn(0x7f8d4c06c100 legacy=0x7f8d4c155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.985+0000 7f8d49d9b640 1 -- 172.21.15.1:0/1684577021 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c0a89f0 con 0x7f8d4c072c20 2024-03-06T07:22:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.985+0000 7f8d4959a640 1 -- 172.21.15.1:0/1684577021 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820021272 0 0) 0x7f8d4c14fa40 con 0x7f8d4c06c100 2024-03-06T07:22:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.985+0000 7f8d4a59c640 1 --2- 172.21.15.1:0/1684577021 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d4c14f660 0x7f8d4c152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:09.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.986+0000 7f8d49d9b640 1 --2- 172.21.15.1:0/1684577021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d4c072c20 0x7f8d4c0a84b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8d400097a0 tx=0x7f8d4002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dfc0904a3799173 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.986+0000 7f8d4959a640 1 -- 172.21.15.1:0/1684577021 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d400315a0 con 0x7f8d4c072c20 2024-03-06T07:22:09.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.986+0000 7f8d4959a640 1 -- 172.21.15.1:0/1684577021 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d40037070 con 0x7f8d4c072c20 2024-03-06T07:22:09.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.986+0000 7f8d4959a640 1 -- 172.21.15.1:0/1684577021 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d40031960 con 0x7f8d4c072c20 2024-03-06T07:22:09.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.987+0000 7f8d50965640 1 -- 172.21.15.1:0/1684577021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d4c072c20 msgr2=0x7f8d4c0a84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.987+0000 7f8d50965640 1 --2- 172.21.15.1:0/1684577021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d4c072c20 0x7f8d4c0a84b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8d400097a0 tx=0x7f8d4002b840 comp rx=0 tx=0).stop 2024-03-06T07:22:09.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.987+0000 7f8d50965640 1 -- 172.21.15.1:0/1684577021 shutdown_connections 2024-03-06T07:22:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.987+0000 7f8d50965640 1 --2- 172.21.15.1:0/1684577021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d4c072c20 0x7f8d4c0a84b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.987+0000 7f8d50965640 1 --2- 172.21.15.1:0/1684577021 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d4c14f660 0x7f8d4c152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.987+0000 7f8d50965640 1 -- 172.21.15.1:0/1684577021 >> 172.21.15.1:0/1684577021 conn(0x7f8d4c071cf0 msgr2=0x7f8d4c0720d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.987+0000 7f8d50965640 1 -- 172.21.15.1:0/1684577021 shutdown_connections 2024-03-06T07:22:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.987+0000 7f8d50965640 1 -- 172.21.15.1:0/1684577021 wait complete. 2024-03-06T07:22:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.988+0000 7f8d50965640 1 Processor -- start 2024-03-06T07:22:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.988+0000 7f8d50965640 1 -- start start 2024-03-06T07:22:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.989+0000 7f8d50965640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d4c06c100 0x7f8d4c145bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.989+0000 7f8d50965640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d4c072c20 0x7f8d4c1df980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.989+0000 7f8d50965640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d4c14f660 0x7f8d4c1eafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.989+0000 7f8d50965640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c152200 con 0x7f8d4c06c100 2024-03-06T07:22:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.989+0000 7f8d50965640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c06fc20 con 0x7f8d4c14f660 2024-03-06T07:22:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.989+0000 7f8d50965640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c1524e0 con 0x7f8d4c072c20 2024-03-06T07:22:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.989+0000 7f8d4a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d4c06c100 0x7f8d4c145bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.989+0000 7f8d4a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d4c06c100 0x7f8d4c145bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59562/0 (socket says 172.21.15.1:59562) 2024-03-06T07:22:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.989+0000 7f8d4a59c640 1 -- 172.21.15.1:0/866285695 learned_addr learned my addr 172.21.15.1:0/866285695 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.989+0000 7f8d49d9b640 1 --2- 172.21.15.1:0/866285695 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d4c072c20 0x7f8d4c1df980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.989+0000 7f8d4ad9d640 1 --2- 172.21.15.1:0/866285695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d4c14f660 0x7f8d4c1eafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.990+0000 7f8d4a59c640 1 -- 172.21.15.1:0/866285695 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d4c072c20 msgr2=0x7f8d4c1df980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.990+0000 7f8d4a59c640 1 --2- 172.21.15.1:0/866285695 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d4c072c20 0x7f8d4c1df980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.990+0000 7f8d4a59c640 1 -- 172.21.15.1:0/866285695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d4c14f660 msgr2=0x7f8d4c1eafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.990+0000 7f8d4a59c640 1 --2- 172.21.15.1:0/866285695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d4c14f660 0x7f8d4c1eafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.990+0000 7f8d4a59c640 1 -- 172.21.15.1:0/866285695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c1ea8e0 con 0x7f8d4c06c100 2024-03-06T07:22:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.990+0000 7f8d4a59c640 1 --2- 172.21.15.1:0/866285695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d4c06c100 0x7f8d4c145bc0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00ebe0 tx=0x7f8d3c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.990+0000 7f8d3affd640 1 -- 172.21.15.1:0/866285695 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c017070 con 0x7f8d4c06c100 2024-03-06T07:22:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.990+0000 7f8d3affd640 1 -- 172.21.15.1:0/866285695 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3c00cc30 con 0x7f8d4c06c100 2024-03-06T07:22:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.990+0000 7f8d50965640 1 -- 172.21.15.1:0/866285695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d4c14b010 con 0x7f8d4c06c100 2024-03-06T07:22:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.990+0000 7f8d3affd640 1 -- 172.21.15.1:0/866285695 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c013470 con 0x7f8d4c06c100 2024-03-06T07:22:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.990+0000 7f8d50965640 1 -- 172.21.15.1:0/866285695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4c14b8b0 con 0x7f8d4c06c100 2024-03-06T07:22:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.991+0000 7f8d50965640 1 -- 172.21.15.1:0/866285695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4c0672b0 con 0x7f8d4c06c100 2024-03-06T07:22:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.991+0000 7f8d3affd640 1 -- 172.21.15.1:0/866285695 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d3c01a070 con 0x7f8d4c06c100 2024-03-06T07:22:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.992+0000 7f8d3affd640 1 --2- 172.21.15.1:0/866285695 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d2c06b4d0 0x7f8d2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.992+0000 7f8d3affd640 1 -- 172.21.15.1:0/866285695 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8d3c08a1b0 con 0x7f8d4c06c100 2024-03-06T07:22:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.992+0000 7f8d49d9b640 1 --2- 172.21.15.1:0/866285695 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d2c06b4d0 0x7f8d2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.993+0000 7f8d49d9b640 1 --2- 172.21.15.1:0/866285695 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d2c06b4d0 0x7f8d2c06d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8d40036000 tx=0x7f8d4003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:09.996+0000 7f8d3affd640 1 -- 172.21.15.1:0/866285695 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d3c0588e0 con 0x7f8d4c06c100 2024-03-06T07:22:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.173+0000 7f8d50965640 1 -- 172.21.15.1:0/866285695 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8d4c0a8650 con 0x7f8d2c06b4d0 2024-03-06T07:22:10.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.179+0000 7f8d3affd640 1 -- 172.21.15.1:0/866285695 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240099 (secure 0 0 0) 0x7f8d4c0a8650 con 0x7f8d2c06b4d0 2024-03-06T07:22:10.185 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:10.187 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:22:10.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.182+0000 7f8d50965640 1 -- 172.21.15.1:0/866285695 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d2c06b4d0 msgr2=0x7f8d2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.182+0000 7f8d50965640 1 --2- 172.21.15.1:0/866285695 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d2c06b4d0 0x7f8d2c06d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8d40036000 tx=0x7f8d4003d040 comp rx=0 tx=0).stop 2024-03-06T07:22:10.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.182+0000 7f8d50965640 1 -- 172.21.15.1:0/866285695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d4c06c100 msgr2=0x7f8d4c145bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.182+0000 7f8d50965640 1 --2- 172.21.15.1:0/866285695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d4c06c100 0x7f8d4c145bc0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00ebe0 tx=0x7f8d3c00c520 comp rx=0 tx=0).stop 2024-03-06T07:22:10.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.182+0000 7f8d50965640 1 -- 172.21.15.1:0/866285695 shutdown_connections 2024-03-06T07:22:10.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.182+0000 7f8d50965640 1 --2- 172.21.15.1:0/866285695 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d2c06b4d0 0x7f8d2c06d990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.182+0000 7f8d50965640 1 --2- 172.21.15.1:0/866285695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d4c14f660 0x7f8d4c1eafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.182+0000 7f8d50965640 1 --2- 172.21.15.1:0/866285695 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d4c072c20 0x7f8d4c1df980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.182+0000 7f8d50965640 1 --2- 172.21.15.1:0/866285695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d4c06c100 0x7f8d4c145bc0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.182+0000 7f8d50965640 1 -- 172.21.15.1:0/866285695 >> 172.21.15.1:0/866285695 conn(0x7f8d4c071cf0 msgr2=0x7f8d4c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.182+0000 7f8d50965640 1 -- 172.21.15.1:0/866285695 shutdown_connections 2024-03-06T07:22:10.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.182+0000 7f8d50965640 1 -- 172.21.15.1:0/866285695 wait complete. 2024-03-06T07:22:10.194 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-03-06T07:22:08.713858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237452,"kb_used_data":4600,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737268,"statfs":{"total":773094113280,"available":772850962432,"internally_reserved":0,"allocated":4710400,"data_stored":3199506,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.419004"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.301253+0000","last_change":"2024-03-06T07:22:04.301253+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.301253+0000","last_undegraded":"2024-03-06T07:22:04.301253+0000","last_fullsized":"2024-03-06T07:22:04.301253+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296454+0000","last_change":"2024-03-06T07:22:04.296454+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296454+0000","last_undegraded":"2024-03-06T07:22:04.296454+0000","last_fullsized":"2024-03-06T07:22:04.296454+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296482+0000","last_change":"2024-03-06T07:22:04.296482+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296482+0000","last_undegraded":"2024-03-06T07:22:04.296482+0000","last_fullsized":"2024-03-06T07:22:04.296482+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302512+0000","last_change":"2024-03-06T07:22:04.302512+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302512+0000","last_undegraded":"2024-03-06T07:22:04.302512+0000","last_fullsized":"2024-03-06T07:22:04.302512+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302956+0000","last_change":"2024-03-06T07:22:04.302956+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302956+0000","last_undegraded":"2024-03-06T07:22:04.302956+0000","last_fullsized":"2024-03-06T07:22:04.302956+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.305118+0000","last_change":"2024-03-06T07:22:04.305118+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.305118+0000","last_undegraded":"2024-03-06T07:22:04.305118+0000","last_fullsized":"2024-03-06T07:22:04.305118+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.300365+0000","last_change":"2024-03-06T07:22:04.300365+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.300365+0000","last_undegraded":"2024-03-06T07:22:04.300365+0000","last_fullsized":"2024-03-06T07:22:04.300365+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302506+0000","last_change":"2024-03-06T07:22:04.302506+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302506+0000","last_undegraded":"2024-03-06T07:22:04.302506+0000","last_fullsized":"2024-03-06T07:22:04.302506+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299764+0000","last_change":"2024-03-06T07:22:04.299764+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299764+0000","last_undegraded":"2024-03-06T07:22:04.299764+0000","last_fullsized":"2024-03-06T07:22:04.299764+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.305086+0000","last_change":"2024-03-06T07:22:04.305086+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.305086+0000","last_undegraded":"2024-03-06T07:22:04.305086+0000","last_fullsized":"2024-03-06T07:22:04.305086+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.300440+0000","last_change":"2024-03-06T07:22:04.300440+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.300440+0000","last_undegraded":"2024-03-06T07:22:04.300440+0000","last_fullsized":"2024-03-06T07:22:04.300440+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296462+0000","last_change":"2024-03-06T07:22:04.296462+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296462+0000","last_undegraded":"2024-03-06T07:22:04.296462+0000","last_fullsized":"2024-03-06T07:22:04.296462+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302506+0000","last_change":"2024-03-06T07:22:04.302506+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302506+0000","last_undegraded":"2024-03-06T07:22:04.302506+0000","last_fullsized":"2024-03-06T07:22:04.302506+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299827+0000","last_change":"2024-03-06T07:22:04.299827+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299827+0000","last_undegraded":"2024-03-06T07:22:04.299827+0000","last_fullsized":"2024-03-06T07:22:04.299827+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299759+0000","last_change":"2024-03-06T07:22:04.299759+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299759+0000","last_undegraded":"2024-03-06T07:22:04.299759+0000","last_fullsized":"2024-03-06T07:22:04.299759+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.297059+0000","last_change":"2024-03-06T07:22:04.297059+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.297059+0000","last_undegraded":"2024-03-06T07:22:04.297059+0000","last_fullsized":"2024-03-06T07:22:04.297059+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.303050+0000","last_change":"2024-03-06T07:22:04.303050+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.303050+0000","last_undegraded":"2024-03-06T07:22:04.303050+0000","last_fullsized":"2024-03-06T07:22:04.303050+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.297784+0000","last_change":"2024-03-06T07:22:04.297784+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.297784+0000","last_undegraded":"2024-03-06T07:22:04.297784+0000","last_fullsized":"2024-03-06T07:22:04.297784+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.300752+0000","last_change":"2024-03-06T07:22:04.300752+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.300752+0000","last_undegraded":"2024-03-06T07:22:04.300752+0000","last_fullsized":"2024-03-06T07:22:04.300752+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299956+0000","last_change":"2024-03-06T07:22:04.299956+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299956+0000","last_undegraded":"2024-03-06T07:22:04.299956+0000","last_fullsized":"2024-03-06T07:22:04.299956+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296314+0000","last_change":"2024-03-06T07:22:04.296314+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296314+0000","last_undegraded":"2024-03-06T07:22:04.296314+0000","last_fullsized":"2024-03-06T07:22:04.296314+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.297197+0000","last_change":"2024-03-06T07:22:04.297197+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.297197+0000","last_undegraded":"2024-03-06T07:22:04.297197+0000","last_fullsized":"2024-03-06T07:22:04.297197+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302534+0000","last_change":"2024-03-06T07:22:04.302534+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302534+0000","last_undegraded":"2024-03-06T07:22:04.302534+0000","last_fullsized":"2024-03-06T07:22:04.302534+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.297668+0000","last_change":"2024-03-06T07:22:04.297668+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.297668+0000","last_undegraded":"2024-03-06T07:22:04.297668+0000","last_fullsized":"2024-03-06T07:22:04.297668+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.300822+0000","last_change":"2024-03-06T07:22:04.300822+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.300822+0000","last_undegraded":"2024-03-06T07:22:04.300822+0000","last_fullsized":"2024-03-06T07:22:04.300822+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299878+0000","last_change":"2024-03-06T07:22:04.299878+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299878+0000","last_undegraded":"2024-03-06T07:22:04.299878+0000","last_fullsized":"2024-03-06T07:22:04.299878+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302664+0000","last_change":"2024-03-06T07:22:04.302664+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302664+0000","last_undegraded":"2024-03-06T07:22:04.302664+0000","last_fullsized":"2024-03-06T07:22:04.302664+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296474+0000","last_change":"2024-03-06T07:22:04.296474+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296474+0000","last_undegraded":"2024-03-06T07:22:04.296474+0000","last_fullsized":"2024-03-06T07:22:04.296474+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.296739+0000","last_change":"2024-03-06T07:22:04.296739+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.296739+0000","last_undegraded":"2024-03-06T07:22:04.296739+0000","last_fullsized":"2024-03-06T07:22:04.296739+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.305047+0000","last_change":"2024-03-06T07:22:04.305047+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.305047+0000","last_undegraded":"2024-03-06T07:22:04.305047+0000","last_fullsized":"2024-03-06T07:22:04.305047+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.299860+0000","last_change":"2024-03-06T07:22:04.299860+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.299860+0000","last_undegraded":"2024-03-06T07:22:04.299860+0000","last_fullsized":"2024-03-06T07:22:04.299860+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:05.294528+0000","last_change":"2024-03-06T07:22:05.294528+0000","last_active":"2024-03-06T07:22:05.294528+0000","last_peered":"2024-03-06T07:22:05.294528+0000","last_clean":"2024-03-06T07:22:05.294528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:05.294528+0000","last_undegraded":"2024-03-06T07:22:05.294528+0000","last_fullsized":"2024-03-06T07:22:05.294528+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:22:04.302620+0000","last_change":"2024-03-06T07:22:04.302620+0000","last_active":"2024-03-06T07:22:04.289606+0000","last_peered":"2024-03-06T07:22:04.289606+0000","last_clean":"2024-03-06T07:22:04.289606+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:04.302620+0000","last_undegraded":"2024-03-06T07:22:04.302620+0000","last_fullsized":"2024-03-06T07:22:04.302620+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":224,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-06T07:22:04.301339+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:22:04.301339+0000","last_peered":"2024-03-06T07:22:04.301339+0000","last_clean":"2024-03-06T07:22:04.301339+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:22:04.301339+0000","last_undegraded":"2024-03-06T07:22:04.301339+0000","last_fullsized":"2024-03-06T07:22:04.301339+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":634880,"data_stored":841162,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27584,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344256,"statfs":{"total":96636764160,"available":96608518144,"internally_reserved":0,"allocated":569344,"data_stored":250794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27584,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344256,"statfs":{"total":96636764160,"available":96608518144,"internally_reserved":0,"allocated":569344,"data_stored":250794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":569344,"data_stored":250794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31748,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340092,"statfs":{"total":96636764160,"available":96604254208,"internally_reserved":0,"allocated":581632,"data_stored":257003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31736,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340104,"statfs":{"total":96636764160,"available":96604266496,"internally_reserved":0,"allocated":569344,"data_stored":250794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31748,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340092,"statfs":{"total":96636764160,"available":96604254208,"internally_reserved":0,"allocated":581632,"data_stored":257003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":634880,"data_stored":841162,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:10.198 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3f is not active+clean 2024-03-06T07:22:10.198 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.198 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3e is not active+clean 2024-03-06T07:22:10.199 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.199 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3d is not active+clean 2024-03-06T07:22:10.199 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.199 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3c is not active+clean 2024-03-06T07:22:10.199 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.199 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3b is not active+clean 2024-03-06T07:22:10.199 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.200 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3a is not active+clean 2024-03-06T07:22:10.200 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.200 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.39 is not active+clean 2024-03-06T07:22:10.200 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.201 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.38 is not active+clean 2024-03-06T07:22:10.201 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.201 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.37 is not active+clean 2024-03-06T07:22:10.201 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.201 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.36 is not active+clean 2024-03-06T07:22:10.202 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.202 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.35 is not active+clean 2024-03-06T07:22:10.202 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.202 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.34 is not active+clean 2024-03-06T07:22:10.202 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.202 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.33 is not active+clean 2024-03-06T07:22:10.202 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.203 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.32 is not active+clean 2024-03-06T07:22:10.203 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.203 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.31 is not active+clean 2024-03-06T07:22:10.203 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.203 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.30 is not active+clean 2024-03-06T07:22:10.203 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.203 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2f is not active+clean 2024-03-06T07:22:10.203 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.204 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2e is not active+clean 2024-03-06T07:22:10.204 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.204 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2d is not active+clean 2024-03-06T07:22:10.204 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.204 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2c is not active+clean 2024-03-06T07:22:10.204 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.204 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2b is not active+clean 2024-03-06T07:22:10.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.205 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2a is not active+clean 2024-03-06T07:22:10.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.205 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.29 is not active+clean 2024-03-06T07:22:10.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.205 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.28 is not active+clean 2024-03-06T07:22:10.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.205 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.27 is not active+clean 2024-03-06T07:22:10.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.205 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.26 is not active+clean 2024-03-06T07:22:10.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.25 is not active+clean 2024-03-06T07:22:10.206 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.24 is not active+clean 2024-03-06T07:22:10.206 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.23 is not active+clean 2024-03-06T07:22:10.206 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.22 is not active+clean 2024-03-06T07:22:10.206 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.21 is not active+clean 2024-03-06T07:22:10.206 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.20 is not active+clean 2024-03-06T07:22:10.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.10 is not active+clean 2024-03-06T07:22:10.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1f is not active+clean 2024-03-06T07:22:10.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.301253+0000', 'last_change': '2024-03-06T07:22:04.301253+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.301253+0000', 'last_undegraded': '2024-03-06T07:22:04.301253+0000', 'last_fullsized': '2024-03-06T07:22:04.301253+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:10.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.11 is not active+clean 2024-03-06T07:22:10.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1e is not active+clean 2024-03-06T07:22:10.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.296454+0000', 'last_change': '2024-03-06T07:22:04.296454+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.296454+0000', 'last_undegraded': '2024-03-06T07:22:04.296454+0000', 'last_fullsized': '2024-03-06T07:22:04.296454+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:10.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.12 is not active+clean 2024-03-06T07:22:10.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.208 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1d is not active+clean 2024-03-06T07:22:10.208 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.296482+0000', 'last_change': '2024-03-06T07:22:04.296482+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.296482+0000', 'last_undegraded': '2024-03-06T07:22:04.296482+0000', 'last_fullsized': '2024-03-06T07:22:04.296482+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:10.208 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.13 is not active+clean 2024-03-06T07:22:10.208 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.208 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1c is not active+clean 2024-03-06T07:22:10.208 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.302512+0000', 'last_change': '2024-03-06T07:22:04.302512+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.302512+0000', 'last_undegraded': '2024-03-06T07:22:04.302512+0000', 'last_fullsized': '2024-03-06T07:22:04.302512+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:22:10.208 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.14 is not active+clean 2024-03-06T07:22:10.208 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.209 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1b is not active+clean 2024-03-06T07:22:10.209 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.302956+0000', 'last_change': '2024-03-06T07:22:04.302956+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.302956+0000', 'last_undegraded': '2024-03-06T07:22:04.302956+0000', 'last_fullsized': '2024-03-06T07:22:04.302956+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:22:10.209 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.15 is not active+clean 2024-03-06T07:22:10.209 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.209 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1a is not active+clean 2024-03-06T07:22:10.209 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.305118+0000', 'last_change': '2024-03-06T07:22:04.305118+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.305118+0000', 'last_undegraded': '2024-03-06T07:22:04.305118+0000', 'last_fullsized': '2024-03-06T07:22:04.305118+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:22:10.209 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.16 is not active+clean 2024-03-06T07:22:10.209 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.209 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.19 is not active+clean 2024-03-06T07:22:10.209 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.300365+0000', 'last_change': '2024-03-06T07:22:04.300365+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.300365+0000', 'last_undegraded': '2024-03-06T07:22:04.300365+0000', 'last_fullsized': '2024-03-06T07:22:04.300365+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:10.210 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.17 is not active+clean 2024-03-06T07:22:10.210 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.210 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.18 is not active+clean 2024-03-06T07:22:10.210 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.302506+0000', 'last_change': '2024-03-06T07:22:04.302506+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.302506+0000', 'last_undegraded': '2024-03-06T07:22:04.302506+0000', 'last_fullsized': '2024-03-06T07:22:04.302506+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:22:10.210 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.18 is not active+clean 2024-03-06T07:22:10.210 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.210 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.17 is not active+clean 2024-03-06T07:22:10.210 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.299764+0000', 'last_change': '2024-03-06T07:22:04.299764+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.299764+0000', 'last_undegraded': '2024-03-06T07:22:04.299764+0000', 'last_fullsized': '2024-03-06T07:22:04.299764+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:10.210 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.19 is not active+clean 2024-03-06T07:22:10.210 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.210 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.16 is not active+clean 2024-03-06T07:22:10.210 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.305086+0000', 'last_change': '2024-03-06T07:22:04.305086+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.305086+0000', 'last_undegraded': '2024-03-06T07:22:04.305086+0000', 'last_fullsized': '2024-03-06T07:22:04.305086+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:22:10.211 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1a is not active+clean 2024-03-06T07:22:10.211 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.211 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.15 is not active+clean 2024-03-06T07:22:10.211 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.300440+0000', 'last_change': '2024-03-06T07:22:04.300440+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.300440+0000', 'last_undegraded': '2024-03-06T07:22:04.300440+0000', 'last_fullsized': '2024-03-06T07:22:04.300440+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:10.211 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1b is not active+clean 2024-03-06T07:22:10.211 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.211 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.14 is not active+clean 2024-03-06T07:22:10.211 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.296462+0000', 'last_change': '2024-03-06T07:22:04.296462+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.296462+0000', 'last_undegraded': '2024-03-06T07:22:04.296462+0000', 'last_fullsized': '2024-03-06T07:22:04.296462+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:22:10.211 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1c is not active+clean 2024-03-06T07:22:10.211 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.211 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.13 is not active+clean 2024-03-06T07:22:10.212 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.302506+0000', 'last_change': '2024-03-06T07:22:04.302506+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.302506+0000', 'last_undegraded': '2024-03-06T07:22:04.302506+0000', 'last_fullsized': '2024-03-06T07:22:04.302506+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:22:10.212 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1d is not active+clean 2024-03-06T07:22:10.212 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.212 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.12 is not active+clean 2024-03-06T07:22:10.212 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.299827+0000', 'last_change': '2024-03-06T07:22:04.299827+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.299827+0000', 'last_undegraded': '2024-03-06T07:22:04.299827+0000', 'last_fullsized': '2024-03-06T07:22:04.299827+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:10.212 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1e is not active+clean 2024-03-06T07:22:10.212 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.212 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.11 is not active+clean 2024-03-06T07:22:10.212 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.299759+0000', 'last_change': '2024-03-06T07:22:04.299759+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.299759+0000', 'last_undegraded': '2024-03-06T07:22:04.299759+0000', 'last_fullsized': '2024-03-06T07:22:04.299759+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:10.212 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1f is not active+clean 2024-03-06T07:22:10.212 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.213 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.10 is not active+clean 2024-03-06T07:22:10.213 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.297059+0000', 'last_change': '2024-03-06T07:22:04.297059+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.297059+0000', 'last_undegraded': '2024-03-06T07:22:04.297059+0000', 'last_fullsized': '2024-03-06T07:22:04.297059+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:22:10.213 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.0 is not active+clean 2024-03-06T07:22:10.213 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.213 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.f is not active+clean 2024-03-06T07:22:10.213 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.303050+0000', 'last_change': '2024-03-06T07:22:04.303050+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.303050+0000', 'last_undegraded': '2024-03-06T07:22:04.303050+0000', 'last_fullsized': '2024-03-06T07:22:04.303050+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:22:10.213 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1 is not active+clean 2024-03-06T07:22:10.213 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.213 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.e is not active+clean 2024-03-06T07:22:10.213 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.297784+0000', 'last_change': '2024-03-06T07:22:04.297784+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.297784+0000', 'last_undegraded': '2024-03-06T07:22:04.297784+0000', 'last_fullsized': '2024-03-06T07:22:04.297784+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:22:10.213 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2 is not active+clean 2024-03-06T07:22:10.213 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.214 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.d is not active+clean 2024-03-06T07:22:10.214 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.300752+0000', 'last_change': '2024-03-06T07:22:04.300752+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.300752+0000', 'last_undegraded': '2024-03-06T07:22:04.300752+0000', 'last_fullsized': '2024-03-06T07:22:04.300752+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:10.214 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3 is not active+clean 2024-03-06T07:22:10.214 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.214 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.c is not active+clean 2024-03-06T07:22:10.214 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.299956+0000', 'last_change': '2024-03-06T07:22:04.299956+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.299956+0000', 'last_undegraded': '2024-03-06T07:22:04.299956+0000', 'last_fullsized': '2024-03-06T07:22:04.299956+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:10.214 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.4 is not active+clean 2024-03-06T07:22:10.214 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.214 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.b is not active+clean 2024-03-06T07:22:10.214 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.296314+0000', 'last_change': '2024-03-06T07:22:04.296314+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.296314+0000', 'last_undegraded': '2024-03-06T07:22:04.296314+0000', 'last_fullsized': '2024-03-06T07:22:04.296314+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:10.214 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.5 is not active+clean 2024-03-06T07:22:10.215 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.215 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.a is not active+clean 2024-03-06T07:22:10.215 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.297197+0000', 'last_change': '2024-03-06T07:22:04.297197+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.297197+0000', 'last_undegraded': '2024-03-06T07:22:04.297197+0000', 'last_fullsized': '2024-03-06T07:22:04.297197+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:22:10.215 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.6 is not active+clean 2024-03-06T07:22:10.215 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.215 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.9 is not active+clean 2024-03-06T07:22:10.215 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.302534+0000', 'last_change': '2024-03-06T07:22:04.302534+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.302534+0000', 'last_undegraded': '2024-03-06T07:22:04.302534+0000', 'last_fullsized': '2024-03-06T07:22:04.302534+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:22:10.215 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.7 is not active+clean 2024-03-06T07:22:10.215 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.215 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.8 is not active+clean 2024-03-06T07:22:10.216 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.297668+0000', 'last_change': '2024-03-06T07:22:04.297668+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.297668+0000', 'last_undegraded': '2024-03-06T07:22:04.297668+0000', 'last_fullsized': '2024-03-06T07:22:04.297668+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:22:10.216 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.8 is not active+clean 2024-03-06T07:22:10.216 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.216 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.7 is not active+clean 2024-03-06T07:22:10.216 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.300822+0000', 'last_change': '2024-03-06T07:22:04.300822+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.300822+0000', 'last_undegraded': '2024-03-06T07:22:04.300822+0000', 'last_fullsized': '2024-03-06T07:22:04.300822+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:10.216 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.a is not active+clean 2024-03-06T07:22:10.216 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.216 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.5 is not active+clean 2024-03-06T07:22:10.216 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.299878+0000', 'last_change': '2024-03-06T07:22:04.299878+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.299878+0000', 'last_undegraded': '2024-03-06T07:22:04.299878+0000', 'last_fullsized': '2024-03-06T07:22:04.299878+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:10.216 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.b is not active+clean 2024-03-06T07:22:10.216 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.217 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.4 is not active+clean 2024-03-06T07:22:10.217 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.302664+0000', 'last_change': '2024-03-06T07:22:04.302664+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.302664+0000', 'last_undegraded': '2024-03-06T07:22:04.302664+0000', 'last_fullsized': '2024-03-06T07:22:04.302664+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:22:10.217 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.c is not active+clean 2024-03-06T07:22:10.217 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.217 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3 is not active+clean 2024-03-06T07:22:10.217 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.296474+0000', 'last_change': '2024-03-06T07:22:04.296474+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.296474+0000', 'last_undegraded': '2024-03-06T07:22:04.296474+0000', 'last_fullsized': '2024-03-06T07:22:04.296474+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:22:10.217 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.d is not active+clean 2024-03-06T07:22:10.217 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.217 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2 is not active+clean 2024-03-06T07:22:10.217 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.296739+0000', 'last_change': '2024-03-06T07:22:04.296739+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.296739+0000', 'last_undegraded': '2024-03-06T07:22:04.296739+0000', 'last_fullsized': '2024-03-06T07:22:04.296739+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:22:10.217 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.e is not active+clean 2024-03-06T07:22:10.217 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.218 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1 is not active+clean 2024-03-06T07:22:10.218 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.305047+0000', 'last_change': '2024-03-06T07:22:04.305047+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.305047+0000', 'last_undegraded': '2024-03-06T07:22:04.305047+0000', 'last_fullsized': '2024-03-06T07:22:04.305047+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:22:10.218 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.f is not active+clean 2024-03-06T07:22:10.218 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.218 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.0 is not active+clean 2024-03-06T07:22:10.218 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.299860+0000', 'last_change': '2024-03-06T07:22:04.299860+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.299860+0000', 'last_undegraded': '2024-03-06T07:22:04.299860+0000', 'last_fullsized': '2024-03-06T07:22:04.299860+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:22:10.218 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.9 is not active+clean 2024-03-06T07:22:10.218 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:22:05.294528+0000', 'last_change': '2024-03-06T07:22:05.294528+0000', 'last_active': '2024-03-06T07:22:05.294528+0000', 'last_peered': '2024-03-06T07:22:05.294528+0000', 'last_clean': '2024-03-06T07:22:05.294528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:05.294528+0000', 'last_undegraded': '2024-03-06T07:22:05.294528+0000', 'last_fullsized': '2024-03-06T07:22:05.294528+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:05.294528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:10.218 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.6 is not active+clean 2024-03-06T07:22:10.218 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:04.302620+0000', 'last_change': '2024-03-06T07:22:04.302620+0000', 'last_active': '2024-03-06T07:22:04.289606+0000', 'last_peered': '2024-03-06T07:22:04.289606+0000', 'last_clean': '2024-03-06T07:22:04.289606+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:04.302620+0000', 'last_undegraded': '2024-03-06T07:22:04.302620+0000', 'last_fullsized': '2024-03-06T07:22:04.302620+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:04.289606+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:22:10.219 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:22:10.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.333+0000 7f108e9ac640 1 Processor -- start 2024-03-06T07:22:10.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.333+0000 7f108e9ac640 1 -- start start 2024-03-06T07:22:10.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.334+0000 7f108e9ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10880af6f0 0x7f10880c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.334+0000 7f108e9ac640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f10880c7930 0x7f108815fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.334+0000 7f108e9ac640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10880c7180 con 0x7f1088152820 2024-03-06T07:22:10.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.334+0000 7f108e9ac640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1088155460 con 0x7f10880c7930 2024-03-06T07:22:10.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.334+0000 7f108e9ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10880c6bc0 con 0x7f10880af6f0 2024-03-06T07:22:10.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.334+0000 7f108cf49640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f10880c7930 0x7f108815fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.334+0000 7f108cf49640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f10880c7930 0x7f108815fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59578/0 (socket says 172.21.15.1:59578) 2024-03-06T07:22:10.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.334+0000 7f108cf49640 1 -- 172.21.15.1:0/1592651183 learned_addr learned my addr 172.21.15.1:0/1592651183 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:10.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.335+0000 7f10877fe640 1 --2- 172.21.15.1:0/1592651183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10880af6f0 0x7f10880c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.335+0000 7f108cf49640 1 -- 172.21.15.1:0/1592651183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10880af6f0 msgr2=0x7f10880c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.335+0000 7f108cf49640 1 --2- 172.21.15.1:0/1592651183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10880af6f0 0x7f10880c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.335+0000 7f108cf49640 1 -- 172.21.15.1:0/1592651183 >> v1:172.21.15.98:6789/0 conn(0x7f1088152820 legacy=0x7f10880c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.335+0000 7f108cf49640 1 -- 172.21.15.1:0/1592651183 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10880c6ea0 con 0x7f10880c7930 2024-03-06T07:22:10.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.335+0000 7f10877fe640 1 --2- 172.21.15.1:0/1592651183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10880af6f0 0x7f10880c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:10.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.336+0000 7f108cf49640 1 --2- 172.21.15.1:0/1592651183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10880c7930 0x7f108815fb60 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1080009e50 tx=0x7f108002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ce542aaab6c1f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.336+0000 7f1086ffd640 1 -- 172.21.15.1:0/1592651183 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108002ca70 con 0x7f10880c7930 2024-03-06T07:22:10.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.336+0000 7f1086ffd640 1 -- 172.21.15.1:0/1592651183 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108002cc10 con 0x7f10880c7930 2024-03-06T07:22:10.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.336+0000 7f1086ffd640 1 -- 172.21.15.1:0/1592651183 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10800343f0 con 0x7f10880c7930 2024-03-06T07:22:10.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.336+0000 7f108e9ac640 1 -- 172.21.15.1:0/1592651183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10880c7930 msgr2=0x7f108815fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.336+0000 7f108e9ac640 1 --2- 172.21.15.1:0/1592651183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10880c7930 0x7f108815fb60 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1080009e50 tx=0x7f108002bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.336+0000 7f108e9ac640 1 -- 172.21.15.1:0/1592651183 shutdown_connections 2024-03-06T07:22:10.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.336+0000 7f108e9ac640 1 --2- 172.21.15.1:0/1592651183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10880c7930 0x7f108815fb60 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.336+0000 7f108e9ac640 1 --2- 172.21.15.1:0/1592651183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10880af6f0 0x7f10880c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.336+0000 7f108e9ac640 1 -- 172.21.15.1:0/1592651183 >> 172.21.15.1:0/1592651183 conn(0x7f1088072720 msgr2=0x7f1088072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.336+0000 7f108e9ac640 1 -- 172.21.15.1:0/1592651183 shutdown_connections 2024-03-06T07:22:10.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.336+0000 7f108e9ac640 1 -- 172.21.15.1:0/1592651183 wait complete. 2024-03-06T07:22:10.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.338+0000 7f108e9ac640 1 Processor -- start 2024-03-06T07:22:10.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.338+0000 7f108e9ac640 1 -- start start 2024-03-06T07:22:10.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.339+0000 7f108e9ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10880af6f0 0x7f1088157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.339+0000 7f108e9ac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10880c7930 0x7f10881ede60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.339+0000 7f108e9ac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1088152820 0x7f108814b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.339+0000 7f108e9ac640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1088152c00 con 0x7f10880c7930 2024-03-06T07:22:10.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.339+0000 7f108e9ac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10880c0840 con 0x7f1088152820 2024-03-06T07:22:10.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.339+0000 7f108e9ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1088152d80 con 0x7f10880af6f0 2024-03-06T07:22:10.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.339+0000 7f10877fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10880c7930 0x7f10881ede60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.339+0000 7f10877fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10880c7930 0x7f10881ede60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59592/0 (socket says 172.21.15.1:59592) 2024-03-06T07:22:10.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.339+0000 7f10877fe640 1 -- 172.21.15.1:0/1495811085 learned_addr learned my addr 172.21.15.1:0/1495811085 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:10.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.339+0000 7f108cf49640 1 --2- 172.21.15.1:0/1495811085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1088152820 0x7f108814b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.339+0000 7f10877fe640 1 -- 172.21.15.1:0/1495811085 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10880af6f0 msgr2=0x7f1088157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.339+0000 7f1087fff640 1 --2- 172.21.15.1:0/1495811085 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10880af6f0 0x7f1088157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.339+0000 7f10877fe640 1 --2- 172.21.15.1:0/1495811085 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10880af6f0 0x7f1088157e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.339+0000 7f10877fe640 1 -- 172.21.15.1:0/1495811085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1088152820 msgr2=0x7f108814b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.339+0000 7f10877fe640 1 --2- 172.21.15.1:0/1495811085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1088152820 0x7f108814b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.339+0000 7f10877fe640 1 -- 172.21.15.1:0/1495811085 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10880ca370 con 0x7f10880c7930 2024-03-06T07:22:10.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.339+0000 7f108cf49640 1 --2- 172.21.15.1:0/1495811085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1088152820 0x7f108814b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:10.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.340+0000 7f10877fe640 1 --2- 172.21.15.1:0/1495811085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10880c7930 0x7f10881ede60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f107c00c7f0 tx=0x7f107c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.340+0000 7f1084ff9640 1 -- 172.21.15.1:0/1495811085 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107c015070 con 0x7f10880c7930 2024-03-06T07:22:10.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.340+0000 7f1084ff9640 1 -- 172.21.15.1:0/1495811085 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f107c007a30 con 0x7f10880c7930 2024-03-06T07:22:10.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.340+0000 7f108e9ac640 1 -- 172.21.15.1:0/1495811085 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10880ca650 con 0x7f10880c7930 2024-03-06T07:22:10.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.340+0000 7f1084ff9640 1 -- 172.21.15.1:0/1495811085 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107c007e00 con 0x7f10880c7930 2024-03-06T07:22:10.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.340+0000 7f108e9ac640 1 -- 172.21.15.1:0/1495811085 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10880bff60 con 0x7f10880c7930 2024-03-06T07:22:10.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.341+0000 7f108e9ac640 1 -- 172.21.15.1:0/1495811085 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10880686e0 con 0x7f10880c7930 2024-03-06T07:22:10.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.342+0000 7f1084ff9640 1 -- 172.21.15.1:0/1495811085 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f107c018070 con 0x7f10880c7930 2024-03-06T07:22:10.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.342+0000 7f1084ff9640 1 --2- 172.21.15.1:0/1495811085 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f107406b4d0 0x7f107406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.342+0000 7f1084ff9640 1 -- 172.21.15.1:0/1495811085 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f107c08a380 con 0x7f10880c7930 2024-03-06T07:22:10.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.342+0000 7f1087fff640 1 --2- 172.21.15.1:0/1495811085 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f107406b4d0 0x7f107406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.343+0000 7f1087fff640 1 --2- 172.21.15.1:0/1495811085 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f107406b4d0 0x7f107406d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1078006fd0 tx=0x7f1078008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.350+0000 7f1084ff9640 1 -- 172.21.15.1:0/1495811085 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f107c059080 con 0x7f10880c7930 2024-03-06T07:22:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.615+0000 7f108e9ac640 1 -- 172.21.15.1:0/1495811085 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f10880c5b40 con 0x7f10880c7930 2024-03-06T07:22:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.615+0000 7f1084ff9640 1 -- 172.21.15.1:0/1495811085 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1270 (secure 0 0 0) 0x7f107c058980 con 0x7f10880c7930 2024-03-06T07:22:10.618 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:10.618 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":126,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":52,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709614,"num_in_osds":8,"osd_in_since":1709709606,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":243150848,"bytes_avail":772850962432,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":28,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"c","status":"up:active","gid":5778}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-03-06T07:21:16.701529+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:22:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.617+0000 7f108e9ac640 1 -- 172.21.15.1:0/1495811085 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f107406b4d0 msgr2=0x7f107406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.617+0000 7f108e9ac640 1 --2- 172.21.15.1:0/1495811085 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f107406b4d0 0x7f107406d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1078006fd0 tx=0x7f1078008040 comp rx=0 tx=0).stop 2024-03-06T07:22:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.617+0000 7f108e9ac640 1 -- 172.21.15.1:0/1495811085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10880c7930 msgr2=0x7f10881ede60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.617+0000 7f108e9ac640 1 --2- 172.21.15.1:0/1495811085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10880c7930 0x7f10881ede60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f107c00c7f0 tx=0x7f107c00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.617+0000 7f108e9ac640 1 -- 172.21.15.1:0/1495811085 shutdown_connections 2024-03-06T07:22:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.617+0000 7f108e9ac640 1 --2- 172.21.15.1:0/1495811085 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f107406b4d0 0x7f107406d990 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.617+0000 7f108e9ac640 1 --2- 172.21.15.1:0/1495811085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1088152820 0x7f108814b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.617+0000 7f108e9ac640 1 --2- 172.21.15.1:0/1495811085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10880c7930 0x7f10881ede60 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.617+0000 7f108e9ac640 1 --2- 172.21.15.1:0/1495811085 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10880af6f0 0x7f1088157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.617+0000 7f108e9ac640 1 -- 172.21.15.1:0/1495811085 >> 172.21.15.1:0/1495811085 conn(0x7f1088072720 msgr2=0x7f108814ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.617+0000 7f108e9ac640 1 -- 172.21.15.1:0/1495811085 shutdown_connections 2024-03-06T07:22:10.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.617+0000 7f108e9ac640 1 -- 172.21.15.1:0/1495811085 wait complete. 2024-03-06T07:22:10.630 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:22:10.630 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:10.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.729+0000 7f67b641c640 1 Processor -- start 2024-03-06T07:22:10.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.730+0000 7f67b641c640 1 -- start start 2024-03-06T07:22:10.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.730+0000 7f67b641c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f67b00c5960 0x7f67b00c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.730+0000 7f67b641c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67b0153ab0 0x7f67b0070b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.730+0000 7f67b641c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b00c67c0 con 0x7f67b00c4ec0 2024-03-06T07:22:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.730+0000 7f67b641c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67b00c6a60 con 0x7f67b00c5960 2024-03-06T07:22:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.730+0000 7f67b641c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b00c6d40 con 0x7f67b0153ab0 2024-03-06T07:22:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.730+0000 7f67affff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f67b00c5960 0x7f67b00c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.730+0000 7f67affff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f67b00c5960 0x7f67b00c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59604/0 (socket says 172.21.15.1:59604) 2024-03-06T07:22:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.730+0000 7f67affff640 1 -- 172.21.15.1:0/4077399563 learned_addr learned my addr 172.21.15.1:0/4077399563 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.731+0000 7f67af7fe640 1 --2- 172.21.15.1:0/4077399563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67b0153ab0 0x7f67b0070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.731+0000 7f67affff640 1 -- 172.21.15.1:0/4077399563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67b0153ab0 msgr2=0x7f67b0070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.731+0000 7f67affff640 1 --2- 172.21.15.1:0/4077399563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67b0153ab0 0x7f67b0070b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.731+0000 7f67affff640 1 -- 172.21.15.1:0/4077399563 >> v1:172.21.15.98:6789/0 conn(0x7f67b00c4ec0 legacy=0x7f67b0156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.731+0000 7f67affff640 1 -- 172.21.15.1:0/4077399563 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b00c7020 con 0x7f67b00c5960 2024-03-06T07:22:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.731+0000 7f67affff640 1 --2- 172.21.15.1:0/4077399563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67b00c5960 0x7f67b00c4980 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f67a0009870 tx=0x7f67a002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=329ce7cc6bf95dd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.732+0000 7f67aeffd640 1 -- 172.21.15.1:0/4077399563 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a0035020 con 0x7f67b00c5960 2024-03-06T07:22:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.732+0000 7f67aeffd640 1 -- 172.21.15.1:0/4077399563 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a0031640 con 0x7f67b00c5960 2024-03-06T07:22:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.732+0000 7f67aeffd640 1 -- 172.21.15.1:0/4077399563 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a002d040 con 0x7f67b00c5960 2024-03-06T07:22:10.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.732+0000 7f67b641c640 1 -- 172.21.15.1:0/4077399563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67b00c5960 msgr2=0x7f67b00c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.732+0000 7f67b641c640 1 --2- 172.21.15.1:0/4077399563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67b00c5960 0x7f67b00c4980 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f67a0009870 tx=0x7f67a002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.732+0000 7f67b641c640 1 -- 172.21.15.1:0/4077399563 shutdown_connections 2024-03-06T07:22:10.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.732+0000 7f67b641c640 1 --2- 172.21.15.1:0/4077399563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67b0153ab0 0x7f67b0070b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.732+0000 7f67b641c640 1 --2- 172.21.15.1:0/4077399563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67b00c5960 0x7f67b00c4980 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.732+0000 7f67b641c640 1 -- 172.21.15.1:0/4077399563 >> 172.21.15.1:0/4077399563 conn(0x7f67b00aa550 msgr2=0x7f67b00aa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.732+0000 7f67b641c640 1 -- 172.21.15.1:0/4077399563 shutdown_connections 2024-03-06T07:22:10.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.732+0000 7f67b641c640 1 -- 172.21.15.1:0/4077399563 wait complete. 2024-03-06T07:22:10.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.733+0000 7f67b641c640 1 Processor -- start 2024-03-06T07:22:10.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.734+0000 7f67b641c640 1 -- start start 2024-03-06T07:22:10.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.734+0000 7f67b641c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67b00c4ec0 0x7f67b01dee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.734+0000 7f67b641c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67b00c5960 0x7f67b00be660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.734+0000 7f67b641c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67b0153ab0 0x7f67b01dc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.734+0000 7f67b641c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b006f550 con 0x7f67b00c4ec0 2024-03-06T07:22:10.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.734+0000 7f67b641c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b006f3d0 con 0x7f67b00c5960 2024-03-06T07:22:10.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.734+0000 7f67b641c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b00c5d40 con 0x7f67b0153ab0 2024-03-06T07:22:10.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.734+0000 7f67affff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67b00c4ec0 0x7f67b01dee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.734+0000 7f67affff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67b00c4ec0 0x7f67b01dee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59614/0 (socket says 172.21.15.1:59614) 2024-03-06T07:22:10.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.734+0000 7f67affff640 1 -- 172.21.15.1:0/444194417 learned_addr learned my addr 172.21.15.1:0/444194417 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:10.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.735+0000 7f67af7fe640 1 --2- 172.21.15.1:0/444194417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67b00c5960 0x7f67b00be660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.735+0000 7f67b49b9640 1 --2- 172.21.15.1:0/444194417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67b0153ab0 0x7f67b01dc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.735+0000 7f67affff640 1 -- 172.21.15.1:0/444194417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67b0153ab0 msgr2=0x7f67b01dc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.735+0000 7f67affff640 1 --2- 172.21.15.1:0/444194417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67b0153ab0 0x7f67b01dc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.735+0000 7f67affff640 1 -- 172.21.15.1:0/444194417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67b00c5960 msgr2=0x7f67b00be660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.735+0000 7f67affff640 1 --2- 172.21.15.1:0/444194417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67b00c5960 0x7f67b00be660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.735+0000 7f67affff640 1 -- 172.21.15.1:0/444194417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b00beba0 con 0x7f67b00c4ec0 2024-03-06T07:22:10.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.735+0000 7f67af7fe640 1 --2- 172.21.15.1:0/444194417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67b00c5960 0x7f67b00be660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:10.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.735+0000 7f67b49b9640 1 --2- 172.21.15.1:0/444194417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67b0153ab0 0x7f67b01dc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:10.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.735+0000 7f67affff640 1 --2- 172.21.15.1:0/444194417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67b00c4ec0 0x7f67b01dee00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f67a002b930 tx=0x7f67a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.735+0000 7f67acff9640 1 -- 172.21.15.1:0/444194417 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a0040070 con 0x7f67b00c4ec0 2024-03-06T07:22:10.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.735+0000 7f67acff9640 1 -- 172.21.15.1:0/444194417 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a0039c10 con 0x7f67b00c4ec0 2024-03-06T07:22:10.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.735+0000 7f67b641c640 1 -- 172.21.15.1:0/444194417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b01ecd80 con 0x7f67b00c4ec0 2024-03-06T07:22:10.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.735+0000 7f67acff9640 1 -- 172.21.15.1:0/444194417 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a0035040 con 0x7f67b00c4ec0 2024-03-06T07:22:10.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.735+0000 7f67b641c640 1 -- 172.21.15.1:0/444194417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b01df860 con 0x7f67b00c4ec0 2024-03-06T07:22:10.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.736+0000 7f67b641c640 1 -- 172.21.15.1:0/444194417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67b0067fd0 con 0x7f67b00c4ec0 2024-03-06T07:22:10.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.736+0000 7f67acff9640 1 -- 172.21.15.1:0/444194417 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67a004b060 con 0x7f67b00c4ec0 2024-03-06T07:22:10.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.737+0000 7f67acff9640 1 --2- 172.21.15.1:0/444194417 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f679006b4d0 0x7f679006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.737+0000 7f67acff9640 1 -- 172.21.15.1:0/444194417 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f67a00abe70 con 0x7f67b00c4ec0 2024-03-06T07:22:10.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.737+0000 7f67af7fe640 1 --2- 172.21.15.1:0/444194417 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f679006b4d0 0x7f679006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.738+0000 7f67af7fe640 1 --2- 172.21.15.1:0/444194417 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f679006b4d0 0x7f679006d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f67b01df340 tx=0x7f6798008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.741+0000 7f67acff9640 1 -- 172.21.15.1:0/444194417 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67a002f300 con 0x7f67b00c4ec0 2024-03-06T07:22:10.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.919+0000 7f67b641c640 1 -- 172.21.15.1:0/444194417 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f67b00c7a70 con 0x7f679006b4d0 2024-03-06T07:22:10.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.926+0000 7f67acff9640 1 -- 172.21.15.1:0/444194417 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252030 (secure 0 0 0) 0x7f67b00c7a70 con 0x7f679006b4d0 2024-03-06T07:22:10.930 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:22:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.929+0000 7f67b641c640 1 -- 172.21.15.1:0/444194417 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f679006b4d0 msgr2=0x7f679006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.929+0000 7f67b641c640 1 --2- 172.21.15.1:0/444194417 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f679006b4d0 0x7f679006d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f67b01df340 tx=0x7f6798008040 comp rx=0 tx=0).stop 2024-03-06T07:22:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.929+0000 7f67b641c640 1 -- 172.21.15.1:0/444194417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67b00c4ec0 msgr2=0x7f67b01dee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.929+0000 7f67b641c640 1 --2- 172.21.15.1:0/444194417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67b00c4ec0 0x7f67b01dee00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f67a002b930 tx=0x7f67a00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.929+0000 7f67b641c640 1 -- 172.21.15.1:0/444194417 shutdown_connections 2024-03-06T07:22:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.929+0000 7f67b641c640 1 --2- 172.21.15.1:0/444194417 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f679006b4d0 0x7f679006d990 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.929+0000 7f67b641c640 1 --2- 172.21.15.1:0/444194417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67b0153ab0 0x7f67b01dc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.929+0000 7f67b641c640 1 --2- 172.21.15.1:0/444194417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67b00c5960 0x7f67b00be660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.929+0000 7f67b641c640 1 --2- 172.21.15.1:0/444194417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67b00c4ec0 0x7f67b01dee00 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.929+0000 7f67b641c640 1 -- 172.21.15.1:0/444194417 >> 172.21.15.1:0/444194417 conn(0x7f67b00aa550 msgr2=0x7f67b0151920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.929+0000 7f67b641c640 1 -- 172.21.15.1:0/444194417 shutdown_connections 2024-03-06T07:22:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:10.929+0000 7f67b641c640 1 -- 172.21.15.1:0/444194417 wait complete. 2024-03-06T07:22:10.942 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-03-06T07:22:10.714297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":186,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237832,"kb_used_data":5024,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736888,"statfs":{"total":773094113280,"available":772850573312,"internally_reserved":0,"allocated":5144576,"data_stored":3405788,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001204"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317049+0000","last_change":"2024-03-06T07:22:06.323021+0000","last_active":"2024-03-06T07:22:07.317049+0000","last_peered":"2024-03-06T07:22:07.317049+0000","last_clean":"2024-03-06T07:22:07.317049+0000","last_became_active":"2024-03-06T07:22:06.319646+0000","last_became_peered":"2024-03-06T07:22:06.319646+0000","last_unstale":"2024-03-06T07:22:07.317049+0000","last_undegraded":"2024-03-06T07:22:07.317049+0000","last_fullsized":"2024-03-06T07:22:07.317049+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:51.508838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316898+0000","last_change":"2024-03-06T07:22:06.325529+0000","last_active":"2024-03-06T07:22:07.316898+0000","last_peered":"2024-03-06T07:22:07.316898+0000","last_clean":"2024-03-06T07:22:07.316898+0000","last_became_active":"2024-03-06T07:22:06.324758+0000","last_became_peered":"2024-03-06T07:22:06.324758+0000","last_unstale":"2024-03-06T07:22:07.316898+0000","last_undegraded":"2024-03-06T07:22:07.316898+0000","last_fullsized":"2024-03-06T07:22:07.316898+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:02.558815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315668+0000","last_change":"2024-03-06T07:22:06.321470+0000","last_active":"2024-03-06T07:22:07.315668+0000","last_peered":"2024-03-06T07:22:07.315668+0000","last_clean":"2024-03-06T07:22:07.315668+0000","last_became_active":"2024-03-06T07:22:06.318716+0000","last_became_peered":"2024-03-06T07:22:06.318716+0000","last_unstale":"2024-03-06T07:22:07.315668+0000","last_undegraded":"2024-03-06T07:22:07.315668+0000","last_fullsized":"2024-03-06T07:22:07.315668+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:16.706902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316698+0000","last_change":"2024-03-06T07:22:06.321775+0000","last_active":"2024-03-06T07:22:07.316698+0000","last_peered":"2024-03-06T07:22:07.316698+0000","last_clean":"2024-03-06T07:22:07.316698+0000","last_became_active":"2024-03-06T07:22:06.321437+0000","last_became_peered":"2024-03-06T07:22:06.321437+0000","last_unstale":"2024-03-06T07:22:07.316698+0000","last_undegraded":"2024-03-06T07:22:07.316698+0000","last_fullsized":"2024-03-06T07:22:07.316698+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:53.049705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517963+0000","last_change":"2024-03-06T07:22:06.317556+0000","last_active":"2024-03-06T07:22:07.517963+0000","last_peered":"2024-03-06T07:22:07.517963+0000","last_clean":"2024-03-06T07:22:07.517963+0000","last_became_active":"2024-03-06T07:22:06.317298+0000","last_became_peered":"2024-03-06T07:22:06.317298+0000","last_unstale":"2024-03-06T07:22:07.517963+0000","last_undegraded":"2024-03-06T07:22:07.517963+0000","last_fullsized":"2024-03-06T07:22:07.517963+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:45.397748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316863+0000","last_change":"2024-03-06T07:22:06.323812+0000","last_active":"2024-03-06T07:22:07.316863+0000","last_peered":"2024-03-06T07:22:07.316863+0000","last_clean":"2024-03-06T07:22:07.316863+0000","last_became_active":"2024-03-06T07:22:06.323558+0000","last_became_peered":"2024-03-06T07:22:06.323558+0000","last_unstale":"2024-03-06T07:22:07.316863+0000","last_undegraded":"2024-03-06T07:22:07.316863+0000","last_fullsized":"2024-03-06T07:22:07.316863+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:37.954594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315285+0000","last_change":"2024-03-06T07:22:06.322447+0000","last_active":"2024-03-06T07:22:07.315285+0000","last_peered":"2024-03-06T07:22:07.315285+0000","last_clean":"2024-03-06T07:22:07.315285+0000","last_became_active":"2024-03-06T07:22:06.318369+0000","last_became_peered":"2024-03-06T07:22:06.318369+0000","last_unstale":"2024-03-06T07:22:07.315285+0000","last_undegraded":"2024-03-06T07:22:07.315285+0000","last_fullsized":"2024-03-06T07:22:07.315285+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:50.150401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516696+0000","last_change":"2024-03-06T07:22:06.328376+0000","last_active":"2024-03-06T07:22:07.516696+0000","last_peered":"2024-03-06T07:22:07.516696+0000","last_clean":"2024-03-06T07:22:07.516696+0000","last_became_active":"2024-03-06T07:22:06.327042+0000","last_became_peered":"2024-03-06T07:22:06.327042+0000","last_unstale":"2024-03-06T07:22:07.516696+0000","last_undegraded":"2024-03-06T07:22:07.516696+0000","last_fullsized":"2024-03-06T07:22:07.516696+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:34.670786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517253+0000","last_change":"2024-03-06T07:22:06.335876+0000","last_active":"2024-03-06T07:22:07.517253+0000","last_peered":"2024-03-06T07:22:07.517253+0000","last_clean":"2024-03-06T07:22:07.517253+0000","last_became_active":"2024-03-06T07:22:06.335575+0000","last_became_peered":"2024-03-06T07:22:06.335575+0000","last_unstale":"2024-03-06T07:22:07.517253+0000","last_undegraded":"2024-03-06T07:22:07.517253+0000","last_fullsized":"2024-03-06T07:22:07.517253+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:19.402946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317040+0000","last_change":"2024-03-06T07:22:06.320896+0000","last_active":"2024-03-06T07:22:07.317040+0000","last_peered":"2024-03-06T07:22:07.317040+0000","last_clean":"2024-03-06T07:22:07.317040+0000","last_became_active":"2024-03-06T07:22:06.320643+0000","last_became_peered":"2024-03-06T07:22:06.320643+0000","last_unstale":"2024-03-06T07:22:07.317040+0000","last_undegraded":"2024-03-06T07:22:07.317040+0000","last_fullsized":"2024-03-06T07:22:07.317040+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:09.367201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516602+0000","last_change":"2024-03-06T07:22:06.322171+0000","last_active":"2024-03-06T07:22:07.516602+0000","last_peered":"2024-03-06T07:22:07.516602+0000","last_clean":"2024-03-06T07:22:07.516602+0000","last_became_active":"2024-03-06T07:22:06.321891+0000","last_became_peered":"2024-03-06T07:22:06.321891+0000","last_unstale":"2024-03-06T07:22:07.516602+0000","last_undegraded":"2024-03-06T07:22:07.516602+0000","last_fullsized":"2024-03-06T07:22:07.516602+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:02.261930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517100+0000","last_change":"2024-03-06T07:22:06.321590+0000","last_active":"2024-03-06T07:22:07.517100+0000","last_peered":"2024-03-06T07:22:07.517100+0000","last_clean":"2024-03-06T07:22:07.517100+0000","last_became_active":"2024-03-06T07:22:06.321334+0000","last_became_peered":"2024-03-06T07:22:06.321334+0000","last_unstale":"2024-03-06T07:22:07.517100+0000","last_undegraded":"2024-03-06T07:22:07.517100+0000","last_fullsized":"2024-03-06T07:22:07.517100+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:49.931180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316738+0000","last_change":"2024-03-06T07:22:06.328682+0000","last_active":"2024-03-06T07:22:07.316738+0000","last_peered":"2024-03-06T07:22:07.316738+0000","last_clean":"2024-03-06T07:22:07.316738+0000","last_became_active":"2024-03-06T07:22:06.328418+0000","last_became_peered":"2024-03-06T07:22:06.328418+0000","last_unstale":"2024-03-06T07:22:07.316738+0000","last_undegraded":"2024-03-06T07:22:07.316738+0000","last_fullsized":"2024-03-06T07:22:07.316738+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:52.638347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315858+0000","last_change":"2024-03-06T07:22:06.330122+0000","last_active":"2024-03-06T07:22:07.315858+0000","last_peered":"2024-03-06T07:22:07.315858+0000","last_clean":"2024-03-06T07:22:07.315858+0000","last_became_active":"2024-03-06T07:22:06.329780+0000","last_became_peered":"2024-03-06T07:22:06.329780+0000","last_unstale":"2024-03-06T07:22:07.315858+0000","last_undegraded":"2024-03-06T07:22:07.315858+0000","last_fullsized":"2024-03-06T07:22:07.315858+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:38.898151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317089+0000","last_change":"2024-03-06T07:22:06.321523+0000","last_active":"2024-03-06T07:22:07.317089+0000","last_peered":"2024-03-06T07:22:07.317089+0000","last_clean":"2024-03-06T07:22:07.317089+0000","last_became_active":"2024-03-06T07:22:06.321224+0000","last_became_peered":"2024-03-06T07:22:06.321224+0000","last_unstale":"2024-03-06T07:22:07.317089+0000","last_undegraded":"2024-03-06T07:22:07.317089+0000","last_fullsized":"2024-03-06T07:22:07.317089+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:05.731939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316709+0000","last_change":"2024-03-06T07:22:06.314914+0000","last_active":"2024-03-06T07:22:07.316709+0000","last_peered":"2024-03-06T07:22:07.316709+0000","last_clean":"2024-03-06T07:22:07.316709+0000","last_became_active":"2024-03-06T07:22:06.314654+0000","last_became_peered":"2024-03-06T07:22:06.314654+0000","last_unstale":"2024-03-06T07:22:07.316709+0000","last_undegraded":"2024-03-06T07:22:07.316709+0000","last_fullsized":"2024-03-06T07:22:07.316709+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:18.848063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316467+0000","last_change":"2024-03-06T07:22:06.323974+0000","last_active":"2024-03-06T07:22:07.316467+0000","last_peered":"2024-03-06T07:22:07.316467+0000","last_clean":"2024-03-06T07:22:07.316467+0000","last_became_active":"2024-03-06T07:22:06.323718+0000","last_became_peered":"2024-03-06T07:22:06.323718+0000","last_unstale":"2024-03-06T07:22:07.316467+0000","last_undegraded":"2024-03-06T07:22:07.316467+0000","last_fullsized":"2024-03-06T07:22:07.316467+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:46.981658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316834+0000","last_change":"2024-03-06T07:22:06.325388+0000","last_active":"2024-03-06T07:22:07.316834+0000","last_peered":"2024-03-06T07:22:07.316834+0000","last_clean":"2024-03-06T07:22:07.316834+0000","last_became_active":"2024-03-06T07:22:06.325109+0000","last_became_peered":"2024-03-06T07:22:06.325109+0000","last_unstale":"2024-03-06T07:22:07.316834+0000","last_undegraded":"2024-03-06T07:22:07.316834+0000","last_fullsized":"2024-03-06T07:22:07.316834+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:51.508838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315801+0000","last_change":"2024-03-06T07:22:06.329771+0000","last_active":"2024-03-06T07:22:07.315801+0000","last_peered":"2024-03-06T07:22:07.315801+0000","last_clean":"2024-03-06T07:22:07.315801+0000","last_became_active":"2024-03-06T07:22:06.318483+0000","last_became_peered":"2024-03-06T07:22:06.318483+0000","last_unstale":"2024-03-06T07:22:07.315801+0000","last_undegraded":"2024-03-06T07:22:07.315801+0000","last_fullsized":"2024-03-06T07:22:07.315801+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:17.568813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516970+0000","last_change":"2024-03-06T07:22:06.317746+0000","last_active":"2024-03-06T07:22:07.516970+0000","last_peered":"2024-03-06T07:22:07.516970+0000","last_clean":"2024-03-06T07:22:07.516970+0000","last_became_active":"2024-03-06T07:22:06.317465+0000","last_became_peered":"2024-03-06T07:22:06.317465+0000","last_unstale":"2024-03-06T07:22:07.516970+0000","last_undegraded":"2024-03-06T07:22:07.516970+0000","last_fullsized":"2024-03-06T07:22:07.516970+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:07.308296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516958+0000","last_change":"2024-03-06T07:22:06.322670+0000","last_active":"2024-03-06T07:22:07.516958+0000","last_peered":"2024-03-06T07:22:07.516958+0000","last_clean":"2024-03-06T07:22:07.516958+0000","last_became_active":"2024-03-06T07:22:06.322382+0000","last_became_peered":"2024-03-06T07:22:06.322382+0000","last_unstale":"2024-03-06T07:22:07.516958+0000","last_undegraded":"2024-03-06T07:22:07.516958+0000","last_fullsized":"2024-03-06T07:22:07.516958+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:02.940410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516289+0000","last_change":"2024-03-06T07:22:06.331711+0000","last_active":"2024-03-06T07:22:07.516289+0000","last_peered":"2024-03-06T07:22:07.516289+0000","last_clean":"2024-03-06T07:22:07.516289+0000","last_became_active":"2024-03-06T07:22:06.326618+0000","last_became_peered":"2024-03-06T07:22:06.326618+0000","last_unstale":"2024-03-06T07:22:07.516289+0000","last_undegraded":"2024-03-06T07:22:07.516289+0000","last_fullsized":"2024-03-06T07:22:07.516289+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:15.334381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516859+0000","last_change":"2024-03-06T07:22:06.318320+0000","last_active":"2024-03-06T07:22:07.516859+0000","last_peered":"2024-03-06T07:22:07.516859+0000","last_clean":"2024-03-06T07:22:07.516859+0000","last_became_active":"2024-03-06T07:22:06.318065+0000","last_became_peered":"2024-03-06T07:22:06.318065+0000","last_unstale":"2024-03-06T07:22:07.516859+0000","last_undegraded":"2024-03-06T07:22:07.516859+0000","last_fullsized":"2024-03-06T07:22:07.516859+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:00.505615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315007+0000","last_change":"2024-03-06T07:22:06.318170+0000","last_active":"2024-03-06T07:22:07.315007+0000","last_peered":"2024-03-06T07:22:07.315007+0000","last_clean":"2024-03-06T07:22:07.315007+0000","last_became_active":"2024-03-06T07:22:06.317903+0000","last_became_peered":"2024-03-06T07:22:06.317903+0000","last_unstale":"2024-03-06T07:22:07.315007+0000","last_undegraded":"2024-03-06T07:22:07.315007+0000","last_fullsized":"2024-03-06T07:22:07.315007+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:25:11.151255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316491+0000","last_change":"2024-03-06T07:22:06.333955+0000","last_active":"2024-03-06T07:22:07.316491+0000","last_peered":"2024-03-06T07:22:07.316491+0000","last_clean":"2024-03-06T07:22:07.316491+0000","last_became_active":"2024-03-06T07:22:06.333703+0000","last_became_peered":"2024-03-06T07:22:06.333703+0000","last_unstale":"2024-03-06T07:22:07.316491+0000","last_undegraded":"2024-03-06T07:22:07.316491+0000","last_fullsized":"2024-03-06T07:22:07.316491+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:34.670786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517246+0000","last_change":"2024-03-06T07:22:06.320349+0000","last_active":"2024-03-06T07:22:07.517246+0000","last_peered":"2024-03-06T07:22:07.517246+0000","last_clean":"2024-03-06T07:22:07.517246+0000","last_became_active":"2024-03-06T07:22:06.320094+0000","last_became_peered":"2024-03-06T07:22:06.320094+0000","last_unstale":"2024-03-06T07:22:07.517246+0000","last_undegraded":"2024-03-06T07:22:07.517246+0000","last_fullsized":"2024-03-06T07:22:07.517246+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:48.224301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317246+0000","last_change":"2024-03-06T07:22:06.319987+0000","last_active":"2024-03-06T07:22:07.317246+0000","last_peered":"2024-03-06T07:22:07.317246+0000","last_clean":"2024-03-06T07:22:07.317246+0000","last_became_active":"2024-03-06T07:22:06.319703+0000","last_became_peered":"2024-03-06T07:22:06.319703+0000","last_unstale":"2024-03-06T07:22:07.317246+0000","last_undegraded":"2024-03-06T07:22:07.317246+0000","last_fullsized":"2024-03-06T07:22:07.317246+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:02.558815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517174+0000","last_change":"2024-03-06T07:22:06.318435+0000","last_active":"2024-03-06T07:22:07.517174+0000","last_peered":"2024-03-06T07:22:07.517174+0000","last_clean":"2024-03-06T07:22:07.517174+0000","last_became_active":"2024-03-06T07:22:06.318147+0000","last_became_peered":"2024-03-06T07:22:06.318147+0000","last_unstale":"2024-03-06T07:22:07.517174+0000","last_undegraded":"2024-03-06T07:22:07.517174+0000","last_fullsized":"2024-03-06T07:22:07.517174+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:20.858614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315255+0000","last_change":"2024-03-06T07:22:06.335154+0000","last_active":"2024-03-06T07:22:07.315255+0000","last_peered":"2024-03-06T07:22:07.315255+0000","last_clean":"2024-03-06T07:22:07.315255+0000","last_became_active":"2024-03-06T07:22:06.334855+0000","last_became_peered":"2024-03-06T07:22:06.334855+0000","last_unstale":"2024-03-06T07:22:07.315255+0000","last_undegraded":"2024-03-06T07:22:07.315255+0000","last_fullsized":"2024-03-06T07:22:07.315255+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:54.652557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316793+0000","last_change":"2024-03-06T07:22:06.330141+0000","last_active":"2024-03-06T07:22:07.316793+0000","last_peered":"2024-03-06T07:22:07.316793+0000","last_clean":"2024-03-06T07:22:07.316793+0000","last_became_active":"2024-03-06T07:22:06.329487+0000","last_became_peered":"2024-03-06T07:22:06.329487+0000","last_unstale":"2024-03-06T07:22:07.316793+0000","last_undegraded":"2024-03-06T07:22:07.316793+0000","last_fullsized":"2024-03-06T07:22:07.316793+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:09.367201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315416+0000","last_change":"2024-03-06T07:22:06.322218+0000","last_active":"2024-03-06T07:22:07.315416+0000","last_peered":"2024-03-06T07:22:07.315416+0000","last_clean":"2024-03-06T07:22:07.315416+0000","last_became_active":"2024-03-06T07:22:06.318026+0000","last_became_peered":"2024-03-06T07:22:06.318026+0000","last_unstale":"2024-03-06T07:22:07.315416+0000","last_undegraded":"2024-03-06T07:22:07.315416+0000","last_fullsized":"2024-03-06T07:22:07.315416+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:16.809640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516831+0000","last_change":"2024-03-06T07:22:06.322067+0000","last_active":"2024-03-06T07:22:07.516831+0000","last_peered":"2024-03-06T07:22:07.516831+0000","last_clean":"2024-03-06T07:22:07.516831+0000","last_became_active":"2024-03-06T07:22:06.321797+0000","last_became_peered":"2024-03-06T07:22:06.321797+0000","last_unstale":"2024-03-06T07:22:07.516831+0000","last_undegraded":"2024-03-06T07:22:07.516831+0000","last_fullsized":"2024-03-06T07:22:07.516831+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:36.139439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316549+0000","last_change":"2024-03-06T07:22:06.320372+0000","last_active":"2024-03-06T07:22:07.316549+0000","last_peered":"2024-03-06T07:22:07.316549+0000","last_clean":"2024-03-06T07:22:07.316549+0000","last_became_active":"2024-03-06T07:22:06.320109+0000","last_became_peered":"2024-03-06T07:22:06.320109+0000","last_unstale":"2024-03-06T07:22:07.316549+0000","last_undegraded":"2024-03-06T07:22:07.316549+0000","last_fullsized":"2024-03-06T07:22:07.316549+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:37.954594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":18,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516998+0000","last_change":"2024-03-06T07:22:05.317240+0000","last_active":"2024-03-06T07:22:07.516998+0000","last_peered":"2024-03-06T07:22:07.516998+0000","last_clean":"2024-03-06T07:22:07.516998+0000","last_became_active":"2024-03-06T07:22:05.316908+0000","last_became_peered":"2024-03-06T07:22:05.316908+0000","last_unstale":"2024-03-06T07:22:07.516998+0000","last_undegraded":"2024-03-06T07:22:07.516998+0000","last_fullsized":"2024-03-06T07:22:07.516998+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:18.512707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517456+0000","last_change":"2024-03-06T07:22:06.324306+0000","last_active":"2024-03-06T07:22:07.517456+0000","last_peered":"2024-03-06T07:22:07.517456+0000","last_clean":"2024-03-06T07:22:07.517456+0000","last_became_active":"2024-03-06T07:22:06.323979+0000","last_became_peered":"2024-03-06T07:22:06.323979+0000","last_unstale":"2024-03-06T07:22:07.517456+0000","last_undegraded":"2024-03-06T07:22:07.517456+0000","last_fullsized":"2024-03-06T07:22:07.517456+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:05.731939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317093+0000","last_change":"2024-03-06T07:22:05.321228+0000","last_active":"2024-03-06T07:22:07.317093+0000","last_peered":"2024-03-06T07:22:07.317093+0000","last_clean":"2024-03-06T07:22:07.317093+0000","last_became_active":"2024-03-06T07:22:05.320958+0000","last_became_peered":"2024-03-06T07:22:05.320958+0000","last_unstale":"2024-03-06T07:22:07.317093+0000","last_undegraded":"2024-03-06T07:22:07.317093+0000","last_fullsized":"2024-03-06T07:22:07.317093+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:43.288752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516599+0000","last_change":"2024-03-06T07:22:06.323135+0000","last_active":"2024-03-06T07:22:07.516599+0000","last_peered":"2024-03-06T07:22:07.516599+0000","last_clean":"2024-03-06T07:22:07.516599+0000","last_became_active":"2024-03-06T07:22:06.322873+0000","last_became_peered":"2024-03-06T07:22:06.322873+0000","last_unstale":"2024-03-06T07:22:07.516599+0000","last_undegraded":"2024-03-06T07:22:07.516599+0000","last_fullsized":"2024-03-06T07:22:07.516599+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:32.712920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317243+0000","last_change":"2024-03-06T07:22:05.321069+0000","last_active":"2024-03-06T07:22:07.317243+0000","last_peered":"2024-03-06T07:22:07.317243+0000","last_clean":"2024-03-06T07:22:07.317243+0000","last_became_active":"2024-03-06T07:22:05.320722+0000","last_became_peered":"2024-03-06T07:22:05.320722+0000","last_unstale":"2024-03-06T07:22:07.317243+0000","last_undegraded":"2024-03-06T07:22:07.317243+0000","last_fullsized":"2024-03-06T07:22:07.317243+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:26.446709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516359+0000","last_change":"2024-03-06T07:22:06.318876+0000","last_active":"2024-03-06T07:22:07.516359+0000","last_peered":"2024-03-06T07:22:07.516359+0000","last_clean":"2024-03-06T07:22:07.516359+0000","last_became_active":"2024-03-06T07:22:06.318574+0000","last_became_peered":"2024-03-06T07:22:06.318574+0000","last_unstale":"2024-03-06T07:22:07.516359+0000","last_undegraded":"2024-03-06T07:22:07.516359+0000","last_fullsized":"2024-03-06T07:22:07.516359+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:34.284715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517667+0000","last_change":"2024-03-06T07:22:05.316288+0000","last_active":"2024-03-06T07:22:07.517667+0000","last_peered":"2024-03-06T07:22:07.517667+0000","last_clean":"2024-03-06T07:22:07.517667+0000","last_became_active":"2024-03-06T07:22:05.315965+0000","last_became_peered":"2024-03-06T07:22:05.315965+0000","last_unstale":"2024-03-06T07:22:07.517667+0000","last_undegraded":"2024-03-06T07:22:07.517667+0000","last_fullsized":"2024-03-06T07:22:07.517667+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:19.545421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517766+0000","last_change":"2024-03-06T07:22:06.319384+0000","last_active":"2024-03-06T07:22:07.517766+0000","last_peered":"2024-03-06T07:22:07.517766+0000","last_clean":"2024-03-06T07:22:07.517766+0000","last_became_active":"2024-03-06T07:22:06.318885+0000","last_became_peered":"2024-03-06T07:22:06.318885+0000","last_unstale":"2024-03-06T07:22:07.517766+0000","last_undegraded":"2024-03-06T07:22:07.517766+0000","last_fullsized":"2024-03-06T07:22:07.517766+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:05.680579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315305+0000","last_change":"2024-03-06T07:22:05.319423+0000","last_active":"2024-03-06T07:22:07.315305+0000","last_peered":"2024-03-06T07:22:07.315305+0000","last_clean":"2024-03-06T07:22:07.315305+0000","last_became_active":"2024-03-06T07:22:05.319055+0000","last_became_peered":"2024-03-06T07:22:05.319055+0000","last_unstale":"2024-03-06T07:22:07.315305+0000","last_undegraded":"2024-03-06T07:22:07.315305+0000","last_fullsized":"2024-03-06T07:22:07.315305+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:19.853691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317008+0000","last_change":"2024-03-06T07:22:06.330770+0000","last_active":"2024-03-06T07:22:07.317008+0000","last_peered":"2024-03-06T07:22:07.317008+0000","last_clean":"2024-03-06T07:22:07.317008+0000","last_became_active":"2024-03-06T07:22:06.330404+0000","last_became_peered":"2024-03-06T07:22:06.330404+0000","last_unstale":"2024-03-06T07:22:07.317008+0000","last_undegraded":"2024-03-06T07:22:07.317008+0000","last_fullsized":"2024-03-06T07:22:07.317008+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:36.139439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517204+0000","last_change":"2024-03-06T07:22:05.313908+0000","last_active":"2024-03-06T07:22:07.517204+0000","last_peered":"2024-03-06T07:22:07.517204+0000","last_clean":"2024-03-06T07:22:07.517204+0000","last_became_active":"2024-03-06T07:22:05.313638+0000","last_became_peered":"2024-03-06T07:22:05.313638+0000","last_unstale":"2024-03-06T07:22:07.517204+0000","last_undegraded":"2024-03-06T07:22:07.517204+0000","last_fullsized":"2024-03-06T07:22:07.517204+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:46.701129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517266+0000","last_change":"2024-03-06T07:22:06.320596+0000","last_active":"2024-03-06T07:22:07.517266+0000","last_peered":"2024-03-06T07:22:07.517266+0000","last_clean":"2024-03-06T07:22:07.517266+0000","last_became_active":"2024-03-06T07:22:06.319430+0000","last_became_peered":"2024-03-06T07:22:06.319430+0000","last_unstale":"2024-03-06T07:22:07.517266+0000","last_undegraded":"2024-03-06T07:22:07.517266+0000","last_fullsized":"2024-03-06T07:22:07.517266+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:49.581292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517316+0000","last_change":"2024-03-06T07:22:05.320922+0000","last_active":"2024-03-06T07:22:07.517316+0000","last_peered":"2024-03-06T07:22:07.517316+0000","last_clean":"2024-03-06T07:22:07.517316+0000","last_became_active":"2024-03-06T07:22:05.320638+0000","last_became_peered":"2024-03-06T07:22:05.320638+0000","last_unstale":"2024-03-06T07:22:07.517316+0000","last_undegraded":"2024-03-06T07:22:07.517316+0000","last_fullsized":"2024-03-06T07:22:07.517316+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:13.798736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316765+0000","last_change":"2024-03-06T07:22:06.335202+0000","last_active":"2024-03-06T07:22:07.316765+0000","last_peered":"2024-03-06T07:22:07.316765+0000","last_clean":"2024-03-06T07:22:07.316765+0000","last_became_active":"2024-03-06T07:22:06.334922+0000","last_became_peered":"2024-03-06T07:22:06.334922+0000","last_unstale":"2024-03-06T07:22:07.316765+0000","last_undegraded":"2024-03-06T07:22:07.316765+0000","last_fullsized":"2024-03-06T07:22:07.316765+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:01.204740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315003+0000","last_change":"2024-03-06T07:22:05.326648+0000","last_active":"2024-03-06T07:22:07.315003+0000","last_peered":"2024-03-06T07:22:07.315003+0000","last_clean":"2024-03-06T07:22:07.315003+0000","last_became_active":"2024-03-06T07:22:05.326423+0000","last_became_peered":"2024-03-06T07:22:05.326423+0000","last_unstale":"2024-03-06T07:22:07.315003+0000","last_undegraded":"2024-03-06T07:22:07.315003+0000","last_fullsized":"2024-03-06T07:22:07.315003+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:04.675657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517225+0000","last_change":"2024-03-06T07:22:06.326561+0000","last_active":"2024-03-06T07:22:07.517225+0000","last_peered":"2024-03-06T07:22:07.517225+0000","last_clean":"2024-03-06T07:22:07.517225+0000","last_became_active":"2024-03-06T07:22:06.326295+0000","last_became_peered":"2024-03-06T07:22:06.326295+0000","last_unstale":"2024-03-06T07:22:07.517225+0000","last_undegraded":"2024-03-06T07:22:07.517225+0000","last_fullsized":"2024-03-06T07:22:07.517225+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:36.941825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316569+0000","last_change":"2024-03-06T07:22:05.319695+0000","last_active":"2024-03-06T07:22:07.316569+0000","last_peered":"2024-03-06T07:22:07.316569+0000","last_clean":"2024-03-06T07:22:07.316569+0000","last_became_active":"2024-03-06T07:22:05.319410+0000","last_became_peered":"2024-03-06T07:22:05.319410+0000","last_unstale":"2024-03-06T07:22:07.316569+0000","last_undegraded":"2024-03-06T07:22:07.316569+0000","last_fullsized":"2024-03-06T07:22:07.316569+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:06.303374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516554+0000","last_change":"2024-03-06T07:22:06.314348+0000","last_active":"2024-03-06T07:22:07.516554+0000","last_peered":"2024-03-06T07:22:07.516554+0000","last_clean":"2024-03-06T07:22:07.516554+0000","last_became_active":"2024-03-06T07:22:06.314044+0000","last_became_peered":"2024-03-06T07:22:06.314044+0000","last_unstale":"2024-03-06T07:22:07.516554+0000","last_undegraded":"2024-03-06T07:22:07.516554+0000","last_fullsized":"2024-03-06T07:22:07.516554+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:02.558815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517557+0000","last_change":"2024-03-06T07:22:05.321049+0000","last_active":"2024-03-06T07:22:07.517557+0000","last_peered":"2024-03-06T07:22:07.517557+0000","last_clean":"2024-03-06T07:22:07.517557+0000","last_became_active":"2024-03-06T07:22:05.320696+0000","last_became_peered":"2024-03-06T07:22:05.320696+0000","last_unstale":"2024-03-06T07:22:07.517557+0000","last_undegraded":"2024-03-06T07:22:07.517557+0000","last_fullsized":"2024-03-06T07:22:07.517557+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:26.446709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315896+0000","last_change":"2024-03-06T07:22:06.330920+0000","last_active":"2024-03-06T07:22:07.315896+0000","last_peered":"2024-03-06T07:22:07.315896+0000","last_clean":"2024-03-06T07:22:07.315896+0000","last_became_active":"2024-03-06T07:22:06.330331+0000","last_became_peered":"2024-03-06T07:22:06.330331+0000","last_unstale":"2024-03-06T07:22:07.315896+0000","last_undegraded":"2024-03-06T07:22:07.315896+0000","last_fullsized":"2024-03-06T07:22:07.315896+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:18.720650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":17,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317000+0000","last_change":"2024-03-06T07:22:05.321995+0000","last_active":"2024-03-06T07:22:07.317000+0000","last_peered":"2024-03-06T07:22:07.317000+0000","last_clean":"2024-03-06T07:22:07.317000+0000","last_became_active":"2024-03-06T07:22:05.321734+0000","last_became_peered":"2024-03-06T07:22:05.321734+0000","last_unstale":"2024-03-06T07:22:07.317000+0000","last_undegraded":"2024-03-06T07:22:07.317000+0000","last_fullsized":"2024-03-06T07:22:07.317000+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:40.725817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517991+0000","last_change":"2024-03-06T07:22:06.308003+0000","last_active":"2024-03-06T07:22:07.517991+0000","last_peered":"2024-03-06T07:22:07.517991+0000","last_clean":"2024-03-06T07:22:07.517991+0000","last_became_active":"2024-03-06T07:22:06.307696+0000","last_became_peered":"2024-03-06T07:22:06.307696+0000","last_unstale":"2024-03-06T07:22:07.517991+0000","last_undegraded":"2024-03-06T07:22:07.517991+0000","last_fullsized":"2024-03-06T07:22:07.517991+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:20.550343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":20,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517060+0000","last_change":"2024-03-06T07:22:05.313388+0000","last_active":"2024-03-06T07:22:07.517060+0000","last_peered":"2024-03-06T07:22:07.517060+0000","last_clean":"2024-03-06T07:22:07.517060+0000","last_became_active":"2024-03-06T07:22:05.312146+0000","last_became_peered":"2024-03-06T07:22:05.312146+0000","last_unstale":"2024-03-06T07:22:07.517060+0000","last_undegraded":"2024-03-06T07:22:07.517060+0000","last_fullsized":"2024-03-06T07:22:07.517060+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:16.630462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517645+0000","last_change":"2024-03-06T07:22:06.324830+0000","last_active":"2024-03-06T07:22:07.517645+0000","last_peered":"2024-03-06T07:22:07.517645+0000","last_clean":"2024-03-06T07:22:07.517645+0000","last_became_active":"2024-03-06T07:22:06.324574+0000","last_became_peered":"2024-03-06T07:22:06.324574+0000","last_unstale":"2024-03-06T07:22:07.517645+0000","last_undegraded":"2024-03-06T07:22:07.517645+0000","last_fullsized":"2024-03-06T07:22:07.517645+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:22.443979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315403+0000","last_change":"2024-03-06T07:22:05.326946+0000","last_active":"2024-03-06T07:22:07.315403+0000","last_peered":"2024-03-06T07:22:07.315403+0000","last_clean":"2024-03-06T07:22:07.315403+0000","last_became_active":"2024-03-06T07:22:05.326675+0000","last_became_peered":"2024-03-06T07:22:05.326675+0000","last_unstale":"2024-03-06T07:22:07.315403+0000","last_undegraded":"2024-03-06T07:22:07.315403+0000","last_fullsized":"2024-03-06T07:22:07.315403+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:35.936903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315948+0000","last_change":"2024-03-06T07:22:06.329315+0000","last_active":"2024-03-06T07:22:07.315948+0000","last_peered":"2024-03-06T07:22:07.315948+0000","last_clean":"2024-03-06T07:22:07.315948+0000","last_became_active":"2024-03-06T07:22:06.322815+0000","last_became_peered":"2024-03-06T07:22:06.322815+0000","last_unstale":"2024-03-06T07:22:07.315948+0000","last_undegraded":"2024-03-06T07:22:07.315948+0000","last_fullsized":"2024-03-06T07:22:07.315948+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:15.214675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516686+0000","last_change":"2024-03-06T07:22:05.324367+0000","last_active":"2024-03-06T07:22:07.516686+0000","last_peered":"2024-03-06T07:22:07.516686+0000","last_clean":"2024-03-06T07:22:07.516686+0000","last_became_active":"2024-03-06T07:22:05.324086+0000","last_became_peered":"2024-03-06T07:22:05.324086+0000","last_unstale":"2024-03-06T07:22:07.516686+0000","last_undegraded":"2024-03-06T07:22:07.516686+0000","last_fullsized":"2024-03-06T07:22:07.516686+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:31.061844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517116+0000","last_change":"2024-03-06T07:22:06.322371+0000","last_active":"2024-03-06T07:22:07.517116+0000","last_peered":"2024-03-06T07:22:07.517116+0000","last_clean":"2024-03-06T07:22:07.517116+0000","last_became_active":"2024-03-06T07:22:06.322099+0000","last_became_peered":"2024-03-06T07:22:06.322099+0000","last_unstale":"2024-03-06T07:22:07.517116+0000","last_undegraded":"2024-03-06T07:22:07.517116+0000","last_fullsized":"2024-03-06T07:22:07.517116+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:02.237473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317064+0000","last_change":"2024-03-06T07:22:05.320305+0000","last_active":"2024-03-06T07:22:07.317064+0000","last_peered":"2024-03-06T07:22:07.317064+0000","last_clean":"2024-03-06T07:22:07.317064+0000","last_became_active":"2024-03-06T07:22:05.319970+0000","last_became_peered":"2024-03-06T07:22:05.319970+0000","last_unstale":"2024-03-06T07:22:07.317064+0000","last_undegraded":"2024-03-06T07:22:07.317064+0000","last_fullsized":"2024-03-06T07:22:07.317064+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:01.935488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316347+0000","last_change":"2024-03-06T07:22:06.318628+0000","last_active":"2024-03-06T07:22:07.316347+0000","last_peered":"2024-03-06T07:22:07.316347+0000","last_clean":"2024-03-06T07:22:07.316347+0000","last_became_active":"2024-03-06T07:22:06.318324+0000","last_became_peered":"2024-03-06T07:22:06.318324+0000","last_unstale":"2024-03-06T07:22:07.316347+0000","last_undegraded":"2024-03-06T07:22:07.316347+0000","last_fullsized":"2024-03-06T07:22:07.316347+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:09.827959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":22,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516634+0000","last_change":"2024-03-06T07:22:05.316578+0000","last_active":"2024-03-06T07:22:07.516634+0000","last_peered":"2024-03-06T07:22:07.516634+0000","last_clean":"2024-03-06T07:22:07.516634+0000","last_became_active":"2024-03-06T07:22:05.316285+0000","last_became_peered":"2024-03-06T07:22:05.316285+0000","last_unstale":"2024-03-06T07:22:07.516634+0000","last_undegraded":"2024-03-06T07:22:07.516634+0000","last_fullsized":"2024-03-06T07:22:07.516634+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:21.439057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516581+0000","last_change":"2024-03-06T07:22:06.328590+0000","last_active":"2024-03-06T07:22:07.516581+0000","last_peered":"2024-03-06T07:22:07.516581+0000","last_clean":"2024-03-06T07:22:07.516581+0000","last_became_active":"2024-03-06T07:22:06.327594+0000","last_became_peered":"2024-03-06T07:22:06.327594+0000","last_unstale":"2024-03-06T07:22:07.516581+0000","last_undegraded":"2024-03-06T07:22:07.516581+0000","last_fullsized":"2024-03-06T07:22:07.516581+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:46.981658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316517+0000","last_change":"2024-03-06T07:22:05.325528+0000","last_active":"2024-03-06T07:22:07.316517+0000","last_peered":"2024-03-06T07:22:07.316517+0000","last_clean":"2024-03-06T07:22:07.316517+0000","last_became_active":"2024-03-06T07:22:05.325254+0000","last_became_peered":"2024-03-06T07:22:05.325254+0000","last_unstale":"2024-03-06T07:22:07.316517+0000","last_undegraded":"2024-03-06T07:22:07.316517+0000","last_fullsized":"2024-03-06T07:22:07.316517+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:44.392826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316730+0000","last_change":"2024-03-06T07:22:06.320394+0000","last_active":"2024-03-06T07:22:07.316730+0000","last_peered":"2024-03-06T07:22:07.316730+0000","last_clean":"2024-03-06T07:22:07.316730+0000","last_became_active":"2024-03-06T07:22:06.320135+0000","last_became_peered":"2024-03-06T07:22:06.320135+0000","last_unstale":"2024-03-06T07:22:07.316730+0000","last_undegraded":"2024-03-06T07:22:07.316730+0000","last_fullsized":"2024-03-06T07:22:07.316730+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:34.766315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516581+0000","last_change":"2024-03-06T07:22:05.314843+0000","last_active":"2024-03-06T07:22:07.516581+0000","last_peered":"2024-03-06T07:22:07.516581+0000","last_clean":"2024-03-06T07:22:07.516581+0000","last_became_active":"2024-03-06T07:22:05.314567+0000","last_became_peered":"2024-03-06T07:22:05.314567+0000","last_unstale":"2024-03-06T07:22:07.516581+0000","last_undegraded":"2024-03-06T07:22:07.516581+0000","last_fullsized":"2024-03-06T07:22:07.516581+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:23.891871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517144+0000","last_change":"2024-03-06T07:22:06.321026+0000","last_active":"2024-03-06T07:22:07.517144+0000","last_peered":"2024-03-06T07:22:07.517144+0000","last_clean":"2024-03-06T07:22:07.517144+0000","last_became_active":"2024-03-06T07:22:06.320749+0000","last_became_peered":"2024-03-06T07:22:06.320749+0000","last_unstale":"2024-03-06T07:22:07.517144+0000","last_undegraded":"2024-03-06T07:22:07.517144+0000","last_fullsized":"2024-03-06T07:22:07.517144+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:59.907792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517116+0000","last_change":"2024-03-06T07:22:05.325044+0000","last_active":"2024-03-06T07:22:07.517116+0000","last_peered":"2024-03-06T07:22:07.517116+0000","last_clean":"2024-03-06T07:22:07.517116+0000","last_became_active":"2024-03-06T07:22:05.324731+0000","last_became_peered":"2024-03-06T07:22:05.324731+0000","last_unstale":"2024-03-06T07:22:07.517116+0000","last_undegraded":"2024-03-06T07:22:07.517116+0000","last_fullsized":"2024-03-06T07:22:07.517116+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:50.594146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316958+0000","last_change":"2024-03-06T07:22:06.327972+0000","last_active":"2024-03-06T07:22:07.316958+0000","last_peered":"2024-03-06T07:22:07.316958+0000","last_clean":"2024-03-06T07:22:07.316958+0000","last_became_active":"2024-03-06T07:22:06.327676+0000","last_became_peered":"2024-03-06T07:22:06.327676+0000","last_unstale":"2024-03-06T07:22:07.316958+0000","last_undegraded":"2024-03-06T07:22:07.316958+0000","last_fullsized":"2024-03-06T07:22:07.316958+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:53.049705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517228+0000","last_change":"2024-03-06T07:22:05.318191+0000","last_active":"2024-03-06T07:22:07.517228+0000","last_peered":"2024-03-06T07:22:07.517228+0000","last_clean":"2024-03-06T07:22:07.517228+0000","last_became_active":"2024-03-06T07:22:05.317984+0000","last_became_peered":"2024-03-06T07:22:05.317984+0000","last_unstale":"2024-03-06T07:22:07.517228+0000","last_undegraded":"2024-03-06T07:22:07.517228+0000","last_fullsized":"2024-03-06T07:22:07.517228+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:48.457911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517619+0000","last_change":"2024-03-06T07:22:06.317413+0000","last_active":"2024-03-06T07:22:07.517619+0000","last_peered":"2024-03-06T07:22:07.517619+0000","last_clean":"2024-03-06T07:22:07.517619+0000","last_became_active":"2024-03-06T07:22:06.317126+0000","last_became_peered":"2024-03-06T07:22:06.317126+0000","last_unstale":"2024-03-06T07:22:07.517619+0000","last_undegraded":"2024-03-06T07:22:07.517619+0000","last_fullsized":"2024-03-06T07:22:07.517619+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:17.635384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316869+0000","last_change":"2024-03-06T07:22:05.320901+0000","last_active":"2024-03-06T07:22:07.316869+0000","last_peered":"2024-03-06T07:22:07.316869+0000","last_clean":"2024-03-06T07:22:07.316869+0000","last_became_active":"2024-03-06T07:22:05.320579+0000","last_became_peered":"2024-03-06T07:22:05.320579+0000","last_unstale":"2024-03-06T07:22:07.316869+0000","last_undegraded":"2024-03-06T07:22:07.316869+0000","last_fullsized":"2024-03-06T07:22:07.316869+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:46.701129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517070+0000","last_change":"2024-03-06T07:22:06.321030+0000","last_active":"2024-03-06T07:22:07.517070+0000","last_peered":"2024-03-06T07:22:07.517070+0000","last_clean":"2024-03-06T07:22:07.517070+0000","last_became_active":"2024-03-06T07:22:06.320686+0000","last_became_peered":"2024-03-06T07:22:06.320686+0000","last_unstale":"2024-03-06T07:22:07.517070+0000","last_undegraded":"2024-03-06T07:22:07.517070+0000","last_fullsized":"2024-03-06T07:22:07.517070+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:41.730739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516237+0000","last_change":"2024-03-06T07:22:05.316761+0000","last_active":"2024-03-06T07:22:07.516237+0000","last_peered":"2024-03-06T07:22:07.516237+0000","last_clean":"2024-03-06T07:22:07.516237+0000","last_became_active":"2024-03-06T07:22:05.316088+0000","last_became_peered":"2024-03-06T07:22:05.316088+0000","last_unstale":"2024-03-06T07:22:07.516237+0000","last_undegraded":"2024-03-06T07:22:07.516237+0000","last_fullsized":"2024-03-06T07:22:07.516237+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:35.134517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316158+0000","last_change":"2024-03-06T07:22:06.313796+0000","last_active":"2024-03-06T07:22:07.316158+0000","last_peered":"2024-03-06T07:22:07.316158+0000","last_clean":"2024-03-06T07:22:07.316158+0000","last_became_active":"2024-03-06T07:22:06.313534+0000","last_became_peered":"2024-03-06T07:22:06.313534+0000","last_unstale":"2024-03-06T07:22:07.316158+0000","last_undegraded":"2024-03-06T07:22:07.316158+0000","last_fullsized":"2024-03-06T07:22:07.316158+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:34.284715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315462+0000","last_change":"2024-03-06T07:22:05.324175+0000","last_active":"2024-03-06T07:22:07.315462+0000","last_peered":"2024-03-06T07:22:07.315462+0000","last_clean":"2024-03-06T07:22:07.315462+0000","last_became_active":"2024-03-06T07:22:05.323587+0000","last_became_peered":"2024-03-06T07:22:05.323587+0000","last_unstale":"2024-03-06T07:22:07.315462+0000","last_undegraded":"2024-03-06T07:22:07.315462+0000","last_fullsized":"2024-03-06T07:22:07.315462+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:01.935488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317313+0000","last_change":"2024-03-06T07:22:06.325375+0000","last_active":"2024-03-06T07:22:07.317313+0000","last_peered":"2024-03-06T07:22:07.317313+0000","last_clean":"2024-03-06T07:22:07.317313+0000","last_became_active":"2024-03-06T07:22:06.325108+0000","last_became_peered":"2024-03-06T07:22:06.325108+0000","last_unstale":"2024-03-06T07:22:07.317313+0000","last_undegraded":"2024-03-06T07:22:07.317313+0000","last_fullsized":"2024-03-06T07:22:07.317313+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:36.139439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516789+0000","last_change":"2024-03-06T07:22:05.317252+0000","last_active":"2024-03-06T07:22:07.516789+0000","last_peered":"2024-03-06T07:22:07.516789+0000","last_clean":"2024-03-06T07:22:07.516789+0000","last_became_active":"2024-03-06T07:22:05.316964+0000","last_became_peered":"2024-03-06T07:22:05.316964+0000","last_unstale":"2024-03-06T07:22:07.516789+0000","last_undegraded":"2024-03-06T07:22:07.516789+0000","last_fullsized":"2024-03-06T07:22:07.516789+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:34.517096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516532+0000","last_change":"2024-03-06T07:22:06.325564+0000","last_active":"2024-03-06T07:22:07.516532+0000","last_peered":"2024-03-06T07:22:07.516532+0000","last_clean":"2024-03-06T07:22:07.516532+0000","last_became_active":"2024-03-06T07:22:06.325298+0000","last_became_peered":"2024-03-06T07:22:06.325298+0000","last_unstale":"2024-03-06T07:22:07.516532+0000","last_undegraded":"2024-03-06T07:22:07.516532+0000","last_fullsized":"2024-03-06T07:22:07.516532+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:51.508838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517050+0000","last_change":"2024-03-06T07:22:05.326276+0000","last_active":"2024-03-06T07:22:07.517050+0000","last_peered":"2024-03-06T07:22:07.517050+0000","last_clean":"2024-03-06T07:22:07.517050+0000","last_became_active":"2024-03-06T07:22:05.325993+0000","last_became_peered":"2024-03-06T07:22:05.325993+0000","last_unstale":"2024-03-06T07:22:07.517050+0000","last_undegraded":"2024-03-06T07:22:07.517050+0000","last_fullsized":"2024-03-06T07:22:07.517050+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:19.366175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316881+0000","last_change":"2024-03-06T07:22:06.323310+0000","last_active":"2024-03-06T07:22:07.316881+0000","last_peered":"2024-03-06T07:22:07.316881+0000","last_clean":"2024-03-06T07:22:07.316881+0000","last_became_active":"2024-03-06T07:22:06.322311+0000","last_became_peered":"2024-03-06T07:22:06.322311+0000","last_unstale":"2024-03-06T07:22:07.316881+0000","last_undegraded":"2024-03-06T07:22:07.316881+0000","last_fullsized":"2024-03-06T07:22:07.316881+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:01.204740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317078+0000","last_change":"2024-03-06T07:22:05.319638+0000","last_active":"2024-03-06T07:22:07.317078+0000","last_peered":"2024-03-06T07:22:07.317078+0000","last_clean":"2024-03-06T07:22:07.317078+0000","last_became_active":"2024-03-06T07:22:05.319307+0000","last_became_peered":"2024-03-06T07:22:05.319307+0000","last_unstale":"2024-03-06T07:22:07.317078+0000","last_undegraded":"2024-03-06T07:22:07.317078+0000","last_fullsized":"2024-03-06T07:22:07.317078+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:19.853691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516496+0000","last_change":"2024-03-06T07:22:06.315689+0000","last_active":"2024-03-06T07:22:07.516496+0000","last_peered":"2024-03-06T07:22:07.516496+0000","last_clean":"2024-03-06T07:22:07.516496+0000","last_became_active":"2024-03-06T07:22:06.315395+0000","last_became_peered":"2024-03-06T07:22:06.315395+0000","last_unstale":"2024-03-06T07:22:07.516496+0000","last_undegraded":"2024-03-06T07:22:07.516496+0000","last_fullsized":"2024-03-06T07:22:07.516496+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:09.827959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315324+0000","last_change":"2024-03-06T07:22:05.319307+0000","last_active":"2024-03-06T07:22:07.315324+0000","last_peered":"2024-03-06T07:22:07.315324+0000","last_clean":"2024-03-06T07:22:07.315324+0000","last_became_active":"2024-03-06T07:22:05.318839+0000","last_became_peered":"2024-03-06T07:22:05.318839+0000","last_unstale":"2024-03-06T07:22:07.315324+0000","last_undegraded":"2024-03-06T07:22:07.315324+0000","last_fullsized":"2024-03-06T07:22:07.315324+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:40.725817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315277+0000","last_change":"2024-03-06T07:22:06.331443+0000","last_active":"2024-03-06T07:22:07.315277+0000","last_peered":"2024-03-06T07:22:07.315277+0000","last_clean":"2024-03-06T07:22:07.315277+0000","last_became_active":"2024-03-06T07:22:06.331189+0000","last_became_peered":"2024-03-06T07:22:06.331189+0000","last_unstale":"2024-03-06T07:22:07.315277+0000","last_undegraded":"2024-03-06T07:22:07.315277+0000","last_fullsized":"2024-03-06T07:22:07.315277+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:34.295379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516542+0000","last_change":"2024-03-06T07:22:05.315558+0000","last_active":"2024-03-06T07:22:07.516542+0000","last_peered":"2024-03-06T07:22:07.516542+0000","last_clean":"2024-03-06T07:22:07.516542+0000","last_became_active":"2024-03-06T07:22:05.315263+0000","last_became_peered":"2024-03-06T07:22:05.315263+0000","last_unstale":"2024-03-06T07:22:07.516542+0000","last_undegraded":"2024-03-06T07:22:07.516542+0000","last_fullsized":"2024-03-06T07:22:07.516542+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:01.553893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517825+0000","last_change":"2024-03-06T07:22:06.324288+0000","last_active":"2024-03-06T07:22:07.517825+0000","last_peered":"2024-03-06T07:22:07.517825+0000","last_clean":"2024-03-06T07:22:07.517825+0000","last_became_active":"2024-03-06T07:22:06.323925+0000","last_became_peered":"2024-03-06T07:22:06.323925+0000","last_unstale":"2024-03-06T07:22:07.517825+0000","last_undegraded":"2024-03-06T07:22:07.517825+0000","last_fullsized":"2024-03-06T07:22:07.517825+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:49.900223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516342+0000","last_change":"2024-03-06T07:22:05.322782+0000","last_active":"2024-03-06T07:22:07.516342+0000","last_peered":"2024-03-06T07:22:07.516342+0000","last_clean":"2024-03-06T07:22:07.516342+0000","last_became_active":"2024-03-06T07:22:05.322484+0000","last_became_peered":"2024-03-06T07:22:05.322484+0000","last_unstale":"2024-03-06T07:22:07.516342+0000","last_undegraded":"2024-03-06T07:22:07.516342+0000","last_fullsized":"2024-03-06T07:22:07.516342+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:04.727017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316019+0000","last_change":"2024-03-06T07:22:06.321965+0000","last_active":"2024-03-06T07:22:07.316019+0000","last_peered":"2024-03-06T07:22:07.316019+0000","last_clean":"2024-03-06T07:22:07.316019+0000","last_became_active":"2024-03-06T07:22:06.320167+0000","last_became_peered":"2024-03-06T07:22:06.320167+0000","last_unstale":"2024-03-06T07:22:07.316019+0000","last_undegraded":"2024-03-06T07:22:07.316019+0000","last_fullsized":"2024-03-06T07:22:07.316019+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:15.334381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517279+0000","last_change":"2024-03-06T07:22:05.312784+0000","last_active":"2024-03-06T07:22:07.517279+0000","last_peered":"2024-03-06T07:22:07.517279+0000","last_clean":"2024-03-06T07:22:07.517279+0000","last_became_active":"2024-03-06T07:22:05.312472+0000","last_became_peered":"2024-03-06T07:22:05.312472+0000","last_unstale":"2024-03-06T07:22:07.517279+0000","last_undegraded":"2024-03-06T07:22:07.517279+0000","last_fullsized":"2024-03-06T07:22:07.517279+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:43.288752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517339+0000","last_change":"2024-03-06T07:22:06.319779+0000","last_active":"2024-03-06T07:22:07.517339+0000","last_peered":"2024-03-06T07:22:07.517339+0000","last_clean":"2024-03-06T07:22:07.517339+0000","last_became_active":"2024-03-06T07:22:06.319415+0000","last_became_peered":"2024-03-06T07:22:06.319415+0000","last_unstale":"2024-03-06T07:22:07.517339+0000","last_undegraded":"2024-03-06T07:22:07.517339+0000","last_fullsized":"2024-03-06T07:22:07.517339+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:20.884386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516711+0000","last_change":"2024-03-06T07:22:05.322202+0000","last_active":"2024-03-06T07:22:07.516711+0000","last_peered":"2024-03-06T07:22:07.516711+0000","last_clean":"2024-03-06T07:22:07.516711+0000","last_became_active":"2024-03-06T07:22:05.321896+0000","last_became_peered":"2024-03-06T07:22:05.321896+0000","last_unstale":"2024-03-06T07:22:07.516711+0000","last_undegraded":"2024-03-06T07:22:07.516711+0000","last_fullsized":"2024-03-06T07:22:07.516711+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:44.392826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516613+0000","last_change":"2024-03-06T07:22:06.328678+0000","last_active":"2024-03-06T07:22:07.516613+0000","last_peered":"2024-03-06T07:22:07.516613+0000","last_clean":"2024-03-06T07:22:07.516613+0000","last_became_active":"2024-03-06T07:22:06.328362+0000","last_became_peered":"2024-03-06T07:22:06.328362+0000","last_unstale":"2024-03-06T07:22:07.516613+0000","last_undegraded":"2024-03-06T07:22:07.516613+0000","last_fullsized":"2024-03-06T07:22:07.516613+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:27.383059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"52'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316130+0000","last_change":"2024-03-06T07:22:05.326257+0000","last_active":"2024-03-06T07:22:07.316130+0000","last_peered":"2024-03-06T07:22:07.316130+0000","last_clean":"2024-03-06T07:22:07.316130+0000","last_became_active":"2024-03-06T07:22:05.325920+0000","last_became_peered":"2024-03-06T07:22:05.325920+0000","last_unstale":"2024-03-06T07:22:07.316130+0000","last_undegraded":"2024-03-06T07:22:07.316130+0000","last_fullsized":"2024-03-06T07:22:07.316130+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:58.902870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":230,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316623+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:22:07.316623+0000","last_peered":"2024-03-06T07:22:07.316623+0000","last_clean":"2024-03-06T07:22:07.316623+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:22:07.316623+0000","last_undegraded":"2024-03-06T07:22:07.316623+0000","last_fullsized":"2024-03-06T07:22:07.316623+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":868021,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":278181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":618496,"data_stored":277529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":622592,"data_stored":278111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":634880,"data_stored":279163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31804,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":630784,"data_stored":279141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":626688,"data_stored":277689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":692224,"data_stored":867953,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:11.593 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-06T07:22:11.591+0000 7fb8e6f7e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:13.947 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:14.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.047+0000 7f94cb5e2640 1 Processor -- start 2024-03-06T07:22:14.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.047+0000 7f94cb5e2640 1 -- start start 2024-03-06T07:22:14.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.048+0000 7f94cb5e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c40c6b10 0x7f94c4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.048+0000 7f94cb5e2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f94c40bf9d0 0x7f94c40c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.048+0000 7f94cb5e2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c40c2fc0 con 0x7f94c4152650 2024-03-06T07:22:14.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.048+0000 7f94cb5e2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c4155800 con 0x7f94c40bf9d0 2024-03-06T07:22:14.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.048+0000 7f94cb5e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c4158da0 con 0x7f94c40c6b10 2024-03-06T07:22:14.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.048+0000 7f94c937e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f94c4152650 0x7f94c4158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45370/0 (socket says 172.21.15.1:45370) 2024-03-06T07:22:14.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.048+0000 7f94c937e640 1 -- 172.21.15.1:0/1216212640 learned_addr learned my addr 172.21.15.1:0/1216212640 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:14.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.048+0000 7f94c9b7f640 1 --2- 172.21.15.1:0/1216212640 >> v2:172.21.15.98:3300/0 conn(0x7f94c40bf9d0 0x7f94c40c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.048+0000 7f94c8b7d640 1 --2- 172.21.15.1:0/1216212640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c40c6b10 0x7f94c4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.048+0000 7f94bbfff640 1 -- 172.21.15.1:0/1216212640 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245906977 0 0) 0x7f94c40c2fc0 con 0x7f94c4152650 2024-03-06T07:22:14.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.048+0000 7f94bbfff640 1 -- 172.21.15.1:0/1216212640 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94b0003520 con 0x7f94c4152650 2024-03-06T07:22:14.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.048+0000 7f94bbfff640 1 -- 172.21.15.1:0/1216212640 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1609823825 0 0) 0x7f94b0003520 con 0x7f94c4152650 2024-03-06T07:22:14.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.048+0000 7f94c9b7f640 1 -- 172.21.15.1:0/1216212640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c40c6b10 msgr2=0x7f94c4155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:14.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.048+0000 7f94c9b7f640 1 --2- 172.21.15.1:0/1216212640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c40c6b10 0x7f94c4155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.048+0000 7f94c9b7f640 1 -- 172.21.15.1:0/1216212640 >> v1:172.21.15.1:6789/0 conn(0x7f94c4152650 legacy=0x7f94c4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.048+0000 7f94c9b7f640 1 -- 172.21.15.1:0/1216212640 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c40c6690 con 0x7f94c40bf9d0 2024-03-06T07:22:14.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.049+0000 7f94c8b7d640 1 --2- 172.21.15.1:0/1216212640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c40c6b10 0x7f94c4155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:14.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.050+0000 7f94c9b7f640 1 --2- 172.21.15.1:0/1216212640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c40bf9d0 0x7f94c40c1460 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009d90 tx=0x7f94bc02bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdd7170890faade server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.050+0000 7f94bbfff640 1 -- 172.21.15.1:0/1216212640 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc039070 con 0x7f94c40bf9d0 2024-03-06T07:22:14.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.050+0000 7f94bbfff640 1 -- 172.21.15.1:0/1216212640 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94bc02c810 con 0x7f94c40bf9d0 2024-03-06T07:22:14.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.050+0000 7f94bbfff640 1 -- 172.21.15.1:0/1216212640 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c0001670 con 0x7f94c40bf9d0 2024-03-06T07:22:14.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.050+0000 7f94cb5e2640 1 -- 172.21.15.1:0/1216212640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c40bf9d0 msgr2=0x7f94c40c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.050+0000 7f94cb5e2640 1 --2- 172.21.15.1:0/1216212640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c40bf9d0 0x7f94c40c1460 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009d90 tx=0x7f94bc02bce0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.050+0000 7f94cb5e2640 1 -- 172.21.15.1:0/1216212640 shutdown_connections 2024-03-06T07:22:14.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.050+0000 7f94cb5e2640 1 --2- 172.21.15.1:0/1216212640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c40bf9d0 0x7f94c40c1460 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.050+0000 7f94cb5e2640 1 --2- 172.21.15.1:0/1216212640 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c40c6b10 0x7f94c4155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.050+0000 7f94cb5e2640 1 -- 172.21.15.1:0/1216212640 >> 172.21.15.1:0/1216212640 conn(0x7f94c4066ee0 msgr2=0x7f94c40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.051+0000 7f94cb5e2640 1 -- 172.21.15.1:0/1216212640 shutdown_connections 2024-03-06T07:22:14.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.051+0000 7f94cb5e2640 1 -- 172.21.15.1:0/1216212640 wait complete. 2024-03-06T07:22:14.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.052+0000 7f94cb5e2640 1 Processor -- start 2024-03-06T07:22:14.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.052+0000 7f94cb5e2640 1 -- start start 2024-03-06T07:22:14.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.052+0000 7f94cb5e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c40bf9d0 0x7f94c414d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.052+0000 7f94cb5e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c4152650 0x7f94c414a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.052+0000 7f94cb5e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c414b670 0x7f94c414aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.052+0000 7f94cb5e2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c41551b0 con 0x7f94c414b670 2024-03-06T07:22:14.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.052+0000 7f94cb5e2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c40c6fa0 con 0x7f94c40bf9d0 2024-03-06T07:22:14.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.052+0000 7f94cb5e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c4155490 con 0x7f94c4152650 2024-03-06T07:22:14.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.052+0000 7f94c9b7f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c414b670 0x7f94c414aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.052+0000 7f94c9b7f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c414b670 0x7f94c414aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59620/0 (socket says 172.21.15.1:59620) 2024-03-06T07:22:14.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.052+0000 7f94c9b7f640 1 -- 172.21.15.1:0/4137868872 learned_addr learned my addr 172.21.15.1:0/4137868872 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:14.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.052+0000 7f94c937e640 1 --2- 172.21.15.1:0/4137868872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c40bf9d0 0x7f94c414d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.053+0000 7f94c8b7d640 1 --2- 172.21.15.1:0/4137868872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c4152650 0x7f94c414a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.053+0000 7f94c9b7f640 1 -- 172.21.15.1:0/4137868872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c4152650 msgr2=0x7f94c414a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.053+0000 7f94c9b7f640 1 --2- 172.21.15.1:0/4137868872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c4152650 0x7f94c414a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.053+0000 7f94c9b7f640 1 -- 172.21.15.1:0/4137868872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c40bf9d0 msgr2=0x7f94c414d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.053+0000 7f94c9b7f640 1 --2- 172.21.15.1:0/4137868872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c40bf9d0 0x7f94c414d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.053+0000 7f94c9b7f640 1 -- 172.21.15.1:0/4137868872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c41e22d0 con 0x7f94c414b670 2024-03-06T07:22:14.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.053+0000 7f94c9b7f640 1 --2- 172.21.15.1:0/4137868872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c414b670 0x7f94c414aad0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009eb0 tx=0x7f94bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.053+0000 7f94b9ffb640 1 -- 172.21.15.1:0/4137868872 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc039050 con 0x7f94c414b670 2024-03-06T07:22:14.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.053+0000 7f94b9ffb640 1 -- 172.21.15.1:0/4137868872 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94bc045070 con 0x7f94c414b670 2024-03-06T07:22:14.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.053+0000 7f94cb5e2640 1 -- 172.21.15.1:0/4137868872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c4152bb0 con 0x7f94c414b670 2024-03-06T07:22:14.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.053+0000 7f94b9ffb640 1 -- 172.21.15.1:0/4137868872 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc009450 con 0x7f94c414b670 2024-03-06T07:22:14.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.053+0000 7f94cb5e2640 1 -- 172.21.15.1:0/4137868872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c41548a0 con 0x7f94c414b670 2024-03-06T07:22:14.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.054+0000 7f94cb5e2640 1 -- 172.21.15.1:0/4137868872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c4067fd0 con 0x7f94c414b670 2024-03-06T07:22:14.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.055+0000 7f94b9ffb640 1 -- 172.21.15.1:0/4137868872 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94bc040690 con 0x7f94c414b670 2024-03-06T07:22:14.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.055+0000 7f94b9ffb640 1 --2- 172.21.15.1:0/4137868872 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94a806b500 0x7f94a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.055+0000 7f94b9ffb640 1 -- 172.21.15.1:0/4137868872 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f94bc0ac9a0 con 0x7f94c414b670 2024-03-06T07:22:14.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.056+0000 7f94c937e640 1 --2- 172.21.15.1:0/4137868872 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94a806b500 0x7f94a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.056+0000 7f94c937e640 1 --2- 172.21.15.1:0/4137868872 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94a806b500 0x7f94a806d9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f94c00013c0 tx=0x7f94c0001350 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.064+0000 7f94b9ffb640 1 -- 172.21.15.1:0/4137868872 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94c41e22d0 con 0x7f94c414b670 2024-03-06T07:22:14.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.241+0000 7f94cb5e2640 1 -- 172.21.15.1:0/4137868872 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f94c4155ed0 con 0x7f94a806b500 2024-03-06T07:22:14.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.247+0000 7f94b9ffb640 1 -- 172.21.15.1:0/4137868872 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252030 (secure 0 0 0) 0x7f94c4155ed0 con 0x7f94a806b500 2024-03-06T07:22:14.252 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:14.253 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:22:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.250+0000 7f94cb5e2640 1 -- 172.21.15.1:0/4137868872 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94a806b500 msgr2=0x7f94a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.250+0000 7f94cb5e2640 1 --2- 172.21.15.1:0/4137868872 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94a806b500 0x7f94a806d9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f94c00013c0 tx=0x7f94c0001350 comp rx=0 tx=0).stop 2024-03-06T07:22:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.250+0000 7f94cb5e2640 1 -- 172.21.15.1:0/4137868872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c414b670 msgr2=0x7f94c414aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.250+0000 7f94cb5e2640 1 --2- 172.21.15.1:0/4137868872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c414b670 0x7f94c414aad0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009eb0 tx=0x7f94bc004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.250+0000 7f94cb5e2640 1 -- 172.21.15.1:0/4137868872 shutdown_connections 2024-03-06T07:22:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.250+0000 7f94cb5e2640 1 --2- 172.21.15.1:0/4137868872 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94a806b500 0x7f94a806d9c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.250+0000 7f94cb5e2640 1 --2- 172.21.15.1:0/4137868872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c414b670 0x7f94c414aad0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.250+0000 7f94cb5e2640 1 --2- 172.21.15.1:0/4137868872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c4152650 0x7f94c414a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.250+0000 7f94cb5e2640 1 --2- 172.21.15.1:0/4137868872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c40bf9d0 0x7f94c414d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.250+0000 7f94cb5e2640 1 -- 172.21.15.1:0/4137868872 >> 172.21.15.1:0/4137868872 conn(0x7f94c4066ee0 msgr2=0x7f94c40c3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.250+0000 7f94cb5e2640 1 -- 172.21.15.1:0/4137868872 shutdown_connections 2024-03-06T07:22:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.250+0000 7f94cb5e2640 1 -- 172.21.15.1:0/4137868872 wait complete. 2024-03-06T07:22:14.263 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-03-06T07:22:12.714899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":186,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237832,"kb_used_data":5024,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736888,"statfs":{"total":773094113280,"available":772850573312,"internally_reserved":0,"allocated":5144576,"data_stored":3405788,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.414612"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317049+0000","last_change":"2024-03-06T07:22:06.323021+0000","last_active":"2024-03-06T07:22:07.317049+0000","last_peered":"2024-03-06T07:22:07.317049+0000","last_clean":"2024-03-06T07:22:07.317049+0000","last_became_active":"2024-03-06T07:22:06.319646+0000","last_became_peered":"2024-03-06T07:22:06.319646+0000","last_unstale":"2024-03-06T07:22:07.317049+0000","last_undegraded":"2024-03-06T07:22:07.317049+0000","last_fullsized":"2024-03-06T07:22:07.317049+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:51.508838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316898+0000","last_change":"2024-03-06T07:22:06.325529+0000","last_active":"2024-03-06T07:22:07.316898+0000","last_peered":"2024-03-06T07:22:07.316898+0000","last_clean":"2024-03-06T07:22:07.316898+0000","last_became_active":"2024-03-06T07:22:06.324758+0000","last_became_peered":"2024-03-06T07:22:06.324758+0000","last_unstale":"2024-03-06T07:22:07.316898+0000","last_undegraded":"2024-03-06T07:22:07.316898+0000","last_fullsized":"2024-03-06T07:22:07.316898+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:02.558815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315668+0000","last_change":"2024-03-06T07:22:06.321470+0000","last_active":"2024-03-06T07:22:07.315668+0000","last_peered":"2024-03-06T07:22:07.315668+0000","last_clean":"2024-03-06T07:22:07.315668+0000","last_became_active":"2024-03-06T07:22:06.318716+0000","last_became_peered":"2024-03-06T07:22:06.318716+0000","last_unstale":"2024-03-06T07:22:07.315668+0000","last_undegraded":"2024-03-06T07:22:07.315668+0000","last_fullsized":"2024-03-06T07:22:07.315668+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:16.706902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316698+0000","last_change":"2024-03-06T07:22:06.321775+0000","last_active":"2024-03-06T07:22:07.316698+0000","last_peered":"2024-03-06T07:22:07.316698+0000","last_clean":"2024-03-06T07:22:07.316698+0000","last_became_active":"2024-03-06T07:22:06.321437+0000","last_became_peered":"2024-03-06T07:22:06.321437+0000","last_unstale":"2024-03-06T07:22:07.316698+0000","last_undegraded":"2024-03-06T07:22:07.316698+0000","last_fullsized":"2024-03-06T07:22:07.316698+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:53.049705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517963+0000","last_change":"2024-03-06T07:22:06.317556+0000","last_active":"2024-03-06T07:22:07.517963+0000","last_peered":"2024-03-06T07:22:07.517963+0000","last_clean":"2024-03-06T07:22:07.517963+0000","last_became_active":"2024-03-06T07:22:06.317298+0000","last_became_peered":"2024-03-06T07:22:06.317298+0000","last_unstale":"2024-03-06T07:22:07.517963+0000","last_undegraded":"2024-03-06T07:22:07.517963+0000","last_fullsized":"2024-03-06T07:22:07.517963+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:45.397748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316863+0000","last_change":"2024-03-06T07:22:06.323812+0000","last_active":"2024-03-06T07:22:07.316863+0000","last_peered":"2024-03-06T07:22:07.316863+0000","last_clean":"2024-03-06T07:22:07.316863+0000","last_became_active":"2024-03-06T07:22:06.323558+0000","last_became_peered":"2024-03-06T07:22:06.323558+0000","last_unstale":"2024-03-06T07:22:07.316863+0000","last_undegraded":"2024-03-06T07:22:07.316863+0000","last_fullsized":"2024-03-06T07:22:07.316863+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:37.954594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315285+0000","last_change":"2024-03-06T07:22:06.322447+0000","last_active":"2024-03-06T07:22:07.315285+0000","last_peered":"2024-03-06T07:22:07.315285+0000","last_clean":"2024-03-06T07:22:07.315285+0000","last_became_active":"2024-03-06T07:22:06.318369+0000","last_became_peered":"2024-03-06T07:22:06.318369+0000","last_unstale":"2024-03-06T07:22:07.315285+0000","last_undegraded":"2024-03-06T07:22:07.315285+0000","last_fullsized":"2024-03-06T07:22:07.315285+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:50.150401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516696+0000","last_change":"2024-03-06T07:22:06.328376+0000","last_active":"2024-03-06T07:22:07.516696+0000","last_peered":"2024-03-06T07:22:07.516696+0000","last_clean":"2024-03-06T07:22:07.516696+0000","last_became_active":"2024-03-06T07:22:06.327042+0000","last_became_peered":"2024-03-06T07:22:06.327042+0000","last_unstale":"2024-03-06T07:22:07.516696+0000","last_undegraded":"2024-03-06T07:22:07.516696+0000","last_fullsized":"2024-03-06T07:22:07.516696+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:34.670786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517253+0000","last_change":"2024-03-06T07:22:06.335876+0000","last_active":"2024-03-06T07:22:07.517253+0000","last_peered":"2024-03-06T07:22:07.517253+0000","last_clean":"2024-03-06T07:22:07.517253+0000","last_became_active":"2024-03-06T07:22:06.335575+0000","last_became_peered":"2024-03-06T07:22:06.335575+0000","last_unstale":"2024-03-06T07:22:07.517253+0000","last_undegraded":"2024-03-06T07:22:07.517253+0000","last_fullsized":"2024-03-06T07:22:07.517253+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:19.402946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317040+0000","last_change":"2024-03-06T07:22:06.320896+0000","last_active":"2024-03-06T07:22:07.317040+0000","last_peered":"2024-03-06T07:22:07.317040+0000","last_clean":"2024-03-06T07:22:07.317040+0000","last_became_active":"2024-03-06T07:22:06.320643+0000","last_became_peered":"2024-03-06T07:22:06.320643+0000","last_unstale":"2024-03-06T07:22:07.317040+0000","last_undegraded":"2024-03-06T07:22:07.317040+0000","last_fullsized":"2024-03-06T07:22:07.317040+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:09.367201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516602+0000","last_change":"2024-03-06T07:22:06.322171+0000","last_active":"2024-03-06T07:22:07.516602+0000","last_peered":"2024-03-06T07:22:07.516602+0000","last_clean":"2024-03-06T07:22:07.516602+0000","last_became_active":"2024-03-06T07:22:06.321891+0000","last_became_peered":"2024-03-06T07:22:06.321891+0000","last_unstale":"2024-03-06T07:22:07.516602+0000","last_undegraded":"2024-03-06T07:22:07.516602+0000","last_fullsized":"2024-03-06T07:22:07.516602+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:02.261930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517100+0000","last_change":"2024-03-06T07:22:06.321590+0000","last_active":"2024-03-06T07:22:07.517100+0000","last_peered":"2024-03-06T07:22:07.517100+0000","last_clean":"2024-03-06T07:22:07.517100+0000","last_became_active":"2024-03-06T07:22:06.321334+0000","last_became_peered":"2024-03-06T07:22:06.321334+0000","last_unstale":"2024-03-06T07:22:07.517100+0000","last_undegraded":"2024-03-06T07:22:07.517100+0000","last_fullsized":"2024-03-06T07:22:07.517100+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:49.931180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316738+0000","last_change":"2024-03-06T07:22:06.328682+0000","last_active":"2024-03-06T07:22:07.316738+0000","last_peered":"2024-03-06T07:22:07.316738+0000","last_clean":"2024-03-06T07:22:07.316738+0000","last_became_active":"2024-03-06T07:22:06.328418+0000","last_became_peered":"2024-03-06T07:22:06.328418+0000","last_unstale":"2024-03-06T07:22:07.316738+0000","last_undegraded":"2024-03-06T07:22:07.316738+0000","last_fullsized":"2024-03-06T07:22:07.316738+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:52.638347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315858+0000","last_change":"2024-03-06T07:22:06.330122+0000","last_active":"2024-03-06T07:22:07.315858+0000","last_peered":"2024-03-06T07:22:07.315858+0000","last_clean":"2024-03-06T07:22:07.315858+0000","last_became_active":"2024-03-06T07:22:06.329780+0000","last_became_peered":"2024-03-06T07:22:06.329780+0000","last_unstale":"2024-03-06T07:22:07.315858+0000","last_undegraded":"2024-03-06T07:22:07.315858+0000","last_fullsized":"2024-03-06T07:22:07.315858+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:38.898151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317089+0000","last_change":"2024-03-06T07:22:06.321523+0000","last_active":"2024-03-06T07:22:07.317089+0000","last_peered":"2024-03-06T07:22:07.317089+0000","last_clean":"2024-03-06T07:22:07.317089+0000","last_became_active":"2024-03-06T07:22:06.321224+0000","last_became_peered":"2024-03-06T07:22:06.321224+0000","last_unstale":"2024-03-06T07:22:07.317089+0000","last_undegraded":"2024-03-06T07:22:07.317089+0000","last_fullsized":"2024-03-06T07:22:07.317089+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:05.731939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316709+0000","last_change":"2024-03-06T07:22:06.314914+0000","last_active":"2024-03-06T07:22:07.316709+0000","last_peered":"2024-03-06T07:22:07.316709+0000","last_clean":"2024-03-06T07:22:07.316709+0000","last_became_active":"2024-03-06T07:22:06.314654+0000","last_became_peered":"2024-03-06T07:22:06.314654+0000","last_unstale":"2024-03-06T07:22:07.316709+0000","last_undegraded":"2024-03-06T07:22:07.316709+0000","last_fullsized":"2024-03-06T07:22:07.316709+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:18.848063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316467+0000","last_change":"2024-03-06T07:22:06.323974+0000","last_active":"2024-03-06T07:22:07.316467+0000","last_peered":"2024-03-06T07:22:07.316467+0000","last_clean":"2024-03-06T07:22:07.316467+0000","last_became_active":"2024-03-06T07:22:06.323718+0000","last_became_peered":"2024-03-06T07:22:06.323718+0000","last_unstale":"2024-03-06T07:22:07.316467+0000","last_undegraded":"2024-03-06T07:22:07.316467+0000","last_fullsized":"2024-03-06T07:22:07.316467+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:46.981658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316834+0000","last_change":"2024-03-06T07:22:06.325388+0000","last_active":"2024-03-06T07:22:07.316834+0000","last_peered":"2024-03-06T07:22:07.316834+0000","last_clean":"2024-03-06T07:22:07.316834+0000","last_became_active":"2024-03-06T07:22:06.325109+0000","last_became_peered":"2024-03-06T07:22:06.325109+0000","last_unstale":"2024-03-06T07:22:07.316834+0000","last_undegraded":"2024-03-06T07:22:07.316834+0000","last_fullsized":"2024-03-06T07:22:07.316834+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:51.508838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315801+0000","last_change":"2024-03-06T07:22:06.329771+0000","last_active":"2024-03-06T07:22:07.315801+0000","last_peered":"2024-03-06T07:22:07.315801+0000","last_clean":"2024-03-06T07:22:07.315801+0000","last_became_active":"2024-03-06T07:22:06.318483+0000","last_became_peered":"2024-03-06T07:22:06.318483+0000","last_unstale":"2024-03-06T07:22:07.315801+0000","last_undegraded":"2024-03-06T07:22:07.315801+0000","last_fullsized":"2024-03-06T07:22:07.315801+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:17.568813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516970+0000","last_change":"2024-03-06T07:22:06.317746+0000","last_active":"2024-03-06T07:22:07.516970+0000","last_peered":"2024-03-06T07:22:07.516970+0000","last_clean":"2024-03-06T07:22:07.516970+0000","last_became_active":"2024-03-06T07:22:06.317465+0000","last_became_peered":"2024-03-06T07:22:06.317465+0000","last_unstale":"2024-03-06T07:22:07.516970+0000","last_undegraded":"2024-03-06T07:22:07.516970+0000","last_fullsized":"2024-03-06T07:22:07.516970+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:07.308296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516958+0000","last_change":"2024-03-06T07:22:06.322670+0000","last_active":"2024-03-06T07:22:07.516958+0000","last_peered":"2024-03-06T07:22:07.516958+0000","last_clean":"2024-03-06T07:22:07.516958+0000","last_became_active":"2024-03-06T07:22:06.322382+0000","last_became_peered":"2024-03-06T07:22:06.322382+0000","last_unstale":"2024-03-06T07:22:07.516958+0000","last_undegraded":"2024-03-06T07:22:07.516958+0000","last_fullsized":"2024-03-06T07:22:07.516958+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:02.940410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516289+0000","last_change":"2024-03-06T07:22:06.331711+0000","last_active":"2024-03-06T07:22:07.516289+0000","last_peered":"2024-03-06T07:22:07.516289+0000","last_clean":"2024-03-06T07:22:07.516289+0000","last_became_active":"2024-03-06T07:22:06.326618+0000","last_became_peered":"2024-03-06T07:22:06.326618+0000","last_unstale":"2024-03-06T07:22:07.516289+0000","last_undegraded":"2024-03-06T07:22:07.516289+0000","last_fullsized":"2024-03-06T07:22:07.516289+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:15.334381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516859+0000","last_change":"2024-03-06T07:22:06.318320+0000","last_active":"2024-03-06T07:22:07.516859+0000","last_peered":"2024-03-06T07:22:07.516859+0000","last_clean":"2024-03-06T07:22:07.516859+0000","last_became_active":"2024-03-06T07:22:06.318065+0000","last_became_peered":"2024-03-06T07:22:06.318065+0000","last_unstale":"2024-03-06T07:22:07.516859+0000","last_undegraded":"2024-03-06T07:22:07.516859+0000","last_fullsized":"2024-03-06T07:22:07.516859+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:00.505615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315007+0000","last_change":"2024-03-06T07:22:06.318170+0000","last_active":"2024-03-06T07:22:07.315007+0000","last_peered":"2024-03-06T07:22:07.315007+0000","last_clean":"2024-03-06T07:22:07.315007+0000","last_became_active":"2024-03-06T07:22:06.317903+0000","last_became_peered":"2024-03-06T07:22:06.317903+0000","last_unstale":"2024-03-06T07:22:07.315007+0000","last_undegraded":"2024-03-06T07:22:07.315007+0000","last_fullsized":"2024-03-06T07:22:07.315007+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:25:11.151255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316491+0000","last_change":"2024-03-06T07:22:06.333955+0000","last_active":"2024-03-06T07:22:07.316491+0000","last_peered":"2024-03-06T07:22:07.316491+0000","last_clean":"2024-03-06T07:22:07.316491+0000","last_became_active":"2024-03-06T07:22:06.333703+0000","last_became_peered":"2024-03-06T07:22:06.333703+0000","last_unstale":"2024-03-06T07:22:07.316491+0000","last_undegraded":"2024-03-06T07:22:07.316491+0000","last_fullsized":"2024-03-06T07:22:07.316491+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:34.670786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517246+0000","last_change":"2024-03-06T07:22:06.320349+0000","last_active":"2024-03-06T07:22:07.517246+0000","last_peered":"2024-03-06T07:22:07.517246+0000","last_clean":"2024-03-06T07:22:07.517246+0000","last_became_active":"2024-03-06T07:22:06.320094+0000","last_became_peered":"2024-03-06T07:22:06.320094+0000","last_unstale":"2024-03-06T07:22:07.517246+0000","last_undegraded":"2024-03-06T07:22:07.517246+0000","last_fullsized":"2024-03-06T07:22:07.517246+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:48.224301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317246+0000","last_change":"2024-03-06T07:22:06.319987+0000","last_active":"2024-03-06T07:22:07.317246+0000","last_peered":"2024-03-06T07:22:07.317246+0000","last_clean":"2024-03-06T07:22:07.317246+0000","last_became_active":"2024-03-06T07:22:06.319703+0000","last_became_peered":"2024-03-06T07:22:06.319703+0000","last_unstale":"2024-03-06T07:22:07.317246+0000","last_undegraded":"2024-03-06T07:22:07.317246+0000","last_fullsized":"2024-03-06T07:22:07.317246+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:02.558815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517174+0000","last_change":"2024-03-06T07:22:06.318435+0000","last_active":"2024-03-06T07:22:07.517174+0000","last_peered":"2024-03-06T07:22:07.517174+0000","last_clean":"2024-03-06T07:22:07.517174+0000","last_became_active":"2024-03-06T07:22:06.318147+0000","last_became_peered":"2024-03-06T07:22:06.318147+0000","last_unstale":"2024-03-06T07:22:07.517174+0000","last_undegraded":"2024-03-06T07:22:07.517174+0000","last_fullsized":"2024-03-06T07:22:07.517174+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:20.858614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315255+0000","last_change":"2024-03-06T07:22:06.335154+0000","last_active":"2024-03-06T07:22:07.315255+0000","last_peered":"2024-03-06T07:22:07.315255+0000","last_clean":"2024-03-06T07:22:07.315255+0000","last_became_active":"2024-03-06T07:22:06.334855+0000","last_became_peered":"2024-03-06T07:22:06.334855+0000","last_unstale":"2024-03-06T07:22:07.315255+0000","last_undegraded":"2024-03-06T07:22:07.315255+0000","last_fullsized":"2024-03-06T07:22:07.315255+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:54.652557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316793+0000","last_change":"2024-03-06T07:22:06.330141+0000","last_active":"2024-03-06T07:22:07.316793+0000","last_peered":"2024-03-06T07:22:07.316793+0000","last_clean":"2024-03-06T07:22:07.316793+0000","last_became_active":"2024-03-06T07:22:06.329487+0000","last_became_peered":"2024-03-06T07:22:06.329487+0000","last_unstale":"2024-03-06T07:22:07.316793+0000","last_undegraded":"2024-03-06T07:22:07.316793+0000","last_fullsized":"2024-03-06T07:22:07.316793+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:09.367201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315416+0000","last_change":"2024-03-06T07:22:06.322218+0000","last_active":"2024-03-06T07:22:07.315416+0000","last_peered":"2024-03-06T07:22:07.315416+0000","last_clean":"2024-03-06T07:22:07.315416+0000","last_became_active":"2024-03-06T07:22:06.318026+0000","last_became_peered":"2024-03-06T07:22:06.318026+0000","last_unstale":"2024-03-06T07:22:07.315416+0000","last_undegraded":"2024-03-06T07:22:07.315416+0000","last_fullsized":"2024-03-06T07:22:07.315416+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:16.809640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516831+0000","last_change":"2024-03-06T07:22:06.322067+0000","last_active":"2024-03-06T07:22:07.516831+0000","last_peered":"2024-03-06T07:22:07.516831+0000","last_clean":"2024-03-06T07:22:07.516831+0000","last_became_active":"2024-03-06T07:22:06.321797+0000","last_became_peered":"2024-03-06T07:22:06.321797+0000","last_unstale":"2024-03-06T07:22:07.516831+0000","last_undegraded":"2024-03-06T07:22:07.516831+0000","last_fullsized":"2024-03-06T07:22:07.516831+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:36.139439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316549+0000","last_change":"2024-03-06T07:22:06.320372+0000","last_active":"2024-03-06T07:22:07.316549+0000","last_peered":"2024-03-06T07:22:07.316549+0000","last_clean":"2024-03-06T07:22:07.316549+0000","last_became_active":"2024-03-06T07:22:06.320109+0000","last_became_peered":"2024-03-06T07:22:06.320109+0000","last_unstale":"2024-03-06T07:22:07.316549+0000","last_undegraded":"2024-03-06T07:22:07.316549+0000","last_fullsized":"2024-03-06T07:22:07.316549+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:37.954594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":18,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516998+0000","last_change":"2024-03-06T07:22:05.317240+0000","last_active":"2024-03-06T07:22:07.516998+0000","last_peered":"2024-03-06T07:22:07.516998+0000","last_clean":"2024-03-06T07:22:07.516998+0000","last_became_active":"2024-03-06T07:22:05.316908+0000","last_became_peered":"2024-03-06T07:22:05.316908+0000","last_unstale":"2024-03-06T07:22:07.516998+0000","last_undegraded":"2024-03-06T07:22:07.516998+0000","last_fullsized":"2024-03-06T07:22:07.516998+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:18.512707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517456+0000","last_change":"2024-03-06T07:22:06.324306+0000","last_active":"2024-03-06T07:22:07.517456+0000","last_peered":"2024-03-06T07:22:07.517456+0000","last_clean":"2024-03-06T07:22:07.517456+0000","last_became_active":"2024-03-06T07:22:06.323979+0000","last_became_peered":"2024-03-06T07:22:06.323979+0000","last_unstale":"2024-03-06T07:22:07.517456+0000","last_undegraded":"2024-03-06T07:22:07.517456+0000","last_fullsized":"2024-03-06T07:22:07.517456+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:05.731939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317093+0000","last_change":"2024-03-06T07:22:05.321228+0000","last_active":"2024-03-06T07:22:07.317093+0000","last_peered":"2024-03-06T07:22:07.317093+0000","last_clean":"2024-03-06T07:22:07.317093+0000","last_became_active":"2024-03-06T07:22:05.320958+0000","last_became_peered":"2024-03-06T07:22:05.320958+0000","last_unstale":"2024-03-06T07:22:07.317093+0000","last_undegraded":"2024-03-06T07:22:07.317093+0000","last_fullsized":"2024-03-06T07:22:07.317093+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:43.288752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516599+0000","last_change":"2024-03-06T07:22:06.323135+0000","last_active":"2024-03-06T07:22:07.516599+0000","last_peered":"2024-03-06T07:22:07.516599+0000","last_clean":"2024-03-06T07:22:07.516599+0000","last_became_active":"2024-03-06T07:22:06.322873+0000","last_became_peered":"2024-03-06T07:22:06.322873+0000","last_unstale":"2024-03-06T07:22:07.516599+0000","last_undegraded":"2024-03-06T07:22:07.516599+0000","last_fullsized":"2024-03-06T07:22:07.516599+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:32.712920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317243+0000","last_change":"2024-03-06T07:22:05.321069+0000","last_active":"2024-03-06T07:22:07.317243+0000","last_peered":"2024-03-06T07:22:07.317243+0000","last_clean":"2024-03-06T07:22:07.317243+0000","last_became_active":"2024-03-06T07:22:05.320722+0000","last_became_peered":"2024-03-06T07:22:05.320722+0000","last_unstale":"2024-03-06T07:22:07.317243+0000","last_undegraded":"2024-03-06T07:22:07.317243+0000","last_fullsized":"2024-03-06T07:22:07.317243+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:26.446709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516359+0000","last_change":"2024-03-06T07:22:06.318876+0000","last_active":"2024-03-06T07:22:07.516359+0000","last_peered":"2024-03-06T07:22:07.516359+0000","last_clean":"2024-03-06T07:22:07.516359+0000","last_became_active":"2024-03-06T07:22:06.318574+0000","last_became_peered":"2024-03-06T07:22:06.318574+0000","last_unstale":"2024-03-06T07:22:07.516359+0000","last_undegraded":"2024-03-06T07:22:07.516359+0000","last_fullsized":"2024-03-06T07:22:07.516359+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:34.284715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517667+0000","last_change":"2024-03-06T07:22:05.316288+0000","last_active":"2024-03-06T07:22:07.517667+0000","last_peered":"2024-03-06T07:22:07.517667+0000","last_clean":"2024-03-06T07:22:07.517667+0000","last_became_active":"2024-03-06T07:22:05.315965+0000","last_became_peered":"2024-03-06T07:22:05.315965+0000","last_unstale":"2024-03-06T07:22:07.517667+0000","last_undegraded":"2024-03-06T07:22:07.517667+0000","last_fullsized":"2024-03-06T07:22:07.517667+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:19.545421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517766+0000","last_change":"2024-03-06T07:22:06.319384+0000","last_active":"2024-03-06T07:22:07.517766+0000","last_peered":"2024-03-06T07:22:07.517766+0000","last_clean":"2024-03-06T07:22:07.517766+0000","last_became_active":"2024-03-06T07:22:06.318885+0000","last_became_peered":"2024-03-06T07:22:06.318885+0000","last_unstale":"2024-03-06T07:22:07.517766+0000","last_undegraded":"2024-03-06T07:22:07.517766+0000","last_fullsized":"2024-03-06T07:22:07.517766+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:05.680579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315305+0000","last_change":"2024-03-06T07:22:05.319423+0000","last_active":"2024-03-06T07:22:07.315305+0000","last_peered":"2024-03-06T07:22:07.315305+0000","last_clean":"2024-03-06T07:22:07.315305+0000","last_became_active":"2024-03-06T07:22:05.319055+0000","last_became_peered":"2024-03-06T07:22:05.319055+0000","last_unstale":"2024-03-06T07:22:07.315305+0000","last_undegraded":"2024-03-06T07:22:07.315305+0000","last_fullsized":"2024-03-06T07:22:07.315305+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:19.853691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317008+0000","last_change":"2024-03-06T07:22:06.330770+0000","last_active":"2024-03-06T07:22:07.317008+0000","last_peered":"2024-03-06T07:22:07.317008+0000","last_clean":"2024-03-06T07:22:07.317008+0000","last_became_active":"2024-03-06T07:22:06.330404+0000","last_became_peered":"2024-03-06T07:22:06.330404+0000","last_unstale":"2024-03-06T07:22:07.317008+0000","last_undegraded":"2024-03-06T07:22:07.317008+0000","last_fullsized":"2024-03-06T07:22:07.317008+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:36.139439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517204+0000","last_change":"2024-03-06T07:22:05.313908+0000","last_active":"2024-03-06T07:22:07.517204+0000","last_peered":"2024-03-06T07:22:07.517204+0000","last_clean":"2024-03-06T07:22:07.517204+0000","last_became_active":"2024-03-06T07:22:05.313638+0000","last_became_peered":"2024-03-06T07:22:05.313638+0000","last_unstale":"2024-03-06T07:22:07.517204+0000","last_undegraded":"2024-03-06T07:22:07.517204+0000","last_fullsized":"2024-03-06T07:22:07.517204+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:46.701129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517266+0000","last_change":"2024-03-06T07:22:06.320596+0000","last_active":"2024-03-06T07:22:07.517266+0000","last_peered":"2024-03-06T07:22:07.517266+0000","last_clean":"2024-03-06T07:22:07.517266+0000","last_became_active":"2024-03-06T07:22:06.319430+0000","last_became_peered":"2024-03-06T07:22:06.319430+0000","last_unstale":"2024-03-06T07:22:07.517266+0000","last_undegraded":"2024-03-06T07:22:07.517266+0000","last_fullsized":"2024-03-06T07:22:07.517266+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:49.581292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517316+0000","last_change":"2024-03-06T07:22:05.320922+0000","last_active":"2024-03-06T07:22:07.517316+0000","last_peered":"2024-03-06T07:22:07.517316+0000","last_clean":"2024-03-06T07:22:07.517316+0000","last_became_active":"2024-03-06T07:22:05.320638+0000","last_became_peered":"2024-03-06T07:22:05.320638+0000","last_unstale":"2024-03-06T07:22:07.517316+0000","last_undegraded":"2024-03-06T07:22:07.517316+0000","last_fullsized":"2024-03-06T07:22:07.517316+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:13.798736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316765+0000","last_change":"2024-03-06T07:22:06.335202+0000","last_active":"2024-03-06T07:22:07.316765+0000","last_peered":"2024-03-06T07:22:07.316765+0000","last_clean":"2024-03-06T07:22:07.316765+0000","last_became_active":"2024-03-06T07:22:06.334922+0000","last_became_peered":"2024-03-06T07:22:06.334922+0000","last_unstale":"2024-03-06T07:22:07.316765+0000","last_undegraded":"2024-03-06T07:22:07.316765+0000","last_fullsized":"2024-03-06T07:22:07.316765+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:01.204740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315003+0000","last_change":"2024-03-06T07:22:05.326648+0000","last_active":"2024-03-06T07:22:07.315003+0000","last_peered":"2024-03-06T07:22:07.315003+0000","last_clean":"2024-03-06T07:22:07.315003+0000","last_became_active":"2024-03-06T07:22:05.326423+0000","last_became_peered":"2024-03-06T07:22:05.326423+0000","last_unstale":"2024-03-06T07:22:07.315003+0000","last_undegraded":"2024-03-06T07:22:07.315003+0000","last_fullsized":"2024-03-06T07:22:07.315003+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:04.675657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517225+0000","last_change":"2024-03-06T07:22:06.326561+0000","last_active":"2024-03-06T07:22:07.517225+0000","last_peered":"2024-03-06T07:22:07.517225+0000","last_clean":"2024-03-06T07:22:07.517225+0000","last_became_active":"2024-03-06T07:22:06.326295+0000","last_became_peered":"2024-03-06T07:22:06.326295+0000","last_unstale":"2024-03-06T07:22:07.517225+0000","last_undegraded":"2024-03-06T07:22:07.517225+0000","last_fullsized":"2024-03-06T07:22:07.517225+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:36.941825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316569+0000","last_change":"2024-03-06T07:22:05.319695+0000","last_active":"2024-03-06T07:22:07.316569+0000","last_peered":"2024-03-06T07:22:07.316569+0000","last_clean":"2024-03-06T07:22:07.316569+0000","last_became_active":"2024-03-06T07:22:05.319410+0000","last_became_peered":"2024-03-06T07:22:05.319410+0000","last_unstale":"2024-03-06T07:22:07.316569+0000","last_undegraded":"2024-03-06T07:22:07.316569+0000","last_fullsized":"2024-03-06T07:22:07.316569+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:06.303374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516554+0000","last_change":"2024-03-06T07:22:06.314348+0000","last_active":"2024-03-06T07:22:07.516554+0000","last_peered":"2024-03-06T07:22:07.516554+0000","last_clean":"2024-03-06T07:22:07.516554+0000","last_became_active":"2024-03-06T07:22:06.314044+0000","last_became_peered":"2024-03-06T07:22:06.314044+0000","last_unstale":"2024-03-06T07:22:07.516554+0000","last_undegraded":"2024-03-06T07:22:07.516554+0000","last_fullsized":"2024-03-06T07:22:07.516554+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:02.558815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517557+0000","last_change":"2024-03-06T07:22:05.321049+0000","last_active":"2024-03-06T07:22:07.517557+0000","last_peered":"2024-03-06T07:22:07.517557+0000","last_clean":"2024-03-06T07:22:07.517557+0000","last_became_active":"2024-03-06T07:22:05.320696+0000","last_became_peered":"2024-03-06T07:22:05.320696+0000","last_unstale":"2024-03-06T07:22:07.517557+0000","last_undegraded":"2024-03-06T07:22:07.517557+0000","last_fullsized":"2024-03-06T07:22:07.517557+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:26.446709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315896+0000","last_change":"2024-03-06T07:22:06.330920+0000","last_active":"2024-03-06T07:22:07.315896+0000","last_peered":"2024-03-06T07:22:07.315896+0000","last_clean":"2024-03-06T07:22:07.315896+0000","last_became_active":"2024-03-06T07:22:06.330331+0000","last_became_peered":"2024-03-06T07:22:06.330331+0000","last_unstale":"2024-03-06T07:22:07.315896+0000","last_undegraded":"2024-03-06T07:22:07.315896+0000","last_fullsized":"2024-03-06T07:22:07.315896+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:18.720650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":17,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317000+0000","last_change":"2024-03-06T07:22:05.321995+0000","last_active":"2024-03-06T07:22:07.317000+0000","last_peered":"2024-03-06T07:22:07.317000+0000","last_clean":"2024-03-06T07:22:07.317000+0000","last_became_active":"2024-03-06T07:22:05.321734+0000","last_became_peered":"2024-03-06T07:22:05.321734+0000","last_unstale":"2024-03-06T07:22:07.317000+0000","last_undegraded":"2024-03-06T07:22:07.317000+0000","last_fullsized":"2024-03-06T07:22:07.317000+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:40.725817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517991+0000","last_change":"2024-03-06T07:22:06.308003+0000","last_active":"2024-03-06T07:22:07.517991+0000","last_peered":"2024-03-06T07:22:07.517991+0000","last_clean":"2024-03-06T07:22:07.517991+0000","last_became_active":"2024-03-06T07:22:06.307696+0000","last_became_peered":"2024-03-06T07:22:06.307696+0000","last_unstale":"2024-03-06T07:22:07.517991+0000","last_undegraded":"2024-03-06T07:22:07.517991+0000","last_fullsized":"2024-03-06T07:22:07.517991+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:20.550343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":20,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517060+0000","last_change":"2024-03-06T07:22:05.313388+0000","last_active":"2024-03-06T07:22:07.517060+0000","last_peered":"2024-03-06T07:22:07.517060+0000","last_clean":"2024-03-06T07:22:07.517060+0000","last_became_active":"2024-03-06T07:22:05.312146+0000","last_became_peered":"2024-03-06T07:22:05.312146+0000","last_unstale":"2024-03-06T07:22:07.517060+0000","last_undegraded":"2024-03-06T07:22:07.517060+0000","last_fullsized":"2024-03-06T07:22:07.517060+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:16.630462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517645+0000","last_change":"2024-03-06T07:22:06.324830+0000","last_active":"2024-03-06T07:22:07.517645+0000","last_peered":"2024-03-06T07:22:07.517645+0000","last_clean":"2024-03-06T07:22:07.517645+0000","last_became_active":"2024-03-06T07:22:06.324574+0000","last_became_peered":"2024-03-06T07:22:06.324574+0000","last_unstale":"2024-03-06T07:22:07.517645+0000","last_undegraded":"2024-03-06T07:22:07.517645+0000","last_fullsized":"2024-03-06T07:22:07.517645+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:22.443979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315403+0000","last_change":"2024-03-06T07:22:05.326946+0000","last_active":"2024-03-06T07:22:07.315403+0000","last_peered":"2024-03-06T07:22:07.315403+0000","last_clean":"2024-03-06T07:22:07.315403+0000","last_became_active":"2024-03-06T07:22:05.326675+0000","last_became_peered":"2024-03-06T07:22:05.326675+0000","last_unstale":"2024-03-06T07:22:07.315403+0000","last_undegraded":"2024-03-06T07:22:07.315403+0000","last_fullsized":"2024-03-06T07:22:07.315403+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:35.936903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315948+0000","last_change":"2024-03-06T07:22:06.329315+0000","last_active":"2024-03-06T07:22:07.315948+0000","last_peered":"2024-03-06T07:22:07.315948+0000","last_clean":"2024-03-06T07:22:07.315948+0000","last_became_active":"2024-03-06T07:22:06.322815+0000","last_became_peered":"2024-03-06T07:22:06.322815+0000","last_unstale":"2024-03-06T07:22:07.315948+0000","last_undegraded":"2024-03-06T07:22:07.315948+0000","last_fullsized":"2024-03-06T07:22:07.315948+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:15.214675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516686+0000","last_change":"2024-03-06T07:22:05.324367+0000","last_active":"2024-03-06T07:22:07.516686+0000","last_peered":"2024-03-06T07:22:07.516686+0000","last_clean":"2024-03-06T07:22:07.516686+0000","last_became_active":"2024-03-06T07:22:05.324086+0000","last_became_peered":"2024-03-06T07:22:05.324086+0000","last_unstale":"2024-03-06T07:22:07.516686+0000","last_undegraded":"2024-03-06T07:22:07.516686+0000","last_fullsized":"2024-03-06T07:22:07.516686+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:31.061844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517116+0000","last_change":"2024-03-06T07:22:06.322371+0000","last_active":"2024-03-06T07:22:07.517116+0000","last_peered":"2024-03-06T07:22:07.517116+0000","last_clean":"2024-03-06T07:22:07.517116+0000","last_became_active":"2024-03-06T07:22:06.322099+0000","last_became_peered":"2024-03-06T07:22:06.322099+0000","last_unstale":"2024-03-06T07:22:07.517116+0000","last_undegraded":"2024-03-06T07:22:07.517116+0000","last_fullsized":"2024-03-06T07:22:07.517116+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:02.237473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317064+0000","last_change":"2024-03-06T07:22:05.320305+0000","last_active":"2024-03-06T07:22:07.317064+0000","last_peered":"2024-03-06T07:22:07.317064+0000","last_clean":"2024-03-06T07:22:07.317064+0000","last_became_active":"2024-03-06T07:22:05.319970+0000","last_became_peered":"2024-03-06T07:22:05.319970+0000","last_unstale":"2024-03-06T07:22:07.317064+0000","last_undegraded":"2024-03-06T07:22:07.317064+0000","last_fullsized":"2024-03-06T07:22:07.317064+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:01.935488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316347+0000","last_change":"2024-03-06T07:22:06.318628+0000","last_active":"2024-03-06T07:22:07.316347+0000","last_peered":"2024-03-06T07:22:07.316347+0000","last_clean":"2024-03-06T07:22:07.316347+0000","last_became_active":"2024-03-06T07:22:06.318324+0000","last_became_peered":"2024-03-06T07:22:06.318324+0000","last_unstale":"2024-03-06T07:22:07.316347+0000","last_undegraded":"2024-03-06T07:22:07.316347+0000","last_fullsized":"2024-03-06T07:22:07.316347+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:09.827959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":22,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516634+0000","last_change":"2024-03-06T07:22:05.316578+0000","last_active":"2024-03-06T07:22:07.516634+0000","last_peered":"2024-03-06T07:22:07.516634+0000","last_clean":"2024-03-06T07:22:07.516634+0000","last_became_active":"2024-03-06T07:22:05.316285+0000","last_became_peered":"2024-03-06T07:22:05.316285+0000","last_unstale":"2024-03-06T07:22:07.516634+0000","last_undegraded":"2024-03-06T07:22:07.516634+0000","last_fullsized":"2024-03-06T07:22:07.516634+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:21.439057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516581+0000","last_change":"2024-03-06T07:22:06.328590+0000","last_active":"2024-03-06T07:22:07.516581+0000","last_peered":"2024-03-06T07:22:07.516581+0000","last_clean":"2024-03-06T07:22:07.516581+0000","last_became_active":"2024-03-06T07:22:06.327594+0000","last_became_peered":"2024-03-06T07:22:06.327594+0000","last_unstale":"2024-03-06T07:22:07.516581+0000","last_undegraded":"2024-03-06T07:22:07.516581+0000","last_fullsized":"2024-03-06T07:22:07.516581+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:46.981658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316517+0000","last_change":"2024-03-06T07:22:05.325528+0000","last_active":"2024-03-06T07:22:07.316517+0000","last_peered":"2024-03-06T07:22:07.316517+0000","last_clean":"2024-03-06T07:22:07.316517+0000","last_became_active":"2024-03-06T07:22:05.325254+0000","last_became_peered":"2024-03-06T07:22:05.325254+0000","last_unstale":"2024-03-06T07:22:07.316517+0000","last_undegraded":"2024-03-06T07:22:07.316517+0000","last_fullsized":"2024-03-06T07:22:07.316517+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:44.392826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316730+0000","last_change":"2024-03-06T07:22:06.320394+0000","last_active":"2024-03-06T07:22:07.316730+0000","last_peered":"2024-03-06T07:22:07.316730+0000","last_clean":"2024-03-06T07:22:07.316730+0000","last_became_active":"2024-03-06T07:22:06.320135+0000","last_became_peered":"2024-03-06T07:22:06.320135+0000","last_unstale":"2024-03-06T07:22:07.316730+0000","last_undegraded":"2024-03-06T07:22:07.316730+0000","last_fullsized":"2024-03-06T07:22:07.316730+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:34.766315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516581+0000","last_change":"2024-03-06T07:22:05.314843+0000","last_active":"2024-03-06T07:22:07.516581+0000","last_peered":"2024-03-06T07:22:07.516581+0000","last_clean":"2024-03-06T07:22:07.516581+0000","last_became_active":"2024-03-06T07:22:05.314567+0000","last_became_peered":"2024-03-06T07:22:05.314567+0000","last_unstale":"2024-03-06T07:22:07.516581+0000","last_undegraded":"2024-03-06T07:22:07.516581+0000","last_fullsized":"2024-03-06T07:22:07.516581+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:23.891871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517144+0000","last_change":"2024-03-06T07:22:06.321026+0000","last_active":"2024-03-06T07:22:07.517144+0000","last_peered":"2024-03-06T07:22:07.517144+0000","last_clean":"2024-03-06T07:22:07.517144+0000","last_became_active":"2024-03-06T07:22:06.320749+0000","last_became_peered":"2024-03-06T07:22:06.320749+0000","last_unstale":"2024-03-06T07:22:07.517144+0000","last_undegraded":"2024-03-06T07:22:07.517144+0000","last_fullsized":"2024-03-06T07:22:07.517144+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:59.907792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517116+0000","last_change":"2024-03-06T07:22:05.325044+0000","last_active":"2024-03-06T07:22:07.517116+0000","last_peered":"2024-03-06T07:22:07.517116+0000","last_clean":"2024-03-06T07:22:07.517116+0000","last_became_active":"2024-03-06T07:22:05.324731+0000","last_became_peered":"2024-03-06T07:22:05.324731+0000","last_unstale":"2024-03-06T07:22:07.517116+0000","last_undegraded":"2024-03-06T07:22:07.517116+0000","last_fullsized":"2024-03-06T07:22:07.517116+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:50.594146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316958+0000","last_change":"2024-03-06T07:22:06.327972+0000","last_active":"2024-03-06T07:22:07.316958+0000","last_peered":"2024-03-06T07:22:07.316958+0000","last_clean":"2024-03-06T07:22:07.316958+0000","last_became_active":"2024-03-06T07:22:06.327676+0000","last_became_peered":"2024-03-06T07:22:06.327676+0000","last_unstale":"2024-03-06T07:22:07.316958+0000","last_undegraded":"2024-03-06T07:22:07.316958+0000","last_fullsized":"2024-03-06T07:22:07.316958+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:53.049705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517228+0000","last_change":"2024-03-06T07:22:05.318191+0000","last_active":"2024-03-06T07:22:07.517228+0000","last_peered":"2024-03-06T07:22:07.517228+0000","last_clean":"2024-03-06T07:22:07.517228+0000","last_became_active":"2024-03-06T07:22:05.317984+0000","last_became_peered":"2024-03-06T07:22:05.317984+0000","last_unstale":"2024-03-06T07:22:07.517228+0000","last_undegraded":"2024-03-06T07:22:07.517228+0000","last_fullsized":"2024-03-06T07:22:07.517228+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:48.457911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517619+0000","last_change":"2024-03-06T07:22:06.317413+0000","last_active":"2024-03-06T07:22:07.517619+0000","last_peered":"2024-03-06T07:22:07.517619+0000","last_clean":"2024-03-06T07:22:07.517619+0000","last_became_active":"2024-03-06T07:22:06.317126+0000","last_became_peered":"2024-03-06T07:22:06.317126+0000","last_unstale":"2024-03-06T07:22:07.517619+0000","last_undegraded":"2024-03-06T07:22:07.517619+0000","last_fullsized":"2024-03-06T07:22:07.517619+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:17.635384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316869+0000","last_change":"2024-03-06T07:22:05.320901+0000","last_active":"2024-03-06T07:22:07.316869+0000","last_peered":"2024-03-06T07:22:07.316869+0000","last_clean":"2024-03-06T07:22:07.316869+0000","last_became_active":"2024-03-06T07:22:05.320579+0000","last_became_peered":"2024-03-06T07:22:05.320579+0000","last_unstale":"2024-03-06T07:22:07.316869+0000","last_undegraded":"2024-03-06T07:22:07.316869+0000","last_fullsized":"2024-03-06T07:22:07.316869+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:46.701129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517070+0000","last_change":"2024-03-06T07:22:06.321030+0000","last_active":"2024-03-06T07:22:07.517070+0000","last_peered":"2024-03-06T07:22:07.517070+0000","last_clean":"2024-03-06T07:22:07.517070+0000","last_became_active":"2024-03-06T07:22:06.320686+0000","last_became_peered":"2024-03-06T07:22:06.320686+0000","last_unstale":"2024-03-06T07:22:07.517070+0000","last_undegraded":"2024-03-06T07:22:07.517070+0000","last_fullsized":"2024-03-06T07:22:07.517070+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:41.730739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516237+0000","last_change":"2024-03-06T07:22:05.316761+0000","last_active":"2024-03-06T07:22:07.516237+0000","last_peered":"2024-03-06T07:22:07.516237+0000","last_clean":"2024-03-06T07:22:07.516237+0000","last_became_active":"2024-03-06T07:22:05.316088+0000","last_became_peered":"2024-03-06T07:22:05.316088+0000","last_unstale":"2024-03-06T07:22:07.516237+0000","last_undegraded":"2024-03-06T07:22:07.516237+0000","last_fullsized":"2024-03-06T07:22:07.516237+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:35.134517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316158+0000","last_change":"2024-03-06T07:22:06.313796+0000","last_active":"2024-03-06T07:22:07.316158+0000","last_peered":"2024-03-06T07:22:07.316158+0000","last_clean":"2024-03-06T07:22:07.316158+0000","last_became_active":"2024-03-06T07:22:06.313534+0000","last_became_peered":"2024-03-06T07:22:06.313534+0000","last_unstale":"2024-03-06T07:22:07.316158+0000","last_undegraded":"2024-03-06T07:22:07.316158+0000","last_fullsized":"2024-03-06T07:22:07.316158+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:34.284715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315462+0000","last_change":"2024-03-06T07:22:05.324175+0000","last_active":"2024-03-06T07:22:07.315462+0000","last_peered":"2024-03-06T07:22:07.315462+0000","last_clean":"2024-03-06T07:22:07.315462+0000","last_became_active":"2024-03-06T07:22:05.323587+0000","last_became_peered":"2024-03-06T07:22:05.323587+0000","last_unstale":"2024-03-06T07:22:07.315462+0000","last_undegraded":"2024-03-06T07:22:07.315462+0000","last_fullsized":"2024-03-06T07:22:07.315462+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:01.935488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317313+0000","last_change":"2024-03-06T07:22:06.325375+0000","last_active":"2024-03-06T07:22:07.317313+0000","last_peered":"2024-03-06T07:22:07.317313+0000","last_clean":"2024-03-06T07:22:07.317313+0000","last_became_active":"2024-03-06T07:22:06.325108+0000","last_became_peered":"2024-03-06T07:22:06.325108+0000","last_unstale":"2024-03-06T07:22:07.317313+0000","last_undegraded":"2024-03-06T07:22:07.317313+0000","last_fullsized":"2024-03-06T07:22:07.317313+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:36.139439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516789+0000","last_change":"2024-03-06T07:22:05.317252+0000","last_active":"2024-03-06T07:22:07.516789+0000","last_peered":"2024-03-06T07:22:07.516789+0000","last_clean":"2024-03-06T07:22:07.516789+0000","last_became_active":"2024-03-06T07:22:05.316964+0000","last_became_peered":"2024-03-06T07:22:05.316964+0000","last_unstale":"2024-03-06T07:22:07.516789+0000","last_undegraded":"2024-03-06T07:22:07.516789+0000","last_fullsized":"2024-03-06T07:22:07.516789+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:34.517096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516532+0000","last_change":"2024-03-06T07:22:06.325564+0000","last_active":"2024-03-06T07:22:07.516532+0000","last_peered":"2024-03-06T07:22:07.516532+0000","last_clean":"2024-03-06T07:22:07.516532+0000","last_became_active":"2024-03-06T07:22:06.325298+0000","last_became_peered":"2024-03-06T07:22:06.325298+0000","last_unstale":"2024-03-06T07:22:07.516532+0000","last_undegraded":"2024-03-06T07:22:07.516532+0000","last_fullsized":"2024-03-06T07:22:07.516532+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:51.508838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517050+0000","last_change":"2024-03-06T07:22:05.326276+0000","last_active":"2024-03-06T07:22:07.517050+0000","last_peered":"2024-03-06T07:22:07.517050+0000","last_clean":"2024-03-06T07:22:07.517050+0000","last_became_active":"2024-03-06T07:22:05.325993+0000","last_became_peered":"2024-03-06T07:22:05.325993+0000","last_unstale":"2024-03-06T07:22:07.517050+0000","last_undegraded":"2024-03-06T07:22:07.517050+0000","last_fullsized":"2024-03-06T07:22:07.517050+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:19.366175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316881+0000","last_change":"2024-03-06T07:22:06.323310+0000","last_active":"2024-03-06T07:22:07.316881+0000","last_peered":"2024-03-06T07:22:07.316881+0000","last_clean":"2024-03-06T07:22:07.316881+0000","last_became_active":"2024-03-06T07:22:06.322311+0000","last_became_peered":"2024-03-06T07:22:06.322311+0000","last_unstale":"2024-03-06T07:22:07.316881+0000","last_undegraded":"2024-03-06T07:22:07.316881+0000","last_fullsized":"2024-03-06T07:22:07.316881+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:01.204740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.317078+0000","last_change":"2024-03-06T07:22:05.319638+0000","last_active":"2024-03-06T07:22:07.317078+0000","last_peered":"2024-03-06T07:22:07.317078+0000","last_clean":"2024-03-06T07:22:07.317078+0000","last_became_active":"2024-03-06T07:22:05.319307+0000","last_became_peered":"2024-03-06T07:22:05.319307+0000","last_unstale":"2024-03-06T07:22:07.317078+0000","last_undegraded":"2024-03-06T07:22:07.317078+0000","last_fullsized":"2024-03-06T07:22:07.317078+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:19.853691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516496+0000","last_change":"2024-03-06T07:22:06.315689+0000","last_active":"2024-03-06T07:22:07.516496+0000","last_peered":"2024-03-06T07:22:07.516496+0000","last_clean":"2024-03-06T07:22:07.516496+0000","last_became_active":"2024-03-06T07:22:06.315395+0000","last_became_peered":"2024-03-06T07:22:06.315395+0000","last_unstale":"2024-03-06T07:22:07.516496+0000","last_undegraded":"2024-03-06T07:22:07.516496+0000","last_fullsized":"2024-03-06T07:22:07.516496+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:09.827959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315324+0000","last_change":"2024-03-06T07:22:05.319307+0000","last_active":"2024-03-06T07:22:07.315324+0000","last_peered":"2024-03-06T07:22:07.315324+0000","last_clean":"2024-03-06T07:22:07.315324+0000","last_became_active":"2024-03-06T07:22:05.318839+0000","last_became_peered":"2024-03-06T07:22:05.318839+0000","last_unstale":"2024-03-06T07:22:07.315324+0000","last_undegraded":"2024-03-06T07:22:07.315324+0000","last_fullsized":"2024-03-06T07:22:07.315324+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:40.725817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.315277+0000","last_change":"2024-03-06T07:22:06.331443+0000","last_active":"2024-03-06T07:22:07.315277+0000","last_peered":"2024-03-06T07:22:07.315277+0000","last_clean":"2024-03-06T07:22:07.315277+0000","last_became_active":"2024-03-06T07:22:06.331189+0000","last_became_peered":"2024-03-06T07:22:06.331189+0000","last_unstale":"2024-03-06T07:22:07.315277+0000","last_undegraded":"2024-03-06T07:22:07.315277+0000","last_fullsized":"2024-03-06T07:22:07.315277+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:34.295379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516542+0000","last_change":"2024-03-06T07:22:05.315558+0000","last_active":"2024-03-06T07:22:07.516542+0000","last_peered":"2024-03-06T07:22:07.516542+0000","last_clean":"2024-03-06T07:22:07.516542+0000","last_became_active":"2024-03-06T07:22:05.315263+0000","last_became_peered":"2024-03-06T07:22:05.315263+0000","last_unstale":"2024-03-06T07:22:07.516542+0000","last_undegraded":"2024-03-06T07:22:07.516542+0000","last_fullsized":"2024-03-06T07:22:07.516542+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:01.553893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517825+0000","last_change":"2024-03-06T07:22:06.324288+0000","last_active":"2024-03-06T07:22:07.517825+0000","last_peered":"2024-03-06T07:22:07.517825+0000","last_clean":"2024-03-06T07:22:07.517825+0000","last_became_active":"2024-03-06T07:22:06.323925+0000","last_became_peered":"2024-03-06T07:22:06.323925+0000","last_unstale":"2024-03-06T07:22:07.517825+0000","last_undegraded":"2024-03-06T07:22:07.517825+0000","last_fullsized":"2024-03-06T07:22:07.517825+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:49.900223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516342+0000","last_change":"2024-03-06T07:22:05.322782+0000","last_active":"2024-03-06T07:22:07.516342+0000","last_peered":"2024-03-06T07:22:07.516342+0000","last_clean":"2024-03-06T07:22:07.516342+0000","last_became_active":"2024-03-06T07:22:05.322484+0000","last_became_peered":"2024-03-06T07:22:05.322484+0000","last_unstale":"2024-03-06T07:22:07.516342+0000","last_undegraded":"2024-03-06T07:22:07.516342+0000","last_fullsized":"2024-03-06T07:22:07.516342+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:04.727017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316019+0000","last_change":"2024-03-06T07:22:06.321965+0000","last_active":"2024-03-06T07:22:07.316019+0000","last_peered":"2024-03-06T07:22:07.316019+0000","last_clean":"2024-03-06T07:22:07.316019+0000","last_became_active":"2024-03-06T07:22:06.320167+0000","last_became_peered":"2024-03-06T07:22:06.320167+0000","last_unstale":"2024-03-06T07:22:07.316019+0000","last_undegraded":"2024-03-06T07:22:07.316019+0000","last_fullsized":"2024-03-06T07:22:07.316019+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:15.334381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517279+0000","last_change":"2024-03-06T07:22:05.312784+0000","last_active":"2024-03-06T07:22:07.517279+0000","last_peered":"2024-03-06T07:22:07.517279+0000","last_clean":"2024-03-06T07:22:07.517279+0000","last_became_active":"2024-03-06T07:22:05.312472+0000","last_became_peered":"2024-03-06T07:22:05.312472+0000","last_unstale":"2024-03-06T07:22:07.517279+0000","last_undegraded":"2024-03-06T07:22:07.517279+0000","last_fullsized":"2024-03-06T07:22:07.517279+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:43.288752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.517339+0000","last_change":"2024-03-06T07:22:06.319779+0000","last_active":"2024-03-06T07:22:07.517339+0000","last_peered":"2024-03-06T07:22:07.517339+0000","last_clean":"2024-03-06T07:22:07.517339+0000","last_became_active":"2024-03-06T07:22:06.319415+0000","last_became_peered":"2024-03-06T07:22:06.319415+0000","last_unstale":"2024-03-06T07:22:07.517339+0000","last_undegraded":"2024-03-06T07:22:07.517339+0000","last_fullsized":"2024-03-06T07:22:07.517339+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:20.884386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516711+0000","last_change":"2024-03-06T07:22:05.322202+0000","last_active":"2024-03-06T07:22:07.516711+0000","last_peered":"2024-03-06T07:22:07.516711+0000","last_clean":"2024-03-06T07:22:07.516711+0000","last_became_active":"2024-03-06T07:22:05.321896+0000","last_became_peered":"2024-03-06T07:22:05.321896+0000","last_unstale":"2024-03-06T07:22:07.516711+0000","last_undegraded":"2024-03-06T07:22:07.516711+0000","last_fullsized":"2024-03-06T07:22:07.516711+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:44.392826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.516613+0000","last_change":"2024-03-06T07:22:06.328678+0000","last_active":"2024-03-06T07:22:07.516613+0000","last_peered":"2024-03-06T07:22:07.516613+0000","last_clean":"2024-03-06T07:22:07.516613+0000","last_became_active":"2024-03-06T07:22:06.328362+0000","last_became_peered":"2024-03-06T07:22:06.328362+0000","last_unstale":"2024-03-06T07:22:07.516613+0000","last_undegraded":"2024-03-06T07:22:07.516613+0000","last_fullsized":"2024-03-06T07:22:07.516613+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:27.383059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"52'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316130+0000","last_change":"2024-03-06T07:22:05.326257+0000","last_active":"2024-03-06T07:22:07.316130+0000","last_peered":"2024-03-06T07:22:07.316130+0000","last_clean":"2024-03-06T07:22:07.316130+0000","last_became_active":"2024-03-06T07:22:05.325920+0000","last_became_peered":"2024-03-06T07:22:05.325920+0000","last_unstale":"2024-03-06T07:22:07.316130+0000","last_undegraded":"2024-03-06T07:22:07.316130+0000","last_fullsized":"2024-03-06T07:22:07.316130+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:58.902870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":230,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.316623+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:22:07.316623+0000","last_peered":"2024-03-06T07:22:07.316623+0000","last_clean":"2024-03-06T07:22:07.316623+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:22:07.316623+0000","last_undegraded":"2024-03-06T07:22:07.316623+0000","last_fullsized":"2024-03-06T07:22:07.316623+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":868021,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":278181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":618496,"data_stored":277529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":622592,"data_stored":278111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":634880,"data_stored":279163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31804,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":630784,"data_stored":279141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":626688,"data_stored":277689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":692224,"data_stored":867953,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:14.267 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:22:14.267 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:22:14.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.414+0000 7f0458155640 1 Processor -- start 2024-03-06T07:22:14.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.415+0000 7f0458155640 1 -- start start 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.416+0000 7f0458155640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0450156cd0 0x7f0450157120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.416+0000 7f0458155640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04500c3300 con 0x7f04500c6730 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.416+0000 7f0458155640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04500c35a0 con 0x7f0450152690 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.416+0000 7f0458155640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04500c3880 con 0x7f0450156cd0 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.416+0000 7f04566f2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0450156cd0 0x7f0450157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.416+0000 7f04556f0640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f04500c6730 0x7f04500c57d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51634/0 (socket says 172.21.15.1:51634) 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.416+0000 7f04556f0640 1 -- 172.21.15.1:0/1301065840 learned_addr learned my addr 172.21.15.1:0/1301065840 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:14.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.416+0000 7f04566f2640 1 -- 172.21.15.1:0/1301065840 >> v1:172.21.15.98:6789/0 conn(0x7f0450152690 legacy=0x7f0450155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.416+0000 7f04566f2640 1 -- 172.21.15.1:0/1301065840 >> v1:172.21.15.1:6789/0 conn(0x7f04500c6730 legacy=0x7f04500c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.416+0000 7f04566f2640 1 -- 172.21.15.1:0/1301065840 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04500c3b60 con 0x7f0450156cd0 2024-03-06T07:22:14.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.416+0000 7f0454eef640 1 -- 172.21.15.1:0/1301065840 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862066156 0 0) 0x7f04500c3300 con 0x7f04500c6730 2024-03-06T07:22:14.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.417+0000 7f04566f2640 1 --2- 172.21.15.1:0/1301065840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0450156cd0 0x7f0450157120 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0448009d90 tx=0x7f044802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a2411ee63d8122 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.417+0000 7f0454eef640 1 -- 172.21.15.1:0/1301065840 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0448039070 con 0x7f0450156cd0 2024-03-06T07:22:14.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.417+0000 7f0454eef640 1 -- 172.21.15.1:0/1301065840 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044802cc60 con 0x7f0450156cd0 2024-03-06T07:22:14.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.417+0000 7f0454eef640 1 -- 172.21.15.1:0/1301065840 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0448034660 con 0x7f0450156cd0 2024-03-06T07:22:14.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.418+0000 7f0458155640 1 -- 172.21.15.1:0/1301065840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0450156cd0 msgr2=0x7f0450157120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.418+0000 7f0458155640 1 --2- 172.21.15.1:0/1301065840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0450156cd0 0x7f0450157120 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0448009d90 tx=0x7f044802beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.418+0000 7f0458155640 1 -- 172.21.15.1:0/1301065840 shutdown_connections 2024-03-06T07:22:14.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.418+0000 7f0458155640 1 --2- 172.21.15.1:0/1301065840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0450156cd0 0x7f0450157120 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.418+0000 7f0458155640 1 -- 172.21.15.1:0/1301065840 >> 172.21.15.1:0/1301065840 conn(0x7f045006b410 msgr2=0x7f045006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.418+0000 7f0458155640 1 -- 172.21.15.1:0/1301065840 shutdown_connections 2024-03-06T07:22:14.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.418+0000 7f0458155640 1 -- 172.21.15.1:0/1301065840 wait complete. 2024-03-06T07:22:14.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.419+0000 7f0458155640 1 Processor -- start 2024-03-06T07:22:14.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.419+0000 7f0458155640 1 -- start start 2024-03-06T07:22:14.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f0458155640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04500c6730 0x7f0450151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f0458155640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0450152690 0x7f04500aa580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f0458155640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0450156cd0 0x7f04500aaac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f0458155640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04500c6060 con 0x7f04500c6730 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f0458155640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04500c5ee0 con 0x7f0450156cd0 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f0458155640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0450152a70 con 0x7f0450152690 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f0455ef1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04500c6730 0x7f0450151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f0455ef1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04500c6730 0x7f0450151010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37294/0 (socket says 172.21.15.1:37294) 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f0455ef1640 1 -- 172.21.15.1:0/4071783017 learned_addr learned my addr 172.21.15.1:0/4071783017 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f04566f2640 1 --2- 172.21.15.1:0/4071783017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0450156cd0 0x7f04500aaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f04556f0640 1 --2- 172.21.15.1:0/4071783017 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0450152690 0x7f04500aa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f0455ef1640 1 -- 172.21.15.1:0/4071783017 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0450152690 msgr2=0x7f04500aa580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f0455ef1640 1 --2- 172.21.15.1:0/4071783017 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0450152690 0x7f04500aa580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f0455ef1640 1 -- 172.21.15.1:0/4071783017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0450156cd0 msgr2=0x7f04500aaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f0455ef1640 1 --2- 172.21.15.1:0/4071783017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0450156cd0 0x7f04500aaac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f0455ef1640 1 -- 172.21.15.1:0/4071783017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045014f140 con 0x7f04500c6730 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f04566f2640 1 --2- 172.21.15.1:0/4071783017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0450156cd0 0x7f04500aaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f04556f0640 1 --2- 172.21.15.1:0/4071783017 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0450152690 0x7f04500aa580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f0455ef1640 1 --2- 172.21.15.1:0/4071783017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04500c6730 0x7f0450151010 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f044000c400 tx=0x7f044000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f04467fc640 1 -- 172.21.15.1:0/4071783017 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0440017070 con 0x7f04500c6730 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f04467fc640 1 -- 172.21.15.1:0/4071783017 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04400052d0 con 0x7f04500c6730 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f0458155640 1 -- 172.21.15.1:0/4071783017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04501ee230 con 0x7f04500c6730 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f04467fc640 1 -- 172.21.15.1:0/4071783017 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0440003830 con 0x7f04500c6730 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.420+0000 7f0458155640 1 -- 172.21.15.1:0/4071783017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0450151ab0 con 0x7f04500c6730 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.421+0000 7f0458155640 1 -- 172.21.15.1:0/4071783017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0450067fd0 con 0x7f04500c6730 2024-03-06T07:22:14.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.422+0000 7f04467fc640 1 -- 172.21.15.1:0/4071783017 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f044001a070 con 0x7f04500c6730 2024-03-06T07:22:14.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.423+0000 7f04467fc640 1 --2- 172.21.15.1:0/4071783017 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f043406b4d0 0x7f043406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.423+0000 7f04467fc640 1 -- 172.21.15.1:0/4071783017 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0440089660 con 0x7f04500c6730 2024-03-06T07:22:14.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.423+0000 7f04556f0640 1 --2- 172.21.15.1:0/4071783017 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f043406b4d0 0x7f043406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.425+0000 7f04556f0640 1 --2- 172.21.15.1:0/4071783017 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f043406b4d0 0x7f043406d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0450151e40 tx=0x7f044c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.428+0000 7f04467fc640 1 -- 172.21.15.1:0/4071783017 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04400583e0 con 0x7f04500c6730 2024-03-06T07:22:14.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.678+0000 7f0458155640 1 -- 172.21.15.1:0/4071783017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f04500c2c40 con 0x7f04500c6730 2024-03-06T07:22:14.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.680+0000 7f04467fc640 1 -- 172.21.15.1:0/4071783017 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7f0440087e20 con 0x7f04500c6730 2024-03-06T07:22:14.682 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-06T07:22:14.682 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng== 2024-03-06T07:22:14.682 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-06T07:22:14.682 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-06T07:22:14.683 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:22:14.683 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-06T07:22:14.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.682+0000 7f0458155640 1 -- 172.21.15.1:0/4071783017 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f043406b4d0 msgr2=0x7f043406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.682+0000 7f0458155640 1 --2- 172.21.15.1:0/4071783017 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f043406b4d0 0x7f043406d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0450151e40 tx=0x7f044c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:14.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.683+0000 7f0458155640 1 -- 172.21.15.1:0/4071783017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04500c6730 msgr2=0x7f0450151010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.683+0000 7f0458155640 1 --2- 172.21.15.1:0/4071783017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04500c6730 0x7f0450151010 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f044000c400 tx=0x7f044000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.683+0000 7f0458155640 1 -- 172.21.15.1:0/4071783017 shutdown_connections 2024-03-06T07:22:14.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.683+0000 7f0458155640 1 --2- 172.21.15.1:0/4071783017 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f043406b4d0 0x7f043406d990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.683+0000 7f0458155640 1 --2- 172.21.15.1:0/4071783017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0450156cd0 0x7f04500aaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.683+0000 7f0458155640 1 --2- 172.21.15.1:0/4071783017 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0450152690 0x7f04500aa580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.683+0000 7f0458155640 1 --2- 172.21.15.1:0/4071783017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04500c6730 0x7f0450151010 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.683+0000 7f0458155640 1 -- 172.21.15.1:0/4071783017 >> 172.21.15.1:0/4071783017 conn(0x7f045006b410 msgr2=0x7f045006b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.683+0000 7f0458155640 1 -- 172.21.15.1:0/4071783017 shutdown_connections 2024-03-06T07:22:14.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.683+0000 7f0458155640 1 -- 172.21.15.1:0/4071783017 wait complete. 2024-03-06T07:22:14.695 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:14.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.796+0000 7f02a13dd640 1 Processor -- start 2024-03-06T07:22:14.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.796+0000 7f02a13dd640 1 -- start start 2024-03-06T07:22:14.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.797+0000 7f02a13dd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f029c0c8cd0 0x7f029c0c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.797+0000 7f02a13dd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f029c155b70 0x7f029c159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.797+0000 7f02a13dd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029c0c2b90 con 0x7f029c0c3260 2024-03-06T07:22:14.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.797+0000 7f02a13dd640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029c156020 con 0x7f029c155b70 2024-03-06T07:22:14.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.797+0000 7f02a13dd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029c0c37b0 con 0x7f029c0c8cd0 2024-03-06T07:22:14.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.797+0000 7f029b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f029c155b70 0x7f029c159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.797+0000 7f029b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f029c155b70 0x7f029c159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37310/0 (socket says 172.21.15.1:37310) 2024-03-06T07:22:14.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.797+0000 7f029b7fe640 1 -- 172.21.15.1:0/2694164479 learned_addr learned my addr 172.21.15.1:0/2694164479 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:14.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.797+0000 7f029affd640 1 --2- 172.21.15.1:0/2694164479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f029c0c8cd0 0x7f029c0c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.797+0000 7f029b7fe640 1 -- 172.21.15.1:0/2694164479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f029c0c8cd0 msgr2=0x7f029c0c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:14.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.797+0000 7f029b7fe640 1 --2- 172.21.15.1:0/2694164479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f029c0c8cd0 0x7f029c0c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.797+0000 7f029b7fe640 1 -- 172.21.15.1:0/2694164479 >> v1:172.21.15.98:6789/0 conn(0x7f029c0c3260 legacy=0x7f029c0c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:14.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.797+0000 7f029b7fe640 1 -- 172.21.15.1:0/2694164479 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c0c3a90 con 0x7f029c155b70 2024-03-06T07:22:14.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.797+0000 7f029affd640 1 --2- 172.21.15.1:0/2694164479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f029c0c8cd0 0x7f029c0c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:14.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.798+0000 7f029b7fe640 1 --2- 172.21.15.1:0/2694164479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c155b70 0x7f029c159010 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0294009e50 tx=0x7f029402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fd2ea9181019b09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.798+0000 7f0299ffb640 1 -- 172.21.15.1:0/2694164479 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029402c9e0 con 0x7f029c155b70 2024-03-06T07:22:14.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.798+0000 7f0299ffb640 1 -- 172.21.15.1:0/2694164479 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029402cb80 con 0x7f029c155b70 2024-03-06T07:22:14.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.798+0000 7f0299ffb640 1 -- 172.21.15.1:0/2694164479 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02940343f0 con 0x7f029c155b70 2024-03-06T07:22:14.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.799+0000 7f02a13dd640 1 -- 172.21.15.1:0/2694164479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c155b70 msgr2=0x7f029c159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.799+0000 7f02a13dd640 1 --2- 172.21.15.1:0/2694164479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c155b70 0x7f029c159010 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0294009e50 tx=0x7f029402be40 comp rx=0 tx=0).stop 2024-03-06T07:22:14.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.799+0000 7f02a13dd640 1 -- 172.21.15.1:0/2694164479 shutdown_connections 2024-03-06T07:22:14.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.799+0000 7f02a13dd640 1 --2- 172.21.15.1:0/2694164479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c155b70 0x7f029c159010 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.799+0000 7f02a13dd640 1 --2- 172.21.15.1:0/2694164479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f029c0c8cd0 0x7f029c0c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.799+0000 7f02a13dd640 1 -- 172.21.15.1:0/2694164479 >> 172.21.15.1:0/2694164479 conn(0x7f029c06b3e0 msgr2=0x7f029c06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.799+0000 7f02a13dd640 1 -- 172.21.15.1:0/2694164479 shutdown_connections 2024-03-06T07:22:14.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.799+0000 7f02a13dd640 1 -- 172.21.15.1:0/2694164479 wait complete. 2024-03-06T07:22:14.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.800+0000 7f02a13dd640 1 Processor -- start 2024-03-06T07:22:14.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.800+0000 7f02a13dd640 1 -- start start 2024-03-06T07:22:14.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f02a13dd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f029c0c3260 0x7f029c14fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f02a13dd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f029c0c8cd0 0x7f029c153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f02a13dd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c155b70 0x7f029c1ee560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f02a13dd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c159550 con 0x7f029c155b70 2024-03-06T07:22:14.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f02a13dd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c158af0 con 0x7f029c0c3260 2024-03-06T07:22:14.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f02a13dd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029c0c8560 con 0x7f029c0c8cd0 2024-03-06T07:22:14.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f029b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c155b70 0x7f029c1ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f029b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c155b70 0x7f029c1ee560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37316/0 (socket says 172.21.15.1:37316) 2024-03-06T07:22:14.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f029b7fe640 1 -- 172.21.15.1:0/889536643 learned_addr learned my addr 172.21.15.1:0/889536643 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:14.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f029a7fc640 1 --2- 172.21.15.1:0/889536643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f029c0c8cd0 0x7f029c153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f029affd640 1 --2- 172.21.15.1:0/889536643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f029c0c3260 0x7f029c14fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f029b7fe640 1 -- 172.21.15.1:0/889536643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f029c0c8cd0 msgr2=0x7f029c153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f029b7fe640 1 --2- 172.21.15.1:0/889536643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f029c0c8cd0 0x7f029c153290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f029b7fe640 1 -- 172.21.15.1:0/889536643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f029c0c3260 msgr2=0x7f029c14fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f029b7fe640 1 --2- 172.21.15.1:0/889536643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f029c0c3260 0x7f029c14fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f029b7fe640 1 -- 172.21.15.1:0/889536643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c1ef050 con 0x7f029c155b70 2024-03-06T07:22:14.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f029b7fe640 1 --2- 172.21.15.1:0/889536643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c155b70 0x7f029c1ee560 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0294009f70 tx=0x7f0294004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.801+0000 7f02837fe640 1 -- 172.21.15.1:0/889536643 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0294043020 con 0x7f029c155b70 2024-03-06T07:22:14.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.802+0000 7f02837fe640 1 -- 172.21.15.1:0/889536643 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029403e450 con 0x7f029c155b70 2024-03-06T07:22:14.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.802+0000 7f02a13dd640 1 -- 172.21.15.1:0/889536643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029c152100 con 0x7f029c155b70 2024-03-06T07:22:14.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.802+0000 7f02837fe640 1 -- 172.21.15.1:0/889536643 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029403e820 con 0x7f029c155b70 2024-03-06T07:22:14.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.802+0000 7f02a13dd640 1 -- 172.21.15.1:0/889536643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029c155130 con 0x7f029c155b70 2024-03-06T07:22:14.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.802+0000 7f02a13dd640 1 -- 172.21.15.1:0/889536643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f029c151ef0 con 0x7f029c155b70 2024-03-06T07:22:14.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.803+0000 7f02837fe640 1 -- 172.21.15.1:0/889536643 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0294046020 con 0x7f029c155b70 2024-03-06T07:22:14.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.803+0000 7f02837fe640 1 --2- 172.21.15.1:0/889536643 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f028806b4d0 0x7f028806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.804+0000 7f02837fe640 1 -- 172.21.15.1:0/889536643 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f02940acf70 con 0x7f029c155b70 2024-03-06T07:22:14.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.804+0000 7f029affd640 1 --2- 172.21.15.1:0/889536643 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f028806b4d0 0x7f028806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.804+0000 7f029affd640 1 --2- 172.21.15.1:0/889536643 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f028806b4d0 0x7f028806d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f028c00aac0 tx=0x7f028c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:14.811+0000 7f02837fe640 1 -- 172.21.15.1:0/889536643 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f029404d030 con 0x7f029c155b70 2024-03-06T07:22:15.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.030+0000 7f02a13dd640 1 -- 172.21.15.1:0/889536643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f029c067fd0 con 0x7f029c155b70 2024-03-06T07:22:15.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.030+0000 7f02837fe640 1 -- 172.21.15.1:0/889536643 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4968 (secure 0 0 0) 0x7f029407b8a0 con 0x7f029c155b70 2024-03-06T07:22:15.032 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:15.032 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5084,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2937068915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2937068915},{"type":"v1","addr":"172.21.15.98:6836","nonce":2937068915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5087,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2828783606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2828783606},{"type":"v1","addr":"172.21.15.98:6837","nonce":2828783606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/3845300501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":3845300501},{"type":"v1","addr":"172.21.15.1:6835","nonce":3845300501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:05.671602+0000","modified":"2024-03-06T07:22:07.663851+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5778":{"gid":5778,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/131367647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":131367647},{"type":"v1","addr":"172.21.15.1:6834","nonce":131367647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5778,"qdb_cluster":[5778]},"id":3}]} 2024-03-06T07:22:15.033 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-03-06T07:22:15.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.032+0000 7f02a13dd640 1 -- 172.21.15.1:0/889536643 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f028806b4d0 msgr2=0x7f028806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.032+0000 7f02a13dd640 1 --2- 172.21.15.1:0/889536643 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f028806b4d0 0x7f028806d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f028c00aac0 tx=0x7f028c008040 comp rx=0 tx=0).stop 2024-03-06T07:22:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.032+0000 7f02a13dd640 1 -- 172.21.15.1:0/889536643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c155b70 msgr2=0x7f029c1ee560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.032+0000 7f02a13dd640 1 --2- 172.21.15.1:0/889536643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c155b70 0x7f029c1ee560 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0294009f70 tx=0x7f0294004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.032+0000 7f02a13dd640 1 -- 172.21.15.1:0/889536643 shutdown_connections 2024-03-06T07:22:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.032+0000 7f02a13dd640 1 --2- 172.21.15.1:0/889536643 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f028806b4d0 0x7f028806d990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.032+0000 7f02a13dd640 1 --2- 172.21.15.1:0/889536643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c155b70 0x7f029c1ee560 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.032+0000 7f02a13dd640 1 --2- 172.21.15.1:0/889536643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f029c0c8cd0 0x7f029c153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.032+0000 7f02a13dd640 1 --2- 172.21.15.1:0/889536643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f029c0c3260 0x7f029c14fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.032+0000 7f02a13dd640 1 -- 172.21.15.1:0/889536643 >> 172.21.15.1:0/889536643 conn(0x7f029c06b3e0 msgr2=0x7f029c156ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.032+0000 7f02a13dd640 1 -- 172.21.15.1:0/889536643 shutdown_connections 2024-03-06T07:22:15.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.032+0000 7f02a13dd640 1 -- 172.21.15.1:0/889536643 wait complete. 2024-03-06T07:22:15.447 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:22:05.671602+0000', 'modified': '2024-03-06T07:22:07.663851+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5778}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5778': {'gid': 5778, 'name': 'c', 'rank': 0, 'incarnation': 26, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6834/131367647', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 131367647}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 131367647}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5778, 'qdb_cluster': [5778]} 2024-03-06T07:22:15.447 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:22:15.447 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:22:15.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.553+0000 7fa41cf56640 1 Processor -- start 2024-03-06T07:22:15.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.553+0000 7fa41cf56640 1 -- start start 2024-03-06T07:22:15.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.554+0000 7fa41cf56640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4180c3a90 0x7fa418159250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.554+0000 7fa41cf56640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa418155e50 0x7fa4181524f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.554+0000 7fa41cf56640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4180c5970 con 0x7fa4180c5440 2024-03-06T07:22:15.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.554+0000 7fa41cf56640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4180c5c50 con 0x7fa418155e50 2024-03-06T07:22:15.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.554+0000 7fa41cf56640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4181559c0 con 0x7fa4180c3a90 2024-03-06T07:22:15.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.554+0000 7fa416d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa418155e50 0x7fa4181524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.554+0000 7fa40ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa4180c5440 0x7fa4180c67f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51636/0 (socket says 172.21.15.1:51636) 2024-03-06T07:22:15.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.554+0000 7fa40ffff640 1 -- 172.21.15.1:0/1458161888 learned_addr learned my addr 172.21.15.1:0/1458161888 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:15.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.554+0000 7fa41659c640 1 --2- 172.21.15.1:0/1458161888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4180c3a90 0x7fa418159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.554+0000 7fa416d9d640 1 -- 172.21.15.1:0/1458161888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4180c3a90 msgr2=0x7fa418159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.554+0000 7fa416d9d640 1 --2- 172.21.15.1:0/1458161888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4180c3a90 0x7fa418159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.554+0000 7fa416d9d640 1 -- 172.21.15.1:0/1458161888 >> v1:172.21.15.1:6789/0 conn(0x7fa4180c5440 legacy=0x7fa4180c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.554+0000 7fa415d9b640 1 -- 172.21.15.1:0/1458161888 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3428734615 0 0) 0x7fa4180c5970 con 0x7fa4180c5440 2024-03-06T07:22:15.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.554+0000 7fa416d9d640 1 -- 172.21.15.1:0/1458161888 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa418155370 con 0x7fa418155e50 2024-03-06T07:22:15.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.555+0000 7fa416d9d640 1 --2- 172.21.15.1:0/1458161888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa418155e50 0x7fa4181524f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa410009d90 tx=0x7fa41002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb5f9d5f772f1fad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.555+0000 7fa415d9b640 1 -- 172.21.15.1:0/1458161888 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa410039070 con 0x7fa418155e50 2024-03-06T07:22:15.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.555+0000 7fa415d9b640 1 -- 172.21.15.1:0/1458161888 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa41002cbc0 con 0x7fa418155e50 2024-03-06T07:22:15.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.555+0000 7fa415d9b640 1 -- 172.21.15.1:0/1458161888 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa410034440 con 0x7fa418155e50 2024-03-06T07:22:15.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.556+0000 7fa41cf56640 1 -- 172.21.15.1:0/1458161888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa418155e50 msgr2=0x7fa4181524f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.556+0000 7fa41cf56640 1 --2- 172.21.15.1:0/1458161888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa418155e50 0x7fa4181524f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa410009d90 tx=0x7fa41002be10 comp rx=0 tx=0).stop 2024-03-06T07:22:15.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.556+0000 7fa41cf56640 1 -- 172.21.15.1:0/1458161888 shutdown_connections 2024-03-06T07:22:15.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.556+0000 7fa41cf56640 1 --2- 172.21.15.1:0/1458161888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa418155e50 0x7fa4181524f0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.556+0000 7fa41cf56640 1 --2- 172.21.15.1:0/1458161888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4180c3a90 0x7fa418159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.556+0000 7fa41cf56640 1 -- 172.21.15.1:0/1458161888 >> 172.21.15.1:0/1458161888 conn(0x7fa4180c88f0 msgr2=0x7fa4180c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.556+0000 7fa41cf56640 1 -- 172.21.15.1:0/1458161888 shutdown_connections 2024-03-06T07:22:15.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.556+0000 7fa41cf56640 1 -- 172.21.15.1:0/1458161888 wait complete. 2024-03-06T07:22:15.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.557+0000 7fa41cf56640 1 Processor -- start 2024-03-06T07:22:15.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.557+0000 7fa41cf56640 1 -- start start 2024-03-06T07:22:15.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa41cf56640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4180c3a90 0x7fa4180ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa41cf56640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4180c5440 0x7fa4180ac460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa41cf56640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa418155e50 0x7fa4180ac9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa41cf56640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4180c6f00 con 0x7fa4180c5440 2024-03-06T07:22:15.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa41cf56640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4180c84a0 con 0x7fa418155e50 2024-03-06T07:22:15.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa41cf56640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41814a4e0 con 0x7fa4180c3a90 2024-03-06T07:22:15.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa40ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4180c5440 0x7fa4180ac460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa40ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4180c5440 0x7fa4180ac460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37328/0 (socket says 172.21.15.1:37328) 2024-03-06T07:22:15.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa40ffff640 1 -- 172.21.15.1:0/3090556759 learned_addr learned my addr 172.21.15.1:0/3090556759 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:15.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa41659c640 1 --2- 172.21.15.1:0/3090556759 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4180c3a90 0x7fa4180ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa416d9d640 1 --2- 172.21.15.1:0/3090556759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa418155e50 0x7fa4180ac9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa40ffff640 1 -- 172.21.15.1:0/3090556759 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4180c3a90 msgr2=0x7fa4180ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa40ffff640 1 --2- 172.21.15.1:0/3090556759 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4180c3a90 0x7fa4180ad880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa40ffff640 1 -- 172.21.15.1:0/3090556759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa418155e50 msgr2=0x7fa4180ac9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa40ffff640 1 --2- 172.21.15.1:0/3090556759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa418155e50 0x7fa4180ac9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa40ffff640 1 -- 172.21.15.1:0/3090556759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4180abd40 con 0x7fa4180c5440 2024-03-06T07:22:15.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa40ffff640 1 --2- 172.21.15.1:0/3090556759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4180c5440 0x7fa4180ac460 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa40000c4d0 tx=0x7fa40000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa40effd640 1 -- 172.21.15.1:0/3090556759 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400018070 con 0x7fa4180c5440 2024-03-06T07:22:15.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa40effd640 1 -- 172.21.15.1:0/3090556759 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa400007500 con 0x7fa4180c5440 2024-03-06T07:22:15.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa40effd640 1 -- 172.21.15.1:0/3090556759 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400004a20 con 0x7fa4180c5440 2024-03-06T07:22:15.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa41cf56640 1 -- 172.21.15.1:0/3090556759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa41814c6f0 con 0x7fa4180c5440 2024-03-06T07:22:15.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.558+0000 7fa41cf56640 1 -- 172.21.15.1:0/3090556759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41814a890 con 0x7fa4180c5440 2024-03-06T07:22:15.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.560+0000 7fa40effd640 1 -- 172.21.15.1:0/3090556759 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa40001b070 con 0x7fa4180c5440 2024-03-06T07:22:15.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.561+0000 7fa40effd640 1 --2- 172.21.15.1:0/3090556759 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3f806b500 0x7fa3f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.561+0000 7fa40effd640 1 -- 172.21.15.1:0/3090556759 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa400089560 con 0x7fa4180c5440 2024-03-06T07:22:15.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.561+0000 7fa41659c640 1 --2- 172.21.15.1:0/3090556759 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3f806b500 0x7fa3f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.561+0000 7fa41659c640 1 --2- 172.21.15.1:0/3090556759 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3f806b500 0x7fa3f806d9c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa408006fd0 tx=0x7fa408008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.562+0000 7fa41fc1b740 1 Processor -- start 2024-03-06T07:22:15.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.562+0000 7fa41fc1b740 1 -- start start 2024-03-06T07:22:15.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.563+0000 7fa41fc1b740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b2905c8f50 0x55b2905c6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.563+0000 7fa41fc1b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c9330 0x55b2905c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.563+0000 7fa41fc1b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c74b0 0x55b2905c3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.563+0000 7fa41fc1b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b29052d6b0 con 0x55b2905c9330 2024-03-06T07:22:15.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.563+0000 7fa41fc1b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2904d6a60 con 0x55b2905c74b0 2024-03-06T07:22:15.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.563+0000 7fa41fc1b740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b290529750 con 0x55b2905c8f50 2024-03-06T07:22:15.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.563+0000 7fa40ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c9330 0x55b2905c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.563+0000 7fa40ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c9330 0x55b2905c6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37340/0 (socket says 172.21.15.1:37340) 2024-03-06T07:22:15.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.563+0000 7fa40ffff640 1 -- 172.21.15.1:0/839853394 learned_addr learned my addr 172.21.15.1:0/839853394 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:15.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.563+0000 7fa41659c640 1 --2- 172.21.15.1:0/839853394 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b2905c8f50 0x55b2905c6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.563+0000 7fa416d9d640 1 --2- 172.21.15.1:0/839853394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c74b0 0x55b2905c3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.563+0000 7fa40ffff640 1 -- 172.21.15.1:0/839853394 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b2905c8f50 msgr2=0x55b2905c6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.563+0000 7fa40ffff640 1 --2- 172.21.15.1:0/839853394 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b2905c8f50 0x55b2905c6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.563+0000 7fa40ffff640 1 -- 172.21.15.1:0/839853394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c74b0 msgr2=0x55b2905c3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.563+0000 7fa40ffff640 1 --2- 172.21.15.1:0/839853394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c74b0 0x55b2905c3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.563+0000 7fa40ffff640 1 -- 172.21.15.1:0/839853394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b2904f4f30 con 0x55b2905c9330 2024-03-06T07:22:15.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.563+0000 7fa416d9d640 1 --2- 172.21.15.1:0/839853394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c74b0 0x55b2905c3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:15.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.564+0000 7fa40ffff640 1 --2- 172.21.15.1:0/839853394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c9330 0x55b2905c6f70 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa40005fdc0 tx=0x7fa400057d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=474557bb15f15cc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.564+0000 7fa3f7fff640 1 -- 172.21.15.1:0/839853394 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400055f90 con 0x55b2905c9330 2024-03-06T07:22:15.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.564+0000 7fa3f7fff640 1 -- 172.21.15.1:0/839853394 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa400054300 con 0x55b2905c9330 2024-03-06T07:22:15.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.564+0000 7fa3f7fff640 1 -- 172.21.15.1:0/839853394 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400004dc0 con 0x55b2905c9330 2024-03-06T07:22:15.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.564+0000 7fa41fc1b740 1 -- 172.21.15.1:0/839853394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c9330 msgr2=0x55b2905c6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.564+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/839853394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c9330 0x55b2905c6f70 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa40005fdc0 tx=0x7fa400057d50 comp rx=0 tx=0).stop 2024-03-06T07:22:15.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.565+0000 7fa41fc1b740 1 -- 172.21.15.1:0/839853394 shutdown_connections 2024-03-06T07:22:15.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.565+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/839853394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c74b0 0x55b2905c3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.565+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/839853394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c9330 0x55b2905c6f70 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.565+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/839853394 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b2905c8f50 0x55b2905c6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.565+0000 7fa41fc1b740 1 -- 172.21.15.1:0/839853394 >> 172.21.15.1:0/839853394 conn(0x55b2905c57d0 msgr2=0x55b2905ea3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.565+0000 7fa41fc1b740 1 -- 172.21.15.1:0/839853394 shutdown_connections 2024-03-06T07:22:15.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.565+0000 7fa41fc1b740 1 -- 172.21.15.1:0/839853394 wait complete. 2024-03-06T07:22:15.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.565+0000 7fa41fc1b740 1 -- start start 2024-03-06T07:22:15.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.565+0000 7fa41fc1b740 1 Processor -- start 2024-03-06T07:22:15.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.566+0000 7fa41fc1b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c8f50 0x55b2905e97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.566+0000 7fa41fc1b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c9330 0x55b2905eb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.566+0000 7fa41fc1b740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b290602a70 0x55b2905e5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.566+0000 7fa41fc1b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2905eb230 con 0x55b2905c9330 2024-03-06T07:22:15.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.566+0000 7fa41fc1b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2904e9ea0 con 0x55b2905c8f50 2024-03-06T07:22:15.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.566+0000 7fa41fc1b740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b2905eac70 con 0x55b290602a70 2024-03-06T07:22:15.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.566+0000 7fa41fc1b740 10 client.0 _ll_register_callbacks cb 0x55b2904e9870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:15.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.566+0000 7fa40ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c9330 0x55b2905eb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.566+0000 7fa40ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c9330 0x55b2905eb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37342/0 (socket says 172.21.15.1:37342) 2024-03-06T07:22:15.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.566+0000 7fa40ffff640 1 -- 172.21.15.1:0/1747856201 learned_addr learned my addr 172.21.15.1:0/1747856201 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:15.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.566+0000 7fa416d9d640 1 --2- 172.21.15.1:0/1747856201 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b290602a70 0x55b2905e5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.566+0000 7fa41659c640 1 --2- 172.21.15.1:0/1747856201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c8f50 0x55b2905e97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa40ffff640 1 -- 172.21.15.1:0/1747856201 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b290602a70 msgr2=0x55b2905e5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa40ffff640 1 --2- 172.21.15.1:0/1747856201 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b290602a70 0x55b2905e5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa40ffff640 1 -- 172.21.15.1:0/1747856201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c8f50 msgr2=0x55b2905e97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa40ffff640 1 --2- 172.21.15.1:0/1747856201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c8f50 0x55b2905e97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa40ffff640 1 -- 172.21.15.1:0/1747856201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b2904ea270 con 0x55b2905c9330 2024-03-06T07:22:15.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa40ffff640 1 --2- 172.21.15.1:0/1747856201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c9330 0x55b2905eb600 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fa4000011e0 tx=0x7fa40005a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1155d117ecce0c5f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa3f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:22:15.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa3f57fa640 1 -- 172.21.15.1:0/1747856201 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40006bda0 con 0x55b2905c9330 2024-03-06T07:22:15.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa3f57fa640 1 -- 172.21.15.1:0/1747856201 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa400085c80 con 0x55b2905c9330 2024-03-06T07:22:15.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa41fc1b740 1 -- 172.21.15.1:0/1747856201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa400054300 con 0x55b2905c9330 2024-03-06T07:22:15.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa3f57fa640 1 -- 172.21.15.1:0/1747856201 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4000890c0 con 0x55b2905c9330 2024-03-06T07:22:15.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa41fc1b740 10 client.5958 fetch_fsmap learned FSMap version 28 2024-03-06T07:22:15.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa41fc1b740 1 -- 172.21.15.1:0/1747856201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55b2902422d0 con 0x55b2905c9330 2024-03-06T07:22:15.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa3f57fa640 1 -- 172.21.15.1:0/1747856201 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa400054300 con 0x55b2905c9330 2024-03-06T07:22:15.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa3f57fa640 1 -- 172.21.15.1:0/1747856201 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 28) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa400082070 con 0x55b2905c9330 2024-03-06T07:22:15.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa41fc1b740 10 client.5958 fetch_fsmap finished waiting for FSMap version 28 2024-03-06T07:22:15.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa41fc1b740 10 client.5958 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:22:15.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa41fc1b740 20 client.5958 populate_metadata read hostname 'smithi001' 2024-03-06T07:22:15.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/1747856201 >> [v2:172.21.15.1:6832/131367647,v1:172.21.15.1:6834/131367647] conn(0x55b290615aa0 0x55b290615e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa41fc1b740 4 client.5958 mds_command: new command op to 5778 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:22:15.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.567+0000 7fa41fc1b740 1 -- 172.21.15.1:0/1747856201 --> [v2:172.21.15.1:6832/131367647,v1:172.21.15.1:6834/131367647] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b29053abf0 con 0x55b290615aa0 2024-03-06T07:22:15.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.568+0000 7fa41659c640 1 --2- 172.21.15.1:0/1747856201 >> [v2:172.21.15.1:6832/131367647,v1:172.21.15.1:6834/131367647] conn(0x55b290615aa0 0x55b290615e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.568+0000 7fa3f57fa640 10 client.5958 ms_handle_connect on v2:172.21.15.1:6832/131367647 2024-03-06T07:22:15.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.568+0000 7fa41659c640 1 --2- 172.21.15.1:0/1747856201 >> [v2:172.21.15.1:6832/131367647,v1:172.21.15.1:6834/131367647] conn(0x55b290615aa0 0x55b290615e80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b579282cf4976b58 server_cookie=1d4fc63acb7fddc6 in_seq=0 out_seq=0 2024-03-06T07:22:15.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.568+0000 7fa3f57fa640 1 -- 172.21.15.1:0/1747856201 <== mds.0 v2:172.21.15.1:6832/131367647 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b29053abf0 con 0x55b290615aa0 2024-03-06T07:22:15.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.568+0000 7fa3f57fa640 10 client.5958 handle_command_reply: tid=0 2024-03-06T07:22:15.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.568+0000 7fa41fc1b740 1 client.5958 shutdown 2024-03-06T07:22:15.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.569+0000 7fa41fc1b740 1 -- 172.21.15.1:0/1747856201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c9330 msgr2=0x55b2905eb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.569+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/1747856201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c9330 0x55b2905eb600 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fa4000011e0 tx=0x7fa40005a040 comp rx=0 tx=0).stop 2024-03-06T07:22:15.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.569+0000 7fa41fc1b740 1 -- 172.21.15.1:0/1747856201 shutdown_connections 2024-03-06T07:22:15.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.569+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/1747856201 >> [v2:172.21.15.1:6832/131367647,v1:172.21.15.1:6834/131367647] conn(0x55b290615aa0 0x55b290615e80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.569+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/1747856201 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b290602a70 0x55b2905e5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.569+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/1747856201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c9330 0x55b2905eb600 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.569+0000 7fa3f57fa640 0 client.5958 ms_handle_reset on v2:172.21.15.1:6832/131367647 2024-03-06T07:22:15.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.569+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/1747856201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c8f50 0x55b2905e97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.569+0000 7fa41fc1b740 1 -- 172.21.15.1:0/1747856201 >> 172.21.15.1:0/1747856201 conn(0x55b2905c74b0 msgr2=0x55b2905c78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.569+0000 7fa41fc1b740 1 -- 172.21.15.1:0/1747856201 shutdown_connections 2024-03-06T07:22:15.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.569+0000 7fa41fc1b740 1 -- 172.21.15.1:0/1747856201 wait complete. 2024-03-06T07:22:15.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.569+0000 7fa41fc1b740 20 client.5958 trim_cache size 0 max 16384 2024-03-06T07:22:15.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.586+0000 7fa41fc1b740 1 Processor -- start 2024-03-06T07:22:15.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.586+0000 7fa41fc1b740 1 -- start start 2024-03-06T07:22:15.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.586+0000 7fa41fc1b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c8f50 0x55b2905eeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.586+0000 7fa41fc1b740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b2905c9330 0x55b2905c2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.586+0000 7fa41fc1b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b290602a70 0x55b2905c2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.586+0000 7fa41fc1b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4181f79e0 con 0x55b290602a70 2024-03-06T07:22:15.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.586+0000 7fa41fc1b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4181f6b00 con 0x55b2905c8f50 2024-03-06T07:22:15.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.586+0000 7fa41fc1b740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa418151930 con 0x55b2905c9330 2024-03-06T07:22:15.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.586+0000 7fa416d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b290602a70 0x55b2905c2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.586+0000 7fa416d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b290602a70 0x55b2905c2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37358/0 (socket says 172.21.15.1:37358) 2024-03-06T07:22:15.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.586+0000 7fa416d9d640 1 -- 172.21.15.1:0/3426381500 learned_addr learned my addr 172.21.15.1:0/3426381500 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:15.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.587+0000 7fa41659c640 1 --2- 172.21.15.1:0/3426381500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c8f50 0x55b2905eeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.587+0000 7fa40ffff640 1 --2- 172.21.15.1:0/3426381500 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b2905c9330 0x55b2905c2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.587+0000 7fa416d9d640 1 -- 172.21.15.1:0/3426381500 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b2905c9330 msgr2=0x55b2905c2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.587+0000 7fa416d9d640 1 --2- 172.21.15.1:0/3426381500 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b2905c9330 0x55b2905c2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.587+0000 7fa416d9d640 1 -- 172.21.15.1:0/3426381500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c8f50 msgr2=0x55b2905eeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.587+0000 7fa416d9d640 1 --2- 172.21.15.1:0/3426381500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c8f50 0x55b2905eeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.587+0000 7fa416d9d640 1 -- 172.21.15.1:0/3426381500 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b29030c6b0 con 0x55b290602a70 2024-03-06T07:22:15.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.587+0000 7fa416d9d640 1 --2- 172.21.15.1:0/3426381500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b290602a70 0x55b2905c2a50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fa410009eb0 tx=0x7fa41002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87e3c32a54b73f12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.587+0000 7fa3f77fe640 1 -- 172.21.15.1:0/3426381500 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa410039040 con 0x55b290602a70 2024-03-06T07:22:15.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.587+0000 7fa3f77fe640 1 -- 172.21.15.1:0/3426381500 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa410003b90 con 0x55b290602a70 2024-03-06T07:22:15.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.587+0000 7fa3f77fe640 1 -- 172.21.15.1:0/3426381500 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa410004040 con 0x55b290602a70 2024-03-06T07:22:15.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.588+0000 7fa41fc1b740 1 -- 172.21.15.1:0/3426381500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b290602a70 msgr2=0x55b2905c2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.588+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/3426381500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b290602a70 0x55b2905c2a50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fa410009eb0 tx=0x7fa41002e000 comp rx=0 tx=0).stop 2024-03-06T07:22:15.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.588+0000 7fa41fc1b740 1 -- 172.21.15.1:0/3426381500 shutdown_connections 2024-03-06T07:22:15.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.588+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/3426381500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b290602a70 0x55b2905c2a50 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.588+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/3426381500 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b2905c9330 0x55b2905c2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.588+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/3426381500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c8f50 0x55b2905eeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.588+0000 7fa41fc1b740 1 -- 172.21.15.1:0/3426381500 >> 172.21.15.1:0/3426381500 conn(0x55b2905c74b0 msgr2=0x55b2905e95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.588+0000 7fa41fc1b740 1 -- 172.21.15.1:0/3426381500 shutdown_connections 2024-03-06T07:22:15.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.588+0000 7fa41fc1b740 1 -- 172.21.15.1:0/3426381500 wait complete. 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.588+0000 7fa41fc1b740 1 -- start start 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.588+0000 7fa41fc1b740 1 Processor -- start 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.589+0000 7fa41fc1b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c8f50 0x55b2906289e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.589+0000 7fa41fc1b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c9330 0x55b2905c2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.589+0000 7fa41fc1b740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b290602a70 0x55b2906717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.589+0000 7fa41fc1b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2905eb230 con 0x55b2905c8f50 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.589+0000 7fa41fc1b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2904e9ea0 con 0x55b2905c9330 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.589+0000 7fa41fc1b740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b2905ebe70 con 0x55b290602a70 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.589+0000 7fa41fc1b740 10 client.0 _ll_register_callbacks cb 0x7fa400004f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.589+0000 7fa41659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c8f50 0x55b2906289e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.589+0000 7fa41659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c8f50 0x55b2906289e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37374/0 (socket says 172.21.15.1:37374) 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.589+0000 7fa41659c640 1 -- 172.21.15.1:0/2535997986 learned_addr learned my addr 172.21.15.1:0/2535997986 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.589+0000 7fa416d9d640 1 --2- 172.21.15.1:0/2535997986 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b290602a70 0x55b2906717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.589+0000 7fa40ffff640 1 --2- 172.21.15.1:0/2535997986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c9330 0x55b2905c2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa41659c640 1 -- 172.21.15.1:0/2535997986 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b290602a70 msgr2=0x55b2906717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa41659c640 1 --2- 172.21.15.1:0/2535997986 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b290602a70 0x55b2906717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa41659c640 1 -- 172.21.15.1:0/2535997986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c9330 msgr2=0x55b2905c2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa41659c640 1 --2- 172.21.15.1:0/2535997986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c9330 0x55b2905c2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa41659c640 1 -- 172.21.15.1:0/2535997986 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b2905eeef0 con 0x55b2905c8f50 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa41659c640 1 --2- 172.21.15.1:0/2535997986 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c8f50 0x55b2906289e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa408007860 tx=0x7fa408013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9b5f9f12db6146 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa3f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa3f57fa640 1 -- 172.21.15.1:0/2535997986 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40801c040 con 0x55b2905c8f50 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa3f57fa640 1 -- 172.21.15.1:0/2535997986 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa408004240 con 0x55b2905c8f50 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa41fc1b740 1 -- 172.21.15.1:0/2535997986 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa410003b90 con 0x55b2905c8f50 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa3f57fa640 1 -- 172.21.15.1:0/2535997986 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa408004440 con 0x55b2905c8f50 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa41fc1b740 10 client.5964 fetch_fsmap learned FSMap version 28 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa41fc1b740 1 -- 172.21.15.1:0/2535997986 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55b290628fb0 con 0x55b2905c8f50 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa3f57fa640 1 -- 172.21.15.1:0/2535997986 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa410003b90 con 0x55b2905c8f50 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa3f57fa640 1 -- 172.21.15.1:0/2535997986 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 28) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa408005990 con 0x55b2905c8f50 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa41fc1b740 10 client.5964 fetch_fsmap finished waiting for FSMap version 28 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa41fc1b740 10 client.5964 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa41fc1b740 20 client.5964 populate_metadata read hostname 'smithi001' 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/2535997986 >> [v2:172.21.15.1:6832/131367647,v1:172.21.15.1:6834/131367647] conn(0x55b2906779e0 0x55b290677dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa41fc1b740 4 client.5964 mds_command: new command op to 5778 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:22:15.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa41fc1b740 1 -- 172.21.15.1:0/2535997986 --> [v2:172.21.15.1:6832/131367647,v1:172.21.15.1:6834/131367647] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b2905ea880 con 0x55b2906779e0 2024-03-06T07:22:15.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.590+0000 7fa40ffff640 1 --2- 172.21.15.1:0/2535997986 >> [v2:172.21.15.1:6832/131367647,v1:172.21.15.1:6834/131367647] conn(0x55b2906779e0 0x55b290677dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.591+0000 7fa40ffff640 1 --2- 172.21.15.1:0/2535997986 >> [v2:172.21.15.1:6832/131367647,v1:172.21.15.1:6834/131367647] conn(0x55b2906779e0 0x55b290677dc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cf03e69ed62fa09 server_cookie=8c7d66ba47d33b6a in_seq=0 out_seq=0 2024-03-06T07:22:15.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.591+0000 7fa3f57fa640 10 client.5964 ms_handle_connect on v2:172.21.15.1:6832/131367647 2024-03-06T07:22:15.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.591+0000 7fa3f57fa640 1 -- 172.21.15.1:0/2535997986 <== mds.0 v2:172.21.15.1:6832/131367647 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55b2905ea880 con 0x55b2906779e0 2024-03-06T07:22:15.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.591+0000 7fa3f57fa640 10 client.5964 handle_command_reply: tid=0 2024-03-06T07:22:15.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.591+0000 7fa41fc1b740 1 client.5964 shutdown 2024-03-06T07:22:15.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.591+0000 7fa41fc1b740 1 -- 172.21.15.1:0/2535997986 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c8f50 msgr2=0x55b2906289e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.591+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/2535997986 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c8f50 0x55b2906289e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa408007860 tx=0x7fa408013000 comp rx=0 tx=0).stop 2024-03-06T07:22:15.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.591+0000 7fa41fc1b740 1 -- 172.21.15.1:0/2535997986 shutdown_connections 2024-03-06T07:22:15.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.591+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/2535997986 >> [v2:172.21.15.1:6832/131367647,v1:172.21.15.1:6834/131367647] conn(0x55b2906779e0 0x55b290677dc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.591+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/2535997986 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55b290602a70 0x55b2906717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.591+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/2535997986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b2905c9330 0x55b2905c2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.591+0000 7fa3f57fa640 0 client.5964 ms_handle_reset on v2:172.21.15.1:6832/131367647 2024-03-06T07:22:15.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.591+0000 7fa41fc1b740 1 --2- 172.21.15.1:0/2535997986 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b2905c8f50 0x55b2906289e0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.591+0000 7fa41fc1b740 1 -- 172.21.15.1:0/2535997986 >> 172.21.15.1:0/2535997986 conn(0x55b2905c74b0 msgr2=0x55b2905e89b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.591+0000 7fa41fc1b740 1 -- 172.21.15.1:0/2535997986 shutdown_connections 2024-03-06T07:22:15.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.591+0000 7fa41fc1b740 1 -- 172.21.15.1:0/2535997986 wait complete. 2024-03-06T07:22:15.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.591+0000 7fa41fc1b740 20 client.5964 trim_cache size 0 max 16384 2024-03-06T07:22:15.598 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:22:15.598 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:22:15.598 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:22:15.598 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 5778, 2024-03-06T07:22:15.598 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:22:15.598 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:22:15.599 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:22:15.599 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 9.2898571640000007, 2024-03-06T07:22:15.599 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 28, 2024-03-06T07:22:15.599 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 52, 2024-03-06T07:22:15.599 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 52, 2024-03-06T07:22:15.599 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 14.355852594 2024-03-06T07:22:15.599 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:22:15.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.592+0000 7fa41cf56640 1 -- 172.21.15.1:0/3090556759 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3f806b500 msgr2=0x7fa3f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.592+0000 7fa41cf56640 1 --2- 172.21.15.1:0/3090556759 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3f806b500 0x7fa3f806d9c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa408006fd0 tx=0x7fa408008040 comp rx=0 tx=0).stop 2024-03-06T07:22:15.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.592+0000 7fa41cf56640 1 -- 172.21.15.1:0/3090556759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4180c5440 msgr2=0x7fa4180ac460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.592+0000 7fa41cf56640 1 --2- 172.21.15.1:0/3090556759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4180c5440 0x7fa4180ac460 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa40000c4d0 tx=0x7fa40000c990 comp rx=0 tx=0).stop 2024-03-06T07:22:15.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.592+0000 7fa41cf56640 1 -- 172.21.15.1:0/3090556759 shutdown_connections 2024-03-06T07:22:15.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.592+0000 7fa41cf56640 1 --2- 172.21.15.1:0/3090556759 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3f806b500 0x7fa3f806d9c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.592+0000 7fa41cf56640 1 --2- 172.21.15.1:0/3090556759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa418155e50 0x7fa4180ac9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.592+0000 7fa41cf56640 1 --2- 172.21.15.1:0/3090556759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4180c5440 0x7fa4180ac460 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.592+0000 7fa41cf56640 1 --2- 172.21.15.1:0/3090556759 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4180c3a90 0x7fa4180ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.592+0000 7fa41cf56640 1 -- 172.21.15.1:0/3090556759 >> 172.21.15.1:0/3090556759 conn(0x7fa4180c88f0 msgr2=0x7fa4181528a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.592+0000 7fa41cf56640 1 -- 172.21.15.1:0/3090556759 shutdown_connections 2024-03-06T07:22:15.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.592+0000 7fa41cf56640 1 -- 172.21.15.1:0/3090556759 wait complete. 2024-03-06T07:22:15.604 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:15.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.704+0000 7f5beb93e640 1 Processor -- start 2024-03-06T07:22:15.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.704+0000 7f5beb93e640 1 -- start start 2024-03-06T07:22:15.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.705+0000 7f5beb93e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be4156500 0x7f5be4152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.705+0000 7f5beb93e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5be406b510 0x7f5be406b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.705+0000 7f5beb93e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be41569e0 con 0x7f5be414cdc0 2024-03-06T07:22:15.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.705+0000 7f5beb93e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5be40c2dd0 con 0x7f5be406b510 2024-03-06T07:22:15.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.705+0000 7f5beb93e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be4156cc0 con 0x7f5be4156500 2024-03-06T07:22:15.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.705+0000 7f5be96da640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5be414cdc0 0x7f5be40c2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51652/0 (socket says 172.21.15.1:51652) 2024-03-06T07:22:15.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.705+0000 7f5be96da640 1 -- 172.21.15.1:0/2010474761 learned_addr learned my addr 172.21.15.1:0/2010474761 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:15.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.705+0000 7f5be9edb640 1 --2- 172.21.15.1:0/2010474761 >> v2:172.21.15.98:3300/0 conn(0x7f5be406b510 0x7f5be406b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.705+0000 7f5be8ed9640 1 --2- 172.21.15.1:0/2010474761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be4156500 0x7f5be4152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.705+0000 7f5bdbfff640 1 -- 172.21.15.1:0/2010474761 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484742870 0 0) 0x7f5be41569e0 con 0x7f5be414cdc0 2024-03-06T07:22:15.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5bdbfff640 1 -- 172.21.15.1:0/2010474761 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd0003520 con 0x7f5be414cdc0 2024-03-06T07:22:15.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5bdbfff640 1 -- 172.21.15.1:0/2010474761 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1698996015 0 0) 0x7f5bd0003520 con 0x7f5be414cdc0 2024-03-06T07:22:15.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5bdbfff640 1 -- 172.21.15.1:0/2010474761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be4156500 msgr2=0x7f5be4152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5bdbfff640 1 --2- 172.21.15.1:0/2010474761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be4156500 0x7f5be4152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5bdbfff640 1 -- 172.21.15.1:0/2010474761 >> v2:172.21.15.98:3300/0 conn(0x7f5be406b510 msgr2=0x7f5be406b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5bdbfff640 1 --2- 172.21.15.1:0/2010474761 >> v2:172.21.15.98:3300/0 conn(0x7f5be406b510 0x7f5be406b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5bdbfff640 1 -- 172.21.15.1:0/2010474761 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be4156fa0 con 0x7f5be414cdc0 2024-03-06T07:22:15.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5bdbfff640 1 -- 172.21.15.1:0/2010474761 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f5be0001670 con 0x7f5be414cdc0 2024-03-06T07:22:15.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5bdbfff640 1 -- 172.21.15.1:0/2010474761 >> v1:172.21.15.1:6789/0 conn(0x7f5be414cdc0 legacy=0x7f5be40c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5bdbfff640 1 --2- 172.21.15.1:0/2010474761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bd00039b0 0x7f5bd0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5bdbfff640 1 --2- 172.21.15.1:0/2010474761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bd00043a0 0x7f5bd0004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5bdbfff640 1 --2- 172.21.15.1:0/2010474761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd0004ef0 0x7f5bd000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5bdbfff640 1 -- 172.21.15.1:0/2010474761 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be4156cc0 con 0x7f5bd00043a0 2024-03-06T07:22:15.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5bdbfff640 1 -- 172.21.15.1:0/2010474761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd000bf00 con 0x7f5bd0004ef0 2024-03-06T07:22:15.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5bdbfff640 1 -- 172.21.15.1:0/2010474761 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd000c080 con 0x7f5bd00039b0 2024-03-06T07:22:15.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5be9edb640 1 --2- 172.21.15.1:0/2010474761 >> v2:172.21.15.98:3300/0 conn(0x7f5be406b510 0x7f5be406b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:15.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5be8ed9640 1 --2- 172.21.15.1:0/2010474761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be4156500 0x7f5be4152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:15.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5be96da640 1 --2- 172.21.15.1:0/2010474761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bd00039b0 0x7f5bd0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5be8ed9640 1 --2- 172.21.15.1:0/2010474761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bd00043a0 0x7f5bd0004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5be9edb640 1 --2- 172.21.15.1:0/2010474761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd0004ef0 0x7f5bd000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5be8ed9640 1 -- 172.21.15.1:0/2010474761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bd00039b0 msgr2=0x7f5bd0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5be8ed9640 1 --2- 172.21.15.1:0/2010474761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5bd00039b0 0x7f5bd0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5be8ed9640 1 -- 172.21.15.1:0/2010474761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd0004ef0 msgr2=0x7f5bd000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5be8ed9640 1 --2- 172.21.15.1:0/2010474761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd0004ef0 0x7f5bd000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5be8ed9640 1 -- 172.21.15.1:0/2010474761 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd0003520 con 0x7f5bd00043a0 2024-03-06T07:22:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.706+0000 7f5be9edb640 1 --2- 172.21.15.1:0/2010474761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bd0004ef0 0x7f5bd000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.707+0000 7f5be8ed9640 1 --2- 172.21.15.1:0/2010474761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bd00043a0 0x7f5bd0004880 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f5be4072570 tx=0x7f5bd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c6cbe47dcd837f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.707+0000 7f5bdbfff640 1 -- 172.21.15.1:0/2010474761 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd40097d0 con 0x7f5bd00043a0 2024-03-06T07:22:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.707+0000 7f5bdbfff640 1 -- 172.21.15.1:0/2010474761 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd4004030 con 0x7f5bd00043a0 2024-03-06T07:22:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.707+0000 7f5bdbfff640 1 -- 172.21.15.1:0/2010474761 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4004400 con 0x7f5bd00043a0 2024-03-06T07:22:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.708+0000 7f5beb93e640 1 -- 172.21.15.1:0/2010474761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bd00043a0 msgr2=0x7f5bd0004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.708+0000 7f5beb93e640 1 --2- 172.21.15.1:0/2010474761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bd00043a0 0x7f5bd0004880 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f5be4072570 tx=0x7f5bd0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.708+0000 7f5be9edb640 1 -- 172.21.15.1:0/2010474761 reap_dead start 2024-03-06T07:22:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.708+0000 7f5beb93e640 1 -- 172.21.15.1:0/2010474761 shutdown_connections 2024-03-06T07:22:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.708+0000 7f5beb93e640 1 -- 172.21.15.1:0/2010474761 >> 172.21.15.1:0/2010474761 conn(0x7f5be40a9700 msgr2=0x7f5be40a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.708+0000 7f5beb93e640 1 -- 172.21.15.1:0/2010474761 shutdown_connections 2024-03-06T07:22:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.708+0000 7f5beb93e640 1 -- 172.21.15.1:0/2010474761 wait complete. 2024-03-06T07:22:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.710+0000 7f5beb93e640 1 Processor -- start 2024-03-06T07:22:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.710+0000 7f5beb93e640 1 -- start start 2024-03-06T07:22:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.711+0000 7f5beb93e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be40c11f0 0x7f5be40bca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.711+0000 7f5beb93e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be414c2a0 0x7f5be4141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.711+0000 7f5beb93e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be4141dc0 0x7f5be40bdde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.711+0000 7f5beb93e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be40c5a70 con 0x7f5be4141dc0 2024-03-06T07:22:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.711+0000 7f5beb93e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be406bec0 con 0x7f5be40c11f0 2024-03-06T07:22:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.711+0000 7f5beb93e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be40c65e0 con 0x7f5be414c2a0 2024-03-06T07:22:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.711+0000 7f5be9edb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be4141dc0 0x7f5be40bdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.711+0000 7f5be9edb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be4141dc0 0x7f5be40bdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37388/0 (socket says 172.21.15.1:37388) 2024-03-06T07:22:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.711+0000 7f5be9edb640 1 -- 172.21.15.1:0/2379965396 learned_addr learned my addr 172.21.15.1:0/2379965396 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.711+0000 7f5be96da640 1 --2- 172.21.15.1:0/2379965396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be40c11f0 0x7f5be40bca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.711+0000 7f5be8ed9640 1 --2- 172.21.15.1:0/2379965396 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be414c2a0 0x7f5be4141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.711+0000 7f5be9edb640 1 -- 172.21.15.1:0/2379965396 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be414c2a0 msgr2=0x7f5be4141880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.711+0000 7f5be9edb640 1 --2- 172.21.15.1:0/2379965396 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be414c2a0 0x7f5be4141880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.711+0000 7f5be9edb640 1 -- 172.21.15.1:0/2379965396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be40c11f0 msgr2=0x7f5be40bca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.711+0000 7f5be9edb640 1 --2- 172.21.15.1:0/2379965396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be40c11f0 0x7f5be40bca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.711+0000 7f5be9edb640 1 -- 172.21.15.1:0/2379965396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be40acb50 con 0x7f5be4141dc0 2024-03-06T07:22:15.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.711+0000 7f5be96da640 1 --2- 172.21.15.1:0/2379965396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be40c11f0 0x7f5be40bca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:15.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.712+0000 7f5be9edb640 1 --2- 172.21.15.1:0/2379965396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be4141dc0 0x7f5be40bdde0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f5be406d080 tx=0x7f5bdc0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.712+0000 7f5bd9ffb640 1 -- 172.21.15.1:0/2379965396 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc003990 con 0x7f5be4141dc0 2024-03-06T07:22:15.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.712+0000 7f5bd9ffb640 1 -- 172.21.15.1:0/2379965396 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bdc01b070 con 0x7f5be4141dc0 2024-03-06T07:22:15.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.712+0000 7f5beb93e640 1 -- 172.21.15.1:0/2379965396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be40abdd0 con 0x7f5be4141dc0 2024-03-06T07:22:15.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.712+0000 7f5bd9ffb640 1 -- 172.21.15.1:0/2379965396 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc003bb0 con 0x7f5be4141dc0 2024-03-06T07:22:15.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.712+0000 7f5beb93e640 1 -- 172.21.15.1:0/2379965396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be40ac350 con 0x7f5be4141dc0 2024-03-06T07:22:15.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.713+0000 7f5beb93e640 1 -- 172.21.15.1:0/2379965396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be4067fd0 con 0x7f5be4141dc0 2024-03-06T07:22:15.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.714+0000 7f5bd9ffb640 1 -- 172.21.15.1:0/2379965396 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5bdc005d10 con 0x7f5be4141dc0 2024-03-06T07:22:15.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.714+0000 7f5bd9ffb640 1 --2- 172.21.15.1:0/2379965396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bc806b4d0 0x7f5bc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.715+0000 7f5bd9ffb640 1 -- 172.21.15.1:0/2379965396 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5bdc08a900 con 0x7f5be4141dc0 2024-03-06T07:22:15.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.715+0000 7f5be96da640 1 --2- 172.21.15.1:0/2379965396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bc806b4d0 0x7f5bc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.715+0000 7f5be96da640 1 --2- 172.21.15.1:0/2379965396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bc806b4d0 0x7f5bc806d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5be0000c00 tx=0x7f5be0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.719+0000 7f5bd9ffb640 1 -- 172.21.15.1:0/2379965396 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5be40acb50 con 0x7f5be4141dc0 2024-03-06T07:22:16.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.998+0000 7f5beb93e640 1 -- 172.21.15.1:0/2379965396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5be4153e00 con 0x7f5be4141dc0 2024-03-06T07:22:16.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:15.999+0000 7f5bd9ffb640 1 -- 172.21.15.1:0/2379965396 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4968 (secure 0 0 0) 0x7f5bdc059680 con 0x7f5be4141dc0 2024-03-06T07:22:16.001 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:16.001 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5084,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2937068915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2937068915},{"type":"v1","addr":"172.21.15.98:6836","nonce":2937068915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5087,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2828783606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2828783606},{"type":"v1","addr":"172.21.15.98:6837","nonce":2828783606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/3845300501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":3845300501},{"type":"v1","addr":"172.21.15.1:6835","nonce":3845300501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:05.671602+0000","modified":"2024-03-06T07:22:07.663851+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5778":{"gid":5778,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/131367647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":131367647},{"type":"v1","addr":"172.21.15.1:6834","nonce":131367647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5778,"qdb_cluster":[5778]},"id":3}]} 2024-03-06T07:22:16.001 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-03-06T07:22:16.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.001+0000 7f5beb93e640 1 -- 172.21.15.1:0/2379965396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bc806b4d0 msgr2=0x7f5bc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.001+0000 7f5beb93e640 1 --2- 172.21.15.1:0/2379965396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bc806b4d0 0x7f5bc806d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5be0000c00 tx=0x7f5be0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:16.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.001+0000 7f5beb93e640 1 -- 172.21.15.1:0/2379965396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be4141dc0 msgr2=0x7f5be40bdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.001+0000 7f5beb93e640 1 --2- 172.21.15.1:0/2379965396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be4141dc0 0x7f5be40bdde0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f5be406d080 tx=0x7f5bdc0115a0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.001+0000 7f5beb93e640 1 -- 172.21.15.1:0/2379965396 shutdown_connections 2024-03-06T07:22:16.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.001+0000 7f5beb93e640 1 --2- 172.21.15.1:0/2379965396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bc806b4d0 0x7f5bc806d990 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.001+0000 7f5beb93e640 1 --2- 172.21.15.1:0/2379965396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be4141dc0 0x7f5be40bdde0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.001+0000 7f5beb93e640 1 --2- 172.21.15.1:0/2379965396 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be414c2a0 0x7f5be4141880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.001+0000 7f5beb93e640 1 --2- 172.21.15.1:0/2379965396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be40c11f0 0x7f5be40bca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.001+0000 7f5beb93e640 1 -- 172.21.15.1:0/2379965396 >> 172.21.15.1:0/2379965396 conn(0x7f5be40a9700 msgr2=0x7f5be41558c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.001+0000 7f5beb93e640 1 -- 172.21.15.1:0/2379965396 shutdown_connections 2024-03-06T07:22:16.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.001+0000 7f5beb93e640 1 -- 172.21.15.1:0/2379965396 wait complete. 2024-03-06T07:22:16.014 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 3, "epoch": 28, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 3, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:22:05.671602+0000", "damaged": [], "data_pools": [ 8 ], "enabled": true, "epoch": 28, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5778": { "addr": "172.21.15.1:6834/131367647", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 131367647, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 131367647, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5778, "incarnation": 26, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 7, "modified": "2024-03-06T07:22:07.663851+0000", "qdb_cluster": [ 5778 ], "qdb_leader": 5778, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5778 } } } ], "standbys": [ { "addr": "172.21.15.98:6836/2937068915", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 2937068915, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 2937068915, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 25, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5084, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6837/2828783606", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 2828783606, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2828783606, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 25, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5087, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6835/3845300501", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 3845300501, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 3845300501, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 25, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5344, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:22:16.014 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:22:16.015 INFO:tasks.cephfs.test_failover:current = [{'gid': 5344, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/3845300501', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 3845300501}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 3845300501}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 25}, {'gid': 5084, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/2937068915', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2937068915}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2937068915}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 25}, {'gid': 5778, 'name': 'c', 'rank': 0, 'incarnation': 26, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6834/131367647', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 131367647}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 131367647}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5087, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/2828783606', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 2828783606}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2828783606}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 25}] 2024-03-06T07:22:16.015 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs2 2024-03-06T07:22:16.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.120+0000 7fd8c0381640 1 Processor -- start 2024-03-06T07:22:16.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.120+0000 7fd8c0381640 1 -- start start 2024-03-06T07:22:16.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.121+0000 7fd8c0381640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8b8072bd0 0x7fd8b80c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.121+0000 7fd8c0381640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd8b8152b70 0x7fd8b80c3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.121+0000 7fd8c0381640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b8151dd0 con 0x7fd8b80c2b30 2024-03-06T07:22:16.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.121+0000 7fd8c0381640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b81526e0 con 0x7fd8b8072bd0 2024-03-06T07:22:16.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.121+0000 7fd8c0381640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b8156190 con 0x7fd8b8152b70 2024-03-06T07:22:16.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.121+0000 7fd8bd91c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8b8072bd0 0x7fd8b80c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.121+0000 7fd8be11d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd8b80c2b30 0x7fd8b80724c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51664/0 (socket says 172.21.15.1:51664) 2024-03-06T07:22:16.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.121+0000 7fd8be11d640 1 -- 172.21.15.1:0/1874066360 learned_addr learned my addr 172.21.15.1:0/1874066360 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:16.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.121+0000 7fd8bd11b640 1 -- 172.21.15.1:0/1874066360 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529390074 0 0) 0x7fd8b8151dd0 con 0x7fd8b80c2b30 2024-03-06T07:22:16.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.121+0000 7fd8be91e640 1 --2- 172.21.15.1:0/1874066360 >> v2:172.21.15.98:3300/0 conn(0x7fd8b8152b70 0x7fd8b80c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.121+0000 7fd8bd11b640 1 -- 172.21.15.1:0/1874066360 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8a4003520 con 0x7fd8b80c2b30 2024-03-06T07:22:16.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.121+0000 7fd8bd91c640 1 -- 172.21.15.1:0/1874066360 >> v2:172.21.15.98:3300/0 conn(0x7fd8b8152b70 msgr2=0x7fd8b80c3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:16.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.121+0000 7fd8bd91c640 1 --2- 172.21.15.1:0/1874066360 >> v2:172.21.15.98:3300/0 conn(0x7fd8b8152b70 0x7fd8b80c3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.121+0000 7fd8bd91c640 1 -- 172.21.15.1:0/1874066360 >> v1:172.21.15.1:6789/0 conn(0x7fd8b80c2b30 legacy=0x7fd8b80724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.121+0000 7fd8bd91c640 1 -- 172.21.15.1:0/1874066360 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b8152090 con 0x7fd8b8072bd0 2024-03-06T07:22:16.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.122+0000 7fd8bd91c640 1 --2- 172.21.15.1:0/1874066360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b8072bd0 0x7fd8b80c32f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fd8b806f710 tx=0x7fd8a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d64c1c0ce468710 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.123+0000 7fd8bd11b640 1 -- 172.21.15.1:0/1874066360 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8035020 con 0x7fd8b8072bd0 2024-03-06T07:22:16.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.123+0000 7fd8bd11b640 1 -- 172.21.15.1:0/1874066360 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a8031830 con 0x7fd8b8072bd0 2024-03-06T07:22:16.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.123+0000 7fd8bd11b640 1 -- 172.21.15.1:0/1874066360 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8031c00 con 0x7fd8b8072bd0 2024-03-06T07:22:16.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.123+0000 7fd8c0381640 1 -- 172.21.15.1:0/1874066360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b8072bd0 msgr2=0x7fd8b80c32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.123+0000 7fd8c0381640 1 --2- 172.21.15.1:0/1874066360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b8072bd0 0x7fd8b80c32f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fd8b806f710 tx=0x7fd8a802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.123+0000 7fd8c0381640 1 -- 172.21.15.1:0/1874066360 shutdown_connections 2024-03-06T07:22:16.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.123+0000 7fd8c0381640 1 --2- 172.21.15.1:0/1874066360 >> v2:172.21.15.98:3300/0 conn(0x7fd8b8152b70 0x7fd8b80c3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.123+0000 7fd8c0381640 1 --2- 172.21.15.1:0/1874066360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b8072bd0 0x7fd8b80c32f0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.123+0000 7fd8c0381640 1 -- 172.21.15.1:0/1874066360 >> 172.21.15.1:0/1874066360 conn(0x7fd8b80c6860 msgr2=0x7fd8b80c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.123+0000 7fd8c0381640 1 -- 172.21.15.1:0/1874066360 shutdown_connections 2024-03-06T07:22:16.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.123+0000 7fd8c0381640 1 -- 172.21.15.1:0/1874066360 wait complete. 2024-03-06T07:22:16.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.124+0000 7fd8c0381640 1 Processor -- start 2024-03-06T07:22:16.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.124+0000 7fd8c0381640 1 -- start start 2024-03-06T07:22:16.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.125+0000 7fd8c0381640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8b8072bd0 0x7fd8b81eef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.125+0000 7fd8c0381640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b80c2b30 0x7fd8b80bff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.125+0000 7fd8c0381640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8b8152b70 0x7fd8b815b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.125+0000 7fd8c0381640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b80c3ef0 con 0x7fd8b80c2b30 2024-03-06T07:22:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.125+0000 7fd8c0381640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b80c3d70 con 0x7fd8b8072bd0 2024-03-06T07:22:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.125+0000 7fd8c0381640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b8155d80 con 0x7fd8b8152b70 2024-03-06T07:22:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.125+0000 7fd8bd91c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b80c2b30 0x7fd8b80bff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.125+0000 7fd8bd91c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b80c2b30 0x7fd8b80bff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37412/0 (socket says 172.21.15.1:37412) 2024-03-06T07:22:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.125+0000 7fd8bd91c640 1 -- 172.21.15.1:0/540011589 learned_addr learned my addr 172.21.15.1:0/540011589 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.125+0000 7fd8be11d640 1 --2- 172.21.15.1:0/540011589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8b8072bd0 0x7fd8b81eef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.125+0000 7fd8be91e640 1 --2- 172.21.15.1:0/540011589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8b8152b70 0x7fd8b815b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.125+0000 7fd8bd91c640 1 -- 172.21.15.1:0/540011589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8b8152b70 msgr2=0x7fd8b815b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.125+0000 7fd8bd91c640 1 --2- 172.21.15.1:0/540011589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8b8152b70 0x7fd8b815b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.125+0000 7fd8bd91c640 1 -- 172.21.15.1:0/540011589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8b8072bd0 msgr2=0x7fd8b81eef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.125+0000 7fd8bd91c640 1 --2- 172.21.15.1:0/540011589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8b8072bd0 0x7fd8b81eef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.125+0000 7fd8bd91c640 1 -- 172.21.15.1:0/540011589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b81568c0 con 0x7fd8b80c2b30 2024-03-06T07:22:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.126+0000 7fd8bd91c640 1 --2- 172.21.15.1:0/540011589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b80c2b30 0x7fd8b80bff00 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fd8a802bee0 tx=0x7fd8a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.126+0000 7fd8aeffd640 1 -- 172.21.15.1:0/540011589 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8041070 con 0x7fd8b80c2b30 2024-03-06T07:22:16.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.126+0000 7fd8aeffd640 1 -- 172.21.15.1:0/540011589 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a803f070 con 0x7fd8b80c2b30 2024-03-06T07:22:16.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.126+0000 7fd8c0381640 1 -- 172.21.15.1:0/540011589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b8156ba0 con 0x7fd8b80c2b30 2024-03-06T07:22:16.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.126+0000 7fd8aeffd640 1 -- 172.21.15.1:0/540011589 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8035040 con 0x7fd8b80c2b30 2024-03-06T07:22:16.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.126+0000 7fd8c0381640 1 -- 172.21.15.1:0/540011589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b80c0440 con 0x7fd8b80c2b30 2024-03-06T07:22:16.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.127+0000 7fd8c0381640 1 -- 172.21.15.1:0/540011589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b8067fd0 con 0x7fd8b80c2b30 2024-03-06T07:22:16.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.127+0000 7fd8aeffd640 1 -- 172.21.15.1:0/540011589 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8a8038020 con 0x7fd8b80c2b30 2024-03-06T07:22:16.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.128+0000 7fd8aeffd640 1 --2- 172.21.15.1:0/540011589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd89c06b4d0 0x7fd89c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.128+0000 7fd8aeffd640 1 -- 172.21.15.1:0/540011589 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd8a80ab970 con 0x7fd8b80c2b30 2024-03-06T07:22:16.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.128+0000 7fd8be11d640 1 --2- 172.21.15.1:0/540011589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd89c06b4d0 0x7fd89c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.128+0000 7fd8be11d640 1 --2- 172.21.15.1:0/540011589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd89c06b4d0 0x7fd89c06d990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0005880 tx=0x7fd8b0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.132+0000 7fd8aeffd640 1 -- 172.21.15.1:0/540011589 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8a807a650 con 0x7fd8b80c2b30 2024-03-06T07:22:16.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.313+0000 7fd8c0381640 1 -- 172.21.15.1:0/540011589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fd8b8071790 con 0x7fd8b80c2b30 2024-03-06T07:22:16.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.316+0000 7fd8aeffd640 1 -- 172.21.15.1:0/540011589 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v16)=0 v16) v1 ==== 117+0+0 (secure 0 0 0) 0x7fd8a8031800 con 0x7fd8b80c2b30 2024-03-06T07:22:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.318+0000 7fd8c0381640 1 -- 172.21.15.1:0/540011589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd89c06b4d0 msgr2=0x7fd89c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.318+0000 7fd8c0381640 1 --2- 172.21.15.1:0/540011589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd89c06b4d0 0x7fd89c06d990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0005880 tx=0x7fd8b0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.318+0000 7fd8c0381640 1 -- 172.21.15.1:0/540011589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b80c2b30 msgr2=0x7fd8b80bff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.318+0000 7fd8c0381640 1 --2- 172.21.15.1:0/540011589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b80c2b30 0x7fd8b80bff00 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fd8a802bee0 tx=0x7fd8a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.318+0000 7fd8c0381640 1 -- 172.21.15.1:0/540011589 shutdown_connections 2024-03-06T07:22:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.318+0000 7fd8c0381640 1 --2- 172.21.15.1:0/540011589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd89c06b4d0 0x7fd89c06d990 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.318+0000 7fd8c0381640 1 --2- 172.21.15.1:0/540011589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8b8152b70 0x7fd8b815b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.318+0000 7fd8c0381640 1 --2- 172.21.15.1:0/540011589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b80c2b30 0x7fd8b80bff00 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.318+0000 7fd8c0381640 1 --2- 172.21.15.1:0/540011589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8b8072bd0 0x7fd8b81eef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.318+0000 7fd8c0381640 1 -- 172.21.15.1:0/540011589 >> 172.21.15.1:0/540011589 conn(0x7fd8b80c6860 msgr2=0x7fd8b814fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.318+0000 7fd8c0381640 1 -- 172.21.15.1:0/540011589 shutdown_connections 2024-03-06T07:22:16.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.318+0000 7fd8c0381640 1 -- 172.21.15.1:0/540011589 wait complete. 2024-03-06T07:22:16.331 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs2 2024-03-06T07:22:16.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.437+0000 7fae0e3b6640 1 Processor -- start 2024-03-06T07:22:16.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.437+0000 7fae0e3b6640 1 -- start start 2024-03-06T07:22:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.438+0000 7fae0e3b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae080a7050 0x7fae080c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.438+0000 7fae0e3b6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae080c4a20 0x7fae08156460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.438+0000 7fae0e3b6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae08153a00 con 0x7fae081569a0 2024-03-06T07:22:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.438+0000 7fae0e3b6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae08153ce0 con 0x7fae080c4a20 2024-03-06T07:22:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.438+0000 7fae0e3b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae080c3cb0 con 0x7fae080a7050 2024-03-06T07:22:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.438+0000 7fae07fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae080a7050 0x7fae080c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.438+0000 7fae077fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae080c4a20 0x7fae08156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.438+0000 7fae077fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae080c4a20 0x7fae08156460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:36408/0 (socket says 172.21.15.1:36408) 2024-03-06T07:22:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.438+0000 7fae077fe640 1 -- 172.21.15.1:0/1225612466 learned_addr learned my addr 172.21.15.1:0/1225612466 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.438+0000 7fae07fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae080a7050 0x7fae080c44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:44350/0 (socket says 172.21.15.1:44350) 2024-03-06T07:22:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.438+0000 7fae0c953640 1 --1- 172.21.15.1:0/1225612466 >> v1:172.21.15.98:6789/0 conn(0x7fae081569a0 0x7fae0806aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:54870/0 (socket says 172.21.15.1:54870) 2024-03-06T07:22:16.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.439+0000 7fae07fff640 1 -- 172.21.15.1:0/1225612466 >> v2:172.21.15.98:3300/0 conn(0x7fae080c4a20 msgr2=0x7fae08156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:16.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.439+0000 7fae07fff640 1 --2- 172.21.15.1:0/1225612466 >> v2:172.21.15.98:3300/0 conn(0x7fae080c4a20 0x7fae08156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.439+0000 7fae07fff640 1 -- 172.21.15.1:0/1225612466 >> v1:172.21.15.98:6789/0 conn(0x7fae081569a0 legacy=0x7fae0806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.439+0000 7fae07fff640 1 -- 172.21.15.1:0/1225612466 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae080c3f90 con 0x7fae080a7050 2024-03-06T07:22:16.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.439+0000 7fae077fe640 1 --2- 172.21.15.1:0/1225612466 >> v2:172.21.15.98:3300/0 conn(0x7fae080c4a20 0x7fae08156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:16.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.440+0000 7fae07fff640 1 --2- 172.21.15.1:0/1225612466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae080a7050 0x7fae080c44e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fadf80097a0 tx=0x7fadf802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13cd669554de7dae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.440+0000 7fae06ffd640 1 -- 172.21.15.1:0/1225612466 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf8038070 con 0x7fae080a7050 2024-03-06T07:22:16.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.440+0000 7fae06ffd640 1 -- 172.21.15.1:0/1225612466 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf8031700 con 0x7fae080a7050 2024-03-06T07:22:16.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.440+0000 7fae06ffd640 1 -- 172.21.15.1:0/1225612466 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf8031ad0 con 0x7fae080a7050 2024-03-06T07:22:16.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.440+0000 7fae0e3b6640 1 -- 172.21.15.1:0/1225612466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae080a7050 msgr2=0x7fae080c44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.440+0000 7fae0e3b6640 1 --2- 172.21.15.1:0/1225612466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae080a7050 0x7fae080c44e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fadf80097a0 tx=0x7fadf802b7a0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.440+0000 7fae0e3b6640 1 -- 172.21.15.1:0/1225612466 shutdown_connections 2024-03-06T07:22:16.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.440+0000 7fae0e3b6640 1 --2- 172.21.15.1:0/1225612466 >> v2:172.21.15.98:3300/0 conn(0x7fae080c4a20 0x7fae08156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.440+0000 7fae0e3b6640 1 --2- 172.21.15.1:0/1225612466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae080a7050 0x7fae080c44e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.440+0000 7fae0e3b6640 1 -- 172.21.15.1:0/1225612466 >> 172.21.15.1:0/1225612466 conn(0x7fae080c7a50 msgr2=0x7fae080c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.440+0000 7fae0e3b6640 1 -- 172.21.15.1:0/1225612466 shutdown_connections 2024-03-06T07:22:16.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.440+0000 7fae0e3b6640 1 -- 172.21.15.1:0/1225612466 wait complete. 2024-03-06T07:22:16.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.441+0000 7fae0e3b6640 1 Processor -- start 2024-03-06T07:22:16.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.442+0000 7fae0e3b6640 1 -- start start 2024-03-06T07:22:16.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.442+0000 7fae0e3b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae080a7050 0x7fae0814daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.442+0000 7fae0e3b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae080c4a20 0x7fae08155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.442+0000 7fae0e3b6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae081569a0 0x7fae081e9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.442+0000 7fae0e3b6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae08072f60 con 0x7fae081569a0 2024-03-06T07:22:16.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.442+0000 7fae0e3b6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae080c4ea0 con 0x7fae080c4a20 2024-03-06T07:22:16.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.442+0000 7fae0e3b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae080c2450 con 0x7fae080a7050 2024-03-06T07:22:16.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.442+0000 7fae0c953640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae081569a0 0x7fae081e9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.442+0000 7fae0c953640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae081569a0 0x7fae081e9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37416/0 (socket says 172.21.15.1:37416) 2024-03-06T07:22:16.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.442+0000 7fae0c953640 1 -- 172.21.15.1:0/4180121299 learned_addr learned my addr 172.21.15.1:0/4180121299 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:16.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.443+0000 7fae07fff640 1 --2- 172.21.15.1:0/4180121299 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae080a7050 0x7fae0814daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.443+0000 7fae077fe640 1 --2- 172.21.15.1:0/4180121299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae080c4a20 0x7fae08155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.443+0000 7fae0c953640 1 -- 172.21.15.1:0/4180121299 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae080a7050 msgr2=0x7fae0814daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.443+0000 7fae0c953640 1 --2- 172.21.15.1:0/4180121299 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae080a7050 0x7fae0814daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.443+0000 7fae0c953640 1 -- 172.21.15.1:0/4180121299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae080c4a20 msgr2=0x7fae08155370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.443+0000 7fae0c953640 1 --2- 172.21.15.1:0/4180121299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae080c4a20 0x7fae08155370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.443+0000 7fae0c953640 1 -- 172.21.15.1:0/4180121299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae080c1860 con 0x7fae081569a0 2024-03-06T07:22:16.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.443+0000 7fae0c953640 1 --2- 172.21.15.1:0/4180121299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae081569a0 0x7fae081e9310 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fae0000c7b0 tx=0x7fae0000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.443+0000 7fae04ff9640 1 -- 172.21.15.1:0/4180121299 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae00017070 con 0x7fae081569a0 2024-03-06T07:22:16.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.443+0000 7fae04ff9640 1 -- 172.21.15.1:0/4180121299 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0000a640 con 0x7fae081569a0 2024-03-06T07:22:16.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.443+0000 7fae04ff9640 1 -- 172.21.15.1:0/4180121299 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0000a9f0 con 0x7fae081569a0 2024-03-06T07:22:16.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.443+0000 7fae0e3b6640 1 -- 172.21.15.1:0/4180121299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae080c1b40 con 0x7fae081569a0 2024-03-06T07:22:16.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.443+0000 7fae0e3b6640 1 -- 172.21.15.1:0/4180121299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae081e8df0 con 0x7fae081569a0 2024-03-06T07:22:16.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.444+0000 7fae0e3b6640 1 -- 172.21.15.1:0/4180121299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae080686e0 con 0x7fae081569a0 2024-03-06T07:22:16.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.445+0000 7fae04ff9640 1 -- 172.21.15.1:0/4180121299 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae0001a070 con 0x7fae081569a0 2024-03-06T07:22:16.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.445+0000 7fae04ff9640 1 --2- 172.21.15.1:0/4180121299 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fadf406b500 0x7fadf406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.445+0000 7fae04ff9640 1 -- 172.21.15.1:0/4180121299 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fae00089860 con 0x7fae081569a0 2024-03-06T07:22:16.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.446+0000 7fae07fff640 1 --2- 172.21.15.1:0/4180121299 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fadf406b500 0x7fadf406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.446+0000 7fae07fff640 1 --2- 172.21.15.1:0/4180121299 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fadf406b500 0x7fadf406d9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fadf8035000 tx=0x7fadf8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.450+0000 7fae04ff9640 1 -- 172.21.15.1:0/4180121299 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae00057f90 con 0x7fae081569a0 2024-03-06T07:22:16.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.669+0000 7fae0e3b6640 1 -- 172.21.15.1:0/4180121299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fae08072b40 con 0x7fae081569a0 2024-03-06T07:22:16.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.672+0000 7fae04ff9640 1 -- 172.21.15.1:0/4180121299 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v17)=0 v17) v1 ==== 117+0+0 (secure 0 0 0) 0x7fae0005c030 con 0x7fae081569a0 2024-03-06T07:22:16.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.675+0000 7fae0e3b6640 1 -- 172.21.15.1:0/4180121299 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fadf406b500 msgr2=0x7fadf406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.675+0000 7fae0e3b6640 1 --2- 172.21.15.1:0/4180121299 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fadf406b500 0x7fadf406d9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fadf8035000 tx=0x7fadf8008030 comp rx=0 tx=0).stop 2024-03-06T07:22:16.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.675+0000 7fae0e3b6640 1 -- 172.21.15.1:0/4180121299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae081569a0 msgr2=0x7fae081e9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.675+0000 7fae0e3b6640 1 --2- 172.21.15.1:0/4180121299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae081569a0 0x7fae081e9310 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fae0000c7b0 tx=0x7fae0000cc70 comp rx=0 tx=0).stop 2024-03-06T07:22:16.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.675+0000 7fae0e3b6640 1 -- 172.21.15.1:0/4180121299 shutdown_connections 2024-03-06T07:22:16.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.675+0000 7fae0e3b6640 1 --2- 172.21.15.1:0/4180121299 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fadf406b500 0x7fadf406d9c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.675+0000 7fae0e3b6640 1 --2- 172.21.15.1:0/4180121299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae081569a0 0x7fae081e9310 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.675+0000 7fae0e3b6640 1 --2- 172.21.15.1:0/4180121299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae080c4a20 0x7fae08155370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.675+0000 7fae0e3b6640 1 --2- 172.21.15.1:0/4180121299 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae080a7050 0x7fae0814daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.675+0000 7fae0e3b6640 1 -- 172.21.15.1:0/4180121299 >> 172.21.15.1:0/4180121299 conn(0x7fae080c7a50 msgr2=0x7fae0806e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.675+0000 7fae0e3b6640 1 -- 172.21.15.1:0/4180121299 shutdown_connections 2024-03-06T07:22:16.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.675+0000 7fae0e3b6640 1 -- 172.21.15.1:0/4180121299 wait complete. 2024-03-06T07:22:16.687 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.a mds_join_fs cephfs2 2024-03-06T07:22:16.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.790+0000 7f88cd2e1640 1 Processor -- start 2024-03-06T07:22:16.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.790+0000 7f88cd2e1640 1 -- start start 2024-03-06T07:22:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.791+0000 7f88cd2e1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88c8152650 0x7f88c8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.791+0000 7f88cd2e1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f88c80bf9d0 0x7f88c80c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.791+0000 7f88cd2e1640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c80c2fc0 con 0x7f88c80c6b10 2024-03-06T07:22:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.791+0000 7f88cd2e1640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c8155800 con 0x7f88c8152650 2024-03-06T07:22:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.791+0000 7f88cd2e1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c8158da0 con 0x7f88c80bf9d0 2024-03-06T07:22:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.791+0000 7f88bffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f88c80c6b10 0x7f88c8158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51668/0 (socket says 172.21.15.1:51668) 2024-03-06T07:22:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.791+0000 7f88c6ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88c8152650 0x7f88c8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.791+0000 7f88bffff640 1 -- 172.21.15.1:0/511802 learned_addr learned my addr 172.21.15.1:0/511802 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.792+0000 7f88c77fe640 1 --2- 172.21.15.1:0/511802 >> v2:172.21.15.98:3300/0 conn(0x7f88c80bf9d0 0x7f88c80c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.792+0000 7f88c6ffd640 1 -- 172.21.15.1:0/511802 >> v2:172.21.15.98:3300/0 conn(0x7f88c80bf9d0 msgr2=0x7f88c80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.792+0000 7f88c6ffd640 1 --2- 172.21.15.1:0/511802 >> v2:172.21.15.98:3300/0 conn(0x7f88c80bf9d0 0x7f88c80c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.792+0000 7f88c6ffd640 1 -- 172.21.15.1:0/511802 >> v1:172.21.15.1:6789/0 conn(0x7f88c80c6b10 legacy=0x7f88c8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.792+0000 7f88c6ffd640 1 -- 172.21.15.1:0/511802 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c80c6690 con 0x7f88c8152650 2024-03-06T07:22:16.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.792+0000 7f88c67fc640 1 -- 172.21.15.1:0/511802 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526875463 0 0) 0x7f88c80c2fc0 con 0x7f88c80c6b10 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.792+0000 7f88c6ffd640 1 --2- 172.21.15.1:0/511802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88c8152650 0x7f88c8155c90 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f88c81597b0 tx=0x7f88b802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1203d303aa161da4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.793+0000 7f88c67fc640 1 -- 172.21.15.1:0/511802 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b80315a0 con 0x7f88c8152650 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.793+0000 7f88c67fc640 1 -- 172.21.15.1:0/511802 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b8037070 con 0x7f88c8152650 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.793+0000 7f88c67fc640 1 -- 172.21.15.1:0/511802 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8031960 con 0x7f88c8152650 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.793+0000 7f88cd2e1640 1 -- 172.21.15.1:0/511802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88c8152650 msgr2=0x7f88c8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.793+0000 7f88cd2e1640 1 --2- 172.21.15.1:0/511802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88c8152650 0x7f88c8155c90 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f88c81597b0 tx=0x7f88b802b840 comp rx=0 tx=0).stop 2024-03-06T07:22:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.793+0000 7f88cd2e1640 1 -- 172.21.15.1:0/511802 shutdown_connections 2024-03-06T07:22:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.793+0000 7f88cd2e1640 1 --2- 172.21.15.1:0/511802 >> v2:172.21.15.98:3300/0 conn(0x7f88c80bf9d0 0x7f88c80c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.793+0000 7f88cd2e1640 1 --2- 172.21.15.1:0/511802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88c8152650 0x7f88c8155c90 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.793+0000 7f88cd2e1640 1 -- 172.21.15.1:0/511802 >> 172.21.15.1:0/511802 conn(0x7f88c8066ee0 msgr2=0x7f88c80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.793+0000 7f88cd2e1640 1 -- 172.21.15.1:0/511802 shutdown_connections 2024-03-06T07:22:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.793+0000 7f88cd2e1640 1 -- 172.21.15.1:0/511802 wait complete. 2024-03-06T07:22:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.794+0000 7f88cd2e1640 1 Processor -- start 2024-03-06T07:22:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.794+0000 7f88cd2e1640 1 -- start start 2024-03-06T07:22:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.795+0000 7f88cd2e1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c80bf9d0 0x7f88c814b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.795+0000 7f88cd2e1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88c8152650 0x7f88c81e18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.795+0000 7f88cd2e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88c81e1e30 0x7f88c80c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.795+0000 7f88cd2e1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c81551b0 con 0x7f88c8152650 2024-03-06T07:22:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.795+0000 7f88cd2e1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80c6fa0 con 0x7f88c81e1e30 2024-03-06T07:22:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.795+0000 7f88cd2e1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c8155490 con 0x7f88c80bf9d0 2024-03-06T07:22:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.795+0000 7f88bffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88c8152650 0x7f88c81e18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.795+0000 7f88bffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88c8152650 0x7f88c81e18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37440/0 (socket says 172.21.15.1:37440) 2024-03-06T07:22:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.795+0000 7f88bffff640 1 -- 172.21.15.1:0/1581912811 learned_addr learned my addr 172.21.15.1:0/1581912811 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.795+0000 7f88c77fe640 1 --2- 172.21.15.1:0/1581912811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88c81e1e30 0x7f88c80c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.795+0000 7f88c6ffd640 1 --2- 172.21.15.1:0/1581912811 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c80bf9d0 0x7f88c814b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.795+0000 7f88bffff640 1 -- 172.21.15.1:0/1581912811 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c80bf9d0 msgr2=0x7f88c814b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.795+0000 7f88bffff640 1 --2- 172.21.15.1:0/1581912811 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c80bf9d0 0x7f88c814b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.795+0000 7f88bffff640 1 -- 172.21.15.1:0/1581912811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88c81e1e30 msgr2=0x7f88c80c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.795+0000 7f88bffff640 1 --2- 172.21.15.1:0/1581912811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88c81e1e30 0x7f88c80c9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.795+0000 7f88bffff640 1 -- 172.21.15.1:0/1581912811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c81ead50 con 0x7f88c8152650 2024-03-06T07:22:16.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.795+0000 7f88c6ffd640 1 --2- 172.21.15.1:0/1581912811 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c80bf9d0 0x7f88c814b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.796+0000 7f88bffff640 1 --2- 172.21.15.1:0/1581912811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88c8152650 0x7f88c81e18f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f88b000c500 tx=0x7f88b000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.796+0000 7f88bf7fe640 1 -- 172.21.15.1:0/1581912811 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b0018070 con 0x7f88c8152650 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.796+0000 7f88bf7fe640 1 -- 172.21.15.1:0/1581912811 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b0007580 con 0x7f88c8152650 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.796+0000 7f88cd2e1640 1 -- 172.21.15.1:0/1581912811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c81eb030 con 0x7f88c8152650 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.796+0000 7f88bf7fe640 1 -- 172.21.15.1:0/1581912811 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b0004a20 con 0x7f88c8152650 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.796+0000 7f88cd2e1640 1 -- 172.21.15.1:0/1581912811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c81eb5f0 con 0x7f88c8152650 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.797+0000 7f88cd2e1640 1 -- 172.21.15.1:0/1581912811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c8067fd0 con 0x7f88c8152650 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.797+0000 7f88bf7fe640 1 -- 172.21.15.1:0/1581912811 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88b001b070 con 0x7f88c8152650 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.798+0000 7f88bf7fe640 1 --2- 172.21.15.1:0/1581912811 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88a806b500 0x7f88a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.798+0000 7f88bf7fe640 1 -- 172.21.15.1:0/1581912811 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f88b00895e0 con 0x7f88c8152650 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.798+0000 7f88c6ffd640 1 --2- 172.21.15.1:0/1581912811 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88a806b500 0x7f88a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.798+0000 7f88c6ffd640 1 --2- 172.21.15.1:0/1581912811 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88a806b500 0x7f88a806d9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f88b8036000 tx=0x7f88b8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.802+0000 7f88bf7fe640 1 -- 172.21.15.1:0/1581912811 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88b0058360 con 0x7f88c8152650 2024-03-06T07:22:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.986+0000 7f88cd2e1640 1 -- 172.21.15.1:0/1581912811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f88c80b06f0 con 0x7f88c8152650 2024-03-06T07:22:16.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.989+0000 7f88bf7fe640 1 -- 172.21.15.1:0/1581912811 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v18)=0 v18) v1 ==== 117+0+0 (secure 0 0 0) 0x7f88b0057c60 con 0x7f88c8152650 2024-03-06T07:22:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.991+0000 7f88cd2e1640 1 -- 172.21.15.1:0/1581912811 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88a806b500 msgr2=0x7f88a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.991+0000 7f88cd2e1640 1 --2- 172.21.15.1:0/1581912811 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88a806b500 0x7f88a806d9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f88b8036000 tx=0x7f88b8008030 comp rx=0 tx=0).stop 2024-03-06T07:22:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.992+0000 7f88cd2e1640 1 -- 172.21.15.1:0/1581912811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88c8152650 msgr2=0x7f88c81e18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.992+0000 7f88cd2e1640 1 --2- 172.21.15.1:0/1581912811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88c8152650 0x7f88c81e18f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f88b000c500 tx=0x7f88b000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.992+0000 7f88cd2e1640 1 -- 172.21.15.1:0/1581912811 shutdown_connections 2024-03-06T07:22:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.992+0000 7f88cd2e1640 1 --2- 172.21.15.1:0/1581912811 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88a806b500 0x7f88a806d9c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.992+0000 7f88cd2e1640 1 --2- 172.21.15.1:0/1581912811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88c81e1e30 0x7f88c80c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.992+0000 7f88cd2e1640 1 --2- 172.21.15.1:0/1581912811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88c8152650 0x7f88c81e18f0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.992+0000 7f88cd2e1640 1 --2- 172.21.15.1:0/1581912811 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c80bf9d0 0x7f88c814b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.992+0000 7f88cd2e1640 1 -- 172.21.15.1:0/1581912811 >> 172.21.15.1:0/1581912811 conn(0x7f88c8066ee0 msgr2=0x7f88c80c5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.992+0000 7f88cd2e1640 1 -- 172.21.15.1:0/1581912811 shutdown_connections 2024-03-06T07:22:16.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:16.992+0000 7f88cd2e1640 1 -- 172.21.15.1:0/1581912811 wait complete. 2024-03-06T07:22:17.008 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:17.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.107+0000 7f3451433640 1 Processor -- start 2024-03-06T07:22:17.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.107+0000 7f3451433640 1 -- start start 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.108+0000 7f3451433640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f344c06d060 0x7f344c1558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.108+0000 7f3451433640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f344c0af640 0x7f344c155df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.108+0000 7f3451433640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f344c0c71c0 con 0x7f344c0c3eb0 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.108+0000 7f3451433640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f344c0c59b0 con 0x7f344c06d060 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.108+0000 7f3451433640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344c0c4c00 con 0x7f344c0af640 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.108+0000 7f344b7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f344c0af640 0x7f344c155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.108+0000 7f3443fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f344c06d060 0x7f344c1558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.109+0000 7f3443fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f344c06d060 0x7f344c1558b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:36458/0 (socket says 172.21.15.1:36458) 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.109+0000 7f344b7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f344c0af640 0x7f344c155df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:44378/0 (socket says 172.21.15.1:44378) 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.109+0000 7f3443fff640 1 -- 172.21.15.1:0/4253935541 learned_addr learned my addr 172.21.15.1:0/4253935541 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.109+0000 7f344affd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f344c0c3eb0 0x7f344c0c7400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:54882/0 (socket says 172.21.15.1:54882) 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.109+0000 7f3443fff640 1 -- 172.21.15.1:0/4253935541 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f344c0af640 msgr2=0x7f344c155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.109+0000 7f3443fff640 1 --2- 172.21.15.1:0/4253935541 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f344c0af640 0x7f344c155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.109+0000 7f3443fff640 1 -- 172.21.15.1:0/4253935541 >> v1:172.21.15.98:6789/0 conn(0x7f344c0c3eb0 legacy=0x7f344c0c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.109+0000 7f3443fff640 1 -- 172.21.15.1:0/4253935541 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344c0c4e50 con 0x7f344c06d060 2024-03-06T07:22:17.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.109+0000 7f344b7fe640 1 --2- 172.21.15.1:0/4253935541 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f344c0af640 0x7f344c155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.110+0000 7f3443fff640 1 --2- 172.21.15.1:0/4253935541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f344c06d060 0x7f344c1558b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3434009870 tx=0x7f343402b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=199af5bdb28b9ed7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.110+0000 7f344a7fc640 1 -- 172.21.15.1:0/4253935541 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34340317c0 con 0x7f344c06d060 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.110+0000 7f344a7fc640 1 -- 172.21.15.1:0/4253935541 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3434037070 con 0x7f344c06d060 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.110+0000 7f344a7fc640 1 -- 172.21.15.1:0/4253935541 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3434031b80 con 0x7f344c06d060 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.110+0000 7f3451433640 1 -- 172.21.15.1:0/4253935541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f344c06d060 msgr2=0x7f344c1558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.110+0000 7f3451433640 1 --2- 172.21.15.1:0/4253935541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f344c06d060 0x7f344c1558b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3434009870 tx=0x7f343402b860 comp rx=0 tx=0).stop 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.110+0000 7f3451433640 1 -- 172.21.15.1:0/4253935541 shutdown_connections 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.110+0000 7f3451433640 1 --2- 172.21.15.1:0/4253935541 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f344c0af640 0x7f344c155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.110+0000 7f3451433640 1 --2- 172.21.15.1:0/4253935541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f344c06d060 0x7f344c1558b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.110+0000 7f3451433640 1 -- 172.21.15.1:0/4253935541 >> 172.21.15.1:0/4253935541 conn(0x7f344c066d00 msgr2=0x7f344c067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:17.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.111+0000 7f3451433640 1 -- 172.21.15.1:0/4253935541 shutdown_connections 2024-03-06T07:22:17.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.111+0000 7f3451433640 1 -- 172.21.15.1:0/4253935541 wait complete. 2024-03-06T07:22:17.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.112+0000 7f3451433640 1 Processor -- start 2024-03-06T07:22:17.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.112+0000 7f3451433640 1 -- start start 2024-03-06T07:22:17.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.112+0000 7f3451433640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f344c06d060 0x7f344c1eeae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.112+0000 7f3451433640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f344c0af640 0x7f344c1e0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.112+0000 7f3451433640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f344c0c3eb0 0x7f344c1517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.112+0000 7f3451433640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c0c7b10 con 0x7f344c0c3eb0 2024-03-06T07:22:17.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.112+0000 7f3451433640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c155420 con 0x7f344c0af640 2024-03-06T07:22:17.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.112+0000 7f3451433640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344c0c7c90 con 0x7f344c06d060 2024-03-06T07:22:17.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.112+0000 7f344b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f344c0c3eb0 0x7f344c1517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.112+0000 7f344b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f344c0c3eb0 0x7f344c1517a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37446/0 (socket says 172.21.15.1:37446) 2024-03-06T07:22:17.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.112+0000 7f344b7fe640 1 -- 172.21.15.1:0/1546463423 learned_addr learned my addr 172.21.15.1:0/1546463423 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:17.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.112+0000 7f344affd640 1 --2- 172.21.15.1:0/1546463423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f344c06d060 0x7f344c1eeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.113+0000 7f3443fff640 1 --2- 172.21.15.1:0/1546463423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f344c0af640 0x7f344c1e0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.113+0000 7f344b7fe640 1 -- 172.21.15.1:0/1546463423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f344c06d060 msgr2=0x7f344c1eeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.113+0000 7f344b7fe640 1 --2- 172.21.15.1:0/1546463423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f344c06d060 0x7f344c1eeae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.113+0000 7f344b7fe640 1 -- 172.21.15.1:0/1546463423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f344c0af640 msgr2=0x7f344c1e0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.113+0000 7f344b7fe640 1 --2- 172.21.15.1:0/1546463423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f344c0af640 0x7f344c1e0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.113+0000 7f344b7fe640 1 -- 172.21.15.1:0/1546463423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344c1e11d0 con 0x7f344c0c3eb0 2024-03-06T07:22:17.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.113+0000 7f344b7fe640 1 --2- 172.21.15.1:0/1546463423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f344c0c3eb0 0x7f344c1517a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3444009e60 tx=0x7f344400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.113+0000 7f34437fe640 1 -- 172.21.15.1:0/1546463423 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3444018030 con 0x7f344c0c3eb0 2024-03-06T07:22:17.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.113+0000 7f34437fe640 1 -- 172.21.15.1:0/1546463423 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344400cdf0 con 0x7f344c0c3eb0 2024-03-06T07:22:17.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.113+0000 7f3451433640 1 -- 172.21.15.1:0/1546463423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f344c1e14b0 con 0x7f344c0c3eb0 2024-03-06T07:22:17.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.113+0000 7f34437fe640 1 -- 172.21.15.1:0/1546463423 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3444003bb0 con 0x7f344c0c3eb0 2024-03-06T07:22:17.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.113+0000 7f3451433640 1 -- 172.21.15.1:0/1546463423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f344c1ee5c0 con 0x7f344c0c3eb0 2024-03-06T07:22:17.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.114+0000 7f3451433640 1 -- 172.21.15.1:0/1546463423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344c067fd0 con 0x7f344c0c3eb0 2024-03-06T07:22:17.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.115+0000 7f34437fe640 1 -- 172.21.15.1:0/1546463423 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f344401b070 con 0x7f344c0c3eb0 2024-03-06T07:22:17.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.115+0000 7f34437fe640 1 --2- 172.21.15.1:0/1546463423 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f342c06b4d0 0x7f342c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.115+0000 7f34437fe640 1 -- 172.21.15.1:0/1546463423 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f344408ab70 con 0x7f344c0c3eb0 2024-03-06T07:22:17.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.115+0000 7f344affd640 1 --2- 172.21.15.1:0/1546463423 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f342c06b4d0 0x7f342c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.116+0000 7f344affd640 1 --2- 172.21.15.1:0/1546463423 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f342c06b4d0 0x7f342c06d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f343c007fd0 tx=0x7f343c007e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.119+0000 7f34437fe640 1 -- 172.21.15.1:0/1546463423 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3444010040 con 0x7f344c0c3eb0 2024-03-06T07:22:17.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.302+0000 7f3451433640 1 -- 172.21.15.1:0/1546463423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f344c0c0a80 con 0x7f344c0c3eb0 2024-03-06T07:22:17.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.303+0000 7f34437fe640 1 -- 172.21.15.1:0/1546463423 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16822 (secure 0 0 0) 0x7f34440592a0 con 0x7f344c0c3eb0 2024-03-06T07:22:17.305 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:17.305 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":52,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:07.309375+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:03.930333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:04.634569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:17.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.305+0000 7f3451433640 1 -- 172.21.15.1:0/1546463423 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f342c06b4d0 msgr2=0x7f342c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.305+0000 7f3451433640 1 --2- 172.21.15.1:0/1546463423 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f342c06b4d0 0x7f342c06d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f343c007fd0 tx=0x7f343c007e80 comp rx=0 tx=0).stop 2024-03-06T07:22:17.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.305+0000 7f3451433640 1 -- 172.21.15.1:0/1546463423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f344c0c3eb0 msgr2=0x7f344c1517a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.305+0000 7f3451433640 1 --2- 172.21.15.1:0/1546463423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f344c0c3eb0 0x7f344c1517a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3444009e60 tx=0x7f344400c490 comp rx=0 tx=0).stop 2024-03-06T07:22:17.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.305+0000 7f3451433640 1 -- 172.21.15.1:0/1546463423 shutdown_connections 2024-03-06T07:22:17.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.305+0000 7f3451433640 1 --2- 172.21.15.1:0/1546463423 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f342c06b4d0 0x7f342c06d990 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.305+0000 7f3451433640 1 --2- 172.21.15.1:0/1546463423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f344c0c3eb0 0x7f344c1517a0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.305+0000 7f3451433640 1 --2- 172.21.15.1:0/1546463423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f344c0af640 0x7f344c1e0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.305+0000 7f3451433640 1 --2- 172.21.15.1:0/1546463423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f344c06d060 0x7f344c1eeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.305+0000 7f3451433640 1 -- 172.21.15.1:0/1546463423 >> 172.21.15.1:0/1546463423 conn(0x7f344c066d00 msgr2=0x7f344c1557b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:17.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.305+0000 7f3451433640 1 -- 172.21.15.1:0/1546463423 shutdown_connections 2024-03-06T07:22:17.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.305+0000 7f3451433640 1 -- 172.21.15.1:0/1546463423 wait complete. 2024-03-06T07:22:17.318 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:03.930333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:04.634569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:17.318 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:17.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.420+0000 7f347655e640 1 Processor -- start 2024-03-06T07:22:17.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.420+0000 7f347655e640 1 -- start start 2024-03-06T07:22:17.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.421+0000 7f347655e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3470152660 0x7f34700a9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f347655e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34700c3030 con 0x7f347006f800 2024-03-06T07:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f347655e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34700c3310 con 0x7f3470152a40 2024-03-06T07:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f347655e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34700c35f0 con 0x7f3470152660 2024-03-06T07:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f347006f800 0x7f34700c10b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51674/0 (socket says 172.21.15.1:51674) 2024-03-06T07:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346f7fe640 1 -- 172.21.15.1:0/2128660078 learned_addr learned my addr 172.21.15.1:0/2128660078 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346effd640 1 -- 172.21.15.1:0/2128660078 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111958283 0 0) 0x7f34700c3030 con 0x7f347006f800 2024-03-06T07:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346ffff640 1 --2- 172.21.15.1:0/2128660078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3470152660 0x7f34700a9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346effd640 1 -- 172.21.15.1:0/2128660078 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3458003520 con 0x7f347006f800 2024-03-06T07:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346effd640 1 -- 172.21.15.1:0/2128660078 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2804939037 0 0) 0x7f3458003520 con 0x7f347006f800 2024-03-06T07:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346effd640 1 -- 172.21.15.1:0/2128660078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3470152660 msgr2=0x7f34700a9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346effd640 1 --2- 172.21.15.1:0/2128660078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3470152660 0x7f34700a9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346effd640 1 -- 172.21.15.1:0/2128660078 >> v1:172.21.15.98:6789/0 conn(0x7f3470152a40 legacy=0x7f3470155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346effd640 1 -- 172.21.15.1:0/2128660078 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34700c38d0 con 0x7f347006f800 2024-03-06T07:22:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346effd640 1 -- 172.21.15.1:0/2128660078 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f3464001670 con 0x7f347006f800 2024-03-06T07:22:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346effd640 1 -- 172.21.15.1:0/2128660078 >> v1:172.21.15.1:6789/0 conn(0x7f347006f800 legacy=0x7f34700c10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346effd640 1 --2- 172.21.15.1:0/2128660078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3470152660 0x7f34580039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346effd640 1 --2- 172.21.15.1:0/2128660078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3458003f60 0x7f3458004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346effd640 1 --2- 172.21.15.1:0/2128660078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3458004d40 0x7f34580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346effd640 1 -- 172.21.15.1:0/2128660078 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34700c35f0 con 0x7f3458004d40 2024-03-06T07:22:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346effd640 1 -- 172.21.15.1:0/2128660078 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345800bb50 con 0x7f3458003f60 2024-03-06T07:22:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.422+0000 7f346effd640 1 -- 172.21.15.1:0/2128660078 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345800bd50 con 0x7f3470152660 2024-03-06T07:22:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.423+0000 7f3474afb640 1 --2- 172.21.15.1:0/2128660078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3458004d40 0x7f34580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.423+0000 7f346ffff640 1 --2- 172.21.15.1:0/2128660078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3470152660 0x7f34580039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.423+0000 7f346f7fe640 1 --2- 172.21.15.1:0/2128660078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3458003f60 0x7f3458004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.423+0000 7f346ffff640 1 -- 172.21.15.1:0/2128660078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3458003f60 msgr2=0x7f3458004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.423+0000 7f346ffff640 1 --2- 172.21.15.1:0/2128660078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3458003f60 0x7f3458004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.423+0000 7f346ffff640 1 -- 172.21.15.1:0/2128660078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3458004d40 msgr2=0x7f34580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.423+0000 7f346ffff640 1 --2- 172.21.15.1:0/2128660078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3458004d40 0x7f34580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.423+0000 7f346ffff640 1 -- 172.21.15.1:0/2128660078 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3458003520 con 0x7f3470152660 2024-03-06T07:22:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.423+0000 7f3474afb640 1 --2- 172.21.15.1:0/2128660078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3458004d40 0x7f34580051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.424+0000 7f346ffff640 1 --2- 172.21.15.1:0/2128660078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3470152660 0x7f34580039e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3460009af0 tx=0x7f3458005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f471449a1edc6f63 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.424+0000 7f346effd640 1 -- 172.21.15.1:0/2128660078 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3460039070 con 0x7f3470152660 2024-03-06T07:22:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.424+0000 7f346effd640 1 -- 172.21.15.1:0/2128660078 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346002c710 con 0x7f3470152660 2024-03-06T07:22:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.424+0000 7f346effd640 1 -- 172.21.15.1:0/2128660078 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3460009690 con 0x7f3470152660 2024-03-06T07:22:17.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.424+0000 7f347655e640 1 -- 172.21.15.1:0/2128660078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3470152660 msgr2=0x7f34580039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.424+0000 7f347655e640 1 --2- 172.21.15.1:0/2128660078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3470152660 0x7f34580039e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3460009af0 tx=0x7f3458005830 comp rx=0 tx=0).stop 2024-03-06T07:22:17.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.424+0000 7f3474afb640 1 -- 172.21.15.1:0/2128660078 reap_dead start 2024-03-06T07:22:17.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.424+0000 7f347655e640 1 -- 172.21.15.1:0/2128660078 shutdown_connections 2024-03-06T07:22:17.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.424+0000 7f347655e640 1 -- 172.21.15.1:0/2128660078 >> 172.21.15.1:0/2128660078 conn(0x7f34700c3cc0 msgr2=0x7f34700c40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:17.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.424+0000 7f347655e640 1 -- 172.21.15.1:0/2128660078 shutdown_connections 2024-03-06T07:22:17.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.424+0000 7f347655e640 1 -- 172.21.15.1:0/2128660078 wait complete. 2024-03-06T07:22:17.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.425+0000 7f347655e640 1 Processor -- start 2024-03-06T07:22:17.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.426+0000 7f347655e640 1 -- start start 2024-03-06T07:22:17.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.426+0000 7f347655e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3470152660 0x7f347014d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.426+0000 7f347655e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3470150bd0 0x7f34701537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.426+0000 7f347655e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f347014c4e0 0x7f347014c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.426+0000 7f347655e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34700c17c0 con 0x7f3470150bd0 2024-03-06T07:22:17.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.426+0000 7f347655e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34700a84b0 con 0x7f3470152660 2024-03-06T07:22:17.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.426+0000 7f347655e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34700c1940 con 0x7f347014c4e0 2024-03-06T07:22:17.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.426+0000 7f346f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3470150bd0 0x7f34701537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.426+0000 7f346f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3470150bd0 0x7f34701537b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37464/0 (socket says 172.21.15.1:37464) 2024-03-06T07:22:17.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.426+0000 7f346f7fe640 1 -- 172.21.15.1:0/1748220482 learned_addr learned my addr 172.21.15.1:0/1748220482 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:17.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.427+0000 7f346ffff640 1 --2- 172.21.15.1:0/1748220482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3470152660 0x7f347014d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.427+0000 7f3474afb640 1 --2- 172.21.15.1:0/1748220482 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f347014c4e0 0x7f347014c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.427+0000 7f346f7fe640 1 -- 172.21.15.1:0/1748220482 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f347014c4e0 msgr2=0x7f347014c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.427+0000 7f346f7fe640 1 --2- 172.21.15.1:0/1748220482 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f347014c4e0 0x7f347014c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.427+0000 7f346f7fe640 1 -- 172.21.15.1:0/1748220482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3470152660 msgr2=0x7f347014d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.427+0000 7f346f7fe640 1 --2- 172.21.15.1:0/1748220482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3470152660 0x7f347014d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.427+0000 7f346f7fe640 1 -- 172.21.15.1:0/1748220482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3470151050 con 0x7f3470150bd0 2024-03-06T07:22:17.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.427+0000 7f3474afb640 1 --2- 172.21.15.1:0/1748220482 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f347014c4e0 0x7f347014c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:17.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.427+0000 7f346f7fe640 1 --2- 172.21.15.1:0/1748220482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3470150bd0 0x7f34701537b0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f3464007370 tx=0x7f3464007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.427+0000 7f346cff9640 1 -- 172.21.15.1:0/1748220482 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346401a070 con 0x7f3470150bd0 2024-03-06T07:22:17.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.427+0000 7f346cff9640 1 -- 172.21.15.1:0/1748220482 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346400f8e0 con 0x7f3470150bd0 2024-03-06T07:22:17.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.427+0000 7f347655e640 1 -- 172.21.15.1:0/1748220482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34701ee970 con 0x7f3470150bd0 2024-03-06T07:22:17.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.427+0000 7f346cff9640 1 -- 172.21.15.1:0/1748220482 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346400fcb0 con 0x7f3470150bd0 2024-03-06T07:22:17.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.427+0000 7f347655e640 1 -- 172.21.15.1:0/1748220482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347014fdd0 con 0x7f3470150bd0 2024-03-06T07:22:17.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.428+0000 7f347655e640 1 -- 172.21.15.1:0/1748220482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34700686e0 con 0x7f3470150bd0 2024-03-06T07:22:17.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.429+0000 7f346cff9640 1 -- 172.21.15.1:0/1748220482 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f346401d070 con 0x7f3470150bd0 2024-03-06T07:22:17.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.429+0000 7f346cff9640 1 --2- 172.21.15.1:0/1748220482 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f345c06b4d0 0x7f345c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.429+0000 7f346cff9640 1 -- 172.21.15.1:0/1748220482 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3464089bd0 con 0x7f3470150bd0 2024-03-06T07:22:17.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.429+0000 7f346ffff640 1 --2- 172.21.15.1:0/1748220482 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f345c06b4d0 0x7f345c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.430+0000 7f346ffff640 1 --2- 172.21.15.1:0/1748220482 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f345c06b4d0 0x7f345c06d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3460036000 tx=0x7f3460030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.433+0000 7f346cff9640 1 -- 172.21.15.1:0/1748220482 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3464058270 con 0x7f3470150bd0 2024-03-06T07:22:17.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.614+0000 7f347655e640 1 -- 172.21.15.1:0/1748220482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f347006ea10 con 0x7f3470150bd0 2024-03-06T07:22:17.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.614+0000 7f346cff9640 1 -- 172.21.15.1:0/1748220482 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7f346405c310 con 0x7f3470150bd0 2024-03-06T07:22:17.616 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:22:17.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.616+0000 7f347655e640 1 -- 172.21.15.1:0/1748220482 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f345c06b4d0 msgr2=0x7f345c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.616+0000 7f347655e640 1 --2- 172.21.15.1:0/1748220482 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f345c06b4d0 0x7f345c06d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3460036000 tx=0x7f3460030000 comp rx=0 tx=0).stop 2024-03-06T07:22:17.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.616+0000 7f347655e640 1 -- 172.21.15.1:0/1748220482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3470150bd0 msgr2=0x7f34701537b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.616+0000 7f347655e640 1 --2- 172.21.15.1:0/1748220482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3470150bd0 0x7f34701537b0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f3464007370 tx=0x7f3464007830 comp rx=0 tx=0).stop 2024-03-06T07:22:17.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.616+0000 7f347655e640 1 -- 172.21.15.1:0/1748220482 shutdown_connections 2024-03-06T07:22:17.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.616+0000 7f347655e640 1 --2- 172.21.15.1:0/1748220482 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f345c06b4d0 0x7f345c06d990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.616+0000 7f347655e640 1 --2- 172.21.15.1:0/1748220482 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f347014c4e0 0x7f347014c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.616+0000 7f347655e640 1 --2- 172.21.15.1:0/1748220482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3470150bd0 0x7f34701537b0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.616+0000 7f347655e640 1 --2- 172.21.15.1:0/1748220482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3470152660 0x7f347014d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.616+0000 7f347655e640 1 -- 172.21.15.1:0/1748220482 >> 172.21.15.1:0/1748220482 conn(0x7f34700c3cc0 msgr2=0x7f34700c40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:17.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.616+0000 7f347655e640 1 -- 172.21.15.1:0/1748220482 shutdown_connections 2024-03-06T07:22:17.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.616+0000 7f347655e640 1 -- 172.21.15.1:0/1748220482 wait complete. 2024-03-06T07:22:17.629 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:17.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.732+0000 7ff250b1e640 1 Processor -- start 2024-03-06T07:22:17.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.732+0000 7ff250b1e640 1 -- start start 2024-03-06T07:22:17.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.733+0000 7ff250b1e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff24c150520 0x7ff24c1543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.733+0000 7ff250b1e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff24c0c32f0 0x7ff24c1537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.733+0000 7ff250b1e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c150900 con 0x7ff24c070c80 2024-03-06T07:22:17.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.733+0000 7ff250b1e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff24c071790 con 0x7ff24c150520 2024-03-06T07:22:17.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.733+0000 7ff250b1e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c157050 con 0x7ff24c0c32f0 2024-03-06T07:22:17.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.733+0000 7ff24a59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff24c150520 0x7ff24c1543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.733+0000 7ff243fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff24c0c32f0 0x7ff24c1537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.733+0000 7ff243fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff24c0c32f0 0x7ff24c1537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:44428/0 (socket says 172.21.15.1:44428) 2024-03-06T07:22:17.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.733+0000 7ff24a59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff24c150520 0x7ff24c1543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:36494/0 (socket says 172.21.15.1:36494) 2024-03-06T07:22:17.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.733+0000 7ff243fff640 1 -- 172.21.15.1:0/166362762 learned_addr learned my addr 172.21.15.1:0/166362762 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:17.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.733+0000 7ff24ad9d640 1 --1- 172.21.15.1:0/166362762 >> v1:172.21.15.98:6789/0 conn(0x7ff24c070c80 0x7ff24c0c67f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:54898/0 (socket says 172.21.15.1:54898) 2024-03-06T07:22:17.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.734+0000 7ff243fff640 1 -- 172.21.15.1:0/166362762 >> v2:172.21.15.98:3300/0 conn(0x7ff24c150520 msgr2=0x7ff24c1543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:17.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.734+0000 7ff243fff640 1 --2- 172.21.15.1:0/166362762 >> v2:172.21.15.98:3300/0 conn(0x7ff24c150520 0x7ff24c1543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.734+0000 7ff243fff640 1 -- 172.21.15.1:0/166362762 >> v1:172.21.15.98:6789/0 conn(0x7ff24c070c80 legacy=0x7ff24c0c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.734+0000 7ff243fff640 1 -- 172.21.15.1:0/166362762 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c152fc0 con 0x7ff24c0c32f0 2024-03-06T07:22:17.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.734+0000 7ff24a59c640 1 --2- 172.21.15.1:0/166362762 >> v2:172.21.15.98:3300/0 conn(0x7ff24c150520 0x7ff24c1543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:17.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.734+0000 7ff243fff640 1 --2- 172.21.15.1:0/166362762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff24c0c32f0 0x7ff24c1537c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff234009870 tx=0x7ff23402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fad750a1b0840844 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.735+0000 7ff249d9b640 1 -- 172.21.15.1:0/166362762 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff234034070 con 0x7ff24c0c32f0 2024-03-06T07:22:17.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.735+0000 7ff249d9b640 1 -- 172.21.15.1:0/166362762 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff234030840 con 0x7ff24c0c32f0 2024-03-06T07:22:17.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.735+0000 7ff249d9b640 1 -- 172.21.15.1:0/166362762 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff234030c10 con 0x7ff24c0c32f0 2024-03-06T07:22:17.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.735+0000 7ff250b1e640 1 -- 172.21.15.1:0/166362762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff24c0c32f0 msgr2=0x7ff24c1537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.735+0000 7ff250b1e640 1 --2- 172.21.15.1:0/166362762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff24c0c32f0 0x7ff24c1537c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff234009870 tx=0x7ff23402b980 comp rx=0 tx=0).stop 2024-03-06T07:22:17.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.735+0000 7ff250b1e640 1 -- 172.21.15.1:0/166362762 shutdown_connections 2024-03-06T07:22:17.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.735+0000 7ff250b1e640 1 --2- 172.21.15.1:0/166362762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff24c0c32f0 0x7ff24c1537c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.735+0000 7ff250b1e640 1 --2- 172.21.15.1:0/166362762 >> v2:172.21.15.98:3300/0 conn(0x7ff24c150520 0x7ff24c1543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.735+0000 7ff250b1e640 1 -- 172.21.15.1:0/166362762 >> 172.21.15.1:0/166362762 conn(0x7ff24c0c88f0 msgr2=0x7ff24c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:17.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.735+0000 7ff250b1e640 1 -- 172.21.15.1:0/166362762 shutdown_connections 2024-03-06T07:22:17.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.735+0000 7ff250b1e640 1 -- 172.21.15.1:0/166362762 wait complete. 2024-03-06T07:22:17.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.736+0000 7ff250b1e640 1 Processor -- start 2024-03-06T07:22:17.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.737+0000 7ff250b1e640 1 -- start start 2024-03-06T07:22:17.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.737+0000 7ff250b1e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff24c070c80 0x7ff24c155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.737+0000 7ff250b1e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff24c0c32f0 0x7ff24c0c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.737+0000 7ff250b1e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff24c150520 0x7ff24c1e88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.737+0000 7ff250b1e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c0c7ec0 con 0x7ff24c0c32f0 2024-03-06T07:22:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.737+0000 7ff250b1e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c157330 con 0x7ff24c150520 2024-03-06T07:22:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.737+0000 7ff250b1e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c153d00 con 0x7ff24c070c80 2024-03-06T07:22:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.737+0000 7ff243fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff24c0c32f0 0x7ff24c0c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.737+0000 7ff243fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff24c0c32f0 0x7ff24c0c49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37470/0 (socket says 172.21.15.1:37470) 2024-03-06T07:22:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.737+0000 7ff243fff640 1 -- 172.21.15.1:0/1774277651 learned_addr learned my addr 172.21.15.1:0/1774277651 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.737+0000 7ff24ad9d640 1 --2- 172.21.15.1:0/1774277651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff24c150520 0x7ff24c1e88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.737+0000 7ff24a59c640 1 --2- 172.21.15.1:0/1774277651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff24c070c80 0x7ff24c155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.738+0000 7ff243fff640 1 -- 172.21.15.1:0/1774277651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff24c070c80 msgr2=0x7ff24c155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.738+0000 7ff243fff640 1 --2- 172.21.15.1:0/1774277651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff24c070c80 0x7ff24c155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.738+0000 7ff243fff640 1 -- 172.21.15.1:0/1774277651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff24c150520 msgr2=0x7ff24c1e88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.738+0000 7ff243fff640 1 --2- 172.21.15.1:0/1774277651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff24c150520 0x7ff24c1e88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.738+0000 7ff243fff640 1 -- 172.21.15.1:0/1774277651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c1460e0 con 0x7ff24c0c32f0 2024-03-06T07:22:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.738+0000 7ff24a59c640 1 --2- 172.21.15.1:0/1774277651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff24c070c80 0x7ff24c155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.738+0000 7ff243fff640 1 --2- 172.21.15.1:0/1774277651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff24c0c32f0 0x7ff24c0c49b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff23402b9c0 tx=0x7ff234004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.738+0000 7ff242ffd640 1 -- 172.21.15.1:0/1774277651 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff234048070 con 0x7ff24c0c32f0 2024-03-06T07:22:17.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.738+0000 7ff242ffd640 1 -- 172.21.15.1:0/1774277651 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff234030640 con 0x7ff24c0c32f0 2024-03-06T07:22:17.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.738+0000 7ff250b1e640 1 -- 172.21.15.1:0/1774277651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24c1eca70 con 0x7ff24c0c32f0 2024-03-06T07:22:17.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.738+0000 7ff242ffd640 1 -- 172.21.15.1:0/1774277651 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff234034040 con 0x7ff24c0c32f0 2024-03-06T07:22:17.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.738+0000 7ff250b1e640 1 -- 172.21.15.1:0/1774277651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24c1469e0 con 0x7ff24c0c32f0 2024-03-06T07:22:17.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.739+0000 7ff250b1e640 1 -- 172.21.15.1:0/1774277651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff24c0686e0 con 0x7ff24c0c32f0 2024-03-06T07:22:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.740+0000 7ff242ffd640 1 -- 172.21.15.1:0/1774277651 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff23404b070 con 0x7ff24c0c32f0 2024-03-06T07:22:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.740+0000 7ff242ffd640 1 --2- 172.21.15.1:0/1774277651 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff22c06b4d0 0x7ff22c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.741+0000 7ff242ffd640 1 -- 172.21.15.1:0/1774277651 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff2340ac380 con 0x7ff24c0c32f0 2024-03-06T07:22:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.741+0000 7ff24a59c640 1 --2- 172.21.15.1:0/1774277651 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff22c06b4d0 0x7ff22c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.741+0000 7ff24a59c640 1 --2- 172.21.15.1:0/1774277651 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff22c06b4d0 0x7ff22c06d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009560 tx=0x7ff23c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.749+0000 7ff242ffd640 1 -- 172.21.15.1:0/1774277651 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff23407ab40 con 0x7ff24c0c32f0 2024-03-06T07:22:17.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.931+0000 7ff250b1e640 1 -- 172.21.15.1:0/1774277651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff24c155cd0 con 0x7ff24c0c32f0 2024-03-06T07:22:17.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.932+0000 7ff242ffd640 1 -- 172.21.15.1:0/1774277651 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff23407ebe0 con 0x7ff24c0c32f0 2024-03-06T07:22:17.935 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:22:17.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.933+0000 7ff250b1e640 1 -- 172.21.15.1:0/1774277651 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff22c06b4d0 msgr2=0x7ff22c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.934+0000 7ff250b1e640 1 --2- 172.21.15.1:0/1774277651 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff22c06b4d0 0x7ff22c06d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009560 tx=0x7ff23c009190 comp rx=0 tx=0).stop 2024-03-06T07:22:17.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.934+0000 7ff250b1e640 1 -- 172.21.15.1:0/1774277651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff24c0c32f0 msgr2=0x7ff24c0c49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.934+0000 7ff250b1e640 1 --2- 172.21.15.1:0/1774277651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff24c0c32f0 0x7ff24c0c49b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff23402b9c0 tx=0x7ff234004730 comp rx=0 tx=0).stop 2024-03-06T07:22:17.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.934+0000 7ff250b1e640 1 -- 172.21.15.1:0/1774277651 shutdown_connections 2024-03-06T07:22:17.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.934+0000 7ff250b1e640 1 --2- 172.21.15.1:0/1774277651 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff22c06b4d0 0x7ff22c06d990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.934+0000 7ff250b1e640 1 --2- 172.21.15.1:0/1774277651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff24c150520 0x7ff24c1e88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.934+0000 7ff250b1e640 1 --2- 172.21.15.1:0/1774277651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff24c0c32f0 0x7ff24c0c49b0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.934+0000 7ff250b1e640 1 --2- 172.21.15.1:0/1774277651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff24c070c80 0x7ff24c155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.934+0000 7ff250b1e640 1 -- 172.21.15.1:0/1774277651 >> 172.21.15.1:0/1774277651 conn(0x7ff24c0c88f0 msgr2=0x7ff24c14b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:17.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.934+0000 7ff250b1e640 1 -- 172.21.15.1:0/1774277651 shutdown_connections 2024-03-06T07:22:17.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:17.934+0000 7ff250b1e640 1 -- 172.21.15.1:0/1774277651 wait complete. 2024-03-06T07:22:17.946 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:18.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.048+0000 7f7c841d6640 1 Processor -- start 2024-03-06T07:22:18.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.048+0000 7f7c841d6640 1 -- start start 2024-03-06T07:22:18.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.049+0000 7f7c841d6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7c7c152650 0x7f7c7c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.049+0000 7f7c841d6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c7c0bf9d0 0x7f7c7c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.049+0000 7f7c841d6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7c0c2fc0 con 0x7f7c7c0c6b10 2024-03-06T07:22:18.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.049+0000 7f7c841d6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c7c155800 con 0x7f7c7c0bf9d0 2024-03-06T07:22:18.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.049+0000 7f7c841d6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c7c158da0 con 0x7f7c7c152650 2024-03-06T07:22:18.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.049+0000 7f7c82773640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c7c0bf9d0 0x7f7c7c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.049+0000 7f7c82773640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c7c0bf9d0 0x7f7c7c0c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37478/0 (socket says 172.21.15.1:37478) 2024-03-06T07:22:18.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.049+0000 7f7c82773640 1 -- 172.21.15.1:0/4254090847 learned_addr learned my addr 172.21.15.1:0/4254090847 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:18.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.050+0000 7f7c82773640 1 -- 172.21.15.1:0/4254090847 >> v2:172.21.15.98:3300/0 conn(0x7f7c7c152650 msgr2=0x7f7c7c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.050+0000 7f7c82773640 1 --2- 172.21.15.1:0/4254090847 >> v2:172.21.15.98:3300/0 conn(0x7f7c7c152650 0x7f7c7c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.050+0000 7f7c82773640 1 -- 172.21.15.1:0/4254090847 >> v1:172.21.15.98:6789/0 conn(0x7f7c7c0c6b10 legacy=0x7f7c7c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.050+0000 7f7c82773640 1 -- 172.21.15.1:0/4254090847 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c0c6690 con 0x7f7c7c0bf9d0 2024-03-06T07:22:18.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.050+0000 7f7c82773640 1 --2- 172.21.15.1:0/4254090847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c7c0bf9d0 0x7f7c7c0c1460 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c0cac90 tx=0x7f7c6c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1fff6b34f655ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.051+0000 7f7c80f70640 1 -- 172.21.15.1:0/4254090847 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c02c9f0 con 0x7f7c7c0bf9d0 2024-03-06T07:22:18.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.051+0000 7f7c80f70640 1 -- 172.21.15.1:0/4254090847 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6c02cb90 con 0x7f7c7c0bf9d0 2024-03-06T07:22:18.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.051+0000 7f7c80f70640 1 -- 172.21.15.1:0/4254090847 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0343f0 con 0x7f7c7c0bf9d0 2024-03-06T07:22:18.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.051+0000 7f7c841d6640 1 -- 172.21.15.1:0/4254090847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c7c0bf9d0 msgr2=0x7f7c7c0c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.051+0000 7f7c841d6640 1 --2- 172.21.15.1:0/4254090847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c7c0bf9d0 0x7f7c7c0c1460 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c0cac90 tx=0x7f7c6c02be50 comp rx=0 tx=0).stop 2024-03-06T07:22:18.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.051+0000 7f7c841d6640 1 -- 172.21.15.1:0/4254090847 shutdown_connections 2024-03-06T07:22:18.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.051+0000 7f7c841d6640 1 --2- 172.21.15.1:0/4254090847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c7c0bf9d0 0x7f7c7c0c1460 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.051+0000 7f7c841d6640 1 --2- 172.21.15.1:0/4254090847 >> v2:172.21.15.98:3300/0 conn(0x7f7c7c152650 0x7f7c7c155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.051+0000 7f7c841d6640 1 -- 172.21.15.1:0/4254090847 >> 172.21.15.1:0/4254090847 conn(0x7f7c7c066ee0 msgr2=0x7f7c7c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.051+0000 7f7c841d6640 1 -- 172.21.15.1:0/4254090847 shutdown_connections 2024-03-06T07:22:18.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.051+0000 7f7c841d6640 1 -- 172.21.15.1:0/4254090847 wait complete. 2024-03-06T07:22:18.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.052+0000 7f7c841d6640 1 Processor -- start 2024-03-06T07:22:18.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.052+0000 7f7c841d6640 1 -- start start 2024-03-06T07:22:18.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.053+0000 7f7c841d6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c7c0bf9d0 0x7f7c7c153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.053+0000 7f7c841d6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c7c152650 0x7f7c7c14b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.053+0000 7f7c841d6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c7c14e580 0x7f7c7c1ee680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.053+0000 7f7c841d6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c1551b0 con 0x7f7c7c14e580 2024-03-06T07:22:18.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.053+0000 7f7c841d6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c0c6fa0 con 0x7f7c7c0bf9d0 2024-03-06T07:22:18.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.053+0000 7f7c841d6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c155490 con 0x7f7c7c152650 2024-03-06T07:22:18.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.053+0000 7f7c82773640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c7c14e580 0x7f7c7c1ee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.053+0000 7f7c82773640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c7c14e580 0x7f7c7c1ee680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37488/0 (socket says 172.21.15.1:37488) 2024-03-06T07:22:18.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.053+0000 7f7c82773640 1 -- 172.21.15.1:0/3924657020 learned_addr learned my addr 172.21.15.1:0/3924657020 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:18.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.053+0000 7f7c81f72640 1 --2- 172.21.15.1:0/3924657020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c7c0bf9d0 0x7f7c7c153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.053+0000 7f7c81771640 1 --2- 172.21.15.1:0/3924657020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c7c152650 0x7f7c7c14b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.053+0000 7f7c82773640 1 -- 172.21.15.1:0/3924657020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c7c152650 msgr2=0x7f7c7c14b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.053+0000 7f7c82773640 1 --2- 172.21.15.1:0/3924657020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c7c152650 0x7f7c7c14b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.053+0000 7f7c82773640 1 -- 172.21.15.1:0/3924657020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c7c0bf9d0 msgr2=0x7f7c7c153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.053+0000 7f7c82773640 1 --2- 172.21.15.1:0/3924657020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c7c0bf9d0 0x7f7c7c153bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.053+0000 7f7c82773640 1 -- 172.21.15.1:0/3924657020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c14d5d0 con 0x7f7c7c14e580 2024-03-06T07:22:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.053+0000 7f7c81f72640 1 --2- 172.21.15.1:0/3924657020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c7c0bf9d0 0x7f7c7c153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.054+0000 7f7c82773640 1 --2- 172.21.15.1:0/3924657020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c7c14e580 0x7f7c7c1ee680 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c036040 tx=0x7f7c6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.054+0000 7f7c7a7fc640 1 -- 172.21.15.1:0/3924657020 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c043020 con 0x7f7c7c14e580 2024-03-06T07:22:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.054+0000 7f7c7a7fc640 1 -- 172.21.15.1:0/3924657020 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6c03e450 con 0x7f7c7c14e580 2024-03-06T07:22:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.054+0000 7f7c841d6640 1 -- 172.21.15.1:0/3924657020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7c14d8b0 con 0x7f7c7c14e580 2024-03-06T07:22:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.054+0000 7f7c7a7fc640 1 -- 172.21.15.1:0/3924657020 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c009510 con 0x7f7c7c14e580 2024-03-06T07:22:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.054+0000 7f7c841d6640 1 -- 172.21.15.1:0/3924657020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7c14ea00 con 0x7f7c7c14e580 2024-03-06T07:22:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.055+0000 7f7c841d6640 1 -- 172.21.15.1:0/3924657020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7c067fd0 con 0x7f7c7c14e580 2024-03-06T07:22:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.055+0000 7f7c7a7fc640 1 -- 172.21.15.1:0/3924657020 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c6c046020 con 0x7f7c7c14e580 2024-03-06T07:22:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.055+0000 7f7c7a7fc640 1 --2- 172.21.15.1:0/3924657020 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c6806b4d0 0x7f7c6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.056+0000 7f7c7a7fc640 1 -- 172.21.15.1:0/3924657020 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7c6c0acca0 con 0x7f7c7c14e580 2024-03-06T07:22:18.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.056+0000 7f7c81f72640 1 --2- 172.21.15.1:0/3924657020 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c6806b4d0 0x7f7c6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.056+0000 7f7c81f72640 1 --2- 172.21.15.1:0/3924657020 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c6806b4d0 0x7f7c6806d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7c74006fd0 tx=0x7f7c74008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.064+0000 7f7c7a7fc640 1 -- 172.21.15.1:0/3924657020 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c6c04d030 con 0x7f7c7c14e580 2024-03-06T07:22:18.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.244+0000 7f7c841d6640 1 -- 172.21.15.1:0/3924657020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7c7c156040 con 0x7f7c7c14e580 2024-03-06T07:22:18.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.244+0000 7f7c7a7fc640 1 -- 172.21.15.1:0/3924657020 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7c6c07ba20 con 0x7f7c7c14e580 2024-03-06T07:22:18.246 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:22:18.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.246+0000 7f7c841d6640 1 -- 172.21.15.1:0/3924657020 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c6806b4d0 msgr2=0x7f7c6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.246+0000 7f7c841d6640 1 --2- 172.21.15.1:0/3924657020 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c6806b4d0 0x7f7c6806d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7c74006fd0 tx=0x7f7c74008040 comp rx=0 tx=0).stop 2024-03-06T07:22:18.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.246+0000 7f7c841d6640 1 -- 172.21.15.1:0/3924657020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c7c14e580 msgr2=0x7f7c7c1ee680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.246+0000 7f7c841d6640 1 --2- 172.21.15.1:0/3924657020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c7c14e580 0x7f7c7c1ee680 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c036040 tx=0x7f7c6c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:18.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.246+0000 7f7c841d6640 1 -- 172.21.15.1:0/3924657020 shutdown_connections 2024-03-06T07:22:18.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.246+0000 7f7c841d6640 1 --2- 172.21.15.1:0/3924657020 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c6806b4d0 0x7f7c6806d990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.246+0000 7f7c841d6640 1 --2- 172.21.15.1:0/3924657020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c7c14e580 0x7f7c7c1ee680 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.246+0000 7f7c841d6640 1 --2- 172.21.15.1:0/3924657020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c7c152650 0x7f7c7c14b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.246+0000 7f7c841d6640 1 --2- 172.21.15.1:0/3924657020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c7c0bf9d0 0x7f7c7c153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.246+0000 7f7c841d6640 1 -- 172.21.15.1:0/3924657020 >> 172.21.15.1:0/3924657020 conn(0x7f7c7c066ee0 msgr2=0x7f7c7c0bfdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.246+0000 7f7c841d6640 1 -- 172.21.15.1:0/3924657020 shutdown_connections 2024-03-06T07:22:18.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.246+0000 7f7c841d6640 1 -- 172.21.15.1:0/3924657020 wait complete. 2024-03-06T07:22:18.259 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:22:18.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.358+0000 7fb720a5c640 1 Processor -- start 2024-03-06T07:22:18.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.358+0000 7fb720a5c640 1 -- start start 2024-03-06T07:22:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.359+0000 7fb720a5c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb71c150520 0x7fb71c1543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.359+0000 7fb720a5c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c0c32f0 0x7fb71c1537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.359+0000 7fb720a5c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb71c070c80 0x7fb71c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.359+0000 7fb720a5c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb71c150900 con 0x7fb71c070c80 2024-03-06T07:22:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.359+0000 7fb720a5c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb71c071790 con 0x7fb71c150520 2024-03-06T07:22:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.359+0000 7fb720a5c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c157050 con 0x7fb71c0c32f0 2024-03-06T07:22:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.359+0000 7fb71ad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb71c070c80 0x7fb71c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.359+0000 7fb71ad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb71c070c80 0x7fb71c0710f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37494/0 (socket says 172.21.15.1:37494) 2024-03-06T07:22:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.359+0000 7fb71ad9d640 1 -- 172.21.15.1:0/3224465092 learned_addr learned my addr 172.21.15.1:0/3224465092 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.360+0000 7fb71ad9d640 1 -- 172.21.15.1:0/3224465092 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c0c32f0 msgr2=0x7fb71c1537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.360+0000 7fb719d9b640 1 --2- 172.21.15.1:0/3224465092 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c0c32f0 0x7fb71c1537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.360+0000 7fb71a59c640 1 --2- 172.21.15.1:0/3224465092 >> v2:172.21.15.98:3300/0 conn(0x7fb71c150520 0x7fb71c1543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.360+0000 7fb71ad9d640 1 --2- 172.21.15.1:0/3224465092 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c0c32f0 0x7fb71c1537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.360+0000 7fb71ad9d640 1 -- 172.21.15.1:0/3224465092 >> v2:172.21.15.98:3300/0 conn(0x7fb71c150520 msgr2=0x7fb71c1543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.360+0000 7fb71ad9d640 1 --2- 172.21.15.1:0/3224465092 >> v2:172.21.15.98:3300/0 conn(0x7fb71c150520 0x7fb71c1543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.360+0000 7fb71ad9d640 1 -- 172.21.15.1:0/3224465092 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71c152fc0 con 0x7fb71c070c80 2024-03-06T07:22:18.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.360+0000 7fb719d9b640 1 --2- 172.21.15.1:0/3224465092 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c0c32f0 0x7fb71c1537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:18.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.360+0000 7fb71a59c640 1 --2- 172.21.15.1:0/3224465092 >> v2:172.21.15.98:3300/0 conn(0x7fb71c150520 0x7fb71c1543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:18.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.360+0000 7fb71ad9d640 1 --2- 172.21.15.1:0/3224465092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c070c80 0x7fb71c0710f0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb71c06e6c0 tx=0x7fb70c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c72627dc1770765c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.361+0000 7fb71959a640 1 -- 172.21.15.1:0/3224465092 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c02ca70 con 0x7fb71c070c80 2024-03-06T07:22:18.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.361+0000 7fb71959a640 1 -- 172.21.15.1:0/3224465092 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70c02cc10 con 0x7fb71c070c80 2024-03-06T07:22:18.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.361+0000 7fb71959a640 1 -- 172.21.15.1:0/3224465092 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c0343f0 con 0x7fb71c070c80 2024-03-06T07:22:18.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.361+0000 7fb720a5c640 1 -- 172.21.15.1:0/3224465092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c070c80 msgr2=0x7fb71c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.361+0000 7fb720a5c640 1 --2- 172.21.15.1:0/3224465092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c070c80 0x7fb71c0710f0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb71c06e6c0 tx=0x7fb70c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.361+0000 7fb720a5c640 1 -- 172.21.15.1:0/3224465092 shutdown_connections 2024-03-06T07:22:18.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.361+0000 7fb720a5c640 1 --2- 172.21.15.1:0/3224465092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c070c80 0x7fb71c0710f0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.361+0000 7fb720a5c640 1 --2- 172.21.15.1:0/3224465092 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c0c32f0 0x7fb71c1537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.361+0000 7fb720a5c640 1 --2- 172.21.15.1:0/3224465092 >> v2:172.21.15.98:3300/0 conn(0x7fb71c150520 0x7fb71c1543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.361+0000 7fb720a5c640 1 -- 172.21.15.1:0/3224465092 >> 172.21.15.1:0/3224465092 conn(0x7fb71c0c88f0 msgr2=0x7fb71c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.361+0000 7fb720a5c640 1 -- 172.21.15.1:0/3224465092 shutdown_connections 2024-03-06T07:22:18.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.361+0000 7fb720a5c640 1 -- 172.21.15.1:0/3224465092 wait complete. 2024-03-06T07:22:18.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.362+0000 7fb720a5c640 1 Processor -- start 2024-03-06T07:22:18.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.363+0000 7fb720a5c640 1 -- start start 2024-03-06T07:22:18.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.363+0000 7fb720a5c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c070c80 0x7fb71c1eaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.363+0000 7fb720a5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb71c0c32f0 0x7fb71c151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.363+0000 7fb720a5c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c150520 0x7fb71c0c1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.363+0000 7fb720a5c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c0c7ec0 con 0x7fb71c150520 2024-03-06T07:22:18.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.363+0000 7fb720a5c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c157330 con 0x7fb71c0c32f0 2024-03-06T07:22:18.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.363+0000 7fb720a5c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c153d00 con 0x7fb71c070c80 2024-03-06T07:22:18.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.363+0000 7fb71ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c150520 0x7fb71c0c1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.363+0000 7fb71ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c150520 0x7fb71c0c1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37508/0 (socket says 172.21.15.1:37508) 2024-03-06T07:22:18.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.363+0000 7fb71ad9d640 1 -- 172.21.15.1:0/887378097 learned_addr learned my addr 172.21.15.1:0/887378097 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:18.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.363+0000 7fb719d9b640 1 --2- 172.21.15.1:0/887378097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb71c0c32f0 0x7fb71c151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.363+0000 7fb71a59c640 1 --2- 172.21.15.1:0/887378097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c070c80 0x7fb71c1eaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.364+0000 7fb71ad9d640 1 -- 172.21.15.1:0/887378097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c070c80 msgr2=0x7fb71c1eaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.364+0000 7fb71ad9d640 1 --2- 172.21.15.1:0/887378097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c070c80 0x7fb71c1eaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.364+0000 7fb71ad9d640 1 -- 172.21.15.1:0/887378097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb71c0c32f0 msgr2=0x7fb71c151020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.364+0000 7fb71ad9d640 1 --2- 172.21.15.1:0/887378097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb71c0c32f0 0x7fb71c151020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.364+0000 7fb71ad9d640 1 -- 172.21.15.1:0/887378097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71c06c480 con 0x7fb71c150520 2024-03-06T07:22:18.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.364+0000 7fb719d9b640 1 --2- 172.21.15.1:0/887378097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb71c0c32f0 0x7fb71c151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:18.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.364+0000 7fb71ad9d640 1 --2- 172.21.15.1:0/887378097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c150520 0x7fb71c0c1d20 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb70c02c4e0 tx=0x7fb70c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.364+0000 7fb712ffd640 1 -- 172.21.15.1:0/887378097 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c043020 con 0x7fb71c150520 2024-03-06T07:22:18.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.364+0000 7fb712ffd640 1 -- 172.21.15.1:0/887378097 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70c03e450 con 0x7fb71c150520 2024-03-06T07:22:18.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.364+0000 7fb720a5c640 1 -- 172.21.15.1:0/887378097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb71c06c1a0 con 0x7fb71c150520 2024-03-06T07:22:18.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.364+0000 7fb712ffd640 1 -- 172.21.15.1:0/887378097 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c03e840 con 0x7fb71c150520 2024-03-06T07:22:18.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.364+0000 7fb720a5c640 1 -- 172.21.15.1:0/887378097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb71c147010 con 0x7fb71c150520 2024-03-06T07:22:18.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.365+0000 7fb720a5c640 1 -- 172.21.15.1:0/887378097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb71c06e900 con 0x7fb71c150520 2024-03-06T07:22:18.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.365+0000 7fb712ffd640 1 -- 172.21.15.1:0/887378097 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb70c046020 con 0x7fb71c150520 2024-03-06T07:22:18.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.366+0000 7fb712ffd640 1 --2- 172.21.15.1:0/887378097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb6fc06b500 0x7fb6fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.366+0000 7fb712ffd640 1 -- 172.21.15.1:0/887378097 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb70c0acdc0 con 0x7fb71c150520 2024-03-06T07:22:18.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.366+0000 7fb71a59c640 1 --2- 172.21.15.1:0/887378097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb6fc06b500 0x7fb6fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.367+0000 7fb71a59c640 1 --2- 172.21.15.1:0/887378097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb6fc06b500 0x7fb6fc06d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb71400a950 tx=0x7fb714008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.370+0000 7fb712ffd640 1 -- 172.21.15.1:0/887378097 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb71c06e900 con 0x7fb71c150520 2024-03-06T07:22:18.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.572+0000 7fb720a5c640 1 -- 172.21.15.1:0/887378097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb71c14a8c0 con 0x7fb71c150520 2024-03-06T07:22:18.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.572+0000 7fb712ffd640 1 -- 172.21.15.1:0/887378097 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v52) v1 ==== 84+0+207 (secure 0 0 0) 0x7fb70c04d030 con 0x7fb71c150520 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 7, 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": "cephfs_metadata" 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 8, 2024-03-06T07:22:18.577 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": "cephfs_data" 2024-03-06T07:22:18.577 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:18.577 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-06T07:22:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.574+0000 7fb720a5c640 1 -- 172.21.15.1:0/887378097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb6fc06b500 msgr2=0x7fb6fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.574+0000 7fb720a5c640 1 --2- 172.21.15.1:0/887378097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb6fc06b500 0x7fb6fc06d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb71400a950 tx=0x7fb714008040 comp rx=0 tx=0).stop 2024-03-06T07:22:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.574+0000 7fb720a5c640 1 -- 172.21.15.1:0/887378097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c150520 msgr2=0x7fb71c0c1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.574+0000 7fb720a5c640 1 --2- 172.21.15.1:0/887378097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c150520 0x7fb71c0c1d20 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb70c02c4e0 tx=0x7fb70c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.574+0000 7fb720a5c640 1 -- 172.21.15.1:0/887378097 shutdown_connections 2024-03-06T07:22:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.574+0000 7fb720a5c640 1 --2- 172.21.15.1:0/887378097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb6fc06b500 0x7fb6fc06d9c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.574+0000 7fb720a5c640 1 --2- 172.21.15.1:0/887378097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c150520 0x7fb71c0c1d20 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.574+0000 7fb720a5c640 1 --2- 172.21.15.1:0/887378097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb71c0c32f0 0x7fb71c151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.574+0000 7fb720a5c640 1 --2- 172.21.15.1:0/887378097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c070c80 0x7fb71c1eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.574+0000 7fb720a5c640 1 -- 172.21.15.1:0/887378097 >> 172.21.15.1:0/887378097 conn(0x7fb71c0c88f0 msgr2=0x7fb71c071060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.574+0000 7fb720a5c640 1 -- 172.21.15.1:0/887378097 shutdown_connections 2024-03-06T07:22:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.574+0000 7fb720a5c640 1 -- 172.21.15.1:0/887378097 wait complete. 2024-03-06T07:22:18.587 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs2' 2024-03-06T07:22:18.588 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_metadata --pg_num_min 64 2024-03-06T07:22:18.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.692+0000 7f9639284640 1 Processor -- start 2024-03-06T07:22:18.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.692+0000 7f9639284640 1 -- start start 2024-03-06T07:22:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.693+0000 7f9639284640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f96341527d0 0x7f96340c6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.693+0000 7f9639284640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96340704b0 0x7f96340c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.693+0000 7f9639284640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9634155cd0 0x7f96340c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.693+0000 7f9639284640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9634156230 con 0x7f96341527d0 2024-03-06T07:22:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.693+0000 7f9639284640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f963406f730 con 0x7f9634155cd0 2024-03-06T07:22:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.693+0000 7f9639284640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963406f9d0 con 0x7f96340704b0 2024-03-06T07:22:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.693+0000 7f9632ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f96341527d0 0x7f96340c6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.693+0000 7f9632ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f96341527d0 0x7f96340c6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37514/0 (socket says 172.21.15.1:37514) 2024-03-06T07:22:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.693+0000 7f9632ffd640 1 -- 172.21.15.1:0/3642602217 learned_addr learned my addr 172.21.15.1:0/3642602217 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.694+0000 7f96337fe640 1 --2- 172.21.15.1:0/3642602217 >> v2:172.21.15.98:3300/0 conn(0x7f9634155cd0 0x7f96340c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.694+0000 7f962bfff640 1 --2- 172.21.15.1:0/3642602217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96340704b0 0x7f96340c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.694+0000 7f9632ffd640 1 -- 172.21.15.1:0/3642602217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96340704b0 msgr2=0x7f96340c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.694+0000 7f9632ffd640 1 --2- 172.21.15.1:0/3642602217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96340704b0 0x7f96340c58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.694+0000 7f9632ffd640 1 -- 172.21.15.1:0/3642602217 >> v2:172.21.15.98:3300/0 conn(0x7f9634155cd0 msgr2=0x7f96340c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.694+0000 7f9632ffd640 1 --2- 172.21.15.1:0/3642602217 >> v2:172.21.15.98:3300/0 conn(0x7f9634155cd0 0x7f96340c5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.694+0000 7f9632ffd640 1 -- 172.21.15.1:0/3642602217 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963406fcb0 con 0x7f96341527d0 2024-03-06T07:22:18.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.694+0000 7f9632ffd640 1 --2- 172.21.15.1:0/3642602217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96341527d0 0x7f96340c6fa0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f963406e270 tx=0x7f962402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32e142affa72e20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.695+0000 7f96327fc640 1 -- 172.21.15.1:0/3642602217 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962402e070 con 0x7f96341527d0 2024-03-06T07:22:18.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.695+0000 7f96327fc640 1 -- 172.21.15.1:0/3642602217 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9624037070 con 0x7f96341527d0 2024-03-06T07:22:18.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.695+0000 7f96327fc640 1 -- 172.21.15.1:0/3642602217 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9624031a00 con 0x7f96341527d0 2024-03-06T07:22:18.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.695+0000 7f9639284640 1 -- 172.21.15.1:0/3642602217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96341527d0 msgr2=0x7f96340c6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.695+0000 7f9639284640 1 --2- 172.21.15.1:0/3642602217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96341527d0 0x7f96340c6fa0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f963406e270 tx=0x7f962402b880 comp rx=0 tx=0).stop 2024-03-06T07:22:18.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.695+0000 7f9639284640 1 -- 172.21.15.1:0/3642602217 shutdown_connections 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.695+0000 7f9639284640 1 --2- 172.21.15.1:0/3642602217 >> v2:172.21.15.98:3300/0 conn(0x7f9634155cd0 0x7f96340c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.695+0000 7f9639284640 1 --2- 172.21.15.1:0/3642602217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96340704b0 0x7f96340c58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.695+0000 7f9639284640 1 --2- 172.21.15.1:0/3642602217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96341527d0 0x7f96340c6fa0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.695+0000 7f9639284640 1 -- 172.21.15.1:0/3642602217 >> 172.21.15.1:0/3642602217 conn(0x7f96340c44c0 msgr2=0x7f96340c48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.695+0000 7f9639284640 1 -- 172.21.15.1:0/3642602217 shutdown_connections 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.695+0000 7f9639284640 1 -- 172.21.15.1:0/3642602217 wait complete. 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.696+0000 7f9639284640 1 Processor -- start 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.696+0000 7f9639284640 1 -- start start 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.697+0000 7f9639284640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96340704b0 0x7f96340ab440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.697+0000 7f9639284640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96341527d0 0x7f9634150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.697+0000 7f9639284640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9634155cd0 0x7f96340acef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.697+0000 7f9639284640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96340c1570 con 0x7f9634155cd0 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.697+0000 7f9639284640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9634152bb0 con 0x7f96340704b0 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.697+0000 7f9639284640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96340c16f0 con 0x7f96341527d0 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.697+0000 7f96337fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9634155cd0 0x7f96340acef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.697+0000 7f96337fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9634155cd0 0x7f96340acef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37530/0 (socket says 172.21.15.1:37530) 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.697+0000 7f96337fe640 1 -- 172.21.15.1:0/1535326663 learned_addr learned my addr 172.21.15.1:0/1535326663 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.697+0000 7f9632ffd640 1 --2- 172.21.15.1:0/1535326663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96340704b0 0x7f96340ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.697+0000 7f962bfff640 1 --2- 172.21.15.1:0/1535326663 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96341527d0 0x7f9634150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.697+0000 7f96337fe640 1 -- 172.21.15.1:0/1535326663 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96341527d0 msgr2=0x7f9634150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.697+0000 7f96337fe640 1 --2- 172.21.15.1:0/1535326663 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96341527d0 0x7f9634150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.697+0000 7f96337fe640 1 -- 172.21.15.1:0/1535326663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96340704b0 msgr2=0x7f96340ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.697+0000 7f96337fe640 1 --2- 172.21.15.1:0/1535326663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96340704b0 0x7f96340ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.697+0000 7f96337fe640 1 -- 172.21.15.1:0/1535326663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96341597f0 con 0x7f9634155cd0 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.698+0000 7f96337fe640 1 --2- 172.21.15.1:0/1535326663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9634155cd0 0x7f96340acef0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f962c007e60 tx=0x7f962c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.698+0000 7f962b7fe640 1 -- 172.21.15.1:0/1535326663 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c014620 con 0x7f9634155cd0 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.698+0000 7f962b7fe640 1 -- 172.21.15.1:0/1535326663 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962c019070 con 0x7f9634155cd0 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.698+0000 7f9639284640 1 -- 172.21.15.1:0/1535326663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9634147940 con 0x7f9634155cd0 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.698+0000 7f962b7fe640 1 -- 172.21.15.1:0/1535326663 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c014a00 con 0x7f9634155cd0 2024-03-06T07:22:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.698+0000 7f9639284640 1 -- 172.21.15.1:0/1535326663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f963414a5f0 con 0x7f9634155cd0 2024-03-06T07:22:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.699+0000 7f9639284640 1 -- 172.21.15.1:0/1535326663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96340686e0 con 0x7f9634155cd0 2024-03-06T07:22:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.699+0000 7f962b7fe640 1 -- 172.21.15.1:0/1535326663 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f962c014ba0 con 0x7f9634155cd0 2024-03-06T07:22:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.700+0000 7f962b7fe640 1 --2- 172.21.15.1:0/1535326663 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f961406b500 0x7f961406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.700+0000 7f962b7fe640 1 -- 172.21.15.1:0/1535326663 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f962c08abf0 con 0x7f9634155cd0 2024-03-06T07:22:18.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.700+0000 7f9632ffd640 1 --2- 172.21.15.1:0/1535326663 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f961406b500 0x7f961406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.700+0000 7f9632ffd640 1 --2- 172.21.15.1:0/1535326663 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f961406b500 0x7f961406d9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9624036000 tx=0x7f962403d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.704+0000 7f962b7fe640 1 -- 172.21.15.1:0/1535326663 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f962c011260 con 0x7f9634155cd0 2024-03-06T07:22:18.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:18.887+0000 7f9639284640 1 -- 172.21.15.1:0/1535326663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f96340c1fe0 con 0x7f9634155cd0 2024-03-06T07:22:19.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.682+0000 7f962b7fe640 1 -- 172.21.15.1:0/1535326663 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64}]=0 pool 'cephfs2_metadata' created v53) v1 ==== 140+0+0 (secure 0 0 0) 0x7f962c0598e0 con 0x7f9634155cd0 2024-03-06T07:22:19.684 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs2_metadata' created 2024-03-06T07:22:19.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.685+0000 7f9639284640 1 -- 172.21.15.1:0/1535326663 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f961406b500 msgr2=0x7f961406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.685+0000 7f9639284640 1 --2- 172.21.15.1:0/1535326663 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f961406b500 0x7f961406d9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9624036000 tx=0x7f962403d040 comp rx=0 tx=0).stop 2024-03-06T07:22:19.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.685+0000 7f9639284640 1 -- 172.21.15.1:0/1535326663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9634155cd0 msgr2=0x7f96340acef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.685+0000 7f9639284640 1 --2- 172.21.15.1:0/1535326663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9634155cd0 0x7f96340acef0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f962c007e60 tx=0x7f962c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.685+0000 7f9639284640 1 -- 172.21.15.1:0/1535326663 shutdown_connections 2024-03-06T07:22:19.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.685+0000 7f9639284640 1 --2- 172.21.15.1:0/1535326663 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f961406b500 0x7f961406d9c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.685+0000 7f9639284640 1 --2- 172.21.15.1:0/1535326663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9634155cd0 0x7f96340acef0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.685+0000 7f9639284640 1 --2- 172.21.15.1:0/1535326663 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f96341527d0 0x7f9634150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.685+0000 7f9639284640 1 --2- 172.21.15.1:0/1535326663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96340704b0 0x7f96340ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.685+0000 7f9639284640 1 -- 172.21.15.1:0/1535326663 >> 172.21.15.1:0/1535326663 conn(0x7f96340c44c0 msgr2=0x7f96340a76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.686+0000 7f9639284640 1 -- 172.21.15.1:0/1535326663 shutdown_connections 2024-03-06T07:22:19.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.686+0000 7f9639284640 1 -- 172.21.15.1:0/1535326663 wait complete. 2024-03-06T07:22:19.706 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:22:19.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.812+0000 7fea1656c640 1 Processor -- start 2024-03-06T07:22:19.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.812+0000 7fea1656c640 1 -- start start 2024-03-06T07:22:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.813+0000 7fea1656c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fea100c32f0 0x7fea101543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.813+0000 7fea1656c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea10070c80 0x7fea100710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.813+0000 7fea1656c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea10150900 con 0x7fea10150520 2024-03-06T07:22:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.813+0000 7fea1656c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea10153ed0 con 0x7fea100c32f0 2024-03-06T07:22:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.813+0000 7fea1656c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea10071790 con 0x7fea10070c80 2024-03-06T07:22:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.813+0000 7fea0ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fea10150520 0x7fea101537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51678/0 (socket says 172.21.15.1:51678) 2024-03-06T07:22:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.813+0000 7fea0ffff640 1 -- 172.21.15.1:0/684507960 learned_addr learned my addr 172.21.15.1:0/684507960 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.813+0000 7fea0f7fe640 1 --2- 172.21.15.1:0/684507960 >> v2:172.21.15.98:3300/0 conn(0x7fea100c32f0 0x7fea101543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.813+0000 7fea0effd640 1 -- 172.21.15.1:0/684507960 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947941994 0 0) 0x7fea10150900 con 0x7fea10150520 2024-03-06T07:22:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.813+0000 7fea14b09640 1 --2- 172.21.15.1:0/684507960 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea10070c80 0x7fea100710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.813+0000 7fea0effd640 1 -- 172.21.15.1:0/684507960 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9f8003520 con 0x7fea10150520 2024-03-06T07:22:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.813+0000 7fea0effd640 1 -- 172.21.15.1:0/684507960 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2076340664 0 0) 0x7fe9f8003520 con 0x7fea10150520 2024-03-06T07:22:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.813+0000 7fea0effd640 1 -- 172.21.15.1:0/684507960 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea10070c80 msgr2=0x7fea100710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.813+0000 7fea0effd640 1 --2- 172.21.15.1:0/684507960 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea10070c80 0x7fea100710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.813+0000 7fea0effd640 1 -- 172.21.15.1:0/684507960 >> v2:172.21.15.98:3300/0 conn(0x7fea100c32f0 msgr2=0x7fea101543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea0effd640 1 --2- 172.21.15.1:0/684507960 >> v2:172.21.15.98:3300/0 conn(0x7fea100c32f0 0x7fea101543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea0effd640 1 -- 172.21.15.1:0/684507960 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea10157050 con 0x7fea10150520 2024-03-06T07:22:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea0effd640 1 -- 172.21.15.1:0/684507960 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fea00001670 con 0x7fea10150520 2024-03-06T07:22:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea0effd640 1 -- 172.21.15.1:0/684507960 >> v1:172.21.15.1:6789/0 conn(0x7fea10150520 legacy=0x7fea101537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea0effd640 1 --2- 172.21.15.1:0/684507960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9f80039b0 0x7fe9f8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea0effd640 1 --2- 172.21.15.1:0/684507960 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea10070c80 0x7fe9f8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea0effd640 1 --2- 172.21.15.1:0/684507960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9f8004d40 0x7fe9f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea0effd640 1 -- 172.21.15.1:0/684507960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea10153ed0 con 0x7fe9f8004d40 2024-03-06T07:22:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea0effd640 1 -- 172.21.15.1:0/684507960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea10071790 con 0x7fe9f80039b0 2024-03-06T07:22:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea0effd640 1 -- 172.21.15.1:0/684507960 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f800b9c0 con 0x7fea10070c80 2024-03-06T07:22:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea14b09640 1 --2- 172.21.15.1:0/684507960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9f8004d40 0x7fe9f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea0ffff640 1 --2- 172.21.15.1:0/684507960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9f80039b0 0x7fe9f8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea0f7fe640 1 --2- 172.21.15.1:0/684507960 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea10070c80 0x7fe9f8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea14b09640 1 -- 172.21.15.1:0/684507960 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea10070c80 msgr2=0x7fe9f8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea14b09640 1 --2- 172.21.15.1:0/684507960 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea10070c80 0x7fe9f8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea14b09640 1 -- 172.21.15.1:0/684507960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9f80039b0 msgr2=0x7fe9f8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea14b09640 1 --2- 172.21.15.1:0/684507960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9f80039b0 0x7fe9f8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.814+0000 7fea14b09640 1 -- 172.21.15.1:0/684507960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f8003520 con 0x7fe9f8004d40 2024-03-06T07:22:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.815+0000 7fea14b09640 1 --2- 172.21.15.1:0/684507960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9f8004d40 0x7fe9f80051c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fea0800e170 tx=0x7fe9f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6eb345bdbcb0fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.815+0000 7fea0effd640 1 -- 172.21.15.1:0/684507960 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08039070 con 0x7fe9f8004d40 2024-03-06T07:22:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.815+0000 7fea0effd640 1 -- 172.21.15.1:0/684507960 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0802cbf0 con 0x7fe9f8004d40 2024-03-06T07:22:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.815+0000 7fea0effd640 1 -- 172.21.15.1:0/684507960 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08009c10 con 0x7fe9f8004d40 2024-03-06T07:22:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.815+0000 7fea1656c640 1 -- 172.21.15.1:0/684507960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9f8004d40 msgr2=0x7fe9f80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.815+0000 7fea1656c640 1 --2- 172.21.15.1:0/684507960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9f8004d40 0x7fe9f80051c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fea0800e170 tx=0x7fe9f8005830 comp rx=0 tx=0).stop 2024-03-06T07:22:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.815+0000 7fea14b09640 1 -- 172.21.15.1:0/684507960 reap_dead start 2024-03-06T07:22:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.815+0000 7fea1656c640 1 -- 172.21.15.1:0/684507960 shutdown_connections 2024-03-06T07:22:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.815+0000 7fea1656c640 1 -- 172.21.15.1:0/684507960 >> 172.21.15.1:0/684507960 conn(0x7fea100c88f0 msgr2=0x7fea100c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.816+0000 7fea1656c640 1 -- 172.21.15.1:0/684507960 shutdown_connections 2024-03-06T07:22:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.816+0000 7fea1656c640 1 -- 172.21.15.1:0/684507960 wait complete. 2024-03-06T07:22:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.817+0000 7fea1656c640 1 Processor -- start 2024-03-06T07:22:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.817+0000 7fea1656c640 1 -- start start 2024-03-06T07:22:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.817+0000 7fea1656c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea10145a20 0x7fea101562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.817+0000 7fea1656c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea10156820 0x7fea10151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.817+0000 7fea1656c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea10151a70 0x7fea100ac740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.817+0000 7fea1656c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea10157330 con 0x7fea10156820 2024-03-06T07:22:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.817+0000 7fea1656c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea101532a0 con 0x7fea10145a20 2024-03-06T07:22:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.817+0000 7fea1656c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea100c7ec0 con 0x7fea10151a70 2024-03-06T07:22:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.818+0000 7fea0f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea10156820 0x7fea10151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.818+0000 7fea0f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea10156820 0x7fea10151530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37550/0 (socket says 172.21.15.1:37550) 2024-03-06T07:22:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.818+0000 7fea0f7fe640 1 -- 172.21.15.1:0/3947065780 learned_addr learned my addr 172.21.15.1:0/3947065780 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.818+0000 7fea0ffff640 1 --2- 172.21.15.1:0/3947065780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea10145a20 0x7fea101562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.818+0000 7fea14b09640 1 --2- 172.21.15.1:0/3947065780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea10151a70 0x7fea100ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.818+0000 7fea0f7fe640 1 -- 172.21.15.1:0/3947065780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea10151a70 msgr2=0x7fea100ac740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.818+0000 7fea0f7fe640 1 --2- 172.21.15.1:0/3947065780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea10151a70 0x7fea100ac740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.818+0000 7fea0f7fe640 1 -- 172.21.15.1:0/3947065780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea10145a20 msgr2=0x7fea101562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.818+0000 7fea0f7fe640 1 --2- 172.21.15.1:0/3947065780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea10145a20 0x7fea101562e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.818+0000 7fea0f7fe640 1 -- 172.21.15.1:0/3947065780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea100a8620 con 0x7fea10156820 2024-03-06T07:22:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.818+0000 7fea0f7fe640 1 --2- 172.21.15.1:0/3947065780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea10156820 0x7fea10151530 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fea04011a70 tx=0x7fea04011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.818+0000 7fea0cff9640 1 -- 172.21.15.1:0/3947065780 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0400fa90 con 0x7fea10156820 2024-03-06T07:22:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.818+0000 7fea0cff9640 1 -- 172.21.15.1:0/3947065780 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0400fc30 con 0x7fea10156820 2024-03-06T07:22:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.818+0000 7fea0cff9640 1 -- 172.21.15.1:0/3947065780 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea04014440 con 0x7fea10156820 2024-03-06T07:22:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.818+0000 7fea1656c640 1 -- 172.21.15.1:0/3947065780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea101ec700 con 0x7fea10156820 2024-03-06T07:22:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.818+0000 7fea1656c640 1 -- 172.21.15.1:0/3947065780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea101559f0 con 0x7fea10156820 2024-03-06T07:22:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.819+0000 7fea1656c640 1 -- 172.21.15.1:0/3947065780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea100686e0 con 0x7fea10156820 2024-03-06T07:22:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.820+0000 7fea0cff9640 1 -- 172.21.15.1:0/3947065780 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea04012040 con 0x7fea10156820 2024-03-06T07:22:19.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.820+0000 7fea0cff9640 1 --2- 172.21.15.1:0/3947065780 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9fc06b4d0 0x7fe9fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.821+0000 7fea0cff9640 1 -- 172.21.15.1:0/3947065780 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6472+0+0 (secure 0 0 0) 0x7fea0408aa80 con 0x7fea10156820 2024-03-06T07:22:19.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.821+0000 7fea0ffff640 1 --2- 172.21.15.1:0/3947065780 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9fc06b4d0 0x7fe9fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.821+0000 7fea0ffff640 1 --2- 172.21.15.1:0/3947065780 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9fc06b4d0 0x7fe9fc06d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fea00004840 tx=0x7fea000078f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:19.825+0000 7fea0cff9640 1 -- 172.21.15.1:0/3947065780 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea04019030 con 0x7fea10156820 2024-03-06T07:22:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.008+0000 7fea1656c640 1 -- 172.21.15.1:0/3947065780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fea100008d0 con 0x7fea10156820 2024-03-06T07:22:20.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.684+0000 7fea0cff9640 1 -- 172.21.15.1:0/3947065780 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs2_data' created v54) v1 ==== 172+0+0 (secure 0 0 0) 0x7fea04059690 con 0x7fea10156820 2024-03-06T07:22:20.686 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs2_data' created 2024-03-06T07:22:20.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.688+0000 7fea1656c640 1 -- 172.21.15.1:0/3947065780 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9fc06b4d0 msgr2=0x7fe9fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.688+0000 7fea1656c640 1 --2- 172.21.15.1:0/3947065780 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9fc06b4d0 0x7fe9fc06d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fea00004840 tx=0x7fea000078f0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.688+0000 7fea1656c640 1 -- 172.21.15.1:0/3947065780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea10156820 msgr2=0x7fea10151530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.688+0000 7fea1656c640 1 --2- 172.21.15.1:0/3947065780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea10156820 0x7fea10151530 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fea04011a70 tx=0x7fea04011e70 comp rx=0 tx=0).stop 2024-03-06T07:22:20.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.688+0000 7fea1656c640 1 -- 172.21.15.1:0/3947065780 shutdown_connections 2024-03-06T07:22:20.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.688+0000 7fea1656c640 1 --2- 172.21.15.1:0/3947065780 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9fc06b4d0 0x7fe9fc06d990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.688+0000 7fea1656c640 1 --2- 172.21.15.1:0/3947065780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea10151a70 0x7fea100ac740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.688+0000 7fea1656c640 1 --2- 172.21.15.1:0/3947065780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea10156820 0x7fea10151530 secure :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fea04011a70 tx=0x7fea04011e70 comp rx=0 tx=0).stop 2024-03-06T07:22:20.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.688+0000 7fea1656c640 1 --2- 172.21.15.1:0/3947065780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea10145a20 0x7fea101562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.688+0000 7fea1656c640 1 -- 172.21.15.1:0/3947065780 >> 172.21.15.1:0/3947065780 conn(0x7fea100c88f0 msgr2=0x7fea1014f830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.688+0000 7fea1656c640 1 -- 172.21.15.1:0/3947065780 shutdown_connections 2024-03-06T07:22:20.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.688+0000 7fea1656c640 1 -- 172.21.15.1:0/3947065780 wait complete. 2024-03-06T07:22:20.708 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs2 cephfs2_metadata cephfs2_data 2024-03-06T07:22:20.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.812+0000 7fa657f02640 1 Processor -- start 2024-03-06T07:22:20.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.812+0000 7fa657f02640 1 -- start start 2024-03-06T07:22:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.813+0000 7fa657f02640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa650150520 0x7fa6501543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.813+0000 7fa657f02640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa6500c32f0 0x7fa6501537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.813+0000 7fa657f02640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa650150900 con 0x7fa650070c80 2024-03-06T07:22:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.813+0000 7fa657f02640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa650071790 con 0x7fa650150520 2024-03-06T07:22:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.813+0000 7fa657f02640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa650157050 con 0x7fa6500c32f0 2024-03-06T07:22:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.813+0000 7fa655c9e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa650150520 0x7fa6501543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.813+0000 7fa655c9e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa650150520 0x7fa6501543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37566/0 (socket says 172.21.15.1:37566) 2024-03-06T07:22:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.813+0000 7fa655c9e640 1 -- 172.21.15.1:0/1291090728 learned_addr learned my addr 172.21.15.1:0/1291090728 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.813+0000 7fa65549d640 1 --2- 172.21.15.1:0/1291090728 >> v2:172.21.15.98:3300/0 conn(0x7fa6500c32f0 0x7fa6501537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.814+0000 7fa655c9e640 1 -- 172.21.15.1:0/1291090728 >> v2:172.21.15.98:3300/0 conn(0x7fa6500c32f0 msgr2=0x7fa6501537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:20.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.814+0000 7fa655c9e640 1 --2- 172.21.15.1:0/1291090728 >> v2:172.21.15.98:3300/0 conn(0x7fa6500c32f0 0x7fa6501537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.814+0000 7fa655c9e640 1 -- 172.21.15.1:0/1291090728 >> v1:172.21.15.98:6789/0 conn(0x7fa650070c80 legacy=0x7fa6500c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.814+0000 7fa655c9e640 1 -- 172.21.15.1:0/1291090728 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa650152fc0 con 0x7fa650150520 2024-03-06T07:22:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.815+0000 7fa655c9e640 1 --2- 172.21.15.1:0/1291090728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa650150520 0x7fa6501543a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa6500732a0 tx=0x7fa64c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fed3e33796892db7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.815+0000 7fa654c9c640 1 -- 172.21.15.1:0/1291090728 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c034070 con 0x7fa650150520 2024-03-06T07:22:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.815+0000 7fa654c9c640 1 -- 172.21.15.1:0/1291090728 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c030860 con 0x7fa650150520 2024-03-06T07:22:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.815+0000 7fa654c9c640 1 -- 172.21.15.1:0/1291090728 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c030c30 con 0x7fa650150520 2024-03-06T07:22:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.815+0000 7fa657f02640 1 -- 172.21.15.1:0/1291090728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa650150520 msgr2=0x7fa6501543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.815+0000 7fa657f02640 1 --2- 172.21.15.1:0/1291090728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa650150520 0x7fa6501543a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa6500732a0 tx=0x7fa64c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.815+0000 7fa657f02640 1 -- 172.21.15.1:0/1291090728 shutdown_connections 2024-03-06T07:22:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.815+0000 7fa657f02640 1 --2- 172.21.15.1:0/1291090728 >> v2:172.21.15.98:3300/0 conn(0x7fa6500c32f0 0x7fa6501537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.815+0000 7fa657f02640 1 --2- 172.21.15.1:0/1291090728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa650150520 0x7fa6501543a0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.815+0000 7fa657f02640 1 -- 172.21.15.1:0/1291090728 >> 172.21.15.1:0/1291090728 conn(0x7fa6500c88f0 msgr2=0x7fa6500c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.816+0000 7fa657f02640 1 -- 172.21.15.1:0/1291090728 shutdown_connections 2024-03-06T07:22:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.816+0000 7fa657f02640 1 -- 172.21.15.1:0/1291090728 wait complete. 2024-03-06T07:22:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.817+0000 7fa657f02640 1 Processor -- start 2024-03-06T07:22:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.817+0000 7fa657f02640 1 -- start start 2024-03-06T07:22:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.817+0000 7fa657f02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa650070c80 0x7fa6501ea660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.817+0000 7fa657f02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6500c32f0 0x7fa6500c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.817+0000 7fa657f02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa650150520 0x7fa6501549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.817+0000 7fa657f02640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6500c7ec0 con 0x7fa650070c80 2024-03-06T07:22:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.817+0000 7fa657f02640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa650157330 con 0x7fa6500c32f0 2024-03-06T07:22:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.817+0000 7fa657f02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa650153d00 con 0x7fa650150520 2024-03-06T07:22:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.817+0000 7fa655c9e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa650070c80 0x7fa6501ea660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.817+0000 7fa655c9e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa650070c80 0x7fa6501ea660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37578/0 (socket says 172.21.15.1:37578) 2024-03-06T07:22:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.817+0000 7fa655c9e640 1 -- 172.21.15.1:0/4020645935 learned_addr learned my addr 172.21.15.1:0/4020645935 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.818+0000 7fa655c9e640 1 -- 172.21.15.1:0/4020645935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa650150520 msgr2=0x7fa6501549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.818+0000 7fa655c9e640 1 --2- 172.21.15.1:0/4020645935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa650150520 0x7fa6501549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.818+0000 7fa655c9e640 1 -- 172.21.15.1:0/4020645935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6500c32f0 msgr2=0x7fa6500c1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.818+0000 7fa655c9e640 1 --2- 172.21.15.1:0/4020645935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6500c32f0 0x7fa6500c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.818+0000 7fa655c9e640 1 -- 172.21.15.1:0/4020645935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa650156490 con 0x7fa650070c80 2024-03-06T07:22:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.818+0000 7fa655c9e640 1 --2- 172.21.15.1:0/4020645935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa650070c80 0x7fa6501ea660 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa64c02b9e0 tx=0x7fa64c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.818+0000 7fa6467fc640 1 -- 172.21.15.1:0/4020645935 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c034040 con 0x7fa650070c80 2024-03-06T07:22:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.818+0000 7fa6467fc640 1 -- 172.21.15.1:0/4020645935 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c039d90 con 0x7fa650070c80 2024-03-06T07:22:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.818+0000 7fa6467fc640 1 -- 172.21.15.1:0/4020645935 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c043610 con 0x7fa650070c80 2024-03-06T07:22:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.818+0000 7fa657f02640 1 -- 172.21.15.1:0/4020645935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6501ee710 con 0x7fa650070c80 2024-03-06T07:22:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.818+0000 7fa657f02640 1 -- 172.21.15.1:0/4020645935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65006d9e0 con 0x7fa650070c80 2024-03-06T07:22:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.819+0000 7fa657f02640 1 -- 172.21.15.1:0/4020645935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6500686e0 con 0x7fa650070c80 2024-03-06T07:22:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.820+0000 7fa6467fc640 1 -- 172.21.15.1:0/4020645935 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa64c037070 con 0x7fa650070c80 2024-03-06T07:22:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.820+0000 7fa6467fc640 1 --2- 172.21.15.1:0/4020645935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa63406b4d0 0x7fa63406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.820+0000 7fa6467fc640 1 -- 172.21.15.1:0/4020645935 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6855+0+0 (secure 0 0 0) 0x7fa64c0ac660 con 0x7fa650070c80 2024-03-06T07:22:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.820+0000 7fa65549d640 1 --2- 172.21.15.1:0/4020645935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa63406b4d0 0x7fa63406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.821+0000 7fa65549d640 1 --2- 172.21.15.1:0/4020645935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa63406b4d0 0x7fa63406d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa640006a90 tx=0x7fa640006a20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:20.824+0000 7fa6467fc640 1 -- 172.21.15.1:0/4020645935 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa64c03e070 con 0x7fa650070c80 2024-03-06T07:22:21.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.045+0000 7fa657f02640 1 -- 172.21.15.1:0/4020645935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"} v 0) v1 -- 0x7fa6500c7b10 con 0x7fa650070c80 2024-03-06T07:22:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.710+0000 7fa6467fc640 1 -- 172.21.15.1:0/4020645935 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"}]=0 new fs with metadata pool 9 and data pool 10 v29) v1 ==== 176+0+0 (secure 0 0 0) 0x7fa64c07b0f0 con 0x7fa650070c80 2024-03-06T07:22:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 9 and data pool 10 2024-03-06T07:22:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.719+0000 7fa657f02640 1 -- 172.21.15.1:0/4020645935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa63406b4d0 msgr2=0x7fa63406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.719+0000 7fa657f02640 1 --2- 172.21.15.1:0/4020645935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa63406b4d0 0x7fa63406d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa640006a90 tx=0x7fa640006a20 comp rx=0 tx=0).stop 2024-03-06T07:22:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.719+0000 7fa657f02640 1 -- 172.21.15.1:0/4020645935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa650070c80 msgr2=0x7fa6501ea660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.719+0000 7fa657f02640 1 --2- 172.21.15.1:0/4020645935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa650070c80 0x7fa6501ea660 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa64c02b9e0 tx=0x7fa64c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.719+0000 7fa657f02640 1 -- 172.21.15.1:0/4020645935 shutdown_connections 2024-03-06T07:22:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.719+0000 7fa657f02640 1 --2- 172.21.15.1:0/4020645935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa63406b4d0 0x7fa63406d990 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.719+0000 7fa657f02640 1 --2- 172.21.15.1:0/4020645935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa650150520 0x7fa6501549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.719+0000 7fa657f02640 1 --2- 172.21.15.1:0/4020645935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6500c32f0 0x7fa6500c1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.719+0000 7fa657f02640 1 --2- 172.21.15.1:0/4020645935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa650070c80 0x7fa6501ea660 secure :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa64c02b9e0 tx=0x7fa64c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.719+0000 7fa657f02640 1 -- 172.21.15.1:0/4020645935 >> 172.21.15.1:0/4020645935 conn(0x7fa6500c88f0 msgr2=0x7fa650152530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.720+0000 7fa657f02640 1 -- 172.21.15.1:0/4020645935 shutdown_connections 2024-03-06T07:22:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.720+0000 7fa657f02640 1 -- 172.21.15.1:0/4020645935 wait complete. 2024-03-06T07:22:21.739 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:21.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.838+0000 7f0eba3a0640 1 Processor -- start 2024-03-06T07:22:21.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.838+0000 7f0eba3a0640 1 -- start start 2024-03-06T07:22:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.839+0000 7f0eba3a0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0eb40c49b0 0x7f0eb40c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.839+0000 7f0eba3a0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0eb40c7e60 0x7f0eb40c3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.839+0000 7f0eba3a0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb41532a0 con 0x7f0eb40c8a70 2024-03-06T07:22:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.839+0000 7f0eba3a0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb4071530 con 0x7f0eb40c7e60 2024-03-06T07:22:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.839+0000 7f0eba3a0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb4150960 con 0x7f0eb40c49b0 2024-03-06T07:22:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.839+0000 7f0eb9b9f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0eb40c7e60 0x7f0eb40c3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.839+0000 7f0eb9b9f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0eb40c7e60 0x7f0eb40c3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37582/0 (socket says 172.21.15.1:37582) 2024-03-06T07:22:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.839+0000 7f0eb9b9f640 1 -- 172.21.15.1:0/1017488095 learned_addr learned my addr 172.21.15.1:0/1017488095 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.840+0000 7f0eb939e640 1 --2- 172.21.15.1:0/1017488095 >> v2:172.21.15.98:3300/0 conn(0x7f0eb40c49b0 0x7f0eb40c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.840+0000 7f0eb9b9f640 1 -- 172.21.15.1:0/1017488095 >> v2:172.21.15.98:3300/0 conn(0x7f0eb40c49b0 msgr2=0x7f0eb40c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.840+0000 7f0eb9b9f640 1 --2- 172.21.15.1:0/1017488095 >> v2:172.21.15.98:3300/0 conn(0x7f0eb40c49b0 0x7f0eb40c8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.840+0000 7f0eb9b9f640 1 -- 172.21.15.1:0/1017488095 >> v1:172.21.15.98:6789/0 conn(0x7f0eb40c8a70 legacy=0x7f0eb40c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.840+0000 7f0eb9b9f640 1 -- 172.21.15.1:0/1017488095 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb4153580 con 0x7f0eb40c7e60 2024-03-06T07:22:21.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.840+0000 7f0eb9b9f640 1 --2- 172.21.15.1:0/1017488095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb40c7e60 0x7f0eb40c3100 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0eb40c62e0 tx=0x7f0eac02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5769110e881ab9f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.841+0000 7f0eabfff640 1 -- 172.21.15.1:0/1017488095 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac02c9f0 con 0x7f0eb40c7e60 2024-03-06T07:22:21.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.841+0000 7f0eabfff640 1 -- 172.21.15.1:0/1017488095 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eac02cb90 con 0x7f0eb40c7e60 2024-03-06T07:22:21.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.841+0000 7f0eabfff640 1 -- 172.21.15.1:0/1017488095 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac0343f0 con 0x7f0eb40c7e60 2024-03-06T07:22:21.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.841+0000 7f0eba3a0640 1 -- 172.21.15.1:0/1017488095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb40c7e60 msgr2=0x7f0eb40c3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.841+0000 7f0eba3a0640 1 --2- 172.21.15.1:0/1017488095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb40c7e60 0x7f0eb40c3100 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0eb40c62e0 tx=0x7f0eac02be50 comp rx=0 tx=0).stop 2024-03-06T07:22:21.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.841+0000 7f0eba3a0640 1 -- 172.21.15.1:0/1017488095 shutdown_connections 2024-03-06T07:22:21.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.841+0000 7f0eba3a0640 1 --2- 172.21.15.1:0/1017488095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb40c7e60 0x7f0eb40c3100 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.841+0000 7f0eba3a0640 1 --2- 172.21.15.1:0/1017488095 >> v2:172.21.15.98:3300/0 conn(0x7f0eb40c49b0 0x7f0eb40c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.841+0000 7f0eba3a0640 1 -- 172.21.15.1:0/1017488095 >> 172.21.15.1:0/1017488095 conn(0x7f0eb406b620 msgr2=0x7f0eb40aa4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.841+0000 7f0eba3a0640 1 -- 172.21.15.1:0/1017488095 shutdown_connections 2024-03-06T07:22:21.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.841+0000 7f0eba3a0640 1 -- 172.21.15.1:0/1017488095 wait complete. 2024-03-06T07:22:21.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.842+0000 7f0eba3a0640 1 Processor -- start 2024-03-06T07:22:21.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.842+0000 7f0eba3a0640 1 -- start start 2024-03-06T07:22:21.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.843+0000 7f0eba3a0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0eb40c49b0 0x7f0eb4156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.843+0000 7f0eba3a0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb40c7e60 0x7f0eb40c1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.843+0000 7f0eba3a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb40c8a70 0x7f0eb41ee130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.843+0000 7f0eba3a0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb4152f90 con 0x7f0eb40c7e60 2024-03-06T07:22:21.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.843+0000 7f0eba3a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb4153b40 con 0x7f0eb40c8a70 2024-03-06T07:22:21.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.843+0000 7f0eba3a0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb40c4d90 con 0x7f0eb40c49b0 2024-03-06T07:22:21.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.843+0000 7f0eb8b9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb40c7e60 0x7f0eb40c1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.843+0000 7f0eb8b9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb40c7e60 0x7f0eb40c1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37586/0 (socket says 172.21.15.1:37586) 2024-03-06T07:22:21.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.843+0000 7f0eb8b9d640 1 -- 172.21.15.1:0/2210225366 learned_addr learned my addr 172.21.15.1:0/2210225366 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:21.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.843+0000 7f0eb9b9f640 1 --2- 172.21.15.1:0/2210225366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb40c8a70 0x7f0eb41ee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.843+0000 7f0eb939e640 1 --2- 172.21.15.1:0/2210225366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0eb40c49b0 0x7f0eb4156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.843+0000 7f0eb8b9d640 1 -- 172.21.15.1:0/2210225366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0eb40c49b0 msgr2=0x7f0eb4156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.843+0000 7f0eb8b9d640 1 --2- 172.21.15.1:0/2210225366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0eb40c49b0 0x7f0eb4156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.843+0000 7f0eb8b9d640 1 -- 172.21.15.1:0/2210225366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb40c8a70 msgr2=0x7f0eb41ee130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.843+0000 7f0eb8b9d640 1 --2- 172.21.15.1:0/2210225366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb40c8a70 0x7f0eb41ee130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.843+0000 7f0eb8b9d640 1 -- 172.21.15.1:0/2210225366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb406d020 con 0x7f0eb40c7e60 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.843+0000 7f0eb9b9f640 1 --2- 172.21.15.1:0/2210225366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb40c8a70 0x7f0eb41ee130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.844+0000 7f0eb8b9d640 1 --2- 172.21.15.1:0/2210225366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb40c7e60 0x7f0eb40c1da0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0ea400b420 tx=0x7f0ea400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.844+0000 7f0ea9ffb640 1 -- 172.21.15.1:0/2210225366 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4011830 con 0x7f0eb40c7e60 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.844+0000 7f0ea9ffb640 1 -- 172.21.15.1:0/2210225366 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea4016070 con 0x7f0eb40c7e60 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.844+0000 7f0ea9ffb640 1 -- 172.21.15.1:0/2210225366 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4011c10 con 0x7f0eb40c7e60 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.844+0000 7f0eba3a0640 1 -- 172.21.15.1:0/2210225366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb406d2c0 con 0x7f0eb40c7e60 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.844+0000 7f0eba3a0640 1 -- 172.21.15.1:0/2210225366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb41eb1d0 con 0x7f0eb40c7e60 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.845+0000 7f0eba3a0640 1 -- 172.21.15.1:0/2210225366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb40ab060 con 0x7f0eb40c7e60 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.845+0000 7f0ea9ffb640 1 -- 172.21.15.1:0/2210225366 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ea401b460 con 0x7f0eb40c7e60 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.846+0000 7f0ea9ffb640 1 --2- 172.21.15.1:0/2210225366 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0e9806b500 0x7f0e9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.846+0000 7f0ea9ffb640 1 -- 172.21.15.1:0/2210225366 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f0ea408a080 con 0x7f0eb40c7e60 2024-03-06T07:22:21.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.846+0000 7f0eb939e640 1 --2- 172.21.15.1:0/2210225366 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0e9806b500 0x7f0e9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.846+0000 7f0eb939e640 1 --2- 172.21.15.1:0/2210225366 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0e9806b500 0x7f0e9806d9c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0006fd0 tx=0x7f0eb0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:21.850+0000 7f0ea9ffb640 1 -- 172.21.15.1:0/2210225366 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0eb40ab060 con 0x7f0eb40c7e60 2024-03-06T07:22:22.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.034+0000 7f0eba3a0640 1 -- 172.21.15.1:0/2210225366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0eb4150200 con 0x7f0eb40c7e60 2024-03-06T07:22:22.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.036+0000 7f0ea9ffb640 1 -- 172.21.15.1:0/2210225366 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+20559 (secure 0 0 0) 0x7f0ea4020020 con 0x7f0eb40c7e60 2024-03-06T07:22:22.038 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:22.038 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":55,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:21.047038+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:03.930333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:04.634569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:22:18.907422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:22:20.028174+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:22.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.037+0000 7f0eba3a0640 1 -- 172.21.15.1:0/2210225366 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0e9806b500 msgr2=0x7f0e9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.038+0000 7f0eba3a0640 1 --2- 172.21.15.1:0/2210225366 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0e9806b500 0x7f0e9806d9c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0006fd0 tx=0x7f0eb0008040 comp rx=0 tx=0).stop 2024-03-06T07:22:22.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.038+0000 7f0eba3a0640 1 -- 172.21.15.1:0/2210225366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb40c7e60 msgr2=0x7f0eb40c1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.038+0000 7f0eba3a0640 1 --2- 172.21.15.1:0/2210225366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb40c7e60 0x7f0eb40c1da0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0ea400b420 tx=0x7f0ea400b8e0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.038+0000 7f0eba3a0640 1 -- 172.21.15.1:0/2210225366 shutdown_connections 2024-03-06T07:22:22.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.038+0000 7f0eba3a0640 1 --2- 172.21.15.1:0/2210225366 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0e9806b500 0x7f0e9806d9c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.038+0000 7f0eba3a0640 1 --2- 172.21.15.1:0/2210225366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb40c8a70 0x7f0eb41ee130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.038+0000 7f0eba3a0640 1 --2- 172.21.15.1:0/2210225366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb40c7e60 0x7f0eb40c1da0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.038+0000 7f0eba3a0640 1 --2- 172.21.15.1:0/2210225366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0eb40c49b0 0x7f0eb4156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.038+0000 7f0eba3a0640 1 -- 172.21.15.1:0/2210225366 >> 172.21.15.1:0/2210225366 conn(0x7f0eb406b620 msgr2=0x7f0eb4151c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.038+0000 7f0eba3a0640 1 -- 172.21.15.1:0/2210225366 shutdown_connections 2024-03-06T07:22:22.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.038+0000 7f0eba3a0640 1 -- 172.21.15.1:0/2210225366 wait complete. 2024-03-06T07:22:22.051 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:22.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.150+0000 7f668633a640 1 Processor -- start 2024-03-06T07:22:22.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.150+0000 7f668633a640 1 -- start start 2024-03-06T07:22:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.151+0000 7f668633a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f66800c6ce0 0x7f668013e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.151+0000 7f668633a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f66800c1a70 0x7f6680158980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.151+0000 7f668633a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66801551b0 con 0x7f668013e6b0 2024-03-06T07:22:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.151+0000 7f668633a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6680155470 con 0x7f66800c1a70 2024-03-06T07:22:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.151+0000 7f668633a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6680155750 con 0x7f66800c6ce0 2024-03-06T07:22:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.152+0000 7f66848d7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f66800c1a70 0x7f6680158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.152+0000 7f66848d7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f66800c1a70 0x7f6680158980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37602/0 (socket says 172.21.15.1:37602) 2024-03-06T07:22:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.152+0000 7f66848d7640 1 -- 172.21.15.1:0/170168269 learned_addr learned my addr 172.21.15.1:0/170168269 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.152+0000 7f667ffff640 1 --2- 172.21.15.1:0/170168269 >> v2:172.21.15.98:3300/0 conn(0x7f66800c6ce0 0x7f668013e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.152+0000 7f66848d7640 1 -- 172.21.15.1:0/170168269 >> v2:172.21.15.98:3300/0 conn(0x7f66800c6ce0 msgr2=0x7f668013e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.152+0000 7f66848d7640 1 --2- 172.21.15.1:0/170168269 >> v2:172.21.15.98:3300/0 conn(0x7f66800c6ce0 0x7f668013e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.152+0000 7f66848d7640 1 -- 172.21.15.1:0/170168269 >> v1:172.21.15.98:6789/0 conn(0x7f668013e6b0 legacy=0x7f66800c1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.152+0000 7f66848d7640 1 -- 172.21.15.1:0/170168269 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6680155a30 con 0x7f66800c1a70 2024-03-06T07:22:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.153+0000 7f66848d7640 1 --2- 172.21.15.1:0/170168269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66800c1a70 0x7f6680158980 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f66800cb470 tx=0x7f667802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b5d82f5630c2b3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.153+0000 7f667effd640 1 -- 172.21.15.1:0/170168269 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667802ca80 con 0x7f66800c1a70 2024-03-06T07:22:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.153+0000 7f667effd640 1 -- 172.21.15.1:0/170168269 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667802cc20 con 0x7f66800c1a70 2024-03-06T07:22:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.153+0000 7f667effd640 1 -- 172.21.15.1:0/170168269 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66780343f0 con 0x7f66800c1a70 2024-03-06T07:22:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.153+0000 7f668633a640 1 -- 172.21.15.1:0/170168269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66800c1a70 msgr2=0x7f6680158980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.153+0000 7f668633a640 1 --2- 172.21.15.1:0/170168269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66800c1a70 0x7f6680158980 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f66800cb470 tx=0x7f667802bee0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.153+0000 7f668633a640 1 -- 172.21.15.1:0/170168269 shutdown_connections 2024-03-06T07:22:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.153+0000 7f668633a640 1 --2- 172.21.15.1:0/170168269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66800c1a70 0x7f6680158980 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.153+0000 7f668633a640 1 --2- 172.21.15.1:0/170168269 >> v2:172.21.15.98:3300/0 conn(0x7f66800c6ce0 0x7f668013e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.153+0000 7f668633a640 1 -- 172.21.15.1:0/170168269 >> 172.21.15.1:0/170168269 conn(0x7f668006da30 msgr2=0x7f6680066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.153+0000 7f668633a640 1 -- 172.21.15.1:0/170168269 shutdown_connections 2024-03-06T07:22:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.153+0000 7f668633a640 1 -- 172.21.15.1:0/170168269 wait complete. 2024-03-06T07:22:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.154+0000 7f668633a640 1 Processor -- start 2024-03-06T07:22:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.155+0000 7f668633a640 1 -- start start 2024-03-06T07:22:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.155+0000 7f668633a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66800c1a70 0x7f6680136d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.155+0000 7f668633a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66800c6ce0 0x7f668013c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.155+0000 7f668633a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f668013e6b0 0x7f66801eac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.155+0000 7f668633a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66800c70c0 con 0x7f66800c1a70 2024-03-06T07:22:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.155+0000 7f668633a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6680155ff0 con 0x7f668013e6b0 2024-03-06T07:22:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.155+0000 7f668633a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66800c7240 con 0x7f66800c6ce0 2024-03-06T07:22:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.155+0000 7f667ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66800c1a70 0x7f6680136d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.155+0000 7f667ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66800c1a70 0x7f6680136d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37618/0 (socket says 172.21.15.1:37618) 2024-03-06T07:22:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.155+0000 7f667ffff640 1 -- 172.21.15.1:0/314910383 learned_addr learned my addr 172.21.15.1:0/314910383 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.156+0000 7f667f7fe640 1 --2- 172.21.15.1:0/314910383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66800c6ce0 0x7f668013c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.156+0000 7f66848d7640 1 --2- 172.21.15.1:0/314910383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f668013e6b0 0x7f66801eac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.156+0000 7f667ffff640 1 -- 172.21.15.1:0/314910383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66800c6ce0 msgr2=0x7f668013c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.156+0000 7f667ffff640 1 --2- 172.21.15.1:0/314910383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66800c6ce0 0x7f668013c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.156+0000 7f667ffff640 1 -- 172.21.15.1:0/314910383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f668013e6b0 msgr2=0x7f66801eac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.156+0000 7f667ffff640 1 --2- 172.21.15.1:0/314910383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f668013e6b0 0x7f66801eac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.156+0000 7f667ffff640 1 -- 172.21.15.1:0/314910383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6680157e60 con 0x7f66800c1a70 2024-03-06T07:22:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.156+0000 7f667ffff640 1 --2- 172.21.15.1:0/314910383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66800c1a70 0x7f6680136d80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f667000e9b0 tx=0x7f667000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.156+0000 7f667cff9640 1 -- 172.21.15.1:0/314910383 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670017070 con 0x7f66800c1a70 2024-03-06T07:22:22.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.156+0000 7f667cff9640 1 -- 172.21.15.1:0/314910383 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667000c9a0 con 0x7f66800c1a70 2024-03-06T07:22:22.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.156+0000 7f668633a640 1 -- 172.21.15.1:0/314910383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6680138a40 con 0x7f66800c1a70 2024-03-06T07:22:22.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.156+0000 7f667cff9640 1 -- 172.21.15.1:0/314910383 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667000cd90 con 0x7f66800c1a70 2024-03-06T07:22:22.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.156+0000 7f668633a640 1 -- 172.21.15.1:0/314910383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66801392e0 con 0x7f66800c1a70 2024-03-06T07:22:22.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.157+0000 7f668633a640 1 -- 172.21.15.1:0/314910383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6680067fd0 con 0x7f66800c1a70 2024-03-06T07:22:22.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.158+0000 7f667cff9640 1 -- 172.21.15.1:0/314910383 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f667001a070 con 0x7f66800c1a70 2024-03-06T07:22:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.158+0000 7f667cff9640 1 --2- 172.21.15.1:0/314910383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f666c06b4d0 0x7f666c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.158+0000 7f667cff9640 1 -- 172.21.15.1:0/314910383 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f667008a9c0 con 0x7f66800c1a70 2024-03-06T07:22:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.158+0000 7f667f7fe640 1 --2- 172.21.15.1:0/314910383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f666c06b4d0 0x7f666c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.159+0000 7f667f7fe640 1 --2- 172.21.15.1:0/314910383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f666c06b4d0 0x7f666c06d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f66800c09a0 tx=0x7f6674009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.162+0000 7f667cff9640 1 -- 172.21.15.1:0/314910383 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f667008ce20 con 0x7f66800c1a70 2024-03-06T07:22:22.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.363+0000 7f668633a640 1 -- 172.21.15.1:0/314910383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66800c6850 con 0x7f66800c1a70 2024-03-06T07:22:22.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.364+0000 7f667cff9640 1 -- 172.21.15.1:0/314910383 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+20559 (secure 0 0 0) 0x7f6670058d10 con 0x7f66800c1a70 2024-03-06T07:22:22.366 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:22.366 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":55,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:21.047038+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:03.930333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:04.634569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:22:18.907422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:22:20.028174+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:22.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.366+0000 7f668633a640 1 -- 172.21.15.1:0/314910383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f666c06b4d0 msgr2=0x7f666c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.366+0000 7f668633a640 1 --2- 172.21.15.1:0/314910383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f666c06b4d0 0x7f666c06d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f66800c09a0 tx=0x7f6674009040 comp rx=0 tx=0).stop 2024-03-06T07:22:22.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.366+0000 7f668633a640 1 -- 172.21.15.1:0/314910383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66800c1a70 msgr2=0x7f6680136d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.366+0000 7f668633a640 1 --2- 172.21.15.1:0/314910383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66800c1a70 0x7f6680136d80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f667000e9b0 tx=0x7f667000ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.366+0000 7f668633a640 1 -- 172.21.15.1:0/314910383 shutdown_connections 2024-03-06T07:22:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.366+0000 7f668633a640 1 --2- 172.21.15.1:0/314910383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f666c06b4d0 0x7f666c06d990 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.366+0000 7f668633a640 1 --2- 172.21.15.1:0/314910383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f668013e6b0 0x7f66801eac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.366+0000 7f668633a640 1 --2- 172.21.15.1:0/314910383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66800c6ce0 0x7f668013c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.366+0000 7f668633a640 1 --2- 172.21.15.1:0/314910383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66800c1a70 0x7f6680136d80 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.366+0000 7f668633a640 1 -- 172.21.15.1:0/314910383 >> 172.21.15.1:0/314910383 conn(0x7f668006da30 msgr2=0x7f66800ca960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.366+0000 7f668633a640 1 -- 172.21.15.1:0/314910383 shutdown_connections 2024-03-06T07:22:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.366+0000 7f668633a640 1 -- 172.21.15.1:0/314910383 wait complete. 2024-03-06T07:22:22.380 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 standby_count_wanted 0 2024-03-06T07:22:22.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.482+0000 7fc5f3e2a640 1 Processor -- start 2024-03-06T07:22:22.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.482+0000 7fc5f3e2a640 1 -- start start 2024-03-06T07:22:22.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.483+0000 7fc5f3e2a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5ec1543a0 0x7fc5ec15e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.483+0000 7fc5f3e2a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ec150900 con 0x7fc5ec150520 2024-03-06T07:22:22.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.483+0000 7fc5f3e2a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ec071790 con 0x7fc5ec0c32f0 2024-03-06T07:22:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.483+0000 7fc5f3e2a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5ec153ed0 con 0x7fc5ec1543a0 2024-03-06T07:22:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.483+0000 7fc5f23c7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5ec1543a0 0x7fc5ec15e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.483+0000 7fc5f1bc6640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc5ec150520 0x7fc5ec1537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51688/0 (socket says 172.21.15.1:51688) 2024-03-06T07:22:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.483+0000 7fc5f1bc6640 1 -- 172.21.15.1:0/1243900880 learned_addr learned my addr 172.21.15.1:0/1243900880 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.483+0000 7fc5f23c7640 1 -- 172.21.15.1:0/1243900880 >> v1:172.21.15.98:6789/0 conn(0x7fc5ec0c32f0 legacy=0x7fc5ec070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.483+0000 7fc5f23c7640 1 -- 172.21.15.1:0/1243900880 >> v1:172.21.15.1:6789/0 conn(0x7fc5ec150520 legacy=0x7fc5ec1537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.483+0000 7fc5f0bc4640 1 -- 172.21.15.1:0/1243900880 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219266263 0 0) 0x7fc5ec150900 con 0x7fc5ec150520 2024-03-06T07:22:22.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.483+0000 7fc5f23c7640 1 -- 172.21.15.1:0/1243900880 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec157050 con 0x7fc5ec1543a0 2024-03-06T07:22:22.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.484+0000 7fc5f23c7640 1 --2- 172.21.15.1:0/1243900880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5ec1543a0 0x7fc5ec15e840 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009d90 tx=0x7fc5e402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d4aff42f86bc50 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.485+0000 7fc5f0bc4640 1 -- 172.21.15.1:0/1243900880 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4039070 con 0x7fc5ec1543a0 2024-03-06T07:22:22.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.485+0000 7fc5f0bc4640 1 -- 172.21.15.1:0/1243900880 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e402cd00 con 0x7fc5ec1543a0 2024-03-06T07:22:22.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.485+0000 7fc5f0bc4640 1 -- 172.21.15.1:0/1243900880 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4034660 con 0x7fc5ec1543a0 2024-03-06T07:22:22.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.485+0000 7fc5f3e2a640 1 -- 172.21.15.1:0/1243900880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5ec1543a0 msgr2=0x7fc5ec15e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.485+0000 7fc5f3e2a640 1 --2- 172.21.15.1:0/1243900880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5ec1543a0 0x7fc5ec15e840 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009d90 tx=0x7fc5e402bf50 comp rx=0 tx=0).stop 2024-03-06T07:22:22.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.485+0000 7fc5f3e2a640 1 -- 172.21.15.1:0/1243900880 shutdown_connections 2024-03-06T07:22:22.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.485+0000 7fc5f3e2a640 1 --2- 172.21.15.1:0/1243900880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5ec1543a0 0x7fc5ec15e840 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.485+0000 7fc5f3e2a640 1 -- 172.21.15.1:0/1243900880 >> 172.21.15.1:0/1243900880 conn(0x7fc5ec0c88f0 msgr2=0x7fc5ec0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.485+0000 7fc5f3e2a640 1 -- 172.21.15.1:0/1243900880 shutdown_connections 2024-03-06T07:22:22.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.485+0000 7fc5f3e2a640 1 -- 172.21.15.1:0/1243900880 wait complete. 2024-03-06T07:22:22.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.486+0000 7fc5f3e2a640 1 Processor -- start 2024-03-06T07:22:22.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.486+0000 7fc5f3e2a640 1 -- start start 2024-03-06T07:22:22.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.487+0000 7fc5f3e2a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5ec0c32f0 0x7fc5ec147a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.487+0000 7fc5f3e2a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5ec150520 0x7fc5ec0c4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.487+0000 7fc5f3e2a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5ec1543a0 0x7fc5ec0c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.487+0000 7fc5f3e2a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec1532a0 con 0x7fc5ec150520 2024-03-06T07:22:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.487+0000 7fc5f3e2a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec1547e0 con 0x7fc5ec1543a0 2024-03-06T07:22:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.487+0000 7fc5f3e2a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec157330 con 0x7fc5ec0c32f0 2024-03-06T07:22:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.487+0000 7fc5f13c5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5ec150520 0x7fc5ec0c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.487+0000 7fc5f13c5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5ec150520 0x7fc5ec0c4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37636/0 (socket says 172.21.15.1:37636) 2024-03-06T07:22:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.487+0000 7fc5f13c5640 1 -- 172.21.15.1:0/4127137397 learned_addr learned my addr 172.21.15.1:0/4127137397 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.487+0000 7fc5f23c7640 1 --2- 172.21.15.1:0/4127137397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5ec1543a0 0x7fc5ec0c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.487+0000 7fc5f1bc6640 1 --2- 172.21.15.1:0/4127137397 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5ec0c32f0 0x7fc5ec147a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.487+0000 7fc5f13c5640 1 -- 172.21.15.1:0/4127137397 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5ec0c32f0 msgr2=0x7fc5ec147a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.487+0000 7fc5f13c5640 1 --2- 172.21.15.1:0/4127137397 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5ec0c32f0 0x7fc5ec147a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.487+0000 7fc5f13c5640 1 -- 172.21.15.1:0/4127137397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5ec1543a0 msgr2=0x7fc5ec0c4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.487+0000 7fc5f13c5640 1 --2- 172.21.15.1:0/4127137397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5ec1543a0 0x7fc5ec0c4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.487+0000 7fc5f13c5640 1 -- 172.21.15.1:0/4127137397 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec148ef0 con 0x7fc5ec150520 2024-03-06T07:22:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.487+0000 7fc5f23c7640 1 --2- 172.21.15.1:0/4127137397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5ec1543a0 0x7fc5ec0c4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.488+0000 7fc5f13c5640 1 --2- 172.21.15.1:0/4127137397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5ec150520 0x7fc5ec0c4850 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00c480 tx=0x7fc5dc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.488+0000 7fc5e27fc640 1 -- 172.21.15.1:0/4127137397 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc0052d0 con 0x7fc5ec150520 2024-03-06T07:22:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.488+0000 7fc5e27fc640 1 -- 172.21.15.1:0/4127137397 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5dc005470 con 0x7fc5ec150520 2024-03-06T07:22:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.488+0000 7fc5e27fc640 1 -- 172.21.15.1:0/4127137397 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc012870 con 0x7fc5ec150520 2024-03-06T07:22:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.488+0000 7fc5f3e2a640 1 -- 172.21.15.1:0/4127137397 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ec0c52d0 con 0x7fc5ec150520 2024-03-06T07:22:22.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.488+0000 7fc5f3e2a640 1 -- 172.21.15.1:0/4127137397 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ec155070 con 0x7fc5ec150520 2024-03-06T07:22:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.489+0000 7fc5f3e2a640 1 -- 172.21.15.1:0/4127137397 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ec0c9200 con 0x7fc5ec150520 2024-03-06T07:22:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.489+0000 7fc5e27fc640 1 -- 172.21.15.1:0/4127137397 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5dc012a10 con 0x7fc5ec150520 2024-03-06T07:22:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.490+0000 7fc5e27fc640 1 --2- 172.21.15.1:0/4127137397 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5d006b4d0 0x7fc5d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.490+0000 7fc5e27fc640 1 -- 172.21.15.1:0/4127137397 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fc5dc08a2e0 con 0x7fc5ec150520 2024-03-06T07:22:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.490+0000 7fc5f1bc6640 1 --2- 172.21.15.1:0/4127137397 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5d006b4d0 0x7fc5d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.491+0000 7fc5f1bc6640 1 --2- 172.21.15.1:0/4127137397 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5d006b4d0 0x7fc5d006d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8000f60 tx=0x7fc5e8007b90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.494+0000 7fc5e27fc640 1 -- 172.21.15.1:0/4127137397 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5ec0c9200 con 0x7fc5ec150520 2024-03-06T07:22:22.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:22.724+0000 7fc5f3e2a640 1 -- 172.21.15.1:0/4127137397 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc5ec0686e0 con 0x7fc5ec150520 2024-03-06T07:22:23.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.718+0000 7fc5e27fc640 1 -- 172.21.15.1:0/4127137397 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"}]=0 v32) v1 ==== 119+0+0 (secure 0 0 0) 0x7fc5dc058d10 con 0x7fc5ec150520 2024-03-06T07:22:23.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.720+0000 7fc5f3e2a640 1 -- 172.21.15.1:0/4127137397 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5d006b4d0 msgr2=0x7fc5d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.720+0000 7fc5f3e2a640 1 --2- 172.21.15.1:0/4127137397 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5d006b4d0 0x7fc5d006d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8000f60 tx=0x7fc5e8007b90 comp rx=0 tx=0).stop 2024-03-06T07:22:23.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.720+0000 7fc5f3e2a640 1 -- 172.21.15.1:0/4127137397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5ec150520 msgr2=0x7fc5ec0c4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.720+0000 7fc5f3e2a640 1 --2- 172.21.15.1:0/4127137397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5ec150520 0x7fc5ec0c4850 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00c480 tx=0x7fc5dc00c940 comp rx=0 tx=0).stop 2024-03-06T07:22:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.720+0000 7fc5f3e2a640 1 -- 172.21.15.1:0/4127137397 shutdown_connections 2024-03-06T07:22:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.720+0000 7fc5f3e2a640 1 --2- 172.21.15.1:0/4127137397 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5d006b4d0 0x7fc5d006d990 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.720+0000 7fc5f3e2a640 1 --2- 172.21.15.1:0/4127137397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5ec1543a0 0x7fc5ec0c4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.720+0000 7fc5f3e2a640 1 --2- 172.21.15.1:0/4127137397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5ec150520 0x7fc5ec0c4850 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.720+0000 7fc5f3e2a640 1 --2- 172.21.15.1:0/4127137397 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5ec0c32f0 0x7fc5ec147a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.720+0000 7fc5f3e2a640 1 -- 172.21.15.1:0/4127137397 >> 172.21.15.1:0/4127137397 conn(0x7fc5ec0c88f0 msgr2=0x7fc5ec14dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.720+0000 7fc5f3e2a640 1 -- 172.21.15.1:0/4127137397 shutdown_connections 2024-03-06T07:22:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.720+0000 7fc5f3e2a640 1 -- 172.21.15.1:0/4127137397 wait complete. 2024-03-06T07:22:23.733 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:22:23.733 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 allow_standby_replay false 2024-03-06T07:22:23.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.836+0000 7f9ef96d4640 1 Processor -- start 2024-03-06T07:22:23.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.836+0000 7f9ef96d4640 1 -- start start 2024-03-06T07:22:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.837+0000 7f9ef96d4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9ef4150520 0x7f9ef41543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.837+0000 7f9ef96d4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9ef40c32f0 0x7f9ef41537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.837+0000 7f9ef96d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ef4070c80 0x7f9ef40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.837+0000 7f9ef96d4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ef4150900 con 0x7f9ef40c32f0 2024-03-06T07:22:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.837+0000 7f9ef96d4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ef4071790 con 0x7f9ef4150520 2024-03-06T07:22:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.837+0000 7f9ef96d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef4157050 con 0x7f9ef4070c80 2024-03-06T07:22:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.837+0000 7f9eebfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9ef40c32f0 0x7f9ef41537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.837+0000 7f9eebfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9ef40c32f0 0x7f9ef41537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37640/0 (socket says 172.21.15.1:37640) 2024-03-06T07:22:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.837+0000 7f9eebfff640 1 -- 172.21.15.1:0/4184265675 learned_addr learned my addr 172.21.15.1:0/4184265675 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.837+0000 7f9ef2ffd640 1 --2- 172.21.15.1:0/4184265675 >> v2:172.21.15.98:3300/0 conn(0x7f9ef4150520 0x7f9ef41543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.837+0000 7f9ef37fe640 1 --2- 172.21.15.1:0/4184265675 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ef4070c80 0x7f9ef40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.838+0000 7f9eebfff640 1 -- 172.21.15.1:0/4184265675 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ef4070c80 msgr2=0x7f9ef40710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.838+0000 7f9eebfff640 1 --2- 172.21.15.1:0/4184265675 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ef4070c80 0x7f9ef40710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.838+0000 7f9eebfff640 1 -- 172.21.15.1:0/4184265675 >> v2:172.21.15.98:3300/0 conn(0x7f9ef4150520 msgr2=0x7f9ef41543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.838+0000 7f9eebfff640 1 --2- 172.21.15.1:0/4184265675 >> v2:172.21.15.98:3300/0 conn(0x7f9ef4150520 0x7f9ef41543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.838+0000 7f9eebfff640 1 -- 172.21.15.1:0/4184265675 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef4152fc0 con 0x7f9ef40c32f0 2024-03-06T07:22:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.838+0000 7f9ef2ffd640 1 --2- 172.21.15.1:0/4184265675 >> v2:172.21.15.98:3300/0 conn(0x7f9ef4150520 0x7f9ef41543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.838+0000 7f9ef37fe640 1 --2- 172.21.15.1:0/4184265675 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ef4070c80 0x7f9ef40710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.839+0000 7f9eebfff640 1 --2- 172.21.15.1:0/4184265675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ef40c32f0 0x7f9ef41537c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9ef406e6c0 tx=0x7f9edc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95f361cc63684ff8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.839+0000 7f9ef27fc640 1 -- 172.21.15.1:0/4184265675 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc035020 con 0x7f9ef40c32f0 2024-03-06T07:22:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.839+0000 7f9ef27fc640 1 -- 172.21.15.1:0/4184265675 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9edc031870 con 0x7f9ef40c32f0 2024-03-06T07:22:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.839+0000 7f9ef27fc640 1 -- 172.21.15.1:0/4184265675 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc031c40 con 0x7f9ef40c32f0 2024-03-06T07:22:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.840+0000 7f9ef96d4640 1 -- 172.21.15.1:0/4184265675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ef40c32f0 msgr2=0x7f9ef41537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.840+0000 7f9ef96d4640 1 --2- 172.21.15.1:0/4184265675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ef40c32f0 0x7f9ef41537c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9ef406e6c0 tx=0x7f9edc02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.840+0000 7f9ef96d4640 1 -- 172.21.15.1:0/4184265675 shutdown_connections 2024-03-06T07:22:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.840+0000 7f9ef96d4640 1 --2- 172.21.15.1:0/4184265675 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ef4070c80 0x7f9ef40710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.840+0000 7f9ef96d4640 1 --2- 172.21.15.1:0/4184265675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ef40c32f0 0x7f9ef41537c0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.840+0000 7f9ef96d4640 1 --2- 172.21.15.1:0/4184265675 >> v2:172.21.15.98:3300/0 conn(0x7f9ef4150520 0x7f9ef41543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.840+0000 7f9ef96d4640 1 -- 172.21.15.1:0/4184265675 >> 172.21.15.1:0/4184265675 conn(0x7f9ef40c88f0 msgr2=0x7f9ef40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.840+0000 7f9ef96d4640 1 -- 172.21.15.1:0/4184265675 shutdown_connections 2024-03-06T07:22:23.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.840+0000 7f9ef96d4640 1 -- 172.21.15.1:0/4184265675 wait complete. 2024-03-06T07:22:23.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.841+0000 7f9ef96d4640 1 Processor -- start 2024-03-06T07:22:23.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.841+0000 7f9ef96d4640 1 -- start start 2024-03-06T07:22:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9ef96d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ef4070c80 0x7f9ef41eab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9ef96d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ef40c32f0 0x7f9ef40c4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9ef96d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ef4150520 0x7f9ef41ee9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9ef96d4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef40c7ec0 con 0x7f9ef4150520 2024-03-06T07:22:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9ef96d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef4157330 con 0x7f9ef40c32f0 2024-03-06T07:22:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9ef96d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef4153d00 con 0x7f9ef4070c80 2024-03-06T07:22:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9ef37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ef4150520 0x7f9ef41ee9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9ef2ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ef4070c80 0x7f9ef41eab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9ef37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ef4150520 0x7f9ef41ee9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37650/0 (socket says 172.21.15.1:37650) 2024-03-06T07:22:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9ef37fe640 1 -- 172.21.15.1:0/2617258442 learned_addr learned my addr 172.21.15.1:0/2617258442 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9eebfff640 1 --2- 172.21.15.1:0/2617258442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ef40c32f0 0x7f9ef40c4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9ef37fe640 1 -- 172.21.15.1:0/2617258442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ef4070c80 msgr2=0x7f9ef41eab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9ef37fe640 1 --2- 172.21.15.1:0/2617258442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ef4070c80 0x7f9ef41eab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9ef37fe640 1 -- 172.21.15.1:0/2617258442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ef40c32f0 msgr2=0x7f9ef40c4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9ef37fe640 1 --2- 172.21.15.1:0/2617258442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ef40c32f0 0x7f9ef40c4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9ef37fe640 1 -- 172.21.15.1:0/2617258442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef41eb050 con 0x7f9ef4150520 2024-03-06T07:22:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9ef37fe640 1 --2- 172.21.15.1:0/2617258442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ef4150520 0x7f9ef41ee9b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9eec007e10 tx=0x7f9eec00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9eeb7fe640 1 -- 172.21.15.1:0/2617258442 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec01a070 con 0x7f9ef4150520 2024-03-06T07:22:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.842+0000 7f9eeb7fe640 1 -- 172.21.15.1:0/2617258442 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eec003b90 con 0x7f9ef4150520 2024-03-06T07:22:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.843+0000 7f9eeb7fe640 1 -- 172.21.15.1:0/2617258442 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec004030 con 0x7f9ef4150520 2024-03-06T07:22:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.843+0000 7f9ef96d4640 1 -- 172.21.15.1:0/2617258442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ef40c2a20 con 0x7f9ef4150520 2024-03-06T07:22:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.843+0000 7f9ef96d4640 1 -- 172.21.15.1:0/2617258442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef41ee1b0 con 0x7f9ef4150520 2024-03-06T07:22:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.844+0000 7f9ef96d4640 1 -- 172.21.15.1:0/2617258442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef40686e0 con 0x7f9ef4150520 2024-03-06T07:22:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.844+0000 7f9eeb7fe640 1 -- 172.21.15.1:0/2617258442 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9eec01d070 con 0x7f9ef4150520 2024-03-06T07:22:23.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.845+0000 7f9eeb7fe640 1 --2- 172.21.15.1:0/2617258442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ed406b4d0 0x7f9ed406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.845+0000 7f9eeb7fe640 1 -- 172.21.15.1:0/2617258442 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f9eec08a860 con 0x7f9ef4150520 2024-03-06T07:22:23.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.845+0000 7f9ef2ffd640 1 --2- 172.21.15.1:0/2617258442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ed406b4d0 0x7f9ed406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.846+0000 7f9ef2ffd640 1 --2- 172.21.15.1:0/2617258442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ed406b4d0 0x7f9ed406d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4006fd0 tx=0x7f9ee4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:23.849+0000 7f9eeb7fe640 1 -- 172.21.15.1:0/2617258442 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9eec010070 con 0x7f9ef4150520 2024-03-06T07:22:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.068+0000 7f9ef96d4640 1 -- 172.21.15.1:0/2617258442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f9ef40c7b10 con 0x7f9ef4150520 2024-03-06T07:22:24.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.719+0000 7f9eeb7fe640 1 -- 172.21.15.1:0/2617258442 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"}]=0 v33) v1 ==== 123+0+0 (secure 0 0 0) 0x7f9eec058c40 con 0x7f9ef4150520 2024-03-06T07:22:24.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.721+0000 7f9ef96d4640 1 -- 172.21.15.1:0/2617258442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ed406b4d0 msgr2=0x7f9ed406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.721+0000 7f9ef96d4640 1 --2- 172.21.15.1:0/2617258442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ed406b4d0 0x7f9ed406d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4006fd0 tx=0x7f9ee4008040 comp rx=0 tx=0).stop 2024-03-06T07:22:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.721+0000 7f9ef96d4640 1 -- 172.21.15.1:0/2617258442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ef4150520 msgr2=0x7f9ef41ee9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.721+0000 7f9ef96d4640 1 --2- 172.21.15.1:0/2617258442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ef4150520 0x7f9ef41ee9b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9eec007e10 tx=0x7f9eec00d4a0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.721+0000 7f9ef96d4640 1 -- 172.21.15.1:0/2617258442 shutdown_connections 2024-03-06T07:22:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.721+0000 7f9ef96d4640 1 --2- 172.21.15.1:0/2617258442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ed406b4d0 0x7f9ed406d990 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.721+0000 7f9ef96d4640 1 --2- 172.21.15.1:0/2617258442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ef4150520 0x7f9ef41ee9b0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.721+0000 7f9ef96d4640 1 --2- 172.21.15.1:0/2617258442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ef40c32f0 0x7f9ef40c4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.721+0000 7f9ef96d4640 1 --2- 172.21.15.1:0/2617258442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ef4070c80 0x7f9ef41eab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.721+0000 7f9ef96d4640 1 -- 172.21.15.1:0/2617258442 >> 172.21.15.1:0/2617258442 conn(0x7f9ef40c88f0 msgr2=0x7f9ef414aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.721+0000 7f9ef96d4640 1 -- 172.21.15.1:0/2617258442 shutdown_connections 2024-03-06T07:22:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.722+0000 7f9ef96d4640 1 -- 172.21.15.1:0/2617258442 wait complete. 2024-03-06T07:22:24.738 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs2' 2024-03-06T07:22:24.738 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:24.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.840+0000 7fd04d3d0640 1 Processor -- start 2024-03-06T07:22:24.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.840+0000 7fd04d3d0640 1 -- start start 2024-03-06T07:22:24.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.841+0000 7fd04d3d0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd0480c4940 0x7fd0480ac930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.841+0000 7fd04d3d0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0480c7420 con 0x7fd0480c4560 2024-03-06T07:22:24.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.841+0000 7fd04d3d0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0480ab760 con 0x7fd0480c39c0 2024-03-06T07:22:24.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.841+0000 7fd04d3d0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0480c76c0 con 0x7fd0480c4940 2024-03-06T07:22:24.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.841+0000 7fd0477fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd0480c4940 0x7fd0480ac930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.841+0000 7fd0477fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd0480c4940 0x7fd0480ac930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47086/0 (socket says 172.21.15.1:47086) 2024-03-06T07:22:24.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.841+0000 7fd0477fe640 1 -- 172.21.15.1:0/188232703 learned_addr learned my addr 172.21.15.1:0/188232703 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:24.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.841+0000 7fd046ffd640 1 --1- 172.21.15.1:0/188232703 >> v1:172.21.15.1:6789/0 conn(0x7fd0480c4560 0x7fd048155c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43848/0 (socket says 172.21.15.1:43848) 2024-03-06T07:22:24.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.841+0000 7fd0477fe640 1 -- 172.21.15.1:0/188232703 >> v1:172.21.15.98:6789/0 conn(0x7fd0480c39c0 legacy=0x7fd0480ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.841+0000 7fd0477fe640 1 -- 172.21.15.1:0/188232703 >> v1:172.21.15.1:6789/0 conn(0x7fd0480c4560 legacy=0x7fd048155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.841+0000 7fd0477fe640 1 -- 172.21.15.1:0/188232703 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0480aba20 con 0x7fd0480c4940 2024-03-06T07:22:24.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.842+0000 7fd0477fe640 1 --2- 172.21.15.1:0/188232703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0480c4940 0x7fd0480ac930 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fd04000d150 tx=0x7fd04002e200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc49fafa48d38a3d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.842+0000 7fd0467fc640 1 -- 172.21.15.1:0/188232703 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04000b0d0 con 0x7fd0480c4940 2024-03-06T07:22:24.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.842+0000 7fd0467fc640 1 -- 172.21.15.1:0/188232703 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04000b270 con 0x7fd0480c4940 2024-03-06T07:22:24.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.842+0000 7fd0467fc640 1 -- 172.21.15.1:0/188232703 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd040009600 con 0x7fd0480c4940 2024-03-06T07:22:24.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.843+0000 7fd04d3d0640 1 -- 172.21.15.1:0/188232703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0480c4940 msgr2=0x7fd0480ac930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.843+0000 7fd04d3d0640 1 --2- 172.21.15.1:0/188232703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0480c4940 0x7fd0480ac930 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fd04000d150 tx=0x7fd04002e200 comp rx=0 tx=0).stop 2024-03-06T07:22:24.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.843+0000 7fd04d3d0640 1 -- 172.21.15.1:0/188232703 shutdown_connections 2024-03-06T07:22:24.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.843+0000 7fd04d3d0640 1 --2- 172.21.15.1:0/188232703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0480c4940 0x7fd0480ac930 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.843+0000 7fd04d3d0640 1 -- 172.21.15.1:0/188232703 >> 172.21.15.1:0/188232703 conn(0x7fd0480c7a50 msgr2=0x7fd0480c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.843+0000 7fd04d3d0640 1 -- 172.21.15.1:0/188232703 shutdown_connections 2024-03-06T07:22:24.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.843+0000 7fd04d3d0640 1 -- 172.21.15.1:0/188232703 wait complete. 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.844+0000 7fd04d3d0640 1 Processor -- start 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.844+0000 7fd04d3d0640 1 -- start start 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.845+0000 7fd04d3d0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0480c39c0 0x7fd0481e6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.845+0000 7fd04d3d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0480c4560 0x7fd0480ac390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.845+0000 7fd04d3d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0480c4940 0x7fd0480bec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.845+0000 7fd04d3d0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0480acff0 con 0x7fd0480c39c0 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.845+0000 7fd04d3d0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0480ace70 con 0x7fd0480c4940 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.845+0000 7fd04d3d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04806e6d0 con 0x7fd0480c4560 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.845+0000 7fd046ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0480c39c0 0x7fd0481e6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.845+0000 7fd046ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0480c39c0 0x7fd0481e6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47090/0 (socket says 172.21.15.1:47090) 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.845+0000 7fd046ffd640 1 -- 172.21.15.1:0/3097426643 learned_addr learned my addr 172.21.15.1:0/3097426643 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.845+0000 7fd046ffd640 1 -- 172.21.15.1:0/3097426643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0480c4560 msgr2=0x7fd0480ac390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.845+0000 7fd046ffd640 1 --2- 172.21.15.1:0/3097426643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0480c4560 0x7fd0480ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.845+0000 7fd046ffd640 1 -- 172.21.15.1:0/3097426643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0480c4940 msgr2=0x7fd0480bec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.845+0000 7fd046ffd640 1 --2- 172.21.15.1:0/3097426643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0480c4940 0x7fd0480bec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.845+0000 7fd046ffd640 1 -- 172.21.15.1:0/3097426643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0481eab40 con 0x7fd0480c39c0 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.845+0000 7fd046ffd640 1 --2- 172.21.15.1:0/3097426643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0480c39c0 0x7fd0481e6970 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd03800c480 tx=0x7fd03800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.846+0000 7fd03f7fe640 1 -- 172.21.15.1:0/3097426643 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd038018070 con 0x7fd0480c39c0 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.846+0000 7fd03f7fe640 1 -- 172.21.15.1:0/3097426643 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0380074a0 con 0x7fd0480c39c0 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.846+0000 7fd04d3d0640 1 -- 172.21.15.1:0/3097426643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd048152830 con 0x7fd0480c39c0 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.846+0000 7fd03f7fe640 1 -- 172.21.15.1:0/3097426643 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd038004a20 con 0x7fd0480c39c0 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.846+0000 7fd04d3d0640 1 -- 172.21.15.1:0/3097426643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0480c0280 con 0x7fd0480c39c0 2024-03-06T07:22:24.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.847+0000 7fd04d3d0640 1 -- 172.21.15.1:0/3097426643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0480686e0 con 0x7fd0480c39c0 2024-03-06T07:22:24.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.847+0000 7fd03f7fe640 1 -- 172.21.15.1:0/3097426643 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd03801b070 con 0x7fd0480c39c0 2024-03-06T07:22:24.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.848+0000 7fd03f7fe640 1 --2- 172.21.15.1:0/3097426643 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd02806b500 0x7fd02806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.848+0000 7fd03f7fe640 1 -- 172.21.15.1:0/3097426643 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fd038089910 con 0x7fd0480c39c0 2024-03-06T07:22:24.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.848+0000 7fd03ffff640 1 --2- 172.21.15.1:0/3097426643 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd02806b500 0x7fd02806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.849+0000 7fd03ffff640 1 --2- 172.21.15.1:0/3097426643 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd02806b500 0x7fd02806d9c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd030000f60 tx=0x7fd0300079b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:24.852+0000 7fd03f7fe640 1 -- 172.21.15.1:0/3097426643 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd038007650 con 0x7fd0480c39c0 2024-03-06T07:22:25.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.074+0000 7fd04d3d0640 1 -- 172.21.15.1:0/3097426643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0480c24a0 con 0x7fd0480c39c0 2024-03-06T07:22:25.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.074+0000 7fd03f7fe640 1 -- 172.21.15.1:0/3097426643 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+6249 (secure 0 0 0) 0x7fd038058340 con 0x7fd0480c39c0 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":33,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5084,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2937068915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2937068915},{"type":"v1","addr":"172.21.15.98:6836","nonce":2937068915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5087,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2828783606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2828783606},{"type":"v1","addr":"172.21.15.98:6837","nonce":2828783606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:05.671602+0000","modified":"2024-03-06T07:22:07.663851+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5778":{"gid":5778,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/131367647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":131367647},{"type":"v1","addr":"172.21.15.1:6834","nonce":131367647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5778,"qdb_cluster":[5778]},"id":3},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:21.047528+0000","modified":"2024-03-06T07:22:24.069892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5344":{"gid":5344,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":7,"addr":"172.21.15.1:6835/3845300501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":3845300501},{"type":"v1","addr":"172.21.15.1:6835","nonce":3845300501}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5344,"qdb_cluster":[5344]},"id":4}]} 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 33 2024-03-06T07:22:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.077+0000 7fd04d3d0640 1 -- 172.21.15.1:0/3097426643 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd02806b500 msgr2=0x7fd02806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.077+0000 7fd04d3d0640 1 --2- 172.21.15.1:0/3097426643 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd02806b500 0x7fd02806d9c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd030000f60 tx=0x7fd0300079b0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.077+0000 7fd04d3d0640 1 -- 172.21.15.1:0/3097426643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0480c39c0 msgr2=0x7fd0481e6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.077+0000 7fd04d3d0640 1 --2- 172.21.15.1:0/3097426643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0480c39c0 0x7fd0481e6970 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd03800c480 tx=0x7fd03800c940 comp rx=0 tx=0).stop 2024-03-06T07:22:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.077+0000 7fd04d3d0640 1 -- 172.21.15.1:0/3097426643 shutdown_connections 2024-03-06T07:22:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.077+0000 7fd04d3d0640 1 --2- 172.21.15.1:0/3097426643 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd02806b500 0x7fd02806d9c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.077+0000 7fd04d3d0640 1 --2- 172.21.15.1:0/3097426643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0480c4940 0x7fd0480bec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.077+0000 7fd04d3d0640 1 --2- 172.21.15.1:0/3097426643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0480c4560 0x7fd0480ac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.077+0000 7fd04d3d0640 1 --2- 172.21.15.1:0/3097426643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0480c39c0 0x7fd0481e6970 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.077+0000 7fd04d3d0640 1 -- 172.21.15.1:0/3097426643 >> 172.21.15.1:0/3097426643 conn(0x7fd0480c7a50 msgr2=0x7fd0480c7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.077+0000 7fd04d3d0640 1 -- 172.21.15.1:0/3097426643 shutdown_connections 2024-03-06T07:22:25.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.077+0000 7fd04d3d0640 1 -- 172.21.15.1:0/3097426643 wait complete. 2024-03-06T07:22:25.090 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:25.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.189+0000 7f2e350aa640 1 Processor -- start 2024-03-06T07:22:25.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.189+0000 7f2e350aa640 1 -- start start 2024-03-06T07:22:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.190+0000 7f2e350aa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e30150520 0x7f2e301543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.190+0000 7f2e350aa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e30070c80 0x7f2e300710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.190+0000 7f2e350aa640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e30150900 con 0x7f2e300c32f0 2024-03-06T07:22:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.190+0000 7f2e350aa640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e30153ed0 con 0x7f2e30070c80 2024-03-06T07:22:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.190+0000 7f2e350aa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e30071790 con 0x7f2e30150520 2024-03-06T07:22:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.190+0000 7f2e2f59e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e30070c80 0x7f2e300710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.190+0000 7f2e2e59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2e300c32f0 0x7f2e301537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43856/0 (socket says 172.21.15.1:43856) 2024-03-06T07:22:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.190+0000 7f2e2e59c640 1 -- 172.21.15.1:0/1953701371 learned_addr learned my addr 172.21.15.1:0/1953701371 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.190+0000 7f2e2ed9d640 1 --2- 172.21.15.1:0/1953701371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e30150520 0x7f2e301543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.190+0000 7f2e2f59e640 1 -- 172.21.15.1:0/1953701371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e30150520 msgr2=0x7f2e301543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.190+0000 7f2e2f59e640 1 --2- 172.21.15.1:0/1953701371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e30150520 0x7f2e301543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.190+0000 7f2e2f59e640 1 -- 172.21.15.1:0/1953701371 >> v1:172.21.15.1:6789/0 conn(0x7f2e300c32f0 legacy=0x7f2e301537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.190+0000 7f2e2f59e640 1 -- 172.21.15.1:0/1953701371 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e30157050 con 0x7f2e30070c80 2024-03-06T07:22:25.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.191+0000 7f2e2f59e640 1 --2- 172.21.15.1:0/1953701371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e30070c80 0x7f2e300710f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f2e28009d90 tx=0x7f2e2802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2e99b2f222292 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.191+0000 7f2e2dd9b640 1 -- 172.21.15.1:0/1953701371 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28039070 con 0x7f2e30070c80 2024-03-06T07:22:25.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.191+0000 7f2e2dd9b640 1 -- 172.21.15.1:0/1953701371 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2802cc50 con 0x7f2e30070c80 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.191+0000 7f2e2dd9b640 1 -- 172.21.15.1:0/1953701371 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28034440 con 0x7f2e30070c80 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.192+0000 7f2e350aa640 1 -- 172.21.15.1:0/1953701371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e30070c80 msgr2=0x7f2e300710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.192+0000 7f2e350aa640 1 --2- 172.21.15.1:0/1953701371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e30070c80 0x7f2e300710f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f2e28009d90 tx=0x7f2e2802bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.192+0000 7f2e350aa640 1 -- 172.21.15.1:0/1953701371 shutdown_connections 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.192+0000 7f2e350aa640 1 --2- 172.21.15.1:0/1953701371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e30070c80 0x7f2e300710f0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.192+0000 7f2e350aa640 1 --2- 172.21.15.1:0/1953701371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e30150520 0x7f2e301543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.192+0000 7f2e350aa640 1 -- 172.21.15.1:0/1953701371 >> 172.21.15.1:0/1953701371 conn(0x7f2e300c88f0 msgr2=0x7f2e300c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.192+0000 7f2e350aa640 1 -- 172.21.15.1:0/1953701371 shutdown_connections 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.192+0000 7f2e350aa640 1 -- 172.21.15.1:0/1953701371 wait complete. 2024-03-06T07:22:25.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.193+0000 7f2e350aa640 1 Processor -- start 2024-03-06T07:22:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.193+0000 7f2e350aa640 1 -- start start 2024-03-06T07:22:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e350aa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e30070c80 0x7f2e300ac470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e350aa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e300c32f0 0x7f2e30145e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e350aa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e30150520 0x7f2e30146370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e350aa640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e30157330 con 0x7f2e30070c80 2024-03-06T07:22:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e350aa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e301532a0 con 0x7f2e300c32f0 2024-03-06T07:22:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e350aa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300c7ec0 con 0x7f2e30150520 2024-03-06T07:22:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e2ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e30070c80 0x7f2e300ac470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e2ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e30070c80 0x7f2e300ac470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47110/0 (socket says 172.21.15.1:47110) 2024-03-06T07:22:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e2ed9d640 1 -- 172.21.15.1:0/3522983828 learned_addr learned my addr 172.21.15.1:0/3522983828 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e2ed9d640 1 -- 172.21.15.1:0/3522983828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e30150520 msgr2=0x7f2e30146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e2f59e640 1 --2- 172.21.15.1:0/3522983828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e30150520 0x7f2e30146370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e2e59c640 1 --2- 172.21.15.1:0/3522983828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e300c32f0 0x7f2e30145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e2ed9d640 1 --2- 172.21.15.1:0/3522983828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e30150520 0x7f2e30146370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e2ed9d640 1 -- 172.21.15.1:0/3522983828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e300c32f0 msgr2=0x7f2e30145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e2ed9d640 1 --2- 172.21.15.1:0/3522983828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e300c32f0 0x7f2e30145e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e2ed9d640 1 -- 172.21.15.1:0/3522983828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e300ad560 con 0x7f2e30070c80 2024-03-06T07:22:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e2f59e640 1 --2- 172.21.15.1:0/3522983828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e30150520 0x7f2e30146370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e2e59c640 1 --2- 172.21.15.1:0/3522983828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e300c32f0 0x7f2e30145e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.194+0000 7f2e2ed9d640 1 --2- 172.21.15.1:0/3522983828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e30070c80 0x7f2e300ac470 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f2e2000c7f0 tx=0x7f2e2000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.195+0000 7f2e177fe640 1 -- 172.21.15.1:0/3522983828 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e20015070 con 0x7f2e30070c80 2024-03-06T07:22:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.195+0000 7f2e177fe640 1 -- 172.21.15.1:0/3522983828 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e20004220 con 0x7f2e30070c80 2024-03-06T07:22:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.195+0000 7f2e350aa640 1 -- 172.21.15.1:0/3522983828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e30146c10 con 0x7f2e30070c80 2024-03-06T07:22:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.195+0000 7f2e177fe640 1 -- 172.21.15.1:0/3522983828 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e20007620 con 0x7f2e30070c80 2024-03-06T07:22:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.195+0000 7f2e350aa640 1 -- 172.21.15.1:0/3522983828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e300ac9b0 con 0x7f2e30070c80 2024-03-06T07:22:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.196+0000 7f2e350aa640 1 -- 172.21.15.1:0/3522983828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e300ad7e0 con 0x7f2e30070c80 2024-03-06T07:22:25.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.196+0000 7f2e177fe640 1 -- 172.21.15.1:0/3522983828 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e200079d0 con 0x7f2e30070c80 2024-03-06T07:22:25.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.197+0000 7f2e177fe640 1 --2- 172.21.15.1:0/3522983828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e1c06b4d0 0x7f2e1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.197+0000 7f2e177fe640 1 -- 172.21.15.1:0/3522983828 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f2e2008a9b0 con 0x7f2e30070c80 2024-03-06T07:22:25.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.197+0000 7f2e2e59c640 1 --2- 172.21.15.1:0/3522983828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e1c06b4d0 0x7f2e1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.197+0000 7f2e2e59c640 1 --2- 172.21.15.1:0/3522983828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e1c06b4d0 0x7f2e1c06d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2e300acd90 tx=0x7f2e24009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.201+0000 7f2e177fe640 1 -- 172.21.15.1:0/3522983828 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e300ad7e0 con 0x7f2e30070c80 2024-03-06T07:22:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.406+0000 7f2e350aa640 1 -- 172.21.15.1:0/3522983828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e30154990 con 0x7f2e30070c80 2024-03-06T07:22:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.407+0000 7f2e177fe640 1 -- 172.21.15.1:0/3522983828 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+20546 (secure 0 0 0) 0x7f2e20088e20 con 0x7f2e30070c80 2024-03-06T07:22:25.409 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:25.410 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":56,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:22.700511+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:03.930333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:04.634569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:22:18.907422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:22:20.028174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.409+0000 7f2e350aa640 1 -- 172.21.15.1:0/3522983828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e1c06b4d0 msgr2=0x7f2e1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.409+0000 7f2e350aa640 1 --2- 172.21.15.1:0/3522983828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e1c06b4d0 0x7f2e1c06d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2e300acd90 tx=0x7f2e24009040 comp rx=0 tx=0).stop 2024-03-06T07:22:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.409+0000 7f2e350aa640 1 -- 172.21.15.1:0/3522983828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e30070c80 msgr2=0x7f2e300ac470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.409+0000 7f2e350aa640 1 --2- 172.21.15.1:0/3522983828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e30070c80 0x7f2e300ac470 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f2e2000c7f0 tx=0x7f2e2000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.409+0000 7f2e350aa640 1 -- 172.21.15.1:0/3522983828 shutdown_connections 2024-03-06T07:22:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.409+0000 7f2e350aa640 1 --2- 172.21.15.1:0/3522983828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e1c06b4d0 0x7f2e1c06d990 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.409+0000 7f2e350aa640 1 --2- 172.21.15.1:0/3522983828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e30150520 0x7f2e30146370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.409+0000 7f2e350aa640 1 --2- 172.21.15.1:0/3522983828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e300c32f0 0x7f2e30145e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.409+0000 7f2e350aa640 1 --2- 172.21.15.1:0/3522983828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e30070c80 0x7f2e300ac470 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.409+0000 7f2e350aa640 1 -- 172.21.15.1:0/3522983828 >> 172.21.15.1:0/3522983828 conn(0x7f2e300c88f0 msgr2=0x7f2e300c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.410+0000 7f2e350aa640 1 -- 172.21.15.1:0/3522983828 shutdown_connections 2024-03-06T07:22:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.410+0000 7f2e350aa640 1 -- 172.21.15.1:0/3522983828 wait complete. 2024-03-06T07:22:25.423 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:22:25.423 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.526+0000 7fa860aed640 1 Processor -- start 2024-03-06T07:22:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.526+0000 7fa860aed640 1 -- start start 2024-03-06T07:22:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.527+0000 7fa860aed640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa85c0c6b10 0x7fa85c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.527+0000 7fa860aed640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85c0c2fc0 con 0x7fa85c0bf9d0 2024-03-06T07:22:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.527+0000 7fa860aed640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85c158da0 con 0x7fa85c152650 2024-03-06T07:22:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.527+0000 7fa860aed640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa85c155800 con 0x7fa85c0c6b10 2024-03-06T07:22:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.527+0000 7fa853fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa85c0c6b10 0x7fa85c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.527+0000 7fa85ad9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa85c0bf9d0 0x7fa85c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43860/0 (socket says 172.21.15.1:43860) 2024-03-06T07:22:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.527+0000 7fa85ad9d640 1 -- 172.21.15.1:0/2974192598 learned_addr learned my addr 172.21.15.1:0/2974192598 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.527+0000 7fa853fff640 1 -- 172.21.15.1:0/2974192598 >> v1:172.21.15.98:6789/0 conn(0x7fa85c152650 legacy=0x7fa85c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.527+0000 7fa853fff640 1 -- 172.21.15.1:0/2974192598 >> v1:172.21.15.1:6789/0 conn(0x7fa85c0bf9d0 legacy=0x7fa85c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.527+0000 7fa859d9b640 1 -- 172.21.15.1:0/2974192598 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174886355 0 0) 0x7fa85c0c2fc0 con 0x7fa85c0bf9d0 2024-03-06T07:22:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.527+0000 7fa853fff640 1 -- 172.21.15.1:0/2974192598 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c0c6690 con 0x7fa85c0c6b10 2024-03-06T07:22:25.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.528+0000 7fa853fff640 1 --2- 172.21.15.1:0/2974192598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa85c0c6b10 0x7fa85c155c90 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa8440097a0 tx=0x7fa84402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b58e9594f1c5437e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.528+0000 7fa859d9b640 1 -- 172.21.15.1:0/2974192598 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa844035020 con 0x7fa85c0c6b10 2024-03-06T07:22:25.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.529+0000 7fa859d9b640 1 -- 172.21.15.1:0/2974192598 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa844031830 con 0x7fa85c0c6b10 2024-03-06T07:22:25.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.529+0000 7fa859d9b640 1 -- 172.21.15.1:0/2974192598 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa844031c00 con 0x7fa85c0c6b10 2024-03-06T07:22:25.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.529+0000 7fa860aed640 1 -- 172.21.15.1:0/2974192598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa85c0c6b10 msgr2=0x7fa85c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.529+0000 7fa860aed640 1 --2- 172.21.15.1:0/2974192598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa85c0c6b10 0x7fa85c155c90 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa8440097a0 tx=0x7fa84402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.529+0000 7fa860aed640 1 -- 172.21.15.1:0/2974192598 shutdown_connections 2024-03-06T07:22:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.529+0000 7fa860aed640 1 --2- 172.21.15.1:0/2974192598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa85c0c6b10 0x7fa85c155c90 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.529+0000 7fa860aed640 1 -- 172.21.15.1:0/2974192598 >> 172.21.15.1:0/2974192598 conn(0x7fa85c066ee0 msgr2=0x7fa85c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.529+0000 7fa860aed640 1 -- 172.21.15.1:0/2974192598 shutdown_connections 2024-03-06T07:22:25.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.529+0000 7fa860aed640 1 -- 172.21.15.1:0/2974192598 wait complete. 2024-03-06T07:22:25.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.530+0000 7fa860aed640 1 Processor -- start 2024-03-06T07:22:25.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.530+0000 7fa860aed640 1 -- start start 2024-03-06T07:22:25.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.531+0000 7fa860aed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa85c0bf9d0 0x7fa85c154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.531+0000 7fa860aed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa85c152650 0x7fa85c1eaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.531+0000 7fa860aed640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa85c0c6020 0x7fa85c1eb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.531+0000 7fa860aed640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c1551b0 con 0x7fa85c0c6020 2024-03-06T07:22:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.531+0000 7fa860aed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c0c6fa0 con 0x7fa85c152650 2024-03-06T07:22:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.531+0000 7fa860aed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c155490 con 0x7fa85c0bf9d0 2024-03-06T07:22:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.531+0000 7fa85ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa85c0c6020 0x7fa85c1eb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.531+0000 7fa85ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa85c0c6020 0x7fa85c1eb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47124/0 (socket says 172.21.15.1:47124) 2024-03-06T07:22:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.531+0000 7fa85ad9d640 1 -- 172.21.15.1:0/32334599 learned_addr learned my addr 172.21.15.1:0/32334599 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.531+0000 7fa853fff640 1 --2- 172.21.15.1:0/32334599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa85c152650 0x7fa85c1eaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.531+0000 7fa85a59c640 1 --2- 172.21.15.1:0/32334599 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa85c0bf9d0 0x7fa85c154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.531+0000 7fa85ad9d640 1 -- 172.21.15.1:0/32334599 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa85c0bf9d0 msgr2=0x7fa85c154770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.532+0000 7fa85ad9d640 1 --2- 172.21.15.1:0/32334599 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa85c0bf9d0 0x7fa85c154770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.532+0000 7fa85ad9d640 1 -- 172.21.15.1:0/32334599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa85c152650 msgr2=0x7fa85c1eaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.532+0000 7fa85ad9d640 1 --2- 172.21.15.1:0/32334599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa85c152650 0x7fa85c1eaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.532+0000 7fa85ad9d640 1 -- 172.21.15.1:0/32334599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c1ef120 con 0x7fa85c0c6020 2024-03-06T07:22:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.532+0000 7fa85a59c640 1 --2- 172.21.15.1:0/32334599 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa85c0bf9d0 0x7fa85c154770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.532+0000 7fa853fff640 1 --2- 172.21.15.1:0/32334599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa85c152650 0x7fa85c1eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.532+0000 7fa85ad9d640 1 --2- 172.21.15.1:0/32334599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa85c0c6020 0x7fa85c1eb490 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa85400c730 tx=0x7fa85400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.532+0000 7fa852ffd640 1 -- 172.21.15.1:0/32334599 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa854018070 con 0x7fa85c0c6020 2024-03-06T07:22:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.532+0000 7fa852ffd640 1 -- 172.21.15.1:0/32334599 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85400ce40 con 0x7fa85c0c6020 2024-03-06T07:22:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.532+0000 7fa860aed640 1 -- 172.21.15.1:0/32334599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa85c1eee40 con 0x7fa85c0c6020 2024-03-06T07:22:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.532+0000 7fa852ffd640 1 -- 172.21.15.1:0/32334599 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85400a860 con 0x7fa85c0c6020 2024-03-06T07:22:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.532+0000 7fa860aed640 1 -- 172.21.15.1:0/32334599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa85c0c9c10 con 0x7fa85c0c6020 2024-03-06T07:22:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.533+0000 7fa860aed640 1 -- 172.21.15.1:0/32334599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa85c067fd0 con 0x7fa85c0c6020 2024-03-06T07:22:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.534+0000 7fa852ffd640 1 -- 172.21.15.1:0/32334599 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa85401b070 con 0x7fa85c0c6020 2024-03-06T07:22:25.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.534+0000 7fa852ffd640 1 --2- 172.21.15.1:0/32334599 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa83c06b4d0 0x7fa83c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.534+0000 7fa852ffd640 1 -- 172.21.15.1:0/32334599 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fa854089bd0 con 0x7fa85c0c6020 2024-03-06T07:22:25.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.534+0000 7fa85a59c640 1 --2- 172.21.15.1:0/32334599 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa83c06b4d0 0x7fa83c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.535+0000 7fa85a59c640 1 --2- 172.21.15.1:0/32334599 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa83c06b4d0 0x7fa83c06d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0c50b0 tx=0x7fa84c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.541+0000 7fa852ffd640 1 -- 172.21.15.1:0/32334599 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa854058600 con 0x7fa85c0c6020 2024-03-06T07:22:25.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.754+0000 7fa860aed640 1 -- 172.21.15.1:0/32334599 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa85c0b06f0 con 0x7fa83c06b4d0 2024-03-06T07:22:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.766+0000 7fa852ffd640 1 -- 172.21.15.1:0/32334599 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490559 (secure 0 0 0) 0x7fa85c0b06f0 con 0x7fa83c06b4d0 2024-03-06T07:22:25.773 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:25.777 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:22:25.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.771+0000 7fa860aed640 1 -- 172.21.15.1:0/32334599 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa83c06b4d0 msgr2=0x7fa83c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.771+0000 7fa860aed640 1 --2- 172.21.15.1:0/32334599 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa83c06b4d0 0x7fa83c06d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0c50b0 tx=0x7fa84c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.771+0000 7fa860aed640 1 -- 172.21.15.1:0/32334599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa85c0c6020 msgr2=0x7fa85c1eb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.771+0000 7fa860aed640 1 --2- 172.21.15.1:0/32334599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa85c0c6020 0x7fa85c1eb490 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa85400c730 tx=0x7fa85400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.771+0000 7fa860aed640 1 -- 172.21.15.1:0/32334599 shutdown_connections 2024-03-06T07:22:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.771+0000 7fa860aed640 1 --2- 172.21.15.1:0/32334599 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa83c06b4d0 0x7fa83c06d990 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.771+0000 7fa860aed640 1 --2- 172.21.15.1:0/32334599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa85c0c6020 0x7fa85c1eb490 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.771+0000 7fa860aed640 1 --2- 172.21.15.1:0/32334599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa85c152650 0x7fa85c1eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.771+0000 7fa860aed640 1 --2- 172.21.15.1:0/32334599 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa85c0bf9d0 0x7fa85c154770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.771+0000 7fa860aed640 1 -- 172.21.15.1:0/32334599 >> 172.21.15.1:0/32334599 conn(0x7fa85c066ee0 msgr2=0x7fa85c0c3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.771+0000 7fa860aed640 1 -- 172.21.15.1:0/32334599 shutdown_connections 2024-03-06T07:22:25.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.771+0000 7fa860aed640 1 -- 172.21.15.1:0/32334599 wait complete. 2024-03-06T07:22:25.784 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-03-06T07:22:24.717528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":375,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238280,"kb_used_data":5472,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736440,"statfs":{"total":773094113280,"available":772850114560,"internally_reserved":0,"allocated":5603328,"data_stored":3651816,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.033398"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007985+0000","last_change":"2024-03-06T07:22:21.717661+0000","last_active":"2024-03-06T07:22:23.007985+0000","last_peered":"2024-03-06T07:22:23.007985+0000","last_clean":"2024-03-06T07:22:23.007985+0000","last_became_active":"2024-03-06T07:22:21.717087+0000","last_became_peered":"2024-03-06T07:22:21.717087+0000","last_unstale":"2024-03-06T07:22:23.007985+0000","last_undegraded":"2024-03-06T07:22:23.007985+0000","last_fullsized":"2024-03-06T07:22:23.007985+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:01.959177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007560+0000","last_change":"2024-03-06T07:22:06.323021+0000","last_active":"2024-03-06T07:22:23.007560+0000","last_peered":"2024-03-06T07:22:23.007560+0000","last_clean":"2024-03-06T07:22:23.007560+0000","last_became_active":"2024-03-06T07:22:06.319646+0000","last_became_peered":"2024-03-06T07:22:06.319646+0000","last_unstale":"2024-03-06T07:22:23.007560+0000","last_undegraded":"2024-03-06T07:22:23.007560+0000","last_fullsized":"2024-03-06T07:22:23.007560+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:51.508838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.009062+0000","last_change":"2024-03-06T07:22:21.716822+0000","last_active":"2024-03-06T07:22:23.009062+0000","last_peered":"2024-03-06T07:22:23.009062+0000","last_clean":"2024-03-06T07:22:23.009062+0000","last_became_active":"2024-03-06T07:22:21.715576+0000","last_became_peered":"2024-03-06T07:22:21.715576+0000","last_unstale":"2024-03-06T07:22:23.009062+0000","last_undegraded":"2024-03-06T07:22:23.009062+0000","last_fullsized":"2024-03-06T07:22:23.009062+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:05.319823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008495+0000","last_change":"2024-03-06T07:22:06.325529+0000","last_active":"2024-03-06T07:22:23.008495+0000","last_peered":"2024-03-06T07:22:23.008495+0000","last_clean":"2024-03-06T07:22:23.008495+0000","last_became_active":"2024-03-06T07:22:06.324758+0000","last_became_peered":"2024-03-06T07:22:06.324758+0000","last_unstale":"2024-03-06T07:22:23.008495+0000","last_undegraded":"2024-03-06T07:22:23.008495+0000","last_fullsized":"2024-03-06T07:22:23.008495+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:02.558815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007218+0000","last_change":"2024-03-06T07:22:21.725743+0000","last_active":"2024-03-06T07:22:23.007218+0000","last_peered":"2024-03-06T07:22:23.007218+0000","last_clean":"2024-03-06T07:22:23.007218+0000","last_became_active":"2024-03-06T07:22:21.724872+0000","last_became_peered":"2024-03-06T07:22:21.724872+0000","last_unstale":"2024-03-06T07:22:23.007218+0000","last_undegraded":"2024-03-06T07:22:23.007218+0000","last_fullsized":"2024-03-06T07:22:23.007218+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:36.273030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007517+0000","last_change":"2024-03-06T07:22:06.321470+0000","last_active":"2024-03-06T07:22:23.007517+0000","last_peered":"2024-03-06T07:22:23.007517+0000","last_clean":"2024-03-06T07:22:23.007517+0000","last_became_active":"2024-03-06T07:22:06.318716+0000","last_became_peered":"2024-03-06T07:22:06.318716+0000","last_unstale":"2024-03-06T07:22:23.007517+0000","last_undegraded":"2024-03-06T07:22:23.007517+0000","last_fullsized":"2024-03-06T07:22:23.007517+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:16.706902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007730+0000","last_change":"2024-03-06T07:22:21.709292+0000","last_active":"2024-03-06T07:22:23.007730+0000","last_peered":"2024-03-06T07:22:23.007730+0000","last_clean":"2024-03-06T07:22:23.007730+0000","last_became_active":"2024-03-06T07:22:21.709012+0000","last_became_peered":"2024-03-06T07:22:21.709012+0000","last_unstale":"2024-03-06T07:22:23.007730+0000","last_undegraded":"2024-03-06T07:22:23.007730+0000","last_fullsized":"2024-03-06T07:22:23.007730+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:40.345682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008291+0000","last_change":"2024-03-06T07:22:06.321775+0000","last_active":"2024-03-06T07:22:23.008291+0000","last_peered":"2024-03-06T07:22:23.008291+0000","last_clean":"2024-03-06T07:22:23.008291+0000","last_became_active":"2024-03-06T07:22:06.321437+0000","last_became_peered":"2024-03-06T07:22:06.321437+0000","last_unstale":"2024-03-06T07:22:23.008291+0000","last_undegraded":"2024-03-06T07:22:23.008291+0000","last_fullsized":"2024-03-06T07:22:23.008291+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:53.049705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.704022+0000","last_change":"2024-03-06T07:22:21.708405+0000","last_active":"2024-03-06T07:22:22.704022+0000","last_peered":"2024-03-06T07:22:22.704022+0000","last_clean":"2024-03-06T07:22:22.704022+0000","last_became_active":"2024-03-06T07:22:21.708146+0000","last_became_peered":"2024-03-06T07:22:21.708146+0000","last_unstale":"2024-03-06T07:22:22.704022+0000","last_undegraded":"2024-03-06T07:22:22.704022+0000","last_fullsized":"2024-03-06T07:22:22.704022+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:35.759740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008405+0000","last_change":"2024-03-06T07:22:06.317556+0000","last_active":"2024-03-06T07:22:23.008405+0000","last_peered":"2024-03-06T07:22:23.008405+0000","last_clean":"2024-03-06T07:22:23.008405+0000","last_became_active":"2024-03-06T07:22:06.317298+0000","last_became_peered":"2024-03-06T07:22:06.317298+0000","last_unstale":"2024-03-06T07:22:23.008405+0000","last_undegraded":"2024-03-06T07:22:23.008405+0000","last_fullsized":"2024-03-06T07:22:23.008405+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:45.397748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719247+0000","last_change":"2024-03-06T07:22:21.725231+0000","last_active":"2024-03-06T07:22:22.719247+0000","last_peered":"2024-03-06T07:22:22.719247+0000","last_clean":"2024-03-06T07:22:22.719247+0000","last_became_active":"2024-03-06T07:22:21.724959+0000","last_became_peered":"2024-03-06T07:22:21.724959+0000","last_unstale":"2024-03-06T07:22:22.719247+0000","last_undegraded":"2024-03-06T07:22:22.719247+0000","last_fullsized":"2024-03-06T07:22:22.719247+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:11.190593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008143+0000","last_change":"2024-03-06T07:22:06.323812+0000","last_active":"2024-03-06T07:22:23.008143+0000","last_peered":"2024-03-06T07:22:23.008143+0000","last_clean":"2024-03-06T07:22:23.008143+0000","last_became_active":"2024-03-06T07:22:06.323558+0000","last_became_peered":"2024-03-06T07:22:06.323558+0000","last_unstale":"2024-03-06T07:22:23.008143+0000","last_undegraded":"2024-03-06T07:22:23.008143+0000","last_fullsized":"2024-03-06T07:22:23.008143+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:37.954594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705600+0000","last_change":"2024-03-06T07:22:21.727129+0000","last_active":"2024-03-06T07:22:22.705600+0000","last_peered":"2024-03-06T07:22:22.705600+0000","last_clean":"2024-03-06T07:22:22.705600+0000","last_became_active":"2024-03-06T07:22:21.726856+0000","last_became_peered":"2024-03-06T07:22:21.726856+0000","last_unstale":"2024-03-06T07:22:22.705600+0000","last_undegraded":"2024-03-06T07:22:22.705600+0000","last_fullsized":"2024-03-06T07:22:22.705600+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:10.041200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006875+0000","last_change":"2024-03-06T07:22:06.322447+0000","last_active":"2024-03-06T07:22:23.006875+0000","last_peered":"2024-03-06T07:22:23.006875+0000","last_clean":"2024-03-06T07:22:23.006875+0000","last_became_active":"2024-03-06T07:22:06.318369+0000","last_became_peered":"2024-03-06T07:22:06.318369+0000","last_unstale":"2024-03-06T07:22:23.006875+0000","last_undegraded":"2024-03-06T07:22:23.006875+0000","last_fullsized":"2024-03-06T07:22:23.006875+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:50.150401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006916+0000","last_change":"2024-03-06T07:22:21.720925+0000","last_active":"2024-03-06T07:22:23.006916+0000","last_peered":"2024-03-06T07:22:23.006916+0000","last_clean":"2024-03-06T07:22:23.006916+0000","last_became_active":"2024-03-06T07:22:21.720681+0000","last_became_peered":"2024-03-06T07:22:21.720681+0000","last_unstale":"2024-03-06T07:22:23.006916+0000","last_undegraded":"2024-03-06T07:22:23.006916+0000","last_fullsized":"2024-03-06T07:22:23.006916+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:55.798503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705194+0000","last_change":"2024-03-06T07:22:06.328376+0000","last_active":"2024-03-06T07:22:22.705194+0000","last_peered":"2024-03-06T07:22:22.705194+0000","last_clean":"2024-03-06T07:22:22.705194+0000","last_became_active":"2024-03-06T07:22:06.327042+0000","last_became_peered":"2024-03-06T07:22:06.327042+0000","last_unstale":"2024-03-06T07:22:22.705194+0000","last_undegraded":"2024-03-06T07:22:22.705194+0000","last_fullsized":"2024-03-06T07:22:22.705194+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:34.670786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008331+0000","last_change":"2024-03-06T07:22:21.721756+0000","last_active":"2024-03-06T07:22:23.008331+0000","last_peered":"2024-03-06T07:22:23.008331+0000","last_clean":"2024-03-06T07:22:23.008331+0000","last_became_active":"2024-03-06T07:22:21.721498+0000","last_became_peered":"2024-03-06T07:22:21.721498+0000","last_unstale":"2024-03-06T07:22:23.008331+0000","last_undegraded":"2024-03-06T07:22:23.008331+0000","last_fullsized":"2024-03-06T07:22:23.008331+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:02.370302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705869+0000","last_change":"2024-03-06T07:22:06.335876+0000","last_active":"2024-03-06T07:22:22.705869+0000","last_peered":"2024-03-06T07:22:22.705869+0000","last_clean":"2024-03-06T07:22:22.705869+0000","last_became_active":"2024-03-06T07:22:06.335575+0000","last_became_peered":"2024-03-06T07:22:06.335575+0000","last_unstale":"2024-03-06T07:22:22.705869+0000","last_undegraded":"2024-03-06T07:22:22.705869+0000","last_fullsized":"2024-03-06T07:22:22.705869+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:19.402946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719507+0000","last_change":"2024-03-06T07:22:21.724127+0000","last_active":"2024-03-06T07:22:22.719507+0000","last_peered":"2024-03-06T07:22:22.719507+0000","last_clean":"2024-03-06T07:22:22.719507+0000","last_became_active":"2024-03-06T07:22:21.723801+0000","last_became_peered":"2024-03-06T07:22:21.723801+0000","last_unstale":"2024-03-06T07:22:22.719507+0000","last_undegraded":"2024-03-06T07:22:22.719507+0000","last_fullsized":"2024-03-06T07:22:22.719507+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:28.340044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007778+0000","last_change":"2024-03-06T07:22:06.320896+0000","last_active":"2024-03-06T07:22:23.007778+0000","last_peered":"2024-03-06T07:22:23.007778+0000","last_clean":"2024-03-06T07:22:23.007778+0000","last_became_active":"2024-03-06T07:22:06.320643+0000","last_became_peered":"2024-03-06T07:22:06.320643+0000","last_unstale":"2024-03-06T07:22:23.007778+0000","last_undegraded":"2024-03-06T07:22:23.007778+0000","last_fullsized":"2024-03-06T07:22:23.007778+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:09.367201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008906+0000","last_change":"2024-03-06T07:22:21.725159+0000","last_active":"2024-03-06T07:22:23.008906+0000","last_peered":"2024-03-06T07:22:23.008906+0000","last_clean":"2024-03-06T07:22:23.008906+0000","last_became_active":"2024-03-06T07:22:21.724897+0000","last_became_peered":"2024-03-06T07:22:21.724897+0000","last_unstale":"2024-03-06T07:22:23.008906+0000","last_undegraded":"2024-03-06T07:22:23.008906+0000","last_fullsized":"2024-03-06T07:22:23.008906+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:53.042875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.704526+0000","last_change":"2024-03-06T07:22:06.322171+0000","last_active":"2024-03-06T07:22:22.704526+0000","last_peered":"2024-03-06T07:22:22.704526+0000","last_clean":"2024-03-06T07:22:22.704526+0000","last_became_active":"2024-03-06T07:22:06.321891+0000","last_became_peered":"2024-03-06T07:22:06.321891+0000","last_unstale":"2024-03-06T07:22:22.704526+0000","last_undegraded":"2024-03-06T07:22:22.704526+0000","last_fullsized":"2024-03-06T07:22:22.704526+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:02.261930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008176+0000","last_change":"2024-03-06T07:22:21.722673+0000","last_active":"2024-03-06T07:22:23.008176+0000","last_peered":"2024-03-06T07:22:23.008176+0000","last_clean":"2024-03-06T07:22:23.008176+0000","last_became_active":"2024-03-06T07:22:21.722424+0000","last_became_peered":"2024-03-06T07:22:21.722424+0000","last_unstale":"2024-03-06T07:22:23.008176+0000","last_undegraded":"2024-03-06T07:22:23.008176+0000","last_fullsized":"2024-03-06T07:22:23.008176+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:08.186982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705909+0000","last_change":"2024-03-06T07:22:06.321590+0000","last_active":"2024-03-06T07:22:22.705909+0000","last_peered":"2024-03-06T07:22:22.705909+0000","last_clean":"2024-03-06T07:22:22.705909+0000","last_became_active":"2024-03-06T07:22:06.321334+0000","last_became_peered":"2024-03-06T07:22:06.321334+0000","last_unstale":"2024-03-06T07:22:22.705909+0000","last_undegraded":"2024-03-06T07:22:22.705909+0000","last_fullsized":"2024-03-06T07:22:22.705909+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:49.931180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007904+0000","last_change":"2024-03-06T07:22:21.726103+0000","last_active":"2024-03-06T07:22:23.007904+0000","last_peered":"2024-03-06T07:22:23.007904+0000","last_clean":"2024-03-06T07:22:23.007904+0000","last_became_active":"2024-03-06T07:22:21.723373+0000","last_became_peered":"2024-03-06T07:22:21.723373+0000","last_unstale":"2024-03-06T07:22:23.007904+0000","last_undegraded":"2024-03-06T07:22:23.007904+0000","last_fullsized":"2024-03-06T07:22:23.007904+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:28.340044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007337+0000","last_change":"2024-03-06T07:22:06.328682+0000","last_active":"2024-03-06T07:22:23.007337+0000","last_peered":"2024-03-06T07:22:23.007337+0000","last_clean":"2024-03-06T07:22:23.007337+0000","last_became_active":"2024-03-06T07:22:06.328418+0000","last_became_peered":"2024-03-06T07:22:06.328418+0000","last_unstale":"2024-03-06T07:22:23.007337+0000","last_undegraded":"2024-03-06T07:22:23.007337+0000","last_fullsized":"2024-03-06T07:22:23.007337+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:52.638347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.703876+0000","last_change":"2024-03-06T07:22:21.718432+0000","last_active":"2024-03-06T07:22:22.703876+0000","last_peered":"2024-03-06T07:22:22.703876+0000","last_clean":"2024-03-06T07:22:22.703876+0000","last_became_active":"2024-03-06T07:22:21.718193+0000","last_became_peered":"2024-03-06T07:22:21.718193+0000","last_unstale":"2024-03-06T07:22:22.703876+0000","last_undegraded":"2024-03-06T07:22:22.703876+0000","last_fullsized":"2024-03-06T07:22:22.703876+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:47.455410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007482+0000","last_change":"2024-03-06T07:22:06.330122+0000","last_active":"2024-03-06T07:22:23.007482+0000","last_peered":"2024-03-06T07:22:23.007482+0000","last_clean":"2024-03-06T07:22:23.007482+0000","last_became_active":"2024-03-06T07:22:06.329780+0000","last_became_peered":"2024-03-06T07:22:06.329780+0000","last_unstale":"2024-03-06T07:22:23.007482+0000","last_undegraded":"2024-03-06T07:22:23.007482+0000","last_fullsized":"2024-03-06T07:22:23.007482+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:38.898151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719695+0000","last_change":"2024-03-06T07:22:21.730693+0000","last_active":"2024-03-06T07:22:22.719695+0000","last_peered":"2024-03-06T07:22:22.719695+0000","last_clean":"2024-03-06T07:22:22.719695+0000","last_became_active":"2024-03-06T07:22:21.730389+0000","last_became_peered":"2024-03-06T07:22:21.730389+0000","last_unstale":"2024-03-06T07:22:22.719695+0000","last_undegraded":"2024-03-06T07:22:22.719695+0000","last_fullsized":"2024-03-06T07:22:22.719695+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:08.186982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008024+0000","last_change":"2024-03-06T07:22:06.321523+0000","last_active":"2024-03-06T07:22:23.008024+0000","last_peered":"2024-03-06T07:22:23.008024+0000","last_clean":"2024-03-06T07:22:23.008024+0000","last_became_active":"2024-03-06T07:22:06.321224+0000","last_became_peered":"2024-03-06T07:22:06.321224+0000","last_unstale":"2024-03-06T07:22:23.008024+0000","last_undegraded":"2024-03-06T07:22:23.008024+0000","last_fullsized":"2024-03-06T07:22:23.008024+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:05.731939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007761+0000","last_change":"2024-03-06T07:22:21.720005+0000","last_active":"2024-03-06T07:22:23.007761+0000","last_peered":"2024-03-06T07:22:23.007761+0000","last_clean":"2024-03-06T07:22:23.007761+0000","last_became_active":"2024-03-06T07:22:21.719590+0000","last_became_peered":"2024-03-06T07:22:21.719590+0000","last_unstale":"2024-03-06T07:22:23.007761+0000","last_undegraded":"2024-03-06T07:22:23.007761+0000","last_fullsized":"2024-03-06T07:22:23.007761+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:02.330255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006573+0000","last_change":"2024-03-06T07:22:06.314914+0000","last_active":"2024-03-06T07:22:23.006573+0000","last_peered":"2024-03-06T07:22:23.006573+0000","last_clean":"2024-03-06T07:22:23.006573+0000","last_became_active":"2024-03-06T07:22:06.314654+0000","last_became_peered":"2024-03-06T07:22:06.314654+0000","last_unstale":"2024-03-06T07:22:23.006573+0000","last_undegraded":"2024-03-06T07:22:23.006573+0000","last_fullsized":"2024-03-06T07:22:23.006573+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:18.848063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706093+0000","last_change":"2024-03-06T07:22:21.721024+0000","last_active":"2024-03-06T07:22:22.706093+0000","last_peered":"2024-03-06T07:22:22.706093+0000","last_clean":"2024-03-06T07:22:22.706093+0000","last_became_active":"2024-03-06T07:22:21.720521+0000","last_became_peered":"2024-03-06T07:22:21.720521+0000","last_unstale":"2024-03-06T07:22:22.706093+0000","last_undegraded":"2024-03-06T07:22:22.706093+0000","last_fullsized":"2024-03-06T07:22:22.706093+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:32.095545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007773+0000","last_change":"2024-03-06T07:22:06.323974+0000","last_active":"2024-03-06T07:22:23.007773+0000","last_peered":"2024-03-06T07:22:23.007773+0000","last_clean":"2024-03-06T07:22:23.007773+0000","last_became_active":"2024-03-06T07:22:06.323718+0000","last_became_peered":"2024-03-06T07:22:06.323718+0000","last_unstale":"2024-03-06T07:22:23.007773+0000","last_undegraded":"2024-03-06T07:22:23.007773+0000","last_fullsized":"2024-03-06T07:22:23.007773+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:46.981658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705796+0000","last_change":"2024-03-06T07:22:21.714896+0000","last_active":"2024-03-06T07:22:22.705796+0000","last_peered":"2024-03-06T07:22:22.705796+0000","last_clean":"2024-03-06T07:22:22.705796+0000","last_became_active":"2024-03-06T07:22:21.714647+0000","last_became_peered":"2024-03-06T07:22:21.714647+0000","last_unstale":"2024-03-06T07:22:22.705796+0000","last_undegraded":"2024-03-06T07:22:22.705796+0000","last_fullsized":"2024-03-06T07:22:22.705796+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:51.486696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008604+0000","last_change":"2024-03-06T07:22:06.325388+0000","last_active":"2024-03-06T07:22:23.008604+0000","last_peered":"2024-03-06T07:22:23.008604+0000","last_clean":"2024-03-06T07:22:23.008604+0000","last_became_active":"2024-03-06T07:22:06.325109+0000","last_became_peered":"2024-03-06T07:22:06.325109+0000","last_unstale":"2024-03-06T07:22:23.008604+0000","last_undegraded":"2024-03-06T07:22:23.008604+0000","last_fullsized":"2024-03-06T07:22:23.008604+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:51.508838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007352+0000","last_change":"2024-03-06T07:22:21.724840+0000","last_active":"2024-03-06T07:22:23.007352+0000","last_peered":"2024-03-06T07:22:23.007352+0000","last_clean":"2024-03-06T07:22:23.007352+0000","last_became_active":"2024-03-06T07:22:21.724372+0000","last_became_peered":"2024-03-06T07:22:21.724372+0000","last_unstale":"2024-03-06T07:22:23.007352+0000","last_undegraded":"2024-03-06T07:22:23.007352+0000","last_fullsized":"2024-03-06T07:22:23.007352+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:30.701781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007605+0000","last_change":"2024-03-06T07:22:06.329771+0000","last_active":"2024-03-06T07:22:23.007605+0000","last_peered":"2024-03-06T07:22:23.007605+0000","last_clean":"2024-03-06T07:22:23.007605+0000","last_became_active":"2024-03-06T07:22:06.318483+0000","last_became_peered":"2024-03-06T07:22:06.318483+0000","last_unstale":"2024-03-06T07:22:23.007605+0000","last_undegraded":"2024-03-06T07:22:23.007605+0000","last_fullsized":"2024-03-06T07:22:23.007605+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:17.568813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008197+0000","last_change":"2024-03-06T07:22:21.716259+0000","last_active":"2024-03-06T07:22:23.008197+0000","last_peered":"2024-03-06T07:22:23.008197+0000","last_clean":"2024-03-06T07:22:23.008197+0000","last_became_active":"2024-03-06T07:22:21.715977+0000","last_became_peered":"2024-03-06T07:22:21.715977+0000","last_unstale":"2024-03-06T07:22:23.008197+0000","last_undegraded":"2024-03-06T07:22:23.008197+0000","last_fullsized":"2024-03-06T07:22:23.008197+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:25.216603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705889+0000","last_change":"2024-03-06T07:22:06.317746+0000","last_active":"2024-03-06T07:22:22.705889+0000","last_peered":"2024-03-06T07:22:22.705889+0000","last_clean":"2024-03-06T07:22:22.705889+0000","last_became_active":"2024-03-06T07:22:06.317465+0000","last_became_peered":"2024-03-06T07:22:06.317465+0000","last_unstale":"2024-03-06T07:22:22.705889+0000","last_undegraded":"2024-03-06T07:22:22.705889+0000","last_fullsized":"2024-03-06T07:22:22.705889+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:07.308296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705635+0000","last_change":"2024-03-06T07:22:21.724121+0000","last_active":"2024-03-06T07:22:22.705635+0000","last_peered":"2024-03-06T07:22:22.705635+0000","last_clean":"2024-03-06T07:22:22.705635+0000","last_became_active":"2024-03-06T07:22:21.723843+0000","last_became_peered":"2024-03-06T07:22:21.723843+0000","last_unstale":"2024-03-06T07:22:22.705635+0000","last_undegraded":"2024-03-06T07:22:22.705635+0000","last_fullsized":"2024-03-06T07:22:22.705635+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:19.770698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705935+0000","last_change":"2024-03-06T07:22:06.322670+0000","last_active":"2024-03-06T07:22:22.705935+0000","last_peered":"2024-03-06T07:22:22.705935+0000","last_clean":"2024-03-06T07:22:22.705935+0000","last_became_active":"2024-03-06T07:22:06.322382+0000","last_became_peered":"2024-03-06T07:22:06.322382+0000","last_unstale":"2024-03-06T07:22:22.705935+0000","last_undegraded":"2024-03-06T07:22:22.705935+0000","last_fullsized":"2024-03-06T07:22:22.705935+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:02.940410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007334+0000","last_change":"2024-03-06T07:22:21.719094+0000","last_active":"2024-03-06T07:22:23.007334+0000","last_peered":"2024-03-06T07:22:23.007334+0000","last_clean":"2024-03-06T07:22:23.007334+0000","last_became_active":"2024-03-06T07:22:21.718802+0000","last_became_peered":"2024-03-06T07:22:21.718802+0000","last_unstale":"2024-03-06T07:22:23.007334+0000","last_undegraded":"2024-03-06T07:22:23.007334+0000","last_fullsized":"2024-03-06T07:22:23.007334+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:11.190593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705581+0000","last_change":"2024-03-06T07:22:06.331711+0000","last_active":"2024-03-06T07:22:22.705581+0000","last_peered":"2024-03-06T07:22:22.705581+0000","last_clean":"2024-03-06T07:22:22.705581+0000","last_became_active":"2024-03-06T07:22:06.326618+0000","last_became_peered":"2024-03-06T07:22:06.326618+0000","last_unstale":"2024-03-06T07:22:22.705581+0000","last_undegraded":"2024-03-06T07:22:22.705581+0000","last_fullsized":"2024-03-06T07:22:22.705581+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:15.334381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719743+0000","last_change":"2024-03-06T07:22:21.724603+0000","last_active":"2024-03-06T07:22:22.719743+0000","last_peered":"2024-03-06T07:22:22.719743+0000","last_clean":"2024-03-06T07:22:22.719743+0000","last_became_active":"2024-03-06T07:22:21.724259+0000","last_became_peered":"2024-03-06T07:22:21.724259+0000","last_unstale":"2024-03-06T07:22:22.719743+0000","last_undegraded":"2024-03-06T07:22:22.719743+0000","last_fullsized":"2024-03-06T07:22:22.719743+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:17.698053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.704202+0000","last_change":"2024-03-06T07:22:06.318320+0000","last_active":"2024-03-06T07:22:22.704202+0000","last_peered":"2024-03-06T07:22:22.704202+0000","last_clean":"2024-03-06T07:22:22.704202+0000","last_became_active":"2024-03-06T07:22:06.318065+0000","last_became_peered":"2024-03-06T07:22:06.318065+0000","last_unstale":"2024-03-06T07:22:22.704202+0000","last_undegraded":"2024-03-06T07:22:22.704202+0000","last_fullsized":"2024-03-06T07:22:22.704202+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:00.505615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705681+0000","last_change":"2024-03-06T07:22:21.719457+0000","last_active":"2024-03-06T07:22:22.705681+0000","last_peered":"2024-03-06T07:22:22.705681+0000","last_clean":"2024-03-06T07:22:22.705681+0000","last_became_active":"2024-03-06T07:22:21.719177+0000","last_became_peered":"2024-03-06T07:22:21.719177+0000","last_unstale":"2024-03-06T07:22:22.705681+0000","last_undegraded":"2024-03-06T07:22:22.705681+0000","last_fullsized":"2024-03-06T07:22:22.705681+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:54.543771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006682+0000","last_change":"2024-03-06T07:22:06.318170+0000","last_active":"2024-03-06T07:22:23.006682+0000","last_peered":"2024-03-06T07:22:23.006682+0000","last_clean":"2024-03-06T07:22:23.006682+0000","last_became_active":"2024-03-06T07:22:06.317903+0000","last_became_peered":"2024-03-06T07:22:06.317903+0000","last_unstale":"2024-03-06T07:22:23.006682+0000","last_undegraded":"2024-03-06T07:22:23.006682+0000","last_fullsized":"2024-03-06T07:22:23.006682+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:25:11.151255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007891+0000","last_change":"2024-03-06T07:22:21.722788+0000","last_active":"2024-03-06T07:22:23.007891+0000","last_peered":"2024-03-06T07:22:23.007891+0000","last_clean":"2024-03-06T07:22:23.007891+0000","last_became_active":"2024-03-06T07:22:21.722511+0000","last_became_peered":"2024-03-06T07:22:21.722511+0000","last_unstale":"2024-03-06T07:22:23.007891+0000","last_undegraded":"2024-03-06T07:22:23.007891+0000","last_fullsized":"2024-03-06T07:22:23.007891+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:30.603319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007863+0000","last_change":"2024-03-06T07:22:06.333955+0000","last_active":"2024-03-06T07:22:23.007863+0000","last_peered":"2024-03-06T07:22:23.007863+0000","last_clean":"2024-03-06T07:22:23.007863+0000","last_became_active":"2024-03-06T07:22:06.333703+0000","last_became_peered":"2024-03-06T07:22:06.333703+0000","last_unstale":"2024-03-06T07:22:23.007863+0000","last_undegraded":"2024-03-06T07:22:23.007863+0000","last_fullsized":"2024-03-06T07:22:23.007863+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:34.670786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006861+0000","last_change":"2024-03-06T07:22:21.714703+0000","last_active":"2024-03-06T07:22:23.006861+0000","last_peered":"2024-03-06T07:22:23.006861+0000","last_clean":"2024-03-06T07:22:23.006861+0000","last_became_active":"2024-03-06T07:22:21.714435+0000","last_became_peered":"2024-03-06T07:22:21.714435+0000","last_unstale":"2024-03-06T07:22:23.006861+0000","last_undegraded":"2024-03-06T07:22:23.006861+0000","last_fullsized":"2024-03-06T07:22:23.006861+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:03.612944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705558+0000","last_change":"2024-03-06T07:22:06.320349+0000","last_active":"2024-03-06T07:22:22.705558+0000","last_peered":"2024-03-06T07:22:22.705558+0000","last_clean":"2024-03-06T07:22:22.705558+0000","last_became_active":"2024-03-06T07:22:06.320094+0000","last_became_peered":"2024-03-06T07:22:06.320094+0000","last_unstale":"2024-03-06T07:22:22.705558+0000","last_undegraded":"2024-03-06T07:22:22.705558+0000","last_fullsized":"2024-03-06T07:22:22.705558+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:48.224301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719832+0000","last_change":"2024-03-06T07:22:21.720164+0000","last_active":"2024-03-06T07:22:22.719832+0000","last_peered":"2024-03-06T07:22:22.719832+0000","last_clean":"2024-03-06T07:22:22.719832+0000","last_became_active":"2024-03-06T07:22:21.719753+0000","last_became_peered":"2024-03-06T07:22:21.719753+0000","last_unstale":"2024-03-06T07:22:22.719832+0000","last_undegraded":"2024-03-06T07:22:22.719832+0000","last_fullsized":"2024-03-06T07:22:22.719832+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:30.960027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008240+0000","last_change":"2024-03-06T07:22:06.319987+0000","last_active":"2024-03-06T07:22:23.008240+0000","last_peered":"2024-03-06T07:22:23.008240+0000","last_clean":"2024-03-06T07:22:23.008240+0000","last_became_active":"2024-03-06T07:22:06.319703+0000","last_became_peered":"2024-03-06T07:22:06.319703+0000","last_unstale":"2024-03-06T07:22:23.008240+0000","last_undegraded":"2024-03-06T07:22:23.008240+0000","last_fullsized":"2024-03-06T07:22:23.008240+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:02.558815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719894+0000","last_change":"2024-03-06T07:22:21.716584+0000","last_active":"2024-03-06T07:22:22.719894+0000","last_peered":"2024-03-06T07:22:22.719894+0000","last_clean":"2024-03-06T07:22:22.719894+0000","last_became_active":"2024-03-06T07:22:21.716250+0000","last_became_peered":"2024-03-06T07:22:21.716250+0000","last_unstale":"2024-03-06T07:22:22.719894+0000","last_undegraded":"2024-03-06T07:22:22.719894+0000","last_fullsized":"2024-03-06T07:22:22.719894+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:53.042875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705929+0000","last_change":"2024-03-06T07:22:06.318435+0000","last_active":"2024-03-06T07:22:22.705929+0000","last_peered":"2024-03-06T07:22:22.705929+0000","last_clean":"2024-03-06T07:22:22.705929+0000","last_became_active":"2024-03-06T07:22:06.318147+0000","last_became_peered":"2024-03-06T07:22:06.318147+0000","last_unstale":"2024-03-06T07:22:22.705929+0000","last_undegraded":"2024-03-06T07:22:22.705929+0000","last_fullsized":"2024-03-06T07:22:22.705929+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:20.858614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007096+0000","last_change":"2024-03-06T07:22:21.725146+0000","last_active":"2024-03-06T07:22:23.007096+0000","last_peered":"2024-03-06T07:22:23.007096+0000","last_clean":"2024-03-06T07:22:23.007096+0000","last_became_active":"2024-03-06T07:22:21.723713+0000","last_became_peered":"2024-03-06T07:22:21.723713+0000","last_unstale":"2024-03-06T07:22:23.007096+0000","last_undegraded":"2024-03-06T07:22:23.007096+0000","last_fullsized":"2024-03-06T07:22:23.007096+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:34.109293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006649+0000","last_change":"2024-03-06T07:22:06.335154+0000","last_active":"2024-03-06T07:22:23.006649+0000","last_peered":"2024-03-06T07:22:23.006649+0000","last_clean":"2024-03-06T07:22:23.006649+0000","last_became_active":"2024-03-06T07:22:06.334855+0000","last_became_peered":"2024-03-06T07:22:06.334855+0000","last_unstale":"2024-03-06T07:22:23.006649+0000","last_undegraded":"2024-03-06T07:22:23.006649+0000","last_fullsized":"2024-03-06T07:22:23.006649+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:54.652557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006572+0000","last_change":"2024-03-06T07:22:21.716747+0000","last_active":"2024-03-06T07:22:23.006572+0000","last_peered":"2024-03-06T07:22:23.006572+0000","last_clean":"2024-03-06T07:22:23.006572+0000","last_became_active":"2024-03-06T07:22:21.716102+0000","last_became_peered":"2024-03-06T07:22:21.716102+0000","last_unstale":"2024-03-06T07:22:23.006572+0000","last_undegraded":"2024-03-06T07:22:23.006572+0000","last_fullsized":"2024-03-06T07:22:23.006572+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:05.317140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007948+0000","last_change":"2024-03-06T07:22:06.330141+0000","last_active":"2024-03-06T07:22:23.007948+0000","last_peered":"2024-03-06T07:22:23.007948+0000","last_clean":"2024-03-06T07:22:23.007948+0000","last_became_active":"2024-03-06T07:22:06.329487+0000","last_became_peered":"2024-03-06T07:22:06.329487+0000","last_unstale":"2024-03-06T07:22:23.007948+0000","last_undegraded":"2024-03-06T07:22:23.007948+0000","last_fullsized":"2024-03-06T07:22:23.007948+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:09.367201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007606+0000","last_change":"2024-03-06T07:22:21.720384+0000","last_active":"2024-03-06T07:22:23.007606+0000","last_peered":"2024-03-06T07:22:23.007606+0000","last_clean":"2024-03-06T07:22:23.007606+0000","last_became_active":"2024-03-06T07:22:21.719261+0000","last_became_peered":"2024-03-06T07:22:21.719261+0000","last_unstale":"2024-03-06T07:22:23.007606+0000","last_undegraded":"2024-03-06T07:22:23.007606+0000","last_fullsized":"2024-03-06T07:22:23.007606+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:17.353502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007135+0000","last_change":"2024-03-06T07:22:06.322218+0000","last_active":"2024-03-06T07:22:23.007135+0000","last_peered":"2024-03-06T07:22:23.007135+0000","last_clean":"2024-03-06T07:22:23.007135+0000","last_became_active":"2024-03-06T07:22:06.318026+0000","last_became_peered":"2024-03-06T07:22:06.318026+0000","last_unstale":"2024-03-06T07:22:23.007135+0000","last_undegraded":"2024-03-06T07:22:23.007135+0000","last_fullsized":"2024-03-06T07:22:23.007135+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:16.809640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719387+0000","last_change":"2024-03-06T07:22:21.729828+0000","last_active":"2024-03-06T07:22:22.719387+0000","last_peered":"2024-03-06T07:22:22.719387+0000","last_clean":"2024-03-06T07:22:22.719387+0000","last_became_active":"2024-03-06T07:22:21.729562+0000","last_became_peered":"2024-03-06T07:22:21.729562+0000","last_unstale":"2024-03-06T07:22:22.719387+0000","last_undegraded":"2024-03-06T07:22:22.719387+0000","last_fullsized":"2024-03-06T07:22:22.719387+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:02.330255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719197+0000","last_change":"2024-03-06T07:22:06.322067+0000","last_active":"2024-03-06T07:22:22.719197+0000","last_peered":"2024-03-06T07:22:22.719197+0000","last_clean":"2024-03-06T07:22:22.719197+0000","last_became_active":"2024-03-06T07:22:06.321797+0000","last_became_peered":"2024-03-06T07:22:06.321797+0000","last_unstale":"2024-03-06T07:22:22.719197+0000","last_undegraded":"2024-03-06T07:22:22.719197+0000","last_fullsized":"2024-03-06T07:22:22.719197+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:36.139439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705442+0000","last_change":"2024-03-06T07:22:21.720183+0000","last_active":"2024-03-06T07:22:22.705442+0000","last_peered":"2024-03-06T07:22:22.705442+0000","last_clean":"2024-03-06T07:22:22.705442+0000","last_became_active":"2024-03-06T07:22:21.719913+0000","last_became_peered":"2024-03-06T07:22:21.719913+0000","last_unstale":"2024-03-06T07:22:22.705442+0000","last_undegraded":"2024-03-06T07:22:22.705442+0000","last_fullsized":"2024-03-06T07:22:22.705442+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:25:26.539898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007467+0000","last_change":"2024-03-06T07:22:20.702488+0000","last_active":"2024-03-06T07:22:23.007467+0000","last_peered":"2024-03-06T07:22:23.007467+0000","last_clean":"2024-03-06T07:22:23.007467+0000","last_became_active":"2024-03-06T07:22:20.702195+0000","last_became_peered":"2024-03-06T07:22:20.702195+0000","last_unstale":"2024-03-06T07:22:23.007467+0000","last_undegraded":"2024-03-06T07:22:23.007467+0000","last_fullsized":"2024-03-06T07:22:23.007467+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:48.668680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007341+0000","last_change":"2024-03-06T07:22:06.320372+0000","last_active":"2024-03-06T07:22:23.007341+0000","last_peered":"2024-03-06T07:22:23.007341+0000","last_clean":"2024-03-06T07:22:23.007341+0000","last_became_active":"2024-03-06T07:22:06.320109+0000","last_became_peered":"2024-03-06T07:22:06.320109+0000","last_unstale":"2024-03-06T07:22:23.007341+0000","last_undegraded":"2024-03-06T07:22:23.007341+0000","last_fullsized":"2024-03-06T07:22:23.007341+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:37.954594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":26,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706128+0000","last_change":"2024-03-06T07:22:05.317240+0000","last_active":"2024-03-06T07:22:22.706128+0000","last_peered":"2024-03-06T07:22:22.706128+0000","last_clean":"2024-03-06T07:22:22.706128+0000","last_became_active":"2024-03-06T07:22:05.316908+0000","last_became_peered":"2024-03-06T07:22:05.316908+0000","last_unstale":"2024-03-06T07:22:22.706128+0000","last_undegraded":"2024-03-06T07:22:22.706128+0000","last_fullsized":"2024-03-06T07:22:22.706128+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:18.512707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007611+0000","last_change":"2024-03-06T07:22:21.714837+0000","last_active":"2024-03-06T07:22:23.007611+0000","last_peered":"2024-03-06T07:22:23.007611+0000","last_clean":"2024-03-06T07:22:23.007611+0000","last_became_active":"2024-03-06T07:22:21.714547+0000","last_became_peered":"2024-03-06T07:22:21.714547+0000","last_unstale":"2024-03-06T07:22:23.007611+0000","last_undegraded":"2024-03-06T07:22:23.007611+0000","last_fullsized":"2024-03-06T07:22:23.007611+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:52.725112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"55'8","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006888+0000","last_change":"2024-03-06T07:22:20.708393+0000","last_active":"2024-03-06T07:22:23.006888+0000","last_peered":"2024-03-06T07:22:23.006888+0000","last_clean":"2024-03-06T07:22:23.006888+0000","last_became_active":"2024-03-06T07:22:20.708115+0000","last_became_peered":"2024-03-06T07:22:20.708115+0000","last_unstale":"2024-03-06T07:22:23.006888+0000","last_undegraded":"2024-03-06T07:22:23.006888+0000","last_fullsized":"2024-03-06T07:22:23.006888+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:05.892802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719439+0000","last_change":"2024-03-06T07:22:06.324306+0000","last_active":"2024-03-06T07:22:22.719439+0000","last_peered":"2024-03-06T07:22:22.719439+0000","last_clean":"2024-03-06T07:22:22.719439+0000","last_became_active":"2024-03-06T07:22:06.323979+0000","last_became_peered":"2024-03-06T07:22:06.323979+0000","last_unstale":"2024-03-06T07:22:22.719439+0000","last_undegraded":"2024-03-06T07:22:22.719439+0000","last_fullsized":"2024-03-06T07:22:22.719439+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:05.731939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007869+0000","last_change":"2024-03-06T07:22:05.321228+0000","last_active":"2024-03-06T07:22:23.007869+0000","last_peered":"2024-03-06T07:22:23.007869+0000","last_clean":"2024-03-06T07:22:23.007869+0000","last_became_active":"2024-03-06T07:22:05.320958+0000","last_became_peered":"2024-03-06T07:22:05.320958+0000","last_unstale":"2024-03-06T07:22:23.007869+0000","last_undegraded":"2024-03-06T07:22:23.007869+0000","last_fullsized":"2024-03-06T07:22:23.007869+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:43.288752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007035+0000","last_change":"2024-03-06T07:22:21.723577+0000","last_active":"2024-03-06T07:22:23.007035+0000","last_peered":"2024-03-06T07:22:23.007035+0000","last_clean":"2024-03-06T07:22:23.007035+0000","last_became_active":"2024-03-06T07:22:21.723299+0000","last_became_peered":"2024-03-06T07:22:21.723299+0000","last_unstale":"2024-03-06T07:22:23.007035+0000","last_undegraded":"2024-03-06T07:22:23.007035+0000","last_fullsized":"2024-03-06T07:22:23.007035+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:32.957457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008792+0000","last_change":"2024-03-06T07:22:20.709772+0000","last_active":"2024-03-06T07:22:23.008792+0000","last_peered":"2024-03-06T07:22:23.008792+0000","last_clean":"2024-03-06T07:22:23.008792+0000","last_became_active":"2024-03-06T07:22:20.709341+0000","last_became_peered":"2024-03-06T07:22:20.709341+0000","last_unstale":"2024-03-06T07:22:23.008792+0000","last_undegraded":"2024-03-06T07:22:23.008792+0000","last_fullsized":"2024-03-06T07:22:23.008792+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:29.598640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705539+0000","last_change":"2024-03-06T07:22:06.323135+0000","last_active":"2024-03-06T07:22:22.705539+0000","last_peered":"2024-03-06T07:22:22.705539+0000","last_clean":"2024-03-06T07:22:22.705539+0000","last_became_active":"2024-03-06T07:22:06.322873+0000","last_became_peered":"2024-03-06T07:22:06.322873+0000","last_unstale":"2024-03-06T07:22:22.705539+0000","last_undegraded":"2024-03-06T07:22:22.705539+0000","last_fullsized":"2024-03-06T07:22:22.705539+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:32.712920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008278+0000","last_change":"2024-03-06T07:22:05.321069+0000","last_active":"2024-03-06T07:22:23.008278+0000","last_peered":"2024-03-06T07:22:23.008278+0000","last_clean":"2024-03-06T07:22:23.008278+0000","last_became_active":"2024-03-06T07:22:05.320722+0000","last_became_peered":"2024-03-06T07:22:05.320722+0000","last_unstale":"2024-03-06T07:22:23.008278+0000","last_undegraded":"2024-03-06T07:22:23.008278+0000","last_fullsized":"2024-03-06T07:22:23.008278+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:26.446709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719258+0000","last_change":"2024-03-06T07:22:21.720273+0000","last_active":"2024-03-06T07:22:22.719258+0000","last_peered":"2024-03-06T07:22:22.719258+0000","last_clean":"2024-03-06T07:22:22.719258+0000","last_became_active":"2024-03-06T07:22:21.719605+0000","last_became_peered":"2024-03-06T07:22:21.719605+0000","last_unstale":"2024-03-06T07:22:22.719258+0000","last_undegraded":"2024-03-06T07:22:22.719258+0000","last_fullsized":"2024-03-06T07:22:22.719258+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:17.353502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719195+0000","last_change":"2024-03-06T07:22:20.705361+0000","last_active":"2024-03-06T07:22:22.719195+0000","last_peered":"2024-03-06T07:22:22.719195+0000","last_clean":"2024-03-06T07:22:22.719195+0000","last_became_active":"2024-03-06T07:22:20.705057+0000","last_became_peered":"2024-03-06T07:22:20.705057+0000","last_unstale":"2024-03-06T07:22:22.719195+0000","last_undegraded":"2024-03-06T07:22:22.719195+0000","last_fullsized":"2024-03-06T07:22:22.719195+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:29.598640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705979+0000","last_change":"2024-03-06T07:22:06.318876+0000","last_active":"2024-03-06T07:22:22.705979+0000","last_peered":"2024-03-06T07:22:22.705979+0000","last_clean":"2024-03-06T07:22:22.705979+0000","last_became_active":"2024-03-06T07:22:06.318574+0000","last_became_peered":"2024-03-06T07:22:06.318574+0000","last_unstale":"2024-03-06T07:22:22.705979+0000","last_undegraded":"2024-03-06T07:22:22.705979+0000","last_fullsized":"2024-03-06T07:22:22.705979+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:34.284715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719634+0000","last_change":"2024-03-06T07:22:05.316288+0000","last_active":"2024-03-06T07:22:22.719634+0000","last_peered":"2024-03-06T07:22:22.719634+0000","last_clean":"2024-03-06T07:22:22.719634+0000","last_became_active":"2024-03-06T07:22:05.315965+0000","last_became_peered":"2024-03-06T07:22:05.315965+0000","last_unstale":"2024-03-06T07:22:22.719634+0000","last_undegraded":"2024-03-06T07:22:22.719634+0000","last_fullsized":"2024-03-06T07:22:22.719634+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:19.545421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706511+0000","last_change":"2024-03-06T07:22:21.726430+0000","last_active":"2024-03-06T07:22:22.706511+0000","last_peered":"2024-03-06T07:22:22.706511+0000","last_clean":"2024-03-06T07:22:22.706511+0000","last_became_active":"2024-03-06T07:22:21.726174+0000","last_became_peered":"2024-03-06T07:22:21.726174+0000","last_unstale":"2024-03-06T07:22:22.706511+0000","last_undegraded":"2024-03-06T07:22:22.706511+0000","last_fullsized":"2024-03-06T07:22:22.706511+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:30.723025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"55'3","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008508+0000","last_change":"2024-03-06T07:22:20.706738+0000","last_active":"2024-03-06T07:22:23.008508+0000","last_peered":"2024-03-06T07:22:23.008508+0000","last_clean":"2024-03-06T07:22:23.008508+0000","last_became_active":"2024-03-06T07:22:20.706116+0000","last_became_peered":"2024-03-06T07:22:20.706116+0000","last_unstale":"2024-03-06T07:22:23.008508+0000","last_undegraded":"2024-03-06T07:22:23.008508+0000","last_fullsized":"2024-03-06T07:22:23.008508+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:49.054750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008930+0000","last_change":"2024-03-06T07:22:06.319384+0000","last_active":"2024-03-06T07:22:23.008930+0000","last_peered":"2024-03-06T07:22:23.008930+0000","last_clean":"2024-03-06T07:22:23.008930+0000","last_became_active":"2024-03-06T07:22:06.318885+0000","last_became_peered":"2024-03-06T07:22:06.318885+0000","last_unstale":"2024-03-06T07:22:23.008930+0000","last_undegraded":"2024-03-06T07:22:23.008930+0000","last_fullsized":"2024-03-06T07:22:23.008930+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:05.680579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006890+0000","last_change":"2024-03-06T07:22:05.319423+0000","last_active":"2024-03-06T07:22:23.006890+0000","last_peered":"2024-03-06T07:22:23.006890+0000","last_clean":"2024-03-06T07:22:23.006890+0000","last_became_active":"2024-03-06T07:22:05.319055+0000","last_became_peered":"2024-03-06T07:22:05.319055+0000","last_unstale":"2024-03-06T07:22:23.006890+0000","last_undegraded":"2024-03-06T07:22:23.006890+0000","last_fullsized":"2024-03-06T07:22:23.006890+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:19.853691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008178+0000","last_change":"2024-03-06T07:22:21.723296+0000","last_active":"2024-03-06T07:22:23.008178+0000","last_peered":"2024-03-06T07:22:23.008178+0000","last_clean":"2024-03-06T07:22:23.008178+0000","last_became_active":"2024-03-06T07:22:21.722406+0000","last_became_peered":"2024-03-06T07:22:21.722406+0000","last_unstale":"2024-03-06T07:22:23.008178+0000","last_undegraded":"2024-03-06T07:22:23.008178+0000","last_fullsized":"2024-03-06T07:22:23.008178+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:49.673358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"55'6","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007117+0000","last_change":"2024-03-06T07:22:20.710366+0000","last_active":"2024-03-06T07:22:23.007117+0000","last_peered":"2024-03-06T07:22:23.007117+0000","last_clean":"2024-03-06T07:22:23.007117+0000","last_became_active":"2024-03-06T07:22:20.709863+0000","last_became_peered":"2024-03-06T07:22:20.709863+0000","last_unstale":"2024-03-06T07:22:23.007117+0000","last_undegraded":"2024-03-06T07:22:23.007117+0000","last_fullsized":"2024-03-06T07:22:23.007117+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:31.090866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008053+0000","last_change":"2024-03-06T07:22:06.330770+0000","last_active":"2024-03-06T07:22:23.008053+0000","last_peered":"2024-03-06T07:22:23.008053+0000","last_clean":"2024-03-06T07:22:23.008053+0000","last_became_active":"2024-03-06T07:22:06.330404+0000","last_became_peered":"2024-03-06T07:22:06.330404+0000","last_unstale":"2024-03-06T07:22:23.008053+0000","last_undegraded":"2024-03-06T07:22:23.008053+0000","last_fullsized":"2024-03-06T07:22:23.008053+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:36.139439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007658+0000","last_change":"2024-03-06T07:22:05.313908+0000","last_active":"2024-03-06T07:22:23.007658+0000","last_peered":"2024-03-06T07:22:23.007658+0000","last_clean":"2024-03-06T07:22:23.007658+0000","last_became_active":"2024-03-06T07:22:05.313638+0000","last_became_peered":"2024-03-06T07:22:05.313638+0000","last_unstale":"2024-03-06T07:22:23.007658+0000","last_undegraded":"2024-03-06T07:22:23.007658+0000","last_fullsized":"2024-03-06T07:22:23.007658+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:46.701129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006995+0000","last_change":"2024-03-06T07:22:21.713645+0000","last_active":"2024-03-06T07:22:23.006995+0000","last_peered":"2024-03-06T07:22:23.006995+0000","last_clean":"2024-03-06T07:22:23.006995+0000","last_became_active":"2024-03-06T07:22:21.713387+0000","last_became_peered":"2024-03-06T07:22:21.713387+0000","last_unstale":"2024-03-06T07:22:23.006995+0000","last_undegraded":"2024-03-06T07:22:23.006995+0000","last_fullsized":"2024-03-06T07:22:23.006995+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:04.969935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008879+0000","last_change":"2024-03-06T07:22:20.707352+0000","last_active":"2024-03-06T07:22:23.008879+0000","last_peered":"2024-03-06T07:22:23.008879+0000","last_clean":"2024-03-06T07:22:23.008879+0000","last_became_active":"2024-03-06T07:22:20.706288+0000","last_became_peered":"2024-03-06T07:22:20.706288+0000","last_unstale":"2024-03-06T07:22:23.008879+0000","last_undegraded":"2024-03-06T07:22:23.008879+0000","last_fullsized":"2024-03-06T07:22:23.008879+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:04.315144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705659+0000","last_change":"2024-03-06T07:22:06.320596+0000","last_active":"2024-03-06T07:22:22.705659+0000","last_peered":"2024-03-06T07:22:22.705659+0000","last_clean":"2024-03-06T07:22:22.705659+0000","last_became_active":"2024-03-06T07:22:06.319430+0000","last_became_peered":"2024-03-06T07:22:06.319430+0000","last_unstale":"2024-03-06T07:22:22.705659+0000","last_undegraded":"2024-03-06T07:22:22.705659+0000","last_fullsized":"2024-03-06T07:22:22.705659+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:49.581292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705499+0000","last_change":"2024-03-06T07:22:05.320922+0000","last_active":"2024-03-06T07:22:22.705499+0000","last_peered":"2024-03-06T07:22:22.705499+0000","last_clean":"2024-03-06T07:22:22.705499+0000","last_became_active":"2024-03-06T07:22:05.320638+0000","last_became_peered":"2024-03-06T07:22:05.320638+0000","last_unstale":"2024-03-06T07:22:22.705499+0000","last_undegraded":"2024-03-06T07:22:22.705499+0000","last_fullsized":"2024-03-06T07:22:22.705499+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:13.798736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705751+0000","last_change":"2024-03-06T07:22:21.726275+0000","last_active":"2024-03-06T07:22:22.705751+0000","last_peered":"2024-03-06T07:22:22.705751+0000","last_clean":"2024-03-06T07:22:22.705751+0000","last_became_active":"2024-03-06T07:22:21.726006+0000","last_became_peered":"2024-03-06T07:22:21.726006+0000","last_unstale":"2024-03-06T07:22:22.705751+0000","last_undegraded":"2024-03-06T07:22:22.705751+0000","last_fullsized":"2024-03-06T07:22:22.705751+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:54.164947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008025+0000","last_change":"2024-03-06T07:22:20.706922+0000","last_active":"2024-03-06T07:22:23.008025+0000","last_peered":"2024-03-06T07:22:23.008025+0000","last_clean":"2024-03-06T07:22:23.008025+0000","last_became_active":"2024-03-06T07:22:20.706248+0000","last_became_peered":"2024-03-06T07:22:20.706248+0000","last_unstale":"2024-03-06T07:22:23.008025+0000","last_undegraded":"2024-03-06T07:22:23.008025+0000","last_fullsized":"2024-03-06T07:22:23.008025+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:16.942779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.009065+0000","last_change":"2024-03-06T07:22:06.335202+0000","last_active":"2024-03-06T07:22:23.009065+0000","last_peered":"2024-03-06T07:22:23.009065+0000","last_clean":"2024-03-06T07:22:23.009065+0000","last_became_active":"2024-03-06T07:22:06.334922+0000","last_became_peered":"2024-03-06T07:22:06.334922+0000","last_unstale":"2024-03-06T07:22:23.009065+0000","last_undegraded":"2024-03-06T07:22:23.009065+0000","last_fullsized":"2024-03-06T07:22:23.009065+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:01.204740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006707+0000","last_change":"2024-03-06T07:22:05.326648+0000","last_active":"2024-03-06T07:22:23.006707+0000","last_peered":"2024-03-06T07:22:23.006707+0000","last_clean":"2024-03-06T07:22:23.006707+0000","last_became_active":"2024-03-06T07:22:05.326423+0000","last_became_peered":"2024-03-06T07:22:05.326423+0000","last_unstale":"2024-03-06T07:22:23.006707+0000","last_undegraded":"2024-03-06T07:22:23.006707+0000","last_fullsized":"2024-03-06T07:22:23.006707+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:04.675657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705762+0000","last_change":"2024-03-06T07:22:21.722486+0000","last_active":"2024-03-06T07:22:22.705762+0000","last_peered":"2024-03-06T07:22:22.705762+0000","last_clean":"2024-03-06T07:22:22.705762+0000","last_became_active":"2024-03-06T07:22:21.721809+0000","last_became_peered":"2024-03-06T07:22:21.721809+0000","last_unstale":"2024-03-06T07:22:22.705762+0000","last_undegraded":"2024-03-06T07:22:22.705762+0000","last_fullsized":"2024-03-06T07:22:22.705762+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:05.539044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705205+0000","last_change":"2024-03-06T07:22:20.709212+0000","last_active":"2024-03-06T07:22:22.705205+0000","last_peered":"2024-03-06T07:22:22.705205+0000","last_clean":"2024-03-06T07:22:22.705205+0000","last_became_active":"2024-03-06T07:22:20.708913+0000","last_became_peered":"2024-03-06T07:22:20.708913+0000","last_unstale":"2024-03-06T07:22:22.705205+0000","last_undegraded":"2024-03-06T07:22:22.705205+0000","last_fullsized":"2024-03-06T07:22:22.705205+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:52.038196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007099+0000","last_change":"2024-03-06T07:22:06.326561+0000","last_active":"2024-03-06T07:22:23.007099+0000","last_peered":"2024-03-06T07:22:23.007099+0000","last_clean":"2024-03-06T07:22:23.007099+0000","last_became_active":"2024-03-06T07:22:06.326295+0000","last_became_peered":"2024-03-06T07:22:06.326295+0000","last_unstale":"2024-03-06T07:22:23.007099+0000","last_undegraded":"2024-03-06T07:22:23.007099+0000","last_fullsized":"2024-03-06T07:22:23.007099+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:36.941825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.009020+0000","last_change":"2024-03-06T07:22:05.319695+0000","last_active":"2024-03-06T07:22:23.009020+0000","last_peered":"2024-03-06T07:22:23.009020+0000","last_clean":"2024-03-06T07:22:23.009020+0000","last_became_active":"2024-03-06T07:22:05.319410+0000","last_became_peered":"2024-03-06T07:22:05.319410+0000","last_unstale":"2024-03-06T07:22:23.009020+0000","last_undegraded":"2024-03-06T07:22:23.009020+0000","last_fullsized":"2024-03-06T07:22:23.009020+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:06.303374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705769+0000","last_change":"2024-03-06T07:22:21.725276+0000","last_active":"2024-03-06T07:22:22.705769+0000","last_peered":"2024-03-06T07:22:22.705769+0000","last_clean":"2024-03-06T07:22:22.705769+0000","last_became_active":"2024-03-06T07:22:21.725019+0000","last_became_peered":"2024-03-06T07:22:21.725019+0000","last_unstale":"2024-03-06T07:22:22.705769+0000","last_undegraded":"2024-03-06T07:22:22.705769+0000","last_fullsized":"2024-03-06T07:22:22.705769+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:04.407350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705244+0000","last_change":"2024-03-06T07:22:20.708398+0000","last_active":"2024-03-06T07:22:22.705244+0000","last_peered":"2024-03-06T07:22:22.705244+0000","last_clean":"2024-03-06T07:22:22.705244+0000","last_became_active":"2024-03-06T07:22:20.708105+0000","last_became_peered":"2024-03-06T07:22:20.708105+0000","last_unstale":"2024-03-06T07:22:22.705244+0000","last_undegraded":"2024-03-06T07:22:22.705244+0000","last_fullsized":"2024-03-06T07:22:22.705244+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:33.232028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719373+0000","last_change":"2024-03-06T07:22:06.314348+0000","last_active":"2024-03-06T07:22:22.719373+0000","last_peered":"2024-03-06T07:22:22.719373+0000","last_clean":"2024-03-06T07:22:22.719373+0000","last_became_active":"2024-03-06T07:22:06.314044+0000","last_became_peered":"2024-03-06T07:22:06.314044+0000","last_unstale":"2024-03-06T07:22:22.719373+0000","last_undegraded":"2024-03-06T07:22:22.719373+0000","last_fullsized":"2024-03-06T07:22:22.719373+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:02.558815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008323+0000","last_change":"2024-03-06T07:22:05.321049+0000","last_active":"2024-03-06T07:22:23.008323+0000","last_peered":"2024-03-06T07:22:23.008323+0000","last_clean":"2024-03-06T07:22:23.008323+0000","last_became_active":"2024-03-06T07:22:05.320696+0000","last_became_peered":"2024-03-06T07:22:05.320696+0000","last_unstale":"2024-03-06T07:22:23.008323+0000","last_undegraded":"2024-03-06T07:22:23.008323+0000","last_fullsized":"2024-03-06T07:22:23.008323+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:26.446709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.703804+0000","last_change":"2024-03-06T07:22:21.713747+0000","last_active":"2024-03-06T07:22:22.703804+0000","last_peered":"2024-03-06T07:22:22.703804+0000","last_clean":"2024-03-06T07:22:22.703804+0000","last_became_active":"2024-03-06T07:22:21.713490+0000","last_became_peered":"2024-03-06T07:22:21.713490+0000","last_unstale":"2024-03-06T07:22:22.703804+0000","last_undegraded":"2024-03-06T07:22:22.703804+0000","last_fullsized":"2024-03-06T07:22:22.703804+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:30.192301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705353+0000","last_change":"2024-03-06T07:22:20.702222+0000","last_active":"2024-03-06T07:22:22.705353+0000","last_peered":"2024-03-06T07:22:22.705353+0000","last_clean":"2024-03-06T07:22:22.705353+0000","last_became_active":"2024-03-06T07:22:20.701653+0000","last_became_peered":"2024-03-06T07:22:20.701653+0000","last_unstale":"2024-03-06T07:22:22.705353+0000","last_undegraded":"2024-03-06T07:22:22.705353+0000","last_fullsized":"2024-03-06T07:22:22.705353+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:04.312461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007570+0000","last_change":"2024-03-06T07:22:06.330920+0000","last_active":"2024-03-06T07:22:23.007570+0000","last_peered":"2024-03-06T07:22:23.007570+0000","last_clean":"2024-03-06T07:22:23.007570+0000","last_became_active":"2024-03-06T07:22:06.330331+0000","last_became_peered":"2024-03-06T07:22:06.330331+0000","last_unstale":"2024-03-06T07:22:23.007570+0000","last_undegraded":"2024-03-06T07:22:23.007570+0000","last_fullsized":"2024-03-06T07:22:23.007570+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:18.720650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":25,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008329+0000","last_change":"2024-03-06T07:22:05.321995+0000","last_active":"2024-03-06T07:22:23.008329+0000","last_peered":"2024-03-06T07:22:23.008329+0000","last_clean":"2024-03-06T07:22:23.008329+0000","last_became_active":"2024-03-06T07:22:05.321734+0000","last_became_peered":"2024-03-06T07:22:05.321734+0000","last_unstale":"2024-03-06T07:22:23.008329+0000","last_undegraded":"2024-03-06T07:22:23.008329+0000","last_fullsized":"2024-03-06T07:22:23.008329+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:40.725817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006969+0000","last_change":"2024-03-06T07:22:21.723978+0000","last_active":"2024-03-06T07:22:23.006969+0000","last_peered":"2024-03-06T07:22:23.006969+0000","last_clean":"2024-03-06T07:22:23.006969+0000","last_became_active":"2024-03-06T07:22:21.723647+0000","last_became_peered":"2024-03-06T07:22:21.723647+0000","last_unstale":"2024-03-06T07:22:23.006969+0000","last_undegraded":"2024-03-06T07:22:23.006969+0000","last_fullsized":"2024-03-06T07:22:23.006969+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:34.791590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705387+0000","last_change":"2024-03-06T07:22:20.702276+0000","last_active":"2024-03-06T07:22:22.705387+0000","last_peered":"2024-03-06T07:22:22.705387+0000","last_clean":"2024-03-06T07:22:22.705387+0000","last_became_active":"2024-03-06T07:22:20.701967+0000","last_became_peered":"2024-03-06T07:22:20.701967+0000","last_unstale":"2024-03-06T07:22:22.705387+0000","last_undegraded":"2024-03-06T07:22:22.705387+0000","last_fullsized":"2024-03-06T07:22:22.705387+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:16.693374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008489+0000","last_change":"2024-03-06T07:22:06.308003+0000","last_active":"2024-03-06T07:22:23.008489+0000","last_peered":"2024-03-06T07:22:23.008489+0000","last_clean":"2024-03-06T07:22:23.008489+0000","last_became_active":"2024-03-06T07:22:06.307696+0000","last_became_peered":"2024-03-06T07:22:06.307696+0000","last_unstale":"2024-03-06T07:22:23.008489+0000","last_undegraded":"2024-03-06T07:22:23.008489+0000","last_fullsized":"2024-03-06T07:22:23.008489+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:20.550343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":28,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705948+0000","last_change":"2024-03-06T07:22:05.313388+0000","last_active":"2024-03-06T07:22:22.705948+0000","last_peered":"2024-03-06T07:22:22.705948+0000","last_clean":"2024-03-06T07:22:22.705948+0000","last_became_active":"2024-03-06T07:22:05.312146+0000","last_became_peered":"2024-03-06T07:22:05.312146+0000","last_unstale":"2024-03-06T07:22:22.705948+0000","last_undegraded":"2024-03-06T07:22:22.705948+0000","last_fullsized":"2024-03-06T07:22:22.705948+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:16.630462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007976+0000","last_change":"2024-03-06T07:22:21.715675+0000","last_active":"2024-03-06T07:22:23.007976+0000","last_peered":"2024-03-06T07:22:23.007976+0000","last_clean":"2024-03-06T07:22:23.007976+0000","last_became_active":"2024-03-06T07:22:21.715423+0000","last_became_peered":"2024-03-06T07:22:21.715423+0000","last_unstale":"2024-03-06T07:22:23.007976+0000","last_undegraded":"2024-03-06T07:22:23.007976+0000","last_fullsized":"2024-03-06T07:22:23.007976+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:12.856641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007292+0000","last_change":"2024-03-06T07:22:20.697808+0000","last_active":"2024-03-06T07:22:23.007292+0000","last_peered":"2024-03-06T07:22:23.007292+0000","last_clean":"2024-03-06T07:22:23.007292+0000","last_became_active":"2024-03-06T07:22:20.697175+0000","last_became_peered":"2024-03-06T07:22:20.697175+0000","last_unstale":"2024-03-06T07:22:23.007292+0000","last_undegraded":"2024-03-06T07:22:23.007292+0000","last_fullsized":"2024-03-06T07:22:23.007292+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:10.185915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719353+0000","last_change":"2024-03-06T07:22:06.324830+0000","last_active":"2024-03-06T07:22:22.719353+0000","last_peered":"2024-03-06T07:22:22.719353+0000","last_clean":"2024-03-06T07:22:22.719353+0000","last_became_active":"2024-03-06T07:22:06.324574+0000","last_became_peered":"2024-03-06T07:22:06.324574+0000","last_unstale":"2024-03-06T07:22:22.719353+0000","last_undegraded":"2024-03-06T07:22:22.719353+0000","last_fullsized":"2024-03-06T07:22:22.719353+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:22.443979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006926+0000","last_change":"2024-03-06T07:22:05.326946+0000","last_active":"2024-03-06T07:22:23.006926+0000","last_peered":"2024-03-06T07:22:23.006926+0000","last_clean":"2024-03-06T07:22:23.006926+0000","last_became_active":"2024-03-06T07:22:05.326675+0000","last_became_peered":"2024-03-06T07:22:05.326675+0000","last_unstale":"2024-03-06T07:22:23.006926+0000","last_undegraded":"2024-03-06T07:22:23.006926+0000","last_fullsized":"2024-03-06T07:22:23.006926+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:35.936903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008727+0000","last_change":"2024-03-06T07:22:21.723312+0000","last_active":"2024-03-06T07:22:23.008727+0000","last_peered":"2024-03-06T07:22:23.008727+0000","last_clean":"2024-03-06T07:22:23.008727+0000","last_became_active":"2024-03-06T07:22:21.723046+0000","last_became_peered":"2024-03-06T07:22:21.723046+0000","last_unstale":"2024-03-06T07:22:23.008727+0000","last_undegraded":"2024-03-06T07:22:23.008727+0000","last_fullsized":"2024-03-06T07:22:23.008727+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:30.960027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007205+0000","last_change":"2024-03-06T07:22:20.701291+0000","last_active":"2024-03-06T07:22:23.007205+0000","last_peered":"2024-03-06T07:22:23.007205+0000","last_clean":"2024-03-06T07:22:23.007205+0000","last_became_active":"2024-03-06T07:22:20.701012+0000","last_became_peered":"2024-03-06T07:22:20.701012+0000","last_unstale":"2024-03-06T07:22:23.007205+0000","last_undegraded":"2024-03-06T07:22:23.007205+0000","last_fullsized":"2024-03-06T07:22:23.007205+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:01.325576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007690+0000","last_change":"2024-03-06T07:22:06.329315+0000","last_active":"2024-03-06T07:22:23.007690+0000","last_peered":"2024-03-06T07:22:23.007690+0000","last_clean":"2024-03-06T07:22:23.007690+0000","last_became_active":"2024-03-06T07:22:06.322815+0000","last_became_peered":"2024-03-06T07:22:06.322815+0000","last_unstale":"2024-03-06T07:22:23.007690+0000","last_undegraded":"2024-03-06T07:22:23.007690+0000","last_fullsized":"2024-03-06T07:22:23.007690+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:15.214675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706031+0000","last_change":"2024-03-06T07:22:05.324367+0000","last_active":"2024-03-06T07:22:22.706031+0000","last_peered":"2024-03-06T07:22:22.706031+0000","last_clean":"2024-03-06T07:22:22.706031+0000","last_became_active":"2024-03-06T07:22:05.324086+0000","last_became_peered":"2024-03-06T07:22:05.324086+0000","last_unstale":"2024-03-06T07:22:22.706031+0000","last_undegraded":"2024-03-06T07:22:22.706031+0000","last_fullsized":"2024-03-06T07:22:22.706031+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:31.061844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705866+0000","last_change":"2024-03-06T07:22:21.725366+0000","last_active":"2024-03-06T07:22:22.705866+0000","last_peered":"2024-03-06T07:22:22.705866+0000","last_clean":"2024-03-06T07:22:22.705866+0000","last_became_active":"2024-03-06T07:22:21.725118+0000","last_became_peered":"2024-03-06T07:22:21.725118+0000","last_unstale":"2024-03-06T07:22:22.705866+0000","last_undegraded":"2024-03-06T07:22:22.705866+0000","last_fullsized":"2024-03-06T07:22:22.705866+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:32.198283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"55'4","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007459+0000","last_change":"2024-03-06T07:22:20.710921+0000","last_active":"2024-03-06T07:22:23.007459+0000","last_peered":"2024-03-06T07:22:23.007459+0000","last_clean":"2024-03-06T07:22:23.007459+0000","last_became_active":"2024-03-06T07:22:20.710660+0000","last_became_peered":"2024-03-06T07:22:20.710660+0000","last_unstale":"2024-03-06T07:22:23.007459+0000","last_undegraded":"2024-03-06T07:22:23.007459+0000","last_fullsized":"2024-03-06T07:22:23.007459+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:53.282115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705575+0000","last_change":"2024-03-06T07:22:06.322371+0000","last_active":"2024-03-06T07:22:22.705575+0000","last_peered":"2024-03-06T07:22:22.705575+0000","last_clean":"2024-03-06T07:22:22.705575+0000","last_became_active":"2024-03-06T07:22:06.322099+0000","last_became_peered":"2024-03-06T07:22:06.322099+0000","last_unstale":"2024-03-06T07:22:22.705575+0000","last_undegraded":"2024-03-06T07:22:22.705575+0000","last_fullsized":"2024-03-06T07:22:22.705575+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:02.237473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008220+0000","last_change":"2024-03-06T07:22:05.320305+0000","last_active":"2024-03-06T07:22:23.008220+0000","last_peered":"2024-03-06T07:22:23.008220+0000","last_clean":"2024-03-06T07:22:23.008220+0000","last_became_active":"2024-03-06T07:22:05.319970+0000","last_became_peered":"2024-03-06T07:22:05.319970+0000","last_unstale":"2024-03-06T07:22:23.008220+0000","last_undegraded":"2024-03-06T07:22:23.008220+0000","last_fullsized":"2024-03-06T07:22:23.008220+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:01.935488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008091+0000","last_change":"2024-03-06T07:22:21.723887+0000","last_active":"2024-03-06T07:22:23.008091+0000","last_peered":"2024-03-06T07:22:23.008091+0000","last_clean":"2024-03-06T07:22:23.008091+0000","last_became_active":"2024-03-06T07:22:21.723616+0000","last_became_peered":"2024-03-06T07:22:21.723616+0000","last_unstale":"2024-03-06T07:22:23.008091+0000","last_undegraded":"2024-03-06T07:22:23.008091+0000","last_fullsized":"2024-03-06T07:22:23.008091+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:30.723025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007950+0000","last_change":"2024-03-06T07:22:20.697649+0000","last_active":"2024-03-06T07:22:23.007950+0000","last_peered":"2024-03-06T07:22:23.007950+0000","last_clean":"2024-03-06T07:22:23.007950+0000","last_became_active":"2024-03-06T07:22:20.697385+0000","last_became_peered":"2024-03-06T07:22:20.697385+0000","last_unstale":"2024-03-06T07:22:23.007950+0000","last_undegraded":"2024-03-06T07:22:23.007950+0000","last_fullsized":"2024-03-06T07:22:23.007950+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:07.182303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007617+0000","last_change":"2024-03-06T07:22:06.318628+0000","last_active":"2024-03-06T07:22:23.007617+0000","last_peered":"2024-03-06T07:22:23.007617+0000","last_clean":"2024-03-06T07:22:23.007617+0000","last_became_active":"2024-03-06T07:22:06.318324+0000","last_became_peered":"2024-03-06T07:22:06.318324+0000","last_unstale":"2024-03-06T07:22:23.007617+0000","last_undegraded":"2024-03-06T07:22:23.007617+0000","last_fullsized":"2024-03-06T07:22:23.007617+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:09.827959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":30,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705348+0000","last_change":"2024-03-06T07:22:05.316578+0000","last_active":"2024-03-06T07:22:22.705348+0000","last_peered":"2024-03-06T07:22:22.705348+0000","last_clean":"2024-03-06T07:22:22.705348+0000","last_became_active":"2024-03-06T07:22:05.316285+0000","last_became_peered":"2024-03-06T07:22:05.316285+0000","last_unstale":"2024-03-06T07:22:22.705348+0000","last_undegraded":"2024-03-06T07:22:22.705348+0000","last_fullsized":"2024-03-06T07:22:22.705348+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:21.439057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007868+0000","last_change":"2024-03-06T07:22:21.716524+0000","last_active":"2024-03-06T07:22:23.007868+0000","last_peered":"2024-03-06T07:22:23.007868+0000","last_clean":"2024-03-06T07:22:23.007868+0000","last_became_active":"2024-03-06T07:22:21.716242+0000","last_became_peered":"2024-03-06T07:22:21.716242+0000","last_unstale":"2024-03-06T07:22:23.007868+0000","last_undegraded":"2024-03-06T07:22:23.007868+0000","last_fullsized":"2024-03-06T07:22:23.007868+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:13:16.916071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705302+0000","last_change":"2024-03-06T07:22:20.706114+0000","last_active":"2024-03-06T07:22:22.705302+0000","last_peered":"2024-03-06T07:22:22.705302+0000","last_clean":"2024-03-06T07:22:22.705302+0000","last_became_active":"2024-03-06T07:22:20.705831+0000","last_became_peered":"2024-03-06T07:22:20.705831+0000","last_unstale":"2024-03-06T07:22:22.705302+0000","last_undegraded":"2024-03-06T07:22:22.705302+0000","last_fullsized":"2024-03-06T07:22:22.705302+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:28.208506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705286+0000","last_change":"2024-03-06T07:22:06.328590+0000","last_active":"2024-03-06T07:22:22.705286+0000","last_peered":"2024-03-06T07:22:22.705286+0000","last_clean":"2024-03-06T07:22:22.705286+0000","last_became_active":"2024-03-06T07:22:06.327594+0000","last_became_peered":"2024-03-06T07:22:06.327594+0000","last_unstale":"2024-03-06T07:22:22.705286+0000","last_undegraded":"2024-03-06T07:22:22.705286+0000","last_fullsized":"2024-03-06T07:22:22.705286+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:46.981658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007956+0000","last_change":"2024-03-06T07:22:05.325528+0000","last_active":"2024-03-06T07:22:23.007956+0000","last_peered":"2024-03-06T07:22:23.007956+0000","last_clean":"2024-03-06T07:22:23.007956+0000","last_became_active":"2024-03-06T07:22:05.325254+0000","last_became_peered":"2024-03-06T07:22:05.325254+0000","last_unstale":"2024-03-06T07:22:23.007956+0000","last_undegraded":"2024-03-06T07:22:23.007956+0000","last_fullsized":"2024-03-06T07:22:23.007956+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:44.392826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.703958+0000","last_change":"2024-03-06T07:22:21.719104+0000","last_active":"2024-03-06T07:22:22.703958+0000","last_peered":"2024-03-06T07:22:22.703958+0000","last_clean":"2024-03-06T07:22:22.703958+0000","last_became_active":"2024-03-06T07:22:21.718824+0000","last_became_peered":"2024-03-06T07:22:21.718824+0000","last_unstale":"2024-03-06T07:22:22.703958+0000","last_undegraded":"2024-03-06T07:22:22.703958+0000","last_fullsized":"2024-03-06T07:22:22.703958+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:34.906272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006548+0000","last_change":"2024-03-06T07:22:20.709142+0000","last_active":"2024-03-06T07:22:23.006548+0000","last_peered":"2024-03-06T07:22:23.006548+0000","last_clean":"2024-03-06T07:22:23.006548+0000","last_became_active":"2024-03-06T07:22:20.708835+0000","last_became_peered":"2024-03-06T07:22:20.708835+0000","last_unstale":"2024-03-06T07:22:23.006548+0000","last_undegraded":"2024-03-06T07:22:23.006548+0000","last_fullsized":"2024-03-06T07:22:23.006548+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:31.193605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007028+0000","last_change":"2024-03-06T07:22:06.320394+0000","last_active":"2024-03-06T07:22:23.007028+0000","last_peered":"2024-03-06T07:22:23.007028+0000","last_clean":"2024-03-06T07:22:23.007028+0000","last_became_active":"2024-03-06T07:22:06.320135+0000","last_became_peered":"2024-03-06T07:22:06.320135+0000","last_unstale":"2024-03-06T07:22:23.007028+0000","last_undegraded":"2024-03-06T07:22:23.007028+0000","last_fullsized":"2024-03-06T07:22:23.007028+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:34.766315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.704708+0000","last_change":"2024-03-06T07:22:05.314843+0000","last_active":"2024-03-06T07:22:22.704708+0000","last_peered":"2024-03-06T07:22:22.704708+0000","last_clean":"2024-03-06T07:22:22.704708+0000","last_became_active":"2024-03-06T07:22:05.314567+0000","last_became_peered":"2024-03-06T07:22:05.314567+0000","last_unstale":"2024-03-06T07:22:22.704708+0000","last_undegraded":"2024-03-06T07:22:22.704708+0000","last_fullsized":"2024-03-06T07:22:22.704708+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:23.891871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705345+0000","last_change":"2024-03-06T07:22:21.722602+0000","last_active":"2024-03-06T07:22:22.705345+0000","last_peered":"2024-03-06T07:22:22.705345+0000","last_clean":"2024-03-06T07:22:22.705345+0000","last_became_active":"2024-03-06T07:22:21.722323+0000","last_became_peered":"2024-03-06T07:22:21.722323+0000","last_unstale":"2024-03-06T07:22:22.705345+0000","last_undegraded":"2024-03-06T07:22:22.705345+0000","last_fullsized":"2024-03-06T07:22:22.705345+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:05.288867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008914+0000","last_change":"2024-03-06T07:22:20.703676+0000","last_active":"2024-03-06T07:22:23.008914+0000","last_peered":"2024-03-06T07:22:23.008914+0000","last_clean":"2024-03-06T07:22:23.008914+0000","last_became_active":"2024-03-06T07:22:20.703383+0000","last_became_peered":"2024-03-06T07:22:20.703383+0000","last_unstale":"2024-03-06T07:22:23.008914+0000","last_undegraded":"2024-03-06T07:22:23.008914+0000","last_fullsized":"2024-03-06T07:22:23.008914+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:29.718346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007174+0000","last_change":"2024-03-06T07:22:06.321026+0000","last_active":"2024-03-06T07:22:23.007174+0000","last_peered":"2024-03-06T07:22:23.007174+0000","last_clean":"2024-03-06T07:22:23.007174+0000","last_became_active":"2024-03-06T07:22:06.320749+0000","last_became_peered":"2024-03-06T07:22:06.320749+0000","last_unstale":"2024-03-06T07:22:23.007174+0000","last_undegraded":"2024-03-06T07:22:23.007174+0000","last_fullsized":"2024-03-06T07:22:23.007174+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:59.907792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706410+0000","last_change":"2024-03-06T07:22:05.325044+0000","last_active":"2024-03-06T07:22:22.706410+0000","last_peered":"2024-03-06T07:22:22.706410+0000","last_clean":"2024-03-06T07:22:22.706410+0000","last_became_active":"2024-03-06T07:22:05.324731+0000","last_became_peered":"2024-03-06T07:22:05.324731+0000","last_unstale":"2024-03-06T07:22:22.706410+0000","last_undegraded":"2024-03-06T07:22:22.706410+0000","last_fullsized":"2024-03-06T07:22:22.706410+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:50.594146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007073+0000","last_change":"2024-03-06T07:22:21.734429+0000","last_active":"2024-03-06T07:22:23.007073+0000","last_peered":"2024-03-06T07:22:23.007073+0000","last_clean":"2024-03-06T07:22:23.007073+0000","last_became_active":"2024-03-06T07:22:21.734156+0000","last_became_peered":"2024-03-06T07:22:21.734156+0000","last_unstale":"2024-03-06T07:22:23.007073+0000","last_undegraded":"2024-03-06T07:22:23.007073+0000","last_fullsized":"2024-03-06T07:22:23.007073+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:54.286794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705488+0000","last_change":"2024-03-06T07:22:20.703911+0000","last_active":"2024-03-06T07:22:22.705488+0000","last_peered":"2024-03-06T07:22:22.705488+0000","last_clean":"2024-03-06T07:22:22.705488+0000","last_became_active":"2024-03-06T07:22:20.703633+0000","last_became_peered":"2024-03-06T07:22:20.703633+0000","last_unstale":"2024-03-06T07:22:22.705488+0000","last_undegraded":"2024-03-06T07:22:22.705488+0000","last_fullsized":"2024-03-06T07:22:22.705488+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:29.955349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007427+0000","last_change":"2024-03-06T07:22:06.327972+0000","last_active":"2024-03-06T07:22:23.007427+0000","last_peered":"2024-03-06T07:22:23.007427+0000","last_clean":"2024-03-06T07:22:23.007427+0000","last_became_active":"2024-03-06T07:22:06.327676+0000","last_became_peered":"2024-03-06T07:22:06.327676+0000","last_unstale":"2024-03-06T07:22:23.007427+0000","last_undegraded":"2024-03-06T07:22:23.007427+0000","last_fullsized":"2024-03-06T07:22:23.007427+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:53.049705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706086+0000","last_change":"2024-03-06T07:22:05.318191+0000","last_active":"2024-03-06T07:22:22.706086+0000","last_peered":"2024-03-06T07:22:22.706086+0000","last_clean":"2024-03-06T07:22:22.706086+0000","last_became_active":"2024-03-06T07:22:05.317984+0000","last_became_peered":"2024-03-06T07:22:05.317984+0000","last_unstale":"2024-03-06T07:22:22.706086+0000","last_undegraded":"2024-03-06T07:22:22.706086+0000","last_fullsized":"2024-03-06T07:22:22.706086+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:48.457911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719680+0000","last_change":"2024-03-06T07:22:21.728826+0000","last_active":"2024-03-06T07:22:22.719680+0000","last_peered":"2024-03-06T07:22:22.719680+0000","last_clean":"2024-03-06T07:22:22.719680+0000","last_became_active":"2024-03-06T07:22:21.724267+0000","last_became_peered":"2024-03-06T07:22:21.724267+0000","last_unstale":"2024-03-06T07:22:22.719680+0000","last_undegraded":"2024-03-06T07:22:22.719680+0000","last_fullsized":"2024-03-06T07:22:22.719680+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:01.959177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007384+0000","last_change":"2024-03-06T07:22:20.698001+0000","last_active":"2024-03-06T07:22:23.007384+0000","last_peered":"2024-03-06T07:22:23.007384+0000","last_clean":"2024-03-06T07:22:23.007384+0000","last_became_active":"2024-03-06T07:22:20.697566+0000","last_became_peered":"2024-03-06T07:22:20.697566+0000","last_unstale":"2024-03-06T07:22:23.007384+0000","last_undegraded":"2024-03-06T07:22:23.007384+0000","last_fullsized":"2024-03-06T07:22:23.007384+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:16.348823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719488+0000","last_change":"2024-03-06T07:22:06.317413+0000","last_active":"2024-03-06T07:22:22.719488+0000","last_peered":"2024-03-06T07:22:22.719488+0000","last_clean":"2024-03-06T07:22:22.719488+0000","last_became_active":"2024-03-06T07:22:06.317126+0000","last_became_peered":"2024-03-06T07:22:06.317126+0000","last_unstale":"2024-03-06T07:22:22.719488+0000","last_undegraded":"2024-03-06T07:22:22.719488+0000","last_fullsized":"2024-03-06T07:22:22.719488+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:17.635384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007663+0000","last_change":"2024-03-06T07:22:05.320901+0000","last_active":"2024-03-06T07:22:23.007663+0000","last_peered":"2024-03-06T07:22:23.007663+0000","last_clean":"2024-03-06T07:22:23.007663+0000","last_became_active":"2024-03-06T07:22:05.320579+0000","last_became_peered":"2024-03-06T07:22:05.320579+0000","last_unstale":"2024-03-06T07:22:23.007663+0000","last_undegraded":"2024-03-06T07:22:23.007663+0000","last_fullsized":"2024-03-06T07:22:23.007663+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:46.701129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706359+0000","last_change":"2024-03-06T07:22:21.722508+0000","last_active":"2024-03-06T07:22:22.706359+0000","last_peered":"2024-03-06T07:22:22.706359+0000","last_clean":"2024-03-06T07:22:22.706359+0000","last_became_active":"2024-03-06T07:22:21.722215+0000","last_became_peered":"2024-03-06T07:22:21.722215+0000","last_unstale":"2024-03-06T07:22:22.706359+0000","last_undegraded":"2024-03-06T07:22:22.706359+0000","last_fullsized":"2024-03-06T07:22:22.706359+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:30.603319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007217+0000","last_change":"2024-03-06T07:22:20.709740+0000","last_active":"2024-03-06T07:22:23.007217+0000","last_peered":"2024-03-06T07:22:23.007217+0000","last_clean":"2024-03-06T07:22:23.007217+0000","last_became_active":"2024-03-06T07:22:20.709457+0000","last_became_peered":"2024-03-06T07:22:20.709457+0000","last_unstale":"2024-03-06T07:22:23.007217+0000","last_undegraded":"2024-03-06T07:22:23.007217+0000","last_fullsized":"2024-03-06T07:22:23.007217+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:31.952778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706231+0000","last_change":"2024-03-06T07:22:06.321030+0000","last_active":"2024-03-06T07:22:22.706231+0000","last_peered":"2024-03-06T07:22:22.706231+0000","last_clean":"2024-03-06T07:22:22.706231+0000","last_became_active":"2024-03-06T07:22:06.320686+0000","last_became_peered":"2024-03-06T07:22:06.320686+0000","last_unstale":"2024-03-06T07:22:22.706231+0000","last_undegraded":"2024-03-06T07:22:22.706231+0000","last_fullsized":"2024-03-06T07:22:22.706231+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:41.730739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705709+0000","last_change":"2024-03-06T07:22:05.316761+0000","last_active":"2024-03-06T07:22:22.705709+0000","last_peered":"2024-03-06T07:22:22.705709+0000","last_clean":"2024-03-06T07:22:22.705709+0000","last_became_active":"2024-03-06T07:22:05.316088+0000","last_became_peered":"2024-03-06T07:22:05.316088+0000","last_unstale":"2024-03-06T07:22:22.705709+0000","last_undegraded":"2024-03-06T07:22:22.705709+0000","last_fullsized":"2024-03-06T07:22:22.705709+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:35.134517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007153+0000","last_change":"2024-03-06T07:22:21.728348+0000","last_active":"2024-03-06T07:22:23.007153+0000","last_peered":"2024-03-06T07:22:23.007153+0000","last_clean":"2024-03-06T07:22:23.007153+0000","last_became_active":"2024-03-06T07:22:21.728078+0000","last_became_peered":"2024-03-06T07:22:21.728078+0000","last_unstale":"2024-03-06T07:22:23.007153+0000","last_undegraded":"2024-03-06T07:22:23.007153+0000","last_fullsized":"2024-03-06T07:22:23.007153+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:15.237287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719893+0000","last_change":"2024-03-06T07:22:20.700220+0000","last_active":"2024-03-06T07:22:22.719893+0000","last_peered":"2024-03-06T07:22:22.719893+0000","last_clean":"2024-03-06T07:22:22.719893+0000","last_became_active":"2024-03-06T07:22:20.699919+0000","last_became_peered":"2024-03-06T07:22:20.699919+0000","last_unstale":"2024-03-06T07:22:22.719893+0000","last_undegraded":"2024-03-06T07:22:22.719893+0000","last_fullsized":"2024-03-06T07:22:22.719893+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:01.365623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008143+0000","last_change":"2024-03-06T07:22:06.313796+0000","last_active":"2024-03-06T07:22:23.008143+0000","last_peered":"2024-03-06T07:22:23.008143+0000","last_clean":"2024-03-06T07:22:23.008143+0000","last_became_active":"2024-03-06T07:22:06.313534+0000","last_became_peered":"2024-03-06T07:22:06.313534+0000","last_unstale":"2024-03-06T07:22:23.008143+0000","last_undegraded":"2024-03-06T07:22:23.008143+0000","last_fullsized":"2024-03-06T07:22:23.008143+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:34.284715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007048+0000","last_change":"2024-03-06T07:22:05.324175+0000","last_active":"2024-03-06T07:22:23.007048+0000","last_peered":"2024-03-06T07:22:23.007048+0000","last_clean":"2024-03-06T07:22:23.007048+0000","last_became_active":"2024-03-06T07:22:05.323587+0000","last_became_peered":"2024-03-06T07:22:05.323587+0000","last_unstale":"2024-03-06T07:22:23.007048+0000","last_undegraded":"2024-03-06T07:22:23.007048+0000","last_fullsized":"2024-03-06T07:22:23.007048+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:01.935488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.704720+0000","last_change":"2024-03-06T07:22:21.709596+0000","last_active":"2024-03-06T07:22:22.704720+0000","last_peered":"2024-03-06T07:22:22.704720+0000","last_clean":"2024-03-06T07:22:22.704720+0000","last_became_active":"2024-03-06T07:22:21.709338+0000","last_became_peered":"2024-03-06T07:22:21.709338+0000","last_unstale":"2024-03-06T07:22:22.704720+0000","last_undegraded":"2024-03-06T07:22:22.704720+0000","last_fullsized":"2024-03-06T07:22:22.704720+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:06.987711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"56'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719962+0000","last_change":"2024-03-06T07:22:20.701812+0000","last_active":"2024-03-06T07:22:22.719962+0000","last_peered":"2024-03-06T07:22:22.719962+0000","last_clean":"2024-03-06T07:22:22.719962+0000","last_became_active":"2024-03-06T07:22:20.701560+0000","last_became_peered":"2024-03-06T07:22:20.701560+0000","last_unstale":"2024-03-06T07:22:22.719962+0000","last_undegraded":"2024-03-06T07:22:22.719962+0000","last_fullsized":"2024-03-06T07:22:22.719962+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:48.668680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007645+0000","last_change":"2024-03-06T07:22:06.325375+0000","last_active":"2024-03-06T07:22:23.007645+0000","last_peered":"2024-03-06T07:22:23.007645+0000","last_clean":"2024-03-06T07:22:23.007645+0000","last_became_active":"2024-03-06T07:22:06.325108+0000","last_became_peered":"2024-03-06T07:22:06.325108+0000","last_unstale":"2024-03-06T07:22:23.007645+0000","last_undegraded":"2024-03-06T07:22:23.007645+0000","last_fullsized":"2024-03-06T07:22:23.007645+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:36.139439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.703958+0000","last_change":"2024-03-06T07:22:05.317252+0000","last_active":"2024-03-06T07:22:22.703958+0000","last_peered":"2024-03-06T07:22:22.703958+0000","last_clean":"2024-03-06T07:22:22.703958+0000","last_became_active":"2024-03-06T07:22:05.316964+0000","last_became_peered":"2024-03-06T07:22:05.316964+0000","last_unstale":"2024-03-06T07:22:22.703958+0000","last_undegraded":"2024-03-06T07:22:22.703958+0000","last_fullsized":"2024-03-06T07:22:22.703958+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:34.517096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705650+0000","last_change":"2024-03-06T07:22:21.718931+0000","last_active":"2024-03-06T07:22:22.705650+0000","last_peered":"2024-03-06T07:22:22.705650+0000","last_clean":"2024-03-06T07:22:22.705650+0000","last_became_active":"2024-03-06T07:22:21.718656+0000","last_became_peered":"2024-03-06T07:22:21.718656+0000","last_unstale":"2024-03-06T07:22:22.705650+0000","last_undegraded":"2024-03-06T07:22:22.705650+0000","last_fullsized":"2024-03-06T07:22:22.705650+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:49.684022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006731+0000","last_change":"2024-03-06T07:22:20.700497+0000","last_active":"2024-03-06T07:22:23.006731+0000","last_peered":"2024-03-06T07:22:23.006731+0000","last_clean":"2024-03-06T07:22:23.006731+0000","last_became_active":"2024-03-06T07:22:20.700240+0000","last_became_peered":"2024-03-06T07:22:20.700240+0000","last_unstale":"2024-03-06T07:22:23.006731+0000","last_undegraded":"2024-03-06T07:22:23.006731+0000","last_fullsized":"2024-03-06T07:22:23.006731+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:00.954498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719321+0000","last_change":"2024-03-06T07:22:06.325564+0000","last_active":"2024-03-06T07:22:22.719321+0000","last_peered":"2024-03-06T07:22:22.719321+0000","last_clean":"2024-03-06T07:22:22.719321+0000","last_became_active":"2024-03-06T07:22:06.325298+0000","last_became_peered":"2024-03-06T07:22:06.325298+0000","last_unstale":"2024-03-06T07:22:22.719321+0000","last_undegraded":"2024-03-06T07:22:22.719321+0000","last_fullsized":"2024-03-06T07:22:22.719321+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:51.508838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706339+0000","last_change":"2024-03-06T07:22:05.326276+0000","last_active":"2024-03-06T07:22:22.706339+0000","last_peered":"2024-03-06T07:22:22.706339+0000","last_clean":"2024-03-06T07:22:22.706339+0000","last_became_active":"2024-03-06T07:22:05.325993+0000","last_became_peered":"2024-03-06T07:22:05.325993+0000","last_unstale":"2024-03-06T07:22:22.706339+0000","last_undegraded":"2024-03-06T07:22:22.706339+0000","last_fullsized":"2024-03-06T07:22:22.706339+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:19.366175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006647+0000","last_change":"2024-03-06T07:22:21.710303+0000","last_active":"2024-03-06T07:22:23.006647+0000","last_peered":"2024-03-06T07:22:23.006647+0000","last_clean":"2024-03-06T07:22:23.006647+0000","last_became_active":"2024-03-06T07:22:21.710050+0000","last_became_peered":"2024-03-06T07:22:21.710050+0000","last_unstale":"2024-03-06T07:22:23.006647+0000","last_undegraded":"2024-03-06T07:22:23.006647+0000","last_fullsized":"2024-03-06T07:22:23.006647+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:02.816787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006803+0000","last_change":"2024-03-06T07:22:20.709655+0000","last_active":"2024-03-06T07:22:23.006803+0000","last_peered":"2024-03-06T07:22:23.006803+0000","last_clean":"2024-03-06T07:22:23.006803+0000","last_became_active":"2024-03-06T07:22:20.709159+0000","last_became_peered":"2024-03-06T07:22:20.709159+0000","last_unstale":"2024-03-06T07:22:23.006803+0000","last_undegraded":"2024-03-06T07:22:23.006803+0000","last_fullsized":"2024-03-06T07:22:23.006803+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:27.335366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008150+0000","last_change":"2024-03-06T07:22:06.323310+0000","last_active":"2024-03-06T07:22:23.008150+0000","last_peered":"2024-03-06T07:22:23.008150+0000","last_clean":"2024-03-06T07:22:23.008150+0000","last_became_active":"2024-03-06T07:22:06.322311+0000","last_became_peered":"2024-03-06T07:22:06.322311+0000","last_unstale":"2024-03-06T07:22:23.008150+0000","last_undegraded":"2024-03-06T07:22:23.008150+0000","last_fullsized":"2024-03-06T07:22:23.008150+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:01.204740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008469+0000","last_change":"2024-03-06T07:22:05.319638+0000","last_active":"2024-03-06T07:22:23.008469+0000","last_peered":"2024-03-06T07:22:23.008469+0000","last_clean":"2024-03-06T07:22:23.008469+0000","last_became_active":"2024-03-06T07:22:05.319307+0000","last_became_peered":"2024-03-06T07:22:05.319307+0000","last_unstale":"2024-03-06T07:22:23.008469+0000","last_undegraded":"2024-03-06T07:22:23.008469+0000","last_fullsized":"2024-03-06T07:22:23.008469+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:19.853691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719254+0000","last_change":"2024-03-06T07:22:21.720493+0000","last_active":"2024-03-06T07:22:22.719254+0000","last_peered":"2024-03-06T07:22:22.719254+0000","last_clean":"2024-03-06T07:22:22.719254+0000","last_became_active":"2024-03-06T07:22:21.719998+0000","last_became_peered":"2024-03-06T07:22:21.719998+0000","last_unstale":"2024-03-06T07:22:22.719254+0000","last_undegraded":"2024-03-06T07:22:22.719254+0000","last_fullsized":"2024-03-06T07:22:22.719254+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:49.989399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719528+0000","last_change":"2024-03-06T07:22:20.704397+0000","last_active":"2024-03-06T07:22:22.719528+0000","last_peered":"2024-03-06T07:22:22.719528+0000","last_clean":"2024-03-06T07:22:22.719528+0000","last_became_active":"2024-03-06T07:22:20.704078+0000","last_became_peered":"2024-03-06T07:22:20.704078+0000","last_unstale":"2024-03-06T07:22:22.719528+0000","last_undegraded":"2024-03-06T07:22:22.719528+0000","last_fullsized":"2024-03-06T07:22:22.719528+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:29.718346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705485+0000","last_change":"2024-03-06T07:22:06.315689+0000","last_active":"2024-03-06T07:22:22.705485+0000","last_peered":"2024-03-06T07:22:22.705485+0000","last_clean":"2024-03-06T07:22:22.705485+0000","last_became_active":"2024-03-06T07:22:06.315395+0000","last_became_peered":"2024-03-06T07:22:06.315395+0000","last_unstale":"2024-03-06T07:22:22.705485+0000","last_undegraded":"2024-03-06T07:22:22.705485+0000","last_fullsized":"2024-03-06T07:22:22.705485+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:09.827959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007399+0000","last_change":"2024-03-06T07:22:05.319307+0000","last_active":"2024-03-06T07:22:23.007399+0000","last_peered":"2024-03-06T07:22:23.007399+0000","last_clean":"2024-03-06T07:22:23.007399+0000","last_became_active":"2024-03-06T07:22:05.318839+0000","last_became_peered":"2024-03-06T07:22:05.318839+0000","last_unstale":"2024-03-06T07:22:23.007399+0000","last_undegraded":"2024-03-06T07:22:23.007399+0000","last_fullsized":"2024-03-06T07:22:23.007399+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:40.725817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008601+0000","last_change":"2024-03-06T07:22:21.722966+0000","last_active":"2024-03-06T07:22:23.008601+0000","last_peered":"2024-03-06T07:22:23.008601+0000","last_clean":"2024-03-06T07:22:23.008601+0000","last_became_active":"2024-03-06T07:22:21.722659+0000","last_became_peered":"2024-03-06T07:22:21.722659+0000","last_unstale":"2024-03-06T07:22:23.008601+0000","last_undegraded":"2024-03-06T07:22:23.008601+0000","last_fullsized":"2024-03-06T07:22:23.008601+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:50.059429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007410+0000","last_change":"2024-03-06T07:22:20.706953+0000","last_active":"2024-03-06T07:22:23.007410+0000","last_peered":"2024-03-06T07:22:23.007410+0000","last_clean":"2024-03-06T07:22:23.007410+0000","last_became_active":"2024-03-06T07:22:20.706459+0000","last_became_peered":"2024-03-06T07:22:20.706459+0000","last_unstale":"2024-03-06T07:22:23.007410+0000","last_undegraded":"2024-03-06T07:22:23.007410+0000","last_fullsized":"2024-03-06T07:22:23.007410+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:24.211924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007309+0000","last_change":"2024-03-06T07:22:06.331443+0000","last_active":"2024-03-06T07:22:23.007309+0000","last_peered":"2024-03-06T07:22:23.007309+0000","last_clean":"2024-03-06T07:22:23.007309+0000","last_became_active":"2024-03-06T07:22:06.331189+0000","last_became_peered":"2024-03-06T07:22:06.331189+0000","last_unstale":"2024-03-06T07:22:23.007309+0000","last_undegraded":"2024-03-06T07:22:23.007309+0000","last_fullsized":"2024-03-06T07:22:23.007309+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:34.295379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705818+0000","last_change":"2024-03-06T07:22:05.315558+0000","last_active":"2024-03-06T07:22:22.705818+0000","last_peered":"2024-03-06T07:22:22.705818+0000","last_clean":"2024-03-06T07:22:22.705818+0000","last_became_active":"2024-03-06T07:22:05.315263+0000","last_became_peered":"2024-03-06T07:22:05.315263+0000","last_unstale":"2024-03-06T07:22:22.705818+0000","last_undegraded":"2024-03-06T07:22:22.705818+0000","last_fullsized":"2024-03-06T07:22:22.705818+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:01.553893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706276+0000","last_change":"2024-03-06T07:22:21.723229+0000","last_active":"2024-03-06T07:22:22.706276+0000","last_peered":"2024-03-06T07:22:22.706276+0000","last_clean":"2024-03-06T07:22:22.706276+0000","last_became_active":"2024-03-06T07:22:21.722956+0000","last_became_peered":"2024-03-06T07:22:21.722956+0000","last_unstale":"2024-03-06T07:22:22.706276+0000","last_undegraded":"2024-03-06T07:22:22.706276+0000","last_fullsized":"2024-03-06T07:22:22.706276+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:34.109293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705671+0000","last_change":"2024-03-06T07:22:20.703827+0000","last_active":"2024-03-06T07:22:22.705671+0000","last_peered":"2024-03-06T07:22:22.705671+0000","last_clean":"2024-03-06T07:22:22.705671+0000","last_became_active":"2024-03-06T07:22:20.703580+0000","last_became_peered":"2024-03-06T07:22:20.703580+0000","last_unstale":"2024-03-06T07:22:22.705671+0000","last_undegraded":"2024-03-06T07:22:22.705671+0000","last_fullsized":"2024-03-06T07:22:22.705671+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:41.367702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719560+0000","last_change":"2024-03-06T07:22:06.324288+0000","last_active":"2024-03-06T07:22:22.719560+0000","last_peered":"2024-03-06T07:22:22.719560+0000","last_clean":"2024-03-06T07:22:22.719560+0000","last_became_active":"2024-03-06T07:22:06.323925+0000","last_became_peered":"2024-03-06T07:22:06.323925+0000","last_unstale":"2024-03-06T07:22:22.719560+0000","last_undegraded":"2024-03-06T07:22:22.719560+0000","last_fullsized":"2024-03-06T07:22:22.719560+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:49.900223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705737+0000","last_change":"2024-03-06T07:22:05.322782+0000","last_active":"2024-03-06T07:22:22.705737+0000","last_peered":"2024-03-06T07:22:22.705737+0000","last_clean":"2024-03-06T07:22:22.705737+0000","last_became_active":"2024-03-06T07:22:05.322484+0000","last_became_peered":"2024-03-06T07:22:05.322484+0000","last_unstale":"2024-03-06T07:22:22.705737+0000","last_undegraded":"2024-03-06T07:22:22.705737+0000","last_fullsized":"2024-03-06T07:22:22.705737+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:04.727017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008539+0000","last_change":"2024-03-06T07:22:21.721093+0000","last_active":"2024-03-06T07:22:23.008539+0000","last_peered":"2024-03-06T07:22:23.008539+0000","last_clean":"2024-03-06T07:22:23.008539+0000","last_became_active":"2024-03-06T07:22:21.720833+0000","last_became_peered":"2024-03-06T07:22:21.720833+0000","last_unstale":"2024-03-06T07:22:23.008539+0000","last_undegraded":"2024-03-06T07:22:23.008539+0000","last_fullsized":"2024-03-06T07:22:23.008539+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:17.698053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719770+0000","last_change":"2024-03-06T07:22:20.705956+0000","last_active":"2024-03-06T07:22:22.719770+0000","last_peered":"2024-03-06T07:22:22.719770+0000","last_clean":"2024-03-06T07:22:22.719770+0000","last_became_active":"2024-03-06T07:22:20.705630+0000","last_became_peered":"2024-03-06T07:22:20.705630+0000","last_unstale":"2024-03-06T07:22:22.719770+0000","last_undegraded":"2024-03-06T07:22:22.719770+0000","last_fullsized":"2024-03-06T07:22:22.719770+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:24.211924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008030+0000","last_change":"2024-03-06T07:22:06.321965+0000","last_active":"2024-03-06T07:22:23.008030+0000","last_peered":"2024-03-06T07:22:23.008030+0000","last_clean":"2024-03-06T07:22:23.008030+0000","last_became_active":"2024-03-06T07:22:06.320167+0000","last_became_peered":"2024-03-06T07:22:06.320167+0000","last_unstale":"2024-03-06T07:22:23.008030+0000","last_undegraded":"2024-03-06T07:22:23.008030+0000","last_fullsized":"2024-03-06T07:22:23.008030+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:15.334381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007251+0000","last_change":"2024-03-06T07:22:05.312784+0000","last_active":"2024-03-06T07:22:23.007251+0000","last_peered":"2024-03-06T07:22:23.007251+0000","last_clean":"2024-03-06T07:22:23.007251+0000","last_became_active":"2024-03-06T07:22:05.312472+0000","last_became_peered":"2024-03-06T07:22:05.312472+0000","last_unstale":"2024-03-06T07:22:23.007251+0000","last_undegraded":"2024-03-06T07:22:23.007251+0000","last_fullsized":"2024-03-06T07:22:23.007251+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:43.288752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706255+0000","last_change":"2024-03-06T07:22:21.727433+0000","last_active":"2024-03-06T07:22:22.706255+0000","last_peered":"2024-03-06T07:22:22.706255+0000","last_clean":"2024-03-06T07:22:22.706255+0000","last_became_active":"2024-03-06T07:22:21.727182+0000","last_became_peered":"2024-03-06T07:22:21.727182+0000","last_unstale":"2024-03-06T07:22:22.706255+0000","last_undegraded":"2024-03-06T07:22:22.706255+0000","last_fullsized":"2024-03-06T07:22:22.706255+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:54.286794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008239+0000","last_change":"2024-03-06T07:22:20.706627+0000","last_active":"2024-03-06T07:22:23.008239+0000","last_peered":"2024-03-06T07:22:23.008239+0000","last_clean":"2024-03-06T07:22:23.008239+0000","last_became_active":"2024-03-06T07:22:20.706360+0000","last_became_peered":"2024-03-06T07:22:20.706360+0000","last_unstale":"2024-03-06T07:22:23.008239+0000","last_undegraded":"2024-03-06T07:22:23.008239+0000","last_fullsized":"2024-03-06T07:22:23.008239+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:01.365623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705904+0000","last_change":"2024-03-06T07:22:06.319779+0000","last_active":"2024-03-06T07:22:22.705904+0000","last_peered":"2024-03-06T07:22:22.705904+0000","last_clean":"2024-03-06T07:22:22.705904+0000","last_became_active":"2024-03-06T07:22:06.319415+0000","last_became_peered":"2024-03-06T07:22:06.319415+0000","last_unstale":"2024-03-06T07:22:22.705904+0000","last_undegraded":"2024-03-06T07:22:22.705904+0000","last_fullsized":"2024-03-06T07:22:22.705904+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:20.884386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719743+0000","last_change":"2024-03-06T07:22:05.322202+0000","last_active":"2024-03-06T07:22:22.719743+0000","last_peered":"2024-03-06T07:22:22.719743+0000","last_clean":"2024-03-06T07:22:22.719743+0000","last_became_active":"2024-03-06T07:22:05.321896+0000","last_became_peered":"2024-03-06T07:22:05.321896+0000","last_unstale":"2024-03-06T07:22:22.719743+0000","last_undegraded":"2024-03-06T07:22:22.719743+0000","last_fullsized":"2024-03-06T07:22:22.719743+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:44.392826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706135+0000","last_change":"2024-03-06T07:22:21.724699+0000","last_active":"2024-03-06T07:22:22.706135+0000","last_peered":"2024-03-06T07:22:22.706135+0000","last_clean":"2024-03-06T07:22:22.706135+0000","last_became_active":"2024-03-06T07:22:21.724448+0000","last_became_peered":"2024-03-06T07:22:21.724448+0000","last_unstale":"2024-03-06T07:22:22.706135+0000","last_undegraded":"2024-03-06T07:22:22.706135+0000","last_fullsized":"2024-03-06T07:22:22.706135+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:32.957457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705071+0000","last_change":"2024-03-06T07:22:20.700732+0000","last_active":"2024-03-06T07:22:22.705071+0000","last_peered":"2024-03-06T07:22:22.705071+0000","last_clean":"2024-03-06T07:22:22.705071+0000","last_became_active":"2024-03-06T07:22:20.700157+0000","last_became_peered":"2024-03-06T07:22:20.700157+0000","last_unstale":"2024-03-06T07:22:22.705071+0000","last_undegraded":"2024-03-06T07:22:22.705071+0000","last_fullsized":"2024-03-06T07:22:22.705071+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:07.182303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705476+0000","last_change":"2024-03-06T07:22:06.328678+0000","last_active":"2024-03-06T07:22:22.705476+0000","last_peered":"2024-03-06T07:22:22.705476+0000","last_clean":"2024-03-06T07:22:22.705476+0000","last_became_active":"2024-03-06T07:22:06.328362+0000","last_became_peered":"2024-03-06T07:22:06.328362+0000","last_unstale":"2024-03-06T07:22:22.705476+0000","last_undegraded":"2024-03-06T07:22:22.705476+0000","last_fullsized":"2024-03-06T07:22:22.705476+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:27.383059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"52'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008147+0000","last_change":"2024-03-06T07:22:05.326257+0000","last_active":"2024-03-06T07:22:23.008147+0000","last_peered":"2024-03-06T07:22:23.008147+0000","last_clean":"2024-03-06T07:22:23.008147+0000","last_became_active":"2024-03-06T07:22:05.325920+0000","last_became_peered":"2024-03-06T07:22:05.325920+0000","last_unstale":"2024-03-06T07:22:23.008147+0000","last_undegraded":"2024-03-06T07:22:23.008147+0000","last_fullsized":"2024-03-06T07:22:23.008147+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:58.902870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":239,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007474+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:22:23.007474+0000","last_peered":"2024-03-06T07:22:23.007474+0000","last_clean":"2024-03-06T07:22:23.007474+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:22:23.007474+0000","last_undegraded":"2024-03-06T07:22:23.007474+0000","last_fullsized":"2024-03-06T07:22:23.007474+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":757760,"data_stored":898948,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":308366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":307702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":684032,"data_stored":309874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":309314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":679936,"data_stored":309292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":704512,"data_stored":310216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":741376,"data_stored":898104,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:25.793 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.925+0000 7f2fa9ba1640 1 Processor -- start 2024-03-06T07:22:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.925+0000 7f2fa9ba1640 1 -- start start 2024-03-06T07:22:25.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.926+0000 7f2fa9ba1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2fa406c490 0x7f2fa40a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.926+0000 7f2fa9ba1640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa4153e30 con 0x7f2fa40a9090 2024-03-06T07:22:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.926+0000 7f2fa9ba1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa40c1bc0 con 0x7f2fa40c28c0 2024-03-06T07:22:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.926+0000 7f2fa9ba1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa406c870 con 0x7f2fa406c490 2024-03-06T07:22:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.926+0000 7f2fa3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2fa40a9090 0x7f2fa40a94a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43870/0 (socket says 172.21.15.1:43870) 2024-03-06T07:22:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa3fff640 1 -- 172.21.15.1:0/3981123588 learned_addr learned my addr 172.21.15.1:0/3981123588 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa27fc640 1 -- 172.21.15.1:0/3981123588 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489043165 0 0) 0x7f2fa4153e30 con 0x7f2fa40a9090 2024-03-06T07:22:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa27fc640 1 -- 172.21.15.1:0/3981123588 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f8c003520 con 0x7f2fa40a9090 2024-03-06T07:22:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa37fe640 1 --2- 172.21.15.1:0/3981123588 >> v2:172.21.15.98:3300/0 conn(0x7f2fa406c490 0x7f2fa40a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa27fc640 1 -- 172.21.15.1:0/3981123588 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 765956065 0 0) 0x7f2f8c003520 con 0x7f2fa40a9090 2024-03-06T07:22:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa27fc640 1 -- 172.21.15.1:0/3981123588 >> v2:172.21.15.98:3300/0 conn(0x7f2fa406c490 msgr2=0x7f2fa40a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa27fc640 1 --2- 172.21.15.1:0/3981123588 >> v2:172.21.15.98:3300/0 conn(0x7f2fa406c490 0x7f2fa40a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa27fc640 1 -- 172.21.15.1:0/3981123588 >> v1:172.21.15.98:6789/0 conn(0x7f2fa40c28c0 legacy=0x7f2fa406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa27fc640 1 -- 172.21.15.1:0/3981123588 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa40c1ea0 con 0x7f2fa40a9090 2024-03-06T07:22:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa27fc640 1 -- 172.21.15.1:0/3981123588 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f2f9c0019f0 con 0x7f2fa40a9090 2024-03-06T07:22:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa27fc640 1 -- 172.21.15.1:0/3981123588 >> v1:172.21.15.1:6789/0 conn(0x7f2fa40a9090 legacy=0x7f2fa40a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa27fc640 1 --2- 172.21.15.1:0/3981123588 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f8c0039b0 0x7f2f8c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa27fc640 1 --2- 172.21.15.1:0/3981123588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f8c004800 0x7f2f8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa27fc640 1 --2- 172.21.15.1:0/3981123588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f8c0052e0 0x7f2f8c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa27fc640 1 -- 172.21.15.1:0/3981123588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa406c870 con 0x7f2f8c0052e0 2024-03-06T07:22:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa27fc640 1 -- 172.21.15.1:0/3981123588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c00c1b0 con 0x7f2f8c004800 2024-03-06T07:22:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa27fc640 1 -- 172.21.15.1:0/3981123588 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c00c3b0 con 0x7f2f8c0039b0 2024-03-06T07:22:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa3fff640 1 --2- 172.21.15.1:0/3981123588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f8c0052e0 0x7f2f8c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa2ffd640 1 --2- 172.21.15.1:0/3981123588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f8c004800 0x7f2f8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.927+0000 7f2fa37fe640 1 --2- 172.21.15.1:0/3981123588 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f8c0039b0 0x7f2f8c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.928+0000 7f2fa3fff640 1 -- 172.21.15.1:0/3981123588 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f8c0039b0 msgr2=0x7f2f8c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.928+0000 7f2fa3fff640 1 --2- 172.21.15.1:0/3981123588 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f8c0039b0 0x7f2f8c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.928+0000 7f2fa3fff640 1 -- 172.21.15.1:0/3981123588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f8c004800 msgr2=0x7f2f8c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.928+0000 7f2fa3fff640 1 --2- 172.21.15.1:0/3981123588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f8c004800 0x7f2f8c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.928+0000 7f2fa3fff640 1 -- 172.21.15.1:0/3981123588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8c003520 con 0x7f2f8c0052e0 2024-03-06T07:22:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.928+0000 7f2fa3fff640 1 -- 172.21.15.1:0/3981123588 reap_dead start 2024-03-06T07:22:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.928+0000 7f2fa3fff640 1 --2- 172.21.15.1:0/3981123588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f8c0052e0 0x7f2f8c00b9c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00a810 tx=0x7f2f8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4de65c9d663353b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.929+0000 7f2fa27fc640 1 -- 172.21.15.1:0/3981123588 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c00a460 con 0x7f2f8c0052e0 2024-03-06T07:22:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.929+0000 7f2fa27fc640 1 -- 172.21.15.1:0/3981123588 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c034650 con 0x7f2f8c0052e0 2024-03-06T07:22:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.929+0000 7f2fa27fc640 1 -- 172.21.15.1:0/3981123588 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c034a40 con 0x7f2f8c0052e0 2024-03-06T07:22:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.929+0000 7f2fa9ba1640 1 -- 172.21.15.1:0/3981123588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f8c0052e0 msgr2=0x7f2f8c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.929+0000 7f2fa9ba1640 1 --2- 172.21.15.1:0/3981123588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f8c0052e0 0x7f2f8c00b9c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00a810 tx=0x7f2f8c005830 comp rx=0 tx=0).stop 2024-03-06T07:22:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.929+0000 7f2fa9ba1640 1 -- 172.21.15.1:0/3981123588 shutdown_connections 2024-03-06T07:22:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.929+0000 7f2fa9ba1640 1 --2- 172.21.15.1:0/3981123588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f8c0052e0 0x7f2f8c00b9c0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.929+0000 7f2fa9ba1640 1 -- 172.21.15.1:0/3981123588 >> 172.21.15.1:0/3981123588 conn(0x7f2fa40c3cc0 msgr2=0x7f2fa40c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.929+0000 7f2fa9ba1640 1 -- 172.21.15.1:0/3981123588 shutdown_connections 2024-03-06T07:22:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.929+0000 7f2fa9ba1640 1 -- 172.21.15.1:0/3981123588 wait complete. 2024-03-06T07:22:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.930+0000 7f2fa9ba1640 1 Processor -- start 2024-03-06T07:22:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.931+0000 7f2fa9ba1640 1 -- start start 2024-03-06T07:22:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.931+0000 7f2fa9ba1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f8c0052e0 0x7f2fa414daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.931+0000 7f2fa9ba1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fa414eb80 0x7f2fa40ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.931+0000 7f2fa9ba1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa40c6490 0x7f2fa40c6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.931+0000 7f2fa9ba1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa4153900 con 0x7f2fa40c6490 2024-03-06T07:22:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.931+0000 7f2fa9ba1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa40c2f60 con 0x7f2fa414eb80 2024-03-06T07:22:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.931+0000 7f2fa9ba1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa40673f0 con 0x7f2f8c0052e0 2024-03-06T07:22:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.931+0000 7f2fa3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa40c6490 0x7f2fa40c6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.931+0000 7f2fa3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa40c6490 0x7f2fa40c6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47146/0 (socket says 172.21.15.1:47146) 2024-03-06T07:22:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.931+0000 7f2fa3fff640 1 -- 172.21.15.1:0/2833178537 learned_addr learned my addr 172.21.15.1:0/2833178537 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.932+0000 7f2fa37fe640 1 --2- 172.21.15.1:0/2833178537 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f8c0052e0 0x7f2fa414daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.932+0000 7f2fa2ffd640 1 --2- 172.21.15.1:0/2833178537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fa414eb80 0x7f2fa40ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.932+0000 7f2fa3fff640 1 -- 172.21.15.1:0/2833178537 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f8c0052e0 msgr2=0x7f2fa414daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.932+0000 7f2fa3fff640 1 --2- 172.21.15.1:0/2833178537 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f8c0052e0 0x7f2fa414daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.932+0000 7f2fa3fff640 1 -- 172.21.15.1:0/2833178537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fa414eb80 msgr2=0x7f2fa40ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.932+0000 7f2fa3fff640 1 --2- 172.21.15.1:0/2833178537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fa414eb80 0x7f2fa40ab8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.932+0000 7f2fa3fff640 1 -- 172.21.15.1:0/2833178537 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa40c6e80 con 0x7f2fa40c6490 2024-03-06T07:22:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.932+0000 7f2fa3fff640 1 --2- 172.21.15.1:0/2833178537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa40c6490 0x7f2fa40c6940 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00a930 tx=0x7f2fa40c1d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.932+0000 7f2f8bfff640 1 -- 172.21.15.1:0/2833178537 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c039040 con 0x7f2fa40c6490 2024-03-06T07:22:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.932+0000 7f2f8bfff640 1 -- 172.21.15.1:0/2833178537 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c043070 con 0x7f2fa40c6490 2024-03-06T07:22:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.932+0000 7f2f8bfff640 1 -- 172.21.15.1:0/2833178537 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c001fb0 con 0x7f2fa40c6490 2024-03-06T07:22:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.932+0000 7f2fa9ba1640 1 -- 172.21.15.1:0/2833178537 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa40c0a30 con 0x7f2fa40c6490 2024-03-06T07:22:25.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.932+0000 7f2fa9ba1640 1 -- 172.21.15.1:0/2833178537 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa40c0ff0 con 0x7f2fa40c6490 2024-03-06T07:22:25.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.933+0000 7f2fa9ba1640 1 -- 172.21.15.1:0/2833178537 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa40686e0 con 0x7f2fa40c6490 2024-03-06T07:22:25.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.934+0000 7f2f8bfff640 1 -- 172.21.15.1:0/2833178537 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f9c004b60 con 0x7f2fa40c6490 2024-03-06T07:22:25.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.934+0000 7f2f8bfff640 1 --2- 172.21.15.1:0/2833178537 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f9006b4d0 0x7f2f9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.934+0000 7f2f8bfff640 1 -- 172.21.15.1:0/2833178537 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f2f9c0ac2a0 con 0x7f2fa40c6490 2024-03-06T07:22:25.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.934+0000 7f2fa37fe640 1 --2- 172.21.15.1:0/2833178537 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f9006b4d0 0x7f2f9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.935+0000 7f2fa37fe640 1 --2- 172.21.15.1:0/2833178537 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f9006b4d0 0x7f2f9006d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009b40 tx=0x7f2f94009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:25.938+0000 7f2f8bfff640 1 -- 172.21.15.1:0/2833178537 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f9c07acd0 con 0x7f2fa40c6490 2024-03-06T07:22:26.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.115+0000 7f2fa9ba1640 1 -- 172.21.15.1:0/2833178537 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2fa40ac170 con 0x7f2f9006b4d0 2024-03-06T07:22:26.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.126+0000 7f2f8bfff640 1 -- 172.21.15.1:0/2833178537 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490559 (secure 0 0 0) 0x7f2f94014b50 con 0x7f2f9006b4d0 2024-03-06T07:22:26.137 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:26.142 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:22:26.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.132+0000 7f2fa9ba1640 1 -- 172.21.15.1:0/2833178537 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f9006b4d0 msgr2=0x7f2f9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.132+0000 7f2fa9ba1640 1 --2- 172.21.15.1:0/2833178537 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f9006b4d0 0x7f2f9006d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009b40 tx=0x7f2f94009190 comp rx=0 tx=0).stop 2024-03-06T07:22:26.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.132+0000 7f2fa9ba1640 1 -- 172.21.15.1:0/2833178537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa40c6490 msgr2=0x7f2fa40c6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.132+0000 7f2fa9ba1640 1 --2- 172.21.15.1:0/2833178537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa40c6490 0x7f2fa40c6940 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00a930 tx=0x7f2fa40c1d90 comp rx=0 tx=0).stop 2024-03-06T07:22:26.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.132+0000 7f2fa9ba1640 1 -- 172.21.15.1:0/2833178537 shutdown_connections 2024-03-06T07:22:26.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.132+0000 7f2fa9ba1640 1 --2- 172.21.15.1:0/2833178537 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f9006b4d0 0x7f2f9006d990 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.132+0000 7f2fa9ba1640 1 --2- 172.21.15.1:0/2833178537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa40c6490 0x7f2fa40c6940 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.132+0000 7f2fa9ba1640 1 --2- 172.21.15.1:0/2833178537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fa414eb80 0x7f2fa40ab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.132+0000 7f2fa9ba1640 1 --2- 172.21.15.1:0/2833178537 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f8c0052e0 0x7f2fa414daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.132+0000 7f2fa9ba1640 1 -- 172.21.15.1:0/2833178537 >> 172.21.15.1:0/2833178537 conn(0x7f2fa40c3cc0 msgr2=0x7f2fa4151550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.132+0000 7f2fa9ba1640 1 -- 172.21.15.1:0/2833178537 shutdown_connections 2024-03-06T07:22:26.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.132+0000 7f2fa9ba1640 1 -- 172.21.15.1:0/2833178537 wait complete. 2024-03-06T07:22:26.147 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-03-06T07:22:24.717528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":375,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238280,"kb_used_data":5472,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736440,"statfs":{"total":773094113280,"available":772850114560,"internally_reserved":0,"allocated":5603328,"data_stored":3651816,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.033398"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007985+0000","last_change":"2024-03-06T07:22:21.717661+0000","last_active":"2024-03-06T07:22:23.007985+0000","last_peered":"2024-03-06T07:22:23.007985+0000","last_clean":"2024-03-06T07:22:23.007985+0000","last_became_active":"2024-03-06T07:22:21.717087+0000","last_became_peered":"2024-03-06T07:22:21.717087+0000","last_unstale":"2024-03-06T07:22:23.007985+0000","last_undegraded":"2024-03-06T07:22:23.007985+0000","last_fullsized":"2024-03-06T07:22:23.007985+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:01.959177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007560+0000","last_change":"2024-03-06T07:22:06.323021+0000","last_active":"2024-03-06T07:22:23.007560+0000","last_peered":"2024-03-06T07:22:23.007560+0000","last_clean":"2024-03-06T07:22:23.007560+0000","last_became_active":"2024-03-06T07:22:06.319646+0000","last_became_peered":"2024-03-06T07:22:06.319646+0000","last_unstale":"2024-03-06T07:22:23.007560+0000","last_undegraded":"2024-03-06T07:22:23.007560+0000","last_fullsized":"2024-03-06T07:22:23.007560+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:51.508838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.009062+0000","last_change":"2024-03-06T07:22:21.716822+0000","last_active":"2024-03-06T07:22:23.009062+0000","last_peered":"2024-03-06T07:22:23.009062+0000","last_clean":"2024-03-06T07:22:23.009062+0000","last_became_active":"2024-03-06T07:22:21.715576+0000","last_became_peered":"2024-03-06T07:22:21.715576+0000","last_unstale":"2024-03-06T07:22:23.009062+0000","last_undegraded":"2024-03-06T07:22:23.009062+0000","last_fullsized":"2024-03-06T07:22:23.009062+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:05.319823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008495+0000","last_change":"2024-03-06T07:22:06.325529+0000","last_active":"2024-03-06T07:22:23.008495+0000","last_peered":"2024-03-06T07:22:23.008495+0000","last_clean":"2024-03-06T07:22:23.008495+0000","last_became_active":"2024-03-06T07:22:06.324758+0000","last_became_peered":"2024-03-06T07:22:06.324758+0000","last_unstale":"2024-03-06T07:22:23.008495+0000","last_undegraded":"2024-03-06T07:22:23.008495+0000","last_fullsized":"2024-03-06T07:22:23.008495+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:02.558815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007218+0000","last_change":"2024-03-06T07:22:21.725743+0000","last_active":"2024-03-06T07:22:23.007218+0000","last_peered":"2024-03-06T07:22:23.007218+0000","last_clean":"2024-03-06T07:22:23.007218+0000","last_became_active":"2024-03-06T07:22:21.724872+0000","last_became_peered":"2024-03-06T07:22:21.724872+0000","last_unstale":"2024-03-06T07:22:23.007218+0000","last_undegraded":"2024-03-06T07:22:23.007218+0000","last_fullsized":"2024-03-06T07:22:23.007218+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:36.273030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007517+0000","last_change":"2024-03-06T07:22:06.321470+0000","last_active":"2024-03-06T07:22:23.007517+0000","last_peered":"2024-03-06T07:22:23.007517+0000","last_clean":"2024-03-06T07:22:23.007517+0000","last_became_active":"2024-03-06T07:22:06.318716+0000","last_became_peered":"2024-03-06T07:22:06.318716+0000","last_unstale":"2024-03-06T07:22:23.007517+0000","last_undegraded":"2024-03-06T07:22:23.007517+0000","last_fullsized":"2024-03-06T07:22:23.007517+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:16.706902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007730+0000","last_change":"2024-03-06T07:22:21.709292+0000","last_active":"2024-03-06T07:22:23.007730+0000","last_peered":"2024-03-06T07:22:23.007730+0000","last_clean":"2024-03-06T07:22:23.007730+0000","last_became_active":"2024-03-06T07:22:21.709012+0000","last_became_peered":"2024-03-06T07:22:21.709012+0000","last_unstale":"2024-03-06T07:22:23.007730+0000","last_undegraded":"2024-03-06T07:22:23.007730+0000","last_fullsized":"2024-03-06T07:22:23.007730+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:40.345682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008291+0000","last_change":"2024-03-06T07:22:06.321775+0000","last_active":"2024-03-06T07:22:23.008291+0000","last_peered":"2024-03-06T07:22:23.008291+0000","last_clean":"2024-03-06T07:22:23.008291+0000","last_became_active":"2024-03-06T07:22:06.321437+0000","last_became_peered":"2024-03-06T07:22:06.321437+0000","last_unstale":"2024-03-06T07:22:23.008291+0000","last_undegraded":"2024-03-06T07:22:23.008291+0000","last_fullsized":"2024-03-06T07:22:23.008291+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:53.049705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.704022+0000","last_change":"2024-03-06T07:22:21.708405+0000","last_active":"2024-03-06T07:22:22.704022+0000","last_peered":"2024-03-06T07:22:22.704022+0000","last_clean":"2024-03-06T07:22:22.704022+0000","last_became_active":"2024-03-06T07:22:21.708146+0000","last_became_peered":"2024-03-06T07:22:21.708146+0000","last_unstale":"2024-03-06T07:22:22.704022+0000","last_undegraded":"2024-03-06T07:22:22.704022+0000","last_fullsized":"2024-03-06T07:22:22.704022+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:35.759740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008405+0000","last_change":"2024-03-06T07:22:06.317556+0000","last_active":"2024-03-06T07:22:23.008405+0000","last_peered":"2024-03-06T07:22:23.008405+0000","last_clean":"2024-03-06T07:22:23.008405+0000","last_became_active":"2024-03-06T07:22:06.317298+0000","last_became_peered":"2024-03-06T07:22:06.317298+0000","last_unstale":"2024-03-06T07:22:23.008405+0000","last_undegraded":"2024-03-06T07:22:23.008405+0000","last_fullsized":"2024-03-06T07:22:23.008405+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:45.397748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719247+0000","last_change":"2024-03-06T07:22:21.725231+0000","last_active":"2024-03-06T07:22:22.719247+0000","last_peered":"2024-03-06T07:22:22.719247+0000","last_clean":"2024-03-06T07:22:22.719247+0000","last_became_active":"2024-03-06T07:22:21.724959+0000","last_became_peered":"2024-03-06T07:22:21.724959+0000","last_unstale":"2024-03-06T07:22:22.719247+0000","last_undegraded":"2024-03-06T07:22:22.719247+0000","last_fullsized":"2024-03-06T07:22:22.719247+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:11.190593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008143+0000","last_change":"2024-03-06T07:22:06.323812+0000","last_active":"2024-03-06T07:22:23.008143+0000","last_peered":"2024-03-06T07:22:23.008143+0000","last_clean":"2024-03-06T07:22:23.008143+0000","last_became_active":"2024-03-06T07:22:06.323558+0000","last_became_peered":"2024-03-06T07:22:06.323558+0000","last_unstale":"2024-03-06T07:22:23.008143+0000","last_undegraded":"2024-03-06T07:22:23.008143+0000","last_fullsized":"2024-03-06T07:22:23.008143+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:37.954594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705600+0000","last_change":"2024-03-06T07:22:21.727129+0000","last_active":"2024-03-06T07:22:22.705600+0000","last_peered":"2024-03-06T07:22:22.705600+0000","last_clean":"2024-03-06T07:22:22.705600+0000","last_became_active":"2024-03-06T07:22:21.726856+0000","last_became_peered":"2024-03-06T07:22:21.726856+0000","last_unstale":"2024-03-06T07:22:22.705600+0000","last_undegraded":"2024-03-06T07:22:22.705600+0000","last_fullsized":"2024-03-06T07:22:22.705600+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:10.041200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006875+0000","last_change":"2024-03-06T07:22:06.322447+0000","last_active":"2024-03-06T07:22:23.006875+0000","last_peered":"2024-03-06T07:22:23.006875+0000","last_clean":"2024-03-06T07:22:23.006875+0000","last_became_active":"2024-03-06T07:22:06.318369+0000","last_became_peered":"2024-03-06T07:22:06.318369+0000","last_unstale":"2024-03-06T07:22:23.006875+0000","last_undegraded":"2024-03-06T07:22:23.006875+0000","last_fullsized":"2024-03-06T07:22:23.006875+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:50.150401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006916+0000","last_change":"2024-03-06T07:22:21.720925+0000","last_active":"2024-03-06T07:22:23.006916+0000","last_peered":"2024-03-06T07:22:23.006916+0000","last_clean":"2024-03-06T07:22:23.006916+0000","last_became_active":"2024-03-06T07:22:21.720681+0000","last_became_peered":"2024-03-06T07:22:21.720681+0000","last_unstale":"2024-03-06T07:22:23.006916+0000","last_undegraded":"2024-03-06T07:22:23.006916+0000","last_fullsized":"2024-03-06T07:22:23.006916+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:55.798503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705194+0000","last_change":"2024-03-06T07:22:06.328376+0000","last_active":"2024-03-06T07:22:22.705194+0000","last_peered":"2024-03-06T07:22:22.705194+0000","last_clean":"2024-03-06T07:22:22.705194+0000","last_became_active":"2024-03-06T07:22:06.327042+0000","last_became_peered":"2024-03-06T07:22:06.327042+0000","last_unstale":"2024-03-06T07:22:22.705194+0000","last_undegraded":"2024-03-06T07:22:22.705194+0000","last_fullsized":"2024-03-06T07:22:22.705194+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:34.670786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008331+0000","last_change":"2024-03-06T07:22:21.721756+0000","last_active":"2024-03-06T07:22:23.008331+0000","last_peered":"2024-03-06T07:22:23.008331+0000","last_clean":"2024-03-06T07:22:23.008331+0000","last_became_active":"2024-03-06T07:22:21.721498+0000","last_became_peered":"2024-03-06T07:22:21.721498+0000","last_unstale":"2024-03-06T07:22:23.008331+0000","last_undegraded":"2024-03-06T07:22:23.008331+0000","last_fullsized":"2024-03-06T07:22:23.008331+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:02.370302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705869+0000","last_change":"2024-03-06T07:22:06.335876+0000","last_active":"2024-03-06T07:22:22.705869+0000","last_peered":"2024-03-06T07:22:22.705869+0000","last_clean":"2024-03-06T07:22:22.705869+0000","last_became_active":"2024-03-06T07:22:06.335575+0000","last_became_peered":"2024-03-06T07:22:06.335575+0000","last_unstale":"2024-03-06T07:22:22.705869+0000","last_undegraded":"2024-03-06T07:22:22.705869+0000","last_fullsized":"2024-03-06T07:22:22.705869+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:19.402946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719507+0000","last_change":"2024-03-06T07:22:21.724127+0000","last_active":"2024-03-06T07:22:22.719507+0000","last_peered":"2024-03-06T07:22:22.719507+0000","last_clean":"2024-03-06T07:22:22.719507+0000","last_became_active":"2024-03-06T07:22:21.723801+0000","last_became_peered":"2024-03-06T07:22:21.723801+0000","last_unstale":"2024-03-06T07:22:22.719507+0000","last_undegraded":"2024-03-06T07:22:22.719507+0000","last_fullsized":"2024-03-06T07:22:22.719507+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:28.340044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007778+0000","last_change":"2024-03-06T07:22:06.320896+0000","last_active":"2024-03-06T07:22:23.007778+0000","last_peered":"2024-03-06T07:22:23.007778+0000","last_clean":"2024-03-06T07:22:23.007778+0000","last_became_active":"2024-03-06T07:22:06.320643+0000","last_became_peered":"2024-03-06T07:22:06.320643+0000","last_unstale":"2024-03-06T07:22:23.007778+0000","last_undegraded":"2024-03-06T07:22:23.007778+0000","last_fullsized":"2024-03-06T07:22:23.007778+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:09.367201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008906+0000","last_change":"2024-03-06T07:22:21.725159+0000","last_active":"2024-03-06T07:22:23.008906+0000","last_peered":"2024-03-06T07:22:23.008906+0000","last_clean":"2024-03-06T07:22:23.008906+0000","last_became_active":"2024-03-06T07:22:21.724897+0000","last_became_peered":"2024-03-06T07:22:21.724897+0000","last_unstale":"2024-03-06T07:22:23.008906+0000","last_undegraded":"2024-03-06T07:22:23.008906+0000","last_fullsized":"2024-03-06T07:22:23.008906+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:53.042875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.704526+0000","last_change":"2024-03-06T07:22:06.322171+0000","last_active":"2024-03-06T07:22:22.704526+0000","last_peered":"2024-03-06T07:22:22.704526+0000","last_clean":"2024-03-06T07:22:22.704526+0000","last_became_active":"2024-03-06T07:22:06.321891+0000","last_became_peered":"2024-03-06T07:22:06.321891+0000","last_unstale":"2024-03-06T07:22:22.704526+0000","last_undegraded":"2024-03-06T07:22:22.704526+0000","last_fullsized":"2024-03-06T07:22:22.704526+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:02.261930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008176+0000","last_change":"2024-03-06T07:22:21.722673+0000","last_active":"2024-03-06T07:22:23.008176+0000","last_peered":"2024-03-06T07:22:23.008176+0000","last_clean":"2024-03-06T07:22:23.008176+0000","last_became_active":"2024-03-06T07:22:21.722424+0000","last_became_peered":"2024-03-06T07:22:21.722424+0000","last_unstale":"2024-03-06T07:22:23.008176+0000","last_undegraded":"2024-03-06T07:22:23.008176+0000","last_fullsized":"2024-03-06T07:22:23.008176+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:08.186982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705909+0000","last_change":"2024-03-06T07:22:06.321590+0000","last_active":"2024-03-06T07:22:22.705909+0000","last_peered":"2024-03-06T07:22:22.705909+0000","last_clean":"2024-03-06T07:22:22.705909+0000","last_became_active":"2024-03-06T07:22:06.321334+0000","last_became_peered":"2024-03-06T07:22:06.321334+0000","last_unstale":"2024-03-06T07:22:22.705909+0000","last_undegraded":"2024-03-06T07:22:22.705909+0000","last_fullsized":"2024-03-06T07:22:22.705909+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:49.931180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007904+0000","last_change":"2024-03-06T07:22:21.726103+0000","last_active":"2024-03-06T07:22:23.007904+0000","last_peered":"2024-03-06T07:22:23.007904+0000","last_clean":"2024-03-06T07:22:23.007904+0000","last_became_active":"2024-03-06T07:22:21.723373+0000","last_became_peered":"2024-03-06T07:22:21.723373+0000","last_unstale":"2024-03-06T07:22:23.007904+0000","last_undegraded":"2024-03-06T07:22:23.007904+0000","last_fullsized":"2024-03-06T07:22:23.007904+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:28.340044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007337+0000","last_change":"2024-03-06T07:22:06.328682+0000","last_active":"2024-03-06T07:22:23.007337+0000","last_peered":"2024-03-06T07:22:23.007337+0000","last_clean":"2024-03-06T07:22:23.007337+0000","last_became_active":"2024-03-06T07:22:06.328418+0000","last_became_peered":"2024-03-06T07:22:06.328418+0000","last_unstale":"2024-03-06T07:22:23.007337+0000","last_undegraded":"2024-03-06T07:22:23.007337+0000","last_fullsized":"2024-03-06T07:22:23.007337+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:52.638347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.703876+0000","last_change":"2024-03-06T07:22:21.718432+0000","last_active":"2024-03-06T07:22:22.703876+0000","last_peered":"2024-03-06T07:22:22.703876+0000","last_clean":"2024-03-06T07:22:22.703876+0000","last_became_active":"2024-03-06T07:22:21.718193+0000","last_became_peered":"2024-03-06T07:22:21.718193+0000","last_unstale":"2024-03-06T07:22:22.703876+0000","last_undegraded":"2024-03-06T07:22:22.703876+0000","last_fullsized":"2024-03-06T07:22:22.703876+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:47.455410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007482+0000","last_change":"2024-03-06T07:22:06.330122+0000","last_active":"2024-03-06T07:22:23.007482+0000","last_peered":"2024-03-06T07:22:23.007482+0000","last_clean":"2024-03-06T07:22:23.007482+0000","last_became_active":"2024-03-06T07:22:06.329780+0000","last_became_peered":"2024-03-06T07:22:06.329780+0000","last_unstale":"2024-03-06T07:22:23.007482+0000","last_undegraded":"2024-03-06T07:22:23.007482+0000","last_fullsized":"2024-03-06T07:22:23.007482+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:38.898151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719695+0000","last_change":"2024-03-06T07:22:21.730693+0000","last_active":"2024-03-06T07:22:22.719695+0000","last_peered":"2024-03-06T07:22:22.719695+0000","last_clean":"2024-03-06T07:22:22.719695+0000","last_became_active":"2024-03-06T07:22:21.730389+0000","last_became_peered":"2024-03-06T07:22:21.730389+0000","last_unstale":"2024-03-06T07:22:22.719695+0000","last_undegraded":"2024-03-06T07:22:22.719695+0000","last_fullsized":"2024-03-06T07:22:22.719695+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:08.186982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008024+0000","last_change":"2024-03-06T07:22:06.321523+0000","last_active":"2024-03-06T07:22:23.008024+0000","last_peered":"2024-03-06T07:22:23.008024+0000","last_clean":"2024-03-06T07:22:23.008024+0000","last_became_active":"2024-03-06T07:22:06.321224+0000","last_became_peered":"2024-03-06T07:22:06.321224+0000","last_unstale":"2024-03-06T07:22:23.008024+0000","last_undegraded":"2024-03-06T07:22:23.008024+0000","last_fullsized":"2024-03-06T07:22:23.008024+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:05.731939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007761+0000","last_change":"2024-03-06T07:22:21.720005+0000","last_active":"2024-03-06T07:22:23.007761+0000","last_peered":"2024-03-06T07:22:23.007761+0000","last_clean":"2024-03-06T07:22:23.007761+0000","last_became_active":"2024-03-06T07:22:21.719590+0000","last_became_peered":"2024-03-06T07:22:21.719590+0000","last_unstale":"2024-03-06T07:22:23.007761+0000","last_undegraded":"2024-03-06T07:22:23.007761+0000","last_fullsized":"2024-03-06T07:22:23.007761+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:02.330255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006573+0000","last_change":"2024-03-06T07:22:06.314914+0000","last_active":"2024-03-06T07:22:23.006573+0000","last_peered":"2024-03-06T07:22:23.006573+0000","last_clean":"2024-03-06T07:22:23.006573+0000","last_became_active":"2024-03-06T07:22:06.314654+0000","last_became_peered":"2024-03-06T07:22:06.314654+0000","last_unstale":"2024-03-06T07:22:23.006573+0000","last_undegraded":"2024-03-06T07:22:23.006573+0000","last_fullsized":"2024-03-06T07:22:23.006573+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:18.848063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706093+0000","last_change":"2024-03-06T07:22:21.721024+0000","last_active":"2024-03-06T07:22:22.706093+0000","last_peered":"2024-03-06T07:22:22.706093+0000","last_clean":"2024-03-06T07:22:22.706093+0000","last_became_active":"2024-03-06T07:22:21.720521+0000","last_became_peered":"2024-03-06T07:22:21.720521+0000","last_unstale":"2024-03-06T07:22:22.706093+0000","last_undegraded":"2024-03-06T07:22:22.706093+0000","last_fullsized":"2024-03-06T07:22:22.706093+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:32.095545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007773+0000","last_change":"2024-03-06T07:22:06.323974+0000","last_active":"2024-03-06T07:22:23.007773+0000","last_peered":"2024-03-06T07:22:23.007773+0000","last_clean":"2024-03-06T07:22:23.007773+0000","last_became_active":"2024-03-06T07:22:06.323718+0000","last_became_peered":"2024-03-06T07:22:06.323718+0000","last_unstale":"2024-03-06T07:22:23.007773+0000","last_undegraded":"2024-03-06T07:22:23.007773+0000","last_fullsized":"2024-03-06T07:22:23.007773+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:46.981658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705796+0000","last_change":"2024-03-06T07:22:21.714896+0000","last_active":"2024-03-06T07:22:22.705796+0000","last_peered":"2024-03-06T07:22:22.705796+0000","last_clean":"2024-03-06T07:22:22.705796+0000","last_became_active":"2024-03-06T07:22:21.714647+0000","last_became_peered":"2024-03-06T07:22:21.714647+0000","last_unstale":"2024-03-06T07:22:22.705796+0000","last_undegraded":"2024-03-06T07:22:22.705796+0000","last_fullsized":"2024-03-06T07:22:22.705796+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:51.486696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008604+0000","last_change":"2024-03-06T07:22:06.325388+0000","last_active":"2024-03-06T07:22:23.008604+0000","last_peered":"2024-03-06T07:22:23.008604+0000","last_clean":"2024-03-06T07:22:23.008604+0000","last_became_active":"2024-03-06T07:22:06.325109+0000","last_became_peered":"2024-03-06T07:22:06.325109+0000","last_unstale":"2024-03-06T07:22:23.008604+0000","last_undegraded":"2024-03-06T07:22:23.008604+0000","last_fullsized":"2024-03-06T07:22:23.008604+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:51.508838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007352+0000","last_change":"2024-03-06T07:22:21.724840+0000","last_active":"2024-03-06T07:22:23.007352+0000","last_peered":"2024-03-06T07:22:23.007352+0000","last_clean":"2024-03-06T07:22:23.007352+0000","last_became_active":"2024-03-06T07:22:21.724372+0000","last_became_peered":"2024-03-06T07:22:21.724372+0000","last_unstale":"2024-03-06T07:22:23.007352+0000","last_undegraded":"2024-03-06T07:22:23.007352+0000","last_fullsized":"2024-03-06T07:22:23.007352+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:30.701781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007605+0000","last_change":"2024-03-06T07:22:06.329771+0000","last_active":"2024-03-06T07:22:23.007605+0000","last_peered":"2024-03-06T07:22:23.007605+0000","last_clean":"2024-03-06T07:22:23.007605+0000","last_became_active":"2024-03-06T07:22:06.318483+0000","last_became_peered":"2024-03-06T07:22:06.318483+0000","last_unstale":"2024-03-06T07:22:23.007605+0000","last_undegraded":"2024-03-06T07:22:23.007605+0000","last_fullsized":"2024-03-06T07:22:23.007605+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:17.568813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008197+0000","last_change":"2024-03-06T07:22:21.716259+0000","last_active":"2024-03-06T07:22:23.008197+0000","last_peered":"2024-03-06T07:22:23.008197+0000","last_clean":"2024-03-06T07:22:23.008197+0000","last_became_active":"2024-03-06T07:22:21.715977+0000","last_became_peered":"2024-03-06T07:22:21.715977+0000","last_unstale":"2024-03-06T07:22:23.008197+0000","last_undegraded":"2024-03-06T07:22:23.008197+0000","last_fullsized":"2024-03-06T07:22:23.008197+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:25.216603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705889+0000","last_change":"2024-03-06T07:22:06.317746+0000","last_active":"2024-03-06T07:22:22.705889+0000","last_peered":"2024-03-06T07:22:22.705889+0000","last_clean":"2024-03-06T07:22:22.705889+0000","last_became_active":"2024-03-06T07:22:06.317465+0000","last_became_peered":"2024-03-06T07:22:06.317465+0000","last_unstale":"2024-03-06T07:22:22.705889+0000","last_undegraded":"2024-03-06T07:22:22.705889+0000","last_fullsized":"2024-03-06T07:22:22.705889+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:07.308296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705635+0000","last_change":"2024-03-06T07:22:21.724121+0000","last_active":"2024-03-06T07:22:22.705635+0000","last_peered":"2024-03-06T07:22:22.705635+0000","last_clean":"2024-03-06T07:22:22.705635+0000","last_became_active":"2024-03-06T07:22:21.723843+0000","last_became_peered":"2024-03-06T07:22:21.723843+0000","last_unstale":"2024-03-06T07:22:22.705635+0000","last_undegraded":"2024-03-06T07:22:22.705635+0000","last_fullsized":"2024-03-06T07:22:22.705635+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:19.770698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705935+0000","last_change":"2024-03-06T07:22:06.322670+0000","last_active":"2024-03-06T07:22:22.705935+0000","last_peered":"2024-03-06T07:22:22.705935+0000","last_clean":"2024-03-06T07:22:22.705935+0000","last_became_active":"2024-03-06T07:22:06.322382+0000","last_became_peered":"2024-03-06T07:22:06.322382+0000","last_unstale":"2024-03-06T07:22:22.705935+0000","last_undegraded":"2024-03-06T07:22:22.705935+0000","last_fullsized":"2024-03-06T07:22:22.705935+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:02.940410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007334+0000","last_change":"2024-03-06T07:22:21.719094+0000","last_active":"2024-03-06T07:22:23.007334+0000","last_peered":"2024-03-06T07:22:23.007334+0000","last_clean":"2024-03-06T07:22:23.007334+0000","last_became_active":"2024-03-06T07:22:21.718802+0000","last_became_peered":"2024-03-06T07:22:21.718802+0000","last_unstale":"2024-03-06T07:22:23.007334+0000","last_undegraded":"2024-03-06T07:22:23.007334+0000","last_fullsized":"2024-03-06T07:22:23.007334+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:11.190593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705581+0000","last_change":"2024-03-06T07:22:06.331711+0000","last_active":"2024-03-06T07:22:22.705581+0000","last_peered":"2024-03-06T07:22:22.705581+0000","last_clean":"2024-03-06T07:22:22.705581+0000","last_became_active":"2024-03-06T07:22:06.326618+0000","last_became_peered":"2024-03-06T07:22:06.326618+0000","last_unstale":"2024-03-06T07:22:22.705581+0000","last_undegraded":"2024-03-06T07:22:22.705581+0000","last_fullsized":"2024-03-06T07:22:22.705581+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:15.334381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719743+0000","last_change":"2024-03-06T07:22:21.724603+0000","last_active":"2024-03-06T07:22:22.719743+0000","last_peered":"2024-03-06T07:22:22.719743+0000","last_clean":"2024-03-06T07:22:22.719743+0000","last_became_active":"2024-03-06T07:22:21.724259+0000","last_became_peered":"2024-03-06T07:22:21.724259+0000","last_unstale":"2024-03-06T07:22:22.719743+0000","last_undegraded":"2024-03-06T07:22:22.719743+0000","last_fullsized":"2024-03-06T07:22:22.719743+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:17.698053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.704202+0000","last_change":"2024-03-06T07:22:06.318320+0000","last_active":"2024-03-06T07:22:22.704202+0000","last_peered":"2024-03-06T07:22:22.704202+0000","last_clean":"2024-03-06T07:22:22.704202+0000","last_became_active":"2024-03-06T07:22:06.318065+0000","last_became_peered":"2024-03-06T07:22:06.318065+0000","last_unstale":"2024-03-06T07:22:22.704202+0000","last_undegraded":"2024-03-06T07:22:22.704202+0000","last_fullsized":"2024-03-06T07:22:22.704202+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:00.505615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705681+0000","last_change":"2024-03-06T07:22:21.719457+0000","last_active":"2024-03-06T07:22:22.705681+0000","last_peered":"2024-03-06T07:22:22.705681+0000","last_clean":"2024-03-06T07:22:22.705681+0000","last_became_active":"2024-03-06T07:22:21.719177+0000","last_became_peered":"2024-03-06T07:22:21.719177+0000","last_unstale":"2024-03-06T07:22:22.705681+0000","last_undegraded":"2024-03-06T07:22:22.705681+0000","last_fullsized":"2024-03-06T07:22:22.705681+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:54.543771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006682+0000","last_change":"2024-03-06T07:22:06.318170+0000","last_active":"2024-03-06T07:22:23.006682+0000","last_peered":"2024-03-06T07:22:23.006682+0000","last_clean":"2024-03-06T07:22:23.006682+0000","last_became_active":"2024-03-06T07:22:06.317903+0000","last_became_peered":"2024-03-06T07:22:06.317903+0000","last_unstale":"2024-03-06T07:22:23.006682+0000","last_undegraded":"2024-03-06T07:22:23.006682+0000","last_fullsized":"2024-03-06T07:22:23.006682+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:25:11.151255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007891+0000","last_change":"2024-03-06T07:22:21.722788+0000","last_active":"2024-03-06T07:22:23.007891+0000","last_peered":"2024-03-06T07:22:23.007891+0000","last_clean":"2024-03-06T07:22:23.007891+0000","last_became_active":"2024-03-06T07:22:21.722511+0000","last_became_peered":"2024-03-06T07:22:21.722511+0000","last_unstale":"2024-03-06T07:22:23.007891+0000","last_undegraded":"2024-03-06T07:22:23.007891+0000","last_fullsized":"2024-03-06T07:22:23.007891+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:30.603319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007863+0000","last_change":"2024-03-06T07:22:06.333955+0000","last_active":"2024-03-06T07:22:23.007863+0000","last_peered":"2024-03-06T07:22:23.007863+0000","last_clean":"2024-03-06T07:22:23.007863+0000","last_became_active":"2024-03-06T07:22:06.333703+0000","last_became_peered":"2024-03-06T07:22:06.333703+0000","last_unstale":"2024-03-06T07:22:23.007863+0000","last_undegraded":"2024-03-06T07:22:23.007863+0000","last_fullsized":"2024-03-06T07:22:23.007863+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:34.670786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006861+0000","last_change":"2024-03-06T07:22:21.714703+0000","last_active":"2024-03-06T07:22:23.006861+0000","last_peered":"2024-03-06T07:22:23.006861+0000","last_clean":"2024-03-06T07:22:23.006861+0000","last_became_active":"2024-03-06T07:22:21.714435+0000","last_became_peered":"2024-03-06T07:22:21.714435+0000","last_unstale":"2024-03-06T07:22:23.006861+0000","last_undegraded":"2024-03-06T07:22:23.006861+0000","last_fullsized":"2024-03-06T07:22:23.006861+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:03.612944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705558+0000","last_change":"2024-03-06T07:22:06.320349+0000","last_active":"2024-03-06T07:22:22.705558+0000","last_peered":"2024-03-06T07:22:22.705558+0000","last_clean":"2024-03-06T07:22:22.705558+0000","last_became_active":"2024-03-06T07:22:06.320094+0000","last_became_peered":"2024-03-06T07:22:06.320094+0000","last_unstale":"2024-03-06T07:22:22.705558+0000","last_undegraded":"2024-03-06T07:22:22.705558+0000","last_fullsized":"2024-03-06T07:22:22.705558+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:48.224301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719832+0000","last_change":"2024-03-06T07:22:21.720164+0000","last_active":"2024-03-06T07:22:22.719832+0000","last_peered":"2024-03-06T07:22:22.719832+0000","last_clean":"2024-03-06T07:22:22.719832+0000","last_became_active":"2024-03-06T07:22:21.719753+0000","last_became_peered":"2024-03-06T07:22:21.719753+0000","last_unstale":"2024-03-06T07:22:22.719832+0000","last_undegraded":"2024-03-06T07:22:22.719832+0000","last_fullsized":"2024-03-06T07:22:22.719832+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:30.960027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008240+0000","last_change":"2024-03-06T07:22:06.319987+0000","last_active":"2024-03-06T07:22:23.008240+0000","last_peered":"2024-03-06T07:22:23.008240+0000","last_clean":"2024-03-06T07:22:23.008240+0000","last_became_active":"2024-03-06T07:22:06.319703+0000","last_became_peered":"2024-03-06T07:22:06.319703+0000","last_unstale":"2024-03-06T07:22:23.008240+0000","last_undegraded":"2024-03-06T07:22:23.008240+0000","last_fullsized":"2024-03-06T07:22:23.008240+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:02.558815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719894+0000","last_change":"2024-03-06T07:22:21.716584+0000","last_active":"2024-03-06T07:22:22.719894+0000","last_peered":"2024-03-06T07:22:22.719894+0000","last_clean":"2024-03-06T07:22:22.719894+0000","last_became_active":"2024-03-06T07:22:21.716250+0000","last_became_peered":"2024-03-06T07:22:21.716250+0000","last_unstale":"2024-03-06T07:22:22.719894+0000","last_undegraded":"2024-03-06T07:22:22.719894+0000","last_fullsized":"2024-03-06T07:22:22.719894+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:53.042875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705929+0000","last_change":"2024-03-06T07:22:06.318435+0000","last_active":"2024-03-06T07:22:22.705929+0000","last_peered":"2024-03-06T07:22:22.705929+0000","last_clean":"2024-03-06T07:22:22.705929+0000","last_became_active":"2024-03-06T07:22:06.318147+0000","last_became_peered":"2024-03-06T07:22:06.318147+0000","last_unstale":"2024-03-06T07:22:22.705929+0000","last_undegraded":"2024-03-06T07:22:22.705929+0000","last_fullsized":"2024-03-06T07:22:22.705929+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:20.858614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007096+0000","last_change":"2024-03-06T07:22:21.725146+0000","last_active":"2024-03-06T07:22:23.007096+0000","last_peered":"2024-03-06T07:22:23.007096+0000","last_clean":"2024-03-06T07:22:23.007096+0000","last_became_active":"2024-03-06T07:22:21.723713+0000","last_became_peered":"2024-03-06T07:22:21.723713+0000","last_unstale":"2024-03-06T07:22:23.007096+0000","last_undegraded":"2024-03-06T07:22:23.007096+0000","last_fullsized":"2024-03-06T07:22:23.007096+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:34.109293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006649+0000","last_change":"2024-03-06T07:22:06.335154+0000","last_active":"2024-03-06T07:22:23.006649+0000","last_peered":"2024-03-06T07:22:23.006649+0000","last_clean":"2024-03-06T07:22:23.006649+0000","last_became_active":"2024-03-06T07:22:06.334855+0000","last_became_peered":"2024-03-06T07:22:06.334855+0000","last_unstale":"2024-03-06T07:22:23.006649+0000","last_undegraded":"2024-03-06T07:22:23.006649+0000","last_fullsized":"2024-03-06T07:22:23.006649+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:54.652557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006572+0000","last_change":"2024-03-06T07:22:21.716747+0000","last_active":"2024-03-06T07:22:23.006572+0000","last_peered":"2024-03-06T07:22:23.006572+0000","last_clean":"2024-03-06T07:22:23.006572+0000","last_became_active":"2024-03-06T07:22:21.716102+0000","last_became_peered":"2024-03-06T07:22:21.716102+0000","last_unstale":"2024-03-06T07:22:23.006572+0000","last_undegraded":"2024-03-06T07:22:23.006572+0000","last_fullsized":"2024-03-06T07:22:23.006572+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:05.317140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007948+0000","last_change":"2024-03-06T07:22:06.330141+0000","last_active":"2024-03-06T07:22:23.007948+0000","last_peered":"2024-03-06T07:22:23.007948+0000","last_clean":"2024-03-06T07:22:23.007948+0000","last_became_active":"2024-03-06T07:22:06.329487+0000","last_became_peered":"2024-03-06T07:22:06.329487+0000","last_unstale":"2024-03-06T07:22:23.007948+0000","last_undegraded":"2024-03-06T07:22:23.007948+0000","last_fullsized":"2024-03-06T07:22:23.007948+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:09.367201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007606+0000","last_change":"2024-03-06T07:22:21.720384+0000","last_active":"2024-03-06T07:22:23.007606+0000","last_peered":"2024-03-06T07:22:23.007606+0000","last_clean":"2024-03-06T07:22:23.007606+0000","last_became_active":"2024-03-06T07:22:21.719261+0000","last_became_peered":"2024-03-06T07:22:21.719261+0000","last_unstale":"2024-03-06T07:22:23.007606+0000","last_undegraded":"2024-03-06T07:22:23.007606+0000","last_fullsized":"2024-03-06T07:22:23.007606+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:17.353502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007135+0000","last_change":"2024-03-06T07:22:06.322218+0000","last_active":"2024-03-06T07:22:23.007135+0000","last_peered":"2024-03-06T07:22:23.007135+0000","last_clean":"2024-03-06T07:22:23.007135+0000","last_became_active":"2024-03-06T07:22:06.318026+0000","last_became_peered":"2024-03-06T07:22:06.318026+0000","last_unstale":"2024-03-06T07:22:23.007135+0000","last_undegraded":"2024-03-06T07:22:23.007135+0000","last_fullsized":"2024-03-06T07:22:23.007135+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:16.809640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719387+0000","last_change":"2024-03-06T07:22:21.729828+0000","last_active":"2024-03-06T07:22:22.719387+0000","last_peered":"2024-03-06T07:22:22.719387+0000","last_clean":"2024-03-06T07:22:22.719387+0000","last_became_active":"2024-03-06T07:22:21.729562+0000","last_became_peered":"2024-03-06T07:22:21.729562+0000","last_unstale":"2024-03-06T07:22:22.719387+0000","last_undegraded":"2024-03-06T07:22:22.719387+0000","last_fullsized":"2024-03-06T07:22:22.719387+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:02.330255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719197+0000","last_change":"2024-03-06T07:22:06.322067+0000","last_active":"2024-03-06T07:22:22.719197+0000","last_peered":"2024-03-06T07:22:22.719197+0000","last_clean":"2024-03-06T07:22:22.719197+0000","last_became_active":"2024-03-06T07:22:06.321797+0000","last_became_peered":"2024-03-06T07:22:06.321797+0000","last_unstale":"2024-03-06T07:22:22.719197+0000","last_undegraded":"2024-03-06T07:22:22.719197+0000","last_fullsized":"2024-03-06T07:22:22.719197+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:36.139439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705442+0000","last_change":"2024-03-06T07:22:21.720183+0000","last_active":"2024-03-06T07:22:22.705442+0000","last_peered":"2024-03-06T07:22:22.705442+0000","last_clean":"2024-03-06T07:22:22.705442+0000","last_became_active":"2024-03-06T07:22:21.719913+0000","last_became_peered":"2024-03-06T07:22:21.719913+0000","last_unstale":"2024-03-06T07:22:22.705442+0000","last_undegraded":"2024-03-06T07:22:22.705442+0000","last_fullsized":"2024-03-06T07:22:22.705442+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:25:26.539898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007467+0000","last_change":"2024-03-06T07:22:20.702488+0000","last_active":"2024-03-06T07:22:23.007467+0000","last_peered":"2024-03-06T07:22:23.007467+0000","last_clean":"2024-03-06T07:22:23.007467+0000","last_became_active":"2024-03-06T07:22:20.702195+0000","last_became_peered":"2024-03-06T07:22:20.702195+0000","last_unstale":"2024-03-06T07:22:23.007467+0000","last_undegraded":"2024-03-06T07:22:23.007467+0000","last_fullsized":"2024-03-06T07:22:23.007467+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:48.668680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007341+0000","last_change":"2024-03-06T07:22:06.320372+0000","last_active":"2024-03-06T07:22:23.007341+0000","last_peered":"2024-03-06T07:22:23.007341+0000","last_clean":"2024-03-06T07:22:23.007341+0000","last_became_active":"2024-03-06T07:22:06.320109+0000","last_became_peered":"2024-03-06T07:22:06.320109+0000","last_unstale":"2024-03-06T07:22:23.007341+0000","last_undegraded":"2024-03-06T07:22:23.007341+0000","last_fullsized":"2024-03-06T07:22:23.007341+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:37.954594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":26,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706128+0000","last_change":"2024-03-06T07:22:05.317240+0000","last_active":"2024-03-06T07:22:22.706128+0000","last_peered":"2024-03-06T07:22:22.706128+0000","last_clean":"2024-03-06T07:22:22.706128+0000","last_became_active":"2024-03-06T07:22:05.316908+0000","last_became_peered":"2024-03-06T07:22:05.316908+0000","last_unstale":"2024-03-06T07:22:22.706128+0000","last_undegraded":"2024-03-06T07:22:22.706128+0000","last_fullsized":"2024-03-06T07:22:22.706128+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:18.512707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007611+0000","last_change":"2024-03-06T07:22:21.714837+0000","last_active":"2024-03-06T07:22:23.007611+0000","last_peered":"2024-03-06T07:22:23.007611+0000","last_clean":"2024-03-06T07:22:23.007611+0000","last_became_active":"2024-03-06T07:22:21.714547+0000","last_became_peered":"2024-03-06T07:22:21.714547+0000","last_unstale":"2024-03-06T07:22:23.007611+0000","last_undegraded":"2024-03-06T07:22:23.007611+0000","last_fullsized":"2024-03-06T07:22:23.007611+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:52.725112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"55'8","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006888+0000","last_change":"2024-03-06T07:22:20.708393+0000","last_active":"2024-03-06T07:22:23.006888+0000","last_peered":"2024-03-06T07:22:23.006888+0000","last_clean":"2024-03-06T07:22:23.006888+0000","last_became_active":"2024-03-06T07:22:20.708115+0000","last_became_peered":"2024-03-06T07:22:20.708115+0000","last_unstale":"2024-03-06T07:22:23.006888+0000","last_undegraded":"2024-03-06T07:22:23.006888+0000","last_fullsized":"2024-03-06T07:22:23.006888+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:05.892802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719439+0000","last_change":"2024-03-06T07:22:06.324306+0000","last_active":"2024-03-06T07:22:22.719439+0000","last_peered":"2024-03-06T07:22:22.719439+0000","last_clean":"2024-03-06T07:22:22.719439+0000","last_became_active":"2024-03-06T07:22:06.323979+0000","last_became_peered":"2024-03-06T07:22:06.323979+0000","last_unstale":"2024-03-06T07:22:22.719439+0000","last_undegraded":"2024-03-06T07:22:22.719439+0000","last_fullsized":"2024-03-06T07:22:22.719439+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:05.731939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007869+0000","last_change":"2024-03-06T07:22:05.321228+0000","last_active":"2024-03-06T07:22:23.007869+0000","last_peered":"2024-03-06T07:22:23.007869+0000","last_clean":"2024-03-06T07:22:23.007869+0000","last_became_active":"2024-03-06T07:22:05.320958+0000","last_became_peered":"2024-03-06T07:22:05.320958+0000","last_unstale":"2024-03-06T07:22:23.007869+0000","last_undegraded":"2024-03-06T07:22:23.007869+0000","last_fullsized":"2024-03-06T07:22:23.007869+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:43.288752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007035+0000","last_change":"2024-03-06T07:22:21.723577+0000","last_active":"2024-03-06T07:22:23.007035+0000","last_peered":"2024-03-06T07:22:23.007035+0000","last_clean":"2024-03-06T07:22:23.007035+0000","last_became_active":"2024-03-06T07:22:21.723299+0000","last_became_peered":"2024-03-06T07:22:21.723299+0000","last_unstale":"2024-03-06T07:22:23.007035+0000","last_undegraded":"2024-03-06T07:22:23.007035+0000","last_fullsized":"2024-03-06T07:22:23.007035+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:32.957457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008792+0000","last_change":"2024-03-06T07:22:20.709772+0000","last_active":"2024-03-06T07:22:23.008792+0000","last_peered":"2024-03-06T07:22:23.008792+0000","last_clean":"2024-03-06T07:22:23.008792+0000","last_became_active":"2024-03-06T07:22:20.709341+0000","last_became_peered":"2024-03-06T07:22:20.709341+0000","last_unstale":"2024-03-06T07:22:23.008792+0000","last_undegraded":"2024-03-06T07:22:23.008792+0000","last_fullsized":"2024-03-06T07:22:23.008792+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:29.598640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705539+0000","last_change":"2024-03-06T07:22:06.323135+0000","last_active":"2024-03-06T07:22:22.705539+0000","last_peered":"2024-03-06T07:22:22.705539+0000","last_clean":"2024-03-06T07:22:22.705539+0000","last_became_active":"2024-03-06T07:22:06.322873+0000","last_became_peered":"2024-03-06T07:22:06.322873+0000","last_unstale":"2024-03-06T07:22:22.705539+0000","last_undegraded":"2024-03-06T07:22:22.705539+0000","last_fullsized":"2024-03-06T07:22:22.705539+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:32.712920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008278+0000","last_change":"2024-03-06T07:22:05.321069+0000","last_active":"2024-03-06T07:22:23.008278+0000","last_peered":"2024-03-06T07:22:23.008278+0000","last_clean":"2024-03-06T07:22:23.008278+0000","last_became_active":"2024-03-06T07:22:05.320722+0000","last_became_peered":"2024-03-06T07:22:05.320722+0000","last_unstale":"2024-03-06T07:22:23.008278+0000","last_undegraded":"2024-03-06T07:22:23.008278+0000","last_fullsized":"2024-03-06T07:22:23.008278+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:26.446709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719258+0000","last_change":"2024-03-06T07:22:21.720273+0000","last_active":"2024-03-06T07:22:22.719258+0000","last_peered":"2024-03-06T07:22:22.719258+0000","last_clean":"2024-03-06T07:22:22.719258+0000","last_became_active":"2024-03-06T07:22:21.719605+0000","last_became_peered":"2024-03-06T07:22:21.719605+0000","last_unstale":"2024-03-06T07:22:22.719258+0000","last_undegraded":"2024-03-06T07:22:22.719258+0000","last_fullsized":"2024-03-06T07:22:22.719258+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:17.353502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719195+0000","last_change":"2024-03-06T07:22:20.705361+0000","last_active":"2024-03-06T07:22:22.719195+0000","last_peered":"2024-03-06T07:22:22.719195+0000","last_clean":"2024-03-06T07:22:22.719195+0000","last_became_active":"2024-03-06T07:22:20.705057+0000","last_became_peered":"2024-03-06T07:22:20.705057+0000","last_unstale":"2024-03-06T07:22:22.719195+0000","last_undegraded":"2024-03-06T07:22:22.719195+0000","last_fullsized":"2024-03-06T07:22:22.719195+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:29.598640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705979+0000","last_change":"2024-03-06T07:22:06.318876+0000","last_active":"2024-03-06T07:22:22.705979+0000","last_peered":"2024-03-06T07:22:22.705979+0000","last_clean":"2024-03-06T07:22:22.705979+0000","last_became_active":"2024-03-06T07:22:06.318574+0000","last_became_peered":"2024-03-06T07:22:06.318574+0000","last_unstale":"2024-03-06T07:22:22.705979+0000","last_undegraded":"2024-03-06T07:22:22.705979+0000","last_fullsized":"2024-03-06T07:22:22.705979+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:34.284715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719634+0000","last_change":"2024-03-06T07:22:05.316288+0000","last_active":"2024-03-06T07:22:22.719634+0000","last_peered":"2024-03-06T07:22:22.719634+0000","last_clean":"2024-03-06T07:22:22.719634+0000","last_became_active":"2024-03-06T07:22:05.315965+0000","last_became_peered":"2024-03-06T07:22:05.315965+0000","last_unstale":"2024-03-06T07:22:22.719634+0000","last_undegraded":"2024-03-06T07:22:22.719634+0000","last_fullsized":"2024-03-06T07:22:22.719634+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:19.545421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706511+0000","last_change":"2024-03-06T07:22:21.726430+0000","last_active":"2024-03-06T07:22:22.706511+0000","last_peered":"2024-03-06T07:22:22.706511+0000","last_clean":"2024-03-06T07:22:22.706511+0000","last_became_active":"2024-03-06T07:22:21.726174+0000","last_became_peered":"2024-03-06T07:22:21.726174+0000","last_unstale":"2024-03-06T07:22:22.706511+0000","last_undegraded":"2024-03-06T07:22:22.706511+0000","last_fullsized":"2024-03-06T07:22:22.706511+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:30.723025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"55'3","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008508+0000","last_change":"2024-03-06T07:22:20.706738+0000","last_active":"2024-03-06T07:22:23.008508+0000","last_peered":"2024-03-06T07:22:23.008508+0000","last_clean":"2024-03-06T07:22:23.008508+0000","last_became_active":"2024-03-06T07:22:20.706116+0000","last_became_peered":"2024-03-06T07:22:20.706116+0000","last_unstale":"2024-03-06T07:22:23.008508+0000","last_undegraded":"2024-03-06T07:22:23.008508+0000","last_fullsized":"2024-03-06T07:22:23.008508+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:49.054750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008930+0000","last_change":"2024-03-06T07:22:06.319384+0000","last_active":"2024-03-06T07:22:23.008930+0000","last_peered":"2024-03-06T07:22:23.008930+0000","last_clean":"2024-03-06T07:22:23.008930+0000","last_became_active":"2024-03-06T07:22:06.318885+0000","last_became_peered":"2024-03-06T07:22:06.318885+0000","last_unstale":"2024-03-06T07:22:23.008930+0000","last_undegraded":"2024-03-06T07:22:23.008930+0000","last_fullsized":"2024-03-06T07:22:23.008930+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:05.680579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006890+0000","last_change":"2024-03-06T07:22:05.319423+0000","last_active":"2024-03-06T07:22:23.006890+0000","last_peered":"2024-03-06T07:22:23.006890+0000","last_clean":"2024-03-06T07:22:23.006890+0000","last_became_active":"2024-03-06T07:22:05.319055+0000","last_became_peered":"2024-03-06T07:22:05.319055+0000","last_unstale":"2024-03-06T07:22:23.006890+0000","last_undegraded":"2024-03-06T07:22:23.006890+0000","last_fullsized":"2024-03-06T07:22:23.006890+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:19.853691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008178+0000","last_change":"2024-03-06T07:22:21.723296+0000","last_active":"2024-03-06T07:22:23.008178+0000","last_peered":"2024-03-06T07:22:23.008178+0000","last_clean":"2024-03-06T07:22:23.008178+0000","last_became_active":"2024-03-06T07:22:21.722406+0000","last_became_peered":"2024-03-06T07:22:21.722406+0000","last_unstale":"2024-03-06T07:22:23.008178+0000","last_undegraded":"2024-03-06T07:22:23.008178+0000","last_fullsized":"2024-03-06T07:22:23.008178+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:49.673358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"55'6","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007117+0000","last_change":"2024-03-06T07:22:20.710366+0000","last_active":"2024-03-06T07:22:23.007117+0000","last_peered":"2024-03-06T07:22:23.007117+0000","last_clean":"2024-03-06T07:22:23.007117+0000","last_became_active":"2024-03-06T07:22:20.709863+0000","last_became_peered":"2024-03-06T07:22:20.709863+0000","last_unstale":"2024-03-06T07:22:23.007117+0000","last_undegraded":"2024-03-06T07:22:23.007117+0000","last_fullsized":"2024-03-06T07:22:23.007117+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:31.090866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008053+0000","last_change":"2024-03-06T07:22:06.330770+0000","last_active":"2024-03-06T07:22:23.008053+0000","last_peered":"2024-03-06T07:22:23.008053+0000","last_clean":"2024-03-06T07:22:23.008053+0000","last_became_active":"2024-03-06T07:22:06.330404+0000","last_became_peered":"2024-03-06T07:22:06.330404+0000","last_unstale":"2024-03-06T07:22:23.008053+0000","last_undegraded":"2024-03-06T07:22:23.008053+0000","last_fullsized":"2024-03-06T07:22:23.008053+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:36.139439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007658+0000","last_change":"2024-03-06T07:22:05.313908+0000","last_active":"2024-03-06T07:22:23.007658+0000","last_peered":"2024-03-06T07:22:23.007658+0000","last_clean":"2024-03-06T07:22:23.007658+0000","last_became_active":"2024-03-06T07:22:05.313638+0000","last_became_peered":"2024-03-06T07:22:05.313638+0000","last_unstale":"2024-03-06T07:22:23.007658+0000","last_undegraded":"2024-03-06T07:22:23.007658+0000","last_fullsized":"2024-03-06T07:22:23.007658+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:46.701129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006995+0000","last_change":"2024-03-06T07:22:21.713645+0000","last_active":"2024-03-06T07:22:23.006995+0000","last_peered":"2024-03-06T07:22:23.006995+0000","last_clean":"2024-03-06T07:22:23.006995+0000","last_became_active":"2024-03-06T07:22:21.713387+0000","last_became_peered":"2024-03-06T07:22:21.713387+0000","last_unstale":"2024-03-06T07:22:23.006995+0000","last_undegraded":"2024-03-06T07:22:23.006995+0000","last_fullsized":"2024-03-06T07:22:23.006995+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:04.969935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008879+0000","last_change":"2024-03-06T07:22:20.707352+0000","last_active":"2024-03-06T07:22:23.008879+0000","last_peered":"2024-03-06T07:22:23.008879+0000","last_clean":"2024-03-06T07:22:23.008879+0000","last_became_active":"2024-03-06T07:22:20.706288+0000","last_became_peered":"2024-03-06T07:22:20.706288+0000","last_unstale":"2024-03-06T07:22:23.008879+0000","last_undegraded":"2024-03-06T07:22:23.008879+0000","last_fullsized":"2024-03-06T07:22:23.008879+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:04.315144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705659+0000","last_change":"2024-03-06T07:22:06.320596+0000","last_active":"2024-03-06T07:22:22.705659+0000","last_peered":"2024-03-06T07:22:22.705659+0000","last_clean":"2024-03-06T07:22:22.705659+0000","last_became_active":"2024-03-06T07:22:06.319430+0000","last_became_peered":"2024-03-06T07:22:06.319430+0000","last_unstale":"2024-03-06T07:22:22.705659+0000","last_undegraded":"2024-03-06T07:22:22.705659+0000","last_fullsized":"2024-03-06T07:22:22.705659+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:49.581292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705499+0000","last_change":"2024-03-06T07:22:05.320922+0000","last_active":"2024-03-06T07:22:22.705499+0000","last_peered":"2024-03-06T07:22:22.705499+0000","last_clean":"2024-03-06T07:22:22.705499+0000","last_became_active":"2024-03-06T07:22:05.320638+0000","last_became_peered":"2024-03-06T07:22:05.320638+0000","last_unstale":"2024-03-06T07:22:22.705499+0000","last_undegraded":"2024-03-06T07:22:22.705499+0000","last_fullsized":"2024-03-06T07:22:22.705499+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:13.798736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705751+0000","last_change":"2024-03-06T07:22:21.726275+0000","last_active":"2024-03-06T07:22:22.705751+0000","last_peered":"2024-03-06T07:22:22.705751+0000","last_clean":"2024-03-06T07:22:22.705751+0000","last_became_active":"2024-03-06T07:22:21.726006+0000","last_became_peered":"2024-03-06T07:22:21.726006+0000","last_unstale":"2024-03-06T07:22:22.705751+0000","last_undegraded":"2024-03-06T07:22:22.705751+0000","last_fullsized":"2024-03-06T07:22:22.705751+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:54.164947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008025+0000","last_change":"2024-03-06T07:22:20.706922+0000","last_active":"2024-03-06T07:22:23.008025+0000","last_peered":"2024-03-06T07:22:23.008025+0000","last_clean":"2024-03-06T07:22:23.008025+0000","last_became_active":"2024-03-06T07:22:20.706248+0000","last_became_peered":"2024-03-06T07:22:20.706248+0000","last_unstale":"2024-03-06T07:22:23.008025+0000","last_undegraded":"2024-03-06T07:22:23.008025+0000","last_fullsized":"2024-03-06T07:22:23.008025+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:16.942779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.009065+0000","last_change":"2024-03-06T07:22:06.335202+0000","last_active":"2024-03-06T07:22:23.009065+0000","last_peered":"2024-03-06T07:22:23.009065+0000","last_clean":"2024-03-06T07:22:23.009065+0000","last_became_active":"2024-03-06T07:22:06.334922+0000","last_became_peered":"2024-03-06T07:22:06.334922+0000","last_unstale":"2024-03-06T07:22:23.009065+0000","last_undegraded":"2024-03-06T07:22:23.009065+0000","last_fullsized":"2024-03-06T07:22:23.009065+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:01.204740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006707+0000","last_change":"2024-03-06T07:22:05.326648+0000","last_active":"2024-03-06T07:22:23.006707+0000","last_peered":"2024-03-06T07:22:23.006707+0000","last_clean":"2024-03-06T07:22:23.006707+0000","last_became_active":"2024-03-06T07:22:05.326423+0000","last_became_peered":"2024-03-06T07:22:05.326423+0000","last_unstale":"2024-03-06T07:22:23.006707+0000","last_undegraded":"2024-03-06T07:22:23.006707+0000","last_fullsized":"2024-03-06T07:22:23.006707+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:04.675657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705762+0000","last_change":"2024-03-06T07:22:21.722486+0000","last_active":"2024-03-06T07:22:22.705762+0000","last_peered":"2024-03-06T07:22:22.705762+0000","last_clean":"2024-03-06T07:22:22.705762+0000","last_became_active":"2024-03-06T07:22:21.721809+0000","last_became_peered":"2024-03-06T07:22:21.721809+0000","last_unstale":"2024-03-06T07:22:22.705762+0000","last_undegraded":"2024-03-06T07:22:22.705762+0000","last_fullsized":"2024-03-06T07:22:22.705762+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:05.539044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705205+0000","last_change":"2024-03-06T07:22:20.709212+0000","last_active":"2024-03-06T07:22:22.705205+0000","last_peered":"2024-03-06T07:22:22.705205+0000","last_clean":"2024-03-06T07:22:22.705205+0000","last_became_active":"2024-03-06T07:22:20.708913+0000","last_became_peered":"2024-03-06T07:22:20.708913+0000","last_unstale":"2024-03-06T07:22:22.705205+0000","last_undegraded":"2024-03-06T07:22:22.705205+0000","last_fullsized":"2024-03-06T07:22:22.705205+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:52.038196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007099+0000","last_change":"2024-03-06T07:22:06.326561+0000","last_active":"2024-03-06T07:22:23.007099+0000","last_peered":"2024-03-06T07:22:23.007099+0000","last_clean":"2024-03-06T07:22:23.007099+0000","last_became_active":"2024-03-06T07:22:06.326295+0000","last_became_peered":"2024-03-06T07:22:06.326295+0000","last_unstale":"2024-03-06T07:22:23.007099+0000","last_undegraded":"2024-03-06T07:22:23.007099+0000","last_fullsized":"2024-03-06T07:22:23.007099+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:36.941825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.009020+0000","last_change":"2024-03-06T07:22:05.319695+0000","last_active":"2024-03-06T07:22:23.009020+0000","last_peered":"2024-03-06T07:22:23.009020+0000","last_clean":"2024-03-06T07:22:23.009020+0000","last_became_active":"2024-03-06T07:22:05.319410+0000","last_became_peered":"2024-03-06T07:22:05.319410+0000","last_unstale":"2024-03-06T07:22:23.009020+0000","last_undegraded":"2024-03-06T07:22:23.009020+0000","last_fullsized":"2024-03-06T07:22:23.009020+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:06.303374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705769+0000","last_change":"2024-03-06T07:22:21.725276+0000","last_active":"2024-03-06T07:22:22.705769+0000","last_peered":"2024-03-06T07:22:22.705769+0000","last_clean":"2024-03-06T07:22:22.705769+0000","last_became_active":"2024-03-06T07:22:21.725019+0000","last_became_peered":"2024-03-06T07:22:21.725019+0000","last_unstale":"2024-03-06T07:22:22.705769+0000","last_undegraded":"2024-03-06T07:22:22.705769+0000","last_fullsized":"2024-03-06T07:22:22.705769+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:04.407350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705244+0000","last_change":"2024-03-06T07:22:20.708398+0000","last_active":"2024-03-06T07:22:22.705244+0000","last_peered":"2024-03-06T07:22:22.705244+0000","last_clean":"2024-03-06T07:22:22.705244+0000","last_became_active":"2024-03-06T07:22:20.708105+0000","last_became_peered":"2024-03-06T07:22:20.708105+0000","last_unstale":"2024-03-06T07:22:22.705244+0000","last_undegraded":"2024-03-06T07:22:22.705244+0000","last_fullsized":"2024-03-06T07:22:22.705244+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:33.232028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719373+0000","last_change":"2024-03-06T07:22:06.314348+0000","last_active":"2024-03-06T07:22:22.719373+0000","last_peered":"2024-03-06T07:22:22.719373+0000","last_clean":"2024-03-06T07:22:22.719373+0000","last_became_active":"2024-03-06T07:22:06.314044+0000","last_became_peered":"2024-03-06T07:22:06.314044+0000","last_unstale":"2024-03-06T07:22:22.719373+0000","last_undegraded":"2024-03-06T07:22:22.719373+0000","last_fullsized":"2024-03-06T07:22:22.719373+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:02.558815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008323+0000","last_change":"2024-03-06T07:22:05.321049+0000","last_active":"2024-03-06T07:22:23.008323+0000","last_peered":"2024-03-06T07:22:23.008323+0000","last_clean":"2024-03-06T07:22:23.008323+0000","last_became_active":"2024-03-06T07:22:05.320696+0000","last_became_peered":"2024-03-06T07:22:05.320696+0000","last_unstale":"2024-03-06T07:22:23.008323+0000","last_undegraded":"2024-03-06T07:22:23.008323+0000","last_fullsized":"2024-03-06T07:22:23.008323+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:26.446709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.703804+0000","last_change":"2024-03-06T07:22:21.713747+0000","last_active":"2024-03-06T07:22:22.703804+0000","last_peered":"2024-03-06T07:22:22.703804+0000","last_clean":"2024-03-06T07:22:22.703804+0000","last_became_active":"2024-03-06T07:22:21.713490+0000","last_became_peered":"2024-03-06T07:22:21.713490+0000","last_unstale":"2024-03-06T07:22:22.703804+0000","last_undegraded":"2024-03-06T07:22:22.703804+0000","last_fullsized":"2024-03-06T07:22:22.703804+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:30.192301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705353+0000","last_change":"2024-03-06T07:22:20.702222+0000","last_active":"2024-03-06T07:22:22.705353+0000","last_peered":"2024-03-06T07:22:22.705353+0000","last_clean":"2024-03-06T07:22:22.705353+0000","last_became_active":"2024-03-06T07:22:20.701653+0000","last_became_peered":"2024-03-06T07:22:20.701653+0000","last_unstale":"2024-03-06T07:22:22.705353+0000","last_undegraded":"2024-03-06T07:22:22.705353+0000","last_fullsized":"2024-03-06T07:22:22.705353+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:04.312461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007570+0000","last_change":"2024-03-06T07:22:06.330920+0000","last_active":"2024-03-06T07:22:23.007570+0000","last_peered":"2024-03-06T07:22:23.007570+0000","last_clean":"2024-03-06T07:22:23.007570+0000","last_became_active":"2024-03-06T07:22:06.330331+0000","last_became_peered":"2024-03-06T07:22:06.330331+0000","last_unstale":"2024-03-06T07:22:23.007570+0000","last_undegraded":"2024-03-06T07:22:23.007570+0000","last_fullsized":"2024-03-06T07:22:23.007570+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:18.720650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":25,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008329+0000","last_change":"2024-03-06T07:22:05.321995+0000","last_active":"2024-03-06T07:22:23.008329+0000","last_peered":"2024-03-06T07:22:23.008329+0000","last_clean":"2024-03-06T07:22:23.008329+0000","last_became_active":"2024-03-06T07:22:05.321734+0000","last_became_peered":"2024-03-06T07:22:05.321734+0000","last_unstale":"2024-03-06T07:22:23.008329+0000","last_undegraded":"2024-03-06T07:22:23.008329+0000","last_fullsized":"2024-03-06T07:22:23.008329+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:40.725817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006969+0000","last_change":"2024-03-06T07:22:21.723978+0000","last_active":"2024-03-06T07:22:23.006969+0000","last_peered":"2024-03-06T07:22:23.006969+0000","last_clean":"2024-03-06T07:22:23.006969+0000","last_became_active":"2024-03-06T07:22:21.723647+0000","last_became_peered":"2024-03-06T07:22:21.723647+0000","last_unstale":"2024-03-06T07:22:23.006969+0000","last_undegraded":"2024-03-06T07:22:23.006969+0000","last_fullsized":"2024-03-06T07:22:23.006969+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:34.791590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705387+0000","last_change":"2024-03-06T07:22:20.702276+0000","last_active":"2024-03-06T07:22:22.705387+0000","last_peered":"2024-03-06T07:22:22.705387+0000","last_clean":"2024-03-06T07:22:22.705387+0000","last_became_active":"2024-03-06T07:22:20.701967+0000","last_became_peered":"2024-03-06T07:22:20.701967+0000","last_unstale":"2024-03-06T07:22:22.705387+0000","last_undegraded":"2024-03-06T07:22:22.705387+0000","last_fullsized":"2024-03-06T07:22:22.705387+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:16.693374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008489+0000","last_change":"2024-03-06T07:22:06.308003+0000","last_active":"2024-03-06T07:22:23.008489+0000","last_peered":"2024-03-06T07:22:23.008489+0000","last_clean":"2024-03-06T07:22:23.008489+0000","last_became_active":"2024-03-06T07:22:06.307696+0000","last_became_peered":"2024-03-06T07:22:06.307696+0000","last_unstale":"2024-03-06T07:22:23.008489+0000","last_undegraded":"2024-03-06T07:22:23.008489+0000","last_fullsized":"2024-03-06T07:22:23.008489+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:20.550343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":28,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705948+0000","last_change":"2024-03-06T07:22:05.313388+0000","last_active":"2024-03-06T07:22:22.705948+0000","last_peered":"2024-03-06T07:22:22.705948+0000","last_clean":"2024-03-06T07:22:22.705948+0000","last_became_active":"2024-03-06T07:22:05.312146+0000","last_became_peered":"2024-03-06T07:22:05.312146+0000","last_unstale":"2024-03-06T07:22:22.705948+0000","last_undegraded":"2024-03-06T07:22:22.705948+0000","last_fullsized":"2024-03-06T07:22:22.705948+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:16.630462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007976+0000","last_change":"2024-03-06T07:22:21.715675+0000","last_active":"2024-03-06T07:22:23.007976+0000","last_peered":"2024-03-06T07:22:23.007976+0000","last_clean":"2024-03-06T07:22:23.007976+0000","last_became_active":"2024-03-06T07:22:21.715423+0000","last_became_peered":"2024-03-06T07:22:21.715423+0000","last_unstale":"2024-03-06T07:22:23.007976+0000","last_undegraded":"2024-03-06T07:22:23.007976+0000","last_fullsized":"2024-03-06T07:22:23.007976+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:12.856641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007292+0000","last_change":"2024-03-06T07:22:20.697808+0000","last_active":"2024-03-06T07:22:23.007292+0000","last_peered":"2024-03-06T07:22:23.007292+0000","last_clean":"2024-03-06T07:22:23.007292+0000","last_became_active":"2024-03-06T07:22:20.697175+0000","last_became_peered":"2024-03-06T07:22:20.697175+0000","last_unstale":"2024-03-06T07:22:23.007292+0000","last_undegraded":"2024-03-06T07:22:23.007292+0000","last_fullsized":"2024-03-06T07:22:23.007292+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:10.185915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719353+0000","last_change":"2024-03-06T07:22:06.324830+0000","last_active":"2024-03-06T07:22:22.719353+0000","last_peered":"2024-03-06T07:22:22.719353+0000","last_clean":"2024-03-06T07:22:22.719353+0000","last_became_active":"2024-03-06T07:22:06.324574+0000","last_became_peered":"2024-03-06T07:22:06.324574+0000","last_unstale":"2024-03-06T07:22:22.719353+0000","last_undegraded":"2024-03-06T07:22:22.719353+0000","last_fullsized":"2024-03-06T07:22:22.719353+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:22.443979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006926+0000","last_change":"2024-03-06T07:22:05.326946+0000","last_active":"2024-03-06T07:22:23.006926+0000","last_peered":"2024-03-06T07:22:23.006926+0000","last_clean":"2024-03-06T07:22:23.006926+0000","last_became_active":"2024-03-06T07:22:05.326675+0000","last_became_peered":"2024-03-06T07:22:05.326675+0000","last_unstale":"2024-03-06T07:22:23.006926+0000","last_undegraded":"2024-03-06T07:22:23.006926+0000","last_fullsized":"2024-03-06T07:22:23.006926+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:35.936903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008727+0000","last_change":"2024-03-06T07:22:21.723312+0000","last_active":"2024-03-06T07:22:23.008727+0000","last_peered":"2024-03-06T07:22:23.008727+0000","last_clean":"2024-03-06T07:22:23.008727+0000","last_became_active":"2024-03-06T07:22:21.723046+0000","last_became_peered":"2024-03-06T07:22:21.723046+0000","last_unstale":"2024-03-06T07:22:23.008727+0000","last_undegraded":"2024-03-06T07:22:23.008727+0000","last_fullsized":"2024-03-06T07:22:23.008727+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:30.960027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007205+0000","last_change":"2024-03-06T07:22:20.701291+0000","last_active":"2024-03-06T07:22:23.007205+0000","last_peered":"2024-03-06T07:22:23.007205+0000","last_clean":"2024-03-06T07:22:23.007205+0000","last_became_active":"2024-03-06T07:22:20.701012+0000","last_became_peered":"2024-03-06T07:22:20.701012+0000","last_unstale":"2024-03-06T07:22:23.007205+0000","last_undegraded":"2024-03-06T07:22:23.007205+0000","last_fullsized":"2024-03-06T07:22:23.007205+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:01.325576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007690+0000","last_change":"2024-03-06T07:22:06.329315+0000","last_active":"2024-03-06T07:22:23.007690+0000","last_peered":"2024-03-06T07:22:23.007690+0000","last_clean":"2024-03-06T07:22:23.007690+0000","last_became_active":"2024-03-06T07:22:06.322815+0000","last_became_peered":"2024-03-06T07:22:06.322815+0000","last_unstale":"2024-03-06T07:22:23.007690+0000","last_undegraded":"2024-03-06T07:22:23.007690+0000","last_fullsized":"2024-03-06T07:22:23.007690+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:15.214675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706031+0000","last_change":"2024-03-06T07:22:05.324367+0000","last_active":"2024-03-06T07:22:22.706031+0000","last_peered":"2024-03-06T07:22:22.706031+0000","last_clean":"2024-03-06T07:22:22.706031+0000","last_became_active":"2024-03-06T07:22:05.324086+0000","last_became_peered":"2024-03-06T07:22:05.324086+0000","last_unstale":"2024-03-06T07:22:22.706031+0000","last_undegraded":"2024-03-06T07:22:22.706031+0000","last_fullsized":"2024-03-06T07:22:22.706031+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:31.061844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705866+0000","last_change":"2024-03-06T07:22:21.725366+0000","last_active":"2024-03-06T07:22:22.705866+0000","last_peered":"2024-03-06T07:22:22.705866+0000","last_clean":"2024-03-06T07:22:22.705866+0000","last_became_active":"2024-03-06T07:22:21.725118+0000","last_became_peered":"2024-03-06T07:22:21.725118+0000","last_unstale":"2024-03-06T07:22:22.705866+0000","last_undegraded":"2024-03-06T07:22:22.705866+0000","last_fullsized":"2024-03-06T07:22:22.705866+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:32.198283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"55'4","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007459+0000","last_change":"2024-03-06T07:22:20.710921+0000","last_active":"2024-03-06T07:22:23.007459+0000","last_peered":"2024-03-06T07:22:23.007459+0000","last_clean":"2024-03-06T07:22:23.007459+0000","last_became_active":"2024-03-06T07:22:20.710660+0000","last_became_peered":"2024-03-06T07:22:20.710660+0000","last_unstale":"2024-03-06T07:22:23.007459+0000","last_undegraded":"2024-03-06T07:22:23.007459+0000","last_fullsized":"2024-03-06T07:22:23.007459+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:53.282115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705575+0000","last_change":"2024-03-06T07:22:06.322371+0000","last_active":"2024-03-06T07:22:22.705575+0000","last_peered":"2024-03-06T07:22:22.705575+0000","last_clean":"2024-03-06T07:22:22.705575+0000","last_became_active":"2024-03-06T07:22:06.322099+0000","last_became_peered":"2024-03-06T07:22:06.322099+0000","last_unstale":"2024-03-06T07:22:22.705575+0000","last_undegraded":"2024-03-06T07:22:22.705575+0000","last_fullsized":"2024-03-06T07:22:22.705575+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:02.237473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008220+0000","last_change":"2024-03-06T07:22:05.320305+0000","last_active":"2024-03-06T07:22:23.008220+0000","last_peered":"2024-03-06T07:22:23.008220+0000","last_clean":"2024-03-06T07:22:23.008220+0000","last_became_active":"2024-03-06T07:22:05.319970+0000","last_became_peered":"2024-03-06T07:22:05.319970+0000","last_unstale":"2024-03-06T07:22:23.008220+0000","last_undegraded":"2024-03-06T07:22:23.008220+0000","last_fullsized":"2024-03-06T07:22:23.008220+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:01.935488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008091+0000","last_change":"2024-03-06T07:22:21.723887+0000","last_active":"2024-03-06T07:22:23.008091+0000","last_peered":"2024-03-06T07:22:23.008091+0000","last_clean":"2024-03-06T07:22:23.008091+0000","last_became_active":"2024-03-06T07:22:21.723616+0000","last_became_peered":"2024-03-06T07:22:21.723616+0000","last_unstale":"2024-03-06T07:22:23.008091+0000","last_undegraded":"2024-03-06T07:22:23.008091+0000","last_fullsized":"2024-03-06T07:22:23.008091+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:30.723025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007950+0000","last_change":"2024-03-06T07:22:20.697649+0000","last_active":"2024-03-06T07:22:23.007950+0000","last_peered":"2024-03-06T07:22:23.007950+0000","last_clean":"2024-03-06T07:22:23.007950+0000","last_became_active":"2024-03-06T07:22:20.697385+0000","last_became_peered":"2024-03-06T07:22:20.697385+0000","last_unstale":"2024-03-06T07:22:23.007950+0000","last_undegraded":"2024-03-06T07:22:23.007950+0000","last_fullsized":"2024-03-06T07:22:23.007950+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:07.182303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007617+0000","last_change":"2024-03-06T07:22:06.318628+0000","last_active":"2024-03-06T07:22:23.007617+0000","last_peered":"2024-03-06T07:22:23.007617+0000","last_clean":"2024-03-06T07:22:23.007617+0000","last_became_active":"2024-03-06T07:22:06.318324+0000","last_became_peered":"2024-03-06T07:22:06.318324+0000","last_unstale":"2024-03-06T07:22:23.007617+0000","last_undegraded":"2024-03-06T07:22:23.007617+0000","last_fullsized":"2024-03-06T07:22:23.007617+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:09.827959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":30,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705348+0000","last_change":"2024-03-06T07:22:05.316578+0000","last_active":"2024-03-06T07:22:22.705348+0000","last_peered":"2024-03-06T07:22:22.705348+0000","last_clean":"2024-03-06T07:22:22.705348+0000","last_became_active":"2024-03-06T07:22:05.316285+0000","last_became_peered":"2024-03-06T07:22:05.316285+0000","last_unstale":"2024-03-06T07:22:22.705348+0000","last_undegraded":"2024-03-06T07:22:22.705348+0000","last_fullsized":"2024-03-06T07:22:22.705348+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:21.439057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007868+0000","last_change":"2024-03-06T07:22:21.716524+0000","last_active":"2024-03-06T07:22:23.007868+0000","last_peered":"2024-03-06T07:22:23.007868+0000","last_clean":"2024-03-06T07:22:23.007868+0000","last_became_active":"2024-03-06T07:22:21.716242+0000","last_became_peered":"2024-03-06T07:22:21.716242+0000","last_unstale":"2024-03-06T07:22:23.007868+0000","last_undegraded":"2024-03-06T07:22:23.007868+0000","last_fullsized":"2024-03-06T07:22:23.007868+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:13:16.916071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705302+0000","last_change":"2024-03-06T07:22:20.706114+0000","last_active":"2024-03-06T07:22:22.705302+0000","last_peered":"2024-03-06T07:22:22.705302+0000","last_clean":"2024-03-06T07:22:22.705302+0000","last_became_active":"2024-03-06T07:22:20.705831+0000","last_became_peered":"2024-03-06T07:22:20.705831+0000","last_unstale":"2024-03-06T07:22:22.705302+0000","last_undegraded":"2024-03-06T07:22:22.705302+0000","last_fullsized":"2024-03-06T07:22:22.705302+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:28.208506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705286+0000","last_change":"2024-03-06T07:22:06.328590+0000","last_active":"2024-03-06T07:22:22.705286+0000","last_peered":"2024-03-06T07:22:22.705286+0000","last_clean":"2024-03-06T07:22:22.705286+0000","last_became_active":"2024-03-06T07:22:06.327594+0000","last_became_peered":"2024-03-06T07:22:06.327594+0000","last_unstale":"2024-03-06T07:22:22.705286+0000","last_undegraded":"2024-03-06T07:22:22.705286+0000","last_fullsized":"2024-03-06T07:22:22.705286+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:46.981658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007956+0000","last_change":"2024-03-06T07:22:05.325528+0000","last_active":"2024-03-06T07:22:23.007956+0000","last_peered":"2024-03-06T07:22:23.007956+0000","last_clean":"2024-03-06T07:22:23.007956+0000","last_became_active":"2024-03-06T07:22:05.325254+0000","last_became_peered":"2024-03-06T07:22:05.325254+0000","last_unstale":"2024-03-06T07:22:23.007956+0000","last_undegraded":"2024-03-06T07:22:23.007956+0000","last_fullsized":"2024-03-06T07:22:23.007956+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:44.392826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.703958+0000","last_change":"2024-03-06T07:22:21.719104+0000","last_active":"2024-03-06T07:22:22.703958+0000","last_peered":"2024-03-06T07:22:22.703958+0000","last_clean":"2024-03-06T07:22:22.703958+0000","last_became_active":"2024-03-06T07:22:21.718824+0000","last_became_peered":"2024-03-06T07:22:21.718824+0000","last_unstale":"2024-03-06T07:22:22.703958+0000","last_undegraded":"2024-03-06T07:22:22.703958+0000","last_fullsized":"2024-03-06T07:22:22.703958+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:34.906272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006548+0000","last_change":"2024-03-06T07:22:20.709142+0000","last_active":"2024-03-06T07:22:23.006548+0000","last_peered":"2024-03-06T07:22:23.006548+0000","last_clean":"2024-03-06T07:22:23.006548+0000","last_became_active":"2024-03-06T07:22:20.708835+0000","last_became_peered":"2024-03-06T07:22:20.708835+0000","last_unstale":"2024-03-06T07:22:23.006548+0000","last_undegraded":"2024-03-06T07:22:23.006548+0000","last_fullsized":"2024-03-06T07:22:23.006548+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:31.193605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007028+0000","last_change":"2024-03-06T07:22:06.320394+0000","last_active":"2024-03-06T07:22:23.007028+0000","last_peered":"2024-03-06T07:22:23.007028+0000","last_clean":"2024-03-06T07:22:23.007028+0000","last_became_active":"2024-03-06T07:22:06.320135+0000","last_became_peered":"2024-03-06T07:22:06.320135+0000","last_unstale":"2024-03-06T07:22:23.007028+0000","last_undegraded":"2024-03-06T07:22:23.007028+0000","last_fullsized":"2024-03-06T07:22:23.007028+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:34.766315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.704708+0000","last_change":"2024-03-06T07:22:05.314843+0000","last_active":"2024-03-06T07:22:22.704708+0000","last_peered":"2024-03-06T07:22:22.704708+0000","last_clean":"2024-03-06T07:22:22.704708+0000","last_became_active":"2024-03-06T07:22:05.314567+0000","last_became_peered":"2024-03-06T07:22:05.314567+0000","last_unstale":"2024-03-06T07:22:22.704708+0000","last_undegraded":"2024-03-06T07:22:22.704708+0000","last_fullsized":"2024-03-06T07:22:22.704708+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:23.891871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705345+0000","last_change":"2024-03-06T07:22:21.722602+0000","last_active":"2024-03-06T07:22:22.705345+0000","last_peered":"2024-03-06T07:22:22.705345+0000","last_clean":"2024-03-06T07:22:22.705345+0000","last_became_active":"2024-03-06T07:22:21.722323+0000","last_became_peered":"2024-03-06T07:22:21.722323+0000","last_unstale":"2024-03-06T07:22:22.705345+0000","last_undegraded":"2024-03-06T07:22:22.705345+0000","last_fullsized":"2024-03-06T07:22:22.705345+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:05.288867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008914+0000","last_change":"2024-03-06T07:22:20.703676+0000","last_active":"2024-03-06T07:22:23.008914+0000","last_peered":"2024-03-06T07:22:23.008914+0000","last_clean":"2024-03-06T07:22:23.008914+0000","last_became_active":"2024-03-06T07:22:20.703383+0000","last_became_peered":"2024-03-06T07:22:20.703383+0000","last_unstale":"2024-03-06T07:22:23.008914+0000","last_undegraded":"2024-03-06T07:22:23.008914+0000","last_fullsized":"2024-03-06T07:22:23.008914+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:29.718346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007174+0000","last_change":"2024-03-06T07:22:06.321026+0000","last_active":"2024-03-06T07:22:23.007174+0000","last_peered":"2024-03-06T07:22:23.007174+0000","last_clean":"2024-03-06T07:22:23.007174+0000","last_became_active":"2024-03-06T07:22:06.320749+0000","last_became_peered":"2024-03-06T07:22:06.320749+0000","last_unstale":"2024-03-06T07:22:23.007174+0000","last_undegraded":"2024-03-06T07:22:23.007174+0000","last_fullsized":"2024-03-06T07:22:23.007174+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:59.907792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706410+0000","last_change":"2024-03-06T07:22:05.325044+0000","last_active":"2024-03-06T07:22:22.706410+0000","last_peered":"2024-03-06T07:22:22.706410+0000","last_clean":"2024-03-06T07:22:22.706410+0000","last_became_active":"2024-03-06T07:22:05.324731+0000","last_became_peered":"2024-03-06T07:22:05.324731+0000","last_unstale":"2024-03-06T07:22:22.706410+0000","last_undegraded":"2024-03-06T07:22:22.706410+0000","last_fullsized":"2024-03-06T07:22:22.706410+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:50.594146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007073+0000","last_change":"2024-03-06T07:22:21.734429+0000","last_active":"2024-03-06T07:22:23.007073+0000","last_peered":"2024-03-06T07:22:23.007073+0000","last_clean":"2024-03-06T07:22:23.007073+0000","last_became_active":"2024-03-06T07:22:21.734156+0000","last_became_peered":"2024-03-06T07:22:21.734156+0000","last_unstale":"2024-03-06T07:22:23.007073+0000","last_undegraded":"2024-03-06T07:22:23.007073+0000","last_fullsized":"2024-03-06T07:22:23.007073+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:54.286794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705488+0000","last_change":"2024-03-06T07:22:20.703911+0000","last_active":"2024-03-06T07:22:22.705488+0000","last_peered":"2024-03-06T07:22:22.705488+0000","last_clean":"2024-03-06T07:22:22.705488+0000","last_became_active":"2024-03-06T07:22:20.703633+0000","last_became_peered":"2024-03-06T07:22:20.703633+0000","last_unstale":"2024-03-06T07:22:22.705488+0000","last_undegraded":"2024-03-06T07:22:22.705488+0000","last_fullsized":"2024-03-06T07:22:22.705488+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:29.955349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007427+0000","last_change":"2024-03-06T07:22:06.327972+0000","last_active":"2024-03-06T07:22:23.007427+0000","last_peered":"2024-03-06T07:22:23.007427+0000","last_clean":"2024-03-06T07:22:23.007427+0000","last_became_active":"2024-03-06T07:22:06.327676+0000","last_became_peered":"2024-03-06T07:22:06.327676+0000","last_unstale":"2024-03-06T07:22:23.007427+0000","last_undegraded":"2024-03-06T07:22:23.007427+0000","last_fullsized":"2024-03-06T07:22:23.007427+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:53.049705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706086+0000","last_change":"2024-03-06T07:22:05.318191+0000","last_active":"2024-03-06T07:22:22.706086+0000","last_peered":"2024-03-06T07:22:22.706086+0000","last_clean":"2024-03-06T07:22:22.706086+0000","last_became_active":"2024-03-06T07:22:05.317984+0000","last_became_peered":"2024-03-06T07:22:05.317984+0000","last_unstale":"2024-03-06T07:22:22.706086+0000","last_undegraded":"2024-03-06T07:22:22.706086+0000","last_fullsized":"2024-03-06T07:22:22.706086+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:48.457911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719680+0000","last_change":"2024-03-06T07:22:21.728826+0000","last_active":"2024-03-06T07:22:22.719680+0000","last_peered":"2024-03-06T07:22:22.719680+0000","last_clean":"2024-03-06T07:22:22.719680+0000","last_became_active":"2024-03-06T07:22:21.724267+0000","last_became_peered":"2024-03-06T07:22:21.724267+0000","last_unstale":"2024-03-06T07:22:22.719680+0000","last_undegraded":"2024-03-06T07:22:22.719680+0000","last_fullsized":"2024-03-06T07:22:22.719680+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:01.959177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007384+0000","last_change":"2024-03-06T07:22:20.698001+0000","last_active":"2024-03-06T07:22:23.007384+0000","last_peered":"2024-03-06T07:22:23.007384+0000","last_clean":"2024-03-06T07:22:23.007384+0000","last_became_active":"2024-03-06T07:22:20.697566+0000","last_became_peered":"2024-03-06T07:22:20.697566+0000","last_unstale":"2024-03-06T07:22:23.007384+0000","last_undegraded":"2024-03-06T07:22:23.007384+0000","last_fullsized":"2024-03-06T07:22:23.007384+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:16.348823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719488+0000","last_change":"2024-03-06T07:22:06.317413+0000","last_active":"2024-03-06T07:22:22.719488+0000","last_peered":"2024-03-06T07:22:22.719488+0000","last_clean":"2024-03-06T07:22:22.719488+0000","last_became_active":"2024-03-06T07:22:06.317126+0000","last_became_peered":"2024-03-06T07:22:06.317126+0000","last_unstale":"2024-03-06T07:22:22.719488+0000","last_undegraded":"2024-03-06T07:22:22.719488+0000","last_fullsized":"2024-03-06T07:22:22.719488+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:17.635384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007663+0000","last_change":"2024-03-06T07:22:05.320901+0000","last_active":"2024-03-06T07:22:23.007663+0000","last_peered":"2024-03-06T07:22:23.007663+0000","last_clean":"2024-03-06T07:22:23.007663+0000","last_became_active":"2024-03-06T07:22:05.320579+0000","last_became_peered":"2024-03-06T07:22:05.320579+0000","last_unstale":"2024-03-06T07:22:23.007663+0000","last_undegraded":"2024-03-06T07:22:23.007663+0000","last_fullsized":"2024-03-06T07:22:23.007663+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:46.701129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706359+0000","last_change":"2024-03-06T07:22:21.722508+0000","last_active":"2024-03-06T07:22:22.706359+0000","last_peered":"2024-03-06T07:22:22.706359+0000","last_clean":"2024-03-06T07:22:22.706359+0000","last_became_active":"2024-03-06T07:22:21.722215+0000","last_became_peered":"2024-03-06T07:22:21.722215+0000","last_unstale":"2024-03-06T07:22:22.706359+0000","last_undegraded":"2024-03-06T07:22:22.706359+0000","last_fullsized":"2024-03-06T07:22:22.706359+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:30.603319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007217+0000","last_change":"2024-03-06T07:22:20.709740+0000","last_active":"2024-03-06T07:22:23.007217+0000","last_peered":"2024-03-06T07:22:23.007217+0000","last_clean":"2024-03-06T07:22:23.007217+0000","last_became_active":"2024-03-06T07:22:20.709457+0000","last_became_peered":"2024-03-06T07:22:20.709457+0000","last_unstale":"2024-03-06T07:22:23.007217+0000","last_undegraded":"2024-03-06T07:22:23.007217+0000","last_fullsized":"2024-03-06T07:22:23.007217+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:31.952778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706231+0000","last_change":"2024-03-06T07:22:06.321030+0000","last_active":"2024-03-06T07:22:22.706231+0000","last_peered":"2024-03-06T07:22:22.706231+0000","last_clean":"2024-03-06T07:22:22.706231+0000","last_became_active":"2024-03-06T07:22:06.320686+0000","last_became_peered":"2024-03-06T07:22:06.320686+0000","last_unstale":"2024-03-06T07:22:22.706231+0000","last_undegraded":"2024-03-06T07:22:22.706231+0000","last_fullsized":"2024-03-06T07:22:22.706231+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:41.730739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705709+0000","last_change":"2024-03-06T07:22:05.316761+0000","last_active":"2024-03-06T07:22:22.705709+0000","last_peered":"2024-03-06T07:22:22.705709+0000","last_clean":"2024-03-06T07:22:22.705709+0000","last_became_active":"2024-03-06T07:22:05.316088+0000","last_became_peered":"2024-03-06T07:22:05.316088+0000","last_unstale":"2024-03-06T07:22:22.705709+0000","last_undegraded":"2024-03-06T07:22:22.705709+0000","last_fullsized":"2024-03-06T07:22:22.705709+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:35.134517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007153+0000","last_change":"2024-03-06T07:22:21.728348+0000","last_active":"2024-03-06T07:22:23.007153+0000","last_peered":"2024-03-06T07:22:23.007153+0000","last_clean":"2024-03-06T07:22:23.007153+0000","last_became_active":"2024-03-06T07:22:21.728078+0000","last_became_peered":"2024-03-06T07:22:21.728078+0000","last_unstale":"2024-03-06T07:22:23.007153+0000","last_undegraded":"2024-03-06T07:22:23.007153+0000","last_fullsized":"2024-03-06T07:22:23.007153+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:15.237287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719893+0000","last_change":"2024-03-06T07:22:20.700220+0000","last_active":"2024-03-06T07:22:22.719893+0000","last_peered":"2024-03-06T07:22:22.719893+0000","last_clean":"2024-03-06T07:22:22.719893+0000","last_became_active":"2024-03-06T07:22:20.699919+0000","last_became_peered":"2024-03-06T07:22:20.699919+0000","last_unstale":"2024-03-06T07:22:22.719893+0000","last_undegraded":"2024-03-06T07:22:22.719893+0000","last_fullsized":"2024-03-06T07:22:22.719893+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:01.365623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008143+0000","last_change":"2024-03-06T07:22:06.313796+0000","last_active":"2024-03-06T07:22:23.008143+0000","last_peered":"2024-03-06T07:22:23.008143+0000","last_clean":"2024-03-06T07:22:23.008143+0000","last_became_active":"2024-03-06T07:22:06.313534+0000","last_became_peered":"2024-03-06T07:22:06.313534+0000","last_unstale":"2024-03-06T07:22:23.008143+0000","last_undegraded":"2024-03-06T07:22:23.008143+0000","last_fullsized":"2024-03-06T07:22:23.008143+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:34.284715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007048+0000","last_change":"2024-03-06T07:22:05.324175+0000","last_active":"2024-03-06T07:22:23.007048+0000","last_peered":"2024-03-06T07:22:23.007048+0000","last_clean":"2024-03-06T07:22:23.007048+0000","last_became_active":"2024-03-06T07:22:05.323587+0000","last_became_peered":"2024-03-06T07:22:05.323587+0000","last_unstale":"2024-03-06T07:22:23.007048+0000","last_undegraded":"2024-03-06T07:22:23.007048+0000","last_fullsized":"2024-03-06T07:22:23.007048+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:01.935488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.704720+0000","last_change":"2024-03-06T07:22:21.709596+0000","last_active":"2024-03-06T07:22:22.704720+0000","last_peered":"2024-03-06T07:22:22.704720+0000","last_clean":"2024-03-06T07:22:22.704720+0000","last_became_active":"2024-03-06T07:22:21.709338+0000","last_became_peered":"2024-03-06T07:22:21.709338+0000","last_unstale":"2024-03-06T07:22:22.704720+0000","last_undegraded":"2024-03-06T07:22:22.704720+0000","last_fullsized":"2024-03-06T07:22:22.704720+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:06.987711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"56'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719962+0000","last_change":"2024-03-06T07:22:20.701812+0000","last_active":"2024-03-06T07:22:22.719962+0000","last_peered":"2024-03-06T07:22:22.719962+0000","last_clean":"2024-03-06T07:22:22.719962+0000","last_became_active":"2024-03-06T07:22:20.701560+0000","last_became_peered":"2024-03-06T07:22:20.701560+0000","last_unstale":"2024-03-06T07:22:22.719962+0000","last_undegraded":"2024-03-06T07:22:22.719962+0000","last_fullsized":"2024-03-06T07:22:22.719962+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:48.668680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007645+0000","last_change":"2024-03-06T07:22:06.325375+0000","last_active":"2024-03-06T07:22:23.007645+0000","last_peered":"2024-03-06T07:22:23.007645+0000","last_clean":"2024-03-06T07:22:23.007645+0000","last_became_active":"2024-03-06T07:22:06.325108+0000","last_became_peered":"2024-03-06T07:22:06.325108+0000","last_unstale":"2024-03-06T07:22:23.007645+0000","last_undegraded":"2024-03-06T07:22:23.007645+0000","last_fullsized":"2024-03-06T07:22:23.007645+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:36.139439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.703958+0000","last_change":"2024-03-06T07:22:05.317252+0000","last_active":"2024-03-06T07:22:22.703958+0000","last_peered":"2024-03-06T07:22:22.703958+0000","last_clean":"2024-03-06T07:22:22.703958+0000","last_became_active":"2024-03-06T07:22:05.316964+0000","last_became_peered":"2024-03-06T07:22:05.316964+0000","last_unstale":"2024-03-06T07:22:22.703958+0000","last_undegraded":"2024-03-06T07:22:22.703958+0000","last_fullsized":"2024-03-06T07:22:22.703958+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:34.517096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705650+0000","last_change":"2024-03-06T07:22:21.718931+0000","last_active":"2024-03-06T07:22:22.705650+0000","last_peered":"2024-03-06T07:22:22.705650+0000","last_clean":"2024-03-06T07:22:22.705650+0000","last_became_active":"2024-03-06T07:22:21.718656+0000","last_became_peered":"2024-03-06T07:22:21.718656+0000","last_unstale":"2024-03-06T07:22:22.705650+0000","last_undegraded":"2024-03-06T07:22:22.705650+0000","last_fullsized":"2024-03-06T07:22:22.705650+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:49.684022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006731+0000","last_change":"2024-03-06T07:22:20.700497+0000","last_active":"2024-03-06T07:22:23.006731+0000","last_peered":"2024-03-06T07:22:23.006731+0000","last_clean":"2024-03-06T07:22:23.006731+0000","last_became_active":"2024-03-06T07:22:20.700240+0000","last_became_peered":"2024-03-06T07:22:20.700240+0000","last_unstale":"2024-03-06T07:22:23.006731+0000","last_undegraded":"2024-03-06T07:22:23.006731+0000","last_fullsized":"2024-03-06T07:22:23.006731+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:00.954498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719321+0000","last_change":"2024-03-06T07:22:06.325564+0000","last_active":"2024-03-06T07:22:22.719321+0000","last_peered":"2024-03-06T07:22:22.719321+0000","last_clean":"2024-03-06T07:22:22.719321+0000","last_became_active":"2024-03-06T07:22:06.325298+0000","last_became_peered":"2024-03-06T07:22:06.325298+0000","last_unstale":"2024-03-06T07:22:22.719321+0000","last_undegraded":"2024-03-06T07:22:22.719321+0000","last_fullsized":"2024-03-06T07:22:22.719321+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:51.508838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706339+0000","last_change":"2024-03-06T07:22:05.326276+0000","last_active":"2024-03-06T07:22:22.706339+0000","last_peered":"2024-03-06T07:22:22.706339+0000","last_clean":"2024-03-06T07:22:22.706339+0000","last_became_active":"2024-03-06T07:22:05.325993+0000","last_became_peered":"2024-03-06T07:22:05.325993+0000","last_unstale":"2024-03-06T07:22:22.706339+0000","last_undegraded":"2024-03-06T07:22:22.706339+0000","last_fullsized":"2024-03-06T07:22:22.706339+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:19.366175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006647+0000","last_change":"2024-03-06T07:22:21.710303+0000","last_active":"2024-03-06T07:22:23.006647+0000","last_peered":"2024-03-06T07:22:23.006647+0000","last_clean":"2024-03-06T07:22:23.006647+0000","last_became_active":"2024-03-06T07:22:21.710050+0000","last_became_peered":"2024-03-06T07:22:21.710050+0000","last_unstale":"2024-03-06T07:22:23.006647+0000","last_undegraded":"2024-03-06T07:22:23.006647+0000","last_fullsized":"2024-03-06T07:22:23.006647+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:02.816787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.006803+0000","last_change":"2024-03-06T07:22:20.709655+0000","last_active":"2024-03-06T07:22:23.006803+0000","last_peered":"2024-03-06T07:22:23.006803+0000","last_clean":"2024-03-06T07:22:23.006803+0000","last_became_active":"2024-03-06T07:22:20.709159+0000","last_became_peered":"2024-03-06T07:22:20.709159+0000","last_unstale":"2024-03-06T07:22:23.006803+0000","last_undegraded":"2024-03-06T07:22:23.006803+0000","last_fullsized":"2024-03-06T07:22:23.006803+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:27.335366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008150+0000","last_change":"2024-03-06T07:22:06.323310+0000","last_active":"2024-03-06T07:22:23.008150+0000","last_peered":"2024-03-06T07:22:23.008150+0000","last_clean":"2024-03-06T07:22:23.008150+0000","last_became_active":"2024-03-06T07:22:06.322311+0000","last_became_peered":"2024-03-06T07:22:06.322311+0000","last_unstale":"2024-03-06T07:22:23.008150+0000","last_undegraded":"2024-03-06T07:22:23.008150+0000","last_fullsized":"2024-03-06T07:22:23.008150+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:01.204740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008469+0000","last_change":"2024-03-06T07:22:05.319638+0000","last_active":"2024-03-06T07:22:23.008469+0000","last_peered":"2024-03-06T07:22:23.008469+0000","last_clean":"2024-03-06T07:22:23.008469+0000","last_became_active":"2024-03-06T07:22:05.319307+0000","last_became_peered":"2024-03-06T07:22:05.319307+0000","last_unstale":"2024-03-06T07:22:23.008469+0000","last_undegraded":"2024-03-06T07:22:23.008469+0000","last_fullsized":"2024-03-06T07:22:23.008469+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:19.853691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719254+0000","last_change":"2024-03-06T07:22:21.720493+0000","last_active":"2024-03-06T07:22:22.719254+0000","last_peered":"2024-03-06T07:22:22.719254+0000","last_clean":"2024-03-06T07:22:22.719254+0000","last_became_active":"2024-03-06T07:22:21.719998+0000","last_became_peered":"2024-03-06T07:22:21.719998+0000","last_unstale":"2024-03-06T07:22:22.719254+0000","last_undegraded":"2024-03-06T07:22:22.719254+0000","last_fullsized":"2024-03-06T07:22:22.719254+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:49.989399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719528+0000","last_change":"2024-03-06T07:22:20.704397+0000","last_active":"2024-03-06T07:22:22.719528+0000","last_peered":"2024-03-06T07:22:22.719528+0000","last_clean":"2024-03-06T07:22:22.719528+0000","last_became_active":"2024-03-06T07:22:20.704078+0000","last_became_peered":"2024-03-06T07:22:20.704078+0000","last_unstale":"2024-03-06T07:22:22.719528+0000","last_undegraded":"2024-03-06T07:22:22.719528+0000","last_fullsized":"2024-03-06T07:22:22.719528+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:29.718346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705485+0000","last_change":"2024-03-06T07:22:06.315689+0000","last_active":"2024-03-06T07:22:22.705485+0000","last_peered":"2024-03-06T07:22:22.705485+0000","last_clean":"2024-03-06T07:22:22.705485+0000","last_became_active":"2024-03-06T07:22:06.315395+0000","last_became_peered":"2024-03-06T07:22:06.315395+0000","last_unstale":"2024-03-06T07:22:22.705485+0000","last_undegraded":"2024-03-06T07:22:22.705485+0000","last_fullsized":"2024-03-06T07:22:22.705485+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:09.827959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007399+0000","last_change":"2024-03-06T07:22:05.319307+0000","last_active":"2024-03-06T07:22:23.007399+0000","last_peered":"2024-03-06T07:22:23.007399+0000","last_clean":"2024-03-06T07:22:23.007399+0000","last_became_active":"2024-03-06T07:22:05.318839+0000","last_became_peered":"2024-03-06T07:22:05.318839+0000","last_unstale":"2024-03-06T07:22:23.007399+0000","last_undegraded":"2024-03-06T07:22:23.007399+0000","last_fullsized":"2024-03-06T07:22:23.007399+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:40.725817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008601+0000","last_change":"2024-03-06T07:22:21.722966+0000","last_active":"2024-03-06T07:22:23.008601+0000","last_peered":"2024-03-06T07:22:23.008601+0000","last_clean":"2024-03-06T07:22:23.008601+0000","last_became_active":"2024-03-06T07:22:21.722659+0000","last_became_peered":"2024-03-06T07:22:21.722659+0000","last_unstale":"2024-03-06T07:22:23.008601+0000","last_undegraded":"2024-03-06T07:22:23.008601+0000","last_fullsized":"2024-03-06T07:22:23.008601+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:50.059429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007410+0000","last_change":"2024-03-06T07:22:20.706953+0000","last_active":"2024-03-06T07:22:23.007410+0000","last_peered":"2024-03-06T07:22:23.007410+0000","last_clean":"2024-03-06T07:22:23.007410+0000","last_became_active":"2024-03-06T07:22:20.706459+0000","last_became_peered":"2024-03-06T07:22:20.706459+0000","last_unstale":"2024-03-06T07:22:23.007410+0000","last_undegraded":"2024-03-06T07:22:23.007410+0000","last_fullsized":"2024-03-06T07:22:23.007410+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:24.211924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007309+0000","last_change":"2024-03-06T07:22:06.331443+0000","last_active":"2024-03-06T07:22:23.007309+0000","last_peered":"2024-03-06T07:22:23.007309+0000","last_clean":"2024-03-06T07:22:23.007309+0000","last_became_active":"2024-03-06T07:22:06.331189+0000","last_became_peered":"2024-03-06T07:22:06.331189+0000","last_unstale":"2024-03-06T07:22:23.007309+0000","last_undegraded":"2024-03-06T07:22:23.007309+0000","last_fullsized":"2024-03-06T07:22:23.007309+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:34.295379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705818+0000","last_change":"2024-03-06T07:22:05.315558+0000","last_active":"2024-03-06T07:22:22.705818+0000","last_peered":"2024-03-06T07:22:22.705818+0000","last_clean":"2024-03-06T07:22:22.705818+0000","last_became_active":"2024-03-06T07:22:05.315263+0000","last_became_peered":"2024-03-06T07:22:05.315263+0000","last_unstale":"2024-03-06T07:22:22.705818+0000","last_undegraded":"2024-03-06T07:22:22.705818+0000","last_fullsized":"2024-03-06T07:22:22.705818+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:01.553893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706276+0000","last_change":"2024-03-06T07:22:21.723229+0000","last_active":"2024-03-06T07:22:22.706276+0000","last_peered":"2024-03-06T07:22:22.706276+0000","last_clean":"2024-03-06T07:22:22.706276+0000","last_became_active":"2024-03-06T07:22:21.722956+0000","last_became_peered":"2024-03-06T07:22:21.722956+0000","last_unstale":"2024-03-06T07:22:22.706276+0000","last_undegraded":"2024-03-06T07:22:22.706276+0000","last_fullsized":"2024-03-06T07:22:22.706276+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:34.109293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705671+0000","last_change":"2024-03-06T07:22:20.703827+0000","last_active":"2024-03-06T07:22:22.705671+0000","last_peered":"2024-03-06T07:22:22.705671+0000","last_clean":"2024-03-06T07:22:22.705671+0000","last_became_active":"2024-03-06T07:22:20.703580+0000","last_became_peered":"2024-03-06T07:22:20.703580+0000","last_unstale":"2024-03-06T07:22:22.705671+0000","last_undegraded":"2024-03-06T07:22:22.705671+0000","last_fullsized":"2024-03-06T07:22:22.705671+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:41.367702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719560+0000","last_change":"2024-03-06T07:22:06.324288+0000","last_active":"2024-03-06T07:22:22.719560+0000","last_peered":"2024-03-06T07:22:22.719560+0000","last_clean":"2024-03-06T07:22:22.719560+0000","last_became_active":"2024-03-06T07:22:06.323925+0000","last_became_peered":"2024-03-06T07:22:06.323925+0000","last_unstale":"2024-03-06T07:22:22.719560+0000","last_undegraded":"2024-03-06T07:22:22.719560+0000","last_fullsized":"2024-03-06T07:22:22.719560+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:49.900223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705737+0000","last_change":"2024-03-06T07:22:05.322782+0000","last_active":"2024-03-06T07:22:22.705737+0000","last_peered":"2024-03-06T07:22:22.705737+0000","last_clean":"2024-03-06T07:22:22.705737+0000","last_became_active":"2024-03-06T07:22:05.322484+0000","last_became_peered":"2024-03-06T07:22:05.322484+0000","last_unstale":"2024-03-06T07:22:22.705737+0000","last_undegraded":"2024-03-06T07:22:22.705737+0000","last_fullsized":"2024-03-06T07:22:22.705737+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:04.727017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008539+0000","last_change":"2024-03-06T07:22:21.721093+0000","last_active":"2024-03-06T07:22:23.008539+0000","last_peered":"2024-03-06T07:22:23.008539+0000","last_clean":"2024-03-06T07:22:23.008539+0000","last_became_active":"2024-03-06T07:22:21.720833+0000","last_became_peered":"2024-03-06T07:22:21.720833+0000","last_unstale":"2024-03-06T07:22:23.008539+0000","last_undegraded":"2024-03-06T07:22:23.008539+0000","last_fullsized":"2024-03-06T07:22:23.008539+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:17.698053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719770+0000","last_change":"2024-03-06T07:22:20.705956+0000","last_active":"2024-03-06T07:22:22.719770+0000","last_peered":"2024-03-06T07:22:22.719770+0000","last_clean":"2024-03-06T07:22:22.719770+0000","last_became_active":"2024-03-06T07:22:20.705630+0000","last_became_peered":"2024-03-06T07:22:20.705630+0000","last_unstale":"2024-03-06T07:22:22.719770+0000","last_undegraded":"2024-03-06T07:22:22.719770+0000","last_fullsized":"2024-03-06T07:22:22.719770+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:24.211924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008030+0000","last_change":"2024-03-06T07:22:06.321965+0000","last_active":"2024-03-06T07:22:23.008030+0000","last_peered":"2024-03-06T07:22:23.008030+0000","last_clean":"2024-03-06T07:22:23.008030+0000","last_became_active":"2024-03-06T07:22:06.320167+0000","last_became_peered":"2024-03-06T07:22:06.320167+0000","last_unstale":"2024-03-06T07:22:23.008030+0000","last_undegraded":"2024-03-06T07:22:23.008030+0000","last_fullsized":"2024-03-06T07:22:23.008030+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:15.334381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007251+0000","last_change":"2024-03-06T07:22:05.312784+0000","last_active":"2024-03-06T07:22:23.007251+0000","last_peered":"2024-03-06T07:22:23.007251+0000","last_clean":"2024-03-06T07:22:23.007251+0000","last_became_active":"2024-03-06T07:22:05.312472+0000","last_became_peered":"2024-03-06T07:22:05.312472+0000","last_unstale":"2024-03-06T07:22:23.007251+0000","last_undegraded":"2024-03-06T07:22:23.007251+0000","last_fullsized":"2024-03-06T07:22:23.007251+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:43.288752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706255+0000","last_change":"2024-03-06T07:22:21.727433+0000","last_active":"2024-03-06T07:22:22.706255+0000","last_peered":"2024-03-06T07:22:22.706255+0000","last_clean":"2024-03-06T07:22:22.706255+0000","last_became_active":"2024-03-06T07:22:21.727182+0000","last_became_peered":"2024-03-06T07:22:21.727182+0000","last_unstale":"2024-03-06T07:22:22.706255+0000","last_undegraded":"2024-03-06T07:22:22.706255+0000","last_fullsized":"2024-03-06T07:22:22.706255+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:54.286794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008239+0000","last_change":"2024-03-06T07:22:20.706627+0000","last_active":"2024-03-06T07:22:23.008239+0000","last_peered":"2024-03-06T07:22:23.008239+0000","last_clean":"2024-03-06T07:22:23.008239+0000","last_became_active":"2024-03-06T07:22:20.706360+0000","last_became_peered":"2024-03-06T07:22:20.706360+0000","last_unstale":"2024-03-06T07:22:23.008239+0000","last_undegraded":"2024-03-06T07:22:23.008239+0000","last_fullsized":"2024-03-06T07:22:23.008239+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:01.365623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705904+0000","last_change":"2024-03-06T07:22:06.319779+0000","last_active":"2024-03-06T07:22:22.705904+0000","last_peered":"2024-03-06T07:22:22.705904+0000","last_clean":"2024-03-06T07:22:22.705904+0000","last_became_active":"2024-03-06T07:22:06.319415+0000","last_became_peered":"2024-03-06T07:22:06.319415+0000","last_unstale":"2024-03-06T07:22:22.705904+0000","last_undegraded":"2024-03-06T07:22:22.705904+0000","last_fullsized":"2024-03-06T07:22:22.705904+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:20.884386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.719743+0000","last_change":"2024-03-06T07:22:05.322202+0000","last_active":"2024-03-06T07:22:22.719743+0000","last_peered":"2024-03-06T07:22:22.719743+0000","last_clean":"2024-03-06T07:22:22.719743+0000","last_became_active":"2024-03-06T07:22:05.321896+0000","last_became_peered":"2024-03-06T07:22:05.321896+0000","last_unstale":"2024-03-06T07:22:22.719743+0000","last_undegraded":"2024-03-06T07:22:22.719743+0000","last_fullsized":"2024-03-06T07:22:22.719743+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:44.392826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.706135+0000","last_change":"2024-03-06T07:22:21.724699+0000","last_active":"2024-03-06T07:22:22.706135+0000","last_peered":"2024-03-06T07:22:22.706135+0000","last_clean":"2024-03-06T07:22:22.706135+0000","last_became_active":"2024-03-06T07:22:21.724448+0000","last_became_peered":"2024-03-06T07:22:21.724448+0000","last_unstale":"2024-03-06T07:22:22.706135+0000","last_undegraded":"2024-03-06T07:22:22.706135+0000","last_fullsized":"2024-03-06T07:22:22.706135+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:20.683171+0000","last_clean_scrub_stamp":"2024-03-06T07:22:20.683171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:32.957457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705071+0000","last_change":"2024-03-06T07:22:20.700732+0000","last_active":"2024-03-06T07:22:22.705071+0000","last_peered":"2024-03-06T07:22:22.705071+0000","last_clean":"2024-03-06T07:22:22.705071+0000","last_became_active":"2024-03-06T07:22:20.700157+0000","last_became_peered":"2024-03-06T07:22:20.700157+0000","last_unstale":"2024-03-06T07:22:22.705071+0000","last_undegraded":"2024-03-06T07:22:22.705071+0000","last_fullsized":"2024-03-06T07:22:22.705071+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:19.678493+0000","last_clean_scrub_stamp":"2024-03-06T07:22:19.678493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:07.182303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:22.705476+0000","last_change":"2024-03-06T07:22:06.328678+0000","last_active":"2024-03-06T07:22:22.705476+0000","last_peered":"2024-03-06T07:22:22.705476+0000","last_clean":"2024-03-06T07:22:22.705476+0000","last_became_active":"2024-03-06T07:22:06.328362+0000","last_became_peered":"2024-03-06T07:22:06.328362+0000","last_unstale":"2024-03-06T07:22:22.705476+0000","last_undegraded":"2024-03-06T07:22:22.705476+0000","last_fullsized":"2024-03-06T07:22:22.705476+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.294528+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.294528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:27.383059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"52'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.008147+0000","last_change":"2024-03-06T07:22:05.326257+0000","last_active":"2024-03-06T07:22:23.008147+0000","last_peered":"2024-03-06T07:22:23.008147+0000","last_clean":"2024-03-06T07:22:23.008147+0000","last_became_active":"2024-03-06T07:22:05.325920+0000","last_became_peered":"2024-03-06T07:22:05.325920+0000","last_unstale":"2024-03-06T07:22:23.008147+0000","last_undegraded":"2024-03-06T07:22:23.008147+0000","last_fullsized":"2024-03-06T07:22:23.008147+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.289606+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.289606+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:58.902870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":239,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:22:23.007474+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:22:23.007474+0000","last_peered":"2024-03-06T07:22:23.007474+0000","last_clean":"2024-03-06T07:22:23.007474+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:22:23.007474+0000","last_undegraded":"2024-03-06T07:22:23.007474+0000","last_fullsized":"2024-03-06T07:22:23.007474+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":757760,"data_stored":898948,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":308366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":307702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":684032,"data_stored":309874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":309314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":679936,"data_stored":309292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":704512,"data_stored":310216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":741376,"data_stored":898104,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:26.155 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:22:26.155 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 3:0 2024-03-06T07:22:26.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.286+0000 7f5b0b97a640 1 Processor -- start 2024-03-06T07:22:26.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.286+0000 7f5b0b97a640 1 -- start start 2024-03-06T07:22:26.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.287+0000 7f5b0b97a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5b040c6b10 0x7f5b04155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.287+0000 7f5b0b97a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b040c2fc0 con 0x7f5b04152650 2024-03-06T07:22:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.287+0000 7f5b0b97a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b04158da0 con 0x7f5b040bf9d0 2024-03-06T07:22:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.287+0000 7f5b0b97a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b04155800 con 0x7f5b040c6b10 2024-03-06T07:22:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5b09716640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5b04152650 0x7f5b04158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43872/0 (socket says 172.21.15.1:43872) 2024-03-06T07:22:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5b09716640 1 -- 172.21.15.1:0/2218401702 learned_addr learned my addr 172.21.15.1:0/2218401702 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5b08f15640 1 --2- 172.21.15.1:0/2218401702 >> v2:172.21.15.98:3300/0 conn(0x7f5b040c6b10 0x7f5b04155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5afbfff640 1 -- 172.21.15.1:0/2218401702 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222026875 0 0) 0x7f5b040c2fc0 con 0x7f5b04152650 2024-03-06T07:22:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5afbfff640 1 -- 172.21.15.1:0/2218401702 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5af0003520 con 0x7f5b04152650 2024-03-06T07:22:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5afbfff640 1 -- 172.21.15.1:0/2218401702 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1875914705 0 0) 0x7f5af0003520 con 0x7f5b04152650 2024-03-06T07:22:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5afbfff640 1 -- 172.21.15.1:0/2218401702 >> v2:172.21.15.98:3300/0 conn(0x7f5b040c6b10 msgr2=0x7f5b04155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5afbfff640 1 --2- 172.21.15.1:0/2218401702 >> v2:172.21.15.98:3300/0 conn(0x7f5b040c6b10 0x7f5b04155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5afbfff640 1 -- 172.21.15.1:0/2218401702 >> v1:172.21.15.98:6789/0 conn(0x7f5b040bf9d0 legacy=0x7f5b040c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5afbfff640 1 -- 172.21.15.1:0/2218401702 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b040c6690 con 0x7f5b04152650 2024-03-06T07:22:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5afbfff640 1 -- 172.21.15.1:0/2218401702 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f5b00001670 con 0x7f5b04152650 2024-03-06T07:22:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5afbfff640 1 -- 172.21.15.1:0/2218401702 >> v1:172.21.15.1:6789/0 conn(0x7f5b04152650 legacy=0x7f5b04158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5afbfff640 1 --2- 172.21.15.1:0/2218401702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af00039b0 0x7f5af0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5afbfff640 1 --2- 172.21.15.1:0/2218401702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5af0004800 0x7f5af0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5afbfff640 1 --2- 172.21.15.1:0/2218401702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5af00051c0 0x7f5af000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5afbfff640 1 -- 172.21.15.1:0/2218401702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b04155800 con 0x7f5af00039b0 2024-03-06T07:22:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5afbfff640 1 -- 172.21.15.1:0/2218401702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af000c160 con 0x7f5af00051c0 2024-03-06T07:22:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5afbfff640 1 -- 172.21.15.1:0/2218401702 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af000c360 con 0x7f5af0004800 2024-03-06T07:22:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.288+0000 7f5b09716640 1 --2- 172.21.15.1:0/2218401702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af00039b0 0x7f5af0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.289+0000 7f5b09f17640 1 --2- 172.21.15.1:0/2218401702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5af00051c0 0x7f5af000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.289+0000 7f5b08f15640 1 --2- 172.21.15.1:0/2218401702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5af0004800 0x7f5af0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.289+0000 7f5b09716640 1 -- 172.21.15.1:0/2218401702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5af0004800 msgr2=0x7f5af0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.289+0000 7f5b09716640 1 --2- 172.21.15.1:0/2218401702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5af0004800 0x7f5af0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.289+0000 7f5b09716640 1 -- 172.21.15.1:0/2218401702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5af00051c0 msgr2=0x7f5af000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.289+0000 7f5b09716640 1 --2- 172.21.15.1:0/2218401702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5af00051c0 0x7f5af000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.289+0000 7f5b09716640 1 -- 172.21.15.1:0/2218401702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af0003520 con 0x7f5af00039b0 2024-03-06T07:22:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.289+0000 7f5b09f17640 1 -- 172.21.15.1:0/2218401702 reap_dead start 2024-03-06T07:22:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.290+0000 7f5b09716640 1 --2- 172.21.15.1:0/2218401702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af00039b0 0x7f5af0003dd0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5b0000c240 tx=0x7f5af0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac4b711c68c32d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.290+0000 7f5afbfff640 1 -- 172.21.15.1:0/2218401702 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00007cf0 con 0x7f5af00039b0 2024-03-06T07:22:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.290+0000 7f5afbfff640 1 -- 172.21.15.1:0/2218401702 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b00030070 con 0x7f5af00039b0 2024-03-06T07:22:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.290+0000 7f5afbfff640 1 -- 172.21.15.1:0/2218401702 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00005d70 con 0x7f5af00039b0 2024-03-06T07:22:26.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.290+0000 7f5b0b97a640 1 -- 172.21.15.1:0/2218401702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af00039b0 msgr2=0x7f5af0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.290+0000 7f5b0b97a640 1 --2- 172.21.15.1:0/2218401702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af00039b0 0x7f5af0003dd0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5b0000c240 tx=0x7f5af0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:26.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.290+0000 7f5b0b97a640 1 -- 172.21.15.1:0/2218401702 shutdown_connections 2024-03-06T07:22:26.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.290+0000 7f5b0b97a640 1 --2- 172.21.15.1:0/2218401702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af00039b0 0x7f5af0003dd0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.290+0000 7f5b0b97a640 1 -- 172.21.15.1:0/2218401702 >> 172.21.15.1:0/2218401702 conn(0x7f5b04066ee0 msgr2=0x7f5b040672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.291+0000 7f5b0b97a640 1 -- 172.21.15.1:0/2218401702 shutdown_connections 2024-03-06T07:22:26.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.291+0000 7f5b0b97a640 1 -- 172.21.15.1:0/2218401702 wait complete. 2024-03-06T07:22:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.292+0000 7f5b0b97a640 1 Processor -- start 2024-03-06T07:22:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.292+0000 7f5b0b97a640 1 -- start start 2024-03-06T07:22:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.293+0000 7f5b0b97a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5af00039b0 0x7f5b04151330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.293+0000 7f5b0b97a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b0414c910 0x7f5b041ee500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.293+0000 7f5b0b97a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b0414b310 0x7f5b0414a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.293+0000 7f5b0b97a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b041551b0 con 0x7f5b0414c910 2024-03-06T07:22:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.293+0000 7f5b0b97a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b040c6fa0 con 0x7f5b0414b310 2024-03-06T07:22:26.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.293+0000 7f5b0b97a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b04155490 con 0x7f5af00039b0 2024-03-06T07:22:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.293+0000 7f5b08f15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b0414c910 0x7f5b041ee500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.294+0000 7f5b08f15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b0414c910 0x7f5b041ee500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47160/0 (socket says 172.21.15.1:47160) 2024-03-06T07:22:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.294+0000 7f5b08f15640 1 -- 172.21.15.1:0/1043809220 learned_addr learned my addr 172.21.15.1:0/1043809220 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.294+0000 7f5b09f17640 1 --2- 172.21.15.1:0/1043809220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b0414b310 0x7f5b0414a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.294+0000 7f5b09716640 1 --2- 172.21.15.1:0/1043809220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5af00039b0 0x7f5b04151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.294+0000 7f5b08f15640 1 -- 172.21.15.1:0/1043809220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5af00039b0 msgr2=0x7f5b04151330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.294+0000 7f5b08f15640 1 --2- 172.21.15.1:0/1043809220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5af00039b0 0x7f5b04151330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.294+0000 7f5b08f15640 1 -- 172.21.15.1:0/1043809220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b0414b310 msgr2=0x7f5b0414a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.294+0000 7f5b08f15640 1 --2- 172.21.15.1:0/1043809220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b0414b310 0x7f5b0414a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.294+0000 7f5b08f15640 1 -- 172.21.15.1:0/1043809220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0414b030 con 0x7f5b0414c910 2024-03-06T07:22:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.294+0000 7f5b09f17640 1 --2- 172.21.15.1:0/1043809220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b0414b310 0x7f5b0414a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.294+0000 7f5b08f15640 1 --2- 172.21.15.1:0/1043809220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b0414c910 0x7f5b041ee500 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5af4011aa0 tx=0x7f5af4011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.294+0000 7f5af9ffb640 1 -- 172.21.15.1:0/1043809220 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af400fb80 con 0x7f5b0414c910 2024-03-06T07:22:26.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.294+0000 7f5af9ffb640 1 -- 172.21.15.1:0/1043809220 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af400fd20 con 0x7f5b0414c910 2024-03-06T07:22:26.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.294+0000 7f5b0b97a640 1 -- 172.21.15.1:0/1043809220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b041e2150 con 0x7f5b0414c910 2024-03-06T07:22:26.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.294+0000 7f5af9ffb640 1 -- 172.21.15.1:0/1043809220 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af4014660 con 0x7f5b0414c910 2024-03-06T07:22:26.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.295+0000 7f5b0b97a640 1 -- 172.21.15.1:0/1043809220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b041edfe0 con 0x7f5b0414c910 2024-03-06T07:22:26.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.295+0000 7f5b0b97a640 1 -- 172.21.15.1:0/1043809220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b040c45d0 con 0x7f5b0414c910 2024-03-06T07:22:26.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.296+0000 7f5af9ffb640 1 -- 172.21.15.1:0/1043809220 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5af4012040 con 0x7f5b0414c910 2024-03-06T07:22:26.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.297+0000 7f5af9ffb640 1 --2- 172.21.15.1:0/1043809220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ae806b500 0x7f5ae806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.297+0000 7f5af9ffb640 1 -- 172.21.15.1:0/1043809220 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f5af408ad50 con 0x7f5b0414c910 2024-03-06T07:22:26.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.297+0000 7f5b09716640 1 --2- 172.21.15.1:0/1043809220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ae806b500 0x7f5ae806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.298+0000 7f5b09716640 1 --2- 172.21.15.1:0/1043809220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ae806b500 0x7f5ae806d9c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5b00034000 tx=0x7f5b00039040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.301+0000 7f5af9ffb640 1 -- 172.21.15.1:0/1043809220 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b040c45d0 con 0x7f5b0414c910 2024-03-06T07:22:26.594 INFO:tasks.ceph.mds.a.smithi001.stderr:2024-03-06T07:22:26.592+0000 7f4939bcd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:26.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.738+0000 7f5b0b97a640 1 -- 172.21.15.1:0/1043809220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "3:0"} v 0) v1 -- 0x7f5b04067fd0 con 0x7f5b0414c910 2024-03-06T07:22:26.741 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:22:26.739+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:22:26.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.740+0000 7f5af9ffb640 1 -- 172.21.15.1:0/1043809220 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "3:0"}]=0 failed mds gid 5778 v35) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5af408e030 con 0x7f5b0414c910 2024-03-06T07:22:26.742 INFO:teuthology.orchestra.run.smithi001.stderr:failed mds gid 5778 2024-03-06T07:22:26.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.743+0000 7f5b0b97a640 1 -- 172.21.15.1:0/1043809220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ae806b500 msgr2=0x7f5ae806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.743+0000 7f5b0b97a640 1 --2- 172.21.15.1:0/1043809220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ae806b500 0x7f5ae806d9c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5b00034000 tx=0x7f5b00039040 comp rx=0 tx=0).stop 2024-03-06T07:22:26.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.743+0000 7f5b0b97a640 1 -- 172.21.15.1:0/1043809220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b0414c910 msgr2=0x7f5b041ee500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.743+0000 7f5b0b97a640 1 --2- 172.21.15.1:0/1043809220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b0414c910 0x7f5b041ee500 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5af4011aa0 tx=0x7f5af4011f60 comp rx=0 tx=0).stop 2024-03-06T07:22:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.744+0000 7f5b0b97a640 1 -- 172.21.15.1:0/1043809220 shutdown_connections 2024-03-06T07:22:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.744+0000 7f5b0b97a640 1 --2- 172.21.15.1:0/1043809220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ae806b500 0x7f5ae806d9c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.744+0000 7f5b0b97a640 1 --2- 172.21.15.1:0/1043809220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b0414b310 0x7f5b0414a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.744+0000 7f5b0b97a640 1 --2- 172.21.15.1:0/1043809220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b0414c910 0x7f5b041ee500 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.744+0000 7f5b0b97a640 1 --2- 172.21.15.1:0/1043809220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5af00039b0 0x7f5b04151330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.744+0000 7f5b0b97a640 1 -- 172.21.15.1:0/1043809220 >> 172.21.15.1:0/1043809220 conn(0x7f5b04066ee0 msgr2=0x7f5b040c0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.745+0000 7f5b0b97a640 1 -- 172.21.15.1:0/1043809220 shutdown_connections 2024-03-06T07:22:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.745+0000 7f5b0b97a640 1 -- 172.21.15.1:0/1043809220 wait complete. 2024-03-06T07:22:26.748 INFO:tasks.ceph.mds.c.smithi001.stderr: -523> 2024-03-06T07:22:11.591+0000 7fb8e6f7e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:26.764 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:26.775 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:22:26.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.865+0000 7fc460678640 1 Processor -- start 2024-03-06T07:22:26.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.865+0000 7fc460678640 1 -- start start 2024-03-06T07:22:26.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.866+0000 7fc460678640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc4580c6730 0x7fc4580c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.866+0000 7fc460678640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4580c3300 con 0x7fc4580c5d10 2024-03-06T07:22:26.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.866+0000 7fc460678640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4580c35a0 con 0x7fc458152690 2024-03-06T07:22:26.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.866+0000 7fc460678640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4580c3880 con 0x7fc4580c6730 2024-03-06T07:22:26.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.866+0000 7fc45ec15640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc4580c5d10 0x7fc458156cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43892/0 (socket says 172.21.15.1:43892) 2024-03-06T07:22:26.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.866+0000 7fc45ec15640 1 -- 172.21.15.1:0/997387156 learned_addr learned my addr 172.21.15.1:0/997387156 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:26.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.866+0000 7fc45dc13640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc4580c6730 0x7fc4580c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.866+0000 7fc45dc13640 1 -- 172.21.15.1:0/997387156 >> v1:172.21.15.98:6789/0 conn(0x7fc458152690 legacy=0x7fc458155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.866+0000 7fc45dc13640 1 -- 172.21.15.1:0/997387156 >> v1:172.21.15.1:6789/0 conn(0x7fc4580c5d10 legacy=0x7fc458156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.866+0000 7fc45dc13640 1 -- 172.21.15.1:0/997387156 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4580c3b60 con 0x7fc4580c6730 2024-03-06T07:22:26.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.866+0000 7fc45d412640 1 -- 172.21.15.1:0/997387156 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119785780 0 0) 0x7fc4580c3300 con 0x7fc4580c5d10 2024-03-06T07:22:26.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.867+0000 7fc45dc13640 1 --2- 172.21.15.1:0/997387156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4580c6730 0x7fc4580c57d0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc4480097a0 tx=0x7fc44802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6d0c21f8bd4577 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.867+0000 7fc45d412640 1 -- 172.21.15.1:0/997387156 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448035020 con 0x7fc4580c6730 2024-03-06T07:22:26.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.867+0000 7fc45d412640 1 -- 172.21.15.1:0/997387156 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc448031830 con 0x7fc4580c6730 2024-03-06T07:22:26.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.867+0000 7fc45d412640 1 -- 172.21.15.1:0/997387156 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448031c00 con 0x7fc4580c6730 2024-03-06T07:22:26.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.868+0000 7fc460678640 1 -- 172.21.15.1:0/997387156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4580c6730 msgr2=0x7fc4580c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.868+0000 7fc460678640 1 --2- 172.21.15.1:0/997387156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4580c6730 0x7fc4580c57d0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc4480097a0 tx=0x7fc44802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.868+0000 7fc460678640 1 -- 172.21.15.1:0/997387156 shutdown_connections 2024-03-06T07:22:26.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.868+0000 7fc460678640 1 --2- 172.21.15.1:0/997387156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4580c6730 0x7fc4580c57d0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.868+0000 7fc460678640 1 -- 172.21.15.1:0/997387156 >> 172.21.15.1:0/997387156 conn(0x7fc45806b410 msgr2=0x7fc45806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.868+0000 7fc460678640 1 -- 172.21.15.1:0/997387156 shutdown_connections 2024-03-06T07:22:26.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.868+0000 7fc460678640 1 -- 172.21.15.1:0/997387156 wait complete. 2024-03-06T07:22:26.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.869+0000 7fc460678640 1 Processor -- start 2024-03-06T07:22:26.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.869+0000 7fc460678640 1 -- start start 2024-03-06T07:22:26.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.869+0000 7fc460678640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4580c5d10 0x7fc4580c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.869+0000 7fc460678640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4580c6730 0x7fc458153050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc460678640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc458152690 0x7fc4580a6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc460678640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc458152bf0 con 0x7fc458152690 2024-03-06T07:22:26.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc460678640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc458152a70 con 0x7fc4580c5d10 2024-03-06T07:22:26.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc460678640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4581573e0 con 0x7fc4580c6730 2024-03-06T07:22:26.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc45ec15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc458152690 0x7fc4580a6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc45ec15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc458152690 0x7fc4580a6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47188/0 (socket says 172.21.15.1:47188) 2024-03-06T07:22:26.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc45ec15640 1 -- 172.21.15.1:0/4031231773 learned_addr learned my addr 172.21.15.1:0/4031231773 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:26.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc45e414640 1 --2- 172.21.15.1:0/4031231773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4580c5d10 0x7fc4580c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc45dc13640 1 --2- 172.21.15.1:0/4031231773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4580c6730 0x7fc458153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc45ec15640 1 -- 172.21.15.1:0/4031231773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4580c6730 msgr2=0x7fc458153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc45ec15640 1 --2- 172.21.15.1:0/4031231773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4580c6730 0x7fc458153050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc45ec15640 1 -- 172.21.15.1:0/4031231773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4580c5d10 msgr2=0x7fc4580c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc45ec15640 1 --2- 172.21.15.1:0/4031231773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4580c5d10 0x7fc4580c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc45ec15640 1 -- 172.21.15.1:0/4031231773 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4581ea410 con 0x7fc458152690 2024-03-06T07:22:26.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc45ec15640 1 --2- 172.21.15.1:0/4031231773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc458152690 0x7fc4580a6b20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc45000caf0 tx=0x7fc4500021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc44effd640 1 -- 172.21.15.1:0/4031231773 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45000ac00 con 0x7fc458152690 2024-03-06T07:22:26.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc44effd640 1 -- 172.21.15.1:0/4031231773 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45000ada0 con 0x7fc458152690 2024-03-06T07:22:26.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc460678640 1 -- 172.21.15.1:0/4031231773 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4580ad610 con 0x7fc458152690 2024-03-06T07:22:26.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc44effd640 1 -- 172.21.15.1:0/4031231773 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450012640 con 0x7fc458152690 2024-03-06T07:22:26.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.870+0000 7fc460678640 1 -- 172.21.15.1:0/4031231773 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4581ea970 con 0x7fc458152690 2024-03-06T07:22:26.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.872+0000 7fc460678640 1 -- 172.21.15.1:0/4031231773 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc458067fd0 con 0x7fc458152690 2024-03-06T07:22:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.872+0000 7fc44effd640 1 -- 172.21.15.1:0/4031231773 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc4500127e0 con 0x7fc458152690 2024-03-06T07:22:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.872+0000 7fc44effd640 1 --2- 172.21.15.1:0/4031231773 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc43c06b4d0 0x7fc43c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.872+0000 7fc44effd640 1 -- 172.21.15.1:0/4031231773 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fc45008ab90 con 0x7fc458152690 2024-03-06T07:22:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.873+0000 7fc45e414640 1 --2- 172.21.15.1:0/4031231773 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc43c06b4d0 0x7fc43c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.873+0000 7fc45e414640 1 --2- 172.21.15.1:0/4031231773 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc43c06b4d0 0x7fc43c06d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc454000f60 tx=0x7fc454009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:26.877+0000 7fc44effd640 1 -- 172.21.15.1:0/4031231773 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc4581ea410 con 0x7fc458152690 2024-03-06T07:22:27.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.129+0000 7fc460678640 1 -- 172.21.15.1:0/4031231773 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4580c83f0 con 0x7fc458152690 2024-03-06T07:22:27.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.130+0000 7fc44effd640 1 -- 172.21.15.1:0/4031231773 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5456 (secure 0 0 0) 0x7fc450058e80 con 0x7fc458152690 2024-03-06T07:22:27.132 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:27.132 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5087,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.98:6837/2828783606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2828783606},{"type":"v1","addr":"172.21.15.98:6837","nonce":2828783606}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:05.671602+0000","modified":"2024-03-06T07:22:26.744431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5084":{"gid":5084,"name":"b","rank":0,"incarnation":36,"state":"up:replay","state_seq":7,"addr":"172.21.15.98:6836/2937068915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2937068915},{"type":"v1","addr":"172.21.15.98:6836","nonce":2937068915}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:21.047528+0000","modified":"2024-03-06T07:22:24.069892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5344":{"gid":5344,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":7,"addr":"172.21.15.1:6835/3845300501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":3845300501},{"type":"v1","addr":"172.21.15.1:6835","nonce":3845300501}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5344,"qdb_cluster":[5344]},"id":4}]} 2024-03-06T07:22:27.132 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 36 2024-03-06T07:22:27.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.132+0000 7fc460678640 1 -- 172.21.15.1:0/4031231773 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc43c06b4d0 msgr2=0x7fc43c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.132+0000 7fc460678640 1 --2- 172.21.15.1:0/4031231773 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc43c06b4d0 0x7fc43c06d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc454000f60 tx=0x7fc454009040 comp rx=0 tx=0).stop 2024-03-06T07:22:27.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.132+0000 7fc460678640 1 -- 172.21.15.1:0/4031231773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc458152690 msgr2=0x7fc4580a6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.132+0000 7fc460678640 1 --2- 172.21.15.1:0/4031231773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc458152690 0x7fc4580a6b20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc45000caf0 tx=0x7fc4500021c0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.132+0000 7fc460678640 1 -- 172.21.15.1:0/4031231773 shutdown_connections 2024-03-06T07:22:27.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.132+0000 7fc460678640 1 --2- 172.21.15.1:0/4031231773 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc43c06b4d0 0x7fc43c06d990 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.132+0000 7fc460678640 1 --2- 172.21.15.1:0/4031231773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc458152690 0x7fc4580a6b20 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.132+0000 7fc460678640 1 --2- 172.21.15.1:0/4031231773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4580c6730 0x7fc458153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.132+0000 7fc460678640 1 --2- 172.21.15.1:0/4031231773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4580c5d10 0x7fc4580c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.132+0000 7fc460678640 1 -- 172.21.15.1:0/4031231773 >> 172.21.15.1:0/4031231773 conn(0x7fc45806b410 msgr2=0x7fc45806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.132+0000 7fc460678640 1 -- 172.21.15.1:0/4031231773 shutdown_connections 2024-03-06T07:22:27.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.132+0000 7fc460678640 1 -- 172.21.15.1:0/4031231773 wait complete. 2024-03-06T07:22:27.145 INFO:tasks.cephfs.test_failover:Waiting for former active to reclaim its spot 2024-03-06T07:22:27.145 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-03-06T07:22:27.293 INFO:teuthology.orchestra.run.smithi001.stdout:{"mds_beacon_grace":"15.000000"} 2024-03-06T07:22:27.293 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-03-06T07:22:27.293 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:27.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.393+0000 7f87f4cd4640 1 Processor -- start 2024-03-06T07:22:27.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.393+0000 7f87f4cd4640 1 -- start start 2024-03-06T07:22:27.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.393+0000 7f87f4cd4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87f00a7280 0x7f87f0072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.393+0000 7f87f4cd4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87f00c4ef0 0x7f87f01564e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.393+0000 7f87f4cd4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f87f00732b0 0x7f87f0156a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.393+0000 7f87f4cd4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f0152fb0 con 0x7f87f00c4ef0 2024-03-06T07:22:27.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.393+0000 7f87f4cd4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f0153290 con 0x7f87f00732b0 2024-03-06T07:22:27.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.393+0000 7f87f4cd4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f0153570 con 0x7f87f00a7280 2024-03-06T07:22:27.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.394+0000 7f87edd9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87f00c4ef0 0x7f87f01564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.394+0000 7f87edd9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87f00c4ef0 0x7f87f01564e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47200/0 (socket says 172.21.15.1:47200) 2024-03-06T07:22:27.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.394+0000 7f87edd9b640 1 -- 172.21.15.1:0/1961046645 learned_addr learned my addr 172.21.15.1:0/1961046645 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:27.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.394+0000 7f87eed9d640 1 --2- 172.21.15.1:0/1961046645 >> v2:172.21.15.98:3300/0 conn(0x7f87f00732b0 0x7f87f0156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.394+0000 7f87ee59c640 1 --2- 172.21.15.1:0/1961046645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87f00a7280 0x7f87f0072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.394+0000 7f87edd9b640 1 -- 172.21.15.1:0/1961046645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87f00a7280 msgr2=0x7f87f0072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.394+0000 7f87edd9b640 1 --2- 172.21.15.1:0/1961046645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87f00a7280 0x7f87f0072d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.394+0000 7f87edd9b640 1 -- 172.21.15.1:0/1961046645 >> v2:172.21.15.98:3300/0 conn(0x7f87f00732b0 msgr2=0x7f87f0156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.394+0000 7f87edd9b640 1 --2- 172.21.15.1:0/1961046645 >> v2:172.21.15.98:3300/0 conn(0x7f87f00732b0 0x7f87f0156a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.394+0000 7f87edd9b640 1 -- 172.21.15.1:0/1961046645 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f0153850 con 0x7f87f00c4ef0 2024-03-06T07:22:27.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.395+0000 7f87edd9b640 1 --2- 172.21.15.1:0/1961046645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87f00c4ef0 0x7f87f01564e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f87f00c1f30 tx=0x7f87d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1acf281ec169d31d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.395+0000 7f87ed59a640 1 -- 172.21.15.1:0/1961046645 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8035020 con 0x7f87f00c4ef0 2024-03-06T07:22:27.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.395+0000 7f87ed59a640 1 -- 172.21.15.1:0/1961046645 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d8031870 con 0x7f87f00c4ef0 2024-03-06T07:22:27.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.395+0000 7f87ed59a640 1 -- 172.21.15.1:0/1961046645 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8031c40 con 0x7f87f00c4ef0 2024-03-06T07:22:27.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.395+0000 7f87f4cd4640 1 -- 172.21.15.1:0/1961046645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87f00c4ef0 msgr2=0x7f87f01564e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.395+0000 7f87f4cd4640 1 --2- 172.21.15.1:0/1961046645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87f00c4ef0 0x7f87f01564e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f87f00c1f30 tx=0x7f87d802b910 comp rx=0 tx=0).stop 2024-03-06T07:22:27.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.395+0000 7f87f4cd4640 1 -- 172.21.15.1:0/1961046645 shutdown_connections 2024-03-06T07:22:27.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.395+0000 7f87f4cd4640 1 --2- 172.21.15.1:0/1961046645 >> v2:172.21.15.98:3300/0 conn(0x7f87f00732b0 0x7f87f0156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.395+0000 7f87f4cd4640 1 --2- 172.21.15.1:0/1961046645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87f00c4ef0 0x7f87f01564e0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.395+0000 7f87f4cd4640 1 --2- 172.21.15.1:0/1961046645 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87f00a7280 0x7f87f0072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.395+0000 7f87f4cd4640 1 -- 172.21.15.1:0/1961046645 >> 172.21.15.1:0/1961046645 conn(0x7f87f00c7a50 msgr2=0x7f87f00c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.396+0000 7f87f4cd4640 1 -- 172.21.15.1:0/1961046645 shutdown_connections 2024-03-06T07:22:27.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.396+0000 7f87f4cd4640 1 -- 172.21.15.1:0/1961046645 wait complete. 2024-03-06T07:22:27.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.397+0000 7f87f4cd4640 1 Processor -- start 2024-03-06T07:22:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.397+0000 7f87f4cd4640 1 -- start start 2024-03-06T07:22:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.397+0000 7f87f4cd4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87f00732b0 0x7f87f01e1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.397+0000 7f87f4cd4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87f00a7280 0x7f87f01ddf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.397+0000 7f87f4cd4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00c4ef0 0x7f87f01de4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.397+0000 7f87f4cd4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f00c2660 con 0x7f87f00a7280 2024-03-06T07:22:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.397+0000 7f87f4cd4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f00c24e0 con 0x7f87f00c4ef0 2024-03-06T07:22:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.397+0000 7f87f4cd4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f0156f60 con 0x7f87f00732b0 2024-03-06T07:22:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.398+0000 7f87edd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87f00a7280 0x7f87f01ddf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.398+0000 7f87edd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87f00a7280 0x7f87f01ddf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47208/0 (socket says 172.21.15.1:47208) 2024-03-06T07:22:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.398+0000 7f87edd9b640 1 -- 172.21.15.1:0/1780729409 learned_addr learned my addr 172.21.15.1:0/1780729409 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.398+0000 7f87ee59c640 1 --2- 172.21.15.1:0/1780729409 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87f00732b0 0x7f87f01e1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.398+0000 7f87eed9d640 1 --2- 172.21.15.1:0/1780729409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00c4ef0 0x7f87f01de4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.398+0000 7f87edd9b640 1 -- 172.21.15.1:0/1780729409 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87f00732b0 msgr2=0x7f87f01e1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.398+0000 7f87edd9b640 1 --2- 172.21.15.1:0/1780729409 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87f00732b0 0x7f87f01e1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.398+0000 7f87edd9b640 1 -- 172.21.15.1:0/1780729409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00c4ef0 msgr2=0x7f87f01de4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.398+0000 7f87edd9b640 1 --2- 172.21.15.1:0/1780729409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00c4ef0 0x7f87f01de4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.398+0000 7f87edd9b640 1 -- 172.21.15.1:0/1780729409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f01eb380 con 0x7f87f00a7280 2024-03-06T07:22:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.398+0000 7f87eed9d640 1 --2- 172.21.15.1:0/1780729409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00c4ef0 0x7f87f01de4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.398+0000 7f87edd9b640 1 --2- 172.21.15.1:0/1780729409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87f00a7280 0x7f87f01ddf80 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f87d802b950 tx=0x7f87d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.398+0000 7f87e6ffd640 1 -- 172.21.15.1:0/1780729409 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8040070 con 0x7f87f00a7280 2024-03-06T07:22:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.398+0000 7f87e6ffd640 1 -- 172.21.15.1:0/1780729409 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d8030d00 con 0x7f87f00a7280 2024-03-06T07:22:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.398+0000 7f87f4cd4640 1 -- 172.21.15.1:0/1780729409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f01eb660 con 0x7f87f00a7280 2024-03-06T07:22:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.398+0000 7f87e6ffd640 1 -- 172.21.15.1:0/1780729409 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8035040 con 0x7f87f00a7280 2024-03-06T07:22:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.398+0000 7f87f4cd4640 1 -- 172.21.15.1:0/1780729409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f01ebc20 con 0x7f87f00a7280 2024-03-06T07:22:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.399+0000 7f87f4cd4640 1 -- 172.21.15.1:0/1780729409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f00686e0 con 0x7f87f00a7280 2024-03-06T07:22:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.400+0000 7f87e6ffd640 1 -- 172.21.15.1:0/1780729409 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87d804b060 con 0x7f87f00a7280 2024-03-06T07:22:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.401+0000 7f87e6ffd640 1 --2- 172.21.15.1:0/1780729409 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f87d006b4d0 0x7f87d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.401+0000 7f87e6ffd640 1 -- 172.21.15.1:0/1780729409 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f87d80ac490 con 0x7f87f00a7280 2024-03-06T07:22:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.401+0000 7f87ee59c640 1 --2- 172.21.15.1:0/1780729409 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f87d006b4d0 0x7f87d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.401+0000 7f87ee59c640 1 --2- 172.21.15.1:0/1780729409 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f87d006b4d0 0x7f87d006d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f87f00c81f0 tx=0x7f87e0009830 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.409+0000 7f87e6ffd640 1 -- 172.21.15.1:0/1780729409 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87d807ae60 con 0x7f87f00a7280 2024-03-06T07:22:27.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.629+0000 7f87f4cd4640 1 -- 172.21.15.1:0/1780729409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87f0072b50 con 0x7f87f00a7280 2024-03-06T07:22:27.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.629+0000 7f87e6ffd640 1 -- 172.21.15.1:0/1780729409 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5456 (secure 0 0 0) 0x7f87d807a760 con 0x7f87f00a7280 2024-03-06T07:22:27.631 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:27.632 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5087,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.98:6837/2828783606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2828783606},{"type":"v1","addr":"172.21.15.98:6837","nonce":2828783606}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:05.671602+0000","modified":"2024-03-06T07:22:26.744431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5084":{"gid":5084,"name":"b","rank":0,"incarnation":36,"state":"up:replay","state_seq":7,"addr":"172.21.15.98:6836/2937068915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2937068915},{"type":"v1","addr":"172.21.15.98:6836","nonce":2937068915}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:21.047528+0000","modified":"2024-03-06T07:22:24.069892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5344":{"gid":5344,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":7,"addr":"172.21.15.1:6835/3845300501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":3845300501},{"type":"v1","addr":"172.21.15.1:6835","nonce":3845300501}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5344,"qdb_cluster":[5344]},"id":4}]} 2024-03-06T07:22:27.632 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 36 2024-03-06T07:22:27.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.632+0000 7f87f4cd4640 1 -- 172.21.15.1:0/1780729409 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f87d006b4d0 msgr2=0x7f87d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.632+0000 7f87f4cd4640 1 --2- 172.21.15.1:0/1780729409 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f87d006b4d0 0x7f87d006d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f87f00c81f0 tx=0x7f87e0009830 comp rx=0 tx=0).stop 2024-03-06T07:22:27.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.632+0000 7f87f4cd4640 1 -- 172.21.15.1:0/1780729409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87f00a7280 msgr2=0x7f87f01ddf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.632+0000 7f87f4cd4640 1 --2- 172.21.15.1:0/1780729409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87f00a7280 0x7f87f01ddf80 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f87d802b950 tx=0x7f87d80046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.632+0000 7f87f4cd4640 1 -- 172.21.15.1:0/1780729409 shutdown_connections 2024-03-06T07:22:27.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.632+0000 7f87f4cd4640 1 --2- 172.21.15.1:0/1780729409 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f87d006b4d0 0x7f87d006d990 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.632+0000 7f87f4cd4640 1 --2- 172.21.15.1:0/1780729409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00c4ef0 0x7f87f01de4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.632+0000 7f87f4cd4640 1 --2- 172.21.15.1:0/1780729409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87f00a7280 0x7f87f01ddf80 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.632+0000 7f87f4cd4640 1 --2- 172.21.15.1:0/1780729409 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87f00732b0 0x7f87f01e1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.632+0000 7f87f4cd4640 1 -- 172.21.15.1:0/1780729409 >> 172.21.15.1:0/1780729409 conn(0x7f87f00c7a50 msgr2=0x7f87f00a7660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.632+0000 7f87f4cd4640 1 -- 172.21.15.1:0/1780729409 shutdown_connections 2024-03-06T07:22:27.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:27.632+0000 7f87f4cd4640 1 -- 172.21.15.1:0/1780729409 wait complete. 2024-03-06T07:22:27.644 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-03-06T07:22:29.281 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:22:30.922 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:30.924 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:32.646 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:32.753 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-06T07:22:32.751+0000 7fd085d40640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.752+0000 7fed56a15640 1 Processor -- start 2024-03-06T07:22:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.752+0000 7fed56a15640 1 -- start start 2024-03-06T07:22:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.753+0000 7fed56a15640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fed500bf9d0 0x7fed50155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.753+0000 7fed56a15640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed500c2fc0 con 0x7fed500c6b10 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.753+0000 7fed56a15640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed50158da0 con 0x7fed50152650 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.753+0000 7fed56a15640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed50155800 con 0x7fed500bf9d0 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.753+0000 7fed4f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fed500c6b10 0x7fed500c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43902/0 (socket says 172.21.15.1:43902) 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.753+0000 7fed4f7fe640 1 -- 172.21.15.1:0/3187685612 learned_addr learned my addr 172.21.15.1:0/3187685612 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.753+0000 7fed54fb2640 1 --2- 172.21.15.1:0/3187685612 >> v2:172.21.15.98:3300/0 conn(0x7fed500bf9d0 0x7fed50155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.754+0000 7fed4effd640 1 -- 172.21.15.1:0/3187685612 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163366069 0 0) 0x7fed500c2fc0 con 0x7fed500c6b10 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.754+0000 7fed4effd640 1 -- 172.21.15.1:0/3187685612 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed38003520 con 0x7fed500c6b10 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.754+0000 7fed4effd640 1 -- 172.21.15.1:0/3187685612 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146971253 0 0) 0x7fed50158da0 con 0x7fed50152650 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.754+0000 7fed4effd640 1 -- 172.21.15.1:0/3187685612 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed500c2fc0 con 0x7fed50152650 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.754+0000 7fed54fb2640 1 -- 172.21.15.1:0/3187685612 >> v1:172.21.15.98:6789/0 conn(0x7fed50152650 legacy=0x7fed50158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.754+0000 7fed54fb2640 1 -- 172.21.15.1:0/3187685612 >> v1:172.21.15.1:6789/0 conn(0x7fed500c6b10 legacy=0x7fed500c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.754+0000 7fed54fb2640 1 -- 172.21.15.1:0/3187685612 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed500c6690 con 0x7fed500bf9d0 2024-03-06T07:22:32.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.755+0000 7fed54fb2640 1 --2- 172.21.15.1:0/3187685612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed500bf9d0 0x7fed50155c90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fed48009d90 tx=0x7fed4802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a1272a9da047c50 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.755+0000 7fed4effd640 1 -- 172.21.15.1:0/3187685612 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed48039070 con 0x7fed500bf9d0 2024-03-06T07:22:32.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.755+0000 7fed4effd640 1 -- 172.21.15.1:0/3187685612 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4802cc60 con 0x7fed500bf9d0 2024-03-06T07:22:32.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.755+0000 7fed4effd640 1 -- 172.21.15.1:0/3187685612 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed48034640 con 0x7fed500bf9d0 2024-03-06T07:22:32.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.756+0000 7fed56a15640 1 -- 172.21.15.1:0/3187685612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed500bf9d0 msgr2=0x7fed50155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.756+0000 7fed56a15640 1 --2- 172.21.15.1:0/3187685612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed500bf9d0 0x7fed50155c90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fed48009d90 tx=0x7fed4802beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.756+0000 7fed56a15640 1 -- 172.21.15.1:0/3187685612 shutdown_connections 2024-03-06T07:22:32.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.756+0000 7fed56a15640 1 --2- 172.21.15.1:0/3187685612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed500bf9d0 0x7fed50155c90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.756+0000 7fed56a15640 1 -- 172.21.15.1:0/3187685612 >> 172.21.15.1:0/3187685612 conn(0x7fed50066ee0 msgr2=0x7fed500672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.756+0000 7fed56a15640 1 -- 172.21.15.1:0/3187685612 shutdown_connections 2024-03-06T07:22:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.756+0000 7fed56a15640 1 -- 172.21.15.1:0/3187685612 wait complete. 2024-03-06T07:22:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.757+0000 7fed56a15640 1 Processor -- start 2024-03-06T07:22:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.757+0000 7fed56a15640 1 -- start start 2024-03-06T07:22:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.758+0000 7fed56a15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed500bf9d0 0x7fed50153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.758+0000 7fed56a15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed50152650 0x7fed501eaff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.758+0000 7fed56a15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed501eb530 0x7fed5014cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.758+0000 7fed56a15640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed501551b0 con 0x7fed50152650 2024-03-06T07:22:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.758+0000 7fed56a15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed500c6fa0 con 0x7fed500bf9d0 2024-03-06T07:22:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.758+0000 7fed56a15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed50155490 con 0x7fed501eb530 2024-03-06T07:22:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.758+0000 7fed4f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed50152650 0x7fed501eaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.758+0000 7fed4f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed50152650 0x7fed501eaff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47222/0 (socket says 172.21.15.1:47222) 2024-03-06T07:22:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.758+0000 7fed4f7fe640 1 -- 172.21.15.1:0/3436130900 learned_addr learned my addr 172.21.15.1:0/3436130900 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.758+0000 7fed54fb2640 1 --2- 172.21.15.1:0/3436130900 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed501eb530 0x7fed5014cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.758+0000 7fed4ffff640 1 --2- 172.21.15.1:0/3436130900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed500bf9d0 0x7fed50153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.758+0000 7fed4f7fe640 1 -- 172.21.15.1:0/3436130900 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed501eb530 msgr2=0x7fed5014cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.758+0000 7fed4f7fe640 1 --2- 172.21.15.1:0/3436130900 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed501eb530 0x7fed5014cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.758+0000 7fed4f7fe640 1 -- 172.21.15.1:0/3436130900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed500bf9d0 msgr2=0x7fed50153cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.758+0000 7fed4f7fe640 1 --2- 172.21.15.1:0/3436130900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed500bf9d0 0x7fed50153cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.758+0000 7fed4f7fe640 1 -- 172.21.15.1:0/3436130900 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed501ef170 con 0x7fed50152650 2024-03-06T07:22:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.758+0000 7fed4f7fe640 1 --2- 172.21.15.1:0/3436130900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed50152650 0x7fed501eaff0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fed4400c530 tx=0x7fed4400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.759+0000 7fed4cff9640 1 -- 172.21.15.1:0/3436130900 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44017070 con 0x7fed50152650 2024-03-06T07:22:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.759+0000 7fed4cff9640 1 -- 172.21.15.1:0/3436130900 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed44005ce0 con 0x7fed50152650 2024-03-06T07:22:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.759+0000 7fed56a15640 1 -- 172.21.15.1:0/3436130900 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed501ee560 con 0x7fed50152650 2024-03-06T07:22:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.759+0000 7fed4cff9640 1 -- 172.21.15.1:0/3436130900 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44007270 con 0x7fed50152650 2024-03-06T07:22:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.759+0000 7fed56a15640 1 -- 172.21.15.1:0/3436130900 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed501eeb20 con 0x7fed50152650 2024-03-06T07:22:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.760+0000 7fed56a15640 1 -- 172.21.15.1:0/3436130900 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed50067fd0 con 0x7fed50152650 2024-03-06T07:22:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.760+0000 7fed4cff9640 1 -- 172.21.15.1:0/3436130900 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed4401a070 con 0x7fed50152650 2024-03-06T07:22:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.761+0000 7fed4cff9640 1 --2- 172.21.15.1:0/3436130900 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fed3c06b4d0 0x7fed3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.761+0000 7fed4cff9640 1 -- 172.21.15.1:0/3436130900 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fed44089aa0 con 0x7fed50152650 2024-03-06T07:22:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.761+0000 7fed4ffff640 1 --2- 172.21.15.1:0/3436130900 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fed3c06b4d0 0x7fed3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.761+0000 7fed4ffff640 1 --2- 172.21.15.1:0/3436130900 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fed3c06b4d0 0x7fed3c06d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fed400011e0 tx=0x7fed40009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:32.765+0000 7fed4cff9640 1 -- 172.21.15.1:0/3436130900 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed44057cc0 con 0x7fed50152650 2024-03-06T07:22:33.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.136+0000 7fed56a15640 1 -- 172.21.15.1:0/3436130900 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed50154e60 con 0x7fed50152650 2024-03-06T07:22:33.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.137+0000 7fed4cff9640 1 -- 172.21.15.1:0/3436130900 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6251 (secure 0 0 0) 0x7fed4405bd60 con 0x7fed50152650 2024-03-06T07:22:33.139 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:33.139 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":42,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5087,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.98:6837/2828783606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2828783606},{"type":"v1","addr":"172.21.15.98:6837","nonce":2828783606}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":5258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2604497680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2604497680},{"type":"v1","addr":"172.21.15.98:6836","nonce":2604497680}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:05.671602+0000","modified":"2024-03-06T07:22:32.254637+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6105":{"gid":6105,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":4,"addr":"172.21.15.1:6834/4244709430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4244709430},{"type":"v1","addr":"172.21.15.1:6834","nonce":4244709430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6105,"qdb_cluster":[6105]},"id":3},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:21.047528+0000","modified":"2024-03-06T07:22:24.069892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5344":{"gid":5344,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":7,"addr":"172.21.15.1:6835/3845300501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":3845300501},{"type":"v1","addr":"172.21.15.1:6835","nonce":3845300501}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5344,"qdb_cluster":[5344]},"id":4}]} 2024-03-06T07:22:33.139 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 42 2024-03-06T07:22:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.139+0000 7fed56a15640 1 -- 172.21.15.1:0/3436130900 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fed3c06b4d0 msgr2=0x7fed3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.139+0000 7fed56a15640 1 --2- 172.21.15.1:0/3436130900 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fed3c06b4d0 0x7fed3c06d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fed400011e0 tx=0x7fed40009040 comp rx=0 tx=0).stop 2024-03-06T07:22:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.139+0000 7fed56a15640 1 -- 172.21.15.1:0/3436130900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed50152650 msgr2=0x7fed501eaff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.139+0000 7fed56a15640 1 --2- 172.21.15.1:0/3436130900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed50152650 0x7fed501eaff0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fed4400c530 tx=0x7fed4400c9f0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.139+0000 7fed56a15640 1 -- 172.21.15.1:0/3436130900 shutdown_connections 2024-03-06T07:22:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.139+0000 7fed56a15640 1 --2- 172.21.15.1:0/3436130900 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fed3c06b4d0 0x7fed3c06d990 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.139+0000 7fed56a15640 1 --2- 172.21.15.1:0/3436130900 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed501eb530 0x7fed5014cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.139+0000 7fed56a15640 1 --2- 172.21.15.1:0/3436130900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed50152650 0x7fed501eaff0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.139+0000 7fed56a15640 1 --2- 172.21.15.1:0/3436130900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed500bf9d0 0x7fed50153cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.139+0000 7fed56a15640 1 -- 172.21.15.1:0/3436130900 >> 172.21.15.1:0/3436130900 conn(0x7fed50066ee0 msgr2=0x7fed500caf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.139+0000 7fed56a15640 1 -- 172.21.15.1:0/3436130900 shutdown_connections 2024-03-06T07:22:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.139+0000 7fed56a15640 1 -- 172.21.15.1:0/3436130900 wait complete. 2024-03-06T07:22:33.151 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:22:33.152 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:22:33.152 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:33.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:33.232 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:22:33.232 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:22:33.233 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:33.247 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:22:33.247 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:22:33.247 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:33.289 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:22:33.290 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:33.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.390+0000 7ff4b86ed640 1 Processor -- start 2024-03-06T07:22:33.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.390+0000 7ff4b86ed640 1 -- start start 2024-03-06T07:22:33.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.391+0000 7ff4b86ed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4b0152650 0x7ff4b0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.391+0000 7ff4b86ed640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff4b00bf9d0 0x7ff4b00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.391+0000 7ff4b86ed640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b00c2fc0 con 0x7ff4b00c6b10 2024-03-06T07:22:33.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.391+0000 7ff4b86ed640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4b0155800 con 0x7ff4b00bf9d0 2024-03-06T07:22:33.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.391+0000 7ff4b86ed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b0158da0 con 0x7ff4b0152650 2024-03-06T07:22:33.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.391+0000 7ff4b6489640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4b0152650 0x7ff4b0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.391+0000 7ff4b6c8a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff4b00bf9d0 0x7ff4b00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.391+0000 7ff4b5c88640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff4b00c6b10 0x7ff4b0158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:42052/0 (socket says 172.21.15.1:42052) 2024-03-06T07:22:33.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.391+0000 7ff4b5c88640 1 -- 172.21.15.1:0/940415244 learned_addr learned my addr 172.21.15.1:0/940415244 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:33.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.392+0000 7ff4b5487640 1 -- 172.21.15.1:0/940415244 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601504770 0 0) 0x7ff4b00c2fc0 con 0x7ff4b00c6b10 2024-03-06T07:22:33.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.392+0000 7ff4b5487640 1 -- 172.21.15.1:0/940415244 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff498003520 con 0x7ff4b00c6b10 2024-03-06T07:22:33.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.392+0000 7ff4b6489640 1 -- 172.21.15.1:0/940415244 >> v2:172.21.15.98:3300/0 conn(0x7ff4b00bf9d0 msgr2=0x7ff4b00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.392+0000 7ff4b6489640 1 --2- 172.21.15.1:0/940415244 >> v2:172.21.15.98:3300/0 conn(0x7ff4b00bf9d0 0x7ff4b00c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.392+0000 7ff4b6489640 1 -- 172.21.15.1:0/940415244 >> v1:172.21.15.98:6789/0 conn(0x7ff4b00c6b10 legacy=0x7ff4b0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.392+0000 7ff4b6489640 1 -- 172.21.15.1:0/940415244 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b00c6690 con 0x7ff4b0152650 2024-03-06T07:22:33.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.392+0000 7ff4b6c8a640 1 --2- 172.21.15.1:0/940415244 >> v2:172.21.15.98:3300/0 conn(0x7ff4b00bf9d0 0x7ff4b00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:33.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.393+0000 7ff4b6489640 1 --2- 172.21.15.1:0/940415244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4b0152650 0x7ff4b0155c90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff4a80097a0 tx=0x7ff4a802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5db0af217eb22ae2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.393+0000 7ff4b5487640 1 -- 172.21.15.1:0/940415244 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a80315a0 con 0x7ff4b0152650 2024-03-06T07:22:33.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.393+0000 7ff4b5487640 1 -- 172.21.15.1:0/940415244 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a8037070 con 0x7ff4b0152650 2024-03-06T07:22:33.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.393+0000 7ff4b5487640 1 -- 172.21.15.1:0/940415244 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8031960 con 0x7ff4b0152650 2024-03-06T07:22:33.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.394+0000 7ff4b86ed640 1 -- 172.21.15.1:0/940415244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4b0152650 msgr2=0x7ff4b0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.394+0000 7ff4b86ed640 1 --2- 172.21.15.1:0/940415244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4b0152650 0x7ff4b0155c90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff4a80097a0 tx=0x7ff4a802b840 comp rx=0 tx=0).stop 2024-03-06T07:22:33.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.394+0000 7ff4b86ed640 1 -- 172.21.15.1:0/940415244 shutdown_connections 2024-03-06T07:22:33.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.394+0000 7ff4b86ed640 1 --2- 172.21.15.1:0/940415244 >> v2:172.21.15.98:3300/0 conn(0x7ff4b00bf9d0 0x7ff4b00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.394+0000 7ff4b86ed640 1 --2- 172.21.15.1:0/940415244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4b0152650 0x7ff4b0155c90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.394+0000 7ff4b86ed640 1 -- 172.21.15.1:0/940415244 >> 172.21.15.1:0/940415244 conn(0x7ff4b0066ee0 msgr2=0x7ff4b00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.394+0000 7ff4b86ed640 1 -- 172.21.15.1:0/940415244 shutdown_connections 2024-03-06T07:22:33.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.394+0000 7ff4b86ed640 1 -- 172.21.15.1:0/940415244 wait complete. 2024-03-06T07:22:33.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.395+0000 7ff4b86ed640 1 Processor -- start 2024-03-06T07:22:33.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.395+0000 7ff4b86ed640 1 -- start start 2024-03-06T07:22:33.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.396+0000 7ff4b86ed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4b00bf9d0 0x7ff4b0156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.396+0000 7ff4b86ed640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4b0152650 0x7ff4b01ee470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.396+0000 7ff4b86ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4b01ee9b0 0x7ff4b01eac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.396+0000 7ff4b86ed640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b01551b0 con 0x7ff4b0152650 2024-03-06T07:22:33.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.396+0000 7ff4b86ed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b00c6fa0 con 0x7ff4b01ee9b0 2024-03-06T07:22:33.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.396+0000 7ff4b86ed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b0155490 con 0x7ff4b00bf9d0 2024-03-06T07:22:33.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.397+0000 7ff4b5c88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4b0152650 0x7ff4b01ee470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.397+0000 7ff4b5c88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4b0152650 0x7ff4b01ee470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47228/0 (socket says 172.21.15.1:47228) 2024-03-06T07:22:33.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.397+0000 7ff4b5c88640 1 -- 172.21.15.1:0/1994003313 learned_addr learned my addr 172.21.15.1:0/1994003313 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:33.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.397+0000 7ff4b6489640 1 --2- 172.21.15.1:0/1994003313 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4b00bf9d0 0x7ff4b0156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.397+0000 7ff4b6c8a640 1 --2- 172.21.15.1:0/1994003313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4b01ee9b0 0x7ff4b01eac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.397+0000 7ff4b5c88640 1 -- 172.21.15.1:0/1994003313 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4b00bf9d0 msgr2=0x7ff4b0156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.397+0000 7ff4b5c88640 1 --2- 172.21.15.1:0/1994003313 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4b00bf9d0 0x7ff4b0156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.397+0000 7ff4b5c88640 1 -- 172.21.15.1:0/1994003313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4b01ee9b0 msgr2=0x7ff4b01eac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.397+0000 7ff4b5c88640 1 --2- 172.21.15.1:0/1994003313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4b01ee9b0 0x7ff4b01eac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.397+0000 7ff4b5c88640 1 -- 172.21.15.1:0/1994003313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b01eee60 con 0x7ff4b0152650 2024-03-06T07:22:33.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.397+0000 7ff4b5c88640 1 --2- 172.21.15.1:0/1994003313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4b0152650 0x7ff4b01ee470 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400c3c0 tx=0x7ff4a400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.397+0000 7ff4aeffd640 1 -- 172.21.15.1:0/1994003313 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4017070 con 0x7ff4b0152650 2024-03-06T07:22:33.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.397+0000 7ff4aeffd640 1 -- 172.21.15.1:0/1994003313 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a4005ce0 con 0x7ff4b0152650 2024-03-06T07:22:33.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.397+0000 7ff4b86ed640 1 -- 172.21.15.1:0/1994003313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b01e0e30 con 0x7ff4b0152650 2024-03-06T07:22:33.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.397+0000 7ff4aeffd640 1 -- 172.21.15.1:0/1994003313 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a40072f0 con 0x7ff4b0152650 2024-03-06T07:22:33.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.397+0000 7ff4b86ed640 1 -- 172.21.15.1:0/1994003313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b014dbe0 con 0x7ff4b0152650 2024-03-06T07:22:33.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.399+0000 7ff4b86ed640 1 -- 172.21.15.1:0/1994003313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b0067fd0 con 0x7ff4b0152650 2024-03-06T07:22:33.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.399+0000 7ff4aeffd640 1 -- 172.21.15.1:0/1994003313 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4a401a070 con 0x7ff4b0152650 2024-03-06T07:22:33.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.399+0000 7ff4aeffd640 1 --2- 172.21.15.1:0/1994003313 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff49c06b4d0 0x7ff49c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.399+0000 7ff4aeffd640 1 -- 172.21.15.1:0/1994003313 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7ff4a40899f0 con 0x7ff4b0152650 2024-03-06T07:22:33.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.400+0000 7ff4b6489640 1 --2- 172.21.15.1:0/1994003313 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff49c06b4d0 0x7ff49c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.400+0000 7ff4b6489640 1 --2- 172.21.15.1:0/1994003313 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff49c06b4d0 0x7ff49c06d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8036000 tx=0x7ff4a803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.404+0000 7ff4aeffd640 1 -- 172.21.15.1:0/1994003313 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4a4057c80 con 0x7ff4b0152650 2024-03-06T07:22:33.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.727+0000 7ff4b86ed640 1 -- 172.21.15.1:0/1994003313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4b0154ec0 con 0x7ff4b0152650 2024-03-06T07:22:33.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.728+0000 7ff4aeffd640 1 -- 172.21.15.1:0/1994003313 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6251 (secure 0 0 0) 0x7ff4a405bd20 con 0x7ff4b0152650 2024-03-06T07:22:33.730 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:33.730 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":42,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5087,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.98:6837/2828783606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2828783606},{"type":"v1","addr":"172.21.15.98:6837","nonce":2828783606}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":5258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2604497680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2604497680},{"type":"v1","addr":"172.21.15.98:6836","nonce":2604497680}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:05.671602+0000","modified":"2024-03-06T07:22:32.254637+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6105":{"gid":6105,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":4,"addr":"172.21.15.1:6834/4244709430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4244709430},{"type":"v1","addr":"172.21.15.1:6834","nonce":4244709430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6105,"qdb_cluster":[6105]},"id":3},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:21.047528+0000","modified":"2024-03-06T07:22:24.069892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5344":{"gid":5344,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":7,"addr":"172.21.15.1:6835/3845300501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":3845300501},{"type":"v1","addr":"172.21.15.1:6835","nonce":3845300501}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5344,"qdb_cluster":[5344]},"id":4}]} 2024-03-06T07:22:33.730 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 42 2024-03-06T07:22:33.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.730+0000 7ff4b86ed640 1 -- 172.21.15.1:0/1994003313 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff49c06b4d0 msgr2=0x7ff49c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.730+0000 7ff4b86ed640 1 --2- 172.21.15.1:0/1994003313 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff49c06b4d0 0x7ff49c06d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8036000 tx=0x7ff4a803d040 comp rx=0 tx=0).stop 2024-03-06T07:22:33.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.730+0000 7ff4b86ed640 1 -- 172.21.15.1:0/1994003313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4b0152650 msgr2=0x7ff4b01ee470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.730+0000 7ff4b86ed640 1 --2- 172.21.15.1:0/1994003313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4b0152650 0x7ff4b01ee470 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400c3c0 tx=0x7ff4a400c7c0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.730+0000 7ff4b86ed640 1 -- 172.21.15.1:0/1994003313 shutdown_connections 2024-03-06T07:22:33.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.730+0000 7ff4b86ed640 1 --2- 172.21.15.1:0/1994003313 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff49c06b4d0 0x7ff49c06d990 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.730+0000 7ff4b86ed640 1 --2- 172.21.15.1:0/1994003313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4b01ee9b0 0x7ff4b01eac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.730+0000 7ff4b86ed640 1 --2- 172.21.15.1:0/1994003313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4b0152650 0x7ff4b01ee470 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.730+0000 7ff4b86ed640 1 --2- 172.21.15.1:0/1994003313 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4b00bf9d0 0x7ff4b0156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.730+0000 7ff4b86ed640 1 -- 172.21.15.1:0/1994003313 >> 172.21.15.1:0/1994003313 conn(0x7ff4b0066ee0 msgr2=0x7ff4b00c5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.730+0000 7ff4b86ed640 1 -- 172.21.15.1:0/1994003313 shutdown_connections 2024-03-06T07:22:33.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.730+0000 7ff4b86ed640 1 -- 172.21.15.1:0/1994003313 wait complete. 2024-03-06T07:22:33.743 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:33.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.846+0000 7f210aa6f640 1 Processor -- start 2024-03-06T07:22:33.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.846+0000 7f210aa6f640 1 -- start start 2024-03-06T07:22:33.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.847+0000 7f210aa6f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f21040c6b10 0x7f2104155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.847+0000 7f210aa6f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21040c2fc0 con 0x7f21040bf9d0 2024-03-06T07:22:33.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.847+0000 7f210aa6f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2104158da0 con 0x7f2104152650 2024-03-06T07:22:33.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.847+0000 7f210aa6f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2104155800 con 0x7f21040c6b10 2024-03-06T07:22:33.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.847+0000 7f2103fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f21040c6b10 0x7f2104155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.847+0000 7f210900c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f21040bf9d0 0x7f21040c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43918/0 (socket says 172.21.15.1:43918) 2024-03-06T07:22:33.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.847+0000 7f210900c640 1 -- 172.21.15.1:0/3633211 learned_addr learned my addr 172.21.15.1:0/3633211 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:33.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.847+0000 7f2103fff640 1 -- 172.21.15.1:0/3633211 >> v1:172.21.15.98:6789/0 conn(0x7f2104152650 legacy=0x7f2104158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.847+0000 7f2103fff640 1 -- 172.21.15.1:0/3633211 >> v1:172.21.15.1:6789/0 conn(0x7f21040bf9d0 legacy=0x7f21040c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.847+0000 7f2103fff640 1 -- 172.21.15.1:0/3633211 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21040c6690 con 0x7f21040c6b10 2024-03-06T07:22:33.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.847+0000 7f21037fe640 1 -- 172.21.15.1:0/3633211 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021087130 0 0) 0x7f21040c2fc0 con 0x7f21040bf9d0 2024-03-06T07:22:33.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.848+0000 7f2103fff640 1 --2- 172.21.15.1:0/3633211 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21040c6b10 0x7f2104155c90 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f20f00097a0 tx=0x7f20f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef9859e63861cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.848+0000 7f21037fe640 1 -- 172.21.15.1:0/3633211 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f0035020 con 0x7f21040c6b10 2024-03-06T07:22:33.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.848+0000 7f21037fe640 1 -- 172.21.15.1:0/3633211 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f0031830 con 0x7f21040c6b10 2024-03-06T07:22:33.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.848+0000 7f21037fe640 1 -- 172.21.15.1:0/3633211 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f0031c00 con 0x7f21040c6b10 2024-03-06T07:22:33.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.849+0000 7f210aa6f640 1 -- 172.21.15.1:0/3633211 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21040c6b10 msgr2=0x7f2104155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.849+0000 7f210aa6f640 1 --2- 172.21.15.1:0/3633211 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21040c6b10 0x7f2104155c90 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f20f00097a0 tx=0x7f20f002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.849+0000 7f210aa6f640 1 -- 172.21.15.1:0/3633211 shutdown_connections 2024-03-06T07:22:33.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.849+0000 7f210aa6f640 1 --2- 172.21.15.1:0/3633211 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21040c6b10 0x7f2104155c90 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.849+0000 7f210aa6f640 1 -- 172.21.15.1:0/3633211 >> 172.21.15.1:0/3633211 conn(0x7f2104066ee0 msgr2=0x7f21040672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.849+0000 7f210aa6f640 1 -- 172.21.15.1:0/3633211 shutdown_connections 2024-03-06T07:22:33.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.849+0000 7f210aa6f640 1 -- 172.21.15.1:0/3633211 wait complete. 2024-03-06T07:22:33.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.850+0000 7f210aa6f640 1 Processor -- start 2024-03-06T07:22:33.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.850+0000 7f210aa6f640 1 -- start start 2024-03-06T07:22:33.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.851+0000 7f210aa6f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21040bf9d0 0x7f2104154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.851+0000 7f210aa6f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2104152650 0x7f21041eaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.851+0000 7f210aa6f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21040c6020 0x7f21041eb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.851+0000 7f210aa6f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21041551b0 con 0x7f21040c6020 2024-03-06T07:22:33.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.851+0000 7f210aa6f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21040c6fa0 con 0x7f21040bf9d0 2024-03-06T07:22:33.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.851+0000 7f210aa6f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2104155490 con 0x7f2104152650 2024-03-06T07:22:33.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.851+0000 7f210900c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21040c6020 0x7f21041eb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.851+0000 7f210900c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21040c6020 0x7f21041eb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47242/0 (socket says 172.21.15.1:47242) 2024-03-06T07:22:33.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.851+0000 7f210900c640 1 -- 172.21.15.1:0/1600709304 learned_addr learned my addr 172.21.15.1:0/1600709304 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:33.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.852+0000 7f210880b640 1 --2- 172.21.15.1:0/1600709304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21040bf9d0 0x7f2104154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.852+0000 7f2103fff640 1 --2- 172.21.15.1:0/1600709304 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2104152650 0x7f21041eaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.852+0000 7f210900c640 1 -- 172.21.15.1:0/1600709304 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2104152650 msgr2=0x7f21041eaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.852+0000 7f210900c640 1 --2- 172.21.15.1:0/1600709304 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2104152650 0x7f21041eaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.852+0000 7f210900c640 1 -- 172.21.15.1:0/1600709304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21040bf9d0 msgr2=0x7f2104154770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.852+0000 7f210900c640 1 --2- 172.21.15.1:0/1600709304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21040bf9d0 0x7f2104154770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.852+0000 7f210900c640 1 -- 172.21.15.1:0/1600709304 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21041ef120 con 0x7f21040c6020 2024-03-06T07:22:33.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.852+0000 7f210900c640 1 --2- 172.21.15.1:0/1600709304 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21040c6020 0x7f21041eb490 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00c730 tx=0x7f20fc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.852+0000 7f21017fa640 1 -- 172.21.15.1:0/1600709304 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc018070 con 0x7f21040c6020 2024-03-06T07:22:33.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.852+0000 7f21017fa640 1 -- 172.21.15.1:0/1600709304 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20fc00ce40 con 0x7f21040c6020 2024-03-06T07:22:33.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.852+0000 7f210aa6f640 1 -- 172.21.15.1:0/1600709304 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21041eee40 con 0x7f21040c6020 2024-03-06T07:22:33.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.852+0000 7f21017fa640 1 -- 172.21.15.1:0/1600709304 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc00a860 con 0x7f21040c6020 2024-03-06T07:22:33.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.852+0000 7f210aa6f640 1 -- 172.21.15.1:0/1600709304 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21040c9c10 con 0x7f21040c6020 2024-03-06T07:22:33.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.853+0000 7f210aa6f640 1 -- 172.21.15.1:0/1600709304 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2104067fd0 con 0x7f21040c6020 2024-03-06T07:22:33.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.854+0000 7f21017fa640 1 -- 172.21.15.1:0/1600709304 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f20fc01b070 con 0x7f21040c6020 2024-03-06T07:22:33.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.855+0000 7f21017fa640 1 --2- 172.21.15.1:0/1600709304 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20e806b4d0 0x7f20e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.855+0000 7f21017fa640 1 -- 172.21.15.1:0/1600709304 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f20fc089c60 con 0x7f21040c6020 2024-03-06T07:22:33.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.855+0000 7f210880b640 1 --2- 172.21.15.1:0/1600709304 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20e806b4d0 0x7f20e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.856+0000 7f210880b640 1 --2- 172.21.15.1:0/1600709304 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20e806b4d0 0x7f20e806d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f21040c50b0 tx=0x7f20f8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:33.863+0000 7f21017fa640 1 -- 172.21.15.1:0/1600709304 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f20fc0585e0 con 0x7f21040c6020 2024-03-06T07:22:34.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.067+0000 7f210aa6f640 1 -- 172.21.15.1:0/1600709304 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21040b06f0 con 0x7f21040c6020 2024-03-06T07:22:34.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.069+0000 7f21017fa640 1 -- 172.21.15.1:0/1600709304 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+20801 (secure 0 0 0) 0x7f20fc057ee0 con 0x7f21040c6020 2024-03-06T07:22:34.072 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:34.072 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":58,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:29.247339+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:03.930333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:04.634569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:22:18.907422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:22:20.028174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6836/2937068915":"2024-03-07T07:22:29.247312+0000","172.21.15.98:6834/2937068915":"2024-03-07T07:22:29.247312+0000","172.21.15.1:6834/131367647":"2024-03-07T07:22:26.740110+0000","172.21.15.1:6832/131367647":"2024-03-07T07:22:26.740110+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:34.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.072+0000 7f210aa6f640 1 -- 172.21.15.1:0/1600709304 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20e806b4d0 msgr2=0x7f20e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.072+0000 7f210aa6f640 1 --2- 172.21.15.1:0/1600709304 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20e806b4d0 0x7f20e806d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f21040c50b0 tx=0x7f20f8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:34.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.072+0000 7f210aa6f640 1 -- 172.21.15.1:0/1600709304 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21040c6020 msgr2=0x7f21041eb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.072+0000 7f210aa6f640 1 --2- 172.21.15.1:0/1600709304 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21040c6020 0x7f21041eb490 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00c730 tx=0x7f20fc00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.072+0000 7f210aa6f640 1 -- 172.21.15.1:0/1600709304 shutdown_connections 2024-03-06T07:22:34.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.072+0000 7f210aa6f640 1 --2- 172.21.15.1:0/1600709304 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20e806b4d0 0x7f20e806d990 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.072+0000 7f210aa6f640 1 --2- 172.21.15.1:0/1600709304 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21040c6020 0x7f21041eb490 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.072+0000 7f210aa6f640 1 --2- 172.21.15.1:0/1600709304 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2104152650 0x7f21041eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.072+0000 7f210aa6f640 1 --2- 172.21.15.1:0/1600709304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21040bf9d0 0x7f2104154770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.072+0000 7f210aa6f640 1 -- 172.21.15.1:0/1600709304 >> 172.21.15.1:0/1600709304 conn(0x7f2104066ee0 msgr2=0x7f21040c3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.072+0000 7f210aa6f640 1 -- 172.21.15.1:0/1600709304 shutdown_connections 2024-03-06T07:22:34.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.072+0000 7f210aa6f640 1 -- 172.21.15.1:0/1600709304 wait complete. 2024-03-06T07:22:34.089 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:03.930333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:04.634569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-06T07:22:18.907422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-06T07:22:20.028174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:34.090 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.192+0000 7f5e6d9ac640 1 Processor -- start 2024-03-06T07:22:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.192+0000 7f5e6d9ac640 1 -- start start 2024-03-06T07:22:34.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.193+0000 7f5e6d9ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e680c6b10 0x7f5e68155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.193+0000 7f5e6d9ac640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5e680bf9d0 0x7f5e680c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.193+0000 7f5e6d9ac640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e680c2fc0 con 0x7f5e68152650 2024-03-06T07:22:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.193+0000 7f5e6d9ac640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e68155800 con 0x7f5e680bf9d0 2024-03-06T07:22:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.193+0000 7f5e6d9ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e68158da0 con 0x7f5e680c6b10 2024-03-06T07:22:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.193+0000 7f5e677fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5e680bf9d0 0x7f5e680c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.193+0000 7f5e677fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5e680bf9d0 0x7f5e680c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47254/0 (socket says 172.21.15.1:47254) 2024-03-06T07:22:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.193+0000 7f5e677fe640 1 -- 172.21.15.1:0/414204074 learned_addr learned my addr 172.21.15.1:0/414204074 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.193+0000 7f5e667fc640 1 --2- 172.21.15.1:0/414204074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e680c6b10 0x7f5e68155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.193+0000 7f5e677fe640 1 -- 172.21.15.1:0/414204074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e680c6b10 msgr2=0x7f5e68155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.193+0000 7f5e677fe640 1 --2- 172.21.15.1:0/414204074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e680c6b10 0x7f5e68155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.193+0000 7f5e677fe640 1 -- 172.21.15.1:0/414204074 >> v1:172.21.15.98:6789/0 conn(0x7f5e68152650 legacy=0x7f5e68158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.193+0000 7f5e677fe640 1 -- 172.21.15.1:0/414204074 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e680c6690 con 0x7f5e680bf9d0 2024-03-06T07:22:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.194+0000 7f5e677fe640 1 --2- 172.21.15.1:0/414204074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e680bf9d0 0x7f5e680c1460 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f5e60009e50 tx=0x7f5e6002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d6848127d080551 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.194+0000 7f5e65ffb640 1 -- 172.21.15.1:0/414204074 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6002ca70 con 0x7f5e680bf9d0 2024-03-06T07:22:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.194+0000 7f5e65ffb640 1 -- 172.21.15.1:0/414204074 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6002cc10 con 0x7f5e680bf9d0 2024-03-06T07:22:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.194+0000 7f5e65ffb640 1 -- 172.21.15.1:0/414204074 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e600343f0 con 0x7f5e680bf9d0 2024-03-06T07:22:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.195+0000 7f5e6d9ac640 1 -- 172.21.15.1:0/414204074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e680bf9d0 msgr2=0x7f5e680c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.195+0000 7f5e6d9ac640 1 --2- 172.21.15.1:0/414204074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e680bf9d0 0x7f5e680c1460 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f5e60009e50 tx=0x7f5e6002bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.195+0000 7f5e6d9ac640 1 -- 172.21.15.1:0/414204074 shutdown_connections 2024-03-06T07:22:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.195+0000 7f5e6d9ac640 1 --2- 172.21.15.1:0/414204074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e680bf9d0 0x7f5e680c1460 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.195+0000 7f5e6d9ac640 1 --2- 172.21.15.1:0/414204074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e680c6b10 0x7f5e68155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.195+0000 7f5e6d9ac640 1 -- 172.21.15.1:0/414204074 >> 172.21.15.1:0/414204074 conn(0x7f5e68066ee0 msgr2=0x7f5e680672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.195+0000 7f5e6d9ac640 1 -- 172.21.15.1:0/414204074 shutdown_connections 2024-03-06T07:22:34.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.195+0000 7f5e6d9ac640 1 -- 172.21.15.1:0/414204074 wait complete. 2024-03-06T07:22:34.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.196+0000 7f5e6d9ac640 1 Processor -- start 2024-03-06T07:22:34.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.196+0000 7f5e6d9ac640 1 -- start start 2024-03-06T07:22:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.197+0000 7f5e6d9ac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e680bf9d0 0x7f5e681eaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.197+0000 7f5e6d9ac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e68152650 0x7f5e680af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.197+0000 7f5e6d9ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e6814ba60 0x7f5e680afc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.197+0000 7f5e6d9ac640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e681551b0 con 0x7f5e680bf9d0 2024-03-06T07:22:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.197+0000 7f5e6d9ac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e680c6fa0 con 0x7f5e68152650 2024-03-06T07:22:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.197+0000 7f5e6d9ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e68155490 con 0x7f5e6814ba60 2024-03-06T07:22:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.197+0000 7f5e66ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e680bf9d0 0x7f5e681eaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.197+0000 7f5e66ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e680bf9d0 0x7f5e681eaf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47264/0 (socket says 172.21.15.1:47264) 2024-03-06T07:22:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.197+0000 7f5e66ffd640 1 -- 172.21.15.1:0/1121951585 learned_addr learned my addr 172.21.15.1:0/1121951585 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.197+0000 7f5e677fe640 1 --2- 172.21.15.1:0/1121951585 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e6814ba60 0x7f5e680afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.197+0000 7f5e667fc640 1 --2- 172.21.15.1:0/1121951585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e68152650 0x7f5e680af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.197+0000 7f5e66ffd640 1 -- 172.21.15.1:0/1121951585 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e6814ba60 msgr2=0x7f5e680afc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.197+0000 7f5e66ffd640 1 --2- 172.21.15.1:0/1121951585 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e6814ba60 0x7f5e680afc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.197+0000 7f5e66ffd640 1 -- 172.21.15.1:0/1121951585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e68152650 msgr2=0x7f5e680af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.197+0000 7f5e66ffd640 1 --2- 172.21.15.1:0/1121951585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e68152650 0x7f5e680af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.197+0000 7f5e66ffd640 1 -- 172.21.15.1:0/1121951585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e68156530 con 0x7f5e680bf9d0 2024-03-06T07:22:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.198+0000 7f5e66ffd640 1 --2- 172.21.15.1:0/1121951585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e680bf9d0 0x7f5e681eaf70 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c00c430 tx=0x7f5e5c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.198+0000 7f5e4ffff640 1 -- 172.21.15.1:0/1121951585 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c0052d0 con 0x7f5e680bf9d0 2024-03-06T07:22:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.198+0000 7f5e4ffff640 1 -- 172.21.15.1:0/1121951585 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e5c005490 con 0x7f5e680bf9d0 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.198+0000 7f5e6d9ac640 1 -- 172.21.15.1:0/1121951585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e681eb8b0 con 0x7f5e680bf9d0 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.198+0000 7f5e4ffff640 1 -- 172.21.15.1:0/1121951585 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c007260 con 0x7f5e680bf9d0 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.198+0000 7f5e6d9ac640 1 -- 172.21.15.1:0/1121951585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e681ebb90 con 0x7f5e680bf9d0 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.199+0000 7f5e6d9ac640 1 -- 172.21.15.1:0/1121951585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e68067fd0 con 0x7f5e680bf9d0 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.199+0000 7f5e4ffff640 1 -- 172.21.15.1:0/1121951585 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e5c007400 con 0x7f5e680bf9d0 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.200+0000 7f5e4ffff640 1 --2- 172.21.15.1:0/1121951585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e5406b4d0 0x7f5e5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.200+0000 7f5e4ffff640 1 -- 172.21.15.1:0/1121951585 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f5e5c089c30 con 0x7f5e680bf9d0 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.200+0000 7f5e667fc640 1 --2- 172.21.15.1:0/1121951585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e5406b4d0 0x7f5e5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.201+0000 7f5e667fc640 1 --2- 172.21.15.1:0/1121951585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e5406b4d0 0x7f5e5406d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5e68151940 tx=0x7f5e58008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.204+0000 7f5e4ffff640 1 -- 172.21.15.1:0/1121951585 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e5c0585b0 con 0x7f5e680bf9d0 2024-03-06T07:22:34.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.387+0000 7f5e6d9ac640 1 -- 172.21.15.1:0/1121951585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5e680b01f0 con 0x7f5e680bf9d0 2024-03-06T07:22:34.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.388+0000 7f5e4ffff640 1 -- 172.21.15.1:0/1121951585 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5e5c057eb0 con 0x7f5e680bf9d0 2024-03-06T07:22:34.390 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:22:34.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.390+0000 7f5e6d9ac640 1 -- 172.21.15.1:0/1121951585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e5406b4d0 msgr2=0x7f5e5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.390+0000 7f5e6d9ac640 1 --2- 172.21.15.1:0/1121951585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e5406b4d0 0x7f5e5406d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5e68151940 tx=0x7f5e58008040 comp rx=0 tx=0).stop 2024-03-06T07:22:34.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.390+0000 7f5e6d9ac640 1 -- 172.21.15.1:0/1121951585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e680bf9d0 msgr2=0x7f5e681eaf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.390+0000 7f5e6d9ac640 1 --2- 172.21.15.1:0/1121951585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e680bf9d0 0x7f5e681eaf70 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c00c430 tx=0x7f5e5c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.390+0000 7f5e6d9ac640 1 -- 172.21.15.1:0/1121951585 shutdown_connections 2024-03-06T07:22:34.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.390+0000 7f5e6d9ac640 1 --2- 172.21.15.1:0/1121951585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e5406b4d0 0x7f5e5406d990 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.390+0000 7f5e6d9ac640 1 --2- 172.21.15.1:0/1121951585 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e6814ba60 0x7f5e680afc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.390+0000 7f5e6d9ac640 1 --2- 172.21.15.1:0/1121951585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e68152650 0x7f5e680af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.390+0000 7f5e6d9ac640 1 --2- 172.21.15.1:0/1121951585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e680bf9d0 0x7f5e681eaf70 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.390+0000 7f5e6d9ac640 1 -- 172.21.15.1:0/1121951585 >> 172.21.15.1:0/1121951585 conn(0x7f5e68066ee0 msgr2=0x7f5e680cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.390+0000 7f5e6d9ac640 1 -- 172.21.15.1:0/1121951585 shutdown_connections 2024-03-06T07:22:34.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.390+0000 7f5e6d9ac640 1 -- 172.21.15.1:0/1121951585 wait complete. 2024-03-06T07:22:34.403 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:34.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.506+0000 7fdd111f6640 1 Processor -- start 2024-03-06T07:22:34.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.506+0000 7fdd111f6640 1 -- start start 2024-03-06T07:22:34.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd111f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd0c0c60b0 0x7fdd0c0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd111f6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd0c0c5e40 con 0x7fdd0c152840 2024-03-06T07:22:34.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd111f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd0c0c6a40 con 0x7fdd0c0c3970 2024-03-06T07:22:34.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd111f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c0c67f0 con 0x7fdd0c0c60b0 2024-03-06T07:22:34.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd0a59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdd0c152840 0x7fdd0c1553d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37790/0 (socket says 172.21.15.1:37790) 2024-03-06T07:22:34.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd0a59c640 1 -- 172.21.15.1:0/3285851715 learned_addr learned my addr 172.21.15.1:0/3285851715 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:34.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd09d9b640 1 -- 172.21.15.1:0/3285851715 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947296285 0 0) 0x7fdd0c0c5e40 con 0x7fdd0c152840 2024-03-06T07:22:34.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd0b59e640 1 --2- 172.21.15.1:0/3285851715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd0c0c60b0 0x7fdd0c0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd09d9b640 1 -- 172.21.15.1:0/3285851715 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcf4003520 con 0x7fdd0c152840 2024-03-06T07:22:34.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd09d9b640 1 -- 172.21.15.1:0/3285851715 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2752209639 0 0) 0x7fdcf4003520 con 0x7fdd0c152840 2024-03-06T07:22:34.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd09d9b640 1 -- 172.21.15.1:0/3285851715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd0c0c60b0 msgr2=0x7fdd0c0c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd09d9b640 1 --2- 172.21.15.1:0/3285851715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd0c0c60b0 0x7fdd0c0c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd09d9b640 1 -- 172.21.15.1:0/3285851715 >> v1:172.21.15.98:6789/0 conn(0x7fdd0c0c3970 legacy=0x7fdd0c152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd09d9b640 1 -- 172.21.15.1:0/3285851715 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0c0c6d20 con 0x7fdd0c152840 2024-03-06T07:22:34.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd09d9b640 1 -- 172.21.15.1:0/3285851715 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fdd00001670 con 0x7fdd0c152840 2024-03-06T07:22:34.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd09d9b640 1 -- 172.21.15.1:0/3285851715 >> v1:172.21.15.1:6789/0 conn(0x7fdd0c152840 legacy=0x7fdd0c1553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd09d9b640 1 --2- 172.21.15.1:0/3285851715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40039e0 0x7fdcf4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd09d9b640 1 --2- 172.21.15.1:0/3285851715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd0c0c60b0 0x7fdcf4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd09d9b640 1 --2- 172.21.15.1:0/3285851715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf4004d40 0x7fdcf40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd09d9b640 1 -- 172.21.15.1:0/3285851715 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c0c67f0 con 0x7fdcf40039e0 2024-03-06T07:22:34.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd09d9b640 1 -- 172.21.15.1:0/3285851715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf400b9c0 con 0x7fdcf4004d40 2024-03-06T07:22:34.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.508+0000 7fdd09d9b640 1 -- 172.21.15.1:0/3285851715 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf400bbc0 con 0x7fdd0c0c60b0 2024-03-06T07:22:34.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.509+0000 7fdd0ad9d640 1 --2- 172.21.15.1:0/3285851715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40039e0 0x7fdcf4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.509+0000 7fdd0b59e640 1 --2- 172.21.15.1:0/3285851715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf4004d40 0x7fdcf40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.509+0000 7fdd0a59c640 1 --2- 172.21.15.1:0/3285851715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd0c0c60b0 0x7fdcf4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.509+0000 7fdd0ad9d640 1 -- 172.21.15.1:0/3285851715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd0c0c60b0 msgr2=0x7fdcf4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.509+0000 7fdd0ad9d640 1 --2- 172.21.15.1:0/3285851715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd0c0c60b0 0x7fdcf4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.509+0000 7fdd0ad9d640 1 -- 172.21.15.1:0/3285851715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf4004d40 msgr2=0x7fdcf40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.509+0000 7fdd0ad9d640 1 --2- 172.21.15.1:0/3285851715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf4004d40 0x7fdcf40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.509+0000 7fdd0ad9d640 1 -- 172.21.15.1:0/3285851715 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf4003520 con 0x7fdcf40039e0 2024-03-06T07:22:34.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.509+0000 7fdd0a59c640 1 --2- 172.21.15.1:0/3285851715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd0c0c60b0 0x7fdcf4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:34.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.510+0000 7fdd0ad9d640 1 --2- 172.21.15.1:0/3285851715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40039e0 0x7fdcf4003dc0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc001860 tx=0x7fdcf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=628a5309447d97d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.510+0000 7fdd09d9b640 1 -- 172.21.15.1:0/3285851715 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc038070 con 0x7fdcf40039e0 2024-03-06T07:22:34.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.510+0000 7fdd09d9b640 1 -- 172.21.15.1:0/3285851715 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcfc033660 con 0x7fdcf40039e0 2024-03-06T07:22:34.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.510+0000 7fdd09d9b640 1 -- 172.21.15.1:0/3285851715 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc001430 con 0x7fdcf40039e0 2024-03-06T07:22:34.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.510+0000 7fdd111f6640 1 -- 172.21.15.1:0/3285851715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40039e0 msgr2=0x7fdcf4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.510+0000 7fdd111f6640 1 --2- 172.21.15.1:0/3285851715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40039e0 0x7fdcf4003dc0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc001860 tx=0x7fdcf4005830 comp rx=0 tx=0).stop 2024-03-06T07:22:34.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.510+0000 7fdd0b59e640 1 -- 172.21.15.1:0/3285851715 reap_dead start 2024-03-06T07:22:34.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.510+0000 7fdd111f6640 1 -- 172.21.15.1:0/3285851715 shutdown_connections 2024-03-06T07:22:34.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.510+0000 7fdd111f6640 1 -- 172.21.15.1:0/3285851715 >> 172.21.15.1:0/3285851715 conn(0x7fdd0c0c88f0 msgr2=0x7fdd0c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.510+0000 7fdd111f6640 1 -- 172.21.15.1:0/3285851715 shutdown_connections 2024-03-06T07:22:34.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.510+0000 7fdd111f6640 1 -- 172.21.15.1:0/3285851715 wait complete. 2024-03-06T07:22:34.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.512+0000 7fdd111f6640 1 Processor -- start 2024-03-06T07:22:34.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.512+0000 7fdd111f6640 1 -- start start 2024-03-06T07:22:34.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.513+0000 7fdd111f6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40039e0 0x7fdd0c1eacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.513+0000 7fdd111f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd0c0abfe0 0x7fdd0c1575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.513+0000 7fdd111f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd0c157b20 0x7fdd0c153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.513+0000 7fdd111f6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c0c75c0 con 0x7fdcf40039e0 2024-03-06T07:22:34.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.513+0000 7fdd111f6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c0c72e0 con 0x7fdd0c0abfe0 2024-03-06T07:22:34.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.513+0000 7fdd111f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c0c3d50 con 0x7fdd0c157b20 2024-03-06T07:22:34.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.513+0000 7fdd0ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40039e0 0x7fdd0c1eacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.513+0000 7fdd0ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40039e0 0x7fdd0c1eacb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57058/0 (socket says 172.21.15.1:57058) 2024-03-06T07:22:34.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.513+0000 7fdd0ad9d640 1 -- 172.21.15.1:0/3716034979 learned_addr learned my addr 172.21.15.1:0/3716034979 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:34.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.513+0000 7fdd0b59e640 1 --2- 172.21.15.1:0/3716034979 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd0c157b20 0x7fdd0c153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.513+0000 7fdd0a59c640 1 --2- 172.21.15.1:0/3716034979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd0c0abfe0 0x7fdd0c1575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.514+0000 7fdd0ad9d640 1 -- 172.21.15.1:0/3716034979 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd0c157b20 msgr2=0x7fdd0c153f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.514+0000 7fdd0ad9d640 1 --2- 172.21.15.1:0/3716034979 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd0c157b20 0x7fdd0c153f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.514+0000 7fdd0ad9d640 1 -- 172.21.15.1:0/3716034979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd0c0abfe0 msgr2=0x7fdd0c1575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.514+0000 7fdd0ad9d640 1 --2- 172.21.15.1:0/3716034979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd0c0abfe0 0x7fdd0c1575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.514+0000 7fdd0ad9d640 1 -- 172.21.15.1:0/3716034979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0c1eb1f0 con 0x7fdcf40039e0 2024-03-06T07:22:34.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.514+0000 7fdd0ad9d640 1 --2- 172.21.15.1:0/3716034979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40039e0 0x7fdd0c1eacb0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc02bd30 tx=0x7fdcfc001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.514+0000 7fdcf37fe640 1 -- 172.21.15.1:0/3716034979 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc038040 con 0x7fdcf40039e0 2024-03-06T07:22:34.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.514+0000 7fdcf37fe640 1 -- 172.21.15.1:0/3716034979 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcfc042070 con 0x7fdcf40039e0 2024-03-06T07:22:34.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.514+0000 7fdcf37fe640 1 -- 172.21.15.1:0/3716034979 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc005470 con 0x7fdcf40039e0 2024-03-06T07:22:34.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.514+0000 7fdd111f6640 1 -- 172.21.15.1:0/3716034979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd0c1ee4d0 con 0x7fdcf40039e0 2024-03-06T07:22:34.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.514+0000 7fdd111f6640 1 -- 172.21.15.1:0/3716034979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0c1ee1a0 con 0x7fdcf40039e0 2024-03-06T07:22:34.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.515+0000 7fdd111f6640 1 -- 172.21.15.1:0/3716034979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd0c1ee6f0 con 0x7fdcf40039e0 2024-03-06T07:22:34.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.516+0000 7fdcf37fe640 1 -- 172.21.15.1:0/3716034979 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdcfc007050 con 0x7fdcf40039e0 2024-03-06T07:22:34.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.516+0000 7fdcf37fe640 1 --2- 172.21.15.1:0/3716034979 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdcf806b4d0 0x7fdcf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.517+0000 7fdcf37fe640 1 -- 172.21.15.1:0/3716034979 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fdcfc0abb10 con 0x7fdcf40039e0 2024-03-06T07:22:34.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.517+0000 7fdd0a59c640 1 --2- 172.21.15.1:0/3716034979 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdcf806b4d0 0x7fdcf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.517+0000 7fdd0a59c640 1 --2- 172.21.15.1:0/3716034979 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdcf806b4d0 0x7fdcf806d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdd000017e0 tx=0x7fdd00007040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.525+0000 7fdcf37fe640 1 -- 172.21.15.1:0/3716034979 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd0c1ee6f0 con 0x7fdcf40039e0 2024-03-06T07:22:34.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.704+0000 7fdd111f6640 1 -- 172.21.15.1:0/3716034979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdd0c0686e0 con 0x7fdcf40039e0 2024-03-06T07:22:34.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.704+0000 7fdcf37fe640 1 -- 172.21.15.1:0/3716034979 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdd0c1eb1f0 con 0x7fdcf40039e0 2024-03-06T07:22:34.706 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:22:34.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.707+0000 7fdd111f6640 1 -- 172.21.15.1:0/3716034979 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdcf806b4d0 msgr2=0x7fdcf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.707+0000 7fdd111f6640 1 --2- 172.21.15.1:0/3716034979 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdcf806b4d0 0x7fdcf806d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdd000017e0 tx=0x7fdd00007040 comp rx=0 tx=0).stop 2024-03-06T07:22:34.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.707+0000 7fdd111f6640 1 -- 172.21.15.1:0/3716034979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40039e0 msgr2=0x7fdd0c1eacb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.707+0000 7fdd111f6640 1 --2- 172.21.15.1:0/3716034979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40039e0 0x7fdd0c1eacb0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc02bd30 tx=0x7fdcfc001230 comp rx=0 tx=0).stop 2024-03-06T07:22:34.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.707+0000 7fdd111f6640 1 -- 172.21.15.1:0/3716034979 shutdown_connections 2024-03-06T07:22:34.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.707+0000 7fdd111f6640 1 --2- 172.21.15.1:0/3716034979 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdcf806b4d0 0x7fdcf806d990 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.707+0000 7fdd111f6640 1 --2- 172.21.15.1:0/3716034979 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd0c157b20 0x7fdd0c153f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.707+0000 7fdd111f6640 1 --2- 172.21.15.1:0/3716034979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd0c0abfe0 0x7fdd0c1575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.707+0000 7fdd111f6640 1 --2- 172.21.15.1:0/3716034979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40039e0 0x7fdd0c1eacb0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.707+0000 7fdd111f6640 1 -- 172.21.15.1:0/3716034979 >> 172.21.15.1:0/3716034979 conn(0x7fdd0c0c88f0 msgr2=0x7fdd0c0c2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.707+0000 7fdd111f6640 1 -- 172.21.15.1:0/3716034979 shutdown_connections 2024-03-06T07:22:34.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.707+0000 7fdd111f6640 1 -- 172.21.15.1:0/3716034979 wait complete. 2024-03-06T07:22:34.720 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:34.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.841+0000 7f09d715c640 1 Processor -- start 2024-03-06T07:22:34.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.841+0000 7f09d715c640 1 -- start start 2024-03-06T07:22:34.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.842+0000 7f09d715c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09d0152650 0x7f09d0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.842+0000 7f09d715c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d00c2fc0 con 0x7f09d00c6b10 2024-03-06T07:22:34.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.842+0000 7f09d715c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d0158da0 con 0x7f09d00bf9d0 2024-03-06T07:22:34.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.842+0000 7f09d715c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d0155800 con 0x7f09d0152650 2024-03-06T07:22:34.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.843+0000 7f09cffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f09d00c6b10 0x7f09d0158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37804/0 (socket says 172.21.15.1:37804) 2024-03-06T07:22:34.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.843+0000 7f09cffff640 1 -- 172.21.15.1:0/2394484267 learned_addr learned my addr 172.21.15.1:0/2394484267 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:34.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.843+0000 7f09d4ef8640 1 --2- 172.21.15.1:0/2394484267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09d0152650 0x7f09d0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.843+0000 7f09cf7fe640 1 -- 172.21.15.1:0/2394484267 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792059251 0 0) 0x7f09d00c2fc0 con 0x7f09d00c6b10 2024-03-06T07:22:34.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.843+0000 7f09cf7fe640 1 -- 172.21.15.1:0/2394484267 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c0003520 con 0x7f09d00c6b10 2024-03-06T07:22:34.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.843+0000 7f09cf7fe640 1 -- 172.21.15.1:0/2394484267 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895726090 0 0) 0x7f09d0158da0 con 0x7f09d00bf9d0 2024-03-06T07:22:34.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.843+0000 7f09cf7fe640 1 -- 172.21.15.1:0/2394484267 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d00c2fc0 con 0x7f09d00bf9d0 2024-03-06T07:22:34.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.843+0000 7f09cf7fe640 1 -- 172.21.15.1:0/2394484267 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3294634453 0 0) 0x7f09c0003520 con 0x7f09d00c6b10 2024-03-06T07:22:34.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.843+0000 7f09d4ef8640 1 -- 172.21.15.1:0/2394484267 >> v1:172.21.15.98:6789/0 conn(0x7f09d00bf9d0 legacy=0x7f09d00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.843+0000 7f09d4ef8640 1 -- 172.21.15.1:0/2394484267 >> v1:172.21.15.1:6789/0 conn(0x7f09d00c6b10 legacy=0x7f09d0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.843+0000 7f09d4ef8640 1 -- 172.21.15.1:0/2394484267 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d00c6690 con 0x7f09d0152650 2024-03-06T07:22:34.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.844+0000 7f09d4ef8640 1 --2- 172.21.15.1:0/2394484267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09d0152650 0x7f09d0155c90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f09c40097d0 tx=0x7f09c402b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9707b24998778c35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.844+0000 7f09cf7fe640 1 -- 172.21.15.1:0/2394484267 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c4037070 con 0x7f09d0152650 2024-03-06T07:22:34.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.844+0000 7f09cf7fe640 1 -- 172.21.15.1:0/2394484267 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c4030510 con 0x7f09d0152650 2024-03-06T07:22:34.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.844+0000 7f09cf7fe640 1 -- 172.21.15.1:0/2394484267 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc001670 con 0x7f09d0152650 2024-03-06T07:22:34.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.845+0000 7f09d715c640 1 -- 172.21.15.1:0/2394484267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09d0152650 msgr2=0x7f09d0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.845+0000 7f09d715c640 1 --2- 172.21.15.1:0/2394484267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09d0152650 0x7f09d0155c90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f09c40097d0 tx=0x7f09c402b770 comp rx=0 tx=0).stop 2024-03-06T07:22:34.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.845+0000 7f09d715c640 1 -- 172.21.15.1:0/2394484267 shutdown_connections 2024-03-06T07:22:34.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.845+0000 7f09d715c640 1 --2- 172.21.15.1:0/2394484267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09d0152650 0x7f09d0155c90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.845+0000 7f09d715c640 1 -- 172.21.15.1:0/2394484267 >> 172.21.15.1:0/2394484267 conn(0x7f09d0066ee0 msgr2=0x7f09d00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.845+0000 7f09d715c640 1 -- 172.21.15.1:0/2394484267 shutdown_connections 2024-03-06T07:22:34.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.845+0000 7f09d715c640 1 -- 172.21.15.1:0/2394484267 wait complete. 2024-03-06T07:22:34.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.846+0000 7f09d715c640 1 Processor -- start 2024-03-06T07:22:34.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.846+0000 7f09d715c640 1 -- start start 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.847+0000 7f09d715c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09d00bf9d0 0x7f09d00c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.847+0000 7f09d715c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d0152650 0x7f09d014aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.847+0000 7f09d715c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09d01566a0 0x7f09d014b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.847+0000 7f09d715c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d01551b0 con 0x7f09d0152650 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.847+0000 7f09d715c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00c6fa0 con 0x7f09d01566a0 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.847+0000 7f09d715c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d0155490 con 0x7f09d00bf9d0 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.847+0000 7f09cffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d0152650 0x7f09d014aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.847+0000 7f09cffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d0152650 0x7f09d014aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57066/0 (socket says 172.21.15.1:57066) 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.847+0000 7f09cffff640 1 -- 172.21.15.1:0/100549515 learned_addr learned my addr 172.21.15.1:0/100549515 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.847+0000 7f09d4ef8640 1 --2- 172.21.15.1:0/100549515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09d00bf9d0 0x7f09d00c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.847+0000 7f09d56f9640 1 --2- 172.21.15.1:0/100549515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09d01566a0 0x7f09d014b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.847+0000 7f09cffff640 1 -- 172.21.15.1:0/100549515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09d00bf9d0 msgr2=0x7f09d00c9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.847+0000 7f09cffff640 1 --2- 172.21.15.1:0/100549515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09d00bf9d0 0x7f09d00c9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.847+0000 7f09cffff640 1 -- 172.21.15.1:0/100549515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09d01566a0 msgr2=0x7f09d014b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.847+0000 7f09cffff640 1 --2- 172.21.15.1:0/100549515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09d01566a0 0x7f09d014b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.847+0000 7f09cffff640 1 -- 172.21.15.1:0/100549515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d014b810 con 0x7f09d0152650 2024-03-06T07:22:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.847+0000 7f09d56f9640 1 --2- 172.21.15.1:0/100549515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09d01566a0 0x7f09d014b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.848+0000 7f09cffff640 1 --2- 172.21.15.1:0/100549515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d0152650 0x7f09d014aaf0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00c730 tx=0x7f09bc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.848+0000 7f09cd7fa640 1 -- 172.21.15.1:0/100549515 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc017070 con 0x7f09d0152650 2024-03-06T07:22:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.848+0000 7f09cd7fa640 1 -- 172.21.15.1:0/100549515 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09bc005ad0 con 0x7f09d0152650 2024-03-06T07:22:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.848+0000 7f09d715c640 1 -- 172.21.15.1:0/100549515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d014baf0 con 0x7f09d0152650 2024-03-06T07:22:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.848+0000 7f09cd7fa640 1 -- 172.21.15.1:0/100549515 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc007070 con 0x7f09d0152650 2024-03-06T07:22:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.848+0000 7f09d715c640 1 -- 172.21.15.1:0/100549515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d00ca3a0 con 0x7f09d0152650 2024-03-06T07:22:34.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.849+0000 7f09d715c640 1 -- 172.21.15.1:0/100549515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d0067fd0 con 0x7f09d0152650 2024-03-06T07:22:34.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.850+0000 7f09cd7fa640 1 -- 172.21.15.1:0/100549515 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09bc01a070 con 0x7f09d0152650 2024-03-06T07:22:34.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.850+0000 7f09cd7fa640 1 --2- 172.21.15.1:0/100549515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f09b406b4d0 0x7f09b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.850+0000 7f09cd7fa640 1 -- 172.21.15.1:0/100549515 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f09bc089d20 con 0x7f09d0152650 2024-03-06T07:22:34.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.850+0000 7f09d4ef8640 1 --2- 172.21.15.1:0/100549515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f09b406b4d0 0x7f09b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.851+0000 7f09d4ef8640 1 --2- 172.21.15.1:0/100549515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f09b406b4d0 0x7f09b406d990 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f09c4034000 tx=0x7f09c4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:34.854+0000 7f09cd7fa640 1 -- 172.21.15.1:0/100549515 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09bc0586a0 con 0x7f09d0152650 2024-03-06T07:22:35.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.049+0000 7f09d715c640 1 -- 172.21.15.1:0/100549515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f09d00b06f0 con 0x7f09d0152650 2024-03-06T07:22:35.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.049+0000 7f09cd7fa640 1 -- 172.21.15.1:0/100549515 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7f09bc057fa0 con 0x7f09d0152650 2024-03-06T07:22:35.051 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:22:35.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.052+0000 7f09d715c640 1 -- 172.21.15.1:0/100549515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f09b406b4d0 msgr2=0x7f09b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.052+0000 7f09d715c640 1 --2- 172.21.15.1:0/100549515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f09b406b4d0 0x7f09b406d990 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f09c4034000 tx=0x7f09c4008030 comp rx=0 tx=0).stop 2024-03-06T07:22:35.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.052+0000 7f09d715c640 1 -- 172.21.15.1:0/100549515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d0152650 msgr2=0x7f09d014aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.052+0000 7f09d715c640 1 --2- 172.21.15.1:0/100549515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d0152650 0x7f09d014aaf0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00c730 tx=0x7f09bc00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.052+0000 7f09d715c640 1 -- 172.21.15.1:0/100549515 shutdown_connections 2024-03-06T07:22:35.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.052+0000 7f09d715c640 1 --2- 172.21.15.1:0/100549515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f09b406b4d0 0x7f09b406d990 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.052+0000 7f09d715c640 1 --2- 172.21.15.1:0/100549515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09d01566a0 0x7f09d014b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.052+0000 7f09d715c640 1 --2- 172.21.15.1:0/100549515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d0152650 0x7f09d014aaf0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.052+0000 7f09d715c640 1 --2- 172.21.15.1:0/100549515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09d00bf9d0 0x7f09d00c9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.052+0000 7f09d715c640 1 -- 172.21.15.1:0/100549515 >> 172.21.15.1:0/100549515 conn(0x7f09d0066ee0 msgr2=0x7f09d00c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.052+0000 7f09d715c640 1 -- 172.21.15.1:0/100549515 shutdown_connections 2024-03-06T07:22:35.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.052+0000 7f09d715c640 1 -- 172.21.15.1:0/100549515 wait complete. 2024-03-06T07:22:35.065 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-06T07:22:35.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.167+0000 7f3a3e81c640 1 Processor -- start 2024-03-06T07:22:35.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.167+0000 7f3a3e81c640 1 -- start start 2024-03-06T07:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.168+0000 7f3a3e81c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3a380bf9d0 0x7f3a38155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.168+0000 7f3a3e81c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a380c2fc0 con 0x7f3a380c6b10 2024-03-06T07:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.168+0000 7f3a3e81c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a38158da0 con 0x7f3a38152650 2024-03-06T07:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.168+0000 7f3a3e81c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a38155800 con 0x7f3a380bf9d0 2024-03-06T07:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.169+0000 7f3a3cdb9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3a380bf9d0 0x7f3a38155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.169+0000 7f3a377fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3a380c6b10 0x7f3a380c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37810/0 (socket says 172.21.15.1:37810) 2024-03-06T07:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.169+0000 7f3a377fe640 1 -- 172.21.15.1:0/687344426 learned_addr learned my addr 172.21.15.1:0/687344426 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.169+0000 7f3a3cdb9640 1 -- 172.21.15.1:0/687344426 >> v1:172.21.15.98:6789/0 conn(0x7f3a38152650 legacy=0x7f3a38158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.169+0000 7f3a3cdb9640 1 -- 172.21.15.1:0/687344426 >> v1:172.21.15.1:6789/0 conn(0x7f3a380c6b10 legacy=0x7f3a380c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.169+0000 7f3a3cdb9640 1 -- 172.21.15.1:0/687344426 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a380c6690 con 0x7f3a380bf9d0 2024-03-06T07:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.169+0000 7f3a36ffd640 1 -- 172.21.15.1:0/687344426 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354098813 0 0) 0x7f3a380c2fc0 con 0x7f3a380c6b10 2024-03-06T07:22:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.170+0000 7f3a3cdb9640 1 --2- 172.21.15.1:0/687344426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a380bf9d0 0x7f3a38155c90 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f3a30009d90 tx=0x7f3a3002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f34d8567021951 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.170+0000 7f3a36ffd640 1 -- 172.21.15.1:0/687344426 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a30039070 con 0x7f3a380bf9d0 2024-03-06T07:22:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.170+0000 7f3a36ffd640 1 -- 172.21.15.1:0/687344426 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3002cc60 con 0x7f3a380bf9d0 2024-03-06T07:22:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.170+0000 7f3a36ffd640 1 -- 172.21.15.1:0/687344426 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a30034660 con 0x7f3a380bf9d0 2024-03-06T07:22:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.171+0000 7f3a3e81c640 1 -- 172.21.15.1:0/687344426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a380bf9d0 msgr2=0x7f3a38155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.171+0000 7f3a3e81c640 1 --2- 172.21.15.1:0/687344426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a380bf9d0 0x7f3a38155c90 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f3a30009d90 tx=0x7f3a3002beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.171+0000 7f3a3e81c640 1 -- 172.21.15.1:0/687344426 shutdown_connections 2024-03-06T07:22:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.171+0000 7f3a3e81c640 1 --2- 172.21.15.1:0/687344426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a380bf9d0 0x7f3a38155c90 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.171+0000 7f3a3e81c640 1 -- 172.21.15.1:0/687344426 >> 172.21.15.1:0/687344426 conn(0x7f3a38066ee0 msgr2=0x7f3a380672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.171+0000 7f3a3e81c640 1 -- 172.21.15.1:0/687344426 shutdown_connections 2024-03-06T07:22:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.171+0000 7f3a3e81c640 1 -- 172.21.15.1:0/687344426 wait complete. 2024-03-06T07:22:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.173+0000 7f3a3e81c640 1 Processor -- start 2024-03-06T07:22:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.173+0000 7f3a3e81c640 1 -- start start 2024-03-06T07:22:35.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a3e81c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a380bf9d0 0x7f3a381e2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a3e81c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a38152650 0x7f3a381ee5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a3e81c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a38156490 0x7f3a3814d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a3e81c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a381551b0 con 0x7f3a38152650 2024-03-06T07:22:35.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a3e81c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a380c6fa0 con 0x7f3a380bf9d0 2024-03-06T07:22:35.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a3e81c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a38155490 con 0x7f3a38156490 2024-03-06T07:22:35.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a377fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a38152650 0x7f3a381ee5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a377fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a38152650 0x7f3a381ee5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57080/0 (socket says 172.21.15.1:57080) 2024-03-06T07:22:35.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a377fe640 1 -- 172.21.15.1:0/4151210974 learned_addr learned my addr 172.21.15.1:0/4151210974 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:35.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a3cdb9640 1 --2- 172.21.15.1:0/4151210974 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a38156490 0x7f3a3814d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a37fff640 1 --2- 172.21.15.1:0/4151210974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a380bf9d0 0x7f3a381e2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a377fe640 1 -- 172.21.15.1:0/4151210974 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a38156490 msgr2=0x7f3a3814d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a377fe640 1 --2- 172.21.15.1:0/4151210974 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a38156490 0x7f3a3814d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a377fe640 1 -- 172.21.15.1:0/4151210974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a380bf9d0 msgr2=0x7f3a381e2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a377fe640 1 --2- 172.21.15.1:0/4151210974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a380bf9d0 0x7f3a381e2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a377fe640 1 -- 172.21.15.1:0/4151210974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a381eede0 con 0x7f3a38152650 2024-03-06T07:22:35.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a37fff640 1 --2- 172.21.15.1:0/4151210974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a380bf9d0 0x7f3a381e2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:35.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a3cdb9640 1 --2- 172.21.15.1:0/4151210974 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a38156490 0x7f3a3814d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:35.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.174+0000 7f3a377fe640 1 --2- 172.21.15.1:0/4151210974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a38152650 0x7f3a381ee5a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c00c4d0 tx=0x7f3a2c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.175+0000 7f3a34ff9640 1 -- 172.21.15.1:0/4151210974 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c018070 con 0x7f3a38152650 2024-03-06T07:22:35.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.175+0000 7f3a34ff9640 1 -- 172.21.15.1:0/4151210974 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2c0074e0 con 0x7f3a38152650 2024-03-06T07:22:35.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.175+0000 7f3a34ff9640 1 -- 172.21.15.1:0/4151210974 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c004a20 con 0x7f3a38152650 2024-03-06T07:22:35.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.175+0000 7f3a3e81c640 1 -- 172.21.15.1:0/4151210974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a381ef0c0 con 0x7f3a38152650 2024-03-06T07:22:35.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.175+0000 7f3a3e81c640 1 -- 172.21.15.1:0/4151210974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a381dfaf0 con 0x7f3a38152650 2024-03-06T07:22:35.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.176+0000 7f3a3e81c640 1 -- 172.21.15.1:0/4151210974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a38067fd0 con 0x7f3a38152650 2024-03-06T07:22:35.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.176+0000 7f3a34ff9640 1 -- 172.21.15.1:0/4151210974 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a2c01b070 con 0x7f3a38152650 2024-03-06T07:22:35.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.177+0000 7f3a34ff9640 1 --2- 172.21.15.1:0/4151210974 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a2406b4d0 0x7f3a2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.177+0000 7f3a34ff9640 1 -- 172.21.15.1:0/4151210974 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f3a2c089b20 con 0x7f3a38152650 2024-03-06T07:22:35.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.177+0000 7f3a37fff640 1 --2- 172.21.15.1:0/4151210974 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a2406b4d0 0x7f3a2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.177+0000 7f3a37fff640 1 --2- 172.21.15.1:0/4151210974 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a2406b4d0 0x7f3a2406d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3a28000f60 tx=0x7f3a28009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.184+0000 7f3a34ff9640 1 -- 172.21.15.1:0/4151210974 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a2c058380 con 0x7f3a38152650 2024-03-06T07:22:35.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.365+0000 7f3a3e81c640 1 -- 172.21.15.1:0/4151210974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3a380b06f0 con 0x7f3a38152650 2024-03-06T07:22:35.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.366+0000 7f3a34ff9640 1 -- 172.21.15.1:0/4151210974 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v58) v1 ==== 105+0+11 (secure 0 0 0) 0x7f3a2c057c80 con 0x7f3a38152650 2024-03-06T07:22:35.368 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:22:35.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.368+0000 7f3a3e81c640 1 -- 172.21.15.1:0/4151210974 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a2406b4d0 msgr2=0x7f3a2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.368+0000 7f3a3e81c640 1 --2- 172.21.15.1:0/4151210974 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a2406b4d0 0x7f3a2406d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3a28000f60 tx=0x7f3a28009040 comp rx=0 tx=0).stop 2024-03-06T07:22:35.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.368+0000 7f3a3e81c640 1 -- 172.21.15.1:0/4151210974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a38152650 msgr2=0x7f3a381ee5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.368+0000 7f3a3e81c640 1 --2- 172.21.15.1:0/4151210974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a38152650 0x7f3a381ee5a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c00c4d0 tx=0x7f3a2c00c990 comp rx=0 tx=0).stop 2024-03-06T07:22:35.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.368+0000 7f3a3e81c640 1 -- 172.21.15.1:0/4151210974 shutdown_connections 2024-03-06T07:22:35.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.368+0000 7f3a3e81c640 1 --2- 172.21.15.1:0/4151210974 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a2406b4d0 0x7f3a2406d990 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.368+0000 7f3a3e81c640 1 --2- 172.21.15.1:0/4151210974 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a38156490 0x7f3a3814d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.368+0000 7f3a3e81c640 1 --2- 172.21.15.1:0/4151210974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a38152650 0x7f3a381ee5a0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.368+0000 7f3a3e81c640 1 --2- 172.21.15.1:0/4151210974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a380bf9d0 0x7f3a381e2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.368+0000 7f3a3e81c640 1 -- 172.21.15.1:0/4151210974 >> 172.21.15.1:0/4151210974 conn(0x7f3a38066ee0 msgr2=0x7f3a380cadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.368+0000 7f3a3e81c640 1 -- 172.21.15.1:0/4151210974 shutdown_connections 2024-03-06T07:22:35.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.368+0000 7f3a3e81c640 1 -- 172.21.15.1:0/4151210974 wait complete. 2024-03-06T07:22:35.381 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-06T07:22:35.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.492+0000 7f56e2bd7640 1 Processor -- start 2024-03-06T07:22:35.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.492+0000 7f56e2bd7640 1 -- start start 2024-03-06T07:22:35.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.493+0000 7f56e2bd7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f56dc152820 0x7f56dc0c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.493+0000 7f56e2bd7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f56dc0af6f0 0x7f56dc15fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.493+0000 7f56e2bd7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56dc0c7180 con 0x7f56dc0c0670 2024-03-06T07:22:35.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.493+0000 7f56e2bd7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56dc155460 con 0x7f56dc0af6f0 2024-03-06T07:22:35.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.493+0000 7f56e2bd7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56dc0c6bc0 con 0x7f56dc152820 2024-03-06T07:22:35.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.493+0000 7f56e1174640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f56dc0af6f0 0x7f56dc15fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.493+0000 7f56dbfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f56dc0c0670 0x7f56dc0c73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37814/0 (socket says 172.21.15.1:37814) 2024-03-06T07:22:35.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.493+0000 7f56e1174640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f56dc0af6f0 0x7f56dc15fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57094/0 (socket says 172.21.15.1:57094) 2024-03-06T07:22:35.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.493+0000 7f56dbfff640 1 -- 172.21.15.1:0/1709687802 learned_addr learned my addr 172.21.15.1:0/1709687802 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:35.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.493+0000 7f56e0973640 1 --2- 172.21.15.1:0/1709687802 >> v2:172.21.15.98:3300/0 conn(0x7f56dc152820 0x7f56dc0c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.493+0000 7f56e1174640 1 -- 172.21.15.1:0/1709687802 >> v2:172.21.15.98:3300/0 conn(0x7f56dc152820 msgr2=0x7f56dc0c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.493+0000 7f56e1174640 1 --2- 172.21.15.1:0/1709687802 >> v2:172.21.15.98:3300/0 conn(0x7f56dc152820 0x7f56dc0c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.493+0000 7f56e1174640 1 -- 172.21.15.1:0/1709687802 >> v1:172.21.15.1:6789/0 conn(0x7f56dc0c0670 legacy=0x7f56dc0c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.494+0000 7f56e1174640 1 -- 172.21.15.1:0/1709687802 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56dc0c6ea0 con 0x7f56dc0af6f0 2024-03-06T07:22:35.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.494+0000 7f56db7fe640 1 -- 172.21.15.1:0/1709687802 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079635441 0 0) 0x7f56dc0c7180 con 0x7f56dc0c0670 2024-03-06T07:22:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.495+0000 7f56e1174640 1 --2- 172.21.15.1:0/1709687802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc0af6f0 0x7f56dc15fb60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f56dc08ccc0 tx=0x7f56d402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf8f3c11ab5fcf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.495+0000 7f56db7fe640 1 -- 172.21.15.1:0/1709687802 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d4039070 con 0x7f56dc0af6f0 2024-03-06T07:22:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.495+0000 7f56db7fe640 1 -- 172.21.15.1:0/1709687802 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d402cc80 con 0x7f56dc0af6f0 2024-03-06T07:22:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.495+0000 7f56db7fe640 1 -- 172.21.15.1:0/1709687802 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d4034660 con 0x7f56dc0af6f0 2024-03-06T07:22:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.495+0000 7f56e2bd7640 1 -- 172.21.15.1:0/1709687802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc0af6f0 msgr2=0x7f56dc15fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.495+0000 7f56e2bd7640 1 --2- 172.21.15.1:0/1709687802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc0af6f0 0x7f56dc15fb60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f56dc08ccc0 tx=0x7f56d402bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.496+0000 7f56e2bd7640 1 -- 172.21.15.1:0/1709687802 shutdown_connections 2024-03-06T07:22:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.496+0000 7f56e2bd7640 1 --2- 172.21.15.1:0/1709687802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc0af6f0 0x7f56dc15fb60 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.496+0000 7f56e2bd7640 1 --2- 172.21.15.1:0/1709687802 >> v2:172.21.15.98:3300/0 conn(0x7f56dc152820 0x7f56dc0c0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.496+0000 7f56e2bd7640 1 -- 172.21.15.1:0/1709687802 >> 172.21.15.1:0/1709687802 conn(0x7f56dc072720 msgr2=0x7f56dc072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.496+0000 7f56e2bd7640 1 -- 172.21.15.1:0/1709687802 shutdown_connections 2024-03-06T07:22:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.496+0000 7f56e2bd7640 1 -- 172.21.15.1:0/1709687802 wait complete. 2024-03-06T07:22:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.497+0000 7f56e2bd7640 1 Processor -- start 2024-03-06T07:22:35.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.497+0000 7f56e2bd7640 1 -- start start 2024-03-06T07:22:35.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56e2bd7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56dc0af6f0 0x7f56dc1591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56e2bd7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56dc0c0670 0x7f56dc1eaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56e2bd7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc152820 0x7f56dc1543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56e2bd7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc0c7c80 con 0x7f56dc152820 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56e2bd7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc0c7b00 con 0x7f56dc0af6f0 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56e2bd7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc152c00 con 0x7f56dc0c0670 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56e1174640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc152820 0x7f56dc1543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56e1174640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc152820 0x7f56dc1543f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57110/0 (socket says 172.21.15.1:57110) 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56e1174640 1 -- 172.21.15.1:0/3613528383 learned_addr learned my addr 172.21.15.1:0/3613528383 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56e0973640 1 --2- 172.21.15.1:0/3613528383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56dc0af6f0 0x7f56dc1591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56dbfff640 1 --2- 172.21.15.1:0/3613528383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56dc0c0670 0x7f56dc1eaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56e1174640 1 -- 172.21.15.1:0/3613528383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56dc0c0670 msgr2=0x7f56dc1eaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56e1174640 1 --2- 172.21.15.1:0/3613528383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56dc0c0670 0x7f56dc1eaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56e1174640 1 -- 172.21.15.1:0/3613528383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56dc0af6f0 msgr2=0x7f56dc1591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56e1174640 1 --2- 172.21.15.1:0/3613528383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56dc0af6f0 0x7f56dc1591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56e1174640 1 -- 172.21.15.1:0/3613528383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56dc158d90 con 0x7f56dc152820 2024-03-06T07:22:35.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56e1174640 1 --2- 172.21.15.1:0/3613528383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc152820 0x7f56dc1543f0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009e80 tx=0x7f56d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.499+0000 7f56d97fa640 1 -- 172.21.15.1:0/3613528383 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d4039050 con 0x7f56dc152820 2024-03-06T07:22:35.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.500+0000 7f56d97fa640 1 -- 172.21.15.1:0/3613528383 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d4045070 con 0x7f56dc152820 2024-03-06T07:22:35.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.500+0000 7f56e2bd7640 1 -- 172.21.15.1:0/3613528383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56dc0af0e0 con 0x7f56dc152820 2024-03-06T07:22:35.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.500+0000 7f56d97fa640 1 -- 172.21.15.1:0/3613528383 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d403e830 con 0x7f56dc152820 2024-03-06T07:22:35.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.500+0000 7f56e2bd7640 1 -- 172.21.15.1:0/3613528383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56dc1ea6e0 con 0x7f56dc152820 2024-03-06T07:22:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.501+0000 7f56e2bd7640 1 -- 172.21.15.1:0/3613528383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56dc0686e0 con 0x7f56dc152820 2024-03-06T07:22:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.501+0000 7f56d97fa640 1 -- 172.21.15.1:0/3613528383 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56d403e9d0 con 0x7f56dc152820 2024-03-06T07:22:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.502+0000 7f56d97fa640 1 --2- 172.21.15.1:0/3613528383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f56c006b4d0 0x7f56c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.502+0000 7f56d97fa640 1 -- 172.21.15.1:0/3613528383 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f56d40ad4e0 con 0x7f56dc152820 2024-03-06T07:22:35.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.502+0000 7f56e0973640 1 --2- 172.21.15.1:0/3613528383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f56c006b4d0 0x7f56c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.502+0000 7f56e0973640 1 --2- 172.21.15.1:0/3613528383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f56c006b4d0 0x7f56c006d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f56dc1598f0 tx=0x7f56d0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.506+0000 7f56d97fa640 1 -- 172.21.15.1:0/3613528383 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56d407b930 con 0x7f56dc152820 2024-03-06T07:22:35.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.753+0000 7f56e2bd7640 1 -- 172.21.15.1:0/3613528383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7f56dc14fd00 con 0x7f56dc152820 2024-03-06T07:22:35.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.753+0000 7f56d97fa640 1 -- 172.21.15.1:0/3613528383 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v58) v1 ==== 101+0+11 (secure 0 0 0) 0x7f56d407f9d0 con 0x7f56dc152820 2024-03-06T07:22:35.756 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:22:35.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.755+0000 7f56e2bd7640 1 -- 172.21.15.1:0/3613528383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f56c006b4d0 msgr2=0x7f56c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.755+0000 7f56e2bd7640 1 --2- 172.21.15.1:0/3613528383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f56c006b4d0 0x7f56c006d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f56dc1598f0 tx=0x7f56d0008040 comp rx=0 tx=0).stop 2024-03-06T07:22:35.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.755+0000 7f56e2bd7640 1 -- 172.21.15.1:0/3613528383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc152820 msgr2=0x7f56dc1543f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.755+0000 7f56e2bd7640 1 --2- 172.21.15.1:0/3613528383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc152820 0x7f56dc1543f0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009e80 tx=0x7f56d4004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:35.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.755+0000 7f56e2bd7640 1 -- 172.21.15.1:0/3613528383 shutdown_connections 2024-03-06T07:22:35.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.755+0000 7f56e2bd7640 1 --2- 172.21.15.1:0/3613528383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f56c006b4d0 0x7f56c006d990 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.755+0000 7f56e2bd7640 1 --2- 172.21.15.1:0/3613528383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc152820 0x7f56dc1543f0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.755+0000 7f56e2bd7640 1 --2- 172.21.15.1:0/3613528383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56dc0c0670 0x7f56dc1eaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.755+0000 7f56e2bd7640 1 --2- 172.21.15.1:0/3613528383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56dc0af6f0 0x7f56dc1591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.755+0000 7f56e2bd7640 1 -- 172.21.15.1:0/3613528383 >> 172.21.15.1:0/3613528383 conn(0x7f56dc072720 msgr2=0x7f56dc06cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.755+0000 7f56e2bd7640 1 -- 172.21.15.1:0/3613528383 shutdown_connections 2024-03-06T07:22:35.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.755+0000 7f56e2bd7640 1 -- 172.21.15.1:0/3613528383 wait complete. 2024-03-06T07:22:35.771 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:35.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.872+0000 7feb908ed640 1 Processor -- start 2024-03-06T07:22:35.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.872+0000 7feb908ed640 1 -- start start 2024-03-06T07:22:35.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.873+0000 7feb908ed640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feb8c152650 0x7feb8c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.873+0000 7feb908ed640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb8c0bf9d0 0x7feb8c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.873+0000 7feb908ed640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8c0c2fc0 con 0x7feb8c0c6b10 2024-03-06T07:22:35.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.873+0000 7feb908ed640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb8c155800 con 0x7feb8c152650 2024-03-06T07:22:35.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.873+0000 7feb908ed640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb8c158da0 con 0x7feb8c0bf9d0 2024-03-06T07:22:35.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.873+0000 7feb8a59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feb8c152650 0x7feb8c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.873+0000 7feb8a59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feb8c152650 0x7feb8c155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57112/0 (socket says 172.21.15.1:57112) 2024-03-06T07:22:35.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.873+0000 7feb8a59c640 1 -- 172.21.15.1:0/2371469183 learned_addr learned my addr 172.21.15.1:0/2371469183 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:35.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.873+0000 7feb8ad9d640 1 --2- 172.21.15.1:0/2371469183 >> v2:172.21.15.98:3300/0 conn(0x7feb8c0bf9d0 0x7feb8c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.873+0000 7feb8a59c640 1 -- 172.21.15.1:0/2371469183 >> v2:172.21.15.98:3300/0 conn(0x7feb8c0bf9d0 msgr2=0x7feb8c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.873+0000 7feb8a59c640 1 --2- 172.21.15.1:0/2371469183 >> v2:172.21.15.98:3300/0 conn(0x7feb8c0bf9d0 0x7feb8c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.873+0000 7feb8a59c640 1 -- 172.21.15.1:0/2371469183 >> v1:172.21.15.98:6789/0 conn(0x7feb8c0c6b10 legacy=0x7feb8c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.873+0000 7feb8a59c640 1 -- 172.21.15.1:0/2371469183 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c0c6690 con 0x7feb8c152650 2024-03-06T07:22:35.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.874+0000 7feb8a59c640 1 --2- 172.21.15.1:0/2371469183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb8c152650 0x7feb8c155c90 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7feb8c1597b0 tx=0x7feb7c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49401dfb7be1aeef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.874+0000 7feb8959a640 1 -- 172.21.15.1:0/2371469183 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c035020 con 0x7feb8c152650 2024-03-06T07:22:35.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.874+0000 7feb8959a640 1 -- 172.21.15.1:0/2371469183 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7c031870 con 0x7feb8c152650 2024-03-06T07:22:35.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.874+0000 7feb8959a640 1 -- 172.21.15.1:0/2371469183 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c031c40 con 0x7feb8c152650 2024-03-06T07:22:35.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.875+0000 7feb908ed640 1 -- 172.21.15.1:0/2371469183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb8c152650 msgr2=0x7feb8c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.875+0000 7feb908ed640 1 --2- 172.21.15.1:0/2371469183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb8c152650 0x7feb8c155c90 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7feb8c1597b0 tx=0x7feb7c02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:35.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.875+0000 7feb908ed640 1 -- 172.21.15.1:0/2371469183 shutdown_connections 2024-03-06T07:22:35.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.875+0000 7feb908ed640 1 --2- 172.21.15.1:0/2371469183 >> v2:172.21.15.98:3300/0 conn(0x7feb8c0bf9d0 0x7feb8c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.875+0000 7feb908ed640 1 --2- 172.21.15.1:0/2371469183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb8c152650 0x7feb8c155c90 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.875+0000 7feb908ed640 1 -- 172.21.15.1:0/2371469183 >> 172.21.15.1:0/2371469183 conn(0x7feb8c066ee0 msgr2=0x7feb8c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.875+0000 7feb908ed640 1 -- 172.21.15.1:0/2371469183 shutdown_connections 2024-03-06T07:22:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.875+0000 7feb908ed640 1 -- 172.21.15.1:0/2371469183 wait complete. 2024-03-06T07:22:35.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.877+0000 7feb908ed640 1 Processor -- start 2024-03-06T07:22:35.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.877+0000 7feb908ed640 1 -- start start 2024-03-06T07:22:35.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.878+0000 7feb908ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb8c0bf9d0 0x7feb8c158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.878+0000 7feb908ed640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb8c152650 0x7feb8c14ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.878+0000 7feb908ed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb8c14b360 0x7feb8c1ee1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.878+0000 7feb908ed640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c1551b0 con 0x7feb8c152650 2024-03-06T07:22:35.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.878+0000 7feb908ed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c0c6fa0 con 0x7feb8c0bf9d0 2024-03-06T07:22:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.878+0000 7feb908ed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c155490 con 0x7feb8c14b360 2024-03-06T07:22:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.878+0000 7feb89d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb8c152650 0x7feb8c14ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.878+0000 7feb89d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb8c152650 0x7feb8c14ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57116/0 (socket says 172.21.15.1:57116) 2024-03-06T07:22:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.878+0000 7feb89d9b640 1 -- 172.21.15.1:0/1685020193 learned_addr learned my addr 172.21.15.1:0/1685020193 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.878+0000 7feb8a59c640 1 --2- 172.21.15.1:0/1685020193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb8c0bf9d0 0x7feb8c158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.878+0000 7feb8ad9d640 1 --2- 172.21.15.1:0/1685020193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb8c14b360 0x7feb8c1ee1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.878+0000 7feb89d9b640 1 -- 172.21.15.1:0/1685020193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb8c14b360 msgr2=0x7feb8c1ee1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.878+0000 7feb89d9b640 1 --2- 172.21.15.1:0/1685020193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb8c14b360 0x7feb8c1ee1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.878+0000 7feb89d9b640 1 -- 172.21.15.1:0/1685020193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb8c0bf9d0 msgr2=0x7feb8c158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.878+0000 7feb89d9b640 1 --2- 172.21.15.1:0/1685020193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb8c0bf9d0 0x7feb8c158790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.878+0000 7feb89d9b640 1 -- 172.21.15.1:0/1685020193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c156e90 con 0x7feb8c152650 2024-03-06T07:22:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.878+0000 7feb8ad9d640 1 --2- 172.21.15.1:0/1685020193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb8c14b360 0x7feb8c1ee1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.879+0000 7feb89d9b640 1 --2- 172.21.15.1:0/1685020193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb8c152650 0x7feb8c14ae20 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7feb8000c400 tx=0x7feb8000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.879+0000 7feb7affd640 1 -- 172.21.15.1:0/1685020193 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb800052d0 con 0x7feb8c152650 2024-03-06T07:22:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.879+0000 7feb7affd640 1 -- 172.21.15.1:0/1685020193 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb80005490 con 0x7feb8c152650 2024-03-06T07:22:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.879+0000 7feb7affd640 1 -- 172.21.15.1:0/1685020193 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb80007260 con 0x7feb8c152650 2024-03-06T07:22:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.879+0000 7feb908ed640 1 -- 172.21.15.1:0/1685020193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb8c157170 con 0x7feb8c152650 2024-03-06T07:22:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.879+0000 7feb908ed640 1 -- 172.21.15.1:0/1685020193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8c151d80 con 0x7feb8c152650 2024-03-06T07:22:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.880+0000 7feb908ed640 1 -- 172.21.15.1:0/1685020193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb8c067fd0 con 0x7feb8c152650 2024-03-06T07:22:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.880+0000 7feb7affd640 1 -- 172.21.15.1:0/1685020193 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb80007400 con 0x7feb8c152650 2024-03-06T07:22:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.881+0000 7feb7affd640 1 --2- 172.21.15.1:0/1685020193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb6c06b500 0x7feb6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.881+0000 7feb7affd640 1 -- 172.21.15.1:0/1685020193 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7feb80089c10 con 0x7feb8c152650 2024-03-06T07:22:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.881+0000 7feb8a59c640 1 --2- 172.21.15.1:0/1685020193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb6c06b500 0x7feb6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.882+0000 7feb8a59c640 1 --2- 172.21.15.1:0/1685020193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb6c06b500 0x7feb6c06d9c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7feb7c036000 tx=0x7feb7c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:35.889+0000 7feb7affd640 1 -- 172.21.15.1:0/1685020193 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb80057e20 con 0x7feb8c152650 2024-03-06T07:22:36.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.107+0000 7feb908ed640 1 -- 172.21.15.1:0/1685020193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb8c0b06f0 con 0x7feb8c152650 2024-03-06T07:22:36.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.107+0000 7feb7affd640 1 -- 172.21.15.1:0/1685020193 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6251 (secure 0 0 0) 0x7feb8005bec0 con 0x7feb8c152650 2024-03-06T07:22:36.110 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:36.110 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":42,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5087,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.98:6837/2828783606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2828783606},{"type":"v1","addr":"172.21.15.98:6837","nonce":2828783606}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":5258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2604497680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2604497680},{"type":"v1","addr":"172.21.15.98:6836","nonce":2604497680}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:05.671602+0000","modified":"2024-03-06T07:22:32.254637+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6105":{"gid":6105,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":4,"addr":"172.21.15.1:6834/4244709430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4244709430},{"type":"v1","addr":"172.21.15.1:6834","nonce":4244709430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6105,"qdb_cluster":[6105]},"id":3},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:21.047528+0000","modified":"2024-03-06T07:22:24.069892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5344":{"gid":5344,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":7,"addr":"172.21.15.1:6835/3845300501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":3845300501},{"type":"v1","addr":"172.21.15.1:6835","nonce":3845300501}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5344,"qdb_cluster":[5344]},"id":4}]} 2024-03-06T07:22:36.110 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 42 2024-03-06T07:22:36.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.109+0000 7feb908ed640 1 -- 172.21.15.1:0/1685020193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb6c06b500 msgr2=0x7feb6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.109+0000 7feb908ed640 1 --2- 172.21.15.1:0/1685020193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb6c06b500 0x7feb6c06d9c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7feb7c036000 tx=0x7feb7c03c040 comp rx=0 tx=0).stop 2024-03-06T07:22:36.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.109+0000 7feb908ed640 1 -- 172.21.15.1:0/1685020193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb8c152650 msgr2=0x7feb8c14ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.109+0000 7feb908ed640 1 --2- 172.21.15.1:0/1685020193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb8c152650 0x7feb8c14ae20 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7feb8000c400 tx=0x7feb8000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.109+0000 7feb908ed640 1 -- 172.21.15.1:0/1685020193 shutdown_connections 2024-03-06T07:22:36.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.109+0000 7feb908ed640 1 --2- 172.21.15.1:0/1685020193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb6c06b500 0x7feb6c06d9c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.109+0000 7feb908ed640 1 --2- 172.21.15.1:0/1685020193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb8c14b360 0x7feb8c1ee1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.109+0000 7feb908ed640 1 --2- 172.21.15.1:0/1685020193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb8c152650 0x7feb8c14ae20 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.109+0000 7feb908ed640 1 --2- 172.21.15.1:0/1685020193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb8c0bf9d0 0x7feb8c158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.109+0000 7feb908ed640 1 -- 172.21.15.1:0/1685020193 >> 172.21.15.1:0/1685020193 conn(0x7feb8c066ee0 msgr2=0x7feb8c0c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.109+0000 7feb908ed640 1 -- 172.21.15.1:0/1685020193 shutdown_connections 2024-03-06T07:22:36.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.109+0000 7feb908ed640 1 -- 172.21.15.1:0/1685020193 wait complete. 2024-03-06T07:22:36.122 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:36.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.236+0000 7f6e55556640 1 Processor -- start 2024-03-06T07:22:36.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.236+0000 7f6e55556640 1 -- start start 2024-03-06T07:22:36.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.237+0000 7f6e55556640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e50152820 0x7f6e500c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.237+0000 7f6e55556640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6e500c0670 0x7f6e500c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.237+0000 7f6e55556640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e500c7180 con 0x7f6e500c7930 2024-03-06T07:22:36.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.237+0000 7f6e55556640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e50155460 con 0x7f6e500c0670 2024-03-06T07:22:36.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.237+0000 7f6e55556640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e500c6bc0 con 0x7f6e50152820 2024-03-06T07:22:36.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.237+0000 7f6e4effd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e50152820 0x7f6e500c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.237+0000 7f6e47fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6e500c0670 0x7f6e500c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.237+0000 7f6e4effd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e50152820 0x7f6e500c0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:35820/0 (socket says 172.21.15.1:35820) 2024-03-06T07:22:36.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.237+0000 7f6e47fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6e500c0670 0x7f6e500c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:33100/0 (socket says 172.21.15.1:33100) 2024-03-06T07:22:36.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.237+0000 7f6e4effd640 1 -- 172.21.15.1:0/294415080 learned_addr learned my addr 172.21.15.1:0/294415080 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:36.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.237+0000 7f6e47fff640 1 -- 172.21.15.1:0/294415080 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e50152820 msgr2=0x7f6e500c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.237+0000 7f6e47fff640 1 --2- 172.21.15.1:0/294415080 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e50152820 0x7f6e500c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.237+0000 7f6e47fff640 1 -- 172.21.15.1:0/294415080 >> v1:172.21.15.98:6789/0 conn(0x7f6e500c7930 legacy=0x7f6e50158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.237+0000 7f6e4e7fc640 1 -- 172.21.15.1:0/294415080 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727202451 0 0) 0x7f6e500c7180 con 0x7f6e500c7930 2024-03-06T07:22:36.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.237+0000 7f6e47fff640 1 -- 172.21.15.1:0/294415080 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e500c6ea0 con 0x7f6e500c0670 2024-03-06T07:22:36.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.239+0000 7f6e47fff640 1 --2- 172.21.15.1:0/294415080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e500c0670 0x7f6e500c73f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6e380097a0 tx=0x7f6e3802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee1dfa06ab9f5fc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.239+0000 7f6e4e7fc640 1 -- 172.21.15.1:0/294415080 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e38035020 con 0x7f6e500c0670 2024-03-06T07:22:36.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.239+0000 7f6e4e7fc640 1 -- 172.21.15.1:0/294415080 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e38031830 con 0x7f6e500c0670 2024-03-06T07:22:36.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.239+0000 7f6e4e7fc640 1 -- 172.21.15.1:0/294415080 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e38031c00 con 0x7f6e500c0670 2024-03-06T07:22:36.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.239+0000 7f6e55556640 1 -- 172.21.15.1:0/294415080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e500c0670 msgr2=0x7f6e500c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.239+0000 7f6e55556640 1 --2- 172.21.15.1:0/294415080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e500c0670 0x7f6e500c73f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6e380097a0 tx=0x7f6e3802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.239+0000 7f6e55556640 1 -- 172.21.15.1:0/294415080 shutdown_connections 2024-03-06T07:22:36.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.239+0000 7f6e55556640 1 --2- 172.21.15.1:0/294415080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e500c0670 0x7f6e500c73f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.239+0000 7f6e55556640 1 --2- 172.21.15.1:0/294415080 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e50152820 0x7f6e500c0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.239+0000 7f6e55556640 1 -- 172.21.15.1:0/294415080 >> 172.21.15.1:0/294415080 conn(0x7f6e50072720 msgr2=0x7f6e50072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.239+0000 7f6e55556640 1 -- 172.21.15.1:0/294415080 shutdown_connections 2024-03-06T07:22:36.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.239+0000 7f6e55556640 1 -- 172.21.15.1:0/294415080 wait complete. 2024-03-06T07:22:36.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.241+0000 7f6e55556640 1 Processor -- start 2024-03-06T07:22:36.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.241+0000 7f6e55556640 1 -- start start 2024-03-06T07:22:36.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.242+0000 7f6e55556640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e500c0670 0x7f6e500ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.242+0000 7f6e55556640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e500c7930 0x7f6e501482e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.242+0000 7f6e55556640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e50152820 0x7f6e50148820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.242+0000 7f6e55556640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e50152d80 con 0x7f6e500c0670 2024-03-06T07:22:36.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.242+0000 7f6e55556640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e50152c00 con 0x7f6e500c7930 2024-03-06T07:22:36.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.242+0000 7f6e55556640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e500af6f0 con 0x7f6e50152820 2024-03-06T07:22:36.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.242+0000 7f6e4effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e500c0670 0x7f6e500ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.242+0000 7f6e4effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e500c0670 0x7f6e500ca060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57124/0 (socket says 172.21.15.1:57124) 2024-03-06T07:22:36.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.242+0000 7f6e4effd640 1 -- 172.21.15.1:0/1201747527 learned_addr learned my addr 172.21.15.1:0/1201747527 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:36.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.243+0000 7f6e47fff640 1 --2- 172.21.15.1:0/1201747527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e500c7930 0x7f6e501482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.243+0000 7f6e4f7fe640 1 --2- 172.21.15.1:0/1201747527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e50152820 0x7f6e50148820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.243+0000 7f6e4effd640 1 -- 172.21.15.1:0/1201747527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e50152820 msgr2=0x7f6e50148820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.243+0000 7f6e4effd640 1 --2- 172.21.15.1:0/1201747527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e50152820 0x7f6e50148820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.243+0000 7f6e4effd640 1 -- 172.21.15.1:0/1201747527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e500c7930 msgr2=0x7f6e501482e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.243+0000 7f6e4effd640 1 --2- 172.21.15.1:0/1201747527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e500c7930 0x7f6e501482e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.243+0000 7f6e4effd640 1 -- 172.21.15.1:0/1201747527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e501ebc00 con 0x7f6e500c0670 2024-03-06T07:22:36.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.243+0000 7f6e4effd640 1 --2- 172.21.15.1:0/1201747527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e500c0670 0x7f6e500ca060 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000e840 tx=0x7f6e4000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.243+0000 7f6e477fe640 1 -- 172.21.15.1:0/1201747527 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e40017070 con 0x7f6e500c0670 2024-03-06T07:22:36.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.243+0000 7f6e477fe640 1 -- 172.21.15.1:0/1201747527 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e4000c8e0 con 0x7f6e500c0670 2024-03-06T07:22:36.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.243+0000 7f6e477fe640 1 -- 172.21.15.1:0/1201747527 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4000ccd0 con 0x7f6e500c0670 2024-03-06T07:22:36.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.243+0000 7f6e55556640 1 -- 172.21.15.1:0/1201747527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5014abe0 con 0x7f6e500c0670 2024-03-06T07:22:36.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.243+0000 7f6e55556640 1 -- 172.21.15.1:0/1201747527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e500ca5a0 con 0x7f6e500c0670 2024-03-06T07:22:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.244+0000 7f6e55556640 1 -- 172.21.15.1:0/1201747527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e500686e0 con 0x7f6e500c0670 2024-03-06T07:22:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.245+0000 7f6e477fe640 1 -- 172.21.15.1:0/1201747527 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e4001a070 con 0x7f6e500c0670 2024-03-06T07:22:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.245+0000 7f6e477fe640 1 --2- 172.21.15.1:0/1201747527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e3006b4d0 0x7f6e3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.245+0000 7f6e477fe640 1 -- 172.21.15.1:0/1201747527 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f6e4008a4c0 con 0x7f6e500c0670 2024-03-06T07:22:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.245+0000 7f6e47fff640 1 --2- 172.21.15.1:0/1201747527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e3006b4d0 0x7f6e3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.246+0000 7f6e47fff640 1 --2- 172.21.15.1:0/1201747527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e3006b4d0 0x7f6e3006d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6e38036000 tx=0x7f6e3803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.249+0000 7f6e477fe640 1 -- 172.21.15.1:0/1201747527 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e400587f0 con 0x7f6e500c0670 2024-03-06T07:22:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.451+0000 7f6e55556640 1 -- 172.21.15.1:0/1201747527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e5006db80 con 0x7f6e500c0670 2024-03-06T07:22:36.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.453+0000 7f6e477fe640 1 -- 172.21.15.1:0/1201747527 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+20801 (secure 0 0 0) 0x7f6e4005c890 con 0x7f6e500c0670 2024-03-06T07:22:36.455 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:36.455 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":58,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:29.247339+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:03.930333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:04.634569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:22:18.907422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:22:20.028174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6836/2937068915":"2024-03-07T07:22:29.247312+0000","172.21.15.98:6834/2937068915":"2024-03-07T07:22:29.247312+0000","172.21.15.1:6834/131367647":"2024-03-07T07:22:26.740110+0000","172.21.15.1:6832/131367647":"2024-03-07T07:22:26.740110+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.455+0000 7f6e55556640 1 -- 172.21.15.1:0/1201747527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e3006b4d0 msgr2=0x7f6e3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.455+0000 7f6e55556640 1 --2- 172.21.15.1:0/1201747527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e3006b4d0 0x7f6e3006d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6e38036000 tx=0x7f6e3803d040 comp rx=0 tx=0).stop 2024-03-06T07:22:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.455+0000 7f6e55556640 1 -- 172.21.15.1:0/1201747527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e500c0670 msgr2=0x7f6e500ca060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.455+0000 7f6e55556640 1 --2- 172.21.15.1:0/1201747527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e500c0670 0x7f6e500ca060 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000e840 tx=0x7f6e4000ed00 comp rx=0 tx=0).stop 2024-03-06T07:22:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.455+0000 7f6e55556640 1 -- 172.21.15.1:0/1201747527 shutdown_connections 2024-03-06T07:22:36.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.455+0000 7f6e55556640 1 --2- 172.21.15.1:0/1201747527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e3006b4d0 0x7f6e3006d990 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.455+0000 7f6e55556640 1 --2- 172.21.15.1:0/1201747527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e50152820 0x7f6e50148820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.455+0000 7f6e55556640 1 --2- 172.21.15.1:0/1201747527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e500c7930 0x7f6e501482e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.455+0000 7f6e55556640 1 --2- 172.21.15.1:0/1201747527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e500c0670 0x7f6e500ca060 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.455+0000 7f6e55556640 1 -- 172.21.15.1:0/1201747527 >> 172.21.15.1:0/1201747527 conn(0x7f6e50072720 msgr2=0x7f6e5014f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.455+0000 7f6e55556640 1 -- 172.21.15.1:0/1201747527 shutdown_connections 2024-03-06T07:22:36.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.455+0000 7f6e55556640 1 -- 172.21.15.1:0/1201747527 wait complete. 2024-03-06T07:22:36.472 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:22:36.472 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:36.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.571+0000 7fc5beedf640 1 Processor -- start 2024-03-06T07:22:36.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.572+0000 7fc5beedf640 1 -- start start 2024-03-06T07:22:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.573+0000 7fc5beedf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc5b8152650 0x7fc5b8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.573+0000 7fc5beedf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b80c6b10 0x7fc5b8158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.573+0000 7fc5beedf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b80c2fc0 con 0x7fc5b80bf9d0 2024-03-06T07:22:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.573+0000 7fc5beedf640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b8155800 con 0x7fc5b8152650 2024-03-06T07:22:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.573+0000 7fc5beedf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80c6690 con 0x7fc5b80c6b10 2024-03-06T07:22:36.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.573+0000 7fc5bcc7b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc5b8152650 0x7fc5b8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.573+0000 7fc5affff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b80c6b10 0x7fc5b8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.573+0000 7fc5bcc7b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc5b8152650 0x7fc5b8155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:33114/0 (socket says 172.21.15.1:33114) 2024-03-06T07:22:36.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.573+0000 7fc5bd47c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc5b80bf9d0 0x7fc5b80c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:56146/0 (socket says 172.21.15.1:56146) 2024-03-06T07:22:36.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.573+0000 7fc5bcc7b640 1 -- 172.21.15.1:0/1606674018 learned_addr learned my addr 172.21.15.1:0/1606674018 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:36.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.574+0000 7fc5bcc7b640 1 -- 172.21.15.1:0/1606674018 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b80c6b10 msgr2=0x7fc5b8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.574+0000 7fc5bcc7b640 1 --2- 172.21.15.1:0/1606674018 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b80c6b10 0x7fc5b8158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.574+0000 7fc5bcc7b640 1 -- 172.21.15.1:0/1606674018 >> v1:172.21.15.98:6789/0 conn(0x7fc5b80bf9d0 legacy=0x7fc5b80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.574+0000 7fc5bcc7b640 1 -- 172.21.15.1:0/1606674018 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b80c7280 con 0x7fc5b8152650 2024-03-06T07:22:36.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.574+0000 7fc5bcc7b640 1 --2- 172.21.15.1:0/1606674018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b8152650 0x7fc5b8155c90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009870 tx=0x7fc5b402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8264fb395df5297 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.575+0000 7fc5af7fe640 1 -- 172.21.15.1:0/1606674018 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4035020 con 0x7fc5b8152650 2024-03-06T07:22:36.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.575+0000 7fc5af7fe640 1 -- 172.21.15.1:0/1606674018 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b4031640 con 0x7fc5b8152650 2024-03-06T07:22:36.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.575+0000 7fc5af7fe640 1 -- 172.21.15.1:0/1606674018 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b402d040 con 0x7fc5b8152650 2024-03-06T07:22:36.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.575+0000 7fc5beedf640 1 -- 172.21.15.1:0/1606674018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b8152650 msgr2=0x7fc5b8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.575+0000 7fc5beedf640 1 --2- 172.21.15.1:0/1606674018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b8152650 0x7fc5b8155c90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009870 tx=0x7fc5b402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.575+0000 7fc5beedf640 1 -- 172.21.15.1:0/1606674018 shutdown_connections 2024-03-06T07:22:36.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.575+0000 7fc5beedf640 1 --2- 172.21.15.1:0/1606674018 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b80c6b10 0x7fc5b8158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.575+0000 7fc5beedf640 1 --2- 172.21.15.1:0/1606674018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b8152650 0x7fc5b8155c90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.575+0000 7fc5beedf640 1 -- 172.21.15.1:0/1606674018 >> 172.21.15.1:0/1606674018 conn(0x7fc5b8066ee0 msgr2=0x7fc5b80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.575+0000 7fc5beedf640 1 -- 172.21.15.1:0/1606674018 shutdown_connections 2024-03-06T07:22:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.575+0000 7fc5beedf640 1 -- 172.21.15.1:0/1606674018 wait complete. 2024-03-06T07:22:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.576+0000 7fc5beedf640 1 Processor -- start 2024-03-06T07:22:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.576+0000 7fc5beedf640 1 -- start start 2024-03-06T07:22:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.578+0000 7fc5beedf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b80bf9d0 0x7fc5b80c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.578+0000 7fc5beedf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b8152650 0x7fc5b81eea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.578+0000 7fc5beedf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5b814ed90 0x7fc5b8141d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.578+0000 7fc5beedf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b8155490 con 0x7fc5b814ed90 2024-03-06T07:22:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.578+0000 7fc5beedf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b81551b0 con 0x7fc5b8152650 2024-03-06T07:22:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.578+0000 7fc5beedf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b8152a30 con 0x7fc5b80bf9d0 2024-03-06T07:22:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.578+0000 7fc5bd47c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5b814ed90 0x7fc5b8141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.578+0000 7fc5bd47c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5b814ed90 0x7fc5b8141d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57134/0 (socket says 172.21.15.1:57134) 2024-03-06T07:22:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.578+0000 7fc5bd47c640 1 -- 172.21.15.1:0/282029540 learned_addr learned my addr 172.21.15.1:0/282029540 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.578+0000 7fc5bcc7b640 1 --2- 172.21.15.1:0/282029540 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b80bf9d0 0x7fc5b80c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.578+0000 7fc5affff640 1 --2- 172.21.15.1:0/282029540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b8152650 0x7fc5b81eea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.578+0000 7fc5bd47c640 1 -- 172.21.15.1:0/282029540 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b80bf9d0 msgr2=0x7fc5b80c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.578+0000 7fc5bd47c640 1 --2- 172.21.15.1:0/282029540 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b80bf9d0 0x7fc5b80c9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.578+0000 7fc5bd47c640 1 -- 172.21.15.1:0/282029540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b8152650 msgr2=0x7fc5b81eea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.578+0000 7fc5bd47c640 1 --2- 172.21.15.1:0/282029540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b8152650 0x7fc5b81eea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.578+0000 7fc5bd47c640 1 -- 172.21.15.1:0/282029540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b8154f00 con 0x7fc5b814ed90 2024-03-06T07:22:36.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.578+0000 7fc5bd47c640 1 --2- 172.21.15.1:0/282029540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5b814ed90 0x7fc5b8141d90 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000c730 tx=0x7fc5b000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.579+0000 7fc5ad7fa640 1 -- 172.21.15.1:0/282029540 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0017070 con 0x7fc5b814ed90 2024-03-06T07:22:36.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.579+0000 7fc5ad7fa640 1 -- 172.21.15.1:0/282029540 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b000ce40 con 0x7fc5b814ed90 2024-03-06T07:22:36.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.579+0000 7fc5ad7fa640 1 -- 172.21.15.1:0/282029540 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b000a840 con 0x7fc5b814ed90 2024-03-06T07:22:36.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.579+0000 7fc5beedf640 1 -- 172.21.15.1:0/282029540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b814dc30 con 0x7fc5b814ed90 2024-03-06T07:22:36.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.579+0000 7fc5beedf640 1 -- 172.21.15.1:0/282029540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b814c4b0 con 0x7fc5b814ed90 2024-03-06T07:22:36.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.580+0000 7fc5beedf640 1 -- 172.21.15.1:0/282029540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b8067fd0 con 0x7fc5b814ed90 2024-03-06T07:22:36.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.581+0000 7fc5ad7fa640 1 -- 172.21.15.1:0/282029540 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5b001a070 con 0x7fc5b814ed90 2024-03-06T07:22:36.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.581+0000 7fc5ad7fa640 1 --2- 172.21.15.1:0/282029540 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5a406b500 0x7fc5a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.581+0000 7fc5ad7fa640 1 -- 172.21.15.1:0/282029540 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fc5b0089b00 con 0x7fc5b814ed90 2024-03-06T07:22:36.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.581+0000 7fc5bcc7b640 1 --2- 172.21.15.1:0/282029540 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5a406b500 0x7fc5a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.582+0000 7fc5bcc7b640 1 --2- 172.21.15.1:0/282029540 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5a406b500 0x7fc5a406d9c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4036000 tx=0x7fc5b403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.586+0000 7fc5ad7fa640 1 -- 172.21.15.1:0/282029540 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5b0057e30 con 0x7fc5b814ed90 2024-03-06T07:22:36.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.823+0000 7fc5beedf640 1 -- 172.21.15.1:0/282029540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5b80c5410 con 0x7fc5b814ed90 2024-03-06T07:22:36.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.824+0000 7fc5ad7fa640 1 -- 172.21.15.1:0/282029540 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6251 (secure 0 0 0) 0x7fc5b005bed0 con 0x7fc5b814ed90 2024-03-06T07:22:36.826 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:36.826 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":42,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5087,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.98:6837/2828783606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2828783606},{"type":"v1","addr":"172.21.15.98:6837","nonce":2828783606}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":5258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2604497680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2604497680},{"type":"v1","addr":"172.21.15.98:6836","nonce":2604497680}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:05.671602+0000","modified":"2024-03-06T07:22:32.254637+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6105":{"gid":6105,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":4,"addr":"172.21.15.1:6834/4244709430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4244709430},{"type":"v1","addr":"172.21.15.1:6834","nonce":4244709430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6105,"qdb_cluster":[6105]},"id":3},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:21.047528+0000","modified":"2024-03-06T07:22:24.069892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5344":{"gid":5344,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":7,"addr":"172.21.15.1:6835/3845300501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":3845300501},{"type":"v1","addr":"172.21.15.1:6835","nonce":3845300501}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5344,"qdb_cluster":[5344]},"id":4}]} 2024-03-06T07:22:36.826 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 42 2024-03-06T07:22:36.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.826+0000 7fc5beedf640 1 -- 172.21.15.1:0/282029540 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5a406b500 msgr2=0x7fc5a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.826+0000 7fc5beedf640 1 --2- 172.21.15.1:0/282029540 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5a406b500 0x7fc5a406d9c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4036000 tx=0x7fc5b403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:36.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.826+0000 7fc5beedf640 1 -- 172.21.15.1:0/282029540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5b814ed90 msgr2=0x7fc5b8141d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.826+0000 7fc5beedf640 1 --2- 172.21.15.1:0/282029540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5b814ed90 0x7fc5b8141d90 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000c730 tx=0x7fc5b000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.826+0000 7fc5beedf640 1 -- 172.21.15.1:0/282029540 shutdown_connections 2024-03-06T07:22:36.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.826+0000 7fc5beedf640 1 --2- 172.21.15.1:0/282029540 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5a406b500 0x7fc5a406d9c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.826+0000 7fc5beedf640 1 --2- 172.21.15.1:0/282029540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5b814ed90 0x7fc5b8141d90 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.826+0000 7fc5beedf640 1 --2- 172.21.15.1:0/282029540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5b8152650 0x7fc5b81eea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.826+0000 7fc5beedf640 1 --2- 172.21.15.1:0/282029540 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5b80bf9d0 0x7fc5b80c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.826+0000 7fc5beedf640 1 -- 172.21.15.1:0/282029540 >> 172.21.15.1:0/282029540 conn(0x7fc5b8066ee0 msgr2=0x7fc5b80c5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.826+0000 7fc5beedf640 1 -- 172.21.15.1:0/282029540 shutdown_connections 2024-03-06T07:22:36.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.826+0000 7fc5beedf640 1 -- 172.21.15.1:0/282029540 wait complete. 2024-03-06T07:22:36.839 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:36.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.942+0000 7f848390c640 1 Processor -- start 2024-03-06T07:22:36.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.942+0000 7f848390c640 1 -- start start 2024-03-06T07:22:36.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.943+0000 7f848390c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f847c152650 0x7f847c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.943+0000 7f848390c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c0c2fc0 con 0x7f847c0c6b10 2024-03-06T07:22:36.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.943+0000 7f848390c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c158da0 con 0x7f847c0bf9d0 2024-03-06T07:22:36.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.943+0000 7f848390c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f847c155800 con 0x7f847c152650 2024-03-06T07:22:36.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.943+0000 7f8480ea7640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f847c0c6b10 0x7f847c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37824/0 (socket says 172.21.15.1:37824) 2024-03-06T07:22:36.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.943+0000 7f8480ea7640 1 -- 172.21.15.1:0/2972743383 learned_addr learned my addr 172.21.15.1:0/2972743383 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:36.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.943+0000 7f84816a8640 1 --2- 172.21.15.1:0/2972743383 >> v2:172.21.15.98:3300/0 conn(0x7f847c152650 0x7f847c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.943+0000 7f8473fff640 1 -- 172.21.15.1:0/2972743383 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858794465 0 0) 0x7f847c0c2fc0 con 0x7f847c0c6b10 2024-03-06T07:22:36.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.943+0000 7f8473fff640 1 -- 172.21.15.1:0/2972743383 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8468003520 con 0x7f847c0c6b10 2024-03-06T07:22:36.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.943+0000 7f84816a8640 1 -- 172.21.15.1:0/2972743383 >> v1:172.21.15.98:6789/0 conn(0x7f847c0bf9d0 legacy=0x7f847c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.943+0000 7f84816a8640 1 -- 172.21.15.1:0/2972743383 >> v1:172.21.15.1:6789/0 conn(0x7f847c0c6b10 legacy=0x7f847c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.943+0000 7f84816a8640 1 -- 172.21.15.1:0/2972743383 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c0c6690 con 0x7f847c152650 2024-03-06T07:22:36.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.944+0000 7f84816a8640 1 --2- 172.21.15.1:0/2972743383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f847c152650 0x7f847c155c90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f84780097a0 tx=0x7f847802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8abbbfc32d689137 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.944+0000 7f8473fff640 1 -- 172.21.15.1:0/2972743383 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8478035020 con 0x7f847c152650 2024-03-06T07:22:36.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.944+0000 7f8473fff640 1 -- 172.21.15.1:0/2972743383 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8478031830 con 0x7f847c152650 2024-03-06T07:22:36.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.944+0000 7f8473fff640 1 -- 172.21.15.1:0/2972743383 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8478031be0 con 0x7f847c152650 2024-03-06T07:22:36.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.945+0000 7f848390c640 1 -- 172.21.15.1:0/2972743383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f847c152650 msgr2=0x7f847c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.945+0000 7f848390c640 1 --2- 172.21.15.1:0/2972743383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f847c152650 0x7f847c155c90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f84780097a0 tx=0x7f847802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.945+0000 7f848390c640 1 -- 172.21.15.1:0/2972743383 shutdown_connections 2024-03-06T07:22:36.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.945+0000 7f848390c640 1 --2- 172.21.15.1:0/2972743383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f847c152650 0x7f847c155c90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.945+0000 7f848390c640 1 -- 172.21.15.1:0/2972743383 >> 172.21.15.1:0/2972743383 conn(0x7f847c066ee0 msgr2=0x7f847c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.945+0000 7f848390c640 1 -- 172.21.15.1:0/2972743383 shutdown_connections 2024-03-06T07:22:36.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.945+0000 7f848390c640 1 -- 172.21.15.1:0/2972743383 wait complete. 2024-03-06T07:22:36.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.946+0000 7f848390c640 1 Processor -- start 2024-03-06T07:22:36.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.947+0000 7f848390c640 1 -- start start 2024-03-06T07:22:36.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.948+0000 7f848390c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f847c0bf9d0 0x7f847c14cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.948+0000 7f848390c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f847c152650 0x7f847c14d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.948+0000 7f848390c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c156490 0x7f847c1e1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.948+0000 7f848390c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c1551b0 con 0x7f847c156490 2024-03-06T07:22:36.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.948+0000 7f848390c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c0c6fa0 con 0x7f847c152650 2024-03-06T07:22:36.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.948+0000 7f848390c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c155490 con 0x7f847c0bf9d0 2024-03-06T07:22:36.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.948+0000 7f8481ea9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c156490 0x7f847c1e1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.948+0000 7f8481ea9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c156490 0x7f847c1e1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57150/0 (socket says 172.21.15.1:57150) 2024-03-06T07:22:36.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.948+0000 7f8481ea9640 1 -- 172.21.15.1:0/3013369494 learned_addr learned my addr 172.21.15.1:0/3013369494 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:36.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.948+0000 7f84816a8640 1 --2- 172.21.15.1:0/3013369494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f847c0bf9d0 0x7f847c14cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.948+0000 7f8480ea7640 1 --2- 172.21.15.1:0/3013369494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f847c152650 0x7f847c14d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.948+0000 7f8481ea9640 1 -- 172.21.15.1:0/3013369494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f847c0bf9d0 msgr2=0x7f847c14cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.948+0000 7f8481ea9640 1 --2- 172.21.15.1:0/3013369494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f847c0bf9d0 0x7f847c14cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.948+0000 7f8481ea9640 1 -- 172.21.15.1:0/3013369494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f847c152650 msgr2=0x7f847c14d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.948+0000 7f8481ea9640 1 --2- 172.21.15.1:0/3013369494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f847c152650 0x7f847c14d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.948+0000 7f8481ea9640 1 -- 172.21.15.1:0/3013369494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c14da40 con 0x7f847c156490 2024-03-06T07:22:36.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.948+0000 7f8481ea9640 1 --2- 172.21.15.1:0/3013369494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c156490 0x7f847c1e1280 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f847400c6e0 tx=0x7f847400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.949+0000 7f8471ffb640 1 -- 172.21.15.1:0/3013369494 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474003b70 con 0x7f847c156490 2024-03-06T07:22:36.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.949+0000 7f8471ffb640 1 -- 172.21.15.1:0/3013369494 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8474003d30 con 0x7f847c156490 2024-03-06T07:22:36.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.949+0000 7f848390c640 1 -- 172.21.15.1:0/3013369494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847c1df480 con 0x7f847c156490 2024-03-06T07:22:36.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.949+0000 7f8471ffb640 1 -- 172.21.15.1:0/3013369494 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474005310 con 0x7f847c156490 2024-03-06T07:22:36.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.949+0000 7f848390c640 1 -- 172.21.15.1:0/3013369494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c157500 con 0x7f847c156490 2024-03-06T07:22:36.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.950+0000 7f848390c640 1 -- 172.21.15.1:0/3013369494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847c0c2f50 con 0x7f847c156490 2024-03-06T07:22:36.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.950+0000 7f8471ffb640 1 -- 172.21.15.1:0/3013369494 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84740054b0 con 0x7f847c156490 2024-03-06T07:22:36.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.950+0000 7f8471ffb640 1 --2- 172.21.15.1:0/3013369494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f846006b4d0 0x7f846006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.951+0000 7f8471ffb640 1 -- 172.21.15.1:0/3013369494 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f847408a3f0 con 0x7f847c156490 2024-03-06T07:22:36.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.951+0000 7f84816a8640 1 --2- 172.21.15.1:0/3013369494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f846006b4d0 0x7f846006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.951+0000 7f84816a8640 1 --2- 172.21.15.1:0/3013369494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f846006b4d0 0x7f846006d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8478036000 tx=0x7f847803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:36.955+0000 7f8471ffb640 1 -- 172.21.15.1:0/3013369494 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f847c0c2f50 con 0x7f847c156490 2024-03-06T07:22:37.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.175+0000 7f848390c640 1 -- 172.21.15.1:0/3013369494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f847c067fd0 con 0x7f847c156490 2024-03-06T07:22:37.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.176+0000 7f8471ffb640 1 -- 172.21.15.1:0/3013369494 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6251 (secure 0 0 0) 0x7f8474058720 con 0x7f847c156490 2024-03-06T07:22:37.178 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:37.178 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":42,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5087,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.98:6837/2828783606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2828783606},{"type":"v1","addr":"172.21.15.98:6837","nonce":2828783606}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":5258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2604497680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2604497680},{"type":"v1","addr":"172.21.15.98:6836","nonce":2604497680}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:05.671602+0000","modified":"2024-03-06T07:22:32.254637+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6105":{"gid":6105,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":4,"addr":"172.21.15.1:6834/4244709430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4244709430},{"type":"v1","addr":"172.21.15.1:6834","nonce":4244709430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6105,"qdb_cluster":[6105]},"id":3},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:21.047528+0000","modified":"2024-03-06T07:22:24.069892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5344":{"gid":5344,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":7,"addr":"172.21.15.1:6835/3845300501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":3845300501},{"type":"v1","addr":"172.21.15.1:6835","nonce":3845300501}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5344,"qdb_cluster":[5344]},"id":4}]} 2024-03-06T07:22:37.178 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 42 2024-03-06T07:22:37.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.178+0000 7f848390c640 1 -- 172.21.15.1:0/3013369494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f846006b4d0 msgr2=0x7f846006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.178+0000 7f848390c640 1 --2- 172.21.15.1:0/3013369494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f846006b4d0 0x7f846006d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8478036000 tx=0x7f847803d040 comp rx=0 tx=0).stop 2024-03-06T07:22:37.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.178+0000 7f848390c640 1 -- 172.21.15.1:0/3013369494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c156490 msgr2=0x7f847c1e1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.178+0000 7f848390c640 1 --2- 172.21.15.1:0/3013369494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c156490 0x7f847c1e1280 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f847400c6e0 tx=0x7f847400cba0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.178+0000 7f848390c640 1 -- 172.21.15.1:0/3013369494 shutdown_connections 2024-03-06T07:22:37.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.178+0000 7f848390c640 1 --2- 172.21.15.1:0/3013369494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f846006b4d0 0x7f846006d990 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.178+0000 7f848390c640 1 --2- 172.21.15.1:0/3013369494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c156490 0x7f847c1e1280 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.178+0000 7f848390c640 1 --2- 172.21.15.1:0/3013369494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f847c152650 0x7f847c14d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.178+0000 7f848390c640 1 --2- 172.21.15.1:0/3013369494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f847c0bf9d0 0x7f847c14cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.178+0000 7f848390c640 1 -- 172.21.15.1:0/3013369494 >> 172.21.15.1:0/3013369494 conn(0x7f847c066ee0 msgr2=0x7f847c0c04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.178+0000 7f848390c640 1 -- 172.21.15.1:0/3013369494 shutdown_connections 2024-03-06T07:22:37.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.178+0000 7f848390c640 1 -- 172.21.15.1:0/3013369494 wait complete. 2024-03-06T07:22:37.192 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:37.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.300+0000 7f31ac622640 1 Processor -- start 2024-03-06T07:22:37.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.300+0000 7f31ac622640 1 -- start start 2024-03-06T07:22:37.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.301+0000 7f31ac622640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a4152650 0x7f31a4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.301+0000 7f31ac622640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f31a40c6b10 0x7f31a4158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.301+0000 7f31ac622640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a40c2fc0 con 0x7f31a40bf9d0 2024-03-06T07:22:37.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.301+0000 7f31ac622640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a4155800 con 0x7f31a40c6b10 2024-03-06T07:22:37.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.301+0000 7f31ac622640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a40c6690 con 0x7f31a4152650 2024-03-06T07:22:37.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.301+0000 7f31aa3be640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a4152650 0x7f31a4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.301+0000 7f31aa3be640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a4152650 0x7f31a4155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:35868/0 (socket says 172.21.15.1:35868) 2024-03-06T07:22:37.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.301+0000 7f31aa3be640 1 -- 172.21.15.1:0/1997747214 learned_addr learned my addr 172.21.15.1:0/1997747214 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:37.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.301+0000 7f31a9bbd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f31a40c6b10 0x7f31a4158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.301+0000 7f31aa3be640 1 -- 172.21.15.1:0/1997747214 >> v2:172.21.15.98:3300/0 conn(0x7f31a40c6b10 msgr2=0x7f31a4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:37.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.301+0000 7f31aa3be640 1 --2- 172.21.15.1:0/1997747214 >> v2:172.21.15.98:3300/0 conn(0x7f31a40c6b10 0x7f31a4158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.301+0000 7f31aa3be640 1 -- 172.21.15.1:0/1997747214 >> v1:172.21.15.98:6789/0 conn(0x7f31a40bf9d0 legacy=0x7f31a40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.301+0000 7f31aa3be640 1 -- 172.21.15.1:0/1997747214 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a40c7280 con 0x7f31a4152650 2024-03-06T07:22:37.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.302+0000 7f31aa3be640 1 --2- 172.21.15.1:0/1997747214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a4152650 0x7f31a4155c90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009870 tx=0x7f31a002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5207bb7eab2f48ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.302+0000 7f31a93bc640 1 -- 172.21.15.1:0/1997747214 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a0035020 con 0x7f31a4152650 2024-03-06T07:22:37.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.302+0000 7f31a93bc640 1 -- 172.21.15.1:0/1997747214 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a0031640 con 0x7f31a4152650 2024-03-06T07:22:37.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.302+0000 7f31a93bc640 1 -- 172.21.15.1:0/1997747214 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a002d040 con 0x7f31a4152650 2024-03-06T07:22:37.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.303+0000 7f31ac622640 1 -- 172.21.15.1:0/1997747214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a4152650 msgr2=0x7f31a4155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.303+0000 7f31ac622640 1 --2- 172.21.15.1:0/1997747214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a4152650 0x7f31a4155c90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009870 tx=0x7f31a002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.303+0000 7f31ac622640 1 -- 172.21.15.1:0/1997747214 shutdown_connections 2024-03-06T07:22:37.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.303+0000 7f31ac622640 1 --2- 172.21.15.1:0/1997747214 >> v2:172.21.15.98:3300/0 conn(0x7f31a40c6b10 0x7f31a4158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.303+0000 7f31ac622640 1 --2- 172.21.15.1:0/1997747214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a4152650 0x7f31a4155c90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.303+0000 7f31ac622640 1 -- 172.21.15.1:0/1997747214 >> 172.21.15.1:0/1997747214 conn(0x7f31a4066ee0 msgr2=0x7f31a40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.303+0000 7f31ac622640 1 -- 172.21.15.1:0/1997747214 shutdown_connections 2024-03-06T07:22:37.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.303+0000 7f31ac622640 1 -- 172.21.15.1:0/1997747214 wait complete. 2024-03-06T07:22:37.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.304+0000 7f31ac622640 1 Processor -- start 2024-03-06T07:22:37.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.304+0000 7f31ac622640 1 -- start start 2024-03-06T07:22:37.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31ac622640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a40bf9d0 0x7f31a41e1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31ac622640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a4152650 0x7f31a41eb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31ac622640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a40ca420 0x7f31a41ebae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31ac622640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a4155490 con 0x7f31a4152650 2024-03-06T07:22:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31ac622640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a41551b0 con 0x7f31a40bf9d0 2024-03-06T07:22:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31ac622640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a4152a30 con 0x7f31a40ca420 2024-03-06T07:22:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31a9bbd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a4152650 0x7f31a41eb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31a9bbd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a4152650 0x7f31a41eb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57160/0 (socket says 172.21.15.1:57160) 2024-03-06T07:22:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31a9bbd640 1 -- 172.21.15.1:0/2119655379 learned_addr learned my addr 172.21.15.1:0/2119655379 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31aa3be640 1 --2- 172.21.15.1:0/2119655379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a40bf9d0 0x7f31a41e1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31aabbf640 1 --2- 172.21.15.1:0/2119655379 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a40ca420 0x7f31a41ebae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31a9bbd640 1 -- 172.21.15.1:0/2119655379 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a40ca420 msgr2=0x7f31a41ebae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31a9bbd640 1 --2- 172.21.15.1:0/2119655379 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a40ca420 0x7f31a41ebae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31a9bbd640 1 -- 172.21.15.1:0/2119655379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a40bf9d0 msgr2=0x7f31a41e1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31a9bbd640 1 --2- 172.21.15.1:0/2119655379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a40bf9d0 0x7f31a41e1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31a9bbd640 1 -- 172.21.15.1:0/2119655379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a41eb130 con 0x7f31a4152650 2024-03-06T07:22:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31aabbf640 1 --2- 172.21.15.1:0/2119655379 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a40ca420 0x7f31a41ebae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31aa3be640 1 --2- 172.21.15.1:0/2119655379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a40bf9d0 0x7f31a41e1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.305+0000 7f31a9bbd640 1 --2- 172.21.15.1:0/2119655379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a4152650 0x7f31a41eb5a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f319400e9b0 tx=0x7f319400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.306+0000 7f319affd640 1 -- 172.21.15.1:0/2119655379 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3194017070 con 0x7f31a4152650 2024-03-06T07:22:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.306+0000 7f319affd640 1 -- 172.21.15.1:0/2119655379 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319400c9a0 con 0x7f31a4152650 2024-03-06T07:22:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.306+0000 7f319affd640 1 -- 172.21.15.1:0/2119655379 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319400cd90 con 0x7f31a4152650 2024-03-06T07:22:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.306+0000 7f31ac622640 1 -- 172.21.15.1:0/2119655379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a414de20 con 0x7f31a4152650 2024-03-06T07:22:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.306+0000 7f31ac622640 1 -- 172.21.15.1:0/2119655379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a40c8c30 con 0x7f31a4152650 2024-03-06T07:22:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.307+0000 7f31ac622640 1 -- 172.21.15.1:0/2119655379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a4067fd0 con 0x7f31a4152650 2024-03-06T07:22:37.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.307+0000 7f319affd640 1 -- 172.21.15.1:0/2119655379 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f319401a070 con 0x7f31a4152650 2024-03-06T07:22:37.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.307+0000 7f319affd640 1 --2- 172.21.15.1:0/2119655379 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f318806b4d0 0x7f318806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.307+0000 7f319affd640 1 -- 172.21.15.1:0/2119655379 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f319408a740 con 0x7f31a4152650 2024-03-06T07:22:37.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.308+0000 7f31aa3be640 1 --2- 172.21.15.1:0/2119655379 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f318806b4d0 0x7f318806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.308+0000 7f31aa3be640 1 --2- 172.21.15.1:0/2119655379 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f318806b4d0 0x7f318806d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f31a0036000 tx=0x7f31a0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.316+0000 7f319affd640 1 -- 172.21.15.1:0/2119655379 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31940590c0 con 0x7f31a4152650 2024-03-06T07:22:37.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.517+0000 7f31ac622640 1 -- 172.21.15.1:0/2119655379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31a41eb300 con 0x7f31a4152650 2024-03-06T07:22:37.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.518+0000 7f319affd640 1 -- 172.21.15.1:0/2119655379 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+20801 (secure 0 0 0) 0x7f31940589c0 con 0x7f31a4152650 2024-03-06T07:22:37.520 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:37.520 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":58,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:29.247339+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:03.930333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:04.634569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:22:18.907422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:22:20.028174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6836/2937068915":"2024-03-07T07:22:29.247312+0000","172.21.15.98:6834/2937068915":"2024-03-07T07:22:29.247312+0000","172.21.15.1:6834/131367647":"2024-03-07T07:22:26.740110+0000","172.21.15.1:6832/131367647":"2024-03-07T07:22:26.740110+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.521+0000 7f31ac622640 1 -- 172.21.15.1:0/2119655379 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f318806b4d0 msgr2=0x7f318806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.521+0000 7f31ac622640 1 --2- 172.21.15.1:0/2119655379 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f318806b4d0 0x7f318806d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f31a0036000 tx=0x7f31a0008030 comp rx=0 tx=0).stop 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.521+0000 7f31ac622640 1 -- 172.21.15.1:0/2119655379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a4152650 msgr2=0x7f31a41eb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.521+0000 7f31ac622640 1 --2- 172.21.15.1:0/2119655379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a4152650 0x7f31a41eb5a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f319400e9b0 tx=0x7f319400ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.521+0000 7f31ac622640 1 -- 172.21.15.1:0/2119655379 shutdown_connections 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.521+0000 7f31ac622640 1 --2- 172.21.15.1:0/2119655379 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f318806b4d0 0x7f318806d990 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.521+0000 7f31ac622640 1 --2- 172.21.15.1:0/2119655379 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31a40ca420 0x7f31a41ebae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.521+0000 7f31ac622640 1 --2- 172.21.15.1:0/2119655379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a4152650 0x7f31a41eb5a0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.521+0000 7f31ac622640 1 --2- 172.21.15.1:0/2119655379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a40bf9d0 0x7f31a41e1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.521+0000 7f31ac622640 1 -- 172.21.15.1:0/2119655379 >> 172.21.15.1:0/2119655379 conn(0x7f31a4066ee0 msgr2=0x7f31a40c3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.521+0000 7f31ac622640 1 -- 172.21.15.1:0/2119655379 shutdown_connections 2024-03-06T07:22:37.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.521+0000 7f31ac622640 1 -- 172.21.15.1:0/2119655379 wait complete. 2024-03-06T07:22:37.534 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:22:37.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.642+0000 7ffa4ebb6640 1 Processor -- start 2024-03-06T07:22:37.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.642+0000 7ffa4ebb6640 1 -- start start 2024-03-06T07:22:37.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.642+0000 7ffa4ebb6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffa480c5960 0x7ffa480c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.642+0000 7ffa4ebb6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffa480c4ec0 0x7ffa4815d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.642+0000 7ffa4ebb6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa480c67c0 con 0x7ffa48153ab0 2024-03-06T07:22:37.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.642+0000 7ffa4ebb6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa480c6a60 con 0x7ffa480c4ec0 2024-03-06T07:22:37.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.642+0000 7ffa4ebb6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa480c6d40 con 0x7ffa480c5960 2024-03-06T07:22:37.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.643+0000 7ffa4d153640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffa480c4ec0 0x7ffa4815d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.643+0000 7ffa4d153640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffa480c4ec0 0x7ffa4815d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57170/0 (socket says 172.21.15.1:57170) 2024-03-06T07:22:37.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.643+0000 7ffa4d153640 1 -- 172.21.15.1:0/1352629676 learned_addr learned my addr 172.21.15.1:0/1352629676 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:37.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.643+0000 7ffa4c952640 1 --2- 172.21.15.1:0/1352629676 >> v2:172.21.15.98:3300/0 conn(0x7ffa480c5960 0x7ffa480c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.643+0000 7ffa4d153640 1 -- 172.21.15.1:0/1352629676 >> v2:172.21.15.98:3300/0 conn(0x7ffa480c5960 msgr2=0x7ffa480c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:37.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.643+0000 7ffa4d153640 1 --2- 172.21.15.1:0/1352629676 >> v2:172.21.15.98:3300/0 conn(0x7ffa480c5960 0x7ffa480c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.643+0000 7ffa4d153640 1 -- 172.21.15.1:0/1352629676 >> v1:172.21.15.98:6789/0 conn(0x7ffa48153ab0 legacy=0x7ffa48070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:37.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.643+0000 7ffa4d153640 1 -- 172.21.15.1:0/1352629676 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa480c7020 con 0x7ffa480c4ec0 2024-03-06T07:22:37.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.644+0000 7ffa4d153640 1 --2- 172.21.15.1:0/1352629676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa480c4ec0 0x7ffa4815d9b0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ffa4806fef0 tx=0x7ffa4002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a93efbae27059f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.644+0000 7ffa477fe640 1 -- 172.21.15.1:0/1352629676 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4002ca70 con 0x7ffa480c4ec0 2024-03-06T07:22:37.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.644+0000 7ffa477fe640 1 -- 172.21.15.1:0/1352629676 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4002cc10 con 0x7ffa480c4ec0 2024-03-06T07:22:37.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.644+0000 7ffa477fe640 1 -- 172.21.15.1:0/1352629676 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa400343f0 con 0x7ffa480c4ec0 2024-03-06T07:22:37.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.644+0000 7ffa4ebb6640 1 -- 172.21.15.1:0/1352629676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa480c4ec0 msgr2=0x7ffa4815d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.644+0000 7ffa4ebb6640 1 --2- 172.21.15.1:0/1352629676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa480c4ec0 0x7ffa4815d9b0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ffa4806fef0 tx=0x7ffa4002bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.644+0000 7ffa4ebb6640 1 -- 172.21.15.1:0/1352629676 shutdown_connections 2024-03-06T07:22:37.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.644+0000 7ffa4ebb6640 1 --2- 172.21.15.1:0/1352629676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa480c4ec0 0x7ffa4815d9b0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.644+0000 7ffa4ebb6640 1 --2- 172.21.15.1:0/1352629676 >> v2:172.21.15.98:3300/0 conn(0x7ffa480c5960 0x7ffa480c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.644+0000 7ffa4ebb6640 1 -- 172.21.15.1:0/1352629676 >> 172.21.15.1:0/1352629676 conn(0x7ffa480aa550 msgr2=0x7ffa480aa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.645+0000 7ffa4ebb6640 1 -- 172.21.15.1:0/1352629676 shutdown_connections 2024-03-06T07:22:37.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.645+0000 7ffa4ebb6640 1 -- 172.21.15.1:0/1352629676 wait complete. 2024-03-06T07:22:37.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.646+0000 7ffa4ebb6640 1 Processor -- start 2024-03-06T07:22:37.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.646+0000 7ffa4ebb6640 1 -- start start 2024-03-06T07:22:37.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.646+0000 7ffa4ebb6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa480c4ec0 0x7ffa480c1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.646+0000 7ffa4ebb6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa480c5960 0x7ffa4806d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.646+0000 7ffa4ebb6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa48153ab0 0x7ffa4806d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.646+0000 7ffa4ebb6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4806f550 con 0x7ffa480c5960 2024-03-06T07:22:37.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.646+0000 7ffa4ebb6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4806f3d0 con 0x7ffa480c4ec0 2024-03-06T07:22:37.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.646+0000 7ffa4ebb6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa48071210 con 0x7ffa48153ab0 2024-03-06T07:22:37.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.647+0000 7ffa47fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa480c5960 0x7ffa4806d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.647+0000 7ffa47fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa480c5960 0x7ffa4806d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57172/0 (socket says 172.21.15.1:57172) 2024-03-06T07:22:37.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.647+0000 7ffa47fff640 1 -- 172.21.15.1:0/1788606624 learned_addr learned my addr 172.21.15.1:0/1788606624 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:37.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.647+0000 7ffa4c952640 1 --2- 172.21.15.1:0/1788606624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa480c4ec0 0x7ffa480c1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.647+0000 7ffa4d153640 1 --2- 172.21.15.1:0/1788606624 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa48153ab0 0x7ffa4806d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.647+0000 7ffa47fff640 1 -- 172.21.15.1:0/1788606624 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa48153ab0 msgr2=0x7ffa4806d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.647+0000 7ffa47fff640 1 --2- 172.21.15.1:0/1788606624 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa48153ab0 0x7ffa4806d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.647+0000 7ffa47fff640 1 -- 172.21.15.1:0/1788606624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa480c4ec0 msgr2=0x7ffa480c1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.647+0000 7ffa47fff640 1 --2- 172.21.15.1:0/1788606624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa480c4ec0 0x7ffa480c1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.647+0000 7ffa47fff640 1 -- 172.21.15.1:0/1788606624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4806c280 con 0x7ffa480c5960 2024-03-06T07:22:37.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.647+0000 7ffa47fff640 1 --2- 172.21.15.1:0/1788606624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa480c5960 0x7ffa4806d020 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00c440 tx=0x7ffa3c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.647+0000 7ffa457fa640 1 -- 172.21.15.1:0/1788606624 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c017070 con 0x7ffa480c5960 2024-03-06T07:22:37.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.647+0000 7ffa457fa640 1 -- 172.21.15.1:0/1788606624 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3c0035f0 con 0x7ffa480c5960 2024-03-06T07:22:37.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.647+0000 7ffa4ebb6640 1 -- 172.21.15.1:0/1788606624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4806c540 con 0x7ffa480c5960 2024-03-06T07:22:37.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.647+0000 7ffa457fa640 1 -- 172.21.15.1:0/1788606624 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c0039e0 con 0x7ffa480c5960 2024-03-06T07:22:37.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.647+0000 7ffa4ebb6640 1 -- 172.21.15.1:0/1788606624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4806cb00 con 0x7ffa480c5960 2024-03-06T07:22:37.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.648+0000 7ffa4ebb6640 1 -- 172.21.15.1:0/1788606624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa48067fd0 con 0x7ffa480c5960 2024-03-06T07:22:37.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.649+0000 7ffa457fa640 1 -- 172.21.15.1:0/1788606624 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa3c01a070 con 0x7ffa480c5960 2024-03-06T07:22:37.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.649+0000 7ffa457fa640 1 --2- 172.21.15.1:0/1788606624 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa3406b4d0 0x7ffa3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.649+0000 7ffa457fa640 1 -- 172.21.15.1:0/1788606624 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7ffa3c089a80 con 0x7ffa480c5960 2024-03-06T07:22:37.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.650+0000 7ffa4c952640 1 --2- 172.21.15.1:0/1788606624 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa3406b4d0 0x7ffa3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.650+0000 7ffa4c952640 1 --2- 172.21.15.1:0/1788606624 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa3406b4d0 0x7ffa3406d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ffa38006fd0 tx=0x7ffa38008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.654+0000 7ffa457fa640 1 -- 172.21.15.1:0/1788606624 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa3c058400 con 0x7ffa480c5960 2024-03-06T07:22:37.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.953+0000 7ffa4ebb6640 1 -- 172.21.15.1:0/1788606624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ffa4806ccd0 con 0x7ffa480c5960 2024-03-06T07:22:37.956 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:22:37.954+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:22:37.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.955+0000 7ffa457fa640 1 -- 172.21.15.1:0/1788606624 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) v1 ==== 161+0+0 (secure 0 0 0) 0x7ffa3c0010c0 con 0x7ffa480c5960 2024-03-06T07:22:37.957 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:22:37.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.958+0000 7ffa4ebb6640 1 -- 172.21.15.1:0/1788606624 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa3406b4d0 msgr2=0x7ffa3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.958+0000 7ffa4ebb6640 1 --2- 172.21.15.1:0/1788606624 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa3406b4d0 0x7ffa3406d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ffa38006fd0 tx=0x7ffa38008040 comp rx=0 tx=0).stop 2024-03-06T07:22:37.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.958+0000 7ffa4ebb6640 1 -- 172.21.15.1:0/1788606624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa480c5960 msgr2=0x7ffa4806d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.958+0000 7ffa4ebb6640 1 --2- 172.21.15.1:0/1788606624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa480c5960 0x7ffa4806d020 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00c440 tx=0x7ffa3c00c900 comp rx=0 tx=0).stop 2024-03-06T07:22:37.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.959+0000 7ffa4ebb6640 1 -- 172.21.15.1:0/1788606624 shutdown_connections 2024-03-06T07:22:37.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.959+0000 7ffa4ebb6640 1 --2- 172.21.15.1:0/1788606624 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa3406b4d0 0x7ffa3406d990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.959+0000 7ffa4ebb6640 1 --2- 172.21.15.1:0/1788606624 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa48153ab0 0x7ffa4806d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.959+0000 7ffa4ebb6640 1 --2- 172.21.15.1:0/1788606624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa480c5960 0x7ffa4806d020 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.959+0000 7ffa4ebb6640 1 --2- 172.21.15.1:0/1788606624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa480c4ec0 0x7ffa480c1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.959+0000 7ffa4ebb6640 1 -- 172.21.15.1:0/1788606624 >> 172.21.15.1:0/1788606624 conn(0x7ffa480aa550 msgr2=0x7ffa480ab640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.959+0000 7ffa4ebb6640 1 -- 172.21.15.1:0/1788606624 shutdown_connections 2024-03-06T07:22:37.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:37.959+0000 7ffa4ebb6640 1 -- 172.21.15.1:0/1788606624 wait complete. 2024-03-06T07:22:37.962 INFO:tasks.ceph.mds.c.smithi001.stderr: -168> 2024-03-06T07:22:32.751+0000 7fd085d40640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:37.972 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:22:37.991 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:22:38.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.073+0000 7f3d0fd77640 1 Processor -- start 2024-03-06T07:22:38.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.074+0000 7f3d0fd77640 1 -- start start 2024-03-06T07:22:38.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.074+0000 7f3d0fd77640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3d080afc40 0x7f3d08160b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.074+0000 7f3d0fd77640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d080c7180 con 0x7f3d080af6f0 2024-03-06T07:22:38.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.074+0000 7f3d0fd77640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d08155460 con 0x7f3d08152820 2024-03-06T07:22:38.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.074+0000 7f3d0fd77640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d080c6bc0 con 0x7f3d080afc40 2024-03-06T07:22:38.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.075+0000 7f3d0e314640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3d080afc40 0x7f3d08160b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.075+0000 7f3d0d312640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3d080af6f0 0x7f3d080c73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37850/0 (socket says 172.21.15.1:37850) 2024-03-06T07:22:38.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.075+0000 7f3d0d312640 1 -- 172.21.15.1:0/482155341 learned_addr learned my addr 172.21.15.1:0/482155341 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:38.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.075+0000 7f3d0e314640 1 -- 172.21.15.1:0/482155341 >> v1:172.21.15.98:6789/0 conn(0x7f3d08152820 legacy=0x7f3d080c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:38.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.075+0000 7f3d0e314640 1 -- 172.21.15.1:0/482155341 >> v1:172.21.15.1:6789/0 conn(0x7f3d080af6f0 legacy=0x7f3d080c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.075+0000 7f3d0cb11640 1 -- 172.21.15.1:0/482155341 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306115430 0 0) 0x7f3d080c7180 con 0x7f3d080af6f0 2024-03-06T07:22:38.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.075+0000 7f3d0e314640 1 -- 172.21.15.1:0/482155341 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d080c6ea0 con 0x7f3d080afc40 2024-03-06T07:22:38.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.076+0000 7f3d0e314640 1 --2- 172.21.15.1:0/482155341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d080afc40 0x7f3d08160b70 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f3d00009d90 tx=0x7f3d0002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248c400b5550ccf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.076+0000 7f3d0cb11640 1 -- 172.21.15.1:0/482155341 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00039070 con 0x7f3d080afc40 2024-03-06T07:22:38.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.076+0000 7f3d0cb11640 1 -- 172.21.15.1:0/482155341 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0002cc60 con 0x7f3d080afc40 2024-03-06T07:22:38.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.076+0000 7f3d0cb11640 1 -- 172.21.15.1:0/482155341 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00034660 con 0x7f3d080afc40 2024-03-06T07:22:38.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.076+0000 7f3d0fd77640 1 -- 172.21.15.1:0/482155341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d080afc40 msgr2=0x7f3d08160b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.076+0000 7f3d0fd77640 1 --2- 172.21.15.1:0/482155341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d080afc40 0x7f3d08160b70 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f3d00009d90 tx=0x7f3d0002beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.076+0000 7f3d0fd77640 1 -- 172.21.15.1:0/482155341 shutdown_connections 2024-03-06T07:22:38.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.076+0000 7f3d0fd77640 1 --2- 172.21.15.1:0/482155341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d080afc40 0x7f3d08160b70 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.076+0000 7f3d0fd77640 1 -- 172.21.15.1:0/482155341 >> 172.21.15.1:0/482155341 conn(0x7f3d08072720 msgr2=0x7f3d08072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.077+0000 7f3d0fd77640 1 -- 172.21.15.1:0/482155341 shutdown_connections 2024-03-06T07:22:38.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.077+0000 7f3d0fd77640 1 -- 172.21.15.1:0/482155341 wait complete. 2024-03-06T07:22:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.078+0000 7f3d0fd77640 1 Processor -- start 2024-03-06T07:22:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.078+0000 7f3d0fd77640 1 -- start start 2024-03-06T07:22:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.078+0000 7f3d0fd77640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d080af6f0 0x7f3d08157d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.078+0000 7f3d0fd77640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d080afc40 0x7f3d080c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.078+0000 7f3d0fd77640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d08152820 0x7f3d0814b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.078+0000 7f3d0fd77640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d080c7b00 con 0x7f3d080afc40 2024-03-06T07:22:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.078+0000 7f3d0fd77640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d080c0840 con 0x7f3d080af6f0 2024-03-06T07:22:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.078+0000 7f3d0fd77640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d080c7c80 con 0x7f3d08152820 2024-03-06T07:22:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.078+0000 7f3d0d312640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d080afc40 0x7f3d080c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.078+0000 7f3d0d312640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d080afc40 0x7f3d080c9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57192/0 (socket says 172.21.15.1:57192) 2024-03-06T07:22:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.078+0000 7f3d0d312640 1 -- 172.21.15.1:0/13603247 learned_addr learned my addr 172.21.15.1:0/13603247 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.079+0000 7f3d0d312640 1 -- 172.21.15.1:0/13603247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d08152820 msgr2=0x7f3d0814b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.079+0000 7f3d0db13640 1 --2- 172.21.15.1:0/13603247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d080af6f0 0x7f3d08157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.079+0000 7f3d0e314640 1 --2- 172.21.15.1:0/13603247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d08152820 0x7f3d0814b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.079+0000 7f3d0d312640 1 --2- 172.21.15.1:0/13603247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d08152820 0x7f3d0814b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.079+0000 7f3d0d312640 1 -- 172.21.15.1:0/13603247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d080af6f0 msgr2=0x7f3d08157d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.079+0000 7f3d0d312640 1 --2- 172.21.15.1:0/13603247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d080af6f0 0x7f3d08157d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.079+0000 7f3d0d312640 1 -- 172.21.15.1:0/13603247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d081eac60 con 0x7f3d080afc40 2024-03-06T07:22:38.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.079+0000 7f3d0d312640 1 --2- 172.21.15.1:0/13603247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d080afc40 0x7f3d080c9d30 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3cf800c4d0 tx=0x7f3cf800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.079+0000 7f3cfe7fc640 1 -- 172.21.15.1:0/13603247 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8018070 con 0x7f3d080afc40 2024-03-06T07:22:38.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.079+0000 7f3cfe7fc640 1 -- 172.21.15.1:0/13603247 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf8007500 con 0x7f3d080afc40 2024-03-06T07:22:38.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.079+0000 7f3cfe7fc640 1 -- 172.21.15.1:0/13603247 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8004a20 con 0x7f3d080afc40 2024-03-06T07:22:38.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.079+0000 7f3d0fd77640 1 -- 172.21.15.1:0/13603247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d081eaf40 con 0x7f3d080afc40 2024-03-06T07:22:38.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.079+0000 7f3d0fd77640 1 -- 172.21.15.1:0/13603247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d081eb500 con 0x7f3d080afc40 2024-03-06T07:22:38.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.080+0000 7f3d0fd77640 1 -- 172.21.15.1:0/13603247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d080686e0 con 0x7f3d080afc40 2024-03-06T07:22:38.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.080+0000 7f3cfe7fc640 1 -- 172.21.15.1:0/13603247 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3cf801b070 con 0x7f3d080afc40 2024-03-06T07:22:38.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.081+0000 7f3cfe7fc640 1 --2- 172.21.15.1:0/13603247 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cec06b4d0 0x7f3cec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.081+0000 7f3cfe7fc640 1 -- 172.21.15.1:0/13603247 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7215+0+0 (secure 0 0 0) 0x7f3cf8089a60 con 0x7f3d080afc40 2024-03-06T07:22:38.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.081+0000 7f3d0db13640 1 --2- 172.21.15.1:0/13603247 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cec06b4d0 0x7f3cec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.082+0000 7f3d0db13640 1 --2- 172.21.15.1:0/13603247 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cec06b4d0 0x7f3cec06d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3d04006fd0 tx=0x7f3d04008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.085+0000 7f3cfe7fc640 1 -- 172.21.15.1:0/13603247 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3cf8058390 con 0x7f3d080afc40 2024-03-06T07:22:38.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.303+0000 7f3d0fd77640 1 -- 172.21.15.1:0/13603247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3d081eb6d0 con 0x7f3d080afc40 2024-03-06T07:22:38.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.956+0000 7f3cfe7fc640 1 -- 172.21.15.1:0/13603247 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3cf8057c90 con 0x7f3d080afc40 2024-03-06T07:22:38.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.959+0000 7f3d0fd77640 1 -- 172.21.15.1:0/13603247 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cec06b4d0 msgr2=0x7f3cec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.959+0000 7f3d0fd77640 1 --2- 172.21.15.1:0/13603247 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cec06b4d0 0x7f3cec06d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3d04006fd0 tx=0x7f3d04008040 comp rx=0 tx=0).stop 2024-03-06T07:22:38.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.959+0000 7f3d0fd77640 1 -- 172.21.15.1:0/13603247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d080afc40 msgr2=0x7f3d080c9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.959+0000 7f3d0fd77640 1 --2- 172.21.15.1:0/13603247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d080afc40 0x7f3d080c9d30 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3cf800c4d0 tx=0x7f3cf800c990 comp rx=0 tx=0).stop 2024-03-06T07:22:38.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.959+0000 7f3d0fd77640 1 -- 172.21.15.1:0/13603247 shutdown_connections 2024-03-06T07:22:38.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.959+0000 7f3d0fd77640 1 --2- 172.21.15.1:0/13603247 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cec06b4d0 0x7f3cec06d990 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.959+0000 7f3d0fd77640 1 --2- 172.21.15.1:0/13603247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d08152820 0x7f3d0814b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.959+0000 7f3d0fd77640 1 --2- 172.21.15.1:0/13603247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d080afc40 0x7f3d080c9d30 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.959+0000 7f3d0fd77640 1 --2- 172.21.15.1:0/13603247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d080af6f0 0x7f3d08157d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.959+0000 7f3d0fd77640 1 -- 172.21.15.1:0/13603247 >> 172.21.15.1:0/13603247 conn(0x7f3d08072720 msgr2=0x7f3d0814f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.959+0000 7f3d0fd77640 1 -- 172.21.15.1:0/13603247 shutdown_connections 2024-03-06T07:22:38.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:38.959+0000 7f3d0fd77640 1 -- 172.21.15.1:0/13603247 wait complete. 2024-03-06T07:22:38.967 INFO:tasks.ceph.mds.a.smithi001.stderr: -434> 2024-03-06T07:22:26.592+0000 7f4939bcd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:38.975 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:22:38.993 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:22:39.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.096+0000 7effe76eb640 1 Processor -- start 2024-03-06T07:22:39.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.096+0000 7effe76eb640 1 -- start start 2024-03-06T07:22:39.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.097+0000 7effe76eb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7effe0152780 0x7effe00c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.097+0000 7effe76eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7effe0155e50 0x7effe00cab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.097+0000 7effe76eb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe00c69b0 con 0x7effe00c1010 2024-03-06T07:22:39.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.097+0000 7effe76eb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effe00c6c90 con 0x7effe0152780 2024-03-06T07:22:39.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.097+0000 7effe76eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe0155370 con 0x7effe0155e50 2024-03-06T07:22:39.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.097+0000 7effe5487640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7effe0152780 0x7effe00c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.097+0000 7effe5c88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7effe0155e50 0x7effe00cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.097+0000 7effe5487640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7effe0152780 0x7effe00c6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:33230/0 (socket says 172.21.15.1:33230) 2024-03-06T07:22:39.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.097+0000 7effe4c86640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7effe00c1010 0x7effe00c6fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:56218/0 (socket says 172.21.15.1:56218) 2024-03-06T07:22:39.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.097+0000 7effe5c88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7effe0155e50 0x7effe00cab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:35924/0 (socket says 172.21.15.1:35924) 2024-03-06T07:22:39.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.097+0000 7effe5487640 1 -- 172.21.15.1:0/3536619975 learned_addr learned my addr 172.21.15.1:0/3536619975 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:39.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.098+0000 7effe5487640 1 -- 172.21.15.1:0/3536619975 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7effe0155e50 msgr2=0x7effe00cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:39.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.098+0000 7effe5487640 1 --2- 172.21.15.1:0/3536619975 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7effe0155e50 0x7effe00cab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.098+0000 7effe5487640 1 -- 172.21.15.1:0/3536619975 >> v1:172.21.15.98:6789/0 conn(0x7effe00c1010 legacy=0x7effe00c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.098+0000 7effe5487640 1 -- 172.21.15.1:0/3536619975 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe0155650 con 0x7effe0152780 2024-03-06T07:22:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.098+0000 7effe5487640 1 --2- 172.21.15.1:0/3536619975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effe0152780 0x7effe00c6000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7effd0009870 tx=0x7effd002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e68e310068312a46 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.099+0000 7effd7fff640 1 -- 172.21.15.1:0/3536619975 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd0035020 con 0x7effe0152780 2024-03-06T07:22:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.099+0000 7effd7fff640 1 -- 172.21.15.1:0/3536619975 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd0031640 con 0x7effe0152780 2024-03-06T07:22:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.099+0000 7effd7fff640 1 -- 172.21.15.1:0/3536619975 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd002d040 con 0x7effe0152780 2024-03-06T07:22:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.099+0000 7effe76eb640 1 -- 172.21.15.1:0/3536619975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effe0152780 msgr2=0x7effe00c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.099+0000 7effe76eb640 1 --2- 172.21.15.1:0/3536619975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effe0152780 0x7effe00c6000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7effd0009870 tx=0x7effd002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.099+0000 7effe76eb640 1 -- 172.21.15.1:0/3536619975 shutdown_connections 2024-03-06T07:22:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.099+0000 7effe76eb640 1 --2- 172.21.15.1:0/3536619975 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7effe0155e50 0x7effe00cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.099+0000 7effe76eb640 1 --2- 172.21.15.1:0/3536619975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effe0152780 0x7effe00c6000 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.099+0000 7effe76eb640 1 -- 172.21.15.1:0/3536619975 >> 172.21.15.1:0/3536619975 conn(0x7effe008b720 msgr2=0x7effe006d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.099+0000 7effe76eb640 1 -- 172.21.15.1:0/3536619975 shutdown_connections 2024-03-06T07:22:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.099+0000 7effe76eb640 1 -- 172.21.15.1:0/3536619975 wait complete. 2024-03-06T07:22:39.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.100+0000 7effe76eb640 1 Processor -- start 2024-03-06T07:22:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.100+0000 7effe76eb640 1 -- start start 2024-03-06T07:22:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.101+0000 7effe76eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effe00c1010 0x7effe00c3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.101+0000 7effe76eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7effe0152780 0x7effe00c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.101+0000 7effe76eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effe0155e50 0x7effe0147710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.101+0000 7effe76eb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe0152b60 con 0x7effe0155e50 2024-03-06T07:22:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.101+0000 7effe76eb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe00c76d0 con 0x7effe00c1010 2024-03-06T07:22:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.101+0000 7effe76eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe00c6540 con 0x7effe0152780 2024-03-06T07:22:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.101+0000 7effe5c88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effe0155e50 0x7effe0147710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.101+0000 7effe5c88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effe0155e50 0x7effe0147710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57222/0 (socket says 172.21.15.1:57222) 2024-03-06T07:22:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.101+0000 7effe5c88640 1 -- 172.21.15.1:0/1340032999 learned_addr learned my addr 172.21.15.1:0/1340032999 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.101+0000 7effe5487640 1 --2- 172.21.15.1:0/1340032999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effe00c1010 0x7effe00c3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.101+0000 7effe4c86640 1 --2- 172.21.15.1:0/1340032999 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7effe0152780 0x7effe00c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.101+0000 7effe5c88640 1 -- 172.21.15.1:0/1340032999 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7effe0152780 msgr2=0x7effe00c99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.101+0000 7effe5c88640 1 --2- 172.21.15.1:0/1340032999 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7effe0152780 0x7effe00c99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.101+0000 7effe5c88640 1 -- 172.21.15.1:0/1340032999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effe00c1010 msgr2=0x7effe00c3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.101+0000 7effe5c88640 1 --2- 172.21.15.1:0/1340032999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effe00c1010 0x7effe00c3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.101+0000 7effe5c88640 1 -- 172.21.15.1:0/1340032999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe01f21a0 con 0x7effe0155e50 2024-03-06T07:22:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.101+0000 7effe5487640 1 --2- 172.21.15.1:0/1340032999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effe00c1010 0x7effe00c3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.102+0000 7effe5c88640 1 --2- 172.21.15.1:0/1340032999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effe0155e50 0x7effe0147710 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7effd8009d30 tx=0x7effd800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.102+0000 7effd5ffb640 1 -- 172.21.15.1:0/1340032999 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd8018020 con 0x7effe0155e50 2024-03-06T07:22:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.102+0000 7effd5ffb640 1 -- 172.21.15.1:0/1340032999 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd8014480 con 0x7effe0155e50 2024-03-06T07:22:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.102+0000 7effe76eb640 1 -- 172.21.15.1:0/1340032999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe01f2440 con 0x7effe0155e50 2024-03-06T07:22:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.102+0000 7effd5ffb640 1 -- 172.21.15.1:0/1340032999 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd8014870 con 0x7effe0155e50 2024-03-06T07:22:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.102+0000 7effe76eb640 1 -- 172.21.15.1:0/1340032999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe014b3e0 con 0x7effe0155e50 2024-03-06T07:22:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.103+0000 7effe76eb640 1 -- 172.21.15.1:0/1340032999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe00680b0 con 0x7effe0155e50 2024-03-06T07:22:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.103+0000 7effd5ffb640 1 -- 172.21.15.1:0/1340032999 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7effd8005d10 con 0x7effe0155e50 2024-03-06T07:22:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.104+0000 7effd5ffb640 1 --2- 172.21.15.1:0/1340032999 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7effc406b4d0 0x7effc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.104+0000 7effd5ffb640 1 -- 172.21.15.1:0/1340032999 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7301+0+0 (secure 0 0 0) 0x7effd808b390 con 0x7effe0155e50 2024-03-06T07:22:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.104+0000 7effe5487640 1 --2- 172.21.15.1:0/1340032999 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7effc406b4d0 0x7effc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.105+0000 7effe5487640 1 --2- 172.21.15.1:0/1340032999 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7effc406b4d0 0x7effc406d990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7effd0036000 tx=0x7effd003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.108+0000 7effd5ffb640 1 -- 172.21.15.1:0/1340032999 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7effd8022070 con 0x7effe0155e50 2024-03-06T07:22:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:39.289+0000 7effe76eb640 1 -- 172.21.15.1:0/1340032999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7effe00c5de0 con 0x7effe0155e50 2024-03-06T07:22:40.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.281+0000 7effd5ffb640 1 -- 172.21.15.1:0/1340032999 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v61) v1 ==== 181+0+0 (secure 0 0 0) 0x7effd8010040 con 0x7effe0155e50 2024-03-06T07:22:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:22:40.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.290+0000 7effe76eb640 1 -- 172.21.15.1:0/1340032999 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7effc406b4d0 msgr2=0x7effc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.290+0000 7effe76eb640 1 --2- 172.21.15.1:0/1340032999 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7effc406b4d0 0x7effc406d990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7effd0036000 tx=0x7effd003c040 comp rx=0 tx=0).stop 2024-03-06T07:22:40.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.290+0000 7effe76eb640 1 -- 172.21.15.1:0/1340032999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effe0155e50 msgr2=0x7effe0147710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.290+0000 7effe76eb640 1 --2- 172.21.15.1:0/1340032999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effe0155e50 0x7effe0147710 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7effd8009d30 tx=0x7effd800c490 comp rx=0 tx=0).stop 2024-03-06T07:22:40.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.292+0000 7effe76eb640 1 -- 172.21.15.1:0/1340032999 shutdown_connections 2024-03-06T07:22:40.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.292+0000 7effe76eb640 1 --2- 172.21.15.1:0/1340032999 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7effc406b4d0 0x7effc406d990 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.292+0000 7effe76eb640 1 --2- 172.21.15.1:0/1340032999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effe0155e50 0x7effe0147710 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.292+0000 7effe76eb640 1 --2- 172.21.15.1:0/1340032999 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7effe0152780 0x7effe00c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.292+0000 7effe76eb640 1 --2- 172.21.15.1:0/1340032999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effe00c1010 0x7effe00c3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.292+0000 7effe76eb640 1 -- 172.21.15.1:0/1340032999 >> 172.21.15.1:0/1340032999 conn(0x7effe008b720 msgr2=0x7effe00c78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.293+0000 7effe76eb640 1 -- 172.21.15.1:0/1340032999 shutdown_connections 2024-03-06T07:22:40.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.293+0000 7effe76eb640 1 -- 172.21.15.1:0/1340032999 wait complete. 2024-03-06T07:22:40.309 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:22:40.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.418+0000 7f5578af4640 1 Processor -- start 2024-03-06T07:22:40.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.418+0000 7f5578af4640 1 -- start start 2024-03-06T07:22:40.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.419+0000 7f5578af4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5574158180 0x7f55740c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.419+0000 7f5578af4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f55740c5e20 0x7f5574156a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.419+0000 7f5578af4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5574158560 con 0x7f55740c6800 2024-03-06T07:22:40.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.419+0000 7f5578af4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5574156210 con 0x7f55740c5e20 2024-03-06T07:22:40.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.419+0000 7f5578af4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55740c3ea0 con 0x7f5574158180 2024-03-06T07:22:40.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.419+0000 7f5571d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f55740c5e20 0x7f5574156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.419+0000 7f557259c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5574158180 0x7f55740c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.420+0000 7f5572d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f55740c6800 0x7f5574156f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:56234/0 (socket says 172.21.15.1:56234) 2024-03-06T07:22:40.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.420+0000 7f5572d9d640 1 -- 172.21.15.1:0/2536133196 learned_addr learned my addr 172.21.15.1:0/2536133196 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:40.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.420+0000 7f557159a640 1 -- 172.21.15.1:0/2536133196 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724103082 0 0) 0x7f5574158560 con 0x7f55740c6800 2024-03-06T07:22:40.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.420+0000 7f557159a640 1 -- 172.21.15.1:0/2536133196 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5560003520 con 0x7f55740c6800 2024-03-06T07:22:40.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.420+0000 7f557259c640 1 -- 172.21.15.1:0/2536133196 >> v2:172.21.15.98:3300/0 conn(0x7f55740c5e20 msgr2=0x7f5574156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.420+0000 7f557259c640 1 --2- 172.21.15.1:0/2536133196 >> v2:172.21.15.98:3300/0 conn(0x7f55740c5e20 0x7f5574156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.420+0000 7f557259c640 1 -- 172.21.15.1:0/2536133196 >> v1:172.21.15.98:6789/0 conn(0x7f55740c6800 legacy=0x7f5574156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.420+0000 7f557259c640 1 -- 172.21.15.1:0/2536133196 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5574155600 con 0x7f5574158180 2024-03-06T07:22:40.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.420+0000 7f5571d9b640 1 --2- 172.21.15.1:0/2536133196 >> v2:172.21.15.98:3300/0 conn(0x7f55740c5e20 0x7f5574156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:40.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.421+0000 7f557259c640 1 --2- 172.21.15.1:0/2536133196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5574158180 0x7f55740c58e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f556c0097a0 tx=0x7f556c02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c55f16bf3623ab27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.421+0000 7f557159a640 1 -- 172.21.15.1:0/2536133196 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556c038070 con 0x7f5574158180 2024-03-06T07:22:40.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.421+0000 7f557159a640 1 -- 172.21.15.1:0/2536133196 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556c031700 con 0x7f5574158180 2024-03-06T07:22:40.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.421+0000 7f557159a640 1 -- 172.21.15.1:0/2536133196 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556c031ab0 con 0x7f5574158180 2024-03-06T07:22:40.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.421+0000 7f5578af4640 1 -- 172.21.15.1:0/2536133196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5574158180 msgr2=0x7f55740c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.421+0000 7f5578af4640 1 --2- 172.21.15.1:0/2536133196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5574158180 0x7f55740c58e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f556c0097a0 tx=0x7f556c02b7a0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.422+0000 7f5578af4640 1 -- 172.21.15.1:0/2536133196 shutdown_connections 2024-03-06T07:22:40.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.422+0000 7f5578af4640 1 --2- 172.21.15.1:0/2536133196 >> v2:172.21.15.98:3300/0 conn(0x7f55740c5e20 0x7f5574156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.422+0000 7f5578af4640 1 --2- 172.21.15.1:0/2536133196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5574158180 0x7f55740c58e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.422+0000 7f5578af4640 1 -- 172.21.15.1:0/2536133196 >> 172.21.15.1:0/2536133196 conn(0x7f557406b410 msgr2=0x7f557406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.422+0000 7f5578af4640 1 -- 172.21.15.1:0/2536133196 shutdown_connections 2024-03-06T07:22:40.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.422+0000 7f5578af4640 1 -- 172.21.15.1:0/2536133196 wait complete. 2024-03-06T07:22:40.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.423+0000 7f5578af4640 1 Processor -- start 2024-03-06T07:22:40.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.423+0000 7f5578af4640 1 -- start start 2024-03-06T07:22:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.423+0000 7f5578af4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55740c5e20 0x7f55741495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.423+0000 7f5578af4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55740c6800 0x7f557414cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.423+0000 7f5578af4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5574158180 0x7f55741edcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.423+0000 7f5578af4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5574155ea0 con 0x7f55740c6800 2024-03-06T07:22:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.423+0000 7f5578af4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5574155bc0 con 0x7f5574158180 2024-03-06T07:22:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.423+0000 7f5578af4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55741576a0 con 0x7f55740c5e20 2024-03-06T07:22:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.424+0000 7f5571d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55740c6800 0x7f557414cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.424+0000 7f5571d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55740c6800 0x7f557414cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57226/0 (socket says 172.21.15.1:57226) 2024-03-06T07:22:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.424+0000 7f5571d9b640 1 -- 172.21.15.1:0/390603030 learned_addr learned my addr 172.21.15.1:0/390603030 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.424+0000 7f557259c640 1 --2- 172.21.15.1:0/390603030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55740c5e20 0x7f55741495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.424+0000 7f5572d9d640 1 --2- 172.21.15.1:0/390603030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5574158180 0x7f55741edcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.424+0000 7f5571d9b640 1 -- 172.21.15.1:0/390603030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55740c5e20 msgr2=0x7f55741495d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.424+0000 7f5571d9b640 1 --2- 172.21.15.1:0/390603030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55740c5e20 0x7f55741495d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.424+0000 7f5571d9b640 1 -- 172.21.15.1:0/390603030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5574158180 msgr2=0x7f55741edcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.424+0000 7f5571d9b640 1 --2- 172.21.15.1:0/390603030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5574158180 0x7f55741edcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.424+0000 7f5571d9b640 1 -- 172.21.15.1:0/390603030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55740c8500 con 0x7f55740c6800 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.424+0000 7f5571d9b640 1 --2- 172.21.15.1:0/390603030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55740c6800 0x7f557414cb70 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f555c00e990 tx=0x7f555c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.424+0000 7f556affd640 1 -- 172.21.15.1:0/390603030 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c017070 con 0x7f55740c6800 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.424+0000 7f556affd640 1 -- 172.21.15.1:0/390603030 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555c00c980 con 0x7f55740c6800 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.424+0000 7f556affd640 1 -- 172.21.15.1:0/390603030 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c00cd50 con 0x7f55740c6800 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.424+0000 7f5578af4640 1 -- 172.21.15.1:0/390603030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55740c87e0 con 0x7f55740c6800 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.424+0000 7f5578af4640 1 -- 172.21.15.1:0/390603030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5574148dd0 con 0x7f55740c6800 2024-03-06T07:22:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.425+0000 7f5578af4640 1 -- 172.21.15.1:0/390603030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55740680b0 con 0x7f55740c6800 2024-03-06T07:22:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.426+0000 7f556affd640 1 -- 172.21.15.1:0/390603030 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f555c01a070 con 0x7f55740c6800 2024-03-06T07:22:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.426+0000 7f556affd640 1 --2- 172.21.15.1:0/390603030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f555406b500 0x7f555406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.426+0000 7f556affd640 1 -- 172.21.15.1:0/390603030 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6863+0+0 (secure 0 0 0) 0x7f555c08a700 con 0x7f55740c6800 2024-03-06T07:22:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.426+0000 7f557259c640 1 --2- 172.21.15.1:0/390603030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f555406b500 0x7f555406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.427+0000 7f557259c640 1 --2- 172.21.15.1:0/390603030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f555406b500 0x7f555406d9c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f556c0097a0 tx=0x7f556c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.431+0000 7f556affd640 1 -- 172.21.15.1:0/390603030 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f555c058b40 con 0x7f55740c6800 2024-03-06T07:22:40.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:40.614+0000 7f5578af4640 1 -- 172.21.15.1:0/390603030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f55740c2df0 con 0x7f55740c6800 2024-03-06T07:22:41.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.287+0000 7f556affd640 1 -- 172.21.15.1:0/390603030 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v62) v1 ==== 169+0+0 (secure 0 0 0) 0x7f555c05cbe0 con 0x7f55740c6800 2024-03-06T07:22:41.292 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-06T07:22:41.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.291+0000 7f5578af4640 1 -- 172.21.15.1:0/390603030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f555406b500 msgr2=0x7f555406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.291+0000 7f5578af4640 1 --2- 172.21.15.1:0/390603030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f555406b500 0x7f555406d9c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f556c0097a0 tx=0x7f556c008030 comp rx=0 tx=0).stop 2024-03-06T07:22:41.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.291+0000 7f5578af4640 1 -- 172.21.15.1:0/390603030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55740c6800 msgr2=0x7f557414cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.291+0000 7f5578af4640 1 --2- 172.21.15.1:0/390603030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55740c6800 0x7f557414cb70 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f555c00e990 tx=0x7f555c00ee50 comp rx=0 tx=0).stop 2024-03-06T07:22:41.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.291+0000 7f5578af4640 1 -- 172.21.15.1:0/390603030 shutdown_connections 2024-03-06T07:22:41.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.291+0000 7f5578af4640 1 --2- 172.21.15.1:0/390603030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f555406b500 0x7f555406d9c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.291+0000 7f5578af4640 1 --2- 172.21.15.1:0/390603030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5574158180 0x7f55741edcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.291+0000 7f5578af4640 1 --2- 172.21.15.1:0/390603030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55740c6800 0x7f557414cb70 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.291+0000 7f5578af4640 1 --2- 172.21.15.1:0/390603030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55740c5e20 0x7f55741495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.291+0000 7f5578af4640 1 -- 172.21.15.1:0/390603030 >> 172.21.15.1:0/390603030 conn(0x7f557406b410 msgr2=0x7f55740c3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.291+0000 7f5578af4640 1 -- 172.21.15.1:0/390603030 shutdown_connections 2024-03-06T07:22:41.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.291+0000 7f5578af4640 1 -- 172.21.15.1:0/390603030 wait complete. 2024-03-06T07:22:41.308 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:41.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.419+0000 7f47d77bc640 1 Processor -- start 2024-03-06T07:22:41.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.420+0000 7f47d77bc640 1 -- start start 2024-03-06T07:22:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.420+0000 7f47d77bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d00c1390 0x7f47d00c17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.420+0000 7f47d77bc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d01588e0 con 0x7f47d0152730 2024-03-06T07:22:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.420+0000 7f47d77bc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d0158bc0 con 0x7f47d01552e0 2024-03-06T07:22:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.420+0000 7f47d77bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d00c1d30 con 0x7f47d00c1390 2024-03-06T07:22:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47d5558640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f47d0152730 0x7f47d00c6a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37866/0 (socket says 172.21.15.1:37866) 2024-03-06T07:22:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47d5558640 1 -- 172.21.15.1:0/2898913016 learned_addr learned my addr 172.21.15.1:0/2898913016 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47d4d57640 1 --2- 172.21.15.1:0/2898913016 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d00c1390 0x7f47d00c17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47c7fff640 1 -- 172.21.15.1:0/2898913016 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691923125 0 0) 0x7f47d01588e0 con 0x7f47d0152730 2024-03-06T07:22:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47c7fff640 1 -- 172.21.15.1:0/2898913016 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47bc003520 con 0x7f47d0152730 2024-03-06T07:22:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47c7fff640 1 -- 172.21.15.1:0/2898913016 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1199893344 0 0) 0x7f47bc003520 con 0x7f47d0152730 2024-03-06T07:22:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47c7fff640 1 -- 172.21.15.1:0/2898913016 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d00c1390 msgr2=0x7f47d00c17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47c7fff640 1 --2- 172.21.15.1:0/2898913016 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d00c1390 0x7f47d00c17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47c7fff640 1 -- 172.21.15.1:0/2898913016 >> v1:172.21.15.98:6789/0 conn(0x7f47d01552e0 legacy=0x7f47d0155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47c7fff640 1 -- 172.21.15.1:0/2898913016 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d0158ea0 con 0x7f47d0152730 2024-03-06T07:22:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47c7fff640 1 -- 172.21.15.1:0/2898913016 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f47cc001690 con 0x7f47d0152730 2024-03-06T07:22:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47c7fff640 1 -- 172.21.15.1:0/2898913016 >> v1:172.21.15.1:6789/0 conn(0x7f47d0152730 legacy=0x7f47d00c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47c7fff640 1 --2- 172.21.15.1:0/2898913016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47bc0039e0 0x7f47bc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47c7fff640 1 --2- 172.21.15.1:0/2898913016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47bc004800 0x7f47bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47c7fff640 1 --2- 172.21.15.1:0/2898913016 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d00c1390 0x7f47bc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47c7fff640 1 -- 172.21.15.1:0/2898913016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d00c1d30 con 0x7f47bc0039e0 2024-03-06T07:22:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47c7fff640 1 -- 172.21.15.1:0/2898913016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc00baf0 con 0x7f47bc004800 2024-03-06T07:22:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47c7fff640 1 -- 172.21.15.1:0/2898913016 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc00bcf0 con 0x7f47d00c1390 2024-03-06T07:22:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.421+0000 7f47d5558640 1 --2- 172.21.15.1:0/2898913016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47bc0039e0 0x7f47bc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.422+0000 7f47d5d59640 1 --2- 172.21.15.1:0/2898913016 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d00c1390 0x7f47bc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.422+0000 7f47d4d57640 1 --2- 172.21.15.1:0/2898913016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47bc004800 0x7f47bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.422+0000 7f47d5558640 1 -- 172.21.15.1:0/2898913016 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d00c1390 msgr2=0x7f47bc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.422+0000 7f47d5558640 1 --2- 172.21.15.1:0/2898913016 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d00c1390 0x7f47bc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.422+0000 7f47d5558640 1 -- 172.21.15.1:0/2898913016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47bc004800 msgr2=0x7f47bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.422+0000 7f47d5558640 1 --2- 172.21.15.1:0/2898913016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47bc004800 0x7f47bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.422+0000 7f47d5558640 1 -- 172.21.15.1:0/2898913016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47bc003520 con 0x7f47bc0039e0 2024-03-06T07:22:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.423+0000 7f47d5558640 1 --2- 172.21.15.1:0/2898913016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47bc0039e0 0x7f47bc003dc0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f47cc00c210 tx=0x7f47bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f991fd15d8cbf4e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.423+0000 7f47c7fff640 1 -- 172.21.15.1:0/2898913016 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc007d60 con 0x7f47bc0039e0 2024-03-06T07:22:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.423+0000 7f47c7fff640 1 -- 172.21.15.1:0/2898913016 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47cc030070 con 0x7f47bc0039e0 2024-03-06T07:22:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.423+0000 7f47c7fff640 1 -- 172.21.15.1:0/2898913016 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc02cd20 con 0x7f47bc0039e0 2024-03-06T07:22:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.423+0000 7f47d77bc640 1 -- 172.21.15.1:0/2898913016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47bc0039e0 msgr2=0x7f47bc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.423+0000 7f47d77bc640 1 --2- 172.21.15.1:0/2898913016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47bc0039e0 0x7f47bc003dc0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f47cc00c210 tx=0x7f47bc005830 comp rx=0 tx=0).stop 2024-03-06T07:22:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.423+0000 7f47d5d59640 1 -- 172.21.15.1:0/2898913016 reap_dead start 2024-03-06T07:22:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.423+0000 7f47d77bc640 1 -- 172.21.15.1:0/2898913016 shutdown_connections 2024-03-06T07:22:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.423+0000 7f47d77bc640 1 -- 172.21.15.1:0/2898913016 >> 172.21.15.1:0/2898913016 conn(0x7f47d0066d00 msgr2=0x7f47d0067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.423+0000 7f47d77bc640 1 -- 172.21.15.1:0/2898913016 shutdown_connections 2024-03-06T07:22:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.423+0000 7f47d77bc640 1 -- 172.21.15.1:0/2898913016 wait complete. 2024-03-06T07:22:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.424+0000 7f47d77bc640 1 Processor -- start 2024-03-06T07:22:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.425+0000 7f47d77bc640 1 -- start start 2024-03-06T07:22:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.425+0000 7f47d77bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47bc0039e0 0x7f47d00c0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.425+0000 7f47d77bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d0158160 0x7f47d014ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.425+0000 7f47d77bc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47d014cf40 0x7f47d014a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.425+0000 7f47d77bc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d0152b10 con 0x7f47d014cf40 2024-03-06T07:22:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.425+0000 7f47d77bc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d0159460 con 0x7f47bc0039e0 2024-03-06T07:22:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.425+0000 7f47d77bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d00c71a0 con 0x7f47d0158160 2024-03-06T07:22:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.425+0000 7f47d5d59640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47d014cf40 0x7f47d014a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.425+0000 7f47d5d59640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47d014cf40 0x7f47d014a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57236/0 (socket says 172.21.15.1:57236) 2024-03-06T07:22:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.425+0000 7f47d5d59640 1 -- 172.21.15.1:0/1736904857 learned_addr learned my addr 172.21.15.1:0/1736904857 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.426+0000 7f47d5558640 1 --2- 172.21.15.1:0/1736904857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47bc0039e0 0x7f47d00c0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.426+0000 7f47d4d57640 1 --2- 172.21.15.1:0/1736904857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d0158160 0x7f47d014ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.426+0000 7f47d5d59640 1 -- 172.21.15.1:0/1736904857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d0158160 msgr2=0x7f47d014ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.426+0000 7f47d5d59640 1 --2- 172.21.15.1:0/1736904857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d0158160 0x7f47d014ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.426+0000 7f47d5d59640 1 -- 172.21.15.1:0/1736904857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47bc0039e0 msgr2=0x7f47d00c0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.426+0000 7f47d5d59640 1 --2- 172.21.15.1:0/1736904857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47bc0039e0 0x7f47d00c0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.426+0000 7f47d5d59640 1 -- 172.21.15.1:0/1736904857 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d00c6f70 con 0x7f47d014cf40 2024-03-06T07:22:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.426+0000 7f47d4d57640 1 --2- 172.21.15.1:0/1736904857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d0158160 0x7f47d014ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.426+0000 7f47d5558640 1 --2- 172.21.15.1:0/1736904857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47bc0039e0 0x7f47d00c0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.426+0000 7f47d5d59640 1 --2- 172.21.15.1:0/1736904857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47d014cf40 0x7f47d014a330 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f47d008c310 tx=0x7f47c800ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.426+0000 7f47c5ffb640 1 -- 172.21.15.1:0/1736904857 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c8003b80 con 0x7f47d014cf40 2024-03-06T07:22:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.426+0000 7f47c5ffb640 1 -- 172.21.15.1:0/1736904857 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c8018070 con 0x7f47d014cf40 2024-03-06T07:22:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.426+0000 7f47d77bc640 1 -- 172.21.15.1:0/1736904857 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d00c9400 con 0x7f47d014cf40 2024-03-06T07:22:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.426+0000 7f47c5ffb640 1 -- 172.21.15.1:0/1736904857 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c80050d0 con 0x7f47d014cf40 2024-03-06T07:22:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.426+0000 7f47d77bc640 1 -- 172.21.15.1:0/1736904857 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d0156700 con 0x7f47d014cf40 2024-03-06T07:22:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.427+0000 7f47d77bc640 1 -- 172.21.15.1:0/1736904857 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d0067fd0 con 0x7f47d014cf40 2024-03-06T07:22:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.427+0000 7f47c5ffb640 1 -- 172.21.15.1:0/1736904857 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f47c8005270 con 0x7f47d014cf40 2024-03-06T07:22:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.428+0000 7f47c5ffb640 1 --2- 172.21.15.1:0/1736904857 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47b406b4d0 0x7f47b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.428+0000 7f47c5ffb640 1 -- 172.21.15.1:0/1736904857 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f47c808a4d0 con 0x7f47d014cf40 2024-03-06T07:22:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.428+0000 7f47d5558640 1 --2- 172.21.15.1:0/1736904857 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47b406b4d0 0x7f47b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.429+0000 7f47d5558640 1 --2- 172.21.15.1:0/1736904857 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47b406b4d0 0x7f47b406d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f47cc034000 tx=0x7f47cc009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.432+0000 7f47c5ffb640 1 -- 172.21.15.1:0/1736904857 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f47c8059060 con 0x7f47d014cf40 2024-03-06T07:22:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.615+0000 7f47d77bc640 1 -- 172.21.15.1:0/1736904857 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47d0156530 con 0x7f47d014cf40 2024-03-06T07:22:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.616+0000 7f47c5ffb640 1 -- 172.21.15.1:0/1736904857 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+17279 (secure 0 0 0) 0x7f47c8058960 con 0x7f47d014cf40 2024-03-06T07:22:41.619 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:41.619 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":62,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:41.282730+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:22:18.907422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:22:20.028174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/3845300501":"2024-03-07T07:22:38.960489+0000","172.21.15.1:6834/4244709430":"2024-03-07T07:22:37.955221+0000","172.21.15.1:6832/4244709430":"2024-03-07T07:22:37.955221+0000","172.21.15.98:6836/2937068915":"2024-03-07T07:22:29.247312+0000","172.21.15.1:6835/3845300501":"2024-03-07T07:22:38.960489+0000","172.21.15.98:6834/2937068915":"2024-03-07T07:22:29.247312+0000","172.21.15.1:6834/131367647":"2024-03-07T07:22:26.740110+0000","172.21.15.1:6832/131367647":"2024-03-07T07:22:26.740110+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.618+0000 7f47d77bc640 1 -- 172.21.15.1:0/1736904857 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47b406b4d0 msgr2=0x7f47b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.618+0000 7f47d77bc640 1 --2- 172.21.15.1:0/1736904857 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47b406b4d0 0x7f47b406d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f47cc034000 tx=0x7f47cc009030 comp rx=0 tx=0).stop 2024-03-06T07:22:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.618+0000 7f47d77bc640 1 -- 172.21.15.1:0/1736904857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47d014cf40 msgr2=0x7f47d014a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.618+0000 7f47d77bc640 1 --2- 172.21.15.1:0/1736904857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47d014cf40 0x7f47d014a330 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f47d008c310 tx=0x7f47c800ff70 comp rx=0 tx=0).stop 2024-03-06T07:22:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.618+0000 7f47d77bc640 1 -- 172.21.15.1:0/1736904857 shutdown_connections 2024-03-06T07:22:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.619+0000 7f47d77bc640 1 --2- 172.21.15.1:0/1736904857 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47b406b4d0 0x7f47b406d990 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.619+0000 7f47d77bc640 1 --2- 172.21.15.1:0/1736904857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47d014cf40 0x7f47d014a330 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.619+0000 7f47d77bc640 1 --2- 172.21.15.1:0/1736904857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d0158160 0x7f47d014ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.619+0000 7f47d77bc640 1 --2- 172.21.15.1:0/1736904857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47bc0039e0 0x7f47d00c0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.619+0000 7f47d77bc640 1 -- 172.21.15.1:0/1736904857 >> 172.21.15.1:0/1736904857 conn(0x7f47d0066d00 msgr2=0x7f47d00c34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.619+0000 7f47d77bc640 1 -- 172.21.15.1:0/1736904857 shutdown_connections 2024-03-06T07:22:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.619+0000 7f47d77bc640 1 -- 172.21.15.1:0/1736904857 wait complete. 2024-03-06T07:22:41.631 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-06T07:22:18.907422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-06T07:22:20.028174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:41.632 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:41.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.734+0000 7f4bcc250640 1 Processor -- start 2024-03-06T07:22:41.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.734+0000 7f4bcc250640 1 -- start start 2024-03-06T07:22:41.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.735+0000 7f4bcc250640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4bc4152730 0x7f4bc40c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.735+0000 7f4bcc250640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4bc40c6fd0 0x7f4bc40c1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.735+0000 7f4bcc250640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc40c1cf0 con 0x7f4bc40c18d0 2024-03-06T07:22:41.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.735+0000 7f4bcc250640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc41588e0 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.735+0000 7f4bcc250640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc4158bc0 con 0x7f4bc4152730 2024-03-06T07:22:41.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.735+0000 7f4bc97eb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4bc40c6fd0 0x7f4bc40c1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.735+0000 7f4bc97eb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4bc40c6fd0 0x7f4bc40c1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57244/0 (socket says 172.21.15.1:57244) 2024-03-06T07:22:41.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.735+0000 7f4bc97eb640 1 -- 172.21.15.1:0/1928410091 learned_addr learned my addr 172.21.15.1:0/1928410091 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:41.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.735+0000 7f4bc9fec640 1 --2- 172.21.15.1:0/1928410091 >> v2:172.21.15.98:3300/0 conn(0x7f4bc4152730 0x7f4bc40c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.735+0000 7f4bc97eb640 1 -- 172.21.15.1:0/1928410091 >> v2:172.21.15.98:3300/0 conn(0x7f4bc4152730 msgr2=0x7f4bc40c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.736+0000 7f4bc97eb640 1 --2- 172.21.15.1:0/1928410091 >> v2:172.21.15.98:3300/0 conn(0x7f4bc4152730 0x7f4bc40c6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.736+0000 7f4bc97eb640 1 -- 172.21.15.1:0/1928410091 >> v1:172.21.15.98:6789/0 conn(0x7f4bc40c18d0 legacy=0x7f4bc41552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.736+0000 7f4bc97eb640 1 -- 172.21.15.1:0/1928410091 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc4158ea0 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.736+0000 7f4bc9fec640 1 --2- 172.21.15.1:0/1928410091 >> v2:172.21.15.98:3300/0 conn(0x7f4bc4152730 0x7f4bc40c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:41.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.736+0000 7f4bc97eb640 1 --2- 172.21.15.1:0/1928410091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc40c6fd0 0x7f4bc40c1390 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4bc408c220 tx=0x7f4bb802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df319227e2222cfe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.736+0000 7f4bc8fea640 1 -- 172.21.15.1:0/1928410091 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8038070 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.736+0000 7f4bc8fea640 1 -- 172.21.15.1:0/1928410091 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb802f050 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.736+0000 7f4bc8fea640 1 -- 172.21.15.1:0/1928410091 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8031970 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.737+0000 7f4bcc250640 1 -- 172.21.15.1:0/1928410091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc40c6fd0 msgr2=0x7f4bc40c1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.737+0000 7f4bcc250640 1 --2- 172.21.15.1:0/1928410091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc40c6fd0 0x7f4bc40c1390 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4bc408c220 tx=0x7f4bb802b7f0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.737+0000 7f4bcc250640 1 -- 172.21.15.1:0/1928410091 shutdown_connections 2024-03-06T07:22:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.737+0000 7f4bcc250640 1 --2- 172.21.15.1:0/1928410091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc40c6fd0 0x7f4bc40c1390 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.737+0000 7f4bcc250640 1 --2- 172.21.15.1:0/1928410091 >> v2:172.21.15.98:3300/0 conn(0x7f4bc4152730 0x7f4bc40c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.737+0000 7f4bcc250640 1 -- 172.21.15.1:0/1928410091 >> 172.21.15.1:0/1928410091 conn(0x7f4bc4066d00 msgr2=0x7f4bc4067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.737+0000 7f4bcc250640 1 -- 172.21.15.1:0/1928410091 shutdown_connections 2024-03-06T07:22:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.737+0000 7f4bcc250640 1 -- 172.21.15.1:0/1928410091 wait complete. 2024-03-06T07:22:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.738+0000 7f4bcc250640 1 Processor -- start 2024-03-06T07:22:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.738+0000 7f4bcc250640 1 -- start start 2024-03-06T07:22:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.739+0000 7f4bcc250640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bc40c18d0 0x7f4bc40c35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.739+0000 7f4bcc250640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc40c6fd0 0x7f4bc41ee6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.739+0000 7f4bcc250640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bc4152730 0x7f4bc41eec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.739+0000 7f4bcc250640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc4152b10 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.739+0000 7f4bcc250640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc4159460 con 0x7f4bc4152730 2024-03-06T07:22:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.739+0000 7f4bcc250640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc41559f0 con 0x7f4bc40c18d0 2024-03-06T07:22:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.739+0000 7f4bc97eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc40c6fd0 0x7f4bc41ee6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.739+0000 7f4bc97eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc40c6fd0 0x7f4bc41ee6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57254/0 (socket says 172.21.15.1:57254) 2024-03-06T07:22:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.739+0000 7f4bc97eb640 1 -- 172.21.15.1:0/3865746554 learned_addr learned my addr 172.21.15.1:0/3865746554 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:41.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.739+0000 7f4bc9fec640 1 --2- 172.21.15.1:0/3865746554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bc40c18d0 0x7f4bc40c35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.739+0000 7f4bca7ed640 1 --2- 172.21.15.1:0/3865746554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bc4152730 0x7f4bc41eec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.739+0000 7f4bc97eb640 1 -- 172.21.15.1:0/3865746554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bc40c18d0 msgr2=0x7f4bc40c35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.739+0000 7f4bc97eb640 1 --2- 172.21.15.1:0/3865746554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bc40c18d0 0x7f4bc40c35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.739+0000 7f4bc97eb640 1 -- 172.21.15.1:0/3865746554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bc4152730 msgr2=0x7f4bc41eec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.739+0000 7f4bc97eb640 1 --2- 172.21.15.1:0/3865746554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bc4152730 0x7f4bc41eec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.739+0000 7f4bc97eb640 1 -- 172.21.15.1:0/3865746554 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc41ef170 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.740+0000 7f4bc97eb640 1 --2- 172.21.15.1:0/3865746554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc40c6fd0 0x7f4bc41ee6f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8000c00 tx=0x7f4bb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.740+0000 7f4bc27fc640 1 -- 172.21.15.1:0/3865746554 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8042070 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.740+0000 7f4bc27fc640 1 -- 172.21.15.1:0/3865746554 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb803d870 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.740+0000 7f4bcc250640 1 -- 172.21.15.1:0/3865746554 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc41de970 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.740+0000 7f4bc27fc640 1 -- 172.21.15.1:0/3865746554 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8038040 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.740+0000 7f4bcc250640 1 -- 172.21.15.1:0/3865746554 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc41e1030 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.741+0000 7f4bcc250640 1 -- 172.21.15.1:0/3865746554 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc4067fd0 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.741+0000 7f4bc27fc640 1 -- 172.21.15.1:0/3865746554 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4bb8045070 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.741+0000 7f4bc27fc640 1 --2- 172.21.15.1:0/3865746554 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4bb006b500 0x7f4bb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.742+0000 7f4bc27fc640 1 -- 172.21.15.1:0/3865746554 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f4bb80ac3c0 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.742+0000 7f4bc9fec640 1 --2- 172.21.15.1:0/3865746554 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4bb006b500 0x7f4bb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.742+0000 7f4bc9fec640 1 --2- 172.21.15.1:0/3865746554 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4bb006b500 0x7f4bb006d9c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00aac0 tx=0x7f4bbc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.746+0000 7f4bc27fc640 1 -- 172.21.15.1:0/3865746554 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4bc41ef170 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.926+0000 7f4bcc250640 1 -- 172.21.15.1:0/3865746554 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4bc40c6870 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.927+0000 7f4bc27fc640 1 -- 172.21.15.1:0/3865746554 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4bb807aab0 con 0x7f4bc40c6fd0 2024-03-06T07:22:41.929 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:22:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.929+0000 7f4bcc250640 1 -- 172.21.15.1:0/3865746554 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4bb006b500 msgr2=0x7f4bb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.929+0000 7f4bcc250640 1 --2- 172.21.15.1:0/3865746554 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4bb006b500 0x7f4bb006d9c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00aac0 tx=0x7f4bbc008040 comp rx=0 tx=0).stop 2024-03-06T07:22:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.929+0000 7f4bcc250640 1 -- 172.21.15.1:0/3865746554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc40c6fd0 msgr2=0x7f4bc41ee6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.929+0000 7f4bcc250640 1 --2- 172.21.15.1:0/3865746554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc40c6fd0 0x7f4bc41ee6f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8000c00 tx=0x7f4bb80046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.929+0000 7f4bcc250640 1 -- 172.21.15.1:0/3865746554 shutdown_connections 2024-03-06T07:22:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.929+0000 7f4bcc250640 1 --2- 172.21.15.1:0/3865746554 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4bb006b500 0x7f4bb006d9c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.929+0000 7f4bcc250640 1 --2- 172.21.15.1:0/3865746554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bc4152730 0x7f4bc41eec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.929+0000 7f4bcc250640 1 --2- 172.21.15.1:0/3865746554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bc40c6fd0 0x7f4bc41ee6f0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.929+0000 7f4bcc250640 1 --2- 172.21.15.1:0/3865746554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bc40c18d0 0x7f4bc40c35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.929+0000 7f4bcc250640 1 -- 172.21.15.1:0/3865746554 >> 172.21.15.1:0/3865746554 conn(0x7f4bc4066d00 msgr2=0x7f4bc40670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.929+0000 7f4bcc250640 1 -- 172.21.15.1:0/3865746554 shutdown_connections 2024-03-06T07:22:41.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:41.929+0000 7f4bcc250640 1 -- 172.21.15.1:0/3865746554 wait complete. 2024-03-06T07:22:41.941 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-06T07:22:42.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.045+0000 7f0df7b3a640 1 Processor -- start 2024-03-06T07:22:42.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.045+0000 7f0df7b3a640 1 -- start start 2024-03-06T07:22:42.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.046+0000 7f0df7b3a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0df006e670 0x7f0df006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.046+0000 7f0df7b3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0df00a8520 0x7f0df00c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.046+0000 7f0df7b3a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0df0150290 0x7f0df0150740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.046+0000 7f0df7b3a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0df00c2ca0 con 0x7f0df006e670 2024-03-06T07:22:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.046+0000 7f0df7b3a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0df006f060 con 0x7f0df0150290 2024-03-06T07:22:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.046+0000 7f0df7b3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df00c26e0 con 0x7f0df00a8520 2024-03-06T07:22:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.046+0000 7f0df58d6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0df006e670 0x7f0df006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.046+0000 7f0df58d6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0df006e670 0x7f0df006ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57256/0 (socket says 172.21.15.1:57256) 2024-03-06T07:22:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.046+0000 7f0df58d6640 1 -- 172.21.15.1:0/4143134472 learned_addr learned my addr 172.21.15.1:0/4143134472 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.046+0000 7f0df60d7640 1 --2- 172.21.15.1:0/4143134472 >> v2:172.21.15.98:3300/0 conn(0x7f0df0150290 0x7f0df0150740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.046+0000 7f0df50d5640 1 --2- 172.21.15.1:0/4143134472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0df00a8520 0x7f0df00c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.046+0000 7f0df58d6640 1 -- 172.21.15.1:0/4143134472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0df00a8520 msgr2=0x7f0df00c1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.046+0000 7f0df58d6640 1 --2- 172.21.15.1:0/4143134472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0df00a8520 0x7f0df00c1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.046+0000 7f0df58d6640 1 -- 172.21.15.1:0/4143134472 >> v2:172.21.15.98:3300/0 conn(0x7f0df0150290 msgr2=0x7f0df0150740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.046+0000 7f0df58d6640 1 --2- 172.21.15.1:0/4143134472 >> v2:172.21.15.98:3300/0 conn(0x7f0df0150290 0x7f0df0150740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.046+0000 7f0df58d6640 1 -- 172.21.15.1:0/4143134472 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df00c29c0 con 0x7f0df006e670 2024-03-06T07:22:42.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.047+0000 7f0df58d6640 1 --2- 172.21.15.1:0/4143134472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0df006e670 0x7f0df006ea50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f0df0072dd0 tx=0x7f0de802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e4af50a3d34517 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.047+0000 7f0df48d4640 1 -- 172.21.15.1:0/4143134472 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de802e070 con 0x7f0df006e670 2024-03-06T07:22:42.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.047+0000 7f0df48d4640 1 -- 172.21.15.1:0/4143134472 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de8037070 con 0x7f0df006e670 2024-03-06T07:22:42.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.047+0000 7f0df48d4640 1 -- 172.21.15.1:0/4143134472 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de8031a00 con 0x7f0df006e670 2024-03-06T07:22:42.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.048+0000 7f0df7b3a640 1 -- 172.21.15.1:0/4143134472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0df006e670 msgr2=0x7f0df006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.048+0000 7f0df7b3a640 1 --2- 172.21.15.1:0/4143134472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0df006e670 0x7f0df006ea50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f0df0072dd0 tx=0x7f0de802b880 comp rx=0 tx=0).stop 2024-03-06T07:22:42.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.048+0000 7f0df7b3a640 1 -- 172.21.15.1:0/4143134472 shutdown_connections 2024-03-06T07:22:42.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.048+0000 7f0df7b3a640 1 --2- 172.21.15.1:0/4143134472 >> v2:172.21.15.98:3300/0 conn(0x7f0df0150290 0x7f0df0150740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.048+0000 7f0df7b3a640 1 --2- 172.21.15.1:0/4143134472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0df00a8520 0x7f0df00c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.048+0000 7f0df7b3a640 1 --2- 172.21.15.1:0/4143134472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0df006e670 0x7f0df006ea50 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.048+0000 7f0df7b3a640 1 -- 172.21.15.1:0/4143134472 >> 172.21.15.1:0/4143134472 conn(0x7f0df00c3cc0 msgr2=0x7f0df00c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.048+0000 7f0df7b3a640 1 -- 172.21.15.1:0/4143134472 shutdown_connections 2024-03-06T07:22:42.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.048+0000 7f0df7b3a640 1 -- 172.21.15.1:0/4143134472 wait complete. 2024-03-06T07:22:42.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.049+0000 7f0df7b3a640 1 Processor -- start 2024-03-06T07:22:42.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.049+0000 7f0df7b3a640 1 -- start start 2024-03-06T07:22:42.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df7b3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0df006e670 0x7f0df01555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df7b3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0df00a8520 0x7f0df01e90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df7b3a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0df0150290 0x7f0df014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df7b3a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df0153620 con 0x7f0df0150290 2024-03-06T07:22:42.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df7b3a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df01534a0 con 0x7f0df006e670 2024-03-06T07:22:42.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df7b3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df01537a0 con 0x7f0df00a8520 2024-03-06T07:22:42.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df60d7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0df0150290 0x7f0df014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df60d7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0df0150290 0x7f0df014efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57272/0 (socket says 172.21.15.1:57272) 2024-03-06T07:22:42.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df60d7640 1 -- 172.21.15.1:0/2740757486 learned_addr learned my addr 172.21.15.1:0/2740757486 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:42.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df58d6640 1 --2- 172.21.15.1:0/2740757486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0df006e670 0x7f0df01555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df50d5640 1 --2- 172.21.15.1:0/2740757486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0df00a8520 0x7f0df01e90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df60d7640 1 -- 172.21.15.1:0/2740757486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0df00a8520 msgr2=0x7f0df01e90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df60d7640 1 --2- 172.21.15.1:0/2740757486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0df00a8520 0x7f0df01e90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df60d7640 1 -- 172.21.15.1:0/2740757486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0df006e670 msgr2=0x7f0df01555f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df60d7640 1 --2- 172.21.15.1:0/2740757486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0df006e670 0x7f0df01555f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df60d7640 1 -- 172.21.15.1:0/2740757486 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df01e9600 con 0x7f0df0150290 2024-03-06T07:22:42.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df60d7640 1 --2- 172.21.15.1:0/2740757486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0df0150290 0x7f0df014efe0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f0de000efc0 tx=0x7f0de000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0dee7fc640 1 -- 172.21.15.1:0/2740757486 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de0018020 con 0x7f0df0150290 2024-03-06T07:22:42.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0dee7fc640 1 -- 172.21.15.1:0/2740757486 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de0014480 con 0x7f0df0150290 2024-03-06T07:22:42.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df7b3a640 1 -- 172.21.15.1:0/2740757486 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df01527c0 con 0x7f0df0150290 2024-03-06T07:22:42.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0dee7fc640 1 -- 172.21.15.1:0/2740757486 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de0014870 con 0x7f0df0150290 2024-03-06T07:22:42.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.050+0000 7f0df7b3a640 1 -- 172.21.15.1:0/2740757486 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df0150e00 con 0x7f0df0150290 2024-03-06T07:22:42.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.051+0000 7f0df7b3a640 1 -- 172.21.15.1:0/2740757486 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df014f9d0 con 0x7f0df0150290 2024-03-06T07:22:42.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.052+0000 7f0dee7fc640 1 -- 172.21.15.1:0/2740757486 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0de0005d10 con 0x7f0df0150290 2024-03-06T07:22:42.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.052+0000 7f0dee7fc640 1 --2- 172.21.15.1:0/2740757486 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ddc06b4d0 0x7f0ddc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.052+0000 7f0dee7fc640 1 -- 172.21.15.1:0/2740757486 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f0de008b0b0 con 0x7f0df0150290 2024-03-06T07:22:42.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.052+0000 7f0df58d6640 1 --2- 172.21.15.1:0/2740757486 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ddc06b4d0 0x7f0ddc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.053+0000 7f0df58d6640 1 --2- 172.21.15.1:0/2740757486 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ddc06b4d0 0x7f0ddc06d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0de8036000 tx=0x7f0de803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.057+0000 7f0dee7fc640 1 -- 172.21.15.1:0/2740757486 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0df014f9d0 con 0x7f0df0150290 2024-03-06T07:22:42.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.238+0000 7f0df7b3a640 1 -- 172.21.15.1:0/2740757486 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0df0067fd0 con 0x7f0df0150290 2024-03-06T07:22:42.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.239+0000 7f0dee7fc640 1 -- 172.21.15.1:0/2740757486 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v62) v1 ==== 105+0+11 (secure 0 0 0) 0x7f0de0022030 con 0x7f0df0150290 2024-03-06T07:22:42.241 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:22:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.241+0000 7f0df7b3a640 1 -- 172.21.15.1:0/2740757486 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ddc06b4d0 msgr2=0x7f0ddc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.241+0000 7f0df7b3a640 1 --2- 172.21.15.1:0/2740757486 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ddc06b4d0 0x7f0ddc06d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0de8036000 tx=0x7f0de803d040 comp rx=0 tx=0).stop 2024-03-06T07:22:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.241+0000 7f0df7b3a640 1 -- 172.21.15.1:0/2740757486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0df0150290 msgr2=0x7f0df014efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.241+0000 7f0df7b3a640 1 --2- 172.21.15.1:0/2740757486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0df0150290 0x7f0df014efe0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f0de000efc0 tx=0x7f0de000c490 comp rx=0 tx=0).stop 2024-03-06T07:22:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.241+0000 7f0df7b3a640 1 -- 172.21.15.1:0/2740757486 shutdown_connections 2024-03-06T07:22:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.241+0000 7f0df7b3a640 1 --2- 172.21.15.1:0/2740757486 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ddc06b4d0 0x7f0ddc06d990 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.241+0000 7f0df7b3a640 1 --2- 172.21.15.1:0/2740757486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0df0150290 0x7f0df014efe0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.241+0000 7f0df7b3a640 1 --2- 172.21.15.1:0/2740757486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0df00a8520 0x7f0df01e90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.241+0000 7f0df7b3a640 1 --2- 172.21.15.1:0/2740757486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0df006e670 0x7f0df01555f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.241+0000 7f0df7b3a640 1 -- 172.21.15.1:0/2740757486 >> 172.21.15.1:0/2740757486 conn(0x7f0df00c3cc0 msgr2=0x7f0df00c40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.241+0000 7f0df7b3a640 1 -- 172.21.15.1:0/2740757486 shutdown_connections 2024-03-06T07:22:42.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.241+0000 7f0df7b3a640 1 -- 172.21.15.1:0/2740757486 wait complete. 2024-03-06T07:22:42.253 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-06T07:22:42.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.356+0000 7fa8b7642640 1 Processor -- start 2024-03-06T07:22:42.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.356+0000 7fa8b7642640 1 -- start start 2024-03-06T07:22:42.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.357+0000 7fa8b7642640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa8b00c2b30 0x7fa8b00724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.357+0000 7fa8b7642640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8b0152b70 0x7fa8b0072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.357+0000 7fa8b7642640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b0151dd0 con 0x7fa8b00a5fa0 2024-03-06T07:22:42.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.357+0000 7fa8b7642640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b01526e0 con 0x7fa8b00c2b30 2024-03-06T07:22:42.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.357+0000 7fa8b7642640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b0156190 con 0x7fa8b0152b70 2024-03-06T07:22:42.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.357+0000 7fa8b53de640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa8b00c2b30 0x7fa8b00724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.357+0000 7fa8b53de640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa8b00c2b30 0x7fa8b00724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57280/0 (socket says 172.21.15.1:57280) 2024-03-06T07:22:42.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.357+0000 7fa8b53de640 1 -- 172.21.15.1:0/2677977854 learned_addr learned my addr 172.21.15.1:0/2677977854 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:42.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.357+0000 7fa8b4bdd640 1 --2- 172.21.15.1:0/2677977854 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8b0152b70 0x7fa8b0072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.357+0000 7fa8b53de640 1 -- 172.21.15.1:0/2677977854 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8b0152b70 msgr2=0x7fa8b0072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.357+0000 7fa8b53de640 1 --2- 172.21.15.1:0/2677977854 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8b0152b70 0x7fa8b0072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.357+0000 7fa8b53de640 1 -- 172.21.15.1:0/2677977854 >> v1:172.21.15.98:6789/0 conn(0x7fa8b00a5fa0 legacy=0x7fa8b00c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.357+0000 7fa8b53de640 1 -- 172.21.15.1:0/2677977854 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b0152090 con 0x7fa8b00c2b30 2024-03-06T07:22:42.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.358+0000 7fa8b53de640 1 --2- 172.21.15.1:0/2677977854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8b00c2b30 0x7fa8b00724c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8009870 tx=0x7fa8a802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2940ec12259220d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.358+0000 7fa8affff640 1 -- 172.21.15.1:0/2677977854 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8034070 con 0x7fa8b00c2b30 2024-03-06T07:22:42.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.358+0000 7fa8affff640 1 -- 172.21.15.1:0/2677977854 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a8030840 con 0x7fa8b00c2b30 2024-03-06T07:22:42.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.358+0000 7fa8affff640 1 -- 172.21.15.1:0/2677977854 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8030c10 con 0x7fa8b00c2b30 2024-03-06T07:22:42.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.359+0000 7fa8b7642640 1 -- 172.21.15.1:0/2677977854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8b00c2b30 msgr2=0x7fa8b00724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.359+0000 7fa8b7642640 1 --2- 172.21.15.1:0/2677977854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8b00c2b30 0x7fa8b00724c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8009870 tx=0x7fa8a802b980 comp rx=0 tx=0).stop 2024-03-06T07:22:42.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.359+0000 7fa8b7642640 1 -- 172.21.15.1:0/2677977854 shutdown_connections 2024-03-06T07:22:42.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.359+0000 7fa8b7642640 1 --2- 172.21.15.1:0/2677977854 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8b0152b70 0x7fa8b0072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.359+0000 7fa8b7642640 1 --2- 172.21.15.1:0/2677977854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8b00c2b30 0x7fa8b00724c0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.359+0000 7fa8b7642640 1 -- 172.21.15.1:0/2677977854 >> 172.21.15.1:0/2677977854 conn(0x7fa8b00c6860 msgr2=0x7fa8b00c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.359+0000 7fa8b7642640 1 -- 172.21.15.1:0/2677977854 shutdown_connections 2024-03-06T07:22:42.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.359+0000 7fa8b7642640 1 -- 172.21.15.1:0/2677977854 wait complete. 2024-03-06T07:22:42.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.360+0000 7fa8b7642640 1 Processor -- start 2024-03-06T07:22:42.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.360+0000 7fa8b7642640 1 -- start start 2024-03-06T07:22:42.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b7642640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8b00a5fa0 0x7fa8b01ecce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b7642640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8b00c2b30 0x7fa8b00bc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b7642640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8b0152b70 0x7fa8b014b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b7642640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b0155f00 con 0x7fa8b00c2b30 2024-03-06T07:22:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b7642640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b0155d80 con 0x7fa8b0152b70 2024-03-06T07:22:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b7642640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b00c3a00 con 0x7fa8b00a5fa0 2024-03-06T07:22:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b4bdd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8b00c2b30 0x7fa8b00bc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b4bdd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8b00c2b30 0x7fa8b00bc110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57292/0 (socket says 172.21.15.1:57292) 2024-03-06T07:22:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b4bdd640 1 -- 172.21.15.1:0/1437099760 learned_addr learned my addr 172.21.15.1:0/1437099760 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b53de640 1 --2- 172.21.15.1:0/1437099760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8b00a5fa0 0x7fa8b01ecce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b5bdf640 1 --2- 172.21.15.1:0/1437099760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8b0152b70 0x7fa8b014b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b4bdd640 1 -- 172.21.15.1:0/1437099760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8b00a5fa0 msgr2=0x7fa8b01ecce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b4bdd640 1 --2- 172.21.15.1:0/1437099760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8b00a5fa0 0x7fa8b01ecce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b4bdd640 1 -- 172.21.15.1:0/1437099760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8b0152b70 msgr2=0x7fa8b014b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b4bdd640 1 --2- 172.21.15.1:0/1437099760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8b0152b70 0x7fa8b014b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b4bdd640 1 -- 172.21.15.1:0/1437099760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b00c0480 con 0x7fa8b00c2b30 2024-03-06T07:22:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b4bdd640 1 --2- 172.21.15.1:0/1437099760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8b00c2b30 0x7fa8b00bc110 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fa8a400e870 tx=0x7fa8a400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8adffb640 1 -- 172.21.15.1:0/1437099760 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a4017070 con 0x7fa8b00c2b30 2024-03-06T07:22:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8adffb640 1 -- 172.21.15.1:0/1437099760 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a400c940 con 0x7fa8b00c2b30 2024-03-06T07:22:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8b7642640 1 -- 172.21.15.1:0/1437099760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b00c0760 con 0x7fa8b00c2b30 2024-03-06T07:22:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.361+0000 7fa8adffb640 1 -- 172.21.15.1:0/1437099760 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a400cd30 con 0x7fa8b00c2b30 2024-03-06T07:22:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.362+0000 7fa8b7642640 1 -- 172.21.15.1:0/1437099760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b01ec7c0 con 0x7fa8b00c2b30 2024-03-06T07:22:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.362+0000 7fa8b7642640 1 -- 172.21.15.1:0/1437099760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b0072d60 con 0x7fa8b00c2b30 2024-03-06T07:22:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.363+0000 7fa8adffb640 1 -- 172.21.15.1:0/1437099760 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8a401a070 con 0x7fa8b00c2b30 2024-03-06T07:22:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.363+0000 7fa8adffb640 1 --2- 172.21.15.1:0/1437099760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa89c06b500 0x7fa89c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.363+0000 7fa8adffb640 1 -- 172.21.15.1:0/1437099760 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7fa8a408a2b0 con 0x7fa8b00c2b30 2024-03-06T07:22:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.364+0000 7fa8b53de640 1 --2- 172.21.15.1:0/1437099760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa89c06b500 0x7fa89c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.364+0000 7fa8b53de640 1 --2- 172.21.15.1:0/1437099760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa89c06b500 0x7fa89c06d9c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8036000 tx=0x7fa8a8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.368+0000 7fa8adffb640 1 -- 172.21.15.1:0/1437099760 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8b0072d60 con 0x7fa8b00c2b30 2024-03-06T07:22:42.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.548+0000 7fa8b7642640 1 -- 172.21.15.1:0/1437099760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7fa8b01ec5b0 con 0x7fa8b00c2b30 2024-03-06T07:22:42.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.548+0000 7fa8adffb640 1 -- 172.21.15.1:0/1437099760 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v62) v1 ==== 101+0+11 (secure 0 0 0) 0x7fa8a40587f0 con 0x7fa8b00c2b30 2024-03-06T07:22:42.550 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:22:42.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.550+0000 7fa8b7642640 1 -- 172.21.15.1:0/1437099760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa89c06b500 msgr2=0x7fa89c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.550+0000 7fa8b7642640 1 --2- 172.21.15.1:0/1437099760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa89c06b500 0x7fa89c06d9c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8036000 tx=0x7fa8a8008030 comp rx=0 tx=0).stop 2024-03-06T07:22:42.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.550+0000 7fa8b7642640 1 -- 172.21.15.1:0/1437099760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8b00c2b30 msgr2=0x7fa8b00bc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.550+0000 7fa8b7642640 1 --2- 172.21.15.1:0/1437099760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8b00c2b30 0x7fa8b00bc110 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fa8a400e870 tx=0x7fa8a400ed30 comp rx=0 tx=0).stop 2024-03-06T07:22:42.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.550+0000 7fa8b7642640 1 -- 172.21.15.1:0/1437099760 shutdown_connections 2024-03-06T07:22:42.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.550+0000 7fa8b7642640 1 --2- 172.21.15.1:0/1437099760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa89c06b500 0x7fa89c06d9c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.550+0000 7fa8b7642640 1 --2- 172.21.15.1:0/1437099760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8b0152b70 0x7fa8b014b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.550+0000 7fa8b7642640 1 --2- 172.21.15.1:0/1437099760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8b00c2b30 0x7fa8b00bc110 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.550+0000 7fa8b7642640 1 --2- 172.21.15.1:0/1437099760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8b00a5fa0 0x7fa8b01ecce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.550+0000 7fa8b7642640 1 -- 172.21.15.1:0/1437099760 >> 172.21.15.1:0/1437099760 conn(0x7fa8b00c6860 msgr2=0x7fa8b014fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.550+0000 7fa8b7642640 1 -- 172.21.15.1:0/1437099760 shutdown_connections 2024-03-06T07:22:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.550+0000 7fa8b7642640 1 -- 172.21.15.1:0/1437099760 wait complete. 2024-03-06T07:22:42.563 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.662+0000 7ff61c313640 1 Processor -- start 2024-03-06T07:22:42.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.662+0000 7ff61c313640 1 -- start start 2024-03-06T07:22:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.663+0000 7ff61c313640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff61414cdc0 0x7ff614152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.663+0000 7ff61c313640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff61406b510 0x7ff61406b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.663+0000 7ff61c313640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6141569e0 con 0x7ff614156500 2024-03-06T07:22:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.663+0000 7ff61c313640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6140c2dd0 con 0x7ff61414cdc0 2024-03-06T07:22:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.663+0000 7ff61c313640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff614156cc0 con 0x7ff61406b510 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.663+0000 7ff61a0af640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff61414cdc0 0x7ff614152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.663+0000 7ff6198ae640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff614156500 0x7ff6140c2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37868/0 (socket says 172.21.15.1:37868) 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.663+0000 7ff6198ae640 1 -- 172.21.15.1:0/311486018 learned_addr learned my addr 172.21.15.1:0/311486018 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.663+0000 7ff61a8b0640 1 --2- 172.21.15.1:0/311486018 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff61406b510 0x7ff61406b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.663+0000 7ff61a0af640 1 -- 172.21.15.1:0/311486018 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff61406b510 msgr2=0x7ff61406b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.663+0000 7ff61a0af640 1 --2- 172.21.15.1:0/311486018 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff61406b510 0x7ff61406b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.663+0000 7ff61a0af640 1 -- 172.21.15.1:0/311486018 >> v1:172.21.15.1:6789/0 conn(0x7ff614156500 legacy=0x7ff6140c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.663+0000 7ff61a0af640 1 -- 172.21.15.1:0/311486018 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff614156fa0 con 0x7ff61414cdc0 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.663+0000 7ff6190ad640 1 -- 172.21.15.1:0/311486018 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745107777 0 0) 0x7ff6141569e0 con 0x7ff614156500 2024-03-06T07:22:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.664+0000 7ff61a0af640 1 --2- 172.21.15.1:0/311486018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff61414cdc0 0x7ff614152f10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff6100097a0 tx=0x7ff61002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03ac21b0252aec3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.664+0000 7ff6190ad640 1 -- 172.21.15.1:0/311486018 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610035020 con 0x7ff61414cdc0 2024-03-06T07:22:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.664+0000 7ff6190ad640 1 -- 172.21.15.1:0/311486018 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff610031830 con 0x7ff61414cdc0 2024-03-06T07:22:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.664+0000 7ff6190ad640 1 -- 172.21.15.1:0/311486018 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610031c00 con 0x7ff61414cdc0 2024-03-06T07:22:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.665+0000 7ff61c313640 1 -- 172.21.15.1:0/311486018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff61414cdc0 msgr2=0x7ff614152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.665+0000 7ff61c313640 1 --2- 172.21.15.1:0/311486018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff61414cdc0 0x7ff614152f10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff6100097a0 tx=0x7ff61002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.665+0000 7ff61c313640 1 -- 172.21.15.1:0/311486018 shutdown_connections 2024-03-06T07:22:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.665+0000 7ff61c313640 1 --2- 172.21.15.1:0/311486018 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff61406b510 0x7ff61406b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.665+0000 7ff61c313640 1 --2- 172.21.15.1:0/311486018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff61414cdc0 0x7ff614152f10 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.665+0000 7ff61c313640 1 -- 172.21.15.1:0/311486018 >> 172.21.15.1:0/311486018 conn(0x7ff6140a9700 msgr2=0x7ff6140a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.665+0000 7ff61c313640 1 -- 172.21.15.1:0/311486018 shutdown_connections 2024-03-06T07:22:42.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.665+0000 7ff61c313640 1 -- 172.21.15.1:0/311486018 wait complete. 2024-03-06T07:22:42.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.666+0000 7ff61c313640 1 Processor -- start 2024-03-06T07:22:42.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.666+0000 7ff61c313640 1 -- start start 2024-03-06T07:22:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.666+0000 7ff61c313640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff61406b510 0x7ff61414b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.666+0000 7ff61c313640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff61414cdc0 0x7ff6140c2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.666+0000 7ff61c313640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff614156500 0x7ff61414e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.666+0000 7ff61c313640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61406bea0 con 0x7ff61414cdc0 2024-03-06T07:22:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.666+0000 7ff61c313640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6140c5a70 con 0x7ff614156500 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.666+0000 7ff61c313640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6140c65e0 con 0x7ff61406b510 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.667+0000 7ff6198ae640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff61414cdc0 0x7ff6140c2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.667+0000 7ff6198ae640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff61414cdc0 0x7ff6140c2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57310/0 (socket says 172.21.15.1:57310) 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.667+0000 7ff6198ae640 1 -- 172.21.15.1:0/2602989228 learned_addr learned my addr 172.21.15.1:0/2602989228 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.667+0000 7ff61a0af640 1 --2- 172.21.15.1:0/2602989228 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff61406b510 0x7ff61414b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.667+0000 7ff61a8b0640 1 --2- 172.21.15.1:0/2602989228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff614156500 0x7ff61414e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.667+0000 7ff6198ae640 1 -- 172.21.15.1:0/2602989228 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff61406b510 msgr2=0x7ff61414b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.667+0000 7ff6198ae640 1 --2- 172.21.15.1:0/2602989228 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff61406b510 0x7ff61414b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.667+0000 7ff6198ae640 1 -- 172.21.15.1:0/2602989228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff614156500 msgr2=0x7ff61414e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.667+0000 7ff6198ae640 1 --2- 172.21.15.1:0/2602989228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff614156500 0x7ff61414e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.667+0000 7ff6198ae640 1 -- 172.21.15.1:0/2602989228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6141ec9f0 con 0x7ff61414cdc0 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.667+0000 7ff6198ae640 1 --2- 172.21.15.1:0/2602989228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff61414cdc0 0x7ff6140c2250 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff60400c430 tx=0x7ff60400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.667+0000 7ff60affd640 1 -- 172.21.15.1:0/2602989228 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6040074e0 con 0x7ff61414cdc0 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.667+0000 7ff60affd640 1 -- 172.21.15.1:0/2602989228 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6040047e0 con 0x7ff61414cdc0 2024-03-06T07:22:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.667+0000 7ff61c313640 1 -- 172.21.15.1:0/2602989228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff614150170 con 0x7ff61414cdc0 2024-03-06T07:22:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.667+0000 7ff60affd640 1 -- 172.21.15.1:0/2602989228 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604004bd0 con 0x7ff61414cdc0 2024-03-06T07:22:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.667+0000 7ff61c313640 1 -- 172.21.15.1:0/2602989228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6141eccd0 con 0x7ff61414cdc0 2024-03-06T07:22:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.668+0000 7ff61c313640 1 -- 172.21.15.1:0/2602989228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff614067fd0 con 0x7ff61414cdc0 2024-03-06T07:22:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.669+0000 7ff60affd640 1 -- 172.21.15.1:0/2602989228 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff604011070 con 0x7ff61414cdc0 2024-03-06T07:22:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.669+0000 7ff60affd640 1 --2- 172.21.15.1:0/2602989228 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff5f806b4d0 0x7ff5f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.669+0000 7ff60affd640 1 -- 172.21.15.1:0/2602989228 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7ff604089f90 con 0x7ff61414cdc0 2024-03-06T07:22:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.669+0000 7ff61a0af640 1 --2- 172.21.15.1:0/2602989228 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff5f806b4d0 0x7ff5f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.670+0000 7ff61a0af640 1 --2- 172.21.15.1:0/2602989228 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff5f806b4d0 0x7ff5f806d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff610036000 tx=0x7ff61003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.674+0000 7ff60affd640 1 -- 172.21.15.1:0/2602989228 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff604058560 con 0x7ff61414cdc0 2024-03-06T07:22:42.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.892+0000 7ff61c313640 1 -- 172.21.15.1:0/2602989228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff614152dc0 con 0x7ff61414cdc0 2024-03-06T07:22:42.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.893+0000 7ff60affd640 1 -- 172.21.15.1:0/2602989228 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4973 (secure 0 0 0) 0x7ff60405c600 con 0x7ff61414cdc0 2024-03-06T07:22:42.895 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:42.895 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":48,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2604497680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2604497680},{"type":"v1","addr":"172.21.15.98:6836","nonce":2604497680}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/1450734721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1450734721},{"type":"v1","addr":"172.21.15.1:6834","nonce":1450734721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4269513324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4269513324},{"type":"v1","addr":"172.21.15.1:6835","nonce":4269513324}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:21.047528+0000","modified":"2024-03-06T07:22:41.965083+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5087":{"gid":5087,"name":"d","rank":0,"incarnation":45,"state":"up:active","state_seq":13,"addr":"172.21.15.98:6837/2828783606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2828783606},{"type":"v1","addr":"172.21.15.98:6837","nonce":2828783606}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5087,"qdb_cluster":[5087]},"id":4}]} 2024-03-06T07:22:42.895 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 48 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.895+0000 7ff61c313640 1 -- 172.21.15.1:0/2602989228 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff5f806b4d0 msgr2=0x7ff5f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.895+0000 7ff61c313640 1 --2- 172.21.15.1:0/2602989228 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff5f806b4d0 0x7ff5f806d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff610036000 tx=0x7ff61003d040 comp rx=0 tx=0).stop 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.895+0000 7ff61c313640 1 -- 172.21.15.1:0/2602989228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff61414cdc0 msgr2=0x7ff6140c2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.895+0000 7ff61c313640 1 --2- 172.21.15.1:0/2602989228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff61414cdc0 0x7ff6140c2250 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff60400c430 tx=0x7ff60400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.895+0000 7ff61c313640 1 -- 172.21.15.1:0/2602989228 shutdown_connections 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.895+0000 7ff61c313640 1 --2- 172.21.15.1:0/2602989228 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff5f806b4d0 0x7ff5f806d990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.895+0000 7ff61c313640 1 --2- 172.21.15.1:0/2602989228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff614156500 0x7ff61414e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.895+0000 7ff61c313640 1 --2- 172.21.15.1:0/2602989228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff61414cdc0 0x7ff6140c2250 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.895+0000 7ff61c313640 1 --2- 172.21.15.1:0/2602989228 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff61406b510 0x7ff61414b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.895+0000 7ff61c313640 1 -- 172.21.15.1:0/2602989228 >> 172.21.15.1:0/2602989228 conn(0x7ff6140a9700 msgr2=0x7ff614153580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.895+0000 7ff61c313640 1 -- 172.21.15.1:0/2602989228 shutdown_connections 2024-03-06T07:22:42.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:42.895+0000 7ff61c313640 1 -- 172.21.15.1:0/2602989228 wait complete. 2024-03-06T07:22:42.908 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:43.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.007+0000 7f31339b6640 1 Processor -- start 2024-03-06T07:22:43.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.007+0000 7f31339b6640 1 -- start start 2024-03-06T07:22:43.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.008+0000 7f31339b6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f312c072bd0 0x7f312c0c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.008+0000 7f31339b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f312c152b70 0x7f312c0c3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.008+0000 7f31339b6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312c151dd0 con 0x7f312c0c2b30 2024-03-06T07:22:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.008+0000 7f31339b6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f312c1526e0 con 0x7f312c072bd0 2024-03-06T07:22:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.008+0000 7f31339b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312c156190 con 0x7f312c152b70 2024-03-06T07:22:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.008+0000 7f3131f53640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f312c152b70 0x7f312c0c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.008+0000 7f3130f51640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f312c072bd0 0x7f312c0c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.008+0000 7f3131f53640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f312c152b70 0x7f312c0c3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:36028/0 (socket says 172.21.15.1:36028) 2024-03-06T07:22:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.008+0000 7f3130f51640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f312c072bd0 0x7f312c0c32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:33354/0 (socket says 172.21.15.1:33354) 2024-03-06T07:22:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.008+0000 7f3131f53640 1 -- 172.21.15.1:0/3395671771 learned_addr learned my addr 172.21.15.1:0/3395671771 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.008+0000 7f3130f51640 1 -- 172.21.15.1:0/3395671771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f312c152b70 msgr2=0x7f312c0c3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.008+0000 7f3130f51640 1 --2- 172.21.15.1:0/3395671771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f312c152b70 0x7f312c0c3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.008+0000 7f3130f51640 1 -- 172.21.15.1:0/3395671771 >> v1:172.21.15.98:6789/0 conn(0x7f312c0c2b30 legacy=0x7f312c0724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.008+0000 7f3130f51640 1 -- 172.21.15.1:0/3395671771 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c152090 con 0x7f312c072bd0 2024-03-06T07:22:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.008+0000 7f3131f53640 1 --2- 172.21.15.1:0/3395671771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f312c152b70 0x7f312c0c3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.009+0000 7f3130f51640 1 --2- 172.21.15.1:0/3395671771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f312c072bd0 0x7f312c0c32f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f312c152860 tx=0x7f311c02b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=489eb1e632e96fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.009+0000 7f3123fff640 1 -- 172.21.15.1:0/3395671771 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c035020 con 0x7f312c072bd0 2024-03-06T07:22:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.009+0000 7f3123fff640 1 -- 172.21.15.1:0/3395671771 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311c031800 con 0x7f312c072bd0 2024-03-06T07:22:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.009+0000 7f3123fff640 1 -- 172.21.15.1:0/3395671771 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c031bd0 con 0x7f312c072bd0 2024-03-06T07:22:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.010+0000 7f31339b6640 1 -- 172.21.15.1:0/3395671771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f312c072bd0 msgr2=0x7f312c0c32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.010+0000 7f31339b6640 1 --2- 172.21.15.1:0/3395671771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f312c072bd0 0x7f312c0c32f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f312c152860 tx=0x7f311c02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.010+0000 7f31339b6640 1 -- 172.21.15.1:0/3395671771 shutdown_connections 2024-03-06T07:22:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.010+0000 7f31339b6640 1 --2- 172.21.15.1:0/3395671771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f312c152b70 0x7f312c0c3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.010+0000 7f31339b6640 1 --2- 172.21.15.1:0/3395671771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f312c072bd0 0x7f312c0c32f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.010+0000 7f31339b6640 1 -- 172.21.15.1:0/3395671771 >> 172.21.15.1:0/3395671771 conn(0x7f312c0c6860 msgr2=0x7f312c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.010+0000 7f31339b6640 1 -- 172.21.15.1:0/3395671771 shutdown_connections 2024-03-06T07:22:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.010+0000 7f31339b6640 1 -- 172.21.15.1:0/3395671771 wait complete. 2024-03-06T07:22:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.011+0000 7f31339b6640 1 Processor -- start 2024-03-06T07:22:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.011+0000 7f31339b6640 1 -- start start 2024-03-06T07:22:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f31339b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f312c072bd0 0x7f312c149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f31339b6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f312c0c2b30 0x7f312c14fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f31339b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f312c152b70 0x7f312c14e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f31339b6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c0c3ef0 con 0x7f312c0c2b30 2024-03-06T07:22:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f31339b6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c0c3d70 con 0x7f312c152b70 2024-03-06T07:22:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f31339b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312c155d80 con 0x7f312c072bd0 2024-03-06T07:22:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f3130f51640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f312c0c2b30 0x7f312c14fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f3130f51640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f312c0c2b30 0x7f312c14fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57324/0 (socket says 172.21.15.1:57324) 2024-03-06T07:22:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f3130f51640 1 -- 172.21.15.1:0/2327493131 learned_addr learned my addr 172.21.15.1:0/2327493131 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f3131752640 1 --2- 172.21.15.1:0/2327493131 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f312c072bd0 0x7f312c149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f3131f53640 1 --2- 172.21.15.1:0/2327493131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f312c152b70 0x7f312c14e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f3130f51640 1 -- 172.21.15.1:0/2327493131 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f312c072bd0 msgr2=0x7f312c149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f3130f51640 1 --2- 172.21.15.1:0/2327493131 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f312c072bd0 0x7f312c149f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f3130f51640 1 -- 172.21.15.1:0/2327493131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f312c152b70 msgr2=0x7f312c14e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f3130f51640 1 --2- 172.21.15.1:0/2327493131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f312c152b70 0x7f312c14e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f3130f51640 1 -- 172.21.15.1:0/2327493131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c14a4b0 con 0x7f312c0c2b30 2024-03-06T07:22:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f3131752640 1 --2- 172.21.15.1:0/2327493131 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f312c072bd0 0x7f312c149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f3131f53640 1 --2- 172.21.15.1:0/2327493131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f312c152b70 0x7f312c14e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.012+0000 7f3130f51640 1 --2- 172.21.15.1:0/2327493131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f312c0c2b30 0x7f312c14fb00 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f311c009890 tx=0x7f311c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.013+0000 7f3121ffb640 1 -- 172.21.15.1:0/2327493131 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c041070 con 0x7f312c0c2b30 2024-03-06T07:22:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.013+0000 7f3121ffb640 1 -- 172.21.15.1:0/2327493131 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311c03ad00 con 0x7f312c0c2b30 2024-03-06T07:22:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.013+0000 7f3121ffb640 1 -- 172.21.15.1:0/2327493131 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c035040 con 0x7f312c0c2b30 2024-03-06T07:22:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.013+0000 7f31339b6640 1 -- 172.21.15.1:0/2327493131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f312c0aca40 con 0x7f312c0c2b30 2024-03-06T07:22:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.013+0000 7f31339b6640 1 -- 172.21.15.1:0/2327493131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f312c0c4990 con 0x7f312c0c2b30 2024-03-06T07:22:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.014+0000 7f31339b6640 1 -- 172.21.15.1:0/2327493131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312c067fd0 con 0x7f312c0c2b30 2024-03-06T07:22:43.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.014+0000 7f3121ffb640 1 -- 172.21.15.1:0/2327493131 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f311c038020 con 0x7f312c0c2b30 2024-03-06T07:22:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.014+0000 7f3121ffb640 1 --2- 172.21.15.1:0/2327493131 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f311006b4d0 0x7f311006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.015+0000 7f3121ffb640 1 -- 172.21.15.1:0/2327493131 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f311c0abe80 con 0x7f312c0c2b30 2024-03-06T07:22:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.015+0000 7f3131752640 1 --2- 172.21.15.1:0/2327493131 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f311006b4d0 0x7f311006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.015+0000 7f3131752640 1 --2- 172.21.15.1:0/2327493131 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f311006b4d0 0x7f311006d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f312c0c09a0 tx=0x7f3124009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.019+0000 7f3121ffb640 1 -- 172.21.15.1:0/2327493131 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f312c14a4b0 con 0x7f312c0c2b30 2024-03-06T07:22:43.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.200+0000 7f31339b6640 1 -- 172.21.15.1:0/2327493131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f312c071790 con 0x7f312c0c2b30 2024-03-06T07:22:43.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.201+0000 7f3121ffb640 1 -- 172.21.15.1:0/2327493131 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+17279 (secure 0 0 0) 0x7f311c07a450 con 0x7f312c0c2b30 2024-03-06T07:22:43.203 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:43.203 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":62,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:41.282730+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:22:18.907422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:22:20.028174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/3845300501":"2024-03-07T07:22:38.960489+0000","172.21.15.1:6834/4244709430":"2024-03-07T07:22:37.955221+0000","172.21.15.1:6832/4244709430":"2024-03-07T07:22:37.955221+0000","172.21.15.98:6836/2937068915":"2024-03-07T07:22:29.247312+0000","172.21.15.1:6835/3845300501":"2024-03-07T07:22:38.960489+0000","172.21.15.98:6834/2937068915":"2024-03-07T07:22:29.247312+0000","172.21.15.1:6834/131367647":"2024-03-07T07:22:26.740110+0000","172.21.15.1:6832/131367647":"2024-03-07T07:22:26.740110+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.203+0000 7f31339b6640 1 -- 172.21.15.1:0/2327493131 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f311006b4d0 msgr2=0x7f311006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.203+0000 7f31339b6640 1 --2- 172.21.15.1:0/2327493131 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f311006b4d0 0x7f311006d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f312c0c09a0 tx=0x7f3124009040 comp rx=0 tx=0).stop 2024-03-06T07:22:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.203+0000 7f31339b6640 1 -- 172.21.15.1:0/2327493131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f312c0c2b30 msgr2=0x7f312c14fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.203+0000 7f31339b6640 1 --2- 172.21.15.1:0/2327493131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f312c0c2b30 0x7f312c14fb00 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f311c009890 tx=0x7f311c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.203+0000 7f31339b6640 1 -- 172.21.15.1:0/2327493131 shutdown_connections 2024-03-06T07:22:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.203+0000 7f31339b6640 1 --2- 172.21.15.1:0/2327493131 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f311006b4d0 0x7f311006d990 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.203+0000 7f31339b6640 1 --2- 172.21.15.1:0/2327493131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f312c152b70 0x7f312c14e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.203+0000 7f31339b6640 1 --2- 172.21.15.1:0/2327493131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f312c0c2b30 0x7f312c14fb00 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.203+0000 7f31339b6640 1 --2- 172.21.15.1:0/2327493131 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f312c072bd0 0x7f312c149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.203+0000 7f31339b6640 1 -- 172.21.15.1:0/2327493131 >> 172.21.15.1:0/2327493131 conn(0x7f312c0c6860 msgr2=0x7f312c150ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.203+0000 7f31339b6640 1 -- 172.21.15.1:0/2327493131 shutdown_connections 2024-03-06T07:22:43.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.203+0000 7f31339b6640 1 -- 172.21.15.1:0/2327493131 wait complete. 2024-03-06T07:22:43.216 INFO:tasks.cephfs.filesystem:Destroying file system cephfs2 and related pools 2024-03-06T07:22:43.216 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.317+0000 7f38762df640 1 Processor -- start 2024-03-06T07:22:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.317+0000 7f38762df640 1 -- start start 2024-03-06T07:22:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.317+0000 7f38762df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3870152690 0x7f3870155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.317+0000 7f38762df640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3870156cd0 0x7f3870157160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.317+0000 7f38762df640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38701561e0 con 0x7f38700c6730 2024-03-06T07:22:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.317+0000 7f38762df640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38700c3300 con 0x7f3870156cd0 2024-03-06T07:22:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.317+0000 7f38762df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38700c35a0 con 0x7f3870152690 2024-03-06T07:22:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.318+0000 7f387487c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3870156cd0 0x7f3870157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.318+0000 7f386f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f38700c6730 0x7f38700c57d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37880/0 (socket says 172.21.15.1:37880) 2024-03-06T07:22:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.318+0000 7f386f7fe640 1 -- 172.21.15.1:0/3171693915 learned_addr learned my addr 172.21.15.1:0/3171693915 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.318+0000 7f386ffff640 1 --2- 172.21.15.1:0/3171693915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3870152690 0x7f3870155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.318+0000 7f387487c640 1 -- 172.21.15.1:0/3171693915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3870152690 msgr2=0x7f3870155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.318+0000 7f387487c640 1 --2- 172.21.15.1:0/3171693915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3870152690 0x7f3870155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.318+0000 7f387487c640 1 -- 172.21.15.1:0/3171693915 >> v1:172.21.15.1:6789/0 conn(0x7f38700c6730 legacy=0x7f38700c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.318+0000 7f387487c640 1 -- 172.21.15.1:0/3171693915 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38700c3880 con 0x7f3870156cd0 2024-03-06T07:22:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.318+0000 7f386effd640 1 -- 172.21.15.1:0/3171693915 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563842694 0 0) 0x7f3864001110 con 0x7f38700c6730 2024-03-06T07:22:43.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.319+0000 7f387487c640 1 --2- 172.21.15.1:0/3171693915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3870156cd0 0x7f3870157160 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f3868009d90 tx=0x7f386802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c64ddf682bd827f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.319+0000 7f386effd640 1 -- 172.21.15.1:0/3171693915 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3868039070 con 0x7f3870156cd0 2024-03-06T07:22:43.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.319+0000 7f386effd640 1 -- 172.21.15.1:0/3171693915 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386802cbc0 con 0x7f3870156cd0 2024-03-06T07:22:43.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.319+0000 7f386effd640 1 -- 172.21.15.1:0/3171693915 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3868034440 con 0x7f3870156cd0 2024-03-06T07:22:43.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.319+0000 7f38762df640 1 -- 172.21.15.1:0/3171693915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3870156cd0 msgr2=0x7f3870157160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.319+0000 7f38762df640 1 --2- 172.21.15.1:0/3171693915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3870156cd0 0x7f3870157160 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f3868009d90 tx=0x7f386802be10 comp rx=0 tx=0).stop 2024-03-06T07:22:43.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.319+0000 7f38762df640 1 -- 172.21.15.1:0/3171693915 shutdown_connections 2024-03-06T07:22:43.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.319+0000 7f38762df640 1 --2- 172.21.15.1:0/3171693915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3870156cd0 0x7f3870157160 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.319+0000 7f38762df640 1 --2- 172.21.15.1:0/3171693915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3870152690 0x7f3870155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.319+0000 7f38762df640 1 -- 172.21.15.1:0/3171693915 >> 172.21.15.1:0/3171693915 conn(0x7f387006b410 msgr2=0x7f387006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.319+0000 7f38762df640 1 -- 172.21.15.1:0/3171693915 shutdown_connections 2024-03-06T07:22:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.319+0000 7f38762df640 1 -- 172.21.15.1:0/3171693915 wait complete. 2024-03-06T07:22:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.320+0000 7f38762df640 1 Processor -- start 2024-03-06T07:22:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.321+0000 7f38762df640 1 -- start start 2024-03-06T07:22:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.321+0000 7f38762df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38700c6730 0x7f38700c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.321+0000 7f38762df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3870152690 0x7f38701ee990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.321+0000 7f38762df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3870156cd0 0x7f38701eeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.321+0000 7f38762df640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38700c5ee0 con 0x7f38700c6730 2024-03-06T07:22:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.321+0000 7f38762df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38701567b0 con 0x7f3870152690 2024-03-06T07:22:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.321+0000 7f38762df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38700c6060 con 0x7f3870156cd0 2024-03-06T07:22:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.321+0000 7f386ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38700c6730 0x7f38700c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.321+0000 7f386ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38700c6730 0x7f38700c82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57348/0 (socket says 172.21.15.1:57348) 2024-03-06T07:22:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.321+0000 7f386ffff640 1 -- 172.21.15.1:0/2654166928 learned_addr learned my addr 172.21.15.1:0/2654166928 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.321+0000 7f387487c640 1 --2- 172.21.15.1:0/2654166928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3870156cd0 0x7f38701eeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.322+0000 7f386f7fe640 1 --2- 172.21.15.1:0/2654166928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3870152690 0x7f38701ee990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.322+0000 7f386ffff640 1 -- 172.21.15.1:0/2654166928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3870156cd0 msgr2=0x7f38701eeed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.322+0000 7f386ffff640 1 --2- 172.21.15.1:0/2654166928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3870156cd0 0x7f38701eeed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.322+0000 7f386ffff640 1 -- 172.21.15.1:0/2654166928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3870152690 msgr2=0x7f38701ee990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.322+0000 7f386ffff640 1 --2- 172.21.15.1:0/2654166928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3870152690 0x7f38701ee990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.322+0000 7f386ffff640 1 -- 172.21.15.1:0/2654166928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387006d4d0 con 0x7f38700c6730 2024-03-06T07:22:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.322+0000 7f386ffff640 1 --2- 172.21.15.1:0/2654166928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38700c6730 0x7f38700c82d0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f386000e8a0 tx=0x7f386000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.322+0000 7f386cff9640 1 -- 172.21.15.1:0/2654166928 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3860017070 con 0x7f38700c6730 2024-03-06T07:22:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.322+0000 7f386cff9640 1 -- 172.21.15.1:0/2654166928 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386000c940 con 0x7f38700c6730 2024-03-06T07:22:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.322+0000 7f386cff9640 1 -- 172.21.15.1:0/2654166928 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386000cd30 con 0x7f38700c6730 2024-03-06T07:22:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.322+0000 7f38762df640 1 -- 172.21.15.1:0/2654166928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38700ad9a0 con 0x7f38700c6730 2024-03-06T07:22:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.322+0000 7f38762df640 1 -- 172.21.15.1:0/2654166928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f387014dae0 con 0x7f38700c6730 2024-03-06T07:22:43.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.323+0000 7f38762df640 1 -- 172.21.15.1:0/2654166928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3870067fd0 con 0x7f38700c6730 2024-03-06T07:22:43.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.324+0000 7f386cff9640 1 -- 172.21.15.1:0/2654166928 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f386001a070 con 0x7f38700c6730 2024-03-06T07:22:43.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.324+0000 7f386cff9640 1 --2- 172.21.15.1:0/2654166928 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f385c06b4d0 0x7f385c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.324+0000 7f386cff9640 1 -- 172.21.15.1:0/2654166928 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f386008a220 con 0x7f38700c6730 2024-03-06T07:22:43.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.324+0000 7f386f7fe640 1 --2- 172.21.15.1:0/2654166928 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f385c06b4d0 0x7f385c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.325+0000 7f386f7fe640 1 --2- 172.21.15.1:0/2654166928 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f385c06b4d0 0x7f385c06d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f38700c8030 tx=0x7f3864007de0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.328+0000 7f386cff9640 1 -- 172.21.15.1:0/2654166928 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f387006d4d0 con 0x7f38700c6730 2024-03-06T07:22:43.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.545+0000 7f38762df640 1 -- 172.21.15.1:0/2654166928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38700c2c40 con 0x7f38700c6730 2024-03-06T07:22:43.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.545+0000 7f386cff9640 1 -- 172.21.15.1:0/2654166928 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f38600587f0 con 0x7f38700c6730 2024-03-06T07:22:43.548 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:43.548 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":48,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2604497680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2604497680},{"type":"v1","addr":"172.21.15.98:6836","nonce":2604497680}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/1450734721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1450734721},{"type":"v1","addr":"172.21.15.1:6834","nonce":1450734721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4269513324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4269513324},{"type":"v1","addr":"172.21.15.1:6835","nonce":4269513324}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:21.047528+0000","modified":"2024-03-06T07:22:41.965083+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5087":{"gid":5087,"name":"d","rank":0,"incarnation":45,"state":"up:active","state_seq":13,"addr":"172.21.15.98:6837/2828783606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2828783606},{"type":"v1","addr":"172.21.15.98:6837","nonce":2828783606}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5087,"qdb_cluster":[5087]},"id":4}]} 2024-03-06T07:22:43.548 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 48 2024-03-06T07:22:43.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.547+0000 7f38762df640 1 -- 172.21.15.1:0/2654166928 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f385c06b4d0 msgr2=0x7f385c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.547+0000 7f38762df640 1 --2- 172.21.15.1:0/2654166928 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f385c06b4d0 0x7f385c06d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f38700c8030 tx=0x7f3864007de0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.547+0000 7f38762df640 1 -- 172.21.15.1:0/2654166928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38700c6730 msgr2=0x7f38700c82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.547+0000 7f38762df640 1 --2- 172.21.15.1:0/2654166928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38700c6730 0x7f38700c82d0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f386000e8a0 tx=0x7f386000ed60 comp rx=0 tx=0).stop 2024-03-06T07:22:43.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.547+0000 7f38762df640 1 -- 172.21.15.1:0/2654166928 shutdown_connections 2024-03-06T07:22:43.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.547+0000 7f38762df640 1 --2- 172.21.15.1:0/2654166928 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f385c06b4d0 0x7f385c06d990 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.547+0000 7f38762df640 1 --2- 172.21.15.1:0/2654166928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3870156cd0 0x7f38701eeed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.547+0000 7f38762df640 1 --2- 172.21.15.1:0/2654166928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3870152690 0x7f38701ee990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.547+0000 7f38762df640 1 --2- 172.21.15.1:0/2654166928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38700c6730 0x7f38700c82d0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.547+0000 7f38762df640 1 -- 172.21.15.1:0/2654166928 >> 172.21.15.1:0/2654166928 conn(0x7f387006b410 msgr2=0x7f3870153850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.547+0000 7f38762df640 1 -- 172.21.15.1:0/2654166928 shutdown_connections 2024-03-06T07:22:43.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.547+0000 7f38762df640 1 -- 172.21.15.1:0/2654166928 wait complete. 2024-03-06T07:22:43.560 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:43.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.660+0000 7fb83c57a640 1 Processor -- start 2024-03-06T07:22:43.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.660+0000 7fb83c57a640 1 -- start start 2024-03-06T07:22:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.661+0000 7fb83c57a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb8340c5850 0x7fb8340c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.661+0000 7fb83c57a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb834155220 0x7fb8341556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.661+0000 7fb83c57a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb834158910 con 0x7fb834152920 2024-03-06T07:22:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.661+0000 7fb83c57a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb834152d00 con 0x7fb8340c5850 2024-03-06T07:22:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.661+0000 7fb83c57a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8341596d0 con 0x7fb834155220 2024-03-06T07:22:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.661+0000 7fb83a316640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb834152920 0x7fb83406aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37892/0 (socket says 172.21.15.1:37892) 2024-03-06T07:22:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.661+0000 7fb83a316640 1 -- 172.21.15.1:0/400184919 learned_addr learned my addr 172.21.15.1:0/400184919 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.661+0000 7fb83ab17640 1 --2- 172.21.15.1:0/400184919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb834155220 0x7fb8341556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.661+0000 7fb839b15640 1 --2- 172.21.15.1:0/400184919 >> v2:172.21.15.98:3300/0 conn(0x7fb8340c5850 0x7fb8340c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.661+0000 7fb839314640 1 -- 172.21.15.1:0/400184919 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695502170 0 0) 0x7fb834158910 con 0x7fb834152920 2024-03-06T07:22:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.661+0000 7fb839314640 1 -- 172.21.15.1:0/400184919 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb820003520 con 0x7fb834152920 2024-03-06T07:22:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.661+0000 7fb83ab17640 1 -- 172.21.15.1:0/400184919 >> v2:172.21.15.98:3300/0 conn(0x7fb8340c5850 msgr2=0x7fb8340c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.661+0000 7fb83ab17640 1 --2- 172.21.15.1:0/400184919 >> v2:172.21.15.98:3300/0 conn(0x7fb8340c5850 0x7fb8340c5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.661+0000 7fb83ab17640 1 -- 172.21.15.1:0/400184919 >> v1:172.21.15.1:6789/0 conn(0x7fb834152920 legacy=0x7fb83406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.661+0000 7fb83ab17640 1 -- 172.21.15.1:0/400184919 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb834158bf0 con 0x7fb834155220 2024-03-06T07:22:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.661+0000 7fb839b15640 1 --2- 172.21.15.1:0/400184919 >> v2:172.21.15.98:3300/0 conn(0x7fb8340c5850 0x7fb8340c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.662+0000 7fb83ab17640 1 --2- 172.21.15.1:0/400184919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb834155220 0x7fb8341556b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb82c009d90 tx=0x7fb82c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=548703da860149f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.662+0000 7fb839314640 1 -- 172.21.15.1:0/400184919 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c039070 con 0x7fb834155220 2024-03-06T07:22:43.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.662+0000 7fb839314640 1 -- 172.21.15.1:0/400184919 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c02cbc0 con 0x7fb834155220 2024-03-06T07:22:43.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.662+0000 7fb839314640 1 -- 172.21.15.1:0/400184919 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c034440 con 0x7fb834155220 2024-03-06T07:22:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.663+0000 7fb83c57a640 1 -- 172.21.15.1:0/400184919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb834155220 msgr2=0x7fb8341556b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.663+0000 7fb83c57a640 1 --2- 172.21.15.1:0/400184919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb834155220 0x7fb8341556b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb82c009d90 tx=0x7fb82c02be10 comp rx=0 tx=0).stop 2024-03-06T07:22:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.663+0000 7fb83c57a640 1 -- 172.21.15.1:0/400184919 shutdown_connections 2024-03-06T07:22:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.663+0000 7fb83c57a640 1 --2- 172.21.15.1:0/400184919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb834155220 0x7fb8341556b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.663+0000 7fb83c57a640 1 --2- 172.21.15.1:0/400184919 >> v2:172.21.15.98:3300/0 conn(0x7fb8340c5850 0x7fb8340c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.663+0000 7fb83c57a640 1 -- 172.21.15.1:0/400184919 >> 172.21.15.1:0/400184919 conn(0x7fb8340a9700 msgr2=0x7fb8340a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.663+0000 7fb83c57a640 1 -- 172.21.15.1:0/400184919 shutdown_connections 2024-03-06T07:22:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.663+0000 7fb83c57a640 1 -- 172.21.15.1:0/400184919 wait complete. 2024-03-06T07:22:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.664+0000 7fb83c57a640 1 Processor -- start 2024-03-06T07:22:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.664+0000 7fb83c57a640 1 -- start start 2024-03-06T07:22:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.664+0000 7fb83c57a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8340c5850 0x7fb83406e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.664+0000 7fb83c57a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb834152920 0x7fb83406fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.664+0000 7fb83c57a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb834155220 0x7fb834151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.664+0000 7fb83c57a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8340c6380 con 0x7fb8340c5850 2024-03-06T07:22:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.664+0000 7fb83c57a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb834159240 con 0x7fb834152920 2024-03-06T07:22:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.664+0000 7fb83c57a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83406b700 con 0x7fb834155220 2024-03-06T07:22:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.665+0000 7fb83a316640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8340c5850 0x7fb83406e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.665+0000 7fb83a316640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8340c5850 0x7fb83406e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57362/0 (socket says 172.21.15.1:57362) 2024-03-06T07:22:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.665+0000 7fb83a316640 1 -- 172.21.15.1:0/2367676009 learned_addr learned my addr 172.21.15.1:0/2367676009 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.665+0000 7fb83ab17640 1 --2- 172.21.15.1:0/2367676009 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb834155220 0x7fb834151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.665+0000 7fb839b15640 1 --2- 172.21.15.1:0/2367676009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb834152920 0x7fb83406fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.665+0000 7fb83a316640 1 -- 172.21.15.1:0/2367676009 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb834155220 msgr2=0x7fb834151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.665+0000 7fb83a316640 1 --2- 172.21.15.1:0/2367676009 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb834155220 0x7fb834151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.665+0000 7fb83a316640 1 -- 172.21.15.1:0/2367676009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb834152920 msgr2=0x7fb83406fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.665+0000 7fb83a316640 1 --2- 172.21.15.1:0/2367676009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb834152920 0x7fb83406fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.665+0000 7fb83a316640 1 -- 172.21.15.1:0/2367676009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb834152100 con 0x7fb8340c5850 2024-03-06T07:22:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.665+0000 7fb83ab17640 1 --2- 172.21.15.1:0/2367676009 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb834155220 0x7fb834151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.665+0000 7fb83a316640 1 --2- 172.21.15.1:0/2367676009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8340c5850 0x7fb83406e3b0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb83000c430 tx=0x7fb83000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.665+0000 7fb82affd640 1 -- 172.21.15.1:0/2367676009 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb830017070 con 0x7fb8340c5850 2024-03-06T07:22:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.665+0000 7fb82affd640 1 -- 172.21.15.1:0/2367676009 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb830005ce0 con 0x7fb8340c5850 2024-03-06T07:22:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.665+0000 7fb82affd640 1 -- 172.21.15.1:0/2367676009 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8300072f0 con 0x7fb8340c5850 2024-03-06T07:22:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.665+0000 7fb83c57a640 1 -- 172.21.15.1:0/2367676009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83406ea40 con 0x7fb8340c5850 2024-03-06T07:22:43.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.665+0000 7fb83c57a640 1 -- 172.21.15.1:0/2367676009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb83406f030 con 0x7fb8340c5850 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.666+0000 7fb83c57a640 1 -- 172.21.15.1:0/2367676009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb834067fd0 con 0x7fb8340c5850 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.667+0000 7fb82affd640 1 -- 172.21.15.1:0/2367676009 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb83001a070 con 0x7fb8340c5850 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.667+0000 7fb82affd640 1 --2- 172.21.15.1:0/2367676009 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb81806b500 0x7fb81806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.667+0000 7fb82affd640 1 -- 172.21.15.1:0/2367676009 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7fb830089840 con 0x7fb8340c5850 2024-03-06T07:22:43.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.667+0000 7fb839b15640 1 --2- 172.21.15.1:0/2367676009 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb81806b500 0x7fb81806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.668+0000 7fb839b15640 1 --2- 172.21.15.1:0/2367676009 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb81806b500 0x7fb81806d9c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb824009560 tx=0x7fb824009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.672+0000 7fb82affd640 1 -- 172.21.15.1:0/2367676009 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb830057e10 con 0x7fb8340c5850 2024-03-06T07:22:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.891+0000 7fb83c57a640 1 -- 172.21.15.1:0/2367676009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb83406bdf0 con 0x7fb8340c5850 2024-03-06T07:22:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.892+0000 7fb82affd640 1 -- 172.21.15.1:0/2367676009 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fb83005beb0 con 0x7fb8340c5850 2024-03-06T07:22:43.894 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:43.894 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":48,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2604497680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2604497680},{"type":"v1","addr":"172.21.15.98:6836","nonce":2604497680}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/1450734721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1450734721},{"type":"v1","addr":"172.21.15.1:6834","nonce":1450734721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4269513324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4269513324},{"type":"v1","addr":"172.21.15.1:6835","nonce":4269513324}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:21.047528+0000","modified":"2024-03-06T07:22:41.965083+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5087":{"gid":5087,"name":"d","rank":0,"incarnation":45,"state":"up:active","state_seq":13,"addr":"172.21.15.98:6837/2828783606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2828783606},{"type":"v1","addr":"172.21.15.98:6837","nonce":2828783606}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5087,"qdb_cluster":[5087]},"id":4}]} 2024-03-06T07:22:43.894 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 48 2024-03-06T07:22:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.893+0000 7fb83c57a640 1 -- 172.21.15.1:0/2367676009 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb81806b500 msgr2=0x7fb81806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.893+0000 7fb83c57a640 1 --2- 172.21.15.1:0/2367676009 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb81806b500 0x7fb81806d9c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb824009560 tx=0x7fb824009190 comp rx=0 tx=0).stop 2024-03-06T07:22:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.893+0000 7fb83c57a640 1 -- 172.21.15.1:0/2367676009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8340c5850 msgr2=0x7fb83406e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.893+0000 7fb83c57a640 1 --2- 172.21.15.1:0/2367676009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8340c5850 0x7fb83406e3b0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb83000c430 tx=0x7fb83000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.893+0000 7fb83c57a640 1 -- 172.21.15.1:0/2367676009 shutdown_connections 2024-03-06T07:22:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.893+0000 7fb83c57a640 1 --2- 172.21.15.1:0/2367676009 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb81806b500 0x7fb81806d9c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.893+0000 7fb83c57a640 1 --2- 172.21.15.1:0/2367676009 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb834155220 0x7fb834151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.893+0000 7fb83c57a640 1 --2- 172.21.15.1:0/2367676009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb834152920 0x7fb83406fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.893+0000 7fb83c57a640 1 --2- 172.21.15.1:0/2367676009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8340c5850 0x7fb83406e3b0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.893+0000 7fb83c57a640 1 -- 172.21.15.1:0/2367676009 >> 172.21.15.1:0/2367676009 conn(0x7fb8340a9700 msgr2=0x7fb83414aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.894+0000 7fb83c57a640 1 -- 172.21.15.1:0/2367676009 shutdown_connections 2024-03-06T07:22:43.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:43.894+0000 7fb83c57a640 1 -- 172.21.15.1:0/2367676009 wait complete. 2024-03-06T07:22:43.906 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:44.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.005+0000 7ff3f0876640 1 Processor -- start 2024-03-06T07:22:44.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.005+0000 7ff3f0876640 1 -- start start 2024-03-06T07:22:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.006+0000 7ff3f0876640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3ec0c39c0 0x7ff3ec0ac220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.006+0000 7ff3f0876640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec0c7420 con 0x7ff3ec0c4940 2024-03-06T07:22:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.006+0000 7ff3f0876640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec0ab760 con 0x7ff3ec0c4560 2024-03-06T07:22:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.006+0000 7ff3f0876640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec0c76c0 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.006+0000 7ff3ead9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff3ec0c4940 0x7ff3ec0ac760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37908/0 (socket says 172.21.15.1:37908) 2024-03-06T07:22:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.006+0000 7ff3ead9d640 1 -- 172.21.15.1:0/2579286939 learned_addr learned my addr 172.21.15.1:0/2579286939 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.006+0000 7ff3e9d9b640 1 --2- 172.21.15.1:0/2579286939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3ec0c39c0 0x7ff3ec0ac220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.006+0000 7ff3e959a640 1 -- 172.21.15.1:0/2579286939 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304088596 0 0) 0x7ff3ec0c7420 con 0x7ff3ec0c4940 2024-03-06T07:22:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.007+0000 7ff3e959a640 1 -- 172.21.15.1:0/2579286939 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d8003520 con 0x7ff3ec0c4940 2024-03-06T07:22:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.007+0000 7ff3e959a640 1 -- 172.21.15.1:0/2579286939 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068948688 0 0) 0x7ff3ec0ab760 con 0x7ff3ec0c4560 2024-03-06T07:22:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.007+0000 7ff3e959a640 1 -- 172.21.15.1:0/2579286939 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ec0c7420 con 0x7ff3ec0c4560 2024-03-06T07:22:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.007+0000 7ff3e9d9b640 1 -- 172.21.15.1:0/2579286939 >> v1:172.21.15.98:6789/0 conn(0x7ff3ec0c4560 legacy=0x7ff3ec155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.007+0000 7ff3e9d9b640 1 -- 172.21.15.1:0/2579286939 >> v1:172.21.15.1:6789/0 conn(0x7ff3ec0c4940 legacy=0x7ff3ec0ac760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.007+0000 7ff3e9d9b640 1 -- 172.21.15.1:0/2579286939 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec0aba20 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.007+0000 7ff3e9d9b640 1 --2- 172.21.15.1:0/2579286939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3ec0c39c0 0x7ff3ec0ac220 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff3d40097d0 tx=0x7ff3d402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1269a5096200f9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.008+0000 7ff3e959a640 1 -- 172.21.15.1:0/2579286939 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4035020 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.008+0000 7ff3e959a640 1 -- 172.21.15.1:0/2579286939 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d4031890 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.008+0000 7ff3e959a640 1 -- 172.21.15.1:0/2579286939 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4031c60 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.008+0000 7ff3f0876640 1 -- 172.21.15.1:0/2579286939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3ec0c39c0 msgr2=0x7ff3ec0ac220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.008+0000 7ff3f0876640 1 --2- 172.21.15.1:0/2579286939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3ec0c39c0 0x7ff3ec0ac220 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff3d40097d0 tx=0x7ff3d402b930 comp rx=0 tx=0).stop 2024-03-06T07:22:44.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.008+0000 7ff3f0876640 1 -- 172.21.15.1:0/2579286939 shutdown_connections 2024-03-06T07:22:44.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.008+0000 7ff3f0876640 1 --2- 172.21.15.1:0/2579286939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3ec0c39c0 0x7ff3ec0ac220 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.008+0000 7ff3f0876640 1 -- 172.21.15.1:0/2579286939 >> 172.21.15.1:0/2579286939 conn(0x7ff3ec0c7a50 msgr2=0x7ff3ec0c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.008+0000 7ff3f0876640 1 -- 172.21.15.1:0/2579286939 shutdown_connections 2024-03-06T07:22:44.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.008+0000 7ff3f0876640 1 -- 172.21.15.1:0/2579286939 wait complete. 2024-03-06T07:22:44.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.009+0000 7ff3f0876640 1 Processor -- start 2024-03-06T07:22:44.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.010+0000 7ff3f0876640 1 -- start start 2024-03-06T07:22:44.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.010+0000 7ff3f0876640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3ec0c39c0 0x7ff3ec14af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.010+0000 7ff3f0876640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3ec0c4560 0x7ff3ec1eec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.010+0000 7ff3f0876640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3ec0c4940 0x7ff3ec06fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.010+0000 7ff3f0876640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec0acff0 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.010+0000 7ff3f0876640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec0ace70 con 0x7ff3ec0c4560 2024-03-06T07:22:44.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.010+0000 7ff3f0876640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec06e6d0 con 0x7ff3ec0c4940 2024-03-06T07:22:44.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.010+0000 7ff3ea59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3ec0c39c0 0x7ff3ec14af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.010+0000 7ff3ea59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3ec0c39c0 0x7ff3ec14af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57372/0 (socket says 172.21.15.1:57372) 2024-03-06T07:22:44.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.010+0000 7ff3ea59c640 1 -- 172.21.15.1:0/616969398 learned_addr learned my addr 172.21.15.1:0/616969398 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:44.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.010+0000 7ff3e9d9b640 1 --2- 172.21.15.1:0/616969398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3ec0c4560 0x7ff3ec1eec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.010+0000 7ff3ead9d640 1 --2- 172.21.15.1:0/616969398 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3ec0c4940 0x7ff3ec06fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.011+0000 7ff3ea59c640 1 -- 172.21.15.1:0/616969398 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3ec0c4940 msgr2=0x7ff3ec06fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.011+0000 7ff3ea59c640 1 --2- 172.21.15.1:0/616969398 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3ec0c4940 0x7ff3ec06fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.011+0000 7ff3ea59c640 1 -- 172.21.15.1:0/616969398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3ec0c4560 msgr2=0x7ff3ec1eec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.011+0000 7ff3ea59c640 1 --2- 172.21.15.1:0/616969398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3ec0c4560 0x7ff3ec1eec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.011+0000 7ff3ea59c640 1 -- 172.21.15.1:0/616969398 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec1eac80 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.011+0000 7ff3ea59c640 1 --2- 172.21.15.1:0/616969398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3ec0c39c0 0x7ff3ec14af20 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff3e400c530 tx=0x7ff3e400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.011+0000 7ff3e2ffd640 1 -- 172.21.15.1:0/616969398 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e40052d0 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.011+0000 7ff3e2ffd640 1 -- 172.21.15.1:0/616969398 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e4005490 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.011+0000 7ff3f0876640 1 -- 172.21.15.1:0/616969398 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ec15edf0 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.011+0000 7ff3e2ffd640 1 -- 172.21.15.1:0/616969398 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e4012870 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.011+0000 7ff3f0876640 1 -- 172.21.15.1:0/616969398 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ec15f3b0 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.012+0000 7ff3f0876640 1 -- 172.21.15.1:0/616969398 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ec0686e0 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.013+0000 7ff3e2ffd640 1 -- 172.21.15.1:0/616969398 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3e4012a10 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.013+0000 7ff3e2ffd640 1 --2- 172.21.15.1:0/616969398 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff3cc06b4d0 0x7ff3cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.013+0000 7ff3e2ffd640 1 -- 172.21.15.1:0/616969398 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7ff3e4089fb0 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.013+0000 7ff3e9d9b640 1 --2- 172.21.15.1:0/616969398 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff3cc06b4d0 0x7ff3cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.014+0000 7ff3e9d9b640 1 --2- 172.21.15.1:0/616969398 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff3cc06b4d0 0x7ff3cc06d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4036000 tx=0x7ff3d4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.017+0000 7ff3e2ffd640 1 -- 172.21.15.1:0/616969398 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3e4058500 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.200+0000 7ff3f0876640 1 -- 172.21.15.1:0/616969398 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3ec0c6180 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.201+0000 7ff3e2ffd640 1 -- 172.21.15.1:0/616969398 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+17279 (secure 0 0 0) 0x7ff3e405c5a0 con 0x7ff3ec0c39c0 2024-03-06T07:22:44.203 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:44.203 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":62,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:41.282730+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:22:18.907422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:22:20.028174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/3845300501":"2024-03-07T07:22:38.960489+0000","172.21.15.1:6834/4244709430":"2024-03-07T07:22:37.955221+0000","172.21.15.1:6832/4244709430":"2024-03-07T07:22:37.955221+0000","172.21.15.98:6836/2937068915":"2024-03-07T07:22:29.247312+0000","172.21.15.1:6835/3845300501":"2024-03-07T07:22:38.960489+0000","172.21.15.98:6834/2937068915":"2024-03-07T07:22:29.247312+0000","172.21.15.1:6834/131367647":"2024-03-07T07:22:26.740110+0000","172.21.15.1:6832/131367647":"2024-03-07T07:22:26.740110+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.203+0000 7ff3f0876640 1 -- 172.21.15.1:0/616969398 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff3cc06b4d0 msgr2=0x7ff3cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.203+0000 7ff3f0876640 1 --2- 172.21.15.1:0/616969398 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff3cc06b4d0 0x7ff3cc06d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4036000 tx=0x7ff3d4008030 comp rx=0 tx=0).stop 2024-03-06T07:22:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.203+0000 7ff3f0876640 1 -- 172.21.15.1:0/616969398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3ec0c39c0 msgr2=0x7ff3ec14af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.203+0000 7ff3f0876640 1 --2- 172.21.15.1:0/616969398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3ec0c39c0 0x7ff3ec14af20 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff3e400c530 tx=0x7ff3e400c9f0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.203+0000 7ff3f0876640 1 -- 172.21.15.1:0/616969398 shutdown_connections 2024-03-06T07:22:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.203+0000 7ff3f0876640 1 --2- 172.21.15.1:0/616969398 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff3cc06b4d0 0x7ff3cc06d990 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.203+0000 7ff3f0876640 1 --2- 172.21.15.1:0/616969398 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3ec0c4940 0x7ff3ec06fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.203+0000 7ff3f0876640 1 --2- 172.21.15.1:0/616969398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3ec0c4560 0x7ff3ec1eec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.203+0000 7ff3f0876640 1 --2- 172.21.15.1:0/616969398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3ec0c39c0 0x7ff3ec14af20 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.203+0000 7ff3f0876640 1 -- 172.21.15.1:0/616969398 >> 172.21.15.1:0/616969398 conn(0x7ff3ec0c7a50 msgr2=0x7ff3ec0c63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.203+0000 7ff3f0876640 1 -- 172.21.15.1:0/616969398 shutdown_connections 2024-03-06T07:22:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.203+0000 7ff3f0876640 1 -- 172.21.15.1:0/616969398 wait complete. 2024-03-06T07:22:44.216 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs2 2024-03-06T07:22:44.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.318+0000 7f4cd157a640 1 Processor -- start 2024-03-06T07:22:44.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.318+0000 7f4cd157a640 1 -- start start 2024-03-06T07:22:44.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.319+0000 7f4cd157a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4ccc0a7f50 0x7f4ccc154300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.319+0000 7f4cd157a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4ccc0c75a0 0x7f4ccc0c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.319+0000 7f4cd157a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ccc0c8120 con 0x7f4ccc150640 2024-03-06T07:22:44.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.319+0000 7f4cd157a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ccc153e30 con 0x7f4ccc0c75a0 2024-03-06T07:22:44.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.319+0000 7f4cd157a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc0c7b60 con 0x7f4ccc0a7f50 2024-03-06T07:22:44.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.319+0000 7f4ccaffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4ccc150640 0x7f4ccc153720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37920/0 (socket says 172.21.15.1:37920) 2024-03-06T07:22:44.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.319+0000 7f4ccaffd640 1 -- 172.21.15.1:0/2741606889 learned_addr learned my addr 172.21.15.1:0/2741606889 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:44.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.319+0000 7f4cc3fff640 1 --2- 172.21.15.1:0/2741606889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4ccc0a7f50 0x7f4ccc154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.319+0000 7f4ccb7fe640 1 --2- 172.21.15.1:0/2741606889 >> v2:172.21.15.98:3300/0 conn(0x7f4ccc0c75a0 0x7f4ccc0c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.319+0000 7f4cca7fc640 1 -- 172.21.15.1:0/2741606889 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913009668 0 0) 0x7f4ccc0c8120 con 0x7f4ccc150640 2024-03-06T07:22:44.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.319+0000 7f4cca7fc640 1 -- 172.21.15.1:0/2741606889 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb8003520 con 0x7f4ccc150640 2024-03-06T07:22:44.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.319+0000 7f4ccb7fe640 1 -- 172.21.15.1:0/2741606889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4ccc0a7f50 msgr2=0x7f4ccc154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.319+0000 7f4ccb7fe640 1 --2- 172.21.15.1:0/2741606889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4ccc0a7f50 0x7f4ccc154300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.319+0000 7f4ccb7fe640 1 -- 172.21.15.1:0/2741606889 >> v1:172.21.15.1:6789/0 conn(0x7f4ccc150640 legacy=0x7f4ccc153720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.319+0000 7f4ccb7fe640 1 -- 172.21.15.1:0/2741606889 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ccc0c7e40 con 0x7f4ccc0c75a0 2024-03-06T07:22:44.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.319+0000 7f4cc3fff640 1 --2- 172.21.15.1:0/2741606889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4ccc0a7f50 0x7f4ccc154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:44.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.320+0000 7f4ccb7fe640 1 --2- 172.21.15.1:0/2741606889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ccc0c75a0 0x7f4ccc0c8d60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4009d90 tx=0x7f4cc402bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44a36d227817807a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.320+0000 7f4cca7fc640 1 -- 172.21.15.1:0/2741606889 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4039070 con 0x7f4ccc0c75a0 2024-03-06T07:22:44.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.320+0000 7f4cca7fc640 1 -- 172.21.15.1:0/2741606889 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc402cb30 con 0x7f4ccc0c75a0 2024-03-06T07:22:44.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.320+0000 7f4cca7fc640 1 -- 172.21.15.1:0/2741606889 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4034440 con 0x7f4ccc0c75a0 2024-03-06T07:22:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.321+0000 7f4cd157a640 1 -- 172.21.15.1:0/2741606889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ccc0c75a0 msgr2=0x7f4ccc0c8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.321+0000 7f4cd157a640 1 --2- 172.21.15.1:0/2741606889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ccc0c75a0 0x7f4ccc0c8d60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4009d90 tx=0x7f4cc402bd80 comp rx=0 tx=0).stop 2024-03-06T07:22:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.321+0000 7f4cd157a640 1 -- 172.21.15.1:0/2741606889 shutdown_connections 2024-03-06T07:22:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.321+0000 7f4cd157a640 1 --2- 172.21.15.1:0/2741606889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ccc0c75a0 0x7f4ccc0c8d60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.321+0000 7f4cd157a640 1 --2- 172.21.15.1:0/2741606889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4ccc0a7f50 0x7f4ccc154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.321+0000 7f4cd157a640 1 -- 172.21.15.1:0/2741606889 >> 172.21.15.1:0/2741606889 conn(0x7f4ccc06b3e0 msgr2=0x7f4ccc06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.321+0000 7f4cd157a640 1 -- 172.21.15.1:0/2741606889 shutdown_connections 2024-03-06T07:22:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.321+0000 7f4cd157a640 1 -- 172.21.15.1:0/2741606889 wait complete. 2024-03-06T07:22:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.322+0000 7f4cd157a640 1 Processor -- start 2024-03-06T07:22:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.322+0000 7f4cd157a640 1 -- start start 2024-03-06T07:22:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.322+0000 7f4cd157a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ccc0a7f50 0x7f4ccc1ec4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.322+0000 7f4cd157a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4ccc0c75a0 0x7f4ccc1dd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.322+0000 7f4cd157a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ccc150640 0x7f4ccc14cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.322+0000 7f4cd157a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc150a20 con 0x7f4ccc0a7f50 2024-03-06T07:22:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.322+0000 7f4cd157a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc153200 con 0x7f4ccc150640 2024-03-06T07:22:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.322+0000 7f4cd157a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc071790 con 0x7f4ccc0c75a0 2024-03-06T07:22:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.323+0000 7f4ccaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ccc0a7f50 0x7f4ccc1ec4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.323+0000 7f4ccaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ccc0a7f50 0x7f4ccc1ec4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57376/0 (socket says 172.21.15.1:57376) 2024-03-06T07:22:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.323+0000 7f4ccaffd640 1 -- 172.21.15.1:0/298316574 learned_addr learned my addr 172.21.15.1:0/298316574 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.323+0000 7f4ccb7fe640 1 --2- 172.21.15.1:0/298316574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ccc150640 0x7f4ccc14cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.323+0000 7f4cc3fff640 1 --2- 172.21.15.1:0/298316574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4ccc0c75a0 0x7f4ccc1dd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.323+0000 7f4ccaffd640 1 -- 172.21.15.1:0/298316574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4ccc0c75a0 msgr2=0x7f4ccc1dd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.323+0000 7f4ccaffd640 1 --2- 172.21.15.1:0/298316574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4ccc0c75a0 0x7f4ccc1dd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.323+0000 7f4ccaffd640 1 -- 172.21.15.1:0/298316574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ccc150640 msgr2=0x7f4ccc14cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.323+0000 7f4ccaffd640 1 --2- 172.21.15.1:0/298316574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ccc150640 0x7f4ccc14cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.323+0000 7f4ccaffd640 1 -- 172.21.15.1:0/298316574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ccc1eca20 con 0x7f4ccc0a7f50 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.323+0000 7f4ccaffd640 1 --2- 172.21.15.1:0/298316574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ccc0a7f50 0x7f4ccc1ec4e0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc00c390 tx=0x7f4cbc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.323+0000 7f4cc37fe640 1 -- 172.21.15.1:0/298316574 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc017070 con 0x7f4ccc0a7f50 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.323+0000 7f4cc37fe640 1 -- 172.21.15.1:0/298316574 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cbc005ce0 con 0x7f4ccc0a7f50 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.323+0000 7f4cd157a640 1 -- 172.21.15.1:0/298316574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ccc0a9cf0 con 0x7f4ccc0a7f50 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.323+0000 7f4cc37fe640 1 -- 172.21.15.1:0/298316574 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc0072f0 con 0x7f4ccc0a7f50 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.323+0000 7f4cd157a640 1 -- 172.21.15.1:0/298316574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ccc1ebce0 con 0x7f4ccc0a7f50 2024-03-06T07:22:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.324+0000 7f4cd157a640 1 -- 172.21.15.1:0/298316574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ccc067fd0 con 0x7f4ccc0a7f50 2024-03-06T07:22:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.325+0000 7f4cc37fe640 1 -- 172.21.15.1:0/298316574 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4cbc01a070 con 0x7f4ccc0a7f50 2024-03-06T07:22:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.325+0000 7f4cc37fe640 1 --2- 172.21.15.1:0/298316574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cac06b4d0 0x7f4cac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.325+0000 7f4cc37fe640 1 -- 172.21.15.1:0/298316574 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f4cbc089900 con 0x7f4ccc0a7f50 2024-03-06T07:22:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.325+0000 7f4cc3fff640 1 --2- 172.21.15.1:0/298316574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cac06b4d0 0x7f4cac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.326+0000 7f4cc3fff640 1 --2- 172.21.15.1:0/298316574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cac06b4d0 0x7f4cac06d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009560 tx=0x7f4cb4009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.329+0000 7f4cc37fe640 1 -- 172.21.15.1:0/298316574 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4cbc057db0 con 0x7f4ccc0a7f50 2024-03-06T07:22:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.549+0000 7f4cd157a640 1 -- 172.21.15.1:0/298316574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs2"} v 0) v1 -- 0x7f4ccc1dd8f0 con 0x7f4ccc0a7f50 2024-03-06T07:22:44.551 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:22:44.550+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:22:44.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.551+0000 7f4cc37fe640 1 -- 172.21.15.1:0/298316574 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs2"}]=0 cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v49) v1 ==== 163+0+0 (secure 0 0 0) 0x7f4cbc05be50 con 0x7f4ccc0a7f50 2024-03-06T07:22:44.552 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:22:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.553+0000 7f4cd157a640 1 -- 172.21.15.1:0/298316574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cac06b4d0 msgr2=0x7f4cac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.553+0000 7f4cd157a640 1 --2- 172.21.15.1:0/298316574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cac06b4d0 0x7f4cac06d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009560 tx=0x7f4cb4009190 comp rx=0 tx=0).stop 2024-03-06T07:22:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.553+0000 7f4cd157a640 1 -- 172.21.15.1:0/298316574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ccc0a7f50 msgr2=0x7f4ccc1ec4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.553+0000 7f4cd157a640 1 --2- 172.21.15.1:0/298316574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ccc0a7f50 0x7f4ccc1ec4e0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc00c390 tx=0x7f4cbc00c850 comp rx=0 tx=0).stop 2024-03-06T07:22:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.553+0000 7f4cd157a640 1 -- 172.21.15.1:0/298316574 shutdown_connections 2024-03-06T07:22:44.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.553+0000 7f4cd157a640 1 --2- 172.21.15.1:0/298316574 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cac06b4d0 0x7f4cac06d990 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.553+0000 7f4cd157a640 1 --2- 172.21.15.1:0/298316574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ccc150640 0x7f4ccc14cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.553+0000 7f4cd157a640 1 --2- 172.21.15.1:0/298316574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4ccc0c75a0 0x7f4ccc1dd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.553+0000 7f4cd157a640 1 --2- 172.21.15.1:0/298316574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ccc0a7f50 0x7f4ccc1ec4e0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.553+0000 7f4cd157a640 1 -- 172.21.15.1:0/298316574 >> 172.21.15.1:0/298316574 conn(0x7f4ccc06b3e0 msgr2=0x7f4ccc06b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.553+0000 7f4cd157a640 1 -- 172.21.15.1:0/298316574 shutdown_connections 2024-03-06T07:22:44.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.553+0000 7f4cd157a640 1 -- 172.21.15.1:0/298316574 wait complete. 2024-03-06T07:22:44.566 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs2 --yes-i-really-mean-it 2024-03-06T07:22:44.581 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:22:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.667+0000 7fc4c1102640 1 Processor -- start 2024-03-06T07:22:44.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.668+0000 7fc4c1102640 1 -- start start 2024-03-06T07:22:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.668+0000 7fc4c1102640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4bc152650 0x7fc4bc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.668+0000 7fc4c1102640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc4bc0c6b10 0x7fc4bc158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.668+0000 7fc4c1102640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc0c2fc0 con 0x7fc4bc0bf9d0 2024-03-06T07:22:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.668+0000 7fc4c1102640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4bc155800 con 0x7fc4bc0c6b10 2024-03-06T07:22:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.668+0000 7fc4c1102640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc0c6690 con 0x7fc4bc152650 2024-03-06T07:22:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.668+0000 7fc4b3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc4bc0c6b10 0x7fc4bc158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.668+0000 7fc4b3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc4bc0c6b10 0x7fc4bc158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45600/0 (socket says 172.21.15.1:45600) 2024-03-06T07:22:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.668+0000 7fc4b3fff640 1 -- 172.21.15.1:0/3211681867 learned_addr learned my addr 172.21.15.1:0/3211681867 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.669+0000 7fc4bad9d640 1 --2- 172.21.15.1:0/3211681867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4bc152650 0x7fc4bc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.669+0000 7fc4b3fff640 1 -- 172.21.15.1:0/3211681867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4bc152650 msgr2=0x7fc4bc155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.669+0000 7fc4b3fff640 1 --2- 172.21.15.1:0/3211681867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4bc152650 0x7fc4bc155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.669+0000 7fc4b3fff640 1 -- 172.21.15.1:0/3211681867 >> v1:172.21.15.98:6789/0 conn(0x7fc4bc0bf9d0 legacy=0x7fc4bc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.669+0000 7fc4b3fff640 1 -- 172.21.15.1:0/3211681867 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc0c7280 con 0x7fc4bc0c6b10 2024-03-06T07:22:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.669+0000 7fc4bad9d640 1 --2- 172.21.15.1:0/3211681867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4bc152650 0x7fc4bc155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.669+0000 7fc4b3fff640 1 --2- 172.21.15.1:0/3211681867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4bc0c6b10 0x7fc4bc158690 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4009870 tx=0x7fc4a402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6176c605f3e217d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.670+0000 7fc4ba59c640 1 -- 172.21.15.1:0/3211681867 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40317c0 con 0x7fc4bc0c6b10 2024-03-06T07:22:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.670+0000 7fc4ba59c640 1 -- 172.21.15.1:0/3211681867 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a4037070 con 0x7fc4bc0c6b10 2024-03-06T07:22:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.670+0000 7fc4ba59c640 1 -- 172.21.15.1:0/3211681867 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4031b80 con 0x7fc4bc0c6b10 2024-03-06T07:22:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.670+0000 7fc4c1102640 1 -- 172.21.15.1:0/3211681867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4bc0c6b10 msgr2=0x7fc4bc158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.670+0000 7fc4c1102640 1 --2- 172.21.15.1:0/3211681867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4bc0c6b10 0x7fc4bc158690 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4009870 tx=0x7fc4a402b860 comp rx=0 tx=0).stop 2024-03-06T07:22:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.670+0000 7fc4c1102640 1 -- 172.21.15.1:0/3211681867 shutdown_connections 2024-03-06T07:22:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.670+0000 7fc4c1102640 1 --2- 172.21.15.1:0/3211681867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4bc0c6b10 0x7fc4bc158690 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.670+0000 7fc4c1102640 1 --2- 172.21.15.1:0/3211681867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4bc152650 0x7fc4bc155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.670+0000 7fc4c1102640 1 -- 172.21.15.1:0/3211681867 >> 172.21.15.1:0/3211681867 conn(0x7fc4bc066ee0 msgr2=0x7fc4bc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.670+0000 7fc4c1102640 1 -- 172.21.15.1:0/3211681867 shutdown_connections 2024-03-06T07:22:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.670+0000 7fc4c1102640 1 -- 172.21.15.1:0/3211681867 wait complete. 2024-03-06T07:22:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.671+0000 7fc4c1102640 1 Processor -- start 2024-03-06T07:22:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.671+0000 7fc4c1102640 1 -- start start 2024-03-06T07:22:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.672+0000 7fc4c1102640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4bc0bf9d0 0x7fc4bc142b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.672+0000 7fc4c1102640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4bc152650 0x7fc4bc149240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.672+0000 7fc4c1102640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4bc0c8d90 0x7fc4bc1ee0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.672+0000 7fc4c1102640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc155490 con 0x7fc4bc0c8d90 2024-03-06T07:22:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.672+0000 7fc4c1102640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc1551b0 con 0x7fc4bc152650 2024-03-06T07:22:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.672+0000 7fc4c1102640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc152a30 con 0x7fc4bc0bf9d0 2024-03-06T07:22:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.672+0000 7fc4bb59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4bc0c8d90 0x7fc4bc1ee0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.672+0000 7fc4bb59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4bc0c8d90 0x7fc4bc1ee0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45606/0 (socket says 172.21.15.1:45606) 2024-03-06T07:22:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.672+0000 7fc4bb59e640 1 -- 172.21.15.1:0/3430130418 learned_addr learned my addr 172.21.15.1:0/3430130418 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.672+0000 7fc4b3fff640 1 --2- 172.21.15.1:0/3430130418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4bc152650 0x7fc4bc149240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.672+0000 7fc4bad9d640 1 --2- 172.21.15.1:0/3430130418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4bc0bf9d0 0x7fc4bc142b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.672+0000 7fc4bb59e640 1 -- 172.21.15.1:0/3430130418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4bc0bf9d0 msgr2=0x7fc4bc142b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.672+0000 7fc4bb59e640 1 --2- 172.21.15.1:0/3430130418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4bc0bf9d0 0x7fc4bc142b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.672+0000 7fc4bb59e640 1 -- 172.21.15.1:0/3430130418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4bc152650 msgr2=0x7fc4bc149240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.672+0000 7fc4bb59e640 1 --2- 172.21.15.1:0/3430130418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4bc152650 0x7fc4bc149240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.672+0000 7fc4bb59e640 1 -- 172.21.15.1:0/3430130418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc1546e0 con 0x7fc4bc0c8d90 2024-03-06T07:22:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.672+0000 7fc4b3fff640 1 --2- 172.21.15.1:0/3430130418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4bc152650 0x7fc4bc149240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.673+0000 7fc4bb59e640 1 --2- 172.21.15.1:0/3430130418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4bc0c8d90 0x7fc4bc1ee0b0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400b970 tx=0x7fc4b400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.673+0000 7fc4b37fe640 1 -- 172.21.15.1:0/3430130418 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4019070 con 0x7fc4bc0c8d90 2024-03-06T07:22:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.673+0000 7fc4b37fe640 1 -- 172.21.15.1:0/3430130418 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b400cae0 con 0x7fc4bc0c8d90 2024-03-06T07:22:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.673+0000 7fc4c1102640 1 -- 172.21.15.1:0/3430130418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4bc154c00 con 0x7fc4bc0c8d90 2024-03-06T07:22:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.673+0000 7fc4b37fe640 1 -- 172.21.15.1:0/3430130418 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4014440 con 0x7fc4bc0c8d90 2024-03-06T07:22:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.673+0000 7fc4c1102640 1 -- 172.21.15.1:0/3430130418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4bc1430a0 con 0x7fc4bc0c8d90 2024-03-06T07:22:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.674+0000 7fc4c1102640 1 -- 172.21.15.1:0/3430130418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4bc067fd0 con 0x7fc4bc0c8d90 2024-03-06T07:22:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.674+0000 7fc4b37fe640 1 -- 172.21.15.1:0/3430130418 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc4b401c070 con 0x7fc4bc0c8d90 2024-03-06T07:22:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.675+0000 7fc4b37fe640 1 --2- 172.21.15.1:0/3430130418 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc49c06b4d0 0x7fc49c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.675+0000 7fc4b37fe640 1 -- 172.21.15.1:0/3430130418 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6535+0+0 (secure 0 0 0) 0x7fc4b408a130 con 0x7fc4bc0c8d90 2024-03-06T07:22:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.675+0000 7fc4bad9d640 1 --2- 172.21.15.1:0/3430130418 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc49c06b4d0 0x7fc49c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.675+0000 7fc4bad9d640 1 --2- 172.21.15.1:0/3430130418 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc49c06b4d0 0x7fc49c06d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac008850 tx=0x7fc4ac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.679+0000 7fc4b37fe640 1 -- 172.21.15.1:0/3430130418 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc4bc1546e0 con 0x7fc4bc0c8d90 2024-03-06T07:22:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:44.903+0000 7fc4c1102640 1 -- 172.21.15.1:0/3430130418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc4bc0b06f0 con 0x7fc4bc0c8d90 2024-03-06T07:22:45.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.556+0000 7fc4b37fe640 1 -- 172.21.15.1:0/3430130418 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true}]=0 v50) v1 ==== 105+0+0 (secure 0 0 0) 0x7fc4b4058d00 con 0x7fc4bc0c8d90 2024-03-06T07:22:45.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.559+0000 7fc4c1102640 1 -- 172.21.15.1:0/3430130418 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc49c06b4d0 msgr2=0x7fc49c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.559+0000 7fc4c1102640 1 --2- 172.21.15.1:0/3430130418 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc49c06b4d0 0x7fc49c06d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac008850 tx=0x7fc4ac008040 comp rx=0 tx=0).stop 2024-03-06T07:22:45.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.559+0000 7fc4c1102640 1 -- 172.21.15.1:0/3430130418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4bc0c8d90 msgr2=0x7fc4bc1ee0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.559+0000 7fc4c1102640 1 --2- 172.21.15.1:0/3430130418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4bc0c8d90 0x7fc4bc1ee0b0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400b970 tx=0x7fc4b400be30 comp rx=0 tx=0).stop 2024-03-06T07:22:45.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.559+0000 7fc4c1102640 1 -- 172.21.15.1:0/3430130418 shutdown_connections 2024-03-06T07:22:45.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.559+0000 7fc4c1102640 1 --2- 172.21.15.1:0/3430130418 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc49c06b4d0 0x7fc49c06d990 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.559+0000 7fc4c1102640 1 --2- 172.21.15.1:0/3430130418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4bc0c8d90 0x7fc4bc1ee0b0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.559+0000 7fc4c1102640 1 --2- 172.21.15.1:0/3430130418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4bc152650 0x7fc4bc149240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.559+0000 7fc4c1102640 1 --2- 172.21.15.1:0/3430130418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4bc0bf9d0 0x7fc4bc142b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.559+0000 7fc4c1102640 1 -- 172.21.15.1:0/3430130418 >> 172.21.15.1:0/3430130418 conn(0x7fc4bc066ee0 msgr2=0x7fc4bc14e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.559+0000 7fc4c1102640 1 -- 172.21.15.1:0/3430130418 shutdown_connections 2024-03-06T07:22:45.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.559+0000 7fc4c1102640 1 -- 172.21.15.1:0/3430130418 wait complete. 2024-03-06T07:22:45.575 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_metadata cephfs2_metadata --yes-i-really-really-mean-it 2024-03-06T07:22:45.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.684+0000 7fb407f0c640 1 Processor -- start 2024-03-06T07:22:45.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.685+0000 7fb407f0c640 1 -- start start 2024-03-06T07:22:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.685+0000 7fb407f0c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb400155c60 0x7fb400160c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.685+0000 7fb407f0c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb400158890 con 0x7fb4000c16e0 2024-03-06T07:22:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.685+0000 7fb407f0c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb400158b70 con 0x7fb4000c70c0 2024-03-06T07:22:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.685+0000 7fb407f0c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb400158e50 con 0x7fb400155c60 2024-03-06T07:22:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.685+0000 7fb4064a9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb400155c60 0x7fb400160c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.685+0000 7fb4064a9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb400155c60 0x7fb400160c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45612/0 (socket says 172.21.15.1:45612) 2024-03-06T07:22:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.685+0000 7fb4064a9640 1 -- 172.21.15.1:0/2914874822 learned_addr learned my addr 172.21.15.1:0/2914874822 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.685+0000 7fb405ca8640 1 --1- 172.21.15.1:0/2914874822 >> v1:172.21.15.1:6789/0 conn(0x7fb4000c16e0 0x7fb400152470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58800/0 (socket says 172.21.15.1:58800) 2024-03-06T07:22:45.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.686+0000 7fb4064a9640 1 -- 172.21.15.1:0/2914874822 >> v1:172.21.15.98:6789/0 conn(0x7fb4000c70c0 legacy=0x7fb400155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.686+0000 7fb4064a9640 1 -- 172.21.15.1:0/2914874822 >> v1:172.21.15.1:6789/0 conn(0x7fb4000c16e0 legacy=0x7fb400152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.686+0000 7fb4064a9640 1 -- 172.21.15.1:0/2914874822 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb400159130 con 0x7fb400155c60 2024-03-06T07:22:45.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.686+0000 7fb404ca6640 1 -- 172.21.15.1:0/2914874822 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773391697 0 0) 0x7fb400158890 con 0x7fb4000c16e0 2024-03-06T07:22:45.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.687+0000 7fb4064a9640 1 --2- 172.21.15.1:0/2914874822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb400155c60 0x7fb400160c30 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009e50 tx=0x7fb3f002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c81641db3a412a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.687+0000 7fb404ca6640 1 -- 172.21.15.1:0/2914874822 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f002ca80 con 0x7fb400155c60 2024-03-06T07:22:45.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.687+0000 7fb404ca6640 1 -- 172.21.15.1:0/2914874822 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f002cc20 con 0x7fb400155c60 2024-03-06T07:22:45.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.687+0000 7fb404ca6640 1 -- 172.21.15.1:0/2914874822 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f00343f0 con 0x7fb400155c60 2024-03-06T07:22:45.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.687+0000 7fb407f0c640 1 -- 172.21.15.1:0/2914874822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb400155c60 msgr2=0x7fb400160c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.687+0000 7fb407f0c640 1 --2- 172.21.15.1:0/2914874822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb400155c60 0x7fb400160c30 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009e50 tx=0x7fb3f002bee0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.687+0000 7fb407f0c640 1 -- 172.21.15.1:0/2914874822 shutdown_connections 2024-03-06T07:22:45.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.687+0000 7fb407f0c640 1 --2- 172.21.15.1:0/2914874822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb400155c60 0x7fb400160c30 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.687+0000 7fb407f0c640 1 -- 172.21.15.1:0/2914874822 >> 172.21.15.1:0/2914874822 conn(0x7fb400066d00 msgr2=0x7fb400067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.687+0000 7fb407f0c640 1 -- 172.21.15.1:0/2914874822 shutdown_connections 2024-03-06T07:22:45.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.687+0000 7fb407f0c640 1 -- 172.21.15.1:0/2914874822 wait complete. 2024-03-06T07:22:45.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.688+0000 7fb407f0c640 1 Processor -- start 2024-03-06T07:22:45.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.688+0000 7fb407f0c640 1 -- start start 2024-03-06T07:22:45.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.689+0000 7fb407f0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4000c16e0 0x7fb4001583d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.689+0000 7fb407f0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4000c70c0 0x7fb4001ee5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.689+0000 7fb407f0c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb400155c60 0x7fb400157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.689+0000 7fb407f0c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4000c68c0 con 0x7fb400155c60 2024-03-06T07:22:45.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.689+0000 7fb407f0c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4001596f0 con 0x7fb4000c70c0 2024-03-06T07:22:45.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.689+0000 7fb407f0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4000cb470 con 0x7fb4000c16e0 2024-03-06T07:22:45.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.689+0000 7fb4064a9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb400155c60 0x7fb400157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.689+0000 7fb4064a9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb400155c60 0x7fb400157740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45614/0 (socket says 172.21.15.1:45614) 2024-03-06T07:22:45.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.689+0000 7fb4064a9640 1 -- 172.21.15.1:0/3653772274 learned_addr learned my addr 172.21.15.1:0/3653772274 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:45.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.689+0000 7fb405ca8640 1 --2- 172.21.15.1:0/3653772274 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4000c16e0 0x7fb4001583d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.689+0000 7fb4064a9640 1 -- 172.21.15.1:0/3653772274 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4000c16e0 msgr2=0x7fb4001583d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.689+0000 7fb4054a7640 1 --2- 172.21.15.1:0/3653772274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4000c70c0 0x7fb4001ee5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.690+0000 7fb4064a9640 1 --2- 172.21.15.1:0/3653772274 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4000c16e0 0x7fb4001583d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.690+0000 7fb4064a9640 1 -- 172.21.15.1:0/3653772274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4000c70c0 msgr2=0x7fb4001ee5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.690+0000 7fb4064a9640 1 --2- 172.21.15.1:0/3653772274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4000c70c0 0x7fb4001ee5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.690+0000 7fb4064a9640 1 -- 172.21.15.1:0/3653772274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb400153740 con 0x7fb400155c60 2024-03-06T07:22:45.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.690+0000 7fb405ca8640 1 --2- 172.21.15.1:0/3653772274 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4000c16e0 0x7fb4001583d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:45.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.690+0000 7fb4054a7640 1 --2- 172.21.15.1:0/3653772274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4000c70c0 0x7fb4001ee5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:45.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.690+0000 7fb4064a9640 1 --2- 172.21.15.1:0/3653772274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb400155c60 0x7fb400157740 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009e50 tx=0x7fb3f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.690+0000 7fb3fe7fc640 1 -- 172.21.15.1:0/3653772274 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0043020 con 0x7fb400155c60 2024-03-06T07:22:45.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.690+0000 7fb3fe7fc640 1 -- 172.21.15.1:0/3653772274 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f003e450 con 0x7fb400155c60 2024-03-06T07:22:45.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.690+0000 7fb3fe7fc640 1 -- 172.21.15.1:0/3653772274 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0009510 con 0x7fb400155c60 2024-03-06T07:22:45.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.690+0000 7fb407f0c640 1 -- 172.21.15.1:0/3653772274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4001eeee0 con 0x7fb400155c60 2024-03-06T07:22:45.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.690+0000 7fb407f0c640 1 -- 172.21.15.1:0/3653772274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4001ef1c0 con 0x7fb400155c60 2024-03-06T07:22:45.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.691+0000 7fb407f0c640 1 -- 172.21.15.1:0/3653772274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4000680b0 con 0x7fb400155c60 2024-03-06T07:22:45.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.692+0000 7fb3fe7fc640 1 -- 172.21.15.1:0/3653772274 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3f0046020 con 0x7fb400155c60 2024-03-06T07:22:45.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.692+0000 7fb3fe7fc640 1 --2- 172.21.15.1:0/3653772274 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb3ec06b500 0x7fb3ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.692+0000 7fb3fe7fc640 1 -- 172.21.15.1:0/3653772274 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6535+0+0 (secure 0 0 0) 0x7fb3f00acbe0 con 0x7fb400155c60 2024-03-06T07:22:45.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.692+0000 7fb405ca8640 1 --2- 172.21.15.1:0/3653772274 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb3ec06b500 0x7fb3ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.693+0000 7fb405ca8640 1 --2- 172.21.15.1:0/3653772274 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb3ec06b500 0x7fb3ec06d9c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8000f60 tx=0x7fb3f8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.696+0000 7fb3fe7fc640 1 -- 172.21.15.1:0/3653772274 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3f004d070 con 0x7fb400155c60 2024-03-06T07:22:45.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:45.878+0000 7fb407f0c640 1 -- 172.21.15.1:0/3653772274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb4000b0880 con 0x7fb400155c60 2024-03-06T07:22:46.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.558+0000 7fb3fe7fc640 1 -- 172.21.15.1:0/3653772274 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_metadata' removed v64) v1 ==== 184+0+0 (secure 0 0 0) 0x7fb3f007b720 con 0x7fb400155c60 2024-03-06T07:22:46.560 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs2_metadata' removed 2024-03-06T07:22:46.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.565+0000 7fb407f0c640 1 -- 172.21.15.1:0/3653772274 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb3ec06b500 msgr2=0x7fb3ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.565+0000 7fb407f0c640 1 --2- 172.21.15.1:0/3653772274 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb3ec06b500 0x7fb3ec06d9c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8000f60 tx=0x7fb3f8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:46.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.565+0000 7fb407f0c640 1 -- 172.21.15.1:0/3653772274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb400155c60 msgr2=0x7fb400157740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.565+0000 7fb407f0c640 1 --2- 172.21.15.1:0/3653772274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb400155c60 0x7fb400157740 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009e50 tx=0x7fb3f0004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.566+0000 7fb407f0c640 1 -- 172.21.15.1:0/3653772274 shutdown_connections 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.566+0000 7fb407f0c640 1 --2- 172.21.15.1:0/3653772274 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb3ec06b500 0x7fb3ec06d9c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.566+0000 7fb407f0c640 1 --2- 172.21.15.1:0/3653772274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb400155c60 0x7fb400157740 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.566+0000 7fb407f0c640 1 --2- 172.21.15.1:0/3653772274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4000c70c0 0x7fb4001ee5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.566+0000 7fb407f0c640 1 --2- 172.21.15.1:0/3653772274 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4000c16e0 0x7fb4001583d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.566+0000 7fb407f0c640 1 -- 172.21.15.1:0/3653772274 >> 172.21.15.1:0/3653772274 conn(0x7fb400066d00 msgr2=0x7fb400155550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.567+0000 7fb407f0c640 1 -- 172.21.15.1:0/3653772274 shutdown_connections 2024-03-06T07:22:46.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.567+0000 7fb407f0c640 1 -- 172.21.15.1:0/3653772274 wait complete. 2024-03-06T07:22:46.583 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_data cephfs2_data --yes-i-really-really-mean-it 2024-03-06T07:22:46.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.691+0000 7f05fae5c640 1 Processor -- start 2024-03-06T07:22:46.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.691+0000 7f05fae5c640 1 -- start start 2024-03-06T07:22:46.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.692+0000 7f05fae5c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f05f40c0df0 0x7f05f40c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.692+0000 7f05fae5c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05f4155200 con 0x7f05f40a7150 2024-03-06T07:22:46.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.692+0000 7f05fae5c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05f4155470 con 0x7f05f40c58c0 2024-03-06T07:22:46.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.692+0000 7f05fae5c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f40a7530 con 0x7f05f40c0df0 2024-03-06T07:22:46.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.692+0000 7f05f3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f05f40c0df0 0x7f05f40c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.692+0000 7f05f3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f05f40c0df0 0x7f05f40c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45616/0 (socket says 172.21.15.1:45616) 2024-03-06T07:22:46.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.692+0000 7f05f3fff640 1 -- 172.21.15.1:0/944436177 learned_addr learned my addr 172.21.15.1:0/944436177 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:46.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.692+0000 7f05f8bf8640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f05f40a7150 0x7f05f40c6ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58810/0 (socket says 172.21.15.1:58810) 2024-03-06T07:22:46.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.692+0000 7f05f3fff640 1 -- 172.21.15.1:0/944436177 >> v1:172.21.15.98:6789/0 conn(0x7f05f40c58c0 legacy=0x7f05f40c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.692+0000 7f05f3fff640 1 -- 172.21.15.1:0/944436177 >> v1:172.21.15.1:6789/0 conn(0x7f05f40a7150 legacy=0x7f05f40c6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.692+0000 7f05f3fff640 1 -- 172.21.15.1:0/944436177 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f4155750 con 0x7f05f40c0df0 2024-03-06T07:22:46.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.692+0000 7f05f37fe640 1 -- 172.21.15.1:0/944436177 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36602309 0 0) 0x7f05f4155200 con 0x7f05f40a7150 2024-03-06T07:22:46.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.693+0000 7f05f3fff640 1 --2- 172.21.15.1:0/944436177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05f40c0df0 0x7f05f40c12a0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009870 tx=0x7f05e002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d32f42a6bac82263 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.693+0000 7f05f37fe640 1 -- 172.21.15.1:0/944436177 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e002e070 con 0x7f05f40c0df0 2024-03-06T07:22:46.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.693+0000 7f05f37fe640 1 -- 172.21.15.1:0/944436177 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e0037070 con 0x7f05f40c0df0 2024-03-06T07:22:46.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.693+0000 7f05f37fe640 1 -- 172.21.15.1:0/944436177 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0031a00 con 0x7f05f40c0df0 2024-03-06T07:22:46.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.694+0000 7f05fae5c640 1 -- 172.21.15.1:0/944436177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05f40c0df0 msgr2=0x7f05f40c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.694+0000 7f05fae5c640 1 --2- 172.21.15.1:0/944436177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05f40c0df0 0x7f05f40c12a0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009870 tx=0x7f05e002b880 comp rx=0 tx=0).stop 2024-03-06T07:22:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.694+0000 7f05fae5c640 1 -- 172.21.15.1:0/944436177 shutdown_connections 2024-03-06T07:22:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.694+0000 7f05fae5c640 1 --2- 172.21.15.1:0/944436177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05f40c0df0 0x7f05f40c12a0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.694+0000 7f05fae5c640 1 -- 172.21.15.1:0/944436177 >> 172.21.15.1:0/944436177 conn(0x7f05f40c44c0 msgr2=0x7f05f40c48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.694+0000 7f05fae5c640 1 -- 172.21.15.1:0/944436177 shutdown_connections 2024-03-06T07:22:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.694+0000 7f05fae5c640 1 -- 172.21.15.1:0/944436177 wait complete. 2024-03-06T07:22:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.695+0000 7f05fae5c640 1 Processor -- start 2024-03-06T07:22:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.695+0000 7f05fae5c640 1 -- start start 2024-03-06T07:22:46.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05fae5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05f40a7150 0x7f05f414f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05fae5c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05f40c0df0 0x7f05f41eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05fae5c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05f40c58c0 0x7f05f414de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05fae5c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f4155f50 con 0x7f05f40c0df0 2024-03-06T07:22:46.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05fae5c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f40c7710 con 0x7f05f40a7150 2024-03-06T07:22:46.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05fae5c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f414d490 con 0x7f05f40c58c0 2024-03-06T07:22:46.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05f3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05f40c0df0 0x7f05f41eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05f3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05f40c0df0 0x7f05f41eebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45622/0 (socket says 172.21.15.1:45622) 2024-03-06T07:22:46.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05f3fff640 1 -- 172.21.15.1:0/3164658933 learned_addr learned my addr 172.21.15.1:0/3164658933 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:46.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05f8bf8640 1 --2- 172.21.15.1:0/3164658933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05f40a7150 0x7f05f414f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05f93f9640 1 --2- 172.21.15.1:0/3164658933 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05f40c58c0 0x7f05f414de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05f3fff640 1 -- 172.21.15.1:0/3164658933 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05f40c58c0 msgr2=0x7f05f414de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05f3fff640 1 --2- 172.21.15.1:0/3164658933 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05f40c58c0 0x7f05f414de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05f3fff640 1 -- 172.21.15.1:0/3164658933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05f40a7150 msgr2=0x7f05f414f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05f3fff640 1 --2- 172.21.15.1:0/3164658933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05f40a7150 0x7f05f414f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05f3fff640 1 -- 172.21.15.1:0/3164658933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f414f1c0 con 0x7f05f40c0df0 2024-03-06T07:22:46.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05f8bf8640 1 --2- 172.21.15.1:0/3164658933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05f40a7150 0x7f05f414f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:46.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05f3fff640 1 --2- 172.21.15.1:0/3164658933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05f40c0df0 0x7f05f41eebb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009870 tx=0x7f05e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.696+0000 7f05f17fa640 1 -- 172.21.15.1:0/3164658933 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0032880 con 0x7f05f40c0df0 2024-03-06T07:22:46.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.697+0000 7f05f17fa640 1 -- 172.21.15.1:0/3164658933 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e0032a40 con 0x7f05f40c0df0 2024-03-06T07:22:46.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.697+0000 7f05f17fa640 1 -- 172.21.15.1:0/3164658933 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0032e30 con 0x7f05f40c0df0 2024-03-06T07:22:46.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.697+0000 7f05fae5c640 1 -- 172.21.15.1:0/3164658933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f414ec00 con 0x7f05f40c0df0 2024-03-06T07:22:46.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.697+0000 7f05fae5c640 1 -- 172.21.15.1:0/3164658933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f41ead40 con 0x7f05f40c0df0 2024-03-06T07:22:46.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.697+0000 7f05fae5c640 1 -- 172.21.15.1:0/3164658933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f40680b0 con 0x7f05f40c0df0 2024-03-06T07:22:46.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.698+0000 7f05f17fa640 1 -- 172.21.15.1:0/3164658933 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05e004c060 con 0x7f05f40c0df0 2024-03-06T07:22:46.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.698+0000 7f05f17fa640 1 --2- 172.21.15.1:0/3164658933 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f05d806b4d0 0x7f05d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.698+0000 7f05f17fa640 1 -- 172.21.15.1:0/3164658933 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6095+0+0 (secure 0 0 0) 0x7f05e00abff0 con 0x7f05f40c0df0 2024-03-06T07:22:46.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.699+0000 7f05f8bf8640 1 --2- 172.21.15.1:0/3164658933 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f05d806b4d0 0x7f05d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.699+0000 7f05f8bf8640 1 --2- 172.21.15.1:0/3164658933 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f05d806b4d0 0x7f05d806d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f05e8000f60 tx=0x7f05e8007c10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.703+0000 7f05f17fa640 1 -- 172.21.15.1:0/3164658933 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05e0040070 con 0x7f05f40c0df0 2024-03-06T07:22:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:46.885+0000 7f05fae5c640 1 -- 172.21.15.1:0/3164658933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f05f40c4d90 con 0x7f05f40c0df0 2024-03-06T07:22:47.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.563+0000 7f05f17fa640 1 -- 172.21.15.1:0/3164658933 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_data' removed v65) v1 ==== 172+0+0 (secure 0 0 0) 0x7f05e007a6f0 con 0x7f05f40c0df0 2024-03-06T07:22:47.565 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs2_data' removed 2024-03-06T07:22:47.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.567+0000 7f05fae5c640 1 -- 172.21.15.1:0/3164658933 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f05d806b4d0 msgr2=0x7f05d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.567+0000 7f05fae5c640 1 --2- 172.21.15.1:0/3164658933 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f05d806b4d0 0x7f05d806d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f05e8000f60 tx=0x7f05e8007c10 comp rx=0 tx=0).stop 2024-03-06T07:22:47.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.567+0000 7f05fae5c640 1 -- 172.21.15.1:0/3164658933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05f40c0df0 msgr2=0x7f05f41eebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.567+0000 7f05fae5c640 1 --2- 172.21.15.1:0/3164658933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05f40c0df0 0x7f05f41eebb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009870 tx=0x7f05e00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.567+0000 7f05fae5c640 1 -- 172.21.15.1:0/3164658933 shutdown_connections 2024-03-06T07:22:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.567+0000 7f05fae5c640 1 --2- 172.21.15.1:0/3164658933 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f05d806b4d0 0x7f05d806d990 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.567+0000 7f05fae5c640 1 --2- 172.21.15.1:0/3164658933 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05f40c58c0 0x7f05f414de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.567+0000 7f05fae5c640 1 --2- 172.21.15.1:0/3164658933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05f40c0df0 0x7f05f41eebb0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.567+0000 7f05fae5c640 1 --2- 172.21.15.1:0/3164658933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05f40a7150 0x7f05f414f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.567+0000 7f05fae5c640 1 -- 172.21.15.1:0/3164658933 >> 172.21.15.1:0/3164658933 conn(0x7f05f40c44c0 msgr2=0x7f05f40a7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.567+0000 7f05fae5c640 1 -- 172.21.15.1:0/3164658933 shutdown_connections 2024-03-06T07:22:47.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.567+0000 7f05fae5c640 1 -- 172.21.15.1:0/3164658933 wait complete. 2024-03-06T07:22:47.586 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:22:47.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.691+0000 7f03c9d56640 1 Processor -- start 2024-03-06T07:22:47.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.691+0000 7f03c9d56640 1 -- start start 2024-03-06T07:22:47.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.692+0000 7f03c9d56640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f03c40c69b0 0x7f03c40c2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.692+0000 7f03c9d56640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03c40c7510 0x7f03c40c3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.692+0000 7f03c9d56640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c4072d00 con 0x7f03c4159140 2024-03-06T07:22:47.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.692+0000 7f03c9d56640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c40c0e80 con 0x7f03c40c7510 2024-03-06T07:22:47.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.692+0000 7f03c9d56640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c40c13b0 con 0x7f03c40c69b0 2024-03-06T07:22:47.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.692+0000 7f03c3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03c40c7510 0x7f03c40c3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.692+0000 7f03c37fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f03c4159140 0x7f03c40c4490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58814/0 (socket says 172.21.15.1:58814) 2024-03-06T07:22:47.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.692+0000 7f03c37fe640 1 -- 172.21.15.1:0/1276277558 learned_addr learned my addr 172.21.15.1:0/1276277558 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:47.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.692+0000 7f03c3fff640 1 -- 172.21.15.1:0/1276277558 >> v2:172.21.15.98:3300/0 conn(0x7f03c40c69b0 msgr2=0x7f03c40c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.692+0000 7f03c3fff640 1 --2- 172.21.15.1:0/1276277558 >> v2:172.21.15.98:3300/0 conn(0x7f03c40c69b0 0x7f03c40c2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.692+0000 7f03c3fff640 1 -- 172.21.15.1:0/1276277558 >> v1:172.21.15.1:6789/0 conn(0x7f03c4159140 legacy=0x7f03c40c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.692+0000 7f03c3fff640 1 -- 172.21.15.1:0/1276277558 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c4073180 con 0x7f03c40c7510 2024-03-06T07:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.692+0000 7f03c27fc640 1 -- 172.21.15.1:0/1276277558 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483484814 0 0) 0x7f03c4072d00 con 0x7f03c4159140 2024-03-06T07:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.693+0000 7f03c3fff640 1 --2- 172.21.15.1:0/1276277558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c40c7510 0x7f03c40c3070 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f03c40a7000 tx=0x7f03bc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67072008ed758f87 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.693+0000 7f03c27fc640 1 -- 172.21.15.1:0/1276277558 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc039070 con 0x7f03c40c7510 2024-03-06T07:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.693+0000 7f03c27fc640 1 -- 172.21.15.1:0/1276277558 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03bc02cc00 con 0x7f03c40c7510 2024-03-06T07:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.693+0000 7f03c27fc640 1 -- 172.21.15.1:0/1276277558 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc034440 con 0x7f03c40c7510 2024-03-06T07:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.694+0000 7f03c9d56640 1 -- 172.21.15.1:0/1276277558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c40c7510 msgr2=0x7f03c40c3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.694+0000 7f03c9d56640 1 --2- 172.21.15.1:0/1276277558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c40c7510 0x7f03c40c3070 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f03c40a7000 tx=0x7f03bc02be50 comp rx=0 tx=0).stop 2024-03-06T07:22:47.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.694+0000 7f03c9d56640 1 -- 172.21.15.1:0/1276277558 shutdown_connections 2024-03-06T07:22:47.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.694+0000 7f03c9d56640 1 --2- 172.21.15.1:0/1276277558 >> v2:172.21.15.98:3300/0 conn(0x7f03c40c69b0 0x7f03c40c2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.694+0000 7f03c9d56640 1 --2- 172.21.15.1:0/1276277558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c40c7510 0x7f03c40c3070 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.694+0000 7f03c9d56640 1 -- 172.21.15.1:0/1276277558 >> 172.21.15.1:0/1276277558 conn(0x7f03c40c7a50 msgr2=0x7f03c40c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.694+0000 7f03c9d56640 1 -- 172.21.15.1:0/1276277558 shutdown_connections 2024-03-06T07:22:47.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.694+0000 7f03c9d56640 1 -- 172.21.15.1:0/1276277558 wait complete. 2024-03-06T07:22:47.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.695+0000 7f03c9d56640 1 Processor -- start 2024-03-06T07:22:47.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.695+0000 7f03c9d56640 1 -- start start 2024-03-06T07:22:47.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.696+0000 7f03c9d56640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c40c69b0 0x7f03c4151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.696+0000 7f03c9d56640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03c40c7510 0x7f03c40ab850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.696+0000 7f03c9d56640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03c4159140 0x7f03c4148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.696+0000 7f03c9d56640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c40c35b0 con 0x7f03c40c69b0 2024-03-06T07:22:47.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.696+0000 7f03c9d56640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c40c6520 con 0x7f03c4159140 2024-03-06T07:22:47.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.696+0000 7f03c9d56640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c40c10d0 con 0x7f03c40c7510 2024-03-06T07:22:47.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.696+0000 7f03c37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c40c69b0 0x7f03c4151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.696+0000 7f03c37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c40c69b0 0x7f03c4151700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45640/0 (socket says 172.21.15.1:45640) 2024-03-06T07:22:47.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.696+0000 7f03c37fe640 1 -- 172.21.15.1:0/1813347434 learned_addr learned my addr 172.21.15.1:0/1813347434 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:47.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.696+0000 7f03c37fe640 1 -- 172.21.15.1:0/1813347434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03c40c7510 msgr2=0x7f03c40ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.696+0000 7f03c37fe640 1 --2- 172.21.15.1:0/1813347434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03c40c7510 0x7f03c40ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.696+0000 7f03c37fe640 1 -- 172.21.15.1:0/1813347434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03c4159140 msgr2=0x7f03c4148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.696+0000 7f03c37fe640 1 --2- 172.21.15.1:0/1813347434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03c4159140 0x7f03c4148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.696+0000 7f03c37fe640 1 -- 172.21.15.1:0/1813347434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c40aaa90 con 0x7f03c40c69b0 2024-03-06T07:22:47.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.696+0000 7f03c37fe640 1 --2- 172.21.15.1:0/1813347434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c40c69b0 0x7f03c4151700 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f03b800c430 tx=0x7f03b800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.697+0000 7f03abfff640 1 -- 172.21.15.1:0/1813347434 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b80074e0 con 0x7f03c40c69b0 2024-03-06T07:22:47.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.697+0000 7f03abfff640 1 -- 172.21.15.1:0/1813347434 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b80047e0 con 0x7f03c40c69b0 2024-03-06T07:22:47.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.697+0000 7f03abfff640 1 -- 172.21.15.1:0/1813347434 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8004bd0 con 0x7f03c40c69b0 2024-03-06T07:22:47.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.697+0000 7f03c9d56640 1 -- 172.21.15.1:0/1813347434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c40aad70 con 0x7f03c40c69b0 2024-03-06T07:22:47.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.697+0000 7f03c9d56640 1 -- 172.21.15.1:0/1813347434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c40ab330 con 0x7f03c40c69b0 2024-03-06T07:22:47.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.698+0000 7f03c9d56640 1 -- 172.21.15.1:0/1813347434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c40686e0 con 0x7f03c40c69b0 2024-03-06T07:22:47.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.698+0000 7f03abfff640 1 -- 172.21.15.1:0/1813347434 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03b8011070 con 0x7f03c40c69b0 2024-03-06T07:22:47.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.698+0000 7f03abfff640 1 --2- 172.21.15.1:0/1813347434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f03b006b4d0 0x7f03b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.698+0000 7f03abfff640 1 -- 172.21.15.1:0/1813347434 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f03b8089e30 con 0x7f03c40c69b0 2024-03-06T07:22:47.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.699+0000 7f03c2ffd640 1 --2- 172.21.15.1:0/1813347434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f03b006b4d0 0x7f03b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.699+0000 7f03c2ffd640 1 --2- 172.21.15.1:0/1813347434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f03b006b4d0 0x7f03b006d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f03c40711f0 tx=0x7f03b4006c90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.703+0000 7f03abfff640 1 -- 172.21.15.1:0/1813347434 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f03b800e280 con 0x7f03c40c69b0 2024-03-06T07:22:47.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.886+0000 7f03c9d56640 1 -- 172.21.15.1:0/1813347434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f03c4072f60 con 0x7f03c40c69b0 2024-03-06T07:22:47.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.889+0000 7f03abfff640 1 -- 172.21.15.1:0/1813347434 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f03b8058d60 con 0x7f03c40c69b0 2024-03-06T07:22:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.891+0000 7f03c9d56640 1 -- 172.21.15.1:0/1813347434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f03b006b4d0 msgr2=0x7f03b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.891+0000 7f03c9d56640 1 --2- 172.21.15.1:0/1813347434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f03b006b4d0 0x7f03b006d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f03c40711f0 tx=0x7f03b4006c90 comp rx=0 tx=0).stop 2024-03-06T07:22:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.891+0000 7f03c9d56640 1 -- 172.21.15.1:0/1813347434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c40c69b0 msgr2=0x7f03c4151700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.891+0000 7f03c9d56640 1 --2- 172.21.15.1:0/1813347434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c40c69b0 0x7f03c4151700 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f03b800c430 tx=0x7f03b800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.891+0000 7f03c9d56640 1 -- 172.21.15.1:0/1813347434 shutdown_connections 2024-03-06T07:22:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.891+0000 7f03c9d56640 1 --2- 172.21.15.1:0/1813347434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f03b006b4d0 0x7f03b006d990 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.891+0000 7f03c9d56640 1 --2- 172.21.15.1:0/1813347434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03c4159140 0x7f03c4148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.891+0000 7f03c9d56640 1 --2- 172.21.15.1:0/1813347434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03c40c7510 0x7f03c40ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.891+0000 7f03c9d56640 1 --2- 172.21.15.1:0/1813347434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c40c69b0 0x7f03c4151700 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.891+0000 7f03c9d56640 1 -- 172.21.15.1:0/1813347434 >> 172.21.15.1:0/1813347434 conn(0x7f03c40c7a50 msgr2=0x7f03c406f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.891+0000 7f03c9d56640 1 -- 172.21.15.1:0/1813347434 shutdown_connections 2024-03-06T07:22:47.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:47.891+0000 7f03c9d56640 1 -- 172.21.15.1:0/1813347434 wait complete. 2024-03-06T07:22:47.904 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-03-06T07:22:48.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.005+0000 7f83ad639640 1 Processor -- start 2024-03-06T07:22:48.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.005+0000 7f83ad639640 1 -- start start 2024-03-06T07:22:48.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.006+0000 7f83ad639640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f83a814cdc0 0x7f83a8152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.006+0000 7f83ad639640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a81569e0 con 0x7f83a806b510 2024-03-06T07:22:48.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.006+0000 7f83ad639640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a8156cc0 con 0x7f83a8156500 2024-03-06T07:22:48.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.006+0000 7f83ad639640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83a80c2dd0 con 0x7f83a814cdc0 2024-03-06T07:22:48.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.006+0000 7f83a6ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f83a814cdc0 0x7f83a8152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.006+0000 7f83a6ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f83a814cdc0 0x7f83a8152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45648/0 (socket says 172.21.15.1:45648) 2024-03-06T07:22:48.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.006+0000 7f83a6ffd640 1 -- 172.21.15.1:0/3851750804 learned_addr learned my addr 172.21.15.1:0/3851750804 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:48.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.006+0000 7f83a77fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f83a806b510 0x7f83a806b960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58826/0 (socket says 172.21.15.1:58826) 2024-03-06T07:22:48.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.007+0000 7f83a6ffd640 1 -- 172.21.15.1:0/3851750804 >> v1:172.21.15.98:6789/0 conn(0x7f83a8156500 legacy=0x7f83a80c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.007+0000 7f83a6ffd640 1 -- 172.21.15.1:0/3851750804 >> v1:172.21.15.1:6789/0 conn(0x7f83a806b510 legacy=0x7f83a806b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.007+0000 7f83a6ffd640 1 -- 172.21.15.1:0/3851750804 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a8156fa0 con 0x7f83a814cdc0 2024-03-06T07:22:48.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.007+0000 7f83a6ffd640 1 --2- 172.21.15.1:0/3851750804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83a814cdc0 0x7f83a8152f10 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009870 tx=0x7f83a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a87849b6c0806c40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.008+0000 7f83a5ffb640 1 -- 172.21.15.1:0/3851750804 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a0035020 con 0x7f83a814cdc0 2024-03-06T07:22:48.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.008+0000 7f83a5ffb640 1 -- 172.21.15.1:0/3851750804 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a0031870 con 0x7f83a814cdc0 2024-03-06T07:22:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.008+0000 7f83a5ffb640 1 -- 172.21.15.1:0/3851750804 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a0031c40 con 0x7f83a814cdc0 2024-03-06T07:22:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.008+0000 7f83ad639640 1 -- 172.21.15.1:0/3851750804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83a814cdc0 msgr2=0x7f83a8152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.008+0000 7f83ad639640 1 --2- 172.21.15.1:0/3851750804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83a814cdc0 0x7f83a8152f10 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009870 tx=0x7f83a002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.008+0000 7f83ad639640 1 -- 172.21.15.1:0/3851750804 shutdown_connections 2024-03-06T07:22:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.008+0000 7f83ad639640 1 --2- 172.21.15.1:0/3851750804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83a814cdc0 0x7f83a8152f10 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.008+0000 7f83ad639640 1 -- 172.21.15.1:0/3851750804 >> 172.21.15.1:0/3851750804 conn(0x7f83a80a9700 msgr2=0x7f83a80a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.008+0000 7f83ad639640 1 -- 172.21.15.1:0/3851750804 shutdown_connections 2024-03-06T07:22:48.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.008+0000 7f83ad639640 1 -- 172.21.15.1:0/3851750804 wait complete. 2024-03-06T07:22:48.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.009+0000 7f83ad639640 1 Processor -- start 2024-03-06T07:22:48.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.009+0000 7f83ad639640 1 -- start start 2024-03-06T07:22:48.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83ad639640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a806b510 0x7f83a814d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83ad639640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83a814cdc0 0x7f83a8149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83ad639640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83a8156500 0x7f83a814a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83ad639640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a80c65e0 con 0x7f83a8156500 2024-03-06T07:22:48.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83ad639640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a80c5a70 con 0x7f83a806b510 2024-03-06T07:22:48.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83ad639640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a80c2a60 con 0x7f83a814cdc0 2024-03-06T07:22:48.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83a77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83a8156500 0x7f83a814a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83a77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83a8156500 0x7f83a814a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45654/0 (socket says 172.21.15.1:45654) 2024-03-06T07:22:48.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83a77fe640 1 -- 172.21.15.1:0/4292188583 learned_addr learned my addr 172.21.15.1:0/4292188583 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:48.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83a6ffd640 1 --2- 172.21.15.1:0/4292188583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a806b510 0x7f83a814d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83a67fc640 1 --2- 172.21.15.1:0/4292188583 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83a814cdc0 0x7f83a8149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83a77fe640 1 -- 172.21.15.1:0/4292188583 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83a814cdc0 msgr2=0x7f83a8149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83a77fe640 1 --2- 172.21.15.1:0/4292188583 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83a814cdc0 0x7f83a8149c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83a77fe640 1 -- 172.21.15.1:0/4292188583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a806b510 msgr2=0x7f83a814d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83a77fe640 1 --2- 172.21.15.1:0/4292188583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a806b510 0x7f83a814d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83a77fe640 1 -- 172.21.15.1:0/4292188583 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a81ea640 con 0x7f83a8156500 2024-03-06T07:22:48.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83a6ffd640 1 --2- 172.21.15.1:0/4292188583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a806b510 0x7f83a814d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:48.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.010+0000 7f83a67fc640 1 --2- 172.21.15.1:0/4292188583 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83a814cdc0 0x7f83a8149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:48.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.011+0000 7f83a77fe640 1 --2- 172.21.15.1:0/4292188583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83a8156500 0x7f83a814a160 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f839800c710 tx=0x7f839800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.011+0000 7f839f7fe640 1 -- 172.21.15.1:0/4292188583 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8398017070 con 0x7f83a8156500 2024-03-06T07:22:48.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.011+0000 7f839f7fe640 1 -- 172.21.15.1:0/4292188583 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839800ce20 con 0x7f83a8156500 2024-03-06T07:22:48.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.011+0000 7f83ad639640 1 -- 172.21.15.1:0/4292188583 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a80acbc0 con 0x7f83a8156500 2024-03-06T07:22:48.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.011+0000 7f839f7fe640 1 -- 172.21.15.1:0/4292188583 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839800a860 con 0x7f83a8156500 2024-03-06T07:22:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.011+0000 7f83ad639640 1 -- 172.21.15.1:0/4292188583 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a814db90 con 0x7f83a8156500 2024-03-06T07:22:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.012+0000 7f83ad639640 1 -- 172.21.15.1:0/4292188583 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a80c3e60 con 0x7f83a8156500 2024-03-06T07:22:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.012+0000 7f839f7fe640 1 -- 172.21.15.1:0/4292188583 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f839801a070 con 0x7f83a8156500 2024-03-06T07:22:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.013+0000 7f839f7fe640 1 --2- 172.21.15.1:0/4292188583 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f838806b500 0x7f838806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.013+0000 7f839f7fe640 1 -- 172.21.15.1:0/4292188583 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f8398089770 con 0x7f83a8156500 2024-03-06T07:22:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.013+0000 7f83a6ffd640 1 --2- 172.21.15.1:0/4292188583 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f838806b500 0x7f838806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.013+0000 7f83a6ffd640 1 --2- 172.21.15.1:0/4292188583 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f838806b500 0x7f838806d9c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f83a0036000 tx=0x7f83a0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.017+0000 7f839f7fe640 1 -- 172.21.15.1:0/4292188583 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83a80c3e60 con 0x7f83a8156500 2024-03-06T07:22:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.198+0000 7f83ad639640 1 -- 172.21.15.1:0/4292188583 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7f83a80a7210 con 0x7f83a8156500 2024-03-06T07:22:48.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.201+0000 7f839f7fe640 1 -- 172.21.15.1:0/4292188583 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v20) v1 ==== 96+0+0 (secure 0 0 0) 0x7f83a80a7210 con 0x7f83a8156500 2024-03-06T07:22:48.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.203+0000 7f83ad639640 1 -- 172.21.15.1:0/4292188583 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f838806b500 msgr2=0x7f838806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.203+0000 7f83ad639640 1 --2- 172.21.15.1:0/4292188583 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f838806b500 0x7f838806d9c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f83a0036000 tx=0x7f83a0008030 comp rx=0 tx=0).stop 2024-03-06T07:22:48.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.203+0000 7f83ad639640 1 -- 172.21.15.1:0/4292188583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83a8156500 msgr2=0x7f83a814a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.203+0000 7f83ad639640 1 --2- 172.21.15.1:0/4292188583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83a8156500 0x7f83a814a160 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f839800c710 tx=0x7f839800cbd0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.203+0000 7f83ad639640 1 -- 172.21.15.1:0/4292188583 shutdown_connections 2024-03-06T07:22:48.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.203+0000 7f83ad639640 1 --2- 172.21.15.1:0/4292188583 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f838806b500 0x7f838806d9c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.203+0000 7f83ad639640 1 --2- 172.21.15.1:0/4292188583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83a8156500 0x7f83a814a160 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.203+0000 7f83ad639640 1 --2- 172.21.15.1:0/4292188583 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83a814cdc0 0x7f83a8149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.203+0000 7f83ad639640 1 --2- 172.21.15.1:0/4292188583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83a806b510 0x7f83a814d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.203+0000 7f83ad639640 1 -- 172.21.15.1:0/4292188583 >> 172.21.15.1:0/4292188583 conn(0x7f83a80a9700 msgr2=0x7f83a8072e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.203+0000 7f83ad639640 1 -- 172.21.15.1:0/4292188583 shutdown_connections 2024-03-06T07:22:48.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.203+0000 7f83ad639640 1 -- 172.21.15.1:0/4292188583 wait complete. 2024-03-06T07:22:48.217 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.a mds_join_fs 2024-03-06T07:22:48.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.318+0000 7fe95a440640 1 Processor -- start 2024-03-06T07:22:48.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.318+0000 7fe95a440640 1 -- start start 2024-03-06T07:22:48.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.318+0000 7fe95a440640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe9540af640 0x7fe954160aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe95a440640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9540c71c0 con 0x7fe9540c3eb0 2024-03-06T07:22:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe95a440640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9540c4e50 con 0x7fe95406d060 2024-03-06T07:22:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe95a440640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9540c59b0 con 0x7fe9540af640 2024-03-06T07:22:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe953fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe9540c3eb0 0x7fe9540c7400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58842/0 (socket says 172.21.15.1:58842) 2024-03-06T07:22:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe953fff640 1 -- 172.21.15.1:0/3923542046 learned_addr learned my addr 172.21.15.1:0/3923542046 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe9589dd640 1 --2- 172.21.15.1:0/3923542046 >> v2:172.21.15.98:3300/0 conn(0x7fe9540af640 0x7fe954160aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe952ffd640 1 -- 172.21.15.1:0/3923542046 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571704039 0 0) 0x7fe9540c71c0 con 0x7fe9540c3eb0 2024-03-06T07:22:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe952ffd640 1 -- 172.21.15.1:0/3923542046 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe93c003520 con 0x7fe9540c3eb0 2024-03-06T07:22:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe952ffd640 1 -- 172.21.15.1:0/3923542046 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4129031040 0 0) 0x7fe93c003520 con 0x7fe9540c3eb0 2024-03-06T07:22:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe952ffd640 1 -- 172.21.15.1:0/3923542046 >> v2:172.21.15.98:3300/0 conn(0x7fe9540af640 msgr2=0x7fe954160aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe952ffd640 1 --2- 172.21.15.1:0/3923542046 >> v2:172.21.15.98:3300/0 conn(0x7fe9540af640 0x7fe954160aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe952ffd640 1 -- 172.21.15.1:0/3923542046 >> v1:172.21.15.98:6789/0 conn(0x7fe95406d060 legacy=0x7fe9541558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe952ffd640 1 -- 172.21.15.1:0/3923542046 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9540c6ee0 con 0x7fe9540c3eb0 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe952ffd640 1 -- 172.21.15.1:0/3923542046 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fe944001670 con 0x7fe9540c3eb0 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe952ffd640 1 -- 172.21.15.1:0/3923542046 >> v1:172.21.15.1:6789/0 conn(0x7fe9540c3eb0 legacy=0x7fe9540c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe952ffd640 1 --2- 172.21.15.1:0/3923542046 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe93c0039b0 0x7fe93c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe952ffd640 1 --2- 172.21.15.1:0/3923542046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe93c004800 0x7fe93c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe952ffd640 1 --2- 172.21.15.1:0/3923542046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe93c0052e0 0x7fe93c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe952ffd640 1 -- 172.21.15.1:0/3923542046 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9540c59b0 con 0x7fe93c0052e0 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe952ffd640 1 -- 172.21.15.1:0/3923542046 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c00c1b0 con 0x7fe93c004800 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe952ffd640 1 -- 172.21.15.1:0/3923542046 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c00c3b0 con 0x7fe93c0039b0 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.319+0000 7fe9589dd640 1 --2- 172.21.15.1:0/3923542046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe93c0052e0 0x7fe93c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.320+0000 7fe953fff640 1 --2- 172.21.15.1:0/3923542046 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe93c0039b0 0x7fe93c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.320+0000 7fe9537fe640 1 --2- 172.21.15.1:0/3923542046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe93c004800 0x7fe93c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.320+0000 7fe9589dd640 1 -- 172.21.15.1:0/3923542046 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe93c0039b0 msgr2=0x7fe93c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.320+0000 7fe9589dd640 1 --2- 172.21.15.1:0/3923542046 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe93c0039b0 0x7fe93c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.320+0000 7fe9589dd640 1 -- 172.21.15.1:0/3923542046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe93c004800 msgr2=0x7fe93c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.320+0000 7fe9589dd640 1 --2- 172.21.15.1:0/3923542046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe93c004800 0x7fe93c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.320+0000 7fe9589dd640 1 -- 172.21.15.1:0/3923542046 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c003520 con 0x7fe93c0052e0 2024-03-06T07:22:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.320+0000 7fe9589dd640 1 -- 172.21.15.1:0/3923542046 reap_dead start 2024-03-06T07:22:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.321+0000 7fe9589dd640 1 --2- 172.21.15.1:0/3923542046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe93c0052e0 0x7fe93c00b9c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe94c009f90 tx=0x7fe93c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eacbac939833201d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.321+0000 7fe952ffd640 1 -- 172.21.15.1:0/3923542046 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94c009c10 con 0x7fe93c0052e0 2024-03-06T07:22:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.321+0000 7fe952ffd640 1 -- 172.21.15.1:0/3923542046 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94c02cc20 con 0x7fe93c0052e0 2024-03-06T07:22:48.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.321+0000 7fe952ffd640 1 -- 172.21.15.1:0/3923542046 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94c003990 con 0x7fe93c0052e0 2024-03-06T07:22:48.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.321+0000 7fe95a440640 1 -- 172.21.15.1:0/3923542046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe93c0052e0 msgr2=0x7fe93c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.321+0000 7fe95a440640 1 --2- 172.21.15.1:0/3923542046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe93c0052e0 0x7fe93c00b9c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe94c009f90 tx=0x7fe93c005830 comp rx=0 tx=0).stop 2024-03-06T07:22:48.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.321+0000 7fe95a440640 1 -- 172.21.15.1:0/3923542046 shutdown_connections 2024-03-06T07:22:48.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.321+0000 7fe95a440640 1 --2- 172.21.15.1:0/3923542046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe93c0052e0 0x7fe93c00b9c0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.321+0000 7fe95a440640 1 -- 172.21.15.1:0/3923542046 >> 172.21.15.1:0/3923542046 conn(0x7fe954066d00 msgr2=0x7fe954067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.321+0000 7fe95a440640 1 -- 172.21.15.1:0/3923542046 shutdown_connections 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.321+0000 7fe95a440640 1 -- 172.21.15.1:0/3923542046 wait complete. 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.322+0000 7fe95a440640 1 Processor -- start 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.323+0000 7fe95a440640 1 -- start start 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.323+0000 7fe95a440640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe93c0052e0 0x7fe954157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.323+0000 7fe95a440640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9541eb680 0x7fe95414df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.323+0000 7fe95a440640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9541eded0 0x7fe9541ee380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.323+0000 7fe95a440640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe954155420 con 0x7fe9541eb680 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.323+0000 7fe95a440640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9540c4c00 con 0x7fe9541eded0 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.323+0000 7fe95a440640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9540afb70 con 0x7fe93c0052e0 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.323+0000 7fe9537fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9541eb680 0x7fe95414df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.323+0000 7fe9537fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9541eb680 0x7fe95414df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45670/0 (socket says 172.21.15.1:45670) 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.323+0000 7fe9537fe640 1 -- 172.21.15.1:0/1029739993 learned_addr learned my addr 172.21.15.1:0/1029739993 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.323+0000 7fe9537fe640 1 -- 172.21.15.1:0/1029739993 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe93c0052e0 msgr2=0x7fe954157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.323+0000 7fe9537fe640 1 --2- 172.21.15.1:0/1029739993 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe93c0052e0 0x7fe954157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.324+0000 7fe9537fe640 1 -- 172.21.15.1:0/1029739993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9541eded0 msgr2=0x7fe9541ee380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.324+0000 7fe9537fe640 1 --2- 172.21.15.1:0/1029739993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9541eded0 0x7fe9541ee380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.324+0000 7fe9537fe640 1 -- 172.21.15.1:0/1029739993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9541ea8e0 con 0x7fe9541eb680 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.324+0000 7fe9537fe640 1 --2- 172.21.15.1:0/1029739993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9541eb680 0x7fe95414df70 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe94800f5c0 tx=0x7fe94800fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.324+0000 7fe950ff9640 1 -- 172.21.15.1:0/1029739993 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9480052e0 con 0x7fe9541eb680 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.324+0000 7fe950ff9640 1 -- 172.21.15.1:0/1029739993 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe948018070 con 0x7fe9541eb680 2024-03-06T07:22:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.324+0000 7fe95a440640 1 -- 172.21.15.1:0/1029739993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe954157f70 con 0x7fe9541eb680 2024-03-06T07:22:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.324+0000 7fe950ff9640 1 -- 172.21.15.1:0/1029739993 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe948003830 con 0x7fe9541eb680 2024-03-06T07:22:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.324+0000 7fe95a440640 1 -- 172.21.15.1:0/1029739993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9541eae80 con 0x7fe9541eb680 2024-03-06T07:22:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.325+0000 7fe95a440640 1 -- 172.21.15.1:0/1029739993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe95406ce80 con 0x7fe9541eb680 2024-03-06T07:22:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.325+0000 7fe950ff9640 1 -- 172.21.15.1:0/1029739993 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9480039d0 con 0x7fe9541eb680 2024-03-06T07:22:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.326+0000 7fe950ff9640 1 --2- 172.21.15.1:0/1029739993 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe94006b4d0 0x7fe94006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.326+0000 7fe950ff9640 1 -- 172.21.15.1:0/1029739993 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fe948089f00 con 0x7fe9541eb680 2024-03-06T07:22:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.326+0000 7fe953fff640 1 --2- 172.21.15.1:0/1029739993 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe94006b4d0 0x7fe94006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.327+0000 7fe953fff640 1 --2- 172.21.15.1:0/1029739993 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe94006b4d0 0x7fe94006d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe944001b60 tx=0x7fe944001ad0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.330+0000 7fe950ff9640 1 -- 172.21.15.1:0/1029739993 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe95406ce80 con 0x7fe9541eb680 2024-03-06T07:22:48.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.510+0000 7fe95a440640 1 -- 172.21.15.1:0/1029739993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"} v 0) v1 -- 0x7fe9540c7400 con 0x7fe9541eb680 2024-03-06T07:22:48.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.513+0000 7fe950ff9640 1 -- 172.21.15.1:0/1029739993 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"}]=0 v21) v1 ==== 96+0+0 (secure 0 0 0) 0x7fe9540c7400 con 0x7fe9541eb680 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.515+0000 7fe95a440640 1 -- 172.21.15.1:0/1029739993 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe94006b4d0 msgr2=0x7fe94006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.515+0000 7fe95a440640 1 --2- 172.21.15.1:0/1029739993 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe94006b4d0 0x7fe94006d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe944001b60 tx=0x7fe944001ad0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.515+0000 7fe95a440640 1 -- 172.21.15.1:0/1029739993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9541eb680 msgr2=0x7fe95414df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.515+0000 7fe95a440640 1 --2- 172.21.15.1:0/1029739993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9541eb680 0x7fe95414df70 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe94800f5c0 tx=0x7fe94800fa80 comp rx=0 tx=0).stop 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.515+0000 7fe95a440640 1 -- 172.21.15.1:0/1029739993 shutdown_connections 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.515+0000 7fe95a440640 1 --2- 172.21.15.1:0/1029739993 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe94006b4d0 0x7fe94006d990 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.515+0000 7fe95a440640 1 --2- 172.21.15.1:0/1029739993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9541eded0 0x7fe9541ee380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.515+0000 7fe95a440640 1 --2- 172.21.15.1:0/1029739993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9541eb680 0x7fe95414df70 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.515+0000 7fe95a440640 1 --2- 172.21.15.1:0/1029739993 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe93c0052e0 0x7fe954157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.515+0000 7fe95a440640 1 -- 172.21.15.1:0/1029739993 >> 172.21.15.1:0/1029739993 conn(0x7fe954066d00 msgr2=0x7fe9541512b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.515+0000 7fe95a440640 1 -- 172.21.15.1:0/1029739993 shutdown_connections 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.515+0000 7fe95a440640 1 -- 172.21.15.1:0/1029739993 wait complete. 2024-03-06T07:22:48.528 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-03-06T07:22:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.630+0000 7f3061d81640 1 Processor -- start 2024-03-06T07:22:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.630+0000 7f3061d81640 1 -- start start 2024-03-06T07:22:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f3061d81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f305c06d060 0x7f305c1558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f3061d81640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f305c0c71c0 con 0x7f305c0c3eb0 2024-03-06T07:22:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f3061d81640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f305c0c4e50 con 0x7f305c0af640 2024-03-06T07:22:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f3061d81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f305c0c59b0 con 0x7f305c06d060 2024-03-06T07:22:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f305b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f305c0c3eb0 0x7f305c0c7400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58854/0 (socket says 172.21.15.1:58854) 2024-03-06T07:22:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f305b7fe640 1 -- 172.21.15.1:0/359312302 learned_addr learned my addr 172.21.15.1:0/359312302 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f305affd640 1 -- 172.21.15.1:0/359312302 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971202491 0 0) 0x7f305c0c71c0 con 0x7f305c0c3eb0 2024-03-06T07:22:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f305affd640 1 -- 172.21.15.1:0/359312302 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3048003520 con 0x7f305c0c3eb0 2024-03-06T07:22:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f3053fff640 1 --2- 172.21.15.1:0/359312302 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f305c06d060 0x7f305c1558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f305affd640 1 -- 172.21.15.1:0/359312302 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3303174956 0 0) 0x7f3048003520 con 0x7f305c0c3eb0 2024-03-06T07:22:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f305affd640 1 -- 172.21.15.1:0/359312302 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f305c06d060 msgr2=0x7f305c1558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f305affd640 1 --2- 172.21.15.1:0/359312302 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f305c06d060 0x7f305c1558b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f305affd640 1 -- 172.21.15.1:0/359312302 >> v1:172.21.15.98:6789/0 conn(0x7f305c0af640 legacy=0x7f305c15fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f305affd640 1 -- 172.21.15.1:0/359312302 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c0c6ee0 con 0x7f305c0c3eb0 2024-03-06T07:22:48.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f305affd640 1 -- 172.21.15.1:0/359312302 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f304c001670 con 0x7f305c0c3eb0 2024-03-06T07:22:48.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f305affd640 1 -- 172.21.15.1:0/359312302 >> v1:172.21.15.1:6789/0 conn(0x7f305c0c3eb0 legacy=0x7f305c0c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f305affd640 1 --2- 172.21.15.1:0/359312302 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f305c06d060 0x7f30480039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f305affd640 1 --2- 172.21.15.1:0/359312302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3048003f60 0x7f3048004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f305affd640 1 --2- 172.21.15.1:0/359312302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3048004d40 0x7f30480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f305affd640 1 -- 172.21.15.1:0/359312302 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c0c59b0 con 0x7f3048003f60 2024-03-06T07:22:48.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f305affd640 1 -- 172.21.15.1:0/359312302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304800bc00 con 0x7f3048004d40 2024-03-06T07:22:48.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.631+0000 7f305affd640 1 -- 172.21.15.1:0/359312302 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304800be00 con 0x7f305c06d060 2024-03-06T07:22:48.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.632+0000 7f3053fff640 1 --2- 172.21.15.1:0/359312302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3048003f60 0x7f3048004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.632+0000 7f305b7fe640 1 --2- 172.21.15.1:0/359312302 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f305c06d060 0x7f30480039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.632+0000 7f305bfff640 1 --2- 172.21.15.1:0/359312302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3048004d40 0x7f30480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.632+0000 7f3053fff640 1 -- 172.21.15.1:0/359312302 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f305c06d060 msgr2=0x7f30480039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.632+0000 7f3053fff640 1 --2- 172.21.15.1:0/359312302 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f305c06d060 0x7f30480039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.632+0000 7f3053fff640 1 -- 172.21.15.1:0/359312302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3048004d40 msgr2=0x7f30480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.632+0000 7f3053fff640 1 --2- 172.21.15.1:0/359312302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3048004d40 0x7f30480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.632+0000 7f3053fff640 1 -- 172.21.15.1:0/359312302 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3048003520 con 0x7f3048003f60 2024-03-06T07:22:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.633+0000 7f3053fff640 1 --2- 172.21.15.1:0/359312302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3048003f60 0x7f3048004800 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f3044009a90 tx=0x7f3048005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a34f6c80d990087 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.633+0000 7f305affd640 1 -- 172.21.15.1:0/359312302 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3044039070 con 0x7f3048003f60 2024-03-06T07:22:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.633+0000 7f305affd640 1 -- 172.21.15.1:0/359312302 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304402c680 con 0x7f3048003f60 2024-03-06T07:22:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.633+0000 7f305affd640 1 -- 172.21.15.1:0/359312302 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3044009690 con 0x7f3048003f60 2024-03-06T07:22:48.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.633+0000 7f3061d81640 1 -- 172.21.15.1:0/359312302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3048003f60 msgr2=0x7f3048004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.633+0000 7f3061d81640 1 --2- 172.21.15.1:0/359312302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3048003f60 0x7f3048004800 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f3044009a90 tx=0x7f3048005830 comp rx=0 tx=0).stop 2024-03-06T07:22:48.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.633+0000 7f305bfff640 1 -- 172.21.15.1:0/359312302 reap_dead start 2024-03-06T07:22:48.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.633+0000 7f3061d81640 1 -- 172.21.15.1:0/359312302 shutdown_connections 2024-03-06T07:22:48.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.633+0000 7f3061d81640 1 -- 172.21.15.1:0/359312302 >> 172.21.15.1:0/359312302 conn(0x7f305c066d00 msgr2=0x7f305c067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.633+0000 7f3061d81640 1 -- 172.21.15.1:0/359312302 shutdown_connections 2024-03-06T07:22:48.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.633+0000 7f3061d81640 1 -- 172.21.15.1:0/359312302 wait complete. 2024-03-06T07:22:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.634+0000 7f3061d81640 1 Processor -- start 2024-03-06T07:22:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.635+0000 7f3061d81640 1 -- start start 2024-03-06T07:22:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.635+0000 7f3061d81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3048003f60 0x7f305c1ee040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.635+0000 7f3061d81640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f305c14bf30 0x7f305c151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.635+0000 7f3061d81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f305c1eb2e0 0x7f305c1e0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.635+0000 7f3061d81640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c155420 con 0x7f305c14bf30 2024-03-06T07:22:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.635+0000 7f3061d81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c0c4c00 con 0x7f305c1eb2e0 2024-03-06T07:22:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.635+0000 7f3061d81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f305c0c4290 con 0x7f3048003f60 2024-03-06T07:22:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.635+0000 7f3053fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f305c14bf30 0x7f305c151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.635+0000 7f3053fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f305c14bf30 0x7f305c151750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45686/0 (socket says 172.21.15.1:45686) 2024-03-06T07:22:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.635+0000 7f3053fff640 1 -- 172.21.15.1:0/943201823 learned_addr learned my addr 172.21.15.1:0/943201823 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.635+0000 7f305b7fe640 1 --2- 172.21.15.1:0/943201823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3048003f60 0x7f305c1ee040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.635+0000 7f305bfff640 1 --2- 172.21.15.1:0/943201823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f305c1eb2e0 0x7f305c1e0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.636+0000 7f3053fff640 1 -- 172.21.15.1:0/943201823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3048003f60 msgr2=0x7f305c1ee040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.636+0000 7f3053fff640 1 --2- 172.21.15.1:0/943201823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3048003f60 0x7f305c1ee040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.636+0000 7f3053fff640 1 -- 172.21.15.1:0/943201823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f305c1eb2e0 msgr2=0x7f305c1e0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.636+0000 7f3053fff640 1 --2- 172.21.15.1:0/943201823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f305c1eb2e0 0x7f305c1e0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.636+0000 7f3053fff640 1 -- 172.21.15.1:0/943201823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c159450 con 0x7f305c14bf30 2024-03-06T07:22:48.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.636+0000 7f305b7fe640 1 --2- 172.21.15.1:0/943201823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3048003f60 0x7f305c1ee040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:48.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.636+0000 7f305bfff640 1 --2- 172.21.15.1:0/943201823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f305c1eb2e0 0x7f305c1e0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:48.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.636+0000 7f3053fff640 1 --2- 172.21.15.1:0/943201823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f305c14bf30 0x7f305c151750 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f3044000c00 tx=0x7f3044004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.636+0000 7f3058ff9640 1 -- 172.21.15.1:0/943201823 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3044042020 con 0x7f305c14bf30 2024-03-06T07:22:48.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.636+0000 7f3058ff9640 1 -- 172.21.15.1:0/943201823 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304403a040 con 0x7f305c14bf30 2024-03-06T07:22:48.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.636+0000 7f3061d81640 1 -- 172.21.15.1:0/943201823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f305c0ca930 con 0x7f305c14bf30 2024-03-06T07:22:48.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.636+0000 7f3058ff9640 1 -- 172.21.15.1:0/943201823 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3044039070 con 0x7f305c14bf30 2024-03-06T07:22:48.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.636+0000 7f3061d81640 1 -- 172.21.15.1:0/943201823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f305c0caef0 con 0x7f305c14bf30 2024-03-06T07:22:48.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.637+0000 7f3061d81640 1 -- 172.21.15.1:0/943201823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f305c0c3a60 con 0x7f305c14bf30 2024-03-06T07:22:48.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.637+0000 7f3058ff9640 1 -- 172.21.15.1:0/943201823 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3044045020 con 0x7f305c14bf30 2024-03-06T07:22:48.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.638+0000 7f3058ff9640 1 --2- 172.21.15.1:0/943201823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f303c06b500 0x7f303c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.638+0000 7f3058ff9640 1 -- 172.21.15.1:0/943201823 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f30440ab5b0 con 0x7f305c14bf30 2024-03-06T07:22:48.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.638+0000 7f305b7fe640 1 --2- 172.21.15.1:0/943201823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f303c06b500 0x7f303c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.638+0000 7f305b7fe640 1 --2- 172.21.15.1:0/943201823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f303c06b500 0x7f303c06d9c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f305c14e150 tx=0x7f304c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.642+0000 7f3058ff9640 1 -- 172.21.15.1:0/943201823 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f305c0c3a60 con 0x7f305c14bf30 2024-03-06T07:22:48.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.823+0000 7f3061d81640 1 -- 172.21.15.1:0/943201823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7f305c067fd0 con 0x7f305c14bf30 2024-03-06T07:22:48.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.826+0000 7f3058ff9640 1 -- 172.21.15.1:0/943201823 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v22) v1 ==== 96+0+0 (secure 0 0 0) 0x7f304404c070 con 0x7f305c14bf30 2024-03-06T07:22:48.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.828+0000 7f3061d81640 1 -- 172.21.15.1:0/943201823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f303c06b500 msgr2=0x7f303c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.828+0000 7f3061d81640 1 --2- 172.21.15.1:0/943201823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f303c06b500 0x7f303c06d9c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f305c14e150 tx=0x7f304c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:48.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.828+0000 7f3061d81640 1 -- 172.21.15.1:0/943201823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f305c14bf30 msgr2=0x7f305c151750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.828+0000 7f3061d81640 1 --2- 172.21.15.1:0/943201823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f305c14bf30 0x7f305c151750 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f3044000c00 tx=0x7f3044004730 comp rx=0 tx=0).stop 2024-03-06T07:22:48.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.828+0000 7f3061d81640 1 -- 172.21.15.1:0/943201823 shutdown_connections 2024-03-06T07:22:48.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.828+0000 7f3061d81640 1 --2- 172.21.15.1:0/943201823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f303c06b500 0x7f303c06d9c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.828+0000 7f3061d81640 1 --2- 172.21.15.1:0/943201823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f305c1eb2e0 0x7f305c1e0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.828+0000 7f3061d81640 1 --2- 172.21.15.1:0/943201823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f305c14bf30 0x7f305c151750 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.828+0000 7f3061d81640 1 --2- 172.21.15.1:0/943201823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3048003f60 0x7f305c1ee040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.828+0000 7f3061d81640 1 -- 172.21.15.1:0/943201823 >> 172.21.15.1:0/943201823 conn(0x7f305c066d00 msgr2=0x7f305c0c0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.828+0000 7f3061d81640 1 -- 172.21.15.1:0/943201823 shutdown_connections 2024-03-06T07:22:48.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.828+0000 7f3061d81640 1 -- 172.21.15.1:0/943201823 wait complete. 2024-03-06T07:22:48.841 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort' 2024-03-06T07:22:48.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.940+0000 7fe47dfd7640 1 Processor -- start 2024-03-06T07:22:48.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.940+0000 7fe47dfd7640 1 -- start start 2024-03-06T07:22:48.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.941+0000 7fe47dfd7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe478152650 0x7fe478155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.941+0000 7fe47dfd7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4780bf9d0 0x7fe4780c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.941+0000 7fe47dfd7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4780c2fc0 con 0x7fe4780c6b10 2024-03-06T07:22:48.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.941+0000 7fe47dfd7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe478155800 con 0x7fe478152650 2024-03-06T07:22:48.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.941+0000 7fe47dfd7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe478158da0 con 0x7fe4780bf9d0 2024-03-06T07:22:48.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.941+0000 7fe4777fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe478152650 0x7fe478155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.941+0000 7fe477fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4780bf9d0 0x7fe4780c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.941+0000 7fe4777fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe478152650 0x7fe478155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:53200/0 (socket says 172.21.15.1:53200) 2024-03-06T07:22:48.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.941+0000 7fe4777fe640 1 -- 172.21.15.1:0/3229829189 learned_addr learned my addr 172.21.15.1:0/3229829189 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:48.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.941+0000 7fe477fff640 1 -- 172.21.15.1:0/3229829189 >> v2:172.21.15.98:3300/0 conn(0x7fe478152650 msgr2=0x7fe478155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.941+0000 7fe477fff640 1 --2- 172.21.15.1:0/3229829189 >> v2:172.21.15.98:3300/0 conn(0x7fe478152650 0x7fe478155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.941+0000 7fe477fff640 1 -- 172.21.15.1:0/3229829189 >> v1:172.21.15.98:6789/0 conn(0x7fe4780c6b10 legacy=0x7fe478158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.941+0000 7fe477fff640 1 -- 172.21.15.1:0/3229829189 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4780c6690 con 0x7fe4780bf9d0 2024-03-06T07:22:48.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.941+0000 7fe4777fe640 1 --2- 172.21.15.1:0/3229829189 >> v2:172.21.15.98:3300/0 conn(0x7fe478152650 0x7fe478155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:48.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.942+0000 7fe477fff640 1 --2- 172.21.15.1:0/3229829189 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4780bf9d0 0x7fe4780c1460 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe470009d90 tx=0x7fe47002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f118ccd061fdaf36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.942+0000 7fe4767fc640 1 -- 172.21.15.1:0/3229829189 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470039070 con 0x7fe4780bf9d0 2024-03-06T07:22:48.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.942+0000 7fe4767fc640 1 -- 172.21.15.1:0/3229829189 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47002cbc0 con 0x7fe4780bf9d0 2024-03-06T07:22:48.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.942+0000 7fe4767fc640 1 -- 172.21.15.1:0/3229829189 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470034440 con 0x7fe4780bf9d0 2024-03-06T07:22:48.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.943+0000 7fe47dfd7640 1 -- 172.21.15.1:0/3229829189 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4780bf9d0 msgr2=0x7fe4780c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.943+0000 7fe47dfd7640 1 --2- 172.21.15.1:0/3229829189 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4780bf9d0 0x7fe4780c1460 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe470009d90 tx=0x7fe47002be10 comp rx=0 tx=0).stop 2024-03-06T07:22:48.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.943+0000 7fe47dfd7640 1 -- 172.21.15.1:0/3229829189 shutdown_connections 2024-03-06T07:22:48.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.943+0000 7fe47dfd7640 1 --2- 172.21.15.1:0/3229829189 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4780bf9d0 0x7fe4780c1460 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.943+0000 7fe47dfd7640 1 --2- 172.21.15.1:0/3229829189 >> v2:172.21.15.98:3300/0 conn(0x7fe478152650 0x7fe478155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.943+0000 7fe47dfd7640 1 -- 172.21.15.1:0/3229829189 >> 172.21.15.1:0/3229829189 conn(0x7fe478066ee0 msgr2=0x7fe4780672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.943+0000 7fe47dfd7640 1 -- 172.21.15.1:0/3229829189 shutdown_connections 2024-03-06T07:22:48.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.943+0000 7fe47dfd7640 1 -- 172.21.15.1:0/3229829189 wait complete. 2024-03-06T07:22:48.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.944+0000 7fe47dfd7640 1 Processor -- start 2024-03-06T07:22:48.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.944+0000 7fe47dfd7640 1 -- start start 2024-03-06T07:22:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe47dfd7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4780bf9d0 0x7fe4781586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe47dfd7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe478152650 0x7fe478151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe47dfd7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4781506d0 0x7fe4781ee310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe47dfd7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4781551b0 con 0x7fe478152650 2024-03-06T07:22:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe47dfd7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4780c6fa0 con 0x7fe4780bf9d0 2024-03-06T07:22:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe47dfd7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe478155490 con 0x7fe4781506d0 2024-03-06T07:22:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe476ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe478152650 0x7fe478151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe476ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe478152650 0x7fe478151a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45690/0 (socket says 172.21.15.1:45690) 2024-03-06T07:22:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe476ffd640 1 -- 172.21.15.1:0/4175621520 learned_addr learned my addr 172.21.15.1:0/4175621520 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe4777fe640 1 --2- 172.21.15.1:0/4175621520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4780bf9d0 0x7fe4781586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe477fff640 1 --2- 172.21.15.1:0/4175621520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4781506d0 0x7fe4781ee310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe476ffd640 1 -- 172.21.15.1:0/4175621520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4781506d0 msgr2=0x7fe4781ee310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe476ffd640 1 --2- 172.21.15.1:0/4175621520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4781506d0 0x7fe4781ee310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe476ffd640 1 -- 172.21.15.1:0/4175621520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4780bf9d0 msgr2=0x7fe4781586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe476ffd640 1 --2- 172.21.15.1:0/4175621520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4780bf9d0 0x7fe4781586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe476ffd640 1 -- 172.21.15.1:0/4175621520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4781eedc0 con 0x7fe478152650 2024-03-06T07:22:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe477fff640 1 --2- 172.21.15.1:0/4175621520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4781506d0 0x7fe4781ee310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe476ffd640 1 --2- 172.21.15.1:0/4175621520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe478152650 0x7fe478151a60 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fe46800c430 tx=0x7fe46800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe45ffff640 1 -- 172.21.15.1:0/4175621520 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468017070 con 0x7fe478152650 2024-03-06T07:22:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.945+0000 7fe45ffff640 1 -- 172.21.15.1:0/4175621520 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4680052d0 con 0x7fe478152650 2024-03-06T07:22:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.946+0000 7fe47dfd7640 1 -- 172.21.15.1:0/4175621520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4781ef0a0 con 0x7fe478152650 2024-03-06T07:22:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.946+0000 7fe45ffff640 1 -- 172.21.15.1:0/4175621520 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468003850 con 0x7fe478152650 2024-03-06T07:22:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.946+0000 7fe47dfd7640 1 -- 172.21.15.1:0/4175621520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe478157040 con 0x7fe478152650 2024-03-06T07:22:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.947+0000 7fe47dfd7640 1 -- 172.21.15.1:0/4175621520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4780b06f0 con 0x7fe478152650 2024-03-06T07:22:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.947+0000 7fe45ffff640 1 -- 172.21.15.1:0/4175621520 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe46801a070 con 0x7fe478152650 2024-03-06T07:22:48.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.947+0000 7fe45ffff640 1 --2- 172.21.15.1:0/4175621520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe46406b500 0x7fe46406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.947+0000 7fe45ffff640 1 -- 172.21.15.1:0/4175621520 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fe468089440 con 0x7fe478152650 2024-03-06T07:22:48.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.948+0000 7fe4777fe640 1 --2- 172.21.15.1:0/4175621520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe46406b500 0x7fe46406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.948+0000 7fe4777fe640 1 --2- 172.21.15.1:0/4175621520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe46406b500 0x7fe46406d9c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe46c009630 tx=0x7fe46c009240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:48.952+0000 7fe45ffff640 1 -- 172.21.15.1:0/4175621520 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe468057d20 con 0x7fe478152650 2024-03-06T07:22:49.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.202+0000 7fe47dfd7640 1 -- 172.21.15.1:0/4175621520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]} v 0) v1 -- 0x7fe4781566a0 con 0x7fe478152650 2024-03-06T07:22:49.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.569+0000 7fe45ffff640 1 -- 172.21.15.1:0/4175621520 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]}]=0 v157) v1 ==== 150+0+0 (secure 0 0 0) 0x7fe46805bdc0 con 0x7fe478152650 2024-03-06T07:22:49.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.571+0000 7fe47dfd7640 1 -- 172.21.15.1:0/4175621520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe46406b500 msgr2=0x7fe46406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.571+0000 7fe47dfd7640 1 --2- 172.21.15.1:0/4175621520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe46406b500 0x7fe46406d9c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe46c009630 tx=0x7fe46c009240 comp rx=0 tx=0).stop 2024-03-06T07:22:49.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.571+0000 7fe47dfd7640 1 -- 172.21.15.1:0/4175621520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe478152650 msgr2=0x7fe478151a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.571+0000 7fe47dfd7640 1 --2- 172.21.15.1:0/4175621520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe478152650 0x7fe478151a60 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fe46800c430 tx=0x7fe46800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.571+0000 7fe47dfd7640 1 -- 172.21.15.1:0/4175621520 shutdown_connections 2024-03-06T07:22:49.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.571+0000 7fe47dfd7640 1 --2- 172.21.15.1:0/4175621520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe46406b500 0x7fe46406d9c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.571+0000 7fe47dfd7640 1 --2- 172.21.15.1:0/4175621520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4781506d0 0x7fe4781ee310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.571+0000 7fe47dfd7640 1 --2- 172.21.15.1:0/4175621520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe478152650 0x7fe478151a60 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.571+0000 7fe47dfd7640 1 --2- 172.21.15.1:0/4175621520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4780bf9d0 0x7fe4781586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.571+0000 7fe47dfd7640 1 -- 172.21.15.1:0/4175621520 >> 172.21.15.1:0/4175621520 conn(0x7fe478066ee0 msgr2=0x7fe4780c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.571+0000 7fe47dfd7640 1 -- 172.21.15.1:0/4175621520 shutdown_connections 2024-03-06T07:22:49.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.571+0000 7fe47dfd7640 1 -- 172.21.15.1:0/4175621520 wait complete. 2024-03-06T07:22:49.585 INFO:tasks.cephfs_test_runner:That a standby with mds_join_fs set to another fs is still used if necessary. ... ok 2024-03-06T07:22:49.585 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_last_resort_refused (tasks.cephfs.test_failover.TestClusterAffinity) That a standby with mds_join_fs set to another fs is not used if refuse_standby_for_another_fs is set. 2024-03-06T07:22:49.585 INFO:tasks.cephfs_test_runner:test_join_fs_last_resort_refused (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-06T07:22:49.586 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:49.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.687+0000 7f57ba9a0640 1 Processor -- start 2024-03-06T07:22:49.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.687+0000 7f57ba9a0640 1 -- start start 2024-03-06T07:22:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.688+0000 7f57ba9a0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f57b41505b0 0x7f57b40c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.688+0000 7f57ba9a0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f57b40c16e0 0x7f57b4156df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.688+0000 7f57ba9a0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b406f5c0 con 0x7f57b40c1260 2024-03-06T07:22:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.688+0000 7f57ba9a0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57b4152f80 con 0x7f57b40c16e0 2024-03-06T07:22:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.688+0000 7f57ba9a0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57b4150990 con 0x7f57b41505b0 2024-03-06T07:22:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.688+0000 7f57b8f3d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f57b40c16e0 0x7f57b4156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.688+0000 7f57b8f3d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f57b40c16e0 0x7f57b4156df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45700/0 (socket says 172.21.15.1:45700) 2024-03-06T07:22:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.688+0000 7f57b8f3d640 1 -- 172.21.15.1:0/437222668 learned_addr learned my addr 172.21.15.1:0/437222668 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.688+0000 7f57b3fff640 1 --2- 172.21.15.1:0/437222668 >> v2:172.21.15.98:3300/0 conn(0x7f57b41505b0 0x7f57b40c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.688+0000 7f57b8f3d640 1 -- 172.21.15.1:0/437222668 >> v2:172.21.15.98:3300/0 conn(0x7f57b41505b0 msgr2=0x7f57b40c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.688+0000 7f57b8f3d640 1 --2- 172.21.15.1:0/437222668 >> v2:172.21.15.98:3300/0 conn(0x7f57b41505b0 0x7f57b40c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.688+0000 7f57b8f3d640 1 -- 172.21.15.1:0/437222668 >> v1:172.21.15.98:6789/0 conn(0x7f57b40c1260 legacy=0x7f57b41566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.688+0000 7f57b8f3d640 1 -- 172.21.15.1:0/437222668 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b4153240 con 0x7f57b40c16e0 2024-03-06T07:22:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.689+0000 7f57b8f3d640 1 --2- 172.21.15.1:0/437222668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b40c16e0 0x7f57b4156df0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f57b40c1130 tx=0x7f57ac02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06bd6f3ce2e12c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.689+0000 7f57b2ffd640 1 -- 172.21.15.1:0/437222668 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac02ca80 con 0x7f57b40c16e0 2024-03-06T07:22:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.689+0000 7f57b2ffd640 1 -- 172.21.15.1:0/437222668 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ac02cc20 con 0x7f57b40c16e0 2024-03-06T07:22:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.689+0000 7f57b2ffd640 1 -- 172.21.15.1:0/437222668 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac0343f0 con 0x7f57b40c16e0 2024-03-06T07:22:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.690+0000 7f57ba9a0640 1 -- 172.21.15.1:0/437222668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b40c16e0 msgr2=0x7f57b4156df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.690+0000 7f57ba9a0640 1 --2- 172.21.15.1:0/437222668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b40c16e0 0x7f57b4156df0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f57b40c1130 tx=0x7f57ac02bee0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.690+0000 7f57ba9a0640 1 -- 172.21.15.1:0/437222668 shutdown_connections 2024-03-06T07:22:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.690+0000 7f57ba9a0640 1 --2- 172.21.15.1:0/437222668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b40c16e0 0x7f57b4156df0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.690+0000 7f57ba9a0640 1 --2- 172.21.15.1:0/437222668 >> v2:172.21.15.98:3300/0 conn(0x7f57b41505b0 0x7f57b40c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.690+0000 7f57ba9a0640 1 -- 172.21.15.1:0/437222668 >> 172.21.15.1:0/437222668 conn(0x7f57b40c6860 msgr2=0x7f57b40c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.690+0000 7f57ba9a0640 1 -- 172.21.15.1:0/437222668 shutdown_connections 2024-03-06T07:22:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.690+0000 7f57ba9a0640 1 -- 172.21.15.1:0/437222668 wait complete. 2024-03-06T07:22:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.691+0000 7f57ba9a0640 1 Processor -- start 2024-03-06T07:22:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.691+0000 7f57ba9a0640 1 -- start start 2024-03-06T07:22:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57ba9a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b40c1260 0x7f57b4073110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57ba9a0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b40c16e0 0x7f57b40bff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57ba9a0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b41505b0 0x7f57b41ee270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57ba9a0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b4153ae0 con 0x7f57b41505b0 2024-03-06T07:22:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57ba9a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b4153800 con 0x7f57b40c1260 2024-03-06T07:22:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57ba9a0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b4157330 con 0x7f57b40c16e0 2024-03-06T07:22:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57b8f3d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b41505b0 0x7f57b41ee270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57b8f3d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b41505b0 0x7f57b41ee270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45710/0 (socket says 172.21.15.1:45710) 2024-03-06T07:22:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57b8f3d640 1 -- 172.21.15.1:0/2259454316 learned_addr learned my addr 172.21.15.1:0/2259454316 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57b3fff640 1 --2- 172.21.15.1:0/2259454316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b40c1260 0x7f57b4073110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57b37fe640 1 --2- 172.21.15.1:0/2259454316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b40c16e0 0x7f57b40bff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57b8f3d640 1 -- 172.21.15.1:0/2259454316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b40c16e0 msgr2=0x7f57b40bff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57b8f3d640 1 --2- 172.21.15.1:0/2259454316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b40c16e0 0x7f57b40bff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57b8f3d640 1 -- 172.21.15.1:0/2259454316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b40c1260 msgr2=0x7f57b4073110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57b8f3d640 1 --2- 172.21.15.1:0/2259454316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b40c1260 0x7f57b4073110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57b8f3d640 1 -- 172.21.15.1:0/2259454316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b41eba60 con 0x7f57b41505b0 2024-03-06T07:22:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57b3fff640 1 --2- 172.21.15.1:0/2259454316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b40c1260 0x7f57b4073110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57b37fe640 1 --2- 172.21.15.1:0/2259454316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b40c16e0 0x7f57b40bff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57b8f3d640 1 --2- 172.21.15.1:0/2259454316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b41505b0 0x7f57b41ee270 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f57ac009f40 tx=0x7f57ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57b0ff9640 1 -- 172.21.15.1:0/2259454316 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac043020 con 0x7f57b41505b0 2024-03-06T07:22:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57b0ff9640 1 -- 172.21.15.1:0/2259454316 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ac03e450 con 0x7f57b41505b0 2024-03-06T07:22:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57b0ff9640 1 -- 172.21.15.1:0/2259454316 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac009510 con 0x7f57b41505b0 2024-03-06T07:22:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.692+0000 7f57ba9a0640 1 -- 172.21.15.1:0/2259454316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b4154280 con 0x7f57b41505b0 2024-03-06T07:22:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.693+0000 7f57ba9a0640 1 -- 172.21.15.1:0/2259454316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b40c0440 con 0x7f57b41505b0 2024-03-06T07:22:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.694+0000 7f57ba9a0640 1 -- 172.21.15.1:0/2259454316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b4067fd0 con 0x7f57b41505b0 2024-03-06T07:22:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.694+0000 7f57b0ff9640 1 -- 172.21.15.1:0/2259454316 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f57ac046020 con 0x7f57b41505b0 2024-03-06T07:22:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.694+0000 7f57b0ff9640 1 --2- 172.21.15.1:0/2259454316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f57a006b4d0 0x7f57a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.694+0000 7f57b0ff9640 1 -- 172.21.15.1:0/2259454316 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f57ac0ac8e0 con 0x7f57b41505b0 2024-03-06T07:22:49.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.695+0000 7f57b3fff640 1 --2- 172.21.15.1:0/2259454316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f57a006b4d0 0x7f57a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.695+0000 7f57b3fff640 1 --2- 172.21.15.1:0/2259454316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f57a006b4d0 0x7f57a006d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f57b4151f90 tx=0x7f57a8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.699+0000 7f57b0ff9640 1 -- 172.21.15.1:0/2259454316 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f57ac04d030 con 0x7f57b41505b0 2024-03-06T07:22:49.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.881+0000 7f57ba9a0640 1 -- 172.21.15.1:0/2259454316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57b406f280 con 0x7f57b41505b0 2024-03-06T07:22:49.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.882+0000 7f57b0ff9640 1 -- 172.21.15.1:0/2259454316 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+13686 (secure 0 0 0) 0x7f57ac07b810 con 0x7f57b41505b0 2024-03-06T07:22:49.885 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:49.885 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":65,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:47.559544+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6835/2828783606":"2024-03-07T07:22:44.550883+0000","172.21.15.1:6833/3845300501":"2024-03-07T07:22:38.960489+0000","172.21.15.1:6834/4244709430":"2024-03-07T07:22:37.955221+0000","172.21.15.1:6832/4244709430":"2024-03-07T07:22:37.955221+0000","172.21.15.98:6836/2937068915":"2024-03-07T07:22:29.247312+0000","172.21.15.1:6835/3845300501":"2024-03-07T07:22:38.960489+0000","172.21.15.98:6834/2937068915":"2024-03-07T07:22:29.247312+0000","172.21.15.98:6837/2828783606":"2024-03-07T07:22:44.550883+0000","172.21.15.1:6834/131367647":"2024-03-07T07:22:26.740110+0000","172.21.15.1:6832/131367647":"2024-03-07T07:22:26.740110+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:49.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.884+0000 7f57ba9a0640 1 -- 172.21.15.1:0/2259454316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f57a006b4d0 msgr2=0x7f57a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.884+0000 7f57ba9a0640 1 --2- 172.21.15.1:0/2259454316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f57a006b4d0 0x7f57a006d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f57b4151f90 tx=0x7f57a8008040 comp rx=0 tx=0).stop 2024-03-06T07:22:49.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.884+0000 7f57ba9a0640 1 -- 172.21.15.1:0/2259454316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b41505b0 msgr2=0x7f57b41ee270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.884+0000 7f57ba9a0640 1 --2- 172.21.15.1:0/2259454316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b41505b0 0x7f57b41ee270 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f57ac009f40 tx=0x7f57ac004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:49.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.884+0000 7f57ba9a0640 1 -- 172.21.15.1:0/2259454316 shutdown_connections 2024-03-06T07:22:49.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.884+0000 7f57ba9a0640 1 --2- 172.21.15.1:0/2259454316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f57a006b4d0 0x7f57a006d990 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.884+0000 7f57ba9a0640 1 --2- 172.21.15.1:0/2259454316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57b41505b0 0x7f57b41ee270 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.884+0000 7f57ba9a0640 1 --2- 172.21.15.1:0/2259454316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57b40c16e0 0x7f57b40bff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.884+0000 7f57ba9a0640 1 --2- 172.21.15.1:0/2259454316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57b40c1260 0x7f57b4073110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.884+0000 7f57ba9a0640 1 -- 172.21.15.1:0/2259454316 >> 172.21.15.1:0/2259454316 conn(0x7f57b40c6860 msgr2=0x7f57b40c5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.884+0000 7f57ba9a0640 1 -- 172.21.15.1:0/2259454316 shutdown_connections 2024-03-06T07:22:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:49.884+0000 7f57ba9a0640 1 -- 172.21.15.1:0/2259454316 wait complete. 2024-03-06T07:22:49.898 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:49.898 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:50.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.000+0000 7fdac3aa4640 1 Processor -- start 2024-03-06T07:22:50.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.001+0000 7fdac3aa4640 1 -- start start 2024-03-06T07:22:50.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.001+0000 7fdac3aa4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdabc152690 0x7fdabc155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.001+0000 7fdac3aa4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdabc0c6730 0x7fdabc0c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.001+0000 7fdac3aa4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdabc1561e0 con 0x7fdabc0c5d10 2024-03-06T07:22:50.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.001+0000 7fdac3aa4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdabc0c3300 con 0x7fdabc0c6730 2024-03-06T07:22:50.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.001+0000 7fdac3aa4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdabc0c35a0 con 0x7fdabc152690 2024-03-06T07:22:50.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.002+0000 7fdac103f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdabc0c6730 0x7fdabc0c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.002+0000 7fdac103f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdabc0c6730 0x7fdabc0c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45724/0 (socket says 172.21.15.1:45724) 2024-03-06T07:22:50.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.002+0000 7fdac2041640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdabc0c5d10 0x7fdabc156cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58866/0 (socket says 172.21.15.1:58866) 2024-03-06T07:22:50.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.002+0000 7fdac103f640 1 -- 172.21.15.1:0/2745569137 learned_addr learned my addr 172.21.15.1:0/2745569137 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:50.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.002+0000 7fdac1840640 1 --2- 172.21.15.1:0/2745569137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdabc152690 0x7fdabc155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.002+0000 7fdac103f640 1 -- 172.21.15.1:0/2745569137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdabc152690 msgr2=0x7fdabc155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:50.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.002+0000 7fdac103f640 1 --2- 172.21.15.1:0/2745569137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdabc152690 0x7fdabc155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.002+0000 7fdac103f640 1 -- 172.21.15.1:0/2745569137 >> v1:172.21.15.1:6789/0 conn(0x7fdabc0c5d10 legacy=0x7fdabc156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.002+0000 7fdac103f640 1 -- 172.21.15.1:0/2745569137 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdabc0c3880 con 0x7fdabc0c6730 2024-03-06T07:22:50.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.002+0000 7fdac083e640 1 -- 172.21.15.1:0/2745569137 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162991167 0 0) 0x7fdaac001490 con 0x7fdabc0c5d10 2024-03-06T07:22:50.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.002+0000 7fdac1840640 1 --2- 172.21.15.1:0/2745569137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdabc152690 0x7fdabc155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:50.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.003+0000 7fdac103f640 1 --2- 172.21.15.1:0/2745569137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdabc0c6730 0x7fdabc0c57d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdab0009870 tx=0x7fdab002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3aff906dd98e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.003+0000 7fdac083e640 1 -- 172.21.15.1:0/2745569137 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab0035020 con 0x7fdabc0c6730 2024-03-06T07:22:50.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.003+0000 7fdac083e640 1 -- 172.21.15.1:0/2745569137 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab0031640 con 0x7fdabc0c6730 2024-03-06T07:22:50.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.003+0000 7fdac083e640 1 -- 172.21.15.1:0/2745569137 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab002d040 con 0x7fdabc0c6730 2024-03-06T07:22:50.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.003+0000 7fdac3aa4640 1 -- 172.21.15.1:0/2745569137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdabc0c6730 msgr2=0x7fdabc0c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.003+0000 7fdac3aa4640 1 --2- 172.21.15.1:0/2745569137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdabc0c6730 0x7fdabc0c57d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdab0009870 tx=0x7fdab002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.003+0000 7fdac3aa4640 1 -- 172.21.15.1:0/2745569137 shutdown_connections 2024-03-06T07:22:50.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.003+0000 7fdac3aa4640 1 --2- 172.21.15.1:0/2745569137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdabc0c6730 0x7fdabc0c57d0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.003+0000 7fdac3aa4640 1 --2- 172.21.15.1:0/2745569137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdabc152690 0x7fdabc155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.003+0000 7fdac3aa4640 1 -- 172.21.15.1:0/2745569137 >> 172.21.15.1:0/2745569137 conn(0x7fdabc06b410 msgr2=0x7fdabc06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.003+0000 7fdac3aa4640 1 -- 172.21.15.1:0/2745569137 shutdown_connections 2024-03-06T07:22:50.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.003+0000 7fdac3aa4640 1 -- 172.21.15.1:0/2745569137 wait complete. 2024-03-06T07:22:50.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.004+0000 7fdac3aa4640 1 Processor -- start 2024-03-06T07:22:50.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.004+0000 7fdac3aa4640 1 -- start start 2024-03-06T07:22:50.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.005+0000 7fdac3aa4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdabc0c5d10 0x7fdabc1eeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.005+0000 7fdac3aa4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdabc0c6730 0x7fdabc158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.005+0000 7fdac3aa4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdabc152690 0x7fdabc14f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.005+0000 7fdac3aa4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdabc152a70 con 0x7fdabc0c6730 2024-03-06T07:22:50.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.005+0000 7fdac3aa4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdabc1567b0 con 0x7fdabc152690 2024-03-06T07:22:50.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.005+0000 7fdac3aa4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdabc152bf0 con 0x7fdabc0c5d10 2024-03-06T07:22:50.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.005+0000 7fdac103f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdabc0c6730 0x7fdabc158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.005+0000 7fdac103f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdabc0c6730 0x7fdabc158a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45732/0 (socket says 172.21.15.1:45732) 2024-03-06T07:22:50.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.005+0000 7fdac103f640 1 -- 172.21.15.1:0/2228109829 learned_addr learned my addr 172.21.15.1:0/2228109829 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:50.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.005+0000 7fdac1840640 1 --2- 172.21.15.1:0/2228109829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdabc0c5d10 0x7fdabc1eeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.005+0000 7fdac2041640 1 --2- 172.21.15.1:0/2228109829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdabc152690 0x7fdabc14f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.005+0000 7fdac103f640 1 -- 172.21.15.1:0/2228109829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdabc0c5d10 msgr2=0x7fdabc1eeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.006+0000 7fdac103f640 1 --2- 172.21.15.1:0/2228109829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdabc0c5d10 0x7fdabc1eeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.006+0000 7fdac103f640 1 -- 172.21.15.1:0/2228109829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdabc152690 msgr2=0x7fdabc14f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.006+0000 7fdac103f640 1 --2- 172.21.15.1:0/2228109829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdabc152690 0x7fdabc14f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.006+0000 7fdac103f640 1 -- 172.21.15.1:0/2228109829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdabc0c7b50 con 0x7fdabc0c6730 2024-03-06T07:22:50.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.006+0000 7fdac2041640 1 --2- 172.21.15.1:0/2228109829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdabc152690 0x7fdabc14f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:50.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.006+0000 7fdac103f640 1 --2- 172.21.15.1:0/2228109829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdabc0c6730 0x7fdabc158a10 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdab002b930 tx=0x7fdab00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.006+0000 7fdaba7fc640 1 -- 172.21.15.1:0/2228109829 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab0035040 con 0x7fdabc0c6730 2024-03-06T07:22:50.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.006+0000 7fdaba7fc640 1 -- 172.21.15.1:0/2228109829 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab0039bf0 con 0x7fdabc0c6730 2024-03-06T07:22:50.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.006+0000 7fdac3aa4640 1 -- 172.21.15.1:0/2228109829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdabc0c7e30 con 0x7fdabc0c6730 2024-03-06T07:22:50.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.006+0000 7fdaba7fc640 1 -- 172.21.15.1:0/2228109829 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab0045480 con 0x7fdabc0c6730 2024-03-06T07:22:50.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.006+0000 7fdac3aa4640 1 -- 172.21.15.1:0/2228109829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdabc0c7850 con 0x7fdabc0c6730 2024-03-06T07:22:50.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.007+0000 7fdac3aa4640 1 -- 172.21.15.1:0/2228109829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdabc067fd0 con 0x7fdabc0c6730 2024-03-06T07:22:50.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.007+0000 7fdaba7fc640 1 -- 172.21.15.1:0/2228109829 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdab0038020 con 0x7fdabc0c6730 2024-03-06T07:22:50.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.008+0000 7fdaba7fc640 1 --2- 172.21.15.1:0/2228109829 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdaa806b500 0x7fdaa806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.008+0000 7fdaba7fc640 1 -- 172.21.15.1:0/2228109829 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fdab00ac020 con 0x7fdabc0c6730 2024-03-06T07:22:50.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.008+0000 7fdac1840640 1 --2- 172.21.15.1:0/2228109829 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdaa806b500 0x7fdaa806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.008+0000 7fdac1840640 1 --2- 172.21.15.1:0/2228109829 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdaa806b500 0x7fdaa806d9c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdab400aac0 tx=0x7fdab4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.012+0000 7fdaba7fc640 1 -- 172.21.15.1:0/2228109829 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdab0031610 con 0x7fdabc0c6730 2024-03-06T07:22:50.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.194+0000 7fdac3aa4640 1 -- 172.21.15.1:0/2228109829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdabc0c2c40 con 0x7fdabc0c6730 2024-03-06T07:22:50.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.195+0000 7fdaba7fc640 1 -- 172.21.15.1:0/2228109829 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdab007af50 con 0x7fdabc0c6730 2024-03-06T07:22:50.197 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:22:50.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.197+0000 7fdac3aa4640 1 -- 172.21.15.1:0/2228109829 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdaa806b500 msgr2=0x7fdaa806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.197+0000 7fdac3aa4640 1 --2- 172.21.15.1:0/2228109829 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdaa806b500 0x7fdaa806d9c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdab400aac0 tx=0x7fdab4008040 comp rx=0 tx=0).stop 2024-03-06T07:22:50.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.197+0000 7fdac3aa4640 1 -- 172.21.15.1:0/2228109829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdabc0c6730 msgr2=0x7fdabc158a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.197+0000 7fdac3aa4640 1 --2- 172.21.15.1:0/2228109829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdabc0c6730 0x7fdabc158a10 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdab002b930 tx=0x7fdab00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.197+0000 7fdac3aa4640 1 -- 172.21.15.1:0/2228109829 shutdown_connections 2024-03-06T07:22:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.197+0000 7fdac3aa4640 1 --2- 172.21.15.1:0/2228109829 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdaa806b500 0x7fdaa806d9c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.197+0000 7fdac3aa4640 1 --2- 172.21.15.1:0/2228109829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdabc152690 0x7fdabc14f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.197+0000 7fdac3aa4640 1 --2- 172.21.15.1:0/2228109829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdabc0c6730 0x7fdabc158a10 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.197+0000 7fdac3aa4640 1 --2- 172.21.15.1:0/2228109829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdabc0c5d10 0x7fdabc1eeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.197+0000 7fdac3aa4640 1 -- 172.21.15.1:0/2228109829 >> 172.21.15.1:0/2228109829 conn(0x7fdabc06b410 msgr2=0x7fdabc157390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.197+0000 7fdac3aa4640 1 -- 172.21.15.1:0/2228109829 shutdown_connections 2024-03-06T07:22:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.197+0000 7fdac3aa4640 1 -- 172.21.15.1:0/2228109829 wait complete. 2024-03-06T07:22:50.210 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused' 2024-03-06T07:22:50.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.309+0000 7f713b4ed640 1 Processor -- start 2024-03-06T07:22:50.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.309+0000 7f713b4ed640 1 -- start start 2024-03-06T07:22:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.310+0000 7f713b4ed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7134152650 0x7f7134155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.310+0000 7f713b4ed640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f71340c6b10 0x7f7134158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.310+0000 7f713b4ed640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f71340bf9d0 0x7f71340c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.310+0000 7f713b4ed640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7134155800 con 0x7f71340bf9d0 2024-03-06T07:22:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.310+0000 7f713b4ed640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71340c2fc0 con 0x7f71340c6b10 2024-03-06T07:22:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.310+0000 7f713b4ed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71340c6690 con 0x7f7134152650 2024-03-06T07:22:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.310+0000 7f7139a8a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f71340bf9d0 0x7f71340c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.310+0000 7f7139a8a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f71340bf9d0 0x7f71340c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45746/0 (socket says 172.21.15.1:45746) 2024-03-06T07:22:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.310+0000 7f7139a8a640 1 -- 172.21.15.1:0/3367125754 learned_addr learned my addr 172.21.15.1:0/3367125754 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.311+0000 7f7139289640 1 --2- 172.21.15.1:0/3367125754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7134152650 0x7f7134155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.311+0000 7f7138a88640 1 --2- 172.21.15.1:0/3367125754 >> v2:172.21.15.98:3300/0 conn(0x7f71340c6b10 0x7f7134158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.311+0000 7f7139a8a640 1 -- 172.21.15.1:0/3367125754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7134152650 msgr2=0x7f7134155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.311+0000 7f7139a8a640 1 --2- 172.21.15.1:0/3367125754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7134152650 0x7f7134155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.311+0000 7f7139a8a640 1 -- 172.21.15.1:0/3367125754 >> v2:172.21.15.98:3300/0 conn(0x7f71340c6b10 msgr2=0x7f7134158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.311+0000 7f7139a8a640 1 --2- 172.21.15.1:0/3367125754 >> v2:172.21.15.98:3300/0 conn(0x7f71340c6b10 0x7f7134158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.311+0000 7f7139a8a640 1 -- 172.21.15.1:0/3367125754 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71340c7280 con 0x7f71340bf9d0 2024-03-06T07:22:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.311+0000 7f7138a88640 1 --2- 172.21.15.1:0/3367125754 >> v2:172.21.15.98:3300/0 conn(0x7f71340c6b10 0x7f7134158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.311+0000 7f7139a8a640 1 --2- 172.21.15.1:0/3367125754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71340bf9d0 0x7f71340c1460 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f713408c500 tx=0x7f712c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67da253aa64c12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.312+0000 7f712bfff640 1 -- 172.21.15.1:0/3367125754 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712c02c9e0 con 0x7f71340bf9d0 2024-03-06T07:22:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.312+0000 7f712bfff640 1 -- 172.21.15.1:0/3367125754 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712c02cb80 con 0x7f71340bf9d0 2024-03-06T07:22:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.312+0000 7f712bfff640 1 -- 172.21.15.1:0/3367125754 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712c0343f0 con 0x7f71340bf9d0 2024-03-06T07:22:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.312+0000 7f713b4ed640 1 -- 172.21.15.1:0/3367125754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71340bf9d0 msgr2=0x7f71340c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.312+0000 7f713b4ed640 1 --2- 172.21.15.1:0/3367125754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71340bf9d0 0x7f71340c1460 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f713408c500 tx=0x7f712c02be40 comp rx=0 tx=0).stop 2024-03-06T07:22:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.312+0000 7f713b4ed640 1 -- 172.21.15.1:0/3367125754 shutdown_connections 2024-03-06T07:22:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.312+0000 7f713b4ed640 1 --2- 172.21.15.1:0/3367125754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71340bf9d0 0x7f71340c1460 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.312+0000 7f713b4ed640 1 --2- 172.21.15.1:0/3367125754 >> v2:172.21.15.98:3300/0 conn(0x7f71340c6b10 0x7f7134158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.312+0000 7f713b4ed640 1 --2- 172.21.15.1:0/3367125754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7134152650 0x7f7134155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.312+0000 7f713b4ed640 1 -- 172.21.15.1:0/3367125754 >> 172.21.15.1:0/3367125754 conn(0x7f7134066ee0 msgr2=0x7f71340672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.312+0000 7f713b4ed640 1 -- 172.21.15.1:0/3367125754 shutdown_connections 2024-03-06T07:22:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.312+0000 7f713b4ed640 1 -- 172.21.15.1:0/3367125754 wait complete. 2024-03-06T07:22:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.313+0000 7f713b4ed640 1 Processor -- start 2024-03-06T07:22:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.313+0000 7f713b4ed640 1 -- start start 2024-03-06T07:22:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f713b4ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71340bf9d0 0x7f71340ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f713b4ed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7134152650 0x7f71340c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f713b4ed640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f713414e5f0 0x7f71341e0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f713b4ed640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7134155490 con 0x7f713414e5f0 2024-03-06T07:22:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f713b4ed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71341551b0 con 0x7f71340bf9d0 2024-03-06T07:22:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f713b4ed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7134152a30 con 0x7f7134152650 2024-03-06T07:22:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f7139a8a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f713414e5f0 0x7f71341e0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f7139a8a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f713414e5f0 0x7f71341e0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45754/0 (socket says 172.21.15.1:45754) 2024-03-06T07:22:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f7139a8a640 1 -- 172.21.15.1:0/2528708414 learned_addr learned my addr 172.21.15.1:0/2528708414 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f7139289640 1 --2- 172.21.15.1:0/2528708414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71340bf9d0 0x7f71340ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f7138a88640 1 --2- 172.21.15.1:0/2528708414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7134152650 0x7f71340c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f7139a8a640 1 -- 172.21.15.1:0/2528708414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7134152650 msgr2=0x7f71340c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f7139a8a640 1 --2- 172.21.15.1:0/2528708414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7134152650 0x7f71340c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f7139a8a640 1 -- 172.21.15.1:0/2528708414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71340bf9d0 msgr2=0x7f71340ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f7139a8a640 1 --2- 172.21.15.1:0/2528708414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71340bf9d0 0x7f71340ca400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f7139a8a640 1 -- 172.21.15.1:0/2528708414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713414c400 con 0x7f713414e5f0 2024-03-06T07:22:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f7139289640 1 --2- 172.21.15.1:0/2528708414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71340bf9d0 0x7f71340ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.314+0000 7f7138a88640 1 --2- 172.21.15.1:0/2528708414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7134152650 0x7f71340c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.315+0000 7f7139a8a640 1 --2- 172.21.15.1:0/2528708414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f713414e5f0 0x7f71341e0e50 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f712c02c450 tx=0x7f712c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.315+0000 7f7129ffb640 1 -- 172.21.15.1:0/2528708414 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712c043020 con 0x7f713414e5f0 2024-03-06T07:22:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.315+0000 7f7129ffb640 1 -- 172.21.15.1:0/2528708414 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712c03e450 con 0x7f713414e5f0 2024-03-06T07:22:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.315+0000 7f713b4ed640 1 -- 172.21.15.1:0/2528708414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71341e02c0 con 0x7f713414e5f0 2024-03-06T07:22:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.315+0000 7f7129ffb640 1 -- 172.21.15.1:0/2528708414 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712c009510 con 0x7f713414e5f0 2024-03-06T07:22:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.315+0000 7f713b4ed640 1 -- 172.21.15.1:0/2528708414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f713414c6e0 con 0x7f713414e5f0 2024-03-06T07:22:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.316+0000 7f713b4ed640 1 -- 172.21.15.1:0/2528708414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7134067fd0 con 0x7f713414e5f0 2024-03-06T07:22:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.316+0000 7f7129ffb640 1 -- 172.21.15.1:0/2528708414 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f712c046020 con 0x7f713414e5f0 2024-03-06T07:22:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.317+0000 7f7129ffb640 1 --2- 172.21.15.1:0/2528708414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f711806b4d0 0x7f711806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.317+0000 7f7129ffb640 1 -- 172.21.15.1:0/2528708414 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f712c0ac8b0 con 0x7f713414e5f0 2024-03-06T07:22:50.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.317+0000 7f7139289640 1 --2- 172.21.15.1:0/2528708414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f711806b4d0 0x7f711806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.318+0000 7f7139289640 1 --2- 172.21.15.1:0/2528708414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f711806b4d0 0x7f711806d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f71340c0cc0 tx=0x7f7130008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.321+0000 7f7129ffb640 1 -- 172.21.15.1:0/2528708414 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f712c04d030 con 0x7f713414e5f0 2024-03-06T07:22:50.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:50.571+0000 7f713b4ed640 1 -- 172.21.15.1:0/2528708414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]} v 0) v1 -- 0x7f71340b06f0 con 0x7f713414e5f0 2024-03-06T07:22:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.575+0000 7f7129ffb640 1 -- 172.21.15.1:0/2528708414 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]}]=0 v159) v1 ==== 161+0+0 (secure 0 0 0) 0x7f712c07b7e0 con 0x7f713414e5f0 2024-03-06T07:22:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.578+0000 7f713b4ed640 1 -- 172.21.15.1:0/2528708414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f711806b4d0 msgr2=0x7f711806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.578+0000 7f713b4ed640 1 --2- 172.21.15.1:0/2528708414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f711806b4d0 0x7f711806d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f71340c0cc0 tx=0x7f7130008040 comp rx=0 tx=0).stop 2024-03-06T07:22:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.578+0000 7f713b4ed640 1 -- 172.21.15.1:0/2528708414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f713414e5f0 msgr2=0x7f71341e0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.578+0000 7f713b4ed640 1 --2- 172.21.15.1:0/2528708414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f713414e5f0 0x7f71341e0e50 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f712c02c450 tx=0x7f712c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.578+0000 7f713b4ed640 1 -- 172.21.15.1:0/2528708414 shutdown_connections 2024-03-06T07:22:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.578+0000 7f713b4ed640 1 --2- 172.21.15.1:0/2528708414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f711806b4d0 0x7f711806d990 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.578+0000 7f713b4ed640 1 --2- 172.21.15.1:0/2528708414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f713414e5f0 0x7f71341e0e50 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.578+0000 7f713b4ed640 1 --2- 172.21.15.1:0/2528708414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7134152650 0x7f71340c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.578+0000 7f713b4ed640 1 --2- 172.21.15.1:0/2528708414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71340bf9d0 0x7f71340ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.578+0000 7f713b4ed640 1 -- 172.21.15.1:0/2528708414 >> 172.21.15.1:0/2528708414 conn(0x7f7134066ee0 msgr2=0x7f71340672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.578+0000 7f713b4ed640 1 -- 172.21.15.1:0/2528708414 shutdown_connections 2024-03-06T07:22:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.578+0000 7f713b4ed640 1 -- 172.21.15.1:0/2528708414 wait complete. 2024-03-06T07:22:51.593 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:22:51.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.701+0000 7f1e2d781640 1 Processor -- start 2024-03-06T07:22:51.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.701+0000 7f1e2d781640 1 -- start start 2024-03-06T07:22:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.702+0000 7f1e2d781640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e280c5850 0x7f1e280c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.702+0000 7f1e2d781640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1e28155220 0x7f1e281556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.702+0000 7f1e2d781640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e28158910 con 0x7f1e28152920 2024-03-06T07:22:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.702+0000 7f1e2d781640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e28152d00 con 0x7f1e28155220 2024-03-06T07:22:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.702+0000 7f1e2d781640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e281596d0 con 0x7f1e280c5850 2024-03-06T07:22:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.702+0000 7f1e26ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1e28152920 0x7f1e2806aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58876/0 (socket says 172.21.15.1:58876) 2024-03-06T07:22:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.702+0000 7f1e26ffd640 1 -- 172.21.15.1:0/2445281901 learned_addr learned my addr 172.21.15.1:0/2445281901 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.702+0000 7f1e25ffb640 1 -- 172.21.15.1:0/2445281901 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203640634 0 0) 0x7f1e28158910 con 0x7f1e28152920 2024-03-06T07:22:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.702+0000 7f1e25ffb640 1 -- 172.21.15.1:0/2445281901 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e14003520 con 0x7f1e28152920 2024-03-06T07:22:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.702+0000 7f1e267fc640 1 --2- 172.21.15.1:0/2445281901 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e280c5850 0x7f1e280c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.702+0000 7f1e277fe640 1 --2- 172.21.15.1:0/2445281901 >> v2:172.21.15.98:3300/0 conn(0x7f1e28155220 0x7f1e281556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e25ffb640 1 -- 172.21.15.1:0/2445281901 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2840880011 0 0) 0x7f1e14003520 con 0x7f1e28152920 2024-03-06T07:22:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e25ffb640 1 -- 172.21.15.1:0/2445281901 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e280c5850 msgr2=0x7f1e280c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e25ffb640 1 --2- 172.21.15.1:0/2445281901 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e280c5850 0x7f1e280c5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e25ffb640 1 -- 172.21.15.1:0/2445281901 >> v2:172.21.15.98:3300/0 conn(0x7f1e28155220 msgr2=0x7f1e281556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e25ffb640 1 --2- 172.21.15.1:0/2445281901 >> v2:172.21.15.98:3300/0 conn(0x7f1e28155220 0x7f1e281556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e25ffb640 1 -- 172.21.15.1:0/2445281901 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e28158bf0 con 0x7f1e28152920 2024-03-06T07:22:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e25ffb640 1 -- 172.21.15.1:0/2445281901 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f1e20001670 con 0x7f1e28152920 2024-03-06T07:22:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e25ffb640 1 -- 172.21.15.1:0/2445281901 >> v1:172.21.15.1:6789/0 conn(0x7f1e28152920 legacy=0x7f1e2806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e25ffb640 1 --2- 172.21.15.1:0/2445281901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e140039b0 0x7f1e14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e25ffb640 1 --2- 172.21.15.1:0/2445281901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e14004800 0x7f1e14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e25ffb640 1 --2- 172.21.15.1:0/2445281901 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e280c5850 0x7f1e140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e25ffb640 1 -- 172.21.15.1:0/2445281901 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e28152d00 con 0x7f1e14004800 2024-03-06T07:22:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e25ffb640 1 -- 172.21.15.1:0/2445281901 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e281596d0 con 0x7f1e140039b0 2024-03-06T07:22:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e25ffb640 1 -- 172.21.15.1:0/2445281901 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1400b9c0 con 0x7f1e280c5850 2024-03-06T07:22:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e267fc640 1 --2- 172.21.15.1:0/2445281901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e14004800 0x7f1e14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e26ffd640 1 --2- 172.21.15.1:0/2445281901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e140039b0 0x7f1e14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e277fe640 1 --2- 172.21.15.1:0/2445281901 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e280c5850 0x7f1e140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e267fc640 1 -- 172.21.15.1:0/2445281901 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e280c5850 msgr2=0x7f1e140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e267fc640 1 --2- 172.21.15.1:0/2445281901 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e280c5850 0x7f1e140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e267fc640 1 -- 172.21.15.1:0/2445281901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e140039b0 msgr2=0x7f1e14003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e267fc640 1 --2- 172.21.15.1:0/2445281901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e140039b0 0x7f1e14003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e267fc640 1 -- 172.21.15.1:0/2445281901 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e14003520 con 0x7f1e14004800 2024-03-06T07:22:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e277fe640 1 --2- 172.21.15.1:0/2445281901 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e280c5850 0x7f1e140051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.703+0000 7f1e26ffd640 1 --2- 172.21.15.1:0/2445281901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e140039b0 0x7f1e14003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.704+0000 7f1e267fc640 1 --2- 172.21.15.1:0/2445281901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e14004800 0x7f1e14004c80 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009af0 tx=0x7f1e14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e88968768830966a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.704+0000 7f1e25ffb640 1 -- 172.21.15.1:0/2445281901 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e10039070 con 0x7f1e14004800 2024-03-06T07:22:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.704+0000 7f1e25ffb640 1 -- 172.21.15.1:0/2445281901 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1002c6b0 con 0x7f1e14004800 2024-03-06T07:22:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.704+0000 7f1e25ffb640 1 -- 172.21.15.1:0/2445281901 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e10009690 con 0x7f1e14004800 2024-03-06T07:22:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.705+0000 7f1e2d781640 1 -- 172.21.15.1:0/2445281901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e14004800 msgr2=0x7f1e14004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.705+0000 7f1e2d781640 1 --2- 172.21.15.1:0/2445281901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e14004800 0x7f1e14004c80 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009af0 tx=0x7f1e14005830 comp rx=0 tx=0).stop 2024-03-06T07:22:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.705+0000 7f1e277fe640 1 -- 172.21.15.1:0/2445281901 reap_dead start 2024-03-06T07:22:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.705+0000 7f1e2d781640 1 -- 172.21.15.1:0/2445281901 shutdown_connections 2024-03-06T07:22:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.705+0000 7f1e2d781640 1 -- 172.21.15.1:0/2445281901 >> 172.21.15.1:0/2445281901 conn(0x7f1e280a9700 msgr2=0x7f1e280a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.705+0000 7f1e2d781640 1 -- 172.21.15.1:0/2445281901 shutdown_connections 2024-03-06T07:22:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.705+0000 7f1e2d781640 1 -- 172.21.15.1:0/2445281901 wait complete. 2024-03-06T07:22:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.706+0000 7f1e2d781640 1 Processor -- start 2024-03-06T07:22:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.706+0000 7f1e2d781640 1 -- start start 2024-03-06T07:22:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e2d781640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e14004800 0x7f1e28157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e2d781640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e281ee170 0x7f1e281eb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e2d781640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e280c4dc0 0x7f1e281ee750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e2d781640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e280c6380 con 0x7f1e14004800 2024-03-06T07:22:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e2d781640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e28159240 con 0x7f1e281ee170 2024-03-06T07:22:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e2d781640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2806b700 con 0x7f1e280c4dc0 2024-03-06T07:22:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e26ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e14004800 0x7f1e28157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e26ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e14004800 0x7f1e28157ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45772/0 (socket says 172.21.15.1:45772) 2024-03-06T07:22:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e26ffd640 1 -- 172.21.15.1:0/2472994278 learned_addr learned my addr 172.21.15.1:0/2472994278 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e267fc640 1 --2- 172.21.15.1:0/2472994278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e281ee170 0x7f1e281eb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e277fe640 1 --2- 172.21.15.1:0/2472994278 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e280c4dc0 0x7f1e281ee750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e26ffd640 1 -- 172.21.15.1:0/2472994278 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e280c4dc0 msgr2=0x7f1e281ee750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e26ffd640 1 --2- 172.21.15.1:0/2472994278 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e280c4dc0 0x7f1e281ee750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e26ffd640 1 -- 172.21.15.1:0/2472994278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e281ee170 msgr2=0x7f1e281eb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e26ffd640 1 --2- 172.21.15.1:0/2472994278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e281ee170 0x7f1e281eb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e26ffd640 1 -- 172.21.15.1:0/2472994278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e280acc90 con 0x7f1e14004800 2024-03-06T07:22:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e277fe640 1 --2- 172.21.15.1:0/2472994278 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e280c4dc0 0x7f1e281ee750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e26ffd640 1 --2- 172.21.15.1:0/2472994278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e14004800 0x7f1e28157ab0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000e7e0 tx=0x7f1e2000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e1f7fe640 1 -- 172.21.15.1:0/2472994278 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2000c870 con 0x7f1e14004800 2024-03-06T07:22:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e1f7fe640 1 -- 172.21.15.1:0/2472994278 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e20018070 con 0x7f1e14004800 2024-03-06T07:22:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e2d781640 1 -- 172.21.15.1:0/2472994278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e280aad30 con 0x7f1e14004800 2024-03-06T07:22:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e1f7fe640 1 -- 172.21.15.1:0/2472994278 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2000cc30 con 0x7f1e14004800 2024-03-06T07:22:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.707+0000 7f1e2d781640 1 -- 172.21.15.1:0/2472994278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e280aa770 con 0x7f1e14004800 2024-03-06T07:22:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.708+0000 7f1e2d781640 1 -- 172.21.15.1:0/2472994278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e28067fd0 con 0x7f1e14004800 2024-03-06T07:22:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.709+0000 7f1e1f7fe640 1 -- 172.21.15.1:0/2472994278 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e20013420 con 0x7f1e14004800 2024-03-06T07:22:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.709+0000 7f1e1f7fe640 1 --2- 172.21.15.1:0/2472994278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e0806b500 0x7f1e0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.709+0000 7f1e1f7fe640 1 -- 172.21.15.1:0/2472994278 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f1e20089cb0 con 0x7f1e14004800 2024-03-06T07:22:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.710+0000 7f1e267fc640 1 --2- 172.21.15.1:0/2472994278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e0806b500 0x7f1e0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.710+0000 7f1e267fc640 1 --2- 172.21.15.1:0/2472994278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e0806b500 0x7f1e0806d9c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1e10036000 tx=0x7f1e10030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.714+0000 7f1e1f7fe640 1 -- 172.21.15.1:0/2472994278 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e20058be0 con 0x7f1e14004800 2024-03-06T07:22:51.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.893+0000 7f1e2d781640 1 -- 172.21.15.1:0/2472994278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1e280705e0 con 0x7f1e14004800 2024-03-06T07:22:51.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.896+0000 7f1e1f7fe640 1 -- 172.21.15.1:0/2472994278 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v23)=0 v23) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1e200584e0 con 0x7f1e14004800 2024-03-06T07:22:51.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.898+0000 7f1e2d781640 1 -- 172.21.15.1:0/2472994278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e0806b500 msgr2=0x7f1e0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.898+0000 7f1e2d781640 1 --2- 172.21.15.1:0/2472994278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e0806b500 0x7f1e0806d9c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1e10036000 tx=0x7f1e10030000 comp rx=0 tx=0).stop 2024-03-06T07:22:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.898+0000 7f1e2d781640 1 -- 172.21.15.1:0/2472994278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e14004800 msgr2=0x7f1e28157ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.898+0000 7f1e2d781640 1 --2- 172.21.15.1:0/2472994278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e14004800 0x7f1e28157ab0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000e7e0 tx=0x7f1e2000eca0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.898+0000 7f1e2d781640 1 -- 172.21.15.1:0/2472994278 shutdown_connections 2024-03-06T07:22:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.898+0000 7f1e2d781640 1 --2- 172.21.15.1:0/2472994278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e0806b500 0x7f1e0806d9c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.898+0000 7f1e2d781640 1 --2- 172.21.15.1:0/2472994278 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e280c4dc0 0x7f1e281ee750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.898+0000 7f1e2d781640 1 --2- 172.21.15.1:0/2472994278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e281ee170 0x7f1e281eb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.898+0000 7f1e2d781640 1 --2- 172.21.15.1:0/2472994278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e14004800 0x7f1e28157ab0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.898+0000 7f1e2d781640 1 -- 172.21.15.1:0/2472994278 >> 172.21.15.1:0/2472994278 conn(0x7f1e280a9700 msgr2=0x7f1e28154760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.898+0000 7f1e2d781640 1 -- 172.21.15.1:0/2472994278 shutdown_connections 2024-03-06T07:22:51.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:51.898+0000 7f1e2d781640 1 -- 172.21.15.1:0/2472994278 wait complete. 2024-03-06T07:22:51.911 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:22:51.911 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:51.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:52.000 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:22:52.000 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:22:52.001 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:52.015 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:22:52.016 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:22:52.016 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:52.034 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:22:52.034 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:22:52.034 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.154+0000 7f7bf3684640 1 Processor -- start 2024-03-06T07:22:52.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.155+0000 7f7bf3684640 1 -- start start 2024-03-06T07:22:52.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.155+0000 7f7bf3684640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7bec0c2b30 0x7f7bec0724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.155+0000 7f7bf3684640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7bec0a5fa0 0x7f7bec072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.155+0000 7f7bf3684640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bec151dd0 con 0x7f7bec152b70 2024-03-06T07:22:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.155+0000 7f7bf3684640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bec1526e0 con 0x7f7bec0c2b30 2024-03-06T07:22:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.155+0000 7f7bf3684640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec156190 con 0x7f7bec0a5fa0 2024-03-06T07:22:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.155+0000 7f7bf1420640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7bec0c2b30 0x7f7bec0724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.156+0000 7f7bf1420640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7bec0c2b30 0x7f7bec0724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45778/0 (socket says 172.21.15.1:45778) 2024-03-06T07:22:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.156+0000 7f7bf1420640 1 -- 172.21.15.1:0/494234950 learned_addr learned my addr 172.21.15.1:0/494234950 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.156+0000 7f7bf1c21640 1 --2- 172.21.15.1:0/494234950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7bec0a5fa0 0x7f7bec072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.156+0000 7f7bf1420640 1 -- 172.21.15.1:0/494234950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7bec0a5fa0 msgr2=0x7f7bec072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.156+0000 7f7bf1420640 1 --2- 172.21.15.1:0/494234950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7bec0a5fa0 0x7f7bec072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.156+0000 7f7bf1420640 1 -- 172.21.15.1:0/494234950 >> v1:172.21.15.98:6789/0 conn(0x7f7bec152b70 legacy=0x7f7bec0c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.156+0000 7f7bf1420640 1 -- 172.21.15.1:0/494234950 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bec152090 con 0x7f7bec0c2b30 2024-03-06T07:22:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.156+0000 7f7bf1c21640 1 --2- 172.21.15.1:0/494234950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7bec0a5fa0 0x7f7bec072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.157+0000 7f7bf1420640 1 --2- 172.21.15.1:0/494234950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bec0c2b30 0x7f7bec0724c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7be8009870 tx=0x7f7be802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1575057d95a32670 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.157+0000 7f7be3fff640 1 -- 172.21.15.1:0/494234950 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be8035020 con 0x7f7bec0c2b30 2024-03-06T07:22:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.157+0000 7f7be3fff640 1 -- 172.21.15.1:0/494234950 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be8031640 con 0x7f7bec0c2b30 2024-03-06T07:22:52.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.157+0000 7f7be3fff640 1 -- 172.21.15.1:0/494234950 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be802d040 con 0x7f7bec0c2b30 2024-03-06T07:22:52.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.157+0000 7f7bf3684640 1 -- 172.21.15.1:0/494234950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bec0c2b30 msgr2=0x7f7bec0724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.157+0000 7f7bf3684640 1 --2- 172.21.15.1:0/494234950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bec0c2b30 0x7f7bec0724c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7be8009870 tx=0x7f7be802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.157+0000 7f7bf3684640 1 -- 172.21.15.1:0/494234950 shutdown_connections 2024-03-06T07:22:52.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.157+0000 7f7bf3684640 1 --2- 172.21.15.1:0/494234950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7bec0a5fa0 0x7f7bec072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.157+0000 7f7bf3684640 1 --2- 172.21.15.1:0/494234950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bec0c2b30 0x7f7bec0724c0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.157+0000 7f7bf3684640 1 -- 172.21.15.1:0/494234950 >> 172.21.15.1:0/494234950 conn(0x7f7bec0c6860 msgr2=0x7f7bec0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.157+0000 7f7bf3684640 1 -- 172.21.15.1:0/494234950 shutdown_connections 2024-03-06T07:22:52.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.157+0000 7f7bf3684640 1 -- 172.21.15.1:0/494234950 wait complete. 2024-03-06T07:22:52.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.158+0000 7f7bf3684640 1 Processor -- start 2024-03-06T07:22:52.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.159+0000 7f7bf3684640 1 -- start start 2024-03-06T07:22:52.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.159+0000 7f7bf3684640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bec0a5fa0 0x7f7bec0bde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.159+0000 7f7bf3684640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7bec0c2b30 0x7f7bec1ded60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.159+0000 7f7bf3684640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bec152b70 0x7f7bec0ab3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.159+0000 7f7bf3684640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec155f00 con 0x7f7bec152b70 2024-03-06T07:22:52.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.159+0000 7f7bf3684640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec155d80 con 0x7f7bec0a5fa0 2024-03-06T07:22:52.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.159+0000 7f7bf3684640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec0c3a00 con 0x7f7bec0c2b30 2024-03-06T07:22:52.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.159+0000 7f7bf1c21640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bec152b70 0x7f7bec0ab3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.159+0000 7f7bf1c21640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bec152b70 0x7f7bec0ab3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45780/0 (socket says 172.21.15.1:45780) 2024-03-06T07:22:52.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.159+0000 7f7bf1c21640 1 -- 172.21.15.1:0/217248488 learned_addr learned my addr 172.21.15.1:0/217248488 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:52.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.159+0000 7f7bf1420640 1 --2- 172.21.15.1:0/217248488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bec0a5fa0 0x7f7bec0bde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.159+0000 7f7bf0c1f640 1 --2- 172.21.15.1:0/217248488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7bec0c2b30 0x7f7bec1ded60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.160+0000 7f7bf1c21640 1 -- 172.21.15.1:0/217248488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7bec0c2b30 msgr2=0x7f7bec1ded60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.160+0000 7f7bf1c21640 1 --2- 172.21.15.1:0/217248488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7bec0c2b30 0x7f7bec1ded60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.160+0000 7f7bf1c21640 1 -- 172.21.15.1:0/217248488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bec0a5fa0 msgr2=0x7f7bec0bde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.160+0000 7f7bf1c21640 1 --2- 172.21.15.1:0/217248488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bec0a5fa0 0x7f7bec0bde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.160+0000 7f7bf1c21640 1 -- 172.21.15.1:0/217248488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bec0be380 con 0x7f7bec152b70 2024-03-06T07:22:52.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.160+0000 7f7bf1c21640 1 --2- 172.21.15.1:0/217248488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bec152b70 0x7f7bec0ab3e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7be400cf20 tx=0x7f7be400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.160+0000 7f7be1ffb640 1 -- 172.21.15.1:0/217248488 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be401a070 con 0x7f7bec152b70 2024-03-06T07:22:52.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.160+0000 7f7be1ffb640 1 -- 172.21.15.1:0/217248488 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be400d840 con 0x7f7bec152b70 2024-03-06T07:22:52.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.160+0000 7f7be1ffb640 1 -- 172.21.15.1:0/217248488 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be400dc30 con 0x7f7bec152b70 2024-03-06T07:22:52.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.160+0000 7f7bf3684640 1 -- 172.21.15.1:0/217248488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bec0bb880 con 0x7f7bec152b70 2024-03-06T07:22:52.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.160+0000 7f7bf3684640 1 -- 172.21.15.1:0/217248488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bec0aabe0 con 0x7f7bec152b70 2024-03-06T07:22:52.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.161+0000 7f7bf3684640 1 -- 172.21.15.1:0/217248488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bec067fd0 con 0x7f7bec152b70 2024-03-06T07:22:52.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.162+0000 7f7be1ffb640 1 -- 172.21.15.1:0/217248488 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7be401d070 con 0x7f7bec152b70 2024-03-06T07:22:52.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.162+0000 7f7be1ffb640 1 --2- 172.21.15.1:0/217248488 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bd006b500 0x7f7bd006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.162+0000 7f7be1ffb640 1 -- 172.21.15.1:0/217248488 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f7be408a930 con 0x7f7bec152b70 2024-03-06T07:22:52.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.162+0000 7f7bf1420640 1 --2- 172.21.15.1:0/217248488 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bd006b500 0x7f7bd006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.163+0000 7f7bf1420640 1 --2- 172.21.15.1:0/217248488 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bd006b500 0x7f7bd006d9c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7be8036000 tx=0x7f7be803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.166+0000 7f7be1ffb640 1 -- 172.21.15.1:0/217248488 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7be4059210 con 0x7f7bec152b70 2024-03-06T07:22:52.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.384+0000 7f7bf3684640 1 -- 172.21.15.1:0/217248488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bec071790 con 0x7f7bec152b70 2024-03-06T07:22:52.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.385+0000 7f7be1ffb640 1 -- 172.21.15.1:0/217248488 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+3693 (secure 0 0 0) 0x7f7be405d2b0 con 0x7f7bec152b70 2024-03-06T07:22:52.387 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:52.387 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":50,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2604497680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2604497680},{"type":"v1","addr":"172.21.15.98:6836","nonce":2604497680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":5776,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3387109635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3387109635},{"type":"v1","addr":"172.21.15.98:6837","nonce":3387109635}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/1450734721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1450734721},{"type":"v1","addr":"172.21.15.1:6834","nonce":1450734721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4269513324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4269513324},{"type":"v1","addr":"172.21.15.1:6835","nonce":4269513324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[]} 2024-03-06T07:22:52.387 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 50 2024-03-06T07:22:52.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.386+0000 7f7bf3684640 1 -- 172.21.15.1:0/217248488 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bd006b500 msgr2=0x7f7bd006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.386+0000 7f7bf3684640 1 --2- 172.21.15.1:0/217248488 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bd006b500 0x7f7bd006d9c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7be8036000 tx=0x7f7be803c040 comp rx=0 tx=0).stop 2024-03-06T07:22:52.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.386+0000 7f7bf3684640 1 -- 172.21.15.1:0/217248488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bec152b70 msgr2=0x7f7bec0ab3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.386+0000 7f7bf3684640 1 --2- 172.21.15.1:0/217248488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bec152b70 0x7f7bec0ab3e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7be400cf20 tx=0x7f7be400a7d0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.387+0000 7f7bf3684640 1 -- 172.21.15.1:0/217248488 shutdown_connections 2024-03-06T07:22:52.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.387+0000 7f7bf3684640 1 --2- 172.21.15.1:0/217248488 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bd006b500 0x7f7bd006d9c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.387+0000 7f7bf3684640 1 --2- 172.21.15.1:0/217248488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bec152b70 0x7f7bec0ab3e0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.387+0000 7f7bf3684640 1 --2- 172.21.15.1:0/217248488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7bec0c2b30 0x7f7bec1ded60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.387+0000 7f7bf3684640 1 --2- 172.21.15.1:0/217248488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bec0a5fa0 0x7f7bec0bde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.387+0000 7f7bf3684640 1 -- 172.21.15.1:0/217248488 >> 172.21.15.1:0/217248488 conn(0x7f7bec0c6860 msgr2=0x7f7bec0c6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.387+0000 7f7bf3684640 1 -- 172.21.15.1:0/217248488 shutdown_connections 2024-03-06T07:22:52.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.387+0000 7f7bf3684640 1 -- 172.21.15.1:0/217248488 wait complete. 2024-03-06T07:22:52.399 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:22:52.400 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:22:52.400 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:22:52.400 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:52.400 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:22:52.400 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:22:52.400 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:22:52.400 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:52.401 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:22:52.401 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:22:52.401 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:22:52.401 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:52.401 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:22:52.401 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:22:52.401 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:22:52.401 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:52.411 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:22:52.411 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:22:52.412 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:22:52.412 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:22:52.413 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:22:52.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:22:52.414 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:22:52.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:22:52.454 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:22:52.454 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:52.455 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:22:52.455 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:52.456 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:22:52.457 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:52.457 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:22:52.457 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:52.458 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:22:52.530 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:22:52.530 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:22:52.531 INFO:tasks.ceph.mds.a.smithi001.stderr:2024-03-06T07:22:52.529+0000 7fc62fb3b640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-03-06T07:22:52.539 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:22:52.539 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:22:52.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.560+0000 7f894958a640 1 Processor -- start 2024-03-06T07:22:52.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.561+0000 7f894958a640 1 -- start start 2024-03-06T07:22:52.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.561+0000 7f894958a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f89440c6b10 0x7f8944155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.561+0000 7f894958a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f89440bf9d0 0x7f89440c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.561+0000 7f894958a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89440c2fc0 con 0x7f8944152650 2024-03-06T07:22:52.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.561+0000 7f894958a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8944155800 con 0x7f89440bf9d0 2024-03-06T07:22:52.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.561+0000 7f894958a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8944158da0 con 0x7f89440c6b10 2024-03-06T07:22:52.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.561+0000 7f89437fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f89440bf9d0 0x7f89440c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.561+0000 7f89437fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f89440bf9d0 0x7f89440c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45820/0 (socket says 172.21.15.1:45820) 2024-03-06T07:22:52.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.561+0000 7f89437fe640 1 -- 172.21.15.1:0/2766251586 learned_addr learned my addr 172.21.15.1:0/2766251586 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:52.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.562+0000 7f89427fc640 1 --2- 172.21.15.1:0/2766251586 >> v2:172.21.15.98:3300/0 conn(0x7f89440c6b10 0x7f8944155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.562+0000 7f89437fe640 1 -- 172.21.15.1:0/2766251586 >> v2:172.21.15.98:3300/0 conn(0x7f89440c6b10 msgr2=0x7f8944155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.562+0000 7f89437fe640 1 --2- 172.21.15.1:0/2766251586 >> v2:172.21.15.98:3300/0 conn(0x7f89440c6b10 0x7f8944155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.562+0000 7f89437fe640 1 -- 172.21.15.1:0/2766251586 >> v1:172.21.15.98:6789/0 conn(0x7f8944152650 legacy=0x7f8944158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.562+0000 7f89437fe640 1 -- 172.21.15.1:0/2766251586 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89440c6690 con 0x7f89440bf9d0 2024-03-06T07:22:52.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.562+0000 7f89437fe640 1 --2- 172.21.15.1:0/2766251586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89440bf9d0 0x7f89440c1460 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f89440cb8e0 tx=0x7f893c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3a428b42e6050d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.563+0000 7f8941ffb640 1 -- 172.21.15.1:0/2766251586 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c02c9f0 con 0x7f89440bf9d0 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.563+0000 7f8941ffb640 1 -- 172.21.15.1:0/2766251586 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893c02cb90 con 0x7f89440bf9d0 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.563+0000 7f8941ffb640 1 -- 172.21.15.1:0/2766251586 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c0343f0 con 0x7f89440bf9d0 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.563+0000 7f894958a640 1 -- 172.21.15.1:0/2766251586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89440bf9d0 msgr2=0x7f89440c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.563+0000 7f894958a640 1 --2- 172.21.15.1:0/2766251586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89440bf9d0 0x7f89440c1460 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f89440cb8e0 tx=0x7f893c02be50 comp rx=0 tx=0).stop 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.563+0000 7f894958a640 1 -- 172.21.15.1:0/2766251586 shutdown_connections 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.563+0000 7f894958a640 1 --2- 172.21.15.1:0/2766251586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89440bf9d0 0x7f89440c1460 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.563+0000 7f894958a640 1 --2- 172.21.15.1:0/2766251586 >> v2:172.21.15.98:3300/0 conn(0x7f89440c6b10 0x7f8944155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.563+0000 7f894958a640 1 -- 172.21.15.1:0/2766251586 >> 172.21.15.1:0/2766251586 conn(0x7f8944066ee0 msgr2=0x7f89440672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.563+0000 7f894958a640 1 -- 172.21.15.1:0/2766251586 shutdown_connections 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.563+0000 7f894958a640 1 -- 172.21.15.1:0/2766251586 wait complete. 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.564+0000 7f894958a640 1 Processor -- start 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.564+0000 7f894958a640 1 -- start start 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.565+0000 7f894958a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89440bf9d0 0x7f894414b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.565+0000 7f894958a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8944152650 0x7f8944158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.565+0000 7f894958a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89440c5a70 0x7f89440c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.565+0000 7f894958a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89441551b0 con 0x7f8944152650 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.565+0000 7f894958a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89440c6fa0 con 0x7f89440bf9d0 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.565+0000 7f894958a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8944155490 con 0x7f89440c5a70 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.565+0000 7f89427fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8944152650 0x7f8944158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.565+0000 7f89427fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8944152650 0x7f8944158070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45834/0 (socket says 172.21.15.1:45834) 2024-03-06T07:22:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.565+0000 7f89427fc640 1 -- 172.21.15.1:0/812650197 learned_addr learned my addr 172.21.15.1:0/812650197 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.565+0000 7f89437fe640 1 --2- 172.21.15.1:0/812650197 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89440c5a70 0x7f89440c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.565+0000 7f8942ffd640 1 --2- 172.21.15.1:0/812650197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89440bf9d0 0x7f894414b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.565+0000 7f89427fc640 1 -- 172.21.15.1:0/812650197 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89440c5a70 msgr2=0x7f89440c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.565+0000 7f89427fc640 1 --2- 172.21.15.1:0/812650197 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89440c5a70 0x7f89440c5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.565+0000 7f89427fc640 1 -- 172.21.15.1:0/812650197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89440bf9d0 msgr2=0x7f894414b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.565+0000 7f89427fc640 1 --2- 172.21.15.1:0/812650197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89440bf9d0 0x7f894414b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.565+0000 7f89427fc640 1 -- 172.21.15.1:0/812650197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89441567d0 con 0x7f8944152650 2024-03-06T07:22:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.565+0000 7f8942ffd640 1 --2- 172.21.15.1:0/812650197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89440bf9d0 0x7f894414b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.566+0000 7f89427fc640 1 --2- 172.21.15.1:0/812650197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8944152650 0x7f8944158070 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f893000d830 tx=0x7f893000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.566+0000 7f893b7fe640 1 -- 172.21.15.1:0/812650197 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893000b660 con 0x7f8944152650 2024-03-06T07:22:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.566+0000 7f893b7fe640 1 -- 172.21.15.1:0/812650197 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8930019070 con 0x7f8944152650 2024-03-06T07:22:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.566+0000 7f894958a640 1 -- 172.21.15.1:0/812650197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89441ee0c0 con 0x7f8944152650 2024-03-06T07:22:52.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.566+0000 7f893b7fe640 1 -- 172.21.15.1:0/812650197 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893000ba20 con 0x7f8944152650 2024-03-06T07:22:52.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.566+0000 7f894958a640 1 -- 172.21.15.1:0/812650197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89441ee680 con 0x7f8944152650 2024-03-06T07:22:52.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.567+0000 7f894958a640 1 -- 172.21.15.1:0/812650197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f894406b570 con 0x7f8944152650 2024-03-06T07:22:52.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.567+0000 7f893b7fe640 1 -- 172.21.15.1:0/812650197 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f893000bbc0 con 0x7f8944152650 2024-03-06T07:22:52.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.568+0000 7f893b7fe640 1 --2- 172.21.15.1:0/812650197 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f892c06b500 0x7f892c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.568+0000 7f893b7fe640 1 -- 172.21.15.1:0/812650197 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f893008a5b0 con 0x7f8944152650 2024-03-06T07:22:52.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.568+0000 7f8942ffd640 1 --2- 172.21.15.1:0/812650197 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f892c06b500 0x7f892c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.568+0000 7f8942ffd640 1 --2- 172.21.15.1:0/812650197 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f892c06b500 0x7f892c06d9c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8934003eb0 tx=0x7f8934009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.572+0000 7f893b7fe640 1 -- 172.21.15.1:0/812650197 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f894406b570 con 0x7f8944152650 2024-03-06T07:22:52.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:52.769+0000 7f894958a640 1 -- 172.21.15.1:0/812650197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f89440b06f0 con 0x7f8944152650 2024-03-06T07:22:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.592+0000 7f893b7fe640 1 -- 172.21.15.1:0/812650197 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v70) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8930059440 con 0x7f8944152650 2024-03-06T07:22:53.594 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-06T07:22:53.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.595+0000 7f894958a640 1 -- 172.21.15.1:0/812650197 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f892c06b500 msgr2=0x7f892c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.595+0000 7f894958a640 1 --2- 172.21.15.1:0/812650197 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f892c06b500 0x7f892c06d9c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8934003eb0 tx=0x7f8934009040 comp rx=0 tx=0).stop 2024-03-06T07:22:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.595+0000 7f894958a640 1 -- 172.21.15.1:0/812650197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8944152650 msgr2=0x7f8944158070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.595+0000 7f894958a640 1 --2- 172.21.15.1:0/812650197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8944152650 0x7f8944158070 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f893000d830 tx=0x7f893000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.595+0000 7f894958a640 1 -- 172.21.15.1:0/812650197 shutdown_connections 2024-03-06T07:22:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.595+0000 7f894958a640 1 --2- 172.21.15.1:0/812650197 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f892c06b500 0x7f892c06d9c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.595+0000 7f894958a640 1 --2- 172.21.15.1:0/812650197 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89440c5a70 0x7f89440c5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.595+0000 7f894958a640 1 --2- 172.21.15.1:0/812650197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8944152650 0x7f8944158070 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.595+0000 7f894958a640 1 --2- 172.21.15.1:0/812650197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89440bf9d0 0x7f894414b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.595+0000 7f894958a640 1 -- 172.21.15.1:0/812650197 >> 172.21.15.1:0/812650197 conn(0x7f8944066ee0 msgr2=0x7f89440c3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.595+0000 7f894958a640 1 -- 172.21.15.1:0/812650197 shutdown_connections 2024-03-06T07:22:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.595+0000 7f894958a640 1 -- 172.21.15.1:0/812650197 wait complete. 2024-03-06T07:22:53.612 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:22:53.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.718+0000 7f6d0d678640 1 Processor -- start 2024-03-06T07:22:53.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.718+0000 7f6d0d678640 1 -- start start 2024-03-06T07:22:53.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.719+0000 7f6d0d678640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6d080c5520 0x7f6d081550d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.719+0000 7f6d0d678640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d080c84c0 0x7f6d08158d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.719+0000 7f6d0d678640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d080c7650 con 0x7f6d08159240 2024-03-06T07:22:53.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.719+0000 7f6d0d678640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d080c78f0 con 0x7f6d080c5520 2024-03-06T07:22:53.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.719+0000 7f6d0d678640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d081587e0 con 0x7f6d080c84c0 2024-03-06T07:22:53.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.719+0000 7f6d06ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6d080c5520 0x7f6d081550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.719+0000 7f6d06ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6d080c5520 0x7f6d081550d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45844/0 (socket says 172.21.15.1:45844) 2024-03-06T07:22:53.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.719+0000 7f6d06ffd640 1 -- 172.21.15.1:0/4060507923 learned_addr learned my addr 172.21.15.1:0/4060507923 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:53.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.719+0000 7f6cfffff640 1 --2- 172.21.15.1:0/4060507923 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d080c84c0 0x7f6d08158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.719+0000 7f6d06ffd640 1 -- 172.21.15.1:0/4060507923 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d080c84c0 msgr2=0x7f6d08158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.719+0000 7f6d06ffd640 1 --2- 172.21.15.1:0/4060507923 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d080c84c0 0x7f6d08158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.719+0000 7f6d06ffd640 1 -- 172.21.15.1:0/4060507923 >> v1:172.21.15.98:6789/0 conn(0x7f6d08159240 legacy=0x7f6d0806da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.719+0000 7f6d06ffd640 1 -- 172.21.15.1:0/4060507923 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d080c3ce0 con 0x7f6d080c5520 2024-03-06T07:22:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.720+0000 7f6d06ffd640 1 --2- 172.21.15.1:0/4060507923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d080c5520 0x7f6d081550d0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009870 tx=0x7f6cf802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7511396da7865d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.720+0000 7f6d067fc640 1 -- 172.21.15.1:0/4060507923 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8035020 con 0x7f6d080c5520 2024-03-06T07:22:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.720+0000 7f6d067fc640 1 -- 172.21.15.1:0/4060507923 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf8031640 con 0x7f6d080c5520 2024-03-06T07:22:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.720+0000 7f6d067fc640 1 -- 172.21.15.1:0/4060507923 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf802d040 con 0x7f6d080c5520 2024-03-06T07:22:53.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.721+0000 7f6d0d678640 1 -- 172.21.15.1:0/4060507923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d080c5520 msgr2=0x7f6d081550d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.721+0000 7f6d0d678640 1 --2- 172.21.15.1:0/4060507923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d080c5520 0x7f6d081550d0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009870 tx=0x7f6cf802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.721+0000 7f6d0d678640 1 -- 172.21.15.1:0/4060507923 shutdown_connections 2024-03-06T07:22:53.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.721+0000 7f6d0d678640 1 --2- 172.21.15.1:0/4060507923 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d080c84c0 0x7f6d08158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.721+0000 7f6d0d678640 1 --2- 172.21.15.1:0/4060507923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d080c5520 0x7f6d081550d0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.721+0000 7f6d0d678640 1 -- 172.21.15.1:0/4060507923 >> 172.21.15.1:0/4060507923 conn(0x7f6d080c88f0 msgr2=0x7f6d080c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.721+0000 7f6d0d678640 1 -- 172.21.15.1:0/4060507923 shutdown_connections 2024-03-06T07:22:53.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.721+0000 7f6d0d678640 1 -- 172.21.15.1:0/4060507923 wait complete. 2024-03-06T07:22:53.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.722+0000 7f6d0d678640 1 Processor -- start 2024-03-06T07:22:53.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.722+0000 7f6d0d678640 1 -- start start 2024-03-06T07:22:53.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6d0d678640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d080c5520 0x7f6d081eb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6d0d678640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d080c84c0 0x7f6d081559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6d0d678640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d08159240 0x7f6d0814d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6d0d678640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0806e2c0 con 0x7f6d08159240 2024-03-06T07:22:53.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6d0d678640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0806e140 con 0x7f6d080c5520 2024-03-06T07:22:53.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6d0d678640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080c5900 con 0x7f6d080c84c0 2024-03-06T07:22:53.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6d077fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d08159240 0x7f6d0814d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6d077fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d08159240 0x7f6d0814d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45854/0 (socket says 172.21.15.1:45854) 2024-03-06T07:22:53.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6d077fe640 1 -- 172.21.15.1:0/739196129 learned_addr learned my addr 172.21.15.1:0/739196129 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:53.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6d06ffd640 1 --2- 172.21.15.1:0/739196129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d080c5520 0x7f6d081eb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6cfffff640 1 --2- 172.21.15.1:0/739196129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d080c84c0 0x7f6d081559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6d077fe640 1 -- 172.21.15.1:0/739196129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d080c84c0 msgr2=0x7f6d081559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6d077fe640 1 --2- 172.21.15.1:0/739196129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d080c84c0 0x7f6d081559a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6d077fe640 1 -- 172.21.15.1:0/739196129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d080c5520 msgr2=0x7f6d081eb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6d077fe640 1 --2- 172.21.15.1:0/739196129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d080c5520 0x7f6d081eb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6d077fe640 1 -- 172.21.15.1:0/739196129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d081539f0 con 0x7f6d08159240 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6d06ffd640 1 --2- 172.21.15.1:0/739196129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d080c5520 0x7f6d081eb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6d077fe640 1 --2- 172.21.15.1:0/739196129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d08159240 0x7f6d0814d9f0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000c6b0 tx=0x7f6d0000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6cff7fe640 1 -- 172.21.15.1:0/739196129 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d00003b70 con 0x7f6d08159240 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.723+0000 7f6cff7fe640 1 -- 172.21.15.1:0/739196129 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d00003d30 con 0x7f6d08159240 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.724+0000 7f6d0d678640 1 -- 172.21.15.1:0/739196129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d08153cd0 con 0x7f6d08159240 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.724+0000 7f6cff7fe640 1 -- 172.21.15.1:0/739196129 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d00005310 con 0x7f6d08159240 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.724+0000 7f6d0d678640 1 -- 172.21.15.1:0/739196129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d08156690 con 0x7f6d08159240 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.725+0000 7f6d0d678640 1 -- 172.21.15.1:0/739196129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d08066d00 con 0x7f6d08159240 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.725+0000 7f6cff7fe640 1 -- 172.21.15.1:0/739196129 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d000054b0 con 0x7f6d08159240 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.725+0000 7f6cff7fe640 1 --2- 172.21.15.1:0/739196129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ce806b500 0x7f6ce806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.725+0000 7f6cff7fe640 1 -- 172.21.15.1:0/739196129 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f6d00089dd0 con 0x7f6d08159240 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.726+0000 7f6d06ffd640 1 --2- 172.21.15.1:0/739196129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ce806b500 0x7f6ce806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.726+0000 7f6d06ffd640 1 --2- 172.21.15.1:0/739196129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ce806b500 0x7f6ce806d9c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8036000 tx=0x7f6cf803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.730+0000 7f6cff7fe640 1 -- 172.21.15.1:0/739196129 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d08066d00 con 0x7f6d08159240 2024-03-06T07:22:53.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.980+0000 7f6d0d678640 1 -- 172.21.15.1:0/739196129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6d080686e0 con 0x7f6d08159240 2024-03-06T07:22:53.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.980+0000 7f6cff7fe640 1 -- 172.21.15.1:0/739196129 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f6d0008b090 con 0x7f6d08159240 2024-03-06T07:22:53.982 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:53.982 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:22:53.982 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-06T07:22:53.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:53.983 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-06T07:22:53.983 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlnxyWARAAKIPXU0IafNQ1RzBPX+3DYw==", 2024-03-06T07:22:53.983 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:53.983 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:22:53.983 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:22:53.983 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:22:53.983 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:53.983 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:53.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:53.983 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-06T07:22:53.983 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl2fXxDRAAuLqH+HpKcN1JTts9ZtDTXA==", 2024-03-06T07:22:53.983 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:53.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:22:53.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:22:53.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:22:53.984 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:53.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:53.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:53.984 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-06T07:22:53.984 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl+IzqCBAAmgLkarFGt9Ihs/8kJYicBQ==", 2024-03-06T07:22:53.984 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:53.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:22:53.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:22:53.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:22:53.984 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:53.985 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:53.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:53.985 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-06T07:22:53.985 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlipdRFRAAA34vrsTEoX3L1CKb1YAV9g==", 2024-03-06T07:22:53.985 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:53.985 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:22:53.985 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:22:53.985 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:22:53.985 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:53.985 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:53.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:53.985 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAGGehlQK39LxAAOpPIZvl3FqQwCgyo3yoOGQ==", 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAJGehlTxwDLhAAMwN9E0+3Jh5XNXnLo+fw3g==", 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:53.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:53.987 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-06T07:22:53.987 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAMGehl4qrlLBAAm0VWQRNvSQIpDx5xf4Fyig==", 2024-03-06T07:22:53.987 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:53.987 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:53.987 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:22:53.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:53.987 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:53.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:53.987 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-06T07:22:53.987 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAPGehllO5HKxAAozm6ycG+fRxatoZTogeJTA==", 2024-03-06T07:22:53.987 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:53.987 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:53.987 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAVGehlLtLBMhAAUrV6q6ZCQluIcoo/S/NdbA==", 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAYGehlHp59MRAAxxgpSPTEZkYKjl3fr0xeeQ==", 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAbGehlTnXcMBAAFOBqcnh5ubAz+o6AGjccWQ==", 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:53.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:53.990 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-06T07:22:53.990 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAeGehl1qH1LhAAaSl7Aj7qlN6d4xt5SKAmcg==", 2024-03-06T07:22:53.990 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:53.990 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:53.990 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:22:53.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:53.990 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:53.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:53.990 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-06T07:22:53.990 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng==", 2024-03-06T07:22:53.990 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:53.991 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:22:53.991 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-06T07:22:53.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:22:53.991 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:53.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:53.991 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-06T07:22:53.991 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlMyv7HBAAi7vt7aZWEaZuMpFrB5ofZw==", 2024-03-06T07:22:53.991 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:53.991 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:22:53.991 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-06T07:22:53.991 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-06T07:22:53.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-06T07:22:53.991 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:53.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:53.992 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-06T07:22:53.992 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlBS/vIBAAN86N2K4ze5PJB8kR869Lfg==", 2024-03-06T07:22:53.992 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:53.992 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:22:53.992 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:22:53.992 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:22:53.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:53.992 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:53.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:53.992 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-06T07:22:53.992 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlJ/jPORAAhLk5MTGafsGKhZSYhvhBZg==", 2024-03-06T07:22:53.992 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:53.992 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:22:53.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:53.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:53.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:22:53.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:53.993 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-06T07:22:53.993 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlAl8rNhAAvdU0Qvo/FJYnxc4/H8MIqQ==", 2024-03-06T07:22:53.993 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:22:53.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:22:53.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:53.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:22:53.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:53.993 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-06T07:22:53.993 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:22:53.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.982+0000 7f6d0d678640 1 -- 172.21.15.1:0/739196129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ce806b500 msgr2=0x7f6ce806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.982+0000 7f6d0d678640 1 --2- 172.21.15.1:0/739196129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ce806b500 0x7f6ce806d9c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8036000 tx=0x7f6cf803c040 comp rx=0 tx=0).stop 2024-03-06T07:22:53.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.982+0000 7f6d0d678640 1 -- 172.21.15.1:0/739196129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d08159240 msgr2=0x7f6d0814d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.982+0000 7f6d0d678640 1 --2- 172.21.15.1:0/739196129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d08159240 0x7f6d0814d9f0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000c6b0 tx=0x7f6d0000cb70 comp rx=0 tx=0).stop 2024-03-06T07:22:53.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.982+0000 7f6d0d678640 1 -- 172.21.15.1:0/739196129 shutdown_connections 2024-03-06T07:22:53.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.982+0000 7f6d0d678640 1 --2- 172.21.15.1:0/739196129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ce806b500 0x7f6ce806d9c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.982+0000 7f6d0d678640 1 --2- 172.21.15.1:0/739196129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d08159240 0x7f6d0814d9f0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.982+0000 7f6d0d678640 1 --2- 172.21.15.1:0/739196129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d080c84c0 0x7f6d081559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.982+0000 7f6d0d678640 1 --2- 172.21.15.1:0/739196129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d080c5520 0x7f6d081eb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.982+0000 7f6d0d678640 1 -- 172.21.15.1:0/739196129 >> 172.21.15.1:0/739196129 conn(0x7f6d080c88f0 msgr2=0x7f6d080c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.982+0000 7f6d0d678640 1 -- 172.21.15.1:0/739196129 shutdown_connections 2024-03-06T07:22:53.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:53.982+0000 7f6d0d678640 1 -- 172.21.15.1:0/739196129 wait complete. 2024-03-06T07:22:53.996 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.095+0000 7f6b23fac640 1 Processor -- start 2024-03-06T07:22:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.095+0000 7f6b23fac640 1 -- start start 2024-03-06T07:22:54.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.096+0000 7f6b23fac640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6b1c0c6b10 0x7f6b1c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.096+0000 7f6b23fac640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1c0c2fc0 con 0x7f6b1c0bf9d0 2024-03-06T07:22:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.096+0000 7f6b23fac640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1c158da0 con 0x7f6b1c152650 2024-03-06T07:22:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.096+0000 7f6b23fac640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b1c155800 con 0x7f6b1c0c6b10 2024-03-06T07:22:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.096+0000 7f6b22549640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6b1c0bf9d0 0x7f6b1c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58900/0 (socket says 172.21.15.1:58900) 2024-03-06T07:22:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.096+0000 7f6b22549640 1 -- 172.21.15.1:0/3579899152 learned_addr learned my addr 172.21.15.1:0/3579899152 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b20d46640 1 -- 172.21.15.1:0/3579899152 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658533824 0 0) 0x7f6b1c0c2fc0 con 0x7f6b1c0bf9d0 2024-03-06T07:22:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b21547640 1 --2- 172.21.15.1:0/3579899152 >> v2:172.21.15.98:3300/0 conn(0x7f6b1c0c6b10 0x7f6b1c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b20d46640 1 -- 172.21.15.1:0/3579899152 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b04003520 con 0x7f6b1c0bf9d0 2024-03-06T07:22:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b20d46640 1 -- 172.21.15.1:0/3579899152 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1661737204 0 0) 0x7f6b04003520 con 0x7f6b1c0bf9d0 2024-03-06T07:22:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b20d46640 1 -- 172.21.15.1:0/3579899152 >> v2:172.21.15.98:3300/0 conn(0x7f6b1c0c6b10 msgr2=0x7f6b1c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b20d46640 1 --2- 172.21.15.1:0/3579899152 >> v2:172.21.15.98:3300/0 conn(0x7f6b1c0c6b10 0x7f6b1c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b20d46640 1 -- 172.21.15.1:0/3579899152 >> v1:172.21.15.98:6789/0 conn(0x7f6b1c152650 legacy=0x7f6b1c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b20d46640 1 -- 172.21.15.1:0/3579899152 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c0c6690 con 0x7f6b1c0bf9d0 2024-03-06T07:22:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b20d46640 1 -- 172.21.15.1:0/3579899152 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f6b0c0019f0 con 0x7f6b1c0bf9d0 2024-03-06T07:22:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b20d46640 1 -- 172.21.15.1:0/3579899152 >> v1:172.21.15.1:6789/0 conn(0x7f6b1c0bf9d0 legacy=0x7f6b1c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b20d46640 1 --2- 172.21.15.1:0/3579899152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b040039b0 0x7f6b04003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b20d46640 1 --2- 172.21.15.1:0/3579899152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b04004800 0x7f6b04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b20d46640 1 --2- 172.21.15.1:0/3579899152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b040051c0 0x7f6b0400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b20d46640 1 -- 172.21.15.1:0/3579899152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c155800 con 0x7f6b040039b0 2024-03-06T07:22:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b20d46640 1 -- 172.21.15.1:0/3579899152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0400c160 con 0x7f6b04004800 2024-03-06T07:22:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b20d46640 1 -- 172.21.15.1:0/3579899152 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0400c360 con 0x7f6b040051c0 2024-03-06T07:22:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b21d48640 1 --2- 172.21.15.1:0/3579899152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b040039b0 0x7f6b04003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b21547640 1 --2- 172.21.15.1:0/3579899152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b04004800 0x7f6b04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b22549640 1 --2- 172.21.15.1:0/3579899152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b040051c0 0x7f6b0400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b21d48640 1 -- 172.21.15.1:0/3579899152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b040051c0 msgr2=0x7f6b0400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b21d48640 1 --2- 172.21.15.1:0/3579899152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b040051c0 0x7f6b0400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b21d48640 1 -- 172.21.15.1:0/3579899152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b04004800 msgr2=0x7f6b04004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b21d48640 1 --2- 172.21.15.1:0/3579899152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b04004800 0x7f6b04004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b21d48640 1 -- 172.21.15.1:0/3579899152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b04003520 con 0x7f6b040039b0 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b22549640 1 --2- 172.21.15.1:0/3579899152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b040051c0 0x7f6b0400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.097+0000 7f6b22549640 1 -- 172.21.15.1:0/3579899152 reap_dead start 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.098+0000 7f6b21d48640 1 --2- 172.21.15.1:0/3579899152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b040039b0 0x7f6b04003dd0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f6b14001750 tx=0x7f6b04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e57862c8ef73761 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.098+0000 7f6b20d46640 1 -- 172.21.15.1:0/3579899152 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b14001370 con 0x7f6b040039b0 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.098+0000 7f6b20d46640 1 -- 172.21.15.1:0/3579899152 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b140052d0 con 0x7f6b040039b0 2024-03-06T07:22:54.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.098+0000 7f6b20d46640 1 -- 172.21.15.1:0/3579899152 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b14003830 con 0x7f6b040039b0 2024-03-06T07:22:54.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.099+0000 7f6b23fac640 1 -- 172.21.15.1:0/3579899152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b040039b0 msgr2=0x7f6b04003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.099+0000 7f6b23fac640 1 --2- 172.21.15.1:0/3579899152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b040039b0 0x7f6b04003dd0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f6b14001750 tx=0x7f6b04005830 comp rx=0 tx=0).stop 2024-03-06T07:22:54.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.099+0000 7f6b23fac640 1 -- 172.21.15.1:0/3579899152 shutdown_connections 2024-03-06T07:22:54.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.099+0000 7f6b23fac640 1 --2- 172.21.15.1:0/3579899152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b040039b0 0x7f6b04003dd0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.099+0000 7f6b23fac640 1 -- 172.21.15.1:0/3579899152 >> 172.21.15.1:0/3579899152 conn(0x7f6b1c066ee0 msgr2=0x7f6b1c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.099+0000 7f6b23fac640 1 -- 172.21.15.1:0/3579899152 shutdown_connections 2024-03-06T07:22:54.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.099+0000 7f6b23fac640 1 -- 172.21.15.1:0/3579899152 wait complete. 2024-03-06T07:22:54.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.100+0000 7f6b23fac640 1 Processor -- start 2024-03-06T07:22:54.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.100+0000 7f6b23fac640 1 -- start start 2024-03-06T07:22:54.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.100+0000 7f6b23fac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b040039b0 0x7f6b1c0c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.100+0000 7f6b23fac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b1c1e1310 0x7f6b1c157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.100+0000 7f6b23fac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b1c158090 0x7f6b1c152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.100+0000 7f6b23fac640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c1551b0 con 0x7f6b040039b0 2024-03-06T07:22:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b23fac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c0c6fa0 con 0x7f6b1c1e1310 2024-03-06T07:22:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b23fac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c155490 con 0x7f6b1c158090 2024-03-06T07:22:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b21d48640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b040039b0 0x7f6b1c0c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b21d48640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b040039b0 0x7f6b1c0c5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45862/0 (socket says 172.21.15.1:45862) 2024-03-06T07:22:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b21d48640 1 -- 172.21.15.1:0/2373876457 learned_addr learned my addr 172.21.15.1:0/2373876457 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b22549640 1 --2- 172.21.15.1:0/2373876457 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b1c158090 0x7f6b1c152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b21547640 1 --2- 172.21.15.1:0/2373876457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b1c1e1310 0x7f6b1c157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b21d48640 1 -- 172.21.15.1:0/2373876457 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b1c158090 msgr2=0x7f6b1c152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b21d48640 1 --2- 172.21.15.1:0/2373876457 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b1c158090 0x7f6b1c152ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b21d48640 1 -- 172.21.15.1:0/2373876457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b1c1e1310 msgr2=0x7f6b1c157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b21d48640 1 --2- 172.21.15.1:0/2373876457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b1c1e1310 0x7f6b1c157b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b21d48640 1 -- 172.21.15.1:0/2373876457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c1e1810 con 0x7f6b040039b0 2024-03-06T07:22:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b21d48640 1 --2- 172.21.15.1:0/2373876457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b040039b0 0x7f6b1c0c5ac0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6b14005040 tx=0x7f6b14001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b1a7fc640 1 -- 172.21.15.1:0/2373876457 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b14041020 con 0x7f6b040039b0 2024-03-06T07:22:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b1a7fc640 1 -- 172.21.15.1:0/2373876457 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b14005480 con 0x7f6b040039b0 2024-03-06T07:22:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b1a7fc640 1 -- 172.21.15.1:0/2373876457 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b14007a30 con 0x7f6b040039b0 2024-03-06T07:22:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b23fac640 1 -- 172.21.15.1:0/2373876457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b1c14d7f0 con 0x7f6b040039b0 2024-03-06T07:22:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.101+0000 7f6b23fac640 1 -- 172.21.15.1:0/2373876457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b1c0c6000 con 0x7f6b040039b0 2024-03-06T07:22:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.102+0000 7f6b23fac640 1 -- 172.21.15.1:0/2373876457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b1c067fd0 con 0x7f6b040039b0 2024-03-06T07:22:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.103+0000 7f6b1a7fc640 1 -- 172.21.15.1:0/2373876457 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b14044020 con 0x7f6b040039b0 2024-03-06T07:22:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.103+0000 7f6b1a7fc640 1 --2- 172.21.15.1:0/2373876457 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b0806b500 0x7f6b0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.103+0000 7f6b1a7fc640 1 -- 172.21.15.1:0/2373876457 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f6b140ab810 con 0x7f6b040039b0 2024-03-06T07:22:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.103+0000 7f6b21547640 1 --2- 172.21.15.1:0/2373876457 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b0806b500 0x7f6b0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.104+0000 7f6b21547640 1 --2- 172.21.15.1:0/2373876457 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b0806b500 0x7f6b0806d9c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c1ef280 tx=0x7f6b10009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.107+0000 7f6b1a7fc640 1 -- 172.21.15.1:0/2373876457 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b1404b030 con 0x7f6b040039b0 2024-03-06T07:22:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.288+0000 7f6b23fac640 1 -- 172.21.15.1:0/2373876457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b1c0b06f0 con 0x7f6b040039b0 2024-03-06T07:22:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.289+0000 7f6b1a7fc640 1 -- 172.21.15.1:0/2373876457 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+13045 (secure 0 0 0) 0x7f6b1407a850 con 0x7f6b040039b0 2024-03-06T07:22:54.291 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:54.291 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":70,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:53.591368+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:54.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.291+0000 7f6b23fac640 1 -- 172.21.15.1:0/2373876457 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b0806b500 msgr2=0x7f6b0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.291+0000 7f6b23fac640 1 --2- 172.21.15.1:0/2373876457 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b0806b500 0x7f6b0806d9c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c1ef280 tx=0x7f6b10009190 comp rx=0 tx=0).stop 2024-03-06T07:22:54.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.291+0000 7f6b23fac640 1 -- 172.21.15.1:0/2373876457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b040039b0 msgr2=0x7f6b1c0c5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.291+0000 7f6b23fac640 1 --2- 172.21.15.1:0/2373876457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b040039b0 0x7f6b1c0c5ac0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6b14005040 tx=0x7f6b14001120 comp rx=0 tx=0).stop 2024-03-06T07:22:54.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.291+0000 7f6b23fac640 1 -- 172.21.15.1:0/2373876457 shutdown_connections 2024-03-06T07:22:54.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.291+0000 7f6b23fac640 1 --2- 172.21.15.1:0/2373876457 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b0806b500 0x7f6b0806d9c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.291+0000 7f6b23fac640 1 --2- 172.21.15.1:0/2373876457 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b1c158090 0x7f6b1c152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.291+0000 7f6b23fac640 1 --2- 172.21.15.1:0/2373876457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b1c1e1310 0x7f6b1c157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.291+0000 7f6b23fac640 1 --2- 172.21.15.1:0/2373876457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b040039b0 0x7f6b1c0c5ac0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.291+0000 7f6b23fac640 1 -- 172.21.15.1:0/2373876457 >> 172.21.15.1:0/2373876457 conn(0x7f6b1c066ee0 msgr2=0x7f6b1c14e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.291+0000 7f6b23fac640 1 -- 172.21.15.1:0/2373876457 shutdown_connections 2024-03-06T07:22:54.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.291+0000 7f6b23fac640 1 -- 172.21.15.1:0/2373876457 wait complete. 2024-03-06T07:22:54.304 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:54.304 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:54.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.406+0000 7f1e1ff4b640 1 Processor -- start 2024-03-06T07:22:54.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.407+0000 7f1e1ff4b640 1 -- start start 2024-03-06T07:22:54.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.407+0000 7f1e1ff4b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e18150fe0 0x7f1e181513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.407+0000 7f1e1ff4b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1e18071f90 0x7f1e18072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.407+0000 7f1e1ff4b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e180a6140 con 0x7f1e18072f20 2024-03-06T07:22:54.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.407+0000 7f1e1ff4b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e181526b0 con 0x7f1e18071f90 2024-03-06T07:22:54.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.407+0000 7f1e1ff4b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e18151aa0 con 0x7f1e18150fe0 2024-03-06T07:22:54.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.408+0000 7f1e1dce7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e18150fe0 0x7f1e181513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.408+0000 7f1e1e4e8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1e18071f90 0x7f1e18072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.408+0000 7f1e1e4e8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1e18071f90 0x7f1e18072420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:57598/0 (socket says 172.21.15.1:57598) 2024-03-06T07:22:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.408+0000 7f1e1e4e8640 1 -- 172.21.15.1:0/2576424514 learned_addr learned my addr 172.21.15.1:0/2576424514 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.408+0000 7f1e1dce7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e18150fe0 0x7f1e181513c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:47452/0 (socket says 172.21.15.1:47452) 2024-03-06T07:22:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.408+0000 7f1e1dce7640 1 -- 172.21.15.1:0/2576424514 >> v2:172.21.15.98:3300/0 conn(0x7f1e18071f90 msgr2=0x7f1e18072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.408+0000 7f1e1dce7640 1 --2- 172.21.15.1:0/2576424514 >> v2:172.21.15.98:3300/0 conn(0x7f1e18071f90 0x7f1e18072420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.408+0000 7f1e1dce7640 1 -- 172.21.15.1:0/2576424514 >> v1:172.21.15.98:6789/0 conn(0x7f1e18072f20 legacy=0x7f1e1806edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.408+0000 7f1e1dce7640 1 -- 172.21.15.1:0/2576424514 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e18151d80 con 0x7f1e18150fe0 2024-03-06T07:22:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.408+0000 7f1e1e4e8640 1 --2- 172.21.15.1:0/2576424514 >> v2:172.21.15.98:3300/0 conn(0x7f1e18071f90 0x7f1e18072420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.409+0000 7f1e1dce7640 1 --2- 172.21.15.1:0/2576424514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e18150fe0 0x7f1e181513c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1e140097a0 tx=0x7f1e1402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=826045ff73a809fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.409+0000 7f1e1cce5640 1 -- 172.21.15.1:0/2576424514 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e140315a0 con 0x7f1e18150fe0 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.409+0000 7f1e1cce5640 1 -- 172.21.15.1:0/2576424514 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e14037070 con 0x7f1e18150fe0 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.409+0000 7f1e1cce5640 1 -- 172.21.15.1:0/2576424514 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e14031960 con 0x7f1e18150fe0 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.410+0000 7f1e1ff4b640 1 -- 172.21.15.1:0/2576424514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e18150fe0 msgr2=0x7f1e181513c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.410+0000 7f1e1ff4b640 1 --2- 172.21.15.1:0/2576424514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e18150fe0 0x7f1e181513c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1e140097a0 tx=0x7f1e1402b840 comp rx=0 tx=0).stop 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.410+0000 7f1e1ff4b640 1 -- 172.21.15.1:0/2576424514 shutdown_connections 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.410+0000 7f1e1ff4b640 1 --2- 172.21.15.1:0/2576424514 >> v2:172.21.15.98:3300/0 conn(0x7f1e18071f90 0x7f1e18072420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.410+0000 7f1e1ff4b640 1 --2- 172.21.15.1:0/2576424514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e18150fe0 0x7f1e181513c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.410+0000 7f1e1ff4b640 1 -- 172.21.15.1:0/2576424514 >> 172.21.15.1:0/2576424514 conn(0x7f1e180c6860 msgr2=0x7f1e180c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.410+0000 7f1e1ff4b640 1 -- 172.21.15.1:0/2576424514 shutdown_connections 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.410+0000 7f1e1ff4b640 1 -- 172.21.15.1:0/2576424514 wait complete. 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.411+0000 7f1e1ff4b640 1 Processor -- start 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.411+0000 7f1e1ff4b640 1 -- start start 2024-03-06T07:22:54.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.411+0000 7f1e1ff4b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e18071f90 0x7f1e18154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.411+0000 7f1e1ff4b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e18072f20 0x7f1e180bca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.411+0000 7f1e1ff4b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e18150fe0 0x7f1e181488a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.411+0000 7f1e1ff4b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1806f4c0 con 0x7f1e18072f20 2024-03-06T07:22:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.411+0000 7f1e1ff4b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e18152340 con 0x7f1e18071f90 2024-03-06T07:22:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.412+0000 7f1e1ff4b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1806f640 con 0x7f1e18150fe0 2024-03-06T07:22:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.412+0000 7f1e1d4e6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e18072f20 0x7f1e180bca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.412+0000 7f1e1d4e6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e18072f20 0x7f1e180bca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57552/0 (socket says 172.21.15.1:57552) 2024-03-06T07:22:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.412+0000 7f1e1d4e6640 1 -- 172.21.15.1:0/3078798741 learned_addr learned my addr 172.21.15.1:0/3078798741 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.412+0000 7f1e1e4e8640 1 --2- 172.21.15.1:0/3078798741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e18150fe0 0x7f1e181488a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.412+0000 7f1e1dce7640 1 --2- 172.21.15.1:0/3078798741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e18071f90 0x7f1e18154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.412+0000 7f1e1d4e6640 1 -- 172.21.15.1:0/3078798741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e18150fe0 msgr2=0x7f1e181488a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.412+0000 7f1e1d4e6640 1 --2- 172.21.15.1:0/3078798741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e18150fe0 0x7f1e181488a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.412+0000 7f1e1d4e6640 1 -- 172.21.15.1:0/3078798741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e18071f90 msgr2=0x7f1e18154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.412+0000 7f1e1d4e6640 1 --2- 172.21.15.1:0/3078798741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e18071f90 0x7f1e18154f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.412+0000 7f1e1d4e6640 1 -- 172.21.15.1:0/3078798741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e181e80b0 con 0x7f1e18072f20 2024-03-06T07:22:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.412+0000 7f1e1d4e6640 1 --2- 172.21.15.1:0/3078798741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e18072f20 0x7f1e180bca90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f1e0800c400 tx=0x7f1e0800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.412+0000 7f1e0e7fc640 1 -- 172.21.15.1:0/3078798741 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e080052d0 con 0x7f1e18072f20 2024-03-06T07:22:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.412+0000 7f1e0e7fc640 1 -- 172.21.15.1:0/3078798741 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e08005490 con 0x7f1e18072f20 2024-03-06T07:22:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.412+0000 7f1e0e7fc640 1 -- 172.21.15.1:0/3078798741 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e08007260 con 0x7f1e18072f20 2024-03-06T07:22:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.412+0000 7f1e1ff4b640 1 -- 172.21.15.1:0/3078798741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e18155660 con 0x7f1e18072f20 2024-03-06T07:22:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.413+0000 7f1e1ff4b640 1 -- 172.21.15.1:0/3078798741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e180bcfd0 con 0x7f1e18072f20 2024-03-06T07:22:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.413+0000 7f1e1ff4b640 1 -- 172.21.15.1:0/3078798741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e180c2bd0 con 0x7f1e18072f20 2024-03-06T07:22:54.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.414+0000 7f1e0e7fc640 1 -- 172.21.15.1:0/3078798741 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e08007400 con 0x7f1e18072f20 2024-03-06T07:22:54.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.414+0000 7f1e0e7fc640 1 --2- 172.21.15.1:0/3078798741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1dfc06b500 0x7f1dfc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.414+0000 7f1e0e7fc640 1 -- 172.21.15.1:0/3078798741 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f1e080895b0 con 0x7f1e18072f20 2024-03-06T07:22:54.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.415+0000 7f1e1dce7640 1 --2- 172.21.15.1:0/3078798741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1dfc06b500 0x7f1dfc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.415+0000 7f1e1dce7640 1 --2- 172.21.15.1:0/3078798741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1dfc06b500 0x7f1dfc06d9c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1e14036000 tx=0x7f1e1403d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.419+0000 7f1e0e7fc640 1 -- 172.21.15.1:0/3078798741 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e180c2bd0 con 0x7f1e18072f20 2024-03-06T07:22:54.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.599+0000 7f1e1ff4b640 1 -- 172.21.15.1:0/3078798741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1e18155940 con 0x7f1e18072f20 2024-03-06T07:22:54.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.599+0000 7f1e0e7fc640 1 -- 172.21.15.1:0/3078798741 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1e18155940 con 0x7f1e18072f20 2024-03-06T07:22:54.601 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:22:54.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.601+0000 7f1e1ff4b640 1 -- 172.21.15.1:0/3078798741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1dfc06b500 msgr2=0x7f1dfc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.601+0000 7f1e1ff4b640 1 --2- 172.21.15.1:0/3078798741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1dfc06b500 0x7f1dfc06d9c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1e14036000 tx=0x7f1e1403d040 comp rx=0 tx=0).stop 2024-03-06T07:22:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.601+0000 7f1e1ff4b640 1 -- 172.21.15.1:0/3078798741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e18072f20 msgr2=0x7f1e180bca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.601+0000 7f1e1ff4b640 1 --2- 172.21.15.1:0/3078798741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e18072f20 0x7f1e180bca90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f1e0800c400 tx=0x7f1e0800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.601+0000 7f1e1ff4b640 1 -- 172.21.15.1:0/3078798741 shutdown_connections 2024-03-06T07:22:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.601+0000 7f1e1ff4b640 1 --2- 172.21.15.1:0/3078798741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1dfc06b500 0x7f1dfc06d9c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.601+0000 7f1e1ff4b640 1 --2- 172.21.15.1:0/3078798741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e18150fe0 0x7f1e181488a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.601+0000 7f1e1ff4b640 1 --2- 172.21.15.1:0/3078798741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e18072f20 0x7f1e180bca90 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.601+0000 7f1e1ff4b640 1 --2- 172.21.15.1:0/3078798741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e18071f90 0x7f1e18154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.601+0000 7f1e1ff4b640 1 -- 172.21.15.1:0/3078798741 >> 172.21.15.1:0/3078798741 conn(0x7f1e180c6860 msgr2=0x7f1e1806df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.601+0000 7f1e1ff4b640 1 -- 172.21.15.1:0/3078798741 shutdown_connections 2024-03-06T07:22:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.601+0000 7f1e1ff4b640 1 -- 172.21.15.1:0/3078798741 wait complete. 2024-03-06T07:22:54.614 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:22:54.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.714+0000 7fe3ee066640 1 Processor -- start 2024-03-06T07:22:54.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.714+0000 7fe3ee066640 1 -- start start 2024-03-06T07:22:54.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.714+0000 7fe3ee066640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3e8072bd0 0x7fe3e80c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.714+0000 7fe3ee066640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe3e8152b70 0x7fe3e80c3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.714+0000 7fe3ee066640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3e8151dd0 con 0x7fe3e80c2b30 2024-03-06T07:22:54.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.714+0000 7fe3ee066640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3e81526e0 con 0x7fe3e8152b70 2024-03-06T07:22:54.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.714+0000 7fe3ee066640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e8156190 con 0x7fe3e8072bd0 2024-03-06T07:22:54.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.715+0000 7fe3e6ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3e8072bd0 0x7fe3e80c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.715+0000 7fe3e7fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe3e8152b70 0x7fe3e80c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.715+0000 7fe3e6ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3e8072bd0 0x7fe3e80c32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:47468/0 (socket says 172.21.15.1:47468) 2024-03-06T07:22:54.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.715+0000 7fe3e77fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe3e80c2b30 0x7fe3e80724c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:58354/0 (socket says 172.21.15.1:58354) 2024-03-06T07:22:54.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.715+0000 7fe3e6ffd640 1 -- 172.21.15.1:0/1509446299 learned_addr learned my addr 172.21.15.1:0/1509446299 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:54.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.715+0000 7fe3e7fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe3e8152b70 0x7fe3e80c3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:57622/0 (socket says 172.21.15.1:57622) 2024-03-06T07:22:54.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.715+0000 7fe3e6ffd640 1 -- 172.21.15.1:0/1509446299 >> v2:172.21.15.98:3300/0 conn(0x7fe3e8152b70 msgr2=0x7fe3e80c3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.715+0000 7fe3e6ffd640 1 --2- 172.21.15.1:0/1509446299 >> v2:172.21.15.98:3300/0 conn(0x7fe3e8152b70 0x7fe3e80c3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.715+0000 7fe3e6ffd640 1 -- 172.21.15.1:0/1509446299 >> v1:172.21.15.98:6789/0 conn(0x7fe3e80c2b30 legacy=0x7fe3e80724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.715+0000 7fe3e6ffd640 1 -- 172.21.15.1:0/1509446299 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e8152090 con 0x7fe3e8072bd0 2024-03-06T07:22:54.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.715+0000 7fe3e7fff640 1 --2- 172.21.15.1:0/1509446299 >> v2:172.21.15.98:3300/0 conn(0x7fe3e8152b70 0x7fe3e80c3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:54.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.716+0000 7fe3e6ffd640 1 --2- 172.21.15.1:0/1509446299 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3e8072bd0 0x7fe3e80c32f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc00bb10 tx=0x7fe3dc02e150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f42ae3a68701731 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.716+0000 7fe3e67fc640 1 -- 172.21.15.1:0/1509446299 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc03b070 con 0x7fe3e8072bd0 2024-03-06T07:22:54.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.716+0000 7fe3e67fc640 1 -- 172.21.15.1:0/1509446299 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3dc036440 con 0x7fe3e8072bd0 2024-03-06T07:22:54.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.716+0000 7fe3e67fc640 1 -- 172.21.15.1:0/1509446299 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc036810 con 0x7fe3e8072bd0 2024-03-06T07:22:54.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.717+0000 7fe3ee066640 1 -- 172.21.15.1:0/1509446299 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3e8072bd0 msgr2=0x7fe3e80c32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.717+0000 7fe3ee066640 1 --2- 172.21.15.1:0/1509446299 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3e8072bd0 0x7fe3e80c32f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc00bb10 tx=0x7fe3dc02e150 comp rx=0 tx=0).stop 2024-03-06T07:22:54.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.717+0000 7fe3ee066640 1 -- 172.21.15.1:0/1509446299 shutdown_connections 2024-03-06T07:22:54.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.717+0000 7fe3ee066640 1 --2- 172.21.15.1:0/1509446299 >> v2:172.21.15.98:3300/0 conn(0x7fe3e8152b70 0x7fe3e80c3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.717+0000 7fe3ee066640 1 --2- 172.21.15.1:0/1509446299 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3e8072bd0 0x7fe3e80c32f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.717+0000 7fe3ee066640 1 -- 172.21.15.1:0/1509446299 >> 172.21.15.1:0/1509446299 conn(0x7fe3e80c6860 msgr2=0x7fe3e80c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.717+0000 7fe3ee066640 1 -- 172.21.15.1:0/1509446299 shutdown_connections 2024-03-06T07:22:54.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.717+0000 7fe3ee066640 1 -- 172.21.15.1:0/1509446299 wait complete. 2024-03-06T07:22:54.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.718+0000 7fe3ee066640 1 Processor -- start 2024-03-06T07:22:54.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.718+0000 7fe3ee066640 1 -- start start 2024-03-06T07:22:54.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3ee066640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3e8072bd0 0x7fe3e8154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3ee066640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e80c2b30 0x7fe3e80bfe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3ee066640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3e8152b70 0x7fe3e8149b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3ee066640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e80c3ef0 con 0x7fe3e8072bd0 2024-03-06T07:22:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3ee066640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e80c3d70 con 0x7fe3e80c2b30 2024-03-06T07:22:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3ee066640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e8155d80 con 0x7fe3e8152b70 2024-03-06T07:22:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3e77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3e8072bd0 0x7fe3e8154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3e77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3e8072bd0 0x7fe3e8154480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57556/0 (socket says 172.21.15.1:57556) 2024-03-06T07:22:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3e77fe640 1 -- 172.21.15.1:0/3033494170 learned_addr learned my addr 172.21.15.1:0/3033494170 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3e7fff640 1 --2- 172.21.15.1:0/3033494170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3e8152b70 0x7fe3e8149b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3e6ffd640 1 --2- 172.21.15.1:0/3033494170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e80c2b30 0x7fe3e80bfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3e77fe640 1 -- 172.21.15.1:0/3033494170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3e8152b70 msgr2=0x7fe3e8149b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3e77fe640 1 --2- 172.21.15.1:0/3033494170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3e8152b70 0x7fe3e8149b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3e77fe640 1 -- 172.21.15.1:0/3033494170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e80c2b30 msgr2=0x7fe3e80bfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3e77fe640 1 --2- 172.21.15.1:0/3033494170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e80c2b30 0x7fe3e80bfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3e77fe640 1 -- 172.21.15.1:0/3033494170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e80ab200 con 0x7fe3e8072bd0 2024-03-06T07:22:54.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3e77fe640 1 --2- 172.21.15.1:0/3033494170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3e8072bd0 0x7fe3e8154480 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800c400 tx=0x7fe3d800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3cffff640 1 -- 172.21.15.1:0/3033494170 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d80074e0 con 0x7fe3e8072bd0 2024-03-06T07:22:54.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.719+0000 7fe3cffff640 1 -- 172.21.15.1:0/3033494170 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d8007680 con 0x7fe3e8072bd0 2024-03-06T07:22:54.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.720+0000 7fe3ee066640 1 -- 172.21.15.1:0/3033494170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e80ab4e0 con 0x7fe3e8072bd0 2024-03-06T07:22:54.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.720+0000 7fe3cffff640 1 -- 172.21.15.1:0/3033494170 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8004a00 con 0x7fe3e8072bd0 2024-03-06T07:22:54.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.720+0000 7fe3ee066640 1 -- 172.21.15.1:0/3033494170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e80abaa0 con 0x7fe3e8072bd0 2024-03-06T07:22:54.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.721+0000 7fe3ee066640 1 -- 172.21.15.1:0/3033494170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e8067fd0 con 0x7fe3e8072bd0 2024-03-06T07:22:54.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.721+0000 7fe3cffff640 1 -- 172.21.15.1:0/3033494170 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe3d801a020 con 0x7fe3e8072bd0 2024-03-06T07:22:54.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.721+0000 7fe3cffff640 1 --2- 172.21.15.1:0/3033494170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe3d406b500 0x7fe3d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.722+0000 7fe3cffff640 1 -- 172.21.15.1:0/3033494170 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fe3d80897b0 con 0x7fe3e8072bd0 2024-03-06T07:22:54.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.722+0000 7fe3e6ffd640 1 --2- 172.21.15.1:0/3033494170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe3d406b500 0x7fe3d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.722+0000 7fe3e6ffd640 1 --2- 172.21.15.1:0/3033494170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe3d406b500 0x7fe3d406d9c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc038000 tx=0x7fe3dc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.726+0000 7fe3cffff640 1 -- 172.21.15.1:0/3033494170 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3d8010300 con 0x7fe3e8072bd0 2024-03-06T07:22:54.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.909+0000 7fe3ee066640 1 -- 172.21.15.1:0/3033494170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe3e8071790 con 0x7fe3e8072bd0 2024-03-06T07:22:54.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.910+0000 7fe3cffff640 1 -- 172.21.15.1:0/3033494170 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v70) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe3d8058880 con 0x7fe3e8072bd0 2024-03-06T07:22:54.912 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:54.912 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-06T07:22:54.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:22:54.912 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-06T07:22:54.912 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-06T07:22:54.912 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:22:54.912 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-06T07:22:54.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.912+0000 7fe3ee066640 1 -- 172.21.15.1:0/3033494170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe3d406b500 msgr2=0x7fe3d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.912+0000 7fe3ee066640 1 --2- 172.21.15.1:0/3033494170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe3d406b500 0x7fe3d406d9c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc038000 tx=0x7fe3dc008040 comp rx=0 tx=0).stop 2024-03-06T07:22:54.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.912+0000 7fe3ee066640 1 -- 172.21.15.1:0/3033494170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3e8072bd0 msgr2=0x7fe3e8154480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.912+0000 7fe3ee066640 1 --2- 172.21.15.1:0/3033494170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3e8072bd0 0x7fe3e8154480 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800c400 tx=0x7fe3d800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.912+0000 7fe3ee066640 1 -- 172.21.15.1:0/3033494170 shutdown_connections 2024-03-06T07:22:54.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.912+0000 7fe3ee066640 1 --2- 172.21.15.1:0/3033494170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe3d406b500 0x7fe3d406d9c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.912+0000 7fe3ee066640 1 --2- 172.21.15.1:0/3033494170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3e8152b70 0x7fe3e8149b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.912+0000 7fe3ee066640 1 --2- 172.21.15.1:0/3033494170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e80c2b30 0x7fe3e80bfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.912+0000 7fe3ee066640 1 --2- 172.21.15.1:0/3033494170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3e8072bd0 0x7fe3e8154480 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.912+0000 7fe3ee066640 1 -- 172.21.15.1:0/3033494170 >> 172.21.15.1:0/3033494170 conn(0x7fe3e80c6860 msgr2=0x7fe3e80c3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.912+0000 7fe3ee066640 1 -- 172.21.15.1:0/3033494170 shutdown_connections 2024-03-06T07:22:54.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:54.912+0000 7fe3ee066640 1 -- 172.21.15.1:0/3033494170 wait complete. 2024-03-06T07:22:54.924 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:22:54.925 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:22:55.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.027+0000 7f84256c9640 1 Processor -- start 2024-03-06T07:22:55.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.028+0000 7f84256c9640 1 -- start start 2024-03-06T07:22:55.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.028+0000 7f84256c9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8420150820 0x7f842006c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.028+0000 7f84256c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8420067520 0x7f8420153090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.028+0000 7f84256c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84200c33a0 con 0x7f84200c38f0 2024-03-06T07:22:55.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.028+0000 7f84256c9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f842006c1e0 con 0x7f8420150820 2024-03-06T07:22:55.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.028+0000 7f84256c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84200a8550 con 0x7f8420067520 2024-03-06T07:22:55.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.029+0000 7f841e7fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8420067520 0x7f8420153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.029+0000 7f841effd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8420150820 0x7f842006c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.029+0000 7f841e7fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8420067520 0x7f8420153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:47494/0 (socket says 172.21.15.1:47494) 2024-03-06T07:22:55.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.029+0000 7f841e7fc640 1 -- 172.21.15.1:0/3944758145 learned_addr learned my addr 172.21.15.1:0/3944758145 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:55.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.029+0000 7f841e7fc640 1 -- 172.21.15.1:0/3944758145 >> v2:172.21.15.98:3300/0 conn(0x7f8420150820 msgr2=0x7f842006c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.029+0000 7f841e7fc640 1 --2- 172.21.15.1:0/3944758145 >> v2:172.21.15.98:3300/0 conn(0x7f8420150820 0x7f842006c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.029+0000 7f841e7fc640 1 -- 172.21.15.1:0/3944758145 >> v1:172.21.15.98:6789/0 conn(0x7f84200c38f0 legacy=0x7f842006f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.029+0000 7f841e7fc640 1 -- 172.21.15.1:0/3944758145 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84200a8830 con 0x7f8420067520 2024-03-06T07:22:55.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.030+0000 7f841e7fc640 1 --2- 172.21.15.1:0/3944758145 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8420067520 0x7f8420153090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8414009870 tx=0x7f841402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb72a8635b536b52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.030+0000 7f841dffb640 1 -- 172.21.15.1:0/3944758145 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8414035020 con 0x7f8420067520 2024-03-06T07:22:55.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.030+0000 7f841dffb640 1 -- 172.21.15.1:0/3944758145 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8414031640 con 0x7f8420067520 2024-03-06T07:22:55.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.030+0000 7f841dffb640 1 -- 172.21.15.1:0/3944758145 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841402d040 con 0x7f8420067520 2024-03-06T07:22:55.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.030+0000 7f84256c9640 1 -- 172.21.15.1:0/3944758145 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8420067520 msgr2=0x7f8420153090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.030+0000 7f84256c9640 1 --2- 172.21.15.1:0/3944758145 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8420067520 0x7f8420153090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8414009870 tx=0x7f841402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.030+0000 7f84256c9640 1 -- 172.21.15.1:0/3944758145 shutdown_connections 2024-03-06T07:22:55.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.030+0000 7f84256c9640 1 --2- 172.21.15.1:0/3944758145 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8420067520 0x7f8420153090 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.030+0000 7f84256c9640 1 --2- 172.21.15.1:0/3944758145 >> v2:172.21.15.98:3300/0 conn(0x7f8420150820 0x7f842006c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.031+0000 7f84256c9640 1 -- 172.21.15.1:0/3944758145 >> 172.21.15.1:0/3944758145 conn(0x7f84200c76b0 msgr2=0x7f84200c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.031+0000 7f84256c9640 1 -- 172.21.15.1:0/3944758145 shutdown_connections 2024-03-06T07:22:55.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.031+0000 7f84256c9640 1 -- 172.21.15.1:0/3944758145 wait complete. 2024-03-06T07:22:55.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.032+0000 7f84256c9640 1 Processor -- start 2024-03-06T07:22:55.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.032+0000 7f84256c9640 1 -- start start 2024-03-06T07:22:55.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.032+0000 7f84256c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8420067520 0x7f8420147b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.032+0000 7f84256c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84200c38f0 0x7f8420156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.032+0000 7f84256c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8420150820 0x7f8420146c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.032+0000 7f84256c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84200a8f70 con 0x7f8420067520 2024-03-06T07:22:55.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.032+0000 7f84256c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84200a8b10 con 0x7f84200c38f0 2024-03-06T07:22:55.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.032+0000 7f84256c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f842006f810 con 0x7f8420150820 2024-03-06T07:22:55.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f841effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8420067520 0x7f8420147b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f841effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8420067520 0x7f8420147b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57572/0 (socket says 172.21.15.1:57572) 2024-03-06T07:22:55.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f841effd640 1 -- 172.21.15.1:0/3913892161 learned_addr learned my addr 172.21.15.1:0/3913892161 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:55.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f841e7fc640 1 --2- 172.21.15.1:0/3913892161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84200c38f0 0x7f8420156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f841f7fe640 1 --2- 172.21.15.1:0/3913892161 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8420150820 0x7f8420146c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f841effd640 1 -- 172.21.15.1:0/3913892161 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8420150820 msgr2=0x7f8420146c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f841effd640 1 --2- 172.21.15.1:0/3913892161 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8420150820 0x7f8420146c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f841effd640 1 -- 172.21.15.1:0/3913892161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84200c38f0 msgr2=0x7f8420156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f841effd640 1 --2- 172.21.15.1:0/3913892161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84200c38f0 0x7f8420156840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f841effd640 1 -- 172.21.15.1:0/3913892161 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8420147860 con 0x7f8420067520 2024-03-06T07:22:55.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f841f7fe640 1 --2- 172.21.15.1:0/3913892161 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8420150820 0x7f8420146c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:55.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f841e7fc640 1 --2- 172.21.15.1:0/3913892161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84200c38f0 0x7f8420156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:55.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f841effd640 1 --2- 172.21.15.1:0/3913892161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8420067520 0x7f8420147b40 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f841000e8a0 tx=0x7f841000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f84077fe640 1 -- 172.21.15.1:0/3913892161 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410017070 con 0x7f8420067520 2024-03-06T07:22:55.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f84077fe640 1 -- 172.21.15.1:0/3913892161 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841000c940 con 0x7f8420067520 2024-03-06T07:22:55.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f84077fe640 1 -- 172.21.15.1:0/3913892161 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841000cd30 con 0x7f8420067520 2024-03-06T07:22:55.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f84256c9640 1 -- 172.21.15.1:0/3913892161 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f842014b280 con 0x7f8420067520 2024-03-06T07:22:55.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.033+0000 7f84256c9640 1 -- 172.21.15.1:0/3913892161 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8420153d50 con 0x7f8420067520 2024-03-06T07:22:55.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.034+0000 7f84256c9640 1 -- 172.21.15.1:0/3913892161 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84200680b0 con 0x7f8420067520 2024-03-06T07:22:55.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.035+0000 7f84077fe640 1 -- 172.21.15.1:0/3913892161 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f841001a070 con 0x7f8420067520 2024-03-06T07:22:55.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.035+0000 7f84077fe640 1 --2- 172.21.15.1:0/3913892161 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f840c06b500 0x7f840c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.035+0000 7f84077fe640 1 -- 172.21.15.1:0/3913892161 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f841008a270 con 0x7f8420067520 2024-03-06T07:22:55.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.035+0000 7f841e7fc640 1 --2- 172.21.15.1:0/3913892161 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f840c06b500 0x7f840c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.036+0000 7f841e7fc640 1 --2- 172.21.15.1:0/3913892161 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f840c06b500 0x7f840c06d9c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8414036000 tx=0x7f841403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.039+0000 7f84077fe640 1 -- 172.21.15.1:0/3913892161 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84100592b0 con 0x7f8420067520 2024-03-06T07:22:55.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.221+0000 7f84256c9640 1 -- 172.21.15.1:0/3913892161 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8420148080 con 0x7f8420067520 2024-03-06T07:22:55.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.598+0000 7f84077fe640 1 -- 172.21.15.1:0/3913892161 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8410058bb0 con 0x7f8420067520 2024-03-06T07:22:55.600 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-06T07:22:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.602+0000 7f84256c9640 1 -- 172.21.15.1:0/3913892161 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f840c06b500 msgr2=0x7f840c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.602+0000 7f84256c9640 1 --2- 172.21.15.1:0/3913892161 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f840c06b500 0x7f840c06d9c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8414036000 tx=0x7f841403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.602+0000 7f84256c9640 1 -- 172.21.15.1:0/3913892161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8420067520 msgr2=0x7f8420147b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.602+0000 7f84256c9640 1 --2- 172.21.15.1:0/3913892161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8420067520 0x7f8420147b40 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f841000e8a0 tx=0x7f841000ed60 comp rx=0 tx=0).stop 2024-03-06T07:22:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.603+0000 7f84256c9640 1 -- 172.21.15.1:0/3913892161 shutdown_connections 2024-03-06T07:22:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.603+0000 7f84256c9640 1 --2- 172.21.15.1:0/3913892161 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f840c06b500 0x7f840c06d9c0 secure :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8414036000 tx=0x7f841403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.603+0000 7f84256c9640 1 --2- 172.21.15.1:0/3913892161 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8420150820 0x7f8420146c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.603+0000 7f84256c9640 1 --2- 172.21.15.1:0/3913892161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84200c38f0 0x7f8420156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.603+0000 7f84256c9640 1 --2- 172.21.15.1:0/3913892161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8420067520 0x7f8420147b40 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.603+0000 7f84256c9640 1 -- 172.21.15.1:0/3913892161 >> 172.21.15.1:0/3913892161 conn(0x7f84200c76b0 msgr2=0x7f8420148e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.603+0000 7f84256c9640 1 -- 172.21.15.1:0/3913892161 shutdown_connections 2024-03-06T07:22:55.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.603+0000 7f84256c9640 1 -- 172.21.15.1:0/3913892161 wait complete. 2024-03-06T07:22:55.618 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:22:55.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.727+0000 7f7944bc7640 1 Processor -- start 2024-03-06T07:22:55.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.727+0000 7f7944bc7640 1 -- start start 2024-03-06T07:22:55.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.728+0000 7f7944bc7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7940152730 0x7f79400c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.728+0000 7f7944bc7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79401588e0 con 0x7f79400c1aa0 2024-03-06T07:22:55.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.728+0000 7f7944bc7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7940158bc0 con 0x7f79400c6fd0 2024-03-06T07:22:55.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.728+0000 7f7944bc7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7940158ea0 con 0x7f7940152730 2024-03-06T07:22:55.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.728+0000 7f793ed9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f79400c1aa0 0x7f79401552e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53570/0 (socket says 172.21.15.1:53570) 2024-03-06T07:22:55.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.728+0000 7f793ed9d640 1 -- 172.21.15.1:0/581210753 learned_addr learned my addr 172.21.15.1:0/581210753 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:55.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.728+0000 7f793e59c640 1 --2- 172.21.15.1:0/581210753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7940152730 0x7f79400c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.728+0000 7f793d59a640 1 -- 172.21.15.1:0/581210753 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721881102 0 0) 0x7f79401588e0 con 0x7f79400c1aa0 2024-03-06T07:22:55.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.728+0000 7f793d59a640 1 -- 172.21.15.1:0/581210753 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f792c003520 con 0x7f79400c1aa0 2024-03-06T07:22:55.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.728+0000 7f793d59a640 1 -- 172.21.15.1:0/581210753 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355893344 0 0) 0x7f7940158bc0 con 0x7f79400c6fd0 2024-03-06T07:22:55.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.728+0000 7f793d59a640 1 -- 172.21.15.1:0/581210753 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79401588e0 con 0x7f79400c6fd0 2024-03-06T07:22:55.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.729+0000 7f793e59c640 1 -- 172.21.15.1:0/581210753 >> v1:172.21.15.98:6789/0 conn(0x7f79400c6fd0 legacy=0x7f79400c1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.729+0000 7f793e59c640 1 -- 172.21.15.1:0/581210753 >> v1:172.21.15.1:6789/0 conn(0x7f79400c1aa0 legacy=0x7f79401552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.729+0000 7f793e59c640 1 -- 172.21.15.1:0/581210753 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7940159180 con 0x7f7940152730 2024-03-06T07:22:55.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.729+0000 7f793e59c640 1 --2- 172.21.15.1:0/581210753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7940152730 0x7f79400c6a90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f79380097d0 tx=0x7f793802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed6cbe6546e07dde server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.730+0000 7f793d59a640 1 -- 172.21.15.1:0/581210753 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938031770 con 0x7f7940152730 2024-03-06T07:22:55.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.730+0000 7f793d59a640 1 -- 172.21.15.1:0/581210753 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7938037070 con 0x7f7940152730 2024-03-06T07:22:55.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.730+0000 7f793d59a640 1 -- 172.21.15.1:0/581210753 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938031b10 con 0x7f7940152730 2024-03-06T07:22:55.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.730+0000 7f7944bc7640 1 -- 172.21.15.1:0/581210753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7940152730 msgr2=0x7f79400c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.730+0000 7f7944bc7640 1 --2- 172.21.15.1:0/581210753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7940152730 0x7f79400c6a90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f79380097d0 tx=0x7f793802b810 comp rx=0 tx=0).stop 2024-03-06T07:22:55.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.730+0000 7f7944bc7640 1 -- 172.21.15.1:0/581210753 shutdown_connections 2024-03-06T07:22:55.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.730+0000 7f7944bc7640 1 --2- 172.21.15.1:0/581210753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7940152730 0x7f79400c6a90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.730+0000 7f7944bc7640 1 -- 172.21.15.1:0/581210753 >> 172.21.15.1:0/581210753 conn(0x7f7940066d00 msgr2=0x7f7940067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.730+0000 7f7944bc7640 1 -- 172.21.15.1:0/581210753 shutdown_connections 2024-03-06T07:22:55.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.730+0000 7f7944bc7640 1 -- 172.21.15.1:0/581210753 wait complete. 2024-03-06T07:22:55.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.731+0000 7f7944bc7640 1 Processor -- start 2024-03-06T07:22:55.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.731+0000 7f7944bc7640 1 -- start start 2024-03-06T07:22:55.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.732+0000 7f7944bc7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79400c1aa0 0x7f79401ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.732+0000 7f7944bc7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79400c6fd0 0x7f79401539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.732+0000 7f7944bc7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7940152730 0x7f7940153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.732+0000 7f7944bc7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79401559f0 con 0x7f79400c6fd0 2024-03-06T07:22:55.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.732+0000 7f7944bc7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7940152b10 con 0x7f7940152730 2024-03-06T07:22:55.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.732+0000 7f7944bc7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7940155b70 con 0x7f79400c1aa0 2024-03-06T07:22:55.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.732+0000 7f793dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79400c6fd0 0x7f79401539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.732+0000 7f793dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79400c6fd0 0x7f79401539f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57576/0 (socket says 172.21.15.1:57576) 2024-03-06T07:22:55.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.732+0000 7f793dd9b640 1 -- 172.21.15.1:0/3710271187 learned_addr learned my addr 172.21.15.1:0/3710271187 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:55.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.732+0000 7f793dd9b640 1 -- 172.21.15.1:0/3710271187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79400c1aa0 msgr2=0x7f79401ee5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.732+0000 7f793dd9b640 1 --2- 172.21.15.1:0/3710271187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79400c1aa0 0x7f79401ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.732+0000 7f793dd9b640 1 -- 172.21.15.1:0/3710271187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7940152730 msgr2=0x7f7940153f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.732+0000 7f793dd9b640 1 --2- 172.21.15.1:0/3710271187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7940152730 0x7f7940153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.732+0000 7f793dd9b640 1 -- 172.21.15.1:0/3710271187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79400ca790 con 0x7f79400c6fd0 2024-03-06T07:22:55.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.733+0000 7f793dd9b640 1 --2- 172.21.15.1:0/3710271187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79400c6fd0 0x7f79401539f0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f792800c500 tx=0x7f792800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.733+0000 7f7936ffd640 1 -- 172.21.15.1:0/3710271187 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7928017070 con 0x7f79400c6fd0 2024-03-06T07:22:55.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.733+0000 7f7936ffd640 1 -- 172.21.15.1:0/3710271187 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7928005310 con 0x7f79400c6fd0 2024-03-06T07:22:55.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.733+0000 7f7944bc7640 1 -- 172.21.15.1:0/3710271187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79401ef030 con 0x7f79400c6fd0 2024-03-06T07:22:55.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.733+0000 7f7936ffd640 1 -- 172.21.15.1:0/3710271187 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7928003830 con 0x7f79400c6fd0 2024-03-06T07:22:55.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.733+0000 7f7944bc7640 1 -- 172.21.15.1:0/3710271187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794014a3a0 con 0x7f79400c6fd0 2024-03-06T07:22:55.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.734+0000 7f7944bc7640 1 -- 172.21.15.1:0/3710271187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7940067fd0 con 0x7f79400c6fd0 2024-03-06T07:22:55.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.734+0000 7f7936ffd640 1 -- 172.21.15.1:0/3710271187 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f792801a070 con 0x7f79400c6fd0 2024-03-06T07:22:55.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.735+0000 7f7936ffd640 1 --2- 172.21.15.1:0/3710271187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f792006b4d0 0x7f792006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.735+0000 7f7936ffd640 1 -- 172.21.15.1:0/3710271187 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f79280894b0 con 0x7f79400c6fd0 2024-03-06T07:22:55.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.735+0000 7f793e59c640 1 --2- 172.21.15.1:0/3710271187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f792006b4d0 0x7f792006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.735+0000 7f793e59c640 1 --2- 172.21.15.1:0/3710271187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f792006b4d0 0x7f792006d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7938036000 tx=0x7f7938030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.739+0000 7f7936ffd640 1 -- 172.21.15.1:0/3710271187 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7928057ca0 con 0x7f79400c6fd0 2024-03-06T07:22:55.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:55.920+0000 7f7944bc7640 1 -- 172.21.15.1:0/3710271187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f794004f380 con 0x7f79400c6fd0 2024-03-06T07:22:56.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.600+0000 7f7936ffd640 1 -- 172.21.15.1:0/3710271187 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v72) v1 ==== 170+0+0 (secure 0 0 0) 0x7f792805bd40 con 0x7f79400c6fd0 2024-03-06T07:22:56.602 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-06T07:22:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.603+0000 7f7944bc7640 1 -- 172.21.15.1:0/3710271187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f792006b4d0 msgr2=0x7f792006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.603+0000 7f7944bc7640 1 --2- 172.21.15.1:0/3710271187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f792006b4d0 0x7f792006d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7938036000 tx=0x7f7938030000 comp rx=0 tx=0).stop 2024-03-06T07:22:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.603+0000 7f7944bc7640 1 -- 172.21.15.1:0/3710271187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79400c6fd0 msgr2=0x7f79401539f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.603+0000 7f7944bc7640 1 --2- 172.21.15.1:0/3710271187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79400c6fd0 0x7f79401539f0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f792800c500 tx=0x7f792800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.604+0000 7f7944bc7640 1 -- 172.21.15.1:0/3710271187 shutdown_connections 2024-03-06T07:22:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.604+0000 7f7944bc7640 1 --2- 172.21.15.1:0/3710271187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f792006b4d0 0x7f792006d990 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.604+0000 7f7944bc7640 1 --2- 172.21.15.1:0/3710271187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7940152730 0x7f7940153f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.604+0000 7f7944bc7640 1 --2- 172.21.15.1:0/3710271187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79400c6fd0 0x7f79401539f0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.604+0000 7f7944bc7640 1 --2- 172.21.15.1:0/3710271187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79400c1aa0 0x7f79401ee5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.604+0000 7f7944bc7640 1 -- 172.21.15.1:0/3710271187 >> 172.21.15.1:0/3710271187 conn(0x7f7940066d00 msgr2=0x7f79400c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.606+0000 7f7944bc7640 1 -- 172.21.15.1:0/3710271187 shutdown_connections 2024-03-06T07:22:56.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.608+0000 7f7944bc7640 1 -- 172.21.15.1:0/3710271187 wait complete. 2024-03-06T07:22:56.632 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:22:56.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.735+0000 7fe757033640 1 Processor -- start 2024-03-06T07:22:56.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.736+0000 7fe757033640 1 -- start start 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.736+0000 7fe757033640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7500af6f0 0x7fe7500c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.736+0000 7fe757033640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe7500c7930 0x7fe75015fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.736+0000 7fe757033640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7500c7180 con 0x7fe750152820 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.736+0000 7fe757033640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe750155460 con 0x7fe7500c7930 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.736+0000 7fe757033640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7500c6bc0 con 0x7fe7500af6f0 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.736+0000 7fe7555d0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe7500c7930 0x7fe75015fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.736+0000 7fe7555d0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe7500c7930 0x7fe75015fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57582/0 (socket says 172.21.15.1:57582) 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.736+0000 7fe7555d0640 1 -- 172.21.15.1:0/1692666578 learned_addr learned my addr 172.21.15.1:0/1692666578 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.737+0000 7fe74ffff640 1 --2- 172.21.15.1:0/1692666578 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7500af6f0 0x7fe7500c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.737+0000 7fe7555d0640 1 -- 172.21.15.1:0/1692666578 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7500af6f0 msgr2=0x7fe7500c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.737+0000 7fe7555d0640 1 --2- 172.21.15.1:0/1692666578 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7500af6f0 0x7fe7500c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.737+0000 7fe7555d0640 1 -- 172.21.15.1:0/1692666578 >> v1:172.21.15.98:6789/0 conn(0x7fe750152820 legacy=0x7fe7500c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.737+0000 7fe7555d0640 1 -- 172.21.15.1:0/1692666578 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7500c6ea0 con 0x7fe7500c7930 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.737+0000 7fe74ffff640 1 --2- 172.21.15.1:0/1692666578 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7500af6f0 0x7fe7500c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.737+0000 7fe7555d0640 1 --2- 172.21.15.1:0/1692666578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7500c7930 0x7fe75015fb60 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fe748009e50 tx=0x7fe74802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=205b284afff8ab2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.738+0000 7fe74f7fe640 1 -- 172.21.15.1:0/1692666578 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74802ca70 con 0x7fe7500c7930 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.738+0000 7fe74f7fe640 1 -- 172.21.15.1:0/1692666578 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74802cc10 con 0x7fe7500c7930 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.738+0000 7fe74f7fe640 1 -- 172.21.15.1:0/1692666578 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7480343f0 con 0x7fe7500c7930 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.738+0000 7fe757033640 1 -- 172.21.15.1:0/1692666578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7500c7930 msgr2=0x7fe75015fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.738+0000 7fe757033640 1 --2- 172.21.15.1:0/1692666578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7500c7930 0x7fe75015fb60 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fe748009e50 tx=0x7fe74802bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.738+0000 7fe757033640 1 -- 172.21.15.1:0/1692666578 shutdown_connections 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.738+0000 7fe757033640 1 --2- 172.21.15.1:0/1692666578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7500c7930 0x7fe75015fb60 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.738+0000 7fe757033640 1 --2- 172.21.15.1:0/1692666578 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7500af6f0 0x7fe7500c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.738+0000 7fe757033640 1 -- 172.21.15.1:0/1692666578 >> 172.21.15.1:0/1692666578 conn(0x7fe750072720 msgr2=0x7fe750072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.738+0000 7fe757033640 1 -- 172.21.15.1:0/1692666578 shutdown_connections 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.738+0000 7fe757033640 1 -- 172.21.15.1:0/1692666578 wait complete. 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.739+0000 7fe757033640 1 Processor -- start 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.739+0000 7fe757033640 1 -- start start 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.740+0000 7fe757033640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7500af6f0 0x7fe7501eec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.740+0000 7fe757033640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7500c7930 0x7fe7500c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.740+0000 7fe757033640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe750152820 0x7fe7501eb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.740+0000 7fe757033640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe750152c00 con 0x7fe7500af6f0 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.740+0000 7fe757033640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7500c0840 con 0x7fe7500c7930 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.740+0000 7fe757033640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe750152d80 con 0x7fe750152820 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.740+0000 7fe754dcf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7500af6f0 0x7fe7501eec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.740+0000 7fe754dcf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7500af6f0 0x7fe7501eec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57586/0 (socket says 172.21.15.1:57586) 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.740+0000 7fe754dcf640 1 -- 172.21.15.1:0/3660116392 learned_addr learned my addr 172.21.15.1:0/3660116392 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.740+0000 7fe7555d0640 1 --2- 172.21.15.1:0/3660116392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe750152820 0x7fe7501eb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.740+0000 7fe74ffff640 1 --2- 172.21.15.1:0/3660116392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7500c7930 0x7fe7500c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.741+0000 7fe754dcf640 1 -- 172.21.15.1:0/3660116392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe750152820 msgr2=0x7fe7501eb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.741+0000 7fe754dcf640 1 --2- 172.21.15.1:0/3660116392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe750152820 0x7fe7501eb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.741+0000 7fe754dcf640 1 -- 172.21.15.1:0/3660116392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7500c7930 msgr2=0x7fe7500c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.741+0000 7fe754dcf640 1 --2- 172.21.15.1:0/3660116392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7500c7930 0x7fe7500c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.741+0000 7fe754dcf640 1 -- 172.21.15.1:0/3660116392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7500c3f90 con 0x7fe7500af6f0 2024-03-06T07:22:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.741+0000 7fe754dcf640 1 --2- 172.21.15.1:0/3660116392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7500af6f0 0x7fe7501eec90 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe74400b3f0 tx=0x7fe74400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.741+0000 7fe74d7fa640 1 -- 172.21.15.1:0/3660116392 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe744015020 con 0x7fe7500af6f0 2024-03-06T07:22:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.741+0000 7fe74d7fa640 1 -- 172.21.15.1:0/3660116392 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe744011830 con 0x7fe7500af6f0 2024-03-06T07:22:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.741+0000 7fe74d7fa640 1 -- 172.21.15.1:0/3660116392 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe744011c20 con 0x7fe7500af6f0 2024-03-06T07:22:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.741+0000 7fe757033640 1 -- 172.21.15.1:0/3660116392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7500c4270 con 0x7fe7500af6f0 2024-03-06T07:22:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.741+0000 7fe757033640 1 -- 172.21.15.1:0/3660116392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7501eaf70 con 0x7fe7500af6f0 2024-03-06T07:22:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.742+0000 7fe757033640 1 -- 172.21.15.1:0/3660116392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7501ead60 con 0x7fe7500af6f0 2024-03-06T07:22:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.742+0000 7fe74d7fa640 1 -- 172.21.15.1:0/3660116392 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe744018020 con 0x7fe7500af6f0 2024-03-06T07:22:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.743+0000 7fe74d7fa640 1 --2- 172.21.15.1:0/3660116392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe73406b4d0 0x7fe73406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.743+0000 7fe74d7fa640 1 -- 172.21.15.1:0/3660116392 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fe74408a010 con 0x7fe7500af6f0 2024-03-06T07:22:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.743+0000 7fe74ffff640 1 --2- 172.21.15.1:0/3660116392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe73406b4d0 0x7fe73406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.744+0000 7fe74ffff640 1 --2- 172.21.15.1:0/3660116392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe73406b4d0 0x7fe73406d990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe750154780 tx=0x7fe73c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.747+0000 7fe74d7fa640 1 -- 172.21.15.1:0/3660116392 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe74401f030 con 0x7fe7500af6f0 2024-03-06T07:22:56.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:56.967+0000 7fe757033640 1 -- 172.21.15.1:0/3660116392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe7500686e0 con 0x7fe7500af6f0 2024-03-06T07:22:57.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.608+0000 7fe74d7fa640 1 -- 172.21.15.1:0/3660116392 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v52) v1 ==== 174+0+0 (secure 0 0 0) 0x7fe7440172c0 con 0x7fe7500af6f0 2024-03-06T07:22:57.611 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 11 and data pool 12 2024-03-06T07:22:57.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.614+0000 7fe757033640 1 -- 172.21.15.1:0/3660116392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe73406b4d0 msgr2=0x7fe73406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.614+0000 7fe757033640 1 --2- 172.21.15.1:0/3660116392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe73406b4d0 0x7fe73406d990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe750154780 tx=0x7fe73c008040 comp rx=0 tx=0).stop 2024-03-06T07:22:57.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.614+0000 7fe757033640 1 -- 172.21.15.1:0/3660116392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7500af6f0 msgr2=0x7fe7501eec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.614+0000 7fe757033640 1 --2- 172.21.15.1:0/3660116392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7500af6f0 0x7fe7501eec90 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe74400b3f0 tx=0x7fe74400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.615+0000 7fe757033640 1 -- 172.21.15.1:0/3660116392 shutdown_connections 2024-03-06T07:22:57.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.615+0000 7fe757033640 1 --2- 172.21.15.1:0/3660116392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe73406b4d0 0x7fe73406d990 secure :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe750154780 tx=0x7fe73c008040 comp rx=0 tx=0).stop 2024-03-06T07:22:57.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.615+0000 7fe757033640 1 --2- 172.21.15.1:0/3660116392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe750152820 0x7fe7501eb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.615+0000 7fe757033640 1 --2- 172.21.15.1:0/3660116392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7500c7930 0x7fe7500c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.615+0000 7fe757033640 1 --2- 172.21.15.1:0/3660116392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7500af6f0 0x7fe7501eec90 secure :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe74400b3f0 tx=0x7fe74400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.615+0000 7fe757033640 1 -- 172.21.15.1:0/3660116392 >> 172.21.15.1:0/3660116392 conn(0x7fe750072720 msgr2=0x7fe75014fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.620+0000 7fe757033640 1 -- 172.21.15.1:0/3660116392 shutdown_connections 2024-03-06T07:22:57.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.620+0000 7fe757033640 1 -- 172.21.15.1:0/3660116392 wait complete. 2024-03-06T07:22:57.654 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:57.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.756+0000 7fb0ee288640 1 Processor -- start 2024-03-06T07:22:57.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.756+0000 7fb0ee288640 1 -- start start 2024-03-06T07:22:57.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.757+0000 7fb0ee288640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0e81543a0 0x7fb0e815e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.757+0000 7fb0ee288640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e8150900 con 0x7fb0e8150520 2024-03-06T07:22:57.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.757+0000 7fb0ee288640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e8071790 con 0x7fb0e80c32f0 2024-03-06T07:22:57.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.757+0000 7fb0ee288640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e8153ed0 con 0x7fb0e81543a0 2024-03-06T07:22:57.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.757+0000 7fb0e7fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb0e8150520 0x7fb0e81537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53572/0 (socket says 172.21.15.1:53572) 2024-03-06T07:22:57.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.757+0000 7fb0e7fff640 1 -- 172.21.15.1:0/2908384411 learned_addr learned my addr 172.21.15.1:0/2908384411 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.757+0000 7fb0ec825640 1 --2- 172.21.15.1:0/2908384411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0e81543a0 0x7fb0e815e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e6ffd640 1 -- 172.21.15.1:0/2908384411 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220378856 0 0) 0x7fb0e8150900 con 0x7fb0e8150520 2024-03-06T07:22:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e6ffd640 1 -- 172.21.15.1:0/2908384411 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d0003520 con 0x7fb0e8150520 2024-03-06T07:22:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e6ffd640 1 -- 172.21.15.1:0/2908384411 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126506771 0 0) 0x7fb0e8071790 con 0x7fb0e80c32f0 2024-03-06T07:22:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e6ffd640 1 -- 172.21.15.1:0/2908384411 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e8150900 con 0x7fb0e80c32f0 2024-03-06T07:22:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e6ffd640 1 -- 172.21.15.1:0/2908384411 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1414699513 0 0) 0x7fb0d0003520 con 0x7fb0e8150520 2024-03-06T07:22:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e6ffd640 1 -- 172.21.15.1:0/2908384411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0e81543a0 msgr2=0x7fb0e815e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e6ffd640 1 --2- 172.21.15.1:0/2908384411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0e81543a0 0x7fb0e815e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e6ffd640 1 -- 172.21.15.1:0/2908384411 >> v1:172.21.15.98:6789/0 conn(0x7fb0e80c32f0 legacy=0x7fb0e8070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e6ffd640 1 -- 172.21.15.1:0/2908384411 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e8157050 con 0x7fb0e8150520 2024-03-06T07:22:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e6ffd640 1 -- 172.21.15.1:0/2908384411 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fb0d8001670 con 0x7fb0e8150520 2024-03-06T07:22:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e6ffd640 1 -- 172.21.15.1:0/2908384411 >> v1:172.21.15.1:6789/0 conn(0x7fb0e8150520 legacy=0x7fb0e81537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e6ffd640 1 --2- 172.21.15.1:0/2908384411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0e81543a0 0x7fb0d0003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e6ffd640 1 --2- 172.21.15.1:0/2908384411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d00040d0 0x7fb0d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e6ffd640 1 --2- 172.21.15.1:0/2908384411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d0004d40 0x7fb0d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e6ffd640 1 -- 172.21.15.1:0/2908384411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e8153ed0 con 0x7fb0d00040d0 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e6ffd640 1 -- 172.21.15.1:0/2908384411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d000baa0 con 0x7fb0d0004d40 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e6ffd640 1 -- 172.21.15.1:0/2908384411 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d000bca0 con 0x7fb0e81543a0 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e77fe640 1 --2- 172.21.15.1:0/2908384411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d00040d0 0x7fb0d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e7fff640 1 --2- 172.21.15.1:0/2908384411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0e81543a0 0x7fb0d0003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0ec825640 1 --2- 172.21.15.1:0/2908384411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d0004d40 0x7fb0d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e77fe640 1 -- 172.21.15.1:0/2908384411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0e81543a0 msgr2=0x7fb0d0003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e77fe640 1 --2- 172.21.15.1:0/2908384411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0e81543a0 0x7fb0d0003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e77fe640 1 -- 172.21.15.1:0/2908384411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d0004d40 msgr2=0x7fb0d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e77fe640 1 --2- 172.21.15.1:0/2908384411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d0004d40 0x7fb0d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.758+0000 7fb0e77fe640 1 -- 172.21.15.1:0/2908384411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d0003520 con 0x7fb0d00040d0 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.759+0000 7fb0e77fe640 1 --2- 172.21.15.1:0/2908384411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d00040d0 0x7fb0d0004800 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc0018d0 tx=0x7fb0d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=497c447f54fa1baf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.759+0000 7fb0e6ffd640 1 -- 172.21.15.1:0/2908384411 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc037070 con 0x7fb0d00040d0 2024-03-06T07:22:57.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.759+0000 7fb0e6ffd640 1 -- 172.21.15.1:0/2908384411 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0dc0052d0 con 0x7fb0d00040d0 2024-03-06T07:22:57.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.759+0000 7fb0e6ffd640 1 -- 172.21.15.1:0/2908384411 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc0014f0 con 0x7fb0d00040d0 2024-03-06T07:22:57.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.760+0000 7fb0ee288640 1 -- 172.21.15.1:0/2908384411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d00040d0 msgr2=0x7fb0d0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.760+0000 7fb0ee288640 1 --2- 172.21.15.1:0/2908384411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d00040d0 0x7fb0d0004800 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc0018d0 tx=0x7fb0d0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:57.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.760+0000 7fb0ec825640 1 -- 172.21.15.1:0/2908384411 reap_dead start 2024-03-06T07:22:57.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.760+0000 7fb0ee288640 1 -- 172.21.15.1:0/2908384411 shutdown_connections 2024-03-06T07:22:57.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.760+0000 7fb0ee288640 1 -- 172.21.15.1:0/2908384411 >> 172.21.15.1:0/2908384411 conn(0x7fb0e80c88f0 msgr2=0x7fb0e80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.760+0000 7fb0ee288640 1 -- 172.21.15.1:0/2908384411 shutdown_connections 2024-03-06T07:22:57.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.760+0000 7fb0ee288640 1 -- 172.21.15.1:0/2908384411 wait complete. 2024-03-06T07:22:57.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.761+0000 7fb0ee288640 1 Processor -- start 2024-03-06T07:22:57.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.761+0000 7fb0ee288640 1 -- start start 2024-03-06T07:22:57.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0ee288640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d00040d0 0x7fb0e81555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0ee288640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e81562a0 0x7fb0e81e9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0ee288640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0e80ac9b0 0x7fb0e81e9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0ee288640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e81532a0 con 0x7fb0e80ac9b0 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0ee288640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e81547e0 con 0x7fb0e81562a0 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0ee288640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e8157330 con 0x7fb0d00040d0 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0ec825640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0e80ac9b0 0x7fb0e81e9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0ec825640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0e80ac9b0 0x7fb0e81e9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57596/0 (socket says 172.21.15.1:57596) 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0ec825640 1 -- 172.21.15.1:0/1447083947 learned_addr learned my addr 172.21.15.1:0/1447083947 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0e7fff640 1 --2- 172.21.15.1:0/1447083947 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d00040d0 0x7fb0e81555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0e77fe640 1 --2- 172.21.15.1:0/1447083947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e81562a0 0x7fb0e81e9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0ec825640 1 -- 172.21.15.1:0/1447083947 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d00040d0 msgr2=0x7fb0e81555a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0ec825640 1 --2- 172.21.15.1:0/1447083947 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d00040d0 0x7fb0e81555a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0ec825640 1 -- 172.21.15.1:0/1447083947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e81562a0 msgr2=0x7fb0e81e9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0ec825640 1 --2- 172.21.15.1:0/1447083947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e81562a0 0x7fb0e81e9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0ec825640 1 -- 172.21.15.1:0/1447083947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e81e6450 con 0x7fb0e80ac9b0 2024-03-06T07:22:57.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0e77fe640 1 --2- 172.21.15.1:0/1447083947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e81562a0 0x7fb0e81e9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:57.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.762+0000 7fb0ec825640 1 --2- 172.21.15.1:0/1447083947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0e80ac9b0 0x7fb0e81e9d00 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb0e80c7ae0 tx=0x7fb0e000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.763+0000 7fb0e4ff9640 1 -- 172.21.15.1:0/1447083947 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e000fde0 con 0x7fb0e80ac9b0 2024-03-06T07:22:57.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.763+0000 7fb0e4ff9640 1 -- 172.21.15.1:0/1447083947 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e0015430 con 0x7fb0e80ac9b0 2024-03-06T07:22:57.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.763+0000 7fb0ee288640 1 -- 172.21.15.1:0/1447083947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e8156ad0 con 0x7fb0e80ac9b0 2024-03-06T07:22:57.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.763+0000 7fb0e4ff9640 1 -- 172.21.15.1:0/1447083947 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e0015820 con 0x7fb0e80ac9b0 2024-03-06T07:22:57.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.763+0000 7fb0ee288640 1 -- 172.21.15.1:0/1447083947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e8154ac0 con 0x7fb0e80ac9b0 2024-03-06T07:22:57.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.764+0000 7fb0ee288640 1 -- 172.21.15.1:0/1447083947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e806f4e0 con 0x7fb0e80ac9b0 2024-03-06T07:22:57.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.764+0000 7fb0e4ff9640 1 -- 172.21.15.1:0/1447083947 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0e00159c0 con 0x7fb0e80ac9b0 2024-03-06T07:22:57.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.765+0000 7fb0e4ff9640 1 --2- 172.21.15.1:0/1447083947 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0d406b500 0x7fb0d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.765+0000 7fb0e4ff9640 1 -- 172.21.15.1:0/1447083947 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb0e008b110 con 0x7fb0e80ac9b0 2024-03-06T07:22:57.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.765+0000 7fb0e7fff640 1 --2- 172.21.15.1:0/1447083947 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0d406b500 0x7fb0d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.766+0000 7fb0e7fff640 1 --2- 172.21.15.1:0/1447083947 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0d406b500 0x7fb0d406d9c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8001eb0 tx=0x7fb0d8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.769+0000 7fb0e4ff9640 1 -- 172.21.15.1:0/1447083947 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0e806f4e0 con 0x7fb0e80ac9b0 2024-03-06T07:22:57.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.952+0000 7fb0ee288640 1 -- 172.21.15.1:0/1447083947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0e80686e0 con 0x7fb0e80ac9b0 2024-03-06T07:22:57.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.953+0000 7fb0e4ff9640 1 -- 172.21.15.1:0/1447083947 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+16838 (secure 0 0 0) 0x7fb0e81e6450 con 0x7fb0e80ac9b0 2024-03-06T07:22:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:57.955 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":73,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:56.968816+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:55.240857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:55.939811+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:57.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.955+0000 7fb0ee288640 1 -- 172.21.15.1:0/1447083947 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0d406b500 msgr2=0x7fb0d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.955+0000 7fb0ee288640 1 --2- 172.21.15.1:0/1447083947 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0d406b500 0x7fb0d406d9c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8001eb0 tx=0x7fb0d8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:57.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.955+0000 7fb0ee288640 1 -- 172.21.15.1:0/1447083947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0e80ac9b0 msgr2=0x7fb0e81e9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.955+0000 7fb0ee288640 1 --2- 172.21.15.1:0/1447083947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0e80ac9b0 0x7fb0e81e9d00 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb0e80c7ae0 tx=0x7fb0e000f580 comp rx=0 tx=0).stop 2024-03-06T07:22:57.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.955+0000 7fb0ee288640 1 -- 172.21.15.1:0/1447083947 shutdown_connections 2024-03-06T07:22:57.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.955+0000 7fb0ee288640 1 --2- 172.21.15.1:0/1447083947 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0d406b500 0x7fb0d406d9c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.955+0000 7fb0ee288640 1 --2- 172.21.15.1:0/1447083947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0e80ac9b0 0x7fb0e81e9d00 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.955+0000 7fb0ee288640 1 --2- 172.21.15.1:0/1447083947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e81562a0 0x7fb0e81e9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.955+0000 7fb0ee288640 1 --2- 172.21.15.1:0/1447083947 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d00040d0 0x7fb0e81555a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.955+0000 7fb0ee288640 1 -- 172.21.15.1:0/1447083947 >> 172.21.15.1:0/1447083947 conn(0x7fb0e80c88f0 msgr2=0x7fb0e8070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.955+0000 7fb0ee288640 1 -- 172.21.15.1:0/1447083947 shutdown_connections 2024-03-06T07:22:57.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:57.956+0000 7fb0ee288640 1 -- 172.21.15.1:0/1447083947 wait complete. 2024-03-06T07:22:57.969 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:58.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.068+0000 7f36a37f7640 1 Processor -- start 2024-03-06T07:22:58.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.068+0000 7f36a37f7640 1 -- start start 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.069+0000 7f36a37f7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f369c072f20 0x7f369c1513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.069+0000 7f36a37f7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f369c071f90 0x7f369c072400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.069+0000 7f36a37f7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369c0a6140 con 0x7f369c150fe0 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.069+0000 7f36a37f7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f369c1526b0 con 0x7f369c072f20 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.069+0000 7f36a37f7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f369c151aa0 con 0x7f369c071f90 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.069+0000 7f36a0d92640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f369c072f20 0x7f369c1513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.069+0000 7f36a0d92640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f369c072f20 0x7f369c1513c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57600/0 (socket says 172.21.15.1:57600) 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.069+0000 7f36a1593640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f369c150fe0 0x7f369c06edb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53578/0 (socket says 172.21.15.1:53578) 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.069+0000 7f36a0d92640 1 -- 172.21.15.1:0/2703263552 learned_addr learned my addr 172.21.15.1:0/2703263552 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.069+0000 7f36a1d94640 1 --2- 172.21.15.1:0/2703263552 >> v2:172.21.15.98:3300/0 conn(0x7f369c071f90 0x7f369c072400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.069+0000 7f36a0d92640 1 -- 172.21.15.1:0/2703263552 >> v2:172.21.15.98:3300/0 conn(0x7f369c071f90 msgr2=0x7f369c072400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.069+0000 7f36a0d92640 1 --2- 172.21.15.1:0/2703263552 >> v2:172.21.15.98:3300/0 conn(0x7f369c071f90 0x7f369c072400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.069+0000 7f36a0d92640 1 -- 172.21.15.1:0/2703263552 >> v1:172.21.15.1:6789/0 conn(0x7f369c150fe0 legacy=0x7f369c06edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.069+0000 7f369bfff640 1 -- 172.21.15.1:0/2703263552 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524471626 0 0) 0x7f369c0a6140 con 0x7f369c150fe0 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.069+0000 7f36a0d92640 1 -- 172.21.15.1:0/2703263552 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c151d80 con 0x7f369c072f20 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.070+0000 7f36a0d92640 1 --2- 172.21.15.1:0/2703263552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f369c072f20 0x7f369c1513c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f369c0714d0 tx=0x7f369002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=277330b65939e25f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.070+0000 7f369bfff640 1 -- 172.21.15.1:0/2703263552 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3690035020 con 0x7f369c072f20 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.070+0000 7f369bfff640 1 -- 172.21.15.1:0/2703263552 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3690031870 con 0x7f369c072f20 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.070+0000 7f369bfff640 1 -- 172.21.15.1:0/2703263552 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3690031c40 con 0x7f369c072f20 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.071+0000 7f36a37f7640 1 -- 172.21.15.1:0/2703263552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f369c072f20 msgr2=0x7f369c1513c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.071+0000 7f36a37f7640 1 --2- 172.21.15.1:0/2703263552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f369c072f20 0x7f369c1513c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f369c0714d0 tx=0x7f369002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.071+0000 7f36a37f7640 1 -- 172.21.15.1:0/2703263552 shutdown_connections 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.071+0000 7f36a37f7640 1 --2- 172.21.15.1:0/2703263552 >> v2:172.21.15.98:3300/0 conn(0x7f369c071f90 0x7f369c072400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.071+0000 7f36a37f7640 1 --2- 172.21.15.1:0/2703263552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f369c072f20 0x7f369c1513c0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.071+0000 7f36a37f7640 1 -- 172.21.15.1:0/2703263552 >> 172.21.15.1:0/2703263552 conn(0x7f369c0c6860 msgr2=0x7f369c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.071+0000 7f36a37f7640 1 -- 172.21.15.1:0/2703263552 shutdown_connections 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.071+0000 7f36a37f7640 1 -- 172.21.15.1:0/2703263552 wait complete. 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.072+0000 7f36a37f7640 1 Processor -- start 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.072+0000 7f36a37f7640 1 -- start start 2024-03-06T07:22:58.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a37f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f369c071f90 0x7f369c1ec500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a37f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f369c072f20 0x7f369c1dc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a37f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f369c150fe0 0x7f369c1dc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a37f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c06f4c0 con 0x7f369c150fe0 2024-03-06T07:22:58.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a37f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c152340 con 0x7f369c071f90 2024-03-06T07:22:58.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a37f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f369c06f640 con 0x7f369c072f20 2024-03-06T07:22:58.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a1d94640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f369c150fe0 0x7f369c1dc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a1d94640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f369c150fe0 0x7f369c1dc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57608/0 (socket says 172.21.15.1:57608) 2024-03-06T07:22:58.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a1d94640 1 -- 172.21.15.1:0/1355055187 learned_addr learned my addr 172.21.15.1:0/1355055187 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:58.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a1593640 1 --2- 172.21.15.1:0/1355055187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f369c071f90 0x7f369c1ec500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a0d92640 1 --2- 172.21.15.1:0/1355055187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f369c072f20 0x7f369c1dc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a1d94640 1 -- 172.21.15.1:0/1355055187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f369c072f20 msgr2=0x7f369c1dc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a1d94640 1 --2- 172.21.15.1:0/1355055187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f369c072f20 0x7f369c1dc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a1d94640 1 -- 172.21.15.1:0/1355055187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f369c071f90 msgr2=0x7f369c1ec500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a1d94640 1 --2- 172.21.15.1:0/1355055187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f369c071f90 0x7f369c1ec500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a1d94640 1 -- 172.21.15.1:0/1355055187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c1de580 con 0x7f369c150fe0 2024-03-06T07:22:58.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a1d94640 1 --2- 172.21.15.1:0/1355055187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f369c150fe0 0x7f369c1dc8a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f368c00c550 tx=0x7f368c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f3699ffb640 1 -- 172.21.15.1:0/1355055187 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368c015070 con 0x7f369c150fe0 2024-03-06T07:22:58.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f3699ffb640 1 -- 172.21.15.1:0/1355055187 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368c004220 con 0x7f369c150fe0 2024-03-06T07:22:58.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f36a37f7640 1 -- 172.21.15.1:0/1355055187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f369c1ecf80 con 0x7f369c150fe0 2024-03-06T07:22:58.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.073+0000 7f3699ffb640 1 -- 172.21.15.1:0/1355055187 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368c003610 con 0x7f369c150fe0 2024-03-06T07:22:58.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.074+0000 7f36a37f7640 1 -- 172.21.15.1:0/1355055187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369c1ddc50 con 0x7f369c150fe0 2024-03-06T07:22:58.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.075+0000 7f36a37f7640 1 -- 172.21.15.1:0/1355055187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f369c067fd0 con 0x7f369c150fe0 2024-03-06T07:22:58.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.075+0000 7f3699ffb640 1 -- 172.21.15.1:0/1355055187 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f368c0037b0 con 0x7f369c150fe0 2024-03-06T07:22:58.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.075+0000 7f3699ffb640 1 --2- 172.21.15.1:0/1355055187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f368806b4d0 0x7f368806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.075+0000 7f3699ffb640 1 -- 172.21.15.1:0/1355055187 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f368c089f90 con 0x7f369c150fe0 2024-03-06T07:22:58.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.075+0000 7f36a1593640 1 --2- 172.21.15.1:0/1355055187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f368806b4d0 0x7f368806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.076+0000 7f36a1593640 1 --2- 172.21.15.1:0/1355055187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f368806b4d0 0x7f368806d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f36940012e0 tx=0x7f3694001700 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.079+0000 7f3699ffb640 1 -- 172.21.15.1:0/1355055187 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f368c0586c0 con 0x7f369c150fe0 2024-03-06T07:22:58.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.261+0000 7f36a37f7640 1 -- 172.21.15.1:0/1355055187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f369c0ac9a0 con 0x7f369c150fe0 2024-03-06T07:22:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.262+0000 7f3699ffb640 1 -- 172.21.15.1:0/1355055187 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+16838 (secure 0 0 0) 0x7f368c05c760 con 0x7f369c150fe0 2024-03-06T07:22:58.264 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:22:58.264 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":73,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:56.968816+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:55.240857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:55.939811+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.264+0000 7f36a37f7640 1 -- 172.21.15.1:0/1355055187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f368806b4d0 msgr2=0x7f368806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.264+0000 7f36a37f7640 1 --2- 172.21.15.1:0/1355055187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f368806b4d0 0x7f368806d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f36940012e0 tx=0x7f3694001700 comp rx=0 tx=0).stop 2024-03-06T07:22:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.264+0000 7f36a37f7640 1 -- 172.21.15.1:0/1355055187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f369c150fe0 msgr2=0x7f369c1dc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.264+0000 7f36a37f7640 1 --2- 172.21.15.1:0/1355055187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f369c150fe0 0x7f369c1dc8a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f368c00c550 tx=0x7f368c00ca10 comp rx=0 tx=0).stop 2024-03-06T07:22:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.264+0000 7f36a37f7640 1 -- 172.21.15.1:0/1355055187 shutdown_connections 2024-03-06T07:22:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.264+0000 7f36a37f7640 1 --2- 172.21.15.1:0/1355055187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f368806b4d0 0x7f368806d990 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.264+0000 7f36a37f7640 1 --2- 172.21.15.1:0/1355055187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f369c150fe0 0x7f369c1dc8a0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.264+0000 7f36a37f7640 1 --2- 172.21.15.1:0/1355055187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f369c072f20 0x7f369c1dc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.264+0000 7f36a37f7640 1 --2- 172.21.15.1:0/1355055187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f369c071f90 0x7f369c1ec500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.264+0000 7f36a37f7640 1 -- 172.21.15.1:0/1355055187 >> 172.21.15.1:0/1355055187 conn(0x7f369c0c6860 msgr2=0x7f369c14dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.264+0000 7f36a37f7640 1 -- 172.21.15.1:0/1355055187 shutdown_connections 2024-03-06T07:22:58.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.264+0000 7f36a37f7640 1 -- 172.21.15.1:0/1355055187 wait complete. 2024-03-06T07:22:58.277 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:22:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.379+0000 7f6d90415640 1 Processor -- start 2024-03-06T07:22:58.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.379+0000 7f6d90415640 1 -- start start 2024-03-06T07:22:58.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.380+0000 7f6d90415640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6d880bf9d0 0x7f6d88155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.380+0000 7f6d90415640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d880c2fc0 con 0x7f6d88152650 2024-03-06T07:22:58.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.380+0000 7f6d90415640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d88158da0 con 0x7f6d880c6b10 2024-03-06T07:22:58.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.380+0000 7f6d90415640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d88155800 con 0x7f6d880bf9d0 2024-03-06T07:22:58.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.380+0000 7f6d8e1b1640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6d88152650 0x7f6d88158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53590/0 (socket says 172.21.15.1:53590) 2024-03-06T07:22:58.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.380+0000 7f6d8e1b1640 1 -- 172.21.15.1:0/1849608449 learned_addr learned my addr 172.21.15.1:0/1849608449 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:58.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.380+0000 7f6d8d1af640 1 -- 172.21.15.1:0/1849608449 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988329150 0 0) 0x7f6d880c2fc0 con 0x7f6d88152650 2024-03-06T07:22:58.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.380+0000 7f6d8d1af640 1 -- 172.21.15.1:0/1849608449 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d70003520 con 0x7f6d88152650 2024-03-06T07:22:58.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.380+0000 7f6d8e9b2640 1 --2- 172.21.15.1:0/1849608449 >> v2:172.21.15.98:3300/0 conn(0x7f6d880bf9d0 0x7f6d88155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d1af640 1 -- 172.21.15.1:0/1849608449 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2166902263 0 0) 0x7f6d70003520 con 0x7f6d88152650 2024-03-06T07:22:58.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d1af640 1 -- 172.21.15.1:0/1849608449 >> v2:172.21.15.98:3300/0 conn(0x7f6d880bf9d0 msgr2=0x7f6d88155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d1af640 1 --2- 172.21.15.1:0/1849608449 >> v2:172.21.15.98:3300/0 conn(0x7f6d880bf9d0 0x7f6d88155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d1af640 1 -- 172.21.15.1:0/1849608449 >> v1:172.21.15.98:6789/0 conn(0x7f6d880c6b10 legacy=0x7f6d880c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d1af640 1 -- 172.21.15.1:0/1849608449 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d880c6690 con 0x7f6d88152650 2024-03-06T07:22:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d1af640 1 -- 172.21.15.1:0/1849608449 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f6d80001670 con 0x7f6d88152650 2024-03-06T07:22:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d1af640 1 -- 172.21.15.1:0/1849608449 >> v1:172.21.15.1:6789/0 conn(0x7f6d88152650 legacy=0x7f6d88158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d1af640 1 --2- 172.21.15.1:0/1849608449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d700039b0 0x7f6d70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d1af640 1 --2- 172.21.15.1:0/1849608449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d70004800 0x7f6d70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d1af640 1 --2- 172.21.15.1:0/1849608449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d700052e0 0x7f6d7000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d1af640 1 -- 172.21.15.1:0/1849608449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d88155800 con 0x7f6d70004800 2024-03-06T07:22:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d1af640 1 -- 172.21.15.1:0/1849608449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7000c1b0 con 0x7f6d700039b0 2024-03-06T07:22:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d1af640 1 -- 172.21.15.1:0/1849608449 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7000c3b0 con 0x7f6d700052e0 2024-03-06T07:22:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d9b0640 1 --2- 172.21.15.1:0/1849608449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d70004800 0x7f6d70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8e1b1640 1 --2- 172.21.15.1:0/1849608449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d700039b0 0x7f6d70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8e9b2640 1 --2- 172.21.15.1:0/1849608449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d700052e0 0x7f6d7000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d9b0640 1 -- 172.21.15.1:0/1849608449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d700052e0 msgr2=0x7f6d7000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d9b0640 1 --2- 172.21.15.1:0/1849608449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d700052e0 0x7f6d7000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d9b0640 1 -- 172.21.15.1:0/1849608449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d700039b0 msgr2=0x7f6d70003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d9b0640 1 --2- 172.21.15.1:0/1849608449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d700039b0 0x7f6d70003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8d9b0640 1 -- 172.21.15.1:0/1849608449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d70003520 con 0x7f6d70004800 2024-03-06T07:22:58.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.381+0000 7f6d8e9b2640 1 -- 172.21.15.1:0/1849608449 reap_dead start 2024-03-06T07:22:58.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.382+0000 7f6d8d9b0640 1 --2- 172.21.15.1:0/1849608449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d70004800 0x7f6d70004c80 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c001720 tx=0x7f6d70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba153c98a82b2bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.382+0000 7f6d8d1af640 1 -- 172.21.15.1:0/1849608449 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c001370 con 0x7f6d70004800 2024-03-06T07:22:58.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.382+0000 7f6d8d1af640 1 -- 172.21.15.1:0/1849608449 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7c0052d0 con 0x7f6d70004800 2024-03-06T07:22:58.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.382+0000 7f6d8d1af640 1 -- 172.21.15.1:0/1849608449 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c003830 con 0x7f6d70004800 2024-03-06T07:22:58.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.383+0000 7f6d90415640 1 -- 172.21.15.1:0/1849608449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d70004800 msgr2=0x7f6d70004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.383+0000 7f6d90415640 1 --2- 172.21.15.1:0/1849608449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d70004800 0x7f6d70004c80 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c001720 tx=0x7f6d70005830 comp rx=0 tx=0).stop 2024-03-06T07:22:58.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.383+0000 7f6d90415640 1 -- 172.21.15.1:0/1849608449 shutdown_connections 2024-03-06T07:22:58.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.383+0000 7f6d90415640 1 --2- 172.21.15.1:0/1849608449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d70004800 0x7f6d70004c80 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.383+0000 7f6d90415640 1 -- 172.21.15.1:0/1849608449 >> 172.21.15.1:0/1849608449 conn(0x7f6d88066ee0 msgr2=0x7f6d880672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.383+0000 7f6d90415640 1 -- 172.21.15.1:0/1849608449 shutdown_connections 2024-03-06T07:22:58.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.383+0000 7f6d90415640 1 -- 172.21.15.1:0/1849608449 wait complete. 2024-03-06T07:22:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.384+0000 7f6d90415640 1 Processor -- start 2024-03-06T07:22:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.384+0000 7f6d90415640 1 -- start start 2024-03-06T07:22:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.384+0000 7f6d90415640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d70004800 0x7f6d880c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.384+0000 7f6d90415640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d881def80 0x7f6d881eb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.384+0000 7f6d90415640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d881ee410 0x7f6d881eb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.384+0000 7f6d90415640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d881551b0 con 0x7f6d70004800 2024-03-06T07:22:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.384+0000 7f6d90415640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d880c6fa0 con 0x7f6d881def80 2024-03-06T07:22:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.384+0000 7f6d90415640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d88155490 con 0x7f6d881ee410 2024-03-06T07:22:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.385+0000 7f6d8e1b1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d70004800 0x7f6d880c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.385+0000 7f6d8e1b1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d70004800 0x7f6d880c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57612/0 (socket says 172.21.15.1:57612) 2024-03-06T07:22:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.385+0000 7f6d8e1b1640 1 -- 172.21.15.1:0/1042060989 learned_addr learned my addr 172.21.15.1:0/1042060989 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.385+0000 7f6d8d9b0640 1 --2- 172.21.15.1:0/1042060989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d881def80 0x7f6d881eb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.385+0000 7f6d8e9b2640 1 --2- 172.21.15.1:0/1042060989 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d881ee410 0x7f6d881eb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.385+0000 7f6d8e1b1640 1 -- 172.21.15.1:0/1042060989 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d881ee410 msgr2=0x7f6d881eb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.385+0000 7f6d8e1b1640 1 --2- 172.21.15.1:0/1042060989 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d881ee410 0x7f6d881eb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.385+0000 7f6d8e1b1640 1 -- 172.21.15.1:0/1042060989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d881def80 msgr2=0x7f6d881eb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.385+0000 7f6d8e1b1640 1 --2- 172.21.15.1:0/1042060989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d881def80 0x7f6d881eb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.385+0000 7f6d8e1b1640 1 -- 172.21.15.1:0/1042060989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d881ee980 con 0x7f6d70004800 2024-03-06T07:22:58.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.385+0000 7f6d8e9b2640 1 --2- 172.21.15.1:0/1042060989 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d881ee410 0x7f6d881eb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:58.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.385+0000 7f6d8e1b1640 1 --2- 172.21.15.1:0/1042060989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d70004800 0x7f6d880c9a90 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f6d800072c0 tx=0x7f6d80007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.385+0000 7f6d86ffd640 1 -- 172.21.15.1:0/1042060989 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d80001660 con 0x7f6d70004800 2024-03-06T07:22:58.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.385+0000 7f6d86ffd640 1 -- 172.21.15.1:0/1042060989 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d80018070 con 0x7f6d70004800 2024-03-06T07:22:58.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.385+0000 7f6d90415640 1 -- 172.21.15.1:0/1042060989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d881eec60 con 0x7f6d70004800 2024-03-06T07:22:58.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.385+0000 7f6d86ffd640 1 -- 172.21.15.1:0/1042060989 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8000fc00 con 0x7f6d70004800 2024-03-06T07:22:58.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.385+0000 7f6d90415640 1 -- 172.21.15.1:0/1042060989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d881ef220 con 0x7f6d70004800 2024-03-06T07:22:58.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.386+0000 7f6d90415640 1 -- 172.21.15.1:0/1042060989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d880c48b0 con 0x7f6d70004800 2024-03-06T07:22:58.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.387+0000 7f6d86ffd640 1 -- 172.21.15.1:0/1042060989 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d8001d460 con 0x7f6d70004800 2024-03-06T07:22:58.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.387+0000 7f6d86ffd640 1 --2- 172.21.15.1:0/1042060989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6d7406b6a0 0x7f6d7406db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.387+0000 7f6d86ffd640 1 -- 172.21.15.1:0/1042060989 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6d8008a250 con 0x7f6d70004800 2024-03-06T07:22:58.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.387+0000 7f6d8d9b0640 1 --2- 172.21.15.1:0/1042060989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6d7406b6a0 0x7f6d7406db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.388+0000 7f6d8d9b0640 1 --2- 172.21.15.1:0/1042060989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6d7406b6a0 0x7f6d7406db60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c033000 tx=0x7f6d7c009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.391+0000 7f6d86ffd640 1 -- 172.21.15.1:0/1042060989 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d880c48b0 con 0x7f6d70004800 2024-03-06T07:22:58.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.613+0000 7f6d90415640 1 -- 172.21.15.1:0/1042060989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6d88154e40 con 0x7f6d70004800 2024-03-06T07:22:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.626+0000 7f6d86ffd640 1 -- 172.21.15.1:0/1042060989 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v53) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6d80058fd0 con 0x7f6d70004800 2024-03-06T07:22:58.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.629+0000 7f6d90415640 1 -- 172.21.15.1:0/1042060989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6d7406b6a0 msgr2=0x7f6d7406db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.629+0000 7f6d90415640 1 --2- 172.21.15.1:0/1042060989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6d7406b6a0 0x7f6d7406db60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c033000 tx=0x7f6d7c009030 comp rx=0 tx=0).stop 2024-03-06T07:22:58.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.629+0000 7f6d90415640 1 -- 172.21.15.1:0/1042060989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d70004800 msgr2=0x7f6d880c9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.629+0000 7f6d90415640 1 --2- 172.21.15.1:0/1042060989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d70004800 0x7f6d880c9a90 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f6d800072c0 tx=0x7f6d80007780 comp rx=0 tx=0).stop 2024-03-06T07:22:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.629+0000 7f6d90415640 1 -- 172.21.15.1:0/1042060989 shutdown_connections 2024-03-06T07:22:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.629+0000 7f6d90415640 1 --2- 172.21.15.1:0/1042060989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6d7406b6a0 0x7f6d7406db60 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.629+0000 7f6d90415640 1 --2- 172.21.15.1:0/1042060989 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d881ee410 0x7f6d881eb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.629+0000 7f6d90415640 1 --2- 172.21.15.1:0/1042060989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d881def80 0x7f6d881eb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.629+0000 7f6d90415640 1 --2- 172.21.15.1:0/1042060989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d70004800 0x7f6d880c9a90 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.629+0000 7f6d90415640 1 -- 172.21.15.1:0/1042060989 >> 172.21.15.1:0/1042060989 conn(0x7f6d88066ee0 msgr2=0x7f6d880c4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.629+0000 7f6d90415640 1 -- 172.21.15.1:0/1042060989 shutdown_connections 2024-03-06T07:22:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.629+0000 7f6d90415640 1 -- 172.21.15.1:0/1042060989 wait complete. 2024-03-06T07:22:58.645 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:22:58.645 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:22:58.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.747+0000 7fbdc311e640 1 Processor -- start 2024-03-06T07:22:58.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.747+0000 7fbdc311e640 1 -- start start 2024-03-06T07:22:58.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.748+0000 7fbdc311e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbdbc152a40 0x7fbdbc159450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.748+0000 7fbdc311e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbdbc0c6610 0x7fbdbc1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.748+0000 7fbdc311e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbdbc1558e0 0x7fbdbc0c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.748+0000 7fbdc311e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdbc0c20c0 con 0x7fbdbc0c6610 2024-03-06T07:22:58.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.748+0000 7fbdc311e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdbc152e20 con 0x7fbdbc152a40 2024-03-06T07:22:58.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.748+0000 7fbdc311e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc1589b0 con 0x7fbdbc1558e0 2024-03-06T07:22:58.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.748+0000 7fbdbbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbdbc0c6610 0x7fbdbc1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.748+0000 7fbdbbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbdbc0c6610 0x7fbdbc1553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57620/0 (socket says 172.21.15.1:57620) 2024-03-06T07:22:58.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.748+0000 7fbdbbfff640 1 -- 172.21.15.1:0/229669381 learned_addr learned my addr 172.21.15.1:0/229669381 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:58.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.748+0000 7fbdc0eba640 1 --2- 172.21.15.1:0/229669381 >> v2:172.21.15.98:3300/0 conn(0x7fbdbc152a40 0x7fbdbc159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.748+0000 7fbdc16bb640 1 --2- 172.21.15.1:0/229669381 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbdbc1558e0 0x7fbdbc0c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.748+0000 7fbdbbfff640 1 -- 172.21.15.1:0/229669381 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbdbc1558e0 msgr2=0x7fbdbc0c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.748+0000 7fbdbbfff640 1 --2- 172.21.15.1:0/229669381 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbdbc1558e0 0x7fbdbc0c58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.748+0000 7fbdbbfff640 1 -- 172.21.15.1:0/229669381 >> v2:172.21.15.98:3300/0 conn(0x7fbdbc152a40 msgr2=0x7fbdbc159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.748+0000 7fbdbbfff640 1 --2- 172.21.15.1:0/229669381 >> v2:172.21.15.98:3300/0 conn(0x7fbdbc152a40 0x7fbdbc159450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.748+0000 7fbdbbfff640 1 -- 172.21.15.1:0/229669381 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdbc0c6e60 con 0x7fbdbc0c6610 2024-03-06T07:22:58.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.748+0000 7fbdc0eba640 1 --2- 172.21.15.1:0/229669381 >> v2:172.21.15.98:3300/0 conn(0x7fbdbc152a40 0x7fbdbc159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.749+0000 7fbdbbfff640 1 --2- 172.21.15.1:0/229669381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdbc0c6610 0x7fbdbc1553a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc073660 tx=0x7fbda802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84442aa19d874a66 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.749+0000 7fbdbb7fe640 1 -- 172.21.15.1:0/229669381 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda8038070 con 0x7fbdbc0c6610 2024-03-06T07:22:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.749+0000 7fbdbb7fe640 1 -- 172.21.15.1:0/229669381 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda802f050 con 0x7fbdbc0c6610 2024-03-06T07:22:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.749+0000 7fbdbb7fe640 1 -- 172.21.15.1:0/229669381 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda8031970 con 0x7fbdbc0c6610 2024-03-06T07:22:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.750+0000 7fbdc311e640 1 -- 172.21.15.1:0/229669381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdbc0c6610 msgr2=0x7fbdbc1553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.750+0000 7fbdc311e640 1 --2- 172.21.15.1:0/229669381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdbc0c6610 0x7fbdbc1553a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc073660 tx=0x7fbda802b7f0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.750+0000 7fbdc311e640 1 -- 172.21.15.1:0/229669381 shutdown_connections 2024-03-06T07:22:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.750+0000 7fbdc311e640 1 --2- 172.21.15.1:0/229669381 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbdbc1558e0 0x7fbdbc0c58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.750+0000 7fbdc311e640 1 --2- 172.21.15.1:0/229669381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdbc0c6610 0x7fbdbc1553a0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.750+0000 7fbdc311e640 1 --2- 172.21.15.1:0/229669381 >> v2:172.21.15.98:3300/0 conn(0x7fbdbc152a40 0x7fbdbc159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.750+0000 7fbdc311e640 1 -- 172.21.15.1:0/229669381 >> 172.21.15.1:0/229669381 conn(0x7fbdbc0c79c0 msgr2=0x7fbdbc0c7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.750+0000 7fbdc311e640 1 -- 172.21.15.1:0/229669381 shutdown_connections 2024-03-06T07:22:58.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.750+0000 7fbdc311e640 1 -- 172.21.15.1:0/229669381 wait complete. 2024-03-06T07:22:58.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.751+0000 7fbdc311e640 1 Processor -- start 2024-03-06T07:22:58.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.751+0000 7fbdc311e640 1 -- start start 2024-03-06T07:22:58.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdc311e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbdbc0c6610 0x7fbdbc0aa810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdc311e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdbc152a40 0x7fbdbc1e8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdc311e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbdbc1558e0 0x7fbdbc1e9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdc311e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc0c7140 con 0x7fbdbc152a40 2024-03-06T07:22:58.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdc311e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc158f30 con 0x7fbdbc1558e0 2024-03-06T07:22:58.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdc311e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc0c72c0 con 0x7fbdbc0c6610 2024-03-06T07:22:58.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdbbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdbc152a40 0x7fbdbc1e8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdbbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdbc152a40 0x7fbdbc1e8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57632/0 (socket says 172.21.15.1:57632) 2024-03-06T07:22:58.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdbbfff640 1 -- 172.21.15.1:0/3878290303 learned_addr learned my addr 172.21.15.1:0/3878290303 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:58.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdc0eba640 1 --2- 172.21.15.1:0/3878290303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbdbc0c6610 0x7fbdbc0aa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdc16bb640 1 --2- 172.21.15.1:0/3878290303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbdbc1558e0 0x7fbdbc1e9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdbbfff640 1 -- 172.21.15.1:0/3878290303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbdbc0c6610 msgr2=0x7fbdbc0aa810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdbbfff640 1 --2- 172.21.15.1:0/3878290303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbdbc0c6610 0x7fbdbc0aa810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdbbfff640 1 -- 172.21.15.1:0/3878290303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbdbc1558e0 msgr2=0x7fbdbc1e9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdbbfff640 1 --2- 172.21.15.1:0/3878290303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbdbc1558e0 0x7fbdbc1e9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdbbfff640 1 -- 172.21.15.1:0/3878290303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdbc0a9c10 con 0x7fbdbc152a40 2024-03-06T07:22:58.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdc16bb640 1 --2- 172.21.15.1:0/3878290303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbdbc1558e0 0x7fbdbc1e9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:58.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdc0eba640 1 --2- 172.21.15.1:0/3878290303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbdbc0c6610 0x7fbdbc0aa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.752+0000 7fbdbbfff640 1 --2- 172.21.15.1:0/3878290303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdbc152a40 0x7fbdbc1e8d40 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fbda8009960 tx=0x7fbda80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.753+0000 7fbdb97fa640 1 -- 172.21.15.1:0/3878290303 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda8042020 con 0x7fbdbc152a40 2024-03-06T07:22:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.753+0000 7fbdb97fa640 1 -- 172.21.15.1:0/3878290303 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda803d890 con 0x7fbdbc152a40 2024-03-06T07:22:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.753+0000 7fbdc311e640 1 -- 172.21.15.1:0/3878290303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdbc0aaea0 con 0x7fbdbc152a40 2024-03-06T07:22:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.753+0000 7fbdb97fa640 1 -- 172.21.15.1:0/3878290303 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda8038040 con 0x7fbdbc152a40 2024-03-06T07:22:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.753+0000 7fbdc311e640 1 -- 172.21.15.1:0/3878290303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdbc0a8a00 con 0x7fbdbc152a40 2024-03-06T07:22:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.754+0000 7fbdc311e640 1 -- 172.21.15.1:0/3878290303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdbc0686e0 con 0x7fbdbc152a40 2024-03-06T07:22:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.754+0000 7fbdb97fa640 1 -- 172.21.15.1:0/3878290303 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbda8045020 con 0x7fbdbc152a40 2024-03-06T07:22:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.755+0000 7fbdb97fa640 1 --2- 172.21.15.1:0/3878290303 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbda006b4d0 0x7fbda006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.755+0000 7fbdb97fa640 1 -- 172.21.15.1:0/3878290303 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbda80abd90 con 0x7fbdbc152a40 2024-03-06T07:22:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.755+0000 7fbdc0eba640 1 --2- 172.21.15.1:0/3878290303 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbda006b4d0 0x7fbda006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.755+0000 7fbdc0eba640 1 --2- 172.21.15.1:0/3878290303 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbda006b4d0 0x7fbda006d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0006fd0 tx=0x7fbdb0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.759+0000 7fbdb97fa640 1 -- 172.21.15.1:0/3878290303 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbda804c030 con 0x7fbdbc152a40 2024-03-06T07:22:58.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:58.979+0000 7fbdc311e640 1 -- 172.21.15.1:0/3878290303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbdbc0c0df0 con 0x7fbdbc152a40 2024-03-06T07:22:59.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.630+0000 7fbdb97fa640 1 -- 172.21.15.1:0/3878290303 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v54) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbda807ab10 con 0x7fbdbc152a40 2024-03-06T07:22:59.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.632+0000 7fbdc311e640 1 -- 172.21.15.1:0/3878290303 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbda006b4d0 msgr2=0x7fbda006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.632+0000 7fbdc311e640 1 --2- 172.21.15.1:0/3878290303 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbda006b4d0 0x7fbda006d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0006fd0 tx=0x7fbdb0008040 comp rx=0 tx=0).stop 2024-03-06T07:22:59.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.632+0000 7fbdc311e640 1 -- 172.21.15.1:0/3878290303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdbc152a40 msgr2=0x7fbdbc1e8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.632+0000 7fbdc311e640 1 --2- 172.21.15.1:0/3878290303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdbc152a40 0x7fbdbc1e8d40 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fbda8009960 tx=0x7fbda80046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.632+0000 7fbdc311e640 1 -- 172.21.15.1:0/3878290303 shutdown_connections 2024-03-06T07:22:59.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.632+0000 7fbdc311e640 1 --2- 172.21.15.1:0/3878290303 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbda006b4d0 0x7fbda006d990 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.632+0000 7fbdc311e640 1 --2- 172.21.15.1:0/3878290303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbdbc1558e0 0x7fbdbc1e9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.632+0000 7fbdc311e640 1 --2- 172.21.15.1:0/3878290303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdbc152a40 0x7fbdbc1e8d40 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.632+0000 7fbdc311e640 1 --2- 172.21.15.1:0/3878290303 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbdbc0c6610 0x7fbdbc0aa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.632+0000 7fbdc311e640 1 -- 172.21.15.1:0/3878290303 >> 172.21.15.1:0/3878290303 conn(0x7fbdbc0c79c0 msgr2=0x7fbdbc1551a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.632+0000 7fbdc311e640 1 -- 172.21.15.1:0/3878290303 shutdown_connections 2024-03-06T07:22:59.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.632+0000 7fbdc311e640 1 -- 172.21.15.1:0/3878290303 wait complete. 2024-03-06T07:22:59.645 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:22:59.645 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:59.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.744+0000 7f513c0e1640 1 Processor -- start 2024-03-06T07:22:59.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.744+0000 7f513c0e1640 1 -- start start 2024-03-06T07:22:59.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.745+0000 7f513c0e1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f51340c84c0 0x7f51341550d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.745+0000 7f513c0e1640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51340c7650 con 0x7f51340c3700 2024-03-06T07:22:59.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.745+0000 7f513c0e1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51340c78f0 con 0x7f51340c5520 2024-03-06T07:22:59.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.745+0000 7f513c0e1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51341587e0 con 0x7f51340c84c0 2024-03-06T07:22:59.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.745+0000 7f513a67e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f51340c3700 0x7f513406da30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53602/0 (socket says 172.21.15.1:53602) 2024-03-06T07:22:59.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.745+0000 7f513a67e640 1 -- 172.21.15.1:0/3654430455 learned_addr learned my addr 172.21.15.1:0/3654430455 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:22:59.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.745+0000 7f513967c640 1 --2- 172.21.15.1:0/3654430455 >> v2:172.21.15.98:3300/0 conn(0x7f51340c84c0 0x7f51341550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5138e7b640 1 -- 172.21.15.1:0/3654430455 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156929872 0 0) 0x7f51340c7650 con 0x7f51340c3700 2024-03-06T07:22:59.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5138e7b640 1 -- 172.21.15.1:0/3654430455 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5120003520 con 0x7f51340c3700 2024-03-06T07:22:59.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5138e7b640 1 -- 172.21.15.1:0/3654430455 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457487727 0 0) 0x7f51340c78f0 con 0x7f51340c5520 2024-03-06T07:22:59.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5138e7b640 1 -- 172.21.15.1:0/3654430455 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51340c7650 con 0x7f51340c5520 2024-03-06T07:22:59.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5138e7b640 1 -- 172.21.15.1:0/3654430455 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 829821102 0 0) 0x7f5120003520 con 0x7f51340c3700 2024-03-06T07:22:59.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5138e7b640 1 -- 172.21.15.1:0/3654430455 >> v2:172.21.15.98:3300/0 conn(0x7f51340c84c0 msgr2=0x7f51341550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5138e7b640 1 --2- 172.21.15.1:0/3654430455 >> v2:172.21.15.98:3300/0 conn(0x7f51340c84c0 0x7f51341550d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5138e7b640 1 -- 172.21.15.1:0/3654430455 >> v1:172.21.15.98:6789/0 conn(0x7f51340c5520 legacy=0x7f5134158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5138e7b640 1 -- 172.21.15.1:0/3654430455 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5134159410 con 0x7f51340c3700 2024-03-06T07:22:59.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5138e7b640 1 -- 172.21.15.1:0/3654430455 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f512c0019f0 con 0x7f51340c3700 2024-03-06T07:22:59.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5138e7b640 1 -- 172.21.15.1:0/3654430455 >> v1:172.21.15.1:6789/0 conn(0x7f51340c3700 legacy=0x7f513406da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5138e7b640 1 --2- 172.21.15.1:0/3654430455 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5120003b40 0x7f5120003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5138e7b640 1 --2- 172.21.15.1:0/3654430455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5120004800 0x7f5120004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5138e7b640 1 --2- 172.21.15.1:0/3654430455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51200051c0 0x7f512000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5138e7b640 1 -- 172.21.15.1:0/3654430455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51341587e0 con 0x7f5120004800 2024-03-06T07:22:59.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5138e7b640 1 -- 172.21.15.1:0/3654430455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512000c160 con 0x7f51200051c0 2024-03-06T07:22:59.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5138e7b640 1 -- 172.21.15.1:0/3654430455 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f512000c360 con 0x7f5120003b40 2024-03-06T07:22:59.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f513967c640 1 --2- 172.21.15.1:0/3654430455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5120004800 0x7f5120004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f5139e7d640 1 --2- 172.21.15.1:0/3654430455 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5120003b40 0x7f5120003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f513a67e640 1 --2- 172.21.15.1:0/3654430455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51200051c0 0x7f512000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f513967c640 1 -- 172.21.15.1:0/3654430455 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5120003b40 msgr2=0x7f5120003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f513967c640 1 --2- 172.21.15.1:0/3654430455 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5120003b40 0x7f5120003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f513967c640 1 -- 172.21.15.1:0/3654430455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51200051c0 msgr2=0x7f512000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f513967c640 1 --2- 172.21.15.1:0/3654430455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51200051c0 0x7f512000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f513a67e640 1 -- 172.21.15.1:0/3654430455 reap_dead start 2024-03-06T07:22:59.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.746+0000 7f513967c640 1 -- 172.21.15.1:0/3654430455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5120003520 con 0x7f5120004800 2024-03-06T07:22:59.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.747+0000 7f513967c640 1 --2- 172.21.15.1:0/3654430455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5120004800 0x7f5120004c80 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f5128009a90 tx=0x7f5120005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c092bcb7c0ffc990 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.748+0000 7f5138e7b640 1 -- 172.21.15.1:0/3654430455 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5128039070 con 0x7f5120004800 2024-03-06T07:22:59.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.748+0000 7f5138e7b640 1 -- 172.21.15.1:0/3654430455 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512802c680 con 0x7f5120004800 2024-03-06T07:22:59.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.748+0000 7f5138e7b640 1 -- 172.21.15.1:0/3654430455 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5128009690 con 0x7f5120004800 2024-03-06T07:22:59.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.748+0000 7f513c0e1640 1 -- 172.21.15.1:0/3654430455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5120004800 msgr2=0x7f5120004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.748+0000 7f513c0e1640 1 --2- 172.21.15.1:0/3654430455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5120004800 0x7f5120004c80 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f5128009a90 tx=0x7f5120005830 comp rx=0 tx=0).stop 2024-03-06T07:22:59.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.748+0000 7f513c0e1640 1 -- 172.21.15.1:0/3654430455 shutdown_connections 2024-03-06T07:22:59.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.748+0000 7f513c0e1640 1 --2- 172.21.15.1:0/3654430455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5120004800 0x7f5120004c80 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.748+0000 7f513c0e1640 1 -- 172.21.15.1:0/3654430455 >> 172.21.15.1:0/3654430455 conn(0x7f51340c88f0 msgr2=0x7f51340c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.748+0000 7f513c0e1640 1 -- 172.21.15.1:0/3654430455 shutdown_connections 2024-03-06T07:22:59.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.748+0000 7f513c0e1640 1 -- 172.21.15.1:0/3654430455 wait complete. 2024-03-06T07:22:59.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.749+0000 7f513c0e1640 1 Processor -- start 2024-03-06T07:22:59.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.749+0000 7f513c0e1640 1 -- start start 2024-03-06T07:22:59.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f513c0e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5120004800 0x7f51340ac500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f513c0e1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51340a9900 0x7f513406f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f513c0e1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f513406fcd0 0x7f51341496f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f513c0e1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51340c4220 con 0x7f51340a9900 2024-03-06T07:22:59.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f513c0e1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51340c40a0 con 0x7f5120004800 2024-03-06T07:22:59.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f513c0e1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51340c5900 con 0x7f513406fcd0 2024-03-06T07:22:59.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f513967c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51340a9900 0x7f513406f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f513967c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51340a9900 0x7f513406f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57642/0 (socket says 172.21.15.1:57642) 2024-03-06T07:22:59.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f513967c640 1 -- 172.21.15.1:0/2887880817 learned_addr learned my addr 172.21.15.1:0/2887880817 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:22:59.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f5139e7d640 1 --2- 172.21.15.1:0/2887880817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5120004800 0x7f51340ac500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f513a67e640 1 --2- 172.21.15.1:0/2887880817 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f513406fcd0 0x7f51341496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f513967c640 1 -- 172.21.15.1:0/2887880817 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f513406fcd0 msgr2=0x7f51341496f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f513967c640 1 --2- 172.21.15.1:0/2887880817 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f513406fcd0 0x7f51341496f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f513967c640 1 -- 172.21.15.1:0/2887880817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5120004800 msgr2=0x7f51340ac500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f513967c640 1 --2- 172.21.15.1:0/2887880817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5120004800 0x7f51340ac500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f513967c640 1 -- 172.21.15.1:0/2887880817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51340adb90 con 0x7f51340a9900 2024-03-06T07:22:59.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f5139e7d640 1 --2- 172.21.15.1:0/2887880817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5120004800 0x7f51340ac500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:59.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.750+0000 7f513a67e640 1 --2- 172.21.15.1:0/2887880817 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f513406fcd0 0x7f51341496f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:59.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.751+0000 7f513967c640 1 --2- 172.21.15.1:0/2887880817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51340a9900 0x7f513406f790 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f5128009a90 tx=0x7f5128004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.751+0000 7f51267fc640 1 -- 172.21.15.1:0/2887880817 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5128039050 con 0x7f51340a9900 2024-03-06T07:22:59.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.751+0000 7f51267fc640 1 -- 172.21.15.1:0/2887880817 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5128045070 con 0x7f51340a9900 2024-03-06T07:22:59.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.751+0000 7f513c0e1640 1 -- 172.21.15.1:0/2887880817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5134070b30 con 0x7f51340a9900 2024-03-06T07:22:59.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.751+0000 7f51267fc640 1 -- 172.21.15.1:0/2887880817 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51280044e0 con 0x7f51340a9900 2024-03-06T07:22:59.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.751+0000 7f513c0e1640 1 -- 172.21.15.1:0/2887880817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51340710f0 con 0x7f51340a9900 2024-03-06T07:22:59.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.752+0000 7f513c0e1640 1 -- 172.21.15.1:0/2887880817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51340686e0 con 0x7f51340a9900 2024-03-06T07:22:59.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.752+0000 7f51267fc640 1 -- 172.21.15.1:0/2887880817 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5128003850 con 0x7f51340a9900 2024-03-06T07:22:59.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.752+0000 7f51267fc640 1 --2- 172.21.15.1:0/2887880817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f511806b500 0x7f511806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.753+0000 7f51267fc640 1 -- 172.21.15.1:0/2887880817 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f51280abdc0 con 0x7f51340a9900 2024-03-06T07:22:59.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.753+0000 7f5139e7d640 1 --2- 172.21.15.1:0/2887880817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f511806b500 0x7f511806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.753+0000 7f5139e7d640 1 --2- 172.21.15.1:0/2887880817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f511806b500 0x7f511806d9c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5130000c00 tx=0x7f5130009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:22:59.757+0000 7f51267fc640 1 -- 172.21.15.1:0/2887880817 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f512804e020 con 0x7f51340a9900 2024-03-06T07:23:00.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.036+0000 7f513c0e1640 1 -- 172.21.15.1:0/2887880817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51340745f0 con 0x7f51340a9900 2024-03-06T07:23:00.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.036+0000 7f51267fc640 1 -- 172.21.15.1:0/2887880817 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4970 (secure 0 0 0) 0x7f512807ac60 con 0x7f51340a9900 2024-03-06T07:23:00.039 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:00.039 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":54,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2685225325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2685225325},{"type":"v1","addr":"172.21.15.1:6835","nonce":2685225325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":5842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/4069609682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4069609682},{"type":"v1","addr":"172.21.15.98:6836","nonce":4069609682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":5845,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/114998998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":114998998},{"type":"v1","addr":"172.21.15.98:6837","nonce":114998998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.969135+0000","modified":"2024-03-06T07:22:58.980537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6405},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6405":{"gid":6405,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1494116991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1494116991},{"type":"v1","addr":"172.21.15.1:6834","nonce":1494116991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6405,"qdb_cluster":[6405]},"id":5}]} 2024-03-06T07:23:00.039 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 54 2024-03-06T07:23:00.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.038+0000 7f513c0e1640 1 -- 172.21.15.1:0/2887880817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f511806b500 msgr2=0x7f511806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.038+0000 7f513c0e1640 1 --2- 172.21.15.1:0/2887880817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f511806b500 0x7f511806d9c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5130000c00 tx=0x7f5130009040 comp rx=0 tx=0).stop 2024-03-06T07:23:00.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.038+0000 7f513c0e1640 1 -- 172.21.15.1:0/2887880817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51340a9900 msgr2=0x7f513406f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.038+0000 7f513c0e1640 1 --2- 172.21.15.1:0/2887880817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51340a9900 0x7f513406f790 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f5128009a90 tx=0x7f5128004730 comp rx=0 tx=0).stop 2024-03-06T07:23:00.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.038+0000 7f513c0e1640 1 -- 172.21.15.1:0/2887880817 shutdown_connections 2024-03-06T07:23:00.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.038+0000 7f513c0e1640 1 --2- 172.21.15.1:0/2887880817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f511806b500 0x7f511806d9c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.038+0000 7f513c0e1640 1 --2- 172.21.15.1:0/2887880817 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f513406fcd0 0x7f51341496f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.038+0000 7f513c0e1640 1 --2- 172.21.15.1:0/2887880817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51340a9900 0x7f513406f790 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.038+0000 7f513c0e1640 1 --2- 172.21.15.1:0/2887880817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5120004800 0x7f51340ac500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.038+0000 7f513c0e1640 1 -- 172.21.15.1:0/2887880817 >> 172.21.15.1:0/2887880817 conn(0x7f51340c88f0 msgr2=0x7f51341542c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.039+0000 7f513c0e1640 1 -- 172.21.15.1:0/2887880817 shutdown_connections 2024-03-06T07:23:00.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.039+0000 7f513c0e1640 1 -- 172.21.15.1:0/2887880817 wait complete. 2024-03-06T07:23:00.051 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:00.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.151+0000 7feb29637640 1 Processor -- start 2024-03-06T07:23:00.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.151+0000 7feb29637640 1 -- start start 2024-03-06T07:23:00.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.152+0000 7feb29637640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb24152650 0x7feb24155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.152+0000 7feb29637640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb240c6b10 0x7feb24158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.152+0000 7feb29637640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb240c2fc0 con 0x7feb240bf9d0 2024-03-06T07:23:00.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.152+0000 7feb29637640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb24155800 con 0x7feb24152650 2024-03-06T07:23:00.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.152+0000 7feb29637640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb240c6690 con 0x7feb240c6b10 2024-03-06T07:23:00.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.152+0000 7feb1bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb240c6b10 0x7feb24158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.152+0000 7feb22ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb24152650 0x7feb24155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.152+0000 7feb1bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb240c6b10 0x7feb24158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:47616/0 (socket says 172.21.15.1:47616) 2024-03-06T07:23:00.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.152+0000 7feb1bfff640 1 -- 172.21.15.1:0/1974723201 learned_addr learned my addr 172.21.15.1:0/1974723201 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:00.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.153+0000 7feb1bfff640 1 -- 172.21.15.1:0/1974723201 >> v2:172.21.15.98:3300/0 conn(0x7feb24152650 msgr2=0x7feb24155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:00.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.153+0000 7feb1bfff640 1 --2- 172.21.15.1:0/1974723201 >> v2:172.21.15.98:3300/0 conn(0x7feb24152650 0x7feb24155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.153+0000 7feb1bfff640 1 -- 172.21.15.1:0/1974723201 >> v1:172.21.15.98:6789/0 conn(0x7feb240bf9d0 legacy=0x7feb240c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.153+0000 7feb1bfff640 1 -- 172.21.15.1:0/1974723201 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb240c7280 con 0x7feb240c6b10 2024-03-06T07:23:00.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.153+0000 7feb1bfff640 1 --2- 172.21.15.1:0/1974723201 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb240c6b10 0x7feb24158690 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009870 tx=0x7feb0c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9514dd9fc57354b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.154+0000 7feb227fc640 1 -- 172.21.15.1:0/1974723201 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c035020 con 0x7feb240c6b10 2024-03-06T07:23:00.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.154+0000 7feb227fc640 1 -- 172.21.15.1:0/1974723201 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0c031640 con 0x7feb240c6b10 2024-03-06T07:23:00.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.154+0000 7feb227fc640 1 -- 172.21.15.1:0/1974723201 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c02d040 con 0x7feb240c6b10 2024-03-06T07:23:00.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.154+0000 7feb29637640 1 -- 172.21.15.1:0/1974723201 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb240c6b10 msgr2=0x7feb24158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.154+0000 7feb29637640 1 --2- 172.21.15.1:0/1974723201 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb240c6b10 0x7feb24158690 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009870 tx=0x7feb0c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.154+0000 7feb29637640 1 -- 172.21.15.1:0/1974723201 shutdown_connections 2024-03-06T07:23:00.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.154+0000 7feb29637640 1 --2- 172.21.15.1:0/1974723201 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb240c6b10 0x7feb24158690 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.154+0000 7feb29637640 1 --2- 172.21.15.1:0/1974723201 >> v2:172.21.15.98:3300/0 conn(0x7feb24152650 0x7feb24155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.154+0000 7feb29637640 1 -- 172.21.15.1:0/1974723201 >> 172.21.15.1:0/1974723201 conn(0x7feb24066ee0 msgr2=0x7feb240672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.154+0000 7feb29637640 1 -- 172.21.15.1:0/1974723201 shutdown_connections 2024-03-06T07:23:00.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.154+0000 7feb29637640 1 -- 172.21.15.1:0/1974723201 wait complete. 2024-03-06T07:23:00.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.156+0000 7feb29637640 1 Processor -- start 2024-03-06T07:23:00.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.156+0000 7feb29637640 1 -- start start 2024-03-06T07:23:00.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.157+0000 7feb29637640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb240bf9d0 0x7feb241480b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.157+0000 7feb29637640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb24152650 0x7feb241eb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.157+0000 7feb29637640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb241e12e0 0x7feb241ebc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.157+0000 7feb29637640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb24155490 con 0x7feb241e12e0 2024-03-06T07:23:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.157+0000 7feb29637640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb241551b0 con 0x7feb240bf9d0 2024-03-06T07:23:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.157+0000 7feb29637640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb24152a30 con 0x7feb24152650 2024-03-06T07:23:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.157+0000 7feb237fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb241e12e0 0x7feb241ebc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.157+0000 7feb237fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb241e12e0 0x7feb241ebc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57656/0 (socket says 172.21.15.1:57656) 2024-03-06T07:23:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.157+0000 7feb237fe640 1 -- 172.21.15.1:0/581530838 learned_addr learned my addr 172.21.15.1:0/581530838 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.157+0000 7feb1bfff640 1 --2- 172.21.15.1:0/581530838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb24152650 0x7feb241eb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.158+0000 7feb22ffd640 1 --2- 172.21.15.1:0/581530838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb240bf9d0 0x7feb241480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.158+0000 7feb237fe640 1 -- 172.21.15.1:0/581530838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb24152650 msgr2=0x7feb241eb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.158+0000 7feb237fe640 1 --2- 172.21.15.1:0/581530838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb24152650 0x7feb241eb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.158+0000 7feb237fe640 1 -- 172.21.15.1:0/581530838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb240bf9d0 msgr2=0x7feb241480b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.158+0000 7feb237fe640 1 --2- 172.21.15.1:0/581530838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb240bf9d0 0x7feb241480b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.158+0000 7feb237fe640 1 -- 172.21.15.1:0/581530838 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb24154f70 con 0x7feb241e12e0 2024-03-06T07:23:00.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.158+0000 7feb237fe640 1 --2- 172.21.15.1:0/581530838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb241e12e0 0x7feb241ebc20 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00c6e0 tx=0x7feb1c00cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.158+0000 7feb1b7fe640 1 -- 172.21.15.1:0/581530838 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c003b70 con 0x7feb241e12e0 2024-03-06T07:23:00.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.158+0000 7feb1b7fe640 1 -- 172.21.15.1:0/581530838 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1c003d30 con 0x7feb241e12e0 2024-03-06T07:23:00.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.158+0000 7feb29637640 1 -- 172.21.15.1:0/581530838 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb24152fb0 con 0x7feb241e12e0 2024-03-06T07:23:00.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.158+0000 7feb1b7fe640 1 -- 172.21.15.1:0/581530838 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c005310 con 0x7feb241e12e0 2024-03-06T07:23:00.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.158+0000 7feb29637640 1 -- 172.21.15.1:0/581530838 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb24153570 con 0x7feb241e12e0 2024-03-06T07:23:00.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.159+0000 7feb29637640 1 -- 172.21.15.1:0/581530838 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2414f5e0 con 0x7feb241e12e0 2024-03-06T07:23:00.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.160+0000 7feb1b7fe640 1 -- 172.21.15.1:0/581530838 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb1c0054b0 con 0x7feb241e12e0 2024-03-06T07:23:00.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.160+0000 7feb1b7fe640 1 --2- 172.21.15.1:0/581530838 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb0406b4d0 0x7feb0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.160+0000 7feb1b7fe640 1 -- 172.21.15.1:0/581530838 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feb1c089ff0 con 0x7feb241e12e0 2024-03-06T07:23:00.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.160+0000 7feb22ffd640 1 --2- 172.21.15.1:0/581530838 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb0406b4d0 0x7feb0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.161+0000 7feb22ffd640 1 --2- 172.21.15.1:0/581530838 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb0406b4d0 0x7feb0406d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7feb241433a0 tx=0x7feb14009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.169+0000 7feb1b7fe640 1 -- 172.21.15.1:0/581530838 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb1c058720 con 0x7feb241e12e0 2024-03-06T07:23:00.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.350+0000 7feb29637640 1 -- 172.21.15.1:0/581530838 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb24150cb0 con 0x7feb241e12e0 2024-03-06T07:23:00.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.352+0000 7feb1b7fe640 1 -- 172.21.15.1:0/581530838 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+16825 (secure 0 0 0) 0x7feb1c05c7c0 con 0x7feb241e12e0 2024-03-06T07:23:00.354 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:00.354 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":74,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:58.618600+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:55.240857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:55.939811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:00.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.353+0000 7feb29637640 1 -- 172.21.15.1:0/581530838 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb0406b4d0 msgr2=0x7feb0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.353+0000 7feb29637640 1 --2- 172.21.15.1:0/581530838 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb0406b4d0 0x7feb0406d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7feb241433a0 tx=0x7feb14009190 comp rx=0 tx=0).stop 2024-03-06T07:23:00.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.353+0000 7feb29637640 1 -- 172.21.15.1:0/581530838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb241e12e0 msgr2=0x7feb241ebc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.353+0000 7feb29637640 1 --2- 172.21.15.1:0/581530838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb241e12e0 0x7feb241ebc20 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00c6e0 tx=0x7feb1c00cba0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.353+0000 7feb29637640 1 -- 172.21.15.1:0/581530838 shutdown_connections 2024-03-06T07:23:00.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.353+0000 7feb29637640 1 --2- 172.21.15.1:0/581530838 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb0406b4d0 0x7feb0406d990 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.353+0000 7feb29637640 1 --2- 172.21.15.1:0/581530838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb241e12e0 0x7feb241ebc20 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.353+0000 7feb29637640 1 --2- 172.21.15.1:0/581530838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb24152650 0x7feb241eb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.353+0000 7feb29637640 1 --2- 172.21.15.1:0/581530838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb240bf9d0 0x7feb241480b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.354+0000 7feb29637640 1 -- 172.21.15.1:0/581530838 >> 172.21.15.1:0/581530838 conn(0x7feb24066ee0 msgr2=0x7feb240cac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.354+0000 7feb29637640 1 -- 172.21.15.1:0/581530838 shutdown_connections 2024-03-06T07:23:00.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.354+0000 7feb29637640 1 -- 172.21.15.1:0/581530838 wait complete. 2024-03-06T07:23:00.366 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:23:00.367 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:00.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.467+0000 7fedea1bd640 1 Processor -- start 2024-03-06T07:23:00.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.467+0000 7fedea1bd640 1 -- start start 2024-03-06T07:23:00.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.467+0000 7fedea1bd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fede40c6b10 0x7fede4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.467+0000 7fedea1bd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede40bf9d0 0x7fede40c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.467+0000 7fedea1bd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede40c2fc0 con 0x7fede4152650 2024-03-06T07:23:00.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.467+0000 7fedea1bd640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede4155800 con 0x7fede40c6b10 2024-03-06T07:23:00.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.467+0000 7fedea1bd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede4158da0 con 0x7fede40bf9d0 2024-03-06T07:23:00.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.467+0000 7fede2ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fede40c6b10 0x7fede4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.468+0000 7fede2ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fede40c6b10 0x7fede4155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57664/0 (socket says 172.21.15.1:57664) 2024-03-06T07:23:00.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.468+0000 7fede2ffd640 1 -- 172.21.15.1:0/3901889098 learned_addr learned my addr 172.21.15.1:0/3901889098 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:00.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.468+0000 7fede3fff640 1 --2- 172.21.15.1:0/3901889098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede40bf9d0 0x7fede40c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.468+0000 7fede2ffd640 1 -- 172.21.15.1:0/3901889098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede40bf9d0 msgr2=0x7fede40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:00.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.468+0000 7fede2ffd640 1 --2- 172.21.15.1:0/3901889098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede40bf9d0 0x7fede40c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.468+0000 7fede2ffd640 1 -- 172.21.15.1:0/3901889098 >> v1:172.21.15.98:6789/0 conn(0x7fede4152650 legacy=0x7fede4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.468+0000 7fede2ffd640 1 -- 172.21.15.1:0/3901889098 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede40c6690 con 0x7fede40c6b10 2024-03-06T07:23:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.469+0000 7fede2ffd640 1 --2- 172.21.15.1:0/3901889098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede40c6b10 0x7fede4155c90 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009870 tx=0x7fedd802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f2c568093161a74 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.469+0000 7fede27fc640 1 -- 172.21.15.1:0/3901889098 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd8035020 con 0x7fede40c6b10 2024-03-06T07:23:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.469+0000 7fede27fc640 1 -- 172.21.15.1:0/3901889098 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd8031640 con 0x7fede40c6b10 2024-03-06T07:23:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.469+0000 7fede27fc640 1 -- 172.21.15.1:0/3901889098 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd802d040 con 0x7fede40c6b10 2024-03-06T07:23:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.469+0000 7fedea1bd640 1 -- 172.21.15.1:0/3901889098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede40c6b10 msgr2=0x7fede4155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.469+0000 7fedea1bd640 1 --2- 172.21.15.1:0/3901889098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede40c6b10 0x7fede4155c90 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009870 tx=0x7fedd802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.469+0000 7fedea1bd640 1 -- 172.21.15.1:0/3901889098 shutdown_connections 2024-03-06T07:23:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.469+0000 7fedea1bd640 1 --2- 172.21.15.1:0/3901889098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede40bf9d0 0x7fede40c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.469+0000 7fedea1bd640 1 --2- 172.21.15.1:0/3901889098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede40c6b10 0x7fede4155c90 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.469+0000 7fedea1bd640 1 -- 172.21.15.1:0/3901889098 >> 172.21.15.1:0/3901889098 conn(0x7fede4066ee0 msgr2=0x7fede40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.469+0000 7fedea1bd640 1 -- 172.21.15.1:0/3901889098 shutdown_connections 2024-03-06T07:23:00.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.469+0000 7fedea1bd640 1 -- 172.21.15.1:0/3901889098 wait complete. 2024-03-06T07:23:00.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.470+0000 7fedea1bd640 1 Processor -- start 2024-03-06T07:23:00.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.471+0000 7fedea1bd640 1 -- start start 2024-03-06T07:23:00.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.471+0000 7fedea1bd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede40bf9d0 0x7fede414fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.471+0000 7fedea1bd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede4152650 0x7fede4150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.471+0000 7fedea1bd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede4156b30 0x7fede4156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.471+0000 7fedea1bd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede41551b0 con 0x7fede4156b30 2024-03-06T07:23:00.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.471+0000 7fedea1bd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede40c6fa0 con 0x7fede4152650 2024-03-06T07:23:00.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.471+0000 7fedea1bd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede4155490 con 0x7fede40bf9d0 2024-03-06T07:23:00.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.471+0000 7fede3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede4156b30 0x7fede4156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.471+0000 7fede3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede4156b30 0x7fede4156fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57672/0 (socket says 172.21.15.1:57672) 2024-03-06T07:23:00.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.471+0000 7fede3fff640 1 -- 172.21.15.1:0/3652333151 learned_addr learned my addr 172.21.15.1:0/3652333151 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:00.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.472+0000 7fede37fe640 1 --2- 172.21.15.1:0/3652333151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede40bf9d0 0x7fede414fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.472+0000 7fede2ffd640 1 --2- 172.21.15.1:0/3652333151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede4152650 0x7fede4150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.472+0000 7fede3fff640 1 -- 172.21.15.1:0/3652333151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede40bf9d0 msgr2=0x7fede414fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.472+0000 7fede3fff640 1 --2- 172.21.15.1:0/3652333151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede40bf9d0 0x7fede414fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.472+0000 7fede3fff640 1 -- 172.21.15.1:0/3652333151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede4152650 msgr2=0x7fede4150370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.472+0000 7fede3fff640 1 --2- 172.21.15.1:0/3652333151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede4152650 0x7fede4150370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.472+0000 7fede3fff640 1 -- 172.21.15.1:0/3652333151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede4158480 con 0x7fede4156b30 2024-03-06T07:23:00.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.472+0000 7fede3fff640 1 --2- 172.21.15.1:0/3652333151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede4156b30 0x7fede4156fc0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7feddc009d30 tx=0x7feddc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.472+0000 7fedcbfff640 1 -- 172.21.15.1:0/3652333151 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc018020 con 0x7fede4156b30 2024-03-06T07:23:00.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.472+0000 7fedea1bd640 1 -- 172.21.15.1:0/3652333151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede414b9c0 con 0x7fede4156b30 2024-03-06T07:23:00.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.472+0000 7fedea1bd640 1 -- 172.21.15.1:0/3652333151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede414b530 con 0x7fede4156b30 2024-03-06T07:23:00.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.472+0000 7fedcbfff640 1 -- 172.21.15.1:0/3652333151 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feddc014480 con 0x7fede4156b30 2024-03-06T07:23:00.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.472+0000 7fedcbfff640 1 -- 172.21.15.1:0/3652333151 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc014830 con 0x7fede4156b30 2024-03-06T07:23:00.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.473+0000 7fedea1bd640 1 -- 172.21.15.1:0/3652333151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc4005290 con 0x7fede4156b30 2024-03-06T07:23:00.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.474+0000 7fedcbfff640 1 -- 172.21.15.1:0/3652333151 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feddc005d10 con 0x7fede4156b30 2024-03-06T07:23:00.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.474+0000 7fedcbfff640 1 --2- 172.21.15.1:0/3652333151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fedd006b500 0x7fedd006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.474+0000 7fedcbfff640 1 -- 172.21.15.1:0/3652333151 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feddc08acc0 con 0x7fede4156b30 2024-03-06T07:23:00.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.474+0000 7fede37fe640 1 --2- 172.21.15.1:0/3652333151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fedd006b500 0x7fedd006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.475+0000 7fede37fe640 1 --2- 172.21.15.1:0/3652333151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fedd006b500 0x7fedd006d9c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fedd4000f60 tx=0x7fedd4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.479+0000 7fedcbfff640 1 -- 172.21.15.1:0/3652333151 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feddc022070 con 0x7fede4156b30 2024-03-06T07:23:00.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.683+0000 7fedea1bd640 1 -- 172.21.15.1:0/3652333151 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fedc4000bc0 con 0x7fedd006b500 2024-03-06T07:23:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.688+0000 7fedcbfff640 1 -- 172.21.15.1:0/3652333151 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237299 (secure 0 0 0) 0x7fedc4000bc0 con 0x7fedd006b500 2024-03-06T07:23:00.693 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:23:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.692+0000 7fedea1bd640 1 -- 172.21.15.1:0/3652333151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fedd006b500 msgr2=0x7fedd006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.692+0000 7fedea1bd640 1 --2- 172.21.15.1:0/3652333151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fedd006b500 0x7fedd006d9c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fedd4000f60 tx=0x7fedd4009040 comp rx=0 tx=0).stop 2024-03-06T07:23:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.692+0000 7fedea1bd640 1 -- 172.21.15.1:0/3652333151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede4156b30 msgr2=0x7fede4156fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.692+0000 7fedea1bd640 1 --2- 172.21.15.1:0/3652333151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede4156b30 0x7fede4156fc0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7feddc009d30 tx=0x7feddc00c490 comp rx=0 tx=0).stop 2024-03-06T07:23:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.692+0000 7fedea1bd640 1 -- 172.21.15.1:0/3652333151 shutdown_connections 2024-03-06T07:23:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.692+0000 7fedea1bd640 1 --2- 172.21.15.1:0/3652333151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fedd006b500 0x7fedd006d9c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.692+0000 7fedea1bd640 1 --2- 172.21.15.1:0/3652333151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede4156b30 0x7fede4156fc0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.692+0000 7fedea1bd640 1 --2- 172.21.15.1:0/3652333151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede4152650 0x7fede4150370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.692+0000 7fedea1bd640 1 --2- 172.21.15.1:0/3652333151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede40bf9d0 0x7fede414fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.692+0000 7fedea1bd640 1 -- 172.21.15.1:0/3652333151 >> 172.21.15.1:0/3652333151 conn(0x7fede4066ee0 msgr2=0x7fede40c59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.692+0000 7fedea1bd640 1 -- 172.21.15.1:0/3652333151 shutdown_connections 2024-03-06T07:23:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.692+0000 7fedea1bd640 1 -- 172.21.15.1:0/3652333151 wait complete. 2024-03-06T07:23:00.705 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-03-06T07:22:58.726807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":239672,"kb_used_data":6816,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735048,"statfs":{"total":773094113280,"available":772848689152,"internally_reserved":0,"allocated":6979584,"data_stored":4390928,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000951"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:56.599511+0000","last_change":"2024-03-06T07:22:56.599511+0000","last_active":"2024-03-06T07:22:56.599511+0000","last_peered":"2024-03-06T07:22:56.599511+0000","last_clean":"2024-03-06T07:22:56.599511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:56.599511+0000","last_undegraded":"2024-03-06T07:22:56.599511+0000","last_fullsized":"2024-03-06T07:22:56.599511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:55.594505+0000","last_change":"2024-03-06T07:22:55.594505+0000","last_active":"2024-03-06T07:22:55.594505+0000","last_peered":"2024-03-06T07:22:55.594505+0000","last_clean":"2024-03-06T07:22:55.594505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:55.594505+0000","last_undegraded":"2024-03-06T07:22:55.594505+0000","last_fullsized":"2024-03-06T07:22:55.594505+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":266,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:22:53.595429+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:22:53.595429+0000","last_peered":"2024-03-06T07:22:53.595429+0000","last_clean":"2024-03-06T07:22:53.595429+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:22:53.595429+0000","last_undegraded":"2024-03-06T07:22:53.595429+0000","last_fullsized":"2024-03-06T07:22:53.595429+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":921600,"data_stored":991642,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27860,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":856064,"data_stored":401274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27860,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":856064,"data_stored":401274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27868,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":856064,"data_stored":401274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32024,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":856064,"data_stored":401274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32016,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":856064,"data_stored":401274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32024,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":856064,"data_stored":401274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32088,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339752,"statfs":{"total":96636764160,"available":96603906048,"internally_reserved":0,"allocated":921600,"data_stored":991642,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:00.709 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:00.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.847+0000 7f1ead193640 1 Processor -- start 2024-03-06T07:23:00.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.847+0000 7f1ead193640 1 -- start start 2024-03-06T07:23:00.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.848+0000 7f1ead193640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1ea80af6f0 0x7f1ea80c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.848+0000 7f1ead193640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1ea80c7930 0x7f1ea815fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.848+0000 7f1ead193640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea80c7180 con 0x7f1ea8152820 2024-03-06T07:23:00.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.848+0000 7f1ead193640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ea8155460 con 0x7f1ea80af6f0 2024-03-06T07:23:00.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.848+0000 7f1ead193640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ea80c6bc0 con 0x7f1ea80c7930 2024-03-06T07:23:00.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.848+0000 7f1e9ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1ea80af6f0 0x7f1ea80c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.848+0000 7f1ea6d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1ea8152820 0x7f1ea80c0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53610/0 (socket says 172.21.15.1:53610) 2024-03-06T07:23:00.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.848+0000 7f1ea6d9d640 1 -- 172.21.15.1:0/1103665222 learned_addr learned my addr 172.21.15.1:0/1103665222 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:00.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.848+0000 7f1ea759e640 1 --2- 172.21.15.1:0/1103665222 >> v2:172.21.15.98:3300/0 conn(0x7f1ea80c7930 0x7f1ea815fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.848+0000 7f1e9ffff640 1 -- 172.21.15.1:0/1103665222 >> v2:172.21.15.98:3300/0 conn(0x7f1ea80c7930 msgr2=0x7f1ea815fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:00.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.848+0000 7f1e9ffff640 1 --2- 172.21.15.1:0/1103665222 >> v2:172.21.15.98:3300/0 conn(0x7f1ea80c7930 0x7f1ea815fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.848+0000 7f1e9ffff640 1 -- 172.21.15.1:0/1103665222 >> v1:172.21.15.1:6789/0 conn(0x7f1ea8152820 legacy=0x7f1ea80c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.848+0000 7f1e9ffff640 1 -- 172.21.15.1:0/1103665222 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea80c6ea0 con 0x7f1ea80af6f0 2024-03-06T07:23:00.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.848+0000 7f1ea659c640 1 -- 172.21.15.1:0/1103665222 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055658073 0 0) 0x7f1ea80c7180 con 0x7f1ea8152820 2024-03-06T07:23:00.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.848+0000 7f1ea759e640 1 --2- 172.21.15.1:0/1103665222 >> v2:172.21.15.98:3300/0 conn(0x7f1ea80c7930 0x7f1ea815fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:00.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.849+0000 7f1e9ffff640 1 --2- 172.21.15.1:0/1103665222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ea80af6f0 0x7f1ea80c73f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f1ea806c030 tx=0x7f1e9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a482655ac7915198 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.849+0000 7f1ea659c640 1 -- 172.21.15.1:0/1103665222 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e90035020 con 0x7f1ea80af6f0 2024-03-06T07:23:00.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.849+0000 7f1ea659c640 1 -- 172.21.15.1:0/1103665222 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e90031830 con 0x7f1ea80af6f0 2024-03-06T07:23:00.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.849+0000 7f1ea659c640 1 -- 172.21.15.1:0/1103665222 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e90031c00 con 0x7f1ea80af6f0 2024-03-06T07:23:00.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.850+0000 7f1ead193640 1 -- 172.21.15.1:0/1103665222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ea80af6f0 msgr2=0x7f1ea80c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.850+0000 7f1ead193640 1 --2- 172.21.15.1:0/1103665222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ea80af6f0 0x7f1ea80c73f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f1ea806c030 tx=0x7f1e9002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.850+0000 7f1ead193640 1 -- 172.21.15.1:0/1103665222 shutdown_connections 2024-03-06T07:23:00.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.850+0000 7f1ead193640 1 --2- 172.21.15.1:0/1103665222 >> v2:172.21.15.98:3300/0 conn(0x7f1ea80c7930 0x7f1ea815fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.850+0000 7f1ead193640 1 --2- 172.21.15.1:0/1103665222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ea80af6f0 0x7f1ea80c73f0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.850+0000 7f1ead193640 1 -- 172.21.15.1:0/1103665222 >> 172.21.15.1:0/1103665222 conn(0x7f1ea8072720 msgr2=0x7f1ea8072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.850+0000 7f1ead193640 1 -- 172.21.15.1:0/1103665222 shutdown_connections 2024-03-06T07:23:00.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.850+0000 7f1ead193640 1 -- 172.21.15.1:0/1103665222 wait complete. 2024-03-06T07:23:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.851+0000 7f1ead193640 1 Processor -- start 2024-03-06T07:23:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.851+0000 7f1ead193640 1 -- start start 2024-03-06T07:23:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1ead193640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ea80af6f0 0x7f1ea8159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1ead193640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ea80c7930 0x7f1ea8154a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1ead193640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ea8152820 0x7f1ea814a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1ead193640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea8152c00 con 0x7f1ea8152820 2024-03-06T07:23:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1ead193640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea80c0840 con 0x7f1ea80c7930 2024-03-06T07:23:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1ead193640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea8152d80 con 0x7f1ea80af6f0 2024-03-06T07:23:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1ea759e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ea8152820 0x7f1ea814a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1ea759e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ea8152820 0x7f1ea814a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57684/0 (socket says 172.21.15.1:57684) 2024-03-06T07:23:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1ea759e640 1 -- 172.21.15.1:0/4003585935 learned_addr learned my addr 172.21.15.1:0/4003585935 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1ea6d9d640 1 --2- 172.21.15.1:0/4003585935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ea80af6f0 0x7f1ea8159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1e9ffff640 1 --2- 172.21.15.1:0/4003585935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ea80c7930 0x7f1ea8154a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1ea759e640 1 -- 172.21.15.1:0/4003585935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ea80af6f0 msgr2=0x7f1ea8159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1ea759e640 1 --2- 172.21.15.1:0/4003585935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ea80af6f0 0x7f1ea8159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1ea759e640 1 -- 172.21.15.1:0/4003585935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ea80c7930 msgr2=0x7f1ea8154a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1ea759e640 1 --2- 172.21.15.1:0/4003585935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ea80c7930 0x7f1ea8154a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1ea759e640 1 -- 172.21.15.1:0/4003585935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea80c3c60 con 0x7f1ea8152820 2024-03-06T07:23:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1e9ffff640 1 --2- 172.21.15.1:0/4003585935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ea80c7930 0x7f1ea8154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.852+0000 7f1ea759e640 1 --2- 172.21.15.1:0/4003585935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ea8152820 0x7f1ea814a8c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f1ea000cef0 tx=0x7f1ea000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.853+0000 7f1e9f7fe640 1 -- 172.21.15.1:0/4003585935 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea001a070 con 0x7f1ea8152820 2024-03-06T07:23:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.853+0000 7f1e9f7fe640 1 -- 172.21.15.1:0/4003585935 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea000d860 con 0x7f1ea8152820 2024-03-06T07:23:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.853+0000 7f1ead193640 1 -- 172.21.15.1:0/4003585935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea814b410 con 0x7f1ea8152820 2024-03-06T07:23:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.853+0000 7f1e9f7fe640 1 -- 172.21.15.1:0/4003585935 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea000dc50 con 0x7f1ea8152820 2024-03-06T07:23:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.853+0000 7f1ead193640 1 -- 172.21.15.1:0/4003585935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea8156a40 con 0x7f1ea8152820 2024-03-06T07:23:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.854+0000 7f1ead193640 1 -- 172.21.15.1:0/4003585935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea80686e0 con 0x7f1ea8152820 2024-03-06T07:23:00.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.854+0000 7f1e9f7fe640 1 -- 172.21.15.1:0/4003585935 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ea001d070 con 0x7f1ea8152820 2024-03-06T07:23:00.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.855+0000 7f1e9f7fe640 1 --2- 172.21.15.1:0/4003585935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e8806b4d0 0x7f1e8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.855+0000 7f1e9f7fe640 1 -- 172.21.15.1:0/4003585935 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1ea008ab70 con 0x7f1ea8152820 2024-03-06T07:23:00.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.855+0000 7f1ea6d9d640 1 --2- 172.21.15.1:0/4003585935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e8806b4d0 0x7f1e8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.856+0000 7f1ea6d9d640 1 --2- 172.21.15.1:0/4003585935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e8806b4d0 0x7f1e8806d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1e98000f60 tx=0x7f1e98007c00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:00.863+0000 7f1e9f7fe640 1 -- 172.21.15.1:0/4003585935 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ea00598f0 con 0x7f1ea8152820 2024-03-06T07:23:01.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.041+0000 7f1ead193640 1 -- 172.21.15.1:0/4003585935 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1ea8156410 con 0x7f1e8806b4d0 2024-03-06T07:23:01.049 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:01.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.047+0000 7f1e9f7fe640 1 -- 172.21.15.1:0/4003585935 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252142 (secure 0 0 0) 0x7f1ea8156410 con 0x7f1e8806b4d0 2024-03-06T07:23:01.052 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:01.054 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:23:01.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.050+0000 7f1ead193640 1 -- 172.21.15.1:0/4003585935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e8806b4d0 msgr2=0x7f1e8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.050+0000 7f1ead193640 1 --2- 172.21.15.1:0/4003585935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e8806b4d0 0x7f1e8806d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1e98000f60 tx=0x7f1e98007c00 comp rx=0 tx=0).stop 2024-03-06T07:23:01.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.050+0000 7f1ead193640 1 -- 172.21.15.1:0/4003585935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ea8152820 msgr2=0x7f1ea814a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.050+0000 7f1ead193640 1 --2- 172.21.15.1:0/4003585935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ea8152820 0x7f1ea814a8c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f1ea000cef0 tx=0x7f1ea000a710 comp rx=0 tx=0).stop 2024-03-06T07:23:01.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.050+0000 7f1ead193640 1 -- 172.21.15.1:0/4003585935 shutdown_connections 2024-03-06T07:23:01.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.050+0000 7f1ead193640 1 --2- 172.21.15.1:0/4003585935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e8806b4d0 0x7f1e8806d990 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.050+0000 7f1ead193640 1 --2- 172.21.15.1:0/4003585935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ea8152820 0x7f1ea814a8c0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.050+0000 7f1ead193640 1 --2- 172.21.15.1:0/4003585935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ea80c7930 0x7f1ea8154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.050+0000 7f1ead193640 1 --2- 172.21.15.1:0/4003585935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ea80af6f0 0x7f1ea8159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.050+0000 7f1ead193640 1 -- 172.21.15.1:0/4003585935 >> 172.21.15.1:0/4003585935 conn(0x7f1ea8072720 msgr2=0x7f1ea806cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.050+0000 7f1ead193640 1 -- 172.21.15.1:0/4003585935 shutdown_connections 2024-03-06T07:23:01.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.050+0000 7f1ead193640 1 -- 172.21.15.1:0/4003585935 wait complete. 2024-03-06T07:23:01.063 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-03-06T07:23:00.727243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244228,"kb_used_data":7252,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754730492,"statfs":{"total":773094113280,"available":772844023808,"internally_reserved":0,"allocated":7426048,"data_stored":4602936,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.126779"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634665+0000","last_change":"2024-03-06T07:22:57.627301+0000","last_active":"2024-03-06T07:22:58.634665+0000","last_peered":"2024-03-06T07:22:58.634665+0000","last_clean":"2024-03-06T07:22:58.634665+0000","last_became_active":"2024-03-06T07:22:57.626618+0000","last_became_peered":"2024-03-06T07:22:57.626618+0000","last_unstale":"2024-03-06T07:22:58.634665+0000","last_undegraded":"2024-03-06T07:22:58.634665+0000","last_fullsized":"2024-03-06T07:22:58.634665+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:10.595979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.825124+0000","last_change":"2024-03-06T07:22:57.636108+0000","last_active":"2024-03-06T07:22:58.825124+0000","last_peered":"2024-03-06T07:22:58.825124+0000","last_clean":"2024-03-06T07:22:58.825124+0000","last_became_active":"2024-03-06T07:22:57.635820+0000","last_became_peered":"2024-03-06T07:22:57.635820+0000","last_unstale":"2024-03-06T07:22:58.825124+0000","last_undegraded":"2024-03-06T07:22:58.825124+0000","last_fullsized":"2024-03-06T07:22:58.825124+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:42.232078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824692+0000","last_change":"2024-03-06T07:22:57.627190+0000","last_active":"2024-03-06T07:22:58.824692+0000","last_peered":"2024-03-06T07:22:58.824692+0000","last_clean":"2024-03-06T07:22:58.824692+0000","last_became_active":"2024-03-06T07:22:57.626893+0000","last_became_peered":"2024-03-06T07:22:57.626893+0000","last_unstale":"2024-03-06T07:22:58.824692+0000","last_undegraded":"2024-03-06T07:22:58.824692+0000","last_fullsized":"2024-03-06T07:22:58.824692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:04.450678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824313+0000","last_change":"2024-03-06T07:22:57.632863+0000","last_active":"2024-03-06T07:22:58.824313+0000","last_peered":"2024-03-06T07:22:58.824313+0000","last_clean":"2024-03-06T07:22:58.824313+0000","last_became_active":"2024-03-06T07:22:57.632435+0000","last_became_peered":"2024-03-06T07:22:57.632435+0000","last_unstale":"2024-03-06T07:22:58.824313+0000","last_undegraded":"2024-03-06T07:22:58.824313+0000","last_fullsized":"2024-03-06T07:22:58.824313+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:46.427684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.629679+0000","last_change":"2024-03-06T07:22:57.622627+0000","last_active":"2024-03-06T07:22:58.629679+0000","last_peered":"2024-03-06T07:22:58.629679+0000","last_clean":"2024-03-06T07:22:58.629679+0000","last_became_active":"2024-03-06T07:22:57.622319+0000","last_became_peered":"2024-03-06T07:22:57.622319+0000","last_unstale":"2024-03-06T07:22:58.629679+0000","last_undegraded":"2024-03-06T07:22:58.629679+0000","last_fullsized":"2024-03-06T07:22:58.629679+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:01.132943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.825063+0000","last_change":"2024-03-06T07:22:57.633886+0000","last_active":"2024-03-06T07:22:58.825063+0000","last_peered":"2024-03-06T07:22:58.825063+0000","last_clean":"2024-03-06T07:22:58.825063+0000","last_became_active":"2024-03-06T07:22:57.633567+0000","last_became_peered":"2024-03-06T07:22:57.633567+0000","last_unstale":"2024-03-06T07:22:58.825063+0000","last_undegraded":"2024-03-06T07:22:58.825063+0000","last_fullsized":"2024-03-06T07:22:58.825063+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:39.710112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824500+0000","last_change":"2024-03-06T07:22:57.637662+0000","last_active":"2024-03-06T07:22:58.824500+0000","last_peered":"2024-03-06T07:22:58.824500+0000","last_clean":"2024-03-06T07:22:58.824500+0000","last_became_active":"2024-03-06T07:22:57.637320+0000","last_became_peered":"2024-03-06T07:22:57.637320+0000","last_unstale":"2024-03-06T07:22:58.824500+0000","last_undegraded":"2024-03-06T07:22:58.824500+0000","last_fullsized":"2024-03-06T07:22:58.824500+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:48.772981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.629048+0000","last_change":"2024-03-06T07:22:57.623809+0000","last_active":"2024-03-06T07:22:58.629048+0000","last_peered":"2024-03-06T07:22:58.629048+0000","last_clean":"2024-03-06T07:22:58.629048+0000","last_became_active":"2024-03-06T07:22:57.623535+0000","last_became_peered":"2024-03-06T07:22:57.623535+0000","last_unstale":"2024-03-06T07:22:58.629048+0000","last_undegraded":"2024-03-06T07:22:58.629048+0000","last_fullsized":"2024-03-06T07:22:58.629048+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:08.011885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634989+0000","last_change":"2024-03-06T07:22:57.635037+0000","last_active":"2024-03-06T07:22:58.634989+0000","last_peered":"2024-03-06T07:22:58.634989+0000","last_clean":"2024-03-06T07:22:58.634989+0000","last_became_active":"2024-03-06T07:22:57.634754+0000","last_became_peered":"2024-03-06T07:22:57.634754+0000","last_unstale":"2024-03-06T07:22:58.634989+0000","last_undegraded":"2024-03-06T07:22:58.634989+0000","last_fullsized":"2024-03-06T07:22:58.634989+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:55.687038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824448+0000","last_change":"2024-03-06T07:22:57.632537+0000","last_active":"2024-03-06T07:22:58.824448+0000","last_peered":"2024-03-06T07:22:58.824448+0000","last_clean":"2024-03-06T07:22:58.824448+0000","last_became_active":"2024-03-06T07:22:57.632230+0000","last_became_peered":"2024-03-06T07:22:57.632230+0000","last_unstale":"2024-03-06T07:22:58.824448+0000","last_undegraded":"2024-03-06T07:22:58.824448+0000","last_fullsized":"2024-03-06T07:22:58.824448+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:43.442566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824734+0000","last_change":"2024-03-06T07:22:57.633314+0000","last_active":"2024-03-06T07:22:58.824734+0000","last_peered":"2024-03-06T07:22:58.824734+0000","last_clean":"2024-03-06T07:22:58.824734+0000","last_became_active":"2024-03-06T07:22:57.633041+0000","last_became_peered":"2024-03-06T07:22:57.633041+0000","last_unstale":"2024-03-06T07:22:58.824734+0000","last_undegraded":"2024-03-06T07:22:58.824734+0000","last_fullsized":"2024-03-06T07:22:58.824734+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:46.034056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824382+0000","last_change":"2024-03-06T07:22:57.632052+0000","last_active":"2024-03-06T07:22:58.824382+0000","last_peered":"2024-03-06T07:22:58.824382+0000","last_clean":"2024-03-06T07:22:58.824382+0000","last_became_active":"2024-03-06T07:22:57.630733+0000","last_became_peered":"2024-03-06T07:22:57.630733+0000","last_unstale":"2024-03-06T07:22:58.824382+0000","last_undegraded":"2024-03-06T07:22:58.824382+0000","last_fullsized":"2024-03-06T07:22:58.824382+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:38.629913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.629783+0000","last_change":"2024-03-06T07:22:57.644158+0000","last_active":"2024-03-06T07:22:58.629783+0000","last_peered":"2024-03-06T07:22:58.629783+0000","last_clean":"2024-03-06T07:22:58.629783+0000","last_became_active":"2024-03-06T07:22:57.643252+0000","last_became_peered":"2024-03-06T07:22:57.643252+0000","last_unstale":"2024-03-06T07:22:58.629783+0000","last_undegraded":"2024-03-06T07:22:58.629783+0000","last_fullsized":"2024-03-06T07:22:58.629783+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:38.286642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.825112+0000","last_change":"2024-03-06T07:22:57.633872+0000","last_active":"2024-03-06T07:22:58.825112+0000","last_peered":"2024-03-06T07:22:58.825112+0000","last_clean":"2024-03-06T07:22:58.825112+0000","last_became_active":"2024-03-06T07:22:57.633442+0000","last_became_peered":"2024-03-06T07:22:57.633442+0000","last_unstale":"2024-03-06T07:22:58.825112+0000","last_undegraded":"2024-03-06T07:22:58.825112+0000","last_fullsized":"2024-03-06T07:22:58.825112+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:34.380891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.823671+0000","last_change":"2024-03-06T07:22:57.624521+0000","last_active":"2024-03-06T07:22:58.823671+0000","last_peered":"2024-03-06T07:22:58.823671+0000","last_clean":"2024-03-06T07:22:58.823671+0000","last_became_active":"2024-03-06T07:22:57.624228+0000","last_became_peered":"2024-03-06T07:22:57.624228+0000","last_unstale":"2024-03-06T07:22:58.823671+0000","last_undegraded":"2024-03-06T07:22:58.823671+0000","last_fullsized":"2024-03-06T07:22:58.823671+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:26.279644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.633931+0000","last_change":"2024-03-06T07:22:57.634006+0000","last_active":"2024-03-06T07:22:58.633931+0000","last_peered":"2024-03-06T07:22:58.633931+0000","last_clean":"2024-03-06T07:22:58.633931+0000","last_became_active":"2024-03-06T07:22:57.633669+0000","last_became_peered":"2024-03-06T07:22:57.633669+0000","last_unstale":"2024-03-06T07:22:58.633931+0000","last_undegraded":"2024-03-06T07:22:58.633931+0000","last_fullsized":"2024-03-06T07:22:58.633931+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:48.906789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.629511+0000","last_change":"2024-03-06T07:22:57.635578+0000","last_active":"2024-03-06T07:22:58.629511+0000","last_peered":"2024-03-06T07:22:58.629511+0000","last_clean":"2024-03-06T07:22:58.629511+0000","last_became_active":"2024-03-06T07:22:57.635275+0000","last_became_peered":"2024-03-06T07:22:57.635275+0000","last_unstale":"2024-03-06T07:22:58.629511+0000","last_undegraded":"2024-03-06T07:22:58.629511+0000","last_fullsized":"2024-03-06T07:22:58.629511+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:25.589698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824343+0000","last_change":"2024-03-06T07:22:57.628862+0000","last_active":"2024-03-06T07:22:58.824343+0000","last_peered":"2024-03-06T07:22:58.824343+0000","last_clean":"2024-03-06T07:22:58.824343+0000","last_became_active":"2024-03-06T07:22:57.628584+0000","last_became_peered":"2024-03-06T07:22:57.628584+0000","last_unstale":"2024-03-06T07:22:58.824343+0000","last_undegraded":"2024-03-06T07:22:58.824343+0000","last_fullsized":"2024-03-06T07:22:58.824343+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:54.432088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824630+0000","last_change":"2024-03-06T07:22:57.643101+0000","last_active":"2024-03-06T07:22:58.824630+0000","last_peered":"2024-03-06T07:22:58.824630+0000","last_clean":"2024-03-06T07:22:58.824630+0000","last_became_active":"2024-03-06T07:22:57.642820+0000","last_became_peered":"2024-03-06T07:22:57.642820+0000","last_unstale":"2024-03-06T07:22:58.824630+0000","last_undegraded":"2024-03-06T07:22:58.824630+0000","last_fullsized":"2024-03-06T07:22:58.824630+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:36.548315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.629221+0000","last_change":"2024-03-06T07:22:57.636081+0000","last_active":"2024-03-06T07:22:58.629221+0000","last_peered":"2024-03-06T07:22:58.629221+0000","last_clean":"2024-03-06T07:22:58.629221+0000","last_became_active":"2024-03-06T07:22:57.635798+0000","last_became_peered":"2024-03-06T07:22:57.635798+0000","last_unstale":"2024-03-06T07:22:58.629221+0000","last_undegraded":"2024-03-06T07:22:58.629221+0000","last_fullsized":"2024-03-06T07:22:58.629221+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:30.203134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.635121+0000","last_change":"2024-03-06T07:22:57.626588+0000","last_active":"2024-03-06T07:22:58.635121+0000","last_peered":"2024-03-06T07:22:58.635121+0000","last_clean":"2024-03-06T07:22:58.635121+0000","last_became_active":"2024-03-06T07:22:57.625992+0000","last_became_peered":"2024-03-06T07:22:57.625992+0000","last_unstale":"2024-03-06T07:22:58.635121+0000","last_undegraded":"2024-03-06T07:22:58.635121+0000","last_fullsized":"2024-03-06T07:22:58.635121+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:30.081287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.629227+0000","last_change":"2024-03-06T07:22:57.636470+0000","last_active":"2024-03-06T07:22:58.629227+0000","last_peered":"2024-03-06T07:22:58.629227+0000","last_clean":"2024-03-06T07:22:58.629227+0000","last_became_active":"2024-03-06T07:22:57.636207+0000","last_became_peered":"2024-03-06T07:22:57.636207+0000","last_unstale":"2024-03-06T07:22:58.629227+0000","last_undegraded":"2024-03-06T07:22:58.629227+0000","last_fullsized":"2024-03-06T07:22:58.629227+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:10.025633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634184+0000","last_change":"2024-03-06T07:22:57.632469+0000","last_active":"2024-03-06T07:22:58.634184+0000","last_peered":"2024-03-06T07:22:58.634184+0000","last_clean":"2024-03-06T07:22:58.634184+0000","last_became_active":"2024-03-06T07:22:57.631771+0000","last_became_peered":"2024-03-06T07:22:57.631771+0000","last_unstale":"2024-03-06T07:22:58.634184+0000","last_undegraded":"2024-03-06T07:22:58.634184+0000","last_fullsized":"2024-03-06T07:22:58.634184+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:31.714843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824054+0000","last_change":"2024-03-06T07:22:57.635044+0000","last_active":"2024-03-06T07:22:58.824054+0000","last_peered":"2024-03-06T07:22:58.824054+0000","last_clean":"2024-03-06T07:22:58.824054+0000","last_became_active":"2024-03-06T07:22:57.634107+0000","last_became_peered":"2024-03-06T07:22:57.634107+0000","last_unstale":"2024-03-06T07:22:58.824054+0000","last_undegraded":"2024-03-06T07:22:58.824054+0000","last_fullsized":"2024-03-06T07:22:58.824054+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:23.298584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824433+0000","last_change":"2024-03-06T07:22:57.633281+0000","last_active":"2024-03-06T07:22:58.824433+0000","last_peered":"2024-03-06T07:22:58.824433+0000","last_clean":"2024-03-06T07:22:58.824433+0000","last_became_active":"2024-03-06T07:22:57.632984+0000","last_became_peered":"2024-03-06T07:22:57.632984+0000","last_unstale":"2024-03-06T07:22:58.824433+0000","last_undegraded":"2024-03-06T07:22:58.824433+0000","last_fullsized":"2024-03-06T07:22:58.824433+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:39.710112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824432+0000","last_change":"2024-03-06T07:22:57.631914+0000","last_active":"2024-03-06T07:22:58.824432+0000","last_peered":"2024-03-06T07:22:58.824432+0000","last_clean":"2024-03-06T07:22:58.824432+0000","last_became_active":"2024-03-06T07:22:57.630752+0000","last_became_peered":"2024-03-06T07:22:57.630752+0000","last_unstale":"2024-03-06T07:22:58.824432+0000","last_undegraded":"2024-03-06T07:22:58.824432+0000","last_fullsized":"2024-03-06T07:22:58.824432+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:42.188842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634867+0000","last_change":"2024-03-06T07:22:57.631461+0000","last_active":"2024-03-06T07:22:58.634867+0000","last_peered":"2024-03-06T07:22:58.634867+0000","last_clean":"2024-03-06T07:22:58.634867+0000","last_became_active":"2024-03-06T07:22:57.631177+0000","last_became_peered":"2024-03-06T07:22:57.631177+0000","last_unstale":"2024-03-06T07:22:58.634867+0000","last_undegraded":"2024-03-06T07:22:58.634867+0000","last_fullsized":"2024-03-06T07:22:58.634867+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:59.182493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.629257+0000","last_change":"2024-03-06T07:22:57.636696+0000","last_active":"2024-03-06T07:22:58.629257+0000","last_peered":"2024-03-06T07:22:58.629257+0000","last_clean":"2024-03-06T07:22:58.629257+0000","last_became_active":"2024-03-06T07:22:57.636414+0000","last_became_peered":"2024-03-06T07:22:57.636414+0000","last_unstale":"2024-03-06T07:22:58.629257+0000","last_undegraded":"2024-03-06T07:22:58.629257+0000","last_fullsized":"2024-03-06T07:22:58.629257+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:06.519659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824307+0000","last_change":"2024-03-06T07:22:57.620687+0000","last_active":"2024-03-06T07:22:58.824307+0000","last_peered":"2024-03-06T07:22:58.824307+0000","last_clean":"2024-03-06T07:22:58.824307+0000","last_became_active":"2024-03-06T07:22:57.620416+0000","last_became_peered":"2024-03-06T07:22:57.620416+0000","last_unstale":"2024-03-06T07:22:58.824307+0000","last_undegraded":"2024-03-06T07:22:58.824307+0000","last_fullsized":"2024-03-06T07:22:58.824307+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:30.943768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634436+0000","last_change":"2024-03-06T07:22:57.622616+0000","last_active":"2024-03-06T07:22:58.634436+0000","last_peered":"2024-03-06T07:22:58.634436+0000","last_clean":"2024-03-06T07:22:58.634436+0000","last_became_active":"2024-03-06T07:22:57.622314+0000","last_became_peered":"2024-03-06T07:22:57.622314+0000","last_unstale":"2024-03-06T07:22:58.634436+0000","last_undegraded":"2024-03-06T07:22:58.634436+0000","last_fullsized":"2024-03-06T07:22:58.634436+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:51.972559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.823693+0000","last_change":"2024-03-06T07:22:57.632401+0000","last_active":"2024-03-06T07:22:58.823693+0000","last_peered":"2024-03-06T07:22:58.823693+0000","last_clean":"2024-03-06T07:22:58.823693+0000","last_became_active":"2024-03-06T07:22:57.632063+0000","last_became_peered":"2024-03-06T07:22:57.632063+0000","last_unstale":"2024-03-06T07:22:58.823693+0000","last_undegraded":"2024-03-06T07:22:58.823693+0000","last_fullsized":"2024-03-06T07:22:58.823693+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:40.323690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.633940+0000","last_change":"2024-03-06T07:22:57.626369+0000","last_active":"2024-03-06T07:22:58.633940+0000","last_peered":"2024-03-06T07:22:58.633940+0000","last_clean":"2024-03-06T07:22:58.633940+0000","last_became_active":"2024-03-06T07:22:57.626083+0000","last_became_peered":"2024-03-06T07:22:57.626083+0000","last_unstale":"2024-03-06T07:22:58.633940+0000","last_undegraded":"2024-03-06T07:22:58.633940+0000","last_fullsized":"2024-03-06T07:22:58.633940+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:14.548525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.823714+0000","last_change":"2024-03-06T07:22:57.626460+0000","last_active":"2024-03-06T07:22:58.823714+0000","last_peered":"2024-03-06T07:22:58.823714+0000","last_clean":"2024-03-06T07:22:58.823714+0000","last_became_active":"2024-03-06T07:22:57.626207+0000","last_became_peered":"2024-03-06T07:22:57.626207+0000","last_unstale":"2024-03-06T07:22:58.823714+0000","last_undegraded":"2024-03-06T07:22:58.823714+0000","last_fullsized":"2024-03-06T07:22:58.823714+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:27.403036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"73'4","reported_seq":18,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.629478+0000","last_change":"2024-03-06T07:22:56.618360+0000","last_active":"2024-03-06T07:22:58.629478+0000","last_peered":"2024-03-06T07:22:58.629478+0000","last_clean":"2024-03-06T07:22:58.629478+0000","last_became_active":"2024-03-06T07:22:56.618085+0000","last_became_peered":"2024-03-06T07:22:56.618085+0000","last_unstale":"2024-03-06T07:22:58.629478+0000","last_undegraded":"2024-03-06T07:22:58.629478+0000","last_fullsized":"2024-03-06T07:22:58.629478+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:56.031916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.823826+0000","last_change":"2024-03-06T07:22:57.632027+0000","last_active":"2024-03-06T07:22:58.823826+0000","last_peered":"2024-03-06T07:22:58.823826+0000","last_clean":"2024-03-06T07:22:58.823826+0000","last_became_active":"2024-03-06T07:22:57.631736+0000","last_became_peered":"2024-03-06T07:22:57.631736+0000","last_unstale":"2024-03-06T07:22:58.823826+0000","last_undegraded":"2024-03-06T07:22:58.823826+0000","last_fullsized":"2024-03-06T07:22:58.823826+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:52.824136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"73'1","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634743+0000","last_change":"2024-03-06T07:22:56.623793+0000","last_active":"2024-03-06T07:22:58.634743+0000","last_peered":"2024-03-06T07:22:58.634743+0000","last_clean":"2024-03-06T07:22:58.634743+0000","last_became_active":"2024-03-06T07:22:56.623471+0000","last_became_peered":"2024-03-06T07:22:56.623471+0000","last_unstale":"2024-03-06T07:22:58.634743+0000","last_undegraded":"2024-03-06T07:22:58.634743+0000","last_fullsized":"2024-03-06T07:22:58.634743+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634934+0000","last_change":"2024-03-06T07:22:57.635705+0000","last_active":"2024-03-06T07:22:58.634934+0000","last_peered":"2024-03-06T07:22:58.634934+0000","last_clean":"2024-03-06T07:22:58.634934+0000","last_became_active":"2024-03-06T07:22:57.635457+0000","last_became_peered":"2024-03-06T07:22:57.635457+0000","last_unstale":"2024-03-06T07:22:58.634934+0000","last_undegraded":"2024-03-06T07:22:58.634934+0000","last_fullsized":"2024-03-06T07:22:58.634934+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:00.705050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634812+0000","last_change":"2024-03-06T07:22:56.614741+0000","last_active":"2024-03-06T07:22:58.634812+0000","last_peered":"2024-03-06T07:22:58.634812+0000","last_clean":"2024-03-06T07:22:58.634812+0000","last_became_active":"2024-03-06T07:22:56.614497+0000","last_became_peered":"2024-03-06T07:22:56.614497+0000","last_unstale":"2024-03-06T07:22:58.634812+0000","last_undegraded":"2024-03-06T07:22:58.634812+0000","last_fullsized":"2024-03-06T07:22:58.634812+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:42.726714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634266+0000","last_change":"2024-03-06T07:22:57.625999+0000","last_active":"2024-03-06T07:22:58.634266+0000","last_peered":"2024-03-06T07:22:58.634266+0000","last_clean":"2024-03-06T07:22:58.634266+0000","last_became_active":"2024-03-06T07:22:57.625719+0000","last_became_peered":"2024-03-06T07:22:57.625719+0000","last_unstale":"2024-03-06T07:22:58.634266+0000","last_undegraded":"2024-03-06T07:22:58.634266+0000","last_fullsized":"2024-03-06T07:22:58.634266+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:51.153627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824909+0000","last_change":"2024-03-06T07:22:56.626144+0000","last_active":"2024-03-06T07:22:58.824909+0000","last_peered":"2024-03-06T07:22:58.824909+0000","last_clean":"2024-03-06T07:22:58.824909+0000","last_became_active":"2024-03-06T07:22:56.625739+0000","last_became_peered":"2024-03-06T07:22:56.625739+0000","last_unstale":"2024-03-06T07:22:58.824909+0000","last_undegraded":"2024-03-06T07:22:58.824909+0000","last_fullsized":"2024-03-06T07:22:58.824909+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:52.264835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.823781+0000","last_change":"2024-03-06T07:22:57.633974+0000","last_active":"2024-03-06T07:22:58.823781+0000","last_peered":"2024-03-06T07:22:58.823781+0000","last_clean":"2024-03-06T07:22:58.823781+0000","last_became_active":"2024-03-06T07:22:57.633687+0000","last_became_peered":"2024-03-06T07:22:57.633687+0000","last_unstale":"2024-03-06T07:22:58.823781+0000","last_undegraded":"2024-03-06T07:22:58.823781+0000","last_fullsized":"2024-03-06T07:22:58.823781+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:49.816828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"73'1","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634378+0000","last_change":"2024-03-06T07:22:56.612153+0000","last_active":"2024-03-06T07:22:58.634378+0000","last_peered":"2024-03-06T07:22:58.634378+0000","last_clean":"2024-03-06T07:22:58.634378+0000","last_became_active":"2024-03-06T07:22:56.611872+0000","last_became_peered":"2024-03-06T07:22:56.611872+0000","last_unstale":"2024-03-06T07:22:58.634378+0000","last_undegraded":"2024-03-06T07:22:58.634378+0000","last_fullsized":"2024-03-06T07:22:58.634378+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:03.251378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824861+0000","last_change":"2024-03-06T07:22:57.632247+0000","last_active":"2024-03-06T07:22:58.824861+0000","last_peered":"2024-03-06T07:22:58.824861+0000","last_clean":"2024-03-06T07:22:58.824861+0000","last_became_active":"2024-03-06T07:22:57.631945+0000","last_became_peered":"2024-03-06T07:22:57.631945+0000","last_unstale":"2024-03-06T07:22:58.824861+0000","last_undegraded":"2024-03-06T07:22:58.824861+0000","last_fullsized":"2024-03-06T07:22:58.824861+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:11.711479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"73'1","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824008+0000","last_change":"2024-03-06T07:22:56.616078+0000","last_active":"2024-03-06T07:22:58.824008+0000","last_peered":"2024-03-06T07:22:58.824008+0000","last_clean":"2024-03-06T07:22:58.824008+0000","last_became_active":"2024-03-06T07:22:56.615754+0000","last_became_peered":"2024-03-06T07:22:56.615754+0000","last_unstale":"2024-03-06T07:22:58.824008+0000","last_undegraded":"2024-03-06T07:22:58.824008+0000","last_fullsized":"2024-03-06T07:22:58.824008+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:53.427081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634640+0000","last_change":"2024-03-06T07:22:57.635125+0000","last_active":"2024-03-06T07:22:58.634640+0000","last_peered":"2024-03-06T07:22:58.634640+0000","last_clean":"2024-03-06T07:22:58.634640+0000","last_became_active":"2024-03-06T07:22:57.634829+0000","last_became_peered":"2024-03-06T07:22:57.634829+0000","last_unstale":"2024-03-06T07:22:58.634640+0000","last_undegraded":"2024-03-06T07:22:58.634640+0000","last_fullsized":"2024-03-06T07:22:58.634640+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:48.772981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.823923+0000","last_change":"2024-03-06T07:22:56.612376+0000","last_active":"2024-03-06T07:22:58.823923+0000","last_peered":"2024-03-06T07:22:58.823923+0000","last_clean":"2024-03-06T07:22:58.823923+0000","last_became_active":"2024-03-06T07:22:56.612127+0000","last_became_peered":"2024-03-06T07:22:56.612127+0000","last_unstale":"2024-03-06T07:22:58.823923+0000","last_undegraded":"2024-03-06T07:22:58.823923+0000","last_fullsized":"2024-03-06T07:22:58.823923+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:12.032942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824921+0000","last_change":"2024-03-06T07:22:57.636772+0000","last_active":"2024-03-06T07:22:58.824921+0000","last_peered":"2024-03-06T07:22:58.824921+0000","last_clean":"2024-03-06T07:22:58.824921+0000","last_became_active":"2024-03-06T07:22:57.636474+0000","last_became_peered":"2024-03-06T07:22:57.636474+0000","last_unstale":"2024-03-06T07:22:58.824921+0000","last_undegraded":"2024-03-06T07:22:58.824921+0000","last_fullsized":"2024-03-06T07:22:58.824921+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:13.011040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.629057+0000","last_change":"2024-03-06T07:22:56.619449+0000","last_active":"2024-03-06T07:22:58.629057+0000","last_peered":"2024-03-06T07:22:58.629057+0000","last_clean":"2024-03-06T07:22:58.629057+0000","last_became_active":"2024-03-06T07:22:56.619182+0000","last_became_peered":"2024-03-06T07:22:56.619182+0000","last_unstale":"2024-03-06T07:22:58.629057+0000","last_undegraded":"2024-03-06T07:22:58.629057+0000","last_fullsized":"2024-03-06T07:22:58.629057+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:43.349682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824203+0000","last_change":"2024-03-06T07:22:57.631217+0000","last_active":"2024-03-06T07:22:58.824203+0000","last_peered":"2024-03-06T07:22:58.824203+0000","last_clean":"2024-03-06T07:22:58.824203+0000","last_became_active":"2024-03-06T07:22:57.630879+0000","last_became_peered":"2024-03-06T07:22:57.630879+0000","last_unstale":"2024-03-06T07:22:58.824203+0000","last_undegraded":"2024-03-06T07:22:58.824203+0000","last_fullsized":"2024-03-06T07:22:58.824203+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:36.548315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634797+0000","last_change":"2024-03-06T07:22:56.623979+0000","last_active":"2024-03-06T07:22:58.634797+0000","last_peered":"2024-03-06T07:22:58.634797+0000","last_clean":"2024-03-06T07:22:58.634797+0000","last_became_active":"2024-03-06T07:22:56.623692+0000","last_became_peered":"2024-03-06T07:22:56.623692+0000","last_unstale":"2024-03-06T07:22:58.634797+0000","last_undegraded":"2024-03-06T07:22:58.634797+0000","last_fullsized":"2024-03-06T07:22:58.634797+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:12.032942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824233+0000","last_change":"2024-03-06T07:22:57.629315+0000","last_active":"2024-03-06T07:22:58.824233+0000","last_peered":"2024-03-06T07:22:58.824233+0000","last_clean":"2024-03-06T07:22:58.824233+0000","last_became_active":"2024-03-06T07:22:57.629010+0000","last_became_peered":"2024-03-06T07:22:57.629010+0000","last_unstale":"2024-03-06T07:22:58.824233+0000","last_undegraded":"2024-03-06T07:22:58.824233+0000","last_fullsized":"2024-03-06T07:22:58.824233+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:54.685052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824539+0000","last_change":"2024-03-06T07:22:56.615557+0000","last_active":"2024-03-06T07:22:58.824539+0000","last_peered":"2024-03-06T07:22:58.824539+0000","last_clean":"2024-03-06T07:22:58.824539+0000","last_became_active":"2024-03-06T07:22:56.615299+0000","last_became_peered":"2024-03-06T07:22:56.615299+0000","last_unstale":"2024-03-06T07:22:58.824539+0000","last_undegraded":"2024-03-06T07:22:58.824539+0000","last_fullsized":"2024-03-06T07:22:58.824539+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:03.445671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.825244+0000","last_change":"2024-03-06T07:22:57.642696+0000","last_active":"2024-03-06T07:22:58.825244+0000","last_peered":"2024-03-06T07:22:58.825244+0000","last_clean":"2024-03-06T07:22:58.825244+0000","last_became_active":"2024-03-06T07:22:57.642201+0000","last_became_peered":"2024-03-06T07:22:57.642201+0000","last_unstale":"2024-03-06T07:22:58.825244+0000","last_undegraded":"2024-03-06T07:22:58.825244+0000","last_fullsized":"2024-03-06T07:22:58.825244+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:46.034056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"73'3","reported_seq":17,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.629474+0000","last_change":"2024-03-06T07:22:56.620401+0000","last_active":"2024-03-06T07:22:58.629474+0000","last_peered":"2024-03-06T07:22:58.629474+0000","last_clean":"2024-03-06T07:22:58.629474+0000","last_became_active":"2024-03-06T07:22:56.620134+0000","last_became_peered":"2024-03-06T07:22:56.620134+0000","last_unstale":"2024-03-06T07:22:58.629474+0000","last_undegraded":"2024-03-06T07:22:58.629474+0000","last_fullsized":"2024-03-06T07:22:58.629474+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:48:07.935360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824042+0000","last_change":"2024-03-06T07:22:57.625109+0000","last_active":"2024-03-06T07:22:58.824042+0000","last_peered":"2024-03-06T07:22:58.824042+0000","last_clean":"2024-03-06T07:22:58.824042+0000","last_became_active":"2024-03-06T07:22:57.624843+0000","last_became_peered":"2024-03-06T07:22:57.624843+0000","last_unstale":"2024-03-06T07:22:58.824042+0000","last_undegraded":"2024-03-06T07:22:58.824042+0000","last_fullsized":"2024-03-06T07:22:58.824042+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:13.011040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"73'6","reported_seq":20,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824704+0000","last_change":"2024-03-06T07:22:56.623474+0000","last_active":"2024-03-06T07:22:58.824704+0000","last_peered":"2024-03-06T07:22:58.824704+0000","last_clean":"2024-03-06T07:22:58.824704+0000","last_became_active":"2024-03-06T07:22:56.623169+0000","last_became_peered":"2024-03-06T07:22:56.623169+0000","last_unstale":"2024-03-06T07:22:58.824704+0000","last_undegraded":"2024-03-06T07:22:58.824704+0000","last_fullsized":"2024-03-06T07:22:58.824704+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:37.241588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824656+0000","last_change":"2024-03-06T07:22:57.635263+0000","last_active":"2024-03-06T07:22:58.824656+0000","last_peered":"2024-03-06T07:22:58.824656+0000","last_clean":"2024-03-06T07:22:58.824656+0000","last_became_active":"2024-03-06T07:22:57.635002+0000","last_became_peered":"2024-03-06T07:22:57.635002+0000","last_unstale":"2024-03-06T07:22:58.824656+0000","last_undegraded":"2024-03-06T07:22:58.824656+0000","last_fullsized":"2024-03-06T07:22:58.824656+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:48.906789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824053+0000","last_change":"2024-03-06T07:22:56.613079+0000","last_active":"2024-03-06T07:22:58.824053+0000","last_peered":"2024-03-06T07:22:58.824053+0000","last_clean":"2024-03-06T07:22:58.824053+0000","last_became_active":"2024-03-06T07:22:56.612788+0000","last_became_peered":"2024-03-06T07:22:56.612788+0000","last_unstale":"2024-03-06T07:22:58.824053+0000","last_undegraded":"2024-03-06T07:22:58.824053+0000","last_fullsized":"2024-03-06T07:22:58.824053+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:10.684793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824648+0000","last_change":"2024-03-06T07:22:57.623532+0000","last_active":"2024-03-06T07:22:58.824648+0000","last_peered":"2024-03-06T07:22:58.824648+0000","last_clean":"2024-03-06T07:22:58.824648+0000","last_became_active":"2024-03-06T07:22:57.623226+0000","last_became_peered":"2024-03-06T07:22:57.623226+0000","last_unstale":"2024-03-06T07:22:58.824648+0000","last_undegraded":"2024-03-06T07:22:58.824648+0000","last_fullsized":"2024-03-06T07:22:58.824648+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:55.687038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634543+0000","last_change":"2024-03-06T07:22:56.615633+0000","last_active":"2024-03-06T07:22:58.634543+0000","last_peered":"2024-03-06T07:22:58.634543+0000","last_clean":"2024-03-06T07:22:58.634543+0000","last_became_active":"2024-03-06T07:22:56.615358+0000","last_became_peered":"2024-03-06T07:22:56.615358+0000","last_unstale":"2024-03-06T07:22:58.634543+0000","last_undegraded":"2024-03-06T07:22:58.634543+0000","last_fullsized":"2024-03-06T07:22:58.634543+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:03.445671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824896+0000","last_change":"2024-03-06T07:22:57.628870+0000","last_active":"2024-03-06T07:22:58.824896+0000","last_peered":"2024-03-06T07:22:58.824896+0000","last_clean":"2024-03-06T07:22:58.824896+0000","last_became_active":"2024-03-06T07:22:57.627870+0000","last_became_peered":"2024-03-06T07:22:57.627870+0000","last_unstale":"2024-03-06T07:22:58.824896+0000","last_undegraded":"2024-03-06T07:22:58.824896+0000","last_fullsized":"2024-03-06T07:22:58.824896+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:40.323690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.629096+0000","last_change":"2024-03-06T07:22:56.615747+0000","last_active":"2024-03-06T07:22:58.629096+0000","last_peered":"2024-03-06T07:22:58.629096+0000","last_clean":"2024-03-06T07:22:58.629096+0000","last_became_active":"2024-03-06T07:22:56.615466+0000","last_became_peered":"2024-03-06T07:22:56.615466+0000","last_unstale":"2024-03-06T07:22:58.629096+0000","last_undegraded":"2024-03-06T07:22:58.629096+0000","last_fullsized":"2024-03-06T07:22:58.629096+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:51.504717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824920+0000","last_change":"2024-03-06T07:22:57.634493+0000","last_active":"2024-03-06T07:22:58.824920+0000","last_peered":"2024-03-06T07:22:58.824920+0000","last_clean":"2024-03-06T07:22:58.824920+0000","last_became_active":"2024-03-06T07:22:57.634215+0000","last_became_peered":"2024-03-06T07:22:57.634215+0000","last_unstale":"2024-03-06T07:22:58.824920+0000","last_undegraded":"2024-03-06T07:22:58.824920+0000","last_fullsized":"2024-03-06T07:22:58.824920+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:27.403036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"73'8","reported_seq":22,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824306+0000","last_change":"2024-03-06T07:22:56.623664+0000","last_active":"2024-03-06T07:22:58.824306+0000","last_peered":"2024-03-06T07:22:58.824306+0000","last_clean":"2024-03-06T07:22:58.824306+0000","last_became_active":"2024-03-06T07:22:56.623344+0000","last_became_peered":"2024-03-06T07:22:56.623344+0000","last_unstale":"2024-03-06T07:22:58.824306+0000","last_undegraded":"2024-03-06T07:22:58.824306+0000","last_fullsized":"2024-03-06T07:22:58.824306+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:37.765804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824859+0000","last_change":"2024-03-06T07:22:57.644383+0000","last_active":"2024-03-06T07:22:58.824859+0000","last_peered":"2024-03-06T07:22:58.824859+0000","last_clean":"2024-03-06T07:22:58.824859+0000","last_became_active":"2024-03-06T07:22:57.643011+0000","last_became_peered":"2024-03-06T07:22:57.643011+0000","last_unstale":"2024-03-06T07:22:58.824859+0000","last_undegraded":"2024-03-06T07:22:58.824859+0000","last_fullsized":"2024-03-06T07:22:58.824859+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:30.943768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"73'1","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.635012+0000","last_change":"2024-03-06T07:22:56.622514+0000","last_active":"2024-03-06T07:22:58.635012+0000","last_peered":"2024-03-06T07:22:58.635012+0000","last_clean":"2024-03-06T07:22:58.635012+0000","last_became_active":"2024-03-06T07:22:56.622187+0000","last_became_peered":"2024-03-06T07:22:56.622187+0000","last_unstale":"2024-03-06T07:22:58.635012+0000","last_undegraded":"2024-03-06T07:22:58.635012+0000","last_fullsized":"2024-03-06T07:22:58.635012+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:37.765804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.629122+0000","last_change":"2024-03-06T07:22:57.631467+0000","last_active":"2024-03-06T07:22:58.629122+0000","last_peered":"2024-03-06T07:22:58.629122+0000","last_clean":"2024-03-06T07:22:58.629122+0000","last_became_active":"2024-03-06T07:22:57.631215+0000","last_became_peered":"2024-03-06T07:22:57.631215+0000","last_unstale":"2024-03-06T07:22:58.629122+0000","last_undegraded":"2024-03-06T07:22:58.629122+0000","last_fullsized":"2024-03-06T07:22:58.629122+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:08.873797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634758+0000","last_change":"2024-03-06T07:22:56.619907+0000","last_active":"2024-03-06T07:22:58.634758+0000","last_peered":"2024-03-06T07:22:58.634758+0000","last_clean":"2024-03-06T07:22:58.634758+0000","last_became_active":"2024-03-06T07:22:56.619637+0000","last_became_peered":"2024-03-06T07:22:56.619637+0000","last_unstale":"2024-03-06T07:22:58.634758+0000","last_undegraded":"2024-03-06T07:22:58.634758+0000","last_fullsized":"2024-03-06T07:22:58.634758+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:37.241588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.629371+0000","last_change":"2024-03-06T07:22:57.625588+0000","last_active":"2024-03-06T07:22:58.629371+0000","last_peered":"2024-03-06T07:22:58.629371+0000","last_clean":"2024-03-06T07:22:58.629371+0000","last_became_active":"2024-03-06T07:22:57.625218+0000","last_became_peered":"2024-03-06T07:22:57.625218+0000","last_unstale":"2024-03-06T07:22:58.629371+0000","last_undegraded":"2024-03-06T07:22:58.629371+0000","last_fullsized":"2024-03-06T07:22:58.629371+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:06.639365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"73'1","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.629547+0000","last_change":"2024-03-06T07:22:56.618771+0000","last_active":"2024-03-06T07:22:58.629547+0000","last_peered":"2024-03-06T07:22:58.629547+0000","last_clean":"2024-03-06T07:22:58.629547+0000","last_became_active":"2024-03-06T07:22:56.618384+0000","last_became_peered":"2024-03-06T07:22:56.618384+0000","last_unstale":"2024-03-06T07:22:58.629547+0000","last_undegraded":"2024-03-06T07:22:58.629547+0000","last_fullsized":"2024-03-06T07:22:58.629547+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:04:26.439416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824228+0000","last_change":"2024-03-06T07:22:57.644130+0000","last_active":"2024-03-06T07:22:58.824228+0000","last_peered":"2024-03-06T07:22:58.824228+0000","last_clean":"2024-03-06T07:22:58.824228+0000","last_became_active":"2024-03-06T07:22:57.642544+0000","last_became_peered":"2024-03-06T07:22:57.642544+0000","last_unstale":"2024-03-06T07:22:58.824228+0000","last_undegraded":"2024-03-06T07:22:58.824228+0000","last_fullsized":"2024-03-06T07:22:58.824228+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:23.298584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824132+0000","last_change":"2024-03-06T07:22:56.622412+0000","last_active":"2024-03-06T07:22:58.824132+0000","last_peered":"2024-03-06T07:22:58.824132+0000","last_clean":"2024-03-06T07:22:58.824132+0000","last_became_active":"2024-03-06T07:22:56.622140+0000","last_became_peered":"2024-03-06T07:22:56.622140+0000","last_unstale":"2024-03-06T07:22:58.824132+0000","last_undegraded":"2024-03-06T07:22:58.824132+0000","last_fullsized":"2024-03-06T07:22:58.824132+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:37.728121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.823939+0000","last_change":"2024-03-06T07:22:57.639832+0000","last_active":"2024-03-06T07:22:58.823939+0000","last_peered":"2024-03-06T07:22:58.823939+0000","last_clean":"2024-03-06T07:22:58.823939+0000","last_became_active":"2024-03-06T07:22:57.639055+0000","last_became_peered":"2024-03-06T07:22:57.639055+0000","last_unstale":"2024-03-06T07:22:58.823939+0000","last_undegraded":"2024-03-06T07:22:58.823939+0000","last_fullsized":"2024-03-06T07:22:58.823939+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:00.140244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824325+0000","last_change":"2024-03-06T07:22:56.619240+0000","last_active":"2024-03-06T07:22:58.824325+0000","last_peered":"2024-03-06T07:22:58.824325+0000","last_clean":"2024-03-06T07:22:58.824325+0000","last_became_active":"2024-03-06T07:22:56.619024+0000","last_became_peered":"2024-03-06T07:22:56.619024+0000","last_unstale":"2024-03-06T07:22:58.824325+0000","last_undegraded":"2024-03-06T07:22:58.824325+0000","last_fullsized":"2024-03-06T07:22:58.824325+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:33.101169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.825190+0000","last_change":"2024-03-06T07:22:57.626085+0000","last_active":"2024-03-06T07:22:58.825190+0000","last_peered":"2024-03-06T07:22:58.825190+0000","last_clean":"2024-03-06T07:22:58.825190+0000","last_became_active":"2024-03-06T07:22:57.625810+0000","last_became_peered":"2024-03-06T07:22:57.625810+0000","last_unstale":"2024-03-06T07:22:58.825190+0000","last_undegraded":"2024-03-06T07:22:58.825190+0000","last_fullsized":"2024-03-06T07:22:58.825190+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:33.166265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.823983+0000","last_change":"2024-03-06T07:22:56.623577+0000","last_active":"2024-03-06T07:22:58.823983+0000","last_peered":"2024-03-06T07:22:58.823983+0000","last_clean":"2024-03-06T07:22:58.823983+0000","last_became_active":"2024-03-06T07:22:56.623267+0000","last_became_peered":"2024-03-06T07:22:56.623267+0000","last_unstale":"2024-03-06T07:22:58.823983+0000","last_undegraded":"2024-03-06T07:22:58.823983+0000","last_fullsized":"2024-03-06T07:22:58.823983+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:09.590972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.825089+0000","last_change":"2024-03-06T07:22:57.631438+0000","last_active":"2024-03-06T07:22:58.825089+0000","last_peered":"2024-03-06T07:22:58.825089+0000","last_clean":"2024-03-06T07:22:58.825089+0000","last_became_active":"2024-03-06T07:22:57.631174+0000","last_became_peered":"2024-03-06T07:22:57.631174+0000","last_unstale":"2024-03-06T07:22:58.825089+0000","last_undegraded":"2024-03-06T07:22:58.825089+0000","last_fullsized":"2024-03-06T07:22:58.825089+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:54.432088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824944+0000","last_change":"2024-03-06T07:22:56.621728+0000","last_active":"2024-03-06T07:22:58.824944+0000","last_peered":"2024-03-06T07:22:58.824944+0000","last_clean":"2024-03-06T07:22:58.824944+0000","last_became_active":"2024-03-06T07:22:56.621425+0000","last_became_peered":"2024-03-06T07:22:56.621425+0000","last_unstale":"2024-03-06T07:22:58.824944+0000","last_undegraded":"2024-03-06T07:22:58.824944+0000","last_fullsized":"2024-03-06T07:22:58.824944+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:50.967553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634051+0000","last_change":"2024-03-06T07:22:57.629119+0000","last_active":"2024-03-06T07:22:58.634051+0000","last_peered":"2024-03-06T07:22:58.634051+0000","last_clean":"2024-03-06T07:22:58.634051+0000","last_became_active":"2024-03-06T07:22:57.628813+0000","last_became_peered":"2024-03-06T07:22:57.628813+0000","last_unstale":"2024-03-06T07:22:58.634051+0000","last_undegraded":"2024-03-06T07:22:58.634051+0000","last_fullsized":"2024-03-06T07:22:58.634051+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:38.733127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824226+0000","last_change":"2024-03-06T07:22:56.619641+0000","last_active":"2024-03-06T07:22:58.824226+0000","last_peered":"2024-03-06T07:22:58.824226+0000","last_clean":"2024-03-06T07:22:58.824226+0000","last_became_active":"2024-03-06T07:22:56.619369+0000","last_became_peered":"2024-03-06T07:22:56.619369+0000","last_unstale":"2024-03-06T07:22:58.824226+0000","last_undegraded":"2024-03-06T07:22:58.824226+0000","last_fullsized":"2024-03-06T07:22:58.824226+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824797+0000","last_change":"2024-03-06T07:22:57.634967+0000","last_active":"2024-03-06T07:22:58.824797+0000","last_peered":"2024-03-06T07:22:58.824797+0000","last_clean":"2024-03-06T07:22:58.824797+0000","last_became_active":"2024-03-06T07:22:57.634367+0000","last_became_peered":"2024-03-06T07:22:57.634367+0000","last_unstale":"2024-03-06T07:22:58.824797+0000","last_undegraded":"2024-03-06T07:22:58.824797+0000","last_fullsized":"2024-03-06T07:22:58.824797+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:30.081287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"74'1","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.616605+0000","last_change":"2024-03-06T07:22:56.609890+0000","last_active":"2024-03-06T07:22:59.616605+0000","last_peered":"2024-03-06T07:22:59.616605+0000","last_clean":"2024-03-06T07:22:59.616605+0000","last_became_active":"2024-03-06T07:22:56.609470+0000","last_became_peered":"2024-03-06T07:22:56.609470+0000","last_unstale":"2024-03-06T07:22:59.616605+0000","last_undegraded":"2024-03-06T07:22:59.616605+0000","last_fullsized":"2024-03-06T07:22:59.616605+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:56.348901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634997+0000","last_change":"2024-03-06T07:22:57.634820+0000","last_active":"2024-03-06T07:22:58.634997+0000","last_peered":"2024-03-06T07:22:58.634997+0000","last_clean":"2024-03-06T07:22:58.634997+0000","last_became_active":"2024-03-06T07:22:57.634538+0000","last_became_peered":"2024-03-06T07:22:57.634538+0000","last_unstale":"2024-03-06T07:22:58.634997+0000","last_undegraded":"2024-03-06T07:22:58.634997+0000","last_fullsized":"2024-03-06T07:22:58.634997+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:30.460111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634974+0000","last_change":"2024-03-06T07:22:56.616457+0000","last_active":"2024-03-06T07:22:58.634974+0000","last_peered":"2024-03-06T07:22:58.634974+0000","last_clean":"2024-03-06T07:22:58.634974+0000","last_became_active":"2024-03-06T07:22:56.616201+0000","last_became_peered":"2024-03-06T07:22:56.616201+0000","last_unstale":"2024-03-06T07:22:58.634974+0000","last_undegraded":"2024-03-06T07:22:58.634974+0000","last_fullsized":"2024-03-06T07:22:58.634974+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:59.678591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824571+0000","last_change":"2024-03-06T07:22:57.630052+0000","last_active":"2024-03-06T07:22:58.824571+0000","last_peered":"2024-03-06T07:22:58.824571+0000","last_clean":"2024-03-06T07:22:58.824571+0000","last_became_active":"2024-03-06T07:22:57.629722+0000","last_became_peered":"2024-03-06T07:22:57.629722+0000","last_unstale":"2024-03-06T07:22:58.824571+0000","last_undegraded":"2024-03-06T07:22:58.824571+0000","last_fullsized":"2024-03-06T07:22:58.824571+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:34.380891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634871+0000","last_change":"2024-03-06T07:22:56.624178+0000","last_active":"2024-03-06T07:22:58.634871+0000","last_peered":"2024-03-06T07:22:58.634871+0000","last_clean":"2024-03-06T07:22:58.634871+0000","last_became_active":"2024-03-06T07:22:56.623706+0000","last_became_peered":"2024-03-06T07:22:56.623706+0000","last_unstale":"2024-03-06T07:22:58.634871+0000","last_undegraded":"2024-03-06T07:22:58.634871+0000","last_fullsized":"2024-03-06T07:22:58.634871+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:47.258382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824530+0000","last_change":"2024-03-06T07:22:57.633736+0000","last_active":"2024-03-06T07:22:58.824530+0000","last_peered":"2024-03-06T07:22:58.824530+0000","last_clean":"2024-03-06T07:22:58.824530+0000","last_became_active":"2024-03-06T07:22:57.633502+0000","last_became_peered":"2024-03-06T07:22:57.633502+0000","last_unstale":"2024-03-06T07:22:58.824530+0000","last_undegraded":"2024-03-06T07:22:58.824530+0000","last_fullsized":"2024-03-06T07:22:58.824530+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:00.705050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.825193+0000","last_change":"2024-03-06T07:22:56.626141+0000","last_active":"2024-03-06T07:22:58.825193+0000","last_peered":"2024-03-06T07:22:58.825193+0000","last_clean":"2024-03-06T07:22:58.825193+0000","last_became_active":"2024-03-06T07:22:56.625760+0000","last_became_peered":"2024-03-06T07:22:56.625760+0000","last_unstale":"2024-03-06T07:22:58.825193+0000","last_undegraded":"2024-03-06T07:22:58.825193+0000","last_fullsized":"2024-03-06T07:22:58.825193+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:03.251378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824711+0000","last_change":"2024-03-06T07:22:57.628631+0000","last_active":"2024-03-06T07:22:58.824711+0000","last_peered":"2024-03-06T07:22:58.824711+0000","last_clean":"2024-03-06T07:22:58.824711+0000","last_became_active":"2024-03-06T07:22:57.627708+0000","last_became_peered":"2024-03-06T07:22:57.627708+0000","last_unstale":"2024-03-06T07:22:58.824711+0000","last_undegraded":"2024-03-06T07:22:58.824711+0000","last_fullsized":"2024-03-06T07:22:58.824711+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:30.460111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.629460+0000","last_change":"2024-03-06T07:22:56.615838+0000","last_active":"2024-03-06T07:22:58.629460+0000","last_peered":"2024-03-06T07:22:58.629460+0000","last_clean":"2024-03-06T07:22:58.629460+0000","last_became_active":"2024-03-06T07:22:56.615546+0000","last_became_peered":"2024-03-06T07:22:56.615546+0000","last_unstale":"2024-03-06T07:22:58.629460+0000","last_undegraded":"2024-03-06T07:22:58.629460+0000","last_fullsized":"2024-03-06T07:22:58.629460+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:52.858792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824476+0000","last_change":"2024-03-06T07:22:57.625874+0000","last_active":"2024-03-06T07:22:58.824476+0000","last_peered":"2024-03-06T07:22:58.824476+0000","last_clean":"2024-03-06T07:22:58.824476+0000","last_became_active":"2024-03-06T07:22:57.625574+0000","last_became_peered":"2024-03-06T07:22:57.625574+0000","last_unstale":"2024-03-06T07:22:58.824476+0000","last_undegraded":"2024-03-06T07:22:58.824476+0000","last_fullsized":"2024-03-06T07:22:58.824476+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:28.600878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824902+0000","last_change":"2024-03-06T07:22:56.620322+0000","last_active":"2024-03-06T07:22:58.824902+0000","last_peered":"2024-03-06T07:22:58.824902+0000","last_clean":"2024-03-06T07:22:58.824902+0000","last_became_active":"2024-03-06T07:22:56.620052+0000","last_became_peered":"2024-03-06T07:22:56.620052+0000","last_unstale":"2024-03-06T07:22:58.824902+0000","last_undegraded":"2024-03-06T07:22:58.824902+0000","last_fullsized":"2024-03-06T07:22:58.824902+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:36.870510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.824410+0000","last_change":"2024-03-06T07:22:57.624542+0000","last_active":"2024-03-06T07:22:58.824410+0000","last_peered":"2024-03-06T07:22:58.824410+0000","last_clean":"2024-03-06T07:22:58.824410+0000","last_became_active":"2024-03-06T07:22:57.624258+0000","last_became_peered":"2024-03-06T07:22:57.624258+0000","last_unstale":"2024-03-06T07:22:58.824410+0000","last_undegraded":"2024-03-06T07:22:58.824410+0000","last_fullsized":"2024-03-06T07:22:58.824410+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:11.689800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.823807+0000","last_change":"2024-03-06T07:22:56.621595+0000","last_active":"2024-03-06T07:22:58.823807+0000","last_peered":"2024-03-06T07:22:58.823807+0000","last_clean":"2024-03-06T07:22:58.823807+0000","last_became_active":"2024-03-06T07:22:56.621316+0000","last_became_peered":"2024-03-06T07:22:56.621316+0000","last_unstale":"2024-03-06T07:22:58.823807+0000","last_undegraded":"2024-03-06T07:22:58.823807+0000","last_fullsized":"2024-03-06T07:22:58.823807+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:40.228473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634980+0000","last_change":"2024-03-06T07:22:57.623523+0000","last_active":"2024-03-06T07:22:58.634980+0000","last_peered":"2024-03-06T07:22:58.634980+0000","last_clean":"2024-03-06T07:22:58.634980+0000","last_became_active":"2024-03-06T07:22:57.623281+0000","last_became_peered":"2024-03-06T07:22:57.623281+0000","last_unstale":"2024-03-06T07:22:58.634980+0000","last_undegraded":"2024-03-06T07:22:58.634980+0000","last_fullsized":"2024-03-06T07:22:58.634980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:34.106175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.825271+0000","last_change":"2024-03-06T07:22:56.616740+0000","last_active":"2024-03-06T07:22:58.825271+0000","last_peered":"2024-03-06T07:22:58.825271+0000","last_clean":"2024-03-06T07:22:58.825271+0000","last_became_active":"2024-03-06T07:22:56.614750+0000","last_became_peered":"2024-03-06T07:22:56.614750+0000","last_unstale":"2024-03-06T07:22:58.825271+0000","last_undegraded":"2024-03-06T07:22:58.825271+0000","last_fullsized":"2024-03-06T07:22:58.825271+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:58.177487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":274,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-06T07:22:58.634425+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:22:58.634425+0000","last_peered":"2024-03-06T07:22:58.634425+0000","last_clean":"2024-03-06T07:22:58.634425+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:22:58.634425+0000","last_undegraded":"2024-03-06T07:22:58.634425+0000","last_fullsized":"2024-03-06T07:22:58.634425+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":974848,"data_stored":1018399,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":917504,"data_stored":429621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27908,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343932,"statfs":{"total":96636764160,"available":96608186368,"internally_reserved":0,"allocated":909312,"data_stored":428043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27924,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343916,"statfs":{"total":96636764160,"available":96608169984,"internally_reserved":0,"allocated":925696,"data_stored":429725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":909312,"data_stored":428591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574883,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32084,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339756,"statfs":{"total":96636764160,"available":96603910144,"internally_reserved":0,"allocated":917504,"data_stored":428751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32072,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339768,"statfs":{"total":96636764160,"available":96603922432,"internally_reserved":0,"allocated":913408,"data_stored":428121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":958464,"data_stored":1011685,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:01.067 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:23:01.067 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:23:01.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:01.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.210+0000 7f671eba0640 1 Processor -- start 2024-03-06T07:23:01.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.211+0000 7f671eba0640 1 -- start start 2024-03-06T07:23:01.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.212+0000 7f671eba0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67180c6b10 0x7f6718155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.212+0000 7f671eba0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f67180bf9d0 0x7f67180c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.212+0000 7f671eba0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67180c2fc0 con 0x7f6718152650 2024-03-06T07:23:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.212+0000 7f671eba0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6718155800 con 0x7f67180bf9d0 2024-03-06T07:23:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.212+0000 7f671eba0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6718158da0 con 0x7f67180c6b10 2024-03-06T07:23:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.212+0000 7f671d13d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f67180bf9d0 0x7f67180c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.212+0000 7f671d13d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f67180bf9d0 0x7f67180c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57688/0 (socket says 172.21.15.1:57688) 2024-03-06T07:23:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.212+0000 7f671d13d640 1 -- 172.21.15.1:0/3821316587 learned_addr learned my addr 172.21.15.1:0/3821316587 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.212+0000 7f6717fff640 1 --2- 172.21.15.1:0/3821316587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67180c6b10 0x7f6718155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.212+0000 7f671d13d640 1 -- 172.21.15.1:0/3821316587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67180c6b10 msgr2=0x7f6718155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.212+0000 7f671d13d640 1 --2- 172.21.15.1:0/3821316587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67180c6b10 0x7f6718155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.212+0000 7f671d13d640 1 -- 172.21.15.1:0/3821316587 >> v1:172.21.15.98:6789/0 conn(0x7f6718152650 legacy=0x7f6718158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.212+0000 7f671d13d640 1 -- 172.21.15.1:0/3821316587 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67180c6690 con 0x7f67180bf9d0 2024-03-06T07:23:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.212+0000 7f6717fff640 1 --2- 172.21.15.1:0/3821316587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67180c6b10 0x7f6718155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:01.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.213+0000 7f671d13d640 1 --2- 172.21.15.1:0/3821316587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67180bf9d0 0x7f67180c1460 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f6710009870 tx=0x7f671002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aaced49336a975b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.214+0000 7f67177fe640 1 -- 172.21.15.1:0/3821316587 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67100317c0 con 0x7f67180bf9d0 2024-03-06T07:23:01.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.214+0000 7f67177fe640 1 -- 172.21.15.1:0/3821316587 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6710037070 con 0x7f67180bf9d0 2024-03-06T07:23:01.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.214+0000 7f67177fe640 1 -- 172.21.15.1:0/3821316587 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710031b80 con 0x7f67180bf9d0 2024-03-06T07:23:01.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.214+0000 7f671eba0640 1 -- 172.21.15.1:0/3821316587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67180bf9d0 msgr2=0x7f67180c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.214+0000 7f671eba0640 1 --2- 172.21.15.1:0/3821316587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67180bf9d0 0x7f67180c1460 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f6710009870 tx=0x7f671002b860 comp rx=0 tx=0).stop 2024-03-06T07:23:01.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.214+0000 7f671eba0640 1 -- 172.21.15.1:0/3821316587 shutdown_connections 2024-03-06T07:23:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.214+0000 7f671eba0640 1 --2- 172.21.15.1:0/3821316587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67180bf9d0 0x7f67180c1460 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.214+0000 7f671eba0640 1 --2- 172.21.15.1:0/3821316587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67180c6b10 0x7f6718155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.214+0000 7f671eba0640 1 -- 172.21.15.1:0/3821316587 >> 172.21.15.1:0/3821316587 conn(0x7f6718066ee0 msgr2=0x7f67180672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.214+0000 7f671eba0640 1 -- 172.21.15.1:0/3821316587 shutdown_connections 2024-03-06T07:23:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.214+0000 7f671eba0640 1 -- 172.21.15.1:0/3821316587 wait complete. 2024-03-06T07:23:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.215+0000 7f671eba0640 1 Processor -- start 2024-03-06T07:23:01.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.215+0000 7f671eba0640 1 -- start start 2024-03-06T07:23:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.216+0000 7f671eba0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67180bf9d0 0x7f6718152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.216+0000 7f671eba0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6718152650 0x7f67181ee730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.216+0000 7f671eba0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6718153180 0x7f67181eec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.216+0000 7f671eba0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67181551b0 con 0x7f6718152650 2024-03-06T07:23:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.216+0000 7f671eba0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67180c6fa0 con 0x7f6718153180 2024-03-06T07:23:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.216+0000 7f671eba0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6718155490 con 0x7f67180bf9d0 2024-03-06T07:23:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.216+0000 7f6717fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6718152650 0x7f67181ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.216+0000 7f6717fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6718152650 0x7f67181ee730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57694/0 (socket says 172.21.15.1:57694) 2024-03-06T07:23:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.216+0000 7f6717fff640 1 -- 172.21.15.1:0/3473986829 learned_addr learned my addr 172.21.15.1:0/3473986829 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.216+0000 7f671c93c640 1 --2- 172.21.15.1:0/3473986829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67180bf9d0 0x7f6718152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.216+0000 7f6717fff640 1 -- 172.21.15.1:0/3473986829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67180bf9d0 msgr2=0x7f6718152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.216+0000 7f6717fff640 1 --2- 172.21.15.1:0/3473986829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67180bf9d0 0x7f6718152c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.217+0000 7f6717fff640 1 -- 172.21.15.1:0/3473986829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6718153180 msgr2=0x7f67181eec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.217+0000 7f6717fff640 1 --2- 172.21.15.1:0/3473986829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6718153180 0x7f67181eec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.217+0000 7f6717fff640 1 -- 172.21.15.1:0/3473986829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671814fc60 con 0x7f6718152650 2024-03-06T07:23:01.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.217+0000 7f6717fff640 1 --2- 172.21.15.1:0/3473986829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6718152650 0x7f67181ee730 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f670800ce00 tx=0x7f670800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.217+0000 7f67157fa640 1 -- 172.21.15.1:0/3473986829 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670800f890 con 0x7f6718152650 2024-03-06T07:23:01.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.217+0000 7f67157fa640 1 -- 172.21.15.1:0/3473986829 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670800fa30 con 0x7f6718152650 2024-03-06T07:23:01.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.217+0000 7f671eba0640 1 -- 172.21.15.1:0/3473986829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67181ef1b0 con 0x7f6718152650 2024-03-06T07:23:01.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.217+0000 7f67157fa640 1 -- 172.21.15.1:0/3473986829 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670800fe00 con 0x7f6718152650 2024-03-06T07:23:01.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.217+0000 7f671eba0640 1 -- 172.21.15.1:0/3473986829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67181516f0 con 0x7f6718152650 2024-03-06T07:23:01.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.218+0000 7f671eba0640 1 -- 172.21.15.1:0/3473986829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6718067fd0 con 0x7f6718152650 2024-03-06T07:23:01.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.218+0000 7f67157fa640 1 -- 172.21.15.1:0/3473986829 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6708018400 con 0x7f6718152650 2024-03-06T07:23:01.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.219+0000 7f67157fa640 1 --2- 172.21.15.1:0/3473986829 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f66f806b4d0 0x7f66f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.219+0000 7f67157fa640 1 -- 172.21.15.1:0/3473986829 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f670808b1c0 con 0x7f6718152650 2024-03-06T07:23:01.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.219+0000 7f671c93c640 1 --2- 172.21.15.1:0/3473986829 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f66f806b4d0 0x7f66f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.219+0000 7f671c93c640 1 --2- 172.21.15.1:0/3473986829 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f66f806b4d0 0x7f66f806d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f67180c42d0 tx=0x7f6700009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.223+0000 7f67157fa640 1 -- 172.21.15.1:0/3473986829 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67181ef1b0 con 0x7f6718152650 2024-03-06T07:23:01.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.471+0000 7f671eba0640 1 -- 172.21.15.1:0/3473986829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f67180b06f0 con 0x7f6718152650 2024-03-06T07:23:01.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.473+0000 7f67157fa640 1 -- 172.21.15.1:0/3473986829 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f6708014030 con 0x7f6718152650 2024-03-06T07:23:01.475 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-06T07:23:01.475 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng== 2024-03-06T07:23:01.475 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-06T07:23:01.475 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-06T07:23:01.475 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:01.475 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-06T07:23:01.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.475+0000 7f671eba0640 1 -- 172.21.15.1:0/3473986829 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f66f806b4d0 msgr2=0x7f66f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.475+0000 7f671eba0640 1 --2- 172.21.15.1:0/3473986829 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f66f806b4d0 0x7f66f806d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f67180c42d0 tx=0x7f6700009040 comp rx=0 tx=0).stop 2024-03-06T07:23:01.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.475+0000 7f671eba0640 1 -- 172.21.15.1:0/3473986829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6718152650 msgr2=0x7f67181ee730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.475+0000 7f671eba0640 1 --2- 172.21.15.1:0/3473986829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6718152650 0x7f67181ee730 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f670800ce00 tx=0x7f670800a650 comp rx=0 tx=0).stop 2024-03-06T07:23:01.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.475+0000 7f671eba0640 1 -- 172.21.15.1:0/3473986829 shutdown_connections 2024-03-06T07:23:01.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.475+0000 7f671eba0640 1 --2- 172.21.15.1:0/3473986829 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f66f806b4d0 0x7f66f806d990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.475+0000 7f671eba0640 1 --2- 172.21.15.1:0/3473986829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6718153180 0x7f67181eec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.475+0000 7f671eba0640 1 --2- 172.21.15.1:0/3473986829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6718152650 0x7f67181ee730 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.475+0000 7f671eba0640 1 --2- 172.21.15.1:0/3473986829 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67180bf9d0 0x7f6718152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.476+0000 7f671eba0640 1 -- 172.21.15.1:0/3473986829 >> 172.21.15.1:0/3473986829 conn(0x7f6718066ee0 msgr2=0x7f67180c5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.476+0000 7f671eba0640 1 -- 172.21.15.1:0/3473986829 shutdown_connections 2024-03-06T07:23:01.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.476+0000 7f671eba0640 1 -- 172.21.15.1:0/3473986829 wait complete. 2024-03-06T07:23:01.492 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:01.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.611+0000 7fcffd8d4640 1 Processor -- start 2024-03-06T07:23:01.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.611+0000 7fcffd8d4640 1 -- start start 2024-03-06T07:23:01.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.612+0000 7fcffd8d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff8150fe0 0x7fcff81513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.612+0000 7fcffd8d4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcff8072f20 0x7fcff806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.612+0000 7fcffd8d4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff80a6140 con 0x7fcff806f2f0 2024-03-06T07:23:01.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.612+0000 7fcffd8d4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcff81526b0 con 0x7fcff8072f20 2024-03-06T07:23:01.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.612+0000 7fcffd8d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff8151aa0 con 0x7fcff8150fe0 2024-03-06T07:23:01.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.612+0000 7fcff6ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff8150fe0 0x7fcff81513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.612+0000 7fcff67fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcff8072f20 0x7fcff806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.612+0000 7fcff77fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fcff806f2f0 0x7fcff8071f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:58420/0 (socket says 172.21.15.1:58420) 2024-03-06T07:23:01.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.612+0000 7fcff67fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcff8072f20 0x7fcff806edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:57808/0 (socket says 172.21.15.1:57808) 2024-03-06T07:23:01.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.612+0000 7fcff77fe640 1 -- 172.21.15.1:0/2042648766 learned_addr learned my addr 172.21.15.1:0/2042648766 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:01.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.612+0000 7fcff6ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff8150fe0 0x7fcff81513c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:47678/0 (socket says 172.21.15.1:47678) 2024-03-06T07:23:01.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.612+0000 7fcff6ffd640 1 -- 172.21.15.1:0/2042648766 >> v2:172.21.15.98:3300/0 conn(0x7fcff8072f20 msgr2=0x7fcff806edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.612+0000 7fcff6ffd640 1 --2- 172.21.15.1:0/2042648766 >> v2:172.21.15.98:3300/0 conn(0x7fcff8072f20 0x7fcff806edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.612+0000 7fcff6ffd640 1 -- 172.21.15.1:0/2042648766 >> v1:172.21.15.98:6789/0 conn(0x7fcff806f2f0 legacy=0x7fcff8071f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.612+0000 7fcff6ffd640 1 -- 172.21.15.1:0/2042648766 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff8151d80 con 0x7fcff8150fe0 2024-03-06T07:23:01.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.612+0000 7fcff67fc640 1 --2- 172.21.15.1:0/2042648766 >> v2:172.21.15.98:3300/0 conn(0x7fcff8072f20 0x7fcff806edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:01.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.613+0000 7fcff6ffd640 1 --2- 172.21.15.1:0/2042648766 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff8150fe0 0x7fcff81513c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80097a0 tx=0x7fcfe802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a0fdbaa4600bc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.613+0000 7fcff5ffb640 1 -- 172.21.15.1:0/2042648766 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe80315a0 con 0x7fcff8150fe0 2024-03-06T07:23:01.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.613+0000 7fcff5ffb640 1 -- 172.21.15.1:0/2042648766 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe8037070 con 0x7fcff8150fe0 2024-03-06T07:23:01.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.613+0000 7fcff5ffb640 1 -- 172.21.15.1:0/2042648766 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8031960 con 0x7fcff8150fe0 2024-03-06T07:23:01.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.614+0000 7fcffd8d4640 1 -- 172.21.15.1:0/2042648766 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff8150fe0 msgr2=0x7fcff81513c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.614+0000 7fcffd8d4640 1 --2- 172.21.15.1:0/2042648766 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff8150fe0 0x7fcff81513c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80097a0 tx=0x7fcfe802b840 comp rx=0 tx=0).stop 2024-03-06T07:23:01.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.614+0000 7fcffd8d4640 1 -- 172.21.15.1:0/2042648766 shutdown_connections 2024-03-06T07:23:01.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.614+0000 7fcffd8d4640 1 --2- 172.21.15.1:0/2042648766 >> v2:172.21.15.98:3300/0 conn(0x7fcff8072f20 0x7fcff806edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.614+0000 7fcffd8d4640 1 --2- 172.21.15.1:0/2042648766 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff8150fe0 0x7fcff81513c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.614+0000 7fcffd8d4640 1 -- 172.21.15.1:0/2042648766 >> 172.21.15.1:0/2042648766 conn(0x7fcff80c6860 msgr2=0x7fcff80c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.614+0000 7fcffd8d4640 1 -- 172.21.15.1:0/2042648766 shutdown_connections 2024-03-06T07:23:01.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.614+0000 7fcffd8d4640 1 -- 172.21.15.1:0/2042648766 wait complete. 2024-03-06T07:23:01.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.616+0000 7fcffd8d4640 1 Processor -- start 2024-03-06T07:23:01.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.616+0000 7fcffd8d4640 1 -- start start 2024-03-06T07:23:01.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.617+0000 7fcffd8d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff806f2f0 0x7fcff8142fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.617+0000 7fcffd8d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff8072f20 0x7fcff81551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.617+0000 7fcffd8d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff8150fe0 0x7fcff8141560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.617+0000 7fcffd8d4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff80726a0 con 0x7fcff8072f20 2024-03-06T07:23:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.617+0000 7fcffd8d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff8152340 con 0x7fcff8150fe0 2024-03-06T07:23:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.617+0000 7fcffd8d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff8072820 con 0x7fcff806f2f0 2024-03-06T07:23:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.617+0000 7fcff67fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff8072f20 0x7fcff81551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.617+0000 7fcff67fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff8072f20 0x7fcff81551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57704/0 (socket says 172.21.15.1:57704) 2024-03-06T07:23:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.617+0000 7fcff67fc640 1 -- 172.21.15.1:0/3025082514 learned_addr learned my addr 172.21.15.1:0/3025082514 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.618+0000 7fcff6ffd640 1 --2- 172.21.15.1:0/3025082514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff806f2f0 0x7fcff8142fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.618+0000 7fcff77fe640 1 --2- 172.21.15.1:0/3025082514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff8150fe0 0x7fcff8141560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.618+0000 7fcff67fc640 1 -- 172.21.15.1:0/3025082514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff806f2f0 msgr2=0x7fcff8142fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.618+0000 7fcff67fc640 1 --2- 172.21.15.1:0/3025082514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff806f2f0 0x7fcff8142fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.618+0000 7fcff67fc640 1 -- 172.21.15.1:0/3025082514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff8150fe0 msgr2=0x7fcff8141560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.618+0000 7fcff67fc640 1 --2- 172.21.15.1:0/3025082514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff8150fe0 0x7fcff8141560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.618+0000 7fcff67fc640 1 -- 172.21.15.1:0/3025082514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff8144b60 con 0x7fcff8072f20 2024-03-06T07:23:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.618+0000 7fcff67fc640 1 --2- 172.21.15.1:0/3025082514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff8072f20 0x7fcff81551a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fcfec00ec10 tx=0x7fcfec00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.618+0000 7fcfdffff640 1 -- 172.21.15.1:0/3025082514 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec017070 con 0x7fcff8072f20 2024-03-06T07:23:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.618+0000 7fcfdffff640 1 -- 172.21.15.1:0/3025082514 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfec00cc30 con 0x7fcff8072f20 2024-03-06T07:23:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.618+0000 7fcfdffff640 1 -- 172.21.15.1:0/3025082514 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec013470 con 0x7fcff8072f20 2024-03-06T07:23:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.618+0000 7fcffd8d4640 1 -- 172.21.15.1:0/3025082514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff8144360 con 0x7fcff8072f20 2024-03-06T07:23:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.618+0000 7fcffd8d4640 1 -- 172.21.15.1:0/3025082514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff81556e0 con 0x7fcff8072f20 2024-03-06T07:23:01.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.619+0000 7fcffd8d4640 1 -- 172.21.15.1:0/3025082514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff8067fd0 con 0x7fcff8072f20 2024-03-06T07:23:01.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.619+0000 7fcfdffff640 1 -- 172.21.15.1:0/3025082514 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcfec01a070 con 0x7fcff8072f20 2024-03-06T07:23:01.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.620+0000 7fcfdffff640 1 --2- 172.21.15.1:0/3025082514 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcfe406b4d0 0x7fcfe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.620+0000 7fcfdffff640 1 -- 172.21.15.1:0/3025082514 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcfec08a280 con 0x7fcff8072f20 2024-03-06T07:23:01.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.620+0000 7fcff6ffd640 1 --2- 172.21.15.1:0/3025082514 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcfe406b4d0 0x7fcfe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.621+0000 7fcff6ffd640 1 --2- 172.21.15.1:0/3025082514 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcfe406b4d0 0x7fcfe406d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8036000 tx=0x7fcfe803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.624+0000 7fcfdffff640 1 -- 172.21.15.1:0/3025082514 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcfec059000 con 0x7fcff8072f20 2024-03-06T07:23:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.843+0000 7fcffd8d4640 1 -- 172.21.15.1:0/3025082514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcff80c7630 con 0x7fcff8072f20 2024-03-06T07:23:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.844+0000 7fcfdffff640 1 -- 172.21.15.1:0/3025082514 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4970 (secure 0 0 0) 0x7fcfec058900 con 0x7fcff8072f20 2024-03-06T07:23:01.846 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:01.846 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":54,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2685225325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2685225325},{"type":"v1","addr":"172.21.15.1:6835","nonce":2685225325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":5842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/4069609682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4069609682},{"type":"v1","addr":"172.21.15.98:6836","nonce":4069609682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":5845,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/114998998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":114998998},{"type":"v1","addr":"172.21.15.98:6837","nonce":114998998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.969135+0000","modified":"2024-03-06T07:22:58.980537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6405},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6405":{"gid":6405,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1494116991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1494116991},{"type":"v1","addr":"172.21.15.1:6834","nonce":1494116991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6405,"qdb_cluster":[6405]},"id":5}]} 2024-03-06T07:23:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 54 2024-03-06T07:23:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.846+0000 7fcffd8d4640 1 -- 172.21.15.1:0/3025082514 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcfe406b4d0 msgr2=0x7fcfe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.846+0000 7fcffd8d4640 1 --2- 172.21.15.1:0/3025082514 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcfe406b4d0 0x7fcfe406d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8036000 tx=0x7fcfe803d040 comp rx=0 tx=0).stop 2024-03-06T07:23:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.846+0000 7fcffd8d4640 1 -- 172.21.15.1:0/3025082514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff8072f20 msgr2=0x7fcff81551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.846+0000 7fcffd8d4640 1 --2- 172.21.15.1:0/3025082514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff8072f20 0x7fcff81551a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fcfec00ec10 tx=0x7fcfec00c520 comp rx=0 tx=0).stop 2024-03-06T07:23:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.846+0000 7fcffd8d4640 1 -- 172.21.15.1:0/3025082514 shutdown_connections 2024-03-06T07:23:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.846+0000 7fcffd8d4640 1 --2- 172.21.15.1:0/3025082514 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcfe406b4d0 0x7fcfe406d990 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.846+0000 7fcffd8d4640 1 --2- 172.21.15.1:0/3025082514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcff8150fe0 0x7fcff8141560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.846+0000 7fcffd8d4640 1 --2- 172.21.15.1:0/3025082514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff8072f20 0x7fcff81551a0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.846+0000 7fcffd8d4640 1 --2- 172.21.15.1:0/3025082514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcff806f2f0 0x7fcff8142fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.846+0000 7fcffd8d4640 1 -- 172.21.15.1:0/3025082514 >> 172.21.15.1:0/3025082514 conn(0x7fcff80c6860 msgr2=0x7fcff80c4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.847+0000 7fcffd8d4640 1 -- 172.21.15.1:0/3025082514 shutdown_connections 2024-03-06T07:23:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:01.847+0000 7fcffd8d4640 1 -- 172.21.15.1:0/3025082514 wait complete. 2024-03-06T07:23:02.266 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 54, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:22:56.969135+0000', 'modified': '2024-03-06T07:22:58.980537+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6405}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6405': {'gid': 6405, 'name': 'c', 'rank': 0, 'incarnation': 52, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6834/1494116991', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1494116991}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1494116991}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6405, 'qdb_cluster': [6405]} 2024-03-06T07:23:02.266 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:02.266 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:23:02.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.367+0000 7ff56433e640 1 Processor -- start 2024-03-06T07:23:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.367+0000 7ff56433e640 1 -- start start 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.367+0000 7ff56433e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff55c152650 0x7ff55c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.367+0000 7ff56433e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff55c0c6b10 0x7ff55c158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.367+0000 7ff56433e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff55c0c2fc0 con 0x7ff55c0bf9d0 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.367+0000 7ff56433e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff55c155800 con 0x7ff55c152650 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.367+0000 7ff56433e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c0c6690 con 0x7ff55c0c6b10 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.368+0000 7ff5620da640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff55c152650 0x7ff55c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.368+0000 7ff5628db640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff55c0bf9d0 0x7ff55c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53616/0 (socket says 172.21.15.1:53616) 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.368+0000 7ff5628db640 1 -- 172.21.15.1:0/3946540502 learned_addr learned my addr 172.21.15.1:0/3946540502 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.368+0000 7ff5618d9640 1 --2- 172.21.15.1:0/3946540502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff55c0c6b10 0x7ff55c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.368+0000 7ff5620da640 1 -- 172.21.15.1:0/3946540502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff55c0c6b10 msgr2=0x7ff55c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.368+0000 7ff5620da640 1 --2- 172.21.15.1:0/3946540502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff55c0c6b10 0x7ff55c158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.368+0000 7ff5620da640 1 -- 172.21.15.1:0/3946540502 >> v1:172.21.15.1:6789/0 conn(0x7ff55c0bf9d0 legacy=0x7ff55c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.368+0000 7ff5610d8640 1 -- 172.21.15.1:0/3946540502 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587798374 0 0) 0x7ff55c0c2fc0 con 0x7ff55c0bf9d0 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.368+0000 7ff5620da640 1 -- 172.21.15.1:0/3946540502 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55c0c7280 con 0x7ff55c152650 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.368+0000 7ff5618d9640 1 --2- 172.21.15.1:0/3946540502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff55c0c6b10 0x7ff55c158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.369+0000 7ff5620da640 1 --2- 172.21.15.1:0/3946540502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff55c152650 0x7ff55c155c90 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff54c009d90 tx=0x7ff54c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16291132ebac6a20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.369+0000 7ff5610d8640 1 -- 172.21.15.1:0/3946540502 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c039070 con 0x7ff55c152650 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.369+0000 7ff5610d8640 1 -- 172.21.15.1:0/3946540502 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54c02cbc0 con 0x7ff55c152650 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.369+0000 7ff5610d8640 1 -- 172.21.15.1:0/3946540502 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c034440 con 0x7ff55c152650 2024-03-06T07:23:02.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.369+0000 7ff56433e640 1 -- 172.21.15.1:0/3946540502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff55c152650 msgr2=0x7ff55c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.369+0000 7ff56433e640 1 --2- 172.21.15.1:0/3946540502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff55c152650 0x7ff55c155c90 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff54c009d90 tx=0x7ff54c02be10 comp rx=0 tx=0).stop 2024-03-06T07:23:02.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.370+0000 7ff56433e640 1 -- 172.21.15.1:0/3946540502 shutdown_connections 2024-03-06T07:23:02.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.370+0000 7ff56433e640 1 --2- 172.21.15.1:0/3946540502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff55c0c6b10 0x7ff55c158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.370+0000 7ff56433e640 1 --2- 172.21.15.1:0/3946540502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff55c152650 0x7ff55c155c90 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.370+0000 7ff56433e640 1 -- 172.21.15.1:0/3946540502 >> 172.21.15.1:0/3946540502 conn(0x7ff55c066ee0 msgr2=0x7ff55c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.370+0000 7ff56433e640 1 -- 172.21.15.1:0/3946540502 shutdown_connections 2024-03-06T07:23:02.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.370+0000 7ff56433e640 1 -- 172.21.15.1:0/3946540502 wait complete. 2024-03-06T07:23:02.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.371+0000 7ff56433e640 1 Processor -- start 2024-03-06T07:23:02.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.371+0000 7ff56433e640 1 -- start start 2024-03-06T07:23:02.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.371+0000 7ff56433e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff55c0bf9d0 0x7ff55c151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.371+0000 7ff56433e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff55c152650 0x7ff55c149d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.371+0000 7ff56433e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff55c0c9ef0 0x7ff55c14a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.371+0000 7ff56433e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c155490 con 0x7ff55c0bf9d0 2024-03-06T07:23:02.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.371+0000 7ff56433e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c1551b0 con 0x7ff55c0c9ef0 2024-03-06T07:23:02.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.371+0000 7ff56433e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c152a30 con 0x7ff55c152650 2024-03-06T07:23:02.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.371+0000 7ff5620da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff55c0bf9d0 0x7ff55c151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.371+0000 7ff5620da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff55c0bf9d0 0x7ff55c151320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57710/0 (socket says 172.21.15.1:57710) 2024-03-06T07:23:02.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.371+0000 7ff5620da640 1 -- 172.21.15.1:0/3274087078 learned_addr learned my addr 172.21.15.1:0/3274087078 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:02.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.372+0000 7ff5628db640 1 --2- 172.21.15.1:0/3274087078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff55c0c9ef0 0x7ff55c14a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.372+0000 7ff5618d9640 1 --2- 172.21.15.1:0/3274087078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff55c152650 0x7ff55c149d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.372+0000 7ff5620da640 1 -- 172.21.15.1:0/3274087078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff55c152650 msgr2=0x7ff55c149d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.372+0000 7ff5620da640 1 --2- 172.21.15.1:0/3274087078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff55c152650 0x7ff55c149d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.372+0000 7ff5620da640 1 -- 172.21.15.1:0/3274087078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff55c0c9ef0 msgr2=0x7ff55c14a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.372+0000 7ff5620da640 1 --2- 172.21.15.1:0/3274087078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff55c0c9ef0 0x7ff55c14a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.372+0000 7ff5620da640 1 -- 172.21.15.1:0/3274087078 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55c153e20 con 0x7ff55c0bf9d0 2024-03-06T07:23:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.372+0000 7ff5628db640 1 --2- 172.21.15.1:0/3274087078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff55c0c9ef0 0x7ff55c14a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.372+0000 7ff5620da640 1 --2- 172.21.15.1:0/3274087078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff55c0bf9d0 0x7ff55c151320 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7ff54c009d90 tx=0x7ff54c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.372+0000 7ff55affd640 1 -- 172.21.15.1:0/3274087078 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c039050 con 0x7ff55c0bf9d0 2024-03-06T07:23:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.372+0000 7ff55affd640 1 -- 172.21.15.1:0/3274087078 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54c045070 con 0x7ff55c0bf9d0 2024-03-06T07:23:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.372+0000 7ff56433e640 1 -- 172.21.15.1:0/3274087078 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff55c1ee120 con 0x7ff55c0bf9d0 2024-03-06T07:23:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.372+0000 7ff55affd640 1 -- 172.21.15.1:0/3274087078 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c03e850 con 0x7ff55c0bf9d0 2024-03-06T07:23:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.372+0000 7ff56433e640 1 -- 172.21.15.1:0/3274087078 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff55c1ee6e0 con 0x7ff55c0bf9d0 2024-03-06T07:23:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.374+0000 7ff55affd640 1 -- 172.21.15.1:0/3274087078 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff54c03e9f0 con 0x7ff55c0bf9d0 2024-03-06T07:23:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.374+0000 7ff55affd640 1 --2- 172.21.15.1:0/3274087078 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff54806b4d0 0x7ff54806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.374+0000 7ff55affd640 1 -- 172.21.15.1:0/3274087078 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff54c0acfc0 con 0x7ff55c0bf9d0 2024-03-06T07:23:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.374+0000 7ff5618d9640 1 --2- 172.21.15.1:0/3274087078 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff54806b4d0 0x7ff54806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.375+0000 7ff5618d9640 1 --2- 172.21.15.1:0/3274087078 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff54806b4d0 0x7ff54806d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff55c14ec20 tx=0x7ff550008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.376+0000 7ff566ffe740 1 Processor -- start 2024-03-06T07:23:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.376+0000 7ff566ffe740 1 -- start start 2024-03-06T07:23:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.377+0000 7ff566ffe740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cdca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.377+0000 7ff566ffe740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1cdf330 0x5587d1cdcf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.377+0000 7ff566ffe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1cdd4b0 0x5587d1cd9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.377+0000 7ff566ffe740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587d1c436b0 con 0x5587d1cdef50 2024-03-06T07:23:02.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.377+0000 7ff566ffe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587d1beca60 con 0x5587d1cdd4b0 2024-03-06T07:23:02.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.377+0000 7ff566ffe740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5587d1c3f750 con 0x5587d1cdf330 2024-03-06T07:23:02.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.377+0000 7ff5620da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cdca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.377+0000 7ff5620da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cdca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57722/0 (socket says 172.21.15.1:57722) 2024-03-06T07:23:02.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.377+0000 7ff5620da640 1 -- 172.21.15.1:0/93528750 learned_addr learned my addr 172.21.15.1:0/93528750 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:02.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.377+0000 7ff5618d9640 1 --2- 172.21.15.1:0/93528750 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1cdf330 0x5587d1cdcf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.377+0000 7ff5628db640 1 --2- 172.21.15.1:0/93528750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1cdd4b0 0x5587d1cd9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.377+0000 7ff5620da640 1 -- 172.21.15.1:0/93528750 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1cdf330 msgr2=0x5587d1cdcf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.377+0000 7ff5620da640 1 --2- 172.21.15.1:0/93528750 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1cdf330 0x5587d1cdcf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.377+0000 7ff5620da640 1 -- 172.21.15.1:0/93528750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1cdd4b0 msgr2=0x5587d1cd9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.377+0000 7ff5620da640 1 --2- 172.21.15.1:0/93528750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1cdd4b0 0x5587d1cd9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.377+0000 7ff5620da640 1 -- 172.21.15.1:0/93528750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587d1c0af30 con 0x5587d1cdef50 2024-03-06T07:23:02.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.378+0000 7ff5620da640 1 --2- 172.21.15.1:0/93528750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cdca30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ff54c0ab360 tx=0x7ff54c07b180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d826a816468104 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.378+0000 7ff543fff640 1 -- 172.21.15.1:0/93528750 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c07f8b0 con 0x5587d1cdef50 2024-03-06T07:23:02.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.378+0000 7ff543fff640 1 -- 172.21.15.1:0/93528750 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54c03a040 con 0x5587d1cdef50 2024-03-06T07:23:02.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.378+0000 7ff543fff640 1 -- 172.21.15.1:0/93528750 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c083e70 con 0x5587d1cdef50 2024-03-06T07:23:02.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.379+0000 7ff566ffe740 1 -- 172.21.15.1:0/93528750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 msgr2=0x5587d1cdca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.379+0000 7ff566ffe740 1 --2- 172.21.15.1:0/93528750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cdca30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ff54c0ab360 tx=0x7ff54c07b180 comp rx=0 tx=0).stop 2024-03-06T07:23:02.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.379+0000 7ff566ffe740 1 -- 172.21.15.1:0/93528750 shutdown_connections 2024-03-06T07:23:02.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.379+0000 7ff566ffe740 1 --2- 172.21.15.1:0/93528750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1cdd4b0 0x5587d1cd9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.379+0000 7ff566ffe740 1 --2- 172.21.15.1:0/93528750 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1cdf330 0x5587d1cdcf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.379+0000 7ff566ffe740 1 --2- 172.21.15.1:0/93528750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cdca30 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.379+0000 7ff566ffe740 1 -- 172.21.15.1:0/93528750 >> 172.21.15.1:0/93528750 conn(0x5587d1cdb7d0 msgr2=0x5587d1d003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.379+0000 7ff566ffe740 1 -- 172.21.15.1:0/93528750 shutdown_connections 2024-03-06T07:23:02.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.379+0000 7ff566ffe740 1 -- 172.21.15.1:0/93528750 wait complete. 2024-03-06T07:23:02.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.379+0000 7ff566ffe740 1 -- start start 2024-03-06T07:23:02.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.379+0000 7ff566ffe740 1 Processor -- start 2024-03-06T07:23:02.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.380+0000 7ff566ffe740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1d1b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.380+0000 7ff566ffe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1cdf330 0x5587d1d2ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.380+0000 7ff566ffe740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1d2f320 0x5587d1cfbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.380+0000 7ff566ffe740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587d1c436b0 con 0x5587d1cdef50 2024-03-06T07:23:02.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.380+0000 7ff566ffe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587d1bffea0 con 0x5587d1cdf330 2024-03-06T07:23:02.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.380+0000 7ff566ffe740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5587d1d01230 con 0x5587d1d2f320 2024-03-06T07:23:02.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.380+0000 7ff566ffe740 10 client.0 _ll_register_callbacks cb 0x5587d1bff870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:02.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.380+0000 7ff5620da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1d1b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.380+0000 7ff5620da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1d1b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57732/0 (socket says 172.21.15.1:57732) 2024-03-06T07:23:02.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.380+0000 7ff5620da640 1 -- 172.21.15.1:0/3794685362 learned_addr learned my addr 172.21.15.1:0/3794685362 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:02.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.380+0000 7ff5628db640 1 --2- 172.21.15.1:0/3794685362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1d2f320 0x5587d1cfbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.380+0000 7ff5618d9640 1 --2- 172.21.15.1:0/3794685362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1cdf330 0x5587d1d2ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff5620da640 1 -- 172.21.15.1:0/3794685362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1d2f320 msgr2=0x5587d1cfbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff5620da640 1 --2- 172.21.15.1:0/3794685362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1d2f320 0x5587d1cfbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff5620da640 1 -- 172.21.15.1:0/3794685362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1cdf330 msgr2=0x5587d1d2ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff5620da640 1 --2- 172.21.15.1:0/3794685362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1cdf330 0x5587d1d2ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff5620da640 1 -- 172.21.15.1:0/3794685362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5587d1c00270 con 0x5587d1cdef50 2024-03-06T07:23:02.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff5620da640 1 --2- 172.21.15.1:0/3794685362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1d1b210 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7ff54c000f80 tx=0x7ff54c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2965779d638ad9ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff5417fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:23:02.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff5417fa640 1 -- 172.21.15.1:0/3794685362 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c087070 con 0x5587d1cdef50 2024-03-06T07:23:02.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff5417fa640 1 -- 172.21.15.1:0/3794685362 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff54c03ece0 con 0x5587d1cdef50 2024-03-06T07:23:02.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff566ffe740 1 -- 172.21.15.1:0/3794685362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff54c03a040 con 0x5587d1cdef50 2024-03-06T07:23:02.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff5417fa640 1 -- 172.21.15.1:0/3794685362 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff54c083e70 con 0x5587d1cdef50 2024-03-06T07:23:02.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff566ffe740 10 client.6531 fetch_fsmap learned FSMap version 54 2024-03-06T07:23:02.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff566ffe740 1 -- 172.21.15.1:0/3794685362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) v3 -- 0x5587d19582d0 con 0x5587d1cdef50 2024-03-06T07:23:02.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff5417fa640 1 -- 172.21.15.1:0/3794685362 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54c03a040 con 0x5587d1cdef50 2024-03-06T07:23:02.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff5417fa640 1 -- 172.21.15.1:0/3794685362 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 54) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff54c08a030 con 0x5587d1cdef50 2024-03-06T07:23:02.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff566ffe740 10 client.6531 fetch_fsmap finished waiting for FSMap version 54 2024-03-06T07:23:02.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff566ffe740 10 client.6531 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:23:02.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff566ffe740 20 client.6531 populate_metadata read hostname 'smithi001' 2024-03-06T07:23:02.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff566ffe740 1 --2- 172.21.15.1:0/3794685362 >> [v2:172.21.15.1:6832/1494116991,v1:172.21.15.1:6834/1494116991] conn(0x5587d1d01b60 0x5587d1d01f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff566ffe740 4 client.6531 mds_command: new command op to 6405 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:02.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.381+0000 7ff566ffe740 1 -- 172.21.15.1:0/3794685362 --> [v2:172.21.15.1:6832/1494116991,v1:172.21.15.1:6834/1494116991] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5587d1c50bf0 con 0x5587d1d01b60 2024-03-06T07:23:02.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.382+0000 7ff5618d9640 1 --2- 172.21.15.1:0/3794685362 >> [v2:172.21.15.1:6832/1494116991,v1:172.21.15.1:6834/1494116991] conn(0x5587d1d01b60 0x5587d1d01f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.382+0000 7ff5417fa640 10 client.6531 ms_handle_connect on v2:172.21.15.1:6832/1494116991 2024-03-06T07:23:02.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.382+0000 7ff5618d9640 1 --2- 172.21.15.1:0/3794685362 >> [v2:172.21.15.1:6832/1494116991,v1:172.21.15.1:6834/1494116991] conn(0x5587d1d01b60 0x5587d1d01f40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75c1d36d3939e730 server_cookie=874e1e7c1630dbe in_seq=0 out_seq=0 2024-03-06T07:23:02.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.383+0000 7ff5417fa640 1 -- 172.21.15.1:0/3794685362 <== mds.0 v2:172.21.15.1:6832/1494116991 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5587d1c50bf0 con 0x5587d1d01b60 2024-03-06T07:23:02.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.383+0000 7ff5417fa640 10 client.6531 handle_command_reply: tid=0 2024-03-06T07:23:02.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.383+0000 7ff566ffe740 1 client.6531 shutdown 2024-03-06T07:23:02.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.383+0000 7ff566ffe740 1 -- 172.21.15.1:0/3794685362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 msgr2=0x5587d1d1b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.383+0000 7ff566ffe740 1 --2- 172.21.15.1:0/3794685362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1d1b210 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7ff54c000f80 tx=0x7ff54c02be10 comp rx=0 tx=0).stop 2024-03-06T07:23:02.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.383+0000 7ff566ffe740 1 -- 172.21.15.1:0/3794685362 shutdown_connections 2024-03-06T07:23:02.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.383+0000 7ff566ffe740 1 --2- 172.21.15.1:0/3794685362 >> [v2:172.21.15.1:6832/1494116991,v1:172.21.15.1:6834/1494116991] conn(0x5587d1d01b60 0x5587d1d01f40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.383+0000 7ff566ffe740 1 --2- 172.21.15.1:0/3794685362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1d2f320 0x5587d1cfbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.383+0000 7ff566ffe740 1 --2- 172.21.15.1:0/3794685362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1cdf330 0x5587d1d2ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.383+0000 7ff566ffe740 1 --2- 172.21.15.1:0/3794685362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1d1b210 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.383+0000 7ff5417fa640 0 client.6531 ms_handle_reset on v2:172.21.15.1:6832/1494116991 2024-03-06T07:23:02.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.383+0000 7ff566ffe740 1 -- 172.21.15.1:0/3794685362 >> 172.21.15.1:0/3794685362 conn(0x5587d1cdd4b0 msgr2=0x5587d1cd8510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.383+0000 7ff566ffe740 1 -- 172.21.15.1:0/3794685362 shutdown_connections 2024-03-06T07:23:02.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.383+0000 7ff566ffe740 1 -- 172.21.15.1:0/3794685362 wait complete. 2024-03-06T07:23:02.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.383+0000 7ff566ffe740 20 client.6531 trim_cache size 0 max 16384 2024-03-06T07:23:02.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.400+0000 7ff566ffe740 1 Processor -- start 2024-03-06T07:23:02.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.400+0000 7ff566ffe740 1 -- start start 2024-03-06T07:23:02.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff566ffe740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cda750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff566ffe740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1cdf330 0x5587d1cdac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff566ffe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1d2f320 0x5587d1d28a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff566ffe740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c1f6cd0 con 0x5587d1cdef50 2024-03-06T07:23:02.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff566ffe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c0c5d70 con 0x5587d1d2f320 2024-03-06T07:23:02.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff566ffe740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c1f6700 con 0x5587d1cdf330 2024-03-06T07:23:02.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff5620da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cda750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff5620da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cda750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57748/0 (socket says 172.21.15.1:57748) 2024-03-06T07:23:02.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff5620da640 1 -- 172.21.15.1:0/3350858351 learned_addr learned my addr 172.21.15.1:0/3350858351 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:02.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff5618d9640 1 --2- 172.21.15.1:0/3350858351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1cdf330 0x5587d1cdac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff5620da640 1 -- 172.21.15.1:0/3350858351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1cdf330 msgr2=0x5587d1cdac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff5628db640 1 --2- 172.21.15.1:0/3350858351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1d2f320 0x5587d1d28a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff5620da640 1 --2- 172.21.15.1:0/3350858351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1cdf330 0x5587d1cdac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff5620da640 1 -- 172.21.15.1:0/3350858351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1d2f320 msgr2=0x5587d1d28a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff5620da640 1 --2- 172.21.15.1:0/3350858351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1d2f320 0x5587d1d28a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff5620da640 1 -- 172.21.15.1:0/3350858351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587d1d06c90 con 0x5587d1cdef50 2024-03-06T07:23:02.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff5618d9640 1 --2- 172.21.15.1:0/3350858351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1cdf330 0x5587d1cdac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:02.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff5628db640 1 --2- 172.21.15.1:0/3350858351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1d2f320 0x5587d1d28a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:02.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff5620da640 1 --2- 172.21.15.1:0/3350858351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cda750 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7ff54c0ab440 tx=0x7ff54c0a6070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abe9d67c5a5c7305 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff5437fe640 1 -- 172.21.15.1:0/3350858351 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c097020 con 0x5587d1cdef50 2024-03-06T07:23:02.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff5437fe640 1 -- 172.21.15.1:0/3350858351 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54c0a8070 con 0x5587d1cdef50 2024-03-06T07:23:02.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.401+0000 7ff5437fe640 1 -- 172.21.15.1:0/3350858351 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c087070 con 0x5587d1cdef50 2024-03-06T07:23:02.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.402+0000 7ff566ffe740 1 -- 172.21.15.1:0/3350858351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 msgr2=0x5587d1cda750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.402+0000 7ff566ffe740 1 --2- 172.21.15.1:0/3350858351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cda750 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7ff54c0ab440 tx=0x7ff54c0a6070 comp rx=0 tx=0).stop 2024-03-06T07:23:02.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.402+0000 7ff566ffe740 1 -- 172.21.15.1:0/3350858351 shutdown_connections 2024-03-06T07:23:02.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.402+0000 7ff566ffe740 1 --2- 172.21.15.1:0/3350858351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1d2f320 0x5587d1d28a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.402+0000 7ff566ffe740 1 --2- 172.21.15.1:0/3350858351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1cdf330 0x5587d1cdac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.402+0000 7ff566ffe740 1 --2- 172.21.15.1:0/3350858351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cda750 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.402+0000 7ff566ffe740 1 -- 172.21.15.1:0/3350858351 >> 172.21.15.1:0/3350858351 conn(0x5587d1cdd4b0 msgr2=0x5587d1d05e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.402+0000 7ff566ffe740 1 -- 172.21.15.1:0/3350858351 shutdown_connections 2024-03-06T07:23:02.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.402+0000 7ff566ffe740 1 -- 172.21.15.1:0/3350858351 wait complete. 2024-03-06T07:23:02.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.402+0000 7ff566ffe740 1 -- start start 2024-03-06T07:23:02.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.402+0000 7ff566ffe740 1 Processor -- start 2024-03-06T07:23:02.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.403+0000 7ff566ffe740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cdc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.403+0000 7ff566ffe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1cdf330 0x5587d1cf8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.403+0000 7ff566ffe740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1d2f320 0x5587d1d8a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.403+0000 7ff566ffe740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587d1c436b0 con 0x5587d1cdef50 2024-03-06T07:23:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.403+0000 7ff566ffe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587d1bffea0 con 0x5587d1cdf330 2024-03-06T07:23:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.403+0000 7ff566ffe740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5587d1d03030 con 0x5587d1d2f320 2024-03-06T07:23:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.403+0000 7ff566ffe740 10 client.0 _ll_register_callbacks cb 0x7ff554007d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.403+0000 7ff5620da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cdc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.403+0000 7ff5620da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cdc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57762/0 (socket says 172.21.15.1:57762) 2024-03-06T07:23:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.403+0000 7ff5620da640 1 -- 172.21.15.1:0/973403261 learned_addr learned my addr 172.21.15.1:0/973403261 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff5628db640 1 --2- 172.21.15.1:0/973403261 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1d2f320 0x5587d1d8a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff5618d9640 1 --2- 172.21.15.1:0/973403261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1cdf330 0x5587d1cf8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff5620da640 1 -- 172.21.15.1:0/973403261 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1d2f320 msgr2=0x5587d1d8a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff5620da640 1 --2- 172.21.15.1:0/973403261 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1d2f320 0x5587d1d8a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff5620da640 1 -- 172.21.15.1:0/973403261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1cdf330 msgr2=0x5587d1cf8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff5620da640 1 --2- 172.21.15.1:0/973403261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1cdf330 0x5587d1cf8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff5620da640 1 -- 172.21.15.1:0/973403261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5587d1a226b0 con 0x5587d1cdef50 2024-03-06T07:23:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff5618d9640 1 --2- 172.21.15.1:0/973403261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1cdf330 0x5587d1cf8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff5620da640 1 --2- 172.21.15.1:0/973403261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cdc9a0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7ff54c07b460 tx=0x7ff54c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e331356dda943554 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff5417fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:23:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff5417fa640 1 -- 172.21.15.1:0/973403261 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c097070 con 0x5587d1cdef50 2024-03-06T07:23:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff5417fa640 1 -- 172.21.15.1:0/973403261 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff54c09fca0 con 0x5587d1cdef50 2024-03-06T07:23:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff566ffe740 1 -- 172.21.15.1:0/973403261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff54c0a8070 con 0x5587d1cdef50 2024-03-06T07:23:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff5417fa640 1 -- 172.21.15.1:0/973403261 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff54c09fea0 con 0x5587d1cdef50 2024-03-06T07:23:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff566ffe740 10 client.6537 fetch_fsmap learned FSMap version 54 2024-03-06T07:23:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff566ffe740 1 -- 172.21.15.1:0/973403261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) v3 -- 0x5587d1a226b0 con 0x5587d1cdef50 2024-03-06T07:23:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff5417fa640 1 -- 172.21.15.1:0/973403261 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54c0a8070 con 0x5587d1cdef50 2024-03-06T07:23:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff5417fa640 1 -- 172.21.15.1:0/973403261 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 54) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff54c08a040 con 0x5587d1cdef50 2024-03-06T07:23:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff566ffe740 10 client.6537 fetch_fsmap finished waiting for FSMap version 54 2024-03-06T07:23:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff566ffe740 10 client.6537 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:23:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff566ffe740 20 client.6537 populate_metadata read hostname 'smithi001' 2024-03-06T07:23:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff566ffe740 1 --2- 172.21.15.1:0/973403261 >> [v2:172.21.15.1:6832/1494116991,v1:172.21.15.1:6834/1494116991] conn(0x5587d1cf88a0 0x5587d1d8ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff566ffe740 4 client.6537 mds_command: new command op to 6405 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.404+0000 7ff566ffe740 1 -- 172.21.15.1:0/973403261 --> [v2:172.21.15.1:6832/1494116991,v1:172.21.15.1:6834/1494116991] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5587d1c4af40 con 0x5587d1cf88a0 2024-03-06T07:23:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff5618d9640 1 --2- 172.21.15.1:0/973403261 >> [v2:172.21.15.1:6832/1494116991,v1:172.21.15.1:6834/1494116991] conn(0x5587d1cf88a0 0x5587d1d8ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff5618d9640 1 --2- 172.21.15.1:0/973403261 >> [v2:172.21.15.1:6832/1494116991,v1:172.21.15.1:6834/1494116991] conn(0x5587d1cf88a0 0x5587d1d8ab20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4965d2a2d2395c5b server_cookie=3617299afcfd1296 in_seq=0 out_seq=0 2024-03-06T07:23:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff5417fa640 10 client.6537 ms_handle_connect on v2:172.21.15.1:6832/1494116991 2024-03-06T07:23:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff5417fa640 1 -- 172.21.15.1:0/973403261 <== mds.0 v2:172.21.15.1:6832/1494116991 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5587d1c4af40 con 0x5587d1cf88a0 2024-03-06T07:23:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff5417fa640 10 client.6537 handle_command_reply: tid=0 2024-03-06T07:23:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff566ffe740 1 client.6537 shutdown 2024-03-06T07:23:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff566ffe740 1 -- 172.21.15.1:0/973403261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 msgr2=0x5587d1cdc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff566ffe740 1 --2- 172.21.15.1:0/973403261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cdc9a0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7ff54c07b460 tx=0x7ff54c009750 comp rx=0 tx=0).stop 2024-03-06T07:23:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff566ffe740 1 -- 172.21.15.1:0/973403261 shutdown_connections 2024-03-06T07:23:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff566ffe740 1 --2- 172.21.15.1:0/973403261 >> [v2:172.21.15.1:6832/1494116991,v1:172.21.15.1:6834/1494116991] conn(0x5587d1cf88a0 0x5587d1d8ab20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff566ffe740 1 --2- 172.21.15.1:0/973403261 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5587d1d2f320 0x5587d1d8a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff566ffe740 1 --2- 172.21.15.1:0/973403261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587d1cdf330 0x5587d1cf8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff566ffe740 1 --2- 172.21.15.1:0/973403261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5587d1cdef50 0x5587d1cdc9a0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff5417fa640 0 client.6537 ms_handle_reset on v2:172.21.15.1:6832/1494116991 2024-03-06T07:23:02.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff566ffe740 1 -- 172.21.15.1:0/973403261 >> 172.21.15.1:0/973403261 conn(0x5587d1cdd4b0 msgr2=0x5587d1d05e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff566ffe740 1 -- 172.21.15.1:0/973403261 shutdown_connections 2024-03-06T07:23:02.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff566ffe740 1 -- 172.21.15.1:0/973403261 wait complete. 2024-03-06T07:23:02.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.405+0000 7ff566ffe740 20 client.6537 trim_cache size 0 max 16384 2024-03-06T07:23:02.414 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:23:02.414 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:23:02.415 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:23:02.415 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 6405, 2024-03-06T07:23:02.415 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:23:02.415 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:23:02.415 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:23:02.415 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 4.7958337740000001, 2024-03-06T07:23:02.415 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 54, 2024-03-06T07:23:02.415 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 74, 2024-03-06T07:23:02.415 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 74, 2024-03-06T07:23:02.415 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 9.8749168950000001 2024-03-06T07:23:02.415 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:23:02.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.406+0000 7ff56433e640 1 -- 172.21.15.1:0/3274087078 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff54806b4d0 msgr2=0x7ff54806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.406+0000 7ff56433e640 1 --2- 172.21.15.1:0/3274087078 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff54806b4d0 0x7ff54806d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff55c14ec20 tx=0x7ff550008040 comp rx=0 tx=0).stop 2024-03-06T07:23:02.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.406+0000 7ff56433e640 1 -- 172.21.15.1:0/3274087078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff55c0bf9d0 msgr2=0x7ff55c151320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.406+0000 7ff56433e640 1 --2- 172.21.15.1:0/3274087078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff55c0bf9d0 0x7ff55c151320 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7ff54c009d90 tx=0x7ff54c004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.406+0000 7ff56433e640 1 -- 172.21.15.1:0/3274087078 shutdown_connections 2024-03-06T07:23:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.406+0000 7ff56433e640 1 --2- 172.21.15.1:0/3274087078 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff54806b4d0 0x7ff54806d990 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.406+0000 7ff56433e640 1 --2- 172.21.15.1:0/3274087078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff55c0c9ef0 0x7ff55c14a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.406+0000 7ff56433e640 1 --2- 172.21.15.1:0/3274087078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff55c152650 0x7ff55c149d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.406+0000 7ff56433e640 1 --2- 172.21.15.1:0/3274087078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff55c0bf9d0 0x7ff55c151320 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.406+0000 7ff56433e640 1 -- 172.21.15.1:0/3274087078 >> 172.21.15.1:0/3274087078 conn(0x7ff55c066ee0 msgr2=0x7ff55c1506a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.406+0000 7ff56433e640 1 -- 172.21.15.1:0/3274087078 shutdown_connections 2024-03-06T07:23:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.406+0000 7ff56433e640 1 -- 172.21.15.1:0/3274087078 wait complete. 2024-03-06T07:23:02.422 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.521+0000 7f7ce23f6640 1 Processor -- start 2024-03-06T07:23:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.521+0000 7f7ce23f6640 1 -- start start 2024-03-06T07:23:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.522+0000 7f7ce23f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7cdc0bf9d0 0x7f7cdc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.522+0000 7f7ce23f6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cdc0c2fc0 con 0x7f7cdc152650 2024-03-06T07:23:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.522+0000 7f7ce23f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cdc158da0 con 0x7f7cdc0c6b10 2024-03-06T07:23:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.522+0000 7f7ce23f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc155800 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.522+0000 7f7cdbfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7cdc152650 0x7f7cdc158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53630/0 (socket says 172.21.15.1:53630) 2024-03-06T07:23:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.522+0000 7f7cdbfff640 1 -- 172.21.15.1:0/389659843 learned_addr learned my addr 172.21.15.1:0/389659843 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.522+0000 7f7ce0993640 1 --2- 172.21.15.1:0/389659843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7cdc0bf9d0 0x7f7cdc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.522+0000 7f7cdaffd640 1 -- 172.21.15.1:0/389659843 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644217715 0 0) 0x7f7cdc0c2fc0 con 0x7f7cdc152650 2024-03-06T07:23:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.522+0000 7f7cdaffd640 1 -- 172.21.15.1:0/389659843 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cc4003520 con 0x7f7cdc152650 2024-03-06T07:23:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.522+0000 7f7cdaffd640 1 -- 172.21.15.1:0/389659843 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610865840 0 0) 0x7f7cdc158da0 con 0x7f7cdc0c6b10 2024-03-06T07:23:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.522+0000 7f7cdaffd640 1 -- 172.21.15.1:0/389659843 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cdc0c2fc0 con 0x7f7cdc0c6b10 2024-03-06T07:23:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.522+0000 7f7ce0993640 1 -- 172.21.15.1:0/389659843 >> v1:172.21.15.98:6789/0 conn(0x7f7cdc0c6b10 legacy=0x7f7cdc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.522+0000 7f7ce0993640 1 -- 172.21.15.1:0/389659843 >> v1:172.21.15.1:6789/0 conn(0x7f7cdc152650 legacy=0x7f7cdc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.522+0000 7f7ce0993640 1 -- 172.21.15.1:0/389659843 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cdc0c6690 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.523+0000 7f7ce0993640 1 --2- 172.21.15.1:0/389659843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7cdc0bf9d0 0x7f7cdc155c90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009dc0 tx=0x7f7cd402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c30dd946783cb77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.523+0000 7f7cdaffd640 1 -- 172.21.15.1:0/389659843 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4039070 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.523+0000 7f7cdaffd640 1 -- 172.21.15.1:0/389659843 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd402cc40 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.523+0000 7f7cdaffd640 1 -- 172.21.15.1:0/389659843 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4034440 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.524+0000 7f7ce23f6640 1 -- 172.21.15.1:0/389659843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7cdc0bf9d0 msgr2=0x7f7cdc155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.524+0000 7f7ce23f6640 1 --2- 172.21.15.1:0/389659843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7cdc0bf9d0 0x7f7cdc155c90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009dc0 tx=0x7f7cd402be90 comp rx=0 tx=0).stop 2024-03-06T07:23:02.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.524+0000 7f7ce23f6640 1 -- 172.21.15.1:0/389659843 shutdown_connections 2024-03-06T07:23:02.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.524+0000 7f7ce23f6640 1 --2- 172.21.15.1:0/389659843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7cdc0bf9d0 0x7f7cdc155c90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.524+0000 7f7ce23f6640 1 -- 172.21.15.1:0/389659843 >> 172.21.15.1:0/389659843 conn(0x7f7cdc066ee0 msgr2=0x7f7cdc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.524+0000 7f7ce23f6640 1 -- 172.21.15.1:0/389659843 shutdown_connections 2024-03-06T07:23:02.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.524+0000 7f7ce23f6640 1 -- 172.21.15.1:0/389659843 wait complete. 2024-03-06T07:23:02.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.525+0000 7f7ce23f6640 1 Processor -- start 2024-03-06T07:23:02.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.525+0000 7f7ce23f6640 1 -- start start 2024-03-06T07:23:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.526+0000 7f7ce23f6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc0bf9d0 0x7f7cdc0c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.526+0000 7f7ce23f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7cdc152650 0x7f7cdc14fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.526+0000 7f7ce23f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cdc151c10 0x7f7cdc1e15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.526+0000 7f7ce23f6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc1551b0 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.526+0000 7f7ce23f6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc0c6fa0 con 0x7f7cdc151c10 2024-03-06T07:23:02.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.526+0000 7f7ce23f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc155490 con 0x7f7cdc152650 2024-03-06T07:23:02.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.526+0000 7f7cdbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc0bf9d0 0x7f7cdc0c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.526+0000 7f7cdbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc0bf9d0 0x7f7cdc0c9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57772/0 (socket says 172.21.15.1:57772) 2024-03-06T07:23:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.526+0000 7f7cdbfff640 1 -- 172.21.15.1:0/3662040344 learned_addr learned my addr 172.21.15.1:0/3662040344 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.526+0000 7f7cdb7fe640 1 --2- 172.21.15.1:0/3662040344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7cdc152650 0x7f7cdc14fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.526+0000 7f7ce0993640 1 --2- 172.21.15.1:0/3662040344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cdc151c10 0x7f7cdc1e15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.526+0000 7f7cdbfff640 1 -- 172.21.15.1:0/3662040344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7cdc152650 msgr2=0x7f7cdc14fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.526+0000 7f7cdbfff640 1 --2- 172.21.15.1:0/3662040344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7cdc152650 0x7f7cdc14fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.526+0000 7f7cdbfff640 1 -- 172.21.15.1:0/3662040344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cdc151c10 msgr2=0x7f7cdc1e15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.526+0000 7f7cdbfff640 1 --2- 172.21.15.1:0/3662040344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cdc151c10 0x7f7cdc1e15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.526+0000 7f7cdbfff640 1 -- 172.21.15.1:0/3662040344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cdc14b3d0 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.526+0000 7f7cdbfff640 1 --2- 172.21.15.1:0/3662040344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc0bf9d0 0x7f7cdc0c9fa0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00c500 tx=0x7f7ccc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.527+0000 7f7cd8ff9640 1 -- 172.21.15.1:0/3662040344 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc017070 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.527+0000 7f7cd8ff9640 1 -- 172.21.15.1:0/3662040344 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ccc005ce0 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.527+0000 7f7cd8ff9640 1 -- 172.21.15.1:0/3662040344 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc0072f0 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.527+0000 7f7ce23f6640 1 -- 172.21.15.1:0/3662040344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cdc14b130 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.527+0000 7f7ce23f6640 1 -- 172.21.15.1:0/3662040344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cdc1eaf80 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.528+0000 7f7ce23f6640 1 -- 172.21.15.1:0/3662040344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cdc067fd0 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.528+0000 7f7cd8ff9640 1 -- 172.21.15.1:0/3662040344 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ccc01a070 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.528+0000 7f7cd8ff9640 1 --2- 172.21.15.1:0/3662040344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7cc806b500 0x7f7cc806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.529+0000 7f7cd8ff9640 1 -- 172.21.15.1:0/3662040344 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7ccc089640 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.529+0000 7f7cdb7fe640 1 --2- 172.21.15.1:0/3662040344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7cc806b500 0x7f7cc806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.529+0000 7f7cdb7fe640 1 --2- 172.21.15.1:0/3662040344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7cc806b500 0x7f7cc806d9c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0007fd0 tx=0x7f7cd0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.533+0000 7f7cd8ff9640 1 -- 172.21.15.1:0/3662040344 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ccc057cf0 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.753+0000 7f7ce23f6640 1 -- 172.21.15.1:0/3662040344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cdc0b06f0 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.754+0000 7f7cd8ff9640 1 -- 172.21.15.1:0/3662040344 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4970 (secure 0 0 0) 0x7f7ccc05bd90 con 0x7f7cdc0bf9d0 2024-03-06T07:23:02.758 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:02.758 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":54,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2685225325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2685225325},{"type":"v1","addr":"172.21.15.1:6835","nonce":2685225325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":5842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/4069609682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4069609682},{"type":"v1","addr":"172.21.15.98:6836","nonce":4069609682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":5845,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/114998998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":114998998},{"type":"v1","addr":"172.21.15.98:6837","nonce":114998998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.969135+0000","modified":"2024-03-06T07:22:58.980537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6405},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6405":{"gid":6405,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1494116991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1494116991},{"type":"v1","addr":"172.21.15.1:6834","nonce":1494116991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6405,"qdb_cluster":[6405]},"id":5}]} 2024-03-06T07:23:02.759 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 54 2024-03-06T07:23:02.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.756+0000 7f7ce23f6640 1 -- 172.21.15.1:0/3662040344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7cc806b500 msgr2=0x7f7cc806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.756+0000 7f7ce23f6640 1 --2- 172.21.15.1:0/3662040344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7cc806b500 0x7f7cc806d9c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0007fd0 tx=0x7f7cd0009040 comp rx=0 tx=0).stop 2024-03-06T07:23:02.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.756+0000 7f7ce23f6640 1 -- 172.21.15.1:0/3662040344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc0bf9d0 msgr2=0x7f7cdc0c9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.756+0000 7f7ce23f6640 1 --2- 172.21.15.1:0/3662040344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc0bf9d0 0x7f7cdc0c9fa0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00c500 tx=0x7f7ccc00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.756+0000 7f7ce23f6640 1 -- 172.21.15.1:0/3662040344 shutdown_connections 2024-03-06T07:23:02.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.756+0000 7f7ce23f6640 1 --2- 172.21.15.1:0/3662040344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7cc806b500 0x7f7cc806d9c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.756+0000 7f7ce23f6640 1 --2- 172.21.15.1:0/3662040344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cdc151c10 0x7f7cdc1e15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.756+0000 7f7ce23f6640 1 --2- 172.21.15.1:0/3662040344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7cdc152650 0x7f7cdc14fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.756+0000 7f7ce23f6640 1 --2- 172.21.15.1:0/3662040344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc0bf9d0 0x7f7cdc0c9fa0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.756+0000 7f7ce23f6640 1 -- 172.21.15.1:0/3662040344 >> 172.21.15.1:0/3662040344 conn(0x7f7cdc066ee0 msgr2=0x7f7cdc0c3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.756+0000 7f7ce23f6640 1 -- 172.21.15.1:0/3662040344 shutdown_connections 2024-03-06T07:23:02.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.756+0000 7f7ce23f6640 1 -- 172.21.15.1:0/3662040344 wait complete. 2024-03-06T07:23:02.769 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 5, "epoch": 54, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 5, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:22:56.969135+0000", "damaged": [], "data_pools": [ 12 ], "enabled": true, "epoch": 54, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6405": { "addr": "172.21.15.1:6834/1494116991", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 1494116991, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 1494116991, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6405, "incarnation": 52, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 11, "modified": "2024-03-06T07:22:58.980537+0000", "qdb_cluster": [ 6405 ], "qdb_leader": 6405, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6405 } } } ], "standbys": [ { "addr": "172.21.15.1:6835/2685225325", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 2685225325, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 2685225325, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 51, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5827, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6836/4069609682", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 4069609682, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 4069609682, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 51, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5842, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6837/114998998", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 114998998, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 114998998, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 51, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5845, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:23:02.769 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:23:02.769 INFO:tasks.cephfs.test_failover:current = [{'gid': 5827, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/2685225325', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 2685225325}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2685225325}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 51}, {'gid': 5842, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/4069609682', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 4069609682}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 4069609682}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 51}, {'gid': 6405, 'name': 'c', 'rank': 0, 'incarnation': 52, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6834/1494116991', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1494116991}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1494116991}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5845, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/114998998', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 114998998}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 114998998}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 51}] 2024-03-06T07:23:02.770 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.a mds_join_fs cephfs2 2024-03-06T07:23:02.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.872+0000 7ff2cf80e640 1 Processor -- start 2024-03-06T07:23:02.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.872+0000 7ff2cf80e640 1 -- start start 2024-03-06T07:23:02.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.873+0000 7ff2cf80e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff2c8152a30 0x7ff2c80c7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.873+0000 7ff2cf80e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2c806d770 0x7ff2c815fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.873+0000 7ff2cf80e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c8155260 con 0x7ff2c80afcb0 2024-03-06T07:23:02.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.873+0000 7ff2cf80e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2c80c6fb0 con 0x7ff2c8152a30 2024-03-06T07:23:02.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.873+0000 7ff2cf80e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c8155520 con 0x7ff2c806d770 2024-03-06T07:23:02.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.874+0000 7ff2cddab640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2c806d770 0x7ff2c815fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.874+0000 7ff2cd5aa640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff2c8152a30 0x7ff2c80c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.874+0000 7ff2cddab640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2c806d770 0x7ff2c815fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:47766/0 (socket says 172.21.15.1:47766) 2024-03-06T07:23:02.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.874+0000 7ff2cddab640 1 -- 172.21.15.1:0/2752678886 learned_addr learned my addr 172.21.15.1:0/2752678886 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:02.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.874+0000 7ff2cd5aa640 1 -- 172.21.15.1:0/2752678886 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2c806d770 msgr2=0x7ff2c815fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.874+0000 7ff2cd5aa640 1 --2- 172.21.15.1:0/2752678886 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2c806d770 0x7ff2c815fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.874+0000 7ff2cd5aa640 1 -- 172.21.15.1:0/2752678886 >> v1:172.21.15.98:6789/0 conn(0x7ff2c80afcb0 legacy=0x7ff2c806d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.874+0000 7ff2cd5aa640 1 -- 172.21.15.1:0/2752678886 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c8155800 con 0x7ff2c8152a30 2024-03-06T07:23:02.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.874+0000 7ff2cddab640 1 --2- 172.21.15.1:0/2752678886 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2c806d770 0x7ff2c815fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:02.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.875+0000 7ff2cd5aa640 1 --2- 172.21.15.1:0/2752678886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c8152a30 0x7ff2c80c7400 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff2c40097a0 tx=0x7ff2c402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f16d02c111ef5891 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.875+0000 7ff2bffff640 1 -- 172.21.15.1:0/2752678886 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c40315a0 con 0x7ff2c8152a30 2024-03-06T07:23:02.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.875+0000 7ff2bffff640 1 -- 172.21.15.1:0/2752678886 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c4037070 con 0x7ff2c8152a30 2024-03-06T07:23:02.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.875+0000 7ff2bffff640 1 -- 172.21.15.1:0/2752678886 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4031960 con 0x7ff2c8152a30 2024-03-06T07:23:02.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.876+0000 7ff2cf80e640 1 -- 172.21.15.1:0/2752678886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c8152a30 msgr2=0x7ff2c80c7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.876+0000 7ff2cf80e640 1 --2- 172.21.15.1:0/2752678886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c8152a30 0x7ff2c80c7400 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff2c40097a0 tx=0x7ff2c402b840 comp rx=0 tx=0).stop 2024-03-06T07:23:02.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.876+0000 7ff2cf80e640 1 -- 172.21.15.1:0/2752678886 shutdown_connections 2024-03-06T07:23:02.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.876+0000 7ff2cf80e640 1 --2- 172.21.15.1:0/2752678886 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2c806d770 0x7ff2c815fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.876+0000 7ff2cf80e640 1 --2- 172.21.15.1:0/2752678886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c8152a30 0x7ff2c80c7400 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.876+0000 7ff2cf80e640 1 -- 172.21.15.1:0/2752678886 >> 172.21.15.1:0/2752678886 conn(0x7ff2c808b8e0 msgr2=0x7ff2c8066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.876+0000 7ff2cf80e640 1 -- 172.21.15.1:0/2752678886 shutdown_connections 2024-03-06T07:23:02.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.876+0000 7ff2cf80e640 1 -- 172.21.15.1:0/2752678886 wait complete. 2024-03-06T07:23:02.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.878+0000 7ff2cf80e640 1 Processor -- start 2024-03-06T07:23:02.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.878+0000 7ff2cf80e640 1 -- start start 2024-03-06T07:23:02.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.879+0000 7ff2cf80e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2c80afcb0 0x7ff2c8148b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.879+0000 7ff2cf80e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2c8152a30 0x7ff2c814abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.879+0000 7ff2cf80e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c8158c90 0x7ff2c8159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.879+0000 7ff2cf80e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c80c6cc0 con 0x7ff2c80afcb0 2024-03-06T07:23:02.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.879+0000 7ff2cf80e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c80c6b40 con 0x7ff2c8158c90 2024-03-06T07:23:02.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.879+0000 7ff2cf80e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c80c7940 con 0x7ff2c8152a30 2024-03-06T07:23:02.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.879+0000 7ff2cd5aa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2c80afcb0 0x7ff2c8148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.879+0000 7ff2cd5aa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2c80afcb0 0x7ff2c8148b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57782/0 (socket says 172.21.15.1:57782) 2024-03-06T07:23:02.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.879+0000 7ff2cd5aa640 1 -- 172.21.15.1:0/1098422899 learned_addr learned my addr 172.21.15.1:0/1098422899 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:02.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.879+0000 7ff2cddab640 1 --2- 172.21.15.1:0/1098422899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c8158c90 0x7ff2c8159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.879+0000 7ff2ccda9640 1 --2- 172.21.15.1:0/1098422899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2c8152a30 0x7ff2c814abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.879+0000 7ff2cd5aa640 1 -- 172.21.15.1:0/1098422899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2c8152a30 msgr2=0x7ff2c814abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.879+0000 7ff2cd5aa640 1 --2- 172.21.15.1:0/1098422899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2c8152a30 0x7ff2c814abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.879+0000 7ff2cd5aa640 1 -- 172.21.15.1:0/1098422899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c8158c90 msgr2=0x7ff2c8159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.879+0000 7ff2cd5aa640 1 --2- 172.21.15.1:0/1098422899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c8158c90 0x7ff2c8159110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.879+0000 7ff2cd5aa640 1 -- 172.21.15.1:0/1098422899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c80c3fa0 con 0x7ff2c80afcb0 2024-03-06T07:23:02.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.880+0000 7ff2cd5aa640 1 --2- 172.21.15.1:0/1098422899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2c80afcb0 0x7ff2c8148b50 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4034040 tx=0x7ff2c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.880+0000 7ff2bdffb640 1 -- 172.21.15.1:0/1098422899 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4032630 con 0x7ff2c80afcb0 2024-03-06T07:23:02.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.880+0000 7ff2bdffb640 1 -- 172.21.15.1:0/1098422899 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c40327d0 con 0x7ff2c80afcb0 2024-03-06T07:23:02.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.880+0000 7ff2cf80e640 1 -- 172.21.15.1:0/1098422899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c80c32d0 con 0x7ff2c80afcb0 2024-03-06T07:23:02.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.880+0000 7ff2bdffb640 1 -- 172.21.15.1:0/1098422899 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4032bc0 con 0x7ff2c80afcb0 2024-03-06T07:23:02.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.880+0000 7ff2cf80e640 1 -- 172.21.15.1:0/1098422899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c80c4260 con 0x7ff2c80afcb0 2024-03-06T07:23:02.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.881+0000 7ff2cf80e640 1 -- 172.21.15.1:0/1098422899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c8067fd0 con 0x7ff2c80afcb0 2024-03-06T07:23:02.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.882+0000 7ff2bdffb640 1 -- 172.21.15.1:0/1098422899 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2c404d3f0 con 0x7ff2c80afcb0 2024-03-06T07:23:02.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.882+0000 7ff2bdffb640 1 --2- 172.21.15.1:0/1098422899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff2ac06b4d0 0x7ff2ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.882+0000 7ff2bdffb640 1 -- 172.21.15.1:0/1098422899 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff2c402f030 con 0x7ff2c80afcb0 2024-03-06T07:23:02.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.882+0000 7ff2ccda9640 1 --2- 172.21.15.1:0/1098422899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff2ac06b4d0 0x7ff2ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.883+0000 7ff2ccda9640 1 --2- 172.21.15.1:0/1098422899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff2ac06b4d0 0x7ff2ac06d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8149620 tx=0x7ff2b8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:02.890+0000 7ff2bdffb640 1 -- 172.21.15.1:0/1098422899 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2c40ade20 con 0x7ff2c80afcb0 2024-03-06T07:23:02.903 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-06T07:23:02.902+0000 7f78dfb7e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:03.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.071+0000 7ff2cf80e640 1 -- 172.21.15.1:0/1098422899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7ff2c80c5b50 con 0x7ff2c80afcb0 2024-03-06T07:23:03.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.074+0000 7ff2bdffb640 1 -- 172.21.15.1:0/1098422899 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v24)=0 v24) v1 ==== 117+0+0 (secure 0 0 0) 0x7ff2c4040070 con 0x7ff2c80afcb0 2024-03-06T07:23:03.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.076+0000 7ff2cf80e640 1 -- 172.21.15.1:0/1098422899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff2ac06b4d0 msgr2=0x7ff2ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.076+0000 7ff2cf80e640 1 --2- 172.21.15.1:0/1098422899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff2ac06b4d0 0x7ff2ac06d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8149620 tx=0x7ff2b8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:03.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.076+0000 7ff2cf80e640 1 -- 172.21.15.1:0/1098422899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2c80afcb0 msgr2=0x7ff2c8148b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.076+0000 7ff2cf80e640 1 --2- 172.21.15.1:0/1098422899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2c80afcb0 0x7ff2c8148b50 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4034040 tx=0x7ff2c40046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.076+0000 7ff2cf80e640 1 -- 172.21.15.1:0/1098422899 shutdown_connections 2024-03-06T07:23:03.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.076+0000 7ff2cf80e640 1 --2- 172.21.15.1:0/1098422899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff2ac06b4d0 0x7ff2ac06d990 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.076+0000 7ff2cf80e640 1 --2- 172.21.15.1:0/1098422899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c8158c90 0x7ff2c8159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.076+0000 7ff2cf80e640 1 --2- 172.21.15.1:0/1098422899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2c8152a30 0x7ff2c814abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.076+0000 7ff2cf80e640 1 --2- 172.21.15.1:0/1098422899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2c80afcb0 0x7ff2c8148b50 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.076+0000 7ff2cf80e640 1 -- 172.21.15.1:0/1098422899 >> 172.21.15.1:0/1098422899 conn(0x7ff2c808b8e0 msgr2=0x7ff2c8066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.076+0000 7ff2cf80e640 1 -- 172.21.15.1:0/1098422899 shutdown_connections 2024-03-06T07:23:03.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.076+0000 7ff2cf80e640 1 -- 172.21.15.1:0/1098422899 wait complete. 2024-03-06T07:23:03.089 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs2 2024-03-06T07:23:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.191+0000 7f2346abb640 1 Processor -- start 2024-03-06T07:23:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.191+0000 7f2346abb640 1 -- start start 2024-03-06T07:23:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.192+0000 7f2346abb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2340152650 0x7f2340155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.192+0000 7f2346abb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23400c6b10 0x7f2340158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.192+0000 7f2346abb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23400bf9d0 0x7f23400c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.192+0000 7f2346abb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2340155800 con 0x7f23400bf9d0 2024-03-06T07:23:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.192+0000 7f2346abb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23400c2fc0 con 0x7f2340152650 2024-03-06T07:23:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.192+0000 7f2346abb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23400c6690 con 0x7f23400c6b10 2024-03-06T07:23:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.192+0000 7f2345058640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23400bf9d0 0x7f23400c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.192+0000 7f2345058640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23400bf9d0 0x7f23400c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57798/0 (socket says 172.21.15.1:57798) 2024-03-06T07:23:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.192+0000 7f2345058640 1 -- 172.21.15.1:0/3223737818 learned_addr learned my addr 172.21.15.1:0/3223737818 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.192+0000 7f2344857640 1 --2- 172.21.15.1:0/3223737818 >> v2:172.21.15.98:3300/0 conn(0x7f2340152650 0x7f2340155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.192+0000 7f233ffff640 1 --2- 172.21.15.1:0/3223737818 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23400c6b10 0x7f2340158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.192+0000 7f2345058640 1 -- 172.21.15.1:0/3223737818 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23400c6b10 msgr2=0x7f2340158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.192+0000 7f2345058640 1 --2- 172.21.15.1:0/3223737818 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23400c6b10 0x7f2340158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.192+0000 7f2345058640 1 -- 172.21.15.1:0/3223737818 >> v2:172.21.15.98:3300/0 conn(0x7f2340152650 msgr2=0x7f2340155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.192+0000 7f2345058640 1 --2- 172.21.15.1:0/3223737818 >> v2:172.21.15.98:3300/0 conn(0x7f2340152650 0x7f2340155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.192+0000 7f2345058640 1 -- 172.21.15.1:0/3223737818 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23400c7280 con 0x7f23400bf9d0 2024-03-06T07:23:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.192+0000 7f233ffff640 1 --2- 172.21.15.1:0/3223737818 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23400c6b10 0x7f2340158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.193+0000 7f2345058640 1 --2- 172.21.15.1:0/3223737818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23400bf9d0 0x7f23400c1460 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f23400cac90 tx=0x7f233802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe4753ea00a8a2c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.193+0000 7f233f7fe640 1 -- 172.21.15.1:0/3223737818 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233802c9e0 con 0x7f23400bf9d0 2024-03-06T07:23:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.193+0000 7f233f7fe640 1 -- 172.21.15.1:0/3223737818 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f233802cb80 con 0x7f23400bf9d0 2024-03-06T07:23:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.193+0000 7f233f7fe640 1 -- 172.21.15.1:0/3223737818 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23380343f0 con 0x7f23400bf9d0 2024-03-06T07:23:03.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.194+0000 7f2346abb640 1 -- 172.21.15.1:0/3223737818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23400bf9d0 msgr2=0x7f23400c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.194+0000 7f2346abb640 1 --2- 172.21.15.1:0/3223737818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23400bf9d0 0x7f23400c1460 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f23400cac90 tx=0x7f233802be40 comp rx=0 tx=0).stop 2024-03-06T07:23:03.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.194+0000 7f2346abb640 1 -- 172.21.15.1:0/3223737818 shutdown_connections 2024-03-06T07:23:03.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.194+0000 7f2346abb640 1 --2- 172.21.15.1:0/3223737818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23400bf9d0 0x7f23400c1460 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.194+0000 7f2346abb640 1 --2- 172.21.15.1:0/3223737818 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23400c6b10 0x7f2340158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.194+0000 7f2346abb640 1 --2- 172.21.15.1:0/3223737818 >> v2:172.21.15.98:3300/0 conn(0x7f2340152650 0x7f2340155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.194+0000 7f2346abb640 1 -- 172.21.15.1:0/3223737818 >> 172.21.15.1:0/3223737818 conn(0x7f2340066ee0 msgr2=0x7f23400672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.194+0000 7f2346abb640 1 -- 172.21.15.1:0/3223737818 shutdown_connections 2024-03-06T07:23:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.194+0000 7f2346abb640 1 -- 172.21.15.1:0/3223737818 wait complete. 2024-03-06T07:23:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.195+0000 7f2346abb640 1 Processor -- start 2024-03-06T07:23:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.195+0000 7f2346abb640 1 -- start start 2024-03-06T07:23:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2346abb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23400bf9d0 0x7f234014c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2346abb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2340152650 0x7f2340142420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2346abb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2340156490 0x7f234014ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2346abb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2340155490 con 0x7f2340156490 2024-03-06T07:23:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2346abb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23401551b0 con 0x7f23400bf9d0 2024-03-06T07:23:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2346abb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2340152a30 con 0x7f2340152650 2024-03-06T07:23:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2345058640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2340156490 0x7f234014ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2345058640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2340156490 0x7f234014ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57804/0 (socket says 172.21.15.1:57804) 2024-03-06T07:23:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2345058640 1 -- 172.21.15.1:0/233155434 learned_addr learned my addr 172.21.15.1:0/233155434 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2344857640 1 --2- 172.21.15.1:0/233155434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23400bf9d0 0x7f234014c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f233ffff640 1 --2- 172.21.15.1:0/233155434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2340152650 0x7f2340142420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2345058640 1 -- 172.21.15.1:0/233155434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2340152650 msgr2=0x7f2340142420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2345058640 1 --2- 172.21.15.1:0/233155434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2340152650 0x7f2340142420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2345058640 1 -- 172.21.15.1:0/233155434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23400bf9d0 msgr2=0x7f234014c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2345058640 1 --2- 172.21.15.1:0/233155434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23400bf9d0 0x7f234014c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2345058640 1 -- 172.21.15.1:0/233155434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23400ca370 con 0x7f2340156490 2024-03-06T07:23:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f233ffff640 1 --2- 172.21.15.1:0/233155434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2340152650 0x7f2340142420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2345058640 1 --2- 172.21.15.1:0/233155434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2340156490 0x7f234014ce50 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f2338036040 tx=0x7f2338004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f233d7fa640 1 -- 172.21.15.1:0/233155434 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2338043020 con 0x7f2340156490 2024-03-06T07:23:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f233d7fa640 1 -- 172.21.15.1:0/233155434 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f233803e450 con 0x7f2340156490 2024-03-06T07:23:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2346abb640 1 -- 172.21.15.1:0/233155434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2340152fb0 con 0x7f2340156490 2024-03-06T07:23:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f233d7fa640 1 -- 172.21.15.1:0/233155434 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2338009510 con 0x7f2340156490 2024-03-06T07:23:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.196+0000 7f2346abb640 1 -- 172.21.15.1:0/233155434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2340153570 con 0x7f2340156490 2024-03-06T07:23:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.197+0000 7f2346abb640 1 -- 172.21.15.1:0/233155434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2340067fd0 con 0x7f2340156490 2024-03-06T07:23:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.198+0000 7f233d7fa640 1 -- 172.21.15.1:0/233155434 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2338046020 con 0x7f2340156490 2024-03-06T07:23:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.198+0000 7f233d7fa640 1 --2- 172.21.15.1:0/233155434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f232406b500 0x7f232406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.199+0000 7f233d7fa640 1 -- 172.21.15.1:0/233155434 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f23380ac730 con 0x7f2340156490 2024-03-06T07:23:03.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.199+0000 7f2344857640 1 --2- 172.21.15.1:0/233155434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f232406b500 0x7f232406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.199+0000 7f2344857640 1 --2- 172.21.15.1:0/233155434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f232406b500 0x7f232406d9c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2334009b40 tx=0x7f2334009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.203+0000 7f233d7fa640 1 -- 172.21.15.1:0/233155434 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f233804d070 con 0x7f2340156490 2024-03-06T07:23:03.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.383+0000 7f2346abb640 1 -- 172.21.15.1:0/233155434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f23400b06f0 con 0x7f2340156490 2024-03-06T07:23:03.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.386+0000 7f233d7fa640 1 -- 172.21.15.1:0/233155434 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v25)=0 v25) v1 ==== 117+0+0 (secure 0 0 0) 0x7f233807b450 con 0x7f2340156490 2024-03-06T07:23:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.388+0000 7f2346abb640 1 -- 172.21.15.1:0/233155434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f232406b500 msgr2=0x7f232406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.388+0000 7f2346abb640 1 --2- 172.21.15.1:0/233155434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f232406b500 0x7f232406d9c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2334009b40 tx=0x7f2334009190 comp rx=0 tx=0).stop 2024-03-06T07:23:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.388+0000 7f2346abb640 1 -- 172.21.15.1:0/233155434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2340156490 msgr2=0x7f234014ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.388+0000 7f2346abb640 1 --2- 172.21.15.1:0/233155434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2340156490 0x7f234014ce50 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f2338036040 tx=0x7f2338004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.389+0000 7f2346abb640 1 -- 172.21.15.1:0/233155434 shutdown_connections 2024-03-06T07:23:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.389+0000 7f2346abb640 1 --2- 172.21.15.1:0/233155434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f232406b500 0x7f232406d9c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.389+0000 7f2346abb640 1 --2- 172.21.15.1:0/233155434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2340156490 0x7f234014ce50 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.389+0000 7f2346abb640 1 --2- 172.21.15.1:0/233155434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2340152650 0x7f2340142420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.389+0000 7f2346abb640 1 --2- 172.21.15.1:0/233155434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23400bf9d0 0x7f234014c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.389+0000 7f2346abb640 1 -- 172.21.15.1:0/233155434 >> 172.21.15.1:0/233155434 conn(0x7f2340066ee0 msgr2=0x7f23400bffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.389+0000 7f2346abb640 1 -- 172.21.15.1:0/233155434 shutdown_connections 2024-03-06T07:23:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.389+0000 7f2346abb640 1 -- 172.21.15.1:0/233155434 wait complete. 2024-03-06T07:23:03.401 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs2 2024-03-06T07:23:03.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.504+0000 7f5a6dc95640 1 Processor -- start 2024-03-06T07:23:03.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.504+0000 7f5a6dc95640 1 -- start start 2024-03-06T07:23:03.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.504+0000 7f5a6dc95640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a68152690 0x7f5a68155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.504+0000 7f5a6dc95640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5a680c6730 0x7f5a680c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.504+0000 7f5a6dc95640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a681561e0 con 0x7f5a680c5d10 2024-03-06T07:23:03.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.504+0000 7f5a6dc95640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a680c3300 con 0x7f5a680c6730 2024-03-06T07:23:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.504+0000 7f5a6dc95640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a680c35a0 con 0x7f5a68152690 2024-03-06T07:23:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.505+0000 7f5a66ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5a680c6730 0x7f5a680c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.505+0000 7f5a66ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5a680c6730 0x7f5a680c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57806/0 (socket says 172.21.15.1:57806) 2024-03-06T07:23:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.505+0000 7f5a66ffd640 1 -- 172.21.15.1:0/185803403 learned_addr learned my addr 172.21.15.1:0/185803403 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.505+0000 7f5a677fe640 1 --2- 172.21.15.1:0/185803403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a68152690 0x7f5a68155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.505+0000 7f5a66ffd640 1 -- 172.21.15.1:0/185803403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a68152690 msgr2=0x7f5a68155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.505+0000 7f5a66ffd640 1 --2- 172.21.15.1:0/185803403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a68152690 0x7f5a68155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.505+0000 7f5a66ffd640 1 -- 172.21.15.1:0/185803403 >> v1:172.21.15.98:6789/0 conn(0x7f5a680c5d10 legacy=0x7f5a68156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.505+0000 7f5a66ffd640 1 -- 172.21.15.1:0/185803403 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a680c3880 con 0x7f5a680c6730 2024-03-06T07:23:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.506+0000 7f5a66ffd640 1 --2- 172.21.15.1:0/185803403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a680c6730 0x7f5a680c57d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c009870 tx=0x7f5a5c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bfd6a32849a5186 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.506+0000 7f5a667fc640 1 -- 172.21.15.1:0/185803403 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c035020 con 0x7f5a680c6730 2024-03-06T07:23:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.506+0000 7f5a667fc640 1 -- 172.21.15.1:0/185803403 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a5c031640 con 0x7f5a680c6730 2024-03-06T07:23:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.506+0000 7f5a667fc640 1 -- 172.21.15.1:0/185803403 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c02d040 con 0x7f5a680c6730 2024-03-06T07:23:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.506+0000 7f5a6dc95640 1 -- 172.21.15.1:0/185803403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a680c6730 msgr2=0x7f5a680c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.506+0000 7f5a6dc95640 1 --2- 172.21.15.1:0/185803403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a680c6730 0x7f5a680c57d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c009870 tx=0x7f5a5c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.506+0000 7f5a6dc95640 1 -- 172.21.15.1:0/185803403 shutdown_connections 2024-03-06T07:23:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.506+0000 7f5a6dc95640 1 --2- 172.21.15.1:0/185803403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a680c6730 0x7f5a680c57d0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.506+0000 7f5a6dc95640 1 --2- 172.21.15.1:0/185803403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a68152690 0x7f5a68155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.506+0000 7f5a6dc95640 1 -- 172.21.15.1:0/185803403 >> 172.21.15.1:0/185803403 conn(0x7f5a6806b410 msgr2=0x7f5a6806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.507+0000 7f5a6dc95640 1 -- 172.21.15.1:0/185803403 shutdown_connections 2024-03-06T07:23:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.507+0000 7f5a6dc95640 1 -- 172.21.15.1:0/185803403 wait complete. 2024-03-06T07:23:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.508+0000 7f5a6dc95640 1 Processor -- start 2024-03-06T07:23:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.508+0000 7f5a6dc95640 1 -- start start 2024-03-06T07:23:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.508+0000 7f5a6dc95640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a680c5d10 0x7f5a68150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.508+0000 7f5a6dc95640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a680c6730 0x7f5a6814f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.508+0000 7f5a6dc95640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a68152690 0x7f5a6814fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.508+0000 7f5a6dc95640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a68152a70 con 0x7f5a680c6730 2024-03-06T07:23:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.508+0000 7f5a6dc95640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a681567b0 con 0x7f5a68152690 2024-03-06T07:23:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.508+0000 7f5a6dc95640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a68152bf0 con 0x7f5a680c5d10 2024-03-06T07:23:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.509+0000 7f5a66ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a680c6730 0x7f5a6814f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.509+0000 7f5a66ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a680c6730 0x7f5a6814f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57812/0 (socket says 172.21.15.1:57812) 2024-03-06T07:23:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.509+0000 7f5a66ffd640 1 -- 172.21.15.1:0/3733262644 learned_addr learned my addr 172.21.15.1:0/3733262644 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.509+0000 7f5a677fe640 1 --2- 172.21.15.1:0/3733262644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a680c5d10 0x7f5a68150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.509+0000 7f5a67fff640 1 --2- 172.21.15.1:0/3733262644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a68152690 0x7f5a6814fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.509+0000 7f5a66ffd640 1 -- 172.21.15.1:0/3733262644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a680c5d10 msgr2=0x7f5a68150950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.509+0000 7f5a66ffd640 1 --2- 172.21.15.1:0/3733262644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a680c5d10 0x7f5a68150950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.509+0000 7f5a66ffd640 1 -- 172.21.15.1:0/3733262644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a68152690 msgr2=0x7f5a6814fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.509+0000 7f5a66ffd640 1 --2- 172.21.15.1:0/3733262644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a68152690 0x7f5a6814fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.509+0000 7f5a66ffd640 1 -- 172.21.15.1:0/3733262644 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a681eefe0 con 0x7f5a680c6730 2024-03-06T07:23:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.509+0000 7f5a66ffd640 1 --2- 172.21.15.1:0/3733262644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a680c6730 0x7f5a6814f890 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c000c00 tx=0x7f5a5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.509+0000 7f5a4ffff640 1 -- 172.21.15.1:0/3733262644 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c040070 con 0x7f5a680c6730 2024-03-06T07:23:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.509+0000 7f5a4ffff640 1 -- 172.21.15.1:0/3733262644 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a5c0399f0 con 0x7f5a680c6730 2024-03-06T07:23:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.509+0000 7f5a6dc95640 1 -- 172.21.15.1:0/3733262644 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a681ec370 con 0x7f5a680c6730 2024-03-06T07:23:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.509+0000 7f5a4ffff640 1 -- 172.21.15.1:0/3733262644 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c035040 con 0x7f5a680c6730 2024-03-06T07:23:03.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.509+0000 7f5a6dc95640 1 -- 172.21.15.1:0/3733262644 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a681ec930 con 0x7f5a680c6730 2024-03-06T07:23:03.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.510+0000 7f5a6dc95640 1 -- 172.21.15.1:0/3733262644 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a681ec160 con 0x7f5a680c6730 2024-03-06T07:23:03.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.511+0000 7f5a4ffff640 1 -- 172.21.15.1:0/3733262644 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a5c04b060 con 0x7f5a680c6730 2024-03-06T07:23:03.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.511+0000 7f5a4ffff640 1 --2- 172.21.15.1:0/3733262644 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a5406b4d0 0x7f5a5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.511+0000 7f5a4ffff640 1 -- 172.21.15.1:0/3733262644 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5a5c0aba30 con 0x7f5a680c6730 2024-03-06T07:23:03.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.511+0000 7f5a677fe640 1 --2- 172.21.15.1:0/3733262644 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a5406b4d0 0x7f5a5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.512+0000 7f5a677fe640 1 --2- 172.21.15.1:0/3733262644 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a5406b4d0 0x7f5a5406d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5a58009aa0 tx=0x7f5a58009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.516+0000 7f5a4ffff640 1 -- 172.21.15.1:0/3733262644 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a5c07a0d0 con 0x7f5a680c6730 2024-03-06T07:23:03.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.696+0000 7f5a6dc95640 1 -- 172.21.15.1:0/3733262644 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f5a68067fd0 con 0x7f5a680c6730 2024-03-06T07:23:03.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.698+0000 7f5a4ffff640 1 -- 172.21.15.1:0/3733262644 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v26)=0 v26) v1 ==== 117+0+0 (secure 0 0 0) 0x7f5a5c07e170 con 0x7f5a680c6730 2024-03-06T07:23:03.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.700+0000 7f5a6dc95640 1 -- 172.21.15.1:0/3733262644 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a5406b4d0 msgr2=0x7f5a5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.700+0000 7f5a6dc95640 1 --2- 172.21.15.1:0/3733262644 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a5406b4d0 0x7f5a5406d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5a58009aa0 tx=0x7f5a58009190 comp rx=0 tx=0).stop 2024-03-06T07:23:03.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.700+0000 7f5a6dc95640 1 -- 172.21.15.1:0/3733262644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a680c6730 msgr2=0x7f5a6814f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.700+0000 7f5a6dc95640 1 --2- 172.21.15.1:0/3733262644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a680c6730 0x7f5a6814f890 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c000c00 tx=0x7f5a5c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.700+0000 7f5a6dc95640 1 -- 172.21.15.1:0/3733262644 shutdown_connections 2024-03-06T07:23:03.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.700+0000 7f5a6dc95640 1 --2- 172.21.15.1:0/3733262644 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a5406b4d0 0x7f5a5406d990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.700+0000 7f5a6dc95640 1 --2- 172.21.15.1:0/3733262644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a68152690 0x7f5a6814fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.700+0000 7f5a6dc95640 1 --2- 172.21.15.1:0/3733262644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a680c6730 0x7f5a6814f890 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.700+0000 7f5a6dc95640 1 --2- 172.21.15.1:0/3733262644 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a680c5d10 0x7f5a68150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.700+0000 7f5a6dc95640 1 -- 172.21.15.1:0/3733262644 >> 172.21.15.1:0/3733262644 conn(0x7f5a6806b410 msgr2=0x7f5a6806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.701+0000 7f5a6dc95640 1 -- 172.21.15.1:0/3733262644 shutdown_connections 2024-03-06T07:23:03.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.701+0000 7f5a6dc95640 1 -- 172.21.15.1:0/3733262644 wait complete. 2024-03-06T07:23:03.713 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:03.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.813+0000 7f778db0c640 1 Processor -- start 2024-03-06T07:23:03.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.813+0000 7f778db0c640 1 -- start start 2024-03-06T07:23:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.814+0000 7f778db0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7788152650 0x7f7788155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.814+0000 7f778db0c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f77880c6b10 0x7f7788158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.814+0000 7f778db0c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77880c2fc0 con 0x7f77880bf9d0 2024-03-06T07:23:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.814+0000 7f778db0c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7788155800 con 0x7f77880c6b10 2024-03-06T07:23:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.814+0000 7f778db0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77880c6690 con 0x7f7788152650 2024-03-06T07:23:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.814+0000 7f7786ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f77880c6b10 0x7f7788158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.814+0000 7f7786ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f77880c6b10 0x7f7788158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57822/0 (socket says 172.21.15.1:57822) 2024-03-06T07:23:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.814+0000 7f7786ffd640 1 -- 172.21.15.1:0/877359319 learned_addr learned my addr 172.21.15.1:0/877359319 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.814+0000 7f77877fe640 1 --2- 172.21.15.1:0/877359319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7788152650 0x7f7788155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.814+0000 7f7786ffd640 1 -- 172.21.15.1:0/877359319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7788152650 msgr2=0x7f7788155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.814+0000 7f7786ffd640 1 --2- 172.21.15.1:0/877359319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7788152650 0x7f7788155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.814+0000 7f7786ffd640 1 -- 172.21.15.1:0/877359319 >> v1:172.21.15.98:6789/0 conn(0x7f77880bf9d0 legacy=0x7f77880c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.814+0000 7f7786ffd640 1 -- 172.21.15.1:0/877359319 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77880c7280 con 0x7f77880c6b10 2024-03-06T07:23:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.814+0000 7f77877fe640 1 --2- 172.21.15.1:0/877359319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7788152650 0x7f7788155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.815+0000 7f7786ffd640 1 --2- 172.21.15.1:0/877359319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77880c6b10 0x7f7788158690 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f7770009870 tx=0x7f777002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986e5ce38971ac29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.815+0000 7f77867fc640 1 -- 172.21.15.1:0/877359319 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77700317c0 con 0x7f77880c6b10 2024-03-06T07:23:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.815+0000 7f77867fc640 1 -- 172.21.15.1:0/877359319 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7770037070 con 0x7f77880c6b10 2024-03-06T07:23:03.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.815+0000 7f77867fc640 1 -- 172.21.15.1:0/877359319 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7770031b80 con 0x7f77880c6b10 2024-03-06T07:23:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.816+0000 7f778db0c640 1 -- 172.21.15.1:0/877359319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77880c6b10 msgr2=0x7f7788158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.816+0000 7f778db0c640 1 --2- 172.21.15.1:0/877359319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77880c6b10 0x7f7788158690 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f7770009870 tx=0x7f777002b860 comp rx=0 tx=0).stop 2024-03-06T07:23:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.816+0000 7f778db0c640 1 -- 172.21.15.1:0/877359319 shutdown_connections 2024-03-06T07:23:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.816+0000 7f778db0c640 1 --2- 172.21.15.1:0/877359319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77880c6b10 0x7f7788158690 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.816+0000 7f778db0c640 1 --2- 172.21.15.1:0/877359319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7788152650 0x7f7788155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.816+0000 7f778db0c640 1 -- 172.21.15.1:0/877359319 >> 172.21.15.1:0/877359319 conn(0x7f7788066ee0 msgr2=0x7f77880672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.816+0000 7f778db0c640 1 -- 172.21.15.1:0/877359319 shutdown_connections 2024-03-06T07:23:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.816+0000 7f778db0c640 1 -- 172.21.15.1:0/877359319 wait complete. 2024-03-06T07:23:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.817+0000 7f778db0c640 1 Processor -- start 2024-03-06T07:23:03.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.817+0000 7f778db0c640 1 -- start start 2024-03-06T07:23:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f778db0c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77880bf9d0 0x7f77880c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f778db0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7788152650 0x7f7788148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f778db0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77880c9da0 0x7f7788149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f778db0c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7788155490 con 0x7f77880bf9d0 2024-03-06T07:23:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f778db0c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77881551b0 con 0x7f77880c9da0 2024-03-06T07:23:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f778db0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7788152a30 con 0x7f7788152650 2024-03-06T07:23:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f77877fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77880bf9d0 0x7f77880c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f77877fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77880bf9d0 0x7f77880c8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57838/0 (socket says 172.21.15.1:57838) 2024-03-06T07:23:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f77877fe640 1 -- 172.21.15.1:0/3016889589 learned_addr learned my addr 172.21.15.1:0/3016889589 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f7786ffd640 1 --2- 172.21.15.1:0/3016889589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7788152650 0x7f7788148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f7787fff640 1 --2- 172.21.15.1:0/3016889589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77880c9da0 0x7f7788149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f77877fe640 1 -- 172.21.15.1:0/3016889589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7788152650 msgr2=0x7f7788148e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f77877fe640 1 --2- 172.21.15.1:0/3016889589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7788152650 0x7f7788148e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f77877fe640 1 -- 172.21.15.1:0/3016889589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77880c9da0 msgr2=0x7f7788149340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f77877fe640 1 --2- 172.21.15.1:0/3016889589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77880c9da0 0x7f7788149340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f77877fe640 1 -- 172.21.15.1:0/3016889589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778814de20 con 0x7f77880bf9d0 2024-03-06T07:23:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f7787fff640 1 --2- 172.21.15.1:0/3016889589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77880c9da0 0x7f7788149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f77877fe640 1 --2- 172.21.15.1:0/3016889589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77880bf9d0 0x7f77880c8a40 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f777800c820 tx=0x7f777800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f777ffff640 1 -- 172.21.15.1:0/3016889589 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778015070 con 0x7f77880bf9d0 2024-03-06T07:23:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f777ffff640 1 -- 172.21.15.1:0/3016889589 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7778007a30 con 0x7f77880bf9d0 2024-03-06T07:23:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f778db0c640 1 -- 172.21.15.1:0/3016889589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77881ee540 con 0x7f77880bf9d0 2024-03-06T07:23:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f777ffff640 1 -- 172.21.15.1:0/3016889589 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778007e00 con 0x7f77880bf9d0 2024-03-06T07:23:03.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.818+0000 7f778db0c640 1 -- 172.21.15.1:0/3016889589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77880c8f80 con 0x7f77880bf9d0 2024-03-06T07:23:03.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.820+0000 7f778db0c640 1 -- 172.21.15.1:0/3016889589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7788067fd0 con 0x7f77880bf9d0 2024-03-06T07:23:03.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.820+0000 7f777ffff640 1 -- 172.21.15.1:0/3016889589 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7778018070 con 0x7f77880bf9d0 2024-03-06T07:23:03.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.820+0000 7f777ffff640 1 --2- 172.21.15.1:0/3016889589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f776806b4d0 0x7f776806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.820+0000 7f777ffff640 1 -- 172.21.15.1:0/3016889589 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f777808a330 con 0x7f77880bf9d0 2024-03-06T07:23:03.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.821+0000 7f7786ffd640 1 --2- 172.21.15.1:0/3016889589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f776806b4d0 0x7f776806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.821+0000 7f7786ffd640 1 --2- 172.21.15.1:0/3016889589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f776806b4d0 0x7f776806d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7770036000 tx=0x7f777003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:03.825+0000 7f777ffff640 1 -- 172.21.15.1:0/3016889589 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7778059010 con 0x7f77880bf9d0 2024-03-06T07:23:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.008+0000 7f778db0c640 1 -- 172.21.15.1:0/3016889589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77880c1170 con 0x7f77880bf9d0 2024-03-06T07:23:04.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.009+0000 7f777ffff640 1 -- 172.21.15.1:0/3016889589 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+16825 (secure 0 0 0) 0x7f7778058910 con 0x7f77880bf9d0 2024-03-06T07:23:04.011 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:04.011 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":74,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:22:58.618600+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:55.240857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:55.939811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:04.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.010+0000 7f778db0c640 1 -- 172.21.15.1:0/3016889589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f776806b4d0 msgr2=0x7f776806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.010+0000 7f778db0c640 1 --2- 172.21.15.1:0/3016889589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f776806b4d0 0x7f776806d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7770036000 tx=0x7f777003d040 comp rx=0 tx=0).stop 2024-03-06T07:23:04.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.011+0000 7f778db0c640 1 -- 172.21.15.1:0/3016889589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77880bf9d0 msgr2=0x7f77880c8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.011+0000 7f778db0c640 1 --2- 172.21.15.1:0/3016889589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77880bf9d0 0x7f77880c8a40 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f777800c820 tx=0x7f777800cc20 comp rx=0 tx=0).stop 2024-03-06T07:23:04.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.011+0000 7f778db0c640 1 -- 172.21.15.1:0/3016889589 shutdown_connections 2024-03-06T07:23:04.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.011+0000 7f778db0c640 1 --2- 172.21.15.1:0/3016889589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f776806b4d0 0x7f776806d990 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.011+0000 7f778db0c640 1 --2- 172.21.15.1:0/3016889589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77880c9da0 0x7f7788149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.011+0000 7f778db0c640 1 --2- 172.21.15.1:0/3016889589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7788152650 0x7f7788148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.011+0000 7f778db0c640 1 --2- 172.21.15.1:0/3016889589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77880bf9d0 0x7f77880c8a40 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.011+0000 7f778db0c640 1 -- 172.21.15.1:0/3016889589 >> 172.21.15.1:0/3016889589 conn(0x7f7788066ee0 msgr2=0x7f778814e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:04.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.011+0000 7f778db0c640 1 -- 172.21.15.1:0/3016889589 shutdown_connections 2024-03-06T07:23:04.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.011+0000 7f778db0c640 1 -- 172.21.15.1:0/3016889589 wait complete. 2024-03-06T07:23:04.025 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:55.240857+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:55.939811+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:04.025 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:04.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.127+0000 7f1fe41f7640 1 Processor -- start 2024-03-06T07:23:04.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.127+0000 7f1fe41f7640 1 -- start start 2024-03-06T07:23:04.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.128+0000 7f1fe41f7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1fdc150520 0x7f1fdc1543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.128+0000 7f1fe41f7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fdc150900 con 0x7f1fdc070c80 2024-03-06T07:23:04.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.128+0000 7f1fe41f7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fdc071790 con 0x7f1fdc0c32f0 2024-03-06T07:23:04.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.128+0000 7f1fe41f7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fdc153ed0 con 0x7f1fdc150520 2024-03-06T07:23:04.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.128+0000 7f1fe1f93640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1fdc150520 0x7f1fdc1543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.128+0000 7f1fe2794640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1fdc070c80 0x7f1fdc15d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53632/0 (socket says 172.21.15.1:53632) 2024-03-06T07:23:04.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.128+0000 7f1fe2794640 1 -- 172.21.15.1:0/3232404781 learned_addr learned my addr 172.21.15.1:0/3232404781 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:04.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.128+0000 7f1fe0f91640 1 -- 172.21.15.1:0/3232404781 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594956556 0 0) 0x7f1fdc150900 con 0x7f1fdc070c80 2024-03-06T07:23:04.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.128+0000 7f1fe0f91640 1 -- 172.21.15.1:0/3232404781 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc8003520 con 0x7f1fdc070c80 2024-03-06T07:23:04.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.129+0000 7f1fe1f93640 1 -- 172.21.15.1:0/3232404781 >> v1:172.21.15.98:6789/0 conn(0x7f1fdc0c32f0 legacy=0x7f1fdc1537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.129+0000 7f1fe1f93640 1 -- 172.21.15.1:0/3232404781 >> v1:172.21.15.1:6789/0 conn(0x7f1fdc070c80 legacy=0x7f1fdc15d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.129+0000 7f1fe1f93640 1 -- 172.21.15.1:0/3232404781 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc157050 con 0x7f1fdc150520 2024-03-06T07:23:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.129+0000 7f1fe1f93640 1 --2- 172.21.15.1:0/3232404781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fdc150520 0x7f1fdc1543a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f1fd80097a0 tx=0x7f1fd802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e9a7ca1867064e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.129+0000 7f1fe0f91640 1 -- 172.21.15.1:0/3232404781 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8034070 con 0x7f1fdc150520 2024-03-06T07:23:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.129+0000 7f1fe0f91640 1 -- 172.21.15.1:0/3232404781 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd8030610 con 0x7f1fdc150520 2024-03-06T07:23:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.129+0000 7f1fe0f91640 1 -- 172.21.15.1:0/3232404781 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd80309e0 con 0x7f1fdc150520 2024-03-06T07:23:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.130+0000 7f1fe41f7640 1 -- 172.21.15.1:0/3232404781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fdc150520 msgr2=0x7f1fdc1543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.130+0000 7f1fe41f7640 1 --2- 172.21.15.1:0/3232404781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fdc150520 0x7f1fdc1543a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f1fd80097a0 tx=0x7f1fd802b960 comp rx=0 tx=0).stop 2024-03-06T07:23:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.130+0000 7f1fe41f7640 1 -- 172.21.15.1:0/3232404781 shutdown_connections 2024-03-06T07:23:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.130+0000 7f1fe41f7640 1 --2- 172.21.15.1:0/3232404781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fdc150520 0x7f1fdc1543a0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.130+0000 7f1fe41f7640 1 -- 172.21.15.1:0/3232404781 >> 172.21.15.1:0/3232404781 conn(0x7f1fdc0c88f0 msgr2=0x7f1fdc0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.130+0000 7f1fe41f7640 1 -- 172.21.15.1:0/3232404781 shutdown_connections 2024-03-06T07:23:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.130+0000 7f1fe41f7640 1 -- 172.21.15.1:0/3232404781 wait complete. 2024-03-06T07:23:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.131+0000 7f1fe41f7640 1 Processor -- start 2024-03-06T07:23:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.131+0000 7f1fe41f7640 1 -- start start 2024-03-06T07:23:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.132+0000 7f1fe41f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fdc070c80 0x7f1fdc1eb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.132+0000 7f1fe41f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc0c32f0 0x7f1fdc0acb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.132+0000 7f1fe41f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fdc150520 0x7f1fdc146090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.132+0000 7f1fe41f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc157330 con 0x7f1fdc070c80 2024-03-06T07:23:04.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.132+0000 7f1fe41f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc1532a0 con 0x7f1fdc150520 2024-03-06T07:23:04.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.132+0000 7f1fe41f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc0c7ec0 con 0x7f1fdc0c32f0 2024-03-06T07:23:04.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.132+0000 7f1fe1f93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fdc070c80 0x7f1fdc1eb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.132+0000 7f1fe1f93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fdc070c80 0x7f1fdc1eb0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57864/0 (socket says 172.21.15.1:57864) 2024-03-06T07:23:04.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.132+0000 7f1fe1f93640 1 -- 172.21.15.1:0/1456404057 learned_addr learned my addr 172.21.15.1:0/1456404057 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:04.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.132+0000 7f1fe1792640 1 --2- 172.21.15.1:0/1456404057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc0c32f0 0x7f1fdc0acb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.132+0000 7f1fe2794640 1 --2- 172.21.15.1:0/1456404057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fdc150520 0x7f1fdc146090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.132+0000 7f1fe1f93640 1 -- 172.21.15.1:0/1456404057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc0c32f0 msgr2=0x7f1fdc0acb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.132+0000 7f1fe1f93640 1 --2- 172.21.15.1:0/1456404057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc0c32f0 0x7f1fdc0acb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.132+0000 7f1fe1f93640 1 -- 172.21.15.1:0/1456404057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fdc150520 msgr2=0x7f1fdc146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.132+0000 7f1fe1f93640 1 --2- 172.21.15.1:0/1456404057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fdc150520 0x7f1fdc146090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.132+0000 7f1fe1f93640 1 -- 172.21.15.1:0/1456404057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc06db70 con 0x7f1fdc070c80 2024-03-06T07:23:04.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.133+0000 7f1fe1f93640 1 --2- 172.21.15.1:0/1456404057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fdc070c80 0x7f1fdc1eb0e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f1fd80097a0 tx=0x7f1fd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.133+0000 7f1fd27fc640 1 -- 172.21.15.1:0/1456404057 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8047070 con 0x7f1fdc070c80 2024-03-06T07:23:04.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.133+0000 7f1fd27fc640 1 -- 172.21.15.1:0/1456404057 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd8030410 con 0x7f1fdc070c80 2024-03-06T07:23:04.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.133+0000 7f1fd27fc640 1 -- 172.21.15.1:0/1456404057 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8034040 con 0x7f1fdc070c80 2024-03-06T07:23:04.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.133+0000 7f1fe41f7640 1 -- 172.21.15.1:0/1456404057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fdc06de50 con 0x7f1fdc070c80 2024-03-06T07:23:04.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.133+0000 7f1fe41f7640 1 -- 172.21.15.1:0/1456404057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fdc0ad0d0 con 0x7f1fdc070c80 2024-03-06T07:23:04.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.134+0000 7f1fe41f7640 1 -- 172.21.15.1:0/1456404057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fdc06f4e0 con 0x7f1fdc070c80 2024-03-06T07:23:04.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.134+0000 7f1fd27fc640 1 -- 172.21.15.1:0/1456404057 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fd804a070 con 0x7f1fdc070c80 2024-03-06T07:23:04.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.134+0000 7f1fd27fc640 1 --2- 172.21.15.1:0/1456404057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1fc006b4d0 0x7f1fc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.135+0000 7f1fd27fc640 1 -- 172.21.15.1:0/1456404057 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1fd80abbf0 con 0x7f1fdc070c80 2024-03-06T07:23:04.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.135+0000 7f1fe1792640 1 --2- 172.21.15.1:0/1456404057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1fc006b4d0 0x7f1fc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.135+0000 7f1fe1792640 1 --2- 172.21.15.1:0/1456404057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1fc006b4d0 0x7f1fc006d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0c2eb0 tx=0x7f1fcc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.139+0000 7f1fd27fc640 1 -- 172.21.15.1:0/1456404057 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1fdc06f4e0 con 0x7f1fdc070c80 2024-03-06T07:23:04.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.318+0000 7f1fe41f7640 1 -- 172.21.15.1:0/1456404057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1fdc0686e0 con 0x7f1fdc070c80 2024-03-06T07:23:04.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.318+0000 7f1fd27fc640 1 -- 172.21.15.1:0/1456404057 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1fd807a320 con 0x7f1fdc070c80 2024-03-06T07:23:04.321 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:23:04.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.320+0000 7f1fe41f7640 1 -- 172.21.15.1:0/1456404057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1fc006b4d0 msgr2=0x7f1fc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.320+0000 7f1fe41f7640 1 --2- 172.21.15.1:0/1456404057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1fc006b4d0 0x7f1fc006d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0c2eb0 tx=0x7f1fcc009040 comp rx=0 tx=0).stop 2024-03-06T07:23:04.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.320+0000 7f1fe41f7640 1 -- 172.21.15.1:0/1456404057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fdc070c80 msgr2=0x7f1fdc1eb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.320+0000 7f1fe41f7640 1 --2- 172.21.15.1:0/1456404057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fdc070c80 0x7f1fdc1eb0e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f1fd80097a0 tx=0x7f1fd80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.320+0000 7f1fe41f7640 1 -- 172.21.15.1:0/1456404057 shutdown_connections 2024-03-06T07:23:04.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.320+0000 7f1fe41f7640 1 --2- 172.21.15.1:0/1456404057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1fc006b4d0 0x7f1fc006d990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.320+0000 7f1fe41f7640 1 --2- 172.21.15.1:0/1456404057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fdc150520 0x7f1fdc146090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.320+0000 7f1fe41f7640 1 --2- 172.21.15.1:0/1456404057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc0c32f0 0x7f1fdc0acb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.320+0000 7f1fe41f7640 1 --2- 172.21.15.1:0/1456404057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fdc070c80 0x7f1fdc1eb0e0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.320+0000 7f1fe41f7640 1 -- 172.21.15.1:0/1456404057 >> 172.21.15.1:0/1456404057 conn(0x7f1fdc0c88f0 msgr2=0x7f1fdc14b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:04.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.320+0000 7f1fe41f7640 1 -- 172.21.15.1:0/1456404057 shutdown_connections 2024-03-06T07:23:04.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.320+0000 7f1fe41f7640 1 -- 172.21.15.1:0/1456404057 wait complete. 2024-03-06T07:23:04.333 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:04.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.436+0000 7f2bd61a1640 1 Processor -- start 2024-03-06T07:23:04.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.436+0000 7f2bd61a1640 1 -- start start 2024-03-06T07:23:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.437+0000 7f2bd61a1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2bd0152650 0x7f2bd0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.437+0000 7f2bd61a1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2bd00c6b10 0x7f2bd0158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.437+0000 7f2bd61a1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2bd00bf9d0 0x7f2bd00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.437+0000 7f2bd61a1640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd0155800 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.437+0000 7f2bd61a1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd00c2fc0 con 0x7f2bd00c6b10 2024-03-06T07:23:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.437+0000 7f2bd61a1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd00c6690 con 0x7f2bd0152650 2024-03-06T07:23:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.438+0000 7f2bcffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2bd00bf9d0 0x7f2bd00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.438+0000 7f2bcffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2bd00bf9d0 0x7f2bd00c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42408/0 (socket says 172.21.15.1:42408) 2024-03-06T07:23:04.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.438+0000 7f2bcffff640 1 -- 172.21.15.1:0/1602313591 learned_addr learned my addr 172.21.15.1:0/1602313591 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:04.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.438+0000 7f2bcf7fe640 1 --2- 172.21.15.1:0/1602313591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2bd0152650 0x7f2bd0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.438+0000 7f2bc7fff640 1 --2- 172.21.15.1:0/1602313591 >> v2:172.21.15.98:3300/0 conn(0x7f2bd00c6b10 0x7f2bd0158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.438+0000 7f2bcffff640 1 -- 172.21.15.1:0/1602313591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2bd0152650 msgr2=0x7f2bd0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:04.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.438+0000 7f2bcffff640 1 --2- 172.21.15.1:0/1602313591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2bd0152650 0x7f2bd0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.438+0000 7f2bcffff640 1 -- 172.21.15.1:0/1602313591 >> v2:172.21.15.98:3300/0 conn(0x7f2bd00c6b10 msgr2=0x7f2bd0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:04.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.438+0000 7f2bcffff640 1 --2- 172.21.15.1:0/1602313591 >> v2:172.21.15.98:3300/0 conn(0x7f2bd00c6b10 0x7f2bd0158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.438+0000 7f2bcffff640 1 -- 172.21.15.1:0/1602313591 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd00c7280 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.439+0000 7f2bcffff640 1 --2- 172.21.15.1:0/1602313591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd00bf9d0 0x7f2bd00c1460 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f2bd008c500 tx=0x7f2bc802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a17a56f4c7def300 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.439+0000 7f2bceffd640 1 -- 172.21.15.1:0/1602313591 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc802c9e0 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.439+0000 7f2bceffd640 1 -- 172.21.15.1:0/1602313591 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc802cb80 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.439+0000 7f2bceffd640 1 -- 172.21.15.1:0/1602313591 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc80343f0 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.439+0000 7f2bd61a1640 1 -- 172.21.15.1:0/1602313591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd00bf9d0 msgr2=0x7f2bd00c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.439+0000 7f2bd61a1640 1 --2- 172.21.15.1:0/1602313591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd00bf9d0 0x7f2bd00c1460 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f2bd008c500 tx=0x7f2bc802be40 comp rx=0 tx=0).stop 2024-03-06T07:23:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.439+0000 7f2bd61a1640 1 -- 172.21.15.1:0/1602313591 shutdown_connections 2024-03-06T07:23:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.439+0000 7f2bd61a1640 1 --2- 172.21.15.1:0/1602313591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd00bf9d0 0x7f2bd00c1460 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.439+0000 7f2bd61a1640 1 --2- 172.21.15.1:0/1602313591 >> v2:172.21.15.98:3300/0 conn(0x7f2bd00c6b10 0x7f2bd0158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.439+0000 7f2bd61a1640 1 --2- 172.21.15.1:0/1602313591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2bd0152650 0x7f2bd0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.439+0000 7f2bd61a1640 1 -- 172.21.15.1:0/1602313591 >> 172.21.15.1:0/1602313591 conn(0x7f2bd0066ee0 msgr2=0x7f2bd00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.439+0000 7f2bd61a1640 1 -- 172.21.15.1:0/1602313591 shutdown_connections 2024-03-06T07:23:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.439+0000 7f2bd61a1640 1 -- 172.21.15.1:0/1602313591 wait complete. 2024-03-06T07:23:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.441+0000 7f2bd61a1640 1 Processor -- start 2024-03-06T07:23:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.441+0000 7f2bd61a1640 1 -- start start 2024-03-06T07:23:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.442+0000 7f2bd61a1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd00bf9d0 0x7f2bd00ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.442+0000 7f2bd61a1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bd0152650 0x7f2bd00c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.442+0000 7f2bd61a1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2bd014e5f0 0x7f2bd01e0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.442+0000 7f2bd61a1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd0155490 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.442+0000 7f2bd61a1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd01551b0 con 0x7f2bd0152650 2024-03-06T07:23:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.442+0000 7f2bd61a1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd0152a30 con 0x7f2bd014e5f0 2024-03-06T07:23:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.442+0000 7f2bcf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd00bf9d0 0x7f2bd00ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.442+0000 7f2bcf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd00bf9d0 0x7f2bd00ca400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42414/0 (socket says 172.21.15.1:42414) 2024-03-06T07:23:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.442+0000 7f2bcf7fe640 1 -- 172.21.15.1:0/1447028287 learned_addr learned my addr 172.21.15.1:0/1447028287 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.442+0000 7f2bc7fff640 1 --2- 172.21.15.1:0/1447028287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bd0152650 0x7f2bd00c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.442+0000 7f2bcffff640 1 --2- 172.21.15.1:0/1447028287 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2bd014e5f0 0x7f2bd01e0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.442+0000 7f2bcf7fe640 1 -- 172.21.15.1:0/1447028287 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2bd014e5f0 msgr2=0x7f2bd01e0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.442+0000 7f2bcf7fe640 1 --2- 172.21.15.1:0/1447028287 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2bd014e5f0 0x7f2bd01e0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.442+0000 7f2bcf7fe640 1 -- 172.21.15.1:0/1447028287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bd0152650 msgr2=0x7f2bd00c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.442+0000 7f2bcf7fe640 1 --2- 172.21.15.1:0/1447028287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bd0152650 0x7f2bd00c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.442+0000 7f2bcf7fe640 1 -- 172.21.15.1:0/1447028287 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd014c400 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.442+0000 7f2bcffff640 1 --2- 172.21.15.1:0/1447028287 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2bd014e5f0 0x7f2bd01e0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.443+0000 7f2bcf7fe640 1 --2- 172.21.15.1:0/1447028287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd00bf9d0 0x7f2bd00ca400 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f2bc000d830 tx=0x7f2bc000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.443+0000 7f2bccff9640 1 -- 172.21.15.1:0/1447028287 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc000b6c0 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.443+0000 7f2bccff9640 1 -- 172.21.15.1:0/1447028287 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc0019070 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.443+0000 7f2bd61a1640 1 -- 172.21.15.1:0/1447028287 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd01e02c0 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.443+0000 7f2bccff9640 1 -- 172.21.15.1:0/1447028287 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc000ba80 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.443+0000 7f2bd61a1640 1 -- 172.21.15.1:0/1447028287 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd014c6e0 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.444+0000 7f2bd61a1640 1 -- 172.21.15.1:0/1447028287 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd0067fd0 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.444+0000 7f2bccff9640 1 -- 172.21.15.1:0/1447028287 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2bc001e460 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.445+0000 7f2bccff9640 1 --2- 172.21.15.1:0/1447028287 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2bb006b4d0 0x7f2bb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.445+0000 7f2bccff9640 1 -- 172.21.15.1:0/1447028287 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2bc008a940 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.445+0000 7f2bc7fff640 1 --2- 172.21.15.1:0/1447028287 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2bb006b4d0 0x7f2bb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.445+0000 7f2bc7fff640 1 --2- 172.21.15.1:0/1447028287 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2bb006b4d0 0x7f2bb006d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0144d70 tx=0x7f2bb8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.449+0000 7f2bccff9640 1 -- 172.21.15.1:0/1447028287 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2bc00596c0 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.632+0000 7f2bd61a1640 1 -- 172.21.15.1:0/1447028287 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2bd00b06f0 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.632+0000 7f2bccff9640 1 -- 172.21.15.1:0/1447028287 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v74) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2bc0058fc0 con 0x7f2bd00bf9d0 2024-03-06T07:23:04.634 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:23:04.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.634+0000 7f2bd61a1640 1 -- 172.21.15.1:0/1447028287 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2bb006b4d0 msgr2=0x7f2bb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.634+0000 7f2bd61a1640 1 --2- 172.21.15.1:0/1447028287 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2bb006b4d0 0x7f2bb006d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0144d70 tx=0x7f2bb8008040 comp rx=0 tx=0).stop 2024-03-06T07:23:04.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.634+0000 7f2bd61a1640 1 -- 172.21.15.1:0/1447028287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd00bf9d0 msgr2=0x7f2bd00ca400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.634+0000 7f2bd61a1640 1 --2- 172.21.15.1:0/1447028287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd00bf9d0 0x7f2bd00ca400 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f2bc000d830 tx=0x7f2bc000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.634+0000 7f2bd61a1640 1 -- 172.21.15.1:0/1447028287 shutdown_connections 2024-03-06T07:23:04.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.634+0000 7f2bd61a1640 1 --2- 172.21.15.1:0/1447028287 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2bb006b4d0 0x7f2bb006d990 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.634+0000 7f2bd61a1640 1 --2- 172.21.15.1:0/1447028287 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2bd014e5f0 0x7f2bd01e0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.634+0000 7f2bd61a1640 1 --2- 172.21.15.1:0/1447028287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bd0152650 0x7f2bd00c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.634+0000 7f2bd61a1640 1 --2- 172.21.15.1:0/1447028287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd00bf9d0 0x7f2bd00ca400 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.634+0000 7f2bd61a1640 1 -- 172.21.15.1:0/1447028287 >> 172.21.15.1:0/1447028287 conn(0x7f2bd0066ee0 msgr2=0x7f2bd00672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:04.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.634+0000 7f2bd61a1640 1 -- 172.21.15.1:0/1447028287 shutdown_connections 2024-03-06T07:23:04.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.634+0000 7f2bd61a1640 1 -- 172.21.15.1:0/1447028287 wait complete. 2024-03-06T07:23:04.648 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:04.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.764+0000 7f7bdf888640 1 Processor -- start 2024-03-06T07:23:04.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.764+0000 7f7bdf888640 1 -- start start 2024-03-06T07:23:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.765+0000 7f7bdf888640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7bd8152690 0x7f7bd8155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.765+0000 7f7bdf888640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7bd8156cd0 0x7f7bd8157120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.765+0000 7f7bdf888640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd81561e0 con 0x7f7bd80c6730 2024-03-06T07:23:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.765+0000 7f7bdf888640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bd80c3300 con 0x7f7bd8156cd0 2024-03-06T07:23:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.765+0000 7f7bdf888640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bd80c35a0 con 0x7f7bd8152690 2024-03-06T07:23:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.765+0000 7f7bdde25640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7bd8156cd0 0x7f7bd8157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.765+0000 7f7bdde25640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7bd8156cd0 0x7f7bd8157120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42428/0 (socket says 172.21.15.1:42428) 2024-03-06T07:23:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.765+0000 7f7bdde25640 1 -- 172.21.15.1:0/1466229301 learned_addr learned my addr 172.21.15.1:0/1466229301 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.765+0000 7f7bdd624640 1 --2- 172.21.15.1:0/1466229301 >> v2:172.21.15.98:3300/0 conn(0x7f7bd8152690 0x7f7bd8155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.765+0000 7f7bdde25640 1 -- 172.21.15.1:0/1466229301 >> v2:172.21.15.98:3300/0 conn(0x7f7bd8152690 msgr2=0x7f7bd8155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.765+0000 7f7bdde25640 1 --2- 172.21.15.1:0/1466229301 >> v2:172.21.15.98:3300/0 conn(0x7f7bd8152690 0x7f7bd8155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.765+0000 7f7bdde25640 1 -- 172.21.15.1:0/1466229301 >> v1:172.21.15.98:6789/0 conn(0x7f7bd80c6730 legacy=0x7f7bd80c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.765+0000 7f7bdde25640 1 -- 172.21.15.1:0/1466229301 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd80c3880 con 0x7f7bd8156cd0 2024-03-06T07:23:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.767+0000 7f7bdde25640 1 --2- 172.21.15.1:0/1466229301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bd8156cd0 0x7f7bd8157120 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f7bd80ab5f0 tx=0x7f7bc802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe880cb931eb5e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.767+0000 7f7bd7fff640 1 -- 172.21.15.1:0/1466229301 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc802e070 con 0x7f7bd8156cd0 2024-03-06T07:23:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.767+0000 7f7bd7fff640 1 -- 172.21.15.1:0/1466229301 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc8037070 con 0x7f7bd8156cd0 2024-03-06T07:23:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.767+0000 7f7bd7fff640 1 -- 172.21.15.1:0/1466229301 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8031a00 con 0x7f7bd8156cd0 2024-03-06T07:23:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.767+0000 7f7bdf888640 1 -- 172.21.15.1:0/1466229301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bd8156cd0 msgr2=0x7f7bd8157120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.767+0000 7f7bdf888640 1 --2- 172.21.15.1:0/1466229301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bd8156cd0 0x7f7bd8157120 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f7bd80ab5f0 tx=0x7f7bc802b880 comp rx=0 tx=0).stop 2024-03-06T07:23:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.768+0000 7f7bdf888640 1 -- 172.21.15.1:0/1466229301 shutdown_connections 2024-03-06T07:23:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.768+0000 7f7bdf888640 1 --2- 172.21.15.1:0/1466229301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bd8156cd0 0x7f7bd8157120 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.768+0000 7f7bdf888640 1 --2- 172.21.15.1:0/1466229301 >> v2:172.21.15.98:3300/0 conn(0x7f7bd8152690 0x7f7bd8155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.768+0000 7f7bdf888640 1 -- 172.21.15.1:0/1466229301 >> 172.21.15.1:0/1466229301 conn(0x7f7bd806b410 msgr2=0x7f7bd806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:04.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.768+0000 7f7bdf888640 1 -- 172.21.15.1:0/1466229301 shutdown_connections 2024-03-06T07:23:04.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.768+0000 7f7bdf888640 1 -- 172.21.15.1:0/1466229301 wait complete. 2024-03-06T07:23:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.769+0000 7f7bdf888640 1 Processor -- start 2024-03-06T07:23:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.769+0000 7f7bdf888640 1 -- start start 2024-03-06T07:23:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.769+0000 7f7bdf888640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bd80c6730 0x7f7bd80c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.769+0000 7f7bdf888640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bd8152690 0x7f7bd80c8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.769+0000 7f7bdf888640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7bd8156cd0 0x7f7bd814ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.769+0000 7f7bdf888640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd80c5ee0 con 0x7f7bd80c6730 2024-03-06T07:23:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.769+0000 7f7bdf888640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd81567b0 con 0x7f7bd8152690 2024-03-06T07:23:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.769+0000 7f7bdf888640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd80c6060 con 0x7f7bd8156cd0 2024-03-06T07:23:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.769+0000 7f7bdd624640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bd80c6730 0x7f7bd80c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.769+0000 7f7bdd624640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bd80c6730 0x7f7bd80c7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42436/0 (socket says 172.21.15.1:42436) 2024-03-06T07:23:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.769+0000 7f7bdd624640 1 -- 172.21.15.1:0/2372252391 learned_addr learned my addr 172.21.15.1:0/2372252391 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.770+0000 7f7bdce23640 1 --2- 172.21.15.1:0/2372252391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bd8152690 0x7f7bd80c8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.770+0000 7f7bdde25640 1 --2- 172.21.15.1:0/2372252391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7bd8156cd0 0x7f7bd814ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.770+0000 7f7bdd624640 1 -- 172.21.15.1:0/2372252391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7bd8156cd0 msgr2=0x7f7bd814ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.770+0000 7f7bdd624640 1 --2- 172.21.15.1:0/2372252391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7bd8156cd0 0x7f7bd814ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.770+0000 7f7bdd624640 1 -- 172.21.15.1:0/2372252391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bd8152690 msgr2=0x7f7bd80c8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.770+0000 7f7bdd624640 1 --2- 172.21.15.1:0/2372252391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bd8152690 0x7f7bd80c8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.770+0000 7f7bdd624640 1 -- 172.21.15.1:0/2372252391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd81ee9c0 con 0x7f7bd80c6730 2024-03-06T07:23:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.770+0000 7f7bdce23640 1 --2- 172.21.15.1:0/2372252391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bd8152690 0x7f7bd80c8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.770+0000 7f7bdd624640 1 --2- 172.21.15.1:0/2372252391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bd80c6730 0x7f7bd80c7fe0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f7bd000ccd0 tx=0x7f7bd000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.770+0000 7f7bd5ffb640 1 -- 172.21.15.1:0/2372252391 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd0003990 con 0x7f7bd80c6730 2024-03-06T07:23:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.770+0000 7f7bd5ffb640 1 -- 172.21.15.1:0/2372252391 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd0003b30 con 0x7f7bd80c6730 2024-03-06T07:23:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.770+0000 7f7bdf888640 1 -- 172.21.15.1:0/2372252391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd8154490 con 0x7f7bd80c6730 2024-03-06T07:23:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.770+0000 7f7bd5ffb640 1 -- 172.21.15.1:0/2372252391 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd0004020 con 0x7f7bd80c6730 2024-03-06T07:23:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.770+0000 7f7bdf888640 1 -- 172.21.15.1:0/2372252391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd8155590 con 0x7f7bd80c6730 2024-03-06T07:23:04.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.771+0000 7f7bdf888640 1 -- 172.21.15.1:0/2372252391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd80c2eb0 con 0x7f7bd80c6730 2024-03-06T07:23:04.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.772+0000 7f7bd5ffb640 1 -- 172.21.15.1:0/2372252391 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7bd00041c0 con 0x7f7bd80c6730 2024-03-06T07:23:04.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.772+0000 7f7bd5ffb640 1 --2- 172.21.15.1:0/2372252391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bc406b500 0x7f7bc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.772+0000 7f7bd5ffb640 1 -- 172.21.15.1:0/2372252391 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7bd008a7a0 con 0x7f7bd80c6730 2024-03-06T07:23:04.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.772+0000 7f7bdce23640 1 --2- 172.21.15.1:0/2372252391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bc406b500 0x7f7bc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.773+0000 7f7bdce23640 1 --2- 172.21.15.1:0/2372252391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bc406b500 0x7f7bc406d9c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc006fd0 tx=0x7f7bcc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.777+0000 7f7bd5ffb640 1 -- 172.21.15.1:0/2372252391 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7bd80c2eb0 con 0x7f7bd80c6730 2024-03-06T07:23:04.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.957+0000 7f7bdf888640 1 -- 172.21.15.1:0/2372252391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7bd8067fd0 con 0x7f7bd80c6730 2024-03-06T07:23:04.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.957+0000 7f7bd5ffb640 1 -- 172.21.15.1:0/2372252391 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v74) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7bd81ee9c0 con 0x7f7bd80c6730 2024-03-06T07:23:04.959 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:23:04.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.959+0000 7f7bdf888640 1 -- 172.21.15.1:0/2372252391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bc406b500 msgr2=0x7f7bc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.959+0000 7f7bdf888640 1 --2- 172.21.15.1:0/2372252391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bc406b500 0x7f7bc406d9c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc006fd0 tx=0x7f7bcc008040 comp rx=0 tx=0).stop 2024-03-06T07:23:04.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.959+0000 7f7bdf888640 1 -- 172.21.15.1:0/2372252391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bd80c6730 msgr2=0x7f7bd80c7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.959+0000 7f7bdf888640 1 --2- 172.21.15.1:0/2372252391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bd80c6730 0x7f7bd80c7fe0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f7bd000ccd0 tx=0x7f7bd000a5c0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.959+0000 7f7bdf888640 1 -- 172.21.15.1:0/2372252391 shutdown_connections 2024-03-06T07:23:04.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.959+0000 7f7bdf888640 1 --2- 172.21.15.1:0/2372252391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bc406b500 0x7f7bc406d9c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.959+0000 7f7bdf888640 1 --2- 172.21.15.1:0/2372252391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7bd8156cd0 0x7f7bd814ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.959+0000 7f7bdf888640 1 --2- 172.21.15.1:0/2372252391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bd8152690 0x7f7bd80c8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.959+0000 7f7bdf888640 1 --2- 172.21.15.1:0/2372252391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bd80c6730 0x7f7bd80c7fe0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.959+0000 7f7bdf888640 1 -- 172.21.15.1:0/2372252391 >> 172.21.15.1:0/2372252391 conn(0x7f7bd806b410 msgr2=0x7f7bd81570e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:04.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.959+0000 7f7bdf888640 1 -- 172.21.15.1:0/2372252391 shutdown_connections 2024-03-06T07:23:04.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:04.959+0000 7f7bdf888640 1 -- 172.21.15.1:0/2372252391 wait complete. 2024-03-06T07:23:04.973 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:23:05.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.072+0000 7f2acf177640 1 Processor -- start 2024-03-06T07:23:05.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.072+0000 7f2acf177640 1 -- start start 2024-03-06T07:23:05.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.073+0000 7f2acf177640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2ac8152650 0x7f2ac8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.073+0000 7f2acf177640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2ac80c6b10 0x7f2ac8158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.073+0000 7f2acf177640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac80c2fc0 con 0x7f2ac80bf9d0 2024-03-06T07:23:05.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.073+0000 7f2acf177640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac8155800 con 0x7f2ac80c6b10 2024-03-06T07:23:05.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.073+0000 7f2acf177640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac80c6690 con 0x7f2ac8152650 2024-03-06T07:23:05.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.073+0000 7f2ac7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2ac80c6b10 0x7f2ac8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.073+0000 7f2ac7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2ac80c6b10 0x7f2ac8158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42440/0 (socket says 172.21.15.1:42440) 2024-03-06T07:23:05.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.073+0000 7f2ac7fff640 1 -- 172.21.15.1:0/1640910248 learned_addr learned my addr 172.21.15.1:0/1640910248 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:05.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.074+0000 7f2accf13640 1 --2- 172.21.15.1:0/1640910248 >> v2:172.21.15.98:3300/0 conn(0x7f2ac8152650 0x7f2ac8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.074+0000 7f2ac7fff640 1 -- 172.21.15.1:0/1640910248 >> v2:172.21.15.98:3300/0 conn(0x7f2ac8152650 msgr2=0x7f2ac8155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:05.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.074+0000 7f2ac7fff640 1 --2- 172.21.15.1:0/1640910248 >> v2:172.21.15.98:3300/0 conn(0x7f2ac8152650 0x7f2ac8155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.074+0000 7f2ac7fff640 1 -- 172.21.15.1:0/1640910248 >> v1:172.21.15.98:6789/0 conn(0x7f2ac80bf9d0 legacy=0x7f2ac80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:05.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.074+0000 7f2ac7fff640 1 -- 172.21.15.1:0/1640910248 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac80c7280 con 0x7f2ac80c6b10 2024-03-06T07:23:05.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.074+0000 7f2accf13640 1 --2- 172.21.15.1:0/1640910248 >> v2:172.21.15.98:3300/0 conn(0x7f2ac8152650 0x7f2ac8155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:05.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.074+0000 7f2ac7fff640 1 --2- 172.21.15.1:0/1640910248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac80c6b10 0x7f2ac8158690 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f2ac80cac90 tx=0x7f2ab402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cabd4f237a3b6b36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.075+0000 7f2ac77fe640 1 -- 172.21.15.1:0/1640910248 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab402e070 con 0x7f2ac80c6b10 2024-03-06T07:23:05.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.075+0000 7f2ac77fe640 1 -- 172.21.15.1:0/1640910248 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab4037070 con 0x7f2ac80c6b10 2024-03-06T07:23:05.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.075+0000 7f2ac77fe640 1 -- 172.21.15.1:0/1640910248 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4031a00 con 0x7f2ac80c6b10 2024-03-06T07:23:05.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.075+0000 7f2acf177640 1 -- 172.21.15.1:0/1640910248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac80c6b10 msgr2=0x7f2ac8158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.075+0000 7f2acf177640 1 --2- 172.21.15.1:0/1640910248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac80c6b10 0x7f2ac8158690 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f2ac80cac90 tx=0x7f2ab402b880 comp rx=0 tx=0).stop 2024-03-06T07:23:05.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.075+0000 7f2acf177640 1 -- 172.21.15.1:0/1640910248 shutdown_connections 2024-03-06T07:23:05.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.075+0000 7f2acf177640 1 --2- 172.21.15.1:0/1640910248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac80c6b10 0x7f2ac8158690 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.075+0000 7f2acf177640 1 --2- 172.21.15.1:0/1640910248 >> v2:172.21.15.98:3300/0 conn(0x7f2ac8152650 0x7f2ac8155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.075+0000 7f2acf177640 1 -- 172.21.15.1:0/1640910248 >> 172.21.15.1:0/1640910248 conn(0x7f2ac8066ee0 msgr2=0x7f2ac80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.075+0000 7f2acf177640 1 -- 172.21.15.1:0/1640910248 shutdown_connections 2024-03-06T07:23:05.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.075+0000 7f2acf177640 1 -- 172.21.15.1:0/1640910248 wait complete. 2024-03-06T07:23:05.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.076+0000 7f2acf177640 1 Processor -- start 2024-03-06T07:23:05.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.076+0000 7f2acf177640 1 -- start start 2024-03-06T07:23:05.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.077+0000 7f2acf177640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ac80bf9d0 0x7f2ac8149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.077+0000 7f2acf177640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac8152650 0x7f2ac81eb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.077+0000 7f2acf177640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac814a330 0x7f2ac81eb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.077+0000 7f2acf177640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac8155490 con 0x7f2ac814a330 2024-03-06T07:23:05.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.077+0000 7f2acf177640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac81551b0 con 0x7f2ac80bf9d0 2024-03-06T07:23:05.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.077+0000 7f2acf177640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac8152a30 con 0x7f2ac8152650 2024-03-06T07:23:05.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.077+0000 7f2acd714640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac814a330 0x7f2ac81eb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.077+0000 7f2acd714640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac814a330 0x7f2ac81eb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42442/0 (socket says 172.21.15.1:42442) 2024-03-06T07:23:05.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.077+0000 7f2acd714640 1 -- 172.21.15.1:0/1740111779 learned_addr learned my addr 172.21.15.1:0/1740111779 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:05.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.077+0000 7f2accf13640 1 --2- 172.21.15.1:0/1740111779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ac80bf9d0 0x7f2ac8149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.077+0000 7f2ac7fff640 1 --2- 172.21.15.1:0/1740111779 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac8152650 0x7f2ac81eb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.078+0000 7f2acd714640 1 -- 172.21.15.1:0/1740111779 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac8152650 msgr2=0x7f2ac81eb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.078+0000 7f2acd714640 1 --2- 172.21.15.1:0/1740111779 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac8152650 0x7f2ac81eb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.078+0000 7f2acd714640 1 -- 172.21.15.1:0/1740111779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ac80bf9d0 msgr2=0x7f2ac8149df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.078+0000 7f2acd714640 1 --2- 172.21.15.1:0/1740111779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ac80bf9d0 0x7f2ac8149df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.078+0000 7f2acd714640 1 -- 172.21.15.1:0/1740111779 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac81ee770 con 0x7f2ac814a330 2024-03-06T07:23:05.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.078+0000 7f2ac7fff640 1 --2- 172.21.15.1:0/1740111779 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac8152650 0x7f2ac81eb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:05.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.078+0000 7f2acd714640 1 --2- 172.21.15.1:0/1740111779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac814a330 0x7f2ac81eb740 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000b970 tx=0x7f2ac000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.078+0000 7f2ac57fa640 1 -- 172.21.15.1:0/1740111779 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0019070 con 0x7f2ac814a330 2024-03-06T07:23:05.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.078+0000 7f2ac57fa640 1 -- 172.21.15.1:0/1740111779 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac000cae0 con 0x7f2ac814a330 2024-03-06T07:23:05.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.078+0000 7f2acf177640 1 -- 172.21.15.1:0/1740111779 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac81e1900 con 0x7f2ac814a330 2024-03-06T07:23:05.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.078+0000 7f2ac57fa640 1 -- 172.21.15.1:0/1740111779 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0014440 con 0x7f2ac814a330 2024-03-06T07:23:05.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.078+0000 7f2acf177640 1 -- 172.21.15.1:0/1740111779 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac81eed10 con 0x7f2ac814a330 2024-03-06T07:23:05.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.079+0000 7f2acf177640 1 -- 172.21.15.1:0/1740111779 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac8155c90 con 0x7f2ac814a330 2024-03-06T07:23:05.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.080+0000 7f2ac57fa640 1 -- 172.21.15.1:0/1740111779 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ac001c070 con 0x7f2ac814a330 2024-03-06T07:23:05.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.080+0000 7f2ac57fa640 1 --2- 172.21.15.1:0/1740111779 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2aac06b4d0 0x7f2aac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.080+0000 7f2ac57fa640 1 -- 172.21.15.1:0/1740111779 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2ac0089df0 con 0x7f2ac814a330 2024-03-06T07:23:05.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.080+0000 7f2accf13640 1 --2- 172.21.15.1:0/1740111779 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2aac06b4d0 0x7f2aac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.081+0000 7f2accf13640 1 --2- 172.21.15.1:0/1740111779 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2aac06b4d0 0x7f2aac06d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00ab60 tx=0x7f2abc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.085+0000 7f2ac57fa640 1 -- 172.21.15.1:0/1740111779 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ac8155c90 con 0x7f2ac814a330 2024-03-06T07:23:05.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.267+0000 7f2acf177640 1 -- 172.21.15.1:0/1740111779 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2ac80c85a0 con 0x7f2ac814a330 2024-03-06T07:23:05.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.267+0000 7f2ac57fa640 1 -- 172.21.15.1:0/1740111779 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v74) v1 ==== 84+0+209 (secure 0 0 0) 0x7f2ac80c85a0 con 0x7f2ac814a330 2024-03-06T07:23:05.270 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:05.270 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-06T07:23:05.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:05.270 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-06T07:23:05.270 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-06T07:23:05.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:05.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:05.270 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 11, 2024-03-06T07:23:05.270 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": "cephfs_metadata" 2024-03-06T07:23:05.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:05.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:05.270 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 12, 2024-03-06T07:23:05.271 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": "cephfs_data" 2024-03-06T07:23:05.271 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:05.271 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.269+0000 7f2acf177640 1 -- 172.21.15.1:0/1740111779 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2aac06b4d0 msgr2=0x7f2aac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.269+0000 7f2acf177640 1 --2- 172.21.15.1:0/1740111779 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2aac06b4d0 0x7f2aac06d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00ab60 tx=0x7f2abc008040 comp rx=0 tx=0).stop 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.269+0000 7f2acf177640 1 -- 172.21.15.1:0/1740111779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac814a330 msgr2=0x7f2ac81eb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.269+0000 7f2acf177640 1 --2- 172.21.15.1:0/1740111779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac814a330 0x7f2ac81eb740 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000b970 tx=0x7f2ac000be30 comp rx=0 tx=0).stop 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.269+0000 7f2acf177640 1 -- 172.21.15.1:0/1740111779 shutdown_connections 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.269+0000 7f2acf177640 1 --2- 172.21.15.1:0/1740111779 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2aac06b4d0 0x7f2aac06d990 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.269+0000 7f2acf177640 1 --2- 172.21.15.1:0/1740111779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac814a330 0x7f2ac81eb740 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.269+0000 7f2acf177640 1 --2- 172.21.15.1:0/1740111779 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac8152650 0x7f2ac81eb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.269+0000 7f2acf177640 1 --2- 172.21.15.1:0/1740111779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ac80bf9d0 0x7f2ac8149df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.269+0000 7f2acf177640 1 -- 172.21.15.1:0/1740111779 >> 172.21.15.1:0/1740111779 conn(0x7f2ac8066ee0 msgr2=0x7f2ac80c0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.269+0000 7f2acf177640 1 -- 172.21.15.1:0/1740111779 shutdown_connections 2024-03-06T07:23:05.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.269+0000 7f2acf177640 1 -- 172.21.15.1:0/1740111779 wait complete. 2024-03-06T07:23:05.282 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs2' 2024-03-06T07:23:05.282 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_metadata --pg_num_min 64 2024-03-06T07:23:05.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.385+0000 7fd874cfa640 1 Processor -- start 2024-03-06T07:23:05.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.385+0000 7fd874cfa640 1 -- start start 2024-03-06T07:23:05.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.386+0000 7fd874cfa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8700706f0 0x7fd87015d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.386+0000 7fd874cfa640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd870152d50 con 0x7fd8700c75f0 2024-03-06T07:23:05.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.386+0000 7fd874cfa640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd870151f90 con 0x7fd8700a7380 2024-03-06T07:23:05.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.386+0000 7fd874cfa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd870152270 con 0x7fd8700706f0 2024-03-06T07:23:05.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.386+0000 7fd86dd9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd8700c75f0 0x7fd87006ffe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45158/0 (socket says 172.21.15.1:45158) 2024-03-06T07:23:05.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.386+0000 7fd86dd9b640 1 -- 172.21.15.1:0/866841647 learned_addr learned my addr 172.21.15.1:0/866841647 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:05.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86ed9d640 1 --2- 172.21.15.1:0/866841647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8700706f0 0x7fd87015d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86d59a640 1 -- 172.21.15.1:0/866841647 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169800805 0 0) 0x7fd870152d50 con 0x7fd8700c75f0 2024-03-06T07:23:05.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86d59a640 1 -- 172.21.15.1:0/866841647 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd85c003520 con 0x7fd8700c75f0 2024-03-06T07:23:05.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86d59a640 1 -- 172.21.15.1:0/866841647 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2796811826 0 0) 0x7fd85c003520 con 0x7fd8700c75f0 2024-03-06T07:23:05.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86d59a640 1 -- 172.21.15.1:0/866841647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8700706f0 msgr2=0x7fd87015d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:05.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86d59a640 1 --2- 172.21.15.1:0/866841647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8700706f0 0x7fd87015d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86d59a640 1 -- 172.21.15.1:0/866841647 >> v1:172.21.15.98:6789/0 conn(0x7fd8700a7380 legacy=0x7fd87006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86d59a640 1 -- 172.21.15.1:0/866841647 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd870152550 con 0x7fd8700c75f0 2024-03-06T07:23:05.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86d59a640 1 -- 172.21.15.1:0/866841647 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fd858001670 con 0x7fd8700c75f0 2024-03-06T07:23:05.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86d59a640 1 -- 172.21.15.1:0/866841647 >> v1:172.21.15.1:6789/0 conn(0x7fd8700c75f0 legacy=0x7fd87006ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86d59a640 1 --2- 172.21.15.1:0/866841647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8700706f0 0x7fd85c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86d59a640 1 --2- 172.21.15.1:0/866841647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c003f20 0x7fd85c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86d59a640 1 --2- 172.21.15.1:0/866841647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c004d40 0x7fd85c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86d59a640 1 -- 172.21.15.1:0/866841647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd870152270 con 0x7fd85c004d40 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86d59a640 1 -- 172.21.15.1:0/866841647 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c00ba20 con 0x7fd85c003f20 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86d59a640 1 -- 172.21.15.1:0/866841647 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c00bc20 con 0x7fd8700706f0 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86ed9d640 1 --2- 172.21.15.1:0/866841647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c004d40 0x7fd85c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86dd9b640 1 --2- 172.21.15.1:0/866841647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c003f20 0x7fd85c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.387+0000 7fd86e59c640 1 --2- 172.21.15.1:0/866841647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8700706f0 0x7fd85c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.388+0000 7fd86ed9d640 1 -- 172.21.15.1:0/866841647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8700706f0 msgr2=0x7fd85c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.388+0000 7fd86ed9d640 1 --2- 172.21.15.1:0/866841647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8700706f0 0x7fd85c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.388+0000 7fd86ed9d640 1 -- 172.21.15.1:0/866841647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c003f20 msgr2=0x7fd85c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.388+0000 7fd86ed9d640 1 --2- 172.21.15.1:0/866841647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c003f20 0x7fd85c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.388+0000 7fd86ed9d640 1 -- 172.21.15.1:0/866841647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85c003520 con 0x7fd85c004d40 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.388+0000 7fd86ed9d640 1 --2- 172.21.15.1:0/866841647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c004d40 0x7fd85c0051c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fd860009ac0 tx=0x7fd85c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34fb28c55677933c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.388+0000 7fd86d59a640 1 -- 172.21.15.1:0/866841647 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86003b070 con 0x7fd85c004d40 2024-03-06T07:23:05.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.388+0000 7fd86d59a640 1 -- 172.21.15.1:0/866841647 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86002e8c0 con 0x7fd85c004d40 2024-03-06T07:23:05.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.388+0000 7fd86d59a640 1 -- 172.21.15.1:0/866841647 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860009690 con 0x7fd85c004d40 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.389+0000 7fd874cfa640 1 -- 172.21.15.1:0/866841647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c004d40 msgr2=0x7fd85c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.389+0000 7fd874cfa640 1 --2- 172.21.15.1:0/866841647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c004d40 0x7fd85c0051c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fd860009ac0 tx=0x7fd85c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.389+0000 7fd86ed9d640 1 -- 172.21.15.1:0/866841647 reap_dead start 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.389+0000 7fd874cfa640 1 -- 172.21.15.1:0/866841647 shutdown_connections 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.389+0000 7fd874cfa640 1 -- 172.21.15.1:0/866841647 >> 172.21.15.1:0/866841647 conn(0x7fd8700c45c0 msgr2=0x7fd8700c49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.389+0000 7fd874cfa640 1 -- 172.21.15.1:0/866841647 shutdown_connections 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.389+0000 7fd874cfa640 1 -- 172.21.15.1:0/866841647 wait complete. 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.391+0000 7fd874cfa640 1 Processor -- start 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.391+0000 7fd874cfa640 1 -- start start 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.392+0000 7fd874cfa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c004d40 0x7fd8701ea880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.392+0000 7fd874cfa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8701500d0 0x7fd87014c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.392+0000 7fd874cfa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8701e78f0 0x7fd8701e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.392+0000 7fd874cfa640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87006f7a0 con 0x7fd85c004d40 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.392+0000 7fd874cfa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8700c6bf0 con 0x7fd8701e78f0 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.392+0000 7fd874cfa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87006c810 con 0x7fd8701500d0 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.392+0000 7fd86e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c004d40 0x7fd8701ea880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.392+0000 7fd86e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c004d40 0x7fd8701ea880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42450/0 (socket says 172.21.15.1:42450) 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.392+0000 7fd86e59c640 1 -- 172.21.15.1:0/848328514 learned_addr learned my addr 172.21.15.1:0/848328514 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.392+0000 7fd86ed9d640 1 --2- 172.21.15.1:0/848328514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8701e78f0 0x7fd8701e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.392+0000 7fd86dd9b640 1 --2- 172.21.15.1:0/848328514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8701500d0 0x7fd87014c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.393+0000 7fd86e59c640 1 -- 172.21.15.1:0/848328514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8701500d0 msgr2=0x7fd87014c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.393+0000 7fd86e59c640 1 --2- 172.21.15.1:0/848328514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8701500d0 0x7fd87014c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.393+0000 7fd86e59c640 1 -- 172.21.15.1:0/848328514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8701e78f0 msgr2=0x7fd8701e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.393+0000 7fd86e59c640 1 --2- 172.21.15.1:0/848328514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8701e78f0 0x7fd8701e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.393+0000 7fd86e59c640 1 -- 172.21.15.1:0/848328514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8701e82b0 con 0x7fd85c004d40 2024-03-06T07:23:05.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.393+0000 7fd86dd9b640 1 --2- 172.21.15.1:0/848328514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8701500d0 0x7fd87014c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:05.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.393+0000 7fd86e59c640 1 --2- 172.21.15.1:0/848328514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c004d40 0x7fd8701ea880 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd86800fb60 tx=0x7fd86800d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.393+0000 7fd866ffd640 1 -- 172.21.15.1:0/848328514 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86800dc30 con 0x7fd85c004d40 2024-03-06T07:23:05.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.393+0000 7fd866ffd640 1 -- 172.21.15.1:0/848328514 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd868018070 con 0x7fd85c004d40 2024-03-06T07:23:05.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.393+0000 7fd874cfa640 1 -- 172.21.15.1:0/848328514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8701ede70 con 0x7fd85c004d40 2024-03-06T07:23:05.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.393+0000 7fd866ffd640 1 -- 172.21.15.1:0/848328514 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868003970 con 0x7fd85c004d40 2024-03-06T07:23:05.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.393+0000 7fd874cfa640 1 -- 172.21.15.1:0/848328514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8701eb140 con 0x7fd85c004d40 2024-03-06T07:23:05.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.394+0000 7fd874cfa640 1 -- 172.21.15.1:0/848328514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8700686e0 con 0x7fd85c004d40 2024-03-06T07:23:05.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.395+0000 7fd866ffd640 1 -- 172.21.15.1:0/848328514 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd868003b10 con 0x7fd85c004d40 2024-03-06T07:23:05.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.395+0000 7fd866ffd640 1 --2- 172.21.15.1:0/848328514 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd85006b4d0 0x7fd85006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.395+0000 7fd866ffd640 1 -- 172.21.15.1:0/848328514 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd86808a580 con 0x7fd85c004d40 2024-03-06T07:23:05.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.395+0000 7fd86dd9b640 1 --2- 172.21.15.1:0/848328514 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd85006b4d0 0x7fd85006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.396+0000 7fd86dd9b640 1 --2- 172.21.15.1:0/848328514 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd85006b4d0 0x7fd85006d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd858001050 tx=0x7fd858009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.399+0000 7fd866ffd640 1 -- 172.21.15.1:0/848328514 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd868059300 con 0x7fd85c004d40 2024-03-06T07:23:05.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.578+0000 7fd874cfa640 1 -- 172.21.15.1:0/848328514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd87006d170 con 0x7fd85c004d40 2024-03-06T07:23:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.706+0000 7fd866ffd640 1 -- 172.21.15.1:0/848328514 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64}]=0 pool 'cephfs2_metadata' created v75) v1 ==== 140+0+0 (secure 0 0 0) 0x7fd868058c00 con 0x7fd85c004d40 2024-03-06T07:23:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs2_metadata' created 2024-03-06T07:23:05.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.709+0000 7fd874cfa640 1 -- 172.21.15.1:0/848328514 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd85006b4d0 msgr2=0x7fd85006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.709+0000 7fd874cfa640 1 --2- 172.21.15.1:0/848328514 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd85006b4d0 0x7fd85006d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd858001050 tx=0x7fd858009040 comp rx=0 tx=0).stop 2024-03-06T07:23:05.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.709+0000 7fd874cfa640 1 -- 172.21.15.1:0/848328514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c004d40 msgr2=0x7fd8701ea880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.709+0000 7fd874cfa640 1 --2- 172.21.15.1:0/848328514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c004d40 0x7fd8701ea880 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd86800fb60 tx=0x7fd86800d520 comp rx=0 tx=0).stop 2024-03-06T07:23:05.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.709+0000 7fd874cfa640 1 -- 172.21.15.1:0/848328514 shutdown_connections 2024-03-06T07:23:05.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.709+0000 7fd874cfa640 1 --2- 172.21.15.1:0/848328514 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd85006b4d0 0x7fd85006d990 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.709+0000 7fd874cfa640 1 --2- 172.21.15.1:0/848328514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8701e78f0 0x7fd8701e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.709+0000 7fd874cfa640 1 --2- 172.21.15.1:0/848328514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8701500d0 0x7fd87014c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.709+0000 7fd874cfa640 1 --2- 172.21.15.1:0/848328514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c004d40 0x7fd8701ea880 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.709+0000 7fd874cfa640 1 -- 172.21.15.1:0/848328514 >> 172.21.15.1:0/848328514 conn(0x7fd8700c45c0 msgr2=0x7fd870070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.709+0000 7fd874cfa640 1 -- 172.21.15.1:0/848328514 shutdown_connections 2024-03-06T07:23:05.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.709+0000 7fd874cfa640 1 -- 172.21.15.1:0/848328514 wait complete. 2024-03-06T07:23:05.727 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:23:05.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.833+0000 7f0ba2dc9640 1 Processor -- start 2024-03-06T07:23:05.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.833+0000 7f0ba2dc9640 1 -- start start 2024-03-06T07:23:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.834+0000 7f0ba2dc9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0b9c0af6f0 0x7f0b9c0c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.834+0000 7f0ba2dc9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9c0c7180 con 0x7f0b9c152820 2024-03-06T07:23:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.834+0000 7f0ba2dc9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9c155460 con 0x7f0b9c0c7930 2024-03-06T07:23:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.834+0000 7f0ba2dc9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b9c0c6bc0 con 0x7f0b9c0af6f0 2024-03-06T07:23:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.834+0000 7f0ba0b65640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0b9c152820 0x7f0b9c0c0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45170/0 (socket says 172.21.15.1:45170) 2024-03-06T07:23:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.834+0000 7f0ba0b65640 1 -- 172.21.15.1:0/1785138576 learned_addr learned my addr 172.21.15.1:0/1785138576 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0b9bfff640 1 --2- 172.21.15.1:0/1785138576 >> v2:172.21.15.98:3300/0 conn(0x7f0b9c0af6f0 0x7f0b9c0c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0b9b7fe640 1 -- 172.21.15.1:0/1785138576 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198908912 0 0) 0x7f0b9c0c7180 con 0x7f0b9c152820 2024-03-06T07:23:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0b9b7fe640 1 -- 172.21.15.1:0/1785138576 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b8c003520 con 0x7f0b9c152820 2024-03-06T07:23:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0b9b7fe640 1 -- 172.21.15.1:0/1785138576 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4071834902 0 0) 0x7f0b8c003520 con 0x7f0b9c152820 2024-03-06T07:23:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0b9b7fe640 1 -- 172.21.15.1:0/1785138576 >> v2:172.21.15.98:3300/0 conn(0x7f0b9c0af6f0 msgr2=0x7f0b9c0c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0b9b7fe640 1 --2- 172.21.15.1:0/1785138576 >> v2:172.21.15.98:3300/0 conn(0x7f0b9c0af6f0 0x7f0b9c0c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0b9b7fe640 1 -- 172.21.15.1:0/1785138576 >> v1:172.21.15.98:6789/0 conn(0x7f0b9c0c7930 legacy=0x7f0b9c15fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0b9b7fe640 1 -- 172.21.15.1:0/1785138576 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c0c6ea0 con 0x7f0b9c152820 2024-03-06T07:23:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0b9b7fe640 1 -- 172.21.15.1:0/1785138576 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f0b90001670 con 0x7f0b9c152820 2024-03-06T07:23:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0b9b7fe640 1 -- 172.21.15.1:0/1785138576 >> v1:172.21.15.1:6789/0 conn(0x7f0b9c152820 legacy=0x7f0b9c0c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0b9b7fe640 1 --2- 172.21.15.1:0/1785138576 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b8c0039b0 0x7f0b8c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0b9b7fe640 1 --2- 172.21.15.1:0/1785138576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c004800 0x7f0b8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0b9b7fe640 1 --2- 172.21.15.1:0/1785138576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b8c0051c0 0x7f0b8c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0b9b7fe640 1 -- 172.21.15.1:0/1785138576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c0c6bc0 con 0x7f0b8c0051c0 2024-03-06T07:23:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0b9b7fe640 1 -- 172.21.15.1:0/1785138576 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8c00c060 con 0x7f0b8c004800 2024-03-06T07:23:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0b9b7fe640 1 -- 172.21.15.1:0/1785138576 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8c00c260 con 0x7f0b8c0039b0 2024-03-06T07:23:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0ba1366640 1 --2- 172.21.15.1:0/1785138576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b8c0051c0 0x7f0b8c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0b9bfff640 1 --2- 172.21.15.1:0/1785138576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c004800 0x7f0b8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0ba0b65640 1 --2- 172.21.15.1:0/1785138576 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b8c0039b0 0x7f0b8c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0ba1366640 1 -- 172.21.15.1:0/1785138576 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b8c0039b0 msgr2=0x7f0b8c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0ba1366640 1 --2- 172.21.15.1:0/1785138576 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b8c0039b0 0x7f0b8c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0ba1366640 1 -- 172.21.15.1:0/1785138576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c004800 msgr2=0x7f0b8c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0ba1366640 1 --2- 172.21.15.1:0/1785138576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c004800 0x7f0b8c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.835+0000 7f0ba1366640 1 -- 172.21.15.1:0/1785138576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8c003520 con 0x7f0b8c0051c0 2024-03-06T07:23:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.836+0000 7f0ba1366640 1 -- 172.21.15.1:0/1785138576 reap_dead start 2024-03-06T07:23:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.836+0000 7f0ba1366640 1 --2- 172.21.15.1:0/1785138576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b8c0051c0 0x7f0b8c00b9c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0b94001e50 tx=0x7f0b8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472ab5a8ed4f4d73 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.836+0000 7f0b9b7fe640 1 -- 172.21.15.1:0/1785138576 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b94001a70 con 0x7f0b8c0051c0 2024-03-06T07:23:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.836+0000 7f0b9b7fe640 1 -- 172.21.15.1:0/1785138576 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b94003bb0 con 0x7f0b8c0051c0 2024-03-06T07:23:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.836+0000 7f0b9b7fe640 1 -- 172.21.15.1:0/1785138576 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b940050d0 con 0x7f0b8c0051c0 2024-03-06T07:23:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.837+0000 7f0ba2dc9640 1 -- 172.21.15.1:0/1785138576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b8c0051c0 msgr2=0x7f0b8c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.837+0000 7f0ba2dc9640 1 --2- 172.21.15.1:0/1785138576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b8c0051c0 0x7f0b8c00b9c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0b94001e50 tx=0x7f0b8c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.837+0000 7f0ba2dc9640 1 -- 172.21.15.1:0/1785138576 shutdown_connections 2024-03-06T07:23:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.837+0000 7f0ba2dc9640 1 --2- 172.21.15.1:0/1785138576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b8c0051c0 0x7f0b8c00b9c0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.837+0000 7f0ba2dc9640 1 -- 172.21.15.1:0/1785138576 >> 172.21.15.1:0/1785138576 conn(0x7f0b9c072720 msgr2=0x7f0b9c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.837+0000 7f0ba2dc9640 1 -- 172.21.15.1:0/1785138576 shutdown_connections 2024-03-06T07:23:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.837+0000 7f0ba2dc9640 1 -- 172.21.15.1:0/1785138576 wait complete. 2024-03-06T07:23:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.838+0000 7f0ba2dc9640 1 Processor -- start 2024-03-06T07:23:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.838+0000 7f0ba2dc9640 1 -- start start 2024-03-06T07:23:05.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.839+0000 7f0ba2dc9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c0051c0 0x7f0b9c1edbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.839+0000 7f0ba2dc9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b9c14b960 0x7f0b9c1eaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.839+0000 7f0ba2dc9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b9c1eb3f0 0x7f0b9c14cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.839+0000 7f0ba2dc9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c152d80 con 0x7f0b9c14b960 2024-03-06T07:23:05.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.839+0000 7f0ba2dc9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c152c00 con 0x7f0b8c0051c0 2024-03-06T07:23:05.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.839+0000 7f0ba2dc9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c0afc40 con 0x7f0b9c1eb3f0 2024-03-06T07:23:05.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.839+0000 7f0b9bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b9c14b960 0x7f0b9c1eaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.839+0000 7f0b9bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b9c14b960 0x7f0b9c1eaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42462/0 (socket says 172.21.15.1:42462) 2024-03-06T07:23:05.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.839+0000 7f0b9bfff640 1 -- 172.21.15.1:0/2845865929 learned_addr learned my addr 172.21.15.1:0/2845865929 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:05.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.839+0000 7f0ba1366640 1 --2- 172.21.15.1:0/2845865929 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b9c1eb3f0 0x7f0b9c14cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.839+0000 7f0ba0b65640 1 --2- 172.21.15.1:0/2845865929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c0051c0 0x7f0b9c1edbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.839+0000 7f0b9bfff640 1 -- 172.21.15.1:0/2845865929 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b9c1eb3f0 msgr2=0x7f0b9c14cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.839+0000 7f0b9bfff640 1 --2- 172.21.15.1:0/2845865929 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b9c1eb3f0 0x7f0b9c14cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.839+0000 7f0b9bfff640 1 -- 172.21.15.1:0/2845865929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c0051c0 msgr2=0x7f0b9c1edbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.839+0000 7f0b9bfff640 1 --2- 172.21.15.1:0/2845865929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c0051c0 0x7f0b9c1edbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.839+0000 7f0b9bfff640 1 -- 172.21.15.1:0/2845865929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c1595b0 con 0x7f0b9c14b960 2024-03-06T07:23:05.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.839+0000 7f0b9bfff640 1 --2- 172.21.15.1:0/2845865929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b9c14b960 0x7f0b9c1eaeb0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f0b88011910 tx=0x7f0b88011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.840+0000 7f0b997fa640 1 -- 172.21.15.1:0/2845865929 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8800fa30 con 0x7f0b9c14b960 2024-03-06T07:23:05.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.840+0000 7f0b997fa640 1 -- 172.21.15.1:0/2845865929 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8800fbd0 con 0x7f0b9c14b960 2024-03-06T07:23:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.840+0000 7f0ba2dc9640 1 -- 172.21.15.1:0/2845865929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9c0aefd0 con 0x7f0b9c14b960 2024-03-06T07:23:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.840+0000 7f0b997fa640 1 -- 172.21.15.1:0/2845865929 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b88014440 con 0x7f0b9c14b960 2024-03-06T07:23:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.840+0000 7f0ba2dc9640 1 -- 172.21.15.1:0/2845865929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9c1ea680 con 0x7f0b9c14b960 2024-03-06T07:23:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.841+0000 7f0ba2dc9640 1 -- 172.21.15.1:0/2845865929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b9c0af1a0 con 0x7f0b9c14b960 2024-03-06T07:23:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.841+0000 7f0b997fa640 1 -- 172.21.15.1:0/2845865929 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b88012040 con 0x7f0b9c14b960 2024-03-06T07:23:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.841+0000 7f0b997fa640 1 --2- 172.21.15.1:0/2845865929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0b8006b4d0 0x7f0b8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.842+0000 7f0b997fa640 1 -- 172.21.15.1:0/2845865929 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6472+0+0 (secure 0 0 0) 0x7f0b8808a920 con 0x7f0b9c14b960 2024-03-06T07:23:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.842+0000 7f0ba0b65640 1 --2- 172.21.15.1:0/2845865929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0b8006b4d0 0x7f0b8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.842+0000 7f0ba0b65640 1 --2- 172.21.15.1:0/2845865929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0b8006b4d0 0x7f0b8006d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0b90007cf0 tx=0x7f0b900078f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:05.846+0000 7f0b997fa640 1 -- 172.21.15.1:0/2845865929 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b88019030 con 0x7f0b9c14b960 2024-03-06T07:23:06.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.186+0000 7f0ba2dc9640 1 -- 172.21.15.1:0/2845865929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0b9c0686e0 con 0x7f0b9c14b960 2024-03-06T07:23:06.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.708+0000 7f0b997fa640 1 -- 172.21.15.1:0/2845865929 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs2_data' created v76) v1 ==== 172+0+0 (secure 0 0 0) 0x7f0b88059530 con 0x7f0b9c14b960 2024-03-06T07:23:06.710 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs2_data' created 2024-03-06T07:23:06.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.711+0000 7f0ba2dc9640 1 -- 172.21.15.1:0/2845865929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0b8006b4d0 msgr2=0x7f0b8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.711+0000 7f0ba2dc9640 1 --2- 172.21.15.1:0/2845865929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0b8006b4d0 0x7f0b8006d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0b90007cf0 tx=0x7f0b900078f0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.711+0000 7f0ba2dc9640 1 -- 172.21.15.1:0/2845865929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b9c14b960 msgr2=0x7f0b9c1eaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.711+0000 7f0ba2dc9640 1 --2- 172.21.15.1:0/2845865929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b9c14b960 0x7f0b9c1eaeb0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f0b88011910 tx=0x7f0b88011d10 comp rx=0 tx=0).stop 2024-03-06T07:23:06.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.712+0000 7f0ba2dc9640 1 -- 172.21.15.1:0/2845865929 shutdown_connections 2024-03-06T07:23:06.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.712+0000 7f0ba2dc9640 1 --2- 172.21.15.1:0/2845865929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0b8006b4d0 0x7f0b8006d990 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.712+0000 7f0ba2dc9640 1 --2- 172.21.15.1:0/2845865929 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b9c1eb3f0 0x7f0b9c14cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.712+0000 7f0ba2dc9640 1 --2- 172.21.15.1:0/2845865929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b9c14b960 0x7f0b9c1eaeb0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.712+0000 7f0ba2dc9640 1 --2- 172.21.15.1:0/2845865929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c0051c0 0x7f0b9c1edbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.712+0000 7f0ba2dc9640 1 -- 172.21.15.1:0/2845865929 >> 172.21.15.1:0/2845865929 conn(0x7f0b9c072720 msgr2=0x7f0b9c150a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.712+0000 7f0ba2dc9640 1 -- 172.21.15.1:0/2845865929 shutdown_connections 2024-03-06T07:23:06.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.713+0000 7f0ba2dc9640 1 -- 172.21.15.1:0/2845865929 wait complete. 2024-03-06T07:23:06.735 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs2 cephfs2_metadata cephfs2_data 2024-03-06T07:23:06.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.839+0000 7f5758a4e640 1 Processor -- start 2024-03-06T07:23:06.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.839+0000 7f5758a4e640 1 -- start start 2024-03-06T07:23:06.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.840+0000 7f5758a4e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5754152650 0x7f5754155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.840+0000 7f5758a4e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57540c2fc0 con 0x7f57540bf9d0 2024-03-06T07:23:06.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.840+0000 7f5758a4e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5754158da0 con 0x7f57540c6b10 2024-03-06T07:23:06.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.840+0000 7f5758a4e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5754155800 con 0x7f5754152650 2024-03-06T07:23:06.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.840+0000 7f5752d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f57540bf9d0 0x7f57540c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45184/0 (socket says 172.21.15.1:45184) 2024-03-06T07:23:06.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.840+0000 7f5752d9d640 1 -- 172.21.15.1:0/4125056557 learned_addr learned my addr 172.21.15.1:0/4125056557 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:06.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.840+0000 7f575259c640 1 --2- 172.21.15.1:0/4125056557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5754152650 0x7f5754155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.840+0000 7f575159a640 1 -- 172.21.15.1:0/4125056557 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744623445 0 0) 0x7f57540c2fc0 con 0x7f57540bf9d0 2024-03-06T07:23:06.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.840+0000 7f575159a640 1 -- 172.21.15.1:0/4125056557 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f573c003520 con 0x7f57540bf9d0 2024-03-06T07:23:06.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.840+0000 7f575159a640 1 -- 172.21.15.1:0/4125056557 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 439679392 0 0) 0x7f573c003520 con 0x7f57540bf9d0 2024-03-06T07:23:06.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.840+0000 7f575159a640 1 -- 172.21.15.1:0/4125056557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5754152650 msgr2=0x7f5754155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.840+0000 7f575159a640 1 --2- 172.21.15.1:0/4125056557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5754152650 0x7f5754155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.840+0000 7f575159a640 1 -- 172.21.15.1:0/4125056557 >> v1:172.21.15.98:6789/0 conn(0x7f57540c6b10 legacy=0x7f5754158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.840+0000 7f575159a640 1 -- 172.21.15.1:0/4125056557 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57540c6690 con 0x7f57540bf9d0 2024-03-06T07:23:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.840+0000 7f575159a640 1 -- 172.21.15.1:0/4125056557 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f574c0019f0 con 0x7f57540bf9d0 2024-03-06T07:23:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.840+0000 7f575159a640 1 -- 172.21.15.1:0/4125056557 >> v1:172.21.15.1:6789/0 conn(0x7f57540bf9d0 legacy=0x7f57540c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.841+0000 7f575159a640 1 --2- 172.21.15.1:0/4125056557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f573c0039e0 0x7f573c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.841+0000 7f575159a640 1 --2- 172.21.15.1:0/4125056557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f573c004390 0x7f573c004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.841+0000 7f575159a640 1 --2- 172.21.15.1:0/4125056557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5754152650 0x7f573c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.841+0000 7f575159a640 1 -- 172.21.15.1:0/4125056557 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5754155800 con 0x7f573c004390 2024-03-06T07:23:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.841+0000 7f575159a640 1 -- 172.21.15.1:0/4125056557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c00ba70 con 0x7f573c0039e0 2024-03-06T07:23:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.841+0000 7f575159a640 1 -- 172.21.15.1:0/4125056557 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c00bc70 con 0x7f5754152650 2024-03-06T07:23:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.841+0000 7f5751d9b640 1 --2- 172.21.15.1:0/4125056557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f573c004390 0x7f573c004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.841+0000 7f575259c640 1 --2- 172.21.15.1:0/4125056557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f573c0039e0 0x7f573c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.841+0000 7f5752d9d640 1 --2- 172.21.15.1:0/4125056557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5754152650 0x7f573c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.841+0000 7f5751d9b640 1 -- 172.21.15.1:0/4125056557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5754152650 msgr2=0x7f573c005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.841+0000 7f5751d9b640 1 --2- 172.21.15.1:0/4125056557 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5754152650 0x7f573c005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.841+0000 7f5751d9b640 1 -- 172.21.15.1:0/4125056557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f573c0039e0 msgr2=0x7f573c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.841+0000 7f5751d9b640 1 --2- 172.21.15.1:0/4125056557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f573c0039e0 0x7f573c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.841+0000 7f5751d9b640 1 -- 172.21.15.1:0/4125056557 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c003520 con 0x7f573c004390 2024-03-06T07:23:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.842+0000 7f5751d9b640 1 --2- 172.21.15.1:0/4125056557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f573c004390 0x7f573c004860 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5748001860 tx=0x7f573c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2c3540420aebf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.842+0000 7f575159a640 1 -- 172.21.15.1:0/4125056557 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748038070 con 0x7f573c004390 2024-03-06T07:23:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.842+0000 7f575159a640 1 -- 172.21.15.1:0/4125056557 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5748033410 con 0x7f573c004390 2024-03-06T07:23:06.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.842+0000 7f575159a640 1 -- 172.21.15.1:0/4125056557 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748001430 con 0x7f573c004390 2024-03-06T07:23:06.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.842+0000 7f5758a4e640 1 -- 172.21.15.1:0/4125056557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f573c004390 msgr2=0x7f573c004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.842+0000 7f5758a4e640 1 --2- 172.21.15.1:0/4125056557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f573c004390 0x7f573c004860 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5748001860 tx=0x7f573c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:06.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.842+0000 7f5752d9d640 1 -- 172.21.15.1:0/4125056557 reap_dead start 2024-03-06T07:23:06.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.842+0000 7f5758a4e640 1 -- 172.21.15.1:0/4125056557 shutdown_connections 2024-03-06T07:23:06.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.842+0000 7f5758a4e640 1 -- 172.21.15.1:0/4125056557 >> 172.21.15.1:0/4125056557 conn(0x7f5754066ee0 msgr2=0x7f57540672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.843+0000 7f5758a4e640 1 -- 172.21.15.1:0/4125056557 shutdown_connections 2024-03-06T07:23:06.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.843+0000 7f5758a4e640 1 -- 172.21.15.1:0/4125056557 wait complete. 2024-03-06T07:23:06.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.844+0000 7f5758a4e640 1 Processor -- start 2024-03-06T07:23:06.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.844+0000 7f5758a4e640 1 -- start start 2024-03-06T07:23:06.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.844+0000 7f5758a4e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f573c004390 0x7f575414fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.844+0000 7f5758a4e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5754150ee0 0x7f5754151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f5758a4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57541457d0 0x7f575414ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f5758a4e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57541551b0 con 0x7f573c004390 2024-03-06T07:23:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f5758a4e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57540c6fa0 con 0x7f57541457d0 2024-03-06T07:23:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f5758a4e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5754155490 con 0x7f5754150ee0 2024-03-06T07:23:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f575259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f573c004390 0x7f575414fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f575259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f573c004390 0x7f575414fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42488/0 (socket says 172.21.15.1:42488) 2024-03-06T07:23:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f575259c640 1 -- 172.21.15.1:0/1101789907 learned_addr learned my addr 172.21.15.1:0/1101789907 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f5751d9b640 1 --2- 172.21.15.1:0/1101789907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5754150ee0 0x7f5754151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f5752d9d640 1 --2- 172.21.15.1:0/1101789907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57541457d0 0x7f575414ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f575259c640 1 -- 172.21.15.1:0/1101789907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5754150ee0 msgr2=0x7f5754151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f575259c640 1 --2- 172.21.15.1:0/1101789907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5754150ee0 0x7f5754151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f575259c640 1 -- 172.21.15.1:0/1101789907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57541457d0 msgr2=0x7f575414ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f575259c640 1 --2- 172.21.15.1:0/1101789907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57541457d0 0x7f575414ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f575259c640 1 -- 172.21.15.1:0/1101789907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5754150000 con 0x7f573c004390 2024-03-06T07:23:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f575259c640 1 --2- 172.21.15.1:0/1101789907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f573c004390 0x7f575414fac0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f5744011a00 tx=0x7f5744011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f5742ffd640 1 -- 172.21.15.1:0/1101789907 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574400fb50 con 0x7f573c004390 2024-03-06T07:23:06.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f5742ffd640 1 -- 172.21.15.1:0/1101789907 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574400fcf0 con 0x7f573c004390 2024-03-06T07:23:06.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f5742ffd640 1 -- 172.21.15.1:0/1101789907 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574400fea0 con 0x7f573c004390 2024-03-06T07:23:06.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f5758a4e640 1 -- 172.21.15.1:0/1101789907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57540ca730 con 0x7f573c004390 2024-03-06T07:23:06.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.845+0000 7f5758a4e640 1 -- 172.21.15.1:0/1101789907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57540ca450 con 0x7f573c004390 2024-03-06T07:23:06.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.846+0000 7f5758a4e640 1 -- 172.21.15.1:0/1101789907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575406b540 con 0x7f573c004390 2024-03-06T07:23:06.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.847+0000 7f5742ffd640 1 -- 172.21.15.1:0/1101789907 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5744012040 con 0x7f573c004390 2024-03-06T07:23:06.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.847+0000 7f5742ffd640 1 --2- 172.21.15.1:0/1101789907 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f573406b500 0x7f573406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.847+0000 7f5742ffd640 1 -- 172.21.15.1:0/1101789907 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6855+0+0 (secure 0 0 0) 0x7f574408ab90 con 0x7f573c004390 2024-03-06T07:23:06.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.848+0000 7f5751d9b640 1 --2- 172.21.15.1:0/1101789907 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f573406b500 0x7f573406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.848+0000 7f5751d9b640 1 --2- 172.21.15.1:0/1101789907 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f573406b500 0x7f573406d9c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5748035000 tx=0x7f5748009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:06.852+0000 7f5742ffd640 1 -- 172.21.15.1:0/1101789907 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f575406b540 con 0x7f573c004390 2024-03-06T07:23:07.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.072+0000 7f5758a4e640 1 -- 172.21.15.1:0/1101789907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"} v 0) v1 -- 0x7f5754067fd0 con 0x7f573c004390 2024-03-06T07:23:07.739 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 13 and data pool 14 2024-03-06T07:23:07.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.736+0000 7f5742ffd640 1 -- 172.21.15.1:0/1101789907 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"}]=0 new fs with metadata pool 13 and data pool 14 v55) v1 ==== 177+0+0 (secure 0 0 0) 0x7f5744019030 con 0x7f573c004390 2024-03-06T07:23:07.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.742+0000 7f5758a4e640 1 -- 172.21.15.1:0/1101789907 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f573406b500 msgr2=0x7f573406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.742+0000 7f5758a4e640 1 --2- 172.21.15.1:0/1101789907 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f573406b500 0x7f573406d9c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5748035000 tx=0x7f5748009030 comp rx=0 tx=0).stop 2024-03-06T07:23:07.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.742+0000 7f5758a4e640 1 -- 172.21.15.1:0/1101789907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f573c004390 msgr2=0x7f575414fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.742+0000 7f5758a4e640 1 --2- 172.21.15.1:0/1101789907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f573c004390 0x7f575414fac0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f5744011a00 tx=0x7f5744011ec0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.742+0000 7f5758a4e640 1 -- 172.21.15.1:0/1101789907 shutdown_connections 2024-03-06T07:23:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.742+0000 7f5758a4e640 1 --2- 172.21.15.1:0/1101789907 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f573406b500 0x7f573406d9c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.742+0000 7f5758a4e640 1 --2- 172.21.15.1:0/1101789907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57541457d0 0x7f575414ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.742+0000 7f5758a4e640 1 --2- 172.21.15.1:0/1101789907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5754150ee0 0x7f5754151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.742+0000 7f5758a4e640 1 --2- 172.21.15.1:0/1101789907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f573c004390 0x7f575414fac0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.742+0000 7f5758a4e640 1 -- 172.21.15.1:0/1101789907 >> 172.21.15.1:0/1101789907 conn(0x7f5754066ee0 msgr2=0x7f57540c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.743+0000 7f5758a4e640 1 -- 172.21.15.1:0/1101789907 shutdown_connections 2024-03-06T07:23:07.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.743+0000 7f5758a4e640 1 -- 172.21.15.1:0/1101789907 wait complete. 2024-03-06T07:23:07.763 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:07.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.862+0000 7fdfe924e640 1 Processor -- start 2024-03-06T07:23:07.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.862+0000 7fdfe924e640 1 -- start start 2024-03-06T07:23:07.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.863+0000 7fdfe924e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdfe40bf9d0 0x7fdfe4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.863+0000 7fdfe924e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe40c2fc0 con 0x7fdfe40c6b10 2024-03-06T07:23:07.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.863+0000 7fdfe924e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe4158da0 con 0x7fdfe4152650 2024-03-06T07:23:07.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.863+0000 7fdfe924e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfe4155800 con 0x7fdfe40bf9d0 2024-03-06T07:23:07.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.863+0000 7fdfe359e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdfe40bf9d0 0x7fdfe4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.863+0000 7fdfe259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdfe40c6b10 0x7fdfe40c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45192/0 (socket says 172.21.15.1:45192) 2024-03-06T07:23:07.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.863+0000 7fdfe359e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdfe40bf9d0 0x7fdfe4155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42490/0 (socket says 172.21.15.1:42490) 2024-03-06T07:23:07.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.863+0000 7fdfe259c640 1 -- 172.21.15.1:0/459812631 learned_addr learned my addr 172.21.15.1:0/459812631 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:07.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.863+0000 7fdfe359e640 1 -- 172.21.15.1:0/459812631 >> v1:172.21.15.98:6789/0 conn(0x7fdfe4152650 legacy=0x7fdfe4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:07.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.863+0000 7fdfe359e640 1 -- 172.21.15.1:0/459812631 >> v1:172.21.15.1:6789/0 conn(0x7fdfe40c6b10 legacy=0x7fdfe40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.863+0000 7fdfe1d9b640 1 -- 172.21.15.1:0/459812631 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904047170 0 0) 0x7fdfe40c2fc0 con 0x7fdfe40c6b10 2024-03-06T07:23:07.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.864+0000 7fdfe359e640 1 -- 172.21.15.1:0/459812631 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe40c6690 con 0x7fdfe40bf9d0 2024-03-06T07:23:07.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.864+0000 7fdfe359e640 1 --2- 172.21.15.1:0/459812631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe40bf9d0 0x7fdfe4155c90 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009d90 tx=0x7fdfdc02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc64d61cceba2df1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.864+0000 7fdfe1d9b640 1 -- 172.21.15.1:0/459812631 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc039070 con 0x7fdfe40bf9d0 2024-03-06T07:23:07.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.864+0000 7fdfe1d9b640 1 -- 172.21.15.1:0/459812631 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfdc02cbd0 con 0x7fdfe40bf9d0 2024-03-06T07:23:07.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.864+0000 7fdfe1d9b640 1 -- 172.21.15.1:0/459812631 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc034440 con 0x7fdfe40bf9d0 2024-03-06T07:23:07.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.865+0000 7fdfe924e640 1 -- 172.21.15.1:0/459812631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe40bf9d0 msgr2=0x7fdfe4155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.865+0000 7fdfe924e640 1 --2- 172.21.15.1:0/459812631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe40bf9d0 0x7fdfe4155c90 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009d90 tx=0x7fdfdc02be20 comp rx=0 tx=0).stop 2024-03-06T07:23:07.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.865+0000 7fdfe924e640 1 -- 172.21.15.1:0/459812631 shutdown_connections 2024-03-06T07:23:07.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.865+0000 7fdfe924e640 1 --2- 172.21.15.1:0/459812631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe40bf9d0 0x7fdfe4155c90 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.865+0000 7fdfe924e640 1 -- 172.21.15.1:0/459812631 >> 172.21.15.1:0/459812631 conn(0x7fdfe4066ee0 msgr2=0x7fdfe40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.865+0000 7fdfe924e640 1 -- 172.21.15.1:0/459812631 shutdown_connections 2024-03-06T07:23:07.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.865+0000 7fdfe924e640 1 -- 172.21.15.1:0/459812631 wait complete. 2024-03-06T07:23:07.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.866+0000 7fdfe924e640 1 Processor -- start 2024-03-06T07:23:07.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.866+0000 7fdfe924e640 1 -- start start 2024-03-06T07:23:07.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.867+0000 7fdfe924e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfe40bf9d0 0x7fdfe4153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.867+0000 7fdfe924e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4152650 0x7fdfe41eb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.867+0000 7fdfe924e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe41588d0 0x7fdfe41eb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.867+0000 7fdfe924e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe41551b0 con 0x7fdfe41588d0 2024-03-06T07:23:07.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.867+0000 7fdfe924e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe40c6fa0 con 0x7fdfe4152650 2024-03-06T07:23:07.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.867+0000 7fdfe924e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe4155490 con 0x7fdfe40bf9d0 2024-03-06T07:23:07.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.867+0000 7fdfe359e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe41588d0 0x7fdfe41eb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.867+0000 7fdfe359e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe41588d0 0x7fdfe41eb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42500/0 (socket says 172.21.15.1:42500) 2024-03-06T07:23:07.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.867+0000 7fdfe359e640 1 -- 172.21.15.1:0/3521975631 learned_addr learned my addr 172.21.15.1:0/3521975631 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:07.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.867+0000 7fdfe259c640 1 --2- 172.21.15.1:0/3521975631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4152650 0x7fdfe41eb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.867+0000 7fdfe2d9d640 1 --2- 172.21.15.1:0/3521975631 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfe40bf9d0 0x7fdfe4153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.867+0000 7fdfe359e640 1 -- 172.21.15.1:0/3521975631 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfe40bf9d0 msgr2=0x7fdfe4153cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.867+0000 7fdfe359e640 1 --2- 172.21.15.1:0/3521975631 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfe40bf9d0 0x7fdfe4153cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.868+0000 7fdfe359e640 1 -- 172.21.15.1:0/3521975631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4152650 msgr2=0x7fdfe41eb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.868+0000 7fdfe359e640 1 --2- 172.21.15.1:0/3521975631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4152650 0x7fdfe41eb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.868+0000 7fdfe359e640 1 -- 172.21.15.1:0/3521975631 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe41ee230 con 0x7fdfe41588d0 2024-03-06T07:23:07.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.868+0000 7fdfe359e640 1 --2- 172.21.15.1:0/3521975631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe41588d0 0x7fdfe41eb550 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009d90 tx=0x7fdfdc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.868+0000 7fdfcb7fe640 1 -- 172.21.15.1:0/3521975631 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc044020 con 0x7fdfe41588d0 2024-03-06T07:23:07.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.868+0000 7fdfcb7fe640 1 -- 172.21.15.1:0/3521975631 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfdc03f470 con 0x7fdfe41588d0 2024-03-06T07:23:07.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.868+0000 7fdfe924e640 1 -- 172.21.15.1:0/3521975631 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe41ee510 con 0x7fdfe41588d0 2024-03-06T07:23:07.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.868+0000 7fdfcb7fe640 1 -- 172.21.15.1:0/3521975631 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc039040 con 0x7fdfe41588d0 2024-03-06T07:23:07.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.868+0000 7fdfe924e640 1 -- 172.21.15.1:0/3521975631 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe41eead0 con 0x7fdfe41588d0 2024-03-06T07:23:07.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.869+0000 7fdfe924e640 1 -- 172.21.15.1:0/3521975631 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe4067fd0 con 0x7fdfe41588d0 2024-03-06T07:23:07.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.869+0000 7fdfcb7fe640 1 -- 172.21.15.1:0/3521975631 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdfdc047020 con 0x7fdfe41588d0 2024-03-06T07:23:07.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.870+0000 7fdfcb7fe640 1 --2- 172.21.15.1:0/3521975631 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfd006b4d0 0x7fdfd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.870+0000 7fdfcb7fe640 1 -- 172.21.15.1:0/3521975631 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fdfdc0ad2d0 con 0x7fdfe41588d0 2024-03-06T07:23:07.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.870+0000 7fdfe2d9d640 1 --2- 172.21.15.1:0/3521975631 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfd006b4d0 0x7fdfd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.871+0000 7fdfe2d9d640 1 --2- 172.21.15.1:0/3521975631 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfd006b4d0 0x7fdfd006d990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40c16a0 tx=0x7fdfd4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:07.878+0000 7fdfcb7fe640 1 -- 172.21.15.1:0/3521975631 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdfdc04e030 con 0x7fdfe41588d0 2024-03-06T07:23:08.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.060+0000 7fdfe924e640 1 -- 172.21.15.1:0/3521975631 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfe40b01a0 con 0x7fdfe41588d0 2024-03-06T07:23:08.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.062+0000 7fdfcb7fe640 1 -- 172.21.15.1:0/3521975631 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+20562 (secure 0 0 0) 0x7fdfdc07bd00 con 0x7fdfe41588d0 2024-03-06T07:23:08.064 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:08.064 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":77,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:23:07.074166+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:55.240857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:55.939811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:23:05.599046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:23:06.205539+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.064+0000 7fdfe924e640 1 -- 172.21.15.1:0/3521975631 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfd006b4d0 msgr2=0x7fdfd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.064+0000 7fdfe924e640 1 --2- 172.21.15.1:0/3521975631 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfd006b4d0 0x7fdfd006d990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40c16a0 tx=0x7fdfd4009040 comp rx=0 tx=0).stop 2024-03-06T07:23:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.064+0000 7fdfe924e640 1 -- 172.21.15.1:0/3521975631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe41588d0 msgr2=0x7fdfe41eb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.064+0000 7fdfe924e640 1 --2- 172.21.15.1:0/3521975631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe41588d0 0x7fdfe41eb550 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009d90 tx=0x7fdfdc004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.064+0000 7fdfe924e640 1 -- 172.21.15.1:0/3521975631 shutdown_connections 2024-03-06T07:23:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.064+0000 7fdfe924e640 1 --2- 172.21.15.1:0/3521975631 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfd006b4d0 0x7fdfd006d990 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.064+0000 7fdfe924e640 1 --2- 172.21.15.1:0/3521975631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfe41588d0 0x7fdfe41eb550 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.064+0000 7fdfe924e640 1 --2- 172.21.15.1:0/3521975631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4152650 0x7fdfe41eb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.064+0000 7fdfe924e640 1 --2- 172.21.15.1:0/3521975631 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdfe40bf9d0 0x7fdfe4153cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.064+0000 7fdfe924e640 1 -- 172.21.15.1:0/3521975631 >> 172.21.15.1:0/3521975631 conn(0x7fdfe4066ee0 msgr2=0x7fdfe40c5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.064+0000 7fdfe924e640 1 -- 172.21.15.1:0/3521975631 shutdown_connections 2024-03-06T07:23:08.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.064+0000 7fdfe924e640 1 -- 172.21.15.1:0/3521975631 wait complete. 2024-03-06T07:23:08.077 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:08.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.176+0000 7f7c22ac2640 1 Processor -- start 2024-03-06T07:23:08.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.176+0000 7f7c22ac2640 1 -- start start 2024-03-06T07:23:08.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.177+0000 7f7c22ac2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c1c152650 0x7f7c1c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.177+0000 7f7c22ac2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c1c0c6b10 0x7f7c1c158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.177+0000 7f7c22ac2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1c0c2fc0 con 0x7f7c1c0bf9d0 2024-03-06T07:23:08.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.177+0000 7f7c22ac2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c1c155800 con 0x7f7c1c152650 2024-03-06T07:23:08.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.177+0000 7f7c22ac2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c0c6690 con 0x7f7c1c0c6b10 2024-03-06T07:23:08.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.177+0000 7f7c2085e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c1c152650 0x7f7c1c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.177+0000 7f7c2085e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c1c152650 0x7f7c1c155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42502/0 (socket says 172.21.15.1:42502) 2024-03-06T07:23:08.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.177+0000 7f7c2085e640 1 -- 172.21.15.1:0/189430927 learned_addr learned my addr 172.21.15.1:0/189430927 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:08.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.178+0000 7f7c1bfff640 1 --2- 172.21.15.1:0/189430927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c1c0c6b10 0x7f7c1c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.178+0000 7f7c2085e640 1 -- 172.21.15.1:0/189430927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c1c0c6b10 msgr2=0x7f7c1c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:08.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.178+0000 7f7c2085e640 1 --2- 172.21.15.1:0/189430927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c1c0c6b10 0x7f7c1c158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.178+0000 7f7c2085e640 1 -- 172.21.15.1:0/189430927 >> v1:172.21.15.98:6789/0 conn(0x7f7c1c0bf9d0 legacy=0x7f7c1c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:08.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.178+0000 7f7c2085e640 1 -- 172.21.15.1:0/189430927 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c0c7280 con 0x7f7c1c152650 2024-03-06T07:23:08.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.178+0000 7f7c2085e640 1 --2- 172.21.15.1:0/189430927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c1c152650 0x7f7c1c155c90 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f7c10009870 tx=0x7f7c1002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa52b4d58e74421d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.179+0000 7f7c1b7fe640 1 -- 172.21.15.1:0/189430927 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c100317c0 con 0x7f7c1c152650 2024-03-06T07:23:08.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.179+0000 7f7c1b7fe640 1 -- 172.21.15.1:0/189430927 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c10037070 con 0x7f7c1c152650 2024-03-06T07:23:08.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.179+0000 7f7c1b7fe640 1 -- 172.21.15.1:0/189430927 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10031b80 con 0x7f7c1c152650 2024-03-06T07:23:08.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.179+0000 7f7c22ac2640 1 -- 172.21.15.1:0/189430927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c1c152650 msgr2=0x7f7c1c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.179+0000 7f7c22ac2640 1 --2- 172.21.15.1:0/189430927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c1c152650 0x7f7c1c155c90 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f7c10009870 tx=0x7f7c1002b860 comp rx=0 tx=0).stop 2024-03-06T07:23:08.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.179+0000 7f7c22ac2640 1 -- 172.21.15.1:0/189430927 shutdown_connections 2024-03-06T07:23:08.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.179+0000 7f7c22ac2640 1 --2- 172.21.15.1:0/189430927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c1c0c6b10 0x7f7c1c158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.179+0000 7f7c22ac2640 1 --2- 172.21.15.1:0/189430927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c1c152650 0x7f7c1c155c90 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.179+0000 7f7c22ac2640 1 -- 172.21.15.1:0/189430927 >> 172.21.15.1:0/189430927 conn(0x7f7c1c066ee0 msgr2=0x7f7c1c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.179+0000 7f7c22ac2640 1 -- 172.21.15.1:0/189430927 shutdown_connections 2024-03-06T07:23:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.179+0000 7f7c22ac2640 1 -- 172.21.15.1:0/189430927 wait complete. 2024-03-06T07:23:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.181+0000 7f7c22ac2640 1 Processor -- start 2024-03-06T07:23:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.181+0000 7f7c22ac2640 1 -- start start 2024-03-06T07:23:08.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.182+0000 7f7c22ac2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c1c0bf9d0 0x7f7c1c1ee480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.182+0000 7f7c22ac2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c1c152650 0x7f7c1c1495c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.182+0000 7f7c22ac2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c1c14c500 0x7f7c1c1eb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.182+0000 7f7c22ac2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c155490 con 0x7f7c1c0bf9d0 2024-03-06T07:23:08.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.182+0000 7f7c22ac2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c1551b0 con 0x7f7c1c152650 2024-03-06T07:23:08.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.182+0000 7f7c22ac2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c152a30 con 0x7f7c1c14c500 2024-03-06T07:23:08.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.182+0000 7f7c2085e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c1c0bf9d0 0x7f7c1c1ee480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.182+0000 7f7c2085e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c1c0bf9d0 0x7f7c1c1ee480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42508/0 (socket says 172.21.15.1:42508) 2024-03-06T07:23:08.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.182+0000 7f7c2085e640 1 -- 172.21.15.1:0/1732309853 learned_addr learned my addr 172.21.15.1:0/1732309853 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:08.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.183+0000 7f7c1bfff640 1 --2- 172.21.15.1:0/1732309853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c1c152650 0x7f7c1c1495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.183+0000 7f7c2105f640 1 --2- 172.21.15.1:0/1732309853 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c1c14c500 0x7f7c1c1eb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.183+0000 7f7c2085e640 1 -- 172.21.15.1:0/1732309853 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c1c14c500 msgr2=0x7f7c1c1eb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.183+0000 7f7c2085e640 1 --2- 172.21.15.1:0/1732309853 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c1c14c500 0x7f7c1c1eb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.183+0000 7f7c2085e640 1 -- 172.21.15.1:0/1732309853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c1c152650 msgr2=0x7f7c1c1495c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.183+0000 7f7c2085e640 1 --2- 172.21.15.1:0/1732309853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c1c152650 0x7f7c1c1495c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.183+0000 7f7c2085e640 1 -- 172.21.15.1:0/1732309853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c1de9e0 con 0x7f7c1c0bf9d0 2024-03-06T07:23:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.183+0000 7f7c2085e640 1 --2- 172.21.15.1:0/1732309853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c1c0bf9d0 0x7f7c1c1ee480 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f7c10009870 tx=0x7f7c100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.183+0000 7f7c197fa640 1 -- 172.21.15.1:0/1732309853 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10032650 con 0x7f7c1c0bf9d0 2024-03-06T07:23:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.183+0000 7f7c197fa640 1 -- 172.21.15.1:0/1732309853 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c100327f0 con 0x7f7c1c0bf9d0 2024-03-06T07:23:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.183+0000 7f7c22ac2640 1 -- 172.21.15.1:0/1732309853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c1c14cfb0 con 0x7f7c1c0bf9d0 2024-03-06T07:23:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.183+0000 7f7c197fa640 1 -- 172.21.15.1:0/1732309853 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10032be0 con 0x7f7c1c0bf9d0 2024-03-06T07:23:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.183+0000 7f7c22ac2640 1 -- 172.21.15.1:0/1732309853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1c149b00 con 0x7f7c1c0bf9d0 2024-03-06T07:23:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.184+0000 7f7c22ac2640 1 -- 172.21.15.1:0/1732309853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1c067fd0 con 0x7f7c1c0bf9d0 2024-03-06T07:23:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.185+0000 7f7c197fa640 1 -- 172.21.15.1:0/1732309853 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c1004c060 con 0x7f7c1c0bf9d0 2024-03-06T07:23:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.185+0000 7f7c197fa640 1 --2- 172.21.15.1:0/1732309853 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c0006b4d0 0x7f7c0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.185+0000 7f7c197fa640 1 -- 172.21.15.1:0/1732309853 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f7c100ac210 con 0x7f7c1c0bf9d0 2024-03-06T07:23:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.185+0000 7f7c1bfff640 1 --2- 172.21.15.1:0/1732309853 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c0006b4d0 0x7f7c0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.186+0000 7f7c1bfff640 1 --2- 172.21.15.1:0/1732309853 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c0006b4d0 0x7f7c0006d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c1de7c0 tx=0x7f7c08008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.190+0000 7f7c197fa640 1 -- 172.21.15.1:0/1732309853 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c10040070 con 0x7f7c1c0bf9d0 2024-03-06T07:23:08.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.371+0000 7f7c22ac2640 1 -- 172.21.15.1:0/1732309853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c1c0b06f0 con 0x7f7c1c0bf9d0 2024-03-06T07:23:08.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.372+0000 7f7c197fa640 1 -- 172.21.15.1:0/1732309853 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+20562 (secure 0 0 0) 0x7f7c1007ad60 con 0x7f7c1c0bf9d0 2024-03-06T07:23:08.374 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:08.375 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":77,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:23:07.074166+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:55.240857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:55.939811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:23:05.599046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:23:06.205539+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:08.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.374+0000 7f7c22ac2640 1 -- 172.21.15.1:0/1732309853 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c0006b4d0 msgr2=0x7f7c0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.374+0000 7f7c22ac2640 1 --2- 172.21.15.1:0/1732309853 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c0006b4d0 0x7f7c0006d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c1de7c0 tx=0x7f7c08008040 comp rx=0 tx=0).stop 2024-03-06T07:23:08.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.374+0000 7f7c22ac2640 1 -- 172.21.15.1:0/1732309853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c1c0bf9d0 msgr2=0x7f7c1c1ee480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.374+0000 7f7c22ac2640 1 --2- 172.21.15.1:0/1732309853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c1c0bf9d0 0x7f7c1c1ee480 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f7c10009870 tx=0x7f7c100046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.374+0000 7f7c22ac2640 1 -- 172.21.15.1:0/1732309853 shutdown_connections 2024-03-06T07:23:08.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.374+0000 7f7c22ac2640 1 --2- 172.21.15.1:0/1732309853 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c0006b4d0 0x7f7c0006d990 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.374+0000 7f7c22ac2640 1 --2- 172.21.15.1:0/1732309853 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c1c14c500 0x7f7c1c1eb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.374+0000 7f7c22ac2640 1 --2- 172.21.15.1:0/1732309853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c1c152650 0x7f7c1c1495c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.374+0000 7f7c22ac2640 1 --2- 172.21.15.1:0/1732309853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c1c0bf9d0 0x7f7c1c1ee480 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.374+0000 7f7c22ac2640 1 -- 172.21.15.1:0/1732309853 >> 172.21.15.1:0/1732309853 conn(0x7f7c1c066ee0 msgr2=0x7f7c1c1508a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.374+0000 7f7c22ac2640 1 -- 172.21.15.1:0/1732309853 shutdown_connections 2024-03-06T07:23:08.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.374+0000 7f7c22ac2640 1 -- 172.21.15.1:0/1732309853 wait complete. 2024-03-06T07:23:08.388 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 standby_count_wanted 0 2024-03-06T07:23:08.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.490+0000 7fd9db20e640 1 Processor -- start 2024-03-06T07:23:08.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.490+0000 7fd9db20e640 1 -- start start 2024-03-06T07:23:08.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.491+0000 7fd9db20e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd9d4152650 0x7fd9d4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.491+0000 7fd9db20e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d40c2fc0 con 0x7fd9d40c6b10 2024-03-06T07:23:08.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.491+0000 7fd9db20e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d4158da0 con 0x7fd9d40bf9d0 2024-03-06T07:23:08.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.491+0000 7fd9db20e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d4155800 con 0x7fd9d4152650 2024-03-06T07:23:08.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.491+0000 7fd9d8faa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd9d4152650 0x7fd9d4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.491+0000 7fd9d8faa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd9d4152650 0x7fd9d4155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42516/0 (socket says 172.21.15.1:42516) 2024-03-06T07:23:08.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.491+0000 7fd9d8faa640 1 -- 172.21.15.1:0/1108133626 learned_addr learned my addr 172.21.15.1:0/1108133626 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:08.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.492+0000 7fd9d8faa640 1 -- 172.21.15.1:0/1108133626 >> v1:172.21.15.98:6789/0 conn(0x7fd9d40bf9d0 legacy=0x7fd9d40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.492+0000 7fd9d8faa640 1 -- 172.21.15.1:0/1108133626 >> v1:172.21.15.1:6789/0 conn(0x7fd9d40c6b10 legacy=0x7fd9d4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.492+0000 7fd9d8faa640 1 -- 172.21.15.1:0/1108133626 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d40c6690 con 0x7fd9d4152650 2024-03-06T07:23:08.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.492+0000 7fd9d8faa640 1 --2- 172.21.15.1:0/1108133626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4152650 0x7fd9d4155c90 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8009870 tx=0x7fd9c802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa471928ba7d213 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.492+0000 7fd9d37fe640 1 -- 172.21.15.1:0/1108133626 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8034070 con 0x7fd9d4152650 2024-03-06T07:23:08.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.492+0000 7fd9d37fe640 1 -- 172.21.15.1:0/1108133626 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c8030860 con 0x7fd9d4152650 2024-03-06T07:23:08.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.492+0000 7fd9d37fe640 1 -- 172.21.15.1:0/1108133626 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8030c30 con 0x7fd9d4152650 2024-03-06T07:23:08.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.493+0000 7fd9db20e640 1 -- 172.21.15.1:0/1108133626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4152650 msgr2=0x7fd9d4155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.493+0000 7fd9db20e640 1 --2- 172.21.15.1:0/1108133626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4152650 0x7fd9d4155c90 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8009870 tx=0x7fd9c802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.493+0000 7fd9db20e640 1 -- 172.21.15.1:0/1108133626 shutdown_connections 2024-03-06T07:23:08.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.493+0000 7fd9db20e640 1 --2- 172.21.15.1:0/1108133626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4152650 0x7fd9d4155c90 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.493+0000 7fd9db20e640 1 -- 172.21.15.1:0/1108133626 >> 172.21.15.1:0/1108133626 conn(0x7fd9d4066ee0 msgr2=0x7fd9d40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.493+0000 7fd9db20e640 1 -- 172.21.15.1:0/1108133626 shutdown_connections 2024-03-06T07:23:08.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.493+0000 7fd9db20e640 1 -- 172.21.15.1:0/1108133626 wait complete. 2024-03-06T07:23:08.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.494+0000 7fd9db20e640 1 Processor -- start 2024-03-06T07:23:08.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.494+0000 7fd9db20e640 1 -- start start 2024-03-06T07:23:08.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.495+0000 7fd9db20e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d40bf9d0 0x7fd9d414b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.495+0000 7fd9db20e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4152650 0x7fd9d414e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.495+0000 7fd9db20e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d414fe70 0x7fd9d41502f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.495+0000 7fd9db20e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d41551b0 con 0x7fd9d4152650 2024-03-06T07:23:08.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.495+0000 7fd9db20e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d40c6fa0 con 0x7fd9d414fe70 2024-03-06T07:23:08.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.495+0000 7fd9db20e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d4155490 con 0x7fd9d40bf9d0 2024-03-06T07:23:08.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.495+0000 7fd9d3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4152650 0x7fd9d414e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.495+0000 7fd9d3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4152650 0x7fd9d414e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42530/0 (socket says 172.21.15.1:42530) 2024-03-06T07:23:08.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.495+0000 7fd9d3fff640 1 -- 172.21.15.1:0/3668830977 learned_addr learned my addr 172.21.15.1:0/3668830977 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:08.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.495+0000 7fd9d8faa640 1 --2- 172.21.15.1:0/3668830977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d40bf9d0 0x7fd9d414b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.495+0000 7fd9d97ab640 1 --2- 172.21.15.1:0/3668830977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d414fe70 0x7fd9d41502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.495+0000 7fd9d3fff640 1 -- 172.21.15.1:0/3668830977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d40bf9d0 msgr2=0x7fd9d414b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.495+0000 7fd9d3fff640 1 --2- 172.21.15.1:0/3668830977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d40bf9d0 0x7fd9d414b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.495+0000 7fd9d3fff640 1 -- 172.21.15.1:0/3668830977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d414fe70 msgr2=0x7fd9d41502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.495+0000 7fd9d3fff640 1 --2- 172.21.15.1:0/3668830977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d414fe70 0x7fd9d41502f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.495+0000 7fd9d3fff640 1 -- 172.21.15.1:0/3668830977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d4152e10 con 0x7fd9d4152650 2024-03-06T07:23:08.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.496+0000 7fd9d3fff640 1 --2- 172.21.15.1:0/3668830977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4152650 0x7fd9d414e680 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fd9c000c480 tx=0x7fd9c000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.496+0000 7fd9d17fa640 1 -- 172.21.15.1:0/3668830977 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c00052d0 con 0x7fd9d4152650 2024-03-06T07:23:08.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.496+0000 7fd9d17fa640 1 -- 172.21.15.1:0/3668830977 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c0005470 con 0x7fd9d4152650 2024-03-06T07:23:08.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.496+0000 7fd9db20e640 1 -- 172.21.15.1:0/3668830977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d4154500 con 0x7fd9d4152650 2024-03-06T07:23:08.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.496+0000 7fd9d17fa640 1 -- 172.21.15.1:0/3668830977 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0012870 con 0x7fd9d4152650 2024-03-06T07:23:08.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.496+0000 7fd9db20e640 1 -- 172.21.15.1:0/3668830977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d41547c0 con 0x7fd9d4152650 2024-03-06T07:23:08.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.497+0000 7fd9db20e640 1 -- 172.21.15.1:0/3668830977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d41549c0 con 0x7fd9d4152650 2024-03-06T07:23:08.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.497+0000 7fd9d17fa640 1 -- 172.21.15.1:0/3668830977 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9c0012a10 con 0x7fd9d4152650 2024-03-06T07:23:08.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.498+0000 7fd9d17fa640 1 --2- 172.21.15.1:0/3668830977 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd9b806b4d0 0x7fd9b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.498+0000 7fd9d17fa640 1 -- 172.21.15.1:0/3668830977 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fd9c008a2e0 con 0x7fd9d4152650 2024-03-06T07:23:08.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.498+0000 7fd9d8faa640 1 --2- 172.21.15.1:0/3668830977 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd9b806b4d0 0x7fd9b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.498+0000 7fd9d8faa640 1 --2- 172.21.15.1:0/3668830977 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd9b806b4d0 0x7fd9b806d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8036000 tx=0x7fd9c803b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.502+0000 7fd9d17fa640 1 -- 172.21.15.1:0/3668830977 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9d41549c0 con 0x7fd9d4152650 2024-03-06T07:23:08.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:08.771+0000 7fd9db20e640 1 -- 172.21.15.1:0/3668830977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd9d41542f0 con 0x7fd9d4152650 2024-03-06T07:23:09.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.743+0000 7fd9d17fa640 1 -- 172.21.15.1:0/3668830977 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"}]=0 v58) v1 ==== 119+0+0 (secure 0 0 0) 0x7fd9c0058d10 con 0x7fd9d4152650 2024-03-06T07:23:09.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.745+0000 7fd9db20e640 1 -- 172.21.15.1:0/3668830977 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd9b806b4d0 msgr2=0x7fd9b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.745+0000 7fd9db20e640 1 --2- 172.21.15.1:0/3668830977 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd9b806b4d0 0x7fd9b806d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8036000 tx=0x7fd9c803b040 comp rx=0 tx=0).stop 2024-03-06T07:23:09.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.745+0000 7fd9db20e640 1 -- 172.21.15.1:0/3668830977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4152650 msgr2=0x7fd9d414e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.745+0000 7fd9db20e640 1 --2- 172.21.15.1:0/3668830977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4152650 0x7fd9d414e680 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fd9c000c480 tx=0x7fd9c000c940 comp rx=0 tx=0).stop 2024-03-06T07:23:09.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.745+0000 7fd9db20e640 1 -- 172.21.15.1:0/3668830977 shutdown_connections 2024-03-06T07:23:09.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.745+0000 7fd9db20e640 1 --2- 172.21.15.1:0/3668830977 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd9b806b4d0 0x7fd9b806d990 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.745+0000 7fd9db20e640 1 --2- 172.21.15.1:0/3668830977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d414fe70 0x7fd9d41502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.745+0000 7fd9db20e640 1 --2- 172.21.15.1:0/3668830977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4152650 0x7fd9d414e680 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.745+0000 7fd9db20e640 1 --2- 172.21.15.1:0/3668830977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d40bf9d0 0x7fd9d414b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.745+0000 7fd9db20e640 1 -- 172.21.15.1:0/3668830977 >> 172.21.15.1:0/3668830977 conn(0x7fd9d4066ee0 msgr2=0x7fd9d40672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.745+0000 7fd9db20e640 1 -- 172.21.15.1:0/3668830977 shutdown_connections 2024-03-06T07:23:09.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.745+0000 7fd9db20e640 1 -- 172.21.15.1:0/3668830977 wait complete. 2024-03-06T07:23:09.757 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:23:09.758 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 allow_standby_replay false 2024-03-06T07:23:09.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.861+0000 7ffb51e50640 1 Processor -- start 2024-03-06T07:23:09.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.861+0000 7ffb51e50640 1 -- start start 2024-03-06T07:23:09.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.862+0000 7ffb51e50640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffb4c152650 0x7ffb4c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.862+0000 7ffb51e50640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4c0c2fc0 con 0x7ffb4c0bf9d0 2024-03-06T07:23:09.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.862+0000 7ffb51e50640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4c158da0 con 0x7ffb4c0c6b10 2024-03-06T07:23:09.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.862+0000 7ffb51e50640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb4c155800 con 0x7ffb4c152650 2024-03-06T07:23:09.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.862+0000 7ffb4bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffb4c0bf9d0 0x7ffb4c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45220/0 (socket says 172.21.15.1:45220) 2024-03-06T07:23:09.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.862+0000 7ffb4bfff640 1 -- 172.21.15.1:0/4237926157 learned_addr learned my addr 172.21.15.1:0/4237926157 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:09.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.862+0000 7ffb4b7fe640 1 --2- 172.21.15.1:0/4237926157 >> v2:172.21.15.98:3300/0 conn(0x7ffb4c152650 0x7ffb4c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.862+0000 7ffb4a7fc640 1 -- 172.21.15.1:0/4237926157 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404651873 0 0) 0x7ffb4c0c2fc0 con 0x7ffb4c0bf9d0 2024-03-06T07:23:09.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.862+0000 7ffb4a7fc640 1 -- 172.21.15.1:0/4237926157 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb38003520 con 0x7ffb4c0bf9d0 2024-03-06T07:23:09.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.862+0000 7ffb4b7fe640 1 -- 172.21.15.1:0/4237926157 >> v1:172.21.15.98:6789/0 conn(0x7ffb4c0c6b10 legacy=0x7ffb4c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.862+0000 7ffb4b7fe640 1 -- 172.21.15.1:0/4237926157 >> v1:172.21.15.1:6789/0 conn(0x7ffb4c0bf9d0 legacy=0x7ffb4c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.862+0000 7ffb4a7fc640 1 -- 172.21.15.1:0/4237926157 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462245800 0 0) 0x7ffb4c158da0 con 0x7ffb4c0c6b10 2024-03-06T07:23:09.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.862+0000 7ffb4b7fe640 1 -- 172.21.15.1:0/4237926157 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c0c6690 con 0x7ffb4c152650 2024-03-06T07:23:09.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.863+0000 7ffb4b7fe640 1 --2- 172.21.15.1:0/4237926157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb4c152650 0x7ffb4c155c90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ffb340097a0 tx=0x7ffb3402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95f3da872155779 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.863+0000 7ffb4a7fc640 1 -- 172.21.15.1:0/4237926157 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34035020 con 0x7ffb4c152650 2024-03-06T07:23:09.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.863+0000 7ffb4a7fc640 1 -- 172.21.15.1:0/4237926157 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb34031830 con 0x7ffb4c152650 2024-03-06T07:23:09.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.863+0000 7ffb4a7fc640 1 -- 172.21.15.1:0/4237926157 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34031c00 con 0x7ffb4c152650 2024-03-06T07:23:09.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.864+0000 7ffb51e50640 1 -- 172.21.15.1:0/4237926157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb4c152650 msgr2=0x7ffb4c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.864+0000 7ffb51e50640 1 --2- 172.21.15.1:0/4237926157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb4c152650 0x7ffb4c155c90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ffb340097a0 tx=0x7ffb3402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.864+0000 7ffb51e50640 1 -- 172.21.15.1:0/4237926157 shutdown_connections 2024-03-06T07:23:09.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.864+0000 7ffb51e50640 1 --2- 172.21.15.1:0/4237926157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb4c152650 0x7ffb4c155c90 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.864+0000 7ffb51e50640 1 -- 172.21.15.1:0/4237926157 >> 172.21.15.1:0/4237926157 conn(0x7ffb4c066ee0 msgr2=0x7ffb4c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.864+0000 7ffb51e50640 1 -- 172.21.15.1:0/4237926157 shutdown_connections 2024-03-06T07:23:09.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.864+0000 7ffb51e50640 1 -- 172.21.15.1:0/4237926157 wait complete. 2024-03-06T07:23:09.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.865+0000 7ffb51e50640 1 Processor -- start 2024-03-06T07:23:09.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.865+0000 7ffb51e50640 1 -- start start 2024-03-06T07:23:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb51e50640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c0bf9d0 0x7ffb4c14d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb51e50640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb4c152650 0x7ffb4c14ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb51e50640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb4c0ca2e0 0x7ffb4c14b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb51e50640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c1551b0 con 0x7ffb4c0bf9d0 2024-03-06T07:23:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb51e50640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c0c6fa0 con 0x7ffb4c0ca2e0 2024-03-06T07:23:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb51e50640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c155490 con 0x7ffb4c152650 2024-03-06T07:23:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb4b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c0bf9d0 0x7ffb4c14d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb4b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c0bf9d0 0x7ffb4c14d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42542/0 (socket says 172.21.15.1:42542) 2024-03-06T07:23:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb4b7fe640 1 -- 172.21.15.1:0/497099740 learned_addr learned my addr 172.21.15.1:0/497099740 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb4bfff640 1 --2- 172.21.15.1:0/497099740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb4c0ca2e0 0x7ffb4c14b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb4affd640 1 --2- 172.21.15.1:0/497099740 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb4c152650 0x7ffb4c14ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb4b7fe640 1 -- 172.21.15.1:0/497099740 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb4c152650 msgr2=0x7ffb4c14ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb4b7fe640 1 --2- 172.21.15.1:0/497099740 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb4c152650 0x7ffb4c14ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb4b7fe640 1 -- 172.21.15.1:0/497099740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb4c0ca2e0 msgr2=0x7ffb4c14b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb4b7fe640 1 --2- 172.21.15.1:0/497099740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb4c0ca2e0 0x7ffb4c14b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb4b7fe640 1 -- 172.21.15.1:0/497099740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c156530 con 0x7ffb4c0bf9d0 2024-03-06T07:23:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb4b7fe640 1 --2- 172.21.15.1:0/497099740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c0bf9d0 0x7ffb4c14d1f0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7ffb340098c0 tx=0x7ffb340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.866+0000 7ffb43fff640 1 -- 172.21.15.1:0/497099740 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34035040 con 0x7ffb4c0bf9d0 2024-03-06T07:23:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.867+0000 7ffb43fff640 1 -- 172.21.15.1:0/497099740 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3403a840 con 0x7ffb4c0bf9d0 2024-03-06T07:23:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.867+0000 7ffb51e50640 1 -- 172.21.15.1:0/497099740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4c0c61f0 con 0x7ffb4c0bf9d0 2024-03-06T07:23:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.867+0000 7ffb43fff640 1 -- 172.21.15.1:0/497099740 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3403ac10 con 0x7ffb4c0bf9d0 2024-03-06T07:23:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.867+0000 7ffb51e50640 1 -- 172.21.15.1:0/497099740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4c0c9b70 con 0x7ffb4c0bf9d0 2024-03-06T07:23:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.868+0000 7ffb51e50640 1 -- 172.21.15.1:0/497099740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb4c0b06f0 con 0x7ffb4c0bf9d0 2024-03-06T07:23:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.868+0000 7ffb43fff640 1 -- 172.21.15.1:0/497099740 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb34038020 con 0x7ffb4c0bf9d0 2024-03-06T07:23:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.869+0000 7ffb43fff640 1 --2- 172.21.15.1:0/497099740 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb2c06b4d0 0x7ffb2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.869+0000 7ffb43fff640 1 -- 172.21.15.1:0/497099740 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7ffb340ac700 con 0x7ffb4c0bf9d0 2024-03-06T07:23:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.869+0000 7ffb4affd640 1 --2- 172.21.15.1:0/497099740 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb2c06b4d0 0x7ffb2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.869+0000 7ffb4affd640 1 --2- 172.21.15.1:0/497099740 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb2c06b4d0 0x7ffb2c06d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0c39d0 tx=0x7ffb3c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:09.873+0000 7ffb43fff640 1 -- 172.21.15.1:0/497099740 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb3407aae0 con 0x7ffb4c0bf9d0 2024-03-06T07:23:10.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.094+0000 7ffb51e50640 1 -- 172.21.15.1:0/497099740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ffb4c14b880 con 0x7ffb4c0bf9d0 2024-03-06T07:23:10.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.747+0000 7ffb43fff640 1 -- 172.21.15.1:0/497099740 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"}]=0 v59) v1 ==== 123+0+0 (secure 0 0 0) 0x7ffb3407eb80 con 0x7ffb4c0bf9d0 2024-03-06T07:23:10.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.749+0000 7ffb51e50640 1 -- 172.21.15.1:0/497099740 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb2c06b4d0 msgr2=0x7ffb2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.749+0000 7ffb51e50640 1 --2- 172.21.15.1:0/497099740 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb2c06b4d0 0x7ffb2c06d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0c39d0 tx=0x7ffb3c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:10.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.749+0000 7ffb51e50640 1 -- 172.21.15.1:0/497099740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c0bf9d0 msgr2=0x7ffb4c14d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.749+0000 7ffb51e50640 1 --2- 172.21.15.1:0/497099740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c0bf9d0 0x7ffb4c14d1f0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7ffb340098c0 tx=0x7ffb340046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.749+0000 7ffb51e50640 1 -- 172.21.15.1:0/497099740 shutdown_connections 2024-03-06T07:23:10.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.749+0000 7ffb51e50640 1 --2- 172.21.15.1:0/497099740 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb2c06b4d0 0x7ffb2c06d990 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.749+0000 7ffb51e50640 1 --2- 172.21.15.1:0/497099740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb4c0ca2e0 0x7ffb4c14b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.749+0000 7ffb51e50640 1 --2- 172.21.15.1:0/497099740 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb4c152650 0x7ffb4c14ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.749+0000 7ffb51e50640 1 --2- 172.21.15.1:0/497099740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c0bf9d0 0x7ffb4c14d1f0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.749+0000 7ffb51e50640 1 -- 172.21.15.1:0/497099740 >> 172.21.15.1:0/497099740 conn(0x7ffb4c066ee0 msgr2=0x7ffb4c0c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.749+0000 7ffb51e50640 1 -- 172.21.15.1:0/497099740 shutdown_connections 2024-03-06T07:23:10.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.749+0000 7ffb51e50640 1 -- 172.21.15.1:0/497099740 wait complete. 2024-03-06T07:23:10.766 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs2' 2024-03-06T07:23:10.766 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:10.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.866+0000 7f55b9a39640 1 Processor -- start 2024-03-06T07:23:10.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.866+0000 7f55b9a39640 1 -- start start 2024-03-06T07:23:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.867+0000 7f55b9a39640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55b40c3eb0 0x7f55b40c7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.867+0000 7f55b9a39640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f55b40c7940 0x7f55b415fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.867+0000 7f55b9a39640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b40c71c0 con 0x7f55b406d060 2024-03-06T07:23:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.867+0000 7f55b9a39640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b40c59b0 con 0x7f55b40c7940 2024-03-06T07:23:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.867+0000 7f55b9a39640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b40c4c00 con 0x7f55b40c3eb0 2024-03-06T07:23:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.867+0000 7f55b37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f55b40c7940 0x7f55b415fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.867+0000 7f55b37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f55b40c7940 0x7f55b415fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42548/0 (socket says 172.21.15.1:42548) 2024-03-06T07:23:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.867+0000 7f55b27fc640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f55b406d060 0x7f55b41558b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:52510/0 (socket says 172.21.15.1:52510) 2024-03-06T07:23:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.867+0000 7f55b37fe640 1 -- 172.21.15.1:0/1537184907 learned_addr learned my addr 172.21.15.1:0/1537184907 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.867+0000 7f55b2ffd640 1 --2- 172.21.15.1:0/1537184907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55b40c3eb0 0x7f55b40c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.867+0000 7f55b1ffb640 1 -- 172.21.15.1:0/1537184907 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897088675 0 0) 0x7f55b40c71c0 con 0x7f55b406d060 2024-03-06T07:23:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.867+0000 7f55b37fe640 1 -- 172.21.15.1:0/1537184907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55b40c3eb0 msgr2=0x7f55b40c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.867+0000 7f55b37fe640 1 --2- 172.21.15.1:0/1537184907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55b40c3eb0 0x7f55b40c7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.867+0000 7f55b37fe640 1 -- 172.21.15.1:0/1537184907 >> v1:172.21.15.98:6789/0 conn(0x7f55b406d060 legacy=0x7f55b41558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.867+0000 7f55b37fe640 1 -- 172.21.15.1:0/1537184907 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b40c4e50 con 0x7f55b40c7940 2024-03-06T07:23:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.867+0000 7f55b2ffd640 1 --2- 172.21.15.1:0/1537184907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55b40c3eb0 0x7f55b40c7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.868+0000 7f55b37fe640 1 --2- 172.21.15.1:0/1537184907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55b40c7940 0x7f55b415fa90 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009e50 tx=0x7f55ac02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63250f449fc00ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.868+0000 7f55b1ffb640 1 -- 172.21.15.1:0/1537184907 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac02c9e0 con 0x7f55b40c7940 2024-03-06T07:23:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.868+0000 7f55b1ffb640 1 -- 172.21.15.1:0/1537184907 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac02cb80 con 0x7f55b40c7940 2024-03-06T07:23:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.868+0000 7f55b1ffb640 1 -- 172.21.15.1:0/1537184907 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac0343f0 con 0x7f55b40c7940 2024-03-06T07:23:10.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.869+0000 7f55b9a39640 1 -- 172.21.15.1:0/1537184907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55b40c7940 msgr2=0x7f55b415fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.869+0000 7f55b9a39640 1 --2- 172.21.15.1:0/1537184907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55b40c7940 0x7f55b415fa90 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009e50 tx=0x7f55ac02be40 comp rx=0 tx=0).stop 2024-03-06T07:23:10.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.869+0000 7f55b9a39640 1 -- 172.21.15.1:0/1537184907 shutdown_connections 2024-03-06T07:23:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.869+0000 7f55b9a39640 1 --2- 172.21.15.1:0/1537184907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55b40c7940 0x7f55b415fa90 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.869+0000 7f55b9a39640 1 --2- 172.21.15.1:0/1537184907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55b40c3eb0 0x7f55b40c7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.869+0000 7f55b9a39640 1 -- 172.21.15.1:0/1537184907 >> 172.21.15.1:0/1537184907 conn(0x7f55b4066d00 msgr2=0x7f55b4067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.869+0000 7f55b9a39640 1 -- 172.21.15.1:0/1537184907 shutdown_connections 2024-03-06T07:23:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.869+0000 7f55b9a39640 1 -- 172.21.15.1:0/1537184907 wait complete. 2024-03-06T07:23:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.871+0000 7f55b9a39640 1 Processor -- start 2024-03-06T07:23:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.871+0000 7f55b9a39640 1 -- start start 2024-03-06T07:23:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.872+0000 7f55b9a39640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55b406d060 0x7f55b414dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.872+0000 7f55b9a39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b40c3eb0 0x7f55b41547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.872+0000 7f55b9a39640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55b40c7940 0x7f55b41431a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.872+0000 7f55b9a39640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b40c4290 con 0x7f55b406d060 2024-03-06T07:23:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.872+0000 7f55b9a39640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b4155420 con 0x7f55b40c3eb0 2024-03-06T07:23:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.872+0000 7f55b9a39640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b40abd60 con 0x7f55b40c7940 2024-03-06T07:23:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.872+0000 7f55b2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55b406d060 0x7f55b414dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.872+0000 7f55b2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55b406d060 0x7f55b414dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42560/0 (socket says 172.21.15.1:42560) 2024-03-06T07:23:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.872+0000 7f55b2ffd640 1 -- 172.21.15.1:0/3420935782 learned_addr learned my addr 172.21.15.1:0/3420935782 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.872+0000 7f55b27fc640 1 --2- 172.21.15.1:0/3420935782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b40c3eb0 0x7f55b41547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.872+0000 7f55b37fe640 1 --2- 172.21.15.1:0/3420935782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55b40c7940 0x7f55b41431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.872+0000 7f55b2ffd640 1 -- 172.21.15.1:0/3420935782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55b40c7940 msgr2=0x7f55b41431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.872+0000 7f55b2ffd640 1 --2- 172.21.15.1:0/3420935782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55b40c7940 0x7f55b41431a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.872+0000 7f55b2ffd640 1 -- 172.21.15.1:0/3420935782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b40c3eb0 msgr2=0x7f55b41547c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.872+0000 7f55b2ffd640 1 --2- 172.21.15.1:0/3420935782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b40c3eb0 0x7f55b41547c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.872+0000 7f55b2ffd640 1 -- 172.21.15.1:0/3420935782 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b41ebaa0 con 0x7f55b406d060 2024-03-06T07:23:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.873+0000 7f55b2ffd640 1 --2- 172.21.15.1:0/3420935782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55b406d060 0x7f55b414dd20 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f55a400e990 tx=0x7f55a400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.873+0000 7f559bfff640 1 -- 172.21.15.1:0/3420935782 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a4017070 con 0x7f55b406d060 2024-03-06T07:23:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.873+0000 7f559bfff640 1 -- 172.21.15.1:0/3420935782 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a400c9e0 con 0x7f55b406d060 2024-03-06T07:23:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.873+0000 7f55b9a39640 1 -- 172.21.15.1:0/3420935782 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b41eaf20 con 0x7f55b406d060 2024-03-06T07:23:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.873+0000 7f559bfff640 1 -- 172.21.15.1:0/3420935782 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a400cdb0 con 0x7f55b406d060 2024-03-06T07:23:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.873+0000 7f55b9a39640 1 -- 172.21.15.1:0/3420935782 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b41eb4e0 con 0x7f55b406d060 2024-03-06T07:23:10.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.874+0000 7f55b9a39640 1 -- 172.21.15.1:0/3420935782 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b4067fd0 con 0x7f55b406d060 2024-03-06T07:23:10.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.874+0000 7f559bfff640 1 -- 172.21.15.1:0/3420935782 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55a401a070 con 0x7f55b406d060 2024-03-06T07:23:10.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.875+0000 7f559bfff640 1 --2- 172.21.15.1:0/3420935782 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55a006b4d0 0x7f55a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.875+0000 7f559bfff640 1 -- 172.21.15.1:0/3420935782 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f55a408a460 con 0x7f55b406d060 2024-03-06T07:23:10.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.875+0000 7f55b27fc640 1 --2- 172.21.15.1:0/3420935782 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55a006b4d0 0x7f55a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.875+0000 7f55b27fc640 1 --2- 172.21.15.1:0/3420935782 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55a006b4d0 0x7f55a006d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f55b4153e90 tx=0x7f55a8007de0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:10.883+0000 7f559bfff640 1 -- 172.21.15.1:0/3420935782 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f55a4058840 con 0x7f55b406d060 2024-03-06T07:23:11.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.130+0000 7f55b9a39640 1 -- 172.21.15.1:0/3420935782 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55b40c5b30 con 0x7f55b406d060 2024-03-06T07:23:11.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.130+0000 7f559bfff640 1 -- 172.21.15.1:0/3420935782 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6250 (secure 0 0 0) 0x7f55a405c8e0 con 0x7f55b406d060 2024-03-06T07:23:11.133 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:11.133 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":59,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.1:6835/2685225325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2685225325},{"type":"v1","addr":"172.21.15.1:6835","nonce":2685225325}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.98:6836/4069609682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4069609682},{"type":"v1","addr":"172.21.15.98:6836","nonce":4069609682}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.969135+0000","modified":"2024-03-06T07:22:58.980537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6405},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6405":{"gid":6405,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1494116991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1494116991},{"type":"v1","addr":"172.21.15.1:6834","nonce":1494116991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6405,"qdb_cluster":[6405]},"id":5},{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:07.074540+0000","modified":"2024-03-06T07:23:10.095697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5845":{"gid":5845,"name":"d","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/114998998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":114998998},{"type":"v1","addr":"172.21.15.98:6837","nonce":114998998}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5845,"qdb_cluster":[5845]},"id":6}]} 2024-03-06T07:23:11.134 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 59 2024-03-06T07:23:11.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.132+0000 7f55b9a39640 1 -- 172.21.15.1:0/3420935782 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55a006b4d0 msgr2=0x7f55a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.132+0000 7f55b9a39640 1 --2- 172.21.15.1:0/3420935782 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55a006b4d0 0x7f55a006d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f55b4153e90 tx=0x7f55a8007de0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.132+0000 7f55b9a39640 1 -- 172.21.15.1:0/3420935782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55b406d060 msgr2=0x7f55b414dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.132+0000 7f55b9a39640 1 --2- 172.21.15.1:0/3420935782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55b406d060 0x7f55b414dd20 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f55a400e990 tx=0x7f55a400ee50 comp rx=0 tx=0).stop 2024-03-06T07:23:11.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.132+0000 7f55b9a39640 1 -- 172.21.15.1:0/3420935782 shutdown_connections 2024-03-06T07:23:11.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.132+0000 7f55b9a39640 1 --2- 172.21.15.1:0/3420935782 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55a006b4d0 0x7f55a006d990 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.132+0000 7f55b9a39640 1 --2- 172.21.15.1:0/3420935782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55b40c7940 0x7f55b41431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.132+0000 7f55b9a39640 1 --2- 172.21.15.1:0/3420935782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b40c3eb0 0x7f55b41547c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.132+0000 7f55b9a39640 1 --2- 172.21.15.1:0/3420935782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55b406d060 0x7f55b414dd20 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.132+0000 7f55b9a39640 1 -- 172.21.15.1:0/3420935782 >> 172.21.15.1:0/3420935782 conn(0x7f55b4066d00 msgr2=0x7f55b41536c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.132+0000 7f55b9a39640 1 -- 172.21.15.1:0/3420935782 shutdown_connections 2024-03-06T07:23:11.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.132+0000 7f55b9a39640 1 -- 172.21.15.1:0/3420935782 wait complete. 2024-03-06T07:23:11.146 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:11.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.245+0000 7fd75138a640 1 Processor -- start 2024-03-06T07:23:11.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.245+0000 7fd75138a640 1 -- start start 2024-03-06T07:23:11.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.246+0000 7fd75138a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd74c152690 0x7fd74c155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.246+0000 7fd75138a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd74c156cd0 0x7fd74c157160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.246+0000 7fd75138a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd74c1561e0 con 0x7fd74c0c6730 2024-03-06T07:23:11.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.246+0000 7fd75138a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd74c0c3300 con 0x7fd74c156cd0 2024-03-06T07:23:11.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.246+0000 7fd75138a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c0c35a0 con 0x7fd74c152690 2024-03-06T07:23:11.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.246+0000 7fd74b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd74c156cd0 0x7fd74c157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.246+0000 7fd74b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd74c156cd0 0x7fd74c157160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42562/0 (socket says 172.21.15.1:42562) 2024-03-06T07:23:11.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.246+0000 7fd74b7fe640 1 -- 172.21.15.1:0/1314890335 learned_addr learned my addr 172.21.15.1:0/1314890335 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:11.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.246+0000 7fd74affd640 1 --2- 172.21.15.1:0/1314890335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd74c152690 0x7fd74c155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.246+0000 7fd74b7fe640 1 -- 172.21.15.1:0/1314890335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd74c152690 msgr2=0x7fd74c155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.246+0000 7fd74b7fe640 1 --2- 172.21.15.1:0/1314890335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd74c152690 0x7fd74c155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.246+0000 7fd749ffb640 1 -- 172.21.15.1:0/1314890335 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876024578 0 0) 0x7fd740001110 con 0x7fd74c0c6730 2024-03-06T07:23:11.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.246+0000 7fd74b7fe640 1 -- 172.21.15.1:0/1314890335 >> v1:172.21.15.98:6789/0 conn(0x7fd74c0c6730 legacy=0x7fd74c0c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.246+0000 7fd74b7fe640 1 -- 172.21.15.1:0/1314890335 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c0c3880 con 0x7fd74c156cd0 2024-03-06T07:23:11.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.247+0000 7fd74b7fe640 1 --2- 172.21.15.1:0/1314890335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd74c156cd0 0x7fd74c157160 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fd744009e50 tx=0x7fd74402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c60fe337e1e07031 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.247+0000 7fd749ffb640 1 -- 172.21.15.1:0/1314890335 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74402ca70 con 0x7fd74c156cd0 2024-03-06T07:23:11.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.247+0000 7fd749ffb640 1 -- 172.21.15.1:0/1314890335 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74402cc10 con 0x7fd74c156cd0 2024-03-06T07:23:11.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.247+0000 7fd749ffb640 1 -- 172.21.15.1:0/1314890335 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7440343f0 con 0x7fd74c156cd0 2024-03-06T07:23:11.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.248+0000 7fd75138a640 1 -- 172.21.15.1:0/1314890335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd74c156cd0 msgr2=0x7fd74c157160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.248+0000 7fd75138a640 1 --2- 172.21.15.1:0/1314890335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd74c156cd0 0x7fd74c157160 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fd744009e50 tx=0x7fd74402bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.248+0000 7fd75138a640 1 -- 172.21.15.1:0/1314890335 shutdown_connections 2024-03-06T07:23:11.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.248+0000 7fd75138a640 1 --2- 172.21.15.1:0/1314890335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd74c156cd0 0x7fd74c157160 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.248+0000 7fd75138a640 1 --2- 172.21.15.1:0/1314890335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd74c152690 0x7fd74c155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.248+0000 7fd75138a640 1 -- 172.21.15.1:0/1314890335 >> 172.21.15.1:0/1314890335 conn(0x7fd74c06b410 msgr2=0x7fd74c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.248+0000 7fd75138a640 1 -- 172.21.15.1:0/1314890335 shutdown_connections 2024-03-06T07:23:11.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.248+0000 7fd75138a640 1 -- 172.21.15.1:0/1314890335 wait complete. 2024-03-06T07:23:11.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.250+0000 7fd75138a640 1 Processor -- start 2024-03-06T07:23:11.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.250+0000 7fd75138a640 1 -- start start 2024-03-06T07:23:11.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.251+0000 7fd75138a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd74c0c6730 0x7fd74c146840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.251+0000 7fd75138a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd74c152690 0x7fd74c150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.251+0000 7fd75138a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd74c156cd0 0x7fd74c1ee1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.251+0000 7fd75138a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c0c5ee0 con 0x7fd74c152690 2024-03-06T07:23:11.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.251+0000 7fd75138a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c1567b0 con 0x7fd74c0c6730 2024-03-06T07:23:11.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.251+0000 7fd75138a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c0c6060 con 0x7fd74c156cd0 2024-03-06T07:23:11.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.251+0000 7fd74a7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd74c152690 0x7fd74c150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.251+0000 7fd74a7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd74c152690 0x7fd74c150e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42576/0 (socket says 172.21.15.1:42576) 2024-03-06T07:23:11.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.251+0000 7fd74a7fc640 1 -- 172.21.15.1:0/3019235282 learned_addr learned my addr 172.21.15.1:0/3019235282 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:11.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.251+0000 7fd74affd640 1 --2- 172.21.15.1:0/3019235282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd74c0c6730 0x7fd74c146840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.251+0000 7fd74b7fe640 1 --2- 172.21.15.1:0/3019235282 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd74c156cd0 0x7fd74c1ee1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.251+0000 7fd74a7fc640 1 -- 172.21.15.1:0/3019235282 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd74c156cd0 msgr2=0x7fd74c1ee1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.251+0000 7fd74a7fc640 1 --2- 172.21.15.1:0/3019235282 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd74c156cd0 0x7fd74c1ee1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.251+0000 7fd74a7fc640 1 -- 172.21.15.1:0/3019235282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd74c0c6730 msgr2=0x7fd74c146840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.251+0000 7fd74a7fc640 1 --2- 172.21.15.1:0/3019235282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd74c0c6730 0x7fd74c146840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.251+0000 7fd74a7fc640 1 -- 172.21.15.1:0/3019235282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c0c6ee0 con 0x7fd74c152690 2024-03-06T07:23:11.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.251+0000 7fd74a7fc640 1 --2- 172.21.15.1:0/3019235282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd74c152690 0x7fd74c150e20 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd74000c410 tx=0x7fd74000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.252+0000 7fd7337fe640 1 -- 172.21.15.1:0/3019235282 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd740017070 con 0x7fd74c152690 2024-03-06T07:23:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.252+0000 7fd7337fe640 1 -- 172.21.15.1:0/3019235282 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7400035f0 con 0x7fd74c152690 2024-03-06T07:23:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.252+0000 7fd75138a640 1 -- 172.21.15.1:0/3019235282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74c14a940 con 0x7fd74c152690 2024-03-06T07:23:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.252+0000 7fd7337fe640 1 -- 172.21.15.1:0/3019235282 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7400039e0 con 0x7fd74c152690 2024-03-06T07:23:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.252+0000 7fd75138a640 1 -- 172.21.15.1:0/3019235282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74c0c7b30 con 0x7fd74c152690 2024-03-06T07:23:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.253+0000 7fd75138a640 1 -- 172.21.15.1:0/3019235282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd74c067fd0 con 0x7fd74c152690 2024-03-06T07:23:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.253+0000 7fd7337fe640 1 -- 172.21.15.1:0/3019235282 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd74001a070 con 0x7fd74c152690 2024-03-06T07:23:11.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.254+0000 7fd7337fe640 1 --2- 172.21.15.1:0/3019235282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd73806b4d0 0x7fd73806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.254+0000 7fd7337fe640 1 -- 172.21.15.1:0/3019235282 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fd740089990 con 0x7fd74c152690 2024-03-06T07:23:11.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.254+0000 7fd74affd640 1 --2- 172.21.15.1:0/3019235282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd73806b4d0 0x7fd73806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.254+0000 7fd74affd640 1 --2- 172.21.15.1:0/3019235282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd73806b4d0 0x7fd73806d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd73c009aa0 tx=0x7fd73c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.258+0000 7fd7337fe640 1 -- 172.21.15.1:0/3019235282 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd74c1561e0 con 0x7fd74c152690 2024-03-06T07:23:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.443+0000 7fd75138a640 1 -- 172.21.15.1:0/3019235282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd74c071830 con 0x7fd74c152690 2024-03-06T07:23:11.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.445+0000 7fd7337fe640 1 -- 172.21.15.1:0/3019235282 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+20549 (secure 0 0 0) 0x7fd740057d70 con 0x7fd74c152690 2024-03-06T07:23:11.447 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:11.447 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":78,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:23:08.733625+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:55.240857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:55.939811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:23:05.599046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:23:06.205539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:11.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.447+0000 7fd75138a640 1 -- 172.21.15.1:0/3019235282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd73806b4d0 msgr2=0x7fd73806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.447+0000 7fd75138a640 1 --2- 172.21.15.1:0/3019235282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd73806b4d0 0x7fd73806d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd73c009aa0 tx=0x7fd73c009190 comp rx=0 tx=0).stop 2024-03-06T07:23:11.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.447+0000 7fd75138a640 1 -- 172.21.15.1:0/3019235282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd74c152690 msgr2=0x7fd74c150e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.447+0000 7fd75138a640 1 --2- 172.21.15.1:0/3019235282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd74c152690 0x7fd74c150e20 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd74000c410 tx=0x7fd74000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.447+0000 7fd75138a640 1 -- 172.21.15.1:0/3019235282 shutdown_connections 2024-03-06T07:23:11.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.447+0000 7fd75138a640 1 --2- 172.21.15.1:0/3019235282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd73806b4d0 0x7fd73806d990 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.447+0000 7fd75138a640 1 --2- 172.21.15.1:0/3019235282 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd74c156cd0 0x7fd74c1ee1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.447+0000 7fd75138a640 1 --2- 172.21.15.1:0/3019235282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd74c152690 0x7fd74c150e20 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.447+0000 7fd75138a640 1 --2- 172.21.15.1:0/3019235282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd74c0c6730 0x7fd74c146840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.447+0000 7fd75138a640 1 -- 172.21.15.1:0/3019235282 >> 172.21.15.1:0/3019235282 conn(0x7fd74c06b410 msgr2=0x7fd74c154af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.447+0000 7fd75138a640 1 -- 172.21.15.1:0/3019235282 shutdown_connections 2024-03-06T07:23:11.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.447+0000 7fd75138a640 1 -- 172.21.15.1:0/3019235282 wait complete. 2024-03-06T07:23:11.460 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:23:11.461 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:11.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.561+0000 7f8948eeb640 1 Processor -- start 2024-03-06T07:23:11.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.561+0000 7f8948eeb640 1 -- start start 2024-03-06T07:23:11.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.561+0000 7f8948eeb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f89440c4560 0x7f8944155c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.561+0000 7f8948eeb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89440c39c0 0x7f89441561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.561+0000 7f8948eeb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89440c7420 con 0x7f89440c4940 2024-03-06T07:23:11.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.561+0000 7f8948eeb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89441566f0 con 0x7f89440c4560 2024-03-06T07:23:11.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.561+0000 7f8948eeb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89440ab760 con 0x7f89440c39c0 2024-03-06T07:23:11.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.561+0000 7f894259c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f89440c4560 0x7f8944155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.561+0000 7f894259c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f89440c4560 0x7f8944155c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42578/0 (socket says 172.21.15.1:42578) 2024-03-06T07:23:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.561+0000 7f894259c640 1 -- 172.21.15.1:0/3233361894 learned_addr learned my addr 172.21.15.1:0/3233361894 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.562+0000 7f8939d9b640 1 --2- 172.21.15.1:0/3233361894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89440c39c0 0x7f89441561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.562+0000 7f894259c640 1 -- 172.21.15.1:0/3233361894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89440c39c0 msgr2=0x7f89441561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.562+0000 7f894259c640 1 --2- 172.21.15.1:0/3233361894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89440c39c0 0x7f89441561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.562+0000 7f894259c640 1 -- 172.21.15.1:0/3233361894 >> v1:172.21.15.98:6789/0 conn(0x7f89440c4940 legacy=0x7f89440ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.562+0000 7f894259c640 1 -- 172.21.15.1:0/3233361894 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89440c76c0 con 0x7f89440c4560 2024-03-06T07:23:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.562+0000 7f8941d9b640 1 -- 172.21.15.1:0/3233361894 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787275854 0 0) 0x7f89440c7420 con 0x7f89440c4940 2024-03-06T07:23:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.562+0000 7f894259c640 1 --2- 172.21.15.1:0/3233361894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89440c4560 0x7f8944155c70 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f8934009870 tx=0x7f893402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6397b00868eea8b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.563+0000 7f8941d9b640 1 -- 172.21.15.1:0/3233361894 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934034070 con 0x7f89440c4560 2024-03-06T07:23:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.563+0000 7f8941d9b640 1 -- 172.21.15.1:0/3233361894 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8934030840 con 0x7f89440c4560 2024-03-06T07:23:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.563+0000 7f8941d9b640 1 -- 172.21.15.1:0/3233361894 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934030c10 con 0x7f89440c4560 2024-03-06T07:23:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.563+0000 7f8948eeb640 1 -- 172.21.15.1:0/3233361894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89440c4560 msgr2=0x7f8944155c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.563+0000 7f8948eeb640 1 --2- 172.21.15.1:0/3233361894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89440c4560 0x7f8944155c70 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f8934009870 tx=0x7f893402b980 comp rx=0 tx=0).stop 2024-03-06T07:23:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.563+0000 7f8948eeb640 1 -- 172.21.15.1:0/3233361894 shutdown_connections 2024-03-06T07:23:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.563+0000 7f8948eeb640 1 --2- 172.21.15.1:0/3233361894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89440c39c0 0x7f89441561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.563+0000 7f8948eeb640 1 --2- 172.21.15.1:0/3233361894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89440c4560 0x7f8944155c70 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.563+0000 7f8948eeb640 1 -- 172.21.15.1:0/3233361894 >> 172.21.15.1:0/3233361894 conn(0x7f89440c7a50 msgr2=0x7f89440c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.563+0000 7f8948eeb640 1 -- 172.21.15.1:0/3233361894 shutdown_connections 2024-03-06T07:23:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.563+0000 7f8948eeb640 1 -- 172.21.15.1:0/3233361894 wait complete. 2024-03-06T07:23:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.565+0000 7f8948eeb640 1 Processor -- start 2024-03-06T07:23:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.565+0000 7f8948eeb640 1 -- start start 2024-03-06T07:23:11.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.566+0000 7f8948eeb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89440c39c0 0x7f89440c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.566+0000 7f8948eeb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89440c4560 0x7f894406aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.566+0000 7f8948eeb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89440c4940 0x7f89440a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.566+0000 7f8948eeb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894406e6d0 con 0x7f89440c39c0 2024-03-06T07:23:11.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.566+0000 7f8948eeb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89440abd00 con 0x7f89440c4940 2024-03-06T07:23:11.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.566+0000 7f8948eeb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f894406e850 con 0x7f89440c4560 2024-03-06T07:23:11.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.566+0000 7f894259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89440c39c0 0x7f89440c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.567+0000 7f894259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89440c39c0 0x7f89440c1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42586/0 (socket says 172.21.15.1:42586) 2024-03-06T07:23:11.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.567+0000 7f894259c640 1 -- 172.21.15.1:0/1335195830 learned_addr learned my addr 172.21.15.1:0/1335195830 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:11.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.567+0000 7f8939d9b640 1 --2- 172.21.15.1:0/1335195830 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89440c4560 0x7f894406aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.567+0000 7f8942d9d640 1 --2- 172.21.15.1:0/1335195830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89440c4940 0x7f89440a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.567+0000 7f894259c640 1 -- 172.21.15.1:0/1335195830 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89440c4560 msgr2=0x7f894406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.567+0000 7f894259c640 1 --2- 172.21.15.1:0/1335195830 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89440c4560 0x7f894406aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.567+0000 7f894259c640 1 -- 172.21.15.1:0/1335195830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89440c4940 msgr2=0x7f89440a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.567+0000 7f894259c640 1 --2- 172.21.15.1:0/1335195830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89440c4940 0x7f89440a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.567+0000 7f894259c640 1 -- 172.21.15.1:0/1335195830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89440c0280 con 0x7f89440c39c0 2024-03-06T07:23:11.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.567+0000 7f894259c640 1 --2- 172.21.15.1:0/1335195830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89440c39c0 0x7f89440c1d50 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f89340090b0 tx=0x7f89340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.567+0000 7f893b7fe640 1 -- 172.21.15.1:0/1335195830 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934034040 con 0x7f89440c39c0 2024-03-06T07:23:11.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.567+0000 7f893b7fe640 1 -- 172.21.15.1:0/1335195830 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8934030640 con 0x7f89440c39c0 2024-03-06T07:23:11.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.567+0000 7f893b7fe640 1 -- 172.21.15.1:0/1335195830 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934039d40 con 0x7f89440c39c0 2024-03-06T07:23:11.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.567+0000 7f8948eeb640 1 -- 172.21.15.1:0/1335195830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89440bf700 con 0x7f89440c39c0 2024-03-06T07:23:11.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.567+0000 7f8948eeb640 1 -- 172.21.15.1:0/1335195830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89440bfcc0 con 0x7f89440c39c0 2024-03-06T07:23:11.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.568+0000 7f8948eeb640 1 -- 172.21.15.1:0/1335195830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89440686e0 con 0x7f89440c39c0 2024-03-06T07:23:11.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.569+0000 7f893b7fe640 1 -- 172.21.15.1:0/1335195830 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8934037070 con 0x7f89440c39c0 2024-03-06T07:23:11.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.569+0000 7f893b7fe640 1 --2- 172.21.15.1:0/1335195830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f892406b4d0 0x7f892406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.569+0000 7f893b7fe640 1 -- 172.21.15.1:0/1335195830 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f89340ac2b0 con 0x7f89440c39c0 2024-03-06T07:23:11.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.569+0000 7f8939d9b640 1 --2- 172.21.15.1:0/1335195830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f892406b4d0 0x7f892406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.570+0000 7f8939d9b640 1 --2- 172.21.15.1:0/1335195830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f892406b4d0 0x7f892406d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f89440a8b20 tx=0x7f892c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.573+0000 7f893b7fe640 1 -- 172.21.15.1:0/1335195830 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f893407a7b0 con 0x7f89440c39c0 2024-03-06T07:23:11.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.751+0000 7f8948eeb640 1 -- 172.21.15.1:0/1335195830 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f89440a7eb0 con 0x7f892406b4d0 2024-03-06T07:23:11.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.763+0000 7f893b7fe640 1 -- 172.21.15.1:0/1335195830 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490719 (secure 0 0 0) 0x7f89440a7eb0 con 0x7f892406b4d0 2024-03-06T07:23:11.771 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:11.775 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:23:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.768+0000 7f8948eeb640 1 -- 172.21.15.1:0/1335195830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f892406b4d0 msgr2=0x7f892406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.768+0000 7f8948eeb640 1 --2- 172.21.15.1:0/1335195830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f892406b4d0 0x7f892406d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f89440a8b20 tx=0x7f892c009190 comp rx=0 tx=0).stop 2024-03-06T07:23:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.768+0000 7f8948eeb640 1 -- 172.21.15.1:0/1335195830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89440c39c0 msgr2=0x7f89440c1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.768+0000 7f8948eeb640 1 --2- 172.21.15.1:0/1335195830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89440c39c0 0x7f89440c1d50 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f89340090b0 tx=0x7f89340046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.768+0000 7f8948eeb640 1 -- 172.21.15.1:0/1335195830 shutdown_connections 2024-03-06T07:23:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.768+0000 7f8948eeb640 1 --2- 172.21.15.1:0/1335195830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f892406b4d0 0x7f892406d990 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.768+0000 7f8948eeb640 1 --2- 172.21.15.1:0/1335195830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89440c4940 0x7f89440a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.768+0000 7f8948eeb640 1 --2- 172.21.15.1:0/1335195830 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89440c4560 0x7f894406aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.768+0000 7f8948eeb640 1 --2- 172.21.15.1:0/1335195830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89440c39c0 0x7f89440c1d50 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.768+0000 7f8948eeb640 1 -- 172.21.15.1:0/1335195830 >> 172.21.15.1:0/1335195830 conn(0x7f89440c7a50 msgr2=0x7f8944155170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.768+0000 7f8948eeb640 1 -- 172.21.15.1:0/1335195830 shutdown_connections 2024-03-06T07:23:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.768+0000 7f8948eeb640 1 -- 172.21.15.1:0/1335195830 wait complete. 2024-03-06T07:23:11.782 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-03-06T07:23:10.729719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":345,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244684,"kb_used_data":7700,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754730036,"statfs":{"total":773094113280,"available":772843556864,"internally_reserved":0,"allocated":7884800,"data_stored":4848171,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.020920"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.128111+0000","last_change":"2024-03-06T07:23:07.749704+0000","last_active":"2024-03-06T07:23:09.128111+0000","last_peered":"2024-03-06T07:23:09.128111+0000","last_clean":"2024-03-06T07:23:09.128111+0000","last_became_active":"2024-03-06T07:23:07.749457+0000","last_became_peered":"2024-03-06T07:23:07.749457+0000","last_unstale":"2024-03-06T07:23:09.128111+0000","last_undegraded":"2024-03-06T07:23:09.128111+0000","last_fullsized":"2024-03-06T07:23:09.128111+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:45.512631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743244+0000","last_change":"2024-03-06T07:22:57.627301+0000","last_active":"2024-03-06T07:23:08.743244+0000","last_peered":"2024-03-06T07:23:08.743244+0000","last_clean":"2024-03-06T07:23:08.743244+0000","last_became_active":"2024-03-06T07:22:57.626618+0000","last_became_peered":"2024-03-06T07:22:57.626618+0000","last_unstale":"2024-03-06T07:23:08.743244+0000","last_undegraded":"2024-03-06T07:23:08.743244+0000","last_fullsized":"2024-03-06T07:23:08.743244+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:10.595979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126919+0000","last_change":"2024-03-06T07:23:07.743004+0000","last_active":"2024-03-06T07:23:09.126919+0000","last_peered":"2024-03-06T07:23:09.126919+0000","last_clean":"2024-03-06T07:23:09.126919+0000","last_became_active":"2024-03-06T07:23:07.742748+0000","last_became_peered":"2024-03-06T07:23:07.742748+0000","last_unstale":"2024-03-06T07:23:09.126919+0000","last_undegraded":"2024-03-06T07:23:09.126919+0000","last_fullsized":"2024-03-06T07:23:09.126919+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:41.051713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126115+0000","last_change":"2024-03-06T07:22:57.636108+0000","last_active":"2024-03-06T07:23:09.126115+0000","last_peered":"2024-03-06T07:23:09.126115+0000","last_clean":"2024-03-06T07:23:09.126115+0000","last_became_active":"2024-03-06T07:22:57.635820+0000","last_became_peered":"2024-03-06T07:22:57.635820+0000","last_unstale":"2024-03-06T07:23:09.126115+0000","last_undegraded":"2024-03-06T07:23:09.126115+0000","last_fullsized":"2024-03-06T07:23:09.126115+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:42.232078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.128180+0000","last_change":"2024-03-06T07:23:07.738159+0000","last_active":"2024-03-06T07:23:09.128180+0000","last_peered":"2024-03-06T07:23:09.128180+0000","last_clean":"2024-03-06T07:23:09.128180+0000","last_became_active":"2024-03-06T07:23:07.737901+0000","last_became_peered":"2024-03-06T07:23:07.737901+0000","last_unstale":"2024-03-06T07:23:09.128180+0000","last_undegraded":"2024-03-06T07:23:09.128180+0000","last_fullsized":"2024-03-06T07:23:09.128180+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:03.919513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126505+0000","last_change":"2024-03-06T07:22:57.627190+0000","last_active":"2024-03-06T07:23:09.126505+0000","last_peered":"2024-03-06T07:23:09.126505+0000","last_clean":"2024-03-06T07:23:09.126505+0000","last_became_active":"2024-03-06T07:22:57.626893+0000","last_became_peered":"2024-03-06T07:22:57.626893+0000","last_unstale":"2024-03-06T07:23:09.126505+0000","last_undegraded":"2024-03-06T07:23:09.126505+0000","last_fullsized":"2024-03-06T07:23:09.126505+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:04.450678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742506+0000","last_change":"2024-03-06T07:23:07.738946+0000","last_active":"2024-03-06T07:23:08.742506+0000","last_peered":"2024-03-06T07:23:08.742506+0000","last_clean":"2024-03-06T07:23:08.742506+0000","last_became_active":"2024-03-06T07:23:07.738650+0000","last_became_peered":"2024-03-06T07:23:07.738650+0000","last_unstale":"2024-03-06T07:23:08.742506+0000","last_undegraded":"2024-03-06T07:23:08.742506+0000","last_fullsized":"2024-03-06T07:23:08.742506+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:13.483301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741825+0000","last_change":"2024-03-06T07:22:57.632863+0000","last_active":"2024-03-06T07:23:08.741825+0000","last_peered":"2024-03-06T07:23:08.741825+0000","last_clean":"2024-03-06T07:23:08.741825+0000","last_became_active":"2024-03-06T07:22:57.632435+0000","last_became_peered":"2024-03-06T07:22:57.632435+0000","last_unstale":"2024-03-06T07:23:08.741825+0000","last_undegraded":"2024-03-06T07:23:08.741825+0000","last_fullsized":"2024-03-06T07:23:08.741825+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:46.427684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741458+0000","last_change":"2024-03-06T07:23:07.752531+0000","last_active":"2024-03-06T07:23:08.741458+0000","last_peered":"2024-03-06T07:23:08.741458+0000","last_clean":"2024-03-06T07:23:08.741458+0000","last_became_active":"2024-03-06T07:23:07.752246+0000","last_became_peered":"2024-03-06T07:23:07.752246+0000","last_unstale":"2024-03-06T07:23:08.741458+0000","last_undegraded":"2024-03-06T07:23:08.741458+0000","last_fullsized":"2024-03-06T07:23:08.741458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:39.208886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740208+0000","last_change":"2024-03-06T07:22:57.622627+0000","last_active":"2024-03-06T07:23:08.740208+0000","last_peered":"2024-03-06T07:23:08.740208+0000","last_clean":"2024-03-06T07:23:08.740208+0000","last_became_active":"2024-03-06T07:22:57.622319+0000","last_became_peered":"2024-03-06T07:22:57.622319+0000","last_unstale":"2024-03-06T07:23:08.740208+0000","last_undegraded":"2024-03-06T07:23:08.740208+0000","last_fullsized":"2024-03-06T07:23:08.740208+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:01.132943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126931+0000","last_change":"2024-03-06T07:23:07.754142+0000","last_active":"2024-03-06T07:23:09.126931+0000","last_peered":"2024-03-06T07:23:09.126931+0000","last_clean":"2024-03-06T07:23:09.126931+0000","last_became_active":"2024-03-06T07:23:07.753832+0000","last_became_peered":"2024-03-06T07:23:07.753832+0000","last_unstale":"2024-03-06T07:23:09.126931+0000","last_undegraded":"2024-03-06T07:23:09.126931+0000","last_fullsized":"2024-03-06T07:23:09.126931+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:49.818057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127683+0000","last_change":"2024-03-06T07:22:57.633886+0000","last_active":"2024-03-06T07:23:09.127683+0000","last_peered":"2024-03-06T07:23:09.127683+0000","last_clean":"2024-03-06T07:23:09.127683+0000","last_became_active":"2024-03-06T07:22:57.633567+0000","last_became_peered":"2024-03-06T07:22:57.633567+0000","last_unstale":"2024-03-06T07:23:09.127683+0000","last_undegraded":"2024-03-06T07:23:09.127683+0000","last_fullsized":"2024-03-06T07:23:09.127683+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:39.710112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745825+0000","last_change":"2024-03-06T07:23:07.749384+0000","last_active":"2024-03-06T07:23:08.745825+0000","last_peered":"2024-03-06T07:23:08.745825+0000","last_clean":"2024-03-06T07:23:08.745825+0000","last_became_active":"2024-03-06T07:23:07.749122+0000","last_became_peered":"2024-03-06T07:23:07.749122+0000","last_unstale":"2024-03-06T07:23:08.745825+0000","last_undegraded":"2024-03-06T07:23:08.745825+0000","last_fullsized":"2024-03-06T07:23:08.745825+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:33.406528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125674+0000","last_change":"2024-03-06T07:22:57.637662+0000","last_active":"2024-03-06T07:23:09.125674+0000","last_peered":"2024-03-06T07:23:09.125674+0000","last_clean":"2024-03-06T07:23:09.125674+0000","last_became_active":"2024-03-06T07:22:57.637320+0000","last_became_peered":"2024-03-06T07:22:57.637320+0000","last_unstale":"2024-03-06T07:23:09.125674+0000","last_undegraded":"2024-03-06T07:23:09.125674+0000","last_fullsized":"2024-03-06T07:23:09.125674+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:48.772981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126613+0000","last_change":"2024-03-06T07:23:07.746767+0000","last_active":"2024-03-06T07:23:09.126613+0000","last_peered":"2024-03-06T07:23:09.126613+0000","last_clean":"2024-03-06T07:23:09.126613+0000","last_became_active":"2024-03-06T07:23:07.746252+0000","last_became_peered":"2024-03-06T07:23:07.746252+0000","last_unstale":"2024-03-06T07:23:09.126613+0000","last_undegraded":"2024-03-06T07:23:09.126613+0000","last_fullsized":"2024-03-06T07:23:09.126613+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:40.831625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739605+0000","last_change":"2024-03-06T07:22:57.623809+0000","last_active":"2024-03-06T07:23:08.739605+0000","last_peered":"2024-03-06T07:23:08.739605+0000","last_clean":"2024-03-06T07:23:08.739605+0000","last_became_active":"2024-03-06T07:22:57.623535+0000","last_became_peered":"2024-03-06T07:22:57.623535+0000","last_unstale":"2024-03-06T07:23:08.739605+0000","last_undegraded":"2024-03-06T07:23:08.739605+0000","last_fullsized":"2024-03-06T07:23:08.739605+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:08.011885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746672+0000","last_change":"2024-03-06T07:23:07.749717+0000","last_active":"2024-03-06T07:23:08.746672+0000","last_peered":"2024-03-06T07:23:08.746672+0000","last_clean":"2024-03-06T07:23:08.746672+0000","last_became_active":"2024-03-06T07:23:07.745609+0000","last_became_peered":"2024-03-06T07:23:07.745609+0000","last_unstale":"2024-03-06T07:23:08.746672+0000","last_undegraded":"2024-03-06T07:23:08.746672+0000","last_fullsized":"2024-03-06T07:23:08.746672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:52.340022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746604+0000","last_change":"2024-03-06T07:22:57.635037+0000","last_active":"2024-03-06T07:23:08.746604+0000","last_peered":"2024-03-06T07:23:08.746604+0000","last_clean":"2024-03-06T07:23:08.746604+0000","last_became_active":"2024-03-06T07:22:57.634754+0000","last_became_peered":"2024-03-06T07:22:57.634754+0000","last_unstale":"2024-03-06T07:23:08.746604+0000","last_undegraded":"2024-03-06T07:23:08.746604+0000","last_fullsized":"2024-03-06T07:23:08.746604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:55.687038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739829+0000","last_change":"2024-03-06T07:23:07.736724+0000","last_active":"2024-03-06T07:23:08.739829+0000","last_peered":"2024-03-06T07:23:08.739829+0000","last_clean":"2024-03-06T07:23:08.739829+0000","last_became_active":"2024-03-06T07:23:07.736451+0000","last_became_peered":"2024-03-06T07:23:07.736451+0000","last_unstale":"2024-03-06T07:23:08.739829+0000","last_undegraded":"2024-03-06T07:23:08.739829+0000","last_fullsized":"2024-03-06T07:23:08.739829+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:48.354539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742621+0000","last_change":"2024-03-06T07:22:57.632537+0000","last_active":"2024-03-06T07:23:08.742621+0000","last_peered":"2024-03-06T07:23:08.742621+0000","last_clean":"2024-03-06T07:23:08.742621+0000","last_became_active":"2024-03-06T07:22:57.632230+0000","last_became_peered":"2024-03-06T07:22:57.632230+0000","last_unstale":"2024-03-06T07:23:08.742621+0000","last_undegraded":"2024-03-06T07:23:08.742621+0000","last_fullsized":"2024-03-06T07:23:08.742621+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:43.442566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746502+0000","last_change":"2024-03-06T07:23:07.748940+0000","last_active":"2024-03-06T07:23:08.746502+0000","last_peered":"2024-03-06T07:23:08.746502+0000","last_clean":"2024-03-06T07:23:08.746502+0000","last_became_active":"2024-03-06T07:23:07.745255+0000","last_became_peered":"2024-03-06T07:23:07.745255+0000","last_unstale":"2024-03-06T07:23:08.746502+0000","last_undegraded":"2024-03-06T07:23:08.746502+0000","last_fullsized":"2024-03-06T07:23:08.746502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:56.142001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126946+0000","last_change":"2024-03-06T07:22:57.633314+0000","last_active":"2024-03-06T07:23:09.126946+0000","last_peered":"2024-03-06T07:23:09.126946+0000","last_clean":"2024-03-06T07:23:09.126946+0000","last_became_active":"2024-03-06T07:22:57.633041+0000","last_became_peered":"2024-03-06T07:22:57.633041+0000","last_unstale":"2024-03-06T07:23:09.126946+0000","last_undegraded":"2024-03-06T07:23:09.126946+0000","last_fullsized":"2024-03-06T07:23:09.126946+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:46.034056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740004+0000","last_change":"2024-03-06T07:23:07.746640+0000","last_active":"2024-03-06T07:23:08.740004+0000","last_peered":"2024-03-06T07:23:08.740004+0000","last_clean":"2024-03-06T07:23:08.740004+0000","last_became_active":"2024-03-06T07:23:07.746380+0000","last_became_peered":"2024-03-06T07:23:07.746380+0000","last_unstale":"2024-03-06T07:23:08.740004+0000","last_undegraded":"2024-03-06T07:23:08.740004+0000","last_fullsized":"2024-03-06T07:23:08.740004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:14.364329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741911+0000","last_change":"2024-03-06T07:22:57.632052+0000","last_active":"2024-03-06T07:23:08.741911+0000","last_peered":"2024-03-06T07:23:08.741911+0000","last_clean":"2024-03-06T07:23:08.741911+0000","last_became_active":"2024-03-06T07:22:57.630733+0000","last_became_peered":"2024-03-06T07:22:57.630733+0000","last_unstale":"2024-03-06T07:23:08.741911+0000","last_undegraded":"2024-03-06T07:23:08.741911+0000","last_fullsized":"2024-03-06T07:23:08.741911+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:38.629913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126140+0000","last_change":"2024-03-06T07:23:07.739707+0000","last_active":"2024-03-06T07:23:09.126140+0000","last_peered":"2024-03-06T07:23:09.126140+0000","last_clean":"2024-03-06T07:23:09.126140+0000","last_became_active":"2024-03-06T07:23:07.739440+0000","last_became_peered":"2024-03-06T07:23:07.739440+0000","last_unstale":"2024-03-06T07:23:09.126140+0000","last_undegraded":"2024-03-06T07:23:09.126140+0000","last_fullsized":"2024-03-06T07:23:09.126140+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:10.248189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740297+0000","last_change":"2024-03-06T07:22:57.644158+0000","last_active":"2024-03-06T07:23:08.740297+0000","last_peered":"2024-03-06T07:23:08.740297+0000","last_clean":"2024-03-06T07:23:08.740297+0000","last_became_active":"2024-03-06T07:22:57.643252+0000","last_became_peered":"2024-03-06T07:22:57.643252+0000","last_unstale":"2024-03-06T07:23:08.740297+0000","last_undegraded":"2024-03-06T07:23:08.740297+0000","last_fullsized":"2024-03-06T07:23:08.740297+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:38.286642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125769+0000","last_change":"2024-03-06T07:23:07.740246+0000","last_active":"2024-03-06T07:23:09.125769+0000","last_peered":"2024-03-06T07:23:09.125769+0000","last_clean":"2024-03-06T07:23:09.125769+0000","last_became_active":"2024-03-06T07:23:07.739848+0000","last_became_peered":"2024-03-06T07:23:07.739848+0000","last_unstale":"2024-03-06T07:23:09.125769+0000","last_undegraded":"2024-03-06T07:23:09.125769+0000","last_fullsized":"2024-03-06T07:23:09.125769+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:12.770620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127715+0000","last_change":"2024-03-06T07:22:57.633872+0000","last_active":"2024-03-06T07:23:09.127715+0000","last_peered":"2024-03-06T07:23:09.127715+0000","last_clean":"2024-03-06T07:23:09.127715+0000","last_became_active":"2024-03-06T07:22:57.633442+0000","last_became_peered":"2024-03-06T07:22:57.633442+0000","last_unstale":"2024-03-06T07:23:09.127715+0000","last_undegraded":"2024-03-06T07:23:09.127715+0000","last_fullsized":"2024-03-06T07:23:09.127715+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:34.380891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126897+0000","last_change":"2024-03-06T07:23:07.754105+0000","last_active":"2024-03-06T07:23:09.126897+0000","last_peered":"2024-03-06T07:23:09.126897+0000","last_clean":"2024-03-06T07:23:09.126897+0000","last_became_active":"2024-03-06T07:23:07.753800+0000","last_became_peered":"2024-03-06T07:23:07.753800+0000","last_unstale":"2024-03-06T07:23:09.126897+0000","last_undegraded":"2024-03-06T07:23:09.126897+0000","last_fullsized":"2024-03-06T07:23:09.126897+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:46.656259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126155+0000","last_change":"2024-03-06T07:22:57.624521+0000","last_active":"2024-03-06T07:23:09.126155+0000","last_peered":"2024-03-06T07:23:09.126155+0000","last_clean":"2024-03-06T07:23:09.126155+0000","last_became_active":"2024-03-06T07:22:57.624228+0000","last_became_peered":"2024-03-06T07:22:57.624228+0000","last_unstale":"2024-03-06T07:23:09.126155+0000","last_undegraded":"2024-03-06T07:23:09.126155+0000","last_fullsized":"2024-03-06T07:23:09.126155+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:26.279644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745658+0000","last_change":"2024-03-06T07:23:07.750667+0000","last_active":"2024-03-06T07:23:08.745658+0000","last_peered":"2024-03-06T07:23:08.745658+0000","last_clean":"2024-03-06T07:23:08.745658+0000","last_became_active":"2024-03-06T07:23:07.750394+0000","last_became_peered":"2024-03-06T07:23:07.750394+0000","last_unstale":"2024-03-06T07:23:08.745658+0000","last_undegraded":"2024-03-06T07:23:08.745658+0000","last_fullsized":"2024-03-06T07:23:08.745658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:02.932081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741485+0000","last_change":"2024-03-06T07:22:57.634006+0000","last_active":"2024-03-06T07:23:08.741485+0000","last_peered":"2024-03-06T07:23:08.741485+0000","last_clean":"2024-03-06T07:23:08.741485+0000","last_became_active":"2024-03-06T07:22:57.633669+0000","last_became_peered":"2024-03-06T07:22:57.633669+0000","last_unstale":"2024-03-06T07:23:08.741485+0000","last_undegraded":"2024-03-06T07:23:08.741485+0000","last_fullsized":"2024-03-06T07:23:08.741485+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:48.906789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746785+0000","last_change":"2024-03-06T07:23:07.751696+0000","last_active":"2024-03-06T07:23:08.746785+0000","last_peered":"2024-03-06T07:23:08.746785+0000","last_clean":"2024-03-06T07:23:08.746785+0000","last_became_active":"2024-03-06T07:23:07.751126+0000","last_became_peered":"2024-03-06T07:23:07.751126+0000","last_unstale":"2024-03-06T07:23:08.746785+0000","last_undegraded":"2024-03-06T07:23:08.746785+0000","last_fullsized":"2024-03-06T07:23:08.746785+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:43.274210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740453+0000","last_change":"2024-03-06T07:22:57.635578+0000","last_active":"2024-03-06T07:23:08.740453+0000","last_peered":"2024-03-06T07:23:08.740453+0000","last_clean":"2024-03-06T07:23:08.740453+0000","last_became_active":"2024-03-06T07:22:57.635275+0000","last_became_peered":"2024-03-06T07:22:57.635275+0000","last_unstale":"2024-03-06T07:23:08.740453+0000","last_undegraded":"2024-03-06T07:23:08.740453+0000","last_fullsized":"2024-03-06T07:23:08.740453+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:25.589698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127083+0000","last_change":"2024-03-06T07:23:07.740364+0000","last_active":"2024-03-06T07:23:09.127083+0000","last_peered":"2024-03-06T07:23:09.127083+0000","last_clean":"2024-03-06T07:23:09.127083+0000","last_became_active":"2024-03-06T07:23:07.740105+0000","last_became_peered":"2024-03-06T07:23:07.740105+0000","last_unstale":"2024-03-06T07:23:09.127083+0000","last_undegraded":"2024-03-06T07:23:09.127083+0000","last_fullsized":"2024-03-06T07:23:09.127083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:56.142001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126758+0000","last_change":"2024-03-06T07:22:57.628862+0000","last_active":"2024-03-06T07:23:09.126758+0000","last_peered":"2024-03-06T07:23:09.126758+0000","last_clean":"2024-03-06T07:23:09.126758+0000","last_became_active":"2024-03-06T07:22:57.628584+0000","last_became_peered":"2024-03-06T07:22:57.628584+0000","last_unstale":"2024-03-06T07:23:09.126758+0000","last_undegraded":"2024-03-06T07:23:09.126758+0000","last_fullsized":"2024-03-06T07:23:09.126758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:54.432088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126183+0000","last_change":"2024-03-06T07:23:07.738520+0000","last_active":"2024-03-06T07:23:09.126183+0000","last_peered":"2024-03-06T07:23:09.126183+0000","last_clean":"2024-03-06T07:23:09.126183+0000","last_became_active":"2024-03-06T07:23:07.738246+0000","last_became_peered":"2024-03-06T07:23:07.738246+0000","last_unstale":"2024-03-06T07:23:09.126183+0000","last_undegraded":"2024-03-06T07:23:09.126183+0000","last_fullsized":"2024-03-06T07:23:09.126183+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:00.643965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.128159+0000","last_change":"2024-03-06T07:22:57.643101+0000","last_active":"2024-03-06T07:23:09.128159+0000","last_peered":"2024-03-06T07:23:09.128159+0000","last_clean":"2024-03-06T07:23:09.128159+0000","last_became_active":"2024-03-06T07:22:57.642820+0000","last_became_peered":"2024-03-06T07:22:57.642820+0000","last_unstale":"2024-03-06T07:23:09.128159+0000","last_undegraded":"2024-03-06T07:23:09.128159+0000","last_fullsized":"2024-03-06T07:23:09.128159+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:36.548315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.128192+0000","last_change":"2024-03-06T07:23:07.743885+0000","last_active":"2024-03-06T07:23:09.128192+0000","last_peered":"2024-03-06T07:23:09.128192+0000","last_clean":"2024-03-06T07:23:09.128192+0000","last_became_active":"2024-03-06T07:23:07.743576+0000","last_became_peered":"2024-03-06T07:23:07.743576+0000","last_unstale":"2024-03-06T07:23:09.128192+0000","last_undegraded":"2024-03-06T07:23:09.128192+0000","last_fullsized":"2024-03-06T07:23:09.128192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:05.205498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739779+0000","last_change":"2024-03-06T07:22:57.636081+0000","last_active":"2024-03-06T07:23:08.739779+0000","last_peered":"2024-03-06T07:23:08.739779+0000","last_clean":"2024-03-06T07:23:08.739779+0000","last_became_active":"2024-03-06T07:22:57.635798+0000","last_became_peered":"2024-03-06T07:22:57.635798+0000","last_unstale":"2024-03-06T07:23:08.739779+0000","last_undegraded":"2024-03-06T07:23:08.739779+0000","last_fullsized":"2024-03-06T07:23:08.739779+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:30.203134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125773+0000","last_change":"2024-03-06T07:23:07.745095+0000","last_active":"2024-03-06T07:23:09.125773+0000","last_peered":"2024-03-06T07:23:09.125773+0000","last_clean":"2024-03-06T07:23:09.125773+0000","last_became_active":"2024-03-06T07:23:07.744832+0000","last_became_peered":"2024-03-06T07:23:07.744832+0000","last_unstale":"2024-03-06T07:23:09.125773+0000","last_undegraded":"2024-03-06T07:23:09.125773+0000","last_fullsized":"2024-03-06T07:23:09.125773+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:09.290438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746287+0000","last_change":"2024-03-06T07:22:57.626588+0000","last_active":"2024-03-06T07:23:08.746287+0000","last_peered":"2024-03-06T07:23:08.746287+0000","last_clean":"2024-03-06T07:23:08.746287+0000","last_became_active":"2024-03-06T07:22:57.625992+0000","last_became_peered":"2024-03-06T07:22:57.625992+0000","last_unstale":"2024-03-06T07:23:08.746287+0000","last_undegraded":"2024-03-06T07:23:08.746287+0000","last_fullsized":"2024-03-06T07:23:08.746287+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:30.081287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127066+0000","last_change":"2024-03-06T07:23:07.749479+0000","last_active":"2024-03-06T07:23:09.127066+0000","last_peered":"2024-03-06T07:23:09.127066+0000","last_clean":"2024-03-06T07:23:09.127066+0000","last_became_active":"2024-03-06T07:23:07.749047+0000","last_became_peered":"2024-03-06T07:23:07.749047+0000","last_unstale":"2024-03-06T07:23:09.127066+0000","last_undegraded":"2024-03-06T07:23:09.127066+0000","last_fullsized":"2024-03-06T07:23:09.127066+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:01.336551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740234+0000","last_change":"2024-03-06T07:22:57.636470+0000","last_active":"2024-03-06T07:23:08.740234+0000","last_peered":"2024-03-06T07:23:08.740234+0000","last_clean":"2024-03-06T07:23:08.740234+0000","last_became_active":"2024-03-06T07:22:57.636207+0000","last_became_peered":"2024-03-06T07:22:57.636207+0000","last_unstale":"2024-03-06T07:23:08.740234+0000","last_undegraded":"2024-03-06T07:23:08.740234+0000","last_fullsized":"2024-03-06T07:23:08.740234+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:10.025633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746334+0000","last_change":"2024-03-06T07:23:07.753036+0000","last_active":"2024-03-06T07:23:08.746334+0000","last_peered":"2024-03-06T07:23:08.746334+0000","last_clean":"2024-03-06T07:23:08.746334+0000","last_became_active":"2024-03-06T07:23:07.745694+0000","last_became_peered":"2024-03-06T07:23:07.745694+0000","last_unstale":"2024-03-06T07:23:08.746334+0000","last_undegraded":"2024-03-06T07:23:08.746334+0000","last_fullsized":"2024-03-06T07:23:08.746334+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:44.488836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745662+0000","last_change":"2024-03-06T07:22:57.632469+0000","last_active":"2024-03-06T07:23:08.745662+0000","last_peered":"2024-03-06T07:23:08.745662+0000","last_clean":"2024-03-06T07:23:08.745662+0000","last_became_active":"2024-03-06T07:22:57.631771+0000","last_became_peered":"2024-03-06T07:22:57.631771+0000","last_unstale":"2024-03-06T07:23:08.745662+0000","last_undegraded":"2024-03-06T07:23:08.745662+0000","last_fullsized":"2024-03-06T07:23:08.745662+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:31.714843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745851+0000","last_change":"2024-03-06T07:23:07.742518+0000","last_active":"2024-03-06T07:23:08.745851+0000","last_peered":"2024-03-06T07:23:08.745851+0000","last_clean":"2024-03-06T07:23:08.745851+0000","last_became_active":"2024-03-06T07:23:07.742222+0000","last_became_peered":"2024-03-06T07:23:07.742222+0000","last_unstale":"2024-03-06T07:23:08.745851+0000","last_undegraded":"2024-03-06T07:23:08.745851+0000","last_fullsized":"2024-03-06T07:23:08.745851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:46.656259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125817+0000","last_change":"2024-03-06T07:22:57.635044+0000","last_active":"2024-03-06T07:23:09.125817+0000","last_peered":"2024-03-06T07:23:09.125817+0000","last_clean":"2024-03-06T07:23:09.125817+0000","last_became_active":"2024-03-06T07:22:57.634107+0000","last_became_peered":"2024-03-06T07:22:57.634107+0000","last_unstale":"2024-03-06T07:23:09.125817+0000","last_undegraded":"2024-03-06T07:23:09.125817+0000","last_fullsized":"2024-03-06T07:23:09.125817+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:23.298584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740301+0000","last_change":"2024-03-06T07:23:07.741639+0000","last_active":"2024-03-06T07:23:08.740301+0000","last_peered":"2024-03-06T07:23:08.740301+0000","last_clean":"2024-03-06T07:23:08.740301+0000","last_became_active":"2024-03-06T07:23:07.741400+0000","last_became_peered":"2024-03-06T07:23:07.741400+0000","last_unstale":"2024-03-06T07:23:08.740301+0000","last_undegraded":"2024-03-06T07:23:08.740301+0000","last_fullsized":"2024-03-06T07:23:08.740301+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:03.722338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126375+0000","last_change":"2024-03-06T07:22:57.633281+0000","last_active":"2024-03-06T07:23:09.126375+0000","last_peered":"2024-03-06T07:23:09.126375+0000","last_clean":"2024-03-06T07:23:09.126375+0000","last_became_active":"2024-03-06T07:22:57.632984+0000","last_became_peered":"2024-03-06T07:22:57.632984+0000","last_unstale":"2024-03-06T07:23:09.126375+0000","last_undegraded":"2024-03-06T07:23:09.126375+0000","last_fullsized":"2024-03-06T07:23:09.126375+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:39.710112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127627+0000","last_change":"2024-03-06T07:23:07.746234+0000","last_active":"2024-03-06T07:23:09.127627+0000","last_peered":"2024-03-06T07:23:09.127627+0000","last_clean":"2024-03-06T07:23:09.127627+0000","last_became_active":"2024-03-06T07:23:07.745925+0000","last_became_peered":"2024-03-06T07:23:07.745925+0000","last_unstale":"2024-03-06T07:23:09.127627+0000","last_undegraded":"2024-03-06T07:23:09.127627+0000","last_fullsized":"2024-03-06T07:23:09.127627+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:43.274210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742984+0000","last_change":"2024-03-06T07:22:57.631914+0000","last_active":"2024-03-06T07:23:08.742984+0000","last_peered":"2024-03-06T07:23:08.742984+0000","last_clean":"2024-03-06T07:23:08.742984+0000","last_became_active":"2024-03-06T07:22:57.630752+0000","last_became_peered":"2024-03-06T07:22:57.630752+0000","last_unstale":"2024-03-06T07:23:08.742984+0000","last_undegraded":"2024-03-06T07:23:08.742984+0000","last_fullsized":"2024-03-06T07:23:08.742984+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:42.188842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741671+0000","last_change":"2024-03-06T07:23:07.742010+0000","last_active":"2024-03-06T07:23:08.741671+0000","last_peered":"2024-03-06T07:23:08.741671+0000","last_clean":"2024-03-06T07:23:08.741671+0000","last_became_active":"2024-03-06T07:23:07.741755+0000","last_became_peered":"2024-03-06T07:23:07.741755+0000","last_unstale":"2024-03-06T07:23:08.741671+0000","last_undegraded":"2024-03-06T07:23:08.741671+0000","last_fullsized":"2024-03-06T07:23:08.741671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:43.274210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742378+0000","last_change":"2024-03-06T07:22:57.631461+0000","last_active":"2024-03-06T07:23:08.742378+0000","last_peered":"2024-03-06T07:23:08.742378+0000","last_clean":"2024-03-06T07:23:08.742378+0000","last_became_active":"2024-03-06T07:22:57.631177+0000","last_became_peered":"2024-03-06T07:22:57.631177+0000","last_unstale":"2024-03-06T07:23:08.742378+0000","last_undegraded":"2024-03-06T07:23:08.742378+0000","last_fullsized":"2024-03-06T07:23:08.742378+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:59.182493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746127+0000","last_change":"2024-03-06T07:23:07.751228+0000","last_active":"2024-03-06T07:23:08.746127+0000","last_peered":"2024-03-06T07:23:08.746127+0000","last_clean":"2024-03-06T07:23:08.746127+0000","last_became_active":"2024-03-06T07:23:07.750661+0000","last_became_peered":"2024-03-06T07:23:07.750661+0000","last_unstale":"2024-03-06T07:23:08.746127+0000","last_undegraded":"2024-03-06T07:23:08.746127+0000","last_fullsized":"2024-03-06T07:23:08.746127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:49.818057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739887+0000","last_change":"2024-03-06T07:22:57.636696+0000","last_active":"2024-03-06T07:23:08.739887+0000","last_peered":"2024-03-06T07:23:08.739887+0000","last_clean":"2024-03-06T07:23:08.739887+0000","last_became_active":"2024-03-06T07:22:57.636414+0000","last_became_peered":"2024-03-06T07:22:57.636414+0000","last_unstale":"2024-03-06T07:23:08.739887+0000","last_undegraded":"2024-03-06T07:23:08.739887+0000","last_fullsized":"2024-03-06T07:23:08.739887+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:06.519659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741358+0000","last_change":"2024-03-06T07:23:07.736701+0000","last_active":"2024-03-06T07:23:08.741358+0000","last_peered":"2024-03-06T07:23:08.741358+0000","last_clean":"2024-03-06T07:23:08.741358+0000","last_became_active":"2024-03-06T07:23:07.736441+0000","last_became_peered":"2024-03-06T07:23:07.736441+0000","last_unstale":"2024-03-06T07:23:08.741358+0000","last_undegraded":"2024-03-06T07:23:08.741358+0000","last_fullsized":"2024-03-06T07:23:08.741358+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:52.340022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126167+0000","last_change":"2024-03-06T07:22:57.620687+0000","last_active":"2024-03-06T07:23:09.126167+0000","last_peered":"2024-03-06T07:23:09.126167+0000","last_clean":"2024-03-06T07:23:09.126167+0000","last_became_active":"2024-03-06T07:22:57.620416+0000","last_became_peered":"2024-03-06T07:22:57.620416+0000","last_unstale":"2024-03-06T07:23:09.126167+0000","last_undegraded":"2024-03-06T07:23:09.126167+0000","last_fullsized":"2024-03-06T07:23:09.126167+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:30.943768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125626+0000","last_change":"2024-03-06T07:23:07.739482+0000","last_active":"2024-03-06T07:23:09.125626+0000","last_peered":"2024-03-06T07:23:09.125626+0000","last_clean":"2024-03-06T07:23:09.125626+0000","last_became_active":"2024-03-06T07:23:07.738945+0000","last_became_peered":"2024-03-06T07:23:07.738945+0000","last_unstale":"2024-03-06T07:23:09.125626+0000","last_undegraded":"2024-03-06T07:23:09.125626+0000","last_fullsized":"2024-03-06T07:23:09.125626+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:37.230059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741073+0000","last_change":"2024-03-06T07:22:57.622616+0000","last_active":"2024-03-06T07:23:08.741073+0000","last_peered":"2024-03-06T07:23:08.741073+0000","last_clean":"2024-03-06T07:23:08.741073+0000","last_became_active":"2024-03-06T07:22:57.622314+0000","last_became_peered":"2024-03-06T07:22:57.622314+0000","last_unstale":"2024-03-06T07:23:08.741073+0000","last_undegraded":"2024-03-06T07:23:08.741073+0000","last_fullsized":"2024-03-06T07:23:08.741073+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:51.972559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742047+0000","last_change":"2024-03-06T07:23:07.748947+0000","last_active":"2024-03-06T07:23:08.742047+0000","last_peered":"2024-03-06T07:23:08.742047+0000","last_clean":"2024-03-06T07:23:08.742047+0000","last_became_active":"2024-03-06T07:23:07.748384+0000","last_became_peered":"2024-03-06T07:23:07.748384+0000","last_unstale":"2024-03-06T07:23:08.742047+0000","last_undegraded":"2024-03-06T07:23:08.742047+0000","last_fullsized":"2024-03-06T07:23:08.742047+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:43.387518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126393+0000","last_change":"2024-03-06T07:22:57.632401+0000","last_active":"2024-03-06T07:23:09.126393+0000","last_peered":"2024-03-06T07:23:09.126393+0000","last_clean":"2024-03-06T07:23:09.126393+0000","last_became_active":"2024-03-06T07:22:57.632063+0000","last_became_peered":"2024-03-06T07:22:57.632063+0000","last_unstale":"2024-03-06T07:23:09.126393+0000","last_undegraded":"2024-03-06T07:23:09.126393+0000","last_fullsized":"2024-03-06T07:23:09.126393+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:40.323690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745681+0000","last_change":"2024-03-06T07:23:07.748734+0000","last_active":"2024-03-06T07:23:08.745681+0000","last_peered":"2024-03-06T07:23:08.745681+0000","last_clean":"2024-03-06T07:23:08.745681+0000","last_became_active":"2024-03-06T07:23:07.748420+0000","last_became_peered":"2024-03-06T07:23:07.748420+0000","last_unstale":"2024-03-06T07:23:08.745681+0000","last_undegraded":"2024-03-06T07:23:08.745681+0000","last_fullsized":"2024-03-06T07:23:08.745681+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:10.248189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741568+0000","last_change":"2024-03-06T07:22:57.626369+0000","last_active":"2024-03-06T07:23:08.741568+0000","last_peered":"2024-03-06T07:23:08.741568+0000","last_clean":"2024-03-06T07:23:08.741568+0000","last_became_active":"2024-03-06T07:22:57.626083+0000","last_became_peered":"2024-03-06T07:22:57.626083+0000","last_unstale":"2024-03-06T07:23:08.741568+0000","last_undegraded":"2024-03-06T07:23:08.741568+0000","last_fullsized":"2024-03-06T07:23:08.741568+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:14.548525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126302+0000","last_change":"2024-03-06T07:23:07.746560+0000","last_active":"2024-03-06T07:23:09.126302+0000","last_peered":"2024-03-06T07:23:09.126302+0000","last_clean":"2024-03-06T07:23:09.126302+0000","last_became_active":"2024-03-06T07:23:07.745840+0000","last_became_peered":"2024-03-06T07:23:07.745840+0000","last_unstale":"2024-03-06T07:23:09.126302+0000","last_undegraded":"2024-03-06T07:23:09.126302+0000","last_fullsized":"2024-03-06T07:23:09.126302+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:59.539170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127491+0000","last_change":"2024-03-06T07:23:06.721815+0000","last_active":"2024-03-06T07:23:09.127491+0000","last_peered":"2024-03-06T07:23:09.127491+0000","last_clean":"2024-03-06T07:23:09.127491+0000","last_became_active":"2024-03-06T07:23:06.721564+0000","last_became_peered":"2024-03-06T07:23:06.721564+0000","last_unstale":"2024-03-06T07:23:09.127491+0000","last_undegraded":"2024-03-06T07:23:09.127491+0000","last_fullsized":"2024-03-06T07:23:09.127491+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:58.705339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127163+0000","last_change":"2024-03-06T07:22:57.626460+0000","last_active":"2024-03-06T07:23:09.127163+0000","last_peered":"2024-03-06T07:23:09.127163+0000","last_clean":"2024-03-06T07:23:09.127163+0000","last_became_active":"2024-03-06T07:22:57.626207+0000","last_became_peered":"2024-03-06T07:22:57.626207+0000","last_unstale":"2024-03-06T07:23:09.127163+0000","last_undegraded":"2024-03-06T07:23:09.127163+0000","last_fullsized":"2024-03-06T07:23:09.127163+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:27.403036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"73'4","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740486+0000","last_change":"2024-03-06T07:22:56.618360+0000","last_active":"2024-03-06T07:23:08.740486+0000","last_peered":"2024-03-06T07:23:08.740486+0000","last_clean":"2024-03-06T07:23:08.740486+0000","last_became_active":"2024-03-06T07:22:56.618085+0000","last_became_peered":"2024-03-06T07:22:56.618085+0000","last_unstale":"2024-03-06T07:23:08.740486+0000","last_undegraded":"2024-03-06T07:23:08.740486+0000","last_fullsized":"2024-03-06T07:23:08.740486+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:56.031916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127039+0000","last_change":"2024-03-06T07:23:07.742378+0000","last_active":"2024-03-06T07:23:09.127039+0000","last_peered":"2024-03-06T07:23:09.127039+0000","last_clean":"2024-03-06T07:23:09.127039+0000","last_became_active":"2024-03-06T07:23:07.742132+0000","last_became_peered":"2024-03-06T07:23:07.742132+0000","last_unstale":"2024-03-06T07:23:09.127039+0000","last_undegraded":"2024-03-06T07:23:09.127039+0000","last_fullsized":"2024-03-06T07:23:09.127039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:44.488836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741435+0000","last_change":"2024-03-06T07:23:06.726498+0000","last_active":"2024-03-06T07:23:08.741435+0000","last_peered":"2024-03-06T07:23:08.741435+0000","last_clean":"2024-03-06T07:23:08.741435+0000","last_became_active":"2024-03-06T07:23:06.726180+0000","last_became_peered":"2024-03-06T07:23:06.726180+0000","last_unstale":"2024-03-06T07:23:08.741435+0000","last_undegraded":"2024-03-06T07:23:08.741435+0000","last_fullsized":"2024-03-06T07:23:08.741435+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:51.332694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125783+0000","last_change":"2024-03-06T07:22:57.632027+0000","last_active":"2024-03-06T07:23:09.125783+0000","last_peered":"2024-03-06T07:23:09.125783+0000","last_clean":"2024-03-06T07:23:09.125783+0000","last_became_active":"2024-03-06T07:22:57.631736+0000","last_became_peered":"2024-03-06T07:22:57.631736+0000","last_unstale":"2024-03-06T07:23:09.125783+0000","last_undegraded":"2024-03-06T07:23:09.125783+0000","last_fullsized":"2024-03-06T07:23:09.125783+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:52.824136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"73'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743041+0000","last_change":"2024-03-06T07:22:56.623793+0000","last_active":"2024-03-06T07:23:08.743041+0000","last_peered":"2024-03-06T07:23:08.743041+0000","last_clean":"2024-03-06T07:23:08.743041+0000","last_became_active":"2024-03-06T07:22:56.623471+0000","last_became_peered":"2024-03-06T07:22:56.623471+0000","last_unstale":"2024-03-06T07:23:08.743041+0000","last_undegraded":"2024-03-06T07:23:08.743041+0000","last_fullsized":"2024-03-06T07:23:08.743041+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127086+0000","last_change":"2024-03-06T07:23:07.740395+0000","last_active":"2024-03-06T07:23:09.127086+0000","last_peered":"2024-03-06T07:23:09.127086+0000","last_clean":"2024-03-06T07:23:09.127086+0000","last_became_active":"2024-03-06T07:23:07.740122+0000","last_became_peered":"2024-03-06T07:23:07.740122+0000","last_unstale":"2024-03-06T07:23:09.127086+0000","last_undegraded":"2024-03-06T07:23:09.127086+0000","last_fullsized":"2024-03-06T07:23:09.127086+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:21.819423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745906+0000","last_change":"2024-03-06T07:23:06.725121+0000","last_active":"2024-03-06T07:23:08.745906+0000","last_peered":"2024-03-06T07:23:08.745906+0000","last_clean":"2024-03-06T07:23:08.745906+0000","last_became_active":"2024-03-06T07:23:06.723943+0000","last_became_peered":"2024-03-06T07:23:06.723943+0000","last_unstale":"2024-03-06T07:23:08.745906+0000","last_undegraded":"2024-03-06T07:23:08.745906+0000","last_fullsized":"2024-03-06T07:23:08.745906+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:57.364005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745836+0000","last_change":"2024-03-06T07:22:57.635705+0000","last_active":"2024-03-06T07:23:08.745836+0000","last_peered":"2024-03-06T07:23:08.745836+0000","last_clean":"2024-03-06T07:23:08.745836+0000","last_became_active":"2024-03-06T07:22:57.635457+0000","last_became_peered":"2024-03-06T07:22:57.635457+0000","last_unstale":"2024-03-06T07:23:08.745836+0000","last_undegraded":"2024-03-06T07:23:08.745836+0000","last_fullsized":"2024-03-06T07:23:08.745836+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:00.705050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742444+0000","last_change":"2024-03-06T07:22:56.614741+0000","last_active":"2024-03-06T07:23:08.742444+0000","last_peered":"2024-03-06T07:23:08.742444+0000","last_clean":"2024-03-06T07:23:08.742444+0000","last_became_active":"2024-03-06T07:22:56.614497+0000","last_became_peered":"2024-03-06T07:22:56.614497+0000","last_unstale":"2024-03-06T07:23:08.742444+0000","last_undegraded":"2024-03-06T07:23:08.742444+0000","last_fullsized":"2024-03-06T07:23:08.742444+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:42.726714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125998+0000","last_change":"2024-03-06T07:23:07.742049+0000","last_active":"2024-03-06T07:23:09.125998+0000","last_peered":"2024-03-06T07:23:09.125998+0000","last_clean":"2024-03-06T07:23:09.125998+0000","last_became_active":"2024-03-06T07:23:07.740767+0000","last_became_peered":"2024-03-06T07:23:07.740767+0000","last_unstale":"2024-03-06T07:23:09.125998+0000","last_undegraded":"2024-03-06T07:23:09.125998+0000","last_fullsized":"2024-03-06T07:23:09.125998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:55.819255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127839+0000","last_change":"2024-03-06T07:23:06.731842+0000","last_active":"2024-03-06T07:23:09.127839+0000","last_peered":"2024-03-06T07:23:09.127839+0000","last_clean":"2024-03-06T07:23:09.127839+0000","last_became_active":"2024-03-06T07:23:06.731449+0000","last_became_peered":"2024-03-06T07:23:06.731449+0000","last_unstale":"2024-03-06T07:23:09.127839+0000","last_undegraded":"2024-03-06T07:23:09.127839+0000","last_fullsized":"2024-03-06T07:23:09.127839+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:47.730530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746466+0000","last_change":"2024-03-06T07:22:57.625999+0000","last_active":"2024-03-06T07:23:08.746466+0000","last_peered":"2024-03-06T07:23:08.746466+0000","last_clean":"2024-03-06T07:23:08.746466+0000","last_became_active":"2024-03-06T07:22:57.625719+0000","last_became_peered":"2024-03-06T07:22:57.625719+0000","last_unstale":"2024-03-06T07:23:08.746466+0000","last_undegraded":"2024-03-06T07:23:08.746466+0000","last_fullsized":"2024-03-06T07:23:08.746466+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:51.153627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126204+0000","last_change":"2024-03-06T07:22:56.626144+0000","last_active":"2024-03-06T07:23:09.126204+0000","last_peered":"2024-03-06T07:23:09.126204+0000","last_clean":"2024-03-06T07:23:09.126204+0000","last_became_active":"2024-03-06T07:22:56.625739+0000","last_became_peered":"2024-03-06T07:22:56.625739+0000","last_unstale":"2024-03-06T07:23:09.126204+0000","last_undegraded":"2024-03-06T07:23:09.126204+0000","last_fullsized":"2024-03-06T07:23:09.126204+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:52.264835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742948+0000","last_change":"2024-03-06T07:23:07.747849+0000","last_active":"2024-03-06T07:23:08.742948+0000","last_peered":"2024-03-06T07:23:08.742948+0000","last_clean":"2024-03-06T07:23:08.742948+0000","last_became_active":"2024-03-06T07:23:07.747533+0000","last_became_peered":"2024-03-06T07:23:07.747533+0000","last_unstale":"2024-03-06T07:23:08.742948+0000","last_undegraded":"2024-03-06T07:23:08.742948+0000","last_fullsized":"2024-03-06T07:23:08.742948+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:31.516144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742386+0000","last_change":"2024-03-06T07:23:06.727025+0000","last_active":"2024-03-06T07:23:08.742386+0000","last_peered":"2024-03-06T07:23:08.742386+0000","last_clean":"2024-03-06T07:23:08.742386+0000","last_became_active":"2024-03-06T07:23:06.726749+0000","last_became_peered":"2024-03-06T07:23:06.726749+0000","last_unstale":"2024-03-06T07:23:08.742386+0000","last_undegraded":"2024-03-06T07:23:08.742386+0000","last_fullsized":"2024-03-06T07:23:08.742386+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:43.481508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126271+0000","last_change":"2024-03-06T07:22:57.633974+0000","last_active":"2024-03-06T07:23:09.126271+0000","last_peered":"2024-03-06T07:23:09.126271+0000","last_clean":"2024-03-06T07:23:09.126271+0000","last_became_active":"2024-03-06T07:22:57.633687+0000","last_became_peered":"2024-03-06T07:22:57.633687+0000","last_unstale":"2024-03-06T07:23:09.126271+0000","last_undegraded":"2024-03-06T07:23:09.126271+0000","last_fullsized":"2024-03-06T07:23:09.126271+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:49.816828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"73'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746623+0000","last_change":"2024-03-06T07:22:56.612153+0000","last_active":"2024-03-06T07:23:08.746623+0000","last_peered":"2024-03-06T07:23:08.746623+0000","last_clean":"2024-03-06T07:23:08.746623+0000","last_became_active":"2024-03-06T07:22:56.611872+0000","last_became_peered":"2024-03-06T07:22:56.611872+0000","last_unstale":"2024-03-06T07:23:08.746623+0000","last_undegraded":"2024-03-06T07:23:08.746623+0000","last_fullsized":"2024-03-06T07:23:08.746623+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:03.251378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126326+0000","last_change":"2024-03-06T07:23:07.742241+0000","last_active":"2024-03-06T07:23:09.126326+0000","last_peered":"2024-03-06T07:23:09.126326+0000","last_clean":"2024-03-06T07:23:09.126326+0000","last_became_active":"2024-03-06T07:23:07.741966+0000","last_became_peered":"2024-03-06T07:23:07.741966+0000","last_unstale":"2024-03-06T07:23:09.126326+0000","last_undegraded":"2024-03-06T07:23:09.126326+0000","last_fullsized":"2024-03-06T07:23:09.126326+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:50.360597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125991+0000","last_change":"2024-03-06T07:23:06.732613+0000","last_active":"2024-03-06T07:23:09.125991+0000","last_peered":"2024-03-06T07:23:09.125991+0000","last_clean":"2024-03-06T07:23:09.125991+0000","last_became_active":"2024-03-06T07:23:06.732320+0000","last_became_peered":"2024-03-06T07:23:06.732320+0000","last_unstale":"2024-03-06T07:23:09.125991+0000","last_undegraded":"2024-03-06T07:23:09.125991+0000","last_fullsized":"2024-03-06T07:23:09.125991+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:47.871427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127036+0000","last_change":"2024-03-06T07:22:57.632247+0000","last_active":"2024-03-06T07:23:09.127036+0000","last_peered":"2024-03-06T07:23:09.127036+0000","last_clean":"2024-03-06T07:23:09.127036+0000","last_became_active":"2024-03-06T07:22:57.631945+0000","last_became_peered":"2024-03-06T07:22:57.631945+0000","last_unstale":"2024-03-06T07:23:09.127036+0000","last_undegraded":"2024-03-06T07:23:09.127036+0000","last_fullsized":"2024-03-06T07:23:09.127036+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:11.711479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"73'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742656+0000","last_change":"2024-03-06T07:22:56.616078+0000","last_active":"2024-03-06T07:23:08.742656+0000","last_peered":"2024-03-06T07:23:08.742656+0000","last_clean":"2024-03-06T07:23:08.742656+0000","last_became_active":"2024-03-06T07:22:56.615754+0000","last_became_peered":"2024-03-06T07:22:56.615754+0000","last_unstale":"2024-03-06T07:23:08.742656+0000","last_undegraded":"2024-03-06T07:23:08.742656+0000","last_fullsized":"2024-03-06T07:23:08.742656+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:53.427081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125794+0000","last_change":"2024-03-06T07:23:07.754537+0000","last_active":"2024-03-06T07:23:09.125794+0000","last_peered":"2024-03-06T07:23:09.125794+0000","last_clean":"2024-03-06T07:23:09.125794+0000","last_became_active":"2024-03-06T07:23:07.754304+0000","last_became_peered":"2024-03-06T07:23:07.754304+0000","last_unstale":"2024-03-06T07:23:09.125794+0000","last_undegraded":"2024-03-06T07:23:09.125794+0000","last_fullsized":"2024-03-06T07:23:09.125794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:37.651359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"77'4","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126203+0000","last_change":"2024-03-06T07:23:06.727053+0000","last_active":"2024-03-06T07:23:09.126203+0000","last_peered":"2024-03-06T07:23:09.126203+0000","last_clean":"2024-03-06T07:23:09.126203+0000","last_became_active":"2024-03-06T07:23:06.726737+0000","last_became_peered":"2024-03-06T07:23:06.726737+0000","last_unstale":"2024-03-06T07:23:09.126203+0000","last_undegraded":"2024-03-06T07:23:09.126203+0000","last_fullsized":"2024-03-06T07:23:09.126203+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:55.528300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746802+0000","last_change":"2024-03-06T07:22:57.635125+0000","last_active":"2024-03-06T07:23:08.746802+0000","last_peered":"2024-03-06T07:23:08.746802+0000","last_clean":"2024-03-06T07:23:08.746802+0000","last_became_active":"2024-03-06T07:22:57.634829+0000","last_became_peered":"2024-03-06T07:22:57.634829+0000","last_unstale":"2024-03-06T07:23:08.746802+0000","last_undegraded":"2024-03-06T07:23:08.746802+0000","last_fullsized":"2024-03-06T07:23:08.746802+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:48.772981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125889+0000","last_change":"2024-03-06T07:22:56.612376+0000","last_active":"2024-03-06T07:23:09.125889+0000","last_peered":"2024-03-06T07:23:09.125889+0000","last_clean":"2024-03-06T07:23:09.125889+0000","last_became_active":"2024-03-06T07:22:56.612127+0000","last_became_peered":"2024-03-06T07:22:56.612127+0000","last_unstale":"2024-03-06T07:23:09.125889+0000","last_undegraded":"2024-03-06T07:23:09.125889+0000","last_fullsized":"2024-03-06T07:23:09.125889+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:12.032942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126271+0000","last_change":"2024-03-06T07:23:07.738816+0000","last_active":"2024-03-06T07:23:09.126271+0000","last_peered":"2024-03-06T07:23:09.126271+0000","last_clean":"2024-03-06T07:23:09.126271+0000","last_became_active":"2024-03-06T07:23:07.738316+0000","last_became_peered":"2024-03-06T07:23:07.738316+0000","last_unstale":"2024-03-06T07:23:09.126271+0000","last_undegraded":"2024-03-06T07:23:09.126271+0000","last_fullsized":"2024-03-06T07:23:09.126271+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:38.771898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746220+0000","last_change":"2024-03-06T07:23:06.724709+0000","last_active":"2024-03-06T07:23:08.746220+0000","last_peered":"2024-03-06T07:23:08.746220+0000","last_clean":"2024-03-06T07:23:08.746220+0000","last_became_active":"2024-03-06T07:23:06.724433+0000","last_became_peered":"2024-03-06T07:23:06.724433+0000","last_unstale":"2024-03-06T07:23:08.746220+0000","last_undegraded":"2024-03-06T07:23:08.746220+0000","last_fullsized":"2024-03-06T07:23:08.746220+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:09.784214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126325+0000","last_change":"2024-03-06T07:22:57.636772+0000","last_active":"2024-03-06T07:23:09.126325+0000","last_peered":"2024-03-06T07:23:09.126325+0000","last_clean":"2024-03-06T07:23:09.126325+0000","last_became_active":"2024-03-06T07:22:57.636474+0000","last_became_peered":"2024-03-06T07:22:57.636474+0000","last_unstale":"2024-03-06T07:23:09.126325+0000","last_undegraded":"2024-03-06T07:23:09.126325+0000","last_fullsized":"2024-03-06T07:23:09.126325+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:13.011040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739653+0000","last_change":"2024-03-06T07:22:56.619449+0000","last_active":"2024-03-06T07:23:08.739653+0000","last_peered":"2024-03-06T07:23:08.739653+0000","last_clean":"2024-03-06T07:23:08.739653+0000","last_became_active":"2024-03-06T07:22:56.619182+0000","last_became_peered":"2024-03-06T07:22:56.619182+0000","last_unstale":"2024-03-06T07:23:08.739653+0000","last_undegraded":"2024-03-06T07:23:08.739653+0000","last_fullsized":"2024-03-06T07:23:08.739653+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:43.349682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740162+0000","last_change":"2024-03-06T07:23:07.736967+0000","last_active":"2024-03-06T07:23:08.740162+0000","last_peered":"2024-03-06T07:23:08.740162+0000","last_clean":"2024-03-06T07:23:08.740162+0000","last_became_active":"2024-03-06T07:23:07.736723+0000","last_became_peered":"2024-03-06T07:23:07.736723+0000","last_unstale":"2024-03-06T07:23:08.740162+0000","last_undegraded":"2024-03-06T07:23:08.740162+0000","last_fullsized":"2024-03-06T07:23:08.740162+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:54.211266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126295+0000","last_change":"2024-03-06T07:23:06.737914+0000","last_active":"2024-03-06T07:23:09.126295+0000","last_peered":"2024-03-06T07:23:09.126295+0000","last_clean":"2024-03-06T07:23:09.126295+0000","last_became_active":"2024-03-06T07:23:06.737589+0000","last_became_peered":"2024-03-06T07:23:06.737589+0000","last_unstale":"2024-03-06T07:23:09.126295+0000","last_undegraded":"2024-03-06T07:23:09.126295+0000","last_fullsized":"2024-03-06T07:23:09.126295+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:52.543182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126390+0000","last_change":"2024-03-06T07:22:57.631217+0000","last_active":"2024-03-06T07:23:09.126390+0000","last_peered":"2024-03-06T07:23:09.126390+0000","last_clean":"2024-03-06T07:23:09.126390+0000","last_became_active":"2024-03-06T07:22:57.630879+0000","last_became_peered":"2024-03-06T07:22:57.630879+0000","last_unstale":"2024-03-06T07:23:09.126390+0000","last_undegraded":"2024-03-06T07:23:09.126390+0000","last_fullsized":"2024-03-06T07:23:09.126390+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:36.548315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743410+0000","last_change":"2024-03-06T07:22:56.623979+0000","last_active":"2024-03-06T07:23:08.743410+0000","last_peered":"2024-03-06T07:23:08.743410+0000","last_clean":"2024-03-06T07:23:08.743410+0000","last_became_active":"2024-03-06T07:22:56.623692+0000","last_became_peered":"2024-03-06T07:22:56.623692+0000","last_unstale":"2024-03-06T07:23:08.743410+0000","last_undegraded":"2024-03-06T07:23:08.743410+0000","last_fullsized":"2024-03-06T07:23:08.743410+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:12.032942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740161+0000","last_change":"2024-03-06T07:23:07.751324+0000","last_active":"2024-03-06T07:23:08.740161+0000","last_peered":"2024-03-06T07:23:08.740161+0000","last_clean":"2024-03-06T07:23:08.740161+0000","last_became_active":"2024-03-06T07:23:07.751055+0000","last_became_peered":"2024-03-06T07:23:07.751055+0000","last_unstale":"2024-03-06T07:23:08.740161+0000","last_undegraded":"2024-03-06T07:23:08.740161+0000","last_fullsized":"2024-03-06T07:23:08.740161+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:47.983462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"77'8","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127192+0000","last_change":"2024-03-06T07:23:06.732577+0000","last_active":"2024-03-06T07:23:09.127192+0000","last_peered":"2024-03-06T07:23:09.127192+0000","last_clean":"2024-03-06T07:23:09.127192+0000","last_became_active":"2024-03-06T07:23:06.732232+0000","last_became_peered":"2024-03-06T07:23:06.732232+0000","last_unstale":"2024-03-06T07:23:09.127192+0000","last_undegraded":"2024-03-06T07:23:09.127192+0000","last_fullsized":"2024-03-06T07:23:09.127192+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:51.289458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743207+0000","last_change":"2024-03-06T07:22:57.629315+0000","last_active":"2024-03-06T07:23:08.743207+0000","last_peered":"2024-03-06T07:23:08.743207+0000","last_clean":"2024-03-06T07:23:08.743207+0000","last_became_active":"2024-03-06T07:22:57.629010+0000","last_became_peered":"2024-03-06T07:22:57.629010+0000","last_unstale":"2024-03-06T07:23:08.743207+0000","last_undegraded":"2024-03-06T07:23:08.743207+0000","last_fullsized":"2024-03-06T07:23:08.743207+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:54.685052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743060+0000","last_change":"2024-03-06T07:22:56.615557+0000","last_active":"2024-03-06T07:23:08.743060+0000","last_peered":"2024-03-06T07:23:08.743060+0000","last_clean":"2024-03-06T07:23:08.743060+0000","last_became_active":"2024-03-06T07:22:56.615299+0000","last_became_peered":"2024-03-06T07:22:56.615299+0000","last_unstale":"2024-03-06T07:23:08.743060+0000","last_undegraded":"2024-03-06T07:23:08.743060+0000","last_fullsized":"2024-03-06T07:23:08.743060+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:03.445671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127126+0000","last_change":"2024-03-06T07:23:07.755149+0000","last_active":"2024-03-06T07:23:09.127126+0000","last_peered":"2024-03-06T07:23:09.127126+0000","last_clean":"2024-03-06T07:23:09.127126+0000","last_became_active":"2024-03-06T07:23:07.753813+0000","last_became_peered":"2024-03-06T07:23:07.753813+0000","last_unstale":"2024-03-06T07:23:09.127126+0000","last_undegraded":"2024-03-06T07:23:09.127126+0000","last_fullsized":"2024-03-06T07:23:09.127126+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:52.340022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742167+0000","last_change":"2024-03-06T07:23:06.728804+0000","last_active":"2024-03-06T07:23:08.742167+0000","last_peered":"2024-03-06T07:23:08.742167+0000","last_clean":"2024-03-06T07:23:08.742167+0000","last_became_active":"2024-03-06T07:23:06.728373+0000","last_became_peered":"2024-03-06T07:23:06.728373+0000","last_unstale":"2024-03-06T07:23:08.742167+0000","last_undegraded":"2024-03-06T07:23:08.742167+0000","last_fullsized":"2024-03-06T07:23:08.742167+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:55.134673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127805+0000","last_change":"2024-03-06T07:22:57.642696+0000","last_active":"2024-03-06T07:23:09.127805+0000","last_peered":"2024-03-06T07:23:09.127805+0000","last_clean":"2024-03-06T07:23:09.127805+0000","last_became_active":"2024-03-06T07:22:57.642201+0000","last_became_peered":"2024-03-06T07:22:57.642201+0000","last_unstale":"2024-03-06T07:23:09.127805+0000","last_undegraded":"2024-03-06T07:23:09.127805+0000","last_fullsized":"2024-03-06T07:23:09.127805+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:46.034056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"73'3","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740301+0000","last_change":"2024-03-06T07:22:56.620401+0000","last_active":"2024-03-06T07:23:08.740301+0000","last_peered":"2024-03-06T07:23:08.740301+0000","last_clean":"2024-03-06T07:23:08.740301+0000","last_became_active":"2024-03-06T07:22:56.620134+0000","last_became_peered":"2024-03-06T07:22:56.620134+0000","last_unstale":"2024-03-06T07:23:08.740301+0000","last_undegraded":"2024-03-06T07:23:08.740301+0000","last_fullsized":"2024-03-06T07:23:08.740301+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:48:07.935360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745514+0000","last_change":"2024-03-06T07:23:07.744492+0000","last_active":"2024-03-06T07:23:08.745514+0000","last_peered":"2024-03-06T07:23:08.745514+0000","last_clean":"2024-03-06T07:23:08.745514+0000","last_became_active":"2024-03-06T07:23:07.744029+0000","last_became_peered":"2024-03-06T07:23:07.744029+0000","last_unstale":"2024-03-06T07:23:08.745514+0000","last_undegraded":"2024-03-06T07:23:08.745514+0000","last_fullsized":"2024-03-06T07:23:08.745514+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:59.924773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126094+0000","last_change":"2024-03-06T07:23:06.725559+0000","last_active":"2024-03-06T07:23:09.126094+0000","last_peered":"2024-03-06T07:23:09.126094+0000","last_clean":"2024-03-06T07:23:09.126094+0000","last_became_active":"2024-03-06T07:23:06.725301+0000","last_became_peered":"2024-03-06T07:23:06.725301+0000","last_unstale":"2024-03-06T07:23:09.126094+0000","last_undegraded":"2024-03-06T07:23:09.126094+0000","last_fullsized":"2024-03-06T07:23:09.126094+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:38.201558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127145+0000","last_change":"2024-03-06T07:22:57.625109+0000","last_active":"2024-03-06T07:23:09.127145+0000","last_peered":"2024-03-06T07:23:09.127145+0000","last_clean":"2024-03-06T07:23:09.127145+0000","last_became_active":"2024-03-06T07:22:57.624843+0000","last_became_peered":"2024-03-06T07:22:57.624843+0000","last_unstale":"2024-03-06T07:23:09.127145+0000","last_undegraded":"2024-03-06T07:23:09.127145+0000","last_fullsized":"2024-03-06T07:23:09.127145+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:13.011040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"73'6","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126271+0000","last_change":"2024-03-06T07:22:56.623474+0000","last_active":"2024-03-06T07:23:09.126271+0000","last_peered":"2024-03-06T07:23:09.126271+0000","last_clean":"2024-03-06T07:23:09.126271+0000","last_became_active":"2024-03-06T07:22:56.623169+0000","last_became_peered":"2024-03-06T07:22:56.623169+0000","last_unstale":"2024-03-06T07:23:09.126271+0000","last_undegraded":"2024-03-06T07:23:09.126271+0000","last_fullsized":"2024-03-06T07:23:09.126271+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:37.241588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.128328+0000","last_change":"2024-03-06T07:23:07.737764+0000","last_active":"2024-03-06T07:23:09.128328+0000","last_peered":"2024-03-06T07:23:09.128328+0000","last_clean":"2024-03-06T07:23:09.128328+0000","last_became_active":"2024-03-06T07:23:07.737501+0000","last_became_peered":"2024-03-06T07:23:07.737501+0000","last_unstale":"2024-03-06T07:23:09.128328+0000","last_undegraded":"2024-03-06T07:23:09.128328+0000","last_fullsized":"2024-03-06T07:23:09.128328+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:34.624409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"77'3","reported_seq":17,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746877+0000","last_change":"2024-03-06T07:23:06.724961+0000","last_active":"2024-03-06T07:23:08.746877+0000","last_peered":"2024-03-06T07:23:08.746877+0000","last_clean":"2024-03-06T07:23:08.746877+0000","last_became_active":"2024-03-06T07:23:06.724704+0000","last_became_peered":"2024-03-06T07:23:06.724704+0000","last_unstale":"2024-03-06T07:23:08.746877+0000","last_undegraded":"2024-03-06T07:23:08.746877+0000","last_fullsized":"2024-03-06T07:23:08.746877+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:47.871427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742000+0000","last_change":"2024-03-06T07:22:57.635263+0000","last_active":"2024-03-06T07:23:08.742000+0000","last_peered":"2024-03-06T07:23:08.742000+0000","last_clean":"2024-03-06T07:23:08.742000+0000","last_became_active":"2024-03-06T07:22:57.635002+0000","last_became_peered":"2024-03-06T07:22:57.635002+0000","last_unstale":"2024-03-06T07:23:08.742000+0000","last_undegraded":"2024-03-06T07:23:08.742000+0000","last_fullsized":"2024-03-06T07:23:08.742000+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:48.906789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741550+0000","last_change":"2024-03-06T07:22:56.613079+0000","last_active":"2024-03-06T07:23:08.741550+0000","last_peered":"2024-03-06T07:23:08.741550+0000","last_clean":"2024-03-06T07:23:08.741550+0000","last_became_active":"2024-03-06T07:22:56.612788+0000","last_became_peered":"2024-03-06T07:22:56.612788+0000","last_unstale":"2024-03-06T07:23:08.741550+0000","last_undegraded":"2024-03-06T07:23:08.741550+0000","last_fullsized":"2024-03-06T07:23:08.741550+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:10.684793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126651+0000","last_change":"2024-03-06T07:23:07.744337+0000","last_active":"2024-03-06T07:23:09.126651+0000","last_peered":"2024-03-06T07:23:09.126651+0000","last_clean":"2024-03-06T07:23:09.126651+0000","last_became_active":"2024-03-06T07:23:07.744056+0000","last_became_peered":"2024-03-06T07:23:07.744056+0000","last_unstale":"2024-03-06T07:23:09.126651+0000","last_undegraded":"2024-03-06T07:23:09.126651+0000","last_fullsized":"2024-03-06T07:23:09.126651+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:33.406528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"77'6","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126159+0000","last_change":"2024-03-06T07:23:06.730832+0000","last_active":"2024-03-06T07:23:09.126159+0000","last_peered":"2024-03-06T07:23:09.126159+0000","last_clean":"2024-03-06T07:23:09.126159+0000","last_became_active":"2024-03-06T07:23:06.730568+0000","last_became_peered":"2024-03-06T07:23:06.730568+0000","last_unstale":"2024-03-06T07:23:09.126159+0000","last_undegraded":"2024-03-06T07:23:09.126159+0000","last_fullsized":"2024-03-06T07:23:09.126159+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:23:44.475424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125894+0000","last_change":"2024-03-06T07:22:57.623532+0000","last_active":"2024-03-06T07:23:09.125894+0000","last_peered":"2024-03-06T07:23:09.125894+0000","last_clean":"2024-03-06T07:23:09.125894+0000","last_became_active":"2024-03-06T07:22:57.623226+0000","last_became_peered":"2024-03-06T07:22:57.623226+0000","last_unstale":"2024-03-06T07:23:09.125894+0000","last_undegraded":"2024-03-06T07:23:09.125894+0000","last_fullsized":"2024-03-06T07:23:09.125894+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:55.687038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741204+0000","last_change":"2024-03-06T07:22:56.615633+0000","last_active":"2024-03-06T07:23:08.741204+0000","last_peered":"2024-03-06T07:23:08.741204+0000","last_clean":"2024-03-06T07:23:08.741204+0000","last_became_active":"2024-03-06T07:22:56.615358+0000","last_became_peered":"2024-03-06T07:22:56.615358+0000","last_unstale":"2024-03-06T07:23:08.741204+0000","last_undegraded":"2024-03-06T07:23:08.741204+0000","last_fullsized":"2024-03-06T07:23:08.741204+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:03.445671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746075+0000","last_change":"2024-03-06T07:23:07.744878+0000","last_active":"2024-03-06T07:23:08.746075+0000","last_peered":"2024-03-06T07:23:08.746075+0000","last_clean":"2024-03-06T07:23:08.746075+0000","last_became_active":"2024-03-06T07:23:07.744484+0000","last_became_peered":"2024-03-06T07:23:07.744484+0000","last_unstale":"2024-03-06T07:23:08.746075+0000","last_undegraded":"2024-03-06T07:23:08.746075+0000","last_fullsized":"2024-03-06T07:23:08.746075+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:41.051713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743331+0000","last_change":"2024-03-06T07:23:06.727628+0000","last_active":"2024-03-06T07:23:08.743331+0000","last_peered":"2024-03-06T07:23:08.743331+0000","last_clean":"2024-03-06T07:23:08.743331+0000","last_became_active":"2024-03-06T07:23:06.727351+0000","last_became_peered":"2024-03-06T07:23:06.727351+0000","last_unstale":"2024-03-06T07:23:08.743331+0000","last_undegraded":"2024-03-06T07:23:08.743331+0000","last_fullsized":"2024-03-06T07:23:08.743331+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:09.240860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126110+0000","last_change":"2024-03-06T07:22:57.628870+0000","last_active":"2024-03-06T07:23:09.126110+0000","last_peered":"2024-03-06T07:23:09.126110+0000","last_clean":"2024-03-06T07:23:09.126110+0000","last_became_active":"2024-03-06T07:22:57.627870+0000","last_became_peered":"2024-03-06T07:22:57.627870+0000","last_unstale":"2024-03-06T07:23:09.126110+0000","last_undegraded":"2024-03-06T07:23:09.126110+0000","last_fullsized":"2024-03-06T07:23:09.126110+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:40.323690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739568+0000","last_change":"2024-03-06T07:22:56.615747+0000","last_active":"2024-03-06T07:23:08.739568+0000","last_peered":"2024-03-06T07:23:08.739568+0000","last_clean":"2024-03-06T07:23:08.739568+0000","last_became_active":"2024-03-06T07:22:56.615466+0000","last_became_peered":"2024-03-06T07:22:56.615466+0000","last_unstale":"2024-03-06T07:23:08.739568+0000","last_undegraded":"2024-03-06T07:23:08.739568+0000","last_fullsized":"2024-03-06T07:23:08.739568+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:51.504717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742653+0000","last_change":"2024-03-06T07:23:07.747622+0000","last_active":"2024-03-06T07:23:08.742653+0000","last_peered":"2024-03-06T07:23:08.742653+0000","last_clean":"2024-03-06T07:23:08.742653+0000","last_became_active":"2024-03-06T07:23:07.747325+0000","last_became_peered":"2024-03-06T07:23:07.747325+0000","last_unstale":"2024-03-06T07:23:08.742653+0000","last_undegraded":"2024-03-06T07:23:08.742653+0000","last_fullsized":"2024-03-06T07:23:08.742653+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:32.325806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.128435+0000","last_change":"2024-03-06T07:23:06.729387+0000","last_active":"2024-03-06T07:23:09.128435+0000","last_peered":"2024-03-06T07:23:09.128435+0000","last_clean":"2024-03-06T07:23:09.128435+0000","last_became_active":"2024-03-06T07:23:06.729095+0000","last_became_peered":"2024-03-06T07:23:06.729095+0000","last_unstale":"2024-03-06T07:23:09.128435+0000","last_undegraded":"2024-03-06T07:23:09.128435+0000","last_fullsized":"2024-03-06T07:23:09.128435+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:54.647252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126345+0000","last_change":"2024-03-06T07:22:57.634493+0000","last_active":"2024-03-06T07:23:09.126345+0000","last_peered":"2024-03-06T07:23:09.126345+0000","last_clean":"2024-03-06T07:23:09.126345+0000","last_became_active":"2024-03-06T07:22:57.634215+0000","last_became_peered":"2024-03-06T07:22:57.634215+0000","last_unstale":"2024-03-06T07:23:09.126345+0000","last_undegraded":"2024-03-06T07:23:09.126345+0000","last_fullsized":"2024-03-06T07:23:09.126345+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:27.403036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"73'8","reported_seq":30,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126563+0000","last_change":"2024-03-06T07:22:56.623664+0000","last_active":"2024-03-06T07:23:09.126563+0000","last_peered":"2024-03-06T07:23:09.126563+0000","last_clean":"2024-03-06T07:23:09.126563+0000","last_became_active":"2024-03-06T07:22:56.623344+0000","last_became_peered":"2024-03-06T07:22:56.623344+0000","last_unstale":"2024-03-06T07:23:09.126563+0000","last_undegraded":"2024-03-06T07:23:09.126563+0000","last_fullsized":"2024-03-06T07:23:09.126563+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:37.765804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742608+0000","last_change":"2024-03-06T07:23:07.745122+0000","last_active":"2024-03-06T07:23:08.742608+0000","last_peered":"2024-03-06T07:23:08.742608+0000","last_clean":"2024-03-06T07:23:08.742608+0000","last_became_active":"2024-03-06T07:23:07.744847+0000","last_became_peered":"2024-03-06T07:23:07.744847+0000","last_unstale":"2024-03-06T07:23:08.742608+0000","last_undegraded":"2024-03-06T07:23:08.742608+0000","last_fullsized":"2024-03-06T07:23:08.742608+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:56.424088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741222+0000","last_change":"2024-03-06T07:23:06.726615+0000","last_active":"2024-03-06T07:23:08.741222+0000","last_peered":"2024-03-06T07:23:08.741222+0000","last_clean":"2024-03-06T07:23:08.741222+0000","last_became_active":"2024-03-06T07:23:06.726367+0000","last_became_peered":"2024-03-06T07:23:06.726367+0000","last_unstale":"2024-03-06T07:23:08.741222+0000","last_undegraded":"2024-03-06T07:23:08.741222+0000","last_fullsized":"2024-03-06T07:23:08.741222+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:01.924753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127302+0000","last_change":"2024-03-06T07:22:57.644383+0000","last_active":"2024-03-06T07:23:09.127302+0000","last_peered":"2024-03-06T07:23:09.127302+0000","last_clean":"2024-03-06T07:23:09.127302+0000","last_became_active":"2024-03-06T07:22:57.643011+0000","last_became_peered":"2024-03-06T07:22:57.643011+0000","last_unstale":"2024-03-06T07:23:09.127302+0000","last_undegraded":"2024-03-06T07:23:09.127302+0000","last_fullsized":"2024-03-06T07:23:09.127302+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:30.943768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"73'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743502+0000","last_change":"2024-03-06T07:22:56.622514+0000","last_active":"2024-03-06T07:23:08.743502+0000","last_peered":"2024-03-06T07:23:08.743502+0000","last_clean":"2024-03-06T07:23:08.743502+0000","last_became_active":"2024-03-06T07:22:56.622187+0000","last_became_peered":"2024-03-06T07:22:56.622187+0000","last_unstale":"2024-03-06T07:23:08.743502+0000","last_undegraded":"2024-03-06T07:23:08.743502+0000","last_fullsized":"2024-03-06T07:23:08.743502+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:37.765804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126378+0000","last_change":"2024-03-06T07:23:07.738572+0000","last_active":"2024-03-06T07:23:09.126378+0000","last_peered":"2024-03-06T07:23:09.126378+0000","last_clean":"2024-03-06T07:23:09.126378+0000","last_became_active":"2024-03-06T07:23:07.738291+0000","last_became_peered":"2024-03-06T07:23:07.738291+0000","last_unstale":"2024-03-06T07:23:09.126378+0000","last_undegraded":"2024-03-06T07:23:09.126378+0000","last_fullsized":"2024-03-06T07:23:09.126378+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:31.777109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739551+0000","last_change":"2024-03-06T07:23:06.722149+0000","last_active":"2024-03-06T07:23:08.739551+0000","last_peered":"2024-03-06T07:23:08.739551+0000","last_clean":"2024-03-06T07:23:08.739551+0000","last_became_active":"2024-03-06T07:23:06.721883+0000","last_became_peered":"2024-03-06T07:23:06.721883+0000","last_unstale":"2024-03-06T07:23:08.739551+0000","last_undegraded":"2024-03-06T07:23:08.739551+0000","last_fullsized":"2024-03-06T07:23:08.739551+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:19.808546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739567+0000","last_change":"2024-03-06T07:22:57.631467+0000","last_active":"2024-03-06T07:23:08.739567+0000","last_peered":"2024-03-06T07:23:08.739567+0000","last_clean":"2024-03-06T07:23:08.739567+0000","last_became_active":"2024-03-06T07:22:57.631215+0000","last_became_peered":"2024-03-06T07:22:57.631215+0000","last_unstale":"2024-03-06T07:23:08.739567+0000","last_undegraded":"2024-03-06T07:23:08.739567+0000","last_fullsized":"2024-03-06T07:23:08.739567+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:08.873797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746806+0000","last_change":"2024-03-06T07:22:56.619907+0000","last_active":"2024-03-06T07:23:08.746806+0000","last_peered":"2024-03-06T07:23:08.746806+0000","last_clean":"2024-03-06T07:23:08.746806+0000","last_became_active":"2024-03-06T07:22:56.619637+0000","last_became_peered":"2024-03-06T07:22:56.619637+0000","last_unstale":"2024-03-06T07:23:08.746806+0000","last_undegraded":"2024-03-06T07:23:08.746806+0000","last_fullsized":"2024-03-06T07:23:08.746806+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:37.241588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126511+0000","last_change":"2024-03-06T07:23:07.744643+0000","last_active":"2024-03-06T07:23:09.126511+0000","last_peered":"2024-03-06T07:23:09.126511+0000","last_clean":"2024-03-06T07:23:09.126511+0000","last_became_active":"2024-03-06T07:23:07.744365+0000","last_became_peered":"2024-03-06T07:23:07.744365+0000","last_unstale":"2024-03-06T07:23:09.126511+0000","last_undegraded":"2024-03-06T07:23:09.126511+0000","last_fullsized":"2024-03-06T07:23:09.126511+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:48.486294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741499+0000","last_change":"2024-03-06T07:23:06.721067+0000","last_active":"2024-03-06T07:23:08.741499+0000","last_peered":"2024-03-06T07:23:08.741499+0000","last_clean":"2024-03-06T07:23:08.741499+0000","last_became_active":"2024-03-06T07:23:06.720792+0000","last_became_peered":"2024-03-06T07:23:06.720792+0000","last_unstale":"2024-03-06T07:23:08.741499+0000","last_undegraded":"2024-03-06T07:23:08.741499+0000","last_fullsized":"2024-03-06T07:23:08.741499+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:35.380261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740231+0000","last_change":"2024-03-06T07:22:57.625588+0000","last_active":"2024-03-06T07:23:08.740231+0000","last_peered":"2024-03-06T07:23:08.740231+0000","last_clean":"2024-03-06T07:23:08.740231+0000","last_became_active":"2024-03-06T07:22:57.625218+0000","last_became_peered":"2024-03-06T07:22:57.625218+0000","last_unstale":"2024-03-06T07:23:08.740231+0000","last_undegraded":"2024-03-06T07:23:08.740231+0000","last_fullsized":"2024-03-06T07:23:08.740231+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:06.639365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"73'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740402+0000","last_change":"2024-03-06T07:22:56.618771+0000","last_active":"2024-03-06T07:23:08.740402+0000","last_peered":"2024-03-06T07:23:08.740402+0000","last_clean":"2024-03-06T07:23:08.740402+0000","last_became_active":"2024-03-06T07:22:56.618384+0000","last_became_peered":"2024-03-06T07:22:56.618384+0000","last_unstale":"2024-03-06T07:23:08.740402+0000","last_undegraded":"2024-03-06T07:23:08.740402+0000","last_fullsized":"2024-03-06T07:23:08.740402+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:04:26.439416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741142+0000","last_change":"2024-03-06T07:23:07.738983+0000","last_active":"2024-03-06T07:23:08.741142+0000","last_peered":"2024-03-06T07:23:08.741142+0000","last_clean":"2024-03-06T07:23:08.741142+0000","last_became_active":"2024-03-06T07:23:07.738682+0000","last_became_peered":"2024-03-06T07:23:07.738682+0000","last_unstale":"2024-03-06T07:23:08.741142+0000","last_undegraded":"2024-03-06T07:23:08.741142+0000","last_fullsized":"2024-03-06T07:23:08.741142+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:21.819423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740117+0000","last_change":"2024-03-06T07:23:06.730351+0000","last_active":"2024-03-06T07:23:08.740117+0000","last_peered":"2024-03-06T07:23:08.740117+0000","last_clean":"2024-03-06T07:23:08.740117+0000","last_became_active":"2024-03-06T07:23:06.729955+0000","last_became_peered":"2024-03-06T07:23:06.729955+0000","last_unstale":"2024-03-06T07:23:08.740117+0000","last_undegraded":"2024-03-06T07:23:08.740117+0000","last_fullsized":"2024-03-06T07:23:08.740117+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:49.986892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127914+0000","last_change":"2024-03-06T07:22:57.644130+0000","last_active":"2024-03-06T07:23:09.127914+0000","last_peered":"2024-03-06T07:23:09.127914+0000","last_clean":"2024-03-06T07:23:09.127914+0000","last_became_active":"2024-03-06T07:22:57.642544+0000","last_became_peered":"2024-03-06T07:22:57.642544+0000","last_unstale":"2024-03-06T07:23:09.127914+0000","last_undegraded":"2024-03-06T07:23:09.127914+0000","last_fullsized":"2024-03-06T07:23:09.127914+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:23.298584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127312+0000","last_change":"2024-03-06T07:22:56.622412+0000","last_active":"2024-03-06T07:23:09.127312+0000","last_peered":"2024-03-06T07:23:09.127312+0000","last_clean":"2024-03-06T07:23:09.127312+0000","last_became_active":"2024-03-06T07:22:56.622140+0000","last_became_peered":"2024-03-06T07:22:56.622140+0000","last_unstale":"2024-03-06T07:23:09.127312+0000","last_undegraded":"2024-03-06T07:23:09.127312+0000","last_fullsized":"2024-03-06T07:23:09.127312+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:37.728121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746664+0000","last_change":"2024-03-06T07:23:07.745239+0000","last_active":"2024-03-06T07:23:08.746664+0000","last_peered":"2024-03-06T07:23:08.746664+0000","last_clean":"2024-03-06T07:23:08.746664+0000","last_became_active":"2024-03-06T07:23:07.744624+0000","last_became_peered":"2024-03-06T07:23:07.744624+0000","last_unstale":"2024-03-06T07:23:08.746664+0000","last_undegraded":"2024-03-06T07:23:08.746664+0000","last_fullsized":"2024-03-06T07:23:08.746664+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:21.819423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746949+0000","last_change":"2024-03-06T07:23:06.729258+0000","last_active":"2024-03-06T07:23:08.746949+0000","last_peered":"2024-03-06T07:23:08.746949+0000","last_clean":"2024-03-06T07:23:08.746949+0000","last_became_active":"2024-03-06T07:23:06.728956+0000","last_became_peered":"2024-03-06T07:23:06.728956+0000","last_unstale":"2024-03-06T07:23:08.746949+0000","last_undegraded":"2024-03-06T07:23:08.746949+0000","last_fullsized":"2024-03-06T07:23:08.746949+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:06.454524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126728+0000","last_change":"2024-03-06T07:22:57.639832+0000","last_active":"2024-03-06T07:23:09.126728+0000","last_peered":"2024-03-06T07:23:09.126728+0000","last_clean":"2024-03-06T07:23:09.126728+0000","last_became_active":"2024-03-06T07:22:57.639055+0000","last_became_peered":"2024-03-06T07:22:57.639055+0000","last_unstale":"2024-03-06T07:23:09.126728+0000","last_undegraded":"2024-03-06T07:23:09.126728+0000","last_fullsized":"2024-03-06T07:23:09.126728+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:00.140244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.128024+0000","last_change":"2024-03-06T07:22:56.619240+0000","last_active":"2024-03-06T07:23:09.128024+0000","last_peered":"2024-03-06T07:23:09.128024+0000","last_clean":"2024-03-06T07:23:09.128024+0000","last_became_active":"2024-03-06T07:22:56.619024+0000","last_became_peered":"2024-03-06T07:22:56.619024+0000","last_unstale":"2024-03-06T07:23:09.128024+0000","last_undegraded":"2024-03-06T07:23:09.128024+0000","last_fullsized":"2024-03-06T07:23:09.128024+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:33.101169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125940+0000","last_change":"2024-03-06T07:23:07.745115+0000","last_active":"2024-03-06T07:23:09.125940+0000","last_peered":"2024-03-06T07:23:09.125940+0000","last_clean":"2024-03-06T07:23:09.125940+0000","last_became_active":"2024-03-06T07:23:07.744857+0000","last_became_peered":"2024-03-06T07:23:07.744857+0000","last_unstale":"2024-03-06T07:23:09.125940+0000","last_undegraded":"2024-03-06T07:23:09.125940+0000","last_fullsized":"2024-03-06T07:23:09.125940+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:02.205164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126305+0000","last_change":"2024-03-06T07:23:06.732714+0000","last_active":"2024-03-06T07:23:09.126305+0000","last_peered":"2024-03-06T07:23:09.126305+0000","last_clean":"2024-03-06T07:23:09.126305+0000","last_became_active":"2024-03-06T07:23:06.732402+0000","last_became_peered":"2024-03-06T07:23:06.732402+0000","last_unstale":"2024-03-06T07:23:09.126305+0000","last_undegraded":"2024-03-06T07:23:09.126305+0000","last_fullsized":"2024-03-06T07:23:09.126305+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:47.730530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126292+0000","last_change":"2024-03-06T07:22:57.626085+0000","last_active":"2024-03-06T07:23:09.126292+0000","last_peered":"2024-03-06T07:23:09.126292+0000","last_clean":"2024-03-06T07:23:09.126292+0000","last_became_active":"2024-03-06T07:22:57.625810+0000","last_became_peered":"2024-03-06T07:22:57.625810+0000","last_unstale":"2024-03-06T07:23:09.126292+0000","last_undegraded":"2024-03-06T07:23:09.126292+0000","last_fullsized":"2024-03-06T07:23:09.126292+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:33.166265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127788+0000","last_change":"2024-03-06T07:22:56.623577+0000","last_active":"2024-03-06T07:23:09.127788+0000","last_peered":"2024-03-06T07:23:09.127788+0000","last_clean":"2024-03-06T07:23:09.127788+0000","last_became_active":"2024-03-06T07:22:56.623267+0000","last_became_peered":"2024-03-06T07:22:56.623267+0000","last_unstale":"2024-03-06T07:23:09.127788+0000","last_undegraded":"2024-03-06T07:23:09.127788+0000","last_fullsized":"2024-03-06T07:23:09.127788+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:09.590972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125972+0000","last_change":"2024-03-06T07:23:07.740441+0000","last_active":"2024-03-06T07:23:09.125972+0000","last_peered":"2024-03-06T07:23:09.125972+0000","last_clean":"2024-03-06T07:23:09.125972+0000","last_became_active":"2024-03-06T07:23:07.740155+0000","last_became_peered":"2024-03-06T07:23:07.740155+0000","last_unstale":"2024-03-06T07:23:09.125972+0000","last_undegraded":"2024-03-06T07:23:09.125972+0000","last_fullsized":"2024-03-06T07:23:09.125972+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:44.214120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742971+0000","last_change":"2024-03-06T07:23:06.727098+0000","last_active":"2024-03-06T07:23:08.742971+0000","last_peered":"2024-03-06T07:23:08.742971+0000","last_clean":"2024-03-06T07:23:08.742971+0000","last_became_active":"2024-03-06T07:23:06.726830+0000","last_became_peered":"2024-03-06T07:23:06.726830+0000","last_unstale":"2024-03-06T07:23:08.742971+0000","last_undegraded":"2024-03-06T07:23:08.742971+0000","last_fullsized":"2024-03-06T07:23:08.742971+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:20.812095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126375+0000","last_change":"2024-03-06T07:22:57.631438+0000","last_active":"2024-03-06T07:23:09.126375+0000","last_peered":"2024-03-06T07:23:09.126375+0000","last_clean":"2024-03-06T07:23:09.126375+0000","last_became_active":"2024-03-06T07:22:57.631174+0000","last_became_peered":"2024-03-06T07:22:57.631174+0000","last_unstale":"2024-03-06T07:23:09.126375+0000","last_undegraded":"2024-03-06T07:23:09.126375+0000","last_fullsized":"2024-03-06T07:23:09.126375+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:54.432088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127395+0000","last_change":"2024-03-06T07:22:56.621728+0000","last_active":"2024-03-06T07:23:09.127395+0000","last_peered":"2024-03-06T07:23:09.127395+0000","last_clean":"2024-03-06T07:23:09.127395+0000","last_became_active":"2024-03-06T07:22:56.621425+0000","last_became_peered":"2024-03-06T07:22:56.621425+0000","last_unstale":"2024-03-06T07:23:09.127395+0000","last_undegraded":"2024-03-06T07:23:09.127395+0000","last_fullsized":"2024-03-06T07:23:09.127395+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:50.967553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125742+0000","last_change":"2024-03-06T07:23:07.745096+0000","last_active":"2024-03-06T07:23:09.125742+0000","last_peered":"2024-03-06T07:23:09.125742+0000","last_clean":"2024-03-06T07:23:09.125742+0000","last_became_active":"2024-03-06T07:23:07.744872+0000","last_became_peered":"2024-03-06T07:23:07.744872+0000","last_unstale":"2024-03-06T07:23:09.125742+0000","last_undegraded":"2024-03-06T07:23:09.125742+0000","last_fullsized":"2024-03-06T07:23:09.125742+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:20.703923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125560+0000","last_change":"2024-03-06T07:23:06.728087+0000","last_active":"2024-03-06T07:23:09.125560+0000","last_peered":"2024-03-06T07:23:09.125560+0000","last_clean":"2024-03-06T07:23:09.125560+0000","last_became_active":"2024-03-06T07:23:06.727827+0000","last_became_peered":"2024-03-06T07:23:06.727827+0000","last_unstale":"2024-03-06T07:23:09.125560+0000","last_undegraded":"2024-03-06T07:23:09.125560+0000","last_fullsized":"2024-03-06T07:23:09.125560+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:03.785668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745458+0000","last_change":"2024-03-06T07:22:57.629119+0000","last_active":"2024-03-06T07:23:08.745458+0000","last_peered":"2024-03-06T07:23:08.745458+0000","last_clean":"2024-03-06T07:23:08.745458+0000","last_became_active":"2024-03-06T07:22:57.628813+0000","last_became_peered":"2024-03-06T07:22:57.628813+0000","last_unstale":"2024-03-06T07:23:08.745458+0000","last_undegraded":"2024-03-06T07:23:08.745458+0000","last_fullsized":"2024-03-06T07:23:08.745458+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:38.733127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743517+0000","last_change":"2024-03-06T07:22:56.619641+0000","last_active":"2024-03-06T07:23:08.743517+0000","last_peered":"2024-03-06T07:23:08.743517+0000","last_clean":"2024-03-06T07:23:08.743517+0000","last_became_active":"2024-03-06T07:22:56.619369+0000","last_became_peered":"2024-03-06T07:22:56.619369+0000","last_unstale":"2024-03-06T07:23:08.743517+0000","last_undegraded":"2024-03-06T07:23:08.743517+0000","last_fullsized":"2024-03-06T07:23:08.743517+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126001+0000","last_change":"2024-03-06T07:23:07.738105+0000","last_active":"2024-03-06T07:23:09.126001+0000","last_peered":"2024-03-06T07:23:09.126001+0000","last_clean":"2024-03-06T07:23:09.126001+0000","last_became_active":"2024-03-06T07:23:07.737822+0000","last_became_peered":"2024-03-06T07:23:07.737822+0000","last_unstale":"2024-03-06T07:23:09.126001+0000","last_undegraded":"2024-03-06T07:23:09.126001+0000","last_fullsized":"2024-03-06T07:23:09.126001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:52.763326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741161+0000","last_change":"2024-03-06T07:23:06.732290+0000","last_active":"2024-03-06T07:23:08.741161+0000","last_peered":"2024-03-06T07:23:08.741161+0000","last_clean":"2024-03-06T07:23:08.741161+0000","last_became_active":"2024-03-06T07:23:06.731993+0000","last_became_peered":"2024-03-06T07:23:06.731993+0000","last_unstale":"2024-03-06T07:23:08.741161+0000","last_undegraded":"2024-03-06T07:23:08.741161+0000","last_fullsized":"2024-03-06T07:23:08.741161+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:12.475972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125800+0000","last_change":"2024-03-06T07:22:57.634967+0000","last_active":"2024-03-06T07:23:09.125800+0000","last_peered":"2024-03-06T07:23:09.125800+0000","last_clean":"2024-03-06T07:23:09.125800+0000","last_became_active":"2024-03-06T07:22:57.634367+0000","last_became_peered":"2024-03-06T07:22:57.634367+0000","last_unstale":"2024-03-06T07:23:09.125800+0000","last_undegraded":"2024-03-06T07:23:09.125800+0000","last_fullsized":"2024-03-06T07:23:09.125800+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:30.081287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"74'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126399+0000","last_change":"2024-03-06T07:22:56.609890+0000","last_active":"2024-03-06T07:23:09.126399+0000","last_peered":"2024-03-06T07:23:09.126399+0000","last_clean":"2024-03-06T07:23:09.126399+0000","last_became_active":"2024-03-06T07:22:56.609470+0000","last_became_peered":"2024-03-06T07:22:56.609470+0000","last_unstale":"2024-03-06T07:23:09.126399+0000","last_undegraded":"2024-03-06T07:23:09.126399+0000","last_fullsized":"2024-03-06T07:23:09.126399+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:56.348901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739545+0000","last_change":"2024-03-06T07:23:07.732754+0000","last_active":"2024-03-06T07:23:08.739545+0000","last_peered":"2024-03-06T07:23:08.739545+0000","last_clean":"2024-03-06T07:23:08.739545+0000","last_became_active":"2024-03-06T07:23:07.732499+0000","last_became_peered":"2024-03-06T07:23:07.732499+0000","last_unstale":"2024-03-06T07:23:08.739545+0000","last_undegraded":"2024-03-06T07:23:08.739545+0000","last_fullsized":"2024-03-06T07:23:08.739545+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:03.377787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126054+0000","last_change":"2024-03-06T07:23:06.727720+0000","last_active":"2024-03-06T07:23:09.126054+0000","last_peered":"2024-03-06T07:23:09.126054+0000","last_clean":"2024-03-06T07:23:09.126054+0000","last_became_active":"2024-03-06T07:23:06.726098+0000","last_became_peered":"2024-03-06T07:23:06.726098+0000","last_unstale":"2024-03-06T07:23:09.126054+0000","last_undegraded":"2024-03-06T07:23:09.126054+0000","last_fullsized":"2024-03-06T07:23:09.126054+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:42.380189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745908+0000","last_change":"2024-03-06T07:22:57.634820+0000","last_active":"2024-03-06T07:23:08.745908+0000","last_peered":"2024-03-06T07:23:08.745908+0000","last_clean":"2024-03-06T07:23:08.745908+0000","last_became_active":"2024-03-06T07:22:57.634538+0000","last_became_peered":"2024-03-06T07:22:57.634538+0000","last_unstale":"2024-03-06T07:23:08.745908+0000","last_undegraded":"2024-03-06T07:23:08.745908+0000","last_fullsized":"2024-03-06T07:23:08.745908+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:30.460111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742805+0000","last_change":"2024-03-06T07:22:56.616457+0000","last_active":"2024-03-06T07:23:08.742805+0000","last_peered":"2024-03-06T07:23:08.742805+0000","last_clean":"2024-03-06T07:23:08.742805+0000","last_became_active":"2024-03-06T07:22:56.616201+0000","last_became_peered":"2024-03-06T07:22:56.616201+0000","last_unstale":"2024-03-06T07:23:08.742805+0000","last_undegraded":"2024-03-06T07:23:08.742805+0000","last_fullsized":"2024-03-06T07:23:08.742805+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:59.678591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127735+0000","last_change":"2024-03-06T07:23:07.745128+0000","last_active":"2024-03-06T07:23:09.127735+0000","last_peered":"2024-03-06T07:23:09.127735+0000","last_clean":"2024-03-06T07:23:09.127735+0000","last_became_active":"2024-03-06T07:23:07.744649+0000","last_became_peered":"2024-03-06T07:23:07.744649+0000","last_unstale":"2024-03-06T07:23:09.127735+0000","last_undegraded":"2024-03-06T07:23:09.127735+0000","last_fullsized":"2024-03-06T07:23:09.127735+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:27.698854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742463+0000","last_change":"2024-03-06T07:23:06.724656+0000","last_active":"2024-03-06T07:23:08.742463+0000","last_peered":"2024-03-06T07:23:08.742463+0000","last_clean":"2024-03-06T07:23:08.742463+0000","last_became_active":"2024-03-06T07:23:06.724337+0000","last_became_peered":"2024-03-06T07:23:06.724337+0000","last_unstale":"2024-03-06T07:23:08.742463+0000","last_undegraded":"2024-03-06T07:23:08.742463+0000","last_fullsized":"2024-03-06T07:23:08.742463+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:58.917445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126425+0000","last_change":"2024-03-06T07:22:57.630052+0000","last_active":"2024-03-06T07:23:09.126425+0000","last_peered":"2024-03-06T07:23:09.126425+0000","last_clean":"2024-03-06T07:23:09.126425+0000","last_became_active":"2024-03-06T07:22:57.629722+0000","last_became_peered":"2024-03-06T07:22:57.629722+0000","last_unstale":"2024-03-06T07:23:09.126425+0000","last_undegraded":"2024-03-06T07:23:09.126425+0000","last_fullsized":"2024-03-06T07:23:09.126425+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:34.380891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741889+0000","last_change":"2024-03-06T07:22:56.624178+0000","last_active":"2024-03-06T07:23:08.741889+0000","last_peered":"2024-03-06T07:23:08.741889+0000","last_clean":"2024-03-06T07:23:08.741889+0000","last_became_active":"2024-03-06T07:22:56.623706+0000","last_became_peered":"2024-03-06T07:22:56.623706+0000","last_unstale":"2024-03-06T07:23:08.741889+0000","last_undegraded":"2024-03-06T07:23:08.741889+0000","last_fullsized":"2024-03-06T07:23:08.741889+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:47.258382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125890+0000","last_change":"2024-03-06T07:23:07.737927+0000","last_active":"2024-03-06T07:23:09.125890+0000","last_peered":"2024-03-06T07:23:09.125890+0000","last_clean":"2024-03-06T07:23:09.125890+0000","last_became_active":"2024-03-06T07:23:07.737675+0000","last_became_peered":"2024-03-06T07:23:07.737675+0000","last_unstale":"2024-03-06T07:23:09.125890+0000","last_undegraded":"2024-03-06T07:23:09.125890+0000","last_fullsized":"2024-03-06T07:23:09.125890+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:15.011419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746966+0000","last_change":"2024-03-06T07:23:06.724789+0000","last_active":"2024-03-06T07:23:08.746966+0000","last_peered":"2024-03-06T07:23:08.746966+0000","last_clean":"2024-03-06T07:23:08.746966+0000","last_became_active":"2024-03-06T07:23:06.724461+0000","last_became_peered":"2024-03-06T07:23:06.724461+0000","last_unstale":"2024-03-06T07:23:08.746966+0000","last_undegraded":"2024-03-06T07:23:08.746966+0000","last_fullsized":"2024-03-06T07:23:08.746966+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:03.785668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126100+0000","last_change":"2024-03-06T07:22:57.633736+0000","last_active":"2024-03-06T07:23:09.126100+0000","last_peered":"2024-03-06T07:23:09.126100+0000","last_clean":"2024-03-06T07:23:09.126100+0000","last_became_active":"2024-03-06T07:22:57.633502+0000","last_became_peered":"2024-03-06T07:22:57.633502+0000","last_unstale":"2024-03-06T07:23:09.126100+0000","last_undegraded":"2024-03-06T07:23:09.126100+0000","last_fullsized":"2024-03-06T07:23:09.126100+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:00.705050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125760+0000","last_change":"2024-03-06T07:22:56.626141+0000","last_active":"2024-03-06T07:23:09.125760+0000","last_peered":"2024-03-06T07:23:09.125760+0000","last_clean":"2024-03-06T07:23:09.125760+0000","last_became_active":"2024-03-06T07:22:56.625760+0000","last_became_peered":"2024-03-06T07:22:56.625760+0000","last_unstale":"2024-03-06T07:23:09.125760+0000","last_undegraded":"2024-03-06T07:23:09.125760+0000","last_fullsized":"2024-03-06T07:23:09.125760+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:03.251378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743336+0000","last_change":"2024-03-06T07:23:07.754097+0000","last_active":"2024-03-06T07:23:08.743336+0000","last_peered":"2024-03-06T07:23:08.743336+0000","last_clean":"2024-03-06T07:23:08.743336+0000","last_became_active":"2024-03-06T07:23:07.753821+0000","last_became_peered":"2024-03-06T07:23:07.753821+0000","last_unstale":"2024-03-06T07:23:08.743336+0000","last_undegraded":"2024-03-06T07:23:08.743336+0000","last_fullsized":"2024-03-06T07:23:08.743336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:59.712667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126113+0000","last_change":"2024-03-06T07:23:06.728083+0000","last_active":"2024-03-06T07:23:09.126113+0000","last_peered":"2024-03-06T07:23:09.126113+0000","last_clean":"2024-03-06T07:23:09.126113+0000","last_became_active":"2024-03-06T07:23:06.727874+0000","last_became_peered":"2024-03-06T07:23:06.727874+0000","last_unstale":"2024-03-06T07:23:09.126113+0000","last_undegraded":"2024-03-06T07:23:09.126113+0000","last_fullsized":"2024-03-06T07:23:09.126113+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:06.454524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126000+0000","last_change":"2024-03-06T07:22:57.628631+0000","last_active":"2024-03-06T07:23:09.126000+0000","last_peered":"2024-03-06T07:23:09.126000+0000","last_clean":"2024-03-06T07:23:09.126000+0000","last_became_active":"2024-03-06T07:22:57.627708+0000","last_became_peered":"2024-03-06T07:22:57.627708+0000","last_unstale":"2024-03-06T07:23:09.126000+0000","last_undegraded":"2024-03-06T07:23:09.126000+0000","last_fullsized":"2024-03-06T07:23:09.126000+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:30.460111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740318+0000","last_change":"2024-03-06T07:22:56.615838+0000","last_active":"2024-03-06T07:23:08.740318+0000","last_peered":"2024-03-06T07:23:08.740318+0000","last_clean":"2024-03-06T07:23:08.740318+0000","last_became_active":"2024-03-06T07:22:56.615546+0000","last_became_peered":"2024-03-06T07:22:56.615546+0000","last_unstale":"2024-03-06T07:23:08.740318+0000","last_undegraded":"2024-03-06T07:23:08.740318+0000","last_fullsized":"2024-03-06T07:23:08.740318+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:52.858792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126305+0000","last_change":"2024-03-06T07:23:07.755901+0000","last_active":"2024-03-06T07:23:09.126305+0000","last_peered":"2024-03-06T07:23:09.126305+0000","last_clean":"2024-03-06T07:23:09.126305+0000","last_became_active":"2024-03-06T07:23:07.753264+0000","last_became_peered":"2024-03-06T07:23:07.753264+0000","last_unstale":"2024-03-06T07:23:09.126305+0000","last_undegraded":"2024-03-06T07:23:09.126305+0000","last_fullsized":"2024-03-06T07:23:09.126305+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:03.844554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743133+0000","last_change":"2024-03-06T07:23:06.730657+0000","last_active":"2024-03-06T07:23:08.743133+0000","last_peered":"2024-03-06T07:23:08.743133+0000","last_clean":"2024-03-06T07:23:08.743133+0000","last_became_active":"2024-03-06T07:23:06.730347+0000","last_became_peered":"2024-03-06T07:23:06.730347+0000","last_unstale":"2024-03-06T07:23:08.743133+0000","last_undegraded":"2024-03-06T07:23:08.743133+0000","last_fullsized":"2024-03-06T07:23:08.743133+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:42.266882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126840+0000","last_change":"2024-03-06T07:22:57.625874+0000","last_active":"2024-03-06T07:23:09.126840+0000","last_peered":"2024-03-06T07:23:09.126840+0000","last_clean":"2024-03-06T07:23:09.126840+0000","last_became_active":"2024-03-06T07:22:57.625574+0000","last_became_peered":"2024-03-06T07:22:57.625574+0000","last_unstale":"2024-03-06T07:23:09.126840+0000","last_undegraded":"2024-03-06T07:23:09.126840+0000","last_fullsized":"2024-03-06T07:23:09.126840+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:28.600878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125942+0000","last_change":"2024-03-06T07:22:56.620322+0000","last_active":"2024-03-06T07:23:09.125942+0000","last_peered":"2024-03-06T07:23:09.125942+0000","last_clean":"2024-03-06T07:23:09.125942+0000","last_became_active":"2024-03-06T07:22:56.620052+0000","last_became_peered":"2024-03-06T07:22:56.620052+0000","last_unstale":"2024-03-06T07:23:09.125942+0000","last_undegraded":"2024-03-06T07:23:09.125942+0000","last_fullsized":"2024-03-06T07:23:09.125942+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:36.870510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126672+0000","last_change":"2024-03-06T07:23:07.743019+0000","last_active":"2024-03-06T07:23:09.126672+0000","last_peered":"2024-03-06T07:23:09.126672+0000","last_clean":"2024-03-06T07:23:09.126672+0000","last_became_active":"2024-03-06T07:23:07.742795+0000","last_became_peered":"2024-03-06T07:23:07.742795+0000","last_unstale":"2024-03-06T07:23:09.126672+0000","last_undegraded":"2024-03-06T07:23:09.126672+0000","last_fullsized":"2024-03-06T07:23:09.126672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:22.867840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"78'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.748183+0000","last_change":"2024-03-06T07:23:06.729756+0000","last_active":"2024-03-06T07:23:08.748183+0000","last_peered":"2024-03-06T07:23:08.748183+0000","last_clean":"2024-03-06T07:23:08.748183+0000","last_became_active":"2024-03-06T07:23:06.729524+0000","last_became_peered":"2024-03-06T07:23:06.729524+0000","last_unstale":"2024-03-06T07:23:08.748183+0000","last_undegraded":"2024-03-06T07:23:08.748183+0000","last_fullsized":"2024-03-06T07:23:08.748183+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:22.138565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126758+0000","last_change":"2024-03-06T07:22:57.624542+0000","last_active":"2024-03-06T07:23:09.126758+0000","last_peered":"2024-03-06T07:23:09.126758+0000","last_clean":"2024-03-06T07:23:09.126758+0000","last_became_active":"2024-03-06T07:22:57.624258+0000","last_became_peered":"2024-03-06T07:22:57.624258+0000","last_unstale":"2024-03-06T07:23:09.126758+0000","last_undegraded":"2024-03-06T07:23:09.126758+0000","last_fullsized":"2024-03-06T07:23:09.126758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:11.689800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127249+0000","last_change":"2024-03-06T07:22:56.621595+0000","last_active":"2024-03-06T07:23:09.127249+0000","last_peered":"2024-03-06T07:23:09.127249+0000","last_clean":"2024-03-06T07:23:09.127249+0000","last_became_active":"2024-03-06T07:22:56.621316+0000","last_became_peered":"2024-03-06T07:22:56.621316+0000","last_unstale":"2024-03-06T07:23:09.127249+0000","last_undegraded":"2024-03-06T07:23:09.127249+0000","last_fullsized":"2024-03-06T07:23:09.127249+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:40.228473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743440+0000","last_change":"2024-03-06T07:23:07.749592+0000","last_active":"2024-03-06T07:23:08.743440+0000","last_peered":"2024-03-06T07:23:08.743440+0000","last_clean":"2024-03-06T07:23:08.743440+0000","last_became_active":"2024-03-06T07:23:07.747627+0000","last_became_peered":"2024-03-06T07:23:07.747627+0000","last_unstale":"2024-03-06T07:23:08.743440+0000","last_undegraded":"2024-03-06T07:23:08.743440+0000","last_fullsized":"2024-03-06T07:23:08.743440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:01.336551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127200+0000","last_change":"2024-03-06T07:23:06.737910+0000","last_active":"2024-03-06T07:23:09.127200+0000","last_peered":"2024-03-06T07:23:09.127200+0000","last_clean":"2024-03-06T07:23:09.127200+0000","last_became_active":"2024-03-06T07:23:06.737677+0000","last_became_peered":"2024-03-06T07:23:06.737677+0000","last_unstale":"2024-03-06T07:23:09.127200+0000","last_undegraded":"2024-03-06T07:23:09.127200+0000","last_fullsized":"2024-03-06T07:23:09.127200+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:58.007406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742706+0000","last_change":"2024-03-06T07:22:57.623523+0000","last_active":"2024-03-06T07:23:08.742706+0000","last_peered":"2024-03-06T07:23:08.742706+0000","last_clean":"2024-03-06T07:23:08.742706+0000","last_became_active":"2024-03-06T07:22:57.623281+0000","last_became_peered":"2024-03-06T07:22:57.623281+0000","last_unstale":"2024-03-06T07:23:08.742706+0000","last_undegraded":"2024-03-06T07:23:08.742706+0000","last_fullsized":"2024-03-06T07:23:08.742706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:34.106175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127887+0000","last_change":"2024-03-06T07:22:56.616740+0000","last_active":"2024-03-06T07:23:09.127887+0000","last_peered":"2024-03-06T07:23:09.127887+0000","last_clean":"2024-03-06T07:23:09.127887+0000","last_became_active":"2024-03-06T07:22:56.614750+0000","last_became_peered":"2024-03-06T07:22:56.614750+0000","last_unstale":"2024-03-06T07:23:09.127887+0000","last_undegraded":"2024-03-06T07:23:09.127887+0000","last_fullsized":"2024-03-06T07:23:09.127887+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:58.177487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":282,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741659+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:23:08.741659+0000","last_peered":"2024-03-06T07:23:08.741659+0000","last_clean":"2024-03-06T07:23:08.741659+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:23:08.741659+0000","last_undegraded":"2024-03-06T07:23:08.741659+0000","last_fullsized":"2024-03-06T07:23:08.741659+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339648,"statfs":{"total":96636764160,"available":96603799552,"internally_reserved":0,"allocated":1032192,"data_stored":1048614,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":974848,"data_stored":454447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":958464,"data_stored":458202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574886,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":983040,"data_stored":460044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339720,"statfs":{"total":96636764160,"available":96603873280,"internally_reserved":0,"allocated":958464,"data_stored":458750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":966656,"data_stored":458910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574886,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":966656,"data_stored":458350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1044480,"data_stored":1050854,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:11.791 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:11.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.921+0000 7f71729f7640 1 Processor -- start 2024-03-06T07:23:11.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.921+0000 7f71729f7640 1 -- start start 2024-03-06T07:23:11.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.922+0000 7f71729f7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f716c152650 0x7f716c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.922+0000 7f71729f7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f716c0c2fc0 con 0x7f716c0c6b10 2024-03-06T07:23:11.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.922+0000 7f71729f7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f716c158da0 con 0x7f716c0bf9d0 2024-03-06T07:23:11.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.922+0000 7f71729f7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f716c155800 con 0x7f716c152650 2024-03-06T07:23:11.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.922+0000 7f716b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f716c0c6b10 0x7f716c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45234/0 (socket says 172.21.15.1:45234) 2024-03-06T07:23:11.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.922+0000 7f716b7fe640 1 -- 172.21.15.1:0/1402245926 learned_addr learned my addr 172.21.15.1:0/1402245926 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:11.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.922+0000 7f716bfff640 1 --2- 172.21.15.1:0/1402245926 >> v2:172.21.15.98:3300/0 conn(0x7f716c152650 0x7f716c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.923+0000 7f716affd640 1 -- 172.21.15.1:0/1402245926 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165659409 0 0) 0x7f716c0c2fc0 con 0x7f716c0c6b10 2024-03-06T07:23:11.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.923+0000 7f716affd640 1 -- 172.21.15.1:0/1402245926 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7154003520 con 0x7f716c0c6b10 2024-03-06T07:23:11.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.923+0000 7f716bfff640 1 -- 172.21.15.1:0/1402245926 >> v1:172.21.15.98:6789/0 conn(0x7f716c0bf9d0 legacy=0x7f716c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.923+0000 7f716affd640 1 -- 172.21.15.1:0/1402245926 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303059105 0 0) 0x7f716c158da0 con 0x7f716c0bf9d0 2024-03-06T07:23:11.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.923+0000 7f716bfff640 1 -- 172.21.15.1:0/1402245926 >> v1:172.21.15.1:6789/0 conn(0x7f716c0c6b10 legacy=0x7f716c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.923+0000 7f716bfff640 1 -- 172.21.15.1:0/1402245926 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716c0c6690 con 0x7f716c152650 2024-03-06T07:23:11.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.924+0000 7f716bfff640 1 --2- 172.21.15.1:0/1402245926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c152650 0x7f716c155c90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f715c0097a0 tx=0x7f715c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd54c4449af3e0f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.924+0000 7f716affd640 1 -- 172.21.15.1:0/1402245926 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715c035020 con 0x7f716c152650 2024-03-06T07:23:11.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.924+0000 7f716affd640 1 -- 172.21.15.1:0/1402245926 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715c031830 con 0x7f716c152650 2024-03-06T07:23:11.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.924+0000 7f716affd640 1 -- 172.21.15.1:0/1402245926 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715c031c00 con 0x7f716c152650 2024-03-06T07:23:11.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.924+0000 7f71729f7640 1 -- 172.21.15.1:0/1402245926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c152650 msgr2=0x7f716c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.924+0000 7f71729f7640 1 --2- 172.21.15.1:0/1402245926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c152650 0x7f716c155c90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f715c0097a0 tx=0x7f715c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.924+0000 7f71729f7640 1 -- 172.21.15.1:0/1402245926 shutdown_connections 2024-03-06T07:23:11.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.924+0000 7f71729f7640 1 --2- 172.21.15.1:0/1402245926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c152650 0x7f716c155c90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.924+0000 7f71729f7640 1 -- 172.21.15.1:0/1402245926 >> 172.21.15.1:0/1402245926 conn(0x7f716c066ee0 msgr2=0x7f716c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.924+0000 7f71729f7640 1 -- 172.21.15.1:0/1402245926 shutdown_connections 2024-03-06T07:23:11.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.924+0000 7f71729f7640 1 -- 172.21.15.1:0/1402245926 wait complete. 2024-03-06T07:23:11.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.926+0000 7f71729f7640 1 Processor -- start 2024-03-06T07:23:11.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.926+0000 7f71729f7640 1 -- start start 2024-03-06T07:23:11.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.927+0000 7f71729f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f716c0bf9d0 0x7f716c1df430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.927+0000 7f71729f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f716c152650 0x7f716c1f2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.927+0000 7f71729f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c1e1a90 0x7f716c0c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.927+0000 7f71729f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716c1551b0 con 0x7f716c152650 2024-03-06T07:23:11.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.927+0000 7f71729f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0c6fa0 con 0x7f716c1e1a90 2024-03-06T07:23:11.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.927+0000 7f71729f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716c155490 con 0x7f716c0bf9d0 2024-03-06T07:23:11.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.927+0000 7f716b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f716c152650 0x7f716c1f2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.928+0000 7f716b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f716c152650 0x7f716c1f2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42588/0 (socket says 172.21.15.1:42588) 2024-03-06T07:23:11.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.928+0000 7f716b7fe640 1 -- 172.21.15.1:0/382833601 learned_addr learned my addr 172.21.15.1:0/382833601 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:11.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.928+0000 7f716bfff640 1 --2- 172.21.15.1:0/382833601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f716c0bf9d0 0x7f716c1df430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.928+0000 7f7170f94640 1 --2- 172.21.15.1:0/382833601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c1e1a90 0x7f716c0c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.928+0000 7f716b7fe640 1 -- 172.21.15.1:0/382833601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f716c0bf9d0 msgr2=0x7f716c1df430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.928+0000 7f716b7fe640 1 --2- 172.21.15.1:0/382833601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f716c0bf9d0 0x7f716c1df430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.928+0000 7f716b7fe640 1 -- 172.21.15.1:0/382833601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c1e1a90 msgr2=0x7f716c0c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.928+0000 7f716b7fe640 1 --2- 172.21.15.1:0/382833601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c1e1a90 0x7f716c0c9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.928+0000 7f716b7fe640 1 -- 172.21.15.1:0/382833601 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716c1e0ff0 con 0x7f716c152650 2024-03-06T07:23:11.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.928+0000 7f716bfff640 1 --2- 172.21.15.1:0/382833601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f716c0bf9d0 0x7f716c1df430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:11.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.928+0000 7f716b7fe640 1 --2- 172.21.15.1:0/382833601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f716c152650 0x7f716c1f2370 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f716000c430 tx=0x7f716000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.928+0000 7f7168ff9640 1 -- 172.21.15.1:0/382833601 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7160017070 con 0x7f716c152650 2024-03-06T07:23:11.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.928+0000 7f7168ff9640 1 -- 172.21.15.1:0/382833601 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7160005ce0 con 0x7f716c152650 2024-03-06T07:23:11.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.928+0000 7f71729f7640 1 -- 172.21.15.1:0/382833601 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f716c1de8a0 con 0x7f716c152650 2024-03-06T07:23:11.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.928+0000 7f7168ff9640 1 -- 172.21.15.1:0/382833601 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71600072f0 con 0x7f716c152650 2024-03-06T07:23:11.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.928+0000 7f71729f7640 1 -- 172.21.15.1:0/382833601 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f716c1df970 con 0x7f716c152650 2024-03-06T07:23:11.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.929+0000 7f71729f7640 1 -- 172.21.15.1:0/382833601 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f716c067fd0 con 0x7f716c152650 2024-03-06T07:23:11.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.930+0000 7f7168ff9640 1 -- 172.21.15.1:0/382833601 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f716001a070 con 0x7f716c152650 2024-03-06T07:23:11.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.931+0000 7f7168ff9640 1 --2- 172.21.15.1:0/382833601 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f715806b500 0x7f715806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.931+0000 7f7168ff9640 1 -- 172.21.15.1:0/382833601 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f7160089a20 con 0x7f716c152650 2024-03-06T07:23:11.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.931+0000 7f716bfff640 1 --2- 172.21.15.1:0/382833601 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f715806b500 0x7f715806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.931+0000 7f716bfff640 1 --2- 172.21.15.1:0/382833601 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f715806b500 0x7f715806d9c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f715c036000 tx=0x7f715c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:11.935+0000 7f7168ff9640 1 -- 172.21.15.1:0/382833601 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7160057e00 con 0x7f716c152650 2024-03-06T07:23:12.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.113+0000 7f71729f7640 1 -- 172.21.15.1:0/382833601 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f716c0c3930 con 0x7f715806b500 2024-03-06T07:23:12.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.124+0000 7f7168ff9640 1 -- 172.21.15.1:0/382833601 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490719 (secure 0 0 0) 0x7f716c0c3930 con 0x7f715806b500 2024-03-06T07:23:12.133 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:12.136 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:23:12.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.129+0000 7f71729f7640 1 -- 172.21.15.1:0/382833601 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f715806b500 msgr2=0x7f715806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.129+0000 7f71729f7640 1 --2- 172.21.15.1:0/382833601 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f715806b500 0x7f715806d9c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f715c036000 tx=0x7f715c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:12.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.129+0000 7f71729f7640 1 -- 172.21.15.1:0/382833601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f716c152650 msgr2=0x7f716c1f2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.129+0000 7f71729f7640 1 --2- 172.21.15.1:0/382833601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f716c152650 0x7f716c1f2370 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f716000c430 tx=0x7f716000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.129+0000 7f71729f7640 1 -- 172.21.15.1:0/382833601 shutdown_connections 2024-03-06T07:23:12.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.129+0000 7f71729f7640 1 --2- 172.21.15.1:0/382833601 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f715806b500 0x7f715806d9c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.129+0000 7f71729f7640 1 --2- 172.21.15.1:0/382833601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c1e1a90 0x7f716c0c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.129+0000 7f71729f7640 1 --2- 172.21.15.1:0/382833601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f716c152650 0x7f716c1f2370 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.129+0000 7f71729f7640 1 --2- 172.21.15.1:0/382833601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f716c0bf9d0 0x7f716c1df430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.129+0000 7f71729f7640 1 -- 172.21.15.1:0/382833601 >> 172.21.15.1:0/382833601 conn(0x7f716c066ee0 msgr2=0x7f716c0c5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:12.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.129+0000 7f71729f7640 1 -- 172.21.15.1:0/382833601 shutdown_connections 2024-03-06T07:23:12.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.129+0000 7f71729f7640 1 -- 172.21.15.1:0/382833601 wait complete. 2024-03-06T07:23:12.142 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-03-06T07:23:10.729719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":345,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244684,"kb_used_data":7700,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754730036,"statfs":{"total":773094113280,"available":772843556864,"internally_reserved":0,"allocated":7884800,"data_stored":4848171,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.020920"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.128111+0000","last_change":"2024-03-06T07:23:07.749704+0000","last_active":"2024-03-06T07:23:09.128111+0000","last_peered":"2024-03-06T07:23:09.128111+0000","last_clean":"2024-03-06T07:23:09.128111+0000","last_became_active":"2024-03-06T07:23:07.749457+0000","last_became_peered":"2024-03-06T07:23:07.749457+0000","last_unstale":"2024-03-06T07:23:09.128111+0000","last_undegraded":"2024-03-06T07:23:09.128111+0000","last_fullsized":"2024-03-06T07:23:09.128111+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:45.512631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743244+0000","last_change":"2024-03-06T07:22:57.627301+0000","last_active":"2024-03-06T07:23:08.743244+0000","last_peered":"2024-03-06T07:23:08.743244+0000","last_clean":"2024-03-06T07:23:08.743244+0000","last_became_active":"2024-03-06T07:22:57.626618+0000","last_became_peered":"2024-03-06T07:22:57.626618+0000","last_unstale":"2024-03-06T07:23:08.743244+0000","last_undegraded":"2024-03-06T07:23:08.743244+0000","last_fullsized":"2024-03-06T07:23:08.743244+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:10.595979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126919+0000","last_change":"2024-03-06T07:23:07.743004+0000","last_active":"2024-03-06T07:23:09.126919+0000","last_peered":"2024-03-06T07:23:09.126919+0000","last_clean":"2024-03-06T07:23:09.126919+0000","last_became_active":"2024-03-06T07:23:07.742748+0000","last_became_peered":"2024-03-06T07:23:07.742748+0000","last_unstale":"2024-03-06T07:23:09.126919+0000","last_undegraded":"2024-03-06T07:23:09.126919+0000","last_fullsized":"2024-03-06T07:23:09.126919+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:41.051713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126115+0000","last_change":"2024-03-06T07:22:57.636108+0000","last_active":"2024-03-06T07:23:09.126115+0000","last_peered":"2024-03-06T07:23:09.126115+0000","last_clean":"2024-03-06T07:23:09.126115+0000","last_became_active":"2024-03-06T07:22:57.635820+0000","last_became_peered":"2024-03-06T07:22:57.635820+0000","last_unstale":"2024-03-06T07:23:09.126115+0000","last_undegraded":"2024-03-06T07:23:09.126115+0000","last_fullsized":"2024-03-06T07:23:09.126115+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:42.232078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.128180+0000","last_change":"2024-03-06T07:23:07.738159+0000","last_active":"2024-03-06T07:23:09.128180+0000","last_peered":"2024-03-06T07:23:09.128180+0000","last_clean":"2024-03-06T07:23:09.128180+0000","last_became_active":"2024-03-06T07:23:07.737901+0000","last_became_peered":"2024-03-06T07:23:07.737901+0000","last_unstale":"2024-03-06T07:23:09.128180+0000","last_undegraded":"2024-03-06T07:23:09.128180+0000","last_fullsized":"2024-03-06T07:23:09.128180+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:03.919513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126505+0000","last_change":"2024-03-06T07:22:57.627190+0000","last_active":"2024-03-06T07:23:09.126505+0000","last_peered":"2024-03-06T07:23:09.126505+0000","last_clean":"2024-03-06T07:23:09.126505+0000","last_became_active":"2024-03-06T07:22:57.626893+0000","last_became_peered":"2024-03-06T07:22:57.626893+0000","last_unstale":"2024-03-06T07:23:09.126505+0000","last_undegraded":"2024-03-06T07:23:09.126505+0000","last_fullsized":"2024-03-06T07:23:09.126505+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:04.450678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742506+0000","last_change":"2024-03-06T07:23:07.738946+0000","last_active":"2024-03-06T07:23:08.742506+0000","last_peered":"2024-03-06T07:23:08.742506+0000","last_clean":"2024-03-06T07:23:08.742506+0000","last_became_active":"2024-03-06T07:23:07.738650+0000","last_became_peered":"2024-03-06T07:23:07.738650+0000","last_unstale":"2024-03-06T07:23:08.742506+0000","last_undegraded":"2024-03-06T07:23:08.742506+0000","last_fullsized":"2024-03-06T07:23:08.742506+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:13.483301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741825+0000","last_change":"2024-03-06T07:22:57.632863+0000","last_active":"2024-03-06T07:23:08.741825+0000","last_peered":"2024-03-06T07:23:08.741825+0000","last_clean":"2024-03-06T07:23:08.741825+0000","last_became_active":"2024-03-06T07:22:57.632435+0000","last_became_peered":"2024-03-06T07:22:57.632435+0000","last_unstale":"2024-03-06T07:23:08.741825+0000","last_undegraded":"2024-03-06T07:23:08.741825+0000","last_fullsized":"2024-03-06T07:23:08.741825+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:46.427684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741458+0000","last_change":"2024-03-06T07:23:07.752531+0000","last_active":"2024-03-06T07:23:08.741458+0000","last_peered":"2024-03-06T07:23:08.741458+0000","last_clean":"2024-03-06T07:23:08.741458+0000","last_became_active":"2024-03-06T07:23:07.752246+0000","last_became_peered":"2024-03-06T07:23:07.752246+0000","last_unstale":"2024-03-06T07:23:08.741458+0000","last_undegraded":"2024-03-06T07:23:08.741458+0000","last_fullsized":"2024-03-06T07:23:08.741458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:39.208886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740208+0000","last_change":"2024-03-06T07:22:57.622627+0000","last_active":"2024-03-06T07:23:08.740208+0000","last_peered":"2024-03-06T07:23:08.740208+0000","last_clean":"2024-03-06T07:23:08.740208+0000","last_became_active":"2024-03-06T07:22:57.622319+0000","last_became_peered":"2024-03-06T07:22:57.622319+0000","last_unstale":"2024-03-06T07:23:08.740208+0000","last_undegraded":"2024-03-06T07:23:08.740208+0000","last_fullsized":"2024-03-06T07:23:08.740208+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:01.132943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126931+0000","last_change":"2024-03-06T07:23:07.754142+0000","last_active":"2024-03-06T07:23:09.126931+0000","last_peered":"2024-03-06T07:23:09.126931+0000","last_clean":"2024-03-06T07:23:09.126931+0000","last_became_active":"2024-03-06T07:23:07.753832+0000","last_became_peered":"2024-03-06T07:23:07.753832+0000","last_unstale":"2024-03-06T07:23:09.126931+0000","last_undegraded":"2024-03-06T07:23:09.126931+0000","last_fullsized":"2024-03-06T07:23:09.126931+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:49.818057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127683+0000","last_change":"2024-03-06T07:22:57.633886+0000","last_active":"2024-03-06T07:23:09.127683+0000","last_peered":"2024-03-06T07:23:09.127683+0000","last_clean":"2024-03-06T07:23:09.127683+0000","last_became_active":"2024-03-06T07:22:57.633567+0000","last_became_peered":"2024-03-06T07:22:57.633567+0000","last_unstale":"2024-03-06T07:23:09.127683+0000","last_undegraded":"2024-03-06T07:23:09.127683+0000","last_fullsized":"2024-03-06T07:23:09.127683+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:39.710112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745825+0000","last_change":"2024-03-06T07:23:07.749384+0000","last_active":"2024-03-06T07:23:08.745825+0000","last_peered":"2024-03-06T07:23:08.745825+0000","last_clean":"2024-03-06T07:23:08.745825+0000","last_became_active":"2024-03-06T07:23:07.749122+0000","last_became_peered":"2024-03-06T07:23:07.749122+0000","last_unstale":"2024-03-06T07:23:08.745825+0000","last_undegraded":"2024-03-06T07:23:08.745825+0000","last_fullsized":"2024-03-06T07:23:08.745825+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:33.406528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125674+0000","last_change":"2024-03-06T07:22:57.637662+0000","last_active":"2024-03-06T07:23:09.125674+0000","last_peered":"2024-03-06T07:23:09.125674+0000","last_clean":"2024-03-06T07:23:09.125674+0000","last_became_active":"2024-03-06T07:22:57.637320+0000","last_became_peered":"2024-03-06T07:22:57.637320+0000","last_unstale":"2024-03-06T07:23:09.125674+0000","last_undegraded":"2024-03-06T07:23:09.125674+0000","last_fullsized":"2024-03-06T07:23:09.125674+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:48.772981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126613+0000","last_change":"2024-03-06T07:23:07.746767+0000","last_active":"2024-03-06T07:23:09.126613+0000","last_peered":"2024-03-06T07:23:09.126613+0000","last_clean":"2024-03-06T07:23:09.126613+0000","last_became_active":"2024-03-06T07:23:07.746252+0000","last_became_peered":"2024-03-06T07:23:07.746252+0000","last_unstale":"2024-03-06T07:23:09.126613+0000","last_undegraded":"2024-03-06T07:23:09.126613+0000","last_fullsized":"2024-03-06T07:23:09.126613+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:40.831625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739605+0000","last_change":"2024-03-06T07:22:57.623809+0000","last_active":"2024-03-06T07:23:08.739605+0000","last_peered":"2024-03-06T07:23:08.739605+0000","last_clean":"2024-03-06T07:23:08.739605+0000","last_became_active":"2024-03-06T07:22:57.623535+0000","last_became_peered":"2024-03-06T07:22:57.623535+0000","last_unstale":"2024-03-06T07:23:08.739605+0000","last_undegraded":"2024-03-06T07:23:08.739605+0000","last_fullsized":"2024-03-06T07:23:08.739605+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:08.011885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746672+0000","last_change":"2024-03-06T07:23:07.749717+0000","last_active":"2024-03-06T07:23:08.746672+0000","last_peered":"2024-03-06T07:23:08.746672+0000","last_clean":"2024-03-06T07:23:08.746672+0000","last_became_active":"2024-03-06T07:23:07.745609+0000","last_became_peered":"2024-03-06T07:23:07.745609+0000","last_unstale":"2024-03-06T07:23:08.746672+0000","last_undegraded":"2024-03-06T07:23:08.746672+0000","last_fullsized":"2024-03-06T07:23:08.746672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:52.340022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746604+0000","last_change":"2024-03-06T07:22:57.635037+0000","last_active":"2024-03-06T07:23:08.746604+0000","last_peered":"2024-03-06T07:23:08.746604+0000","last_clean":"2024-03-06T07:23:08.746604+0000","last_became_active":"2024-03-06T07:22:57.634754+0000","last_became_peered":"2024-03-06T07:22:57.634754+0000","last_unstale":"2024-03-06T07:23:08.746604+0000","last_undegraded":"2024-03-06T07:23:08.746604+0000","last_fullsized":"2024-03-06T07:23:08.746604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:55.687038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739829+0000","last_change":"2024-03-06T07:23:07.736724+0000","last_active":"2024-03-06T07:23:08.739829+0000","last_peered":"2024-03-06T07:23:08.739829+0000","last_clean":"2024-03-06T07:23:08.739829+0000","last_became_active":"2024-03-06T07:23:07.736451+0000","last_became_peered":"2024-03-06T07:23:07.736451+0000","last_unstale":"2024-03-06T07:23:08.739829+0000","last_undegraded":"2024-03-06T07:23:08.739829+0000","last_fullsized":"2024-03-06T07:23:08.739829+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:48.354539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742621+0000","last_change":"2024-03-06T07:22:57.632537+0000","last_active":"2024-03-06T07:23:08.742621+0000","last_peered":"2024-03-06T07:23:08.742621+0000","last_clean":"2024-03-06T07:23:08.742621+0000","last_became_active":"2024-03-06T07:22:57.632230+0000","last_became_peered":"2024-03-06T07:22:57.632230+0000","last_unstale":"2024-03-06T07:23:08.742621+0000","last_undegraded":"2024-03-06T07:23:08.742621+0000","last_fullsized":"2024-03-06T07:23:08.742621+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:43.442566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746502+0000","last_change":"2024-03-06T07:23:07.748940+0000","last_active":"2024-03-06T07:23:08.746502+0000","last_peered":"2024-03-06T07:23:08.746502+0000","last_clean":"2024-03-06T07:23:08.746502+0000","last_became_active":"2024-03-06T07:23:07.745255+0000","last_became_peered":"2024-03-06T07:23:07.745255+0000","last_unstale":"2024-03-06T07:23:08.746502+0000","last_undegraded":"2024-03-06T07:23:08.746502+0000","last_fullsized":"2024-03-06T07:23:08.746502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:56.142001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126946+0000","last_change":"2024-03-06T07:22:57.633314+0000","last_active":"2024-03-06T07:23:09.126946+0000","last_peered":"2024-03-06T07:23:09.126946+0000","last_clean":"2024-03-06T07:23:09.126946+0000","last_became_active":"2024-03-06T07:22:57.633041+0000","last_became_peered":"2024-03-06T07:22:57.633041+0000","last_unstale":"2024-03-06T07:23:09.126946+0000","last_undegraded":"2024-03-06T07:23:09.126946+0000","last_fullsized":"2024-03-06T07:23:09.126946+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:46.034056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740004+0000","last_change":"2024-03-06T07:23:07.746640+0000","last_active":"2024-03-06T07:23:08.740004+0000","last_peered":"2024-03-06T07:23:08.740004+0000","last_clean":"2024-03-06T07:23:08.740004+0000","last_became_active":"2024-03-06T07:23:07.746380+0000","last_became_peered":"2024-03-06T07:23:07.746380+0000","last_unstale":"2024-03-06T07:23:08.740004+0000","last_undegraded":"2024-03-06T07:23:08.740004+0000","last_fullsized":"2024-03-06T07:23:08.740004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:14.364329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741911+0000","last_change":"2024-03-06T07:22:57.632052+0000","last_active":"2024-03-06T07:23:08.741911+0000","last_peered":"2024-03-06T07:23:08.741911+0000","last_clean":"2024-03-06T07:23:08.741911+0000","last_became_active":"2024-03-06T07:22:57.630733+0000","last_became_peered":"2024-03-06T07:22:57.630733+0000","last_unstale":"2024-03-06T07:23:08.741911+0000","last_undegraded":"2024-03-06T07:23:08.741911+0000","last_fullsized":"2024-03-06T07:23:08.741911+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:38.629913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126140+0000","last_change":"2024-03-06T07:23:07.739707+0000","last_active":"2024-03-06T07:23:09.126140+0000","last_peered":"2024-03-06T07:23:09.126140+0000","last_clean":"2024-03-06T07:23:09.126140+0000","last_became_active":"2024-03-06T07:23:07.739440+0000","last_became_peered":"2024-03-06T07:23:07.739440+0000","last_unstale":"2024-03-06T07:23:09.126140+0000","last_undegraded":"2024-03-06T07:23:09.126140+0000","last_fullsized":"2024-03-06T07:23:09.126140+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:10.248189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740297+0000","last_change":"2024-03-06T07:22:57.644158+0000","last_active":"2024-03-06T07:23:08.740297+0000","last_peered":"2024-03-06T07:23:08.740297+0000","last_clean":"2024-03-06T07:23:08.740297+0000","last_became_active":"2024-03-06T07:22:57.643252+0000","last_became_peered":"2024-03-06T07:22:57.643252+0000","last_unstale":"2024-03-06T07:23:08.740297+0000","last_undegraded":"2024-03-06T07:23:08.740297+0000","last_fullsized":"2024-03-06T07:23:08.740297+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:38.286642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125769+0000","last_change":"2024-03-06T07:23:07.740246+0000","last_active":"2024-03-06T07:23:09.125769+0000","last_peered":"2024-03-06T07:23:09.125769+0000","last_clean":"2024-03-06T07:23:09.125769+0000","last_became_active":"2024-03-06T07:23:07.739848+0000","last_became_peered":"2024-03-06T07:23:07.739848+0000","last_unstale":"2024-03-06T07:23:09.125769+0000","last_undegraded":"2024-03-06T07:23:09.125769+0000","last_fullsized":"2024-03-06T07:23:09.125769+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:12.770620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127715+0000","last_change":"2024-03-06T07:22:57.633872+0000","last_active":"2024-03-06T07:23:09.127715+0000","last_peered":"2024-03-06T07:23:09.127715+0000","last_clean":"2024-03-06T07:23:09.127715+0000","last_became_active":"2024-03-06T07:22:57.633442+0000","last_became_peered":"2024-03-06T07:22:57.633442+0000","last_unstale":"2024-03-06T07:23:09.127715+0000","last_undegraded":"2024-03-06T07:23:09.127715+0000","last_fullsized":"2024-03-06T07:23:09.127715+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:34.380891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126897+0000","last_change":"2024-03-06T07:23:07.754105+0000","last_active":"2024-03-06T07:23:09.126897+0000","last_peered":"2024-03-06T07:23:09.126897+0000","last_clean":"2024-03-06T07:23:09.126897+0000","last_became_active":"2024-03-06T07:23:07.753800+0000","last_became_peered":"2024-03-06T07:23:07.753800+0000","last_unstale":"2024-03-06T07:23:09.126897+0000","last_undegraded":"2024-03-06T07:23:09.126897+0000","last_fullsized":"2024-03-06T07:23:09.126897+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:46.656259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126155+0000","last_change":"2024-03-06T07:22:57.624521+0000","last_active":"2024-03-06T07:23:09.126155+0000","last_peered":"2024-03-06T07:23:09.126155+0000","last_clean":"2024-03-06T07:23:09.126155+0000","last_became_active":"2024-03-06T07:22:57.624228+0000","last_became_peered":"2024-03-06T07:22:57.624228+0000","last_unstale":"2024-03-06T07:23:09.126155+0000","last_undegraded":"2024-03-06T07:23:09.126155+0000","last_fullsized":"2024-03-06T07:23:09.126155+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:26.279644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745658+0000","last_change":"2024-03-06T07:23:07.750667+0000","last_active":"2024-03-06T07:23:08.745658+0000","last_peered":"2024-03-06T07:23:08.745658+0000","last_clean":"2024-03-06T07:23:08.745658+0000","last_became_active":"2024-03-06T07:23:07.750394+0000","last_became_peered":"2024-03-06T07:23:07.750394+0000","last_unstale":"2024-03-06T07:23:08.745658+0000","last_undegraded":"2024-03-06T07:23:08.745658+0000","last_fullsized":"2024-03-06T07:23:08.745658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:02.932081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741485+0000","last_change":"2024-03-06T07:22:57.634006+0000","last_active":"2024-03-06T07:23:08.741485+0000","last_peered":"2024-03-06T07:23:08.741485+0000","last_clean":"2024-03-06T07:23:08.741485+0000","last_became_active":"2024-03-06T07:22:57.633669+0000","last_became_peered":"2024-03-06T07:22:57.633669+0000","last_unstale":"2024-03-06T07:23:08.741485+0000","last_undegraded":"2024-03-06T07:23:08.741485+0000","last_fullsized":"2024-03-06T07:23:08.741485+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:48.906789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746785+0000","last_change":"2024-03-06T07:23:07.751696+0000","last_active":"2024-03-06T07:23:08.746785+0000","last_peered":"2024-03-06T07:23:08.746785+0000","last_clean":"2024-03-06T07:23:08.746785+0000","last_became_active":"2024-03-06T07:23:07.751126+0000","last_became_peered":"2024-03-06T07:23:07.751126+0000","last_unstale":"2024-03-06T07:23:08.746785+0000","last_undegraded":"2024-03-06T07:23:08.746785+0000","last_fullsized":"2024-03-06T07:23:08.746785+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:43.274210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740453+0000","last_change":"2024-03-06T07:22:57.635578+0000","last_active":"2024-03-06T07:23:08.740453+0000","last_peered":"2024-03-06T07:23:08.740453+0000","last_clean":"2024-03-06T07:23:08.740453+0000","last_became_active":"2024-03-06T07:22:57.635275+0000","last_became_peered":"2024-03-06T07:22:57.635275+0000","last_unstale":"2024-03-06T07:23:08.740453+0000","last_undegraded":"2024-03-06T07:23:08.740453+0000","last_fullsized":"2024-03-06T07:23:08.740453+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:25.589698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127083+0000","last_change":"2024-03-06T07:23:07.740364+0000","last_active":"2024-03-06T07:23:09.127083+0000","last_peered":"2024-03-06T07:23:09.127083+0000","last_clean":"2024-03-06T07:23:09.127083+0000","last_became_active":"2024-03-06T07:23:07.740105+0000","last_became_peered":"2024-03-06T07:23:07.740105+0000","last_unstale":"2024-03-06T07:23:09.127083+0000","last_undegraded":"2024-03-06T07:23:09.127083+0000","last_fullsized":"2024-03-06T07:23:09.127083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:56.142001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126758+0000","last_change":"2024-03-06T07:22:57.628862+0000","last_active":"2024-03-06T07:23:09.126758+0000","last_peered":"2024-03-06T07:23:09.126758+0000","last_clean":"2024-03-06T07:23:09.126758+0000","last_became_active":"2024-03-06T07:22:57.628584+0000","last_became_peered":"2024-03-06T07:22:57.628584+0000","last_unstale":"2024-03-06T07:23:09.126758+0000","last_undegraded":"2024-03-06T07:23:09.126758+0000","last_fullsized":"2024-03-06T07:23:09.126758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:54.432088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126183+0000","last_change":"2024-03-06T07:23:07.738520+0000","last_active":"2024-03-06T07:23:09.126183+0000","last_peered":"2024-03-06T07:23:09.126183+0000","last_clean":"2024-03-06T07:23:09.126183+0000","last_became_active":"2024-03-06T07:23:07.738246+0000","last_became_peered":"2024-03-06T07:23:07.738246+0000","last_unstale":"2024-03-06T07:23:09.126183+0000","last_undegraded":"2024-03-06T07:23:09.126183+0000","last_fullsized":"2024-03-06T07:23:09.126183+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:00.643965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.128159+0000","last_change":"2024-03-06T07:22:57.643101+0000","last_active":"2024-03-06T07:23:09.128159+0000","last_peered":"2024-03-06T07:23:09.128159+0000","last_clean":"2024-03-06T07:23:09.128159+0000","last_became_active":"2024-03-06T07:22:57.642820+0000","last_became_peered":"2024-03-06T07:22:57.642820+0000","last_unstale":"2024-03-06T07:23:09.128159+0000","last_undegraded":"2024-03-06T07:23:09.128159+0000","last_fullsized":"2024-03-06T07:23:09.128159+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:36.548315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.128192+0000","last_change":"2024-03-06T07:23:07.743885+0000","last_active":"2024-03-06T07:23:09.128192+0000","last_peered":"2024-03-06T07:23:09.128192+0000","last_clean":"2024-03-06T07:23:09.128192+0000","last_became_active":"2024-03-06T07:23:07.743576+0000","last_became_peered":"2024-03-06T07:23:07.743576+0000","last_unstale":"2024-03-06T07:23:09.128192+0000","last_undegraded":"2024-03-06T07:23:09.128192+0000","last_fullsized":"2024-03-06T07:23:09.128192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:05.205498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739779+0000","last_change":"2024-03-06T07:22:57.636081+0000","last_active":"2024-03-06T07:23:08.739779+0000","last_peered":"2024-03-06T07:23:08.739779+0000","last_clean":"2024-03-06T07:23:08.739779+0000","last_became_active":"2024-03-06T07:22:57.635798+0000","last_became_peered":"2024-03-06T07:22:57.635798+0000","last_unstale":"2024-03-06T07:23:08.739779+0000","last_undegraded":"2024-03-06T07:23:08.739779+0000","last_fullsized":"2024-03-06T07:23:08.739779+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:30.203134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125773+0000","last_change":"2024-03-06T07:23:07.745095+0000","last_active":"2024-03-06T07:23:09.125773+0000","last_peered":"2024-03-06T07:23:09.125773+0000","last_clean":"2024-03-06T07:23:09.125773+0000","last_became_active":"2024-03-06T07:23:07.744832+0000","last_became_peered":"2024-03-06T07:23:07.744832+0000","last_unstale":"2024-03-06T07:23:09.125773+0000","last_undegraded":"2024-03-06T07:23:09.125773+0000","last_fullsized":"2024-03-06T07:23:09.125773+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:09.290438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746287+0000","last_change":"2024-03-06T07:22:57.626588+0000","last_active":"2024-03-06T07:23:08.746287+0000","last_peered":"2024-03-06T07:23:08.746287+0000","last_clean":"2024-03-06T07:23:08.746287+0000","last_became_active":"2024-03-06T07:22:57.625992+0000","last_became_peered":"2024-03-06T07:22:57.625992+0000","last_unstale":"2024-03-06T07:23:08.746287+0000","last_undegraded":"2024-03-06T07:23:08.746287+0000","last_fullsized":"2024-03-06T07:23:08.746287+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:30.081287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127066+0000","last_change":"2024-03-06T07:23:07.749479+0000","last_active":"2024-03-06T07:23:09.127066+0000","last_peered":"2024-03-06T07:23:09.127066+0000","last_clean":"2024-03-06T07:23:09.127066+0000","last_became_active":"2024-03-06T07:23:07.749047+0000","last_became_peered":"2024-03-06T07:23:07.749047+0000","last_unstale":"2024-03-06T07:23:09.127066+0000","last_undegraded":"2024-03-06T07:23:09.127066+0000","last_fullsized":"2024-03-06T07:23:09.127066+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:01.336551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740234+0000","last_change":"2024-03-06T07:22:57.636470+0000","last_active":"2024-03-06T07:23:08.740234+0000","last_peered":"2024-03-06T07:23:08.740234+0000","last_clean":"2024-03-06T07:23:08.740234+0000","last_became_active":"2024-03-06T07:22:57.636207+0000","last_became_peered":"2024-03-06T07:22:57.636207+0000","last_unstale":"2024-03-06T07:23:08.740234+0000","last_undegraded":"2024-03-06T07:23:08.740234+0000","last_fullsized":"2024-03-06T07:23:08.740234+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:10.025633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746334+0000","last_change":"2024-03-06T07:23:07.753036+0000","last_active":"2024-03-06T07:23:08.746334+0000","last_peered":"2024-03-06T07:23:08.746334+0000","last_clean":"2024-03-06T07:23:08.746334+0000","last_became_active":"2024-03-06T07:23:07.745694+0000","last_became_peered":"2024-03-06T07:23:07.745694+0000","last_unstale":"2024-03-06T07:23:08.746334+0000","last_undegraded":"2024-03-06T07:23:08.746334+0000","last_fullsized":"2024-03-06T07:23:08.746334+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:44.488836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745662+0000","last_change":"2024-03-06T07:22:57.632469+0000","last_active":"2024-03-06T07:23:08.745662+0000","last_peered":"2024-03-06T07:23:08.745662+0000","last_clean":"2024-03-06T07:23:08.745662+0000","last_became_active":"2024-03-06T07:22:57.631771+0000","last_became_peered":"2024-03-06T07:22:57.631771+0000","last_unstale":"2024-03-06T07:23:08.745662+0000","last_undegraded":"2024-03-06T07:23:08.745662+0000","last_fullsized":"2024-03-06T07:23:08.745662+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:31.714843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745851+0000","last_change":"2024-03-06T07:23:07.742518+0000","last_active":"2024-03-06T07:23:08.745851+0000","last_peered":"2024-03-06T07:23:08.745851+0000","last_clean":"2024-03-06T07:23:08.745851+0000","last_became_active":"2024-03-06T07:23:07.742222+0000","last_became_peered":"2024-03-06T07:23:07.742222+0000","last_unstale":"2024-03-06T07:23:08.745851+0000","last_undegraded":"2024-03-06T07:23:08.745851+0000","last_fullsized":"2024-03-06T07:23:08.745851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:46.656259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125817+0000","last_change":"2024-03-06T07:22:57.635044+0000","last_active":"2024-03-06T07:23:09.125817+0000","last_peered":"2024-03-06T07:23:09.125817+0000","last_clean":"2024-03-06T07:23:09.125817+0000","last_became_active":"2024-03-06T07:22:57.634107+0000","last_became_peered":"2024-03-06T07:22:57.634107+0000","last_unstale":"2024-03-06T07:23:09.125817+0000","last_undegraded":"2024-03-06T07:23:09.125817+0000","last_fullsized":"2024-03-06T07:23:09.125817+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:23.298584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740301+0000","last_change":"2024-03-06T07:23:07.741639+0000","last_active":"2024-03-06T07:23:08.740301+0000","last_peered":"2024-03-06T07:23:08.740301+0000","last_clean":"2024-03-06T07:23:08.740301+0000","last_became_active":"2024-03-06T07:23:07.741400+0000","last_became_peered":"2024-03-06T07:23:07.741400+0000","last_unstale":"2024-03-06T07:23:08.740301+0000","last_undegraded":"2024-03-06T07:23:08.740301+0000","last_fullsized":"2024-03-06T07:23:08.740301+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:03.722338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126375+0000","last_change":"2024-03-06T07:22:57.633281+0000","last_active":"2024-03-06T07:23:09.126375+0000","last_peered":"2024-03-06T07:23:09.126375+0000","last_clean":"2024-03-06T07:23:09.126375+0000","last_became_active":"2024-03-06T07:22:57.632984+0000","last_became_peered":"2024-03-06T07:22:57.632984+0000","last_unstale":"2024-03-06T07:23:09.126375+0000","last_undegraded":"2024-03-06T07:23:09.126375+0000","last_fullsized":"2024-03-06T07:23:09.126375+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:39.710112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127627+0000","last_change":"2024-03-06T07:23:07.746234+0000","last_active":"2024-03-06T07:23:09.127627+0000","last_peered":"2024-03-06T07:23:09.127627+0000","last_clean":"2024-03-06T07:23:09.127627+0000","last_became_active":"2024-03-06T07:23:07.745925+0000","last_became_peered":"2024-03-06T07:23:07.745925+0000","last_unstale":"2024-03-06T07:23:09.127627+0000","last_undegraded":"2024-03-06T07:23:09.127627+0000","last_fullsized":"2024-03-06T07:23:09.127627+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:43.274210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742984+0000","last_change":"2024-03-06T07:22:57.631914+0000","last_active":"2024-03-06T07:23:08.742984+0000","last_peered":"2024-03-06T07:23:08.742984+0000","last_clean":"2024-03-06T07:23:08.742984+0000","last_became_active":"2024-03-06T07:22:57.630752+0000","last_became_peered":"2024-03-06T07:22:57.630752+0000","last_unstale":"2024-03-06T07:23:08.742984+0000","last_undegraded":"2024-03-06T07:23:08.742984+0000","last_fullsized":"2024-03-06T07:23:08.742984+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:42.188842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741671+0000","last_change":"2024-03-06T07:23:07.742010+0000","last_active":"2024-03-06T07:23:08.741671+0000","last_peered":"2024-03-06T07:23:08.741671+0000","last_clean":"2024-03-06T07:23:08.741671+0000","last_became_active":"2024-03-06T07:23:07.741755+0000","last_became_peered":"2024-03-06T07:23:07.741755+0000","last_unstale":"2024-03-06T07:23:08.741671+0000","last_undegraded":"2024-03-06T07:23:08.741671+0000","last_fullsized":"2024-03-06T07:23:08.741671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:43.274210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742378+0000","last_change":"2024-03-06T07:22:57.631461+0000","last_active":"2024-03-06T07:23:08.742378+0000","last_peered":"2024-03-06T07:23:08.742378+0000","last_clean":"2024-03-06T07:23:08.742378+0000","last_became_active":"2024-03-06T07:22:57.631177+0000","last_became_peered":"2024-03-06T07:22:57.631177+0000","last_unstale":"2024-03-06T07:23:08.742378+0000","last_undegraded":"2024-03-06T07:23:08.742378+0000","last_fullsized":"2024-03-06T07:23:08.742378+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:59.182493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746127+0000","last_change":"2024-03-06T07:23:07.751228+0000","last_active":"2024-03-06T07:23:08.746127+0000","last_peered":"2024-03-06T07:23:08.746127+0000","last_clean":"2024-03-06T07:23:08.746127+0000","last_became_active":"2024-03-06T07:23:07.750661+0000","last_became_peered":"2024-03-06T07:23:07.750661+0000","last_unstale":"2024-03-06T07:23:08.746127+0000","last_undegraded":"2024-03-06T07:23:08.746127+0000","last_fullsized":"2024-03-06T07:23:08.746127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:49.818057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739887+0000","last_change":"2024-03-06T07:22:57.636696+0000","last_active":"2024-03-06T07:23:08.739887+0000","last_peered":"2024-03-06T07:23:08.739887+0000","last_clean":"2024-03-06T07:23:08.739887+0000","last_became_active":"2024-03-06T07:22:57.636414+0000","last_became_peered":"2024-03-06T07:22:57.636414+0000","last_unstale":"2024-03-06T07:23:08.739887+0000","last_undegraded":"2024-03-06T07:23:08.739887+0000","last_fullsized":"2024-03-06T07:23:08.739887+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:06.519659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741358+0000","last_change":"2024-03-06T07:23:07.736701+0000","last_active":"2024-03-06T07:23:08.741358+0000","last_peered":"2024-03-06T07:23:08.741358+0000","last_clean":"2024-03-06T07:23:08.741358+0000","last_became_active":"2024-03-06T07:23:07.736441+0000","last_became_peered":"2024-03-06T07:23:07.736441+0000","last_unstale":"2024-03-06T07:23:08.741358+0000","last_undegraded":"2024-03-06T07:23:08.741358+0000","last_fullsized":"2024-03-06T07:23:08.741358+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:52.340022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126167+0000","last_change":"2024-03-06T07:22:57.620687+0000","last_active":"2024-03-06T07:23:09.126167+0000","last_peered":"2024-03-06T07:23:09.126167+0000","last_clean":"2024-03-06T07:23:09.126167+0000","last_became_active":"2024-03-06T07:22:57.620416+0000","last_became_peered":"2024-03-06T07:22:57.620416+0000","last_unstale":"2024-03-06T07:23:09.126167+0000","last_undegraded":"2024-03-06T07:23:09.126167+0000","last_fullsized":"2024-03-06T07:23:09.126167+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:30.943768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125626+0000","last_change":"2024-03-06T07:23:07.739482+0000","last_active":"2024-03-06T07:23:09.125626+0000","last_peered":"2024-03-06T07:23:09.125626+0000","last_clean":"2024-03-06T07:23:09.125626+0000","last_became_active":"2024-03-06T07:23:07.738945+0000","last_became_peered":"2024-03-06T07:23:07.738945+0000","last_unstale":"2024-03-06T07:23:09.125626+0000","last_undegraded":"2024-03-06T07:23:09.125626+0000","last_fullsized":"2024-03-06T07:23:09.125626+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:37.230059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741073+0000","last_change":"2024-03-06T07:22:57.622616+0000","last_active":"2024-03-06T07:23:08.741073+0000","last_peered":"2024-03-06T07:23:08.741073+0000","last_clean":"2024-03-06T07:23:08.741073+0000","last_became_active":"2024-03-06T07:22:57.622314+0000","last_became_peered":"2024-03-06T07:22:57.622314+0000","last_unstale":"2024-03-06T07:23:08.741073+0000","last_undegraded":"2024-03-06T07:23:08.741073+0000","last_fullsized":"2024-03-06T07:23:08.741073+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:51.972559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742047+0000","last_change":"2024-03-06T07:23:07.748947+0000","last_active":"2024-03-06T07:23:08.742047+0000","last_peered":"2024-03-06T07:23:08.742047+0000","last_clean":"2024-03-06T07:23:08.742047+0000","last_became_active":"2024-03-06T07:23:07.748384+0000","last_became_peered":"2024-03-06T07:23:07.748384+0000","last_unstale":"2024-03-06T07:23:08.742047+0000","last_undegraded":"2024-03-06T07:23:08.742047+0000","last_fullsized":"2024-03-06T07:23:08.742047+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:43.387518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126393+0000","last_change":"2024-03-06T07:22:57.632401+0000","last_active":"2024-03-06T07:23:09.126393+0000","last_peered":"2024-03-06T07:23:09.126393+0000","last_clean":"2024-03-06T07:23:09.126393+0000","last_became_active":"2024-03-06T07:22:57.632063+0000","last_became_peered":"2024-03-06T07:22:57.632063+0000","last_unstale":"2024-03-06T07:23:09.126393+0000","last_undegraded":"2024-03-06T07:23:09.126393+0000","last_fullsized":"2024-03-06T07:23:09.126393+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:40.323690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745681+0000","last_change":"2024-03-06T07:23:07.748734+0000","last_active":"2024-03-06T07:23:08.745681+0000","last_peered":"2024-03-06T07:23:08.745681+0000","last_clean":"2024-03-06T07:23:08.745681+0000","last_became_active":"2024-03-06T07:23:07.748420+0000","last_became_peered":"2024-03-06T07:23:07.748420+0000","last_unstale":"2024-03-06T07:23:08.745681+0000","last_undegraded":"2024-03-06T07:23:08.745681+0000","last_fullsized":"2024-03-06T07:23:08.745681+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:10.248189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741568+0000","last_change":"2024-03-06T07:22:57.626369+0000","last_active":"2024-03-06T07:23:08.741568+0000","last_peered":"2024-03-06T07:23:08.741568+0000","last_clean":"2024-03-06T07:23:08.741568+0000","last_became_active":"2024-03-06T07:22:57.626083+0000","last_became_peered":"2024-03-06T07:22:57.626083+0000","last_unstale":"2024-03-06T07:23:08.741568+0000","last_undegraded":"2024-03-06T07:23:08.741568+0000","last_fullsized":"2024-03-06T07:23:08.741568+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:14.548525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126302+0000","last_change":"2024-03-06T07:23:07.746560+0000","last_active":"2024-03-06T07:23:09.126302+0000","last_peered":"2024-03-06T07:23:09.126302+0000","last_clean":"2024-03-06T07:23:09.126302+0000","last_became_active":"2024-03-06T07:23:07.745840+0000","last_became_peered":"2024-03-06T07:23:07.745840+0000","last_unstale":"2024-03-06T07:23:09.126302+0000","last_undegraded":"2024-03-06T07:23:09.126302+0000","last_fullsized":"2024-03-06T07:23:09.126302+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:59.539170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127491+0000","last_change":"2024-03-06T07:23:06.721815+0000","last_active":"2024-03-06T07:23:09.127491+0000","last_peered":"2024-03-06T07:23:09.127491+0000","last_clean":"2024-03-06T07:23:09.127491+0000","last_became_active":"2024-03-06T07:23:06.721564+0000","last_became_peered":"2024-03-06T07:23:06.721564+0000","last_unstale":"2024-03-06T07:23:09.127491+0000","last_undegraded":"2024-03-06T07:23:09.127491+0000","last_fullsized":"2024-03-06T07:23:09.127491+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:58.705339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127163+0000","last_change":"2024-03-06T07:22:57.626460+0000","last_active":"2024-03-06T07:23:09.127163+0000","last_peered":"2024-03-06T07:23:09.127163+0000","last_clean":"2024-03-06T07:23:09.127163+0000","last_became_active":"2024-03-06T07:22:57.626207+0000","last_became_peered":"2024-03-06T07:22:57.626207+0000","last_unstale":"2024-03-06T07:23:09.127163+0000","last_undegraded":"2024-03-06T07:23:09.127163+0000","last_fullsized":"2024-03-06T07:23:09.127163+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:27.403036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"73'4","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740486+0000","last_change":"2024-03-06T07:22:56.618360+0000","last_active":"2024-03-06T07:23:08.740486+0000","last_peered":"2024-03-06T07:23:08.740486+0000","last_clean":"2024-03-06T07:23:08.740486+0000","last_became_active":"2024-03-06T07:22:56.618085+0000","last_became_peered":"2024-03-06T07:22:56.618085+0000","last_unstale":"2024-03-06T07:23:08.740486+0000","last_undegraded":"2024-03-06T07:23:08.740486+0000","last_fullsized":"2024-03-06T07:23:08.740486+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:56.031916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127039+0000","last_change":"2024-03-06T07:23:07.742378+0000","last_active":"2024-03-06T07:23:09.127039+0000","last_peered":"2024-03-06T07:23:09.127039+0000","last_clean":"2024-03-06T07:23:09.127039+0000","last_became_active":"2024-03-06T07:23:07.742132+0000","last_became_peered":"2024-03-06T07:23:07.742132+0000","last_unstale":"2024-03-06T07:23:09.127039+0000","last_undegraded":"2024-03-06T07:23:09.127039+0000","last_fullsized":"2024-03-06T07:23:09.127039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:44.488836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741435+0000","last_change":"2024-03-06T07:23:06.726498+0000","last_active":"2024-03-06T07:23:08.741435+0000","last_peered":"2024-03-06T07:23:08.741435+0000","last_clean":"2024-03-06T07:23:08.741435+0000","last_became_active":"2024-03-06T07:23:06.726180+0000","last_became_peered":"2024-03-06T07:23:06.726180+0000","last_unstale":"2024-03-06T07:23:08.741435+0000","last_undegraded":"2024-03-06T07:23:08.741435+0000","last_fullsized":"2024-03-06T07:23:08.741435+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:51.332694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125783+0000","last_change":"2024-03-06T07:22:57.632027+0000","last_active":"2024-03-06T07:23:09.125783+0000","last_peered":"2024-03-06T07:23:09.125783+0000","last_clean":"2024-03-06T07:23:09.125783+0000","last_became_active":"2024-03-06T07:22:57.631736+0000","last_became_peered":"2024-03-06T07:22:57.631736+0000","last_unstale":"2024-03-06T07:23:09.125783+0000","last_undegraded":"2024-03-06T07:23:09.125783+0000","last_fullsized":"2024-03-06T07:23:09.125783+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:52.824136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"73'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743041+0000","last_change":"2024-03-06T07:22:56.623793+0000","last_active":"2024-03-06T07:23:08.743041+0000","last_peered":"2024-03-06T07:23:08.743041+0000","last_clean":"2024-03-06T07:23:08.743041+0000","last_became_active":"2024-03-06T07:22:56.623471+0000","last_became_peered":"2024-03-06T07:22:56.623471+0000","last_unstale":"2024-03-06T07:23:08.743041+0000","last_undegraded":"2024-03-06T07:23:08.743041+0000","last_fullsized":"2024-03-06T07:23:08.743041+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127086+0000","last_change":"2024-03-06T07:23:07.740395+0000","last_active":"2024-03-06T07:23:09.127086+0000","last_peered":"2024-03-06T07:23:09.127086+0000","last_clean":"2024-03-06T07:23:09.127086+0000","last_became_active":"2024-03-06T07:23:07.740122+0000","last_became_peered":"2024-03-06T07:23:07.740122+0000","last_unstale":"2024-03-06T07:23:09.127086+0000","last_undegraded":"2024-03-06T07:23:09.127086+0000","last_fullsized":"2024-03-06T07:23:09.127086+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:21.819423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745906+0000","last_change":"2024-03-06T07:23:06.725121+0000","last_active":"2024-03-06T07:23:08.745906+0000","last_peered":"2024-03-06T07:23:08.745906+0000","last_clean":"2024-03-06T07:23:08.745906+0000","last_became_active":"2024-03-06T07:23:06.723943+0000","last_became_peered":"2024-03-06T07:23:06.723943+0000","last_unstale":"2024-03-06T07:23:08.745906+0000","last_undegraded":"2024-03-06T07:23:08.745906+0000","last_fullsized":"2024-03-06T07:23:08.745906+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:57.364005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745836+0000","last_change":"2024-03-06T07:22:57.635705+0000","last_active":"2024-03-06T07:23:08.745836+0000","last_peered":"2024-03-06T07:23:08.745836+0000","last_clean":"2024-03-06T07:23:08.745836+0000","last_became_active":"2024-03-06T07:22:57.635457+0000","last_became_peered":"2024-03-06T07:22:57.635457+0000","last_unstale":"2024-03-06T07:23:08.745836+0000","last_undegraded":"2024-03-06T07:23:08.745836+0000","last_fullsized":"2024-03-06T07:23:08.745836+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:00.705050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742444+0000","last_change":"2024-03-06T07:22:56.614741+0000","last_active":"2024-03-06T07:23:08.742444+0000","last_peered":"2024-03-06T07:23:08.742444+0000","last_clean":"2024-03-06T07:23:08.742444+0000","last_became_active":"2024-03-06T07:22:56.614497+0000","last_became_peered":"2024-03-06T07:22:56.614497+0000","last_unstale":"2024-03-06T07:23:08.742444+0000","last_undegraded":"2024-03-06T07:23:08.742444+0000","last_fullsized":"2024-03-06T07:23:08.742444+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:42.726714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125998+0000","last_change":"2024-03-06T07:23:07.742049+0000","last_active":"2024-03-06T07:23:09.125998+0000","last_peered":"2024-03-06T07:23:09.125998+0000","last_clean":"2024-03-06T07:23:09.125998+0000","last_became_active":"2024-03-06T07:23:07.740767+0000","last_became_peered":"2024-03-06T07:23:07.740767+0000","last_unstale":"2024-03-06T07:23:09.125998+0000","last_undegraded":"2024-03-06T07:23:09.125998+0000","last_fullsized":"2024-03-06T07:23:09.125998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:55.819255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127839+0000","last_change":"2024-03-06T07:23:06.731842+0000","last_active":"2024-03-06T07:23:09.127839+0000","last_peered":"2024-03-06T07:23:09.127839+0000","last_clean":"2024-03-06T07:23:09.127839+0000","last_became_active":"2024-03-06T07:23:06.731449+0000","last_became_peered":"2024-03-06T07:23:06.731449+0000","last_unstale":"2024-03-06T07:23:09.127839+0000","last_undegraded":"2024-03-06T07:23:09.127839+0000","last_fullsized":"2024-03-06T07:23:09.127839+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:47.730530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746466+0000","last_change":"2024-03-06T07:22:57.625999+0000","last_active":"2024-03-06T07:23:08.746466+0000","last_peered":"2024-03-06T07:23:08.746466+0000","last_clean":"2024-03-06T07:23:08.746466+0000","last_became_active":"2024-03-06T07:22:57.625719+0000","last_became_peered":"2024-03-06T07:22:57.625719+0000","last_unstale":"2024-03-06T07:23:08.746466+0000","last_undegraded":"2024-03-06T07:23:08.746466+0000","last_fullsized":"2024-03-06T07:23:08.746466+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:51.153627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126204+0000","last_change":"2024-03-06T07:22:56.626144+0000","last_active":"2024-03-06T07:23:09.126204+0000","last_peered":"2024-03-06T07:23:09.126204+0000","last_clean":"2024-03-06T07:23:09.126204+0000","last_became_active":"2024-03-06T07:22:56.625739+0000","last_became_peered":"2024-03-06T07:22:56.625739+0000","last_unstale":"2024-03-06T07:23:09.126204+0000","last_undegraded":"2024-03-06T07:23:09.126204+0000","last_fullsized":"2024-03-06T07:23:09.126204+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:52.264835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742948+0000","last_change":"2024-03-06T07:23:07.747849+0000","last_active":"2024-03-06T07:23:08.742948+0000","last_peered":"2024-03-06T07:23:08.742948+0000","last_clean":"2024-03-06T07:23:08.742948+0000","last_became_active":"2024-03-06T07:23:07.747533+0000","last_became_peered":"2024-03-06T07:23:07.747533+0000","last_unstale":"2024-03-06T07:23:08.742948+0000","last_undegraded":"2024-03-06T07:23:08.742948+0000","last_fullsized":"2024-03-06T07:23:08.742948+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:31.516144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742386+0000","last_change":"2024-03-06T07:23:06.727025+0000","last_active":"2024-03-06T07:23:08.742386+0000","last_peered":"2024-03-06T07:23:08.742386+0000","last_clean":"2024-03-06T07:23:08.742386+0000","last_became_active":"2024-03-06T07:23:06.726749+0000","last_became_peered":"2024-03-06T07:23:06.726749+0000","last_unstale":"2024-03-06T07:23:08.742386+0000","last_undegraded":"2024-03-06T07:23:08.742386+0000","last_fullsized":"2024-03-06T07:23:08.742386+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:43.481508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126271+0000","last_change":"2024-03-06T07:22:57.633974+0000","last_active":"2024-03-06T07:23:09.126271+0000","last_peered":"2024-03-06T07:23:09.126271+0000","last_clean":"2024-03-06T07:23:09.126271+0000","last_became_active":"2024-03-06T07:22:57.633687+0000","last_became_peered":"2024-03-06T07:22:57.633687+0000","last_unstale":"2024-03-06T07:23:09.126271+0000","last_undegraded":"2024-03-06T07:23:09.126271+0000","last_fullsized":"2024-03-06T07:23:09.126271+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:49.816828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"73'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746623+0000","last_change":"2024-03-06T07:22:56.612153+0000","last_active":"2024-03-06T07:23:08.746623+0000","last_peered":"2024-03-06T07:23:08.746623+0000","last_clean":"2024-03-06T07:23:08.746623+0000","last_became_active":"2024-03-06T07:22:56.611872+0000","last_became_peered":"2024-03-06T07:22:56.611872+0000","last_unstale":"2024-03-06T07:23:08.746623+0000","last_undegraded":"2024-03-06T07:23:08.746623+0000","last_fullsized":"2024-03-06T07:23:08.746623+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:03.251378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126326+0000","last_change":"2024-03-06T07:23:07.742241+0000","last_active":"2024-03-06T07:23:09.126326+0000","last_peered":"2024-03-06T07:23:09.126326+0000","last_clean":"2024-03-06T07:23:09.126326+0000","last_became_active":"2024-03-06T07:23:07.741966+0000","last_became_peered":"2024-03-06T07:23:07.741966+0000","last_unstale":"2024-03-06T07:23:09.126326+0000","last_undegraded":"2024-03-06T07:23:09.126326+0000","last_fullsized":"2024-03-06T07:23:09.126326+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:50.360597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125991+0000","last_change":"2024-03-06T07:23:06.732613+0000","last_active":"2024-03-06T07:23:09.125991+0000","last_peered":"2024-03-06T07:23:09.125991+0000","last_clean":"2024-03-06T07:23:09.125991+0000","last_became_active":"2024-03-06T07:23:06.732320+0000","last_became_peered":"2024-03-06T07:23:06.732320+0000","last_unstale":"2024-03-06T07:23:09.125991+0000","last_undegraded":"2024-03-06T07:23:09.125991+0000","last_fullsized":"2024-03-06T07:23:09.125991+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:47.871427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127036+0000","last_change":"2024-03-06T07:22:57.632247+0000","last_active":"2024-03-06T07:23:09.127036+0000","last_peered":"2024-03-06T07:23:09.127036+0000","last_clean":"2024-03-06T07:23:09.127036+0000","last_became_active":"2024-03-06T07:22:57.631945+0000","last_became_peered":"2024-03-06T07:22:57.631945+0000","last_unstale":"2024-03-06T07:23:09.127036+0000","last_undegraded":"2024-03-06T07:23:09.127036+0000","last_fullsized":"2024-03-06T07:23:09.127036+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:11.711479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"73'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742656+0000","last_change":"2024-03-06T07:22:56.616078+0000","last_active":"2024-03-06T07:23:08.742656+0000","last_peered":"2024-03-06T07:23:08.742656+0000","last_clean":"2024-03-06T07:23:08.742656+0000","last_became_active":"2024-03-06T07:22:56.615754+0000","last_became_peered":"2024-03-06T07:22:56.615754+0000","last_unstale":"2024-03-06T07:23:08.742656+0000","last_undegraded":"2024-03-06T07:23:08.742656+0000","last_fullsized":"2024-03-06T07:23:08.742656+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:53.427081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125794+0000","last_change":"2024-03-06T07:23:07.754537+0000","last_active":"2024-03-06T07:23:09.125794+0000","last_peered":"2024-03-06T07:23:09.125794+0000","last_clean":"2024-03-06T07:23:09.125794+0000","last_became_active":"2024-03-06T07:23:07.754304+0000","last_became_peered":"2024-03-06T07:23:07.754304+0000","last_unstale":"2024-03-06T07:23:09.125794+0000","last_undegraded":"2024-03-06T07:23:09.125794+0000","last_fullsized":"2024-03-06T07:23:09.125794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:37.651359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"77'4","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126203+0000","last_change":"2024-03-06T07:23:06.727053+0000","last_active":"2024-03-06T07:23:09.126203+0000","last_peered":"2024-03-06T07:23:09.126203+0000","last_clean":"2024-03-06T07:23:09.126203+0000","last_became_active":"2024-03-06T07:23:06.726737+0000","last_became_peered":"2024-03-06T07:23:06.726737+0000","last_unstale":"2024-03-06T07:23:09.126203+0000","last_undegraded":"2024-03-06T07:23:09.126203+0000","last_fullsized":"2024-03-06T07:23:09.126203+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:55.528300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746802+0000","last_change":"2024-03-06T07:22:57.635125+0000","last_active":"2024-03-06T07:23:08.746802+0000","last_peered":"2024-03-06T07:23:08.746802+0000","last_clean":"2024-03-06T07:23:08.746802+0000","last_became_active":"2024-03-06T07:22:57.634829+0000","last_became_peered":"2024-03-06T07:22:57.634829+0000","last_unstale":"2024-03-06T07:23:08.746802+0000","last_undegraded":"2024-03-06T07:23:08.746802+0000","last_fullsized":"2024-03-06T07:23:08.746802+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:48.772981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125889+0000","last_change":"2024-03-06T07:22:56.612376+0000","last_active":"2024-03-06T07:23:09.125889+0000","last_peered":"2024-03-06T07:23:09.125889+0000","last_clean":"2024-03-06T07:23:09.125889+0000","last_became_active":"2024-03-06T07:22:56.612127+0000","last_became_peered":"2024-03-06T07:22:56.612127+0000","last_unstale":"2024-03-06T07:23:09.125889+0000","last_undegraded":"2024-03-06T07:23:09.125889+0000","last_fullsized":"2024-03-06T07:23:09.125889+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:12.032942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126271+0000","last_change":"2024-03-06T07:23:07.738816+0000","last_active":"2024-03-06T07:23:09.126271+0000","last_peered":"2024-03-06T07:23:09.126271+0000","last_clean":"2024-03-06T07:23:09.126271+0000","last_became_active":"2024-03-06T07:23:07.738316+0000","last_became_peered":"2024-03-06T07:23:07.738316+0000","last_unstale":"2024-03-06T07:23:09.126271+0000","last_undegraded":"2024-03-06T07:23:09.126271+0000","last_fullsized":"2024-03-06T07:23:09.126271+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:38.771898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746220+0000","last_change":"2024-03-06T07:23:06.724709+0000","last_active":"2024-03-06T07:23:08.746220+0000","last_peered":"2024-03-06T07:23:08.746220+0000","last_clean":"2024-03-06T07:23:08.746220+0000","last_became_active":"2024-03-06T07:23:06.724433+0000","last_became_peered":"2024-03-06T07:23:06.724433+0000","last_unstale":"2024-03-06T07:23:08.746220+0000","last_undegraded":"2024-03-06T07:23:08.746220+0000","last_fullsized":"2024-03-06T07:23:08.746220+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:09.784214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126325+0000","last_change":"2024-03-06T07:22:57.636772+0000","last_active":"2024-03-06T07:23:09.126325+0000","last_peered":"2024-03-06T07:23:09.126325+0000","last_clean":"2024-03-06T07:23:09.126325+0000","last_became_active":"2024-03-06T07:22:57.636474+0000","last_became_peered":"2024-03-06T07:22:57.636474+0000","last_unstale":"2024-03-06T07:23:09.126325+0000","last_undegraded":"2024-03-06T07:23:09.126325+0000","last_fullsized":"2024-03-06T07:23:09.126325+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:13.011040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739653+0000","last_change":"2024-03-06T07:22:56.619449+0000","last_active":"2024-03-06T07:23:08.739653+0000","last_peered":"2024-03-06T07:23:08.739653+0000","last_clean":"2024-03-06T07:23:08.739653+0000","last_became_active":"2024-03-06T07:22:56.619182+0000","last_became_peered":"2024-03-06T07:22:56.619182+0000","last_unstale":"2024-03-06T07:23:08.739653+0000","last_undegraded":"2024-03-06T07:23:08.739653+0000","last_fullsized":"2024-03-06T07:23:08.739653+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:43.349682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740162+0000","last_change":"2024-03-06T07:23:07.736967+0000","last_active":"2024-03-06T07:23:08.740162+0000","last_peered":"2024-03-06T07:23:08.740162+0000","last_clean":"2024-03-06T07:23:08.740162+0000","last_became_active":"2024-03-06T07:23:07.736723+0000","last_became_peered":"2024-03-06T07:23:07.736723+0000","last_unstale":"2024-03-06T07:23:08.740162+0000","last_undegraded":"2024-03-06T07:23:08.740162+0000","last_fullsized":"2024-03-06T07:23:08.740162+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:54.211266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126295+0000","last_change":"2024-03-06T07:23:06.737914+0000","last_active":"2024-03-06T07:23:09.126295+0000","last_peered":"2024-03-06T07:23:09.126295+0000","last_clean":"2024-03-06T07:23:09.126295+0000","last_became_active":"2024-03-06T07:23:06.737589+0000","last_became_peered":"2024-03-06T07:23:06.737589+0000","last_unstale":"2024-03-06T07:23:09.126295+0000","last_undegraded":"2024-03-06T07:23:09.126295+0000","last_fullsized":"2024-03-06T07:23:09.126295+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:52.543182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126390+0000","last_change":"2024-03-06T07:22:57.631217+0000","last_active":"2024-03-06T07:23:09.126390+0000","last_peered":"2024-03-06T07:23:09.126390+0000","last_clean":"2024-03-06T07:23:09.126390+0000","last_became_active":"2024-03-06T07:22:57.630879+0000","last_became_peered":"2024-03-06T07:22:57.630879+0000","last_unstale":"2024-03-06T07:23:09.126390+0000","last_undegraded":"2024-03-06T07:23:09.126390+0000","last_fullsized":"2024-03-06T07:23:09.126390+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:36.548315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743410+0000","last_change":"2024-03-06T07:22:56.623979+0000","last_active":"2024-03-06T07:23:08.743410+0000","last_peered":"2024-03-06T07:23:08.743410+0000","last_clean":"2024-03-06T07:23:08.743410+0000","last_became_active":"2024-03-06T07:22:56.623692+0000","last_became_peered":"2024-03-06T07:22:56.623692+0000","last_unstale":"2024-03-06T07:23:08.743410+0000","last_undegraded":"2024-03-06T07:23:08.743410+0000","last_fullsized":"2024-03-06T07:23:08.743410+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:12.032942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740161+0000","last_change":"2024-03-06T07:23:07.751324+0000","last_active":"2024-03-06T07:23:08.740161+0000","last_peered":"2024-03-06T07:23:08.740161+0000","last_clean":"2024-03-06T07:23:08.740161+0000","last_became_active":"2024-03-06T07:23:07.751055+0000","last_became_peered":"2024-03-06T07:23:07.751055+0000","last_unstale":"2024-03-06T07:23:08.740161+0000","last_undegraded":"2024-03-06T07:23:08.740161+0000","last_fullsized":"2024-03-06T07:23:08.740161+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:47.983462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"77'8","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127192+0000","last_change":"2024-03-06T07:23:06.732577+0000","last_active":"2024-03-06T07:23:09.127192+0000","last_peered":"2024-03-06T07:23:09.127192+0000","last_clean":"2024-03-06T07:23:09.127192+0000","last_became_active":"2024-03-06T07:23:06.732232+0000","last_became_peered":"2024-03-06T07:23:06.732232+0000","last_unstale":"2024-03-06T07:23:09.127192+0000","last_undegraded":"2024-03-06T07:23:09.127192+0000","last_fullsized":"2024-03-06T07:23:09.127192+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:51.289458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743207+0000","last_change":"2024-03-06T07:22:57.629315+0000","last_active":"2024-03-06T07:23:08.743207+0000","last_peered":"2024-03-06T07:23:08.743207+0000","last_clean":"2024-03-06T07:23:08.743207+0000","last_became_active":"2024-03-06T07:22:57.629010+0000","last_became_peered":"2024-03-06T07:22:57.629010+0000","last_unstale":"2024-03-06T07:23:08.743207+0000","last_undegraded":"2024-03-06T07:23:08.743207+0000","last_fullsized":"2024-03-06T07:23:08.743207+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:54.685052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743060+0000","last_change":"2024-03-06T07:22:56.615557+0000","last_active":"2024-03-06T07:23:08.743060+0000","last_peered":"2024-03-06T07:23:08.743060+0000","last_clean":"2024-03-06T07:23:08.743060+0000","last_became_active":"2024-03-06T07:22:56.615299+0000","last_became_peered":"2024-03-06T07:22:56.615299+0000","last_unstale":"2024-03-06T07:23:08.743060+0000","last_undegraded":"2024-03-06T07:23:08.743060+0000","last_fullsized":"2024-03-06T07:23:08.743060+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:03.445671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127126+0000","last_change":"2024-03-06T07:23:07.755149+0000","last_active":"2024-03-06T07:23:09.127126+0000","last_peered":"2024-03-06T07:23:09.127126+0000","last_clean":"2024-03-06T07:23:09.127126+0000","last_became_active":"2024-03-06T07:23:07.753813+0000","last_became_peered":"2024-03-06T07:23:07.753813+0000","last_unstale":"2024-03-06T07:23:09.127126+0000","last_undegraded":"2024-03-06T07:23:09.127126+0000","last_fullsized":"2024-03-06T07:23:09.127126+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:52.340022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742167+0000","last_change":"2024-03-06T07:23:06.728804+0000","last_active":"2024-03-06T07:23:08.742167+0000","last_peered":"2024-03-06T07:23:08.742167+0000","last_clean":"2024-03-06T07:23:08.742167+0000","last_became_active":"2024-03-06T07:23:06.728373+0000","last_became_peered":"2024-03-06T07:23:06.728373+0000","last_unstale":"2024-03-06T07:23:08.742167+0000","last_undegraded":"2024-03-06T07:23:08.742167+0000","last_fullsized":"2024-03-06T07:23:08.742167+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:55.134673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127805+0000","last_change":"2024-03-06T07:22:57.642696+0000","last_active":"2024-03-06T07:23:09.127805+0000","last_peered":"2024-03-06T07:23:09.127805+0000","last_clean":"2024-03-06T07:23:09.127805+0000","last_became_active":"2024-03-06T07:22:57.642201+0000","last_became_peered":"2024-03-06T07:22:57.642201+0000","last_unstale":"2024-03-06T07:23:09.127805+0000","last_undegraded":"2024-03-06T07:23:09.127805+0000","last_fullsized":"2024-03-06T07:23:09.127805+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:46.034056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"73'3","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740301+0000","last_change":"2024-03-06T07:22:56.620401+0000","last_active":"2024-03-06T07:23:08.740301+0000","last_peered":"2024-03-06T07:23:08.740301+0000","last_clean":"2024-03-06T07:23:08.740301+0000","last_became_active":"2024-03-06T07:22:56.620134+0000","last_became_peered":"2024-03-06T07:22:56.620134+0000","last_unstale":"2024-03-06T07:23:08.740301+0000","last_undegraded":"2024-03-06T07:23:08.740301+0000","last_fullsized":"2024-03-06T07:23:08.740301+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:48:07.935360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745514+0000","last_change":"2024-03-06T07:23:07.744492+0000","last_active":"2024-03-06T07:23:08.745514+0000","last_peered":"2024-03-06T07:23:08.745514+0000","last_clean":"2024-03-06T07:23:08.745514+0000","last_became_active":"2024-03-06T07:23:07.744029+0000","last_became_peered":"2024-03-06T07:23:07.744029+0000","last_unstale":"2024-03-06T07:23:08.745514+0000","last_undegraded":"2024-03-06T07:23:08.745514+0000","last_fullsized":"2024-03-06T07:23:08.745514+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:59.924773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126094+0000","last_change":"2024-03-06T07:23:06.725559+0000","last_active":"2024-03-06T07:23:09.126094+0000","last_peered":"2024-03-06T07:23:09.126094+0000","last_clean":"2024-03-06T07:23:09.126094+0000","last_became_active":"2024-03-06T07:23:06.725301+0000","last_became_peered":"2024-03-06T07:23:06.725301+0000","last_unstale":"2024-03-06T07:23:09.126094+0000","last_undegraded":"2024-03-06T07:23:09.126094+0000","last_fullsized":"2024-03-06T07:23:09.126094+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:38.201558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127145+0000","last_change":"2024-03-06T07:22:57.625109+0000","last_active":"2024-03-06T07:23:09.127145+0000","last_peered":"2024-03-06T07:23:09.127145+0000","last_clean":"2024-03-06T07:23:09.127145+0000","last_became_active":"2024-03-06T07:22:57.624843+0000","last_became_peered":"2024-03-06T07:22:57.624843+0000","last_unstale":"2024-03-06T07:23:09.127145+0000","last_undegraded":"2024-03-06T07:23:09.127145+0000","last_fullsized":"2024-03-06T07:23:09.127145+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:13.011040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"73'6","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126271+0000","last_change":"2024-03-06T07:22:56.623474+0000","last_active":"2024-03-06T07:23:09.126271+0000","last_peered":"2024-03-06T07:23:09.126271+0000","last_clean":"2024-03-06T07:23:09.126271+0000","last_became_active":"2024-03-06T07:22:56.623169+0000","last_became_peered":"2024-03-06T07:22:56.623169+0000","last_unstale":"2024-03-06T07:23:09.126271+0000","last_undegraded":"2024-03-06T07:23:09.126271+0000","last_fullsized":"2024-03-06T07:23:09.126271+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:37.241588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.128328+0000","last_change":"2024-03-06T07:23:07.737764+0000","last_active":"2024-03-06T07:23:09.128328+0000","last_peered":"2024-03-06T07:23:09.128328+0000","last_clean":"2024-03-06T07:23:09.128328+0000","last_became_active":"2024-03-06T07:23:07.737501+0000","last_became_peered":"2024-03-06T07:23:07.737501+0000","last_unstale":"2024-03-06T07:23:09.128328+0000","last_undegraded":"2024-03-06T07:23:09.128328+0000","last_fullsized":"2024-03-06T07:23:09.128328+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:34.624409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"77'3","reported_seq":17,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746877+0000","last_change":"2024-03-06T07:23:06.724961+0000","last_active":"2024-03-06T07:23:08.746877+0000","last_peered":"2024-03-06T07:23:08.746877+0000","last_clean":"2024-03-06T07:23:08.746877+0000","last_became_active":"2024-03-06T07:23:06.724704+0000","last_became_peered":"2024-03-06T07:23:06.724704+0000","last_unstale":"2024-03-06T07:23:08.746877+0000","last_undegraded":"2024-03-06T07:23:08.746877+0000","last_fullsized":"2024-03-06T07:23:08.746877+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:47.871427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742000+0000","last_change":"2024-03-06T07:22:57.635263+0000","last_active":"2024-03-06T07:23:08.742000+0000","last_peered":"2024-03-06T07:23:08.742000+0000","last_clean":"2024-03-06T07:23:08.742000+0000","last_became_active":"2024-03-06T07:22:57.635002+0000","last_became_peered":"2024-03-06T07:22:57.635002+0000","last_unstale":"2024-03-06T07:23:08.742000+0000","last_undegraded":"2024-03-06T07:23:08.742000+0000","last_fullsized":"2024-03-06T07:23:08.742000+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:48.906789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741550+0000","last_change":"2024-03-06T07:22:56.613079+0000","last_active":"2024-03-06T07:23:08.741550+0000","last_peered":"2024-03-06T07:23:08.741550+0000","last_clean":"2024-03-06T07:23:08.741550+0000","last_became_active":"2024-03-06T07:22:56.612788+0000","last_became_peered":"2024-03-06T07:22:56.612788+0000","last_unstale":"2024-03-06T07:23:08.741550+0000","last_undegraded":"2024-03-06T07:23:08.741550+0000","last_fullsized":"2024-03-06T07:23:08.741550+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:10.684793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126651+0000","last_change":"2024-03-06T07:23:07.744337+0000","last_active":"2024-03-06T07:23:09.126651+0000","last_peered":"2024-03-06T07:23:09.126651+0000","last_clean":"2024-03-06T07:23:09.126651+0000","last_became_active":"2024-03-06T07:23:07.744056+0000","last_became_peered":"2024-03-06T07:23:07.744056+0000","last_unstale":"2024-03-06T07:23:09.126651+0000","last_undegraded":"2024-03-06T07:23:09.126651+0000","last_fullsized":"2024-03-06T07:23:09.126651+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:33.406528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"77'6","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126159+0000","last_change":"2024-03-06T07:23:06.730832+0000","last_active":"2024-03-06T07:23:09.126159+0000","last_peered":"2024-03-06T07:23:09.126159+0000","last_clean":"2024-03-06T07:23:09.126159+0000","last_became_active":"2024-03-06T07:23:06.730568+0000","last_became_peered":"2024-03-06T07:23:06.730568+0000","last_unstale":"2024-03-06T07:23:09.126159+0000","last_undegraded":"2024-03-06T07:23:09.126159+0000","last_fullsized":"2024-03-06T07:23:09.126159+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:23:44.475424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125894+0000","last_change":"2024-03-06T07:22:57.623532+0000","last_active":"2024-03-06T07:23:09.125894+0000","last_peered":"2024-03-06T07:23:09.125894+0000","last_clean":"2024-03-06T07:23:09.125894+0000","last_became_active":"2024-03-06T07:22:57.623226+0000","last_became_peered":"2024-03-06T07:22:57.623226+0000","last_unstale":"2024-03-06T07:23:09.125894+0000","last_undegraded":"2024-03-06T07:23:09.125894+0000","last_fullsized":"2024-03-06T07:23:09.125894+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:55.687038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741204+0000","last_change":"2024-03-06T07:22:56.615633+0000","last_active":"2024-03-06T07:23:08.741204+0000","last_peered":"2024-03-06T07:23:08.741204+0000","last_clean":"2024-03-06T07:23:08.741204+0000","last_became_active":"2024-03-06T07:22:56.615358+0000","last_became_peered":"2024-03-06T07:22:56.615358+0000","last_unstale":"2024-03-06T07:23:08.741204+0000","last_undegraded":"2024-03-06T07:23:08.741204+0000","last_fullsized":"2024-03-06T07:23:08.741204+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:03.445671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746075+0000","last_change":"2024-03-06T07:23:07.744878+0000","last_active":"2024-03-06T07:23:08.746075+0000","last_peered":"2024-03-06T07:23:08.746075+0000","last_clean":"2024-03-06T07:23:08.746075+0000","last_became_active":"2024-03-06T07:23:07.744484+0000","last_became_peered":"2024-03-06T07:23:07.744484+0000","last_unstale":"2024-03-06T07:23:08.746075+0000","last_undegraded":"2024-03-06T07:23:08.746075+0000","last_fullsized":"2024-03-06T07:23:08.746075+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:41.051713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743331+0000","last_change":"2024-03-06T07:23:06.727628+0000","last_active":"2024-03-06T07:23:08.743331+0000","last_peered":"2024-03-06T07:23:08.743331+0000","last_clean":"2024-03-06T07:23:08.743331+0000","last_became_active":"2024-03-06T07:23:06.727351+0000","last_became_peered":"2024-03-06T07:23:06.727351+0000","last_unstale":"2024-03-06T07:23:08.743331+0000","last_undegraded":"2024-03-06T07:23:08.743331+0000","last_fullsized":"2024-03-06T07:23:08.743331+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:09.240860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126110+0000","last_change":"2024-03-06T07:22:57.628870+0000","last_active":"2024-03-06T07:23:09.126110+0000","last_peered":"2024-03-06T07:23:09.126110+0000","last_clean":"2024-03-06T07:23:09.126110+0000","last_became_active":"2024-03-06T07:22:57.627870+0000","last_became_peered":"2024-03-06T07:22:57.627870+0000","last_unstale":"2024-03-06T07:23:09.126110+0000","last_undegraded":"2024-03-06T07:23:09.126110+0000","last_fullsized":"2024-03-06T07:23:09.126110+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:40.323690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739568+0000","last_change":"2024-03-06T07:22:56.615747+0000","last_active":"2024-03-06T07:23:08.739568+0000","last_peered":"2024-03-06T07:23:08.739568+0000","last_clean":"2024-03-06T07:23:08.739568+0000","last_became_active":"2024-03-06T07:22:56.615466+0000","last_became_peered":"2024-03-06T07:22:56.615466+0000","last_unstale":"2024-03-06T07:23:08.739568+0000","last_undegraded":"2024-03-06T07:23:08.739568+0000","last_fullsized":"2024-03-06T07:23:08.739568+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:51.504717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742653+0000","last_change":"2024-03-06T07:23:07.747622+0000","last_active":"2024-03-06T07:23:08.742653+0000","last_peered":"2024-03-06T07:23:08.742653+0000","last_clean":"2024-03-06T07:23:08.742653+0000","last_became_active":"2024-03-06T07:23:07.747325+0000","last_became_peered":"2024-03-06T07:23:07.747325+0000","last_unstale":"2024-03-06T07:23:08.742653+0000","last_undegraded":"2024-03-06T07:23:08.742653+0000","last_fullsized":"2024-03-06T07:23:08.742653+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:32.325806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.128435+0000","last_change":"2024-03-06T07:23:06.729387+0000","last_active":"2024-03-06T07:23:09.128435+0000","last_peered":"2024-03-06T07:23:09.128435+0000","last_clean":"2024-03-06T07:23:09.128435+0000","last_became_active":"2024-03-06T07:23:06.729095+0000","last_became_peered":"2024-03-06T07:23:06.729095+0000","last_unstale":"2024-03-06T07:23:09.128435+0000","last_undegraded":"2024-03-06T07:23:09.128435+0000","last_fullsized":"2024-03-06T07:23:09.128435+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:54.647252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126345+0000","last_change":"2024-03-06T07:22:57.634493+0000","last_active":"2024-03-06T07:23:09.126345+0000","last_peered":"2024-03-06T07:23:09.126345+0000","last_clean":"2024-03-06T07:23:09.126345+0000","last_became_active":"2024-03-06T07:22:57.634215+0000","last_became_peered":"2024-03-06T07:22:57.634215+0000","last_unstale":"2024-03-06T07:23:09.126345+0000","last_undegraded":"2024-03-06T07:23:09.126345+0000","last_fullsized":"2024-03-06T07:23:09.126345+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:27.403036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"73'8","reported_seq":30,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126563+0000","last_change":"2024-03-06T07:22:56.623664+0000","last_active":"2024-03-06T07:23:09.126563+0000","last_peered":"2024-03-06T07:23:09.126563+0000","last_clean":"2024-03-06T07:23:09.126563+0000","last_became_active":"2024-03-06T07:22:56.623344+0000","last_became_peered":"2024-03-06T07:22:56.623344+0000","last_unstale":"2024-03-06T07:23:09.126563+0000","last_undegraded":"2024-03-06T07:23:09.126563+0000","last_fullsized":"2024-03-06T07:23:09.126563+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:37.765804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742608+0000","last_change":"2024-03-06T07:23:07.745122+0000","last_active":"2024-03-06T07:23:08.742608+0000","last_peered":"2024-03-06T07:23:08.742608+0000","last_clean":"2024-03-06T07:23:08.742608+0000","last_became_active":"2024-03-06T07:23:07.744847+0000","last_became_peered":"2024-03-06T07:23:07.744847+0000","last_unstale":"2024-03-06T07:23:08.742608+0000","last_undegraded":"2024-03-06T07:23:08.742608+0000","last_fullsized":"2024-03-06T07:23:08.742608+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:56.424088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741222+0000","last_change":"2024-03-06T07:23:06.726615+0000","last_active":"2024-03-06T07:23:08.741222+0000","last_peered":"2024-03-06T07:23:08.741222+0000","last_clean":"2024-03-06T07:23:08.741222+0000","last_became_active":"2024-03-06T07:23:06.726367+0000","last_became_peered":"2024-03-06T07:23:06.726367+0000","last_unstale":"2024-03-06T07:23:08.741222+0000","last_undegraded":"2024-03-06T07:23:08.741222+0000","last_fullsized":"2024-03-06T07:23:08.741222+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:01.924753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127302+0000","last_change":"2024-03-06T07:22:57.644383+0000","last_active":"2024-03-06T07:23:09.127302+0000","last_peered":"2024-03-06T07:23:09.127302+0000","last_clean":"2024-03-06T07:23:09.127302+0000","last_became_active":"2024-03-06T07:22:57.643011+0000","last_became_peered":"2024-03-06T07:22:57.643011+0000","last_unstale":"2024-03-06T07:23:09.127302+0000","last_undegraded":"2024-03-06T07:23:09.127302+0000","last_fullsized":"2024-03-06T07:23:09.127302+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:30.943768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"73'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743502+0000","last_change":"2024-03-06T07:22:56.622514+0000","last_active":"2024-03-06T07:23:08.743502+0000","last_peered":"2024-03-06T07:23:08.743502+0000","last_clean":"2024-03-06T07:23:08.743502+0000","last_became_active":"2024-03-06T07:22:56.622187+0000","last_became_peered":"2024-03-06T07:22:56.622187+0000","last_unstale":"2024-03-06T07:23:08.743502+0000","last_undegraded":"2024-03-06T07:23:08.743502+0000","last_fullsized":"2024-03-06T07:23:08.743502+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:37.765804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126378+0000","last_change":"2024-03-06T07:23:07.738572+0000","last_active":"2024-03-06T07:23:09.126378+0000","last_peered":"2024-03-06T07:23:09.126378+0000","last_clean":"2024-03-06T07:23:09.126378+0000","last_became_active":"2024-03-06T07:23:07.738291+0000","last_became_peered":"2024-03-06T07:23:07.738291+0000","last_unstale":"2024-03-06T07:23:09.126378+0000","last_undegraded":"2024-03-06T07:23:09.126378+0000","last_fullsized":"2024-03-06T07:23:09.126378+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:31.777109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739551+0000","last_change":"2024-03-06T07:23:06.722149+0000","last_active":"2024-03-06T07:23:08.739551+0000","last_peered":"2024-03-06T07:23:08.739551+0000","last_clean":"2024-03-06T07:23:08.739551+0000","last_became_active":"2024-03-06T07:23:06.721883+0000","last_became_peered":"2024-03-06T07:23:06.721883+0000","last_unstale":"2024-03-06T07:23:08.739551+0000","last_undegraded":"2024-03-06T07:23:08.739551+0000","last_fullsized":"2024-03-06T07:23:08.739551+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:19.808546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739567+0000","last_change":"2024-03-06T07:22:57.631467+0000","last_active":"2024-03-06T07:23:08.739567+0000","last_peered":"2024-03-06T07:23:08.739567+0000","last_clean":"2024-03-06T07:23:08.739567+0000","last_became_active":"2024-03-06T07:22:57.631215+0000","last_became_peered":"2024-03-06T07:22:57.631215+0000","last_unstale":"2024-03-06T07:23:08.739567+0000","last_undegraded":"2024-03-06T07:23:08.739567+0000","last_fullsized":"2024-03-06T07:23:08.739567+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:08.873797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746806+0000","last_change":"2024-03-06T07:22:56.619907+0000","last_active":"2024-03-06T07:23:08.746806+0000","last_peered":"2024-03-06T07:23:08.746806+0000","last_clean":"2024-03-06T07:23:08.746806+0000","last_became_active":"2024-03-06T07:22:56.619637+0000","last_became_peered":"2024-03-06T07:22:56.619637+0000","last_unstale":"2024-03-06T07:23:08.746806+0000","last_undegraded":"2024-03-06T07:23:08.746806+0000","last_fullsized":"2024-03-06T07:23:08.746806+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:37.241588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126511+0000","last_change":"2024-03-06T07:23:07.744643+0000","last_active":"2024-03-06T07:23:09.126511+0000","last_peered":"2024-03-06T07:23:09.126511+0000","last_clean":"2024-03-06T07:23:09.126511+0000","last_became_active":"2024-03-06T07:23:07.744365+0000","last_became_peered":"2024-03-06T07:23:07.744365+0000","last_unstale":"2024-03-06T07:23:09.126511+0000","last_undegraded":"2024-03-06T07:23:09.126511+0000","last_fullsized":"2024-03-06T07:23:09.126511+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:48.486294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741499+0000","last_change":"2024-03-06T07:23:06.721067+0000","last_active":"2024-03-06T07:23:08.741499+0000","last_peered":"2024-03-06T07:23:08.741499+0000","last_clean":"2024-03-06T07:23:08.741499+0000","last_became_active":"2024-03-06T07:23:06.720792+0000","last_became_peered":"2024-03-06T07:23:06.720792+0000","last_unstale":"2024-03-06T07:23:08.741499+0000","last_undegraded":"2024-03-06T07:23:08.741499+0000","last_fullsized":"2024-03-06T07:23:08.741499+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:35.380261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740231+0000","last_change":"2024-03-06T07:22:57.625588+0000","last_active":"2024-03-06T07:23:08.740231+0000","last_peered":"2024-03-06T07:23:08.740231+0000","last_clean":"2024-03-06T07:23:08.740231+0000","last_became_active":"2024-03-06T07:22:57.625218+0000","last_became_peered":"2024-03-06T07:22:57.625218+0000","last_unstale":"2024-03-06T07:23:08.740231+0000","last_undegraded":"2024-03-06T07:23:08.740231+0000","last_fullsized":"2024-03-06T07:23:08.740231+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:06.639365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"73'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740402+0000","last_change":"2024-03-06T07:22:56.618771+0000","last_active":"2024-03-06T07:23:08.740402+0000","last_peered":"2024-03-06T07:23:08.740402+0000","last_clean":"2024-03-06T07:23:08.740402+0000","last_became_active":"2024-03-06T07:22:56.618384+0000","last_became_peered":"2024-03-06T07:22:56.618384+0000","last_unstale":"2024-03-06T07:23:08.740402+0000","last_undegraded":"2024-03-06T07:23:08.740402+0000","last_fullsized":"2024-03-06T07:23:08.740402+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:04:26.439416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741142+0000","last_change":"2024-03-06T07:23:07.738983+0000","last_active":"2024-03-06T07:23:08.741142+0000","last_peered":"2024-03-06T07:23:08.741142+0000","last_clean":"2024-03-06T07:23:08.741142+0000","last_became_active":"2024-03-06T07:23:07.738682+0000","last_became_peered":"2024-03-06T07:23:07.738682+0000","last_unstale":"2024-03-06T07:23:08.741142+0000","last_undegraded":"2024-03-06T07:23:08.741142+0000","last_fullsized":"2024-03-06T07:23:08.741142+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:21.819423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740117+0000","last_change":"2024-03-06T07:23:06.730351+0000","last_active":"2024-03-06T07:23:08.740117+0000","last_peered":"2024-03-06T07:23:08.740117+0000","last_clean":"2024-03-06T07:23:08.740117+0000","last_became_active":"2024-03-06T07:23:06.729955+0000","last_became_peered":"2024-03-06T07:23:06.729955+0000","last_unstale":"2024-03-06T07:23:08.740117+0000","last_undegraded":"2024-03-06T07:23:08.740117+0000","last_fullsized":"2024-03-06T07:23:08.740117+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:49.986892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127914+0000","last_change":"2024-03-06T07:22:57.644130+0000","last_active":"2024-03-06T07:23:09.127914+0000","last_peered":"2024-03-06T07:23:09.127914+0000","last_clean":"2024-03-06T07:23:09.127914+0000","last_became_active":"2024-03-06T07:22:57.642544+0000","last_became_peered":"2024-03-06T07:22:57.642544+0000","last_unstale":"2024-03-06T07:23:09.127914+0000","last_undegraded":"2024-03-06T07:23:09.127914+0000","last_fullsized":"2024-03-06T07:23:09.127914+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:23.298584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127312+0000","last_change":"2024-03-06T07:22:56.622412+0000","last_active":"2024-03-06T07:23:09.127312+0000","last_peered":"2024-03-06T07:23:09.127312+0000","last_clean":"2024-03-06T07:23:09.127312+0000","last_became_active":"2024-03-06T07:22:56.622140+0000","last_became_peered":"2024-03-06T07:22:56.622140+0000","last_unstale":"2024-03-06T07:23:09.127312+0000","last_undegraded":"2024-03-06T07:23:09.127312+0000","last_fullsized":"2024-03-06T07:23:09.127312+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:37.728121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746664+0000","last_change":"2024-03-06T07:23:07.745239+0000","last_active":"2024-03-06T07:23:08.746664+0000","last_peered":"2024-03-06T07:23:08.746664+0000","last_clean":"2024-03-06T07:23:08.746664+0000","last_became_active":"2024-03-06T07:23:07.744624+0000","last_became_peered":"2024-03-06T07:23:07.744624+0000","last_unstale":"2024-03-06T07:23:08.746664+0000","last_undegraded":"2024-03-06T07:23:08.746664+0000","last_fullsized":"2024-03-06T07:23:08.746664+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:21.819423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746949+0000","last_change":"2024-03-06T07:23:06.729258+0000","last_active":"2024-03-06T07:23:08.746949+0000","last_peered":"2024-03-06T07:23:08.746949+0000","last_clean":"2024-03-06T07:23:08.746949+0000","last_became_active":"2024-03-06T07:23:06.728956+0000","last_became_peered":"2024-03-06T07:23:06.728956+0000","last_unstale":"2024-03-06T07:23:08.746949+0000","last_undegraded":"2024-03-06T07:23:08.746949+0000","last_fullsized":"2024-03-06T07:23:08.746949+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:06.454524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126728+0000","last_change":"2024-03-06T07:22:57.639832+0000","last_active":"2024-03-06T07:23:09.126728+0000","last_peered":"2024-03-06T07:23:09.126728+0000","last_clean":"2024-03-06T07:23:09.126728+0000","last_became_active":"2024-03-06T07:22:57.639055+0000","last_became_peered":"2024-03-06T07:22:57.639055+0000","last_unstale":"2024-03-06T07:23:09.126728+0000","last_undegraded":"2024-03-06T07:23:09.126728+0000","last_fullsized":"2024-03-06T07:23:09.126728+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:00.140244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.128024+0000","last_change":"2024-03-06T07:22:56.619240+0000","last_active":"2024-03-06T07:23:09.128024+0000","last_peered":"2024-03-06T07:23:09.128024+0000","last_clean":"2024-03-06T07:23:09.128024+0000","last_became_active":"2024-03-06T07:22:56.619024+0000","last_became_peered":"2024-03-06T07:22:56.619024+0000","last_unstale":"2024-03-06T07:23:09.128024+0000","last_undegraded":"2024-03-06T07:23:09.128024+0000","last_fullsized":"2024-03-06T07:23:09.128024+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:33.101169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125940+0000","last_change":"2024-03-06T07:23:07.745115+0000","last_active":"2024-03-06T07:23:09.125940+0000","last_peered":"2024-03-06T07:23:09.125940+0000","last_clean":"2024-03-06T07:23:09.125940+0000","last_became_active":"2024-03-06T07:23:07.744857+0000","last_became_peered":"2024-03-06T07:23:07.744857+0000","last_unstale":"2024-03-06T07:23:09.125940+0000","last_undegraded":"2024-03-06T07:23:09.125940+0000","last_fullsized":"2024-03-06T07:23:09.125940+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:02.205164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126305+0000","last_change":"2024-03-06T07:23:06.732714+0000","last_active":"2024-03-06T07:23:09.126305+0000","last_peered":"2024-03-06T07:23:09.126305+0000","last_clean":"2024-03-06T07:23:09.126305+0000","last_became_active":"2024-03-06T07:23:06.732402+0000","last_became_peered":"2024-03-06T07:23:06.732402+0000","last_unstale":"2024-03-06T07:23:09.126305+0000","last_undegraded":"2024-03-06T07:23:09.126305+0000","last_fullsized":"2024-03-06T07:23:09.126305+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:47.730530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126292+0000","last_change":"2024-03-06T07:22:57.626085+0000","last_active":"2024-03-06T07:23:09.126292+0000","last_peered":"2024-03-06T07:23:09.126292+0000","last_clean":"2024-03-06T07:23:09.126292+0000","last_became_active":"2024-03-06T07:22:57.625810+0000","last_became_peered":"2024-03-06T07:22:57.625810+0000","last_unstale":"2024-03-06T07:23:09.126292+0000","last_undegraded":"2024-03-06T07:23:09.126292+0000","last_fullsized":"2024-03-06T07:23:09.126292+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:33.166265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127788+0000","last_change":"2024-03-06T07:22:56.623577+0000","last_active":"2024-03-06T07:23:09.127788+0000","last_peered":"2024-03-06T07:23:09.127788+0000","last_clean":"2024-03-06T07:23:09.127788+0000","last_became_active":"2024-03-06T07:22:56.623267+0000","last_became_peered":"2024-03-06T07:22:56.623267+0000","last_unstale":"2024-03-06T07:23:09.127788+0000","last_undegraded":"2024-03-06T07:23:09.127788+0000","last_fullsized":"2024-03-06T07:23:09.127788+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:09.590972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125972+0000","last_change":"2024-03-06T07:23:07.740441+0000","last_active":"2024-03-06T07:23:09.125972+0000","last_peered":"2024-03-06T07:23:09.125972+0000","last_clean":"2024-03-06T07:23:09.125972+0000","last_became_active":"2024-03-06T07:23:07.740155+0000","last_became_peered":"2024-03-06T07:23:07.740155+0000","last_unstale":"2024-03-06T07:23:09.125972+0000","last_undegraded":"2024-03-06T07:23:09.125972+0000","last_fullsized":"2024-03-06T07:23:09.125972+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:44.214120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742971+0000","last_change":"2024-03-06T07:23:06.727098+0000","last_active":"2024-03-06T07:23:08.742971+0000","last_peered":"2024-03-06T07:23:08.742971+0000","last_clean":"2024-03-06T07:23:08.742971+0000","last_became_active":"2024-03-06T07:23:06.726830+0000","last_became_peered":"2024-03-06T07:23:06.726830+0000","last_unstale":"2024-03-06T07:23:08.742971+0000","last_undegraded":"2024-03-06T07:23:08.742971+0000","last_fullsized":"2024-03-06T07:23:08.742971+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:20.812095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126375+0000","last_change":"2024-03-06T07:22:57.631438+0000","last_active":"2024-03-06T07:23:09.126375+0000","last_peered":"2024-03-06T07:23:09.126375+0000","last_clean":"2024-03-06T07:23:09.126375+0000","last_became_active":"2024-03-06T07:22:57.631174+0000","last_became_peered":"2024-03-06T07:22:57.631174+0000","last_unstale":"2024-03-06T07:23:09.126375+0000","last_undegraded":"2024-03-06T07:23:09.126375+0000","last_fullsized":"2024-03-06T07:23:09.126375+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:54.432088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127395+0000","last_change":"2024-03-06T07:22:56.621728+0000","last_active":"2024-03-06T07:23:09.127395+0000","last_peered":"2024-03-06T07:23:09.127395+0000","last_clean":"2024-03-06T07:23:09.127395+0000","last_became_active":"2024-03-06T07:22:56.621425+0000","last_became_peered":"2024-03-06T07:22:56.621425+0000","last_unstale":"2024-03-06T07:23:09.127395+0000","last_undegraded":"2024-03-06T07:23:09.127395+0000","last_fullsized":"2024-03-06T07:23:09.127395+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:50.967553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125742+0000","last_change":"2024-03-06T07:23:07.745096+0000","last_active":"2024-03-06T07:23:09.125742+0000","last_peered":"2024-03-06T07:23:09.125742+0000","last_clean":"2024-03-06T07:23:09.125742+0000","last_became_active":"2024-03-06T07:23:07.744872+0000","last_became_peered":"2024-03-06T07:23:07.744872+0000","last_unstale":"2024-03-06T07:23:09.125742+0000","last_undegraded":"2024-03-06T07:23:09.125742+0000","last_fullsized":"2024-03-06T07:23:09.125742+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:20.703923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125560+0000","last_change":"2024-03-06T07:23:06.728087+0000","last_active":"2024-03-06T07:23:09.125560+0000","last_peered":"2024-03-06T07:23:09.125560+0000","last_clean":"2024-03-06T07:23:09.125560+0000","last_became_active":"2024-03-06T07:23:06.727827+0000","last_became_peered":"2024-03-06T07:23:06.727827+0000","last_unstale":"2024-03-06T07:23:09.125560+0000","last_undegraded":"2024-03-06T07:23:09.125560+0000","last_fullsized":"2024-03-06T07:23:09.125560+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:03.785668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745458+0000","last_change":"2024-03-06T07:22:57.629119+0000","last_active":"2024-03-06T07:23:08.745458+0000","last_peered":"2024-03-06T07:23:08.745458+0000","last_clean":"2024-03-06T07:23:08.745458+0000","last_became_active":"2024-03-06T07:22:57.628813+0000","last_became_peered":"2024-03-06T07:22:57.628813+0000","last_unstale":"2024-03-06T07:23:08.745458+0000","last_undegraded":"2024-03-06T07:23:08.745458+0000","last_fullsized":"2024-03-06T07:23:08.745458+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:38.733127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743517+0000","last_change":"2024-03-06T07:22:56.619641+0000","last_active":"2024-03-06T07:23:08.743517+0000","last_peered":"2024-03-06T07:23:08.743517+0000","last_clean":"2024-03-06T07:23:08.743517+0000","last_became_active":"2024-03-06T07:22:56.619369+0000","last_became_peered":"2024-03-06T07:22:56.619369+0000","last_unstale":"2024-03-06T07:23:08.743517+0000","last_undegraded":"2024-03-06T07:23:08.743517+0000","last_fullsized":"2024-03-06T07:23:08.743517+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126001+0000","last_change":"2024-03-06T07:23:07.738105+0000","last_active":"2024-03-06T07:23:09.126001+0000","last_peered":"2024-03-06T07:23:09.126001+0000","last_clean":"2024-03-06T07:23:09.126001+0000","last_became_active":"2024-03-06T07:23:07.737822+0000","last_became_peered":"2024-03-06T07:23:07.737822+0000","last_unstale":"2024-03-06T07:23:09.126001+0000","last_undegraded":"2024-03-06T07:23:09.126001+0000","last_fullsized":"2024-03-06T07:23:09.126001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:52.763326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741161+0000","last_change":"2024-03-06T07:23:06.732290+0000","last_active":"2024-03-06T07:23:08.741161+0000","last_peered":"2024-03-06T07:23:08.741161+0000","last_clean":"2024-03-06T07:23:08.741161+0000","last_became_active":"2024-03-06T07:23:06.731993+0000","last_became_peered":"2024-03-06T07:23:06.731993+0000","last_unstale":"2024-03-06T07:23:08.741161+0000","last_undegraded":"2024-03-06T07:23:08.741161+0000","last_fullsized":"2024-03-06T07:23:08.741161+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:12.475972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125800+0000","last_change":"2024-03-06T07:22:57.634967+0000","last_active":"2024-03-06T07:23:09.125800+0000","last_peered":"2024-03-06T07:23:09.125800+0000","last_clean":"2024-03-06T07:23:09.125800+0000","last_became_active":"2024-03-06T07:22:57.634367+0000","last_became_peered":"2024-03-06T07:22:57.634367+0000","last_unstale":"2024-03-06T07:23:09.125800+0000","last_undegraded":"2024-03-06T07:23:09.125800+0000","last_fullsized":"2024-03-06T07:23:09.125800+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:30.081287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"74'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126399+0000","last_change":"2024-03-06T07:22:56.609890+0000","last_active":"2024-03-06T07:23:09.126399+0000","last_peered":"2024-03-06T07:23:09.126399+0000","last_clean":"2024-03-06T07:23:09.126399+0000","last_became_active":"2024-03-06T07:22:56.609470+0000","last_became_peered":"2024-03-06T07:22:56.609470+0000","last_unstale":"2024-03-06T07:23:09.126399+0000","last_undegraded":"2024-03-06T07:23:09.126399+0000","last_fullsized":"2024-03-06T07:23:09.126399+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:56.348901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.739545+0000","last_change":"2024-03-06T07:23:07.732754+0000","last_active":"2024-03-06T07:23:08.739545+0000","last_peered":"2024-03-06T07:23:08.739545+0000","last_clean":"2024-03-06T07:23:08.739545+0000","last_became_active":"2024-03-06T07:23:07.732499+0000","last_became_peered":"2024-03-06T07:23:07.732499+0000","last_unstale":"2024-03-06T07:23:08.739545+0000","last_undegraded":"2024-03-06T07:23:08.739545+0000","last_fullsized":"2024-03-06T07:23:08.739545+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:03.377787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126054+0000","last_change":"2024-03-06T07:23:06.727720+0000","last_active":"2024-03-06T07:23:09.126054+0000","last_peered":"2024-03-06T07:23:09.126054+0000","last_clean":"2024-03-06T07:23:09.126054+0000","last_became_active":"2024-03-06T07:23:06.726098+0000","last_became_peered":"2024-03-06T07:23:06.726098+0000","last_unstale":"2024-03-06T07:23:09.126054+0000","last_undegraded":"2024-03-06T07:23:09.126054+0000","last_fullsized":"2024-03-06T07:23:09.126054+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:42.380189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.745908+0000","last_change":"2024-03-06T07:22:57.634820+0000","last_active":"2024-03-06T07:23:08.745908+0000","last_peered":"2024-03-06T07:23:08.745908+0000","last_clean":"2024-03-06T07:23:08.745908+0000","last_became_active":"2024-03-06T07:22:57.634538+0000","last_became_peered":"2024-03-06T07:22:57.634538+0000","last_unstale":"2024-03-06T07:23:08.745908+0000","last_undegraded":"2024-03-06T07:23:08.745908+0000","last_fullsized":"2024-03-06T07:23:08.745908+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:30.460111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742805+0000","last_change":"2024-03-06T07:22:56.616457+0000","last_active":"2024-03-06T07:23:08.742805+0000","last_peered":"2024-03-06T07:23:08.742805+0000","last_clean":"2024-03-06T07:23:08.742805+0000","last_became_active":"2024-03-06T07:22:56.616201+0000","last_became_peered":"2024-03-06T07:22:56.616201+0000","last_unstale":"2024-03-06T07:23:08.742805+0000","last_undegraded":"2024-03-06T07:23:08.742805+0000","last_fullsized":"2024-03-06T07:23:08.742805+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:59.678591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127735+0000","last_change":"2024-03-06T07:23:07.745128+0000","last_active":"2024-03-06T07:23:09.127735+0000","last_peered":"2024-03-06T07:23:09.127735+0000","last_clean":"2024-03-06T07:23:09.127735+0000","last_became_active":"2024-03-06T07:23:07.744649+0000","last_became_peered":"2024-03-06T07:23:07.744649+0000","last_unstale":"2024-03-06T07:23:09.127735+0000","last_undegraded":"2024-03-06T07:23:09.127735+0000","last_fullsized":"2024-03-06T07:23:09.127735+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:27.698854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742463+0000","last_change":"2024-03-06T07:23:06.724656+0000","last_active":"2024-03-06T07:23:08.742463+0000","last_peered":"2024-03-06T07:23:08.742463+0000","last_clean":"2024-03-06T07:23:08.742463+0000","last_became_active":"2024-03-06T07:23:06.724337+0000","last_became_peered":"2024-03-06T07:23:06.724337+0000","last_unstale":"2024-03-06T07:23:08.742463+0000","last_undegraded":"2024-03-06T07:23:08.742463+0000","last_fullsized":"2024-03-06T07:23:08.742463+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:58.917445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126425+0000","last_change":"2024-03-06T07:22:57.630052+0000","last_active":"2024-03-06T07:23:09.126425+0000","last_peered":"2024-03-06T07:23:09.126425+0000","last_clean":"2024-03-06T07:23:09.126425+0000","last_became_active":"2024-03-06T07:22:57.629722+0000","last_became_peered":"2024-03-06T07:22:57.629722+0000","last_unstale":"2024-03-06T07:23:09.126425+0000","last_undegraded":"2024-03-06T07:23:09.126425+0000","last_fullsized":"2024-03-06T07:23:09.126425+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:34.380891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741889+0000","last_change":"2024-03-06T07:22:56.624178+0000","last_active":"2024-03-06T07:23:08.741889+0000","last_peered":"2024-03-06T07:23:08.741889+0000","last_clean":"2024-03-06T07:23:08.741889+0000","last_became_active":"2024-03-06T07:22:56.623706+0000","last_became_peered":"2024-03-06T07:22:56.623706+0000","last_unstale":"2024-03-06T07:23:08.741889+0000","last_undegraded":"2024-03-06T07:23:08.741889+0000","last_fullsized":"2024-03-06T07:23:08.741889+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:47.258382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125890+0000","last_change":"2024-03-06T07:23:07.737927+0000","last_active":"2024-03-06T07:23:09.125890+0000","last_peered":"2024-03-06T07:23:09.125890+0000","last_clean":"2024-03-06T07:23:09.125890+0000","last_became_active":"2024-03-06T07:23:07.737675+0000","last_became_peered":"2024-03-06T07:23:07.737675+0000","last_unstale":"2024-03-06T07:23:09.125890+0000","last_undegraded":"2024-03-06T07:23:09.125890+0000","last_fullsized":"2024-03-06T07:23:09.125890+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:15.011419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.746966+0000","last_change":"2024-03-06T07:23:06.724789+0000","last_active":"2024-03-06T07:23:08.746966+0000","last_peered":"2024-03-06T07:23:08.746966+0000","last_clean":"2024-03-06T07:23:08.746966+0000","last_became_active":"2024-03-06T07:23:06.724461+0000","last_became_peered":"2024-03-06T07:23:06.724461+0000","last_unstale":"2024-03-06T07:23:08.746966+0000","last_undegraded":"2024-03-06T07:23:08.746966+0000","last_fullsized":"2024-03-06T07:23:08.746966+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:03.785668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126100+0000","last_change":"2024-03-06T07:22:57.633736+0000","last_active":"2024-03-06T07:23:09.126100+0000","last_peered":"2024-03-06T07:23:09.126100+0000","last_clean":"2024-03-06T07:23:09.126100+0000","last_became_active":"2024-03-06T07:22:57.633502+0000","last_became_peered":"2024-03-06T07:22:57.633502+0000","last_unstale":"2024-03-06T07:23:09.126100+0000","last_undegraded":"2024-03-06T07:23:09.126100+0000","last_fullsized":"2024-03-06T07:23:09.126100+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:00.705050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125760+0000","last_change":"2024-03-06T07:22:56.626141+0000","last_active":"2024-03-06T07:23:09.125760+0000","last_peered":"2024-03-06T07:23:09.125760+0000","last_clean":"2024-03-06T07:23:09.125760+0000","last_became_active":"2024-03-06T07:22:56.625760+0000","last_became_peered":"2024-03-06T07:22:56.625760+0000","last_unstale":"2024-03-06T07:23:09.125760+0000","last_undegraded":"2024-03-06T07:23:09.125760+0000","last_fullsized":"2024-03-06T07:23:09.125760+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:03.251378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743336+0000","last_change":"2024-03-06T07:23:07.754097+0000","last_active":"2024-03-06T07:23:08.743336+0000","last_peered":"2024-03-06T07:23:08.743336+0000","last_clean":"2024-03-06T07:23:08.743336+0000","last_became_active":"2024-03-06T07:23:07.753821+0000","last_became_peered":"2024-03-06T07:23:07.753821+0000","last_unstale":"2024-03-06T07:23:08.743336+0000","last_undegraded":"2024-03-06T07:23:08.743336+0000","last_fullsized":"2024-03-06T07:23:08.743336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:59.712667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126113+0000","last_change":"2024-03-06T07:23:06.728083+0000","last_active":"2024-03-06T07:23:09.126113+0000","last_peered":"2024-03-06T07:23:09.126113+0000","last_clean":"2024-03-06T07:23:09.126113+0000","last_became_active":"2024-03-06T07:23:06.727874+0000","last_became_peered":"2024-03-06T07:23:06.727874+0000","last_unstale":"2024-03-06T07:23:09.126113+0000","last_undegraded":"2024-03-06T07:23:09.126113+0000","last_fullsized":"2024-03-06T07:23:09.126113+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:06.454524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126000+0000","last_change":"2024-03-06T07:22:57.628631+0000","last_active":"2024-03-06T07:23:09.126000+0000","last_peered":"2024-03-06T07:23:09.126000+0000","last_clean":"2024-03-06T07:23:09.126000+0000","last_became_active":"2024-03-06T07:22:57.627708+0000","last_became_peered":"2024-03-06T07:22:57.627708+0000","last_unstale":"2024-03-06T07:23:09.126000+0000","last_undegraded":"2024-03-06T07:23:09.126000+0000","last_fullsized":"2024-03-06T07:23:09.126000+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:30.460111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.740318+0000","last_change":"2024-03-06T07:22:56.615838+0000","last_active":"2024-03-06T07:23:08.740318+0000","last_peered":"2024-03-06T07:23:08.740318+0000","last_clean":"2024-03-06T07:23:08.740318+0000","last_became_active":"2024-03-06T07:22:56.615546+0000","last_became_peered":"2024-03-06T07:22:56.615546+0000","last_unstale":"2024-03-06T07:23:08.740318+0000","last_undegraded":"2024-03-06T07:23:08.740318+0000","last_fullsized":"2024-03-06T07:23:08.740318+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:52.858792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126305+0000","last_change":"2024-03-06T07:23:07.755901+0000","last_active":"2024-03-06T07:23:09.126305+0000","last_peered":"2024-03-06T07:23:09.126305+0000","last_clean":"2024-03-06T07:23:09.126305+0000","last_became_active":"2024-03-06T07:23:07.753264+0000","last_became_peered":"2024-03-06T07:23:07.753264+0000","last_unstale":"2024-03-06T07:23:09.126305+0000","last_undegraded":"2024-03-06T07:23:09.126305+0000","last_fullsized":"2024-03-06T07:23:09.126305+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:03.844554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743133+0000","last_change":"2024-03-06T07:23:06.730657+0000","last_active":"2024-03-06T07:23:08.743133+0000","last_peered":"2024-03-06T07:23:08.743133+0000","last_clean":"2024-03-06T07:23:08.743133+0000","last_became_active":"2024-03-06T07:23:06.730347+0000","last_became_peered":"2024-03-06T07:23:06.730347+0000","last_unstale":"2024-03-06T07:23:08.743133+0000","last_undegraded":"2024-03-06T07:23:08.743133+0000","last_fullsized":"2024-03-06T07:23:08.743133+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:42.266882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126840+0000","last_change":"2024-03-06T07:22:57.625874+0000","last_active":"2024-03-06T07:23:09.126840+0000","last_peered":"2024-03-06T07:23:09.126840+0000","last_clean":"2024-03-06T07:23:09.126840+0000","last_became_active":"2024-03-06T07:22:57.625574+0000","last_became_peered":"2024-03-06T07:22:57.625574+0000","last_unstale":"2024-03-06T07:23:09.126840+0000","last_undegraded":"2024-03-06T07:23:09.126840+0000","last_fullsized":"2024-03-06T07:23:09.126840+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:28.600878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.125942+0000","last_change":"2024-03-06T07:22:56.620322+0000","last_active":"2024-03-06T07:23:09.125942+0000","last_peered":"2024-03-06T07:23:09.125942+0000","last_clean":"2024-03-06T07:23:09.125942+0000","last_became_active":"2024-03-06T07:22:56.620052+0000","last_became_peered":"2024-03-06T07:22:56.620052+0000","last_unstale":"2024-03-06T07:23:09.125942+0000","last_undegraded":"2024-03-06T07:23:09.125942+0000","last_fullsized":"2024-03-06T07:23:09.125942+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:36.870510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126672+0000","last_change":"2024-03-06T07:23:07.743019+0000","last_active":"2024-03-06T07:23:09.126672+0000","last_peered":"2024-03-06T07:23:09.126672+0000","last_clean":"2024-03-06T07:23:09.126672+0000","last_became_active":"2024-03-06T07:23:07.742795+0000","last_became_peered":"2024-03-06T07:23:07.742795+0000","last_unstale":"2024-03-06T07:23:09.126672+0000","last_undegraded":"2024-03-06T07:23:09.126672+0000","last_fullsized":"2024-03-06T07:23:09.126672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:22.867840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"78'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.748183+0000","last_change":"2024-03-06T07:23:06.729756+0000","last_active":"2024-03-06T07:23:08.748183+0000","last_peered":"2024-03-06T07:23:08.748183+0000","last_clean":"2024-03-06T07:23:08.748183+0000","last_became_active":"2024-03-06T07:23:06.729524+0000","last_became_peered":"2024-03-06T07:23:06.729524+0000","last_unstale":"2024-03-06T07:23:08.748183+0000","last_undegraded":"2024-03-06T07:23:08.748183+0000","last_fullsized":"2024-03-06T07:23:08.748183+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:22.138565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.126758+0000","last_change":"2024-03-06T07:22:57.624542+0000","last_active":"2024-03-06T07:23:09.126758+0000","last_peered":"2024-03-06T07:23:09.126758+0000","last_clean":"2024-03-06T07:23:09.126758+0000","last_became_active":"2024-03-06T07:22:57.624258+0000","last_became_peered":"2024-03-06T07:22:57.624258+0000","last_unstale":"2024-03-06T07:23:09.126758+0000","last_undegraded":"2024-03-06T07:23:09.126758+0000","last_fullsized":"2024-03-06T07:23:09.126758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:11.689800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127249+0000","last_change":"2024-03-06T07:22:56.621595+0000","last_active":"2024-03-06T07:23:09.127249+0000","last_peered":"2024-03-06T07:23:09.127249+0000","last_clean":"2024-03-06T07:23:09.127249+0000","last_became_active":"2024-03-06T07:22:56.621316+0000","last_became_peered":"2024-03-06T07:22:56.621316+0000","last_unstale":"2024-03-06T07:23:09.127249+0000","last_undegraded":"2024-03-06T07:23:09.127249+0000","last_fullsized":"2024-03-06T07:23:09.127249+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:40.228473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.743440+0000","last_change":"2024-03-06T07:23:07.749592+0000","last_active":"2024-03-06T07:23:08.743440+0000","last_peered":"2024-03-06T07:23:08.743440+0000","last_clean":"2024-03-06T07:23:08.743440+0000","last_became_active":"2024-03-06T07:23:07.747627+0000","last_became_peered":"2024-03-06T07:23:07.747627+0000","last_unstale":"2024-03-06T07:23:08.743440+0000","last_undegraded":"2024-03-06T07:23:08.743440+0000","last_fullsized":"2024-03-06T07:23:08.743440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:06.707456+0000","last_clean_scrub_stamp":"2024-03-06T07:23:06.707456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:01.336551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127200+0000","last_change":"2024-03-06T07:23:06.737910+0000","last_active":"2024-03-06T07:23:09.127200+0000","last_peered":"2024-03-06T07:23:09.127200+0000","last_clean":"2024-03-06T07:23:09.127200+0000","last_became_active":"2024-03-06T07:23:06.737677+0000","last_became_peered":"2024-03-06T07:23:06.737677+0000","last_unstale":"2024-03-06T07:23:09.127200+0000","last_undegraded":"2024-03-06T07:23:09.127200+0000","last_fullsized":"2024-03-06T07:23:09.127200+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:05.700128+0000","last_clean_scrub_stamp":"2024-03-06T07:23:05.700128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:58.007406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.742706+0000","last_change":"2024-03-06T07:22:57.623523+0000","last_active":"2024-03-06T07:23:08.742706+0000","last_peered":"2024-03-06T07:23:08.742706+0000","last_clean":"2024-03-06T07:23:08.742706+0000","last_became_active":"2024-03-06T07:22:57.623281+0000","last_became_peered":"2024-03-06T07:22:57.623281+0000","last_unstale":"2024-03-06T07:23:08.742706+0000","last_undegraded":"2024-03-06T07:23:08.742706+0000","last_fullsized":"2024-03-06T07:23:08.742706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.599511+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.599511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:34.106175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:09.127887+0000","last_change":"2024-03-06T07:22:56.616740+0000","last_active":"2024-03-06T07:23:09.127887+0000","last_peered":"2024-03-06T07:23:09.127887+0000","last_clean":"2024-03-06T07:23:09.127887+0000","last_became_active":"2024-03-06T07:22:56.614750+0000","last_became_peered":"2024-03-06T07:22:56.614750+0000","last_unstale":"2024-03-06T07:23:09.127887+0000","last_undegraded":"2024-03-06T07:23:09.127887+0000","last_fullsized":"2024-03-06T07:23:09.127887+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.594505+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.594505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:58.177487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":282,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:23:08.741659+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:23:08.741659+0000","last_peered":"2024-03-06T07:23:08.741659+0000","last_clean":"2024-03-06T07:23:08.741659+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:23:08.741659+0000","last_undegraded":"2024-03-06T07:23:08.741659+0000","last_fullsized":"2024-03-06T07:23:08.741659+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339648,"statfs":{"total":96636764160,"available":96603799552,"internally_reserved":0,"allocated":1032192,"data_stored":1048614,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":974848,"data_stored":454447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":958464,"data_stored":458202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574886,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":983040,"data_stored":460044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339720,"statfs":{"total":96636764160,"available":96603873280,"internally_reserved":0,"allocated":958464,"data_stored":458750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":966656,"data_stored":458910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574886,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":966656,"data_stored":458350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1044480,"data_stored":1050854,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:12.149 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:23:12.150 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs refuse_standby_for_another_fs true 2024-03-06T07:23:12.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.285+0000 7f20a4d0f640 1 Processor -- start 2024-03-06T07:23:12.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.285+0000 7f20a4d0f640 1 -- start start 2024-03-06T07:23:12.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.286+0000 7f20a4d0f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20a00c39c0 0x7f20a00ac220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.286+0000 7f20a4d0f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a00c7420 con 0x7f20a00c4940 2024-03-06T07:23:12.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.286+0000 7f20a4d0f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a00ab760 con 0x7f20a00c4560 2024-03-06T07:23:12.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.286+0000 7f20a4d0f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a00c76c0 con 0x7f20a00c39c0 2024-03-06T07:23:12.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.286+0000 7f209ed9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f20a00c4940 0x7f20a00ac760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45238/0 (socket says 172.21.15.1:45238) 2024-03-06T07:23:12.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.286+0000 7f209ed9d640 1 -- 172.21.15.1:0/1476475185 learned_addr learned my addr 172.21.15.1:0/1476475185 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:12.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.286+0000 7f2097fff640 1 --2- 172.21.15.1:0/1476475185 >> v2:172.21.15.98:3300/0 conn(0x7f20a00c39c0 0x7f20a00ac220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.286+0000 7f209dd9b640 1 -- 172.21.15.1:0/1476475185 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140945954 0 0) 0x7f20a00c7420 con 0x7f20a00c4940 2024-03-06T07:23:12.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.286+0000 7f209dd9b640 1 -- 172.21.15.1:0/1476475185 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f208c005730 con 0x7f20a00c4940 2024-03-06T07:23:12.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.286+0000 7f2097fff640 1 -- 172.21.15.1:0/1476475185 >> v1:172.21.15.98:6789/0 conn(0x7f20a00c4560 legacy=0x7f20a0155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.286+0000 7f2097fff640 1 -- 172.21.15.1:0/1476475185 >> v1:172.21.15.1:6789/0 conn(0x7f20a00c4940 legacy=0x7f20a00ac760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.286+0000 7f2097fff640 1 -- 172.21.15.1:0/1476475185 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a00aba20 con 0x7f20a00c39c0 2024-03-06T07:23:12.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.287+0000 7f2097fff640 1 --2- 172.21.15.1:0/1476475185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20a00c39c0 0x7f20a00ac220 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f20880097a0 tx=0x7f208802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c840937c6e26b563 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.287+0000 7f209dd9b640 1 -- 172.21.15.1:0/1476475185 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2088034070 con 0x7f20a00c39c0 2024-03-06T07:23:12.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.287+0000 7f209dd9b640 1 -- 172.21.15.1:0/1476475185 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2088030610 con 0x7f20a00c39c0 2024-03-06T07:23:12.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.287+0000 7f209dd9b640 1 -- 172.21.15.1:0/1476475185 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20880309e0 con 0x7f20a00c39c0 2024-03-06T07:23:12.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.288+0000 7f20a4d0f640 1 -- 172.21.15.1:0/1476475185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20a00c39c0 msgr2=0x7f20a00ac220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.288+0000 7f20a4d0f640 1 --2- 172.21.15.1:0/1476475185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20a00c39c0 0x7f20a00ac220 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f20880097a0 tx=0x7f208802b960 comp rx=0 tx=0).stop 2024-03-06T07:23:12.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.288+0000 7f20a4d0f640 1 -- 172.21.15.1:0/1476475185 shutdown_connections 2024-03-06T07:23:12.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.288+0000 7f20a4d0f640 1 --2- 172.21.15.1:0/1476475185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20a00c39c0 0x7f20a00ac220 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.288+0000 7f20a4d0f640 1 -- 172.21.15.1:0/1476475185 >> 172.21.15.1:0/1476475185 conn(0x7f20a00c7a50 msgr2=0x7f20a00c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:12.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.288+0000 7f20a4d0f640 1 -- 172.21.15.1:0/1476475185 shutdown_connections 2024-03-06T07:23:12.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.288+0000 7f20a4d0f640 1 -- 172.21.15.1:0/1476475185 wait complete. 2024-03-06T07:23:12.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.290+0000 7f20a4d0f640 1 Processor -- start 2024-03-06T07:23:12.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.290+0000 7f20a4d0f640 1 -- start start 2024-03-06T07:23:12.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.291+0000 7f20a4d0f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20a00c39c0 0x7f20a00c01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.291+0000 7f20a4d0f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20a00c4560 0x7f20a0152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.291+0000 7f20a4d0f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20a00c4940 0x7f20a014fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.291+0000 7f20a4d0f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a00acff0 con 0x7f20a00c4940 2024-03-06T07:23:12.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.291+0000 7f20a4d0f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a00ace70 con 0x7f20a00c39c0 2024-03-06T07:23:12.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.291+0000 7f20a4d0f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a006e6d0 con 0x7f20a00c4560 2024-03-06T07:23:12.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.291+0000 7f209ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20a00c4940 0x7f20a014fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.291+0000 7f209ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20a00c4940 0x7f20a014fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42604/0 (socket says 172.21.15.1:42604) 2024-03-06T07:23:12.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.291+0000 7f209ed9d640 1 -- 172.21.15.1:0/3405176863 learned_addr learned my addr 172.21.15.1:0/3405176863 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:12.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.291+0000 7f2097fff640 1 --2- 172.21.15.1:0/3405176863 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20a00c4560 0x7f20a0152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.291+0000 7f209e59c640 1 --2- 172.21.15.1:0/3405176863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20a00c39c0 0x7f20a00c01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.291+0000 7f209ed9d640 1 -- 172.21.15.1:0/3405176863 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20a00c4560 msgr2=0x7f20a0152bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.291+0000 7f209ed9d640 1 --2- 172.21.15.1:0/3405176863 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20a00c4560 0x7f20a0152bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.291+0000 7f209ed9d640 1 -- 172.21.15.1:0/3405176863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20a00c39c0 msgr2=0x7f20a00c01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.291+0000 7f209ed9d640 1 --2- 172.21.15.1:0/3405176863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20a00c39c0 0x7f20a00c01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.291+0000 7f209ed9d640 1 -- 172.21.15.1:0/3405176863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a01e6b60 con 0x7f20a00c4940 2024-03-06T07:23:12.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.291+0000 7f209ed9d640 1 --2- 172.21.15.1:0/3405176863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20a00c4940 0x7f20a014fd80 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f209800caa0 tx=0x7f209800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.292+0000 7f2096ffd640 1 -- 172.21.15.1:0/3405176863 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098017070 con 0x7f20a00c4940 2024-03-06T07:23:12.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.292+0000 7f2096ffd640 1 -- 172.21.15.1:0/3405176863 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209800ac00 con 0x7f20a00c4940 2024-03-06T07:23:12.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.292+0000 7f20a4d0f640 1 -- 172.21.15.1:0/3405176863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a01ea130 con 0x7f20a00c4940 2024-03-06T07:23:12.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.292+0000 7f2096ffd640 1 -- 172.21.15.1:0/3405176863 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098012400 con 0x7f20a00c4940 2024-03-06T07:23:12.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.292+0000 7f20a4d0f640 1 -- 172.21.15.1:0/3405176863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a0153110 con 0x7f20a00c4940 2024-03-06T07:23:12.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.293+0000 7f20a4d0f640 1 -- 172.21.15.1:0/3405176863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a00686e0 con 0x7f20a00c4940 2024-03-06T07:23:12.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.293+0000 7f2096ffd640 1 -- 172.21.15.1:0/3405176863 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f209801a070 con 0x7f20a00c4940 2024-03-06T07:23:12.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.294+0000 7f2096ffd640 1 --2- 172.21.15.1:0/3405176863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f208006b500 0x7f208006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.294+0000 7f2096ffd640 1 -- 172.21.15.1:0/3405176863 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f209808a2b0 con 0x7f20a00c4940 2024-03-06T07:23:12.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.294+0000 7f209e59c640 1 --2- 172.21.15.1:0/3405176863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f208006b500 0x7f208006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.294+0000 7f209e59c640 1 --2- 172.21.15.1:0/3405176863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f208006b500 0x7f208006d9c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f20a0154ef0 tx=0x7f2090009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.298+0000 7f2096ffd640 1 -- 172.21.15.1:0/3405176863 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2098058690 con 0x7f20a00c4940 2024-03-06T07:23:12.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.570+0000 7f20a4d0f640 1 -- 172.21.15.1:0/3405176863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "refuse_standby_for_another_fs", "val": "true"} v 0) v1 -- 0x7f20a00c6180 con 0x7f20a00c4940 2024-03-06T07:23:12.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.749+0000 7f2096ffd640 1 -- 172.21.15.1:0/3405176863 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "refuse_standby_for_another_fs", "val": "true"}]=0 set to refuse standby for another fs v60) v1 ==== 166+0+0 (secure 0 0 0) 0x7f209805c730 con 0x7f20a00c4940 2024-03-06T07:23:12.751 INFO:teuthology.orchestra.run.smithi001.stderr:set to refuse standby for another fs 2024-03-06T07:23:12.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.751+0000 7f20a4d0f640 1 -- 172.21.15.1:0/3405176863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f208006b500 msgr2=0x7f208006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.751+0000 7f20a4d0f640 1 --2- 172.21.15.1:0/3405176863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f208006b500 0x7f208006d9c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f20a0154ef0 tx=0x7f2090009040 comp rx=0 tx=0).stop 2024-03-06T07:23:12.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.751+0000 7f20a4d0f640 1 -- 172.21.15.1:0/3405176863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20a00c4940 msgr2=0x7f20a014fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.751+0000 7f20a4d0f640 1 --2- 172.21.15.1:0/3405176863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20a00c4940 0x7f20a014fd80 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f209800caa0 tx=0x7f209800cf60 comp rx=0 tx=0).stop 2024-03-06T07:23:12.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.751+0000 7f20a4d0f640 1 -- 172.21.15.1:0/3405176863 shutdown_connections 2024-03-06T07:23:12.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.751+0000 7f20a4d0f640 1 --2- 172.21.15.1:0/3405176863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f208006b500 0x7f208006d9c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.751+0000 7f20a4d0f640 1 --2- 172.21.15.1:0/3405176863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20a00c4940 0x7f20a014fd80 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.751+0000 7f20a4d0f640 1 --2- 172.21.15.1:0/3405176863 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20a00c4560 0x7f20a0152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.751+0000 7f20a4d0f640 1 --2- 172.21.15.1:0/3405176863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20a00c39c0 0x7f20a00c01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.751+0000 7f20a4d0f640 1 -- 172.21.15.1:0/3405176863 >> 172.21.15.1:0/3405176863 conn(0x7f20a00c7a50 msgr2=0x7f20a00c63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.751+0000 7f20a4d0f640 1 -- 172.21.15.1:0/3405176863 shutdown_connections 2024-03-06T07:23:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.751+0000 7f20a4d0f640 1 -- 172.21.15.1:0/3405176863 wait complete. 2024-03-06T07:23:12.765 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 5:0 2024-03-06T07:23:12.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.865+0000 7f52a910d640 1 Processor -- start 2024-03-06T07:23:12.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.865+0000 7f52a910d640 1 -- start start 2024-03-06T07:23:12.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.866+0000 7f52a910d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52a40c60b0 0x7f52a40c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.866+0000 7f52a910d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a40c5e40 con 0x7f52a40c3970 2024-03-06T07:23:12.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.866+0000 7f52a910d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a40c6a40 con 0x7f52a4152840 2024-03-06T07:23:12.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.866+0000 7f52a910d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a40c67f0 con 0x7f52a40c60b0 2024-03-06T07:23:12.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.866+0000 7f52a2d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f52a40c3970 0x7f52a4152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45250/0 (socket says 172.21.15.1:45250) 2024-03-06T07:23:12.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.866+0000 7f52a2d9d640 1 -- 172.21.15.1:0/3454170563 learned_addr learned my addr 172.21.15.1:0/3454170563 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:12.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.866+0000 7f52a359e640 1 --2- 172.21.15.1:0/3454170563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52a40c60b0 0x7f52a40c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.867+0000 7f52a1d9b640 1 -- 172.21.15.1:0/3454170563 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941450890 0 0) 0x7f52a40c5e40 con 0x7f52a40c3970 2024-03-06T07:23:12.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.867+0000 7f52a359e640 1 -- 172.21.15.1:0/3454170563 >> v1:172.21.15.98:6789/0 conn(0x7f52a4152840 legacy=0x7f52a41553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.867+0000 7f52a359e640 1 -- 172.21.15.1:0/3454170563 >> v1:172.21.15.1:6789/0 conn(0x7f52a40c3970 legacy=0x7f52a4152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.867+0000 7f52a359e640 1 -- 172.21.15.1:0/3454170563 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a40c6d20 con 0x7f52a40c60b0 2024-03-06T07:23:12.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.867+0000 7f52a359e640 1 --2- 172.21.15.1:0/3454170563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52a40c60b0 0x7f52a40c8170 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5294009dc0 tx=0x7f529402bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc49514831cd0ae8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.868+0000 7f52a1d9b640 1 -- 172.21.15.1:0/3454170563 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5294039070 con 0x7f52a40c60b0 2024-03-06T07:23:12.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.868+0000 7f52a1d9b640 1 -- 172.21.15.1:0/3454170563 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529402caa0 con 0x7f52a40c60b0 2024-03-06T07:23:12.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.868+0000 7f52a1d9b640 1 -- 172.21.15.1:0/3454170563 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5294034440 con 0x7f52a40c60b0 2024-03-06T07:23:12.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.868+0000 7f52a910d640 1 -- 172.21.15.1:0/3454170563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52a40c60b0 msgr2=0x7f52a40c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.868+0000 7f52a910d640 1 --2- 172.21.15.1:0/3454170563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52a40c60b0 0x7f52a40c8170 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5294009dc0 tx=0x7f529402bd70 comp rx=0 tx=0).stop 2024-03-06T07:23:12.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.868+0000 7f52a910d640 1 -- 172.21.15.1:0/3454170563 shutdown_connections 2024-03-06T07:23:12.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.868+0000 7f52a910d640 1 --2- 172.21.15.1:0/3454170563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52a40c60b0 0x7f52a40c8170 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.868+0000 7f52a910d640 1 -- 172.21.15.1:0/3454170563 >> 172.21.15.1:0/3454170563 conn(0x7f52a40c88f0 msgr2=0x7f52a40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:12.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.868+0000 7f52a910d640 1 -- 172.21.15.1:0/3454170563 shutdown_connections 2024-03-06T07:23:12.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.868+0000 7f52a910d640 1 -- 172.21.15.1:0/3454170563 wait complete. 2024-03-06T07:23:12.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.870+0000 7f52a910d640 1 Processor -- start 2024-03-06T07:23:12.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.870+0000 7f52a910d640 1 -- start start 2024-03-06T07:23:12.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.870+0000 7f52a910d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52a40c3970 0x7f52a40ac520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.870+0000 7f52a910d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52a40c60b0 0x7f52a4157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.870+0000 7f52a910d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52a4152840 0x7f52a40a9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.870+0000 7f52a910d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a40c75c0 con 0x7f52a40c3970 2024-03-06T07:23:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.870+0000 7f52a910d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a40c72e0 con 0x7f52a40c60b0 2024-03-06T07:23:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.870+0000 7f52a910d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a40c3d50 con 0x7f52a4152840 2024-03-06T07:23:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.870+0000 7f52a2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52a40c3970 0x7f52a40ac520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.870+0000 7f52a2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52a40c3970 0x7f52a40ac520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42618/0 (socket says 172.21.15.1:42618) 2024-03-06T07:23:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.870+0000 7f52a2d9d640 1 -- 172.21.15.1:0/3995513110 learned_addr learned my addr 172.21.15.1:0/3995513110 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.870+0000 7f52a359e640 1 --2- 172.21.15.1:0/3995513110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52a4152840 0x7f52a40a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.870+0000 7f52a259c640 1 --2- 172.21.15.1:0/3995513110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52a40c60b0 0x7f52a4157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.871+0000 7f52a2d9d640 1 -- 172.21.15.1:0/3995513110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52a4152840 msgr2=0x7f52a40a9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.871+0000 7f52a2d9d640 1 --2- 172.21.15.1:0/3995513110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52a4152840 0x7f52a40a9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.871+0000 7f52a2d9d640 1 -- 172.21.15.1:0/3995513110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52a40c60b0 msgr2=0x7f52a4157a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.871+0000 7f52a2d9d640 1 --2- 172.21.15.1:0/3995513110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52a40c60b0 0x7f52a4157a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.871+0000 7f52a2d9d640 1 -- 172.21.15.1:0/3995513110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a41eae70 con 0x7f52a40c3970 2024-03-06T07:23:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.871+0000 7f52a2d9d640 1 --2- 172.21.15.1:0/3995513110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52a40c3970 0x7f52a40ac520 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f529c00c4a0 tx=0x7f529c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.871+0000 7f529b7fe640 1 -- 172.21.15.1:0/3995513110 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c018070 con 0x7f52a40c3970 2024-03-06T07:23:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.871+0000 7f52a910d640 1 -- 172.21.15.1:0/3995513110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a41eb150 con 0x7f52a40c3970 2024-03-06T07:23:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.871+0000 7f529b7fe640 1 -- 172.21.15.1:0/3995513110 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529c0074e0 con 0x7f52a40c3970 2024-03-06T07:23:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.871+0000 7f529b7fe640 1 -- 172.21.15.1:0/3995513110 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c004a20 con 0x7f52a40c3970 2024-03-06T07:23:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.871+0000 7f52a910d640 1 -- 172.21.15.1:0/3995513110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a41eb710 con 0x7f52a40c3970 2024-03-06T07:23:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.872+0000 7f52a910d640 1 -- 172.21.15.1:0/3995513110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a406e900 con 0x7f52a40c3970 2024-03-06T07:23:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.873+0000 7f529b7fe640 1 -- 172.21.15.1:0/3995513110 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f529c004bc0 con 0x7f52a40c3970 2024-03-06T07:23:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.873+0000 7f529b7fe640 1 --2- 172.21.15.1:0/3995513110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f528406b4d0 0x7f528406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.873+0000 7f529b7fe640 1 -- 172.21.15.1:0/3995513110 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f529c0899e0 con 0x7f52a40c3970 2024-03-06T07:23:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.873+0000 7f52a259c640 1 --2- 172.21.15.1:0/3995513110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f528406b4d0 0x7f528406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.874+0000 7f52a259c640 1 --2- 172.21.15.1:0/3995513110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f528406b4d0 0x7f528406d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f52a40aca60 tx=0x7f528c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:12.877+0000 7f529b7fe640 1 -- 172.21.15.1:0/3995513110 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52a406e900 con 0x7f52a40c3970 2024-03-06T07:23:12.905 INFO:tasks.ceph.mds.d.smithi098.stderr:2024-03-06T07:23:12.903+0000 7f4eca4e9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.132+0000 7f52a910d640 1 -- 172.21.15.1:0/3995513110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "5:0"} v 0) v1 -- 0x7f52a40c50c0 con 0x7f52a40c3970 2024-03-06T07:23:13.134 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:23:13.132+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:23:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.133+0000 7f529b7fe640 1 -- 172.21.15.1:0/3995513110 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "5:0"}]=0 failed mds gid 6405 v61) v1 ==== 97+0+0 (secure 0 0 0) 0x7f52a40c50c0 con 0x7f52a40c3970 2024-03-06T07:23:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:failed mds gid 6405 2024-03-06T07:23:13.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.137+0000 7f52a910d640 1 -- 172.21.15.1:0/3995513110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f528406b4d0 msgr2=0x7f528406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.137+0000 7f52a910d640 1 --2- 172.21.15.1:0/3995513110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f528406b4d0 0x7f528406d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f52a40aca60 tx=0x7f528c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:13.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.137+0000 7f52a910d640 1 -- 172.21.15.1:0/3995513110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52a40c3970 msgr2=0x7f52a40ac520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.137+0000 7f52a910d640 1 --2- 172.21.15.1:0/3995513110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52a40c3970 0x7f52a40ac520 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f529c00c4a0 tx=0x7f529c00c960 comp rx=0 tx=0).stop 2024-03-06T07:23:13.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.138+0000 7f52a910d640 1 -- 172.21.15.1:0/3995513110 shutdown_connections 2024-03-06T07:23:13.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.138+0000 7f52a910d640 1 --2- 172.21.15.1:0/3995513110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f528406b4d0 0x7f528406d990 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.138+0000 7f52a910d640 1 --2- 172.21.15.1:0/3995513110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52a4152840 0x7f52a40a9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.138+0000 7f52a910d640 1 --2- 172.21.15.1:0/3995513110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52a40c60b0 0x7f52a4157a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.138+0000 7f52a910d640 1 --2- 172.21.15.1:0/3995513110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52a40c3970 0x7f52a40ac520 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.138+0000 7f52a910d640 1 -- 172.21.15.1:0/3995513110 >> 172.21.15.1:0/3995513110 conn(0x7f52a40c88f0 msgr2=0x7f52a41553d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.138+0000 7f52a910d640 1 -- 172.21.15.1:0/3995513110 shutdown_connections 2024-03-06T07:23:13.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.138+0000 7f52a910d640 1 -- 172.21.15.1:0/3995513110 wait complete. 2024-03-06T07:23:13.141 INFO:tasks.ceph.mds.c.smithi001.stderr: -360> 2024-03-06T07:23:02.902+0000 7f78dfb7e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:13.152 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:13.164 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:23:13.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.252+0000 7f36936fb640 1 Processor -- start 2024-03-06T07:23:13.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.252+0000 7f36936fb640 1 -- start start 2024-03-06T07:23:13.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.253+0000 7f36936fb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f368c0c3260 0x7f368c0c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.253+0000 7f36936fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f368c155b70 0x7f368c159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.253+0000 7f36936fb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f368c0c2b90 con 0x7f368c0c8cd0 2024-03-06T07:23:13.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.253+0000 7f36936fb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f368c156020 con 0x7f368c0c3260 2024-03-06T07:23:13.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.253+0000 7f36936fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f368c0c37b0 con 0x7f368c155b70 2024-03-06T07:23:13.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.253+0000 7f3690c96640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f368c0c3260 0x7f368c0c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.253+0000 7f3690c96640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f368c0c3260 0x7f368c0c90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42646/0 (socket says 172.21.15.1:42646) 2024-03-06T07:23:13.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.253+0000 7f3690c96640 1 -- 172.21.15.1:0/1271170862 learned_addr learned my addr 172.21.15.1:0/1271170862 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:13.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.253+0000 7f3691497640 1 --1- 172.21.15.1:0/1271170862 >> v1:172.21.15.98:6789/0 conn(0x7f368c0c8cd0 0x7f368c0c7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:52560/0 (socket says 172.21.15.1:52560) 2024-03-06T07:23:13.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.253+0000 7f3691c98640 1 --2- 172.21.15.1:0/1271170862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f368c155b70 0x7f368c159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.253+0000 7f3683fff640 1 -- 172.21.15.1:0/1271170862 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284685146 0 0) 0x7f368c0c2b90 con 0x7f368c0c8cd0 2024-03-06T07:23:13.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.253+0000 7f3690c96640 1 -- 172.21.15.1:0/1271170862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f368c155b70 msgr2=0x7f368c159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:13.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.253+0000 7f3690c96640 1 --2- 172.21.15.1:0/1271170862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f368c155b70 0x7f368c159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.253+0000 7f3690c96640 1 -- 172.21.15.1:0/1271170862 >> v1:172.21.15.98:6789/0 conn(0x7f368c0c8cd0 legacy=0x7f368c0c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.253+0000 7f3690c96640 1 -- 172.21.15.1:0/1271170862 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f368c0c3a90 con 0x7f368c0c3260 2024-03-06T07:23:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.254+0000 7f3690c96640 1 --2- 172.21.15.1:0/1271170862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f368c0c3260 0x7f368c0c90b0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f3688009e50 tx=0x7f368802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68824405de786f2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.254+0000 7f3683fff640 1 -- 172.21.15.1:0/1271170862 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368802ca70 con 0x7f368c0c3260 2024-03-06T07:23:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.254+0000 7f3683fff640 1 -- 172.21.15.1:0/1271170862 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368802cc10 con 0x7f368c0c3260 2024-03-06T07:23:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.254+0000 7f3683fff640 1 -- 172.21.15.1:0/1271170862 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36880343f0 con 0x7f368c0c3260 2024-03-06T07:23:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.255+0000 7f36936fb640 1 -- 172.21.15.1:0/1271170862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f368c0c3260 msgr2=0x7f368c0c90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.255+0000 7f36936fb640 1 --2- 172.21.15.1:0/1271170862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f368c0c3260 0x7f368c0c90b0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f3688009e50 tx=0x7f368802bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.255+0000 7f36936fb640 1 -- 172.21.15.1:0/1271170862 shutdown_connections 2024-03-06T07:23:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.255+0000 7f36936fb640 1 --2- 172.21.15.1:0/1271170862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f368c155b70 0x7f368c159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.255+0000 7f36936fb640 1 --2- 172.21.15.1:0/1271170862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f368c0c3260 0x7f368c0c90b0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.255+0000 7f36936fb640 1 -- 172.21.15.1:0/1271170862 >> 172.21.15.1:0/1271170862 conn(0x7f368c06b3e0 msgr2=0x7f368c06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.255+0000 7f36936fb640 1 -- 172.21.15.1:0/1271170862 shutdown_connections 2024-03-06T07:23:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.255+0000 7f36936fb640 1 -- 172.21.15.1:0/1271170862 wait complete. 2024-03-06T07:23:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.256+0000 7f36936fb640 1 Processor -- start 2024-03-06T07:23:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.256+0000 7f36936fb640 1 -- start start 2024-03-06T07:23:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f36936fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f368c0c3260 0x7f368c14ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f36936fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f368c0c8cd0 0x7f368c151b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f36936fb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f368c155b70 0x7f368c14de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f36936fb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368c159550 con 0x7f368c0c3260 2024-03-06T07:23:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f36936fb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368c158af0 con 0x7f368c155b70 2024-03-06T07:23:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f36936fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f368c0c8560 con 0x7f368c0c8cd0 2024-03-06T07:23:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f3691497640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f368c0c3260 0x7f368c14ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f3691497640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f368c0c3260 0x7f368c14ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42648/0 (socket says 172.21.15.1:42648) 2024-03-06T07:23:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f3691497640 1 -- 172.21.15.1:0/457019282 learned_addr learned my addr 172.21.15.1:0/457019282 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f3691c98640 1 --2- 172.21.15.1:0/457019282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f368c155b70 0x7f368c14de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f3690c96640 1 --2- 172.21.15.1:0/457019282 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f368c0c8cd0 0x7f368c151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f3691497640 1 -- 172.21.15.1:0/457019282 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f368c0c8cd0 msgr2=0x7f368c151b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f3691497640 1 --2- 172.21.15.1:0/457019282 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f368c0c8cd0 0x7f368c151b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f3691497640 1 -- 172.21.15.1:0/457019282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f368c155b70 msgr2=0x7f368c14de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f3691497640 1 --2- 172.21.15.1:0/457019282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f368c155b70 0x7f368c14de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f3691497640 1 -- 172.21.15.1:0/457019282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f368c14f1a0 con 0x7f368c0c3260 2024-03-06T07:23:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f3690c96640 1 --2- 172.21.15.1:0/457019282 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f368c0c8cd0 0x7f368c151b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f3691497640 1 --2- 172.21.15.1:0/457019282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f368c0c3260 0x7f368c14ec60 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f367c00c4d0 tx=0x7f367c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f3681ffb640 1 -- 172.21.15.1:0/457019282 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367c018070 con 0x7f368c0c3260 2024-03-06T07:23:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f3681ffb640 1 -- 172.21.15.1:0/457019282 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367c007500 con 0x7f368c0c3260 2024-03-06T07:23:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f36936fb640 1 -- 172.21.15.1:0/457019282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f368c14e9c0 con 0x7f368c0c3260 2024-03-06T07:23:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.257+0000 7f3681ffb640 1 -- 172.21.15.1:0/457019282 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367c004a20 con 0x7f368c0c3260 2024-03-06T07:23:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.258+0000 7f36936fb640 1 -- 172.21.15.1:0/457019282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f368c1eeee0 con 0x7f368c0c3260 2024-03-06T07:23:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.259+0000 7f36936fb640 1 -- 172.21.15.1:0/457019282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f368c067fd0 con 0x7f368c0c3260 2024-03-06T07:23:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.259+0000 7f3681ffb640 1 -- 172.21.15.1:0/457019282 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f367c01b070 con 0x7f368c0c3260 2024-03-06T07:23:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.259+0000 7f3681ffb640 1 --2- 172.21.15.1:0/457019282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f367006b500 0x7f367006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.260+0000 7f3681ffb640 1 -- 172.21.15.1:0/457019282 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f367c089930 con 0x7f368c0c3260 2024-03-06T07:23:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.260+0000 7f3690c96640 1 --2- 172.21.15.1:0/457019282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f367006b500 0x7f367006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.260+0000 7f3690c96640 1 --2- 172.21.15.1:0/457019282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f367006b500 0x7f367006d9c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3688035000 tx=0x7f3688030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.264+0000 7f3681ffb640 1 -- 172.21.15.1:0/457019282 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f368c14f1a0 con 0x7f368c0c3260 2024-03-06T07:23:13.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.485+0000 7f36936fb640 1 -- 172.21.15.1:0/457019282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f368c0c2d80 con 0x7f368c0c3260 2024-03-06T07:23:13.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.485+0000 7f3681ffb640 1 -- 172.21.15.1:0/457019282 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5447 (secure 0 0 0) 0x7f367c057c10 con 0x7f368c0c3260 2024-03-06T07:23:13.487 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:13.488 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":61,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.1:6835/2685225325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2685225325},{"type":"v1","addr":"172.21.15.1:6835","nonce":2685225325}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.98:6836/4069609682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4069609682},{"type":"v1","addr":"172.21.15.98:6836","nonce":4069609682}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.969135+0000","modified":"2024-03-06T07:23:13.133580+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5},{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:07.074540+0000","modified":"2024-03-06T07:23:10.095697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5845":{"gid":5845,"name":"d","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/114998998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":114998998},{"type":"v1","addr":"172.21.15.98:6837","nonce":114998998}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5845,"qdb_cluster":[5845]},"id":6}]} 2024-03-06T07:23:13.488 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 61 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.487+0000 7f36936fb640 1 -- 172.21.15.1:0/457019282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f367006b500 msgr2=0x7f367006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.487+0000 7f36936fb640 1 --2- 172.21.15.1:0/457019282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f367006b500 0x7f367006d9c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3688035000 tx=0x7f3688030000 comp rx=0 tx=0).stop 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.487+0000 7f36936fb640 1 -- 172.21.15.1:0/457019282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f368c0c3260 msgr2=0x7f368c14ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.487+0000 7f36936fb640 1 --2- 172.21.15.1:0/457019282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f368c0c3260 0x7f368c14ec60 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f367c00c4d0 tx=0x7f367c00c990 comp rx=0 tx=0).stop 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.487+0000 7f36936fb640 1 -- 172.21.15.1:0/457019282 shutdown_connections 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.487+0000 7f36936fb640 1 --2- 172.21.15.1:0/457019282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f367006b500 0x7f367006d9c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.487+0000 7f36936fb640 1 --2- 172.21.15.1:0/457019282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f368c155b70 0x7f368c14de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.487+0000 7f36936fb640 1 --2- 172.21.15.1:0/457019282 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f368c0c8cd0 0x7f368c151b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.487+0000 7f36936fb640 1 --2- 172.21.15.1:0/457019282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f368c0c3260 0x7f368c14ec60 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.487+0000 7f36936fb640 1 -- 172.21.15.1:0/457019282 >> 172.21.15.1:0/457019282 conn(0x7f368c06b3e0 msgr2=0x7f368c0c7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.487+0000 7f36936fb640 1 -- 172.21.15.1:0/457019282 shutdown_connections 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.487+0000 7f36936fb640 1 -- 172.21.15.1:0/457019282 wait complete. 2024-03-06T07:23:13.500 INFO:tasks.cephfs.test_failover:Waiting for former active to reclaim its spot 2024-03-06T07:23:13.500 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-03-06T07:23:13.652 INFO:teuthology.orchestra.run.smithi001.stdout:{"mds_beacon_grace":"15.000000"} 2024-03-06T07:23:13.652 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-03-06T07:23:13.652 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:13.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.751+0000 7f9343f0c640 1 Processor -- start 2024-03-06T07:23:13.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.751+0000 7f9343f0c640 1 -- start start 2024-03-06T07:23:13.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.752+0000 7f9343f0c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f933c152650 0x7f933c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.752+0000 7f9343f0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f933c0c6b10 0x7f933c158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.752+0000 7f9343f0c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c0c2fc0 con 0x7f933c0bf9d0 2024-03-06T07:23:13.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.752+0000 7f9343f0c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f933c155800 con 0x7f933c152650 2024-03-06T07:23:13.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.752+0000 7f9343f0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933c0c6690 con 0x7f933c0c6b10 2024-03-06T07:23:13.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.752+0000 7f9341ca8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f933c152650 0x7f933c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.752+0000 7f93424a9640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f933c0bf9d0 0x7f933c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45274/0 (socket says 172.21.15.1:45274) 2024-03-06T07:23:13.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.752+0000 7f9341ca8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f933c152650 0x7f933c155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42650/0 (socket says 172.21.15.1:42650) 2024-03-06T07:23:13.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.752+0000 7f93424a9640 1 -- 172.21.15.1:0/93034000 learned_addr learned my addr 172.21.15.1:0/93034000 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:13.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.752+0000 7f93414a7640 1 --2- 172.21.15.1:0/93034000 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f933c0c6b10 0x7f933c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.753+0000 7f9341ca8640 1 -- 172.21.15.1:0/93034000 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f933c0c6b10 msgr2=0x7f933c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:13.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.753+0000 7f9341ca8640 1 --2- 172.21.15.1:0/93034000 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f933c0c6b10 0x7f933c158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.753+0000 7f9341ca8640 1 -- 172.21.15.1:0/93034000 >> v1:172.21.15.1:6789/0 conn(0x7f933c0bf9d0 legacy=0x7f933c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.753+0000 7f9340ca6640 1 -- 172.21.15.1:0/93034000 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85345429 0 0) 0x7f933c0c2fc0 con 0x7f933c0bf9d0 2024-03-06T07:23:13.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.753+0000 7f9341ca8640 1 -- 172.21.15.1:0/93034000 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c0c7280 con 0x7f933c152650 2024-03-06T07:23:13.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.753+0000 7f9341ca8640 1 --2- 172.21.15.1:0/93034000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f933c152650 0x7f933c155c90 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f93340097a0 tx=0x7f933402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a911022c9309df6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.753+0000 7f9340ca6640 1 -- 172.21.15.1:0/93034000 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93340315a0 con 0x7f933c152650 2024-03-06T07:23:13.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.753+0000 7f9340ca6640 1 -- 172.21.15.1:0/93034000 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9334037070 con 0x7f933c152650 2024-03-06T07:23:13.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.753+0000 7f9340ca6640 1 -- 172.21.15.1:0/93034000 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9334031960 con 0x7f933c152650 2024-03-06T07:23:13.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.754+0000 7f9343f0c640 1 -- 172.21.15.1:0/93034000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f933c152650 msgr2=0x7f933c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.754+0000 7f9343f0c640 1 --2- 172.21.15.1:0/93034000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f933c152650 0x7f933c155c90 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f93340097a0 tx=0x7f933402b840 comp rx=0 tx=0).stop 2024-03-06T07:23:13.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.754+0000 7f9343f0c640 1 -- 172.21.15.1:0/93034000 shutdown_connections 2024-03-06T07:23:13.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.754+0000 7f9343f0c640 1 --2- 172.21.15.1:0/93034000 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f933c0c6b10 0x7f933c158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.754+0000 7f9343f0c640 1 --2- 172.21.15.1:0/93034000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f933c152650 0x7f933c155c90 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.754+0000 7f9343f0c640 1 -- 172.21.15.1:0/93034000 >> 172.21.15.1:0/93034000 conn(0x7f933c066ee0 msgr2=0x7f933c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.754+0000 7f9343f0c640 1 -- 172.21.15.1:0/93034000 shutdown_connections 2024-03-06T07:23:13.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.754+0000 7f9343f0c640 1 -- 172.21.15.1:0/93034000 wait complete. 2024-03-06T07:23:13.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.755+0000 7f9343f0c640 1 Processor -- start 2024-03-06T07:23:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.755+0000 7f9343f0c640 1 -- start start 2024-03-06T07:23:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.756+0000 7f9343f0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f933c0bf9d0 0x7f933c0c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.756+0000 7f9343f0c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f933c152650 0x7f933c152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.756+0000 7f9343f0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c1eb840 0x7f933c1533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.756+0000 7f9343f0c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c155490 con 0x7f933c152650 2024-03-06T07:23:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.756+0000 7f9343f0c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c1551b0 con 0x7f933c1eb840 2024-03-06T07:23:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.756+0000 7f9343f0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933c152a30 con 0x7f933c0bf9d0 2024-03-06T07:23:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.756+0000 7f93414a7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f933c152650 0x7f933c152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.756+0000 7f93414a7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f933c152650 0x7f933c152e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42664/0 (socket says 172.21.15.1:42664) 2024-03-06T07:23:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.756+0000 7f93414a7640 1 -- 172.21.15.1:0/3698446888 learned_addr learned my addr 172.21.15.1:0/3698446888 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.756+0000 7f9341ca8640 1 --2- 172.21.15.1:0/3698446888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f933c0bf9d0 0x7f933c0c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.756+0000 7f93424a9640 1 --2- 172.21.15.1:0/3698446888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c1eb840 0x7f933c1533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.756+0000 7f93414a7640 1 -- 172.21.15.1:0/3698446888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f933c0bf9d0 msgr2=0x7f933c0c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.756+0000 7f93414a7640 1 --2- 172.21.15.1:0/3698446888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f933c0bf9d0 0x7f933c0c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.756+0000 7f93414a7640 1 -- 172.21.15.1:0/3698446888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c1eb840 msgr2=0x7f933c1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.756+0000 7f93414a7640 1 --2- 172.21.15.1:0/3698446888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c1eb840 0x7f933c1533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.756+0000 7f93414a7640 1 -- 172.21.15.1:0/3698446888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c1544f0 con 0x7f933c152650 2024-03-06T07:23:13.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.756+0000 7f93414a7640 1 --2- 172.21.15.1:0/3698446888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f933c152650 0x7f933c152e90 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f933000e9b0 tx=0x7f933000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.758+0000 7f933a7fc640 1 -- 172.21.15.1:0/3698446888 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9330017070 con 0x7f933c152650 2024-03-06T07:23:13.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.758+0000 7f933a7fc640 1 -- 172.21.15.1:0/3698446888 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933000c9a0 con 0x7f933c152650 2024-03-06T07:23:13.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.758+0000 7f9343f0c640 1 -- 172.21.15.1:0/3698446888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933c1547d0 con 0x7f933c152650 2024-03-06T07:23:13.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.758+0000 7f933a7fc640 1 -- 172.21.15.1:0/3698446888 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933000cd90 con 0x7f933c152650 2024-03-06T07:23:13.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.758+0000 7f9343f0c640 1 -- 172.21.15.1:0/3698446888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933c154d90 con 0x7f933c152650 2024-03-06T07:23:13.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.759+0000 7f9343f0c640 1 -- 172.21.15.1:0/3698446888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f933c067fd0 con 0x7f933c152650 2024-03-06T07:23:13.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.759+0000 7f933a7fc640 1 -- 172.21.15.1:0/3698446888 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f933001a070 con 0x7f933c152650 2024-03-06T07:23:13.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.760+0000 7f933a7fc640 1 --2- 172.21.15.1:0/3698446888 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f932806b4d0 0x7f932806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.760+0000 7f933a7fc640 1 -- 172.21.15.1:0/3698446888 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f933008a990 con 0x7f933c152650 2024-03-06T07:23:13.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.760+0000 7f9341ca8640 1 --2- 172.21.15.1:0/3698446888 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f932806b4d0 0x7f932806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.761+0000 7f9341ca8640 1 --2- 172.21.15.1:0/3698446888 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f932806b4d0 0x7f932806d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9334036000 tx=0x7f9334008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.764+0000 7f933a7fc640 1 -- 172.21.15.1:0/3698446888 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9330058d10 con 0x7f933c152650 2024-03-06T07:23:13.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.983+0000 7f9343f0c640 1 -- 172.21.15.1:0/3698446888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f933c0b06f0 con 0x7f933c152650 2024-03-06T07:23:13.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.983+0000 7f933a7fc640 1 -- 172.21.15.1:0/3698446888 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5447 (secure 0 0 0) 0x7f933005cdb0 con 0x7f933c152650 2024-03-06T07:23:13.986 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:13.986 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":61,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.1:6835/2685225325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2685225325},{"type":"v1","addr":"172.21.15.1:6835","nonce":2685225325}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.98:6836/4069609682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4069609682},{"type":"v1","addr":"172.21.15.98:6836","nonce":4069609682}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.969135+0000","modified":"2024-03-06T07:23:13.133580+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5},{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:07.074540+0000","modified":"2024-03-06T07:23:10.095697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5845":{"gid":5845,"name":"d","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/114998998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":114998998},{"type":"v1","addr":"172.21.15.98:6837","nonce":114998998}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5845,"qdb_cluster":[5845]},"id":6}]} 2024-03-06T07:23:13.986 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 61 2024-03-06T07:23:13.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.985+0000 7f9343f0c640 1 -- 172.21.15.1:0/3698446888 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f932806b4d0 msgr2=0x7f932806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.985+0000 7f9343f0c640 1 --2- 172.21.15.1:0/3698446888 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f932806b4d0 0x7f932806d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9334036000 tx=0x7f9334008030 comp rx=0 tx=0).stop 2024-03-06T07:23:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.985+0000 7f9343f0c640 1 -- 172.21.15.1:0/3698446888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f933c152650 msgr2=0x7f933c152e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.985+0000 7f9343f0c640 1 --2- 172.21.15.1:0/3698446888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f933c152650 0x7f933c152e90 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f933000e9b0 tx=0x7f933000ee70 comp rx=0 tx=0).stop 2024-03-06T07:23:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.985+0000 7f9343f0c640 1 -- 172.21.15.1:0/3698446888 shutdown_connections 2024-03-06T07:23:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.985+0000 7f9343f0c640 1 --2- 172.21.15.1:0/3698446888 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f932806b4d0 0x7f932806d990 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.985+0000 7f9343f0c640 1 --2- 172.21.15.1:0/3698446888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c1eb840 0x7f933c1533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.985+0000 7f9343f0c640 1 --2- 172.21.15.1:0/3698446888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f933c152650 0x7f933c152e90 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.985+0000 7f9343f0c640 1 --2- 172.21.15.1:0/3698446888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f933c0bf9d0 0x7f933c0c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.985+0000 7f9343f0c640 1 -- 172.21.15.1:0/3698446888 >> 172.21.15.1:0/3698446888 conn(0x7f933c066ee0 msgr2=0x7f933c0c5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.985+0000 7f9343f0c640 1 -- 172.21.15.1:0/3698446888 shutdown_connections 2024-03-06T07:23:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:13.985+0000 7f9343f0c640 1 -- 172.21.15.1:0/3698446888 wait complete. 2024-03-06T07:23:13.998 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-03-06T07:23:18.999 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:19.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.099+0000 7f52704fb640 1 Processor -- start 2024-03-06T07:23:19.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.099+0000 7f52704fb640 1 -- start start 2024-03-06T07:23:19.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.099+0000 7f52704fb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f52680c3970 0x7f52680c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.099+0000 7f52704fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52680c60b0 0x7f5268152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.100+0000 7f52704fb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52680c5e40 con 0x7f5268152670 2024-03-06T07:23:19.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.100+0000 7f52704fb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52680c67f0 con 0x7f52680c3970 2024-03-06T07:23:19.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.100+0000 7f52704fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52680c6a40 con 0x7f52680c60b0 2024-03-06T07:23:19.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.100+0000 7f526da96640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52680c60b0 0x7f5268152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.100+0000 7f526e297640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f52680c3970 0x7f52680c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.100+0000 7f526da96640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52680c60b0 0x7f5268152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:57394/0 (socket says 172.21.15.1:57394) 2024-03-06T07:23:19.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.100+0000 7f526da96640 1 -- 172.21.15.1:0/4224593091 learned_addr learned my addr 172.21.15.1:0/4224593091 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:19.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.100+0000 7f526ea98640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5268152670 0x7f52681553d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:55070/0 (socket says 172.21.15.1:55070) 2024-03-06T07:23:19.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.100+0000 7f526da96640 1 -- 172.21.15.1:0/4224593091 >> v2:172.21.15.98:3300/0 conn(0x7f52680c3970 msgr2=0x7f52680c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:19.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.100+0000 7f526da96640 1 --2- 172.21.15.1:0/4224593091 >> v2:172.21.15.98:3300/0 conn(0x7f52680c3970 0x7f52680c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.100+0000 7f526da96640 1 -- 172.21.15.1:0/4224593091 >> v1:172.21.15.98:6789/0 conn(0x7f5268152670 legacy=0x7f52681553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.100+0000 7f526da96640 1 -- 172.21.15.1:0/4224593091 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52680c6d20 con 0x7f52680c60b0 2024-03-06T07:23:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.100+0000 7f526e297640 1 --2- 172.21.15.1:0/4224593091 >> v2:172.21.15.98:3300/0 conn(0x7f52680c3970 0x7f52680c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:19.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.101+0000 7f526da96640 1 --2- 172.21.15.1:0/4224593091 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52680c60b0 0x7f5268152130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5258009870 tx=0x7f525802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa95863dd2ed0271 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.101+0000 7f526d295640 1 -- 172.21.15.1:0/4224593091 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258034070 con 0x7f52680c60b0 2024-03-06T07:23:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.101+0000 7f526d295640 1 -- 172.21.15.1:0/4224593091 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5258030840 con 0x7f52680c60b0 2024-03-06T07:23:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.101+0000 7f526d295640 1 -- 172.21.15.1:0/4224593091 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258030c10 con 0x7f52680c60b0 2024-03-06T07:23:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.102+0000 7f52704fb640 1 -- 172.21.15.1:0/4224593091 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52680c60b0 msgr2=0x7f5268152130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.102+0000 7f52704fb640 1 --2- 172.21.15.1:0/4224593091 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52680c60b0 0x7f5268152130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5258009870 tx=0x7f525802b980 comp rx=0 tx=0).stop 2024-03-06T07:23:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.102+0000 7f52704fb640 1 -- 172.21.15.1:0/4224593091 shutdown_connections 2024-03-06T07:23:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.102+0000 7f52704fb640 1 --2- 172.21.15.1:0/4224593091 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52680c60b0 0x7f5268152130 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.102+0000 7f52704fb640 1 --2- 172.21.15.1:0/4224593091 >> v2:172.21.15.98:3300/0 conn(0x7f52680c3970 0x7f52680c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.102+0000 7f52704fb640 1 -- 172.21.15.1:0/4224593091 >> 172.21.15.1:0/4224593091 conn(0x7f52680c88f0 msgr2=0x7f52680c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.102+0000 7f52704fb640 1 -- 172.21.15.1:0/4224593091 shutdown_connections 2024-03-06T07:23:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.102+0000 7f52704fb640 1 -- 172.21.15.1:0/4224593091 wait complete. 2024-03-06T07:23:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.103+0000 7f52704fb640 1 Processor -- start 2024-03-06T07:23:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.103+0000 7f52704fb640 1 -- start start 2024-03-06T07:23:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.103+0000 7f52704fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52680c3970 0x7f526806fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.103+0000 7f52704fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52680c60b0 0x7f5268156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f52704fb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5268152670 0x7f5268156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f52704fb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52680c75c0 con 0x7f52680c3970 2024-03-06T07:23:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f52704fb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52680c72e0 con 0x7f5268152670 2024-03-06T07:23:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f52704fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52680c3d50 con 0x7f52680c60b0 2024-03-06T07:23:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f526e297640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52680c3970 0x7f526806fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f526e297640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52680c3970 0x7f526806fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49594/0 (socket says 172.21.15.1:49594) 2024-03-06T07:23:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f526e297640 1 -- 172.21.15.1:0/765165138 learned_addr learned my addr 172.21.15.1:0/765165138 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f526da96640 1 --2- 172.21.15.1:0/765165138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52680c60b0 0x7f5268156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f526ea98640 1 --2- 172.21.15.1:0/765165138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5268152670 0x7f5268156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f526e297640 1 -- 172.21.15.1:0/765165138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52680c60b0 msgr2=0x7f5268156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f526e297640 1 --2- 172.21.15.1:0/765165138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52680c60b0 0x7f5268156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f526e297640 1 -- 172.21.15.1:0/765165138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5268152670 msgr2=0x7f5268156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f526e297640 1 --2- 172.21.15.1:0/765165138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5268152670 0x7f5268156ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f526e297640 1 -- 172.21.15.1:0/765165138 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52681ea9d0 con 0x7f52680c3970 2024-03-06T07:23:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f526e297640 1 --2- 172.21.15.1:0/765165138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52680c3970 0x7f526806fe40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f526400ebe0 tx=0x7f526400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f525effd640 1 -- 172.21.15.1:0/765165138 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5264017070 con 0x7f52680c3970 2024-03-06T07:23:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f525effd640 1 -- 172.21.15.1:0/765165138 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526400cc30 con 0x7f52680c3970 2024-03-06T07:23:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f525effd640 1 -- 172.21.15.1:0/765165138 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5264013470 con 0x7f52680c3970 2024-03-06T07:23:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f52704fb640 1 -- 172.21.15.1:0/765165138 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52681eacb0 con 0x7f52680c3970 2024-03-06T07:23:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.104+0000 7f52704fb640 1 -- 172.21.15.1:0/765165138 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52681eb270 con 0x7f52680c3970 2024-03-06T07:23:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.106+0000 7f52704fb640 1 -- 172.21.15.1:0/765165138 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52680686e0 con 0x7f52680c3970 2024-03-06T07:23:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.106+0000 7f525effd640 1 -- 172.21.15.1:0/765165138 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f526401a070 con 0x7f52680c3970 2024-03-06T07:23:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.106+0000 7f525effd640 1 --2- 172.21.15.1:0/765165138 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f524c06b500 0x7f524c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.106+0000 7f525effd640 1 -- 172.21.15.1:0/765165138 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f526408a5e0 con 0x7f52680c3970 2024-03-06T07:23:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.106+0000 7f526da96640 1 --2- 172.21.15.1:0/765165138 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f524c06b500 0x7f524c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.107+0000 7f526da96640 1 --2- 172.21.15.1:0/765165138 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f524c06b500 0x7f524c06d9c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5258036000 tx=0x7f5258008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.111+0000 7f525effd640 1 -- 172.21.15.1:0/765165138 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52640588c0 con 0x7f52680c3970 2024-03-06T07:23:19.143 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-06T07:23:19.141+0000 7f12f8071640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:19.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.329+0000 7f52704fb640 1 -- 172.21.15.1:0/765165138 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52680aab70 con 0x7f52680c3970 2024-03-06T07:23:19.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.330+0000 7f525effd640 1 -- 172.21.15.1:0/765165138 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6251 (secure 0 0 0) 0x7f526405c960 con 0x7f52680c3970 2024-03-06T07:23:19.333 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:19.333 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":66,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.1:6835/2685225325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2685225325},{"type":"v1","addr":"172.21.15.1:6835","nonce":2685225325}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.98:6836/4069609682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4069609682},{"type":"v1","addr":"172.21.15.98:6836","nonce":4069609682}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":66,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.969135+0000","modified":"2024-03-06T07:23:17.152713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.1:6834/3919123103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3919123103},{"type":"v1","addr":"172.21.15.1:6834","nonce":3919123103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6684,"qdb_cluster":[6684]},"id":5},{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:07.074540+0000","modified":"2024-03-06T07:23:10.095697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5845":{"gid":5845,"name":"d","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/114998998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":114998998},{"type":"v1","addr":"172.21.15.98:6837","nonce":114998998}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5845,"qdb_cluster":[5845]},"id":6}]} 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 66 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.332+0000 7f52704fb640 1 -- 172.21.15.1:0/765165138 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f524c06b500 msgr2=0x7f524c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.332+0000 7f52704fb640 1 --2- 172.21.15.1:0/765165138 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f524c06b500 0x7f524c06d9c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5258036000 tx=0x7f5258008030 comp rx=0 tx=0).stop 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.332+0000 7f52704fb640 1 -- 172.21.15.1:0/765165138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52680c3970 msgr2=0x7f526806fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.332+0000 7f52704fb640 1 --2- 172.21.15.1:0/765165138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52680c3970 0x7f526806fe40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f526400ebe0 tx=0x7f526400c520 comp rx=0 tx=0).stop 2024-03-06T07:23:19.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.332+0000 7f52704fb640 1 -- 172.21.15.1:0/765165138 shutdown_connections 2024-03-06T07:23:19.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.332+0000 7f52704fb640 1 --2- 172.21.15.1:0/765165138 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f524c06b500 0x7f524c06d9c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.332+0000 7f52704fb640 1 --2- 172.21.15.1:0/765165138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5268152670 0x7f5268156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.332+0000 7f52704fb640 1 --2- 172.21.15.1:0/765165138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52680c60b0 0x7f5268156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.332+0000 7f52704fb640 1 --2- 172.21.15.1:0/765165138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52680c3970 0x7f526806fe40 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.332+0000 7f52704fb640 1 -- 172.21.15.1:0/765165138 >> 172.21.15.1:0/765165138 conn(0x7f52680c88f0 msgr2=0x7f52680ab020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.332+0000 7f52704fb640 1 -- 172.21.15.1:0/765165138 shutdown_connections 2024-03-06T07:23:19.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.332+0000 7f52704fb640 1 -- 172.21.15.1:0/765165138 wait complete. 2024-03-06T07:23:19.345 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:23:19.345 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:19.345 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:19.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:19.437 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:19.437 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:23:19.438 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:19.453 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:23:19.454 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:23:19.454 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:19.469 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:23:19.469 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:19.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.592+0000 7f054eec3640 1 Processor -- start 2024-03-06T07:23:19.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.592+0000 7f054eec3640 1 -- start start 2024-03-06T07:23:19.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.593+0000 7f054eec3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0548152650 0x7f0548155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.594+0000 7f054eec3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05480c2fc0 con 0x7f05480bf9d0 2024-03-06T07:23:19.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.594+0000 7f054eec3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0548158da0 con 0x7f05480c6b10 2024-03-06T07:23:19.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.594+0000 7f054eec3640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0548155800 con 0x7f0548152650 2024-03-06T07:23:19.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.594+0000 7f054d460640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f05480bf9d0 0x7f05480c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47052/0 (socket says 172.21.15.1:47052) 2024-03-06T07:23:19.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.594+0000 7f054d460640 1 -- 172.21.15.1:0/1207178118 learned_addr learned my addr 172.21.15.1:0/1207178118 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:19.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.594+0000 7f054cc5f640 1 --2- 172.21.15.1:0/1207178118 >> v2:172.21.15.98:3300/0 conn(0x7f0548152650 0x7f0548155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.594+0000 7f053f7fe640 1 -- 172.21.15.1:0/1207178118 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873243119 0 0) 0x7f05480c2fc0 con 0x7f05480bf9d0 2024-03-06T07:23:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.594+0000 7f053f7fe640 1 -- 172.21.15.1:0/1207178118 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0530003520 con 0x7f05480bf9d0 2024-03-06T07:23:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.594+0000 7f054cc5f640 1 -- 172.21.15.1:0/1207178118 >> v1:172.21.15.98:6789/0 conn(0x7f05480c6b10 legacy=0x7f0548158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.594+0000 7f054cc5f640 1 -- 172.21.15.1:0/1207178118 >> v1:172.21.15.1:6789/0 conn(0x7f05480bf9d0 legacy=0x7f05480c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.594+0000 7f054cc5f640 1 -- 172.21.15.1:0/1207178118 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05480c6690 con 0x7f0548152650 2024-03-06T07:23:19.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.595+0000 7f054cc5f640 1 --2- 172.21.15.1:0/1207178118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0548152650 0x7f0548155c90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f05440097a0 tx=0x7f054402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3aaed9e8f00df7d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.595+0000 7f053f7fe640 1 -- 172.21.15.1:0/1207178118 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0544035020 con 0x7f0548152650 2024-03-06T07:23:19.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.595+0000 7f053f7fe640 1 -- 172.21.15.1:0/1207178118 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0544031830 con 0x7f0548152650 2024-03-06T07:23:19.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.595+0000 7f053f7fe640 1 -- 172.21.15.1:0/1207178118 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0544031c00 con 0x7f0548152650 2024-03-06T07:23:19.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.596+0000 7f054eec3640 1 -- 172.21.15.1:0/1207178118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0548152650 msgr2=0x7f0548155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.596+0000 7f054eec3640 1 --2- 172.21.15.1:0/1207178118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0548152650 0x7f0548155c90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f05440097a0 tx=0x7f054402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.596+0000 7f054eec3640 1 -- 172.21.15.1:0/1207178118 shutdown_connections 2024-03-06T07:23:19.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.596+0000 7f054eec3640 1 --2- 172.21.15.1:0/1207178118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0548152650 0x7f0548155c90 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.596+0000 7f054eec3640 1 -- 172.21.15.1:0/1207178118 >> 172.21.15.1:0/1207178118 conn(0x7f0548066ee0 msgr2=0x7f05480672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.596+0000 7f054eec3640 1 -- 172.21.15.1:0/1207178118 shutdown_connections 2024-03-06T07:23:19.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.596+0000 7f054eec3640 1 -- 172.21.15.1:0/1207178118 wait complete. 2024-03-06T07:23:19.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.597+0000 7f054eec3640 1 Processor -- start 2024-03-06T07:23:19.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.597+0000 7f054eec3640 1 -- start start 2024-03-06T07:23:19.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f054eec3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05480bf9d0 0x7f05480c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f054eec3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0548152650 0x7f05481533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f054eec3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05481eab60 0x7f05481538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f054eec3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05481551b0 con 0x7f0548152650 2024-03-06T07:23:19.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f054eec3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05480c6fa0 con 0x7f05480bf9d0 2024-03-06T07:23:19.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f054eec3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0548155490 con 0x7f05481eab60 2024-03-06T07:23:19.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f053ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0548152650 0x7f05481533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f053ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0548152650 0x7f05481533a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49600/0 (socket says 172.21.15.1:49600) 2024-03-06T07:23:19.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f053ffff640 1 -- 172.21.15.1:0/3291105253 learned_addr learned my addr 172.21.15.1:0/3291105253 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:19.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f054cc5f640 1 --2- 172.21.15.1:0/3291105253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05480bf9d0 0x7f05480c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f054d460640 1 --2- 172.21.15.1:0/3291105253 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05481eab60 0x7f05481538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f053ffff640 1 -- 172.21.15.1:0/3291105253 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05481eab60 msgr2=0x7f05481538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f053ffff640 1 --2- 172.21.15.1:0/3291105253 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05481eab60 0x7f05481538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f053ffff640 1 -- 172.21.15.1:0/3291105253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05480bf9d0 msgr2=0x7f05480c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f053ffff640 1 --2- 172.21.15.1:0/3291105253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05480bf9d0 0x7f05480c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f053ffff640 1 -- 172.21.15.1:0/3291105253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05481eb0b0 con 0x7f0548152650 2024-03-06T07:23:19.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f054cc5f640 1 --2- 172.21.15.1:0/3291105253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05480bf9d0 0x7f05480c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:19.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f053ffff640 1 --2- 172.21.15.1:0/3291105253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0548152650 0x7f05481533a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f053800c400 tx=0x7f053800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f053d7fa640 1 -- 172.21.15.1:0/3291105253 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05380052d0 con 0x7f0548152650 2024-03-06T07:23:19.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f053d7fa640 1 -- 172.21.15.1:0/3291105253 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0538005490 con 0x7f0548152650 2024-03-06T07:23:19.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f054eec3640 1 -- 172.21.15.1:0/3291105253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05481eb390 con 0x7f0548152650 2024-03-06T07:23:19.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f053d7fa640 1 -- 172.21.15.1:0/3291105253 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0538007260 con 0x7f0548152650 2024-03-06T07:23:19.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.598+0000 7f054eec3640 1 -- 172.21.15.1:0/3291105253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05481eb950 con 0x7f0548152650 2024-03-06T07:23:19.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.599+0000 7f054eec3640 1 -- 172.21.15.1:0/3291105253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05480c4e90 con 0x7f0548152650 2024-03-06T07:23:19.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.600+0000 7f053d7fa640 1 -- 172.21.15.1:0/3291105253 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0538007400 con 0x7f0548152650 2024-03-06T07:23:19.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.601+0000 7f053d7fa640 1 --2- 172.21.15.1:0/3291105253 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f053406b4d0 0x7f053406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.601+0000 7f053d7fa640 1 -- 172.21.15.1:0/3291105253 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f0538089b80 con 0x7f0548152650 2024-03-06T07:23:19.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.601+0000 7f054cc5f640 1 --2- 172.21.15.1:0/3291105253 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f053406b4d0 0x7f053406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.601+0000 7f054cc5f640 1 --2- 172.21.15.1:0/3291105253 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f053406b4d0 0x7f053406d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0544036000 tx=0x7f0544008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.605+0000 7f053d7fa640 1 -- 172.21.15.1:0/3291105253 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05480c4e90 con 0x7f0548152650 2024-03-06T07:23:19.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.825+0000 7f054eec3640 1 -- 172.21.15.1:0/3291105253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0548067fd0 con 0x7f0548152650 2024-03-06T07:23:19.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.826+0000 7f053d7fa640 1 -- 172.21.15.1:0/3291105253 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6251 (secure 0 0 0) 0x7f0538057f00 con 0x7f0548152650 2024-03-06T07:23:19.828 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:19.828 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":66,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.1:6835/2685225325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2685225325},{"type":"v1","addr":"172.21.15.1:6835","nonce":2685225325}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.98:6836/4069609682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4069609682},{"type":"v1","addr":"172.21.15.98:6836","nonce":4069609682}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":66,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.969135+0000","modified":"2024-03-06T07:23:17.152713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.1:6834/3919123103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3919123103},{"type":"v1","addr":"172.21.15.1:6834","nonce":3919123103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6684,"qdb_cluster":[6684]},"id":5},{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:07.074540+0000","modified":"2024-03-06T07:23:10.095697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5845":{"gid":5845,"name":"d","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/114998998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":114998998},{"type":"v1","addr":"172.21.15.98:6837","nonce":114998998}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5845,"qdb_cluster":[5845]},"id":6}]} 2024-03-06T07:23:19.828 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 66 2024-03-06T07:23:19.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.828+0000 7f054eec3640 1 -- 172.21.15.1:0/3291105253 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f053406b4d0 msgr2=0x7f053406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.828+0000 7f054eec3640 1 --2- 172.21.15.1:0/3291105253 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f053406b4d0 0x7f053406d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0544036000 tx=0x7f0544008030 comp rx=0 tx=0).stop 2024-03-06T07:23:19.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.828+0000 7f054eec3640 1 -- 172.21.15.1:0/3291105253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0548152650 msgr2=0x7f05481533a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.828+0000 7f054eec3640 1 --2- 172.21.15.1:0/3291105253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0548152650 0x7f05481533a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f053800c400 tx=0x7f053800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.828+0000 7f054eec3640 1 -- 172.21.15.1:0/3291105253 shutdown_connections 2024-03-06T07:23:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.828+0000 7f054eec3640 1 --2- 172.21.15.1:0/3291105253 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f053406b4d0 0x7f053406d990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.828+0000 7f054eec3640 1 --2- 172.21.15.1:0/3291105253 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05481eab60 0x7f05481538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.828+0000 7f054eec3640 1 --2- 172.21.15.1:0/3291105253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0548152650 0x7f05481533a0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.828+0000 7f054eec3640 1 --2- 172.21.15.1:0/3291105253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05480bf9d0 0x7f05480c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.828+0000 7f054eec3640 1 -- 172.21.15.1:0/3291105253 >> 172.21.15.1:0/3291105253 conn(0x7f0548066ee0 msgr2=0x7f05480c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.828+0000 7f054eec3640 1 -- 172.21.15.1:0/3291105253 shutdown_connections 2024-03-06T07:23:19.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.828+0000 7f054eec3640 1 -- 172.21.15.1:0/3291105253 wait complete. 2024-03-06T07:23:19.841 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:19.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.944+0000 7efc2254b640 1 Processor -- start 2024-03-06T07:23:19.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.944+0000 7efc2254b640 1 -- start start 2024-03-06T07:23:19.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.945+0000 7efc2254b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc1c0c05f0 0x7efc1c0c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.945+0000 7efc2254b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efc1c06f640 0x7efc1c1551b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.945+0000 7efc2254b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1c0c31f0 con 0x7efc1c0c14c0 2024-03-06T07:23:19.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.945+0000 7efc2254b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc1c06fe90 con 0x7efc1c06f640 2024-03-06T07:23:19.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.945+0000 7efc2254b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c0c2fc0 con 0x7efc1c0c05f0 2024-03-06T07:23:19.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.945+0000 7efc1bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc1c0c05f0 0x7efc1c0c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.945+0000 7efc1b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efc1c06f640 0x7efc1c1551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.945+0000 7efc20ae8640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efc1c0c14c0 0x7efc1c1556f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:55086/0 (socket says 172.21.15.1:55086) 2024-03-06T07:23:19.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.945+0000 7efc1bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc1c0c05f0 0x7efc1c0c09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:57422/0 (socket says 172.21.15.1:57422) 2024-03-06T07:23:19.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.945+0000 7efc20ae8640 1 -- 172.21.15.1:0/3165320539 learned_addr learned my addr 172.21.15.1:0/3165320539 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:19.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.945+0000 7efc1bfff640 1 -- 172.21.15.1:0/3165320539 >> v2:172.21.15.98:3300/0 conn(0x7efc1c06f640 msgr2=0x7efc1c1551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:19.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.945+0000 7efc1bfff640 1 --2- 172.21.15.1:0/3165320539 >> v2:172.21.15.98:3300/0 conn(0x7efc1c06f640 0x7efc1c1551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.945+0000 7efc1bfff640 1 -- 172.21.15.1:0/3165320539 >> v1:172.21.15.98:6789/0 conn(0x7efc1c0c14c0 legacy=0x7efc1c1556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.945+0000 7efc1bfff640 1 -- 172.21.15.1:0/3165320539 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c155e00 con 0x7efc1c0c05f0 2024-03-06T07:23:19.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.946+0000 7efc1bfff640 1 --2- 172.21.15.1:0/3165320539 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc1c0c05f0 0x7efc1c0c09d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efc100097a0 tx=0x7efc1002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65d5bd454fa4b5a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.946+0000 7efc1affd640 1 -- 172.21.15.1:0/3165320539 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc10035020 con 0x7efc1c0c05f0 2024-03-06T07:23:19.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.946+0000 7efc1affd640 1 -- 172.21.15.1:0/3165320539 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc10031830 con 0x7efc1c0c05f0 2024-03-06T07:23:19.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.946+0000 7efc1affd640 1 -- 172.21.15.1:0/3165320539 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc10031c00 con 0x7efc1c0c05f0 2024-03-06T07:23:19.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.947+0000 7efc2254b640 1 -- 172.21.15.1:0/3165320539 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc1c0c05f0 msgr2=0x7efc1c0c09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.947+0000 7efc2254b640 1 --2- 172.21.15.1:0/3165320539 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc1c0c05f0 0x7efc1c0c09d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efc100097a0 tx=0x7efc1002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.947+0000 7efc2254b640 1 -- 172.21.15.1:0/3165320539 shutdown_connections 2024-03-06T07:23:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.947+0000 7efc2254b640 1 --2- 172.21.15.1:0/3165320539 >> v2:172.21.15.98:3300/0 conn(0x7efc1c06f640 0x7efc1c1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.947+0000 7efc2254b640 1 --2- 172.21.15.1:0/3165320539 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc1c0c05f0 0x7efc1c0c09d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.947+0000 7efc2254b640 1 -- 172.21.15.1:0/3165320539 >> 172.21.15.1:0/3165320539 conn(0x7efc1c0c3cc0 msgr2=0x7efc1c0c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.947+0000 7efc2254b640 1 -- 172.21.15.1:0/3165320539 shutdown_connections 2024-03-06T07:23:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.947+0000 7efc2254b640 1 -- 172.21.15.1:0/3165320539 wait complete. 2024-03-06T07:23:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.948+0000 7efc2254b640 1 Processor -- start 2024-03-06T07:23:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.948+0000 7efc2254b640 1 -- start start 2024-03-06T07:23:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc2254b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc1c06f640 0x7efc1c154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc2254b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc1c0c05f0 0x7efc1c0c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc2254b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc1c0c14c0 0x7efc1c13f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc2254b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c0c3770 con 0x7efc1c0c05f0 2024-03-06T07:23:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc2254b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c0c3490 con 0x7efc1c06f640 2024-03-06T07:23:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc2254b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c0c1040 con 0x7efc1c0c14c0 2024-03-06T07:23:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc1b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc1c0c05f0 0x7efc1c0c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc1bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc1c06f640 0x7efc1c154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc20ae8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc1c0c14c0 0x7efc1c13f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc1bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc1c06f640 0x7efc1c154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:37368/0 (socket says 172.21.15.1:37368) 2024-03-06T07:23:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc1bfff640 1 -- 172.21.15.1:0/976751316 learned_addr learned my addr 172.21.15.1:0/976751316 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc20ae8640 1 -- 172.21.15.1:0/976751316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc1c06f640 msgr2=0x7efc1c154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc20ae8640 1 --2- 172.21.15.1:0/976751316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc1c06f640 0x7efc1c154170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc20ae8640 1 -- 172.21.15.1:0/976751316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc1c0c05f0 msgr2=0x7efc1c0c4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc20ae8640 1 --2- 172.21.15.1:0/976751316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc1c0c05f0 0x7efc1c0c4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc20ae8640 1 -- 172.21.15.1:0/976751316 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c1eb6d0 con 0x7efc1c0c14c0 2024-03-06T07:23:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc20ae8640 1 --2- 172.21.15.1:0/976751316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc1c0c14c0 0x7efc1c13f080 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efc1400caa0 tx=0x7efc1400cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.949+0000 7efc18ff9640 1 -- 172.21.15.1:0/976751316 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc14017070 con 0x7efc1c0c14c0 2024-03-06T07:23:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.950+0000 7efc18ff9640 1 -- 172.21.15.1:0/976751316 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc1400aba0 con 0x7efc1c0c14c0 2024-03-06T07:23:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.950+0000 7efc2254b640 1 -- 172.21.15.1:0/976751316 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1c14cc80 con 0x7efc1c0c14c0 2024-03-06T07:23:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.950+0000 7efc18ff9640 1 -- 172.21.15.1:0/976751316 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc14012400 con 0x7efc1c0c14c0 2024-03-06T07:23:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.950+0000 7efc2254b640 1 -- 172.21.15.1:0/976751316 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1c1eb950 con 0x7efc1c0c14c0 2024-03-06T07:23:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.950+0000 7efc2254b640 1 -- 172.21.15.1:0/976751316 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1c067fd0 con 0x7efc1c0c14c0 2024-03-06T07:23:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.951+0000 7efc18ff9640 1 -- 172.21.15.1:0/976751316 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc1401a070 con 0x7efc1c0c14c0 2024-03-06T07:23:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.951+0000 7efc18ff9640 1 --2- 172.21.15.1:0/976751316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc0806b500 0x7efc0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.952+0000 7efc18ff9640 1 -- 172.21.15.1:0/976751316 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7efc1408a1f0 con 0x7efc1c0c14c0 2024-03-06T07:23:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.952+0000 7efc1bfff640 1 --2- 172.21.15.1:0/976751316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc0806b500 0x7efc0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.952+0000 7efc1bfff640 1 --2- 172.21.15.1:0/976751316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc0806b500 0x7efc0806d9c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7efc1002b910 tx=0x7efc10008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:19.956+0000 7efc18ff9640 1 -- 172.21.15.1:0/976751316 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc1c1eb6d0 con 0x7efc1c0c14c0 2024-03-06T07:23:20.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.138+0000 7efc2254b640 1 -- 172.21.15.1:0/976751316 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc1c0ac750 con 0x7efc1c0c14c0 2024-03-06T07:23:20.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.139+0000 7efc18ff9640 1 -- 172.21.15.1:0/976751316 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+20676 (secure 0 0 0) 0x7efc14058550 con 0x7efc1c0c14c0 2024-03-06T07:23:20.142 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:20.142 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":79,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:23:13.133419+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:55.240857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:55.939811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:23:05.599046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:23:06.205539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6834/1494116991":"2024-03-07T07:23:13.133394+0000","172.21.15.1:6832/1494116991":"2024-03-07T07:23:13.133394+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:20.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.142+0000 7efc2254b640 1 -- 172.21.15.1:0/976751316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc0806b500 msgr2=0x7efc0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.142+0000 7efc2254b640 1 --2- 172.21.15.1:0/976751316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc0806b500 0x7efc0806d9c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7efc1002b910 tx=0x7efc10008030 comp rx=0 tx=0).stop 2024-03-06T07:23:20.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.142+0000 7efc2254b640 1 -- 172.21.15.1:0/976751316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc1c0c14c0 msgr2=0x7efc1c13f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.142+0000 7efc2254b640 1 --2- 172.21.15.1:0/976751316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc1c0c14c0 0x7efc1c13f080 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efc1400caa0 tx=0x7efc1400cf60 comp rx=0 tx=0).stop 2024-03-06T07:23:20.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.142+0000 7efc2254b640 1 -- 172.21.15.1:0/976751316 shutdown_connections 2024-03-06T07:23:20.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.142+0000 7efc2254b640 1 --2- 172.21.15.1:0/976751316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc0806b500 0x7efc0806d9c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.142+0000 7efc2254b640 1 --2- 172.21.15.1:0/976751316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc1c0c14c0 0x7efc1c13f080 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.142+0000 7efc2254b640 1 --2- 172.21.15.1:0/976751316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc1c0c05f0 0x7efc1c0c4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.142+0000 7efc2254b640 1 --2- 172.21.15.1:0/976751316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc1c06f640 0x7efc1c154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.142+0000 7efc2254b640 1 -- 172.21.15.1:0/976751316 >> 172.21.15.1:0/976751316 conn(0x7efc1c0c3cc0 msgr2=0x7efc1c14a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.142+0000 7efc2254b640 1 -- 172.21.15.1:0/976751316 shutdown_connections 2024-03-06T07:23:20.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.142+0000 7efc2254b640 1 -- 172.21.15.1:0/976751316 wait complete. 2024-03-06T07:23:20.155 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:55.240857+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:55.939811+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-06T07:23:05.599046+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-06T07:23:06.205539+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:20.156 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:20.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.258+0000 7f338bd9a640 1 Processor -- start 2024-03-06T07:23:20.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.258+0000 7f338bd9a640 1 -- start start 2024-03-06T07:23:20.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.259+0000 7f338bd9a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f33840c32f0 0x7f33841543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.259+0000 7f338bd9a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3384150900 con 0x7f3384150520 2024-03-06T07:23:20.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.259+0000 7f338bd9a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3384071790 con 0x7f3384070c80 2024-03-06T07:23:20.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.259+0000 7f338bd9a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3384153ed0 con 0x7f33840c32f0 2024-03-06T07:23:20.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.259+0000 7f3389335640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f33840c32f0 0x7f33841543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.259+0000 7f3389b36640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3384150520 0x7f33841537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47064/0 (socket says 172.21.15.1:47064) 2024-03-06T07:23:20.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.259+0000 7f3389b36640 1 -- 172.21.15.1:0/2444970259 learned_addr learned my addr 172.21.15.1:0/2444970259 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:20.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.260+0000 7f3389335640 1 -- 172.21.15.1:0/2444970259 >> v1:172.21.15.98:6789/0 conn(0x7f3384070c80 legacy=0x7f338415d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.260+0000 7f3389335640 1 -- 172.21.15.1:0/2444970259 >> v1:172.21.15.1:6789/0 conn(0x7f3384150520 legacy=0x7f33841537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.260+0000 7f3388b34640 1 -- 172.21.15.1:0/2444970259 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498497042 0 0) 0x7f3384150900 con 0x7f3384150520 2024-03-06T07:23:20.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.260+0000 7f3389335640 1 -- 172.21.15.1:0/2444970259 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3384157050 con 0x7f33840c32f0 2024-03-06T07:23:20.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.260+0000 7f3389335640 1 --2- 172.21.15.1:0/2444970259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33840c32f0 0x7f33841543a0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f3374009720 tx=0x7f337402b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=386eb54c76c74acc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.260+0000 7f3388b34640 1 -- 172.21.15.1:0/2444970259 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3374037070 con 0x7f33840c32f0 2024-03-06T07:23:20.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.261+0000 7f3388b34640 1 -- 172.21.15.1:0/2444970259 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3374030710 con 0x7f33840c32f0 2024-03-06T07:23:20.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.261+0000 7f3388b34640 1 -- 172.21.15.1:0/2444970259 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3374030ae0 con 0x7f33840c32f0 2024-03-06T07:23:20.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.261+0000 7f338bd9a640 1 -- 172.21.15.1:0/2444970259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33840c32f0 msgr2=0x7f33841543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.261+0000 7f338bd9a640 1 --2- 172.21.15.1:0/2444970259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33840c32f0 0x7f33841543a0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f3374009720 tx=0x7f337402b730 comp rx=0 tx=0).stop 2024-03-06T07:23:20.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.261+0000 7f338bd9a640 1 -- 172.21.15.1:0/2444970259 shutdown_connections 2024-03-06T07:23:20.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.261+0000 7f338bd9a640 1 --2- 172.21.15.1:0/2444970259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33840c32f0 0x7f33841543a0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.261+0000 7f338bd9a640 1 -- 172.21.15.1:0/2444970259 >> 172.21.15.1:0/2444970259 conn(0x7f33840c88f0 msgr2=0x7f33840c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.261+0000 7f338bd9a640 1 -- 172.21.15.1:0/2444970259 shutdown_connections 2024-03-06T07:23:20.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.261+0000 7f338bd9a640 1 -- 172.21.15.1:0/2444970259 wait complete. 2024-03-06T07:23:20.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.262+0000 7f338bd9a640 1 Processor -- start 2024-03-06T07:23:20.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.262+0000 7f338bd9a640 1 -- start start 2024-03-06T07:23:20.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.263+0000 7f338bd9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3384070c80 0x7f33841e8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.263+0000 7f338bd9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33840c32f0 0x7f33841565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.263+0000 7f338bd9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3384150520 0x7f3384154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.263+0000 7f338bd9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3384157330 con 0x7f3384070c80 2024-03-06T07:23:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.263+0000 7f338bd9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33841532a0 con 0x7f3384150520 2024-03-06T07:23:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.263+0000 7f338bd9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33840c7ec0 con 0x7f33840c32f0 2024-03-06T07:23:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.263+0000 7f3389b36640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3384070c80 0x7f33841e8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.263+0000 7f3389b36640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3384070c80 0x7f33841e8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49640/0 (socket says 172.21.15.1:49640) 2024-03-06T07:23:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.263+0000 7f3389b36640 1 -- 172.21.15.1:0/90522616 learned_addr learned my addr 172.21.15.1:0/90522616 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.263+0000 7f3389335640 1 --2- 172.21.15.1:0/90522616 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33840c32f0 0x7f33841565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.263+0000 7f338a337640 1 --2- 172.21.15.1:0/90522616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3384150520 0x7f3384154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.263+0000 7f3389b36640 1 -- 172.21.15.1:0/90522616 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33840c32f0 msgr2=0x7f33841565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.263+0000 7f3389b36640 1 --2- 172.21.15.1:0/90522616 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33840c32f0 0x7f33841565c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.263+0000 7f3389b36640 1 -- 172.21.15.1:0/90522616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3384150520 msgr2=0x7f3384154fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.263+0000 7f3389b36640 1 --2- 172.21.15.1:0/90522616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3384150520 0x7f3384154fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.263+0000 7f3389b36640 1 -- 172.21.15.1:0/90522616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3384146a70 con 0x7f3384070c80 2024-03-06T07:23:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.264+0000 7f3389b36640 1 --2- 172.21.15.1:0/90522616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3384070c80 0x7f33841e8aa0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f338000c470 tx=0x7f338000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.264+0000 7f337a7fc640 1 -- 172.21.15.1:0/90522616 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3380018070 con 0x7f3384070c80 2024-03-06T07:23:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.264+0000 7f337a7fc640 1 -- 172.21.15.1:0/90522616 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33800074e0 con 0x7f3384070c80 2024-03-06T07:23:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.264+0000 7f338bd9a640 1 -- 172.21.15.1:0/90522616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33841e91a0 con 0x7f3384070c80 2024-03-06T07:23:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.264+0000 7f337a7fc640 1 -- 172.21.15.1:0/90522616 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3380004a20 con 0x7f3384070c80 2024-03-06T07:23:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.264+0000 7f338bd9a640 1 -- 172.21.15.1:0/90522616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33841e9760 con 0x7f3384070c80 2024-03-06T07:23:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.265+0000 7f338bd9a640 1 -- 172.21.15.1:0/90522616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33840686e0 con 0x7f3384070c80 2024-03-06T07:23:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.265+0000 7f337a7fc640 1 -- 172.21.15.1:0/90522616 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f338001b070 con 0x7f3384070c80 2024-03-06T07:23:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.266+0000 7f337a7fc640 1 --2- 172.21.15.1:0/90522616 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f336806b4d0 0x7f336806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.266+0000 7f337a7fc640 1 -- 172.21.15.1:0/90522616 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f3380089860 con 0x7f3384070c80 2024-03-06T07:23:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.266+0000 7f3389335640 1 --2- 172.21.15.1:0/90522616 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f336806b4d0 0x7f336806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.266+0000 7f3389335640 1 --2- 172.21.15.1:0/90522616 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f336806b4d0 0x7f336806d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3374034000 tx=0x7f3374008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.274+0000 7f337a7fc640 1 -- 172.21.15.1:0/90522616 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3380057be0 con 0x7f3384070c80 2024-03-06T07:23:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.555+0000 7f338bd9a640 1 -- 172.21.15.1:0/90522616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f33840c5c60 con 0x7f3384070c80 2024-03-06T07:23:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.555+0000 7f337a7fc640 1 -- 172.21.15.1:0/90522616 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f338005bc80 con 0x7f3384070c80 2024-03-06T07:23:20.557 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:23:20.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.558+0000 7f338bd9a640 1 -- 172.21.15.1:0/90522616 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f336806b4d0 msgr2=0x7f336806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.558+0000 7f338bd9a640 1 --2- 172.21.15.1:0/90522616 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f336806b4d0 0x7f336806d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3374034000 tx=0x7f3374008030 comp rx=0 tx=0).stop 2024-03-06T07:23:20.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.558+0000 7f338bd9a640 1 -- 172.21.15.1:0/90522616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3384070c80 msgr2=0x7f33841e8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.558+0000 7f338bd9a640 1 --2- 172.21.15.1:0/90522616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3384070c80 0x7f33841e8aa0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f338000c470 tx=0x7f338000c930 comp rx=0 tx=0).stop 2024-03-06T07:23:20.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.558+0000 7f338bd9a640 1 -- 172.21.15.1:0/90522616 shutdown_connections 2024-03-06T07:23:20.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.558+0000 7f338bd9a640 1 --2- 172.21.15.1:0/90522616 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f336806b4d0 0x7f336806d990 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.558+0000 7f338bd9a640 1 --2- 172.21.15.1:0/90522616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3384150520 0x7f3384154fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.558+0000 7f338bd9a640 1 --2- 172.21.15.1:0/90522616 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33840c32f0 0x7f33841565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.558+0000 7f338bd9a640 1 --2- 172.21.15.1:0/90522616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3384070c80 0x7f33841e8aa0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.558+0000 7f338bd9a640 1 -- 172.21.15.1:0/90522616 >> 172.21.15.1:0/90522616 conn(0x7f33840c88f0 msgr2=0x7f338414c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.558+0000 7f338bd9a640 1 -- 172.21.15.1:0/90522616 shutdown_connections 2024-03-06T07:23:20.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.558+0000 7f338bd9a640 1 -- 172.21.15.1:0/90522616 wait complete. 2024-03-06T07:23:20.571 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:20.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.702+0000 7fda30bf7640 1 Processor -- start 2024-03-06T07:23:20.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.702+0000 7fda30bf7640 1 -- start start 2024-03-06T07:23:20.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.703+0000 7fda30bf7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda2c0bf9d0 0x7fda2c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.703+0000 7fda30bf7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2c0c2fc0 con 0x7fda2c152650 2024-03-06T07:23:20.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.703+0000 7fda30bf7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2c158da0 con 0x7fda2c0c6b10 2024-03-06T07:23:20.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.703+0000 7fda30bf7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c155800 con 0x7fda2c0bf9d0 2024-03-06T07:23:20.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.703+0000 7fda2a59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fda2c152650 0x7fda2c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47070/0 (socket says 172.21.15.1:47070) 2024-03-06T07:23:20.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.703+0000 7fda2a59c640 1 -- 172.21.15.1:0/3172654793 learned_addr learned my addr 172.21.15.1:0/3172654793 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:20.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.703+0000 7fda2ad9d640 1 --2- 172.21.15.1:0/3172654793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda2c0bf9d0 0x7fda2c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.704+0000 7fda2959a640 1 -- 172.21.15.1:0/3172654793 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410010739 0 0) 0x7fda2c0c2fc0 con 0x7fda2c152650 2024-03-06T07:23:20.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.704+0000 7fda2959a640 1 -- 172.21.15.1:0/3172654793 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda14003520 con 0x7fda2c152650 2024-03-06T07:23:20.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.704+0000 7fda2959a640 1 -- 172.21.15.1:0/3172654793 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966718641 0 0) 0x7fda2c158da0 con 0x7fda2c0c6b10 2024-03-06T07:23:20.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.704+0000 7fda2959a640 1 -- 172.21.15.1:0/3172654793 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda2c0c2fc0 con 0x7fda2c0c6b10 2024-03-06T07:23:20.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.704+0000 7fda2ad9d640 1 -- 172.21.15.1:0/3172654793 >> v1:172.21.15.98:6789/0 conn(0x7fda2c0c6b10 legacy=0x7fda2c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.704+0000 7fda2ad9d640 1 -- 172.21.15.1:0/3172654793 >> v1:172.21.15.1:6789/0 conn(0x7fda2c152650 legacy=0x7fda2c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.704+0000 7fda2ad9d640 1 -- 172.21.15.1:0/3172654793 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2c0c6690 con 0x7fda2c0bf9d0 2024-03-06T07:23:20.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.704+0000 7fda2ad9d640 1 --2- 172.21.15.1:0/3172654793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda2c0bf9d0 0x7fda2c155c90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fda1c009dc0 tx=0x7fda1c02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9fa4abc569adbf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.705+0000 7fda2959a640 1 -- 172.21.15.1:0/3172654793 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c039070 con 0x7fda2c0bf9d0 2024-03-06T07:23:20.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.705+0000 7fda2959a640 1 -- 172.21.15.1:0/3172654793 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c02cc40 con 0x7fda2c0bf9d0 2024-03-06T07:23:20.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.705+0000 7fda2959a640 1 -- 172.21.15.1:0/3172654793 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c034440 con 0x7fda2c0bf9d0 2024-03-06T07:23:20.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.705+0000 7fda30bf7640 1 -- 172.21.15.1:0/3172654793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda2c0bf9d0 msgr2=0x7fda2c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.705+0000 7fda30bf7640 1 --2- 172.21.15.1:0/3172654793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda2c0bf9d0 0x7fda2c155c90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fda1c009dc0 tx=0x7fda1c02be90 comp rx=0 tx=0).stop 2024-03-06T07:23:20.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.705+0000 7fda30bf7640 1 -- 172.21.15.1:0/3172654793 shutdown_connections 2024-03-06T07:23:20.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.705+0000 7fda30bf7640 1 --2- 172.21.15.1:0/3172654793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda2c0bf9d0 0x7fda2c155c90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.705+0000 7fda30bf7640 1 -- 172.21.15.1:0/3172654793 >> 172.21.15.1:0/3172654793 conn(0x7fda2c066ee0 msgr2=0x7fda2c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.706+0000 7fda30bf7640 1 -- 172.21.15.1:0/3172654793 shutdown_connections 2024-03-06T07:23:20.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.706+0000 7fda30bf7640 1 -- 172.21.15.1:0/3172654793 wait complete. 2024-03-06T07:23:20.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.707+0000 7fda30bf7640 1 Processor -- start 2024-03-06T07:23:20.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.707+0000 7fda30bf7640 1 -- start start 2024-03-06T07:23:20.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.708+0000 7fda30bf7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda2c0bf9d0 0x7fda2c153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.708+0000 7fda30bf7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda2c152650 0x7fda2c158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.708+0000 7fda30bf7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda2c150720 0x7fda2c1ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.708+0000 7fda30bf7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c1551b0 con 0x7fda2c152650 2024-03-06T07:23:20.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.708+0000 7fda30bf7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c0c6fa0 con 0x7fda2c150720 2024-03-06T07:23:20.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.708+0000 7fda30bf7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c155490 con 0x7fda2c0bf9d0 2024-03-06T07:23:20.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.709+0000 7fda29d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda2c152650 0x7fda2c158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.709+0000 7fda29d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda2c152650 0x7fda2c158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49646/0 (socket says 172.21.15.1:49646) 2024-03-06T07:23:20.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.709+0000 7fda29d9b640 1 -- 172.21.15.1:0/1794152221 learned_addr learned my addr 172.21.15.1:0/1794152221 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:20.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.709+0000 7fda2a59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda2c0bf9d0 0x7fda2c153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.709+0000 7fda2ad9d640 1 --2- 172.21.15.1:0/1794152221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda2c150720 0x7fda2c1ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.709+0000 7fda29d9b640 1 -- 172.21.15.1:0/1794152221 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda2c0bf9d0 msgr2=0x7fda2c153680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.709+0000 7fda29d9b640 1 --2- 172.21.15.1:0/1794152221 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda2c0bf9d0 0x7fda2c153680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.709+0000 7fda29d9b640 1 -- 172.21.15.1:0/1794152221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda2c150720 msgr2=0x7fda2c1ee590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.709+0000 7fda29d9b640 1 --2- 172.21.15.1:0/1794152221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda2c150720 0x7fda2c1ee590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.709+0000 7fda29d9b640 1 -- 172.21.15.1:0/1794152221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2c156490 con 0x7fda2c152650 2024-03-06T07:23:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.709+0000 7fda2a59c640 1 --2- 172.21.15.1:0/1794152221 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda2c0bf9d0 0x7fda2c153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.709+0000 7fda29d9b640 1 --2- 172.21.15.1:0/1794152221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda2c152650 0x7fda2c158350 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fda2000c530 tx=0x7fda2000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.709+0000 7fda1affd640 1 -- 172.21.15.1:0/1794152221 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda200052d0 con 0x7fda2c152650 2024-03-06T07:23:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.709+0000 7fda1affd640 1 -- 172.21.15.1:0/1794152221 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda20005490 con 0x7fda2c152650 2024-03-06T07:23:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.709+0000 7fda1affd640 1 -- 172.21.15.1:0/1794152221 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda20012870 con 0x7fda2c152650 2024-03-06T07:23:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.709+0000 7fda30bf7640 1 -- 172.21.15.1:0/1794152221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda2c156770 con 0x7fda2c152650 2024-03-06T07:23:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.709+0000 7fda30bf7640 1 -- 172.21.15.1:0/1794152221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda2c156d30 con 0x7fda2c152650 2024-03-06T07:23:20.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.710+0000 7fda30bf7640 1 -- 172.21.15.1:0/1794152221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2c067fd0 con 0x7fda2c152650 2024-03-06T07:23:20.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.711+0000 7fda1affd640 1 -- 172.21.15.1:0/1794152221 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda20012a10 con 0x7fda2c152650 2024-03-06T07:23:20.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.712+0000 7fda1affd640 1 --2- 172.21.15.1:0/1794152221 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda0c06b4d0 0x7fda0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.712+0000 7fda1affd640 1 -- 172.21.15.1:0/1794152221 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fda2008a240 con 0x7fda2c152650 2024-03-06T07:23:20.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.712+0000 7fda2a59c640 1 --2- 172.21.15.1:0/1794152221 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda0c06b4d0 0x7fda0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.712+0000 7fda2a59c640 1 --2- 172.21.15.1:0/1794152221 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda0c06b4d0 0x7fda0c06d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fda24000f60 tx=0x7fda24009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.716+0000 7fda1affd640 1 -- 172.21.15.1:0/1794152221 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda20058540 con 0x7fda2c152650 2024-03-06T07:23:20.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.898+0000 7fda30bf7640 1 -- 172.21.15.1:0/1794152221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fda2c0c39a0 con 0x7fda2c152650 2024-03-06T07:23:20.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.899+0000 7fda1affd640 1 -- 172.21.15.1:0/1794152221 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v79) v1 ==== 104+0+11 (secure 0 0 0) 0x7fda2005c5e0 con 0x7fda2c152650 2024-03-06T07:23:20.901 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:23:20.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.901+0000 7fda30bf7640 1 -- 172.21.15.1:0/1794152221 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda0c06b4d0 msgr2=0x7fda0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.901+0000 7fda30bf7640 1 --2- 172.21.15.1:0/1794152221 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda0c06b4d0 0x7fda0c06d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fda24000f60 tx=0x7fda24009040 comp rx=0 tx=0).stop 2024-03-06T07:23:20.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.901+0000 7fda30bf7640 1 -- 172.21.15.1:0/1794152221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda2c152650 msgr2=0x7fda2c158350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.901+0000 7fda30bf7640 1 --2- 172.21.15.1:0/1794152221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda2c152650 0x7fda2c158350 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fda2000c530 tx=0x7fda2000c9f0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.901+0000 7fda30bf7640 1 -- 172.21.15.1:0/1794152221 shutdown_connections 2024-03-06T07:23:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.901+0000 7fda30bf7640 1 --2- 172.21.15.1:0/1794152221 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda0c06b4d0 0x7fda0c06d990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.901+0000 7fda30bf7640 1 --2- 172.21.15.1:0/1794152221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda2c150720 0x7fda2c1ee590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.901+0000 7fda30bf7640 1 --2- 172.21.15.1:0/1794152221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda2c152650 0x7fda2c158350 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.901+0000 7fda30bf7640 1 --2- 172.21.15.1:0/1794152221 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda2c0bf9d0 0x7fda2c153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.901+0000 7fda30bf7640 1 -- 172.21.15.1:0/1794152221 >> 172.21.15.1:0/1794152221 conn(0x7fda2c066ee0 msgr2=0x7fda2c0c0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.901+0000 7fda30bf7640 1 -- 172.21.15.1:0/1794152221 shutdown_connections 2024-03-06T07:23:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:20.901+0000 7fda30bf7640 1 -- 172.21.15.1:0/1794152221 wait complete. 2024-03-06T07:23:20.914 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:21.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.016+0000 7f063c5ee640 1 Processor -- start 2024-03-06T07:23:21.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.016+0000 7f063c5ee640 1 -- start start 2024-03-06T07:23:21.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.017+0000 7f063c5ee640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0634152650 0x7f0634155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.017+0000 7f063c5ee640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06340c2fc0 con 0x7f06340c6b10 2024-03-06T07:23:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.017+0000 7f063c5ee640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0634158da0 con 0x7f06340bf9d0 2024-03-06T07:23:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.017+0000 7f063c5ee640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0634155800 con 0x7f0634152650 2024-03-06T07:23:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.017+0000 7f0639b89640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f06340c6b10 0x7f0634158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47072/0 (socket says 172.21.15.1:47072) 2024-03-06T07:23:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.017+0000 7f0639b89640 1 -- 172.21.15.1:0/1244449044 learned_addr learned my addr 172.21.15.1:0/1244449044 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.017+0000 7f063a38a640 1 --2- 172.21.15.1:0/1244449044 >> v2:172.21.15.1:3300/0 conn(0x7f0634152650 0x7f0634155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.017+0000 7f0639388640 1 -- 172.21.15.1:0/1244449044 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073567109 0 0) 0x7f06340c2fc0 con 0x7f06340c6b10 2024-03-06T07:23:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.018+0000 7f0639388640 1 -- 172.21.15.1:0/1244449044 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0620003520 con 0x7f06340c6b10 2024-03-06T07:23:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.018+0000 7f0639388640 1 -- 172.21.15.1:0/1244449044 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924081490 0 0) 0x7f0634158da0 con 0x7f06340bf9d0 2024-03-06T07:23:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.018+0000 7f063a38a640 1 -- 172.21.15.1:0/1244449044 >> v1:172.21.15.98:6789/0 conn(0x7f06340bf9d0 legacy=0x7f06340c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.018+0000 7f063a38a640 1 -- 172.21.15.1:0/1244449044 >> v1:172.21.15.1:6789/0 conn(0x7f06340c6b10 legacy=0x7f0634158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.018+0000 7f063a38a640 1 -- 172.21.15.1:0/1244449044 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06340c6690 con 0x7f0634152650 2024-03-06T07:23:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.018+0000 7f063a38a640 1 --2- 172.21.15.1:0/1244449044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0634152650 0x7f0634155c90 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f06300097a0 tx=0x7f063002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd20b444e7e10d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.018+0000 7f0639388640 1 -- 172.21.15.1:0/1244449044 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0630035020 con 0x7f0634152650 2024-03-06T07:23:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.018+0000 7f0639388640 1 -- 172.21.15.1:0/1244449044 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0630031830 con 0x7f0634152650 2024-03-06T07:23:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.018+0000 7f0639388640 1 -- 172.21.15.1:0/1244449044 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0630031be0 con 0x7f0634152650 2024-03-06T07:23:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.019+0000 7f063c5ee640 1 -- 172.21.15.1:0/1244449044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0634152650 msgr2=0x7f0634155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.019+0000 7f063c5ee640 1 --2- 172.21.15.1:0/1244449044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0634152650 0x7f0634155c90 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f06300097a0 tx=0x7f063002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.019+0000 7f063c5ee640 1 -- 172.21.15.1:0/1244449044 shutdown_connections 2024-03-06T07:23:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.019+0000 7f063c5ee640 1 --2- 172.21.15.1:0/1244449044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0634152650 0x7f0634155c90 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.019+0000 7f063c5ee640 1 -- 172.21.15.1:0/1244449044 >> 172.21.15.1:0/1244449044 conn(0x7f0634066ee0 msgr2=0x7f06340672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.019+0000 7f063c5ee640 1 -- 172.21.15.1:0/1244449044 shutdown_connections 2024-03-06T07:23:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.019+0000 7f063c5ee640 1 -- 172.21.15.1:0/1244449044 wait complete. 2024-03-06T07:23:21.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.020+0000 7f063c5ee640 1 Processor -- start 2024-03-06T07:23:21.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.020+0000 7f063c5ee640 1 -- start start 2024-03-06T07:23:21.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f063c5ee640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06340bf9d0 0x7f063414b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f063c5ee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0634152650 0x7f06340c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f063c5ee640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06340c3ec0 0x7f06341eaa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f063c5ee640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06341551b0 con 0x7f06340c3ec0 2024-03-06T07:23:21.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f063c5ee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06340c6fa0 con 0x7f0634152650 2024-03-06T07:23:21.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f063c5ee640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0634155490 con 0x7f06340bf9d0 2024-03-06T07:23:21.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f063ab8b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06340c3ec0 0x7f06341eaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f063ab8b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06340c3ec0 0x7f06341eaa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49674/0 (socket says 172.21.15.1:49674) 2024-03-06T07:23:21.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f063ab8b640 1 -- 172.21.15.1:0/2772117624 learned_addr learned my addr 172.21.15.1:0/2772117624 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:21.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f063a38a640 1 --2- 172.21.15.1:0/2772117624 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06340bf9d0 0x7f063414b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f0639b89640 1 --2- 172.21.15.1:0/2772117624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0634152650 0x7f06340c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f063ab8b640 1 -- 172.21.15.1:0/2772117624 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06340bf9d0 msgr2=0x7f063414b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f063ab8b640 1 --2- 172.21.15.1:0/2772117624 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06340bf9d0 0x7f063414b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f063ab8b640 1 -- 172.21.15.1:0/2772117624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0634152650 msgr2=0x7f06340c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f063ab8b640 1 --2- 172.21.15.1:0/2772117624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0634152650 0x7f06340c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f063ab8b640 1 -- 172.21.15.1:0/2772117624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06340ca5f0 con 0x7f06340c3ec0 2024-03-06T07:23:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f063a38a640 1 --2- 172.21.15.1:0/2772117624 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06340bf9d0 0x7f063414b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f0639b89640 1 --2- 172.21.15.1:0/2772117624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0634152650 0x7f06340c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.021+0000 7f063ab8b640 1 --2- 172.21.15.1:0/2772117624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06340c3ec0 0x7f06341eaa40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f062c00c780 tx=0x7f062c00cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.022+0000 7f0626ffd640 1 -- 172.21.15.1:0/2772117624 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c003b70 con 0x7f06340c3ec0 2024-03-06T07:23:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.022+0000 7f0626ffd640 1 -- 172.21.15.1:0/2772117624 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062c003d10 con 0x7f06340c3ec0 2024-03-06T07:23:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.022+0000 7f063c5ee640 1 -- 172.21.15.1:0/2772117624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06341ee1c0 con 0x7f06340c3ec0 2024-03-06T07:23:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.022+0000 7f0626ffd640 1 -- 172.21.15.1:0/2772117624 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c005270 con 0x7f06340c3ec0 2024-03-06T07:23:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.022+0000 7f063c5ee640 1 -- 172.21.15.1:0/2772117624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06340c4340 con 0x7f06340c3ec0 2024-03-06T07:23:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.023+0000 7f063c5ee640 1 -- 172.21.15.1:0/2772117624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06341edfb0 con 0x7f06340c3ec0 2024-03-06T07:23:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.023+0000 7f0626ffd640 1 -- 172.21.15.1:0/2772117624 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f062c005410 con 0x7f06340c3ec0 2024-03-06T07:23:21.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.023+0000 7f0626ffd640 1 --2- 172.21.15.1:0/2772117624 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f061806b4d0 0x7f061806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.023+0000 7f0626ffd640 1 -- 172.21.15.1:0/2772117624 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f062c08a440 con 0x7f06340c3ec0 2024-03-06T07:23:21.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.024+0000 7f063a38a640 1 --2- 172.21.15.1:0/2772117624 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f061806b4d0 0x7f061806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.024+0000 7f063a38a640 1 --2- 172.21.15.1:0/2772117624 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f061806b4d0 0x7f061806d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0630036000 tx=0x7f0630008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.028+0000 7f0626ffd640 1 -- 172.21.15.1:0/2772117624 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f062c0587c0 con 0x7f06340c3ec0 2024-03-06T07:23:21.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.208+0000 7f063c5ee640 1 -- 172.21.15.1:0/2772117624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0634067fd0 con 0x7f06340c3ec0 2024-03-06T07:23:21.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.208+0000 7f0626ffd640 1 -- 172.21.15.1:0/2772117624 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v79) v1 ==== 100+0+11 (secure 0 0 0) 0x7f062c05c860 con 0x7f06340c3ec0 2024-03-06T07:23:21.210 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:23:21.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.210+0000 7f063c5ee640 1 -- 172.21.15.1:0/2772117624 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f061806b4d0 msgr2=0x7f061806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.210+0000 7f063c5ee640 1 --2- 172.21.15.1:0/2772117624 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f061806b4d0 0x7f061806d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0630036000 tx=0x7f0630008030 comp rx=0 tx=0).stop 2024-03-06T07:23:21.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.210+0000 7f063c5ee640 1 -- 172.21.15.1:0/2772117624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06340c3ec0 msgr2=0x7f06341eaa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.210+0000 7f063c5ee640 1 --2- 172.21.15.1:0/2772117624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06340c3ec0 0x7f06341eaa40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f062c00c780 tx=0x7f062c00cb80 comp rx=0 tx=0).stop 2024-03-06T07:23:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.210+0000 7f063c5ee640 1 -- 172.21.15.1:0/2772117624 shutdown_connections 2024-03-06T07:23:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.210+0000 7f063c5ee640 1 --2- 172.21.15.1:0/2772117624 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f061806b4d0 0x7f061806d990 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.210+0000 7f063c5ee640 1 --2- 172.21.15.1:0/2772117624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06340c3ec0 0x7f06341eaa40 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.210+0000 7f063c5ee640 1 --2- 172.21.15.1:0/2772117624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0634152650 0x7f06340c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.210+0000 7f063c5ee640 1 --2- 172.21.15.1:0/2772117624 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f06340bf9d0 0x7f063414b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.210+0000 7f063c5ee640 1 -- 172.21.15.1:0/2772117624 >> 172.21.15.1:0/2772117624 conn(0x7f0634066ee0 msgr2=0x7f0634151990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.210+0000 7f063c5ee640 1 -- 172.21.15.1:0/2772117624 shutdown_connections 2024-03-06T07:23:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.210+0000 7f063c5ee640 1 -- 172.21.15.1:0/2772117624 wait complete. 2024-03-06T07:23:21.226 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-06T07:23:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.328+0000 7f8ea3d5c640 1 Processor -- start 2024-03-06T07:23:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.328+0000 7f8ea3d5c640 1 -- start start 2024-03-06T07:23:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.329+0000 7f8ea3d5c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8e9c152820 0x7f8e9c0c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.330+0000 7f8ea3d5c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e9c0af6f0 0x7f8e9c15fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.330+0000 7f8ea3d5c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0c7180 con 0x7f8e9c0c0670 2024-03-06T07:23:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.330+0000 7f8ea3d5c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e9c155460 con 0x7f8e9c152820 2024-03-06T07:23:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.330+0000 7f8ea3d5c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c0c6bc0 con 0x7f8e9c0af6f0 2024-03-06T07:23:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.330+0000 7f8ea12f7640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8e9c0c0670 0x7f8e9c0c73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47080/0 (socket says 172.21.15.1:47080) 2024-03-06T07:23:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.330+0000 7f8ea12f7640 1 -- 172.21.15.1:0/2738285321 learned_addr learned my addr 172.21.15.1:0/2738285321 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.330+0000 7f8ea1af8640 1 --2- 172.21.15.1:0/2738285321 >> v2:172.21.15.98:3300/0 conn(0x7f8e9c152820 0x7f8e9c0c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.330+0000 7f8ea22f9640 1 --2- 172.21.15.1:0/2738285321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e9c0af6f0 0x7f8e9c15fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.330+0000 7f8ea0af6640 1 -- 172.21.15.1:0/2738285321 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624050910 0 0) 0x7f8e9c0c7180 con 0x7f8e9c0c0670 2024-03-06T07:23:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.330+0000 7f8ea0af6640 1 -- 172.21.15.1:0/2738285321 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e88003520 con 0x7f8e9c0c0670 2024-03-06T07:23:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.330+0000 7f8ea1af8640 1 -- 172.21.15.1:0/2738285321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e9c0af6f0 msgr2=0x7f8e9c15fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.330+0000 7f8ea1af8640 1 --2- 172.21.15.1:0/2738285321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e9c0af6f0 0x7f8e9c15fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.330+0000 7f8ea1af8640 1 -- 172.21.15.1:0/2738285321 >> v1:172.21.15.1:6789/0 conn(0x7f8e9c0c0670 legacy=0x7f8e9c0c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.330+0000 7f8ea1af8640 1 -- 172.21.15.1:0/2738285321 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c0c6ea0 con 0x7f8e9c152820 2024-03-06T07:23:21.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.330+0000 7f8ea22f9640 1 --2- 172.21.15.1:0/2738285321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e9c0af6f0 0x7f8e9c15fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:21.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.331+0000 7f8ea1af8640 1 --2- 172.21.15.1:0/2738285321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e9c152820 0x7f8e9c0c0130 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c0097a0 tx=0x7f8e8c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f8df3f762e1e6b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.331+0000 7f8ea0af6640 1 -- 172.21.15.1:0/2738285321 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c035020 con 0x7f8e9c152820 2024-03-06T07:23:21.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.331+0000 7f8ea0af6640 1 -- 172.21.15.1:0/2738285321 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e8c031830 con 0x7f8e9c152820 2024-03-06T07:23:21.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.331+0000 7f8ea0af6640 1 -- 172.21.15.1:0/2738285321 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c031be0 con 0x7f8e9c152820 2024-03-06T07:23:21.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.332+0000 7f8ea3d5c640 1 -- 172.21.15.1:0/2738285321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e9c152820 msgr2=0x7f8e9c0c0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.332+0000 7f8ea3d5c640 1 --2- 172.21.15.1:0/2738285321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e9c152820 0x7f8e9c0c0130 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c0097a0 tx=0x7f8e8c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.332+0000 7f8ea3d5c640 1 -- 172.21.15.1:0/2738285321 shutdown_connections 2024-03-06T07:23:21.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.332+0000 7f8ea3d5c640 1 --2- 172.21.15.1:0/2738285321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e9c0af6f0 0x7f8e9c15fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.332+0000 7f8ea3d5c640 1 --2- 172.21.15.1:0/2738285321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e9c152820 0x7f8e9c0c0130 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.332+0000 7f8ea3d5c640 1 -- 172.21.15.1:0/2738285321 >> 172.21.15.1:0/2738285321 conn(0x7f8e9c072720 msgr2=0x7f8e9c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.332+0000 7f8ea3d5c640 1 -- 172.21.15.1:0/2738285321 shutdown_connections 2024-03-06T07:23:21.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.332+0000 7f8ea3d5c640 1 -- 172.21.15.1:0/2738285321 wait complete. 2024-03-06T07:23:21.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.334+0000 7f8ea3d5c640 1 Processor -- start 2024-03-06T07:23:21.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.334+0000 7f8ea3d5c640 1 -- start start 2024-03-06T07:23:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea3d5c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e9c0af6f0 0x7f8e9c0c3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea3d5c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e9c0c0670 0x7f8e9c14b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea3d5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e9c152820 0x7f8e9c14b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea3d5c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c0c7c80 con 0x7f8e9c0af6f0 2024-03-06T07:23:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea3d5c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c0c7b00 con 0x7f8e9c152820 2024-03-06T07:23:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea3d5c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c152c00 con 0x7f8e9c0c0670 2024-03-06T07:23:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea1af8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e9c0af6f0 0x7f8e9c0c3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea1af8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e9c0af6f0 0x7f8e9c0c3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49684/0 (socket says 172.21.15.1:49684) 2024-03-06T07:23:21.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea1af8640 1 -- 172.21.15.1:0/4265105131 learned_addr learned my addr 172.21.15.1:0/4265105131 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea22f9640 1 --2- 172.21.15.1:0/4265105131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e9c152820 0x7f8e9c14b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea12f7640 1 --2- 172.21.15.1:0/4265105131 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e9c0c0670 0x7f8e9c14b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea1af8640 1 -- 172.21.15.1:0/4265105131 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e9c0c0670 msgr2=0x7f8e9c14b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea1af8640 1 --2- 172.21.15.1:0/4265105131 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e9c0c0670 0x7f8e9c14b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea1af8640 1 -- 172.21.15.1:0/4265105131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e9c152820 msgr2=0x7f8e9c14b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea1af8640 1 --2- 172.21.15.1:0/4265105131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e9c152820 0x7f8e9c14b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea1af8640 1 -- 172.21.15.1:0/4265105131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c1eec80 con 0x7f8e9c0af6f0 2024-03-06T07:23:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea22f9640 1 --2- 172.21.15.1:0/4265105131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e9c152820 0x7f8e9c14b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.335+0000 7f8ea1af8640 1 --2- 172.21.15.1:0/4265105131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e9c0af6f0 0x7f8e9c0c3920 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c02b910 tx=0x7f8e8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.336+0000 7f8e927fc640 1 -- 172.21.15.1:0/4265105131 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c040070 con 0x7f8e9c0af6f0 2024-03-06T07:23:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.336+0000 7f8e927fc640 1 -- 172.21.15.1:0/4265105131 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e8c03a910 con 0x7f8e9c0af6f0 2024-03-06T07:23:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.336+0000 7f8ea3d5c640 1 -- 172.21.15.1:0/4265105131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e9c06d850 con 0x7f8e9c0af6f0 2024-03-06T07:23:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.336+0000 7f8e927fc640 1 -- 172.21.15.1:0/4265105131 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c035040 con 0x7f8e9c0af6f0 2024-03-06T07:23:21.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.336+0000 7f8ea3d5c640 1 -- 172.21.15.1:0/4265105131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e9c1ee9a0 con 0x7f8e9c0af6f0 2024-03-06T07:23:21.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.337+0000 7f8ea3d5c640 1 -- 172.21.15.1:0/4265105131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e9c1ee790 con 0x7f8e9c0af6f0 2024-03-06T07:23:21.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.337+0000 7f8e927fc640 1 -- 172.21.15.1:0/4265105131 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e8c04b060 con 0x7f8e9c0af6f0 2024-03-06T07:23:21.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.337+0000 7f8e927fc640 1 --2- 172.21.15.1:0/4265105131 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8e8006b500 0x7f8e8006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.338+0000 7f8e927fc640 1 -- 172.21.15.1:0/4265105131 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f8e8c0abc90 con 0x7f8e9c0af6f0 2024-03-06T07:23:21.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.338+0000 7f8ea12f7640 1 --2- 172.21.15.1:0/4265105131 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8e8006b500 0x7f8e8006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.338+0000 7f8ea12f7640 1 --2- 172.21.15.1:0/4265105131 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8e8006b500 0x7f8e8006d9c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0c7460 tx=0x7f8e98009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.346+0000 7f8e927fc640 1 -- 172.21.15.1:0/4265105131 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e8c07a010 con 0x7f8e9c0af6f0 2024-03-06T07:23:21.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.528+0000 7f8ea3d5c640 1 -- 172.21.15.1:0/4265105131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8e9c0c5b40 con 0x7f8e9c0af6f0 2024-03-06T07:23:21.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.528+0000 7f8e927fc640 1 -- 172.21.15.1:0/4265105131 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v79) v1 ==== 105+0+11 (secure 0 0 0) 0x7f8e8c07e0b0 con 0x7f8e9c0af6f0 2024-03-06T07:23:21.530 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:23:21.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.530+0000 7f8ea3d5c640 1 -- 172.21.15.1:0/4265105131 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8e8006b500 msgr2=0x7f8e8006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.530+0000 7f8ea3d5c640 1 --2- 172.21.15.1:0/4265105131 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8e8006b500 0x7f8e8006d9c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0c7460 tx=0x7f8e98009040 comp rx=0 tx=0).stop 2024-03-06T07:23:21.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.530+0000 7f8ea3d5c640 1 -- 172.21.15.1:0/4265105131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e9c0af6f0 msgr2=0x7f8e9c0c3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.530+0000 7f8ea3d5c640 1 --2- 172.21.15.1:0/4265105131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e9c0af6f0 0x7f8e9c0c3920 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c02b910 tx=0x7f8e8c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.530+0000 7f8ea3d5c640 1 -- 172.21.15.1:0/4265105131 shutdown_connections 2024-03-06T07:23:21.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.530+0000 7f8ea3d5c640 1 --2- 172.21.15.1:0/4265105131 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8e8006b500 0x7f8e8006d9c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.530+0000 7f8ea3d5c640 1 --2- 172.21.15.1:0/4265105131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e9c152820 0x7f8e9c14b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.530+0000 7f8ea3d5c640 1 --2- 172.21.15.1:0/4265105131 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e9c0c0670 0x7f8e9c14b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.530+0000 7f8ea3d5c640 1 --2- 172.21.15.1:0/4265105131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e9c0af6f0 0x7f8e9c0c3920 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.530+0000 7f8ea3d5c640 1 -- 172.21.15.1:0/4265105131 >> 172.21.15.1:0/4265105131 conn(0x7f8e9c072720 msgr2=0x7f8e9c06cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.531+0000 7f8ea3d5c640 1 -- 172.21.15.1:0/4265105131 shutdown_connections 2024-03-06T07:23:21.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.531+0000 7f8ea3d5c640 1 -- 172.21.15.1:0/4265105131 wait complete. 2024-03-06T07:23:21.543 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-06T07:23:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.649+0000 7f2a7a3c7640 1 Processor -- start 2024-03-06T07:23:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.649+0000 7f2a7a3c7640 1 -- start start 2024-03-06T07:23:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.650+0000 7f2a7a3c7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740c3970 0x7f2a740c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.650+0000 7f2a7a3c7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a740c5e40 con 0x7f2a740c60b0 2024-03-06T07:23:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.650+0000 7f2a7a3c7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a740c6a40 con 0x7f2a74152840 2024-03-06T07:23:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.650+0000 7f2a7a3c7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a740c67f0 con 0x7f2a740c3970 2024-03-06T07:23:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.650+0000 7f2a737fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2a740c60b0 0x7f2a74152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47086/0 (socket says 172.21.15.1:47086) 2024-03-06T07:23:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.650+0000 7f2a737fe640 1 -- 172.21.15.1:0/1081512213 learned_addr learned my addr 172.21.15.1:0/1081512213 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.650+0000 7f2a73fff640 1 --2- 172.21.15.1:0/1081512213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740c3970 0x7f2a740c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.650+0000 7f2a72ffd640 1 -- 172.21.15.1:0/1081512213 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309752116 0 0) 0x7f2a740c5e40 con 0x7f2a740c60b0 2024-03-06T07:23:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.650+0000 7f2a72ffd640 1 -- 172.21.15.1:0/1081512213 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a5c003520 con 0x7f2a740c60b0 2024-03-06T07:23:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.650+0000 7f2a72ffd640 1 -- 172.21.15.1:0/1081512213 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590799169 0 0) 0x7f2a740c6a40 con 0x7f2a74152840 2024-03-06T07:23:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.650+0000 7f2a72ffd640 1 -- 172.21.15.1:0/1081512213 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a740c5e40 con 0x7f2a74152840 2024-03-06T07:23:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.650+0000 7f2a73fff640 1 -- 172.21.15.1:0/1081512213 >> v1:172.21.15.98:6789/0 conn(0x7f2a74152840 legacy=0x7f2a741553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.650+0000 7f2a73fff640 1 -- 172.21.15.1:0/1081512213 >> v1:172.21.15.1:6789/0 conn(0x7f2a740c60b0 legacy=0x7f2a74152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.650+0000 7f2a73fff640 1 -- 172.21.15.1:0/1081512213 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a740c6d20 con 0x7f2a740c3970 2024-03-06T07:23:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.651+0000 7f2a73fff640 1 --2- 172.21.15.1:0/1081512213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740c3970 0x7f2a740c8170 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2a640097d0 tx=0x7f2a6402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=908674b614993e66 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.651+0000 7f2a72ffd640 1 -- 172.21.15.1:0/1081512213 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64035020 con 0x7f2a740c3970 2024-03-06T07:23:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.651+0000 7f2a72ffd640 1 -- 172.21.15.1:0/1081512213 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a64031890 con 0x7f2a740c3970 2024-03-06T07:23:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.651+0000 7f2a72ffd640 1 -- 172.21.15.1:0/1081512213 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64031c40 con 0x7f2a740c3970 2024-03-06T07:23:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.652+0000 7f2a7a3c7640 1 -- 172.21.15.1:0/1081512213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740c3970 msgr2=0x7f2a740c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.652+0000 7f2a7a3c7640 1 --2- 172.21.15.1:0/1081512213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740c3970 0x7f2a740c8170 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2a640097d0 tx=0x7f2a6402b930 comp rx=0 tx=0).stop 2024-03-06T07:23:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.652+0000 7f2a7a3c7640 1 -- 172.21.15.1:0/1081512213 shutdown_connections 2024-03-06T07:23:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.652+0000 7f2a7a3c7640 1 --2- 172.21.15.1:0/1081512213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740c3970 0x7f2a740c8170 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.652+0000 7f2a7a3c7640 1 -- 172.21.15.1:0/1081512213 >> 172.21.15.1:0/1081512213 conn(0x7f2a740c88f0 msgr2=0x7f2a740c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.652+0000 7f2a7a3c7640 1 -- 172.21.15.1:0/1081512213 shutdown_connections 2024-03-06T07:23:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.652+0000 7f2a7a3c7640 1 -- 172.21.15.1:0/1081512213 wait complete. 2024-03-06T07:23:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.653+0000 7f2a7a3c7640 1 Processor -- start 2024-03-06T07:23:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.653+0000 7f2a7a3c7640 1 -- start start 2024-03-06T07:23:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a7a3c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a740c3970 0x7f2a7414f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a7a3c7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740c60b0 0x7f2a740a9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a7a3c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a74152840 0x7f2a740aa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a7a3c7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a740c75c0 con 0x7f2a740c3970 2024-03-06T07:23:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a7a3c7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a740c72e0 con 0x7f2a74152840 2024-03-06T07:23:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a7a3c7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a740c3d50 con 0x7f2a740c60b0 2024-03-06T07:23:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a73fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a740c3970 0x7f2a7414f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a73fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a740c3970 0x7f2a7414f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49698/0 (socket says 172.21.15.1:49698) 2024-03-06T07:23:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a73fff640 1 -- 172.21.15.1:0/3760089478 learned_addr learned my addr 172.21.15.1:0/3760089478 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a737fe640 1 --2- 172.21.15.1:0/3760089478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740c60b0 0x7f2a740a9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a78964640 1 --2- 172.21.15.1:0/3760089478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a74152840 0x7f2a740aa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a73fff640 1 -- 172.21.15.1:0/3760089478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740c60b0 msgr2=0x7f2a740a9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a73fff640 1 --2- 172.21.15.1:0/3760089478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740c60b0 0x7f2a740a9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a73fff640 1 -- 172.21.15.1:0/3760089478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a74152840 msgr2=0x7f2a740aa3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a73fff640 1 --2- 172.21.15.1:0/3760089478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a74152840 0x7f2a740aa3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a73fff640 1 -- 172.21.15.1:0/3760089478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a74157f10 con 0x7f2a740c3970 2024-03-06T07:23:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a73fff640 1 --2- 172.21.15.1:0/3760089478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a740c3970 0x7f2a7414f930 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f2a6402b970 tx=0x7f2a640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a70ff9640 1 -- 172.21.15.1:0/3760089478 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64035040 con 0x7f2a740c3970 2024-03-06T07:23:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a70ff9640 1 -- 172.21.15.1:0/3760089478 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a64003820 con 0x7f2a740c3970 2024-03-06T07:23:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a7a3c7640 1 -- 172.21.15.1:0/3760089478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a740c7ee0 con 0x7f2a740c3970 2024-03-06T07:23:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a70ff9640 1 -- 172.21.15.1:0/3760089478 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64003bd0 con 0x7f2a740c3970 2024-03-06T07:23:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.654+0000 7f2a7a3c7640 1 -- 172.21.15.1:0/3760089478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7414fe70 con 0x7f2a740c3970 2024-03-06T07:23:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.655+0000 7f2a7a3c7640 1 -- 172.21.15.1:0/3760089478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a740686e0 con 0x7f2a740c3970 2024-03-06T07:23:21.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.656+0000 7f2a70ff9640 1 -- 172.21.15.1:0/3760089478 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a64038020 con 0x7f2a740c3970 2024-03-06T07:23:21.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.656+0000 7f2a70ff9640 1 --2- 172.21.15.1:0/3760089478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a6006b500 0x7f2a6006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.657+0000 7f2a70ff9640 1 -- 172.21.15.1:0/3760089478 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f2a640acca0 con 0x7f2a740c3970 2024-03-06T07:23:21.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.657+0000 7f2a737fe640 1 --2- 172.21.15.1:0/3760089478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a6006b500 0x7f2a6006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.657+0000 7f2a737fe640 1 --2- 172.21.15.1:0/3760089478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a6006b500 0x7f2a6006d9c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2a74154fd0 tx=0x7f2a68007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.661+0000 7f2a70ff9640 1 -- 172.21.15.1:0/3760089478 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a6402d040 con 0x7f2a740c3970 2024-03-06T07:23:21.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.841+0000 7f2a7a3c7640 1 -- 172.21.15.1:0/3760089478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7f2a7406cf00 con 0x7f2a740c3970 2024-03-06T07:23:21.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.841+0000 7f2a70ff9640 1 -- 172.21.15.1:0/3760089478 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v79) v1 ==== 101+0+11 (secure 0 0 0) 0x7f2a6407af00 con 0x7f2a740c3970 2024-03-06T07:23:21.843 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:23:21.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.843+0000 7f2a7a3c7640 1 -- 172.21.15.1:0/3760089478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a6006b500 msgr2=0x7f2a6006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.843+0000 7f2a7a3c7640 1 --2- 172.21.15.1:0/3760089478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a6006b500 0x7f2a6006d9c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2a74154fd0 tx=0x7f2a68007d50 comp rx=0 tx=0).stop 2024-03-06T07:23:21.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.843+0000 7f2a7a3c7640 1 -- 172.21.15.1:0/3760089478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a740c3970 msgr2=0x7f2a7414f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.843+0000 7f2a7a3c7640 1 --2- 172.21.15.1:0/3760089478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a740c3970 0x7f2a7414f930 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f2a6402b970 tx=0x7f2a640046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.843+0000 7f2a7a3c7640 1 -- 172.21.15.1:0/3760089478 shutdown_connections 2024-03-06T07:23:21.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.843+0000 7f2a7a3c7640 1 --2- 172.21.15.1:0/3760089478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2a6006b500 0x7f2a6006d9c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.843+0000 7f2a7a3c7640 1 --2- 172.21.15.1:0/3760089478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a74152840 0x7f2a740aa3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.843+0000 7f2a7a3c7640 1 --2- 172.21.15.1:0/3760089478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2a740c60b0 0x7f2a740a9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.843+0000 7f2a7a3c7640 1 --2- 172.21.15.1:0/3760089478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a740c3970 0x7f2a7414f930 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.843+0000 7f2a7a3c7640 1 -- 172.21.15.1:0/3760089478 >> 172.21.15.1:0/3760089478 conn(0x7f2a740c88f0 msgr2=0x7f2a740c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.843+0000 7f2a7a3c7640 1 -- 172.21.15.1:0/3760089478 shutdown_connections 2024-03-06T07:23:21.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.843+0000 7f2a7a3c7640 1 -- 172.21.15.1:0/3760089478 wait complete. 2024-03-06T07:23:21.856 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:21.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.958+0000 7fd8eb4a4640 1 Processor -- start 2024-03-06T07:23:21.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.958+0000 7fd8eb4a4640 1 -- start start 2024-03-06T07:23:21.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.959+0000 7fd8eb4a4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8e4152650 0x7fd8e4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.959+0000 7fd8eb4a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e40bf9d0 0x7fd8e40c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.959+0000 7fd8eb4a4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e40c2fc0 con 0x7fd8e40c6b10 2024-03-06T07:23:21.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.959+0000 7fd8eb4a4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e4155800 con 0x7fd8e4152650 2024-03-06T07:23:21.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.959+0000 7fd8eb4a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e4158da0 con 0x7fd8e40bf9d0 2024-03-06T07:23:21.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.959+0000 7fd8e9240640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8e4152650 0x7fd8e4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.959+0000 7fd8e8a3f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd8e40c6b10 0x7fd8e4158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47102/0 (socket says 172.21.15.1:47102) 2024-03-06T07:23:21.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.959+0000 7fd8e8a3f640 1 -- 172.21.15.1:0/3176727856 learned_addr learned my addr 172.21.15.1:0/3176727856 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:21.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.959+0000 7fd8e9a41640 1 --2- 172.21.15.1:0/3176727856 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e40bf9d0 0x7fd8e40c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.959+0000 7fd8e9240640 1 -- 172.21.15.1:0/3176727856 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e40bf9d0 msgr2=0x7fd8e40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:21.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.959+0000 7fd8e9240640 1 --2- 172.21.15.1:0/3176727856 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e40bf9d0 0x7fd8e40c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.959+0000 7fd8e9240640 1 -- 172.21.15.1:0/3176727856 >> v1:172.21.15.1:6789/0 conn(0x7fd8e40c6b10 legacy=0x7fd8e4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.959+0000 7fd8e3fff640 1 -- 172.21.15.1:0/3176727856 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197243150 0 0) 0x7fd8e40c2fc0 con 0x7fd8e40c6b10 2024-03-06T07:23:21.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.959+0000 7fd8e9240640 1 -- 172.21.15.1:0/3176727856 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e40c6690 con 0x7fd8e4152650 2024-03-06T07:23:21.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.960+0000 7fd8e9240640 1 --2- 172.21.15.1:0/3176727856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e4152650 0x7fd8e4155c90 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc0097a0 tx=0x7fd8dc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f13611366476baa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.960+0000 7fd8e3fff640 1 -- 172.21.15.1:0/3176727856 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc0315a0 con 0x7fd8e4152650 2024-03-06T07:23:21.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.960+0000 7fd8e3fff640 1 -- 172.21.15.1:0/3176727856 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc037070 con 0x7fd8e4152650 2024-03-06T07:23:21.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.960+0000 7fd8e3fff640 1 -- 172.21.15.1:0/3176727856 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc031960 con 0x7fd8e4152650 2024-03-06T07:23:21.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.961+0000 7fd8eb4a4640 1 -- 172.21.15.1:0/3176727856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e4152650 msgr2=0x7fd8e4155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.961+0000 7fd8eb4a4640 1 --2- 172.21.15.1:0/3176727856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e4152650 0x7fd8e4155c90 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc0097a0 tx=0x7fd8dc02b840 comp rx=0 tx=0).stop 2024-03-06T07:23:21.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.961+0000 7fd8eb4a4640 1 -- 172.21.15.1:0/3176727856 shutdown_connections 2024-03-06T07:23:21.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.961+0000 7fd8eb4a4640 1 --2- 172.21.15.1:0/3176727856 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e40bf9d0 0x7fd8e40c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.961+0000 7fd8eb4a4640 1 --2- 172.21.15.1:0/3176727856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e4152650 0x7fd8e4155c90 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.961+0000 7fd8eb4a4640 1 -- 172.21.15.1:0/3176727856 >> 172.21.15.1:0/3176727856 conn(0x7fd8e4066ee0 msgr2=0x7fd8e40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.961+0000 7fd8eb4a4640 1 -- 172.21.15.1:0/3176727856 shutdown_connections 2024-03-06T07:23:21.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.961+0000 7fd8eb4a4640 1 -- 172.21.15.1:0/3176727856 wait complete. 2024-03-06T07:23:21.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.962+0000 7fd8eb4a4640 1 Processor -- start 2024-03-06T07:23:21.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.962+0000 7fd8eb4a4640 1 -- start start 2024-03-06T07:23:21.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8eb4a4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e40bf9d0 0x7fd8e414d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8eb4a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e4152650 0x7fd8e41584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8eb4a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e4151420 0x7fd8e414c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8eb4a4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e41551b0 con 0x7fd8e40bf9d0 2024-03-06T07:23:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8eb4a4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e40c6fa0 con 0x7fd8e4152650 2024-03-06T07:23:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8eb4a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e4155490 con 0x7fd8e4151420 2024-03-06T07:23:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8e9240640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e40bf9d0 0x7fd8e414d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8e9240640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e40bf9d0 0x7fd8e414d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49712/0 (socket says 172.21.15.1:49712) 2024-03-06T07:23:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8e9240640 1 -- 172.21.15.1:0/4063041543 learned_addr learned my addr 172.21.15.1:0/4063041543 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8e9a41640 1 --2- 172.21.15.1:0/4063041543 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e4151420 0x7fd8e414c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8e8a3f640 1 --2- 172.21.15.1:0/4063041543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e4152650 0x7fd8e41584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8e9240640 1 -- 172.21.15.1:0/4063041543 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e4151420 msgr2=0x7fd8e414c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8e9240640 1 --2- 172.21.15.1:0/4063041543 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e4151420 0x7fd8e414c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8e9240640 1 -- 172.21.15.1:0/4063041543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e4152650 msgr2=0x7fd8e41584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8e9240640 1 --2- 172.21.15.1:0/4063041543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e4152650 0x7fd8e41584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8e9240640 1 -- 172.21.15.1:0/4063041543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e41ead80 con 0x7fd8e40bf9d0 2024-03-06T07:23:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8e9240640 1 --2- 172.21.15.1:0/4063041543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e40bf9d0 0x7fd8e414d500 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc034040 tx=0x7fd8dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8e1ffb640 1 -- 172.21.15.1:0/4063041543 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc032630 con 0x7fd8e40bf9d0 2024-03-06T07:23:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8e1ffb640 1 -- 172.21.15.1:0/4063041543 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc0327f0 con 0x7fd8e40bf9d0 2024-03-06T07:23:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.963+0000 7fd8eb4a4640 1 -- 172.21.15.1:0/4063041543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e41eb060 con 0x7fd8e40bf9d0 2024-03-06T07:23:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.964+0000 7fd8e1ffb640 1 -- 172.21.15.1:0/4063041543 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc032be0 con 0x7fd8e40bf9d0 2024-03-06T07:23:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.964+0000 7fd8eb4a4640 1 -- 172.21.15.1:0/4063041543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e41eb620 con 0x7fd8e40bf9d0 2024-03-06T07:23:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.965+0000 7fd8eb4a4640 1 -- 172.21.15.1:0/4063041543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e4067fd0 con 0x7fd8e40bf9d0 2024-03-06T07:23:21.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.965+0000 7fd8e1ffb640 1 -- 172.21.15.1:0/4063041543 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8dc04e430 con 0x7fd8e40bf9d0 2024-03-06T07:23:21.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.965+0000 7fd8e1ffb640 1 --2- 172.21.15.1:0/4063041543 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8d006b4d0 0x7fd8d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.966+0000 7fd8e1ffb640 1 -- 172.21.15.1:0/4063041543 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fd8dc02f030 con 0x7fd8e40bf9d0 2024-03-06T07:23:21.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.966+0000 7fd8e8a3f640 1 --2- 172.21.15.1:0/4063041543 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8d006b4d0 0x7fd8d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.966+0000 7fd8e8a3f640 1 --2- 172.21.15.1:0/4063041543 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8d006b4d0 0x7fd8d006d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd8e414de90 tx=0x7fd8d8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:21.970+0000 7fd8e1ffb640 1 -- 172.21.15.1:0/4063041543 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8dc040070 con 0x7fd8e40bf9d0 2024-03-06T07:23:22.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.190+0000 7fd8eb4a4640 1 -- 172.21.15.1:0/4063041543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8e40b01a0 con 0x7fd8e40bf9d0 2024-03-06T07:23:22.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.191+0000 7fd8e1ffb640 1 -- 172.21.15.1:0/4063041543 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6251 (secure 0 0 0) 0x7fd8dc07a770 con 0x7fd8e40bf9d0 2024-03-06T07:23:22.193 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:22.193 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":66,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.1:6835/2685225325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2685225325},{"type":"v1","addr":"172.21.15.1:6835","nonce":2685225325}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.98:6836/4069609682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4069609682},{"type":"v1","addr":"172.21.15.98:6836","nonce":4069609682}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":66,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.969135+0000","modified":"2024-03-06T07:23:17.152713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.1:6834/3919123103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3919123103},{"type":"v1","addr":"172.21.15.1:6834","nonce":3919123103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6684,"qdb_cluster":[6684]},"id":5},{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:07.074540+0000","modified":"2024-03-06T07:23:10.095697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5845":{"gid":5845,"name":"d","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/114998998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":114998998},{"type":"v1","addr":"172.21.15.98:6837","nonce":114998998}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5845,"qdb_cluster":[5845]},"id":6}]} 2024-03-06T07:23:22.193 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 66 2024-03-06T07:23:22.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.192+0000 7fd8eb4a4640 1 -- 172.21.15.1:0/4063041543 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8d006b4d0 msgr2=0x7fd8d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.192+0000 7fd8eb4a4640 1 --2- 172.21.15.1:0/4063041543 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8d006b4d0 0x7fd8d006d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd8e414de90 tx=0x7fd8d8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:22.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.192+0000 7fd8eb4a4640 1 -- 172.21.15.1:0/4063041543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e40bf9d0 msgr2=0x7fd8e414d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.192+0000 7fd8eb4a4640 1 --2- 172.21.15.1:0/4063041543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e40bf9d0 0x7fd8e414d500 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc034040 tx=0x7fd8dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.193+0000 7fd8eb4a4640 1 -- 172.21.15.1:0/4063041543 shutdown_connections 2024-03-06T07:23:22.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.193+0000 7fd8eb4a4640 1 --2- 172.21.15.1:0/4063041543 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8d006b4d0 0x7fd8d006d990 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.193+0000 7fd8eb4a4640 1 --2- 172.21.15.1:0/4063041543 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e4151420 0x7fd8e414c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.193+0000 7fd8eb4a4640 1 --2- 172.21.15.1:0/4063041543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8e4152650 0x7fd8e41584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.193+0000 7fd8eb4a4640 1 --2- 172.21.15.1:0/4063041543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e40bf9d0 0x7fd8e414d500 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.193+0000 7fd8eb4a4640 1 -- 172.21.15.1:0/4063041543 >> 172.21.15.1:0/4063041543 conn(0x7fd8e4066ee0 msgr2=0x7fd8e40672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.193+0000 7fd8eb4a4640 1 -- 172.21.15.1:0/4063041543 shutdown_connections 2024-03-06T07:23:22.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.193+0000 7fd8eb4a4640 1 -- 172.21.15.1:0/4063041543 wait complete. 2024-03-06T07:23:22.205 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.304+0000 7f51cb284640 1 Processor -- start 2024-03-06T07:23:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.304+0000 7f51cb284640 1 -- start start 2024-03-06T07:23:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.305+0000 7f51cb284640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51c40c6b10 0x7f51c4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.305+0000 7f51cb284640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f51c40bf9d0 0x7f51c40c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.305+0000 7f51cb284640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c40c2fc0 con 0x7f51c4152650 2024-03-06T07:23:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.305+0000 7f51cb284640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c4155800 con 0x7f51c40c6b10 2024-03-06T07:23:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.305+0000 7f51cb284640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c4158da0 con 0x7f51c40bf9d0 2024-03-06T07:23:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.305+0000 7f51c881f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51c40c6b10 0x7f51c4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.305+0000 7f51c881f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51c40c6b10 0x7f51c4155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49728/0 (socket says 172.21.15.1:49728) 2024-03-06T07:23:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.305+0000 7f51c881f640 1 -- 172.21.15.1:0/1817613883 learned_addr learned my addr 172.21.15.1:0/1817613883 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.305+0000 7f51c9821640 1 --2- 172.21.15.1:0/1817613883 >> v2:172.21.15.98:3300/0 conn(0x7f51c40bf9d0 0x7f51c40c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.306+0000 7f51c881f640 1 -- 172.21.15.1:0/1817613883 >> v2:172.21.15.98:3300/0 conn(0x7f51c40bf9d0 msgr2=0x7f51c40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.306+0000 7f51c881f640 1 --2- 172.21.15.1:0/1817613883 >> v2:172.21.15.98:3300/0 conn(0x7f51c40bf9d0 0x7f51c40c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.306+0000 7f51c881f640 1 -- 172.21.15.1:0/1817613883 >> v1:172.21.15.98:6789/0 conn(0x7f51c4152650 legacy=0x7f51c4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.306+0000 7f51c881f640 1 -- 172.21.15.1:0/1817613883 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c40c6690 con 0x7f51c40c6b10 2024-03-06T07:23:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.306+0000 7f51c881f640 1 --2- 172.21.15.1:0/1817613883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c40c6b10 0x7f51c4155c90 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f51c41597b0 tx=0x7f51b402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebdada002ad4c810 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.306+0000 7f51bbfff640 1 -- 172.21.15.1:0/1817613883 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b4035020 con 0x7f51c40c6b10 2024-03-06T07:23:22.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.306+0000 7f51bbfff640 1 -- 172.21.15.1:0/1817613883 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b4031870 con 0x7f51c40c6b10 2024-03-06T07:23:22.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.307+0000 7f51bbfff640 1 -- 172.21.15.1:0/1817613883 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b4031c40 con 0x7f51c40c6b10 2024-03-06T07:23:22.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.307+0000 7f51cb284640 1 -- 172.21.15.1:0/1817613883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c40c6b10 msgr2=0x7f51c4155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.307+0000 7f51cb284640 1 --2- 172.21.15.1:0/1817613883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c40c6b10 0x7f51c4155c90 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f51c41597b0 tx=0x7f51b402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:22.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.307+0000 7f51cb284640 1 -- 172.21.15.1:0/1817613883 shutdown_connections 2024-03-06T07:23:22.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.307+0000 7f51cb284640 1 --2- 172.21.15.1:0/1817613883 >> v2:172.21.15.98:3300/0 conn(0x7f51c40bf9d0 0x7f51c40c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.307+0000 7f51cb284640 1 --2- 172.21.15.1:0/1817613883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c40c6b10 0x7f51c4155c90 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.307+0000 7f51cb284640 1 -- 172.21.15.1:0/1817613883 >> 172.21.15.1:0/1817613883 conn(0x7f51c4066ee0 msgr2=0x7f51c40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.307+0000 7f51cb284640 1 -- 172.21.15.1:0/1817613883 shutdown_connections 2024-03-06T07:23:22.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.307+0000 7f51cb284640 1 -- 172.21.15.1:0/1817613883 wait complete. 2024-03-06T07:23:22.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.308+0000 7f51cb284640 1 Processor -- start 2024-03-06T07:23:22.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.308+0000 7f51cb284640 1 -- start start 2024-03-06T07:23:22.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.309+0000 7f51cb284640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51c40bf9d0 0x7f51c4157460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.309+0000 7f51cb284640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c4152650 0x7f51c40c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.309+0000 7f51cb284640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51c4158660 0x7f51c41eea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.309+0000 7f51cb284640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c41551b0 con 0x7f51c4152650 2024-03-06T07:23:22.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.309+0000 7f51cb284640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c40c6fa0 con 0x7f51c4158660 2024-03-06T07:23:22.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.309+0000 7f51cb284640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c4155490 con 0x7f51c40bf9d0 2024-03-06T07:23:22.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.309+0000 7f51c881f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c4152650 0x7f51c40c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.309+0000 7f51c881f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c4152650 0x7f51c40c8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49740/0 (socket says 172.21.15.1:49740) 2024-03-06T07:23:22.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.309+0000 7f51c881f640 1 -- 172.21.15.1:0/4058599204 learned_addr learned my addr 172.21.15.1:0/4058599204 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:22.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.309+0000 7f51c9020640 1 --2- 172.21.15.1:0/4058599204 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51c40bf9d0 0x7f51c4157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.309+0000 7f51c9821640 1 --2- 172.21.15.1:0/4058599204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51c4158660 0x7f51c41eea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.309+0000 7f51c881f640 1 -- 172.21.15.1:0/4058599204 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51c40bf9d0 msgr2=0x7f51c4157460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.309+0000 7f51c881f640 1 --2- 172.21.15.1:0/4058599204 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51c40bf9d0 0x7f51c4157460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.309+0000 7f51c881f640 1 -- 172.21.15.1:0/4058599204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51c4158660 msgr2=0x7f51c41eea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.309+0000 7f51c881f640 1 --2- 172.21.15.1:0/4058599204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51c4158660 0x7f51c41eea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.309+0000 7f51c881f640 1 -- 172.21.15.1:0/4058599204 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c41579a0 con 0x7f51c4152650 2024-03-06T07:23:22.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.309+0000 7f51c881f640 1 --2- 172.21.15.1:0/4058599204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c4152650 0x7f51c40c8a30 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f51b402b950 tx=0x7f51b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.310+0000 7f51b9ffb640 1 -- 172.21.15.1:0/4058599204 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b4040070 con 0x7f51c4152650 2024-03-06T07:23:22.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.310+0000 7f51b9ffb640 1 -- 172.21.15.1:0/4058599204 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b4030d00 con 0x7f51c4152650 2024-03-06T07:23:22.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.310+0000 7f51cb284640 1 -- 172.21.15.1:0/4058599204 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c4153810 con 0x7f51c4152650 2024-03-06T07:23:22.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.310+0000 7f51b9ffb640 1 -- 172.21.15.1:0/4058599204 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b4035040 con 0x7f51c4152650 2024-03-06T07:23:22.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.310+0000 7f51cb284640 1 -- 172.21.15.1:0/4058599204 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c4153dd0 con 0x7f51c4152650 2024-03-06T07:23:22.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.311+0000 7f51cb284640 1 -- 172.21.15.1:0/4058599204 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c406b540 con 0x7f51c4152650 2024-03-06T07:23:22.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.311+0000 7f51b9ffb640 1 -- 172.21.15.1:0/4058599204 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f51b404b060 con 0x7f51c4152650 2024-03-06T07:23:22.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.312+0000 7f51b9ffb640 1 --2- 172.21.15.1:0/4058599204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51a806b4d0 0x7f51a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.312+0000 7f51b9ffb640 1 -- 172.21.15.1:0/4058599204 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f51b40ac280 con 0x7f51c4152650 2024-03-06T07:23:22.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.312+0000 7f51c9020640 1 --2- 172.21.15.1:0/4058599204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51a806b4d0 0x7f51a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.312+0000 7f51c9020640 1 --2- 172.21.15.1:0/4058599204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51a806b4d0 0x7f51a806d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f51c0001780 tx=0x7f51c0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.316+0000 7f51b9ffb640 1 -- 172.21.15.1:0/4058599204 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51c406b540 con 0x7f51c4152650 2024-03-06T07:23:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.501+0000 7f51cb284640 1 -- 172.21.15.1:0/4058599204 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51c40b06f0 con 0x7f51c4152650 2024-03-06T07:23:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.502+0000 7f51b9ffb640 1 -- 172.21.15.1:0/4058599204 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+20676 (secure 0 0 0) 0x7f51b407a600 con 0x7f51c4152650 2024-03-06T07:23:22.504 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:22.505 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":79,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:23:13.133419+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:55.240857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:55.939811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:23:05.599046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:23:06.205539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6834/1494116991":"2024-03-07T07:23:13.133394+0000","172.21.15.1:6832/1494116991":"2024-03-07T07:23:13.133394+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.504+0000 7f51cb284640 1 -- 172.21.15.1:0/4058599204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51a806b4d0 msgr2=0x7f51a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.504+0000 7f51cb284640 1 --2- 172.21.15.1:0/4058599204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51a806b4d0 0x7f51a806d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f51c0001780 tx=0x7f51c0009040 comp rx=0 tx=0).stop 2024-03-06T07:23:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.504+0000 7f51cb284640 1 -- 172.21.15.1:0/4058599204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c4152650 msgr2=0x7f51c40c8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.504+0000 7f51cb284640 1 --2- 172.21.15.1:0/4058599204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c4152650 0x7f51c40c8a30 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f51b402b950 tx=0x7f51b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.504+0000 7f51cb284640 1 -- 172.21.15.1:0/4058599204 shutdown_connections 2024-03-06T07:23:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.504+0000 7f51cb284640 1 --2- 172.21.15.1:0/4058599204 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51a806b4d0 0x7f51a806d990 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.504+0000 7f51cb284640 1 --2- 172.21.15.1:0/4058599204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51c4158660 0x7f51c41eea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.504+0000 7f51cb284640 1 --2- 172.21.15.1:0/4058599204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c4152650 0x7f51c40c8a30 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.504+0000 7f51cb284640 1 --2- 172.21.15.1:0/4058599204 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51c40bf9d0 0x7f51c4157460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.504+0000 7f51cb284640 1 -- 172.21.15.1:0/4058599204 >> 172.21.15.1:0/4058599204 conn(0x7f51c4066ee0 msgr2=0x7f51c40672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.505+0000 7f51cb284640 1 -- 172.21.15.1:0/4058599204 shutdown_connections 2024-03-06T07:23:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.505+0000 7f51cb284640 1 -- 172.21.15.1:0/4058599204 wait complete. 2024-03-06T07:23:22.517 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:23:22.518 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:22.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.617+0000 7f3677fa1640 1 Processor -- start 2024-03-06T07:23:22.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.617+0000 7f3677fa1640 1 -- start start 2024-03-06T07:23:22.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.617+0000 7f3677fa1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3670152650 0x7f3670155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.617+0000 7f3677fa1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36700bf9d0 0x7f36700c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.617+0000 7f3677fa1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36700c2fc0 con 0x7f36700c6b10 2024-03-06T07:23:22.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.617+0000 7f3677fa1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3670155800 con 0x7f36700bf9d0 2024-03-06T07:23:22.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.617+0000 7f3677fa1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3670158da0 con 0x7f3670152650 2024-03-06T07:23:22.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.618+0000 7f3675d3d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3670152650 0x7f3670155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.618+0000 7f367653e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36700bf9d0 0x7f36700c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.618+0000 7f367553c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f36700c6b10 0x7f3670158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:55128/0 (socket says 172.21.15.1:55128) 2024-03-06T07:23:22.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.618+0000 7f367553c640 1 -- 172.21.15.1:0/1940042972 learned_addr learned my addr 172.21.15.1:0/1940042972 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:22.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.618+0000 7f367653e640 1 -- 172.21.15.1:0/1940042972 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3670152650 msgr2=0x7f3670155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:22.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.618+0000 7f367653e640 1 --2- 172.21.15.1:0/1940042972 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3670152650 0x7f3670155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.618+0000 7f367653e640 1 -- 172.21.15.1:0/1940042972 >> v1:172.21.15.98:6789/0 conn(0x7f36700c6b10 legacy=0x7f3670158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.618+0000 7f367653e640 1 -- 172.21.15.1:0/1940042972 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36700c6690 con 0x7f36700bf9d0 2024-03-06T07:23:22.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.619+0000 7f367653e640 1 --2- 172.21.15.1:0/1940042972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36700bf9d0 0x7f36700c1460 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f36680097a0 tx=0x7f366802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e29c4dd97d1e95c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.619+0000 7f3674d3b640 1 -- 172.21.15.1:0/1940042972 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36680315a0 con 0x7f36700bf9d0 2024-03-06T07:23:22.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.619+0000 7f3674d3b640 1 -- 172.21.15.1:0/1940042972 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3668037070 con 0x7f36700bf9d0 2024-03-06T07:23:22.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.619+0000 7f3674d3b640 1 -- 172.21.15.1:0/1940042972 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668031960 con 0x7f36700bf9d0 2024-03-06T07:23:22.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.620+0000 7f3677fa1640 1 -- 172.21.15.1:0/1940042972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36700bf9d0 msgr2=0x7f36700c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.620+0000 7f3677fa1640 1 --2- 172.21.15.1:0/1940042972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36700bf9d0 0x7f36700c1460 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f36680097a0 tx=0x7f366802b840 comp rx=0 tx=0).stop 2024-03-06T07:23:22.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.620+0000 7f3677fa1640 1 -- 172.21.15.1:0/1940042972 shutdown_connections 2024-03-06T07:23:22.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.620+0000 7f3677fa1640 1 --2- 172.21.15.1:0/1940042972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36700bf9d0 0x7f36700c1460 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.620+0000 7f3677fa1640 1 --2- 172.21.15.1:0/1940042972 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3670152650 0x7f3670155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.620+0000 7f3677fa1640 1 -- 172.21.15.1:0/1940042972 >> 172.21.15.1:0/1940042972 conn(0x7f3670066ee0 msgr2=0x7f36700672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.620+0000 7f3677fa1640 1 -- 172.21.15.1:0/1940042972 shutdown_connections 2024-03-06T07:23:22.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.620+0000 7f3677fa1640 1 -- 172.21.15.1:0/1940042972 wait complete. 2024-03-06T07:23:22.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.621+0000 7f3677fa1640 1 Processor -- start 2024-03-06T07:23:22.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.621+0000 7f3677fa1640 1 -- start start 2024-03-06T07:23:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.622+0000 7f3677fa1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36700bf9d0 0x7f36700c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.622+0000 7f3677fa1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3670152650 0x7f367014d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.622+0000 7f3677fa1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3670157b90 0x7f36700c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.622+0000 7f3677fa1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36701551b0 con 0x7f36700bf9d0 2024-03-06T07:23:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f3677fa1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36700c6fa0 con 0x7f3670157b90 2024-03-06T07:23:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f3677fa1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3670155490 con 0x7f3670152650 2024-03-06T07:23:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f3675d3d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36700bf9d0 0x7f36700c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f3675d3d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36700bf9d0 0x7f36700c9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49746/0 (socket says 172.21.15.1:49746) 2024-03-06T07:23:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f3675d3d640 1 -- 172.21.15.1:0/3047406396 learned_addr learned my addr 172.21.15.1:0/3047406396 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f367653e640 1 --2- 172.21.15.1:0/3047406396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3670157b90 0x7f36700c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f367553c640 1 --2- 172.21.15.1:0/3047406396 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3670152650 0x7f367014d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f367653e640 1 -- 172.21.15.1:0/3047406396 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3670152650 msgr2=0x7f367014d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f367653e640 1 --2- 172.21.15.1:0/3047406396 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3670152650 0x7f367014d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f367653e640 1 -- 172.21.15.1:0/3047406396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36700bf9d0 msgr2=0x7f36700c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f367653e640 1 --2- 172.21.15.1:0/3047406396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36700bf9d0 0x7f36700c9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f367653e640 1 -- 172.21.15.1:0/3047406396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3670158690 con 0x7f3670157b90 2024-03-06T07:23:22.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f3675d3d640 1 --2- 172.21.15.1:0/3047406396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36700bf9d0 0x7f36700c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:22.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f367653e640 1 --2- 172.21.15.1:0/3047406396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3670157b90 0x7f36700c54e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3668033000 tx=0x7f36680046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f366e7fc640 1 -- 172.21.15.1:0/3047406396 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668032630 con 0x7f3670157b90 2024-03-06T07:23:22.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f366e7fc640 1 -- 172.21.15.1:0/3047406396 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36680327f0 con 0x7f3670157b90 2024-03-06T07:23:22.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f3677fa1640 1 -- 172.21.15.1:0/3047406396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3670158970 con 0x7f3670157b90 2024-03-06T07:23:22.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f366e7fc640 1 -- 172.21.15.1:0/3047406396 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668032be0 con 0x7f3670157b90 2024-03-06T07:23:22.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.623+0000 7f3677fa1640 1 -- 172.21.15.1:0/3047406396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36700c5a20 con 0x7f3670157b90 2024-03-06T07:23:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.624+0000 7f3677fa1640 1 -- 172.21.15.1:0/3047406396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3670067fd0 con 0x7f3670157b90 2024-03-06T07:23:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.625+0000 7f366e7fc640 1 -- 172.21.15.1:0/3047406396 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f366804e430 con 0x7f3670157b90 2024-03-06T07:23:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.625+0000 7f366e7fc640 1 --2- 172.21.15.1:0/3047406396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f365c06b4d0 0x7f365c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.625+0000 7f366e7fc640 1 -- 172.21.15.1:0/3047406396 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f366802f030 con 0x7f3670157b90 2024-03-06T07:23:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.625+0000 7f3675d3d640 1 --2- 172.21.15.1:0/3047406396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f365c06b4d0 0x7f365c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.626+0000 7f3675d3d640 1 --2- 172.21.15.1:0/3047406396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f365c06b4d0 0x7f365c06d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3660009a40 tx=0x7f366000b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.630+0000 7f366e7fc640 1 -- 172.21.15.1:0/3047406396 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3668040070 con 0x7f3670157b90 2024-03-06T07:23:22.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.869+0000 7f3677fa1640 1 -- 172.21.15.1:0/3047406396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3670158b40 con 0x7f3670157b90 2024-03-06T07:23:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.870+0000 7f366e7fc640 1 -- 172.21.15.1:0/3047406396 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6251 (secure 0 0 0) 0x7f366807a6f0 con 0x7f3670157b90 2024-03-06T07:23:22.872 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:22.872 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":66,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.1:6835/2685225325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2685225325},{"type":"v1","addr":"172.21.15.1:6835","nonce":2685225325}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.98:6836/4069609682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4069609682},{"type":"v1","addr":"172.21.15.98:6836","nonce":4069609682}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":66,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.969135+0000","modified":"2024-03-06T07:23:17.152713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.1:6834/3919123103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3919123103},{"type":"v1","addr":"172.21.15.1:6834","nonce":3919123103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6684,"qdb_cluster":[6684]},"id":5},{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:07.074540+0000","modified":"2024-03-06T07:23:10.095697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5845":{"gid":5845,"name":"d","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/114998998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":114998998},{"type":"v1","addr":"172.21.15.98:6837","nonce":114998998}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5845,"qdb_cluster":[5845]},"id":6}]} 2024-03-06T07:23:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 66 2024-03-06T07:23:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.872+0000 7f3677fa1640 1 -- 172.21.15.1:0/3047406396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f365c06b4d0 msgr2=0x7f365c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.872+0000 7f3677fa1640 1 --2- 172.21.15.1:0/3047406396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f365c06b4d0 0x7f365c06d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3660009a40 tx=0x7f366000b040 comp rx=0 tx=0).stop 2024-03-06T07:23:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.872+0000 7f3677fa1640 1 -- 172.21.15.1:0/3047406396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3670157b90 msgr2=0x7f36700c54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.872+0000 7f3677fa1640 1 --2- 172.21.15.1:0/3047406396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3670157b90 0x7f36700c54e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3668033000 tx=0x7f36680046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.872+0000 7f3677fa1640 1 -- 172.21.15.1:0/3047406396 shutdown_connections 2024-03-06T07:23:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.872+0000 7f3677fa1640 1 --2- 172.21.15.1:0/3047406396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f365c06b4d0 0x7f365c06d990 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.872+0000 7f3677fa1640 1 --2- 172.21.15.1:0/3047406396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3670157b90 0x7f36700c54e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.872+0000 7f3677fa1640 1 --2- 172.21.15.1:0/3047406396 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3670152650 0x7f367014d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.872+0000 7f3677fa1640 1 --2- 172.21.15.1:0/3047406396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36700bf9d0 0x7f36700c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.872+0000 7f3677fa1640 1 -- 172.21.15.1:0/3047406396 >> 172.21.15.1:0/3047406396 conn(0x7f3670066ee0 msgr2=0x7f36700cae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.872+0000 7f3677fa1640 1 -- 172.21.15.1:0/3047406396 shutdown_connections 2024-03-06T07:23:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.872+0000 7f3677fa1640 1 -- 172.21.15.1:0/3047406396 wait complete. 2024-03-06T07:23:22.888 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:22.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.988+0000 7f34d289a640 1 Processor -- start 2024-03-06T07:23:22.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.988+0000 7f34d289a640 1 -- start start 2024-03-06T07:23:22.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.988+0000 7f34d289a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f34cc0c2b30 0x7f34cc0724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.988+0000 7f34d289a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f34cc0a5fa0 0x7f34cc072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.988+0000 7f34d289a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc151dd0 con 0x7f34cc152b70 2024-03-06T07:23:22.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.988+0000 7f34d289a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34cc1526e0 con 0x7f34cc0a5fa0 2024-03-06T07:23:22.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.988+0000 7f34d289a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34cc156190 con 0x7f34cc0c2b30 2024-03-06T07:23:22.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.988+0000 7f34d0e37640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f34cc0a5fa0 0x7f34cc072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.989+0000 7f34d0e37640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f34cc0a5fa0 0x7f34cc072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49752/0 (socket says 172.21.15.1:49752) 2024-03-06T07:23:22.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.989+0000 7f34d0e37640 1 -- 172.21.15.1:0/2994664715 learned_addr learned my addr 172.21.15.1:0/2994664715 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:22.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.989+0000 7f34cbfff640 1 --2- 172.21.15.1:0/2994664715 >> v2:172.21.15.98:3300/0 conn(0x7f34cc0c2b30 0x7f34cc0724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.989+0000 7f34cbfff640 1 -- 172.21.15.1:0/2994664715 >> v2:172.21.15.1:3300/0 conn(0x7f34cc0a5fa0 msgr2=0x7f34cc072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:22.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.989+0000 7f34cbfff640 1 --2- 172.21.15.1:0/2994664715 >> v2:172.21.15.1:3300/0 conn(0x7f34cc0a5fa0 0x7f34cc072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.989+0000 7f34cbfff640 1 -- 172.21.15.1:0/2994664715 >> v1:172.21.15.98:6789/0 conn(0x7f34cc152b70 legacy=0x7f34cc0c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.989+0000 7f34cbfff640 1 -- 172.21.15.1:0/2994664715 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc152090 con 0x7f34cc0c2b30 2024-03-06T07:23:22.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.989+0000 7f34d0e37640 1 --2- 172.21.15.1:0/2994664715 >> v2:172.21.15.1:3300/0 conn(0x7f34cc0a5fa0 0x7f34cc072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:22.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.989+0000 7f34caffd640 1 -- 172.21.15.1:0/2994664715 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256785935 0 0) 0x7f34cc151dd0 con 0x7f34cc152b70 2024-03-06T07:23:22.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.990+0000 7f34cbfff640 1 --2- 172.21.15.1:0/2994664715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc0c2b30 0x7f34cc0724c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f34cc0c6570 tx=0x7f34bc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0f1b5ec55fa6526 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.990+0000 7f34caffd640 1 -- 172.21.15.1:0/2994664715 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc035020 con 0x7f34cc0c2b30 2024-03-06T07:23:22.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.990+0000 7f34caffd640 1 -- 172.21.15.1:0/2994664715 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34bc031830 con 0x7f34cc0c2b30 2024-03-06T07:23:22.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.990+0000 7f34caffd640 1 -- 172.21.15.1:0/2994664715 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc031c00 con 0x7f34cc0c2b30 2024-03-06T07:23:22.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.990+0000 7f34d289a640 1 -- 172.21.15.1:0/2994664715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc0c2b30 msgr2=0x7f34cc0724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.990+0000 7f34d289a640 1 --2- 172.21.15.1:0/2994664715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc0c2b30 0x7f34cc0724c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f34cc0c6570 tx=0x7f34bc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.990+0000 7f34d289a640 1 -- 172.21.15.1:0/2994664715 shutdown_connections 2024-03-06T07:23:22.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.990+0000 7f34d289a640 1 --2- 172.21.15.1:0/2994664715 >> v2:172.21.15.1:3300/0 conn(0x7f34cc0a5fa0 0x7f34cc072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.990+0000 7f34d289a640 1 --2- 172.21.15.1:0/2994664715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc0c2b30 0x7f34cc0724c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.990+0000 7f34d289a640 1 -- 172.21.15.1:0/2994664715 >> 172.21.15.1:0/2994664715 conn(0x7f34cc0c6860 msgr2=0x7f34cc0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.990+0000 7f34d289a640 1 -- 172.21.15.1:0/2994664715 shutdown_connections 2024-03-06T07:23:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.990+0000 7f34d289a640 1 -- 172.21.15.1:0/2994664715 wait complete. 2024-03-06T07:23:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.991+0000 7f34d289a640 1 Processor -- start 2024-03-06T07:23:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.992+0000 7f34d289a640 1 -- start start 2024-03-06T07:23:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.992+0000 7f34d289a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34cc0a5fa0 0x7f34cc1ebc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.992+0000 7f34d289a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34cc0c2b30 0x7f34cc1dfda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.992+0000 7f34d289a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc152b70 0x7f34cc0bff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.992+0000 7f34d289a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc155f00 con 0x7f34cc0a5fa0 2024-03-06T07:23:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.992+0000 7f34d289a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc155d80 con 0x7f34cc152b70 2024-03-06T07:23:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.992+0000 7f34d289a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc0c3a00 con 0x7f34cc0c2b30 2024-03-06T07:23:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.992+0000 7f34cbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34cc0a5fa0 0x7f34cc1ebc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.992+0000 7f34cbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34cc0a5fa0 0x7f34cc1ebc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49760/0 (socket says 172.21.15.1:49760) 2024-03-06T07:23:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.992+0000 7f34cbfff640 1 -- 172.21.15.1:0/3128626869 learned_addr learned my addr 172.21.15.1:0/3128626869 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.992+0000 7f34d0e37640 1 --2- 172.21.15.1:0/3128626869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc152b70 0x7f34cc0bff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.992+0000 7f34cb7fe640 1 --2- 172.21.15.1:0/3128626869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34cc0c2b30 0x7f34cc1dfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.993+0000 7f34cbfff640 1 -- 172.21.15.1:0/3128626869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34cc0c2b30 msgr2=0x7f34cc1dfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.993+0000 7f34cbfff640 1 --2- 172.21.15.1:0/3128626869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34cc0c2b30 0x7f34cc1dfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.993+0000 7f34cbfff640 1 -- 172.21.15.1:0/3128626869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc152b70 msgr2=0x7f34cc0bff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.993+0000 7f34cbfff640 1 --2- 172.21.15.1:0/3128626869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc152b70 0x7f34cc0bff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.993+0000 7f34cbfff640 1 -- 172.21.15.1:0/3128626869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc1eb170 con 0x7f34cc0a5fa0 2024-03-06T07:23:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.993+0000 7f34cbfff640 1 --2- 172.21.15.1:0/3128626869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34cc0a5fa0 0x7f34cc1ebc10 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f34bc009890 tx=0x7f34bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.993+0000 7f34c8ff9640 1 -- 172.21.15.1:0/3128626869 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc041070 con 0x7f34cc0a5fa0 2024-03-06T07:23:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.993+0000 7f34c8ff9640 1 -- 172.21.15.1:0/3128626869 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34bc03b920 con 0x7f34cc0a5fa0 2024-03-06T07:23:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.993+0000 7f34d289a640 1 -- 172.21.15.1:0/3128626869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34cc1e02e0 con 0x7f34cc0a5fa0 2024-03-06T07:23:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.993+0000 7f34c8ff9640 1 -- 172.21.15.1:0/3128626869 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc035040 con 0x7f34cc0a5fa0 2024-03-06T07:23:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.993+0000 7f34d289a640 1 -- 172.21.15.1:0/3128626869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34cc1eb410 con 0x7f34cc0a5fa0 2024-03-06T07:23:22.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.994+0000 7f34d289a640 1 -- 172.21.15.1:0/3128626869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34cc067fd0 con 0x7f34cc0a5fa0 2024-03-06T07:23:22.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.995+0000 7f34c8ff9640 1 -- 172.21.15.1:0/3128626869 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f34bc038020 con 0x7f34cc0a5fa0 2024-03-06T07:23:22.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.995+0000 7f34c8ff9640 1 --2- 172.21.15.1:0/3128626869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34b806b4d0 0x7f34b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.996+0000 7f34c8ff9640 1 -- 172.21.15.1:0/3128626869 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f34bc0abea0 con 0x7f34cc0a5fa0 2024-03-06T07:23:22.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.996+0000 7f34cb7fe640 1 --2- 172.21.15.1:0/3128626869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34b806b4d0 0x7f34b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:22.996+0000 7f34cb7fe640 1 --2- 172.21.15.1:0/3128626869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34b806b4d0 0x7f34b806d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f34cc0a7f00 tx=0x7f34c0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.000+0000 7f34c8ff9640 1 -- 172.21.15.1:0/3128626869 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f34bc031800 con 0x7f34cc0a5fa0 2024-03-06T07:23:23.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.221+0000 7f34d289a640 1 -- 172.21.15.1:0/3128626869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34cc071790 con 0x7f34cc0a5fa0 2024-03-06T07:23:23.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.222+0000 7f34c8ff9640 1 -- 172.21.15.1:0/3128626869 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6251 (secure 0 0 0) 0x7f34bc07a220 con 0x7f34cc0a5fa0 2024-03-06T07:23:23.224 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:23.224 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":66,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.1:6835/2685225325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2685225325},{"type":"v1","addr":"172.21.15.1:6835","nonce":2685225325}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.98:6836/4069609682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4069609682},{"type":"v1","addr":"172.21.15.98:6836","nonce":4069609682}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":66,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.969135+0000","modified":"2024-03-06T07:23:17.152713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.1:6834/3919123103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3919123103},{"type":"v1","addr":"172.21.15.1:6834","nonce":3919123103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6684,"qdb_cluster":[6684]},"id":5},{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:07.074540+0000","modified":"2024-03-06T07:23:10.095697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5845":{"gid":5845,"name":"d","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/114998998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":114998998},{"type":"v1","addr":"172.21.15.98:6837","nonce":114998998}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5845,"qdb_cluster":[5845]},"id":6}]} 2024-03-06T07:23:23.224 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 66 2024-03-06T07:23:23.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.224+0000 7f34d289a640 1 -- 172.21.15.1:0/3128626869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34b806b4d0 msgr2=0x7f34b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.224+0000 7f34d289a640 1 --2- 172.21.15.1:0/3128626869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34b806b4d0 0x7f34b806d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f34cc0a7f00 tx=0x7f34c0009040 comp rx=0 tx=0).stop 2024-03-06T07:23:23.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.224+0000 7f34d289a640 1 -- 172.21.15.1:0/3128626869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34cc0a5fa0 msgr2=0x7f34cc1ebc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.224+0000 7f34d289a640 1 --2- 172.21.15.1:0/3128626869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34cc0a5fa0 0x7f34cc1ebc10 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f34bc009890 tx=0x7f34bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.224+0000 7f34d289a640 1 -- 172.21.15.1:0/3128626869 shutdown_connections 2024-03-06T07:23:23.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.224+0000 7f34d289a640 1 --2- 172.21.15.1:0/3128626869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34b806b4d0 0x7f34b806d990 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.224+0000 7f34d289a640 1 --2- 172.21.15.1:0/3128626869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc152b70 0x7f34cc0bff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.224+0000 7f34d289a640 1 --2- 172.21.15.1:0/3128626869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34cc0c2b30 0x7f34cc1dfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.224+0000 7f34d289a640 1 --2- 172.21.15.1:0/3128626869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34cc0a5fa0 0x7f34cc1ebc10 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.224+0000 7f34d289a640 1 -- 172.21.15.1:0/3128626869 >> 172.21.15.1:0/3128626869 conn(0x7f34cc0c6860 msgr2=0x7f34cc06eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.224+0000 7f34d289a640 1 -- 172.21.15.1:0/3128626869 shutdown_connections 2024-03-06T07:23:23.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.224+0000 7f34d289a640 1 -- 172.21.15.1:0/3128626869 wait complete. 2024-03-06T07:23:23.237 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:23.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.336+0000 7f5b02afb640 1 Processor -- start 2024-03-06T07:23:23.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.336+0000 7f5b02afb640 1 -- start start 2024-03-06T07:23:23.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.337+0000 7f5b02afb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5afc0c8cd0 0x7f5afc0c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.337+0000 7f5b02afb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc0c2b90 con 0x7f5afc0c3260 2024-03-06T07:23:23.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.337+0000 7f5b02afb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc156020 con 0x7f5afc155b70 2024-03-06T07:23:23.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.337+0000 7f5b02afb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc0c37b0 con 0x7f5afc0c8cd0 2024-03-06T07:23:23.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.337+0000 7f5afbfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5afc0c3260 0x7f5afc0c7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47116/0 (socket says 172.21.15.1:47116) 2024-03-06T07:23:23.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.337+0000 7f5afbfff640 1 -- 172.21.15.1:0/1124185976 learned_addr learned my addr 172.21.15.1:0/1124185976 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:23.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.337+0000 7f5b00897640 1 --2- 172.21.15.1:0/1124185976 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5afc0c8cd0 0x7f5afc0c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.337+0000 7f5afb7fe640 1 -- 172.21.15.1:0/1124185976 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548555275 0 0) 0x7f5afc0c2b90 con 0x7f5afc0c3260 2024-03-06T07:23:23.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.337+0000 7f5afb7fe640 1 -- 172.21.15.1:0/1124185976 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae4003520 con 0x7f5afc0c3260 2024-03-06T07:23:23.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.337+0000 7f5afb7fe640 1 -- 172.21.15.1:0/1124185976 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993655029 0 0) 0x7f5afc156020 con 0x7f5afc155b70 2024-03-06T07:23:23.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.337+0000 7f5afb7fe640 1 -- 172.21.15.1:0/1124185976 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5afc0c2b90 con 0x7f5afc155b70 2024-03-06T07:23:23.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.337+0000 7f5b00897640 1 -- 172.21.15.1:0/1124185976 >> v1:172.21.15.98:6789/0 conn(0x7f5afc155b70 legacy=0x7f5afc159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.337+0000 7f5b00897640 1 -- 172.21.15.1:0/1124185976 >> v1:172.21.15.1:6789/0 conn(0x7f5afc0c3260 legacy=0x7f5afc0c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.337+0000 7f5b00897640 1 -- 172.21.15.1:0/1124185976 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc0c3a90 con 0x7f5afc0c8cd0 2024-03-06T07:23:23.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.338+0000 7f5b00897640 1 --2- 172.21.15.1:0/1124185976 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5afc0c8cd0 0x7f5afc0c90b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0097d0 tx=0x7f5aec02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4623c8609616aaa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.338+0000 7f5afb7fe640 1 -- 172.21.15.1:0/1124185976 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec035020 con 0x7f5afc0c8cd0 2024-03-06T07:23:23.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.338+0000 7f5afb7fe640 1 -- 172.21.15.1:0/1124185976 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aec031800 con 0x7f5afc0c8cd0 2024-03-06T07:23:23.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.338+0000 7f5afb7fe640 1 -- 172.21.15.1:0/1124185976 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec031bd0 con 0x7f5afc0c8cd0 2024-03-06T07:23:23.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.339+0000 7f5b02afb640 1 -- 172.21.15.1:0/1124185976 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5afc0c8cd0 msgr2=0x7f5afc0c90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.339+0000 7f5b02afb640 1 --2- 172.21.15.1:0/1124185976 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5afc0c8cd0 0x7f5afc0c90b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0097d0 tx=0x7f5aec02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.339+0000 7f5b02afb640 1 -- 172.21.15.1:0/1124185976 shutdown_connections 2024-03-06T07:23:23.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.339+0000 7f5b02afb640 1 --2- 172.21.15.1:0/1124185976 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5afc0c8cd0 0x7f5afc0c90b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.339+0000 7f5b02afb640 1 -- 172.21.15.1:0/1124185976 >> 172.21.15.1:0/1124185976 conn(0x7f5afc06b3e0 msgr2=0x7f5afc06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.339+0000 7f5b02afb640 1 -- 172.21.15.1:0/1124185976 shutdown_connections 2024-03-06T07:23:23.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.339+0000 7f5b02afb640 1 -- 172.21.15.1:0/1124185976 wait complete. 2024-03-06T07:23:23.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.340+0000 7f5b02afb640 1 Processor -- start 2024-03-06T07:23:23.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.340+0000 7f5b02afb640 1 -- start start 2024-03-06T07:23:23.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.341+0000 7f5b02afb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc0c3260 0x7f5afc1ee640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.341+0000 7f5b02afb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5afc0c8cd0 0x7f5afc0a8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.341+0000 7f5b02afb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5afc155b70 0x7f5afc14e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.341+0000 7f5b02afb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc0c8560 con 0x7f5afc0c3260 2024-03-06T07:23:23.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.341+0000 7f5b02afb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc158af0 con 0x7f5afc155b70 2024-03-06T07:23:23.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.341+0000 7f5b02afb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc0c86e0 con 0x7f5afc0c8cd0 2024-03-06T07:23:23.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.341+0000 7f5b00897640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc0c3260 0x7f5afc1ee640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.341+0000 7f5b00897640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc0c3260 0x7f5afc1ee640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49776/0 (socket says 172.21.15.1:49776) 2024-03-06T07:23:23.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.341+0000 7f5b00897640 1 -- 172.21.15.1:0/1674497589 learned_addr learned my addr 172.21.15.1:0/1674497589 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:23.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.341+0000 7f5b01098640 1 --2- 172.21.15.1:0/1674497589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5afc155b70 0x7f5afc14e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.341+0000 7f5afbfff640 1 --2- 172.21.15.1:0/1674497589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5afc0c8cd0 0x7f5afc0a8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.341+0000 7f5b00897640 1 -- 172.21.15.1:0/1674497589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5afc0c8cd0 msgr2=0x7f5afc0a8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.341+0000 7f5b00897640 1 --2- 172.21.15.1:0/1674497589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5afc0c8cd0 0x7f5afc0a8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.341+0000 7f5b00897640 1 -- 172.21.15.1:0/1674497589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5afc155b70 msgr2=0x7f5afc14e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.341+0000 7f5b00897640 1 --2- 172.21.15.1:0/1674497589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5afc155b70 0x7f5afc14e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.341+0000 7f5b00897640 1 -- 172.21.15.1:0/1674497589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc1ef010 con 0x7f5afc0c3260 2024-03-06T07:23:23.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.342+0000 7f5b00897640 1 --2- 172.21.15.1:0/1674497589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc0c3260 0x7f5afc1ee640 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f5aec02beb0 tx=0x7f5aec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.342+0000 7f5af97fa640 1 -- 172.21.15.1:0/1674497589 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec035040 con 0x7f5afc0c3260 2024-03-06T07:23:23.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.342+0000 7f5af97fa640 1 -- 172.21.15.1:0/1674497589 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aec03ad00 con 0x7f5afc0c3260 2024-03-06T07:23:23.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.342+0000 7f5af97fa640 1 -- 172.21.15.1:0/1674497589 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec0466c0 con 0x7f5afc0c3260 2024-03-06T07:23:23.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.342+0000 7f5b02afb640 1 -- 172.21.15.1:0/1674497589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5afc1ef2b0 con 0x7f5afc0c3260 2024-03-06T07:23:23.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.342+0000 7f5b02afb640 1 -- 172.21.15.1:0/1674497589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5afc1e1470 con 0x7f5afc0c3260 2024-03-06T07:23:23.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.343+0000 7f5b02afb640 1 -- 172.21.15.1:0/1674497589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5afc0c8110 con 0x7f5afc0c3260 2024-03-06T07:23:23.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.344+0000 7f5af97fa640 1 -- 172.21.15.1:0/1674497589 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5aec038020 con 0x7f5afc0c3260 2024-03-06T07:23:23.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.344+0000 7f5af97fa640 1 --2- 172.21.15.1:0/1674497589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ae806b4d0 0x7f5ae806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.344+0000 7f5af97fa640 1 -- 172.21.15.1:0/1674497589 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f5aec0ac690 con 0x7f5afc0c3260 2024-03-06T07:23:23.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.344+0000 7f5afbfff640 1 --2- 172.21.15.1:0/1674497589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ae806b4d0 0x7f5ae806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.345+0000 7f5afbfff640 1 --2- 172.21.15.1:0/1674497589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ae806b4d0 0x7f5ae806d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5afc0a8080 tx=0x7f5af0007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.352+0000 7f5af97fa640 1 -- 172.21.15.1:0/1674497589 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5afc0c8110 con 0x7f5afc0c3260 2024-03-06T07:23:23.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.534+0000 7f5b02afb640 1 -- 172.21.15.1:0/1674497589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5afc1580d0 con 0x7f5afc0c3260 2024-03-06T07:23:23.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.536+0000 7f5af97fa640 1 -- 172.21.15.1:0/1674497589 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+20676 (secure 0 0 0) 0x7f5afc1580d0 con 0x7f5afc0c3260 2024-03-06T07:23:23.538 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:23.538 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":79,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:23:13.133419+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:55.240857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:55.939811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:23:05.599046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:23:06.205539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6834/1494116991":"2024-03-07T07:23:13.133394+0000","172.21.15.1:6832/1494116991":"2024-03-07T07:23:13.133394+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:23.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.538+0000 7f5b02afb640 1 -- 172.21.15.1:0/1674497589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ae806b4d0 msgr2=0x7f5ae806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.538+0000 7f5b02afb640 1 --2- 172.21.15.1:0/1674497589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ae806b4d0 0x7f5ae806d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5afc0a8080 tx=0x7f5af0007d30 comp rx=0 tx=0).stop 2024-03-06T07:23:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.538+0000 7f5b02afb640 1 -- 172.21.15.1:0/1674497589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc0c3260 msgr2=0x7f5afc1ee640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.538+0000 7f5b02afb640 1 --2- 172.21.15.1:0/1674497589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc0c3260 0x7f5afc1ee640 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f5aec02beb0 tx=0x7f5aec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.538+0000 7f5b02afb640 1 -- 172.21.15.1:0/1674497589 shutdown_connections 2024-03-06T07:23:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.538+0000 7f5b02afb640 1 --2- 172.21.15.1:0/1674497589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ae806b4d0 0x7f5ae806d990 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.538+0000 7f5b02afb640 1 --2- 172.21.15.1:0/1674497589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5afc155b70 0x7f5afc14e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.538+0000 7f5b02afb640 1 --2- 172.21.15.1:0/1674497589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5afc0c8cd0 0x7f5afc0a8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.538+0000 7f5b02afb640 1 --2- 172.21.15.1:0/1674497589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc0c3260 0x7f5afc1ee640 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.538+0000 7f5b02afb640 1 -- 172.21.15.1:0/1674497589 >> 172.21.15.1:0/1674497589 conn(0x7f5afc06b3e0 msgr2=0x7f5afc06b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.538+0000 7f5b02afb640 1 -- 172.21.15.1:0/1674497589 shutdown_connections 2024-03-06T07:23:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.538+0000 7f5b02afb640 1 -- 172.21.15.1:0/1674497589 wait complete. 2024-03-06T07:23:23.551 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:23:23.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.651+0000 7f0ab582a640 1 Processor -- start 2024-03-06T07:23:23.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.651+0000 7f0ab582a640 1 -- start start 2024-03-06T07:23:23.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.652+0000 7f0ab582a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ab00c32f0 0x7f0ab01543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.652+0000 7f0ab582a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ab0070c80 0x7f0ab00710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.652+0000 7f0ab582a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab0150900 con 0x7f0ab0150520 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.652+0000 7f0ab582a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab0153ed0 con 0x7f0ab0070c80 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.652+0000 7f0ab582a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab0071790 con 0x7f0ab00c32f0 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.652+0000 7f0aaf7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ab0070c80 0x7f0ab00710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.652+0000 7f0aaeffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0ab0150520 0x7f0ab01537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47130/0 (socket says 172.21.15.1:47130) 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.652+0000 7f0aaeffd640 1 -- 172.21.15.1:0/1078993891 learned_addr learned my addr 172.21.15.1:0/1078993891 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.652+0000 7f0aae7fc640 1 --2- 172.21.15.1:0/1078993891 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ab00c32f0 0x7f0ab01543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.653+0000 7f0aaf7fe640 1 -- 172.21.15.1:0/1078993891 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ab00c32f0 msgr2=0x7f0ab01543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.653+0000 7f0aaf7fe640 1 --2- 172.21.15.1:0/1078993891 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ab00c32f0 0x7f0ab01543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.653+0000 7f0aaf7fe640 1 -- 172.21.15.1:0/1078993891 >> v1:172.21.15.1:6789/0 conn(0x7f0ab0150520 legacy=0x7f0ab01537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.653+0000 7f0aaf7fe640 1 -- 172.21.15.1:0/1078993891 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab0157050 con 0x7f0ab0070c80 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.653+0000 7f0aadffb640 1 -- 172.21.15.1:0/1078993891 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084653045 0 0) 0x7f0ab0150900 con 0x7f0ab0150520 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.653+0000 7f0aaf7fe640 1 --2- 172.21.15.1:0/1078993891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ab0070c80 0x7f0ab00710f0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8009d90 tx=0x7f0aa802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f420f2326d79a6a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.654+0000 7f0aadffb640 1 -- 172.21.15.1:0/1078993891 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8039070 con 0x7f0ab0070c80 2024-03-06T07:23:23.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.654+0000 7f0aadffb640 1 -- 172.21.15.1:0/1078993891 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa802cc50 con 0x7f0ab0070c80 2024-03-06T07:23:23.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.654+0000 7f0aadffb640 1 -- 172.21.15.1:0/1078993891 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8034440 con 0x7f0ab0070c80 2024-03-06T07:23:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.654+0000 7f0ab582a640 1 -- 172.21.15.1:0/1078993891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ab0070c80 msgr2=0x7f0ab00710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.654+0000 7f0ab582a640 1 --2- 172.21.15.1:0/1078993891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ab0070c80 0x7f0ab00710f0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8009d90 tx=0x7f0aa802bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.654+0000 7f0ab582a640 1 -- 172.21.15.1:0/1078993891 shutdown_connections 2024-03-06T07:23:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.654+0000 7f0ab582a640 1 --2- 172.21.15.1:0/1078993891 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ab00c32f0 0x7f0ab01543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.654+0000 7f0ab582a640 1 --2- 172.21.15.1:0/1078993891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ab0070c80 0x7f0ab00710f0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.654+0000 7f0ab582a640 1 -- 172.21.15.1:0/1078993891 >> 172.21.15.1:0/1078993891 conn(0x7f0ab00c88f0 msgr2=0x7f0ab00c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.654+0000 7f0ab582a640 1 -- 172.21.15.1:0/1078993891 shutdown_connections 2024-03-06T07:23:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.654+0000 7f0ab582a640 1 -- 172.21.15.1:0/1078993891 wait complete. 2024-03-06T07:23:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.655+0000 7f0ab582a640 1 Processor -- start 2024-03-06T07:23:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.655+0000 7f0ab582a640 1 -- start start 2024-03-06T07:23:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.656+0000 7f0ab582a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ab0070c80 0x7f0ab00c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.656+0000 7f0ab582a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ab00c32f0 0x7f0ab01df5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.656+0000 7f0ab582a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ab0150520 0x7f0ab01dfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.656+0000 7f0ab582a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab0157330 con 0x7f0ab00c32f0 2024-03-06T07:23:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.656+0000 7f0ab582a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab01532a0 con 0x7f0ab0150520 2024-03-06T07:23:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.656+0000 7f0ab582a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab00c7ec0 con 0x7f0ab0070c80 2024-03-06T07:23:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.656+0000 7f0aae7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ab00c32f0 0x7f0ab01df5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.656+0000 7f0aae7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ab00c32f0 0x7f0ab01df5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49788/0 (socket says 172.21.15.1:49788) 2024-03-06T07:23:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.656+0000 7f0aae7fc640 1 -- 172.21.15.1:0/4080051954 learned_addr learned my addr 172.21.15.1:0/4080051954 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.656+0000 7f0aaf7fe640 1 --2- 172.21.15.1:0/4080051954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ab0150520 0x7f0ab01dfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.656+0000 7f0aaeffd640 1 --2- 172.21.15.1:0/4080051954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ab0070c80 0x7f0ab00c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.656+0000 7f0aae7fc640 1 -- 172.21.15.1:0/4080051954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ab0070c80 msgr2=0x7f0ab00c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.656+0000 7f0aae7fc640 1 --2- 172.21.15.1:0/4080051954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ab0070c80 0x7f0ab00c49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.656+0000 7f0aae7fc640 1 -- 172.21.15.1:0/4080051954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ab0150520 msgr2=0x7f0ab01dfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.656+0000 7f0aae7fc640 1 --2- 172.21.15.1:0/4080051954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ab0150520 0x7f0ab01dfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.656+0000 7f0aae7fc640 1 -- 172.21.15.1:0/4080051954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab01dcd20 con 0x7f0ab00c32f0 2024-03-06T07:23:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.657+0000 7f0aae7fc640 1 --2- 172.21.15.1:0/4080051954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ab00c32f0 0x7f0ab01df5e0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f0aa400e9b0 tx=0x7f0aa400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.657+0000 7f0a97fff640 1 -- 172.21.15.1:0/4080051954 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa4017070 con 0x7f0ab00c32f0 2024-03-06T07:23:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.657+0000 7f0a97fff640 1 -- 172.21.15.1:0/4080051954 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa400c9a0 con 0x7f0ab00c32f0 2024-03-06T07:23:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.657+0000 7f0ab582a640 1 -- 172.21.15.1:0/4080051954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab01dca40 con 0x7f0ab00c32f0 2024-03-06T07:23:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.657+0000 7f0a97fff640 1 -- 172.21.15.1:0/4080051954 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa400cd90 con 0x7f0ab00c32f0 2024-03-06T07:23:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.657+0000 7f0ab582a640 1 -- 172.21.15.1:0/4080051954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab0149da0 con 0x7f0ab00c32f0 2024-03-06T07:23:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.658+0000 7f0ab582a640 1 -- 172.21.15.1:0/4080051954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab00686e0 con 0x7f0ab00c32f0 2024-03-06T07:23:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.658+0000 7f0a97fff640 1 -- 172.21.15.1:0/4080051954 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0aa401a070 con 0x7f0ab00c32f0 2024-03-06T07:23:23.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.659+0000 7f0a97fff640 1 --2- 172.21.15.1:0/4080051954 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a9c06b4d0 0x7f0a9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.659+0000 7f0a97fff640 1 -- 172.21.15.1:0/4080051954 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f0aa408aa20 con 0x7f0ab00c32f0 2024-03-06T07:23:23.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.659+0000 7f0aaeffd640 1 --2- 172.21.15.1:0/4080051954 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a9c06b4d0 0x7f0a9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.659+0000 7f0aaeffd640 1 --2- 172.21.15.1:0/4080051954 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a9c06b4d0 0x7f0a9c06d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0007fd0 tx=0x7f0aa0007de0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.663+0000 7f0a97fff640 1 -- 172.21.15.1:0/4080051954 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ab01dcd20 con 0x7f0ab00c32f0 2024-03-06T07:23:23.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.884+0000 7f0ab582a640 1 -- 172.21.15.1:0/4080051954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0ab00c2e70 con 0x7f0ab00c32f0 2024-03-06T07:23:23.887 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:23:23.885+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:23:23.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.886+0000 7f0a97fff640 1 -- 172.21.15.1:0/4080051954 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v67) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0aa4058d10 con 0x7f0ab00c32f0 2024-03-06T07:23:23.888 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.889+0000 7f0ab582a640 1 -- 172.21.15.1:0/4080051954 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a9c06b4d0 msgr2=0x7f0a9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.889+0000 7f0ab582a640 1 --2- 172.21.15.1:0/4080051954 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a9c06b4d0 0x7f0a9c06d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0007fd0 tx=0x7f0aa0007de0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.889+0000 7f0ab582a640 1 -- 172.21.15.1:0/4080051954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ab00c32f0 msgr2=0x7f0ab01df5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.889+0000 7f0ab582a640 1 --2- 172.21.15.1:0/4080051954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ab00c32f0 0x7f0ab01df5e0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f0aa400e9b0 tx=0x7f0aa400ee70 comp rx=0 tx=0).stop 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.889+0000 7f0ab582a640 1 -- 172.21.15.1:0/4080051954 shutdown_connections 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.889+0000 7f0ab582a640 1 --2- 172.21.15.1:0/4080051954 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a9c06b4d0 0x7f0a9c06d990 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.889+0000 7f0ab582a640 1 --2- 172.21.15.1:0/4080051954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ab0150520 0x7f0ab01dfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.889+0000 7f0ab582a640 1 --2- 172.21.15.1:0/4080051954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ab00c32f0 0x7f0ab01df5e0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.889+0000 7f0ab582a640 1 --2- 172.21.15.1:0/4080051954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ab0070c80 0x7f0ab00c49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.889+0000 7f0ab582a640 1 -- 172.21.15.1:0/4080051954 >> 172.21.15.1:0/4080051954 conn(0x7f0ab00c88f0 msgr2=0x7f0ab00c6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.889+0000 7f0ab582a640 1 -- 172.21.15.1:0/4080051954 shutdown_connections 2024-03-06T07:23:23.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:23.889+0000 7f0ab582a640 1 -- 172.21.15.1:0/4080051954 wait complete. 2024-03-06T07:23:23.893 INFO:tasks.ceph.mds.c.smithi001.stderr: -160> 2024-03-06T07:23:19.141+0000 7f12f8071640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:23.903 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:23:23.916 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:23:24.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.005+0000 7f5746a64640 1 Processor -- start 2024-03-06T07:23:24.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.005+0000 7f5746a64640 1 -- start start 2024-03-06T07:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.006+0000 7f5746a64640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f57400c6730 0x7f57400c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.006+0000 7f5746a64640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57400c3300 con 0x7f5740152690 2024-03-06T07:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.006+0000 7f5746a64640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57400c35a0 con 0x7f57400c5d10 2024-03-06T07:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.006+0000 7f5746a64640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57400c3880 con 0x7f57400c6730 2024-03-06T07:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.006+0000 7f573ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f57400c6730 0x7f57400c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.006+0000 7f5744800640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5740152690 0x7f5740155ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47136/0 (socket says 172.21.15.1:47136) 2024-03-06T07:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.006+0000 7f5744800640 1 -- 172.21.15.1:0/3562219689 learned_addr learned my addr 172.21.15.1:0/3562219689 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.006+0000 7f573ffff640 1 -- 172.21.15.1:0/3562219689 >> v1:172.21.15.98:6789/0 conn(0x7f57400c5d10 legacy=0x7f5740156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.006+0000 7f573ffff640 1 -- 172.21.15.1:0/3562219689 >> v1:172.21.15.1:6789/0 conn(0x7f5740152690 legacy=0x7f5740155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.006+0000 7f573ffff640 1 -- 172.21.15.1:0/3562219689 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57400c3b60 con 0x7f57400c6730 2024-03-06T07:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.006+0000 7f573f7fe640 1 -- 172.21.15.1:0/3562219689 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127870557 0 0) 0x7f57400c3300 con 0x7f5740152690 2024-03-06T07:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.007+0000 7f573ffff640 1 --2- 172.21.15.1:0/3562219689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57400c6730 0x7f57400c57d0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f57280097a0 tx=0x7f572802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0784cb6d20b383a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.007+0000 7f573f7fe640 1 -- 172.21.15.1:0/3562219689 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5728035020 con 0x7f57400c6730 2024-03-06T07:23:24.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.007+0000 7f573f7fe640 1 -- 172.21.15.1:0/3562219689 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5728031830 con 0x7f57400c6730 2024-03-06T07:23:24.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.007+0000 7f573f7fe640 1 -- 172.21.15.1:0/3562219689 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5728031c00 con 0x7f57400c6730 2024-03-06T07:23:24.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.008+0000 7f5746a64640 1 -- 172.21.15.1:0/3562219689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57400c6730 msgr2=0x7f57400c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.008+0000 7f5746a64640 1 --2- 172.21.15.1:0/3562219689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57400c6730 0x7f57400c57d0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f57280097a0 tx=0x7f572802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.008+0000 7f5746a64640 1 -- 172.21.15.1:0/3562219689 shutdown_connections 2024-03-06T07:23:24.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.008+0000 7f5746a64640 1 --2- 172.21.15.1:0/3562219689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57400c6730 0x7f57400c57d0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.008+0000 7f5746a64640 1 -- 172.21.15.1:0/3562219689 >> 172.21.15.1:0/3562219689 conn(0x7f574006b410 msgr2=0x7f574006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.008+0000 7f5746a64640 1 -- 172.21.15.1:0/3562219689 shutdown_connections 2024-03-06T07:23:24.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.008+0000 7f5746a64640 1 -- 172.21.15.1:0/3562219689 wait complete. 2024-03-06T07:23:24.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.009+0000 7f5746a64640 1 Processor -- start 2024-03-06T07:23:24.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.009+0000 7f5746a64640 1 -- start start 2024-03-06T07:23:24.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f5746a64640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57400c5d10 0x7f5740153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f5746a64640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57400c6730 0x7f574006d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f5746a64640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5740152690 0x7f574014eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f5746a64640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5740152bf0 con 0x7f5740152690 2024-03-06T07:23:24.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f5746a64640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5740152a70 con 0x7f57400c6730 2024-03-06T07:23:24.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f5746a64640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57401573e0 con 0x7f57400c5d10 2024-03-06T07:23:24.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f5745001640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5740152690 0x7f574014eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f5745001640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5740152690 0x7f574014eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49836/0 (socket says 172.21.15.1:49836) 2024-03-06T07:23:24.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f5744800640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57400c5d10 0x7f5740153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f5745001640 1 -- 172.21.15.1:0/3847202961 learned_addr learned my addr 172.21.15.1:0/3847202961 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:24.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f573ffff640 1 --2- 172.21.15.1:0/3847202961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57400c6730 0x7f574006d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f5745001640 1 -- 172.21.15.1:0/3847202961 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57400c5d10 msgr2=0x7f5740153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f5745001640 1 --2- 172.21.15.1:0/3847202961 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57400c5d10 0x7f5740153d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f5745001640 1 -- 172.21.15.1:0/3847202961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57400c6730 msgr2=0x7f574006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f5745001640 1 --2- 172.21.15.1:0/3847202961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57400c6730 0x7f574006d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f5745001640 1 -- 172.21.15.1:0/3847202961 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5740153ac0 con 0x7f5740152690 2024-03-06T07:23:24.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f573ffff640 1 --2- 172.21.15.1:0/3847202961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57400c6730 0x7f574006d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:24.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f5745001640 1 --2- 172.21.15.1:0/3847202961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5740152690 0x7f574014eef0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f573800c3c0 tx=0x7f573800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f573d7fa640 1 -- 172.21.15.1:0/3847202961 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738017070 con 0x7f5740152690 2024-03-06T07:23:24.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.010+0000 7f573d7fa640 1 -- 172.21.15.1:0/3847202961 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57380052d0 con 0x7f5740152690 2024-03-06T07:23:24.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.011+0000 7f5746a64640 1 -- 172.21.15.1:0/3847202961 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57401ee480 con 0x7f5740152690 2024-03-06T07:23:24.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.011+0000 7f573d7fa640 1 -- 172.21.15.1:0/3847202961 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738003810 con 0x7f5740152690 2024-03-06T07:23:24.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.011+0000 7f5746a64640 1 -- 172.21.15.1:0/3847202961 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57400a7130 con 0x7f5740152690 2024-03-06T07:23:24.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.012+0000 7f5746a64640 1 -- 172.21.15.1:0/3847202961 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5740067fd0 con 0x7f5740152690 2024-03-06T07:23:24.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.012+0000 7f573d7fa640 1 -- 172.21.15.1:0/3847202961 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f573801a070 con 0x7f5740152690 2024-03-06T07:23:24.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.013+0000 7f573d7fa640 1 --2- 172.21.15.1:0/3847202961 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f572006b4d0 0x7f572006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.013+0000 7f573d7fa640 1 -- 172.21.15.1:0/3847202961 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f5738089a30 con 0x7f5740152690 2024-03-06T07:23:24.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.013+0000 7f5744800640 1 --2- 172.21.15.1:0/3847202961 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f572006b4d0 0x7f572006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.013+0000 7f5744800640 1 --2- 172.21.15.1:0/3847202961 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f572006b4d0 0x7f572006d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5730001560 tx=0x7f57300014b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.017+0000 7f573d7fa640 1 -- 172.21.15.1:0/3847202961 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5740153ac0 con 0x7f5740152690 2024-03-06T07:23:24.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.235+0000 7f5746a64640 1 -- 172.21.15.1:0/3847202961 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f57400c2c40 con 0x7f5740152690 2024-03-06T07:23:24.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.254+0000 7f573d7fa640 1 -- 172.21.15.1:0/3847202961 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5738057cc0 con 0x7f5740152690 2024-03-06T07:23:24.259 INFO:tasks.ceph.mds.d.smithi098.stderr: -355> 2024-03-06T07:23:12.903+0000 7f4eca4e9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:24.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.259+0000 7f5746a64640 1 -- 172.21.15.1:0/3847202961 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f572006b4d0 msgr2=0x7f572006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.259+0000 7f5746a64640 1 --2- 172.21.15.1:0/3847202961 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f572006b4d0 0x7f572006d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5730001560 tx=0x7f57300014b0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.259+0000 7f5746a64640 1 -- 172.21.15.1:0/3847202961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5740152690 msgr2=0x7f574014eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.259+0000 7f5746a64640 1 --2- 172.21.15.1:0/3847202961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5740152690 0x7f574014eef0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f573800c3c0 tx=0x7f573800c880 comp rx=0 tx=0).stop 2024-03-06T07:23:24.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.260+0000 7f5746a64640 1 -- 172.21.15.1:0/3847202961 shutdown_connections 2024-03-06T07:23:24.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.260+0000 7f5746a64640 1 --2- 172.21.15.1:0/3847202961 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f572006b4d0 0x7f572006d990 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.260+0000 7f5746a64640 1 --2- 172.21.15.1:0/3847202961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5740152690 0x7f574014eef0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.260+0000 7f5746a64640 1 --2- 172.21.15.1:0/3847202961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57400c6730 0x7f574006d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.260+0000 7f5746a64640 1 --2- 172.21.15.1:0/3847202961 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f57400c5d10 0x7f5740153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.260+0000 7f5746a64640 1 -- 172.21.15.1:0/3847202961 >> 172.21.15.1:0/3847202961 conn(0x7f574006b410 msgr2=0x7f57400c1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.260+0000 7f5746a64640 1 -- 172.21.15.1:0/3847202961 shutdown_connections 2024-03-06T07:23:24.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.260+0000 7f5746a64640 1 -- 172.21.15.1:0/3847202961 wait complete. 2024-03-06T07:23:24.275 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:23:24.287 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:23:24.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.418+0000 7f74442b8640 1 Processor -- start 2024-03-06T07:23:24.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.418+0000 7f74442b8640 1 -- start start 2024-03-06T07:23:24.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.419+0000 7f74442b8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f743c158180 0x7f743c0c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.419+0000 7f74442b8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f743c0c6800 0x7f743c157160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.419+0000 7f74442b8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f743c158560 con 0x7f743c0c5e20 2024-03-06T07:23:24.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.419+0000 7f74442b8640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f743c156210 con 0x7f743c158180 2024-03-06T07:23:24.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.419+0000 7f74442b8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f743c0c3ea0 con 0x7f743c0c6800 2024-03-06T07:23:24.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.419+0000 7f7442855640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f743c0c6800 0x7f743c157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.419+0000 7f7442054640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f743c158180 0x7f743c0c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.419+0000 7f7441853640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f743c0c5e20 0x7f743c156a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:55358/0 (socket says 172.21.15.1:55358) 2024-03-06T07:23:24.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.419+0000 7f7441853640 1 -- 172.21.15.1:0/3103784851 learned_addr learned my addr 172.21.15.1:0/3103784851 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:24.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.420+0000 7f7442855640 1 -- 172.21.15.1:0/3103784851 >> v2:172.21.15.98:3300/0 conn(0x7f743c158180 msgr2=0x7f743c0c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.420+0000 7f7442855640 1 --2- 172.21.15.1:0/3103784851 >> v2:172.21.15.98:3300/0 conn(0x7f743c158180 0x7f743c0c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.420+0000 7f7442855640 1 -- 172.21.15.1:0/3103784851 >> v1:172.21.15.98:6789/0 conn(0x7f743c0c5e20 legacy=0x7f743c156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.419+0000 7f7441052640 1 -- 172.21.15.1:0/3103784851 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840101753 0 0) 0x7f743c158560 con 0x7f743c0c5e20 2024-03-06T07:23:24.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.420+0000 7f7442855640 1 -- 172.21.15.1:0/3103784851 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743c155600 con 0x7f743c0c6800 2024-03-06T07:23:24.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.420+0000 7f7442855640 1 --2- 172.21.15.1:0/3103784851 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f743c0c6800 0x7f743c157160 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7438009dc0 tx=0x7f743802bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bd1949c49a2f2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.420+0000 7f7441052640 1 -- 172.21.15.1:0/3103784851 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7438039070 con 0x7f743c0c6800 2024-03-06T07:23:24.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.421+0000 7f7441052640 1 -- 172.21.15.1:0/3103784851 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743802cb20 con 0x7f743c0c6800 2024-03-06T07:23:24.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.421+0000 7f7441052640 1 -- 172.21.15.1:0/3103784851 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7438034440 con 0x7f743c0c6800 2024-03-06T07:23:24.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.421+0000 7f74442b8640 1 -- 172.21.15.1:0/3103784851 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f743c0c6800 msgr2=0x7f743c157160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.421+0000 7f74442b8640 1 --2- 172.21.15.1:0/3103784851 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f743c0c6800 0x7f743c157160 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7438009dc0 tx=0x7f743802bd70 comp rx=0 tx=0).stop 2024-03-06T07:23:24.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.421+0000 7f74442b8640 1 -- 172.21.15.1:0/3103784851 shutdown_connections 2024-03-06T07:23:24.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.421+0000 7f74442b8640 1 --2- 172.21.15.1:0/3103784851 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f743c0c6800 0x7f743c157160 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.421+0000 7f74442b8640 1 --2- 172.21.15.1:0/3103784851 >> v2:172.21.15.98:3300/0 conn(0x7f743c158180 0x7f743c0c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.421+0000 7f74442b8640 1 -- 172.21.15.1:0/3103784851 >> 172.21.15.1:0/3103784851 conn(0x7f743c06b410 msgr2=0x7f743c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.421+0000 7f74442b8640 1 -- 172.21.15.1:0/3103784851 shutdown_connections 2024-03-06T07:23:24.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.421+0000 7f74442b8640 1 -- 172.21.15.1:0/3103784851 wait complete. 2024-03-06T07:23:24.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.422+0000 7f74442b8640 1 Processor -- start 2024-03-06T07:23:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.422+0000 7f74442b8640 1 -- start start 2024-03-06T07:23:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.423+0000 7f74442b8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f743c0c5e20 0x7f743c14d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.423+0000 7f74442b8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c0c6800 0x7f743c0a8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.423+0000 7f74442b8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f743c158180 0x7f743c0a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.423+0000 7f74442b8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743c155ea0 con 0x7f743c0c6800 2024-03-06T07:23:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.423+0000 7f74442b8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743c155bc0 con 0x7f743c0c5e20 2024-03-06T07:23:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.423+0000 7f74442b8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f743c1576a0 con 0x7f743c158180 2024-03-06T07:23:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.423+0000 7f7441853640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c0c6800 0x7f743c0a8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.423+0000 7f7441853640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c0c6800 0x7f743c0a8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45198/0 (socket says 172.21.15.1:45198) 2024-03-06T07:23:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.423+0000 7f7441853640 1 -- 172.21.15.1:0/2307859760 learned_addr learned my addr 172.21.15.1:0/2307859760 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.423+0000 7f7442855640 1 --2- 172.21.15.1:0/2307859760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f743c158180 0x7f743c0a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.423+0000 7f7442054640 1 --2- 172.21.15.1:0/2307859760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f743c0c5e20 0x7f743c14d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.423+0000 7f7441853640 1 -- 172.21.15.1:0/2307859760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f743c158180 msgr2=0x7f743c0a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.423+0000 7f7441853640 1 --2- 172.21.15.1:0/2307859760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f743c158180 0x7f743c0a94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.423+0000 7f7441853640 1 -- 172.21.15.1:0/2307859760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f743c0c5e20 msgr2=0x7f743c14d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.423+0000 7f7441853640 1 --2- 172.21.15.1:0/2307859760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f743c0c5e20 0x7f743c14d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.423+0000 7f7441853640 1 -- 172.21.15.1:0/2307859760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743c1eb740 con 0x7f743c0c6800 2024-03-06T07:23:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.423+0000 7f7441853640 1 --2- 172.21.15.1:0/2307859760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c0c6800 0x7f743c0a8f70 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f742c00c420 tx=0x7f742c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.424+0000 7f7432ffd640 1 -- 172.21.15.1:0/2307859760 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c0074e0 con 0x7f743c0c6800 2024-03-06T07:23:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.424+0000 7f7432ffd640 1 -- 172.21.15.1:0/2307859760 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742c0047e0 con 0x7f743c0c6800 2024-03-06T07:23:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.424+0000 7f74442b8640 1 -- 172.21.15.1:0/2307859760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f743c1eb9e0 con 0x7f743c0c6800 2024-03-06T07:23:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.424+0000 7f7432ffd640 1 -- 172.21.15.1:0/2307859760 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c004c10 con 0x7f743c0c6800 2024-03-06T07:23:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.424+0000 7f74442b8640 1 -- 172.21.15.1:0/2307859760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f743c0a8a50 con 0x7f743c0c6800 2024-03-06T07:23:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.425+0000 7f74442b8640 1 -- 172.21.15.1:0/2307859760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f743c0680b0 con 0x7f743c0c6800 2024-03-06T07:23:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.425+0000 7f7432ffd640 1 -- 172.21.15.1:0/2307859760 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f742c011070 con 0x7f743c0c6800 2024-03-06T07:23:24.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.425+0000 7f7432ffd640 1 --2- 172.21.15.1:0/2307859760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f742006b4d0 0x7f742006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.425+0000 7f7432ffd640 1 -- 172.21.15.1:0/2307859760 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7215+0+0 (secure 0 0 0) 0x7f742c08a200 con 0x7f743c0c6800 2024-03-06T07:23:24.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.426+0000 7f7442054640 1 --2- 172.21.15.1:0/2307859760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f742006b4d0 0x7f742006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.426+0000 7f7442054640 1 --2- 172.21.15.1:0/2307859760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f742006b4d0 0x7f742006d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7434009b40 tx=0x7f7434009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.430+0000 7f7432ffd640 1 -- 172.21.15.1:0/2307859760 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f742c058460 con 0x7f743c0c6800 2024-03-06T07:23:24.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:24.611+0000 7f74442b8640 1 -- 172.21.15.1:0/2307859760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f743c0c2df0 con 0x7f743c0c6800 2024-03-06T07:23:25.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.259+0000 7f7432ffd640 1 -- 172.21.15.1:0/2307859760 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v82) v1 ==== 181+0+0 (secure 0 0 0) 0x7f742c05c500 con 0x7f743c0c6800 2024-03-06T07:23:25.261 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:23:25.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.262+0000 7f74442b8640 1 -- 172.21.15.1:0/2307859760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f742006b4d0 msgr2=0x7f742006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.262+0000 7f74442b8640 1 --2- 172.21.15.1:0/2307859760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f742006b4d0 0x7f742006d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7434009b40 tx=0x7f7434009190 comp rx=0 tx=0).stop 2024-03-06T07:23:25.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.262+0000 7f74442b8640 1 -- 172.21.15.1:0/2307859760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c0c6800 msgr2=0x7f743c0a8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.262+0000 7f74442b8640 1 --2- 172.21.15.1:0/2307859760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c0c6800 0x7f743c0a8f70 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f742c00c420 tx=0x7f742c00c8e0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.262+0000 7f74442b8640 1 -- 172.21.15.1:0/2307859760 shutdown_connections 2024-03-06T07:23:25.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.262+0000 7f74442b8640 1 --2- 172.21.15.1:0/2307859760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f742006b4d0 0x7f742006d990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.262+0000 7f74442b8640 1 --2- 172.21.15.1:0/2307859760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f743c158180 0x7f743c0a94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.262+0000 7f74442b8640 1 --2- 172.21.15.1:0/2307859760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c0c6800 0x7f743c0a8f70 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.262+0000 7f74442b8640 1 --2- 172.21.15.1:0/2307859760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f743c0c5e20 0x7f743c14d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.262+0000 7f74442b8640 1 -- 172.21.15.1:0/2307859760 >> 172.21.15.1:0/2307859760 conn(0x7f743c06b410 msgr2=0x7f743c1545f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.262+0000 7f74442b8640 1 -- 172.21.15.1:0/2307859760 shutdown_connections 2024-03-06T07:23:25.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.263+0000 7f74442b8640 1 -- 172.21.15.1:0/2307859760 wait complete. 2024-03-06T07:23:25.277 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:23:25.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.383+0000 7fc8153ae640 1 Processor -- start 2024-03-06T07:23:25.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.383+0000 7fc8153ae640 1 -- start start 2024-03-06T07:23:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.384+0000 7fc8153ae640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc81006f1f0 0x7fc8100c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.384+0000 7fc8153ae640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8100728c0 con 0x7fc8100c5950 2024-03-06T07:23:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.384+0000 7fc8153ae640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc810072ba0 con 0x7fc8100a60b0 2024-03-06T07:23:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.384+0000 7fc8153ae640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc810072690 con 0x7fc81006f1f0 2024-03-06T07:23:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.384+0000 7fc80f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc81006f1f0 0x7fc8100c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.384+0000 7fc80effd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc8100c5950 0x7fc810152910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50182/0 (socket says 172.21.15.1:50182) 2024-03-06T07:23:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.384+0000 7fc80effd640 1 -- 172.21.15.1:0/451098321 learned_addr learned my addr 172.21.15.1:0/451098321 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.384+0000 7fc80f7fe640 1 -- 172.21.15.1:0/451098321 >> v1:172.21.15.98:6789/0 conn(0x7fc8100a60b0 legacy=0x7fc81006eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.384+0000 7fc80f7fe640 1 -- 172.21.15.1:0/451098321 >> v1:172.21.15.1:6789/0 conn(0x7fc8100c5950 legacy=0x7fc810152910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.384+0000 7fc80f7fe640 1 -- 172.21.15.1:0/451098321 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc810072e80 con 0x7fc81006f1f0 2024-03-06T07:23:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.384+0000 7fc80e7fc640 1 -- 172.21.15.1:0/451098321 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092463015 0 0) 0x7fc8100728c0 con 0x7fc8100c5950 2024-03-06T07:23:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.385+0000 7fc80f7fe640 1 --2- 172.21.15.1:0/451098321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81006f1f0 0x7fc8100c5d30 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fc808009d90 tx=0x7fc80802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e065ebbd074c6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.385+0000 7fc80e7fc640 1 -- 172.21.15.1:0/451098321 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc808039070 con 0x7fc81006f1f0 2024-03-06T07:23:25.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.385+0000 7fc80e7fc640 1 -- 172.21.15.1:0/451098321 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80802cd00 con 0x7fc81006f1f0 2024-03-06T07:23:25.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.385+0000 7fc80e7fc640 1 -- 172.21.15.1:0/451098321 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc808034660 con 0x7fc81006f1f0 2024-03-06T07:23:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.385+0000 7fc8153ae640 1 -- 172.21.15.1:0/451098321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81006f1f0 msgr2=0x7fc8100c5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.385+0000 7fc8153ae640 1 --2- 172.21.15.1:0/451098321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81006f1f0 0x7fc8100c5d30 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fc808009d90 tx=0x7fc80802bf50 comp rx=0 tx=0).stop 2024-03-06T07:23:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.385+0000 7fc8153ae640 1 -- 172.21.15.1:0/451098321 shutdown_connections 2024-03-06T07:23:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.385+0000 7fc8153ae640 1 --2- 172.21.15.1:0/451098321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81006f1f0 0x7fc8100c5d30 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.385+0000 7fc8153ae640 1 -- 172.21.15.1:0/451098321 >> 172.21.15.1:0/451098321 conn(0x7fc8100a8e10 msgr2=0x7fc8100c3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.385+0000 7fc8153ae640 1 -- 172.21.15.1:0/451098321 shutdown_connections 2024-03-06T07:23:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.386+0000 7fc8153ae640 1 -- 172.21.15.1:0/451098321 wait complete. 2024-03-06T07:23:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.387+0000 7fc8153ae640 1 Processor -- start 2024-03-06T07:23:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.387+0000 7fc8153ae640 1 -- start start 2024-03-06T07:23:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.387+0000 7fc8153ae640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81006f1f0 0x7fc8101ec020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.387+0000 7fc8153ae640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8100a60b0 0x7fc8101476b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.387+0000 7fc8153ae640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8100c5950 0x7fc8101e8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.387+0000 7fc8153ae640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8100c18a0 con 0x7fc81006f1f0 2024-03-06T07:23:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.387+0000 7fc8153ae640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8100c1720 con 0x7fc8100a60b0 2024-03-06T07:23:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.387+0000 7fc8153ae640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8100c1a20 con 0x7fc8100c5950 2024-03-06T07:23:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.387+0000 7fc80effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81006f1f0 0x7fc8101ec020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.387+0000 7fc80effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81006f1f0 0x7fc8101ec020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45206/0 (socket says 172.21.15.1:45206) 2024-03-06T07:23:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.387+0000 7fc80effd640 1 -- 172.21.15.1:0/3012526393 learned_addr learned my addr 172.21.15.1:0/3012526393 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.388+0000 7fc807fff640 1 --2- 172.21.15.1:0/3012526393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8100a60b0 0x7fc8101476b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.388+0000 7fc80f7fe640 1 --2- 172.21.15.1:0/3012526393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8100c5950 0x7fc8101e8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.388+0000 7fc80effd640 1 -- 172.21.15.1:0/3012526393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8100c5950 msgr2=0x7fc8101e8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.388+0000 7fc80effd640 1 --2- 172.21.15.1:0/3012526393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8100c5950 0x7fc8101e8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.388+0000 7fc80effd640 1 -- 172.21.15.1:0/3012526393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8100a60b0 msgr2=0x7fc8101476b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.388+0000 7fc80effd640 1 --2- 172.21.15.1:0/3012526393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8100a60b0 0x7fc8101476b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.388+0000 7fc80effd640 1 -- 172.21.15.1:0/3012526393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8101e5fc0 con 0x7fc81006f1f0 2024-03-06T07:23:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.388+0000 7fc807fff640 1 --2- 172.21.15.1:0/3012526393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8100a60b0 0x7fc8101476b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.388+0000 7fc80effd640 1 --2- 172.21.15.1:0/3012526393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81006f1f0 0x7fc8101ec020 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fc80000c4d0 tx=0x7fc80000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.388+0000 7fc8077fe640 1 -- 172.21.15.1:0/3012526393 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc800018070 con 0x7fc81006f1f0 2024-03-06T07:23:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.388+0000 7fc8077fe640 1 -- 172.21.15.1:0/3012526393 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8000074e0 con 0x7fc81006f1f0 2024-03-06T07:23:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.388+0000 7fc8153ae640 1 -- 172.21.15.1:0/3012526393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8101e6280 con 0x7fc81006f1f0 2024-03-06T07:23:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.388+0000 7fc8077fe640 1 -- 172.21.15.1:0/3012526393 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc800004a20 con 0x7fc81006f1f0 2024-03-06T07:23:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.388+0000 7fc8153ae640 1 -- 172.21.15.1:0/3012526393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8100a9e50 con 0x7fc81006f1f0 2024-03-06T07:23:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.389+0000 7fc8153ae640 1 -- 172.21.15.1:0/3012526393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8100680b0 con 0x7fc81006f1f0 2024-03-06T07:23:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.390+0000 7fc8077fe640 1 -- 172.21.15.1:0/3012526393 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc80001b070 con 0x7fc81006f1f0 2024-03-06T07:23:25.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.390+0000 7fc8077fe640 1 --2- 172.21.15.1:0/3012526393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc7f006b4d0 0x7fc7f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.390+0000 7fc8077fe640 1 -- 172.21.15.1:0/3012526393 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6777+0+0 (secure 0 0 0) 0x7fc800089820 con 0x7fc81006f1f0 2024-03-06T07:23:25.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.390+0000 7fc807fff640 1 --2- 172.21.15.1:0/3012526393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc7f006b4d0 0x7fc7f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.391+0000 7fc807fff640 1 --2- 172.21.15.1:0/3012526393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc7f006b4d0 0x7fc7f006d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8007fd0 tx=0x7fc7f8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.394+0000 7fc8077fe640 1 -- 172.21.15.1:0/3012526393 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc800058300 con 0x7fc81006f1f0 2024-03-06T07:23:25.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:25.581+0000 7fc8153ae640 1 -- 172.21.15.1:0/3012526393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc81006ca60 con 0x7fc81006f1f0 2024-03-06T07:23:26.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.265+0000 7fc8077fe640 1 -- 172.21.15.1:0/3012526393 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v83) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc800057c00 con 0x7fc81006f1f0 2024-03-06T07:23:26.267 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-06T07:23:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.269+0000 7fc8153ae640 1 -- 172.21.15.1:0/3012526393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc7f006b4d0 msgr2=0x7fc7f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.269+0000 7fc8153ae640 1 --2- 172.21.15.1:0/3012526393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc7f006b4d0 0x7fc7f006d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8007fd0 tx=0x7fc7f8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.269+0000 7fc8153ae640 1 -- 172.21.15.1:0/3012526393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81006f1f0 msgr2=0x7fc8101ec020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.269+0000 7fc8153ae640 1 --2- 172.21.15.1:0/3012526393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81006f1f0 0x7fc8101ec020 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fc80000c4d0 tx=0x7fc80000c990 comp rx=0 tx=0).stop 2024-03-06T07:23:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.269+0000 7fc8153ae640 1 -- 172.21.15.1:0/3012526393 shutdown_connections 2024-03-06T07:23:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.269+0000 7fc8153ae640 1 --2- 172.21.15.1:0/3012526393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc7f006b4d0 0x7fc7f006d990 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.269+0000 7fc8153ae640 1 --2- 172.21.15.1:0/3012526393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8100c5950 0x7fc8101e8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.269+0000 7fc8153ae640 1 --2- 172.21.15.1:0/3012526393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8100a60b0 0x7fc8101476b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.269+0000 7fc8153ae640 1 --2- 172.21.15.1:0/3012526393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81006f1f0 0x7fc8101ec020 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.269+0000 7fc8153ae640 1 -- 172.21.15.1:0/3012526393 >> 172.21.15.1:0/3012526393 conn(0x7fc8100a8e10 msgr2=0x7fc8101528a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.269+0000 7fc8153ae640 1 -- 172.21.15.1:0/3012526393 shutdown_connections 2024-03-06T07:23:26.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.269+0000 7fc8153ae640 1 -- 172.21.15.1:0/3012526393 wait complete. 2024-03-06T07:23:26.286 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:26.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.398+0000 7fbad4608640 1 Processor -- start 2024-03-06T07:23:26.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.399+0000 7fbad4608640 1 -- start start 2024-03-06T07:23:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.399+0000 7fbad4608640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbacc0a7f50 0x7fbacc154300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.399+0000 7fbad4608640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbacc0c75a0 0x7fbacc0c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.399+0000 7fbad4608640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbacc0c8120 con 0x7fbacc150640 2024-03-06T07:23:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.399+0000 7fbad4608640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbacc153e30 con 0x7fbacc0c75a0 2024-03-06T07:23:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.399+0000 7fbad4608640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc0c7b60 con 0x7fbacc0a7f50 2024-03-06T07:23:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.400+0000 7fbad2ba5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbacc0c75a0 0x7fbacc0c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.400+0000 7fbad2ba5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbacc0c75a0 0x7fbacc0c8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45210/0 (socket says 172.21.15.1:45210) 2024-03-06T07:23:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.400+0000 7fbad2ba5640 1 -- 172.21.15.1:0/3587300892 learned_addr learned my addr 172.21.15.1:0/3587300892 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.400+0000 7fbad1ba3640 1 --2- 172.21.15.1:0/3587300892 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbacc0a7f50 0x7fbacc154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.400+0000 7fbad2ba5640 1 -- 172.21.15.1:0/3587300892 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbacc0a7f50 msgr2=0x7fbacc154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.400+0000 7fbad2ba5640 1 --2- 172.21.15.1:0/3587300892 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbacc0a7f50 0x7fbacc154300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.400+0000 7fbad2ba5640 1 -- 172.21.15.1:0/3587300892 >> v1:172.21.15.98:6789/0 conn(0x7fbacc150640 legacy=0x7fbacc153720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.400+0000 7fbad2ba5640 1 -- 172.21.15.1:0/3587300892 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbacc0c7e40 con 0x7fbacc0c75a0 2024-03-06T07:23:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.400+0000 7fbad1ba3640 1 --2- 172.21.15.1:0/3587300892 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbacc0a7f50 0x7fbacc154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.401+0000 7fbad2ba5640 1 --2- 172.21.15.1:0/3587300892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbacc0c75a0 0x7fbacc0c8d60 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009e50 tx=0x7fbabc02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5641b1f40d0eab9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.401+0000 7fbad13a2640 1 -- 172.21.15.1:0/3587300892 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc02c950 con 0x7fbacc0c75a0 2024-03-06T07:23:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.401+0000 7fbad13a2640 1 -- 172.21.15.1:0/3587300892 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbabc02caf0 con 0x7fbacc0c75a0 2024-03-06T07:23:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.401+0000 7fbad13a2640 1 -- 172.21.15.1:0/3587300892 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc0343f0 con 0x7fbacc0c75a0 2024-03-06T07:23:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.401+0000 7fbad4608640 1 -- 172.21.15.1:0/3587300892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbacc0c75a0 msgr2=0x7fbacc0c8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.401+0000 7fbad4608640 1 --2- 172.21.15.1:0/3587300892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbacc0c75a0 0x7fbacc0c8d60 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009e50 tx=0x7fbabc02bdb0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.401+0000 7fbad4608640 1 -- 172.21.15.1:0/3587300892 shutdown_connections 2024-03-06T07:23:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.401+0000 7fbad4608640 1 --2- 172.21.15.1:0/3587300892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbacc0c75a0 0x7fbacc0c8d60 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.401+0000 7fbad4608640 1 --2- 172.21.15.1:0/3587300892 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbacc0a7f50 0x7fbacc154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.402+0000 7fbad4608640 1 -- 172.21.15.1:0/3587300892 >> 172.21.15.1:0/3587300892 conn(0x7fbacc06b3e0 msgr2=0x7fbacc06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.402+0000 7fbad4608640 1 -- 172.21.15.1:0/3587300892 shutdown_connections 2024-03-06T07:23:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.402+0000 7fbad4608640 1 -- 172.21.15.1:0/3587300892 wait complete. 2024-03-06T07:23:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.403+0000 7fbad4608640 1 Processor -- start 2024-03-06T07:23:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.403+0000 7fbad4608640 1 -- start start 2024-03-06T07:23:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.403+0000 7fbad4608640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbacc0a7f50 0x7fbacc14d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.403+0000 7fbad4608640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbacc0c75a0 0x7fbacc0c1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.403+0000 7fbad4608640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbacc150640 0x7fbacc1ee510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.403+0000 7fbad4608640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc150a20 con 0x7fbacc0a7f50 2024-03-06T07:23:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.403+0000 7fbad4608640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc153200 con 0x7fbacc0c75a0 2024-03-06T07:23:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.403+0000 7fbad4608640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc071790 con 0x7fbacc150640 2024-03-06T07:23:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.403+0000 7fbad23a4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbacc0a7f50 0x7fbacc14d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.403+0000 7fbad23a4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbacc0a7f50 0x7fbacc14d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45222/0 (socket says 172.21.15.1:45222) 2024-03-06T07:23:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.403+0000 7fbad23a4640 1 -- 172.21.15.1:0/2717313101 learned_addr learned my addr 172.21.15.1:0/2717313101 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:26.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.404+0000 7fbad2ba5640 1 --2- 172.21.15.1:0/2717313101 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbacc150640 0x7fbacc1ee510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.404+0000 7fbad23a4640 1 -- 172.21.15.1:0/2717313101 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbacc150640 msgr2=0x7fbacc1ee510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.404+0000 7fbad1ba3640 1 --2- 172.21.15.1:0/2717313101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbacc0c75a0 0x7fbacc0c1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.404+0000 7fbad23a4640 1 --2- 172.21.15.1:0/2717313101 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbacc150640 0x7fbacc1ee510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.404+0000 7fbad23a4640 1 -- 172.21.15.1:0/2717313101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbacc0c75a0 msgr2=0x7fbacc0c1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.404+0000 7fbad23a4640 1 --2- 172.21.15.1:0/2717313101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbacc0c75a0 0x7fbacc0c1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.404+0000 7fbad23a4640 1 -- 172.21.15.1:0/2717313101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbacc0c44e0 con 0x7fbacc0a7f50 2024-03-06T07:23:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.404+0000 7fbad23a4640 1 --2- 172.21.15.1:0/2717313101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbacc0a7f50 0x7fbacc14d860 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fbac400b3f0 tx=0x7fbac400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.404+0000 7fbacaffd640 1 -- 172.21.15.1:0/2717313101 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac4015020 con 0x7fbacc0a7f50 2024-03-06T07:23:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.404+0000 7fbacaffd640 1 -- 172.21.15.1:0/2717313101 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac4011830 con 0x7fbacc0a7f50 2024-03-06T07:23:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.404+0000 7fbad4608640 1 -- 172.21.15.1:0/2717313101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbacc0c47c0 con 0x7fbacc0a7f50 2024-03-06T07:23:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.404+0000 7fbacaffd640 1 -- 172.21.15.1:0/2717313101 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac4011c20 con 0x7fbacc0a7f50 2024-03-06T07:23:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.404+0000 7fbad4608640 1 -- 172.21.15.1:0/2717313101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbacc0c5320 con 0x7fbacc0a7f50 2024-03-06T07:23:26.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.405+0000 7fbad4608640 1 -- 172.21.15.1:0/2717313101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbacc0c4a00 con 0x7fbacc0a7f50 2024-03-06T07:23:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.406+0000 7fbacaffd640 1 -- 172.21.15.1:0/2717313101 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbac4018020 con 0x7fbacc0a7f50 2024-03-06T07:23:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.406+0000 7fbacaffd640 1 --2- 172.21.15.1:0/2717313101 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbab806b4d0 0x7fbab806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.406+0000 7fbacaffd640 1 -- 172.21.15.1:0/2717313101 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7fbac408a040 con 0x7fbacc0a7f50 2024-03-06T07:23:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.406+0000 7fbad1ba3640 1 --2- 172.21.15.1:0/2717313101 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbab806b4d0 0x7fbab806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.407+0000 7fbad1ba3640 1 --2- 172.21.15.1:0/2717313101 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbab806b4d0 0x7fbab806d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbacc0add80 tx=0x7fbac0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.410+0000 7fbacaffd640 1 -- 172.21.15.1:0/2717313101 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbacc0c4a00 con 0x7fbacc0a7f50 2024-03-06T07:23:26.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.591+0000 7fbad4608640 1 -- 172.21.15.1:0/2717313101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbacc067fd0 con 0x7fbacc0a7f50 2024-03-06T07:23:26.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.593+0000 7fbacaffd640 1 -- 172.21.15.1:0/2717313101 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+17152 (secure 0 0 0) 0x7fbac401f030 con 0x7fbacc0a7f50 2024-03-06T07:23:26.595 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:26.595 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":83,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:23:26.260431+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:23:05.599046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:23:06.205539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6834/3919123103":"2024-03-07T07:23:23.886065+0000","172.21.15.98:6835/114998998":"2024-03-07T07:23:24.253819+0000","172.21.15.1:6832/3919123103":"2024-03-07T07:23:23.886065+0000","172.21.15.1:6834/1494116991":"2024-03-07T07:23:13.133394+0000","172.21.15.98:6837/114998998":"2024-03-07T07:23:24.253819+0000","172.21.15.1:6832/1494116991":"2024-03-07T07:23:13.133394+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:26.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.594+0000 7fbad4608640 1 -- 172.21.15.1:0/2717313101 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbab806b4d0 msgr2=0x7fbab806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.594+0000 7fbad4608640 1 --2- 172.21.15.1:0/2717313101 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbab806b4d0 0x7fbab806d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbacc0add80 tx=0x7fbac0008040 comp rx=0 tx=0).stop 2024-03-06T07:23:26.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.594+0000 7fbad4608640 1 -- 172.21.15.1:0/2717313101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbacc0a7f50 msgr2=0x7fbacc14d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.595+0000 7fbad4608640 1 --2- 172.21.15.1:0/2717313101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbacc0a7f50 0x7fbacc14d860 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fbac400b3f0 tx=0x7fbac400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.595+0000 7fbad4608640 1 -- 172.21.15.1:0/2717313101 shutdown_connections 2024-03-06T07:23:26.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.595+0000 7fbad4608640 1 --2- 172.21.15.1:0/2717313101 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbab806b4d0 0x7fbab806d990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.595+0000 7fbad4608640 1 --2- 172.21.15.1:0/2717313101 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbacc150640 0x7fbacc1ee510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.595+0000 7fbad4608640 1 --2- 172.21.15.1:0/2717313101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbacc0c75a0 0x7fbacc0c1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.595+0000 7fbad4608640 1 --2- 172.21.15.1:0/2717313101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbacc0a7f50 0x7fbacc14d860 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.595+0000 7fbad4608640 1 -- 172.21.15.1:0/2717313101 >> 172.21.15.1:0/2717313101 conn(0x7fbacc06b3e0 msgr2=0x7fbacc14e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.595+0000 7fbad4608640 1 -- 172.21.15.1:0/2717313101 shutdown_connections 2024-03-06T07:23:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.595+0000 7fbad4608640 1 -- 172.21.15.1:0/2717313101 wait complete. 2024-03-06T07:23:26.607 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-06T07:23:05.599046+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-06T07:23:06.205539+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:26.608 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:26.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.710+0000 7f9469fd7640 1 Processor -- start 2024-03-06T07:23:26.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.710+0000 7f9469fd7640 1 -- start start 2024-03-06T07:23:26.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.711+0000 7f9469fd7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94640c05f0 0x7f94640c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.711+0000 7f9469fd7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94640c31f0 con 0x7f946406f640 2024-03-06T07:23:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.711+0000 7f9469fd7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946406fbb0 con 0x7f94640c14c0 2024-03-06T07:23:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.711+0000 7f9469fd7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946406fe90 con 0x7f94640c05f0 2024-03-06T07:23:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.711+0000 7f9462ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f946406f640 0x7f94641551b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50190/0 (socket says 172.21.15.1:50190) 2024-03-06T07:23:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.711+0000 7f9462ffd640 1 -- 172.21.15.1:0/3447234549 learned_addr learned my addr 172.21.15.1:0/3447234549 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.711+0000 7f94637fe640 1 --2- 172.21.15.1:0/3447234549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94640c05f0 0x7f94640c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.711+0000 7f94627fc640 1 -- 172.21.15.1:0/3447234549 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812338197 0 0) 0x7f94640c31f0 con 0x7f946406f640 2024-03-06T07:23:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.711+0000 7f94627fc640 1 -- 172.21.15.1:0/3447234549 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f944c003520 con 0x7f946406f640 2024-03-06T07:23:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.712+0000 7f94627fc640 1 -- 172.21.15.1:0/3447234549 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 138291592 0 0) 0x7f944c003520 con 0x7f946406f640 2024-03-06T07:23:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.712+0000 7f94637fe640 1 -- 172.21.15.1:0/3447234549 >> v1:172.21.15.98:6789/0 conn(0x7f94640c14c0 legacy=0x7f94641558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.712+0000 7f94637fe640 1 -- 172.21.15.1:0/3447234549 >> v1:172.21.15.1:6789/0 conn(0x7f946406f640 legacy=0x7f94641551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.712+0000 7f94637fe640 1 -- 172.21.15.1:0/3447234549 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94640c3490 con 0x7f94640c05f0 2024-03-06T07:23:26.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.712+0000 7f94637fe640 1 --2- 172.21.15.1:0/3447234549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94640c05f0 0x7f94640c09d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f94540097d0 tx=0x7f945402b660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c57feac60c208a4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.713+0000 7f94627fc640 1 -- 172.21.15.1:0/3447234549 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9454037070 con 0x7f94640c05f0 2024-03-06T07:23:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.713+0000 7f94627fc640 1 -- 172.21.15.1:0/3447234549 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9454030510 con 0x7f94640c05f0 2024-03-06T07:23:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.713+0000 7f94627fc640 1 -- 172.21.15.1:0/3447234549 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945402be60 con 0x7f94640c05f0 2024-03-06T07:23:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.713+0000 7f9469fd7640 1 -- 172.21.15.1:0/3447234549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94640c05f0 msgr2=0x7f94640c09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.713+0000 7f9469fd7640 1 --2- 172.21.15.1:0/3447234549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94640c05f0 0x7f94640c09d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f94540097d0 tx=0x7f945402b660 comp rx=0 tx=0).stop 2024-03-06T07:23:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.713+0000 7f9469fd7640 1 -- 172.21.15.1:0/3447234549 shutdown_connections 2024-03-06T07:23:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.713+0000 7f9469fd7640 1 --2- 172.21.15.1:0/3447234549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94640c05f0 0x7f94640c09d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.713+0000 7f9469fd7640 1 -- 172.21.15.1:0/3447234549 >> 172.21.15.1:0/3447234549 conn(0x7f94640c3cc0 msgr2=0x7f94640c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.713+0000 7f9469fd7640 1 -- 172.21.15.1:0/3447234549 shutdown_connections 2024-03-06T07:23:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.713+0000 7f9469fd7640 1 -- 172.21.15.1:0/3447234549 wait complete. 2024-03-06T07:23:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.714+0000 7f9469fd7640 1 Processor -- start 2024-03-06T07:23:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.714+0000 7f9469fd7640 1 -- start start 2024-03-06T07:23:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.715+0000 7f9469fd7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f946406f640 0x7f9464143c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.715+0000 7f9469fd7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94640c05f0 0x7f94640c6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.715+0000 7f9469fd7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94640c14c0 0x7f94641574f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.715+0000 7f9469fd7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94640c1040 con 0x7f94640c14c0 2024-03-06T07:23:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.715+0000 7f9469fd7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946414f460 con 0x7f946406f640 2024-03-06T07:23:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.715+0000 7f9469fd7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94641609d0 con 0x7f94640c05f0 2024-03-06T07:23:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.715+0000 7f9463fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94640c14c0 0x7f94641574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.715+0000 7f9463fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94640c14c0 0x7f94641574f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45232/0 (socket says 172.21.15.1:45232) 2024-03-06T07:23:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.715+0000 7f9463fff640 1 -- 172.21.15.1:0/2312312631 learned_addr learned my addr 172.21.15.1:0/2312312631 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.715+0000 7f94637fe640 1 --2- 172.21.15.1:0/2312312631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f946406f640 0x7f9464143c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.715+0000 7f9462ffd640 1 --2- 172.21.15.1:0/2312312631 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94640c05f0 0x7f94640c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.716+0000 7f9463fff640 1 -- 172.21.15.1:0/2312312631 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94640c05f0 msgr2=0x7f94640c6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.716+0000 7f9463fff640 1 --2- 172.21.15.1:0/2312312631 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94640c05f0 0x7f94640c6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.716+0000 7f9463fff640 1 -- 172.21.15.1:0/2312312631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f946406f640 msgr2=0x7f9464143c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.716+0000 7f9463fff640 1 --2- 172.21.15.1:0/2312312631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f946406f640 0x7f9464143c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.716+0000 7f9463fff640 1 -- 172.21.15.1:0/2312312631 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9464158060 con 0x7f94640c14c0 2024-03-06T07:23:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.716+0000 7f94637fe640 1 --2- 172.21.15.1:0/2312312631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f946406f640 0x7f9464143c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.716+0000 7f9462ffd640 1 --2- 172.21.15.1:0/2312312631 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94640c05f0 0x7f94640c6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.716+0000 7f9463fff640 1 --2- 172.21.15.1:0/2312312631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94640c14c0 0x7f94641574f0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f945c00c840 tx=0x7f945c00cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.716+0000 7f944bfff640 1 -- 172.21.15.1:0/2312312631 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c017070 con 0x7f94640c14c0 2024-03-06T07:23:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.716+0000 7f944bfff640 1 -- 172.21.15.1:0/2312312631 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945c003da0 con 0x7f94640c14c0 2024-03-06T07:23:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.716+0000 7f9469fd7640 1 -- 172.21.15.1:0/2312312631 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94640c5bf0 con 0x7f94640c14c0 2024-03-06T07:23:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.716+0000 7f944bfff640 1 -- 172.21.15.1:0/2312312631 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c005290 con 0x7f94640c14c0 2024-03-06T07:23:26.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.716+0000 7f9469fd7640 1 -- 172.21.15.1:0/2312312631 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9464156d30 con 0x7f94640c14c0 2024-03-06T07:23:26.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.717+0000 7f9469fd7640 1 -- 172.21.15.1:0/2312312631 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946406b750 con 0x7f94640c14c0 2024-03-06T07:23:26.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.717+0000 7f944bfff640 1 -- 172.21.15.1:0/2312312631 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f945c01a070 con 0x7f94640c14c0 2024-03-06T07:23:26.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.718+0000 7f944bfff640 1 --2- 172.21.15.1:0/2312312631 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f945006b4d0 0x7f945006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.718+0000 7f944bfff640 1 -- 172.21.15.1:0/2312312631 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f945c089c20 con 0x7f94640c14c0 2024-03-06T07:23:26.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.718+0000 7f94637fe640 1 --2- 172.21.15.1:0/2312312631 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f945006b4d0 0x7f945006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.719+0000 7f94637fe640 1 --2- 172.21.15.1:0/2312312631 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f945006b4d0 0x7f945006d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9454034000 tx=0x7f9454008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.722+0000 7f944bfff640 1 -- 172.21.15.1:0/2312312631 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f946406b750 con 0x7f94640c14c0 2024-03-06T07:23:26.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.903+0000 7f9469fd7640 1 -- 172.21.15.1:0/2312312631 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9464067fd0 con 0x7f94640c14c0 2024-03-06T07:23:26.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.903+0000 7f944bfff640 1 -- 172.21.15.1:0/2312312631 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9464158060 con 0x7f94640c14c0 2024-03-06T07:23:26.905 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:23:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.905+0000 7f9469fd7640 1 -- 172.21.15.1:0/2312312631 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f945006b4d0 msgr2=0x7f945006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.905+0000 7f9469fd7640 1 --2- 172.21.15.1:0/2312312631 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f945006b4d0 0x7f945006d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9454034000 tx=0x7f9454008030 comp rx=0 tx=0).stop 2024-03-06T07:23:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.905+0000 7f9469fd7640 1 -- 172.21.15.1:0/2312312631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94640c14c0 msgr2=0x7f94641574f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.905+0000 7f9469fd7640 1 --2- 172.21.15.1:0/2312312631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94640c14c0 0x7f94641574f0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f945c00c840 tx=0x7f945c00cd00 comp rx=0 tx=0).stop 2024-03-06T07:23:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.906+0000 7f9469fd7640 1 -- 172.21.15.1:0/2312312631 shutdown_connections 2024-03-06T07:23:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.906+0000 7f9469fd7640 1 --2- 172.21.15.1:0/2312312631 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f945006b4d0 0x7f945006d990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.906+0000 7f9469fd7640 1 --2- 172.21.15.1:0/2312312631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94640c14c0 0x7f94641574f0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.906+0000 7f9469fd7640 1 --2- 172.21.15.1:0/2312312631 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94640c05f0 0x7f94640c6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.906+0000 7f9469fd7640 1 --2- 172.21.15.1:0/2312312631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f946406f640 0x7f9464143c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.906+0000 7f9469fd7640 1 -- 172.21.15.1:0/2312312631 >> 172.21.15.1:0/2312312631 conn(0x7f94640c3cc0 msgr2=0x7f94640a95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.906+0000 7f9469fd7640 1 -- 172.21.15.1:0/2312312631 shutdown_connections 2024-03-06T07:23:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:26.906+0000 7f9469fd7640 1 -- 172.21.15.1:0/2312312631 wait complete. 2024-03-06T07:23:26.918 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-06T07:23:27.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.020+0000 7f4f93e29640 1 Processor -- start 2024-03-06T07:23:27.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.020+0000 7f4f93e29640 1 -- start start 2024-03-06T07:23:27.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.021+0000 7f4f93e29640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4f8c152650 0x7f4f8c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.021+0000 7f4f93e29640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c0bf9d0 0x7f4f8c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.021+0000 7f4f93e29640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c0c2fc0 con 0x7f4f8c0c6b10 2024-03-06T07:23:27.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.021+0000 7f4f93e29640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f8c155800 con 0x7f4f8c152650 2024-03-06T07:23:27.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.021+0000 7f4f93e29640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c158da0 con 0x7f4f8c0bf9d0 2024-03-06T07:23:27.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.021+0000 7f4f913c4640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4f8c0c6b10 0x7f4f8c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50204/0 (socket says 172.21.15.1:50204) 2024-03-06T07:23:27.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.021+0000 7f4f913c4640 1 -- 172.21.15.1:0/374683200 learned_addr learned my addr 172.21.15.1:0/374683200 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:27.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.021+0000 7f4f923c6640 1 --2- 172.21.15.1:0/374683200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c0bf9d0 0x7f4f8c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.021+0000 7f4f90bc3640 1 -- 172.21.15.1:0/374683200 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668215980 0 0) 0x7f4f8c0c2fc0 con 0x7f4f8c0c6b10 2024-03-06T07:23:27.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.021+0000 7f4f91bc5640 1 --2- 172.21.15.1:0/374683200 >> v2:172.21.15.98:3300/0 conn(0x7f4f8c152650 0x7f4f8c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.021+0000 7f4f90bc3640 1 -- 172.21.15.1:0/374683200 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f74003520 con 0x7f4f8c0c6b10 2024-03-06T07:23:27.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f90bc3640 1 -- 172.21.15.1:0/374683200 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1368320394 0 0) 0x7f4f74003520 con 0x7f4f8c0c6b10 2024-03-06T07:23:27.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f90bc3640 1 -- 172.21.15.1:0/374683200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c0bf9d0 msgr2=0x7f4f8c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f90bc3640 1 --2- 172.21.15.1:0/374683200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c0bf9d0 0x7f4f8c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f90bc3640 1 -- 172.21.15.1:0/374683200 >> v2:172.21.15.98:3300/0 conn(0x7f4f8c152650 msgr2=0x7f4f8c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f90bc3640 1 --2- 172.21.15.1:0/374683200 >> v2:172.21.15.98:3300/0 conn(0x7f4f8c152650 0x7f4f8c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f90bc3640 1 -- 172.21.15.1:0/374683200 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c0c6690 con 0x7f4f8c0c6b10 2024-03-06T07:23:27.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f90bc3640 1 -- 172.21.15.1:0/374683200 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f4f80001690 con 0x7f4f8c0c6b10 2024-03-06T07:23:27.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f90bc3640 1 -- 172.21.15.1:0/374683200 >> v1:172.21.15.1:6789/0 conn(0x7f4f8c0c6b10 legacy=0x7f4f8c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f90bc3640 1 --2- 172.21.15.1:0/374683200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f740039b0 0x7f4f74003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f90bc3640 1 --2- 172.21.15.1:0/374683200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f74004800 0x7f4f74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f90bc3640 1 --2- 172.21.15.1:0/374683200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c0bf9d0 0x7f4f740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f90bc3640 1 -- 172.21.15.1:0/374683200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c155800 con 0x7f4f740039b0 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f90bc3640 1 -- 172.21.15.1:0/374683200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c158da0 con 0x7f4f74004800 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f91bc5640 1 --2- 172.21.15.1:0/374683200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f740039b0 0x7f4f74003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f90bc3640 1 -- 172.21.15.1:0/374683200 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7400b9f0 con 0x7f4f8c0bf9d0 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f913c4640 1 --2- 172.21.15.1:0/374683200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f74004800 0x7f4f74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f923c6640 1 --2- 172.21.15.1:0/374683200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c0bf9d0 0x7f4f740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f91bc5640 1 -- 172.21.15.1:0/374683200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c0bf9d0 msgr2=0x7f4f740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f91bc5640 1 --2- 172.21.15.1:0/374683200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c0bf9d0 0x7f4f740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f91bc5640 1 -- 172.21.15.1:0/374683200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f74004800 msgr2=0x7f4f74004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f91bc5640 1 --2- 172.21.15.1:0/374683200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f74004800 0x7f4f74004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f91bc5640 1 -- 172.21.15.1:0/374683200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f74003520 con 0x7f4f740039b0 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f913c4640 1 --2- 172.21.15.1:0/374683200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f74004800 0x7f4f74004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.022+0000 7f4f923c6640 1 --2- 172.21.15.1:0/374683200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c0bf9d0 0x7f4f740051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.023+0000 7f4f91bc5640 1 --2- 172.21.15.1:0/374683200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f740039b0 0x7f4f74003dd0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f4f84009ac0 tx=0x7f4f74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212a825fd701336c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.023+0000 7f4f90bc3640 1 -- 172.21.15.1:0/374683200 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84039070 con 0x7f4f740039b0 2024-03-06T07:23:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.023+0000 7f4f90bc3640 1 -- 172.21.15.1:0/374683200 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8402c6d0 con 0x7f4f740039b0 2024-03-06T07:23:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.023+0000 7f4f90bc3640 1 -- 172.21.15.1:0/374683200 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84009690 con 0x7f4f740039b0 2024-03-06T07:23:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.024+0000 7f4f93e29640 1 -- 172.21.15.1:0/374683200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f740039b0 msgr2=0x7f4f74003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.024+0000 7f4f93e29640 1 --2- 172.21.15.1:0/374683200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f740039b0 0x7f4f74003dd0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f4f84009ac0 tx=0x7f4f74005830 comp rx=0 tx=0).stop 2024-03-06T07:23:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.024+0000 7f4f923c6640 1 -- 172.21.15.1:0/374683200 reap_dead start 2024-03-06T07:23:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.024+0000 7f4f93e29640 1 -- 172.21.15.1:0/374683200 shutdown_connections 2024-03-06T07:23:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.024+0000 7f4f93e29640 1 -- 172.21.15.1:0/374683200 >> 172.21.15.1:0/374683200 conn(0x7f4f8c066ee0 msgr2=0x7f4f8c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.024+0000 7f4f93e29640 1 -- 172.21.15.1:0/374683200 shutdown_connections 2024-03-06T07:23:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.024+0000 7f4f93e29640 1 -- 172.21.15.1:0/374683200 wait complete. 2024-03-06T07:23:27.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.025+0000 7f4f93e29640 1 Processor -- start 2024-03-06T07:23:27.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.025+0000 7f4f93e29640 1 -- start start 2024-03-06T07:23:27.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.025+0000 7f4f93e29640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f740039b0 0x7f4f8c0ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.025+0000 7f4f93e29640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f8c14fb10 0x7f4f8c14b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f93e29640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c14c160 0x7f4f8c1536b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f93e29640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c1551b0 con 0x7f4f8c14fb10 2024-03-06T07:23:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f93e29640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c0c6fa0 con 0x7f4f740039b0 2024-03-06T07:23:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f93e29640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c155490 con 0x7f4f8c14c160 2024-03-06T07:23:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f913c4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f8c14fb10 0x7f4f8c14b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f913c4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f8c14fb10 0x7f4f8c14b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45258/0 (socket says 172.21.15.1:45258) 2024-03-06T07:23:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f913c4640 1 -- 172.21.15.1:0/2698566982 learned_addr learned my addr 172.21.15.1:0/2698566982 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f923c6640 1 --2- 172.21.15.1:0/2698566982 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c14c160 0x7f4f8c1536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f91bc5640 1 --2- 172.21.15.1:0/2698566982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f740039b0 0x7f4f8c0ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f913c4640 1 -- 172.21.15.1:0/2698566982 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c14c160 msgr2=0x7f4f8c1536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f913c4640 1 --2- 172.21.15.1:0/2698566982 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c14c160 0x7f4f8c1536b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f913c4640 1 -- 172.21.15.1:0/2698566982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f740039b0 msgr2=0x7f4f8c0ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f913c4640 1 --2- 172.21.15.1:0/2698566982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f740039b0 0x7f4f8c0ca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f913c4640 1 -- 172.21.15.1:0/2698566982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c14bbb0 con 0x7f4f8c14fb10 2024-03-06T07:23:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f923c6640 1 --2- 172.21.15.1:0/2698566982 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c14c160 0x7f4f8c1536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f91bc5640 1 --2- 172.21.15.1:0/2698566982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f740039b0 0x7f4f8c0ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:27.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f913c4640 1 --2- 172.21.15.1:0/2698566982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f8c14fb10 0x7f4f8c14b670 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000e7b0 tx=0x7f4f8000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f8a7fc640 1 -- 172.21.15.1:0/2698566982 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8000c870 con 0x7f4f8c14fb10 2024-03-06T07:23:27.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f8a7fc640 1 -- 172.21.15.1:0/2698566982 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f80018070 con 0x7f4f8c14fb10 2024-03-06T07:23:27.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f93e29640 1 -- 172.21.15.1:0/2698566982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f8c14b3d0 con 0x7f4f8c14fb10 2024-03-06T07:23:27.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f8a7fc640 1 -- 172.21.15.1:0/2698566982 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8000cc30 con 0x7f4f8c14fb10 2024-03-06T07:23:27.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.026+0000 7f4f93e29640 1 -- 172.21.15.1:0/2698566982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f8c1ebbe0 con 0x7f4f8c14fb10 2024-03-06T07:23:27.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.027+0000 7f4f93e29640 1 -- 172.21.15.1:0/2698566982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f8c0c2f50 con 0x7f4f8c14fb10 2024-03-06T07:23:27.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.028+0000 7f4f8a7fc640 1 -- 172.21.15.1:0/2698566982 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f80013420 con 0x7f4f8c14fb10 2024-03-06T07:23:27.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.028+0000 7f4f8a7fc640 1 --2- 172.21.15.1:0/2698566982 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4f7806b500 0x7f4f7806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.028+0000 7f4f8a7fc640 1 -- 172.21.15.1:0/2698566982 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f4f80089f20 con 0x7f4f8c14fb10 2024-03-06T07:23:27.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.028+0000 7f4f91bc5640 1 --2- 172.21.15.1:0/2698566982 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4f7806b500 0x7f4f7806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.029+0000 7f4f91bc5640 1 --2- 172.21.15.1:0/2698566982 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4f7806b500 0x7f4f7806d9c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4f84036000 tx=0x7f4f84030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.032+0000 7f4f8a7fc640 1 -- 172.21.15.1:0/2698566982 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f8c0c2f50 con 0x7f4f8c14fb10 2024-03-06T07:23:27.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.211+0000 7f4f93e29640 1 -- 172.21.15.1:0/2698566982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4f8c067fd0 con 0x7f4f8c14fb10 2024-03-06T07:23:27.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.212+0000 7f4f8a7fc640 1 -- 172.21.15.1:0/2698566982 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v83) v1 ==== 105+0+11 (secure 0 0 0) 0x7f4f8c14bbb0 con 0x7f4f8c14fb10 2024-03-06T07:23:27.214 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.214+0000 7f4f93e29640 1 -- 172.21.15.1:0/2698566982 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4f7806b500 msgr2=0x7f4f7806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.214+0000 7f4f93e29640 1 --2- 172.21.15.1:0/2698566982 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4f7806b500 0x7f4f7806d9c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4f84036000 tx=0x7f4f84030000 comp rx=0 tx=0).stop 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.214+0000 7f4f93e29640 1 -- 172.21.15.1:0/2698566982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f8c14fb10 msgr2=0x7f4f8c14b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.214+0000 7f4f93e29640 1 --2- 172.21.15.1:0/2698566982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f8c14fb10 0x7f4f8c14b670 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000e7b0 tx=0x7f4f8000ec70 comp rx=0 tx=0).stop 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.214+0000 7f4f93e29640 1 -- 172.21.15.1:0/2698566982 shutdown_connections 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.214+0000 7f4f93e29640 1 --2- 172.21.15.1:0/2698566982 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4f7806b500 0x7f4f7806d9c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.214+0000 7f4f93e29640 1 --2- 172.21.15.1:0/2698566982 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f8c14c160 0x7f4f8c1536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.214+0000 7f4f93e29640 1 --2- 172.21.15.1:0/2698566982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f8c14fb10 0x7f4f8c14b670 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.214+0000 7f4f93e29640 1 --2- 172.21.15.1:0/2698566982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f740039b0 0x7f4f8c0ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.214+0000 7f4f93e29640 1 -- 172.21.15.1:0/2698566982 >> 172.21.15.1:0/2698566982 conn(0x7f4f8c066ee0 msgr2=0x7f4f8c0cabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.214+0000 7f4f93e29640 1 -- 172.21.15.1:0/2698566982 shutdown_connections 2024-03-06T07:23:27.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.214+0000 7f4f93e29640 1 -- 172.21.15.1:0/2698566982 wait complete. 2024-03-06T07:23:27.227 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-06T07:23:27.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.331+0000 7f1c324af640 1 Processor -- start 2024-03-06T07:23:27.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.331+0000 7f1c324af640 1 -- start start 2024-03-06T07:23:27.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.331+0000 7f1c324af640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1c2c0af6f0 0x7f1c2c0c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.331+0000 7f1c324af640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c0c7180 con 0x7f1c2c0c7930 2024-03-06T07:23:27.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.331+0000 7f1c324af640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c155460 con 0x7f1c2c152820 2024-03-06T07:23:27.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.331+0000 7f1c324af640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c2c0c6bc0 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.332+0000 7f1c2b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1c2c0af6f0 0x7f1c2c0c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.332+0000 7f1c30a4c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1c2c0c7930 0x7f1c2c15fb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50212/0 (socket says 172.21.15.1:50212) 2024-03-06T07:23:27.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.332+0000 7f1c30a4c640 1 -- 172.21.15.1:0/163711704 learned_addr learned my addr 172.21.15.1:0/163711704 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:27.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.332+0000 7f1c2b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1c2c0af6f0 0x7f1c2c0c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45260/0 (socket says 172.21.15.1:45260) 2024-03-06T07:23:27.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.332+0000 7f1c2b7fe640 1 -- 172.21.15.1:0/163711704 >> v1:172.21.15.98:6789/0 conn(0x7f1c2c152820 legacy=0x7f1c2c0c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.332+0000 7f1c2b7fe640 1 -- 172.21.15.1:0/163711704 >> v1:172.21.15.1:6789/0 conn(0x7f1c2c0c7930 legacy=0x7f1c2c15fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.332+0000 7f1c2affd640 1 -- 172.21.15.1:0/163711704 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871202121 0 0) 0x7f1c2c0c7180 con 0x7f1c2c0c7930 2024-03-06T07:23:27.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.332+0000 7f1c2b7fe640 1 -- 172.21.15.1:0/163711704 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c0c6ea0 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.333+0000 7f1c2b7fe640 1 --2- 172.21.15.1:0/163711704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2c0af6f0 0x7f1c2c0c73f0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f1c200097a0 tx=0x7f1c2002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3d80372263d52a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.333+0000 7f1c2affd640 1 -- 172.21.15.1:0/163711704 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c20035020 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.333+0000 7f1c2affd640 1 -- 172.21.15.1:0/163711704 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c20031830 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.333+0000 7f1c2affd640 1 -- 172.21.15.1:0/163711704 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c20031c00 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.333+0000 7f1c324af640 1 -- 172.21.15.1:0/163711704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2c0af6f0 msgr2=0x7f1c2c0c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.333+0000 7f1c324af640 1 --2- 172.21.15.1:0/163711704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2c0af6f0 0x7f1c2c0c73f0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f1c200097a0 tx=0x7f1c2002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.333+0000 7f1c324af640 1 -- 172.21.15.1:0/163711704 shutdown_connections 2024-03-06T07:23:27.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.333+0000 7f1c324af640 1 --2- 172.21.15.1:0/163711704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2c0af6f0 0x7f1c2c0c73f0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.333+0000 7f1c324af640 1 -- 172.21.15.1:0/163711704 >> 172.21.15.1:0/163711704 conn(0x7f1c2c072720 msgr2=0x7f1c2c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.333+0000 7f1c324af640 1 -- 172.21.15.1:0/163711704 shutdown_connections 2024-03-06T07:23:27.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.333+0000 7f1c324af640 1 -- 172.21.15.1:0/163711704 wait complete. 2024-03-06T07:23:27.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.334+0000 7f1c324af640 1 Processor -- start 2024-03-06T07:23:27.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.335+0000 7f1c324af640 1 -- start start 2024-03-06T07:23:27.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.335+0000 7f1c324af640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2c0af6f0 0x7f1c2c0c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.335+0000 7f1c324af640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c2c0c7930 0x7f1c2c1eaf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.335+0000 7f1c324af640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c2c152820 0x7f1c2c1eb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.335+0000 7f1c324af640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c152c00 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.335+0000 7f1c324af640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c0c0840 con 0x7f1c2c0c7930 2024-03-06T07:23:27.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.335+0000 7f1c324af640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c152d80 con 0x7f1c2c152820 2024-03-06T07:23:27.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.335+0000 7f1c2bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2c0af6f0 0x7f1c2c0c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.335+0000 7f1c2bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2c0af6f0 0x7f1c2c0c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45276/0 (socket says 172.21.15.1:45276) 2024-03-06T07:23:27.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.335+0000 7f1c2bfff640 1 -- 172.21.15.1:0/2822622402 learned_addr learned my addr 172.21.15.1:0/2822622402 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.336+0000 7f1c2b7fe640 1 --2- 172.21.15.1:0/2822622402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c2c0c7930 0x7f1c2c1eaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.336+0000 7f1c30a4c640 1 --2- 172.21.15.1:0/2822622402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c2c152820 0x7f1c2c1eb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.336+0000 7f1c2bfff640 1 -- 172.21.15.1:0/2822622402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c2c152820 msgr2=0x7f1c2c1eb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.336+0000 7f1c2bfff640 1 --2- 172.21.15.1:0/2822622402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c2c152820 0x7f1c2c1eb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.336+0000 7f1c2bfff640 1 -- 172.21.15.1:0/2822622402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c2c0c7930 msgr2=0x7f1c2c1eaf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.336+0000 7f1c2bfff640 1 --2- 172.21.15.1:0/2822622402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c2c0c7930 0x7f1c2c1eaf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.336+0000 7f1c2bfff640 1 -- 172.21.15.1:0/2822622402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c1592c0 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.336+0000 7f1c2bfff640 1 --2- 172.21.15.1:0/2822622402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2c0af6f0 0x7f1c2c0c3e30 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c00b3f0 tx=0x7f1c1c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.336+0000 7f1c28ff9640 1 -- 172.21.15.1:0/2822622402 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c015020 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.336+0000 7f1c28ff9640 1 -- 172.21.15.1:0/2822622402 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1c011830 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.336+0000 7f1c324af640 1 -- 172.21.15.1:0/2822622402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2c1ee700 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.336+0000 7f1c28ff9640 1 -- 172.21.15.1:0/2822622402 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c011c20 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.336+0000 7f1c324af640 1 -- 172.21.15.1:0/2822622402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2c1eecc0 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.337+0000 7f1c324af640 1 -- 172.21.15.1:0/2822622402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2c0686e0 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.338+0000 7f1c28ff9640 1 -- 172.21.15.1:0/2822622402 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c1c018020 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.338+0000 7f1c28ff9640 1 --2- 172.21.15.1:0/2822622402 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c1806b4d0 0x7f1c1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.339+0000 7f1c28ff9640 1 -- 172.21.15.1:0/2822622402 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f1c1c08a180 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.339+0000 7f1c2b7fe640 1 --2- 172.21.15.1:0/2822622402 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c1806b4d0 0x7f1c1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.339+0000 7f1c2b7fe640 1 --2- 172.21.15.1:0/2822622402 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c1806b4d0 0x7f1c1806d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1c20036000 tx=0x7f1c2003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.343+0000 7f1c28ff9640 1 -- 172.21.15.1:0/2822622402 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c1c01f030 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.522+0000 7f1c324af640 1 -- 172.21.15.1:0/2822622402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7f1c2c0c5460 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.523+0000 7f1c28ff9640 1 -- 172.21.15.1:0/2822622402 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v83) v1 ==== 101+0+11 (secure 0 0 0) 0x7f1c1c0172c0 con 0x7f1c2c0af6f0 2024-03-06T07:23:27.525 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:23:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.524+0000 7f1c324af640 1 -- 172.21.15.1:0/2822622402 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c1806b4d0 msgr2=0x7f1c1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.524+0000 7f1c324af640 1 --2- 172.21.15.1:0/2822622402 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c1806b4d0 0x7f1c1806d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1c20036000 tx=0x7f1c2003c040 comp rx=0 tx=0).stop 2024-03-06T07:23:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.524+0000 7f1c324af640 1 -- 172.21.15.1:0/2822622402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2c0af6f0 msgr2=0x7f1c2c0c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.524+0000 7f1c324af640 1 --2- 172.21.15.1:0/2822622402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2c0af6f0 0x7f1c2c0c3e30 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c00b3f0 tx=0x7f1c1c00b8b0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.525+0000 7f1c324af640 1 -- 172.21.15.1:0/2822622402 shutdown_connections 2024-03-06T07:23:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.525+0000 7f1c324af640 1 --2- 172.21.15.1:0/2822622402 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c1806b4d0 0x7f1c1806d990 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.525+0000 7f1c324af640 1 --2- 172.21.15.1:0/2822622402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c2c152820 0x7f1c2c1eb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.525+0000 7f1c324af640 1 --2- 172.21.15.1:0/2822622402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c2c0c7930 0x7f1c2c1eaf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.525+0000 7f1c324af640 1 --2- 172.21.15.1:0/2822622402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2c0af6f0 0x7f1c2c0c3e30 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.525+0000 7f1c324af640 1 -- 172.21.15.1:0/2822622402 >> 172.21.15.1:0/2822622402 conn(0x7f1c2c072720 msgr2=0x7f1c2c14fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.525+0000 7f1c324af640 1 -- 172.21.15.1:0/2822622402 shutdown_connections 2024-03-06T07:23:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.525+0000 7f1c324af640 1 -- 172.21.15.1:0/2822622402 wait complete. 2024-03-06T07:23:27.537 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.659+0000 7f01ab93e640 1 Processor -- start 2024-03-06T07:23:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.659+0000 7f01ab93e640 1 -- start start 2024-03-06T07:23:27.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.660+0000 7f01ab93e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f01a40c4dd0 0x7f01a40c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.660+0000 7f01ab93e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01a40c79c0 0x7f01a4152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.660+0000 7f01ab93e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a41566e0 con 0x7f01a40c5d80 2024-03-06T07:23:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.660+0000 7f01ab93e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a4156490 con 0x7f01a40c4dd0 2024-03-06T07:23:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.660+0000 7f01ab93e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a4157010 con 0x7f01a40c79c0 2024-03-06T07:23:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.660+0000 7f01a96da640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f01a40c4dd0 0x7f01a40c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.660+0000 7f01a96da640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f01a40c4dd0 0x7f01a40c5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45282/0 (socket says 172.21.15.1:45282) 2024-03-06T07:23:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.660+0000 7f01a96da640 1 -- 172.21.15.1:0/1163083024 learned_addr learned my addr 172.21.15.1:0/1163083024 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.661+0000 7f01a8ed9640 1 --2- 172.21.15.1:0/1163083024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01a40c79c0 0x7f01a4152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.661+0000 7f01a96da640 1 -- 172.21.15.1:0/1163083024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01a40c79c0 msgr2=0x7f01a4152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.661+0000 7f01a96da640 1 --2- 172.21.15.1:0/1163083024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01a40c79c0 0x7f01a4152ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.661+0000 7f01a96da640 1 -- 172.21.15.1:0/1163083024 >> v1:172.21.15.98:6789/0 conn(0x7f01a40c5d80 legacy=0x7f01a41533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.661+0000 7f01a96da640 1 -- 172.21.15.1:0/1163083024 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a41569c0 con 0x7f01a40c4dd0 2024-03-06T07:23:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.661+0000 7f01a96da640 1 --2- 172.21.15.1:0/1163083024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a40c4dd0 0x7f01a40c5840 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f01a0009870 tx=0x7f01a002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d29ab22209f3393 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.662+0000 7f019bfff640 1 -- 172.21.15.1:0/1163083024 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a0038070 con 0x7f01a40c4dd0 2024-03-06T07:23:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.662+0000 7f019bfff640 1 -- 172.21.15.1:0/1163083024 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a0031740 con 0x7f01a40c4dd0 2024-03-06T07:23:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.662+0000 7f019bfff640 1 -- 172.21.15.1:0/1163083024 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a002f050 con 0x7f01a40c4dd0 2024-03-06T07:23:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.662+0000 7f01ab93e640 1 -- 172.21.15.1:0/1163083024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a40c4dd0 msgr2=0x7f01a40c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.662+0000 7f01ab93e640 1 --2- 172.21.15.1:0/1163083024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a40c4dd0 0x7f01a40c5840 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f01a0009870 tx=0x7f01a002b7e0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.662+0000 7f01ab93e640 1 -- 172.21.15.1:0/1163083024 shutdown_connections 2024-03-06T07:23:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.662+0000 7f01ab93e640 1 --2- 172.21.15.1:0/1163083024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01a40c79c0 0x7f01a4152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.662+0000 7f01ab93e640 1 --2- 172.21.15.1:0/1163083024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a40c4dd0 0x7f01a40c5840 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.662+0000 7f01ab93e640 1 -- 172.21.15.1:0/1163083024 >> 172.21.15.1:0/1163083024 conn(0x7f01a406b410 msgr2=0x7f01a406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.662+0000 7f01ab93e640 1 -- 172.21.15.1:0/1163083024 shutdown_connections 2024-03-06T07:23:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.662+0000 7f01ab93e640 1 -- 172.21.15.1:0/1163083024 wait complete. 2024-03-06T07:23:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.663+0000 7f01ab93e640 1 Processor -- start 2024-03-06T07:23:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.663+0000 7f01ab93e640 1 -- start start 2024-03-06T07:23:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.664+0000 7f01ab93e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a40c4dd0 0x7f01a4155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.664+0000 7f01ab93e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01a40c5d80 0x7f01a41eeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.664+0000 7f01ab93e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01a40c1e90 0x7f01a41eaed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.664+0000 7f01ab93e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a40715f0 con 0x7f01a40c4dd0 2024-03-06T07:23:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.664+0000 7f01ab93e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a4153af0 con 0x7f01a40c1e90 2024-03-06T07:23:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.664+0000 7f01ab93e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a4071770 con 0x7f01a40c5d80 2024-03-06T07:23:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.664+0000 7f01a96da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a40c4dd0 0x7f01a4155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.664+0000 7f01a96da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a40c4dd0 0x7f01a4155bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45292/0 (socket says 172.21.15.1:45292) 2024-03-06T07:23:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.664+0000 7f01a96da640 1 -- 172.21.15.1:0/3828032584 learned_addr learned my addr 172.21.15.1:0/3828032584 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.664+0000 7f01a8ed9640 1 --2- 172.21.15.1:0/3828032584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01a40c5d80 0x7f01a41eeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.664+0000 7f01a9edb640 1 --2- 172.21.15.1:0/3828032584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01a40c1e90 0x7f01a41eaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.664+0000 7f01a96da640 1 -- 172.21.15.1:0/3828032584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01a40c5d80 msgr2=0x7f01a41eeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.664+0000 7f01a96da640 1 --2- 172.21.15.1:0/3828032584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01a40c5d80 0x7f01a41eeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.664+0000 7f01a96da640 1 -- 172.21.15.1:0/3828032584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01a40c1e90 msgr2=0x7f01a41eaed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.664+0000 7f01a96da640 1 --2- 172.21.15.1:0/3828032584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01a40c1e90 0x7f01a41eaed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.664+0000 7f01a96da640 1 -- 172.21.15.1:0/3828032584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a406c9a0 con 0x7f01a40c4dd0 2024-03-06T07:23:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.665+0000 7f01a9edb640 1 --2- 172.21.15.1:0/3828032584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01a40c1e90 0x7f01a41eaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.665+0000 7f01a96da640 1 --2- 172.21.15.1:0/3828032584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a40c4dd0 0x7f01a4155bf0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f01a0009870 tx=0x7f01a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.665+0000 7f0199ffb640 1 -- 172.21.15.1:0/3828032584 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a0042070 con 0x7f01a40c4dd0 2024-03-06T07:23:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.665+0000 7f0199ffb640 1 -- 172.21.15.1:0/3828032584 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a0033a70 con 0x7f01a40c4dd0 2024-03-06T07:23:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.665+0000 7f01ab93e640 1 -- 172.21.15.1:0/3828032584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a406cc80 con 0x7f01a40c4dd0 2024-03-06T07:23:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.665+0000 7f0199ffb640 1 -- 172.21.15.1:0/3828032584 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a0038040 con 0x7f01a40c4dd0 2024-03-06T07:23:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.665+0000 7f01ab93e640 1 -- 172.21.15.1:0/3828032584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a406dfd0 con 0x7f01a40c4dd0 2024-03-06T07:23:27.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.666+0000 7f01ab93e640 1 -- 172.21.15.1:0/3828032584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a41eac70 con 0x7f01a40c4dd0 2024-03-06T07:23:27.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.666+0000 7f0199ffb640 1 -- 172.21.15.1:0/3828032584 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01a0045070 con 0x7f01a40c4dd0 2024-03-06T07:23:27.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.667+0000 7f0199ffb640 1 --2- 172.21.15.1:0/3828032584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f018806b4d0 0x7f018806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.667+0000 7f0199ffb640 1 -- 172.21.15.1:0/3828032584 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f01a00ac180 con 0x7f01a40c4dd0 2024-03-06T07:23:27.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.667+0000 7f01a8ed9640 1 --2- 172.21.15.1:0/3828032584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f018806b4d0 0x7f018806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.668+0000 7f01a8ed9640 1 --2- 172.21.15.1:0/3828032584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f018806b4d0 0x7f018806d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f01a41535d0 tx=0x7f0194008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.671+0000 7f0199ffb640 1 -- 172.21.15.1:0/3828032584 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01a004c030 con 0x7f01a40c4dd0 2024-03-06T07:23:27.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.889+0000 7f01ab93e640 1 -- 172.21.15.1:0/3828032584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01a4067fd0 con 0x7f01a40c4dd0 2024-03-06T07:23:27.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.890+0000 7f0199ffb640 1 -- 172.21.15.1:0/3828032584 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f01a0045e40 con 0x7f01a40c4dd0 2024-03-06T07:23:27.892 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:27.892 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":71,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.98:6836/4069609682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4069609682},{"type":"v1","addr":"172.21.15.98:6836","nonce":4069609682}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6196,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3962693342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3962693342},{"type":"v1","addr":"172.21.15.1:6834","nonce":3962693342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6211,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3130835916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3130835916},{"type":"v1","addr":"172.21.15.98:6837","nonce":3130835916}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:07.074540+0000","modified":"2024-03-06T07:23:27.261401+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5827":{"gid":5827,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":11,"addr":"172.21.15.1:6835/2685225325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2685225325},{"type":"v1","addr":"172.21.15.1:6835","nonce":2685225325}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5827,"qdb_cluster":[5827]},"id":6}]} 2024-03-06T07:23:27.892 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 71 2024-03-06T07:23:27.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.892+0000 7f01ab93e640 1 -- 172.21.15.1:0/3828032584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f018806b4d0 msgr2=0x7f018806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.892+0000 7f01ab93e640 1 --2- 172.21.15.1:0/3828032584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f018806b4d0 0x7f018806d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f01a41535d0 tx=0x7f0194008040 comp rx=0 tx=0).stop 2024-03-06T07:23:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.892+0000 7f01ab93e640 1 -- 172.21.15.1:0/3828032584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a40c4dd0 msgr2=0x7f01a4155bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.892+0000 7f01ab93e640 1 --2- 172.21.15.1:0/3828032584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a40c4dd0 0x7f01a4155bf0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f01a0009870 tx=0x7f01a00046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.892+0000 7f01ab93e640 1 -- 172.21.15.1:0/3828032584 shutdown_connections 2024-03-06T07:23:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.892+0000 7f01ab93e640 1 --2- 172.21.15.1:0/3828032584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f018806b4d0 0x7f018806d990 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.892+0000 7f01ab93e640 1 --2- 172.21.15.1:0/3828032584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01a40c1e90 0x7f01a41eaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.892+0000 7f01ab93e640 1 --2- 172.21.15.1:0/3828032584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01a40c5d80 0x7f01a41eeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.892+0000 7f01ab93e640 1 --2- 172.21.15.1:0/3828032584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a40c4dd0 0x7f01a4155bf0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.892+0000 7f01ab93e640 1 -- 172.21.15.1:0/3828032584 >> 172.21.15.1:0/3828032584 conn(0x7f01a406b410 msgr2=0x7f01a40c3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.892+0000 7f01ab93e640 1 -- 172.21.15.1:0/3828032584 shutdown_connections 2024-03-06T07:23:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:27.892+0000 7f01ab93e640 1 -- 172.21.15.1:0/3828032584 wait complete. 2024-03-06T07:23:27.905 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:27.906 INFO:tasks.ceph.mds.a.smithi001.stderr:2024-03-06T07:23:27.903+0000 7fc629b2f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:28.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.004+0000 7f766faf4640 1 Processor -- start 2024-03-06T07:23:28.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.005+0000 7f766faf4640 1 -- start start 2024-03-06T07:23:28.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.005+0000 7f766faf4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f76680c2b30 0x7f76680724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.005+0000 7f766faf4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7668151dd0 con 0x7f76680a5fa0 2024-03-06T07:23:28.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.005+0000 7f766faf4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7668156190 con 0x7f7668152b70 2024-03-06T07:23:28.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.005+0000 7f766faf4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76681526e0 con 0x7f76680c2b30 2024-03-06T07:23:28.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.005+0000 7f766e091640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f76680a5fa0 0x7f766815c750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50216/0 (socket says 172.21.15.1:50216) 2024-03-06T07:23:28.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.005+0000 7f766e091640 1 -- 172.21.15.1:0/153987754 learned_addr learned my addr 172.21.15.1:0/153987754 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:28.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766d890640 1 --2- 172.21.15.1:0/153987754 >> v2:172.21.15.98:3300/0 conn(0x7f76680c2b30 0x7f76680724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766c88e640 1 -- 172.21.15.1:0/153987754 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265011342 0 0) 0x7f7668151dd0 con 0x7f76680a5fa0 2024-03-06T07:23:28.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766c88e640 1 -- 172.21.15.1:0/153987754 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7654003520 con 0x7f76680a5fa0 2024-03-06T07:23:28.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766c88e640 1 -- 172.21.15.1:0/153987754 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 389633548 0 0) 0x7f7654003520 con 0x7f76680a5fa0 2024-03-06T07:23:28.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766c88e640 1 -- 172.21.15.1:0/153987754 >> v2:172.21.15.98:3300/0 conn(0x7f76680c2b30 msgr2=0x7f76680724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766c88e640 1 --2- 172.21.15.1:0/153987754 >> v2:172.21.15.98:3300/0 conn(0x7f76680c2b30 0x7f76680724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766c88e640 1 -- 172.21.15.1:0/153987754 >> v1:172.21.15.98:6789/0 conn(0x7f7668152b70 legacy=0x7f76680c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766c88e640 1 -- 172.21.15.1:0/153987754 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7668152090 con 0x7f76680a5fa0 2024-03-06T07:23:28.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766c88e640 1 -- 172.21.15.1:0/153987754 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f7660001670 con 0x7f76680a5fa0 2024-03-06T07:23:28.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766d890640 1 --2- 172.21.15.1:0/153987754 >> v2:172.21.15.98:3300/0 conn(0x7f76680c2b30 0x7f76680724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:28.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766c88e640 1 -- 172.21.15.1:0/153987754 >> v1:172.21.15.1:6789/0 conn(0x7f76680a5fa0 legacy=0x7f766815c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766c88e640 1 --2- 172.21.15.1:0/153987754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540039b0 0x7f7654003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766c88e640 1 --2- 172.21.15.1:0/153987754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7654004800 0x7f7654004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766c88e640 1 --2- 172.21.15.1:0/153987754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76540051c0 0x7f765400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766c88e640 1 -- 172.21.15.1:0/153987754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76681526e0 con 0x7f76540051c0 2024-03-06T07:23:28.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766c88e640 1 -- 172.21.15.1:0/153987754 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765400c110 con 0x7f7654004800 2024-03-06T07:23:28.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766c88e640 1 -- 172.21.15.1:0/153987754 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765400e520 con 0x7f76540039b0 2024-03-06T07:23:28.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766e091640 1 --2- 172.21.15.1:0/153987754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76540051c0 0x7f765400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766d890640 1 --2- 172.21.15.1:0/153987754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540039b0 0x7f7654003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766d08f640 1 --2- 172.21.15.1:0/153987754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7654004800 0x7f7654004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.006+0000 7f766e091640 1 -- 172.21.15.1:0/153987754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540039b0 msgr2=0x7f7654003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.007+0000 7f766e091640 1 --2- 172.21.15.1:0/153987754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540039b0 0x7f7654003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.007+0000 7f766e091640 1 -- 172.21.15.1:0/153987754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7654004800 msgr2=0x7f7654004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.007+0000 7f766e091640 1 --2- 172.21.15.1:0/153987754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7654004800 0x7f7654004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.007+0000 7f766e091640 1 -- 172.21.15.1:0/153987754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7654003520 con 0x7f76540051c0 2024-03-06T07:23:28.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.008+0000 7f766e091640 1 -- 172.21.15.1:0/153987754 reap_dead start 2024-03-06T07:23:28.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.008+0000 7f766e091640 1 --2- 172.21.15.1:0/153987754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76540051c0 0x7f765400b9c0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f766000c240 tx=0x7f7654005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b439b10b9feeae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.008+0000 7f766c88e640 1 -- 172.21.15.1:0/153987754 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7660007d10 con 0x7f76540051c0 2024-03-06T07:23:28.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.008+0000 7f766c88e640 1 -- 172.21.15.1:0/153987754 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7660030070 con 0x7f76540051c0 2024-03-06T07:23:28.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.008+0000 7f766c88e640 1 -- 172.21.15.1:0/153987754 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766002cd20 con 0x7f76540051c0 2024-03-06T07:23:28.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.008+0000 7f766faf4640 1 -- 172.21.15.1:0/153987754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76540051c0 msgr2=0x7f765400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.008+0000 7f766faf4640 1 --2- 172.21.15.1:0/153987754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76540051c0 0x7f765400b9c0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f766000c240 tx=0x7f7654005830 comp rx=0 tx=0).stop 2024-03-06T07:23:28.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.008+0000 7f766faf4640 1 -- 172.21.15.1:0/153987754 shutdown_connections 2024-03-06T07:23:28.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.008+0000 7f766faf4640 1 --2- 172.21.15.1:0/153987754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76540051c0 0x7f765400b9c0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.008+0000 7f766faf4640 1 -- 172.21.15.1:0/153987754 >> 172.21.15.1:0/153987754 conn(0x7f76680c6860 msgr2=0x7f76680c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.008+0000 7f766faf4640 1 -- 172.21.15.1:0/153987754 shutdown_connections 2024-03-06T07:23:28.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.009+0000 7f766faf4640 1 -- 172.21.15.1:0/153987754 wait complete. 2024-03-06T07:23:28.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.010+0000 7f766faf4640 1 Processor -- start 2024-03-06T07:23:28.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.010+0000 7f766faf4640 1 -- start start 2024-03-06T07:23:28.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.011+0000 7f766faf4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540051c0 0x7f766814d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.011+0000 7f766faf4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76681ea8c0 0x7f7668149240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.011+0000 7f766faf4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7668149780 0x7f76681e6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.011+0000 7f766faf4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7668155f00 con 0x7f7668149780 2024-03-06T07:23:28.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.011+0000 7f766faf4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7668155d80 con 0x7f76681ea8c0 2024-03-06T07:23:28.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.011+0000 7f766faf4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76680724c0 con 0x7f76540051c0 2024-03-06T07:23:28.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.011+0000 7f766e091640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7668149780 0x7f76681e6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.011+0000 7f766e091640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7668149780 0x7f76681e6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45308/0 (socket says 172.21.15.1:45308) 2024-03-06T07:23:28.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.011+0000 7f766e091640 1 -- 172.21.15.1:0/3719391522 learned_addr learned my addr 172.21.15.1:0/3719391522 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:28.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.012+0000 7f766d08f640 1 --2- 172.21.15.1:0/3719391522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76681ea8c0 0x7f7668149240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.012+0000 7f766d890640 1 --2- 172.21.15.1:0/3719391522 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540051c0 0x7f766814d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.012+0000 7f766e091640 1 -- 172.21.15.1:0/3719391522 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540051c0 msgr2=0x7f766814d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.012+0000 7f766e091640 1 --2- 172.21.15.1:0/3719391522 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540051c0 0x7f766814d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.012+0000 7f766e091640 1 -- 172.21.15.1:0/3719391522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76681ea8c0 msgr2=0x7f7668149240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.012+0000 7f766e091640 1 --2- 172.21.15.1:0/3719391522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76681ea8c0 0x7f7668149240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.012+0000 7f766e091640 1 -- 172.21.15.1:0/3719391522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76680bba00 con 0x7f7668149780 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.012+0000 7f766d890640 1 --2- 172.21.15.1:0/3719391522 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540051c0 0x7f766814d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.012+0000 7f766e091640 1 --2- 172.21.15.1:0/3719391522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7668149780 0x7f76681e6940 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f766002de10 tx=0x7f7668156470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.012+0000 7f765a7fc640 1 -- 172.21.15.1:0/3719391522 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7660007d10 con 0x7f7668149780 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.012+0000 7f765a7fc640 1 -- 172.21.15.1:0/3719391522 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7660004bf0 con 0x7f7668149780 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.012+0000 7f766faf4640 1 -- 172.21.15.1:0/3719391522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76681dc390 con 0x7f7668149780 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.012+0000 7f765a7fc640 1 -- 172.21.15.1:0/3719391522 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7660007070 con 0x7f7668149780 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.012+0000 7f766faf4640 1 -- 172.21.15.1:0/3719391522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76681dbe70 con 0x7f7668149780 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.013+0000 7f766faf4640 1 -- 172.21.15.1:0/3719391522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7668067fd0 con 0x7f7668149780 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.014+0000 7f765a7fc640 1 -- 172.21.15.1:0/3719391522 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7660007210 con 0x7f7668149780 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.014+0000 7f765a7fc640 1 --2- 172.21.15.1:0/3719391522 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764c06b4d0 0x7f764c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.014+0000 7f765a7fc640 1 -- 172.21.15.1:0/3719391522 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f76600ab7f0 con 0x7f7668149780 2024-03-06T07:23:28.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.014+0000 7f766d890640 1 --2- 172.21.15.1:0/3719391522 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764c06b4d0 0x7f764c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.015+0000 7f766d890640 1 --2- 172.21.15.1:0/3719391522 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764c06b4d0 0x7f764c06d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7664007380 tx=0x7f766400b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.023+0000 7f765a7fc640 1 -- 172.21.15.1:0/3719391522 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f766007a470 con 0x7f7668149780 2024-03-06T07:23:28.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.208+0000 7f766faf4640 1 -- 172.21.15.1:0/3719391522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7668071790 con 0x7f7668149780 2024-03-06T07:23:28.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.209+0000 7f765a7fc640 1 -- 172.21.15.1:0/3719391522 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+17152 (secure 0 0 0) 0x7f7660079d70 con 0x7f7668149780 2024-03-06T07:23:28.211 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:28.211 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":83,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:23:26.260431+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:23:05.599046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:23:06.205539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6834/3919123103":"2024-03-07T07:23:23.886065+0000","172.21.15.98:6835/114998998":"2024-03-07T07:23:24.253819+0000","172.21.15.1:6832/3919123103":"2024-03-07T07:23:23.886065+0000","172.21.15.1:6834/1494116991":"2024-03-07T07:23:13.133394+0000","172.21.15.98:6837/114998998":"2024-03-07T07:23:24.253819+0000","172.21.15.1:6832/1494116991":"2024-03-07T07:23:13.133394+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:28.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.211+0000 7f766faf4640 1 -- 172.21.15.1:0/3719391522 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764c06b4d0 msgr2=0x7f764c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.211+0000 7f766faf4640 1 --2- 172.21.15.1:0/3719391522 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764c06b4d0 0x7f764c06d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7664007380 tx=0x7f766400b040 comp rx=0 tx=0).stop 2024-03-06T07:23:28.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.211+0000 7f766faf4640 1 -- 172.21.15.1:0/3719391522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7668149780 msgr2=0x7f76681e6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.211+0000 7f766faf4640 1 --2- 172.21.15.1:0/3719391522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7668149780 0x7f76681e6940 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f766002de10 tx=0x7f7668156470 comp rx=0 tx=0).stop 2024-03-06T07:23:28.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.211+0000 7f766faf4640 1 -- 172.21.15.1:0/3719391522 shutdown_connections 2024-03-06T07:23:28.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.211+0000 7f766faf4640 1 --2- 172.21.15.1:0/3719391522 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764c06b4d0 0x7f764c06d990 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.211+0000 7f766faf4640 1 --2- 172.21.15.1:0/3719391522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7668149780 0x7f76681e6940 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.211+0000 7f766faf4640 1 --2- 172.21.15.1:0/3719391522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76681ea8c0 0x7f7668149240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.211+0000 7f766faf4640 1 --2- 172.21.15.1:0/3719391522 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76540051c0 0x7f766814d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.211+0000 7f766faf4640 1 -- 172.21.15.1:0/3719391522 >> 172.21.15.1:0/3719391522 conn(0x7f76680c6860 msgr2=0x7f7668150510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.211+0000 7f766faf4640 1 -- 172.21.15.1:0/3719391522 shutdown_connections 2024-03-06T07:23:28.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.211+0000 7f766faf4640 1 -- 172.21.15.1:0/3719391522 wait complete. 2024-03-06T07:23:28.227 INFO:tasks.cephfs.filesystem:Destroying file system cephfs2 and related pools 2024-03-06T07:23:28.227 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.328+0000 7f719c48f640 1 Processor -- start 2024-03-06T07:23:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.329+0000 7f719c48f640 1 -- start start 2024-03-06T07:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.329+0000 7f719c48f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71940bf9d0 0x7f7194155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.329+0000 7f719c48f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71940c2fc0 con 0x7f71940c6b10 2024-03-06T07:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.329+0000 7f719c48f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7194158da0 con 0x7f7194152650 2024-03-06T07:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.329+0000 7f719c48f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7194155800 con 0x7f71940bf9d0 2024-03-06T07:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.330+0000 7f7199a2a640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f71940c6b10 0x7f71940c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50222/0 (socket says 172.21.15.1:50222) 2024-03-06T07:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.330+0000 7f7199a2a640 1 -- 172.21.15.1:0/3274652782 learned_addr learned my addr 172.21.15.1:0/3274652782 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.330+0000 7f719aa2c640 1 --2- 172.21.15.1:0/3274652782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71940bf9d0 0x7f7194155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.330+0000 7f7199229640 1 -- 172.21.15.1:0/3274652782 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469260911 0 0) 0x7f71940c2fc0 con 0x7f71940c6b10 2024-03-06T07:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.330+0000 7f7199229640 1 -- 172.21.15.1:0/3274652782 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7180003520 con 0x7f71940c6b10 2024-03-06T07:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.330+0000 7f7199229640 1 -- 172.21.15.1:0/3274652782 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325107141 0 0) 0x7f7194158da0 con 0x7f7194152650 2024-03-06T07:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.330+0000 7f7199229640 1 -- 172.21.15.1:0/3274652782 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71940c2fc0 con 0x7f7194152650 2024-03-06T07:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.330+0000 7f719aa2c640 1 -- 172.21.15.1:0/3274652782 >> v1:172.21.15.98:6789/0 conn(0x7f7194152650 legacy=0x7f7194158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.330+0000 7f719aa2c640 1 -- 172.21.15.1:0/3274652782 >> v1:172.21.15.1:6789/0 conn(0x7f71940c6b10 legacy=0x7f71940c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.330+0000 7f719aa2c640 1 -- 172.21.15.1:0/3274652782 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71940c6690 con 0x7f71940bf9d0 2024-03-06T07:23:28.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.331+0000 7f719aa2c640 1 --2- 172.21.15.1:0/3274652782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71940bf9d0 0x7f7194155c90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f718c009dc0 tx=0x7f718c02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea0255265642d127 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.331+0000 7f7199229640 1 -- 172.21.15.1:0/3274652782 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718c039070 con 0x7f71940bf9d0 2024-03-06T07:23:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.331+0000 7f7199229640 1 -- 172.21.15.1:0/3274652782 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718c02cc40 con 0x7f71940bf9d0 2024-03-06T07:23:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.331+0000 7f7199229640 1 -- 172.21.15.1:0/3274652782 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718c034440 con 0x7f71940bf9d0 2024-03-06T07:23:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.332+0000 7f719c48f640 1 -- 172.21.15.1:0/3274652782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71940bf9d0 msgr2=0x7f7194155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.332+0000 7f719c48f640 1 --2- 172.21.15.1:0/3274652782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71940bf9d0 0x7f7194155c90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f718c009dc0 tx=0x7f718c02be90 comp rx=0 tx=0).stop 2024-03-06T07:23:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.332+0000 7f719c48f640 1 -- 172.21.15.1:0/3274652782 shutdown_connections 2024-03-06T07:23:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.332+0000 7f719c48f640 1 --2- 172.21.15.1:0/3274652782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71940bf9d0 0x7f7194155c90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.332+0000 7f719c48f640 1 -- 172.21.15.1:0/3274652782 >> 172.21.15.1:0/3274652782 conn(0x7f7194066ee0 msgr2=0x7f71940672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.332+0000 7f719c48f640 1 -- 172.21.15.1:0/3274652782 shutdown_connections 2024-03-06T07:23:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.332+0000 7f719c48f640 1 -- 172.21.15.1:0/3274652782 wait complete. 2024-03-06T07:23:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.333+0000 7f719c48f640 1 Processor -- start 2024-03-06T07:23:28.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.333+0000 7f719c48f640 1 -- start start 2024-03-06T07:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.333+0000 7f719c48f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71940bf9d0 0x7f71941e0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.333+0000 7f719c48f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7194152650 0x7f719414c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f719c48f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f719414ce10 0x7f71940c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f719c48f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71941551b0 con 0x7f71940bf9d0 2024-03-06T07:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f719c48f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71940c6fa0 con 0x7f719414ce10 2024-03-06T07:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f719c48f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7194155490 con 0x7f7194152650 2024-03-06T07:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f719a22b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71940bf9d0 0x7f71941e0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f719a22b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71940bf9d0 0x7f71941e0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45314/0 (socket says 172.21.15.1:45314) 2024-03-06T07:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f719a22b640 1 -- 172.21.15.1:0/1951549954 learned_addr learned my addr 172.21.15.1:0/1951549954 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f7199a2a640 1 --2- 172.21.15.1:0/1951549954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7194152650 0x7f719414c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f719aa2c640 1 --2- 172.21.15.1:0/1951549954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f719414ce10 0x7f71940c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f719a22b640 1 -- 172.21.15.1:0/1951549954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7194152650 msgr2=0x7f719414c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f719a22b640 1 --2- 172.21.15.1:0/1951549954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7194152650 0x7f719414c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f719a22b640 1 -- 172.21.15.1:0/1951549954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f719414ce10 msgr2=0x7f71940c91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f719a22b640 1 --2- 172.21.15.1:0/1951549954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f719414ce10 0x7f71940c91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f719a22b640 1 -- 172.21.15.1:0/1951549954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719414d2a0 con 0x7f71940bf9d0 2024-03-06T07:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f719a22b640 1 --2- 172.21.15.1:0/1951549954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71940bf9d0 0x7f71941e0590 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f719000c430 tx=0x7f719000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f718affd640 1 -- 172.21.15.1:0/1951549954 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7190017070 con 0x7f71940bf9d0 2024-03-06T07:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f718affd640 1 -- 172.21.15.1:0/1951549954 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71900052f0 con 0x7f71940bf9d0 2024-03-06T07:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f719c48f640 1 -- 172.21.15.1:0/1951549954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71941e0dd0 con 0x7f71940bf9d0 2024-03-06T07:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f718affd640 1 -- 172.21.15.1:0/1951549954 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7190003830 con 0x7f71940bf9d0 2024-03-06T07:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.334+0000 7f719c48f640 1 -- 172.21.15.1:0/1951549954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7194156ad0 con 0x7f71940bf9d0 2024-03-06T07:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.335+0000 7f719c48f640 1 -- 172.21.15.1:0/1951549954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7194067fd0 con 0x7f71940bf9d0 2024-03-06T07:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.336+0000 7f718affd640 1 -- 172.21.15.1:0/1951549954 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f719001a070 con 0x7f71940bf9d0 2024-03-06T07:23:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.336+0000 7f718affd640 1 --2- 172.21.15.1:0/1951549954 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f717806b4d0 0x7f717806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.337+0000 7f718affd640 1 -- 172.21.15.1:0/1951549954 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f71900896a0 con 0x7f71940bf9d0 2024-03-06T07:23:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.337+0000 7f7199a2a640 1 --2- 172.21.15.1:0/1951549954 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f717806b4d0 0x7f717806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.337+0000 7f7199a2a640 1 --2- 172.21.15.1:0/1951549954 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f717806b4d0 0x7f717806d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f71940ca970 tx=0x7f7184009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.341+0000 7f718affd640 1 -- 172.21.15.1:0/1951549954 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7190057cd0 con 0x7f71940bf9d0 2024-03-06T07:23:28.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.628+0000 7f719c48f640 1 -- 172.21.15.1:0/1951549954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71940c5d60 con 0x7f71940bf9d0 2024-03-06T07:23:28.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.628+0000 7f718affd640 1 -- 172.21.15.1:0/1951549954 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f719005bd70 con 0x7f71940bf9d0 2024-03-06T07:23:28.631 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:28.631 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":71,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.98:6836/4069609682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4069609682},{"type":"v1","addr":"172.21.15.98:6836","nonce":4069609682}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6196,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3962693342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3962693342},{"type":"v1","addr":"172.21.15.1:6834","nonce":3962693342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6211,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3130835916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3130835916},{"type":"v1","addr":"172.21.15.98:6837","nonce":3130835916}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:07.074540+0000","modified":"2024-03-06T07:23:27.261401+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5827":{"gid":5827,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":11,"addr":"172.21.15.1:6835/2685225325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2685225325},{"type":"v1","addr":"172.21.15.1:6835","nonce":2685225325}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5827,"qdb_cluster":[5827]},"id":6}]} 2024-03-06T07:23:28.631 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 71 2024-03-06T07:23:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.630+0000 7f719c48f640 1 -- 172.21.15.1:0/1951549954 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f717806b4d0 msgr2=0x7f717806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.630+0000 7f719c48f640 1 --2- 172.21.15.1:0/1951549954 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f717806b4d0 0x7f717806d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f71940ca970 tx=0x7f7184009040 comp rx=0 tx=0).stop 2024-03-06T07:23:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.630+0000 7f719c48f640 1 -- 172.21.15.1:0/1951549954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71940bf9d0 msgr2=0x7f71941e0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.630+0000 7f719c48f640 1 --2- 172.21.15.1:0/1951549954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71940bf9d0 0x7f71941e0590 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f719000c430 tx=0x7f719000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.630+0000 7f719c48f640 1 -- 172.21.15.1:0/1951549954 shutdown_connections 2024-03-06T07:23:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.630+0000 7f719c48f640 1 --2- 172.21.15.1:0/1951549954 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f717806b4d0 0x7f717806d990 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.630+0000 7f719c48f640 1 --2- 172.21.15.1:0/1951549954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f719414ce10 0x7f71940c91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.630+0000 7f719c48f640 1 --2- 172.21.15.1:0/1951549954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7194152650 0x7f719414c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.630+0000 7f719c48f640 1 --2- 172.21.15.1:0/1951549954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71940bf9d0 0x7f71941e0590 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.630+0000 7f719c48f640 1 -- 172.21.15.1:0/1951549954 >> 172.21.15.1:0/1951549954 conn(0x7f7194066ee0 msgr2=0x7f71940caff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.630+0000 7f719c48f640 1 -- 172.21.15.1:0/1951549954 shutdown_connections 2024-03-06T07:23:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.630+0000 7f719c48f640 1 -- 172.21.15.1:0/1951549954 wait complete. 2024-03-06T07:23:28.643 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:28.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.742+0000 7f89e866b640 1 Processor -- start 2024-03-06T07:23:28.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.742+0000 7f89e866b640 1 -- start start 2024-03-06T07:23:28.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.743+0000 7f89e866b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f89e0152650 0x7f89e0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.743+0000 7f89e866b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e00c2fc0 con 0x7f89e00c6b10 2024-03-06T07:23:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.743+0000 7f89e866b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e0158da0 con 0x7f89e00bf9d0 2024-03-06T07:23:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.743+0000 7f89e866b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e0155800 con 0x7f89e0152650 2024-03-06T07:23:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.743+0000 7f89e6407640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f89e0152650 0x7f89e0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.743+0000 7f89e5c06640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f89e00c6b10 0x7f89e0158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50234/0 (socket says 172.21.15.1:50234) 2024-03-06T07:23:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.743+0000 7f89e5c06640 1 -- 172.21.15.1:0/866612145 learned_addr learned my addr 172.21.15.1:0/866612145 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.744+0000 7f89e6407640 1 -- 172.21.15.1:0/866612145 >> v1:172.21.15.98:6789/0 conn(0x7f89e00bf9d0 legacy=0x7f89e00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.744+0000 7f89e6407640 1 -- 172.21.15.1:0/866612145 >> v1:172.21.15.1:6789/0 conn(0x7f89e00c6b10 legacy=0x7f89e0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.744+0000 7f89e6407640 1 -- 172.21.15.1:0/866612145 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e00c6690 con 0x7f89e0152650 2024-03-06T07:23:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.744+0000 7f89e5405640 1 -- 172.21.15.1:0/866612145 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486978107 0 0) 0x7f89e00c2fc0 con 0x7f89e00c6b10 2024-03-06T07:23:28.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.745+0000 7f89e6407640 1 --2- 172.21.15.1:0/866612145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e0152650 0x7f89e0155c90 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f89d40097a0 tx=0x7f89d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8e13cd3534a351 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.745+0000 7f89e5405640 1 -- 172.21.15.1:0/866612145 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d40315a0 con 0x7f89e0152650 2024-03-06T07:23:28.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.745+0000 7f89e5405640 1 -- 172.21.15.1:0/866612145 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d4037070 con 0x7f89e0152650 2024-03-06T07:23:28.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.745+0000 7f89e5405640 1 -- 172.21.15.1:0/866612145 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d4031960 con 0x7f89e0152650 2024-03-06T07:23:28.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.746+0000 7f89e866b640 1 -- 172.21.15.1:0/866612145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e0152650 msgr2=0x7f89e0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.746+0000 7f89e866b640 1 --2- 172.21.15.1:0/866612145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e0152650 0x7f89e0155c90 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f89d40097a0 tx=0x7f89d402b840 comp rx=0 tx=0).stop 2024-03-06T07:23:28.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.746+0000 7f89e866b640 1 -- 172.21.15.1:0/866612145 shutdown_connections 2024-03-06T07:23:28.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.746+0000 7f89e866b640 1 --2- 172.21.15.1:0/866612145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e0152650 0x7f89e0155c90 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.746+0000 7f89e866b640 1 -- 172.21.15.1:0/866612145 >> 172.21.15.1:0/866612145 conn(0x7f89e0066ee0 msgr2=0x7f89e00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.746+0000 7f89e866b640 1 -- 172.21.15.1:0/866612145 shutdown_connections 2024-03-06T07:23:28.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.746+0000 7f89e866b640 1 -- 172.21.15.1:0/866612145 wait complete. 2024-03-06T07:23:28.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.747+0000 7f89e866b640 1 Processor -- start 2024-03-06T07:23:28.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.747+0000 7f89e866b640 1 -- start start 2024-03-06T07:23:28.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.747+0000 7f89e866b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e00bf9d0 0x7f89e01eec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.747+0000 7f89e866b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e0152650 0x7f89e01577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.747+0000 7f89e866b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89e0158720 0x7f89e014d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.747+0000 7f89e866b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e01551b0 con 0x7f89e0152650 2024-03-06T07:23:28.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.747+0000 7f89e866b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e00c6fa0 con 0x7f89e00bf9d0 2024-03-06T07:23:28.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.747+0000 7f89e866b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e0155490 con 0x7f89e0158720 2024-03-06T07:23:28.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.748+0000 7f89e5c06640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e0152650 0x7f89e01577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.748+0000 7f89e5c06640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e0152650 0x7f89e01577d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45344/0 (socket says 172.21.15.1:45344) 2024-03-06T07:23:28.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.748+0000 7f89e5c06640 1 -- 172.21.15.1:0/235606820 learned_addr learned my addr 172.21.15.1:0/235606820 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.748+0000 7f89e6407640 1 --2- 172.21.15.1:0/235606820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e00bf9d0 0x7f89e01eec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.748+0000 7f89e6c08640 1 --2- 172.21.15.1:0/235606820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89e0158720 0x7f89e014d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.748+0000 7f89e5c06640 1 -- 172.21.15.1:0/235606820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89e0158720 msgr2=0x7f89e014d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.748+0000 7f89e5c06640 1 --2- 172.21.15.1:0/235606820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89e0158720 0x7f89e014d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.748+0000 7f89e5c06640 1 -- 172.21.15.1:0/235606820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e00bf9d0 msgr2=0x7f89e01eec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.748+0000 7f89e5c06640 1 --2- 172.21.15.1:0/235606820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e00bf9d0 0x7f89e01eec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.748+0000 7f89e5c06640 1 -- 172.21.15.1:0/235606820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e01eaf60 con 0x7f89e0152650 2024-03-06T07:23:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.748+0000 7f89e5c06640 1 --2- 172.21.15.1:0/235606820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e0152650 0x7f89e01577d0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00c4d0 tx=0x7f89dc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.748+0000 7f89d2ffd640 1 -- 172.21.15.1:0/235606820 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc018070 con 0x7f89e0152650 2024-03-06T07:23:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.748+0000 7f89d2ffd640 1 -- 172.21.15.1:0/235606820 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89dc007500 con 0x7f89e0152650 2024-03-06T07:23:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.748+0000 7f89d2ffd640 1 -- 172.21.15.1:0/235606820 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc004a20 con 0x7f89e0152650 2024-03-06T07:23:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.748+0000 7f89e866b640 1 -- 172.21.15.1:0/235606820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e01eb240 con 0x7f89e0152650 2024-03-06T07:23:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.748+0000 7f89e866b640 1 -- 172.21.15.1:0/235606820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e01eb800 con 0x7f89e0152650 2024-03-06T07:23:28.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.749+0000 7f89e866b640 1 -- 172.21.15.1:0/235606820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e00c2f50 con 0x7f89e0152650 2024-03-06T07:23:28.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.750+0000 7f89d2ffd640 1 -- 172.21.15.1:0/235606820 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89dc01b070 con 0x7f89e0152650 2024-03-06T07:23:28.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.750+0000 7f89d2ffd640 1 --2- 172.21.15.1:0/235606820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f89c406b4d0 0x7f89c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.751+0000 7f89d2ffd640 1 -- 172.21.15.1:0/235606820 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f89dc089720 con 0x7f89e0152650 2024-03-06T07:23:28.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.751+0000 7f89e6407640 1 --2- 172.21.15.1:0/235606820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f89c406b4d0 0x7f89c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.751+0000 7f89e6407640 1 --2- 172.21.15.1:0/235606820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f89c406b4d0 0x7f89c406d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f89d4036000 tx=0x7f89d4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.755+0000 7f89d2ffd640 1 -- 172.21.15.1:0/235606820 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89e00c2f50 con 0x7f89e0152650 2024-03-06T07:23:28.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.973+0000 7f89e866b640 1 -- 172.21.15.1:0/235606820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89e0067fd0 con 0x7f89e0152650 2024-03-06T07:23:28.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.974+0000 7f89d2ffd640 1 -- 172.21.15.1:0/235606820 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f89dc057cc0 con 0x7f89e0152650 2024-03-06T07:23:28.976 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:28.976 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":71,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.98:6836/4069609682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4069609682},{"type":"v1","addr":"172.21.15.98:6836","nonce":4069609682}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6196,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3962693342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3962693342},{"type":"v1","addr":"172.21.15.1:6834","nonce":3962693342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6211,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3130835916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3130835916},{"type":"v1","addr":"172.21.15.98:6837","nonce":3130835916}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:07.074540+0000","modified":"2024-03-06T07:23:27.261401+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5827":{"gid":5827,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":11,"addr":"172.21.15.1:6835/2685225325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2685225325},{"type":"v1","addr":"172.21.15.1:6835","nonce":2685225325}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5827,"qdb_cluster":[5827]},"id":6}]} 2024-03-06T07:23:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 71 2024-03-06T07:23:28.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.975+0000 7f89e866b640 1 -- 172.21.15.1:0/235606820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f89c406b4d0 msgr2=0x7f89c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.975+0000 7f89e866b640 1 --2- 172.21.15.1:0/235606820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f89c406b4d0 0x7f89c406d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f89d4036000 tx=0x7f89d4008030 comp rx=0 tx=0).stop 2024-03-06T07:23:28.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.975+0000 7f89e866b640 1 -- 172.21.15.1:0/235606820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e0152650 msgr2=0x7f89e01577d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.975+0000 7f89e866b640 1 --2- 172.21.15.1:0/235606820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e0152650 0x7f89e01577d0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00c4d0 tx=0x7f89dc00c990 comp rx=0 tx=0).stop 2024-03-06T07:23:28.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.975+0000 7f89e866b640 1 -- 172.21.15.1:0/235606820 shutdown_connections 2024-03-06T07:23:28.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.975+0000 7f89e866b640 1 --2- 172.21.15.1:0/235606820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f89c406b4d0 0x7f89c406d990 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.975+0000 7f89e866b640 1 --2- 172.21.15.1:0/235606820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89e0158720 0x7f89e014d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.975+0000 7f89e866b640 1 --2- 172.21.15.1:0/235606820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e0152650 0x7f89e01577d0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.975+0000 7f89e866b640 1 --2- 172.21.15.1:0/235606820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89e00bf9d0 0x7f89e01eec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.976+0000 7f89e866b640 1 -- 172.21.15.1:0/235606820 >> 172.21.15.1:0/235606820 conn(0x7f89e0066ee0 msgr2=0x7f89e00c3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.976+0000 7f89e866b640 1 -- 172.21.15.1:0/235606820 shutdown_connections 2024-03-06T07:23:28.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:28.976+0000 7f89e866b640 1 -- 172.21.15.1:0/235606820 wait complete. 2024-03-06T07:23:28.988 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:29.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.088+0000 7f6633815640 1 Processor -- start 2024-03-06T07:23:29.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.088+0000 7f6633815640 1 -- start start 2024-03-06T07:23:29.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.089+0000 7f6633815640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f662c152650 0x7f662c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.089+0000 7f6633815640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f662c0bf9d0 0x7f662c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.089+0000 7f6633815640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f662c0c2fc0 con 0x7f662c0c6b10 2024-03-06T07:23:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.089+0000 7f6633815640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f662c155800 con 0x7f662c0bf9d0 2024-03-06T07:23:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.089+0000 7f6633815640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662c158da0 con 0x7f662c152650 2024-03-06T07:23:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.089+0000 7f6631db2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f662c0bf9d0 0x7f662c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.089+0000 7f66315b1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f662c152650 0x7f662c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.089+0000 7f6631db2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f662c0bf9d0 0x7f662c0c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:34018/0 (socket says 172.21.15.1:34018) 2024-03-06T07:23:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.089+0000 7f6631db2640 1 -- 172.21.15.1:0/556032609 learned_addr learned my addr 172.21.15.1:0/556032609 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.090+0000 7f66315b1640 1 -- 172.21.15.1:0/556032609 >> v2:172.21.15.98:3300/0 conn(0x7f662c0bf9d0 msgr2=0x7f662c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.090+0000 7f66315b1640 1 --2- 172.21.15.1:0/556032609 >> v2:172.21.15.98:3300/0 conn(0x7f662c0bf9d0 0x7f662c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.090+0000 7f66315b1640 1 -- 172.21.15.1:0/556032609 >> v1:172.21.15.98:6789/0 conn(0x7f662c0c6b10 legacy=0x7f662c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.090+0000 7f66315b1640 1 -- 172.21.15.1:0/556032609 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662c0c6690 con 0x7f662c152650 2024-03-06T07:23:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.091+0000 7f66315b1640 1 --2- 172.21.15.1:0/556032609 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f662c152650 0x7f662c155c90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f66280097a0 tx=0x7f662802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f352298561031cd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.091+0000 7f6623fff640 1 -- 172.21.15.1:0/556032609 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66280315a0 con 0x7f662c152650 2024-03-06T07:23:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.091+0000 7f6623fff640 1 -- 172.21.15.1:0/556032609 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6628037070 con 0x7f662c152650 2024-03-06T07:23:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.091+0000 7f6623fff640 1 -- 172.21.15.1:0/556032609 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6628031960 con 0x7f662c152650 2024-03-06T07:23:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.091+0000 7f6633815640 1 -- 172.21.15.1:0/556032609 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f662c152650 msgr2=0x7f662c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.091+0000 7f6633815640 1 --2- 172.21.15.1:0/556032609 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f662c152650 0x7f662c155c90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f66280097a0 tx=0x7f662802b840 comp rx=0 tx=0).stop 2024-03-06T07:23:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.091+0000 7f6633815640 1 -- 172.21.15.1:0/556032609 shutdown_connections 2024-03-06T07:23:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.091+0000 7f6633815640 1 --2- 172.21.15.1:0/556032609 >> v2:172.21.15.98:3300/0 conn(0x7f662c0bf9d0 0x7f662c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.091+0000 7f6633815640 1 --2- 172.21.15.1:0/556032609 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f662c152650 0x7f662c155c90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.091+0000 7f6633815640 1 -- 172.21.15.1:0/556032609 >> 172.21.15.1:0/556032609 conn(0x7f662c066ee0 msgr2=0x7f662c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.091+0000 7f6633815640 1 -- 172.21.15.1:0/556032609 shutdown_connections 2024-03-06T07:23:29.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.091+0000 7f6633815640 1 -- 172.21.15.1:0/556032609 wait complete. 2024-03-06T07:23:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.093+0000 7f6633815640 1 Processor -- start 2024-03-06T07:23:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.093+0000 7f6633815640 1 -- start start 2024-03-06T07:23:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.093+0000 7f6633815640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f662c0bf9d0 0x7f662c0c5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.093+0000 7f6633815640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f662c152650 0x7f662c1565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.093+0000 7f6633815640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f662c158330 0x7f662c156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.093+0000 7f6633815640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662c1551b0 con 0x7f662c158330 2024-03-06T07:23:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.093+0000 7f6633815640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662c0c6fa0 con 0x7f662c0bf9d0 2024-03-06T07:23:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.093+0000 7f6633815640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662c155490 con 0x7f662c152650 2024-03-06T07:23:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.093+0000 7f6631db2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f662c158330 0x7f662c156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.093+0000 7f6631db2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f662c158330 0x7f662c156af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45360/0 (socket says 172.21.15.1:45360) 2024-03-06T07:23:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.093+0000 7f6631db2640 1 -- 172.21.15.1:0/2449759151 learned_addr learned my addr 172.21.15.1:0/2449759151 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.094+0000 7f66315b1640 1 --2- 172.21.15.1:0/2449759151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f662c0bf9d0 0x7f662c0c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.094+0000 7f6630db0640 1 --2- 172.21.15.1:0/2449759151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f662c152650 0x7f662c1565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.094+0000 7f6631db2640 1 -- 172.21.15.1:0/2449759151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f662c152650 msgr2=0x7f662c1565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.094+0000 7f6631db2640 1 --2- 172.21.15.1:0/2449759151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f662c152650 0x7f662c1565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.094+0000 7f6631db2640 1 -- 172.21.15.1:0/2449759151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f662c0bf9d0 msgr2=0x7f662c0c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.094+0000 7f6631db2640 1 --2- 172.21.15.1:0/2449759151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f662c0bf9d0 0x7f662c0c5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.094+0000 7f6631db2640 1 -- 172.21.15.1:0/2449759151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662c14b810 con 0x7f662c158330 2024-03-06T07:23:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.094+0000 7f66315b1640 1 --2- 172.21.15.1:0/2449759151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f662c0bf9d0 0x7f662c0c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.094+0000 7f6631db2640 1 --2- 172.21.15.1:0/2449759151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f662c158330 0x7f662c156af0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f662400efc0 tx=0x7f662400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.094+0000 7f6621ffb640 1 -- 172.21.15.1:0/2449759151 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662400cd20 con 0x7f662c158330 2024-03-06T07:23:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.094+0000 7f6621ffb640 1 -- 172.21.15.1:0/2449759151 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6624003990 con 0x7f662c158330 2024-03-06T07:23:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.094+0000 7f6621ffb640 1 -- 172.21.15.1:0/2449759151 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6624003d80 con 0x7f662c158330 2024-03-06T07:23:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.094+0000 7f6633815640 1 -- 172.21.15.1:0/2449759151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f662c14c280 con 0x7f662c158330 2024-03-06T07:23:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.094+0000 7f6633815640 1 -- 172.21.15.1:0/2449759151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f662c0c07a0 con 0x7f662c158330 2024-03-06T07:23:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.095+0000 7f6633815640 1 -- 172.21.15.1:0/2449759151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f662c067fd0 con 0x7f662c158330 2024-03-06T07:23:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.096+0000 7f6621ffb640 1 -- 172.21.15.1:0/2449759151 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6624005d10 con 0x7f662c158330 2024-03-06T07:23:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.097+0000 7f6621ffb640 1 --2- 172.21.15.1:0/2449759151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f661006b4d0 0x7f661006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.097+0000 7f6621ffb640 1 -- 172.21.15.1:0/2449759151 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f662408a8f0 con 0x7f662c158330 2024-03-06T07:23:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.097+0000 7f66315b1640 1 --2- 172.21.15.1:0/2449759151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f661006b4d0 0x7f661006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.097+0000 7f66315b1640 1 --2- 172.21.15.1:0/2449759151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f661006b4d0 0x7f661006d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6628036000 tx=0x7f6628008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.101+0000 7f6621ffb640 1 -- 172.21.15.1:0/2449759151 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6624022020 con 0x7f662c158330 2024-03-06T07:23:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.295+0000 7f6633815640 1 -- 172.21.15.1:0/2449759151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f662c0c37e0 con 0x7f662c158330 2024-03-06T07:23:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.296+0000 7f6621ffb640 1 -- 172.21.15.1:0/2449759151 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+17152 (secure 0 0 0) 0x7f662c14c280 con 0x7f662c158330 2024-03-06T07:23:29.298 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:29.299 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":83,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:23:26.260431+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:23:05.599046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:23:06.205539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6834/3919123103":"2024-03-07T07:23:23.886065+0000","172.21.15.98:6835/114998998":"2024-03-07T07:23:24.253819+0000","172.21.15.1:6832/3919123103":"2024-03-07T07:23:23.886065+0000","172.21.15.1:6834/1494116991":"2024-03-07T07:23:13.133394+0000","172.21.15.98:6837/114998998":"2024-03-07T07:23:24.253819+0000","172.21.15.1:6832/1494116991":"2024-03-07T07:23:13.133394+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:29.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.298+0000 7f6633815640 1 -- 172.21.15.1:0/2449759151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f661006b4d0 msgr2=0x7f661006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.298+0000 7f6633815640 1 --2- 172.21.15.1:0/2449759151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f661006b4d0 0x7f661006d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6628036000 tx=0x7f6628008030 comp rx=0 tx=0).stop 2024-03-06T07:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.298+0000 7f6633815640 1 -- 172.21.15.1:0/2449759151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f662c158330 msgr2=0x7f662c156af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.298+0000 7f6633815640 1 --2- 172.21.15.1:0/2449759151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f662c158330 0x7f662c156af0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f662400efc0 tx=0x7f662400c490 comp rx=0 tx=0).stop 2024-03-06T07:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.298+0000 7f6633815640 1 -- 172.21.15.1:0/2449759151 shutdown_connections 2024-03-06T07:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.298+0000 7f6633815640 1 --2- 172.21.15.1:0/2449759151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f661006b4d0 0x7f661006d990 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.298+0000 7f6633815640 1 --2- 172.21.15.1:0/2449759151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f662c158330 0x7f662c156af0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.298+0000 7f6633815640 1 --2- 172.21.15.1:0/2449759151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f662c152650 0x7f662c1565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.298+0000 7f6633815640 1 --2- 172.21.15.1:0/2449759151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f662c0bf9d0 0x7f662c0c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.298+0000 7f6633815640 1 -- 172.21.15.1:0/2449759151 >> 172.21.15.1:0/2449759151 conn(0x7f662c066ee0 msgr2=0x7f662c0c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.298+0000 7f6633815640 1 -- 172.21.15.1:0/2449759151 shutdown_connections 2024-03-06T07:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.298+0000 7f6633815640 1 -- 172.21.15.1:0/2449759151 wait complete. 2024-03-06T07:23:29.311 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs2 2024-03-06T07:23:29.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.411+0000 7fa5c4647640 1 Processor -- start 2024-03-06T07:23:29.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.411+0000 7fa5c4647640 1 -- start start 2024-03-06T07:23:29.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.412+0000 7fa5c4647640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa5bc152650 0x7fa5bc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.412+0000 7fa5c4647640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa5bc0bf9d0 0x7fa5bc0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.412+0000 7fa5c4647640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5bc0c2fc0 con 0x7fa5bc0c6b10 2024-03-06T07:23:29.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.412+0000 7fa5c4647640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5bc155800 con 0x7fa5bc152650 2024-03-06T07:23:29.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.412+0000 7fa5c4647640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc158da0 con 0x7fa5bc0bf9d0 2024-03-06T07:23:29.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.412+0000 7fa5c23e3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa5bc152650 0x7fa5bc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.412+0000 7fa5c23e3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa5bc152650 0x7fa5bc155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45376/0 (socket says 172.21.15.1:45376) 2024-03-06T07:23:29.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.412+0000 7fa5c23e3640 1 -- 172.21.15.1:0/3505870451 learned_addr learned my addr 172.21.15.1:0/3505870451 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:29.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.412+0000 7fa5c2be4640 1 --2- 172.21.15.1:0/3505870451 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa5bc0bf9d0 0x7fa5bc0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.412+0000 7fa5c23e3640 1 -- 172.21.15.1:0/3505870451 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa5bc0bf9d0 msgr2=0x7fa5bc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.412+0000 7fa5c23e3640 1 --2- 172.21.15.1:0/3505870451 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa5bc0bf9d0 0x7fa5bc0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.412+0000 7fa5c23e3640 1 -- 172.21.15.1:0/3505870451 >> v1:172.21.15.98:6789/0 conn(0x7fa5bc0c6b10 legacy=0x7fa5bc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.412+0000 7fa5c23e3640 1 -- 172.21.15.1:0/3505870451 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc0c6690 con 0x7fa5bc152650 2024-03-06T07:23:29.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.412+0000 7fa5c2be4640 1 --2- 172.21.15.1:0/3505870451 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa5bc0bf9d0 0x7fa5bc0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:29.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.413+0000 7fa5c23e3640 1 --2- 172.21.15.1:0/3505870451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5bc152650 0x7fa5bc155c90 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009870 tx=0x7fa5b802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62b33bf83fb6007 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.414+0000 7fa5c13e1640 1 -- 172.21.15.1:0/3505870451 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8035020 con 0x7fa5bc152650 2024-03-06T07:23:29.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.414+0000 7fa5c13e1640 1 -- 172.21.15.1:0/3505870451 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b8031640 con 0x7fa5bc152650 2024-03-06T07:23:29.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.414+0000 7fa5c13e1640 1 -- 172.21.15.1:0/3505870451 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b802d040 con 0x7fa5bc152650 2024-03-06T07:23:29.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.414+0000 7fa5c4647640 1 -- 172.21.15.1:0/3505870451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5bc152650 msgr2=0x7fa5bc155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.414+0000 7fa5c4647640 1 --2- 172.21.15.1:0/3505870451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5bc152650 0x7fa5bc155c90 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009870 tx=0x7fa5b802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.414+0000 7fa5c4647640 1 -- 172.21.15.1:0/3505870451 shutdown_connections 2024-03-06T07:23:29.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.414+0000 7fa5c4647640 1 --2- 172.21.15.1:0/3505870451 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa5bc0bf9d0 0x7fa5bc0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.414+0000 7fa5c4647640 1 --2- 172.21.15.1:0/3505870451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5bc152650 0x7fa5bc155c90 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.414+0000 7fa5c4647640 1 -- 172.21.15.1:0/3505870451 >> 172.21.15.1:0/3505870451 conn(0x7fa5bc066ee0 msgr2=0x7fa5bc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.414+0000 7fa5c4647640 1 -- 172.21.15.1:0/3505870451 shutdown_connections 2024-03-06T07:23:29.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.414+0000 7fa5c4647640 1 -- 172.21.15.1:0/3505870451 wait complete. 2024-03-06T07:23:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.415+0000 7fa5c4647640 1 Processor -- start 2024-03-06T07:23:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.415+0000 7fa5c4647640 1 -- start start 2024-03-06T07:23:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.416+0000 7fa5c4647640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5bc0bf9d0 0x7fa5bc14b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.416+0000 7fa5c4647640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa5bc152650 0x7fa5bc1df9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.416+0000 7fa5c4647640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5bc14b3c0 0x7fa5bc14a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.416+0000 7fa5c4647640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc1551b0 con 0x7fa5bc0bf9d0 2024-03-06T07:23:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.416+0000 7fa5c4647640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc0c6fa0 con 0x7fa5bc14b3c0 2024-03-06T07:23:29.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.416+0000 7fa5c4647640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc155490 con 0x7fa5bc152650 2024-03-06T07:23:29.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.416+0000 7fa5c23e3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5bc0bf9d0 0x7fa5bc14b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.416+0000 7fa5c23e3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5bc0bf9d0 0x7fa5bc14b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45386/0 (socket says 172.21.15.1:45386) 2024-03-06T07:23:29.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.416+0000 7fa5c23e3640 1 -- 172.21.15.1:0/2508839687 learned_addr learned my addr 172.21.15.1:0/2508839687 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:29.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.416+0000 7fa5c2be4640 1 --2- 172.21.15.1:0/2508839687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5bc14b3c0 0x7fa5bc14a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.416+0000 7fa5c1be2640 1 --2- 172.21.15.1:0/2508839687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa5bc152650 0x7fa5bc1df9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.417+0000 7fa5c23e3640 1 -- 172.21.15.1:0/2508839687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa5bc152650 msgr2=0x7fa5bc1df9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.417+0000 7fa5c23e3640 1 --2- 172.21.15.1:0/2508839687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa5bc152650 0x7fa5bc1df9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.417+0000 7fa5c23e3640 1 -- 172.21.15.1:0/2508839687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5bc14b3c0 msgr2=0x7fa5bc14a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.417+0000 7fa5c23e3640 1 --2- 172.21.15.1:0/2508839687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5bc14b3c0 0x7fa5bc14a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.417+0000 7fa5c23e3640 1 -- 172.21.15.1:0/2508839687 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc1ee480 con 0x7fa5bc0bf9d0 2024-03-06T07:23:29.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.417+0000 7fa5c1be2640 1 --2- 172.21.15.1:0/2508839687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa5bc152650 0x7fa5bc1df9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:29.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.417+0000 7fa5c2be4640 1 --2- 172.21.15.1:0/2508839687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5bc14b3c0 0x7fa5bc14a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:29.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.417+0000 7fa5c23e3640 1 --2- 172.21.15.1:0/2508839687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5bc0bf9d0 0x7fa5bc14b8f0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fa5b802b930 tx=0x7fa5b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.417+0000 7fa5b2ffd640 1 -- 172.21.15.1:0/2508839687 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8040070 con 0x7fa5bc0bf9d0 2024-03-06T07:23:29.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.417+0000 7fa5b2ffd640 1 -- 172.21.15.1:0/2508839687 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b803e070 con 0x7fa5bc0bf9d0 2024-03-06T07:23:29.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.417+0000 7fa5c4647640 1 -- 172.21.15.1:0/2508839687 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5bc1ee760 con 0x7fa5bc0bf9d0 2024-03-06T07:23:29.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.417+0000 7fa5b2ffd640 1 -- 172.21.15.1:0/2508839687 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8035040 con 0x7fa5bc0bf9d0 2024-03-06T07:23:29.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.417+0000 7fa5c4647640 1 -- 172.21.15.1:0/2508839687 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5bc1eed20 con 0x7fa5bc0bf9d0 2024-03-06T07:23:29.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.418+0000 7fa5c4647640 1 -- 172.21.15.1:0/2508839687 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5bc067fd0 con 0x7fa5bc0bf9d0 2024-03-06T07:23:29.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.418+0000 7fa5b2ffd640 1 -- 172.21.15.1:0/2508839687 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5b8038020 con 0x7fa5bc0bf9d0 2024-03-06T07:23:29.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.419+0000 7fa5b2ffd640 1 --2- 172.21.15.1:0/2508839687 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa5a006b4d0 0x7fa5a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.419+0000 7fa5b2ffd640 1 -- 172.21.15.1:0/2508839687 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7fa5b80abf00 con 0x7fa5bc0bf9d0 2024-03-06T07:23:29.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.419+0000 7fa5c1be2640 1 --2- 172.21.15.1:0/2508839687 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa5a006b4d0 0x7fa5a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.420+0000 7fa5c1be2640 1 --2- 172.21.15.1:0/2508839687 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa5a006b4d0 0x7fa5a006d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac007fd0 tx=0x7fa5ac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.423+0000 7fa5b2ffd640 1 -- 172.21.15.1:0/2508839687 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5b807ab00 con 0x7fa5bc0bf9d0 2024-03-06T07:23:29.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.647+0000 7fa5c4647640 1 -- 172.21.15.1:0/2508839687 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs2"} v 0) v1 -- 0x7fa5bc1dfef0 con 0x7fa5bc0bf9d0 2024-03-06T07:23:29.649 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:23:29.648+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:23:29.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.649+0000 7fa5b2ffd640 1 -- 172.21.15.1:0/2508839687 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs2"}]=0 cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v72) v1 ==== 163+0+0 (secure 0 0 0) 0x7fa5b807a400 con 0x7fa5bc0bf9d0 2024-03-06T07:23:29.651 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:23:29.653 INFO:tasks.ceph.mds.a.smithi001.stderr: -1654> 2024-03-06T07:22:52.529+0000 7fc62fb3b640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-03-06T07:23:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.651+0000 7fa5c4647640 1 -- 172.21.15.1:0/2508839687 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa5a006b4d0 msgr2=0x7fa5a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.651+0000 7fa5c4647640 1 --2- 172.21.15.1:0/2508839687 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa5a006b4d0 0x7fa5a006d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac007fd0 tx=0x7fa5ac009040 comp rx=0 tx=0).stop 2024-03-06T07:23:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.651+0000 7fa5c4647640 1 -- 172.21.15.1:0/2508839687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5bc0bf9d0 msgr2=0x7fa5bc14b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.651+0000 7fa5c4647640 1 --2- 172.21.15.1:0/2508839687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5bc0bf9d0 0x7fa5bc14b8f0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fa5b802b930 tx=0x7fa5b80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.651+0000 7fa5c4647640 1 -- 172.21.15.1:0/2508839687 shutdown_connections 2024-03-06T07:23:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.651+0000 7fa5c4647640 1 --2- 172.21.15.1:0/2508839687 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa5a006b4d0 0x7fa5a006d990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.651+0000 7fa5c4647640 1 --2- 172.21.15.1:0/2508839687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5bc14b3c0 0x7fa5bc14a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.651+0000 7fa5c4647640 1 --2- 172.21.15.1:0/2508839687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa5bc152650 0x7fa5bc1df9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.651+0000 7fa5c4647640 1 --2- 172.21.15.1:0/2508839687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5bc0bf9d0 0x7fa5bc14b8f0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.651+0000 7fa5c4647640 1 -- 172.21.15.1:0/2508839687 >> 172.21.15.1:0/2508839687 conn(0x7fa5bc066ee0 msgr2=0x7fa5bc0c3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.652+0000 7fa5c4647640 1 -- 172.21.15.1:0/2508839687 shutdown_connections 2024-03-06T07:23:29.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.652+0000 7fa5c4647640 1 -- 172.21.15.1:0/2508839687 wait complete. 2024-03-06T07:23:29.655 INFO:tasks.ceph.mds.a.smithi001.stderr: -67> 2024-03-06T07:23:27.903+0000 7fc629b2f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:29.668 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs2 --yes-i-really-mean-it 2024-03-06T07:23:29.682 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:23:29.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.770+0000 7f77eb62a640 1 Processor -- start 2024-03-06T07:23:29.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.771+0000 7f77eb62a640 1 -- start start 2024-03-06T07:23:29.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.771+0000 7f77eb62a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77e40c8cd0 0x7f77e40c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.771+0000 7f77eb62a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f77e40c3260 0x7f77e4155b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.771+0000 7f77eb62a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e40c2b90 con 0x7f77e40c7560 2024-03-06T07:23:29.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.771+0000 7f77eb62a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e40c37b0 con 0x7f77e40c3260 2024-03-06T07:23:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.771+0000 7f77eb62a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40c3a90 con 0x7f77e40c8cd0 2024-03-06T07:23:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.772+0000 7f77e8bc5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f77e40c3260 0x7f77e4155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.772+0000 7f77e93c6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77e40c8cd0 0x7f77e40c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.772+0000 7f77e9bc7640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f77e40c7560 0x7f77e4159010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:55468/0 (socket says 172.21.15.1:55468) 2024-03-06T07:23:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.772+0000 7f77e9bc7640 1 -- 172.21.15.1:0/3384409583 learned_addr learned my addr 172.21.15.1:0/3384409583 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.772+0000 7f77e3fff640 1 -- 172.21.15.1:0/3384409583 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228016259 0 0) 0x7f77e40c2b90 con 0x7f77e40c7560 2024-03-06T07:23:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.772+0000 7f77e3fff640 1 -- 172.21.15.1:0/3384409583 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77cc003520 con 0x7f77e40c7560 2024-03-06T07:23:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.772+0000 7f77e93c6640 1 -- 172.21.15.1:0/3384409583 >> v2:172.21.15.98:3300/0 conn(0x7f77e40c3260 msgr2=0x7f77e4155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.772+0000 7f77e93c6640 1 --2- 172.21.15.1:0/3384409583 >> v2:172.21.15.98:3300/0 conn(0x7f77e40c3260 0x7f77e4155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.772+0000 7f77e93c6640 1 -- 172.21.15.1:0/3384409583 >> v1:172.21.15.98:6789/0 conn(0x7f77e40c7560 legacy=0x7f77e4159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.772+0000 7f77e93c6640 1 -- 172.21.15.1:0/3384409583 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e4158810 con 0x7f77e40c8cd0 2024-03-06T07:23:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.772+0000 7f77e8bc5640 1 --2- 172.21.15.1:0/3384409583 >> v2:172.21.15.98:3300/0 conn(0x7f77e40c3260 0x7f77e4155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.773+0000 7f77e93c6640 1 --2- 172.21.15.1:0/3384409583 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77e40c8cd0 0x7f77e40c90b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009d90 tx=0x7f77dc02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=540f14d98408b031 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.773+0000 7f77e3fff640 1 -- 172.21.15.1:0/3384409583 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc039070 con 0x7f77e40c8cd0 2024-03-06T07:23:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.773+0000 7f77e3fff640 1 -- 172.21.15.1:0/3384409583 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77dc02cbc0 con 0x7f77e40c8cd0 2024-03-06T07:23:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.773+0000 7f77e3fff640 1 -- 172.21.15.1:0/3384409583 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc034440 con 0x7f77e40c8cd0 2024-03-06T07:23:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.774+0000 7f77eb62a640 1 -- 172.21.15.1:0/3384409583 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77e40c8cd0 msgr2=0x7f77e40c90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.774+0000 7f77eb62a640 1 --2- 172.21.15.1:0/3384409583 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77e40c8cd0 0x7f77e40c90b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009d90 tx=0x7f77dc02be10 comp rx=0 tx=0).stop 2024-03-06T07:23:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.774+0000 7f77eb62a640 1 -- 172.21.15.1:0/3384409583 shutdown_connections 2024-03-06T07:23:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.774+0000 7f77eb62a640 1 --2- 172.21.15.1:0/3384409583 >> v2:172.21.15.98:3300/0 conn(0x7f77e40c3260 0x7f77e4155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.774+0000 7f77eb62a640 1 --2- 172.21.15.1:0/3384409583 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77e40c8cd0 0x7f77e40c90b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.774+0000 7f77eb62a640 1 -- 172.21.15.1:0/3384409583 >> 172.21.15.1:0/3384409583 conn(0x7f77e406b3e0 msgr2=0x7f77e406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.774+0000 7f77eb62a640 1 -- 172.21.15.1:0/3384409583 shutdown_connections 2024-03-06T07:23:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.774+0000 7f77eb62a640 1 -- 172.21.15.1:0/3384409583 wait complete. 2024-03-06T07:23:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.775+0000 7f77eb62a640 1 Processor -- start 2024-03-06T07:23:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.775+0000 7f77eb62a640 1 -- start start 2024-03-06T07:23:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.775+0000 7f77eb62a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77e40c3260 0x7f77e414f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.775+0000 7f77eb62a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77e40c7560 0x7f77e40aa100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.775+0000 7f77eb62a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77e40c8cd0 0x7f77e414de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.775+0000 7f77eb62a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40c8560 con 0x7f77e40c8cd0 2024-03-06T07:23:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.775+0000 7f77eb62a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40c7a10 con 0x7f77e40c3260 2024-03-06T07:23:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.775+0000 7f77eb62a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40c86e0 con 0x7f77e40c7560 2024-03-06T07:23:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.776+0000 7f77e9bc7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77e40c8cd0 0x7f77e414de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.776+0000 7f77e9bc7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77e40c8cd0 0x7f77e414de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45420/0 (socket says 172.21.15.1:45420) 2024-03-06T07:23:29.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.776+0000 7f77e9bc7640 1 -- 172.21.15.1:0/2391085165 learned_addr learned my addr 172.21.15.1:0/2391085165 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.776+0000 7f77e8bc5640 1 --2- 172.21.15.1:0/2391085165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77e40c7560 0x7f77e40aa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.776+0000 7f77e93c6640 1 --2- 172.21.15.1:0/2391085165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77e40c3260 0x7f77e414f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.776+0000 7f77e9bc7640 1 -- 172.21.15.1:0/2391085165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77e40c7560 msgr2=0x7f77e40aa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.776+0000 7f77e9bc7640 1 --2- 172.21.15.1:0/2391085165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77e40c7560 0x7f77e40aa100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.776+0000 7f77e9bc7640 1 -- 172.21.15.1:0/2391085165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77e40c3260 msgr2=0x7f77e414f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.776+0000 7f77e9bc7640 1 --2- 172.21.15.1:0/2391085165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77e40c3260 0x7f77e414f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.776+0000 7f77e9bc7640 1 -- 172.21.15.1:0/2391085165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e41eee50 con 0x7f77e40c8cd0 2024-03-06T07:23:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.776+0000 7f77e9bc7640 1 --2- 172.21.15.1:0/2391085165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77e40c8cd0 0x7f77e414de20 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f77d400c430 tx=0x7f77d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.776+0000 7f77e1ffb640 1 -- 172.21.15.1:0/2391085165 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d4017070 con 0x7f77e40c8cd0 2024-03-06T07:23:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.776+0000 7f77e1ffb640 1 -- 172.21.15.1:0/2391085165 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d4005ce0 con 0x7f77e40c8cd0 2024-03-06T07:23:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.776+0000 7f77eb62a640 1 -- 172.21.15.1:0/2391085165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e41dfaa0 con 0x7f77e40c8cd0 2024-03-06T07:23:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.776+0000 7f77e1ffb640 1 -- 172.21.15.1:0/2391085165 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d4007310 con 0x7f77e40c8cd0 2024-03-06T07:23:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.777+0000 7f77eb62a640 1 -- 172.21.15.1:0/2391085165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e414eb30 con 0x7f77e40c8cd0 2024-03-06T07:23:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.777+0000 7f77eb62a640 1 -- 172.21.15.1:0/2391085165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e4156400 con 0x7f77e40c8cd0 2024-03-06T07:23:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.778+0000 7f77e1ffb640 1 -- 172.21.15.1:0/2391085165 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f77d401a070 con 0x7f77e40c8cd0 2024-03-06T07:23:29.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.778+0000 7f77e1ffb640 1 --2- 172.21.15.1:0/2391085165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f77d006b4d0 0x7f77d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.778+0000 7f77e1ffb640 1 -- 172.21.15.1:0/2391085165 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f77d4089990 con 0x7f77e40c8cd0 2024-03-06T07:23:29.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.778+0000 7f77e93c6640 1 --2- 172.21.15.1:0/2391085165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f77d006b4d0 0x7f77d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.779+0000 7f77e93c6640 1 --2- 172.21.15.1:0/2391085165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f77d006b4d0 0x7f77d006d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f77dc036000 tx=0x7f77dc030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:29.782+0000 7f77e1ffb640 1 -- 172.21.15.1:0/2391085165 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f77e4156400 con 0x7f77e40c8cd0 2024-03-06T07:23:30.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.002+0000 7f77eb62a640 1 -- 172.21.15.1:0/2391085165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f77e4067fd0 con 0x7f77e40c8cd0 2024-03-06T07:23:30.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.650+0000 7f77e1ffb640 1 -- 172.21.15.1:0/2391085165 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true}]=0 v73) v1 ==== 105+0+0 (secure 0 0 0) 0x7f77d4057e40 con 0x7f77e40c8cd0 2024-03-06T07:23:30.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.654+0000 7f77eb62a640 1 -- 172.21.15.1:0/2391085165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f77d006b4d0 msgr2=0x7f77d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.654+0000 7f77eb62a640 1 --2- 172.21.15.1:0/2391085165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f77d006b4d0 0x7f77d006d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f77dc036000 tx=0x7f77dc030000 comp rx=0 tx=0).stop 2024-03-06T07:23:30.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.654+0000 7f77eb62a640 1 -- 172.21.15.1:0/2391085165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77e40c8cd0 msgr2=0x7f77e414de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.654+0000 7f77eb62a640 1 --2- 172.21.15.1:0/2391085165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77e40c8cd0 0x7f77e414de20 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f77d400c430 tx=0x7f77d400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.654+0000 7f77eb62a640 1 -- 172.21.15.1:0/2391085165 shutdown_connections 2024-03-06T07:23:30.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.654+0000 7f77eb62a640 1 --2- 172.21.15.1:0/2391085165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f77d006b4d0 0x7f77d006d990 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.654+0000 7f77eb62a640 1 --2- 172.21.15.1:0/2391085165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77e40c8cd0 0x7f77e414de20 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.654+0000 7f77eb62a640 1 --2- 172.21.15.1:0/2391085165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77e40c7560 0x7f77e40aa100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.654+0000 7f77eb62a640 1 --2- 172.21.15.1:0/2391085165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77e40c3260 0x7f77e414f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.654+0000 7f77eb62a640 1 -- 172.21.15.1:0/2391085165 >> 172.21.15.1:0/2391085165 conn(0x7f77e406b3e0 msgr2=0x7f77e41579f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.654+0000 7f77eb62a640 1 -- 172.21.15.1:0/2391085165 shutdown_connections 2024-03-06T07:23:30.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.654+0000 7f77eb62a640 1 -- 172.21.15.1:0/2391085165 wait complete. 2024-03-06T07:23:30.670 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_metadata cephfs2_metadata --yes-i-really-really-mean-it 2024-03-06T07:23:30.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.822+0000 7f7d9ac66640 1 Processor -- start 2024-03-06T07:23:30.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.822+0000 7f7d9ac66640 1 -- start start 2024-03-06T07:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.823+0000 7f7d9ac66640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7d940c1010 0x7f7d940c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.823+0000 7f7d9ac66640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d940c7500 0x7f7d940cab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.823+0000 7f7d9ac66640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d940c6710 con 0x7f7d94152780 2024-03-06T07:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.823+0000 7f7d9ac66640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d940c6c90 con 0x7f7d940c7500 2024-03-06T07:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.823+0000 7f7d9ac66640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d940c69b0 con 0x7f7d940c1010 2024-03-06T07:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.823+0000 7f7d99203640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d940c7500 0x7f7d940cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.823+0000 7f7d99203640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d940c7500 0x7f7d940cab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45426/0 (socket says 172.21.15.1:45426) 2024-03-06T07:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.823+0000 7f7d99203640 1 -- 172.21.15.1:0/2598933534 learned_addr learned my addr 172.21.15.1:0/2598933534 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.823+0000 7f7d93fff640 1 --2- 172.21.15.1:0/2598933534 >> v2:172.21.15.98:3300/0 conn(0x7f7d940c1010 0x7f7d940c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.824+0000 7f7d99203640 1 -- 172.21.15.1:0/2598933534 >> v2:172.21.15.98:3300/0 conn(0x7f7d940c1010 msgr2=0x7f7d940c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.824+0000 7f7d99203640 1 --2- 172.21.15.1:0/2598933534 >> v2:172.21.15.98:3300/0 conn(0x7f7d940c1010 0x7f7d940c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.824+0000 7f7d99203640 1 -- 172.21.15.1:0/2598933534 >> v1:172.21.15.98:6789/0 conn(0x7f7d94152780 legacy=0x7f7d940c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.824+0000 7f7d99203640 1 -- 172.21.15.1:0/2598933534 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d94155370 con 0x7f7d940c7500 2024-03-06T07:23:30.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.824+0000 7f7d93fff640 1 --2- 172.21.15.1:0/2598933534 >> v2:172.21.15.98:3300/0 conn(0x7f7d940c1010 0x7f7d940c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:30.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.824+0000 7f7d99203640 1 --2- 172.21.15.1:0/2598933534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d940c7500 0x7f7d940cab00 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f7d940c5d80 tx=0x7f7d7c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f0f02b1a8ba4c27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.825+0000 7f7d937fe640 1 -- 172.21.15.1:0/2598933534 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c035020 con 0x7f7d940c7500 2024-03-06T07:23:30.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.825+0000 7f7d937fe640 1 -- 172.21.15.1:0/2598933534 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7c031870 con 0x7f7d940c7500 2024-03-06T07:23:30.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.825+0000 7f7d937fe640 1 -- 172.21.15.1:0/2598933534 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c031c40 con 0x7f7d940c7500 2024-03-06T07:23:30.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.825+0000 7f7d9ac66640 1 -- 172.21.15.1:0/2598933534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d940c7500 msgr2=0x7f7d940cab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.825+0000 7f7d9ac66640 1 --2- 172.21.15.1:0/2598933534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d940c7500 0x7f7d940cab00 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f7d940c5d80 tx=0x7f7d7c02b910 comp rx=0 tx=0).stop 2024-03-06T07:23:30.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.825+0000 7f7d9ac66640 1 -- 172.21.15.1:0/2598933534 shutdown_connections 2024-03-06T07:23:30.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.825+0000 7f7d9ac66640 1 --2- 172.21.15.1:0/2598933534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d940c7500 0x7f7d940cab00 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.825+0000 7f7d9ac66640 1 --2- 172.21.15.1:0/2598933534 >> v2:172.21.15.98:3300/0 conn(0x7f7d940c1010 0x7f7d940c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.825+0000 7f7d9ac66640 1 -- 172.21.15.1:0/2598933534 >> 172.21.15.1:0/2598933534 conn(0x7f7d9408b720 msgr2=0x7f7d9406d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.825+0000 7f7d9ac66640 1 -- 172.21.15.1:0/2598933534 shutdown_connections 2024-03-06T07:23:30.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.825+0000 7f7d9ac66640 1 -- 172.21.15.1:0/2598933534 wait complete. 2024-03-06T07:23:30.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.826+0000 7f7d9ac66640 1 Processor -- start 2024-03-06T07:23:30.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.826+0000 7f7d9ac66640 1 -- start start 2024-03-06T07:23:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.827+0000 7f7d9ac66640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d940c1010 0x7f7d94150cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.827+0000 7f7d9ac66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d940c7500 0x7f7d941e1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.827+0000 7f7d9ac66640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d94152780 0x7f7d941e1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.827+0000 7f7d9ac66640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d94152b60 con 0x7f7d94152780 2024-03-06T07:23:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.827+0000 7f7d9ac66640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d94155930 con 0x7f7d940c7500 2024-03-06T07:23:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.827+0000 7f7d9ac66640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d940cb040 con 0x7f7d940c1010 2024-03-06T07:23:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.827+0000 7f7d99203640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d94152780 0x7f7d941e1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.827+0000 7f7d99203640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d94152780 0x7f7d941e1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45440/0 (socket says 172.21.15.1:45440) 2024-03-06T07:23:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.827+0000 7f7d99203640 1 -- 172.21.15.1:0/4268051840 learned_addr learned my addr 172.21.15.1:0/4268051840 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.828+0000 7f7d98a02640 1 --2- 172.21.15.1:0/4268051840 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d940c1010 0x7f7d94150cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.828+0000 7f7d93fff640 1 --2- 172.21.15.1:0/4268051840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d940c7500 0x7f7d941e1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.828+0000 7f7d99203640 1 -- 172.21.15.1:0/4268051840 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d940c1010 msgr2=0x7f7d94150cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.828+0000 7f7d99203640 1 --2- 172.21.15.1:0/4268051840 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d940c1010 0x7f7d94150cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.828+0000 7f7d99203640 1 -- 172.21.15.1:0/4268051840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d940c7500 msgr2=0x7f7d941e1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.828+0000 7f7d99203640 1 --2- 172.21.15.1:0/4268051840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d940c7500 0x7f7d941e1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.828+0000 7f7d99203640 1 -- 172.21.15.1:0/4268051840 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d941ef130 con 0x7f7d94152780 2024-03-06T07:23:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.828+0000 7f7d99203640 1 --2- 172.21.15.1:0/4268051840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d94152780 0x7f7d941e1e60 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009960 tx=0x7f7d7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.828+0000 7f7d917fa640 1 -- 172.21.15.1:0/4268051840 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c040070 con 0x7f7d94152780 2024-03-06T07:23:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.828+0000 7f7d917fa640 1 -- 172.21.15.1:0/4268051840 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7c030c50 con 0x7f7d94152780 2024-03-06T07:23:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.828+0000 7f7d9ac66640 1 -- 172.21.15.1:0/4268051840 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d94154ba0 con 0x7f7d94152780 2024-03-06T07:23:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.828+0000 7f7d917fa640 1 -- 172.21.15.1:0/4268051840 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c035040 con 0x7f7d94152780 2024-03-06T07:23:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.828+0000 7f7d9ac66640 1 -- 172.21.15.1:0/4268051840 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d94154e80 con 0x7f7d94152780 2024-03-06T07:23:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.829+0000 7f7d9ac66640 1 -- 172.21.15.1:0/4268051840 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d940680b0 con 0x7f7d94152780 2024-03-06T07:23:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.830+0000 7f7d917fa640 1 -- 172.21.15.1:0/4268051840 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d7c038020 con 0x7f7d94152780 2024-03-06T07:23:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.830+0000 7f7d917fa640 1 --2- 172.21.15.1:0/4268051840 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d7406b4d0 0x7f7d7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.830+0000 7f7d917fa640 1 -- 172.21.15.1:0/4268051840 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f7d7c0ac270 con 0x7f7d94152780 2024-03-06T07:23:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.830+0000 7f7d98a02640 1 --2- 172.21.15.1:0/4268051840 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d7406b4d0 0x7f7d7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.831+0000 7f7d98a02640 1 --2- 172.21.15.1:0/4268051840 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d7406b4d0 0x7f7d7406d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7d94154a00 tx=0x7f7d8c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:30.834+0000 7f7d917fa640 1 -- 172.21.15.1:0/4268051840 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d7c07ae90 con 0x7f7d94152780 2024-03-06T07:23:31.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.014+0000 7f7d9ac66640 1 -- 172.21.15.1:0/4268051840 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7d94155150 con 0x7f7d94152780 2024-03-06T07:23:31.330 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:31.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:31.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.659+0000 7f7d917fa640 1 -- 172.21.15.1:0/4268051840 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_metadata' removed v85) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7d7c07a790 con 0x7f7d94152780 2024-03-06T07:23:31.661 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs2_metadata' removed 2024-03-06T07:23:31.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.663+0000 7f7d9ac66640 1 -- 172.21.15.1:0/4268051840 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d7406b4d0 msgr2=0x7f7d7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.663+0000 7f7d9ac66640 1 --2- 172.21.15.1:0/4268051840 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d7406b4d0 0x7f7d7406d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7d94154a00 tx=0x7f7d8c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:31.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.663+0000 7f7d9ac66640 1 -- 172.21.15.1:0/4268051840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d94152780 msgr2=0x7f7d941e1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.663+0000 7f7d9ac66640 1 --2- 172.21.15.1:0/4268051840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d94152780 0x7f7d941e1e60 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009960 tx=0x7f7d7c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.664+0000 7f7d9ac66640 1 -- 172.21.15.1:0/4268051840 shutdown_connections 2024-03-06T07:23:31.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.664+0000 7f7d9ac66640 1 --2- 172.21.15.1:0/4268051840 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d7406b4d0 0x7f7d7406d990 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.664+0000 7f7d9ac66640 1 --2- 172.21.15.1:0/4268051840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d94152780 0x7f7d941e1e60 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.664+0000 7f7d9ac66640 1 --2- 172.21.15.1:0/4268051840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d940c7500 0x7f7d941e1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.664+0000 7f7d9ac66640 1 --2- 172.21.15.1:0/4268051840 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d940c1010 0x7f7d94150cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.664+0000 7f7d9ac66640 1 -- 172.21.15.1:0/4268051840 >> 172.21.15.1:0/4268051840 conn(0x7f7d9408b720 msgr2=0x7f7d940c8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.664+0000 7f7d9ac66640 1 -- 172.21.15.1:0/4268051840 shutdown_connections 2024-03-06T07:23:31.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.664+0000 7f7d9ac66640 1 -- 172.21.15.1:0/4268051840 wait complete. 2024-03-06T07:23:31.679 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_data cephfs2_data --yes-i-really-really-mean-it 2024-03-06T07:23:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.788+0000 7f3d2ccbb640 1 Processor -- start 2024-03-06T07:23:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.788+0000 7f3d2ccbb640 1 -- start start 2024-03-06T07:23:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.788+0000 7f3d2ccbb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3d280a7150 0x7f3d280c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.788+0000 7f3d2ccbb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280c0df0 0x7f3d280c1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.788+0000 7f3d2ccbb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d28155200 con 0x7f3d280c58c0 2024-03-06T07:23:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.788+0000 7f3d2ccbb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d280a7530 con 0x7f3d280a7150 2024-03-06T07:23:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.788+0000 7f3d2ccbb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d28155470 con 0x7f3d280c0df0 2024-03-06T07:23:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.789+0000 7f3d2659c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3d280a7150 0x7f3d280c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.789+0000 7f3d1ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280c0df0 0x7f3d280c1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.789+0000 7f3d1ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280c0df0 0x7f3d280c1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:45836/0 (socket says 172.21.15.1:45836) 2024-03-06T07:23:31.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.789+0000 7f3d2659c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3d280a7150 0x7f3d280c6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:34114/0 (socket says 172.21.15.1:34114) 2024-03-06T07:23:31.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.789+0000 7f3d1ffff640 1 -- 172.21.15.1:0/729045912 learned_addr learned my addr 172.21.15.1:0/729045912 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:31.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.789+0000 7f3d2659c640 1 -- 172.21.15.1:0/729045912 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280c0df0 msgr2=0x7f3d280c1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:31.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.789+0000 7f3d2659c640 1 --2- 172.21.15.1:0/729045912 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280c0df0 0x7f3d280c1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.789+0000 7f3d2659c640 1 -- 172.21.15.1:0/729045912 >> v1:172.21.15.98:6789/0 conn(0x7f3d280c58c0 legacy=0x7f3d280c5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.789+0000 7f3d2659c640 1 -- 172.21.15.1:0/729045912 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d28155750 con 0x7f3d280a7150 2024-03-06T07:23:31.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.789+0000 7f3d25d9b640 1 -- 172.21.15.1:0/729045912 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843337295 0 0) 0x7f3d28155200 con 0x7f3d280c58c0 2024-03-06T07:23:31.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.789+0000 7f3d1ffff640 1 --2- 172.21.15.1:0/729045912 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280c0df0 0x7f3d280c1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:31.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.790+0000 7f3d2659c640 1 --2- 172.21.15.1:0/729045912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d280a7150 0x7f3d280c6ec0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3d180097a0 tx=0x7f3d1802b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f9a75e90cfeaf32 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.790+0000 7f3d25d9b640 1 -- 172.21.15.1:0/729045912 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d18038070 con 0x7f3d280a7150 2024-03-06T07:23:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.790+0000 7f3d25d9b640 1 -- 172.21.15.1:0/729045912 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d18031700 con 0x7f3d280a7150 2024-03-06T07:23:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.790+0000 7f3d25d9b640 1 -- 172.21.15.1:0/729045912 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d18031ad0 con 0x7f3d280a7150 2024-03-06T07:23:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.791+0000 7f3d2ccbb640 1 -- 172.21.15.1:0/729045912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d280a7150 msgr2=0x7f3d280c6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.791+0000 7f3d2ccbb640 1 --2- 172.21.15.1:0/729045912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d280a7150 0x7f3d280c6ec0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3d180097a0 tx=0x7f3d1802b7a0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.791+0000 7f3d2ccbb640 1 -- 172.21.15.1:0/729045912 shutdown_connections 2024-03-06T07:23:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.791+0000 7f3d2ccbb640 1 --2- 172.21.15.1:0/729045912 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280c0df0 0x7f3d280c1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.791+0000 7f3d2ccbb640 1 --2- 172.21.15.1:0/729045912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d280a7150 0x7f3d280c6ec0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.791+0000 7f3d2ccbb640 1 -- 172.21.15.1:0/729045912 >> 172.21.15.1:0/729045912 conn(0x7f3d280c44c0 msgr2=0x7f3d280c48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.791+0000 7f3d2ccbb640 1 -- 172.21.15.1:0/729045912 shutdown_connections 2024-03-06T07:23:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.791+0000 7f3d2ccbb640 1 -- 172.21.15.1:0/729045912 wait complete. 2024-03-06T07:23:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.792+0000 7f3d2ccbb640 1 Processor -- start 2024-03-06T07:23:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.792+0000 7f3d2ccbb640 1 -- start start 2024-03-06T07:23:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.793+0000 7f3d2ccbb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280a7150 0x7f3d281eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.793+0000 7f3d2ccbb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d280c0df0 0x7f3d280c3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.793+0000 7f3d2ccbb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d280c58c0 0x7f3d281ec0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.793+0000 7f3d2ccbb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d280c7710 con 0x7f3d280c0df0 2024-03-06T07:23:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.793+0000 7f3d2ccbb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d280c7400 con 0x7f3d280c58c0 2024-03-06T07:23:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.793+0000 7f3d2ccbb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d28155f50 con 0x7f3d280a7150 2024-03-06T07:23:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.793+0000 7f3d1ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d280c0df0 0x7f3d280c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.793+0000 7f3d1ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d280c0df0 0x7f3d280c3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45456/0 (socket says 172.21.15.1:45456) 2024-03-06T07:23:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.793+0000 7f3d1ffff640 1 -- 172.21.15.1:0/926175226 learned_addr learned my addr 172.21.15.1:0/926175226 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.793+0000 7f3d2659c640 1 --2- 172.21.15.1:0/926175226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280a7150 0x7f3d281eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.793+0000 7f3d26d9d640 1 --2- 172.21.15.1:0/926175226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d280c58c0 0x7f3d281ec0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.793+0000 7f3d1ffff640 1 -- 172.21.15.1:0/926175226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280a7150 msgr2=0x7f3d281eebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.793+0000 7f3d1ffff640 1 --2- 172.21.15.1:0/926175226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280a7150 0x7f3d281eebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.793+0000 7f3d1ffff640 1 -- 172.21.15.1:0/926175226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d280c58c0 msgr2=0x7f3d281ec0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.793+0000 7f3d1ffff640 1 --2- 172.21.15.1:0/926175226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d280c58c0 0x7f3d281ec0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.793+0000 7f3d1ffff640 1 -- 172.21.15.1:0/926175226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d280acf10 con 0x7f3d280c0df0 2024-03-06T07:23:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.794+0000 7f3d1ffff640 1 --2- 172.21.15.1:0/926175226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d280c0df0 0x7f3d280c3080 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f3d1000e9d0 tx=0x7f3d1000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.794+0000 7f3d1effd640 1 -- 172.21.15.1:0/926175226 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d10017070 con 0x7f3d280c0df0 2024-03-06T07:23:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.794+0000 7f3d1effd640 1 -- 172.21.15.1:0/926175226 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1000ca60 con 0x7f3d280c0df0 2024-03-06T07:23:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.794+0000 7f3d2ccbb640 1 -- 172.21.15.1:0/926175226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d281eae40 con 0x7f3d280c0df0 2024-03-06T07:23:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.794+0000 7f3d1effd640 1 -- 172.21.15.1:0/926175226 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1000ce10 con 0x7f3d280c0df0 2024-03-06T07:23:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.794+0000 7f3d2ccbb640 1 -- 172.21.15.1:0/926175226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d280c35c0 con 0x7f3d280c0df0 2024-03-06T07:23:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.795+0000 7f3d2ccbb640 1 -- 172.21.15.1:0/926175226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d281eac30 con 0x7f3d280c0df0 2024-03-06T07:23:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.795+0000 7f3d1effd640 1 -- 172.21.15.1:0/926175226 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d1001a070 con 0x7f3d280c0df0 2024-03-06T07:23:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.795+0000 7f3d1effd640 1 --2- 172.21.15.1:0/926175226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3d0806b4d0 0x7f3d0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.796+0000 7f3d1effd640 1 -- 172.21.15.1:0/926175226 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6009+0+0 (secure 0 0 0) 0x7f3d1008a5c0 con 0x7f3d280c0df0 2024-03-06T07:23:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.796+0000 7f3d2659c640 1 --2- 172.21.15.1:0/926175226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3d0806b4d0 0x7f3d0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.796+0000 7f3d2659c640 1 --2- 172.21.15.1:0/926175226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3d0806b4d0 0x7f3d0806d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3d18035000 tx=0x7f3d18008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.800+0000 7f3d1effd640 1 -- 172.21.15.1:0/926175226 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d100593a0 con 0x7f3d280c0df0 2024-03-06T07:23:31.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:31.983+0000 7f3d2ccbb640 1 -- 172.21.15.1:0/926175226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3d280680b0 con 0x7f3d280c0df0 2024-03-06T07:23:32.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.660+0000 7f3d1effd640 1 -- 172.21.15.1:0/926175226 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_data' removed v86) v1 ==== 172+0+0 (secure 0 0 0) 0x7f3d10058ca0 con 0x7f3d280c0df0 2024-03-06T07:23:32.663 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs2_data' removed 2024-03-06T07:23:32.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.664+0000 7f3d2ccbb640 1 -- 172.21.15.1:0/926175226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3d0806b4d0 msgr2=0x7f3d0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.664+0000 7f3d2ccbb640 1 --2- 172.21.15.1:0/926175226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3d0806b4d0 0x7f3d0806d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3d18035000 tx=0x7f3d18008030 comp rx=0 tx=0).stop 2024-03-06T07:23:32.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.664+0000 7f3d2ccbb640 1 -- 172.21.15.1:0/926175226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d280c0df0 msgr2=0x7f3d280c3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.664+0000 7f3d2ccbb640 1 --2- 172.21.15.1:0/926175226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d280c0df0 0x7f3d280c3080 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f3d1000e9d0 tx=0x7f3d1000ee90 comp rx=0 tx=0).stop 2024-03-06T07:23:32.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.665+0000 7f3d2ccbb640 1 -- 172.21.15.1:0/926175226 shutdown_connections 2024-03-06T07:23:32.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.665+0000 7f3d2ccbb640 1 --2- 172.21.15.1:0/926175226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3d0806b4d0 0x7f3d0806d990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.665+0000 7f3d2ccbb640 1 --2- 172.21.15.1:0/926175226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d280c58c0 0x7f3d281ec0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.665+0000 7f3d2ccbb640 1 --2- 172.21.15.1:0/926175226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d280c0df0 0x7f3d280c3080 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.665+0000 7f3d2ccbb640 1 --2- 172.21.15.1:0/926175226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3d280a7150 0x7f3d281eebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.665+0000 7f3d2ccbb640 1 -- 172.21.15.1:0/926175226 >> 172.21.15.1:0/926175226 conn(0x7f3d280c44c0 msgr2=0x7f3d2814e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.665+0000 7f3d2ccbb640 1 -- 172.21.15.1:0/926175226 shutdown_connections 2024-03-06T07:23:32.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.665+0000 7f3d2ccbb640 1 -- 172.21.15.1:0/926175226 wait complete. 2024-03-06T07:23:32.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:23:32.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.788+0000 7fda74e2a640 1 Processor -- start 2024-03-06T07:23:32.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.788+0000 7fda74e2a640 1 -- start start 2024-03-06T07:23:32.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.789+0000 7fda74e2a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fda700c79c0 0x7fda70152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.789+0000 7fda74e2a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda700a7ec0 0x7fda701533e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.789+0000 7fda74e2a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda701566e0 con 0x7fda700c4dd0 2024-03-06T07:23:32.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.789+0000 7fda74e2a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda70156490 con 0x7fda700c79c0 2024-03-06T07:23:32.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.789+0000 7fda74e2a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda70157010 con 0x7fda700a7ec0 2024-03-06T07:23:32.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.789+0000 7fda6e59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fda700c4dd0 0x7fda700c5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50246/0 (socket says 172.21.15.1:50246) 2024-03-06T07:23:32.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.789+0000 7fda6e59c640 1 -- 172.21.15.1:0/988360439 learned_addr learned my addr 172.21.15.1:0/988360439 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:32.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.789+0000 7fda6dd9b640 1 --2- 172.21.15.1:0/988360439 >> v2:172.21.15.1:3300/0 conn(0x7fda700c79c0 0x7fda70152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.789+0000 7fda6ed9d640 1 --2- 172.21.15.1:0/988360439 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda700a7ec0 0x7fda701533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.789+0000 7fda6d59a640 1 -- 172.21.15.1:0/988360439 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114330773 0 0) 0x7fda701566e0 con 0x7fda700c4dd0 2024-03-06T07:23:32.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.789+0000 7fda6d59a640 1 -- 172.21.15.1:0/988360439 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda58003520 con 0x7fda700c4dd0 2024-03-06T07:23:32.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.789+0000 7fda6dd9b640 1 -- 172.21.15.1:0/988360439 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda700a7ec0 msgr2=0x7fda701533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.789+0000 7fda6dd9b640 1 --2- 172.21.15.1:0/988360439 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda700a7ec0 0x7fda701533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.789+0000 7fda6dd9b640 1 -- 172.21.15.1:0/988360439 >> v1:172.21.15.1:6789/0 conn(0x7fda700c4dd0 legacy=0x7fda700c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.789+0000 7fda6dd9b640 1 -- 172.21.15.1:0/988360439 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda701569c0 con 0x7fda700c79c0 2024-03-06T07:23:32.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.790+0000 7fda6dd9b640 1 --2- 172.21.15.1:0/988360439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda700c79c0 0x7fda70152ea0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fda64009d90 tx=0x7fda6402bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e5554330ec32fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.790+0000 7fda6d59a640 1 -- 172.21.15.1:0/988360439 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda64039070 con 0x7fda700c79c0 2024-03-06T07:23:32.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.790+0000 7fda6d59a640 1 -- 172.21.15.1:0/988360439 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda6402cb30 con 0x7fda700c79c0 2024-03-06T07:23:32.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.790+0000 7fda6d59a640 1 -- 172.21.15.1:0/988360439 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda64034440 con 0x7fda700c79c0 2024-03-06T07:23:32.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.791+0000 7fda74e2a640 1 -- 172.21.15.1:0/988360439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda700c79c0 msgr2=0x7fda70152ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.791+0000 7fda74e2a640 1 --2- 172.21.15.1:0/988360439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda700c79c0 0x7fda70152ea0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fda64009d90 tx=0x7fda6402bd80 comp rx=0 tx=0).stop 2024-03-06T07:23:32.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.791+0000 7fda74e2a640 1 -- 172.21.15.1:0/988360439 shutdown_connections 2024-03-06T07:23:32.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.791+0000 7fda74e2a640 1 --2- 172.21.15.1:0/988360439 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda700a7ec0 0x7fda701533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.791+0000 7fda74e2a640 1 --2- 172.21.15.1:0/988360439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda700c79c0 0x7fda70152ea0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.791+0000 7fda74e2a640 1 -- 172.21.15.1:0/988360439 >> 172.21.15.1:0/988360439 conn(0x7fda7006b410 msgr2=0x7fda7006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.791+0000 7fda74e2a640 1 -- 172.21.15.1:0/988360439 shutdown_connections 2024-03-06T07:23:32.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.791+0000 7fda74e2a640 1 -- 172.21.15.1:0/988360439 wait complete. 2024-03-06T07:23:32.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.792+0000 7fda74e2a640 1 Processor -- start 2024-03-06T07:23:32.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.792+0000 7fda74e2a640 1 -- start start 2024-03-06T07:23:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda74e2a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda700a7ec0 0x7fda700bddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda74e2a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda700c4dd0 0x7fda70142a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda74e2a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda700be310 0x7fda70142f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda74e2a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda700c60d0 con 0x7fda700be310 2024-03-06T07:23:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda74e2a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda700c5f50 con 0x7fda700c4dd0 2024-03-06T07:23:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda74e2a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda700715f0 con 0x7fda700a7ec0 2024-03-06T07:23:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda6ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda700be310 0x7fda70142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda6ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda700be310 0x7fda70142f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45464/0 (socket says 172.21.15.1:45464) 2024-03-06T07:23:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda6ed9d640 1 -- 172.21.15.1:0/3005116170 learned_addr learned my addr 172.21.15.1:0/3005116170 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda6e59c640 1 --2- 172.21.15.1:0/3005116170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda700a7ec0 0x7fda700bddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda6dd9b640 1 --2- 172.21.15.1:0/3005116170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda700c4dd0 0x7fda70142a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda6ed9d640 1 -- 172.21.15.1:0/3005116170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda700a7ec0 msgr2=0x7fda700bddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda6ed9d640 1 --2- 172.21.15.1:0/3005116170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda700a7ec0 0x7fda700bddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda6ed9d640 1 -- 172.21.15.1:0/3005116170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda700c4dd0 msgr2=0x7fda70142a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda6ed9d640 1 --2- 172.21.15.1:0/3005116170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda700c4dd0 0x7fda70142a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda6ed9d640 1 -- 172.21.15.1:0/3005116170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda70153760 con 0x7fda700be310 2024-03-06T07:23:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda6dd9b640 1 --2- 172.21.15.1:0/3005116170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda700c4dd0 0x7fda70142a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda6ed9d640 1 --2- 172.21.15.1:0/3005116170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda700be310 0x7fda70142f70 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fda6000e7f0 tx=0x7fda6000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.793+0000 7fda5effd640 1 -- 172.21.15.1:0/3005116170 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda60017070 con 0x7fda700be310 2024-03-06T07:23:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.794+0000 7fda5effd640 1 -- 172.21.15.1:0/3005116170 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda6000ee40 con 0x7fda700be310 2024-03-06T07:23:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.794+0000 7fda74e2a640 1 -- 172.21.15.1:0/3005116170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda70153a40 con 0x7fda700be310 2024-03-06T07:23:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.794+0000 7fda5effd640 1 -- 172.21.15.1:0/3005116170 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6000cb80 con 0x7fda700be310 2024-03-06T07:23:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.794+0000 7fda74e2a640 1 -- 172.21.15.1:0/3005116170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda70151590 con 0x7fda700be310 2024-03-06T07:23:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.795+0000 7fda74e2a640 1 -- 172.21.15.1:0/3005116170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda70067fd0 con 0x7fda700be310 2024-03-06T07:23:32.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.795+0000 7fda5effd640 1 -- 172.21.15.1:0/3005116170 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda6001a070 con 0x7fda700be310 2024-03-06T07:23:32.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.796+0000 7fda5effd640 1 --2- 172.21.15.1:0/3005116170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda5006b4d0 0x7fda5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.796+0000 7fda5effd640 1 -- 172.21.15.1:0/3005116170 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fda6008a3b0 con 0x7fda700be310 2024-03-06T07:23:32.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.796+0000 7fda6e59c640 1 --2- 172.21.15.1:0/3005116170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda5006b4d0 0x7fda5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.796+0000 7fda6e59c640 1 --2- 172.21.15.1:0/3005116170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda5006b4d0 0x7fda5006d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fda68005880 tx=0x7fda68009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.800+0000 7fda5effd640 1 -- 172.21.15.1:0/3005116170 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda60059290 con 0x7fda700be310 2024-03-06T07:23:32.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.980+0000 7fda74e2a640 1 -- 172.21.15.1:0/3005116170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fda700a9670 con 0x7fda700be310 2024-03-06T07:23:32.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.983+0000 7fda5effd640 1 -- 172.21.15.1:0/3005116170 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7fda60058b90 con 0x7fda700be310 2024-03-06T07:23:32.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.985+0000 7fda74e2a640 1 -- 172.21.15.1:0/3005116170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda5006b4d0 msgr2=0x7fda5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.985+0000 7fda74e2a640 1 --2- 172.21.15.1:0/3005116170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda5006b4d0 0x7fda5006d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fda68005880 tx=0x7fda68009040 comp rx=0 tx=0).stop 2024-03-06T07:23:32.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.985+0000 7fda74e2a640 1 -- 172.21.15.1:0/3005116170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda700be310 msgr2=0x7fda70142f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.985+0000 7fda74e2a640 1 --2- 172.21.15.1:0/3005116170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda700be310 0x7fda70142f70 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fda6000e7f0 tx=0x7fda6000ebf0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.985+0000 7fda74e2a640 1 -- 172.21.15.1:0/3005116170 shutdown_connections 2024-03-06T07:23:32.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.985+0000 7fda74e2a640 1 --2- 172.21.15.1:0/3005116170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda5006b4d0 0x7fda5006d990 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.985+0000 7fda74e2a640 1 --2- 172.21.15.1:0/3005116170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda700be310 0x7fda70142f70 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.985+0000 7fda74e2a640 1 --2- 172.21.15.1:0/3005116170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda700c4dd0 0x7fda70142a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.985+0000 7fda74e2a640 1 --2- 172.21.15.1:0/3005116170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda700a7ec0 0x7fda700bddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.985+0000 7fda74e2a640 1 -- 172.21.15.1:0/3005116170 >> 172.21.15.1:0/3005116170 conn(0x7fda7006b410 msgr2=0x7fda700a9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.985+0000 7fda74e2a640 1 -- 172.21.15.1:0/3005116170 shutdown_connections 2024-03-06T07:23:32.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:32.985+0000 7fda74e2a640 1 -- 172.21.15.1:0/3005116170 wait complete. 2024-03-06T07:23:32.998 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-03-06T07:23:33.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.102+0000 7f6b1f6eb640 1 Processor -- start 2024-03-06T07:23:33.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.102+0000 7f6b1f6eb640 1 -- start start 2024-03-06T07:23:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.103+0000 7f6b1f6eb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6b18152690 0x7f6b18155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.103+0000 7f6b1f6eb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6b180c6730 0x7f6b180c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.103+0000 7f6b1f6eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b180c5d10 0x7f6b18156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.103+0000 7f6b1f6eb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b181561e0 con 0x7f6b18152690 2024-03-06T07:23:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.103+0000 7f6b1f6eb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b180c3300 con 0x7f6b180c6730 2024-03-06T07:23:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.103+0000 7f6b1f6eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b180c35a0 con 0x7f6b180c5d10 2024-03-06T07:23:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.103+0000 7f6b1d487640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6b18152690 0x7f6b18155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.103+0000 7f6b1d487640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6b18152690 0x7f6b18155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45476/0 (socket says 172.21.15.1:45476) 2024-03-06T07:23:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.103+0000 7f6b1d487640 1 -- 172.21.15.1:0/912464103 learned_addr learned my addr 172.21.15.1:0/912464103 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.103+0000 7f6b1dc88640 1 --2- 172.21.15.1:0/912464103 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b180c5d10 0x7f6b18156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.103+0000 7f6b1d487640 1 -- 172.21.15.1:0/912464103 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b180c5d10 msgr2=0x7f6b18156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.103+0000 7f6b1d487640 1 --2- 172.21.15.1:0/912464103 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b180c5d10 0x7f6b18156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.103+0000 7f6b1d487640 1 -- 172.21.15.1:0/912464103 >> v2:172.21.15.98:3300/0 conn(0x7f6b180c6730 msgr2=0x7f6b180c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.103+0000 7f6b1cc86640 1 --2- 172.21.15.1:0/912464103 >> v2:172.21.15.98:3300/0 conn(0x7f6b180c6730 0x7f6b180c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.103+0000 7f6b1d487640 1 --2- 172.21.15.1:0/912464103 >> v2:172.21.15.98:3300/0 conn(0x7f6b180c6730 0x7f6b180c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.103+0000 7f6b1d487640 1 -- 172.21.15.1:0/912464103 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b180c3880 con 0x7f6b18152690 2024-03-06T07:23:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.103+0000 7f6b1dc88640 1 --2- 172.21.15.1:0/912464103 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b180c5d10 0x7f6b18156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.104+0000 7f6b1d487640 1 --2- 172.21.15.1:0/912464103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b18152690 0x7f6b18155ca0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6b180c98d0 tx=0x7f6b0802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cdefc60339c1dba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.104+0000 7f6b0ffff640 1 -- 172.21.15.1:0/912464103 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0802e070 con 0x7f6b18152690 2024-03-06T07:23:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.104+0000 7f6b0ffff640 1 -- 172.21.15.1:0/912464103 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b08037070 con 0x7f6b18152690 2024-03-06T07:23:33.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.104+0000 7f6b0ffff640 1 -- 172.21.15.1:0/912464103 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b08031a00 con 0x7f6b18152690 2024-03-06T07:23:33.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.105+0000 7f6b1f6eb640 1 -- 172.21.15.1:0/912464103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b18152690 msgr2=0x7f6b18155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.105+0000 7f6b1f6eb640 1 --2- 172.21.15.1:0/912464103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b18152690 0x7f6b18155ca0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6b180c98d0 tx=0x7f6b0802b880 comp rx=0 tx=0).stop 2024-03-06T07:23:33.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.105+0000 7f6b1f6eb640 1 -- 172.21.15.1:0/912464103 shutdown_connections 2024-03-06T07:23:33.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.105+0000 7f6b1f6eb640 1 --2- 172.21.15.1:0/912464103 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b180c5d10 0x7f6b18156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.105+0000 7f6b1f6eb640 1 --2- 172.21.15.1:0/912464103 >> v2:172.21.15.98:3300/0 conn(0x7f6b180c6730 0x7f6b180c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.105+0000 7f6b1f6eb640 1 --2- 172.21.15.1:0/912464103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b18152690 0x7f6b18155ca0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.105+0000 7f6b1f6eb640 1 -- 172.21.15.1:0/912464103 >> 172.21.15.1:0/912464103 conn(0x7f6b1806b410 msgr2=0x7f6b1806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.105+0000 7f6b1f6eb640 1 -- 172.21.15.1:0/912464103 shutdown_connections 2024-03-06T07:23:33.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.105+0000 7f6b1f6eb640 1 -- 172.21.15.1:0/912464103 wait complete. 2024-03-06T07:23:33.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.106+0000 7f6b1f6eb640 1 Processor -- start 2024-03-06T07:23:33.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.106+0000 7f6b1f6eb640 1 -- start start 2024-03-06T07:23:33.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.107+0000 7f6b1f6eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b180c5d10 0x7f6b18070a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.107+0000 7f6b1f6eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b180c6730 0x7f6b1814bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.107+0000 7f6b1f6eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b18152690 0x7f6b1814cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.107+0000 7f6b1f6eb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b18152a70 con 0x7f6b18152690 2024-03-06T07:23:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.107+0000 7f6b1f6eb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b181567b0 con 0x7f6b180c5d10 2024-03-06T07:23:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.107+0000 7f6b1f6eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b18152bf0 con 0x7f6b180c6730 2024-03-06T07:23:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.107+0000 7f6b1dc88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b18152690 0x7f6b1814cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.107+0000 7f6b1dc88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b18152690 0x7f6b1814cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45480/0 (socket says 172.21.15.1:45480) 2024-03-06T07:23:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.107+0000 7f6b1dc88640 1 -- 172.21.15.1:0/487604297 learned_addr learned my addr 172.21.15.1:0/487604297 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.107+0000 7f6b1cc86640 1 --2- 172.21.15.1:0/487604297 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b180c6730 0x7f6b1814bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.107+0000 7f6b1d487640 1 --2- 172.21.15.1:0/487604297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b180c5d10 0x7f6b18070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.107+0000 7f6b1dc88640 1 -- 172.21.15.1:0/487604297 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b180c6730 msgr2=0x7f6b1814bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.107+0000 7f6b1dc88640 1 --2- 172.21.15.1:0/487604297 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b180c6730 0x7f6b1814bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.107+0000 7f6b1dc88640 1 -- 172.21.15.1:0/487604297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b180c5d10 msgr2=0x7f6b18070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.107+0000 7f6b1dc88640 1 --2- 172.21.15.1:0/487604297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b180c5d10 0x7f6b18070a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.107+0000 7f6b1dc88640 1 -- 172.21.15.1:0/487604297 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b180c89d0 con 0x7f6b18152690 2024-03-06T07:23:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.107+0000 7f6b1dc88640 1 --2- 172.21.15.1:0/487604297 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b18152690 0x7f6b1814cec0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000cd10 tx=0x7f6b1000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.108+0000 7f6b0dffb640 1 -- 172.21.15.1:0/487604297 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1000f680 con 0x7f6b18152690 2024-03-06T07:23:33.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.108+0000 7f6b0dffb640 1 -- 172.21.15.1:0/487604297 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1000f840 con 0x7f6b18152690 2024-03-06T07:23:33.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.108+0000 7f6b1f6eb640 1 -- 172.21.15.1:0/487604297 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b180ab620 con 0x7f6b18152690 2024-03-06T07:23:33.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.108+0000 7f6b0dffb640 1 -- 172.21.15.1:0/487604297 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1000fc30 con 0x7f6b18152690 2024-03-06T07:23:33.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.108+0000 7f6b1f6eb640 1 -- 172.21.15.1:0/487604297 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b1814c0a0 con 0x7f6b18152690 2024-03-06T07:23:33.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.109+0000 7f6b1f6eb640 1 -- 172.21.15.1:0/487604297 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b18067fd0 con 0x7f6b18152690 2024-03-06T07:23:33.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.109+0000 7f6b0dffb640 1 -- 172.21.15.1:0/487604297 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b100213f0 con 0x7f6b18152690 2024-03-06T07:23:33.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.109+0000 7f6b0dffb640 1 --2- 172.21.15.1:0/487604297 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6afc06b4d0 0x7f6afc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.110+0000 7f6b0dffb640 1 -- 172.21.15.1:0/487604297 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f6b1008ace0 con 0x7f6b18152690 2024-03-06T07:23:33.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.110+0000 7f6b1d487640 1 --2- 172.21.15.1:0/487604297 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6afc06b4d0 0x7f6afc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.110+0000 7f6b1d487640 1 --2- 172.21.15.1:0/487604297 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6afc06b4d0 0x7f6afc06d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6b08036000 tx=0x7f6b0803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.116+0000 7f6b0dffb640 1 -- 172.21.15.1:0/487604297 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b10014030 con 0x7f6b18152690 2024-03-06T07:23:33.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.301+0000 7f6b1f6eb640 1 -- 172.21.15.1:0/487604297 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7f6b180c2c40 con 0x7f6b18152690 2024-03-06T07:23:33.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.304+0000 7f6b0dffb640 1 -- 172.21.15.1:0/487604297 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v28) v1 ==== 96+0+0 (secure 0 0 0) 0x7f6b10059610 con 0x7f6b18152690 2024-03-06T07:23:33.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.306+0000 7f6b1f6eb640 1 -- 172.21.15.1:0/487604297 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6afc06b4d0 msgr2=0x7f6afc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.306+0000 7f6b1f6eb640 1 --2- 172.21.15.1:0/487604297 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6afc06b4d0 0x7f6afc06d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6b08036000 tx=0x7f6b0803d040 comp rx=0 tx=0).stop 2024-03-06T07:23:33.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.306+0000 7f6b1f6eb640 1 -- 172.21.15.1:0/487604297 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b18152690 msgr2=0x7f6b1814cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.306+0000 7f6b1f6eb640 1 --2- 172.21.15.1:0/487604297 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b18152690 0x7f6b1814cec0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000cd10 tx=0x7f6b1000a650 comp rx=0 tx=0).stop 2024-03-06T07:23:33.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.307+0000 7f6b1f6eb640 1 -- 172.21.15.1:0/487604297 shutdown_connections 2024-03-06T07:23:33.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.307+0000 7f6b1f6eb640 1 --2- 172.21.15.1:0/487604297 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6afc06b4d0 0x7f6afc06d990 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.307+0000 7f6b1f6eb640 1 --2- 172.21.15.1:0/487604297 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b18152690 0x7f6b1814cec0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.307+0000 7f6b1f6eb640 1 --2- 172.21.15.1:0/487604297 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b180c6730 0x7f6b1814bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.307+0000 7f6b1f6eb640 1 --2- 172.21.15.1:0/487604297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b180c5d10 0x7f6b18070a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.307+0000 7f6b1f6eb640 1 -- 172.21.15.1:0/487604297 >> 172.21.15.1:0/487604297 conn(0x7f6b1806b410 msgr2=0x7f6b1806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.307+0000 7f6b1f6eb640 1 -- 172.21.15.1:0/487604297 shutdown_connections 2024-03-06T07:23:33.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.307+0000 7f6b1f6eb640 1 -- 172.21.15.1:0/487604297 wait complete. 2024-03-06T07:23:33.319 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.a mds_join_fs 2024-03-06T07:23:33.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.425+0000 7f7f82215640 1 Processor -- start 2024-03-06T07:23:33.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.425+0000 7f7f82215640 1 -- start start 2024-03-06T07:23:33.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.425+0000 7f7f82215640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f7c14cdc0 0x7f7f7c152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.426+0000 7f7f82215640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7f7c06b510 0x7f7f7c06b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.426+0000 7f7f82215640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f7c1569e0 con 0x7f7f7c156500 2024-03-06T07:23:33.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.426+0000 7f7f82215640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f7c0c2dd0 con 0x7f7f7c06b510 2024-03-06T07:23:33.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.426+0000 7f7f82215640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c156cc0 con 0x7f7f7c14cdc0 2024-03-06T07:23:33.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.426+0000 7f7f7bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7f7c06b510 0x7f7f7c06b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.426+0000 7f7f7bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7f7c06b510 0x7f7f7c06b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45482/0 (socket says 172.21.15.1:45482) 2024-03-06T07:23:33.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.426+0000 7f7f7bfff640 1 -- 172.21.15.1:0/1978070219 learned_addr learned my addr 172.21.15.1:0/1978070219 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:33.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.426+0000 7f7f73fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7f7c156500 0x7f7f7c0c2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50258/0 (socket says 172.21.15.1:50258) 2024-03-06T07:23:33.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.426+0000 7f7f7b7fe640 1 --2- 172.21.15.1:0/1978070219 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f7c14cdc0 0x7f7f7c152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.426+0000 7f7f7bfff640 1 -- 172.21.15.1:0/1978070219 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f7c14cdc0 msgr2=0x7f7f7c152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.426+0000 7f7f7bfff640 1 --2- 172.21.15.1:0/1978070219 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f7c14cdc0 0x7f7f7c152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.426+0000 7f7f7bfff640 1 -- 172.21.15.1:0/1978070219 >> v1:172.21.15.1:6789/0 conn(0x7f7f7c156500 legacy=0x7f7f7c0c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.426+0000 7f7f7bfff640 1 -- 172.21.15.1:0/1978070219 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c156fa0 con 0x7f7f7c06b510 2024-03-06T07:23:33.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.426+0000 7f7f7affd640 1 -- 172.21.15.1:0/1978070219 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482892287 0 0) 0x7f7f7c1569e0 con 0x7f7f7c156500 2024-03-06T07:23:33.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.427+0000 7f7f7bfff640 1 --2- 172.21.15.1:0/1978070219 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f7c06b510 0x7f7f7c06b980 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f7f74009e50 tx=0x7f7f7402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb8995613d39324e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.427+0000 7f7f7affd640 1 -- 172.21.15.1:0/1978070219 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7402cb00 con 0x7f7f7c06b510 2024-03-06T07:23:33.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.427+0000 7f7f7affd640 1 -- 172.21.15.1:0/1978070219 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7402cca0 con 0x7f7f7c06b510 2024-03-06T07:23:33.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.427+0000 7f7f7affd640 1 -- 172.21.15.1:0/1978070219 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f74034610 con 0x7f7f7c06b510 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.427+0000 7f7f82215640 1 -- 172.21.15.1:0/1978070219 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f7c06b510 msgr2=0x7f7f7c06b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.427+0000 7f7f82215640 1 --2- 172.21.15.1:0/1978070219 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f7c06b510 0x7f7f7c06b980 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f7f74009e50 tx=0x7f7f7402bf60 comp rx=0 tx=0).stop 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.428+0000 7f7f82215640 1 -- 172.21.15.1:0/1978070219 shutdown_connections 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.428+0000 7f7f82215640 1 --2- 172.21.15.1:0/1978070219 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f7c06b510 0x7f7f7c06b980 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.428+0000 7f7f82215640 1 --2- 172.21.15.1:0/1978070219 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f7c14cdc0 0x7f7f7c152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.428+0000 7f7f82215640 1 -- 172.21.15.1:0/1978070219 >> 172.21.15.1:0/1978070219 conn(0x7f7f7c0a9700 msgr2=0x7f7f7c0a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.428+0000 7f7f82215640 1 -- 172.21.15.1:0/1978070219 shutdown_connections 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.428+0000 7f7f82215640 1 -- 172.21.15.1:0/1978070219 wait complete. 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.429+0000 7f7f82215640 1 Processor -- start 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.429+0000 7f7f82215640 1 -- start start 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.429+0000 7f7f82215640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f7c06b510 0x7f7f7c1e96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.429+0000 7f7f82215640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f7c14cdc0 0x7f7f7c1520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.429+0000 7f7f82215640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f7c156500 0x7f7f7c14da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.429+0000 7f7f82215640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c0c5a70 con 0x7f7f7c14cdc0 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.429+0000 7f7f82215640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c06bec0 con 0x7f7f7c06b510 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.429+0000 7f7f82215640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c0c65e0 con 0x7f7f7c156500 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.429+0000 7f7f73fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f7c14cdc0 0x7f7f7c1520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.429+0000 7f7f73fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f7c14cdc0 0x7f7f7c1520b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45494/0 (socket says 172.21.15.1:45494) 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.429+0000 7f7f73fff640 1 -- 172.21.15.1:0/1804393914 learned_addr learned my addr 172.21.15.1:0/1804393914 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.430+0000 7f7f7bfff640 1 --2- 172.21.15.1:0/1804393914 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f7c156500 0x7f7f7c14da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.430+0000 7f7f7b7fe640 1 --2- 172.21.15.1:0/1804393914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f7c06b510 0x7f7f7c1e96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.430+0000 7f7f73fff640 1 -- 172.21.15.1:0/1804393914 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f7c156500 msgr2=0x7f7f7c14da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.430+0000 7f7f73fff640 1 --2- 172.21.15.1:0/1804393914 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f7c156500 0x7f7f7c14da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.430+0000 7f7f73fff640 1 -- 172.21.15.1:0/1804393914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f7c06b510 msgr2=0x7f7f7c1e96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.430+0000 7f7f73fff640 1 --2- 172.21.15.1:0/1804393914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f7c06b510 0x7f7f7c1e96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.430+0000 7f7f73fff640 1 -- 172.21.15.1:0/1804393914 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c0ad730 con 0x7f7f7c14cdc0 2024-03-06T07:23:33.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.430+0000 7f7f7bfff640 1 --2- 172.21.15.1:0/1804393914 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f7c156500 0x7f7f7c14da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:33.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.430+0000 7f7f7b7fe640 1 --2- 172.21.15.1:0/1804393914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f7c06b510 0x7f7f7c1e96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:33.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.430+0000 7f7f73fff640 1 --2- 172.21.15.1:0/1804393914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f7c14cdc0 0x7f7f7c1520b0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f7f6400c4d0 tx=0x7f7f6400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.430+0000 7f7f78ff9640 1 -- 172.21.15.1:0/1804393914 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64018070 con 0x7f7f7c14cdc0 2024-03-06T07:23:33.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.430+0000 7f7f78ff9640 1 -- 172.21.15.1:0/1804393914 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f64007500 con 0x7f7f7c14cdc0 2024-03-06T07:23:33.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.430+0000 7f7f78ff9640 1 -- 172.21.15.1:0/1804393914 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64004a20 con 0x7f7f7c14cdc0 2024-03-06T07:23:33.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.430+0000 7f7f82215640 1 -- 172.21.15.1:0/1804393914 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f7c0ad9a0 con 0x7f7f7c14cdc0 2024-03-06T07:23:33.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.430+0000 7f7f82215640 1 -- 172.21.15.1:0/1804393914 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f7c14d240 con 0x7f7f7c14cdc0 2024-03-06T07:23:33.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.431+0000 7f7f82215640 1 -- 172.21.15.1:0/1804393914 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f7c067fd0 con 0x7f7f7c14cdc0 2024-03-06T07:23:33.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.432+0000 7f7f78ff9640 1 -- 172.21.15.1:0/1804393914 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f6401b070 con 0x7f7f7c14cdc0 2024-03-06T07:23:33.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.432+0000 7f7f78ff9640 1 --2- 172.21.15.1:0/1804393914 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7f5c06b4d0 0x7f7f5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.432+0000 7f7f78ff9640 1 -- 172.21.15.1:0/1804393914 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f7f64089410 con 0x7f7f7c14cdc0 2024-03-06T07:23:33.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.432+0000 7f7f7b7fe640 1 --2- 172.21.15.1:0/1804393914 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7f5c06b4d0 0x7f7f5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.433+0000 7f7f7b7fe640 1 --2- 172.21.15.1:0/1804393914 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7f5c06b4d0 0x7f7f5c06d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c006fd0 tx=0x7f7f6c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.436+0000 7f7f78ff9640 1 -- 172.21.15.1:0/1804393914 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f64058390 con 0x7f7f7c14cdc0 2024-03-06T07:23:33.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.617+0000 7f7f82215640 1 -- 172.21.15.1:0/1804393914 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"} v 0) v1 -- 0x7f7f7c0c5850 con 0x7f7f7c14cdc0 2024-03-06T07:23:33.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.620+0000 7f7f78ff9640 1 -- 172.21.15.1:0/1804393914 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"}]=0 v29) v1 ==== 96+0+0 (secure 0 0 0) 0x7f7f64057c90 con 0x7f7f7c14cdc0 2024-03-06T07:23:33.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.622+0000 7f7f82215640 1 -- 172.21.15.1:0/1804393914 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7f5c06b4d0 msgr2=0x7f7f5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.622+0000 7f7f82215640 1 --2- 172.21.15.1:0/1804393914 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7f5c06b4d0 0x7f7f5c06d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c006fd0 tx=0x7f7f6c008040 comp rx=0 tx=0).stop 2024-03-06T07:23:33.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.622+0000 7f7f82215640 1 -- 172.21.15.1:0/1804393914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f7c14cdc0 msgr2=0x7f7f7c1520b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.622+0000 7f7f82215640 1 --2- 172.21.15.1:0/1804393914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f7c14cdc0 0x7f7f7c1520b0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f7f6400c4d0 tx=0x7f7f6400c990 comp rx=0 tx=0).stop 2024-03-06T07:23:33.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.622+0000 7f7f82215640 1 -- 172.21.15.1:0/1804393914 shutdown_connections 2024-03-06T07:23:33.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.622+0000 7f7f82215640 1 --2- 172.21.15.1:0/1804393914 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7f5c06b4d0 0x7f7f5c06d990 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.622+0000 7f7f82215640 1 --2- 172.21.15.1:0/1804393914 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7f7c156500 0x7f7f7c14da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.622+0000 7f7f82215640 1 --2- 172.21.15.1:0/1804393914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f7c14cdc0 0x7f7f7c1520b0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.622+0000 7f7f82215640 1 --2- 172.21.15.1:0/1804393914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f7c06b510 0x7f7f7c1e96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.622+0000 7f7f82215640 1 -- 172.21.15.1:0/1804393914 >> 172.21.15.1:0/1804393914 conn(0x7f7f7c0a9700 msgr2=0x7f7f7c1541d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.622+0000 7f7f82215640 1 -- 172.21.15.1:0/1804393914 shutdown_connections 2024-03-06T07:23:33.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.622+0000 7f7f82215640 1 -- 172.21.15.1:0/1804393914 wait complete. 2024-03-06T07:23:33.635 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-03-06T07:23:33.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.736+0000 7ff630ba2640 1 Processor -- start 2024-03-06T07:23:33.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.737+0000 7ff630ba2640 1 -- start start 2024-03-06T07:23:33.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.737+0000 7ff630ba2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff62c0c6ee0 0x7ff62c0c5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.737+0000 7ff630ba2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff62c155e90 con 0x7ff62c152840 2024-03-06T07:23:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.737+0000 7ff630ba2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff62c0c14d0 con 0x7ff62c0bf900 2024-03-06T07:23:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.737+0000 7ff630ba2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff62c0c1b20 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.737+0000 7ff62affd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff62c0c6ee0 0x7ff62c0c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.737+0000 7ff62b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff62c152840 0x7ff62c0ca890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50260/0 (socket says 172.21.15.1:50260) 2024-03-06T07:23:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.737+0000 7ff62b7fe640 1 -- 172.21.15.1:0/3435080707 learned_addr learned my addr 172.21.15.1:0/3435080707 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.738+0000 7ff62affd640 1 -- 172.21.15.1:0/3435080707 >> v1:172.21.15.98:6789/0 conn(0x7ff62c0bf900 legacy=0x7ff62c0c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.738+0000 7ff62affd640 1 -- 172.21.15.1:0/3435080707 >> v1:172.21.15.1:6789/0 conn(0x7ff62c152840 legacy=0x7ff62c0ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.738+0000 7ff62affd640 1 -- 172.21.15.1:0/3435080707 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62c0c17b0 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.738+0000 7ff62a7fc640 1 -- 172.21.15.1:0/3435080707 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665677581 0 0) 0x7ff62c155e90 con 0x7ff62c152840 2024-03-06T07:23:33.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.738+0000 7ff62affd640 1 --2- 172.21.15.1:0/3435080707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff62c0c6ee0 0x7ff62c0c5490 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7ff6200097a0 tx=0x7ff62002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=944f84250f9d855e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.739+0000 7ff62a7fc640 1 -- 172.21.15.1:0/3435080707 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff620038070 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.739+0000 7ff62a7fc640 1 -- 172.21.15.1:0/3435080707 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff620031700 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.739+0000 7ff62a7fc640 1 -- 172.21.15.1:0/3435080707 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff620031ad0 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.739+0000 7ff630ba2640 1 -- 172.21.15.1:0/3435080707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff62c0c6ee0 msgr2=0x7ff62c0c5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.739+0000 7ff630ba2640 1 --2- 172.21.15.1:0/3435080707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff62c0c6ee0 0x7ff62c0c5490 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7ff6200097a0 tx=0x7ff62002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.739+0000 7ff630ba2640 1 -- 172.21.15.1:0/3435080707 shutdown_connections 2024-03-06T07:23:33.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.739+0000 7ff630ba2640 1 --2- 172.21.15.1:0/3435080707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff62c0c6ee0 0x7ff62c0c5490 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.739+0000 7ff630ba2640 1 -- 172.21.15.1:0/3435080707 >> 172.21.15.1:0/3435080707 conn(0x7ff62c067520 msgr2=0x7ff62c06d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.739+0000 7ff630ba2640 1 -- 172.21.15.1:0/3435080707 shutdown_connections 2024-03-06T07:23:33.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.739+0000 7ff630ba2640 1 -- 172.21.15.1:0/3435080707 wait complete. 2024-03-06T07:23:33.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.740+0000 7ff630ba2640 1 Processor -- start 2024-03-06T07:23:33.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.740+0000 7ff630ba2640 1 -- start start 2024-03-06T07:23:33.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.741+0000 7ff630ba2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff62c0bf900 0x7ff62c151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.741+0000 7ff630ba2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff62c0c6ee0 0x7ff62c156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.741+0000 7ff630ba2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff62c152840 0x7ff62c1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.741+0000 7ff630ba2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c0cafa0 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.741+0000 7ff630ba2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c155b60 con 0x7ff62c152840 2024-03-06T07:23:33.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.741+0000 7ff630ba2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c0cb120 con 0x7ff62c0bf900 2024-03-06T07:23:33.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.741+0000 7ff62affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff62c0c6ee0 0x7ff62c156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.741+0000 7ff62affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff62c0c6ee0 0x7ff62c156b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45514/0 (socket says 172.21.15.1:45514) 2024-03-06T07:23:33.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.741+0000 7ff62affd640 1 -- 172.21.15.1:0/143922181 learned_addr learned my addr 172.21.15.1:0/143922181 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:33.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.741+0000 7ff62b7fe640 1 --2- 172.21.15.1:0/143922181 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff62c0bf900 0x7ff62c151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.741+0000 7ff62bfff640 1 --2- 172.21.15.1:0/143922181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff62c152840 0x7ff62c1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.741+0000 7ff62affd640 1 -- 172.21.15.1:0/143922181 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff62c0bf900 msgr2=0x7ff62c151320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.741+0000 7ff62affd640 1 --2- 172.21.15.1:0/143922181 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff62c0bf900 0x7ff62c151320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.741+0000 7ff62affd640 1 -- 172.21.15.1:0/143922181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff62c152840 msgr2=0x7ff62c1570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.741+0000 7ff62affd640 1 --2- 172.21.15.1:0/143922181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff62c152840 0x7ff62c1570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.741+0000 7ff62affd640 1 -- 172.21.15.1:0/143922181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62c0c9680 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.742+0000 7ff62affd640 1 --2- 172.21.15.1:0/143922181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff62c0c6ee0 0x7ff62c156b60 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7ff6200097a0 tx=0x7ff6200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.742+0000 7ff613fff640 1 -- 172.21.15.1:0/143922181 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff620038040 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.742+0000 7ff613fff640 1 -- 172.21.15.1:0/143922181 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff620033a30 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.742+0000 7ff630ba2640 1 -- 172.21.15.1:0/143922181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff62c0ca040 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.742+0000 7ff613fff640 1 -- 172.21.15.1:0/143922181 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff620031ed0 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.742+0000 7ff630ba2640 1 -- 172.21.15.1:0/143922181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62c0ca600 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.743+0000 7ff630ba2640 1 -- 172.21.15.1:0/143922181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff62c0c69b0 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.743+0000 7ff613fff640 1 -- 172.21.15.1:0/143922181 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff620004030 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.744+0000 7ff613fff640 1 --2- 172.21.15.1:0/143922181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff61806b4d0 0x7ff61806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.744+0000 7ff613fff640 1 -- 172.21.15.1:0/143922181 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7ff6200abf90 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.744+0000 7ff62b7fe640 1 --2- 172.21.15.1:0/143922181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff61806b4d0 0x7ff61806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.744+0000 7ff62b7fe640 1 --2- 172.21.15.1:0/143922181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff61806b4d0 0x7ff61806d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff61c000f60 tx=0x7ff61c007c00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.748+0000 7ff613fff640 1 -- 172.21.15.1:0/143922181 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff620043020 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.927+0000 7ff630ba2640 1 -- 172.21.15.1:0/143922181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7ff62c0c3e70 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.930+0000 7ff613fff640 1 -- 172.21.15.1:0/143922181 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v30) v1 ==== 96+0+0 (secure 0 0 0) 0x7ff620030070 con 0x7ff62c0c6ee0 2024-03-06T07:23:33.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.932+0000 7ff630ba2640 1 -- 172.21.15.1:0/143922181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff61806b4d0 msgr2=0x7ff61806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.932+0000 7ff630ba2640 1 --2- 172.21.15.1:0/143922181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff61806b4d0 0x7ff61806d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff61c000f60 tx=0x7ff61c007c00 comp rx=0 tx=0).stop 2024-03-06T07:23:33.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.933+0000 7ff630ba2640 1 -- 172.21.15.1:0/143922181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff62c0c6ee0 msgr2=0x7ff62c156b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.933+0000 7ff630ba2640 1 --2- 172.21.15.1:0/143922181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff62c0c6ee0 0x7ff62c156b60 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7ff6200097a0 tx=0x7ff6200046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.933+0000 7ff630ba2640 1 -- 172.21.15.1:0/143922181 shutdown_connections 2024-03-06T07:23:33.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.933+0000 7ff630ba2640 1 --2- 172.21.15.1:0/143922181 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff61806b4d0 0x7ff61806d990 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.933+0000 7ff630ba2640 1 --2- 172.21.15.1:0/143922181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff62c152840 0x7ff62c1570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.933+0000 7ff630ba2640 1 --2- 172.21.15.1:0/143922181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff62c0c6ee0 0x7ff62c156b60 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.933+0000 7ff630ba2640 1 --2- 172.21.15.1:0/143922181 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff62c0bf900 0x7ff62c151320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.933+0000 7ff630ba2640 1 -- 172.21.15.1:0/143922181 >> 172.21.15.1:0/143922181 conn(0x7ff62c067520 msgr2=0x7ff62c06d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.933+0000 7ff630ba2640 1 -- 172.21.15.1:0/143922181 shutdown_connections 2024-03-06T07:23:33.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:33.933+0000 7ff630ba2640 1 -- 172.21.15.1:0/143922181 wait complete. 2024-03-06T07:23:33.946 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused' 2024-03-06T07:23:34.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.045+0000 7feeb3f3e640 1 Processor -- start 2024-03-06T07:23:34.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.045+0000 7feeb3f3e640 1 -- start start 2024-03-06T07:23:34.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.046+0000 7feeb3f3e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feeac152690 0x7feeac155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.046+0000 7feeb3f3e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feeac0c6730 0x7feeac0c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.046+0000 7feeb3f3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac0c5d10 0x7feeac156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.046+0000 7feeb3f3e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeac1561e0 con 0x7feeac152690 2024-03-06T07:23:34.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.046+0000 7feeb3f3e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeac0c3300 con 0x7feeac0c6730 2024-03-06T07:23:34.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.046+0000 7feeb3f3e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeac0c35a0 con 0x7feeac0c5d10 2024-03-06T07:23:34.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.046+0000 7feeb1cda640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feeac152690 0x7feeac155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.046+0000 7feeb1cda640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feeac152690 0x7feeac155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45528/0 (socket says 172.21.15.1:45528) 2024-03-06T07:23:34.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.046+0000 7feeb1cda640 1 -- 172.21.15.1:0/1519991878 learned_addr learned my addr 172.21.15.1:0/1519991878 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:34.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.046+0000 7feeb1cda640 1 -- 172.21.15.1:0/1519991878 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac0c5d10 msgr2=0x7feeac156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.046+0000 7feeb1cda640 1 --2- 172.21.15.1:0/1519991878 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac0c5d10 0x7feeac156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.046+0000 7feeb1cda640 1 -- 172.21.15.1:0/1519991878 >> v2:172.21.15.98:3300/0 conn(0x7feeac0c6730 msgr2=0x7feeac0c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.046+0000 7feeb1cda640 1 --2- 172.21.15.1:0/1519991878 >> v2:172.21.15.98:3300/0 conn(0x7feeac0c6730 0x7feeac0c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.046+0000 7feeb1cda640 1 -- 172.21.15.1:0/1519991878 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac0c3880 con 0x7feeac152690 2024-03-06T07:23:34.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.047+0000 7feeb1cda640 1 --2- 172.21.15.1:0/1519991878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac152690 0x7feeac155ca0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7feeac0c98d0 tx=0x7feea802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8435265b6a3d447 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.047+0000 7feeb0cd8640 1 -- 172.21.15.1:0/1519991878 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea802e070 con 0x7feeac152690 2024-03-06T07:23:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.047+0000 7feeb0cd8640 1 -- 172.21.15.1:0/1519991878 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea8037070 con 0x7feeac152690 2024-03-06T07:23:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.047+0000 7feeb0cd8640 1 -- 172.21.15.1:0/1519991878 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea8031a00 con 0x7feeac152690 2024-03-06T07:23:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.047+0000 7feeb3f3e640 1 -- 172.21.15.1:0/1519991878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac152690 msgr2=0x7feeac155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.047+0000 7feeb3f3e640 1 --2- 172.21.15.1:0/1519991878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac152690 0x7feeac155ca0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7feeac0c98d0 tx=0x7feea802b880 comp rx=0 tx=0).stop 2024-03-06T07:23:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.048+0000 7feeb3f3e640 1 -- 172.21.15.1:0/1519991878 shutdown_connections 2024-03-06T07:23:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.048+0000 7feeb3f3e640 1 --2- 172.21.15.1:0/1519991878 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac0c5d10 0x7feeac156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.048+0000 7feeb3f3e640 1 --2- 172.21.15.1:0/1519991878 >> v2:172.21.15.98:3300/0 conn(0x7feeac0c6730 0x7feeac0c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.048+0000 7feeb3f3e640 1 --2- 172.21.15.1:0/1519991878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac152690 0x7feeac155ca0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.048+0000 7feeb3f3e640 1 -- 172.21.15.1:0/1519991878 >> 172.21.15.1:0/1519991878 conn(0x7feeac06b410 msgr2=0x7feeac06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.048+0000 7feeb3f3e640 1 -- 172.21.15.1:0/1519991878 shutdown_connections 2024-03-06T07:23:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.048+0000 7feeb3f3e640 1 -- 172.21.15.1:0/1519991878 wait complete. 2024-03-06T07:23:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.049+0000 7feeb3f3e640 1 Processor -- start 2024-03-06T07:23:34.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.049+0000 7feeb3f3e640 1 -- start start 2024-03-06T07:23:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.049+0000 7feeb3f3e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feeac0c5d10 0x7feeac06d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.049+0000 7feeb3f3e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0c6730 0x7feeac1e7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.049+0000 7feeb3f3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac152690 0x7feeac1ee000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.049+0000 7feeb3f3e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac152a70 con 0x7feeac0c6730 2024-03-06T07:23:34.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.049+0000 7feeb3f3e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac1567b0 con 0x7feeac0c5d10 2024-03-06T07:23:34.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.049+0000 7feeb3f3e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeac152bf0 con 0x7feeac152690 2024-03-06T07:23:34.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.049+0000 7feeb14d9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0c6730 0x7feeac1e7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.049+0000 7feeb14d9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0c6730 0x7feeac1e7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45540/0 (socket says 172.21.15.1:45540) 2024-03-06T07:23:34.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.049+0000 7feeb14d9640 1 -- 172.21.15.1:0/330648951 learned_addr learned my addr 172.21.15.1:0/330648951 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:34.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.050+0000 7feeb24db640 1 --2- 172.21.15.1:0/330648951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac152690 0x7feeac1ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.050+0000 7feeb1cda640 1 --2- 172.21.15.1:0/330648951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feeac0c5d10 0x7feeac06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.050+0000 7feeb14d9640 1 -- 172.21.15.1:0/330648951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac152690 msgr2=0x7feeac1ee000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.050+0000 7feeb14d9640 1 --2- 172.21.15.1:0/330648951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac152690 0x7feeac1ee000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.050+0000 7feeb14d9640 1 -- 172.21.15.1:0/330648951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feeac0c5d10 msgr2=0x7feeac06d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.050+0000 7feeb14d9640 1 --2- 172.21.15.1:0/330648951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feeac0c5d10 0x7feeac06d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.050+0000 7feeb14d9640 1 -- 172.21.15.1:0/330648951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac1eb460 con 0x7feeac0c6730 2024-03-06T07:23:34.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.050+0000 7feeb24db640 1 --2- 172.21.15.1:0/330648951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac152690 0x7feeac1ee000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:34.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.050+0000 7feeb14d9640 1 --2- 172.21.15.1:0/330648951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0c6730 0x7feeac1e7b40 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fee9c00b440 tx=0x7fee9c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.050+0000 7feea27fc640 1 -- 172.21.15.1:0/330648951 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9c015020 con 0x7feeac0c6730 2024-03-06T07:23:34.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.050+0000 7feea27fc640 1 -- 172.21.15.1:0/330648951 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9c011890 con 0x7feeac0c6730 2024-03-06T07:23:34.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.050+0000 7feeb3f3e640 1 -- 172.21.15.1:0/330648951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeac0ac4a0 con 0x7feeac0c6730 2024-03-06T07:23:34.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.050+0000 7feea27fc640 1 -- 172.21.15.1:0/330648951 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9c011c80 con 0x7feeac0c6730 2024-03-06T07:23:34.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.050+0000 7feeb3f3e640 1 -- 172.21.15.1:0/330648951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeac1eb6e0 con 0x7feeac0c6730 2024-03-06T07:23:34.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.051+0000 7feeb3f3e640 1 -- 172.21.15.1:0/330648951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeac067fd0 con 0x7feeac0c6730 2024-03-06T07:23:34.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.052+0000 7feea27fc640 1 -- 172.21.15.1:0/330648951 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee9c018020 con 0x7feeac0c6730 2024-03-06T07:23:34.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.052+0000 7feea27fc640 1 --2- 172.21.15.1:0/330648951 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee9006b4d0 0x7fee9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.052+0000 7feea27fc640 1 -- 172.21.15.1:0/330648951 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fee9c08a250 con 0x7feeac0c6730 2024-03-06T07:23:34.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.052+0000 7feeb1cda640 1 --2- 172.21.15.1:0/330648951 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee9006b4d0 0x7fee9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.053+0000 7feeb1cda640 1 --2- 172.21.15.1:0/330648951 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee9006b4d0 0x7fee9006d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7feea8036000 tx=0x7feea803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.056+0000 7feea27fc640 1 -- 172.21.15.1:0/330648951 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee9c01f020 con 0x7feeac0c6730 2024-03-06T07:23:34.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.306+0000 7feeb3f3e640 1 -- 172.21.15.1:0/330648951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]} v 0) v1 -- 0x7feeac0c2c40 con 0x7feeac0c6730 2024-03-06T07:23:34.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.665+0000 7feea27fc640 1 -- 172.21.15.1:0/330648951 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]}]=0 v200) v1 ==== 158+0+0 (secure 0 0 0) 0x7feeac1eb460 con 0x7feeac0c6730 2024-03-06T07:23:34.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.668+0000 7feeb3f3e640 1 -- 172.21.15.1:0/330648951 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee9006b4d0 msgr2=0x7fee9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.668+0000 7feeb3f3e640 1 --2- 172.21.15.1:0/330648951 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee9006b4d0 0x7fee9006d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7feea8036000 tx=0x7feea803d040 comp rx=0 tx=0).stop 2024-03-06T07:23:34.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.668+0000 7feeb3f3e640 1 -- 172.21.15.1:0/330648951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0c6730 msgr2=0x7feeac1e7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.668+0000 7feeb3f3e640 1 --2- 172.21.15.1:0/330648951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0c6730 0x7feeac1e7b40 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fee9c00b440 tx=0x7fee9c00b900 comp rx=0 tx=0).stop 2024-03-06T07:23:34.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.668+0000 7feeb3f3e640 1 -- 172.21.15.1:0/330648951 shutdown_connections 2024-03-06T07:23:34.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.668+0000 7feeb3f3e640 1 --2- 172.21.15.1:0/330648951 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee9006b4d0 0x7fee9006d990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.668+0000 7feeb3f3e640 1 --2- 172.21.15.1:0/330648951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac152690 0x7feeac1ee000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.668+0000 7feeb3f3e640 1 --2- 172.21.15.1:0/330648951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0c6730 0x7feeac1e7b40 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.668+0000 7feeb3f3e640 1 --2- 172.21.15.1:0/330648951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feeac0c5d10 0x7feeac06d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.668+0000 7feeb3f3e640 1 -- 172.21.15.1:0/330648951 >> 172.21.15.1:0/330648951 conn(0x7feeac06b410 msgr2=0x7feeac06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.668+0000 7feeb3f3e640 1 -- 172.21.15.1:0/330648951 shutdown_connections 2024-03-06T07:23:34.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.668+0000 7feeb3f3e640 1 -- 172.21.15.1:0/330648951 wait complete. 2024-03-06T07:23:34.681 INFO:tasks.cephfs_test_runner:That a standby with mds_join_fs set to another fs is not used if refuse_standby_for_another_fs is set. ... ok 2024-03-06T07:23:34.682 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_runtime (tasks.cephfs.test_failover.TestClusterAffinity) That setting mds_join_fs at runtime affects the cluster layout. 2024-03-06T07:23:34.682 INFO:tasks.cephfs_test_runner:test_join_fs_runtime (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-06T07:23:34.682 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:34.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.783+0000 7ff16fcdf640 1 Processor -- start 2024-03-06T07:23:34.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.783+0000 7ff16fcdf640 1 -- start start 2024-03-06T07:23:34.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.784+0000 7ff16fcdf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff1680c6b10 0x7ff168155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.784+0000 7ff16fcdf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1680bf9d0 0x7ff1680c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.784+0000 7ff16fcdf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1680c2fc0 con 0x7ff168152650 2024-03-06T07:23:34.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.784+0000 7ff16fcdf640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff168155800 con 0x7ff1680c6b10 2024-03-06T07:23:34.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.784+0000 7ff16fcdf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff168158da0 con 0x7ff1680bf9d0 2024-03-06T07:23:34.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.784+0000 7ff16da7b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff168152650 0x7ff168158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54734/0 (socket says 172.21.15.1:54734) 2024-03-06T07:23:34.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.784+0000 7ff16da7b640 1 -- 172.21.15.1:0/4010997532 learned_addr learned my addr 172.21.15.1:0/4010997532 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:34.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.784+0000 7ff16e27c640 1 --2- 172.21.15.1:0/4010997532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1680bf9d0 0x7ff1680c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.784+0000 7ff16d27a640 1 --2- 172.21.15.1:0/4010997532 >> v2:172.21.15.98:3300/0 conn(0x7ff1680c6b10 0x7ff168155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16ca79640 1 -- 172.21.15.1:0/4010997532 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198464453 0 0) 0x7ff1680c2fc0 con 0x7ff168152650 2024-03-06T07:23:34.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16ca79640 1 -- 172.21.15.1:0/4010997532 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff150003520 con 0x7ff168152650 2024-03-06T07:23:34.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16ca79640 1 -- 172.21.15.1:0/4010997532 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3741975828 0 0) 0x7ff150003520 con 0x7ff168152650 2024-03-06T07:23:34.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16ca79640 1 -- 172.21.15.1:0/4010997532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1680bf9d0 msgr2=0x7ff1680c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16ca79640 1 --2- 172.21.15.1:0/4010997532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1680bf9d0 0x7ff1680c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16ca79640 1 -- 172.21.15.1:0/4010997532 >> v2:172.21.15.98:3300/0 conn(0x7ff1680c6b10 msgr2=0x7ff168155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16ca79640 1 --2- 172.21.15.1:0/4010997532 >> v2:172.21.15.98:3300/0 conn(0x7ff1680c6b10 0x7ff168155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16ca79640 1 -- 172.21.15.1:0/4010997532 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1680c6690 con 0x7ff168152650 2024-03-06T07:23:34.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16ca79640 1 -- 172.21.15.1:0/4010997532 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7ff160001690 con 0x7ff168152650 2024-03-06T07:23:34.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16ca79640 1 -- 172.21.15.1:0/4010997532 >> v1:172.21.15.1:6789/0 conn(0x7ff168152650 legacy=0x7ff168158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16ca79640 1 --2- 172.21.15.1:0/4010997532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1680bf9d0 0x7ff1500039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16ca79640 1 --2- 172.21.15.1:0/4010997532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff150003f30 0x7ff150004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16ca79640 1 --2- 172.21.15.1:0/4010997532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff150004d40 0x7ff1500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16ca79640 1 -- 172.21.15.1:0/4010997532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff168155800 con 0x7ff150004d40 2024-03-06T07:23:34.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16ca79640 1 -- 172.21.15.1:0/4010997532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff168158da0 con 0x7ff150003f30 2024-03-06T07:23:34.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16ca79640 1 -- 172.21.15.1:0/4010997532 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff15000ba10 con 0x7ff1680bf9d0 2024-03-06T07:23:34.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16e27c640 1 --2- 172.21.15.1:0/4010997532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff150004d40 0x7ff1500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16da7b640 1 --2- 172.21.15.1:0/4010997532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1680bf9d0 0x7ff1500039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16d27a640 1 --2- 172.21.15.1:0/4010997532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff150003f30 0x7ff150004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16e27c640 1 -- 172.21.15.1:0/4010997532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1680bf9d0 msgr2=0x7ff1500039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16e27c640 1 --2- 172.21.15.1:0/4010997532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1680bf9d0 0x7ff1500039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16e27c640 1 -- 172.21.15.1:0/4010997532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff150003f30 msgr2=0x7ff150004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16e27c640 1 --2- 172.21.15.1:0/4010997532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff150003f30 0x7ff150004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.785+0000 7ff16e27c640 1 -- 172.21.15.1:0/4010997532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff150003520 con 0x7ff150004d40 2024-03-06T07:23:34.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.786+0000 7ff16e27c640 1 --2- 172.21.15.1:0/4010997532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff150004d40 0x7ff1500051c0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7ff15800e170 tx=0x7ff150005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d60498c437a9516 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.786+0000 7ff16ca79640 1 -- 172.21.15.1:0/4010997532 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff158039070 con 0x7ff150004d40 2024-03-06T07:23:34.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.786+0000 7ff16ca79640 1 -- 172.21.15.1:0/4010997532 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff15802cbf0 con 0x7ff150004d40 2024-03-06T07:23:34.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.786+0000 7ff16ca79640 1 -- 172.21.15.1:0/4010997532 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff158009c10 con 0x7ff150004d40 2024-03-06T07:23:34.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.787+0000 7ff16fcdf640 1 -- 172.21.15.1:0/4010997532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff150004d40 msgr2=0x7ff1500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.787+0000 7ff16fcdf640 1 --2- 172.21.15.1:0/4010997532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff150004d40 0x7ff1500051c0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7ff15800e170 tx=0x7ff150005830 comp rx=0 tx=0).stop 2024-03-06T07:23:34.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.787+0000 7ff16e27c640 1 -- 172.21.15.1:0/4010997532 reap_dead start 2024-03-06T07:23:34.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.787+0000 7ff16fcdf640 1 -- 172.21.15.1:0/4010997532 shutdown_connections 2024-03-06T07:23:34.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.787+0000 7ff16fcdf640 1 -- 172.21.15.1:0/4010997532 >> 172.21.15.1:0/4010997532 conn(0x7ff168066ee0 msgr2=0x7ff1680672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.787+0000 7ff16fcdf640 1 -- 172.21.15.1:0/4010997532 shutdown_connections 2024-03-06T07:23:34.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.787+0000 7ff16fcdf640 1 -- 172.21.15.1:0/4010997532 wait complete. 2024-03-06T07:23:34.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.788+0000 7ff16fcdf640 1 Processor -- start 2024-03-06T07:23:34.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.788+0000 7ff16fcdf640 1 -- start start 2024-03-06T07:23:34.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16fcdf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1681ee5a0 0x7ff168150b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16fcdf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff168156490 0x7ff168156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16fcdf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1681ee980 0x7ff16814a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16fcdf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1681551b0 con 0x7ff168156490 2024-03-06T07:23:34.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16fcdf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1680c6fa0 con 0x7ff1681ee980 2024-03-06T07:23:34.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16fcdf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff168155490 con 0x7ff1681ee5a0 2024-03-06T07:23:34.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16d27a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff168156490 0x7ff168156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16d27a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff168156490 0x7ff168156e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49784/0 (socket says 172.21.15.1:49784) 2024-03-06T07:23:34.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16d27a640 1 -- 172.21.15.1:0/2627504519 learned_addr learned my addr 172.21.15.1:0/2627504519 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:34.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16e27c640 1 --2- 172.21.15.1:0/2627504519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1681ee980 0x7ff16814a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16da7b640 1 --2- 172.21.15.1:0/2627504519 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1681ee5a0 0x7ff168150b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16d27a640 1 -- 172.21.15.1:0/2627504519 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1681ee5a0 msgr2=0x7ff168150b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16d27a640 1 --2- 172.21.15.1:0/2627504519 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1681ee5a0 0x7ff168150b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16d27a640 1 -- 172.21.15.1:0/2627504519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1681ee980 msgr2=0x7ff16814a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16d27a640 1 --2- 172.21.15.1:0/2627504519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1681ee980 0x7ff16814a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16d27a640 1 -- 172.21.15.1:0/2627504519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16814b5a0 con 0x7ff168156490 2024-03-06T07:23:34.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16e27c640 1 --2- 172.21.15.1:0/2627504519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1681ee980 0x7ff16814a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:34.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16d27a640 1 --2- 172.21.15.1:0/2627504519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff168156490 0x7ff168156e30 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7ff15c011990 tx=0x7ff15c011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff1667fc640 1 -- 172.21.15.1:0/2627504519 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c00fa30 con 0x7ff168156490 2024-03-06T07:23:34.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff1667fc640 1 -- 172.21.15.1:0/2627504519 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff15c00fbd0 con 0x7ff168156490 2024-03-06T07:23:34.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16fcdf640 1 -- 172.21.15.1:0/2627504519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff168151c10 con 0x7ff168156490 2024-03-06T07:23:34.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff1667fc640 1 -- 172.21.15.1:0/2627504519 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c014440 con 0x7ff168156490 2024-03-06T07:23:34.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.789+0000 7ff16fcdf640 1 -- 172.21.15.1:0/2627504519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff168157370 con 0x7ff168156490 2024-03-06T07:23:34.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.790+0000 7ff16fcdf640 1 -- 172.21.15.1:0/2627504519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff168151a00 con 0x7ff168156490 2024-03-06T07:23:34.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.791+0000 7ff1667fc640 1 -- 172.21.15.1:0/2627504519 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff15c012040 con 0x7ff168156490 2024-03-06T07:23:34.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.791+0000 7ff1667fc640 1 --2- 172.21.15.1:0/2627504519 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff15406b500 0x7ff15406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.791+0000 7ff1667fc640 1 -- 172.21.15.1:0/2627504519 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7ff15c08a700 con 0x7ff168156490 2024-03-06T07:23:34.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.791+0000 7ff16da7b640 1 --2- 172.21.15.1:0/2627504519 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff15406b500 0x7ff15406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.792+0000 7ff16da7b640 1 --2- 172.21.15.1:0/2627504519 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff15406b500 0x7ff15406d9c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff160007fd0 tx=0x7ff160007970 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.795+0000 7ff1667fc640 1 -- 172.21.15.1:0/2627504519 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff15c019030 con 0x7ff168156490 2024-03-06T07:23:34.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.977+0000 7ff16fcdf640 1 -- 172.21.15.1:0/2627504519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff168067fd0 con 0x7ff168156490 2024-03-06T07:23:34.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.978+0000 7ff1667fc640 1 -- 172.21.15.1:0/2627504519 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+13556 (secure 0 0 0) 0x7ff15c059680 con 0x7ff168156490 2024-03-06T07:23:34.980 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:34.980 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":86,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:23:32.660423+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6835/2685225325":"2024-03-07T07:23:29.648723+0000","172.21.15.1:6834/3919123103":"2024-03-07T07:23:23.886065+0000","172.21.15.1:6833/2685225325":"2024-03-07T07:23:29.648723+0000","172.21.15.98:6835/114998998":"2024-03-07T07:23:24.253819+0000","172.21.15.1:6832/3919123103":"2024-03-07T07:23:23.886065+0000","172.21.15.1:6834/1494116991":"2024-03-07T07:23:13.133394+0000","172.21.15.98:6837/114998998":"2024-03-07T07:23:24.253819+0000","172.21.15.1:6832/1494116991":"2024-03-07T07:23:13.133394+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:34.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.980+0000 7ff16fcdf640 1 -- 172.21.15.1:0/2627504519 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff15406b500 msgr2=0x7ff15406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.980+0000 7ff16fcdf640 1 --2- 172.21.15.1:0/2627504519 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff15406b500 0x7ff15406d9c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff160007fd0 tx=0x7ff160007970 comp rx=0 tx=0).stop 2024-03-06T07:23:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.980+0000 7ff16fcdf640 1 -- 172.21.15.1:0/2627504519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff168156490 msgr2=0x7ff168156e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.980+0000 7ff16fcdf640 1 --2- 172.21.15.1:0/2627504519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff168156490 0x7ff168156e30 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7ff15c011990 tx=0x7ff15c011d90 comp rx=0 tx=0).stop 2024-03-06T07:23:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.980+0000 7ff16fcdf640 1 -- 172.21.15.1:0/2627504519 shutdown_connections 2024-03-06T07:23:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.980+0000 7ff16fcdf640 1 --2- 172.21.15.1:0/2627504519 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff15406b500 0x7ff15406d9c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.980+0000 7ff16fcdf640 1 --2- 172.21.15.1:0/2627504519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1681ee980 0x7ff16814a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.980+0000 7ff16fcdf640 1 --2- 172.21.15.1:0/2627504519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff168156490 0x7ff168156e30 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.980+0000 7ff16fcdf640 1 --2- 172.21.15.1:0/2627504519 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1681ee5a0 0x7ff168150b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.980+0000 7ff16fcdf640 1 -- 172.21.15.1:0/2627504519 >> 172.21.15.1:0/2627504519 conn(0x7ff168066ee0 msgr2=0x7ff1680c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.980+0000 7ff16fcdf640 1 -- 172.21.15.1:0/2627504519 shutdown_connections 2024-03-06T07:23:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:34.980+0000 7ff16fcdf640 1 -- 172.21.15.1:0/2627504519 wait complete. 2024-03-06T07:23:34.993 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:34.994 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:35.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.096+0000 7f520748f640 1 Processor -- start 2024-03-06T07:23:35.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.096+0000 7f520748f640 1 -- start start 2024-03-06T07:23:35.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.096+0000 7f520748f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5200152730 0x7f52000c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.096+0000 7f520748f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f52000c1aa0 0x7f52001552e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.097+0000 7f520748f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52001588e0 con 0x7f52000c6fd0 2024-03-06T07:23:35.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.097+0000 7f520748f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5200158bc0 con 0x7f5200152730 2024-03-06T07:23:35.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.097+0000 7f520748f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5200158ea0 con 0x7f52000c1aa0 2024-03-06T07:23:35.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.097+0000 7f520522b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5200152730 0x7f52000c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.097+0000 7f520522b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5200152730 0x7f52000c6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49792/0 (socket says 172.21.15.1:49792) 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.097+0000 7f520522b640 1 -- 172.21.15.1:0/1526911898 learned_addr learned my addr 172.21.15.1:0/1526911898 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.097+0000 7f5205a2c640 1 --2- 172.21.15.1:0/1526911898 >> v2:172.21.15.98:3300/0 conn(0x7f52000c1aa0 0x7f52001552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.097+0000 7f520522b640 1 -- 172.21.15.1:0/1526911898 >> v2:172.21.15.98:3300/0 conn(0x7f52000c1aa0 msgr2=0x7f52001552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.097+0000 7f520522b640 1 --2- 172.21.15.1:0/1526911898 >> v2:172.21.15.98:3300/0 conn(0x7f52000c1aa0 0x7f52001552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.097+0000 7f520522b640 1 -- 172.21.15.1:0/1526911898 >> v1:172.21.15.98:6789/0 conn(0x7f52000c6fd0 legacy=0x7f52000c1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.097+0000 7f520522b640 1 -- 172.21.15.1:0/1526911898 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5200159180 con 0x7f5200152730 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.098+0000 7f520522b640 1 --2- 172.21.15.1:0/1526911898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5200152730 0x7f52000c6a90 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f52000b0440 tx=0x7f51fc02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac87ae37cd6180b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.098+0000 7f51f7fff640 1 -- 172.21.15.1:0/1526911898 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc038070 con 0x7f5200152730 2024-03-06T07:23:35.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.098+0000 7f51f7fff640 1 -- 172.21.15.1:0/1526911898 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51fc02f050 con 0x7f5200152730 2024-03-06T07:23:35.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.098+0000 7f51f7fff640 1 -- 172.21.15.1:0/1526911898 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc031970 con 0x7f5200152730 2024-03-06T07:23:35.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.098+0000 7f520748f640 1 -- 172.21.15.1:0/1526911898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5200152730 msgr2=0x7f52000c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.098+0000 7f520748f640 1 --2- 172.21.15.1:0/1526911898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5200152730 0x7f52000c6a90 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f52000b0440 tx=0x7f51fc02b7f0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.098+0000 7f520748f640 1 -- 172.21.15.1:0/1526911898 shutdown_connections 2024-03-06T07:23:35.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.098+0000 7f520748f640 1 --2- 172.21.15.1:0/1526911898 >> v2:172.21.15.98:3300/0 conn(0x7f52000c1aa0 0x7f52001552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.098+0000 7f520748f640 1 --2- 172.21.15.1:0/1526911898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5200152730 0x7f52000c6a90 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.098+0000 7f520748f640 1 -- 172.21.15.1:0/1526911898 >> 172.21.15.1:0/1526911898 conn(0x7f5200066d00 msgr2=0x7f5200067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.099+0000 7f520748f640 1 -- 172.21.15.1:0/1526911898 shutdown_connections 2024-03-06T07:23:35.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.099+0000 7f520748f640 1 -- 172.21.15.1:0/1526911898 wait complete. 2024-03-06T07:23:35.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.100+0000 7f520748f640 1 Processor -- start 2024-03-06T07:23:35.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.100+0000 7f520748f640 1 -- start start 2024-03-06T07:23:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.100+0000 7f520748f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52000c1aa0 0x7f52001eeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.100+0000 7f520748f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52000c6fd0 0x7f52000c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.100+0000 7f520748f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5200152730 0x7f520014c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.100+0000 7f520748f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5200155820 con 0x7f52000c1aa0 2024-03-06T07:23:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.100+0000 7f520748f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5200152b10 con 0x7f5200152730 2024-03-06T07:23:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.100+0000 7f520748f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52001559a0 con 0x7f52000c6fd0 2024-03-06T07:23:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.100+0000 7f520522b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52000c1aa0 0x7f52001eeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.100+0000 7f520522b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52000c1aa0 0x7f52001eeb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49800/0 (socket says 172.21.15.1:49800) 2024-03-06T07:23:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.100+0000 7f520522b640 1 -- 172.21.15.1:0/2352925883 learned_addr learned my addr 172.21.15.1:0/2352925883 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.101+0000 7f5205a2c640 1 --2- 172.21.15.1:0/2352925883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5200152730 0x7f520014c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.101+0000 7f5204a2a640 1 --2- 172.21.15.1:0/2352925883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52000c6fd0 0x7f52000c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.101+0000 7f520522b640 1 -- 172.21.15.1:0/2352925883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52000c6fd0 msgr2=0x7f52000c3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.101+0000 7f520522b640 1 --2- 172.21.15.1:0/2352925883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52000c6fd0 0x7f52000c3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.101+0000 7f520522b640 1 -- 172.21.15.1:0/2352925883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5200152730 msgr2=0x7f520014c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.101+0000 7f520522b640 1 --2- 172.21.15.1:0/2352925883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5200152730 0x7f520014c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.101+0000 7f520522b640 1 -- 172.21.15.1:0/2352925883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52000c9a30 con 0x7f52000c1aa0 2024-03-06T07:23:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.101+0000 7f520522b640 1 --2- 172.21.15.1:0/2352925883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52000c1aa0 0x7f52001eeb90 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f51fc000c00 tx=0x7f51fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.101+0000 7f51f5ffb640 1 -- 172.21.15.1:0/2352925883 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc042070 con 0x7f52000c1aa0 2024-03-06T07:23:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.101+0000 7f51f5ffb640 1 -- 172.21.15.1:0/2352925883 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51fc03d870 con 0x7f52000c1aa0 2024-03-06T07:23:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.101+0000 7f51f5ffb640 1 -- 172.21.15.1:0/2352925883 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc038040 con 0x7f52000c1aa0 2024-03-06T07:23:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.101+0000 7f520748f640 1 -- 172.21.15.1:0/2352925883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52000ca680 con 0x7f52000c1aa0 2024-03-06T07:23:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.101+0000 7f520748f640 1 -- 172.21.15.1:0/2352925883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52001ee390 con 0x7f52000c1aa0 2024-03-06T07:23:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.102+0000 7f520748f640 1 -- 172.21.15.1:0/2352925883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52000686e0 con 0x7f52000c1aa0 2024-03-06T07:23:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.103+0000 7f51f5ffb640 1 -- 172.21.15.1:0/2352925883 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f51fc045070 con 0x7f52000c1aa0 2024-03-06T07:23:35.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.103+0000 7f51f5ffb640 1 --2- 172.21.15.1:0/2352925883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51e406b500 0x7f51e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.103+0000 7f51f5ffb640 1 -- 172.21.15.1:0/2352925883 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f51fc0abfc0 con 0x7f52000c1aa0 2024-03-06T07:23:35.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.103+0000 7f5204a2a640 1 --2- 172.21.15.1:0/2352925883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51e406b500 0x7f51e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.104+0000 7f5204a2a640 1 --2- 172.21.15.1:0/2352925883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51e406b500 0x7f51e406d9c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f52001ebbd0 tx=0x7f51f0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.107+0000 7f51f5ffb640 1 -- 172.21.15.1:0/2352925883 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52000686e0 con 0x7f52000c1aa0 2024-03-06T07:23:35.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.287+0000 7f520748f640 1 -- 172.21.15.1:0/2352925883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5200067fd0 con 0x7f52000c1aa0 2024-03-06T07:23:35.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.288+0000 7f51f5ffb640 1 -- 172.21.15.1:0/2352925883 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f51fc07b060 con 0x7f52000c1aa0 2024-03-06T07:23:35.290 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:23:35.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.290+0000 7f520748f640 1 -- 172.21.15.1:0/2352925883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51e406b500 msgr2=0x7f51e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.290+0000 7f520748f640 1 --2- 172.21.15.1:0/2352925883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51e406b500 0x7f51e406d9c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f52001ebbd0 tx=0x7f51f0009040 comp rx=0 tx=0).stop 2024-03-06T07:23:35.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.290+0000 7f520748f640 1 -- 172.21.15.1:0/2352925883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52000c1aa0 msgr2=0x7f52001eeb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.290+0000 7f520748f640 1 --2- 172.21.15.1:0/2352925883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52000c1aa0 0x7f52001eeb90 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f51fc000c00 tx=0x7f51fc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.290+0000 7f520748f640 1 -- 172.21.15.1:0/2352925883 shutdown_connections 2024-03-06T07:23:35.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.290+0000 7f520748f640 1 --2- 172.21.15.1:0/2352925883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51e406b500 0x7f51e406d9c0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.290+0000 7f520748f640 1 --2- 172.21.15.1:0/2352925883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5200152730 0x7f520014c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.290+0000 7f520748f640 1 --2- 172.21.15.1:0/2352925883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52000c6fd0 0x7f52000c3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.290+0000 7f520748f640 1 --2- 172.21.15.1:0/2352925883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52000c1aa0 0x7f52001eeb90 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.290+0000 7f520748f640 1 -- 172.21.15.1:0/2352925883 >> 172.21.15.1:0/2352925883 conn(0x7f5200066d00 msgr2=0x7f52000670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.290+0000 7f520748f640 1 -- 172.21.15.1:0/2352925883 shutdown_connections 2024-03-06T07:23:35.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.290+0000 7f520748f640 1 -- 172.21.15.1:0/2352925883 wait complete. 2024-03-06T07:23:35.303 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime' 2024-03-06T07:23:35.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.402+0000 7f50c759e640 1 Processor -- start 2024-03-06T07:23:35.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.403+0000 7f50c759e640 1 -- start start 2024-03-06T07:23:35.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.403+0000 7f50c759e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f50c814b9d0 0x7f50c8155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.403+0000 7f50c759e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f50c80c6980 0x7f50c80c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.403+0000 7f50c759e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c80cb050 con 0x7f50c80c5ef0 2024-03-06T07:23:35.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.403+0000 7f50c759e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50c80cb330 con 0x7f50c80c6980 2024-03-06T07:23:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.403+0000 7f50c759e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50c80c5a60 con 0x7f50c814b9d0 2024-03-06T07:23:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.403+0000 7f50c6d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f50c80c6980 0x7f50c80c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.403+0000 7f50bffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f50c80c5ef0 0x7f50c80ca890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54746/0 (socket says 172.21.15.1:54746) 2024-03-06T07:23:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.403+0000 7f50bffff640 1 -- 172.21.15.1:0/267209555 learned_addr learned my addr 172.21.15.1:0/267209555 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.404+0000 7f50c659c640 1 --2- 172.21.15.1:0/267209555 >> v2:172.21.15.98:3300/0 conn(0x7f50c814b9d0 0x7f50c8155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.404+0000 7f50c6d9d640 1 -- 172.21.15.1:0/267209555 >> v2:172.21.15.98:3300/0 conn(0x7f50c814b9d0 msgr2=0x7f50c8155ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.404+0000 7f50c6d9d640 1 --2- 172.21.15.1:0/267209555 >> v2:172.21.15.98:3300/0 conn(0x7f50c814b9d0 0x7f50c8155ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.404+0000 7f50c6d9d640 1 -- 172.21.15.1:0/267209555 >> v1:172.21.15.1:6789/0 conn(0x7f50c80c5ef0 legacy=0x7f50c80ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.404+0000 7f50c6d9d640 1 -- 172.21.15.1:0/267209555 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c80c5410 con 0x7f50c80c6980 2024-03-06T07:23:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.404+0000 7f50c5d9b640 1 -- 172.21.15.1:0/267209555 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983943538 0 0) 0x7f50c80cb050 con 0x7f50c80c5ef0 2024-03-06T07:23:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.404+0000 7f50c659c640 1 --2- 172.21.15.1:0/267209555 >> v2:172.21.15.98:3300/0 conn(0x7f50c814b9d0 0x7f50c8155ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.404+0000 7f50c6d9d640 1 --2- 172.21.15.1:0/267209555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50c80c6980 0x7f50c80c6d90 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f50c80c7dd0 tx=0x7f50c002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf6215cbc3ab9de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.405+0000 7f50c5d9b640 1 -- 172.21.15.1:0/267209555 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c0039070 con 0x7f50c80c6980 2024-03-06T07:23:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.405+0000 7f50c5d9b640 1 -- 172.21.15.1:0/267209555 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c002cb60 con 0x7f50c80c6980 2024-03-06T07:23:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.405+0000 7f50c5d9b640 1 -- 172.21.15.1:0/267209555 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c0034440 con 0x7f50c80c6980 2024-03-06T07:23:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.405+0000 7f50c759e640 1 -- 172.21.15.1:0/267209555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50c80c6980 msgr2=0x7f50c80c6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.405+0000 7f50c759e640 1 --2- 172.21.15.1:0/267209555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50c80c6980 0x7f50c80c6d90 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f50c80c7dd0 tx=0x7f50c002bdb0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.405+0000 7f50c759e640 1 -- 172.21.15.1:0/267209555 shutdown_connections 2024-03-06T07:23:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.405+0000 7f50c759e640 1 --2- 172.21.15.1:0/267209555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50c80c6980 0x7f50c80c6d90 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.405+0000 7f50c759e640 1 --2- 172.21.15.1:0/267209555 >> v2:172.21.15.98:3300/0 conn(0x7f50c814b9d0 0x7f50c8155ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.405+0000 7f50c759e640 1 -- 172.21.15.1:0/267209555 >> 172.21.15.1:0/267209555 conn(0x7f50c80674f0 msgr2=0x7f50c80c73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.405+0000 7f50c759e640 1 -- 172.21.15.1:0/267209555 shutdown_connections 2024-03-06T07:23:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.405+0000 7f50c759e640 1 -- 172.21.15.1:0/267209555 wait complete. 2024-03-06T07:23:35.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.406+0000 7f50c759e640 1 Processor -- start 2024-03-06T07:23:35.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.406+0000 7f50c759e640 1 -- start start 2024-03-06T07:23:35.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.407+0000 7f50c759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50c80c5ef0 0x7f50c80c1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.407+0000 7f50c759e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50c80c6980 0x7f50c8157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.407+0000 7f50c759e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50c814b9d0 0x7f50c81ea900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.407+0000 7f50c759e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c80c3190 con 0x7f50c814b9d0 2024-03-06T07:23:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.407+0000 7f50c759e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c814bdb0 con 0x7f50c80c5ef0 2024-03-06T07:23:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.407+0000 7f50c759e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c80c3310 con 0x7f50c80c6980 2024-03-06T07:23:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.407+0000 7f50c6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50c814b9d0 0x7f50c81ea900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.407+0000 7f50c6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50c814b9d0 0x7f50c81ea900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49828/0 (socket says 172.21.15.1:49828) 2024-03-06T07:23:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.407+0000 7f50c6d9d640 1 -- 172.21.15.1:0/164276628 learned_addr learned my addr 172.21.15.1:0/164276628 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.407+0000 7f50bffff640 1 --2- 172.21.15.1:0/164276628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50c80c6980 0x7f50c8157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.407+0000 7f50c659c640 1 --2- 172.21.15.1:0/164276628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50c80c5ef0 0x7f50c80c1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.407+0000 7f50c6d9d640 1 -- 172.21.15.1:0/164276628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50c80c6980 msgr2=0x7f50c8157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.407+0000 7f50c6d9d640 1 --2- 172.21.15.1:0/164276628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50c80c6980 0x7f50c8157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.407+0000 7f50c6d9d640 1 -- 172.21.15.1:0/164276628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50c80c5ef0 msgr2=0x7f50c80c1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.407+0000 7f50c6d9d640 1 --2- 172.21.15.1:0/164276628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50c80c5ef0 0x7f50c80c1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.407+0000 7f50c6d9d640 1 -- 172.21.15.1:0/164276628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c8152eb0 con 0x7f50c814b9d0 2024-03-06T07:23:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.408+0000 7f50c6d9d640 1 --2- 172.21.15.1:0/164276628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50c814b9d0 0x7f50c81ea900 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f50c0037040 tx=0x7f50c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.408+0000 7f50beffd640 1 -- 172.21.15.1:0/164276628 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c0044020 con 0x7f50c814b9d0 2024-03-06T07:23:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.408+0000 7f50beffd640 1 -- 172.21.15.1:0/164276628 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c003a040 con 0x7f50c814b9d0 2024-03-06T07:23:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.408+0000 7f50c759e640 1 -- 172.21.15.1:0/164276628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c8153190 con 0x7f50c814b9d0 2024-03-06T07:23:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.408+0000 7f50beffd640 1 -- 172.21.15.1:0/164276628 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c0039050 con 0x7f50c814b9d0 2024-03-06T07:23:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.408+0000 7f50c759e640 1 -- 172.21.15.1:0/164276628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c80c0c30 con 0x7f50c814b9d0 2024-03-06T07:23:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.409+0000 7f50c759e640 1 -- 172.21.15.1:0/164276628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c806b3b0 con 0x7f50c814b9d0 2024-03-06T07:23:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.409+0000 7f50beffd640 1 -- 172.21.15.1:0/164276628 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50c002cd10 con 0x7f50c814b9d0 2024-03-06T07:23:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.410+0000 7f50beffd640 1 --2- 172.21.15.1:0/164276628 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50a806b500 0x7f50a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.410+0000 7f50beffd640 1 -- 172.21.15.1:0/164276628 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f50c00ac780 con 0x7f50c814b9d0 2024-03-06T07:23:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.410+0000 7f50c659c640 1 --2- 172.21.15.1:0/164276628 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50a806b500 0x7f50a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.410+0000 7f50c659c640 1 --2- 172.21.15.1:0/164276628 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50a806b500 0x7f50a806d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f50c80b0280 tx=0x7f50b8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.414+0000 7f50beffd640 1 -- 172.21.15.1:0/164276628 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50c806b3b0 con 0x7f50c814b9d0 2024-03-06T07:23:35.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.663+0000 7f50c759e640 1 -- 172.21.15.1:0/164276628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]} v 0) v1 -- 0x7f50c80c6760 con 0x7f50c814b9d0 2024-03-06T07:23:35.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.670+0000 7f50beffd640 1 -- 172.21.15.1:0/164276628 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]}]=0 v201) v1 ==== 149+0+0 (secure 0 0 0) 0x7f50c004e020 con 0x7f50c814b9d0 2024-03-06T07:23:35.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.671+0000 7f50c759e640 1 -- 172.21.15.1:0/164276628 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50a806b500 msgr2=0x7f50a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.671+0000 7f50c759e640 1 --2- 172.21.15.1:0/164276628 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50a806b500 0x7f50a806d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f50c80b0280 tx=0x7f50b8008040 comp rx=0 tx=0).stop 2024-03-06T07:23:35.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.672+0000 7f50c759e640 1 -- 172.21.15.1:0/164276628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50c814b9d0 msgr2=0x7f50c81ea900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.672+0000 7f50c759e640 1 --2- 172.21.15.1:0/164276628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50c814b9d0 0x7f50c81ea900 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f50c0037040 tx=0x7f50c0004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:35.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.672+0000 7f50c759e640 1 -- 172.21.15.1:0/164276628 shutdown_connections 2024-03-06T07:23:35.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.672+0000 7f50c759e640 1 --2- 172.21.15.1:0/164276628 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50a806b500 0x7f50a806d9c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.672+0000 7f50c759e640 1 --2- 172.21.15.1:0/164276628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50c814b9d0 0x7f50c81ea900 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.672+0000 7f50c759e640 1 --2- 172.21.15.1:0/164276628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50c80c6980 0x7f50c8157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.672+0000 7f50c759e640 1 --2- 172.21.15.1:0/164276628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50c80c5ef0 0x7f50c80c1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.672+0000 7f50c759e640 1 -- 172.21.15.1:0/164276628 >> 172.21.15.1:0/164276628 conn(0x7f50c80674f0 msgr2=0x7f50c814a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.672+0000 7f50c759e640 1 -- 172.21.15.1:0/164276628 shutdown_connections 2024-03-06T07:23:35.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.672+0000 7f50c759e640 1 -- 172.21.15.1:0/164276628 wait complete. 2024-03-06T07:23:35.684 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:23:35.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.786+0000 7f1b63671640 1 Processor -- start 2024-03-06T07:23:35.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.787+0000 7f1b63671640 1 -- start start 2024-03-06T07:23:35.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.787+0000 7f1b63671640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1b5c0c6b10 0x7f1b5c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.787+0000 7f1b63671640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b5c0bf9d0 0x7f1b5c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.787+0000 7f1b63671640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b5c0c2fc0 con 0x7f1b5c152650 2024-03-06T07:23:35.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.787+0000 7f1b63671640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b5c155800 con 0x7f1b5c0c6b10 2024-03-06T07:23:35.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.787+0000 7f1b63671640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c158da0 con 0x7f1b5c0bf9d0 2024-03-06T07:23:35.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.787+0000 7f1b6140d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1b5c152650 0x7f1b5c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54748/0 (socket says 172.21.15.1:54748) 2024-03-06T07:23:35.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.787+0000 7f1b6140d640 1 -- 172.21.15.1:0/3941155821 learned_addr learned my addr 172.21.15.1:0/3941155821 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:35.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b61c0e640 1 --2- 172.21.15.1:0/3941155821 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b5c0bf9d0 0x7f1b5c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b60c0c640 1 --2- 172.21.15.1:0/3941155821 >> v2:172.21.15.98:3300/0 conn(0x7f1b5c0c6b10 0x7f1b5c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b5bfff640 1 -- 172.21.15.1:0/3941155821 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799044746 0 0) 0x7f1b5c0c2fc0 con 0x7f1b5c152650 2024-03-06T07:23:35.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b5bfff640 1 -- 172.21.15.1:0/3941155821 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b44003520 con 0x7f1b5c152650 2024-03-06T07:23:35.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b5bfff640 1 -- 172.21.15.1:0/3941155821 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2908307570 0 0) 0x7f1b44003520 con 0x7f1b5c152650 2024-03-06T07:23:35.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b5bfff640 1 -- 172.21.15.1:0/3941155821 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b5c0bf9d0 msgr2=0x7f1b5c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b5bfff640 1 --2- 172.21.15.1:0/3941155821 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b5c0bf9d0 0x7f1b5c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b5bfff640 1 -- 172.21.15.1:0/3941155821 >> v2:172.21.15.98:3300/0 conn(0x7f1b5c0c6b10 msgr2=0x7f1b5c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b5bfff640 1 --2- 172.21.15.1:0/3941155821 >> v2:172.21.15.98:3300/0 conn(0x7f1b5c0c6b10 0x7f1b5c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b5bfff640 1 -- 172.21.15.1:0/3941155821 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5c0c6690 con 0x7f1b5c152650 2024-03-06T07:23:35.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b5bfff640 1 -- 172.21.15.1:0/3941155821 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f1b54001690 con 0x7f1b5c152650 2024-03-06T07:23:35.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b5bfff640 1 -- 172.21.15.1:0/3941155821 >> v1:172.21.15.1:6789/0 conn(0x7f1b5c152650 legacy=0x7f1b5c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b5bfff640 1 --2- 172.21.15.1:0/3941155821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b440039b0 0x7f1b44003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b5bfff640 1 --2- 172.21.15.1:0/3941155821 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b5c0bf9d0 0x7f1b44004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b5bfff640 1 --2- 172.21.15.1:0/3941155821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b44004d40 0x7f1b440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b5bfff640 1 -- 172.21.15.1:0/3941155821 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c155800 con 0x7f1b440039b0 2024-03-06T07:23:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b5bfff640 1 -- 172.21.15.1:0/3941155821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c158da0 con 0x7f1b44004d40 2024-03-06T07:23:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b5bfff640 1 -- 172.21.15.1:0/3941155821 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4400b9f0 con 0x7f1b5c0bf9d0 2024-03-06T07:23:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b6140d640 1 --2- 172.21.15.1:0/3941155821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b440039b0 0x7f1b44003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b60c0c640 1 --2- 172.21.15.1:0/3941155821 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b5c0bf9d0 0x7f1b44004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b61c0e640 1 --2- 172.21.15.1:0/3941155821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b44004d40 0x7f1b440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b6140d640 1 -- 172.21.15.1:0/3941155821 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b5c0bf9d0 msgr2=0x7f1b44004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b6140d640 1 --2- 172.21.15.1:0/3941155821 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b5c0bf9d0 0x7f1b44004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b6140d640 1 -- 172.21.15.1:0/3941155821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b44004d40 msgr2=0x7f1b440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b6140d640 1 --2- 172.21.15.1:0/3941155821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b44004d40 0x7f1b440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.788+0000 7f1b6140d640 1 -- 172.21.15.1:0/3941155821 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b44003520 con 0x7f1b440039b0 2024-03-06T07:23:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.789+0000 7f1b6140d640 1 --2- 172.21.15.1:0/3941155821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b440039b0 0x7f1b44003dd0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1b54007fd0 tx=0x7f1b44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d59927d99dad3cf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.789+0000 7f1b5bfff640 1 -- 172.21.15.1:0/3941155821 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b54007d50 con 0x7f1b440039b0 2024-03-06T07:23:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.789+0000 7f1b5bfff640 1 -- 172.21.15.1:0/3941155821 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b54030070 con 0x7f1b440039b0 2024-03-06T07:23:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.789+0000 7f1b5bfff640 1 -- 172.21.15.1:0/3941155821 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b54005d30 con 0x7f1b440039b0 2024-03-06T07:23:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.790+0000 7f1b63671640 1 -- 172.21.15.1:0/3941155821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b440039b0 msgr2=0x7f1b44003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.790+0000 7f1b63671640 1 --2- 172.21.15.1:0/3941155821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b440039b0 0x7f1b44003dd0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1b54007fd0 tx=0x7f1b44005830 comp rx=0 tx=0).stop 2024-03-06T07:23:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.790+0000 7f1b61c0e640 1 -- 172.21.15.1:0/3941155821 reap_dead start 2024-03-06T07:23:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.790+0000 7f1b63671640 1 -- 172.21.15.1:0/3941155821 shutdown_connections 2024-03-06T07:23:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.790+0000 7f1b63671640 1 -- 172.21.15.1:0/3941155821 >> 172.21.15.1:0/3941155821 conn(0x7f1b5c066ee0 msgr2=0x7f1b5c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.790+0000 7f1b63671640 1 -- 172.21.15.1:0/3941155821 shutdown_connections 2024-03-06T07:23:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.790+0000 7f1b63671640 1 -- 172.21.15.1:0/3941155821 wait complete. 2024-03-06T07:23:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.791+0000 7f1b63671640 1 Processor -- start 2024-03-06T07:23:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.791+0000 7f1b63671640 1 -- start start 2024-03-06T07:23:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b63671640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b440039b0 0x7f1b5c0c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b63671640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b5c156ea0 0x7f1b5c1edf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b63671640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c157ff0 0x7f1b5c1ee480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b63671640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c1551b0 con 0x7f1b5c156ea0 2024-03-06T07:23:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b63671640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c0c6fa0 con 0x7f1b5c157ff0 2024-03-06T07:23:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b63671640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c155490 con 0x7f1b440039b0 2024-03-06T07:23:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b60c0c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b5c156ea0 0x7f1b5c1edf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b60c0c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b5c156ea0 0x7f1b5c1edf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49850/0 (socket says 172.21.15.1:49850) 2024-03-06T07:23:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b60c0c640 1 -- 172.21.15.1:0/3513019444 learned_addr learned my addr 172.21.15.1:0/3513019444 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b6140d640 1 --2- 172.21.15.1:0/3513019444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b440039b0 0x7f1b5c0c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b61c0e640 1 --2- 172.21.15.1:0/3513019444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c157ff0 0x7f1b5c1ee480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b60c0c640 1 -- 172.21.15.1:0/3513019444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b440039b0 msgr2=0x7f1b5c0c5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b60c0c640 1 --2- 172.21.15.1:0/3513019444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b440039b0 0x7f1b5c0c5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b60c0c640 1 -- 172.21.15.1:0/3513019444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c157ff0 msgr2=0x7f1b5c1ee480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b60c0c640 1 --2- 172.21.15.1:0/3513019444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c157ff0 0x7f1b5c1ee480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b60c0c640 1 -- 172.21.15.1:0/3513019444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5c158680 con 0x7f1b5c156ea0 2024-03-06T07:23:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b60c0c640 1 --2- 172.21.15.1:0/3513019444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b5c156ea0 0x7f1b5c1edf40 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f1b50011600 tx=0x7f1b50011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b59ffb640 1 -- 172.21.15.1:0/3513019444 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5000f920 con 0x7f1b5c156ea0 2024-03-06T07:23:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b59ffb640 1 -- 172.21.15.1:0/3513019444 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5000fac0 con 0x7f1b5c156ea0 2024-03-06T07:23:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b63671640 1 -- 172.21.15.1:0/3513019444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5c14b050 con 0x7f1b5c156ea0 2024-03-06T07:23:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.792+0000 7f1b59ffb640 1 -- 172.21.15.1:0/3513019444 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5000fe70 con 0x7f1b5c156ea0 2024-03-06T07:23:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.793+0000 7f1b63671640 1 -- 172.21.15.1:0/3513019444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5c0c60c0 con 0x7f1b5c156ea0 2024-03-06T07:23:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.794+0000 7f1b63671640 1 -- 172.21.15.1:0/3513019444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b5c067fd0 con 0x7f1b5c156ea0 2024-03-06T07:23:35.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.794+0000 7f1b59ffb640 1 -- 172.21.15.1:0/3513019444 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b50015460 con 0x7f1b5c156ea0 2024-03-06T07:23:35.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.794+0000 7f1b59ffb640 1 --2- 172.21.15.1:0/3513019444 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1b4806b4d0 0x7f1b4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.795+0000 7f1b59ffb640 1 -- 172.21.15.1:0/3513019444 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f1b5008a880 con 0x7f1b5c156ea0 2024-03-06T07:23:35.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.795+0000 7f1b6140d640 1 --2- 172.21.15.1:0/3513019444 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1b4806b4d0 0x7f1b4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.795+0000 7f1b6140d640 1 --2- 172.21.15.1:0/3513019444 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1b4806b4d0 0x7f1b4806d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1b54034000 tx=0x7f1b54007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.799+0000 7f1b59ffb640 1 -- 172.21.15.1:0/3513019444 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b5c158680 con 0x7f1b5c156ea0 2024-03-06T07:23:35.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.980+0000 7f1b63671640 1 -- 172.21.15.1:0/3513019444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1b5c0b06f0 con 0x7f1b5c156ea0 2024-03-06T07:23:35.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.982+0000 7f1b59ffb640 1 -- 172.21.15.1:0/3513019444 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v31)=0 v31) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1b50019030 con 0x7f1b5c156ea0 2024-03-06T07:23:35.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.984+0000 7f1b63671640 1 -- 172.21.15.1:0/3513019444 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1b4806b4d0 msgr2=0x7f1b4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.984+0000 7f1b63671640 1 --2- 172.21.15.1:0/3513019444 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1b4806b4d0 0x7f1b4806d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1b54034000 tx=0x7f1b54007000 comp rx=0 tx=0).stop 2024-03-06T07:23:35.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.985+0000 7f1b63671640 1 -- 172.21.15.1:0/3513019444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b5c156ea0 msgr2=0x7f1b5c1edf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.985+0000 7f1b63671640 1 --2- 172.21.15.1:0/3513019444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b5c156ea0 0x7f1b5c1edf40 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f1b50011600 tx=0x7f1b50011ac0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.985+0000 7f1b63671640 1 -- 172.21.15.1:0/3513019444 shutdown_connections 2024-03-06T07:23:35.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.985+0000 7f1b63671640 1 --2- 172.21.15.1:0/3513019444 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1b4806b4d0 0x7f1b4806d990 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.985+0000 7f1b63671640 1 --2- 172.21.15.1:0/3513019444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c157ff0 0x7f1b5c1ee480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.985+0000 7f1b63671640 1 --2- 172.21.15.1:0/3513019444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b5c156ea0 0x7f1b5c1edf40 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.985+0000 7f1b63671640 1 --2- 172.21.15.1:0/3513019444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b440039b0 0x7f1b5c0c5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.985+0000 7f1b63671640 1 -- 172.21.15.1:0/3513019444 >> 172.21.15.1:0/3513019444 conn(0x7f1b5c066ee0 msgr2=0x7f1b5c0bfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.985+0000 7f1b63671640 1 -- 172.21.15.1:0/3513019444 shutdown_connections 2024-03-06T07:23:35.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:35.985+0000 7f1b63671640 1 -- 172.21.15.1:0/3513019444 wait complete. 2024-03-06T07:23:35.997 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:35.997 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:36.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:36.079 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:36.079 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:23:36.080 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:36.094 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:23:36.094 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:23:36.094 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:36.137 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:23:36.138 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:23:36.138 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:36.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.237+0000 7f9b28847640 1 Processor -- start 2024-03-06T07:23:36.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.237+0000 7f9b28847640 1 -- start start 2024-03-06T07:23:36.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b28847640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9b240c1390 0x7f9b240c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b28847640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b241552e0 0x7f9b24155750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b28847640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b241588e0 con 0x7f9b24152730 2024-03-06T07:23:36.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b28847640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b240c1d60 con 0x7f9b240c1390 2024-03-06T07:23:36.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b28847640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b24158bc0 con 0x7f9b241552e0 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b2259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9b24152730 0x7f9b240c6a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54754/0 (socket says 172.21.15.1:54754) 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b2259c640 1 -- 172.21.15.1:0/219045017 learned_addr learned my addr 172.21.15.1:0/219045017 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b22d9d640 1 --2- 172.21.15.1:0/219045017 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b241552e0 0x7f9b24155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b1bfff640 1 --2- 172.21.15.1:0/219045017 >> v2:172.21.15.98:3300/0 conn(0x7f9b240c1390 0x7f9b240c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b21d9b640 1 -- 172.21.15.1:0/219045017 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132482325 0 0) 0x7f9b241588e0 con 0x7f9b24152730 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b21d9b640 1 -- 172.21.15.1:0/219045017 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b10003520 con 0x7f9b24152730 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b21d9b640 1 -- 172.21.15.1:0/219045017 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3236998753 0 0) 0x7f9b10003520 con 0x7f9b24152730 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b21d9b640 1 -- 172.21.15.1:0/219045017 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b241552e0 msgr2=0x7f9b24155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b21d9b640 1 --2- 172.21.15.1:0/219045017 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b241552e0 0x7f9b24155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b21d9b640 1 -- 172.21.15.1:0/219045017 >> v2:172.21.15.98:3300/0 conn(0x7f9b240c1390 msgr2=0x7f9b240c1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b21d9b640 1 --2- 172.21.15.1:0/219045017 >> v2:172.21.15.98:3300/0 conn(0x7f9b240c1390 0x7f9b240c1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b21d9b640 1 -- 172.21.15.1:0/219045017 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b24158ea0 con 0x7f9b24152730 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b21d9b640 1 -- 172.21.15.1:0/219045017 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f9b14001690 con 0x7f9b24152730 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b21d9b640 1 -- 172.21.15.1:0/219045017 >> v1:172.21.15.1:6789/0 conn(0x7f9b24152730 legacy=0x7f9b240c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b21d9b640 1 --2- 172.21.15.1:0/219045017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b100039b0 0x7f9b10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b21d9b640 1 --2- 172.21.15.1:0/219045017 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b241552e0 0x7f9b10004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b21d9b640 1 --2- 172.21.15.1:0/219045017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b10004d40 0x7f9b100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b21d9b640 1 -- 172.21.15.1:0/219045017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b240c1d60 con 0x7f9b10004d40 2024-03-06T07:23:36.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b21d9b640 1 -- 172.21.15.1:0/219045017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b24158bc0 con 0x7f9b100039b0 2024-03-06T07:23:36.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.238+0000 7f9b21d9b640 1 -- 172.21.15.1:0/219045017 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1000ba40 con 0x7f9b241552e0 2024-03-06T07:23:36.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.239+0000 7f9b22d9d640 1 --2- 172.21.15.1:0/219045017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b10004d40 0x7f9b100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.239+0000 7f9b1bfff640 1 --2- 172.21.15.1:0/219045017 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b241552e0 0x7f9b10004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.239+0000 7f9b2259c640 1 --2- 172.21.15.1:0/219045017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b100039b0 0x7f9b10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.239+0000 7f9b22d9d640 1 -- 172.21.15.1:0/219045017 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b241552e0 msgr2=0x7f9b10004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:36.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.239+0000 7f9b22d9d640 1 --2- 172.21.15.1:0/219045017 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b241552e0 0x7f9b10004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.239+0000 7f9b22d9d640 1 -- 172.21.15.1:0/219045017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b100039b0 msgr2=0x7f9b10003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:36.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.239+0000 7f9b22d9d640 1 --2- 172.21.15.1:0/219045017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b100039b0 0x7f9b10003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.239+0000 7f9b22d9d640 1 -- 172.21.15.1:0/219045017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b10003520 con 0x7f9b10004d40 2024-03-06T07:23:36.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.240+0000 7f9b22d9d640 1 --2- 172.21.15.1:0/219045017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b10004d40 0x7f9b100051c0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c009fc0 tx=0x7f9b10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8776a687df85d029 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.240+0000 7f9b21d9b640 1 -- 172.21.15.1:0/219045017 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c039070 con 0x7f9b10004d40 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.240+0000 7f9b21d9b640 1 -- 172.21.15.1:0/219045017 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1c02cc20 con 0x7f9b10004d40 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.240+0000 7f9b21d9b640 1 -- 172.21.15.1:0/219045017 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c009c10 con 0x7f9b10004d40 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.240+0000 7f9b28847640 1 -- 172.21.15.1:0/219045017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b10004d40 msgr2=0x7f9b100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.240+0000 7f9b28847640 1 --2- 172.21.15.1:0/219045017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b10004d40 0x7f9b100051c0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c009fc0 tx=0x7f9b10005830 comp rx=0 tx=0).stop 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.240+0000 7f9b22d9d640 1 -- 172.21.15.1:0/219045017 reap_dead start 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.240+0000 7f9b28847640 1 -- 172.21.15.1:0/219045017 shutdown_connections 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.240+0000 7f9b28847640 1 -- 172.21.15.1:0/219045017 >> 172.21.15.1:0/219045017 conn(0x7f9b24066d00 msgr2=0x7f9b24067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.240+0000 7f9b28847640 1 -- 172.21.15.1:0/219045017 shutdown_connections 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.240+0000 7f9b28847640 1 -- 172.21.15.1:0/219045017 wait complete. 2024-03-06T07:23:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.242+0000 7f9b28847640 1 Processor -- start 2024-03-06T07:23:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.242+0000 7f9b28847640 1 -- start start 2024-03-06T07:23:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.242+0000 7f9b28847640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b241eabe0 0x7f9b24157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.242+0000 7f9b28847640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b240aa6f0 0x7f9b241eafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.242+0000 7f9b28847640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b240ab850 0x7f9b240c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.242+0000 7f9b28847640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b24152b10 con 0x7f9b241eabe0 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.242+0000 7f9b28847640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b24159460 con 0x7f9b240aa6f0 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.242+0000 7f9b28847640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b240c71a0 con 0x7f9b240ab850 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.242+0000 7f9b2259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b241eabe0 0x7f9b24157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.242+0000 7f9b2259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b241eabe0 0x7f9b24157500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49872/0 (socket says 172.21.15.1:49872) 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.242+0000 7f9b2259c640 1 -- 172.21.15.1:0/3543385732 learned_addr learned my addr 172.21.15.1:0/3543385732 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.243+0000 7f9b22d9d640 1 --2- 172.21.15.1:0/3543385732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b240ab850 0x7f9b240c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.243+0000 7f9b1bfff640 1 --2- 172.21.15.1:0/3543385732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b240aa6f0 0x7f9b241eafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.243+0000 7f9b2259c640 1 -- 172.21.15.1:0/3543385732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b240ab850 msgr2=0x7f9b240c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.243+0000 7f9b2259c640 1 --2- 172.21.15.1:0/3543385732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b240ab850 0x7f9b240c8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.243+0000 7f9b2259c640 1 -- 172.21.15.1:0/3543385732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b240aa6f0 msgr2=0x7f9b241eafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.243+0000 7f9b2259c640 1 --2- 172.21.15.1:0/3543385732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b240aa6f0 0x7f9b241eafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.243+0000 7f9b2259c640 1 -- 172.21.15.1:0/3543385732 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b240abd00 con 0x7f9b241eabe0 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.243+0000 7f9b1bfff640 1 --2- 172.21.15.1:0/3543385732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b240aa6f0 0x7f9b241eafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:36.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.243+0000 7f9b2259c640 1 --2- 172.21.15.1:0/3543385732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b241eabe0 0x7f9b24157500 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f9b140072f0 tx=0x7f9b140077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.243+0000 7f9b1affd640 1 -- 172.21.15.1:0/3543385732 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1401a070 con 0x7f9b241eabe0 2024-03-06T07:23:36.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.243+0000 7f9b1affd640 1 -- 172.21.15.1:0/3543385732 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1400f960 con 0x7f9b241eabe0 2024-03-06T07:23:36.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.243+0000 7f9b28847640 1 -- 172.21.15.1:0/3543385732 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b24158150 con 0x7f9b241eabe0 2024-03-06T07:23:36.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.243+0000 7f9b1affd640 1 -- 172.21.15.1:0/3543385732 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1400fd50 con 0x7f9b241eabe0 2024-03-06T07:23:36.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.243+0000 7f9b28847640 1 -- 172.21.15.1:0/3543385732 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b24156d00 con 0x7f9b241eabe0 2024-03-06T07:23:36.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.244+0000 7f9b28847640 1 -- 172.21.15.1:0/3543385732 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b24067fd0 con 0x7f9b241eabe0 2024-03-06T07:23:36.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.244+0000 7f9b1affd640 1 -- 172.21.15.1:0/3543385732 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b1401d360 con 0x7f9b241eabe0 2024-03-06T07:23:36.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.245+0000 7f9b1affd640 1 --2- 172.21.15.1:0/3543385732 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9b0406b500 0x7f9b0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.245+0000 7f9b1affd640 1 -- 172.21.15.1:0/3543385732 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f9b14089ba0 con 0x7f9b241eabe0 2024-03-06T07:23:36.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.245+0000 7f9b1bfff640 1 --2- 172.21.15.1:0/3543385732 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9b0406b500 0x7f9b0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.246+0000 7f9b1bfff640 1 --2- 172.21.15.1:0/3543385732 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9b0406b500 0x7f9b0406d9c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9b240bfed0 tx=0x7f9b0c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.249+0000 7f9b1affd640 1 -- 172.21.15.1:0/3543385732 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b14058a90 con 0x7f9b241eabe0 2024-03-06T07:23:36.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.469+0000 7f9b28847640 1 -- 172.21.15.1:0/3543385732 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b240c6870 con 0x7f9b241eabe0 2024-03-06T07:23:36.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.470+0000 7f9b1affd640 1 -- 172.21.15.1:0/3543385732 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+3693 (secure 0 0 0) 0x7f9b14058390 con 0x7f9b241eabe0 2024-03-06T07:23:36.472 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:36.472 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":73,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.98:6836/4069609682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4069609682},{"type":"v1","addr":"172.21.15.98:6836","nonce":4069609682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":6196,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3962693342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3962693342},{"type":"v1","addr":"172.21.15.1:6834","nonce":3962693342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6211,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3130835916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3130835916},{"type":"v1","addr":"172.21.15.98:6837","nonce":3130835916}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":6888,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/3853788153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":3853788153},{"type":"v1","addr":"172.21.15.1:6835","nonce":3853788153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[]} 2024-03-06T07:23:36.473 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 73 2024-03-06T07:23:36.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.472+0000 7f9b28847640 1 -- 172.21.15.1:0/3543385732 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9b0406b500 msgr2=0x7f9b0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.472+0000 7f9b28847640 1 --2- 172.21.15.1:0/3543385732 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9b0406b500 0x7f9b0406d9c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9b240bfed0 tx=0x7f9b0c008040 comp rx=0 tx=0).stop 2024-03-06T07:23:36.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.472+0000 7f9b28847640 1 -- 172.21.15.1:0/3543385732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b241eabe0 msgr2=0x7f9b24157500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.472+0000 7f9b28847640 1 --2- 172.21.15.1:0/3543385732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b241eabe0 0x7f9b24157500 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f9b140072f0 tx=0x7f9b140077b0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.472+0000 7f9b28847640 1 -- 172.21.15.1:0/3543385732 shutdown_connections 2024-03-06T07:23:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.472+0000 7f9b28847640 1 --2- 172.21.15.1:0/3543385732 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9b0406b500 0x7f9b0406d9c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.472+0000 7f9b28847640 1 --2- 172.21.15.1:0/3543385732 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9b240ab850 0x7f9b240c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.472+0000 7f9b28847640 1 --2- 172.21.15.1:0/3543385732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b240aa6f0 0x7f9b241eafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.472+0000 7f9b28847640 1 --2- 172.21.15.1:0/3543385732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b241eabe0 0x7f9b24157500 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.472+0000 7f9b28847640 1 -- 172.21.15.1:0/3543385732 >> 172.21.15.1:0/3543385732 conn(0x7f9b24066d00 msgr2=0x7f9b240b0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.472+0000 7f9b28847640 1 -- 172.21.15.1:0/3543385732 shutdown_connections 2024-03-06T07:23:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.472+0000 7f9b28847640 1 -- 172.21.15.1:0/3543385732 wait complete. 2024-03-06T07:23:36.485 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:23:36.486 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:23:36.486 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:23:36.486 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:36.486 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:23:36.487 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:23:36.487 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:23:36.487 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:36.487 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:23:36.487 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:23:36.487 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:23:36.487 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:36.488 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:23:36.488 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:23:36.488 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:23:36.488 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:36.496 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:23:36.497 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:23:36.498 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:23:36.498 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:23:36.499 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:23:36.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:23:36.500 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:23:36.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:23:36.539 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:23:36.539 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:36.540 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:23:36.540 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:36.542 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:23:36.542 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:36.543 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:23:36.543 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:36.543 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:23:36.616 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:23:36.616 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:23:36.625 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:23:36.625 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:23:36.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.649+0000 7f2e8e876640 1 Processor -- start 2024-03-06T07:23:36.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.649+0000 7f2e8e876640 1 -- start start 2024-03-06T07:23:36.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.650+0000 7f2e8e876640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e88152820 0x7f2e880c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.650+0000 7f2e8e876640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2e880c0670 0x7f2e880c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.650+0000 7f2e8e876640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e880c7930 0x7f2e88158830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.650+0000 7f2e8e876640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e880c7180 con 0x7f2e88152820 2024-03-06T07:23:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.650+0000 7f2e8e876640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e88155460 con 0x7f2e880c0670 2024-03-06T07:23:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.650+0000 7f2e8e876640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e880c6bc0 con 0x7f2e880c7930 2024-03-06T07:23:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.650+0000 7f2e87fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e88152820 0x7f2e880c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.650+0000 7f2e87fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e88152820 0x7f2e880c0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49908/0 (socket says 172.21.15.1:49908) 2024-03-06T07:23:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.650+0000 7f2e87fff640 1 -- 172.21.15.1:0/3834932972 learned_addr learned my addr 172.21.15.1:0/3834932972 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.650+0000 7f2e8ce13640 1 --2- 172.21.15.1:0/3834932972 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e880c7930 0x7f2e88158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.650+0000 7f2e877fe640 1 --2- 172.21.15.1:0/3834932972 >> v2:172.21.15.98:3300/0 conn(0x7f2e880c0670 0x7f2e880c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.650+0000 7f2e87fff640 1 -- 172.21.15.1:0/3834932972 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e880c7930 msgr2=0x7f2e88158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.650+0000 7f2e87fff640 1 --2- 172.21.15.1:0/3834932972 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e880c7930 0x7f2e88158830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.650+0000 7f2e87fff640 1 -- 172.21.15.1:0/3834932972 >> v2:172.21.15.98:3300/0 conn(0x7f2e880c0670 msgr2=0x7f2e880c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.650+0000 7f2e87fff640 1 --2- 172.21.15.1:0/3834932972 >> v2:172.21.15.98:3300/0 conn(0x7f2e880c0670 0x7f2e880c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.650+0000 7f2e87fff640 1 -- 172.21.15.1:0/3834932972 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e880c6ea0 con 0x7f2e88152820 2024-03-06T07:23:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.651+0000 7f2e87fff640 1 --2- 172.21.15.1:0/3834932972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e88152820 0x7f2e880c0130 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f2e8806cca0 tx=0x7f2e7802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e07a8d8f9b372a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.651+0000 7f2e86ffd640 1 -- 172.21.15.1:0/3834932972 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e78035020 con 0x7f2e88152820 2024-03-06T07:23:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.651+0000 7f2e86ffd640 1 -- 172.21.15.1:0/3834932972 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e78031870 con 0x7f2e88152820 2024-03-06T07:23:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.651+0000 7f2e86ffd640 1 -- 172.21.15.1:0/3834932972 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e78031c40 con 0x7f2e88152820 2024-03-06T07:23:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.652+0000 7f2e8e876640 1 -- 172.21.15.1:0/3834932972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e88152820 msgr2=0x7f2e880c0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.652+0000 7f2e8e876640 1 --2- 172.21.15.1:0/3834932972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e88152820 0x7f2e880c0130 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f2e8806cca0 tx=0x7f2e7802b910 comp rx=0 tx=0).stop 2024-03-06T07:23:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.652+0000 7f2e8e876640 1 -- 172.21.15.1:0/3834932972 shutdown_connections 2024-03-06T07:23:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.652+0000 7f2e8e876640 1 --2- 172.21.15.1:0/3834932972 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e880c7930 0x7f2e88158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.652+0000 7f2e8e876640 1 --2- 172.21.15.1:0/3834932972 >> v2:172.21.15.98:3300/0 conn(0x7f2e880c0670 0x7f2e880c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.652+0000 7f2e8e876640 1 --2- 172.21.15.1:0/3834932972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e88152820 0x7f2e880c0130 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.652+0000 7f2e8e876640 1 -- 172.21.15.1:0/3834932972 >> 172.21.15.1:0/3834932972 conn(0x7f2e88072720 msgr2=0x7f2e88072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.652+0000 7f2e8e876640 1 -- 172.21.15.1:0/3834932972 shutdown_connections 2024-03-06T07:23:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.652+0000 7f2e8e876640 1 -- 172.21.15.1:0/3834932972 wait complete. 2024-03-06T07:23:36.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.653+0000 7f2e8e876640 1 Processor -- start 2024-03-06T07:23:36.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.653+0000 7f2e8e876640 1 -- start start 2024-03-06T07:23:36.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.654+0000 7f2e8e876640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e880c0670 0x7f2e881eba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.654+0000 7f2e8e876640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e880c7930 0x7f2e88148e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.654+0000 7f2e8e876640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e88152820 0x7f2e88156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.654+0000 7f2e8e876640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e88152d80 con 0x7f2e88152820 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.654+0000 7f2e8e876640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e88152c00 con 0x7f2e880c0670 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.654+0000 7f2e8e876640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e880af6f0 con 0x7f2e880c7930 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.654+0000 7f2e8ce13640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e88152820 0x7f2e88156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.654+0000 7f2e8ce13640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e88152820 0x7f2e88156960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49914/0 (socket says 172.21.15.1:49914) 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.654+0000 7f2e8ce13640 1 -- 172.21.15.1:0/844402801 learned_addr learned my addr 172.21.15.1:0/844402801 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.654+0000 7f2e877fe640 1 --2- 172.21.15.1:0/844402801 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e880c7930 0x7f2e88148e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.654+0000 7f2e87fff640 1 --2- 172.21.15.1:0/844402801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e880c0670 0x7f2e881eba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.654+0000 7f2e8ce13640 1 -- 172.21.15.1:0/844402801 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e880c7930 msgr2=0x7f2e88148e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.654+0000 7f2e8ce13640 1 --2- 172.21.15.1:0/844402801 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e880c7930 0x7f2e88148e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.654+0000 7f2e8ce13640 1 -- 172.21.15.1:0/844402801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e880c0670 msgr2=0x7f2e881eba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.654+0000 7f2e8ce13640 1 --2- 172.21.15.1:0/844402801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e880c0670 0x7f2e881eba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.654+0000 7f2e8ce13640 1 -- 172.21.15.1:0/844402801 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e881e2000 con 0x7f2e88152820 2024-03-06T07:23:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.654+0000 7f2e8ce13640 1 --2- 172.21.15.1:0/844402801 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e88152820 0x7f2e88156960 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f2e80009dd0 tx=0x7f2e8000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.655+0000 7f2e84ff9640 1 -- 172.21.15.1:0/844402801 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e80018020 con 0x7f2e88152820 2024-03-06T07:23:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.655+0000 7f2e84ff9640 1 -- 172.21.15.1:0/844402801 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e80014480 con 0x7f2e88152820 2024-03-06T07:23:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.655+0000 7f2e8e876640 1 -- 172.21.15.1:0/844402801 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e881e22e0 con 0x7f2e88152820 2024-03-06T07:23:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.655+0000 7f2e84ff9640 1 -- 172.21.15.1:0/844402801 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e80014870 con 0x7f2e88152820 2024-03-06T07:23:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.655+0000 7f2e8e876640 1 -- 172.21.15.1:0/844402801 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e88153220 con 0x7f2e88152820 2024-03-06T07:23:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.656+0000 7f2e6e7fc640 1 -- 172.21.15.1:0/844402801 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e60005290 con 0x7f2e88152820 2024-03-06T07:23:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.656+0000 7f2e84ff9640 1 -- 172.21.15.1:0/844402801 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e8001b020 con 0x7f2e88152820 2024-03-06T07:23:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.657+0000 7f2e84ff9640 1 --2- 172.21.15.1:0/844402801 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e7406b4d0 0x7f2e7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.657+0000 7f2e84ff9640 1 -- 172.21.15.1:0/844402801 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f2e8008af90 con 0x7f2e88152820 2024-03-06T07:23:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.657+0000 7f2e87fff640 1 --2- 172.21.15.1:0/844402801 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e7406b4d0 0x7f2e7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.657+0000 7f2e87fff640 1 --2- 172.21.15.1:0/844402801 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e7406b4d0 0x7f2e7406d990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2e78036000 tx=0x7f2e7803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.661+0000 7f2e84ff9640 1 -- 172.21.15.1:0/844402801 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e80022020 con 0x7f2e88152820 2024-03-06T07:23:36.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:36.859+0000 7f2e6e7fc640 1 -- 172.21.15.1:0/844402801 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2e60005580 con 0x7f2e88152820 2024-03-06T07:23:37.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.689+0000 7f2e84ff9640 1 -- 172.21.15.1:0/844402801 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v91) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2e80010040 con 0x7f2e88152820 2024-03-06T07:23:37.691 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-06T07:23:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.692+0000 7f2e6e7fc640 1 -- 172.21.15.1:0/844402801 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e7406b4d0 msgr2=0x7f2e7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.692+0000 7f2e6e7fc640 1 --2- 172.21.15.1:0/844402801 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e7406b4d0 0x7f2e7406d990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2e78036000 tx=0x7f2e7803c040 comp rx=0 tx=0).stop 2024-03-06T07:23:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.692+0000 7f2e6e7fc640 1 -- 172.21.15.1:0/844402801 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e88152820 msgr2=0x7f2e88156960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.692+0000 7f2e6e7fc640 1 --2- 172.21.15.1:0/844402801 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e88152820 0x7f2e88156960 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f2e80009dd0 tx=0x7f2e8000c490 comp rx=0 tx=0).stop 2024-03-06T07:23:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.692+0000 7f2e6e7fc640 1 -- 172.21.15.1:0/844402801 shutdown_connections 2024-03-06T07:23:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.692+0000 7f2e6e7fc640 1 --2- 172.21.15.1:0/844402801 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e7406b4d0 0x7f2e7406d990 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.692+0000 7f2e6e7fc640 1 --2- 172.21.15.1:0/844402801 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e88152820 0x7f2e88156960 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.692+0000 7f2e6e7fc640 1 --2- 172.21.15.1:0/844402801 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e880c7930 0x7f2e88148e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.692+0000 7f2e6e7fc640 1 --2- 172.21.15.1:0/844402801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e880c0670 0x7f2e881eba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.692+0000 7f2e6e7fc640 1 -- 172.21.15.1:0/844402801 >> 172.21.15.1:0/844402801 conn(0x7f2e88072720 msgr2=0x7f2e8806cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.692+0000 7f2e6e7fc640 1 -- 172.21.15.1:0/844402801 shutdown_connections 2024-03-06T07:23:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.692+0000 7f2e6e7fc640 1 -- 172.21.15.1:0/844402801 wait complete. 2024-03-06T07:23:37.708 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:23:37.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.814+0000 7f829b7d7640 1 Processor -- start 2024-03-06T07:23:37.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.814+0000 7f829b7d7640 1 -- start start 2024-03-06T07:23:37.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.815+0000 7f829b7d7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8294072bd0 0x7f82940c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.815+0000 7f829b7d7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8294152b70 0x7f82940c3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.815+0000 7f829b7d7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8294151dd0 con 0x7f82940c2b30 2024-03-06T07:23:37.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.815+0000 7f829b7d7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82941526e0 con 0x7f8294152b70 2024-03-06T07:23:37.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.815+0000 7f829b7d7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8294156190 con 0x7f8294072bd0 2024-03-06T07:23:37.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.816+0000 7f8299d74640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8294152b70 0x7f82940c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.816+0000 7f8298d72640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8294072bd0 0x7f82940c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.816+0000 7f8299573640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f82940c2b30 0x7f82940724c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:32848/0 (socket says 172.21.15.1:32848) 2024-03-06T07:23:37.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.816+0000 7f8299573640 1 -- 172.21.15.1:0/3993139389 learned_addr learned my addr 172.21.15.1:0/3993139389 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:37.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.816+0000 7f828bfff640 1 -- 172.21.15.1:0/3993139389 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563178137 0 0) 0x7f8294151dd0 con 0x7f82940c2b30 2024-03-06T07:23:37.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.816+0000 7f828bfff640 1 -- 172.21.15.1:0/3993139389 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8280003520 con 0x7f82940c2b30 2024-03-06T07:23:37.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.816+0000 7f8299d74640 1 -- 172.21.15.1:0/3993139389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8294072bd0 msgr2=0x7f82940c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:37.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.816+0000 7f8299d74640 1 --2- 172.21.15.1:0/3993139389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8294072bd0 0x7f82940c32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.816+0000 7f8299d74640 1 -- 172.21.15.1:0/3993139389 >> v1:172.21.15.98:6789/0 conn(0x7f82940c2b30 legacy=0x7f82940724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.816+0000 7f8299d74640 1 -- 172.21.15.1:0/3993139389 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8294152090 con 0x7f8294152b70 2024-03-06T07:23:37.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.816+0000 7f8298d72640 1 --2- 172.21.15.1:0/3993139389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8294072bd0 0x7f82940c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:37.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.817+0000 7f8299d74640 1 --2- 172.21.15.1:0/3993139389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8294152b70 0x7f82940c3830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8290009d90 tx=0x7f829002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f65d93cd19cbd25 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.817+0000 7f828bfff640 1 -- 172.21.15.1:0/3993139389 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8290039070 con 0x7f8294152b70 2024-03-06T07:23:37.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.817+0000 7f828bfff640 1 -- 172.21.15.1:0/3993139389 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829002cc50 con 0x7f8294152b70 2024-03-06T07:23:37.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.817+0000 7f828bfff640 1 -- 172.21.15.1:0/3993139389 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8290034440 con 0x7f8294152b70 2024-03-06T07:23:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.817+0000 7f829b7d7640 1 -- 172.21.15.1:0/3993139389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8294152b70 msgr2=0x7f82940c3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.817+0000 7f829b7d7640 1 --2- 172.21.15.1:0/3993139389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8294152b70 0x7f82940c3830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8290009d90 tx=0x7f829002bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.817+0000 7f829b7d7640 1 -- 172.21.15.1:0/3993139389 shutdown_connections 2024-03-06T07:23:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.817+0000 7f829b7d7640 1 --2- 172.21.15.1:0/3993139389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8294152b70 0x7f82940c3830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.817+0000 7f829b7d7640 1 --2- 172.21.15.1:0/3993139389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8294072bd0 0x7f82940c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.817+0000 7f829b7d7640 1 -- 172.21.15.1:0/3993139389 >> 172.21.15.1:0/3993139389 conn(0x7f82940c6860 msgr2=0x7f82940c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.818+0000 7f829b7d7640 1 -- 172.21.15.1:0/3993139389 shutdown_connections 2024-03-06T07:23:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.818+0000 7f829b7d7640 1 -- 172.21.15.1:0/3993139389 wait complete. 2024-03-06T07:23:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.819+0000 7f829b7d7640 1 Processor -- start 2024-03-06T07:23:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.819+0000 7f829b7d7640 1 -- start start 2024-03-06T07:23:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.819+0000 7f829b7d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8294072bd0 0x7f8294150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.819+0000 7f829b7d7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82940c2b30 0x7f82941535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.819+0000 7f829b7d7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8294152b70 0x7f8294153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.819+0000 7f829b7d7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82940c3ef0 con 0x7f82940c2b30 2024-03-06T07:23:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.819+0000 7f829b7d7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82940c3d70 con 0x7f8294072bd0 2024-03-06T07:23:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.819+0000 7f829b7d7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8294155d80 con 0x7f8294152b70 2024-03-06T07:23:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.819+0000 7f8298d72640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82940c2b30 0x7f82941535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.819+0000 7f8298d72640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82940c2b30 0x7f82941535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49924/0 (socket says 172.21.15.1:49924) 2024-03-06T07:23:37.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.819+0000 7f8298d72640 1 -- 172.21.15.1:0/1133354375 learned_addr learned my addr 172.21.15.1:0/1133354375 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:37.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.820+0000 7f8299d74640 1 --2- 172.21.15.1:0/1133354375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8294152b70 0x7f8294153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.820+0000 7f8299573640 1 --2- 172.21.15.1:0/1133354375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8294072bd0 0x7f8294150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.820+0000 7f8298d72640 1 -- 172.21.15.1:0/1133354375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8294152b70 msgr2=0x7f8294153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.820+0000 7f8298d72640 1 --2- 172.21.15.1:0/1133354375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8294152b70 0x7f8294153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.820+0000 7f8298d72640 1 -- 172.21.15.1:0/1133354375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8294072bd0 msgr2=0x7f8294150670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.820+0000 7f8298d72640 1 --2- 172.21.15.1:0/1133354375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8294072bd0 0x7f8294150670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.820+0000 7f8298d72640 1 -- 172.21.15.1:0/1133354375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82941eb750 con 0x7f82940c2b30 2024-03-06T07:23:37.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.820+0000 7f8298d72640 1 --2- 172.21.15.1:0/1133354375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82940c2b30 0x7f82941535a0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f828400e9e0 tx=0x7f828400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.820+0000 7f8289ffb640 1 -- 172.21.15.1:0/1133354375 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284017070 con 0x7f82940c2b30 2024-03-06T07:23:37.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.820+0000 7f8289ffb640 1 -- 172.21.15.1:0/1133354375 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828400ca30 con 0x7f82940c2b30 2024-03-06T07:23:37.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.820+0000 7f829b7d7640 1 -- 172.21.15.1:0/1133354375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8294145930 con 0x7f82940c2b30 2024-03-06T07:23:37.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.820+0000 7f8289ffb640 1 -- 172.21.15.1:0/1133354375 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828400ce20 con 0x7f82940c2b30 2024-03-06T07:23:37.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.820+0000 7f829b7d7640 1 -- 172.21.15.1:0/1133354375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8294154720 con 0x7f82940c2b30 2024-03-06T07:23:37.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.821+0000 7f829b7d7640 1 -- 172.21.15.1:0/1133354375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8294067fd0 con 0x7f82940c2b30 2024-03-06T07:23:37.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.822+0000 7f8289ffb640 1 -- 172.21.15.1:0/1133354375 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f828401a070 con 0x7f82940c2b30 2024-03-06T07:23:37.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.822+0000 7f8289ffb640 1 --2- 172.21.15.1:0/1133354375 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f827806b4d0 0x7f827806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.822+0000 7f8289ffb640 1 -- 172.21.15.1:0/1133354375 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8284089c20 con 0x7f82940c2b30 2024-03-06T07:23:37.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.823+0000 7f8299573640 1 --2- 172.21.15.1:0/1133354375 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f827806b4d0 0x7f827806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.823+0000 7f8299573640 1 --2- 172.21.15.1:0/1133354375 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f827806b4d0 0x7f827806d990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f828c007fd0 tx=0x7f828c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:37.827+0000 7f8289ffb640 1 -- 172.21.15.1:0/1133354375 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8284058c50 con 0x7f82940c2b30 2024-03-06T07:23:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.079+0000 7f829b7d7640 1 -- 172.21.15.1:0/1133354375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f82941531c0 con 0x7f82940c2b30 2024-03-06T07:23:38.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.079+0000 7f8289ffb640 1 -- 172.21.15.1:0/1133354375 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f8284058550 con 0x7f82940c2b30 2024-03-06T07:23:38.081 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:38.081 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:23:38.081 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-06T07:23:38.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:38.082 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-06T07:23:38.082 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlnxyWARAAKIPXU0IafNQ1RzBPX+3DYw==", 2024-03-06T07:23:38.082 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:23:38.082 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:23:38.082 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:23:38.082 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:23:38.082 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:23:38.082 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:38.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:38.082 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-06T07:23:38.082 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl2fXxDRAAuLqH+HpKcN1JTts9ZtDTXA==", 2024-03-06T07:23:38.083 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:23:38.083 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:23:38.083 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:23:38.083 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:23:38.083 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:23:38.083 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:38.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:38.083 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-06T07:23:38.083 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl+IzqCBAAmgLkarFGt9Ihs/8kJYicBQ==", 2024-03-06T07:23:38.083 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:23:38.083 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:23:38.083 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:23:38.083 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:23:38.084 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:23:38.084 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:38.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:38.084 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-06T07:23:38.084 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlipdRFRAAA34vrsTEoX3L1CKb1YAV9g==", 2024-03-06T07:23:38.084 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:23:38.084 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:23:38.084 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:23:38.084 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:23:38.084 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:23:38.084 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:38.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:38.085 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-06T07:23:38.085 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAGGehlQK39LxAAOpPIZvl3FqQwCgyo3yoOGQ==", 2024-03-06T07:23:38.085 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:23:38.085 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:38.085 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:23:38.085 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:23:38.085 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:38.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:38.085 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-06T07:23:38.085 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAJGehlTxwDLhAAMwN9E0+3Jh5XNXnLo+fw3g==", 2024-03-06T07:23:38.085 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:23:38.085 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:38.086 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:23:38.086 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:23:38.086 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:38.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:38.086 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-06T07:23:38.086 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAMGehl4qrlLBAAm0VWQRNvSQIpDx5xf4Fyig==", 2024-03-06T07:23:38.086 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:23:38.086 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:38.086 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:23:38.086 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:23:38.086 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:38.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:38.086 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-06T07:23:38.087 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAPGehllO5HKxAAozm6ycG+fRxatoZTogeJTA==", 2024-03-06T07:23:38.087 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:23:38.087 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:38.087 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:23:38.087 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:23:38.087 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:38.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:38.087 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-06T07:23:38.087 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAVGehlLtLBMhAAUrV6q6ZCQluIcoo/S/NdbA==", 2024-03-06T07:23:38.087 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:23:38.087 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:38.087 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:23:38.088 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:23:38.088 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:38.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:38.088 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-06T07:23:38.088 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAYGehlHp59MRAAxxgpSPTEZkYKjl3fr0xeeQ==", 2024-03-06T07:23:38.088 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:23:38.088 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:38.088 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:23:38.088 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:23:38.088 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:38.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:38.088 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-06T07:23:38.089 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAbGehlTnXcMBAAFOBqcnh5ubAz+o6AGjccWQ==", 2024-03-06T07:23:38.089 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:23:38.089 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:38.089 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:23:38.089 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:23:38.089 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:38.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:38.089 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-06T07:23:38.089 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAeGehl1qH1LhAAaSl7Aj7qlN6d4xt5SKAmcg==", 2024-03-06T07:23:38.089 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:23:38.089 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:38.089 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:23:38.089 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:23:38.090 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:38.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:38.090 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-06T07:23:38.090 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng==", 2024-03-06T07:23:38.090 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:23:38.090 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:23:38.090 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-06T07:23:38.090 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:38.090 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:38.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:38.090 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-06T07:23:38.090 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlMyv7HBAAi7vt7aZWEaZuMpFrB5ofZw==", 2024-03-06T07:23:38.090 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:23:38.091 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:23:38.091 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-06T07:23:38.091 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-06T07:23:38.091 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-06T07:23:38.091 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:38.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:38.091 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-06T07:23:38.091 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlBS/vIBAAN86N2K4ze5PJB8kR869Lfg==", 2024-03-06T07:23:38.091 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:23:38.091 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:23:38.091 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:23:38.091 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:23:38.091 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:23:38.092 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:38.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:38.092 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-06T07:23:38.092 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlJ/jPORAAhLk5MTGafsGKhZSYhvhBZg==", 2024-03-06T07:23:38.092 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:23:38.092 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:23:38.092 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:38.092 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:23:38.092 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:23:38.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:38.092 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-06T07:23:38.092 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlAl8rNhAAvdU0Qvo/FJYnxc4/H8MIqQ==", 2024-03-06T07:23:38.093 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:23:38.093 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:23:38.093 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:38.093 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:23:38.093 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.093 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:38.093 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-06T07:23:38.093 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:23:38.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.081+0000 7f829b7d7640 1 -- 172.21.15.1:0/1133354375 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f827806b4d0 msgr2=0x7f827806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.081+0000 7f829b7d7640 1 --2- 172.21.15.1:0/1133354375 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f827806b4d0 0x7f827806d990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f828c007fd0 tx=0x7f828c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.081+0000 7f829b7d7640 1 -- 172.21.15.1:0/1133354375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82940c2b30 msgr2=0x7f82941535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.081+0000 7f829b7d7640 1 --2- 172.21.15.1:0/1133354375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82940c2b30 0x7f82941535a0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f828400e9e0 tx=0x7f828400eea0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.081+0000 7f829b7d7640 1 -- 172.21.15.1:0/1133354375 shutdown_connections 2024-03-06T07:23:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.081+0000 7f829b7d7640 1 --2- 172.21.15.1:0/1133354375 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f827806b4d0 0x7f827806d990 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.081+0000 7f829b7d7640 1 --2- 172.21.15.1:0/1133354375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8294152b70 0x7f8294153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.081+0000 7f829b7d7640 1 --2- 172.21.15.1:0/1133354375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82940c2b30 0x7f82941535a0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.081+0000 7f829b7d7640 1 --2- 172.21.15.1:0/1133354375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8294072bd0 0x7f8294150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.081+0000 7f829b7d7640 1 -- 172.21.15.1:0/1133354375 >> 172.21.15.1:0/1133354375 conn(0x7f82940c6860 msgr2=0x7f82940c3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.081+0000 7f829b7d7640 1 -- 172.21.15.1:0/1133354375 shutdown_connections 2024-03-06T07:23:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.081+0000 7f829b7d7640 1 -- 172.21.15.1:0/1133354375 wait complete. 2024-03-06T07:23:38.096 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:38.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.195+0000 7fdd8da17640 1 Processor -- start 2024-03-06T07:23:38.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.195+0000 7fdd8da17640 1 -- start start 2024-03-06T07:23:38.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.195+0000 7fdd8da17640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd880c2b30 0x7fdd880724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.195+0000 7fdd8da17640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdd88152b70 0x7fdd88072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.196+0000 7fdd8da17640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdd880a5fa0 0x7fdd880c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.196+0000 7fdd8da17640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd881526e0 con 0x7fdd880a5fa0 2024-03-06T07:23:38.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.196+0000 7fdd8da17640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd88151dd0 con 0x7fdd88152b70 2024-03-06T07:23:38.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.196+0000 7fdd8da17640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd88156190 con 0x7fdd880c2b30 2024-03-06T07:23:38.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.196+0000 7fdd877fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdd880a5fa0 0x7fdd880c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.196+0000 7fdd877fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdd880a5fa0 0x7fdd880c32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49934/0 (socket says 172.21.15.1:49934) 2024-03-06T07:23:38.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.196+0000 7fdd877fe640 1 -- 172.21.15.1:0/622633632 learned_addr learned my addr 172.21.15.1:0/622633632 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:38.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.196+0000 7fdd86ffd640 1 --2- 172.21.15.1:0/622633632 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd880c2b30 0x7fdd880724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.196+0000 7fdd867fc640 1 --2- 172.21.15.1:0/622633632 >> v2:172.21.15.98:3300/0 conn(0x7fdd88152b70 0x7fdd88072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.196+0000 7fdd877fe640 1 -- 172.21.15.1:0/622633632 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd880c2b30 msgr2=0x7fdd880724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.196+0000 7fdd877fe640 1 --2- 172.21.15.1:0/622633632 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd880c2b30 0x7fdd880724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.196+0000 7fdd877fe640 1 -- 172.21.15.1:0/622633632 >> v2:172.21.15.98:3300/0 conn(0x7fdd88152b70 msgr2=0x7fdd88072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.196+0000 7fdd877fe640 1 --2- 172.21.15.1:0/622633632 >> v2:172.21.15.98:3300/0 conn(0x7fdd88152b70 0x7fdd88072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.196+0000 7fdd877fe640 1 -- 172.21.15.1:0/622633632 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd88152090 con 0x7fdd880a5fa0 2024-03-06T07:23:38.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.196+0000 7fdd867fc640 1 --2- 172.21.15.1:0/622633632 >> v2:172.21.15.98:3300/0 conn(0x7fdd88152b70 0x7fdd88072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:38.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.197+0000 7fdd877fe640 1 --2- 172.21.15.1:0/622633632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd880a5fa0 0x7fdd880c32f0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fdd8806fba0 tx=0x7fdd8002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84fb2d9ef91fada server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.197+0000 7fdd85ffb640 1 -- 172.21.15.1:0/622633632 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8002ca70 con 0x7fdd880a5fa0 2024-03-06T07:23:38.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.197+0000 7fdd85ffb640 1 -- 172.21.15.1:0/622633632 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8002cc10 con 0x7fdd880a5fa0 2024-03-06T07:23:38.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.197+0000 7fdd85ffb640 1 -- 172.21.15.1:0/622633632 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd800343f0 con 0x7fdd880a5fa0 2024-03-06T07:23:38.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.197+0000 7fdd8da17640 1 -- 172.21.15.1:0/622633632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd880a5fa0 msgr2=0x7fdd880c32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.197+0000 7fdd8da17640 1 --2- 172.21.15.1:0/622633632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd880a5fa0 0x7fdd880c32f0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fdd8806fba0 tx=0x7fdd8002bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.197+0000 7fdd8da17640 1 -- 172.21.15.1:0/622633632 shutdown_connections 2024-03-06T07:23:38.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.197+0000 7fdd8da17640 1 --2- 172.21.15.1:0/622633632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd880a5fa0 0x7fdd880c32f0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.197+0000 7fdd8da17640 1 --2- 172.21.15.1:0/622633632 >> v2:172.21.15.98:3300/0 conn(0x7fdd88152b70 0x7fdd88072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.197+0000 7fdd8da17640 1 --2- 172.21.15.1:0/622633632 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd880c2b30 0x7fdd880724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.197+0000 7fdd8da17640 1 -- 172.21.15.1:0/622633632 >> 172.21.15.1:0/622633632 conn(0x7fdd880c6860 msgr2=0x7fdd880c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.197+0000 7fdd8da17640 1 -- 172.21.15.1:0/622633632 shutdown_connections 2024-03-06T07:23:38.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.197+0000 7fdd8da17640 1 -- 172.21.15.1:0/622633632 wait complete. 2024-03-06T07:23:38.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.199+0000 7fdd8da17640 1 Processor -- start 2024-03-06T07:23:38.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.199+0000 7fdd8da17640 1 -- start start 2024-03-06T07:23:38.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.199+0000 7fdd8da17640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd880a5fa0 0x7fdd880ac440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.199+0000 7fdd8da17640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd880c2b30 0x7fdd880ab6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.199+0000 7fdd8da17640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd88152b70 0x7fdd880abbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.199+0000 7fdd8da17640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd88155f00 con 0x7fdd88152b70 2024-03-06T07:23:38.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.199+0000 7fdd8da17640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd88155d80 con 0x7fdd880c2b30 2024-03-06T07:23:38.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.199+0000 7fdd8da17640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd880c3830 con 0x7fdd880a5fa0 2024-03-06T07:23:38.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.199+0000 7fdd877fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd88152b70 0x7fdd880abbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.199+0000 7fdd877fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd88152b70 0x7fdd880abbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49948/0 (socket says 172.21.15.1:49948) 2024-03-06T07:23:38.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.199+0000 7fdd877fe640 1 -- 172.21.15.1:0/3351729190 learned_addr learned my addr 172.21.15.1:0/3351729190 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:38.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.200+0000 7fdd86ffd640 1 --2- 172.21.15.1:0/3351729190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd880a5fa0 0x7fdd880ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.200+0000 7fdd877fe640 1 -- 172.21.15.1:0/3351729190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd880a5fa0 msgr2=0x7fdd880ac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.200+0000 7fdd867fc640 1 --2- 172.21.15.1:0/3351729190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd880c2b30 0x7fdd880ab6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.200+0000 7fdd877fe640 1 --2- 172.21.15.1:0/3351729190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd880a5fa0 0x7fdd880ac440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.200+0000 7fdd877fe640 1 -- 172.21.15.1:0/3351729190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd880c2b30 msgr2=0x7fdd880ab6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.200+0000 7fdd877fe640 1 --2- 172.21.15.1:0/3351729190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd880c2b30 0x7fdd880ab6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.200+0000 7fdd877fe640 1 -- 172.21.15.1:0/3351729190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd881dd010 con 0x7fdd88152b70 2024-03-06T07:23:38.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.200+0000 7fdd877fe640 1 --2- 172.21.15.1:0/3351729190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd88152b70 0x7fdd880abbf0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fdd80036040 tx=0x7fdd80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.200+0000 7fdd6ffff640 1 -- 172.21.15.1:0/3351729190 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd80043020 con 0x7fdd88152b70 2024-03-06T07:23:38.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.200+0000 7fdd6ffff640 1 -- 172.21.15.1:0/3351729190 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8003e450 con 0x7fdd88152b70 2024-03-06T07:23:38.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.200+0000 7fdd8da17640 1 -- 172.21.15.1:0/3351729190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd880acb40 con 0x7fdd88152b70 2024-03-06T07:23:38.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.200+0000 7fdd6ffff640 1 -- 172.21.15.1:0/3351729190 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd80009510 con 0x7fdd88152b70 2024-03-06T07:23:38.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.200+0000 7fdd8da17640 1 -- 172.21.15.1:0/3351729190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd881ddf00 con 0x7fdd88152b70 2024-03-06T07:23:38.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.201+0000 7fdd8da17640 1 -- 172.21.15.1:0/3351729190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd8814c790 con 0x7fdd88152b70 2024-03-06T07:23:38.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.202+0000 7fdd6ffff640 1 -- 172.21.15.1:0/3351729190 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd80046020 con 0x7fdd88152b70 2024-03-06T07:23:38.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.202+0000 7fdd6ffff640 1 --2- 172.21.15.1:0/3351729190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd7406b4d0 0x7fdd7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.202+0000 7fdd6ffff640 1 -- 172.21.15.1:0/3351729190 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fdd800ac8c0 con 0x7fdd88152b70 2024-03-06T07:23:38.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.202+0000 7fdd86ffd640 1 --2- 172.21.15.1:0/3351729190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd7406b4d0 0x7fdd7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.203+0000 7fdd86ffd640 1 --2- 172.21.15.1:0/3351729190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd7406b4d0 0x7fdd7406d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdd78006fd0 tx=0x7fdd78008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.206+0000 7fdd6ffff640 1 -- 172.21.15.1:0/3351729190 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd8814c790 con 0x7fdd88152b70 2024-03-06T07:23:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.389+0000 7fdd8da17640 1 -- 172.21.15.1:0/3351729190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd88067fd0 con 0x7fdd88152b70 2024-03-06T07:23:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.390+0000 7fdd6ffff640 1 -- 172.21.15.1:0/3351729190 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+13045 (secure 0 0 0) 0x7fdd8004d030 con 0x7fdd88152b70 2024-03-06T07:23:38.392 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:38.392 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":91,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:23:37.683560+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.392+0000 7fdd8da17640 1 -- 172.21.15.1:0/3351729190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd7406b4d0 msgr2=0x7fdd7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.392+0000 7fdd8da17640 1 --2- 172.21.15.1:0/3351729190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd7406b4d0 0x7fdd7406d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdd78006fd0 tx=0x7fdd78008040 comp rx=0 tx=0).stop 2024-03-06T07:23:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.392+0000 7fdd8da17640 1 -- 172.21.15.1:0/3351729190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd88152b70 msgr2=0x7fdd880abbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.392+0000 7fdd8da17640 1 --2- 172.21.15.1:0/3351729190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd88152b70 0x7fdd880abbf0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fdd80036040 tx=0x7fdd80004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.392+0000 7fdd8da17640 1 -- 172.21.15.1:0/3351729190 shutdown_connections 2024-03-06T07:23:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.392+0000 7fdd8da17640 1 --2- 172.21.15.1:0/3351729190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd7406b4d0 0x7fdd7406d990 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.392+0000 7fdd8da17640 1 --2- 172.21.15.1:0/3351729190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd88152b70 0x7fdd880abbf0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.392+0000 7fdd8da17640 1 --2- 172.21.15.1:0/3351729190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd880c2b30 0x7fdd880ab6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.392+0000 7fdd8da17640 1 --2- 172.21.15.1:0/3351729190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd880a5fa0 0x7fdd880ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.392+0000 7fdd8da17640 1 -- 172.21.15.1:0/3351729190 >> 172.21.15.1:0/3351729190 conn(0x7fdd880c6860 msgr2=0x7fdd8814e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.392+0000 7fdd8da17640 1 -- 172.21.15.1:0/3351729190 shutdown_connections 2024-03-06T07:23:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.392+0000 7fdd8da17640 1 -- 172.21.15.1:0/3351729190 wait complete. 2024-03-06T07:23:38.405 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:38.405 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.507+0000 7fac435eb640 1 Processor -- start 2024-03-06T07:23:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.507+0000 7fac435eb640 1 -- start start 2024-03-06T07:23:38.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.508+0000 7fac435eb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fac3c0af6f0 0x7fac3c0c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.508+0000 7fac435eb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3c0c7180 con 0x7fac3c152820 2024-03-06T07:23:38.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.508+0000 7fac435eb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3c155460 con 0x7fac3c0c7930 2024-03-06T07:23:38.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.508+0000 7fac435eb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac3c0c6bc0 con 0x7fac3c0af6f0 2024-03-06T07:23:38.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.508+0000 7fac41387640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fac3c152820 0x7fac3c0c0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54766/0 (socket says 172.21.15.1:54766) 2024-03-06T07:23:38.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.508+0000 7fac41387640 1 -- 172.21.15.1:0/1955604061 learned_addr learned my addr 172.21.15.1:0/1955604061 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:38.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.508+0000 7fac3bfff640 1 -- 172.21.15.1:0/1955604061 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85152020 0 0) 0x7fac3c0c7180 con 0x7fac3c152820 2024-03-06T07:23:38.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.508+0000 7fac3bfff640 1 -- 172.21.15.1:0/1955604061 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac24003520 con 0x7fac3c152820 2024-03-06T07:23:38.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.508+0000 7fac40b86640 1 --2- 172.21.15.1:0/1955604061 >> v2:172.21.15.98:3300/0 conn(0x7fac3c0af6f0 0x7fac3c0c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac3bfff640 1 -- 172.21.15.1:0/1955604061 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4286733202 0 0) 0x7fac24003520 con 0x7fac3c152820 2024-03-06T07:23:38.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac3bfff640 1 -- 172.21.15.1:0/1955604061 >> v2:172.21.15.98:3300/0 conn(0x7fac3c0af6f0 msgr2=0x7fac3c0c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac3bfff640 1 --2- 172.21.15.1:0/1955604061 >> v2:172.21.15.98:3300/0 conn(0x7fac3c0af6f0 0x7fac3c0c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac3bfff640 1 -- 172.21.15.1:0/1955604061 >> v1:172.21.15.98:6789/0 conn(0x7fac3c0c7930 legacy=0x7fac3c15fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac3bfff640 1 -- 172.21.15.1:0/1955604061 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c0c6ea0 con 0x7fac3c152820 2024-03-06T07:23:38.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac3bfff640 1 -- 172.21.15.1:0/1955604061 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fac34001670 con 0x7fac3c152820 2024-03-06T07:23:38.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac3bfff640 1 -- 172.21.15.1:0/1955604061 >> v1:172.21.15.1:6789/0 conn(0x7fac3c152820 legacy=0x7fac3c0c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac3bfff640 1 --2- 172.21.15.1:0/1955604061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac240039b0 0x7fac24003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac3bfff640 1 --2- 172.21.15.1:0/1955604061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac24004800 0x7fac24004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac3bfff640 1 --2- 172.21.15.1:0/1955604061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac240051c0 0x7fac2400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac3bfff640 1 -- 172.21.15.1:0/1955604061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c0c6bc0 con 0x7fac24004800 2024-03-06T07:23:38.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac3bfff640 1 -- 172.21.15.1:0/1955604061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2400c060 con 0x7fac240051c0 2024-03-06T07:23:38.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac3bfff640 1 -- 172.21.15.1:0/1955604061 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2400c260 con 0x7fac240039b0 2024-03-06T07:23:38.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac40b86640 1 --2- 172.21.15.1:0/1955604061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac24004800 0x7fac24004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac41387640 1 --2- 172.21.15.1:0/1955604061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac240039b0 0x7fac24003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac41b88640 1 --2- 172.21.15.1:0/1955604061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac240051c0 0x7fac2400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac40b86640 1 -- 172.21.15.1:0/1955604061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac240039b0 msgr2=0x7fac24003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac40b86640 1 --2- 172.21.15.1:0/1955604061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac240039b0 0x7fac24003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac40b86640 1 -- 172.21.15.1:0/1955604061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac240051c0 msgr2=0x7fac2400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac40b86640 1 --2- 172.21.15.1:0/1955604061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac240051c0 0x7fac2400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac41b88640 1 -- 172.21.15.1:0/1955604061 reap_dead start 2024-03-06T07:23:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.509+0000 7fac40b86640 1 -- 172.21.15.1:0/1955604061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac24003520 con 0x7fac24004800 2024-03-06T07:23:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.510+0000 7fac40b86640 1 --2- 172.21.15.1:0/1955604061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac24004800 0x7fac24004c80 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fac30009ac0 tx=0x7fac24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f43af47ec6edf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.510+0000 7fac3bfff640 1 -- 172.21.15.1:0/1955604061 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30039070 con 0x7fac24004800 2024-03-06T07:23:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.510+0000 7fac3bfff640 1 -- 172.21.15.1:0/1955604061 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3002c6b0 con 0x7fac24004800 2024-03-06T07:23:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.510+0000 7fac3bfff640 1 -- 172.21.15.1:0/1955604061 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30009690 con 0x7fac24004800 2024-03-06T07:23:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.511+0000 7fac435eb640 1 -- 172.21.15.1:0/1955604061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac24004800 msgr2=0x7fac24004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.511+0000 7fac435eb640 1 --2- 172.21.15.1:0/1955604061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac24004800 0x7fac24004c80 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fac30009ac0 tx=0x7fac24005830 comp rx=0 tx=0).stop 2024-03-06T07:23:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.511+0000 7fac435eb640 1 -- 172.21.15.1:0/1955604061 shutdown_connections 2024-03-06T07:23:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.511+0000 7fac435eb640 1 --2- 172.21.15.1:0/1955604061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac24004800 0x7fac24004c80 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.511+0000 7fac435eb640 1 -- 172.21.15.1:0/1955604061 >> 172.21.15.1:0/1955604061 conn(0x7fac3c072720 msgr2=0x7fac3c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.511+0000 7fac435eb640 1 -- 172.21.15.1:0/1955604061 shutdown_connections 2024-03-06T07:23:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.511+0000 7fac435eb640 1 -- 172.21.15.1:0/1955604061 wait complete. 2024-03-06T07:23:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.512+0000 7fac435eb640 1 Processor -- start 2024-03-06T07:23:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.512+0000 7fac435eb640 1 -- start start 2024-03-06T07:23:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.512+0000 7fac435eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac24004800 0x7fac3c0ac010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.512+0000 7fac435eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3c0aee80 0x7fac3c14f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac435eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3c14f5e0 0x7fac3c1ea610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac435eb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c152d80 con 0x7fac3c0aee80 2024-03-06T07:23:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac435eb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c152c00 con 0x7fac3c14f5e0 2024-03-06T07:23:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac435eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c0afc40 con 0x7fac24004800 2024-03-06T07:23:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac40b86640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3c0aee80 0x7fac3c14f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac40b86640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3c0aee80 0x7fac3c14f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49968/0 (socket says 172.21.15.1:49968) 2024-03-06T07:23:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac40b86640 1 -- 172.21.15.1:0/566378585 learned_addr learned my addr 172.21.15.1:0/566378585 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac41387640 1 --2- 172.21.15.1:0/566378585 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac24004800 0x7fac3c0ac010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac41b88640 1 --2- 172.21.15.1:0/566378585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3c14f5e0 0x7fac3c1ea610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac40b86640 1 -- 172.21.15.1:0/566378585 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac24004800 msgr2=0x7fac3c0ac010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac40b86640 1 --2- 172.21.15.1:0/566378585 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac24004800 0x7fac3c0ac010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac40b86640 1 -- 172.21.15.1:0/566378585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3c14f5e0 msgr2=0x7fac3c1ea610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac40b86640 1 --2- 172.21.15.1:0/566378585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3c14f5e0 0x7fac3c1ea610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac40b86640 1 -- 172.21.15.1:0/566378585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c0ad780 con 0x7fac3c0aee80 2024-03-06T07:23:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac40b86640 1 --2- 172.21.15.1:0/566378585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3c0aee80 0x7fac3c14f0a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fac30037040 tx=0x7fac30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac39ffb640 1 -- 172.21.15.1:0/566378585 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30042020 con 0x7fac3c0aee80 2024-03-06T07:23:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac39ffb640 1 -- 172.21.15.1:0/566378585 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3003a040 con 0x7fac3c0aee80 2024-03-06T07:23:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac435eb640 1 -- 172.21.15.1:0/566378585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac3c1edb80 con 0x7fac3c0aee80 2024-03-06T07:23:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.513+0000 7fac39ffb640 1 -- 172.21.15.1:0/566378585 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30039070 con 0x7fac3c0aee80 2024-03-06T07:23:38.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.514+0000 7fac435eb640 1 -- 172.21.15.1:0/566378585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac3c1ee140 con 0x7fac3c0aee80 2024-03-06T07:23:38.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.515+0000 7fac435eb640 1 -- 172.21.15.1:0/566378585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac3c0686e0 con 0x7fac3c0aee80 2024-03-06T07:23:38.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.515+0000 7fac39ffb640 1 -- 172.21.15.1:0/566378585 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac30045020 con 0x7fac3c0aee80 2024-03-06T07:23:38.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.515+0000 7fac39ffb640 1 --2- 172.21.15.1:0/566378585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac2806b500 0x7fac2806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.515+0000 7fac39ffb640 1 -- 172.21.15.1:0/566378585 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fac300ab3d0 con 0x7fac3c0aee80 2024-03-06T07:23:38.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.516+0000 7fac41387640 1 --2- 172.21.15.1:0/566378585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac2806b500 0x7fac2806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.516+0000 7fac41387640 1 --2- 172.21.15.1:0/566378585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac2806b500 0x7fac2806d9c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fac3c157dd0 tx=0x7fac34009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.520+0000 7fac39ffb640 1 -- 172.21.15.1:0/566378585 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac3004c070 con 0x7fac3c0aee80 2024-03-06T07:23:38.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.699+0000 7fac435eb640 1 -- 172.21.15.1:0/566378585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fac3c0c5b40 con 0x7fac3c0aee80 2024-03-06T07:23:38.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.700+0000 7fac39ffb640 1 -- 172.21.15.1:0/566378585 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fac3007a4a0 con 0x7fac3c0aee80 2024-03-06T07:23:38.702 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:23:38.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.702+0000 7fac435eb640 1 -- 172.21.15.1:0/566378585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac2806b500 msgr2=0x7fac2806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.702+0000 7fac435eb640 1 --2- 172.21.15.1:0/566378585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac2806b500 0x7fac2806d9c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fac3c157dd0 tx=0x7fac34009040 comp rx=0 tx=0).stop 2024-03-06T07:23:38.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.702+0000 7fac435eb640 1 -- 172.21.15.1:0/566378585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3c0aee80 msgr2=0x7fac3c14f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.702+0000 7fac435eb640 1 --2- 172.21.15.1:0/566378585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3c0aee80 0x7fac3c14f0a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fac30037040 tx=0x7fac30004730 comp rx=0 tx=0).stop 2024-03-06T07:23:38.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.702+0000 7fac435eb640 1 -- 172.21.15.1:0/566378585 shutdown_connections 2024-03-06T07:23:38.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.702+0000 7fac435eb640 1 --2- 172.21.15.1:0/566378585 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac2806b500 0x7fac2806d9c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.702+0000 7fac435eb640 1 --2- 172.21.15.1:0/566378585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3c14f5e0 0x7fac3c1ea610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.702+0000 7fac435eb640 1 --2- 172.21.15.1:0/566378585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3c0aee80 0x7fac3c14f0a0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.702+0000 7fac435eb640 1 --2- 172.21.15.1:0/566378585 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac24004800 0x7fac3c0ac010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.702+0000 7fac435eb640 1 -- 172.21.15.1:0/566378585 >> 172.21.15.1:0/566378585 conn(0x7fac3c072720 msgr2=0x7fac3c0c4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.702+0000 7fac435eb640 1 -- 172.21.15.1:0/566378585 shutdown_connections 2024-03-06T07:23:38.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.702+0000 7fac435eb640 1 -- 172.21.15.1:0/566378585 wait complete. 2024-03-06T07:23:38.715 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:23:38.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.814+0000 7f17c9b13640 1 Processor -- start 2024-03-06T07:23:38.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.814+0000 7f17c9b13640 1 -- start start 2024-03-06T07:23:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c9b13640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f17c40c2b30 0x7f17c40724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c9b13640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c4151dd0 con 0x7f17c40a5fa0 2024-03-06T07:23:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c9b13640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c4156190 con 0x7f17c4152b70 2024-03-06T07:23:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c9b13640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c41526e0 con 0x7f17c40c2b30 2024-03-06T07:23:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f17c40a5fa0 0x7f17c415c750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54782/0 (socket says 172.21.15.1:54782) 2024-03-06T07:23:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c3fff640 1 -- 172.21.15.1:0/3250808069 learned_addr learned my addr 172.21.15.1:0/3250808069 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c27fc640 1 -- 172.21.15.1:0/3250808069 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090858303 0 0) 0x7f17c4151dd0 con 0x7f17c40a5fa0 2024-03-06T07:23:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c27fc640 1 -- 172.21.15.1:0/3250808069 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17ac003520 con 0x7f17c40a5fa0 2024-03-06T07:23:38.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c37fe640 1 --2- 172.21.15.1:0/3250808069 >> v2:172.21.15.98:3300/0 conn(0x7f17c40c2b30 0x7f17c40724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c27fc640 1 -- 172.21.15.1:0/3250808069 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3673424447 0 0) 0x7f17ac003520 con 0x7f17c40a5fa0 2024-03-06T07:23:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c27fc640 1 -- 172.21.15.1:0/3250808069 >> v2:172.21.15.98:3300/0 conn(0x7f17c40c2b30 msgr2=0x7f17c40724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c27fc640 1 --2- 172.21.15.1:0/3250808069 >> v2:172.21.15.98:3300/0 conn(0x7f17c40c2b30 0x7f17c40724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c27fc640 1 -- 172.21.15.1:0/3250808069 >> v1:172.21.15.98:6789/0 conn(0x7f17c4152b70 legacy=0x7f17c40c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c27fc640 1 -- 172.21.15.1:0/3250808069 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c4152090 con 0x7f17c40a5fa0 2024-03-06T07:23:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c27fc640 1 -- 172.21.15.1:0/3250808069 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f17bc0019f0 con 0x7f17c40a5fa0 2024-03-06T07:23:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c27fc640 1 -- 172.21.15.1:0/3250808069 >> v1:172.21.15.1:6789/0 conn(0x7f17c40a5fa0 legacy=0x7f17c415c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c27fc640 1 --2- 172.21.15.1:0/3250808069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17ac0039b0 0x7f17ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c27fc640 1 --2- 172.21.15.1:0/3250808069 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ac004800 0x7f17ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c27fc640 1 --2- 172.21.15.1:0/3250808069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac0051c0 0x7f17ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c27fc640 1 -- 172.21.15.1:0/3250808069 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c41526e0 con 0x7f17ac0051c0 2024-03-06T07:23:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c27fc640 1 -- 172.21.15.1:0/3250808069 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac00e270 con 0x7f17ac0039b0 2024-03-06T07:23:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c27fc640 1 -- 172.21.15.1:0/3250808069 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac00e470 con 0x7f17ac004800 2024-03-06T07:23:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c3fff640 1 --2- 172.21.15.1:0/3250808069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac0051c0 0x7f17ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c37fe640 1 --2- 172.21.15.1:0/3250808069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17ac0039b0 0x7f17ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c2ffd640 1 --2- 172.21.15.1:0/3250808069 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ac004800 0x7f17ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c3fff640 1 -- 172.21.15.1:0/3250808069 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ac004800 msgr2=0x7f17ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c3fff640 1 --2- 172.21.15.1:0/3250808069 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ac004800 0x7f17ac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c3fff640 1 -- 172.21.15.1:0/3250808069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17ac0039b0 msgr2=0x7f17ac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c3fff640 1 --2- 172.21.15.1:0/3250808069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17ac0039b0 0x7f17ac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c3fff640 1 -- 172.21.15.1:0/3250808069 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ac003520 con 0x7f17ac0051c0 2024-03-06T07:23:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.816+0000 7f17c2ffd640 1 --2- 172.21.15.1:0/3250808069 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17ac004800 0x7f17ac004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:38.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.817+0000 7f17c3fff640 1 -- 172.21.15.1:0/3250808069 reap_dead start 2024-03-06T07:23:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.817+0000 7f17c3fff640 1 --2- 172.21.15.1:0/3250808069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac0051c0 0x7f17ac00b9c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00a860 tx=0x7f17ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce02b30c6e6a8aeb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.817+0000 7f17c27fc640 1 -- 172.21.15.1:0/3250808069 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc00a480 con 0x7f17ac0051c0 2024-03-06T07:23:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.817+0000 7f17c27fc640 1 -- 172.21.15.1:0/3250808069 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17bc034630 con 0x7f17ac0051c0 2024-03-06T07:23:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.817+0000 7f17c27fc640 1 -- 172.21.15.1:0/3250808069 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc034a20 con 0x7f17ac0051c0 2024-03-06T07:23:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.817+0000 7f17c9b13640 1 -- 172.21.15.1:0/3250808069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac0051c0 msgr2=0x7f17ac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.817+0000 7f17c9b13640 1 --2- 172.21.15.1:0/3250808069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac0051c0 0x7f17ac00b9c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00a860 tx=0x7f17ac005830 comp rx=0 tx=0).stop 2024-03-06T07:23:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.817+0000 7f17c9b13640 1 -- 172.21.15.1:0/3250808069 shutdown_connections 2024-03-06T07:23:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.817+0000 7f17c9b13640 1 --2- 172.21.15.1:0/3250808069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac0051c0 0x7f17ac00b9c0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.817+0000 7f17c9b13640 1 -- 172.21.15.1:0/3250808069 >> 172.21.15.1:0/3250808069 conn(0x7f17c40c6860 msgr2=0x7f17c40c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.817+0000 7f17c9b13640 1 -- 172.21.15.1:0/3250808069 shutdown_connections 2024-03-06T07:23:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.817+0000 7f17c9b13640 1 -- 172.21.15.1:0/3250808069 wait complete. 2024-03-06T07:23:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.818+0000 7f17c9b13640 1 Processor -- start 2024-03-06T07:23:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.819+0000 7f17c9b13640 1 -- start start 2024-03-06T07:23:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.819+0000 7f17c9b13640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac0051c0 0x7f17c40c4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.819+0000 7f17c9b13640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c40ac1b0 0x7f17c41eadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.819+0000 7f17c9b13640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17c40ac630 0x7f17c41eb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.819+0000 7f17c9b13640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c4155f00 con 0x7f17ac0051c0 2024-03-06T07:23:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.819+0000 7f17c9b13640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c4155d80 con 0x7f17c40ac1b0 2024-03-06T07:23:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.819+0000 7f17c9b13640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c40724c0 con 0x7f17c40ac630 2024-03-06T07:23:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.819+0000 7f17c37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac0051c0 0x7f17c40c4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.819+0000 7f17c37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac0051c0 0x7f17c40c4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49994/0 (socket says 172.21.15.1:49994) 2024-03-06T07:23:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.819+0000 7f17c37fe640 1 -- 172.21.15.1:0/1749996432 learned_addr learned my addr 172.21.15.1:0/1749996432 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.820+0000 7f17c2ffd640 1 --2- 172.21.15.1:0/1749996432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c40ac1b0 0x7f17c41eadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.820+0000 7f17c37fe640 1 -- 172.21.15.1:0/1749996432 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17c40ac630 msgr2=0x7f17c41eb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.820+0000 7f17c37fe640 1 --2- 172.21.15.1:0/1749996432 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17c40ac630 0x7f17c41eb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.820+0000 7f17c37fe640 1 -- 172.21.15.1:0/1749996432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c40ac1b0 msgr2=0x7f17c41eadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.820+0000 7f17c37fe640 1 --2- 172.21.15.1:0/1749996432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c40ac1b0 0x7f17c41eadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.820+0000 7f17c37fe640 1 -- 172.21.15.1:0/1749996432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c40abdb0 con 0x7f17ac0051c0 2024-03-06T07:23:38.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.820+0000 7f17c37fe640 1 --2- 172.21.15.1:0/1749996432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac0051c0 0x7f17c40c4680 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f17b40118e0 tx=0x7f17b4011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.820+0000 7f17abfff640 1 -- 172.21.15.1:0/1749996432 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b400faf0 con 0x7f17ac0051c0 2024-03-06T07:23:38.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.820+0000 7f17abfff640 1 -- 172.21.15.1:0/1749996432 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b400fc90 con 0x7f17ac0051c0 2024-03-06T07:23:38.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.820+0000 7f17c9b13640 1 -- 172.21.15.1:0/1749996432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c40acae0 con 0x7f17ac0051c0 2024-03-06T07:23:38.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.820+0000 7f17abfff640 1 -- 172.21.15.1:0/1749996432 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b4014440 con 0x7f17ac0051c0 2024-03-06T07:23:38.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.820+0000 7f17c9b13640 1 -- 172.21.15.1:0/1749996432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c40ab5b0 con 0x7f17ac0051c0 2024-03-06T07:23:38.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.821+0000 7f17c9b13640 1 -- 172.21.15.1:0/1749996432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c4067fd0 con 0x7f17ac0051c0 2024-03-06T07:23:38.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.821+0000 7f17abfff640 1 -- 172.21.15.1:0/1749996432 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17b4012040 con 0x7f17ac0051c0 2024-03-06T07:23:38.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.822+0000 7f17abfff640 1 --2- 172.21.15.1:0/1749996432 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17b006b4d0 0x7f17b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.822+0000 7f17abfff640 1 -- 172.21.15.1:0/1749996432 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f17b408a740 con 0x7f17ac0051c0 2024-03-06T07:23:38.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.822+0000 7f17c2ffd640 1 --2- 172.21.15.1:0/1749996432 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17b006b4d0 0x7f17b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.822+0000 7f17c2ffd640 1 --2- 172.21.15.1:0/1749996432 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17b006b4d0 0x7f17b006d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f17b8007fd0 tx=0x7f17b8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:38.826+0000 7f17abfff640 1 -- 172.21.15.1:0/1749996432 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17b4019030 con 0x7f17ac0051c0 2024-03-06T07:23:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.009+0000 7f17c9b13640 1 -- 172.21.15.1:0/1749996432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f17c4071790 con 0x7f17ac0051c0 2024-03-06T07:23:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.010+0000 7f17abfff640 1 -- 172.21.15.1:0/1749996432 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v91) v1 ==== 84+0+65 (secure 0 0 0) 0x7f17b4059130 con 0x7f17ac0051c0 2024-03-06T07:23:39.012 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:39.012 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-06T07:23:39.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:23:39.012 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-06T07:23:39.012 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-06T07:23:39.012 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:23:39.012 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-06T07:23:39.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.012+0000 7f17c9b13640 1 -- 172.21.15.1:0/1749996432 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17b006b4d0 msgr2=0x7f17b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.012+0000 7f17c9b13640 1 --2- 172.21.15.1:0/1749996432 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17b006b4d0 0x7f17b006d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f17b8007fd0 tx=0x7f17b8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.012+0000 7f17c9b13640 1 -- 172.21.15.1:0/1749996432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac0051c0 msgr2=0x7f17c40c4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.012+0000 7f17c9b13640 1 --2- 172.21.15.1:0/1749996432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac0051c0 0x7f17c40c4680 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f17b40118e0 tx=0x7f17b4011da0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.012+0000 7f17c9b13640 1 -- 172.21.15.1:0/1749996432 shutdown_connections 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.012+0000 7f17c9b13640 1 --2- 172.21.15.1:0/1749996432 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f17b006b4d0 0x7f17b006d990 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.012+0000 7f17c9b13640 1 --2- 172.21.15.1:0/1749996432 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17c40ac630 0x7f17c41eb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.012+0000 7f17c9b13640 1 --2- 172.21.15.1:0/1749996432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c40ac1b0 0x7f17c41eadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.012+0000 7f17c9b13640 1 --2- 172.21.15.1:0/1749996432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17ac0051c0 0x7f17c40c4680 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.012+0000 7f17c9b13640 1 -- 172.21.15.1:0/1749996432 >> 172.21.15.1:0/1749996432 conn(0x7f17c40c6860 msgr2=0x7f17c40c5ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.012+0000 7f17c9b13640 1 -- 172.21.15.1:0/1749996432 shutdown_connections 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.012+0000 7f17c9b13640 1 -- 172.21.15.1:0/1749996432 wait complete. 2024-03-06T07:23:39.024 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:23:39.025 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:23:39.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.129+0000 7f5393ad4640 1 Processor -- start 2024-03-06T07:23:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.129+0000 7f5393ad4640 1 -- start start 2024-03-06T07:23:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.129+0000 7f5393ad4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f538c1527a0 0x7f538c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.129+0000 7f5393ad4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f538c0c3ec0 0x7f538c0c4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.129+0000 7f5393ad4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538c152b80 con 0x7f538c0c2900 2024-03-06T07:23:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.129+0000 7f5393ad4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f538c158760 con 0x7f538c1527a0 2024-03-06T07:23:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.129+0000 7f5393ad4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538c1592e0 con 0x7f538c0c3ec0 2024-03-06T07:23:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.130+0000 7f5391870640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f538c1527a0 0x7f538c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.130+0000 7f5391870640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f538c1527a0 0x7f538c155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49996/0 (socket says 172.21.15.1:49996) 2024-03-06T07:23:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.130+0000 7f539106f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f538c0c2900 0x7f538c06f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54792/0 (socket says 172.21.15.1:54792) 2024-03-06T07:23:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.130+0000 7f5391870640 1 -- 172.21.15.1:0/860974549 learned_addr learned my addr 172.21.15.1:0/860974549 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.130+0000 7f5392071640 1 --2- 172.21.15.1:0/860974549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f538c0c3ec0 0x7f538c0c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.130+0000 7f5391870640 1 -- 172.21.15.1:0/860974549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f538c0c3ec0 msgr2=0x7f538c0c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.130+0000 7f5391870640 1 --2- 172.21.15.1:0/860974549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f538c0c3ec0 0x7f538c0c4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.130+0000 7f5391870640 1 -- 172.21.15.1:0/860974549 >> v1:172.21.15.1:6789/0 conn(0x7f538c0c2900 legacy=0x7f538c06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.130+0000 7f5391870640 1 -- 172.21.15.1:0/860974549 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538c1589b0 con 0x7f538c1527a0 2024-03-06T07:23:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.130+0000 7f539086e640 1 -- 172.21.15.1:0/860974549 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785451160 0 0) 0x7f538c152b80 con 0x7f538c0c2900 2024-03-06T07:23:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.130+0000 7f5392071640 1 --2- 172.21.15.1:0/860974549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f538c0c3ec0 0x7f538c0c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.131+0000 7f5391870640 1 --2- 172.21.15.1:0/860974549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f538c1527a0 0x7f538c155c10 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f5388009870 tx=0x7f538802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3930f10093b4632 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.131+0000 7f539086e640 1 -- 172.21.15.1:0/860974549 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5388035020 con 0x7f538c1527a0 2024-03-06T07:23:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.131+0000 7f539086e640 1 -- 172.21.15.1:0/860974549 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5388031640 con 0x7f538c1527a0 2024-03-06T07:23:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.131+0000 7f539086e640 1 -- 172.21.15.1:0/860974549 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538802d040 con 0x7f538c1527a0 2024-03-06T07:23:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.131+0000 7f5393ad4640 1 -- 172.21.15.1:0/860974549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f538c1527a0 msgr2=0x7f538c155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.131+0000 7f5393ad4640 1 --2- 172.21.15.1:0/860974549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f538c1527a0 0x7f538c155c10 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f5388009870 tx=0x7f538802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.131+0000 7f5393ad4640 1 -- 172.21.15.1:0/860974549 shutdown_connections 2024-03-06T07:23:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.131+0000 7f5393ad4640 1 --2- 172.21.15.1:0/860974549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f538c0c3ec0 0x7f538c0c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.131+0000 7f5393ad4640 1 --2- 172.21.15.1:0/860974549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f538c1527a0 0x7f538c155c10 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.131+0000 7f5393ad4640 1 -- 172.21.15.1:0/860974549 >> 172.21.15.1:0/860974549 conn(0x7f538c0aa6a0 msgr2=0x7f538c0aaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.131+0000 7f5393ad4640 1 -- 172.21.15.1:0/860974549 shutdown_connections 2024-03-06T07:23:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.131+0000 7f5393ad4640 1 -- 172.21.15.1:0/860974549 wait complete. 2024-03-06T07:23:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.132+0000 7f5393ad4640 1 Processor -- start 2024-03-06T07:23:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.133+0000 7f5393ad4640 1 -- start start 2024-03-06T07:23:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.133+0000 7f5393ad4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538c0c2900 0x7f538c14d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.133+0000 7f5393ad4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f538c0c3ec0 0x7f538c143cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.133+0000 7f5393ad4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f538c1527a0 0x7f538c1441f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.133+0000 7f5393ad4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538c0704b0 con 0x7f538c0c3ec0 2024-03-06T07:23:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.133+0000 7f5393ad4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538c158f70 con 0x7f538c0c2900 2024-03-06T07:23:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.133+0000 7f5393ad4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538c06fc60 con 0x7f538c1527a0 2024-03-06T07:23:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.133+0000 7f539106f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f538c0c3ec0 0x7f538c143cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.133+0000 7f539106f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f538c0c3ec0 0x7f538c143cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50002/0 (socket says 172.21.15.1:50002) 2024-03-06T07:23:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.133+0000 7f539106f640 1 -- 172.21.15.1:0/480064958 learned_addr learned my addr 172.21.15.1:0/480064958 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.134+0000 7f539106f640 1 -- 172.21.15.1:0/480064958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f538c1527a0 msgr2=0x7f538c1441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.134+0000 7f5391870640 1 --2- 172.21.15.1:0/480064958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538c0c2900 0x7f538c14d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.134+0000 7f5392071640 1 --2- 172.21.15.1:0/480064958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f538c1527a0 0x7f538c1441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.134+0000 7f539106f640 1 --2- 172.21.15.1:0/480064958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f538c1527a0 0x7f538c1441f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.134+0000 7f539106f640 1 -- 172.21.15.1:0/480064958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538c0c2900 msgr2=0x7f538c14d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.134+0000 7f539106f640 1 --2- 172.21.15.1:0/480064958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538c0c2900 0x7f538c14d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.134+0000 7f539106f640 1 -- 172.21.15.1:0/480064958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538c158030 con 0x7f538c0c3ec0 2024-03-06T07:23:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.134+0000 7f5391870640 1 --2- 172.21.15.1:0/480064958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538c0c2900 0x7f538c14d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.134+0000 7f5392071640 1 --2- 172.21.15.1:0/480064958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f538c1527a0 0x7f538c1441f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.134+0000 7f539106f640 1 --2- 172.21.15.1:0/480064958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f538c0c3ec0 0x7f538c143cb0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f537c00c500 tx=0x7f537c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.134+0000 7f53827fc640 1 -- 172.21.15.1:0/480064958 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537c018070 con 0x7f538c0c3ec0 2024-03-06T07:23:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.134+0000 7f53827fc640 1 -- 172.21.15.1:0/480064958 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f537c007580 con 0x7f538c0c3ec0 2024-03-06T07:23:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.134+0000 7f5393ad4640 1 -- 172.21.15.1:0/480064958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f538c1eaa80 con 0x7f538c0c3ec0 2024-03-06T07:23:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.134+0000 7f53827fc640 1 -- 172.21.15.1:0/480064958 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537c004a20 con 0x7f538c0c3ec0 2024-03-06T07:23:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.134+0000 7f5393ad4640 1 -- 172.21.15.1:0/480064958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f538c1eb040 con 0x7f538c0c3ec0 2024-03-06T07:23:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.135+0000 7f5393ad4640 1 -- 172.21.15.1:0/480064958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f538c0680b0 con 0x7f538c0c3ec0 2024-03-06T07:23:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.135+0000 7f53827fc640 1 -- 172.21.15.1:0/480064958 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f537c01b070 con 0x7f538c0c3ec0 2024-03-06T07:23:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.136+0000 7f53827fc640 1 --2- 172.21.15.1:0/480064958 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f537006b4d0 0x7f537006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.136+0000 7f53827fc640 1 -- 172.21.15.1:0/480064958 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f537c0892b0 con 0x7f538c0c3ec0 2024-03-06T07:23:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.136+0000 7f5391870640 1 --2- 172.21.15.1:0/480064958 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f537006b4d0 0x7f537006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.136+0000 7f5391870640 1 --2- 172.21.15.1:0/480064958 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f537006b4d0 0x7f537006d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5388036000 tx=0x7f5388008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.140+0000 7f53827fc640 1 -- 172.21.15.1:0/480064958 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f538c158030 con 0x7f538c0c3ec0 2024-03-06T07:23:39.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.323+0000 7f5393ad4640 1 -- 172.21.15.1:0/480064958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f538c1505f0 con 0x7f538c0c3ec0 2024-03-06T07:23:39.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.698+0000 7f53827fc640 1 -- 172.21.15.1:0/480064958 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v92) v1 ==== 138+0+0 (secure 0 0 0) 0x7f537c058360 con 0x7f538c0c3ec0 2024-03-06T07:23:39.700 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-06T07:23:39.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.700+0000 7f5393ad4640 1 -- 172.21.15.1:0/480064958 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f537006b4d0 msgr2=0x7f537006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.700+0000 7f5393ad4640 1 --2- 172.21.15.1:0/480064958 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f537006b4d0 0x7f537006d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5388036000 tx=0x7f5388008030 comp rx=0 tx=0).stop 2024-03-06T07:23:39.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.700+0000 7f5393ad4640 1 -- 172.21.15.1:0/480064958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f538c0c3ec0 msgr2=0x7f538c143cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.700+0000 7f5393ad4640 1 --2- 172.21.15.1:0/480064958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f538c0c3ec0 0x7f538c143cb0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f537c00c500 tx=0x7f537c00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.700+0000 7f5393ad4640 1 -- 172.21.15.1:0/480064958 shutdown_connections 2024-03-06T07:23:39.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.700+0000 7f5393ad4640 1 --2- 172.21.15.1:0/480064958 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f537006b4d0 0x7f537006d990 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.700+0000 7f5393ad4640 1 --2- 172.21.15.1:0/480064958 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f538c1527a0 0x7f538c1441f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.700+0000 7f5393ad4640 1 --2- 172.21.15.1:0/480064958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f538c0c3ec0 0x7f538c143cb0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.700+0000 7f5393ad4640 1 --2- 172.21.15.1:0/480064958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538c0c2900 0x7f538c14d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.700+0000 7f5393ad4640 1 -- 172.21.15.1:0/480064958 >> 172.21.15.1:0/480064958 conn(0x7f538c0aa6a0 msgr2=0x7f538c0c5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.700+0000 7f5393ad4640 1 -- 172.21.15.1:0/480064958 shutdown_connections 2024-03-06T07:23:39.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.700+0000 7f5393ad4640 1 -- 172.21.15.1:0/480064958 wait complete. 2024-03-06T07:23:39.717 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:23:39.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.827+0000 7f6e27bf7640 1 Processor -- start 2024-03-06T07:23:39.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.827+0000 7f6e27bf7640 1 -- start start 2024-03-06T07:23:39.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.827+0000 7f6e27bf7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6e20152690 0x7f6e20155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.827+0000 7f6e27bf7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6e20156cd0 0x7f6e20157120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.827+0000 7f6e27bf7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e201561e0 con 0x7f6e200c6730 2024-03-06T07:23:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.827+0000 7f6e27bf7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e200c3300 con 0x7f6e20152690 2024-03-06T07:23:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.827+0000 7f6e27bf7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e200c35a0 con 0x7f6e20156cd0 2024-03-06T07:23:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.827+0000 7f6e25993640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6e20152690 0x7f6e20155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.828+0000 7f6e25993640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6e20152690 0x7f6e20155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50006/0 (socket says 172.21.15.1:50006) 2024-03-06T07:23:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.828+0000 7f6e25993640 1 -- 172.21.15.1:0/2409612048 learned_addr learned my addr 172.21.15.1:0/2409612048 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.828+0000 7f6e25993640 1 -- 172.21.15.1:0/2409612048 >> v2:172.21.15.98:3300/0 conn(0x7f6e20156cd0 msgr2=0x7f6e20157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.828+0000 7f6e25993640 1 --2- 172.21.15.1:0/2409612048 >> v2:172.21.15.98:3300/0 conn(0x7f6e20156cd0 0x7f6e20157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.828+0000 7f6e25993640 1 -- 172.21.15.1:0/2409612048 >> v1:172.21.15.98:6789/0 conn(0x7f6e200c6730 legacy=0x7f6e200c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.828+0000 7f6e25993640 1 -- 172.21.15.1:0/2409612048 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e200c3880 con 0x7f6e20152690 2024-03-06T07:23:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.828+0000 7f6e25993640 1 --2- 172.21.15.1:0/2409612048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e20152690 0x7f6e20155ca0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f6e200c6350 tx=0x7f6e1002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=648c95b48140530f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.829+0000 7f6e24991640 1 -- 172.21.15.1:0/2409612048 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1002e070 con 0x7f6e20152690 2024-03-06T07:23:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.829+0000 7f6e24991640 1 -- 172.21.15.1:0/2409612048 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e10037070 con 0x7f6e20152690 2024-03-06T07:23:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.829+0000 7f6e24991640 1 -- 172.21.15.1:0/2409612048 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e10031a00 con 0x7f6e20152690 2024-03-06T07:23:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.829+0000 7f6e27bf7640 1 -- 172.21.15.1:0/2409612048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e20152690 msgr2=0x7f6e20155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.829+0000 7f6e27bf7640 1 --2- 172.21.15.1:0/2409612048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e20152690 0x7f6e20155ca0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f6e200c6350 tx=0x7f6e1002b880 comp rx=0 tx=0).stop 2024-03-06T07:23:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.829+0000 7f6e27bf7640 1 -- 172.21.15.1:0/2409612048 shutdown_connections 2024-03-06T07:23:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.829+0000 7f6e27bf7640 1 --2- 172.21.15.1:0/2409612048 >> v2:172.21.15.98:3300/0 conn(0x7f6e20156cd0 0x7f6e20157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.829+0000 7f6e27bf7640 1 --2- 172.21.15.1:0/2409612048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e20152690 0x7f6e20155ca0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.829+0000 7f6e27bf7640 1 -- 172.21.15.1:0/2409612048 >> 172.21.15.1:0/2409612048 conn(0x7f6e2006b410 msgr2=0x7f6e2006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.829+0000 7f6e27bf7640 1 -- 172.21.15.1:0/2409612048 shutdown_connections 2024-03-06T07:23:39.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.829+0000 7f6e27bf7640 1 -- 172.21.15.1:0/2409612048 wait complete. 2024-03-06T07:23:39.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.830+0000 7f6e27bf7640 1 Processor -- start 2024-03-06T07:23:39.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.830+0000 7f6e27bf7640 1 -- start start 2024-03-06T07:23:39.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.831+0000 7f6e27bf7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e200c6730 0x7f6e2014c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.831+0000 7f6e27bf7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e20152690 0x7f6e201ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.831+0000 7f6e27bf7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e20156cd0 0x7f6e201eeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.831+0000 7f6e27bf7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e200c5ee0 con 0x7f6e20152690 2024-03-06T07:23:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.831+0000 7f6e27bf7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e201567b0 con 0x7f6e200c6730 2024-03-06T07:23:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.831+0000 7f6e27bf7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e200c6060 con 0x7f6e20156cd0 2024-03-06T07:23:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.831+0000 7f6e25192640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e20152690 0x7f6e201ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.831+0000 7f6e25192640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e20152690 0x7f6e201ee5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50012/0 (socket says 172.21.15.1:50012) 2024-03-06T07:23:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.831+0000 7f6e25192640 1 -- 172.21.15.1:0/1202578043 learned_addr learned my addr 172.21.15.1:0/1202578043 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.831+0000 7f6e25993640 1 --2- 172.21.15.1:0/1202578043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e200c6730 0x7f6e2014c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.831+0000 7f6e26194640 1 --2- 172.21.15.1:0/1202578043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e20156cd0 0x7f6e201eeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.831+0000 7f6e25192640 1 -- 172.21.15.1:0/1202578043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e20156cd0 msgr2=0x7f6e201eeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.831+0000 7f6e25192640 1 --2- 172.21.15.1:0/1202578043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e20156cd0 0x7f6e201eeb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.831+0000 7f6e25192640 1 -- 172.21.15.1:0/1202578043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e200c6730 msgr2=0x7f6e2014c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.831+0000 7f6e25192640 1 --2- 172.21.15.1:0/1202578043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e200c6730 0x7f6e2014c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.831+0000 7f6e25192640 1 -- 172.21.15.1:0/1202578043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e200ab850 con 0x7f6e20152690 2024-03-06T07:23:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.831+0000 7f6e26194640 1 --2- 172.21.15.1:0/1202578043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e20156cd0 0x7f6e201eeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.832+0000 7f6e25192640 1 --2- 172.21.15.1:0/1202578043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e20152690 0x7f6e201ee5f0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c00b3e0 tx=0x7f6e1c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.832+0000 7f6e167fc640 1 -- 172.21.15.1:0/1202578043 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c015020 con 0x7f6e20152690 2024-03-06T07:23:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.832+0000 7f6e167fc640 1 -- 172.21.15.1:0/1202578043 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1c011850 con 0x7f6e20152690 2024-03-06T07:23:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.832+0000 7f6e27bf7640 1 -- 172.21.15.1:0/1202578043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e201ef170 con 0x7f6e20152690 2024-03-06T07:23:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.832+0000 7f6e167fc640 1 -- 172.21.15.1:0/1202578043 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c011c40 con 0x7f6e20152690 2024-03-06T07:23:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.832+0000 7f6e27bf7640 1 -- 172.21.15.1:0/1202578043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e20155ea0 con 0x7f6e20152690 2024-03-06T07:23:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.833+0000 7f6e27bf7640 1 -- 172.21.15.1:0/1202578043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e20067fd0 con 0x7f6e20152690 2024-03-06T07:23:39.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.833+0000 7f6e167fc640 1 -- 172.21.15.1:0/1202578043 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e1c018020 con 0x7f6e20152690 2024-03-06T07:23:39.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.834+0000 7f6e167fc640 1 --2- 172.21.15.1:0/1202578043 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e0406b500 0x7f6e0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.834+0000 7f6e167fc640 1 -- 172.21.15.1:0/1202578043 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f6e1c08a120 con 0x7f6e20152690 2024-03-06T07:23:39.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.834+0000 7f6e25993640 1 --2- 172.21.15.1:0/1202578043 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e0406b500 0x7f6e0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.835+0000 7f6e25993640 1 --2- 172.21.15.1:0/1202578043 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e0406b500 0x7f6e0406d9c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f6e10036000 tx=0x7f6e1003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:39.839+0000 7f6e167fc640 1 -- 172.21.15.1:0/1202578043 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e1c01f020 con 0x7f6e20152690 2024-03-06T07:23:40.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.025+0000 7f6e27bf7640 1 -- 172.21.15.1:0/1202578043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f6e200c2c40 con 0x7f6e20152690 2024-03-06T07:23:40.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.701+0000 7f6e167fc640 1 -- 172.21.15.1:0/1202578043 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v93) v1 ==== 170+0+0 (secure 0 0 0) 0x7f6e1c0172c0 con 0x7f6e20152690 2024-03-06T07:23:40.702 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-06T07:23:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.709+0000 7f6e27bf7640 1 -- 172.21.15.1:0/1202578043 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e0406b500 msgr2=0x7f6e0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.709+0000 7f6e27bf7640 1 --2- 172.21.15.1:0/1202578043 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e0406b500 0x7f6e0406d9c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f6e10036000 tx=0x7f6e1003d040 comp rx=0 tx=0).stop 2024-03-06T07:23:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.709+0000 7f6e27bf7640 1 -- 172.21.15.1:0/1202578043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e20152690 msgr2=0x7f6e201ee5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.709+0000 7f6e27bf7640 1 --2- 172.21.15.1:0/1202578043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e20152690 0x7f6e201ee5f0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c00b3e0 tx=0x7f6e1c00b8a0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.712+0000 7f6e27bf7640 1 -- 172.21.15.1:0/1202578043 shutdown_connections 2024-03-06T07:23:40.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.712+0000 7f6e27bf7640 1 --2- 172.21.15.1:0/1202578043 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e0406b500 0x7f6e0406d9c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.712+0000 7f6e27bf7640 1 --2- 172.21.15.1:0/1202578043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e20156cd0 0x7f6e201eeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.712+0000 7f6e27bf7640 1 --2- 172.21.15.1:0/1202578043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e20152690 0x7f6e201ee5f0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.712+0000 7f6e27bf7640 1 --2- 172.21.15.1:0/1202578043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e200c6730 0x7f6e2014c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.712+0000 7f6e27bf7640 1 -- 172.21.15.1:0/1202578043 >> 172.21.15.1:0/1202578043 conn(0x7f6e2006b410 msgr2=0x7f6e2006b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:40.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.714+0000 7f6e27bf7640 1 -- 172.21.15.1:0/1202578043 shutdown_connections 2024-03-06T07:23:40.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.714+0000 7f6e27bf7640 1 -- 172.21.15.1:0/1202578043 wait complete. 2024-03-06T07:23:40.735 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:23:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.843+0000 7f64537f6640 1 Processor -- start 2024-03-06T07:23:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.844+0000 7f64537f6640 1 -- start start 2024-03-06T07:23:40.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.844+0000 7f64537f6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f644c159140 0x7f644c0c4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.844+0000 7f64537f6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644c072d00 con 0x7f644c0c7510 2024-03-06T07:23:40.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.844+0000 7f64537f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644c073180 con 0x7f644c0c69b0 2024-03-06T07:23:40.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.844+0000 7f64537f6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f644c0c0e80 con 0x7f644c159140 2024-03-06T07:23:40.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.845+0000 7f6451d93640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f644c0c7510 0x7f644c15fb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54808/0 (socket says 172.21.15.1:54808) 2024-03-06T07:23:40.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.845+0000 7f6451d93640 1 -- 172.21.15.1:0/2891217043 learned_addr learned my addr 172.21.15.1:0/2891217043 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:40.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.845+0000 7f6451592640 1 --2- 172.21.15.1:0/2891217043 >> v2:172.21.15.1:3300/0 conn(0x7f644c159140 0x7f644c0c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.845+0000 7f6443fff640 1 -- 172.21.15.1:0/2891217043 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283803579 0 0) 0x7f644c072d00 con 0x7f644c0c7510 2024-03-06T07:23:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.845+0000 7f6443fff640 1 -- 172.21.15.1:0/2891217043 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6438003520 con 0x7f644c0c7510 2024-03-06T07:23:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.845+0000 7f6451592640 1 -- 172.21.15.1:0/2891217043 >> v1:172.21.15.98:6789/0 conn(0x7f644c0c69b0 legacy=0x7f644c0c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.845+0000 7f6451592640 1 -- 172.21.15.1:0/2891217043 >> v1:172.21.15.1:6789/0 conn(0x7f644c0c7510 legacy=0x7f644c15fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.845+0000 7f6451592640 1 -- 172.21.15.1:0/2891217043 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c073460 con 0x7f644c159140 2024-03-06T07:23:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.846+0000 7f6451592640 1 --2- 172.21.15.1:0/2891217043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f644c159140 0x7f644c0c4490 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f64480097a0 tx=0x7f644802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47901b9b6dcb25c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.846+0000 7f6443fff640 1 -- 172.21.15.1:0/2891217043 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6448035020 con 0x7f644c159140 2024-03-06T07:23:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.846+0000 7f6443fff640 1 -- 172.21.15.1:0/2891217043 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6448031830 con 0x7f644c159140 2024-03-06T07:23:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.846+0000 7f6443fff640 1 -- 172.21.15.1:0/2891217043 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6448031c00 con 0x7f644c159140 2024-03-06T07:23:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.846+0000 7f64537f6640 1 -- 172.21.15.1:0/2891217043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f644c159140 msgr2=0x7f644c0c4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.846+0000 7f64537f6640 1 --2- 172.21.15.1:0/2891217043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f644c159140 0x7f644c0c4490 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f64480097a0 tx=0x7f644802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.846+0000 7f64537f6640 1 -- 172.21.15.1:0/2891217043 shutdown_connections 2024-03-06T07:23:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.846+0000 7f64537f6640 1 --2- 172.21.15.1:0/2891217043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f644c159140 0x7f644c0c4490 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.846+0000 7f64537f6640 1 -- 172.21.15.1:0/2891217043 >> 172.21.15.1:0/2891217043 conn(0x7f644c0c7a50 msgr2=0x7f644c0c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.847+0000 7f64537f6640 1 -- 172.21.15.1:0/2891217043 shutdown_connections 2024-03-06T07:23:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.847+0000 7f64537f6640 1 -- 172.21.15.1:0/2891217043 wait complete. 2024-03-06T07:23:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.848+0000 7f64537f6640 1 Processor -- start 2024-03-06T07:23:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.848+0000 7f64537f6640 1 -- start start 2024-03-06T07:23:40.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.848+0000 7f64537f6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f644c0c69b0 0x7f644c06f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.848+0000 7f64537f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f644c0c7510 0x7f644c1e7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.848+0000 7f64537f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f644c159140 0x7f644c1e7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.848+0000 7f64537f6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c0c6520 con 0x7f644c0c69b0 2024-03-06T07:23:40.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.848+0000 7f64537f6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c0c13b0 con 0x7f644c159140 2024-03-06T07:23:40.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.848+0000 7f64537f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644c159520 con 0x7f644c0c7510 2024-03-06T07:23:40.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.848+0000 7f6451592640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f644c0c69b0 0x7f644c06f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.848+0000 7f6451592640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f644c0c69b0 0x7f644c06f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50030/0 (socket says 172.21.15.1:50030) 2024-03-06T07:23:40.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.848+0000 7f6451592640 1 -- 172.21.15.1:0/1574774948 learned_addr learned my addr 172.21.15.1:0/1574774948 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:40.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.849+0000 7f6451d93640 1 --2- 172.21.15.1:0/1574774948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f644c159140 0x7f644c1e7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.849+0000 7f6450d91640 1 --2- 172.21.15.1:0/1574774948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f644c0c7510 0x7f644c1e7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.849+0000 7f6451592640 1 -- 172.21.15.1:0/1574774948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f644c0c7510 msgr2=0x7f644c1e7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.849+0000 7f6451592640 1 --2- 172.21.15.1:0/1574774948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f644c0c7510 0x7f644c1e7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.849+0000 7f6451592640 1 -- 172.21.15.1:0/1574774948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f644c159140 msgr2=0x7f644c1e7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.849+0000 7f6451592640 1 --2- 172.21.15.1:0/1574774948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f644c159140 0x7f644c1e7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.849+0000 7f6451592640 1 -- 172.21.15.1:0/1574774948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c1ede00 con 0x7f644c0c69b0 2024-03-06T07:23:40.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.849+0000 7f6451592640 1 --2- 172.21.15.1:0/1574774948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f644c0c69b0 0x7f644c06f100 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f64480098c0 tx=0x7f64480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.849+0000 7f6441ffb640 1 -- 172.21.15.1:0/1574774948 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6448040070 con 0x7f644c0c69b0 2024-03-06T07:23:40.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.849+0000 7f6441ffb640 1 -- 172.21.15.1:0/1574774948 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644803e070 con 0x7f644c0c69b0 2024-03-06T07:23:40.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.849+0000 7f64537f6640 1 -- 172.21.15.1:0/1574774948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f644c1ee0e0 con 0x7f644c0c69b0 2024-03-06T07:23:40.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.849+0000 7f6441ffb640 1 -- 172.21.15.1:0/1574774948 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6448035040 con 0x7f644c0c69b0 2024-03-06T07:23:40.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.849+0000 7f64537f6640 1 -- 172.21.15.1:0/1574774948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f644c1ee6a0 con 0x7f644c0c69b0 2024-03-06T07:23:40.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.850+0000 7f64537f6640 1 -- 172.21.15.1:0/1574774948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f644c0686e0 con 0x7f644c0c69b0 2024-03-06T07:23:40.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.851+0000 7f6441ffb640 1 -- 172.21.15.1:0/1574774948 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6448038020 con 0x7f644c0c69b0 2024-03-06T07:23:40.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.851+0000 7f6441ffb640 1 --2- 172.21.15.1:0/1574774948 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f643006b4d0 0x7f643006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.851+0000 7f6441ffb640 1 -- 172.21.15.1:0/1574774948 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f64480aba60 con 0x7f644c0c69b0 2024-03-06T07:23:40.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.851+0000 7f6450d91640 1 --2- 172.21.15.1:0/1574774948 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f643006b4d0 0x7f643006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.852+0000 7f6450d91640 1 --2- 172.21.15.1:0/1574774948 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f643006b4d0 0x7f643006d990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f644c14ae90 tx=0x7f643c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:40.856+0000 7f6441ffb640 1 -- 172.21.15.1:0/1574774948 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6448031800 con 0x7f644c0c69b0 2024-03-06T07:23:41.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.076+0000 7f64537f6640 1 -- 172.21.15.1:0/1574774948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f644c072f60 con 0x7f644c0c69b0 2024-03-06T07:23:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.720+0000 7f6441ffb640 1 -- 172.21.15.1:0/1574774948 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v75) v1 ==== 174+0+0 (secure 0 0 0) 0x7f6448080020 con 0x7f644c0c69b0 2024-03-06T07:23:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 15 and data pool 16 2024-03-06T07:23:41.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.733+0000 7f64537f6640 1 -- 172.21.15.1:0/1574774948 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f643006b4d0 msgr2=0x7f643006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.733+0000 7f64537f6640 1 --2- 172.21.15.1:0/1574774948 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f643006b4d0 0x7f643006d990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f644c14ae90 tx=0x7f643c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:41.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.733+0000 7f64537f6640 1 -- 172.21.15.1:0/1574774948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f644c0c69b0 msgr2=0x7f644c06f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.733+0000 7f64537f6640 1 --2- 172.21.15.1:0/1574774948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f644c0c69b0 0x7f644c06f100 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f64480098c0 tx=0x7f64480046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.735+0000 7f64537f6640 1 -- 172.21.15.1:0/1574774948 shutdown_connections 2024-03-06T07:23:41.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.735+0000 7f64537f6640 1 --2- 172.21.15.1:0/1574774948 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f643006b4d0 0x7f643006d990 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.735+0000 7f64537f6640 1 --2- 172.21.15.1:0/1574774948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f644c159140 0x7f644c1e7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.735+0000 7f64537f6640 1 --2- 172.21.15.1:0/1574774948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f644c0c7510 0x7f644c1e7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.735+0000 7f64537f6640 1 --2- 172.21.15.1:0/1574774948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f644c0c69b0 0x7f644c06f100 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.735+0000 7f64537f6640 1 -- 172.21.15.1:0/1574774948 >> 172.21.15.1:0/1574774948 conn(0x7f644c0c7a50 msgr2=0x7f644c0707c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:41.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.736+0000 7f64537f6640 1 -- 172.21.15.1:0/1574774948 shutdown_connections 2024-03-06T07:23:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.739+0000 7f64537f6640 1 -- 172.21.15.1:0/1574774948 wait complete. 2024-03-06T07:23:41.759 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:41.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.863+0000 7fb241bf6640 1 Processor -- start 2024-03-06T07:23:41.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.863+0000 7fb241bf6640 1 -- start start 2024-03-06T07:23:41.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.864+0000 7fb241bf6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb23c152920 0x7fb23c06aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.864+0000 7fb241bf6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb23c155220 0x7fb23c1556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.864+0000 7fb241bf6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb23c158910 con 0x7fb23c06b530 2024-03-06T07:23:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.864+0000 7fb241bf6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb23c152d00 con 0x7fb23c152920 2024-03-06T07:23:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.864+0000 7fb241bf6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c1596d0 con 0x7fb23c155220 2024-03-06T07:23:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.864+0000 7fb23affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb23c06b530 0x7fb23c0c5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54822/0 (socket says 172.21.15.1:54822) 2024-03-06T07:23:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.864+0000 7fb23affd640 1 -- 172.21.15.1:0/1540511191 learned_addr learned my addr 172.21.15.1:0/1540511191 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.864+0000 7fb23b7fe640 1 --2- 172.21.15.1:0/1540511191 >> v2:172.21.15.98:3300/0 conn(0x7fb23c152920 0x7fb23c06aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.864+0000 7fb23bfff640 1 --2- 172.21.15.1:0/1540511191 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb23c155220 0x7fb23c1556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.864+0000 7fb23a7fc640 1 -- 172.21.15.1:0/1540511191 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065231236 0 0) 0x7fb23c158910 con 0x7fb23c06b530 2024-03-06T07:23:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.864+0000 7fb23a7fc640 1 -- 172.21.15.1:0/1540511191 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb224003520 con 0x7fb23c06b530 2024-03-06T07:23:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.864+0000 7fb23b7fe640 1 -- 172.21.15.1:0/1540511191 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb23c155220 msgr2=0x7fb23c1556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.864+0000 7fb23b7fe640 1 --2- 172.21.15.1:0/1540511191 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb23c155220 0x7fb23c1556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.864+0000 7fb23b7fe640 1 -- 172.21.15.1:0/1540511191 >> v1:172.21.15.1:6789/0 conn(0x7fb23c06b530 legacy=0x7fb23c0c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.864+0000 7fb23b7fe640 1 -- 172.21.15.1:0/1540511191 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c158bf0 con 0x7fb23c152920 2024-03-06T07:23:41.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.864+0000 7fb23bfff640 1 --2- 172.21.15.1:0/1540511191 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb23c155220 0x7fb23c1556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:41.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.865+0000 7fb23b7fe640 1 --2- 172.21.15.1:0/1540511191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb23c152920 0x7fb23c06aff0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb22c0097a0 tx=0x7fb22c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=354fcf49c0258360 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.865+0000 7fb23a7fc640 1 -- 172.21.15.1:0/1540511191 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c0315a0 con 0x7fb23c152920 2024-03-06T07:23:41.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.865+0000 7fb23a7fc640 1 -- 172.21.15.1:0/1540511191 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22c037070 con 0x7fb23c152920 2024-03-06T07:23:41.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.865+0000 7fb23a7fc640 1 -- 172.21.15.1:0/1540511191 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c031940 con 0x7fb23c152920 2024-03-06T07:23:41.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.866+0000 7fb241bf6640 1 -- 172.21.15.1:0/1540511191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb23c152920 msgr2=0x7fb23c06aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.866+0000 7fb241bf6640 1 --2- 172.21.15.1:0/1540511191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb23c152920 0x7fb23c06aff0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb22c0097a0 tx=0x7fb22c02b840 comp rx=0 tx=0).stop 2024-03-06T07:23:41.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.866+0000 7fb241bf6640 1 -- 172.21.15.1:0/1540511191 shutdown_connections 2024-03-06T07:23:41.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.866+0000 7fb241bf6640 1 --2- 172.21.15.1:0/1540511191 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb23c155220 0x7fb23c1556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.866+0000 7fb241bf6640 1 --2- 172.21.15.1:0/1540511191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb23c152920 0x7fb23c06aff0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.866+0000 7fb241bf6640 1 -- 172.21.15.1:0/1540511191 >> 172.21.15.1:0/1540511191 conn(0x7fb23c0a9700 msgr2=0x7fb23c0a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:41.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.866+0000 7fb241bf6640 1 -- 172.21.15.1:0/1540511191 shutdown_connections 2024-03-06T07:23:41.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.866+0000 7fb241bf6640 1 -- 172.21.15.1:0/1540511191 wait complete. 2024-03-06T07:23:41.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.867+0000 7fb241bf6640 1 Processor -- start 2024-03-06T07:23:41.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.867+0000 7fb241bf6640 1 -- start start 2024-03-06T07:23:41.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.868+0000 7fb241bf6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb23c06b530 0x7fb23c06f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.868+0000 7fb241bf6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23c152920 0x7fb23c1ead60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.868+0000 7fb241bf6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb23c155220 0x7fb23c1eb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.868+0000 7fb241bf6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c0c6380 con 0x7fb23c152920 2024-03-06T07:23:41.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.868+0000 7fb241bf6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c159240 con 0x7fb23c06b530 2024-03-06T07:23:41.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.868+0000 7fb241bf6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c0c5f60 con 0x7fb23c155220 2024-03-06T07:23:41.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.868+0000 7fb23affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23c152920 0x7fb23c1ead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.868+0000 7fb23affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23c152920 0x7fb23c1ead60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50046/0 (socket says 172.21.15.1:50046) 2024-03-06T07:23:41.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.868+0000 7fb23affd640 1 -- 172.21.15.1:0/3240447228 learned_addr learned my addr 172.21.15.1:0/3240447228 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:41.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.868+0000 7fb23b7fe640 1 --2- 172.21.15.1:0/3240447228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb23c06b530 0x7fb23c06f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.868+0000 7fb23bfff640 1 --2- 172.21.15.1:0/3240447228 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb23c155220 0x7fb23c1eb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.868+0000 7fb23affd640 1 -- 172.21.15.1:0/3240447228 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb23c155220 msgr2=0x7fb23c1eb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.868+0000 7fb23affd640 1 --2- 172.21.15.1:0/3240447228 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb23c155220 0x7fb23c1eb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.868+0000 7fb23affd640 1 -- 172.21.15.1:0/3240447228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb23c06b530 msgr2=0x7fb23c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.868+0000 7fb23affd640 1 --2- 172.21.15.1:0/3240447228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb23c06b530 0x7fb23c06f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.868+0000 7fb23affd640 1 -- 172.21.15.1:0/3240447228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c0aa910 con 0x7fb23c152920 2024-03-06T07:23:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.868+0000 7fb23affd640 1 --2- 172.21.15.1:0/3240447228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23c152920 0x7fb23c1ead60 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb23000c400 tx=0x7fb23000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.869+0000 7fb223fff640 1 -- 172.21.15.1:0/3240447228 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb230017070 con 0x7fb23c152920 2024-03-06T07:23:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.869+0000 7fb223fff640 1 -- 172.21.15.1:0/3240447228 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb230005d00 con 0x7fb23c152920 2024-03-06T07:23:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.869+0000 7fb223fff640 1 -- 172.21.15.1:0/3240447228 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb230007310 con 0x7fb23c152920 2024-03-06T07:23:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.869+0000 7fb241bf6640 1 -- 172.21.15.1:0/3240447228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23c0aabf0 con 0x7fb23c152920 2024-03-06T07:23:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.869+0000 7fb241bf6640 1 -- 172.21.15.1:0/3240447228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23c0ab1b0 con 0x7fb23c152920 2024-03-06T07:23:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.870+0000 7fb241bf6640 1 -- 172.21.15.1:0/3240447228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb23c0a7270 con 0x7fb23c152920 2024-03-06T07:23:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.870+0000 7fb223fff640 1 -- 172.21.15.1:0/3240447228 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb23001a070 con 0x7fb23c152920 2024-03-06T07:23:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.870+0000 7fb223fff640 1 --2- 172.21.15.1:0/3240447228 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb22806b500 0x7fb22806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.871+0000 7fb223fff640 1 -- 172.21.15.1:0/3240447228 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb2300897d0 con 0x7fb23c152920 2024-03-06T07:23:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.871+0000 7fb23b7fe640 1 --2- 172.21.15.1:0/3240447228 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb22806b500 0x7fb22806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.871+0000 7fb23b7fe640 1 --2- 172.21.15.1:0/3240447228 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb22806b500 0x7fb22806d9c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb22c036000 tx=0x7fb22c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:41.875+0000 7fb223fff640 1 -- 172.21.15.1:0/3240447228 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb23c0a7270 con 0x7fb23c152920 2024-03-06T07:23:42.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.058+0000 7fb241bf6640 1 -- 172.21.15.1:0/3240447228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb23c067fd0 con 0x7fb23c152920 2024-03-06T07:23:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.059+0000 7fb223fff640 1 -- 172.21.15.1:0/3240447228 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16766 (secure 0 0 0) 0x7fb2300584d0 con 0x7fb23c152920 2024-03-06T07:23:42.061 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:42.061 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":94,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:23:41.076663+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:39.342951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:40.043775+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.061+0000 7fb241bf6640 1 -- 172.21.15.1:0/3240447228 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb22806b500 msgr2=0x7fb22806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.061+0000 7fb241bf6640 1 --2- 172.21.15.1:0/3240447228 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb22806b500 0x7fb22806d9c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb22c036000 tx=0x7fb22c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.061+0000 7fb241bf6640 1 -- 172.21.15.1:0/3240447228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23c152920 msgr2=0x7fb23c1ead60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.061+0000 7fb241bf6640 1 --2- 172.21.15.1:0/3240447228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23c152920 0x7fb23c1ead60 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb23000c400 tx=0x7fb23000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.061+0000 7fb241bf6640 1 -- 172.21.15.1:0/3240447228 shutdown_connections 2024-03-06T07:23:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.061+0000 7fb241bf6640 1 --2- 172.21.15.1:0/3240447228 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb22806b500 0x7fb22806d9c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.061+0000 7fb241bf6640 1 --2- 172.21.15.1:0/3240447228 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb23c155220 0x7fb23c1eb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.061+0000 7fb241bf6640 1 --2- 172.21.15.1:0/3240447228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23c152920 0x7fb23c1ead60 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.061+0000 7fb241bf6640 1 --2- 172.21.15.1:0/3240447228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb23c06b530 0x7fb23c06f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.061+0000 7fb241bf6640 1 -- 172.21.15.1:0/3240447228 >> 172.21.15.1:0/3240447228 conn(0x7fb23c0a9700 msgr2=0x7fb23c155690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.061+0000 7fb241bf6640 1 -- 172.21.15.1:0/3240447228 shutdown_connections 2024-03-06T07:23:42.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.061+0000 7fb241bf6640 1 -- 172.21.15.1:0/3240447228 wait complete. 2024-03-06T07:23:42.074 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:42.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.173+0000 7f330efc9640 1 Processor -- start 2024-03-06T07:23:42.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.173+0000 7f330efc9640 1 -- start start 2024-03-06T07:23:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.174+0000 7f330efc9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3308150520 0x7f33081543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.174+0000 7f330efc9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33080c32f0 0x7f33081537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.174+0000 7f330efc9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3308150900 con 0x7f3308070c80 2024-03-06T07:23:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.174+0000 7f330efc9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3308071790 con 0x7f3308150520 2024-03-06T07:23:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.174+0000 7f330efc9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3308157050 con 0x7f33080c32f0 2024-03-06T07:23:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.175+0000 7f330cd65640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3308150520 0x7f33081543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.175+0000 7f3307fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33080c32f0 0x7f33081537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.175+0000 7f330d566640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3308070c80 0x7f33080c67f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:32908/0 (socket says 172.21.15.1:32908) 2024-03-06T07:23:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.175+0000 7f330d566640 1 -- 172.21.15.1:0/3547229294 learned_addr learned my addr 172.21.15.1:0/3547229294 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.175+0000 7f330cd65640 1 -- 172.21.15.1:0/3547229294 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33080c32f0 msgr2=0x7f33081537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.175+0000 7f330cd65640 1 --2- 172.21.15.1:0/3547229294 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33080c32f0 0x7f33081537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.175+0000 7f330cd65640 1 -- 172.21.15.1:0/3547229294 >> v1:172.21.15.98:6789/0 conn(0x7f3308070c80 legacy=0x7f33080c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.175+0000 7f330cd65640 1 -- 172.21.15.1:0/3547229294 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3308152fc0 con 0x7f3308150520 2024-03-06T07:23:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.175+0000 7f3307fff640 1 --2- 172.21.15.1:0/3547229294 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33080c32f0 0x7f33081537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.176+0000 7f330cd65640 1 --2- 172.21.15.1:0/3547229294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3308150520 0x7f33081543a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3300009d90 tx=0x7f330002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2af7e3d1453f90a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.176+0000 7f33077fe640 1 -- 172.21.15.1:0/3547229294 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3300039070 con 0x7f3308150520 2024-03-06T07:23:42.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.176+0000 7f33077fe640 1 -- 172.21.15.1:0/3547229294 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330002cc50 con 0x7f3308150520 2024-03-06T07:23:42.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.176+0000 7f33077fe640 1 -- 172.21.15.1:0/3547229294 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3300034440 con 0x7f3308150520 2024-03-06T07:23:42.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.176+0000 7f330efc9640 1 -- 172.21.15.1:0/3547229294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3308150520 msgr2=0x7f33081543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.176+0000 7f330efc9640 1 --2- 172.21.15.1:0/3547229294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3308150520 0x7f33081543a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3300009d90 tx=0x7f330002bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.176+0000 7f330efc9640 1 -- 172.21.15.1:0/3547229294 shutdown_connections 2024-03-06T07:23:42.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.176+0000 7f330efc9640 1 --2- 172.21.15.1:0/3547229294 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33080c32f0 0x7f33081537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.176+0000 7f330efc9640 1 --2- 172.21.15.1:0/3547229294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3308150520 0x7f33081543a0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.176+0000 7f330efc9640 1 -- 172.21.15.1:0/3547229294 >> 172.21.15.1:0/3547229294 conn(0x7f33080c88f0 msgr2=0x7f33080c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.176+0000 7f330efc9640 1 -- 172.21.15.1:0/3547229294 shutdown_connections 2024-03-06T07:23:42.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.176+0000 7f330efc9640 1 -- 172.21.15.1:0/3547229294 wait complete. 2024-03-06T07:23:42.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.177+0000 7f330efc9640 1 Processor -- start 2024-03-06T07:23:42.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.178+0000 7f330efc9640 1 -- start start 2024-03-06T07:23:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.178+0000 7f330efc9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3308070c80 0x7f330814b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.178+0000 7f330efc9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33080c32f0 0x7f33080ac150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.178+0000 7f330efc9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3308150520 0x7f33080ac690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.178+0000 7f330efc9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33080c7ec0 con 0x7f3308150520 2024-03-06T07:23:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.178+0000 7f330efc9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3308157330 con 0x7f3308070c80 2024-03-06T07:23:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.178+0000 7f330efc9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3308153d00 con 0x7f33080c32f0 2024-03-06T07:23:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.178+0000 7f330d566640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3308150520 0x7f33080ac690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.178+0000 7f330d566640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3308150520 0x7f33080ac690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50058/0 (socket says 172.21.15.1:50058) 2024-03-06T07:23:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.178+0000 7f330d566640 1 -- 172.21.15.1:0/2750651038 learned_addr learned my addr 172.21.15.1:0/2750651038 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.179+0000 7f330cd65640 1 --2- 172.21.15.1:0/2750651038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3308070c80 0x7f330814b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.179+0000 7f3307fff640 1 --2- 172.21.15.1:0/2750651038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33080c32f0 0x7f33080ac150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.179+0000 7f330d566640 1 -- 172.21.15.1:0/2750651038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33080c32f0 msgr2=0x7f33080ac150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.179+0000 7f330d566640 1 --2- 172.21.15.1:0/2750651038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33080c32f0 0x7f33080ac150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.179+0000 7f330d566640 1 -- 172.21.15.1:0/2750651038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3308070c80 msgr2=0x7f330814b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.179+0000 7f330d566640 1 --2- 172.21.15.1:0/2750651038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3308070c80 0x7f330814b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.179+0000 7f330d566640 1 -- 172.21.15.1:0/2750651038 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33081ebca0 con 0x7f3308150520 2024-03-06T07:23:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.179+0000 7f330d566640 1 --2- 172.21.15.1:0/2750651038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3308150520 0x7f33080ac690 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f32f800c420 tx=0x7f32f800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.179+0000 7f33057fa640 1 -- 172.21.15.1:0/2750651038 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f8017070 con 0x7f3308150520 2024-03-06T07:23:42.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.179+0000 7f33057fa640 1 -- 172.21.15.1:0/2750651038 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f8005ce0 con 0x7f3308150520 2024-03-06T07:23:42.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.179+0000 7f33057fa640 1 -- 172.21.15.1:0/2750651038 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f80072d0 con 0x7f3308150520 2024-03-06T07:23:42.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.179+0000 7f330efc9640 1 -- 172.21.15.1:0/2750651038 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33081ebf40 con 0x7f3308150520 2024-03-06T07:23:42.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.179+0000 7f330efc9640 1 -- 172.21.15.1:0/2750651038 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33081ec500 con 0x7f3308150520 2024-03-06T07:23:42.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.180+0000 7f330efc9640 1 -- 172.21.15.1:0/2750651038 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33080686e0 con 0x7f3308150520 2024-03-06T07:23:42.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.181+0000 7f33057fa640 1 -- 172.21.15.1:0/2750651038 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f32f801a070 con 0x7f3308150520 2024-03-06T07:23:42.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.181+0000 7f33057fa640 1 --2- 172.21.15.1:0/2750651038 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32f406b4d0 0x7f32f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.181+0000 7f33057fa640 1 -- 172.21.15.1:0/2750651038 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f32f8089650 con 0x7f3308150520 2024-03-06T07:23:42.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.181+0000 7f330cd65640 1 --2- 172.21.15.1:0/2750651038 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32f406b4d0 0x7f32f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.182+0000 7f330cd65640 1 --2- 172.21.15.1:0/2750651038 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32f406b4d0 0x7f32f406d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3300036000 tx=0x7f3300030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.185+0000 7f33057fa640 1 -- 172.21.15.1:0/2750651038 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32f8057c70 con 0x7f3308150520 2024-03-06T07:23:42.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.368+0000 7f330efc9640 1 -- 172.21.15.1:0/2750651038 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33080c5fb0 con 0x7f3308150520 2024-03-06T07:23:42.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.369+0000 7f33057fa640 1 -- 172.21.15.1:0/2750651038 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16766 (secure 0 0 0) 0x7f32f805bd10 con 0x7f3308150520 2024-03-06T07:23:42.371 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:42.371 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":94,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:23:41.076663+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:39.342951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:40.043775+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:42.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.371+0000 7f330efc9640 1 -- 172.21.15.1:0/2750651038 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32f406b4d0 msgr2=0x7f32f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.371+0000 7f330efc9640 1 --2- 172.21.15.1:0/2750651038 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32f406b4d0 0x7f32f406d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3300036000 tx=0x7f3300030000 comp rx=0 tx=0).stop 2024-03-06T07:23:42.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.371+0000 7f330efc9640 1 -- 172.21.15.1:0/2750651038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3308150520 msgr2=0x7f33080ac690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.371+0000 7f330efc9640 1 --2- 172.21.15.1:0/2750651038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3308150520 0x7f33080ac690 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f32f800c420 tx=0x7f32f800c8e0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.371+0000 7f330efc9640 1 -- 172.21.15.1:0/2750651038 shutdown_connections 2024-03-06T07:23:42.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.371+0000 7f330efc9640 1 --2- 172.21.15.1:0/2750651038 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32f406b4d0 0x7f32f406d990 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.371+0000 7f330efc9640 1 --2- 172.21.15.1:0/2750651038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3308150520 0x7f33080ac690 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.371+0000 7f330efc9640 1 --2- 172.21.15.1:0/2750651038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33080c32f0 0x7f33080ac150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.371+0000 7f330efc9640 1 --2- 172.21.15.1:0/2750651038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3308070c80 0x7f330814b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.371+0000 7f330efc9640 1 -- 172.21.15.1:0/2750651038 >> 172.21.15.1:0/2750651038 conn(0x7f33080c88f0 msgr2=0x7f330814a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.371+0000 7f330efc9640 1 -- 172.21.15.1:0/2750651038 shutdown_connections 2024-03-06T07:23:42.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.371+0000 7f330efc9640 1 -- 172.21.15.1:0/2750651038 wait complete. 2024-03-06T07:23:42.384 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:23:42.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.487+0000 7fe3a8de9640 1 Processor -- start 2024-03-06T07:23:42.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.487+0000 7fe3a8de9640 1 -- start start 2024-03-06T07:23:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.488+0000 7fe3a8de9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe3a406e670 0x7fe3a406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.488+0000 7fe3a8de9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3a40c1bc0 0x7fe3a4153290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.488+0000 7fe3a8de9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a406f060 con 0x7fe3a40a8520 2024-03-06T07:23:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.488+0000 7fe3a8de9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a40c2ca0 con 0x7fe3a406e670 2024-03-06T07:23:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.488+0000 7fe3a8de9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a40c26e0 con 0x7fe3a40c1bc0 2024-03-06T07:23:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.488+0000 7fe3a259c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe3a406e670 0x7fe3a406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.488+0000 7fe3a259c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe3a406e670 0x7fe3a406ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50064/0 (socket says 172.21.15.1:50064) 2024-03-06T07:23:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.488+0000 7fe3a259c640 1 -- 172.21.15.1:0/2914407108 learned_addr learned my addr 172.21.15.1:0/2914407108 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.488+0000 7fe3a259c640 1 -- 172.21.15.1:0/2914407108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3a40c1bc0 msgr2=0x7fe3a4153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.488+0000 7fe3a259c640 1 --2- 172.21.15.1:0/2914407108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3a40c1bc0 0x7fe3a4153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.488+0000 7fe3a259c640 1 -- 172.21.15.1:0/2914407108 >> v1:172.21.15.98:6789/0 conn(0x7fe3a40a8520 legacy=0x7fe3a4150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.488+0000 7fe3a259c640 1 -- 172.21.15.1:0/2914407108 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a40c29c0 con 0x7fe3a406e670 2024-03-06T07:23:42.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.489+0000 7fe3a259c640 1 --2- 172.21.15.1:0/2914407108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3a406e670 0x7fe3a406ea50 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fe394009870 tx=0x7fe39402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928a9262c4f95b7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.489+0000 7fe3a1d9b640 1 -- 172.21.15.1:0/2914407108 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3940317c0 con 0x7fe3a406e670 2024-03-06T07:23:42.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.489+0000 7fe3a1d9b640 1 -- 172.21.15.1:0/2914407108 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe394037070 con 0x7fe3a406e670 2024-03-06T07:23:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.489+0000 7fe3a1d9b640 1 -- 172.21.15.1:0/2914407108 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe394031b80 con 0x7fe3a406e670 2024-03-06T07:23:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.489+0000 7fe3a8de9640 1 -- 172.21.15.1:0/2914407108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3a406e670 msgr2=0x7fe3a406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.489+0000 7fe3a8de9640 1 --2- 172.21.15.1:0/2914407108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3a406e670 0x7fe3a406ea50 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fe394009870 tx=0x7fe39402b860 comp rx=0 tx=0).stop 2024-03-06T07:23:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.489+0000 7fe3a8de9640 1 -- 172.21.15.1:0/2914407108 shutdown_connections 2024-03-06T07:23:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.489+0000 7fe3a8de9640 1 --2- 172.21.15.1:0/2914407108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3a40c1bc0 0x7fe3a4153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.489+0000 7fe3a8de9640 1 --2- 172.21.15.1:0/2914407108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3a406e670 0x7fe3a406ea50 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.489+0000 7fe3a8de9640 1 -- 172.21.15.1:0/2914407108 >> 172.21.15.1:0/2914407108 conn(0x7fe3a40c3cc0 msgr2=0x7fe3a40c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.490+0000 7fe3a8de9640 1 -- 172.21.15.1:0/2914407108 shutdown_connections 2024-03-06T07:23:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.490+0000 7fe3a8de9640 1 -- 172.21.15.1:0/2914407108 wait complete. 2024-03-06T07:23:42.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.491+0000 7fe3a8de9640 1 Processor -- start 2024-03-06T07:23:42.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.491+0000 7fe3a8de9640 1 -- start start 2024-03-06T07:23:42.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.491+0000 7fe3a8de9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3a406e670 0x7fe3a4145e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.491+0000 7fe3a8de9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3a40a8520 0x7fe3a41e8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.491+0000 7fe3a8de9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3a40c1bc0 0x7fe3a41ecc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.491+0000 7fe3a8de9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a4153950 con 0x7fe3a40c1bc0 2024-03-06T07:23:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.491+0000 7fe3a8de9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a41537d0 con 0x7fe3a40a8520 2024-03-06T07:23:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.491+0000 7fe3a8de9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a4153ad0 con 0x7fe3a406e670 2024-03-06T07:23:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.492+0000 7fe3a2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3a40c1bc0 0x7fe3a41ecc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.492+0000 7fe3a2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3a40c1bc0 0x7fe3a41ecc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50070/0 (socket says 172.21.15.1:50070) 2024-03-06T07:23:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.492+0000 7fe3a259c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3a406e670 0x7fe3a4145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.492+0000 7fe3a2d9d640 1 -- 172.21.15.1:0/776125518 learned_addr learned my addr 172.21.15.1:0/776125518 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.492+0000 7fe39bfff640 1 --2- 172.21.15.1:0/776125518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3a40a8520 0x7fe3a41e8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.492+0000 7fe3a2d9d640 1 -- 172.21.15.1:0/776125518 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3a406e670 msgr2=0x7fe3a4145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.492+0000 7fe3a2d9d640 1 --2- 172.21.15.1:0/776125518 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3a406e670 0x7fe3a4145e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.492+0000 7fe3a2d9d640 1 -- 172.21.15.1:0/776125518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3a40a8520 msgr2=0x7fe3a41e8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.492+0000 7fe3a2d9d640 1 --2- 172.21.15.1:0/776125518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3a40a8520 0x7fe3a41e8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.492+0000 7fe3a2d9d640 1 -- 172.21.15.1:0/776125518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a40a9770 con 0x7fe3a40c1bc0 2024-03-06T07:23:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.492+0000 7fe3a2d9d640 1 --2- 172.21.15.1:0/776125518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3a40c1bc0 0x7fe3a41ecc00 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00b9a0 tx=0x7fe39c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.492+0000 7fe39affd640 1 -- 172.21.15.1:0/776125518 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c019070 con 0x7fe3a40c1bc0 2024-03-06T07:23:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.492+0000 7fe39affd640 1 -- 172.21.15.1:0/776125518 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39c00cbc0 con 0x7fe3a40c1bc0 2024-03-06T07:23:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.492+0000 7fe3a8de9640 1 -- 172.21.15.1:0/776125518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a41e92a0 con 0x7fe3a40c1bc0 2024-03-06T07:23:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.492+0000 7fe39affd640 1 -- 172.21.15.1:0/776125518 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c014440 con 0x7fe3a40c1bc0 2024-03-06T07:23:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.492+0000 7fe3a8de9640 1 -- 172.21.15.1:0/776125518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a4152940 con 0x7fe3a40c1bc0 2024-03-06T07:23:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.493+0000 7fe3a8de9640 1 -- 172.21.15.1:0/776125518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a4067fd0 con 0x7fe3a40c1bc0 2024-03-06T07:23:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.494+0000 7fe39affd640 1 -- 172.21.15.1:0/776125518 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe39c01c070 con 0x7fe3a40c1bc0 2024-03-06T07:23:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.494+0000 7fe39affd640 1 --2- 172.21.15.1:0/776125518 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe38406b4d0 0x7fe38406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.494+0000 7fe39affd640 1 -- 172.21.15.1:0/776125518 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe39c089fd0 con 0x7fe3a40c1bc0 2024-03-06T07:23:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.495+0000 7fe3a259c640 1 --2- 172.21.15.1:0/776125518 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe38406b4d0 0x7fe38406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.495+0000 7fe3a259c640 1 --2- 172.21.15.1:0/776125518 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe38406b4d0 0x7fe38406d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe394036000 tx=0x7fe39403d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.498+0000 7fe39affd640 1 -- 172.21.15.1:0/776125518 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe39c058700 con 0x7fe3a40c1bc0 2024-03-06T07:23:42.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.716+0000 7fe3a8de9640 1 -- 172.21.15.1:0/776125518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe3a4157060 con 0x7fe3a40c1bc0 2024-03-06T07:23:42.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.727+0000 7fe39affd640 1 -- 172.21.15.1:0/776125518 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v76) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe39c05c7a0 con 0x7fe3a40c1bc0 2024-03-06T07:23:42.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.729+0000 7fe3a8de9640 1 -- 172.21.15.1:0/776125518 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe38406b4d0 msgr2=0x7fe38406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.729+0000 7fe3a8de9640 1 --2- 172.21.15.1:0/776125518 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe38406b4d0 0x7fe38406d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe394036000 tx=0x7fe39403d040 comp rx=0 tx=0).stop 2024-03-06T07:23:42.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.729+0000 7fe3a8de9640 1 -- 172.21.15.1:0/776125518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3a40c1bc0 msgr2=0x7fe3a41ecc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.729+0000 7fe3a8de9640 1 --2- 172.21.15.1:0/776125518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3a40c1bc0 0x7fe3a41ecc00 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00b9a0 tx=0x7fe39c00be60 comp rx=0 tx=0).stop 2024-03-06T07:23:42.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.729+0000 7fe3a8de9640 1 -- 172.21.15.1:0/776125518 shutdown_connections 2024-03-06T07:23:42.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.729+0000 7fe3a8de9640 1 --2- 172.21.15.1:0/776125518 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe38406b4d0 0x7fe38406d990 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.729+0000 7fe3a8de9640 1 --2- 172.21.15.1:0/776125518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3a40c1bc0 0x7fe3a41ecc00 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.729+0000 7fe3a8de9640 1 --2- 172.21.15.1:0/776125518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3a40a8520 0x7fe3a41e8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.729+0000 7fe3a8de9640 1 --2- 172.21.15.1:0/776125518 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3a406e670 0x7fe3a4145e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.729+0000 7fe3a8de9640 1 -- 172.21.15.1:0/776125518 >> 172.21.15.1:0/776125518 conn(0x7fe3a40c3cc0 msgr2=0x7fe3a414ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.729+0000 7fe3a8de9640 1 -- 172.21.15.1:0/776125518 shutdown_connections 2024-03-06T07:23:42.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.729+0000 7fe3a8de9640 1 -- 172.21.15.1:0/776125518 wait complete. 2024-03-06T07:23:42.741 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:23:42.742 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:23:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.844+0000 7f4d47102640 1 Processor -- start 2024-03-06T07:23:42.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.844+0000 7f4d47102640 1 -- start start 2024-03-06T07:23:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.845+0000 7f4d47102640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d400c2b30 0x7f4d400724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.845+0000 7f4d47102640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d40151dd0 con 0x7f4d40152b70 2024-03-06T07:23:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.845+0000 7f4d47102640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d40156190 con 0x7f4d400a5fa0 2024-03-06T07:23:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.845+0000 7f4d47102640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d401526e0 con 0x7f4d400c2b30 2024-03-06T07:23:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.845+0000 7f4d3ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4d40152b70 0x7f4d400c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54828/0 (socket says 172.21.15.1:54828) 2024-03-06T07:23:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.845+0000 7f4d3ffff640 1 -- 172.21.15.1:0/221285057 learned_addr learned my addr 172.21.15.1:0/221285057 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.845+0000 7f4d44e9e640 1 --2- 172.21.15.1:0/221285057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d400c2b30 0x7f4d400724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.845+0000 7f4d3f7fe640 1 -- 172.21.15.1:0/221285057 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091886331 0 0) 0x7f4d40151dd0 con 0x7f4d40152b70 2024-03-06T07:23:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.845+0000 7f4d3f7fe640 1 -- 172.21.15.1:0/221285057 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d30003520 con 0x7f4d40152b70 2024-03-06T07:23:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.845+0000 7f4d3f7fe640 1 -- 172.21.15.1:0/221285057 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283679119 0 0) 0x7f4d40156190 con 0x7f4d400a5fa0 2024-03-06T07:23:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.845+0000 7f4d3f7fe640 1 -- 172.21.15.1:0/221285057 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d40151dd0 con 0x7f4d400a5fa0 2024-03-06T07:23:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.845+0000 7f4d44e9e640 1 -- 172.21.15.1:0/221285057 >> v1:172.21.15.98:6789/0 conn(0x7f4d400a5fa0 legacy=0x7f4d4015c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.845+0000 7f4d44e9e640 1 -- 172.21.15.1:0/221285057 >> v1:172.21.15.1:6789/0 conn(0x7f4d40152b70 legacy=0x7f4d400c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.845+0000 7f4d44e9e640 1 -- 172.21.15.1:0/221285057 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d40152090 con 0x7f4d400c2b30 2024-03-06T07:23:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.846+0000 7f4d44e9e640 1 --2- 172.21.15.1:0/221285057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d400c2b30 0x7f4d400724c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4d340097d0 tx=0x7f4d3402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6671c903d6c296d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.846+0000 7f4d3f7fe640 1 -- 172.21.15.1:0/221285057 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d34035020 con 0x7f4d400c2b30 2024-03-06T07:23:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.846+0000 7f4d3f7fe640 1 -- 172.21.15.1:0/221285057 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d34031890 con 0x7f4d400c2b30 2024-03-06T07:23:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.846+0000 7f4d3f7fe640 1 -- 172.21.15.1:0/221285057 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d34031c60 con 0x7f4d400c2b30 2024-03-06T07:23:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.847+0000 7f4d47102640 1 -- 172.21.15.1:0/221285057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d400c2b30 msgr2=0x7f4d400724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.847+0000 7f4d47102640 1 --2- 172.21.15.1:0/221285057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d400c2b30 0x7f4d400724c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4d340097d0 tx=0x7f4d3402b930 comp rx=0 tx=0).stop 2024-03-06T07:23:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.847+0000 7f4d47102640 1 -- 172.21.15.1:0/221285057 shutdown_connections 2024-03-06T07:23:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.847+0000 7f4d47102640 1 --2- 172.21.15.1:0/221285057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d400c2b30 0x7f4d400724c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.847+0000 7f4d47102640 1 -- 172.21.15.1:0/221285057 >> 172.21.15.1:0/221285057 conn(0x7f4d400c6860 msgr2=0x7f4d400c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.847+0000 7f4d47102640 1 -- 172.21.15.1:0/221285057 shutdown_connections 2024-03-06T07:23:42.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.847+0000 7f4d47102640 1 -- 172.21.15.1:0/221285057 wait complete. 2024-03-06T07:23:42.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.848+0000 7f4d47102640 1 Processor -- start 2024-03-06T07:23:42.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.848+0000 7f4d47102640 1 -- start start 2024-03-06T07:23:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d47102640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d400a5fa0 0x7f4d401df030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d47102640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d400c2b30 0x7f4d40154340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d47102640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d40152b70 0x7f4d401ee6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d47102640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d40155f00 con 0x7f4d400c2b30 2024-03-06T07:23:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d47102640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d40155d80 con 0x7f4d40152b70 2024-03-06T07:23:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d47102640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d40072a00 con 0x7f4d400a5fa0 2024-03-06T07:23:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d3ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d400c2b30 0x7f4d40154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d3ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d400c2b30 0x7f4d40154340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50074/0 (socket says 172.21.15.1:50074) 2024-03-06T07:23:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d3ffff640 1 -- 172.21.15.1:0/4227454943 learned_addr learned my addr 172.21.15.1:0/4227454943 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d44e9e640 1 --2- 172.21.15.1:0/4227454943 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d400a5fa0 0x7f4d401df030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d4569f640 1 --2- 172.21.15.1:0/4227454943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d40152b70 0x7f4d401ee6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d3ffff640 1 -- 172.21.15.1:0/4227454943 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d400a5fa0 msgr2=0x7f4d401df030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d3ffff640 1 --2- 172.21.15.1:0/4227454943 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d400a5fa0 0x7f4d401df030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d3ffff640 1 -- 172.21.15.1:0/4227454943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d40152b70 msgr2=0x7f4d401ee6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d3ffff640 1 --2- 172.21.15.1:0/4227454943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d40152b70 0x7f4d401ee6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d3ffff640 1 -- 172.21.15.1:0/4227454943 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d40154ea0 con 0x7f4d400c2b30 2024-03-06T07:23:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d3ffff640 1 --2- 172.21.15.1:0/4227454943 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d400c2b30 0x7f4d40154340 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c00c440 tx=0x7f4d2c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d3d7fa640 1 -- 172.21.15.1:0/4227454943 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c017070 con 0x7f4d400c2b30 2024-03-06T07:23:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d3d7fa640 1 -- 172.21.15.1:0/4227454943 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2c005ce0 con 0x7f4d400c2b30 2024-03-06T07:23:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d47102640 1 -- 172.21.15.1:0/4227454943 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d40155180 con 0x7f4d400c2b30 2024-03-06T07:23:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d3d7fa640 1 -- 172.21.15.1:0/4227454943 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c007270 con 0x7f4d400c2b30 2024-03-06T07:23:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.849+0000 7f4d47102640 1 -- 172.21.15.1:0/4227454943 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4006ebb0 con 0x7f4d400c2b30 2024-03-06T07:23:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.851+0000 7f4d47102640 1 -- 172.21.15.1:0/4227454943 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d40071790 con 0x7f4d400c2b30 2024-03-06T07:23:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.851+0000 7f4d3d7fa640 1 -- 172.21.15.1:0/4227454943 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d2c01a070 con 0x7f4d400c2b30 2024-03-06T07:23:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.851+0000 7f4d3d7fa640 1 --2- 172.21.15.1:0/4227454943 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4d2406b4d0 0x7f4d2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.851+0000 7f4d3d7fa640 1 -- 172.21.15.1:0/4227454943 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4d2c089690 con 0x7f4d400c2b30 2024-03-06T07:23:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.851+0000 7f4d44e9e640 1 --2- 172.21.15.1:0/4227454943 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4d2406b4d0 0x7f4d2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.852+0000 7f4d44e9e640 1 --2- 172.21.15.1:0/4227454943 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4d2406b4d0 0x7f4d2406d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4d34036000 tx=0x7f4d34008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:42.856+0000 7f4d3d7fa640 1 -- 172.21.15.1:0/4227454943 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d2c057c10 con 0x7f4d400c2b30 2024-03-06T07:23:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.076+0000 7f4d47102640 1 -- 172.21.15.1:0/4227454943 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4d4006f460 con 0x7f4d400c2b30 2024-03-06T07:23:43.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.734+0000 7f4d3d7fa640 1 -- 172.21.15.1:0/4227454943 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v77) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4d2c05bcb0 con 0x7f4d400c2b30 2024-03-06T07:23:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.737+0000 7f4d47102640 1 -- 172.21.15.1:0/4227454943 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4d2406b4d0 msgr2=0x7f4d2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.737+0000 7f4d47102640 1 --2- 172.21.15.1:0/4227454943 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4d2406b4d0 0x7f4d2406d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4d34036000 tx=0x7f4d34008030 comp rx=0 tx=0).stop 2024-03-06T07:23:43.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.737+0000 7f4d47102640 1 -- 172.21.15.1:0/4227454943 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d400c2b30 msgr2=0x7f4d40154340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.737+0000 7f4d47102640 1 --2- 172.21.15.1:0/4227454943 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d400c2b30 0x7f4d40154340 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c00c440 tx=0x7f4d2c00c900 comp rx=0 tx=0).stop 2024-03-06T07:23:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.737+0000 7f4d47102640 1 -- 172.21.15.1:0/4227454943 shutdown_connections 2024-03-06T07:23:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.737+0000 7f4d47102640 1 --2- 172.21.15.1:0/4227454943 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4d2406b4d0 0x7f4d2406d990 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.737+0000 7f4d47102640 1 --2- 172.21.15.1:0/4227454943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d40152b70 0x7f4d401ee6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.737+0000 7f4d47102640 1 --2- 172.21.15.1:0/4227454943 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d400c2b30 0x7f4d40154340 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.737+0000 7f4d47102640 1 --2- 172.21.15.1:0/4227454943 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d400a5fa0 0x7f4d401df030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.737+0000 7f4d47102640 1 -- 172.21.15.1:0/4227454943 >> 172.21.15.1:0/4227454943 conn(0x7f4d400c6860 msgr2=0x7f4d40150a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.737+0000 7f4d47102640 1 -- 172.21.15.1:0/4227454943 shutdown_connections 2024-03-06T07:23:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.737+0000 7f4d47102640 1 -- 172.21.15.1:0/4227454943 wait complete. 2024-03-06T07:23:43.752 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:23:43.753 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:43.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.858+0000 7fc16fb81640 1 Processor -- start 2024-03-06T07:23:43.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.858+0000 7fc16fb81640 1 -- start start 2024-03-06T07:23:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.859+0000 7fc16fb81640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc1680c2b30 0x7fc1680724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.859+0000 7fc16fb81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc168152b70 0x7fc168072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.859+0000 7fc16fb81640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc168151dd0 con 0x7fc1680a5fa0 2024-03-06T07:23:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.859+0000 7fc16fb81640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1681526e0 con 0x7fc1680c2b30 2024-03-06T07:23:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.859+0000 7fc16fb81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc168156190 con 0x7fc168152b70 2024-03-06T07:23:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.859+0000 7fc16d11c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc168152b70 0x7fc168072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.859+0000 7fc16d91d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc1680c2b30 0x7fc1680724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.859+0000 7fc16e11e640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc1680a5fa0 0x7fc1680c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:32934/0 (socket says 172.21.15.1:32934) 2024-03-06T07:23:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.859+0000 7fc16e11e640 1 -- 172.21.15.1:0/2692746534 learned_addr learned my addr 172.21.15.1:0/2692746534 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.859+0000 7fc16c91b640 1 -- 172.21.15.1:0/2692746534 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351272002 0 0) 0x7fc168151dd0 con 0x7fc1680a5fa0 2024-03-06T07:23:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.859+0000 7fc16c91b640 1 -- 172.21.15.1:0/2692746534 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc154003520 con 0x7fc1680a5fa0 2024-03-06T07:23:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.859+0000 7fc16d11c640 1 -- 172.21.15.1:0/2692746534 >> v2:172.21.15.98:3300/0 conn(0x7fc1680c2b30 msgr2=0x7fc1680724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.859+0000 7fc16d11c640 1 --2- 172.21.15.1:0/2692746534 >> v2:172.21.15.98:3300/0 conn(0x7fc1680c2b30 0x7fc1680724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.859+0000 7fc16d11c640 1 -- 172.21.15.1:0/2692746534 >> v1:172.21.15.98:6789/0 conn(0x7fc1680a5fa0 legacy=0x7fc1680c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.859+0000 7fc16d11c640 1 -- 172.21.15.1:0/2692746534 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc168152090 con 0x7fc168152b70 2024-03-06T07:23:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.859+0000 7fc16d91d640 1 --2- 172.21.15.1:0/2692746534 >> v2:172.21.15.98:3300/0 conn(0x7fc1680c2b30 0x7fc1680724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.860+0000 7fc16d11c640 1 --2- 172.21.15.1:0/2692746534 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc168152b70 0x7fc168072a00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc1580097a0 tx=0x7fc15802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66d00ad1adae06c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.860+0000 7fc16c91b640 1 -- 172.21.15.1:0/2692746534 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158035020 con 0x7fc168152b70 2024-03-06T07:23:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.860+0000 7fc16c91b640 1 -- 172.21.15.1:0/2692746534 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc158031830 con 0x7fc168152b70 2024-03-06T07:23:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.860+0000 7fc16c91b640 1 -- 172.21.15.1:0/2692746534 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158031c00 con 0x7fc168152b70 2024-03-06T07:23:43.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.861+0000 7fc16fb81640 1 -- 172.21.15.1:0/2692746534 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc168152b70 msgr2=0x7fc168072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.861+0000 7fc16fb81640 1 --2- 172.21.15.1:0/2692746534 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc168152b70 0x7fc168072a00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc1580097a0 tx=0x7fc15802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.861+0000 7fc16fb81640 1 -- 172.21.15.1:0/2692746534 shutdown_connections 2024-03-06T07:23:43.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.861+0000 7fc16fb81640 1 --2- 172.21.15.1:0/2692746534 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc168152b70 0x7fc168072a00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.861+0000 7fc16fb81640 1 --2- 172.21.15.1:0/2692746534 >> v2:172.21.15.98:3300/0 conn(0x7fc1680c2b30 0x7fc1680724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.861+0000 7fc16fb81640 1 -- 172.21.15.1:0/2692746534 >> 172.21.15.1:0/2692746534 conn(0x7fc1680c6860 msgr2=0x7fc1680c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:43.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.861+0000 7fc16fb81640 1 -- 172.21.15.1:0/2692746534 shutdown_connections 2024-03-06T07:23:43.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.861+0000 7fc16fb81640 1 -- 172.21.15.1:0/2692746534 wait complete. 2024-03-06T07:23:43.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.862+0000 7fc16fb81640 1 Processor -- start 2024-03-06T07:23:43.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.862+0000 7fc16fb81640 1 -- start start 2024-03-06T07:23:43.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.863+0000 7fc16fb81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1680a5fa0 0x7fc1681472d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.863+0000 7fc16fb81640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1680c2b30 0x7fc168142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.863+0000 7fc16fb81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc168152b70 0x7fc168142680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.863+0000 7fc16fb81640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc168155f00 con 0x7fc1680c2b30 2024-03-06T07:23:43.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.863+0000 7fc16fb81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc168155d80 con 0x7fc168152b70 2024-03-06T07:23:43.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.863+0000 7fc16fb81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1680c3a00 con 0x7fc1680a5fa0 2024-03-06T07:23:43.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.863+0000 7fc16d11c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1680c2b30 0x7fc168142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.863+0000 7fc16d11c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1680c2b30 0x7fc168142140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50082/0 (socket says 172.21.15.1:50082) 2024-03-06T07:23:43.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.863+0000 7fc16d91d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1680a5fa0 0x7fc1681472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.863+0000 7fc16d11c640 1 -- 172.21.15.1:0/2412465396 learned_addr learned my addr 172.21.15.1:0/2412465396 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.863+0000 7fc16e11e640 1 --2- 172.21.15.1:0/2412465396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc168152b70 0x7fc168142680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.863+0000 7fc16d91d640 1 -- 172.21.15.1:0/2412465396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc168152b70 msgr2=0x7fc168142680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.863+0000 7fc16d91d640 1 --2- 172.21.15.1:0/2412465396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc168152b70 0x7fc168142680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.863+0000 7fc16d91d640 1 -- 172.21.15.1:0/2412465396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1680c2b30 msgr2=0x7fc168142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.863+0000 7fc16d91d640 1 --2- 172.21.15.1:0/2412465396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1680c2b30 0x7fc168142140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.863+0000 7fc16d91d640 1 -- 172.21.15.1:0/2412465396 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc168142e20 con 0x7fc1680a5fa0 2024-03-06T07:23:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.863+0000 7fc16d91d640 1 --2- 172.21.15.1:0/2412465396 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1680a5fa0 0x7fc1681472d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc16400e900 tx=0x7fc16400edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.864+0000 7fc15e7fc640 1 -- 172.21.15.1:0/2412465396 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc164017070 con 0x7fc1680a5fa0 2024-03-06T07:23:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.864+0000 7fc15e7fc640 1 -- 172.21.15.1:0/2412465396 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc16400ca00 con 0x7fc1680a5fa0 2024-03-06T07:23:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.864+0000 7fc15e7fc640 1 -- 172.21.15.1:0/2412465396 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16400cdf0 con 0x7fc1680a5fa0 2024-03-06T07:23:43.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.864+0000 7fc16fb81640 1 -- 172.21.15.1:0/2412465396 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc168149070 con 0x7fc1680a5fa0 2024-03-06T07:23:43.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.864+0000 7fc16fb81640 1 -- 172.21.15.1:0/2412465396 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc168149630 con 0x7fc1680a5fa0 2024-03-06T07:23:43.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.865+0000 7fc16fb81640 1 -- 172.21.15.1:0/2412465396 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc168148e60 con 0x7fc1680a5fa0 2024-03-06T07:23:43.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.865+0000 7fc15e7fc640 1 -- 172.21.15.1:0/2412465396 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc16401a070 con 0x7fc1680a5fa0 2024-03-06T07:23:43.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.865+0000 7fc15e7fc640 1 --2- 172.21.15.1:0/2412465396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc14c06b4d0 0x7fc14c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.865+0000 7fc15e7fc640 1 -- 172.21.15.1:0/2412465396 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc16408a480 con 0x7fc1680a5fa0 2024-03-06T07:23:43.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.866+0000 7fc16d11c640 1 --2- 172.21.15.1:0/2412465396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc14c06b4d0 0x7fc14c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.866+0000 7fc16d11c640 1 --2- 172.21.15.1:0/2412465396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc14c06b4d0 0x7fc14c06d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc15802b910 tx=0x7fc15803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:43.870+0000 7fc15e7fc640 1 -- 172.21.15.1:0/2412465396 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc164058bb0 con 0x7fc1680a5fa0 2024-03-06T07:23:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.090+0000 7fc16fb81640 1 -- 172.21.15.1:0/2412465396 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc168067fd0 con 0x7fc1680a5fa0 2024-03-06T07:23:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.090+0000 7fc15e7fc640 1 -- 172.21.15.1:0/2412465396 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4970 (secure 0 0 0) 0x7fc16405cc50 con 0x7fc1680a5fa0 2024-03-06T07:23:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:44.092 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":77,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1335402413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1335402413},{"type":"v1","addr":"172.21.15.98:6836","nonce":1335402413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6981,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/2118550222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2118550222},{"type":"v1","addr":"172.21.15.1:6834","nonce":2118550222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/543699854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":543699854},{"type":"v1","addr":"172.21.15.1:6835","nonce":543699854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:41.077040+0000","modified":"2024-03-06T07:23:43.077835+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6996},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6996":{"gid":6996,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/3330683497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3330683497},{"type":"v1","addr":"172.21.15.98:6837","nonce":3330683497}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6996,"qdb_cluster":[6996]},"id":7}]} 2024-03-06T07:23:44.093 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 77 2024-03-06T07:23:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.092+0000 7fc16fb81640 1 -- 172.21.15.1:0/2412465396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc14c06b4d0 msgr2=0x7fc14c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.092+0000 7fc16fb81640 1 --2- 172.21.15.1:0/2412465396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc14c06b4d0 0x7fc14c06d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc15802b910 tx=0x7fc15803c040 comp rx=0 tx=0).stop 2024-03-06T07:23:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.092+0000 7fc16fb81640 1 -- 172.21.15.1:0/2412465396 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1680a5fa0 msgr2=0x7fc1681472d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.092+0000 7fc16fb81640 1 --2- 172.21.15.1:0/2412465396 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1680a5fa0 0x7fc1681472d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc16400e900 tx=0x7fc16400edc0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.092+0000 7fc16fb81640 1 -- 172.21.15.1:0/2412465396 shutdown_connections 2024-03-06T07:23:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.092+0000 7fc16fb81640 1 --2- 172.21.15.1:0/2412465396 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc14c06b4d0 0x7fc14c06d990 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.092+0000 7fc16fb81640 1 --2- 172.21.15.1:0/2412465396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc168152b70 0x7fc168142680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.092+0000 7fc16fb81640 1 --2- 172.21.15.1:0/2412465396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1680c2b30 0x7fc168142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.092+0000 7fc16fb81640 1 --2- 172.21.15.1:0/2412465396 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1680a5fa0 0x7fc1681472d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.092+0000 7fc16fb81640 1 -- 172.21.15.1:0/2412465396 >> 172.21.15.1:0/2412465396 conn(0x7fc1680c6860 msgr2=0x7fc16814d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.092+0000 7fc16fb81640 1 -- 172.21.15.1:0/2412465396 shutdown_connections 2024-03-06T07:23:44.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.092+0000 7fc16fb81640 1 -- 172.21.15.1:0/2412465396 wait complete. 2024-03-06T07:23:44.105 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.204+0000 7fc069f9a640 1 Processor -- start 2024-03-06T07:23:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.204+0000 7fc069f9a640 1 -- start start 2024-03-06T07:23:44.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.205+0000 7fc069f9a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc0640c2b30 0x7fc0640724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.205+0000 7fc069f9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc064152b70 0x7fc064072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.205+0000 7fc069f9a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc064151dd0 con 0x7fc0640a5fa0 2024-03-06T07:23:44.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.205+0000 7fc069f9a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0641526e0 con 0x7fc0640c2b30 2024-03-06T07:23:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.205+0000 7fc069f9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc064156190 con 0x7fc064152b70 2024-03-06T07:23:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.205+0000 7fc0637fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc0640c2b30 0x7fc0640724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.205+0000 7fc0637fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc0640c2b30 0x7fc0640724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50098/0 (socket says 172.21.15.1:50098) 2024-03-06T07:23:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.205+0000 7fc0637fe640 1 -- 172.21.15.1:0/1565267488 learned_addr learned my addr 172.21.15.1:0/1565267488 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.205+0000 7fc062ffd640 1 --2- 172.21.15.1:0/1565267488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc064152b70 0x7fc064072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.205+0000 7fc0637fe640 1 -- 172.21.15.1:0/1565267488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc064152b70 msgr2=0x7fc064072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.205+0000 7fc0637fe640 1 --2- 172.21.15.1:0/1565267488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc064152b70 0x7fc064072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.205+0000 7fc0637fe640 1 -- 172.21.15.1:0/1565267488 >> v1:172.21.15.98:6789/0 conn(0x7fc0640a5fa0 legacy=0x7fc0640c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.205+0000 7fc0637fe640 1 -- 172.21.15.1:0/1565267488 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc064152090 con 0x7fc0640c2b30 2024-03-06T07:23:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.206+0000 7fc0637fe640 1 --2- 172.21.15.1:0/1565267488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0640c2b30 0x7fc0640724c0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fc054009870 tx=0x7fc05402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9baae8cd748adf1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.206+0000 7fc0627fc640 1 -- 172.21.15.1:0/1565267488 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc054034070 con 0x7fc0640c2b30 2024-03-06T07:23:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.206+0000 7fc0627fc640 1 -- 172.21.15.1:0/1565267488 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc054030840 con 0x7fc0640c2b30 2024-03-06T07:23:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.206+0000 7fc0627fc640 1 -- 172.21.15.1:0/1565267488 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc054030c10 con 0x7fc0640c2b30 2024-03-06T07:23:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.207+0000 7fc069f9a640 1 -- 172.21.15.1:0/1565267488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0640c2b30 msgr2=0x7fc0640724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.207+0000 7fc069f9a640 1 --2- 172.21.15.1:0/1565267488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0640c2b30 0x7fc0640724c0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fc054009870 tx=0x7fc05402b980 comp rx=0 tx=0).stop 2024-03-06T07:23:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.207+0000 7fc069f9a640 1 -- 172.21.15.1:0/1565267488 shutdown_connections 2024-03-06T07:23:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.207+0000 7fc069f9a640 1 --2- 172.21.15.1:0/1565267488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc064152b70 0x7fc064072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.207+0000 7fc069f9a640 1 --2- 172.21.15.1:0/1565267488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0640c2b30 0x7fc0640724c0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.207+0000 7fc069f9a640 1 -- 172.21.15.1:0/1565267488 >> 172.21.15.1:0/1565267488 conn(0x7fc0640c6860 msgr2=0x7fc0640c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.207+0000 7fc069f9a640 1 -- 172.21.15.1:0/1565267488 shutdown_connections 2024-03-06T07:23:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.207+0000 7fc069f9a640 1 -- 172.21.15.1:0/1565267488 wait complete. 2024-03-06T07:23:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.208+0000 7fc069f9a640 1 Processor -- start 2024-03-06T07:23:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.208+0000 7fc069f9a640 1 -- start start 2024-03-06T07:23:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc069f9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0640a5fa0 0x7fc0641dcec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc069f9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0640c2b30 0x7fc06414df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc069f9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc064152b70 0x7fc06414a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc069f9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc064155f00 con 0x7fc0640a5fa0 2024-03-06T07:23:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc069f9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc064155d80 con 0x7fc0640c2b30 2024-03-06T07:23:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc069f9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0640c3a00 con 0x7fc064152b70 2024-03-06T07:23:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc0637fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0640a5fa0 0x7fc0641dcec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc0637fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0640a5fa0 0x7fc0641dcec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50110/0 (socket says 172.21.15.1:50110) 2024-03-06T07:23:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc0637fe640 1 -- 172.21.15.1:0/2320226515 learned_addr learned my addr 172.21.15.1:0/2320226515 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc062ffd640 1 --2- 172.21.15.1:0/2320226515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0640c2b30 0x7fc06414df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc063fff640 1 --2- 172.21.15.1:0/2320226515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc064152b70 0x7fc06414a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc0637fe640 1 -- 172.21.15.1:0/2320226515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc064152b70 msgr2=0x7fc06414a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc0637fe640 1 --2- 172.21.15.1:0/2320226515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc064152b70 0x7fc06414a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc0637fe640 1 -- 172.21.15.1:0/2320226515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0640c2b30 msgr2=0x7fc06414df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc0637fe640 1 --2- 172.21.15.1:0/2320226515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0640c2b30 0x7fc06414df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc0637fe640 1 -- 172.21.15.1:0/2320226515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0641534d0 con 0x7fc0640a5fa0 2024-03-06T07:23:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc063fff640 1 --2- 172.21.15.1:0/2320226515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc064152b70 0x7fc06414a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.209+0000 7fc0637fe640 1 --2- 172.21.15.1:0/2320226515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0640a5fa0 0x7fc0641dcec0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fc054000c00 tx=0x7fc054004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.210+0000 7fc04bfff640 1 -- 172.21.15.1:0/2320226515 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc054048070 con 0x7fc0640a5fa0 2024-03-06T07:23:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.210+0000 7fc04bfff640 1 -- 172.21.15.1:0/2320226515 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc054030640 con 0x7fc0640a5fa0 2024-03-06T07:23:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.210+0000 7fc069f9a640 1 -- 172.21.15.1:0/2320226515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0641ef010 con 0x7fc0640a5fa0 2024-03-06T07:23:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.210+0000 7fc04bfff640 1 -- 172.21.15.1:0/2320226515 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc054034040 con 0x7fc0640a5fa0 2024-03-06T07:23:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.210+0000 7fc069f9a640 1 -- 172.21.15.1:0/2320226515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc064153ed0 con 0x7fc0640a5fa0 2024-03-06T07:23:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.211+0000 7fc069f9a640 1 -- 172.21.15.1:0/2320226515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc064067fd0 con 0x7fc0640a5fa0 2024-03-06T07:23:44.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.211+0000 7fc04bfff640 1 -- 172.21.15.1:0/2320226515 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc05404b070 con 0x7fc0640a5fa0 2024-03-06T07:23:44.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.211+0000 7fc04bfff640 1 --2- 172.21.15.1:0/2320226515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc05006b4d0 0x7fc05006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.211+0000 7fc04bfff640 1 -- 172.21.15.1:0/2320226515 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc0540ac100 con 0x7fc0640a5fa0 2024-03-06T07:23:44.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.212+0000 7fc062ffd640 1 --2- 172.21.15.1:0/2320226515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc05006b4d0 0x7fc05006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.212+0000 7fc062ffd640 1 --2- 172.21.15.1:0/2320226515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc05006b4d0 0x7fc05006d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc0641eef40 tx=0x7fc058009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.216+0000 7fc04bfff640 1 -- 172.21.15.1:0/2320226515 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc054030810 con 0x7fc0640a5fa0 2024-03-06T07:23:44.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.397+0000 7fc069f9a640 1 -- 172.21.15.1:0/2320226515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc064071790 con 0x7fc0640a5fa0 2024-03-06T07:23:44.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.398+0000 7fc04bfff640 1 -- 172.21.15.1:0/2320226515 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16753 (secure 0 0 0) 0x7fc05407a950 con 0x7fc0640a5fa0 2024-03-06T07:23:44.400 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:44.401 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":95,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:23:42.720499+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:39.342951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:40.043775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.400+0000 7fc069f9a640 1 -- 172.21.15.1:0/2320226515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc05006b4d0 msgr2=0x7fc05006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.400+0000 7fc069f9a640 1 --2- 172.21.15.1:0/2320226515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc05006b4d0 0x7fc05006d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc0641eef40 tx=0x7fc058009190 comp rx=0 tx=0).stop 2024-03-06T07:23:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.400+0000 7fc069f9a640 1 -- 172.21.15.1:0/2320226515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0640a5fa0 msgr2=0x7fc0641dcec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.400+0000 7fc069f9a640 1 --2- 172.21.15.1:0/2320226515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0640a5fa0 0x7fc0641dcec0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fc054000c00 tx=0x7fc054004730 comp rx=0 tx=0).stop 2024-03-06T07:23:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.400+0000 7fc069f9a640 1 -- 172.21.15.1:0/2320226515 shutdown_connections 2024-03-06T07:23:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.400+0000 7fc069f9a640 1 --2- 172.21.15.1:0/2320226515 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc05006b4d0 0x7fc05006d990 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.400+0000 7fc069f9a640 1 --2- 172.21.15.1:0/2320226515 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc064152b70 0x7fc06414a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.400+0000 7fc069f9a640 1 --2- 172.21.15.1:0/2320226515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0640c2b30 0x7fc06414df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.400+0000 7fc069f9a640 1 --2- 172.21.15.1:0/2320226515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0640a5fa0 0x7fc0641dcec0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.400+0000 7fc069f9a640 1 -- 172.21.15.1:0/2320226515 >> 172.21.15.1:0/2320226515 conn(0x7fc0640c6860 msgr2=0x7fc0641512a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.400+0000 7fc069f9a640 1 -- 172.21.15.1:0/2320226515 shutdown_connections 2024-03-06T07:23:44.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.400+0000 7fc069f9a640 1 -- 172.21.15.1:0/2320226515 wait complete. 2024-03-06T07:23:44.414 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:23:44.414 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.514+0000 7f818e38a640 1 Processor -- start 2024-03-06T07:23:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.514+0000 7f818e38a640 1 -- start start 2024-03-06T07:23:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.515+0000 7f818e38a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8188156cd0 0x7f8188157120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.515+0000 7f818e38a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81880c3300 con 0x7f8188152690 2024-03-06T07:23:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.515+0000 7f818e38a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81880c35a0 con 0x7f81880c6730 2024-03-06T07:23:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.515+0000 7f818e38a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81880c3880 con 0x7f8188156cd0 2024-03-06T07:23:44.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.515+0000 7f8187fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8188152690 0x7f8188155ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36014/0 (socket says 172.21.15.1:36014) 2024-03-06T07:23:44.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.515+0000 7f8187fff640 1 -- 172.21.15.1:0/2946502541 learned_addr learned my addr 172.21.15.1:0/2946502541 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:44.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.515+0000 7f818c927640 1 --2- 172.21.15.1:0/2946502541 >> v2:172.21.15.98:3300/0 conn(0x7f8188156cd0 0x7f8188157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.515+0000 7f8186ffd640 1 -- 172.21.15.1:0/2946502541 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18761490 0 0) 0x7f81880c3300 con 0x7f8188152690 2024-03-06T07:23:44.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.515+0000 7f8186ffd640 1 -- 172.21.15.1:0/2946502541 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8170003520 con 0x7f8188152690 2024-03-06T07:23:44.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.515+0000 7f818c927640 1 -- 172.21.15.1:0/2946502541 >> v1:172.21.15.98:6789/0 conn(0x7f81880c6730 legacy=0x7f81880c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.515+0000 7f8186ffd640 1 -- 172.21.15.1:0/2946502541 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124584008 0 0) 0x7f81880c35a0 con 0x7f81880c6730 2024-03-06T07:23:44.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.515+0000 7f818c927640 1 -- 172.21.15.1:0/2946502541 >> v1:172.21.15.1:6789/0 conn(0x7f8188152690 legacy=0x7f8188155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.515+0000 7f818c927640 1 -- 172.21.15.1:0/2946502541 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81880c3b60 con 0x7f8188156cd0 2024-03-06T07:23:44.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.516+0000 7f818c927640 1 --2- 172.21.15.1:0/2946502541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8188156cd0 0x7f8188157120 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8180009d90 tx=0x7f818002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7716d2341a2b216e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.516+0000 7f8186ffd640 1 -- 172.21.15.1:0/2946502541 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180039070 con 0x7f8188156cd0 2024-03-06T07:23:44.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.516+0000 7f8186ffd640 1 -- 172.21.15.1:0/2946502541 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818002cc60 con 0x7f8188156cd0 2024-03-06T07:23:44.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.516+0000 7f8186ffd640 1 -- 172.21.15.1:0/2946502541 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180034640 con 0x7f8188156cd0 2024-03-06T07:23:44.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.517+0000 7f818e38a640 1 -- 172.21.15.1:0/2946502541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8188156cd0 msgr2=0x7f8188157120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.517+0000 7f818e38a640 1 --2- 172.21.15.1:0/2946502541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8188156cd0 0x7f8188157120 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8180009d90 tx=0x7f818002beb0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.517+0000 7f818e38a640 1 -- 172.21.15.1:0/2946502541 shutdown_connections 2024-03-06T07:23:44.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.517+0000 7f818e38a640 1 --2- 172.21.15.1:0/2946502541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8188156cd0 0x7f8188157120 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.517+0000 7f818e38a640 1 -- 172.21.15.1:0/2946502541 >> 172.21.15.1:0/2946502541 conn(0x7f818806b410 msgr2=0x7f818806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.517+0000 7f818e38a640 1 -- 172.21.15.1:0/2946502541 shutdown_connections 2024-03-06T07:23:44.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.517+0000 7f818e38a640 1 -- 172.21.15.1:0/2946502541 wait complete. 2024-03-06T07:23:44.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.518+0000 7f818e38a640 1 Processor -- start 2024-03-06T07:23:44.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.518+0000 7f818e38a640 1 -- start start 2024-03-06T07:23:44.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f818e38a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81880c6730 0x7f81880ada00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f818e38a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8188152690 0x7f81881eb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f818e38a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8188156cd0 0x7f81880abe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f818e38a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81880c6060 con 0x7f81880c6730 2024-03-06T07:23:44.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f818e38a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81880c5ee0 con 0x7f8188152690 2024-03-06T07:23:44.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f818e38a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8188152a70 con 0x7f8188156cd0 2024-03-06T07:23:44.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f8187fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81880c6730 0x7f81880ada00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f8187fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81880c6730 0x7f81880ada00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36862/0 (socket says 172.21.15.1:36862) 2024-03-06T07:23:44.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f8187fff640 1 -- 172.21.15.1:0/611465650 learned_addr learned my addr 172.21.15.1:0/611465650 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:44.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f81877fe640 1 --2- 172.21.15.1:0/611465650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8188152690 0x7f81881eb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f818c927640 1 --2- 172.21.15.1:0/611465650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8188156cd0 0x7f81880abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f8187fff640 1 -- 172.21.15.1:0/611465650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8188156cd0 msgr2=0x7f81880abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f8187fff640 1 --2- 172.21.15.1:0/611465650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8188156cd0 0x7f81880abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f8187fff640 1 -- 172.21.15.1:0/611465650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8188152690 msgr2=0x7f81881eb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f8187fff640 1 --2- 172.21.15.1:0/611465650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8188152690 0x7f81881eb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f8187fff640 1 -- 172.21.15.1:0/611465650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81880c59a0 con 0x7f81880c6730 2024-03-06T07:23:44.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f81877fe640 1 --2- 172.21.15.1:0/611465650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8188152690 0x7f81881eb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:44.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f8187fff640 1 --2- 172.21.15.1:0/611465650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81880c6730 0x7f81880ada00 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f81780010e0 tx=0x7f817800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f8184ff9640 1 -- 172.21.15.1:0/611465650 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8178017070 con 0x7f81880c6730 2024-03-06T07:23:44.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f8184ff9640 1 -- 172.21.15.1:0/611465650 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8178005d00 con 0x7f81880c6730 2024-03-06T07:23:44.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f818e38a640 1 -- 172.21.15.1:0/611465650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8188153860 con 0x7f81880c6730 2024-03-06T07:23:44.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f8184ff9640 1 -- 172.21.15.1:0/611465650 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8178007310 con 0x7f81880c6730 2024-03-06T07:23:44.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.519+0000 7f818e38a640 1 -- 172.21.15.1:0/611465650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81881ea930 con 0x7f81880c6730 2024-03-06T07:23:44.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.520+0000 7f818e38a640 1 -- 172.21.15.1:0/611465650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8188067fd0 con 0x7f81880c6730 2024-03-06T07:23:44.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.521+0000 7f8184ff9640 1 -- 172.21.15.1:0/611465650 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f817801a070 con 0x7f81880c6730 2024-03-06T07:23:44.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.521+0000 7f8184ff9640 1 --2- 172.21.15.1:0/611465650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f817406b4d0 0x7f817406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.521+0000 7f8184ff9640 1 -- 172.21.15.1:0/611465650 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f81780896d0 con 0x7f81880c6730 2024-03-06T07:23:44.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.521+0000 7f81877fe640 1 --2- 172.21.15.1:0/611465650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f817406b4d0 0x7f817406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.522+0000 7f81877fe640 1 --2- 172.21.15.1:0/611465650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f817406b4d0 0x7f817406d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f81880689e0 tx=0x7f817c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.526+0000 7f8184ff9640 1 -- 172.21.15.1:0/611465650 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8178057d70 con 0x7f81880c6730 2024-03-06T07:23:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.705+0000 7f818e38a640 1 -- 172.21.15.1:0/611465650 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f81880c2c40 con 0x7f817406b4d0 2024-03-06T07:23:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.711+0000 7f8184ff9640 1 -- 172.21.15.1:0/611465650 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240223 (secure 0 0 0) 0x7f81880c2c40 con 0x7f817406b4d0 2024-03-06T07:23:44.716 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:44.717 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:23:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.714+0000 7f818e38a640 1 -- 172.21.15.1:0/611465650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f817406b4d0 msgr2=0x7f817406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.714+0000 7f818e38a640 1 --2- 172.21.15.1:0/611465650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f817406b4d0 0x7f817406d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f81880689e0 tx=0x7f817c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.714+0000 7f818e38a640 1 -- 172.21.15.1:0/611465650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81880c6730 msgr2=0x7f81880ada00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.714+0000 7f818e38a640 1 --2- 172.21.15.1:0/611465650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81880c6730 0x7f81880ada00 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f81780010e0 tx=0x7f817800c890 comp rx=0 tx=0).stop 2024-03-06T07:23:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.714+0000 7f818e38a640 1 -- 172.21.15.1:0/611465650 shutdown_connections 2024-03-06T07:23:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.714+0000 7f818e38a640 1 --2- 172.21.15.1:0/611465650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f817406b4d0 0x7f817406d990 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.714+0000 7f818e38a640 1 --2- 172.21.15.1:0/611465650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8188156cd0 0x7f81880abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.714+0000 7f818e38a640 1 --2- 172.21.15.1:0/611465650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8188152690 0x7f81881eb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.714+0000 7f818e38a640 1 --2- 172.21.15.1:0/611465650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81880c6730 0x7f81880ada00 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.714+0000 7f818e38a640 1 -- 172.21.15.1:0/611465650 >> 172.21.15.1:0/611465650 conn(0x7f818806b410 msgr2=0x7f818806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.714+0000 7f818e38a640 1 -- 172.21.15.1:0/611465650 shutdown_connections 2024-03-06T07:23:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.714+0000 7f818e38a640 1 -- 172.21.15.1:0/611465650 wait complete. 2024-03-06T07:23:44.728 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-03-06T07:23:42.737320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254252,"kb_used_data":8948,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754720468,"statfs":{"total":773094113280,"available":772833759232,"internally_reserved":0,"allocated":9162752,"data_stored":5521655,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000802"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.705505+0000","last_change":"2024-03-06T07:23:39.705505+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.705505+0000","last_undegraded":"2024-03-06T07:23:39.705505+0000","last_fullsized":"2024-03-06T07:23:39.705505+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.703061+0000","last_change":"2024-03-06T07:23:39.703061+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.703061+0000","last_undegraded":"2024-03-06T07:23:39.703061+0000","last_fullsized":"2024-03-06T07:23:39.703061+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.706356+0000","last_change":"2024-03-06T07:23:39.706356+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.706356+0000","last_undegraded":"2024-03-06T07:23:39.706356+0000","last_fullsized":"2024-03-06T07:23:39.706356+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.706381+0000","last_change":"2024-03-06T07:23:39.706381+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.706381+0000","last_undegraded":"2024-03-06T07:23:39.706381+0000","last_fullsized":"2024-03-06T07:23:39.706381+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.706971+0000","last_change":"2024-03-06T07:23:39.706971+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.706971+0000","last_undegraded":"2024-03-06T07:23:39.706971+0000","last_fullsized":"2024-03-06T07:23:39.706971+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.706671+0000","last_change":"2024-03-06T07:23:39.706671+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.706671+0000","last_undegraded":"2024-03-06T07:23:39.706671+0000","last_fullsized":"2024-03-06T07:23:39.706671+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.705828+0000","last_change":"2024-03-06T07:23:39.705828+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.705828+0000","last_undegraded":"2024-03-06T07:23:39.705828+0000","last_fullsized":"2024-03-06T07:23:39.705828+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.705167+0000","last_change":"2024-03-06T07:23:39.705167+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.705167+0000","last_undegraded":"2024-03-06T07:23:39.705167+0000","last_fullsized":"2024-03-06T07:23:39.705167+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.706077+0000","last_change":"2024-03-06T07:23:39.706077+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.706077+0000","last_undegraded":"2024-03-06T07:23:39.706077+0000","last_fullsized":"2024-03-06T07:23:39.706077+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.704498+0000","last_change":"2024-03-06T07:23:39.704498+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.704498+0000","last_undegraded":"2024-03-06T07:23:39.704498+0000","last_fullsized":"2024-03-06T07:23:39.704498+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.706286+0000","last_change":"2024-03-06T07:23:39.706286+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.706286+0000","last_undegraded":"2024-03-06T07:23:39.706286+0000","last_fullsized":"2024-03-06T07:23:39.706286+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.710173+0000","last_change":"2024-03-06T07:23:39.710173+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.710173+0000","last_undegraded":"2024-03-06T07:23:39.710173+0000","last_fullsized":"2024-03-06T07:23:39.710173+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.705802+0000","last_change":"2024-03-06T07:23:39.705802+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.705802+0000","last_undegraded":"2024-03-06T07:23:39.705802+0000","last_fullsized":"2024-03-06T07:23:39.705802+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.704452+0000","last_change":"2024-03-06T07:23:39.704452+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.704452+0000","last_undegraded":"2024-03-06T07:23:39.704452+0000","last_fullsized":"2024-03-06T07:23:39.704452+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.702675+0000","last_change":"2024-03-06T07:23:39.702675+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.702675+0000","last_undegraded":"2024-03-06T07:23:39.702675+0000","last_fullsized":"2024-03-06T07:23:39.702675+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.705787+0000","last_change":"2024-03-06T07:23:39.705787+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.705787+0000","last_undegraded":"2024-03-06T07:23:39.705787+0000","last_fullsized":"2024-03-06T07:23:39.705787+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.710171+0000","last_change":"2024-03-06T07:23:39.710171+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.710171+0000","last_undegraded":"2024-03-06T07:23:39.710171+0000","last_fullsized":"2024-03-06T07:23:39.710171+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.705787+0000","last_change":"2024-03-06T07:23:39.705787+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.705787+0000","last_undegraded":"2024-03-06T07:23:39.705787+0000","last_fullsized":"2024-03-06T07:23:39.705787+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.706764+0000","last_change":"2024-03-06T07:23:39.706764+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.706764+0000","last_undegraded":"2024-03-06T07:23:39.706764+0000","last_fullsized":"2024-03-06T07:23:39.706764+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.702205+0000","last_change":"2024-03-06T07:23:39.702205+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.702205+0000","last_undegraded":"2024-03-06T07:23:39.702205+0000","last_fullsized":"2024-03-06T07:23:39.702205+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.702141+0000","last_change":"2024-03-06T07:23:39.702141+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.702141+0000","last_undegraded":"2024-03-06T07:23:39.702141+0000","last_fullsized":"2024-03-06T07:23:39.702141+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.710943+0000","last_change":"2024-03-06T07:23:39.710943+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.710943+0000","last_undegraded":"2024-03-06T07:23:39.710943+0000","last_fullsized":"2024-03-06T07:23:39.710943+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.702143+0000","last_change":"2024-03-06T07:23:39.702143+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.702143+0000","last_undegraded":"2024-03-06T07:23:39.702143+0000","last_fullsized":"2024-03-06T07:23:39.702143+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.705929+0000","last_change":"2024-03-06T07:23:39.705929+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.705929+0000","last_undegraded":"2024-03-06T07:23:39.705929+0000","last_fullsized":"2024-03-06T07:23:39.705929+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.704008+0000","last_change":"2024-03-06T07:23:39.704008+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.704008+0000","last_undegraded":"2024-03-06T07:23:39.704008+0000","last_fullsized":"2024-03-06T07:23:39.704008+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.710160+0000","last_change":"2024-03-06T07:23:39.710160+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.710160+0000","last_undegraded":"2024-03-06T07:23:39.710160+0000","last_fullsized":"2024-03-06T07:23:39.710160+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.705884+0000","last_change":"2024-03-06T07:23:39.705884+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.705884+0000","last_undegraded":"2024-03-06T07:23:39.705884+0000","last_fullsized":"2024-03-06T07:23:39.705884+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.705855+0000","last_change":"2024-03-06T07:23:39.705855+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.705855+0000","last_undegraded":"2024-03-06T07:23:39.705855+0000","last_fullsized":"2024-03-06T07:23:39.705855+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.701627+0000","last_change":"2024-03-06T07:23:39.701627+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.701627+0000","last_undegraded":"2024-03-06T07:23:39.701627+0000","last_fullsized":"2024-03-06T07:23:39.701627+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.710330+0000","last_change":"2024-03-06T07:23:39.710330+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.710330+0000","last_undegraded":"2024-03-06T07:23:39.710330+0000","last_fullsized":"2024-03-06T07:23:39.710330+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.701635+0000","last_change":"2024-03-06T07:23:39.701635+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.701635+0000","last_undegraded":"2024-03-06T07:23:39.701635+0000","last_fullsized":"2024-03-06T07:23:39.701635+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:40.698942+0000","last_change":"2024-03-06T07:23:40.698942+0000","last_active":"2024-03-06T07:23:40.698942+0000","last_peered":"2024-03-06T07:23:40.698942+0000","last_clean":"2024-03-06T07:23:40.698942+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.698942+0000","last_undegraded":"2024-03-06T07:23:40.698942+0000","last_fullsized":"2024-03-06T07:23:40.698942+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":1,"reported_epoch":92,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.702222+0000","last_change":"2024-03-06T07:23:39.702222+0000","last_active":"2024-03-06T07:23:39.694297+0000","last_peered":"2024-03-06T07:23:39.694297+0000","last_clean":"2024-03-06T07:23:39.694297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.702222+0000","last_undegraded":"2024-03-06T07:23:39.702222+0000","last_fullsized":"2024-03-06T07:23:39.702222+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":308,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.706659+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:23:39.706659+0000","last_peered":"2024-03-06T07:23:39.706659+0000","last_clean":"2024-03-06T07:23:39.706659+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:23:39.706659+0000","last_undegraded":"2024-03-06T07:23:39.706659+0000","last_fullsized":"2024-03-06T07:23:39.706659+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32348,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339492,"statfs":{"total":96636764160,"available":96603639808,"internally_reserved":0,"allocated":1196032,"data_stored":1133699,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32288,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339552,"statfs":{"total":96636764160,"available":96603701248,"internally_reserved":0,"allocated":1130496,"data_stored":543331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343712,"statfs":{"total":96636764160,"available":96607961088,"internally_reserved":0,"allocated":1130496,"data_stored":543331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1130496,"data_stored":543331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1130496,"data_stored":543331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1130496,"data_stored":543331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1130496,"data_stored":543331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32344,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339496,"statfs":{"total":96636764160,"available":96603643904,"internally_reserved":0,"allocated":1183744,"data_stored":1127970,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:44.732 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:44.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.867+0000 7fd77f015640 1 Processor -- start 2024-03-06T07:23:44.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.867+0000 7fd77f015640 1 -- start start 2024-03-06T07:23:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.868+0000 7fd77f015640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd778152650 0x7fd778155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.868+0000 7fd77f015640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7780c6b10 0x7fd778158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.868+0000 7fd77f015640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7780bf9d0 0x7fd7780c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.868+0000 7fd77f015640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd778155800 con 0x7fd7780c6b10 2024-03-06T07:23:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.868+0000 7fd77f015640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7780c2fc0 con 0x7fd778152650 2024-03-06T07:23:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.868+0000 7fd77f015640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7780c6690 con 0x7fd7780bf9d0 2024-03-06T07:23:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.868+0000 7fd777fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7780c6b10 0x7fd778158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.868+0000 7fd777fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7780c6b10 0x7fd778158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36864/0 (socket says 172.21.15.1:36864) 2024-03-06T07:23:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.868+0000 7fd777fff640 1 -- 172.21.15.1:0/2205439626 learned_addr learned my addr 172.21.15.1:0/2205439626 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.868+0000 7fd77d5b2640 1 --2- 172.21.15.1:0/2205439626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7780bf9d0 0x7fd7780c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.868+0000 7fd77cdb1640 1 --2- 172.21.15.1:0/2205439626 >> v2:172.21.15.98:3300/0 conn(0x7fd778152650 0x7fd778155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.869+0000 7fd777fff640 1 -- 172.21.15.1:0/2205439626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7780bf9d0 msgr2=0x7fd7780c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.869+0000 7fd777fff640 1 --2- 172.21.15.1:0/2205439626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7780bf9d0 0x7fd7780c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.869+0000 7fd777fff640 1 -- 172.21.15.1:0/2205439626 >> v2:172.21.15.98:3300/0 conn(0x7fd778152650 msgr2=0x7fd778155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.869+0000 7fd777fff640 1 --2- 172.21.15.1:0/2205439626 >> v2:172.21.15.98:3300/0 conn(0x7fd778152650 0x7fd778155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.869+0000 7fd777fff640 1 -- 172.21.15.1:0/2205439626 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7780c7280 con 0x7fd7780c6b10 2024-03-06T07:23:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.869+0000 7fd777fff640 1 --2- 172.21.15.1:0/2205439626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7780c6b10 0x7fd778158690 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fd7780cac90 tx=0x7fd76402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee78cab93eab0b39 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.869+0000 7fd7777fe640 1 -- 172.21.15.1:0/2205439626 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76402e070 con 0x7fd7780c6b10 2024-03-06T07:23:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.870+0000 7fd7777fe640 1 -- 172.21.15.1:0/2205439626 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd764037070 con 0x7fd7780c6b10 2024-03-06T07:23:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.870+0000 7fd7777fe640 1 -- 172.21.15.1:0/2205439626 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd764031a00 con 0x7fd7780c6b10 2024-03-06T07:23:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.870+0000 7fd77f015640 1 -- 172.21.15.1:0/2205439626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7780c6b10 msgr2=0x7fd778158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.870+0000 7fd77f015640 1 --2- 172.21.15.1:0/2205439626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7780c6b10 0x7fd778158690 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fd7780cac90 tx=0x7fd76402b880 comp rx=0 tx=0).stop 2024-03-06T07:23:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.870+0000 7fd77f015640 1 -- 172.21.15.1:0/2205439626 shutdown_connections 2024-03-06T07:23:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.870+0000 7fd77f015640 1 --2- 172.21.15.1:0/2205439626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7780bf9d0 0x7fd7780c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.870+0000 7fd77f015640 1 --2- 172.21.15.1:0/2205439626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7780c6b10 0x7fd778158690 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.870+0000 7fd77f015640 1 --2- 172.21.15.1:0/2205439626 >> v2:172.21.15.98:3300/0 conn(0x7fd778152650 0x7fd778155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.870+0000 7fd77f015640 1 -- 172.21.15.1:0/2205439626 >> 172.21.15.1:0/2205439626 conn(0x7fd778066ee0 msgr2=0x7fd7780672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.870+0000 7fd77f015640 1 -- 172.21.15.1:0/2205439626 shutdown_connections 2024-03-06T07:23:44.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.870+0000 7fd77f015640 1 -- 172.21.15.1:0/2205439626 wait complete. 2024-03-06T07:23:44.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.871+0000 7fd77f015640 1 Processor -- start 2024-03-06T07:23:44.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.871+0000 7fd77f015640 1 -- start start 2024-03-06T07:23:44.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.872+0000 7fd77f015640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7780bf9d0 0x7fd77814d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.872+0000 7fd77f015640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd778152650 0x7fd778141d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.872+0000 7fd77f015640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd778154d70 0x7fd7781422d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.872+0000 7fd77f015640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd778155490 con 0x7fd778154d70 2024-03-06T07:23:44.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.872+0000 7fd77f015640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7781551b0 con 0x7fd7780bf9d0 2024-03-06T07:23:44.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.872+0000 7fd77f015640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd778152a30 con 0x7fd778152650 2024-03-06T07:23:44.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.872+0000 7fd77d5b2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd778154d70 0x7fd7781422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.872+0000 7fd77d5b2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd778154d70 0x7fd7781422d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36880/0 (socket says 172.21.15.1:36880) 2024-03-06T07:23:44.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.872+0000 7fd77d5b2640 1 -- 172.21.15.1:0/1399711905 learned_addr learned my addr 172.21.15.1:0/1399711905 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:44.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.872+0000 7fd777fff640 1 --2- 172.21.15.1:0/1399711905 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd778152650 0x7fd778141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.872+0000 7fd77cdb1640 1 --2- 172.21.15.1:0/1399711905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7780bf9d0 0x7fd77814d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.872+0000 7fd77d5b2640 1 -- 172.21.15.1:0/1399711905 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd778152650 msgr2=0x7fd778141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.872+0000 7fd77d5b2640 1 --2- 172.21.15.1:0/1399711905 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd778152650 0x7fd778141d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.872+0000 7fd77d5b2640 1 -- 172.21.15.1:0/1399711905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7780bf9d0 msgr2=0x7fd77814d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.872+0000 7fd77d5b2640 1 --2- 172.21.15.1:0/1399711905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7780bf9d0 0x7fd77814d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.872+0000 7fd77d5b2640 1 -- 172.21.15.1:0/1399711905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd778148b50 con 0x7fd778154d70 2024-03-06T07:23:44.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.873+0000 7fd77d5b2640 1 --2- 172.21.15.1:0/1399711905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd778154d70 0x7fd7781422d0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fd77000ef00 tx=0x7fd77000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.873+0000 7fd7757fa640 1 -- 172.21.15.1:0/1399711905 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd770018020 con 0x7fd778154d70 2024-03-06T07:23:44.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.873+0000 7fd7757fa640 1 -- 172.21.15.1:0/1399711905 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77000bd20 con 0x7fd778154d70 2024-03-06T07:23:44.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.873+0000 7fd77f015640 1 -- 172.21.15.1:0/1399711905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7781eede0 con 0x7fd778154d70 2024-03-06T07:23:44.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.873+0000 7fd7757fa640 1 -- 172.21.15.1:0/1399711905 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd770014640 con 0x7fd778154d70 2024-03-06T07:23:44.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.873+0000 7fd77f015640 1 -- 172.21.15.1:0/1399711905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd778142a80 con 0x7fd778154d70 2024-03-06T07:23:44.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.874+0000 7fd77f015640 1 -- 172.21.15.1:0/1399711905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd778067fd0 con 0x7fd778154d70 2024-03-06T07:23:44.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.874+0000 7fd7757fa640 1 -- 172.21.15.1:0/1399711905 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd770005d10 con 0x7fd778154d70 2024-03-06T07:23:44.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.875+0000 7fd7757fa640 1 --2- 172.21.15.1:0/1399711905 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd75c06b4d0 0x7fd75c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.875+0000 7fd7757fa640 1 -- 172.21.15.1:0/1399711905 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd77008a8a0 con 0x7fd778154d70 2024-03-06T07:23:44.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.875+0000 7fd77cdb1640 1 --2- 172.21.15.1:0/1399711905 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd75c06b4d0 0x7fd75c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.875+0000 7fd77cdb1640 1 --2- 172.21.15.1:0/1399711905 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd75c06b4d0 0x7fd75c06d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd76c009b40 tx=0x7fd76c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:44.879+0000 7fd7757fa640 1 -- 172.21.15.1:0/1399711905 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd770022070 con 0x7fd778154d70 2024-03-06T07:23:45.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.056+0000 7fd77f015640 1 -- 172.21.15.1:0/1399711905 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd778142830 con 0x7fd75c06b4d0 2024-03-06T07:23:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.062+0000 7fd7757fa640 1 -- 172.21.15.1:0/1399711905 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252158 (secure 0 0 0) 0x7fd778142830 con 0x7fd75c06b4d0 2024-03-06T07:23:45.066 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:23:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.065+0000 7fd77f015640 1 -- 172.21.15.1:0/1399711905 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd75c06b4d0 msgr2=0x7fd75c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.065+0000 7fd77f015640 1 --2- 172.21.15.1:0/1399711905 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd75c06b4d0 0x7fd75c06d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd76c009b40 tx=0x7fd76c009190 comp rx=0 tx=0).stop 2024-03-06T07:23:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.065+0000 7fd77f015640 1 -- 172.21.15.1:0/1399711905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd778154d70 msgr2=0x7fd7781422d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.065+0000 7fd77f015640 1 --2- 172.21.15.1:0/1399711905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd778154d70 0x7fd7781422d0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fd77000ef00 tx=0x7fd77000b480 comp rx=0 tx=0).stop 2024-03-06T07:23:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.065+0000 7fd77f015640 1 -- 172.21.15.1:0/1399711905 shutdown_connections 2024-03-06T07:23:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.065+0000 7fd77f015640 1 --2- 172.21.15.1:0/1399711905 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd75c06b4d0 0x7fd75c06d990 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.065+0000 7fd77f015640 1 --2- 172.21.15.1:0/1399711905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd778154d70 0x7fd7781422d0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.065+0000 7fd77f015640 1 --2- 172.21.15.1:0/1399711905 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd778152650 0x7fd778141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.065+0000 7fd77f015640 1 --2- 172.21.15.1:0/1399711905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7780bf9d0 0x7fd77814d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.065+0000 7fd77f015640 1 -- 172.21.15.1:0/1399711905 >> 172.21.15.1:0/1399711905 conn(0x7fd778066ee0 msgr2=0x7fd7780c3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.065+0000 7fd77f015640 1 -- 172.21.15.1:0/1399711905 shutdown_connections 2024-03-06T07:23:45.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.065+0000 7fd77f015640 1 -- 172.21.15.1:0/1399711905 wait complete. 2024-03-06T07:23:45.077 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-03-06T07:23:44.737639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":134,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258876,"kb_used_data":9408,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715844,"statfs":{"total":773094113280,"available":772829024256,"internally_reserved":0,"allocated":9633792,"data_stored":5746084,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.033499"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.086446+0000","last_change":"2024-03-06T07:23:41.728456+0000","last_active":"2024-03-06T07:23:43.086446+0000","last_peered":"2024-03-06T07:23:43.086446+0000","last_clean":"2024-03-06T07:23:43.086446+0000","last_became_active":"2024-03-06T07:23:41.727804+0000","last_became_peered":"2024-03-06T07:23:41.727804+0000","last_unstale":"2024-03-06T07:23:43.086446+0000","last_undegraded":"2024-03-06T07:23:43.086446+0000","last_fullsized":"2024-03-06T07:23:43.086446+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:38.903702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.700034+0000","last_change":"2024-03-06T07:23:41.740248+0000","last_active":"2024-03-06T07:23:43.700034+0000","last_peered":"2024-03-06T07:23:43.700034+0000","last_clean":"2024-03-06T07:23:43.700034+0000","last_became_active":"2024-03-06T07:23:41.734977+0000","last_became_peered":"2024-03-06T07:23:41.734977+0000","last_unstale":"2024-03-06T07:23:43.700034+0000","last_undegraded":"2024-03-06T07:23:43.700034+0000","last_fullsized":"2024-03-06T07:23:43.700034+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:11.221545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.086342+0000","last_change":"2024-03-06T07:23:41.733652+0000","last_active":"2024-03-06T07:23:43.086342+0000","last_peered":"2024-03-06T07:23:43.086342+0000","last_clean":"2024-03-06T07:23:43.086342+0000","last_became_active":"2024-03-06T07:23:41.733054+0000","last_became_peered":"2024-03-06T07:23:41.733054+0000","last_unstale":"2024-03-06T07:23:43.086342+0000","last_undegraded":"2024-03-06T07:23:43.086342+0000","last_fullsized":"2024-03-06T07:23:43.086342+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:53.953703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.727298+0000","last_change":"2024-03-06T07:23:41.743909+0000","last_active":"2024-03-06T07:23:42.727298+0000","last_peered":"2024-03-06T07:23:42.727298+0000","last_clean":"2024-03-06T07:23:42.727298+0000","last_became_active":"2024-03-06T07:23:41.742418+0000","last_became_peered":"2024-03-06T07:23:41.742418+0000","last_unstale":"2024-03-06T07:23:42.727298+0000","last_undegraded":"2024-03-06T07:23:42.727298+0000","last_fullsized":"2024-03-06T07:23:42.727298+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:46.360845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.726899+0000","last_change":"2024-03-06T07:23:41.744446+0000","last_active":"2024-03-06T07:23:42.726899+0000","last_peered":"2024-03-06T07:23:42.726899+0000","last_clean":"2024-03-06T07:23:42.726899+0000","last_became_active":"2024-03-06T07:23:41.744022+0000","last_became_peered":"2024-03-06T07:23:41.744022+0000","last_unstale":"2024-03-06T07:23:42.726899+0000","last_undegraded":"2024-03-06T07:23:42.726899+0000","last_fullsized":"2024-03-06T07:23:42.726899+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:36.196650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.700110+0000","last_change":"2024-03-06T07:23:41.741789+0000","last_active":"2024-03-06T07:23:43.700110+0000","last_peered":"2024-03-06T07:23:43.700110+0000","last_clean":"2024-03-06T07:23:43.700110+0000","last_became_active":"2024-03-06T07:23:41.740917+0000","last_became_peered":"2024-03-06T07:23:41.740917+0000","last_unstale":"2024-03-06T07:23:43.700110+0000","last_undegraded":"2024-03-06T07:23:43.700110+0000","last_fullsized":"2024-03-06T07:23:43.700110+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:46.762106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699326+0000","last_change":"2024-03-06T07:23:41.736167+0000","last_active":"2024-03-06T07:23:43.699326+0000","last_peered":"2024-03-06T07:23:43.699326+0000","last_clean":"2024-03-06T07:23:43.699326+0000","last_became_active":"2024-03-06T07:23:41.732933+0000","last_became_peered":"2024-03-06T07:23:41.732933+0000","last_unstale":"2024-03-06T07:23:43.699326+0000","last_undegraded":"2024-03-06T07:23:43.699326+0000","last_fullsized":"2024-03-06T07:23:43.699326+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:01.833192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723379+0000","last_change":"2024-03-06T07:23:41.732142+0000","last_active":"2024-03-06T07:23:42.723379+0000","last_peered":"2024-03-06T07:23:42.723379+0000","last_clean":"2024-03-06T07:23:42.723379+0000","last_became_active":"2024-03-06T07:23:41.731872+0000","last_became_peered":"2024-03-06T07:23:41.731872+0000","last_unstale":"2024-03-06T07:23:42.723379+0000","last_undegraded":"2024-03-06T07:23:42.723379+0000","last_fullsized":"2024-03-06T07:23:42.723379+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:29.810740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.700186+0000","last_change":"2024-03-06T07:23:41.736926+0000","last_active":"2024-03-06T07:23:43.700186+0000","last_peered":"2024-03-06T07:23:43.700186+0000","last_clean":"2024-03-06T07:23:43.700186+0000","last_became_active":"2024-03-06T07:23:41.736594+0000","last_became_peered":"2024-03-06T07:23:41.736594+0000","last_unstale":"2024-03-06T07:23:43.700186+0000","last_undegraded":"2024-03-06T07:23:43.700186+0000","last_fullsized":"2024-03-06T07:23:43.700186+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:11.642844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.726923+0000","last_change":"2024-03-06T07:23:41.745876+0000","last_active":"2024-03-06T07:23:42.726923+0000","last_peered":"2024-03-06T07:23:42.726923+0000","last_clean":"2024-03-06T07:23:42.726923+0000","last_became_active":"2024-03-06T07:23:41.745610+0000","last_became_peered":"2024-03-06T07:23:41.745610+0000","last_unstale":"2024-03-06T07:23:42.726923+0000","last_undegraded":"2024-03-06T07:23:42.726923+0000","last_fullsized":"2024-03-06T07:23:42.726923+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:12.763384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699978+0000","last_change":"2024-03-06T07:23:41.738752+0000","last_active":"2024-03-06T07:23:43.699978+0000","last_peered":"2024-03-06T07:23:43.699978+0000","last_clean":"2024-03-06T07:23:43.699978+0000","last_became_active":"2024-03-06T07:23:41.738482+0000","last_became_peered":"2024-03-06T07:23:41.738482+0000","last_unstale":"2024-03-06T07:23:43.699978+0000","last_undegraded":"2024-03-06T07:23:43.699978+0000","last_fullsized":"2024-03-06T07:23:43.699978+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:11.642844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699771+0000","last_change":"2024-03-06T07:23:41.738109+0000","last_active":"2024-03-06T07:23:43.699771+0000","last_peered":"2024-03-06T07:23:43.699771+0000","last_clean":"2024-03-06T07:23:43.699771+0000","last_became_active":"2024-03-06T07:23:41.736780+0000","last_became_peered":"2024-03-06T07:23:41.736780+0000","last_unstale":"2024-03-06T07:23:43.699771+0000","last_undegraded":"2024-03-06T07:23:43.699771+0000","last_fullsized":"2024-03-06T07:23:43.699771+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:33.530656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.727152+0000","last_change":"2024-03-06T07:23:41.731587+0000","last_active":"2024-03-06T07:23:42.727152+0000","last_peered":"2024-03-06T07:23:42.727152+0000","last_clean":"2024-03-06T07:23:42.727152+0000","last_became_active":"2024-03-06T07:23:41.731296+0000","last_became_peered":"2024-03-06T07:23:41.731296+0000","last_unstale":"2024-03-06T07:23:42.727152+0000","last_undegraded":"2024-03-06T07:23:42.727152+0000","last_fullsized":"2024-03-06T07:23:42.727152+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:50.938383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.430241+0000","last_change":"2024-03-06T07:23:41.725406+0000","last_active":"2024-03-06T07:23:43.430241+0000","last_peered":"2024-03-06T07:23:43.430241+0000","last_clean":"2024-03-06T07:23:43.430241+0000","last_became_active":"2024-03-06T07:23:41.725090+0000","last_became_peered":"2024-03-06T07:23:41.725090+0000","last_unstale":"2024-03-06T07:23:43.430241+0000","last_undegraded":"2024-03-06T07:23:43.430241+0000","last_fullsized":"2024-03-06T07:23:43.430241+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:22.477780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.430239+0000","last_change":"2024-03-06T07:23:41.726558+0000","last_active":"2024-03-06T07:23:43.430239+0000","last_peered":"2024-03-06T07:23:43.430239+0000","last_clean":"2024-03-06T07:23:43.430239+0000","last_became_active":"2024-03-06T07:23:41.726300+0000","last_became_peered":"2024-03-06T07:23:41.726300+0000","last_unstale":"2024-03-06T07:23:43.430239+0000","last_undegraded":"2024-03-06T07:23:43.430239+0000","last_fullsized":"2024-03-06T07:23:43.430239+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:14.823111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.430237+0000","last_change":"2024-03-06T07:23:41.727227+0000","last_active":"2024-03-06T07:23:43.430237+0000","last_peered":"2024-03-06T07:23:43.430237+0000","last_clean":"2024-03-06T07:23:43.430237+0000","last_became_active":"2024-03-06T07:23:41.726958+0000","last_became_peered":"2024-03-06T07:23:41.726958+0000","last_unstale":"2024-03-06T07:23:43.430237+0000","last_undegraded":"2024-03-06T07:23:43.430237+0000","last_fullsized":"2024-03-06T07:23:43.430237+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:56.859326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.086514+0000","last_change":"2024-03-06T07:23:41.726195+0000","last_active":"2024-03-06T07:23:43.086514+0000","last_peered":"2024-03-06T07:23:43.086514+0000","last_clean":"2024-03-06T07:23:43.086514+0000","last_became_active":"2024-03-06T07:23:41.725574+0000","last_became_peered":"2024-03-06T07:23:41.725574+0000","last_unstale":"2024-03-06T07:23:43.086514+0000","last_undegraded":"2024-03-06T07:23:43.086514+0000","last_fullsized":"2024-03-06T07:23:43.086514+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:33.323629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.700392+0000","last_change":"2024-03-06T07:23:41.739560+0000","last_active":"2024-03-06T07:23:43.700392+0000","last_peered":"2024-03-06T07:23:43.700392+0000","last_clean":"2024-03-06T07:23:43.700392+0000","last_became_active":"2024-03-06T07:23:41.739096+0000","last_became_peered":"2024-03-06T07:23:41.739096+0000","last_unstale":"2024-03-06T07:23:43.700392+0000","last_undegraded":"2024-03-06T07:23:43.700392+0000","last_fullsized":"2024-03-06T07:23:43.700392+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:26.754812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.086420+0000","last_change":"2024-03-06T07:23:41.740528+0000","last_active":"2024-03-06T07:23:43.086420+0000","last_peered":"2024-03-06T07:23:43.086420+0000","last_clean":"2024-03-06T07:23:43.086420+0000","last_became_active":"2024-03-06T07:23:41.739380+0000","last_became_peered":"2024-03-06T07:23:41.739380+0000","last_unstale":"2024-03-06T07:23:43.086420+0000","last_undegraded":"2024-03-06T07:23:43.086420+0000","last_fullsized":"2024-03-06T07:23:43.086420+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:26.159788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699905+0000","last_change":"2024-03-06T07:23:41.742413+0000","last_active":"2024-03-06T07:23:43.699905+0000","last_peered":"2024-03-06T07:23:43.699905+0000","last_clean":"2024-03-06T07:23:43.699905+0000","last_became_active":"2024-03-06T07:23:41.736408+0000","last_became_peered":"2024-03-06T07:23:41.736408+0000","last_unstale":"2024-03-06T07:23:43.699905+0000","last_undegraded":"2024-03-06T07:23:43.699905+0000","last_fullsized":"2024-03-06T07:23:43.699905+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:05.768595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723602+0000","last_change":"2024-03-06T07:23:41.729152+0000","last_active":"2024-03-06T07:23:42.723602+0000","last_peered":"2024-03-06T07:23:42.723602+0000","last_clean":"2024-03-06T07:23:42.723602+0000","last_became_active":"2024-03-06T07:23:41.728389+0000","last_became_peered":"2024-03-06T07:23:41.728389+0000","last_unstale":"2024-03-06T07:23:42.723602+0000","last_undegraded":"2024-03-06T07:23:42.723602+0000","last_fullsized":"2024-03-06T07:23:42.723602+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:44.239674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.726907+0000","last_change":"2024-03-06T07:23:41.737095+0000","last_active":"2024-03-06T07:23:42.726907+0000","last_peered":"2024-03-06T07:23:42.726907+0000","last_clean":"2024-03-06T07:23:42.726907+0000","last_became_active":"2024-03-06T07:23:41.736805+0000","last_became_peered":"2024-03-06T07:23:41.736805+0000","last_unstale":"2024-03-06T07:23:42.726907+0000","last_undegraded":"2024-03-06T07:23:42.726907+0000","last_fullsized":"2024-03-06T07:23:42.726907+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:34.635451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.086121+0000","last_change":"2024-03-06T07:23:41.730405+0000","last_active":"2024-03-06T07:23:43.086121+0000","last_peered":"2024-03-06T07:23:43.086121+0000","last_clean":"2024-03-06T07:23:43.086121+0000","last_became_active":"2024-03-06T07:23:41.730135+0000","last_became_peered":"2024-03-06T07:23:41.730135+0000","last_unstale":"2024-03-06T07:23:43.086121+0000","last_undegraded":"2024-03-06T07:23:43.086121+0000","last_fullsized":"2024-03-06T07:23:43.086121+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:19.433191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.726897+0000","last_change":"2024-03-06T07:23:41.740813+0000","last_active":"2024-03-06T07:23:42.726897+0000","last_peered":"2024-03-06T07:23:42.726897+0000","last_clean":"2024-03-06T07:23:42.726897+0000","last_became_active":"2024-03-06T07:23:41.739650+0000","last_became_peered":"2024-03-06T07:23:41.739650+0000","last_unstale":"2024-03-06T07:23:42.726897+0000","last_undegraded":"2024-03-06T07:23:42.726897+0000","last_fullsized":"2024-03-06T07:23:42.726897+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:49.002905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699866+0000","last_change":"2024-03-06T07:23:41.740551+0000","last_active":"2024-03-06T07:23:43.699866+0000","last_peered":"2024-03-06T07:23:43.699866+0000","last_clean":"2024-03-06T07:23:43.699866+0000","last_became_active":"2024-03-06T07:23:41.740271+0000","last_became_peered":"2024-03-06T07:23:41.740271+0000","last_unstale":"2024-03-06T07:23:43.699866+0000","last_undegraded":"2024-03-06T07:23:43.699866+0000","last_fullsized":"2024-03-06T07:23:43.699866+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:24.352083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.700173+0000","last_change":"2024-03-06T07:23:41.744601+0000","last_active":"2024-03-06T07:23:43.700173+0000","last_peered":"2024-03-06T07:23:43.700173+0000","last_clean":"2024-03-06T07:23:43.700173+0000","last_became_active":"2024-03-06T07:23:41.744105+0000","last_became_peered":"2024-03-06T07:23:41.744105+0000","last_unstale":"2024-03-06T07:23:43.700173+0000","last_undegraded":"2024-03-06T07:23:43.700173+0000","last_fullsized":"2024-03-06T07:23:43.700173+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:22.477780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.700116+0000","last_change":"2024-03-06T07:23:41.728748+0000","last_active":"2024-03-06T07:23:43.700116+0000","last_peered":"2024-03-06T07:23:43.700116+0000","last_clean":"2024-03-06T07:23:43.700116+0000","last_became_active":"2024-03-06T07:23:41.727886+0000","last_became_peered":"2024-03-06T07:23:41.727886+0000","last_unstale":"2024-03-06T07:23:43.700116+0000","last_undegraded":"2024-03-06T07:23:43.700116+0000","last_fullsized":"2024-03-06T07:23:43.700116+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:49.002905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699542+0000","last_change":"2024-03-06T07:23:41.737471+0000","last_active":"2024-03-06T07:23:43.699542+0000","last_peered":"2024-03-06T07:23:43.699542+0000","last_clean":"2024-03-06T07:23:43.699542+0000","last_became_active":"2024-03-06T07:23:41.737153+0000","last_became_peered":"2024-03-06T07:23:41.737153+0000","last_unstale":"2024-03-06T07:23:43.699542+0000","last_undegraded":"2024-03-06T07:23:43.699542+0000","last_fullsized":"2024-03-06T07:23:43.699542+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:12.417849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723648+0000","last_change":"2024-03-06T07:23:41.721828+0000","last_active":"2024-03-06T07:23:42.723648+0000","last_peered":"2024-03-06T07:23:42.723648+0000","last_clean":"2024-03-06T07:23:42.723648+0000","last_became_active":"2024-03-06T07:23:41.721505+0000","last_became_peered":"2024-03-06T07:23:41.721505+0000","last_unstale":"2024-03-06T07:23:42.723648+0000","last_undegraded":"2024-03-06T07:23:42.723648+0000","last_fullsized":"2024-03-06T07:23:42.723648+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:07.398014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699462+0000","last_change":"2024-03-06T07:23:41.744747+0000","last_active":"2024-03-06T07:23:43.699462+0000","last_peered":"2024-03-06T07:23:43.699462+0000","last_clean":"2024-03-06T07:23:43.699462+0000","last_became_active":"2024-03-06T07:23:41.737448+0000","last_became_peered":"2024-03-06T07:23:41.737448+0000","last_unstale":"2024-03-06T07:23:43.699462+0000","last_undegraded":"2024-03-06T07:23:43.699462+0000","last_fullsized":"2024-03-06T07:23:43.699462+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:34.635451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699477+0000","last_change":"2024-03-06T07:23:41.741564+0000","last_active":"2024-03-06T07:23:43.699477+0000","last_peered":"2024-03-06T07:23:43.699477+0000","last_clean":"2024-03-06T07:23:43.699477+0000","last_became_active":"2024-03-06T07:23:41.741283+0000","last_became_peered":"2024-03-06T07:23:41.741283+0000","last_unstale":"2024-03-06T07:23:43.699477+0000","last_undegraded":"2024-03-06T07:23:43.699477+0000","last_fullsized":"2024-03-06T07:23:43.699477+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:31.272742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723439+0000","last_change":"2024-03-06T07:23:41.737723+0000","last_active":"2024-03-06T07:23:42.723439+0000","last_peered":"2024-03-06T07:23:42.723439+0000","last_clean":"2024-03-06T07:23:42.723439+0000","last_became_active":"2024-03-06T07:23:41.737426+0000","last_became_peered":"2024-03-06T07:23:41.737426+0000","last_unstale":"2024-03-06T07:23:42.723439+0000","last_undegraded":"2024-03-06T07:23:42.723439+0000","last_fullsized":"2024-03-06T07:23:42.723439+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:10.379075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.086280+0000","last_change":"2024-03-06T07:23:41.738093+0000","last_active":"2024-03-06T07:23:43.086280+0000","last_peered":"2024-03-06T07:23:43.086280+0000","last_clean":"2024-03-06T07:23:43.086280+0000","last_became_active":"2024-03-06T07:23:41.736108+0000","last_became_peered":"2024-03-06T07:23:41.736108+0000","last_unstale":"2024-03-06T07:23:43.086280+0000","last_undegraded":"2024-03-06T07:23:43.086280+0000","last_fullsized":"2024-03-06T07:23:43.086280+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:39.987304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"94'4","reported_seq":18,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.727362+0000","last_change":"2024-03-06T07:23:40.717457+0000","last_active":"2024-03-06T07:23:42.727362+0000","last_peered":"2024-03-06T07:23:42.727362+0000","last_clean":"2024-03-06T07:23:42.727362+0000","last_became_active":"2024-03-06T07:23:40.717182+0000","last_became_peered":"2024-03-06T07:23:40.717182+0000","last_unstale":"2024-03-06T07:23:42.727362+0000","last_undegraded":"2024-03-06T07:23:42.727362+0000","last_fullsized":"2024-03-06T07:23:42.727362+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:55.854681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723432+0000","last_change":"2024-03-06T07:23:41.733799+0000","last_active":"2024-03-06T07:23:42.723432+0000","last_peered":"2024-03-06T07:23:42.723432+0000","last_clean":"2024-03-06T07:23:42.723432+0000","last_became_active":"2024-03-06T07:23:41.733522+0000","last_became_peered":"2024-03-06T07:23:41.733522+0000","last_unstale":"2024-03-06T07:23:42.723432+0000","last_undegraded":"2024-03-06T07:23:42.723432+0000","last_fullsized":"2024-03-06T07:23:42.723432+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:37.836040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723781+0000","last_change":"2024-03-06T07:23:40.725609+0000","last_active":"2024-03-06T07:23:42.723781+0000","last_peered":"2024-03-06T07:23:42.723781+0000","last_clean":"2024-03-06T07:23:42.723781+0000","last_became_active":"2024-03-06T07:23:40.725336+0000","last_became_peered":"2024-03-06T07:23:40.725336+0000","last_unstale":"2024-03-06T07:23:42.723781+0000","last_undegraded":"2024-03-06T07:23:42.723781+0000","last_fullsized":"2024-03-06T07:23:42.723781+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:31.358174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699600+0000","last_change":"2024-03-06T07:23:41.737482+0000","last_active":"2024-03-06T07:23:43.699600+0000","last_peered":"2024-03-06T07:23:43.699600+0000","last_clean":"2024-03-06T07:23:43.699600+0000","last_became_active":"2024-03-06T07:23:41.735512+0000","last_became_peered":"2024-03-06T07:23:41.735512+0000","last_unstale":"2024-03-06T07:23:43.699600+0000","last_undegraded":"2024-03-06T07:23:43.699600+0000","last_fullsized":"2024-03-06T07:23:43.699600+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:12.763384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699953+0000","last_change":"2024-03-06T07:23:40.720128+0000","last_active":"2024-03-06T07:23:43.699953+0000","last_peered":"2024-03-06T07:23:43.699953+0000","last_clean":"2024-03-06T07:23:43.699953+0000","last_became_active":"2024-03-06T07:23:40.719823+0000","last_became_peered":"2024-03-06T07:23:40.719823+0000","last_unstale":"2024-03-06T07:23:43.699953+0000","last_undegraded":"2024-03-06T07:23:43.699953+0000","last_fullsized":"2024-03-06T07:23:43.699953+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:07.611250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.700441+0000","last_change":"2024-03-06T07:23:41.739322+0000","last_active":"2024-03-06T07:23:43.700441+0000","last_peered":"2024-03-06T07:23:43.700441+0000","last_clean":"2024-03-06T07:23:43.700441+0000","last_became_active":"2024-03-06T07:23:41.738494+0000","last_became_peered":"2024-03-06T07:23:41.738494+0000","last_unstale":"2024-03-06T07:23:43.700441+0000","last_undegraded":"2024-03-06T07:23:43.700441+0000","last_fullsized":"2024-03-06T07:23:43.700441+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:26.754812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.726754+0000","last_change":"2024-03-06T07:23:40.713614+0000","last_active":"2024-03-06T07:23:42.726754+0000","last_peered":"2024-03-06T07:23:42.726754+0000","last_clean":"2024-03-06T07:23:42.726754+0000","last_became_active":"2024-03-06T07:23:40.713188+0000","last_became_peered":"2024-03-06T07:23:40.713188+0000","last_unstale":"2024-03-06T07:23:42.726754+0000","last_undegraded":"2024-03-06T07:23:42.726754+0000","last_fullsized":"2024-03-06T07:23:42.726754+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:38.192339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.727380+0000","last_change":"2024-03-06T07:23:41.733036+0000","last_active":"2024-03-06T07:23:42.727380+0000","last_peered":"2024-03-06T07:23:42.727380+0000","last_clean":"2024-03-06T07:23:42.727380+0000","last_became_active":"2024-03-06T07:23:41.732766+0000","last_became_peered":"2024-03-06T07:23:41.732766+0000","last_unstale":"2024-03-06T07:23:42.727380+0000","last_undegraded":"2024-03-06T07:23:42.727380+0000","last_fullsized":"2024-03-06T07:23:42.727380+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:18.855336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.430239+0000","last_change":"2024-03-06T07:23:40.717736+0000","last_active":"2024-03-06T07:23:43.430239+0000","last_peered":"2024-03-06T07:23:43.430239+0000","last_clean":"2024-03-06T07:23:43.430239+0000","last_became_active":"2024-03-06T07:23:40.717461+0000","last_became_peered":"2024-03-06T07:23:40.717461+0000","last_unstale":"2024-03-06T07:23:43.430239+0000","last_undegraded":"2024-03-06T07:23:43.430239+0000","last_fullsized":"2024-03-06T07:23:43.430239+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:10.638199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.086554+0000","last_change":"2024-03-06T07:23:41.740310+0000","last_active":"2024-03-06T07:23:43.086554+0000","last_peered":"2024-03-06T07:23:43.086554+0000","last_clean":"2024-03-06T07:23:43.086554+0000","last_became_active":"2024-03-06T07:23:41.739029+0000","last_became_peered":"2024-03-06T07:23:41.739029+0000","last_unstale":"2024-03-06T07:23:43.086554+0000","last_undegraded":"2024-03-06T07:23:43.086554+0000","last_fullsized":"2024-03-06T07:23:43.086554+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:26.104781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699409+0000","last_change":"2024-03-06T07:23:40.718410+0000","last_active":"2024-03-06T07:23:43.699409+0000","last_peered":"2024-03-06T07:23:43.699409+0000","last_clean":"2024-03-06T07:23:43.699409+0000","last_became_active":"2024-03-06T07:23:40.718118+0000","last_became_peered":"2024-03-06T07:23:40.718118+0000","last_unstale":"2024-03-06T07:23:43.699409+0000","last_undegraded":"2024-03-06T07:23:43.699409+0000","last_fullsized":"2024-03-06T07:23:43.699409+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:38.192339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.700389+0000","last_change":"2024-03-06T07:23:41.739346+0000","last_active":"2024-03-06T07:23:43.700389+0000","last_peered":"2024-03-06T07:23:43.700389+0000","last_clean":"2024-03-06T07:23:43.700389+0000","last_became_active":"2024-03-06T07:23:41.738706+0000","last_became_peered":"2024-03-06T07:23:41.738706+0000","last_unstale":"2024-03-06T07:23:43.700389+0000","last_undegraded":"2024-03-06T07:23:43.700389+0000","last_fullsized":"2024-03-06T07:23:43.700389+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:36.196650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699129+0000","last_change":"2024-03-06T07:23:40.720290+0000","last_active":"2024-03-06T07:23:43.699129+0000","last_peered":"2024-03-06T07:23:43.699129+0000","last_clean":"2024-03-06T07:23:43.699129+0000","last_became_active":"2024-03-06T07:23:40.719928+0000","last_became_peered":"2024-03-06T07:23:40.719928+0000","last_unstale":"2024-03-06T07:23:43.699129+0000","last_undegraded":"2024-03-06T07:23:43.699129+0000","last_fullsized":"2024-03-06T07:23:43.699129+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:00.828547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.727461+0000","last_change":"2024-03-06T07:23:41.739526+0000","last_active":"2024-03-06T07:23:42.727461+0000","last_peered":"2024-03-06T07:23:42.727461+0000","last_clean":"2024-03-06T07:23:42.727461+0000","last_became_active":"2024-03-06T07:23:41.739235+0000","last_became_peered":"2024-03-06T07:23:41.739235+0000","last_unstale":"2024-03-06T07:23:42.727461+0000","last_undegraded":"2024-03-06T07:23:42.727461+0000","last_fullsized":"2024-03-06T07:23:42.727461+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:19.729694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.726824+0000","last_change":"2024-03-06T07:23:40.714859+0000","last_active":"2024-03-06T07:23:42.726824+0000","last_peered":"2024-03-06T07:23:42.726824+0000","last_clean":"2024-03-06T07:23:42.726824+0000","last_became_active":"2024-03-06T07:23:40.714576+0000","last_became_peered":"2024-03-06T07:23:40.714576+0000","last_unstale":"2024-03-06T07:23:42.726824+0000","last_undegraded":"2024-03-06T07:23:42.726824+0000","last_fullsized":"2024-03-06T07:23:42.726824+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:21.473135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.086198+0000","last_change":"2024-03-06T07:23:41.735697+0000","last_active":"2024-03-06T07:23:43.086198+0000","last_peered":"2024-03-06T07:23:43.086198+0000","last_clean":"2024-03-06T07:23:43.086198+0000","last_became_active":"2024-03-06T07:23:41.735398+0000","last_became_peered":"2024-03-06T07:23:41.735398+0000","last_unstale":"2024-03-06T07:23:43.086198+0000","last_undegraded":"2024-03-06T07:23:43.086198+0000","last_fullsized":"2024-03-06T07:23:43.086198+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:01.113258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699914+0000","last_change":"2024-03-06T07:23:40.716925+0000","last_active":"2024-03-06T07:23:43.699914+0000","last_peered":"2024-03-06T07:23:43.699914+0000","last_clean":"2024-03-06T07:23:43.699914+0000","last_became_active":"2024-03-06T07:23:40.716644+0000","last_became_peered":"2024-03-06T07:23:40.716644+0000","last_unstale":"2024-03-06T07:23:43.699914+0000","last_undegraded":"2024-03-06T07:23:43.699914+0000","last_fullsized":"2024-03-06T07:23:43.699914+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:30.268097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.726899+0000","last_change":"2024-03-06T07:23:41.737919+0000","last_active":"2024-03-06T07:23:42.726899+0000","last_peered":"2024-03-06T07:23:42.726899+0000","last_clean":"2024-03-06T07:23:42.726899+0000","last_became_active":"2024-03-06T07:23:41.736976+0000","last_became_peered":"2024-03-06T07:23:41.736976+0000","last_unstale":"2024-03-06T07:23:42.726899+0000","last_undegraded":"2024-03-06T07:23:42.726899+0000","last_fullsized":"2024-03-06T07:23:42.726899+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:26.754812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.086276+0000","last_change":"2024-03-06T07:23:40.712494+0000","last_active":"2024-03-06T07:23:43.086276+0000","last_peered":"2024-03-06T07:23:43.086276+0000","last_clean":"2024-03-06T07:23:43.086276+0000","last_became_active":"2024-03-06T07:23:40.712173+0000","last_became_peered":"2024-03-06T07:23:40.712173+0000","last_unstale":"2024-03-06T07:23:43.086276+0000","last_undegraded":"2024-03-06T07:23:43.086276+0000","last_fullsized":"2024-03-06T07:23:43.086276+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:38.984941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723459+0000","last_change":"2024-03-06T07:23:41.734591+0000","last_active":"2024-03-06T07:23:42.723459+0000","last_peered":"2024-03-06T07:23:42.723459+0000","last_clean":"2024-03-06T07:23:42.723459+0000","last_became_active":"2024-03-06T07:23:41.734231+0000","last_became_peered":"2024-03-06T07:23:41.734231+0000","last_unstale":"2024-03-06T07:23:42.723459+0000","last_undegraded":"2024-03-06T07:23:42.723459+0000","last_fullsized":"2024-03-06T07:23:42.723459+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:33.916259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"94'3","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.430316+0000","last_change":"2024-03-06T07:23:40.719744+0000","last_active":"2024-03-06T07:23:43.430316+0000","last_peered":"2024-03-06T07:23:43.430316+0000","last_clean":"2024-03-06T07:23:43.430316+0000","last_became_active":"2024-03-06T07:23:40.719414+0000","last_became_peered":"2024-03-06T07:23:40.719414+0000","last_unstale":"2024-03-06T07:23:43.430316+0000","last_undegraded":"2024-03-06T07:23:43.430316+0000","last_fullsized":"2024-03-06T07:23:43.430316+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:47.998260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.727077+0000","last_change":"2024-03-06T07:23:41.740468+0000","last_active":"2024-03-06T07:23:42.727077+0000","last_peered":"2024-03-06T07:23:42.727077+0000","last_clean":"2024-03-06T07:23:42.727077+0000","last_became_active":"2024-03-06T07:23:41.738893+0000","last_became_peered":"2024-03-06T07:23:41.738893+0000","last_unstale":"2024-03-06T07:23:42.727077+0000","last_undegraded":"2024-03-06T07:23:42.727077+0000","last_fullsized":"2024-03-06T07:23:42.727077+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:24.352083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"94'6","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699590+0000","last_change":"2024-03-06T07:23:40.719400+0000","last_active":"2024-03-06T07:23:43.699590+0000","last_peered":"2024-03-06T07:23:43.699590+0000","last_clean":"2024-03-06T07:23:43.699590+0000","last_became_active":"2024-03-06T07:23:40.719136+0000","last_became_peered":"2024-03-06T07:23:40.719136+0000","last_unstale":"2024-03-06T07:23:43.699590+0000","last_undegraded":"2024-03-06T07:23:43.699590+0000","last_fullsized":"2024-03-06T07:23:43.699590+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:30.165325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699674+0000","last_change":"2024-03-06T07:23:41.726148+0000","last_active":"2024-03-06T07:23:43.699674+0000","last_peered":"2024-03-06T07:23:43.699674+0000","last_clean":"2024-03-06T07:23:43.699674+0000","last_became_active":"2024-03-06T07:23:41.725884+0000","last_became_peered":"2024-03-06T07:23:41.725884+0000","last_unstale":"2024-03-06T07:23:43.699674+0000","last_undegraded":"2024-03-06T07:23:43.699674+0000","last_fullsized":"2024-03-06T07:23:43.699674+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:46.762106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.726988+0000","last_change":"2024-03-06T07:23:40.722697+0000","last_active":"2024-03-06T07:23:42.726988+0000","last_peered":"2024-03-06T07:23:42.726988+0000","last_clean":"2024-03-06T07:23:42.726988+0000","last_became_active":"2024-03-06T07:23:40.721923+0000","last_became_peered":"2024-03-06T07:23:40.721923+0000","last_unstale":"2024-03-06T07:23:42.726988+0000","last_undegraded":"2024-03-06T07:23:42.726988+0000","last_fullsized":"2024-03-06T07:23:42.726988+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:07.872472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699802+0000","last_change":"2024-03-06T07:23:41.736507+0000","last_active":"2024-03-06T07:23:43.699802+0000","last_peered":"2024-03-06T07:23:43.699802+0000","last_clean":"2024-03-06T07:23:43.699802+0000","last_became_active":"2024-03-06T07:23:41.736232+0000","last_became_peered":"2024-03-06T07:23:41.736232+0000","last_unstale":"2024-03-06T07:23:43.699802+0000","last_undegraded":"2024-03-06T07:23:43.699802+0000","last_fullsized":"2024-03-06T07:23:43.699802+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:28.532264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.727105+0000","last_change":"2024-03-06T07:23:40.714424+0000","last_active":"2024-03-06T07:23:42.727105+0000","last_peered":"2024-03-06T07:23:42.727105+0000","last_clean":"2024-03-06T07:23:42.727105+0000","last_became_active":"2024-03-06T07:23:40.714067+0000","last_became_peered":"2024-03-06T07:23:40.714067+0000","last_unstale":"2024-03-06T07:23:42.727105+0000","last_undegraded":"2024-03-06T07:23:42.727105+0000","last_fullsized":"2024-03-06T07:23:42.727105+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:13.818466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723824+0000","last_change":"2024-03-06T07:23:41.738904+0000","last_active":"2024-03-06T07:23:42.723824+0000","last_peered":"2024-03-06T07:23:42.723824+0000","last_clean":"2024-03-06T07:23:42.723824+0000","last_became_active":"2024-03-06T07:23:41.737386+0000","last_became_peered":"2024-03-06T07:23:41.737386+0000","last_unstale":"2024-03-06T07:23:42.723824+0000","last_undegraded":"2024-03-06T07:23:42.723824+0000","last_fullsized":"2024-03-06T07:23:42.723824+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:18.480322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723376+0000","last_change":"2024-03-06T07:23:40.722930+0000","last_active":"2024-03-06T07:23:42.723376+0000","last_peered":"2024-03-06T07:23:42.723376+0000","last_clean":"2024-03-06T07:23:42.723376+0000","last_became_active":"2024-03-06T07:23:40.722608+0000","last_became_peered":"2024-03-06T07:23:40.722608+0000","last_unstale":"2024-03-06T07:23:42.723376+0000","last_undegraded":"2024-03-06T07:23:42.723376+0000","last_fullsized":"2024-03-06T07:23:42.723376+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:43.778383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.086527+0000","last_change":"2024-03-06T07:23:41.728230+0000","last_active":"2024-03-06T07:23:43.086527+0000","last_peered":"2024-03-06T07:23:43.086527+0000","last_clean":"2024-03-06T07:23:43.086527+0000","last_became_active":"2024-03-06T07:23:41.726779+0000","last_became_peered":"2024-03-06T07:23:41.726779+0000","last_unstale":"2024-03-06T07:23:43.086527+0000","last_undegraded":"2024-03-06T07:23:43.086527+0000","last_fullsized":"2024-03-06T07:23:43.086527+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:49.143163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"94'8","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699093+0000","last_change":"2024-03-06T07:23:40.721913+0000","last_active":"2024-03-06T07:23:43.699093+0000","last_peered":"2024-03-06T07:23:43.699093+0000","last_clean":"2024-03-06T07:23:43.699093+0000","last_became_active":"2024-03-06T07:23:40.721578+0000","last_became_peered":"2024-03-06T07:23:40.721578+0000","last_unstale":"2024-03-06T07:23:43.699093+0000","last_undegraded":"2024-03-06T07:23:43.699093+0000","last_fullsized":"2024-03-06T07:23:43.699093+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:07.872472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723529+0000","last_change":"2024-03-06T07:23:41.724342+0000","last_active":"2024-03-06T07:23:42.723529+0000","last_peered":"2024-03-06T07:23:42.723529+0000","last_clean":"2024-03-06T07:23:42.723529+0000","last_became_active":"2024-03-06T07:23:41.722929+0000","last_became_peered":"2024-03-06T07:23:41.722929+0000","last_unstale":"2024-03-06T07:23:42.723529+0000","last_undegraded":"2024-03-06T07:23:42.723529+0000","last_fullsized":"2024-03-06T07:23:42.723529+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:36.923567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.700237+0000","last_change":"2024-03-06T07:23:40.719065+0000","last_active":"2024-03-06T07:23:43.700237+0000","last_peered":"2024-03-06T07:23:43.700237+0000","last_clean":"2024-03-06T07:23:43.700237+0000","last_became_active":"2024-03-06T07:23:40.718655+0000","last_became_peered":"2024-03-06T07:23:40.718655+0000","last_unstale":"2024-03-06T07:23:43.700237+0000","last_undegraded":"2024-03-06T07:23:43.700237+0000","last_fullsized":"2024-03-06T07:23:43.700237+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:30.785028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.700245+0000","last_change":"2024-03-06T07:23:41.732921+0000","last_active":"2024-03-06T07:23:43.700245+0000","last_peered":"2024-03-06T07:23:43.700245+0000","last_clean":"2024-03-06T07:23:43.700245+0000","last_became_active":"2024-03-06T07:23:41.732626+0000","last_became_peered":"2024-03-06T07:23:41.732626+0000","last_unstale":"2024-03-06T07:23:43.700245+0000","last_undegraded":"2024-03-06T07:23:43.700245+0000","last_fullsized":"2024-03-06T07:23:43.700245+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:49.002905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.430372+0000","last_change":"2024-03-06T07:23:40.715435+0000","last_active":"2024-03-06T07:23:43.430372+0000","last_peered":"2024-03-06T07:23:43.430372+0000","last_clean":"2024-03-06T07:23:43.430372+0000","last_became_active":"2024-03-06T07:23:40.715173+0000","last_became_peered":"2024-03-06T07:23:40.715173+0000","last_unstale":"2024-03-06T07:23:43.430372+0000","last_undegraded":"2024-03-06T07:23:43.430372+0000","last_fullsized":"2024-03-06T07:23:43.430372+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:35.192005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.726941+0000","last_change":"2024-03-06T07:23:41.734820+0000","last_active":"2024-03-06T07:23:42.726941+0000","last_peered":"2024-03-06T07:23:42.726941+0000","last_clean":"2024-03-06T07:23:42.726941+0000","last_became_active":"2024-03-06T07:23:41.734519+0000","last_became_peered":"2024-03-06T07:23:41.734519+0000","last_unstale":"2024-03-06T07:23:42.726941+0000","last_undegraded":"2024-03-06T07:23:42.726941+0000","last_fullsized":"2024-03-06T07:23:42.726941+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:33.530656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699720+0000","last_change":"2024-03-06T07:23:40.722169+0000","last_active":"2024-03-06T07:23:43.699720+0000","last_peered":"2024-03-06T07:23:43.699720+0000","last_clean":"2024-03-06T07:23:43.699720+0000","last_became_active":"2024-03-06T07:23:40.721891+0000","last_became_peered":"2024-03-06T07:23:40.721891+0000","last_unstale":"2024-03-06T07:23:43.699720+0000","last_undegraded":"2024-03-06T07:23:43.699720+0000","last_fullsized":"2024-03-06T07:23:43.699720+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:07.611250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.086290+0000","last_change":"2024-03-06T07:23:41.727461+0000","last_active":"2024-03-06T07:23:43.086290+0000","last_peered":"2024-03-06T07:23:43.086290+0000","last_clean":"2024-03-06T07:23:43.086290+0000","last_became_active":"2024-03-06T07:23:41.727134+0000","last_became_peered":"2024-03-06T07:23:41.727134+0000","last_unstale":"2024-03-06T07:23:43.086290+0000","last_undegraded":"2024-03-06T07:23:43.086290+0000","last_fullsized":"2024-03-06T07:23:43.086290+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:07.469416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723661+0000","last_change":"2024-03-06T07:23:40.724191+0000","last_active":"2024-03-06T07:23:42.723661+0000","last_peered":"2024-03-06T07:23:42.723661+0000","last_clean":"2024-03-06T07:23:42.723661+0000","last_became_active":"2024-03-06T07:23:40.723929+0000","last_became_peered":"2024-03-06T07:23:40.723929+0000","last_unstale":"2024-03-06T07:23:42.723661+0000","last_undegraded":"2024-03-06T07:23:42.723661+0000","last_fullsized":"2024-03-06T07:23:42.723661+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:47.545463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.086484+0000","last_change":"2024-03-06T07:23:41.733878+0000","last_active":"2024-03-06T07:23:43.086484+0000","last_peered":"2024-03-06T07:23:43.086484+0000","last_clean":"2024-03-06T07:23:43.086484+0000","last_became_active":"2024-03-06T07:23:41.733397+0000","last_became_peered":"2024-03-06T07:23:41.733397+0000","last_unstale":"2024-03-06T07:23:43.086484+0000","last_undegraded":"2024-03-06T07:23:43.086484+0000","last_fullsized":"2024-03-06T07:23:43.086484+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:41.423204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723294+0000","last_change":"2024-03-06T07:23:40.722294+0000","last_active":"2024-03-06T07:23:42.723294+0000","last_peered":"2024-03-06T07:23:42.723294+0000","last_clean":"2024-03-06T07:23:42.723294+0000","last_became_active":"2024-03-06T07:23:40.722006+0000","last_became_peered":"2024-03-06T07:23:40.722006+0000","last_unstale":"2024-03-06T07:23:42.723294+0000","last_undegraded":"2024-03-06T07:23:42.723294+0000","last_fullsized":"2024-03-06T07:23:42.723294+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:54.784585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.727291+0000","last_change":"2024-03-06T07:23:41.743132+0000","last_active":"2024-03-06T07:23:42.727291+0000","last_peered":"2024-03-06T07:23:42.727291+0000","last_clean":"2024-03-06T07:23:42.727291+0000","last_became_active":"2024-03-06T07:23:41.742852+0000","last_became_peered":"2024-03-06T07:23:41.742852+0000","last_unstale":"2024-03-06T07:23:42.727291+0000","last_undegraded":"2024-03-06T07:23:42.727291+0000","last_fullsized":"2024-03-06T07:23:42.727291+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:25.380520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699526+0000","last_change":"2024-03-06T07:23:40.717736+0000","last_active":"2024-03-06T07:23:43.699526+0000","last_peered":"2024-03-06T07:23:43.699526+0000","last_clean":"2024-03-06T07:23:43.699526+0000","last_became_active":"2024-03-06T07:23:40.717467+0000","last_became_peered":"2024-03-06T07:23:40.717467+0000","last_unstale":"2024-03-06T07:23:43.699526+0000","last_undegraded":"2024-03-06T07:23:43.699526+0000","last_fullsized":"2024-03-06T07:23:43.699526+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:18.469593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723727+0000","last_change":"2024-03-06T07:23:41.733669+0000","last_active":"2024-03-06T07:23:42.723727+0000","last_peered":"2024-03-06T07:23:42.723727+0000","last_clean":"2024-03-06T07:23:42.723727+0000","last_became_active":"2024-03-06T07:23:41.733160+0000","last_became_peered":"2024-03-06T07:23:41.733160+0000","last_unstale":"2024-03-06T07:23:42.723727+0000","last_undegraded":"2024-03-06T07:23:42.723727+0000","last_fullsized":"2024-03-06T07:23:42.723727+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:23.809543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723478+0000","last_change":"2024-03-06T07:23:40.721483+0000","last_active":"2024-03-06T07:23:42.723478+0000","last_peered":"2024-03-06T07:23:42.723478+0000","last_clean":"2024-03-06T07:23:42.723478+0000","last_became_active":"2024-03-06T07:23:40.721205+0000","last_became_peered":"2024-03-06T07:23:40.721205+0000","last_unstale":"2024-03-06T07:23:42.723478+0000","last_undegraded":"2024-03-06T07:23:42.723478+0000","last_fullsized":"2024-03-06T07:23:42.723478+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:56.132734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.086125+0000","last_change":"2024-03-06T07:23:41.726831+0000","last_active":"2024-03-06T07:23:43.086125+0000","last_peered":"2024-03-06T07:23:43.086125+0000","last_clean":"2024-03-06T07:23:43.086125+0000","last_became_active":"2024-03-06T07:23:41.726553+0000","last_became_peered":"2024-03-06T07:23:41.726553+0000","last_unstale":"2024-03-06T07:23:43.086125+0000","last_undegraded":"2024-03-06T07:23:43.086125+0000","last_fullsized":"2024-03-06T07:23:43.086125+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:00.843817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.726707+0000","last_change":"2024-03-06T07:23:40.721441+0000","last_active":"2024-03-06T07:23:42.726707+0000","last_peered":"2024-03-06T07:23:42.726707+0000","last_clean":"2024-03-06T07:23:42.726707+0000","last_became_active":"2024-03-06T07:23:40.721083+0000","last_became_peered":"2024-03-06T07:23:40.721083+0000","last_unstale":"2024-03-06T07:23:42.726707+0000","last_undegraded":"2024-03-06T07:23:42.726707+0000","last_fullsized":"2024-03-06T07:23:42.726707+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:34.043452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699227+0000","last_change":"2024-03-06T07:23:41.735606+0000","last_active":"2024-03-06T07:23:43.699227+0000","last_peered":"2024-03-06T07:23:43.699227+0000","last_clean":"2024-03-06T07:23:43.699227+0000","last_became_active":"2024-03-06T07:23:41.735328+0000","last_became_peered":"2024-03-06T07:23:41.735328+0000","last_unstale":"2024-03-06T07:23:43.699227+0000","last_undegraded":"2024-03-06T07:23:43.699227+0000","last_fullsized":"2024-03-06T07:23:43.699227+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:26.754812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"95'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.086402+0000","last_change":"2024-03-06T07:23:40.715765+0000","last_active":"2024-03-06T07:23:43.086402+0000","last_peered":"2024-03-06T07:23:43.086402+0000","last_clean":"2024-03-06T07:23:43.086402+0000","last_became_active":"2024-03-06T07:23:40.715499+0000","last_became_peered":"2024-03-06T07:23:40.715499+0000","last_unstale":"2024-03-06T07:23:43.086402+0000","last_undegraded":"2024-03-06T07:23:43.086402+0000","last_fullsized":"2024-03-06T07:23:43.086402+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:09.298829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.727055+0000","last_change":"2024-03-06T07:23:41.743420+0000","last_active":"2024-03-06T07:23:42.727055+0000","last_peered":"2024-03-06T07:23:42.727055+0000","last_clean":"2024-03-06T07:23:42.727055+0000","last_became_active":"2024-03-06T07:23:41.741880+0000","last_became_peered":"2024-03-06T07:23:41.741880+0000","last_unstale":"2024-03-06T07:23:42.727055+0000","last_undegraded":"2024-03-06T07:23:42.727055+0000","last_fullsized":"2024-03-06T07:23:42.727055+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:47.862783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699848+0000","last_change":"2024-03-06T07:23:40.722676+0000","last_active":"2024-03-06T07:23:43.699848+0000","last_peered":"2024-03-06T07:23:43.699848+0000","last_clean":"2024-03-06T07:23:43.699848+0000","last_became_active":"2024-03-06T07:23:40.721231+0000","last_became_peered":"2024-03-06T07:23:40.721231+0000","last_unstale":"2024-03-06T07:23:43.699848+0000","last_undegraded":"2024-03-06T07:23:43.699848+0000","last_fullsized":"2024-03-06T07:23:43.699848+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:16.374358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.700358+0000","last_change":"2024-03-06T07:23:41.734428+0000","last_active":"2024-03-06T07:23:43.700358+0000","last_peered":"2024-03-06T07:23:43.700358+0000","last_clean":"2024-03-06T07:23:43.700358+0000","last_became_active":"2024-03-06T07:23:41.733914+0000","last_became_peered":"2024-03-06T07:23:41.733914+0000","last_unstale":"2024-03-06T07:23:43.700358+0000","last_undegraded":"2024-03-06T07:23:43.700358+0000","last_fullsized":"2024-03-06T07:23:43.700358+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:36.196650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.726867+0000","last_change":"2024-03-06T07:23:40.723204+0000","last_active":"2024-03-06T07:23:42.726867+0000","last_peered":"2024-03-06T07:23:42.726867+0000","last_clean":"2024-03-06T07:23:42.726867+0000","last_became_active":"2024-03-06T07:23:40.722931+0000","last_became_peered":"2024-03-06T07:23:40.722931+0000","last_unstale":"2024-03-06T07:23:42.726867+0000","last_undegraded":"2024-03-06T07:23:42.726867+0000","last_fullsized":"2024-03-06T07:23:42.726867+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:07.611250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723708+0000","last_change":"2024-03-06T07:23:41.734517+0000","last_active":"2024-03-06T07:23:42.723708+0000","last_peered":"2024-03-06T07:23:42.723708+0000","last_clean":"2024-03-06T07:23:42.723708+0000","last_became_active":"2024-03-06T07:23:41.733611+0000","last_became_peered":"2024-03-06T07:23:41.733611+0000","last_unstale":"2024-03-06T07:23:42.723708+0000","last_undegraded":"2024-03-06T07:23:42.723708+0000","last_fullsized":"2024-03-06T07:23:42.723708+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:15.043199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699441+0000","last_change":"2024-03-06T07:23:40.715361+0000","last_active":"2024-03-06T07:23:43.699441+0000","last_peered":"2024-03-06T07:23:43.699441+0000","last_clean":"2024-03-06T07:23:43.699441+0000","last_became_active":"2024-03-06T07:23:40.715071+0000","last_became_peered":"2024-03-06T07:23:40.715071+0000","last_unstale":"2024-03-06T07:23:43.699441+0000","last_undegraded":"2024-03-06T07:23:43.699441+0000","last_fullsized":"2024-03-06T07:23:43.699441+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:34.043452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723690+0000","last_change":"2024-03-06T07:23:41.734270+0000","last_active":"2024-03-06T07:23:42.723690+0000","last_peered":"2024-03-06T07:23:42.723690+0000","last_clean":"2024-03-06T07:23:42.723690+0000","last_became_active":"2024-03-06T07:23:41.733992+0000","last_became_peered":"2024-03-06T07:23:41.733992+0000","last_unstale":"2024-03-06T07:23:42.723690+0000","last_undegraded":"2024-03-06T07:23:42.723690+0000","last_fullsized":"2024-03-06T07:23:42.723690+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:20.647745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723497+0000","last_change":"2024-03-06T07:23:40.723885+0000","last_active":"2024-03-06T07:23:42.723497+0000","last_peered":"2024-03-06T07:23:42.723497+0000","last_clean":"2024-03-06T07:23:42.723497+0000","last_became_active":"2024-03-06T07:23:40.723444+0000","last_became_peered":"2024-03-06T07:23:40.723444+0000","last_unstale":"2024-03-06T07:23:42.723497+0000","last_undegraded":"2024-03-06T07:23:42.723497+0000","last_fullsized":"2024-03-06T07:23:42.723497+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:25:26.826506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.430469+0000","last_change":"2024-03-06T07:23:41.726164+0000","last_active":"2024-03-06T07:23:43.430469+0000","last_peered":"2024-03-06T07:23:43.430469+0000","last_clean":"2024-03-06T07:23:43.430469+0000","last_became_active":"2024-03-06T07:23:41.725877+0000","last_became_peered":"2024-03-06T07:23:41.725877+0000","last_unstale":"2024-03-06T07:23:43.430469+0000","last_undegraded":"2024-03-06T07:23:43.430469+0000","last_fullsized":"2024-03-06T07:23:43.430469+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:01.253076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.699619+0000","last_change":"2024-03-06T07:23:40.716579+0000","last_active":"2024-03-06T07:23:43.699619+0000","last_peered":"2024-03-06T07:23:43.699619+0000","last_clean":"2024-03-06T07:23:43.699619+0000","last_became_active":"2024-03-06T07:23:40.716316+0000","last_became_peered":"2024-03-06T07:23:40.716316+0000","last_unstale":"2024-03-06T07:23:43.699619+0000","last_undegraded":"2024-03-06T07:23:43.699619+0000","last_fullsized":"2024-03-06T07:23:43.699619+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:12.195727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.700572+0000","last_change":"2024-03-06T07:23:41.726998+0000","last_active":"2024-03-06T07:23:43.700572+0000","last_peered":"2024-03-06T07:23:43.700572+0000","last_clean":"2024-03-06T07:23:43.700572+0000","last_became_active":"2024-03-06T07:23:41.726749+0000","last_became_peered":"2024-03-06T07:23:41.726749+0000","last_unstale":"2024-03-06T07:23:43.700572+0000","last_undegraded":"2024-03-06T07:23:43.700572+0000","last_fullsized":"2024-03-06T07:23:43.700572+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:34.635451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723535+0000","last_change":"2024-03-06T07:23:40.723580+0000","last_active":"2024-03-06T07:23:42.723535+0000","last_peered":"2024-03-06T07:23:42.723535+0000","last_clean":"2024-03-06T07:23:42.723535+0000","last_became_active":"2024-03-06T07:23:40.723280+0000","last_became_peered":"2024-03-06T07:23:40.723280+0000","last_unstale":"2024-03-06T07:23:42.723535+0000","last_undegraded":"2024-03-06T07:23:42.723535+0000","last_fullsized":"2024-03-06T07:23:42.723535+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:21.865596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.700574+0000","last_change":"2024-03-06T07:23:41.724024+0000","last_active":"2024-03-06T07:23:43.700574+0000","last_peered":"2024-03-06T07:23:43.700574+0000","last_clean":"2024-03-06T07:23:43.700574+0000","last_became_active":"2024-03-06T07:23:41.723754+0000","last_became_peered":"2024-03-06T07:23:41.723754+0000","last_unstale":"2024-03-06T07:23:43.700574+0000","last_undegraded":"2024-03-06T07:23:43.700574+0000","last_fullsized":"2024-03-06T07:23:43.700574+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.698942+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.698942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:12.763384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.723872+0000","last_change":"2024-03-06T07:23:40.722423+0000","last_active":"2024-03-06T07:23:42.723872+0000","last_peered":"2024-03-06T07:23:42.723872+0000","last_clean":"2024-03-06T07:23:42.723872+0000","last_became_active":"2024-03-06T07:23:40.721963+0000","last_became_peered":"2024-03-06T07:23:40.721963+0000","last_unstale":"2024-03-06T07:23:42.723872+0000","last_undegraded":"2024-03-06T07:23:42.723872+0000","last_fullsized":"2024-03-06T07:23:42.723872+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.694297+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.694297+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:11.695663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":314,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.430108+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:23:43.430108+0000","last_peered":"2024-03-06T07:23:43.430108+0000","last_clean":"2024-03-06T07:23:43.430108+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:23:43.430108+0000","last_undegraded":"2024-03-06T07:23:43.430108+0000","last_fullsized":"2024-03-06T07:23:43.430108+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32404,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339436,"statfs":{"total":96636764160,"available":96603582464,"internally_reserved":0,"allocated":1253376,"data_stored":1160594,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32352,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339488,"statfs":{"total":96636764160,"available":96603635712,"internally_reserved":0,"allocated":1200128,"data_stored":572282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574892,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32332,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339508,"statfs":{"total":96636764160,"available":96603656192,"internally_reserved":0,"allocated":1179648,"data_stored":570066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574893,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32340,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339500,"statfs":{"total":96636764160,"available":96603648000,"internally_reserved":0,"allocated":1187840,"data_stored":570158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32344,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339496,"statfs":{"total":96636764160,"available":96603643904,"internally_reserved":0,"allocated":1183744,"data_stored":570648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574892,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32340,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339500,"statfs":{"total":96636764160,"available":96603648000,"internally_reserved":0,"allocated":1183744,"data_stored":570100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32344,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339496,"statfs":{"total":96636764160,"available":96603643904,"internally_reserved":0,"allocated":1183744,"data_stored":570100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574893,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":1261568,"data_stored":1162136,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:45.081 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:23:45.082 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:23:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.224+0000 7f0e0ce5c640 1 Processor -- start 2024-03-06T07:23:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.224+0000 7f0e0ce5c640 1 -- start start 2024-03-06T07:23:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.225+0000 7f0e0ce5c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e080c7560 0x7f0e08155860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.225+0000 7f0e0ce5c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0e08155da0 0x7f0e080c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.225+0000 7f0e0ce5c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e08155080 con 0x7f0e08152890 2024-03-06T07:23:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.225+0000 7f0e0ce5c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e080c35b0 con 0x7f0e08155da0 2024-03-06T07:23:45.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.225+0000 7f0e0ce5c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e080c3890 con 0x7f0e080c7560 2024-03-06T07:23:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.225+0000 7f0e06d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0e08155da0 0x7f0e080c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.225+0000 7f0e0659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0e08152890 0x7f0e08159030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36024/0 (socket says 172.21.15.1:36024) 2024-03-06T07:23:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.225+0000 7f0e0659c640 1 -- 172.21.15.1:0/4237208403 learned_addr learned my addr 172.21.15.1:0/4237208403 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.225+0000 7f0dfffff640 1 --2- 172.21.15.1:0/4237208403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e080c7560 0x7f0e08155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.225+0000 7f0dfffff640 1 -- 172.21.15.1:0/4237208403 >> v2:172.21.15.1:3300/0 conn(0x7f0e08155da0 msgr2=0x7f0e080c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.225+0000 7f0dfffff640 1 --2- 172.21.15.1:0/4237208403 >> v2:172.21.15.1:3300/0 conn(0x7f0e08155da0 0x7f0e080c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.225+0000 7f0dfffff640 1 -- 172.21.15.1:0/4237208403 >> v1:172.21.15.1:6789/0 conn(0x7f0e08152890 legacy=0x7f0e08159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.225+0000 7f0e05d9b640 1 -- 172.21.15.1:0/4237208403 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945231318 0 0) 0x7f0e08155080 con 0x7f0e08152890 2024-03-06T07:23:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.225+0000 7f0dfffff640 1 -- 172.21.15.1:0/4237208403 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e080c3b70 con 0x7f0e080c7560 2024-03-06T07:23:45.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.225+0000 7f0e06d9d640 1 --2- 172.21.15.1:0/4237208403 >> v2:172.21.15.1:3300/0 conn(0x7f0e08155da0 0x7f0e080c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.226+0000 7f0dfffff640 1 --2- 172.21.15.1:0/4237208403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e080c7560 0x7f0e08155860 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0df00097d0 tx=0x7f0df002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b43560d7e7ff4c7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.226+0000 7f0e05d9b640 1 -- 172.21.15.1:0/4237208403 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df0039070 con 0x7f0e080c7560 2024-03-06T07:23:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.226+0000 7f0e05d9b640 1 -- 172.21.15.1:0/4237208403 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df00316f0 con 0x7f0e080c7560 2024-03-06T07:23:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.226+0000 7f0e05d9b640 1 -- 172.21.15.1:0/4237208403 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df0031ac0 con 0x7f0e080c7560 2024-03-06T07:23:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.227+0000 7f0e0ce5c640 1 -- 172.21.15.1:0/4237208403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e080c7560 msgr2=0x7f0e08155860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.227+0000 7f0e0ce5c640 1 --2- 172.21.15.1:0/4237208403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e080c7560 0x7f0e08155860 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0df00097d0 tx=0x7f0df002b790 comp rx=0 tx=0).stop 2024-03-06T07:23:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.227+0000 7f0e0ce5c640 1 -- 172.21.15.1:0/4237208403 shutdown_connections 2024-03-06T07:23:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.227+0000 7f0e0ce5c640 1 --2- 172.21.15.1:0/4237208403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e080c7560 0x7f0e08155860 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.227+0000 7f0e0ce5c640 1 --2- 172.21.15.1:0/4237208403 >> v2:172.21.15.1:3300/0 conn(0x7f0e08155da0 0x7f0e080c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.227+0000 7f0e0ce5c640 1 -- 172.21.15.1:0/4237208403 >> 172.21.15.1:0/4237208403 conn(0x7f0e0806b3e0 msgr2=0x7f0e0806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.227+0000 7f0e0ce5c640 1 -- 172.21.15.1:0/4237208403 shutdown_connections 2024-03-06T07:23:45.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.227+0000 7f0e0ce5c640 1 -- 172.21.15.1:0/4237208403 wait complete. 2024-03-06T07:23:45.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.228+0000 7f0e0ce5c640 1 Processor -- start 2024-03-06T07:23:45.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.228+0000 7f0e0ce5c640 1 -- start start 2024-03-06T07:23:45.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.228+0000 7f0e0ce5c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e080c7560 0x7f0e08151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.228+0000 7f0e0ce5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e08152890 0x7f0e081e19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0e0ce5c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e080acd00 0x7f0e080aa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0e0ce5c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e080a81f0 con 0x7f0e080c7560 2024-03-06T07:23:45.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0e0ce5c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e080c3260 con 0x7f0e08152890 2024-03-06T07:23:45.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0e0ce5c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e080a8370 con 0x7f0e080acd00 2024-03-06T07:23:45.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0e0659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e080c7560 0x7f0e08151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0e0659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e080c7560 0x7f0e08151ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36908/0 (socket says 172.21.15.1:36908) 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0e0659c640 1 -- 172.21.15.1:0/2624734754 learned_addr learned my addr 172.21.15.1:0/2624734754 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0e06d9d640 1 --2- 172.21.15.1:0/2624734754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e080acd00 0x7f0e080aa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0dfffff640 1 --2- 172.21.15.1:0/2624734754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e08152890 0x7f0e081e19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0e0659c640 1 -- 172.21.15.1:0/2624734754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e080acd00 msgr2=0x7f0e080aa5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0e0659c640 1 --2- 172.21.15.1:0/2624734754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e080acd00 0x7f0e080aa5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0e0659c640 1 -- 172.21.15.1:0/2624734754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e08152890 msgr2=0x7f0e081e19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0e0659c640 1 --2- 172.21.15.1:0/2624734754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e08152890 0x7f0e081e19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0e0659c640 1 -- 172.21.15.1:0/2624734754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e081e24d0 con 0x7f0e080c7560 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0e0659c640 1 --2- 172.21.15.1:0/2624734754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e080c7560 0x7f0e08151ed0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f0df800c4d0 tx=0x7f0df800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0dfeffd640 1 -- 172.21.15.1:0/2624734754 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df8018070 con 0x7f0e080c7560 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0dfeffd640 1 -- 172.21.15.1:0/2624734754 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df8007500 con 0x7f0e080c7560 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0e0ce5c640 1 -- 172.21.15.1:0/2624734754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e081e21f0 con 0x7f0e080c7560 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0dfeffd640 1 -- 172.21.15.1:0/2624734754 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df8004a20 con 0x7f0e080c7560 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.229+0000 7f0e0ce5c640 1 -- 172.21.15.1:0/2624734754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e081eb9c0 con 0x7f0e080c7560 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.230+0000 7f0e0ce5c640 1 -- 172.21.15.1:0/2624734754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e08067fd0 con 0x7f0e080c7560 2024-03-06T07:23:45.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.231+0000 7f0dfeffd640 1 -- 172.21.15.1:0/2624734754 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0df801b070 con 0x7f0e080c7560 2024-03-06T07:23:45.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.231+0000 7f0dfeffd640 1 --2- 172.21.15.1:0/2624734754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0de806b4d0 0x7f0de806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.231+0000 7f0dfeffd640 1 -- 172.21.15.1:0/2624734754 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0df8089590 con 0x7f0e080c7560 2024-03-06T07:23:45.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.231+0000 7f0dfffff640 1 --2- 172.21.15.1:0/2624734754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0de806b4d0 0x7f0de806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.232+0000 7f0dfffff640 1 --2- 172.21.15.1:0/2624734754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0de806b4d0 0x7f0de806d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0df0036000 tx=0x7f0df0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.235+0000 7f0dfeffd640 1 -- 172.21.15.1:0/2624734754 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0df8057cc0 con 0x7f0e080c7560 2024-03-06T07:23:45.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.488+0000 7f0e0ce5c640 1 -- 172.21.15.1:0/2624734754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f0e080c2d80 con 0x7f0e080c7560 2024-03-06T07:23:45.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.489+0000 7f0dfeffd640 1 -- 172.21.15.1:0/2624734754 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f0df805bd60 con 0x7f0e080c7560 2024-03-06T07:23:45.491 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-06T07:23:45.492 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng== 2024-03-06T07:23:45.492 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-06T07:23:45.492 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-06T07:23:45.492 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:45.492 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-06T07:23:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.491+0000 7f0e0ce5c640 1 -- 172.21.15.1:0/2624734754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0de806b4d0 msgr2=0x7f0de806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.491+0000 7f0e0ce5c640 1 --2- 172.21.15.1:0/2624734754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0de806b4d0 0x7f0de806d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0df0036000 tx=0x7f0df0008030 comp rx=0 tx=0).stop 2024-03-06T07:23:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.491+0000 7f0e0ce5c640 1 -- 172.21.15.1:0/2624734754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e080c7560 msgr2=0x7f0e08151ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.491+0000 7f0e0ce5c640 1 --2- 172.21.15.1:0/2624734754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e080c7560 0x7f0e08151ed0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f0df800c4d0 tx=0x7f0df800c990 comp rx=0 tx=0).stop 2024-03-06T07:23:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.491+0000 7f0e0ce5c640 1 -- 172.21.15.1:0/2624734754 shutdown_connections 2024-03-06T07:23:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.491+0000 7f0e0ce5c640 1 --2- 172.21.15.1:0/2624734754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0de806b4d0 0x7f0de806d990 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.491+0000 7f0e0ce5c640 1 --2- 172.21.15.1:0/2624734754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e080acd00 0x7f0e080aa5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.491+0000 7f0e0ce5c640 1 --2- 172.21.15.1:0/2624734754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e08152890 0x7f0e081e19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.491+0000 7f0e0ce5c640 1 --2- 172.21.15.1:0/2624734754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e080c7560 0x7f0e08151ed0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.491+0000 7f0e0ce5c640 1 -- 172.21.15.1:0/2624734754 >> 172.21.15.1:0/2624734754 conn(0x7f0e0806b3e0 msgr2=0x7f0e08158c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.492+0000 7f0e0ce5c640 1 -- 172.21.15.1:0/2624734754 shutdown_connections 2024-03-06T07:23:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.492+0000 7f0e0ce5c640 1 -- 172.21.15.1:0/2624734754 wait complete. 2024-03-06T07:23:45.504 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:45.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.603+0000 7ff357f93640 1 Processor -- start 2024-03-06T07:23:45.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.603+0000 7ff357f93640 1 -- start start 2024-03-06T07:23:45.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.604+0000 7ff357f93640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff3500c7560 0x7ff350155860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.604+0000 7ff357f93640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff350155da0 0x7ff3500c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.604+0000 7ff357f93640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff350155080 con 0x7ff350152890 2024-03-06T07:23:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.604+0000 7ff357f93640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3500c35b0 con 0x7ff3500c7560 2024-03-06T07:23:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.604+0000 7ff357f93640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500c3890 con 0x7ff350155da0 2024-03-06T07:23:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.604+0000 7ff355d2f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff350152890 0x7ff350159030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36028/0 (socket says 172.21.15.1:36028) 2024-03-06T07:23:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.604+0000 7ff355d2f640 1 -- 172.21.15.1:0/4039030374 learned_addr learned my addr 172.21.15.1:0/4039030374 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.604+0000 7ff35552e640 1 --2- 172.21.15.1:0/4039030374 >> v2:172.21.15.98:3300/0 conn(0x7ff3500c7560 0x7ff350155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.604+0000 7ff356530640 1 --2- 172.21.15.1:0/4039030374 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff350155da0 0x7ff3500c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.605+0000 7ff354d2d640 1 -- 172.21.15.1:0/4039030374 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111955374 0 0) 0x7ff350155080 con 0x7ff350152890 2024-03-06T07:23:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.605+0000 7ff35552e640 1 -- 172.21.15.1:0/4039030374 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff350155da0 msgr2=0x7ff3500c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.605+0000 7ff35552e640 1 --2- 172.21.15.1:0/4039030374 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff350155da0 0x7ff3500c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.605+0000 7ff35552e640 1 -- 172.21.15.1:0/4039030374 >> v1:172.21.15.1:6789/0 conn(0x7ff350152890 legacy=0x7ff350159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.605+0000 7ff35552e640 1 -- 172.21.15.1:0/4039030374 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3500c3b70 con 0x7ff3500c7560 2024-03-06T07:23:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.605+0000 7ff356530640 1 --2- 172.21.15.1:0/4039030374 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff350155da0 0x7ff3500c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:45.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.605+0000 7ff35552e640 1 --2- 172.21.15.1:0/4039030374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3500c7560 0x7ff350155860 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff3400097a0 tx=0x7ff34002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0c3a525761353f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.606+0000 7ff354d2d640 1 -- 172.21.15.1:0/4039030374 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340039070 con 0x7ff3500c7560 2024-03-06T07:23:45.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.606+0000 7ff354d2d640 1 -- 172.21.15.1:0/4039030374 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3400316f0 con 0x7ff3500c7560 2024-03-06T07:23:45.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.606+0000 7ff354d2d640 1 -- 172.21.15.1:0/4039030374 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340031ac0 con 0x7ff3500c7560 2024-03-06T07:23:45.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.606+0000 7ff357f93640 1 -- 172.21.15.1:0/4039030374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3500c7560 msgr2=0x7ff350155860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.606+0000 7ff357f93640 1 --2- 172.21.15.1:0/4039030374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3500c7560 0x7ff350155860 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff3400097a0 tx=0x7ff34002b790 comp rx=0 tx=0).stop 2024-03-06T07:23:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.606+0000 7ff357f93640 1 -- 172.21.15.1:0/4039030374 shutdown_connections 2024-03-06T07:23:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.606+0000 7ff357f93640 1 --2- 172.21.15.1:0/4039030374 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff350155da0 0x7ff3500c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.606+0000 7ff357f93640 1 --2- 172.21.15.1:0/4039030374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3500c7560 0x7ff350155860 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.606+0000 7ff357f93640 1 -- 172.21.15.1:0/4039030374 >> 172.21.15.1:0/4039030374 conn(0x7ff35006b3e0 msgr2=0x7ff35006b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.606+0000 7ff357f93640 1 -- 172.21.15.1:0/4039030374 shutdown_connections 2024-03-06T07:23:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.606+0000 7ff357f93640 1 -- 172.21.15.1:0/4039030374 wait complete. 2024-03-06T07:23:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.607+0000 7ff357f93640 1 Processor -- start 2024-03-06T07:23:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.607+0000 7ff357f93640 1 -- start start 2024-03-06T07:23:45.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.608+0000 7ff357f93640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3500c7560 0x7ff35006d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.608+0000 7ff357f93640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff350152890 0x7ff350150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.608+0000 7ff357f93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3501520d0 0x7ff350151150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.608+0000 7ff357f93640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500a81f0 con 0x7ff3501520d0 2024-03-06T07:23:45.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.608+0000 7ff357f93640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500c3260 con 0x7ff350152890 2024-03-06T07:23:45.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.608+0000 7ff357f93640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500a8370 con 0x7ff3500c7560 2024-03-06T07:23:45.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.608+0000 7ff356530640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3501520d0 0x7ff350151150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.608+0000 7ff356530640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3501520d0 0x7ff350151150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36910/0 (socket says 172.21.15.1:36910) 2024-03-06T07:23:45.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.608+0000 7ff356530640 1 -- 172.21.15.1:0/3796140623 learned_addr learned my addr 172.21.15.1:0/3796140623 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:45.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.608+0000 7ff355d2f640 1 --2- 172.21.15.1:0/3796140623 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3500c7560 0x7ff35006d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.608+0000 7ff35552e640 1 --2- 172.21.15.1:0/3796140623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff350152890 0x7ff350150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.608+0000 7ff356530640 1 -- 172.21.15.1:0/3796140623 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3500c7560 msgr2=0x7ff35006d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.608+0000 7ff356530640 1 --2- 172.21.15.1:0/3796140623 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3500c7560 0x7ff35006d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.608+0000 7ff356530640 1 -- 172.21.15.1:0/3796140623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff350152890 msgr2=0x7ff350150c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.608+0000 7ff356530640 1 --2- 172.21.15.1:0/3796140623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff350152890 0x7ff350150c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.608+0000 7ff356530640 1 -- 172.21.15.1:0/3796140623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35006d340 con 0x7ff3501520d0 2024-03-06T07:23:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.609+0000 7ff356530640 1 --2- 172.21.15.1:0/3796140623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3501520d0 0x7ff350151150 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7ff34800e8f0 tx=0x7ff34800edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.609+0000 7ff3467fc640 1 -- 172.21.15.1:0/3796140623 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff348017070 con 0x7ff3501520d0 2024-03-06T07:23:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.609+0000 7ff3467fc640 1 -- 172.21.15.1:0/3796140623 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff348004020 con 0x7ff3501520d0 2024-03-06T07:23:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.609+0000 7ff3467fc640 1 -- 172.21.15.1:0/3796140623 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3480043f0 con 0x7ff3501520d0 2024-03-06T07:23:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.609+0000 7ff357f93640 1 -- 172.21.15.1:0/3796140623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3501dfaa0 con 0x7ff3501520d0 2024-03-06T07:23:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.609+0000 7ff357f93640 1 -- 172.21.15.1:0/3796140623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3501e0060 con 0x7ff3501520d0 2024-03-06T07:23:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.610+0000 7ff357f93640 1 -- 172.21.15.1:0/3796140623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff350067fd0 con 0x7ff3501520d0 2024-03-06T07:23:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.610+0000 7ff3467fc640 1 -- 172.21.15.1:0/3796140623 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff34800c940 con 0x7ff3501520d0 2024-03-06T07:23:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.611+0000 7ff3467fc640 1 --2- 172.21.15.1:0/3796140623 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff33406b4d0 0x7ff33406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.611+0000 7ff3467fc640 1 -- 172.21.15.1:0/3796140623 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff34808a480 con 0x7ff3501520d0 2024-03-06T07:23:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.611+0000 7ff355d2f640 1 --2- 172.21.15.1:0/3796140623 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff33406b4d0 0x7ff33406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.611+0000 7ff355d2f640 1 --2- 172.21.15.1:0/3796140623 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff33406b4d0 0x7ff33406d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0012e0 tx=0x7ff34c001630 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.615+0000 7ff3467fc640 1 -- 172.21.15.1:0/3796140623 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff348058b30 con 0x7ff3501520d0 2024-03-06T07:23:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.836+0000 7ff357f93640 1 -- 172.21.15.1:0/3796140623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3500c2d80 con 0x7ff3501520d0 2024-03-06T07:23:45.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.837+0000 7ff3467fc640 1 -- 172.21.15.1:0/3796140623 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4970 (secure 0 0 0) 0x7ff34805cbd0 con 0x7ff3501520d0 2024-03-06T07:23:45.839 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:45.839 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":77,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1335402413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1335402413},{"type":"v1","addr":"172.21.15.98:6836","nonce":1335402413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6981,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/2118550222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2118550222},{"type":"v1","addr":"172.21.15.1:6834","nonce":2118550222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/543699854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":543699854},{"type":"v1","addr":"172.21.15.1:6835","nonce":543699854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:41.077040+0000","modified":"2024-03-06T07:23:43.077835+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6996},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6996":{"gid":6996,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/3330683497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3330683497},{"type":"v1","addr":"172.21.15.98:6837","nonce":3330683497}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6996,"qdb_cluster":[6996]},"id":7}]} 2024-03-06T07:23:45.839 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 77 2024-03-06T07:23:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.839+0000 7ff357f93640 1 -- 172.21.15.1:0/3796140623 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff33406b4d0 msgr2=0x7ff33406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.839+0000 7ff357f93640 1 --2- 172.21.15.1:0/3796140623 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff33406b4d0 0x7ff33406d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0012e0 tx=0x7ff34c001630 comp rx=0 tx=0).stop 2024-03-06T07:23:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.839+0000 7ff357f93640 1 -- 172.21.15.1:0/3796140623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3501520d0 msgr2=0x7ff350151150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.839+0000 7ff357f93640 1 --2- 172.21.15.1:0/3796140623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3501520d0 0x7ff350151150 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7ff34800e8f0 tx=0x7ff34800edb0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.839+0000 7ff357f93640 1 -- 172.21.15.1:0/3796140623 shutdown_connections 2024-03-06T07:23:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.839+0000 7ff357f93640 1 --2- 172.21.15.1:0/3796140623 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff33406b4d0 0x7ff33406d990 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.839+0000 7ff357f93640 1 --2- 172.21.15.1:0/3796140623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3501520d0 0x7ff350151150 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.839+0000 7ff357f93640 1 --2- 172.21.15.1:0/3796140623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff350152890 0x7ff350150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.839+0000 7ff357f93640 1 --2- 172.21.15.1:0/3796140623 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3500c7560 0x7ff35006d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.839+0000 7ff357f93640 1 -- 172.21.15.1:0/3796140623 >> 172.21.15.1:0/3796140623 conn(0x7ff35006b3e0 msgr2=0x7ff3500c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.839+0000 7ff357f93640 1 -- 172.21.15.1:0/3796140623 shutdown_connections 2024-03-06T07:23:45.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:45.839+0000 7ff357f93640 1 -- 172.21.15.1:0/3796140623 wait complete. 2024-03-06T07:23:46.254 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:41.077040+0000', 'modified': '2024-03-06T07:23:43.077835+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6996}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6996': {'gid': 6996, 'name': 'd', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.98:6837/3330683497', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3330683497}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3330683497}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6996, 'qdb_cluster': [6996]} 2024-03-06T07:23:46.254 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:46.255 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:23:46.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.360+0000 7efe089d0640 1 Processor -- start 2024-03-06T07:23:46.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.360+0000 7efe089d0640 1 -- start start 2024-03-06T07:23:46.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.361+0000 7efe089d0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efe04072f20 0x7efe041513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.361+0000 7efe089d0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efe04071f90 0x7efe04072400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.361+0000 7efe089d0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe040a6140 con 0x7efe04150fe0 2024-03-06T07:23:46.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.361+0000 7efe089d0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe041526b0 con 0x7efe04072f20 2024-03-06T07:23:46.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.361+0000 7efe089d0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe04151aa0 con 0x7efe04071f90 2024-03-06T07:23:46.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.361+0000 7efe0259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efe04150fe0 0x7efe0406edb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36044/0 (socket says 172.21.15.1:36044) 2024-03-06T07:23:46.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.361+0000 7efe0259c640 1 -- 172.21.15.1:0/1193164466 learned_addr learned my addr 172.21.15.1:0/1193164466 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:46.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.361+0000 7efdfbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efe04072f20 0x7efe041513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.361+0000 7efe02d9d640 1 --2- 172.21.15.1:0/1193164466 >> v2:172.21.15.98:3300/0 conn(0x7efe04071f90 0x7efe04072400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.361+0000 7efdfbfff640 1 -- 172.21.15.1:0/1193164466 >> v2:172.21.15.98:3300/0 conn(0x7efe04071f90 msgr2=0x7efe04072400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:46.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.361+0000 7efdfbfff640 1 --2- 172.21.15.1:0/1193164466 >> v2:172.21.15.98:3300/0 conn(0x7efe04071f90 0x7efe04072400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.361+0000 7efdfbfff640 1 -- 172.21.15.1:0/1193164466 >> v1:172.21.15.1:6789/0 conn(0x7efe04150fe0 legacy=0x7efe0406edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.361+0000 7efdfbfff640 1 -- 172.21.15.1:0/1193164466 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe04151d80 con 0x7efe04072f20 2024-03-06T07:23:46.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.361+0000 7efe01d9b640 1 -- 172.21.15.1:0/1193164466 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270152921 0 0) 0x7efe040a6140 con 0x7efe04150fe0 2024-03-06T07:23:46.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.361+0000 7efe02d9d640 1 --2- 172.21.15.1:0/1193164466 >> v2:172.21.15.98:3300/0 conn(0x7efe04071f90 0x7efe04072400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:46.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.362+0000 7efdfbfff640 1 --2- 172.21.15.1:0/1193164466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe04072f20 0x7efe041513c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7efe040714d0 tx=0x7efdec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b103c20f9766c9be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.362+0000 7efe01d9b640 1 -- 172.21.15.1:0/1193164466 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec0315a0 con 0x7efe04072f20 2024-03-06T07:23:46.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.362+0000 7efe01d9b640 1 -- 172.21.15.1:0/1193164466 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdec037070 con 0x7efe04072f20 2024-03-06T07:23:46.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.362+0000 7efe01d9b640 1 -- 172.21.15.1:0/1193164466 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec031960 con 0x7efe04072f20 2024-03-06T07:23:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.363+0000 7efe089d0640 1 -- 172.21.15.1:0/1193164466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe04072f20 msgr2=0x7efe041513c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.363+0000 7efe089d0640 1 --2- 172.21.15.1:0/1193164466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe04072f20 0x7efe041513c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7efe040714d0 tx=0x7efdec02b840 comp rx=0 tx=0).stop 2024-03-06T07:23:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.363+0000 7efe089d0640 1 -- 172.21.15.1:0/1193164466 shutdown_connections 2024-03-06T07:23:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.363+0000 7efe089d0640 1 --2- 172.21.15.1:0/1193164466 >> v2:172.21.15.98:3300/0 conn(0x7efe04071f90 0x7efe04072400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.363+0000 7efe089d0640 1 --2- 172.21.15.1:0/1193164466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe04072f20 0x7efe041513c0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.363+0000 7efe089d0640 1 -- 172.21.15.1:0/1193164466 >> 172.21.15.1:0/1193164466 conn(0x7efe040c6860 msgr2=0x7efe040c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.363+0000 7efe089d0640 1 -- 172.21.15.1:0/1193164466 shutdown_connections 2024-03-06T07:23:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.363+0000 7efe089d0640 1 -- 172.21.15.1:0/1193164466 wait complete. 2024-03-06T07:23:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.364+0000 7efe089d0640 1 Processor -- start 2024-03-06T07:23:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.364+0000 7efe089d0640 1 -- start start 2024-03-06T07:23:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efe089d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efe04071f90 0x7efe041468c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efe089d0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe04072f20 0x7efe0415d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efe089d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe04150fe0 0x7efe040bc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efe089d0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0406f4c0 con 0x7efe04072f20 2024-03-06T07:23:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efe089d0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe04152340 con 0x7efe04150fe0 2024-03-06T07:23:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efe089d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0406f640 con 0x7efe04071f90 2024-03-06T07:23:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efdfbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe04072f20 0x7efe0415d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efdfbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe04072f20 0x7efe0415d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36932/0 (socket says 172.21.15.1:36932) 2024-03-06T07:23:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efdfbfff640 1 -- 172.21.15.1:0/397542727 learned_addr learned my addr 172.21.15.1:0/397542727 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efe0259c640 1 --2- 172.21.15.1:0/397542727 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efe04071f90 0x7efe041468c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efe02d9d640 1 --2- 172.21.15.1:0/397542727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe04150fe0 0x7efe040bc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efdfbfff640 1 -- 172.21.15.1:0/397542727 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efe04071f90 msgr2=0x7efe041468c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efdfbfff640 1 --2- 172.21.15.1:0/397542727 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efe04071f90 0x7efe041468c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efdfbfff640 1 -- 172.21.15.1:0/397542727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe04150fe0 msgr2=0x7efe040bc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efdfbfff640 1 --2- 172.21.15.1:0/397542727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe04150fe0 0x7efe040bc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efdfbfff640 1 -- 172.21.15.1:0/397542727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0414e560 con 0x7efe04072f20 2024-03-06T07:23:46.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efdfbfff640 1 --2- 172.21.15.1:0/397542727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe04072f20 0x7efe0415d610 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7efdec034040 tx=0x7efdec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.365+0000 7efdfaffd640 1 -- 172.21.15.1:0/397542727 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec032630 con 0x7efe04072f20 2024-03-06T07:23:46.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.366+0000 7efdfaffd640 1 -- 172.21.15.1:0/397542727 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdec0327f0 con 0x7efe04072f20 2024-03-06T07:23:46.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.366+0000 7efe089d0640 1 -- 172.21.15.1:0/397542727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe0414e840 con 0x7efe04072f20 2024-03-06T07:23:46.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.366+0000 7efdfaffd640 1 -- 172.21.15.1:0/397542727 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec032be0 con 0x7efe04072f20 2024-03-06T07:23:46.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.366+0000 7efe089d0640 1 -- 172.21.15.1:0/397542727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe04146e00 con 0x7efe04072f20 2024-03-06T07:23:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.367+0000 7efdfaffd640 1 -- 172.21.15.1:0/397542727 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efdec04e430 con 0x7efe04072f20 2024-03-06T07:23:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.367+0000 7efdfaffd640 1 --2- 172.21.15.1:0/397542727 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efde406b500 0x7efde406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.367+0000 7efdfaffd640 1 -- 172.21.15.1:0/397542727 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efdec02f030 con 0x7efe04072f20 2024-03-06T07:23:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.367+0000 7efe0259c640 1 --2- 172.21.15.1:0/397542727 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efde406b500 0x7efde406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.368+0000 7efe0259c640 1 --2- 172.21.15.1:0/397542727 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efde406b500 0x7efde406d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7efdf4000f60 tx=0x7efdf4007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.368+0000 7efe0b5b0740 1 Processor -- start 2024-03-06T07:23:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.369+0000 7efe0b5b0740 1 -- start start 2024-03-06T07:23:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.369+0000 7efe0b5b0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fa2f50 0x562b81fa0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.369+0000 7efe0b5b0740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fa3330 0x562b81fa0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.369+0000 7efe0b5b0740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa14b0 0x562b81f9d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.369+0000 7efe0b5b0740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81f076b0 con 0x562b81fa14b0 2024-03-06T07:23:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.369+0000 7efe0b5b0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81eb0a60 con 0x562b81fa2f50 2024-03-06T07:23:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.369+0000 7efe0b5b0740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x562b81f03750 con 0x562b81fa3330 2024-03-06T07:23:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.369+0000 7efe02d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa14b0 0x562b81f9d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.369+0000 7efe02d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa14b0 0x562b81f9d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36936/0 (socket says 172.21.15.1:36936) 2024-03-06T07:23:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.369+0000 7efe0259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fa2f50 0x562b81fa0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.369+0000 7efe02d9d640 1 -- 172.21.15.1:0/2379465442 learned_addr learned my addr 172.21.15.1:0/2379465442 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.369+0000 7efdfbfff640 1 --2- 172.21.15.1:0/2379465442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fa3330 0x562b81fa0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.370+0000 7efe02d9d640 1 -- 172.21.15.1:0/2379465442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fa3330 msgr2=0x562b81fa0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.370+0000 7efe02d9d640 1 --2- 172.21.15.1:0/2379465442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fa3330 0x562b81fa0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.370+0000 7efe02d9d640 1 -- 172.21.15.1:0/2379465442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fa2f50 msgr2=0x562b81fa0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.370+0000 7efe02d9d640 1 --2- 172.21.15.1:0/2379465442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fa2f50 0x562b81fa0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.370+0000 7efe02d9d640 1 -- 172.21.15.1:0/2379465442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b81ecef30 con 0x562b81fa14b0 2024-03-06T07:23:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.370+0000 7efe02d9d640 1 --2- 172.21.15.1:0/2379465442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa14b0 0x562b81f9d680 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7efdfc011360 tx=0x7efdfc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b08588b75d78a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.370+0000 7efde3fff640 1 -- 172.21.15.1:0/2379465442 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc003b90 con 0x562b81fa14b0 2024-03-06T07:23:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.370+0000 7efde3fff640 1 -- 172.21.15.1:0/2379465442 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdfc003d30 con 0x562b81fa14b0 2024-03-06T07:23:46.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.370+0000 7efde3fff640 1 -- 172.21.15.1:0/2379465442 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc004260 con 0x562b81fa14b0 2024-03-06T07:23:46.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.371+0000 7efe0b5b0740 1 -- 172.21.15.1:0/2379465442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa14b0 msgr2=0x562b81f9d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.371+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/2379465442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa14b0 0x562b81f9d680 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7efdfc011360 tx=0x7efdfc009750 comp rx=0 tx=0).stop 2024-03-06T07:23:46.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.371+0000 7efe0b5b0740 1 -- 172.21.15.1:0/2379465442 shutdown_connections 2024-03-06T07:23:46.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.371+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/2379465442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa14b0 0x562b81f9d680 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.371+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/2379465442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fa3330 0x562b81fa0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.371+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/2379465442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fa2f50 0x562b81fa0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.371+0000 7efe0b5b0740 1 -- 172.21.15.1:0/2379465442 >> 172.21.15.1:0/2379465442 conn(0x562b81f9f7d0 msgr2=0x562b81fc43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:46.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.371+0000 7efe0b5b0740 1 -- 172.21.15.1:0/2379465442 shutdown_connections 2024-03-06T07:23:46.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.371+0000 7efe0b5b0740 1 -- 172.21.15.1:0/2379465442 wait complete. 2024-03-06T07:23:46.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.371+0000 7efe0b5b0740 1 -- start start 2024-03-06T07:23:46.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.371+0000 7efe0b5b0740 1 Processor -- start 2024-03-06T07:23:46.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.372+0000 7efe0b5b0740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fa2f50 0x562b81fc37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.372+0000 7efe0b5b0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fa3330 0x562b81fc71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.372+0000 7efe0b5b0740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fc9ba0 0x562b81fbfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.372+0000 7efe0b5b0740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81fc5230 con 0x562b81fc9ba0 2024-03-06T07:23:46.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.372+0000 7efe0b5b0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81ec3ea0 con 0x562b81fa3330 2024-03-06T07:23:46.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.372+0000 7efe0b5b0740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x562b81fc4c70 con 0x562b81fa2f50 2024-03-06T07:23:46.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.372+0000 7efe0b5b0740 10 client.0 _ll_register_callbacks cb 0x562b81ec3870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:46.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.372+0000 7efe02d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fc9ba0 0x562b81fbfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.372+0000 7efe02d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fc9ba0 0x562b81fbfdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36950/0 (socket says 172.21.15.1:36950) 2024-03-06T07:23:46.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.372+0000 7efe02d9d640 1 -- 172.21.15.1:0/3623967103 learned_addr learned my addr 172.21.15.1:0/3623967103 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:46.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.372+0000 7efe0259c640 1 --2- 172.21.15.1:0/3623967103 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fa2f50 0x562b81fc37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.372+0000 7efdfbfff640 1 --2- 172.21.15.1:0/3623967103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fa3330 0x562b81fc71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.372+0000 7efe02d9d640 1 -- 172.21.15.1:0/3623967103 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fa2f50 msgr2=0x562b81fc37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:46.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.372+0000 7efe02d9d640 1 --2- 172.21.15.1:0/3623967103 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fa2f50 0x562b81fc37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efe02d9d640 1 -- 172.21.15.1:0/3623967103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fa3330 msgr2=0x562b81fc71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:46.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efe02d9d640 1 --2- 172.21.15.1:0/3623967103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fa3330 0x562b81fc71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efe02d9d640 1 -- 172.21.15.1:0/3623967103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562b81ec4270 con 0x562b81fc9ba0 2024-03-06T07:23:46.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efe02d9d640 1 --2- 172.21.15.1:0/3623967103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fc9ba0 0x562b81fbfdc0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x562b81fe25d0 tx=0x7efdfc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595333bf65895614 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efde17fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:23:46.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efde17fa640 1 -- 172.21.15.1:0/3623967103 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc02d060 con 0x562b81fc9ba0 2024-03-06T07:23:46.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efde17fa640 1 -- 172.21.15.1:0/3623967103 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efdfc0158f0 con 0x562b81fc9ba0 2024-03-06T07:23:46.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efe0b5b0740 1 -- 172.21.15.1:0/3623967103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efdfc003d30 con 0x562b81fc9ba0 2024-03-06T07:23:46.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efde17fa640 1 -- 172.21.15.1:0/3623967103 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdfc033050 con 0x562b81fc9ba0 2024-03-06T07:23:46.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efe0b5b0740 10 client.7119 fetch_fsmap learned FSMap version 77 2024-03-06T07:23:46.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efe0b5b0740 1 -- 172.21.15.1:0/3623967103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x562b81c1c2d0 con 0x562b81fc9ba0 2024-03-06T07:23:46.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efde17fa640 1 -- 172.21.15.1:0/3623967103 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdfc003d30 con 0x562b81fc9ba0 2024-03-06T07:23:46.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efde17fa640 1 -- 172.21.15.1:0/3623967103 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 77) v1 ==== 3314+0+0 (secure 0 0 0) 0x7efdfc036020 con 0x562b81fc9ba0 2024-03-06T07:23:46.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efe0b5b0740 10 client.7119 fetch_fsmap finished waiting for FSMap version 77 2024-03-06T07:23:46.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efe0b5b0740 10 client.7119 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:23:46.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efe0b5b0740 20 client.7119 populate_metadata read hostname 'smithi001' 2024-03-06T07:23:46.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/3623967103 >> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] conn(0x562b81fc0480 0x562b81fc5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efe0b5b0740 4 client.7119 mds_command: new command op to 6996 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:46.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.373+0000 7efe0b5b0740 1 -- 172.21.15.1:0/3623967103 --> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562b81f14bf0 con 0x562b81fc0480 2024-03-06T07:23:46.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.374+0000 7efe0259c640 1 --2- 172.21.15.1:0/3623967103 >> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] conn(0x562b81fc0480 0x562b81fc5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.374+0000 7efde17fa640 10 client.7119 ms_handle_connect on v2:172.21.15.98:6835/3330683497 2024-03-06T07:23:46.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.374+0000 7efe0259c640 1 --2- 172.21.15.1:0/3623967103 >> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] conn(0x562b81fc0480 0x562b81fc5fe0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8fab54b4341dbda server_cookie=704f8979aa58a511 in_seq=0 out_seq=0 2024-03-06T07:23:46.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.375+0000 7efde17fa640 1 -- 172.21.15.1:0/3623967103 <== mds.0 v2:172.21.15.98:6835/3330683497 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562b81f14bf0 con 0x562b81fc0480 2024-03-06T07:23:46.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.375+0000 7efde17fa640 10 client.7119 handle_command_reply: tid=0 2024-03-06T07:23:46.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.375+0000 7efe0b5b0740 1 client.7119 shutdown 2024-03-06T07:23:46.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.375+0000 7efe0b5b0740 1 -- 172.21.15.1:0/3623967103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fc9ba0 msgr2=0x562b81fbfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.375+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/3623967103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fc9ba0 0x562b81fbfdc0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x562b81fe25d0 tx=0x7efdfc004ab0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.375+0000 7efe0b5b0740 1 -- 172.21.15.1:0/3623967103 shutdown_connections 2024-03-06T07:23:46.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.375+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/3623967103 >> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] conn(0x562b81fc0480 0x562b81fc5fe0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.375+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/3623967103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fc9ba0 0x562b81fbfdc0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.375+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/3623967103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fa3330 0x562b81fc71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.375+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/3623967103 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fa2f50 0x562b81fc37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.375+0000 7efde17fa640 0 client.7119 ms_handle_reset on v2:172.21.15.98:6835/3330683497 2024-03-06T07:23:46.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.375+0000 7efe0b5b0740 1 -- 172.21.15.1:0/3623967103 >> 172.21.15.1:0/3623967103 conn(0x562b81fa14b0 msgr2=0x562b81fa18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:46.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.375+0000 7efe0b5b0740 1 -- 172.21.15.1:0/3623967103 shutdown_connections 2024-03-06T07:23:46.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.375+0000 7efe0b5b0740 1 -- 172.21.15.1:0/3623967103 wait complete. 2024-03-06T07:23:46.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.375+0000 7efe0b5b0740 20 client.7119 trim_cache size 0 max 16384 2024-03-06T07:23:46.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.392+0000 7efe0b5b0740 1 Processor -- start 2024-03-06T07:23:46.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.392+0000 7efe0b5b0740 1 -- start start 2024-03-06T07:23:46.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.392+0000 7efe0b5b0740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa2f50 0x562b81fa1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.392+0000 7efe0b5b0740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fa3330 0x562b81fecc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.392+0000 7efe0b5b0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fc9ba0 0x562b81fa3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.392+0000 7efe0b5b0740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe041f4b40 con 0x562b81fa2f50 2024-03-06T07:23:46.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.392+0000 7efe0b5b0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe04146170 con 0x562b81fc9ba0 2024-03-06T07:23:46.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.392+0000 7efe0b5b0740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe041f4570 con 0x562b81fa3330 2024-03-06T07:23:46.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.392+0000 7efe0259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa2f50 0x562b81fa1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.392+0000 7efe0259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa2f50 0x562b81fa1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36954/0 (socket says 172.21.15.1:36954) 2024-03-06T07:23:46.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.393+0000 7efe0259c640 1 -- 172.21.15.1:0/3047821114 learned_addr learned my addr 172.21.15.1:0/3047821114 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:46.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.393+0000 7efe02d9d640 1 --2- 172.21.15.1:0/3047821114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fc9ba0 0x562b81fa3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.393+0000 7efdfbfff640 1 --2- 172.21.15.1:0/3047821114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fa3330 0x562b81fecc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.393+0000 7efe0259c640 1 -- 172.21.15.1:0/3047821114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fa3330 msgr2=0x562b81fecc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:46.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.393+0000 7efe0259c640 1 --2- 172.21.15.1:0/3047821114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fa3330 0x562b81fecc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.393+0000 7efe0259c640 1 -- 172.21.15.1:0/3047821114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fc9ba0 msgr2=0x562b81fa3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:46.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.393+0000 7efe0259c640 1 --2- 172.21.15.1:0/3047821114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fc9ba0 0x562b81fa3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.393+0000 7efe0259c640 1 -- 172.21.15.1:0/3047821114 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b81ce66b0 con 0x562b81fa2f50 2024-03-06T07:23:46.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.393+0000 7efe0259c640 1 --2- 172.21.15.1:0/3047821114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa2f50 0x562b81fa1890 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x562b81ff3b40 tx=0x7efdf4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a99ab524b16cc536 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.393+0000 7efde37fe640 1 -- 172.21.15.1:0/3047821114 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf402f070 con 0x562b81fa2f50 2024-03-06T07:23:46.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.393+0000 7efde37fe640 1 -- 172.21.15.1:0/3047821114 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf4001c10 con 0x562b81fa2f50 2024-03-06T07:23:46.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.393+0000 7efde37fe640 1 -- 172.21.15.1:0/3047821114 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf40223f0 con 0x562b81fa2f50 2024-03-06T07:23:46.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.394+0000 7efe0b5b0740 1 -- 172.21.15.1:0/3047821114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa2f50 msgr2=0x562b81fa1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.394+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/3047821114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa2f50 0x562b81fa1890 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x562b81ff3b40 tx=0x7efdf4013000 comp rx=0 tx=0).stop 2024-03-06T07:23:46.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.394+0000 7efe0b5b0740 1 -- 172.21.15.1:0/3047821114 shutdown_connections 2024-03-06T07:23:46.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.394+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/3047821114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fc9ba0 0x562b81fa3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.394+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/3047821114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fa3330 0x562b81fecc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.394+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/3047821114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa2f50 0x562b81fa1890 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.394+0000 7efe0b5b0740 1 -- 172.21.15.1:0/3047821114 >> 172.21.15.1:0/3047821114 conn(0x562b81fa14b0 msgr2=0x562b81fc2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:46.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.394+0000 7efe0b5b0740 1 -- 172.21.15.1:0/3047821114 shutdown_connections 2024-03-06T07:23:46.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.394+0000 7efe0b5b0740 1 -- 172.21.15.1:0/3047821114 wait complete. 2024-03-06T07:23:46.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.394+0000 7efe0b5b0740 1 -- start start 2024-03-06T07:23:46.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.394+0000 7efe0b5b0740 1 Processor -- start 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.395+0000 7efe0b5b0740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa2f50 0x562b82002c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.395+0000 7efe0b5b0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fa3330 0x562b82051f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.395+0000 7efe0b5b0740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fc9ba0 0x562b8204bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.395+0000 7efe0b5b0740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81fc0300 con 0x562b81fa2f50 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.395+0000 7efe0b5b0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81ff3930 con 0x562b81fa3330 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.395+0000 7efe0b5b0740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x562b81fc0900 con 0x562b81fc9ba0 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.395+0000 7efe0b5b0740 10 client.0 _ll_register_callbacks cb 0x7efdfc011230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.395+0000 7efe0259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa2f50 0x562b82002c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.395+0000 7efe0259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa2f50 0x562b82002c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36960/0 (socket says 172.21.15.1:36960) 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.395+0000 7efe0259c640 1 -- 172.21.15.1:0/1127722167 learned_addr learned my addr 172.21.15.1:0/1127722167 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.395+0000 7efdfbfff640 1 --2- 172.21.15.1:0/1127722167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fa3330 0x562b82051f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.395+0000 7efe02d9d640 1 --2- 172.21.15.1:0/1127722167 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fc9ba0 0x562b8204bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.395+0000 7efe0259c640 1 -- 172.21.15.1:0/1127722167 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fc9ba0 msgr2=0x562b8204bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.395+0000 7efe0259c640 1 --2- 172.21.15.1:0/1127722167 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fc9ba0 0x562b8204bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.395+0000 7efe0259c640 1 -- 172.21.15.1:0/1127722167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fa3330 msgr2=0x562b82051f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.395+0000 7efe0259c640 1 --2- 172.21.15.1:0/1127722167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fa3330 0x562b82051f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.395+0000 7efe0259c640 1 -- 172.21.15.1:0/1127722167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562b81fdb520 con 0x562b81fa2f50 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.396+0000 7efe0259c640 1 --2- 172.21.15.1:0/1127722167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa2f50 0x562b82002c20 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7efdf4007470 tx=0x7efdf4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1137f1699ab5184f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.396+0000 7efde17fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.396+0000 7efde17fa640 1 -- 172.21.15.1:0/1127722167 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf402f070 con 0x562b81fa2f50 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.396+0000 7efde17fa640 1 -- 172.21.15.1:0/1127722167 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efdf40226c0 con 0x562b81fa2f50 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.396+0000 7efe0b5b0740 1 -- 172.21.15.1:0/1127722167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efdf4001c10 con 0x562b81fa2f50 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.396+0000 7efde17fa640 1 -- 172.21.15.1:0/1127722167 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdf40228c0 con 0x562b81fa2f50 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.396+0000 7efe0b5b0740 10 client.7125 fetch_fsmap learned FSMap version 77 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.396+0000 7efe0b5b0740 1 -- 172.21.15.1:0/1127722167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x562b820032f0 con 0x562b81fa2f50 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.396+0000 7efde17fa640 1 -- 172.21.15.1:0/1127722167 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf4001c10 con 0x562b81fa2f50 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.396+0000 7efde17fa640 1 -- 172.21.15.1:0/1127722167 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 77) v1 ==== 3314+0+0 (secure 0 0 0) 0x7efdf4022a60 con 0x562b81fa2f50 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.396+0000 7efe0b5b0740 10 client.7125 fetch_fsmap finished waiting for FSMap version 77 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.396+0000 7efe0b5b0740 10 client.7125 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.396+0000 7efe0b5b0740 20 client.7125 populate_metadata read hostname 'smithi001' 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.396+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/1127722167 >> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] conn(0x562b81fe2600 0x562b81fe29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.396+0000 7efe0b5b0740 4 client.7125 mds_command: new command op to 6996 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.396+0000 7efe0b5b0740 1 -- 172.21.15.1:0/1127722167 --> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562b81f0ef40 con 0x562b81fe2600 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.396+0000 7efdfbfff640 1 --2- 172.21.15.1:0/1127722167 >> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] conn(0x562b81fe2600 0x562b81fe29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.397+0000 7efdfbfff640 1 --2- 172.21.15.1:0/1127722167 >> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] conn(0x562b81fe2600 0x562b81fe29e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4215b02ab36bf62b server_cookie=e03d5571b4777f75 in_seq=0 out_seq=0 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.397+0000 7efde17fa640 10 client.7125 ms_handle_connect on v2:172.21.15.98:6835/3330683497 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.397+0000 7efde17fa640 1 -- 172.21.15.1:0/1127722167 <== mds.0 v2:172.21.15.98:6835/3330683497 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+315 (crc 0 0 0) 0x562b81f0ef40 con 0x562b81fe2600 2024-03-06T07:23:46.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.397+0000 7efde17fa640 10 client.7125 handle_command_reply: tid=0 2024-03-06T07:23:46.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.397+0000 7efe0b5b0740 1 client.7125 shutdown 2024-03-06T07:23:46.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.397+0000 7efe0b5b0740 1 -- 172.21.15.1:0/1127722167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa2f50 msgr2=0x562b82002c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.397+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/1127722167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa2f50 0x562b82002c20 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7efdf4007470 tx=0x7efdf4012000 comp rx=0 tx=0).stop 2024-03-06T07:23:46.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.397+0000 7efe0b5b0740 1 -- 172.21.15.1:0/1127722167 shutdown_connections 2024-03-06T07:23:46.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.397+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/1127722167 >> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] conn(0x562b81fe2600 0x562b81fe29e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.397+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/1127722167 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x562b81fc9ba0 0x562b8204bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.397+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/1127722167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b81fa3330 0x562b82051f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.397+0000 7efe0b5b0740 1 --2- 172.21.15.1:0/1127722167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562b81fa2f50 0x562b82002c20 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.397+0000 7efde17fa640 0 client.7125 ms_handle_reset on v2:172.21.15.98:6835/3330683497 2024-03-06T07:23:46.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.397+0000 7efe0b5b0740 1 -- 172.21.15.1:0/1127722167 >> 172.21.15.1:0/1127722167 conn(0x562b81fa14b0 msgr2=0x562b81fa1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:46.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.397+0000 7efe0b5b0740 1 -- 172.21.15.1:0/1127722167 shutdown_connections 2024-03-06T07:23:46.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.397+0000 7efe0b5b0740 1 -- 172.21.15.1:0/1127722167 wait complete. 2024-03-06T07:23:46.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.397+0000 7efe0b5b0740 20 client.7125 trim_cache size 0 max 16384 2024-03-06T07:23:46.407 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:23:46.407 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:23:46.407 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:23:46.407 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 6996, 2024-03-06T07:23:46.407 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:23:46.407 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:23:46.407 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:23:46.407 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 4.676504306, 2024-03-06T07:23:46.407 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 77, 2024-03-06T07:23:46.407 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 95, 2024-03-06T07:23:46.407 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 95, 2024-03-06T07:23:46.407 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 9.7720319 2024-03-06T07:23:46.407 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:23:46.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.398+0000 7efe089d0640 1 -- 172.21.15.1:0/397542727 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efde406b500 msgr2=0x7efde406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.398+0000 7efe089d0640 1 --2- 172.21.15.1:0/397542727 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efde406b500 0x7efde406d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7efdf4000f60 tx=0x7efdf4007d50 comp rx=0 tx=0).stop 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.398+0000 7efe089d0640 1 -- 172.21.15.1:0/397542727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe04072f20 msgr2=0x7efe0415d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.398+0000 7efe089d0640 1 --2- 172.21.15.1:0/397542727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe04072f20 0x7efe0415d610 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7efdec034040 tx=0x7efdec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.398+0000 7efe089d0640 1 -- 172.21.15.1:0/397542727 shutdown_connections 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.398+0000 7efe089d0640 1 --2- 172.21.15.1:0/397542727 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efde406b500 0x7efde406d9c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.398+0000 7efe089d0640 1 --2- 172.21.15.1:0/397542727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe04150fe0 0x7efe040bc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.398+0000 7efe089d0640 1 --2- 172.21.15.1:0/397542727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe04072f20 0x7efe0415d610 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.398+0000 7efe089d0640 1 --2- 172.21.15.1:0/397542727 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efe04071f90 0x7efe041468c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.398+0000 7efe089d0640 1 -- 172.21.15.1:0/397542727 >> 172.21.15.1:0/397542727 conn(0x7efe040c6860 msgr2=0x7efe0406edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.398+0000 7efe089d0640 1 -- 172.21.15.1:0/397542727 shutdown_connections 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.398+0000 7efe089d0640 1 -- 172.21.15.1:0/397542727 wait complete. 2024-03-06T07:23:46.410 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:46.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.509+0000 7f0d266b5640 1 Processor -- start 2024-03-06T07:23:46.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.509+0000 7f0d266b5640 1 -- start start 2024-03-06T07:23:46.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.510+0000 7f0d266b5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d20152650 0x7f0d20155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.510+0000 7f0d266b5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d200c2fc0 con 0x7f0d200bf9d0 2024-03-06T07:23:46.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.510+0000 7f0d266b5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d20158da0 con 0x7f0d200c6b10 2024-03-06T07:23:46.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.510+0000 7f0d266b5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d20155800 con 0x7f0d20152650 2024-03-06T07:23:46.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.510+0000 7f0d24c52640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0d200bf9d0 0x7f0d200c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36058/0 (socket says 172.21.15.1:36058) 2024-03-06T07:23:46.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.510+0000 7f0d24c52640 1 -- 172.21.15.1:0/526590810 learned_addr learned my addr 172.21.15.1:0/526590810 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:46.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.510+0000 7f0d1ffff640 1 --2- 172.21.15.1:0/526590810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d20152650 0x7f0d20155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.510+0000 7f0d1effd640 1 -- 172.21.15.1:0/526590810 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586988021 0 0) 0x7f0d200c2fc0 con 0x7f0d200bf9d0 2024-03-06T07:23:46.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.511+0000 7f0d1effd640 1 -- 172.21.15.1:0/526590810 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d08003520 con 0x7f0d200bf9d0 2024-03-06T07:23:46.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.511+0000 7f0d1effd640 1 -- 172.21.15.1:0/526590810 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884602630 0 0) 0x7f0d20158da0 con 0x7f0d200c6b10 2024-03-06T07:23:46.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.511+0000 7f0d1effd640 1 -- 172.21.15.1:0/526590810 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d200c2fc0 con 0x7f0d200c6b10 2024-03-06T07:23:46.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.511+0000 7f0d1ffff640 1 -- 172.21.15.1:0/526590810 >> v1:172.21.15.98:6789/0 conn(0x7f0d200c6b10 legacy=0x7f0d20158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.511+0000 7f0d1ffff640 1 -- 172.21.15.1:0/526590810 >> v1:172.21.15.1:6789/0 conn(0x7f0d200bf9d0 legacy=0x7f0d200c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.511+0000 7f0d1ffff640 1 -- 172.21.15.1:0/526590810 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d200c6690 con 0x7f0d20152650 2024-03-06T07:23:46.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.511+0000 7f0d1ffff640 1 --2- 172.21.15.1:0/526590810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d20152650 0x7f0d20155c90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0d100097d0 tx=0x7f0d1002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ac8b7bb940bc47a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.511+0000 7f0d1effd640 1 -- 172.21.15.1:0/526590810 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10035020 con 0x7f0d20152650 2024-03-06T07:23:46.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.512+0000 7f0d1effd640 1 -- 172.21.15.1:0/526590810 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d10031800 con 0x7f0d20152650 2024-03-06T07:23:46.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.512+0000 7f0d1effd640 1 -- 172.21.15.1:0/526590810 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10031bd0 con 0x7f0d20152650 2024-03-06T07:23:46.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.512+0000 7f0d266b5640 1 -- 172.21.15.1:0/526590810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d20152650 msgr2=0x7f0d20155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.512+0000 7f0d266b5640 1 --2- 172.21.15.1:0/526590810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d20152650 0x7f0d20155c90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0d100097d0 tx=0x7f0d1002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.512+0000 7f0d266b5640 1 -- 172.21.15.1:0/526590810 shutdown_connections 2024-03-06T07:23:46.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.512+0000 7f0d266b5640 1 --2- 172.21.15.1:0/526590810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d20152650 0x7f0d20155c90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.512+0000 7f0d266b5640 1 -- 172.21.15.1:0/526590810 >> 172.21.15.1:0/526590810 conn(0x7f0d20066ee0 msgr2=0x7f0d200672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:46.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.512+0000 7f0d266b5640 1 -- 172.21.15.1:0/526590810 shutdown_connections 2024-03-06T07:23:46.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.512+0000 7f0d266b5640 1 -- 172.21.15.1:0/526590810 wait complete. 2024-03-06T07:23:46.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.513+0000 7f0d266b5640 1 Processor -- start 2024-03-06T07:23:46.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.513+0000 7f0d266b5640 1 -- start start 2024-03-06T07:23:46.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.514+0000 7f0d266b5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d200bf9d0 0x7f0d20150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.514+0000 7f0d266b5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d20152650 0x7f0d201df810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.514+0000 7f0d266b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d201e0e10 0x7f0d201dfd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.514+0000 7f0d266b5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d201551b0 con 0x7f0d200bf9d0 2024-03-06T07:23:46.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.514+0000 7f0d266b5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d200c6fa0 con 0x7f0d201e0e10 2024-03-06T07:23:46.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.514+0000 7f0d266b5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d20155490 con 0x7f0d20152650 2024-03-06T07:23:46.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.514+0000 7f0d1ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d200bf9d0 0x7f0d20150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.514+0000 7f0d1ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d200bf9d0 0x7f0d20150480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36974/0 (socket says 172.21.15.1:36974) 2024-03-06T07:23:46.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.514+0000 7f0d1ffff640 1 -- 172.21.15.1:0/1786973418 learned_addr learned my addr 172.21.15.1:0/1786973418 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:46.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.514+0000 7f0d1f7fe640 1 --2- 172.21.15.1:0/1786973418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d20152650 0x7f0d201df810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.514+0000 7f0d24c52640 1 --2- 172.21.15.1:0/1786973418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d201e0e10 0x7f0d201dfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.514+0000 7f0d1ffff640 1 -- 172.21.15.1:0/1786973418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d20152650 msgr2=0x7f0d201df810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.514+0000 7f0d1ffff640 1 --2- 172.21.15.1:0/1786973418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d20152650 0x7f0d201df810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.514+0000 7f0d1ffff640 1 -- 172.21.15.1:0/1786973418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d201e0e10 msgr2=0x7f0d201dfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.514+0000 7f0d1ffff640 1 --2- 172.21.15.1:0/1786973418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d201e0e10 0x7f0d201dfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.514+0000 7f0d1ffff640 1 -- 172.21.15.1:0/1786973418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d201e1430 con 0x7f0d200bf9d0 2024-03-06T07:23:46.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.515+0000 7f0d1ffff640 1 --2- 172.21.15.1:0/1786973418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d200bf9d0 0x7f0d20150480 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f0d100098f0 tx=0x7f0d100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.515+0000 7f0d1cff9640 1 -- 172.21.15.1:0/1786973418 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10041070 con 0x7f0d200bf9d0 2024-03-06T07:23:46.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.515+0000 7f0d1cff9640 1 -- 172.21.15.1:0/1786973418 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d10031e30 con 0x7f0d200bf9d0 2024-03-06T07:23:46.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.515+0000 7f0d1cff9640 1 -- 172.21.15.1:0/1786973418 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10035040 con 0x7f0d200bf9d0 2024-03-06T07:23:46.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.515+0000 7f0d266b5640 1 -- 172.21.15.1:0/1786973418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d20154730 con 0x7f0d200bf9d0 2024-03-06T07:23:46.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.515+0000 7f0d266b5640 1 -- 172.21.15.1:0/1786973418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d201ebb30 con 0x7f0d200bf9d0 2024-03-06T07:23:46.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.516+0000 7f0d266b5640 1 -- 172.21.15.1:0/1786973418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d20067fd0 con 0x7f0d200bf9d0 2024-03-06T07:23:46.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.516+0000 7f0d1cff9640 1 -- 172.21.15.1:0/1786973418 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d1004c060 con 0x7f0d200bf9d0 2024-03-06T07:23:46.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.516+0000 7f0d1cff9640 1 --2- 172.21.15.1:0/1786973418 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0d0c06b4d0 0x7f0d0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.517+0000 7f0d1cff9640 1 -- 172.21.15.1:0/1786973418 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0d100abdf0 con 0x7f0d200bf9d0 2024-03-06T07:23:46.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.517+0000 7f0d1f7fe640 1 --2- 172.21.15.1:0/1786973418 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0d0c06b4d0 0x7f0d0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.517+0000 7f0d1f7fe640 1 --2- 172.21.15.1:0/1786973418 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0d0c06b4d0 0x7f0d0c06d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0d14007fd0 tx=0x7f0d14009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.521+0000 7f0d1cff9640 1 -- 172.21.15.1:0/1786973418 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d1007a370 con 0x7f0d200bf9d0 2024-03-06T07:23:46.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.739+0000 7f0d266b5640 1 -- 172.21.15.1:0/1786973418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d200b06f0 con 0x7f0d200bf9d0 2024-03-06T07:23:46.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.740+0000 7f0d1cff9640 1 -- 172.21.15.1:0/1786973418 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4970 (secure 0 0 0) 0x7f0d1007e410 con 0x7f0d200bf9d0 2024-03-06T07:23:46.742 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:46.742 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":77,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1335402413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1335402413},{"type":"v1","addr":"172.21.15.98:6836","nonce":1335402413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6981,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/2118550222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2118550222},{"type":"v1","addr":"172.21.15.1:6834","nonce":2118550222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/543699854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":543699854},{"type":"v1","addr":"172.21.15.1:6835","nonce":543699854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:41.077040+0000","modified":"2024-03-06T07:23:43.077835+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6996},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6996":{"gid":6996,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/3330683497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3330683497},{"type":"v1","addr":"172.21.15.98:6837","nonce":3330683497}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6996,"qdb_cluster":[6996]},"id":7}]} 2024-03-06T07:23:46.742 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 77 2024-03-06T07:23:46.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.743+0000 7f0d266b5640 1 -- 172.21.15.1:0/1786973418 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0d0c06b4d0 msgr2=0x7f0d0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.743+0000 7f0d266b5640 1 --2- 172.21.15.1:0/1786973418 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0d0c06b4d0 0x7f0d0c06d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0d14007fd0 tx=0x7f0d14009040 comp rx=0 tx=0).stop 2024-03-06T07:23:46.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.743+0000 7f0d266b5640 1 -- 172.21.15.1:0/1786973418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d200bf9d0 msgr2=0x7f0d20150480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.743+0000 7f0d266b5640 1 --2- 172.21.15.1:0/1786973418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d200bf9d0 0x7f0d20150480 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f0d100098f0 tx=0x7f0d100046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.743+0000 7f0d266b5640 1 -- 172.21.15.1:0/1786973418 shutdown_connections 2024-03-06T07:23:46.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.743+0000 7f0d266b5640 1 --2- 172.21.15.1:0/1786973418 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0d0c06b4d0 0x7f0d0c06d990 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.743+0000 7f0d266b5640 1 --2- 172.21.15.1:0/1786973418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d201e0e10 0x7f0d201dfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.743+0000 7f0d266b5640 1 --2- 172.21.15.1:0/1786973418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d20152650 0x7f0d201df810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.743+0000 7f0d266b5640 1 --2- 172.21.15.1:0/1786973418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d200bf9d0 0x7f0d20150480 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.743+0000 7f0d266b5640 1 -- 172.21.15.1:0/1786973418 >> 172.21.15.1:0/1786973418 conn(0x7f0d20066ee0 msgr2=0x7f0d200bff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:46.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.743+0000 7f0d266b5640 1 -- 172.21.15.1:0/1786973418 shutdown_connections 2024-03-06T07:23:46.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.743+0000 7f0d266b5640 1 -- 172.21.15.1:0/1786973418 wait complete. 2024-03-06T07:23:46.757 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 77, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 7, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:23:41.077040+0000", "damaged": [], "data_pools": [ 16 ], "enabled": true, "epoch": 77, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6996": { "addr": "172.21.15.98:6837/3330683497", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 3330683497, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 3330683497, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6996, "incarnation": 75, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 15, "modified": "2024-03-06T07:23:43.077835+0000", "qdb_cluster": [ 6996 ], "qdb_leader": 6996, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6996 } } } ], "standbys": [ { "addr": "172.21.15.98:6836/1335402413", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 1335402413, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 1335402413, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 74, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5855, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6834/2118550222", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 2118550222, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 2118550222, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 74, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6981, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6835/543699854", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 543699854, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 543699854, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 74, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6984, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:23:46.757 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:23:46.757 INFO:tasks.cephfs.test_failover:current = [{'gid': 6984, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/543699854', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 543699854}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 543699854}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 74}, {'gid': 5855, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/1335402413', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 1335402413}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1335402413}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 74}, {'gid': 6981, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6834/2118550222', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2118550222}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2118550222}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 74}, {'gid': 6996, 'name': 'd', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.98:6837/3330683497', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3330683497}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3330683497}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-03-06T07:23:46.758 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs 2024-03-06T07:23:46.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.860+0000 7fa800233640 1 Processor -- start 2024-03-06T07:23:46.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.860+0000 7fa800233640 1 -- start start 2024-03-06T07:23:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.861+0000 7fa800233640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa7f8152920 0x7fa7f806aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.861+0000 7fa800233640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa7f8155220 0x7fa7f81556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.861+0000 7fa800233640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f8158910 con 0x7fa7f806b530 2024-03-06T07:23:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.861+0000 7fa800233640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7f8152d00 con 0x7fa7f8155220 2024-03-06T07:23:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.861+0000 7fa800233640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7f81596d0 con 0x7fa7f8152920 2024-03-06T07:23:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.861+0000 7fa7fe7d0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa7f8155220 0x7fa7f81556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.861+0000 7fa7fe7d0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa7f8155220 0x7fa7f81556b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36986/0 (socket says 172.21.15.1:36986) 2024-03-06T07:23:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.861+0000 7fa7fdfcf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa7f8152920 0x7fa7f806aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.861+0000 7fa7fe7d0640 1 -- 172.21.15.1:0/3441530564 learned_addr learned my addr 172.21.15.1:0/3441530564 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.861+0000 7fa7fe7d0640 1 -- 172.21.15.1:0/3441530564 >> v2:172.21.15.98:3300/0 conn(0x7fa7f8152920 msgr2=0x7fa7f806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.861+0000 7fa7fe7d0640 1 --2- 172.21.15.1:0/3441530564 >> v2:172.21.15.98:3300/0 conn(0x7fa7f8152920 0x7fa7f806aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.861+0000 7fa7fe7d0640 1 -- 172.21.15.1:0/3441530564 >> v1:172.21.15.98:6789/0 conn(0x7fa7f806b530 legacy=0x7fa7f80c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.861+0000 7fa7fe7d0640 1 -- 172.21.15.1:0/3441530564 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f8158bf0 con 0x7fa7f8155220 2024-03-06T07:23:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.862+0000 7fa7fe7d0640 1 --2- 172.21.15.1:0/3441530564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7f8155220 0x7fa7f81556b0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fa7f806bd50 tx=0x7fa7f002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20666b78469a1d38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.862+0000 7fa7fcfcd640 1 -- 172.21.15.1:0/3441530564 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f002ca80 con 0x7fa7f8155220 2024-03-06T07:23:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.862+0000 7fa7fcfcd640 1 -- 172.21.15.1:0/3441530564 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f002cc20 con 0x7fa7f8155220 2024-03-06T07:23:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.862+0000 7fa7fcfcd640 1 -- 172.21.15.1:0/3441530564 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f00343f0 con 0x7fa7f8155220 2024-03-06T07:23:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.862+0000 7fa800233640 1 -- 172.21.15.1:0/3441530564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7f8155220 msgr2=0x7fa7f81556b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.862+0000 7fa800233640 1 --2- 172.21.15.1:0/3441530564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7f8155220 0x7fa7f81556b0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fa7f806bd50 tx=0x7fa7f002bee0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.862+0000 7fa800233640 1 -- 172.21.15.1:0/3441530564 shutdown_connections 2024-03-06T07:23:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.862+0000 7fa800233640 1 --2- 172.21.15.1:0/3441530564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7f8155220 0x7fa7f81556b0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.862+0000 7fa800233640 1 --2- 172.21.15.1:0/3441530564 >> v2:172.21.15.98:3300/0 conn(0x7fa7f8152920 0x7fa7f806aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.862+0000 7fa800233640 1 -- 172.21.15.1:0/3441530564 >> 172.21.15.1:0/3441530564 conn(0x7fa7f80a9700 msgr2=0x7fa7f80a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.863+0000 7fa800233640 1 -- 172.21.15.1:0/3441530564 shutdown_connections 2024-03-06T07:23:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.863+0000 7fa800233640 1 -- 172.21.15.1:0/3441530564 wait complete. 2024-03-06T07:23:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.864+0000 7fa800233640 1 Processor -- start 2024-03-06T07:23:46.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.864+0000 7fa800233640 1 -- start start 2024-03-06T07:23:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.864+0000 7fa800233640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7f806b530 0x7fa7f81eddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.864+0000 7fa800233640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7f8152920 0x7fa7f8140f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.864+0000 7fa800233640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7f8155220 0x7fa7f8156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.864+0000 7fa800233640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f80c6380 con 0x7fa7f8155220 2024-03-06T07:23:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.864+0000 7fa800233640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f8159240 con 0x7fa7f8152920 2024-03-06T07:23:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.864+0000 7fa800233640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f80c5f60 con 0x7fa7f806b530 2024-03-06T07:23:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.864+0000 7fa7fe7d0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7f8155220 0x7fa7f8156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.864+0000 7fa7fe7d0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7f8155220 0x7fa7f8156a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36996/0 (socket says 172.21.15.1:36996) 2024-03-06T07:23:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.864+0000 7fa7fe7d0640 1 -- 172.21.15.1:0/3506525413 learned_addr learned my addr 172.21.15.1:0/3506525413 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.865+0000 7fa7fdfcf640 1 --2- 172.21.15.1:0/3506525413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7f806b530 0x7fa7f81eddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.865+0000 7fa7fd7ce640 1 --2- 172.21.15.1:0/3506525413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7f8152920 0x7fa7f8140f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.865+0000 7fa7fe7d0640 1 -- 172.21.15.1:0/3506525413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7f806b530 msgr2=0x7fa7f81eddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.865+0000 7fa7fe7d0640 1 --2- 172.21.15.1:0/3506525413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7f806b530 0x7fa7f81eddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.865+0000 7fa7fe7d0640 1 -- 172.21.15.1:0/3506525413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7f8152920 msgr2=0x7fa7f8140f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.865+0000 7fa7fe7d0640 1 --2- 172.21.15.1:0/3506525413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7f8152920 0x7fa7f8140f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.865+0000 7fa7fe7d0640 1 -- 172.21.15.1:0/3506525413 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f8141470 con 0x7fa7f8155220 2024-03-06T07:23:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.865+0000 7fa7fd7ce640 1 --2- 172.21.15.1:0/3506525413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7f8152920 0x7fa7f8140f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.865+0000 7fa7fe7d0640 1 --2- 172.21.15.1:0/3506525413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7f8155220 0x7fa7f8156a00 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0009f40 tx=0x7fa7f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.865+0000 7fa7ee7fc640 1 -- 172.21.15.1:0/3506525413 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f0043020 con 0x7fa7f8155220 2024-03-06T07:23:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.865+0000 7fa7ee7fc640 1 -- 172.21.15.1:0/3506525413 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f003e450 con 0x7fa7f8155220 2024-03-06T07:23:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.865+0000 7fa7ee7fc640 1 -- 172.21.15.1:0/3506525413 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f0009510 con 0x7fa7f8155220 2024-03-06T07:23:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.865+0000 7fa800233640 1 -- 172.21.15.1:0/3506525413 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f8148550 con 0x7fa7f8155220 2024-03-06T07:23:46.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.865+0000 7fa800233640 1 -- 172.21.15.1:0/3506525413 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f8151ad0 con 0x7fa7f8155220 2024-03-06T07:23:46.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.866+0000 7fa800233640 1 -- 172.21.15.1:0/3506525413 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f8067fd0 con 0x7fa7f8155220 2024-03-06T07:23:46.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.867+0000 7fa7ee7fc640 1 -- 172.21.15.1:0/3506525413 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7f0046020 con 0x7fa7f8155220 2024-03-06T07:23:46.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.867+0000 7fa7ee7fc640 1 --2- 172.21.15.1:0/3506525413 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa7dc06b4d0 0x7fa7dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.867+0000 7fa7ee7fc640 1 -- 172.21.15.1:0/3506525413 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa7f00ac780 con 0x7fa7f8155220 2024-03-06T07:23:46.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.867+0000 7fa7fdfcf640 1 --2- 172.21.15.1:0/3506525413 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa7dc06b4d0 0x7fa7dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.868+0000 7fa7fdfcf640 1 --2- 172.21.15.1:0/3506525413 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa7dc06b4d0 0x7fa7dc06d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4009b40 tx=0x7fa7f4009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:46.871+0000 7fa7ee7fc640 1 -- 172.21.15.1:0/3506525413 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7f004d070 con 0x7fa7f8155220 2024-03-06T07:23:46.990 INFO:tasks.ceph.mds.d.smithi098.stderr:2024-03-06T07:23:46.989+0000 7fef69aca640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:47.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.051+0000 7fa800233640 1 -- 172.21.15.1:0/3506525413 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fa7f806bdf0 con 0x7fa7f8155220 2024-03-06T07:23:47.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.054+0000 7fa7ee7fc640 1 -- 172.21.15.1:0/3506525413 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v32)=0 v32) v1 ==== 116+0+0 (secure 0 0 0) 0x7fa7f8141470 con 0x7fa7f8155220 2024-03-06T07:23:47.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.056+0000 7fa800233640 1 -- 172.21.15.1:0/3506525413 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa7dc06b4d0 msgr2=0x7fa7dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.056+0000 7fa800233640 1 --2- 172.21.15.1:0/3506525413 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa7dc06b4d0 0x7fa7dc06d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4009b40 tx=0x7fa7f4009190 comp rx=0 tx=0).stop 2024-03-06T07:23:47.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.056+0000 7fa800233640 1 -- 172.21.15.1:0/3506525413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7f8155220 msgr2=0x7fa7f8156a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.056+0000 7fa800233640 1 --2- 172.21.15.1:0/3506525413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7f8155220 0x7fa7f8156a00 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0009f40 tx=0x7fa7f0004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.056+0000 7fa800233640 1 -- 172.21.15.1:0/3506525413 shutdown_connections 2024-03-06T07:23:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.056+0000 7fa800233640 1 --2- 172.21.15.1:0/3506525413 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa7dc06b4d0 0x7fa7dc06d990 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.056+0000 7fa800233640 1 --2- 172.21.15.1:0/3506525413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7f8155220 0x7fa7f8156a00 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.056+0000 7fa800233640 1 --2- 172.21.15.1:0/3506525413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7f8152920 0x7fa7f8140f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.056+0000 7fa800233640 1 --2- 172.21.15.1:0/3506525413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7f806b530 0x7fa7f81eddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.056+0000 7fa800233640 1 -- 172.21.15.1:0/3506525413 >> 172.21.15.1:0/3506525413 conn(0x7fa7f80a9700 msgr2=0x7fa7f80c6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.056+0000 7fa800233640 1 -- 172.21.15.1:0/3506525413 shutdown_connections 2024-03-06T07:23:47.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.056+0000 7fa800233640 1 -- 172.21.15.1:0/3506525413 wait complete. 2024-03-06T07:23:47.069 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.170+0000 7f0fde5f7640 1 Processor -- start 2024-03-06T07:23:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.170+0000 7f0fde5f7640 1 -- start start 2024-03-06T07:23:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.170+0000 7f0fde5f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fd8152690 0x7f0fd8155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.170+0000 7f0fde5f7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0fd80c6730 0x7f0fd80c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.170+0000 7f0fde5f7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0fd80c5d10 0x7f0fd8156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.170+0000 7f0fde5f7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd81561e0 con 0x7f0fd80c6730 2024-03-06T07:23:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.170+0000 7f0fde5f7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd80c3300 con 0x7f0fd80c5d10 2024-03-06T07:23:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.170+0000 7f0fde5f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80c35a0 con 0x7f0fd8152690 2024-03-06T07:23:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.171+0000 7f0fd77fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0fd80c6730 0x7f0fd80c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.171+0000 7f0fd77fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0fd80c6730 0x7f0fd80c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36998/0 (socket says 172.21.15.1:36998) 2024-03-06T07:23:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.171+0000 7f0fd77fe640 1 -- 172.21.15.1:0/3204721393 learned_addr learned my addr 172.21.15.1:0/3204721393 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.171+0000 7f0fd7fff640 1 --2- 172.21.15.1:0/3204721393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fd8152690 0x7f0fd8155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.171+0000 7f0fdcb94640 1 --2- 172.21.15.1:0/3204721393 >> v2:172.21.15.98:3300/0 conn(0x7f0fd80c5d10 0x7f0fd8156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.171+0000 7f0fd77fe640 1 -- 172.21.15.1:0/3204721393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fd8152690 msgr2=0x7f0fd8155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.171+0000 7f0fd77fe640 1 --2- 172.21.15.1:0/3204721393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fd8152690 0x7f0fd8155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.171+0000 7f0fd77fe640 1 -- 172.21.15.1:0/3204721393 >> v2:172.21.15.98:3300/0 conn(0x7f0fd80c5d10 msgr2=0x7f0fd8156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.171+0000 7f0fd77fe640 1 --2- 172.21.15.1:0/3204721393 >> v2:172.21.15.98:3300/0 conn(0x7f0fd80c5d10 0x7f0fd8156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.171+0000 7f0fd77fe640 1 -- 172.21.15.1:0/3204721393 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd80c3880 con 0x7f0fd80c6730 2024-03-06T07:23:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.172+0000 7f0fd77fe640 1 --2- 172.21.15.1:0/3204721393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c6730 0x7f0fd80c57d0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80c2240 tx=0x7f0fcc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9203fd3e3fc090e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.172+0000 7f0fd6ffd640 1 -- 172.21.15.1:0/3204721393 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc02e070 con 0x7f0fd80c6730 2024-03-06T07:23:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.172+0000 7f0fd6ffd640 1 -- 172.21.15.1:0/3204721393 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fcc037070 con 0x7f0fd80c6730 2024-03-06T07:23:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.172+0000 7f0fd6ffd640 1 -- 172.21.15.1:0/3204721393 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc031a00 con 0x7f0fd80c6730 2024-03-06T07:23:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.172+0000 7f0fde5f7640 1 -- 172.21.15.1:0/3204721393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c6730 msgr2=0x7f0fd80c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.172+0000 7f0fde5f7640 1 --2- 172.21.15.1:0/3204721393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c6730 0x7f0fd80c57d0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80c2240 tx=0x7f0fcc02b880 comp rx=0 tx=0).stop 2024-03-06T07:23:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.172+0000 7f0fde5f7640 1 -- 172.21.15.1:0/3204721393 shutdown_connections 2024-03-06T07:23:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.172+0000 7f0fde5f7640 1 --2- 172.21.15.1:0/3204721393 >> v2:172.21.15.98:3300/0 conn(0x7f0fd80c5d10 0x7f0fd8156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.172+0000 7f0fde5f7640 1 --2- 172.21.15.1:0/3204721393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c6730 0x7f0fd80c57d0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.172+0000 7f0fde5f7640 1 --2- 172.21.15.1:0/3204721393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fd8152690 0x7f0fd8155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.172+0000 7f0fde5f7640 1 -- 172.21.15.1:0/3204721393 >> 172.21.15.1:0/3204721393 conn(0x7f0fd806b410 msgr2=0x7f0fd806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.172+0000 7f0fde5f7640 1 -- 172.21.15.1:0/3204721393 shutdown_connections 2024-03-06T07:23:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.172+0000 7f0fde5f7640 1 -- 172.21.15.1:0/3204721393 wait complete. 2024-03-06T07:23:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.173+0000 7f0fde5f7640 1 Processor -- start 2024-03-06T07:23:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.174+0000 7f0fde5f7640 1 -- start start 2024-03-06T07:23:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.174+0000 7f0fde5f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fd80c5d10 0x7f0fd80ad150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.174+0000 7f0fde5f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c6730 0x7f0fd814a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.174+0000 7f0fde5f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fd8152690 0x7f0fd814f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.174+0000 7f0fde5f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd8152a70 con 0x7f0fd80c6730 2024-03-06T07:23:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.174+0000 7f0fde5f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd81567b0 con 0x7f0fd80c5d10 2024-03-06T07:23:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.174+0000 7f0fde5f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd8152bf0 con 0x7f0fd8152690 2024-03-06T07:23:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.174+0000 7f0fd77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c6730 0x7f0fd814a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.174+0000 7f0fd77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c6730 0x7f0fd814a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37004/0 (socket says 172.21.15.1:37004) 2024-03-06T07:23:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.174+0000 7f0fd77fe640 1 -- 172.21.15.1:0/3512889906 learned_addr learned my addr 172.21.15.1:0/3512889906 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.175+0000 7f0fdcb94640 1 --2- 172.21.15.1:0/3512889906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fd8152690 0x7f0fd814f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.175+0000 7f0fd7fff640 1 --2- 172.21.15.1:0/3512889906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fd80c5d10 0x7f0fd80ad150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.175+0000 7f0fd77fe640 1 -- 172.21.15.1:0/3512889906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fd8152690 msgr2=0x7f0fd814f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.175+0000 7f0fd77fe640 1 --2- 172.21.15.1:0/3512889906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fd8152690 0x7f0fd814f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.175+0000 7f0fd77fe640 1 -- 172.21.15.1:0/3512889906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fd80c5d10 msgr2=0x7f0fd80ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.175+0000 7f0fd77fe640 1 --2- 172.21.15.1:0/3512889906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fd80c5d10 0x7f0fd80ad150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.175+0000 7f0fd77fe640 1 -- 172.21.15.1:0/3512889906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd8154870 con 0x7f0fd80c6730 2024-03-06T07:23:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.175+0000 7f0fd77fe640 1 --2- 172.21.15.1:0/3512889906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c6730 0x7f0fd814a610 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc02b8c0 tx=0x7f0fcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.175+0000 7f0fd4ff9640 1 -- 172.21.15.1:0/3512889906 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc032670 con 0x7f0fd80c6730 2024-03-06T07:23:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.175+0000 7f0fd4ff9640 1 -- 172.21.15.1:0/3512889906 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fcc032810 con 0x7f0fd80c6730 2024-03-06T07:23:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.175+0000 7f0fd4ff9640 1 -- 172.21.15.1:0/3512889906 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc032c00 con 0x7f0fd80c6730 2024-03-06T07:23:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.175+0000 7f0fde5f7640 1 -- 172.21.15.1:0/3512889906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd8154b50 con 0x7f0fd80c6730 2024-03-06T07:23:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.175+0000 7f0fde5f7640 1 -- 172.21.15.1:0/3512889906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd8155110 con 0x7f0fd80c6730 2024-03-06T07:23:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.176+0000 7f0fde5f7640 1 -- 172.21.15.1:0/3512889906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd8067fd0 con 0x7f0fd80c6730 2024-03-06T07:23:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.176+0000 7f0fd4ff9640 1 -- 172.21.15.1:0/3512889906 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0fcc0453f0 con 0x7f0fd80c6730 2024-03-06T07:23:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.177+0000 7f0fd4ff9640 1 --2- 172.21.15.1:0/3512889906 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fc406b500 0x7f0fc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.177+0000 7f0fd4ff9640 1 -- 172.21.15.1:0/3512889906 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0fcc0abf90 con 0x7f0fd80c6730 2024-03-06T07:23:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.177+0000 7f0fd7fff640 1 --2- 172.21.15.1:0/3512889906 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fc406b500 0x7f0fc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.177+0000 7f0fd7fff640 1 --2- 172.21.15.1:0/3512889906 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fc406b500 0x7f0fc406d9c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009aa0 tx=0x7f0fc8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.181+0000 7f0fd4ff9640 1 -- 172.21.15.1:0/3512889906 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0fcc040070 con 0x7f0fd80c6730 2024-03-06T07:23:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.400+0000 7f0fde5f7640 1 -- 172.21.15.1:0/3512889906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fd81556c0 con 0x7f0fd80c6730 2024-03-06T07:23:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.401+0000 7f0fd4ff9640 1 -- 172.21.15.1:0/3512889906 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4970 (secure 0 0 0) 0x7f0fcc07a670 con 0x7f0fd80c6730 2024-03-06T07:23:47.403 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:47.403 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":77,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1335402413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1335402413},{"type":"v1","addr":"172.21.15.98:6836","nonce":1335402413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6981,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/2118550222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2118550222},{"type":"v1","addr":"172.21.15.1:6834","nonce":2118550222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/543699854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":543699854},{"type":"v1","addr":"172.21.15.1:6835","nonce":543699854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:41.077040+0000","modified":"2024-03-06T07:23:43.077835+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6996},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6996":{"gid":6996,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/3330683497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3330683497},{"type":"v1","addr":"172.21.15.98:6837","nonce":3330683497}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6996,"qdb_cluster":[6996]},"id":7}]} 2024-03-06T07:23:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 77 2024-03-06T07:23:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.403+0000 7f0fde5f7640 1 -- 172.21.15.1:0/3512889906 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fc406b500 msgr2=0x7f0fc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.403+0000 7f0fde5f7640 1 --2- 172.21.15.1:0/3512889906 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fc406b500 0x7f0fc406d9c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009aa0 tx=0x7f0fc8009190 comp rx=0 tx=0).stop 2024-03-06T07:23:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.403+0000 7f0fde5f7640 1 -- 172.21.15.1:0/3512889906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c6730 msgr2=0x7f0fd814a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.403+0000 7f0fde5f7640 1 --2- 172.21.15.1:0/3512889906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c6730 0x7f0fd814a610 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc02b8c0 tx=0x7f0fcc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.403+0000 7f0fde5f7640 1 -- 172.21.15.1:0/3512889906 shutdown_connections 2024-03-06T07:23:47.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.403+0000 7f0fde5f7640 1 --2- 172.21.15.1:0/3512889906 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fc406b500 0x7f0fc406d9c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.403+0000 7f0fde5f7640 1 --2- 172.21.15.1:0/3512889906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fd8152690 0x7f0fd814f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.403+0000 7f0fde5f7640 1 --2- 172.21.15.1:0/3512889906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c6730 0x7f0fd814a610 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.403+0000 7f0fde5f7640 1 --2- 172.21.15.1:0/3512889906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fd80c5d10 0x7f0fd80ad150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.403+0000 7f0fde5f7640 1 -- 172.21.15.1:0/3512889906 >> 172.21.15.1:0/3512889906 conn(0x7f0fd806b410 msgr2=0x7f0fd806c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.403+0000 7f0fde5f7640 1 -- 172.21.15.1:0/3512889906 shutdown_connections 2024-03-06T07:23:47.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.403+0000 7f0fde5f7640 1 -- 172.21.15.1:0/3512889906 wait complete. 2024-03-06T07:23:47.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:41.077040+0000', 'modified': '2024-03-06T07:23:43.077835+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6996}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6996': {'gid': 6996, 'name': 'd', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.98:6837/3330683497', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3330683497}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3330683497}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6996, 'qdb_cluster': [6996]} 2024-03-06T07:23:47.819 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:47.819 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:23:47.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.921+0000 7f29e7cd4640 1 Processor -- start 2024-03-06T07:23:47.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.921+0000 7f29e7cd4640 1 -- start start 2024-03-06T07:23:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.922+0000 7f29e7cd4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29e00c6b10 0x7f29e0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.922+0000 7f29e7cd4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e00c2fc0 con 0x7f29e0152650 2024-03-06T07:23:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.922+0000 7f29e7cd4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e0158da0 con 0x7f29e00bf9d0 2024-03-06T07:23:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.922+0000 7f29e7cd4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e0155800 con 0x7f29e00c6b10 2024-03-06T07:23:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.922+0000 7f29e5a70640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f29e0152650 0x7f29e0158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36074/0 (socket says 172.21.15.1:36074) 2024-03-06T07:23:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.922+0000 7f29e5a70640 1 -- 172.21.15.1:0/4294399028 learned_addr learned my addr 172.21.15.1:0/4294399028 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.922+0000 7f29e526f640 1 --2- 172.21.15.1:0/4294399028 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29e00c6b10 0x7f29e0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.923+0000 7f29e4a6e640 1 -- 172.21.15.1:0/4294399028 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529469044 0 0) 0x7f29e00c2fc0 con 0x7f29e0152650 2024-03-06T07:23:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.923+0000 7f29e4a6e640 1 -- 172.21.15.1:0/4294399028 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c8003520 con 0x7f29e0152650 2024-03-06T07:23:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.923+0000 7f29e4a6e640 1 -- 172.21.15.1:0/4294399028 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573478261 0 0) 0x7f29e0158da0 con 0x7f29e00bf9d0 2024-03-06T07:23:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.923+0000 7f29e4a6e640 1 -- 172.21.15.1:0/4294399028 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e00c2fc0 con 0x7f29e00bf9d0 2024-03-06T07:23:47.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.923+0000 7f29e526f640 1 -- 172.21.15.1:0/4294399028 >> v1:172.21.15.98:6789/0 conn(0x7f29e00bf9d0 legacy=0x7f29e00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.923+0000 7f29e526f640 1 -- 172.21.15.1:0/4294399028 >> v1:172.21.15.1:6789/0 conn(0x7f29e0152650 legacy=0x7f29e0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.923+0000 7f29e526f640 1 -- 172.21.15.1:0/4294399028 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e00c6690 con 0x7f29e00c6b10 2024-03-06T07:23:47.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.924+0000 7f29e526f640 1 --2- 172.21.15.1:0/4294399028 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29e00c6b10 0x7f29e0155c90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f29d40097d0 tx=0x7f29d402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5d2aea33e469b66 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.924+0000 7f29e4a6e640 1 -- 172.21.15.1:0/4294399028 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d4035020 con 0x7f29e00c6b10 2024-03-06T07:23:47.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.924+0000 7f29e4a6e640 1 -- 172.21.15.1:0/4294399028 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d4031800 con 0x7f29e00c6b10 2024-03-06T07:23:47.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.924+0000 7f29e4a6e640 1 -- 172.21.15.1:0/4294399028 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d4031bb0 con 0x7f29e00c6b10 2024-03-06T07:23:47.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.924+0000 7f29e7cd4640 1 -- 172.21.15.1:0/4294399028 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29e00c6b10 msgr2=0x7f29e0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.924+0000 7f29e7cd4640 1 --2- 172.21.15.1:0/4294399028 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29e00c6b10 0x7f29e0155c90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f29d40097d0 tx=0x7f29d402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.924+0000 7f29e7cd4640 1 -- 172.21.15.1:0/4294399028 shutdown_connections 2024-03-06T07:23:47.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.924+0000 7f29e7cd4640 1 --2- 172.21.15.1:0/4294399028 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29e00c6b10 0x7f29e0155c90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.924+0000 7f29e7cd4640 1 -- 172.21.15.1:0/4294399028 >> 172.21.15.1:0/4294399028 conn(0x7f29e0066ee0 msgr2=0x7f29e00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.924+0000 7f29e7cd4640 1 -- 172.21.15.1:0/4294399028 shutdown_connections 2024-03-06T07:23:47.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.924+0000 7f29e7cd4640 1 -- 172.21.15.1:0/4294399028 wait complete. 2024-03-06T07:23:47.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.925+0000 7f29e7cd4640 1 Processor -- start 2024-03-06T07:23:47.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.925+0000 7f29e7cd4640 1 -- start start 2024-03-06T07:23:47.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.926+0000 7f29e7cd4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e00bf9d0 0x7f29e0157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.926+0000 7f29e7cd4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29e0152650 0x7f29e0157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.926+0000 7f29e7cd4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0156c90 0x7f29e014d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.926+0000 7f29e7cd4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e01551b0 con 0x7f29e0156c90 2024-03-06T07:23:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.926+0000 7f29e7cd4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e00c6fa0 con 0x7f29e00bf9d0 2024-03-06T07:23:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.926+0000 7f29e7cd4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e0155490 con 0x7f29e0152650 2024-03-06T07:23:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.926+0000 7f29e6271640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0156c90 0x7f29e014d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.926+0000 7f29e6271640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0156c90 0x7f29e014d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37014/0 (socket says 172.21.15.1:37014) 2024-03-06T07:23:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.926+0000 7f29e6271640 1 -- 172.21.15.1:0/4025069360 learned_addr learned my addr 172.21.15.1:0/4025069360 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.926+0000 7f29e526f640 1 --2- 172.21.15.1:0/4025069360 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29e0152650 0x7f29e0157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.926+0000 7f29e5a70640 1 --2- 172.21.15.1:0/4025069360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e00bf9d0 0x7f29e0157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.926+0000 7f29e6271640 1 -- 172.21.15.1:0/4025069360 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29e0152650 msgr2=0x7f29e0157680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.926+0000 7f29e6271640 1 --2- 172.21.15.1:0/4025069360 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29e0152650 0x7f29e0157680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.926+0000 7f29e6271640 1 -- 172.21.15.1:0/4025069360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e00bf9d0 msgr2=0x7f29e0157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.926+0000 7f29e6271640 1 --2- 172.21.15.1:0/4025069360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e00bf9d0 0x7f29e0157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.927+0000 7f29e6271640 1 -- 172.21.15.1:0/4025069360 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e00ca3b0 con 0x7f29e0156c90 2024-03-06T07:23:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.927+0000 7f29e6271640 1 --2- 172.21.15.1:0/4025069360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0156c90 0x7f29e014d070 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f29d800c390 tx=0x7f29d800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.927+0000 7f29de7fc640 1 -- 172.21.15.1:0/4025069360 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d8017070 con 0x7f29e0156c90 2024-03-06T07:23:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.927+0000 7f29de7fc640 1 -- 172.21.15.1:0/4025069360 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d80052d0 con 0x7f29e0156c90 2024-03-06T07:23:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.927+0000 7f29de7fc640 1 -- 172.21.15.1:0/4025069360 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d80037f0 con 0x7f29e0156c90 2024-03-06T07:23:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.927+0000 7f29e7cd4640 1 -- 172.21.15.1:0/4025069360 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e01def90 con 0x7f29e0156c90 2024-03-06T07:23:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.927+0000 7f29e7cd4640 1 -- 172.21.15.1:0/4025069360 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e01ec100 con 0x7f29e0156c90 2024-03-06T07:23:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.928+0000 7f29de7fc640 1 -- 172.21.15.1:0/4025069360 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29d801a070 con 0x7f29e0156c90 2024-03-06T07:23:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.929+0000 7f29de7fc640 1 --2- 172.21.15.1:0/4025069360 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f29cc06b500 0x7f29cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.929+0000 7f29de7fc640 1 -- 172.21.15.1:0/4025069360 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f29d8089510 con 0x7f29e0156c90 2024-03-06T07:23:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.929+0000 7f29e5a70640 1 --2- 172.21.15.1:0/4025069360 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f29cc06b500 0x7f29cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.929+0000 7f29e5a70640 1 --2- 172.21.15.1:0/4025069360 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f29cc06b500 0x7f29cc06d9c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f29d0001560 tx=0x7f29d0001a90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.930+0000 7f29ea998740 1 Processor -- start 2024-03-06T07:23:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.930+0000 7f29ea998740 1 -- start start 2024-03-06T07:23:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.930+0000 7f29ea998740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf15bf50 0x55eeaf159a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.930+0000 7f29ea998740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf159f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.930+0000 7f29ea998740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf15a4b0 0x55eeaf156680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.930+0000 7f29ea998740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeaf0c06b0 con 0x55eeaf15c330 2024-03-06T07:23:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.930+0000 7f29ea998740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeaf069a60 con 0x55eeaf15a4b0 2024-03-06T07:23:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.930+0000 7f29ea998740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eeaf0bc750 con 0x55eeaf15bf50 2024-03-06T07:23:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.930+0000 7f29e526f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.930+0000 7f29e526f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf159f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37018/0 (socket says 172.21.15.1:37018) 2024-03-06T07:23:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.930+0000 7f29e526f640 1 -- 172.21.15.1:0/2281816049 learned_addr learned my addr 172.21.15.1:0/2281816049 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.930+0000 7f29e6271640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf15a4b0 0x55eeaf156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.930+0000 7f29e5a70640 1 --2- 172.21.15.1:0/2281816049 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf15bf50 0x55eeaf159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.931+0000 7f29e526f640 1 -- 172.21.15.1:0/2281816049 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf15bf50 msgr2=0x55eeaf159a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.931+0000 7f29e526f640 1 --2- 172.21.15.1:0/2281816049 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf15bf50 0x55eeaf159a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.931+0000 7f29e526f640 1 -- 172.21.15.1:0/2281816049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf15a4b0 msgr2=0x55eeaf156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.931+0000 7f29e526f640 1 --2- 172.21.15.1:0/2281816049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf15a4b0 0x55eeaf156680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.931+0000 7f29e526f640 1 -- 172.21.15.1:0/2281816049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eeaf087f30 con 0x55eeaf15c330 2024-03-06T07:23:47.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.931+0000 7f29e526f640 1 --2- 172.21.15.1:0/2281816049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf159f70 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f29d4034040 tx=0x7f29d4030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bcc7e942de4be3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.931+0000 7f29c77fe640 1 -- 172.21.15.1:0/2281816049 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d4048070 con 0x55eeaf15c330 2024-03-06T07:23:47.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.931+0000 7f29c77fe640 1 -- 172.21.15.1:0/2281816049 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d4004180 con 0x55eeaf15c330 2024-03-06T07:23:47.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.931+0000 7f29c77fe640 1 -- 172.21.15.1:0/2281816049 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d4035040 con 0x55eeaf15c330 2024-03-06T07:23:47.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.932+0000 7f29ea998740 1 -- 172.21.15.1:0/2281816049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 msgr2=0x55eeaf159f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.932+0000 7f29ea998740 1 --2- 172.21.15.1:0/2281816049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf159f70 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f29d4034040 tx=0x7f29d4030000 comp rx=0 tx=0).stop 2024-03-06T07:23:47.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.932+0000 7f29ea998740 1 -- 172.21.15.1:0/2281816049 shutdown_connections 2024-03-06T07:23:47.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.932+0000 7f29ea998740 1 --2- 172.21.15.1:0/2281816049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf15a4b0 0x55eeaf156680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.932+0000 7f29ea998740 1 --2- 172.21.15.1:0/2281816049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf159f70 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.932+0000 7f29ea998740 1 --2- 172.21.15.1:0/2281816049 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf15bf50 0x55eeaf159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.932+0000 7f29ea998740 1 -- 172.21.15.1:0/2281816049 >> 172.21.15.1:0/2281816049 conn(0x55eeaf1587d0 msgr2=0x55eeaf17d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.932+0000 7f29ea998740 1 -- 172.21.15.1:0/2281816049 shutdown_connections 2024-03-06T07:23:47.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.932+0000 7f29ea998740 1 -- 172.21.15.1:0/2281816049 wait complete. 2024-03-06T07:23:47.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.932+0000 7f29ea998740 1 -- start start 2024-03-06T07:23:47.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.932+0000 7f29ea998740 1 Processor -- start 2024-03-06T07:23:47.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.933+0000 7f29ea998740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15bf50 0x55eeaf182c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.933+0000 7f29ea998740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf15c330 0x55eeaf180dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.933+0000 7f29ea998740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf182360 0x55eeaf194c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.933+0000 7f29ea998740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeaf17e230 con 0x55eeaf15bf50 2024-03-06T07:23:47.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.933+0000 7f29ea998740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeaf07cea0 con 0x55eeaf182360 2024-03-06T07:23:47.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.933+0000 7f29ea998740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eeaf17dc70 con 0x55eeaf15c330 2024-03-06T07:23:47.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.933+0000 7f29ea998740 10 client.0 _ll_register_callbacks cb 0x55eeaf07c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:47.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.933+0000 7f29e5a70640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15bf50 0x55eeaf182c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.933+0000 7f29e5a70640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15bf50 0x55eeaf182c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37032/0 (socket says 172.21.15.1:37032) 2024-03-06T07:23:47.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.933+0000 7f29e5a70640 1 -- 172.21.15.1:0/1299668858 learned_addr learned my addr 172.21.15.1:0/1299668858 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:47.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.933+0000 7f29e526f640 1 --2- 172.21.15.1:0/1299668858 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf15c330 0x55eeaf180dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.933+0000 7f29e6271640 1 --2- 172.21.15.1:0/1299668858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf182360 0x55eeaf194c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29e5a70640 1 -- 172.21.15.1:0/1299668858 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf15c330 msgr2=0x55eeaf180dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29e5a70640 1 --2- 172.21.15.1:0/1299668858 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf15c330 0x55eeaf180dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29e5a70640 1 -- 172.21.15.1:0/1299668858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf182360 msgr2=0x55eeaf194c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29e5a70640 1 --2- 172.21.15.1:0/1299668858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf182360 0x55eeaf194c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29e5a70640 1 -- 172.21.15.1:0/1299668858 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eeaf07d270 con 0x55eeaf15bf50 2024-03-06T07:23:47.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29e526f640 1 --2- 172.21.15.1:0/1299668858 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf15c330 0x55eeaf180dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:47.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29e5a70640 1 --2- 172.21.15.1:0/1299668858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15bf50 0x55eeaf182c40 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f29d0013b40 tx=0x7f29d00014d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f99ffb95069e87 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:23:47.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29c4ff9640 1 -- 172.21.15.1:0/1299668858 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d0012ba0 con 0x55eeaf15bf50 2024-03-06T07:23:47.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29c4ff9640 1 -- 172.21.15.1:0/1299668858 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f29d0012d40 con 0x55eeaf15bf50 2024-03-06T07:23:47.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29ea998740 1 -- 172.21.15.1:0/1299668858 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f29d4004180 con 0x55eeaf15bf50 2024-03-06T07:23:47.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29c4ff9640 1 -- 172.21.15.1:0/1299668858 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29d002d470 con 0x55eeaf15bf50 2024-03-06T07:23:47.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29ea998740 10 client.7155 fetch_fsmap learned FSMap version 77 2024-03-06T07:23:47.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29ea998740 1 -- 172.21.15.1:0/1299668858 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x55eeaedd52d0 con 0x55eeaf15bf50 2024-03-06T07:23:47.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29c4ff9640 1 -- 172.21.15.1:0/1299668858 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d4004180 con 0x55eeaf15bf50 2024-03-06T07:23:47.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29c4ff9640 1 -- 172.21.15.1:0/1299668858 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 77) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f29d0021410 con 0x55eeaf15bf50 2024-03-06T07:23:47.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29ea998740 10 client.7155 fetch_fsmap finished waiting for FSMap version 77 2024-03-06T07:23:47.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29ea998740 10 client.7155 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:23:47.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29ea998740 20 client.7155 populate_metadata read hostname 'smithi001' 2024-03-06T07:23:47.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29ea998740 1 --2- 172.21.15.1:0/1299668858 >> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] conn(0x55eeaf1acc20 0x55eeaf1b7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29ea998740 4 client.7155 mds_command: new command op to 6996 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:47.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.934+0000 7f29ea998740 1 -- 172.21.15.1:0/1299668858 --> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eeaf0cdbf0 con 0x55eeaf1acc20 2024-03-06T07:23:47.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.935+0000 7f29e526f640 1 --2- 172.21.15.1:0/1299668858 >> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] conn(0x55eeaf1acc20 0x55eeaf1b7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.935+0000 7f29e526f640 1 --2- 172.21.15.1:0/1299668858 >> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] conn(0x55eeaf1acc20 0x55eeaf1b7720 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8739fc77c331727 server_cookie=b907bb07f0f3f34c in_seq=0 out_seq=0 2024-03-06T07:23:47.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.935+0000 7f29c4ff9640 10 client.7155 ms_handle_connect on v2:172.21.15.98:6835/3330683497 2024-03-06T07:23:47.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.936+0000 7f29c4ff9640 1 -- 172.21.15.1:0/1299668858 <== mds.0 v2:172.21.15.98:6835/3330683497 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55eeaf0cdbf0 con 0x55eeaf1acc20 2024-03-06T07:23:47.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.936+0000 7f29c4ff9640 10 client.7155 handle_command_reply: tid=0 2024-03-06T07:23:47.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.936+0000 7f29ea998740 1 client.7155 shutdown 2024-03-06T07:23:47.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.936+0000 7f29ea998740 1 -- 172.21.15.1:0/1299668858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15bf50 msgr2=0x55eeaf182c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.936+0000 7f29ea998740 1 --2- 172.21.15.1:0/1299668858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15bf50 0x55eeaf182c40 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f29d0013b40 tx=0x7f29d00014d0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.936+0000 7f29ea998740 1 -- 172.21.15.1:0/1299668858 shutdown_connections 2024-03-06T07:23:47.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.936+0000 7f29ea998740 1 --2- 172.21.15.1:0/1299668858 >> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] conn(0x55eeaf1acc20 0x55eeaf1b7720 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.936+0000 7f29ea998740 1 --2- 172.21.15.1:0/1299668858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf182360 0x55eeaf194c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.936+0000 7f29ea998740 1 --2- 172.21.15.1:0/1299668858 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf15c330 0x55eeaf180dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.936+0000 7f29c4ff9640 0 client.7155 ms_handle_reset on v2:172.21.15.98:6835/3330683497 2024-03-06T07:23:47.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.936+0000 7f29ea998740 1 --2- 172.21.15.1:0/1299668858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15bf50 0x55eeaf182c40 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.936+0000 7f29ea998740 1 -- 172.21.15.1:0/1299668858 >> 172.21.15.1:0/1299668858 conn(0x55eeaf15a4b0 msgr2=0x55eeaf15a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.936+0000 7f29ea998740 1 -- 172.21.15.1:0/1299668858 shutdown_connections 2024-03-06T07:23:47.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.936+0000 7f29ea998740 1 -- 172.21.15.1:0/1299668858 wait complete. 2024-03-06T07:23:47.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.936+0000 7f29ea998740 20 client.7155 trim_cache size 0 max 16384 2024-03-06T07:23:47.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.953+0000 7f29ea998740 1 Processor -- start 2024-03-06T07:23:47.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.953+0000 7f29ea998740 1 -- start start 2024-03-06T07:23:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.953+0000 7f29ea998740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf15bf50 0x55eeaf159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.953+0000 7f29ea998740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf155510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.953+0000 7f29ea998740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf182360 0x55eeaf19b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.953+0000 7f29ea998740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e01f6d00 con 0x55eeaf15c330 2024-03-06T07:23:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.953+0000 7f29ea998740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e01ef4e0 con 0x55eeaf15bf50 2024-03-06T07:23:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.953+0000 7f29ea998740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e01e11f0 con 0x55eeaf182360 2024-03-06T07:23:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.954+0000 7f29e526f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.954+0000 7f29e526f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf155510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37034/0 (socket says 172.21.15.1:37034) 2024-03-06T07:23:47.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.954+0000 7f29e526f640 1 -- 172.21.15.1:0/4094614873 learned_addr learned my addr 172.21.15.1:0/4094614873 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.954+0000 7f29e6271640 1 --2- 172.21.15.1:0/4094614873 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf182360 0x55eeaf19b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.954+0000 7f29e5a70640 1 --2- 172.21.15.1:0/4094614873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf15bf50 0x55eeaf159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.954+0000 7f29e526f640 1 -- 172.21.15.1:0/4094614873 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf182360 msgr2=0x55eeaf19b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.954+0000 7f29e526f640 1 --2- 172.21.15.1:0/4094614873 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf182360 0x55eeaf19b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.954+0000 7f29e526f640 1 -- 172.21.15.1:0/4094614873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf15bf50 msgr2=0x55eeaf159bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.954+0000 7f29e526f640 1 --2- 172.21.15.1:0/4094614873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf15bf50 0x55eeaf159bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.954+0000 7f29e526f640 1 -- 172.21.15.1:0/4094614873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eeaee9f6b0 con 0x55eeaf15c330 2024-03-06T07:23:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.954+0000 7f29e5a70640 1 --2- 172.21.15.1:0/4094614873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf15bf50 0x55eeaf159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.954+0000 7f29e526f640 1 --2- 172.21.15.1:0/4094614873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf155510 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f29d40098f0 tx=0x7f29d4031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19e032cb34f73be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.954+0000 7f29c6ffd640 1 -- 172.21.15.1:0/4094614873 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d4048040 con 0x55eeaf15c330 2024-03-06T07:23:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.954+0000 7f29c6ffd640 1 -- 172.21.15.1:0/4094614873 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d4006a20 con 0x55eeaf15c330 2024-03-06T07:23:47.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.954+0000 7f29c6ffd640 1 -- 172.21.15.1:0/4094614873 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d4035040 con 0x55eeaf15c330 2024-03-06T07:23:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.954+0000 7f29ea998740 1 -- 172.21.15.1:0/4094614873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 msgr2=0x55eeaf155510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.954+0000 7f29ea998740 1 --2- 172.21.15.1:0/4094614873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf155510 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f29d40098f0 tx=0x7f29d4031000 comp rx=0 tx=0).stop 2024-03-06T07:23:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.955+0000 7f29ea998740 1 -- 172.21.15.1:0/4094614873 shutdown_connections 2024-03-06T07:23:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.955+0000 7f29ea998740 1 --2- 172.21.15.1:0/4094614873 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf182360 0x55eeaf19b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.955+0000 7f29ea998740 1 --2- 172.21.15.1:0/4094614873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf155510 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.955+0000 7f29ea998740 1 --2- 172.21.15.1:0/4094614873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf15bf50 0x55eeaf159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.955+0000 7f29ea998740 1 -- 172.21.15.1:0/4094614873 >> 172.21.15.1:0/4094614873 conn(0x55eeaf15a4b0 msgr2=0x55eeaf182a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.955+0000 7f29ea998740 1 -- 172.21.15.1:0/4094614873 shutdown_connections 2024-03-06T07:23:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.955+0000 7f29ea998740 1 -- 172.21.15.1:0/4094614873 wait complete. 2024-03-06T07:23:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.955+0000 7f29ea998740 1 -- start start 2024-03-06T07:23:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.955+0000 7f29ea998740 1 Processor -- start 2024-03-06T07:23:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29ea998740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf15bf50 0x55eeaf1a92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29ea998740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf207e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29ea998740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf182360 0x55eeaf204ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29ea998740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeaf181350 con 0x55eeaf15c330 2024-03-06T07:23:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29ea998740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeaf180400 con 0x55eeaf15bf50 2024-03-06T07:23:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29ea998740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eeaf1ac080 con 0x55eeaf182360 2024-03-06T07:23:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29ea998740 10 client.0 _ll_register_callbacks cb 0x7f29d0013760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29e526f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf207e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29e526f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf207e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37050/0 (socket says 172.21.15.1:37050) 2024-03-06T07:23:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29e526f640 1 -- 172.21.15.1:0/238207675 learned_addr learned my addr 172.21.15.1:0/238207675 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29e5a70640 1 --2- 172.21.15.1:0/238207675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf15bf50 0x55eeaf1a92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29e6271640 1 --2- 172.21.15.1:0/238207675 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf182360 0x55eeaf204ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29e526f640 1 -- 172.21.15.1:0/238207675 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf182360 msgr2=0x55eeaf204ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29e526f640 1 --2- 172.21.15.1:0/238207675 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf182360 0x55eeaf204ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29e526f640 1 -- 172.21.15.1:0/238207675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf15bf50 msgr2=0x55eeaf1a92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29e526f640 1 --2- 172.21.15.1:0/238207675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf15bf50 0x55eeaf1a92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29e526f640 1 -- 172.21.15.1:0/238207675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eeaf182170 con 0x55eeaf15c330 2024-03-06T07:23:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29e526f640 1 --2- 172.21.15.1:0/238207675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf207e30 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f29d4009720 tx=0x7f29d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a30de951406f296 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.956+0000 7f29c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:23:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.957+0000 7f29c4ff9640 1 -- 172.21.15.1:0/238207675 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d4053020 con 0x55eeaf15c330 2024-03-06T07:23:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.957+0000 7f29c4ff9640 1 -- 172.21.15.1:0/238207675 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f29d404f090 con 0x55eeaf15c330 2024-03-06T07:23:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.957+0000 7f29ea998740 1 -- 172.21.15.1:0/238207675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f29d4006a20 con 0x55eeaf15c330 2024-03-06T07:23:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.957+0000 7f29c4ff9640 1 -- 172.21.15.1:0/238207675 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29d405d050 con 0x55eeaf15c330 2024-03-06T07:23:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.957+0000 7f29ea998740 10 client.7161 fetch_fsmap learned FSMap version 77 2024-03-06T07:23:47.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.957+0000 7f29ea998740 1 -- 172.21.15.1:0/238207675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x55eeaf155be0 con 0x55eeaf15c330 2024-03-06T07:23:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.957+0000 7f29c4ff9640 1 -- 172.21.15.1:0/238207675 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d4006a20 con 0x55eeaf15c330 2024-03-06T07:23:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.957+0000 7f29c4ff9640 1 -- 172.21.15.1:0/238207675 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 77) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f29d4060020 con 0x55eeaf15c330 2024-03-06T07:23:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.957+0000 7f29ea998740 10 client.7161 fetch_fsmap finished waiting for FSMap version 77 2024-03-06T07:23:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.957+0000 7f29ea998740 10 client.7161 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:23:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.957+0000 7f29ea998740 20 client.7161 populate_metadata read hostname 'smithi001' 2024-03-06T07:23:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.957+0000 7f29ea998740 1 --2- 172.21.15.1:0/238207675 >> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] conn(0x55eeaf205530 0x55eeaf1bb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.957+0000 7f29ea998740 4 client.7161 mds_command: new command op to 6996 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.957+0000 7f29ea998740 1 -- 172.21.15.1:0/238207675 --> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55eeaf0c7f40 con 0x55eeaf205530 2024-03-06T07:23:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.957+0000 7f29e5a70640 1 --2- 172.21.15.1:0/238207675 >> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] conn(0x55eeaf205530 0x55eeaf1bb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.958+0000 7f29e5a70640 1 --2- 172.21.15.1:0/238207675 >> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] conn(0x55eeaf205530 0x55eeaf1bb740 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c25cefe3bdad210 server_cookie=8a9cec5ade513d08 in_seq=0 out_seq=0 2024-03-06T07:23:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.958+0000 7f29c4ff9640 10 client.7161 ms_handle_connect on v2:172.21.15.98:6835/3330683497 2024-03-06T07:23:47.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.958+0000 7f29c4ff9640 1 -- 172.21.15.1:0/238207675 <== mds.0 v2:172.21.15.98:6835/3330683497 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55eeaf0c7f40 con 0x55eeaf205530 2024-03-06T07:23:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.958+0000 7f29c4ff9640 10 client.7161 handle_command_reply: tid=0 2024-03-06T07:23:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.958+0000 7f29ea998740 1 client.7161 shutdown 2024-03-06T07:23:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.958+0000 7f29ea998740 1 -- 172.21.15.1:0/238207675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 msgr2=0x55eeaf207e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.958+0000 7f29ea998740 1 --2- 172.21.15.1:0/238207675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf207e30 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f29d4009720 tx=0x7f29d4004730 comp rx=0 tx=0).stop 2024-03-06T07:23:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.958+0000 7f29ea998740 1 -- 172.21.15.1:0/238207675 shutdown_connections 2024-03-06T07:23:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.958+0000 7f29ea998740 1 --2- 172.21.15.1:0/238207675 >> [v2:172.21.15.98:6835/3330683497,v1:172.21.15.98:6837/3330683497] conn(0x55eeaf205530 0x55eeaf1bb740 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.958+0000 7f29ea998740 1 --2- 172.21.15.1:0/238207675 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55eeaf182360 0x55eeaf204ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.958+0000 7f29ea998740 1 --2- 172.21.15.1:0/238207675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55eeaf15c330 0x55eeaf207e30 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.958+0000 7f29c4ff9640 0 client.7161 ms_handle_reset on v2:172.21.15.98:6835/3330683497 2024-03-06T07:23:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.958+0000 7f29ea998740 1 --2- 172.21.15.1:0/238207675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eeaf15bf50 0x55eeaf1a92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.958+0000 7f29ea998740 1 -- 172.21.15.1:0/238207675 >> 172.21.15.1:0/238207675 conn(0x55eeaf15a4b0 msgr2=0x55eeaf1578d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.958+0000 7f29ea998740 1 -- 172.21.15.1:0/238207675 shutdown_connections 2024-03-06T07:23:47.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.958+0000 7f29ea998740 1 -- 172.21.15.1:0/238207675 wait complete. 2024-03-06T07:23:47.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.958+0000 7f29ea998740 20 client.7161 trim_cache size 0 max 16384 2024-03-06T07:23:47.968 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:23:47.968 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:23:47.968 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:23:47.968 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 6996, 2024-03-06T07:23:47.968 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:23:47.968 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:23:47.968 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:23:47.968 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 6.2373387889999998, 2024-03-06T07:23:47.969 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 77, 2024-03-06T07:23:47.969 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 95, 2024-03-06T07:23:47.969 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 95, 2024-03-06T07:23:47.969 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 11.332867024 2024-03-06T07:23:47.969 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:23:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.959+0000 7f29e7cd4640 1 -- 172.21.15.1:0/4025069360 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f29cc06b500 msgr2=0x7f29cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.959+0000 7f29e7cd4640 1 --2- 172.21.15.1:0/4025069360 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f29cc06b500 0x7f29cc06d9c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f29d0001560 tx=0x7f29d0001a90 comp rx=0 tx=0).stop 2024-03-06T07:23:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.959+0000 7f29e7cd4640 1 -- 172.21.15.1:0/4025069360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0156c90 msgr2=0x7f29e014d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.959+0000 7f29e7cd4640 1 --2- 172.21.15.1:0/4025069360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0156c90 0x7f29e014d070 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f29d800c390 tx=0x7f29d800c850 comp rx=0 tx=0).stop 2024-03-06T07:23:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.959+0000 7f29e7cd4640 1 -- 172.21.15.1:0/4025069360 shutdown_connections 2024-03-06T07:23:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.959+0000 7f29e7cd4640 1 --2- 172.21.15.1:0/4025069360 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f29cc06b500 0x7f29cc06d9c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.959+0000 7f29e7cd4640 1 --2- 172.21.15.1:0/4025069360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0156c90 0x7f29e014d070 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.959+0000 7f29e7cd4640 1 --2- 172.21.15.1:0/4025069360 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29e0152650 0x7f29e0157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.959+0000 7f29e7cd4640 1 --2- 172.21.15.1:0/4025069360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e00bf9d0 0x7f29e0157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.959+0000 7f29e7cd4640 1 -- 172.21.15.1:0/4025069360 >> 172.21.15.1:0/4025069360 conn(0x7f29e0066ee0 msgr2=0x7f29e00bff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.959+0000 7f29e7cd4640 1 -- 172.21.15.1:0/4025069360 shutdown_connections 2024-03-06T07:23:47.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:47.959+0000 7f29e7cd4640 1 -- 172.21.15.1:0/4025069360 wait complete. 2024-03-06T07:23:47.971 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 77, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 7, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:23:41.077040+0000", "damaged": [], "data_pools": [ 16 ], "enabled": true, "epoch": 77, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6996": { "addr": "172.21.15.98:6837/3330683497", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 3330683497, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 3330683497, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6996, "incarnation": 75, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 15, "modified": "2024-03-06T07:23:43.077835+0000", "qdb_cluster": [ 6996 ], "qdb_leader": 6996, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6996 } } } ], "standbys": [ { "addr": "172.21.15.98:6836/1335402413", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 1335402413, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 1335402413, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 74, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5855, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6834/2118550222", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 2118550222, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 2118550222, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 74, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6981, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6835/543699854", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 543699854, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 543699854, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 74, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6984, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:23:47.972 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 7, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:23:47.973 INFO:tasks.cephfs.test_failover:current = [{'gid': 6984, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/543699854', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 543699854}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 543699854}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 74}, {'gid': 5855, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/1335402413', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 1335402413}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1335402413}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 74}, {'gid': 6981, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6834/2118550222', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2118550222}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2118550222}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 74}, {'gid': 6996, 'name': 'd', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.98:6837/3330683497', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3330683497}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3330683497}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-03-06T07:23:47.973 DEBUG:tasks.cephfs.test_failover:7 != -1 2024-03-06T07:23:47.973 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:23:48.760 INFO:tasks.ceph.mds.d.smithi098.stderr: -104> 2024-03-06T07:23:46.989+0000 7fef69aca640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:48.785 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:23:51.990 INFO:tasks.ceph.mds.b.smithi098.stderr:2024-03-06T07:23:51.988+0000 7f251b5bd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:52.975 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.087+0000 7fc0909d0640 1 Processor -- start 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.087+0000 7fc0909d0640 1 -- start start 2024-03-06T07:23:53.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.088+0000 7fc0909d0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc08c0c6b10 0x7fc08c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.088+0000 7fc0909d0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc08c0bf9d0 0x7fc08c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.088+0000 7fc0909d0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08c0c2fc0 con 0x7fc08c152650 2024-03-06T07:23:53.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.088+0000 7fc0909d0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc08c155800 con 0x7fc08c0bf9d0 2024-03-06T07:23:53.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.088+0000 7fc0909d0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc08c158da0 con 0x7fc08c0c6b10 2024-03-06T07:23:53.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.088+0000 7fc08ad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc08c0bf9d0 0x7fc08c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.088+0000 7fc089d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc08c0c6b10 0x7fc08c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.088+0000 7fc08ad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc08c0bf9d0 0x7fc08c0c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37054/0 (socket says 172.21.15.1:37054) 2024-03-06T07:23:53.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.088+0000 7fc08ad9d640 1 -- 172.21.15.1:0/435452412 learned_addr learned my addr 172.21.15.1:0/435452412 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:53.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.089+0000 7fc089d9b640 1 -- 172.21.15.1:0/435452412 >> v2:172.21.15.1:3300/0 conn(0x7fc08c0bf9d0 msgr2=0x7fc08c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.089+0000 7fc089d9b640 1 --2- 172.21.15.1:0/435452412 >> v2:172.21.15.1:3300/0 conn(0x7fc08c0bf9d0 0x7fc08c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.089+0000 7fc089d9b640 1 -- 172.21.15.1:0/435452412 >> v1:172.21.15.98:6789/0 conn(0x7fc08c152650 legacy=0x7fc08c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.089+0000 7fc089d9b640 1 -- 172.21.15.1:0/435452412 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c0c6690 con 0x7fc08c0c6b10 2024-03-06T07:23:53.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.089+0000 7fc08959a640 1 -- 172.21.15.1:0/435452412 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442232668 0 0) 0x7fc08c0c2fc0 con 0x7fc08c152650 2024-03-06T07:23:53.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.089+0000 7fc089d9b640 1 --2- 172.21.15.1:0/435452412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc08c0c6b10 0x7fc08c155c90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc08c0cb8e0 tx=0x7fc07402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a621957435324721 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.090+0000 7fc08959a640 1 -- 172.21.15.1:0/435452412 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0740315a0 con 0x7fc08c0c6b10 2024-03-06T07:23:53.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.090+0000 7fc08959a640 1 -- 172.21.15.1:0/435452412 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc074037070 con 0x7fc08c0c6b10 2024-03-06T07:23:53.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.090+0000 7fc08959a640 1 -- 172.21.15.1:0/435452412 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc074031960 con 0x7fc08c0c6b10 2024-03-06T07:23:53.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.090+0000 7fc0909d0640 1 -- 172.21.15.1:0/435452412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc08c0c6b10 msgr2=0x7fc08c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.090+0000 7fc0909d0640 1 --2- 172.21.15.1:0/435452412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc08c0c6b10 0x7fc08c155c90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc08c0cb8e0 tx=0x7fc07402b840 comp rx=0 tx=0).stop 2024-03-06T07:23:53.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.090+0000 7fc0909d0640 1 -- 172.21.15.1:0/435452412 shutdown_connections 2024-03-06T07:23:53.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.090+0000 7fc0909d0640 1 --2- 172.21.15.1:0/435452412 >> v2:172.21.15.1:3300/0 conn(0x7fc08c0bf9d0 0x7fc08c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.090+0000 7fc0909d0640 1 --2- 172.21.15.1:0/435452412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc08c0c6b10 0x7fc08c155c90 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.090+0000 7fc0909d0640 1 -- 172.21.15.1:0/435452412 >> 172.21.15.1:0/435452412 conn(0x7fc08c066ee0 msgr2=0x7fc08c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.090+0000 7fc0909d0640 1 -- 172.21.15.1:0/435452412 shutdown_connections 2024-03-06T07:23:53.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.090+0000 7fc0909d0640 1 -- 172.21.15.1:0/435452412 wait complete. 2024-03-06T07:23:53.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.092+0000 7fc0909d0640 1 Processor -- start 2024-03-06T07:23:53.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.092+0000 7fc0909d0640 1 -- start start 2024-03-06T07:23:53.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.092+0000 7fc0909d0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc08c0bf9d0 0x7fc08c153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.092+0000 7fc0909d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc08c152650 0x7fc08c1de700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.092+0000 7fc0909d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc08c158960 0x7fc08c1dec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.092+0000 7fc0909d0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c1551b0 con 0x7fc08c0bf9d0 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.092+0000 7fc0909d0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c0c6fa0 con 0x7fc08c152650 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.092+0000 7fc0909d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c155490 con 0x7fc08c158960 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.092+0000 7fc08a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc08c0bf9d0 0x7fc08c153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.092+0000 7fc08a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc08c0bf9d0 0x7fc08c153c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37058/0 (socket says 172.21.15.1:37058) 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.093+0000 7fc08a59c640 1 -- 172.21.15.1:0/578567439 learned_addr learned my addr 172.21.15.1:0/578567439 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.093+0000 7fc089d9b640 1 --2- 172.21.15.1:0/578567439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc08c152650 0x7fc08c1de700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.093+0000 7fc08ad9d640 1 --2- 172.21.15.1:0/578567439 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc08c158960 0x7fc08c1dec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.093+0000 7fc089d9b640 1 -- 172.21.15.1:0/578567439 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc08c158960 msgr2=0x7fc08c1dec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.093+0000 7fc089d9b640 1 --2- 172.21.15.1:0/578567439 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc08c158960 0x7fc08c1dec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.093+0000 7fc089d9b640 1 -- 172.21.15.1:0/578567439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc08c0bf9d0 msgr2=0x7fc08c153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.093+0000 7fc089d9b640 1 --2- 172.21.15.1:0/578567439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc08c0bf9d0 0x7fc08c153c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.093+0000 7fc089d9b640 1 -- 172.21.15.1:0/578567439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c1eae30 con 0x7fc08c152650 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.093+0000 7fc08a59c640 1 --2- 172.21.15.1:0/578567439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc08c0bf9d0 0x7fc08c153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.093+0000 7fc089d9b640 1 --2- 172.21.15.1:0/578567439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc08c152650 0x7fc08c1de700 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc074033000 tx=0x7fc0740046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.093+0000 7fc082ffd640 1 -- 172.21.15.1:0/578567439 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc074032630 con 0x7fc08c152650 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.093+0000 7fc082ffd640 1 -- 172.21.15.1:0/578567439 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0740327d0 con 0x7fc08c152650 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.093+0000 7fc0909d0640 1 -- 172.21.15.1:0/578567439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08c1eb0f0 con 0x7fc08c152650 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.093+0000 7fc082ffd640 1 -- 172.21.15.1:0/578567439 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc074032ba0 con 0x7fc08c152650 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.093+0000 7fc0909d0640 1 -- 172.21.15.1:0/578567439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08c1e18e0 con 0x7fc08c152650 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.094+0000 7fc0909d0640 1 -- 172.21.15.1:0/578567439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08c152190 con 0x7fc08c152650 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.095+0000 7fc082ffd640 1 -- 172.21.15.1:0/578567439 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc07403b3f0 con 0x7fc08c152650 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.095+0000 7fc082ffd640 1 --2- 172.21.15.1:0/578567439 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc06c06b500 0x7fc06c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.095+0000 7fc082ffd640 1 -- 172.21.15.1:0/578567439 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc07402f030 con 0x7fc08c152650 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.095+0000 7fc08a59c640 1 --2- 172.21.15.1:0/578567439 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc06c06b500 0x7fc06c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.096+0000 7fc08a59c640 1 --2- 172.21.15.1:0/578567439 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc06c06b500 0x7fc06c06d9c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc07c0012e0 tx=0x7fc07c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.100+0000 7fc082ffd640 1 -- 172.21.15.1:0/578567439 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc0740af030 con 0x7fc08c152650 2024-03-06T07:23:53.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.318+0000 7fc0909d0640 1 -- 172.21.15.1:0/578567439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc08c067fd0 con 0x7fc08c152650 2024-03-06T07:23:53.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.318+0000 7fc082ffd640 1 -- 172.21.15.1:0/578567439 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4970 (secure 0 0 0) 0x7fc07404a060 con 0x7fc08c152650 2024-03-06T07:23:53.320 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:53.320 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":82,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3611156352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3611156352},{"type":"v1","addr":"172.21.15.98:6837","nonce":3611156352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6981,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/2118550222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2118550222},{"type":"v1","addr":"172.21.15.1:6834","nonce":2118550222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/543699854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":543699854},{"type":"v1","addr":"172.21.15.1:6835","nonce":543699854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:41.077040+0000","modified":"2024-03-06T07:23:51.762675+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5855},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5855":{"gid":5855,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":7,"addr":"172.21.15.98:6836/1335402413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1335402413},{"type":"v1","addr":"172.21.15.98:6836","nonce":1335402413}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5855,"qdb_cluster":[5855]},"id":7}]} 2024-03-06T07:23:53.321 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 82 2024-03-06T07:23:53.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.320+0000 7fc0909d0640 1 -- 172.21.15.1:0/578567439 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc06c06b500 msgr2=0x7fc06c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.320+0000 7fc0909d0640 1 --2- 172.21.15.1:0/578567439 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc06c06b500 0x7fc06c06d9c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc07c0012e0 tx=0x7fc07c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:53.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.320+0000 7fc0909d0640 1 -- 172.21.15.1:0/578567439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc08c152650 msgr2=0x7fc08c1de700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.320+0000 7fc0909d0640 1 --2- 172.21.15.1:0/578567439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc08c152650 0x7fc08c1de700 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc074033000 tx=0x7fc0740046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.320+0000 7fc0909d0640 1 -- 172.21.15.1:0/578567439 shutdown_connections 2024-03-06T07:23:53.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.320+0000 7fc0909d0640 1 --2- 172.21.15.1:0/578567439 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc06c06b500 0x7fc06c06d9c0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.320+0000 7fc0909d0640 1 --2- 172.21.15.1:0/578567439 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc08c158960 0x7fc08c1dec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.320+0000 7fc0909d0640 1 --2- 172.21.15.1:0/578567439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc08c152650 0x7fc08c1de700 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.320+0000 7fc0909d0640 1 --2- 172.21.15.1:0/578567439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc08c0bf9d0 0x7fc08c153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.320+0000 7fc0909d0640 1 -- 172.21.15.1:0/578567439 >> 172.21.15.1:0/578567439 conn(0x7fc08c066ee0 msgr2=0x7fc08c0c5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.320+0000 7fc0909d0640 1 -- 172.21.15.1:0/578567439 shutdown_connections 2024-03-06T07:23:53.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.320+0000 7fc0909d0640 1 -- 172.21.15.1:0/578567439 wait complete. 2024-03-06T07:23:53.736 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:41.077040+0000', 'modified': '2024-03-06T07:23:51.762675+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5855}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5855': {'gid': 5855, 'name': 'b', 'rank': 0, 'incarnation': 79, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.98:6836/1335402413', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 1335402413}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1335402413}]}, 'join_fscid': 7, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5855, 'qdb_cluster': [5855]} 2024-03-06T07:23:53.736 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:53.736 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:23:53.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.843+0000 7f8f1777a640 1 Processor -- start 2024-03-06T07:23:53.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.843+0000 7f8f1777a640 1 -- start start 2024-03-06T07:23:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.844+0000 7f8f1777a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8f100c6b10 0x7f8f10155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.844+0000 7f8f1777a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8f100bf9d0 0x7f8f100c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.844+0000 7f8f1777a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f100c2fc0 con 0x7f8f10152650 2024-03-06T07:23:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.844+0000 7f8f1777a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f10155800 con 0x7f8f100bf9d0 2024-03-06T07:23:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.844+0000 7f8f1777a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f10158da0 con 0x7f8f100c6b10 2024-03-06T07:23:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.844+0000 7f8f15d17640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8f100bf9d0 0x7f8f100c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.844+0000 7f8f15516640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8f10152650 0x7f8f10158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36080/0 (socket says 172.21.15.1:36080) 2024-03-06T07:23:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.844+0000 7f8f15516640 1 -- 172.21.15.1:0/1012050840 learned_addr learned my addr 172.21.15.1:0/1012050840 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.844+0000 7f8f15d17640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8f100bf9d0 0x7f8f100c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37062/0 (socket says 172.21.15.1:37062) 2024-03-06T07:23:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.844+0000 7f8f14d15640 1 --2- 172.21.15.1:0/1012050840 >> v2:172.21.15.98:3300/0 conn(0x7f8f100c6b10 0x7f8f10155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.845+0000 7f8f15d17640 1 -- 172.21.15.1:0/1012050840 >> v2:172.21.15.98:3300/0 conn(0x7f8f100c6b10 msgr2=0x7f8f10155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.845+0000 7f8f15d17640 1 --2- 172.21.15.1:0/1012050840 >> v2:172.21.15.98:3300/0 conn(0x7f8f100c6b10 0x7f8f10155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.845+0000 7f8f15d17640 1 -- 172.21.15.1:0/1012050840 >> v1:172.21.15.1:6789/0 conn(0x7f8f10152650 legacy=0x7f8f10158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.845+0000 7f8f07fff640 1 -- 172.21.15.1:0/1012050840 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328568289 0 0) 0x7f8f100c2fc0 con 0x7f8f10152650 2024-03-06T07:23:53.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.845+0000 7f8f15d17640 1 -- 172.21.15.1:0/1012050840 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f100c6690 con 0x7f8f100bf9d0 2024-03-06T07:23:53.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.845+0000 7f8f15d17640 1 --2- 172.21.15.1:0/1012050840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f100bf9d0 0x7f8f100c1460 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f8f100cb8e0 tx=0x7f8f0802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92294fa7e3b66008 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.846+0000 7f8f07fff640 1 -- 172.21.15.1:0/1012050840 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f08039070 con 0x7f8f100bf9d0 2024-03-06T07:23:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.846+0000 7f8f07fff640 1 -- 172.21.15.1:0/1012050840 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0802cbf0 con 0x7f8f100bf9d0 2024-03-06T07:23:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.846+0000 7f8f07fff640 1 -- 172.21.15.1:0/1012050840 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f08034440 con 0x7f8f100bf9d0 2024-03-06T07:23:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.846+0000 7f8f1777a640 1 -- 172.21.15.1:0/1012050840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f100bf9d0 msgr2=0x7f8f100c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.846+0000 7f8f1777a640 1 --2- 172.21.15.1:0/1012050840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f100bf9d0 0x7f8f100c1460 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f8f100cb8e0 tx=0x7f8f0802be40 comp rx=0 tx=0).stop 2024-03-06T07:23:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.846+0000 7f8f1777a640 1 -- 172.21.15.1:0/1012050840 shutdown_connections 2024-03-06T07:23:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.846+0000 7f8f1777a640 1 --2- 172.21.15.1:0/1012050840 >> v2:172.21.15.98:3300/0 conn(0x7f8f100c6b10 0x7f8f10155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.846+0000 7f8f1777a640 1 --2- 172.21.15.1:0/1012050840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f100bf9d0 0x7f8f100c1460 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.846+0000 7f8f1777a640 1 -- 172.21.15.1:0/1012050840 >> 172.21.15.1:0/1012050840 conn(0x7f8f10066ee0 msgr2=0x7f8f100672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.846+0000 7f8f1777a640 1 -- 172.21.15.1:0/1012050840 shutdown_connections 2024-03-06T07:23:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.846+0000 7f8f1777a640 1 -- 172.21.15.1:0/1012050840 wait complete. 2024-03-06T07:23:53.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.847+0000 7f8f1777a640 1 Processor -- start 2024-03-06T07:23:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.847+0000 7f8f1777a640 1 -- start start 2024-03-06T07:23:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.848+0000 7f8f1777a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f100bf9d0 0x7f8f101eb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.848+0000 7f8f1777a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f10152650 0x7f8f100c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.848+0000 7f8f1777a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f10153ec0 0x7f8f10156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.848+0000 7f8f1777a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f101551b0 con 0x7f8f10152650 2024-03-06T07:23:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.848+0000 7f8f1777a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f100c6fa0 con 0x7f8f10153ec0 2024-03-06T07:23:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.848+0000 7f8f1777a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f10155490 con 0x7f8f100bf9d0 2024-03-06T07:23:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.848+0000 7f8f14d15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f10152650 0x7f8f100c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.848+0000 7f8f14d15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f10152650 0x7f8f100c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37068/0 (socket says 172.21.15.1:37068) 2024-03-06T07:23:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.848+0000 7f8f14d15640 1 -- 172.21.15.1:0/3654747004 learned_addr learned my addr 172.21.15.1:0/3654747004 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.848+0000 7f8f15516640 1 --2- 172.21.15.1:0/3654747004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f100bf9d0 0x7f8f101eb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.848+0000 7f8f15d17640 1 --2- 172.21.15.1:0/3654747004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f10153ec0 0x7f8f10156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.848+0000 7f8f14d15640 1 -- 172.21.15.1:0/3654747004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f100bf9d0 msgr2=0x7f8f101eb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.848+0000 7f8f14d15640 1 --2- 172.21.15.1:0/3654747004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f100bf9d0 0x7f8f101eb520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.848+0000 7f8f14d15640 1 -- 172.21.15.1:0/3654747004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f10153ec0 msgr2=0x7f8f10156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.848+0000 7f8f14d15640 1 --2- 172.21.15.1:0/3654747004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f10153ec0 0x7f8f10156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.848+0000 7f8f14d15640 1 -- 172.21.15.1:0/3654747004 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f10158b90 con 0x7f8f10152650 2024-03-06T07:23:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.849+0000 7f8f14d15640 1 --2- 172.21.15.1:0/3654747004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f10152650 0x7f8f100c9a90 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f8f0000ea10 tx=0x7f8f0000eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.849+0000 7f8f05ffb640 1 -- 172.21.15.1:0/3654747004 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00017070 con 0x7f8f10152650 2024-03-06T07:23:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.849+0000 7f8f05ffb640 1 -- 172.21.15.1:0/3654747004 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0000c9a0 con 0x7f8f10152650 2024-03-06T07:23:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.849+0000 7f8f1777a640 1 -- 172.21.15.1:0/3654747004 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f101eea50 con 0x7f8f10152650 2024-03-06T07:23:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.849+0000 7f8f05ffb640 1 -- 172.21.15.1:0/3654747004 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0000cd90 con 0x7f8f10152650 2024-03-06T07:23:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.849+0000 7f8f1777a640 1 -- 172.21.15.1:0/3654747004 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f101eaa40 con 0x7f8f10152650 2024-03-06T07:23:53.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.850+0000 7f8f05ffb640 1 -- 172.21.15.1:0/3654747004 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f0001a070 con 0x7f8f10152650 2024-03-06T07:23:53.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.851+0000 7f8f05ffb640 1 --2- 172.21.15.1:0/3654747004 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ef406b500 0x7f8ef406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.851+0000 7f8f05ffb640 1 -- 172.21.15.1:0/3654747004 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8f0008a250 con 0x7f8f10152650 2024-03-06T07:23:53.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.851+0000 7f8f15516640 1 --2- 172.21.15.1:0/3654747004 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ef406b500 0x7f8ef406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.852+0000 7f8f15516640 1 --2- 172.21.15.1:0/3654747004 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ef406b500 0x7f8ef406d9c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c000f60 tx=0x7f8f0c007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.852+0000 7f8f19ffd740 1 Processor -- start 2024-03-06T07:23:53.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.852+0000 7f8f19ffd740 1 -- start start 2024-03-06T07:23:53.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.852+0000 7f8f19ffd740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2612f50 0x5586e2610a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8f19ffd740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2613330 0x5586e2610f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8f19ffd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e26114b0 0x5586e260d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8f19ffd740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586e25776b0 con 0x5586e2612f50 2024-03-06T07:23:53.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8f19ffd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586e2520a60 con 0x5586e26114b0 2024-03-06T07:23:53.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8f19ffd740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5586e2573750 con 0x5586e2613330 2024-03-06T07:23:53.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8f15516640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2612f50 0x5586e2610a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8f15516640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2612f50 0x5586e2610a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37074/0 (socket says 172.21.15.1:37074) 2024-03-06T07:23:53.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8f15516640 1 -- 172.21.15.1:0/2984150508 learned_addr learned my addr 172.21.15.1:0/2984150508 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:53.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8f14d15640 1 --2- 172.21.15.1:0/2984150508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2613330 0x5586e2610f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8f15d17640 1 --2- 172.21.15.1:0/2984150508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e26114b0 0x5586e260d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8f15516640 1 -- 172.21.15.1:0/2984150508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2613330 msgr2=0x5586e2610f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8f15516640 1 --2- 172.21.15.1:0/2984150508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2613330 0x5586e2610f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8f15516640 1 -- 172.21.15.1:0/2984150508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e26114b0 msgr2=0x5586e260d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8f15516640 1 --2- 172.21.15.1:0/2984150508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e26114b0 0x5586e260d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8f15516640 1 -- 172.21.15.1:0/2984150508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5586e253ef30 con 0x5586e2612f50 2024-03-06T07:23:53.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8f15516640 1 --2- 172.21.15.1:0/2984150508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2612f50 0x5586e2610a30 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c007aa0 tx=0x7f8f0c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=834901bb4eff8343 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8efaffd640 1 -- 172.21.15.1:0/2984150508 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c026070 con 0x5586e2612f50 2024-03-06T07:23:53.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8efaffd640 1 -- 172.21.15.1:0/2984150508 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0c013b80 con 0x5586e2612f50 2024-03-06T07:23:53.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.853+0000 7f8efaffd640 1 -- 172.21.15.1:0/2984150508 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c012350 con 0x5586e2612f50 2024-03-06T07:23:53.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.854+0000 7f8f19ffd740 1 -- 172.21.15.1:0/2984150508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2612f50 msgr2=0x5586e2610a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.854+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/2984150508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2612f50 0x5586e2610a30 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c007aa0 tx=0x7f8f0c013000 comp rx=0 tx=0).stop 2024-03-06T07:23:53.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.854+0000 7f8f19ffd740 1 -- 172.21.15.1:0/2984150508 shutdown_connections 2024-03-06T07:23:53.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.854+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/2984150508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e26114b0 0x5586e260d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.854+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/2984150508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2613330 0x5586e2610f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.854+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/2984150508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2612f50 0x5586e2610a30 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.854+0000 7f8f19ffd740 1 -- 172.21.15.1:0/2984150508 >> 172.21.15.1:0/2984150508 conn(0x5586e260f7d0 msgr2=0x5586e26343c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.854+0000 7f8f19ffd740 1 -- 172.21.15.1:0/2984150508 shutdown_connections 2024-03-06T07:23:53.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.854+0000 7f8f19ffd740 1 -- 172.21.15.1:0/2984150508 wait complete. 2024-03-06T07:23:53.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.854+0000 7f8f19ffd740 1 -- start start 2024-03-06T07:23:53.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.854+0000 7f8f19ffd740 1 Processor -- start 2024-03-06T07:23:53.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.855+0000 7f8f19ffd740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2612f50 0x5586e264c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.855+0000 7f8f19ffd740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2613330 0x5586e2636da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.855+0000 7f8f19ffd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e2639790 0x5586e2639c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.855+0000 7f8f19ffd740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586e2635230 con 0x5586e2613330 2024-03-06T07:23:53.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.855+0000 7f8f19ffd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586e2533ea0 con 0x5586e2639790 2024-03-06T07:23:53.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.855+0000 7f8f19ffd740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5586e2634c70 con 0x5586e2612f50 2024-03-06T07:23:53.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.855+0000 7f8f19ffd740 10 client.0 _ll_register_callbacks cb 0x5586e2533870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:53.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.855+0000 7f8f14d15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2613330 0x5586e2636da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.855+0000 7f8f14d15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2613330 0x5586e2636da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37086/0 (socket says 172.21.15.1:37086) 2024-03-06T07:23:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.855+0000 7f8f14d15640 1 -- 172.21.15.1:0/3740341526 learned_addr learned my addr 172.21.15.1:0/3740341526 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.855+0000 7f8f15516640 1 --2- 172.21.15.1:0/3740341526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2612f50 0x5586e264c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.856+0000 7f8f15d17640 1 --2- 172.21.15.1:0/3740341526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e2639790 0x5586e2639c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.856+0000 7f8f14d15640 1 -- 172.21.15.1:0/3740341526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2612f50 msgr2=0x5586e264c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.856+0000 7f8f14d15640 1 --2- 172.21.15.1:0/3740341526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2612f50 0x5586e264c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.856+0000 7f8f14d15640 1 -- 172.21.15.1:0/3740341526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e2639790 msgr2=0x5586e2639c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.856+0000 7f8f14d15640 1 --2- 172.21.15.1:0/3740341526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e2639790 0x5586e2639c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.856+0000 7f8f14d15640 1 -- 172.21.15.1:0/3740341526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5586e2534270 con 0x5586e2613330 2024-03-06T07:23:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.856+0000 7f8f14d15640 1 --2- 172.21.15.1:0/3740341526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2613330 0x5586e2636da0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f8f000097a0 tx=0x7f8f000584e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19a3782e0ff2a4c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.856+0000 7f8eeffff640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:23:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.856+0000 7f8eeffff640 1 -- 172.21.15.1:0/3740341526 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00021030 con 0x5586e2613330 2024-03-06T07:23:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.856+0000 7f8eeffff640 1 -- 172.21.15.1:0/3740341526 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8f00057ce0 con 0x5586e2613330 2024-03-06T07:23:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.856+0000 7f8f19ffd740 1 -- 172.21.15.1:0/3740341526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f0c013b80 con 0x5586e2613330 2024-03-06T07:23:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.856+0000 7f8eeffff640 1 -- 172.21.15.1:0/3740341526 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f00060ef0 con 0x5586e2613330 2024-03-06T07:23:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.856+0000 7f8f19ffd740 10 client.7191 fetch_fsmap learned FSMap version 82 2024-03-06T07:23:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.856+0000 7f8f19ffd740 1 -- 172.21.15.1:0/3740341526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x5586e228c2d0 con 0x5586e2613330 2024-03-06T07:23:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.856+0000 7f8eeffff640 1 -- 172.21.15.1:0/3740341526 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0c013b80 con 0x5586e2613330 2024-03-06T07:23:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.856+0000 7f8eeffff640 1 -- 172.21.15.1:0/3740341526 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 82) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8f00024060 con 0x5586e2613330 2024-03-06T07:23:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.856+0000 7f8f19ffd740 10 client.7191 fetch_fsmap finished waiting for FSMap version 82 2024-03-06T07:23:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.857+0000 7f8f19ffd740 10 client.7191 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:23:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.857+0000 7f8f19ffd740 20 client.7191 populate_metadata read hostname 'smithi001' 2024-03-06T07:23:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.857+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/3740341526 >> [v2:172.21.15.98:6834/1335402413,v1:172.21.15.98:6836/1335402413] conn(0x5586e2635d70 0x5586e266e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.857+0000 7f8f19ffd740 4 client.7191 mds_command: new command op to 5855 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.857+0000 7f8f19ffd740 1 -- 172.21.15.1:0/3740341526 --> [v2:172.21.15.98:6834/1335402413,v1:172.21.15.98:6836/1335402413] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5586e2584bf0 con 0x5586e2635d70 2024-03-06T07:23:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.857+0000 7f8f15516640 1 --2- 172.21.15.1:0/3740341526 >> [v2:172.21.15.98:6834/1335402413,v1:172.21.15.98:6836/1335402413] conn(0x5586e2635d70 0x5586e266e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.857+0000 7f8eeffff640 10 client.7191 ms_handle_connect on v2:172.21.15.98:6834/1335402413 2024-03-06T07:23:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.857+0000 7f8f15516640 1 --2- 172.21.15.1:0/3740341526 >> [v2:172.21.15.98:6834/1335402413,v1:172.21.15.98:6836/1335402413] conn(0x5586e2635d70 0x5586e266e590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79bb9603adff2e45 server_cookie=a1e7f18efdaab39a in_seq=0 out_seq=0 2024-03-06T07:23:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.858+0000 7f8eeffff640 1 -- 172.21.15.1:0/3740341526 <== mds.0 v2:172.21.15.98:6834/1335402413 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5586e2584bf0 con 0x5586e2635d70 2024-03-06T07:23:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.858+0000 7f8eeffff640 10 client.7191 handle_command_reply: tid=0 2024-03-06T07:23:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.858+0000 7f8f19ffd740 1 client.7191 shutdown 2024-03-06T07:23:53.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.858+0000 7f8f19ffd740 1 -- 172.21.15.1:0/3740341526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2613330 msgr2=0x5586e2636da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.858+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/3740341526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2613330 0x5586e2636da0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f8f000097a0 tx=0x7f8f000584e0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.858+0000 7f8f19ffd740 1 -- 172.21.15.1:0/3740341526 shutdown_connections 2024-03-06T07:23:53.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.858+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/3740341526 >> [v2:172.21.15.98:6834/1335402413,v1:172.21.15.98:6836/1335402413] conn(0x5586e2635d70 0x5586e266e590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.858+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/3740341526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e2639790 0x5586e2639c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.858+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/3740341526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2613330 0x5586e2636da0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.858+0000 7f8eeffff640 0 client.7191 ms_handle_reset on v2:172.21.15.98:6834/1335402413 2024-03-06T07:23:53.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.858+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/3740341526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2612f50 0x5586e264c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.858+0000 7f8f19ffd740 1 -- 172.21.15.1:0/3740341526 >> 172.21.15.1:0/3740341526 conn(0x5586e26114b0 msgr2=0x5586e264fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.858+0000 7f8f19ffd740 1 -- 172.21.15.1:0/3740341526 shutdown_connections 2024-03-06T07:23:53.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.858+0000 7f8f19ffd740 1 -- 172.21.15.1:0/3740341526 wait complete. 2024-03-06T07:23:53.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.858+0000 7f8f19ffd740 20 client.7191 trim_cache size 0 max 16384 2024-03-06T07:23:53.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.875+0000 7f8f19ffd740 1 Processor -- start 2024-03-06T07:23:53.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.875+0000 7f8f19ffd740 1 -- start start 2024-03-06T07:23:53.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.875+0000 7f8f19ffd740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2612f50 0x5586e265fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.875+0000 7f8f19ffd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e2613330 0x5586e264be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.875+0000 7f8f19ffd740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2639790 0x5586e264c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.875+0000 7f8f19ffd740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f101f7c70 con 0x5586e2639790 2024-03-06T07:23:53.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.875+0000 7f8f19ffd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f101f6e60 con 0x5586e2613330 2024-03-06T07:23:53.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.875+0000 7f8f19ffd740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f101f63e0 con 0x5586e2612f50 2024-03-06T07:23:53.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.876+0000 7f8f15d17640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2639790 0x5586e264c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.876+0000 7f8f15d17640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2639790 0x5586e264c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37096/0 (socket says 172.21.15.1:37096) 2024-03-06T07:23:53.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.876+0000 7f8f15d17640 1 -- 172.21.15.1:0/1777793637 learned_addr learned my addr 172.21.15.1:0/1777793637 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:53.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.876+0000 7f8f15516640 1 --2- 172.21.15.1:0/1777793637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2612f50 0x5586e265fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.876+0000 7f8f14d15640 1 --2- 172.21.15.1:0/1777793637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e2613330 0x5586e264be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.876+0000 7f8f15d17640 1 -- 172.21.15.1:0/1777793637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2612f50 msgr2=0x5586e265fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.876+0000 7f8f15d17640 1 --2- 172.21.15.1:0/1777793637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2612f50 0x5586e265fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.876+0000 7f8f15d17640 1 -- 172.21.15.1:0/1777793637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e2613330 msgr2=0x5586e264be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.876+0000 7f8f15d17640 1 --2- 172.21.15.1:0/1777793637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e2613330 0x5586e264be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.876+0000 7f8f15d17640 1 -- 172.21.15.1:0/1777793637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5586e2639f80 con 0x5586e2639790 2024-03-06T07:23:53.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.876+0000 7f8f15d17640 1 --2- 172.21.15.1:0/1777793637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2639790 0x5586e264c3b0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f8f08009e80 tx=0x7f8f0802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b77f2dd51f5f78b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.876+0000 7f8efa7fc640 1 -- 172.21.15.1:0/1777793637 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f08039040 con 0x5586e2639790 2024-03-06T07:23:53.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.876+0000 7f8efa7fc640 1 -- 172.21.15.1:0/1777793637 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0803a040 con 0x5586e2639790 2024-03-06T07:23:53.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.876+0000 7f8efa7fc640 1 -- 172.21.15.1:0/1777793637 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f08041660 con 0x5586e2639790 2024-03-06T07:23:53.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.877+0000 7f8f19ffd740 1 -- 172.21.15.1:0/1777793637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2639790 msgr2=0x5586e264c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.877+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/1777793637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2639790 0x5586e264c3b0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f8f08009e80 tx=0x7f8f0802e000 comp rx=0 tx=0).stop 2024-03-06T07:23:53.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.877+0000 7f8f19ffd740 1 -- 172.21.15.1:0/1777793637 shutdown_connections 2024-03-06T07:23:53.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.877+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/1777793637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2639790 0x5586e264c3b0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.877+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/1777793637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e2613330 0x5586e264be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.877+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/1777793637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2612f50 0x5586e265fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.877+0000 7f8f19ffd740 1 -- 172.21.15.1:0/1777793637 >> 172.21.15.1:0/1777793637 conn(0x5586e26114b0 msgr2=0x5586e2636150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.877+0000 7f8f19ffd740 1 -- 172.21.15.1:0/1777793637 shutdown_connections 2024-03-06T07:23:53.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.877+0000 7f8f19ffd740 1 -- 172.21.15.1:0/1777793637 wait complete. 2024-03-06T07:23:53.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.877+0000 7f8f19ffd740 1 -- start start 2024-03-06T07:23:53.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.877+0000 7f8f19ffd740 1 Processor -- start 2024-03-06T07:23:53.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f19ffd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e2612f50 0x5586e26a8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f19ffd740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2613330 0x5586e2652600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f19ffd740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2639790 0x5586e2670980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f19ffd740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586e2635bf0 con 0x5586e2613330 2024-03-06T07:23:53.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f19ffd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f101f6860 con 0x5586e2612f50 2024-03-06T07:23:53.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f19ffd740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5586e2639090 con 0x5586e2639790 2024-03-06T07:23:53.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f19ffd740 10 client.0 _ll_register_callbacks cb 0x7f8f000581f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:53.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f14d15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2613330 0x5586e2652600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f15516640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e2612f50 0x5586e26a8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f14d15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2613330 0x5586e2652600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37102/0 (socket says 172.21.15.1:37102) 2024-03-06T07:23:53.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f15516640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e2612f50 0x5586e26a8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:57726/0 (socket says 172.21.15.1:57726) 2024-03-06T07:23:53.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f14d15640 1 -- 172.21.15.1:0/795544561 learned_addr learned my addr 172.21.15.1:0/795544561 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:53.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f15d17640 1 --2- 172.21.15.1:0/795544561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2639790 0x5586e2670980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f14d15640 1 -- 172.21.15.1:0/795544561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2639790 msgr2=0x5586e2670980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f14d15640 1 --2- 172.21.15.1:0/795544561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2639790 0x5586e2670980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f14d15640 1 -- 172.21.15.1:0/795544561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e2612f50 msgr2=0x5586e26a8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f14d15640 1 --2- 172.21.15.1:0/795544561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e2612f50 0x5586e26a8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f14d15640 1 -- 172.21.15.1:0/795544561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5586e23566b0 con 0x5586e2613330 2024-03-06T07:23:53.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.878+0000 7f8f15516640 1 --2- 172.21.15.1:0/795544561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e2612f50 0x5586e26a8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.879+0000 7f8f14d15640 1 --2- 172.21.15.1:0/795544561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2613330 0x5586e2652600 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f8f0008a1c0 tx=0x7f8f00088460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3eda2febd1cb287 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.879+0000 7f8eeffff640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:23:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.879+0000 7f8eeffff640 1 -- 172.21.15.1:0/795544561 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00021040 con 0x5586e2613330 2024-03-06T07:23:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.879+0000 7f8eeffff640 1 -- 172.21.15.1:0/795544561 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8f00077340 con 0x5586e2613330 2024-03-06T07:23:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.879+0000 7f8f19ffd740 1 -- 172.21.15.1:0/795544561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f0803a040 con 0x5586e2613330 2024-03-06T07:23:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.879+0000 7f8eeffff640 1 -- 172.21.15.1:0/795544561 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f0007e050 con 0x5586e2613330 2024-03-06T07:23:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.879+0000 7f8f19ffd740 10 client.7197 fetch_fsmap learned FSMap version 82 2024-03-06T07:23:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.879+0000 7f8f19ffd740 1 -- 172.21.15.1:0/795544561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x5586e265be80 con 0x5586e2613330 2024-03-06T07:23:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.879+0000 7f8eeffff640 1 -- 172.21.15.1:0/795544561 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0803a040 con 0x5586e2613330 2024-03-06T07:23:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.879+0000 7f8eeffff640 1 -- 172.21.15.1:0/795544561 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 82) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8f00081020 con 0x5586e2613330 2024-03-06T07:23:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.879+0000 7f8f19ffd740 10 client.7197 fetch_fsmap finished waiting for FSMap version 82 2024-03-06T07:23:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.879+0000 7f8f19ffd740 10 client.7197 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:23:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.879+0000 7f8f19ffd740 20 client.7197 populate_metadata read hostname 'smithi001' 2024-03-06T07:23:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.879+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/795544561 >> [v2:172.21.15.98:6834/1335402413,v1:172.21.15.98:6836/1335402413] conn(0x5586e2630bb0 0x5586e2630f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.879+0000 7f8f19ffd740 4 client.7197 mds_command: new command op to 5855 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.879+0000 7f8f19ffd740 1 -- 172.21.15.1:0/795544561 --> [v2:172.21.15.98:6834/1335402413,v1:172.21.15.98:6836/1335402413] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5586e257ef40 con 0x5586e2630bb0 2024-03-06T07:23:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.879+0000 7f8f15516640 1 --2- 172.21.15.1:0/795544561 >> [v2:172.21.15.98:6834/1335402413,v1:172.21.15.98:6836/1335402413] conn(0x5586e2630bb0 0x5586e2630f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.880+0000 7f8f15516640 1 --2- 172.21.15.1:0/795544561 >> [v2:172.21.15.98:6834/1335402413,v1:172.21.15.98:6836/1335402413] conn(0x5586e2630bb0 0x5586e2630f90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba333ca28ad97aa8 server_cookie=de55d300afcbf5d7 in_seq=0 out_seq=0 2024-03-06T07:23:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.880+0000 7f8eeffff640 10 client.7197 ms_handle_connect on v2:172.21.15.98:6834/1335402413 2024-03-06T07:23:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.880+0000 7f8eeffff640 1 -- 172.21.15.1:0/795544561 <== mds.0 v2:172.21.15.98:6834/1335402413 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5586e257ef40 con 0x5586e2630bb0 2024-03-06T07:23:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.880+0000 7f8eeffff640 10 client.7197 handle_command_reply: tid=0 2024-03-06T07:23:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.880+0000 7f8f19ffd740 1 client.7197 shutdown 2024-03-06T07:23:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.880+0000 7f8f19ffd740 1 -- 172.21.15.1:0/795544561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2613330 msgr2=0x5586e2652600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.880+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/795544561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2613330 0x5586e2652600 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f8f0008a1c0 tx=0x7f8f00088460 comp rx=0 tx=0).stop 2024-03-06T07:23:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.880+0000 7f8f19ffd740 1 -- 172.21.15.1:0/795544561 shutdown_connections 2024-03-06T07:23:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.880+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/795544561 >> [v2:172.21.15.98:6834/1335402413,v1:172.21.15.98:6836/1335402413] conn(0x5586e2630bb0 0x5586e2630f90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.880+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/795544561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5586e2639790 0x5586e2670980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.880+0000 7f8eeffff640 0 client.7197 ms_handle_reset on v2:172.21.15.98:6834/1335402413 2024-03-06T07:23:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.880+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/795544561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5586e2613330 0x5586e2652600 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.880+0000 7f8f19ffd740 1 --2- 172.21.15.1:0/795544561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586e2612f50 0x5586e26a8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.880+0000 7f8f19ffd740 1 -- 172.21.15.1:0/795544561 >> 172.21.15.1:0/795544561 conn(0x5586e26114b0 msgr2=0x5586e2633190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.880+0000 7f8f19ffd740 1 -- 172.21.15.1:0/795544561 shutdown_connections 2024-03-06T07:23:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.880+0000 7f8f19ffd740 1 -- 172.21.15.1:0/795544561 wait complete. 2024-03-06T07:23:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.880+0000 7f8f19ffd740 20 client.7197 trim_cache size 0 max 16384 2024-03-06T07:23:53.889 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:23:53.889 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:23:53.889 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:23:53.889 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 5855, 2024-03-06T07:23:53.889 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:23:53.889 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:23:53.889 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:23:53.889 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 5.1218282630000003, 2024-03-06T07:23:53.890 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 82, 2024-03-06T07:23:53.890 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 96, 2024-03-06T07:23:53.890 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 96, 2024-03-06T07:23:53.890 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 17.255113119000001 2024-03-06T07:23:53.890 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:23:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.881+0000 7f8f1777a640 1 -- 172.21.15.1:0/3654747004 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ef406b500 msgr2=0x7f8ef406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.881+0000 7f8f1777a640 1 --2- 172.21.15.1:0/3654747004 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ef406b500 0x7f8ef406d9c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c000f60 tx=0x7f8f0c007d50 comp rx=0 tx=0).stop 2024-03-06T07:23:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.881+0000 7f8f1777a640 1 -- 172.21.15.1:0/3654747004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f10152650 msgr2=0x7f8f100c9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.881+0000 7f8f1777a640 1 --2- 172.21.15.1:0/3654747004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f10152650 0x7f8f100c9a90 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f8f0000ea10 tx=0x7f8f0000eed0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.881+0000 7f8f1777a640 1 -- 172.21.15.1:0/3654747004 shutdown_connections 2024-03-06T07:23:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.881+0000 7f8f1777a640 1 --2- 172.21.15.1:0/3654747004 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ef406b500 0x7f8ef406d9c0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.881+0000 7f8f1777a640 1 --2- 172.21.15.1:0/3654747004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f10153ec0 0x7f8f10156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.881+0000 7f8f1777a640 1 --2- 172.21.15.1:0/3654747004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f10152650 0x7f8f100c9a90 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.881+0000 7f8f1777a640 1 --2- 172.21.15.1:0/3654747004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f100bf9d0 0x7f8f101eb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.881+0000 7f8f1777a640 1 -- 172.21.15.1:0/3654747004 >> 172.21.15.1:0/3654747004 conn(0x7f8f10066ee0 msgr2=0x7f8f100c52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.881+0000 7f8f1777a640 1 -- 172.21.15.1:0/3654747004 shutdown_connections 2024-03-06T07:23:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:53.881+0000 7f8f1777a640 1 -- 172.21.15.1:0/3654747004 wait complete. 2024-03-06T07:23:53.893 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 82, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 7, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:23:41.077040+0000", "damaged": [], "data_pools": [ 16 ], "enabled": true, "epoch": 82, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5855": { "addr": "172.21.15.98:6836/1335402413", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 1335402413, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 1335402413, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5855, "incarnation": 79, "join_fscid": 7, "name": "b", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 96, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 15, "modified": "2024-03-06T07:23:51.762675+0000", "qdb_cluster": [ 5855 ], "qdb_leader": 5855, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5855 } } } ], "standbys": [ { "addr": "172.21.15.98:6837/3611156352", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 3611156352, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 3611156352, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 80, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6514, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6834/2118550222", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 2118550222, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 2118550222, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 74, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6981, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6835/543699854", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 543699854, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 543699854, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 74, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6984, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:23:53.894 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 7, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:23:53.894 INFO:tasks.cephfs.test_failover:current = [{'gid': 6984, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/543699854', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 543699854}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 543699854}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 74}, {'gid': 5855, 'name': 'b', 'rank': 0, 'incarnation': 79, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.98:6836/1335402413', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 1335402413}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1335402413}]}, 'join_fscid': 7, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6981, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6834/2118550222', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2118550222}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2118550222}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 74}, {'gid': 6514, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/3611156352', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3611156352}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3611156352}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 80}] 2024-03-06T07:23:53.895 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:23:53.895 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:53.895 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:53.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:53.981 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:53.981 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:23:53.981 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:53.999 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:23:53.999 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:23:53.999 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:54.037 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:23:54.037 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:54.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.136+0000 7fc2b77f7640 1 Processor -- start 2024-03-06T07:23:54.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.136+0000 7fc2b77f7640 1 -- start start 2024-03-06T07:23:54.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.137+0000 7fc2b77f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2b00c7560 0x7fc2b0155860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.137+0000 7fc2b77f7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2b0155da0 0x7fc2b00c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.137+0000 7fc2b77f7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2b0155080 con 0x7fc2b0152890 2024-03-06T07:23:54.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.137+0000 7fc2b77f7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b00c35b0 con 0x7fc2b0155da0 2024-03-06T07:23:54.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.137+0000 7fc2b77f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b00c3890 con 0x7fc2b00c7560 2024-03-06T07:23:54.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.137+0000 7fc2b5593640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc2b0152890 0x7fc2b0159030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36094/0 (socket says 172.21.15.1:36094) 2024-03-06T07:23:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.137+0000 7fc2b5593640 1 -- 172.21.15.1:0/1904468423 learned_addr learned my addr 172.21.15.1:0/1904468423 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.137+0000 7fc2b4d92640 1 --2- 172.21.15.1:0/1904468423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2b00c7560 0x7fc2b0155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.137+0000 7fc2b5d94640 1 --2- 172.21.15.1:0/1904468423 >> v2:172.21.15.98:3300/0 conn(0x7fc2b0155da0 0x7fc2b00c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.137+0000 7fc2a7fff640 1 -- 172.21.15.1:0/1904468423 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565266116 0 0) 0x7fc2b0155080 con 0x7fc2b0152890 2024-03-06T07:23:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.137+0000 7fc2a7fff640 1 -- 172.21.15.1:0/1904468423 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc29c003520 con 0x7fc2b0152890 2024-03-06T07:23:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.137+0000 7fc2b4d92640 1 -- 172.21.15.1:0/1904468423 >> v2:172.21.15.98:3300/0 conn(0x7fc2b0155da0 msgr2=0x7fc2b00c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.137+0000 7fc2b4d92640 1 --2- 172.21.15.1:0/1904468423 >> v2:172.21.15.98:3300/0 conn(0x7fc2b0155da0 0x7fc2b00c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.137+0000 7fc2b4d92640 1 -- 172.21.15.1:0/1904468423 >> v1:172.21.15.1:6789/0 conn(0x7fc2b0152890 legacy=0x7fc2b0159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.137+0000 7fc2b4d92640 1 -- 172.21.15.1:0/1904468423 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b00c3b70 con 0x7fc2b00c7560 2024-03-06T07:23:54.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.137+0000 7fc2b5d94640 1 --2- 172.21.15.1:0/1904468423 >> v2:172.21.15.98:3300/0 conn(0x7fc2b0155da0 0x7fc2b00c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:54.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.138+0000 7fc2b4d92640 1 --2- 172.21.15.1:0/1904468423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2b00c7560 0x7fc2b0155860 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00097d0 tx=0x7fc2a002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=762354163a432a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.138+0000 7fc2a7fff640 1 -- 172.21.15.1:0/1904468423 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0039070 con 0x7fc2b00c7560 2024-03-06T07:23:54.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.138+0000 7fc2a7fff640 1 -- 172.21.15.1:0/1904468423 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a00316f0 con 0x7fc2b00c7560 2024-03-06T07:23:54.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.138+0000 7fc2a7fff640 1 -- 172.21.15.1:0/1904468423 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0031aa0 con 0x7fc2b00c7560 2024-03-06T07:23:54.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.139+0000 7fc2b77f7640 1 -- 172.21.15.1:0/1904468423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2b00c7560 msgr2=0x7fc2b0155860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.139+0000 7fc2b77f7640 1 --2- 172.21.15.1:0/1904468423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2b00c7560 0x7fc2b0155860 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00097d0 tx=0x7fc2a002b790 comp rx=0 tx=0).stop 2024-03-06T07:23:54.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.139+0000 7fc2b77f7640 1 -- 172.21.15.1:0/1904468423 shutdown_connections 2024-03-06T07:23:54.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.139+0000 7fc2b77f7640 1 --2- 172.21.15.1:0/1904468423 >> v2:172.21.15.98:3300/0 conn(0x7fc2b0155da0 0x7fc2b00c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.139+0000 7fc2b77f7640 1 --2- 172.21.15.1:0/1904468423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2b00c7560 0x7fc2b0155860 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.139+0000 7fc2b77f7640 1 -- 172.21.15.1:0/1904468423 >> 172.21.15.1:0/1904468423 conn(0x7fc2b006b3e0 msgr2=0x7fc2b006b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.139+0000 7fc2b77f7640 1 -- 172.21.15.1:0/1904468423 shutdown_connections 2024-03-06T07:23:54.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.139+0000 7fc2b77f7640 1 -- 172.21.15.1:0/1904468423 wait complete. 2024-03-06T07:23:54.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.140+0000 7fc2b77f7640 1 Processor -- start 2024-03-06T07:23:54.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.140+0000 7fc2b77f7640 1 -- start start 2024-03-06T07:23:54.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.140+0000 7fc2b77f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2b00c7560 0x7fc2b01517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b77f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2b0152890 0x7fc2b01eaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b77f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2b0071590 0x7fc2b01eb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b77f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b00a81f0 con 0x7fc2b0071590 2024-03-06T07:23:54.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b77f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b00c3260 con 0x7fc2b0152890 2024-03-06T07:23:54.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b77f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b00a8370 con 0x7fc2b00c7560 2024-03-06T07:23:54.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b5d94640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2b0071590 0x7fc2b01eb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b5593640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2b00c7560 0x7fc2b01517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b4d92640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2b0152890 0x7fc2b01eaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b5d94640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2b0071590 0x7fc2b01eb420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37110/0 (socket says 172.21.15.1:37110) 2024-03-06T07:23:54.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b5d94640 1 -- 172.21.15.1:0/3154464414 learned_addr learned my addr 172.21.15.1:0/3154464414 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b5d94640 1 -- 172.21.15.1:0/3154464414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2b00c7560 msgr2=0x7fc2b01517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b5d94640 1 --2- 172.21.15.1:0/3154464414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2b00c7560 0x7fc2b01517a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b5d94640 1 -- 172.21.15.1:0/3154464414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2b0152890 msgr2=0x7fc2b01eaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b5d94640 1 --2- 172.21.15.1:0/3154464414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2b0152890 0x7fc2b01eaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b5d94640 1 -- 172.21.15.1:0/3154464414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b00aa5f0 con 0x7fc2b0071590 2024-03-06T07:23:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b5593640 1 --2- 172.21.15.1:0/3154464414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2b00c7560 0x7fc2b01517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b5d94640 1 --2- 172.21.15.1:0/3154464414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2b0071590 0x7fc2b01eb420 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800efc0 tx=0x7fc2a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2a5ffb640 1 -- 172.21.15.1:0/3154464414 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8018020 con 0x7fc2b0071590 2024-03-06T07:23:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2a5ffb640 1 -- 172.21.15.1:0/3154464414 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a8014480 con 0x7fc2b0071590 2024-03-06T07:23:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2b77f7640 1 -- 172.21.15.1:0/3154464414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b00aa8d0 con 0x7fc2b0071590 2024-03-06T07:23:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.141+0000 7fc2a5ffb640 1 -- 172.21.15.1:0/3154464414 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8014870 con 0x7fc2b0071590 2024-03-06T07:23:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.142+0000 7fc2b77f7640 1 -- 172.21.15.1:0/3154464414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b0151ce0 con 0x7fc2b0071590 2024-03-06T07:23:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.142+0000 7fc2b77f7640 1 -- 172.21.15.1:0/3154464414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2b0067fd0 con 0x7fc2b0071590 2024-03-06T07:23:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.143+0000 7fc2a5ffb640 1 -- 172.21.15.1:0/3154464414 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2a8005d10 con 0x7fc2b0071590 2024-03-06T07:23:54.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.143+0000 7fc2a5ffb640 1 --2- 172.21.15.1:0/3154464414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc29406b4d0 0x7fc29406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.143+0000 7fc2a5ffb640 1 -- 172.21.15.1:0/3154464414 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc2a808af50 con 0x7fc2b0071590 2024-03-06T07:23:54.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.143+0000 7fc2b5593640 1 --2- 172.21.15.1:0/3154464414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc29406b4d0 0x7fc29406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.144+0000 7fc2b5593640 1 --2- 172.21.15.1:0/3154464414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc29406b4d0 0x7fc29406d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac005880 tx=0x7fc2ac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.148+0000 7fc2a5ffb640 1 -- 172.21.15.1:0/3154464414 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2a8022030 con 0x7fc2b0071590 2024-03-06T07:23:54.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.368+0000 7fc2b77f7640 1 -- 172.21.15.1:0/3154464414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2b00c2d80 con 0x7fc2b0071590 2024-03-06T07:23:54.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.369+0000 7fc2a5ffb640 1 -- 172.21.15.1:0/3154464414 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4970 (secure 0 0 0) 0x7fc2a8010040 con 0x7fc2b0071590 2024-03-06T07:23:54.371 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:54.371 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":82,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3611156352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3611156352},{"type":"v1","addr":"172.21.15.98:6837","nonce":3611156352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6981,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/2118550222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2118550222},{"type":"v1","addr":"172.21.15.1:6834","nonce":2118550222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/543699854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":543699854},{"type":"v1","addr":"172.21.15.1:6835","nonce":543699854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:41.077040+0000","modified":"2024-03-06T07:23:51.762675+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5855},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5855":{"gid":5855,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":7,"addr":"172.21.15.98:6836/1335402413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1335402413},{"type":"v1","addr":"172.21.15.98:6836","nonce":1335402413}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5855,"qdb_cluster":[5855]},"id":7}]} 2024-03-06T07:23:54.371 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 82 2024-03-06T07:23:54.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.371+0000 7fc2b77f7640 1 -- 172.21.15.1:0/3154464414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc29406b4d0 msgr2=0x7fc29406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.371+0000 7fc2b77f7640 1 --2- 172.21.15.1:0/3154464414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc29406b4d0 0x7fc29406d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac005880 tx=0x7fc2ac009040 comp rx=0 tx=0).stop 2024-03-06T07:23:54.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.371+0000 7fc2b77f7640 1 -- 172.21.15.1:0/3154464414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2b0071590 msgr2=0x7fc2b01eb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.371+0000 7fc2b77f7640 1 --2- 172.21.15.1:0/3154464414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2b0071590 0x7fc2b01eb420 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800efc0 tx=0x7fc2a800c490 comp rx=0 tx=0).stop 2024-03-06T07:23:54.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.371+0000 7fc2b77f7640 1 -- 172.21.15.1:0/3154464414 shutdown_connections 2024-03-06T07:23:54.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.371+0000 7fc2b77f7640 1 --2- 172.21.15.1:0/3154464414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc29406b4d0 0x7fc29406d990 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.371+0000 7fc2b77f7640 1 --2- 172.21.15.1:0/3154464414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2b0071590 0x7fc2b01eb420 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.371+0000 7fc2b77f7640 1 --2- 172.21.15.1:0/3154464414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2b0152890 0x7fc2b01eaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.371+0000 7fc2b77f7640 1 --2- 172.21.15.1:0/3154464414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2b00c7560 0x7fc2b01517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.371+0000 7fc2b77f7640 1 -- 172.21.15.1:0/3154464414 >> 172.21.15.1:0/3154464414 conn(0x7fc2b006b3e0 msgr2=0x7fc2b0156240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.371+0000 7fc2b77f7640 1 -- 172.21.15.1:0/3154464414 shutdown_connections 2024-03-06T07:23:54.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.371+0000 7fc2b77f7640 1 -- 172.21.15.1:0/3154464414 wait complete. 2024-03-06T07:23:54.383 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:54.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.483+0000 7f9428302640 1 Processor -- start 2024-03-06T07:23:54.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.483+0000 7f9428302640 1 -- start start 2024-03-06T07:23:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.484+0000 7f9428302640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94200af6f0 0x7f94200c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.484+0000 7f9428302640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94200c7930 0x7f942015fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.484+0000 7f9428302640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94200c7180 con 0x7f9420152820 2024-03-06T07:23:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.484+0000 7f9428302640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9420155460 con 0x7f94200af6f0 2024-03-06T07:23:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.484+0000 7f9428302640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94200c6bc0 con 0x7f94200c7930 2024-03-06T07:23:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.484+0000 7f942589d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94200af6f0 0x7f94200c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.484+0000 7f942589d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94200af6f0 0x7f94200c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41326/0 (socket says 172.21.15.1:41326) 2024-03-06T07:23:54.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.484+0000 7f942589d640 1 -- 172.21.15.1:0/3902196912 learned_addr learned my addr 172.21.15.1:0/3902196912 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:54.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.484+0000 7f942689f640 1 --2- 172.21.15.1:0/3902196912 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94200c7930 0x7f942015fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.484+0000 7f942589d640 1 -- 172.21.15.1:0/3902196912 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94200c7930 msgr2=0x7f942015fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:54.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.484+0000 7f942589d640 1 --2- 172.21.15.1:0/3902196912 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94200c7930 0x7f942015fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.484+0000 7f942589d640 1 -- 172.21.15.1:0/3902196912 >> v1:172.21.15.98:6789/0 conn(0x7f9420152820 legacy=0x7f94200c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.484+0000 7f942589d640 1 -- 172.21.15.1:0/3902196912 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94200c6ea0 con 0x7f94200af6f0 2024-03-06T07:23:54.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.485+0000 7f942589d640 1 --2- 172.21.15.1:0/3902196912 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94200af6f0 0x7f94200c73f0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f9410009870 tx=0x7f941002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880aede76ae89282 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.485+0000 7f942509c640 1 -- 172.21.15.1:0/3902196912 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9410034070 con 0x7f94200af6f0 2024-03-06T07:23:54.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.485+0000 7f942509c640 1 -- 172.21.15.1:0/3902196912 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9410030840 con 0x7f94200af6f0 2024-03-06T07:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.485+0000 7f942509c640 1 -- 172.21.15.1:0/3902196912 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9410030c10 con 0x7f94200af6f0 2024-03-06T07:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.485+0000 7f9428302640 1 -- 172.21.15.1:0/3902196912 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94200af6f0 msgr2=0x7f94200c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.485+0000 7f9428302640 1 --2- 172.21.15.1:0/3902196912 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94200af6f0 0x7f94200c73f0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f9410009870 tx=0x7f941002b980 comp rx=0 tx=0).stop 2024-03-06T07:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.486+0000 7f9428302640 1 -- 172.21.15.1:0/3902196912 shutdown_connections 2024-03-06T07:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.486+0000 7f9428302640 1 --2- 172.21.15.1:0/3902196912 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94200c7930 0x7f942015fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.486+0000 7f9428302640 1 --2- 172.21.15.1:0/3902196912 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94200af6f0 0x7f94200c73f0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.486+0000 7f9428302640 1 -- 172.21.15.1:0/3902196912 >> 172.21.15.1:0/3902196912 conn(0x7f9420072720 msgr2=0x7f9420072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.486+0000 7f9428302640 1 -- 172.21.15.1:0/3902196912 shutdown_connections 2024-03-06T07:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.486+0000 7f9428302640 1 -- 172.21.15.1:0/3902196912 wait complete. 2024-03-06T07:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.487+0000 7f9428302640 1 Processor -- start 2024-03-06T07:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.487+0000 7f9428302640 1 -- start start 2024-03-06T07:23:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.487+0000 7f9428302640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94200af6f0 0x7f942014eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.487+0000 7f9428302640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94200c7930 0x7f94201ee640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.487+0000 7f9428302640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9420152820 0x7f94201eeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.487+0000 7f9428302640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9420152c00 con 0x7f94200af6f0 2024-03-06T07:23:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.487+0000 7f9428302640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94200c0840 con 0x7f9420152820 2024-03-06T07:23:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.487+0000 7f9428302640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9420152d80 con 0x7f94200c7930 2024-03-06T07:23:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.487+0000 7f942609e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94200af6f0 0x7f942014eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.487+0000 7f942609e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94200af6f0 0x7f942014eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41338/0 (socket says 172.21.15.1:41338) 2024-03-06T07:23:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.487+0000 7f942609e640 1 -- 172.21.15.1:0/398994482 learned_addr learned my addr 172.21.15.1:0/398994482 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.488+0000 7f942689f640 1 --2- 172.21.15.1:0/398994482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9420152820 0x7f94201eeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.488+0000 7f942589d640 1 --2- 172.21.15.1:0/398994482 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94200c7930 0x7f94201ee640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.488+0000 7f942609e640 1 -- 172.21.15.1:0/398994482 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94200c7930 msgr2=0x7f94201ee640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.488+0000 7f942609e640 1 --2- 172.21.15.1:0/398994482 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94200c7930 0x7f94201ee640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.488+0000 7f942609e640 1 -- 172.21.15.1:0/398994482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9420152820 msgr2=0x7f94201eeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.488+0000 7f942609e640 1 --2- 172.21.15.1:0/398994482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9420152820 0x7f94201eeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.488+0000 7f942609e640 1 -- 172.21.15.1:0/398994482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94201eb8e0 con 0x7f94200af6f0 2024-03-06T07:23:54.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.488+0000 7f942609e640 1 --2- 172.21.15.1:0/398994482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94200af6f0 0x7f942014eb30 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f941c00c3f0 tx=0x7f941c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.488+0000 7f9416ffd640 1 -- 172.21.15.1:0/398994482 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941c0052d0 con 0x7f94200af6f0 2024-03-06T07:23:54.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.488+0000 7f9416ffd640 1 -- 172.21.15.1:0/398994482 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941c005490 con 0x7f94200af6f0 2024-03-06T07:23:54.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.488+0000 7f9428302640 1 -- 172.21.15.1:0/398994482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9420156aa0 con 0x7f94200af6f0 2024-03-06T07:23:54.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.488+0000 7f9416ffd640 1 -- 172.21.15.1:0/398994482 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941c007260 con 0x7f94200af6f0 2024-03-06T07:23:54.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.488+0000 7f9428302640 1 -- 172.21.15.1:0/398994482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f942014f360 con 0x7f94200af6f0 2024-03-06T07:23:54.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.489+0000 7f9428302640 1 -- 172.21.15.1:0/398994482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9420156890 con 0x7f94200af6f0 2024-03-06T07:23:54.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.489+0000 7f9416ffd640 1 -- 172.21.15.1:0/398994482 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f941c007400 con 0x7f94200af6f0 2024-03-06T07:23:54.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.490+0000 7f9416ffd640 1 --2- 172.21.15.1:0/398994482 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f940406b4d0 0x7f940406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.490+0000 7f9416ffd640 1 -- 172.21.15.1:0/398994482 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f941c089740 con 0x7f94200af6f0 2024-03-06T07:23:54.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.490+0000 7f942589d640 1 --2- 172.21.15.1:0/398994482 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f940406b4d0 0x7f940406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.491+0000 7f942589d640 1 --2- 172.21.15.1:0/398994482 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f940406b4d0 0x7f940406d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f9410036000 tx=0x7f9410008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.494+0000 7f9416ffd640 1 -- 172.21.15.1:0/398994482 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f941c057e20 con 0x7f94200af6f0 2024-03-06T07:23:54.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.676+0000 7f9428302640 1 -- 172.21.15.1:0/398994482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94200686e0 con 0x7f94200af6f0 2024-03-06T07:23:54.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.677+0000 7f9416ffd640 1 -- 172.21.15.1:0/398994482 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16882 (secure 0 0 0) 0x7f941c05bec0 con 0x7f94200af6f0 2024-03-06T07:23:54.680 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:54.680 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":96,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:23:48.752552+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:39.342951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:40.043775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6837/3330683497":"2024-03-07T07:23:48.752518+0000","172.21.15.98:6835/3330683497":"2024-03-07T07:23:48.752518+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:54.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.679+0000 7f9428302640 1 -- 172.21.15.1:0/398994482 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f940406b4d0 msgr2=0x7f940406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.679+0000 7f9428302640 1 --2- 172.21.15.1:0/398994482 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f940406b4d0 0x7f940406d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f9410036000 tx=0x7f9410008030 comp rx=0 tx=0).stop 2024-03-06T07:23:54.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.679+0000 7f9428302640 1 -- 172.21.15.1:0/398994482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94200af6f0 msgr2=0x7f942014eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.679+0000 7f9428302640 1 --2- 172.21.15.1:0/398994482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94200af6f0 0x7f942014eb30 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f941c00c3f0 tx=0x7f941c00c8b0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.679+0000 7f9428302640 1 -- 172.21.15.1:0/398994482 shutdown_connections 2024-03-06T07:23:54.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.679+0000 7f9428302640 1 --2- 172.21.15.1:0/398994482 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f940406b4d0 0x7f940406d990 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.679+0000 7f9428302640 1 --2- 172.21.15.1:0/398994482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9420152820 0x7f94201eeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.679+0000 7f9428302640 1 --2- 172.21.15.1:0/398994482 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94200c7930 0x7f94201ee640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.679+0000 7f9428302640 1 --2- 172.21.15.1:0/398994482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94200af6f0 0x7f942014eb30 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.679+0000 7f9428302640 1 -- 172.21.15.1:0/398994482 >> 172.21.15.1:0/398994482 conn(0x7f9420072720 msgr2=0x7f9420151930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.680+0000 7f9428302640 1 -- 172.21.15.1:0/398994482 shutdown_connections 2024-03-06T07:23:54.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.680+0000 7f9428302640 1 -- 172.21.15.1:0/398994482 wait complete. 2024-03-06T07:23:54.693 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:39.342951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:40.043775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:54.693 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.796+0000 7f396f239640 1 Processor -- start 2024-03-06T07:23:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.796+0000 7f396f239640 1 -- start start 2024-03-06T07:23:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.797+0000 7f396f239640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f39680c2b30 0x7f39680724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.797+0000 7f396f239640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3968152b70 0x7f3968072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.797+0000 7f396f239640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3968151dd0 con 0x7f39680a5fa0 2024-03-06T07:23:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.797+0000 7f396f239640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39681526e0 con 0x7f39680c2b30 2024-03-06T07:23:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.797+0000 7f396f239640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3968156190 con 0x7f3968152b70 2024-03-06T07:23:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.797+0000 7f396d7d6640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f39680a5fa0 0x7f39680c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37298/0 (socket says 172.21.15.1:37298) 2024-03-06T07:23:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.797+0000 7f396d7d6640 1 -- 172.21.15.1:0/2340761505 learned_addr learned my addr 172.21.15.1:0/2340761505 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.797+0000 7f396cfd5640 1 --2- 172.21.15.1:0/2340761505 >> v2:172.21.15.98:3300/0 conn(0x7f39680c2b30 0x7f39680724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.797+0000 7f3967fff640 1 --2- 172.21.15.1:0/2340761505 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3968152b70 0x7f3968072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.798+0000 7f39677fe640 1 -- 172.21.15.1:0/2340761505 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681023256 0 0) 0x7f3968151dd0 con 0x7f39680a5fa0 2024-03-06T07:23:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.798+0000 7f39677fe640 1 -- 172.21.15.1:0/2340761505 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3958003520 con 0x7f39680a5fa0 2024-03-06T07:23:54.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.798+0000 7f396cfd5640 1 -- 172.21.15.1:0/2340761505 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3968152b70 msgr2=0x7f3968072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:54.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.798+0000 7f396cfd5640 1 --2- 172.21.15.1:0/2340761505 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3968152b70 0x7f3968072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.798+0000 7f396cfd5640 1 -- 172.21.15.1:0/2340761505 >> v1:172.21.15.1:6789/0 conn(0x7f39680a5fa0 legacy=0x7f39680c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.798+0000 7f396cfd5640 1 -- 172.21.15.1:0/2340761505 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3968152090 con 0x7f39680c2b30 2024-03-06T07:23:54.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.798+0000 7f3967fff640 1 --2- 172.21.15.1:0/2340761505 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3968152b70 0x7f3968072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.799+0000 7f396cfd5640 1 --2- 172.21.15.1:0/2340761505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39680c2b30 0x7f39680724c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f395c0097a0 tx=0x7f395c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c24cd0c0f90a7295 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.799+0000 7f39677fe640 1 -- 172.21.15.1:0/2340761505 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c035020 con 0x7f39680c2b30 2024-03-06T07:23:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.799+0000 7f39677fe640 1 -- 172.21.15.1:0/2340761505 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395c031830 con 0x7f39680c2b30 2024-03-06T07:23:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.799+0000 7f39677fe640 1 -- 172.21.15.1:0/2340761505 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c031be0 con 0x7f39680c2b30 2024-03-06T07:23:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.799+0000 7f396f239640 1 -- 172.21.15.1:0/2340761505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39680c2b30 msgr2=0x7f39680724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.799+0000 7f396f239640 1 --2- 172.21.15.1:0/2340761505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39680c2b30 0x7f39680724c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f395c0097a0 tx=0x7f395c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.799+0000 7f396f239640 1 -- 172.21.15.1:0/2340761505 shutdown_connections 2024-03-06T07:23:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.799+0000 7f396f239640 1 --2- 172.21.15.1:0/2340761505 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3968152b70 0x7f3968072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.799+0000 7f396f239640 1 --2- 172.21.15.1:0/2340761505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39680c2b30 0x7f39680724c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.799+0000 7f396f239640 1 -- 172.21.15.1:0/2340761505 >> 172.21.15.1:0/2340761505 conn(0x7f39680c6860 msgr2=0x7f39680c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.799+0000 7f396f239640 1 -- 172.21.15.1:0/2340761505 shutdown_connections 2024-03-06T07:23:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.799+0000 7f396f239640 1 -- 172.21.15.1:0/2340761505 wait complete. 2024-03-06T07:23:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.800+0000 7f396f239640 1 Processor -- start 2024-03-06T07:23:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.801+0000 7f396f239640 1 -- start start 2024-03-06T07:23:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.801+0000 7f396f239640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39680a5fa0 0x7f39680bc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.801+0000 7f396f239640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39680c2b30 0x7f39681ecb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.801+0000 7f396f239640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3968152b70 0x7f396814eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.801+0000 7f396f239640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3968155f00 con 0x7f39680c2b30 2024-03-06T07:23:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.801+0000 7f396f239640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3968155d80 con 0x7f3968152b70 2024-03-06T07:23:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.801+0000 7f396f239640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39680c3a00 con 0x7f39680a5fa0 2024-03-06T07:23:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.801+0000 7f3967fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39680c2b30 0x7f39681ecb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.801+0000 7f3967fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39680c2b30 0x7f39681ecb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41352/0 (socket says 172.21.15.1:41352) 2024-03-06T07:23:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.801+0000 7f3967fff640 1 -- 172.21.15.1:0/1789231951 learned_addr learned my addr 172.21.15.1:0/1789231951 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.801+0000 7f396cfd5640 1 --2- 172.21.15.1:0/1789231951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39680a5fa0 0x7f39680bc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.801+0000 7f396d7d6640 1 --2- 172.21.15.1:0/1789231951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3968152b70 0x7f396814eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.802+0000 7f3967fff640 1 -- 172.21.15.1:0/1789231951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39680a5fa0 msgr2=0x7f39680bc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.802+0000 7f3967fff640 1 --2- 172.21.15.1:0/1789231951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39680a5fa0 0x7f39680bc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.802+0000 7f3967fff640 1 -- 172.21.15.1:0/1789231951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3968152b70 msgr2=0x7f396814eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.802+0000 7f3967fff640 1 --2- 172.21.15.1:0/1789231951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3968152b70 0x7f396814eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.802+0000 7f3967fff640 1 -- 172.21.15.1:0/1789231951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39680bcf90 con 0x7f39680c2b30 2024-03-06T07:23:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.802+0000 7f3967fff640 1 --2- 172.21.15.1:0/1789231951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39680c2b30 0x7f39681ecb10 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f395400e960 tx=0x7f395400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.802+0000 7f39657fa640 1 -- 172.21.15.1:0/1789231951 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3954017070 con 0x7f39680c2b30 2024-03-06T07:23:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.802+0000 7f39657fa640 1 -- 172.21.15.1:0/1789231951 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395400ca00 con 0x7f39680c2b30 2024-03-06T07:23:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.802+0000 7f39657fa640 1 -- 172.21.15.1:0/1789231951 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395400cdd0 con 0x7f39680c2b30 2024-03-06T07:23:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.802+0000 7f396f239640 1 -- 172.21.15.1:0/1789231951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39681e8da0 con 0x7f39680c2b30 2024-03-06T07:23:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.802+0000 7f396f239640 1 -- 172.21.15.1:0/1789231951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396814f0c0 con 0x7f39680c2b30 2024-03-06T07:23:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.803+0000 7f396f239640 1 -- 172.21.15.1:0/1789231951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39681e8b90 con 0x7f39680c2b30 2024-03-06T07:23:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.804+0000 7f39657fa640 1 -- 172.21.15.1:0/1789231951 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f395401a070 con 0x7f39680c2b30 2024-03-06T07:23:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.804+0000 7f39657fa640 1 --2- 172.21.15.1:0/1789231951 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f394c06b4d0 0x7f394c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.804+0000 7f39657fa640 1 -- 172.21.15.1:0/1789231951 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f395408a510 con 0x7f39680c2b30 2024-03-06T07:23:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.804+0000 7f396cfd5640 1 --2- 172.21.15.1:0/1789231951 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f394c06b4d0 0x7f394c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.805+0000 7f396cfd5640 1 --2- 172.21.15.1:0/1789231951 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f394c06b4d0 0x7f394c06d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f395c036000 tx=0x7f395c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.809+0000 7f39657fa640 1 -- 172.21.15.1:0/1789231951 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3954058bf0 con 0x7f39680c2b30 2024-03-06T07:23:54.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.987+0000 7f396f239640 1 -- 172.21.15.1:0/1789231951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3968067fd0 con 0x7f39680c2b30 2024-03-06T07:23:54.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.988+0000 7f39657fa640 1 -- 172.21.15.1:0/1789231951 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) v1 ==== 93+0+10 (secure 0 0 0) 0x7f395405cc90 con 0x7f39680c2b30 2024-03-06T07:23:54.990 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:23:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.990+0000 7f396f239640 1 -- 172.21.15.1:0/1789231951 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f394c06b4d0 msgr2=0x7f394c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.990+0000 7f396f239640 1 --2- 172.21.15.1:0/1789231951 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f394c06b4d0 0x7f394c06d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f395c036000 tx=0x7f395c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.990+0000 7f396f239640 1 -- 172.21.15.1:0/1789231951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39680c2b30 msgr2=0x7f39681ecb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.990+0000 7f396f239640 1 --2- 172.21.15.1:0/1789231951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39680c2b30 0x7f39681ecb10 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f395400e960 tx=0x7f395400ee20 comp rx=0 tx=0).stop 2024-03-06T07:23:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.990+0000 7f396f239640 1 -- 172.21.15.1:0/1789231951 shutdown_connections 2024-03-06T07:23:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.990+0000 7f396f239640 1 --2- 172.21.15.1:0/1789231951 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f394c06b4d0 0x7f394c06d990 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.990+0000 7f396f239640 1 --2- 172.21.15.1:0/1789231951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3968152b70 0x7f396814eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.990+0000 7f396f239640 1 --2- 172.21.15.1:0/1789231951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39680c2b30 0x7f39681ecb10 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.990+0000 7f396f239640 1 --2- 172.21.15.1:0/1789231951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39680a5fa0 0x7f39680bc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.990+0000 7f396f239640 1 -- 172.21.15.1:0/1789231951 >> 172.21.15.1:0/1789231951 conn(0x7f39680c6860 msgr2=0x7f39680c2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.990+0000 7f396f239640 1 -- 172.21.15.1:0/1789231951 shutdown_connections 2024-03-06T07:23:54.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:54.990+0000 7f396f239640 1 -- 172.21.15.1:0/1789231951 wait complete. 2024-03-06T07:23:55.003 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:55.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.105+0000 7ff32bd65640 1 Processor -- start 2024-03-06T07:23:55.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.105+0000 7ff32bd65640 1 -- start start 2024-03-06T07:23:55.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.106+0000 7ff32bd65640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff324152650 0x7ff324155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.106+0000 7ff32bd65640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3240c2fc0 con 0x7ff3240c6b10 2024-03-06T07:23:55.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.106+0000 7ff32bd65640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff324158da0 con 0x7ff3240bf9d0 2024-03-06T07:23:55.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.106+0000 7ff32bd65640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff324155800 con 0x7ff324152650 2024-03-06T07:23:55.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.106+0000 7ff329b01640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff324152650 0x7ff324155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.106+0000 7ff329300640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff3240c6b10 0x7ff324158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37314/0 (socket says 172.21.15.1:37314) 2024-03-06T07:23:55.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.106+0000 7ff329300640 1 -- 172.21.15.1:0/3074357712 learned_addr learned my addr 172.21.15.1:0/3074357712 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:55.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.107+0000 7ff329b01640 1 -- 172.21.15.1:0/3074357712 >> v1:172.21.15.98:6789/0 conn(0x7ff3240bf9d0 legacy=0x7ff3240c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.107+0000 7ff329b01640 1 -- 172.21.15.1:0/3074357712 >> v1:172.21.15.1:6789/0 conn(0x7ff3240c6b10 legacy=0x7ff324158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.107+0000 7ff328aff640 1 -- 172.21.15.1:0/3074357712 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304859815 0 0) 0x7ff3240c2fc0 con 0x7ff3240c6b10 2024-03-06T07:23:55.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.107+0000 7ff329b01640 1 -- 172.21.15.1:0/3074357712 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3240c6690 con 0x7ff324152650 2024-03-06T07:23:55.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.107+0000 7ff329b01640 1 --2- 172.21.15.1:0/3074357712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff324152650 0x7ff324155c90 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009720 tx=0x7ff31c02b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e13bbd92ad6b501 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.107+0000 7ff328aff640 1 -- 172.21.15.1:0/3074357712 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c037070 con 0x7ff324152650 2024-03-06T07:23:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.108+0000 7ff328aff640 1 -- 172.21.15.1:0/3074357712 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31c030710 con 0x7ff324152650 2024-03-06T07:23:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.108+0000 7ff328aff640 1 -- 172.21.15.1:0/3074357712 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c030ae0 con 0x7ff324152650 2024-03-06T07:23:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.108+0000 7ff32bd65640 1 -- 172.21.15.1:0/3074357712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff324152650 msgr2=0x7ff324155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.108+0000 7ff32bd65640 1 --2- 172.21.15.1:0/3074357712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff324152650 0x7ff324155c90 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009720 tx=0x7ff31c02b6a0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.108+0000 7ff32bd65640 1 -- 172.21.15.1:0/3074357712 shutdown_connections 2024-03-06T07:23:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.108+0000 7ff32bd65640 1 --2- 172.21.15.1:0/3074357712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff324152650 0x7ff324155c90 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.108+0000 7ff32bd65640 1 -- 172.21.15.1:0/3074357712 >> 172.21.15.1:0/3074357712 conn(0x7ff324066ee0 msgr2=0x7ff3240672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.108+0000 7ff32bd65640 1 -- 172.21.15.1:0/3074357712 shutdown_connections 2024-03-06T07:23:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.108+0000 7ff32bd65640 1 -- 172.21.15.1:0/3074357712 wait complete. 2024-03-06T07:23:55.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.109+0000 7ff32bd65640 1 Processor -- start 2024-03-06T07:23:55.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.109+0000 7ff32bd65640 1 -- start start 2024-03-06T07:23:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.110+0000 7ff32bd65640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3240bf9d0 0x7ff324158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.110+0000 7ff32bd65640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff324152650 0x7ff324150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.110+0000 7ff32bd65640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3241511e0 0x7ff32414ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.110+0000 7ff32bd65640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3241551b0 con 0x7ff3241511e0 2024-03-06T07:23:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.110+0000 7ff32bd65640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240c6fa0 con 0x7ff3240bf9d0 2024-03-06T07:23:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.110+0000 7ff32bd65640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff324155490 con 0x7ff324152650 2024-03-06T07:23:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.110+0000 7ff32a302640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3241511e0 0x7ff32414ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.110+0000 7ff32a302640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3241511e0 0x7ff32414ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41380/0 (socket says 172.21.15.1:41380) 2024-03-06T07:23:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.110+0000 7ff32a302640 1 -- 172.21.15.1:0/920969883 learned_addr learned my addr 172.21.15.1:0/920969883 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.110+0000 7ff329b01640 1 --2- 172.21.15.1:0/920969883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3240bf9d0 0x7ff324158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.110+0000 7ff329300640 1 --2- 172.21.15.1:0/920969883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff324152650 0x7ff324150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.110+0000 7ff32a302640 1 -- 172.21.15.1:0/920969883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff324152650 msgr2=0x7ff324150ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.110+0000 7ff32a302640 1 --2- 172.21.15.1:0/920969883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff324152650 0x7ff324150ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.110+0000 7ff32a302640 1 -- 172.21.15.1:0/920969883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3240bf9d0 msgr2=0x7ff324158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.110+0000 7ff32a302640 1 --2- 172.21.15.1:0/920969883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3240bf9d0 0x7ff324158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.110+0000 7ff32a302640 1 -- 172.21.15.1:0/920969883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff324157220 con 0x7ff3241511e0 2024-03-06T07:23:55.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.111+0000 7ff32a302640 1 --2- 172.21.15.1:0/920969883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3241511e0 0x7ff32414ae20 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7ff31400c8c0 tx=0x7ff31400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.111+0000 7ff3227fc640 1 -- 172.21.15.1:0/920969883 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314017070 con 0x7ff3241511e0 2024-03-06T07:23:55.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.111+0000 7ff3227fc640 1 -- 172.21.15.1:0/920969883 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff314003da0 con 0x7ff3241511e0 2024-03-06T07:23:55.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.111+0000 7ff32bd65640 1 -- 172.21.15.1:0/920969883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3241eac90 con 0x7ff3241511e0 2024-03-06T07:23:55.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.111+0000 7ff3227fc640 1 -- 172.21.15.1:0/920969883 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314005290 con 0x7ff3241511e0 2024-03-06T07:23:55.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.111+0000 7ff32bd65640 1 -- 172.21.15.1:0/920969883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3241eecb0 con 0x7ff3241511e0 2024-03-06T07:23:55.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.112+0000 7ff32bd65640 1 -- 172.21.15.1:0/920969883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff324067fd0 con 0x7ff3241511e0 2024-03-06T07:23:55.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.112+0000 7ff3227fc640 1 -- 172.21.15.1:0/920969883 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff31401a070 con 0x7ff3241511e0 2024-03-06T07:23:55.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.113+0000 7ff3227fc640 1 --2- 172.21.15.1:0/920969883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff31006b500 0x7ff31006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.113+0000 7ff3227fc640 1 -- 172.21.15.1:0/920969883 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff314089a20 con 0x7ff3241511e0 2024-03-06T07:23:55.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.113+0000 7ff329b01640 1 --2- 172.21.15.1:0/920969883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff31006b500 0x7ff31006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.113+0000 7ff329b01640 1 --2- 172.21.15.1:0/920969883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff31006b500 0x7ff31006d9c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff31c034000 tx=0x7ff31c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.117+0000 7ff3227fc640 1 -- 172.21.15.1:0/920969883 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff314058070 con 0x7ff3241511e0 2024-03-06T07:23:55.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.296+0000 7ff32bd65640 1 -- 172.21.15.1:0/920969883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff324156c80 con 0x7ff3241511e0 2024-03-06T07:23:55.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.296+0000 7ff3227fc640 1 -- 172.21.15.1:0/920969883 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff31405c110 con 0x7ff3241511e0 2024-03-06T07:23:55.298 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:23:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.298+0000 7ff32bd65640 1 -- 172.21.15.1:0/920969883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff31006b500 msgr2=0x7ff31006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.298+0000 7ff32bd65640 1 --2- 172.21.15.1:0/920969883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff31006b500 0x7ff31006d9c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff31c034000 tx=0x7ff31c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.298+0000 7ff32bd65640 1 -- 172.21.15.1:0/920969883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3241511e0 msgr2=0x7ff32414ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.298+0000 7ff32bd65640 1 --2- 172.21.15.1:0/920969883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3241511e0 0x7ff32414ae20 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7ff31400c8c0 tx=0x7ff31400cd80 comp rx=0 tx=0).stop 2024-03-06T07:23:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.298+0000 7ff32bd65640 1 -- 172.21.15.1:0/920969883 shutdown_connections 2024-03-06T07:23:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.298+0000 7ff32bd65640 1 --2- 172.21.15.1:0/920969883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff31006b500 0x7ff31006d9c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.298+0000 7ff32bd65640 1 --2- 172.21.15.1:0/920969883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3241511e0 0x7ff32414ae20 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.298+0000 7ff32bd65640 1 --2- 172.21.15.1:0/920969883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff324152650 0x7ff324150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.298+0000 7ff32bd65640 1 --2- 172.21.15.1:0/920969883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3240bf9d0 0x7ff324158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.298+0000 7ff32bd65640 1 -- 172.21.15.1:0/920969883 >> 172.21.15.1:0/920969883 conn(0x7ff324066ee0 msgr2=0x7ff3240c0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.299+0000 7ff32bd65640 1 -- 172.21.15.1:0/920969883 shutdown_connections 2024-03-06T07:23:55.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.299+0000 7ff32bd65640 1 -- 172.21.15.1:0/920969883 wait complete. 2024-03-06T07:23:55.311 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:55.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.414+0000 7f469cf8c640 1 Processor -- start 2024-03-06T07:23:55.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.414+0000 7f469cf8c640 1 -- start start 2024-03-06T07:23:55.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.415+0000 7f469cf8c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4698150640 0x7f4698154300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.415+0000 7f469cf8c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f46980c75a0 0x7f46980c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.415+0000 7f469cf8c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46980c8120 con 0x7f46980a7f50 2024-03-06T07:23:55.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.415+0000 7f469cf8c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4698153e30 con 0x7f46980c75a0 2024-03-06T07:23:55.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.415+0000 7f469cf8c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46980c7b60 con 0x7f4698150640 2024-03-06T07:23:55.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.415+0000 7f4695d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f46980a7f50 0x7f4698153720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37324/0 (socket says 172.21.15.1:37324) 2024-03-06T07:23:55.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.415+0000 7f4695d9b640 1 -- 172.21.15.1:0/739412830 learned_addr learned my addr 172.21.15.1:0/739412830 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:55.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.415+0000 7f4696d9d640 1 --2- 172.21.15.1:0/739412830 >> v2:172.21.15.98:3300/0 conn(0x7f46980c75a0 0x7f46980c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.415+0000 7f469659c640 1 --2- 172.21.15.1:0/739412830 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4698150640 0x7f4698154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.416+0000 7f469559a640 1 -- 172.21.15.1:0/739412830 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148497172 0 0) 0x7f46980c8120 con 0x7f46980a7f50 2024-03-06T07:23:55.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.416+0000 7f4696d9d640 1 -- 172.21.15.1:0/739412830 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4698150640 msgr2=0x7f4698154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.416+0000 7f4696d9d640 1 --2- 172.21.15.1:0/739412830 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4698150640 0x7f4698154300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.416+0000 7f4696d9d640 1 -- 172.21.15.1:0/739412830 >> v1:172.21.15.1:6789/0 conn(0x7f46980a7f50 legacy=0x7f4698153720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.416+0000 7f4696d9d640 1 -- 172.21.15.1:0/739412830 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46980c7e40 con 0x7f46980c75a0 2024-03-06T07:23:55.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.416+0000 7f4696d9d640 1 --2- 172.21.15.1:0/739412830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46980c75a0 0x7f46980c8d60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4688009d90 tx=0x7f468802bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f77fe455fcd3d2de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.417+0000 7f469559a640 1 -- 172.21.15.1:0/739412830 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4688039070 con 0x7f46980c75a0 2024-03-06T07:23:55.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.417+0000 7f469559a640 1 -- 172.21.15.1:0/739412830 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468802cb30 con 0x7f46980c75a0 2024-03-06T07:23:55.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.417+0000 7f469559a640 1 -- 172.21.15.1:0/739412830 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4688034440 con 0x7f46980c75a0 2024-03-06T07:23:55.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.417+0000 7f469cf8c640 1 -- 172.21.15.1:0/739412830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46980c75a0 msgr2=0x7f46980c8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.417+0000 7f469cf8c640 1 --2- 172.21.15.1:0/739412830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46980c75a0 0x7f46980c8d60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4688009d90 tx=0x7f468802bd80 comp rx=0 tx=0).stop 2024-03-06T07:23:55.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.417+0000 7f469cf8c640 1 -- 172.21.15.1:0/739412830 shutdown_connections 2024-03-06T07:23:55.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.417+0000 7f469cf8c640 1 --2- 172.21.15.1:0/739412830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46980c75a0 0x7f46980c8d60 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.417+0000 7f469cf8c640 1 --2- 172.21.15.1:0/739412830 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4698150640 0x7f4698154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.417+0000 7f469cf8c640 1 -- 172.21.15.1:0/739412830 >> 172.21.15.1:0/739412830 conn(0x7f469806b3e0 msgr2=0x7f469806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.417+0000 7f469cf8c640 1 -- 172.21.15.1:0/739412830 shutdown_connections 2024-03-06T07:23:55.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.417+0000 7f469cf8c640 1 -- 172.21.15.1:0/739412830 wait complete. 2024-03-06T07:23:55.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.418+0000 7f469cf8c640 1 Processor -- start 2024-03-06T07:23:55.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.419+0000 7f469cf8c640 1 -- start start 2024-03-06T07:23:55.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.419+0000 7f469cf8c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46980a7f50 0x7f469814d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.419+0000 7f469cf8c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46980c75a0 0x7f46980abcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.419+0000 7f469cf8c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4698150640 0x7f46981eb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.419+0000 7f469cf8c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4698150a20 con 0x7f4698150640 2024-03-06T07:23:55.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.419+0000 7f469cf8c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4698153200 con 0x7f46980c75a0 2024-03-06T07:23:55.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.419+0000 7f469cf8c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4698071790 con 0x7f46980a7f50 2024-03-06T07:23:55.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.419+0000 7f4696d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4698150640 0x7f46981eb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.419+0000 7f4696d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4698150640 0x7f46981eb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41384/0 (socket says 172.21.15.1:41384) 2024-03-06T07:23:55.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.419+0000 7f4696d9d640 1 -- 172.21.15.1:0/244081711 learned_addr learned my addr 172.21.15.1:0/244081711 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:55.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.419+0000 7f469659c640 1 --2- 172.21.15.1:0/244081711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46980a7f50 0x7f469814d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.419+0000 7f4695d9b640 1 --2- 172.21.15.1:0/244081711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46980c75a0 0x7f46980abcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.420+0000 7f4696d9d640 1 -- 172.21.15.1:0/244081711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46980a7f50 msgr2=0x7f469814d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.420+0000 7f4696d9d640 1 --2- 172.21.15.1:0/244081711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46980a7f50 0x7f469814d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.420+0000 7f4696d9d640 1 -- 172.21.15.1:0/244081711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46980c75a0 msgr2=0x7f46980abcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.420+0000 7f4696d9d640 1 --2- 172.21.15.1:0/244081711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46980c75a0 0x7f46980abcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.420+0000 7f4696d9d640 1 -- 172.21.15.1:0/244081711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469814b740 con 0x7f4698150640 2024-03-06T07:23:55.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.420+0000 7f4696d9d640 1 --2- 172.21.15.1:0/244081711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4698150640 0x7f46981eb8f0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f4688009d90 tx=0x7f4688004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.420+0000 7f468effd640 1 -- 172.21.15.1:0/244081711 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4688039050 con 0x7f4698150640 2024-03-06T07:23:55.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.420+0000 7f468effd640 1 -- 172.21.15.1:0/244081711 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4688045070 con 0x7f4698150640 2024-03-06T07:23:55.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.420+0000 7f468effd640 1 -- 172.21.15.1:0/244081711 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4688009450 con 0x7f4698150640 2024-03-06T07:23:55.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.420+0000 7f469cf8c640 1 -- 172.21.15.1:0/244081711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469814c0a0 con 0x7f4698150640 2024-03-06T07:23:55.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.420+0000 7f469cf8c640 1 -- 172.21.15.1:0/244081711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f469814d850 con 0x7f4698150640 2024-03-06T07:23:55.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.421+0000 7f469cf8c640 1 -- 172.21.15.1:0/244081711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4698067fd0 con 0x7f4698150640 2024-03-06T07:23:55.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.422+0000 7f468effd640 1 -- 172.21.15.1:0/244081711 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f468803e850 con 0x7f4698150640 2024-03-06T07:23:55.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.422+0000 7f468effd640 1 --2- 172.21.15.1:0/244081711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f467806b4d0 0x7f467806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.422+0000 7f468effd640 1 -- 172.21.15.1:0/244081711 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f46880ad130 con 0x7f4698150640 2024-03-06T07:23:55.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.422+0000 7f469659c640 1 --2- 172.21.15.1:0/244081711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f467806b4d0 0x7f467806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.423+0000 7f469659c640 1 --2- 172.21.15.1:0/244081711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f467806b4d0 0x7f467806d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4698151200 tx=0x7f4690009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.426+0000 7f468effd640 1 -- 172.21.15.1:0/244081711 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f468807b930 con 0x7f4698150640 2024-03-06T07:23:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.607+0000 7f469cf8c640 1 -- 172.21.15.1:0/244081711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f469814f570 con 0x7f4698150640 2024-03-06T07:23:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.607+0000 7f468effd640 1 -- 172.21.15.1:0/244081711 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) v1 ==== 100+0+11 (secure 0 0 0) 0x7f468807f9d0 con 0x7f4698150640 2024-03-06T07:23:55.609 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:23:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.609+0000 7f469cf8c640 1 -- 172.21.15.1:0/244081711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f467806b4d0 msgr2=0x7f467806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.609+0000 7f469cf8c640 1 --2- 172.21.15.1:0/244081711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f467806b4d0 0x7f467806d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4698151200 tx=0x7f4690009190 comp rx=0 tx=0).stop 2024-03-06T07:23:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.609+0000 7f469cf8c640 1 -- 172.21.15.1:0/244081711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4698150640 msgr2=0x7f46981eb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.609+0000 7f469cf8c640 1 --2- 172.21.15.1:0/244081711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4698150640 0x7f46981eb8f0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f4688009d90 tx=0x7f4688004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.609+0000 7f469cf8c640 1 -- 172.21.15.1:0/244081711 shutdown_connections 2024-03-06T07:23:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.609+0000 7f469cf8c640 1 --2- 172.21.15.1:0/244081711 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f467806b4d0 0x7f467806d990 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.609+0000 7f469cf8c640 1 --2- 172.21.15.1:0/244081711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4698150640 0x7f46981eb8f0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.609+0000 7f469cf8c640 1 --2- 172.21.15.1:0/244081711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46980c75a0 0x7f46980abcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.609+0000 7f469cf8c640 1 --2- 172.21.15.1:0/244081711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46980a7f50 0x7f469814d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.609+0000 7f469cf8c640 1 -- 172.21.15.1:0/244081711 >> 172.21.15.1:0/244081711 conn(0x7f469806b3e0 msgr2=0x7f46980c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.609+0000 7f469cf8c640 1 -- 172.21.15.1:0/244081711 shutdown_connections 2024-03-06T07:23:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.609+0000 7f469cf8c640 1 -- 172.21.15.1:0/244081711 wait complete. 2024-03-06T07:23:55.622 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:55.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.720+0000 7f0a78a39640 1 Processor -- start 2024-03-06T07:23:55.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.721+0000 7f0a78a39640 1 -- start start 2024-03-06T07:23:55.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.721+0000 7f0a78a39640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a7414cdc0 0x7f0a74152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.721+0000 7f0a78a39640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0a7406b510 0x7f0a7406b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.721+0000 7f0a78a39640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a741569e0 con 0x7f0a74156500 2024-03-06T07:23:55.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.721+0000 7f0a78a39640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a740c2dd0 con 0x7f0a7406b510 2024-03-06T07:23:55.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.721+0000 7f0a78a39640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a74156cc0 con 0x7f0a7414cdc0 2024-03-06T07:23:55.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.722+0000 7f0a72d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0a7406b510 0x7f0a7406b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.722+0000 7f0a72d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0a7406b510 0x7f0a7406b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41396/0 (socket says 172.21.15.1:41396) 2024-03-06T07:23:55.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.722+0000 7f0a7259c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a7414cdc0 0x7f0a74152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.722+0000 7f0a72d9d640 1 -- 172.21.15.1:0/4150715569 learned_addr learned my addr 172.21.15.1:0/4150715569 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:55.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.722+0000 7f0a71d9b640 1 --1- 172.21.15.1:0/4150715569 >> v1:172.21.15.1:6789/0 conn(0x7f0a74156500 0x7f0a740c2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37338/0 (socket says 172.21.15.1:37338) 2024-03-06T07:23:55.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.722+0000 7f0a72d9d640 1 -- 172.21.15.1:0/4150715569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a7414cdc0 msgr2=0x7f0a74152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.722+0000 7f0a72d9d640 1 --2- 172.21.15.1:0/4150715569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a7414cdc0 0x7f0a74152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.722+0000 7f0a72d9d640 1 -- 172.21.15.1:0/4150715569 >> v1:172.21.15.1:6789/0 conn(0x7f0a74156500 legacy=0x7f0a740c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.722+0000 7f0a72d9d640 1 -- 172.21.15.1:0/4150715569 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a74156fa0 con 0x7f0a7406b510 2024-03-06T07:23:55.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.722+0000 7f0a7259c640 1 --2- 172.21.15.1:0/4150715569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a7414cdc0 0x7f0a74152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:55.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.723+0000 7f0a72d9d640 1 --2- 172.21.15.1:0/4150715569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a7406b510 0x7f0a7406b980 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009e50 tx=0x7f0a6402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52386a76a2f17da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.723+0000 7f0a7159a640 1 -- 172.21.15.1:0/4150715569 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6402cb00 con 0x7f0a7406b510 2024-03-06T07:23:55.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.723+0000 7f0a7159a640 1 -- 172.21.15.1:0/4150715569 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6402cca0 con 0x7f0a7406b510 2024-03-06T07:23:55.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.723+0000 7f0a7159a640 1 -- 172.21.15.1:0/4150715569 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64034610 con 0x7f0a7406b510 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.723+0000 7f0a78a39640 1 -- 172.21.15.1:0/4150715569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a7406b510 msgr2=0x7f0a7406b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.723+0000 7f0a78a39640 1 --2- 172.21.15.1:0/4150715569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a7406b510 0x7f0a7406b980 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009e50 tx=0x7f0a6402bf60 comp rx=0 tx=0).stop 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.723+0000 7f0a78a39640 1 -- 172.21.15.1:0/4150715569 shutdown_connections 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.723+0000 7f0a78a39640 1 --2- 172.21.15.1:0/4150715569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a7406b510 0x7f0a7406b980 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.723+0000 7f0a78a39640 1 --2- 172.21.15.1:0/4150715569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a7414cdc0 0x7f0a74152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.723+0000 7f0a78a39640 1 -- 172.21.15.1:0/4150715569 >> 172.21.15.1:0/4150715569 conn(0x7f0a740a9700 msgr2=0x7f0a740a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.723+0000 7f0a78a39640 1 -- 172.21.15.1:0/4150715569 shutdown_connections 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.723+0000 7f0a78a39640 1 -- 172.21.15.1:0/4150715569 wait complete. 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.725+0000 7f0a78a39640 1 Processor -- start 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.725+0000 7f0a78a39640 1 -- start start 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.725+0000 7f0a78a39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a7406b510 0x7f0a7414a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.725+0000 7f0a78a39640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a7414cdc0 0x7f0a741de780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.725+0000 7f0a78a39640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a74156500 0x7f0a740ab170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.725+0000 7f0a78a39640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a740c5a70 con 0x7f0a74156500 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.725+0000 7f0a78a39640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7406bec0 con 0x7f0a7406b510 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.725+0000 7f0a78a39640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a740c65e0 con 0x7f0a7414cdc0 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.726+0000 7f0a72d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a74156500 0x7f0a740ab170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.726+0000 7f0a72d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a74156500 0x7f0a740ab170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41408/0 (socket says 172.21.15.1:41408) 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.726+0000 7f0a72d9d640 1 -- 172.21.15.1:0/1689901708 learned_addr learned my addr 172.21.15.1:0/1689901708 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.726+0000 7f0a7259c640 1 --2- 172.21.15.1:0/1689901708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a7406b510 0x7f0a7414a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.726+0000 7f0a71d9b640 1 --2- 172.21.15.1:0/1689901708 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a7414cdc0 0x7f0a741de780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.726+0000 7f0a72d9d640 1 -- 172.21.15.1:0/1689901708 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a7414cdc0 msgr2=0x7f0a741de780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.726+0000 7f0a72d9d640 1 --2- 172.21.15.1:0/1689901708 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a7414cdc0 0x7f0a741de780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.726+0000 7f0a72d9d640 1 -- 172.21.15.1:0/1689901708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a7406b510 msgr2=0x7f0a7414a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.726+0000 7f0a72d9d640 1 --2- 172.21.15.1:0/1689901708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a7406b510 0x7f0a7414a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.726+0000 7f0a72d9d640 1 -- 172.21.15.1:0/1689901708 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a741ddd80 con 0x7f0a74156500 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.726+0000 7f0a72d9d640 1 --2- 172.21.15.1:0/1689901708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a74156500 0x7f0a740ab170 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f0a64036040 tx=0x7f0a64004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.726+0000 7f0a6affd640 1 -- 172.21.15.1:0/1689901708 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64043020 con 0x7f0a74156500 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.726+0000 7f0a6affd640 1 -- 172.21.15.1:0/1689901708 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6402ce50 con 0x7f0a74156500 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.726+0000 7f0a6affd640 1 -- 172.21.15.1:0/1689901708 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64009510 con 0x7f0a74156500 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.726+0000 7f0a78a39640 1 -- 172.21.15.1:0/1689901708 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a741dd7c0 con 0x7f0a74156500 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.726+0000 7f0a78a39640 1 -- 172.21.15.1:0/1689901708 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a741decc0 con 0x7f0a74156500 2024-03-06T07:23:55.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.727+0000 7f0a78a39640 1 -- 172.21.15.1:0/1689901708 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a74067fd0 con 0x7f0a74156500 2024-03-06T07:23:55.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.728+0000 7f0a6affd640 1 -- 172.21.15.1:0/1689901708 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a64046020 con 0x7f0a74156500 2024-03-06T07:23:55.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.728+0000 7f0a6affd640 1 --2- 172.21.15.1:0/1689901708 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a5406b4d0 0x7f0a5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.728+0000 7f0a6affd640 1 -- 172.21.15.1:0/1689901708 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f0a640ac780 con 0x7f0a74156500 2024-03-06T07:23:55.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.728+0000 7f0a7259c640 1 --2- 172.21.15.1:0/1689901708 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a5406b4d0 0x7f0a5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.729+0000 7f0a7259c640 1 --2- 172.21.15.1:0/1689901708 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a5406b4d0 0x7f0a5406d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009560 tx=0x7f0a6c012040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.732+0000 7f0a6affd640 1 -- 172.21.15.1:0/1689901708 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a6404d030 con 0x7f0a74156500 2024-03-06T07:23:55.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.953+0000 7f0a78a39640 1 -- 172.21.15.1:0/1689901708 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a740c5850 con 0x7f0a74156500 2024-03-06T07:23:55.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.953+0000 7f0a6affd640 1 -- 172.21.15.1:0/1689901708 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4970 (secure 0 0 0) 0x7f0a6407fe20 con 0x7f0a74156500 2024-03-06T07:23:55.956 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:55.956 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":82,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3611156352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3611156352},{"type":"v1","addr":"172.21.15.98:6837","nonce":3611156352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6981,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/2118550222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2118550222},{"type":"v1","addr":"172.21.15.1:6834","nonce":2118550222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/543699854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":543699854},{"type":"v1","addr":"172.21.15.1:6835","nonce":543699854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:41.077040+0000","modified":"2024-03-06T07:23:51.762675+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5855},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5855":{"gid":5855,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":7,"addr":"172.21.15.98:6836/1335402413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1335402413},{"type":"v1","addr":"172.21.15.98:6836","nonce":1335402413}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5855,"qdb_cluster":[5855]},"id":7}]} 2024-03-06T07:23:55.956 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 82 2024-03-06T07:23:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.955+0000 7f0a78a39640 1 -- 172.21.15.1:0/1689901708 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a5406b4d0 msgr2=0x7f0a5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.955+0000 7f0a78a39640 1 --2- 172.21.15.1:0/1689901708 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a5406b4d0 0x7f0a5406d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009560 tx=0x7f0a6c012040 comp rx=0 tx=0).stop 2024-03-06T07:23:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.956+0000 7f0a78a39640 1 -- 172.21.15.1:0/1689901708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a74156500 msgr2=0x7f0a740ab170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.956+0000 7f0a78a39640 1 --2- 172.21.15.1:0/1689901708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a74156500 0x7f0a740ab170 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f0a64036040 tx=0x7f0a64004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.956+0000 7f0a78a39640 1 -- 172.21.15.1:0/1689901708 shutdown_connections 2024-03-06T07:23:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.956+0000 7f0a78a39640 1 --2- 172.21.15.1:0/1689901708 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a5406b4d0 0x7f0a5406d990 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.956+0000 7f0a78a39640 1 --2- 172.21.15.1:0/1689901708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a74156500 0x7f0a740ab170 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.956+0000 7f0a78a39640 1 --2- 172.21.15.1:0/1689901708 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a7414cdc0 0x7f0a741de780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.956+0000 7f0a78a39640 1 --2- 172.21.15.1:0/1689901708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a7406b510 0x7f0a7414a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.956+0000 7f0a78a39640 1 -- 172.21.15.1:0/1689901708 >> 172.21.15.1:0/1689901708 conn(0x7f0a740a9700 msgr2=0x7f0a740a8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.956+0000 7f0a78a39640 1 -- 172.21.15.1:0/1689901708 shutdown_connections 2024-03-06T07:23:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:55.956+0000 7f0a78a39640 1 -- 172.21.15.1:0/1689901708 wait complete. 2024-03-06T07:23:55.968 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:56.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.068+0000 7ff33ad93640 1 Processor -- start 2024-03-06T07:23:56.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.068+0000 7ff33ad93640 1 -- start start 2024-03-06T07:23:56.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.069+0000 7ff33ad93640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff33406d060 0x7ff3341558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.069+0000 7ff33ad93640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3340af640 0x7ff334155df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.069+0000 7ff33ad93640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3340c71c0 con 0x7ff3340c3eb0 2024-03-06T07:23:56.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.069+0000 7ff33ad93640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3340c59b0 con 0x7ff33406d060 2024-03-06T07:23:56.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.069+0000 7ff33ad93640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3340c4c00 con 0x7ff3340af640 2024-03-06T07:23:56.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.069+0000 7ff338b2f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff3340c3eb0 0x7ff3340c7400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37348/0 (socket says 172.21.15.1:37348) 2024-03-06T07:23:56.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.069+0000 7ff338b2f640 1 -- 172.21.15.1:0/3887208024 learned_addr learned my addr 172.21.15.1:0/3887208024 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:56.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.069+0000 7ff339330640 1 --2- 172.21.15.1:0/3887208024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3340af640 0x7ff334155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.069+0000 7ff333fff640 1 --2- 172.21.15.1:0/3887208024 >> v2:172.21.15.98:3300/0 conn(0x7ff33406d060 0x7ff3341558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.069+0000 7ff3337fe640 1 -- 172.21.15.1:0/3887208024 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654769488 0 0) 0x7ff3340c71c0 con 0x7ff3340c3eb0 2024-03-06T07:23:56.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.069+0000 7ff3337fe640 1 -- 172.21.15.1:0/3887208024 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff324003520 con 0x7ff3340c3eb0 2024-03-06T07:23:56.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.069+0000 7ff333fff640 1 -- 172.21.15.1:0/3887208024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3340af640 msgr2=0x7ff334155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.069+0000 7ff333fff640 1 --2- 172.21.15.1:0/3887208024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3340af640 0x7ff334155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.069+0000 7ff333fff640 1 -- 172.21.15.1:0/3887208024 >> v1:172.21.15.1:6789/0 conn(0x7ff3340c3eb0 legacy=0x7ff3340c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.069+0000 7ff333fff640 1 -- 172.21.15.1:0/3887208024 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3340c4e50 con 0x7ff33406d060 2024-03-06T07:23:56.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.069+0000 7ff339330640 1 --2- 172.21.15.1:0/3887208024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3340af640 0x7ff334155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.070+0000 7ff333fff640 1 --2- 172.21.15.1:0/3887208024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33406d060 0x7ff3341558b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff3200097a0 tx=0x7ff32002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5135c48c7d201003 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.070+0000 7ff3337fe640 1 -- 172.21.15.1:0/3887208024 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff320038070 con 0x7ff33406d060 2024-03-06T07:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.070+0000 7ff3337fe640 1 -- 172.21.15.1:0/3887208024 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff320031700 con 0x7ff33406d060 2024-03-06T07:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.070+0000 7ff3337fe640 1 -- 172.21.15.1:0/3887208024 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff320031ad0 con 0x7ff33406d060 2024-03-06T07:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.071+0000 7ff33ad93640 1 -- 172.21.15.1:0/3887208024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33406d060 msgr2=0x7ff3341558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.071+0000 7ff33ad93640 1 --2- 172.21.15.1:0/3887208024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33406d060 0x7ff3341558b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff3200097a0 tx=0x7ff32002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.071+0000 7ff33ad93640 1 -- 172.21.15.1:0/3887208024 shutdown_connections 2024-03-06T07:23:56.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.071+0000 7ff33ad93640 1 --2- 172.21.15.1:0/3887208024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3340af640 0x7ff334155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.071+0000 7ff33ad93640 1 --2- 172.21.15.1:0/3887208024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33406d060 0x7ff3341558b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.071+0000 7ff33ad93640 1 -- 172.21.15.1:0/3887208024 >> 172.21.15.1:0/3887208024 conn(0x7ff334066d00 msgr2=0x7ff334067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.071+0000 7ff33ad93640 1 -- 172.21.15.1:0/3887208024 shutdown_connections 2024-03-06T07:23:56.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.071+0000 7ff33ad93640 1 -- 172.21.15.1:0/3887208024 wait complete. 2024-03-06T07:23:56.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.072+0000 7ff33ad93640 1 Processor -- start 2024-03-06T07:23:56.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.072+0000 7ff33ad93640 1 -- start start 2024-03-06T07:23:56.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.073+0000 7ff33ad93640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33406d060 0x7ff334157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.073+0000 7ff33ad93640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3340af640 0x7ff334158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.073+0000 7ff33ad93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3340c3eb0 0x7ff334156470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.073+0000 7ff33ad93640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3340c7b10 con 0x7ff3340c3eb0 2024-03-06T07:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.073+0000 7ff33ad93640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff334155420 con 0x7ff33406d060 2024-03-06T07:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.073+0000 7ff33ad93640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3340c7c90 con 0x7ff3340af640 2024-03-06T07:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.073+0000 7ff339330640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3340c3eb0 0x7ff334156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.073+0000 7ff339330640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3340c3eb0 0x7ff334156470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41416/0 (socket says 172.21.15.1:41416) 2024-03-06T07:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.073+0000 7ff339330640 1 -- 172.21.15.1:0/1717321939 learned_addr learned my addr 172.21.15.1:0/1717321939 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.073+0000 7ff338b2f640 1 --2- 172.21.15.1:0/1717321939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33406d060 0x7ff334157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.073+0000 7ff333fff640 1 --2- 172.21.15.1:0/1717321939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3340af640 0x7ff334158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.073+0000 7ff339330640 1 -- 172.21.15.1:0/1717321939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3340af640 msgr2=0x7ff334158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.073+0000 7ff339330640 1 --2- 172.21.15.1:0/1717321939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3340af640 0x7ff334158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.073+0000 7ff339330640 1 -- 172.21.15.1:0/1717321939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33406d060 msgr2=0x7ff334157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.073+0000 7ff339330640 1 --2- 172.21.15.1:0/1717321939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33406d060 0x7ff334157b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.073+0000 7ff339330640 1 -- 172.21.15.1:0/1717321939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33414ba90 con 0x7ff3340c3eb0 2024-03-06T07:23:56.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.074+0000 7ff339330640 1 --2- 172.21.15.1:0/1717321939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3340c3eb0 0x7ff334156470 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7ff32c00efc0 tx=0x7ff32c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.074+0000 7ff3317fa640 1 -- 172.21.15.1:0/1717321939 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c018030 con 0x7ff3340c3eb0 2024-03-06T07:23:56.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.074+0000 7ff3317fa640 1 -- 172.21.15.1:0/1717321939 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32c00cdf0 con 0x7ff3340c3eb0 2024-03-06T07:23:56.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.074+0000 7ff3317fa640 1 -- 172.21.15.1:0/1717321939 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c003bb0 con 0x7ff3340c3eb0 2024-03-06T07:23:56.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.074+0000 7ff33ad93640 1 -- 172.21.15.1:0/1717321939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3341ead90 con 0x7ff3340c3eb0 2024-03-06T07:23:56.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.074+0000 7ff33ad93640 1 -- 172.21.15.1:0/1717321939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3341eb350 con 0x7ff3340c3eb0 2024-03-06T07:23:56.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.075+0000 7ff33ad93640 1 -- 172.21.15.1:0/1717321939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff334067fd0 con 0x7ff3340c3eb0 2024-03-06T07:23:56.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.075+0000 7ff3317fa640 1 -- 172.21.15.1:0/1717321939 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff32c01b070 con 0x7ff3340c3eb0 2024-03-06T07:23:56.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.075+0000 7ff3317fa640 1 --2- 172.21.15.1:0/1717321939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff31806b500 0x7ff31806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.076+0000 7ff3317fa640 1 -- 172.21.15.1:0/1717321939 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff32c08ab90 con 0x7ff3340c3eb0 2024-03-06T07:23:56.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.076+0000 7ff338b2f640 1 --2- 172.21.15.1:0/1717321939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff31806b500 0x7ff31806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.076+0000 7ff338b2f640 1 --2- 172.21.15.1:0/1717321939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff31806b500 0x7ff31806d9c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff328005880 tx=0x7ff328009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.080+0000 7ff3317fa640 1 -- 172.21.15.1:0/1717321939 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff32c010040 con 0x7ff3340c3eb0 2024-03-06T07:23:56.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.260+0000 7ff33ad93640 1 -- 172.21.15.1:0/1717321939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3340c5b30 con 0x7ff3340c3eb0 2024-03-06T07:23:56.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.261+0000 7ff3317fa640 1 -- 172.21.15.1:0/1717321939 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16882 (secure 0 0 0) 0x7ff32c059270 con 0x7ff3340c3eb0 2024-03-06T07:23:56.263 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:56.263 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":96,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:23:48.752552+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:39.342951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:40.043775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6837/3330683497":"2024-03-07T07:23:48.752518+0000","172.21.15.98:6835/3330683497":"2024-03-07T07:23:48.752518+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:56.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.263+0000 7ff33ad93640 1 -- 172.21.15.1:0/1717321939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff31806b500 msgr2=0x7ff31806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.263+0000 7ff33ad93640 1 --2- 172.21.15.1:0/1717321939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff31806b500 0x7ff31806d9c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff328005880 tx=0x7ff328009040 comp rx=0 tx=0).stop 2024-03-06T07:23:56.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.263+0000 7ff33ad93640 1 -- 172.21.15.1:0/1717321939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3340c3eb0 msgr2=0x7ff334156470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.263+0000 7ff33ad93640 1 --2- 172.21.15.1:0/1717321939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3340c3eb0 0x7ff334156470 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7ff32c00efc0 tx=0x7ff32c00c490 comp rx=0 tx=0).stop 2024-03-06T07:23:56.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.263+0000 7ff33ad93640 1 -- 172.21.15.1:0/1717321939 shutdown_connections 2024-03-06T07:23:56.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.263+0000 7ff33ad93640 1 --2- 172.21.15.1:0/1717321939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff31806b500 0x7ff31806d9c0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.263+0000 7ff33ad93640 1 --2- 172.21.15.1:0/1717321939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3340c3eb0 0x7ff334156470 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.263+0000 7ff33ad93640 1 --2- 172.21.15.1:0/1717321939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff3340af640 0x7ff334158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.263+0000 7ff33ad93640 1 --2- 172.21.15.1:0/1717321939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33406d060 0x7ff334157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.263+0000 7ff33ad93640 1 -- 172.21.15.1:0/1717321939 >> 172.21.15.1:0/1717321939 conn(0x7ff334066d00 msgr2=0x7ff3340c5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.263+0000 7ff33ad93640 1 -- 172.21.15.1:0/1717321939 shutdown_connections 2024-03-06T07:23:56.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.263+0000 7ff33ad93640 1 -- 172.21.15.1:0/1717321939 wait complete. 2024-03-06T07:23:56.276 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:23:56.276 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:56.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.376+0000 7fd7bd050640 1 Processor -- start 2024-03-06T07:23:56.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.376+0000 7fd7bd050640 1 -- start start 2024-03-06T07:23:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.376+0000 7fd7bd050640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7b80c4dd0 0x7fd7b80c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.376+0000 7fd7bd050640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b80c5d80 0x7fd7b81535b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.376+0000 7fd7bd050640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b81566e0 con 0x7fd7b80c79c0 2024-03-06T07:23:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.376+0000 7fd7bd050640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7b8156490 con 0x7fd7b80c4dd0 2024-03-06T07:23:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.376+0000 7fd7bd050640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b8157010 con 0x7fd7b80c5d80 2024-03-06T07:23:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.377+0000 7fd7b6d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7b80c4dd0 0x7fd7b80c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.377+0000 7fd7b6d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7b80c4dd0 0x7fd7b80c5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41424/0 (socket says 172.21.15.1:41424) 2024-03-06T07:23:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.377+0000 7fd7b659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd7b80c79c0 0x7fd7b8152ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37354/0 (socket says 172.21.15.1:37354) 2024-03-06T07:23:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.377+0000 7fd7b6d9d640 1 -- 172.21.15.1:0/272285577 learned_addr learned my addr 172.21.15.1:0/272285577 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:56.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.377+0000 7fd7b759e640 1 --2- 172.21.15.1:0/272285577 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b80c5d80 0x7fd7b81535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.377+0000 7fd7b6d9d640 1 -- 172.21.15.1:0/272285577 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b80c5d80 msgr2=0x7fd7b81535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.377+0000 7fd7b6d9d640 1 --2- 172.21.15.1:0/272285577 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b80c5d80 0x7fd7b81535b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.377+0000 7fd7b6d9d640 1 -- 172.21.15.1:0/272285577 >> v1:172.21.15.1:6789/0 conn(0x7fd7b80c79c0 legacy=0x7fd7b8152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.377+0000 7fd7b6d9d640 1 -- 172.21.15.1:0/272285577 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b81569c0 con 0x7fd7b80c4dd0 2024-03-06T07:23:56.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.377+0000 7fd7b5d9b640 1 -- 172.21.15.1:0/272285577 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450803115 0 0) 0x7fd7b81566e0 con 0x7fd7b80c79c0 2024-03-06T07:23:56.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.378+0000 7fd7b6d9d640 1 --2- 172.21.15.1:0/272285577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7b80c4dd0 0x7fd7b80c5840 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8009870 tx=0x7fd7a802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac72326448e50c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.378+0000 7fd7b5d9b640 1 -- 172.21.15.1:0/272285577 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a80317c0 con 0x7fd7b80c4dd0 2024-03-06T07:23:56.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.378+0000 7fd7b5d9b640 1 -- 172.21.15.1:0/272285577 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a8037070 con 0x7fd7b80c4dd0 2024-03-06T07:23:56.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.378+0000 7fd7b5d9b640 1 -- 172.21.15.1:0/272285577 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8031b80 con 0x7fd7b80c4dd0 2024-03-06T07:23:56.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.378+0000 7fd7bd050640 1 -- 172.21.15.1:0/272285577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7b80c4dd0 msgr2=0x7fd7b80c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.378+0000 7fd7bd050640 1 --2- 172.21.15.1:0/272285577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7b80c4dd0 0x7fd7b80c5840 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8009870 tx=0x7fd7a802b860 comp rx=0 tx=0).stop 2024-03-06T07:23:56.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.378+0000 7fd7bd050640 1 -- 172.21.15.1:0/272285577 shutdown_connections 2024-03-06T07:23:56.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.378+0000 7fd7bd050640 1 --2- 172.21.15.1:0/272285577 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b80c5d80 0x7fd7b81535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.378+0000 7fd7bd050640 1 --2- 172.21.15.1:0/272285577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7b80c4dd0 0x7fd7b80c5840 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.378+0000 7fd7bd050640 1 -- 172.21.15.1:0/272285577 >> 172.21.15.1:0/272285577 conn(0x7fd7b806b410 msgr2=0x7fd7b806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.378+0000 7fd7bd050640 1 -- 172.21.15.1:0/272285577 shutdown_connections 2024-03-06T07:23:56.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.378+0000 7fd7bd050640 1 -- 172.21.15.1:0/272285577 wait complete. 2024-03-06T07:23:56.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.380+0000 7fd7bd050640 1 Processor -- start 2024-03-06T07:23:56.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.380+0000 7fd7bd050640 1 -- start start 2024-03-06T07:23:56.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.380+0000 7fd7bd050640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7b80c4dd0 0x7fd7b80ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.380+0000 7fd7bd050640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b80c5d80 0x7fd7b813fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.380+0000 7fd7bd050640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7b80c2760 0x7fd7b813fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.380+0000 7fd7bd050640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b80715f0 con 0x7fd7b80c2760 2024-03-06T07:23:56.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.380+0000 7fd7bd050640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b8153af0 con 0x7fd7b80c4dd0 2024-03-06T07:23:56.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.380+0000 7fd7bd050640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b8071770 con 0x7fd7b80c5d80 2024-03-06T07:23:56.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd7b759e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7b80c2760 0x7fd7b813fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd7b6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7b80c4dd0 0x7fd7b80ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd7b6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7b80c4dd0 0x7fd7b80ac8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:33046/0 (socket says 172.21.15.1:33046) 2024-03-06T07:23:56.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd7b759e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7b80c2760 0x7fd7b813fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41440/0 (socket says 172.21.15.1:41440) 2024-03-06T07:23:56.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd7b6d9d640 1 -- 172.21.15.1:0/2872683793 learned_addr learned my addr 172.21.15.1:0/2872683793 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:56.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd7b659c640 1 --2- 172.21.15.1:0/2872683793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b80c5d80 0x7fd7b813fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd7b759e640 1 -- 172.21.15.1:0/2872683793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b80c5d80 msgr2=0x7fd7b813fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd7b759e640 1 --2- 172.21.15.1:0/2872683793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b80c5d80 0x7fd7b813fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd7b759e640 1 -- 172.21.15.1:0/2872683793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7b80c4dd0 msgr2=0x7fd7b80ac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd7b759e640 1 --2- 172.21.15.1:0/2872683793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7b80c4dd0 0x7fd7b80ac8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd7b759e640 1 -- 172.21.15.1:0/2872683793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b806c230 con 0x7fd7b80c2760 2024-03-06T07:23:56.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd7b6d9d640 1 --2- 172.21.15.1:0/2872683793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7b80c4dd0 0x7fd7b80ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:56.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd7b759e640 1 --2- 172.21.15.1:0/2872683793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7b80c2760 0x7fd7b813fff0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fd7b000eba0 tx=0x7fd7b000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd79f7fe640 1 -- 172.21.15.1:0/2872683793 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0018020 con 0x7fd7b80c2760 2024-03-06T07:23:56.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd79f7fe640 1 -- 172.21.15.1:0/2872683793 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b000cd20 con 0x7fd7b80c2760 2024-03-06T07:23:56.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd79f7fe640 1 -- 172.21.15.1:0/2872683793 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b00146c0 con 0x7fd7b80c2760 2024-03-06T07:23:56.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd7bd050640 1 -- 172.21.15.1:0/2872683793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b80aa620 con 0x7fd7b80c2760 2024-03-06T07:23:56.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.381+0000 7fd7bd050640 1 -- 172.21.15.1:0/2872683793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b80aabe0 con 0x7fd7b80c2760 2024-03-06T07:23:56.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.383+0000 7fd7bd050640 1 -- 172.21.15.1:0/2872683793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7b80aade0 con 0x7fd7b80c2760 2024-03-06T07:23:56.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.383+0000 7fd79f7fe640 1 -- 172.21.15.1:0/2872683793 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd7b0005d10 con 0x7fd7b80c2760 2024-03-06T07:23:56.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.383+0000 7fd79f7fe640 1 --2- 172.21.15.1:0/2872683793 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd7a406b4d0 0x7fd7a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.383+0000 7fd79f7fe640 1 -- 172.21.15.1:0/2872683793 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd7b008aeb0 con 0x7fd7b80c2760 2024-03-06T07:23:56.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.383+0000 7fd7b6d9d640 1 --2- 172.21.15.1:0/2872683793 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd7a406b4d0 0x7fd7a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.384+0000 7fd7b6d9d640 1 --2- 172.21.15.1:0/2872683793 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd7a406b4d0 0x7fd7a406d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd7a802b8a0 tx=0x7fd7a8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.388+0000 7fd79f7fe640 1 -- 172.21.15.1:0/2872683793 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd7b80aade0 con 0x7fd7b80c2760 2024-03-06T07:23:56.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.609+0000 7fd7bd050640 1 -- 172.21.15.1:0/2872683793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7b8067fd0 con 0x7fd7b80c2760 2024-03-06T07:23:56.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.610+0000 7fd79f7fe640 1 -- 172.21.15.1:0/2872683793 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4970 (secure 0 0 0) 0x7fd7b0022030 con 0x7fd7b80c2760 2024-03-06T07:23:56.612 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:56.612 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":82,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3611156352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3611156352},{"type":"v1","addr":"172.21.15.98:6837","nonce":3611156352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6981,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/2118550222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2118550222},{"type":"v1","addr":"172.21.15.1:6834","nonce":2118550222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/543699854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":543699854},{"type":"v1","addr":"172.21.15.1:6835","nonce":543699854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:41.077040+0000","modified":"2024-03-06T07:23:51.762675+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5855},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5855":{"gid":5855,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":7,"addr":"172.21.15.98:6836/1335402413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1335402413},{"type":"v1","addr":"172.21.15.98:6836","nonce":1335402413}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5855,"qdb_cluster":[5855]},"id":7}]} 2024-03-06T07:23:56.612 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 82 2024-03-06T07:23:56.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.611+0000 7fd7bd050640 1 -- 172.21.15.1:0/2872683793 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd7a406b4d0 msgr2=0x7fd7a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.611+0000 7fd7bd050640 1 --2- 172.21.15.1:0/2872683793 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd7a406b4d0 0x7fd7a406d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd7a802b8a0 tx=0x7fd7a8008030 comp rx=0 tx=0).stop 2024-03-06T07:23:56.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.611+0000 7fd7bd050640 1 -- 172.21.15.1:0/2872683793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7b80c2760 msgr2=0x7fd7b813fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.611+0000 7fd7bd050640 1 --2- 172.21.15.1:0/2872683793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7b80c2760 0x7fd7b813fff0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fd7b000eba0 tx=0x7fd7b000c490 comp rx=0 tx=0).stop 2024-03-06T07:23:56.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.611+0000 7fd7bd050640 1 -- 172.21.15.1:0/2872683793 shutdown_connections 2024-03-06T07:23:56.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.611+0000 7fd7bd050640 1 --2- 172.21.15.1:0/2872683793 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd7a406b4d0 0x7fd7a406d990 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.611+0000 7fd7bd050640 1 --2- 172.21.15.1:0/2872683793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7b80c2760 0x7fd7b813fff0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.611+0000 7fd7bd050640 1 --2- 172.21.15.1:0/2872683793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7b80c5d80 0x7fd7b813fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.611+0000 7fd7bd050640 1 --2- 172.21.15.1:0/2872683793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7b80c4dd0 0x7fd7b80ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.612+0000 7fd7bd050640 1 -- 172.21.15.1:0/2872683793 >> 172.21.15.1:0/2872683793 conn(0x7fd7b806b410 msgr2=0x7fd7b806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.612+0000 7fd7bd050640 1 -- 172.21.15.1:0/2872683793 shutdown_connections 2024-03-06T07:23:56.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.612+0000 7fd7bd050640 1 -- 172.21.15.1:0/2872683793 wait complete. 2024-03-06T07:23:56.624 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.725+0000 7f5c9cb7a640 1 Processor -- start 2024-03-06T07:23:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.725+0000 7f5c9cb7a640 1 -- start start 2024-03-06T07:23:56.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.726+0000 7f5c9cb7a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5c98152650 0x7f5c98155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.726+0000 7f5c9cb7a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980c6b10 0x7f5c98158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.726+0000 7f5c9cb7a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c980bf9d0 0x7f5c980c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.726+0000 7f5c9cb7a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c98155800 con 0x7f5c980bf9d0 2024-03-06T07:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.726+0000 7f5c9cb7a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c980c2fc0 con 0x7f5c98152650 2024-03-06T07:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.726+0000 7f5c9cb7a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c980c6690 con 0x7f5c980c6b10 2024-03-06T07:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.726+0000 7f5c96d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c980bf9d0 0x7f5c980c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.726+0000 7f5c96d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c980bf9d0 0x7f5c980c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41442/0 (socket says 172.21.15.1:41442) 2024-03-06T07:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.726+0000 7f5c96d9d640 1 -- 172.21.15.1:0/1795198715 learned_addr learned my addr 172.21.15.1:0/1795198715 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.726+0000 7f5c8ffff640 1 --2- 172.21.15.1:0/1795198715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980c6b10 0x7f5c98158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.726+0000 7f5c9659c640 1 --2- 172.21.15.1:0/1795198715 >> v2:172.21.15.98:3300/0 conn(0x7f5c98152650 0x7f5c98155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.726+0000 7f5c96d9d640 1 -- 172.21.15.1:0/1795198715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980c6b10 msgr2=0x7f5c98158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.726+0000 7f5c96d9d640 1 --2- 172.21.15.1:0/1795198715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980c6b10 0x7f5c98158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.726+0000 7f5c96d9d640 1 -- 172.21.15.1:0/1795198715 >> v2:172.21.15.98:3300/0 conn(0x7f5c98152650 msgr2=0x7f5c98155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.726+0000 7f5c96d9d640 1 --2- 172.21.15.1:0/1795198715 >> v2:172.21.15.98:3300/0 conn(0x7f5c98152650 0x7f5c98155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.726+0000 7f5c96d9d640 1 -- 172.21.15.1:0/1795198715 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c980c7280 con 0x7f5c980bf9d0 2024-03-06T07:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.727+0000 7f5c96d9d640 1 --2- 172.21.15.1:0/1795198715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c980bf9d0 0x7f5c980c1460 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f5c980cac90 tx=0x7f5c9002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b4888ad410581c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.727+0000 7f5c95d9b640 1 -- 172.21.15.1:0/1795198715 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9002c9e0 con 0x7f5c980bf9d0 2024-03-06T07:23:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.727+0000 7f5c95d9b640 1 -- 172.21.15.1:0/1795198715 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9002cb80 con 0x7f5c980bf9d0 2024-03-06T07:23:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.727+0000 7f5c95d9b640 1 -- 172.21.15.1:0/1795198715 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c900343f0 con 0x7f5c980bf9d0 2024-03-06T07:23:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.727+0000 7f5c9cb7a640 1 -- 172.21.15.1:0/1795198715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c980bf9d0 msgr2=0x7f5c980c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.727+0000 7f5c9cb7a640 1 --2- 172.21.15.1:0/1795198715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c980bf9d0 0x7f5c980c1460 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f5c980cac90 tx=0x7f5c9002be40 comp rx=0 tx=0).stop 2024-03-06T07:23:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.727+0000 7f5c9cb7a640 1 -- 172.21.15.1:0/1795198715 shutdown_connections 2024-03-06T07:23:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.727+0000 7f5c9cb7a640 1 --2- 172.21.15.1:0/1795198715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c980bf9d0 0x7f5c980c1460 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.727+0000 7f5c9cb7a640 1 --2- 172.21.15.1:0/1795198715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980c6b10 0x7f5c98158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.727+0000 7f5c9cb7a640 1 --2- 172.21.15.1:0/1795198715 >> v2:172.21.15.98:3300/0 conn(0x7f5c98152650 0x7f5c98155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.727+0000 7f5c9cb7a640 1 -- 172.21.15.1:0/1795198715 >> 172.21.15.1:0/1795198715 conn(0x7f5c98066ee0 msgr2=0x7f5c980672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.728+0000 7f5c9cb7a640 1 -- 172.21.15.1:0/1795198715 shutdown_connections 2024-03-06T07:23:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.728+0000 7f5c9cb7a640 1 -- 172.21.15.1:0/1795198715 wait complete. 2024-03-06T07:23:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.729+0000 7f5c9cb7a640 1 Processor -- start 2024-03-06T07:23:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.729+0000 7f5c9cb7a640 1 -- start start 2024-03-06T07:23:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.729+0000 7f5c9cb7a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c980bf9d0 0x7f5c9814c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.729+0000 7f5c9cb7a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c98152650 0x7f5c980c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.729+0000 7f5c9cb7a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980c9df0 0x7f5c98153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.729+0000 7f5c9cb7a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c98155490 con 0x7f5c980bf9d0 2024-03-06T07:23:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.729+0000 7f5c9cb7a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c981551b0 con 0x7f5c98152650 2024-03-06T07:23:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.729+0000 7f5c9cb7a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c98152a30 con 0x7f5c980c9df0 2024-03-06T07:23:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.730+0000 7f5c9659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c980bf9d0 0x7f5c9814c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.730+0000 7f5c9659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c980bf9d0 0x7f5c9814c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41454/0 (socket says 172.21.15.1:41454) 2024-03-06T07:23:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.730+0000 7f5c9659c640 1 -- 172.21.15.1:0/2496336728 learned_addr learned my addr 172.21.15.1:0/2496336728 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.730+0000 7f5c8ffff640 1 --2- 172.21.15.1:0/2496336728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c98152650 0x7f5c980c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.730+0000 7f5c96d9d640 1 --2- 172.21.15.1:0/2496336728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980c9df0 0x7f5c98153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.730+0000 7f5c9659c640 1 -- 172.21.15.1:0/2496336728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980c9df0 msgr2=0x7f5c98153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.730+0000 7f5c9659c640 1 --2- 172.21.15.1:0/2496336728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980c9df0 0x7f5c98153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.730+0000 7f5c9659c640 1 -- 172.21.15.1:0/2496336728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c98152650 msgr2=0x7f5c980c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.730+0000 7f5c9659c640 1 --2- 172.21.15.1:0/2496336728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c98152650 0x7f5c980c98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.730+0000 7f5c9659c640 1 -- 172.21.15.1:0/2496336728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c981ebba0 con 0x7f5c980bf9d0 2024-03-06T07:23:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.730+0000 7f5c9659c640 1 --2- 172.21.15.1:0/2496336728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c980bf9d0 0x7f5c9814c260 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f5c8800e9b0 tx=0x7f5c8800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.730+0000 7f5c8effd640 1 -- 172.21.15.1:0/2496336728 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c88017070 con 0x7f5c980bf9d0 2024-03-06T07:23:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.730+0000 7f5c8effd640 1 -- 172.21.15.1:0/2496336728 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8800c9a0 con 0x7f5c980bf9d0 2024-03-06T07:23:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.730+0000 7f5c9cb7a640 1 -- 172.21.15.1:0/2496336728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c980c8dd0 con 0x7f5c980bf9d0 2024-03-06T07:23:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.730+0000 7f5c8effd640 1 -- 172.21.15.1:0/2496336728 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8800cd90 con 0x7f5c980bf9d0 2024-03-06T07:23:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.730+0000 7f5c9cb7a640 1 -- 172.21.15.1:0/2496336728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c980c9390 con 0x7f5c980bf9d0 2024-03-06T07:23:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.731+0000 7f5c9cb7a640 1 -- 172.21.15.1:0/2496336728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c98067fd0 con 0x7f5c980bf9d0 2024-03-06T07:23:56.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.732+0000 7f5c8effd640 1 -- 172.21.15.1:0/2496336728 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c8801a070 con 0x7f5c980bf9d0 2024-03-06T07:23:56.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.732+0000 7f5c8effd640 1 --2- 172.21.15.1:0/2496336728 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c7806b4d0 0x7f5c7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.732+0000 7f5c8effd640 1 -- 172.21.15.1:0/2496336728 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5c8808a2e0 con 0x7f5c980bf9d0 2024-03-06T07:23:56.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.732+0000 7f5c8ffff640 1 --2- 172.21.15.1:0/2496336728 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c7806b4d0 0x7f5c7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.733+0000 7f5c8ffff640 1 --2- 172.21.15.1:0/2496336728 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c7806b4d0 0x7f5c7806d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f5c98155cb0 tx=0x7f5c80009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.736+0000 7f5c8effd640 1 -- 172.21.15.1:0/2496336728 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c880589c0 con 0x7f5c980bf9d0 2024-03-06T07:23:56.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.961+0000 7f5c9cb7a640 1 -- 172.21.15.1:0/2496336728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c980b06f0 con 0x7f5c980bf9d0 2024-03-06T07:23:56.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.962+0000 7f5c8effd640 1 -- 172.21.15.1:0/2496336728 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4970 (secure 0 0 0) 0x7f5c8805ca60 con 0x7f5c980bf9d0 2024-03-06T07:23:56.965 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:56.965 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":82,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3611156352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3611156352},{"type":"v1","addr":"172.21.15.98:6837","nonce":3611156352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6981,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/2118550222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2118550222},{"type":"v1","addr":"172.21.15.1:6834","nonce":2118550222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/543699854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":543699854},{"type":"v1","addr":"172.21.15.1:6835","nonce":543699854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:41.077040+0000","modified":"2024-03-06T07:23:51.762675+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5855},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5855":{"gid":5855,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":7,"addr":"172.21.15.98:6836/1335402413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1335402413},{"type":"v1","addr":"172.21.15.98:6836","nonce":1335402413}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5855,"qdb_cluster":[5855]},"id":7}]} 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 82 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.964+0000 7f5c9cb7a640 1 -- 172.21.15.1:0/2496336728 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c7806b4d0 msgr2=0x7f5c7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.964+0000 7f5c9cb7a640 1 --2- 172.21.15.1:0/2496336728 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c7806b4d0 0x7f5c7806d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f5c98155cb0 tx=0x7f5c80009190 comp rx=0 tx=0).stop 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.964+0000 7f5c9cb7a640 1 -- 172.21.15.1:0/2496336728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c980bf9d0 msgr2=0x7f5c9814c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.964+0000 7f5c9cb7a640 1 --2- 172.21.15.1:0/2496336728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c980bf9d0 0x7f5c9814c260 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f5c8800e9b0 tx=0x7f5c8800ee70 comp rx=0 tx=0).stop 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.964+0000 7f5c9cb7a640 1 -- 172.21.15.1:0/2496336728 shutdown_connections 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.964+0000 7f5c9cb7a640 1 --2- 172.21.15.1:0/2496336728 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c7806b4d0 0x7f5c7806d990 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.964+0000 7f5c9cb7a640 1 --2- 172.21.15.1:0/2496336728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980c9df0 0x7f5c98153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.964+0000 7f5c9cb7a640 1 --2- 172.21.15.1:0/2496336728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c98152650 0x7f5c980c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.964+0000 7f5c9cb7a640 1 --2- 172.21.15.1:0/2496336728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c980bf9d0 0x7f5c9814c260 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.964+0000 7f5c9cb7a640 1 -- 172.21.15.1:0/2496336728 >> 172.21.15.1:0/2496336728 conn(0x7f5c98066ee0 msgr2=0x7f5c980672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.964+0000 7f5c9cb7a640 1 -- 172.21.15.1:0/2496336728 shutdown_connections 2024-03-06T07:23:56.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:56.964+0000 7f5c9cb7a640 1 -- 172.21.15.1:0/2496336728 wait complete. 2024-03-06T07:23:56.977 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:57.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.077+0000 7fe0df13a640 1 Processor -- start 2024-03-06T07:23:57.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.077+0000 7fe0df13a640 1 -- start start 2024-03-06T07:23:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.077+0000 7fe0df13a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe0d8152690 0x7fe0d8155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.077+0000 7fe0df13a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0d80c6730 0x7fe0d80c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.077+0000 7fe0df13a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe0d80c5d10 0x7fe0d8156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.077+0000 7fe0df13a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d81561e0 con 0x7fe0d8152690 2024-03-06T07:23:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.077+0000 7fe0df13a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d80c3300 con 0x7fe0d80c5d10 2024-03-06T07:23:57.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.077+0000 7fe0df13a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d80c35a0 con 0x7fe0d80c6730 2024-03-06T07:23:57.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.078+0000 7fe0dced6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe0d8152690 0x7fe0d8155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.078+0000 7fe0dced6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe0d8152690 0x7fe0d8155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41458/0 (socket says 172.21.15.1:41458) 2024-03-06T07:23:57.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.078+0000 7fe0dd6d7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe0d80c5d10 0x7fe0d8156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.078+0000 7fe0d7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0d80c6730 0x7fe0d80c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.078+0000 7fe0dced6640 1 -- 172.21.15.1:0/3168193369 learned_addr learned my addr 172.21.15.1:0/3168193369 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:57.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.078+0000 7fe0dced6640 1 -- 172.21.15.1:0/3168193369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0d80c6730 msgr2=0x7fe0d80c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.078+0000 7fe0dced6640 1 --2- 172.21.15.1:0/3168193369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0d80c6730 0x7fe0d80c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.078+0000 7fe0dced6640 1 -- 172.21.15.1:0/3168193369 >> v2:172.21.15.98:3300/0 conn(0x7fe0d80c5d10 msgr2=0x7fe0d8156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.078+0000 7fe0dced6640 1 --2- 172.21.15.1:0/3168193369 >> v2:172.21.15.98:3300/0 conn(0x7fe0d80c5d10 0x7fe0d8156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.078+0000 7fe0dced6640 1 -- 172.21.15.1:0/3168193369 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d80c3880 con 0x7fe0d8152690 2024-03-06T07:23:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.079+0000 7fe0dced6640 1 --2- 172.21.15.1:0/3168193369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0d8152690 0x7fe0d8155ca0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fe0d80c2240 tx=0x7fe0cc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4dedd6a33bcfb8a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.079+0000 7fe0d77fe640 1 -- 172.21.15.1:0/3168193369 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc02e070 con 0x7fe0d8152690 2024-03-06T07:23:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.079+0000 7fe0d77fe640 1 -- 172.21.15.1:0/3168193369 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0cc037070 con 0x7fe0d8152690 2024-03-06T07:23:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.079+0000 7fe0d77fe640 1 -- 172.21.15.1:0/3168193369 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc031a00 con 0x7fe0d8152690 2024-03-06T07:23:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.079+0000 7fe0df13a640 1 -- 172.21.15.1:0/3168193369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0d8152690 msgr2=0x7fe0d8155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.079+0000 7fe0df13a640 1 --2- 172.21.15.1:0/3168193369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0d8152690 0x7fe0d8155ca0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fe0d80c2240 tx=0x7fe0cc02b880 comp rx=0 tx=0).stop 2024-03-06T07:23:57.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.079+0000 7fe0df13a640 1 -- 172.21.15.1:0/3168193369 shutdown_connections 2024-03-06T07:23:57.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.079+0000 7fe0df13a640 1 --2- 172.21.15.1:0/3168193369 >> v2:172.21.15.98:3300/0 conn(0x7fe0d80c5d10 0x7fe0d8156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.079+0000 7fe0df13a640 1 --2- 172.21.15.1:0/3168193369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0d80c6730 0x7fe0d80c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.079+0000 7fe0df13a640 1 --2- 172.21.15.1:0/3168193369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0d8152690 0x7fe0d8155ca0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.079+0000 7fe0df13a640 1 -- 172.21.15.1:0/3168193369 >> 172.21.15.1:0/3168193369 conn(0x7fe0d806b410 msgr2=0x7fe0d806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.079+0000 7fe0df13a640 1 -- 172.21.15.1:0/3168193369 shutdown_connections 2024-03-06T07:23:57.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.079+0000 7fe0df13a640 1 -- 172.21.15.1:0/3168193369 wait complete. 2024-03-06T07:23:57.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.080+0000 7fe0df13a640 1 Processor -- start 2024-03-06T07:23:57.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.081+0000 7fe0df13a640 1 -- start start 2024-03-06T07:23:57.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.081+0000 7fe0df13a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0d80c5d10 0x7fe0d8070ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.081+0000 7fe0df13a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0d80c6730 0x7fe0d8146990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.081+0000 7fe0df13a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0d8152690 0x7fe0d8146ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.081+0000 7fe0df13a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d8152a70 con 0x7fe0d80c5d10 2024-03-06T07:23:57.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.081+0000 7fe0df13a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d81567b0 con 0x7fe0d80c6730 2024-03-06T07:23:57.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.081+0000 7fe0df13a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d8152bf0 con 0x7fe0d8152690 2024-03-06T07:23:57.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.081+0000 7fe0dced6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0d80c5d10 0x7fe0d8070ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.081+0000 7fe0dced6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0d80c5d10 0x7fe0d8070ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41470/0 (socket says 172.21.15.1:41470) 2024-03-06T07:23:57.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.081+0000 7fe0dced6640 1 -- 172.21.15.1:0/198850188 learned_addr learned my addr 172.21.15.1:0/198850188 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:57.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.082+0000 7fe0d7fff640 1 --2- 172.21.15.1:0/198850188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0d80c6730 0x7fe0d8146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.082+0000 7fe0dd6d7640 1 --2- 172.21.15.1:0/198850188 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0d8152690 0x7fe0d8146ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.082+0000 7fe0dced6640 1 -- 172.21.15.1:0/198850188 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0d8152690 msgr2=0x7fe0d8146ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.082+0000 7fe0dced6640 1 --2- 172.21.15.1:0/198850188 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0d8152690 0x7fe0d8146ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.082+0000 7fe0dced6640 1 -- 172.21.15.1:0/198850188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0d80c6730 msgr2=0x7fe0d8146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.082+0000 7fe0dced6640 1 --2- 172.21.15.1:0/198850188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0d80c6730 0x7fe0d8146990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.082+0000 7fe0dced6640 1 -- 172.21.15.1:0/198850188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d8153410 con 0x7fe0d80c5d10 2024-03-06T07:23:57.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.082+0000 7fe0dced6640 1 --2- 172.21.15.1:0/198850188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0d80c5d10 0x7fe0d8070ec0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc000c00 tx=0x7fe0cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.082+0000 7fe0d57fa640 1 -- 172.21.15.1:0/198850188 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc032670 con 0x7fe0d80c5d10 2024-03-06T07:23:57.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.082+0000 7fe0d57fa640 1 -- 172.21.15.1:0/198850188 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0cc032810 con 0x7fe0d80c5d10 2024-03-06T07:23:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.082+0000 7fe0df13a640 1 -- 172.21.15.1:0/198850188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d8152e90 con 0x7fe0d80c5d10 2024-03-06T07:23:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.082+0000 7fe0d57fa640 1 -- 172.21.15.1:0/198850188 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc032c00 con 0x7fe0d80c5d10 2024-03-06T07:23:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.082+0000 7fe0df13a640 1 -- 172.21.15.1:0/198850188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d8155bd0 con 0x7fe0d80c5d10 2024-03-06T07:23:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.083+0000 7fe0df13a640 1 -- 172.21.15.1:0/198850188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d8067fd0 con 0x7fe0d80c5d10 2024-03-06T07:23:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.084+0000 7fe0d57fa640 1 -- 172.21.15.1:0/198850188 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe0cc0453f0 con 0x7fe0d80c5d10 2024-03-06T07:23:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.084+0000 7fe0d57fa640 1 --2- 172.21.15.1:0/198850188 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe0bc06b4d0 0x7fe0bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.084+0000 7fe0d57fa640 1 -- 172.21.15.1:0/198850188 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe0cc0abfa0 con 0x7fe0d80c5d10 2024-03-06T07:23:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.084+0000 7fe0d7fff640 1 --2- 172.21.15.1:0/198850188 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe0bc06b4d0 0x7fe0bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.085+0000 7fe0d7fff640 1 --2- 172.21.15.1:0/198850188 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe0bc06b4d0 0x7fe0bc06d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe0d80adbb0 tx=0x7fe0c4009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.089+0000 7fe0d57fa640 1 -- 172.21.15.1:0/198850188 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe0d8153410 con 0x7fe0d80c5d10 2024-03-06T07:23:57.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.272+0000 7fe0df13a640 1 -- 172.21.15.1:0/198850188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0d80c2c40 con 0x7fe0d80c5d10 2024-03-06T07:23:57.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.274+0000 7fe0d57fa640 1 -- 172.21.15.1:0/198850188 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16882 (secure 0 0 0) 0x7fe0cc040070 con 0x7fe0d80c5d10 2024-03-06T07:23:57.276 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:23:57.276 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":96,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:23:48.752552+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:39.342951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:40.043775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6837/3330683497":"2024-03-07T07:23:48.752518+0000","172.21.15.98:6835/3330683497":"2024-03-07T07:23:48.752518+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:57.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.275+0000 7fe0df13a640 1 -- 172.21.15.1:0/198850188 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe0bc06b4d0 msgr2=0x7fe0bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.275+0000 7fe0df13a640 1 --2- 172.21.15.1:0/198850188 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe0bc06b4d0 0x7fe0bc06d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe0d80adbb0 tx=0x7fe0c4009190 comp rx=0 tx=0).stop 2024-03-06T07:23:57.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.276+0000 7fe0df13a640 1 -- 172.21.15.1:0/198850188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0d80c5d10 msgr2=0x7fe0d8070ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.276+0000 7fe0df13a640 1 --2- 172.21.15.1:0/198850188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0d80c5d10 0x7fe0d8070ec0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc000c00 tx=0x7fe0cc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.276+0000 7fe0df13a640 1 -- 172.21.15.1:0/198850188 shutdown_connections 2024-03-06T07:23:57.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.276+0000 7fe0df13a640 1 --2- 172.21.15.1:0/198850188 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe0bc06b4d0 0x7fe0bc06d990 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.276+0000 7fe0df13a640 1 --2- 172.21.15.1:0/198850188 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0d8152690 0x7fe0d8146ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.276+0000 7fe0df13a640 1 --2- 172.21.15.1:0/198850188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0d80c6730 0x7fe0d8146990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.276+0000 7fe0df13a640 1 --2- 172.21.15.1:0/198850188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0d80c5d10 0x7fe0d8070ec0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.276+0000 7fe0df13a640 1 -- 172.21.15.1:0/198850188 >> 172.21.15.1:0/198850188 conn(0x7fe0d806b410 msgr2=0x7fe0d806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.276+0000 7fe0df13a640 1 -- 172.21.15.1:0/198850188 shutdown_connections 2024-03-06T07:23:57.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.276+0000 7fe0df13a640 1 -- 172.21.15.1:0/198850188 wait complete. 2024-03-06T07:23:57.289 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:23:57.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.389+0000 7f40d215e640 1 Processor -- start 2024-03-06T07:23:57.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.389+0000 7f40d215e640 1 -- start start 2024-03-06T07:23:57.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.390+0000 7f40d215e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40cc152730 0x7f40cc0c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.390+0000 7f40d215e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f40cc0c6fd0 0x7f40cc0c1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.390+0000 7f40d215e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40cc0c18d0 0x7f40cc1552e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.390+0000 7f40d215e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40cc0c1d20 con 0x7f40cc0c18d0 2024-03-06T07:23:57.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.390+0000 7f40d215e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40cc1588e0 con 0x7f40cc0c6fd0 2024-03-06T07:23:57.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.390+0000 7f40d215e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc158bc0 con 0x7f40cc152730 2024-03-06T07:23:57.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.390+0000 7f40cbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40cc0c18d0 0x7f40cc1552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.390+0000 7f40cbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40cc0c18d0 0x7f40cc1552e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41474/0 (socket says 172.21.15.1:41474) 2024-03-06T07:23:57.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.390+0000 7f40cbfff640 1 -- 172.21.15.1:0/2947420789 learned_addr learned my addr 172.21.15.1:0/2947420789 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:57.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.390+0000 7f40caffd640 1 --2- 172.21.15.1:0/2947420789 >> v2:172.21.15.98:3300/0 conn(0x7f40cc0c6fd0 0x7f40cc0c1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.390+0000 7f40cb7fe640 1 --2- 172.21.15.1:0/2947420789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40cc152730 0x7f40cc0c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.390+0000 7f40cbfff640 1 -- 172.21.15.1:0/2947420789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40cc152730 msgr2=0x7f40cc0c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.390+0000 7f40cbfff640 1 --2- 172.21.15.1:0/2947420789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40cc152730 0x7f40cc0c6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.390+0000 7f40cbfff640 1 -- 172.21.15.1:0/2947420789 >> v2:172.21.15.98:3300/0 conn(0x7f40cc0c6fd0 msgr2=0x7f40cc0c1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.390+0000 7f40cbfff640 1 --2- 172.21.15.1:0/2947420789 >> v2:172.21.15.98:3300/0 conn(0x7f40cc0c6fd0 0x7f40cc0c1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.390+0000 7f40cbfff640 1 -- 172.21.15.1:0/2947420789 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40cc158ea0 con 0x7f40cc0c18d0 2024-03-06T07:23:57.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.391+0000 7f40cbfff640 1 --2- 172.21.15.1:0/2947420789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40cc0c18d0 0x7f40cc1552e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f40cc08c090 tx=0x7f40c402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=545a34080ab47f6f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.391+0000 7f40ca7fc640 1 -- 172.21.15.1:0/2947420789 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c402c950 con 0x7f40cc0c18d0 2024-03-06T07:23:57.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.391+0000 7f40ca7fc640 1 -- 172.21.15.1:0/2947420789 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c402caf0 con 0x7f40cc0c18d0 2024-03-06T07:23:57.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.391+0000 7f40ca7fc640 1 -- 172.21.15.1:0/2947420789 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c40343f0 con 0x7f40cc0c18d0 2024-03-06T07:23:57.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.391+0000 7f40d215e640 1 -- 172.21.15.1:0/2947420789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40cc0c18d0 msgr2=0x7f40cc1552e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.391+0000 7f40d215e640 1 --2- 172.21.15.1:0/2947420789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40cc0c18d0 0x7f40cc1552e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f40cc08c090 tx=0x7f40c402bdb0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.391+0000 7f40d215e640 1 -- 172.21.15.1:0/2947420789 shutdown_connections 2024-03-06T07:23:57.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.391+0000 7f40d215e640 1 --2- 172.21.15.1:0/2947420789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40cc0c18d0 0x7f40cc1552e0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.391+0000 7f40d215e640 1 --2- 172.21.15.1:0/2947420789 >> v2:172.21.15.98:3300/0 conn(0x7f40cc0c6fd0 0x7f40cc0c1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.391+0000 7f40d215e640 1 --2- 172.21.15.1:0/2947420789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40cc152730 0x7f40cc0c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.391+0000 7f40d215e640 1 -- 172.21.15.1:0/2947420789 >> 172.21.15.1:0/2947420789 conn(0x7f40cc066d00 msgr2=0x7f40cc067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.392+0000 7f40d215e640 1 -- 172.21.15.1:0/2947420789 shutdown_connections 2024-03-06T07:23:57.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.392+0000 7f40d215e640 1 -- 172.21.15.1:0/2947420789 wait complete. 2024-03-06T07:23:57.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.393+0000 7f40d215e640 1 Processor -- start 2024-03-06T07:23:57.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.393+0000 7f40d215e640 1 -- start start 2024-03-06T07:23:57.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.393+0000 7f40d215e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40cc0c18d0 0x7f40cc14b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.393+0000 7f40d215e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40cc0c6fd0 0x7f40cc157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.393+0000 7f40d215e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40cc152730 0x7f40cc14a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.393+0000 7f40d215e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc152b10 con 0x7f40cc152730 2024-03-06T07:23:57.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.393+0000 7f40d215e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc159460 con 0x7f40cc0c6fd0 2024-03-06T07:23:57.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.393+0000 7f40d215e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc155820 con 0x7f40cc0c18d0 2024-03-06T07:23:57.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.394+0000 7f40cbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40cc152730 0x7f40cc14a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.394+0000 7f40cbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40cc152730 0x7f40cc14a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41488/0 (socket says 172.21.15.1:41488) 2024-03-06T07:23:57.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.394+0000 7f40cbfff640 1 -- 172.21.15.1:0/1692184479 learned_addr learned my addr 172.21.15.1:0/1692184479 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:57.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.394+0000 7f40caffd640 1 --2- 172.21.15.1:0/1692184479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40cc0c6fd0 0x7f40cc157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.394+0000 7f40cb7fe640 1 --2- 172.21.15.1:0/1692184479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40cc0c18d0 0x7f40cc14b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.394+0000 7f40cbfff640 1 -- 172.21.15.1:0/1692184479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40cc0c18d0 msgr2=0x7f40cc14b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.394+0000 7f40cbfff640 1 --2- 172.21.15.1:0/1692184479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40cc0c18d0 0x7f40cc14b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.394+0000 7f40cbfff640 1 -- 172.21.15.1:0/1692184479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40cc0c6fd0 msgr2=0x7f40cc157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.394+0000 7f40cbfff640 1 --2- 172.21.15.1:0/1692184479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40cc0c6fd0 0x7f40cc157580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.394+0000 7f40cbfff640 1 -- 172.21.15.1:0/1692184479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40cc0c8ee0 con 0x7f40cc152730 2024-03-06T07:23:57.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.394+0000 7f40cbfff640 1 --2- 172.21.15.1:0/1692184479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40cc152730 0x7f40cc14a780 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f40c4009f40 tx=0x7f40c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.394+0000 7f40b3fff640 1 -- 172.21.15.1:0/1692184479 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c4043020 con 0x7f40cc152730 2024-03-06T07:23:57.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.394+0000 7f40b3fff640 1 -- 172.21.15.1:0/1692184479 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c403e450 con 0x7f40cc152730 2024-03-06T07:23:57.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.394+0000 7f40d215e640 1 -- 172.21.15.1:0/1692184479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40cc0c91c0 con 0x7f40cc152730 2024-03-06T07:23:57.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.394+0000 7f40b3fff640 1 -- 172.21.15.1:0/1692184479 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c403e840 con 0x7f40cc152730 2024-03-06T07:23:57.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.394+0000 7f40d215e640 1 -- 172.21.15.1:0/1692184479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40cc0c94a0 con 0x7f40cc152730 2024-03-06T07:23:57.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.395+0000 7f40d215e640 1 -- 172.21.15.1:0/1692184479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40cc067fd0 con 0x7f40cc152730 2024-03-06T07:23:57.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.396+0000 7f40b3fff640 1 -- 172.21.15.1:0/1692184479 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f40c4046020 con 0x7f40cc152730 2024-03-06T07:23:57.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.396+0000 7f40b3fff640 1 --2- 172.21.15.1:0/1692184479 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f40b806b4d0 0x7f40b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.396+0000 7f40b3fff640 1 -- 172.21.15.1:0/1692184479 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f40c40ac8b0 con 0x7f40cc152730 2024-03-06T07:23:57.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.396+0000 7f40cb7fe640 1 --2- 172.21.15.1:0/1692184479 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f40b806b4d0 0x7f40b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.397+0000 7f40cb7fe640 1 --2- 172.21.15.1:0/1692184479 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f40b806b4d0 0x7f40b806d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f40cc14d290 tx=0x7f40bc009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.401+0000 7f40b3fff640 1 -- 172.21.15.1:0/1692184479 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f40c404d030 con 0x7f40cc152730 2024-03-06T07:23:57.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.618+0000 7f40d215e640 1 -- 172.21.15.1:0/1692184479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f40cc0b0220 con 0x7f40cc152730 2024-03-06T07:23:57.620 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:23:57.619+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:23:57.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.620+0000 7f40b3fff640 1 -- 172.21.15.1:0/1692184479 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v83) v1 ==== 161+0+0 (secure 0 0 0) 0x7f40c407b0b0 con 0x7f40cc152730 2024-03-06T07:23:57.621 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:23:57.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.623+0000 7f40d215e640 1 -- 172.21.15.1:0/1692184479 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f40b806b4d0 msgr2=0x7f40b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.623+0000 7f40d215e640 1 --2- 172.21.15.1:0/1692184479 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f40b806b4d0 0x7f40b806d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f40cc14d290 tx=0x7f40bc009190 comp rx=0 tx=0).stop 2024-03-06T07:23:57.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.623+0000 7f40d215e640 1 -- 172.21.15.1:0/1692184479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40cc152730 msgr2=0x7f40cc14a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.623+0000 7f40d215e640 1 --2- 172.21.15.1:0/1692184479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40cc152730 0x7f40cc14a780 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f40c4009f40 tx=0x7f40c4004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:57.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.623+0000 7f40d215e640 1 -- 172.21.15.1:0/1692184479 shutdown_connections 2024-03-06T07:23:57.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.623+0000 7f40d215e640 1 --2- 172.21.15.1:0/1692184479 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f40b806b4d0 0x7f40b806d990 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.623+0000 7f40d215e640 1 --2- 172.21.15.1:0/1692184479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40cc152730 0x7f40cc14a780 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.623+0000 7f40d215e640 1 --2- 172.21.15.1:0/1692184479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40cc0c6fd0 0x7f40cc157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.623+0000 7f40d215e640 1 --2- 172.21.15.1:0/1692184479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40cc0c18d0 0x7f40cc14b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.623+0000 7f40d215e640 1 -- 172.21.15.1:0/1692184479 >> 172.21.15.1:0/1692184479 conn(0x7f40cc066d00 msgr2=0x7f40cc0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.623+0000 7f40d215e640 1 -- 172.21.15.1:0/1692184479 shutdown_connections 2024-03-06T07:23:57.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.623+0000 7f40d215e640 1 -- 172.21.15.1:0/1692184479 wait complete. 2024-03-06T07:23:57.626 INFO:tasks.ceph.mds.b.smithi098.stderr: -202> 2024-03-06T07:23:51.988+0000 7f251b5bd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:57.636 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:23:57.648 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:23:57.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.739+0000 7f070f3c9640 1 Processor -- start 2024-03-06T07:23:57.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.739+0000 7f070f3c9640 1 -- start start 2024-03-06T07:23:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f070f3c9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f07080c6b10 0x7f0708155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f070f3c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07080bf9d0 0x7f07080c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f070f3c9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07080c2fc0 con 0x7f0708152650 2024-03-06T07:23:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f070f3c9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0708155800 con 0x7f07080c6b10 2024-03-06T07:23:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f070f3c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0708158da0 con 0x7f07080bf9d0 2024-03-06T07:23:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f070d165640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0708152650 0x7f0708158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37370/0 (socket says 172.21.15.1:37370) 2024-03-06T07:23:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f070d165640 1 -- 172.21.15.1:0/658731050 learned_addr learned my addr 172.21.15.1:0/658731050 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f070c964640 1 --2- 172.21.15.1:0/658731050 >> v2:172.21.15.98:3300/0 conn(0x7f07080c6b10 0x7f0708155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f070d966640 1 --2- 172.21.15.1:0/658731050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07080bf9d0 0x7f07080c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f06fffff640 1 -- 172.21.15.1:0/658731050 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081533913 0 0) 0x7f07080c2fc0 con 0x7f0708152650 2024-03-06T07:23:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f06fffff640 1 -- 172.21.15.1:0/658731050 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f4003520 con 0x7f0708152650 2024-03-06T07:23:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f06fffff640 1 -- 172.21.15.1:0/658731050 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3935910441 0 0) 0x7f06f4003520 con 0x7f0708152650 2024-03-06T07:23:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f06fffff640 1 -- 172.21.15.1:0/658731050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07080bf9d0 msgr2=0x7f07080c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f06fffff640 1 --2- 172.21.15.1:0/658731050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07080bf9d0 0x7f07080c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f06fffff640 1 -- 172.21.15.1:0/658731050 >> v2:172.21.15.98:3300/0 conn(0x7f07080c6b10 msgr2=0x7f0708155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f06fffff640 1 --2- 172.21.15.1:0/658731050 >> v2:172.21.15.98:3300/0 conn(0x7f07080c6b10 0x7f0708155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f06fffff640 1 -- 172.21.15.1:0/658731050 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07080c6690 con 0x7f0708152650 2024-03-06T07:23:57.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f06fffff640 1 -- 172.21.15.1:0/658731050 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f0704001690 con 0x7f0708152650 2024-03-06T07:23:57.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f06fffff640 1 -- 172.21.15.1:0/658731050 >> v1:172.21.15.1:6789/0 conn(0x7f0708152650 legacy=0x7f0708158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f06fffff640 1 --2- 172.21.15.1:0/658731050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06f40039b0 0x7f06f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f06fffff640 1 --2- 172.21.15.1:0/658731050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06f4004800 0x7f06f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f06fffff640 1 --2- 172.21.15.1:0/658731050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07080bf9d0 0x7f06f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f06fffff640 1 -- 172.21.15.1:0/658731050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0708155800 con 0x7f06f40039b0 2024-03-06T07:23:57.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f06fffff640 1 -- 172.21.15.1:0/658731050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0708158da0 con 0x7f06f4004800 2024-03-06T07:23:57.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f06fffff640 1 -- 172.21.15.1:0/658731050 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f400b9f0 con 0x7f07080bf9d0 2024-03-06T07:23:57.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.740+0000 7f070d165640 1 --2- 172.21.15.1:0/658731050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06f40039b0 0x7f06f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.741+0000 7f070c964640 1 --2- 172.21.15.1:0/658731050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06f4004800 0x7f06f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.741+0000 7f070d966640 1 --2- 172.21.15.1:0/658731050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07080bf9d0 0x7f06f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.741+0000 7f070d165640 1 -- 172.21.15.1:0/658731050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07080bf9d0 msgr2=0x7f06f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.741+0000 7f070d165640 1 --2- 172.21.15.1:0/658731050 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07080bf9d0 0x7f06f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.741+0000 7f070d165640 1 -- 172.21.15.1:0/658731050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06f4004800 msgr2=0x7f06f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.741+0000 7f070d165640 1 --2- 172.21.15.1:0/658731050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06f4004800 0x7f06f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.741+0000 7f070d165640 1 -- 172.21.15.1:0/658731050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f4003520 con 0x7f06f40039b0 2024-03-06T07:23:57.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.742+0000 7f070d165640 1 --2- 172.21.15.1:0/658731050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06f40039b0 0x7f06f4003dd0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f0704007fd0 tx=0x7f06f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdbb6948bf10a081 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.742+0000 7f06fffff640 1 -- 172.21.15.1:0/658731050 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0704007d50 con 0x7f06f40039b0 2024-03-06T07:23:57.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.742+0000 7f06fffff640 1 -- 172.21.15.1:0/658731050 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0704030070 con 0x7f06f40039b0 2024-03-06T07:23:57.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.742+0000 7f06fffff640 1 -- 172.21.15.1:0/658731050 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0704005d30 con 0x7f06f40039b0 2024-03-06T07:23:57.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.742+0000 7f070f3c9640 1 -- 172.21.15.1:0/658731050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06f40039b0 msgr2=0x7f06f4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.742+0000 7f070f3c9640 1 --2- 172.21.15.1:0/658731050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06f40039b0 0x7f06f4003dd0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f0704007fd0 tx=0x7f06f4005830 comp rx=0 tx=0).stop 2024-03-06T07:23:57.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.742+0000 7f070d966640 1 -- 172.21.15.1:0/658731050 reap_dead start 2024-03-06T07:23:57.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.742+0000 7f070f3c9640 1 -- 172.21.15.1:0/658731050 shutdown_connections 2024-03-06T07:23:57.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.742+0000 7f070f3c9640 1 -- 172.21.15.1:0/658731050 >> 172.21.15.1:0/658731050 conn(0x7f0708066ee0 msgr2=0x7f07080672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.742+0000 7f070f3c9640 1 -- 172.21.15.1:0/658731050 shutdown_connections 2024-03-06T07:23:57.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.742+0000 7f070f3c9640 1 -- 172.21.15.1:0/658731050 wait complete. 2024-03-06T07:23:57.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.744+0000 7f070f3c9640 1 Processor -- start 2024-03-06T07:23:57.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.744+0000 7f070f3c9640 1 -- start start 2024-03-06T07:23:57.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.744+0000 7f070f3c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06f40039b0 0x7f070814da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.744+0000 7f070f3c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f070814b430 0x7f07081ead30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.744+0000 7f070f3c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f070814fd30 0x7f07081eb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.744+0000 7f070f3c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07081551b0 con 0x7f070814b430 2024-03-06T07:23:57.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.744+0000 7f070f3c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07080c6fa0 con 0x7f06f40039b0 2024-03-06T07:23:57.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.744+0000 7f070f3c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0708155490 con 0x7f070814fd30 2024-03-06T07:23:57.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.744+0000 7f070c964640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f070814b430 0x7f07081ead30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.744+0000 7f070c964640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f070814b430 0x7f07081ead30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41496/0 (socket says 172.21.15.1:41496) 2024-03-06T07:23:57.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.744+0000 7f070c964640 1 -- 172.21.15.1:0/1640488316 learned_addr learned my addr 172.21.15.1:0/1640488316 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:57.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.744+0000 7f070d966640 1 --2- 172.21.15.1:0/1640488316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f070814fd30 0x7f07081eb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.745+0000 7f070d165640 1 --2- 172.21.15.1:0/1640488316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06f40039b0 0x7f070814da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.745+0000 7f070c964640 1 -- 172.21.15.1:0/1640488316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f070814fd30 msgr2=0x7f07081eb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.745+0000 7f070c964640 1 --2- 172.21.15.1:0/1640488316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f070814fd30 0x7f07081eb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.745+0000 7f070c964640 1 -- 172.21.15.1:0/1640488316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06f40039b0 msgr2=0x7f070814da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.745+0000 7f070c964640 1 --2- 172.21.15.1:0/1640488316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06f40039b0 0x7f070814da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.745+0000 7f070c964640 1 -- 172.21.15.1:0/1640488316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0708151800 con 0x7f070814b430 2024-03-06T07:23:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.745+0000 7f070d165640 1 --2- 172.21.15.1:0/1640488316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06f40039b0 0x7f070814da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.745+0000 7f070c964640 1 --2- 172.21.15.1:0/1640488316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f070814b430 0x7f07081ead30 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f06f8011960 tx=0x7f06f8011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.745+0000 7f06fdffb640 1 -- 172.21.15.1:0/1640488316 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f800faf0 con 0x7f070814b430 2024-03-06T07:23:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.745+0000 7f06fdffb640 1 -- 172.21.15.1:0/1640488316 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f800fc90 con 0x7f070814b430 2024-03-06T07:23:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.745+0000 7f070f3c9640 1 -- 172.21.15.1:0/1640488316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f070814b150 con 0x7f070814b430 2024-03-06T07:23:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.745+0000 7f06fdffb640 1 -- 172.21.15.1:0/1640488316 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f8014440 con 0x7f070814b430 2024-03-06T07:23:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.745+0000 7f070f3c9640 1 -- 172.21.15.1:0/1640488316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07080c4360 con 0x7f070814b430 2024-03-06T07:23:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.746+0000 7f070f3c9640 1 -- 172.21.15.1:0/1640488316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0708154610 con 0x7f070814b430 2024-03-06T07:23:57.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.747+0000 7f06fdffb640 1 -- 172.21.15.1:0/1640488316 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f06f8012040 con 0x7f070814b430 2024-03-06T07:23:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.747+0000 7f06fdffb640 1 --2- 172.21.15.1:0/1640488316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f06ec06b4d0 0x7f06ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.747+0000 7f06fdffb640 1 -- 172.21.15.1:0/1640488316 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f06f808a980 con 0x7f070814b430 2024-03-06T07:23:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.747+0000 7f070d165640 1 --2- 172.21.15.1:0/1640488316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f06ec06b4d0 0x7f06ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.748+0000 7f070d165640 1 --2- 172.21.15.1:0/1640488316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f06ec06b4d0 0x7f06ec06d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0704034000 tx=0x7f0704007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.751+0000 7f06fdffb640 1 -- 172.21.15.1:0/1640488316 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0708154610 con 0x7f070814b430 2024-03-06T07:23:57.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:57.969+0000 7f070f3c9640 1 -- 172.21.15.1:0/1640488316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f07081515f0 con 0x7f070814b430 2024-03-06T07:23:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.621+0000 7f06fdffb640 1 -- 172.21.15.1:0/1640488316 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v84) v1 ==== 104+0+0 (secure 0 0 0) 0x7f06f8019030 con 0x7f070814b430 2024-03-06T07:23:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.624+0000 7f070f3c9640 1 -- 172.21.15.1:0/1640488316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f06ec06b4d0 msgr2=0x7f06ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.624+0000 7f070f3c9640 1 --2- 172.21.15.1:0/1640488316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f06ec06b4d0 0x7f06ec06d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0704034000 tx=0x7f0704007000 comp rx=0 tx=0).stop 2024-03-06T07:23:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.624+0000 7f070f3c9640 1 -- 172.21.15.1:0/1640488316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f070814b430 msgr2=0x7f07081ead30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.624+0000 7f070f3c9640 1 --2- 172.21.15.1:0/1640488316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f070814b430 0x7f07081ead30 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f06f8011960 tx=0x7f06f8011e20 comp rx=0 tx=0).stop 2024-03-06T07:23:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.624+0000 7f070f3c9640 1 -- 172.21.15.1:0/1640488316 shutdown_connections 2024-03-06T07:23:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.624+0000 7f070f3c9640 1 --2- 172.21.15.1:0/1640488316 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f06ec06b4d0 0x7f06ec06d990 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.624+0000 7f070f3c9640 1 --2- 172.21.15.1:0/1640488316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f070814fd30 0x7f07081eb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.624+0000 7f070f3c9640 1 --2- 172.21.15.1:0/1640488316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f070814b430 0x7f07081ead30 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.624+0000 7f070f3c9640 1 --2- 172.21.15.1:0/1640488316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06f40039b0 0x7f070814da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.624+0000 7f070f3c9640 1 -- 172.21.15.1:0/1640488316 >> 172.21.15.1:0/1640488316 conn(0x7f0708066ee0 msgr2=0x7f07081de700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.624+0000 7f070f3c9640 1 -- 172.21.15.1:0/1640488316 shutdown_connections 2024-03-06T07:23:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.624+0000 7f070f3c9640 1 -- 172.21.15.1:0/1640488316 wait complete. 2024-03-06T07:23:58.639 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:23:58.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.749+0000 7fe424a76640 1 Processor -- start 2024-03-06T07:23:58.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.749+0000 7fe424a76640 1 -- start start 2024-03-06T07:23:58.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.750+0000 7fe424a76640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4200c70c0 0x7fe420155550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.750+0000 7fe424a76640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe420155a90 0x7fe42015fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.750+0000 7fe424a76640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe420158890 con 0x7fe4200c16e0 2024-03-06T07:23:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.750+0000 7fe424a76640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe420158b70 con 0x7fe420155a90 2024-03-06T07:23:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.750+0000 7fe424a76640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe420158e50 con 0x7fe4200c70c0 2024-03-06T07:23:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.750+0000 7fe41e59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe4200c16e0 0x7fe420152470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37382/0 (socket says 172.21.15.1:37382) 2024-03-06T07:23:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.750+0000 7fe41e59c640 1 -- 172.21.15.1:0/179311044 learned_addr learned my addr 172.21.15.1:0/179311044 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.751+0000 7fe41ed9d640 1 --2- 172.21.15.1:0/179311044 >> v2:172.21.15.98:3300/0 conn(0x7fe420155a90 0x7fe42015fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.751+0000 7fe41dd9b640 1 --2- 172.21.15.1:0/179311044 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4200c70c0 0x7fe420155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.751+0000 7fe41d59a640 1 -- 172.21.15.1:0/179311044 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790746870 0 0) 0x7fe420158890 con 0x7fe4200c16e0 2024-03-06T07:23:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.751+0000 7fe41d59a640 1 -- 172.21.15.1:0/179311044 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe408003520 con 0x7fe4200c16e0 2024-03-06T07:23:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.751+0000 7fe41d59a640 1 -- 172.21.15.1:0/179311044 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 904763534 0 0) 0x7fe408003520 con 0x7fe4200c16e0 2024-03-06T07:23:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.751+0000 7fe41ed9d640 1 -- 172.21.15.1:0/179311044 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4200c70c0 msgr2=0x7fe420155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.751+0000 7fe41ed9d640 1 --2- 172.21.15.1:0/179311044 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4200c70c0 0x7fe420155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.751+0000 7fe41ed9d640 1 -- 172.21.15.1:0/179311044 >> v1:172.21.15.1:6789/0 conn(0x7fe4200c16e0 legacy=0x7fe420152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.751+0000 7fe41ed9d640 1 -- 172.21.15.1:0/179311044 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe420159130 con 0x7fe420155a90 2024-03-06T07:23:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.751+0000 7fe41dd9b640 1 --2- 172.21.15.1:0/179311044 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4200c70c0 0x7fe420155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.752+0000 7fe41ed9d640 1 --2- 172.21.15.1:0/179311044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe420155a90 0x7fe42015fc20 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe418009d90 tx=0x7fe41802bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4b06554feda901a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.752+0000 7fe41d59a640 1 -- 172.21.15.1:0/179311044 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe418039070 con 0x7fe420155a90 2024-03-06T07:23:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.752+0000 7fe41d59a640 1 -- 172.21.15.1:0/179311044 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41802c810 con 0x7fe420155a90 2024-03-06T07:23:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.752+0000 7fe41d59a640 1 -- 172.21.15.1:0/179311044 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe410001670 con 0x7fe420155a90 2024-03-06T07:23:58.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.752+0000 7fe424a76640 1 -- 172.21.15.1:0/179311044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe420155a90 msgr2=0x7fe42015fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.752+0000 7fe424a76640 1 --2- 172.21.15.1:0/179311044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe420155a90 0x7fe42015fc20 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe418009d90 tx=0x7fe41802bce0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.753+0000 7fe424a76640 1 -- 172.21.15.1:0/179311044 shutdown_connections 2024-03-06T07:23:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.753+0000 7fe424a76640 1 --2- 172.21.15.1:0/179311044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe420155a90 0x7fe42015fc20 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.753+0000 7fe424a76640 1 --2- 172.21.15.1:0/179311044 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4200c70c0 0x7fe420155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.753+0000 7fe424a76640 1 -- 172.21.15.1:0/179311044 >> 172.21.15.1:0/179311044 conn(0x7fe420066d00 msgr2=0x7fe420067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.753+0000 7fe424a76640 1 -- 172.21.15.1:0/179311044 shutdown_connections 2024-03-06T07:23:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.753+0000 7fe424a76640 1 -- 172.21.15.1:0/179311044 wait complete. 2024-03-06T07:23:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.754+0000 7fe424a76640 1 Processor -- start 2024-03-06T07:23:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.754+0000 7fe424a76640 1 -- start start 2024-03-06T07:23:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.754+0000 7fe424a76640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4200c16e0 0x7fe420151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.754+0000 7fe424a76640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4200c70c0 0x7fe4201532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.754+0000 7fe424a76640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe420155a90 0x7fe420153800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe424a76640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4200c68c0 con 0x7fe4200c70c0 2024-03-06T07:23:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe424a76640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4201596f0 con 0x7fe4200c16e0 2024-03-06T07:23:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe424a76640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4200cb470 con 0x7fe420155a90 2024-03-06T07:23:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe41dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4200c70c0 0x7fe4201532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe41dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4200c70c0 0x7fe4201532c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41512/0 (socket says 172.21.15.1:41512) 2024-03-06T07:23:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe41dd9b640 1 -- 172.21.15.1:0/803802881 learned_addr learned my addr 172.21.15.1:0/803802881 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe41e59c640 1 --2- 172.21.15.1:0/803802881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4200c16e0 0x7fe420151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe41ed9d640 1 --2- 172.21.15.1:0/803802881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe420155a90 0x7fe420153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe41dd9b640 1 -- 172.21.15.1:0/803802881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe420155a90 msgr2=0x7fe420153800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe41dd9b640 1 --2- 172.21.15.1:0/803802881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe420155a90 0x7fe420153800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe41dd9b640 1 -- 172.21.15.1:0/803802881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4200c16e0 msgr2=0x7fe420151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe41dd9b640 1 --2- 172.21.15.1:0/803802881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4200c16e0 0x7fe420151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe41dd9b640 1 -- 172.21.15.1:0/803802881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4201ebd60 con 0x7fe4200c70c0 2024-03-06T07:23:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe41dd9b640 1 --2- 172.21.15.1:0/803802881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4200c70c0 0x7fe4201532c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fe41400ebe0 tx=0x7fe41400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe40effd640 1 -- 172.21.15.1:0/803802881 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe414017070 con 0x7fe4200c70c0 2024-03-06T07:23:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe40effd640 1 -- 172.21.15.1:0/803802881 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41400cc30 con 0x7fe4200c70c0 2024-03-06T07:23:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe424a76640 1 -- 172.21.15.1:0/803802881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4201545b0 con 0x7fe4200c70c0 2024-03-06T07:23:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe40effd640 1 -- 172.21.15.1:0/803802881 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe414013470 con 0x7fe4200c70c0 2024-03-06T07:23:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.755+0000 7fe424a76640 1 -- 172.21.15.1:0/803802881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe420154b70 con 0x7fe4200c70c0 2024-03-06T07:23:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.756+0000 7fe424a76640 1 -- 172.21.15.1:0/803802881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4200680b0 con 0x7fe4200c70c0 2024-03-06T07:23:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.757+0000 7fe40effd640 1 -- 172.21.15.1:0/803802881 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe41401a070 con 0x7fe4200c70c0 2024-03-06T07:23:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.757+0000 7fe40effd640 1 --2- 172.21.15.1:0/803802881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe40006b500 0x7fe40006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.757+0000 7fe40effd640 1 -- 172.21.15.1:0/803802881 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fe41408a5b0 con 0x7fe4200c70c0 2024-03-06T07:23:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.757+0000 7fe41e59c640 1 --2- 172.21.15.1:0/803802881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe40006b500 0x7fe40006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.758+0000 7fe41e59c640 1 --2- 172.21.15.1:0/803802881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe40006b500 0x7fe40006d9c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe4100013c0 tx=0x7fe410001350 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.762+0000 7fe40effd640 1 -- 172.21.15.1:0/803802881 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe414058c30 con 0x7fe4200c70c0 2024-03-06T07:23:58.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:58.941+0000 7fe424a76640 1 -- 172.21.15.1:0/803802881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe4200b0880 con 0x7fe4200c70c0 2024-03-06T07:23:59.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.792+0000 7fe40effd640 1 -- 172.21.15.1:0/803802881 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v98) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe41405ccd0 con 0x7fe4200c70c0 2024-03-06T07:23:59.794 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:23:59.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.798+0000 7fe424a76640 1 -- 172.21.15.1:0/803802881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe40006b500 msgr2=0x7fe40006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.798+0000 7fe424a76640 1 --2- 172.21.15.1:0/803802881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe40006b500 0x7fe40006d9c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe4100013c0 tx=0x7fe410001350 comp rx=0 tx=0).stop 2024-03-06T07:23:59.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.798+0000 7fe424a76640 1 -- 172.21.15.1:0/803802881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4200c70c0 msgr2=0x7fe4201532c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.798+0000 7fe424a76640 1 --2- 172.21.15.1:0/803802881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4200c70c0 0x7fe4201532c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fe41400ebe0 tx=0x7fe41400c520 comp rx=0 tx=0).stop 2024-03-06T07:23:59.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.801+0000 7fe424a76640 1 -- 172.21.15.1:0/803802881 shutdown_connections 2024-03-06T07:23:59.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.801+0000 7fe424a76640 1 --2- 172.21.15.1:0/803802881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe40006b500 0x7fe40006d9c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.801+0000 7fe424a76640 1 --2- 172.21.15.1:0/803802881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe420155a90 0x7fe420153800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.801+0000 7fe424a76640 1 --2- 172.21.15.1:0/803802881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4200c70c0 0x7fe4201532c0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.801+0000 7fe424a76640 1 --2- 172.21.15.1:0/803802881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4200c16e0 0x7fe420151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.801+0000 7fe424a76640 1 -- 172.21.15.1:0/803802881 >> 172.21.15.1:0/803802881 conn(0x7fe420066d00 msgr2=0x7fe4200c0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.801+0000 7fe424a76640 1 -- 172.21.15.1:0/803802881 shutdown_connections 2024-03-06T07:23:59.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.801+0000 7fe424a76640 1 -- 172.21.15.1:0/803802881 wait complete. 2024-03-06T07:23:59.817 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:23:59.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.925+0000 7f3ae91b5640 1 Processor -- start 2024-03-06T07:23:59.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.925+0000 7f3ae91b5640 1 -- start start 2024-03-06T07:23:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.926+0000 7f3ae91b5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ae41527a0 0x7f3ae4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.926+0000 7f3ae91b5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae4152b80 con 0x7f3ae40c2900 2024-03-06T07:23:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.926+0000 7f3ae91b5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae41589b0 con 0x7f3ae40c3ec0 2024-03-06T07:23:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.926+0000 7f3ae91b5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae4158760 con 0x7f3ae41527a0 2024-03-06T07:23:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.926+0000 7f3ae259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3ae40c2900 0x7f3ae406f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37384/0 (socket says 172.21.15.1:37384) 2024-03-06T07:23:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.926+0000 7f3ae259c640 1 -- 172.21.15.1:0/1980600580 learned_addr learned my addr 172.21.15.1:0/1980600580 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:23:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae2d9d640 1 --2- 172.21.15.1:0/1980600580 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ae41527a0 0x7f3ae4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae1d9b640 1 -- 172.21.15.1:0/1980600580 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148188839 0 0) 0x7f3ae4152b80 con 0x7f3ae40c2900 2024-03-06T07:23:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae1d9b640 1 -- 172.21.15.1:0/1980600580 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3acc003520 con 0x7f3ae40c2900 2024-03-06T07:23:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae1d9b640 1 -- 172.21.15.1:0/1980600580 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3113734054 0 0) 0x7f3acc003520 con 0x7f3ae40c2900 2024-03-06T07:23:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae1d9b640 1 -- 172.21.15.1:0/1980600580 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ae41527a0 msgr2=0x7f3ae4155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae1d9b640 1 --2- 172.21.15.1:0/1980600580 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ae41527a0 0x7f3ae4155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae1d9b640 1 -- 172.21.15.1:0/1980600580 >> v1:172.21.15.98:6789/0 conn(0x7f3ae40c3ec0 legacy=0x7f3ae40c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae1d9b640 1 -- 172.21.15.1:0/1980600580 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae4158c90 con 0x7f3ae40c2900 2024-03-06T07:23:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae1d9b640 1 -- 172.21.15.1:0/1980600580 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f3ad8001670 con 0x7f3ae40c2900 2024-03-06T07:23:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae1d9b640 1 -- 172.21.15.1:0/1980600580 >> v1:172.21.15.1:6789/0 conn(0x7f3ae40c2900 legacy=0x7f3ae406f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae1d9b640 1 --2- 172.21.15.1:0/1980600580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3acc0039e0 0x7f3acc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae1d9b640 1 --2- 172.21.15.1:0/1980600580 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ae41527a0 0x7f3acc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae1d9b640 1 --2- 172.21.15.1:0/1980600580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3acc004d40 0x7f3acc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae1d9b640 1 -- 172.21.15.1:0/1980600580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae4158760 con 0x7f3acc004d40 2024-03-06T07:23:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae1d9b640 1 -- 172.21.15.1:0/1980600580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc00ba70 con 0x7f3acc0039e0 2024-03-06T07:23:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae1d9b640 1 -- 172.21.15.1:0/1980600580 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc00bc70 con 0x7f3ae41527a0 2024-03-06T07:23:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae359e640 1 --2- 172.21.15.1:0/1980600580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3acc004d40 0x7f3acc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae2d9d640 1 --2- 172.21.15.1:0/1980600580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3acc0039e0 0x7f3acc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae259c640 1 --2- 172.21.15.1:0/1980600580 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ae41527a0 0x7f3acc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.927+0000 7f3ae359e640 1 -- 172.21.15.1:0/1980600580 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ae41527a0 msgr2=0x7f3acc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.928+0000 7f3ae359e640 1 --2- 172.21.15.1:0/1980600580 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ae41527a0 0x7f3acc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.928+0000 7f3ae359e640 1 -- 172.21.15.1:0/1980600580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3acc0039e0 msgr2=0x7f3acc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.928+0000 7f3ae359e640 1 --2- 172.21.15.1:0/1980600580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3acc0039e0 0x7f3acc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.928+0000 7f3ae359e640 1 -- 172.21.15.1:0/1980600580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc003520 con 0x7f3acc004d40 2024-03-06T07:23:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.928+0000 7f3ae359e640 1 --2- 172.21.15.1:0/1980600580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3acc004d40 0x7f3acc0051c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f3adc001f60 tx=0x7f3acc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358514f292374a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.928+0000 7f3ae1d9b640 1 -- 172.21.15.1:0/1980600580 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc001b30 con 0x7f3acc004d40 2024-03-06T07:23:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.928+0000 7f3ae1d9b640 1 -- 172.21.15.1:0/1980600580 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3adc02d710 con 0x7f3acc004d40 2024-03-06T07:23:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.928+0000 7f3ae1d9b640 1 -- 172.21.15.1:0/1980600580 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc02db00 con 0x7f3acc004d40 2024-03-06T07:23:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.929+0000 7f3ae91b5640 1 -- 172.21.15.1:0/1980600580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3acc004d40 msgr2=0x7f3acc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.929+0000 7f3ae91b5640 1 --2- 172.21.15.1:0/1980600580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3acc004d40 0x7f3acc0051c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f3adc001f60 tx=0x7f3acc005830 comp rx=0 tx=0).stop 2024-03-06T07:23:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.929+0000 7f3ae359e640 1 -- 172.21.15.1:0/1980600580 reap_dead start 2024-03-06T07:23:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.929+0000 7f3ae91b5640 1 -- 172.21.15.1:0/1980600580 shutdown_connections 2024-03-06T07:23:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.929+0000 7f3ae91b5640 1 -- 172.21.15.1:0/1980600580 >> 172.21.15.1:0/1980600580 conn(0x7f3ae40aa6a0 msgr2=0x7f3ae40aaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.929+0000 7f3ae91b5640 1 -- 172.21.15.1:0/1980600580 shutdown_connections 2024-03-06T07:23:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.929+0000 7f3ae91b5640 1 -- 172.21.15.1:0/1980600580 wait complete. 2024-03-06T07:23:59.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.930+0000 7f3ae91b5640 1 Processor -- start 2024-03-06T07:23:59.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.930+0000 7f3ae91b5640 1 -- start start 2024-03-06T07:23:59.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.931+0000 7f3ae91b5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae40a9d70 0x7f3ae406db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.931+0000 7f3ae91b5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ae4150240 0x7f3ae414a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.931+0000 7f3ae91b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ae414a9f0 0x7f3ae41eadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.931+0000 7f3ae91b5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae40704b0 con 0x7f3ae40a9d70 2024-03-06T07:23:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.931+0000 7f3ae91b5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae41592e0 con 0x7f3ae414a9f0 2024-03-06T07:23:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.931+0000 7f3ae91b5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae40c4a60 con 0x7f3ae4150240 2024-03-06T07:23:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.931+0000 7f3ae2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae40a9d70 0x7f3ae406db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.931+0000 7f3ae2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae40a9d70 0x7f3ae406db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41532/0 (socket says 172.21.15.1:41532) 2024-03-06T07:23:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.931+0000 7f3ae2d9d640 1 -- 172.21.15.1:0/2289645772 learned_addr learned my addr 172.21.15.1:0/2289645772 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:23:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.932+0000 7f3ae259c640 1 --2- 172.21.15.1:0/2289645772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ae4150240 0x7f3ae414a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.932+0000 7f3ae2d9d640 1 -- 172.21.15.1:0/2289645772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ae4150240 msgr2=0x7f3ae414a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.932+0000 7f3ae2d9d640 1 --2- 172.21.15.1:0/2289645772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ae4150240 0x7f3ae414a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.932+0000 7f3ae359e640 1 --2- 172.21.15.1:0/2289645772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ae414a9f0 0x7f3ae41eadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.932+0000 7f3ae2d9d640 1 -- 172.21.15.1:0/2289645772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ae414a9f0 msgr2=0x7f3ae41eadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.932+0000 7f3ae2d9d640 1 --2- 172.21.15.1:0/2289645772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ae414a9f0 0x7f3ae41eadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.932+0000 7f3ae2d9d640 1 -- 172.21.15.1:0/2289645772 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae41ee550 con 0x7f3ae40a9d70 2024-03-06T07:23:59.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.932+0000 7f3ae2d9d640 1 --2- 172.21.15.1:0/2289645772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae40a9d70 0x7f3ae406db20 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f3ad40119a0 tx=0x7f3ad4011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.932+0000 7f3acb7fe640 1 -- 172.21.15.1:0/2289645772 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad400faf0 con 0x7f3ae40a9d70 2024-03-06T07:23:59.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.932+0000 7f3acb7fe640 1 -- 172.21.15.1:0/2289645772 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad400fc90 con 0x7f3ae40a9d70 2024-03-06T07:23:59.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.932+0000 7f3ae91b5640 1 -- 172.21.15.1:0/2289645772 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae41ee830 con 0x7f3ae40a9d70 2024-03-06T07:23:59.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.932+0000 7f3acb7fe640 1 -- 172.21.15.1:0/2289645772 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4014440 con 0x7f3ae40a9d70 2024-03-06T07:23:59.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.932+0000 7f3ae91b5640 1 -- 172.21.15.1:0/2289645772 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae41eedf0 con 0x7f3ae40a9d70 2024-03-06T07:23:59.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.933+0000 7f3ae91b5640 1 -- 172.21.15.1:0/2289645772 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae406e9f0 con 0x7f3ae40a9d70 2024-03-06T07:23:59.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.934+0000 7f3acb7fe640 1 -- 172.21.15.1:0/2289645772 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ad4012040 con 0x7f3ae40a9d70 2024-03-06T07:23:59.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.934+0000 7f3acb7fe640 1 --2- 172.21.15.1:0/2289645772 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3ad006b4d0 0x7f3ad006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.934+0000 7f3acb7fe640 1 -- 172.21.15.1:0/2289645772 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5835+0+0 (secure 0 0 0) 0x7f3ad408a940 con 0x7f3ae40a9d70 2024-03-06T07:23:59.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.934+0000 7f3ae259c640 1 --2- 172.21.15.1:0/2289645772 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3ad006b4d0 0x7f3ad006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.935+0000 7f3ae259c640 1 --2- 172.21.15.1:0/2289645772 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3ad006b4d0 0x7f3ad006d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3ae414fa70 tx=0x7f3ad8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:23:59.938+0000 7f3acb7fe640 1 -- 172.21.15.1:0/2289645772 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ae406e9f0 con 0x7f3ae40a9d70 2024-03-06T07:24:00.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.118+0000 7f3ae91b5640 1 -- 172.21.15.1:0/2289645772 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3ae40680b0 con 0x7f3ae40a9d70 2024-03-06T07:24:00.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.797+0000 7f3acb7fe640 1 -- 172.21.15.1:0/2289645772 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v99) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3ad4019030 con 0x7f3ae40a9d70 2024-03-06T07:24:00.800 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-06T07:24:00.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.801+0000 7f3ae91b5640 1 -- 172.21.15.1:0/2289645772 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3ad006b4d0 msgr2=0x7f3ad006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.801+0000 7f3ae91b5640 1 --2- 172.21.15.1:0/2289645772 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3ad006b4d0 0x7f3ad006d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3ae414fa70 tx=0x7f3ad8009040 comp rx=0 tx=0).stop 2024-03-06T07:24:00.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.801+0000 7f3ae91b5640 1 -- 172.21.15.1:0/2289645772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae40a9d70 msgr2=0x7f3ae406db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.801+0000 7f3ae91b5640 1 --2- 172.21.15.1:0/2289645772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae40a9d70 0x7f3ae406db20 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f3ad40119a0 tx=0x7f3ad4011e60 comp rx=0 tx=0).stop 2024-03-06T07:24:00.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.801+0000 7f3ae91b5640 1 -- 172.21.15.1:0/2289645772 shutdown_connections 2024-03-06T07:24:00.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.801+0000 7f3ae91b5640 1 --2- 172.21.15.1:0/2289645772 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3ad006b4d0 0x7f3ad006d990 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.801+0000 7f3ae91b5640 1 --2- 172.21.15.1:0/2289645772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ae414a9f0 0x7f3ae41eadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.801+0000 7f3ae91b5640 1 --2- 172.21.15.1:0/2289645772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ae4150240 0x7f3ae414a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.801+0000 7f3ae91b5640 1 --2- 172.21.15.1:0/2289645772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae40a9d70 0x7f3ae406db20 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.801+0000 7f3ae91b5640 1 -- 172.21.15.1:0/2289645772 >> 172.21.15.1:0/2289645772 conn(0x7f3ae40aa6a0 msgr2=0x7f3ae41544f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.801+0000 7f3ae91b5640 1 -- 172.21.15.1:0/2289645772 shutdown_connections 2024-03-06T07:24:00.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.801+0000 7f3ae91b5640 1 -- 172.21.15.1:0/2289645772 wait complete. 2024-03-06T07:24:00.821 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:24:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.928+0000 7fd1d92af640 1 Processor -- start 2024-03-06T07:24:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.928+0000 7fd1d92af640 1 -- start start 2024-03-06T07:24:00.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.929+0000 7fd1d92af640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1d4149a90 0x7fd1d40c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.929+0000 7fd1d92af640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1d40c16c0 0x7fd1d4156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.929+0000 7fd1d92af640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d41530a0 con 0x7fd1d40c1260 2024-03-06T07:24:00.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.929+0000 7fd1d92af640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d4153380 con 0x7fd1d40c16c0 2024-03-06T07:24:00.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.929+0000 7fd1d92af640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d4153660 con 0x7fd1d4149a90 2024-03-06T07:24:00.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.929+0000 7fd1d37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1d40c16c0 0x7fd1d4156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.929+0000 7fd1d37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1d40c16c0 0x7fd1d4156cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41546/0 (socket says 172.21.15.1:41546) 2024-03-06T07:24:00.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.929+0000 7fd1d37fe640 1 -- 172.21.15.1:0/2656192680 learned_addr learned my addr 172.21.15.1:0/2656192680 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:00.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.929+0000 7fd1d2ffd640 1 --2- 172.21.15.1:0/2656192680 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1d4149a90 0x7fd1d40c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.929+0000 7fd1d37fe640 1 -- 172.21.15.1:0/2656192680 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1d4149a90 msgr2=0x7fd1d40c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:00.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.929+0000 7fd1d37fe640 1 --2- 172.21.15.1:0/2656192680 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1d4149a90 0x7fd1d40c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.929+0000 7fd1d37fe640 1 -- 172.21.15.1:0/2656192680 >> v1:172.21.15.98:6789/0 conn(0x7fd1d40c1260 legacy=0x7fd1d41565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.929+0000 7fd1d37fe640 1 -- 172.21.15.1:0/2656192680 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d4153940 con 0x7fd1d40c16c0 2024-03-06T07:24:00.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.930+0000 7fd1d37fe640 1 --2- 172.21.15.1:0/2656192680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1d40c16c0 0x7fd1d4156cb0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc009e50 tx=0x7fd1cc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5990b532f08edc36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.930+0000 7fd1d27fc640 1 -- 172.21.15.1:0/2656192680 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc02ca70 con 0x7fd1d40c16c0 2024-03-06T07:24:00.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.930+0000 7fd1d27fc640 1 -- 172.21.15.1:0/2656192680 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1cc02cc10 con 0x7fd1d40c16c0 2024-03-06T07:24:00.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.930+0000 7fd1d27fc640 1 -- 172.21.15.1:0/2656192680 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc0343f0 con 0x7fd1d40c16c0 2024-03-06T07:24:00.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.931+0000 7fd1d92af640 1 -- 172.21.15.1:0/2656192680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1d40c16c0 msgr2=0x7fd1d4156cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.931+0000 7fd1d92af640 1 --2- 172.21.15.1:0/2656192680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1d40c16c0 0x7fd1d4156cb0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc009e50 tx=0x7fd1cc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.931+0000 7fd1d92af640 1 -- 172.21.15.1:0/2656192680 shutdown_connections 2024-03-06T07:24:00.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.931+0000 7fd1d92af640 1 --2- 172.21.15.1:0/2656192680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1d40c16c0 0x7fd1d4156cb0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.931+0000 7fd1d92af640 1 --2- 172.21.15.1:0/2656192680 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1d4149a90 0x7fd1d40c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.931+0000 7fd1d92af640 1 -- 172.21.15.1:0/2656192680 >> 172.21.15.1:0/2656192680 conn(0x7fd1d40c6860 msgr2=0x7fd1d40c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.931+0000 7fd1d92af640 1 -- 172.21.15.1:0/2656192680 shutdown_connections 2024-03-06T07:24:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.931+0000 7fd1d92af640 1 -- 172.21.15.1:0/2656192680 wait complete. 2024-03-06T07:24:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.932+0000 7fd1d92af640 1 Processor -- start 2024-03-06T07:24:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.932+0000 7fd1d92af640 1 -- start start 2024-03-06T07:24:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.932+0000 7fd1d92af640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1d40c1260 0x7fd1d41455b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.932+0000 7fd1d92af640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d40c16c0 0x7fd1d40bfc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.932+0000 7fd1d92af640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1d4149a90 0x7fd1d4145ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.932+0000 7fd1d92af640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d40c4ca0 con 0x7fd1d40c1260 2024-03-06T07:24:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.932+0000 7fd1d92af640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d41571f0 con 0x7fd1d40c16c0 2024-03-06T07:24:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.932+0000 7fd1d92af640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d406f4a0 con 0x7fd1d4149a90 2024-03-06T07:24:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.933+0000 7fd1d2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1d40c1260 0x7fd1d41455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.933+0000 7fd1d2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1d40c1260 0x7fd1d41455b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41558/0 (socket says 172.21.15.1:41558) 2024-03-06T07:24:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.933+0000 7fd1d2ffd640 1 -- 172.21.15.1:0/4212122971 learned_addr learned my addr 172.21.15.1:0/4212122971 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.933+0000 7fd1d37fe640 1 --2- 172.21.15.1:0/4212122971 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1d4149a90 0x7fd1d4145ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.933+0000 7fd1cbfff640 1 --2- 172.21.15.1:0/4212122971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d40c16c0 0x7fd1d40bfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.933+0000 7fd1d2ffd640 1 -- 172.21.15.1:0/4212122971 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1d4149a90 msgr2=0x7fd1d4145ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.933+0000 7fd1d2ffd640 1 --2- 172.21.15.1:0/4212122971 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1d4149a90 0x7fd1d4145ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.933+0000 7fd1d2ffd640 1 -- 172.21.15.1:0/4212122971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d40c16c0 msgr2=0x7fd1d40bfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.933+0000 7fd1d2ffd640 1 --2- 172.21.15.1:0/4212122971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d40c16c0 0x7fd1d40bfc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.933+0000 7fd1d2ffd640 1 -- 172.21.15.1:0/4212122971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d4150c50 con 0x7fd1d40c1260 2024-03-06T07:24:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.933+0000 7fd1d2ffd640 1 --2- 172.21.15.1:0/4212122971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1d40c1260 0x7fd1d41455b0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400d830 tx=0x7fd1c400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.933+0000 7fd1cb7fe640 1 -- 172.21.15.1:0/4212122971 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c400b6c0 con 0x7fd1d40c1260 2024-03-06T07:24:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.933+0000 7fd1cb7fe640 1 -- 172.21.15.1:0/4212122971 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c4019070 con 0x7fd1d40c1260 2024-03-06T07:24:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.933+0000 7fd1d92af640 1 -- 172.21.15.1:0/4212122971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d4150970 con 0x7fd1d40c1260 2024-03-06T07:24:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.933+0000 7fd1cb7fe640 1 -- 172.21.15.1:0/4212122971 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c400bac0 con 0x7fd1d40c1260 2024-03-06T07:24:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.933+0000 7fd1d92af640 1 -- 172.21.15.1:0/4212122971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d41ee3e0 con 0x7fd1d40c1260 2024-03-06T07:24:00.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.934+0000 7fd1d92af640 1 -- 172.21.15.1:0/4212122971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d40c3410 con 0x7fd1d40c1260 2024-03-06T07:24:00.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.935+0000 7fd1cb7fe640 1 -- 172.21.15.1:0/4212122971 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1c401e460 con 0x7fd1d40c1260 2024-03-06T07:24:00.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.935+0000 7fd1cb7fe640 1 --2- 172.21.15.1:0/4212122971 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd1b406b4d0 0x7fd1b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.935+0000 7fd1cb7fe640 1 -- 172.21.15.1:0/4212122971 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fd1c408a810 con 0x7fd1d40c1260 2024-03-06T07:24:00.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.935+0000 7fd1cbfff640 1 --2- 172.21.15.1:0/4212122971 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd1b406b4d0 0x7fd1b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.936+0000 7fd1cbfff640 1 --2- 172.21.15.1:0/4212122971 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd1b406b4d0 0x7fd1b406d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc007fd0 tx=0x7fd1bc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:00.940+0000 7fd1cb7fe640 1 -- 172.21.15.1:0/4212122971 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1d40c3410 con 0x7fd1d40c1260 2024-03-06T07:24:01.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.126+0000 7fd1d92af640 1 -- 172.21.15.1:0/4212122971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd1d4067fd0 con 0x7fd1d40c1260 2024-03-06T07:24:01.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.129+0000 7fd1cb7fe640 1 -- 172.21.15.1:0/4212122971 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd1c40597b0 con 0x7fd1d40c1260 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.131+0000 7fd1d92af640 1 -- 172.21.15.1:0/4212122971 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd1b406b4d0 msgr2=0x7fd1b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.131+0000 7fd1d92af640 1 --2- 172.21.15.1:0/4212122971 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd1b406b4d0 0x7fd1b406d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc007fd0 tx=0x7fd1bc009040 comp rx=0 tx=0).stop 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.131+0000 7fd1d92af640 1 -- 172.21.15.1:0/4212122971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1d40c1260 msgr2=0x7fd1d41455b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.131+0000 7fd1d92af640 1 --2- 172.21.15.1:0/4212122971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1d40c1260 0x7fd1d41455b0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400d830 tx=0x7fd1c400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.131+0000 7fd1d92af640 1 -- 172.21.15.1:0/4212122971 shutdown_connections 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.131+0000 7fd1d92af640 1 --2- 172.21.15.1:0/4212122971 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd1b406b4d0 0x7fd1b406d990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.131+0000 7fd1d92af640 1 --2- 172.21.15.1:0/4212122971 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1d4149a90 0x7fd1d4145ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.131+0000 7fd1d92af640 1 --2- 172.21.15.1:0/4212122971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1d40c16c0 0x7fd1d40bfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.131+0000 7fd1d92af640 1 --2- 172.21.15.1:0/4212122971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1d40c1260 0x7fd1d41455b0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.131+0000 7fd1d92af640 1 -- 172.21.15.1:0/4212122971 >> 172.21.15.1:0/4212122971 conn(0x7fd1d40c6860 msgr2=0x7fd1d4152060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.131+0000 7fd1d92af640 1 -- 172.21.15.1:0/4212122971 shutdown_connections 2024-03-06T07:24:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.131+0000 7fd1d92af640 1 -- 172.21.15.1:0/4212122971 wait complete. 2024-03-06T07:24:01.144 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.245+0000 7f4a67ceb640 1 Processor -- start 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.245+0000 7f4a67ceb640 1 -- start start 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.246+0000 7f4a67ceb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a600c4940 0x7f4a600ac930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.246+0000 7f4a67ceb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a600c7420 con 0x7f4a600c39c0 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.246+0000 7f4a67ceb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a600ab760 con 0x7f4a600c4560 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.246+0000 7f4a67ceb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a600c76c0 con 0x7f4a600c4940 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.246+0000 7f4a65286640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4a600c39c0 0x7f4a600ac220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37388/0 (socket says 172.21.15.1:37388) 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.246+0000 7f4a65286640 1 -- 172.21.15.1:0/558837552 learned_addr learned my addr 172.21.15.1:0/558837552 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.246+0000 7f4a64a85640 1 -- 172.21.15.1:0/558837552 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467992619 0 0) 0x7f4a600c7420 con 0x7f4a600c39c0 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.246+0000 7f4a64a85640 1 -- 172.21.15.1:0/558837552 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a48003520 con 0x7f4a600c39c0 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a66288640 1 --2- 172.21.15.1:0/558837552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a600c4940 0x7f4a600ac930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a64a85640 1 -- 172.21.15.1:0/558837552 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 834683966 0 0) 0x7f4a48003520 con 0x7f4a600c39c0 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a64a85640 1 -- 172.21.15.1:0/558837552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a600c4940 msgr2=0x7f4a600ac930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a64a85640 1 --2- 172.21.15.1:0/558837552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a600c4940 0x7f4a600ac930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a64a85640 1 -- 172.21.15.1:0/558837552 >> v1:172.21.15.98:6789/0 conn(0x7f4a600c4560 legacy=0x7f4a60155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a64a85640 1 -- 172.21.15.1:0/558837552 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a600aba20 con 0x7f4a600c39c0 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a64a85640 1 -- 172.21.15.1:0/558837552 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f4a54001670 con 0x7f4a600c39c0 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a64a85640 1 -- 172.21.15.1:0/558837552 >> v1:172.21.15.1:6789/0 conn(0x7f4a600c39c0 legacy=0x7f4a600ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a64a85640 1 --2- 172.21.15.1:0/558837552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a480039e0 0x7f4a48003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a64a85640 1 --2- 172.21.15.1:0/558837552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a48004800 0x7f4a48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a64a85640 1 --2- 172.21.15.1:0/558837552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a600c4940 0x7f4a480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a64a85640 1 -- 172.21.15.1:0/558837552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a600c76c0 con 0x7f4a48004800 2024-03-06T07:24:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a64a85640 1 -- 172.21.15.1:0/558837552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4800ba70 con 0x7f4a480039e0 2024-03-06T07:24:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a64a85640 1 -- 172.21.15.1:0/558837552 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4800de80 con 0x7f4a600c4940 2024-03-06T07:24:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a65286640 1 --2- 172.21.15.1:0/558837552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a48004800 0x7f4a48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a65a87640 1 --2- 172.21.15.1:0/558837552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a480039e0 0x7f4a48003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a66288640 1 --2- 172.21.15.1:0/558837552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a600c4940 0x7f4a480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a65286640 1 -- 172.21.15.1:0/558837552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a600c4940 msgr2=0x7f4a480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a65286640 1 --2- 172.21.15.1:0/558837552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a600c4940 0x7f4a480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a65286640 1 -- 172.21.15.1:0/558837552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a480039e0 msgr2=0x7f4a48003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a65286640 1 --2- 172.21.15.1:0/558837552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a480039e0 0x7f4a48003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.247+0000 7f4a65286640 1 -- 172.21.15.1:0/558837552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a48003520 con 0x7f4a48004800 2024-03-06T07:24:01.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.248+0000 7f4a65286640 1 --2- 172.21.15.1:0/558837552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a48004800 0x7f4a48004c80 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f4a5400c210 tx=0x7f4a48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb6f48e0a086159 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.248+0000 7f4a64a85640 1 -- 172.21.15.1:0/558837552 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a54007cf0 con 0x7f4a48004800 2024-03-06T07:24:01.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.248+0000 7f4a64a85640 1 -- 172.21.15.1:0/558837552 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a54030070 con 0x7f4a48004800 2024-03-06T07:24:01.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.248+0000 7f4a64a85640 1 -- 172.21.15.1:0/558837552 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a54005d30 con 0x7f4a48004800 2024-03-06T07:24:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.249+0000 7f4a67ceb640 1 -- 172.21.15.1:0/558837552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a48004800 msgr2=0x7f4a48004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.249+0000 7f4a67ceb640 1 --2- 172.21.15.1:0/558837552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a48004800 0x7f4a48004c80 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f4a5400c210 tx=0x7f4a48005830 comp rx=0 tx=0).stop 2024-03-06T07:24:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.249+0000 7f4a66288640 1 -- 172.21.15.1:0/558837552 reap_dead start 2024-03-06T07:24:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.249+0000 7f4a67ceb640 1 -- 172.21.15.1:0/558837552 shutdown_connections 2024-03-06T07:24:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.249+0000 7f4a67ceb640 1 -- 172.21.15.1:0/558837552 >> 172.21.15.1:0/558837552 conn(0x7f4a600c7a50 msgr2=0x7f4a600c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.249+0000 7f4a67ceb640 1 -- 172.21.15.1:0/558837552 shutdown_connections 2024-03-06T07:24:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.249+0000 7f4a67ceb640 1 -- 172.21.15.1:0/558837552 wait complete. 2024-03-06T07:24:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.250+0000 7f4a67ceb640 1 Processor -- start 2024-03-06T07:24:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.250+0000 7f4a67ceb640 1 -- start start 2024-03-06T07:24:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a67ceb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a48004800 0x7f4a60155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a67ceb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a600a6290 0x7f4a601e95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a67ceb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a600a6710 0x7f4a601e9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a67ceb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6006e850 con 0x7f4a600a6710 2024-03-06T07:24:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a67ceb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6006e6d0 con 0x7f4a600a6290 2024-03-06T07:24:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a67ceb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a600c63a0 con 0x7f4a48004800 2024-03-06T07:24:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a66288640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a600a6710 0x7f4a601e9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a66288640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a600a6710 0x7f4a601e9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41574/0 (socket says 172.21.15.1:41574) 2024-03-06T07:24:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a66288640 1 -- 172.21.15.1:0/3770158858 learned_addr learned my addr 172.21.15.1:0/3770158858 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a65286640 1 --2- 172.21.15.1:0/3770158858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a600a6290 0x7f4a601e95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a65a87640 1 --2- 172.21.15.1:0/3770158858 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a48004800 0x7f4a60155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a66288640 1 -- 172.21.15.1:0/3770158858 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a48004800 msgr2=0x7f4a60155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a66288640 1 --2- 172.21.15.1:0/3770158858 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a48004800 0x7f4a60155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a66288640 1 -- 172.21.15.1:0/3770158858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a600a6290 msgr2=0x7f4a601e95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a66288640 1 --2- 172.21.15.1:0/3770158858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a600a6290 0x7f4a601e95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a66288640 1 -- 172.21.15.1:0/3770158858 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a601e6390 con 0x7f4a600a6710 2024-03-06T07:24:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a65a87640 1 --2- 172.21.15.1:0/3770158858 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a48004800 0x7f4a60155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a66288640 1 --2- 172.21.15.1:0/3770158858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a600a6710 0x7f4a601e9af0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f4a50011c80 tx=0x7f4a5000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a5e7fc640 1 -- 172.21.15.1:0/3770158858 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a50019070 con 0x7f4a600a6710 2024-03-06T07:24:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a5e7fc640 1 -- 172.21.15.1:0/3770158858 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a50003990 con 0x7f4a600a6710 2024-03-06T07:24:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a67ceb640 1 -- 172.21.15.1:0/3770158858 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a601e6670 con 0x7f4a600a6710 2024-03-06T07:24:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a5e7fc640 1 -- 172.21.15.1:0/3770158858 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a50003d80 con 0x7f4a600a6710 2024-03-06T07:24:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.251+0000 7f4a67ceb640 1 -- 172.21.15.1:0/3770158858 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a601e6040 con 0x7f4a600a6710 2024-03-06T07:24:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.252+0000 7f4a67ceb640 1 -- 172.21.15.1:0/3770158858 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a600686e0 con 0x7f4a600a6710 2024-03-06T07:24:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.253+0000 7f4a5e7fc640 1 -- 172.21.15.1:0/3770158858 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a5001c070 con 0x7f4a600a6710 2024-03-06T07:24:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.253+0000 7f4a5e7fc640 1 --2- 172.21.15.1:0/3770158858 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a4c06b500 0x7f4a4c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.253+0000 7f4a5e7fc640 1 -- 172.21.15.1:0/3770158858 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f4a5008a970 con 0x7f4a600a6710 2024-03-06T07:24:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.253+0000 7f4a65a87640 1 --2- 172.21.15.1:0/3770158858 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a4c06b500 0x7f4a4c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.254+0000 7f4a65a87640 1 --2- 172.21.15.1:0/3770158858 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a4c06b500 0x7f4a4c06d9c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4a600c5520 tx=0x7f4a58009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.257+0000 7f4a5e7fc640 1 -- 172.21.15.1:0/3770158858 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a50010040 con 0x7f4a600a6710 2024-03-06T07:24:01.394 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:01.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:01.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.442+0000 7f4a67ceb640 1 -- 172.21.15.1:0/3770158858 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7f4a6006db40 con 0x7f4a600a6710 2024-03-06T07:24:01.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.446+0000 7f4a5e7fc640 1 -- 172.21.15.1:0/3770158858 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v34) v1 ==== 96+0+0 (secure 0 0 0) 0x7f4a500599a0 con 0x7f4a600a6710 2024-03-06T07:24:01.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.448+0000 7f4a67ceb640 1 -- 172.21.15.1:0/3770158858 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a4c06b500 msgr2=0x7f4a4c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.448+0000 7f4a67ceb640 1 --2- 172.21.15.1:0/3770158858 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a4c06b500 0x7f4a4c06d9c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4a600c5520 tx=0x7f4a58009040 comp rx=0 tx=0).stop 2024-03-06T07:24:01.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.448+0000 7f4a67ceb640 1 -- 172.21.15.1:0/3770158858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a600a6710 msgr2=0x7f4a601e9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.448+0000 7f4a67ceb640 1 --2- 172.21.15.1:0/3770158858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a600a6710 0x7f4a601e9af0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f4a50011c80 tx=0x7f4a5000f580 comp rx=0 tx=0).stop 2024-03-06T07:24:01.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.448+0000 7f4a67ceb640 1 -- 172.21.15.1:0/3770158858 shutdown_connections 2024-03-06T07:24:01.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.448+0000 7f4a67ceb640 1 --2- 172.21.15.1:0/3770158858 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a4c06b500 0x7f4a4c06d9c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.448+0000 7f4a67ceb640 1 --2- 172.21.15.1:0/3770158858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a600a6710 0x7f4a601e9af0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.448+0000 7f4a67ceb640 1 --2- 172.21.15.1:0/3770158858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a600a6290 0x7f4a601e95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.448+0000 7f4a67ceb640 1 --2- 172.21.15.1:0/3770158858 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a48004800 0x7f4a60155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.448+0000 7f4a67ceb640 1 -- 172.21.15.1:0/3770158858 >> 172.21.15.1:0/3770158858 conn(0x7f4a600c7a50 msgr2=0x7f4a6006f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:01.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.448+0000 7f4a67ceb640 1 -- 172.21.15.1:0/3770158858 shutdown_connections 2024-03-06T07:24:01.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.448+0000 7f4a67ceb640 1 -- 172.21.15.1:0/3770158858 wait complete. 2024-03-06T07:24:01.500 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime' 2024-03-06T07:24:01.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.601+0000 7fe9ce64e640 1 Processor -- start 2024-03-06T07:24:01.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.601+0000 7fe9ce64e640 1 -- start start 2024-03-06T07:24:01.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.602+0000 7fe9ce64e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe9c8156b70 0x7fe9c815e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.602+0000 7fe9ce64e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c8153a00 con 0x7fe9c80c4bf0 2024-03-06T07:24:01.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.602+0000 7fe9ce64e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c8153ce0 con 0x7fe9c80a7050 2024-03-06T07:24:01.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.602+0000 7fe9ce64e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c80c3cb0 con 0x7fe9c8156b70 2024-03-06T07:24:01.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.602+0000 7fe9c77fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe9c80c4bf0 0x7fe9c8156460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37404/0 (socket says 172.21.15.1:37404) 2024-03-06T07:24:01.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.602+0000 7fe9c77fe640 1 -- 172.21.15.1:0/2561804688 learned_addr learned my addr 172.21.15.1:0/2561804688 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:01.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.602+0000 7fe9ccbeb640 1 --2- 172.21.15.1:0/2561804688 >> v2:172.21.15.98:3300/0 conn(0x7fe9c8156b70 0x7fe9c815e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.602+0000 7fe9c6ffd640 1 -- 172.21.15.1:0/2561804688 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171529221 0 0) 0x7fe9c8153a00 con 0x7fe9c80c4bf0 2024-03-06T07:24:01.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.602+0000 7fe9c6ffd640 1 -- 172.21.15.1:0/2561804688 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b0003520 con 0x7fe9c80c4bf0 2024-03-06T07:24:01.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.602+0000 7fe9c6ffd640 1 -- 172.21.15.1:0/2561804688 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3148070971 0 0) 0x7fe9b0003520 con 0x7fe9c80c4bf0 2024-03-06T07:24:01.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.602+0000 7fe9ccbeb640 1 -- 172.21.15.1:0/2561804688 >> v1:172.21.15.98:6789/0 conn(0x7fe9c80a7050 legacy=0x7fe9c80c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.602+0000 7fe9ccbeb640 1 -- 172.21.15.1:0/2561804688 >> v1:172.21.15.1:6789/0 conn(0x7fe9c80c4bf0 legacy=0x7fe9c8156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.602+0000 7fe9ccbeb640 1 -- 172.21.15.1:0/2561804688 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c80c3f90 con 0x7fe9c8156b70 2024-03-06T07:24:01.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.603+0000 7fe9ccbeb640 1 --2- 172.21.15.1:0/2561804688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c8156b70 0x7fe9c815e770 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0009d90 tx=0x7fe9c002bb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddf69b71dad7b91f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.603+0000 7fe9c6ffd640 1 -- 172.21.15.1:0/2561804688 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b8001670 con 0x7fe9c8156b70 2024-03-06T07:24:01.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.603+0000 7fe9c6ffd640 1 -- 172.21.15.1:0/2561804688 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c002d690 con 0x7fe9c8156b70 2024-03-06T07:24:01.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.603+0000 7fe9c6ffd640 1 -- 172.21.15.1:0/2561804688 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c002da40 con 0x7fe9c8156b70 2024-03-06T07:24:01.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.604+0000 7fe9ce64e640 1 -- 172.21.15.1:0/2561804688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c8156b70 msgr2=0x7fe9c815e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.604+0000 7fe9ce64e640 1 --2- 172.21.15.1:0/2561804688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c8156b70 0x7fe9c815e770 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0009d90 tx=0x7fe9c002bb50 comp rx=0 tx=0).stop 2024-03-06T07:24:01.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.604+0000 7fe9ce64e640 1 -- 172.21.15.1:0/2561804688 shutdown_connections 2024-03-06T07:24:01.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.604+0000 7fe9ce64e640 1 --2- 172.21.15.1:0/2561804688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c8156b70 0x7fe9c815e770 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.604+0000 7fe9ce64e640 1 -- 172.21.15.1:0/2561804688 >> 172.21.15.1:0/2561804688 conn(0x7fe9c80c7a50 msgr2=0x7fe9c80c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:01.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.604+0000 7fe9ce64e640 1 -- 172.21.15.1:0/2561804688 shutdown_connections 2024-03-06T07:24:01.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.604+0000 7fe9ce64e640 1 -- 172.21.15.1:0/2561804688 wait complete. 2024-03-06T07:24:01.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.605+0000 7fe9ce64e640 1 Processor -- start 2024-03-06T07:24:01.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.605+0000 7fe9ce64e640 1 -- start start 2024-03-06T07:24:01.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9ce64e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c80a7050 0x7fe9c81dc440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9ce64e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9c80c4bf0 0x7fe9c80c10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9ce64e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c8156b70 0x7fe9c80ac480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9ce64e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c8156fc0 con 0x7fe9c80a7050 2024-03-06T07:24:01.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9ce64e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c8072f60 con 0x7fe9c8156b70 2024-03-06T07:24:01.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9ce64e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c81de040 con 0x7fe9c80c4bf0 2024-03-06T07:24:01.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9c7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c80a7050 0x7fe9c81dc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9c7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c80a7050 0x7fe9c81dc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41578/0 (socket says 172.21.15.1:41578) 2024-03-06T07:24:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9c7fff640 1 -- 172.21.15.1:0/211553049 learned_addr learned my addr 172.21.15.1:0/211553049 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9c77fe640 1 --2- 172.21.15.1:0/211553049 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9c80c4bf0 0x7fe9c80c10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9ccbeb640 1 --2- 172.21.15.1:0/211553049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c8156b70 0x7fe9c80ac480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9c7fff640 1 -- 172.21.15.1:0/211553049 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9c80c4bf0 msgr2=0x7fe9c80c10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9c7fff640 1 --2- 172.21.15.1:0/211553049 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9c80c4bf0 0x7fe9c80c10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9c7fff640 1 -- 172.21.15.1:0/211553049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c8156b70 msgr2=0x7fe9c80ac480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9c7fff640 1 --2- 172.21.15.1:0/211553049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c8156b70 0x7fe9c80ac480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9c7fff640 1 -- 172.21.15.1:0/211553049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c81eac00 con 0x7fe9c80a7050 2024-03-06T07:24:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9ccbeb640 1 --2- 172.21.15.1:0/211553049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c8156b70 0x7fe9c80ac480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.606+0000 7fe9c7fff640 1 --2- 172.21.15.1:0/211553049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c80a7050 0x7fe9c81dc440 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00c550 tx=0x7fe9bc00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.607+0000 7fe9c4ff9640 1 -- 172.21.15.1:0/211553049 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc017070 con 0x7fe9c80a7050 2024-03-06T07:24:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.607+0000 7fe9c4ff9640 1 -- 172.21.15.1:0/211553049 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc0035f0 con 0x7fe9c80a7050 2024-03-06T07:24:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.607+0000 7fe9ce64e640 1 -- 172.21.15.1:0/211553049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c81de2e0 con 0x7fe9c80a7050 2024-03-06T07:24:01.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.607+0000 7fe9c4ff9640 1 -- 172.21.15.1:0/211553049 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc0039e0 con 0x7fe9c80a7050 2024-03-06T07:24:01.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.607+0000 7fe9ce64e640 1 -- 172.21.15.1:0/211553049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c81eaec0 con 0x7fe9c80a7050 2024-03-06T07:24:01.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.608+0000 7fe9ce64e640 1 -- 172.21.15.1:0/211553049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c81eb0c0 con 0x7fe9c80a7050 2024-03-06T07:24:01.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.608+0000 7fe9c4ff9640 1 -- 172.21.15.1:0/211553049 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9bc01a070 con 0x7fe9c80a7050 2024-03-06T07:24:01.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.608+0000 7fe9c4ff9640 1 --2- 172.21.15.1:0/211553049 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9b406b500 0x7fe9b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.608+0000 7fe9c4ff9640 1 -- 172.21.15.1:0/211553049 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fe9bc089430 con 0x7fe9c80a7050 2024-03-06T07:24:01.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.609+0000 7fe9c77fe640 1 --2- 172.21.15.1:0/211553049 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9b406b500 0x7fe9b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.609+0000 7fe9c77fe640 1 --2- 172.21.15.1:0/211553049 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9b406b500 0x7fe9b406d9c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8001eb0 tx=0x7fe9b8001cc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.613+0000 7fe9c4ff9640 1 -- 172.21.15.1:0/211553049 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe9bc058460 con 0x7fe9c80a7050 2024-03-06T07:24:01.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:01.867+0000 7fe9ce64e640 1 -- 172.21.15.1:0/211553049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]} v 0) v1 -- 0x7fe9c80686e0 con 0x7fe9c80a7050 2024-03-06T07:24:02.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.804+0000 7fe9c4ff9640 1 -- 172.21.15.1:0/211553049 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]}]=0 v227) v1 ==== 146+0+0 (secure 0 0 0) 0x7fe9bc057d60 con 0x7fe9c80a7050 2024-03-06T07:24:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.806+0000 7fe9ce64e640 1 -- 172.21.15.1:0/211553049 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9b406b500 msgr2=0x7fe9b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.806+0000 7fe9ce64e640 1 --2- 172.21.15.1:0/211553049 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9b406b500 0x7fe9b406d9c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8001eb0 tx=0x7fe9b8001cc0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.806+0000 7fe9ce64e640 1 -- 172.21.15.1:0/211553049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c80a7050 msgr2=0x7fe9c81dc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.806+0000 7fe9ce64e640 1 --2- 172.21.15.1:0/211553049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c80a7050 0x7fe9c81dc440 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00c550 tx=0x7fe9bc00ca10 comp rx=0 tx=0).stop 2024-03-06T07:24:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.806+0000 7fe9ce64e640 1 -- 172.21.15.1:0/211553049 shutdown_connections 2024-03-06T07:24:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.806+0000 7fe9ce64e640 1 --2- 172.21.15.1:0/211553049 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9b406b500 0x7fe9b406d9c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.806+0000 7fe9ce64e640 1 --2- 172.21.15.1:0/211553049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c8156b70 0x7fe9c80ac480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.806+0000 7fe9ce64e640 1 --2- 172.21.15.1:0/211553049 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9c80c4bf0 0x7fe9c80c10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.806+0000 7fe9ce64e640 1 --2- 172.21.15.1:0/211553049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c80a7050 0x7fe9c81dc440 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.806+0000 7fe9ce64e640 1 -- 172.21.15.1:0/211553049 >> 172.21.15.1:0/211553049 conn(0x7fe9c80c7a50 msgr2=0x7fe9c81559c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:02.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.806+0000 7fe9ce64e640 1 -- 172.21.15.1:0/211553049 shutdown_connections 2024-03-06T07:24:02.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.807+0000 7fe9ce64e640 1 -- 172.21.15.1:0/211553049 wait complete. 2024-03-06T07:24:02.822 INFO:tasks.cephfs_test_runner:That setting mds_join_fs at runtime affects the cluster layout. ... ok 2024-03-06T07:24:02.822 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_standby_replay (tasks.cephfs.test_failover.TestClusterAffinity) That a standby-replay daemon with weak affinity is replaced by a stronger one. 2024-03-06T07:24:02.822 INFO:tasks.cephfs_test_runner:test_join_fs_standby_replay (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-06T07:24:02.823 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:02.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.970+0000 7fc24f18a640 1 Processor -- start 2024-03-06T07:24:02.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.970+0000 7fc24f18a640 1 -- start start 2024-03-06T07:24:02.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.970+0000 7fc24f18a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc24806e670 0x7fc24806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.970+0000 7fc24f18a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc2480a8520 0x7fc2480c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.970+0000 7fc24f18a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc24806f060 con 0x7fc248150290 2024-03-06T07:24:02.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.970+0000 7fc24f18a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2480c2ca0 con 0x7fc2480a8520 2024-03-06T07:24:02.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.970+0000 7fc24f18a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2480c26e0 con 0x7fc24806e670 2024-03-06T07:24:02.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.971+0000 7fc247fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc2480a8520 0x7fc2480c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.971+0000 7fc24d727640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc248150290 0x7fc248153290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37408/0 (socket says 172.21.15.1:37408) 2024-03-06T07:24:02.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.971+0000 7fc24d727640 1 -- 172.21.15.1:0/2537669000 learned_addr learned my addr 172.21.15.1:0/2537669000 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:02.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.971+0000 7fc24cf26640 1 --2- 172.21.15.1:0/2537669000 >> v2:172.21.15.98:3300/0 conn(0x7fc24806e670 0x7fc24806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.971+0000 7fc247fff640 1 -- 172.21.15.1:0/2537669000 >> v2:172.21.15.98:3300/0 conn(0x7fc24806e670 msgr2=0x7fc24806ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:02.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.971+0000 7fc247fff640 1 --2- 172.21.15.1:0/2537669000 >> v2:172.21.15.98:3300/0 conn(0x7fc24806e670 0x7fc24806ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.971+0000 7fc2477fe640 1 -- 172.21.15.1:0/2537669000 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434459983 0 0) 0x7fc24806f060 con 0x7fc248150290 2024-03-06T07:24:02.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.971+0000 7fc247fff640 1 -- 172.21.15.1:0/2537669000 >> v1:172.21.15.1:6789/0 conn(0x7fc248150290 legacy=0x7fc248153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.971+0000 7fc247fff640 1 -- 172.21.15.1:0/2537669000 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2480c29c0 con 0x7fc2480a8520 2024-03-06T07:24:02.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.971+0000 7fc24cf26640 1 --2- 172.21.15.1:0/2537669000 >> v2:172.21.15.98:3300/0 conn(0x7fc24806e670 0x7fc24806ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:02.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.972+0000 7fc247fff640 1 --2- 172.21.15.1:0/2537669000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2480a8520 0x7fc2480c1bc0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fc2480732d0 tx=0x7fc23402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b0e1eb8abe7e0d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.972+0000 7fc2477fe640 1 -- 172.21.15.1:0/2537669000 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2340315a0 con 0x7fc2480a8520 2024-03-06T07:24:02.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.972+0000 7fc2477fe640 1 -- 172.21.15.1:0/2537669000 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc234037070 con 0x7fc2480a8520 2024-03-06T07:24:02.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.972+0000 7fc2477fe640 1 -- 172.21.15.1:0/2537669000 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc234031960 con 0x7fc2480a8520 2024-03-06T07:24:02.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.972+0000 7fc24f18a640 1 -- 172.21.15.1:0/2537669000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2480a8520 msgr2=0x7fc2480c1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.972+0000 7fc24f18a640 1 --2- 172.21.15.1:0/2537669000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2480a8520 0x7fc2480c1bc0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fc2480732d0 tx=0x7fc23402b840 comp rx=0 tx=0).stop 2024-03-06T07:24:02.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.972+0000 7fc24f18a640 1 -- 172.21.15.1:0/2537669000 shutdown_connections 2024-03-06T07:24:02.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.972+0000 7fc24f18a640 1 --2- 172.21.15.1:0/2537669000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2480a8520 0x7fc2480c1bc0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.972+0000 7fc24f18a640 1 --2- 172.21.15.1:0/2537669000 >> v2:172.21.15.98:3300/0 conn(0x7fc24806e670 0x7fc24806ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.972+0000 7fc24f18a640 1 -- 172.21.15.1:0/2537669000 >> 172.21.15.1:0/2537669000 conn(0x7fc2480c3cc0 msgr2=0x7fc2480c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:02.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.972+0000 7fc24f18a640 1 -- 172.21.15.1:0/2537669000 shutdown_connections 2024-03-06T07:24:02.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.972+0000 7fc24f18a640 1 -- 172.21.15.1:0/2537669000 wait complete. 2024-03-06T07:24:02.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.974+0000 7fc24f18a640 1 Processor -- start 2024-03-06T07:24:02.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.974+0000 7fc24f18a640 1 -- start start 2024-03-06T07:24:02.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.974+0000 7fc24f18a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc24806e670 0x7fc2481e8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.974+0000 7fc24f18a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2480a8520 0x7fc24814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.974+0000 7fc24f18a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc248150290 0x7fc24813ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.974+0000 7fc24f18a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc248150ad0 con 0x7fc2480a8520 2024-03-06T07:24:02.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.974+0000 7fc24f18a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc248150950 con 0x7fc24806e670 2024-03-06T07:24:02.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.974+0000 7fc24f18a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2481539a0 con 0x7fc248150290 2024-03-06T07:24:02.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.975+0000 7fc247fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2480a8520 0x7fc24814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.975+0000 7fc247fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2480a8520 0x7fc24814efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41596/0 (socket says 172.21.15.1:41596) 2024-03-06T07:24:02.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.975+0000 7fc247fff640 1 -- 172.21.15.1:0/1698387408 learned_addr learned my addr 172.21.15.1:0/1698387408 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:02.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.975+0000 7fc24cf26640 1 --2- 172.21.15.1:0/1698387408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc24806e670 0x7fc2481e8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.975+0000 7fc24d727640 1 --2- 172.21.15.1:0/1698387408 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc248150290 0x7fc24813ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.975+0000 7fc247fff640 1 -- 172.21.15.1:0/1698387408 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc248150290 msgr2=0x7fc24813ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.975+0000 7fc247fff640 1 --2- 172.21.15.1:0/1698387408 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc248150290 0x7fc24813ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.975+0000 7fc247fff640 1 -- 172.21.15.1:0/1698387408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc24806e670 msgr2=0x7fc2481e8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.975+0000 7fc247fff640 1 --2- 172.21.15.1:0/1698387408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc24806e670 0x7fc2481e8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.975+0000 7fc247fff640 1 -- 172.21.15.1:0/1698387408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2480c6950 con 0x7fc2480a8520 2024-03-06T07:24:02.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.975+0000 7fc247fff640 1 --2- 172.21.15.1:0/1698387408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2480a8520 0x7fc24814efe0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fc234009890 tx=0x7fc2340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.975+0000 7fc2457fa640 1 -- 172.21.15.1:0/1698387408 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc234032840 con 0x7fc2480a8520 2024-03-06T07:24:02.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.975+0000 7fc2457fa640 1 -- 172.21.15.1:0/1698387408 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc234032a00 con 0x7fc2480a8520 2024-03-06T07:24:02.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.975+0000 7fc24f18a640 1 -- 172.21.15.1:0/1698387408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2480c6c30 con 0x7fc2480a8520 2024-03-06T07:24:02.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.975+0000 7fc2457fa640 1 -- 172.21.15.1:0/1698387408 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc234032df0 con 0x7fc2480a8520 2024-03-06T07:24:02.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.975+0000 7fc24f18a640 1 -- 172.21.15.1:0/1698387408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2481571f0 con 0x7fc2480a8520 2024-03-06T07:24:02.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.976+0000 7fc24f18a640 1 -- 172.21.15.1:0/1698387408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2480c3610 con 0x7fc2480a8520 2024-03-06T07:24:02.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.977+0000 7fc2457fa640 1 -- 172.21.15.1:0/1698387408 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc23404c020 con 0x7fc2480a8520 2024-03-06T07:24:02.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.977+0000 7fc2457fa640 1 --2- 172.21.15.1:0/1698387408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc22c06b4d0 0x7fc22c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.977+0000 7fc2457fa640 1 -- 172.21.15.1:0/1698387408 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fc23402f020 con 0x7fc2480a8520 2024-03-06T07:24:02.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.977+0000 7fc24cf26640 1 --2- 172.21.15.1:0/1698387408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc22c06b4d0 0x7fc22c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.978+0000 7fc24cf26640 1 --2- 172.21.15.1:0/1698387408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc22c06b4d0 0x7fc22c06d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc23c00a990 tx=0x7fc23c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:02.981+0000 7fc2457fa640 1 -- 172.21.15.1:0/1698387408 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2480c3610 con 0x7fc2480a8520 2024-03-06T07:24:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.167+0000 7fc24f18a640 1 -- 172.21.15.1:0/1698387408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc248067fd0 con 0x7fc2480a8520 2024-03-06T07:24:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.167+0000 7fc2457fa640 1 -- 172.21.15.1:0/1698387408 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+13304 (secure 0 0 0) 0x7fc234040070 con 0x7fc2480a8520 2024-03-06T07:24:03.169 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:03.170 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":99,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:24:00.793262+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6834/1335402413":"2024-03-07T07:23:57.619692+0000","172.21.15.98:6836/1335402413":"2024-03-07T07:23:57.619692+0000","172.21.15.98:6837/3330683497":"2024-03-07T07:23:48.752518+0000","172.21.15.98:6835/3330683497":"2024-03-07T07:23:48.752518+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:03.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.170+0000 7fc24f18a640 1 -- 172.21.15.1:0/1698387408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc22c06b4d0 msgr2=0x7fc22c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.170+0000 7fc24f18a640 1 --2- 172.21.15.1:0/1698387408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc22c06b4d0 0x7fc22c06d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc23c00a990 tx=0x7fc23c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:03.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.170+0000 7fc24f18a640 1 -- 172.21.15.1:0/1698387408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2480a8520 msgr2=0x7fc24814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.170+0000 7fc24f18a640 1 --2- 172.21.15.1:0/1698387408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2480a8520 0x7fc24814efe0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fc234009890 tx=0x7fc2340046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.170+0000 7fc24f18a640 1 -- 172.21.15.1:0/1698387408 shutdown_connections 2024-03-06T07:24:03.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.170+0000 7fc24f18a640 1 --2- 172.21.15.1:0/1698387408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc22c06b4d0 0x7fc22c06d990 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.170+0000 7fc24f18a640 1 --2- 172.21.15.1:0/1698387408 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc248150290 0x7fc24813ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.170+0000 7fc24f18a640 1 --2- 172.21.15.1:0/1698387408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2480a8520 0x7fc24814efe0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.170+0000 7fc24f18a640 1 --2- 172.21.15.1:0/1698387408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc24806e670 0x7fc2481e8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.170+0000 7fc24f18a640 1 -- 172.21.15.1:0/1698387408 >> 172.21.15.1:0/1698387408 conn(0x7fc2480c3cc0 msgr2=0x7fc248148860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.170+0000 7fc24f18a640 1 -- 172.21.15.1:0/1698387408 shutdown_connections 2024-03-06T07:24:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.170+0000 7fc24f18a640 1 -- 172.21.15.1:0/1698387408 wait complete. 2024-03-06T07:24:03.182 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:03.183 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:03.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.285+0000 7f52a1fbc640 1 Processor -- start 2024-03-06T07:24:03.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.285+0000 7f52a1fbc640 1 -- start start 2024-03-06T07:24:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.285+0000 7f52a1fbc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f529c0c1260 0x7f529c0c16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.285+0000 7f52a1fbc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f529c1565a0 0x7f529c156a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.285+0000 7f52a1fbc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f529c1530a0 con 0x7f529c149a90 2024-03-06T07:24:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.285+0000 7f52a1fbc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f529c153380 con 0x7f529c1565a0 2024-03-06T07:24:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.285+0000 7f52a1fbc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f529c153660 con 0x7f529c0c1260 2024-03-06T07:24:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f529c149a90 0x7f529c0c4760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37420/0 (socket says 172.21.15.1:37420) 2024-03-06T07:24:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529b7fe640 1 -- 172.21.15.1:0/2111805179 learned_addr learned my addr 172.21.15.1:0/2111805179 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529a7fc640 1 -- 172.21.15.1:0/2111805179 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682001591 0 0) 0x7f529c1530a0 con 0x7f529c149a90 2024-03-06T07:24:03.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529a7fc640 1 -- 172.21.15.1:0/2111805179 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5284003520 con 0x7f529c149a90 2024-03-06T07:24:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529bfff640 1 --2- 172.21.15.1:0/2111805179 >> v2:172.21.15.98:3300/0 conn(0x7f529c1565a0 0x7f529c156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529affd640 1 --2- 172.21.15.1:0/2111805179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f529c0c1260 0x7f529c0c16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529a7fc640 1 -- 172.21.15.1:0/2111805179 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2664132820 0 0) 0x7f5284003520 con 0x7f529c149a90 2024-03-06T07:24:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529a7fc640 1 -- 172.21.15.1:0/2111805179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f529c0c1260 msgr2=0x7f529c0c16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529a7fc640 1 --2- 172.21.15.1:0/2111805179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f529c0c1260 0x7f529c0c16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529a7fc640 1 -- 172.21.15.1:0/2111805179 >> v2:172.21.15.98:3300/0 conn(0x7f529c1565a0 msgr2=0x7f529c156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529a7fc640 1 --2- 172.21.15.1:0/2111805179 >> v2:172.21.15.98:3300/0 conn(0x7f529c1565a0 0x7f529c156a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529a7fc640 1 -- 172.21.15.1:0/2111805179 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529c153940 con 0x7f529c149a90 2024-03-06T07:24:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529a7fc640 1 -- 172.21.15.1:0/2111805179 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f528c001670 con 0x7f529c149a90 2024-03-06T07:24:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529a7fc640 1 -- 172.21.15.1:0/2111805179 >> v1:172.21.15.1:6789/0 conn(0x7f529c149a90 legacy=0x7f529c0c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529a7fc640 1 --2- 172.21.15.1:0/2111805179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52840039b0 0x7f5284003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529a7fc640 1 --2- 172.21.15.1:0/2111805179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f529c0c1260 0x7f5284004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529a7fc640 1 --2- 172.21.15.1:0/2111805179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5284004d40 0x7f52840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529a7fc640 1 -- 172.21.15.1:0/2111805179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c153380 con 0x7f52840039b0 2024-03-06T07:24:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529a7fc640 1 -- 172.21.15.1:0/2111805179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c153660 con 0x7f5284004d40 2024-03-06T07:24:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529a7fc640 1 -- 172.21.15.1:0/2111805179 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f528400bb50 con 0x7f529c0c1260 2024-03-06T07:24:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529b7fe640 1 --2- 172.21.15.1:0/2111805179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52840039b0 0x7f5284003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529affd640 1 --2- 172.21.15.1:0/2111805179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f529c0c1260 0x7f5284004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.286+0000 7f529bfff640 1 --2- 172.21.15.1:0/2111805179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5284004d40 0x7f52840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.287+0000 7f529b7fe640 1 -- 172.21.15.1:0/2111805179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f529c0c1260 msgr2=0x7f5284004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.287+0000 7f529b7fe640 1 --2- 172.21.15.1:0/2111805179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f529c0c1260 0x7f5284004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.287+0000 7f529b7fe640 1 -- 172.21.15.1:0/2111805179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5284004d40 msgr2=0x7f52840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.287+0000 7f529b7fe640 1 --2- 172.21.15.1:0/2111805179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5284004d40 0x7f52840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.287+0000 7f529b7fe640 1 -- 172.21.15.1:0/2111805179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5284003520 con 0x7f52840039b0 2024-03-06T07:24:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.287+0000 7f529b7fe640 1 --2- 172.21.15.1:0/2111805179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52840039b0 0x7f5284003dd0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f528c001050 tx=0x7f5284005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d809a97f64929e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.288+0000 7f529a7fc640 1 -- 172.21.15.1:0/2111805179 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c007d70 con 0x7f52840039b0 2024-03-06T07:24:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.288+0000 7f529a7fc640 1 -- 172.21.15.1:0/2111805179 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528c030070 con 0x7f52840039b0 2024-03-06T07:24:03.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.288+0000 7f529a7fc640 1 -- 172.21.15.1:0/2111805179 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c02cd80 con 0x7f52840039b0 2024-03-06T07:24:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.288+0000 7f52a1fbc640 1 -- 172.21.15.1:0/2111805179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52840039b0 msgr2=0x7f5284003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.288+0000 7f52a1fbc640 1 --2- 172.21.15.1:0/2111805179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52840039b0 0x7f5284003dd0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f528c001050 tx=0x7f5284005830 comp rx=0 tx=0).stop 2024-03-06T07:24:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.288+0000 7f529bfff640 1 -- 172.21.15.1:0/2111805179 reap_dead start 2024-03-06T07:24:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.288+0000 7f52a1fbc640 1 -- 172.21.15.1:0/2111805179 shutdown_connections 2024-03-06T07:24:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.288+0000 7f52a1fbc640 1 -- 172.21.15.1:0/2111805179 >> 172.21.15.1:0/2111805179 conn(0x7f529c0c6860 msgr2=0x7f529c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.288+0000 7f52a1fbc640 1 -- 172.21.15.1:0/2111805179 shutdown_connections 2024-03-06T07:24:03.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.288+0000 7f52a1fbc640 1 -- 172.21.15.1:0/2111805179 wait complete. 2024-03-06T07:24:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.289+0000 7f52a1fbc640 1 Processor -- start 2024-03-06T07:24:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.289+0000 7f52a1fbc640 1 -- start start 2024-03-06T07:24:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.290+0000 7f52a1fbc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52840039b0 0x7f529c1ebd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.290+0000 7f52a1fbc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f529c0c0a40 0x7f529c0be980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.290+0000 7f52a1fbc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f529c0beec0 0x7f529c14f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.290+0000 7f52a1fbc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c06f620 con 0x7f52840039b0 2024-03-06T07:24:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.290+0000 7f52a1fbc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c06f4a0 con 0x7f529c0c0a40 2024-03-06T07:24:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.290+0000 7f52a1fbc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f529c156f70 con 0x7f529c0beec0 2024-03-06T07:24:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.290+0000 7f529b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52840039b0 0x7f529c1ebd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.290+0000 7f529b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52840039b0 0x7f529c1ebd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41620/0 (socket says 172.21.15.1:41620) 2024-03-06T07:24:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.290+0000 7f529b7fe640 1 -- 172.21.15.1:0/863712701 learned_addr learned my addr 172.21.15.1:0/863712701 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.290+0000 7f529b7fe640 1 -- 172.21.15.1:0/863712701 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f529c0beec0 msgr2=0x7f529c14f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.290+0000 7f529b7fe640 1 --2- 172.21.15.1:0/863712701 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f529c0beec0 0x7f529c14f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.290+0000 7f529b7fe640 1 -- 172.21.15.1:0/863712701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f529c0c0a40 msgr2=0x7f529c0be980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.290+0000 7f529b7fe640 1 --2- 172.21.15.1:0/863712701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f529c0c0a40 0x7f529c0be980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.290+0000 7f529b7fe640 1 -- 172.21.15.1:0/863712701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529c0bfbd0 con 0x7f52840039b0 2024-03-06T07:24:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.291+0000 7f529b7fe640 1 --2- 172.21.15.1:0/863712701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52840039b0 0x7f529c1ebd30 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f528c00c300 tx=0x7f528c000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.291+0000 7f5283fff640 1 -- 172.21.15.1:0/863712701 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c004bf0 con 0x7f52840039b0 2024-03-06T07:24:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.291+0000 7f5283fff640 1 -- 172.21.15.1:0/863712701 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528c004d90 con 0x7f52840039b0 2024-03-06T07:24:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.291+0000 7f52a1fbc640 1 -- 172.21.15.1:0/863712701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529c0bfe70 con 0x7f52840039b0 2024-03-06T07:24:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.291+0000 7f5283fff640 1 -- 172.21.15.1:0/863712701 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c007270 con 0x7f52840039b0 2024-03-06T07:24:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.291+0000 7f52a1fbc640 1 -- 172.21.15.1:0/863712701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f529c1ec850 con 0x7f52840039b0 2024-03-06T07:24:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.292+0000 7f52a1fbc640 1 -- 172.21.15.1:0/863712701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f529c067fd0 con 0x7f52840039b0 2024-03-06T07:24:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.292+0000 7f5283fff640 1 -- 172.21.15.1:0/863712701 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f528c047020 con 0x7f52840039b0 2024-03-06T07:24:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.292+0000 7f5283fff640 1 --2- 172.21.15.1:0/863712701 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f528806b4d0 0x7f528806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.292+0000 7f5283fff640 1 -- 172.21.15.1:0/863712701 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f528c0ab2c0 con 0x7f52840039b0 2024-03-06T07:24:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.293+0000 7f529affd640 1 --2- 172.21.15.1:0/863712701 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f528806b4d0 0x7f528806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.293+0000 7f529affd640 1 --2- 172.21.15.1:0/863712701 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f528806b4d0 0x7f528806d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5290006a90 tx=0x7f5290006a20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.297+0000 7f5283fff640 1 -- 172.21.15.1:0/863712701 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f528c034e90 con 0x7f52840039b0 2024-03-06T07:24:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.475+0000 7f52a1fbc640 1 -- 172.21.15.1:0/863712701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f529c06f280 con 0x7f52840039b0 2024-03-06T07:24:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.475+0000 7f5283fff640 1 -- 172.21.15.1:0/863712701 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7f528c07a260 con 0x7f52840039b0 2024-03-06T07:24:03.478 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:24:03.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.477+0000 7f52a1fbc640 1 -- 172.21.15.1:0/863712701 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f528806b4d0 msgr2=0x7f528806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.477+0000 7f52a1fbc640 1 --2- 172.21.15.1:0/863712701 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f528806b4d0 0x7f528806d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5290006a90 tx=0x7f5290006a20 comp rx=0 tx=0).stop 2024-03-06T07:24:03.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.477+0000 7f52a1fbc640 1 -- 172.21.15.1:0/863712701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52840039b0 msgr2=0x7f529c1ebd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.477+0000 7f52a1fbc640 1 --2- 172.21.15.1:0/863712701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52840039b0 0x7f529c1ebd30 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f528c00c300 tx=0x7f528c000fe0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.478+0000 7f52a1fbc640 1 -- 172.21.15.1:0/863712701 shutdown_connections 2024-03-06T07:24:03.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.478+0000 7f52a1fbc640 1 --2- 172.21.15.1:0/863712701 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f528806b4d0 0x7f528806d990 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.478+0000 7f52a1fbc640 1 --2- 172.21.15.1:0/863712701 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f529c0beec0 0x7f529c14f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.478+0000 7f52a1fbc640 1 --2- 172.21.15.1:0/863712701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f529c0c0a40 0x7f529c0be980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.478+0000 7f52a1fbc640 1 --2- 172.21.15.1:0/863712701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52840039b0 0x7f529c1ebd30 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.478+0000 7f52a1fbc640 1 -- 172.21.15.1:0/863712701 >> 172.21.15.1:0/863712701 conn(0x7f529c0c6860 msgr2=0x7f529c150ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.478+0000 7f52a1fbc640 1 -- 172.21.15.1:0/863712701 shutdown_connections 2024-03-06T07:24:03.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.478+0000 7f52a1fbc640 1 -- 172.21.15.1:0/863712701 wait complete. 2024-03-06T07:24:03.490 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay' 2024-03-06T07:24:03.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.590+0000 7ff07a095640 1 Processor -- start 2024-03-06T07:24:03.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.590+0000 7ff07a095640 1 -- start start 2024-03-06T07:24:03.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff07a095640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0740c4dd0 0x7ff0740c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff07a095640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff0740c79c0 0x7ff074152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff07a095640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0741566e0 con 0x7ff0740c5d80 2024-03-06T07:24:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff07a095640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff074156490 con 0x7ff0740c79c0 2024-03-06T07:24:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff07a095640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff074157010 con 0x7ff0740c4dd0 2024-03-06T07:24:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff073fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff0740c5d80 0x7ff0741533e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37428/0 (socket says 172.21.15.1:37428) 2024-03-06T07:24:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff073fff640 1 -- 172.21.15.1:0/2916788620 learned_addr learned my addr 172.21.15.1:0/2916788620 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0727fc640 1 -- 172.21.15.1:0/2916788620 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228944170 0 0) 0x7ff0741566e0 con 0x7ff0740c5d80 2024-03-06T07:24:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0727fc640 1 -- 172.21.15.1:0/2916788620 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff05c003520 con 0x7ff0740c5d80 2024-03-06T07:24:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff072ffd640 1 --2- 172.21.15.1:0/2916788620 >> v2:172.21.15.98:3300/0 conn(0x7ff0740c79c0 0x7ff074152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0737fe640 1 --2- 172.21.15.1:0/2916788620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0740c4dd0 0x7ff0740c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0727fc640 1 -- 172.21.15.1:0/2916788620 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2815532556 0 0) 0x7ff05c003520 con 0x7ff0740c5d80 2024-03-06T07:24:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0727fc640 1 -- 172.21.15.1:0/2916788620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0740c4dd0 msgr2=0x7ff0740c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0727fc640 1 --2- 172.21.15.1:0/2916788620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0740c4dd0 0x7ff0740c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0727fc640 1 -- 172.21.15.1:0/2916788620 >> v2:172.21.15.98:3300/0 conn(0x7ff0740c79c0 msgr2=0x7ff074152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0727fc640 1 --2- 172.21.15.1:0/2916788620 >> v2:172.21.15.98:3300/0 conn(0x7ff0740c79c0 0x7ff074152ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0727fc640 1 -- 172.21.15.1:0/2916788620 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0741569c0 con 0x7ff0740c5d80 2024-03-06T07:24:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0727fc640 1 -- 172.21.15.1:0/2916788620 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7ff06c0019f0 con 0x7ff0740c5d80 2024-03-06T07:24:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0737fe640 1 --2- 172.21.15.1:0/2916788620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0740c4dd0 0x7ff0740c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0727fc640 1 -- 172.21.15.1:0/2916788620 >> v1:172.21.15.1:6789/0 conn(0x7ff0740c5d80 legacy=0x7ff0741533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0727fc640 1 --2- 172.21.15.1:0/2916788620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0740c4dd0 0x7ff05c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0727fc640 1 --2- 172.21.15.1:0/2916788620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff05c003ef0 0x7ff05c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0727fc640 1 --2- 172.21.15.1:0/2916788620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff05c004d40 0x7ff05c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0727fc640 1 -- 172.21.15.1:0/2916788620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff074156490 con 0x7ff05c004d40 2024-03-06T07:24:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0727fc640 1 -- 172.21.15.1:0/2916788620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff074157010 con 0x7ff05c003ef0 2024-03-06T07:24:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff0727fc640 1 -- 172.21.15.1:0/2916788620 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c00bb20 con 0x7ff0740c4dd0 2024-03-06T07:24:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.591+0000 7ff073fff640 1 --2- 172.21.15.1:0/2916788620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff05c004d40 0x7ff05c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.592+0000 7ff0737fe640 1 --2- 172.21.15.1:0/2916788620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0740c4dd0 0x7ff05c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.592+0000 7ff072ffd640 1 --2- 172.21.15.1:0/2916788620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff05c003ef0 0x7ff05c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.592+0000 7ff073fff640 1 -- 172.21.15.1:0/2916788620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0740c4dd0 msgr2=0x7ff05c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.592+0000 7ff073fff640 1 --2- 172.21.15.1:0/2916788620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0740c4dd0 0x7ff05c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.592+0000 7ff073fff640 1 -- 172.21.15.1:0/2916788620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff05c003ef0 msgr2=0x7ff05c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.592+0000 7ff073fff640 1 --2- 172.21.15.1:0/2916788620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff05c003ef0 0x7ff05c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.592+0000 7ff073fff640 1 -- 172.21.15.1:0/2916788620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c003520 con 0x7ff05c004d40 2024-03-06T07:24:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.592+0000 7ff073fff640 1 --2- 172.21.15.1:0/2916788620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff05c004d40 0x7ff05c0051c0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7ff06c00a880 tx=0x7ff05c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f390cb137fde815 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.593+0000 7ff0727fc640 1 -- 172.21.15.1:0/2916788620 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c00a4a0 con 0x7ff05c004d40 2024-03-06T07:24:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.593+0000 7ff0727fc640 1 -- 172.21.15.1:0/2916788620 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06c034630 con 0x7ff05c004d40 2024-03-06T07:24:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.593+0000 7ff0727fc640 1 -- 172.21.15.1:0/2916788620 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c034aa0 con 0x7ff05c004d40 2024-03-06T07:24:03.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.593+0000 7ff07a095640 1 -- 172.21.15.1:0/2916788620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff05c004d40 msgr2=0x7ff05c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.593+0000 7ff07a095640 1 --2- 172.21.15.1:0/2916788620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff05c004d40 0x7ff05c0051c0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7ff06c00a880 tx=0x7ff05c005830 comp rx=0 tx=0).stop 2024-03-06T07:24:03.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.593+0000 7ff073fff640 1 -- 172.21.15.1:0/2916788620 reap_dead start 2024-03-06T07:24:03.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.593+0000 7ff07a095640 1 -- 172.21.15.1:0/2916788620 shutdown_connections 2024-03-06T07:24:03.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.593+0000 7ff07a095640 1 -- 172.21.15.1:0/2916788620 >> 172.21.15.1:0/2916788620 conn(0x7ff07406b410 msgr2=0x7ff07406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.593+0000 7ff07a095640 1 -- 172.21.15.1:0/2916788620 shutdown_connections 2024-03-06T07:24:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.593+0000 7ff07a095640 1 -- 172.21.15.1:0/2916788620 wait complete. 2024-03-06T07:24:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.594+0000 7ff07a095640 1 Processor -- start 2024-03-06T07:24:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.594+0000 7ff07a095640 1 -- start start 2024-03-06T07:24:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.595+0000 7ff07a095640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0740bda80 0x7ff07414ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.595+0000 7ff07a095640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0740bde60 0x7ff07414f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.595+0000 7ff07a095640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff07414ff20 0x7ff0741df860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.595+0000 7ff07a095640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0740715f0 con 0x7ff0740bde60 2024-03-06T07:24:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.595+0000 7ff07a095640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff074153af0 con 0x7ff07414ff20 2024-03-06T07:24:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.595+0000 7ff07a095640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff074071770 con 0x7ff0740bda80 2024-03-06T07:24:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.595+0000 7ff072ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0740bde60 0x7ff07414f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.595+0000 7ff072ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0740bde60 0x7ff07414f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41652/0 (socket says 172.21.15.1:41652) 2024-03-06T07:24:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.595+0000 7ff072ffd640 1 -- 172.21.15.1:0/1933210278 learned_addr learned my addr 172.21.15.1:0/1933210278 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.595+0000 7ff0737fe640 1 --2- 172.21.15.1:0/1933210278 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0740bda80 0x7ff07414ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.595+0000 7ff073fff640 1 --2- 172.21.15.1:0/1933210278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff07414ff20 0x7ff0741df860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.596+0000 7ff072ffd640 1 -- 172.21.15.1:0/1933210278 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0740bda80 msgr2=0x7ff07414ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.596+0000 7ff072ffd640 1 --2- 172.21.15.1:0/1933210278 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0740bda80 0x7ff07414ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.596+0000 7ff072ffd640 1 -- 172.21.15.1:0/1933210278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff07414ff20 msgr2=0x7ff0741df860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.596+0000 7ff072ffd640 1 --2- 172.21.15.1:0/1933210278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff07414ff20 0x7ff0741df860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.596+0000 7ff072ffd640 1 -- 172.21.15.1:0/1933210278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0741e9e50 con 0x7ff0740bde60 2024-03-06T07:24:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.596+0000 7ff0737fe640 1 --2- 172.21.15.1:0/1933210278 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0740bda80 0x7ff07414ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.596+0000 7ff073fff640 1 --2- 172.21.15.1:0/1933210278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff07414ff20 0x7ff0741df860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.596+0000 7ff072ffd640 1 --2- 172.21.15.1:0/1933210278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0740bde60 0x7ff07414f3a0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7ff0680119c0 tx=0x7ff068011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.596+0000 7ff05bfff640 1 -- 172.21.15.1:0/1933210278 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06800fa90 con 0x7ff0740bde60 2024-03-06T07:24:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.596+0000 7ff05bfff640 1 -- 172.21.15.1:0/1933210278 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06800fc30 con 0x7ff0740bde60 2024-03-06T07:24:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.596+0000 7ff07a095640 1 -- 172.21.15.1:0/1933210278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07414fc40 con 0x7ff0740bde60 2024-03-06T07:24:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.596+0000 7ff05bfff640 1 -- 172.21.15.1:0/1933210278 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff068014440 con 0x7ff0740bde60 2024-03-06T07:24:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.596+0000 7ff07a095640 1 -- 172.21.15.1:0/1933210278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0741e9740 con 0x7ff0740bde60 2024-03-06T07:24:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.597+0000 7ff07a095640 1 -- 172.21.15.1:0/1933210278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0741e9940 con 0x7ff0740bde60 2024-03-06T07:24:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.597+0000 7ff05bfff640 1 -- 172.21.15.1:0/1933210278 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff068012040 con 0x7ff0740bde60 2024-03-06T07:24:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.598+0000 7ff05bfff640 1 --2- 172.21.15.1:0/1933210278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff06006b500 0x7ff06006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.598+0000 7ff05bfff640 1 -- 172.21.15.1:0/1933210278 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5421+0+0 (secure 0 0 0) 0x7ff06808a5d0 con 0x7ff0740bde60 2024-03-06T07:24:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.598+0000 7ff0737fe640 1 --2- 172.21.15.1:0/1933210278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff06006b500 0x7ff06006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.598+0000 7ff0737fe640 1 --2- 172.21.15.1:0/1933210278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff06006b500 0x7ff06006d9c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff0740c5a80 tx=0x7ff064008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.602+0000 7ff05bfff640 1 -- 172.21.15.1:0/1933210278 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff068019030 con 0x7ff0740bde60 2024-03-06T07:24:03.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:03.857+0000 7ff07a095640 1 -- 172.21.15.1:0/1933210278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]} v 0) v1 -- 0x7ff074067fd0 con 0x7ff0740bde60 2024-03-06T07:24:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.809+0000 7ff05bfff640 1 -- 172.21.15.1:0/1933210278 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]}]=0 v229) v1 ==== 156+0+0 (secure 0 0 0) 0x7ff068059600 con 0x7ff0740bde60 2024-03-06T07:24:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.812+0000 7ff07a095640 1 -- 172.21.15.1:0/1933210278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff06006b500 msgr2=0x7ff06006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.812+0000 7ff07a095640 1 --2- 172.21.15.1:0/1933210278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff06006b500 0x7ff06006d9c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff0740c5a80 tx=0x7ff064008040 comp rx=0 tx=0).stop 2024-03-06T07:24:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.812+0000 7ff07a095640 1 -- 172.21.15.1:0/1933210278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0740bde60 msgr2=0x7ff07414f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.812+0000 7ff07a095640 1 --2- 172.21.15.1:0/1933210278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0740bde60 0x7ff07414f3a0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7ff0680119c0 tx=0x7ff068011e80 comp rx=0 tx=0).stop 2024-03-06T07:24:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.812+0000 7ff07a095640 1 -- 172.21.15.1:0/1933210278 shutdown_connections 2024-03-06T07:24:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.812+0000 7ff07a095640 1 --2- 172.21.15.1:0/1933210278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff06006b500 0x7ff06006d9c0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.812+0000 7ff07a095640 1 --2- 172.21.15.1:0/1933210278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff07414ff20 0x7ff0741df860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.812+0000 7ff07a095640 1 --2- 172.21.15.1:0/1933210278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0740bde60 0x7ff07414f3a0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.812+0000 7ff07a095640 1 --2- 172.21.15.1:0/1933210278 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0740bda80 0x7ff07414ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.812+0000 7ff07a095640 1 -- 172.21.15.1:0/1933210278 >> 172.21.15.1:0/1933210278 conn(0x7ff07406b410 msgr2=0x7ff0740a9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.812+0000 7ff07a095640 1 -- 172.21.15.1:0/1933210278 shutdown_connections 2024-03-06T07:24:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.812+0000 7ff07a095640 1 -- 172.21.15.1:0/1933210278 wait complete. 2024-03-06T07:24:04.828 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:24:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.936+0000 7f0478dac640 1 Processor -- start 2024-03-06T07:24:04.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.936+0000 7f0478dac640 1 -- start start 2024-03-06T07:24:04.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.937+0000 7f0478dac640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0474159140 0x7f04740c4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.937+0000 7f0478dac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04740c69b0 0x7f04740c2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.937+0000 7f0478dac640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0474072d00 con 0x7f04740c7510 2024-03-06T07:24:04.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.937+0000 7f0478dac640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04740c0e80 con 0x7f0474159140 2024-03-06T07:24:04.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.937+0000 7f0478dac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04740c13b0 con 0x7f04740c69b0 2024-03-06T07:24:04.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f0472d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f04740c7510 0x7f047406aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48788/0 (socket says 172.21.15.1:48788) 2024-03-06T07:24:04.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f0472d9d640 1 -- 172.21.15.1:0/1229448074 learned_addr learned my addr 172.21.15.1:0/1229448074 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:04.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047159a640 1 -- 172.21.15.1:0/1229448074 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951510076 0 0) 0x7f0474072d00 con 0x7f04740c7510 2024-03-06T07:24:04.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047159a640 1 -- 172.21.15.1:0/1229448074 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f045c003520 con 0x7f04740c7510 2024-03-06T07:24:04.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047259c640 1 --2- 172.21.15.1:0/1229448074 >> v2:172.21.15.98:3300/0 conn(0x7f0474159140 0x7f04740c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f0471d9b640 1 --2- 172.21.15.1:0/1229448074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04740c69b0 0x7f04740c2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047159a640 1 -- 172.21.15.1:0/1229448074 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3692911855 0 0) 0x7f045c003520 con 0x7f04740c7510 2024-03-06T07:24:04.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047159a640 1 -- 172.21.15.1:0/1229448074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04740c69b0 msgr2=0x7f04740c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047159a640 1 --2- 172.21.15.1:0/1229448074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04740c69b0 0x7f04740c2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047159a640 1 -- 172.21.15.1:0/1229448074 >> v2:172.21.15.98:3300/0 conn(0x7f0474159140 msgr2=0x7f04740c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047159a640 1 --2- 172.21.15.1:0/1229448074 >> v2:172.21.15.98:3300/0 conn(0x7f0474159140 0x7f04740c4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047159a640 1 -- 172.21.15.1:0/1229448074 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0474073180 con 0x7f04740c7510 2024-03-06T07:24:04.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047159a640 1 -- 172.21.15.1:0/1229448074 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f0464001a10 con 0x7f04740c7510 2024-03-06T07:24:04.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047159a640 1 -- 172.21.15.1:0/1229448074 >> v1:172.21.15.1:6789/0 conn(0x7f04740c7510 legacy=0x7f047406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047159a640 1 --2- 172.21.15.1:0/1229448074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f045c0039b0 0x7f045c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047159a640 1 --2- 172.21.15.1:0/1229448074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f045c004800 0x7f045c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047159a640 1 --2- 172.21.15.1:0/1229448074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04740c69b0 0x7f045c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047159a640 1 -- 172.21.15.1:0/1229448074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04740c0e80 con 0x7f045c0039b0 2024-03-06T07:24:04.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047159a640 1 -- 172.21.15.1:0/1229448074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04740c13b0 con 0x7f045c004800 2024-03-06T07:24:04.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047159a640 1 -- 172.21.15.1:0/1229448074 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045c00b9c0 con 0x7f04740c69b0 2024-03-06T07:24:04.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f047259c640 1 --2- 172.21.15.1:0/1229448074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f045c0039b0 0x7f045c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f0472d9d640 1 --2- 172.21.15.1:0/1229448074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04740c69b0 0x7f045c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.938+0000 7f0471d9b640 1 --2- 172.21.15.1:0/1229448074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f045c004800 0x7f045c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.939+0000 7f047259c640 1 -- 172.21.15.1:0/1229448074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04740c69b0 msgr2=0x7f045c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.939+0000 7f047259c640 1 --2- 172.21.15.1:0/1229448074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04740c69b0 0x7f045c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.939+0000 7f047259c640 1 -- 172.21.15.1:0/1229448074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f045c004800 msgr2=0x7f045c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.939+0000 7f047259c640 1 --2- 172.21.15.1:0/1229448074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f045c004800 0x7f045c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.939+0000 7f047259c640 1 -- 172.21.15.1:0/1229448074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045c003520 con 0x7f045c0039b0 2024-03-06T07:24:04.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.939+0000 7f047259c640 1 --2- 172.21.15.1:0/1229448074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f045c0039b0 0x7f045c003dd0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f046c009af0 tx=0x7f045c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894fefdf98dd0d7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.940+0000 7f047159a640 1 -- 172.21.15.1:0/1229448074 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c039070 con 0x7f045c0039b0 2024-03-06T07:24:04.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.940+0000 7f047159a640 1 -- 172.21.15.1:0/1229448074 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046c02c710 con 0x7f045c0039b0 2024-03-06T07:24:04.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.940+0000 7f047159a640 1 -- 172.21.15.1:0/1229448074 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c009690 con 0x7f045c0039b0 2024-03-06T07:24:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.940+0000 7f0478dac640 1 -- 172.21.15.1:0/1229448074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f045c0039b0 msgr2=0x7f045c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.940+0000 7f0478dac640 1 --2- 172.21.15.1:0/1229448074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f045c0039b0 0x7f045c003dd0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f046c009af0 tx=0x7f045c005830 comp rx=0 tx=0).stop 2024-03-06T07:24:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.940+0000 7f0472d9d640 1 -- 172.21.15.1:0/1229448074 reap_dead start 2024-03-06T07:24:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.940+0000 7f0478dac640 1 -- 172.21.15.1:0/1229448074 shutdown_connections 2024-03-06T07:24:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.940+0000 7f0478dac640 1 -- 172.21.15.1:0/1229448074 >> 172.21.15.1:0/1229448074 conn(0x7f04740c7a50 msgr2=0x7f04740c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.940+0000 7f0478dac640 1 -- 172.21.15.1:0/1229448074 shutdown_connections 2024-03-06T07:24:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.940+0000 7f0478dac640 1 -- 172.21.15.1:0/1229448074 wait complete. 2024-03-06T07:24:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.941+0000 7f0478dac640 1 Processor -- start 2024-03-06T07:24:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.941+0000 7f0478dac640 1 -- start start 2024-03-06T07:24:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.942+0000 7f0478dac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f045c0039b0 0x7f04741ee410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.942+0000 7f0478dac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f047414d070 0x7f047414bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.942+0000 7f0478dac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f047414c520 0x7f04740a83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.942+0000 7f0478dac640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04740c10d0 con 0x7f045c0039b0 2024-03-06T07:24:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.942+0000 7f0478dac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04740c6520 con 0x7f047414c520 2024-03-06T07:24:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.942+0000 7f0478dac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0474159520 con 0x7f047414d070 2024-03-06T07:24:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.942+0000 7f047259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f045c0039b0 0x7f04741ee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.942+0000 7f047259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f045c0039b0 0x7f04741ee410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53378/0 (socket says 172.21.15.1:53378) 2024-03-06T07:24:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.942+0000 7f047259c640 1 -- 172.21.15.1:0/1118366183 learned_addr learned my addr 172.21.15.1:0/1118366183 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.942+0000 7f047259c640 1 -- 172.21.15.1:0/1118366183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f047414d070 msgr2=0x7f047414bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.942+0000 7f0471d9b640 1 --2- 172.21.15.1:0/1118366183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f047414d070 0x7f047414bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.942+0000 7f0472d9d640 1 --2- 172.21.15.1:0/1118366183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f047414c520 0x7f04740a83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.943+0000 7f047259c640 1 --2- 172.21.15.1:0/1118366183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f047414d070 0x7f047414bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.943+0000 7f047259c640 1 -- 172.21.15.1:0/1118366183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f047414c520 msgr2=0x7f04740a83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.943+0000 7f047259c640 1 --2- 172.21.15.1:0/1118366183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f047414c520 0x7f04740a83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.943+0000 7f047259c640 1 -- 172.21.15.1:0/1118366183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f047414ccc0 con 0x7f045c0039b0 2024-03-06T07:24:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.943+0000 7f047259c640 1 --2- 172.21.15.1:0/1118366183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f045c0039b0 0x7f04741ee410 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f046c009af0 tx=0x7f046c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.943+0000 7f0462ffd640 1 -- 172.21.15.1:0/1118366183 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c042020 con 0x7f045c0039b0 2024-03-06T07:24:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.943+0000 7f0462ffd640 1 -- 172.21.15.1:0/1118366183 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046c03a040 con 0x7f045c0039b0 2024-03-06T07:24:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.943+0000 7f0478dac640 1 -- 172.21.15.1:0/1118366183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0474151640 con 0x7f045c0039b0 2024-03-06T07:24:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.943+0000 7f0462ffd640 1 -- 172.21.15.1:0/1118366183 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c039070 con 0x7f045c0039b0 2024-03-06T07:24:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.943+0000 7f0478dac640 1 -- 172.21.15.1:0/1118366183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04740a98d0 con 0x7f045c0039b0 2024-03-06T07:24:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.944+0000 7f0478dac640 1 -- 172.21.15.1:0/1118366183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f047414c900 con 0x7f045c0039b0 2024-03-06T07:24:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.944+0000 7f0462ffd640 1 -- 172.21.15.1:0/1118366183 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f046c045020 con 0x7f045c0039b0 2024-03-06T07:24:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.945+0000 7f0462ffd640 1 --2- 172.21.15.1:0/1118366183 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f045406b500 0x7f045406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.945+0000 7f0462ffd640 1 -- 172.21.15.1:0/1118366183 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f046c0aaef0 con 0x7f045c0039b0 2024-03-06T07:24:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.945+0000 7f0471d9b640 1 --2- 172.21.15.1:0/1118366183 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f045406b500 0x7f045406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.945+0000 7f0471d9b640 1 --2- 172.21.15.1:0/1118366183 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f045406b500 0x7f045406d9c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0474158a70 tx=0x7f0468008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:04.949+0000 7f0462ffd640 1 -- 172.21.15.1:0/1118366183 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f046c079f20 con 0x7f045c0039b0 2024-03-06T07:24:05.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.134+0000 7f0478dac640 1 -- 172.21.15.1:0/1118366183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f04740686e0 con 0x7f045c0039b0 2024-03-06T07:24:05.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.137+0000 7f0462ffd640 1 -- 172.21.15.1:0/1118366183 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v35)=0 v35) v1 ==== 122+0+0 (secure 0 0 0) 0x7f046c079820 con 0x7f045c0039b0 2024-03-06T07:24:05.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.139+0000 7f0478dac640 1 -- 172.21.15.1:0/1118366183 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f045406b500 msgr2=0x7f045406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.139+0000 7f0478dac640 1 --2- 172.21.15.1:0/1118366183 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f045406b500 0x7f045406d9c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0474158a70 tx=0x7f0468008040 comp rx=0 tx=0).stop 2024-03-06T07:24:05.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.139+0000 7f0478dac640 1 -- 172.21.15.1:0/1118366183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f045c0039b0 msgr2=0x7f04741ee410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.139+0000 7f0478dac640 1 --2- 172.21.15.1:0/1118366183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f045c0039b0 0x7f04741ee410 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f046c009af0 tx=0x7f046c004730 comp rx=0 tx=0).stop 2024-03-06T07:24:05.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.139+0000 7f0478dac640 1 -- 172.21.15.1:0/1118366183 shutdown_connections 2024-03-06T07:24:05.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.139+0000 7f0478dac640 1 --2- 172.21.15.1:0/1118366183 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f045406b500 0x7f045406d9c0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.139+0000 7f0478dac640 1 --2- 172.21.15.1:0/1118366183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f047414c520 0x7f04740a83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.139+0000 7f0478dac640 1 --2- 172.21.15.1:0/1118366183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f047414d070 0x7f047414bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.139+0000 7f0478dac640 1 --2- 172.21.15.1:0/1118366183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f045c0039b0 0x7f04741ee410 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.139+0000 7f0478dac640 1 -- 172.21.15.1:0/1118366183 >> 172.21.15.1:0/1118366183 conn(0x7f04740c7a50 msgr2=0x7f047406faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.139+0000 7f0478dac640 1 -- 172.21.15.1:0/1118366183 shutdown_connections 2024-03-06T07:24:05.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.139+0000 7f0478dac640 1 -- 172.21.15.1:0/1118366183 wait complete. 2024-03-06T07:24:05.151 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:05.152 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:05.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:05.243 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:05.243 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:24:05.244 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:05.258 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:24:05.258 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:24:05.258 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:05.301 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:24:05.301 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:24:05.302 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:05.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.401+0000 7fb4545c9640 1 Processor -- start 2024-03-06T07:24:05.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.401+0000 7fb4545c9640 1 -- start start 2024-03-06T07:24:05.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.402+0000 7fb4545c9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb44c0c6b10 0x7fb44c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.402+0000 7fb4545c9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c0c2fc0 con 0x7fb44c0bf9d0 2024-03-06T07:24:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.402+0000 7fb4545c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c158da0 con 0x7fb44c152650 2024-03-06T07:24:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.402+0000 7fb4545c9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb44c155800 con 0x7fb44c0c6b10 2024-03-06T07:24:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.402+0000 7fb452b66640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb44c0bf9d0 0x7fb44c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48792/0 (socket says 172.21.15.1:48792) 2024-03-06T07:24:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.402+0000 7fb452b66640 1 -- 172.21.15.1:0/3712844176 learned_addr learned my addr 172.21.15.1:0/3712844176 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.402+0000 7fb451363640 1 -- 172.21.15.1:0/3712844176 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800881993 0 0) 0x7fb44c0c2fc0 con 0x7fb44c0bf9d0 2024-03-06T07:24:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451363640 1 -- 172.21.15.1:0/3712844176 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb434003520 con 0x7fb44c0bf9d0 2024-03-06T07:24:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451b64640 1 --2- 172.21.15.1:0/3712844176 >> v2:172.21.15.98:3300/0 conn(0x7fb44c0c6b10 0x7fb44c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451363640 1 -- 172.21.15.1:0/3712844176 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1116030422 0 0) 0x7fb434003520 con 0x7fb44c0bf9d0 2024-03-06T07:24:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451363640 1 -- 172.21.15.1:0/3712844176 >> v2:172.21.15.98:3300/0 conn(0x7fb44c0c6b10 msgr2=0x7fb44c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451363640 1 --2- 172.21.15.1:0/3712844176 >> v2:172.21.15.98:3300/0 conn(0x7fb44c0c6b10 0x7fb44c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451363640 1 -- 172.21.15.1:0/3712844176 >> v1:172.21.15.98:6789/0 conn(0x7fb44c152650 legacy=0x7fb44c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451363640 1 -- 172.21.15.1:0/3712844176 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44c0c6690 con 0x7fb44c0bf9d0 2024-03-06T07:24:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451363640 1 -- 172.21.15.1:0/3712844176 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fb43c0019f0 con 0x7fb44c0bf9d0 2024-03-06T07:24:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451363640 1 -- 172.21.15.1:0/3712844176 >> v1:172.21.15.1:6789/0 conn(0x7fb44c0bf9d0 legacy=0x7fb44c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451363640 1 --2- 172.21.15.1:0/3712844176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4340039b0 0x7fb434003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451363640 1 --2- 172.21.15.1:0/3712844176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb434004800 0x7fb434004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451363640 1 --2- 172.21.15.1:0/3712844176 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4340051c0 0x7fb43400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451363640 1 -- 172.21.15.1:0/3712844176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c155800 con 0x7fb434004800 2024-03-06T07:24:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451363640 1 -- 172.21.15.1:0/3712844176 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43400c160 con 0x7fb4340039b0 2024-03-06T07:24:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451363640 1 -- 172.21.15.1:0/3712844176 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43400c360 con 0x7fb4340051c0 2024-03-06T07:24:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451b64640 1 --2- 172.21.15.1:0/3712844176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb434004800 0x7fb434004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb452365640 1 --2- 172.21.15.1:0/3712844176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4340039b0 0x7fb434003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb452b66640 1 --2- 172.21.15.1:0/3712844176 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4340051c0 0x7fb43400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451b64640 1 -- 172.21.15.1:0/3712844176 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4340051c0 msgr2=0x7fb43400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451b64640 1 --2- 172.21.15.1:0/3712844176 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4340051c0 0x7fb43400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451b64640 1 -- 172.21.15.1:0/3712844176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4340039b0 msgr2=0x7fb434003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451b64640 1 --2- 172.21.15.1:0/3712844176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4340039b0 0x7fb434003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb451b64640 1 -- 172.21.15.1:0/3712844176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb434003520 con 0x7fb434004800 2024-03-06T07:24:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb452b66640 1 --2- 172.21.15.1:0/3712844176 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4340051c0 0x7fb43400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.403+0000 7fb452b66640 1 -- 172.21.15.1:0/3712844176 reap_dead start 2024-03-06T07:24:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.404+0000 7fb451b64640 1 --2- 172.21.15.1:0/3712844176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb434004800 0x7fb434004c80 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fb440009ac0 tx=0x7fb434005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d9b7a3d5e7f90c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.404+0000 7fb451363640 1 -- 172.21.15.1:0/3712844176 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb440039070 con 0x7fb434004800 2024-03-06T07:24:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.404+0000 7fb451363640 1 -- 172.21.15.1:0/3712844176 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44002c6b0 con 0x7fb434004800 2024-03-06T07:24:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.404+0000 7fb451363640 1 -- 172.21.15.1:0/3712844176 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb440009690 con 0x7fb434004800 2024-03-06T07:24:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.405+0000 7fb4545c9640 1 -- 172.21.15.1:0/3712844176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb434004800 msgr2=0x7fb434004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.405+0000 7fb4545c9640 1 --2- 172.21.15.1:0/3712844176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb434004800 0x7fb434004c80 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fb440009ac0 tx=0x7fb434005830 comp rx=0 tx=0).stop 2024-03-06T07:24:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.405+0000 7fb4545c9640 1 -- 172.21.15.1:0/3712844176 shutdown_connections 2024-03-06T07:24:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.405+0000 7fb4545c9640 1 --2- 172.21.15.1:0/3712844176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb434004800 0x7fb434004c80 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.405+0000 7fb4545c9640 1 -- 172.21.15.1:0/3712844176 >> 172.21.15.1:0/3712844176 conn(0x7fb44c066ee0 msgr2=0x7fb44c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.405+0000 7fb4545c9640 1 -- 172.21.15.1:0/3712844176 shutdown_connections 2024-03-06T07:24:05.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.405+0000 7fb4545c9640 1 -- 172.21.15.1:0/3712844176 wait complete. 2024-03-06T07:24:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.406+0000 7fb4545c9640 1 Processor -- start 2024-03-06T07:24:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.406+0000 7fb4545c9640 1 -- start start 2024-03-06T07:24:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb4545c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb434004800 0x7fb44c1e0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb4545c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb44c0c5520 0x7fb44c1eae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb4545c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb44c14b120 0x7fb44c14b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb4545c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c1551b0 con 0x7fb434004800 2024-03-06T07:24:05.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb4545c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c0c6fa0 con 0x7fb44c0c5520 2024-03-06T07:24:05.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb4545c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c155490 con 0x7fb44c14b120 2024-03-06T07:24:05.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb452365640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb434004800 0x7fb44c1e0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb452365640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb434004800 0x7fb44c1e0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53396/0 (socket says 172.21.15.1:53396) 2024-03-06T07:24:05.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb452365640 1 -- 172.21.15.1:0/280548296 learned_addr learned my addr 172.21.15.1:0/280548296 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:05.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb451b64640 1 --2- 172.21.15.1:0/280548296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb44c0c5520 0x7fb44c1eae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb452b66640 1 --2- 172.21.15.1:0/280548296 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb44c14b120 0x7fb44c14b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb452365640 1 -- 172.21.15.1:0/280548296 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb44c14b120 msgr2=0x7fb44c14b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb452365640 1 --2- 172.21.15.1:0/280548296 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb44c14b120 0x7fb44c14b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb452365640 1 -- 172.21.15.1:0/280548296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb44c0c5520 msgr2=0x7fb44c1eae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb452365640 1 --2- 172.21.15.1:0/280548296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb44c0c5520 0x7fb44c1eae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb452365640 1 -- 172.21.15.1:0/280548296 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44c1e1720 con 0x7fb434004800 2024-03-06T07:24:05.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb451b64640 1 --2- 172.21.15.1:0/280548296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb44c0c5520 0x7fb44c1eae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:05.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb452365640 1 --2- 172.21.15.1:0/280548296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb434004800 0x7fb44c1e0460 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fb44400f820 tx=0x7fb44400fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb44affd640 1 -- 172.21.15.1:0/280548296 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4440050d0 con 0x7fb434004800 2024-03-06T07:24:05.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb44affd640 1 -- 172.21.15.1:0/280548296 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb444018070 con 0x7fb434004800 2024-03-06T07:24:05.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb44affd640 1 -- 172.21.15.1:0/280548296 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb444005490 con 0x7fb434004800 2024-03-06T07:24:05.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb4545c9640 1 -- 172.21.15.1:0/280548296 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb44c1eee20 con 0x7fb434004800 2024-03-06T07:24:05.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.407+0000 7fb4545c9640 1 -- 172.21.15.1:0/280548296 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb44c1e1440 con 0x7fb434004800 2024-03-06T07:24:05.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.408+0000 7fb4545c9640 1 -- 172.21.15.1:0/280548296 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb44c067fd0 con 0x7fb434004800 2024-03-06T07:24:05.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.409+0000 7fb44affd640 1 -- 172.21.15.1:0/280548296 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb44400d730 con 0x7fb434004800 2024-03-06T07:24:05.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.409+0000 7fb44affd640 1 --2- 172.21.15.1:0/280548296 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb43806b4d0 0x7fb43806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.409+0000 7fb44affd640 1 -- 172.21.15.1:0/280548296 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fb444089ec0 con 0x7fb434004800 2024-03-06T07:24:05.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.409+0000 7fb451b64640 1 --2- 172.21.15.1:0/280548296 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb43806b4d0 0x7fb43806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.410+0000 7fb451b64640 1 --2- 172.21.15.1:0/280548296 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb43806b4d0 0x7fb43806d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb440036000 tx=0x7fb440030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.414+0000 7fb44affd640 1 -- 172.21.15.1:0/280548296 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb444058ef0 con 0x7fb434004800 2024-03-06T07:24:05.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.637+0000 7fb4545c9640 1 -- 172.21.15.1:0/280548296 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb44c0b06f0 con 0x7fb434004800 2024-03-06T07:24:05.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.638+0000 7fb44affd640 1 -- 172.21.15.1:0/280548296 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+3690 (secure 0 0 0) 0x7fb4440587f0 con 0x7fb434004800 2024-03-06T07:24:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:05.640 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":84,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6068,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1246018107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1246018107},{"type":"v1","addr":"172.21.15.98:6836","nonce":1246018107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":84},{"gid":6514,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3611156352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3611156352},{"type":"v1","addr":"172.21.15.98:6837","nonce":3611156352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6981,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/2118550222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2118550222},{"type":"v1","addr":"172.21.15.1:6834","nonce":2118550222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/543699854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":543699854},{"type":"v1","addr":"172.21.15.1:6835","nonce":543699854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[]} 2024-03-06T07:24:05.640 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 84 2024-03-06T07:24:05.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.639+0000 7fb4545c9640 1 -- 172.21.15.1:0/280548296 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb43806b4d0 msgr2=0x7fb43806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.639+0000 7fb4545c9640 1 --2- 172.21.15.1:0/280548296 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb43806b4d0 0x7fb43806d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb440036000 tx=0x7fb440030000 comp rx=0 tx=0).stop 2024-03-06T07:24:05.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.639+0000 7fb4545c9640 1 -- 172.21.15.1:0/280548296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb434004800 msgr2=0x7fb44c1e0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.639+0000 7fb4545c9640 1 --2- 172.21.15.1:0/280548296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb434004800 0x7fb44c1e0460 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fb44400f820 tx=0x7fb44400fce0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.639+0000 7fb4545c9640 1 -- 172.21.15.1:0/280548296 shutdown_connections 2024-03-06T07:24:05.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.639+0000 7fb4545c9640 1 --2- 172.21.15.1:0/280548296 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb43806b4d0 0x7fb43806d990 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.639+0000 7fb4545c9640 1 --2- 172.21.15.1:0/280548296 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb44c14b120 0x7fb44c14b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.639+0000 7fb4545c9640 1 --2- 172.21.15.1:0/280548296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb44c0c5520 0x7fb44c1eae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.639+0000 7fb4545c9640 1 --2- 172.21.15.1:0/280548296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb434004800 0x7fb44c1e0460 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.639+0000 7fb4545c9640 1 -- 172.21.15.1:0/280548296 >> 172.21.15.1:0/280548296 conn(0x7fb44c066ee0 msgr2=0x7fb44c0c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.640+0000 7fb4545c9640 1 -- 172.21.15.1:0/280548296 shutdown_connections 2024-03-06T07:24:05.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.640+0000 7fb4545c9640 1 -- 172.21.15.1:0/280548296 wait complete. 2024-03-06T07:24:05.652 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:24:05.653 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:24:05.653 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:24:05.653 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:05.653 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:24:05.653 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:24:05.653 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:24:05.653 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:05.654 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:24:05.654 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:24:05.654 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:24:05.654 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:05.654 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:24:05.654 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:24:05.654 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:24:05.654 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:05.662 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:24:05.662 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:24:05.663 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:24:05.663 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:24:05.665 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:24:05.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:24:05.666 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:24:05.666 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:24:05.705 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:24:05.705 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:05.706 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:24:05.706 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:05.708 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:24:05.708 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:05.709 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:24:05.709 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:05.709 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:24:05.782 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:24:05.782 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:24:05.791 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:24:05.791 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:24:05.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.815+0000 7f20b5439640 1 Processor -- start 2024-03-06T07:24:05.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.815+0000 7f20b5439640 1 -- start start 2024-03-06T07:24:05.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.816+0000 7f20b5439640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f20b00c5960 0x7f20b00c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.816+0000 7f20b5439640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20b0153ab0 0x7f20b0070b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.816+0000 7f20b5439640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b00c67c0 con 0x7f20b00c4ec0 2024-03-06T07:24:05.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.816+0000 7f20b5439640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20b00c6a60 con 0x7f20b00c5960 2024-03-06T07:24:05.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.816+0000 7f20b5439640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20b00c6d40 con 0x7f20b0153ab0 2024-03-06T07:24:05.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.816+0000 7f20aeffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f20b00c5960 0x7f20b00c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.816+0000 7f20af7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f20b00c4ec0 0x7f20b0156620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48814/0 (socket says 172.21.15.1:48814) 2024-03-06T07:24:05.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.816+0000 7f20af7fe640 1 -- 172.21.15.1:0/62477859 learned_addr learned my addr 172.21.15.1:0/62477859 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:05.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.816+0000 7f20a7fff640 1 --2- 172.21.15.1:0/62477859 >> v2:172.21.15.98:3300/0 conn(0x7f20b0153ab0 0x7f20b0070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.817+0000 7f20aeffd640 1 -- 172.21.15.1:0/62477859 >> v2:172.21.15.98:3300/0 conn(0x7f20b0153ab0 msgr2=0x7f20b0070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.817+0000 7f20aeffd640 1 --2- 172.21.15.1:0/62477859 >> v2:172.21.15.98:3300/0 conn(0x7f20b0153ab0 0x7f20b0070b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.817+0000 7f20aeffd640 1 -- 172.21.15.1:0/62477859 >> v1:172.21.15.1:6789/0 conn(0x7f20b00c4ec0 legacy=0x7f20b0156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.817+0000 7f20aeffd640 1 -- 172.21.15.1:0/62477859 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b00c7020 con 0x7f20b00c5960 2024-03-06T07:24:05.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.817+0000 7f20a7fff640 1 --2- 172.21.15.1:0/62477859 >> v2:172.21.15.98:3300/0 conn(0x7f20b0153ab0 0x7f20b0070b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:05.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.818+0000 7f20aeffd640 1 --2- 172.21.15.1:0/62477859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b00c5960 0x7f20b00c4980 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f20b0071840 tx=0x7f20a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e12818a09b6dd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.819+0000 7f20ae7fc640 1 -- 172.21.15.1:0/62477859 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0035020 con 0x7f20b00c5960 2024-03-06T07:24:05.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.819+0000 7f20ae7fc640 1 -- 172.21.15.1:0/62477859 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a0031830 con 0x7f20b00c5960 2024-03-06T07:24:05.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.819+0000 7f20ae7fc640 1 -- 172.21.15.1:0/62477859 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0031c00 con 0x7f20b00c5960 2024-03-06T07:24:05.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.820+0000 7f20b5439640 1 -- 172.21.15.1:0/62477859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b00c5960 msgr2=0x7f20b00c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.820+0000 7f20b5439640 1 --2- 172.21.15.1:0/62477859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b00c5960 0x7f20b00c4980 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f20b0071840 tx=0x7f20a002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.820+0000 7f20b5439640 1 -- 172.21.15.1:0/62477859 shutdown_connections 2024-03-06T07:24:05.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.820+0000 7f20b5439640 1 --2- 172.21.15.1:0/62477859 >> v2:172.21.15.98:3300/0 conn(0x7f20b0153ab0 0x7f20b0070b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.820+0000 7f20b5439640 1 --2- 172.21.15.1:0/62477859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b00c5960 0x7f20b00c4980 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.820+0000 7f20b5439640 1 -- 172.21.15.1:0/62477859 >> 172.21.15.1:0/62477859 conn(0x7f20b00aa550 msgr2=0x7f20b00aa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.820+0000 7f20b5439640 1 -- 172.21.15.1:0/62477859 shutdown_connections 2024-03-06T07:24:05.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.820+0000 7f20b5439640 1 -- 172.21.15.1:0/62477859 wait complete. 2024-03-06T07:24:05.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.821+0000 7f20b5439640 1 Processor -- start 2024-03-06T07:24:05.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.821+0000 7f20b5439640 1 -- start start 2024-03-06T07:24:05.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20b5439640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20b00c4ec0 0x7f20b00c0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20b5439640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b00c5960 0x7f20b00c10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20b5439640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20b0153ab0 0x7f20b014db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20b5439640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b006f550 con 0x7f20b00c5960 2024-03-06T07:24:05.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20b5439640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b006f3d0 con 0x7f20b00c4ec0 2024-03-06T07:24:05.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20b5439640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b00c5d40 con 0x7f20b0153ab0 2024-03-06T07:24:05.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20a7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b00c5960 0x7f20b00c10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20a7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b00c5960 0x7f20b00c10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53462/0 (socket says 172.21.15.1:53462) 2024-03-06T07:24:05.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20a7fff640 1 -- 172.21.15.1:0/2199335794 learned_addr learned my addr 172.21.15.1:0/2199335794 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:05.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20aeffd640 1 --2- 172.21.15.1:0/2199335794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20b00c4ec0 0x7f20b00c0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20af7fe640 1 --2- 172.21.15.1:0/2199335794 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20b0153ab0 0x7f20b014db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20a7fff640 1 -- 172.21.15.1:0/2199335794 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20b0153ab0 msgr2=0x7f20b014db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20a7fff640 1 --2- 172.21.15.1:0/2199335794 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20b0153ab0 0x7f20b014db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20a7fff640 1 -- 172.21.15.1:0/2199335794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20b00c4ec0 msgr2=0x7f20b00c0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20a7fff640 1 --2- 172.21.15.1:0/2199335794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20b00c4ec0 0x7f20b00c0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20a7fff640 1 -- 172.21.15.1:0/2199335794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b00bfc40 con 0x7f20b00c5960 2024-03-06T07:24:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20a7fff640 1 --2- 172.21.15.1:0/2199335794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b00c5960 0x7f20b00c10e0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f209800ec10 tx=0x7f209800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20a77fe640 1 -- 172.21.15.1:0/2199335794 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098017070 con 0x7f20b00c5960 2024-03-06T07:24:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20a77fe640 1 -- 172.21.15.1:0/2199335794 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209800cc30 con 0x7f20b00c5960 2024-03-06T07:24:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20b5439640 1 -- 172.21.15.1:0/2199335794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b00be370 con 0x7f20b00c5960 2024-03-06T07:24:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.822+0000 7f20a77fe640 1 -- 172.21.15.1:0/2199335794 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098013470 con 0x7f20b00c5960 2024-03-06T07:24:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.823+0000 7f20b5439640 1 -- 172.21.15.1:0/2199335794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b00be930 con 0x7f20b00c5960 2024-03-06T07:24:05.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.824+0000 7f20b5439640 1 -- 172.21.15.1:0/2199335794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b0067fd0 con 0x7f20b00c5960 2024-03-06T07:24:05.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.824+0000 7f20a77fe640 1 -- 172.21.15.1:0/2199335794 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f209801a070 con 0x7f20b00c5960 2024-03-06T07:24:05.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.824+0000 7f20a77fe640 1 --2- 172.21.15.1:0/2199335794 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f209006b500 0x7f209006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.824+0000 7f20a77fe640 1 -- 172.21.15.1:0/2199335794 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f209808a2a0 con 0x7f20b00c5960 2024-03-06T07:24:05.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.824+0000 7f20aeffd640 1 --2- 172.21.15.1:0/2199335794 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f209006b500 0x7f209006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.825+0000 7f20aeffd640 1 --2- 172.21.15.1:0/2199335794 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f209006b500 0x7f209006d9c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f20a0036000 tx=0x7f20a0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:05.829+0000 7f20a77fe640 1 -- 172.21.15.1:0/2199335794 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2098058c80 con 0x7f20b00c5960 2024-03-06T07:24:06.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.022+0000 7f20b5439640 1 -- 172.21.15.1:0/2199335794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f20b00c7a70 con 0x7f20b00c5960 2024-03-06T07:24:06.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.836+0000 7f20a77fe640 1 -- 172.21.15.1:0/2199335794 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v104) v1 ==== 97+0+0 (secure 0 0 0) 0x7f209805cd20 con 0x7f20b00c5960 2024-03-06T07:24:06.838 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-06T07:24:06.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.839+0000 7f20b5439640 1 -- 172.21.15.1:0/2199335794 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f209006b500 msgr2=0x7f209006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.839+0000 7f20b5439640 1 --2- 172.21.15.1:0/2199335794 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f209006b500 0x7f209006d9c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f20a0036000 tx=0x7f20a0008030 comp rx=0 tx=0).stop 2024-03-06T07:24:06.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.839+0000 7f20b5439640 1 -- 172.21.15.1:0/2199335794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b00c5960 msgr2=0x7f20b00c10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.839+0000 7f20b5439640 1 --2- 172.21.15.1:0/2199335794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b00c5960 0x7f20b00c10e0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f209800ec10 tx=0x7f209800c520 comp rx=0 tx=0).stop 2024-03-06T07:24:06.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.839+0000 7f20b5439640 1 -- 172.21.15.1:0/2199335794 shutdown_connections 2024-03-06T07:24:06.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.839+0000 7f20b5439640 1 --2- 172.21.15.1:0/2199335794 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f209006b500 0x7f209006d9c0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.839+0000 7f20b5439640 1 --2- 172.21.15.1:0/2199335794 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20b0153ab0 0x7f20b014db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.839+0000 7f20b5439640 1 --2- 172.21.15.1:0/2199335794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b00c5960 0x7f20b00c10e0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.839+0000 7f20b5439640 1 --2- 172.21.15.1:0/2199335794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20b00c4ec0 0x7f20b00c0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.839+0000 7f20b5439640 1 -- 172.21.15.1:0/2199335794 >> 172.21.15.1:0/2199335794 conn(0x7f20b00aa550 msgr2=0x7f20b0151d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.839+0000 7f20b5439640 1 -- 172.21.15.1:0/2199335794 shutdown_connections 2024-03-06T07:24:06.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.839+0000 7f20b5439640 1 -- 172.21.15.1:0/2199335794 wait complete. 2024-03-06T07:24:06.854 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:24:06.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.961+0000 7f9ecf3b6640 1 Processor -- start 2024-03-06T07:24:06.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.961+0000 7f9ecf3b6640 1 -- start start 2024-03-06T07:24:06.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.962+0000 7f9ecf3b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ec8152650 0x7f9ec8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.962+0000 7f9ecf3b6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9ec80bf9d0 0x7f9ec80c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.962+0000 7f9ecf3b6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec80c2fc0 con 0x7f9ec80c6b10 2024-03-06T07:24:06.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.962+0000 7f9ecf3b6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ec8155800 con 0x7f9ec80bf9d0 2024-03-06T07:24:06.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.962+0000 7f9ecf3b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec8158da0 con 0x7f9ec8152650 2024-03-06T07:24:06.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.962+0000 7f9ecc951640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9ec80c6b10 0x7f9ec8158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48820/0 (socket says 172.21.15.1:48820) 2024-03-06T07:24:06.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.962+0000 7f9ecc951640 1 -- 172.21.15.1:0/4210231387 learned_addr learned my addr 172.21.15.1:0/4210231387 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:06.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.962+0000 7f9ecd953640 1 --2- 172.21.15.1:0/4210231387 >> v2:172.21.15.98:3300/0 conn(0x7f9ec80bf9d0 0x7f9ec80c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.962+0000 7f9ecd152640 1 --2- 172.21.15.1:0/4210231387 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ec8152650 0x7f9ec8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.962+0000 7f9ebffff640 1 -- 172.21.15.1:0/4210231387 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116755849 0 0) 0x7f9ec80c2fc0 con 0x7f9ec80c6b10 2024-03-06T07:24:06.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.962+0000 7f9ebffff640 1 -- 172.21.15.1:0/4210231387 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9eb4003520 con 0x7f9ec80c6b10 2024-03-06T07:24:06.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.962+0000 7f9ebffff640 1 -- 172.21.15.1:0/4210231387 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3819155019 0 0) 0x7f9eb4003520 con 0x7f9ec80c6b10 2024-03-06T07:24:06.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ebffff640 1 -- 172.21.15.1:0/4210231387 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ec8152650 msgr2=0x7f9ec8155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ebffff640 1 --2- 172.21.15.1:0/4210231387 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ec8152650 0x7f9ec8155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ebffff640 1 -- 172.21.15.1:0/4210231387 >> v2:172.21.15.98:3300/0 conn(0x7f9ec80bf9d0 msgr2=0x7f9ec80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ebffff640 1 --2- 172.21.15.1:0/4210231387 >> v2:172.21.15.98:3300/0 conn(0x7f9ec80bf9d0 0x7f9ec80c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ebffff640 1 -- 172.21.15.1:0/4210231387 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec80c6690 con 0x7f9ec80c6b10 2024-03-06T07:24:06.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ebffff640 1 -- 172.21.15.1:0/4210231387 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f9eb8001670 con 0x7f9ec80c6b10 2024-03-06T07:24:06.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ebffff640 1 -- 172.21.15.1:0/4210231387 >> v1:172.21.15.1:6789/0 conn(0x7f9ec80c6b10 legacy=0x7f9ec8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ebffff640 1 --2- 172.21.15.1:0/4210231387 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ec8152650 0x7f9eb40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ebffff640 1 --2- 172.21.15.1:0/4210231387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb4003f30 0x7f9eb4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ebffff640 1 --2- 172.21.15.1:0/4210231387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb4004d40 0x7f9eb40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ebffff640 1 -- 172.21.15.1:0/4210231387 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec8155800 con 0x7f9eb4004d40 2024-03-06T07:24:06.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ebffff640 1 -- 172.21.15.1:0/4210231387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec8158da0 con 0x7f9eb4003f30 2024-03-06T07:24:06.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ebffff640 1 -- 172.21.15.1:0/4210231387 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb400ba10 con 0x7f9ec8152650 2024-03-06T07:24:06.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ecd953640 1 --2- 172.21.15.1:0/4210231387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb4004d40 0x7f9eb40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ecd152640 1 --2- 172.21.15.1:0/4210231387 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ec8152650 0x7f9eb40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ecc951640 1 --2- 172.21.15.1:0/4210231387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb4003f30 0x7f9eb4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ecd953640 1 -- 172.21.15.1:0/4210231387 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ec8152650 msgr2=0x7f9eb40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ecd953640 1 --2- 172.21.15.1:0/4210231387 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ec8152650 0x7f9eb40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ecd953640 1 -- 172.21.15.1:0/4210231387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb4003f30 msgr2=0x7f9eb4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ecd953640 1 --2- 172.21.15.1:0/4210231387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb4003f30 0x7f9eb4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.963+0000 7f9ecd953640 1 -- 172.21.15.1:0/4210231387 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb4003520 con 0x7f9eb4004d40 2024-03-06T07:24:06.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.964+0000 7f9ecd953640 1 --2- 172.21.15.1:0/4210231387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb4004d40 0x7f9eb40051c0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f9ec000e170 tx=0x7f9eb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77c2f64b6297da9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.964+0000 7f9ebffff640 1 -- 172.21.15.1:0/4210231387 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec0039070 con 0x7f9eb4004d40 2024-03-06T07:24:06.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.964+0000 7f9ebffff640 1 -- 172.21.15.1:0/4210231387 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec002cc20 con 0x7f9eb4004d40 2024-03-06T07:24:06.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.964+0000 7f9ebffff640 1 -- 172.21.15.1:0/4210231387 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec0009c10 con 0x7f9eb4004d40 2024-03-06T07:24:06.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.964+0000 7f9ecf3b6640 1 -- 172.21.15.1:0/4210231387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb4004d40 msgr2=0x7f9eb40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.964+0000 7f9ecf3b6640 1 --2- 172.21.15.1:0/4210231387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb4004d40 0x7f9eb40051c0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f9ec000e170 tx=0x7f9eb4005830 comp rx=0 tx=0).stop 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.964+0000 7f9ecd953640 1 -- 172.21.15.1:0/4210231387 reap_dead start 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.965+0000 7f9ecf3b6640 1 -- 172.21.15.1:0/4210231387 shutdown_connections 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.965+0000 7f9ecf3b6640 1 -- 172.21.15.1:0/4210231387 >> 172.21.15.1:0/4210231387 conn(0x7f9ec8066ee0 msgr2=0x7f9ec80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.965+0000 7f9ecf3b6640 1 -- 172.21.15.1:0/4210231387 shutdown_connections 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.965+0000 7f9ecf3b6640 1 -- 172.21.15.1:0/4210231387 wait complete. 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.966+0000 7f9ecf3b6640 1 Processor -- start 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.966+0000 7f9ecf3b6640 1 -- start start 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.966+0000 7f9ecf3b6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb4004d40 0x7f9ec8151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.966+0000 7f9ecf3b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ec81eaf80 0x7f9ec814d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.966+0000 7f9ecf3b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ec814d990 0x7f9ec81de700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.966+0000 7f9ecf3b6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec81551b0 con 0x7f9eb4004d40 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.966+0000 7f9ecf3b6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec80c6fa0 con 0x7f9ec814d990 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.966+0000 7f9ecf3b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec8155490 con 0x7f9ec81eaf80 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.966+0000 7f9ecd152640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb4004d40 0x7f9ec8151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.967+0000 7f9ecd152640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb4004d40 0x7f9ec8151070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53478/0 (socket says 172.21.15.1:53478) 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.967+0000 7f9ecd152640 1 -- 172.21.15.1:0/3837395569 learned_addr learned my addr 172.21.15.1:0/3837395569 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.967+0000 7f9ecd953640 1 --2- 172.21.15.1:0/3837395569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ec814d990 0x7f9ec81de700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.967+0000 7f9ecc951640 1 --2- 172.21.15.1:0/3837395569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ec81eaf80 0x7f9ec814d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.967+0000 7f9ecd152640 1 -- 172.21.15.1:0/3837395569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ec81eaf80 msgr2=0x7f9ec814d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.967+0000 7f9ecd152640 1 --2- 172.21.15.1:0/3837395569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ec81eaf80 0x7f9ec814d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.967+0000 7f9ecd152640 1 -- 172.21.15.1:0/3837395569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ec814d990 msgr2=0x7f9ec81de700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.967+0000 7f9ecd152640 1 --2- 172.21.15.1:0/3837395569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ec814d990 0x7f9ec81de700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.967+0000 7f9ecd152640 1 -- 172.21.15.1:0/3837395569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec81e12b0 con 0x7f9eb4004d40 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.967+0000 7f9ecd152640 1 --2- 172.21.15.1:0/3837395569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb4004d40 0x7f9ec8151070 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4011980 tx=0x7f9ec4011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.967+0000 7f9ebdffb640 1 -- 172.21.15.1:0/3837395569 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec400f9d0 con 0x7f9eb4004d40 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.967+0000 7f9ebdffb640 1 -- 172.21.15.1:0/3837395569 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec400fb70 con 0x7f9eb4004d40 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.967+0000 7f9ebdffb640 1 -- 172.21.15.1:0/3837395569 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4014440 con 0x7f9eb4004d40 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.967+0000 7f9ecf3b6640 1 -- 172.21.15.1:0/3837395569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec81e1550 con 0x7f9eb4004d40 2024-03-06T07:24:06.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.967+0000 7f9ecf3b6640 1 -- 172.21.15.1:0/3837395569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec81e1b10 con 0x7f9eb4004d40 2024-03-06T07:24:06.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.968+0000 7f9ecf3b6640 1 -- 172.21.15.1:0/3837395569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec8067fd0 con 0x7f9eb4004d40 2024-03-06T07:24:06.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.969+0000 7f9ebdffb640 1 -- 172.21.15.1:0/3837395569 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ec4012040 con 0x7f9eb4004d40 2024-03-06T07:24:06.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.969+0000 7f9ebdffb640 1 --2- 172.21.15.1:0/3837395569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9eac06b4d0 0x7f9eac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.970+0000 7f9ebdffb640 1 -- 172.21.15.1:0/3837395569 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9ec408a5c0 con 0x7f9eb4004d40 2024-03-06T07:24:06.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.970+0000 7f9ecc951640 1 --2- 172.21.15.1:0/3837395569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9eac06b4d0 0x7f9eac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.970+0000 7f9ecc951640 1 --2- 172.21.15.1:0/3837395569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9eac06b4d0 0x7f9eac06d990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f9ec81eb3a0 tx=0x7f9eb80078f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:06.974+0000 7f9ebdffb640 1 -- 172.21.15.1:0/3837395569 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ec4019030 con 0x7f9eb4004d40 2024-03-06T07:24:07.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.224+0000 7f9ecf3b6640 1 -- 172.21.15.1:0/3837395569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9ec80b06f0 con 0x7f9eb4004d40 2024-03-06T07:24:07.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.224+0000 7f9ebdffb640 1 -- 172.21.15.1:0/3837395569 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f9ec40595f0 con 0x7f9eb4004d40 2024-03-06T07:24:07.226 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:07.226 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:24:07.227 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-06T07:24:07.227 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:07.227 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-06T07:24:07.227 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlnxyWARAAKIPXU0IafNQ1RzBPX+3DYw==", 2024-03-06T07:24:07.227 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:07.227 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:24:07.227 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:24:07.227 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:24:07.227 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:07.227 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.227 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:07.227 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:07.227 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-06T07:24:07.227 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl2fXxDRAAuLqH+HpKcN1JTts9ZtDTXA==", 2024-03-06T07:24:07.228 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:07.228 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:24:07.228 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:24:07.228 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:24:07.228 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:07.228 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.228 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:07.228 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:07.228 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-06T07:24:07.228 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl+IzqCBAAmgLkarFGt9Ihs/8kJYicBQ==", 2024-03-06T07:24:07.228 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:07.228 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:24:07.228 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:24:07.228 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:24:07.229 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:07.229 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.229 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:07.229 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:07.229 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-06T07:24:07.229 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlipdRFRAAA34vrsTEoX3L1CKb1YAV9g==", 2024-03-06T07:24:07.229 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:07.229 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:24:07.229 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:24:07.229 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:24:07.229 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:07.229 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.229 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:07.230 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:07.230 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-06T07:24:07.230 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAGGehlQK39LxAAOpPIZvl3FqQwCgyo3yoOGQ==", 2024-03-06T07:24:07.230 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:07.230 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:07.230 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:24:07.230 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:07.230 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.230 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:07.230 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:07.230 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-06T07:24:07.230 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAJGehlTxwDLhAAMwN9E0+3Jh5XNXnLo+fw3g==", 2024-03-06T07:24:07.230 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:07.230 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:07.231 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:24:07.231 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:07.231 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.231 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:07.231 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:07.231 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-06T07:24:07.231 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAMGehl4qrlLBAAm0VWQRNvSQIpDx5xf4Fyig==", 2024-03-06T07:24:07.231 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:07.231 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:07.231 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:24:07.231 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:07.231 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.231 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:07.231 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:07.232 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-06T07:24:07.232 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAPGehllO5HKxAAozm6ycG+fRxatoZTogeJTA==", 2024-03-06T07:24:07.232 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:07.232 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:07.232 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:24:07.232 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:07.232 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.232 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:07.232 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:07.232 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-06T07:24:07.232 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAVGehlLtLBMhAAUrV6q6ZCQluIcoo/S/NdbA==", 2024-03-06T07:24:07.232 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:07.232 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:07.232 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:24:07.233 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:07.233 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.233 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:07.233 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:07.233 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-06T07:24:07.233 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAYGehlHp59MRAAxxgpSPTEZkYKjl3fr0xeeQ==", 2024-03-06T07:24:07.233 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:07.233 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:07.233 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:24:07.233 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:07.233 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.233 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:07.233 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:07.233 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-06T07:24:07.234 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAbGehlTnXcMBAAFOBqcnh5ubAz+o6AGjccWQ==", 2024-03-06T07:24:07.234 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:07.234 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:07.234 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:24:07.234 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:07.234 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.234 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:07.234 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:07.234 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-06T07:24:07.234 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAeGehl1qH1LhAAaSl7Aj7qlN6d4xt5SKAmcg==", 2024-03-06T07:24:07.234 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:07.234 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:07.234 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:24:07.234 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:07.235 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.235 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:07.235 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:07.235 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-06T07:24:07.235 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng==", 2024-03-06T07:24:07.235 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:07.235 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:24:07.235 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-06T07:24:07.235 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:07.235 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.235 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:07.235 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:07.235 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-06T07:24:07.235 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlMyv7HBAAi7vt7aZWEaZuMpFrB5ofZw==", 2024-03-06T07:24:07.235 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:07.236 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:24:07.236 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-06T07:24:07.236 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-06T07:24:07.236 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-06T07:24:07.236 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.236 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:07.236 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:07.236 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-06T07:24:07.236 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlBS/vIBAAN86N2K4ze5PJB8kR869Lfg==", 2024-03-06T07:24:07.236 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:07.236 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:24:07.236 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:24:07.236 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:24:07.236 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:07.237 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.237 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:07.237 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:07.237 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-06T07:24:07.237 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlJ/jPORAAhLk5MTGafsGKhZSYhvhBZg==", 2024-03-06T07:24:07.237 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:07.237 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:24:07.237 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:07.237 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:07.237 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.237 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:07.237 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:07.237 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-06T07:24:07.237 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlAl8rNhAAvdU0Qvo/FJYnxc4/H8MIqQ==", 2024-03-06T07:24:07.237 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:07.238 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:24:07.238 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:07.238 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:07.238 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.238 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:07.238 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-06T07:24:07.238 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:24:07.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.227+0000 7f9ecf3b6640 1 -- 172.21.15.1:0/3837395569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9eac06b4d0 msgr2=0x7f9eac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.227+0000 7f9ecf3b6640 1 --2- 172.21.15.1:0/3837395569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9eac06b4d0 0x7f9eac06d990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f9ec81eb3a0 tx=0x7f9eb80078f0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.227+0000 7f9ecf3b6640 1 -- 172.21.15.1:0/3837395569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb4004d40 msgr2=0x7f9ec8151070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.227+0000 7f9ecf3b6640 1 --2- 172.21.15.1:0/3837395569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb4004d40 0x7f9ec8151070 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4011980 tx=0x7f9ec4011d80 comp rx=0 tx=0).stop 2024-03-06T07:24:07.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.227+0000 7f9ecf3b6640 1 -- 172.21.15.1:0/3837395569 shutdown_connections 2024-03-06T07:24:07.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.227+0000 7f9ecf3b6640 1 --2- 172.21.15.1:0/3837395569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9eac06b4d0 0x7f9eac06d990 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.227+0000 7f9ecf3b6640 1 --2- 172.21.15.1:0/3837395569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ec814d990 0x7f9ec81de700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.227+0000 7f9ecf3b6640 1 --2- 172.21.15.1:0/3837395569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ec81eaf80 0x7f9ec814d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.227+0000 7f9ecf3b6640 1 --2- 172.21.15.1:0/3837395569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb4004d40 0x7f9ec8151070 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.227+0000 7f9ecf3b6640 1 -- 172.21.15.1:0/3837395569 >> 172.21.15.1:0/3837395569 conn(0x7f9ec8066ee0 msgr2=0x7f9ec814ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.227+0000 7f9ecf3b6640 1 -- 172.21.15.1:0/3837395569 shutdown_connections 2024-03-06T07:24:07.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.227+0000 7f9ecf3b6640 1 -- 172.21.15.1:0/3837395569 wait complete. 2024-03-06T07:24:07.241 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:07.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.341+0000 7f859131e640 1 Processor -- start 2024-03-06T07:24:07.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.341+0000 7f859131e640 1 -- start start 2024-03-06T07:24:07.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.342+0000 7f859131e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f858c06bc20 0x7f858c152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.342+0000 7f859131e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858c1569e0 con 0x7f858c14cdc0 2024-03-06T07:24:07.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.342+0000 7f859131e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858c156cc0 con 0x7f858c156500 2024-03-06T07:24:07.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.342+0000 7f859131e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f858c0c2dd0 con 0x7f858c06bc20 2024-03-06T07:24:07.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.342+0000 7f858b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f858c06bc20 0x7f858c152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.342+0000 7f858affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f858c14cdc0 0x7f858c0c2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48828/0 (socket says 172.21.15.1:48828) 2024-03-06T07:24:07.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.342+0000 7f858b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f858c06bc20 0x7f858c152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53484/0 (socket says 172.21.15.1:53484) 2024-03-06T07:24:07.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.342+0000 7f858affd640 1 -- 172.21.15.1:0/1136657306 learned_addr learned my addr 172.21.15.1:0/1136657306 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:07.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.342+0000 7f858b7fe640 1 -- 172.21.15.1:0/1136657306 >> v1:172.21.15.98:6789/0 conn(0x7f858c156500 legacy=0x7f858c06b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.342+0000 7f858b7fe640 1 -- 172.21.15.1:0/1136657306 >> v1:172.21.15.1:6789/0 conn(0x7f858c14cdc0 legacy=0x7f858c0c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.342+0000 7f8589ffb640 1 -- 172.21.15.1:0/1136657306 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720957147 0 0) 0x7f858c1569e0 con 0x7f858c14cdc0 2024-03-06T07:24:07.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.342+0000 7f858b7fe640 1 -- 172.21.15.1:0/1136657306 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858c156fa0 con 0x7f858c06bc20 2024-03-06T07:24:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.343+0000 7f858b7fe640 1 --2- 172.21.15.1:0/1136657306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f858c06bc20 0x7f858c152f10 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f857c009d90 tx=0x7f857c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff11a77bc3e881c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.343+0000 7f8589ffb640 1 -- 172.21.15.1:0/1136657306 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857c039070 con 0x7f858c06bc20 2024-03-06T07:24:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.343+0000 7f8589ffb640 1 -- 172.21.15.1:0/1136657306 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857c02cc60 con 0x7f858c06bc20 2024-03-06T07:24:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.343+0000 7f8589ffb640 1 -- 172.21.15.1:0/1136657306 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857c034660 con 0x7f858c06bc20 2024-03-06T07:24:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.344+0000 7f859131e640 1 -- 172.21.15.1:0/1136657306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f858c06bc20 msgr2=0x7f858c152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.344+0000 7f859131e640 1 --2- 172.21.15.1:0/1136657306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f858c06bc20 0x7f858c152f10 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f857c009d90 tx=0x7f857c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.344+0000 7f859131e640 1 -- 172.21.15.1:0/1136657306 shutdown_connections 2024-03-06T07:24:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.344+0000 7f859131e640 1 --2- 172.21.15.1:0/1136657306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f858c06bc20 0x7f858c152f10 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.344+0000 7f859131e640 1 -- 172.21.15.1:0/1136657306 >> 172.21.15.1:0/1136657306 conn(0x7f858c0a9700 msgr2=0x7f858c0a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.344+0000 7f859131e640 1 -- 172.21.15.1:0/1136657306 shutdown_connections 2024-03-06T07:24:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.344+0000 7f859131e640 1 -- 172.21.15.1:0/1136657306 wait complete. 2024-03-06T07:24:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.345+0000 7f859131e640 1 Processor -- start 2024-03-06T07:24:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.345+0000 7f859131e640 1 -- start start 2024-03-06T07:24:07.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f859131e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f858c06bc20 0x7f858c1e9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f859131e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f858c14cdc0 0x7f858c0aa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f859131e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f858c156500 0x7f858c14f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f859131e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858c0c65e0 con 0x7f858c14cdc0 2024-03-06T07:24:07.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f859131e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858c0c5a70 con 0x7f858c156500 2024-03-06T07:24:07.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f859131e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f858c0c2a60 con 0x7f858c06bc20 2024-03-06T07:24:07.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f858a7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f858c14cdc0 0x7f858c0aa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f858a7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f858c14cdc0 0x7f858c0aa5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53486/0 (socket says 172.21.15.1:53486) 2024-03-06T07:24:07.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f858a7fc640 1 -- 172.21.15.1:0/3037095566 learned_addr learned my addr 172.21.15.1:0/3037095566 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:07.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f858affd640 1 --2- 172.21.15.1:0/3037095566 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f858c06bc20 0x7f858c1e9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f858b7fe640 1 --2- 172.21.15.1:0/3037095566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f858c156500 0x7f858c14f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f858a7fc640 1 -- 172.21.15.1:0/3037095566 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f858c06bc20 msgr2=0x7f858c1e9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f858a7fc640 1 --2- 172.21.15.1:0/3037095566 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f858c06bc20 0x7f858c1e9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f858a7fc640 1 -- 172.21.15.1:0/3037095566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f858c156500 msgr2=0x7f858c14f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f858a7fc640 1 --2- 172.21.15.1:0/3037095566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f858c156500 0x7f858c14f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f858a7fc640 1 -- 172.21.15.1:0/3037095566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858c1e9c60 con 0x7f858c14cdc0 2024-03-06T07:24:07.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f858a7fc640 1 --2- 172.21.15.1:0/3037095566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f858c14cdc0 0x7f858c0aa5b0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f858000c400 tx=0x7f858000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.346+0000 7f85737fe640 1 -- 172.21.15.1:0/3037095566 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8580017070 con 0x7f858c14cdc0 2024-03-06T07:24:07.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.347+0000 7f85737fe640 1 -- 172.21.15.1:0/3037095566 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85800052d0 con 0x7f858c14cdc0 2024-03-06T07:24:07.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.347+0000 7f859131e640 1 -- 172.21.15.1:0/3037095566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f858c0aaaf0 con 0x7f858c14cdc0 2024-03-06T07:24:07.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.347+0000 7f85737fe640 1 -- 172.21.15.1:0/3037095566 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8580003830 con 0x7f858c14cdc0 2024-03-06T07:24:07.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.347+0000 7f859131e640 1 -- 172.21.15.1:0/3037095566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f858c0ad880 con 0x7f858c14cdc0 2024-03-06T07:24:07.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.348+0000 7f859131e640 1 -- 172.21.15.1:0/3037095566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f858c0adac0 con 0x7f858c14cdc0 2024-03-06T07:24:07.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.348+0000 7f85737fe640 1 -- 172.21.15.1:0/3037095566 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f858001a070 con 0x7f858c14cdc0 2024-03-06T07:24:07.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.348+0000 7f85737fe640 1 --2- 172.21.15.1:0/3037095566 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f857806b4d0 0x7f857806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.348+0000 7f85737fe640 1 -- 172.21.15.1:0/3037095566 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8580089330 con 0x7f858c14cdc0 2024-03-06T07:24:07.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.348+0000 7f858affd640 1 --2- 172.21.15.1:0/3037095566 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f857806b4d0 0x7f857806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.349+0000 7f858affd640 1 --2- 172.21.15.1:0/3037095566 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f857806b4d0 0x7f857806d990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8584000f60 tx=0x7f8584007bb0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.353+0000 7f85737fe640 1 -- 172.21.15.1:0/3037095566 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f858c0adac0 con 0x7f858c14cdc0 2024-03-06T07:24:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.535+0000 7f859131e640 1 -- 172.21.15.1:0/3037095566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f858c152cc0 con 0x7f858c14cdc0 2024-03-06T07:24:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.536+0000 7f85737fe640 1 -- 172.21.15.1:0/3037095566 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+13046 (secure 0 0 0) 0x7f858c152cc0 con 0x7f858c14cdc0 2024-03-06T07:24:07.538 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:07.538 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":104,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:24:06.835376+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:07.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.538+0000 7f859131e640 1 -- 172.21.15.1:0/3037095566 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f857806b4d0 msgr2=0x7f857806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.538+0000 7f859131e640 1 --2- 172.21.15.1:0/3037095566 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f857806b4d0 0x7f857806d990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8584000f60 tx=0x7f8584007bb0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.538+0000 7f859131e640 1 -- 172.21.15.1:0/3037095566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f858c14cdc0 msgr2=0x7f858c0aa5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.538+0000 7f859131e640 1 --2- 172.21.15.1:0/3037095566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f858c14cdc0 0x7f858c0aa5b0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f858000c400 tx=0x7f858000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.538+0000 7f859131e640 1 -- 172.21.15.1:0/3037095566 shutdown_connections 2024-03-06T07:24:07.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.538+0000 7f859131e640 1 --2- 172.21.15.1:0/3037095566 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f857806b4d0 0x7f857806d990 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.538+0000 7f859131e640 1 --2- 172.21.15.1:0/3037095566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f858c156500 0x7f858c14f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.538+0000 7f859131e640 1 --2- 172.21.15.1:0/3037095566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f858c14cdc0 0x7f858c0aa5b0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.538+0000 7f859131e640 1 --2- 172.21.15.1:0/3037095566 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f858c06bc20 0x7f858c1e9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.538+0000 7f859131e640 1 -- 172.21.15.1:0/3037095566 >> 172.21.15.1:0/3037095566 conn(0x7f858c0a9700 msgr2=0x7f858c0c2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.538+0000 7f859131e640 1 -- 172.21.15.1:0/3037095566 shutdown_connections 2024-03-06T07:24:07.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.538+0000 7f859131e640 1 -- 172.21.15.1:0/3037095566 wait complete. 2024-03-06T07:24:07.552 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:07.552 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:07.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.654+0000 7f7fabf81640 1 Processor -- start 2024-03-06T07:24:07.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.654+0000 7f7fabf81640 1 -- start start 2024-03-06T07:24:07.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.655+0000 7f7fabf81640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7fa4152690 0x7f7fa4155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.655+0000 7f7fabf81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fa40c6730 0x7f7fa40c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.655+0000 7f7fabf81640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa41561e0 con 0x7f7fa40c5d10 2024-03-06T07:24:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.655+0000 7f7fabf81640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fa40c3300 con 0x7f7fa4152690 2024-03-06T07:24:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.655+0000 7f7fabf81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa40c35a0 con 0x7f7fa40c6730 2024-03-06T07:24:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.655+0000 7f7fa9d1d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7fa4152690 0x7f7fa4155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.655+0000 7f7faa51e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7fa40c5d10 0x7f7fa4156cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48836/0 (socket says 172.21.15.1:48836) 2024-03-06T07:24:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.655+0000 7f7faa51e640 1 -- 172.21.15.1:0/1728830256 learned_addr learned my addr 172.21.15.1:0/1728830256 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.655+0000 7f7fa951c640 1 --2- 172.21.15.1:0/1728830256 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fa40c6730 0x7f7fa40c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.655+0000 7f7fa9d1d640 1 -- 172.21.15.1:0/1728830256 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fa40c6730 msgr2=0x7f7fa40c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.655+0000 7f7fa9d1d640 1 --2- 172.21.15.1:0/1728830256 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fa40c6730 0x7f7fa40c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.655+0000 7f7fa9d1d640 1 -- 172.21.15.1:0/1728830256 >> v1:172.21.15.1:6789/0 conn(0x7f7fa40c5d10 legacy=0x7f7fa4156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.655+0000 7f7fa9d1d640 1 -- 172.21.15.1:0/1728830256 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa40c3880 con 0x7f7fa4152690 2024-03-06T07:24:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.655+0000 7f7fa8d1b640 1 -- 172.21.15.1:0/1728830256 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64189542 0 0) 0x7f7f94001490 con 0x7f7fa40c5d10 2024-03-06T07:24:07.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.656+0000 7f7fa9d1d640 1 --2- 172.21.15.1:0/1728830256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa4152690 0x7f7fa4155ca0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c0097a0 tx=0x7f7f9c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d19a7910a2c0f9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.656+0000 7f7fa8d1b640 1 -- 172.21.15.1:0/1728830256 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c0315a0 con 0x7f7fa4152690 2024-03-06T07:24:07.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.656+0000 7f7fa8d1b640 1 -- 172.21.15.1:0/1728830256 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9c037070 con 0x7f7fa4152690 2024-03-06T07:24:07.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.656+0000 7f7fa8d1b640 1 -- 172.21.15.1:0/1728830256 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c031960 con 0x7f7fa4152690 2024-03-06T07:24:07.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.657+0000 7f7fabf81640 1 -- 172.21.15.1:0/1728830256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa4152690 msgr2=0x7f7fa4155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.657+0000 7f7fabf81640 1 --2- 172.21.15.1:0/1728830256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa4152690 0x7f7fa4155ca0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c0097a0 tx=0x7f7f9c02b840 comp rx=0 tx=0).stop 2024-03-06T07:24:07.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.657+0000 7f7fabf81640 1 -- 172.21.15.1:0/1728830256 shutdown_connections 2024-03-06T07:24:07.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.657+0000 7f7fabf81640 1 --2- 172.21.15.1:0/1728830256 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fa40c6730 0x7f7fa40c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.657+0000 7f7fabf81640 1 --2- 172.21.15.1:0/1728830256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa4152690 0x7f7fa4155ca0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.657+0000 7f7fabf81640 1 -- 172.21.15.1:0/1728830256 >> 172.21.15.1:0/1728830256 conn(0x7f7fa406b410 msgr2=0x7f7fa406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.657+0000 7f7fabf81640 1 -- 172.21.15.1:0/1728830256 shutdown_connections 2024-03-06T07:24:07.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.657+0000 7f7fabf81640 1 -- 172.21.15.1:0/1728830256 wait complete. 2024-03-06T07:24:07.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.658+0000 7f7fabf81640 1 Processor -- start 2024-03-06T07:24:07.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.658+0000 7f7fabf81640 1 -- start start 2024-03-06T07:24:07.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fabf81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fa40c5d10 0x7f7fa4155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fabf81640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa40c6730 0x7f7fa4155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fabf81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fa4152690 0x7f7fa4153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fabf81640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa4152a70 con 0x7f7fa40c6730 2024-03-06T07:24:07.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fabf81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa41567b0 con 0x7f7fa40c5d10 2024-03-06T07:24:07.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fabf81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa4152bf0 con 0x7f7fa4152690 2024-03-06T07:24:07.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fa951c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa40c6730 0x7f7fa4155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fa951c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa40c6730 0x7f7fa4155ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53492/0 (socket says 172.21.15.1:53492) 2024-03-06T07:24:07.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fa951c640 1 -- 172.21.15.1:0/2826441787 learned_addr learned my addr 172.21.15.1:0/2826441787 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:07.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fa9d1d640 1 --2- 172.21.15.1:0/2826441787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fa40c5d10 0x7f7fa4155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7faa51e640 1 --2- 172.21.15.1:0/2826441787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fa4152690 0x7f7fa4153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fa951c640 1 -- 172.21.15.1:0/2826441787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fa4152690 msgr2=0x7f7fa4153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fa951c640 1 --2- 172.21.15.1:0/2826441787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fa4152690 0x7f7fa4153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fa951c640 1 -- 172.21.15.1:0/2826441787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fa40c5d10 msgr2=0x7f7fa4155570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fa951c640 1 --2- 172.21.15.1:0/2826441787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fa40c5d10 0x7f7fa4155570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fa951c640 1 -- 172.21.15.1:0/2826441787 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa41e1f20 con 0x7f7fa40c6730 2024-03-06T07:24:07.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7faa51e640 1 --2- 172.21.15.1:0/2826441787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fa4152690 0x7f7fa4153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:07.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fa9d1d640 1 --2- 172.21.15.1:0/2826441787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fa40c5d10 0x7f7fa4155570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:07.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fa951c640 1 --2- 172.21.15.1:0/2826441787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa40c6730 0x7f7fa4155ab0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f7f9800d620 tx=0x7f7f9800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fa27fc640 1 -- 172.21.15.1:0/2826441787 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9800b620 con 0x7f7fa40c6730 2024-03-06T07:24:07.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fa27fc640 1 -- 172.21.15.1:0/2826441787 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f98019070 con 0x7f7fa40c6730 2024-03-06T07:24:07.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.659+0000 7f7fabf81640 1 -- 172.21.15.1:0/2826441787 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa41e2200 con 0x7f7fa40c6730 2024-03-06T07:24:07.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.660+0000 7f7fa27fc640 1 -- 172.21.15.1:0/2826441787 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9800ba00 con 0x7f7fa40c6730 2024-03-06T07:24:07.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.660+0000 7f7fabf81640 1 -- 172.21.15.1:0/2826441787 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa40ad010 con 0x7f7fa40c6730 2024-03-06T07:24:07.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.660+0000 7f7fabf81640 1 -- 172.21.15.1:0/2826441787 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa41e2720 con 0x7f7fa40c6730 2024-03-06T07:24:07.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.661+0000 7f7fa27fc640 1 -- 172.21.15.1:0/2826441787 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f9800bba0 con 0x7f7fa40c6730 2024-03-06T07:24:07.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.661+0000 7f7fa27fc640 1 --2- 172.21.15.1:0/2826441787 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7f9006b4d0 0x7f7f9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.661+0000 7f7fa27fc640 1 -- 172.21.15.1:0/2826441787 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f7f9808a390 con 0x7f7fa40c6730 2024-03-06T07:24:07.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.661+0000 7f7fa9d1d640 1 --2- 172.21.15.1:0/2826441787 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7f9006b4d0 0x7f7f9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.662+0000 7f7fa9d1d640 1 --2- 172.21.15.1:0/2826441787 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7f9006b4d0 0x7f7f9006d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c036000 tx=0x7f7f9c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.665+0000 7f7fa27fc640 1 -- 172.21.15.1:0/2826441787 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f98059460 con 0x7f7fa40c6730 2024-03-06T07:24:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.845+0000 7f7fabf81640 1 -- 172.21.15.1:0/2826441787 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7fa4067fd0 con 0x7f7fa40c6730 2024-03-06T07:24:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.846+0000 7f7fa27fc640 1 -- 172.21.15.1:0/2826441787 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7f98058d60 con 0x7f7fa40c6730 2024-03-06T07:24:07.848 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:24:07.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.847+0000 7f7fabf81640 1 -- 172.21.15.1:0/2826441787 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7f9006b4d0 msgr2=0x7f7f9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.847+0000 7f7fabf81640 1 --2- 172.21.15.1:0/2826441787 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7f9006b4d0 0x7f7f9006d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c036000 tx=0x7f7f9c008030 comp rx=0 tx=0).stop 2024-03-06T07:24:07.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.847+0000 7f7fabf81640 1 -- 172.21.15.1:0/2826441787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa40c6730 msgr2=0x7f7fa4155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.848+0000 7f7fabf81640 1 --2- 172.21.15.1:0/2826441787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa40c6730 0x7f7fa4155ab0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f7f9800d620 tx=0x7f7f9800dae0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.848+0000 7f7fabf81640 1 -- 172.21.15.1:0/2826441787 shutdown_connections 2024-03-06T07:24:07.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.848+0000 7f7fabf81640 1 --2- 172.21.15.1:0/2826441787 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7f9006b4d0 0x7f7f9006d990 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.848+0000 7f7fabf81640 1 --2- 172.21.15.1:0/2826441787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fa4152690 0x7f7fa4153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.848+0000 7f7fabf81640 1 --2- 172.21.15.1:0/2826441787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa40c6730 0x7f7fa4155ab0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.848+0000 7f7fabf81640 1 --2- 172.21.15.1:0/2826441787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fa40c5d10 0x7f7fa4155570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.848+0000 7f7fabf81640 1 -- 172.21.15.1:0/2826441787 >> 172.21.15.1:0/2826441787 conn(0x7f7fa406b410 msgr2=0x7f7fa40700e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.848+0000 7f7fabf81640 1 -- 172.21.15.1:0/2826441787 shutdown_connections 2024-03-06T07:24:07.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.848+0000 7f7fabf81640 1 -- 172.21.15.1:0/2826441787 wait complete. 2024-03-06T07:24:07.861 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:24:07.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.960+0000 7f8a6269d640 1 Processor -- start 2024-03-06T07:24:07.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.960+0000 7f8a6269d640 1 -- start start 2024-03-06T07:24:07.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.961+0000 7f8a6269d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a5c0c3970 0x7f8a5c0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.961+0000 7f8a6269d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8a5c152840 0x7f8a5c1553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.961+0000 7f8a6269d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c0c5e40 con 0x7f8a5c0c60b0 2024-03-06T07:24:07.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.961+0000 7f8a6269d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a5c0c67f0 con 0x7f8a5c152840 2024-03-06T07:24:07.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.961+0000 7f8a6269d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c0c6a40 con 0x7f8a5c0c3970 2024-03-06T07:24:07.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.961+0000 7f8a60c3a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8a5c152840 0x7f8a5c1553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.961+0000 7f8a5bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a5c0c3970 0x7f8a5c0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.961+0000 7f8a5b7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8a5c0c60b0 0x7f8a5c152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:47090/0 (socket says 172.21.15.1:47090) 2024-03-06T07:24:07.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.962+0000 7f8a5b7fe640 1 -- 172.21.15.1:0/561907365 learned_addr learned my addr 172.21.15.1:0/561907365 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:07.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.962+0000 7f8a5bfff640 1 -- 172.21.15.1:0/561907365 >> v2:172.21.15.98:3300/0 conn(0x7f8a5c152840 msgr2=0x7f8a5c1553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.962+0000 7f8a5bfff640 1 --2- 172.21.15.1:0/561907365 >> v2:172.21.15.98:3300/0 conn(0x7f8a5c152840 0x7f8a5c1553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.962+0000 7f8a5bfff640 1 -- 172.21.15.1:0/561907365 >> v1:172.21.15.98:6789/0 conn(0x7f8a5c0c60b0 legacy=0x7f8a5c152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.962+0000 7f8a5bfff640 1 -- 172.21.15.1:0/561907365 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c0c6d20 con 0x7f8a5c0c3970 2024-03-06T07:24:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.963+0000 7f8a5bfff640 1 --2- 172.21.15.1:0/561907365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a5c0c3970 0x7f8a5c0c8170 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0097a0 tx=0x7f8a4c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5a3d85d59172b09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.963+0000 7f8a5affd640 1 -- 172.21.15.1:0/561907365 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c035020 con 0x7f8a5c0c3970 2024-03-06T07:24:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.963+0000 7f8a5affd640 1 -- 172.21.15.1:0/561907365 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4c031830 con 0x7f8a5c0c3970 2024-03-06T07:24:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.963+0000 7f8a5affd640 1 -- 172.21.15.1:0/561907365 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c031c00 con 0x7f8a5c0c3970 2024-03-06T07:24:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.963+0000 7f8a6269d640 1 -- 172.21.15.1:0/561907365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a5c0c3970 msgr2=0x7f8a5c0c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.963+0000 7f8a6269d640 1 --2- 172.21.15.1:0/561907365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a5c0c3970 0x7f8a5c0c8170 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0097a0 tx=0x7f8a4c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.963+0000 7f8a6269d640 1 -- 172.21.15.1:0/561907365 shutdown_connections 2024-03-06T07:24:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.963+0000 7f8a6269d640 1 --2- 172.21.15.1:0/561907365 >> v2:172.21.15.98:3300/0 conn(0x7f8a5c152840 0x7f8a5c1553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.963+0000 7f8a6269d640 1 --2- 172.21.15.1:0/561907365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a5c0c3970 0x7f8a5c0c8170 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.963+0000 7f8a6269d640 1 -- 172.21.15.1:0/561907365 >> 172.21.15.1:0/561907365 conn(0x7f8a5c0c88f0 msgr2=0x7f8a5c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.963+0000 7f8a6269d640 1 -- 172.21.15.1:0/561907365 shutdown_connections 2024-03-06T07:24:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.963+0000 7f8a6269d640 1 -- 172.21.15.1:0/561907365 wait complete. 2024-03-06T07:24:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.964+0000 7f8a6269d640 1 Processor -- start 2024-03-06T07:24:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.965+0000 7f8a6269d640 1 -- start start 2024-03-06T07:24:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.965+0000 7f8a6269d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a5c0c3970 0x7f8a5c157730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.965+0000 7f8a6269d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a5c0c60b0 0x7f8a5c06f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.965+0000 7f8a6269d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a5c152840 0x7f8a5c06fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.965+0000 7f8a6269d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c0c75c0 con 0x7f8a5c0c3970 2024-03-06T07:24:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.965+0000 7f8a6269d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c0c72e0 con 0x7f8a5c152840 2024-03-06T07:24:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.965+0000 7f8a6269d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c0c3d50 con 0x7f8a5c0c60b0 2024-03-06T07:24:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.965+0000 7f8a5bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a5c0c3970 0x7f8a5c157730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.965+0000 7f8a5bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a5c0c3970 0x7f8a5c157730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53508/0 (socket says 172.21.15.1:53508) 2024-03-06T07:24:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.966+0000 7f8a5bfff640 1 -- 172.21.15.1:0/1614302016 learned_addr learned my addr 172.21.15.1:0/1614302016 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.966+0000 7f8a60c3a640 1 --2- 172.21.15.1:0/1614302016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a5c152840 0x7f8a5c06fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.966+0000 7f8a5b7fe640 1 --2- 172.21.15.1:0/1614302016 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a5c0c60b0 0x7f8a5c06f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.966+0000 7f8a5bfff640 1 -- 172.21.15.1:0/1614302016 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a5c0c60b0 msgr2=0x7f8a5c06f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.966+0000 7f8a5bfff640 1 --2- 172.21.15.1:0/1614302016 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a5c0c60b0 0x7f8a5c06f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.966+0000 7f8a5bfff640 1 -- 172.21.15.1:0/1614302016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a5c152840 msgr2=0x7f8a5c06fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.966+0000 7f8a5bfff640 1 --2- 172.21.15.1:0/1614302016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a5c152840 0x7f8a5c06fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.966+0000 7f8a5bfff640 1 -- 172.21.15.1:0/1614302016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c1eb300 con 0x7f8a5c0c3970 2024-03-06T07:24:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.966+0000 7f8a60c3a640 1 --2- 172.21.15.1:0/1614302016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a5c152840 0x7f8a5c06fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.966+0000 7f8a5bfff640 1 --2- 172.21.15.1:0/1614302016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a5c0c3970 0x7f8a5c157730 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c02bee0 tx=0x7f8a4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.966+0000 7f8a58ff9640 1 -- 172.21.15.1:0/1614302016 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c035040 con 0x7f8a5c0c3970 2024-03-06T07:24:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.966+0000 7f8a58ff9640 1 -- 172.21.15.1:0/1614302016 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4c03b920 con 0x7f8a5c0c3970 2024-03-06T07:24:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.966+0000 7f8a6269d640 1 -- 172.21.15.1:0/1614302016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5c1eb5e0 con 0x7f8a5c0c3970 2024-03-06T07:24:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.966+0000 7f8a58ff9640 1 -- 172.21.15.1:0/1614302016 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c03bd10 con 0x7f8a5c0c3970 2024-03-06T07:24:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.966+0000 7f8a6269d640 1 -- 172.21.15.1:0/1614302016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5c06ff60 con 0x7f8a5c0c3970 2024-03-06T07:24:07.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.967+0000 7f8a6269d640 1 -- 172.21.15.1:0/1614302016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5c0686e0 con 0x7f8a5c0c3970 2024-03-06T07:24:07.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.968+0000 7f8a58ff9640 1 -- 172.21.15.1:0/1614302016 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a4c038020 con 0x7f8a5c0c3970 2024-03-06T07:24:07.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.968+0000 7f8a58ff9640 1 --2- 172.21.15.1:0/1614302016 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a3c06b500 0x7f8a3c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.968+0000 7f8a58ff9640 1 -- 172.21.15.1:0/1614302016 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8a4c0ac020 con 0x7f8a5c0c3970 2024-03-06T07:24:07.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.968+0000 7f8a5b7fe640 1 --2- 172.21.15.1:0/1614302016 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a3c06b500 0x7f8a3c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.969+0000 7f8a5b7fe640 1 --2- 172.21.15.1:0/1614302016 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a3c06b500 0x7f8a3c06d9c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c0c8fe0 tx=0x7f8a44007e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:07.972+0000 7f8a58ff9640 1 -- 172.21.15.1:0/1614302016 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a4c031800 con 0x7f8a5c0c3970 2024-03-06T07:24:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.155+0000 7f8a6269d640 1 -- 172.21.15.1:0/1614302016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8a5c071050 con 0x7f8a5c0c3970 2024-03-06T07:24:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.155+0000 7f8a58ff9640 1 -- 172.21.15.1:0/1614302016 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v104) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8a4c07aa20 con 0x7f8a5c0c3970 2024-03-06T07:24:08.157 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:08.158 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-06T07:24:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-06T07:24:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-06T07:24:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:08.158 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-06T07:24:08.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.157+0000 7f8a6269d640 1 -- 172.21.15.1:0/1614302016 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a3c06b500 msgr2=0x7f8a3c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.157+0000 7f8a6269d640 1 --2- 172.21.15.1:0/1614302016 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a3c06b500 0x7f8a3c06d9c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c0c8fe0 tx=0x7f8a44007e80 comp rx=0 tx=0).stop 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.157+0000 7f8a6269d640 1 -- 172.21.15.1:0/1614302016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a5c0c3970 msgr2=0x7f8a5c157730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.157+0000 7f8a6269d640 1 --2- 172.21.15.1:0/1614302016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a5c0c3970 0x7f8a5c157730 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c02bee0 tx=0x7f8a4c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.157+0000 7f8a6269d640 1 -- 172.21.15.1:0/1614302016 shutdown_connections 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.157+0000 7f8a6269d640 1 --2- 172.21.15.1:0/1614302016 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a3c06b500 0x7f8a3c06d9c0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.157+0000 7f8a6269d640 1 --2- 172.21.15.1:0/1614302016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a5c152840 0x7f8a5c06fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.157+0000 7f8a6269d640 1 --2- 172.21.15.1:0/1614302016 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a5c0c60b0 0x7f8a5c06f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.157+0000 7f8a6269d640 1 --2- 172.21.15.1:0/1614302016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a5c0c3970 0x7f8a5c157730 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.157+0000 7f8a6269d640 1 -- 172.21.15.1:0/1614302016 >> 172.21.15.1:0/1614302016 conn(0x7f8a5c0c88f0 msgr2=0x7f8a5c158790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.157+0000 7f8a6269d640 1 -- 172.21.15.1:0/1614302016 shutdown_connections 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.157+0000 7f8a6269d640 1 -- 172.21.15.1:0/1614302016 wait complete. 2024-03-06T07:24:08.170 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:24:08.170 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:24:08.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.273+0000 7ffae98c9640 1 Processor -- start 2024-03-06T07:24:08.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.273+0000 7ffae98c9640 1 -- start start 2024-03-06T07:24:08.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.274+0000 7ffae98c9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffae414d180 0x7ffae40c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.274+0000 7ffae98c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae40c5910 0x7ffae40c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.274+0000 7ffae98c9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffae4156580 0x7ffae41569f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.274+0000 7ffae98c9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae4153f60 con 0x7ffae414d180 2024-03-06T07:24:08.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.274+0000 7ffae98c9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae40c80a0 con 0x7ffae4156580 2024-03-06T07:24:08.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.274+0000 7ffae98c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae41531a0 con 0x7ffae40c5910 2024-03-06T07:24:08.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.274+0000 7ffae2ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffae414d180 0x7ffae40c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.274+0000 7ffae2ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffae414d180 0x7ffae40c7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53510/0 (socket says 172.21.15.1:53510) 2024-03-06T07:24:08.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.274+0000 7ffae2ffd640 1 -- 172.21.15.1:0/2540574263 learned_addr learned my addr 172.21.15.1:0/2540574263 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:08.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.274+0000 7ffae37fe640 1 --2- 172.21.15.1:0/2540574263 >> v2:172.21.15.98:3300/0 conn(0x7ffae4156580 0x7ffae41569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.274+0000 7ffae27fc640 1 --2- 172.21.15.1:0/2540574263 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae40c5910 0x7ffae40c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.274+0000 7ffae2ffd640 1 -- 172.21.15.1:0/2540574263 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae40c5910 msgr2=0x7ffae40c5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.274+0000 7ffae2ffd640 1 --2- 172.21.15.1:0/2540574263 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae40c5910 0x7ffae40c5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.274+0000 7ffae2ffd640 1 -- 172.21.15.1:0/2540574263 >> v2:172.21.15.98:3300/0 conn(0x7ffae4156580 msgr2=0x7ffae41569f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.274+0000 7ffae2ffd640 1 --2- 172.21.15.1:0/2540574263 >> v2:172.21.15.98:3300/0 conn(0x7ffae4156580 0x7ffae41569f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.274+0000 7ffae2ffd640 1 -- 172.21.15.1:0/2540574263 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae4153480 con 0x7ffae414d180 2024-03-06T07:24:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.274+0000 7ffae37fe640 1 --2- 172.21.15.1:0/2540574263 >> v2:172.21.15.98:3300/0 conn(0x7ffae4156580 0x7ffae41569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.275+0000 7ffae2ffd640 1 --2- 172.21.15.1:0/2540574263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffae414d180 0x7ffae40c7b60 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7ffae4073630 tx=0x7ffadc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41ad3e0922d3d391 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.275+0000 7ffae1ffb640 1 -- 172.21.15.1:0/2540574263 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc02c9e0 con 0x7ffae414d180 2024-03-06T07:24:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.275+0000 7ffae1ffb640 1 -- 172.21.15.1:0/2540574263 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffadc02cb80 con 0x7ffae414d180 2024-03-06T07:24:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.275+0000 7ffae1ffb640 1 -- 172.21.15.1:0/2540574263 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc0343f0 con 0x7ffae414d180 2024-03-06T07:24:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.276+0000 7ffae98c9640 1 -- 172.21.15.1:0/2540574263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffae414d180 msgr2=0x7ffae40c7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.276+0000 7ffae98c9640 1 --2- 172.21.15.1:0/2540574263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffae414d180 0x7ffae40c7b60 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7ffae4073630 tx=0x7ffadc02be40 comp rx=0 tx=0).stop 2024-03-06T07:24:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.276+0000 7ffae98c9640 1 -- 172.21.15.1:0/2540574263 shutdown_connections 2024-03-06T07:24:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.276+0000 7ffae98c9640 1 --2- 172.21.15.1:0/2540574263 >> v2:172.21.15.98:3300/0 conn(0x7ffae4156580 0x7ffae41569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.276+0000 7ffae98c9640 1 --2- 172.21.15.1:0/2540574263 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae40c5910 0x7ffae40c5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.276+0000 7ffae98c9640 1 --2- 172.21.15.1:0/2540574263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffae414d180 0x7ffae40c7b60 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.276+0000 7ffae98c9640 1 -- 172.21.15.1:0/2540574263 >> 172.21.15.1:0/2540574263 conn(0x7ffae406b410 msgr2=0x7ffae406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.276+0000 7ffae98c9640 1 -- 172.21.15.1:0/2540574263 shutdown_connections 2024-03-06T07:24:08.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.276+0000 7ffae98c9640 1 -- 172.21.15.1:0/2540574263 wait complete. 2024-03-06T07:24:08.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.277+0000 7ffae98c9640 1 Processor -- start 2024-03-06T07:24:08.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.277+0000 7ffae98c9640 1 -- start start 2024-03-06T07:24:08.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae98c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae40c5910 0x7ffae41e9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae98c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffae414d180 0x7ffae4147d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae98c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae4156580 0x7ffae41dca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae98c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae40c76e0 con 0x7ffae414d180 2024-03-06T07:24:08.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae98c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae4153a40 con 0x7ffae40c5910 2024-03-06T07:24:08.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae98c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae4071820 con 0x7ffae4156580 2024-03-06T07:24:08.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae27fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffae414d180 0x7ffae4147d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae27fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffae414d180 0x7ffae4147d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53520/0 (socket says 172.21.15.1:53520) 2024-03-06T07:24:08.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae27fc640 1 -- 172.21.15.1:0/521549784 learned_addr learned my addr 172.21.15.1:0/521549784 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:08.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae27fc640 1 -- 172.21.15.1:0/521549784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae4156580 msgr2=0x7ffae41dca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae27fc640 1 --2- 172.21.15.1:0/521549784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae4156580 0x7ffae41dca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae2ffd640 1 --2- 172.21.15.1:0/521549784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae40c5910 0x7ffae41e9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae27fc640 1 -- 172.21.15.1:0/521549784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae40c5910 msgr2=0x7ffae41e9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae27fc640 1 --2- 172.21.15.1:0/521549784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae40c5910 0x7ffae41e9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae27fc640 1 -- 172.21.15.1:0/521549784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae41dd670 con 0x7ffae414d180 2024-03-06T07:24:08.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae27fc640 1 --2- 172.21.15.1:0/521549784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffae414d180 0x7ffae4147d20 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7ffad800d830 tx=0x7ffad800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffacbfff640 1 -- 172.21.15.1:0/521549784 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad8018030 con 0x7ffae414d180 2024-03-06T07:24:08.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffacbfff640 1 -- 172.21.15.1:0/521549784 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad800be10 con 0x7ffae414d180 2024-03-06T07:24:08.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae98c9640 1 -- 172.21.15.1:0/521549784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae40c2650 con 0x7ffae414d180 2024-03-06T07:24:08.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffacbfff640 1 -- 172.21.15.1:0/521549784 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad80098b0 con 0x7ffae414d180 2024-03-06T07:24:08.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.278+0000 7ffae98c9640 1 -- 172.21.15.1:0/521549784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae414a390 con 0x7ffae414d180 2024-03-06T07:24:08.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.280+0000 7ffae98c9640 1 -- 172.21.15.1:0/521549784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae40c45c0 con 0x7ffae414d180 2024-03-06T07:24:08.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.280+0000 7ffacbfff640 1 -- 172.21.15.1:0/521549784 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffad801b070 con 0x7ffae414d180 2024-03-06T07:24:08.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.280+0000 7ffacbfff640 1 --2- 172.21.15.1:0/521549784 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffad006b4d0 0x7ffad006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.280+0000 7ffacbfff640 1 -- 172.21.15.1:0/521549784 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ffad808a1a0 con 0x7ffae414d180 2024-03-06T07:24:08.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.280+0000 7ffae2ffd640 1 --2- 172.21.15.1:0/521549784 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffad006b4d0 0x7ffad006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.281+0000 7ffae2ffd640 1 --2- 172.21.15.1:0/521549784 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffad006b4d0 0x7ffad006d990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ffadc035000 tx=0x7ffadc030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.285+0000 7ffacbfff640 1 -- 172.21.15.1:0/521549784 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffae40c45c0 con 0x7ffae414d180 2024-03-06T07:24:08.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.464+0000 7ffae98c9640 1 -- 172.21.15.1:0/521549784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ffae40680b0 con 0x7ffae414d180 2024-03-06T07:24:08.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.841+0000 7ffacbfff640 1 -- 172.21.15.1:0/521549784 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v105) v1 ==== 138+0+0 (secure 0 0 0) 0x7ffad8059270 con 0x7ffae414d180 2024-03-06T07:24:08.843 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-06T07:24:08.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.844+0000 7ffae98c9640 1 -- 172.21.15.1:0/521549784 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffad006b4d0 msgr2=0x7ffad006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.844+0000 7ffae98c9640 1 --2- 172.21.15.1:0/521549784 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffad006b4d0 0x7ffad006d990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ffadc035000 tx=0x7ffadc030000 comp rx=0 tx=0).stop 2024-03-06T07:24:08.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.844+0000 7ffae98c9640 1 -- 172.21.15.1:0/521549784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffae414d180 msgr2=0x7ffae4147d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.844+0000 7ffae98c9640 1 --2- 172.21.15.1:0/521549784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffae414d180 0x7ffae4147d20 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7ffad800d830 tx=0x7ffad800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.846+0000 7ffae98c9640 1 -- 172.21.15.1:0/521549784 shutdown_connections 2024-03-06T07:24:08.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.846+0000 7ffae98c9640 1 --2- 172.21.15.1:0/521549784 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffad006b4d0 0x7ffad006d990 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.846+0000 7ffae98c9640 1 --2- 172.21.15.1:0/521549784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae4156580 0x7ffae41dca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.846+0000 7ffae98c9640 1 --2- 172.21.15.1:0/521549784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffae414d180 0x7ffae4147d20 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.846+0000 7ffae98c9640 1 --2- 172.21.15.1:0/521549784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae40c5910 0x7ffae41e9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.846+0000 7ffae98c9640 1 -- 172.21.15.1:0/521549784 >> 172.21.15.1:0/521549784 conn(0x7ffae406b410 msgr2=0x7ffae40ad770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.847+0000 7ffae98c9640 1 -- 172.21.15.1:0/521549784 shutdown_connections 2024-03-06T07:24:08.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.847+0000 7ffae98c9640 1 -- 172.21.15.1:0/521549784 wait complete. 2024-03-06T07:24:08.862 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:24:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.970+0000 7f080e29a640 1 Processor -- start 2024-03-06T07:24:08.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.970+0000 7f080e29a640 1 -- start start 2024-03-06T07:24:08.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.971+0000 7f080e29a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0808150520 0x7f08081543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.971+0000 7f080e29a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f08080c32f0 0x7f08081537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.971+0000 7f080e29a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0808070c80 0x7f08080710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.971+0000 7f080e29a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0808150900 con 0x7f08080c32f0 2024-03-06T07:24:08.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.971+0000 7f080e29a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0808071790 con 0x7f0808070c80 2024-03-06T07:24:08.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.971+0000 7f080e29a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0808157050 con 0x7f0808150520 2024-03-06T07:24:08.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.971+0000 7f08077fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f08080c32f0 0x7f08081537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.971+0000 7f08077fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f08080c32f0 0x7f08081537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53526/0 (socket says 172.21.15.1:53526) 2024-03-06T07:24:08.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.971+0000 7f08077fe640 1 -- 172.21.15.1:0/3304443336 learned_addr learned my addr 172.21.15.1:0/3304443336 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:08.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.972+0000 7f0807fff640 1 --2- 172.21.15.1:0/3304443336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0808150520 0x7f08081543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.972+0000 7f080c837640 1 --2- 172.21.15.1:0/3304443336 >> v2:172.21.15.98:3300/0 conn(0x7f0808070c80 0x7f08080710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.972+0000 7f08077fe640 1 -- 172.21.15.1:0/3304443336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0808150520 msgr2=0x7f08081543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.972+0000 7f08077fe640 1 --2- 172.21.15.1:0/3304443336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0808150520 0x7f08081543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.972+0000 7f08077fe640 1 -- 172.21.15.1:0/3304443336 >> v2:172.21.15.98:3300/0 conn(0x7f0808070c80 msgr2=0x7f08080710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.972+0000 7f08077fe640 1 --2- 172.21.15.1:0/3304443336 >> v2:172.21.15.98:3300/0 conn(0x7f0808070c80 0x7f08080710f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.972+0000 7f08077fe640 1 -- 172.21.15.1:0/3304443336 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0808152fc0 con 0x7f08080c32f0 2024-03-06T07:24:08.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.972+0000 7f08077fe640 1 --2- 172.21.15.1:0/3304443336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08080c32f0 0x7f08081537c0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f08080c9990 tx=0x7f07fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5608d462633d2ad2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.973+0000 7f0806ffd640 1 -- 172.21.15.1:0/3304443336 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc035020 con 0x7f08080c32f0 2024-03-06T07:24:08.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.973+0000 7f0806ffd640 1 -- 172.21.15.1:0/3304443336 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07fc031870 con 0x7f08080c32f0 2024-03-06T07:24:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.973+0000 7f0806ffd640 1 -- 172.21.15.1:0/3304443336 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc031c40 con 0x7f08080c32f0 2024-03-06T07:24:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.973+0000 7f080e29a640 1 -- 172.21.15.1:0/3304443336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08080c32f0 msgr2=0x7f08081537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.973+0000 7f080e29a640 1 --2- 172.21.15.1:0/3304443336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08080c32f0 0x7f08081537c0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f08080c9990 tx=0x7f07fc02b910 comp rx=0 tx=0).stop 2024-03-06T07:24:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.973+0000 7f080e29a640 1 -- 172.21.15.1:0/3304443336 shutdown_connections 2024-03-06T07:24:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.973+0000 7f080e29a640 1 --2- 172.21.15.1:0/3304443336 >> v2:172.21.15.98:3300/0 conn(0x7f0808070c80 0x7f08080710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.973+0000 7f080e29a640 1 --2- 172.21.15.1:0/3304443336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08080c32f0 0x7f08081537c0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.973+0000 7f080e29a640 1 --2- 172.21.15.1:0/3304443336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0808150520 0x7f08081543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.973+0000 7f080e29a640 1 -- 172.21.15.1:0/3304443336 >> 172.21.15.1:0/3304443336 conn(0x7f08080c88f0 msgr2=0x7f08080c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.973+0000 7f080e29a640 1 -- 172.21.15.1:0/3304443336 shutdown_connections 2024-03-06T07:24:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.973+0000 7f080e29a640 1 -- 172.21.15.1:0/3304443336 wait complete. 2024-03-06T07:24:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.974+0000 7f080e29a640 1 Processor -- start 2024-03-06T07:24:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.974+0000 7f080e29a640 1 -- start start 2024-03-06T07:24:08.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.975+0000 7f080e29a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0808070c80 0x7f0808156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.975+0000 7f080e29a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08080c32f0 0x7f080806dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.975+0000 7f080e29a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0808150520 0x7f08080aad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.975+0000 7f080e29a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08080c7ec0 con 0x7f0808150520 2024-03-06T07:24:08.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.975+0000 7f080e29a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0808157330 con 0x7f0808070c80 2024-03-06T07:24:08.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.975+0000 7f080e29a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0808153d00 con 0x7f08080c32f0 2024-03-06T07:24:08.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.975+0000 7f080c837640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0808150520 0x7f08080aad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.975+0000 7f080c837640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0808150520 0x7f08080aad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53536/0 (socket says 172.21.15.1:53536) 2024-03-06T07:24:08.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.975+0000 7f080c837640 1 -- 172.21.15.1:0/2709806325 learned_addr learned my addr 172.21.15.1:0/2709806325 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:08.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.975+0000 7f08077fe640 1 --2- 172.21.15.1:0/2709806325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08080c32f0 0x7f080806dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.975+0000 7f0807fff640 1 --2- 172.21.15.1:0/2709806325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0808070c80 0x7f0808156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.976+0000 7f080c837640 1 -- 172.21.15.1:0/2709806325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08080c32f0 msgr2=0x7f080806dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.976+0000 7f080c837640 1 --2- 172.21.15.1:0/2709806325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08080c32f0 0x7f080806dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.976+0000 7f080c837640 1 -- 172.21.15.1:0/2709806325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0808070c80 msgr2=0x7f0808156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.976+0000 7f080c837640 1 --2- 172.21.15.1:0/2709806325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0808070c80 0x7f0808156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.976+0000 7f080c837640 1 -- 172.21.15.1:0/2709806325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08080ab2d0 con 0x7f0808150520 2024-03-06T07:24:08.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.976+0000 7f0807fff640 1 --2- 172.21.15.1:0/2709806325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0808070c80 0x7f0808156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:08.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.976+0000 7f08077fe640 1 --2- 172.21.15.1:0/2709806325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08080c32f0 0x7f080806dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:08.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.976+0000 7f080c837640 1 --2- 172.21.15.1:0/2709806325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0808150520 0x7f08080aad90 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f0800007e60 tx=0x7f080000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.976+0000 7f0804ff9640 1 -- 172.21.15.1:0/2709806325 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800014620 con 0x7f0808150520 2024-03-06T07:24:08.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.976+0000 7f0804ff9640 1 -- 172.21.15.1:0/2709806325 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0800019070 con 0x7f0808150520 2024-03-06T07:24:08.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.976+0000 7f080e29a640 1 -- 172.21.15.1:0/2709806325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080806d140 con 0x7f0808150520 2024-03-06T07:24:08.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.976+0000 7f0804ff9640 1 -- 172.21.15.1:0/2709806325 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800014a20 con 0x7f0808150520 2024-03-06T07:24:08.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.976+0000 7f080e29a640 1 -- 172.21.15.1:0/2709806325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08080c2810 con 0x7f0808150520 2024-03-06T07:24:08.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.977+0000 7f080e29a640 1 -- 172.21.15.1:0/2709806325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0808154e20 con 0x7f0808150520 2024-03-06T07:24:08.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.977+0000 7f0804ff9640 1 -- 172.21.15.1:0/2709806325 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0800014bc0 con 0x7f0808150520 2024-03-06T07:24:08.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.978+0000 7f0804ff9640 1 --2- 172.21.15.1:0/2709806325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f07f406b4d0 0x7f07f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.978+0000 7f0804ff9640 1 -- 172.21.15.1:0/2709806325 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f080008a9b0 con 0x7f0808150520 2024-03-06T07:24:08.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.978+0000 7f0807fff640 1 --2- 172.21.15.1:0/2709806325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f07f406b4d0 0x7f07f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.979+0000 7f0807fff640 1 --2- 172.21.15.1:0/2709806325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f07f406b4d0 0x7f07f406d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f07f8006fd0 tx=0x7f07f8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:08.982+0000 7f0804ff9640 1 -- 172.21.15.1:0/2709806325 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0808154e20 con 0x7f0808150520 2024-03-06T07:24:09.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.164+0000 7f080e29a640 1 -- 172.21.15.1:0/2709806325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f08080686e0 con 0x7f0808150520 2024-03-06T07:24:09.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.843+0000 7f0804ff9640 1 -- 172.21.15.1:0/2709806325 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v106) v1 ==== 170+0+0 (secure 0 0 0) 0x7f08080ab2d0 con 0x7f0808150520 2024-03-06T07:24:09.846 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-06T07:24:09.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.847+0000 7f080e29a640 1 -- 172.21.15.1:0/2709806325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f07f406b4d0 msgr2=0x7f07f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.847+0000 7f080e29a640 1 --2- 172.21.15.1:0/2709806325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f07f406b4d0 0x7f07f406d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f07f8006fd0 tx=0x7f07f8008040 comp rx=0 tx=0).stop 2024-03-06T07:24:09.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.847+0000 7f080e29a640 1 -- 172.21.15.1:0/2709806325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0808150520 msgr2=0x7f08080aad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.847+0000 7f080e29a640 1 --2- 172.21.15.1:0/2709806325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0808150520 0x7f08080aad90 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f0800007e60 tx=0x7f080000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.853+0000 7f080e29a640 1 -- 172.21.15.1:0/2709806325 shutdown_connections 2024-03-06T07:24:09.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.853+0000 7f080e29a640 1 --2- 172.21.15.1:0/2709806325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f07f406b4d0 0x7f07f406d990 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.853+0000 7f080e29a640 1 --2- 172.21.15.1:0/2709806325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0808150520 0x7f08080aad90 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.853+0000 7f080e29a640 1 --2- 172.21.15.1:0/2709806325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08080c32f0 0x7f080806dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.853+0000 7f080e29a640 1 --2- 172.21.15.1:0/2709806325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0808070c80 0x7f0808156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.853+0000 7f080e29a640 1 -- 172.21.15.1:0/2709806325 >> 172.21.15.1:0/2709806325 conn(0x7f08080c88f0 msgr2=0x7f080814c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.858+0000 7f080e29a640 1 -- 172.21.15.1:0/2709806325 shutdown_connections 2024-03-06T07:24:09.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.858+0000 7f080e29a640 1 -- 172.21.15.1:0/2709806325 wait complete. 2024-03-06T07:24:09.880 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:24:09.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.987+0000 7fb80a7e2640 1 Processor -- start 2024-03-06T07:24:09.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.988+0000 7fb80a7e2640 1 -- start start 2024-03-06T07:24:09.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.988+0000 7fb80a7e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb804072bd0 0x7fb8040c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.988+0000 7fb80a7e2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb804152b70 0x7fb8040c3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.988+0000 7fb80a7e2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb804151dd0 con 0x7fb8040c2b30 2024-03-06T07:24:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.988+0000 7fb80a7e2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8041526e0 con 0x7fb804152b70 2024-03-06T07:24:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.988+0000 7fb80a7e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb804156190 con 0x7fb804072bd0 2024-03-06T07:24:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.988+0000 7fb808d7f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb804152b70 0x7fb8040c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.988+0000 7fb808d7f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb804152b70 0x7fb8040c3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53542/0 (socket says 172.21.15.1:53542) 2024-03-06T07:24:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.988+0000 7fb808d7f640 1 -- 172.21.15.1:0/3338908962 learned_addr learned my addr 172.21.15.1:0/3338908962 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.989+0000 7fb808d7f640 1 -- 172.21.15.1:0/3338908962 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb804072bd0 msgr2=0x7fb8040c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.989+0000 7fb8037fe640 1 --2- 172.21.15.1:0/3338908962 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb804072bd0 0x7fb8040c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.989+0000 7fb808d7f640 1 --2- 172.21.15.1:0/3338908962 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb804072bd0 0x7fb8040c32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.989+0000 7fb808d7f640 1 -- 172.21.15.1:0/3338908962 >> v1:172.21.15.98:6789/0 conn(0x7fb8040c2b30 legacy=0x7fb8040724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.989+0000 7fb808d7f640 1 -- 172.21.15.1:0/3338908962 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb804152090 con 0x7fb804152b70 2024-03-06T07:24:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.989+0000 7fb8037fe640 1 --2- 172.21.15.1:0/3338908962 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb804072bd0 0x7fb8040c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.989+0000 7fb808d7f640 1 --2- 172.21.15.1:0/3338908962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb804152b70 0x7fb8040c3830 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009e50 tx=0x7fb7fc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe30cf177418ba2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.990+0000 7fb802ffd640 1 -- 172.21.15.1:0/3338908962 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc02ca70 con 0x7fb804152b70 2024-03-06T07:24:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.990+0000 7fb802ffd640 1 -- 172.21.15.1:0/3338908962 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7fc02cc10 con 0x7fb804152b70 2024-03-06T07:24:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.990+0000 7fb802ffd640 1 -- 172.21.15.1:0/3338908962 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc0343f0 con 0x7fb804152b70 2024-03-06T07:24:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.990+0000 7fb80a7e2640 1 -- 172.21.15.1:0/3338908962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb804152b70 msgr2=0x7fb8040c3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.990+0000 7fb80a7e2640 1 --2- 172.21.15.1:0/3338908962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb804152b70 0x7fb8040c3830 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009e50 tx=0x7fb7fc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.990+0000 7fb80a7e2640 1 -- 172.21.15.1:0/3338908962 shutdown_connections 2024-03-06T07:24:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.990+0000 7fb80a7e2640 1 --2- 172.21.15.1:0/3338908962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb804152b70 0x7fb8040c3830 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.990+0000 7fb80a7e2640 1 --2- 172.21.15.1:0/3338908962 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb804072bd0 0x7fb8040c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.990+0000 7fb80a7e2640 1 -- 172.21.15.1:0/3338908962 >> 172.21.15.1:0/3338908962 conn(0x7fb8040c6860 msgr2=0x7fb8040c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.990+0000 7fb80a7e2640 1 -- 172.21.15.1:0/3338908962 shutdown_connections 2024-03-06T07:24:09.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.990+0000 7fb80a7e2640 1 -- 172.21.15.1:0/3338908962 wait complete. 2024-03-06T07:24:09.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.991+0000 7fb80a7e2640 1 Processor -- start 2024-03-06T07:24:09.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.992+0000 7fb80a7e2640 1 -- start start 2024-03-06T07:24:09.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.992+0000 7fb80a7e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb804072bd0 0x7fb804145cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.992+0000 7fb80a7e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8040c2b30 0x7fb80414c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.992+0000 7fb80a7e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb804152b70 0x7fb80414c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.992+0000 7fb80a7e2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8040c3ef0 con 0x7fb804152b70 2024-03-06T07:24:09.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.992+0000 7fb80a7e2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8040c3d70 con 0x7fb8040c2b30 2024-03-06T07:24:09.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.992+0000 7fb80a7e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb804155d80 con 0x7fb804072bd0 2024-03-06T07:24:09.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.992+0000 7fb808d7f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb804152b70 0x7fb80414c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.992+0000 7fb808d7f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb804152b70 0x7fb80414c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53546/0 (socket says 172.21.15.1:53546) 2024-03-06T07:24:09.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.992+0000 7fb808d7f640 1 -- 172.21.15.1:0/2124876382 learned_addr learned my addr 172.21.15.1:0/2124876382 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:09.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.993+0000 7fb8037fe640 1 --2- 172.21.15.1:0/2124876382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8040c2b30 0x7fb80414c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.993+0000 7fb803fff640 1 --2- 172.21.15.1:0/2124876382 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb804072bd0 0x7fb804145cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.993+0000 7fb808d7f640 1 -- 172.21.15.1:0/2124876382 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb804072bd0 msgr2=0x7fb804145cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.993+0000 7fb808d7f640 1 --2- 172.21.15.1:0/2124876382 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb804072bd0 0x7fb804145cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.993+0000 7fb808d7f640 1 -- 172.21.15.1:0/2124876382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8040c2b30 msgr2=0x7fb80414c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.993+0000 7fb808d7f640 1 --2- 172.21.15.1:0/2124876382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8040c2b30 0x7fb80414c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.993+0000 7fb808d7f640 1 -- 172.21.15.1:0/2124876382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb804143da0 con 0x7fb804152b70 2024-03-06T07:24:09.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.993+0000 7fb8037fe640 1 --2- 172.21.15.1:0/2124876382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8040c2b30 0x7fb80414c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:09.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.993+0000 7fb808d7f640 1 --2- 172.21.15.1:0/2124876382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb804152b70 0x7fb80414c810 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009e50 tx=0x7fb7fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.993+0000 7fb800ff9640 1 -- 172.21.15.1:0/2124876382 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc043020 con 0x7fb804152b70 2024-03-06T07:24:09.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.993+0000 7fb800ff9640 1 -- 172.21.15.1:0/2124876382 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7fc03e450 con 0x7fb804152b70 2024-03-06T07:24:09.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.993+0000 7fb80a7e2640 1 -- 172.21.15.1:0/2124876382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8040ac890 con 0x7fb804152b70 2024-03-06T07:24:09.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.993+0000 7fb800ff9640 1 -- 172.21.15.1:0/2124876382 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc03e820 con 0x7fb804152b70 2024-03-06T07:24:09.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.993+0000 7fb80a7e2640 1 -- 172.21.15.1:0/2124876382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb804144010 con 0x7fb804152b70 2024-03-06T07:24:09.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.994+0000 7fb80a7e2640 1 -- 172.21.15.1:0/2124876382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb804067fd0 con 0x7fb804152b70 2024-03-06T07:24:09.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.995+0000 7fb800ff9640 1 -- 172.21.15.1:0/2124876382 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7fc046020 con 0x7fb804152b70 2024-03-06T07:24:09.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.995+0000 7fb800ff9640 1 --2- 172.21.15.1:0/2124876382 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7f006b4d0 0x7fb7f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.995+0000 7fb800ff9640 1 -- 172.21.15.1:0/2124876382 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fb7fc0acb90 con 0x7fb804152b70 2024-03-06T07:24:09.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.995+0000 7fb803fff640 1 --2- 172.21.15.1:0/2124876382 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7f006b4d0 0x7fb7f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:09.996+0000 7fb803fff640 1 --2- 172.21.15.1:0/2124876382 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7f006b4d0 0x7fb7f006d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8004ea0 tx=0x7fb7f8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:10.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.000+0000 7fb800ff9640 1 -- 172.21.15.1:0/2124876382 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7fc04d030 con 0x7fb804152b70 2024-03-06T07:24:10.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.218+0000 7fb80a7e2640 1 -- 172.21.15.1:0/2124876382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb804071790 con 0x7fb804152b70 2024-03-06T07:24:10.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.854+0000 7fb800ff9640 1 -- 172.21.15.1:0/2124876382 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v86) v1 ==== 174+0+0 (secure 0 0 0) 0x7fb7fc07b970 con 0x7fb804152b70 2024-03-06T07:24:10.856 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 17 and data pool 18 2024-03-06T07:24:10.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.860+0000 7fb80a7e2640 1 -- 172.21.15.1:0/2124876382 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7f006b4d0 msgr2=0x7fb7f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.860+0000 7fb80a7e2640 1 --2- 172.21.15.1:0/2124876382 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7f006b4d0 0x7fb7f006d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8004ea0 tx=0x7fb7f8009040 comp rx=0 tx=0).stop 2024-03-06T07:24:10.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.860+0000 7fb80a7e2640 1 -- 172.21.15.1:0/2124876382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb804152b70 msgr2=0x7fb80414c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.860+0000 7fb80a7e2640 1 --2- 172.21.15.1:0/2124876382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb804152b70 0x7fb80414c810 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009e50 tx=0x7fb7fc004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.861+0000 7fb80a7e2640 1 -- 172.21.15.1:0/2124876382 shutdown_connections 2024-03-06T07:24:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.861+0000 7fb80a7e2640 1 --2- 172.21.15.1:0/2124876382 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7f006b4d0 0x7fb7f006d990 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.861+0000 7fb80a7e2640 1 --2- 172.21.15.1:0/2124876382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb804152b70 0x7fb80414c810 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.861+0000 7fb80a7e2640 1 --2- 172.21.15.1:0/2124876382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8040c2b30 0x7fb80414c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.861+0000 7fb80a7e2640 1 --2- 172.21.15.1:0/2124876382 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb804072bd0 0x7fb804145cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.861+0000 7fb80a7e2640 1 -- 172.21.15.1:0/2124876382 >> 172.21.15.1:0/2124876382 conn(0x7fb8040c6860 msgr2=0x7fb8040c3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.862+0000 7fb80a7e2640 1 -- 172.21.15.1:0/2124876382 shutdown_connections 2024-03-06T07:24:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.862+0000 7fb80a7e2640 1 -- 172.21.15.1:0/2124876382 wait complete. 2024-03-06T07:24:10.895 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:10.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.996+0000 7f3dc5cfa640 1 Processor -- start 2024-03-06T07:24:10.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.996+0000 7f3dc5cfa640 1 -- start start 2024-03-06T07:24:11.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.997+0000 7f3dc5cfa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3dc0152690 0x7f3dc0155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.997+0000 7f3dc5cfa640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3dc00c6730 0x7f3dc00c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.997+0000 7f3dc5cfa640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc01561e0 con 0x7f3dc00c5d10 2024-03-06T07:24:11.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.997+0000 7f3dc5cfa640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc00c3300 con 0x7f3dc00c6730 2024-03-06T07:24:11.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.997+0000 7f3dc5cfa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc00c35a0 con 0x7f3dc0152690 2024-03-06T07:24:11.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.997+0000 7f3dbffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3dc00c5d10 0x7f3dc0156cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48846/0 (socket says 172.21.15.1:48846) 2024-03-06T07:24:11.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.997+0000 7f3dbffff640 1 -- 172.21.15.1:0/2070474750 learned_addr learned my addr 172.21.15.1:0/2070474750 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:11.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.997+0000 7f3dbe7fc640 1 -- 172.21.15.1:0/2070474750 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993712203 0 0) 0x7f3db8001490 con 0x7f3dc00c5d10 2024-03-06T07:24:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.997+0000 7f3dbe7fc640 1 -- 172.21.15.1:0/2070474750 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da8003520 con 0x7f3dc00c5d10 2024-03-06T07:24:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.997+0000 7f3dbf7fe640 1 --2- 172.21.15.1:0/2070474750 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3dc0152690 0x7f3dc0155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.997+0000 7f3dbeffd640 1 --2- 172.21.15.1:0/2070474750 >> v2:172.21.15.98:3300/0 conn(0x7f3dc00c6730 0x7f3dc00c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.997+0000 7f3dbe7fc640 1 -- 172.21.15.1:0/2070474750 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3857088461 0 0) 0x7f3da8003520 con 0x7f3dc00c5d10 2024-03-06T07:24:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.997+0000 7f3dbe7fc640 1 -- 172.21.15.1:0/2070474750 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3dc0152690 msgr2=0x7f3dc0155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.997+0000 7f3dbe7fc640 1 --2- 172.21.15.1:0/2070474750 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3dc0152690 0x7f3dc0155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.997+0000 7f3dbe7fc640 1 -- 172.21.15.1:0/2070474750 >> v2:172.21.15.98:3300/0 conn(0x7f3dc00c6730 msgr2=0x7f3dc00c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.997+0000 7f3dbe7fc640 1 --2- 172.21.15.1:0/2070474750 >> v2:172.21.15.98:3300/0 conn(0x7f3dc00c6730 0x7f3dc00c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbe7fc640 1 -- 172.21.15.1:0/2070474750 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc00c3880 con 0x7f3dc00c5d10 2024-03-06T07:24:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbe7fc640 1 -- 172.21.15.1:0/2070474750 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f3db8001bc0 con 0x7f3dc00c5d10 2024-03-06T07:24:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbf7fe640 1 --2- 172.21.15.1:0/2070474750 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3dc0152690 0x7f3dc0155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbe7fc640 1 -- 172.21.15.1:0/2070474750 >> v1:172.21.15.1:6789/0 conn(0x7f3dc00c5d10 legacy=0x7f3dc0156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbe7fc640 1 --2- 172.21.15.1:0/2070474750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da80039b0 0x7f3da8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbe7fc640 1 --2- 172.21.15.1:0/2070474750 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3dc0152690 0x7f3da8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbe7fc640 1 --2- 172.21.15.1:0/2070474750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3da8004d40 0x7f3da80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbe7fc640 1 -- 172.21.15.1:0/2070474750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc00c3300 con 0x7f3da8004d40 2024-03-06T07:24:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbe7fc640 1 -- 172.21.15.1:0/2070474750 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc00c35a0 con 0x7f3da80039b0 2024-03-06T07:24:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbe7fc640 1 -- 172.21.15.1:0/2070474750 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3da800bb20 con 0x7f3dc0152690 2024-03-06T07:24:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbeffd640 1 --2- 172.21.15.1:0/2070474750 >> v2:172.21.15.98:3300/0 conn(0x7f3dc00c6730 0x7f3dc00c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbffff640 1 --2- 172.21.15.1:0/2070474750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3da8004d40 0x7f3da80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbf7fe640 1 --2- 172.21.15.1:0/2070474750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da80039b0 0x7f3da8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbeffd640 1 --2- 172.21.15.1:0/2070474750 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3dc0152690 0x7f3da8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbffff640 1 -- 172.21.15.1:0/2070474750 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3dc0152690 msgr2=0x7f3da8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbffff640 1 --2- 172.21.15.1:0/2070474750 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3dc0152690 0x7f3da8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbffff640 1 -- 172.21.15.1:0/2070474750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da80039b0 msgr2=0x7f3da8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbffff640 1 --2- 172.21.15.1:0/2070474750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da80039b0 0x7f3da8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.998+0000 7f3dbffff640 1 -- 172.21.15.1:0/2070474750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da8003520 con 0x7f3da8004d40 2024-03-06T07:24:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.999+0000 7f3dbffff640 1 --2- 172.21.15.1:0/2070474750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3da8004d40 0x7f3da80051c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f3db800a970 tx=0x7f3da8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c4c9279fd2b76a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.999+0000 7f3dbe7fc640 1 -- 172.21.15.1:0/2070474750 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db800a5c0 con 0x7f3da8004d40 2024-03-06T07:24:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.999+0000 7f3dbe7fc640 1 -- 172.21.15.1:0/2070474750 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db8007290 con 0x7f3da8004d40 2024-03-06T07:24:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.999+0000 7f3dbe7fc640 1 -- 172.21.15.1:0/2070474750 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db8007680 con 0x7f3da8004d40 2024-03-06T07:24:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.999+0000 7f3dc5cfa640 1 -- 172.21.15.1:0/2070474750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3da8004d40 msgr2=0x7f3da80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.999+0000 7f3dc5cfa640 1 --2- 172.21.15.1:0/2070474750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3da8004d40 0x7f3da80051c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f3db800a970 tx=0x7f3da8005830 comp rx=0 tx=0).stop 2024-03-06T07:24:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.999+0000 7f3dbffff640 1 -- 172.21.15.1:0/2070474750 reap_dead start 2024-03-06T07:24:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.999+0000 7f3dc5cfa640 1 -- 172.21.15.1:0/2070474750 shutdown_connections 2024-03-06T07:24:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.999+0000 7f3dc5cfa640 1 -- 172.21.15.1:0/2070474750 >> 172.21.15.1:0/2070474750 conn(0x7f3dc006b410 msgr2=0x7f3dc006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.999+0000 7f3dc5cfa640 1 -- 172.21.15.1:0/2070474750 shutdown_connections 2024-03-06T07:24:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:10.999+0000 7f3dc5cfa640 1 -- 172.21.15.1:0/2070474750 wait complete. 2024-03-06T07:24:11.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.000+0000 7f3dc5cfa640 1 Processor -- start 2024-03-06T07:24:11.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.001+0000 7f3dc5cfa640 1 -- start start 2024-03-06T07:24:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.001+0000 7f3dc5cfa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dc00ad400 0x7f3dc0152050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.001+0000 7f3dc5cfa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc01eaf40 0x7f3dc01eea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.001+0000 7f3dc5cfa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3dc01eefb0 0x7f3dc01de880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.001+0000 7f3dc5cfa640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc0152a70 con 0x7f3dc00ad400 2024-03-06T07:24:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.001+0000 7f3dc5cfa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc01567b0 con 0x7f3dc01eaf40 2024-03-06T07:24:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.001+0000 7f3dc5cfa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc0152bf0 con 0x7f3dc01eefb0 2024-03-06T07:24:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.001+0000 7f3dbf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dc00ad400 0x7f3dc0152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.001+0000 7f3dbf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dc00ad400 0x7f3dc0152050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53552/0 (socket says 172.21.15.1:53552) 2024-03-06T07:24:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.001+0000 7f3dbf7fe640 1 -- 172.21.15.1:0/2274658129 learned_addr learned my addr 172.21.15.1:0/2274658129 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.002+0000 7f3dbeffd640 1 --2- 172.21.15.1:0/2274658129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc01eaf40 0x7f3dc01eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.002+0000 7f3dbffff640 1 --2- 172.21.15.1:0/2274658129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3dc01eefb0 0x7f3dc01de880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.002+0000 7f3dbf7fe640 1 -- 172.21.15.1:0/2274658129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3dc01eefb0 msgr2=0x7f3dc01de880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.002+0000 7f3dbf7fe640 1 --2- 172.21.15.1:0/2274658129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3dc01eefb0 0x7f3dc01de880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.002+0000 7f3dbf7fe640 1 -- 172.21.15.1:0/2274658129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc01eaf40 msgr2=0x7f3dc01eea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.002+0000 7f3dbf7fe640 1 --2- 172.21.15.1:0/2274658129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc01eaf40 0x7f3dc01eea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.002+0000 7f3dbf7fe640 1 -- 172.21.15.1:0/2274658129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc0155160 con 0x7f3dc00ad400 2024-03-06T07:24:11.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.002+0000 7f3dbeffd640 1 --2- 172.21.15.1:0/2274658129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc01eaf40 0x7f3dc01eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:11.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.002+0000 7f3dbffff640 1 --2- 172.21.15.1:0/2274658129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3dc01eefb0 0x7f3dc01de880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:11.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.002+0000 7f3dbf7fe640 1 --2- 172.21.15.1:0/2274658129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dc00ad400 0x7f3dc0152050 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f3db0011590 tx=0x7f3db0011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.002+0000 7f3da7fff640 1 -- 172.21.15.1:0/2274658129 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db0019070 con 0x7f3dc00ad400 2024-03-06T07:24:11.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.002+0000 7f3da7fff640 1 -- 172.21.15.1:0/2274658129 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db0004220 con 0x7f3dc00ad400 2024-03-06T07:24:11.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.002+0000 7f3da7fff640 1 -- 172.21.15.1:0/2274658129 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db000f040 con 0x7f3dc00ad400 2024-03-06T07:24:11.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.002+0000 7f3dc5cfa640 1 -- 172.21.15.1:0/2274658129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc0154ba0 con 0x7f3dc00ad400 2024-03-06T07:24:11.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.002+0000 7f3dc5cfa640 1 -- 172.21.15.1:0/2274658129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc0155eb0 con 0x7f3dc00ad400 2024-03-06T07:24:11.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.003+0000 7f3dc5cfa640 1 -- 172.21.15.1:0/2274658129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc0067fd0 con 0x7f3dc00ad400 2024-03-06T07:24:11.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.004+0000 7f3da7fff640 1 -- 172.21.15.1:0/2274658129 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3db0015670 con 0x7f3dc00ad400 2024-03-06T07:24:11.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.004+0000 7f3da7fff640 1 --2- 172.21.15.1:0/2274658129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3dac06b500 0x7f3dac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.004+0000 7f3da7fff640 1 -- 172.21.15.1:0/2274658129 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3db008a500 con 0x7f3dc00ad400 2024-03-06T07:24:11.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.004+0000 7f3dbeffd640 1 --2- 172.21.15.1:0/2274658129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3dac06b500 0x7f3dac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.005+0000 7f3dbeffd640 1 --2- 172.21.15.1:0/2274658129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3dac06b500 0x7f3dac06d9c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3dc00a7dc0 tx=0x7f3db4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.009+0000 7f3da7fff640 1 -- 172.21.15.1:0/2274658129 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3db0059280 con 0x7f3dc00ad400 2024-03-06T07:24:11.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.193+0000 7f3dc5cfa640 1 -- 172.21.15.1:0/2274658129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3dc00add60 con 0x7f3dc00ad400 2024-03-06T07:24:11.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.194+0000 7f3da7fff640 1 -- 172.21.15.1:0/2274658129 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16845 (secure 0 0 0) 0x7f3db0058b80 con 0x7f3dc00ad400 2024-03-06T07:24:11.196 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:11.196 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":107,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:24:10.219881+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:08.484521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.184053+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:11.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.196+0000 7f3dc5cfa640 1 -- 172.21.15.1:0/2274658129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3dac06b500 msgr2=0x7f3dac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.196+0000 7f3dc5cfa640 1 --2- 172.21.15.1:0/2274658129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3dac06b500 0x7f3dac06d9c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3dc00a7dc0 tx=0x7f3db4008040 comp rx=0 tx=0).stop 2024-03-06T07:24:11.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.196+0000 7f3dc5cfa640 1 -- 172.21.15.1:0/2274658129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dc00ad400 msgr2=0x7f3dc0152050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.196+0000 7f3dc5cfa640 1 --2- 172.21.15.1:0/2274658129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dc00ad400 0x7f3dc0152050 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f3db0011590 tx=0x7f3db0011a50 comp rx=0 tx=0).stop 2024-03-06T07:24:11.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.196+0000 7f3dc5cfa640 1 -- 172.21.15.1:0/2274658129 shutdown_connections 2024-03-06T07:24:11.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.196+0000 7f3dc5cfa640 1 --2- 172.21.15.1:0/2274658129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3dac06b500 0x7f3dac06d9c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.196+0000 7f3dc5cfa640 1 --2- 172.21.15.1:0/2274658129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3dc01eefb0 0x7f3dc01de880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.196+0000 7f3dc5cfa640 1 --2- 172.21.15.1:0/2274658129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc01eaf40 0x7f3dc01eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.196+0000 7f3dc5cfa640 1 --2- 172.21.15.1:0/2274658129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dc00ad400 0x7f3dc0152050 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.196+0000 7f3dc5cfa640 1 -- 172.21.15.1:0/2274658129 >> 172.21.15.1:0/2274658129 conn(0x7f3dc006b410 msgr2=0x7f3dc0071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.196+0000 7f3dc5cfa640 1 -- 172.21.15.1:0/2274658129 shutdown_connections 2024-03-06T07:24:11.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.196+0000 7f3dc5cfa640 1 -- 172.21.15.1:0/2274658129 wait complete. 2024-03-06T07:24:11.209 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.309+0000 7ffa543b6640 1 Processor -- start 2024-03-06T07:24:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.309+0000 7ffa543b6640 1 -- start start 2024-03-06T07:24:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.310+0000 7ffa543b6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffa4c0c4dd0 0x7ffa4c0c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.310+0000 7ffa543b6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffa4c0c79c0 0x7ffa4c152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.310+0000 7ffa543b6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c1566e0 con 0x7ffa4c0c5d80 2024-03-06T07:24:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.310+0000 7ffa543b6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa4c156490 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.310+0000 7ffa543b6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa4c157010 con 0x7ffa4c0c79c0 2024-03-06T07:24:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.310+0000 7ffa52152640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffa4c0c4dd0 0x7ffa4c0c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.310+0000 7ffa52152640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffa4c0c4dd0 0x7ffa4c0c5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53556/0 (socket says 172.21.15.1:53556) 2024-03-06T07:24:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.310+0000 7ffa52152640 1 -- 172.21.15.1:0/3777363026 learned_addr learned my addr 172.21.15.1:0/3777363026 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.310+0000 7ffa51951640 1 --2- 172.21.15.1:0/3777363026 >> v2:172.21.15.98:3300/0 conn(0x7ffa4c0c79c0 0x7ffa4c152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.310+0000 7ffa52152640 1 -- 172.21.15.1:0/3777363026 >> v2:172.21.15.98:3300/0 conn(0x7ffa4c0c79c0 msgr2=0x7ffa4c152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.310+0000 7ffa52152640 1 --2- 172.21.15.1:0/3777363026 >> v2:172.21.15.98:3300/0 conn(0x7ffa4c0c79c0 0x7ffa4c152ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.310+0000 7ffa52152640 1 -- 172.21.15.1:0/3777363026 >> v1:172.21.15.98:6789/0 conn(0x7ffa4c0c5d80 legacy=0x7ffa4c1533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.310+0000 7ffa52152640 1 -- 172.21.15.1:0/3777363026 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c1569c0 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.311+0000 7ffa52152640 1 --2- 172.21.15.1:0/3777363026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa4c0c4dd0 0x7ffa4c0c5840 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0c9890 tx=0x7ffa4802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41a4b1ce40c0141 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.311+0000 7ffa51150640 1 -- 172.21.15.1:0/3777363026 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4802e070 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.311+0000 7ffa51150640 1 -- 172.21.15.1:0/3777363026 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa48037070 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.311+0000 7ffa51150640 1 -- 172.21.15.1:0/3777363026 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa48031a00 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.312+0000 7ffa543b6640 1 -- 172.21.15.1:0/3777363026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa4c0c4dd0 msgr2=0x7ffa4c0c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.312+0000 7ffa543b6640 1 --2- 172.21.15.1:0/3777363026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa4c0c4dd0 0x7ffa4c0c5840 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0c9890 tx=0x7ffa4802b880 comp rx=0 tx=0).stop 2024-03-06T07:24:11.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.312+0000 7ffa543b6640 1 -- 172.21.15.1:0/3777363026 shutdown_connections 2024-03-06T07:24:11.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.312+0000 7ffa543b6640 1 --2- 172.21.15.1:0/3777363026 >> v2:172.21.15.98:3300/0 conn(0x7ffa4c0c79c0 0x7ffa4c152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.312+0000 7ffa543b6640 1 --2- 172.21.15.1:0/3777363026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa4c0c4dd0 0x7ffa4c0c5840 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.312+0000 7ffa543b6640 1 -- 172.21.15.1:0/3777363026 >> 172.21.15.1:0/3777363026 conn(0x7ffa4c06b410 msgr2=0x7ffa4c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.312+0000 7ffa543b6640 1 -- 172.21.15.1:0/3777363026 shutdown_connections 2024-03-06T07:24:11.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.312+0000 7ffa543b6640 1 -- 172.21.15.1:0/3777363026 wait complete. 2024-03-06T07:24:11.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.313+0000 7ffa543b6640 1 Processor -- start 2024-03-06T07:24:11.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.313+0000 7ffa543b6640 1 -- start start 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa543b6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa4c0c4dd0 0x7ffa4c0c1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa543b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa4c0c5d80 0x7ffa4c1ea5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa543b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa4c152340 0x7ffa4c1eab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa543b6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c0715f0 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa543b6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c153af0 con 0x7ffa4c152340 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa543b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c071770 con 0x7ffa4c0c5d80 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa52152640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa4c0c4dd0 0x7ffa4c0c1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa52152640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa4c0c4dd0 0x7ffa4c0c1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53564/0 (socket says 172.21.15.1:53564) 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa52152640 1 -- 172.21.15.1:0/2480967967 learned_addr learned my addr 172.21.15.1:0/2480967967 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa52953640 1 --2- 172.21.15.1:0/2480967967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa4c152340 0x7ffa4c1eab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa51951640 1 --2- 172.21.15.1:0/2480967967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa4c0c5d80 0x7ffa4c1ea5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa52152640 1 -- 172.21.15.1:0/2480967967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa4c0c5d80 msgr2=0x7ffa4c1ea5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa52152640 1 --2- 172.21.15.1:0/2480967967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa4c0c5d80 0x7ffa4c1ea5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa52152640 1 -- 172.21.15.1:0/2480967967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa4c152340 msgr2=0x7ffa4c1eab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa52152640 1 --2- 172.21.15.1:0/2480967967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa4c152340 0x7ffa4c1eab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa52152640 1 -- 172.21.15.1:0/2480967967 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c151210 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa52953640 1 --2- 172.21.15.1:0/2480967967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa4c152340 0x7ffa4c1eab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa52152640 1 --2- 172.21.15.1:0/2480967967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa4c0c4dd0 0x7ffa4c0c1e90 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7ffa4802b8c0 tx=0x7ffa480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa42ffd640 1 -- 172.21.15.1:0/2480967967 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa48032670 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa42ffd640 1 -- 172.21.15.1:0/2480967967 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa48032830 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa543b6640 1 -- 172.21.15.1:0/2480967967 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4c1514b0 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa42ffd640 1 -- 172.21.15.1:0/2480967967 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa48032c20 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.314+0000 7ffa543b6640 1 -- 172.21.15.1:0/2480967967 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4c06d090 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.315+0000 7ffa543b6640 1 -- 172.21.15.1:0/2480967967 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4c067fd0 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.316+0000 7ffa42ffd640 1 -- 172.21.15.1:0/2480967967 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa4804c060 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.316+0000 7ffa42ffd640 1 --2- 172.21.15.1:0/2480967967 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa3006b4d0 0x7ffa3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.316+0000 7ffa42ffd640 1 -- 172.21.15.1:0/2480967967 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffa480abe00 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.316+0000 7ffa51951640 1 --2- 172.21.15.1:0/2480967967 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa3006b4d0 0x7ffa3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.317+0000 7ffa51951640 1 --2- 172.21.15.1:0/2480967967 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa3006b4d0 0x7ffa3006d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0c26d0 tx=0x7ffa3c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.321+0000 7ffa42ffd640 1 -- 172.21.15.1:0/2480967967 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa48040070 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.504+0000 7ffa543b6640 1 -- 172.21.15.1:0/2480967967 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa4c06b890 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.505+0000 7ffa42ffd640 1 -- 172.21.15.1:0/2480967967 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16845 (secure 0 0 0) 0x7ffa4807ab80 con 0x7ffa4c0c4dd0 2024-03-06T07:24:11.507 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:11.507 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":107,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:24:10.219881+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:08.484521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.184053+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:11.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.507+0000 7ffa543b6640 1 -- 172.21.15.1:0/2480967967 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa3006b4d0 msgr2=0x7ffa3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.507+0000 7ffa543b6640 1 --2- 172.21.15.1:0/2480967967 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa3006b4d0 0x7ffa3006d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0c26d0 tx=0x7ffa3c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:11.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.507+0000 7ffa543b6640 1 -- 172.21.15.1:0/2480967967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa4c0c4dd0 msgr2=0x7ffa4c0c1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.507+0000 7ffa543b6640 1 --2- 172.21.15.1:0/2480967967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa4c0c4dd0 0x7ffa4c0c1e90 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7ffa4802b8c0 tx=0x7ffa480046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.507+0000 7ffa543b6640 1 -- 172.21.15.1:0/2480967967 shutdown_connections 2024-03-06T07:24:11.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.507+0000 7ffa543b6640 1 --2- 172.21.15.1:0/2480967967 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa3006b4d0 0x7ffa3006d990 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.507+0000 7ffa543b6640 1 --2- 172.21.15.1:0/2480967967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa4c152340 0x7ffa4c1eab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.507+0000 7ffa543b6640 1 --2- 172.21.15.1:0/2480967967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa4c0c5d80 0x7ffa4c1ea5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.507+0000 7ffa543b6640 1 --2- 172.21.15.1:0/2480967967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa4c0c4dd0 0x7ffa4c0c1e90 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.507+0000 7ffa543b6640 1 -- 172.21.15.1:0/2480967967 >> 172.21.15.1:0/2480967967 conn(0x7ffa4c06b410 msgr2=0x7ffa4c0a9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.507+0000 7ffa543b6640 1 -- 172.21.15.1:0/2480967967 shutdown_connections 2024-03-06T07:24:11.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.507+0000 7ffa543b6640 1 -- 172.21.15.1:0/2480967967 wait complete. 2024-03-06T07:24:11.520 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:24:11.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.622+0000 7f81e89f7640 1 Processor -- start 2024-03-06T07:24:11.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.622+0000 7f81e89f7640 1 -- start start 2024-03-06T07:24:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.623+0000 7f81e89f7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81e40c32f0 0x7f81e41543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.623+0000 7f81e89f7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e4150900 con 0x7f81e4070c80 2024-03-06T07:24:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.623+0000 7f81e89f7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e4071790 con 0x7f81e4150520 2024-03-06T07:24:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.623+0000 7f81e89f7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81e4153ed0 con 0x7f81e40c32f0 2024-03-06T07:24:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.623+0000 7f81e1d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81e40c32f0 0x7f81e41543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.623+0000 7f81e2d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f81e4070c80 0x7f81e415d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48858/0 (socket says 172.21.15.1:48858) 2024-03-06T07:24:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.623+0000 7f81e1d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81e40c32f0 0x7f81e41543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53576/0 (socket says 172.21.15.1:53576) 2024-03-06T07:24:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.623+0000 7f81e2d9d640 1 -- 172.21.15.1:0/3730803470 learned_addr learned my addr 172.21.15.1:0/3730803470 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.624+0000 7f81e1d9b640 1 -- 172.21.15.1:0/3730803470 >> v1:172.21.15.98:6789/0 conn(0x7f81e4150520 legacy=0x7f81e41537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.624+0000 7f81e1d9b640 1 -- 172.21.15.1:0/3730803470 >> v1:172.21.15.1:6789/0 conn(0x7f81e4070c80 legacy=0x7f81e415d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.624+0000 7f81e1d9b640 1 -- 172.21.15.1:0/3730803470 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e4157050 con 0x7f81e40c32f0 2024-03-06T07:24:11.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.624+0000 7f81e159a640 1 -- 172.21.15.1:0/3730803470 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666585788 0 0) 0x7f81e4150900 con 0x7f81e4070c80 2024-03-06T07:24:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.624+0000 7f81e1d9b640 1 --2- 172.21.15.1:0/3730803470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81e40c32f0 0x7f81e41543a0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f81d80097a0 tx=0x7f81d802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1140f39739995f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.625+0000 7f81e159a640 1 -- 172.21.15.1:0/3730803470 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d8034070 con 0x7f81e40c32f0 2024-03-06T07:24:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.625+0000 7f81e159a640 1 -- 172.21.15.1:0/3730803470 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d8030610 con 0x7f81e40c32f0 2024-03-06T07:24:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.625+0000 7f81e159a640 1 -- 172.21.15.1:0/3730803470 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d80309e0 con 0x7f81e40c32f0 2024-03-06T07:24:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.625+0000 7f81e89f7640 1 -- 172.21.15.1:0/3730803470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81e40c32f0 msgr2=0x7f81e41543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.625+0000 7f81e89f7640 1 --2- 172.21.15.1:0/3730803470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81e40c32f0 0x7f81e41543a0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f81d80097a0 tx=0x7f81d802b960 comp rx=0 tx=0).stop 2024-03-06T07:24:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.625+0000 7f81e89f7640 1 -- 172.21.15.1:0/3730803470 shutdown_connections 2024-03-06T07:24:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.625+0000 7f81e89f7640 1 --2- 172.21.15.1:0/3730803470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81e40c32f0 0x7f81e41543a0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.625+0000 7f81e89f7640 1 -- 172.21.15.1:0/3730803470 >> 172.21.15.1:0/3730803470 conn(0x7f81e40c88f0 msgr2=0x7f81e40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.625+0000 7f81e89f7640 1 -- 172.21.15.1:0/3730803470 shutdown_connections 2024-03-06T07:24:11.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.625+0000 7f81e89f7640 1 -- 172.21.15.1:0/3730803470 wait complete. 2024-03-06T07:24:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.626+0000 7f81e89f7640 1 Processor -- start 2024-03-06T07:24:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.627+0000 7f81e89f7640 1 -- start start 2024-03-06T07:24:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.627+0000 7f81e89f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81e4070c80 0x7f81e41466a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.627+0000 7f81e89f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81e40c32f0 0x7f81e41dcb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.627+0000 7f81e89f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e4150520 0x7f81e4150e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.627+0000 7f81e89f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e4157330 con 0x7f81e40c32f0 2024-03-06T07:24:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.627+0000 7f81e89f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e41532a0 con 0x7f81e4150520 2024-03-06T07:24:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.627+0000 7f81e89f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e40c7ec0 con 0x7f81e4070c80 2024-03-06T07:24:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.627+0000 7f81e1d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81e40c32f0 0x7f81e41dcb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.627+0000 7f81e1d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81e40c32f0 0x7f81e41dcb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53578/0 (socket says 172.21.15.1:53578) 2024-03-06T07:24:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.627+0000 7f81e1d9b640 1 -- 172.21.15.1:0/1517112870 learned_addr learned my addr 172.21.15.1:0/1517112870 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.627+0000 7f81e2d9d640 1 --2- 172.21.15.1:0/1517112870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e4150520 0x7f81e4150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.627+0000 7f81e259c640 1 --2- 172.21.15.1:0/1517112870 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81e4070c80 0x7f81e41466a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.628+0000 7f81e1d9b640 1 -- 172.21.15.1:0/1517112870 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81e4070c80 msgr2=0x7f81e41466a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.628+0000 7f81e1d9b640 1 --2- 172.21.15.1:0/1517112870 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81e4070c80 0x7f81e41466a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.628+0000 7f81e1d9b640 1 -- 172.21.15.1:0/1517112870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e4150520 msgr2=0x7f81e4150e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.628+0000 7f81e1d9b640 1 --2- 172.21.15.1:0/1517112870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e4150520 0x7f81e4150e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.628+0000 7f81e1d9b640 1 -- 172.21.15.1:0/1517112870 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e41ecae0 con 0x7f81e40c32f0 2024-03-06T07:24:11.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.628+0000 7f81e1d9b640 1 --2- 172.21.15.1:0/1517112870 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81e40c32f0 0x7f81e41dcb70 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f81d8000c00 tx=0x7f81d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.628+0000 7f81d2ffd640 1 -- 172.21.15.1:0/1517112870 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d8034040 con 0x7f81e40c32f0 2024-03-06T07:24:11.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.628+0000 7f81d2ffd640 1 -- 172.21.15.1:0/1517112870 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d8004030 con 0x7f81e40c32f0 2024-03-06T07:24:11.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.628+0000 7f81d2ffd640 1 -- 172.21.15.1:0/1517112870 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d8004420 con 0x7f81e40c32f0 2024-03-06T07:24:11.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.628+0000 7f81e89f7640 1 -- 172.21.15.1:0/1517112870 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e41ecdc0 con 0x7f81e40c32f0 2024-03-06T07:24:11.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.628+0000 7f81e89f7640 1 -- 172.21.15.1:0/1517112870 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e4149de0 con 0x7f81e40c32f0 2024-03-06T07:24:11.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.629+0000 7f81e89f7640 1 -- 172.21.15.1:0/1517112870 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e406f030 con 0x7f81e40c32f0 2024-03-06T07:24:11.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.630+0000 7f81d2ffd640 1 -- 172.21.15.1:0/1517112870 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81d8037070 con 0x7f81e40c32f0 2024-03-06T07:24:11.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.630+0000 7f81d2ffd640 1 --2- 172.21.15.1:0/1517112870 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81c406b4d0 0x7f81c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.630+0000 7f81d2ffd640 1 -- 172.21.15.1:0/1517112870 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f81d80abee0 con 0x7f81e40c32f0 2024-03-06T07:24:11.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.630+0000 7f81e259c640 1 --2- 172.21.15.1:0/1517112870 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81c406b4d0 0x7f81c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.631+0000 7f81e259c640 1 --2- 172.21.15.1:0/1517112870 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81c406b4d0 0x7f81c406d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f81e41e9c90 tx=0x7f81d4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.634+0000 7f81d2ffd640 1 -- 172.21.15.1:0/1517112870 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81e406f030 con 0x7f81e40c32f0 2024-03-06T07:24:11.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.857+0000 7f81e89f7640 1 -- 172.21.15.1:0/1517112870 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f81e406e900 con 0x7f81e40c32f0 2024-03-06T07:24:11.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.873+0000 7f81d2ffd640 1 -- 172.21.15.1:0/1517112870 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v87) v1 ==== 118+0+0 (secure 0 0 0) 0x7f81e406e900 con 0x7f81e40c32f0 2024-03-06T07:24:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.875+0000 7f81e89f7640 1 -- 172.21.15.1:0/1517112870 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81c406b4d0 msgr2=0x7f81c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.875+0000 7f81e89f7640 1 --2- 172.21.15.1:0/1517112870 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81c406b4d0 0x7f81c406d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f81e41e9c90 tx=0x7f81d4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.875+0000 7f81e89f7640 1 -- 172.21.15.1:0/1517112870 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81e40c32f0 msgr2=0x7f81e41dcb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.875+0000 7f81e89f7640 1 --2- 172.21.15.1:0/1517112870 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81e40c32f0 0x7f81e41dcb70 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f81d8000c00 tx=0x7f81d80046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.876+0000 7f81e89f7640 1 -- 172.21.15.1:0/1517112870 shutdown_connections 2024-03-06T07:24:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.876+0000 7f81e89f7640 1 --2- 172.21.15.1:0/1517112870 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81c406b4d0 0x7f81c406d990 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.876+0000 7f81e89f7640 1 --2- 172.21.15.1:0/1517112870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81e4150520 0x7f81e4150e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.876+0000 7f81e89f7640 1 --2- 172.21.15.1:0/1517112870 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81e40c32f0 0x7f81e41dcb70 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.876+0000 7f81e89f7640 1 --2- 172.21.15.1:0/1517112870 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81e4070c80 0x7f81e41466a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.876+0000 7f81e89f7640 1 -- 172.21.15.1:0/1517112870 >> 172.21.15.1:0/1517112870 conn(0x7f81e40c88f0 msgr2=0x7f81e40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.876+0000 7f81e89f7640 1 -- 172.21.15.1:0/1517112870 shutdown_connections 2024-03-06T07:24:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.876+0000 7f81e89f7640 1 -- 172.21.15.1:0/1517112870 wait complete. 2024-03-06T07:24:11.888 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:24:11.888 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:24:11.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.990+0000 7f73f6bf7640 1 Processor -- start 2024-03-06T07:24:11.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.990+0000 7f73f6bf7640 1 -- start start 2024-03-06T07:24:11.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.992+0000 7f73f6bf7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f73f0150520 0x7f73f01543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.992+0000 7f73f6bf7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f73f00c32f0 0x7f73f01537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.992+0000 7f73f6bf7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73f0070c80 0x7f73f00710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.992+0000 7f73f6bf7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73f0150900 con 0x7f73f0150520 2024-03-06T07:24:11.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.992+0000 7f73f6bf7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73f0071790 con 0x7f73f00c32f0 2024-03-06T07:24:11.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.992+0000 7f73f6bf7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f0157050 con 0x7f73f0070c80 2024-03-06T07:24:11.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.992+0000 7f73f4993640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f73f0150520 0x7f73f01543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.992+0000 7f73f4993640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f73f0150520 0x7f73f01543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53586/0 (socket says 172.21.15.1:53586) 2024-03-06T07:24:11.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.992+0000 7f73f4993640 1 -- 172.21.15.1:0/2051565178 learned_addr learned my addr 172.21.15.1:0/2051565178 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:11.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.992+0000 7f73effff640 1 --2- 172.21.15.1:0/2051565178 >> v2:172.21.15.98:3300/0 conn(0x7f73f00c32f0 0x7f73f01537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.992+0000 7f73f5194640 1 --2- 172.21.15.1:0/2051565178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73f0070c80 0x7f73f00710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.992+0000 7f73f4993640 1 -- 172.21.15.1:0/2051565178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73f0070c80 msgr2=0x7f73f00710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.992+0000 7f73f4993640 1 --2- 172.21.15.1:0/2051565178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73f0070c80 0x7f73f00710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.992+0000 7f73f4993640 1 -- 172.21.15.1:0/2051565178 >> v2:172.21.15.98:3300/0 conn(0x7f73f00c32f0 msgr2=0x7f73f01537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.992+0000 7f73f4993640 1 --2- 172.21.15.1:0/2051565178 >> v2:172.21.15.98:3300/0 conn(0x7f73f00c32f0 0x7f73f01537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.992+0000 7f73f4993640 1 -- 172.21.15.1:0/2051565178 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f0152fc0 con 0x7f73f0150520 2024-03-06T07:24:11.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.992+0000 7f73f5194640 1 --2- 172.21.15.1:0/2051565178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73f0070c80 0x7f73f00710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:11.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.993+0000 7f73f4993640 1 --2- 172.21.15.1:0/2051565178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f0150520 0x7f73f01543a0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f73f00732a0 tx=0x7f73e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13747a333a3402ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.993+0000 7f73ef7fe640 1 -- 172.21.15.1:0/2051565178 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e4035020 con 0x7f73f0150520 2024-03-06T07:24:11.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.993+0000 7f73ef7fe640 1 -- 172.21.15.1:0/2051565178 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e4031870 con 0x7f73f0150520 2024-03-06T07:24:11.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.993+0000 7f73ef7fe640 1 -- 172.21.15.1:0/2051565178 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e4031c40 con 0x7f73f0150520 2024-03-06T07:24:11.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.994+0000 7f73f6bf7640 1 -- 172.21.15.1:0/2051565178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f0150520 msgr2=0x7f73f01543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.994+0000 7f73f6bf7640 1 --2- 172.21.15.1:0/2051565178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f0150520 0x7f73f01543a0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f73f00732a0 tx=0x7f73e402b910 comp rx=0 tx=0).stop 2024-03-06T07:24:11.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.994+0000 7f73f6bf7640 1 -- 172.21.15.1:0/2051565178 shutdown_connections 2024-03-06T07:24:11.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.994+0000 7f73f6bf7640 1 --2- 172.21.15.1:0/2051565178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73f0070c80 0x7f73f00710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.994+0000 7f73f6bf7640 1 --2- 172.21.15.1:0/2051565178 >> v2:172.21.15.98:3300/0 conn(0x7f73f00c32f0 0x7f73f01537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.994+0000 7f73f6bf7640 1 --2- 172.21.15.1:0/2051565178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f0150520 0x7f73f01543a0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.994+0000 7f73f6bf7640 1 -- 172.21.15.1:0/2051565178 >> 172.21.15.1:0/2051565178 conn(0x7f73f00c88f0 msgr2=0x7f73f00c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.994+0000 7f73f6bf7640 1 -- 172.21.15.1:0/2051565178 shutdown_connections 2024-03-06T07:24:11.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.994+0000 7f73f6bf7640 1 -- 172.21.15.1:0/2051565178 wait complete. 2024-03-06T07:24:11.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.995+0000 7f73f6bf7640 1 Processor -- start 2024-03-06T07:24:11.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.995+0000 7f73f6bf7640 1 -- start start 2024-03-06T07:24:12.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f6bf7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73f0070c80 0x7f73f01ece10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f6bf7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73f00c32f0 0x7f73f00a9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f6bf7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f0150520 0x7f73f00c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f6bf7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f00c7ec0 con 0x7f73f0150520 2024-03-06T07:24:12.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f6bf7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f0157330 con 0x7f73f00c32f0 2024-03-06T07:24:12.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f6bf7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f0153d00 con 0x7f73f0070c80 2024-03-06T07:24:12.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f5194640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f0150520 0x7f73f00c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f5194640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f0150520 0x7f73f00c49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53588/0 (socket says 172.21.15.1:53588) 2024-03-06T07:24:12.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f5194640 1 -- 172.21.15.1:0/3277817670 learned_addr learned my addr 172.21.15.1:0/3277817670 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:12.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73effff640 1 --2- 172.21.15.1:0/3277817670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73f00c32f0 0x7f73f00a9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f4993640 1 --2- 172.21.15.1:0/3277817670 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73f0070c80 0x7f73f01ece10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f5194640 1 -- 172.21.15.1:0/3277817670 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73f0070c80 msgr2=0x7f73f01ece10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f5194640 1 --2- 172.21.15.1:0/3277817670 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73f0070c80 0x7f73f01ece10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f5194640 1 -- 172.21.15.1:0/3277817670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73f00c32f0 msgr2=0x7f73f00a9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f5194640 1 --2- 172.21.15.1:0/3277817670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73f00c32f0 0x7f73f00a9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f5194640 1 -- 172.21.15.1:0/3277817670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f01e8e90 con 0x7f73f0150520 2024-03-06T07:24:12.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f4993640 1 --2- 172.21.15.1:0/3277817670 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73f0070c80 0x7f73f01ece10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:12.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73effff640 1 --2- 172.21.15.1:0/3277817670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73f00c32f0 0x7f73f00a9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:12.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f5194640 1 --2- 172.21.15.1:0/3277817670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f0150520 0x7f73f00c49b0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f73e800cf20 tx=0x7f73e800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73ed7fa640 1 -- 172.21.15.1:0/3277817670 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e801a070 con 0x7f73f0150520 2024-03-06T07:24:12.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73ed7fa640 1 -- 172.21.15.1:0/3277817670 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e800d840 con 0x7f73f0150520 2024-03-06T07:24:12.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73ed7fa640 1 -- 172.21.15.1:0/3277817670 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e800dc30 con 0x7f73f0150520 2024-03-06T07:24:12.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f6bf7640 1 -- 172.21.15.1:0/3277817670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f00a8690 con 0x7f73f0150520 2024-03-06T07:24:12.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.997+0000 7f73f6bf7640 1 -- 172.21.15.1:0/3277817670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f01e9110 con 0x7f73f0150520 2024-03-06T07:24:12.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.998+0000 7f73f6bf7640 1 -- 172.21.15.1:0/3277817670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f01e9310 con 0x7f73f0150520 2024-03-06T07:24:12.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.999+0000 7f73ed7fa640 1 -- 172.21.15.1:0/3277817670 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73e801d070 con 0x7f73f0150520 2024-03-06T07:24:12.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.999+0000 7f73ed7fa640 1 --2- 172.21.15.1:0/3277817670 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73d406b4d0 0x7f73d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:11.999+0000 7f73ed7fa640 1 -- 172.21.15.1:0/3277817670 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f73e808ab70 con 0x7f73f0150520 2024-03-06T07:24:12.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.000+0000 7f73f4993640 1 --2- 172.21.15.1:0/3277817670 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73d406b4d0 0x7f73d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.000+0000 7f73f4993640 1 --2- 172.21.15.1:0/3277817670 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73d406b4d0 0x7f73d406d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f73e4036000 tx=0x7f73e4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.004+0000 7f73ed7fa640 1 -- 172.21.15.1:0/3277817670 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73f01e9310 con 0x7f73f0150520 2024-03-06T07:24:12.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.223+0000 7f73f6bf7640 1 -- 172.21.15.1:0/3277817670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f73f014c6f0 con 0x7f73f0150520 2024-03-06T07:24:12.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.878+0000 7f73ed7fa640 1 -- 172.21.15.1:0/3277817670 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v88) v1 ==== 122+0+0 (secure 0 0 0) 0x7f73f01e8e90 con 0x7f73f0150520 2024-03-06T07:24:12.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.880+0000 7f73f6bf7640 1 -- 172.21.15.1:0/3277817670 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73d406b4d0 msgr2=0x7f73d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.880+0000 7f73f6bf7640 1 --2- 172.21.15.1:0/3277817670 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73d406b4d0 0x7f73d406d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f73e4036000 tx=0x7f73e4008030 comp rx=0 tx=0).stop 2024-03-06T07:24:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.880+0000 7f73f6bf7640 1 -- 172.21.15.1:0/3277817670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f0150520 msgr2=0x7f73f00c49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.880+0000 7f73f6bf7640 1 --2- 172.21.15.1:0/3277817670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f0150520 0x7f73f00c49b0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f73e800cf20 tx=0x7f73e800a710 comp rx=0 tx=0).stop 2024-03-06T07:24:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.881+0000 7f73f6bf7640 1 -- 172.21.15.1:0/3277817670 shutdown_connections 2024-03-06T07:24:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.881+0000 7f73f6bf7640 1 --2- 172.21.15.1:0/3277817670 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73d406b4d0 0x7f73d406d990 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.881+0000 7f73f6bf7640 1 --2- 172.21.15.1:0/3277817670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f0150520 0x7f73f00c49b0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.881+0000 7f73f6bf7640 1 --2- 172.21.15.1:0/3277817670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73f00c32f0 0x7f73f00a9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.881+0000 7f73f6bf7640 1 --2- 172.21.15.1:0/3277817670 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73f0070c80 0x7f73f01ece10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.881+0000 7f73f6bf7640 1 -- 172.21.15.1:0/3277817670 >> 172.21.15.1:0/3277817670 conn(0x7f73f00c88f0 msgr2=0x7f73f0156870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.881+0000 7f73f6bf7640 1 -- 172.21.15.1:0/3277817670 shutdown_connections 2024-03-06T07:24:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.881+0000 7f73f6bf7640 1 -- 172.21.15.1:0/3277817670 wait complete. 2024-03-06T07:24:12.893 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:24:12.894 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:12.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.993+0000 7f988fc55640 1 Processor -- start 2024-03-06T07:24:12.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.993+0000 7f988fc55640 1 -- start start 2024-03-06T07:24:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.994+0000 7f988fc55640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98880c32f0 0x7f98881543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.994+0000 7f988fc55640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9888070c80 0x7f98880710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.994+0000 7f988fc55640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9888150900 con 0x7f9888150520 2024-03-06T07:24:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.994+0000 7f988fc55640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9888153ed0 con 0x7f9888070c80 2024-03-06T07:24:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.994+0000 7f988fc55640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9888071790 con 0x7f98880c32f0 2024-03-06T07:24:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.994+0000 7f988d9f1640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9888150520 0x7f98881537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48862/0 (socket says 172.21.15.1:48862) 2024-03-06T07:24:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.994+0000 7f988d9f1640 1 -- 172.21.15.1:0/293637011 learned_addr learned my addr 172.21.15.1:0/293637011 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:12.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988e1f2640 1 --2- 172.21.15.1:0/293637011 >> v2:172.21.15.98:3300/0 conn(0x7f9888070c80 0x7f98880710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988d1f0640 1 --2- 172.21.15.1:0/293637011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98880c32f0 0x7f98881543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988c9ef640 1 -- 172.21.15.1:0/293637011 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720549643 0 0) 0x7f9888150900 con 0x7f9888150520 2024-03-06T07:24:13.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988c9ef640 1 -- 172.21.15.1:0/293637011 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9874003520 con 0x7f9888150520 2024-03-06T07:24:13.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988c9ef640 1 -- 172.21.15.1:0/293637011 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2634282884 0 0) 0x7f9874003520 con 0x7f9888150520 2024-03-06T07:24:13.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988c9ef640 1 -- 172.21.15.1:0/293637011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98880c32f0 msgr2=0x7f98881543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988c9ef640 1 --2- 172.21.15.1:0/293637011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98880c32f0 0x7f98881543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988c9ef640 1 -- 172.21.15.1:0/293637011 >> v2:172.21.15.98:3300/0 conn(0x7f9888070c80 msgr2=0x7f98880710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988c9ef640 1 --2- 172.21.15.1:0/293637011 >> v2:172.21.15.98:3300/0 conn(0x7f9888070c80 0x7f98880710f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988c9ef640 1 -- 172.21.15.1:0/293637011 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9888157050 con 0x7f9888150520 2024-03-06T07:24:13.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988c9ef640 1 -- 172.21.15.1:0/293637011 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f9884001670 con 0x7f9888150520 2024-03-06T07:24:13.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988c9ef640 1 -- 172.21.15.1:0/293637011 >> v1:172.21.15.1:6789/0 conn(0x7f9888150520 legacy=0x7f98881537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988c9ef640 1 --2- 172.21.15.1:0/293637011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98740039b0 0x7f9874003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988c9ef640 1 --2- 172.21.15.1:0/293637011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98880c32f0 0x7f9874004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988c9ef640 1 --2- 172.21.15.1:0/293637011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9874004d40 0x7f98740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988c9ef640 1 -- 172.21.15.1:0/293637011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9888153ed0 con 0x7f9874004d40 2024-03-06T07:24:13.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988c9ef640 1 -- 172.21.15.1:0/293637011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9888071790 con 0x7f98740039b0 2024-03-06T07:24:13.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988c9ef640 1 -- 172.21.15.1:0/293637011 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987400b9c0 con 0x7f98880c32f0 2024-03-06T07:24:13.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988e1f2640 1 --2- 172.21.15.1:0/293637011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9874004d40 0x7f98740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988d9f1640 1 --2- 172.21.15.1:0/293637011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98740039b0 0x7f9874003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.995+0000 7f988d1f0640 1 --2- 172.21.15.1:0/293637011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98880c32f0 0x7f9874004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.996+0000 7f988e1f2640 1 -- 172.21.15.1:0/293637011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98880c32f0 msgr2=0x7f9874004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.996+0000 7f988e1f2640 1 --2- 172.21.15.1:0/293637011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98880c32f0 0x7f9874004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.996+0000 7f988e1f2640 1 -- 172.21.15.1:0/293637011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98740039b0 msgr2=0x7f9874003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.996+0000 7f988e1f2640 1 --2- 172.21.15.1:0/293637011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98740039b0 0x7f9874003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.996+0000 7f988e1f2640 1 -- 172.21.15.1:0/293637011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9874003520 con 0x7f9874004d40 2024-03-06T07:24:13.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.996+0000 7f988e1f2640 1 --2- 172.21.15.1:0/293637011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9874004d40 0x7f98740051c0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f9880009fc0 tx=0x7f9874005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b019a52a0a9a89 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.996+0000 7f988c9ef640 1 -- 172.21.15.1:0/293637011 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9880039070 con 0x7f9874004d40 2024-03-06T07:24:13.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.996+0000 7f988c9ef640 1 -- 172.21.15.1:0/293637011 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988002cc60 con 0x7f9874004d40 2024-03-06T07:24:13.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.996+0000 7f988c9ef640 1 -- 172.21.15.1:0/293637011 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9880009c10 con 0x7f9874004d40 2024-03-06T07:24:13.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.997+0000 7f988fc55640 1 -- 172.21.15.1:0/293637011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9874004d40 msgr2=0x7f98740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.997+0000 7f988fc55640 1 --2- 172.21.15.1:0/293637011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9874004d40 0x7f98740051c0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f9880009fc0 tx=0x7f9874005830 comp rx=0 tx=0).stop 2024-03-06T07:24:13.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.997+0000 7f988e1f2640 1 -- 172.21.15.1:0/293637011 reap_dead start 2024-03-06T07:24:13.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.997+0000 7f988fc55640 1 -- 172.21.15.1:0/293637011 shutdown_connections 2024-03-06T07:24:13.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.997+0000 7f988fc55640 1 -- 172.21.15.1:0/293637011 >> 172.21.15.1:0/293637011 conn(0x7f98880c88f0 msgr2=0x7f98880c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.997+0000 7f988fc55640 1 -- 172.21.15.1:0/293637011 shutdown_connections 2024-03-06T07:24:13.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.997+0000 7f988fc55640 1 -- 172.21.15.1:0/293637011 wait complete. 2024-03-06T07:24:13.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.998+0000 7f988fc55640 1 Processor -- start 2024-03-06T07:24:13.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.998+0000 7f988fc55640 1 -- start start 2024-03-06T07:24:13.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.999+0000 7f988fc55640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9874004d40 0x7f9888146290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.999+0000 7f988fc55640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98881ee810 0x7f98880c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.999+0000 7f988fc55640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98881ea900 0x7f988814d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.999+0000 7f988fc55640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9888157330 con 0x7f98881ee810 2024-03-06T07:24:13.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.999+0000 7f988fc55640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98881532a0 con 0x7f98881ea900 2024-03-06T07:24:13.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.999+0000 7f988fc55640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98880c7ec0 con 0x7f9874004d40 2024-03-06T07:24:13.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.999+0000 7f988d1f0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98881ee810 0x7f98880c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.999+0000 7f988d1f0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98881ee810 0x7f98880c1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53608/0 (socket says 172.21.15.1:53608) 2024-03-06T07:24:13.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.999+0000 7f988d1f0640 1 -- 172.21.15.1:0/3279239552 learned_addr learned my addr 172.21.15.1:0/3279239552 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:13.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.999+0000 7f988e1f2640 1 --2- 172.21.15.1:0/3279239552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98881ea900 0x7f988814d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.999+0000 7f988d9f1640 1 --2- 172.21.15.1:0/3279239552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9874004d40 0x7f9888146290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.999+0000 7f988d1f0640 1 -- 172.21.15.1:0/3279239552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9874004d40 msgr2=0x7f9888146290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.999+0000 7f988d1f0640 1 --2- 172.21.15.1:0/3279239552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9874004d40 0x7f9888146290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.999+0000 7f988d1f0640 1 -- 172.21.15.1:0/3279239552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98881ea900 msgr2=0x7f988814d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.999+0000 7f988d1f0640 1 --2- 172.21.15.1:0/3279239552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98881ea900 0x7f988814d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:12.999+0000 7f988d1f0640 1 -- 172.21.15.1:0/3279239552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988814a4a0 con 0x7f98881ee810 2024-03-06T07:24:13.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.000+0000 7f988d1f0640 1 --2- 172.21.15.1:0/3279239552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98881ee810 0x7f98880c1f90 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f98780119a0 tx=0x7f9878011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.000+0000 7f987e7fc640 1 -- 172.21.15.1:0/3279239552 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987800fa30 con 0x7f98881ee810 2024-03-06T07:24:13.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.000+0000 7f987e7fc640 1 -- 172.21.15.1:0/3279239552 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987800fbd0 con 0x7f98881ee810 2024-03-06T07:24:13.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.000+0000 7f987e7fc640 1 -- 172.21.15.1:0/3279239552 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9878014440 con 0x7f98881ee810 2024-03-06T07:24:13.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.000+0000 7f988fc55640 1 -- 172.21.15.1:0/3279239552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98881ee010 con 0x7f98881ee810 2024-03-06T07:24:13.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.000+0000 7f988fc55640 1 -- 172.21.15.1:0/3279239552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98880c24d0 con 0x7f98881ee810 2024-03-06T07:24:13.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.001+0000 7f988fc55640 1 -- 172.21.15.1:0/3279239552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98880686e0 con 0x7f98881ee810 2024-03-06T07:24:13.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.001+0000 7f987e7fc640 1 -- 172.21.15.1:0/3279239552 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9878012040 con 0x7f98881ee810 2024-03-06T07:24:13.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.002+0000 7f987e7fc640 1 --2- 172.21.15.1:0/3279239552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f986c06b4d0 0x7f986c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.002+0000 7f987e7fc640 1 -- 172.21.15.1:0/3279239552 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f987808a550 con 0x7f98881ee810 2024-03-06T07:24:13.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.002+0000 7f988d9f1640 1 --2- 172.21.15.1:0/3279239552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f986c06b4d0 0x7f986c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.003+0000 7f988d9f1640 1 --2- 172.21.15.1:0/3279239552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f986c06b4d0 0x7f986c06d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f98880adbe0 tx=0x7f98840078f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.006+0000 7f987e7fc640 1 -- 172.21.15.1:0/3279239552 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9878019030 con 0x7f98881ee810 2024-03-06T07:24:13.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.225+0000 7f988fc55640 1 -- 172.21.15.1:0/3279239552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98880c7b10 con 0x7f98881ee810 2024-03-06T07:24:13.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.226+0000 7f987e7fc640 1 -- 172.21.15.1:0/3279239552 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f98780592d0 con 0x7f98881ee810 2024-03-06T07:24:13.229 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:13.229 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":88,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6122,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2632566151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2632566151},{"type":"v1","addr":"172.21.15.98:6837","nonce":2632566151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":6670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2244230066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2244230066},{"type":"v1","addr":"172.21.15.1:6835","nonce":2244230066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":6682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1741783790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1741783790},{"type":"v1","addr":"172.21.15.98:6836","nonce":1741783790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:10.220198+0000","modified":"2024-03-06T07:24:12.224326+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6833/2433923518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2433923518},{"type":"v1","addr":"172.21.15.1:6833","nonce":2433923518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":8}]} 2024-03-06T07:24:13.229 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 88 2024-03-06T07:24:13.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.228+0000 7f988fc55640 1 -- 172.21.15.1:0/3279239552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f986c06b4d0 msgr2=0x7f986c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.228+0000 7f988fc55640 1 --2- 172.21.15.1:0/3279239552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f986c06b4d0 0x7f986c06d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f98880adbe0 tx=0x7f98840078f0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.228+0000 7f988fc55640 1 -- 172.21.15.1:0/3279239552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98881ee810 msgr2=0x7f98880c1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.228+0000 7f988fc55640 1 --2- 172.21.15.1:0/3279239552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98881ee810 0x7f98880c1f90 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f98780119a0 tx=0x7f9878011da0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.228+0000 7f988fc55640 1 -- 172.21.15.1:0/3279239552 shutdown_connections 2024-03-06T07:24:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.228+0000 7f988fc55640 1 --2- 172.21.15.1:0/3279239552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f986c06b4d0 0x7f986c06d990 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.228+0000 7f988fc55640 1 --2- 172.21.15.1:0/3279239552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98881ea900 0x7f988814d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.228+0000 7f988fc55640 1 --2- 172.21.15.1:0/3279239552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98881ee810 0x7f98880c1f90 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.228+0000 7f988fc55640 1 --2- 172.21.15.1:0/3279239552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9874004d40 0x7f9888146290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.228+0000 7f988fc55640 1 -- 172.21.15.1:0/3279239552 >> 172.21.15.1:0/3279239552 conn(0x7f98880c88f0 msgr2=0x7f988814f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.228+0000 7f988fc55640 1 -- 172.21.15.1:0/3279239552 shutdown_connections 2024-03-06T07:24:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.228+0000 7f988fc55640 1 -- 172.21.15.1:0/3279239552 wait complete. 2024-03-06T07:24:13.242 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.342+0000 7fa958c4e640 1 Processor -- start 2024-03-06T07:24:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.342+0000 7fa958c4e640 1 -- start start 2024-03-06T07:24:13.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.343+0000 7fa958c4e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa9540c32f0 0x7fa9541543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.343+0000 7fa958c4e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa954150900 con 0x7fa954150520 2024-03-06T07:24:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.343+0000 7fa958c4e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa954071790 con 0x7fa954070c80 2024-03-06T07:24:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.343+0000 7fa958c4e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa954153ed0 con 0x7fa9540c32f0 2024-03-06T07:24:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.343+0000 7fa94bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa9540c32f0 0x7fa9541543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.343+0000 7fa95259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa954150520 0x7fa9541537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48868/0 (socket says 172.21.15.1:48868) 2024-03-06T07:24:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.343+0000 7fa95259c640 1 -- 172.21.15.1:0/3237060394 learned_addr learned my addr 172.21.15.1:0/3237060394 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.343+0000 7fa94bfff640 1 -- 172.21.15.1:0/3237060394 >> v1:172.21.15.98:6789/0 conn(0x7fa954070c80 legacy=0x7fa95415d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.343+0000 7fa94bfff640 1 -- 172.21.15.1:0/3237060394 >> v1:172.21.15.1:6789/0 conn(0x7fa954150520 legacy=0x7fa9541537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.343+0000 7fa94bfff640 1 -- 172.21.15.1:0/3237060394 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa954157050 con 0x7fa9540c32f0 2024-03-06T07:24:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.343+0000 7fa951d9b640 1 -- 172.21.15.1:0/3237060394 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422669248 0 0) 0x7fa954150900 con 0x7fa954150520 2024-03-06T07:24:13.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.344+0000 7fa94bfff640 1 --2- 172.21.15.1:0/3237060394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9540c32f0 0x7fa9541543a0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fa93c0097a0 tx=0x7fa93c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a5f99658b23b1bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.344+0000 7fa951d9b640 1 -- 172.21.15.1:0/3237060394 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c034070 con 0x7fa9540c32f0 2024-03-06T07:24:13.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.344+0000 7fa951d9b640 1 -- 172.21.15.1:0/3237060394 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93c030610 con 0x7fa9540c32f0 2024-03-06T07:24:13.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.344+0000 7fa951d9b640 1 -- 172.21.15.1:0/3237060394 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c0309e0 con 0x7fa9540c32f0 2024-03-06T07:24:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.345+0000 7fa958c4e640 1 -- 172.21.15.1:0/3237060394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9540c32f0 msgr2=0x7fa9541543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.345+0000 7fa958c4e640 1 --2- 172.21.15.1:0/3237060394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9540c32f0 0x7fa9541543a0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fa93c0097a0 tx=0x7fa93c02b960 comp rx=0 tx=0).stop 2024-03-06T07:24:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.345+0000 7fa958c4e640 1 -- 172.21.15.1:0/3237060394 shutdown_connections 2024-03-06T07:24:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.345+0000 7fa958c4e640 1 --2- 172.21.15.1:0/3237060394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9540c32f0 0x7fa9541543a0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.345+0000 7fa958c4e640 1 -- 172.21.15.1:0/3237060394 >> 172.21.15.1:0/3237060394 conn(0x7fa9540c88f0 msgr2=0x7fa9540c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.345+0000 7fa958c4e640 1 -- 172.21.15.1:0/3237060394 shutdown_connections 2024-03-06T07:24:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.345+0000 7fa958c4e640 1 -- 172.21.15.1:0/3237060394 wait complete. 2024-03-06T07:24:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.346+0000 7fa958c4e640 1 Processor -- start 2024-03-06T07:24:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.346+0000 7fa958c4e640 1 -- start start 2024-03-06T07:24:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa958c4e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa954070c80 0x7fa95406dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa958c4e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9540c32f0 0x7fa9540c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa958c4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa954150520 0x7fa9541460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa958c4e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa954157330 con 0x7fa9540c32f0 2024-03-06T07:24:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa958c4e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9541532a0 con 0x7fa954150520 2024-03-06T07:24:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa958c4e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9540c7ec0 con 0x7fa954070c80 2024-03-06T07:24:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa94bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9540c32f0 0x7fa9540c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa94bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9540c32f0 0x7fa9540c1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53622/0 (socket says 172.21.15.1:53622) 2024-03-06T07:24:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa94bfff640 1 -- 172.21.15.1:0/1516843384 learned_addr learned my addr 172.21.15.1:0/1516843384 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa95259c640 1 --2- 172.21.15.1:0/1516843384 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa954070c80 0x7fa95406dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa952d9d640 1 --2- 172.21.15.1:0/1516843384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa954150520 0x7fa9541460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa94bfff640 1 -- 172.21.15.1:0/1516843384 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa954070c80 msgr2=0x7fa95406dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa94bfff640 1 --2- 172.21.15.1:0/1516843384 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa954070c80 0x7fa95406dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa94bfff640 1 -- 172.21.15.1:0/1516843384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa954150520 msgr2=0x7fa9541460e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa94bfff640 1 --2- 172.21.15.1:0/1516843384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa954150520 0x7fa9541460e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa94bfff640 1 -- 172.21.15.1:0/1516843384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9540c43e0 con 0x7fa9540c32f0 2024-03-06T07:24:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa95259c640 1 --2- 172.21.15.1:0/1516843384 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa954070c80 0x7fa95406dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa952d9d640 1 --2- 172.21.15.1:0/1516843384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa954150520 0x7fa9541460e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.347+0000 7fa94bfff640 1 --2- 172.21.15.1:0/1516843384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9540c32f0 0x7fa9540c1f90 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fa93c02bf70 tx=0x7fa93c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.348+0000 7fa94affd640 1 -- 172.21.15.1:0/1516843384 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c034040 con 0x7fa9540c32f0 2024-03-06T07:24:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.348+0000 7fa94affd640 1 -- 172.21.15.1:0/1516843384 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93c039a10 con 0x7fa9540c32f0 2024-03-06T07:24:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.348+0000 7fa958c4e640 1 -- 172.21.15.1:0/1516843384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa954151310 con 0x7fa9540c32f0 2024-03-06T07:24:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.348+0000 7fa94affd640 1 -- 172.21.15.1:0/1516843384 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c039de0 con 0x7fa9540c32f0 2024-03-06T07:24:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.348+0000 7fa958c4e640 1 -- 172.21.15.1:0/1516843384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9541eb590 con 0x7fa9540c32f0 2024-03-06T07:24:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.349+0000 7fa958c4e640 1 -- 172.21.15.1:0/1516843384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9540686e0 con 0x7fa9540c32f0 2024-03-06T07:24:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.349+0000 7fa94affd640 1 -- 172.21.15.1:0/1516843384 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa93c037070 con 0x7fa9540c32f0 2024-03-06T07:24:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.350+0000 7fa94affd640 1 --2- 172.21.15.1:0/1516843384 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa93406b4d0 0x7fa93406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.350+0000 7fa94affd640 1 -- 172.21.15.1:0/1516843384 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa93c0ac3c0 con 0x7fa9540c32f0 2024-03-06T07:24:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.350+0000 7fa95259c640 1 --2- 172.21.15.1:0/1516843384 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa93406b4d0 0x7fa93406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.351+0000 7fa95259c640 1 --2- 172.21.15.1:0/1516843384 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa93406b4d0 0x7fa93406d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa944007fd0 tx=0x7fa944007dc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.354+0000 7fa94affd640 1 -- 172.21.15.1:0/1516843384 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa93c07b140 con 0x7fa9540c32f0 2024-03-06T07:24:13.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.537+0000 7fa958c4e640 1 -- 172.21.15.1:0/1516843384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9540c7b10 con 0x7fa9540c32f0 2024-03-06T07:24:13.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.538+0000 7fa94affd640 1 -- 172.21.15.1:0/1516843384 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16832 (secure 0 0 0) 0x7fa93c07aa40 con 0x7fa9540c32f0 2024-03-06T07:24:13.540 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:13.540 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":108,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:24:11.864352+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:08.484521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.184053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.539+0000 7fa958c4e640 1 -- 172.21.15.1:0/1516843384 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa93406b4d0 msgr2=0x7fa93406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.539+0000 7fa958c4e640 1 --2- 172.21.15.1:0/1516843384 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa93406b4d0 0x7fa93406d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa944007fd0 tx=0x7fa944007dc0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.539+0000 7fa958c4e640 1 -- 172.21.15.1:0/1516843384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9540c32f0 msgr2=0x7fa9540c1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.539+0000 7fa958c4e640 1 --2- 172.21.15.1:0/1516843384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9540c32f0 0x7fa9540c1f90 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fa93c02bf70 tx=0x7fa93c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.540+0000 7fa958c4e640 1 -- 172.21.15.1:0/1516843384 shutdown_connections 2024-03-06T07:24:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.540+0000 7fa958c4e640 1 --2- 172.21.15.1:0/1516843384 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa93406b4d0 0x7fa93406d990 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.540+0000 7fa958c4e640 1 --2- 172.21.15.1:0/1516843384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa954150520 0x7fa9541460e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.540+0000 7fa958c4e640 1 --2- 172.21.15.1:0/1516843384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9540c32f0 0x7fa9540c1f90 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.540+0000 7fa958c4e640 1 --2- 172.21.15.1:0/1516843384 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa954070c80 0x7fa95406dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.540+0000 7fa958c4e640 1 -- 172.21.15.1:0/1516843384 >> 172.21.15.1:0/1516843384 conn(0x7fa9540c88f0 msgr2=0x7fa9540c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.540+0000 7fa958c4e640 1 -- 172.21.15.1:0/1516843384 shutdown_connections 2024-03-06T07:24:13.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.540+0000 7fa958c4e640 1 -- 172.21.15.1:0/1516843384 wait complete. 2024-03-06T07:24:13.553 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:24:13.553 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:13.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.653+0000 7fdce3050640 1 Processor -- start 2024-03-06T07:24:13.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.653+0000 7fdce3050640 1 -- start start 2024-03-06T07:24:13.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.653+0000 7fdce3050640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdcdc0c3970 0x7fdcdc0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.653+0000 7fdce3050640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdcdc152840 0x7fdcdc1553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.654+0000 7fdce3050640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcdc0c5e40 con 0x7fdcdc0c60b0 2024-03-06T07:24:13.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.654+0000 7fdce3050640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcdc0c67f0 con 0x7fdcdc0c3970 2024-03-06T07:24:13.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.654+0000 7fdce3050640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc0c6a40 con 0x7fdcdc152840 2024-03-06T07:24:13.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.654+0000 7fdce0dec640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdcdc0c3970 0x7fdcdc0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.654+0000 7fdce0dec640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdcdc0c3970 0x7fdcdc0c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53636/0 (socket says 172.21.15.1:53636) 2024-03-06T07:24:13.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.654+0000 7fdce0dec640 1 -- 172.21.15.1:0/138235203 learned_addr learned my addr 172.21.15.1:0/138235203 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:13.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.654+0000 7fdce15ed640 1 --2- 172.21.15.1:0/138235203 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdcdc152840 0x7fdcdc1553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.654+0000 7fdce0dec640 1 -- 172.21.15.1:0/138235203 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdcdc152840 msgr2=0x7fdcdc1553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.654+0000 7fdce0dec640 1 --2- 172.21.15.1:0/138235203 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdcdc152840 0x7fdcdc1553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.654+0000 7fdce0dec640 1 -- 172.21.15.1:0/138235203 >> v1:172.21.15.98:6789/0 conn(0x7fdcdc0c60b0 legacy=0x7fdcdc152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.654+0000 7fdce0dec640 1 -- 172.21.15.1:0/138235203 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcdc0c6d20 con 0x7fdcdc0c3970 2024-03-06T07:24:13.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.654+0000 7fdce15ed640 1 --2- 172.21.15.1:0/138235203 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdcdc152840 0x7fdcdc1553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:13.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.655+0000 7fdce0dec640 1 --2- 172.21.15.1:0/138235203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcdc0c3970 0x7fdcdc0c8170 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0009870 tx=0x7fdcd002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb69a6523ca918c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.655+0000 7fdcdb7fe640 1 -- 172.21.15.1:0/138235203 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0034070 con 0x7fdcdc0c3970 2024-03-06T07:24:13.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.655+0000 7fdcdb7fe640 1 -- 172.21.15.1:0/138235203 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd0030840 con 0x7fdcdc0c3970 2024-03-06T07:24:13.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.655+0000 7fdcdb7fe640 1 -- 172.21.15.1:0/138235203 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0030c10 con 0x7fdcdc0c3970 2024-03-06T07:24:13.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.656+0000 7fdce3050640 1 -- 172.21.15.1:0/138235203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcdc0c3970 msgr2=0x7fdcdc0c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.656+0000 7fdce3050640 1 --2- 172.21.15.1:0/138235203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcdc0c3970 0x7fdcdc0c8170 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0009870 tx=0x7fdcd002b980 comp rx=0 tx=0).stop 2024-03-06T07:24:13.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.656+0000 7fdce3050640 1 -- 172.21.15.1:0/138235203 shutdown_connections 2024-03-06T07:24:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.656+0000 7fdce3050640 1 --2- 172.21.15.1:0/138235203 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdcdc152840 0x7fdcdc1553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.656+0000 7fdce3050640 1 --2- 172.21.15.1:0/138235203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcdc0c3970 0x7fdcdc0c8170 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.656+0000 7fdce3050640 1 -- 172.21.15.1:0/138235203 >> 172.21.15.1:0/138235203 conn(0x7fdcdc0c88f0 msgr2=0x7fdcdc0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.656+0000 7fdce3050640 1 -- 172.21.15.1:0/138235203 shutdown_connections 2024-03-06T07:24:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.656+0000 7fdce3050640 1 -- 172.21.15.1:0/138235203 wait complete. 2024-03-06T07:24:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.657+0000 7fdce3050640 1 Processor -- start 2024-03-06T07:24:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.657+0000 7fdce3050640 1 -- start start 2024-03-06T07:24:13.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.658+0000 7fdce3050640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcdc0c3970 0x7fdcdc1502d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.658+0000 7fdce3050640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcdc0c60b0 0x7fdcdc152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.658+0000 7fdce3050640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdcdc152840 0x7fdcdc14bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.658+0000 7fdce3050640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc0c75c0 con 0x7fdcdc0c3970 2024-03-06T07:24:13.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.658+0000 7fdce3050640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc0c72e0 con 0x7fdcdc0c60b0 2024-03-06T07:24:13.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.658+0000 7fdce3050640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc0c3d50 con 0x7fdcdc152840 2024-03-06T07:24:13.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdce0dec640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcdc0c3970 0x7fdcdc1502d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdce0dec640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcdc0c3970 0x7fdcdc1502d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53652/0 (socket says 172.21.15.1:53652) 2024-03-06T07:24:13.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdce0dec640 1 -- 172.21.15.1:0/3318844938 learned_addr learned my addr 172.21.15.1:0/3318844938 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:13.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdcdbfff640 1 --2- 172.21.15.1:0/3318844938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcdc0c60b0 0x7fdcdc152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdce15ed640 1 --2- 172.21.15.1:0/3318844938 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdcdc152840 0x7fdcdc14bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdce0dec640 1 -- 172.21.15.1:0/3318844938 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdcdc152840 msgr2=0x7fdcdc14bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdce0dec640 1 --2- 172.21.15.1:0/3318844938 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdcdc152840 0x7fdcdc14bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdce0dec640 1 -- 172.21.15.1:0/3318844938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcdc0c60b0 msgr2=0x7fdcdc152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdce0dec640 1 --2- 172.21.15.1:0/3318844938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcdc0c60b0 0x7fdcdc152f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdce0dec640 1 -- 172.21.15.1:0/3318844938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcdc06f4e0 con 0x7fdcdc0c3970 2024-03-06T07:24:13.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdcdbfff640 1 --2- 172.21.15.1:0/3318844938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcdc0c60b0 0x7fdcdc152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:13.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdce15ed640 1 --2- 172.21.15.1:0/3318844938 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdcdc152840 0x7fdcdc14bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:13.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdce0dec640 1 --2- 172.21.15.1:0/3318844938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcdc0c3970 0x7fdcdc1502d0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fdcd002b9c0 tx=0x7fdcd0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdcd97fa640 1 -- 172.21.15.1:0/3318844938 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0048070 con 0x7fdcdc0c3970 2024-03-06T07:24:13.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdcd97fa640 1 -- 172.21.15.1:0/3318844938 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd0030640 con 0x7fdcdc0c3970 2024-03-06T07:24:13.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdcd97fa640 1 -- 172.21.15.1:0/3318844938 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0034040 con 0x7fdcdc0c3970 2024-03-06T07:24:13.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdce3050640 1 -- 172.21.15.1:0/3318844938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcdc0a9f80 con 0x7fdcdc0c3970 2024-03-06T07:24:13.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.659+0000 7fdce3050640 1 -- 172.21.15.1:0/3318844938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcdc1eed80 con 0x7fdcdc0c3970 2024-03-06T07:24:13.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.660+0000 7fdce3050640 1 -- 172.21.15.1:0/3318844938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcdc0686e0 con 0x7fdcdc0c3970 2024-03-06T07:24:13.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.661+0000 7fdcd97fa640 1 -- 172.21.15.1:0/3318844938 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdcd004b070 con 0x7fdcdc0c3970 2024-03-06T07:24:13.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.662+0000 7fdcd97fa640 1 --2- 172.21.15.1:0/3318844938 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdcc006b500 0x7fdcc006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.662+0000 7fdcd97fa640 1 -- 172.21.15.1:0/3318844938 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdcd00ac1c0 con 0x7fdcdc0c3970 2024-03-06T07:24:13.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.662+0000 7fdcdbfff640 1 --2- 172.21.15.1:0/3318844938 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdcc006b500 0x7fdcc006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.662+0000 7fdcdbfff640 1 --2- 172.21.15.1:0/3318844938 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdcc006b500 0x7fdcc006d9c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc150030 tx=0x7fdcc8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.666+0000 7fdcd97fa640 1 -- 172.21.15.1:0/3318844938 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdcd007b060 con 0x7fdcdc0c3970 2024-03-06T07:24:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.843+0000 7fdce3050640 1 -- 172.21.15.1:0/3318844938 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdcdc0aa820 con 0x7fdcc006b500 2024-03-06T07:24:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.849+0000 7fdcd97fa640 1 -- 172.21.15.1:0/3318844938 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240335 (secure 0 0 0) 0x7fdcdc0aa820 con 0x7fdcc006b500 2024-03-06T07:24:13.854 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:24:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.851+0000 7fdce3050640 1 -- 172.21.15.1:0/3318844938 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdcc006b500 msgr2=0x7fdcc006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.851+0000 7fdce3050640 1 --2- 172.21.15.1:0/3318844938 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdcc006b500 0x7fdcc006d9c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc150030 tx=0x7fdcc8009040 comp rx=0 tx=0).stop 2024-03-06T07:24:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.851+0000 7fdce3050640 1 -- 172.21.15.1:0/3318844938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcdc0c3970 msgr2=0x7fdcdc1502d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.851+0000 7fdce3050640 1 --2- 172.21.15.1:0/3318844938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcdc0c3970 0x7fdcdc1502d0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fdcd002b9c0 tx=0x7fdcd0004730 comp rx=0 tx=0).stop 2024-03-06T07:24:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.851+0000 7fdce3050640 1 -- 172.21.15.1:0/3318844938 shutdown_connections 2024-03-06T07:24:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.852+0000 7fdce3050640 1 --2- 172.21.15.1:0/3318844938 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdcc006b500 0x7fdcc006d9c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.852+0000 7fdce3050640 1 --2- 172.21.15.1:0/3318844938 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdcdc152840 0x7fdcdc14bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.852+0000 7fdce3050640 1 --2- 172.21.15.1:0/3318844938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcdc0c60b0 0x7fdcdc152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.852+0000 7fdce3050640 1 --2- 172.21.15.1:0/3318844938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcdc0c3970 0x7fdcdc1502d0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.852+0000 7fdce3050640 1 -- 172.21.15.1:0/3318844938 >> 172.21.15.1:0/3318844938 conn(0x7fdcdc0c88f0 msgr2=0x7fdcdc074600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.852+0000 7fdce3050640 1 -- 172.21.15.1:0/3318844938 shutdown_connections 2024-03-06T07:24:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:13.852+0000 7fdce3050640 1 -- 172.21.15.1:0/3318844938 wait complete. 2024-03-06T07:24:13.865 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-03-06T07:24:12.743580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259804,"kb_used_data":10356,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714916,"statfs":{"total":773094113280,"available":772828073984,"internally_reserved":0,"allocated":10604544,"data_stored":6213759,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000912"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.854086+0000","last_change":"2024-03-06T07:24:08.854086+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.854086+0000","last_undegraded":"2024-03-06T07:24:08.854086+0000","last_fullsized":"2024-03-06T07:24:08.854086+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.845274+0000","last_change":"2024-03-06T07:24:08.845274+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.845274+0000","last_undegraded":"2024-03-06T07:24:08.845274+0000","last_fullsized":"2024-03-06T07:24:08.845274+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.850739+0000","last_change":"2024-03-06T07:24:08.850739+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.850739+0000","last_undegraded":"2024-03-06T07:24:08.850739+0000","last_fullsized":"2024-03-06T07:24:08.850739+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.845273+0000","last_change":"2024-03-06T07:24:08.845273+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.845273+0000","last_undegraded":"2024-03-06T07:24:08.845273+0000","last_fullsized":"2024-03-06T07:24:08.845273+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.850067+0000","last_change":"2024-03-06T07:24:08.850067+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.850067+0000","last_undegraded":"2024-03-06T07:24:08.850067+0000","last_fullsized":"2024-03-06T07:24:08.850067+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.847973+0000","last_change":"2024-03-06T07:24:08.847973+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.847973+0000","last_undegraded":"2024-03-06T07:24:08.847973+0000","last_fullsized":"2024-03-06T07:24:08.847973+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.848585+0000","last_change":"2024-03-06T07:24:08.848585+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.848585+0000","last_undegraded":"2024-03-06T07:24:08.848585+0000","last_fullsized":"2024-03-06T07:24:08.848585+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.847916+0000","last_change":"2024-03-06T07:24:08.847916+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.847916+0000","last_undegraded":"2024-03-06T07:24:08.847916+0000","last_fullsized":"2024-03-06T07:24:08.847916+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.853639+0000","last_change":"2024-03-06T07:24:08.853639+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.853639+0000","last_undegraded":"2024-03-06T07:24:08.853639+0000","last_fullsized":"2024-03-06T07:24:08.853639+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.848662+0000","last_change":"2024-03-06T07:24:08.848662+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.848662+0000","last_undegraded":"2024-03-06T07:24:08.848662+0000","last_fullsized":"2024-03-06T07:24:08.848662+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.848578+0000","last_change":"2024-03-06T07:24:08.848578+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.848578+0000","last_undegraded":"2024-03-06T07:24:08.848578+0000","last_fullsized":"2024-03-06T07:24:08.848578+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846438+0000","last_change":"2024-03-06T07:24:08.846438+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846438+0000","last_undegraded":"2024-03-06T07:24:08.846438+0000","last_fullsized":"2024-03-06T07:24:08.846438+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846316+0000","last_change":"2024-03-06T07:24:08.846316+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846316+0000","last_undegraded":"2024-03-06T07:24:08.846316+0000","last_fullsized":"2024-03-06T07:24:08.846316+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846427+0000","last_change":"2024-03-06T07:24:08.846427+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846427+0000","last_undegraded":"2024-03-06T07:24:08.846427+0000","last_fullsized":"2024-03-06T07:24:08.846427+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846992+0000","last_change":"2024-03-06T07:24:08.846992+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846992+0000","last_undegraded":"2024-03-06T07:24:08.846992+0000","last_fullsized":"2024-03-06T07:24:08.846992+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.845307+0000","last_change":"2024-03-06T07:24:08.845307+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.845307+0000","last_undegraded":"2024-03-06T07:24:08.845307+0000","last_fullsized":"2024-03-06T07:24:08.845307+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.847439+0000","last_change":"2024-03-06T07:24:08.847439+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.847439+0000","last_undegraded":"2024-03-06T07:24:08.847439+0000","last_fullsized":"2024-03-06T07:24:08.847439+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.850264+0000","last_change":"2024-03-06T07:24:08.850264+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.850264+0000","last_undegraded":"2024-03-06T07:24:08.850264+0000","last_fullsized":"2024-03-06T07:24:08.850264+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.849826+0000","last_change":"2024-03-06T07:24:08.849826+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.849826+0000","last_undegraded":"2024-03-06T07:24:08.849826+0000","last_fullsized":"2024-03-06T07:24:08.849826+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.847378+0000","last_change":"2024-03-06T07:24:08.847378+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.847378+0000","last_undegraded":"2024-03-06T07:24:08.847378+0000","last_fullsized":"2024-03-06T07:24:08.847378+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.848578+0000","last_change":"2024-03-06T07:24:08.848578+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.848578+0000","last_undegraded":"2024-03-06T07:24:08.848578+0000","last_fullsized":"2024-03-06T07:24:08.848578+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846361+0000","last_change":"2024-03-06T07:24:08.846361+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846361+0000","last_undegraded":"2024-03-06T07:24:08.846361+0000","last_fullsized":"2024-03-06T07:24:08.846361+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.849875+0000","last_change":"2024-03-06T07:24:08.849875+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.849875+0000","last_undegraded":"2024-03-06T07:24:08.849875+0000","last_fullsized":"2024-03-06T07:24:08.849875+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.847320+0000","last_change":"2024-03-06T07:24:08.847320+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.847320+0000","last_undegraded":"2024-03-06T07:24:08.847320+0000","last_fullsized":"2024-03-06T07:24:08.847320+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.853598+0000","last_change":"2024-03-06T07:24:08.853598+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.853598+0000","last_undegraded":"2024-03-06T07:24:08.853598+0000","last_fullsized":"2024-03-06T07:24:08.853598+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.853646+0000","last_change":"2024-03-06T07:24:08.853646+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.853646+0000","last_undegraded":"2024-03-06T07:24:08.853646+0000","last_fullsized":"2024-03-06T07:24:08.853646+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846811+0000","last_change":"2024-03-06T07:24:08.846811+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846811+0000","last_undegraded":"2024-03-06T07:24:08.846811+0000","last_fullsized":"2024-03-06T07:24:08.846811+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.849580+0000","last_change":"2024-03-06T07:24:08.849580+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.849580+0000","last_undegraded":"2024-03-06T07:24:08.849580+0000","last_fullsized":"2024-03-06T07:24:08.849580+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.849483+0000","last_change":"2024-03-06T07:24:08.849483+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.849483+0000","last_undegraded":"2024-03-06T07:24:08.849483+0000","last_fullsized":"2024-03-06T07:24:08.849483+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846300+0000","last_change":"2024-03-06T07:24:08.846300+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846300+0000","last_undegraded":"2024-03-06T07:24:08.846300+0000","last_fullsized":"2024-03-06T07:24:08.846300+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.848679+0000","last_change":"2024-03-06T07:24:08.848679+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.848679+0000","last_undegraded":"2024-03-06T07:24:08.848679+0000","last_fullsized":"2024-03-06T07:24:08.848679+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846681+0000","last_change":"2024-03-06T07:24:08.846681+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846681+0000","last_undegraded":"2024-03-06T07:24:08.846681+0000","last_fullsized":"2024-03-06T07:24:08.846681+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":333,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-06T07:24:08.851102+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:24:08.851102+0000","last_peered":"2024-03-06T07:24:08.851102+0000","last_clean":"2024-03-06T07:24:08.851102+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:24:08.851102+0000","last_undegraded":"2024-03-06T07:24:08.851102+0000","last_fullsized":"2024-03-06T07:24:08.851102+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1376256,"data_stored":1220272,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":629904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1298432,"data_stored":623695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":629904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":629904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":629904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":629904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1376256,"data_stored":1220272,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:13.906 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:14.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.005+0000 7fd446d02640 1 Processor -- start 2024-03-06T07:24:14.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.005+0000 7fd446d02640 1 -- start start 2024-03-06T07:24:14.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.006+0000 7fd446d02640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd4400c32f0 0x7fd4401543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.006+0000 7fd446d02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd440070c80 0x7fd4400710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.006+0000 7fd446d02640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd440150900 con 0x7fd440150520 2024-03-06T07:24:14.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.006+0000 7fd446d02640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd440153ed0 con 0x7fd4400c32f0 2024-03-06T07:24:14.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.006+0000 7fd446d02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd440071790 con 0x7fd440070c80 2024-03-06T07:24:14.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.006+0000 7fd43ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd4400c32f0 0x7fd4401543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.006+0000 7fd43ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd4400c32f0 0x7fd4401543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53668/0 (socket says 172.21.15.1:53668) 2024-03-06T07:24:14.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.006+0000 7fd43ffff640 1 -- 172.21.15.1:0/2731980025 learned_addr learned my addr 172.21.15.1:0/2731980025 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:14.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.006+0000 7fd44529f640 1 --2- 172.21.15.1:0/2731980025 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd440070c80 0x7fd4400710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.006+0000 7fd43ffff640 1 -- 172.21.15.1:0/2731980025 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd440070c80 msgr2=0x7fd4400710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.006+0000 7fd43ffff640 1 --2- 172.21.15.1:0/2731980025 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd440070c80 0x7fd4400710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.006+0000 7fd43ffff640 1 -- 172.21.15.1:0/2731980025 >> v1:172.21.15.98:6789/0 conn(0x7fd440150520 legacy=0x7fd4401537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.006+0000 7fd43ffff640 1 -- 172.21.15.1:0/2731980025 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd440157050 con 0x7fd4400c32f0 2024-03-06T07:24:14.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.007+0000 7fd43ffff640 1 --2- 172.21.15.1:0/2731980025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4400c32f0 0x7fd4401543a0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fd42c009870 tx=0x7fd42c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11a970a4cfccb64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.007+0000 7fd43f7fe640 1 -- 172.21.15.1:0/2731980025 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c034070 con 0x7fd4400c32f0 2024-03-06T07:24:14.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.007+0000 7fd43f7fe640 1 -- 172.21.15.1:0/2731980025 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42c030840 con 0x7fd4400c32f0 2024-03-06T07:24:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.008+0000 7fd43f7fe640 1 -- 172.21.15.1:0/2731980025 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c030c10 con 0x7fd4400c32f0 2024-03-06T07:24:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.008+0000 7fd446d02640 1 -- 172.21.15.1:0/2731980025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4400c32f0 msgr2=0x7fd4401543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.008+0000 7fd446d02640 1 --2- 172.21.15.1:0/2731980025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4400c32f0 0x7fd4401543a0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fd42c009870 tx=0x7fd42c02b980 comp rx=0 tx=0).stop 2024-03-06T07:24:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.008+0000 7fd446d02640 1 -- 172.21.15.1:0/2731980025 shutdown_connections 2024-03-06T07:24:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.008+0000 7fd446d02640 1 --2- 172.21.15.1:0/2731980025 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd440070c80 0x7fd4400710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.008+0000 7fd446d02640 1 --2- 172.21.15.1:0/2731980025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4400c32f0 0x7fd4401543a0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.008+0000 7fd446d02640 1 -- 172.21.15.1:0/2731980025 >> 172.21.15.1:0/2731980025 conn(0x7fd4400c88f0 msgr2=0x7fd4400c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.008+0000 7fd446d02640 1 -- 172.21.15.1:0/2731980025 shutdown_connections 2024-03-06T07:24:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.008+0000 7fd446d02640 1 -- 172.21.15.1:0/2731980025 wait complete. 2024-03-06T07:24:14.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.009+0000 7fd446d02640 1 Processor -- start 2024-03-06T07:24:14.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.009+0000 7fd446d02640 1 -- start start 2024-03-06T07:24:14.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.010+0000 7fd446d02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd440070c80 0x7fd4401ebee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.010+0000 7fd446d02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4400c32f0 0x7fd4401401a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.010+0000 7fd446d02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd440150520 0x7fd4401406e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.010+0000 7fd446d02640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd440157330 con 0x7fd440070c80 2024-03-06T07:24:14.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.010+0000 7fd446d02640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4401532a0 con 0x7fd440150520 2024-03-06T07:24:14.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.010+0000 7fd446d02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4400c7ec0 con 0x7fd4400c32f0 2024-03-06T07:24:14.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.010+0000 7fd444a9e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd440070c80 0x7fd4401ebee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.010+0000 7fd444a9e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd440070c80 0x7fd4401ebee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53676/0 (socket says 172.21.15.1:53676) 2024-03-06T07:24:14.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.010+0000 7fd444a9e640 1 -- 172.21.15.1:0/1618566945 learned_addr learned my addr 172.21.15.1:0/1618566945 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:14.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.010+0000 7fd43ffff640 1 --2- 172.21.15.1:0/1618566945 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4400c32f0 0x7fd4401401a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.010+0000 7fd44529f640 1 --2- 172.21.15.1:0/1618566945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd440150520 0x7fd4401406e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.010+0000 7fd444a9e640 1 -- 172.21.15.1:0/1618566945 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4400c32f0 msgr2=0x7fd4401401a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.010+0000 7fd444a9e640 1 --2- 172.21.15.1:0/1618566945 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4400c32f0 0x7fd4401401a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.010+0000 7fd444a9e640 1 -- 172.21.15.1:0/1618566945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd440150520 msgr2=0x7fd4401406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.010+0000 7fd444a9e640 1 --2- 172.21.15.1:0/1618566945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd440150520 0x7fd4401406e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.010+0000 7fd444a9e640 1 -- 172.21.15.1:0/1618566945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4401ec960 con 0x7fd440070c80 2024-03-06T07:24:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.010+0000 7fd44529f640 1 --2- 172.21.15.1:0/1618566945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd440150520 0x7fd4401406e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.011+0000 7fd444a9e640 1 --2- 172.21.15.1:0/1618566945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd440070c80 0x7fd4401ebee0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fd43400c3f0 tx=0x7fd43400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.011+0000 7fd43d7fa640 1 -- 172.21.15.1:0/1618566945 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4340052d0 con 0x7fd440070c80 2024-03-06T07:24:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.011+0000 7fd43d7fa640 1 -- 172.21.15.1:0/1618566945 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd434005490 con 0x7fd440070c80 2024-03-06T07:24:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.011+0000 7fd43d7fa640 1 -- 172.21.15.1:0/1618566945 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434007260 con 0x7fd440070c80 2024-03-06T07:24:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.011+0000 7fd446d02640 1 -- 172.21.15.1:0/1618566945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd440155520 con 0x7fd440070c80 2024-03-06T07:24:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.011+0000 7fd446d02640 1 -- 172.21.15.1:0/1618566945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd440155d20 con 0x7fd440070c80 2024-03-06T07:24:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.012+0000 7fd446d02640 1 -- 172.21.15.1:0/1618566945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4400686e0 con 0x7fd440070c80 2024-03-06T07:24:14.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.012+0000 7fd43d7fa640 1 -- 172.21.15.1:0/1618566945 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd434007400 con 0x7fd440070c80 2024-03-06T07:24:14.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.013+0000 7fd43d7fa640 1 --2- 172.21.15.1:0/1618566945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd42406b4d0 0x7fd42406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.013+0000 7fd43d7fa640 1 -- 172.21.15.1:0/1618566945 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd4340898c0 con 0x7fd440070c80 2024-03-06T07:24:14.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.013+0000 7fd43ffff640 1 --2- 172.21.15.1:0/1618566945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd42406b4d0 0x7fd42406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.013+0000 7fd43ffff640 1 --2- 172.21.15.1:0/1618566945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd42406b4d0 0x7fd42406d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd42c036000 tx=0x7fd42c03b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.017+0000 7fd43d7fa640 1 -- 172.21.15.1:0/1618566945 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd434058520 con 0x7fd440070c80 2024-03-06T07:24:14.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.198+0000 7fd446d02640 1 -- 172.21.15.1:0/1618566945 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd440155a40 con 0x7fd42406b4d0 2024-03-06T07:24:14.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.204+0000 7fd43d7fa640 1 -- 172.21.15.1:0/1618566945 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240335 (secure 0 0 0) 0x7fd440155a40 con 0x7fd42406b4d0 2024-03-06T07:24:14.208 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:14.209 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:24:14.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.208+0000 7fd446d02640 1 -- 172.21.15.1:0/1618566945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd42406b4d0 msgr2=0x7fd42406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.208+0000 7fd446d02640 1 --2- 172.21.15.1:0/1618566945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd42406b4d0 0x7fd42406d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd42c036000 tx=0x7fd42c03b040 comp rx=0 tx=0).stop 2024-03-06T07:24:14.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.208+0000 7fd446d02640 1 -- 172.21.15.1:0/1618566945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd440070c80 msgr2=0x7fd4401ebee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.208+0000 7fd446d02640 1 --2- 172.21.15.1:0/1618566945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd440070c80 0x7fd4401ebee0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fd43400c3f0 tx=0x7fd43400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.208+0000 7fd446d02640 1 -- 172.21.15.1:0/1618566945 shutdown_connections 2024-03-06T07:24:14.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.208+0000 7fd446d02640 1 --2- 172.21.15.1:0/1618566945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd42406b4d0 0x7fd42406d990 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.208+0000 7fd446d02640 1 --2- 172.21.15.1:0/1618566945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd440150520 0x7fd4401406e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.208+0000 7fd446d02640 1 --2- 172.21.15.1:0/1618566945 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4400c32f0 0x7fd4401401a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.208+0000 7fd446d02640 1 --2- 172.21.15.1:0/1618566945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd440070c80 0x7fd4401ebee0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.208+0000 7fd446d02640 1 -- 172.21.15.1:0/1618566945 >> 172.21.15.1:0/1618566945 conn(0x7fd4400c88f0 msgr2=0x7fd44014a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.208+0000 7fd446d02640 1 -- 172.21.15.1:0/1618566945 shutdown_connections 2024-03-06T07:24:14.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.208+0000 7fd446d02640 1 -- 172.21.15.1:0/1618566945 wait complete. 2024-03-06T07:24:14.221 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-03-06T07:24:12.743580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259804,"kb_used_data":10356,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714916,"statfs":{"total":773094113280,"available":772828073984,"internally_reserved":0,"allocated":10604544,"data_stored":6213759,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000912"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.854086+0000","last_change":"2024-03-06T07:24:08.854086+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.854086+0000","last_undegraded":"2024-03-06T07:24:08.854086+0000","last_fullsized":"2024-03-06T07:24:08.854086+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.845274+0000","last_change":"2024-03-06T07:24:08.845274+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.845274+0000","last_undegraded":"2024-03-06T07:24:08.845274+0000","last_fullsized":"2024-03-06T07:24:08.845274+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.850739+0000","last_change":"2024-03-06T07:24:08.850739+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.850739+0000","last_undegraded":"2024-03-06T07:24:08.850739+0000","last_fullsized":"2024-03-06T07:24:08.850739+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.845273+0000","last_change":"2024-03-06T07:24:08.845273+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.845273+0000","last_undegraded":"2024-03-06T07:24:08.845273+0000","last_fullsized":"2024-03-06T07:24:08.845273+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.850067+0000","last_change":"2024-03-06T07:24:08.850067+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.850067+0000","last_undegraded":"2024-03-06T07:24:08.850067+0000","last_fullsized":"2024-03-06T07:24:08.850067+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.847973+0000","last_change":"2024-03-06T07:24:08.847973+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.847973+0000","last_undegraded":"2024-03-06T07:24:08.847973+0000","last_fullsized":"2024-03-06T07:24:08.847973+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.848585+0000","last_change":"2024-03-06T07:24:08.848585+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.848585+0000","last_undegraded":"2024-03-06T07:24:08.848585+0000","last_fullsized":"2024-03-06T07:24:08.848585+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.847916+0000","last_change":"2024-03-06T07:24:08.847916+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.847916+0000","last_undegraded":"2024-03-06T07:24:08.847916+0000","last_fullsized":"2024-03-06T07:24:08.847916+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.853639+0000","last_change":"2024-03-06T07:24:08.853639+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.853639+0000","last_undegraded":"2024-03-06T07:24:08.853639+0000","last_fullsized":"2024-03-06T07:24:08.853639+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.848662+0000","last_change":"2024-03-06T07:24:08.848662+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.848662+0000","last_undegraded":"2024-03-06T07:24:08.848662+0000","last_fullsized":"2024-03-06T07:24:08.848662+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.848578+0000","last_change":"2024-03-06T07:24:08.848578+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.848578+0000","last_undegraded":"2024-03-06T07:24:08.848578+0000","last_fullsized":"2024-03-06T07:24:08.848578+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846438+0000","last_change":"2024-03-06T07:24:08.846438+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846438+0000","last_undegraded":"2024-03-06T07:24:08.846438+0000","last_fullsized":"2024-03-06T07:24:08.846438+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846316+0000","last_change":"2024-03-06T07:24:08.846316+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846316+0000","last_undegraded":"2024-03-06T07:24:08.846316+0000","last_fullsized":"2024-03-06T07:24:08.846316+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846427+0000","last_change":"2024-03-06T07:24:08.846427+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846427+0000","last_undegraded":"2024-03-06T07:24:08.846427+0000","last_fullsized":"2024-03-06T07:24:08.846427+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846992+0000","last_change":"2024-03-06T07:24:08.846992+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846992+0000","last_undegraded":"2024-03-06T07:24:08.846992+0000","last_fullsized":"2024-03-06T07:24:08.846992+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.845307+0000","last_change":"2024-03-06T07:24:08.845307+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.845307+0000","last_undegraded":"2024-03-06T07:24:08.845307+0000","last_fullsized":"2024-03-06T07:24:08.845307+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.847439+0000","last_change":"2024-03-06T07:24:08.847439+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.847439+0000","last_undegraded":"2024-03-06T07:24:08.847439+0000","last_fullsized":"2024-03-06T07:24:08.847439+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.850264+0000","last_change":"2024-03-06T07:24:08.850264+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.850264+0000","last_undegraded":"2024-03-06T07:24:08.850264+0000","last_fullsized":"2024-03-06T07:24:08.850264+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.849826+0000","last_change":"2024-03-06T07:24:08.849826+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.849826+0000","last_undegraded":"2024-03-06T07:24:08.849826+0000","last_fullsized":"2024-03-06T07:24:08.849826+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.847378+0000","last_change":"2024-03-06T07:24:08.847378+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.847378+0000","last_undegraded":"2024-03-06T07:24:08.847378+0000","last_fullsized":"2024-03-06T07:24:08.847378+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.848578+0000","last_change":"2024-03-06T07:24:08.848578+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.848578+0000","last_undegraded":"2024-03-06T07:24:08.848578+0000","last_fullsized":"2024-03-06T07:24:08.848578+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846361+0000","last_change":"2024-03-06T07:24:08.846361+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846361+0000","last_undegraded":"2024-03-06T07:24:08.846361+0000","last_fullsized":"2024-03-06T07:24:08.846361+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.849875+0000","last_change":"2024-03-06T07:24:08.849875+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.849875+0000","last_undegraded":"2024-03-06T07:24:08.849875+0000","last_fullsized":"2024-03-06T07:24:08.849875+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.847320+0000","last_change":"2024-03-06T07:24:08.847320+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.847320+0000","last_undegraded":"2024-03-06T07:24:08.847320+0000","last_fullsized":"2024-03-06T07:24:08.847320+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.853598+0000","last_change":"2024-03-06T07:24:08.853598+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.853598+0000","last_undegraded":"2024-03-06T07:24:08.853598+0000","last_fullsized":"2024-03-06T07:24:08.853598+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.853646+0000","last_change":"2024-03-06T07:24:08.853646+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.853646+0000","last_undegraded":"2024-03-06T07:24:08.853646+0000","last_fullsized":"2024-03-06T07:24:08.853646+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846811+0000","last_change":"2024-03-06T07:24:08.846811+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846811+0000","last_undegraded":"2024-03-06T07:24:08.846811+0000","last_fullsized":"2024-03-06T07:24:08.846811+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.849580+0000","last_change":"2024-03-06T07:24:08.849580+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.849580+0000","last_undegraded":"2024-03-06T07:24:08.849580+0000","last_fullsized":"2024-03-06T07:24:08.849580+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.849483+0000","last_change":"2024-03-06T07:24:08.849483+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.849483+0000","last_undegraded":"2024-03-06T07:24:08.849483+0000","last_fullsized":"2024-03-06T07:24:08.849483+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846300+0000","last_change":"2024-03-06T07:24:08.846300+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846300+0000","last_undegraded":"2024-03-06T07:24:08.846300+0000","last_fullsized":"2024-03-06T07:24:08.846300+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.848679+0000","last_change":"2024-03-06T07:24:08.848679+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.848679+0000","last_undegraded":"2024-03-06T07:24:08.848679+0000","last_fullsized":"2024-03-06T07:24:08.848679+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846681+0000","last_change":"2024-03-06T07:24:08.846681+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846681+0000","last_undegraded":"2024-03-06T07:24:08.846681+0000","last_fullsized":"2024-03-06T07:24:08.846681+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":333,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-06T07:24:08.851102+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:24:08.851102+0000","last_peered":"2024-03-06T07:24:08.851102+0000","last_clean":"2024-03-06T07:24:08.851102+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:24:08.851102+0000","last_undegraded":"2024-03-06T07:24:08.851102+0000","last_fullsized":"2024-03-06T07:24:08.851102+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1376256,"data_stored":1220272,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":629904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1298432,"data_stored":623695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":629904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":629904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":629904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":629904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1376256,"data_stored":1220272,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:14.224 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:14.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.360+0000 7f32fcd1e640 1 Processor -- start 2024-03-06T07:24:14.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.360+0000 7f32fcd1e640 1 -- start start 2024-03-06T07:24:14.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.361+0000 7f32fcd1e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32f806e670 0x7f32f806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.361+0000 7f32fcd1e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f806f060 con 0x7f32f80c1bc0 2024-03-06T07:24:14.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.361+0000 7f32fcd1e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f80c26e0 con 0x7f32f80a8520 2024-03-06T07:24:14.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.361+0000 7f32fcd1e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80c2050 con 0x7f32f806e670 2024-03-06T07:24:14.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.361+0000 7f32f6d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f32f80c1bc0 0x7f32f8153290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36016/0 (socket says 172.21.15.1:36016) 2024-03-06T07:24:14.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.361+0000 7f32f6d9d640 1 -- 172.21.15.1:0/118329787 learned_addr learned my addr 172.21.15.1:0/118329787 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:14.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.361+0000 7f32f659c640 1 --2- 172.21.15.1:0/118329787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32f806e670 0x7f32f806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.361+0000 7f32f5d9b640 1 -- 172.21.15.1:0/118329787 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542998445 0 0) 0x7f32f806f060 con 0x7f32f80c1bc0 2024-03-06T07:24:14.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.361+0000 7f32f5d9b640 1 -- 172.21.15.1:0/118329787 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e4003520 con 0x7f32f80c1bc0 2024-03-06T07:24:14.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.361+0000 7f32f5d9b640 1 -- 172.21.15.1:0/118329787 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1579381470 0 0) 0x7f32e4003520 con 0x7f32f80c1bc0 2024-03-06T07:24:14.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.361+0000 7f32f5d9b640 1 -- 172.21.15.1:0/118329787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32f806e670 msgr2=0x7f32f806ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.361+0000 7f32f5d9b640 1 --2- 172.21.15.1:0/118329787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32f806e670 0x7f32f806ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.361+0000 7f32f5d9b640 1 -- 172.21.15.1:0/118329787 >> v1:172.21.15.98:6789/0 conn(0x7f32f80a8520 legacy=0x7f32f8150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.361+0000 7f32f5d9b640 1 -- 172.21.15.1:0/118329787 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f80c29c0 con 0x7f32f80c1bc0 2024-03-06T07:24:14.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.361+0000 7f32f5d9b640 1 -- 172.21.15.1:0/118329787 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f32f00019f0 con 0x7f32f80c1bc0 2024-03-06T07:24:14.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.362+0000 7f32f5d9b640 1 -- 172.21.15.1:0/118329787 >> v1:172.21.15.1:6789/0 conn(0x7f32f80c1bc0 legacy=0x7f32f8153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.362+0000 7f32f5d9b640 1 --2- 172.21.15.1:0/118329787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e40039e0 0x7f32e4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.362+0000 7f32f5d9b640 1 --2- 172.21.15.1:0/118329787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e4004390 0x7f32e4004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.362+0000 7f32f5d9b640 1 --2- 172.21.15.1:0/118329787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32f806e670 0x7f32e4005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.362+0000 7f32f5d9b640 1 -- 172.21.15.1:0/118329787 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80c2050 con 0x7f32e40039e0 2024-03-06T07:24:14.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.362+0000 7f32f5d9b640 1 -- 172.21.15.1:0/118329787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e400ba70 con 0x7f32e4004390 2024-03-06T07:24:14.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.362+0000 7f32f5d9b640 1 -- 172.21.15.1:0/118329787 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e400bc70 con 0x7f32f806e670 2024-03-06T07:24:14.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.362+0000 7f32f659c640 1 --2- 172.21.15.1:0/118329787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e40039e0 0x7f32e4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.362+0000 7f32effff640 1 --2- 172.21.15.1:0/118329787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e4004390 0x7f32e4004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.362+0000 7f32f6d9d640 1 --2- 172.21.15.1:0/118329787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32f806e670 0x7f32e4005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.362+0000 7f32f659c640 1 -- 172.21.15.1:0/118329787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32f806e670 msgr2=0x7f32e4005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.362+0000 7f32f659c640 1 --2- 172.21.15.1:0/118329787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32f806e670 0x7f32e4005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.362+0000 7f32f659c640 1 -- 172.21.15.1:0/118329787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e4004390 msgr2=0x7f32e4004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.362+0000 7f32f659c640 1 --2- 172.21.15.1:0/118329787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e4004390 0x7f32e4004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.362+0000 7f32f659c640 1 -- 172.21.15.1:0/118329787 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e4003520 con 0x7f32e40039e0 2024-03-06T07:24:14.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.362+0000 7f32f6d9d640 1 --2- 172.21.15.1:0/118329787 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32f806e670 0x7f32e4005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:14.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.363+0000 7f32f659c640 1 --2- 172.21.15.1:0/118329787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e40039e0 0x7f32e4003dc0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009ac0 tx=0x7f32e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d5d6f7f6d112b30 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.363+0000 7f32f5d9b640 1 -- 172.21.15.1:0/118329787 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e8039070 con 0x7f32e40039e0 2024-03-06T07:24:14.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.363+0000 7f32f5d9b640 1 -- 172.21.15.1:0/118329787 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e802c6b0 con 0x7f32e40039e0 2024-03-06T07:24:14.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.363+0000 7f32f5d9b640 1 -- 172.21.15.1:0/118329787 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e8009690 con 0x7f32e40039e0 2024-03-06T07:24:14.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.363+0000 7f32fcd1e640 1 -- 172.21.15.1:0/118329787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e40039e0 msgr2=0x7f32e4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.363+0000 7f32fcd1e640 1 --2- 172.21.15.1:0/118329787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e40039e0 0x7f32e4003dc0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009ac0 tx=0x7f32e4005830 comp rx=0 tx=0).stop 2024-03-06T07:24:14.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.363+0000 7f32f6d9d640 1 -- 172.21.15.1:0/118329787 reap_dead start 2024-03-06T07:24:14.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.363+0000 7f32fcd1e640 1 -- 172.21.15.1:0/118329787 shutdown_connections 2024-03-06T07:24:14.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.363+0000 7f32fcd1e640 1 -- 172.21.15.1:0/118329787 >> 172.21.15.1:0/118329787 conn(0x7f32f80c3cc0 msgr2=0x7f32f80c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.364+0000 7f32fcd1e640 1 -- 172.21.15.1:0/118329787 shutdown_connections 2024-03-06T07:24:14.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.364+0000 7f32fcd1e640 1 -- 172.21.15.1:0/118329787 wait complete. 2024-03-06T07:24:14.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.365+0000 7f32fcd1e640 1 Processor -- start 2024-03-06T07:24:14.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.365+0000 7f32fcd1e640 1 -- start start 2024-03-06T07:24:14.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.365+0000 7f32fcd1e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e40039e0 0x7f32f8151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.365+0000 7f32fcd1e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32f80aa490 0x7f32f8145ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32fcd1e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32f80aa910 0x7f32f8146020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32fcd1e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f806ea50 con 0x7f32f80aa910 2024-03-06T07:24:14.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32fcd1e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80c2ca0 con 0x7f32e40039e0 2024-03-06T07:24:14.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32fcd1e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f806ebd0 con 0x7f32f80aa490 2024-03-06T07:24:14.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32f6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32f80aa910 0x7f32f8146020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32f659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e40039e0 0x7f32f8151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32f6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32f80aa910 0x7f32f8146020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44264/0 (socket says 172.21.15.1:44264) 2024-03-06T07:24:14.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32f659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e40039e0 0x7f32f8151530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:46650/0 (socket says 172.21.15.1:46650) 2024-03-06T07:24:14.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32f6d9d640 1 -- 172.21.15.1:0/18329775 learned_addr learned my addr 172.21.15.1:0/18329775 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:14.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32effff640 1 --2- 172.21.15.1:0/18329775 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32f80aa490 0x7f32f8145ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32f6d9d640 1 -- 172.21.15.1:0/18329775 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32f80aa490 msgr2=0x7f32f8145ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32f6d9d640 1 --2- 172.21.15.1:0/18329775 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32f80aa490 0x7f32f8145ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32f6d9d640 1 -- 172.21.15.1:0/18329775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e40039e0 msgr2=0x7f32f8151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32f6d9d640 1 --2- 172.21.15.1:0/18329775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e40039e0 0x7f32f8151530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32f6d9d640 1 -- 172.21.15.1:0/18329775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f81eac20 con 0x7f32f80aa910 2024-03-06T07:24:14.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32f6d9d640 1 --2- 172.21.15.1:0/18329775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32f80aa910 0x7f32f8146020 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f32f8072e60 tx=0x7f32f000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32eeffd640 1 -- 172.21.15.1:0/18329775 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f000eb60 con 0x7f32f80aa910 2024-03-06T07:24:14.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32eeffd640 1 -- 172.21.15.1:0/18329775 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f000ed00 con 0x7f32f80aa910 2024-03-06T07:24:14.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32fcd1e640 1 -- 172.21.15.1:0/18329775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f81eaf00 con 0x7f32f80aa910 2024-03-06T07:24:14.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32eeffd640 1 -- 172.21.15.1:0/18329775 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f0014670 con 0x7f32f80aa910 2024-03-06T07:24:14.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.366+0000 7f32fcd1e640 1 -- 172.21.15.1:0/18329775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f81eb4c0 con 0x7f32f80aa910 2024-03-06T07:24:14.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.367+0000 7f32fcd1e640 1 -- 172.21.15.1:0/18329775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f8067fd0 con 0x7f32f80aa910 2024-03-06T07:24:14.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.368+0000 7f32eeffd640 1 -- 172.21.15.1:0/18329775 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f32f0014810 con 0x7f32f80aa910 2024-03-06T07:24:14.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.369+0000 7f32eeffd640 1 --2- 172.21.15.1:0/18329775 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32d806b4d0 0x7f32d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.369+0000 7f32eeffd640 1 -- 172.21.15.1:0/18329775 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f32f008a060 con 0x7f32f80aa910 2024-03-06T07:24:14.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.369+0000 7f32f659c640 1 --2- 172.21.15.1:0/18329775 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32d806b4d0 0x7f32d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.370+0000 7f32f659c640 1 --2- 172.21.15.1:0/18329775 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32d806b4d0 0x7f32d806d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f32e8036000 tx=0x7f32e8030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.373+0000 7f32eeffd640 1 -- 172.21.15.1:0/18329775 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32f0058660 con 0x7f32f80aa910 2024-03-06T07:24:14.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.551+0000 7f32fcd1e640 1 -- 172.21.15.1:0/18329775 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f32f80c18a0 con 0x7f32d806b4d0 2024-03-06T07:24:14.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.557+0000 7f32eeffd640 1 -- 172.21.15.1:0/18329775 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240335 (secure 0 0 0) 0x7f32f80c18a0 con 0x7f32d806b4d0 2024-03-06T07:24:14.561 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:14.562 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:24:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.560+0000 7f32fcd1e640 1 -- 172.21.15.1:0/18329775 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32d806b4d0 msgr2=0x7f32d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.560+0000 7f32fcd1e640 1 --2- 172.21.15.1:0/18329775 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32d806b4d0 0x7f32d806d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f32e8036000 tx=0x7f32e8030000 comp rx=0 tx=0).stop 2024-03-06T07:24:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.560+0000 7f32fcd1e640 1 -- 172.21.15.1:0/18329775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32f80aa910 msgr2=0x7f32f8146020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.560+0000 7f32fcd1e640 1 --2- 172.21.15.1:0/18329775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32f80aa910 0x7f32f8146020 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f32f8072e60 tx=0x7f32f000e530 comp rx=0 tx=0).stop 2024-03-06T07:24:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.560+0000 7f32fcd1e640 1 -- 172.21.15.1:0/18329775 shutdown_connections 2024-03-06T07:24:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.560+0000 7f32fcd1e640 1 --2- 172.21.15.1:0/18329775 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32d806b4d0 0x7f32d806d990 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.560+0000 7f32fcd1e640 1 --2- 172.21.15.1:0/18329775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32f80aa910 0x7f32f8146020 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.560+0000 7f32fcd1e640 1 --2- 172.21.15.1:0/18329775 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32f80aa490 0x7f32f8145ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.560+0000 7f32fcd1e640 1 --2- 172.21.15.1:0/18329775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e40039e0 0x7f32f8151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.560+0000 7f32fcd1e640 1 -- 172.21.15.1:0/18329775 >> 172.21.15.1:0/18329775 conn(0x7f32f80c3cc0 msgr2=0x7f32f8071d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.560+0000 7f32fcd1e640 1 -- 172.21.15.1:0/18329775 shutdown_connections 2024-03-06T07:24:14.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.560+0000 7f32fcd1e640 1 -- 172.21.15.1:0/18329775 wait complete. 2024-03-06T07:24:14.573 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-03-06T07:24:12.743580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259804,"kb_used_data":10356,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714916,"statfs":{"total":773094113280,"available":772828073984,"internally_reserved":0,"allocated":10604544,"data_stored":6213759,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000912"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.854086+0000","last_change":"2024-03-06T07:24:08.854086+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.854086+0000","last_undegraded":"2024-03-06T07:24:08.854086+0000","last_fullsized":"2024-03-06T07:24:08.854086+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.845274+0000","last_change":"2024-03-06T07:24:08.845274+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.845274+0000","last_undegraded":"2024-03-06T07:24:08.845274+0000","last_fullsized":"2024-03-06T07:24:08.845274+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.850739+0000","last_change":"2024-03-06T07:24:08.850739+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.850739+0000","last_undegraded":"2024-03-06T07:24:08.850739+0000","last_fullsized":"2024-03-06T07:24:08.850739+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.845273+0000","last_change":"2024-03-06T07:24:08.845273+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.845273+0000","last_undegraded":"2024-03-06T07:24:08.845273+0000","last_fullsized":"2024-03-06T07:24:08.845273+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.850067+0000","last_change":"2024-03-06T07:24:08.850067+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.850067+0000","last_undegraded":"2024-03-06T07:24:08.850067+0000","last_fullsized":"2024-03-06T07:24:08.850067+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.847973+0000","last_change":"2024-03-06T07:24:08.847973+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.847973+0000","last_undegraded":"2024-03-06T07:24:08.847973+0000","last_fullsized":"2024-03-06T07:24:08.847973+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.848585+0000","last_change":"2024-03-06T07:24:08.848585+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.848585+0000","last_undegraded":"2024-03-06T07:24:08.848585+0000","last_fullsized":"2024-03-06T07:24:08.848585+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.847916+0000","last_change":"2024-03-06T07:24:08.847916+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.847916+0000","last_undegraded":"2024-03-06T07:24:08.847916+0000","last_fullsized":"2024-03-06T07:24:08.847916+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.853639+0000","last_change":"2024-03-06T07:24:08.853639+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.853639+0000","last_undegraded":"2024-03-06T07:24:08.853639+0000","last_fullsized":"2024-03-06T07:24:08.853639+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.848662+0000","last_change":"2024-03-06T07:24:08.848662+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.848662+0000","last_undegraded":"2024-03-06T07:24:08.848662+0000","last_fullsized":"2024-03-06T07:24:08.848662+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.848578+0000","last_change":"2024-03-06T07:24:08.848578+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.848578+0000","last_undegraded":"2024-03-06T07:24:08.848578+0000","last_fullsized":"2024-03-06T07:24:08.848578+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846438+0000","last_change":"2024-03-06T07:24:08.846438+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846438+0000","last_undegraded":"2024-03-06T07:24:08.846438+0000","last_fullsized":"2024-03-06T07:24:08.846438+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846316+0000","last_change":"2024-03-06T07:24:08.846316+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846316+0000","last_undegraded":"2024-03-06T07:24:08.846316+0000","last_fullsized":"2024-03-06T07:24:08.846316+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846427+0000","last_change":"2024-03-06T07:24:08.846427+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846427+0000","last_undegraded":"2024-03-06T07:24:08.846427+0000","last_fullsized":"2024-03-06T07:24:08.846427+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846992+0000","last_change":"2024-03-06T07:24:08.846992+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846992+0000","last_undegraded":"2024-03-06T07:24:08.846992+0000","last_fullsized":"2024-03-06T07:24:08.846992+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.845307+0000","last_change":"2024-03-06T07:24:08.845307+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.845307+0000","last_undegraded":"2024-03-06T07:24:08.845307+0000","last_fullsized":"2024-03-06T07:24:08.845307+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.847439+0000","last_change":"2024-03-06T07:24:08.847439+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.847439+0000","last_undegraded":"2024-03-06T07:24:08.847439+0000","last_fullsized":"2024-03-06T07:24:08.847439+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.850264+0000","last_change":"2024-03-06T07:24:08.850264+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.850264+0000","last_undegraded":"2024-03-06T07:24:08.850264+0000","last_fullsized":"2024-03-06T07:24:08.850264+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.849826+0000","last_change":"2024-03-06T07:24:08.849826+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.849826+0000","last_undegraded":"2024-03-06T07:24:08.849826+0000","last_fullsized":"2024-03-06T07:24:08.849826+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.847378+0000","last_change":"2024-03-06T07:24:08.847378+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.847378+0000","last_undegraded":"2024-03-06T07:24:08.847378+0000","last_fullsized":"2024-03-06T07:24:08.847378+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.848578+0000","last_change":"2024-03-06T07:24:08.848578+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.848578+0000","last_undegraded":"2024-03-06T07:24:08.848578+0000","last_fullsized":"2024-03-06T07:24:08.848578+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846361+0000","last_change":"2024-03-06T07:24:08.846361+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846361+0000","last_undegraded":"2024-03-06T07:24:08.846361+0000","last_fullsized":"2024-03-06T07:24:08.846361+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.849875+0000","last_change":"2024-03-06T07:24:08.849875+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.849875+0000","last_undegraded":"2024-03-06T07:24:08.849875+0000","last_fullsized":"2024-03-06T07:24:08.849875+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.847320+0000","last_change":"2024-03-06T07:24:08.847320+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.847320+0000","last_undegraded":"2024-03-06T07:24:08.847320+0000","last_fullsized":"2024-03-06T07:24:08.847320+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.853598+0000","last_change":"2024-03-06T07:24:08.853598+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.853598+0000","last_undegraded":"2024-03-06T07:24:08.853598+0000","last_fullsized":"2024-03-06T07:24:08.853598+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.853646+0000","last_change":"2024-03-06T07:24:08.853646+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.853646+0000","last_undegraded":"2024-03-06T07:24:08.853646+0000","last_fullsized":"2024-03-06T07:24:08.853646+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846811+0000","last_change":"2024-03-06T07:24:08.846811+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846811+0000","last_undegraded":"2024-03-06T07:24:08.846811+0000","last_fullsized":"2024-03-06T07:24:08.846811+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.849580+0000","last_change":"2024-03-06T07:24:08.849580+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.849580+0000","last_undegraded":"2024-03-06T07:24:08.849580+0000","last_fullsized":"2024-03-06T07:24:08.849580+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.849483+0000","last_change":"2024-03-06T07:24:08.849483+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.849483+0000","last_undegraded":"2024-03-06T07:24:08.849483+0000","last_fullsized":"2024-03-06T07:24:08.849483+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846300+0000","last_change":"2024-03-06T07:24:08.846300+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846300+0000","last_undegraded":"2024-03-06T07:24:08.846300+0000","last_fullsized":"2024-03-06T07:24:08.846300+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.848679+0000","last_change":"2024-03-06T07:24:08.848679+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.848679+0000","last_undegraded":"2024-03-06T07:24:08.848679+0000","last_fullsized":"2024-03-06T07:24:08.848679+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:09.842470+0000","last_change":"2024-03-06T07:24:09.842470+0000","last_active":"2024-03-06T07:24:09.842470+0000","last_peered":"2024-03-06T07:24:09.842470+0000","last_clean":"2024-03-06T07:24:09.842470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:09.842470+0000","last_undegraded":"2024-03-06T07:24:09.842470+0000","last_fullsized":"2024-03-06T07:24:09.842470+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-03-06T07:24:08.846681+0000","last_change":"2024-03-06T07:24:08.846681+0000","last_active":"2024-03-06T07:24:08.838505+0000","last_peered":"2024-03-06T07:24:08.838505+0000","last_clean":"2024-03-06T07:24:08.838505+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:08.846681+0000","last_undegraded":"2024-03-06T07:24:08.846681+0000","last_fullsized":"2024-03-06T07:24:08.846681+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":333,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-06T07:24:08.851102+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:24:08.851102+0000","last_peered":"2024-03-06T07:24:08.851102+0000","last_clean":"2024-03-06T07:24:08.851102+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:24:08.851102+0000","last_undegraded":"2024-03-06T07:24:08.851102+0000","last_fullsized":"2024-03-06T07:24:08.851102+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1376256,"data_stored":1220272,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":629904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1298432,"data_stored":623695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":629904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":629904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":629904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":629904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1376256,"data_stored":1220272,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:14.577 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3f is not active+clean 2024-03-06T07:24:14.577 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.577 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3e is not active+clean 2024-03-06T07:24:14.577 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.577 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3d is not active+clean 2024-03-06T07:24:14.577 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.577 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3c is not active+clean 2024-03-06T07:24:14.578 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.578 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3b is not active+clean 2024-03-06T07:24:14.578 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.578 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3a is not active+clean 2024-03-06T07:24:14.578 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.578 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.39 is not active+clean 2024-03-06T07:24:14.578 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.578 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.38 is not active+clean 2024-03-06T07:24:14.578 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.578 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.37 is not active+clean 2024-03-06T07:24:14.578 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.579 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.36 is not active+clean 2024-03-06T07:24:14.579 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.579 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.35 is not active+clean 2024-03-06T07:24:14.579 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.579 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.34 is not active+clean 2024-03-06T07:24:14.579 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.579 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.33 is not active+clean 2024-03-06T07:24:14.579 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.579 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.32 is not active+clean 2024-03-06T07:24:14.579 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.31 is not active+clean 2024-03-06T07:24:14.580 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.30 is not active+clean 2024-03-06T07:24:14.580 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2f is not active+clean 2024-03-06T07:24:14.580 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2e is not active+clean 2024-03-06T07:24:14.580 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2d is not active+clean 2024-03-06T07:24:14.580 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2c is not active+clean 2024-03-06T07:24:14.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2b is not active+clean 2024-03-06T07:24:14.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2a is not active+clean 2024-03-06T07:24:14.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.29 is not active+clean 2024-03-06T07:24:14.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.28 is not active+clean 2024-03-06T07:24:14.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.582 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.27 is not active+clean 2024-03-06T07:24:14.582 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.582 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.26 is not active+clean 2024-03-06T07:24:14.582 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.582 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.25 is not active+clean 2024-03-06T07:24:14.582 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.582 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.24 is not active+clean 2024-03-06T07:24:14.582 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.582 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.23 is not active+clean 2024-03-06T07:24:14.582 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.582 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.22 is not active+clean 2024-03-06T07:24:14.582 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.583 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.21 is not active+clean 2024-03-06T07:24:14.583 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.583 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.20 is not active+clean 2024-03-06T07:24:14.583 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.583 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1c is not active+clean 2024-03-06T07:24:14.583 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.583 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1f is not active+clean 2024-03-06T07:24:14.583 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.854086+0000', 'last_change': '2024-03-06T07:24:08.854086+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.854086+0000', 'last_undegraded': '2024-03-06T07:24:08.854086+0000', 'last_fullsized': '2024-03-06T07:24:08.854086+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:14.583 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1d is not active+clean 2024-03-06T07:24:14.583 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.584 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1e is not active+clean 2024-03-06T07:24:14.584 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.845274+0000', 'last_change': '2024-03-06T07:24:08.845274+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.845274+0000', 'last_undegraded': '2024-03-06T07:24:08.845274+0000', 'last_fullsized': '2024-03-06T07:24:08.845274+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:14.584 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1e is not active+clean 2024-03-06T07:24:14.584 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.584 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1d is not active+clean 2024-03-06T07:24:14.584 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.850739+0000', 'last_change': '2024-03-06T07:24:08.850739+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.850739+0000', 'last_undegraded': '2024-03-06T07:24:08.850739+0000', 'last_fullsized': '2024-03-06T07:24:08.850739+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:14.584 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1f is not active+clean 2024-03-06T07:24:14.584 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.584 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1c is not active+clean 2024-03-06T07:24:14.584 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.845273+0000', 'last_change': '2024-03-06T07:24:08.845273+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.845273+0000', 'last_undegraded': '2024-03-06T07:24:08.845273+0000', 'last_fullsized': '2024-03-06T07:24:08.845273+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:14.585 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.18 is not active+clean 2024-03-06T07:24:14.585 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.585 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1b is not active+clean 2024-03-06T07:24:14.585 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.850067+0000', 'last_change': '2024-03-06T07:24:08.850067+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.850067+0000', 'last_undegraded': '2024-03-06T07:24:08.850067+0000', 'last_fullsized': '2024-03-06T07:24:08.850067+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:14.585 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.19 is not active+clean 2024-03-06T07:24:14.585 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.585 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1a is not active+clean 2024-03-06T07:24:14.585 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.847973+0000', 'last_change': '2024-03-06T07:24:08.847973+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.847973+0000', 'last_undegraded': '2024-03-06T07:24:08.847973+0000', 'last_fullsized': '2024-03-06T07:24:08.847973+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:14.585 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1a is not active+clean 2024-03-06T07:24:14.585 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.19 is not active+clean 2024-03-06T07:24:14.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.848585+0000', 'last_change': '2024-03-06T07:24:08.848585+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.848585+0000', 'last_undegraded': '2024-03-06T07:24:08.848585+0000', 'last_fullsized': '2024-03-06T07:24:08.848585+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:14.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1b is not active+clean 2024-03-06T07:24:14.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.18 is not active+clean 2024-03-06T07:24:14.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.847916+0000', 'last_change': '2024-03-06T07:24:08.847916+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.847916+0000', 'last_undegraded': '2024-03-06T07:24:08.847916+0000', 'last_fullsized': '2024-03-06T07:24:08.847916+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:14.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.14 is not active+clean 2024-03-06T07:24:14.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.17 is not active+clean 2024-03-06T07:24:14.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.853639+0000', 'last_change': '2024-03-06T07:24:08.853639+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.853639+0000', 'last_undegraded': '2024-03-06T07:24:08.853639+0000', 'last_fullsized': '2024-03-06T07:24:08.853639+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:14.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.15 is not active+clean 2024-03-06T07:24:14.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.16 is not active+clean 2024-03-06T07:24:14.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.848662+0000', 'last_change': '2024-03-06T07:24:08.848662+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.848662+0000', 'last_undegraded': '2024-03-06T07:24:08.848662+0000', 'last_fullsized': '2024-03-06T07:24:08.848662+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:14.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.16 is not active+clean 2024-03-06T07:24:14.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.15 is not active+clean 2024-03-06T07:24:14.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.848578+0000', 'last_change': '2024-03-06T07:24:08.848578+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.848578+0000', 'last_undegraded': '2024-03-06T07:24:08.848578+0000', 'last_fullsized': '2024-03-06T07:24:08.848578+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:14.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.17 is not active+clean 2024-03-06T07:24:14.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.14 is not active+clean 2024-03-06T07:24:14.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.846438+0000', 'last_change': '2024-03-06T07:24:08.846438+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.846438+0000', 'last_undegraded': '2024-03-06T07:24:08.846438+0000', 'last_fullsized': '2024-03-06T07:24:08.846438+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:14.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.10 is not active+clean 2024-03-06T07:24:14.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.13 is not active+clean 2024-03-06T07:24:14.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.846316+0000', 'last_change': '2024-03-06T07:24:08.846316+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.846316+0000', 'last_undegraded': '2024-03-06T07:24:08.846316+0000', 'last_fullsized': '2024-03-06T07:24:08.846316+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:14.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.11 is not active+clean 2024-03-06T07:24:14.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.12 is not active+clean 2024-03-06T07:24:14.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.846427+0000', 'last_change': '2024-03-06T07:24:08.846427+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.846427+0000', 'last_undegraded': '2024-03-06T07:24:08.846427+0000', 'last_fullsized': '2024-03-06T07:24:08.846427+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:14.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.12 is not active+clean 2024-03-06T07:24:14.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.11 is not active+clean 2024-03-06T07:24:14.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.846992+0000', 'last_change': '2024-03-06T07:24:08.846992+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.846992+0000', 'last_undegraded': '2024-03-06T07:24:08.846992+0000', 'last_fullsized': '2024-03-06T07:24:08.846992+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:14.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.c is not active+clean 2024-03-06T07:24:14.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.f is not active+clean 2024-03-06T07:24:14.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.845307+0000', 'last_change': '2024-03-06T07:24:08.845307+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.845307+0000', 'last_undegraded': '2024-03-06T07:24:08.845307+0000', 'last_fullsized': '2024-03-06T07:24:08.845307+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:14.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.d is not active+clean 2024-03-06T07:24:14.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.e is not active+clean 2024-03-06T07:24:14.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.847439+0000', 'last_change': '2024-03-06T07:24:08.847439+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.847439+0000', 'last_undegraded': '2024-03-06T07:24:08.847439+0000', 'last_fullsized': '2024-03-06T07:24:08.847439+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:14.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.e is not active+clean 2024-03-06T07:24:14.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.d is not active+clean 2024-03-06T07:24:14.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.850264+0000', 'last_change': '2024-03-06T07:24:08.850264+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.850264+0000', 'last_undegraded': '2024-03-06T07:24:08.850264+0000', 'last_fullsized': '2024-03-06T07:24:08.850264+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:14.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.f is not active+clean 2024-03-06T07:24:14.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.c is not active+clean 2024-03-06T07:24:14.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.849826+0000', 'last_change': '2024-03-06T07:24:08.849826+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.849826+0000', 'last_undegraded': '2024-03-06T07:24:08.849826+0000', 'last_fullsized': '2024-03-06T07:24:08.849826+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:14.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.8 is not active+clean 2024-03-06T07:24:14.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.b is not active+clean 2024-03-06T07:24:14.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.847378+0000', 'last_change': '2024-03-06T07:24:08.847378+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.847378+0000', 'last_undegraded': '2024-03-06T07:24:08.847378+0000', 'last_fullsized': '2024-03-06T07:24:08.847378+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:14.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.9 is not active+clean 2024-03-06T07:24:14.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.a is not active+clean 2024-03-06T07:24:14.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.848578+0000', 'last_change': '2024-03-06T07:24:08.848578+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.848578+0000', 'last_undegraded': '2024-03-06T07:24:08.848578+0000', 'last_fullsized': '2024-03-06T07:24:08.848578+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:14.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.a is not active+clean 2024-03-06T07:24:14.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.9 is not active+clean 2024-03-06T07:24:14.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.846361+0000', 'last_change': '2024-03-06T07:24:08.846361+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.846361+0000', 'last_undegraded': '2024-03-06T07:24:08.846361+0000', 'last_fullsized': '2024-03-06T07:24:08.846361+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:14.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.b is not active+clean 2024-03-06T07:24:14.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.8 is not active+clean 2024-03-06T07:24:14.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.849875+0000', 'last_change': '2024-03-06T07:24:08.849875+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.849875+0000', 'last_undegraded': '2024-03-06T07:24:08.849875+0000', 'last_fullsized': '2024-03-06T07:24:08.849875+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:14.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.4 is not active+clean 2024-03-06T07:24:14.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.592 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.7 is not active+clean 2024-03-06T07:24:14.592 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.847320+0000', 'last_change': '2024-03-06T07:24:08.847320+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.847320+0000', 'last_undegraded': '2024-03-06T07:24:08.847320+0000', 'last_fullsized': '2024-03-06T07:24:08.847320+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:14.592 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.5 is not active+clean 2024-03-06T07:24:14.592 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.592 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.6 is not active+clean 2024-03-06T07:24:14.592 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.853598+0000', 'last_change': '2024-03-06T07:24:08.853598+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.853598+0000', 'last_undegraded': '2024-03-06T07:24:08.853598+0000', 'last_fullsized': '2024-03-06T07:24:08.853598+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:14.592 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.6 is not active+clean 2024-03-06T07:24:14.592 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.592 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.5 is not active+clean 2024-03-06T07:24:14.592 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.853646+0000', 'last_change': '2024-03-06T07:24:08.853646+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.853646+0000', 'last_undegraded': '2024-03-06T07:24:08.853646+0000', 'last_fullsized': '2024-03-06T07:24:08.853646+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:14.593 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.7 is not active+clean 2024-03-06T07:24:14.593 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.593 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.4 is not active+clean 2024-03-06T07:24:14.593 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.846811+0000', 'last_change': '2024-03-06T07:24:08.846811+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.846811+0000', 'last_undegraded': '2024-03-06T07:24:08.846811+0000', 'last_fullsized': '2024-03-06T07:24:08.846811+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:14.593 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.0 is not active+clean 2024-03-06T07:24:14.593 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.593 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3 is not active+clean 2024-03-06T07:24:14.593 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.849580+0000', 'last_change': '2024-03-06T07:24:08.849580+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.849580+0000', 'last_undegraded': '2024-03-06T07:24:08.849580+0000', 'last_fullsized': '2024-03-06T07:24:08.849580+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:14.593 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1 is not active+clean 2024-03-06T07:24:14.593 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.593 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2 is not active+clean 2024-03-06T07:24:14.594 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.849483+0000', 'last_change': '2024-03-06T07:24:08.849483+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.849483+0000', 'last_undegraded': '2024-03-06T07:24:08.849483+0000', 'last_fullsized': '2024-03-06T07:24:08.849483+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:14.594 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2 is not active+clean 2024-03-06T07:24:14.594 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.594 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1 is not active+clean 2024-03-06T07:24:14.594 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.846300+0000', 'last_change': '2024-03-06T07:24:08.846300+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.846300+0000', 'last_undegraded': '2024-03-06T07:24:08.846300+0000', 'last_fullsized': '2024-03-06T07:24:08.846300+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:14.594 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3 is not active+clean 2024-03-06T07:24:14.594 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.594 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.0 is not active+clean 2024-03-06T07:24:14.594 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.848679+0000', 'last_change': '2024-03-06T07:24:08.848679+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.848679+0000', 'last_undegraded': '2024-03-06T07:24:08.848679+0000', 'last_fullsized': '2024-03-06T07:24:08.848679+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:14.594 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.13 is not active+clean 2024-03-06T07:24:14.594 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:09.842470+0000', 'last_change': '2024-03-06T07:24:09.842470+0000', 'last_active': '2024-03-06T07:24:09.842470+0000', 'last_peered': '2024-03-06T07:24:09.842470+0000', 'last_clean': '2024-03-06T07:24:09.842470+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:09.842470+0000', 'last_undegraded': '2024-03-06T07:24:09.842470+0000', 'last_fullsized': '2024-03-06T07:24:09.842470+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:09.842470+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:14.595 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.10 is not active+clean 2024-03-06T07:24:14.595 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:08.846681+0000', 'last_change': '2024-03-06T07:24:08.846681+0000', 'last_active': '2024-03-06T07:24:08.838505+0000', 'last_peered': '2024-03-06T07:24:08.838505+0000', 'last_clean': '2024-03-06T07:24:08.838505+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:08.846681+0000', 'last_undegraded': '2024-03-06T07:24:08.846681+0000', 'last_fullsized': '2024-03-06T07:24:08.846681+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:08.838505+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:14.595 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:24:14.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.712+0000 7f342b69a640 1 Processor -- start 2024-03-06T07:24:14.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.712+0000 7f342b69a640 1 -- start start 2024-03-06T07:24:14.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.713+0000 7f342b69a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f34240c2b30 0x7f34240724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.713+0000 7f342b69a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3424152b70 0x7f3424072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.713+0000 7f342b69a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f34240a5fa0 0x7f34240c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.713+0000 7f342b69a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34241526e0 con 0x7f34240c2b30 2024-03-06T07:24:14.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.713+0000 7f342b69a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3424151dd0 con 0x7f34240a5fa0 2024-03-06T07:24:14.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.713+0000 7f342b69a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3424156190 con 0x7f3424152b70 2024-03-06T07:24:14.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.713+0000 7f3429436640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f34240c2b30 0x7f34240724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.713+0000 7f3429436640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f34240c2b30 0x7f34240724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44270/0 (socket says 172.21.15.1:44270) 2024-03-06T07:24:14.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.713+0000 7f3429436640 1 -- 172.21.15.1:0/2608690466 learned_addr learned my addr 172.21.15.1:0/2608690466 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:14.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.713+0000 7f3428c35640 1 --2- 172.21.15.1:0/2608690466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3424152b70 0x7f3424072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.713+0000 7f3429c37640 1 --2- 172.21.15.1:0/2608690466 >> v2:172.21.15.98:3300/0 conn(0x7f34240a5fa0 0x7f34240c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.713+0000 7f3429436640 1 -- 172.21.15.1:0/2608690466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3424152b70 msgr2=0x7f3424072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.713+0000 7f3429436640 1 --2- 172.21.15.1:0/2608690466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3424152b70 0x7f3424072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.713+0000 7f3429436640 1 -- 172.21.15.1:0/2608690466 >> v2:172.21.15.98:3300/0 conn(0x7f34240a5fa0 msgr2=0x7f34240c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.713+0000 7f3429436640 1 --2- 172.21.15.1:0/2608690466 >> v2:172.21.15.98:3300/0 conn(0x7f34240a5fa0 0x7f34240c32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.714+0000 7f3429436640 1 -- 172.21.15.1:0/2608690466 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3424152090 con 0x7f34240c2b30 2024-03-06T07:24:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.714+0000 7f3429436640 1 --2- 172.21.15.1:0/2608690466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34240c2b30 0x7f34240724c0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f34240c0e20 tx=0x7f341802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82df55ee9d8a343a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.714+0000 7f3417fff640 1 -- 172.21.15.1:0/2608690466 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418035020 con 0x7f34240c2b30 2024-03-06T07:24:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.714+0000 7f3417fff640 1 -- 172.21.15.1:0/2608690466 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3418031870 con 0x7f34240c2b30 2024-03-06T07:24:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.714+0000 7f3417fff640 1 -- 172.21.15.1:0/2608690466 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418031c40 con 0x7f34240c2b30 2024-03-06T07:24:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.715+0000 7f342b69a640 1 -- 172.21.15.1:0/2608690466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34240c2b30 msgr2=0x7f34240724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.715+0000 7f342b69a640 1 --2- 172.21.15.1:0/2608690466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34240c2b30 0x7f34240724c0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f34240c0e20 tx=0x7f341802b910 comp rx=0 tx=0).stop 2024-03-06T07:24:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.715+0000 7f342b69a640 1 -- 172.21.15.1:0/2608690466 shutdown_connections 2024-03-06T07:24:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.715+0000 7f342b69a640 1 --2- 172.21.15.1:0/2608690466 >> v2:172.21.15.98:3300/0 conn(0x7f34240a5fa0 0x7f34240c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.715+0000 7f342b69a640 1 --2- 172.21.15.1:0/2608690466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3424152b70 0x7f3424072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.715+0000 7f342b69a640 1 --2- 172.21.15.1:0/2608690466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34240c2b30 0x7f34240724c0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.715+0000 7f342b69a640 1 -- 172.21.15.1:0/2608690466 >> 172.21.15.1:0/2608690466 conn(0x7f34240c6860 msgr2=0x7f34240c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.715+0000 7f342b69a640 1 -- 172.21.15.1:0/2608690466 shutdown_connections 2024-03-06T07:24:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.715+0000 7f342b69a640 1 -- 172.21.15.1:0/2608690466 wait complete. 2024-03-06T07:24:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.716+0000 7f342b69a640 1 Processor -- start 2024-03-06T07:24:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.716+0000 7f342b69a640 1 -- start start 2024-03-06T07:24:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.717+0000 7f342b69a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34240a5fa0 0x7f34240c0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.717+0000 7f342b69a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34240c2b30 0x7f342406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.717+0000 7f342b69a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3424152b70 0x7f34241e9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.717+0000 7f342b69a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3424155f00 con 0x7f34240c2b30 2024-03-06T07:24:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.717+0000 7f342b69a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3424155d80 con 0x7f3424152b70 2024-03-06T07:24:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.717+0000 7f342b69a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34240c3830 con 0x7f34240a5fa0 2024-03-06T07:24:14.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.717+0000 7f3428c35640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34240c2b30 0x7f342406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.717+0000 7f3428c35640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34240c2b30 0x7f342406e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44278/0 (socket says 172.21.15.1:44278) 2024-03-06T07:24:14.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.717+0000 7f3428c35640 1 -- 172.21.15.1:0/3870249591 learned_addr learned my addr 172.21.15.1:0/3870249591 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:14.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.717+0000 7f3428c35640 1 -- 172.21.15.1:0/3870249591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34240a5fa0 msgr2=0x7f34240c0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.717+0000 7f3428c35640 1 --2- 172.21.15.1:0/3870249591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34240a5fa0 0x7f34240c0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.717+0000 7f3428c35640 1 -- 172.21.15.1:0/3870249591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3424152b70 msgr2=0x7f34241e9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.717+0000 7f3428c35640 1 --2- 172.21.15.1:0/3870249591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3424152b70 0x7f34241e9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.717+0000 7f3428c35640 1 -- 172.21.15.1:0/3870249591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3424149ab0 con 0x7f34240c2b30 2024-03-06T07:24:14.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.717+0000 7f3428c35640 1 --2- 172.21.15.1:0/3870249591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34240c2b30 0x7f342406e9d0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f342000d620 tx=0x7f342000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.718+0000 7f3415ffb640 1 -- 172.21.15.1:0/3870249591 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342000b620 con 0x7f34240c2b30 2024-03-06T07:24:14.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.718+0000 7f3415ffb640 1 -- 172.21.15.1:0/3870249591 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3420019070 con 0x7f34240c2b30 2024-03-06T07:24:14.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.718+0000 7f342b69a640 1 -- 172.21.15.1:0/3870249591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3424149d50 con 0x7f34240c2b30 2024-03-06T07:24:14.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.718+0000 7f3415ffb640 1 -- 172.21.15.1:0/3870249591 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342000ba00 con 0x7f34240c2b30 2024-03-06T07:24:14.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.718+0000 7f342b69a640 1 -- 172.21.15.1:0/3870249591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f342414a310 con 0x7f34240c2b30 2024-03-06T07:24:14.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.719+0000 7f342b69a640 1 -- 172.21.15.1:0/3870249591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3424067fd0 con 0x7f34240c2b30 2024-03-06T07:24:14.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.719+0000 7f3415ffb640 1 -- 172.21.15.1:0/3870249591 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f342000bba0 con 0x7f34240c2b30 2024-03-06T07:24:14.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.720+0000 7f3415ffb640 1 --2- 172.21.15.1:0/3870249591 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f340806b4d0 0x7f340806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.720+0000 7f3415ffb640 1 -- 172.21.15.1:0/3870249591 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f342008a7b0 con 0x7f34240c2b30 2024-03-06T07:24:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.720+0000 7f3429436640 1 --2- 172.21.15.1:0/3870249591 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f340806b4d0 0x7f340806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.720+0000 7f3429436640 1 --2- 172.21.15.1:0/3870249591 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f340806b4d0 0x7f340806d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3418036000 tx=0x7f3418030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.726+0000 7f3415ffb640 1 -- 172.21.15.1:0/3870249591 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f34200594a0 con 0x7f34240c2b30 2024-03-06T07:24:14.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.979+0000 7f342b69a640 1 -- 172.21.15.1:0/3870249591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f34240a5d10 con 0x7f34240c2b30 2024-03-06T07:24:14.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.980+0000 7f3415ffb640 1 -- 172.21.15.1:0/3870249591 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1433 (secure 0 0 0) 0x7f3420058da0 con 0x7f34240c2b30 2024-03-06T07:24:14.982 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:14.982 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":250,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":108,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709614,"num_in_osds":8,"osd_in_since":1709709606,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":266432512,"bytes_avail":772827680768,"bytes_total":773094113280,"write_bytes_sec":2429,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":88,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"c","status":"up:active","gid":7362}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-03-06T07:23:24.733248+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:24:14.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.982+0000 7f342b69a640 1 -- 172.21.15.1:0/3870249591 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f340806b4d0 msgr2=0x7f340806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.982+0000 7f342b69a640 1 --2- 172.21.15.1:0/3870249591 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f340806b4d0 0x7f340806d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3418036000 tx=0x7f3418030000 comp rx=0 tx=0).stop 2024-03-06T07:24:14.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.982+0000 7f342b69a640 1 -- 172.21.15.1:0/3870249591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34240c2b30 msgr2=0x7f342406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.982+0000 7f342b69a640 1 --2- 172.21.15.1:0/3870249591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34240c2b30 0x7f342406e9d0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f342000d620 tx=0x7f342000dae0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.982+0000 7f342b69a640 1 -- 172.21.15.1:0/3870249591 shutdown_connections 2024-03-06T07:24:14.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.982+0000 7f342b69a640 1 --2- 172.21.15.1:0/3870249591 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f340806b4d0 0x7f340806d990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.982+0000 7f342b69a640 1 --2- 172.21.15.1:0/3870249591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3424152b70 0x7f34241e9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.982+0000 7f342b69a640 1 --2- 172.21.15.1:0/3870249591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34240c2b30 0x7f342406e9d0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.982+0000 7f342b69a640 1 --2- 172.21.15.1:0/3870249591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34240a5fa0 0x7f34240c0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.982+0000 7f342b69a640 1 -- 172.21.15.1:0/3870249591 >> 172.21.15.1:0/3870249591 conn(0x7f34240c6860 msgr2=0x7f3424074450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.982+0000 7f342b69a640 1 -- 172.21.15.1:0/3870249591 shutdown_connections 2024-03-06T07:24:14.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:14.982+0000 7f342b69a640 1 -- 172.21.15.1:0/3870249591 wait complete. 2024-03-06T07:24:14.995 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:24:14.995 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:15.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.094+0000 7ff4cd35c640 1 Processor -- start 2024-03-06T07:24:15.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.094+0000 7ff4cd35c640 1 -- start start 2024-03-06T07:24:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.095+0000 7ff4cd35c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff4c8152650 0x7ff4c8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.095+0000 7ff4cd35c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c80c2fc0 con 0x7ff4c80c6b10 2024-03-06T07:24:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.095+0000 7ff4cd35c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c8158da0 con 0x7ff4c80bf9d0 2024-03-06T07:24:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.095+0000 7ff4cd35c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c8155800 con 0x7ff4c8152650 2024-03-06T07:24:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.095+0000 7ff4c6ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff4c8152650 0x7ff4c8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.095+0000 7ff4c67fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff4c80c6b10 0x7ff4c8158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36030/0 (socket says 172.21.15.1:36030) 2024-03-06T07:24:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.095+0000 7ff4c67fc640 1 -- 172.21.15.1:0/196955328 learned_addr learned my addr 172.21.15.1:0/196955328 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.096+0000 7ff4c6ffd640 1 -- 172.21.15.1:0/196955328 >> v1:172.21.15.98:6789/0 conn(0x7ff4c80bf9d0 legacy=0x7ff4c80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.096+0000 7ff4c6ffd640 1 -- 172.21.15.1:0/196955328 >> v1:172.21.15.1:6789/0 conn(0x7ff4c80c6b10 legacy=0x7ff4c8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.096+0000 7ff4c6ffd640 1 -- 172.21.15.1:0/196955328 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c80c6690 con 0x7ff4c8152650 2024-03-06T07:24:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.096+0000 7ff4c5ffb640 1 -- 172.21.15.1:0/196955328 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716086524 0 0) 0x7ff4c80c2fc0 con 0x7ff4c80c6b10 2024-03-06T07:24:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.096+0000 7ff4c6ffd640 1 --2- 172.21.15.1:0/196955328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c8152650 0x7ff4c8155c90 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80097a0 tx=0x7ff4b802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f1c6ed32b129c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.096+0000 7ff4c5ffb640 1 -- 172.21.15.1:0/196955328 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b80315a0 con 0x7ff4c8152650 2024-03-06T07:24:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.096+0000 7ff4c5ffb640 1 -- 172.21.15.1:0/196955328 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b8037070 con 0x7ff4c8152650 2024-03-06T07:24:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.096+0000 7ff4c5ffb640 1 -- 172.21.15.1:0/196955328 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8031960 con 0x7ff4c8152650 2024-03-06T07:24:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.097+0000 7ff4cd35c640 1 -- 172.21.15.1:0/196955328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c8152650 msgr2=0x7ff4c8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.097+0000 7ff4cd35c640 1 --2- 172.21.15.1:0/196955328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c8152650 0x7ff4c8155c90 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80097a0 tx=0x7ff4b802b840 comp rx=0 tx=0).stop 2024-03-06T07:24:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.097+0000 7ff4cd35c640 1 -- 172.21.15.1:0/196955328 shutdown_connections 2024-03-06T07:24:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.097+0000 7ff4cd35c640 1 --2- 172.21.15.1:0/196955328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c8152650 0x7ff4c8155c90 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.097+0000 7ff4cd35c640 1 -- 172.21.15.1:0/196955328 >> 172.21.15.1:0/196955328 conn(0x7ff4c8066ee0 msgr2=0x7ff4c80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.097+0000 7ff4cd35c640 1 -- 172.21.15.1:0/196955328 shutdown_connections 2024-03-06T07:24:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.097+0000 7ff4cd35c640 1 -- 172.21.15.1:0/196955328 wait complete. 2024-03-06T07:24:15.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.098+0000 7ff4cd35c640 1 Processor -- start 2024-03-06T07:24:15.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.098+0000 7ff4cd35c640 1 -- start start 2024-03-06T07:24:15.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.099+0000 7ff4cd35c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c80bf9d0 0x7ff4c8151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.099+0000 7ff4cd35c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c8152650 0x7ff4c8154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.099+0000 7ff4cd35c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c814fa80 0x7ff4c814b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.099+0000 7ff4cd35c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c81551b0 con 0x7ff4c8152650 2024-03-06T07:24:15.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.099+0000 7ff4cd35c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c80c6fa0 con 0x7ff4c80bf9d0 2024-03-06T07:24:15.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.099+0000 7ff4cd35c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c8155490 con 0x7ff4c814fa80 2024-03-06T07:24:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.099+0000 7ff4c67fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c8152650 0x7ff4c8154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.099+0000 7ff4c67fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c8152650 0x7ff4c8154880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44292/0 (socket says 172.21.15.1:44292) 2024-03-06T07:24:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.099+0000 7ff4c67fc640 1 -- 172.21.15.1:0/2289633267 learned_addr learned my addr 172.21.15.1:0/2289633267 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.099+0000 7ff4c6ffd640 1 --2- 172.21.15.1:0/2289633267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c80bf9d0 0x7ff4c8151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.099+0000 7ff4c77fe640 1 --2- 172.21.15.1:0/2289633267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c814fa80 0x7ff4c814b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.099+0000 7ff4c67fc640 1 -- 172.21.15.1:0/2289633267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c814fa80 msgr2=0x7ff4c814b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.099+0000 7ff4c67fc640 1 --2- 172.21.15.1:0/2289633267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c814fa80 0x7ff4c814b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.099+0000 7ff4c67fc640 1 -- 172.21.15.1:0/2289633267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c80bf9d0 msgr2=0x7ff4c8151410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.099+0000 7ff4c67fc640 1 --2- 172.21.15.1:0/2289633267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c80bf9d0 0x7ff4c8151410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.099+0000 7ff4c67fc640 1 -- 172.21.15.1:0/2289633267 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c81ef5a0 con 0x7ff4c8152650 2024-03-06T07:24:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.100+0000 7ff4c67fc640 1 --2- 172.21.15.1:0/2289633267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c8152650 0x7ff4c8154880 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00c4d0 tx=0x7ff4bc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.100+0000 7ff4af7fe640 1 -- 172.21.15.1:0/2289633267 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc018070 con 0x7ff4c8152650 2024-03-06T07:24:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.100+0000 7ff4af7fe640 1 -- 172.21.15.1:0/2289633267 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4bc007500 con 0x7ff4c8152650 2024-03-06T07:24:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.100+0000 7ff4af7fe640 1 -- 172.21.15.1:0/2289633267 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc004a20 con 0x7ff4c8152650 2024-03-06T07:24:15.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.100+0000 7ff4cd35c640 1 -- 172.21.15.1:0/2289633267 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c8151b30 con 0x7ff4c8152650 2024-03-06T07:24:15.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.100+0000 7ff4cd35c640 1 -- 172.21.15.1:0/2289633267 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c81eeac0 con 0x7ff4c8152650 2024-03-06T07:24:15.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.101+0000 7ff4cd35c640 1 -- 172.21.15.1:0/2289633267 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c80c2f50 con 0x7ff4c8152650 2024-03-06T07:24:15.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.101+0000 7ff4af7fe640 1 -- 172.21.15.1:0/2289633267 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4bc01b070 con 0x7ff4c8152650 2024-03-06T07:24:15.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.102+0000 7ff4af7fe640 1 --2- 172.21.15.1:0/2289633267 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4b406b4d0 0x7ff4b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.102+0000 7ff4af7fe640 1 -- 172.21.15.1:0/2289633267 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff4bc0896a0 con 0x7ff4c8152650 2024-03-06T07:24:15.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.102+0000 7ff4c6ffd640 1 --2- 172.21.15.1:0/2289633267 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4b406b4d0 0x7ff4b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.102+0000 7ff4c6ffd640 1 --2- 172.21.15.1:0/2289633267 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4b406b4d0 0x7ff4b406d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8036000 tx=0x7ff4b8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.106+0000 7ff4af7fe640 1 -- 172.21.15.1:0/2289633267 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4c80c2f50 con 0x7ff4c8152650 2024-03-06T07:24:15.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.283+0000 7ff4cd35c640 1 -- 172.21.15.1:0/2289633267 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4c8067fd0 con 0x7ff4b406b4d0 2024-03-06T07:24:15.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.290+0000 7ff4af7fe640 1 -- 172.21.15.1:0/2289633267 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252565 (secure 0 0 0) 0x7ff4c8067fd0 con 0x7ff4b406b4d0 2024-03-06T07:24:15.294 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:15.296 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:24:15.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.293+0000 7ff4cd35c640 1 -- 172.21.15.1:0/2289633267 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4b406b4d0 msgr2=0x7ff4b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.293+0000 7ff4cd35c640 1 --2- 172.21.15.1:0/2289633267 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4b406b4d0 0x7ff4b406d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8036000 tx=0x7ff4b8008030 comp rx=0 tx=0).stop 2024-03-06T07:24:15.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.293+0000 7ff4cd35c640 1 -- 172.21.15.1:0/2289633267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c8152650 msgr2=0x7ff4c8154880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.293+0000 7ff4cd35c640 1 --2- 172.21.15.1:0/2289633267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c8152650 0x7ff4c8154880 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00c4d0 tx=0x7ff4bc00c990 comp rx=0 tx=0).stop 2024-03-06T07:24:15.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.293+0000 7ff4cd35c640 1 -- 172.21.15.1:0/2289633267 shutdown_connections 2024-03-06T07:24:15.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.293+0000 7ff4cd35c640 1 --2- 172.21.15.1:0/2289633267 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4b406b4d0 0x7ff4b406d990 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.293+0000 7ff4cd35c640 1 --2- 172.21.15.1:0/2289633267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c814fa80 0x7ff4c814b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.293+0000 7ff4cd35c640 1 --2- 172.21.15.1:0/2289633267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c8152650 0x7ff4c8154880 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.293+0000 7ff4cd35c640 1 --2- 172.21.15.1:0/2289633267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c80bf9d0 0x7ff4c8151410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.293+0000 7ff4cd35c640 1 -- 172.21.15.1:0/2289633267 >> 172.21.15.1:0/2289633267 conn(0x7ff4c8066ee0 msgr2=0x7ff4c80cac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.293+0000 7ff4cd35c640 1 -- 172.21.15.1:0/2289633267 shutdown_connections 2024-03-06T07:24:15.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:15.293+0000 7ff4cd35c640 1 -- 172.21.15.1:0/2289633267 wait complete. 2024-03-06T07:24:15.306 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-03-06T07:24:14.743982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260188,"kb_used_data":10720,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714532,"statfs":{"total":773094113280,"available":772827680768,"internally_reserved":0,"allocated":10977280,"data_stored":6388996,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.899987"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871645+0000","last_change":"2024-03-06T07:24:10.872255+0000","last_active":"2024-03-06T07:24:11.871645+0000","last_peered":"2024-03-06T07:24:11.871645+0000","last_clean":"2024-03-06T07:24:11.871645+0000","last_became_active":"2024-03-06T07:24:10.871997+0000","last_became_peered":"2024-03-06T07:24:10.871997+0000","last_unstale":"2024-03-06T07:24:11.871645+0000","last_undegraded":"2024-03-06T07:24:11.871645+0000","last_fullsized":"2024-03-06T07:24:11.871645+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:14.096402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870272+0000","last_change":"2024-03-06T07:24:10.887711+0000","last_active":"2024-03-06T07:24:11.870272+0000","last_peered":"2024-03-06T07:24:11.870272+0000","last_clean":"2024-03-06T07:24:11.870272+0000","last_became_active":"2024-03-06T07:24:10.887166+0000","last_became_peered":"2024-03-06T07:24:10.887166+0000","last_unstale":"2024-03-06T07:24:11.870272+0000","last_undegraded":"2024-03-06T07:24:11.870272+0000","last_fullsized":"2024-03-06T07:24:11.870272+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:59.150271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870413+0000","last_change":"2024-03-06T07:24:10.885027+0000","last_active":"2024-03-06T07:24:11.870413+0000","last_peered":"2024-03-06T07:24:11.870413+0000","last_clean":"2024-03-06T07:24:11.870413+0000","last_became_active":"2024-03-06T07:24:10.884742+0000","last_became_peered":"2024-03-06T07:24:10.884742+0000","last_unstale":"2024-03-06T07:24:11.870413+0000","last_undegraded":"2024-03-06T07:24:11.870413+0000","last_fullsized":"2024-03-06T07:24:11.870413+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:06.270087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435166+0000","last_change":"2024-03-06T07:24:10.872345+0000","last_active":"2024-03-06T07:24:12.435166+0000","last_peered":"2024-03-06T07:24:12.435166+0000","last_clean":"2024-03-06T07:24:12.435166+0000","last_became_active":"2024-03-06T07:24:10.872070+0000","last_became_peered":"2024-03-06T07:24:10.872070+0000","last_unstale":"2024-03-06T07:24:12.435166+0000","last_undegraded":"2024-03-06T07:24:12.435166+0000","last_fullsized":"2024-03-06T07:24:12.435166+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:08.047230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871133+0000","last_change":"2024-03-06T07:24:10.875833+0000","last_active":"2024-03-06T07:24:11.871133+0000","last_peered":"2024-03-06T07:24:11.871133+0000","last_clean":"2024-03-06T07:24:11.871133+0000","last_became_active":"2024-03-06T07:24:10.874654+0000","last_became_peered":"2024-03-06T07:24:10.874654+0000","last_unstale":"2024-03-06T07:24:11.871133+0000","last_undegraded":"2024-03-06T07:24:11.871133+0000","last_fullsized":"2024-03-06T07:24:11.871133+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:16.962306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706910+0000","last_change":"2024-03-06T07:24:10.879327+0000","last_active":"2024-03-06T07:24:12.706910+0000","last_peered":"2024-03-06T07:24:12.706910+0000","last_clean":"2024-03-06T07:24:12.706910+0000","last_became_active":"2024-03-06T07:24:10.879066+0000","last_became_peered":"2024-03-06T07:24:10.879066+0000","last_unstale":"2024-03-06T07:24:12.706910+0000","last_undegraded":"2024-03-06T07:24:12.706910+0000","last_fullsized":"2024-03-06T07:24:12.706910+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:26.886161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706541+0000","last_change":"2024-03-06T07:24:10.874658+0000","last_active":"2024-03-06T07:24:12.706541+0000","last_peered":"2024-03-06T07:24:12.706541+0000","last_clean":"2024-03-06T07:24:12.706541+0000","last_became_active":"2024-03-06T07:24:10.874409+0000","last_became_peered":"2024-03-06T07:24:10.874409+0000","last_unstale":"2024-03-06T07:24:12.706541+0000","last_undegraded":"2024-03-06T07:24:12.706541+0000","last_fullsized":"2024-03-06T07:24:12.706541+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:09:40.994943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.869960+0000","last_change":"2024-03-06T07:24:10.888269+0000","last_active":"2024-03-06T07:24:11.869960+0000","last_peered":"2024-03-06T07:24:11.869960+0000","last_clean":"2024-03-06T07:24:11.869960+0000","last_became_active":"2024-03-06T07:24:10.887087+0000","last_became_peered":"2024-03-06T07:24:10.887087+0000","last_unstale":"2024-03-06T07:24:11.869960+0000","last_undegraded":"2024-03-06T07:24:11.869960+0000","last_fullsized":"2024-03-06T07:24:11.869960+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:41.969138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435403+0000","last_change":"2024-03-06T07:24:10.880426+0000","last_active":"2024-03-06T07:24:12.435403+0000","last_peered":"2024-03-06T07:24:12.435403+0000","last_clean":"2024-03-06T07:24:12.435403+0000","last_became_active":"2024-03-06T07:24:10.880164+0000","last_became_peered":"2024-03-06T07:24:10.880164+0000","last_unstale":"2024-03-06T07:24:12.435403+0000","last_undegraded":"2024-03-06T07:24:12.435403+0000","last_fullsized":"2024-03-06T07:24:12.435403+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:02.467158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706778+0000","last_change":"2024-03-06T07:24:10.873861+0000","last_active":"2024-03-06T07:24:12.706778+0000","last_peered":"2024-03-06T07:24:12.706778+0000","last_clean":"2024-03-06T07:24:12.706778+0000","last_became_active":"2024-03-06T07:24:10.873603+0000","last_became_peered":"2024-03-06T07:24:10.873603+0000","last_unstale":"2024-03-06T07:24:12.706778+0000","last_undegraded":"2024-03-06T07:24:12.706778+0000","last_fullsized":"2024-03-06T07:24:12.706778+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:14.096402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871222+0000","last_change":"2024-03-06T07:24:10.877379+0000","last_active":"2024-03-06T07:24:11.871222+0000","last_peered":"2024-03-06T07:24:11.871222+0000","last_clean":"2024-03-06T07:24:11.871222+0000","last_became_active":"2024-03-06T07:24:10.877118+0000","last_became_peered":"2024-03-06T07:24:10.877118+0000","last_unstale":"2024-03-06T07:24:11.871222+0000","last_undegraded":"2024-03-06T07:24:11.871222+0000","last_fullsized":"2024-03-06T07:24:11.871222+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:12.210056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707202+0000","last_change":"2024-03-06T07:24:10.883581+0000","last_active":"2024-03-06T07:24:12.707202+0000","last_peered":"2024-03-06T07:24:12.707202+0000","last_clean":"2024-03-06T07:24:12.707202+0000","last_became_active":"2024-03-06T07:24:10.883304+0000","last_became_peered":"2024-03-06T07:24:10.883304+0000","last_unstale":"2024-03-06T07:24:12.707202+0000","last_undegraded":"2024-03-06T07:24:12.707202+0000","last_fullsized":"2024-03-06T07:24:12.707202+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:59.901346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706425+0000","last_change":"2024-03-06T07:24:10.873286+0000","last_active":"2024-03-06T07:24:12.706425+0000","last_peered":"2024-03-06T07:24:12.706425+0000","last_clean":"2024-03-06T07:24:12.706425+0000","last_became_active":"2024-03-06T07:24:10.871844+0000","last_became_peered":"2024-03-06T07:24:10.871844+0000","last_unstale":"2024-03-06T07:24:12.706425+0000","last_undegraded":"2024-03-06T07:24:12.706425+0000","last_fullsized":"2024-03-06T07:24:12.706425+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:13.966188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870116+0000","last_change":"2024-03-06T07:24:10.884278+0000","last_active":"2024-03-06T07:24:11.870116+0000","last_peered":"2024-03-06T07:24:11.870116+0000","last_clean":"2024-03-06T07:24:11.870116+0000","last_became_active":"2024-03-06T07:24:10.884006+0000","last_became_peered":"2024-03-06T07:24:10.884006+0000","last_unstale":"2024-03-06T07:24:11.870116+0000","last_undegraded":"2024-03-06T07:24:11.870116+0000","last_fullsized":"2024-03-06T07:24:11.870116+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:47.528123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871242+0000","last_change":"2024-03-06T07:24:10.878061+0000","last_active":"2024-03-06T07:24:11.871242+0000","last_peered":"2024-03-06T07:24:11.871242+0000","last_clean":"2024-03-06T07:24:11.871242+0000","last_became_active":"2024-03-06T07:24:10.877783+0000","last_became_peered":"2024-03-06T07:24:10.877783+0000","last_unstale":"2024-03-06T07:24:11.871242+0000","last_undegraded":"2024-03-06T07:24:11.871242+0000","last_fullsized":"2024-03-06T07:24:11.871242+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:49.026769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706428+0000","last_change":"2024-03-06T07:24:10.869634+0000","last_active":"2024-03-06T07:24:12.706428+0000","last_peered":"2024-03-06T07:24:12.706428+0000","last_clean":"2024-03-06T07:24:12.706428+0000","last_became_active":"2024-03-06T07:24:10.869360+0000","last_became_peered":"2024-03-06T07:24:10.869360+0000","last_unstale":"2024-03-06T07:24:12.706428+0000","last_undegraded":"2024-03-06T07:24:12.706428+0000","last_fullsized":"2024-03-06T07:24:12.706428+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:16.399326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706688+0000","last_change":"2024-03-06T07:24:10.874469+0000","last_active":"2024-03-06T07:24:12.706688+0000","last_peered":"2024-03-06T07:24:12.706688+0000","last_clean":"2024-03-06T07:24:12.706688+0000","last_became_active":"2024-03-06T07:24:10.874095+0000","last_became_peered":"2024-03-06T07:24:10.874095+0000","last_unstale":"2024-03-06T07:24:12.706688+0000","last_undegraded":"2024-03-06T07:24:12.706688+0000","last_fullsized":"2024-03-06T07:24:12.706688+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:06.797882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870253+0000","last_change":"2024-03-06T07:24:10.887496+0000","last_active":"2024-03-06T07:24:11.870253+0000","last_peered":"2024-03-06T07:24:11.870253+0000","last_clean":"2024-03-06T07:24:11.870253+0000","last_became_active":"2024-03-06T07:24:10.887039+0000","last_became_peered":"2024-03-06T07:24:10.887039+0000","last_unstale":"2024-03-06T07:24:11.870253+0000","last_undegraded":"2024-03-06T07:24:11.870253+0000","last_fullsized":"2024-03-06T07:24:11.870253+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:04.010668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707388+0000","last_change":"2024-03-06T07:24:10.869573+0000","last_active":"2024-03-06T07:24:12.707388+0000","last_peered":"2024-03-06T07:24:12.707388+0000","last_clean":"2024-03-06T07:24:12.707388+0000","last_became_active":"2024-03-06T07:24:10.869311+0000","last_became_peered":"2024-03-06T07:24:10.869311+0000","last_unstale":"2024-03-06T07:24:12.707388+0000","last_undegraded":"2024-03-06T07:24:12.707388+0000","last_fullsized":"2024-03-06T07:24:12.707388+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:09.133114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.867384+0000","last_change":"2024-03-06T07:24:10.871671+0000","last_active":"2024-03-06T07:24:11.867384+0000","last_peered":"2024-03-06T07:24:11.867384+0000","last_clean":"2024-03-06T07:24:11.867384+0000","last_became_active":"2024-03-06T07:24:10.871411+0000","last_became_peered":"2024-03-06T07:24:10.871411+0000","last_unstale":"2024-03-06T07:24:11.867384+0000","last_undegraded":"2024-03-06T07:24:11.867384+0000","last_fullsized":"2024-03-06T07:24:11.867384+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:08.340512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435032+0000","last_change":"2024-03-06T07:24:10.868119+0000","last_active":"2024-03-06T07:24:12.435032+0000","last_peered":"2024-03-06T07:24:12.435032+0000","last_clean":"2024-03-06T07:24:12.435032+0000","last_became_active":"2024-03-06T07:24:10.867820+0000","last_became_peered":"2024-03-06T07:24:10.867820+0000","last_unstale":"2024-03-06T07:24:12.435032+0000","last_undegraded":"2024-03-06T07:24:12.435032+0000","last_fullsized":"2024-03-06T07:24:12.435032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:15:23.097231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871551+0000","last_change":"2024-03-06T07:24:10.879942+0000","last_active":"2024-03-06T07:24:11.871551+0000","last_peered":"2024-03-06T07:24:11.871551+0000","last_clean":"2024-03-06T07:24:11.871551+0000","last_became_active":"2024-03-06T07:24:10.879691+0000","last_became_peered":"2024-03-06T07:24:10.879691+0000","last_unstale":"2024-03-06T07:24:11.871551+0000","last_undegraded":"2024-03-06T07:24:11.871551+0000","last_fullsized":"2024-03-06T07:24:11.871551+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:55.863177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.434926+0000","last_change":"2024-03-06T07:24:10.881411+0000","last_active":"2024-03-06T07:24:12.434926+0000","last_peered":"2024-03-06T07:24:12.434926+0000","last_clean":"2024-03-06T07:24:12.434926+0000","last_became_active":"2024-03-06T07:24:10.879424+0000","last_became_peered":"2024-03-06T07:24:10.879424+0000","last_unstale":"2024-03-06T07:24:12.434926+0000","last_undegraded":"2024-03-06T07:24:12.434926+0000","last_fullsized":"2024-03-06T07:24:12.434926+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:36.612944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706892+0000","last_change":"2024-03-06T07:24:10.872049+0000","last_active":"2024-03-06T07:24:12.706892+0000","last_peered":"2024-03-06T07:24:12.706892+0000","last_clean":"2024-03-06T07:24:12.706892+0000","last_became_active":"2024-03-06T07:24:10.871719+0000","last_became_peered":"2024-03-06T07:24:10.871719+0000","last_unstale":"2024-03-06T07:24:12.706892+0000","last_undegraded":"2024-03-06T07:24:12.706892+0000","last_fullsized":"2024-03-06T07:24:12.706892+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:21.464618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707914+0000","last_change":"2024-03-06T07:24:10.869734+0000","last_active":"2024-03-06T07:24:12.707914+0000","last_peered":"2024-03-06T07:24:12.707914+0000","last_clean":"2024-03-06T07:24:12.707914+0000","last_became_active":"2024-03-06T07:24:10.869325+0000","last_became_peered":"2024-03-06T07:24:10.869325+0000","last_unstale":"2024-03-06T07:24:12.707914+0000","last_undegraded":"2024-03-06T07:24:12.707914+0000","last_fullsized":"2024-03-06T07:24:12.707914+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:06.797882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.867410+0000","last_change":"2024-03-06T07:24:10.865909+0000","last_active":"2024-03-06T07:24:11.867410+0000","last_peered":"2024-03-06T07:24:11.867410+0000","last_clean":"2024-03-06T07:24:11.867410+0000","last_became_active":"2024-03-06T07:24:10.865638+0000","last_became_peered":"2024-03-06T07:24:10.865638+0000","last_unstale":"2024-03-06T07:24:11.867410+0000","last_undegraded":"2024-03-06T07:24:11.867410+0000","last_fullsized":"2024-03-06T07:24:11.867410+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:37.759423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707407+0000","last_change":"2024-03-06T07:24:10.869283+0000","last_active":"2024-03-06T07:24:12.707407+0000","last_peered":"2024-03-06T07:24:12.707407+0000","last_clean":"2024-03-06T07:24:12.707407+0000","last_became_active":"2024-03-06T07:24:10.869005+0000","last_became_peered":"2024-03-06T07:24:10.869005+0000","last_unstale":"2024-03-06T07:24:12.707407+0000","last_undegraded":"2024-03-06T07:24:12.707407+0000","last_fullsized":"2024-03-06T07:24:12.707407+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:39.447002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707262+0000","last_change":"2024-03-06T07:24:10.877332+0000","last_active":"2024-03-06T07:24:12.707262+0000","last_peered":"2024-03-06T07:24:12.707262+0000","last_clean":"2024-03-06T07:24:12.707262+0000","last_became_active":"2024-03-06T07:24:10.876975+0000","last_became_peered":"2024-03-06T07:24:10.876975+0000","last_unstale":"2024-03-06T07:24:12.707262+0000","last_undegraded":"2024-03-06T07:24:12.707262+0000","last_fullsized":"2024-03-06T07:24:12.707262+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:22.153347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.867482+0000","last_change":"2024-03-06T07:24:10.878173+0000","last_active":"2024-03-06T07:24:11.867482+0000","last_peered":"2024-03-06T07:24:11.867482+0000","last_clean":"2024-03-06T07:24:11.867482+0000","last_became_active":"2024-03-06T07:24:10.877825+0000","last_became_peered":"2024-03-06T07:24:10.877825+0000","last_unstale":"2024-03-06T07:24:11.867482+0000","last_undegraded":"2024-03-06T07:24:11.867482+0000","last_fullsized":"2024-03-06T07:24:11.867482+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:04.191625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870217+0000","last_change":"2024-03-06T07:24:10.882571+0000","last_active":"2024-03-06T07:24:11.870217+0000","last_peered":"2024-03-06T07:24:11.870217+0000","last_clean":"2024-03-06T07:24:11.870217+0000","last_became_active":"2024-03-06T07:24:10.882244+0000","last_became_peered":"2024-03-06T07:24:10.882244+0000","last_unstale":"2024-03-06T07:24:11.870217+0000","last_undegraded":"2024-03-06T07:24:11.870217+0000","last_fullsized":"2024-03-06T07:24:11.870217+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:42.857654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707336+0000","last_change":"2024-03-06T07:24:10.881361+0000","last_active":"2024-03-06T07:24:12.707336+0000","last_peered":"2024-03-06T07:24:12.707336+0000","last_clean":"2024-03-06T07:24:12.707336+0000","last_became_active":"2024-03-06T07:24:10.879117+0000","last_became_peered":"2024-03-06T07:24:10.879117+0000","last_unstale":"2024-03-06T07:24:12.707336+0000","last_undegraded":"2024-03-06T07:24:12.707336+0000","last_fullsized":"2024-03-06T07:24:12.707336+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:18.273527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707128+0000","last_change":"2024-03-06T07:24:10.871229+0000","last_active":"2024-03-06T07:24:12.707128+0000","last_peered":"2024-03-06T07:24:12.707128+0000","last_clean":"2024-03-06T07:24:12.707128+0000","last_became_active":"2024-03-06T07:24:10.870824+0000","last_became_peered":"2024-03-06T07:24:10.870824+0000","last_unstale":"2024-03-06T07:24:12.707128+0000","last_undegraded":"2024-03-06T07:24:12.707128+0000","last_fullsized":"2024-03-06T07:24:12.707128+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:27.910318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706869+0000","last_change":"2024-03-06T07:24:10.873797+0000","last_active":"2024-03-06T07:24:12.706869+0000","last_peered":"2024-03-06T07:24:12.706869+0000","last_clean":"2024-03-06T07:24:12.706869+0000","last_became_active":"2024-03-06T07:24:10.873518+0000","last_became_peered":"2024-03-06T07:24:10.873518+0000","last_unstale":"2024-03-06T07:24:12.706869+0000","last_undegraded":"2024-03-06T07:24:12.706869+0000","last_fullsized":"2024-03-06T07:24:12.706869+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:48.576719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"107'4","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706897+0000","last_change":"2024-03-06T07:24:09.866650+0000","last_active":"2024-03-06T07:24:12.706897+0000","last_peered":"2024-03-06T07:24:12.706897+0000","last_clean":"2024-03-06T07:24:12.706897+0000","last_became_active":"2024-03-06T07:24:09.866357+0000","last_became_peered":"2024-03-06T07:24:09.866357+0000","last_unstale":"2024-03-06T07:24:12.706897+0000","last_undegraded":"2024-03-06T07:24:12.706897+0000","last_fullsized":"2024-03-06T07:24:12.706897+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:56.308026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706552+0000","last_change":"2024-03-06T07:24:10.879050+0000","last_active":"2024-03-06T07:24:12.706552+0000","last_peered":"2024-03-06T07:24:12.706552+0000","last_clean":"2024-03-06T07:24:12.706552+0000","last_became_active":"2024-03-06T07:24:10.878772+0000","last_became_peered":"2024-03-06T07:24:10.878772+0000","last_unstale":"2024-03-06T07:24:12.706552+0000","last_undegraded":"2024-03-06T07:24:12.706552+0000","last_fullsized":"2024-03-06T07:24:12.706552+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:33.944045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706467+0000","last_change":"2024-03-06T07:24:09.851704+0000","last_active":"2024-03-06T07:24:12.706467+0000","last_peered":"2024-03-06T07:24:12.706467+0000","last_clean":"2024-03-06T07:24:12.706467+0000","last_became_active":"2024-03-06T07:24:09.851423+0000","last_became_peered":"2024-03-06T07:24:09.851423+0000","last_unstale":"2024-03-06T07:24:12.706467+0000","last_undegraded":"2024-03-06T07:24:12.706467+0000","last_fullsized":"2024-03-06T07:24:12.706467+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:14.500408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707794+0000","last_change":"2024-03-06T07:24:10.882779+0000","last_active":"2024-03-06T07:24:12.707794+0000","last_peered":"2024-03-06T07:24:12.707794+0000","last_clean":"2024-03-06T07:24:12.707794+0000","last_became_active":"2024-03-06T07:24:10.881271+0000","last_became_peered":"2024-03-06T07:24:10.881271+0000","last_unstale":"2024-03-06T07:24:12.707794+0000","last_undegraded":"2024-03-06T07:24:12.707794+0000","last_fullsized":"2024-03-06T07:24:12.707794+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:49.026769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435140+0000","last_change":"2024-03-06T07:24:09.865828+0000","last_active":"2024-03-06T07:24:12.435140+0000","last_peered":"2024-03-06T07:24:12.435140+0000","last_clean":"2024-03-06T07:24:12.435140+0000","last_became_active":"2024-03-06T07:24:09.865126+0000","last_became_peered":"2024-03-06T07:24:09.865126+0000","last_unstale":"2024-03-06T07:24:12.435140+0000","last_undegraded":"2024-03-06T07:24:12.435140+0000","last_fullsized":"2024-03-06T07:24:12.435140+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:15.958341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435413+0000","last_change":"2024-03-06T07:24:10.868169+0000","last_active":"2024-03-06T07:24:12.435413+0000","last_peered":"2024-03-06T07:24:12.435413+0000","last_clean":"2024-03-06T07:24:12.435413+0000","last_became_active":"2024-03-06T07:24:10.867842+0000","last_became_peered":"2024-03-06T07:24:10.867842+0000","last_unstale":"2024-03-06T07:24:12.435413+0000","last_undegraded":"2024-03-06T07:24:12.435413+0000","last_fullsized":"2024-03-06T07:24:12.435413+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:23:18.286691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706584+0000","last_change":"2024-03-06T07:24:09.862873+0000","last_active":"2024-03-06T07:24:12.706584+0000","last_peered":"2024-03-06T07:24:12.706584+0000","last_clean":"2024-03-06T07:24:12.706584+0000","last_became_active":"2024-03-06T07:24:09.862595+0000","last_became_peered":"2024-03-06T07:24:09.862595+0000","last_unstale":"2024-03-06T07:24:12.706584+0000","last_undegraded":"2024-03-06T07:24:12.706584+0000","last_fullsized":"2024-03-06T07:24:12.706584+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:46.994899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706301+0000","last_change":"2024-03-06T07:24:10.871344+0000","last_active":"2024-03-06T07:24:12.706301+0000","last_peered":"2024-03-06T07:24:12.706301+0000","last_clean":"2024-03-06T07:24:12.706301+0000","last_became_active":"2024-03-06T07:24:10.871056+0000","last_became_peered":"2024-03-06T07:24:10.871056+0000","last_unstale":"2024-03-06T07:24:12.706301+0000","last_undegraded":"2024-03-06T07:24:12.706301+0000","last_fullsized":"2024-03-06T07:24:12.706301+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:18.273527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871187+0000","last_change":"2024-03-06T07:24:09.861836+0000","last_active":"2024-03-06T07:24:11.871187+0000","last_peered":"2024-03-06T07:24:11.871187+0000","last_clean":"2024-03-06T07:24:11.871187+0000","last_became_active":"2024-03-06T07:24:09.861533+0000","last_became_peered":"2024-03-06T07:24:09.861533+0000","last_unstale":"2024-03-06T07:24:11.871187+0000","last_undegraded":"2024-03-06T07:24:11.871187+0000","last_fullsized":"2024-03-06T07:24:11.871187+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:42.678582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871180+0000","last_change":"2024-03-06T07:24:10.877007+0000","last_active":"2024-03-06T07:24:11.871180+0000","last_peered":"2024-03-06T07:24:11.871180+0000","last_clean":"2024-03-06T07:24:11.871180+0000","last_became_active":"2024-03-06T07:24:10.876741+0000","last_became_peered":"2024-03-06T07:24:10.876741+0000","last_unstale":"2024-03-06T07:24:11.871180+0000","last_undegraded":"2024-03-06T07:24:11.871180+0000","last_fullsized":"2024-03-06T07:24:11.871180+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:13.966188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706950+0000","last_change":"2024-03-06T07:24:09.861809+0000","last_active":"2024-03-06T07:24:12.706950+0000","last_peered":"2024-03-06T07:24:12.706950+0000","last_clean":"2024-03-06T07:24:12.706950+0000","last_became_active":"2024-03-06T07:24:09.861479+0000","last_became_peered":"2024-03-06T07:24:09.861479+0000","last_unstale":"2024-03-06T07:24:12.706950+0000","last_undegraded":"2024-03-06T07:24:12.706950+0000","last_fullsized":"2024-03-06T07:24:12.706950+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:59.826052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870314+0000","last_change":"2024-03-06T07:24:10.877845+0000","last_active":"2024-03-06T07:24:11.870314+0000","last_peered":"2024-03-06T07:24:11.870314+0000","last_clean":"2024-03-06T07:24:11.870314+0000","last_became_active":"2024-03-06T07:24:10.877548+0000","last_became_peered":"2024-03-06T07:24:10.877548+0000","last_unstale":"2024-03-06T07:24:11.870314+0000","last_undegraded":"2024-03-06T07:24:11.870314+0000","last_fullsized":"2024-03-06T07:24:11.870314+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:00.925482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707341+0000","last_change":"2024-03-06T07:24:09.863436+0000","last_active":"2024-03-06T07:24:12.707341+0000","last_peered":"2024-03-06T07:24:12.707341+0000","last_clean":"2024-03-06T07:24:12.707341+0000","last_became_active":"2024-03-06T07:24:09.863108+0000","last_became_peered":"2024-03-06T07:24:09.863108+0000","last_unstale":"2024-03-06T07:24:12.707341+0000","last_undegraded":"2024-03-06T07:24:12.707341+0000","last_fullsized":"2024-03-06T07:24:12.707341+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:19.077946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706562+0000","last_change":"2024-03-06T07:24:10.878783+0000","last_active":"2024-03-06T07:24:12.706562+0000","last_peered":"2024-03-06T07:24:12.706562+0000","last_clean":"2024-03-06T07:24:12.706562+0000","last_became_active":"2024-03-06T07:24:10.877971+0000","last_became_peered":"2024-03-06T07:24:10.877971+0000","last_unstale":"2024-03-06T07:24:12.706562+0000","last_undegraded":"2024-03-06T07:24:12.706562+0000","last_fullsized":"2024-03-06T07:24:12.706562+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:49.026769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706889+0000","last_change":"2024-03-06T07:24:09.865101+0000","last_active":"2024-03-06T07:24:12.706889+0000","last_peered":"2024-03-06T07:24:12.706889+0000","last_clean":"2024-03-06T07:24:12.706889+0000","last_became_active":"2024-03-06T07:24:09.864768+0000","last_became_peered":"2024-03-06T07:24:09.864768+0000","last_unstale":"2024-03-06T07:24:12.706889+0000","last_undegraded":"2024-03-06T07:24:12.706889+0000","last_fullsized":"2024-03-06T07:24:12.706889+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:39.734895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707173+0000","last_change":"2024-03-06T07:24:10.876317+0000","last_active":"2024-03-06T07:24:12.707173+0000","last_peered":"2024-03-06T07:24:12.707173+0000","last_clean":"2024-03-06T07:24:12.707173+0000","last_became_active":"2024-03-06T07:24:10.876038+0000","last_became_peered":"2024-03-06T07:24:10.876038+0000","last_unstale":"2024-03-06T07:24:12.707173+0000","last_undegraded":"2024-03-06T07:24:12.707173+0000","last_fullsized":"2024-03-06T07:24:12.707173+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:43.682548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707067+0000","last_change":"2024-03-06T07:24:09.868915+0000","last_active":"2024-03-06T07:24:12.707067+0000","last_peered":"2024-03-06T07:24:12.707067+0000","last_clean":"2024-03-06T07:24:12.707067+0000","last_became_active":"2024-03-06T07:24:09.868646+0000","last_became_peered":"2024-03-06T07:24:09.868646+0000","last_unstale":"2024-03-06T07:24:12.707067+0000","last_undegraded":"2024-03-06T07:24:12.707067+0000","last_fullsized":"2024-03-06T07:24:12.707067+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:16.002346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707732+0000","last_change":"2024-03-06T07:24:10.884248+0000","last_active":"2024-03-06T07:24:12.707732+0000","last_peered":"2024-03-06T07:24:12.707732+0000","last_clean":"2024-03-06T07:24:12.707732+0000","last_became_active":"2024-03-06T07:24:10.883814+0000","last_became_peered":"2024-03-06T07:24:10.883814+0000","last_unstale":"2024-03-06T07:24:12.707732+0000","last_undegraded":"2024-03-06T07:24:12.707732+0000","last_fullsized":"2024-03-06T07:24:12.707732+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:12.210056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707672+0000","last_change":"2024-03-06T07:24:09.863228+0000","last_active":"2024-03-06T07:24:12.707672+0000","last_peered":"2024-03-06T07:24:12.707672+0000","last_clean":"2024-03-06T07:24:12.707672+0000","last_became_active":"2024-03-06T07:24:09.862939+0000","last_became_peered":"2024-03-06T07:24:09.862939+0000","last_unstale":"2024-03-06T07:24:12.707672+0000","last_undegraded":"2024-03-06T07:24:12.707672+0000","last_fullsized":"2024-03-06T07:24:12.707672+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:16.002346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870346+0000","last_change":"2024-03-06T07:24:10.887763+0000","last_active":"2024-03-06T07:24:11.870346+0000","last_peered":"2024-03-06T07:24:11.870346+0000","last_clean":"2024-03-06T07:24:11.870346+0000","last_became_active":"2024-03-06T07:24:10.887434+0000","last_became_peered":"2024-03-06T07:24:10.887434+0000","last_unstale":"2024-03-06T07:24:11.870346+0000","last_undegraded":"2024-03-06T07:24:11.870346+0000","last_fullsized":"2024-03-06T07:24:11.870346+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:12.441745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"107'3","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707626+0000","last_change":"2024-03-06T07:24:09.865500+0000","last_active":"2024-03-06T07:24:12.707626+0000","last_peered":"2024-03-06T07:24:12.707626+0000","last_clean":"2024-03-06T07:24:12.707626+0000","last_became_active":"2024-03-06T07:24:09.865216+0000","last_became_peered":"2024-03-06T07:24:09.865216+0000","last_unstale":"2024-03-06T07:24:12.707626+0000","last_undegraded":"2024-03-06T07:24:12.707626+0000","last_fullsized":"2024-03-06T07:24:12.707626+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:56.308026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707911+0000","last_change":"2024-03-06T07:24:10.878231+0000","last_active":"2024-03-06T07:24:12.707911+0000","last_peered":"2024-03-06T07:24:12.707911+0000","last_clean":"2024-03-06T07:24:12.707911+0000","last_became_active":"2024-03-06T07:24:10.876336+0000","last_became_peered":"2024-03-06T07:24:10.876336+0000","last_unstale":"2024-03-06T07:24:12.707911+0000","last_undegraded":"2024-03-06T07:24:12.707911+0000","last_fullsized":"2024-03-06T07:24:12.707911+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:55.863177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"107'6","reported_seq":20,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870113+0000","last_change":"2024-03-06T07:24:09.862669+0000","last_active":"2024-03-06T07:24:11.870113+0000","last_peered":"2024-03-06T07:24:11.870113+0000","last_clean":"2024-03-06T07:24:11.870113+0000","last_became_active":"2024-03-06T07:24:09.862371+0000","last_became_peered":"2024-03-06T07:24:09.862371+0000","last_unstale":"2024-03-06T07:24:11.870113+0000","last_undegraded":"2024-03-06T07:24:11.870113+0000","last_fullsized":"2024-03-06T07:24:11.870113+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:47.572754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707472+0000","last_change":"2024-03-06T07:24:10.882090+0000","last_active":"2024-03-06T07:24:12.707472+0000","last_peered":"2024-03-06T07:24:12.707472+0000","last_clean":"2024-03-06T07:24:12.707472+0000","last_became_active":"2024-03-06T07:24:10.881824+0000","last_became_peered":"2024-03-06T07:24:10.881824+0000","last_unstale":"2024-03-06T07:24:12.707472+0000","last_undegraded":"2024-03-06T07:24:12.707472+0000","last_fullsized":"2024-03-06T07:24:12.707472+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:15.401455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870146+0000","last_change":"2024-03-06T07:24:09.863186+0000","last_active":"2024-03-06T07:24:11.870146+0000","last_peered":"2024-03-06T07:24:11.870146+0000","last_clean":"2024-03-06T07:24:11.870146+0000","last_became_active":"2024-03-06T07:24:09.862898+0000","last_became_peered":"2024-03-06T07:24:09.862898+0000","last_unstale":"2024-03-06T07:24:11.870146+0000","last_undegraded":"2024-03-06T07:24:11.870146+0000","last_fullsized":"2024-03-06T07:24:11.870146+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:13.092436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706362+0000","last_change":"2024-03-06T07:24:10.863118+0000","last_active":"2024-03-06T07:24:12.706362+0000","last_peered":"2024-03-06T07:24:12.706362+0000","last_clean":"2024-03-06T07:24:12.706362+0000","last_became_active":"2024-03-06T07:24:10.862829+0000","last_became_peered":"2024-03-06T07:24:10.862829+0000","last_unstale":"2024-03-06T07:24:12.706362+0000","last_undegraded":"2024-03-06T07:24:12.706362+0000","last_fullsized":"2024-03-06T07:24:12.706362+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:16.962306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870390+0000","last_change":"2024-03-06T07:24:09.864203+0000","last_active":"2024-03-06T07:24:11.870390+0000","last_peered":"2024-03-06T07:24:11.870390+0000","last_clean":"2024-03-06T07:24:11.870390+0000","last_became_active":"2024-03-06T07:24:09.863923+0000","last_became_peered":"2024-03-06T07:24:09.863923+0000","last_unstale":"2024-03-06T07:24:11.870390+0000","last_undegraded":"2024-03-06T07:24:11.870390+0000","last_fullsized":"2024-03-06T07:24:11.870390+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:28.983380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.867291+0000","last_change":"2024-03-06T07:24:10.874735+0000","last_active":"2024-03-06T07:24:11.867291+0000","last_peered":"2024-03-06T07:24:11.867291+0000","last_clean":"2024-03-06T07:24:11.867291+0000","last_became_active":"2024-03-06T07:24:10.874478+0000","last_became_peered":"2024-03-06T07:24:10.874478+0000","last_unstale":"2024-03-06T07:24:11.867291+0000","last_undegraded":"2024-03-06T07:24:11.867291+0000","last_fullsized":"2024-03-06T07:24:11.867291+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:07.054527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870038+0000","last_change":"2024-03-06T07:24:09.862545+0000","last_active":"2024-03-06T07:24:11.870038+0000","last_peered":"2024-03-06T07:24:11.870038+0000","last_clean":"2024-03-06T07:24:11.870038+0000","last_became_active":"2024-03-06T07:24:09.862257+0000","last_became_peered":"2024-03-06T07:24:09.862257+0000","last_unstale":"2024-03-06T07:24:11.870038+0000","last_undegraded":"2024-03-06T07:24:11.870038+0000","last_fullsized":"2024-03-06T07:24:11.870038+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:29.252821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707654+0000","last_change":"2024-03-06T07:24:10.878777+0000","last_active":"2024-03-06T07:24:12.707654+0000","last_peered":"2024-03-06T07:24:12.707654+0000","last_clean":"2024-03-06T07:24:12.707654+0000","last_became_active":"2024-03-06T07:24:10.878513+0000","last_became_peered":"2024-03-06T07:24:10.878513+0000","last_unstale":"2024-03-06T07:24:12.707654+0000","last_undegraded":"2024-03-06T07:24:12.707654+0000","last_fullsized":"2024-03-06T07:24:12.707654+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:13.966188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706742+0000","last_change":"2024-03-06T07:24:09.868594+0000","last_active":"2024-03-06T07:24:12.706742+0000","last_peered":"2024-03-06T07:24:12.706742+0000","last_clean":"2024-03-06T07:24:12.706742+0000","last_became_active":"2024-03-06T07:24:09.868301+0000","last_became_peered":"2024-03-06T07:24:09.868301+0000","last_unstale":"2024-03-06T07:24:12.706742+0000","last_undegraded":"2024-03-06T07:24:12.706742+0000","last_fullsized":"2024-03-06T07:24:12.706742+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:53.520082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871428+0000","last_change":"2024-03-06T07:24:10.880703+0000","last_active":"2024-03-06T07:24:11.871428+0000","last_peered":"2024-03-06T07:24:11.871428+0000","last_clean":"2024-03-06T07:24:11.871428+0000","last_became_active":"2024-03-06T07:24:10.880396+0000","last_became_peered":"2024-03-06T07:24:10.880396+0000","last_unstale":"2024-03-06T07:24:11.871428+0000","last_undegraded":"2024-03-06T07:24:11.871428+0000","last_fullsized":"2024-03-06T07:24:11.871428+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:06.797882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.867452+0000","last_change":"2024-03-06T07:24:09.856421+0000","last_active":"2024-03-06T07:24:11.867452+0000","last_peered":"2024-03-06T07:24:11.867452+0000","last_clean":"2024-03-06T07:24:11.867452+0000","last_became_active":"2024-03-06T07:24:09.856150+0000","last_became_peered":"2024-03-06T07:24:09.856150+0000","last_unstale":"2024-03-06T07:24:11.867452+0000","last_undegraded":"2024-03-06T07:24:11.867452+0000","last_fullsized":"2024-03-06T07:24:11.867452+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:41.339935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707136+0000","last_change":"2024-03-06T07:24:10.874131+0000","last_active":"2024-03-06T07:24:12.707136+0000","last_peered":"2024-03-06T07:24:12.707136+0000","last_clean":"2024-03-06T07:24:12.707136+0000","last_became_active":"2024-03-06T07:24:10.873870+0000","last_became_peered":"2024-03-06T07:24:10.873870+0000","last_unstale":"2024-03-06T07:24:12.707136+0000","last_undegraded":"2024-03-06T07:24:12.707136+0000","last_fullsized":"2024-03-06T07:24:12.707136+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:26.211626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435259+0000","last_change":"2024-03-06T07:24:09.866590+0000","last_active":"2024-03-06T07:24:12.435259+0000","last_peered":"2024-03-06T07:24:12.435259+0000","last_clean":"2024-03-06T07:24:12.435259+0000","last_became_active":"2024-03-06T07:24:09.866219+0000","last_became_peered":"2024-03-06T07:24:09.866219+0000","last_unstale":"2024-03-06T07:24:12.435259+0000","last_undegraded":"2024-03-06T07:24:12.435259+0000","last_fullsized":"2024-03-06T07:24:12.435259+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:17.269562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707999+0000","last_change":"2024-03-06T07:24:10.868999+0000","last_active":"2024-03-06T07:24:12.707999+0000","last_peered":"2024-03-06T07:24:12.707999+0000","last_clean":"2024-03-06T07:24:12.707999+0000","last_became_active":"2024-03-06T07:24:10.868721+0000","last_became_peered":"2024-03-06T07:24:10.868721+0000","last_unstale":"2024-03-06T07:24:12.707999+0000","last_undegraded":"2024-03-06T07:24:12.707999+0000","last_fullsized":"2024-03-06T07:24:12.707999+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:14.096402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435054+0000","last_change":"2024-03-06T07:24:09.868739+0000","last_active":"2024-03-06T07:24:12.435054+0000","last_peered":"2024-03-06T07:24:12.435054+0000","last_clean":"2024-03-06T07:24:12.435054+0000","last_became_active":"2024-03-06T07:24:09.868468+0000","last_became_peered":"2024-03-06T07:24:09.868468+0000","last_unstale":"2024-03-06T07:24:12.435054+0000","last_undegraded":"2024-03-06T07:24:12.435054+0000","last_fullsized":"2024-03-06T07:24:12.435054+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:26.906353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706946+0000","last_change":"2024-03-06T07:24:10.873557+0000","last_active":"2024-03-06T07:24:12.706946+0000","last_peered":"2024-03-06T07:24:12.706946+0000","last_clean":"2024-03-06T07:24:12.706946+0000","last_became_active":"2024-03-06T07:24:10.873300+0000","last_became_peered":"2024-03-06T07:24:10.873300+0000","last_unstale":"2024-03-06T07:24:12.706946+0000","last_undegraded":"2024-03-06T07:24:12.706946+0000","last_fullsized":"2024-03-06T07:24:12.706946+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:27.985129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706587+0000","last_change":"2024-03-06T07:24:09.866393+0000","last_active":"2024-03-06T07:24:12.706587+0000","last_peered":"2024-03-06T07:24:12.706587+0000","last_clean":"2024-03-06T07:24:12.706587+0000","last_became_active":"2024-03-06T07:24:09.866039+0000","last_became_peered":"2024-03-06T07:24:09.866039+0000","last_unstale":"2024-03-06T07:24:12.706587+0000","last_undegraded":"2024-03-06T07:24:12.706587+0000","last_fullsized":"2024-03-06T07:24:12.706587+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:13.805366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.869979+0000","last_change":"2024-03-06T07:24:10.889993+0000","last_active":"2024-03-06T07:24:11.869979+0000","last_peered":"2024-03-06T07:24:11.869979+0000","last_clean":"2024-03-06T07:24:11.869979+0000","last_became_active":"2024-03-06T07:24:10.889789+0000","last_became_peered":"2024-03-06T07:24:10.889789+0000","last_unstale":"2024-03-06T07:24:11.869979+0000","last_undegraded":"2024-03-06T07:24:11.869979+0000","last_fullsized":"2024-03-06T07:24:11.869979+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:55.733709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.434752+0000","last_change":"2024-03-06T07:24:09.865474+0000","last_active":"2024-03-06T07:24:12.434752+0000","last_peered":"2024-03-06T07:24:12.434752+0000","last_clean":"2024-03-06T07:24:12.434752+0000","last_became_active":"2024-03-06T07:24:09.865141+0000","last_became_peered":"2024-03-06T07:24:09.865141+0000","last_unstale":"2024-03-06T07:24:12.434752+0000","last_undegraded":"2024-03-06T07:24:12.434752+0000","last_fullsized":"2024-03-06T07:24:12.434752+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:20.460653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870454+0000","last_change":"2024-03-06T07:24:10.883194+0000","last_active":"2024-03-06T07:24:11.870454+0000","last_peered":"2024-03-06T07:24:11.870454+0000","last_clean":"2024-03-06T07:24:11.870454+0000","last_became_active":"2024-03-06T07:24:10.882932+0000","last_became_peered":"2024-03-06T07:24:10.882932+0000","last_unstale":"2024-03-06T07:24:11.870454+0000","last_undegraded":"2024-03-06T07:24:11.870454+0000","last_fullsized":"2024-03-06T07:24:11.870454+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:05.049220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870108+0000","last_change":"2024-03-06T07:24:09.867221+0000","last_active":"2024-03-06T07:24:11.870108+0000","last_peered":"2024-03-06T07:24:11.870108+0000","last_clean":"2024-03-06T07:24:11.870108+0000","last_became_active":"2024-03-06T07:24:09.866916+0000","last_became_peered":"2024-03-06T07:24:09.866916+0000","last_unstale":"2024-03-06T07:24:11.870108+0000","last_undegraded":"2024-03-06T07:24:11.870108+0000","last_fullsized":"2024-03-06T07:24:11.870108+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:08.126867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707572+0000","last_change":"2024-03-06T07:24:10.876647+0000","last_active":"2024-03-06T07:24:12.707572+0000","last_peered":"2024-03-06T07:24:12.707572+0000","last_clean":"2024-03-06T07:24:12.707572+0000","last_became_active":"2024-03-06T07:24:10.876396+0000","last_became_peered":"2024-03-06T07:24:10.876396+0000","last_unstale":"2024-03-06T07:24:12.707572+0000","last_undegraded":"2024-03-06T07:24:12.707572+0000","last_fullsized":"2024-03-06T07:24:12.707572+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:16.962306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.434591+0000","last_change":"2024-03-06T07:24:09.865489+0000","last_active":"2024-03-06T07:24:12.434591+0000","last_peered":"2024-03-06T07:24:12.434591+0000","last_clean":"2024-03-06T07:24:12.434591+0000","last_became_active":"2024-03-06T07:24:09.865110+0000","last_became_peered":"2024-03-06T07:24:09.865110+0000","last_unstale":"2024-03-06T07:24:12.434591+0000","last_undegraded":"2024-03-06T07:24:12.434591+0000","last_fullsized":"2024-03-06T07:24:12.434591+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:08.965620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707339+0000","last_change":"2024-03-06T07:24:10.882191+0000","last_active":"2024-03-06T07:24:12.707339+0000","last_peered":"2024-03-06T07:24:12.707339+0000","last_clean":"2024-03-06T07:24:12.707339+0000","last_became_active":"2024-03-06T07:24:10.881923+0000","last_became_peered":"2024-03-06T07:24:10.881923+0000","last_unstale":"2024-03-06T07:24:12.707339+0000","last_undegraded":"2024-03-06T07:24:12.707339+0000","last_fullsized":"2024-03-06T07:24:12.707339+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:15.519406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706764+0000","last_change":"2024-03-06T07:24:09.867744+0000","last_active":"2024-03-06T07:24:12.706764+0000","last_peered":"2024-03-06T07:24:12.706764+0000","last_clean":"2024-03-06T07:24:12.706764+0000","last_became_active":"2024-03-06T07:24:09.867379+0000","last_became_peered":"2024-03-06T07:24:09.867379+0000","last_unstale":"2024-03-06T07:24:12.706764+0000","last_undegraded":"2024-03-06T07:24:12.706764+0000","last_fullsized":"2024-03-06T07:24:12.706764+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:01.000646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706529+0000","last_change":"2024-03-06T07:24:10.872169+0000","last_active":"2024-03-06T07:24:12.706529+0000","last_peered":"2024-03-06T07:24:12.706529+0000","last_clean":"2024-03-06T07:24:12.706529+0000","last_became_active":"2024-03-06T07:24:10.871741+0000","last_became_peered":"2024-03-06T07:24:10.871741+0000","last_unstale":"2024-03-06T07:24:12.706529+0000","last_undegraded":"2024-03-06T07:24:12.706529+0000","last_fullsized":"2024-03-06T07:24:12.706529+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:12.210056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"108'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707153+0000","last_change":"2024-03-06T07:24:09.867490+0000","last_active":"2024-03-06T07:24:12.707153+0000","last_peered":"2024-03-06T07:24:12.707153+0000","last_clean":"2024-03-06T07:24:12.707153+0000","last_became_active":"2024-03-06T07:24:09.867201+0000","last_became_peered":"2024-03-06T07:24:09.867201+0000","last_unstale":"2024-03-06T07:24:12.707153+0000","last_undegraded":"2024-03-06T07:24:12.707153+0000","last_fullsized":"2024-03-06T07:24:12.707153+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:40.272791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435321+0000","last_change":"2024-03-06T07:24:10.873473+0000","last_active":"2024-03-06T07:24:12.435321+0000","last_peered":"2024-03-06T07:24:12.435321+0000","last_clean":"2024-03-06T07:24:12.435321+0000","last_became_active":"2024-03-06T07:24:10.873196+0000","last_became_peered":"2024-03-06T07:24:10.873196+0000","last_unstale":"2024-03-06T07:24:12.435321+0000","last_undegraded":"2024-03-06T07:24:12.435321+0000","last_fullsized":"2024-03-06T07:24:12.435321+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:10.566732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707436+0000","last_change":"2024-03-06T07:24:09.869094+0000","last_active":"2024-03-06T07:24:12.707436+0000","last_peered":"2024-03-06T07:24:12.707436+0000","last_clean":"2024-03-06T07:24:12.707436+0000","last_became_active":"2024-03-06T07:24:09.868745+0000","last_became_peered":"2024-03-06T07:24:09.868745+0000","last_unstale":"2024-03-06T07:24:12.707436+0000","last_undegraded":"2024-03-06T07:24:12.707436+0000","last_fullsized":"2024-03-06T07:24:12.707436+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:19.077946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.867544+0000","last_change":"2024-03-06T07:24:10.871045+0000","last_active":"2024-03-06T07:24:11.867544+0000","last_peered":"2024-03-06T07:24:11.867544+0000","last_clean":"2024-03-06T07:24:11.867544+0000","last_became_active":"2024-03-06T07:24:10.870734+0000","last_became_peered":"2024-03-06T07:24:10.870734+0000","last_unstale":"2024-03-06T07:24:11.867544+0000","last_undegraded":"2024-03-06T07:24:11.867544+0000","last_fullsized":"2024-03-06T07:24:11.867544+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:38.020645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707380+0000","last_change":"2024-03-06T07:24:09.864124+0000","last_active":"2024-03-06T07:24:12.707380+0000","last_peered":"2024-03-06T07:24:12.707380+0000","last_clean":"2024-03-06T07:24:12.707380+0000","last_became_active":"2024-03-06T07:24:09.863447+0000","last_became_peered":"2024-03-06T07:24:09.863447+0000","last_unstale":"2024-03-06T07:24:12.707380+0000","last_undegraded":"2024-03-06T07:24:12.707380+0000","last_fullsized":"2024-03-06T07:24:12.707380+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:48.704351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706842+0000","last_change":"2024-03-06T07:24:10.881045+0000","last_active":"2024-03-06T07:24:12.706842+0000","last_peered":"2024-03-06T07:24:12.706842+0000","last_clean":"2024-03-06T07:24:12.706842+0000","last_became_active":"2024-03-06T07:24:10.880707+0000","last_became_peered":"2024-03-06T07:24:10.880707+0000","last_unstale":"2024-03-06T07:24:12.706842+0000","last_undegraded":"2024-03-06T07:24:12.706842+0000","last_fullsized":"2024-03-06T07:24:12.706842+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:48.525994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871301+0000","last_change":"2024-03-06T07:24:09.861100+0000","last_active":"2024-03-06T07:24:11.871301+0000","last_peered":"2024-03-06T07:24:11.871301+0000","last_clean":"2024-03-06T07:24:11.871301+0000","last_became_active":"2024-03-06T07:24:09.860790+0000","last_became_peered":"2024-03-06T07:24:09.860790+0000","last_unstale":"2024-03-06T07:24:11.871301+0000","last_undegraded":"2024-03-06T07:24:11.871301+0000","last_fullsized":"2024-03-06T07:24:11.871301+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:08.129149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870054+0000","last_change":"2024-03-06T07:24:10.884752+0000","last_active":"2024-03-06T07:24:11.870054+0000","last_peered":"2024-03-06T07:24:11.870054+0000","last_clean":"2024-03-06T07:24:11.870054+0000","last_became_active":"2024-03-06T07:24:10.884463+0000","last_became_peered":"2024-03-06T07:24:10.884463+0000","last_unstale":"2024-03-06T07:24:11.870054+0000","last_undegraded":"2024-03-06T07:24:11.870054+0000","last_fullsized":"2024-03-06T07:24:11.870054+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:03.883574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871613+0000","last_change":"2024-03-06T07:24:09.866939+0000","last_active":"2024-03-06T07:24:11.871613+0000","last_peered":"2024-03-06T07:24:11.871613+0000","last_clean":"2024-03-06T07:24:11.871613+0000","last_became_active":"2024-03-06T07:24:09.866561+0000","last_became_peered":"2024-03-06T07:24:09.866561+0000","last_unstale":"2024-03-06T07:24:11.871613+0000","last_undegraded":"2024-03-06T07:24:11.871613+0000","last_fullsized":"2024-03-06T07:24:11.871613+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:47.869257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.434859+0000","last_change":"2024-03-06T07:24:10.876783+0000","last_active":"2024-03-06T07:24:12.434859+0000","last_peered":"2024-03-06T07:24:12.434859+0000","last_clean":"2024-03-06T07:24:12.434859+0000","last_became_active":"2024-03-06T07:24:10.875768+0000","last_became_peered":"2024-03-06T07:24:10.875768+0000","last_unstale":"2024-03-06T07:24:12.434859+0000","last_undegraded":"2024-03-06T07:24:12.434859+0000","last_fullsized":"2024-03-06T07:24:12.434859+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:09.130833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706649+0000","last_change":"2024-03-06T07:24:09.866286+0000","last_active":"2024-03-06T07:24:12.706649+0000","last_peered":"2024-03-06T07:24:12.706649+0000","last_clean":"2024-03-06T07:24:12.706649+0000","last_became_active":"2024-03-06T07:24:09.865961+0000","last_became_peered":"2024-03-06T07:24:09.865961+0000","last_unstale":"2024-03-06T07:24:12.706649+0000","last_undegraded":"2024-03-06T07:24:12.706649+0000","last_fullsized":"2024-03-06T07:24:12.706649+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:44.450400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706694+0000","last_change":"2024-03-06T07:24:10.865615+0000","last_active":"2024-03-06T07:24:12.706694+0000","last_peered":"2024-03-06T07:24:12.706694+0000","last_clean":"2024-03-06T07:24:12.706694+0000","last_became_active":"2024-03-06T07:24:10.865340+0000","last_became_peered":"2024-03-06T07:24:10.865340+0000","last_unstale":"2024-03-06T07:24:12.706694+0000","last_undegraded":"2024-03-06T07:24:12.706694+0000","last_fullsized":"2024-03-06T07:24:12.706694+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:55.863177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.434602+0000","last_change":"2024-03-06T07:24:09.865914+0000","last_active":"2024-03-06T07:24:12.434602+0000","last_peered":"2024-03-06T07:24:12.434602+0000","last_clean":"2024-03-06T07:24:12.434602+0000","last_became_active":"2024-03-06T07:24:09.865626+0000","last_became_peered":"2024-03-06T07:24:09.865626+0000","last_unstale":"2024-03-06T07:24:12.434602+0000","last_undegraded":"2024-03-06T07:24:12.434602+0000","last_fullsized":"2024-03-06T07:24:12.434602+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:55.962450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435120+0000","last_change":"2024-03-06T07:24:10.868167+0000","last_active":"2024-03-06T07:24:12.435120+0000","last_peered":"2024-03-06T07:24:12.435120+0000","last_clean":"2024-03-06T07:24:12.435120+0000","last_became_active":"2024-03-06T07:24:10.867887+0000","last_became_peered":"2024-03-06T07:24:10.867887+0000","last_unstale":"2024-03-06T07:24:12.435120+0000","last_undegraded":"2024-03-06T07:24:12.435120+0000","last_fullsized":"2024-03-06T07:24:12.435120+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:55.303316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"107'8","reported_seq":22,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.869896+0000","last_change":"2024-03-06T07:24:09.863372+0000","last_active":"2024-03-06T07:24:11.869896+0000","last_peered":"2024-03-06T07:24:11.869896+0000","last_clean":"2024-03-06T07:24:11.869896+0000","last_became_active":"2024-03-06T07:24:09.863011+0000","last_became_peered":"2024-03-06T07:24:09.863011+0000","last_unstale":"2024-03-06T07:24:11.869896+0000","last_undegraded":"2024-03-06T07:24:11.869896+0000","last_fullsized":"2024-03-06T07:24:11.869896+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:54.859211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":339,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.434740+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:24:12.434740+0000","last_peered":"2024-03-06T07:24:12.434740+0000","last_clean":"2024-03-06T07:24:12.434740+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:24:12.434740+0000","last_undegraded":"2024-03-06T07:24:12.434740+0000","last_fullsized":"2024-03-06T07:24:12.434740+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574899,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1429504,"data_stored":1241574,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1351680,"data_stored":651012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1355776,"data_stored":650590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574899,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1359872,"data_stored":652042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1363968,"data_stored":652076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":650430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1351680,"data_stored":650452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574899,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1417216,"data_stored":1240820,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:16.144 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-06T07:24:16.142+0000 7ffbf309b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:18.312 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:18.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.412+0000 7fddb2a40640 1 Processor -- start 2024-03-06T07:24:18.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.412+0000 7fddb2a40640 1 -- start start 2024-03-06T07:24:18.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.413+0000 7fddb2a40640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fddac152660 0x7fddac0a9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.413+0000 7fddb2a40640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fddac152a40 0x7fddac155640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.413+0000 7fddb2a40640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddac0c3030 con 0x7fddac06f800 2024-03-06T07:24:18.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.413+0000 7fddb2a40640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddac0c3310 con 0x7fddac152660 2024-03-06T07:24:18.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.413+0000 7fddb2a40640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddac0c35f0 con 0x7fddac152a40 2024-03-06T07:24:18.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.414+0000 7fddab7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fddac06f800 0x7fddac0c10b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36032/0 (socket says 172.21.15.1:36032) 2024-03-06T07:24:18.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.414+0000 7fddab7fe640 1 -- 172.21.15.1:0/687745458 learned_addr learned my addr 172.21.15.1:0/687745458 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:18.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.414+0000 7fddabfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fddac152660 0x7fddac0a9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.414+0000 7fddb0fdd640 1 --2- 172.21.15.1:0/687745458 >> v2:172.21.15.98:3300/0 conn(0x7fddac152a40 0x7fddac155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.414+0000 7fddabfff640 1 -- 172.21.15.1:0/687745458 >> v2:172.21.15.98:3300/0 conn(0x7fddac152a40 msgr2=0x7fddac155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:18.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.414+0000 7fddabfff640 1 --2- 172.21.15.1:0/687745458 >> v2:172.21.15.98:3300/0 conn(0x7fddac152a40 0x7fddac155640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.414+0000 7fddabfff640 1 -- 172.21.15.1:0/687745458 >> v1:172.21.15.1:6789/0 conn(0x7fddac06f800 legacy=0x7fddac0c10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.414+0000 7fddabfff640 1 -- 172.21.15.1:0/687745458 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac0c38d0 con 0x7fddac152660 2024-03-06T07:24:18.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.415+0000 7fddabfff640 1 --2- 172.21.15.1:0/687745458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddac152660 0x7fddac0a9760 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fddac06caa0 tx=0x7fdd9c02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134845ab8e8b3b81 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.415+0000 7fddaaffd640 1 -- 172.21.15.1:0/687745458 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c038070 con 0x7fddac152660 2024-03-06T07:24:18.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.415+0000 7fddaaffd640 1 -- 172.21.15.1:0/687745458 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd9c031710 con 0x7fddac152660 2024-03-06T07:24:18.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.415+0000 7fddaaffd640 1 -- 172.21.15.1:0/687745458 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c031ae0 con 0x7fddac152660 2024-03-06T07:24:18.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.416+0000 7fddb2a40640 1 -- 172.21.15.1:0/687745458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddac152660 msgr2=0x7fddac0a9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.416+0000 7fddb2a40640 1 --2- 172.21.15.1:0/687745458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddac152660 0x7fddac0a9760 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fddac06caa0 tx=0x7fdd9c02b7b0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.416+0000 7fddb2a40640 1 -- 172.21.15.1:0/687745458 shutdown_connections 2024-03-06T07:24:18.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.416+0000 7fddb2a40640 1 --2- 172.21.15.1:0/687745458 >> v2:172.21.15.98:3300/0 conn(0x7fddac152a40 0x7fddac155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.416+0000 7fddb2a40640 1 --2- 172.21.15.1:0/687745458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddac152660 0x7fddac0a9760 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.416+0000 7fddb2a40640 1 -- 172.21.15.1:0/687745458 >> 172.21.15.1:0/687745458 conn(0x7fddac0c3cc0 msgr2=0x7fddac0c40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:18.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.416+0000 7fddb2a40640 1 -- 172.21.15.1:0/687745458 shutdown_connections 2024-03-06T07:24:18.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.416+0000 7fddb2a40640 1 -- 172.21.15.1:0/687745458 wait complete. 2024-03-06T07:24:18.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.418+0000 7fddb2a40640 1 Processor -- start 2024-03-06T07:24:18.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.418+0000 7fddb2a40640 1 -- start start 2024-03-06T07:24:18.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddb2a40640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fddac06f800 0x7fddac156470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddb2a40640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddac152660 0x7fddac14d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddb2a40640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddac152a40 0x7fddac0c6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddb2a40640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddac0c17c0 con 0x7fddac152a40 2024-03-06T07:24:18.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddb2a40640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddac0a84b0 con 0x7fddac152660 2024-03-06T07:24:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddb2a40640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddac0c1940 con 0x7fddac06f800 2024-03-06T07:24:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddb0fdd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddac152a40 0x7fddac0c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddb0fdd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddac152a40 0x7fddac0c6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44320/0 (socket says 172.21.15.1:44320) 2024-03-06T07:24:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddb0fdd640 1 -- 172.21.15.1:0/1245138492 learned_addr learned my addr 172.21.15.1:0/1245138492 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddab7fe640 1 --2- 172.21.15.1:0/1245138492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddac152660 0x7fddac14d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddabfff640 1 --2- 172.21.15.1:0/1245138492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fddac06f800 0x7fddac156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddb0fdd640 1 -- 172.21.15.1:0/1245138492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fddac06f800 msgr2=0x7fddac156470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddb0fdd640 1 --2- 172.21.15.1:0/1245138492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fddac06f800 0x7fddac156470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddb0fdd640 1 -- 172.21.15.1:0/1245138492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddac152660 msgr2=0x7fddac14d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddb0fdd640 1 --2- 172.21.15.1:0/1245138492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddac152660 0x7fddac14d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddb0fdd640 1 -- 172.21.15.1:0/1245138492 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac0c68d0 con 0x7fddac152a40 2024-03-06T07:24:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddab7fe640 1 --2- 172.21.15.1:0/1245138492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddac152660 0x7fddac14d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.419+0000 7fddabfff640 1 --2- 172.21.15.1:0/1245138492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fddac06f800 0x7fddac156470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.420+0000 7fddb0fdd640 1 --2- 172.21.15.1:0/1245138492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddac152a40 0x7fddac0c6030 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fdda4007e40 tx=0x7fdda400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.420+0000 7fdda8ff9640 1 -- 172.21.15.1:0/1245138492 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda4014620 con 0x7fddac152a40 2024-03-06T07:24:18.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.420+0000 7fdda8ff9640 1 -- 172.21.15.1:0/1245138492 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda4019070 con 0x7fddac152a40 2024-03-06T07:24:18.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.420+0000 7fddb2a40640 1 -- 172.21.15.1:0/1245138492 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddac1edb40 con 0x7fddac152a40 2024-03-06T07:24:18.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.420+0000 7fdda8ff9640 1 -- 172.21.15.1:0/1245138492 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda4014a20 con 0x7fddac152a40 2024-03-06T07:24:18.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.420+0000 7fddb2a40640 1 -- 172.21.15.1:0/1245138492 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddac0abae0 con 0x7fddac152a40 2024-03-06T07:24:18.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.421+0000 7fddb2a40640 1 -- 172.21.15.1:0/1245138492 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddac0686e0 con 0x7fddac152a40 2024-03-06T07:24:18.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.421+0000 7fdda8ff9640 1 -- 172.21.15.1:0/1245138492 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdda4014bc0 con 0x7fddac152a40 2024-03-06T07:24:18.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.422+0000 7fdda8ff9640 1 --2- 172.21.15.1:0/1245138492 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd9806b4d0 0x7fdd9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.422+0000 7fdda8ff9640 1 -- 172.21.15.1:0/1245138492 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdda408abd0 con 0x7fddac152a40 2024-03-06T07:24:18.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.422+0000 7fddabfff640 1 --2- 172.21.15.1:0/1245138492 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd9806b4d0 0x7fdd9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.423+0000 7fddabfff640 1 --2- 172.21.15.1:0/1245138492 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd9806b4d0 0x7fdd9806d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c035000 tx=0x7fdd9c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.426+0000 7fdda8ff9640 1 -- 172.21.15.1:0/1245138492 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdda4059950 con 0x7fddac152a40 2024-03-06T07:24:18.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.655+0000 7fddb2a40640 1 -- 172.21.15.1:0/1245138492 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fddac0c6570 con 0x7fdd9806b4d0 2024-03-06T07:24:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.661+0000 7fdda8ff9640 1 -- 172.21.15.1:0/1245138492 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252565 (secure 0 0 0) 0x7fddac0c6570 con 0x7fdd9806b4d0 2024-03-06T07:24:18.666 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:24:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.664+0000 7fddb2a40640 1 -- 172.21.15.1:0/1245138492 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd9806b4d0 msgr2=0x7fdd9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.664+0000 7fddb2a40640 1 --2- 172.21.15.1:0/1245138492 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd9806b4d0 0x7fdd9806d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c035000 tx=0x7fdd9c008030 comp rx=0 tx=0).stop 2024-03-06T07:24:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.664+0000 7fddb2a40640 1 -- 172.21.15.1:0/1245138492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddac152a40 msgr2=0x7fddac0c6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.664+0000 7fddb2a40640 1 --2- 172.21.15.1:0/1245138492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddac152a40 0x7fddac0c6030 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fdda4007e40 tx=0x7fdda400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.664+0000 7fddb2a40640 1 -- 172.21.15.1:0/1245138492 shutdown_connections 2024-03-06T07:24:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.664+0000 7fddb2a40640 1 --2- 172.21.15.1:0/1245138492 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd9806b4d0 0x7fdd9806d990 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.664+0000 7fddb2a40640 1 --2- 172.21.15.1:0/1245138492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddac152a40 0x7fddac0c6030 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.664+0000 7fddb2a40640 1 --2- 172.21.15.1:0/1245138492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddac152660 0x7fddac14d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.664+0000 7fddb2a40640 1 --2- 172.21.15.1:0/1245138492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fddac06f800 0x7fddac156470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.664+0000 7fddb2a40640 1 -- 172.21.15.1:0/1245138492 >> 172.21.15.1:0/1245138492 conn(0x7fddac0c3cc0 msgr2=0x7fddac0c05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.664+0000 7fddb2a40640 1 -- 172.21.15.1:0/1245138492 shutdown_connections 2024-03-06T07:24:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.664+0000 7fddb2a40640 1 -- 172.21.15.1:0/1245138492 wait complete. 2024-03-06T07:24:18.677 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-03-06T07:24:16.744498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260188,"kb_used_data":10720,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714532,"statfs":{"total":773094113280,"available":772827680768,"internally_reserved":0,"allocated":10977280,"data_stored":6388996,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.898795"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871645+0000","last_change":"2024-03-06T07:24:10.872255+0000","last_active":"2024-03-06T07:24:11.871645+0000","last_peered":"2024-03-06T07:24:11.871645+0000","last_clean":"2024-03-06T07:24:11.871645+0000","last_became_active":"2024-03-06T07:24:10.871997+0000","last_became_peered":"2024-03-06T07:24:10.871997+0000","last_unstale":"2024-03-06T07:24:11.871645+0000","last_undegraded":"2024-03-06T07:24:11.871645+0000","last_fullsized":"2024-03-06T07:24:11.871645+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:14.096402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870272+0000","last_change":"2024-03-06T07:24:10.887711+0000","last_active":"2024-03-06T07:24:11.870272+0000","last_peered":"2024-03-06T07:24:11.870272+0000","last_clean":"2024-03-06T07:24:11.870272+0000","last_became_active":"2024-03-06T07:24:10.887166+0000","last_became_peered":"2024-03-06T07:24:10.887166+0000","last_unstale":"2024-03-06T07:24:11.870272+0000","last_undegraded":"2024-03-06T07:24:11.870272+0000","last_fullsized":"2024-03-06T07:24:11.870272+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:59.150271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870413+0000","last_change":"2024-03-06T07:24:10.885027+0000","last_active":"2024-03-06T07:24:11.870413+0000","last_peered":"2024-03-06T07:24:11.870413+0000","last_clean":"2024-03-06T07:24:11.870413+0000","last_became_active":"2024-03-06T07:24:10.884742+0000","last_became_peered":"2024-03-06T07:24:10.884742+0000","last_unstale":"2024-03-06T07:24:11.870413+0000","last_undegraded":"2024-03-06T07:24:11.870413+0000","last_fullsized":"2024-03-06T07:24:11.870413+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:06.270087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435166+0000","last_change":"2024-03-06T07:24:10.872345+0000","last_active":"2024-03-06T07:24:12.435166+0000","last_peered":"2024-03-06T07:24:12.435166+0000","last_clean":"2024-03-06T07:24:12.435166+0000","last_became_active":"2024-03-06T07:24:10.872070+0000","last_became_peered":"2024-03-06T07:24:10.872070+0000","last_unstale":"2024-03-06T07:24:12.435166+0000","last_undegraded":"2024-03-06T07:24:12.435166+0000","last_fullsized":"2024-03-06T07:24:12.435166+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:08.047230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871133+0000","last_change":"2024-03-06T07:24:10.875833+0000","last_active":"2024-03-06T07:24:11.871133+0000","last_peered":"2024-03-06T07:24:11.871133+0000","last_clean":"2024-03-06T07:24:11.871133+0000","last_became_active":"2024-03-06T07:24:10.874654+0000","last_became_peered":"2024-03-06T07:24:10.874654+0000","last_unstale":"2024-03-06T07:24:11.871133+0000","last_undegraded":"2024-03-06T07:24:11.871133+0000","last_fullsized":"2024-03-06T07:24:11.871133+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:16.962306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706910+0000","last_change":"2024-03-06T07:24:10.879327+0000","last_active":"2024-03-06T07:24:12.706910+0000","last_peered":"2024-03-06T07:24:12.706910+0000","last_clean":"2024-03-06T07:24:12.706910+0000","last_became_active":"2024-03-06T07:24:10.879066+0000","last_became_peered":"2024-03-06T07:24:10.879066+0000","last_unstale":"2024-03-06T07:24:12.706910+0000","last_undegraded":"2024-03-06T07:24:12.706910+0000","last_fullsized":"2024-03-06T07:24:12.706910+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:26.886161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706541+0000","last_change":"2024-03-06T07:24:10.874658+0000","last_active":"2024-03-06T07:24:12.706541+0000","last_peered":"2024-03-06T07:24:12.706541+0000","last_clean":"2024-03-06T07:24:12.706541+0000","last_became_active":"2024-03-06T07:24:10.874409+0000","last_became_peered":"2024-03-06T07:24:10.874409+0000","last_unstale":"2024-03-06T07:24:12.706541+0000","last_undegraded":"2024-03-06T07:24:12.706541+0000","last_fullsized":"2024-03-06T07:24:12.706541+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:09:40.994943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.869960+0000","last_change":"2024-03-06T07:24:10.888269+0000","last_active":"2024-03-06T07:24:11.869960+0000","last_peered":"2024-03-06T07:24:11.869960+0000","last_clean":"2024-03-06T07:24:11.869960+0000","last_became_active":"2024-03-06T07:24:10.887087+0000","last_became_peered":"2024-03-06T07:24:10.887087+0000","last_unstale":"2024-03-06T07:24:11.869960+0000","last_undegraded":"2024-03-06T07:24:11.869960+0000","last_fullsized":"2024-03-06T07:24:11.869960+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:41.969138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435403+0000","last_change":"2024-03-06T07:24:10.880426+0000","last_active":"2024-03-06T07:24:12.435403+0000","last_peered":"2024-03-06T07:24:12.435403+0000","last_clean":"2024-03-06T07:24:12.435403+0000","last_became_active":"2024-03-06T07:24:10.880164+0000","last_became_peered":"2024-03-06T07:24:10.880164+0000","last_unstale":"2024-03-06T07:24:12.435403+0000","last_undegraded":"2024-03-06T07:24:12.435403+0000","last_fullsized":"2024-03-06T07:24:12.435403+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:02.467158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706778+0000","last_change":"2024-03-06T07:24:10.873861+0000","last_active":"2024-03-06T07:24:12.706778+0000","last_peered":"2024-03-06T07:24:12.706778+0000","last_clean":"2024-03-06T07:24:12.706778+0000","last_became_active":"2024-03-06T07:24:10.873603+0000","last_became_peered":"2024-03-06T07:24:10.873603+0000","last_unstale":"2024-03-06T07:24:12.706778+0000","last_undegraded":"2024-03-06T07:24:12.706778+0000","last_fullsized":"2024-03-06T07:24:12.706778+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:14.096402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871222+0000","last_change":"2024-03-06T07:24:10.877379+0000","last_active":"2024-03-06T07:24:11.871222+0000","last_peered":"2024-03-06T07:24:11.871222+0000","last_clean":"2024-03-06T07:24:11.871222+0000","last_became_active":"2024-03-06T07:24:10.877118+0000","last_became_peered":"2024-03-06T07:24:10.877118+0000","last_unstale":"2024-03-06T07:24:11.871222+0000","last_undegraded":"2024-03-06T07:24:11.871222+0000","last_fullsized":"2024-03-06T07:24:11.871222+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:12.210056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707202+0000","last_change":"2024-03-06T07:24:10.883581+0000","last_active":"2024-03-06T07:24:12.707202+0000","last_peered":"2024-03-06T07:24:12.707202+0000","last_clean":"2024-03-06T07:24:12.707202+0000","last_became_active":"2024-03-06T07:24:10.883304+0000","last_became_peered":"2024-03-06T07:24:10.883304+0000","last_unstale":"2024-03-06T07:24:12.707202+0000","last_undegraded":"2024-03-06T07:24:12.707202+0000","last_fullsized":"2024-03-06T07:24:12.707202+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:59.901346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706425+0000","last_change":"2024-03-06T07:24:10.873286+0000","last_active":"2024-03-06T07:24:12.706425+0000","last_peered":"2024-03-06T07:24:12.706425+0000","last_clean":"2024-03-06T07:24:12.706425+0000","last_became_active":"2024-03-06T07:24:10.871844+0000","last_became_peered":"2024-03-06T07:24:10.871844+0000","last_unstale":"2024-03-06T07:24:12.706425+0000","last_undegraded":"2024-03-06T07:24:12.706425+0000","last_fullsized":"2024-03-06T07:24:12.706425+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:13.966188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870116+0000","last_change":"2024-03-06T07:24:10.884278+0000","last_active":"2024-03-06T07:24:11.870116+0000","last_peered":"2024-03-06T07:24:11.870116+0000","last_clean":"2024-03-06T07:24:11.870116+0000","last_became_active":"2024-03-06T07:24:10.884006+0000","last_became_peered":"2024-03-06T07:24:10.884006+0000","last_unstale":"2024-03-06T07:24:11.870116+0000","last_undegraded":"2024-03-06T07:24:11.870116+0000","last_fullsized":"2024-03-06T07:24:11.870116+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:47.528123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871242+0000","last_change":"2024-03-06T07:24:10.878061+0000","last_active":"2024-03-06T07:24:11.871242+0000","last_peered":"2024-03-06T07:24:11.871242+0000","last_clean":"2024-03-06T07:24:11.871242+0000","last_became_active":"2024-03-06T07:24:10.877783+0000","last_became_peered":"2024-03-06T07:24:10.877783+0000","last_unstale":"2024-03-06T07:24:11.871242+0000","last_undegraded":"2024-03-06T07:24:11.871242+0000","last_fullsized":"2024-03-06T07:24:11.871242+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:49.026769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706428+0000","last_change":"2024-03-06T07:24:10.869634+0000","last_active":"2024-03-06T07:24:12.706428+0000","last_peered":"2024-03-06T07:24:12.706428+0000","last_clean":"2024-03-06T07:24:12.706428+0000","last_became_active":"2024-03-06T07:24:10.869360+0000","last_became_peered":"2024-03-06T07:24:10.869360+0000","last_unstale":"2024-03-06T07:24:12.706428+0000","last_undegraded":"2024-03-06T07:24:12.706428+0000","last_fullsized":"2024-03-06T07:24:12.706428+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:16.399326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706688+0000","last_change":"2024-03-06T07:24:10.874469+0000","last_active":"2024-03-06T07:24:12.706688+0000","last_peered":"2024-03-06T07:24:12.706688+0000","last_clean":"2024-03-06T07:24:12.706688+0000","last_became_active":"2024-03-06T07:24:10.874095+0000","last_became_peered":"2024-03-06T07:24:10.874095+0000","last_unstale":"2024-03-06T07:24:12.706688+0000","last_undegraded":"2024-03-06T07:24:12.706688+0000","last_fullsized":"2024-03-06T07:24:12.706688+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:06.797882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870253+0000","last_change":"2024-03-06T07:24:10.887496+0000","last_active":"2024-03-06T07:24:11.870253+0000","last_peered":"2024-03-06T07:24:11.870253+0000","last_clean":"2024-03-06T07:24:11.870253+0000","last_became_active":"2024-03-06T07:24:10.887039+0000","last_became_peered":"2024-03-06T07:24:10.887039+0000","last_unstale":"2024-03-06T07:24:11.870253+0000","last_undegraded":"2024-03-06T07:24:11.870253+0000","last_fullsized":"2024-03-06T07:24:11.870253+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:04.010668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707388+0000","last_change":"2024-03-06T07:24:10.869573+0000","last_active":"2024-03-06T07:24:12.707388+0000","last_peered":"2024-03-06T07:24:12.707388+0000","last_clean":"2024-03-06T07:24:12.707388+0000","last_became_active":"2024-03-06T07:24:10.869311+0000","last_became_peered":"2024-03-06T07:24:10.869311+0000","last_unstale":"2024-03-06T07:24:12.707388+0000","last_undegraded":"2024-03-06T07:24:12.707388+0000","last_fullsized":"2024-03-06T07:24:12.707388+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:09.133114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.867384+0000","last_change":"2024-03-06T07:24:10.871671+0000","last_active":"2024-03-06T07:24:11.867384+0000","last_peered":"2024-03-06T07:24:11.867384+0000","last_clean":"2024-03-06T07:24:11.867384+0000","last_became_active":"2024-03-06T07:24:10.871411+0000","last_became_peered":"2024-03-06T07:24:10.871411+0000","last_unstale":"2024-03-06T07:24:11.867384+0000","last_undegraded":"2024-03-06T07:24:11.867384+0000","last_fullsized":"2024-03-06T07:24:11.867384+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:08.340512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435032+0000","last_change":"2024-03-06T07:24:10.868119+0000","last_active":"2024-03-06T07:24:12.435032+0000","last_peered":"2024-03-06T07:24:12.435032+0000","last_clean":"2024-03-06T07:24:12.435032+0000","last_became_active":"2024-03-06T07:24:10.867820+0000","last_became_peered":"2024-03-06T07:24:10.867820+0000","last_unstale":"2024-03-06T07:24:12.435032+0000","last_undegraded":"2024-03-06T07:24:12.435032+0000","last_fullsized":"2024-03-06T07:24:12.435032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:15:23.097231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871551+0000","last_change":"2024-03-06T07:24:10.879942+0000","last_active":"2024-03-06T07:24:11.871551+0000","last_peered":"2024-03-06T07:24:11.871551+0000","last_clean":"2024-03-06T07:24:11.871551+0000","last_became_active":"2024-03-06T07:24:10.879691+0000","last_became_peered":"2024-03-06T07:24:10.879691+0000","last_unstale":"2024-03-06T07:24:11.871551+0000","last_undegraded":"2024-03-06T07:24:11.871551+0000","last_fullsized":"2024-03-06T07:24:11.871551+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:55.863177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.434926+0000","last_change":"2024-03-06T07:24:10.881411+0000","last_active":"2024-03-06T07:24:12.434926+0000","last_peered":"2024-03-06T07:24:12.434926+0000","last_clean":"2024-03-06T07:24:12.434926+0000","last_became_active":"2024-03-06T07:24:10.879424+0000","last_became_peered":"2024-03-06T07:24:10.879424+0000","last_unstale":"2024-03-06T07:24:12.434926+0000","last_undegraded":"2024-03-06T07:24:12.434926+0000","last_fullsized":"2024-03-06T07:24:12.434926+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:36.612944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706892+0000","last_change":"2024-03-06T07:24:10.872049+0000","last_active":"2024-03-06T07:24:12.706892+0000","last_peered":"2024-03-06T07:24:12.706892+0000","last_clean":"2024-03-06T07:24:12.706892+0000","last_became_active":"2024-03-06T07:24:10.871719+0000","last_became_peered":"2024-03-06T07:24:10.871719+0000","last_unstale":"2024-03-06T07:24:12.706892+0000","last_undegraded":"2024-03-06T07:24:12.706892+0000","last_fullsized":"2024-03-06T07:24:12.706892+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:21.464618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707914+0000","last_change":"2024-03-06T07:24:10.869734+0000","last_active":"2024-03-06T07:24:12.707914+0000","last_peered":"2024-03-06T07:24:12.707914+0000","last_clean":"2024-03-06T07:24:12.707914+0000","last_became_active":"2024-03-06T07:24:10.869325+0000","last_became_peered":"2024-03-06T07:24:10.869325+0000","last_unstale":"2024-03-06T07:24:12.707914+0000","last_undegraded":"2024-03-06T07:24:12.707914+0000","last_fullsized":"2024-03-06T07:24:12.707914+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:06.797882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.867410+0000","last_change":"2024-03-06T07:24:10.865909+0000","last_active":"2024-03-06T07:24:11.867410+0000","last_peered":"2024-03-06T07:24:11.867410+0000","last_clean":"2024-03-06T07:24:11.867410+0000","last_became_active":"2024-03-06T07:24:10.865638+0000","last_became_peered":"2024-03-06T07:24:10.865638+0000","last_unstale":"2024-03-06T07:24:11.867410+0000","last_undegraded":"2024-03-06T07:24:11.867410+0000","last_fullsized":"2024-03-06T07:24:11.867410+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:37.759423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707407+0000","last_change":"2024-03-06T07:24:10.869283+0000","last_active":"2024-03-06T07:24:12.707407+0000","last_peered":"2024-03-06T07:24:12.707407+0000","last_clean":"2024-03-06T07:24:12.707407+0000","last_became_active":"2024-03-06T07:24:10.869005+0000","last_became_peered":"2024-03-06T07:24:10.869005+0000","last_unstale":"2024-03-06T07:24:12.707407+0000","last_undegraded":"2024-03-06T07:24:12.707407+0000","last_fullsized":"2024-03-06T07:24:12.707407+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:39.447002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707262+0000","last_change":"2024-03-06T07:24:10.877332+0000","last_active":"2024-03-06T07:24:12.707262+0000","last_peered":"2024-03-06T07:24:12.707262+0000","last_clean":"2024-03-06T07:24:12.707262+0000","last_became_active":"2024-03-06T07:24:10.876975+0000","last_became_peered":"2024-03-06T07:24:10.876975+0000","last_unstale":"2024-03-06T07:24:12.707262+0000","last_undegraded":"2024-03-06T07:24:12.707262+0000","last_fullsized":"2024-03-06T07:24:12.707262+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:22.153347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.867482+0000","last_change":"2024-03-06T07:24:10.878173+0000","last_active":"2024-03-06T07:24:11.867482+0000","last_peered":"2024-03-06T07:24:11.867482+0000","last_clean":"2024-03-06T07:24:11.867482+0000","last_became_active":"2024-03-06T07:24:10.877825+0000","last_became_peered":"2024-03-06T07:24:10.877825+0000","last_unstale":"2024-03-06T07:24:11.867482+0000","last_undegraded":"2024-03-06T07:24:11.867482+0000","last_fullsized":"2024-03-06T07:24:11.867482+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:04.191625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870217+0000","last_change":"2024-03-06T07:24:10.882571+0000","last_active":"2024-03-06T07:24:11.870217+0000","last_peered":"2024-03-06T07:24:11.870217+0000","last_clean":"2024-03-06T07:24:11.870217+0000","last_became_active":"2024-03-06T07:24:10.882244+0000","last_became_peered":"2024-03-06T07:24:10.882244+0000","last_unstale":"2024-03-06T07:24:11.870217+0000","last_undegraded":"2024-03-06T07:24:11.870217+0000","last_fullsized":"2024-03-06T07:24:11.870217+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:42.857654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707336+0000","last_change":"2024-03-06T07:24:10.881361+0000","last_active":"2024-03-06T07:24:12.707336+0000","last_peered":"2024-03-06T07:24:12.707336+0000","last_clean":"2024-03-06T07:24:12.707336+0000","last_became_active":"2024-03-06T07:24:10.879117+0000","last_became_peered":"2024-03-06T07:24:10.879117+0000","last_unstale":"2024-03-06T07:24:12.707336+0000","last_undegraded":"2024-03-06T07:24:12.707336+0000","last_fullsized":"2024-03-06T07:24:12.707336+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:18.273527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707128+0000","last_change":"2024-03-06T07:24:10.871229+0000","last_active":"2024-03-06T07:24:12.707128+0000","last_peered":"2024-03-06T07:24:12.707128+0000","last_clean":"2024-03-06T07:24:12.707128+0000","last_became_active":"2024-03-06T07:24:10.870824+0000","last_became_peered":"2024-03-06T07:24:10.870824+0000","last_unstale":"2024-03-06T07:24:12.707128+0000","last_undegraded":"2024-03-06T07:24:12.707128+0000","last_fullsized":"2024-03-06T07:24:12.707128+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:27.910318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706869+0000","last_change":"2024-03-06T07:24:10.873797+0000","last_active":"2024-03-06T07:24:12.706869+0000","last_peered":"2024-03-06T07:24:12.706869+0000","last_clean":"2024-03-06T07:24:12.706869+0000","last_became_active":"2024-03-06T07:24:10.873518+0000","last_became_peered":"2024-03-06T07:24:10.873518+0000","last_unstale":"2024-03-06T07:24:12.706869+0000","last_undegraded":"2024-03-06T07:24:12.706869+0000","last_fullsized":"2024-03-06T07:24:12.706869+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:48.576719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"107'4","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706897+0000","last_change":"2024-03-06T07:24:09.866650+0000","last_active":"2024-03-06T07:24:12.706897+0000","last_peered":"2024-03-06T07:24:12.706897+0000","last_clean":"2024-03-06T07:24:12.706897+0000","last_became_active":"2024-03-06T07:24:09.866357+0000","last_became_peered":"2024-03-06T07:24:09.866357+0000","last_unstale":"2024-03-06T07:24:12.706897+0000","last_undegraded":"2024-03-06T07:24:12.706897+0000","last_fullsized":"2024-03-06T07:24:12.706897+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:56.308026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706552+0000","last_change":"2024-03-06T07:24:10.879050+0000","last_active":"2024-03-06T07:24:12.706552+0000","last_peered":"2024-03-06T07:24:12.706552+0000","last_clean":"2024-03-06T07:24:12.706552+0000","last_became_active":"2024-03-06T07:24:10.878772+0000","last_became_peered":"2024-03-06T07:24:10.878772+0000","last_unstale":"2024-03-06T07:24:12.706552+0000","last_undegraded":"2024-03-06T07:24:12.706552+0000","last_fullsized":"2024-03-06T07:24:12.706552+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:33.944045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706467+0000","last_change":"2024-03-06T07:24:09.851704+0000","last_active":"2024-03-06T07:24:12.706467+0000","last_peered":"2024-03-06T07:24:12.706467+0000","last_clean":"2024-03-06T07:24:12.706467+0000","last_became_active":"2024-03-06T07:24:09.851423+0000","last_became_peered":"2024-03-06T07:24:09.851423+0000","last_unstale":"2024-03-06T07:24:12.706467+0000","last_undegraded":"2024-03-06T07:24:12.706467+0000","last_fullsized":"2024-03-06T07:24:12.706467+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:14.500408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707794+0000","last_change":"2024-03-06T07:24:10.882779+0000","last_active":"2024-03-06T07:24:12.707794+0000","last_peered":"2024-03-06T07:24:12.707794+0000","last_clean":"2024-03-06T07:24:12.707794+0000","last_became_active":"2024-03-06T07:24:10.881271+0000","last_became_peered":"2024-03-06T07:24:10.881271+0000","last_unstale":"2024-03-06T07:24:12.707794+0000","last_undegraded":"2024-03-06T07:24:12.707794+0000","last_fullsized":"2024-03-06T07:24:12.707794+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:49.026769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435140+0000","last_change":"2024-03-06T07:24:09.865828+0000","last_active":"2024-03-06T07:24:12.435140+0000","last_peered":"2024-03-06T07:24:12.435140+0000","last_clean":"2024-03-06T07:24:12.435140+0000","last_became_active":"2024-03-06T07:24:09.865126+0000","last_became_peered":"2024-03-06T07:24:09.865126+0000","last_unstale":"2024-03-06T07:24:12.435140+0000","last_undegraded":"2024-03-06T07:24:12.435140+0000","last_fullsized":"2024-03-06T07:24:12.435140+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:15.958341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435413+0000","last_change":"2024-03-06T07:24:10.868169+0000","last_active":"2024-03-06T07:24:12.435413+0000","last_peered":"2024-03-06T07:24:12.435413+0000","last_clean":"2024-03-06T07:24:12.435413+0000","last_became_active":"2024-03-06T07:24:10.867842+0000","last_became_peered":"2024-03-06T07:24:10.867842+0000","last_unstale":"2024-03-06T07:24:12.435413+0000","last_undegraded":"2024-03-06T07:24:12.435413+0000","last_fullsized":"2024-03-06T07:24:12.435413+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:23:18.286691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706584+0000","last_change":"2024-03-06T07:24:09.862873+0000","last_active":"2024-03-06T07:24:12.706584+0000","last_peered":"2024-03-06T07:24:12.706584+0000","last_clean":"2024-03-06T07:24:12.706584+0000","last_became_active":"2024-03-06T07:24:09.862595+0000","last_became_peered":"2024-03-06T07:24:09.862595+0000","last_unstale":"2024-03-06T07:24:12.706584+0000","last_undegraded":"2024-03-06T07:24:12.706584+0000","last_fullsized":"2024-03-06T07:24:12.706584+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:46.994899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706301+0000","last_change":"2024-03-06T07:24:10.871344+0000","last_active":"2024-03-06T07:24:12.706301+0000","last_peered":"2024-03-06T07:24:12.706301+0000","last_clean":"2024-03-06T07:24:12.706301+0000","last_became_active":"2024-03-06T07:24:10.871056+0000","last_became_peered":"2024-03-06T07:24:10.871056+0000","last_unstale":"2024-03-06T07:24:12.706301+0000","last_undegraded":"2024-03-06T07:24:12.706301+0000","last_fullsized":"2024-03-06T07:24:12.706301+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:18.273527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871187+0000","last_change":"2024-03-06T07:24:09.861836+0000","last_active":"2024-03-06T07:24:11.871187+0000","last_peered":"2024-03-06T07:24:11.871187+0000","last_clean":"2024-03-06T07:24:11.871187+0000","last_became_active":"2024-03-06T07:24:09.861533+0000","last_became_peered":"2024-03-06T07:24:09.861533+0000","last_unstale":"2024-03-06T07:24:11.871187+0000","last_undegraded":"2024-03-06T07:24:11.871187+0000","last_fullsized":"2024-03-06T07:24:11.871187+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:42.678582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871180+0000","last_change":"2024-03-06T07:24:10.877007+0000","last_active":"2024-03-06T07:24:11.871180+0000","last_peered":"2024-03-06T07:24:11.871180+0000","last_clean":"2024-03-06T07:24:11.871180+0000","last_became_active":"2024-03-06T07:24:10.876741+0000","last_became_peered":"2024-03-06T07:24:10.876741+0000","last_unstale":"2024-03-06T07:24:11.871180+0000","last_undegraded":"2024-03-06T07:24:11.871180+0000","last_fullsized":"2024-03-06T07:24:11.871180+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:13.966188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706950+0000","last_change":"2024-03-06T07:24:09.861809+0000","last_active":"2024-03-06T07:24:12.706950+0000","last_peered":"2024-03-06T07:24:12.706950+0000","last_clean":"2024-03-06T07:24:12.706950+0000","last_became_active":"2024-03-06T07:24:09.861479+0000","last_became_peered":"2024-03-06T07:24:09.861479+0000","last_unstale":"2024-03-06T07:24:12.706950+0000","last_undegraded":"2024-03-06T07:24:12.706950+0000","last_fullsized":"2024-03-06T07:24:12.706950+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:59.826052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870314+0000","last_change":"2024-03-06T07:24:10.877845+0000","last_active":"2024-03-06T07:24:11.870314+0000","last_peered":"2024-03-06T07:24:11.870314+0000","last_clean":"2024-03-06T07:24:11.870314+0000","last_became_active":"2024-03-06T07:24:10.877548+0000","last_became_peered":"2024-03-06T07:24:10.877548+0000","last_unstale":"2024-03-06T07:24:11.870314+0000","last_undegraded":"2024-03-06T07:24:11.870314+0000","last_fullsized":"2024-03-06T07:24:11.870314+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:00.925482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707341+0000","last_change":"2024-03-06T07:24:09.863436+0000","last_active":"2024-03-06T07:24:12.707341+0000","last_peered":"2024-03-06T07:24:12.707341+0000","last_clean":"2024-03-06T07:24:12.707341+0000","last_became_active":"2024-03-06T07:24:09.863108+0000","last_became_peered":"2024-03-06T07:24:09.863108+0000","last_unstale":"2024-03-06T07:24:12.707341+0000","last_undegraded":"2024-03-06T07:24:12.707341+0000","last_fullsized":"2024-03-06T07:24:12.707341+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:19.077946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706562+0000","last_change":"2024-03-06T07:24:10.878783+0000","last_active":"2024-03-06T07:24:12.706562+0000","last_peered":"2024-03-06T07:24:12.706562+0000","last_clean":"2024-03-06T07:24:12.706562+0000","last_became_active":"2024-03-06T07:24:10.877971+0000","last_became_peered":"2024-03-06T07:24:10.877971+0000","last_unstale":"2024-03-06T07:24:12.706562+0000","last_undegraded":"2024-03-06T07:24:12.706562+0000","last_fullsized":"2024-03-06T07:24:12.706562+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:49.026769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706889+0000","last_change":"2024-03-06T07:24:09.865101+0000","last_active":"2024-03-06T07:24:12.706889+0000","last_peered":"2024-03-06T07:24:12.706889+0000","last_clean":"2024-03-06T07:24:12.706889+0000","last_became_active":"2024-03-06T07:24:09.864768+0000","last_became_peered":"2024-03-06T07:24:09.864768+0000","last_unstale":"2024-03-06T07:24:12.706889+0000","last_undegraded":"2024-03-06T07:24:12.706889+0000","last_fullsized":"2024-03-06T07:24:12.706889+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:39.734895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707173+0000","last_change":"2024-03-06T07:24:10.876317+0000","last_active":"2024-03-06T07:24:12.707173+0000","last_peered":"2024-03-06T07:24:12.707173+0000","last_clean":"2024-03-06T07:24:12.707173+0000","last_became_active":"2024-03-06T07:24:10.876038+0000","last_became_peered":"2024-03-06T07:24:10.876038+0000","last_unstale":"2024-03-06T07:24:12.707173+0000","last_undegraded":"2024-03-06T07:24:12.707173+0000","last_fullsized":"2024-03-06T07:24:12.707173+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:43.682548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707067+0000","last_change":"2024-03-06T07:24:09.868915+0000","last_active":"2024-03-06T07:24:12.707067+0000","last_peered":"2024-03-06T07:24:12.707067+0000","last_clean":"2024-03-06T07:24:12.707067+0000","last_became_active":"2024-03-06T07:24:09.868646+0000","last_became_peered":"2024-03-06T07:24:09.868646+0000","last_unstale":"2024-03-06T07:24:12.707067+0000","last_undegraded":"2024-03-06T07:24:12.707067+0000","last_fullsized":"2024-03-06T07:24:12.707067+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:16.002346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707732+0000","last_change":"2024-03-06T07:24:10.884248+0000","last_active":"2024-03-06T07:24:12.707732+0000","last_peered":"2024-03-06T07:24:12.707732+0000","last_clean":"2024-03-06T07:24:12.707732+0000","last_became_active":"2024-03-06T07:24:10.883814+0000","last_became_peered":"2024-03-06T07:24:10.883814+0000","last_unstale":"2024-03-06T07:24:12.707732+0000","last_undegraded":"2024-03-06T07:24:12.707732+0000","last_fullsized":"2024-03-06T07:24:12.707732+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:12.210056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707672+0000","last_change":"2024-03-06T07:24:09.863228+0000","last_active":"2024-03-06T07:24:12.707672+0000","last_peered":"2024-03-06T07:24:12.707672+0000","last_clean":"2024-03-06T07:24:12.707672+0000","last_became_active":"2024-03-06T07:24:09.862939+0000","last_became_peered":"2024-03-06T07:24:09.862939+0000","last_unstale":"2024-03-06T07:24:12.707672+0000","last_undegraded":"2024-03-06T07:24:12.707672+0000","last_fullsized":"2024-03-06T07:24:12.707672+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:16.002346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870346+0000","last_change":"2024-03-06T07:24:10.887763+0000","last_active":"2024-03-06T07:24:11.870346+0000","last_peered":"2024-03-06T07:24:11.870346+0000","last_clean":"2024-03-06T07:24:11.870346+0000","last_became_active":"2024-03-06T07:24:10.887434+0000","last_became_peered":"2024-03-06T07:24:10.887434+0000","last_unstale":"2024-03-06T07:24:11.870346+0000","last_undegraded":"2024-03-06T07:24:11.870346+0000","last_fullsized":"2024-03-06T07:24:11.870346+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:12.441745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"107'3","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707626+0000","last_change":"2024-03-06T07:24:09.865500+0000","last_active":"2024-03-06T07:24:12.707626+0000","last_peered":"2024-03-06T07:24:12.707626+0000","last_clean":"2024-03-06T07:24:12.707626+0000","last_became_active":"2024-03-06T07:24:09.865216+0000","last_became_peered":"2024-03-06T07:24:09.865216+0000","last_unstale":"2024-03-06T07:24:12.707626+0000","last_undegraded":"2024-03-06T07:24:12.707626+0000","last_fullsized":"2024-03-06T07:24:12.707626+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:56.308026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707911+0000","last_change":"2024-03-06T07:24:10.878231+0000","last_active":"2024-03-06T07:24:12.707911+0000","last_peered":"2024-03-06T07:24:12.707911+0000","last_clean":"2024-03-06T07:24:12.707911+0000","last_became_active":"2024-03-06T07:24:10.876336+0000","last_became_peered":"2024-03-06T07:24:10.876336+0000","last_unstale":"2024-03-06T07:24:12.707911+0000","last_undegraded":"2024-03-06T07:24:12.707911+0000","last_fullsized":"2024-03-06T07:24:12.707911+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:55.863177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"107'6","reported_seq":20,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870113+0000","last_change":"2024-03-06T07:24:09.862669+0000","last_active":"2024-03-06T07:24:11.870113+0000","last_peered":"2024-03-06T07:24:11.870113+0000","last_clean":"2024-03-06T07:24:11.870113+0000","last_became_active":"2024-03-06T07:24:09.862371+0000","last_became_peered":"2024-03-06T07:24:09.862371+0000","last_unstale":"2024-03-06T07:24:11.870113+0000","last_undegraded":"2024-03-06T07:24:11.870113+0000","last_fullsized":"2024-03-06T07:24:11.870113+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:47.572754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707472+0000","last_change":"2024-03-06T07:24:10.882090+0000","last_active":"2024-03-06T07:24:12.707472+0000","last_peered":"2024-03-06T07:24:12.707472+0000","last_clean":"2024-03-06T07:24:12.707472+0000","last_became_active":"2024-03-06T07:24:10.881824+0000","last_became_peered":"2024-03-06T07:24:10.881824+0000","last_unstale":"2024-03-06T07:24:12.707472+0000","last_undegraded":"2024-03-06T07:24:12.707472+0000","last_fullsized":"2024-03-06T07:24:12.707472+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:15.401455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870146+0000","last_change":"2024-03-06T07:24:09.863186+0000","last_active":"2024-03-06T07:24:11.870146+0000","last_peered":"2024-03-06T07:24:11.870146+0000","last_clean":"2024-03-06T07:24:11.870146+0000","last_became_active":"2024-03-06T07:24:09.862898+0000","last_became_peered":"2024-03-06T07:24:09.862898+0000","last_unstale":"2024-03-06T07:24:11.870146+0000","last_undegraded":"2024-03-06T07:24:11.870146+0000","last_fullsized":"2024-03-06T07:24:11.870146+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:13.092436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706362+0000","last_change":"2024-03-06T07:24:10.863118+0000","last_active":"2024-03-06T07:24:12.706362+0000","last_peered":"2024-03-06T07:24:12.706362+0000","last_clean":"2024-03-06T07:24:12.706362+0000","last_became_active":"2024-03-06T07:24:10.862829+0000","last_became_peered":"2024-03-06T07:24:10.862829+0000","last_unstale":"2024-03-06T07:24:12.706362+0000","last_undegraded":"2024-03-06T07:24:12.706362+0000","last_fullsized":"2024-03-06T07:24:12.706362+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:16.962306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870390+0000","last_change":"2024-03-06T07:24:09.864203+0000","last_active":"2024-03-06T07:24:11.870390+0000","last_peered":"2024-03-06T07:24:11.870390+0000","last_clean":"2024-03-06T07:24:11.870390+0000","last_became_active":"2024-03-06T07:24:09.863923+0000","last_became_peered":"2024-03-06T07:24:09.863923+0000","last_unstale":"2024-03-06T07:24:11.870390+0000","last_undegraded":"2024-03-06T07:24:11.870390+0000","last_fullsized":"2024-03-06T07:24:11.870390+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:28.983380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.867291+0000","last_change":"2024-03-06T07:24:10.874735+0000","last_active":"2024-03-06T07:24:11.867291+0000","last_peered":"2024-03-06T07:24:11.867291+0000","last_clean":"2024-03-06T07:24:11.867291+0000","last_became_active":"2024-03-06T07:24:10.874478+0000","last_became_peered":"2024-03-06T07:24:10.874478+0000","last_unstale":"2024-03-06T07:24:11.867291+0000","last_undegraded":"2024-03-06T07:24:11.867291+0000","last_fullsized":"2024-03-06T07:24:11.867291+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:07.054527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870038+0000","last_change":"2024-03-06T07:24:09.862545+0000","last_active":"2024-03-06T07:24:11.870038+0000","last_peered":"2024-03-06T07:24:11.870038+0000","last_clean":"2024-03-06T07:24:11.870038+0000","last_became_active":"2024-03-06T07:24:09.862257+0000","last_became_peered":"2024-03-06T07:24:09.862257+0000","last_unstale":"2024-03-06T07:24:11.870038+0000","last_undegraded":"2024-03-06T07:24:11.870038+0000","last_fullsized":"2024-03-06T07:24:11.870038+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:29.252821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707654+0000","last_change":"2024-03-06T07:24:10.878777+0000","last_active":"2024-03-06T07:24:12.707654+0000","last_peered":"2024-03-06T07:24:12.707654+0000","last_clean":"2024-03-06T07:24:12.707654+0000","last_became_active":"2024-03-06T07:24:10.878513+0000","last_became_peered":"2024-03-06T07:24:10.878513+0000","last_unstale":"2024-03-06T07:24:12.707654+0000","last_undegraded":"2024-03-06T07:24:12.707654+0000","last_fullsized":"2024-03-06T07:24:12.707654+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:13.966188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706742+0000","last_change":"2024-03-06T07:24:09.868594+0000","last_active":"2024-03-06T07:24:12.706742+0000","last_peered":"2024-03-06T07:24:12.706742+0000","last_clean":"2024-03-06T07:24:12.706742+0000","last_became_active":"2024-03-06T07:24:09.868301+0000","last_became_peered":"2024-03-06T07:24:09.868301+0000","last_unstale":"2024-03-06T07:24:12.706742+0000","last_undegraded":"2024-03-06T07:24:12.706742+0000","last_fullsized":"2024-03-06T07:24:12.706742+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:53.520082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871428+0000","last_change":"2024-03-06T07:24:10.880703+0000","last_active":"2024-03-06T07:24:11.871428+0000","last_peered":"2024-03-06T07:24:11.871428+0000","last_clean":"2024-03-06T07:24:11.871428+0000","last_became_active":"2024-03-06T07:24:10.880396+0000","last_became_peered":"2024-03-06T07:24:10.880396+0000","last_unstale":"2024-03-06T07:24:11.871428+0000","last_undegraded":"2024-03-06T07:24:11.871428+0000","last_fullsized":"2024-03-06T07:24:11.871428+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:06.797882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.867452+0000","last_change":"2024-03-06T07:24:09.856421+0000","last_active":"2024-03-06T07:24:11.867452+0000","last_peered":"2024-03-06T07:24:11.867452+0000","last_clean":"2024-03-06T07:24:11.867452+0000","last_became_active":"2024-03-06T07:24:09.856150+0000","last_became_peered":"2024-03-06T07:24:09.856150+0000","last_unstale":"2024-03-06T07:24:11.867452+0000","last_undegraded":"2024-03-06T07:24:11.867452+0000","last_fullsized":"2024-03-06T07:24:11.867452+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:41.339935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707136+0000","last_change":"2024-03-06T07:24:10.874131+0000","last_active":"2024-03-06T07:24:12.707136+0000","last_peered":"2024-03-06T07:24:12.707136+0000","last_clean":"2024-03-06T07:24:12.707136+0000","last_became_active":"2024-03-06T07:24:10.873870+0000","last_became_peered":"2024-03-06T07:24:10.873870+0000","last_unstale":"2024-03-06T07:24:12.707136+0000","last_undegraded":"2024-03-06T07:24:12.707136+0000","last_fullsized":"2024-03-06T07:24:12.707136+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:26.211626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435259+0000","last_change":"2024-03-06T07:24:09.866590+0000","last_active":"2024-03-06T07:24:12.435259+0000","last_peered":"2024-03-06T07:24:12.435259+0000","last_clean":"2024-03-06T07:24:12.435259+0000","last_became_active":"2024-03-06T07:24:09.866219+0000","last_became_peered":"2024-03-06T07:24:09.866219+0000","last_unstale":"2024-03-06T07:24:12.435259+0000","last_undegraded":"2024-03-06T07:24:12.435259+0000","last_fullsized":"2024-03-06T07:24:12.435259+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:17.269562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707999+0000","last_change":"2024-03-06T07:24:10.868999+0000","last_active":"2024-03-06T07:24:12.707999+0000","last_peered":"2024-03-06T07:24:12.707999+0000","last_clean":"2024-03-06T07:24:12.707999+0000","last_became_active":"2024-03-06T07:24:10.868721+0000","last_became_peered":"2024-03-06T07:24:10.868721+0000","last_unstale":"2024-03-06T07:24:12.707999+0000","last_undegraded":"2024-03-06T07:24:12.707999+0000","last_fullsized":"2024-03-06T07:24:12.707999+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:14.096402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435054+0000","last_change":"2024-03-06T07:24:09.868739+0000","last_active":"2024-03-06T07:24:12.435054+0000","last_peered":"2024-03-06T07:24:12.435054+0000","last_clean":"2024-03-06T07:24:12.435054+0000","last_became_active":"2024-03-06T07:24:09.868468+0000","last_became_peered":"2024-03-06T07:24:09.868468+0000","last_unstale":"2024-03-06T07:24:12.435054+0000","last_undegraded":"2024-03-06T07:24:12.435054+0000","last_fullsized":"2024-03-06T07:24:12.435054+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:26.906353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706946+0000","last_change":"2024-03-06T07:24:10.873557+0000","last_active":"2024-03-06T07:24:12.706946+0000","last_peered":"2024-03-06T07:24:12.706946+0000","last_clean":"2024-03-06T07:24:12.706946+0000","last_became_active":"2024-03-06T07:24:10.873300+0000","last_became_peered":"2024-03-06T07:24:10.873300+0000","last_unstale":"2024-03-06T07:24:12.706946+0000","last_undegraded":"2024-03-06T07:24:12.706946+0000","last_fullsized":"2024-03-06T07:24:12.706946+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:27.985129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706587+0000","last_change":"2024-03-06T07:24:09.866393+0000","last_active":"2024-03-06T07:24:12.706587+0000","last_peered":"2024-03-06T07:24:12.706587+0000","last_clean":"2024-03-06T07:24:12.706587+0000","last_became_active":"2024-03-06T07:24:09.866039+0000","last_became_peered":"2024-03-06T07:24:09.866039+0000","last_unstale":"2024-03-06T07:24:12.706587+0000","last_undegraded":"2024-03-06T07:24:12.706587+0000","last_fullsized":"2024-03-06T07:24:12.706587+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:13.805366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.869979+0000","last_change":"2024-03-06T07:24:10.889993+0000","last_active":"2024-03-06T07:24:11.869979+0000","last_peered":"2024-03-06T07:24:11.869979+0000","last_clean":"2024-03-06T07:24:11.869979+0000","last_became_active":"2024-03-06T07:24:10.889789+0000","last_became_peered":"2024-03-06T07:24:10.889789+0000","last_unstale":"2024-03-06T07:24:11.869979+0000","last_undegraded":"2024-03-06T07:24:11.869979+0000","last_fullsized":"2024-03-06T07:24:11.869979+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:55.733709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.434752+0000","last_change":"2024-03-06T07:24:09.865474+0000","last_active":"2024-03-06T07:24:12.434752+0000","last_peered":"2024-03-06T07:24:12.434752+0000","last_clean":"2024-03-06T07:24:12.434752+0000","last_became_active":"2024-03-06T07:24:09.865141+0000","last_became_peered":"2024-03-06T07:24:09.865141+0000","last_unstale":"2024-03-06T07:24:12.434752+0000","last_undegraded":"2024-03-06T07:24:12.434752+0000","last_fullsized":"2024-03-06T07:24:12.434752+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:20.460653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870454+0000","last_change":"2024-03-06T07:24:10.883194+0000","last_active":"2024-03-06T07:24:11.870454+0000","last_peered":"2024-03-06T07:24:11.870454+0000","last_clean":"2024-03-06T07:24:11.870454+0000","last_became_active":"2024-03-06T07:24:10.882932+0000","last_became_peered":"2024-03-06T07:24:10.882932+0000","last_unstale":"2024-03-06T07:24:11.870454+0000","last_undegraded":"2024-03-06T07:24:11.870454+0000","last_fullsized":"2024-03-06T07:24:11.870454+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:05.049220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870108+0000","last_change":"2024-03-06T07:24:09.867221+0000","last_active":"2024-03-06T07:24:11.870108+0000","last_peered":"2024-03-06T07:24:11.870108+0000","last_clean":"2024-03-06T07:24:11.870108+0000","last_became_active":"2024-03-06T07:24:09.866916+0000","last_became_peered":"2024-03-06T07:24:09.866916+0000","last_unstale":"2024-03-06T07:24:11.870108+0000","last_undegraded":"2024-03-06T07:24:11.870108+0000","last_fullsized":"2024-03-06T07:24:11.870108+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:08.126867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707572+0000","last_change":"2024-03-06T07:24:10.876647+0000","last_active":"2024-03-06T07:24:12.707572+0000","last_peered":"2024-03-06T07:24:12.707572+0000","last_clean":"2024-03-06T07:24:12.707572+0000","last_became_active":"2024-03-06T07:24:10.876396+0000","last_became_peered":"2024-03-06T07:24:10.876396+0000","last_unstale":"2024-03-06T07:24:12.707572+0000","last_undegraded":"2024-03-06T07:24:12.707572+0000","last_fullsized":"2024-03-06T07:24:12.707572+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:16.962306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.434591+0000","last_change":"2024-03-06T07:24:09.865489+0000","last_active":"2024-03-06T07:24:12.434591+0000","last_peered":"2024-03-06T07:24:12.434591+0000","last_clean":"2024-03-06T07:24:12.434591+0000","last_became_active":"2024-03-06T07:24:09.865110+0000","last_became_peered":"2024-03-06T07:24:09.865110+0000","last_unstale":"2024-03-06T07:24:12.434591+0000","last_undegraded":"2024-03-06T07:24:12.434591+0000","last_fullsized":"2024-03-06T07:24:12.434591+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:08.965620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707339+0000","last_change":"2024-03-06T07:24:10.882191+0000","last_active":"2024-03-06T07:24:12.707339+0000","last_peered":"2024-03-06T07:24:12.707339+0000","last_clean":"2024-03-06T07:24:12.707339+0000","last_became_active":"2024-03-06T07:24:10.881923+0000","last_became_peered":"2024-03-06T07:24:10.881923+0000","last_unstale":"2024-03-06T07:24:12.707339+0000","last_undegraded":"2024-03-06T07:24:12.707339+0000","last_fullsized":"2024-03-06T07:24:12.707339+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:15.519406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706764+0000","last_change":"2024-03-06T07:24:09.867744+0000","last_active":"2024-03-06T07:24:12.706764+0000","last_peered":"2024-03-06T07:24:12.706764+0000","last_clean":"2024-03-06T07:24:12.706764+0000","last_became_active":"2024-03-06T07:24:09.867379+0000","last_became_peered":"2024-03-06T07:24:09.867379+0000","last_unstale":"2024-03-06T07:24:12.706764+0000","last_undegraded":"2024-03-06T07:24:12.706764+0000","last_fullsized":"2024-03-06T07:24:12.706764+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:01.000646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706529+0000","last_change":"2024-03-06T07:24:10.872169+0000","last_active":"2024-03-06T07:24:12.706529+0000","last_peered":"2024-03-06T07:24:12.706529+0000","last_clean":"2024-03-06T07:24:12.706529+0000","last_became_active":"2024-03-06T07:24:10.871741+0000","last_became_peered":"2024-03-06T07:24:10.871741+0000","last_unstale":"2024-03-06T07:24:12.706529+0000","last_undegraded":"2024-03-06T07:24:12.706529+0000","last_fullsized":"2024-03-06T07:24:12.706529+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:12.210056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"108'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707153+0000","last_change":"2024-03-06T07:24:09.867490+0000","last_active":"2024-03-06T07:24:12.707153+0000","last_peered":"2024-03-06T07:24:12.707153+0000","last_clean":"2024-03-06T07:24:12.707153+0000","last_became_active":"2024-03-06T07:24:09.867201+0000","last_became_peered":"2024-03-06T07:24:09.867201+0000","last_unstale":"2024-03-06T07:24:12.707153+0000","last_undegraded":"2024-03-06T07:24:12.707153+0000","last_fullsized":"2024-03-06T07:24:12.707153+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:40.272791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435321+0000","last_change":"2024-03-06T07:24:10.873473+0000","last_active":"2024-03-06T07:24:12.435321+0000","last_peered":"2024-03-06T07:24:12.435321+0000","last_clean":"2024-03-06T07:24:12.435321+0000","last_became_active":"2024-03-06T07:24:10.873196+0000","last_became_peered":"2024-03-06T07:24:10.873196+0000","last_unstale":"2024-03-06T07:24:12.435321+0000","last_undegraded":"2024-03-06T07:24:12.435321+0000","last_fullsized":"2024-03-06T07:24:12.435321+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:10.566732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707436+0000","last_change":"2024-03-06T07:24:09.869094+0000","last_active":"2024-03-06T07:24:12.707436+0000","last_peered":"2024-03-06T07:24:12.707436+0000","last_clean":"2024-03-06T07:24:12.707436+0000","last_became_active":"2024-03-06T07:24:09.868745+0000","last_became_peered":"2024-03-06T07:24:09.868745+0000","last_unstale":"2024-03-06T07:24:12.707436+0000","last_undegraded":"2024-03-06T07:24:12.707436+0000","last_fullsized":"2024-03-06T07:24:12.707436+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:19.077946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.867544+0000","last_change":"2024-03-06T07:24:10.871045+0000","last_active":"2024-03-06T07:24:11.867544+0000","last_peered":"2024-03-06T07:24:11.867544+0000","last_clean":"2024-03-06T07:24:11.867544+0000","last_became_active":"2024-03-06T07:24:10.870734+0000","last_became_peered":"2024-03-06T07:24:10.870734+0000","last_unstale":"2024-03-06T07:24:11.867544+0000","last_undegraded":"2024-03-06T07:24:11.867544+0000","last_fullsized":"2024-03-06T07:24:11.867544+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:38.020645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.707380+0000","last_change":"2024-03-06T07:24:09.864124+0000","last_active":"2024-03-06T07:24:12.707380+0000","last_peered":"2024-03-06T07:24:12.707380+0000","last_clean":"2024-03-06T07:24:12.707380+0000","last_became_active":"2024-03-06T07:24:09.863447+0000","last_became_peered":"2024-03-06T07:24:09.863447+0000","last_unstale":"2024-03-06T07:24:12.707380+0000","last_undegraded":"2024-03-06T07:24:12.707380+0000","last_fullsized":"2024-03-06T07:24:12.707380+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:48.704351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706842+0000","last_change":"2024-03-06T07:24:10.881045+0000","last_active":"2024-03-06T07:24:12.706842+0000","last_peered":"2024-03-06T07:24:12.706842+0000","last_clean":"2024-03-06T07:24:12.706842+0000","last_became_active":"2024-03-06T07:24:10.880707+0000","last_became_peered":"2024-03-06T07:24:10.880707+0000","last_unstale":"2024-03-06T07:24:12.706842+0000","last_undegraded":"2024-03-06T07:24:12.706842+0000","last_fullsized":"2024-03-06T07:24:12.706842+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:48.525994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871301+0000","last_change":"2024-03-06T07:24:09.861100+0000","last_active":"2024-03-06T07:24:11.871301+0000","last_peered":"2024-03-06T07:24:11.871301+0000","last_clean":"2024-03-06T07:24:11.871301+0000","last_became_active":"2024-03-06T07:24:09.860790+0000","last_became_peered":"2024-03-06T07:24:09.860790+0000","last_unstale":"2024-03-06T07:24:11.871301+0000","last_undegraded":"2024-03-06T07:24:11.871301+0000","last_fullsized":"2024-03-06T07:24:11.871301+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:08.129149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.870054+0000","last_change":"2024-03-06T07:24:10.884752+0000","last_active":"2024-03-06T07:24:11.870054+0000","last_peered":"2024-03-06T07:24:11.870054+0000","last_clean":"2024-03-06T07:24:11.870054+0000","last_became_active":"2024-03-06T07:24:10.884463+0000","last_became_peered":"2024-03-06T07:24:10.884463+0000","last_unstale":"2024-03-06T07:24:11.870054+0000","last_undegraded":"2024-03-06T07:24:11.870054+0000","last_fullsized":"2024-03-06T07:24:11.870054+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:03.883574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.871613+0000","last_change":"2024-03-06T07:24:09.866939+0000","last_active":"2024-03-06T07:24:11.871613+0000","last_peered":"2024-03-06T07:24:11.871613+0000","last_clean":"2024-03-06T07:24:11.871613+0000","last_became_active":"2024-03-06T07:24:09.866561+0000","last_became_peered":"2024-03-06T07:24:09.866561+0000","last_unstale":"2024-03-06T07:24:11.871613+0000","last_undegraded":"2024-03-06T07:24:11.871613+0000","last_fullsized":"2024-03-06T07:24:11.871613+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:47.869257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.434859+0000","last_change":"2024-03-06T07:24:10.876783+0000","last_active":"2024-03-06T07:24:12.434859+0000","last_peered":"2024-03-06T07:24:12.434859+0000","last_clean":"2024-03-06T07:24:12.434859+0000","last_became_active":"2024-03-06T07:24:10.875768+0000","last_became_peered":"2024-03-06T07:24:10.875768+0000","last_unstale":"2024-03-06T07:24:12.434859+0000","last_undegraded":"2024-03-06T07:24:12.434859+0000","last_fullsized":"2024-03-06T07:24:12.434859+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:09.130833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706649+0000","last_change":"2024-03-06T07:24:09.866286+0000","last_active":"2024-03-06T07:24:12.706649+0000","last_peered":"2024-03-06T07:24:12.706649+0000","last_clean":"2024-03-06T07:24:12.706649+0000","last_became_active":"2024-03-06T07:24:09.865961+0000","last_became_peered":"2024-03-06T07:24:09.865961+0000","last_unstale":"2024-03-06T07:24:12.706649+0000","last_undegraded":"2024-03-06T07:24:12.706649+0000","last_fullsized":"2024-03-06T07:24:12.706649+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:44.450400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.706694+0000","last_change":"2024-03-06T07:24:10.865615+0000","last_active":"2024-03-06T07:24:12.706694+0000","last_peered":"2024-03-06T07:24:12.706694+0000","last_clean":"2024-03-06T07:24:12.706694+0000","last_became_active":"2024-03-06T07:24:10.865340+0000","last_became_peered":"2024-03-06T07:24:10.865340+0000","last_unstale":"2024-03-06T07:24:12.706694+0000","last_undegraded":"2024-03-06T07:24:12.706694+0000","last_fullsized":"2024-03-06T07:24:12.706694+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:55.863177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.434602+0000","last_change":"2024-03-06T07:24:09.865914+0000","last_active":"2024-03-06T07:24:12.434602+0000","last_peered":"2024-03-06T07:24:12.434602+0000","last_clean":"2024-03-06T07:24:12.434602+0000","last_became_active":"2024-03-06T07:24:09.865626+0000","last_became_peered":"2024-03-06T07:24:09.865626+0000","last_unstale":"2024-03-06T07:24:12.434602+0000","last_undegraded":"2024-03-06T07:24:12.434602+0000","last_fullsized":"2024-03-06T07:24:12.434602+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:55.962450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.435120+0000","last_change":"2024-03-06T07:24:10.868167+0000","last_active":"2024-03-06T07:24:12.435120+0000","last_peered":"2024-03-06T07:24:12.435120+0000","last_clean":"2024-03-06T07:24:12.435120+0000","last_became_active":"2024-03-06T07:24:10.867887+0000","last_became_peered":"2024-03-06T07:24:10.867887+0000","last_unstale":"2024-03-06T07:24:12.435120+0000","last_undegraded":"2024-03-06T07:24:12.435120+0000","last_fullsized":"2024-03-06T07:24:12.435120+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.842470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.842470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:55.303316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"107'8","reported_seq":22,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.869896+0000","last_change":"2024-03-06T07:24:09.863372+0000","last_active":"2024-03-06T07:24:11.869896+0000","last_peered":"2024-03-06T07:24:11.869896+0000","last_clean":"2024-03-06T07:24:11.869896+0000","last_became_active":"2024-03-06T07:24:09.863011+0000","last_became_peered":"2024-03-06T07:24:09.863011+0000","last_unstale":"2024-03-06T07:24:11.869896+0000","last_undegraded":"2024-03-06T07:24:11.869896+0000","last_fullsized":"2024-03-06T07:24:11.869896+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:08.838505+0000","last_clean_scrub_stamp":"2024-03-06T07:24:08.838505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:54.859211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":339,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.434740+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:24:12.434740+0000","last_peered":"2024-03-06T07:24:12.434740+0000","last_clean":"2024-03-06T07:24:12.434740+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:24:12.434740+0000","last_undegraded":"2024-03-06T07:24:12.434740+0000","last_fullsized":"2024-03-06T07:24:12.434740+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574899,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1429504,"data_stored":1241574,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1351680,"data_stored":651012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1355776,"data_stored":650590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574899,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1359872,"data_stored":652042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1363968,"data_stored":652076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":650430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1351680,"data_stored":650452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574899,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1417216,"data_stored":1240820,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:18.681 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:24:18.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:24:18.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.824+0000 7f47479f7640 1 Processor -- start 2024-03-06T07:24:18.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.824+0000 7f47479f7640 1 -- start start 2024-03-06T07:24:18.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.825+0000 7f47479f7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4740152650 0x7f4740155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.825+0000 7f47479f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47400bf9d0 0x7f47400c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.825+0000 7f47479f7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47400c2fc0 con 0x7f47400c6b10 2024-03-06T07:24:18.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.825+0000 7f47479f7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4740155800 con 0x7f4740152650 2024-03-06T07:24:18.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.825+0000 7f47479f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4740158da0 con 0x7f47400bf9d0 2024-03-06T07:24:18.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.826+0000 7f4745793640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4740152650 0x7f4740155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.826+0000 7f4745f94640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47400bf9d0 0x7f47400c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.826+0000 7f4745f94640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47400bf9d0 0x7f47400c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:34378/0 (socket says 172.21.15.1:34378) 2024-03-06T07:24:18.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.826+0000 7f4745793640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4740152650 0x7f4740155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:46710/0 (socket says 172.21.15.1:46710) 2024-03-06T07:24:18.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.826+0000 7f4744f92640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f47400c6b10 0x7f4740158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:43770/0 (socket says 172.21.15.1:43770) 2024-03-06T07:24:18.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.826+0000 7f4745f94640 1 -- 172.21.15.1:0/1626510930 learned_addr learned my addr 172.21.15.1:0/1626510930 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.826+0000 7f4745793640 1 -- 172.21.15.1:0/1626510930 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47400bf9d0 msgr2=0x7f47400c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.826+0000 7f4745793640 1 --2- 172.21.15.1:0/1626510930 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47400bf9d0 0x7f47400c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.826+0000 7f4745793640 1 -- 172.21.15.1:0/1626510930 >> v1:172.21.15.98:6789/0 conn(0x7f47400c6b10 legacy=0x7f4740158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.826+0000 7f4737fff640 1 -- 172.21.15.1:0/1626510930 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157056741 0 0) 0x7f47400c2fc0 con 0x7f47400c6b10 2024-03-06T07:24:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.826+0000 7f4745793640 1 -- 172.21.15.1:0/1626510930 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47400c6690 con 0x7f4740152650 2024-03-06T07:24:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.826+0000 7f4745f94640 1 --2- 172.21.15.1:0/1626510930 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47400bf9d0 0x7f47400c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.827+0000 7f4745793640 1 --2- 172.21.15.1:0/1626510930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4740152650 0x7f4740155c90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f473c0097a0 tx=0x7f473c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4abfe071cb7b920 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.827+0000 7f4737fff640 1 -- 172.21.15.1:0/1626510930 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c0315a0 con 0x7f4740152650 2024-03-06T07:24:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.827+0000 7f4737fff640 1 -- 172.21.15.1:0/1626510930 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473c037070 con 0x7f4740152650 2024-03-06T07:24:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.827+0000 7f4737fff640 1 -- 172.21.15.1:0/1626510930 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c031960 con 0x7f4740152650 2024-03-06T07:24:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.828+0000 7f47479f7640 1 -- 172.21.15.1:0/1626510930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4740152650 msgr2=0x7f4740155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.828+0000 7f47479f7640 1 --2- 172.21.15.1:0/1626510930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4740152650 0x7f4740155c90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f473c0097a0 tx=0x7f473c02b840 comp rx=0 tx=0).stop 2024-03-06T07:24:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.828+0000 7f47479f7640 1 -- 172.21.15.1:0/1626510930 shutdown_connections 2024-03-06T07:24:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.828+0000 7f47479f7640 1 --2- 172.21.15.1:0/1626510930 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47400bf9d0 0x7f47400c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.828+0000 7f47479f7640 1 --2- 172.21.15.1:0/1626510930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4740152650 0x7f4740155c90 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.828+0000 7f47479f7640 1 -- 172.21.15.1:0/1626510930 >> 172.21.15.1:0/1626510930 conn(0x7f4740066ee0 msgr2=0x7f47400672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.828+0000 7f47479f7640 1 -- 172.21.15.1:0/1626510930 shutdown_connections 2024-03-06T07:24:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.828+0000 7f47479f7640 1 -- 172.21.15.1:0/1626510930 wait complete. 2024-03-06T07:24:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.829+0000 7f47479f7640 1 Processor -- start 2024-03-06T07:24:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.829+0000 7f47479f7640 1 -- start start 2024-03-06T07:24:18.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f47479f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47400bf9d0 0x7f47401ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f47479f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4740152650 0x7f474014fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f47479f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47400c5a00 0x7f47401e0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f47479f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47401551b0 con 0x7f47400bf9d0 2024-03-06T07:24:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f47479f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47400c6fa0 con 0x7f4740152650 2024-03-06T07:24:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f47479f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4740155490 con 0x7f47400c5a00 2024-03-06T07:24:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f4745793640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47400bf9d0 0x7f47401ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f4745793640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47400bf9d0 0x7f47401ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44328/0 (socket says 172.21.15.1:44328) 2024-03-06T07:24:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f4745793640 1 -- 172.21.15.1:0/106466162 learned_addr learned my addr 172.21.15.1:0/106466162 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f4745f94640 1 --2- 172.21.15.1:0/106466162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47400c5a00 0x7f47401e0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f4744f92640 1 --2- 172.21.15.1:0/106466162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4740152650 0x7f474014fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f4745793640 1 -- 172.21.15.1:0/106466162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47400c5a00 msgr2=0x7f47401e0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f4745793640 1 --2- 172.21.15.1:0/106466162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47400c5a00 0x7f47401e0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f4745793640 1 -- 172.21.15.1:0/106466162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4740152650 msgr2=0x7f474014fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f4745793640 1 --2- 172.21.15.1:0/106466162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4740152650 0x7f474014fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f4745793640 1 -- 172.21.15.1:0/106466162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4740156490 con 0x7f47400bf9d0 2024-03-06T07:24:18.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f4745f94640 1 --2- 172.21.15.1:0/106466162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47400c5a00 0x7f47401e0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:18.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f4744f92640 1 --2- 172.21.15.1:0/106466162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4740152650 0x7f474014fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:18.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.830+0000 7f4745793640 1 --2- 172.21.15.1:0/106466162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47400bf9d0 0x7f47401ee7e0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f473c0097a0 tx=0x7f473c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.831+0000 7f4735ffb640 1 -- 172.21.15.1:0/106466162 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c004230 con 0x7f47400bf9d0 2024-03-06T07:24:18.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.831+0000 7f4735ffb640 1 -- 172.21.15.1:0/106466162 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473c0043d0 con 0x7f47400bf9d0 2024-03-06T07:24:18.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.831+0000 7f47479f7640 1 -- 172.21.15.1:0/106466162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4740156770 con 0x7f47400bf9d0 2024-03-06T07:24:18.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.831+0000 7f4735ffb640 1 -- 172.21.15.1:0/106466162 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c003850 con 0x7f47400bf9d0 2024-03-06T07:24:18.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.831+0000 7f47479f7640 1 -- 172.21.15.1:0/106466162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47401e1460 con 0x7f47400bf9d0 2024-03-06T07:24:18.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.832+0000 7f47479f7640 1 -- 172.21.15.1:0/106466162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4740067fd0 con 0x7f47400bf9d0 2024-03-06T07:24:18.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.833+0000 7f4735ffb640 1 -- 172.21.15.1:0/106466162 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f473c0039f0 con 0x7f47400bf9d0 2024-03-06T07:24:18.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.833+0000 7f4735ffb640 1 --2- 172.21.15.1:0/106466162 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f472406b4d0 0x7f472406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.833+0000 7f4735ffb640 1 -- 172.21.15.1:0/106466162 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f473c02f030 con 0x7f47400bf9d0 2024-03-06T07:24:18.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.833+0000 7f4744f92640 1 --2- 172.21.15.1:0/106466162 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f472406b4d0 0x7f472406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.834+0000 7f4744f92640 1 --2- 172.21.15.1:0/106466162 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f472406b4d0 0x7f472406d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f4730007fd0 tx=0x7f4730009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:18.837+0000 7f4735ffb640 1 -- 172.21.15.1:0/106466162 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f473c041070 con 0x7f47400bf9d0 2024-03-06T07:24:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.097+0000 7f47479f7640 1 -- 172.21.15.1:0/106466162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f4740156940 con 0x7f47400bf9d0 2024-03-06T07:24:19.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.099+0000 7f4735ffb640 1 -- 172.21.15.1:0/106466162 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7f473c07b2a0 con 0x7f47400bf9d0 2024-03-06T07:24:19.101 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-06T07:24:19.101 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng== 2024-03-06T07:24:19.101 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-06T07:24:19.101 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-06T07:24:19.101 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:19.101 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-06T07:24:19.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.101+0000 7f47479f7640 1 -- 172.21.15.1:0/106466162 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f472406b4d0 msgr2=0x7f472406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.101+0000 7f47479f7640 1 --2- 172.21.15.1:0/106466162 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f472406b4d0 0x7f472406d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f4730007fd0 tx=0x7f4730009040 comp rx=0 tx=0).stop 2024-03-06T07:24:19.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.101+0000 7f47479f7640 1 -- 172.21.15.1:0/106466162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47400bf9d0 msgr2=0x7f47401ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.101+0000 7f47479f7640 1 --2- 172.21.15.1:0/106466162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47400bf9d0 0x7f47401ee7e0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f473c0097a0 tx=0x7f473c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.101+0000 7f47479f7640 1 -- 172.21.15.1:0/106466162 shutdown_connections 2024-03-06T07:24:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.101+0000 7f47479f7640 1 --2- 172.21.15.1:0/106466162 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f472406b4d0 0x7f472406d990 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.101+0000 7f47479f7640 1 --2- 172.21.15.1:0/106466162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47400c5a00 0x7f47401e0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.101+0000 7f47479f7640 1 --2- 172.21.15.1:0/106466162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4740152650 0x7f474014fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.101+0000 7f47479f7640 1 --2- 172.21.15.1:0/106466162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47400bf9d0 0x7f47401ee7e0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.101+0000 7f47479f7640 1 -- 172.21.15.1:0/106466162 >> 172.21.15.1:0/106466162 conn(0x7f4740066ee0 msgr2=0x7f47400c3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.101+0000 7f47479f7640 1 -- 172.21.15.1:0/106466162 shutdown_connections 2024-03-06T07:24:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.101+0000 7f47479f7640 1 -- 172.21.15.1:0/106466162 wait complete. 2024-03-06T07:24:19.114 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:19.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.213+0000 7f1177f7a640 1 Processor -- start 2024-03-06T07:24:19.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.213+0000 7f1177f7a640 1 -- start start 2024-03-06T07:24:19.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.214+0000 7f1177f7a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11701543a0 0x7f117015e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.214+0000 7f1177f7a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1170150900 con 0x7f11700c32f0 2024-03-06T07:24:19.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.214+0000 7f1177f7a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1170071790 con 0x7f1170150520 2024-03-06T07:24:19.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.214+0000 7f1177f7a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1170153ed0 con 0x7f11701543a0 2024-03-06T07:24:19.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.214+0000 7f1176517640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11701543a0 0x7f117015e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.214+0000 7f1176517640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11701543a0 0x7f117015e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44344/0 (socket says 172.21.15.1:44344) 2024-03-06T07:24:19.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.214+0000 7f1175515640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f11700c32f0 0x7f1170070c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36034/0 (socket says 172.21.15.1:36034) 2024-03-06T07:24:19.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.214+0000 7f1176517640 1 -- 172.21.15.1:0/1263596469 learned_addr learned my addr 172.21.15.1:0/1263596469 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:19.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.214+0000 7f1176517640 1 -- 172.21.15.1:0/1263596469 >> v1:172.21.15.98:6789/0 conn(0x7f1170150520 legacy=0x7f11701537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.214+0000 7f1176517640 1 -- 172.21.15.1:0/1263596469 >> v1:172.21.15.1:6789/0 conn(0x7f11700c32f0 legacy=0x7f1170070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.214+0000 7f1176517640 1 -- 172.21.15.1:0/1263596469 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1170157050 con 0x7f11701543a0 2024-03-06T07:24:19.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.215+0000 7f1176517640 1 --2- 172.21.15.1:0/1263596469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11701543a0 0x7f117015e840 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f1168009e50 tx=0x7f116802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e16622a66b4e2c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.215+0000 7f1174d14640 1 -- 172.21.15.1:0/1263596469 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116802cb20 con 0x7f11701543a0 2024-03-06T07:24:19.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.215+0000 7f1174d14640 1 -- 172.21.15.1:0/1263596469 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116802ccc0 con 0x7f11701543a0 2024-03-06T07:24:19.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.215+0000 7f1174d14640 1 -- 172.21.15.1:0/1263596469 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1168034610 con 0x7f11701543a0 2024-03-06T07:24:19.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.216+0000 7f1177f7a640 1 -- 172.21.15.1:0/1263596469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11701543a0 msgr2=0x7f117015e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.216+0000 7f1177f7a640 1 --2- 172.21.15.1:0/1263596469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11701543a0 0x7f117015e840 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f1168009e50 tx=0x7f116802bf80 comp rx=0 tx=0).stop 2024-03-06T07:24:19.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.216+0000 7f1177f7a640 1 -- 172.21.15.1:0/1263596469 shutdown_connections 2024-03-06T07:24:19.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.216+0000 7f1177f7a640 1 --2- 172.21.15.1:0/1263596469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11701543a0 0x7f117015e840 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.216+0000 7f1177f7a640 1 -- 172.21.15.1:0/1263596469 >> 172.21.15.1:0/1263596469 conn(0x7f11700c88f0 msgr2=0x7f11700c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.216+0000 7f1177f7a640 1 -- 172.21.15.1:0/1263596469 shutdown_connections 2024-03-06T07:24:19.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.216+0000 7f1177f7a640 1 -- 172.21.15.1:0/1263596469 wait complete. 2024-03-06T07:24:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.217+0000 7f1177f7a640 1 Processor -- start 2024-03-06T07:24:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.217+0000 7f1177f7a640 1 -- start start 2024-03-06T07:24:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.218+0000 7f1177f7a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11700c32f0 0x7f1170156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.218+0000 7f1177f7a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1170150520 0x7f1170154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.218+0000 7f1177f7a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11701543a0 0x7f1170155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.218+0000 7f1177f7a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11701532a0 con 0x7f11701543a0 2024-03-06T07:24:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.218+0000 7f1177f7a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11701547e0 con 0x7f1170150520 2024-03-06T07:24:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.218+0000 7f1177f7a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1170157330 con 0x7f11700c32f0 2024-03-06T07:24:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.218+0000 7f1176517640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11701543a0 0x7f1170155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.218+0000 7f1176517640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11701543a0 0x7f1170155410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44346/0 (socket says 172.21.15.1:44346) 2024-03-06T07:24:19.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.218+0000 7f1176517640 1 -- 172.21.15.1:0/425194449 learned_addr learned my addr 172.21.15.1:0/425194449 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:19.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.218+0000 7f1175515640 1 --2- 172.21.15.1:0/425194449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1170150520 0x7f1170154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.218+0000 7f1175d16640 1 --2- 172.21.15.1:0/425194449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11700c32f0 0x7f1170156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.218+0000 7f1176517640 1 -- 172.21.15.1:0/425194449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11700c32f0 msgr2=0x7f1170156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.218+0000 7f1176517640 1 --2- 172.21.15.1:0/425194449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11700c32f0 0x7f1170156870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.218+0000 7f1176517640 1 -- 172.21.15.1:0/425194449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1170150520 msgr2=0x7f1170154ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.218+0000 7f1176517640 1 --2- 172.21.15.1:0/425194449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1170150520 0x7f1170154ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.218+0000 7f1176517640 1 -- 172.21.15.1:0/425194449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11701ebe80 con 0x7f11701543a0 2024-03-06T07:24:19.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.218+0000 7f1176517640 1 --2- 172.21.15.1:0/425194449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11701543a0 0x7f1170155410 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f1168009f70 tx=0x7f1168004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.219+0000 7f11627fc640 1 -- 172.21.15.1:0/425194449 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1168043020 con 0x7f11701543a0 2024-03-06T07:24:19.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.219+0000 7f11627fc640 1 -- 172.21.15.1:0/425194449 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116803e450 con 0x7f11701543a0 2024-03-06T07:24:19.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.219+0000 7f11627fc640 1 -- 172.21.15.1:0/425194449 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116803e820 con 0x7f11701543a0 2024-03-06T07:24:19.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.219+0000 7f1177f7a640 1 -- 172.21.15.1:0/425194449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11701ec160 con 0x7f11701543a0 2024-03-06T07:24:19.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.219+0000 7f1177f7a640 1 -- 172.21.15.1:0/425194449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11701ec720 con 0x7f11701543a0 2024-03-06T07:24:19.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.220+0000 7f1177f7a640 1 -- 172.21.15.1:0/425194449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11700686e0 con 0x7f11701543a0 2024-03-06T07:24:19.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.220+0000 7f11627fc640 1 -- 172.21.15.1:0/425194449 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1168046020 con 0x7f11701543a0 2024-03-06T07:24:19.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.221+0000 7f11627fc640 1 --2- 172.21.15.1:0/425194449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f115406b4d0 0x7f115406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.221+0000 7f11627fc640 1 -- 172.21.15.1:0/425194449 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f11680acd60 con 0x7f11701543a0 2024-03-06T07:24:19.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.221+0000 7f1175d16640 1 --2- 172.21.15.1:0/425194449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f115406b4d0 0x7f115406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.222+0000 7f1175d16640 1 --2- 172.21.15.1:0/425194449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f115406b4d0 0x7f115406d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f11701562a0 tx=0x7f1164009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.225+0000 7f11627fc640 1 -- 172.21.15.1:0/425194449 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f116804d030 con 0x7f11701543a0 2024-03-06T07:24:19.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.443+0000 7f1177f7a640 1 -- 172.21.15.1:0/425194449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1170156db0 con 0x7f11701543a0 2024-03-06T07:24:19.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.444+0000 7f11627fc640 1 -- 172.21.15.1:0/425194449 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f116807bae0 con 0x7f11701543a0 2024-03-06T07:24:19.446 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:19.446 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":88,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6122,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2632566151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2632566151},{"type":"v1","addr":"172.21.15.98:6837","nonce":2632566151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":6670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2244230066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2244230066},{"type":"v1","addr":"172.21.15.1:6835","nonce":2244230066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":6682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1741783790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1741783790},{"type":"v1","addr":"172.21.15.98:6836","nonce":1741783790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:10.220198+0000","modified":"2024-03-06T07:24:12.224326+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6833/2433923518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2433923518},{"type":"v1","addr":"172.21.15.1:6833","nonce":2433923518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":8}]} 2024-03-06T07:24:19.446 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 88 2024-03-06T07:24:19.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.446+0000 7f1177f7a640 1 -- 172.21.15.1:0/425194449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f115406b4d0 msgr2=0x7f115406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.446+0000 7f1177f7a640 1 --2- 172.21.15.1:0/425194449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f115406b4d0 0x7f115406d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f11701562a0 tx=0x7f1164009040 comp rx=0 tx=0).stop 2024-03-06T07:24:19.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.446+0000 7f1177f7a640 1 -- 172.21.15.1:0/425194449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11701543a0 msgr2=0x7f1170155410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.446+0000 7f1177f7a640 1 --2- 172.21.15.1:0/425194449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11701543a0 0x7f1170155410 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f1168009f70 tx=0x7f1168004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.446+0000 7f1177f7a640 1 -- 172.21.15.1:0/425194449 shutdown_connections 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.446+0000 7f1177f7a640 1 --2- 172.21.15.1:0/425194449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f115406b4d0 0x7f115406d990 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.446+0000 7f1177f7a640 1 --2- 172.21.15.1:0/425194449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11701543a0 0x7f1170155410 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.446+0000 7f1177f7a640 1 --2- 172.21.15.1:0/425194449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1170150520 0x7f1170154ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.446+0000 7f1177f7a640 1 --2- 172.21.15.1:0/425194449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11700c32f0 0x7f1170156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.446+0000 7f1177f7a640 1 -- 172.21.15.1:0/425194449 >> 172.21.15.1:0/425194449 conn(0x7f11700c88f0 msgr2=0x7f11701537c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.446+0000 7f1177f7a640 1 -- 172.21.15.1:0/425194449 shutdown_connections 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.446+0000 7f1177f7a640 1 -- 172.21.15.1:0/425194449 wait complete. 2024-03-06T07:24:19.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:10.220198+0000', 'modified': '2024-03-06T07:24:12.224326+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7362}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7362': {'gid': 7362, 'name': 'c', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6833/2433923518', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2433923518}, {'type': 'v1', 'addr': '172.21.15.1:6833', 'nonce': 2433923518}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7362, 'qdb_cluster': [7362]} 2024-03-06T07:24:19.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:24:19.863 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:24:19.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.963+0000 7f7050148640 1 Processor -- start 2024-03-06T07:24:19.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.963+0000 7f7050148640 1 -- start start 2024-03-06T07:24:19.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.964+0000 7f7050148640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70480c6b10 0x7f7048155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.964+0000 7f7050148640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f70480bf9d0 0x7f70480c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.964+0000 7f7050148640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70480c2fc0 con 0x7f7048152650 2024-03-06T07:24:19.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.964+0000 7f7050148640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7048155800 con 0x7f70480bf9d0 2024-03-06T07:24:19.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.964+0000 7f7050148640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7048158da0 con 0x7f70480c6b10 2024-03-06T07:24:19.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.964+0000 7f704e6e5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f70480bf9d0 0x7f70480c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.964+0000 7f704dee4640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7048152650 0x7f7048158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36040/0 (socket says 172.21.15.1:36040) 2024-03-06T07:24:19.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.964+0000 7f704dee4640 1 -- 172.21.15.1:0/2387058219 learned_addr learned my addr 172.21.15.1:0/2387058219 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:19.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.965+0000 7f704d6e3640 1 --2- 172.21.15.1:0/2387058219 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70480c6b10 0x7f7048155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.965+0000 7f704e6e5640 1 -- 172.21.15.1:0/2387058219 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70480c6b10 msgr2=0x7f7048155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.965+0000 7f704e6e5640 1 --2- 172.21.15.1:0/2387058219 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70480c6b10 0x7f7048155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.965+0000 7f704e6e5640 1 -- 172.21.15.1:0/2387058219 >> v1:172.21.15.1:6789/0 conn(0x7f7048152650 legacy=0x7f7048158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.965+0000 7f704e6e5640 1 -- 172.21.15.1:0/2387058219 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70480c6690 con 0x7f70480bf9d0 2024-03-06T07:24:19.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.965+0000 7f704e6e5640 1 --2- 172.21.15.1:0/2387058219 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70480bf9d0 0x7f70480c1460 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f7040009d90 tx=0x7f704002bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c20ed0632cee3d29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.966+0000 7f704cee2640 1 -- 172.21.15.1:0/2387058219 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7040039070 con 0x7f70480bf9d0 2024-03-06T07:24:19.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.966+0000 7f704cee2640 1 -- 172.21.15.1:0/2387058219 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704002c980 con 0x7f70480bf9d0 2024-03-06T07:24:19.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.966+0000 7f704cee2640 1 -- 172.21.15.1:0/2387058219 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704002cd50 con 0x7f70480bf9d0 2024-03-06T07:24:19.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.966+0000 7f7050148640 1 -- 172.21.15.1:0/2387058219 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70480bf9d0 msgr2=0x7f70480c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.966+0000 7f7050148640 1 --2- 172.21.15.1:0/2387058219 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70480bf9d0 0x7f70480c1460 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f7040009d90 tx=0x7f704002bc50 comp rx=0 tx=0).stop 2024-03-06T07:24:19.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.966+0000 7f7050148640 1 -- 172.21.15.1:0/2387058219 shutdown_connections 2024-03-06T07:24:19.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.966+0000 7f7050148640 1 --2- 172.21.15.1:0/2387058219 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70480c6b10 0x7f7048155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.966+0000 7f7050148640 1 --2- 172.21.15.1:0/2387058219 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70480bf9d0 0x7f70480c1460 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.966+0000 7f7050148640 1 -- 172.21.15.1:0/2387058219 >> 172.21.15.1:0/2387058219 conn(0x7f7048066ee0 msgr2=0x7f70480672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.966+0000 7f7050148640 1 -- 172.21.15.1:0/2387058219 shutdown_connections 2024-03-06T07:24:19.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.966+0000 7f7050148640 1 -- 172.21.15.1:0/2387058219 wait complete. 2024-03-06T07:24:19.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.968+0000 7f7050148640 1 Processor -- start 2024-03-06T07:24:19.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.968+0000 7f7050148640 1 -- start start 2024-03-06T07:24:19.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.969+0000 7f7050148640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70480bf9d0 0x7f70480c1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.969+0000 7f7050148640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048152650 0x7f704814ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.969+0000 7f7050148640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f704814c160 0x7f704814b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.969+0000 7f7050148640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70481551b0 con 0x7f704814c160 2024-03-06T07:24:19.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.969+0000 7f7050148640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70480c6fa0 con 0x7f7048152650 2024-03-06T07:24:19.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.969+0000 7f7050148640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7048155490 con 0x7f70480bf9d0 2024-03-06T07:24:19.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.969+0000 7f704e6e5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f704814c160 0x7f704814b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.969+0000 7f704e6e5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f704814c160 0x7f704814b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44368/0 (socket says 172.21.15.1:44368) 2024-03-06T07:24:19.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.969+0000 7f704e6e5640 1 -- 172.21.15.1:0/235673378 learned_addr learned my addr 172.21.15.1:0/235673378 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:19.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.969+0000 7f704e6e5640 1 -- 172.21.15.1:0/235673378 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70480bf9d0 msgr2=0x7f70480c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.969+0000 7f704e6e5640 1 --2- 172.21.15.1:0/235673378 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70480bf9d0 0x7f70480c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.969+0000 7f704e6e5640 1 -- 172.21.15.1:0/235673378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048152650 msgr2=0x7f704814ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.969+0000 7f704e6e5640 1 --2- 172.21.15.1:0/235673378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048152650 0x7f704814ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.969+0000 7f704e6e5640 1 -- 172.21.15.1:0/235673378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704814bae0 con 0x7f704814c160 2024-03-06T07:24:19.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.970+0000 7f704e6e5640 1 --2- 172.21.15.1:0/235673378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f704814c160 0x7f704814b360 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f7040009eb0 tx=0x7f7040004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.970+0000 7f703e7fc640 1 -- 172.21.15.1:0/235673378 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7040039050 con 0x7f704814c160 2024-03-06T07:24:19.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.970+0000 7f703e7fc640 1 -- 172.21.15.1:0/235673378 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7040045070 con 0x7f704814c160 2024-03-06T07:24:19.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.970+0000 7f7050148640 1 -- 172.21.15.1:0/235673378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7048154040 con 0x7f704814c160 2024-03-06T07:24:19.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.970+0000 7f703e7fc640 1 -- 172.21.15.1:0/235673378 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7040040450 con 0x7f704814c160 2024-03-06T07:24:19.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.970+0000 7f7050148640 1 -- 172.21.15.1:0/235673378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7048154600 con 0x7f704814c160 2024-03-06T07:24:19.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.971+0000 7f703e7fc640 1 -- 172.21.15.1:0/235673378 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70400405f0 con 0x7f704814c160 2024-03-06T07:24:19.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.972+0000 7f703e7fc640 1 --2- 172.21.15.1:0/235673378 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f702c06b4d0 0x7f702c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.972+0000 7f703e7fc640 1 -- 172.21.15.1:0/235673378 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f70400acb30 con 0x7f704814c160 2024-03-06T07:24:19.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.972+0000 7f704dee4640 1 --2- 172.21.15.1:0/235673378 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f702c06b4d0 0x7f702c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.973+0000 7f704dee4640 1 --2- 172.21.15.1:0/235673378 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f702c06b4d0 0x7f702c06d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7044007fd0 tx=0x7f70440079a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.973+0000 7f7052e0d740 1 Processor -- start 2024-03-06T07:24:19.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.973+0000 7f7052e0d740 1 -- start start 2024-03-06T07:24:19.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f7052e0d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c71cf50 0x55c73c71aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f7052e0d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c71d330 0x55c73c71af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f7052e0d740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71b4b0 0x55c73c717680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f7052e0d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c6816b0 con 0x55c73c71cf50 2024-03-06T07:24:19.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f7052e0d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c62aa60 con 0x55c73c71d330 2024-03-06T07:24:19.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f7052e0d740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c73c67d750 con 0x55c73c71b4b0 2024-03-06T07:24:19.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f704dee4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c71cf50 0x55c73c71aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f704dee4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c71cf50 0x55c73c71aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44380/0 (socket says 172.21.15.1:44380) 2024-03-06T07:24:19.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f704dee4640 1 -- 172.21.15.1:0/2114046446 learned_addr learned my addr 172.21.15.1:0/2114046446 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:19.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f704e6e5640 1 --2- 172.21.15.1:0/2114046446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71b4b0 0x55c73c717680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f704d6e3640 1 --2- 172.21.15.1:0/2114046446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c71d330 0x55c73c71af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f704dee4640 1 -- 172.21.15.1:0/2114046446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71b4b0 msgr2=0x55c73c717680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f704dee4640 1 --2- 172.21.15.1:0/2114046446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71b4b0 0x55c73c717680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f704dee4640 1 -- 172.21.15.1:0/2114046446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c71d330 msgr2=0x55c73c71af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f704dee4640 1 --2- 172.21.15.1:0/2114046446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c71d330 0x55c73c71af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f704dee4640 1 -- 172.21.15.1:0/2114046446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c73c648f30 con 0x55c73c71cf50 2024-03-06T07:24:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f704e6e5640 1 --2- 172.21.15.1:0/2114046446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71b4b0 0x55c73c717680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.974+0000 7f704d6e3640 1 --2- 172.21.15.1:0/2114046446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c71d330 0x55c73c71af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.975+0000 7f704dee4640 1 --2- 172.21.15.1:0/2114046446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c71cf50 0x55c73c71aa30 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f7044007420 tx=0x7f704400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e66bd89ccb08713a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.975+0000 7f702b7fe640 1 -- 172.21.15.1:0/2114046446 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7044007450 con 0x55c73c71cf50 2024-03-06T07:24:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.975+0000 7f702b7fe640 1 -- 172.21.15.1:0/2114046446 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70440075f0 con 0x55c73c71cf50 2024-03-06T07:24:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.975+0000 7f702b7fe640 1 -- 172.21.15.1:0/2114046446 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704400ce20 con 0x55c73c71cf50 2024-03-06T07:24:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.975+0000 7f7052e0d740 1 -- 172.21.15.1:0/2114046446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c71cf50 msgr2=0x55c73c71aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.975+0000 7f7052e0d740 1 --2- 172.21.15.1:0/2114046446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c71cf50 0x55c73c71aa30 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f7044007420 tx=0x7f704400f000 comp rx=0 tx=0).stop 2024-03-06T07:24:19.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.975+0000 7f7052e0d740 1 -- 172.21.15.1:0/2114046446 shutdown_connections 2024-03-06T07:24:19.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.975+0000 7f7052e0d740 1 --2- 172.21.15.1:0/2114046446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71b4b0 0x55c73c717680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.975+0000 7f7052e0d740 1 --2- 172.21.15.1:0/2114046446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c71d330 0x55c73c71af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.975+0000 7f7052e0d740 1 --2- 172.21.15.1:0/2114046446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c71cf50 0x55c73c71aa30 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.975+0000 7f7052e0d740 1 -- 172.21.15.1:0/2114046446 >> 172.21.15.1:0/2114046446 conn(0x55c73c7197d0 msgr2=0x55c73c73e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.975+0000 7f7052e0d740 1 -- 172.21.15.1:0/2114046446 shutdown_connections 2024-03-06T07:24:19.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.975+0000 7f7052e0d740 1 -- 172.21.15.1:0/2114046446 wait complete. 2024-03-06T07:24:19.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.976+0000 7f7052e0d740 1 -- start start 2024-03-06T07:24:19.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.976+0000 7f7052e0d740 1 Processor -- start 2024-03-06T07:24:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f7052e0d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c71cf50 0x55c73c73d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f7052e0d740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71d330 0x55c73c742b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f7052e0d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c743070 0x55c73c755750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f7052e0d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c73f230 con 0x55c73c71cf50 2024-03-06T07:24:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f7052e0d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c63dea0 con 0x55c73c743070 2024-03-06T07:24:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f7052e0d740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c73c73ec70 con 0x55c73c71d330 2024-03-06T07:24:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f7052e0d740 10 client.0 _ll_register_callbacks cb 0x55c73c63d870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f704dee4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c71cf50 0x55c73c73d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f704dee4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c71cf50 0x55c73c73d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44396/0 (socket says 172.21.15.1:44396) 2024-03-06T07:24:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f704dee4640 1 -- 172.21.15.1:0/725190752 learned_addr learned my addr 172.21.15.1:0/725190752 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f704d6e3640 1 --2- 172.21.15.1:0/725190752 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71d330 0x55c73c742b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f704e6e5640 1 --2- 172.21.15.1:0/725190752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c743070 0x55c73c755750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f704dee4640 1 -- 172.21.15.1:0/725190752 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71d330 msgr2=0x55c73c742b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f704dee4640 1 --2- 172.21.15.1:0/725190752 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71d330 0x55c73c742b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f704dee4640 1 -- 172.21.15.1:0/725190752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c743070 msgr2=0x55c73c755750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f704dee4640 1 --2- 172.21.15.1:0/725190752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c743070 0x55c73c755750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f704dee4640 1 -- 172.21.15.1:0/725190752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c73c63e270 con 0x55c73c71cf50 2024-03-06T07:24:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f704dee4640 1 --2- 172.21.15.1:0/725190752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c71cf50 0x55c73c73d7b0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f70440077e0 tx=0x7f7044011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58fb20d6ba66e505 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.977+0000 7f7028ff9640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:24:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.978+0000 7f7028ff9640 1 -- 172.21.15.1:0/725190752 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7044029060 con 0x55c73c71cf50 2024-03-06T07:24:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.978+0000 7f7028ff9640 1 -- 172.21.15.1:0/725190752 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7044014a70 con 0x55c73c71cf50 2024-03-06T07:24:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.978+0000 7f7052e0d740 1 -- 172.21.15.1:0/725190752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f70440075f0 con 0x55c73c71cf50 2024-03-06T07:24:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.978+0000 7f7028ff9640 1 -- 172.21.15.1:0/725190752 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f704402f050 con 0x55c73c71cf50 2024-03-06T07:24:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.978+0000 7f7052e0d740 10 client.7542 fetch_fsmap learned FSMap version 88 2024-03-06T07:24:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.978+0000 7f7052e0d740 1 -- 172.21.15.1:0/725190752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) v3 -- 0x55c73c3962d0 con 0x55c73c71cf50 2024-03-06T07:24:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.978+0000 7f7028ff9640 1 -- 172.21.15.1:0/725190752 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70440075f0 con 0x55c73c71cf50 2024-03-06T07:24:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.978+0000 7f7028ff9640 1 -- 172.21.15.1:0/725190752 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 88) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f7044032020 con 0x55c73c71cf50 2024-03-06T07:24:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.978+0000 7f7052e0d740 10 client.7542 fetch_fsmap finished waiting for FSMap version 88 2024-03-06T07:24:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.978+0000 7f7052e0d740 10 client.7542 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:24:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.978+0000 7f7052e0d740 20 client.7542 populate_metadata read hostname 'smithi001' 2024-03-06T07:24:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.978+0000 7f7052e0d740 1 --2- 172.21.15.1:0/725190752 >> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] conn(0x55c73c76a0a0 0x55c73c76a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.978+0000 7f7052e0d740 4 client.7542 mds_command: new command op to 7362 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:24:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.978+0000 7f7052e0d740 1 -- 172.21.15.1:0/725190752 --> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c73c68ebf0 con 0x55c73c76a0a0 2024-03-06T07:24:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.978+0000 7f704d6e3640 1 --2- 172.21.15.1:0/725190752 >> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] conn(0x55c73c76a0a0 0x55c73c76a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.979+0000 7f7028ff9640 10 client.7542 ms_handle_connect on v2:172.21.15.1:6832/2433923518 2024-03-06T07:24:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.979+0000 7f704d6e3640 1 --2- 172.21.15.1:0/725190752 >> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] conn(0x55c73c76a0a0 0x55c73c76a480 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9bd6598fbbf6487 server_cookie=5d3162b335cb10e in_seq=0 out_seq=0 2024-03-06T07:24:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.979+0000 7f7028ff9640 1 -- 172.21.15.1:0/725190752 <== mds.0 v2:172.21.15.1:6832/2433923518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c73c68ebf0 con 0x55c73c76a0a0 2024-03-06T07:24:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.979+0000 7f7028ff9640 10 client.7542 handle_command_reply: tid=0 2024-03-06T07:24:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.979+0000 7f7052e0d740 1 client.7542 shutdown 2024-03-06T07:24:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.979+0000 7f7052e0d740 1 -- 172.21.15.1:0/725190752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c71cf50 msgr2=0x55c73c73d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.979+0000 7f7052e0d740 1 --2- 172.21.15.1:0/725190752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c71cf50 0x55c73c73d7b0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f70440077e0 tx=0x7f7044011000 comp rx=0 tx=0).stop 2024-03-06T07:24:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.980+0000 7f7052e0d740 1 -- 172.21.15.1:0/725190752 shutdown_connections 2024-03-06T07:24:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.980+0000 7f7052e0d740 1 --2- 172.21.15.1:0/725190752 >> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] conn(0x55c73c76a0a0 0x55c73c76a480 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.980+0000 7f7052e0d740 1 --2- 172.21.15.1:0/725190752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c743070 0x55c73c755750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.980+0000 7f7052e0d740 1 --2- 172.21.15.1:0/725190752 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71d330 0x55c73c742b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.980+0000 7f7052e0d740 1 --2- 172.21.15.1:0/725190752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c71cf50 0x55c73c73d7b0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.980+0000 7f7052e0d740 1 -- 172.21.15.1:0/725190752 >> 172.21.15.1:0/725190752 conn(0x55c73c71b4b0 msgr2=0x55c73c759b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.980+0000 7f7028ff9640 0 client.7542 ms_handle_reset on v2:172.21.15.1:6832/2433923518 2024-03-06T07:24:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.980+0000 7f7052e0d740 1 -- 172.21.15.1:0/725190752 shutdown_connections 2024-03-06T07:24:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.980+0000 7f7052e0d740 1 -- 172.21.15.1:0/725190752 wait complete. 2024-03-06T07:24:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.980+0000 7f7052e0d740 20 client.7542 trim_cache size 0 max 16384 2024-03-06T07:24:19.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.996+0000 7f7052e0d740 1 Processor -- start 2024-03-06T07:24:19.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.997+0000 7f7052e0d740 1 -- start start 2024-03-06T07:24:20.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.997+0000 7f7052e0d740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71cf50 0x55c73c7439e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.997+0000 7f7052e0d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c71d330 0x55c73c769fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.997+0000 7f7052e0d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c743070 0x55c73c76a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.997+0000 7f7052e0d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70481f6e00 con 0x55c73c743070 2024-03-06T07:24:20.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.997+0000 7f7052e0d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70481ef520 con 0x55c73c71d330 2024-03-06T07:24:20.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.997+0000 7f7052e0d740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704814ef10 con 0x55c73c71cf50 2024-03-06T07:24:20.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.997+0000 7f704e6e5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c743070 0x55c73c76a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.997+0000 7f704e6e5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c743070 0x55c73c76a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44412/0 (socket says 172.21.15.1:44412) 2024-03-06T07:24:20.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.997+0000 7f704e6e5640 1 -- 172.21.15.1:0/3234453742 learned_addr learned my addr 172.21.15.1:0/3234453742 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:20.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.998+0000 7f704e6e5640 1 -- 172.21.15.1:0/3234453742 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71cf50 msgr2=0x55c73c7439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.998+0000 7f704e6e5640 1 --2- 172.21.15.1:0/3234453742 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71cf50 0x55c73c7439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.998+0000 7f704e6e5640 1 -- 172.21.15.1:0/3234453742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c71d330 msgr2=0x55c73c769fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.998+0000 7f704e6e5640 1 --2- 172.21.15.1:0/3234453742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c71d330 0x55c73c769fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.998+0000 7f704e6e5640 1 -- 172.21.15.1:0/3234453742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c73c4606b0 con 0x55c73c743070 2024-03-06T07:24:20.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.998+0000 7f704e6e5640 1 --2- 172.21.15.1:0/3234453742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c743070 0x55c73c76a4e0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f704002c480 tx=0x7f704007acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eef8fcb67d7117c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.998+0000 7f702affd640 1 -- 172.21.15.1:0/3234453742 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704007f420 con 0x55c73c743070 2024-03-06T07:24:20.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.998+0000 7f702affd640 1 -- 172.21.15.1:0/3234453742 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70400a9070 con 0x55c73c743070 2024-03-06T07:24:20.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.998+0000 7f702affd640 1 -- 172.21.15.1:0/3234453742 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70400839e0 con 0x55c73c743070 2024-03-06T07:24:20.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.999+0000 7f7052e0d740 1 -- 172.21.15.1:0/3234453742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c743070 msgr2=0x55c73c76a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.999+0000 7f7052e0d740 1 --2- 172.21.15.1:0/3234453742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c743070 0x55c73c76a4e0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f704002c480 tx=0x7f704007acf0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.999+0000 7f7052e0d740 1 -- 172.21.15.1:0/3234453742 shutdown_connections 2024-03-06T07:24:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.999+0000 7f7052e0d740 1 --2- 172.21.15.1:0/3234453742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c743070 0x55c73c76a4e0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.999+0000 7f7052e0d740 1 --2- 172.21.15.1:0/3234453742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c71d330 0x55c73c769fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.999+0000 7f7052e0d740 1 --2- 172.21.15.1:0/3234453742 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71cf50 0x55c73c7439e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.999+0000 7f7052e0d740 1 -- 172.21.15.1:0/3234453742 >> 172.21.15.1:0/3234453742 conn(0x55c73c71b4b0 msgr2=0x55c73c73e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.999+0000 7f7052e0d740 1 -- 172.21.15.1:0/3234453742 shutdown_connections 2024-03-06T07:24:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.999+0000 7f7052e0d740 1 -- 172.21.15.1:0/3234453742 wait complete. 2024-03-06T07:24:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:19.999+0000 7f7052e0d740 1 -- start start 2024-03-06T07:24:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.000+0000 7f7052e0d740 1 Processor -- start 2024-03-06T07:24:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f7052e0d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c71cf50 0x55c73c73c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f7052e0d740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71d330 0x55c73c73aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f7052e0d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c743070 0x55c73c7c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f7052e0d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c740040 con 0x55c73c743070 2024-03-06T07:24:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f7052e0d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c73c73fb40 con 0x55c73c71cf50 2024-03-06T07:24:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f7052e0d740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c73c740880 con 0x55c73c71d330 2024-03-06T07:24:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f7052e0d740 10 client.0 _ll_register_callbacks cb 0x7f70440071b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f704e6e5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c743070 0x55c73c7c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f704e6e5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c743070 0x55c73c7c6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44424/0 (socket says 172.21.15.1:44424) 2024-03-06T07:24:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f704e6e5640 1 -- 172.21.15.1:0/3397372637 learned_addr learned my addr 172.21.15.1:0/3397372637 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f704dee4640 1 --2- 172.21.15.1:0/3397372637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c71cf50 0x55c73c73c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f704d6e3640 1 --2- 172.21.15.1:0/3397372637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71d330 0x55c73c73aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f704e6e5640 1 -- 172.21.15.1:0/3397372637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71d330 msgr2=0x55c73c73aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f704e6e5640 1 --2- 172.21.15.1:0/3397372637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71d330 0x55c73c73aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f704e6e5640 1 -- 172.21.15.1:0/3397372637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c71cf50 msgr2=0x55c73c73c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f704e6e5640 1 --2- 172.21.15.1:0/3397372637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c71cf50 0x55c73c73c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f704e6e5640 1 -- 172.21.15.1:0/3397372637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c73c73b030 con 0x55c73c743070 2024-03-06T07:24:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f704e6e5640 1 --2- 172.21.15.1:0/3397372637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c743070 0x55c73c7c6540 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f70400aafb0 tx=0x7f704002cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dc4aeb3204275b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.001+0000 7f7028ff9640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:24:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.002+0000 7f7028ff9640 1 -- 172.21.15.1:0/3397372637 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704007f420 con 0x55c73c743070 2024-03-06T07:24:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.002+0000 7f7028ff9640 1 -- 172.21.15.1:0/3397372637 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f704007b530 con 0x55c73c743070 2024-03-06T07:24:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.002+0000 7f7052e0d740 1 -- 172.21.15.1:0/3397372637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f70400a9070 con 0x55c73c743070 2024-03-06T07:24:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.002+0000 7f7028ff9640 1 -- 172.21.15.1:0/3397372637 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f704007b730 con 0x55c73c743070 2024-03-06T07:24:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.002+0000 7f7052e0d740 10 client.7548 fetch_fsmap learned FSMap version 88 2024-03-06T07:24:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.002+0000 7f7052e0d740 1 -- 172.21.15.1:0/3397372637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) v3 -- 0x55c73c73b240 con 0x55c73c743070 2024-03-06T07:24:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.002+0000 7f7028ff9640 1 -- 172.21.15.1:0/3397372637 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70400a9070 con 0x55c73c743070 2024-03-06T07:24:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.002+0000 7f7028ff9640 1 -- 172.21.15.1:0/3397372637 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 88) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f70400ac040 con 0x55c73c743070 2024-03-06T07:24:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.002+0000 7f7052e0d740 10 client.7548 fetch_fsmap finished waiting for FSMap version 88 2024-03-06T07:24:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.002+0000 7f7052e0d740 10 client.7548 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:24:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.002+0000 7f7052e0d740 20 client.7548 populate_metadata read hostname 'smithi001' 2024-03-06T07:24:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.002+0000 7f7052e0d740 1 --2- 172.21.15.1:0/3397372637 >> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] conn(0x55c73c7c6a80 0x55c73c77c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.002+0000 7f7052e0d740 4 client.7548 mds_command: new command op to 7362 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:24:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.002+0000 7f7052e0d740 1 -- 172.21.15.1:0/3397372637 --> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c73c688f40 con 0x55c73c7c6a80 2024-03-06T07:24:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.002+0000 7f704dee4640 1 --2- 172.21.15.1:0/3397372637 >> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] conn(0x55c73c7c6a80 0x55c73c77c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.003+0000 7f7028ff9640 10 client.7548 ms_handle_connect on v2:172.21.15.1:6832/2433923518 2024-03-06T07:24:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.003+0000 7f704dee4640 1 --2- 172.21.15.1:0/3397372637 >> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] conn(0x55c73c7c6a80 0x55c73c77c5a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1d72a9662ffc48e server_cookie=7052312406e95bdd in_seq=0 out_seq=0 2024-03-06T07:24:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.003+0000 7f7028ff9640 1 -- 172.21.15.1:0/3397372637 <== mds.0 v2:172.21.15.1:6832/2433923518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55c73c688f40 con 0x55c73c7c6a80 2024-03-06T07:24:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.003+0000 7f7028ff9640 10 client.7548 handle_command_reply: tid=0 2024-03-06T07:24:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.003+0000 7f7052e0d740 1 client.7548 shutdown 2024-03-06T07:24:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.003+0000 7f7052e0d740 1 -- 172.21.15.1:0/3397372637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c743070 msgr2=0x55c73c7c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.003+0000 7f7052e0d740 1 --2- 172.21.15.1:0/3397372637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c743070 0x55c73c7c6540 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f70400aafb0 tx=0x7f704002cf90 comp rx=0 tx=0).stop 2024-03-06T07:24:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.003+0000 7f7052e0d740 1 -- 172.21.15.1:0/3397372637 shutdown_connections 2024-03-06T07:24:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.003+0000 7f7052e0d740 1 --2- 172.21.15.1:0/3397372637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c73c743070 0x55c73c7c6540 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.003+0000 7f7052e0d740 1 --2- 172.21.15.1:0/3397372637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55c73c71d330 0x55c73c73aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.003+0000 7f7052e0d740 1 --2- 172.21.15.1:0/3397372637 >> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] conn(0x55c73c7c6a80 0x55c73c77c5a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.003+0000 7f7052e0d740 1 --2- 172.21.15.1:0/3397372637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c73c71cf50 0x55c73c73c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.003+0000 7f7028ff9640 0 client.7548 ms_handle_reset on v2:172.21.15.1:6832/2433923518 2024-03-06T07:24:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.003+0000 7f7052e0d740 1 -- 172.21.15.1:0/3397372637 >> 172.21.15.1:0/3397372637 conn(0x55c73c71b4b0 msgr2=0x55c73c712c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.003+0000 7f7052e0d740 1 -- 172.21.15.1:0/3397372637 shutdown_connections 2024-03-06T07:24:20.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.003+0000 7f7052e0d740 1 -- 172.21.15.1:0/3397372637 wait complete. 2024-03-06T07:24:20.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.003+0000 7f7052e0d740 20 client.7548 trim_cache size 0 max 16384 2024-03-06T07:24:20.011 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:24:20.011 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:24:20.011 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:24:20.011 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 7362, 2024-03-06T07:24:20.011 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:24:20.011 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:24:20.011 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:24:20.011 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 9.1481856029999999, 2024-03-06T07:24:20.011 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 88, 2024-03-06T07:24:20.012 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 108, 2024-03-06T07:24:20.012 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 108, 2024-03-06T07:24:20.012 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 14.221433647 2024-03-06T07:24:20.012 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:24:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.004+0000 7f7050148640 1 -- 172.21.15.1:0/235673378 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f702c06b4d0 msgr2=0x7f702c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.004+0000 7f7050148640 1 --2- 172.21.15.1:0/235673378 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f702c06b4d0 0x7f702c06d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7044007fd0 tx=0x7f70440079a0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.004+0000 7f7050148640 1 -- 172.21.15.1:0/235673378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f704814c160 msgr2=0x7f704814b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.004+0000 7f7050148640 1 --2- 172.21.15.1:0/235673378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f704814c160 0x7f704814b360 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f7040009eb0 tx=0x7f7040004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.004+0000 7f7050148640 1 -- 172.21.15.1:0/235673378 shutdown_connections 2024-03-06T07:24:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.004+0000 7f7050148640 1 --2- 172.21.15.1:0/235673378 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f702c06b4d0 0x7f702c06d990 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.004+0000 7f7050148640 1 --2- 172.21.15.1:0/235673378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f704814c160 0x7f704814b360 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.004+0000 7f7050148640 1 --2- 172.21.15.1:0/235673378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048152650 0x7f704814ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.004+0000 7f7050148640 1 --2- 172.21.15.1:0/235673378 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70480bf9d0 0x7f70480c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.004+0000 7f7050148640 1 -- 172.21.15.1:0/235673378 >> 172.21.15.1:0/235673378 conn(0x7f7048066ee0 msgr2=0x7f70480c5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.004+0000 7f7050148640 1 -- 172.21.15.1:0/235673378 shutdown_connections 2024-03-06T07:24:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.004+0000 7f7050148640 1 -- 172.21.15.1:0/235673378 wait complete. 2024-03-06T07:24:20.020 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:20.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.121+0000 7f2785a55640 1 Processor -- start 2024-03-06T07:24:20.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.121+0000 7f2785a55640 1 -- start start 2024-03-06T07:24:20.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.121+0000 7f2785a55640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27800c6b10 0x7f2780155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.121+0000 7f2785a55640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27800bf9d0 0x7f27800c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.121+0000 7f2785a55640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27800c2fc0 con 0x7f2780152650 2024-03-06T07:24:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.121+0000 7f2785a55640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2780155800 con 0x7f27800c6b10 2024-03-06T07:24:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.121+0000 7f2785a55640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2780158da0 con 0x7f27800bf9d0 2024-03-06T07:24:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.122+0000 7f2777fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27800c6b10 0x7f2780155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.122+0000 7f2777fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27800c6b10 0x7f2780155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44432/0 (socket says 172.21.15.1:44432) 2024-03-06T07:24:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.122+0000 7f2777fff640 1 -- 172.21.15.1:0/3021333474 learned_addr learned my addr 172.21.15.1:0/3021333474 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.122+0000 7f277f7fe640 1 --2- 172.21.15.1:0/3021333474 >> v2:172.21.15.98:3300/0 conn(0x7f27800bf9d0 0x7f27800c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.122+0000 7f2777fff640 1 -- 172.21.15.1:0/3021333474 >> v2:172.21.15.98:3300/0 conn(0x7f27800bf9d0 msgr2=0x7f27800c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.122+0000 7f2777fff640 1 --2- 172.21.15.1:0/3021333474 >> v2:172.21.15.98:3300/0 conn(0x7f27800bf9d0 0x7f27800c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.122+0000 7f2777fff640 1 -- 172.21.15.1:0/3021333474 >> v1:172.21.15.98:6789/0 conn(0x7f2780152650 legacy=0x7f2780158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.122+0000 7f2777fff640 1 -- 172.21.15.1:0/3021333474 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27800c6690 con 0x7f27800c6b10 2024-03-06T07:24:20.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.123+0000 7f2777fff640 1 --2- 172.21.15.1:0/3021333474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27800c6b10 0x7f2780155c90 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f27801597b0 tx=0x7f276802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c5ae165f3d3fb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.123+0000 7f277e7fc640 1 -- 172.21.15.1:0/3021333474 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276802e070 con 0x7f27800c6b10 2024-03-06T07:24:20.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.123+0000 7f277e7fc640 1 -- 172.21.15.1:0/3021333474 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2768037070 con 0x7f27800c6b10 2024-03-06T07:24:20.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.123+0000 7f277e7fc640 1 -- 172.21.15.1:0/3021333474 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2768031a00 con 0x7f27800c6b10 2024-03-06T07:24:20.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.123+0000 7f2785a55640 1 -- 172.21.15.1:0/3021333474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27800c6b10 msgr2=0x7f2780155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.123+0000 7f2785a55640 1 --2- 172.21.15.1:0/3021333474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27800c6b10 0x7f2780155c90 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f27801597b0 tx=0x7f276802b880 comp rx=0 tx=0).stop 2024-03-06T07:24:20.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.123+0000 7f2785a55640 1 -- 172.21.15.1:0/3021333474 shutdown_connections 2024-03-06T07:24:20.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.123+0000 7f2785a55640 1 --2- 172.21.15.1:0/3021333474 >> v2:172.21.15.98:3300/0 conn(0x7f27800bf9d0 0x7f27800c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.123+0000 7f2785a55640 1 --2- 172.21.15.1:0/3021333474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27800c6b10 0x7f2780155c90 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.123+0000 7f2785a55640 1 -- 172.21.15.1:0/3021333474 >> 172.21.15.1:0/3021333474 conn(0x7f2780066ee0 msgr2=0x7f27800672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.123+0000 7f2785a55640 1 -- 172.21.15.1:0/3021333474 shutdown_connections 2024-03-06T07:24:20.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.123+0000 7f2785a55640 1 -- 172.21.15.1:0/3021333474 wait complete. 2024-03-06T07:24:20.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.124+0000 7f2785a55640 1 Processor -- start 2024-03-06T07:24:20.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.125+0000 7f2785a55640 1 -- start start 2024-03-06T07:24:20.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.125+0000 7f2785a55640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27800bf9d0 0x7f27800c5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.125+0000 7f2785a55640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2780152650 0x7f27801eb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.125+0000 7f2785a55640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2780156490 0x7f27801eb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.125+0000 7f2785a55640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27801551b0 con 0x7f2780152650 2024-03-06T07:24:20.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.125+0000 7f2785a55640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27800c6fa0 con 0x7f27800bf9d0 2024-03-06T07:24:20.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.125+0000 7f2785a55640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2780155490 con 0x7f2780156490 2024-03-06T07:24:20.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.125+0000 7f2777fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2780152650 0x7f27801eb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.125+0000 7f2777fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2780152650 0x7f27801eb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44446/0 (socket says 172.21.15.1:44446) 2024-03-06T07:24:20.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.125+0000 7f2777fff640 1 -- 172.21.15.1:0/2223862059 learned_addr learned my addr 172.21.15.1:0/2223862059 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:20.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.125+0000 7f277effd640 1 --2- 172.21.15.1:0/2223862059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27800bf9d0 0x7f27800c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.125+0000 7f277f7fe640 1 --2- 172.21.15.1:0/2223862059 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2780156490 0x7f27801eb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.126+0000 7f2777fff640 1 -- 172.21.15.1:0/2223862059 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2780156490 msgr2=0x7f27801eb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.126+0000 7f2777fff640 1 --2- 172.21.15.1:0/2223862059 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2780156490 0x7f27801eb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.126+0000 7f2777fff640 1 -- 172.21.15.1:0/2223862059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27800bf9d0 msgr2=0x7f27800c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.126+0000 7f2777fff640 1 --2- 172.21.15.1:0/2223862059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27800bf9d0 0x7f27800c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.126+0000 7f2777fff640 1 -- 172.21.15.1:0/2223862059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27801ebf20 con 0x7f2780152650 2024-03-06T07:24:20.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.126+0000 7f2777fff640 1 --2- 172.21.15.1:0/2223862059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2780152650 0x7f27801eb4a0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f276802b8c0 tx=0x7f27680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.126+0000 7f277cff9640 1 -- 172.21.15.1:0/2223862059 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2768032880 con 0x7f2780152650 2024-03-06T07:24:20.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.126+0000 7f277cff9640 1 -- 172.21.15.1:0/2223862059 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2768032a20 con 0x7f2780152650 2024-03-06T07:24:20.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.126+0000 7f2785a55640 1 -- 172.21.15.1:0/2223862059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27801eafa0 con 0x7f2780152650 2024-03-06T07:24:20.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.126+0000 7f277cff9640 1 -- 172.21.15.1:0/2223862059 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2768032e10 con 0x7f2780152650 2024-03-06T07:24:20.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.126+0000 7f2785a55640 1 -- 172.21.15.1:0/2223862059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27801582c0 con 0x7f2780152650 2024-03-06T07:24:20.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.127+0000 7f2785a55640 1 -- 172.21.15.1:0/2223862059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278006b540 con 0x7f2780152650 2024-03-06T07:24:20.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.127+0000 7f277cff9640 1 -- 172.21.15.1:0/2223862059 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f27680453f0 con 0x7f2780152650 2024-03-06T07:24:20.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.128+0000 7f277cff9640 1 --2- 172.21.15.1:0/2223862059 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f276006b500 0x7f276006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.128+0000 7f277cff9640 1 -- 172.21.15.1:0/2223862059 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f27680ac580 con 0x7f2780152650 2024-03-06T07:24:20.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.128+0000 7f277effd640 1 --2- 172.21.15.1:0/2223862059 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f276006b500 0x7f276006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.129+0000 7f277effd640 1 --2- 172.21.15.1:0/2223862059 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f276006b500 0x7f276006d9c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2770004ea0 tx=0x7f2770009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.132+0000 7f277cff9640 1 -- 172.21.15.1:0/2223862059 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f278006b540 con 0x7f2780152650 2024-03-06T07:24:20.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.390+0000 7f2785a55640 1 -- 172.21.15.1:0/2223862059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2780067fd0 con 0x7f2780152650 2024-03-06T07:24:20.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.391+0000 7f277cff9640 1 -- 172.21.15.1:0/2223862059 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f27801ebf20 con 0x7f2780152650 2024-03-06T07:24:20.396 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:20.396 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":88,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6122,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2632566151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2632566151},{"type":"v1","addr":"172.21.15.98:6837","nonce":2632566151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":6670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2244230066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2244230066},{"type":"v1","addr":"172.21.15.1:6835","nonce":2244230066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":6682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1741783790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1741783790},{"type":"v1","addr":"172.21.15.98:6836","nonce":1741783790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:10.220198+0000","modified":"2024-03-06T07:24:12.224326+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6833/2433923518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2433923518},{"type":"v1","addr":"172.21.15.1:6833","nonce":2433923518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":8}]} 2024-03-06T07:24:20.397 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 88 2024-03-06T07:24:20.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.393+0000 7f2785a55640 1 -- 172.21.15.1:0/2223862059 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f276006b500 msgr2=0x7f276006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.393+0000 7f2785a55640 1 --2- 172.21.15.1:0/2223862059 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f276006b500 0x7f276006d9c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2770004ea0 tx=0x7f2770009040 comp rx=0 tx=0).stop 2024-03-06T07:24:20.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.393+0000 7f2785a55640 1 -- 172.21.15.1:0/2223862059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2780152650 msgr2=0x7f27801eb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.393+0000 7f2785a55640 1 --2- 172.21.15.1:0/2223862059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2780152650 0x7f27801eb4a0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f276802b8c0 tx=0x7f27680046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.393+0000 7f2785a55640 1 -- 172.21.15.1:0/2223862059 shutdown_connections 2024-03-06T07:24:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.393+0000 7f2785a55640 1 --2- 172.21.15.1:0/2223862059 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f276006b500 0x7f276006d9c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.393+0000 7f2785a55640 1 --2- 172.21.15.1:0/2223862059 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2780156490 0x7f27801eb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.393+0000 7f2785a55640 1 --2- 172.21.15.1:0/2223862059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2780152650 0x7f27801eb4a0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.393+0000 7f2785a55640 1 --2- 172.21.15.1:0/2223862059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27800bf9d0 0x7f27800c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.393+0000 7f2785a55640 1 -- 172.21.15.1:0/2223862059 >> 172.21.15.1:0/2223862059 conn(0x7f2780066ee0 msgr2=0x7f278014ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.393+0000 7f2785a55640 1 -- 172.21.15.1:0/2223862059 shutdown_connections 2024-03-06T07:24:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.393+0000 7f2785a55640 1 -- 172.21.15.1:0/2223862059 wait complete. 2024-03-06T07:24:20.406 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 88, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:24:10.220198+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 88, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7362": { "addr": "172.21.15.1:6833/2433923518", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 2433923518, "type": "v2" }, { "addr": "172.21.15.1:6833", "nonce": 2433923518, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7362, "incarnation": 86, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-03-06T07:24:12.224326+0000", "qdb_cluster": [ 7362 ], "qdb_leader": 7362, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7362 } } } ], "standbys": [ { "addr": "172.21.15.98:6837/2632566151", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 2632566151, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2632566151, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6122, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6835/2244230066", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6834", "nonce": 2244230066, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 2244230066, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6670, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6836/1741783790", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 1741783790, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 1741783790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6682, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:24:20.406 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:24:20.407 INFO:tasks.cephfs.test_failover:current = [{'gid': 6670, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/2244230066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 2244230066}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2244230066}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}, {'gid': 6122, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/2632566151', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 2632566151}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2632566151}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}, {'gid': 7362, 'name': 'c', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6833/2433923518', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2433923518}, {'type': 'v1', 'addr': '172.21.15.1:6833', 'nonce': 2433923518}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6682, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/1741783790', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 1741783790}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1741783790}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}] 2024-03-06T07:24:20.407 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs 2024-03-06T07:24:20.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.535+0000 7f0dba6eb640 1 Processor -- start 2024-03-06T07:24:20.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.535+0000 7f0dba6eb640 1 -- start start 2024-03-06T07:24:20.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.537+0000 7f0dba6eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db4152650 0x7f0db4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.537+0000 7f0dba6eb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0db40c6b10 0x7f0db4158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.537+0000 7f0dba6eb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0db40bf9d0 0x7f0db40c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.537+0000 7f0dba6eb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db4155800 con 0x7f0db40c6b10 2024-03-06T07:24:20.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.537+0000 7f0dba6eb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db40c2fc0 con 0x7f0db40bf9d0 2024-03-06T07:24:20.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.537+0000 7f0dba6eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db40c6690 con 0x7f0db4152650 2024-03-06T07:24:20.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.537+0000 7f0db37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0db40c6b10 0x7f0db4158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.537+0000 7f0db37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0db40c6b10 0x7f0db4158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44452/0 (socket says 172.21.15.1:44452) 2024-03-06T07:24:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.537+0000 7f0db37fe640 1 -- 172.21.15.1:0/625867554 learned_addr learned my addr 172.21.15.1:0/625867554 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.537+0000 7f0db3fff640 1 --2- 172.21.15.1:0/625867554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db4152650 0x7f0db4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.537+0000 7f0db8c88640 1 --2- 172.21.15.1:0/625867554 >> v2:172.21.15.98:3300/0 conn(0x7f0db40bf9d0 0x7f0db40c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.537+0000 7f0db37fe640 1 -- 172.21.15.1:0/625867554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db4152650 msgr2=0x7f0db4155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.537+0000 7f0db37fe640 1 --2- 172.21.15.1:0/625867554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db4152650 0x7f0db4155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.537+0000 7f0db37fe640 1 -- 172.21.15.1:0/625867554 >> v2:172.21.15.98:3300/0 conn(0x7f0db40bf9d0 msgr2=0x7f0db40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.537+0000 7f0db37fe640 1 --2- 172.21.15.1:0/625867554 >> v2:172.21.15.98:3300/0 conn(0x7f0db40bf9d0 0x7f0db40c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.537+0000 7f0db37fe640 1 -- 172.21.15.1:0/625867554 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db40c7280 con 0x7f0db40c6b10 2024-03-06T07:24:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.538+0000 7f0db37fe640 1 --2- 172.21.15.1:0/625867554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db40c6b10 0x7f0db4158690 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f0db41597b0 tx=0x7f0da802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c54df811ee9453e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.538+0000 7f0db2ffd640 1 -- 172.21.15.1:0/625867554 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da802e070 con 0x7f0db40c6b10 2024-03-06T07:24:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.538+0000 7f0db2ffd640 1 -- 172.21.15.1:0/625867554 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da8037070 con 0x7f0db40c6b10 2024-03-06T07:24:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.538+0000 7f0db2ffd640 1 -- 172.21.15.1:0/625867554 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da8031a00 con 0x7f0db40c6b10 2024-03-06T07:24:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.538+0000 7f0dba6eb640 1 -- 172.21.15.1:0/625867554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db40c6b10 msgr2=0x7f0db4158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.538+0000 7f0dba6eb640 1 --2- 172.21.15.1:0/625867554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db40c6b10 0x7f0db4158690 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f0db41597b0 tx=0x7f0da802b880 comp rx=0 tx=0).stop 2024-03-06T07:24:20.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.539+0000 7f0dba6eb640 1 -- 172.21.15.1:0/625867554 shutdown_connections 2024-03-06T07:24:20.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.539+0000 7f0dba6eb640 1 --2- 172.21.15.1:0/625867554 >> v2:172.21.15.98:3300/0 conn(0x7f0db40bf9d0 0x7f0db40c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.539+0000 7f0dba6eb640 1 --2- 172.21.15.1:0/625867554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db40c6b10 0x7f0db4158690 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.539+0000 7f0dba6eb640 1 --2- 172.21.15.1:0/625867554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db4152650 0x7f0db4155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.539+0000 7f0dba6eb640 1 -- 172.21.15.1:0/625867554 >> 172.21.15.1:0/625867554 conn(0x7f0db4066ee0 msgr2=0x7f0db40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.539+0000 7f0dba6eb640 1 -- 172.21.15.1:0/625867554 shutdown_connections 2024-03-06T07:24:20.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.539+0000 7f0dba6eb640 1 -- 172.21.15.1:0/625867554 wait complete. 2024-03-06T07:24:20.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.540+0000 7f0dba6eb640 1 Processor -- start 2024-03-06T07:24:20.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.540+0000 7f0dba6eb640 1 -- start start 2024-03-06T07:24:20.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.540+0000 7f0dba6eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db40bf9d0 0x7f0db414d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.540+0000 7f0dba6eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0db4152650 0x7f0db41573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.540+0000 7f0dba6eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db40c9e10 0x7f0db4157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.540+0000 7f0dba6eb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db4155490 con 0x7f0db40bf9d0 2024-03-06T07:24:20.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.540+0000 7f0dba6eb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db41551b0 con 0x7f0db4152650 2024-03-06T07:24:20.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.540+0000 7f0dba6eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db4152a30 con 0x7f0db40c9e10 2024-03-06T07:24:20.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.541+0000 7f0db3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db40bf9d0 0x7f0db414d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.541+0000 7f0db3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db40bf9d0 0x7f0db414d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44468/0 (socket says 172.21.15.1:44468) 2024-03-06T07:24:20.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.541+0000 7f0db3fff640 1 -- 172.21.15.1:0/1078788638 learned_addr learned my addr 172.21.15.1:0/1078788638 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:20.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.541+0000 7f0db8c88640 1 --2- 172.21.15.1:0/1078788638 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db40c9e10 0x7f0db4157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.541+0000 7f0db37fe640 1 --2- 172.21.15.1:0/1078788638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0db4152650 0x7f0db41573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.541+0000 7f0db3fff640 1 -- 172.21.15.1:0/1078788638 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db40c9e10 msgr2=0x7f0db4157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.541+0000 7f0db3fff640 1 --2- 172.21.15.1:0/1078788638 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db40c9e10 0x7f0db4157910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.541+0000 7f0db3fff640 1 -- 172.21.15.1:0/1078788638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0db4152650 msgr2=0x7f0db41573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.541+0000 7f0db3fff640 1 --2- 172.21.15.1:0/1078788638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0db4152650 0x7f0db41573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.541+0000 7f0db3fff640 1 -- 172.21.15.1:0/1078788638 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db40c95c0 con 0x7f0db40bf9d0 2024-03-06T07:24:20.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.541+0000 7f0db8c88640 1 --2- 172.21.15.1:0/1078788638 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db40c9e10 0x7f0db4157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:20.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.541+0000 7f0db3fff640 1 --2- 172.21.15.1:0/1078788638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db40bf9d0 0x7f0db414d500 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f0da400d830 tx=0x7f0da400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.541+0000 7f0db0ff9640 1 -- 172.21.15.1:0/1078788638 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da400b6c0 con 0x7f0db40bf9d0 2024-03-06T07:24:20.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.541+0000 7f0db0ff9640 1 -- 172.21.15.1:0/1078788638 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da4019070 con 0x7f0db40bf9d0 2024-03-06T07:24:20.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.541+0000 7f0dba6eb640 1 -- 172.21.15.1:0/1078788638 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db41568f0 con 0x7f0db40bf9d0 2024-03-06T07:24:20.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.541+0000 7f0db0ff9640 1 -- 172.21.15.1:0/1078788638 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da400ba80 con 0x7f0db40bf9d0 2024-03-06T07:24:20.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.541+0000 7f0dba6eb640 1 -- 172.21.15.1:0/1078788638 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db414da40 con 0x7f0db40bf9d0 2024-03-06T07:24:20.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.542+0000 7f0dba6eb640 1 -- 172.21.15.1:0/1078788638 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db4067fd0 con 0x7f0db40bf9d0 2024-03-06T07:24:20.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.543+0000 7f0db0ff9640 1 -- 172.21.15.1:0/1078788638 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0da401e460 con 0x7f0db40bf9d0 2024-03-06T07:24:20.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.543+0000 7f0db0ff9640 1 --2- 172.21.15.1:0/1078788638 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0da006b500 0x7f0da006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.543+0000 7f0db0ff9640 1 -- 172.21.15.1:0/1078788638 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0da408b3e0 con 0x7f0db40bf9d0 2024-03-06T07:24:20.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.544+0000 7f0db37fe640 1 --2- 172.21.15.1:0/1078788638 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0da006b500 0x7f0da006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.544+0000 7f0db37fe640 1 --2- 172.21.15.1:0/1078788638 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0da006b500 0x7f0da006d9c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0da8036000 tx=0x7f0da803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.547+0000 7f0db0ff9640 1 -- 172.21.15.1:0/1078788638 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0da405a240 con 0x7f0db40bf9d0 2024-03-06T07:24:20.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.816+0000 7f0dba6eb640 1 -- 172.21.15.1:0/1078788638 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f0db40b06f0 con 0x7f0db40bf9d0 2024-03-06T07:24:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.819+0000 7f0db0ff9640 1 -- 172.21.15.1:0/1078788638 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v36)=0 v36) v1 ==== 116+0+0 (secure 0 0 0) 0x7f0da4059b40 con 0x7f0db40bf9d0 2024-03-06T07:24:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.821+0000 7f0dba6eb640 1 -- 172.21.15.1:0/1078788638 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0da006b500 msgr2=0x7f0da006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.821+0000 7f0dba6eb640 1 --2- 172.21.15.1:0/1078788638 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0da006b500 0x7f0da006d9c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0da8036000 tx=0x7f0da803d040 comp rx=0 tx=0).stop 2024-03-06T07:24:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.821+0000 7f0dba6eb640 1 -- 172.21.15.1:0/1078788638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db40bf9d0 msgr2=0x7f0db414d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.821+0000 7f0dba6eb640 1 --2- 172.21.15.1:0/1078788638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db40bf9d0 0x7f0db414d500 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f0da400d830 tx=0x7f0da400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.822+0000 7f0dba6eb640 1 -- 172.21.15.1:0/1078788638 shutdown_connections 2024-03-06T07:24:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.822+0000 7f0dba6eb640 1 --2- 172.21.15.1:0/1078788638 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0da006b500 0x7f0da006d9c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.822+0000 7f0dba6eb640 1 --2- 172.21.15.1:0/1078788638 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0db40c9e10 0x7f0db4157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.822+0000 7f0dba6eb640 1 --2- 172.21.15.1:0/1078788638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0db4152650 0x7f0db41573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.822+0000 7f0dba6eb640 1 --2- 172.21.15.1:0/1078788638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db40bf9d0 0x7f0db414d500 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.822+0000 7f0dba6eb640 1 -- 172.21.15.1:0/1078788638 >> 172.21.15.1:0/1078788638 conn(0x7f0db4066ee0 msgr2=0x7f0db40cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.822+0000 7f0dba6eb640 1 -- 172.21.15.1:0/1078788638 shutdown_connections 2024-03-06T07:24:20.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.822+0000 7f0dba6eb640 1 -- 172.21.15.1:0/1078788638 wait complete. 2024-03-06T07:24:20.838 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:20.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.938+0000 7f73eb3eb640 1 Processor -- start 2024-03-06T07:24:20.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.938+0000 7f73eb3eb640 1 -- start start 2024-03-06T07:24:20.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.938+0000 7f73eb3eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73e4152650 0x7f73e4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.938+0000 7f73eb3eb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f73e40c6b10 0x7f73e4158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.938+0000 7f73eb3eb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e40c2fc0 con 0x7f73e40bf9d0 2024-03-06T07:24:20.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.938+0000 7f73eb3eb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e4155800 con 0x7f73e40c6b10 2024-03-06T07:24:20.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.938+0000 7f73eb3eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e40c6690 con 0x7f73e4152650 2024-03-06T07:24:20.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.939+0000 7f73e8986640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f73e40c6b10 0x7f73e4158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.939+0000 7f73e8986640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f73e40c6b10 0x7f73e4158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44482/0 (socket says 172.21.15.1:44482) 2024-03-06T07:24:20.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.939+0000 7f73e8986640 1 -- 172.21.15.1:0/2135096560 learned_addr learned my addr 172.21.15.1:0/2135096560 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:20.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.939+0000 7f73e8986640 1 -- 172.21.15.1:0/2135096560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73e4152650 msgr2=0x7f73e4155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.939+0000 7f73e9187640 1 --2- 172.21.15.1:0/2135096560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73e4152650 0x7f73e4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.939+0000 7f73e8986640 1 --2- 172.21.15.1:0/2135096560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73e4152650 0x7f73e4155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.939+0000 7f73e8986640 1 -- 172.21.15.1:0/2135096560 >> v1:172.21.15.98:6789/0 conn(0x7f73e40bf9d0 legacy=0x7f73e40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.939+0000 7f73e8986640 1 -- 172.21.15.1:0/2135096560 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e40c7280 con 0x7f73e40c6b10 2024-03-06T07:24:20.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.939+0000 7f73e9187640 1 --2- 172.21.15.1:0/2135096560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73e4152650 0x7f73e4155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:20.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.940+0000 7f73e8986640 1 --2- 172.21.15.1:0/2135096560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73e40c6b10 0x7f73e4158690 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f73d4009870 tx=0x7f73d402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3825cecb93db11 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.940+0000 7f73dbfff640 1 -- 172.21.15.1:0/2135096560 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d40317c0 con 0x7f73e40c6b10 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.940+0000 7f73dbfff640 1 -- 172.21.15.1:0/2135096560 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d4037070 con 0x7f73e40c6b10 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.940+0000 7f73dbfff640 1 -- 172.21.15.1:0/2135096560 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d4031b80 con 0x7f73e40c6b10 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.940+0000 7f73eb3eb640 1 -- 172.21.15.1:0/2135096560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73e40c6b10 msgr2=0x7f73e4158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.940+0000 7f73eb3eb640 1 --2- 172.21.15.1:0/2135096560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73e40c6b10 0x7f73e4158690 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f73d4009870 tx=0x7f73d402b860 comp rx=0 tx=0).stop 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.940+0000 7f73eb3eb640 1 -- 172.21.15.1:0/2135096560 shutdown_connections 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.940+0000 7f73eb3eb640 1 --2- 172.21.15.1:0/2135096560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73e40c6b10 0x7f73e4158690 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.940+0000 7f73eb3eb640 1 --2- 172.21.15.1:0/2135096560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73e4152650 0x7f73e4155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.940+0000 7f73eb3eb640 1 -- 172.21.15.1:0/2135096560 >> 172.21.15.1:0/2135096560 conn(0x7f73e4066ee0 msgr2=0x7f73e40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.941+0000 7f73eb3eb640 1 -- 172.21.15.1:0/2135096560 shutdown_connections 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.941+0000 7f73eb3eb640 1 -- 172.21.15.1:0/2135096560 wait complete. 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.942+0000 7f73eb3eb640 1 Processor -- start 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.942+0000 7f73eb3eb640 1 -- start start 2024-03-06T07:24:20.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.942+0000 7f73eb3eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73e40bf9d0 0x7f73e40c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.942+0000 7f73eb3eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73e4152650 0x7f73e4148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.942+0000 7f73eb3eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73e40c9da0 0x7f73e4149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.942+0000 7f73eb3eb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e4155490 con 0x7f73e40c9da0 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.942+0000 7f73eb3eb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e41551b0 con 0x7f73e40bf9d0 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.942+0000 7f73eb3eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e4152a30 con 0x7f73e4152650 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.942+0000 7f73e9988640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73e40c9da0 0x7f73e4149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.942+0000 7f73e9988640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73e40c9da0 0x7f73e4149340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44490/0 (socket says 172.21.15.1:44490) 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.942+0000 7f73e9988640 1 -- 172.21.15.1:0/3345419997 learned_addr learned my addr 172.21.15.1:0/3345419997 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.943+0000 7f73e8986640 1 --2- 172.21.15.1:0/3345419997 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73e4152650 0x7f73e4148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.943+0000 7f73e9187640 1 --2- 172.21.15.1:0/3345419997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73e40bf9d0 0x7f73e40c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.943+0000 7f73e9988640 1 -- 172.21.15.1:0/3345419997 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73e4152650 msgr2=0x7f73e4148e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.943+0000 7f73e9988640 1 --2- 172.21.15.1:0/3345419997 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73e4152650 0x7f73e4148e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.943+0000 7f73e9988640 1 -- 172.21.15.1:0/3345419997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73e40bf9d0 msgr2=0x7f73e40c8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.943+0000 7f73e9988640 1 --2- 172.21.15.1:0/3345419997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73e40bf9d0 0x7f73e40c8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.943+0000 7f73e9988640 1 -- 172.21.15.1:0/3345419997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e414de20 con 0x7f73e40c9da0 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.943+0000 7f73e9187640 1 --2- 172.21.15.1:0/3345419997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73e40bf9d0 0x7f73e40c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.943+0000 7f73e8986640 1 --2- 172.21.15.1:0/3345419997 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73e4152650 0x7f73e4148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.943+0000 7f73e9988640 1 --2- 172.21.15.1:0/3345419997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73e40c9da0 0x7f73e4149340 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f73dc00b9a0 tx=0x7f73dc00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.943+0000 7f73d9ffb640 1 -- 172.21.15.1:0/3345419997 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc019070 con 0x7f73e40c9da0 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.943+0000 7f73d9ffb640 1 -- 172.21.15.1:0/3345419997 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73dc00cba0 con 0x7f73e40c9da0 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.943+0000 7f73d9ffb640 1 -- 172.21.15.1:0/3345419997 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc014440 con 0x7f73e40c9da0 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.943+0000 7f73eb3eb640 1 -- 172.21.15.1:0/3345419997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e41ee540 con 0x7f73e40c9da0 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.943+0000 7f73eb3eb640 1 -- 172.21.15.1:0/3345419997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e40c8f80 con 0x7f73e40c9da0 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.945+0000 7f73eb3eb640 1 -- 172.21.15.1:0/3345419997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e4067fd0 con 0x7f73e40c9da0 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.945+0000 7f73d9ffb640 1 -- 172.21.15.1:0/3345419997 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73dc01c070 con 0x7f73e40c9da0 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.945+0000 7f73d9ffb640 1 --2- 172.21.15.1:0/3345419997 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73c806b4d0 0x7f73c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.945+0000 7f73d9ffb640 1 -- 172.21.15.1:0/3345419997 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f73dc089ef0 con 0x7f73e40c9da0 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.946+0000 7f73e9187640 1 --2- 172.21.15.1:0/3345419997 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73c806b4d0 0x7f73c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.946+0000 7f73e9187640 1 --2- 172.21.15.1:0/3345419997 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73c806b4d0 0x7f73c806d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f73e000aa30 tx=0x7f73e0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:20.950+0000 7f73d9ffb640 1 -- 172.21.15.1:0/3345419997 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73dc058c70 con 0x7f73e40c9da0 2024-03-06T07:24:21.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.219+0000 7f73eb3eb640 1 -- 172.21.15.1:0/3345419997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73e40b06f0 con 0x7f73e40c9da0 2024-03-06T07:24:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.219+0000 7f73d9ffb640 1 -- 172.21.15.1:0/3345419997 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f73dc058570 con 0x7f73e40c9da0 2024-03-06T07:24:21.221 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:21.221 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":88,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6122,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2632566151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2632566151},{"type":"v1","addr":"172.21.15.98:6837","nonce":2632566151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":6670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2244230066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2244230066},{"type":"v1","addr":"172.21.15.1:6835","nonce":2244230066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":6682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1741783790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1741783790},{"type":"v1","addr":"172.21.15.98:6836","nonce":1741783790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:10.220198+0000","modified":"2024-03-06T07:24:12.224326+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6833/2433923518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2433923518},{"type":"v1","addr":"172.21.15.1:6833","nonce":2433923518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":8}]} 2024-03-06T07:24:21.222 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 88 2024-03-06T07:24:21.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.222+0000 7f73eb3eb640 1 -- 172.21.15.1:0/3345419997 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73c806b4d0 msgr2=0x7f73c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.222+0000 7f73eb3eb640 1 --2- 172.21.15.1:0/3345419997 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73c806b4d0 0x7f73c806d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f73e000aa30 tx=0x7f73e0008040 comp rx=0 tx=0).stop 2024-03-06T07:24:21.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.222+0000 7f73eb3eb640 1 -- 172.21.15.1:0/3345419997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73e40c9da0 msgr2=0x7f73e4149340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.222+0000 7f73eb3eb640 1 --2- 172.21.15.1:0/3345419997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73e40c9da0 0x7f73e4149340 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f73dc00b9a0 tx=0x7f73dc00be60 comp rx=0 tx=0).stop 2024-03-06T07:24:21.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.222+0000 7f73eb3eb640 1 -- 172.21.15.1:0/3345419997 shutdown_connections 2024-03-06T07:24:21.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.222+0000 7f73eb3eb640 1 --2- 172.21.15.1:0/3345419997 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73c806b4d0 0x7f73c806d990 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.222+0000 7f73eb3eb640 1 --2- 172.21.15.1:0/3345419997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73e40c9da0 0x7f73e4149340 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.222+0000 7f73eb3eb640 1 --2- 172.21.15.1:0/3345419997 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f73e4152650 0x7f73e4148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.222+0000 7f73eb3eb640 1 --2- 172.21.15.1:0/3345419997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73e40bf9d0 0x7f73e40c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.222+0000 7f73eb3eb640 1 -- 172.21.15.1:0/3345419997 >> 172.21.15.1:0/3345419997 conn(0x7f73e4066ee0 msgr2=0x7f73e414e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.222+0000 7f73eb3eb640 1 -- 172.21.15.1:0/3345419997 shutdown_connections 2024-03-06T07:24:21.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.222+0000 7f73eb3eb640 1 -- 172.21.15.1:0/3345419997 wait complete. 2024-03-06T07:24:21.639 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:10.220198+0000', 'modified': '2024-03-06T07:24:12.224326+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7362}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7362': {'gid': 7362, 'name': 'c', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6833/2433923518', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2433923518}, {'type': 'v1', 'addr': '172.21.15.1:6833', 'nonce': 2433923518}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7362, 'qdb_cluster': [7362]} 2024-03-06T07:24:21.640 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:24:21.640 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:24:21.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.741+0000 7fccba0e0640 1 Processor -- start 2024-03-06T07:24:21.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.741+0000 7fccba0e0640 1 -- start start 2024-03-06T07:24:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.741+0000 7fccba0e0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fccb40c3970 0x7fccb40c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.741+0000 7fccba0e0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccb40c60b0 0x7fccb4152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccba0e0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb40c5e40 con 0x7fccb4152670 2024-03-06T07:24:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccba0e0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb40c67f0 con 0x7fccb40c3970 2024-03-06T07:24:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccba0e0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb40c6a40 con 0x7fccb40c60b0 2024-03-06T07:24:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fccb4152670 0x7fccb41553d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36048/0 (socket says 172.21.15.1:36048) 2024-03-06T07:24:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb3fff640 1 -- 172.21.15.1:0/1129409872 learned_addr learned my addr 172.21.15.1:0/1129409872 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb2ffd640 1 --2- 172.21.15.1:0/1129409872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccb40c60b0 0x7fccb4152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb37fe640 1 --2- 172.21.15.1:0/1129409872 >> v2:172.21.15.98:3300/0 conn(0x7fccb40c3970 0x7fccb40c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb27fc640 1 -- 172.21.15.1:0/1129409872 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291259868 0 0) 0x7fccb40c5e40 con 0x7fccb4152670 2024-03-06T07:24:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb27fc640 1 -- 172.21.15.1:0/1129409872 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc9c003520 con 0x7fccb4152670 2024-03-06T07:24:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb27fc640 1 -- 172.21.15.1:0/1129409872 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1328210992 0 0) 0x7fcc9c003520 con 0x7fccb4152670 2024-03-06T07:24:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb27fc640 1 -- 172.21.15.1:0/1129409872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccb40c60b0 msgr2=0x7fccb4152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb27fc640 1 --2- 172.21.15.1:0/1129409872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccb40c60b0 0x7fccb4152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb27fc640 1 -- 172.21.15.1:0/1129409872 >> v2:172.21.15.98:3300/0 conn(0x7fccb40c3970 msgr2=0x7fccb40c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb27fc640 1 --2- 172.21.15.1:0/1129409872 >> v2:172.21.15.98:3300/0 conn(0x7fccb40c3970 0x7fccb40c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb27fc640 1 -- 172.21.15.1:0/1129409872 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb40c6d20 con 0x7fccb4152670 2024-03-06T07:24:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb27fc640 1 -- 172.21.15.1:0/1129409872 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fccac0019f0 con 0x7fccb4152670 2024-03-06T07:24:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb27fc640 1 -- 172.21.15.1:0/1129409872 >> v1:172.21.15.1:6789/0 conn(0x7fccb4152670 legacy=0x7fccb41553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb27fc640 1 --2- 172.21.15.1:0/1129409872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9c0039b0 0x7fcc9c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb27fc640 1 --2- 172.21.15.1:0/1129409872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccb40c60b0 0x7fcc9c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb27fc640 1 --2- 172.21.15.1:0/1129409872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc9c004d40 0x7fcc9c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb27fc640 1 -- 172.21.15.1:0/1129409872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb40c67f0 con 0x7fcc9c004d40 2024-03-06T07:24:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb27fc640 1 -- 172.21.15.1:0/1129409872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb40c6a40 con 0x7fcc9c0039b0 2024-03-06T07:24:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.742+0000 7fccb27fc640 1 -- 172.21.15.1:0/1129409872 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c00b9c0 con 0x7fccb40c60b0 2024-03-06T07:24:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.743+0000 7fccb3fff640 1 --2- 172.21.15.1:0/1129409872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc9c004d40 0x7fcc9c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.743+0000 7fccb37fe640 1 --2- 172.21.15.1:0/1129409872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9c0039b0 0x7fcc9c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.743+0000 7fccb2ffd640 1 --2- 172.21.15.1:0/1129409872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccb40c60b0 0x7fcc9c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.743+0000 7fccb37fe640 1 -- 172.21.15.1:0/1129409872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccb40c60b0 msgr2=0x7fcc9c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.743+0000 7fccb37fe640 1 --2- 172.21.15.1:0/1129409872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccb40c60b0 0x7fcc9c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.743+0000 7fccb37fe640 1 -- 172.21.15.1:0/1129409872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc9c004d40 msgr2=0x7fcc9c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.743+0000 7fccb37fe640 1 --2- 172.21.15.1:0/1129409872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc9c004d40 0x7fcc9c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.743+0000 7fccb37fe640 1 -- 172.21.15.1:0/1129409872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9c003520 con 0x7fcc9c0039b0 2024-03-06T07:24:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.743+0000 7fccb3fff640 1 --2- 172.21.15.1:0/1129409872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc9c004d40 0x7fcc9c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.743+0000 7fccb2ffd640 1 --2- 172.21.15.1:0/1129409872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccb40c60b0 0x7fcc9c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.744+0000 7fccb37fe640 1 --2- 172.21.15.1:0/1129409872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9c0039b0 0x7fcc9c003dd0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fcca4009ac0 tx=0x7fcc9c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1459c79953179566 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.744+0000 7fccb27fc640 1 -- 172.21.15.1:0/1129409872 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca4039070 con 0x7fcc9c0039b0 2024-03-06T07:24:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.744+0000 7fccb27fc640 1 -- 172.21.15.1:0/1129409872 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca402c710 con 0x7fcc9c0039b0 2024-03-06T07:24:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.744+0000 7fccb27fc640 1 -- 172.21.15.1:0/1129409872 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca4009690 con 0x7fcc9c0039b0 2024-03-06T07:24:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.744+0000 7fccba0e0640 1 -- 172.21.15.1:0/1129409872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9c0039b0 msgr2=0x7fcc9c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.744+0000 7fccba0e0640 1 --2- 172.21.15.1:0/1129409872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc9c0039b0 0x7fcc9c003dd0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fcca4009ac0 tx=0x7fcc9c005830 comp rx=0 tx=0).stop 2024-03-06T07:24:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.744+0000 7fccb3fff640 1 -- 172.21.15.1:0/1129409872 reap_dead start 2024-03-06T07:24:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.744+0000 7fccba0e0640 1 -- 172.21.15.1:0/1129409872 shutdown_connections 2024-03-06T07:24:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.744+0000 7fccba0e0640 1 -- 172.21.15.1:0/1129409872 >> 172.21.15.1:0/1129409872 conn(0x7fccb40c88f0 msgr2=0x7fccb40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.744+0000 7fccba0e0640 1 -- 172.21.15.1:0/1129409872 shutdown_connections 2024-03-06T07:24:21.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.744+0000 7fccba0e0640 1 -- 172.21.15.1:0/1129409872 wait complete. 2024-03-06T07:24:21.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.745+0000 7fccba0e0640 1 Processor -- start 2024-03-06T07:24:21.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.746+0000 7fccba0e0640 1 -- start start 2024-03-06T07:24:21.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccba0e0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc9c0039b0 0x7fccb40a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccba0e0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb4156c50 0x7fccb41eafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccba0e0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccb41eb4f0 0x7fccb406f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccba0e0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb40c75c0 con 0x7fccb4156c50 2024-03-06T07:24:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccba0e0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb40c72e0 con 0x7fccb41eb4f0 2024-03-06T07:24:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccba0e0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb40c3d50 con 0x7fcc9c0039b0 2024-03-06T07:24:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccb2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb4156c50 0x7fccb41eafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccb2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb4156c50 0x7fccb41eafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44504/0 (socket says 172.21.15.1:44504) 2024-03-06T07:24:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccb2ffd640 1 -- 172.21.15.1:0/1480001286 learned_addr learned my addr 172.21.15.1:0/1480001286 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccb37fe640 1 --2- 172.21.15.1:0/1480001286 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc9c0039b0 0x7fccb40a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccb3fff640 1 --2- 172.21.15.1:0/1480001286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccb41eb4f0 0x7fccb406f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccb2ffd640 1 -- 172.21.15.1:0/1480001286 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc9c0039b0 msgr2=0x7fccb40a97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccb2ffd640 1 --2- 172.21.15.1:0/1480001286 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc9c0039b0 0x7fccb40a97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccb2ffd640 1 -- 172.21.15.1:0/1480001286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccb41eb4f0 msgr2=0x7fccb406f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccb2ffd640 1 --2- 172.21.15.1:0/1480001286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccb41eb4f0 0x7fccb406f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccb2ffd640 1 -- 172.21.15.1:0/1480001286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb41eaab0 con 0x7fccb4156c50 2024-03-06T07:24:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccb3fff640 1 --2- 172.21.15.1:0/1480001286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccb41eb4f0 0x7fccb406f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fccb2ffd640 1 --2- 172.21.15.1:0/1480001286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb4156c50 0x7fccb41eafb0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fcca8011a70 tx=0x7fcca8011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.747+0000 7fcc9bfff640 1 -- 172.21.15.1:0/1480001286 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca800fbb0 con 0x7fccb4156c50 2024-03-06T07:24:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.748+0000 7fcc9bfff640 1 -- 172.21.15.1:0/1480001286 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca800fd50 con 0x7fccb4156c50 2024-03-06T07:24:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.748+0000 7fcc9bfff640 1 -- 172.21.15.1:0/1480001286 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8014680 con 0x7fccb4156c50 2024-03-06T07:24:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.748+0000 7fccba0e0640 1 -- 172.21.15.1:0/1480001286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb41eea30 con 0x7fccb4156c50 2024-03-06T07:24:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.748+0000 7fccba0e0640 1 -- 172.21.15.1:0/1480001286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb4071160 con 0x7fccb4156c50 2024-03-06T07:24:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.749+0000 7fcc9bfff640 1 -- 172.21.15.1:0/1480001286 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcca8012040 con 0x7fccb4156c50 2024-03-06T07:24:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.749+0000 7fcc9bfff640 1 --2- 172.21.15.1:0/1480001286 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcca006b4d0 0x7fcca006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.750+0000 7fcc9bfff640 1 -- 172.21.15.1:0/1480001286 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcca808abc0 con 0x7fccb4156c50 2024-03-06T07:24:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.750+0000 7fccb37fe640 1 --2- 172.21.15.1:0/1480001286 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcca006b4d0 0x7fcca006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.750+0000 7fccbcd9d740 1 Processor -- start 2024-03-06T07:24:21.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.750+0000 7fccbcd9d740 1 -- start start 2024-03-06T07:24:21.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.750+0000 7fccb37fe640 1 --2- 172.21.15.1:0/1480001286 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcca006b4d0 0x7fcca006d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fcca4036000 tx=0x7fcca4030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.751+0000 7fccbcd9d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a6bf50 0x564951a69a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.751+0000 7fccbcd9d740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a6c330 0x564951a69f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.751+0000 7fccbcd9d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6a4b0 0x564951a66680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.751+0000 7fccbcd9d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649519d06b0 con 0x564951a6a4b0 2024-03-06T07:24:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.751+0000 7fccbcd9d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x564951979a60 con 0x564951a6bf50 2024-03-06T07:24:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.751+0000 7fccbcd9d740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5649519cc750 con 0x564951a6c330 2024-03-06T07:24:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.751+0000 7fccb3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6a4b0 0x564951a66680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.751+0000 7fccb3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6a4b0 0x564951a66680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44520/0 (socket says 172.21.15.1:44520) 2024-03-06T07:24:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.751+0000 7fccb3fff640 1 -- 172.21.15.1:0/968359190 learned_addr learned my addr 172.21.15.1:0/968359190 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.751+0000 7fccb37fe640 1 --2- 172.21.15.1:0/968359190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a6bf50 0x564951a69a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.751+0000 7fccb2ffd640 1 --2- 172.21.15.1:0/968359190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a6c330 0x564951a69f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.752+0000 7fccb3fff640 1 -- 172.21.15.1:0/968359190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a6c330 msgr2=0x564951a69f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.752+0000 7fccb3fff640 1 --2- 172.21.15.1:0/968359190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a6c330 0x564951a69f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.752+0000 7fccb3fff640 1 -- 172.21.15.1:0/968359190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a6bf50 msgr2=0x564951a69a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.752+0000 7fccb3fff640 1 --2- 172.21.15.1:0/968359190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a6bf50 0x564951a69a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.752+0000 7fccb3fff640 1 -- 172.21.15.1:0/968359190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564951997f30 con 0x564951a6a4b0 2024-03-06T07:24:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.752+0000 7fccb3fff640 1 --2- 172.21.15.1:0/968359190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6a4b0 0x564951a66680 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fccac000f80 tx=0x7fccac00a0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928d03268561fe7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.752+0000 7fcc997fa640 1 -- 172.21.15.1:0/968359190 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac00a4c0 con 0x564951a6a4b0 2024-03-06T07:24:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.752+0000 7fcc997fa640 1 -- 172.21.15.1:0/968359190 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccac01a070 con 0x564951a6a4b0 2024-03-06T07:24:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.752+0000 7fcc997fa640 1 -- 172.21.15.1:0/968359190 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8009690 con 0x564951a6a4b0 2024-03-06T07:24:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.753+0000 7fccbcd9d740 1 -- 172.21.15.1:0/968359190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6a4b0 msgr2=0x564951a66680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.753+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/968359190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6a4b0 0x564951a66680 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fccac000f80 tx=0x7fccac00a0e0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.753+0000 7fccbcd9d740 1 -- 172.21.15.1:0/968359190 shutdown_connections 2024-03-06T07:24:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.753+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/968359190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6a4b0 0x564951a66680 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.753+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/968359190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a6c330 0x564951a69f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.753+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/968359190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a6bf50 0x564951a69a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.753+0000 7fccbcd9d740 1 -- 172.21.15.1:0/968359190 >> 172.21.15.1:0/968359190 conn(0x564951a687d0 msgr2=0x564951a8d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.753+0000 7fccbcd9d740 1 -- 172.21.15.1:0/968359190 shutdown_connections 2024-03-06T07:24:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.753+0000 7fccbcd9d740 1 -- 172.21.15.1:0/968359190 wait complete. 2024-03-06T07:24:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.753+0000 7fccbcd9d740 1 -- start start 2024-03-06T07:24:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.753+0000 7fccbcd9d740 1 Processor -- start 2024-03-06T07:24:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccbcd9d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6bf50 0x564951aaba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccbcd9d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a6c330 0x564951ab87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccbcd9d740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a908c0 0x564951a88e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccbcd9d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x564951a8e230 con 0x564951a6bf50 2024-03-06T07:24:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccbcd9d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x56495198cea0 con 0x564951a6c330 2024-03-06T07:24:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccbcd9d740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x564951a8dc70 con 0x564951a908c0 2024-03-06T07:24:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccbcd9d740 10 client.0 _ll_register_callbacks cb 0x56495198c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccb37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6bf50 0x564951aaba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccb37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6bf50 0x564951aaba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44526/0 (socket says 172.21.15.1:44526) 2024-03-06T07:24:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccb37fe640 1 -- 172.21.15.1:0/2682767041 learned_addr learned my addr 172.21.15.1:0/2682767041 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccb3fff640 1 --2- 172.21.15.1:0/2682767041 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a908c0 0x564951a88e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccb2ffd640 1 --2- 172.21.15.1:0/2682767041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a6c330 0x564951ab87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccb37fe640 1 -- 172.21.15.1:0/2682767041 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a908c0 msgr2=0x564951a88e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccb37fe640 1 --2- 172.21.15.1:0/2682767041 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a908c0 0x564951a88e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccb37fe640 1 -- 172.21.15.1:0/2682767041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a6c330 msgr2=0x564951ab87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccb37fe640 1 --2- 172.21.15.1:0/2682767041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a6c330 0x564951ab87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccb37fe640 1 -- 172.21.15.1:0/2682767041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56495198d270 con 0x564951a6bf50 2024-03-06T07:24:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.754+0000 7fccb2ffd640 1 --2- 172.21.15.1:0/2682767041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a6c330 0x564951ab87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.755+0000 7fccb37fe640 1 --2- 172.21.15.1:0/2682767041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6bf50 0x564951aaba70 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fcca4006f80 tx=0x7fcca40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937e9ea2edc9a6f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.755+0000 7fcc927fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:24:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.755+0000 7fcc927fc640 1 -- 172.21.15.1:0/2682767041 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca404e020 con 0x564951a6bf50 2024-03-06T07:24:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.755+0000 7fcc927fc640 1 -- 172.21.15.1:0/2682767041 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcca402cc60 con 0x564951a6bf50 2024-03-06T07:24:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.755+0000 7fccbcd9d740 1 -- 172.21.15.1:0/2682767041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fccac01a070 con 0x564951a6bf50 2024-03-06T07:24:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.755+0000 7fcc927fc640 1 -- 172.21.15.1:0/2682767041 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcca4058050 con 0x564951a6bf50 2024-03-06T07:24:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.755+0000 7fccbcd9d740 10 client.7578 fetch_fsmap learned FSMap version 88 2024-03-06T07:24:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.755+0000 7fccbcd9d740 1 -- 172.21.15.1:0/2682767041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) v3 -- 0x5649516e52d0 con 0x564951a6bf50 2024-03-06T07:24:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.755+0000 7fcc927fc640 1 -- 172.21.15.1:0/2682767041 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccac01a070 con 0x564951a6bf50 2024-03-06T07:24:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.755+0000 7fcc927fc640 1 -- 172.21.15.1:0/2682767041 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 88) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fcca405b020 con 0x564951a6bf50 2024-03-06T07:24:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.755+0000 7fccbcd9d740 10 client.7578 fetch_fsmap finished waiting for FSMap version 88 2024-03-06T07:24:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.755+0000 7fccbcd9d740 10 client.7578 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:24:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.755+0000 7fccbcd9d740 20 client.7578 populate_metadata read hostname 'smithi001' 2024-03-06T07:24:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.755+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/2682767041 >> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] conn(0x564951abc5d0 0x564951ac7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.755+0000 7fccbcd9d740 4 client.7578 mds_command: new command op to 7362 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:24:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.755+0000 7fccbcd9d740 1 -- 172.21.15.1:0/2682767041 --> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5649519ddbf0 con 0x564951abc5d0 2024-03-06T07:24:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.755+0000 7fccb2ffd640 1 --2- 172.21.15.1:0/2682767041 >> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] conn(0x564951abc5d0 0x564951ac7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.756+0000 7fcc927fc640 10 client.7578 ms_handle_connect on v2:172.21.15.1:6832/2433923518 2024-03-06T07:24:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.756+0000 7fccb2ffd640 1 --2- 172.21.15.1:0/2682767041 >> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] conn(0x564951abc5d0 0x564951ac7360 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cd316fd19c91f78 server_cookie=d615d9a890b5904b in_seq=0 out_seq=0 2024-03-06T07:24:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.756+0000 7fcc927fc640 1 -- 172.21.15.1:0/2682767041 <== mds.0 v2:172.21.15.1:6832/2433923518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5649519ddbf0 con 0x564951abc5d0 2024-03-06T07:24:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.756+0000 7fcc927fc640 10 client.7578 handle_command_reply: tid=0 2024-03-06T07:24:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.756+0000 7fccbcd9d740 1 client.7578 shutdown 2024-03-06T07:24:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.756+0000 7fccbcd9d740 1 -- 172.21.15.1:0/2682767041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6bf50 msgr2=0x564951aaba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.756+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/2682767041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6bf50 0x564951aaba70 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fcca4006f80 tx=0x7fcca40046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.756+0000 7fccbcd9d740 1 -- 172.21.15.1:0/2682767041 shutdown_connections 2024-03-06T07:24:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.756+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/2682767041 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a908c0 0x564951a88e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.756+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/2682767041 >> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] conn(0x564951abc5d0 0x564951ac7360 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.757+0000 7fcc927fc640 0 client.7578 ms_handle_reset on v2:172.21.15.1:6832/2433923518 2024-03-06T07:24:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.757+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/2682767041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a6c330 0x564951ab87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.757+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/2682767041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6bf50 0x564951aaba70 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.757+0000 7fccbcd9d740 1 -- 172.21.15.1:0/2682767041 >> 172.21.15.1:0/2682767041 conn(0x564951a6a4b0 msgr2=0x564951a6a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.757+0000 7fccbcd9d740 1 -- 172.21.15.1:0/2682767041 shutdown_connections 2024-03-06T07:24:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.757+0000 7fccbcd9d740 1 -- 172.21.15.1:0/2682767041 wait complete. 2024-03-06T07:24:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.757+0000 7fccbcd9d740 20 client.7578 trim_cache size 0 max 16384 2024-03-06T07:24:21.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.773+0000 7fccbcd9d740 1 Processor -- start 2024-03-06T07:24:21.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.773+0000 7fccbcd9d740 1 -- start start 2024-03-06T07:24:21.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.774+0000 7fccbcd9d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6bf50 0x564951aa5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.774+0000 7fccbcd9d740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a6c330 0x564951a6a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.774+0000 7fccbcd9d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a908c0 0x564951ab5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.774+0000 7fccbcd9d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb41f7430 con 0x564951a6bf50 2024-03-06T07:24:21.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.774+0000 7fccbcd9d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb41f6860 con 0x564951a908c0 2024-03-06T07:24:21.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.774+0000 7fccbcd9d740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb41f6e60 con 0x564951a6c330 2024-03-06T07:24:21.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.774+0000 7fccb37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6bf50 0x564951aa5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.774+0000 7fccb37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6bf50 0x564951aa5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44538/0 (socket says 172.21.15.1:44538) 2024-03-06T07:24:21.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.774+0000 7fccb37fe640 1 -- 172.21.15.1:0/1885083206 learned_addr learned my addr 172.21.15.1:0/1885083206 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:21.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.774+0000 7fccb37fe640 1 -- 172.21.15.1:0/1885083206 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a6c330 msgr2=0x564951a6a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.774+0000 7fccb37fe640 1 --2- 172.21.15.1:0/1885083206 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a6c330 0x564951a6a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.774+0000 7fccb37fe640 1 -- 172.21.15.1:0/1885083206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a908c0 msgr2=0x564951ab5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.774+0000 7fccb37fe640 1 --2- 172.21.15.1:0/1885083206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a908c0 0x564951ab5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.775+0000 7fccb37fe640 1 -- 172.21.15.1:0/1885083206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649517af6b0 con 0x564951a6bf50 2024-03-06T07:24:21.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.775+0000 7fccb37fe640 1 --2- 172.21.15.1:0/1885083206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6bf50 0x564951aa5790 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fcca4004560 tx=0x7fcca4047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3853cf7b2ccdac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.775+0000 7fcc98ff9640 1 -- 172.21.15.1:0/1885083206 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca404e020 con 0x564951a6bf50 2024-03-06T07:24:21.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.775+0000 7fcc98ff9640 1 -- 172.21.15.1:0/1885083206 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca403a040 con 0x564951a6bf50 2024-03-06T07:24:21.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.775+0000 7fcc98ff9640 1 -- 172.21.15.1:0/1885083206 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca4043040 con 0x564951a6bf50 2024-03-06T07:24:21.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.775+0000 7fccbcd9d740 1 -- 172.21.15.1:0/1885083206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6bf50 msgr2=0x564951aa5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.775+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/1885083206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6bf50 0x564951aa5790 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fcca4004560 tx=0x7fcca4047000 comp rx=0 tx=0).stop 2024-03-06T07:24:21.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.775+0000 7fccbcd9d740 1 -- 172.21.15.1:0/1885083206 shutdown_connections 2024-03-06T07:24:21.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.775+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/1885083206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a908c0 0x564951ab5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.775+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/1885083206 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a6c330 0x564951a6a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.775+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/1885083206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6bf50 0x564951aa5790 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.775+0000 7fccbcd9d740 1 -- 172.21.15.1:0/1885083206 >> 172.21.15.1:0/1885083206 conn(0x564951a6a4b0 msgr2=0x564951a65510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.775+0000 7fccbcd9d740 1 -- 172.21.15.1:0/1885083206 shutdown_connections 2024-03-06T07:24:21.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.776+0000 7fccbcd9d740 1 -- 172.21.15.1:0/1885083206 wait complete. 2024-03-06T07:24:21.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.776+0000 7fccbcd9d740 1 -- start start 2024-03-06T07:24:21.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.776+0000 7fccbcd9d740 1 Processor -- start 2024-03-06T07:24:21.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccbcd9d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a6bf50 0x564951a6ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccbcd9d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6c330 0x564951a8c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccbcd9d740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a908c0 0x564951b1b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccbcd9d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x564951ab8cf0 con 0x564951a6c330 2024-03-06T07:24:21.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccbcd9d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x564951a8fab0 con 0x564951a6bf50 2024-03-06T07:24:21.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccbcd9d740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x564951aac170 con 0x564951a908c0 2024-03-06T07:24:21.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccbcd9d740 10 client.0 _ll_register_callbacks cb 0x7fcca8009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:21.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccb2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6c330 0x564951a8c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccb2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6c330 0x564951a8c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44544/0 (socket says 172.21.15.1:44544) 2024-03-06T07:24:21.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccb2ffd640 1 -- 172.21.15.1:0/4224659181 learned_addr learned my addr 172.21.15.1:0/4224659181 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:21.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccb37fe640 1 --2- 172.21.15.1:0/4224659181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a6bf50 0x564951a6ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccb3fff640 1 --2- 172.21.15.1:0/4224659181 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a908c0 0x564951b1b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccb2ffd640 1 -- 172.21.15.1:0/4224659181 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a908c0 msgr2=0x564951b1b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccb2ffd640 1 --2- 172.21.15.1:0/4224659181 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a908c0 0x564951b1b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccb2ffd640 1 -- 172.21.15.1:0/4224659181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a6bf50 msgr2=0x564951a6ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccb2ffd640 1 --2- 172.21.15.1:0/4224659181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a6bf50 0x564951a6ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccb2ffd640 1 -- 172.21.15.1:0/4224659181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564951aa4370 con 0x564951a6c330 2024-03-06T07:24:21.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccb37fe640 1 --2- 172.21.15.1:0/4224659181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a6bf50 0x564951a6ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:21.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fccb2ffd640 1 --2- 172.21.15.1:0/4224659181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6c330 0x564951a8c5b0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fcca800f6d0 tx=0x7fcca8059810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f32ecd953ed8e17 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fcc927fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:24:21.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.777+0000 7fcc927fc640 1 -- 172.21.15.1:0/4224659181 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8075060 con 0x564951a6c330 2024-03-06T07:24:21.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.778+0000 7fcc927fc640 1 -- 172.21.15.1:0/4224659181 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcca8055780 con 0x564951a6c330 2024-03-06T07:24:21.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.778+0000 7fccbcd9d740 1 -- 172.21.15.1:0/4224659181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcca403a040 con 0x564951a6c330 2024-03-06T07:24:21.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.778+0000 7fcc927fc640 1 -- 172.21.15.1:0/4224659181 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcca805d260 con 0x564951a6c330 2024-03-06T07:24:21.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.778+0000 7fccbcd9d740 10 client.7584 fetch_fsmap learned FSMap version 88 2024-03-06T07:24:21.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.778+0000 7fccbcd9d740 1 -- 172.21.15.1:0/4224659181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) v3 -- 0x564951a8bb80 con 0x564951a6c330 2024-03-06T07:24:21.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.778+0000 7fcc927fc640 1 -- 172.21.15.1:0/4224659181 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca403a040 con 0x564951a6c330 2024-03-06T07:24:21.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.778+0000 7fcc927fc640 1 -- 172.21.15.1:0/4224659181 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 88) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fcca8025030 con 0x564951a6c330 2024-03-06T07:24:21.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.778+0000 7fccbcd9d740 10 client.7584 fetch_fsmap finished waiting for FSMap version 88 2024-03-06T07:24:21.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.778+0000 7fccbcd9d740 10 client.7584 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:24:21.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.778+0000 7fccbcd9d740 20 client.7584 populate_metadata read hostname 'smithi001' 2024-03-06T07:24:21.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.778+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/4224659181 >> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] conn(0x564951b15150 0x564951b15530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.778+0000 7fccbcd9d740 4 client.7584 mds_command: new command op to 7362 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:24:21.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.778+0000 7fccbcd9d740 1 -- 172.21.15.1:0/4224659181 --> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5649519d7f40 con 0x564951b15150 2024-03-06T07:24:21.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.778+0000 7fccb37fe640 1 --2- 172.21.15.1:0/4224659181 >> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] conn(0x564951b15150 0x564951b15530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.778+0000 7fcc927fc640 10 client.7584 ms_handle_connect on v2:172.21.15.1:6832/2433923518 2024-03-06T07:24:21.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.778+0000 7fccb37fe640 1 --2- 172.21.15.1:0/4224659181 >> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] conn(0x564951b15150 0x564951b15530 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=480e282c12a877e server_cookie=f3902da32f0b6616 in_seq=0 out_seq=0 2024-03-06T07:24:21.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fcc927fc640 1 -- 172.21.15.1:0/4224659181 <== mds.0 v2:172.21.15.1:6832/2433923518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5649519d7f40 con 0x564951b15150 2024-03-06T07:24:21.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fcc927fc640 10 client.7584 handle_command_reply: tid=0 2024-03-06T07:24:21.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccbcd9d740 1 client.7584 shutdown 2024-03-06T07:24:21.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccbcd9d740 1 -- 172.21.15.1:0/4224659181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6c330 msgr2=0x564951a8c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/4224659181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6c330 0x564951a8c5b0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fcca800f6d0 tx=0x7fcca8059810 comp rx=0 tx=0).stop 2024-03-06T07:24:21.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccbcd9d740 1 -- 172.21.15.1:0/4224659181 shutdown_connections 2024-03-06T07:24:21.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/4224659181 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x564951a908c0 0x564951b1b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/4224659181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564951a6c330 0x564951a8c5b0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/4224659181 >> [v2:172.21.15.1:6832/2433923518,v1:172.21.15.1:6833/2433923518] conn(0x564951b15150 0x564951b15530 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccbcd9d740 1 --2- 172.21.15.1:0/4224659181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564951a6bf50 0x564951a6ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccbcd9d740 1 -- 172.21.15.1:0/4224659181 >> 172.21.15.1:0/4224659181 conn(0x564951a6a4b0 msgr2=0x564951aaadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fcc927fc640 0 client.7584 ms_handle_reset on v2:172.21.15.1:6832/2433923518 2024-03-06T07:24:21.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccbcd9d740 1 -- 172.21.15.1:0/4224659181 shutdown_connections 2024-03-06T07:24:21.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccbcd9d740 1 -- 172.21.15.1:0/4224659181 wait complete. 2024-03-06T07:24:21.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccbcd9d740 20 client.7584 trim_cache size 0 max 16384 2024-03-06T07:24:21.788 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:24:21.788 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:24:21.789 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:24:21.789 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 7362, 2024-03-06T07:24:21.789 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:24:21.789 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:24:21.789 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:24:21.789 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 10.924035698999999, 2024-03-06T07:24:21.789 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 88, 2024-03-06T07:24:21.789 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 108, 2024-03-06T07:24:21.789 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 108, 2024-03-06T07:24:21.789 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 15.997285882 2024-03-06T07:24:21.789 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:24:21.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccba0e0640 1 -- 172.21.15.1:0/1480001286 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcca006b4d0 msgr2=0x7fcca006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccba0e0640 1 --2- 172.21.15.1:0/1480001286 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcca006b4d0 0x7fcca006d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fcca4036000 tx=0x7fcca4030000 comp rx=0 tx=0).stop 2024-03-06T07:24:21.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccba0e0640 1 -- 172.21.15.1:0/1480001286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb4156c50 msgr2=0x7fccb41eafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccba0e0640 1 --2- 172.21.15.1:0/1480001286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb4156c50 0x7fccb41eafb0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fcca8011a70 tx=0x7fcca8011f30 comp rx=0 tx=0).stop 2024-03-06T07:24:21.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccba0e0640 1 -- 172.21.15.1:0/1480001286 shutdown_connections 2024-03-06T07:24:21.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccba0e0640 1 --2- 172.21.15.1:0/1480001286 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcca006b4d0 0x7fcca006d990 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccba0e0640 1 --2- 172.21.15.1:0/1480001286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccb41eb4f0 0x7fccb406f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccba0e0640 1 --2- 172.21.15.1:0/1480001286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccb4156c50 0x7fccb41eafb0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccba0e0640 1 --2- 172.21.15.1:0/1480001286 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc9c0039b0 0x7fccb40a97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.779+0000 7fccba0e0640 1 -- 172.21.15.1:0/1480001286 >> 172.21.15.1:0/1480001286 conn(0x7fccb40c88f0 msgr2=0x7fccb4150a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.780+0000 7fccba0e0640 1 -- 172.21.15.1:0/1480001286 shutdown_connections 2024-03-06T07:24:21.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:21.780+0000 7fccba0e0640 1 -- 172.21.15.1:0/1480001286 wait complete. 2024-03-06T07:24:21.792 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 88, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:24:10.220198+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 88, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7362": { "addr": "172.21.15.1:6833/2433923518", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 2433923518, "type": "v2" }, { "addr": "172.21.15.1:6833", "nonce": 2433923518, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7362, "incarnation": 86, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-03-06T07:24:12.224326+0000", "qdb_cluster": [ 7362 ], "qdb_leader": 7362, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7362 } } } ], "standbys": [ { "addr": "172.21.15.98:6837/2632566151", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 2632566151, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2632566151, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6122, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6835/2244230066", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6834", "nonce": 2244230066, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 2244230066, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6670, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6836/1741783790", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 1741783790, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 1741783790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6682, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:24:21.793 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 8, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:24:21.793 INFO:tasks.cephfs.test_failover:current = [{'gid': 6670, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/2244230066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 2244230066}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2244230066}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}, {'gid': 6122, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/2632566151', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 2632566151}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2632566151}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}, {'gid': 7362, 'name': 'c', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6833/2433923518', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2433923518}, {'type': 'v1', 'addr': '172.21.15.1:6833', 'nonce': 2433923518}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6682, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/1741783790', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 1741783790}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1741783790}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}] 2024-03-06T07:24:21.794 DEBUG:tasks.cephfs.test_failover:8 != -1 2024-03-06T07:24:21.794 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:24:21.908 INFO:tasks.ceph.mds.c.smithi001.stderr: -264> 2024-03-06T07:24:16.142+0000 7ffbf309b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:21.933 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:24:26.145 INFO:tasks.ceph.mds.b.smithi098.stderr:2024-03-06T07:24:26.143+0000 7f9608b23640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:26.795 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.909+0000 7f5c2561c640 1 Processor -- start 2024-03-06T07:24:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.909+0000 7f5c2561c640 1 -- start start 2024-03-06T07:24:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.910+0000 7f5c2561c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c20152650 0x7f5c20155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.910+0000 7f5c2561c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c200bf9d0 0x7f5c200c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.910+0000 7f5c2561c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c200c2fc0 con 0x7f5c200c6b10 2024-03-06T07:24:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.910+0000 7f5c2561c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c20155800 con 0x7f5c20152650 2024-03-06T07:24:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.910+0000 7f5c2561c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c20158da0 con 0x7f5c200bf9d0 2024-03-06T07:24:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.910+0000 7f5c1effd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c20152650 0x7f5c20155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.910+0000 7f5c1e7fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5c200c6b10 0x7f5c20158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53812/0 (socket says 172.21.15.1:53812) 2024-03-06T07:24:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.910+0000 7f5c1e7fc640 1 -- 172.21.15.1:0/2950399653 learned_addr learned my addr 172.21.15.1:0/2950399653 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.910+0000 7f5c1f7fe640 1 --2- 172.21.15.1:0/2950399653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c200bf9d0 0x7f5c200c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.911+0000 7f5c1dffb640 1 -- 172.21.15.1:0/2950399653 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297391372 0 0) 0x7f5c200c2fc0 con 0x7f5c200c6b10 2024-03-06T07:24:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.911+0000 7f5c1effd640 1 -- 172.21.15.1:0/2950399653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c200bf9d0 msgr2=0x7f5c200c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.911+0000 7f5c1effd640 1 --2- 172.21.15.1:0/2950399653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c200bf9d0 0x7f5c200c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.911+0000 7f5c1effd640 1 -- 172.21.15.1:0/2950399653 >> v1:172.21.15.1:6789/0 conn(0x7f5c200c6b10 legacy=0x7f5c20158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.911+0000 7f5c1effd640 1 -- 172.21.15.1:0/2950399653 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c200c6690 con 0x7f5c20152650 2024-03-06T07:24:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.911+0000 7f5c1effd640 1 --2- 172.21.15.1:0/2950399653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c20152650 0x7f5c20155c90 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f5c180097a0 tx=0x7f5c1802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf9c996be600c2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.911+0000 7f5c1dffb640 1 -- 172.21.15.1:0/2950399653 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c180315a0 con 0x7f5c20152650 2024-03-06T07:24:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.912+0000 7f5c1dffb640 1 -- 172.21.15.1:0/2950399653 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c18037070 con 0x7f5c20152650 2024-03-06T07:24:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.912+0000 7f5c1dffb640 1 -- 172.21.15.1:0/2950399653 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c18031960 con 0x7f5c20152650 2024-03-06T07:24:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.912+0000 7f5c2561c640 1 -- 172.21.15.1:0/2950399653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c20152650 msgr2=0x7f5c20155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.912+0000 7f5c2561c640 1 --2- 172.21.15.1:0/2950399653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c20152650 0x7f5c20155c90 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f5c180097a0 tx=0x7f5c1802b840 comp rx=0 tx=0).stop 2024-03-06T07:24:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.912+0000 7f5c2561c640 1 -- 172.21.15.1:0/2950399653 shutdown_connections 2024-03-06T07:24:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.912+0000 7f5c2561c640 1 --2- 172.21.15.1:0/2950399653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c200bf9d0 0x7f5c200c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.912+0000 7f5c2561c640 1 --2- 172.21.15.1:0/2950399653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c20152650 0x7f5c20155c90 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.912+0000 7f5c2561c640 1 -- 172.21.15.1:0/2950399653 >> 172.21.15.1:0/2950399653 conn(0x7f5c20066ee0 msgr2=0x7f5c200672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.912+0000 7f5c2561c640 1 -- 172.21.15.1:0/2950399653 shutdown_connections 2024-03-06T07:24:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.912+0000 7f5c2561c640 1 -- 172.21.15.1:0/2950399653 wait complete. 2024-03-06T07:24:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.913+0000 7f5c2561c640 1 Processor -- start 2024-03-06T07:24:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.913+0000 7f5c2561c640 1 -- start start 2024-03-06T07:24:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.914+0000 7f5c2561c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c200bf9d0 0x7f5c201edfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.914+0000 7f5c2561c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c20152650 0x7f5c201e2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.914+0000 7f5c2561c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c200c9bd0 0x7f5c201578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.914+0000 7f5c2561c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c201551b0 con 0x7f5c20152650 2024-03-06T07:24:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.914+0000 7f5c2561c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c200c6fa0 con 0x7f5c200c9bd0 2024-03-06T07:24:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.914+0000 7f5c2561c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c20155490 con 0x7f5c200bf9d0 2024-03-06T07:24:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.914+0000 7f5c1e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c20152650 0x7f5c201e2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.914+0000 7f5c1e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c20152650 0x7f5c201e2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35220/0 (socket says 172.21.15.1:35220) 2024-03-06T07:24:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.914+0000 7f5c1e7fc640 1 -- 172.21.15.1:0/2453206446 learned_addr learned my addr 172.21.15.1:0/2453206446 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.914+0000 7f5c1f7fe640 1 --2- 172.21.15.1:0/2453206446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c200c9bd0 0x7f5c201578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.914+0000 7f5c1effd640 1 --2- 172.21.15.1:0/2453206446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c200bf9d0 0x7f5c201edfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.915+0000 7f5c1e7fc640 1 -- 172.21.15.1:0/2453206446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c200bf9d0 msgr2=0x7f5c201edfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.915+0000 7f5c1e7fc640 1 --2- 172.21.15.1:0/2453206446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c200bf9d0 0x7f5c201edfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.915+0000 7f5c1e7fc640 1 -- 172.21.15.1:0/2453206446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c200c9bd0 msgr2=0x7f5c201578f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.915+0000 7f5c1e7fc640 1 --2- 172.21.15.1:0/2453206446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c200c9bd0 0x7f5c201578f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.915+0000 7f5c1e7fc640 1 -- 172.21.15.1:0/2453206446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c20154400 con 0x7f5c20152650 2024-03-06T07:24:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.915+0000 7f5c1e7fc640 1 --2- 172.21.15.1:0/2453206446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c20152650 0x7f5c201e2120 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800c4d0 tx=0x7f5c0800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.915+0000 7f5c177fe640 1 -- 172.21.15.1:0/2453206446 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c08017070 con 0x7f5c20152650 2024-03-06T07:24:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.915+0000 7f5c177fe640 1 -- 172.21.15.1:0/2453206446 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c08005d00 con 0x7f5c20152650 2024-03-06T07:24:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.915+0000 7f5c2561c640 1 -- 172.21.15.1:0/2453206446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c20158590 con 0x7f5c20152650 2024-03-06T07:24:26.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.915+0000 7f5c177fe640 1 -- 172.21.15.1:0/2453206446 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c080072b0 con 0x7f5c20152650 2024-03-06T07:24:26.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.915+0000 7f5c2561c640 1 -- 172.21.15.1:0/2453206446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c201e2660 con 0x7f5c20152650 2024-03-06T07:24:26.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.916+0000 7f5c2561c640 1 -- 172.21.15.1:0/2453206446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c20067fd0 con 0x7f5c20152650 2024-03-06T07:24:26.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.917+0000 7f5c177fe640 1 -- 172.21.15.1:0/2453206446 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c0801a070 con 0x7f5c20152650 2024-03-06T07:24:26.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.917+0000 7f5c177fe640 1 --2- 172.21.15.1:0/2453206446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0006b4d0 0x7f5c0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:26.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.917+0000 7f5c177fe640 1 -- 172.21.15.1:0/2453206446 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5c08089640 con 0x7f5c20152650 2024-03-06T07:24:26.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.917+0000 7f5c1effd640 1 --2- 172.21.15.1:0/2453206446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0006b4d0 0x7f5c0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:26.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.918+0000 7f5c1effd640 1 --2- 172.21.15.1:0/2453206446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0006b4d0 0x7f5c0006d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5c18036000 tx=0x7f5c18008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:26.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:26.922+0000 7f5c177fe640 1 -- 172.21.15.1:0/2453206446 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c08057d20 con 0x7f5c20152650 2024-03-06T07:24:27.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.139+0000 7f5c2561c640 1 -- 172.21.15.1:0/2453206446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c200b01a0 con 0x7f5c20152650 2024-03-06T07:24:27.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.139+0000 7f5c177fe640 1 -- 172.21.15.1:0/2453206446 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f5c0805bdc0 con 0x7f5c20152650 2024-03-06T07:24:27.142 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:27.142 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":93,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2244230066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2244230066},{"type":"v1","addr":"172.21.15.1:6835","nonce":2244230066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":6682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1741783790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1741783790},{"type":"v1","addr":"172.21.15.98:6836","nonce":1741783790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7590,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/550216328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":550216328},{"type":"v1","addr":"172.21.15.1:6833","nonce":550216328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:10.220198+0000","modified":"2024-03-06T07:24:24.911657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6122":{"gid":6122,"name":"b","rank":0,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6837/2632566151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2632566151},{"type":"v1","addr":"172.21.15.98:6837","nonce":2632566151}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6122,"qdb_cluster":[6122]},"id":8}]} 2024-03-06T07:24:27.143 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 93 2024-03-06T07:24:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.141+0000 7f5c2561c640 1 -- 172.21.15.1:0/2453206446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0006b4d0 msgr2=0x7f5c0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.141+0000 7f5c2561c640 1 --2- 172.21.15.1:0/2453206446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0006b4d0 0x7f5c0006d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5c18036000 tx=0x7f5c18008030 comp rx=0 tx=0).stop 2024-03-06T07:24:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.141+0000 7f5c2561c640 1 -- 172.21.15.1:0/2453206446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c20152650 msgr2=0x7f5c201e2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.141+0000 7f5c2561c640 1 --2- 172.21.15.1:0/2453206446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c20152650 0x7f5c201e2120 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800c4d0 tx=0x7f5c0800c990 comp rx=0 tx=0).stop 2024-03-06T07:24:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.141+0000 7f5c2561c640 1 -- 172.21.15.1:0/2453206446 shutdown_connections 2024-03-06T07:24:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.141+0000 7f5c2561c640 1 --2- 172.21.15.1:0/2453206446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0006b4d0 0x7f5c0006d990 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.141+0000 7f5c2561c640 1 --2- 172.21.15.1:0/2453206446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c200c9bd0 0x7f5c201578f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.141+0000 7f5c2561c640 1 --2- 172.21.15.1:0/2453206446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c20152650 0x7f5c201e2120 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.141+0000 7f5c2561c640 1 --2- 172.21.15.1:0/2453206446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c200bf9d0 0x7f5c201edfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.141+0000 7f5c2561c640 1 -- 172.21.15.1:0/2453206446 >> 172.21.15.1:0/2453206446 conn(0x7f5c20066ee0 msgr2=0x7f5c200cae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.141+0000 7f5c2561c640 1 -- 172.21.15.1:0/2453206446 shutdown_connections 2024-03-06T07:24:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.141+0000 7f5c2561c640 1 -- 172.21.15.1:0/2453206446 wait complete. 2024-03-06T07:24:27.557 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 93, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:10.220198+0000', 'modified': '2024-03-06T07:24:24.911657+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 109, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6122}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6122': {'gid': 6122, 'name': 'b', 'rank': 0, 'incarnation': 90, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.98:6837/2632566151', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 2632566151}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2632566151}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6122, 'qdb_cluster': [6122]} 2024-03-06T07:24:27.557 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:24:27.557 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:24:27.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.660+0000 7f2fd1402640 1 Processor -- start 2024-03-06T07:24:27.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.660+0000 7f2fd1402640 1 -- start start 2024-03-06T07:24:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.661+0000 7f2fd1402640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2fcc14f410 0x7f2fcc1527a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.661+0000 7f2fd1402640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2fcc1555a0 0x7f2fcc1559d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.661+0000 7f2fd1402640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc06ffd0 con 0x7f2fcc070270 2024-03-06T07:24:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.661+0000 7f2fd1402640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fcc152ce0 con 0x7f2fcc14f410 2024-03-06T07:24:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.661+0000 7f2fd1402640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fcc14f7f0 con 0x7f2fcc1555a0 2024-03-06T07:24:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.661+0000 7f2fcaffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2fcc14f410 0x7f2fcc1527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.661+0000 7f2fcaffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2fcc14f410 0x7f2fcc1527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35226/0 (socket says 172.21.15.1:35226) 2024-03-06T07:24:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.661+0000 7f2fcaffd640 1 -- 172.21.15.1:0/3708924979 learned_addr learned my addr 172.21.15.1:0/3708924979 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.661+0000 7f2fcb7fe640 1 --2- 172.21.15.1:0/3708924979 >> v2:172.21.15.98:3300/0 conn(0x7f2fcc1555a0 0x7f2fcc1559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.661+0000 7f2fcaffd640 1 -- 172.21.15.1:0/3708924979 >> v2:172.21.15.98:3300/0 conn(0x7f2fcc1555a0 msgr2=0x7f2fcc1559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.661+0000 7f2fcaffd640 1 --2- 172.21.15.1:0/3708924979 >> v2:172.21.15.98:3300/0 conn(0x7f2fcc1555a0 0x7f2fcc1559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.661+0000 7f2fcaffd640 1 -- 172.21.15.1:0/3708924979 >> v1:172.21.15.98:6789/0 conn(0x7f2fcc070270 legacy=0x7f2fcc06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.661+0000 7f2fcaffd640 1 -- 172.21.15.1:0/3708924979 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc06f450 con 0x7f2fcc14f410 2024-03-06T07:24:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.662+0000 7f2fcaffd640 1 --2- 172.21.15.1:0/3708924979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc14f410 0x7f2fcc1527a0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0c2090 tx=0x7f2fc402e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e6c3cddae7099a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.662+0000 7f2fc9ffb640 1 -- 172.21.15.1:0/3708924979 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc402ea00 con 0x7f2fcc14f410 2024-03-06T07:24:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.662+0000 7f2fc9ffb640 1 -- 172.21.15.1:0/3708924979 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc402eba0 con 0x7f2fcc14f410 2024-03-06T07:24:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.662+0000 7f2fc9ffb640 1 -- 172.21.15.1:0/3708924979 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4009600 con 0x7f2fcc14f410 2024-03-06T07:24:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.663+0000 7f2fd1402640 1 -- 172.21.15.1:0/3708924979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc14f410 msgr2=0x7f2fcc1527a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.663+0000 7f2fd1402640 1 --2- 172.21.15.1:0/3708924979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc14f410 0x7f2fcc1527a0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0c2090 tx=0x7f2fc402e060 comp rx=0 tx=0).stop 2024-03-06T07:24:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.663+0000 7f2fd1402640 1 -- 172.21.15.1:0/3708924979 shutdown_connections 2024-03-06T07:24:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.663+0000 7f2fd1402640 1 --2- 172.21.15.1:0/3708924979 >> v2:172.21.15.98:3300/0 conn(0x7f2fcc1555a0 0x7f2fcc1559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.663+0000 7f2fd1402640 1 --2- 172.21.15.1:0/3708924979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc14f410 0x7f2fcc1527a0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.663+0000 7f2fd1402640 1 -- 172.21.15.1:0/3708924979 >> 172.21.15.1:0/3708924979 conn(0x7f2fcc0a95b0 msgr2=0x7f2fcc0a99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.663+0000 7f2fd1402640 1 -- 172.21.15.1:0/3708924979 shutdown_connections 2024-03-06T07:24:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.663+0000 7f2fd1402640 1 -- 172.21.15.1:0/3708924979 wait complete. 2024-03-06T07:24:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.664+0000 7f2fd1402640 1 Processor -- start 2024-03-06T07:24:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.664+0000 7f2fd1402640 1 -- start start 2024-03-06T07:24:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fd1402640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fcc070270 0x7f2fcc0bc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fd1402640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc14f410 0x7f2fcc150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fd1402640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fcc1555a0 0x7f2fcc14adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fd1402640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc06fcf0 con 0x7f2fcc14f410 2024-03-06T07:24:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fd1402640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc06fa10 con 0x7f2fcc070270 2024-03-06T07:24:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fd1402640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc06c810 con 0x7f2fcc1555a0 2024-03-06T07:24:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fca7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc14f410 0x7f2fcc150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fca7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc14f410 0x7f2fcc150e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35242/0 (socket says 172.21.15.1:35242) 2024-03-06T07:24:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fca7fc640 1 -- 172.21.15.1:0/950998754 learned_addr learned my addr 172.21.15.1:0/950998754 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fcaffd640 1 --2- 172.21.15.1:0/950998754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fcc070270 0x7f2fcc0bc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fcb7fe640 1 --2- 172.21.15.1:0/950998754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fcc1555a0 0x7f2fcc14adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fca7fc640 1 -- 172.21.15.1:0/950998754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fcc1555a0 msgr2=0x7f2fcc14adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fca7fc640 1 --2- 172.21.15.1:0/950998754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fcc1555a0 0x7f2fcc14adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fca7fc640 1 -- 172.21.15.1:0/950998754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fcc070270 msgr2=0x7f2fcc0bc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fca7fc640 1 --2- 172.21.15.1:0/950998754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fcc070270 0x7f2fcc0bc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fca7fc640 1 -- 172.21.15.1:0/950998754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc1e6f70 con 0x7f2fcc14f410 2024-03-06T07:24:27.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fcb7fe640 1 --2- 172.21.15.1:0/950998754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fcc1555a0 0x7f2fcc14adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:27.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fca7fc640 1 --2- 172.21.15.1:0/950998754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc14f410 0x7f2fcc150e80 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f2fb400c440 tx=0x7f2fb4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fc37fe640 1 -- 172.21.15.1:0/950998754 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb40052d0 con 0x7f2fcc14f410 2024-03-06T07:24:27.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fc37fe640 1 -- 172.21.15.1:0/950998754 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb4005470 con 0x7f2fcc14f410 2024-03-06T07:24:27.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fd1402640 1 -- 172.21.15.1:0/950998754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc14a2e0 con 0x7f2fcc14f410 2024-03-06T07:24:27.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fc37fe640 1 -- 172.21.15.1:0/950998754 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4007260 con 0x7f2fcc14f410 2024-03-06T07:24:27.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.665+0000 7f2fd1402640 1 -- 172.21.15.1:0/950998754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc14a8a0 con 0x7f2fcc14f410 2024-03-06T07:24:27.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.667+0000 7f2fc37fe640 1 -- 172.21.15.1:0/950998754 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fb4007400 con 0x7f2fcc14f410 2024-03-06T07:24:27.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.667+0000 7f2fc37fe640 1 --2- 172.21.15.1:0/950998754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fac06b500 0x7f2fac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.667+0000 7f2fc37fe640 1 -- 172.21.15.1:0/950998754 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2fb4089790 con 0x7f2fcc14f410 2024-03-06T07:24:27.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.667+0000 7f2fcaffd640 1 --2- 172.21.15.1:0/950998754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fac06b500 0x7f2fac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.668+0000 7f2fcaffd640 1 --2- 172.21.15.1:0/950998754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fac06b500 0x7f2fac06d9c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4037000 tx=0x7f2fc4032000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.668+0000 7f2fd3fe2740 1 Processor -- start 2024-03-06T07:24:27.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.668+0000 7f2fd3fe2740 1 -- start start 2024-03-06T07:24:27.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.669+0000 7f2fd3fe2740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882db6100 0x555882db3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.669+0000 7f2fd3fe2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882db64e0 0x555882db4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.669+0000 7f2fd3fe2740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db4660 0x555882db0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.669+0000 7f2fd3fe2740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x555882d1a6b0 con 0x555882db4660 2024-03-06T07:24:27.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.669+0000 7f2fd3fe2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x555882cc3a60 con 0x555882db64e0 2024-03-06T07:24:27.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.669+0000 7f2fd3fe2740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x555882d16750 con 0x555882db6100 2024-03-06T07:24:27.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.669+0000 7f2fcb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db4660 0x555882db0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.669+0000 7f2fcb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db4660 0x555882db0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35248/0 (socket says 172.21.15.1:35248) 2024-03-06T07:24:27.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.669+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/619608714 learned_addr learned my addr 172.21.15.1:0/619608714 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:27.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.669+0000 7f2fcaffd640 1 --2- 172.21.15.1:0/619608714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882db6100 0x555882db3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.669+0000 7f2fca7fc640 1 --2- 172.21.15.1:0/619608714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882db64e0 0x555882db4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.669+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/619608714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882db6100 msgr2=0x555882db3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:27.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.669+0000 7f2fcb7fe640 1 --2- 172.21.15.1:0/619608714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882db6100 0x555882db3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.669+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/619608714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882db64e0 msgr2=0x555882db4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:27.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.669+0000 7f2fcb7fe640 1 --2- 172.21.15.1:0/619608714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882db64e0 0x555882db4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.669+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/619608714 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555882ce1f30 con 0x555882db4660 2024-03-06T07:24:27.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.670+0000 7f2fcb7fe640 1 --2- 172.21.15.1:0/619608714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db4660 0x555882db0830 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc009b40 tx=0x7f2fbc009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85c01187727fbc68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.670+0000 7f2fc0ff9640 1 -- 172.21.15.1:0/619608714 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc019070 con 0x555882db4660 2024-03-06T07:24:27.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.670+0000 7f2fc0ff9640 1 -- 172.21.15.1:0/619608714 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc011c20 con 0x555882db4660 2024-03-06T07:24:27.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.670+0000 7f2fc0ff9640 1 -- 172.21.15.1:0/619608714 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc015460 con 0x555882db4660 2024-03-06T07:24:27.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.670+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/619608714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db4660 msgr2=0x555882db0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.670+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/619608714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db4660 0x555882db0830 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc009b40 tx=0x7f2fbc009190 comp rx=0 tx=0).stop 2024-03-06T07:24:27.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.670+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/619608714 shutdown_connections 2024-03-06T07:24:27.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.670+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/619608714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db4660 0x555882db0830 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.670+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/619608714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882db64e0 0x555882db4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.670+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/619608714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882db6100 0x555882db3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.670+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/619608714 >> 172.21.15.1:0/619608714 conn(0x555882db2980 msgr2=0x555882dd7570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:27.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.671+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/619608714 shutdown_connections 2024-03-06T07:24:27.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.671+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/619608714 wait complete. 2024-03-06T07:24:27.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.671+0000 7f2fd3fe2740 1 -- start start 2024-03-06T07:24:27.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.671+0000 7f2fd3fe2740 1 Processor -- start 2024-03-06T07:24:27.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fd3fe2740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db6100 0x555882ddcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fd3fe2740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882db64e0 0x555882dd9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fd3fe2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882ddaf20 0x555882ddb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fd3fe2740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x555882dd83e0 con 0x555882db6100 2024-03-06T07:24:27.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fd3fe2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x555882cd6ea0 con 0x555882ddaf20 2024-03-06T07:24:27.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fd3fe2740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x555882dd7e20 con 0x555882db64e0 2024-03-06T07:24:27.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fd3fe2740 10 client.0 _ll_register_callbacks cb 0x555882cd6870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:27.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fcaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db6100 0x555882ddcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fcb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882ddaf20 0x555882ddb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fca7fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882db64e0 0x555882dd9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fcaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db6100 0x555882ddcdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35264/0 (socket says 172.21.15.1:35264) 2024-03-06T07:24:27.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fcaffd640 1 -- 172.21.15.1:0/1650883179 learned_addr learned my addr 172.21.15.1:0/1650883179 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:27.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/1650883179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882db64e0 msgr2=0x555882dd9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:27.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fcb7fe640 1 --2- 172.21.15.1:0/1650883179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882db64e0 0x555882dd9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/1650883179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db6100 msgr2=0x555882ddcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:27.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fcb7fe640 1 --2- 172.21.15.1:0/1650883179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db6100 0x555882ddcdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/1650883179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555882cd7270 con 0x555882ddaf20 2024-03-06T07:24:27.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fcaffd640 1 --2- 172.21.15.1:0/1650883179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db6100 0x555882ddcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:27.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fcb7fe640 1 --2- 172.21.15.1:0/1650883179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882ddaf20 0x555882ddb3a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc01b000 tx=0x7f2fbc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=789400ae9726d90 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.672+0000 7f2fa9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-06T07:24:27.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.673+0000 7f2fa9ffb640 1 -- 172.21.15.1:0/1650883179 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc019040 con 0x555882ddaf20 2024-03-06T07:24:27.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.673+0000 7f2fa9ffb640 1 -- 172.21.15.1:0/1650883179 <== mon.1 v2:172.21.15.98:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2fbc011a20 con 0x555882ddaf20 2024-03-06T07:24:27.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.673+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/1650883179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2fbc011c20 con 0x555882ddaf20 2024-03-06T07:24:27.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.673+0000 7f2fa9ffb640 1 -- 172.21.15.1:0/1650883179 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fbc033050 con 0x555882ddaf20 2024-03-06T07:24:27.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.673+0000 7f2fd3fe2740 10 client.6844 fetch_fsmap learned FSMap version 93 2024-03-06T07:24:27.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.673+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/1650883179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=2+}) v3 -- 0x555882a2f2d0 con 0x555882ddaf20 2024-03-06T07:24:27.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.673+0000 7f2fa9ffb640 1 -- 172.21.15.1:0/1650883179 <== mon.1 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc011c20 con 0x555882ddaf20 2024-03-06T07:24:27.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.673+0000 7f2fa9ffb640 1 -- 172.21.15.1:0/1650883179 <== mon.1 v2:172.21.15.98:3300/0 5 ==== fsmap(e 93) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2fbc036020 con 0x555882ddaf20 2024-03-06T07:24:27.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.673+0000 7f2fd3fe2740 10 client.6844 fetch_fsmap finished waiting for FSMap version 93 2024-03-06T07:24:27.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.673+0000 7f2fd3fe2740 10 client.6844 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:24:27.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.673+0000 7f2fd3fe2740 20 client.6844 populate_metadata read hostname 'smithi001' 2024-03-06T07:24:27.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.673+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/1650883179 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x555882dd8e00 0x555882dd91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.673+0000 7f2fd3fe2740 4 client.6844 mds_command: new command op to 6122 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:24:27.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.673+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/1650883179 --> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555882d27bf0 con 0x555882dd8e00 2024-03-06T07:24:27.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.673+0000 7f2fcaffd640 1 --2- 172.21.15.1:0/1650883179 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x555882dd8e00 0x555882dd91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.674+0000 7f2fa9ffb640 10 client.6844 ms_handle_connect on v2:172.21.15.98:6835/2632566151 2024-03-06T07:24:27.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.674+0000 7f2fcaffd640 1 --2- 172.21.15.1:0/1650883179 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x555882dd8e00 0x555882dd91e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65c02b1d3f9dfdff server_cookie=1b9cf1f94b34107 in_seq=0 out_seq=0 2024-03-06T07:24:27.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.674+0000 7f2fa9ffb640 1 -- 172.21.15.1:0/1650883179 <== mds.0 v2:172.21.15.98:6835/2632566151 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555882d27bf0 con 0x555882dd8e00 2024-03-06T07:24:27.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.674+0000 7f2fa9ffb640 10 client.6844 handle_command_reply: tid=0 2024-03-06T07:24:27.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.674+0000 7f2fd3fe2740 1 client.6844 shutdown 2024-03-06T07:24:27.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.674+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/1650883179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882ddaf20 msgr2=0x555882ddb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.674+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/1650883179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882ddaf20 0x555882ddb3a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc01b000 tx=0x7f2fbc004730 comp rx=0 tx=0).stop 2024-03-06T07:24:27.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.674+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/1650883179 shutdown_connections 2024-03-06T07:24:27.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.674+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/1650883179 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x555882dd8e00 0x555882dd91e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.674+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/1650883179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882ddaf20 0x555882ddb3a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.674+0000 7f2fa9ffb640 0 client.6844 ms_handle_reset on v2:172.21.15.98:6835/2632566151 2024-03-06T07:24:27.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.674+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/1650883179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882db64e0 0x555882dd9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.674+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/1650883179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db6100 0x555882ddcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.674+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/1650883179 >> 172.21.15.1:0/1650883179 conn(0x555882db4660 msgr2=0x555882daf840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:27.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.675+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/1650883179 shutdown_connections 2024-03-06T07:24:27.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.675+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/1650883179 wait complete. 2024-03-06T07:24:27.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.675+0000 7f2fd3fe2740 20 client.6844 trim_cache size 0 max 16384 2024-03-06T07:24:27.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.691+0000 7f2fd3fe2740 1 Processor -- start 2024-03-06T07:24:27.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.691+0000 7f2fd3fe2740 1 -- start start 2024-03-06T07:24:27.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.692+0000 7f2fd3fe2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882db6100 0x555882e02b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.692+0000 7f2fd3fe2740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db64e0 0x555882e157c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.692+0000 7f2fd3fe2740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882ddaf20 0x555882e15d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.692+0000 7f2fd3fe2740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc0ab480 con 0x555882db64e0 2024-03-06T07:24:27.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.692+0000 7f2fd3fe2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1ea5a0 con 0x555882db6100 2024-03-06T07:24:27.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.692+0000 7f2fd3fe2740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc0aae80 con 0x555882ddaf20 2024-03-06T07:24:27.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.692+0000 7f2fca7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db64e0 0x555882e157c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.692+0000 7f2fca7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db64e0 0x555882e157c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35266/0 (socket says 172.21.15.1:35266) 2024-03-06T07:24:27.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.692+0000 7f2fca7fc640 1 -- 172.21.15.1:0/1806084814 learned_addr learned my addr 172.21.15.1:0/1806084814 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:27.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.692+0000 7f2fcb7fe640 1 --2- 172.21.15.1:0/1806084814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882ddaf20 0x555882e15d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.692+0000 7f2fcaffd640 1 --2- 172.21.15.1:0/1806084814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882db6100 0x555882e02b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.692+0000 7f2fca7fc640 1 -- 172.21.15.1:0/1806084814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882ddaf20 msgr2=0x555882e15d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:27.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.692+0000 7f2fca7fc640 1 --2- 172.21.15.1:0/1806084814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882ddaf20 0x555882e15d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.692+0000 7f2fca7fc640 1 -- 172.21.15.1:0/1806084814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882db6100 msgr2=0x555882e02b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:27.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.692+0000 7f2fca7fc640 1 --2- 172.21.15.1:0/1806084814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882db6100 0x555882e02b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.692+0000 7f2fca7fc640 1 -- 172.21.15.1:0/1806084814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555882af96b0 con 0x555882db64e0 2024-03-06T07:24:27.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.692+0000 7f2fca7fc640 1 --2- 172.21.15.1:0/1806084814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db64e0 0x555882e157c0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f2fb405ffa0 tx=0x7f2fb4087a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=631ee9cef89630c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.693+0000 7f2fabfff640 1 -- 172.21.15.1:0/1806084814 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb405e0d0 con 0x555882db64e0 2024-03-06T07:24:27.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.693+0000 7f2fabfff640 1 -- 172.21.15.1:0/1806084814 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb40544e0 con 0x555882db64e0 2024-03-06T07:24:27.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.693+0000 7f2fabfff640 1 -- 172.21.15.1:0/1806084814 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb40574e0 con 0x555882db64e0 2024-03-06T07:24:27.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.693+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/1806084814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db64e0 msgr2=0x555882e157c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.693+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/1806084814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db64e0 0x555882e157c0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f2fb405ffa0 tx=0x7f2fb4087a40 comp rx=0 tx=0).stop 2024-03-06T07:24:27.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.693+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/1806084814 shutdown_connections 2024-03-06T07:24:27.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.693+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/1806084814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882ddaf20 0x555882e15d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.693+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/1806084814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db64e0 0x555882e157c0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.693+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/1806084814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882db6100 0x555882e02b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.693+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/1806084814 >> 172.21.15.1:0/1806084814 conn(0x555882db4660 msgr2=0x555882db7220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:27.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.693+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/1806084814 shutdown_connections 2024-03-06T07:24:27.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.693+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/1806084814 wait complete. 2024-03-06T07:24:27.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.694+0000 7f2fd3fe2740 1 -- start start 2024-03-06T07:24:27.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.694+0000 7f2fd3fe2740 1 Processor -- start 2024-03-06T07:24:27.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fd3fe2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882db6100 0x555882e0bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fd3fe2740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db64e0 0x555882e0c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fd3fe2740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882ddaf20 0x555882e678c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fd3fe2740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x555882dda6e0 con 0x555882db64e0 2024-03-06T07:24:27.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fd3fe2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x555882dd8c80 con 0x555882db6100 2024-03-06T07:24:27.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fd3fe2740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x555882ddc240 con 0x555882ddaf20 2024-03-06T07:24:27.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fd3fe2740 10 client.0 _ll_register_callbacks cb 0x7f2fbc009990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:27.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fca7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db64e0 0x555882e0c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fcaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882db6100 0x555882e0bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fca7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db64e0 0x555882e0c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35278/0 (socket says 172.21.15.1:35278) 2024-03-06T07:24:27.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fca7fc640 1 -- 172.21.15.1:0/2754826414 learned_addr learned my addr 172.21.15.1:0/2754826414 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:27.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fcaffd640 1 --2- 172.21.15.1:0/2754826414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882db6100 0x555882e0bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:57784/0 (socket says 172.21.15.1:57784) 2024-03-06T07:24:27.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fcb7fe640 1 --2- 172.21.15.1:0/2754826414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882ddaf20 0x555882e678c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fca7fc640 1 -- 172.21.15.1:0/2754826414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882ddaf20 msgr2=0x555882e678c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:27.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fca7fc640 1 --2- 172.21.15.1:0/2754826414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882ddaf20 0x555882e678c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fca7fc640 1 -- 172.21.15.1:0/2754826414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882db6100 msgr2=0x555882e0bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fca7fc640 1 --2- 172.21.15.1:0/2754826414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882db6100 0x555882e0bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fca7fc640 1 -- 172.21.15.1:0/2754826414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555882dfffb0 con 0x555882db64e0 2024-03-06T07:24:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fca7fc640 1 --2- 172.21.15.1:0/2754826414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db64e0 0x555882e0c7a0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4057b70 tx=0x7f2fb400a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de024ae9590ea5b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fa9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:24:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fa9ffb640 1 -- 172.21.15.1:0/2754826414 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb406fd50 con 0x555882db64e0 2024-03-06T07:24:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fa9ffb640 1 -- 172.21.15.1:0/2754826414 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2fb40572c0 con 0x555882db64e0 2024-03-06T07:24:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/2754826414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2fb40544e0 con 0x555882db64e0 2024-03-06T07:24:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fa9ffb640 1 -- 172.21.15.1:0/2754826414 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fb40574c0 con 0x555882db64e0 2024-03-06T07:24:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fd3fe2740 10 client.7617 fetch_fsmap learned FSMap version 93 2024-03-06T07:24:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.695+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/2754826414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=2+}) v3 -- 0x555882e683e0 con 0x555882db64e0 2024-03-06T07:24:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fa9ffb640 1 -- 172.21.15.1:0/2754826414 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb40544e0 con 0x555882db64e0 2024-03-06T07:24:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fa9ffb640 1 -- 172.21.15.1:0/2754826414 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 93) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2fb4053bc0 con 0x555882db64e0 2024-03-06T07:24:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fd3fe2740 10 client.7617 fetch_fsmap finished waiting for FSMap version 93 2024-03-06T07:24:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fd3fe2740 10 client.7617 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:24:27.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fd3fe2740 20 client.7617 populate_metadata read hostname 'smithi001' 2024-03-06T07:24:27.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/2754826414 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x555882e63b80 0x555882e66210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fd3fe2740 4 client.7617 mds_command: new command op to 6122 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:24:27.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/2754826414 --> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555882d21f40 con 0x555882e63b80 2024-03-06T07:24:27.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fcaffd640 1 --2- 172.21.15.1:0/2754826414 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x555882e63b80 0x555882e66210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fa9ffb640 10 client.7617 ms_handle_connect on v2:172.21.15.98:6835/2632566151 2024-03-06T07:24:27.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fcaffd640 1 --2- 172.21.15.1:0/2754826414 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x555882e63b80 0x555882e66210 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b23aff16b5990403 server_cookie=9ed7e76de5d89e46 in_seq=0 out_seq=0 2024-03-06T07:24:27.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fa9ffb640 1 -- 172.21.15.1:0/2754826414 <== mds.0 v2:172.21.15.98:6835/2632566151 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x555882d21f40 con 0x555882e63b80 2024-03-06T07:24:27.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fa9ffb640 10 client.7617 handle_command_reply: tid=0 2024-03-06T07:24:27.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fd3fe2740 1 client.7617 shutdown 2024-03-06T07:24:27.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/2754826414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db64e0 msgr2=0x555882e0c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/2754826414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db64e0 0x555882e0c7a0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4057b70 tx=0x7f2fb400a000 comp rx=0 tx=0).stop 2024-03-06T07:24:27.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/2754826414 shutdown_connections 2024-03-06T07:24:27.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/2754826414 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x555882e63b80 0x555882e66210 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/2754826414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555882ddaf20 0x555882e678c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/2754826414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555882db64e0 0x555882e0c7a0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fd3fe2740 1 --2- 172.21.15.1:0/2754826414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555882db6100 0x555882e0bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fa9ffb640 0 client.7617 ms_handle_reset on v2:172.21.15.98:6835/2632566151 2024-03-06T07:24:27.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.696+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/2754826414 >> 172.21.15.1:0/2754826414 conn(0x555882db4660 msgr2=0x555882dff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:27.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.697+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/2754826414 shutdown_connections 2024-03-06T07:24:27.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.697+0000 7f2fd3fe2740 1 -- 172.21.15.1:0/2754826414 wait complete. 2024-03-06T07:24:27.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.697+0000 7f2fd3fe2740 20 client.7617 trim_cache size 0 max 16384 2024-03-06T07:24:27.705 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:24:27.705 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:24:27.705 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:24:27.706 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 6122, 2024-03-06T07:24:27.706 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:24:27.706 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:24:27.706 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:24:27.706 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 5.7900868939999999, 2024-03-06T07:24:27.706 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 93, 2024-03-06T07:24:27.706 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 109, 2024-03-06T07:24:27.706 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 109, 2024-03-06T07:24:27.706 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 21.905930865999999 2024-03-06T07:24:27.706 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:24:27.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.697+0000 7f2fd1402640 1 -- 172.21.15.1:0/950998754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fac06b500 msgr2=0x7f2fac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.697+0000 7f2fd1402640 1 --2- 172.21.15.1:0/950998754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fac06b500 0x7f2fac06d9c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4037000 tx=0x7f2fc4032000 comp rx=0 tx=0).stop 2024-03-06T07:24:27.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.697+0000 7f2fd1402640 1 -- 172.21.15.1:0/950998754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc14f410 msgr2=0x7f2fcc150e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.697+0000 7f2fd1402640 1 --2- 172.21.15.1:0/950998754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc14f410 0x7f2fcc150e80 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f2fb400c440 tx=0x7f2fb4001e70 comp rx=0 tx=0).stop 2024-03-06T07:24:27.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.697+0000 7f2fd1402640 1 -- 172.21.15.1:0/950998754 shutdown_connections 2024-03-06T07:24:27.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.697+0000 7f2fd1402640 1 --2- 172.21.15.1:0/950998754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fac06b500 0x7f2fac06d9c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.697+0000 7f2fd1402640 1 --2- 172.21.15.1:0/950998754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fcc1555a0 0x7f2fcc14adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.697+0000 7f2fd1402640 1 --2- 172.21.15.1:0/950998754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc14f410 0x7f2fcc150e80 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.697+0000 7f2fd1402640 1 --2- 172.21.15.1:0/950998754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fcc070270 0x7f2fcc0bc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.697+0000 7f2fd1402640 1 -- 172.21.15.1:0/950998754 >> 172.21.15.1:0/950998754 conn(0x7f2fcc0a95b0 msgr2=0x7f2fcc0c53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:27.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.697+0000 7f2fd1402640 1 -- 172.21.15.1:0/950998754 shutdown_connections 2024-03-06T07:24:27.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.697+0000 7f2fd1402640 1 -- 172.21.15.1:0/950998754 wait complete. 2024-03-06T07:24:27.710 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 93, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:24:10.220198+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 93, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6122": { "addr": "172.21.15.98:6837/2632566151", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 2632566151, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2632566151, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6122, "incarnation": 90, "join_fscid": 8, "name": "b", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 109, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-03-06T07:24:24.911657+0000", "qdb_cluster": [ 6122 ], "qdb_leader": 6122, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6122 } } } ], "standbys": [ { "addr": "172.21.15.1:6835/2244230066", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6834", "nonce": 2244230066, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 2244230066, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6670, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6836/1741783790", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 1741783790, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 1741783790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6682, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6833/550216328", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 550216328, "type": "v2" }, { "addr": "172.21.15.1:6833", "nonce": 550216328, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 91, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7590, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:24:27.711 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 8, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:24:27.711 INFO:tasks.cephfs.test_failover:current = [{'gid': 6670, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/2244230066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 2244230066}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2244230066}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}, {'gid': 6122, 'name': 'b', 'rank': 0, 'incarnation': 90, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.98:6837/2632566151', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 2632566151}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2632566151}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7590, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6833/550216328', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 550216328}, {'type': 'v1', 'addr': '172.21.15.1:6833', 'nonce': 550216328}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 91}, {'gid': 6682, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/1741783790', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 1741783790}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1741783790}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}] 2024-03-06T07:24:27.711 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:24:27.711 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-06T07:24:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.813+0000 7f6443b65640 1 Processor -- start 2024-03-06T07:24:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.813+0000 7f6443b65640 1 -- start start 2024-03-06T07:24:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.814+0000 7f6443b65640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f643c0c6b10 0x7f643c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.814+0000 7f6443b65640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f643c0bf9d0 0x7f643c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.814+0000 7f6443b65640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643c0c2fc0 con 0x7f643c152650 2024-03-06T07:24:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.814+0000 7f6443b65640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f643c155800 con 0x7f643c0bf9d0 2024-03-06T07:24:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.814+0000 7f6443b65640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f643c158da0 con 0x7f643c0c6b10 2024-03-06T07:24:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.814+0000 7f6441901640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f643c152650 0x7f643c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53818/0 (socket says 172.21.15.1:53818) 2024-03-06T07:24:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.814+0000 7f6441901640 1 -- 172.21.15.1:0/2469629279 learned_addr learned my addr 172.21.15.1:0/2469629279 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.814+0000 7f6441100640 1 --2- 172.21.15.1:0/2469629279 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f643c0c6b10 0x7f643c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.814+0000 7f6442102640 1 --2- 172.21.15.1:0/2469629279 >> v2:172.21.15.98:3300/0 conn(0x7f643c0bf9d0 0x7f643c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.815+0000 7f64408ff640 1 -- 172.21.15.1:0/2469629279 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167131526 0 0) 0x7f643c0c2fc0 con 0x7f643c152650 2024-03-06T07:24:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.815+0000 7f64408ff640 1 -- 172.21.15.1:0/2469629279 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6424003520 con 0x7f643c152650 2024-03-06T07:24:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.815+0000 7f6441100640 1 -- 172.21.15.1:0/2469629279 >> v2:172.21.15.98:3300/0 conn(0x7f643c0bf9d0 msgr2=0x7f643c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.815+0000 7f6441100640 1 --2- 172.21.15.1:0/2469629279 >> v2:172.21.15.98:3300/0 conn(0x7f643c0bf9d0 0x7f643c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.815+0000 7f6441100640 1 -- 172.21.15.1:0/2469629279 >> v1:172.21.15.1:6789/0 conn(0x7f643c152650 legacy=0x7f643c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.815+0000 7f6441100640 1 -- 172.21.15.1:0/2469629279 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c0c6690 con 0x7f643c0c6b10 2024-03-06T07:24:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.815+0000 7f6442102640 1 --2- 172.21.15.1:0/2469629279 >> v2:172.21.15.98:3300/0 conn(0x7f643c0bf9d0 0x7f643c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.815+0000 7f6441100640 1 --2- 172.21.15.1:0/2469629279 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f643c0c6b10 0x7f643c155c90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f64300097a0 tx=0x7f643002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ba4263c1d047168 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.816+0000 7f64408ff640 1 -- 172.21.15.1:0/2469629279 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64300315a0 con 0x7f643c0c6b10 2024-03-06T07:24:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.816+0000 7f64408ff640 1 -- 172.21.15.1:0/2469629279 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6430037070 con 0x7f643c0c6b10 2024-03-06T07:24:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.816+0000 7f64408ff640 1 -- 172.21.15.1:0/2469629279 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6430031940 con 0x7f643c0c6b10 2024-03-06T07:24:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.816+0000 7f6443b65640 1 -- 172.21.15.1:0/2469629279 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f643c0c6b10 msgr2=0x7f643c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.816+0000 7f6443b65640 1 --2- 172.21.15.1:0/2469629279 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f643c0c6b10 0x7f643c155c90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f64300097a0 tx=0x7f643002b840 comp rx=0 tx=0).stop 2024-03-06T07:24:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.816+0000 7f6443b65640 1 -- 172.21.15.1:0/2469629279 shutdown_connections 2024-03-06T07:24:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.816+0000 7f6443b65640 1 --2- 172.21.15.1:0/2469629279 >> v2:172.21.15.98:3300/0 conn(0x7f643c0bf9d0 0x7f643c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.816+0000 7f6443b65640 1 --2- 172.21.15.1:0/2469629279 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f643c0c6b10 0x7f643c155c90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.816+0000 7f6443b65640 1 -- 172.21.15.1:0/2469629279 >> 172.21.15.1:0/2469629279 conn(0x7f643c066ee0 msgr2=0x7f643c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.816+0000 7f6443b65640 1 -- 172.21.15.1:0/2469629279 shutdown_connections 2024-03-06T07:24:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.816+0000 7f6443b65640 1 -- 172.21.15.1:0/2469629279 wait complete. 2024-03-06T07:24:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.817+0000 7f6443b65640 1 Processor -- start 2024-03-06T07:24:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.817+0000 7f6443b65640 1 -- start start 2024-03-06T07:24:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.818+0000 7f6443b65640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f643c0bf9d0 0x7f643c157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.818+0000 7f6443b65640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f643c152650 0x7f643c0c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.818+0000 7f6443b65640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f643c14afb0 0x7f643c1ee220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.818+0000 7f6443b65640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643c1551b0 con 0x7f643c14afb0 2024-03-06T07:24:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.818+0000 7f6443b65640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643c0c6fa0 con 0x7f643c0bf9d0 2024-03-06T07:24:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.818+0000 7f6443b65640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f643c155490 con 0x7f643c152650 2024-03-06T07:24:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.818+0000 7f6442102640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f643c14afb0 0x7f643c1ee220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.818+0000 7f6442102640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f643c14afb0 0x7f643c1ee220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35294/0 (socket says 172.21.15.1:35294) 2024-03-06T07:24:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.818+0000 7f6442102640 1 -- 172.21.15.1:0/2086060147 learned_addr learned my addr 172.21.15.1:0/2086060147 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.818+0000 7f6441901640 1 --2- 172.21.15.1:0/2086060147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f643c0bf9d0 0x7f643c157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.818+0000 7f6441100640 1 --2- 172.21.15.1:0/2086060147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f643c152650 0x7f643c0c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.818+0000 7f6442102640 1 -- 172.21.15.1:0/2086060147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f643c152650 msgr2=0x7f643c0c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.818+0000 7f6442102640 1 --2- 172.21.15.1:0/2086060147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f643c152650 0x7f643c0c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.818+0000 7f6442102640 1 -- 172.21.15.1:0/2086060147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f643c0bf9d0 msgr2=0x7f643c157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.818+0000 7f6442102640 1 --2- 172.21.15.1:0/2086060147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f643c0bf9d0 0x7f643c157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.818+0000 7f6442102640 1 -- 172.21.15.1:0/2086060147 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c1eeeb0 con 0x7f643c14afb0 2024-03-06T07:24:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.819+0000 7f6442102640 1 --2- 172.21.15.1:0/2086060147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f643c14afb0 0x7f643c1ee220 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f642c00efc0 tx=0x7f642c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.819+0000 7f643a7fc640 1 -- 172.21.15.1:0/2086060147 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642c018030 con 0x7f643c14afb0 2024-03-06T07:24:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.819+0000 7f643a7fc640 1 -- 172.21.15.1:0/2086060147 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f642c00cdf0 con 0x7f643c14afb0 2024-03-06T07:24:27.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.819+0000 7f6443b65640 1 -- 172.21.15.1:0/2086060147 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f643c157f60 con 0x7f643c14afb0 2024-03-06T07:24:27.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.819+0000 7f643a7fc640 1 -- 172.21.15.1:0/2086060147 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642c003bb0 con 0x7f643c14afb0 2024-03-06T07:24:27.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.819+0000 7f6443b65640 1 -- 172.21.15.1:0/2086060147 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f643c1eeb40 con 0x7f643c14afb0 2024-03-06T07:24:27.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.820+0000 7f6443b65640 1 -- 172.21.15.1:0/2086060147 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f643c067fd0 con 0x7f643c14afb0 2024-03-06T07:24:27.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.820+0000 7f643a7fc640 1 -- 172.21.15.1:0/2086060147 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f642c01b070 con 0x7f643c14afb0 2024-03-06T07:24:27.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.821+0000 7f643a7fc640 1 --2- 172.21.15.1:0/2086060147 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f642806b500 0x7f642806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.821+0000 7f643a7fc640 1 -- 172.21.15.1:0/2086060147 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f642c08ab90 con 0x7f643c14afb0 2024-03-06T07:24:27.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.821+0000 7f6441901640 1 --2- 172.21.15.1:0/2086060147 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f642806b500 0x7f642806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.821+0000 7f6441901640 1 --2- 172.21.15.1:0/2086060147 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f642806b500 0x7f642806d9c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6434005880 tx=0x7f6434009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:27.825+0000 7f643a7fc640 1 -- 172.21.15.1:0/2086060147 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f642c010040 con 0x7f643c14afb0 2024-03-06T07:24:28.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:28.046+0000 7f6443b65640 1 -- 172.21.15.1:0/2086060147 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f643c1570b0 con 0x7f643c14afb0 2024-03-06T07:24:28.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:28.935+0000 7f643a7fc640 1 -- 172.21.15.1:0/2086060147 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v94) v1 ==== 121+0+0 (secure 0 0 0) 0x7f642c059270 con 0x7f643c14afb0 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:28.938+0000 7f6443b65640 1 -- 172.21.15.1:0/2086060147 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f642806b500 msgr2=0x7f642806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:28.938+0000 7f6443b65640 1 --2- 172.21.15.1:0/2086060147 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f642806b500 0x7f642806d9c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6434005880 tx=0x7f6434009040 comp rx=0 tx=0).stop 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:28.938+0000 7f6443b65640 1 -- 172.21.15.1:0/2086060147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f643c14afb0 msgr2=0x7f643c1ee220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:28.938+0000 7f6443b65640 1 --2- 172.21.15.1:0/2086060147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f643c14afb0 0x7f643c1ee220 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f642c00efc0 tx=0x7f642c00c490 comp rx=0 tx=0).stop 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:28.939+0000 7f6443b65640 1 -- 172.21.15.1:0/2086060147 shutdown_connections 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:28.939+0000 7f6443b65640 1 --2- 172.21.15.1:0/2086060147 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f642806b500 0x7f642806d9c0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:28.939+0000 7f6443b65640 1 --2- 172.21.15.1:0/2086060147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f643c14afb0 0x7f643c1ee220 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:28.939+0000 7f6443b65640 1 --2- 172.21.15.1:0/2086060147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f643c152650 0x7f643c0c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:28.939+0000 7f6443b65640 1 --2- 172.21.15.1:0/2086060147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f643c0bf9d0 0x7f643c157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:28.939+0000 7f6443b65640 1 -- 172.21.15.1:0/2086060147 >> 172.21.15.1:0/2086060147 conn(0x7f643c066ee0 msgr2=0x7f643c0c0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:28.939+0000 7f6443b65640 1 -- 172.21.15.1:0/2086060147 shutdown_connections 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:28.939+0000 7f6443b65640 1 -- 172.21.15.1:0/2086060147 wait complete. 2024-03-06T07:24:28.955 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:29.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.061+0000 7f9aa6135640 1 Processor -- start 2024-03-06T07:24:29.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.061+0000 7f9aa6135640 1 -- start start 2024-03-06T07:24:29.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.062+0000 7f9aa6135640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9aa00c1820 0x7f9aa01559f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.062+0000 7f9aa6135640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa01588e0 con 0x7f9aa0152730 2024-03-06T07:24:29.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.062+0000 7f9aa6135640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa0158bc0 con 0x7f9aa00c1390 2024-03-06T07:24:29.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.062+0000 7f9aa6135640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa0158ea0 con 0x7f9aa00c1820 2024-03-06T07:24:29.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.062+0000 7f9a9f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9aa0152730 0x7f9aa00c6a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53824/0 (socket says 172.21.15.1:53824) 2024-03-06T07:24:29.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.062+0000 7f9a9f7fe640 1 -- 172.21.15.1:0/2008419496 learned_addr learned my addr 172.21.15.1:0/2008419496 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:29.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.062+0000 7f9a9ffff640 1 --2- 172.21.15.1:0/2008419496 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9aa00c1820 0x7f9aa01559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.062+0000 7f9a9effd640 1 --1- 172.21.15.1:0/2008419496 >> v1:172.21.15.98:6789/0 conn(0x7f9aa00c1390 0x7f9aa01552e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:50230/0 (socket says 172.21.15.1:50230) 2024-03-06T07:24:29.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.062+0000 7f9a9e7fc640 1 -- 172.21.15.1:0/2008419496 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441550647 0 0) 0x7f9aa0158bc0 con 0x7f9aa00c1390 2024-03-06T07:24:29.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.062+0000 7f9a9e7fc640 1 -- 172.21.15.1:0/2008419496 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a88003520 con 0x7f9aa00c1390 2024-03-06T07:24:29.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.062+0000 7f9a9e7fc640 1 -- 172.21.15.1:0/2008419496 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1132990503 0 0) 0x7f9aa01588e0 con 0x7f9aa0152730 2024-03-06T07:24:29.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.062+0000 7f9a9ffff640 1 -- 172.21.15.1:0/2008419496 >> v1:172.21.15.98:6789/0 conn(0x7f9aa00c1390 legacy=0x7f9aa01552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.062+0000 7f9a9ffff640 1 -- 172.21.15.1:0/2008419496 >> v1:172.21.15.1:6789/0 conn(0x7f9aa0152730 legacy=0x7f9aa00c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.062+0000 7f9a9ffff640 1 -- 172.21.15.1:0/2008419496 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa0159180 con 0x7f9aa00c1820 2024-03-06T07:24:29.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.063+0000 7f9a9ffff640 1 --2- 172.21.15.1:0/2008419496 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9aa00c1820 0x7f9aa01559f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009dc0 tx=0x7f9a9802be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40381e7f47d5162e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.063+0000 7f9a9e7fc640 1 -- 172.21.15.1:0/2008419496 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98039070 con 0x7f9aa00c1820 2024-03-06T07:24:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.063+0000 7f9a9e7fc640 1 -- 172.21.15.1:0/2008419496 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9802cbb0 con 0x7f9aa00c1820 2024-03-06T07:24:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.063+0000 7f9a9e7fc640 1 -- 172.21.15.1:0/2008419496 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98034440 con 0x7f9aa00c1820 2024-03-06T07:24:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.064+0000 7f9aa6135640 1 -- 172.21.15.1:0/2008419496 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9aa00c1820 msgr2=0x7f9aa01559f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.064+0000 7f9aa6135640 1 --2- 172.21.15.1:0/2008419496 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9aa00c1820 0x7f9aa01559f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009dc0 tx=0x7f9a9802be00 comp rx=0 tx=0).stop 2024-03-06T07:24:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.064+0000 7f9aa6135640 1 -- 172.21.15.1:0/2008419496 shutdown_connections 2024-03-06T07:24:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.064+0000 7f9aa6135640 1 --2- 172.21.15.1:0/2008419496 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9aa00c1820 0x7f9aa01559f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.064+0000 7f9aa6135640 1 -- 172.21.15.1:0/2008419496 >> 172.21.15.1:0/2008419496 conn(0x7f9aa0066d00 msgr2=0x7f9aa0067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.064+0000 7f9aa6135640 1 -- 172.21.15.1:0/2008419496 shutdown_connections 2024-03-06T07:24:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.064+0000 7f9aa6135640 1 -- 172.21.15.1:0/2008419496 wait complete. 2024-03-06T07:24:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.065+0000 7f9aa6135640 1 Processor -- start 2024-03-06T07:24:29.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.065+0000 7f9aa6135640 1 -- start start 2024-03-06T07:24:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9aa6135640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa00c1390 0x7f9aa01eb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9aa6135640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c1820 0x7f9aa00c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9aa6135640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9aa0152730 0x7f9aa00cac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9aa6135640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa0155f30 con 0x7f9aa00c1820 2024-03-06T07:24:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9aa6135640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa00c1c90 con 0x7f9aa00c1390 2024-03-06T07:24:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9aa6135640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa0152b10 con 0x7f9aa0152730 2024-03-06T07:24:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9a9effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c1820 0x7f9aa00c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9a9effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c1820 0x7f9aa00c8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35308/0 (socket says 172.21.15.1:35308) 2024-03-06T07:24:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9a9effd640 1 -- 172.21.15.1:0/3960463035 learned_addr learned my addr 172.21.15.1:0/3960463035 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9a9f7fe640 1 --2- 172.21.15.1:0/3960463035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa00c1390 0x7f9aa01eb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9a9ffff640 1 --2- 172.21.15.1:0/3960463035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9aa0152730 0x7f9aa00cac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9a9effd640 1 -- 172.21.15.1:0/3960463035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9aa0152730 msgr2=0x7f9aa00cac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9a9effd640 1 --2- 172.21.15.1:0/3960463035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9aa0152730 0x7f9aa00cac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9a9effd640 1 -- 172.21.15.1:0/3960463035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa00c1390 msgr2=0x7f9aa01eb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9a9effd640 1 --2- 172.21.15.1:0/3960463035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa00c1390 0x7f9aa01eb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9a9effd640 1 -- 172.21.15.1:0/3960463035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa01ebbd0 con 0x7f9aa00c1820 2024-03-06T07:24:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9a9effd640 1 --2- 172.21.15.1:0/3960463035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c1820 0x7f9aa00c8810 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400c410 tx=0x7f9a9400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9a87fff640 1 -- 172.21.15.1:0/3960463035 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a940074e0 con 0x7f9aa00c1820 2024-03-06T07:24:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.066+0000 7f9a87fff640 1 -- 172.21.15.1:0/3960463035 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a94007680 con 0x7f9aa00c1820 2024-03-06T07:24:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.067+0000 7f9a87fff640 1 -- 172.21.15.1:0/3960463035 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a94004a00 con 0x7f9aa00c1820 2024-03-06T07:24:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.067+0000 7f9aa6135640 1 -- 172.21.15.1:0/3960463035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa0147040 con 0x7f9aa00c1820 2024-03-06T07:24:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.067+0000 7f9aa6135640 1 -- 172.21.15.1:0/3960463035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa00cb140 con 0x7f9aa00c1820 2024-03-06T07:24:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.068+0000 7f9aa6135640 1 -- 172.21.15.1:0/3960463035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa0149dd0 con 0x7f9aa00c1820 2024-03-06T07:24:29.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.068+0000 7f9a87fff640 1 -- 172.21.15.1:0/3960463035 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a9401a020 con 0x7f9aa00c1820 2024-03-06T07:24:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.068+0000 7f9a87fff640 1 --2- 172.21.15.1:0/3960463035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a8c06b4d0 0x7f9a8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.068+0000 7f9a87fff640 1 -- 172.21.15.1:0/3960463035 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f9a94089b70 con 0x7f9aa00c1820 2024-03-06T07:24:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.069+0000 7f9a9f7fe640 1 --2- 172.21.15.1:0/3960463035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a8c06b4d0 0x7f9a8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.069+0000 7f9a9f7fe640 1 --2- 172.21.15.1:0/3960463035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a8c06b4d0 0x7f9a8c06d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9a900011e0 tx=0x7f9a90001130 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.073+0000 7f9a87fff640 1 -- 172.21.15.1:0/3960463035 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a940581f0 con 0x7f9aa00c1820 2024-03-06T07:24:29.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.292+0000 7f9aa6135640 1 -- 172.21.15.1:0/3960463035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9aa00c5e30 con 0x7f9aa00c1820 2024-03-06T07:24:29.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.292+0000 7f9a87fff640 1 -- 172.21.15.1:0/3960463035 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f9a9405c290 con 0x7f9aa00c1820 2024-03-06T07:24:29.295 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:29.295 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":95,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2244230066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2244230066},{"type":"v1","addr":"172.21.15.1:6835","nonce":2244230066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":6682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1741783790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1741783790},{"type":"v1","addr":"172.21.15.98:6836","nonce":1741783790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":95,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-06T07:24:10.220198+0000","modified":"2024-03-06T07:24:28.938794+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6122":{"gid":6122,"name":"b","rank":0,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6837/2632566151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2632566151},{"type":"v1","addr":"172.21.15.98:6837","nonce":2632566151}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7590":{"gid":7590,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.1:6833/550216328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":550216328},{"type":"v1","addr":"172.21.15.1:6833","nonce":550216328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6122,"qdb_cluster":[6122]},"id":8}]} 2024-03-06T07:24:29.295 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 95 2024-03-06T07:24:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.294+0000 7f9aa6135640 1 -- 172.21.15.1:0/3960463035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a8c06b4d0 msgr2=0x7f9a8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.295+0000 7f9aa6135640 1 --2- 172.21.15.1:0/3960463035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a8c06b4d0 0x7f9a8c06d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9a900011e0 tx=0x7f9a90001130 comp rx=0 tx=0).stop 2024-03-06T07:24:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.295+0000 7f9aa6135640 1 -- 172.21.15.1:0/3960463035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c1820 msgr2=0x7f9aa00c8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.295+0000 7f9aa6135640 1 --2- 172.21.15.1:0/3960463035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c1820 0x7f9aa00c8810 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400c410 tx=0x7f9a9400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.295+0000 7f9aa6135640 1 -- 172.21.15.1:0/3960463035 shutdown_connections 2024-03-06T07:24:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.295+0000 7f9aa6135640 1 --2- 172.21.15.1:0/3960463035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a8c06b4d0 0x7f9a8c06d990 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.295+0000 7f9aa6135640 1 --2- 172.21.15.1:0/3960463035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9aa0152730 0x7f9aa00cac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.295+0000 7f9aa6135640 1 --2- 172.21.15.1:0/3960463035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c1820 0x7f9aa00c8810 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.295+0000 7f9aa6135640 1 --2- 172.21.15.1:0/3960463035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa00c1390 0x7f9aa01eb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.295+0000 7f9aa6135640 1 -- 172.21.15.1:0/3960463035 >> 172.21.15.1:0/3960463035 conn(0x7f9aa0066d00 msgr2=0x7f9aa00c3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.295+0000 7f9aa6135640 1 -- 172.21.15.1:0/3960463035 shutdown_connections 2024-03-06T07:24:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.295+0000 7f9aa6135640 1 -- 172.21.15.1:0/3960463035 wait complete. 2024-03-06T07:24:29.308 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.a mds_join_fs cephfs 2024-03-06T07:24:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.410+0000 7f724fafb640 1 Processor -- start 2024-03-06T07:24:29.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.410+0000 7f724fafb640 1 -- start start 2024-03-06T07:24:29.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.411+0000 7f724fafb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7248152650 0x7f7248155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.411+0000 7f724fafb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72480bf9d0 0x7f72480c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.411+0000 7f724fafb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72480c2fc0 con 0x7f72480c6b10 2024-03-06T07:24:29.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.411+0000 7f724fafb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7248155800 con 0x7f7248152650 2024-03-06T07:24:29.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.411+0000 7f724fafb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7248158da0 con 0x7f72480bf9d0 2024-03-06T07:24:29.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.411+0000 7f724d897640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7248152650 0x7f7248155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.411+0000 7f724d897640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7248152650 0x7f7248155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35322/0 (socket says 172.21.15.1:35322) 2024-03-06T07:24:29.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.411+0000 7f724d897640 1 -- 172.21.15.1:0/3684120217 learned_addr learned my addr 172.21.15.1:0/3684120217 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:29.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.411+0000 7f724e098640 1 --2- 172.21.15.1:0/3684120217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72480bf9d0 0x7f72480c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.411+0000 7f724d897640 1 -- 172.21.15.1:0/3684120217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72480bf9d0 msgr2=0x7f72480c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.411+0000 7f724d897640 1 --2- 172.21.15.1:0/3684120217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72480bf9d0 0x7f72480c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.411+0000 7f724d897640 1 -- 172.21.15.1:0/3684120217 >> v1:172.21.15.98:6789/0 conn(0x7f72480c6b10 legacy=0x7f7248158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.411+0000 7f724d897640 1 -- 172.21.15.1:0/3684120217 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72480c6690 con 0x7f7248152650 2024-03-06T07:24:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.412+0000 7f724d897640 1 --2- 172.21.15.1:0/3684120217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7248152650 0x7f7248155c90 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f7240009e50 tx=0x7f724002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c705739f51c12598 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.412+0000 7f724c895640 1 -- 172.21.15.1:0/3684120217 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724002ca70 con 0x7f7248152650 2024-03-06T07:24:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.412+0000 7f724c895640 1 -- 172.21.15.1:0/3684120217 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724002cc10 con 0x7f7248152650 2024-03-06T07:24:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.412+0000 7f724c895640 1 -- 172.21.15.1:0/3684120217 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72400343f0 con 0x7f7248152650 2024-03-06T07:24:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.413+0000 7f724fafb640 1 -- 172.21.15.1:0/3684120217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7248152650 msgr2=0x7f7248155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.413+0000 7f724fafb640 1 --2- 172.21.15.1:0/3684120217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7248152650 0x7f7248155c90 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f7240009e50 tx=0x7f724002bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.413+0000 7f724fafb640 1 -- 172.21.15.1:0/3684120217 shutdown_connections 2024-03-06T07:24:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.413+0000 7f724fafb640 1 --2- 172.21.15.1:0/3684120217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72480bf9d0 0x7f72480c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.413+0000 7f724fafb640 1 --2- 172.21.15.1:0/3684120217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7248152650 0x7f7248155c90 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.413+0000 7f724fafb640 1 -- 172.21.15.1:0/3684120217 >> 172.21.15.1:0/3684120217 conn(0x7f7248066ee0 msgr2=0x7f72480672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.413+0000 7f724fafb640 1 -- 172.21.15.1:0/3684120217 shutdown_connections 2024-03-06T07:24:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.413+0000 7f724fafb640 1 -- 172.21.15.1:0/3684120217 wait complete. 2024-03-06T07:24:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.414+0000 7f724fafb640 1 Processor -- start 2024-03-06T07:24:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.414+0000 7f724fafb640 1 -- start start 2024-03-06T07:24:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.415+0000 7f724fafb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72480bf9d0 0x7f7248158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.415+0000 7f724fafb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7248152650 0x7f7248156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.415+0000 7f724fafb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7248150940 0x7f724814b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.415+0000 7f724fafb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72481551b0 con 0x7f7248150940 2024-03-06T07:24:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.415+0000 7f724fafb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72480c6fa0 con 0x7f7248152650 2024-03-06T07:24:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.415+0000 7f724fafb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7248155490 con 0x7f72480bf9d0 2024-03-06T07:24:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.415+0000 7f724e098640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7248150940 0x7f724814b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.415+0000 7f724e098640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7248150940 0x7f724814b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35334/0 (socket says 172.21.15.1:35334) 2024-03-06T07:24:29.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.415+0000 7f724e098640 1 -- 172.21.15.1:0/654348497 learned_addr learned my addr 172.21.15.1:0/654348497 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:29.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.415+0000 7f724d096640 1 --2- 172.21.15.1:0/654348497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7248152650 0x7f7248156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.415+0000 7f724d897640 1 --2- 172.21.15.1:0/654348497 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72480bf9d0 0x7f7248158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.415+0000 7f724e098640 1 -- 172.21.15.1:0/654348497 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72480bf9d0 msgr2=0x7f7248158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.415+0000 7f724e098640 1 --2- 172.21.15.1:0/654348497 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72480bf9d0 0x7f7248158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.415+0000 7f724e098640 1 -- 172.21.15.1:0/654348497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7248152650 msgr2=0x7f7248156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.415+0000 7f724e098640 1 --2- 172.21.15.1:0/654348497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7248152650 0x7f7248156b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.415+0000 7f724e098640 1 -- 172.21.15.1:0/654348497 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72481eed40 con 0x7f7248150940 2024-03-06T07:24:29.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.415+0000 7f724e098640 1 --2- 172.21.15.1:0/654348497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7248150940 0x7f724814b090 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f723800e7f0 tx=0x7f723800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.416+0000 7f72467fc640 1 -- 172.21.15.1:0/654348497 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7238017070 con 0x7f7248150940 2024-03-06T07:24:29.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.416+0000 7f72467fc640 1 -- 172.21.15.1:0/654348497 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f723800ee40 con 0x7f7248150940 2024-03-06T07:24:29.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.416+0000 7f724fafb640 1 -- 172.21.15.1:0/654348497 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72481ef020 con 0x7f7248150940 2024-03-06T07:24:29.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.416+0000 7f72467fc640 1 -- 172.21.15.1:0/654348497 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723800cb80 con 0x7f7248150940 2024-03-06T07:24:29.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.416+0000 7f724fafb640 1 -- 172.21.15.1:0/654348497 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f724814bbe0 con 0x7f7248150940 2024-03-06T07:24:29.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.417+0000 7f724fafb640 1 -- 172.21.15.1:0/654348497 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7248067fd0 con 0x7f7248150940 2024-03-06T07:24:29.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.417+0000 7f72467fc640 1 -- 172.21.15.1:0/654348497 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f723801a070 con 0x7f7248150940 2024-03-06T07:24:29.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.417+0000 7f72467fc640 1 --2- 172.21.15.1:0/654348497 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f723406b4d0 0x7f723406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.417+0000 7f72467fc640 1 -- 172.21.15.1:0/654348497 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f723808a590 con 0x7f7248150940 2024-03-06T07:24:29.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.418+0000 7f724d897640 1 --2- 172.21.15.1:0/654348497 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f723406b4d0 0x7f723406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.418+0000 7f724d897640 1 --2- 172.21.15.1:0/654348497 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f723406b4d0 0x7f723406d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7240035000 tx=0x7f7240030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.422+0000 7f72467fc640 1 -- 172.21.15.1:0/654348497 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7238058be0 con 0x7f7248150940 2024-03-06T07:24:29.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.604+0000 7f724fafb640 1 -- 172.21.15.1:0/654348497 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f72480b06f0 con 0x7f7248150940 2024-03-06T07:24:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.607+0000 7f72467fc640 1 -- 172.21.15.1:0/654348497 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v37)=0 v37) v1 ==== 116+0+0 (secure 0 0 0) 0x7f723805cc80 con 0x7f7248150940 2024-03-06T07:24:29.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.609+0000 7f724fafb640 1 -- 172.21.15.1:0/654348497 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f723406b4d0 msgr2=0x7f723406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.609+0000 7f724fafb640 1 --2- 172.21.15.1:0/654348497 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f723406b4d0 0x7f723406d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7240035000 tx=0x7f7240030000 comp rx=0 tx=0).stop 2024-03-06T07:24:29.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.609+0000 7f724fafb640 1 -- 172.21.15.1:0/654348497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7248150940 msgr2=0x7f724814b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.609+0000 7f724fafb640 1 --2- 172.21.15.1:0/654348497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7248150940 0x7f724814b090 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f723800e7f0 tx=0x7f723800ebf0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.609+0000 7f724fafb640 1 -- 172.21.15.1:0/654348497 shutdown_connections 2024-03-06T07:24:29.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.609+0000 7f724fafb640 1 --2- 172.21.15.1:0/654348497 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f723406b4d0 0x7f723406d990 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.609+0000 7f724fafb640 1 --2- 172.21.15.1:0/654348497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7248150940 0x7f724814b090 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.609+0000 7f724fafb640 1 --2- 172.21.15.1:0/654348497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7248152650 0x7f7248156b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.609+0000 7f724fafb640 1 --2- 172.21.15.1:0/654348497 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72480bf9d0 0x7f7248158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.609+0000 7f724fafb640 1 -- 172.21.15.1:0/654348497 >> 172.21.15.1:0/654348497 conn(0x7f7248066ee0 msgr2=0x7f72480bfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.609+0000 7f724fafb640 1 -- 172.21.15.1:0/654348497 shutdown_connections 2024-03-06T07:24:29.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.609+0000 7f724fafb640 1 -- 172.21.15.1:0/654348497 wait complete. 2024-03-06T07:24:29.621 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:29.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.722+0000 7f134c39a640 1 Processor -- start 2024-03-06T07:24:29.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.722+0000 7f134c39a640 1 -- start start 2024-03-06T07:24:29.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.723+0000 7f134c39a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f13440c6b10 0x7f1344155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.723+0000 7f134c39a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13440bf9d0 0x7f13440c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.723+0000 7f134c39a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13440c2fc0 con 0x7f1344152650 2024-03-06T07:24:29.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.723+0000 7f134c39a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1344155800 con 0x7f13440c6b10 2024-03-06T07:24:29.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.723+0000 7f134c39a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1344158da0 con 0x7f13440bf9d0 2024-03-06T07:24:29.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.723+0000 7f1349935640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f13440c6b10 0x7f1344155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.723+0000 7f134a136640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1344152650 0x7f1344158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53836/0 (socket says 172.21.15.1:53836) 2024-03-06T07:24:29.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.723+0000 7f134a136640 1 -- 172.21.15.1:0/2675574174 learned_addr learned my addr 172.21.15.1:0/2675574174 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:29.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.724+0000 7f134a937640 1 --2- 172.21.15.1:0/2675574174 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13440bf9d0 0x7f13440c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.724+0000 7f1349935640 1 -- 172.21.15.1:0/2675574174 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13440bf9d0 msgr2=0x7f13440c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.724+0000 7f1349935640 1 --2- 172.21.15.1:0/2675574174 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13440bf9d0 0x7f13440c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.724+0000 7f1349935640 1 -- 172.21.15.1:0/2675574174 >> v1:172.21.15.1:6789/0 conn(0x7f1344152650 legacy=0x7f1344158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.724+0000 7f1349134640 1 -- 172.21.15.1:0/2675574174 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700097546 0 0) 0x7f13440c2fc0 con 0x7f1344152650 2024-03-06T07:24:29.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.724+0000 7f1349935640 1 -- 172.21.15.1:0/2675574174 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13440c6690 con 0x7f13440c6b10 2024-03-06T07:24:29.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.725+0000 7f1349935640 1 --2- 172.21.15.1:0/2675574174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13440c6b10 0x7f1344155c90 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f13340097a0 tx=0x7f133402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d272a450ff3c5ac5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.725+0000 7f1349134640 1 -- 172.21.15.1:0/2675574174 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13340315a0 con 0x7f13440c6b10 2024-03-06T07:24:29.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.725+0000 7f1349134640 1 -- 172.21.15.1:0/2675574174 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1334037070 con 0x7f13440c6b10 2024-03-06T07:24:29.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.725+0000 7f1349134640 1 -- 172.21.15.1:0/2675574174 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1334031960 con 0x7f13440c6b10 2024-03-06T07:24:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.725+0000 7f134c39a640 1 -- 172.21.15.1:0/2675574174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13440c6b10 msgr2=0x7f1344155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.725+0000 7f134c39a640 1 --2- 172.21.15.1:0/2675574174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13440c6b10 0x7f1344155c90 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f13340097a0 tx=0x7f133402b840 comp rx=0 tx=0).stop 2024-03-06T07:24:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.725+0000 7f134c39a640 1 -- 172.21.15.1:0/2675574174 shutdown_connections 2024-03-06T07:24:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.725+0000 7f134c39a640 1 --2- 172.21.15.1:0/2675574174 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13440bf9d0 0x7f13440c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.725+0000 7f134c39a640 1 --2- 172.21.15.1:0/2675574174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13440c6b10 0x7f1344155c90 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.725+0000 7f134c39a640 1 -- 172.21.15.1:0/2675574174 >> 172.21.15.1:0/2675574174 conn(0x7f1344066ee0 msgr2=0x7f13440672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.725+0000 7f134c39a640 1 -- 172.21.15.1:0/2675574174 shutdown_connections 2024-03-06T07:24:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.726+0000 7f134c39a640 1 -- 172.21.15.1:0/2675574174 wait complete. 2024-03-06T07:24:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.727+0000 7f134c39a640 1 Processor -- start 2024-03-06T07:24:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.727+0000 7f134c39a640 1 -- start start 2024-03-06T07:24:29.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.727+0000 7f134c39a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13440bf9d0 0x7f13441eedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.727+0000 7f134c39a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1344152650 0x7f13440c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.727+0000 7f134c39a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1344151190 0x7f13441569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.727+0000 7f134c39a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13441551b0 con 0x7f13440bf9d0 2024-03-06T07:24:29.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.727+0000 7f134c39a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13440c6fa0 con 0x7f1344152650 2024-03-06T07:24:29.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.727+0000 7f134c39a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1344155490 con 0x7f1344151190 2024-03-06T07:24:29.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.727+0000 7f134a136640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13440bf9d0 0x7f13441eedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.727+0000 7f134a136640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13440bf9d0 0x7f13441eedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35344/0 (socket says 172.21.15.1:35344) 2024-03-06T07:24:29.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.727+0000 7f134a136640 1 -- 172.21.15.1:0/1399579405 learned_addr learned my addr 172.21.15.1:0/1399579405 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:29.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.727+0000 7f1349935640 1 --2- 172.21.15.1:0/1399579405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1344152650 0x7f13440c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.727+0000 7f134a937640 1 --2- 172.21.15.1:0/1399579405 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1344151190 0x7f13441569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.728+0000 7f134a136640 1 -- 172.21.15.1:0/1399579405 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1344151190 msgr2=0x7f13441569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.728+0000 7f134a136640 1 --2- 172.21.15.1:0/1399579405 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1344151190 0x7f13441569a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.728+0000 7f134a136640 1 -- 172.21.15.1:0/1399579405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1344152650 msgr2=0x7f13440c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.728+0000 7f134a136640 1 --2- 172.21.15.1:0/1399579405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1344152650 0x7f13440c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.728+0000 7f134a136640 1 -- 172.21.15.1:0/1399579405 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13440ca800 con 0x7f13440bf9d0 2024-03-06T07:24:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.728+0000 7f134a136640 1 --2- 172.21.15.1:0/1399579405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13440bf9d0 0x7f13441eedb0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f134000c4a0 tx=0x7f134000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.728+0000 7f133affd640 1 -- 172.21.15.1:0/1399579405 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1340017070 con 0x7f13440bf9d0 2024-03-06T07:24:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.728+0000 7f133affd640 1 -- 172.21.15.1:0/1399579405 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1340005ce0 con 0x7f13440bf9d0 2024-03-06T07:24:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.728+0000 7f134c39a640 1 -- 172.21.15.1:0/1399579405 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1344151650 con 0x7f13440bf9d0 2024-03-06T07:24:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.728+0000 7f133affd640 1 -- 172.21.15.1:0/1399579405 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13400072b0 con 0x7f13440bf9d0 2024-03-06T07:24:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.728+0000 7f134c39a640 1 -- 172.21.15.1:0/1399579405 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13441eb2b0 con 0x7f13440bf9d0 2024-03-06T07:24:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.729+0000 7f134c39a640 1 -- 172.21.15.1:0/1399579405 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1344067fd0 con 0x7f13440bf9d0 2024-03-06T07:24:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.729+0000 7f133affd640 1 -- 172.21.15.1:0/1399579405 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f134001a070 con 0x7f13440bf9d0 2024-03-06T07:24:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.730+0000 7f133affd640 1 --2- 172.21.15.1:0/1399579405 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f132806b500 0x7f132806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.730+0000 7f133affd640 1 -- 172.21.15.1:0/1399579405 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f13400896c0 con 0x7f13440bf9d0 2024-03-06T07:24:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.730+0000 7f1349935640 1 --2- 172.21.15.1:0/1399579405 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f132806b500 0x7f132806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.730+0000 7f1349935640 1 --2- 172.21.15.1:0/1399579405 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f132806b500 0x7f132806d9c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1334036000 tx=0x7f133403d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.734+0000 7f133affd640 1 -- 172.21.15.1:0/1399579405 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1340057d10 con 0x7f13440bf9d0 2024-03-06T07:24:29.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.952+0000 7f134c39a640 1 -- 172.21.15.1:0/1399579405 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13440b06f0 con 0x7f13440bf9d0 2024-03-06T07:24:29.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.952+0000 7f133affd640 1 -- 172.21.15.1:0/1399579405 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f134005bdb0 con 0x7f13440bf9d0 2024-03-06T07:24:29.955 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:29.956 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":95,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2244230066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2244230066},{"type":"v1","addr":"172.21.15.1:6835","nonce":2244230066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":6682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1741783790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1741783790},{"type":"v1","addr":"172.21.15.98:6836","nonce":1741783790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":95,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-06T07:24:10.220198+0000","modified":"2024-03-06T07:24:28.938794+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6122":{"gid":6122,"name":"b","rank":0,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6837/2632566151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2632566151},{"type":"v1","addr":"172.21.15.98:6837","nonce":2632566151}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7590":{"gid":7590,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.1:6833/550216328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":550216328},{"type":"v1","addr":"172.21.15.1:6833","nonce":550216328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6122,"qdb_cluster":[6122]},"id":8}]} 2024-03-06T07:24:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 95 2024-03-06T07:24:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.954+0000 7f134c39a640 1 -- 172.21.15.1:0/1399579405 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f132806b500 msgr2=0x7f132806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.954+0000 7f134c39a640 1 --2- 172.21.15.1:0/1399579405 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f132806b500 0x7f132806d9c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1334036000 tx=0x7f133403d040 comp rx=0 tx=0).stop 2024-03-06T07:24:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.954+0000 7f134c39a640 1 -- 172.21.15.1:0/1399579405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13440bf9d0 msgr2=0x7f13441eedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.954+0000 7f134c39a640 1 --2- 172.21.15.1:0/1399579405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13440bf9d0 0x7f13441eedb0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f134000c4a0 tx=0x7f134000c960 comp rx=0 tx=0).stop 2024-03-06T07:24:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.954+0000 7f134c39a640 1 -- 172.21.15.1:0/1399579405 shutdown_connections 2024-03-06T07:24:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.954+0000 7f134c39a640 1 --2- 172.21.15.1:0/1399579405 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f132806b500 0x7f132806d9c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.954+0000 7f134c39a640 1 --2- 172.21.15.1:0/1399579405 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1344151190 0x7f13441569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.954+0000 7f134c39a640 1 --2- 172.21.15.1:0/1399579405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1344152650 0x7f13440c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.954+0000 7f134c39a640 1 --2- 172.21.15.1:0/1399579405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13440bf9d0 0x7f13441eedb0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.955+0000 7f134c39a640 1 -- 172.21.15.1:0/1399579405 >> 172.21.15.1:0/1399579405 conn(0x7f1344066ee0 msgr2=0x7f13440c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.955+0000 7f134c39a640 1 -- 172.21.15.1:0/1399579405 shutdown_connections 2024-03-06T07:24:29.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:29.955+0000 7f134c39a640 1 -- 172.21.15.1:0/1399579405 wait complete. 2024-03-06T07:24:30.371 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-06T07:24:10.220198+0000', 'modified': '2024-03-06T07:24:28.938794+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 109, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6122}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6122': {'gid': 6122, 'name': 'b', 'rank': 0, 'incarnation': 90, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.98:6837/2632566151', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 2632566151}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2632566151}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_7590': {'gid': 7590, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.1:6833/550216328', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 550216328}, {'type': 'v1', 'addr': '172.21.15.1:6833', 'nonce': 550216328}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6122, 'qdb_cluster': [6122]} 2024-03-06T07:24:30.371 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:24:30.371 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:24:30.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.475+0000 7f61ae993640 1 Processor -- start 2024-03-06T07:24:30.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.475+0000 7f61ae993640 1 -- start start 2024-03-06T07:24:30.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.476+0000 7f61ae993640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61a814cdc0 0x7f61a8152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.476+0000 7f61ae993640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f61a806b510 0x7f61a806b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.476+0000 7f61ae993640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a81569e0 con 0x7f61a8156500 2024-03-06T07:24:30.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.476+0000 7f61ae993640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a80c2dd0 con 0x7f61a806b510 2024-03-06T07:24:30.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.476+0000 7f61ae993640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a8156cc0 con 0x7f61a814cdc0 2024-03-06T07:24:30.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.476+0000 7f61acf30640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f61a806b510 0x7f61a806b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.476+0000 7f61a7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61a814cdc0 0x7f61a8152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.476+0000 7f61a77fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f61a8156500 0x7f61a80c2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:50242/0 (socket says 172.21.15.1:50242) 2024-03-06T07:24:30.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.476+0000 7f61acf30640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f61a806b510 0x7f61a806b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:57832/0 (socket says 172.21.15.1:57832) 2024-03-06T07:24:30.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.476+0000 7f61a7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61a814cdc0 0x7f61a8152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:52860/0 (socket says 172.21.15.1:52860) 2024-03-06T07:24:30.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.476+0000 7f61a77fe640 1 -- 172.21.15.1:0/3435928209 learned_addr learned my addr 172.21.15.1:0/3435928209 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:30.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.476+0000 7f61acf30640 1 -- 172.21.15.1:0/3435928209 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61a814cdc0 msgr2=0x7f61a8152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.476+0000 7f61acf30640 1 --2- 172.21.15.1:0/3435928209 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61a814cdc0 0x7f61a8152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.476+0000 7f61acf30640 1 -- 172.21.15.1:0/3435928209 >> v1:172.21.15.98:6789/0 conn(0x7f61a8156500 legacy=0x7f61a80c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.476+0000 7f61a6ffd640 1 -- 172.21.15.1:0/3435928209 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348144590 0 0) 0x7f61a81569e0 con 0x7f61a8156500 2024-03-06T07:24:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.476+0000 7f61acf30640 1 -- 172.21.15.1:0/3435928209 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a8156fa0 con 0x7f61a806b510 2024-03-06T07:24:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.477+0000 7f61acf30640 1 --2- 172.21.15.1:0/3435928209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a806b510 0x7f61a806b980 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009d90 tx=0x7f61a002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abddffe0c81e95e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.477+0000 7f61a6ffd640 1 -- 172.21.15.1:0/3435928209 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a0039070 con 0x7f61a806b510 2024-03-06T07:24:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.477+0000 7f61a6ffd640 1 -- 172.21.15.1:0/3435928209 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a002cc50 con 0x7f61a806b510 2024-03-06T07:24:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.477+0000 7f61a6ffd640 1 -- 172.21.15.1:0/3435928209 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a0034440 con 0x7f61a806b510 2024-03-06T07:24:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.478+0000 7f61ae993640 1 -- 172.21.15.1:0/3435928209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a806b510 msgr2=0x7f61a806b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.478+0000 7f61ae993640 1 --2- 172.21.15.1:0/3435928209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a806b510 0x7f61a806b980 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009d90 tx=0x7f61a002bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.478+0000 7f61ae993640 1 -- 172.21.15.1:0/3435928209 shutdown_connections 2024-03-06T07:24:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.478+0000 7f61ae993640 1 --2- 172.21.15.1:0/3435928209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a806b510 0x7f61a806b980 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.478+0000 7f61ae993640 1 --2- 172.21.15.1:0/3435928209 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61a814cdc0 0x7f61a8152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.478+0000 7f61ae993640 1 -- 172.21.15.1:0/3435928209 >> 172.21.15.1:0/3435928209 conn(0x7f61a80a9700 msgr2=0x7f61a80a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.478+0000 7f61ae993640 1 -- 172.21.15.1:0/3435928209 shutdown_connections 2024-03-06T07:24:30.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.478+0000 7f61ae993640 1 -- 172.21.15.1:0/3435928209 wait complete. 2024-03-06T07:24:30.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.479+0000 7f61ae993640 1 Processor -- start 2024-03-06T07:24:30.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.479+0000 7f61ae993640 1 -- start start 2024-03-06T07:24:30.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.480+0000 7f61ae993640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a806b510 0x7f61a8152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.480+0000 7f61ae993640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61a814cdc0 0x7f61a81de7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.480+0000 7f61ae993640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61a8156500 0x7f61a814ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.480+0000 7f61ae993640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a80c5a70 con 0x7f61a8156500 2024-03-06T07:24:30.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.480+0000 7f61ae993640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a806bec0 con 0x7f61a806b510 2024-03-06T07:24:30.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.480+0000 7f61ae993640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a80c65e0 con 0x7f61a814cdc0 2024-03-06T07:24:30.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.480+0000 7f61acf30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61a8156500 0x7f61a814ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.480+0000 7f61acf30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61a8156500 0x7f61a814ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35348/0 (socket says 172.21.15.1:35348) 2024-03-06T07:24:30.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.480+0000 7f61acf30640 1 -- 172.21.15.1:0/667359129 learned_addr learned my addr 172.21.15.1:0/667359129 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:30.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.480+0000 7f61a77fe640 1 --2- 172.21.15.1:0/667359129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61a814cdc0 0x7f61a81de7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.480+0000 7f61a7fff640 1 --2- 172.21.15.1:0/667359129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a806b510 0x7f61a8152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.480+0000 7f61a77fe640 1 -- 172.21.15.1:0/667359129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a806b510 msgr2=0x7f61a8152730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.480+0000 7f61a77fe640 1 --2- 172.21.15.1:0/667359129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a806b510 0x7f61a8152730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.480+0000 7f61a77fe640 1 -- 172.21.15.1:0/667359129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61a8156500 msgr2=0x7f61a814ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.480+0000 7f61a77fe640 1 --2- 172.21.15.1:0/667359129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61a8156500 0x7f61a814ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.480+0000 7f61a77fe640 1 -- 172.21.15.1:0/667359129 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a814ba20 con 0x7f61a814cdc0 2024-03-06T07:24:30.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.480+0000 7f61a77fe640 1 --2- 172.21.15.1:0/667359129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61a814cdc0 0x7f61a81de7a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f619800c480 tx=0x7f619800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.481+0000 7f61a4ff9640 1 -- 172.21.15.1:0/667359129 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6198018070 con 0x7f61a814cdc0 2024-03-06T07:24:30.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.481+0000 7f61a4ff9640 1 -- 172.21.15.1:0/667359129 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61980074e0 con 0x7f61a814cdc0 2024-03-06T07:24:30.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.481+0000 7f61ae993640 1 -- 172.21.15.1:0/667359129 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a81df150 con 0x7f61a814cdc0 2024-03-06T07:24:30.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.481+0000 7f61a4ff9640 1 -- 172.21.15.1:0/667359129 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6198004a20 con 0x7f61a814cdc0 2024-03-06T07:24:30.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.481+0000 7f61ae993640 1 -- 172.21.15.1:0/667359129 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a81ddf60 con 0x7f61a814cdc0 2024-03-06T07:24:30.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.482+0000 7f61a4ff9640 1 -- 172.21.15.1:0/667359129 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f619801b070 con 0x7f61a814cdc0 2024-03-06T07:24:30.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.482+0000 7f61a4ff9640 1 --2- 172.21.15.1:0/667359129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f619406b4d0 0x7f619406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.482+0000 7f61a4ff9640 1 -- 172.21.15.1:0/667359129 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f6198089670 con 0x7f61a814cdc0 2024-03-06T07:24:30.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.483+0000 7f61a7fff640 1 --2- 172.21.15.1:0/667359129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f619406b4d0 0x7f619406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.483+0000 7f61a7fff640 1 --2- 172.21.15.1:0/667359129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f619406b4d0 0x7f619406d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f61a80734b0 tx=0x7f619c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.483+0000 7f61b157c740 1 Processor -- start 2024-03-06T07:24:30.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.483+0000 7f61b157c740 1 -- start start 2024-03-06T07:24:30.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.484+0000 7f61b157c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbaf50 0x55d685db8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.484+0000 7f61b157c740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685dbb330 0x55d685db8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.484+0000 7f61b157c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685db94b0 0x55d685db5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.484+0000 7f61b157c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d685d1f6b0 con 0x55d685db94b0 2024-03-06T07:24:30.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.484+0000 7f61b157c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d685cc8a60 con 0x55d685dbaf50 2024-03-06T07:24:30.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.484+0000 7f61b157c740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d685d1b750 con 0x55d685dbb330 2024-03-06T07:24:30.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.484+0000 7f61acf30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685db94b0 0x55d685db5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.484+0000 7f61acf30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685db94b0 0x55d685db5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35360/0 (socket says 172.21.15.1:35360) 2024-03-06T07:24:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.484+0000 7f61acf30640 1 -- 172.21.15.1:0/508197572 learned_addr learned my addr 172.21.15.1:0/508197572 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.484+0000 7f61a7fff640 1 --2- 172.21.15.1:0/508197572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbaf50 0x55d685db8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.484+0000 7f61a77fe640 1 --2- 172.21.15.1:0/508197572 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685dbb330 0x55d685db8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.484+0000 7f61acf30640 1 -- 172.21.15.1:0/508197572 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685dbb330 msgr2=0x55d685db8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.484+0000 7f61acf30640 1 --2- 172.21.15.1:0/508197572 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685dbb330 0x55d685db8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.484+0000 7f61acf30640 1 -- 172.21.15.1:0/508197572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbaf50 msgr2=0x55d685db8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.484+0000 7f61acf30640 1 --2- 172.21.15.1:0/508197572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbaf50 0x55d685db8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.484+0000 7f61acf30640 1 -- 172.21.15.1:0/508197572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d685ce6f30 con 0x55d685db94b0 2024-03-06T07:24:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.484+0000 7f61acf30640 1 --2- 172.21.15.1:0/508197572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685db94b0 0x55d685db5680 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f61a002c4b0 tx=0x7f61a002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7cbeea6bc15df1e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.485+0000 7f618dffb640 1 -- 172.21.15.1:0/508197572 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a0043030 con 0x55d685db94b0 2024-03-06T07:24:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.485+0000 7f618dffb640 1 -- 172.21.15.1:0/508197572 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a003a040 con 0x55d685db94b0 2024-03-06T07:24:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.485+0000 7f618dffb640 1 -- 172.21.15.1:0/508197572 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a0039070 con 0x55d685db94b0 2024-03-06T07:24:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.485+0000 7f61b157c740 1 -- 172.21.15.1:0/508197572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685db94b0 msgr2=0x55d685db5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.485+0000 7f61b157c740 1 --2- 172.21.15.1:0/508197572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685db94b0 0x55d685db5680 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f61a002c4b0 tx=0x7f61a002e000 comp rx=0 tx=0).stop 2024-03-06T07:24:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.485+0000 7f61b157c740 1 -- 172.21.15.1:0/508197572 shutdown_connections 2024-03-06T07:24:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.485+0000 7f61b157c740 1 --2- 172.21.15.1:0/508197572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685db94b0 0x55d685db5680 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.485+0000 7f61b157c740 1 --2- 172.21.15.1:0/508197572 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685dbb330 0x55d685db8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.485+0000 7f61b157c740 1 --2- 172.21.15.1:0/508197572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbaf50 0x55d685db8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.485+0000 7f61b157c740 1 -- 172.21.15.1:0/508197572 >> 172.21.15.1:0/508197572 conn(0x55d685db77d0 msgr2=0x55d685ddc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.485+0000 7f61b157c740 1 -- 172.21.15.1:0/508197572 shutdown_connections 2024-03-06T07:24:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.485+0000 7f61b157c740 1 -- 172.21.15.1:0/508197572 wait complete. 2024-03-06T07:24:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.486+0000 7f61b157c740 1 -- start start 2024-03-06T07:24:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.486+0000 7f61b157c740 1 Processor -- start 2024-03-06T07:24:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61b157c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685dbaf50 0x55d685ddba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61b157c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbb330 0x55d685ddf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61b157c740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685df39c0 0x55d685df3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61b157c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d685ddd230 con 0x55d685dbaf50 2024-03-06T07:24:30.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61b157c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d685cdbea0 con 0x55d685dbb330 2024-03-06T07:24:30.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61b157c740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d685ddcc70 con 0x55d685df39c0 2024-03-06T07:24:30.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61b157c740 10 client.0 _ll_register_callbacks cb 0x55d685cdb870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:30.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61a7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685dbaf50 0x55d685ddba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61a7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685dbaf50 0x55d685ddba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35362/0 (socket says 172.21.15.1:35362) 2024-03-06T07:24:30.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61a7fff640 1 -- 172.21.15.1:0/1280247426 learned_addr learned my addr 172.21.15.1:0/1280247426 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:30.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61a77fe640 1 --2- 172.21.15.1:0/1280247426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbb330 0x55d685ddf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61acf30640 1 --2- 172.21.15.1:0/1280247426 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685df39c0 0x55d685df3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61a7fff640 1 -- 172.21.15.1:0/1280247426 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685df39c0 msgr2=0x55d685df3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61a7fff640 1 --2- 172.21.15.1:0/1280247426 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685df39c0 0x55d685df3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61a7fff640 1 -- 172.21.15.1:0/1280247426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbb330 msgr2=0x55d685ddf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61a7fff640 1 --2- 172.21.15.1:0/1280247426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbb330 0x55d685ddf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61a7fff640 1 -- 172.21.15.1:0/1280247426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d685cdc270 con 0x55d685dbaf50 2024-03-06T07:24:30.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61a7fff640 1 --2- 172.21.15.1:0/1280247426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685dbaf50 0x55d685ddba60 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f619c003670 tx=0x7f619c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a92adff1d711bcd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f6186ffd640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:24:30.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f6186ffd640 1 -- 172.21.15.1:0/1280247426 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c027070 con 0x55d685dbaf50 2024-03-06T07:24:30.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f6186ffd640 1 -- 172.21.15.1:0/1280247426 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f619c014650 con 0x55d685dbaf50 2024-03-06T07:24:30.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.487+0000 7f61b157c740 1 -- 172.21.15.1:0/1280247426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f61a003a040 con 0x55d685dbaf50 2024-03-06T07:24:30.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.488+0000 7f6186ffd640 1 -- 172.21.15.1:0/1280247426 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) v2 ==== 24+0+0 (secure 0 0 0) 0x7f619c032050 con 0x55d685dbaf50 2024-03-06T07:24:30.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.488+0000 7f61b157c740 10 client.7653 fetch_fsmap learned FSMap version 95 2024-03-06T07:24:30.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.488+0000 7f61b157c740 1 -- 172.21.15.1:0/1280247426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=2+}) v3 -- 0x55d685a342d0 con 0x55d685dbaf50 2024-03-06T07:24:30.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.488+0000 7f6186ffd640 1 -- 172.21.15.1:0/1280247426 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a003a040 con 0x55d685dbaf50 2024-03-06T07:24:30.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.488+0000 7f6186ffd640 1 -- 172.21.15.1:0/1280247426 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 95) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f619c035020 con 0x55d685dbaf50 2024-03-06T07:24:30.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.488+0000 7f61b157c740 10 client.7653 fetch_fsmap finished waiting for FSMap version 95 2024-03-06T07:24:30.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.488+0000 7f61b157c740 10 client.7653 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:24:30.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.488+0000 7f61b157c740 20 client.7653 populate_metadata read hostname 'smithi001' 2024-03-06T07:24:30.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.488+0000 7f61b157c740 1 --2- 172.21.15.1:0/1280247426 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x55d685e14ac0 0x55d685e16eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.488+0000 7f61b157c740 4 client.7653 mds_command: new command op to 6122 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:24:30.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.488+0000 7f61b157c740 1 -- 172.21.15.1:0/1280247426 --> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d685d2cbf0 con 0x55d685e14ac0 2024-03-06T07:24:30.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.488+0000 7f61a77fe640 1 --2- 172.21.15.1:0/1280247426 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x55d685e14ac0 0x55d685e16eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.489+0000 7f6186ffd640 10 client.7653 ms_handle_connect on v2:172.21.15.98:6835/2632566151 2024-03-06T07:24:30.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.488+0000 7f61a77fe640 1 --2- 172.21.15.1:0/1280247426 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x55d685e14ac0 0x55d685e16eb0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=241e09bbcfaa6b0b server_cookie=6eabcdf4680f5ff1 in_seq=0 out_seq=0 2024-03-06T07:24:30.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.489+0000 7f6186ffd640 1 -- 172.21.15.1:0/1280247426 <== mds.0 v2:172.21.15.98:6835/2632566151 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d685d2cbf0 con 0x55d685e14ac0 2024-03-06T07:24:30.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.489+0000 7f6186ffd640 10 client.7653 handle_command_reply: tid=0 2024-03-06T07:24:30.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.489+0000 7f61b157c740 1 client.7653 shutdown 2024-03-06T07:24:30.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.489+0000 7f61b157c740 1 -- 172.21.15.1:0/1280247426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685dbaf50 msgr2=0x55d685ddba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.489+0000 7f61b157c740 1 --2- 172.21.15.1:0/1280247426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685dbaf50 0x55d685ddba60 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f619c003670 tx=0x7f619c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.489+0000 7f61b157c740 1 -- 172.21.15.1:0/1280247426 shutdown_connections 2024-03-06T07:24:30.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.489+0000 7f61b157c740 1 --2- 172.21.15.1:0/1280247426 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x55d685e14ac0 0x55d685e16eb0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.489+0000 7f61b157c740 1 --2- 172.21.15.1:0/1280247426 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685df39c0 0x55d685df3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.489+0000 7f61b157c740 1 --2- 172.21.15.1:0/1280247426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbb330 0x55d685ddf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.489+0000 7f6186ffd640 0 client.7653 ms_handle_reset on v2:172.21.15.98:6835/2632566151 2024-03-06T07:24:30.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.489+0000 7f61b157c740 1 --2- 172.21.15.1:0/1280247426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685dbaf50 0x55d685ddba60 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.489+0000 7f61b157c740 1 -- 172.21.15.1:0/1280247426 >> 172.21.15.1:0/1280247426 conn(0x55d685db94b0 msgr2=0x55d685db98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.489+0000 7f61b157c740 1 -- 172.21.15.1:0/1280247426 shutdown_connections 2024-03-06T07:24:30.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.489+0000 7f61b157c740 1 -- 172.21.15.1:0/1280247426 wait complete. 2024-03-06T07:24:30.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.490+0000 7f61b157c740 20 client.7653 trim_cache size 0 max 16384 2024-03-06T07:24:30.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.506+0000 7f61b157c740 1 Processor -- start 2024-03-06T07:24:30.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.506+0000 7f61b157c740 1 -- start start 2024-03-06T07:24:30.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61b157c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbaf50 0x55d685db8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61b157c740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685dbb330 0x55d685db4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61b157c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685df39c0 0x55d685db4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61b157c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a81f73a0 con 0x55d685df39c0 2024-03-06T07:24:30.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61b157c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a81f7a00 con 0x55d685dbaf50 2024-03-06T07:24:30.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61b157c740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a81eeb40 con 0x55d685dbb330 2024-03-06T07:24:30.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61acf30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685df39c0 0x55d685db4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61acf30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685df39c0 0x55d685db4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35370/0 (socket says 172.21.15.1:35370) 2024-03-06T07:24:30.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61acf30640 1 -- 172.21.15.1:0/3318454930 learned_addr learned my addr 172.21.15.1:0/3318454930 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:30.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61a7fff640 1 --2- 172.21.15.1:0/3318454930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbaf50 0x55d685db8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61a77fe640 1 --2- 172.21.15.1:0/3318454930 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685dbb330 0x55d685db4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61acf30640 1 -- 172.21.15.1:0/3318454930 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685dbb330 msgr2=0x55d685db4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61acf30640 1 --2- 172.21.15.1:0/3318454930 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685dbb330 0x55d685db4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61acf30640 1 -- 172.21.15.1:0/3318454930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbaf50 msgr2=0x55d685db8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61acf30640 1 --2- 172.21.15.1:0/3318454930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbaf50 0x55d685db8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61acf30640 1 -- 172.21.15.1:0/3318454930 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d685df41b0 con 0x55d685df39c0 2024-03-06T07:24:30.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61a7fff640 1 --2- 172.21.15.1:0/3318454930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbaf50 0x55d685db8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:30.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f61acf30640 1 --2- 172.21.15.1:0/3318454930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685df39c0 0x55d685db4a50 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x55d685dfa5d0 tx=0x7f61a0031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1c69245605d78d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f618d7fa640 1 -- 172.21.15.1:0/3318454930 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a0039040 con 0x55d685df39c0 2024-03-06T07:24:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f618d7fa640 1 -- 172.21.15.1:0/3318454930 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a004c070 con 0x55d685df39c0 2024-03-06T07:24:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.507+0000 7f618d7fa640 1 -- 172.21.15.1:0/3318454930 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a0003990 con 0x55d685df39c0 2024-03-06T07:24:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.508+0000 7f61b157c740 1 -- 172.21.15.1:0/3318454930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685df39c0 msgr2=0x55d685db4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.508+0000 7f61b157c740 1 --2- 172.21.15.1:0/3318454930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685df39c0 0x55d685db4a50 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x55d685dfa5d0 tx=0x7f61a0031000 comp rx=0 tx=0).stop 2024-03-06T07:24:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.508+0000 7f61b157c740 1 -- 172.21.15.1:0/3318454930 shutdown_connections 2024-03-06T07:24:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.508+0000 7f61b157c740 1 --2- 172.21.15.1:0/3318454930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685df39c0 0x55d685db4a50 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.508+0000 7f61b157c740 1 --2- 172.21.15.1:0/3318454930 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685dbb330 0x55d685db4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.508+0000 7f61b157c740 1 --2- 172.21.15.1:0/3318454930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbaf50 0x55d685db8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.508+0000 7f61b157c740 1 -- 172.21.15.1:0/3318454930 >> 172.21.15.1:0/3318454930 conn(0x55d685db94b0 msgr2=0x55d685ddb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.508+0000 7f61b157c740 1 -- 172.21.15.1:0/3318454930 shutdown_connections 2024-03-06T07:24:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.508+0000 7f61b157c740 1 -- 172.21.15.1:0/3318454930 wait complete. 2024-03-06T07:24:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.508+0000 7f61b157c740 1 -- start start 2024-03-06T07:24:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.509+0000 7f61b157c740 1 Processor -- start 2024-03-06T07:24:30.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.509+0000 7f61b157c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbaf50 0x55d685db5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.509+0000 7f61b157c740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685dbb330 0x55d685dd9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.509+0000 7f61b157c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685df39c0 0x55d685e03ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.509+0000 7f61b157c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d685e0bcf0 con 0x55d685df39c0 2024-03-06T07:24:30.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.509+0000 7f61b157c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a81f6770 con 0x55d685dbaf50 2024-03-06T07:24:30.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.509+0000 7f61b157c740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d685de0030 con 0x55d685dbb330 2024-03-06T07:24:30.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.509+0000 7f61b157c740 10 client.0 _ll_register_callbacks cb 0x7f619c004320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:30.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.509+0000 7f61acf30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685df39c0 0x55d685e03ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.509+0000 7f61acf30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685df39c0 0x55d685e03ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35374/0 (socket says 172.21.15.1:35374) 2024-03-06T07:24:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.509+0000 7f61acf30640 1 -- 172.21.15.1:0/2230622419 learned_addr learned my addr 172.21.15.1:0/2230622419 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.509+0000 7f61a77fe640 1 --2- 172.21.15.1:0/2230622419 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685dbb330 0x55d685dd9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.509+0000 7f61a7fff640 1 --2- 172.21.15.1:0/2230622419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbaf50 0x55d685db5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f61acf30640 1 -- 172.21.15.1:0/2230622419 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685dbb330 msgr2=0x55d685dd9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f61acf30640 1 --2- 172.21.15.1:0/2230622419 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685dbb330 0x55d685dd9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f61acf30640 1 -- 172.21.15.1:0/2230622419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbaf50 msgr2=0x55d685db5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f61acf30640 1 --2- 172.21.15.1:0/2230622419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbaf50 0x55d685db5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f61acf30640 1 -- 172.21.15.1:0/2230622419 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d685afe6b0 con 0x55d685df39c0 2024-03-06T07:24:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f61acf30640 1 --2- 172.21.15.1:0/2230622419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685df39c0 0x55d685e03ce0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f61a004a040 tx=0x7f61a0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d641d815a0b8463 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f6186ffd640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:24:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f6186ffd640 1 -- 172.21.15.1:0/2230622419 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a0039040 con 0x55d685df39c0 2024-03-06T07:24:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f6186ffd640 1 -- 172.21.15.1:0/2230622419 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f61a0003990 con 0x55d685df39c0 2024-03-06T07:24:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f61b157c740 1 -- 172.21.15.1:0/2230622419 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f61a004c070 con 0x55d685df39c0 2024-03-06T07:24:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f6186ffd640 1 -- 172.21.15.1:0/2230622419 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61a0060050 con 0x55d685df39c0 2024-03-06T07:24:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f61b157c740 10 client.7659 fetch_fsmap learned FSMap version 95 2024-03-06T07:24:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f61b157c740 1 -- 172.21.15.1:0/2230622419 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=2+}) v3 -- 0x55d685e1c4b0 con 0x55d685df39c0 2024-03-06T07:24:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f6186ffd640 1 -- 172.21.15.1:0/2230622419 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a004c070 con 0x55d685df39c0 2024-03-06T07:24:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f6186ffd640 1 -- 172.21.15.1:0/2230622419 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 95) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f61a0063020 con 0x55d685df39c0 2024-03-06T07:24:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f61b157c740 10 client.7659 fetch_fsmap finished waiting for FSMap version 95 2024-03-06T07:24:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f61b157c740 10 client.7659 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:24:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f61b157c740 20 client.7659 populate_metadata read hostname 'smithi001' 2024-03-06T07:24:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f61b157c740 1 --2- 172.21.15.1:0/2230622419 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x55d685e65130 0x55d685dbb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f61b157c740 4 client.7659 mds_command: new command op to 6122 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:24:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.510+0000 7f61b157c740 1 -- 172.21.15.1:0/2230622419 --> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d685ddcf40 con 0x55d685e65130 2024-03-06T07:24:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.511+0000 7f61a7fff640 1 --2- 172.21.15.1:0/2230622419 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x55d685e65130 0x55d685dbb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.511+0000 7f6186ffd640 10 client.7659 ms_handle_connect on v2:172.21.15.98:6835/2632566151 2024-03-06T07:24:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.511+0000 7f61a7fff640 1 --2- 172.21.15.1:0/2230622419 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x55d685e65130 0x55d685dbb710 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae2e467d877558a2 server_cookie=2d3433eac7a8b2f3 in_seq=0 out_seq=0 2024-03-06T07:24:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.511+0000 7f6186ffd640 1 -- 172.21.15.1:0/2230622419 <== mds.0 v2:172.21.15.98:6835/2632566151 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55d685ddcf40 con 0x55d685e65130 2024-03-06T07:24:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.511+0000 7f6186ffd640 10 client.7659 handle_command_reply: tid=0 2024-03-06T07:24:30.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.511+0000 7f61b157c740 1 client.7659 shutdown 2024-03-06T07:24:30.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.511+0000 7f61b157c740 1 -- 172.21.15.1:0/2230622419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685df39c0 msgr2=0x55d685e03ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.511+0000 7f61b157c740 1 --2- 172.21.15.1:0/2230622419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685df39c0 0x55d685e03ce0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f61a004a040 tx=0x7f61a0004ab0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.511+0000 7f61b157c740 1 -- 172.21.15.1:0/2230622419 shutdown_connections 2024-03-06T07:24:30.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.511+0000 7f61b157c740 1 --2- 172.21.15.1:0/2230622419 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x55d685e65130 0x55d685dbb710 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.511+0000 7f61b157c740 1 --2- 172.21.15.1:0/2230622419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d685df39c0 0x55d685e03ce0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.511+0000 7f61b157c740 1 --2- 172.21.15.1:0/2230622419 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d685dbb330 0x55d685dd9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.511+0000 7f6186ffd640 0 client.7659 ms_handle_reset on v2:172.21.15.98:6835/2632566151 2024-03-06T07:24:30.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.511+0000 7f61b157c740 1 --2- 172.21.15.1:0/2230622419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d685dbaf50 0x55d685db5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.511+0000 7f61b157c740 1 -- 172.21.15.1:0/2230622419 >> 172.21.15.1:0/2230622419 conn(0x55d685db94b0 msgr2=0x55d685ddd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.511+0000 7f61b157c740 1 -- 172.21.15.1:0/2230622419 shutdown_connections 2024-03-06T07:24:30.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.511+0000 7f61b157c740 1 -- 172.21.15.1:0/2230622419 wait complete. 2024-03-06T07:24:30.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.512+0000 7f61b157c740 20 client.7659 trim_cache size 0 max 16384 2024-03-06T07:24:30.520 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:24:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:24:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:24:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 6122, 2024-03-06T07:24:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:24:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:24:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:24:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 8.6048973059999998, 2024-03-06T07:24:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 95, 2024-03-06T07:24:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 109, 2024-03-06T07:24:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 109, 2024-03-06T07:24:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 24.720742764000001 2024-03-06T07:24:30.521 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:24:30.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.512+0000 7f61ae993640 1 -- 172.21.15.1:0/667359129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f619406b4d0 msgr2=0x7f619406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.512+0000 7f61ae993640 1 --2- 172.21.15.1:0/667359129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f619406b4d0 0x7f619406d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f61a80734b0 tx=0x7f619c009190 comp rx=0 tx=0).stop 2024-03-06T07:24:30.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.512+0000 7f61ae993640 1 -- 172.21.15.1:0/667359129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61a814cdc0 msgr2=0x7f61a81de7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.512+0000 7f61ae993640 1 --2- 172.21.15.1:0/667359129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61a814cdc0 0x7f61a81de7a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f619800c480 tx=0x7f619800c940 comp rx=0 tx=0).stop 2024-03-06T07:24:30.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.512+0000 7f61ae993640 1 -- 172.21.15.1:0/667359129 shutdown_connections 2024-03-06T07:24:30.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.512+0000 7f61ae993640 1 --2- 172.21.15.1:0/667359129 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f619406b4d0 0x7f619406d990 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.512+0000 7f61ae993640 1 --2- 172.21.15.1:0/667359129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61a8156500 0x7f61a814ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.512+0000 7f61ae993640 1 --2- 172.21.15.1:0/667359129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61a814cdc0 0x7f61a81de7a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.512+0000 7f61ae993640 1 --2- 172.21.15.1:0/667359129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a806b510 0x7f61a8152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.512+0000 7f61ae993640 1 -- 172.21.15.1:0/667359129 >> 172.21.15.1:0/667359129 conn(0x7f61a80a9700 msgr2=0x7f61a80c4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.512+0000 7f61ae993640 1 -- 172.21.15.1:0/667359129 shutdown_connections 2024-03-06T07:24:30.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:30.512+0000 7f61ae993640 1 -- 172.21.15.1:0/667359129 wait complete. 2024-03-06T07:24:30.524 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 95, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:24:10.220198+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 95, "ever_allowed_features": 32, "explicitly_allowed_features": 32, "failed": [], "flags": 50, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": true, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6122": { "addr": "172.21.15.98:6837/2632566151", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 2632566151, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2632566151, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6122, "incarnation": 90, "join_fscid": 8, "name": "b", "rank": 0, "state": "up:active", "state_seq": 8 }, "gid_7590": { "addr": "172.21.15.1:6833/550216328", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 550216328, "type": "v2" }, { "addr": "172.21.15.1:6833", "nonce": 550216328, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7590, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:standby-replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 109, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-03-06T07:24:28.938794+0000", "qdb_cluster": [ 6122 ], "qdb_leader": 6122, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6122 } } } ], "standbys": [ { "addr": "172.21.15.1:6835/2244230066", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6834", "nonce": 2244230066, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 2244230066, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6670, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6836/1741783790", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 1741783790, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 1741783790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6682, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:24:30.526 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 8, 'name': 'a', 'state': 'up:standby-replay'}, {'join_fscid': 8, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:24:30.526 INFO:tasks.cephfs.test_failover:current = [{'gid': 6670, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/2244230066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 2244230066}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2244230066}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}, {'gid': 6122, 'name': 'b', 'rank': 0, 'incarnation': 90, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.98:6837/2632566151', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 2632566151}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2632566151}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7590, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.1:6833/550216328', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 550216328}, {'type': 'v1', 'addr': '172.21.15.1:6833', 'nonce': 550216328}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6682, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/1741783790', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 1741783790}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1741783790}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}] 2024-03-06T07:24:30.526 DEBUG:tasks.cephfs.test_failover:8 != -1 2024-03-06T07:24:30.526 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:24:30.636 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:24:31.581 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:31.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:35.527 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:35.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.638+0000 7f14f8b81640 1 Processor -- start 2024-03-06T07:24:35.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.638+0000 7f14f8b81640 1 -- start start 2024-03-06T07:24:35.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.639+0000 7f14f8b81640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f14f40c6b10 0x7f14f4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.639+0000 7f14f8b81640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f14f40bf9d0 0x7f14f40c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.639+0000 7f14f8b81640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f40c2fc0 con 0x7f14f4152650 2024-03-06T07:24:35.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.639+0000 7f14f8b81640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f4155800 con 0x7f14f40bf9d0 2024-03-06T07:24:35.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.639+0000 7f14f8b81640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f4158da0 con 0x7f14f40c6b10 2024-03-06T07:24:35.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.639+0000 7f14f2d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f14f40bf9d0 0x7f14f40c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.639+0000 7f14f259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f14f4152650 0x7f14f4158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59068/0 (socket says 172.21.15.1:59068) 2024-03-06T07:24:35.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.639+0000 7f14f259c640 1 -- 172.21.15.1:0/2970418742 learned_addr learned my addr 172.21.15.1:0/2970418742 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:35.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.639+0000 7f14f1d9b640 1 --2- 172.21.15.1:0/2970418742 >> v2:172.21.15.98:3300/0 conn(0x7f14f40c6b10 0x7f14f4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.640+0000 7f14f2d9d640 1 -- 172.21.15.1:0/2970418742 >> v2:172.21.15.98:3300/0 conn(0x7f14f40c6b10 msgr2=0x7f14f4155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:35.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.640+0000 7f14f2d9d640 1 --2- 172.21.15.1:0/2970418742 >> v2:172.21.15.98:3300/0 conn(0x7f14f40c6b10 0x7f14f4155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.640+0000 7f14f2d9d640 1 -- 172.21.15.1:0/2970418742 >> v1:172.21.15.1:6789/0 conn(0x7f14f4152650 legacy=0x7f14f4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.640+0000 7f14f2d9d640 1 -- 172.21.15.1:0/2970418742 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f40c6690 con 0x7f14f40bf9d0 2024-03-06T07:24:35.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.640+0000 7f14f159a640 1 -- 172.21.15.1:0/2970418742 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331335367 0 0) 0x7f14f40c2fc0 con 0x7f14f4152650 2024-03-06T07:24:35.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.640+0000 7f14f2d9d640 1 --2- 172.21.15.1:0/2970418742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f40bf9d0 0x7f14f40c1460 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f14f40cb8e0 tx=0x7f14e402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b33e1f329a7dfb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.641+0000 7f14f159a640 1 -- 172.21.15.1:0/2970418742 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e4039070 con 0x7f14f40bf9d0 2024-03-06T07:24:35.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.641+0000 7f14f159a640 1 -- 172.21.15.1:0/2970418742 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e402cbf0 con 0x7f14f40bf9d0 2024-03-06T07:24:35.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.641+0000 7f14f159a640 1 -- 172.21.15.1:0/2970418742 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e4034440 con 0x7f14f40bf9d0 2024-03-06T07:24:35.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.641+0000 7f14f8b81640 1 -- 172.21.15.1:0/2970418742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f40bf9d0 msgr2=0x7f14f40c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.641+0000 7f14f8b81640 1 --2- 172.21.15.1:0/2970418742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f40bf9d0 0x7f14f40c1460 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f14f40cb8e0 tx=0x7f14e402be40 comp rx=0 tx=0).stop 2024-03-06T07:24:35.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.641+0000 7f14f8b81640 1 -- 172.21.15.1:0/2970418742 shutdown_connections 2024-03-06T07:24:35.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.641+0000 7f14f8b81640 1 --2- 172.21.15.1:0/2970418742 >> v2:172.21.15.98:3300/0 conn(0x7f14f40c6b10 0x7f14f4155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.641+0000 7f14f8b81640 1 --2- 172.21.15.1:0/2970418742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f40bf9d0 0x7f14f40c1460 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.641+0000 7f14f8b81640 1 -- 172.21.15.1:0/2970418742 >> 172.21.15.1:0/2970418742 conn(0x7f14f4066ee0 msgr2=0x7f14f40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.641+0000 7f14f8b81640 1 -- 172.21.15.1:0/2970418742 shutdown_connections 2024-03-06T07:24:35.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.641+0000 7f14f8b81640 1 -- 172.21.15.1:0/2970418742 wait complete. 2024-03-06T07:24:35.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.643+0000 7f14f8b81640 1 Processor -- start 2024-03-06T07:24:35.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.643+0000 7f14f8b81640 1 -- start start 2024-03-06T07:24:35.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.643+0000 7f14f8b81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14f40bf9d0 0x7f14f414cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.643+0000 7f14f8b81640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f4152650 0x7f14f41eb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.643+0000 7f14f8b81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14f41e1b00 0x7f14f41eb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.643+0000 7f14f8b81640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f41551b0 con 0x7f14f4152650 2024-03-06T07:24:35.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.643+0000 7f14f8b81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f40c6fa0 con 0x7f14f40bf9d0 2024-03-06T07:24:35.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.643+0000 7f14f8b81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f4155490 con 0x7f14f41e1b00 2024-03-06T07:24:35.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.644+0000 7f14f1d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f4152650 0x7f14f41eb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.644+0000 7f14f1d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f4152650 0x7f14f41eb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57892/0 (socket says 172.21.15.1:57892) 2024-03-06T07:24:35.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.644+0000 7f14f1d9b640 1 -- 172.21.15.1:0/2821445674 learned_addr learned my addr 172.21.15.1:0/2821445674 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:35.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.644+0000 7f14f259c640 1 --2- 172.21.15.1:0/2821445674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14f40bf9d0 0x7f14f414cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.644+0000 7f14f2d9d640 1 --2- 172.21.15.1:0/2821445674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14f41e1b00 0x7f14f41eb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.644+0000 7f14f1d9b640 1 -- 172.21.15.1:0/2821445674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14f41e1b00 msgr2=0x7f14f41eb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.644+0000 7f14f1d9b640 1 --2- 172.21.15.1:0/2821445674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14f41e1b00 0x7f14f41eb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.644+0000 7f14f1d9b640 1 -- 172.21.15.1:0/2821445674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14f40bf9d0 msgr2=0x7f14f414cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.644+0000 7f14f1d9b640 1 --2- 172.21.15.1:0/2821445674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14f40bf9d0 0x7f14f414cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.644+0000 7f14f1d9b640 1 -- 172.21.15.1:0/2821445674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f41de700 con 0x7f14f4152650 2024-03-06T07:24:35.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.644+0000 7f14f1d9b640 1 --2- 172.21.15.1:0/2821445674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f4152650 0x7f14f41eb000 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f14e800e9b0 tx=0x7f14e800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.644+0000 7f14e2ffd640 1 -- 172.21.15.1:0/2821445674 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e8017070 con 0x7f14f4152650 2024-03-06T07:24:35.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.644+0000 7f14e2ffd640 1 -- 172.21.15.1:0/2821445674 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e800c9a0 con 0x7f14f4152650 2024-03-06T07:24:35.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.644+0000 7f14f8b81640 1 -- 172.21.15.1:0/2821445674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f4157220 con 0x7f14f4152650 2024-03-06T07:24:35.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.644+0000 7f14e2ffd640 1 -- 172.21.15.1:0/2821445674 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e800cd90 con 0x7f14f4152650 2024-03-06T07:24:35.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.644+0000 7f14f8b81640 1 -- 172.21.15.1:0/2821445674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f41577e0 con 0x7f14f4152650 2024-03-06T07:24:35.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.645+0000 7f14f8b81640 1 -- 172.21.15.1:0/2821445674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f4067fd0 con 0x7f14f4152650 2024-03-06T07:24:35.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.646+0000 7f14e2ffd640 1 -- 172.21.15.1:0/2821445674 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14e801a070 con 0x7f14f4152650 2024-03-06T07:24:35.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.646+0000 7f14e2ffd640 1 --2- 172.21.15.1:0/2821445674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f14d406b4d0 0x7f14d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.647+0000 7f14e2ffd640 1 -- 172.21.15.1:0/2821445674 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f14e808a680 con 0x7f14f4152650 2024-03-06T07:24:35.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.647+0000 7f14f259c640 1 --2- 172.21.15.1:0/2821445674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f14d406b4d0 0x7f14d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.647+0000 7f14f259c640 1 --2- 172.21.15.1:0/2821445674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f14d406b4d0 0x7f14d406d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f14ec000f60 tx=0x7f14ec007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.651+0000 7f14e2ffd640 1 -- 172.21.15.1:0/2821445674 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14e8058d60 con 0x7f14f4152650 2024-03-06T07:24:35.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.874+0000 7f14f8b81640 1 -- 172.21.15.1:0/2821445674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14f41eba80 con 0x7f14f4152650 2024-03-06T07:24:35.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.875+0000 7f14e2ffd640 1 -- 172.21.15.1:0/2821445674 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f14e805ce00 con 0x7f14f4152650 2024-03-06T07:24:35.877 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:35.877 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":98,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1741783790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1741783790},{"type":"v1","addr":"172.21.15.98:6836","nonce":1741783790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7662,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3096543492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3096543492},{"type":"v1","addr":"172.21.15.1:6833","nonce":3096543492}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-06T07:24:10.220198+0000","modified":"2024-03-06T07:24:30.609444+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6122":{"gid":6122,"name":"b","rank":0,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6837/2632566151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2632566151},{"type":"v1","addr":"172.21.15.98:6837","nonce":2632566151}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6670":{"gid":6670,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.1:6835/2244230066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2244230066},{"type":"v1","addr":"172.21.15.1:6835","nonce":2244230066}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6122,"qdb_cluster":[6122]},"id":8}]} 2024-03-06T07:24:35.877 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 98 2024-03-06T07:24:35.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.877+0000 7f14f8b81640 1 -- 172.21.15.1:0/2821445674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f14d406b4d0 msgr2=0x7f14d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.877+0000 7f14f8b81640 1 --2- 172.21.15.1:0/2821445674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f14d406b4d0 0x7f14d406d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f14ec000f60 tx=0x7f14ec007d50 comp rx=0 tx=0).stop 2024-03-06T07:24:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.877+0000 7f14f8b81640 1 -- 172.21.15.1:0/2821445674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f4152650 msgr2=0x7f14f41eb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.877+0000 7f14f8b81640 1 --2- 172.21.15.1:0/2821445674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f4152650 0x7f14f41eb000 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f14e800e9b0 tx=0x7f14e800ee70 comp rx=0 tx=0).stop 2024-03-06T07:24:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.877+0000 7f14f8b81640 1 -- 172.21.15.1:0/2821445674 shutdown_connections 2024-03-06T07:24:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.877+0000 7f14f8b81640 1 --2- 172.21.15.1:0/2821445674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f14d406b4d0 0x7f14d406d990 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.877+0000 7f14f8b81640 1 --2- 172.21.15.1:0/2821445674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14f41e1b00 0x7f14f41eb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.877+0000 7f14f8b81640 1 --2- 172.21.15.1:0/2821445674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f4152650 0x7f14f41eb000 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.877+0000 7f14f8b81640 1 --2- 172.21.15.1:0/2821445674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14f40bf9d0 0x7f14f414cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.877+0000 7f14f8b81640 1 -- 172.21.15.1:0/2821445674 >> 172.21.15.1:0/2821445674 conn(0x7f14f4066ee0 msgr2=0x7f14f40672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.877+0000 7f14f8b81640 1 -- 172.21.15.1:0/2821445674 shutdown_connections 2024-03-06T07:24:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:35.877+0000 7f14f8b81640 1 -- 172.21.15.1:0/2821445674 wait complete. 2024-03-06T07:24:36.292 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-06T07:24:10.220198+0000', 'modified': '2024-03-06T07:24:30.609444+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6122}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6122': {'gid': 6122, 'name': 'b', 'rank': 0, 'incarnation': 90, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.98:6837/2632566151', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 2632566151}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2632566151}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_6670': {'gid': 6670, 'name': 'a', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 7, 'addr': '172.21.15.1:6835/2244230066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 2244230066}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2244230066}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6122, 'qdb_cluster': [6122]} 2024-03-06T07:24:36.292 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:24:36.293 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:24:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.398+0000 7f7000b3e640 1 Processor -- start 2024-03-06T07:24:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.398+0000 7f7000b3e640 1 -- start start 2024-03-06T07:24:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.399+0000 7f7000b3e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6ffc0c8cd0 0x7f6ffc0c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.399+0000 7f7000b3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ffc0c3260 0x7f6ffc155b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.399+0000 7f7000b3e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ffc0c2b90 con 0x7f6ffc0c7560 2024-03-06T07:24:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.399+0000 7f7000b3e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ffc0c37b0 con 0x7f6ffc0c8cd0 2024-03-06T07:24:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.399+0000 7f7000b3e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc0c3a90 con 0x7f6ffc0c3260 2024-03-06T07:24:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.399+0000 7f6ffad9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6ffc0c7560 0x7f6ffc159010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59080/0 (socket says 172.21.15.1:59080) 2024-03-06T07:24:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.399+0000 7f6ffad9d640 1 -- 172.21.15.1:0/1517176568 learned_addr learned my addr 172.21.15.1:0/1517176568 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.399+0000 7f6ffa59c640 1 --2- 172.21.15.1:0/1517176568 >> v2:172.21.15.98:3300/0 conn(0x7f6ffc0c8cd0 0x7f6ffc0c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.399+0000 7f6ff9d9b640 1 --2- 172.21.15.1:0/1517176568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ffc0c3260 0x7f6ffc155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.399+0000 7f6ff959a640 1 -- 172.21.15.1:0/1517176568 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503815609 0 0) 0x7f6ffc0c2b90 con 0x7f6ffc0c7560 2024-03-06T07:24:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.399+0000 7f6ff959a640 1 -- 172.21.15.1:0/1517176568 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe4003520 con 0x7f6ffc0c7560 2024-03-06T07:24:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.399+0000 7f6ffa59c640 1 -- 172.21.15.1:0/1517176568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ffc0c3260 msgr2=0x7f6ffc155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.399+0000 7f6ffa59c640 1 --2- 172.21.15.1:0/1517176568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ffc0c3260 0x7f6ffc155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.399+0000 7f6ffa59c640 1 -- 172.21.15.1:0/1517176568 >> v1:172.21.15.1:6789/0 conn(0x7f6ffc0c7560 legacy=0x7f6ffc159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.399+0000 7f6ffa59c640 1 -- 172.21.15.1:0/1517176568 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc158810 con 0x7f6ffc0c8cd0 2024-03-06T07:24:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.399+0000 7f6ff9d9b640 1 --2- 172.21.15.1:0/1517176568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ffc0c3260 0x7f6ffc155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.400+0000 7f6ffa59c640 1 --2- 172.21.15.1:0/1517176568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ffc0c8cd0 0x7f6ffc0c90b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40097a0 tx=0x7f6ff402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1f521cec12b2482 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.400+0000 7f6ff959a640 1 -- 172.21.15.1:0/1517176568 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff40315a0 con 0x7f6ffc0c8cd0 2024-03-06T07:24:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.400+0000 7f6ff959a640 1 -- 172.21.15.1:0/1517176568 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff4037070 con 0x7f6ffc0c8cd0 2024-03-06T07:24:36.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.400+0000 7f6ff959a640 1 -- 172.21.15.1:0/1517176568 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4031940 con 0x7f6ffc0c8cd0 2024-03-06T07:24:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.401+0000 7f7000b3e640 1 -- 172.21.15.1:0/1517176568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ffc0c8cd0 msgr2=0x7f6ffc0c90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.401+0000 7f7000b3e640 1 --2- 172.21.15.1:0/1517176568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ffc0c8cd0 0x7f6ffc0c90b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40097a0 tx=0x7f6ff402b840 comp rx=0 tx=0).stop 2024-03-06T07:24:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.401+0000 7f7000b3e640 1 -- 172.21.15.1:0/1517176568 shutdown_connections 2024-03-06T07:24:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.401+0000 7f7000b3e640 1 --2- 172.21.15.1:0/1517176568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ffc0c3260 0x7f6ffc155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.401+0000 7f7000b3e640 1 --2- 172.21.15.1:0/1517176568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ffc0c8cd0 0x7f6ffc0c90b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.401+0000 7f7000b3e640 1 -- 172.21.15.1:0/1517176568 >> 172.21.15.1:0/1517176568 conn(0x7f6ffc06b3e0 msgr2=0x7f6ffc06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.401+0000 7f7000b3e640 1 -- 172.21.15.1:0/1517176568 shutdown_connections 2024-03-06T07:24:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.401+0000 7f7000b3e640 1 -- 172.21.15.1:0/1517176568 wait complete. 2024-03-06T07:24:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.402+0000 7f7000b3e640 1 Processor -- start 2024-03-06T07:24:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.402+0000 7f7000b3e640 1 -- start start 2024-03-06T07:24:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.403+0000 7f7000b3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ffc0c3260 0x7f6ffc14e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.403+0000 7f7000b3e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ffc0c7560 0x7f6ffc1ee040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.403+0000 7f7000b3e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ffc0c8cd0 0x7f6ffc0a9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.403+0000 7f7000b3e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc0c8560 con 0x7f6ffc0c8cd0 2024-03-06T07:24:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.403+0000 7f7000b3e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc0c7a10 con 0x7f6ffc0c7560 2024-03-06T07:24:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.403+0000 7f7000b3e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc0c86e0 con 0x7f6ffc0c3260 2024-03-06T07:24:36.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.403+0000 7f6ffad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ffc0c8cd0 0x7f6ffc0a9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.403+0000 7f6ffad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ffc0c8cd0 0x7f6ffc0a9280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57906/0 (socket says 172.21.15.1:57906) 2024-03-06T07:24:36.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.403+0000 7f6ffad9d640 1 -- 172.21.15.1:0/1530182142 learned_addr learned my addr 172.21.15.1:0/1530182142 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:36.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.403+0000 7f6ffa59c640 1 --2- 172.21.15.1:0/1530182142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ffc0c3260 0x7f6ffc14e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.403+0000 7f6ff9d9b640 1 --2- 172.21.15.1:0/1530182142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ffc0c7560 0x7f6ffc1ee040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.403+0000 7f6ffad9d640 1 -- 172.21.15.1:0/1530182142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ffc0c3260 msgr2=0x7f6ffc14e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.403+0000 7f6ffad9d640 1 --2- 172.21.15.1:0/1530182142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ffc0c3260 0x7f6ffc14e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.403+0000 7f6ffad9d640 1 -- 172.21.15.1:0/1530182142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ffc0c7560 msgr2=0x7f6ffc1ee040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.403+0000 7f6ffad9d640 1 --2- 172.21.15.1:0/1530182142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ffc0c7560 0x7f6ffc1ee040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.403+0000 7f6ffad9d640 1 -- 172.21.15.1:0/1530182142 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc1e2280 con 0x7f6ffc0c8cd0 2024-03-06T07:24:36.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.404+0000 7f6ffad9d640 1 --2- 172.21.15.1:0/1530182142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ffc0c8cd0 0x7f6ffc0a9280 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00cb40 tx=0x7f6fec00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.404+0000 7f6feaffd640 1 -- 172.21.15.1:0/1530182142 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec017070 con 0x7f6ffc0c8cd0 2024-03-06T07:24:36.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.404+0000 7f6feaffd640 1 -- 172.21.15.1:0/1530182142 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fec00aca0 con 0x7f6ffc0c8cd0 2024-03-06T07:24:36.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.404+0000 7f7000b3e640 1 -- 172.21.15.1:0/1530182142 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ffc1e2520 con 0x7f6ffc0c8cd0 2024-03-06T07:24:36.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.404+0000 7f6feaffd640 1 -- 172.21.15.1:0/1530182142 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec012620 con 0x7f6ffc0c8cd0 2024-03-06T07:24:36.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.404+0000 7f7000b3e640 1 -- 172.21.15.1:0/1530182142 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ffc1ee580 con 0x7f6ffc0c8cd0 2024-03-06T07:24:36.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.405+0000 7f6feaffd640 1 -- 172.21.15.1:0/1530182142 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fec01a070 con 0x7f6ffc0c8cd0 2024-03-06T07:24:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.406+0000 7f6feaffd640 1 --2- 172.21.15.1:0/1530182142 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6fdc06b4d0 0x7f6fdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.406+0000 7f6feaffd640 1 -- 172.21.15.1:0/1530182142 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f6fec08a140 con 0x7f6ffc0c8cd0 2024-03-06T07:24:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.406+0000 7f6ffa59c640 1 --2- 172.21.15.1:0/1530182142 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6fdc06b4d0 0x7f6fdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.406+0000 7f6ffa59c640 1 --2- 172.21.15.1:0/1530182142 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6fdc06b4d0 0x7f6fdc06d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4036000 tx=0x7f6ff403d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.407+0000 7f7003801740 1 Processor -- start 2024-03-06T07:24:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.407+0000 7f7003801740 1 -- start start 2024-03-06T07:24:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.407+0000 7f7003801740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111ce7f50 0x563111ce5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.407+0000 7f7003801740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111ce8330 0x563111ce5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.407+0000 7f7003801740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce64b0 0x563111ce2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.407+0000 7f7003801740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x563111c4c6b0 con 0x563111ce64b0 2024-03-06T07:24:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.407+0000 7f7003801740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x563111bf5a60 con 0x563111ce7f50 2024-03-06T07:24:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.407+0000 7f7003801740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x563111c48750 con 0x563111ce8330 2024-03-06T07:24:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.408+0000 7f6ffad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce64b0 0x563111ce2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.408+0000 7f6ffa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111ce7f50 0x563111ce5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.408+0000 7f6ffad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce64b0 0x563111ce2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57914/0 (socket says 172.21.15.1:57914) 2024-03-06T07:24:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.408+0000 7f6ffad9d640 1 -- 172.21.15.1:0/2753957018 learned_addr learned my addr 172.21.15.1:0/2753957018 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.408+0000 7f6ff9d9b640 1 --2- 172.21.15.1:0/2753957018 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111ce8330 0x563111ce5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.408+0000 7f6ffad9d640 1 -- 172.21.15.1:0/2753957018 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111ce8330 msgr2=0x563111ce5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.408+0000 7f6ffad9d640 1 --2- 172.21.15.1:0/2753957018 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111ce8330 0x563111ce5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.408+0000 7f6ffad9d640 1 -- 172.21.15.1:0/2753957018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111ce7f50 msgr2=0x563111ce5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.408+0000 7f6ffad9d640 1 --2- 172.21.15.1:0/2753957018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111ce7f50 0x563111ce5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.408+0000 7f6ffad9d640 1 -- 172.21.15.1:0/2753957018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563111c13f30 con 0x563111ce64b0 2024-03-06T07:24:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.408+0000 7f6ffa59c640 1 --2- 172.21.15.1:0/2753957018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111ce7f50 0x563111ce5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.408+0000 7f6ffad9d640 1 --2- 172.21.15.1:0/2753957018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce64b0 0x563111ce2680 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0608b0 tx=0x7f6fec05c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27010010c9532193 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.408+0000 7f6fdbfff640 1 -- 172.21.15.1:0/2753957018 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec054df0 con 0x563111ce64b0 2024-03-06T07:24:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.408+0000 7f6fdbfff640 1 -- 172.21.15.1:0/2753957018 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fec060950 con 0x563111ce64b0 2024-03-06T07:24:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.408+0000 7f6fdbfff640 1 -- 172.21.15.1:0/2753957018 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec054f90 con 0x563111ce64b0 2024-03-06T07:24:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.409+0000 7f7003801740 1 -- 172.21.15.1:0/2753957018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce64b0 msgr2=0x563111ce2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.409+0000 7f7003801740 1 --2- 172.21.15.1:0/2753957018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce64b0 0x563111ce2680 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0608b0 tx=0x7f6fec05c820 comp rx=0 tx=0).stop 2024-03-06T07:24:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.409+0000 7f7003801740 1 -- 172.21.15.1:0/2753957018 shutdown_connections 2024-03-06T07:24:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.409+0000 7f7003801740 1 --2- 172.21.15.1:0/2753957018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce64b0 0x563111ce2680 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.409+0000 7f7003801740 1 --2- 172.21.15.1:0/2753957018 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111ce8330 0x563111ce5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.409+0000 7f7003801740 1 --2- 172.21.15.1:0/2753957018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111ce7f50 0x563111ce5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.409+0000 7f7003801740 1 -- 172.21.15.1:0/2753957018 >> 172.21.15.1:0/2753957018 conn(0x563111ce47d0 msgr2=0x563111d093c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.409+0000 7f7003801740 1 -- 172.21.15.1:0/2753957018 shutdown_connections 2024-03-06T07:24:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.409+0000 7f7003801740 1 -- 172.21.15.1:0/2753957018 wait complete. 2024-03-06T07:24:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.409+0000 7f7003801740 1 -- start start 2024-03-06T07:24:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.410+0000 7f7003801740 1 Processor -- start 2024-03-06T07:24:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.410+0000 7f7003801740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111ce7f50 0x563111d34b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.410+0000 7f7003801740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111ce8330 0x563111d0e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.410+0000 7f7003801740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111d05680 0x563111d0cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.410+0000 7f7003801740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x563111d0a230 con 0x563111d05680 2024-03-06T07:24:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.410+0000 7f7003801740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x563111c08ea0 con 0x563111ce8330 2024-03-06T07:24:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.410+0000 7f7003801740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x563111d09c70 con 0x563111ce7f50 2024-03-06T07:24:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f7003801740 10 client.0 _ll_register_callbacks cb 0x563111c08870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f6ffad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111d05680 0x563111d0cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f6ffad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111d05680 0x563111d0cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57926/0 (socket says 172.21.15.1:57926) 2024-03-06T07:24:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f6ffad9d640 1 -- 172.21.15.1:0/3977827658 learned_addr learned my addr 172.21.15.1:0/3977827658 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f6ff9d9b640 1 --2- 172.21.15.1:0/3977827658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111ce8330 0x563111d0e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f6ffa59c640 1 --2- 172.21.15.1:0/3977827658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111ce7f50 0x563111d34b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f6ffad9d640 1 -- 172.21.15.1:0/3977827658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111ce7f50 msgr2=0x563111d34b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f6ffad9d640 1 --2- 172.21.15.1:0/3977827658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111ce7f50 0x563111d34b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f6ffad9d640 1 -- 172.21.15.1:0/3977827658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111ce8330 msgr2=0x563111d0e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f6ffad9d640 1 --2- 172.21.15.1:0/3977827658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111ce8330 0x563111d0e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f6ffad9d640 1 -- 172.21.15.1:0/3977827658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563111c09270 con 0x563111d05680 2024-03-06T07:24:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f6ff9d9b640 1 --2- 172.21.15.1:0/3977827658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111ce8330 0x563111d0e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f6ffad9d640 1 --2- 172.21.15.1:0/3977827658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111d05680 0x563111d0cf00 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0608b0 tx=0x7f6fec001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca43662cf4f513 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f6fd97fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:24:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f6fd97fa640 1 -- 172.21.15.1:0/3977827658 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec086060 con 0x563111d05680 2024-03-06T07:24:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f6fd97fa640 1 -- 172.21.15.1:0/3977827658 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f6fec004b60 con 0x563111d05680 2024-03-06T07:24:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f7003801740 1 -- 172.21.15.1:0/3977827658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6fec060950 con 0x563111d05680 2024-03-06T07:24:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f6fd97fa640 1 -- 172.21.15.1:0/3977827658 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fec004d60 con 0x563111d05680 2024-03-06T07:24:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f7003801740 10 client.7683 fetch_fsmap learned FSMap version 98 2024-03-06T07:24:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f7003801740 1 -- 172.21.15.1:0/3977827658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) v3 -- 0x5631119612d0 con 0x563111d05680 2024-03-06T07:24:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.411+0000 7f6fd97fa640 1 -- 172.21.15.1:0/3977827658 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fec060950 con 0x563111d05680 2024-03-06T07:24:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.412+0000 7f6fd97fa640 1 -- 172.21.15.1:0/3977827658 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 98) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f6fec057bd0 con 0x563111d05680 2024-03-06T07:24:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.412+0000 7f7003801740 10 client.7683 fetch_fsmap finished waiting for FSMap version 98 2024-03-06T07:24:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.412+0000 7f7003801740 10 client.7683 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:24:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.412+0000 7f7003801740 20 client.7683 populate_metadata read hostname 'smithi001' 2024-03-06T07:24:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.412+0000 7f7003801740 1 --2- 172.21.15.1:0/3977827658 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x563111d0aad0 0x563111d0aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.412+0000 7f7003801740 4 client.7683 mds_command: new command op to 6122 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:24:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.412+0000 7f7003801740 1 -- 172.21.15.1:0/3977827658 --> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563111c59bf0 con 0x563111d0aad0 2024-03-06T07:24:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.412+0000 7f6ffa59c640 1 --2- 172.21.15.1:0/3977827658 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x563111d0aad0 0x563111d0aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.412+0000 7f6fd97fa640 10 client.7683 ms_handle_connect on v2:172.21.15.98:6835/2632566151 2024-03-06T07:24:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.412+0000 7f6ffa59c640 1 --2- 172.21.15.1:0/3977827658 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x563111d0aad0 0x563111d0aeb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b52de66adc410fd server_cookie=33a2c13bb5ee479c in_seq=0 out_seq=0 2024-03-06T07:24:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.413+0000 7f6fd97fa640 1 -- 172.21.15.1:0/3977827658 <== mds.0 v2:172.21.15.98:6835/2632566151 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563111c59bf0 con 0x563111d0aad0 2024-03-06T07:24:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.413+0000 7f6fd97fa640 10 client.7683 handle_command_reply: tid=0 2024-03-06T07:24:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.413+0000 7f7003801740 1 client.7683 shutdown 2024-03-06T07:24:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.413+0000 7f7003801740 1 -- 172.21.15.1:0/3977827658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111d05680 msgr2=0x563111d0cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.413+0000 7f7003801740 1 --2- 172.21.15.1:0/3977827658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111d05680 0x563111d0cf00 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0608b0 tx=0x7f6fec001250 comp rx=0 tx=0).stop 2024-03-06T07:24:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.413+0000 7f7003801740 1 -- 172.21.15.1:0/3977827658 shutdown_connections 2024-03-06T07:24:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.413+0000 7f7003801740 1 --2- 172.21.15.1:0/3977827658 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x563111d0aad0 0x563111d0aeb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.413+0000 7f7003801740 1 --2- 172.21.15.1:0/3977827658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111d05680 0x563111d0cf00 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.413+0000 7f7003801740 1 --2- 172.21.15.1:0/3977827658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111ce8330 0x563111d0e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.413+0000 7f6fd97fa640 0 client.7683 ms_handle_reset on v2:172.21.15.98:6835/2632566151 2024-03-06T07:24:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.413+0000 7f7003801740 1 --2- 172.21.15.1:0/3977827658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111ce7f50 0x563111d34b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.413+0000 7f7003801740 1 -- 172.21.15.1:0/3977827658 >> 172.21.15.1:0/3977827658 conn(0x563111ce64b0 msgr2=0x563111ce1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.413+0000 7f7003801740 1 -- 172.21.15.1:0/3977827658 shutdown_connections 2024-03-06T07:24:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.413+0000 7f7003801740 1 -- 172.21.15.1:0/3977827658 wait complete. 2024-03-06T07:24:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.413+0000 7f7003801740 20 client.7683 trim_cache size 0 max 16384 2024-03-06T07:24:36.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.430+0000 7f7003801740 1 Processor -- start 2024-03-06T07:24:36.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.430+0000 7f7003801740 1 -- start start 2024-03-06T07:24:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f7003801740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce7f50 0x563111ce2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f7003801740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111ce8330 0x563111d35260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f7003801740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111d05680 0x563111d357a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f7003801740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc1f7a80 con 0x563111ce7f50 2024-03-06T07:24:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f7003801740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc14e3f0 con 0x563111ce8330 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f7003801740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc1f65c0 con 0x563111d05680 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f6ffa59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce7f50 0x563111ce2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f6ffa59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce7f50 0x563111ce2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57934/0 (socket says 172.21.15.1:57934) 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f6ffa59c640 1 -- 172.21.15.1:0/197870380 learned_addr learned my addr 172.21.15.1:0/197870380 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f6ff9d9b640 1 --2- 172.21.15.1:0/197870380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111ce8330 0x563111d35260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f6ffad9d640 1 --2- 172.21.15.1:0/197870380 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111d05680 0x563111d357a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f6ffa59c640 1 -- 172.21.15.1:0/197870380 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111d05680 msgr2=0x563111d357a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f6ffa59c640 1 --2- 172.21.15.1:0/197870380 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111d05680 0x563111d357a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f6ffa59c640 1 -- 172.21.15.1:0/197870380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111ce8330 msgr2=0x563111d35260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f6ffa59c640 1 --2- 172.21.15.1:0/197870380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111ce8330 0x563111d35260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f6ffa59c640 1 -- 172.21.15.1:0/197870380 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563111a2b6b0 con 0x563111ce7f50 2024-03-06T07:24:36.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f6ffa59c640 1 --2- 172.21.15.1:0/197870380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce7f50 0x563111ce2f90 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f6ff402b880 tx=0x7f6ff4030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a74dbd932623d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f6fdb7fe640 1 -- 172.21.15.1:0/197870380 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff405a070 con 0x563111ce7f50 2024-03-06T07:24:36.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f6fdb7fe640 1 -- 172.21.15.1:0/197870380 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff4037040 con 0x563111ce7f50 2024-03-06T07:24:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.431+0000 7f6fdb7fe640 1 -- 172.21.15.1:0/197870380 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4055770 con 0x563111ce7f50 2024-03-06T07:24:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.432+0000 7f7003801740 1 -- 172.21.15.1:0/197870380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce7f50 msgr2=0x563111ce2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.432+0000 7f7003801740 1 --2- 172.21.15.1:0/197870380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce7f50 0x563111ce2f90 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f6ff402b880 tx=0x7f6ff4030000 comp rx=0 tx=0).stop 2024-03-06T07:24:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.432+0000 7f7003801740 1 -- 172.21.15.1:0/197870380 shutdown_connections 2024-03-06T07:24:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.432+0000 7f7003801740 1 --2- 172.21.15.1:0/197870380 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111d05680 0x563111d357a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.432+0000 7f7003801740 1 --2- 172.21.15.1:0/197870380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111ce8330 0x563111d35260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.432+0000 7f7003801740 1 --2- 172.21.15.1:0/197870380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce7f50 0x563111ce2f90 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.432+0000 7f7003801740 1 -- 172.21.15.1:0/197870380 >> 172.21.15.1:0/197870380 conn(0x563111ce64b0 msgr2=0x563111ce6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.432+0000 7f7003801740 1 -- 172.21.15.1:0/197870380 shutdown_connections 2024-03-06T07:24:36.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.432+0000 7f7003801740 1 -- 172.21.15.1:0/197870380 wait complete. 2024-03-06T07:24:36.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.433+0000 7f7003801740 1 -- start start 2024-03-06T07:24:36.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.433+0000 7f7003801740 1 Processor -- start 2024-03-06T07:24:36.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f7003801740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce7f50 0x563111d355b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f7003801740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111ce8330 0x563111d972d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f7003801740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111d05680 0x563111d90ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f7003801740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x563111d0dad0 con 0x563111ce7f50 2024-03-06T07:24:36.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f7003801740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x563111c08ea0 con 0x563111d05680 2024-03-06T07:24:36.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f7003801740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x563111d09c70 con 0x563111ce8330 2024-03-06T07:24:36.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f7003801740 10 client.0 _ll_register_callbacks cb 0x7f6fec058780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:36.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f6ffa59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce7f50 0x563111d355b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f6ffa59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce7f50 0x563111d355b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57936/0 (socket says 172.21.15.1:57936) 2024-03-06T07:24:36.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f6ffa59c640 1 -- 172.21.15.1:0/3696985486 learned_addr learned my addr 172.21.15.1:0/3696985486 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:36.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f6ffad9d640 1 --2- 172.21.15.1:0/3696985486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111d05680 0x563111d90ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f6ff9d9b640 1 --2- 172.21.15.1:0/3696985486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111ce8330 0x563111d972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f6ffa59c640 1 -- 172.21.15.1:0/3696985486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111ce8330 msgr2=0x563111d972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f6ffa59c640 1 --2- 172.21.15.1:0/3696985486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111ce8330 0x563111d972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f6ffa59c640 1 -- 172.21.15.1:0/3696985486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111d05680 msgr2=0x563111d90ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f6ffa59c640 1 --2- 172.21.15.1:0/3696985486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111d05680 0x563111d90ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f6ffa59c640 1 -- 172.21.15.1:0/3696985486 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563111d06730 con 0x563111ce7f50 2024-03-06T07:24:36.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f6ffa59c640 1 --2- 172.21.15.1:0/3696985486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce7f50 0x563111d355b0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4058040 tx=0x7f6ff4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c1c6401a76353a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f6fd97fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:24:36.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f6fd97fa640 1 -- 172.21.15.1:0/3696985486 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff405a070 con 0x563111ce7f50 2024-03-06T07:24:36.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f6fd97fa640 1 -- 172.21.15.1:0/3696985486 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f6ff4061ca0 con 0x563111ce7f50 2024-03-06T07:24:36.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f7003801740 1 -- 172.21.15.1:0/3696985486 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6ff4037040 con 0x563111ce7f50 2024-03-06T07:24:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.434+0000 7f6fd97fa640 1 -- 172.21.15.1:0/3696985486 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ff4068050 con 0x563111ce7f50 2024-03-06T07:24:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.435+0000 7f7003801740 10 client.7689 fetch_fsmap learned FSMap version 98 2024-03-06T07:24:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.435+0000 7f7003801740 1 -- 172.21.15.1:0/3696985486 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) v3 -- 0x563111d0b120 con 0x563111ce7f50 2024-03-06T07:24:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.435+0000 7f6fd97fa640 1 -- 172.21.15.1:0/3696985486 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff4037040 con 0x563111ce7f50 2024-03-06T07:24:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.435+0000 7f6fd97fa640 1 -- 172.21.15.1:0/3696985486 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 98) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f6ff406b020 con 0x563111ce7f50 2024-03-06T07:24:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.435+0000 7f7003801740 10 client.7689 fetch_fsmap finished waiting for FSMap version 98 2024-03-06T07:24:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.435+0000 7f7003801740 10 client.7689 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:24:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.435+0000 7f7003801740 20 client.7689 populate_metadata read hostname 'smithi001' 2024-03-06T07:24:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.435+0000 7f7003801740 1 --2- 172.21.15.1:0/3696985486 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x563111ce5370 0x563111ce5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.435+0000 7f7003801740 4 client.7689 mds_command: new command op to 6122 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:24:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.435+0000 7f7003801740 1 -- 172.21.15.1:0/3696985486 --> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563111c53f40 con 0x563111ce5370 2024-03-06T07:24:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.435+0000 7f6ff9d9b640 1 --2- 172.21.15.1:0/3696985486 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x563111ce5370 0x563111ce5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.435+0000 7f6fd97fa640 10 client.7689 ms_handle_connect on v2:172.21.15.98:6835/2632566151 2024-03-06T07:24:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.435+0000 7f6ff9d9b640 1 --2- 172.21.15.1:0/3696985486 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x563111ce5370 0x563111ce5750 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b0ff001fa6dca95 server_cookie=8869f209257ce912 in_seq=0 out_seq=0 2024-03-06T07:24:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.435+0000 7f6fd97fa640 1 -- 172.21.15.1:0/3696985486 <== mds.0 v2:172.21.15.98:6835/2632566151 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x563111c53f40 con 0x563111ce5370 2024-03-06T07:24:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f6fd97fa640 10 client.7689 handle_command_reply: tid=0 2024-03-06T07:24:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7003801740 1 client.7689 shutdown 2024-03-06T07:24:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7003801740 1 -- 172.21.15.1:0/3696985486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce7f50 msgr2=0x563111d355b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7003801740 1 --2- 172.21.15.1:0/3696985486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce7f50 0x563111d355b0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4058040 tx=0x7f6ff4004730 comp rx=0 tx=0).stop 2024-03-06T07:24:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7003801740 1 -- 172.21.15.1:0/3696985486 shutdown_connections 2024-03-06T07:24:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7003801740 1 --2- 172.21.15.1:0/3696985486 >> [v2:172.21.15.98:6835/2632566151,v1:172.21.15.98:6837/2632566151] conn(0x563111ce5370 0x563111ce5750 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7003801740 1 --2- 172.21.15.1:0/3696985486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563111d05680 0x563111d90ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7003801740 1 --2- 172.21.15.1:0/3696985486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x563111ce8330 0x563111d972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f6fd97fa640 0 client.7689 ms_handle_reset on v2:172.21.15.98:6835/2632566151 2024-03-06T07:24:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7003801740 1 --2- 172.21.15.1:0/3696985486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563111ce7f50 0x563111d355b0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7003801740 1 -- 172.21.15.1:0/3696985486 >> 172.21.15.1:0/3696985486 conn(0x563111ce64b0 msgr2=0x563111d09150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7003801740 1 -- 172.21.15.1:0/3696985486 shutdown_connections 2024-03-06T07:24:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7003801740 1 -- 172.21.15.1:0/3696985486 wait complete. 2024-03-06T07:24:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7003801740 20 client.7689 trim_cache size 0 max 16384 2024-03-06T07:24:36.454 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:24:36.454 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:24:36.454 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:24:36.454 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 6122, 2024-03-06T07:24:36.455 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:24:36.455 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:24:36.455 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:24:36.455 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 14.529261959999999, 2024-03-06T07:24:36.455 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 97, 2024-03-06T07:24:36.455 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 109, 2024-03-06T07:24:36.455 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 109, 2024-03-06T07:24:36.455 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 30.645107025000001 2024-03-06T07:24:36.455 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:24:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7000b3e640 1 -- 172.21.15.1:0/1530182142 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6fdc06b4d0 msgr2=0x7f6fdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7000b3e640 1 --2- 172.21.15.1:0/1530182142 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6fdc06b4d0 0x7f6fdc06d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4036000 tx=0x7f6ff403d040 comp rx=0 tx=0).stop 2024-03-06T07:24:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7000b3e640 1 -- 172.21.15.1:0/1530182142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ffc0c8cd0 msgr2=0x7f6ffc0a9280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7000b3e640 1 --2- 172.21.15.1:0/1530182142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ffc0c8cd0 0x7f6ffc0a9280 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00cb40 tx=0x7f6fec00a060 comp rx=0 tx=0).stop 2024-03-06T07:24:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7000b3e640 1 -- 172.21.15.1:0/1530182142 shutdown_connections 2024-03-06T07:24:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7000b3e640 1 --2- 172.21.15.1:0/1530182142 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6fdc06b4d0 0x7f6fdc06d990 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7000b3e640 1 --2- 172.21.15.1:0/1530182142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ffc0c8cd0 0x7f6ffc0a9280 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7000b3e640 1 --2- 172.21.15.1:0/1530182142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ffc0c7560 0x7f6ffc1ee040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7000b3e640 1 --2- 172.21.15.1:0/1530182142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ffc0c3260 0x7f6ffc14e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7000b3e640 1 -- 172.21.15.1:0/1530182142 >> 172.21.15.1:0/1530182142 conn(0x7f6ffc06b3e0 msgr2=0x7f6ffc156ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.436+0000 7f7000b3e640 1 -- 172.21.15.1:0/1530182142 shutdown_connections 2024-03-06T07:24:36.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.437+0000 7f7000b3e640 1 -- 172.21.15.1:0/1530182142 wait complete. 2024-03-06T07:24:36.458 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 98, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:24:10.220198+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 97, "ever_allowed_features": 32, "explicitly_allowed_features": 32, "failed": [], "flags": 50, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": true, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6122": { "addr": "172.21.15.98:6837/2632566151", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 2632566151, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2632566151, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6122, "incarnation": 90, "join_fscid": 8, "name": "b", "rank": 0, "state": "up:active", "state_seq": 8 }, "gid_6670": { "addr": "172.21.15.1:6835/2244230066", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6834", "nonce": 2244230066, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 2244230066, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6670, "incarnation": 0, "join_fscid": 8, "name": "a", "rank": 0, "state": "up:standby-replay", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-03-06T07:24:30.609444+0000", "qdb_cluster": [ 6122 ], "qdb_leader": 6122, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6122 } } } ], "standbys": [ { "addr": "172.21.15.98:6836/1741783790", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 1741783790, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 1741783790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6682, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6833/3096543492", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 3096543492, "type": "v2" }, { "addr": "172.21.15.1:6833", "nonce": 3096543492, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 98, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7662, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:24:36.459 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 8, 'name': 'a', 'state': 'up:standby-replay'}, {'join_fscid': 8, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:24:36.459 INFO:tasks.cephfs.test_failover:current = [{'gid': 6670, 'name': 'a', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 7, 'addr': '172.21.15.1:6835/2244230066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 2244230066}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2244230066}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6122, 'name': 'b', 'rank': 0, 'incarnation': 90, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.98:6837/2632566151', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 2632566151}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2632566151}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7662, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6833/3096543492', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 3096543492}, {'type': 'v1', 'addr': '172.21.15.1:6833', 'nonce': 3096543492}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 98}, {'gid': 6682, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/1741783790', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 1741783790}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1741783790}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}] 2024-03-06T07:24:36.459 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:24:36.459 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:36.460 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:36.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:36.538 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:36.539 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:24:36.539 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:36.556 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:24:36.556 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:24:36.556 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:36.594 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:24:36.594 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:36.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.694+0000 7fd1f340e640 1 Processor -- start 2024-03-06T07:24:36.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.694+0000 7fd1f340e640 1 -- start start 2024-03-06T07:24:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.695+0000 7fd1f340e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd1ec152650 0x7fd1ec155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.695+0000 7fd1f340e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1ec0c6b10 0x7fd1ec158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.695+0000 7fd1f340e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1ec0bf9d0 0x7fd1ec0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.695+0000 7fd1f340e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1ec155800 con 0x7fd1ec0c6b10 2024-03-06T07:24:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.695+0000 7fd1f340e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1ec0c2fc0 con 0x7fd1ec152650 2024-03-06T07:24:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.695+0000 7fd1f340e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec0c6690 con 0x7fd1ec0bf9d0 2024-03-06T07:24:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.695+0000 7fd1f09a9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1ec0c6b10 0x7fd1ec158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.695+0000 7fd1f09a9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1ec0c6b10 0x7fd1ec158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57938/0 (socket says 172.21.15.1:57938) 2024-03-06T07:24:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.695+0000 7fd1f09a9640 1 -- 172.21.15.1:0/3639384231 learned_addr learned my addr 172.21.15.1:0/3639384231 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.695+0000 7fd1f11aa640 1 --2- 172.21.15.1:0/3639384231 >> v2:172.21.15.98:3300/0 conn(0x7fd1ec152650 0x7fd1ec155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.695+0000 7fd1f19ab640 1 --2- 172.21.15.1:0/3639384231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1ec0bf9d0 0x7fd1ec0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.696+0000 7fd1f09a9640 1 -- 172.21.15.1:0/3639384231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1ec0bf9d0 msgr2=0x7fd1ec0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.696+0000 7fd1f09a9640 1 --2- 172.21.15.1:0/3639384231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1ec0bf9d0 0x7fd1ec0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.696+0000 7fd1f09a9640 1 -- 172.21.15.1:0/3639384231 >> v2:172.21.15.98:3300/0 conn(0x7fd1ec152650 msgr2=0x7fd1ec155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.696+0000 7fd1f09a9640 1 --2- 172.21.15.1:0/3639384231 >> v2:172.21.15.98:3300/0 conn(0x7fd1ec152650 0x7fd1ec155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.696+0000 7fd1f09a9640 1 -- 172.21.15.1:0/3639384231 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1ec0c7280 con 0x7fd1ec0c6b10 2024-03-06T07:24:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.696+0000 7fd1f09a9640 1 --2- 172.21.15.1:0/3639384231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec0c6b10 0x7fd1ec158690 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec0cac90 tx=0x7fd1e002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2589813c36e1b925 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.696+0000 7fd1ebfff640 1 -- 172.21.15.1:0/3639384231 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e002e070 con 0x7fd1ec0c6b10 2024-03-06T07:24:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.697+0000 7fd1ebfff640 1 -- 172.21.15.1:0/3639384231 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e0037070 con 0x7fd1ec0c6b10 2024-03-06T07:24:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.697+0000 7fd1ebfff640 1 -- 172.21.15.1:0/3639384231 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0031a00 con 0x7fd1ec0c6b10 2024-03-06T07:24:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.697+0000 7fd1f340e640 1 -- 172.21.15.1:0/3639384231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec0c6b10 msgr2=0x7fd1ec158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.697+0000 7fd1f340e640 1 --2- 172.21.15.1:0/3639384231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec0c6b10 0x7fd1ec158690 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec0cac90 tx=0x7fd1e002b880 comp rx=0 tx=0).stop 2024-03-06T07:24:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.697+0000 7fd1f340e640 1 -- 172.21.15.1:0/3639384231 shutdown_connections 2024-03-06T07:24:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.697+0000 7fd1f340e640 1 --2- 172.21.15.1:0/3639384231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1ec0bf9d0 0x7fd1ec0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.697+0000 7fd1f340e640 1 --2- 172.21.15.1:0/3639384231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec0c6b10 0x7fd1ec158690 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.697+0000 7fd1f340e640 1 --2- 172.21.15.1:0/3639384231 >> v2:172.21.15.98:3300/0 conn(0x7fd1ec152650 0x7fd1ec155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.697+0000 7fd1f340e640 1 -- 172.21.15.1:0/3639384231 >> 172.21.15.1:0/3639384231 conn(0x7fd1ec066ee0 msgr2=0x7fd1ec0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.697+0000 7fd1f340e640 1 -- 172.21.15.1:0/3639384231 shutdown_connections 2024-03-06T07:24:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.697+0000 7fd1f340e640 1 -- 172.21.15.1:0/3639384231 wait complete. 2024-03-06T07:24:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.698+0000 7fd1f340e640 1 Processor -- start 2024-03-06T07:24:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.698+0000 7fd1f340e640 1 -- start start 2024-03-06T07:24:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.699+0000 7fd1f340e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1ec0bf9d0 0x7fd1ec14d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.699+0000 7fd1f340e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1ec152650 0x7fd1ec141d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.699+0000 7fd1f340e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec154d70 0x7fd1ec1422d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.699+0000 7fd1f340e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec155490 con 0x7fd1ec154d70 2024-03-06T07:24:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.699+0000 7fd1f340e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec1551b0 con 0x7fd1ec0bf9d0 2024-03-06T07:24:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.699+0000 7fd1f340e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec152a30 con 0x7fd1ec152650 2024-03-06T07:24:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.699+0000 7fd1f19ab640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec154d70 0x7fd1ec1422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.699+0000 7fd1f19ab640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec154d70 0x7fd1ec1422d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57946/0 (socket says 172.21.15.1:57946) 2024-03-06T07:24:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.699+0000 7fd1f19ab640 1 -- 172.21.15.1:0/2458052076 learned_addr learned my addr 172.21.15.1:0/2458052076 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.699+0000 7fd1f09a9640 1 --2- 172.21.15.1:0/2458052076 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1ec152650 0x7fd1ec141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.699+0000 7fd1f11aa640 1 --2- 172.21.15.1:0/2458052076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1ec0bf9d0 0x7fd1ec14d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.699+0000 7fd1f19ab640 1 -- 172.21.15.1:0/2458052076 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1ec152650 msgr2=0x7fd1ec141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.699+0000 7fd1f19ab640 1 --2- 172.21.15.1:0/2458052076 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1ec152650 0x7fd1ec141d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.699+0000 7fd1f19ab640 1 -- 172.21.15.1:0/2458052076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1ec0bf9d0 msgr2=0x7fd1ec14d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.699+0000 7fd1f19ab640 1 --2- 172.21.15.1:0/2458052076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1ec0bf9d0 0x7fd1ec14d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.699+0000 7fd1f19ab640 1 -- 172.21.15.1:0/2458052076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1ec148b50 con 0x7fd1ec154d70 2024-03-06T07:24:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.700+0000 7fd1f19ab640 1 --2- 172.21.15.1:0/2458052076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec154d70 0x7fd1ec1422d0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00d600 tx=0x7fd1dc00dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.700+0000 7fd1e9ffb640 1 -- 172.21.15.1:0/2458052076 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc018020 con 0x7fd1ec154d70 2024-03-06T07:24:36.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.700+0000 7fd1e9ffb640 1 -- 172.21.15.1:0/2458052076 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1dc014620 con 0x7fd1ec154d70 2024-03-06T07:24:36.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.700+0000 7fd1f340e640 1 -- 172.21.15.1:0/2458052076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1ec1eede0 con 0x7fd1ec154d70 2024-03-06T07:24:36.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.700+0000 7fd1e9ffb640 1 -- 172.21.15.1:0/2458052076 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc0149f0 con 0x7fd1ec154d70 2024-03-06T07:24:36.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.700+0000 7fd1f340e640 1 -- 172.21.15.1:0/2458052076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1ec142a80 con 0x7fd1ec154d70 2024-03-06T07:24:36.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.701+0000 7fd1f340e640 1 -- 172.21.15.1:0/2458052076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1ec067fd0 con 0x7fd1ec154d70 2024-03-06T07:24:36.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.701+0000 7fd1e9ffb640 1 -- 172.21.15.1:0/2458052076 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1dc01b020 con 0x7fd1ec154d70 2024-03-06T07:24:36.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.702+0000 7fd1e9ffb640 1 --2- 172.21.15.1:0/2458052076 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd1d806b4d0 0x7fd1d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.702+0000 7fd1e9ffb640 1 -- 172.21.15.1:0/2458052076 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd1dc08a400 con 0x7fd1ec154d70 2024-03-06T07:24:36.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.702+0000 7fd1f11aa640 1 --2- 172.21.15.1:0/2458052076 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd1d806b4d0 0x7fd1d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.702+0000 7fd1f11aa640 1 --2- 172.21.15.1:0/2458052076 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd1d806b4d0 0x7fd1d806d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4003b80 tx=0x7fd1e40098d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.706+0000 7fd1e9ffb640 1 -- 172.21.15.1:0/2458052076 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1dc022030 con 0x7fd1ec154d70 2024-03-06T07:24:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.927+0000 7fd1f340e640 1 -- 172.21.15.1:0/2458052076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1ec0b06f0 con 0x7fd1ec154d70 2024-03-06T07:24:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.928+0000 7fd1e9ffb640 1 -- 172.21.15.1:0/2458052076 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4978 (secure 0 0 0) 0x7fd1dc058a50 con 0x7fd1ec154d70 2024-03-06T07:24:36.930 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:36.930 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":98,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1741783790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1741783790},{"type":"v1","addr":"172.21.15.98:6836","nonce":1741783790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7662,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3096543492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3096543492},{"type":"v1","addr":"172.21.15.1:6833","nonce":3096543492}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-06T07:24:10.220198+0000","modified":"2024-03-06T07:24:30.609444+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6122":{"gid":6122,"name":"b","rank":0,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6837/2632566151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2632566151},{"type":"v1","addr":"172.21.15.98:6837","nonce":2632566151}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6670":{"gid":6670,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.1:6835/2244230066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2244230066},{"type":"v1","addr":"172.21.15.1:6835","nonce":2244230066}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6122,"qdb_cluster":[6122]},"id":8}]} 2024-03-06T07:24:36.930 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 98 2024-03-06T07:24:36.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.930+0000 7fd1f340e640 1 -- 172.21.15.1:0/2458052076 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd1d806b4d0 msgr2=0x7fd1d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.930+0000 7fd1f340e640 1 --2- 172.21.15.1:0/2458052076 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd1d806b4d0 0x7fd1d806d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4003b80 tx=0x7fd1e40098d0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.930+0000 7fd1f340e640 1 -- 172.21.15.1:0/2458052076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec154d70 msgr2=0x7fd1ec1422d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.930+0000 7fd1f340e640 1 --2- 172.21.15.1:0/2458052076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec154d70 0x7fd1ec1422d0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00d600 tx=0x7fd1dc00dac0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.930+0000 7fd1f340e640 1 -- 172.21.15.1:0/2458052076 shutdown_connections 2024-03-06T07:24:36.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.930+0000 7fd1f340e640 1 --2- 172.21.15.1:0/2458052076 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd1d806b4d0 0x7fd1d806d990 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.930+0000 7fd1f340e640 1 --2- 172.21.15.1:0/2458052076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec154d70 0x7fd1ec1422d0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.930+0000 7fd1f340e640 1 --2- 172.21.15.1:0/2458052076 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1ec152650 0x7fd1ec141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.930+0000 7fd1f340e640 1 --2- 172.21.15.1:0/2458052076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1ec0bf9d0 0x7fd1ec14d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.930+0000 7fd1f340e640 1 -- 172.21.15.1:0/2458052076 >> 172.21.15.1:0/2458052076 conn(0x7fd1ec066ee0 msgr2=0x7fd1ec0c3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.930+0000 7fd1f340e640 1 -- 172.21.15.1:0/2458052076 shutdown_connections 2024-03-06T07:24:36.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:36.930+0000 7fd1f340e640 1 -- 172.21.15.1:0/2458052076 wait complete. 2024-03-06T07:24:36.943 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:37.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.042+0000 7f7e69e9a640 1 Processor -- start 2024-03-06T07:24:37.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.042+0000 7f7e69e9a640 1 -- start start 2024-03-06T07:24:37.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.043+0000 7f7e69e9a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7e64152730 0x7f7e640c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.043+0000 7f7e69e9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e640c6fd0 0x7f7e640c1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.043+0000 7f7e69e9a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e641588e0 con 0x7f7e640c18d0 2024-03-06T07:24:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.043+0000 7f7e69e9a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e640c1d20 con 0x7f7e64152730 2024-03-06T07:24:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.043+0000 7f7e69e9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e64158bc0 con 0x7f7e640c6fd0 2024-03-06T07:24:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.043+0000 7f7e63fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7e640c18d0 0x7f7e641552e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59086/0 (socket says 172.21.15.1:59086) 2024-03-06T07:24:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.043+0000 7f7e63fff640 1 -- 172.21.15.1:0/660413464 learned_addr learned my addr 172.21.15.1:0/660413464 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.043+0000 7f7e62ffd640 1 --2- 172.21.15.1:0/660413464 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e640c6fd0 0x7f7e640c1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.043+0000 7f7e637fe640 1 --2- 172.21.15.1:0/660413464 >> v2:172.21.15.98:3300/0 conn(0x7f7e64152730 0x7f7e640c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.043+0000 7f7e627fc640 1 -- 172.21.15.1:0/660413464 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035578615 0 0) 0x7f7e641588e0 con 0x7f7e640c18d0 2024-03-06T07:24:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.043+0000 7f7e627fc640 1 -- 172.21.15.1:0/660413464 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e4c003520 con 0x7f7e640c18d0 2024-03-06T07:24:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.044+0000 7f7e637fe640 1 -- 172.21.15.1:0/660413464 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e640c6fd0 msgr2=0x7f7e640c1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.044+0000 7f7e637fe640 1 --2- 172.21.15.1:0/660413464 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e640c6fd0 0x7f7e640c1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.044+0000 7f7e637fe640 1 -- 172.21.15.1:0/660413464 >> v1:172.21.15.1:6789/0 conn(0x7f7e640c18d0 legacy=0x7f7e641552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.044+0000 7f7e637fe640 1 -- 172.21.15.1:0/660413464 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e64158ea0 con 0x7f7e64152730 2024-03-06T07:24:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.044+0000 7f7e62ffd640 1 --2- 172.21.15.1:0/660413464 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e640c6fd0 0x7f7e640c1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.044+0000 7f7e637fe640 1 --2- 172.21.15.1:0/660413464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e64152730 0x7f7e640c6a90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7e540097a0 tx=0x7f7e5402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52002179452c3455 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.044+0000 7f7e627fc640 1 -- 172.21.15.1:0/660413464 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e54038070 con 0x7f7e64152730 2024-03-06T07:24:37.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.045+0000 7f7e627fc640 1 -- 172.21.15.1:0/660413464 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e54031700 con 0x7f7e64152730 2024-03-06T07:24:37.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.045+0000 7f7e627fc640 1 -- 172.21.15.1:0/660413464 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e54031ad0 con 0x7f7e64152730 2024-03-06T07:24:37.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.045+0000 7f7e69e9a640 1 -- 172.21.15.1:0/660413464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e64152730 msgr2=0x7f7e640c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.045+0000 7f7e69e9a640 1 --2- 172.21.15.1:0/660413464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e64152730 0x7f7e640c6a90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7e540097a0 tx=0x7f7e5402b7a0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.045+0000 7f7e69e9a640 1 -- 172.21.15.1:0/660413464 shutdown_connections 2024-03-06T07:24:37.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.045+0000 7f7e69e9a640 1 --2- 172.21.15.1:0/660413464 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e640c6fd0 0x7f7e640c1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.045+0000 7f7e69e9a640 1 --2- 172.21.15.1:0/660413464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e64152730 0x7f7e640c6a90 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.045+0000 7f7e69e9a640 1 -- 172.21.15.1:0/660413464 >> 172.21.15.1:0/660413464 conn(0x7f7e64066d00 msgr2=0x7f7e64067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.045+0000 7f7e69e9a640 1 -- 172.21.15.1:0/660413464 shutdown_connections 2024-03-06T07:24:37.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.045+0000 7f7e69e9a640 1 -- 172.21.15.1:0/660413464 wait complete. 2024-03-06T07:24:37.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.046+0000 7f7e69e9a640 1 Processor -- start 2024-03-06T07:24:37.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.046+0000 7f7e69e9a640 1 -- start start 2024-03-06T07:24:37.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.047+0000 7f7e69e9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e640c18d0 0x7f7e64156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.047+0000 7f7e69e9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e640c6fd0 0x7f7e640c9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.047+0000 7f7e69e9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e64152730 0x7f7e641eb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.047+0000 7f7e69e9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e64152b10 con 0x7f7e640c6fd0 2024-03-06T07:24:37.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.047+0000 7f7e69e9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e64159460 con 0x7f7e640c18d0 2024-03-06T07:24:37.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.047+0000 7f7e69e9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e641559f0 con 0x7f7e64152730 2024-03-06T07:24:37.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.047+0000 7f7e62ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e640c6fd0 0x7f7e640c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.047+0000 7f7e62ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e640c6fd0 0x7f7e640c9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57956/0 (socket says 172.21.15.1:57956) 2024-03-06T07:24:37.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.047+0000 7f7e62ffd640 1 -- 172.21.15.1:0/3728992259 learned_addr learned my addr 172.21.15.1:0/3728992259 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:37.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.047+0000 7f7e637fe640 1 --2- 172.21.15.1:0/3728992259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e640c18d0 0x7f7e64156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.047+0000 7f7e63fff640 1 --2- 172.21.15.1:0/3728992259 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e64152730 0x7f7e641eb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.047+0000 7f7e62ffd640 1 -- 172.21.15.1:0/3728992259 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e64152730 msgr2=0x7f7e641eb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.047+0000 7f7e62ffd640 1 --2- 172.21.15.1:0/3728992259 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e64152730 0x7f7e641eb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.047+0000 7f7e62ffd640 1 -- 172.21.15.1:0/3728992259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e640c18d0 msgr2=0x7f7e64156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.047+0000 7f7e62ffd640 1 --2- 172.21.15.1:0/3728992259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e640c18d0 0x7f7e64156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.047+0000 7f7e62ffd640 1 -- 172.21.15.1:0/3728992259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e640c92b0 con 0x7f7e640c6fd0 2024-03-06T07:24:37.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.048+0000 7f7e62ffd640 1 --2- 172.21.15.1:0/3728992259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e640c6fd0 0x7f7e640c9800 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f7e5800ec10 tx=0x7f7e5800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.048+0000 7f7e4bfff640 1 -- 172.21.15.1:0/3728992259 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e58017070 con 0x7f7e640c6fd0 2024-03-06T07:24:37.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.048+0000 7f7e4bfff640 1 -- 172.21.15.1:0/3728992259 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5800cc30 con 0x7f7e640c6fd0 2024-03-06T07:24:37.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.048+0000 7f7e4bfff640 1 -- 172.21.15.1:0/3728992259 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e58013470 con 0x7f7e640c6fd0 2024-03-06T07:24:37.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.048+0000 7f7e69e9a640 1 -- 172.21.15.1:0/3728992259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e640c8cf0 con 0x7f7e640c6fd0 2024-03-06T07:24:37.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.048+0000 7f7e69e9a640 1 -- 172.21.15.1:0/3728992259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e641eae00 con 0x7f7e640c6fd0 2024-03-06T07:24:37.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.049+0000 7f7e69e9a640 1 -- 172.21.15.1:0/3728992259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e640c6340 con 0x7f7e640c6fd0 2024-03-06T07:24:37.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.049+0000 7f7e4bfff640 1 -- 172.21.15.1:0/3728992259 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e5801a070 con 0x7f7e640c6fd0 2024-03-06T07:24:37.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.050+0000 7f7e4bfff640 1 --2- 172.21.15.1:0/3728992259 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7e5006b500 0x7f7e5006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.050+0000 7f7e4bfff640 1 -- 172.21.15.1:0/3728992259 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f7e5808a5a0 con 0x7f7e640c6fd0 2024-03-06T07:24:37.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.050+0000 7f7e637fe640 1 --2- 172.21.15.1:0/3728992259 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7e5006b500 0x7f7e5006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.050+0000 7f7e637fe640 1 --2- 172.21.15.1:0/3728992259 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7e5006b500 0x7f7e5006d9c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7e54035000 tx=0x7f7e54008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.054+0000 7f7e4bfff640 1 -- 172.21.15.1:0/3728992259 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e58058be0 con 0x7f7e640c6fd0 2024-03-06T07:24:37.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.235+0000 7f7e69e9a640 1 -- 172.21.15.1:0/3728992259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e640c2770 con 0x7f7e640c6fd0 2024-03-06T07:24:37.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.236+0000 7f7e4bfff640 1 -- 172.21.15.1:0/3728992259 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16959 (secure 0 0 0) 0x7f7e5805cc80 con 0x7f7e640c6fd0 2024-03-06T07:24:37.239 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:37.239 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":109,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:24:21.903206+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:08.484521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.184053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/2433923518":"2024-03-07T07:24:21.903185+0000","172.21.15.1:6832/2433923518":"2024-03-07T07:24:21.903185+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.238+0000 7f7e69e9a640 1 -- 172.21.15.1:0/3728992259 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7e5006b500 msgr2=0x7f7e5006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.238+0000 7f7e69e9a640 1 --2- 172.21.15.1:0/3728992259 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7e5006b500 0x7f7e5006d9c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7e54035000 tx=0x7f7e54008030 comp rx=0 tx=0).stop 2024-03-06T07:24:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.238+0000 7f7e69e9a640 1 -- 172.21.15.1:0/3728992259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e640c6fd0 msgr2=0x7f7e640c9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.238+0000 7f7e69e9a640 1 --2- 172.21.15.1:0/3728992259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e640c6fd0 0x7f7e640c9800 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f7e5800ec10 tx=0x7f7e5800c520 comp rx=0 tx=0).stop 2024-03-06T07:24:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.238+0000 7f7e69e9a640 1 -- 172.21.15.1:0/3728992259 shutdown_connections 2024-03-06T07:24:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.238+0000 7f7e69e9a640 1 --2- 172.21.15.1:0/3728992259 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7e5006b500 0x7f7e5006d9c0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.238+0000 7f7e69e9a640 1 --2- 172.21.15.1:0/3728992259 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e64152730 0x7f7e641eb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.238+0000 7f7e69e9a640 1 --2- 172.21.15.1:0/3728992259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e640c6fd0 0x7f7e640c9800 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.238+0000 7f7e69e9a640 1 --2- 172.21.15.1:0/3728992259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e640c18d0 0x7f7e64156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.238+0000 7f7e69e9a640 1 -- 172.21.15.1:0/3728992259 >> 172.21.15.1:0/3728992259 conn(0x7f7e64066d00 msgr2=0x7f7e641519e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.239+0000 7f7e69e9a640 1 -- 172.21.15.1:0/3728992259 shutdown_connections 2024-03-06T07:24:37.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.239+0000 7f7e69e9a640 1 -- 172.21.15.1:0/3728992259 wait complete. 2024-03-06T07:24:37.251 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:08.484521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:09.184053+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:37.252 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.354+0000 7f5c3241c640 1 Processor -- start 2024-03-06T07:24:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.354+0000 7f5c3241c640 1 -- start start 2024-03-06T07:24:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.355+0000 7f5c3241c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c2c152650 0x7f5c2c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.355+0000 7f5c3241c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5c2c0bf9d0 0x7f5c2c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.355+0000 7f5c3241c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c0c2fc0 con 0x7f5c2c0c6b10 2024-03-06T07:24:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.355+0000 7f5c3241c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c2c155800 con 0x7f5c2c0bf9d0 2024-03-06T07:24:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.355+0000 7f5c3241c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c158da0 con 0x7f5c2c152650 2024-03-06T07:24:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.355+0000 7f5c2b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5c2c0c6b10 0x7f5c2c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59094/0 (socket says 172.21.15.1:59094) 2024-03-06T07:24:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.355+0000 7f5c2b7fe640 1 -- 172.21.15.1:0/2569467091 learned_addr learned my addr 172.21.15.1:0/2569467091 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.355+0000 7f5c309b9640 1 --2- 172.21.15.1:0/2569467091 >> v2:172.21.15.98:3300/0 conn(0x7f5c2c0bf9d0 0x7f5c2c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.355+0000 7f5c2bfff640 1 --2- 172.21.15.1:0/2569467091 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c2c152650 0x7f5c2c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.355+0000 7f5c2affd640 1 -- 172.21.15.1:0/2569467091 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516514861 0 0) 0x7f5c2c0c2fc0 con 0x7f5c2c0c6b10 2024-03-06T07:24:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.356+0000 7f5c2affd640 1 -- 172.21.15.1:0/2569467091 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c18003520 con 0x7f5c2c0c6b10 2024-03-06T07:24:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.356+0000 7f5c2bfff640 1 -- 172.21.15.1:0/2569467091 >> v2:172.21.15.98:3300/0 conn(0x7f5c2c0bf9d0 msgr2=0x7f5c2c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.356+0000 7f5c2bfff640 1 --2- 172.21.15.1:0/2569467091 >> v2:172.21.15.98:3300/0 conn(0x7f5c2c0bf9d0 0x7f5c2c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.356+0000 7f5c2bfff640 1 -- 172.21.15.1:0/2569467091 >> v1:172.21.15.1:6789/0 conn(0x7f5c2c0c6b10 legacy=0x7f5c2c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.356+0000 7f5c2bfff640 1 -- 172.21.15.1:0/2569467091 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c0c6690 con 0x7f5c2c152650 2024-03-06T07:24:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.356+0000 7f5c2bfff640 1 --2- 172.21.15.1:0/2569467091 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c2c152650 0x7f5c2c155c90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5c140097a0 tx=0x7f5c1402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2f440fdb7df882d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.357+0000 7f5c2affd640 1 -- 172.21.15.1:0/2569467091 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c140315a0 con 0x7f5c2c152650 2024-03-06T07:24:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.357+0000 7f5c2affd640 1 -- 172.21.15.1:0/2569467091 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c14037070 con 0x7f5c2c152650 2024-03-06T07:24:37.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.357+0000 7f5c2affd640 1 -- 172.21.15.1:0/2569467091 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c14031960 con 0x7f5c2c152650 2024-03-06T07:24:37.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.357+0000 7f5c3241c640 1 -- 172.21.15.1:0/2569467091 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c2c152650 msgr2=0x7f5c2c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.357+0000 7f5c3241c640 1 --2- 172.21.15.1:0/2569467091 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c2c152650 0x7f5c2c155c90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5c140097a0 tx=0x7f5c1402b840 comp rx=0 tx=0).stop 2024-03-06T07:24:37.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.357+0000 7f5c3241c640 1 -- 172.21.15.1:0/2569467091 shutdown_connections 2024-03-06T07:24:37.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.357+0000 7f5c3241c640 1 --2- 172.21.15.1:0/2569467091 >> v2:172.21.15.98:3300/0 conn(0x7f5c2c0bf9d0 0x7f5c2c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.357+0000 7f5c3241c640 1 --2- 172.21.15.1:0/2569467091 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c2c152650 0x7f5c2c155c90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.357+0000 7f5c3241c640 1 -- 172.21.15.1:0/2569467091 >> 172.21.15.1:0/2569467091 conn(0x7f5c2c066ee0 msgr2=0x7f5c2c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.357+0000 7f5c3241c640 1 -- 172.21.15.1:0/2569467091 shutdown_connections 2024-03-06T07:24:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.357+0000 7f5c3241c640 1 -- 172.21.15.1:0/2569467091 wait complete. 2024-03-06T07:24:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.358+0000 7f5c3241c640 1 Processor -- start 2024-03-06T07:24:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.358+0000 7f5c3241c640 1 -- start start 2024-03-06T07:24:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.359+0000 7f5c3241c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c2c0bf9d0 0x7f5c2c14e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.359+0000 7f5c3241c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c2c152650 0x7f5c2c1547f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.359+0000 7f5c3241c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c2c14d860 0x7f5c2c13d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.359+0000 7f5c3241c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c1551b0 con 0x7f5c2c0bf9d0 2024-03-06T07:24:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.359+0000 7f5c3241c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c0c6fa0 con 0x7f5c2c152650 2024-03-06T07:24:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.359+0000 7f5c3241c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c155490 con 0x7f5c2c14d860 2024-03-06T07:24:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.359+0000 7f5c2bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c2c0bf9d0 0x7f5c2c14e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.359+0000 7f5c2bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c2c0bf9d0 0x7f5c2c14e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57968/0 (socket says 172.21.15.1:57968) 2024-03-06T07:24:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.359+0000 7f5c2bfff640 1 -- 172.21.15.1:0/3406156569 learned_addr learned my addr 172.21.15.1:0/3406156569 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.359+0000 7f5c2b7fe640 1 --2- 172.21.15.1:0/3406156569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c2c152650 0x7f5c2c1547f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.359+0000 7f5c309b9640 1 --2- 172.21.15.1:0/3406156569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c2c14d860 0x7f5c2c13d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.359+0000 7f5c2bfff640 1 -- 172.21.15.1:0/3406156569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c2c14d860 msgr2=0x7f5c2c13d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.359+0000 7f5c2bfff640 1 --2- 172.21.15.1:0/3406156569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c2c14d860 0x7f5c2c13d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.359+0000 7f5c2bfff640 1 -- 172.21.15.1:0/3406156569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c2c152650 msgr2=0x7f5c2c1547f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.359+0000 7f5c2bfff640 1 --2- 172.21.15.1:0/3406156569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c2c152650 0x7f5c2c1547f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.359+0000 7f5c2bfff640 1 -- 172.21.15.1:0/3406156569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c14b480 con 0x7f5c2c0bf9d0 2024-03-06T07:24:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.360+0000 7f5c2bfff640 1 --2- 172.21.15.1:0/3406156569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c2c0bf9d0 0x7f5c2c14e460 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f5c140098c0 tx=0x7f5c140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.360+0000 7f5c28ff9640 1 -- 172.21.15.1:0/3406156569 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c14032630 con 0x7f5c2c0bf9d0 2024-03-06T07:24:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.360+0000 7f5c28ff9640 1 -- 172.21.15.1:0/3406156569 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c140327f0 con 0x7f5c2c0bf9d0 2024-03-06T07:24:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.360+0000 7f5c3241c640 1 -- 172.21.15.1:0/3406156569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c2c152ec0 con 0x7f5c2c0bf9d0 2024-03-06T07:24:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.360+0000 7f5c28ff9640 1 -- 172.21.15.1:0/3406156569 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c14032be0 con 0x7f5c2c0bf9d0 2024-03-06T07:24:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.360+0000 7f5c3241c640 1 -- 172.21.15.1:0/3406156569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c2c14e9a0 con 0x7f5c2c0bf9d0 2024-03-06T07:24:37.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.361+0000 7f5c3241c640 1 -- 172.21.15.1:0/3406156569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c2c067fd0 con 0x7f5c2c0bf9d0 2024-03-06T07:24:37.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.361+0000 7f5c28ff9640 1 -- 172.21.15.1:0/3406156569 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c1404e430 con 0x7f5c2c0bf9d0 2024-03-06T07:24:37.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.362+0000 7f5c28ff9640 1 --2- 172.21.15.1:0/3406156569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0c06b4d0 0x7f5c0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.362+0000 7f5c28ff9640 1 -- 172.21.15.1:0/3406156569 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5c1402f030 con 0x7f5c2c0bf9d0 2024-03-06T07:24:37.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.362+0000 7f5c2b7fe640 1 --2- 172.21.15.1:0/3406156569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0c06b4d0 0x7f5c0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.362+0000 7f5c2b7fe640 1 --2- 172.21.15.1:0/3406156569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0c06b4d0 0x7f5c0c06d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c0c5820 tx=0x7f5c1c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.366+0000 7f5c28ff9640 1 -- 172.21.15.1:0/3406156569 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c14040070 con 0x7f5c2c0bf9d0 2024-03-06T07:24:37.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.546+0000 7f5c3241c640 1 -- 172.21.15.1:0/3406156569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5c2c06b500 con 0x7f5c2c0bf9d0 2024-03-06T07:24:37.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.546+0000 7f5c28ff9640 1 -- 172.21.15.1:0/3406156569 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5c1407a6f0 con 0x7f5c2c0bf9d0 2024-03-06T07:24:37.548 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:24:37.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.548+0000 7f5c3241c640 1 -- 172.21.15.1:0/3406156569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0c06b4d0 msgr2=0x7f5c0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.548+0000 7f5c3241c640 1 --2- 172.21.15.1:0/3406156569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0c06b4d0 0x7f5c0c06d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c0c5820 tx=0x7f5c1c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:37.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.548+0000 7f5c3241c640 1 -- 172.21.15.1:0/3406156569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c2c0bf9d0 msgr2=0x7f5c2c14e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.548+0000 7f5c3241c640 1 --2- 172.21.15.1:0/3406156569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c2c0bf9d0 0x7f5c2c14e460 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f5c140098c0 tx=0x7f5c140046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.548+0000 7f5c3241c640 1 -- 172.21.15.1:0/3406156569 shutdown_connections 2024-03-06T07:24:37.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.548+0000 7f5c3241c640 1 --2- 172.21.15.1:0/3406156569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0c06b4d0 0x7f5c0c06d990 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.548+0000 7f5c3241c640 1 --2- 172.21.15.1:0/3406156569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c2c14d860 0x7f5c2c13d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.548+0000 7f5c3241c640 1 --2- 172.21.15.1:0/3406156569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c2c152650 0x7f5c2c1547f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.548+0000 7f5c3241c640 1 --2- 172.21.15.1:0/3406156569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c2c0bf9d0 0x7f5c2c14e460 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.548+0000 7f5c3241c640 1 -- 172.21.15.1:0/3406156569 >> 172.21.15.1:0/3406156569 conn(0x7f5c2c066ee0 msgr2=0x7f5c2c0c5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.548+0000 7f5c3241c640 1 -- 172.21.15.1:0/3406156569 shutdown_connections 2024-03-06T07:24:37.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.548+0000 7f5c3241c640 1 -- 172.21.15.1:0/3406156569 wait complete. 2024-03-06T07:24:37.561 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:37.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.664+0000 7f938f455640 1 Processor -- start 2024-03-06T07:24:37.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.665+0000 7f938f455640 1 -- start start 2024-03-06T07:24:37.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.665+0000 7f938f455640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f93880c90b0 0x7f9388155b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.665+0000 7f938f455640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93880c2b90 con 0x7f93880c8cd0 2024-03-06T07:24:37.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.665+0000 7f938f455640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93880c37b0 con 0x7f93880c3260 2024-03-06T07:24:37.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.665+0000 7f938f455640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93880c9540 con 0x7f93880c90b0 2024-03-06T07:24:37.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.666+0000 7f938d1f1640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f93880c8cd0 0x7f93880c7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59100/0 (socket says 172.21.15.1:59100) 2024-03-06T07:24:37.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.666+0000 7f938d9f2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f93880c90b0 0x7f9388155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.666+0000 7f938d1f1640 1 -- 172.21.15.1:0/3380114952 learned_addr learned my addr 172.21.15.1:0/3380114952 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:37.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.666+0000 7f9387fff640 1 -- 172.21.15.1:0/3380114952 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922828156 0 0) 0x7f93880c2b90 con 0x7f93880c8cd0 2024-03-06T07:24:37.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.666+0000 7f9387fff640 1 -- 172.21.15.1:0/3380114952 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9370003520 con 0x7f93880c8cd0 2024-03-06T07:24:37.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.666+0000 7f938d9f2640 1 -- 172.21.15.1:0/3380114952 >> v1:172.21.15.98:6789/0 conn(0x7f93880c3260 legacy=0x7f9388159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.666+0000 7f938d9f2640 1 -- 172.21.15.1:0/3380114952 >> v1:172.21.15.1:6789/0 conn(0x7f93880c8cd0 legacy=0x7f93880c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.666+0000 7f938d9f2640 1 -- 172.21.15.1:0/3380114952 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93880c3a90 con 0x7f93880c90b0 2024-03-06T07:24:37.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.667+0000 7f938d9f2640 1 --2- 172.21.15.1:0/3380114952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c90b0 0x7f9388155b70 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f9378009d90 tx=0x7f937802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41662578acb4c803 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.667+0000 7f9387fff640 1 -- 172.21.15.1:0/3380114952 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9378039070 con 0x7f93880c90b0 2024-03-06T07:24:37.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.667+0000 7f9387fff640 1 -- 172.21.15.1:0/3380114952 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f937802cc60 con 0x7f93880c90b0 2024-03-06T07:24:37.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.667+0000 7f9387fff640 1 -- 172.21.15.1:0/3380114952 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9378034660 con 0x7f93880c90b0 2024-03-06T07:24:37.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.667+0000 7f938f455640 1 -- 172.21.15.1:0/3380114952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c90b0 msgr2=0x7f9388155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.667+0000 7f938f455640 1 --2- 172.21.15.1:0/3380114952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c90b0 0x7f9388155b70 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f9378009d90 tx=0x7f937802beb0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.667+0000 7f938f455640 1 -- 172.21.15.1:0/3380114952 shutdown_connections 2024-03-06T07:24:37.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.667+0000 7f938f455640 1 --2- 172.21.15.1:0/3380114952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c90b0 0x7f9388155b70 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.667+0000 7f938f455640 1 -- 172.21.15.1:0/3380114952 >> 172.21.15.1:0/3380114952 conn(0x7f938806b3e0 msgr2=0x7f938806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.668+0000 7f938f455640 1 -- 172.21.15.1:0/3380114952 shutdown_connections 2024-03-06T07:24:37.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.668+0000 7f938f455640 1 -- 172.21.15.1:0/3380114952 wait complete. 2024-03-06T07:24:37.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.669+0000 7f938f455640 1 Processor -- start 2024-03-06T07:24:37.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.669+0000 7f938f455640 1 -- start start 2024-03-06T07:24:37.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.669+0000 7f938f455640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93880c3260 0x7f93880ac280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.669+0000 7f938f455640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93880c8cd0 0x7f93880a8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.669+0000 7f938f455640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c90b0 0x7f93880708c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.669+0000 7f938f455640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93880c8560 con 0x7f93880c90b0 2024-03-06T07:24:37.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.669+0000 7f938f455640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9388158af0 con 0x7f93880c8cd0 2024-03-06T07:24:37.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.669+0000 7f938f455640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93880c86e0 con 0x7f93880c3260 2024-03-06T07:24:37.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.669+0000 7f938d9f2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c90b0 0x7f93880708c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.669+0000 7f938d9f2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c90b0 0x7f93880708c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57984/0 (socket says 172.21.15.1:57984) 2024-03-06T07:24:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.669+0000 7f938d9f2640 1 -- 172.21.15.1:0/2207891159 learned_addr learned my addr 172.21.15.1:0/2207891159 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.669+0000 7f938d1f1640 1 --2- 172.21.15.1:0/2207891159 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93880c3260 0x7f93880ac280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.670+0000 7f938c9f0640 1 --2- 172.21.15.1:0/2207891159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93880c8cd0 0x7f93880a8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.670+0000 7f938d9f2640 1 -- 172.21.15.1:0/2207891159 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93880c3260 msgr2=0x7f93880ac280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.670+0000 7f938d9f2640 1 --2- 172.21.15.1:0/2207891159 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93880c3260 0x7f93880ac280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.670+0000 7f938d9f2640 1 -- 172.21.15.1:0/2207891159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93880c8cd0 msgr2=0x7f93880a8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.670+0000 7f938d9f2640 1 --2- 172.21.15.1:0/2207891159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93880c8cd0 0x7f93880a8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.670+0000 7f938d9f2640 1 -- 172.21.15.1:0/2207891159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93880adbb0 con 0x7f93880c90b0 2024-03-06T07:24:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.670+0000 7f938d9f2640 1 --2- 172.21.15.1:0/2207891159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c90b0 0x7f93880708c0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f9378009eb0 tx=0x7f9378004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.670+0000 7f9385ffb640 1 -- 172.21.15.1:0/2207891159 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9378039050 con 0x7f93880c90b0 2024-03-06T07:24:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.670+0000 7f9385ffb640 1 -- 172.21.15.1:0/2207891159 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9378045070 con 0x7f93880c90b0 2024-03-06T07:24:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.670+0000 7f938f455640 1 -- 172.21.15.1:0/2207891159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93880ad8d0 con 0x7f93880c90b0 2024-03-06T07:24:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.670+0000 7f9385ffb640 1 -- 172.21.15.1:0/2207891159 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9378009450 con 0x7f93880c90b0 2024-03-06T07:24:37.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.670+0000 7f938f455640 1 -- 172.21.15.1:0/2207891159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93880aba80 con 0x7f93880c90b0 2024-03-06T07:24:37.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.671+0000 7f938f455640 1 -- 172.21.15.1:0/2207891159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9388067fd0 con 0x7f93880c90b0 2024-03-06T07:24:37.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.672+0000 7f9385ffb640 1 -- 172.21.15.1:0/2207891159 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f937803e850 con 0x7f93880c90b0 2024-03-06T07:24:37.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.672+0000 7f9385ffb640 1 --2- 172.21.15.1:0/2207891159 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f937406b4d0 0x7f937406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.672+0000 7f9385ffb640 1 -- 172.21.15.1:0/2207891159 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f93780ad220 con 0x7f93880c90b0 2024-03-06T07:24:37.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.672+0000 7f938d1f1640 1 --2- 172.21.15.1:0/2207891159 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f937406b4d0 0x7f937406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.673+0000 7f938d1f1640 1 --2- 172.21.15.1:0/2207891159 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f937406b4d0 0x7f937406d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f93800011e0 tx=0x7f9380007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.676+0000 7f9385ffb640 1 -- 172.21.15.1:0/2207891159 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f937807ba20 con 0x7f93880c90b0 2024-03-06T07:24:37.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.859+0000 7f938f455640 1 -- 172.21.15.1:0/2207891159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f93881584c0 con 0x7f93880c90b0 2024-03-06T07:24:37.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.859+0000 7f9385ffb640 1 -- 172.21.15.1:0/2207891159 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) v1 ==== 104+0+11 (secure 0 0 0) 0x7f937807fac0 con 0x7f93880c90b0 2024-03-06T07:24:37.861 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:24:37.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.861+0000 7f938f455640 1 -- 172.21.15.1:0/2207891159 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f937406b4d0 msgr2=0x7f937406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.861+0000 7f938f455640 1 --2- 172.21.15.1:0/2207891159 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f937406b4d0 0x7f937406d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f93800011e0 tx=0x7f9380007d30 comp rx=0 tx=0).stop 2024-03-06T07:24:37.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.861+0000 7f938f455640 1 -- 172.21.15.1:0/2207891159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c90b0 msgr2=0x7f93880708c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.861+0000 7f938f455640 1 --2- 172.21.15.1:0/2207891159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c90b0 0x7f93880708c0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f9378009eb0 tx=0x7f9378004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:37.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.861+0000 7f938f455640 1 -- 172.21.15.1:0/2207891159 shutdown_connections 2024-03-06T07:24:37.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.861+0000 7f938f455640 1 --2- 172.21.15.1:0/2207891159 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f937406b4d0 0x7f937406d990 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.861+0000 7f938f455640 1 --2- 172.21.15.1:0/2207891159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93880c90b0 0x7f93880708c0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.861+0000 7f938f455640 1 --2- 172.21.15.1:0/2207891159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93880c8cd0 0x7f93880a8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.861+0000 7f938f455640 1 --2- 172.21.15.1:0/2207891159 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93880c3260 0x7f93880ac280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.861+0000 7f938f455640 1 -- 172.21.15.1:0/2207891159 >> 172.21.15.1:0/2207891159 conn(0x7f938806b3e0 msgr2=0x7f938806b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.861+0000 7f938f455640 1 -- 172.21.15.1:0/2207891159 shutdown_connections 2024-03-06T07:24:37.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.861+0000 7f938f455640 1 -- 172.21.15.1:0/2207891159 wait complete. 2024-03-06T07:24:37.874 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:37.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.976+0000 7f61bafc7640 1 Processor -- start 2024-03-06T07:24:37.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.976+0000 7f61bafc7640 1 -- start start 2024-03-06T07:24:37.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.977+0000 7f61bafc7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f61b4152840 0x7f61b40c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.977+0000 7f61bafc7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f61b40c60b0 0x7f61b41553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.977+0000 7f61bafc7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b40c5e40 con 0x7f61b40c3970 2024-03-06T07:24:37.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.977+0000 7f61bafc7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b40c67f0 con 0x7f61b40c60b0 2024-03-06T07:24:37.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.977+0000 7f61bafc7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b40c6a40 con 0x7f61b4152840 2024-03-06T07:24:37.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.977+0000 7f61b9564640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f61b40c60b0 0x7f61b41553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.977+0000 7f61b9564640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f61b40c60b0 0x7f61b41553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57986/0 (socket says 172.21.15.1:57986) 2024-03-06T07:24:37.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.977+0000 7f61b9564640 1 -- 172.21.15.1:0/2087521168 learned_addr learned my addr 172.21.15.1:0/2087521168 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:37.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.977+0000 7f61b3fff640 1 --2- 172.21.15.1:0/2087521168 >> v2:172.21.15.98:3300/0 conn(0x7f61b4152840 0x7f61b40c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.978+0000 7f61b9564640 1 -- 172.21.15.1:0/2087521168 >> v2:172.21.15.98:3300/0 conn(0x7f61b4152840 msgr2=0x7f61b40c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:37.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.978+0000 7f61b9564640 1 --2- 172.21.15.1:0/2087521168 >> v2:172.21.15.98:3300/0 conn(0x7f61b4152840 0x7f61b40c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.978+0000 7f61b9564640 1 -- 172.21.15.1:0/2087521168 >> v1:172.21.15.98:6789/0 conn(0x7f61b40c3970 legacy=0x7f61b4152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.978+0000 7f61b9564640 1 -- 172.21.15.1:0/2087521168 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b40c6d20 con 0x7f61b40c60b0 2024-03-06T07:24:37.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.978+0000 7f61b9564640 1 --2- 172.21.15.1:0/2087521168 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b40c60b0 0x7f61b41553d0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009e50 tx=0x7f61ac02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a7eb2e4de3c61 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.979+0000 7f61b37fe640 1 -- 172.21.15.1:0/2087521168 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac02cb20 con 0x7f61b40c60b0 2024-03-06T07:24:37.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.979+0000 7f61b37fe640 1 -- 172.21.15.1:0/2087521168 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ac02ccc0 con 0x7f61b40c60b0 2024-03-06T07:24:37.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.979+0000 7f61b37fe640 1 -- 172.21.15.1:0/2087521168 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac034610 con 0x7f61b40c60b0 2024-03-06T07:24:37.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.979+0000 7f61bafc7640 1 -- 172.21.15.1:0/2087521168 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b40c60b0 msgr2=0x7f61b41553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.979+0000 7f61bafc7640 1 --2- 172.21.15.1:0/2087521168 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b40c60b0 0x7f61b41553d0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009e50 tx=0x7f61ac02bf80 comp rx=0 tx=0).stop 2024-03-06T07:24:37.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.979+0000 7f61bafc7640 1 -- 172.21.15.1:0/2087521168 shutdown_connections 2024-03-06T07:24:37.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.979+0000 7f61bafc7640 1 --2- 172.21.15.1:0/2087521168 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b40c60b0 0x7f61b41553d0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.979+0000 7f61bafc7640 1 --2- 172.21.15.1:0/2087521168 >> v2:172.21.15.98:3300/0 conn(0x7f61b4152840 0x7f61b40c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.979+0000 7f61bafc7640 1 -- 172.21.15.1:0/2087521168 >> 172.21.15.1:0/2087521168 conn(0x7f61b40c88f0 msgr2=0x7f61b40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.979+0000 7f61bafc7640 1 -- 172.21.15.1:0/2087521168 shutdown_connections 2024-03-06T07:24:37.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.979+0000 7f61bafc7640 1 -- 172.21.15.1:0/2087521168 wait complete. 2024-03-06T07:24:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.980+0000 7f61bafc7640 1 Processor -- start 2024-03-06T07:24:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.980+0000 7f61bafc7640 1 -- start start 2024-03-06T07:24:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.981+0000 7f61bafc7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b40c3970 0x7f61b4157730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.981+0000 7f61bafc7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61b40c60b0 0x7f61b414c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.981+0000 7f61bafc7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b4152840 0x7f61b414c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.981+0000 7f61bafc7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40c75c0 con 0x7f61b40c3970 2024-03-06T07:24:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.981+0000 7f61bafc7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40c72e0 con 0x7f61b4152840 2024-03-06T07:24:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.981+0000 7f61bafc7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40c3d50 con 0x7f61b40c60b0 2024-03-06T07:24:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.981+0000 7f61b8d63640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b40c3970 0x7f61b4157730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.981+0000 7f61b8d63640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b40c3970 0x7f61b4157730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58002/0 (socket says 172.21.15.1:58002) 2024-03-06T07:24:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.981+0000 7f61b8d63640 1 -- 172.21.15.1:0/3009724647 learned_addr learned my addr 172.21.15.1:0/3009724647 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.981+0000 7f61b9564640 1 --2- 172.21.15.1:0/3009724647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b4152840 0x7f61b414c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.981+0000 7f61b3fff640 1 --2- 172.21.15.1:0/3009724647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61b40c60b0 0x7f61b414c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.982+0000 7f61b8d63640 1 -- 172.21.15.1:0/3009724647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61b40c60b0 msgr2=0x7f61b414c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.982+0000 7f61b8d63640 1 --2- 172.21.15.1:0/3009724647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61b40c60b0 0x7f61b414c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.982+0000 7f61b8d63640 1 -- 172.21.15.1:0/3009724647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b4152840 msgr2=0x7f61b414c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.982+0000 7f61b8d63640 1 --2- 172.21.15.1:0/3009724647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b4152840 0x7f61b414c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.982+0000 7f61b8d63640 1 -- 172.21.15.1:0/3009724647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b414d270 con 0x7f61b40c3970 2024-03-06T07:24:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.982+0000 7f61b9564640 1 --2- 172.21.15.1:0/3009724647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b4152840 0x7f61b414c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.982+0000 7f61b3fff640 1 --2- 172.21.15.1:0/3009724647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61b40c60b0 0x7f61b414c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.982+0000 7f61b8d63640 1 --2- 172.21.15.1:0/3009724647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b40c3970 0x7f61b4157730 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f61a800c430 tx=0x7f61a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.982+0000 7f61b17fa640 1 -- 172.21.15.1:0/3009724647 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a80052d0 con 0x7f61b40c3970 2024-03-06T07:24:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.982+0000 7f61b17fa640 1 -- 172.21.15.1:0/3009724647 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a8005490 con 0x7f61b40c3970 2024-03-06T07:24:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.982+0000 7f61bafc7640 1 -- 172.21.15.1:0/3009724647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b41eb590 con 0x7f61b40c3970 2024-03-06T07:24:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.982+0000 7f61b17fa640 1 -- 172.21.15.1:0/3009724647 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a8007260 con 0x7f61b40c3970 2024-03-06T07:24:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.982+0000 7f61bafc7640 1 -- 172.21.15.1:0/3009724647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b4157c70 con 0x7f61b40c3970 2024-03-06T07:24:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.983+0000 7f61bafc7640 1 -- 172.21.15.1:0/3009724647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b40686e0 con 0x7f61b40c3970 2024-03-06T07:24:37.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.983+0000 7f61b17fa640 1 -- 172.21.15.1:0/3009724647 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61a8007400 con 0x7f61b40c3970 2024-03-06T07:24:37.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.984+0000 7f61b17fa640 1 --2- 172.21.15.1:0/3009724647 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f619806b4d0 0x7f619806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.984+0000 7f61b17fa640 1 -- 172.21.15.1:0/3009724647 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f61a8089880 con 0x7f61b40c3970 2024-03-06T07:24:37.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.984+0000 7f61b3fff640 1 --2- 172.21.15.1:0/3009724647 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f619806b4d0 0x7f619806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.984+0000 7f61b3fff640 1 --2- 172.21.15.1:0/3009724647 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f619806b4d0 0x7f619806d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f61b4071280 tx=0x7f61a0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:37.988+0000 7f61b17fa640 1 -- 172.21.15.1:0/3009724647 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61a80585b0 con 0x7f61b40c3970 2024-03-06T07:24:38.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.167+0000 7f61bafc7640 1 -- 172.21.15.1:0/3009724647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f61b4074b60 con 0x7f61b40c3970 2024-03-06T07:24:38.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.167+0000 7f61b17fa640 1 -- 172.21.15.1:0/3009724647 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) v1 ==== 100+0+11 (secure 0 0 0) 0x7f61a8057eb0 con 0x7f61b40c3970 2024-03-06T07:24:38.169 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:24:38.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.169+0000 7f61bafc7640 1 -- 172.21.15.1:0/3009724647 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f619806b4d0 msgr2=0x7f619806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.169+0000 7f61bafc7640 1 --2- 172.21.15.1:0/3009724647 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f619806b4d0 0x7f619806d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f61b4071280 tx=0x7f61a0008040 comp rx=0 tx=0).stop 2024-03-06T07:24:38.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.169+0000 7f61bafc7640 1 -- 172.21.15.1:0/3009724647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b40c3970 msgr2=0x7f61b4157730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.169+0000 7f61bafc7640 1 --2- 172.21.15.1:0/3009724647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b40c3970 0x7f61b4157730 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f61a800c430 tx=0x7f61a800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.169+0000 7f61bafc7640 1 -- 172.21.15.1:0/3009724647 shutdown_connections 2024-03-06T07:24:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.169+0000 7f61bafc7640 1 --2- 172.21.15.1:0/3009724647 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f619806b4d0 0x7f619806d990 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.169+0000 7f61bafc7640 1 --2- 172.21.15.1:0/3009724647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b4152840 0x7f61b414c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.169+0000 7f61bafc7640 1 --2- 172.21.15.1:0/3009724647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61b40c60b0 0x7f61b414c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.169+0000 7f61bafc7640 1 --2- 172.21.15.1:0/3009724647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b40c3970 0x7f61b4157730 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.169+0000 7f61bafc7640 1 -- 172.21.15.1:0/3009724647 >> 172.21.15.1:0/3009724647 conn(0x7f61b40c88f0 msgr2=0x7f61b4074450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.169+0000 7f61bafc7640 1 -- 172.21.15.1:0/3009724647 shutdown_connections 2024-03-06T07:24:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.169+0000 7f61bafc7640 1 -- 172.21.15.1:0/3009724647 wait complete. 2024-03-06T07:24:38.182 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.281+0000 7f6ad73f7640 1 Processor -- start 2024-03-06T07:24:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.281+0000 7f6ad73f7640 1 -- start start 2024-03-06T07:24:38.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.282+0000 7f6ad73f7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6ad00c6730 0x7f6ad00c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.282+0000 7f6ad73f7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad00c3300 con 0x7f6ad0152690 2024-03-06T07:24:38.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.282+0000 7f6ad73f7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad00c35a0 con 0x7f6ad00c5d10 2024-03-06T07:24:38.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.282+0000 7f6ad73f7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad00c3880 con 0x7f6ad00c6730 2024-03-06T07:24:38.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.282+0000 7f6ad5193640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6ad0152690 0x7f6ad0155ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59108/0 (socket says 172.21.15.1:59108) 2024-03-06T07:24:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.282+0000 7f6ad5193640 1 -- 172.21.15.1:0/2066130423 learned_addr learned my addr 172.21.15.1:0/2066130423 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.282+0000 7f6ad4992640 1 --2- 172.21.15.1:0/2066130423 >> v2:172.21.15.98:3300/0 conn(0x7f6ad00c6730 0x7f6ad00c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.282+0000 7f6ac7fff640 1 -- 172.21.15.1:0/2066130423 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076041807 0 0) 0x7f6ad00c3300 con 0x7f6ad0152690 2024-03-06T07:24:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.282+0000 7f6ac7fff640 1 -- 172.21.15.1:0/2066130423 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6abc003520 con 0x7f6ad0152690 2024-03-06T07:24:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.282+0000 7f6ac7fff640 1 -- 172.21.15.1:0/2066130423 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518999233 0 0) 0x7f6ad00c35a0 con 0x7f6ad00c5d10 2024-03-06T07:24:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.282+0000 7f6ac7fff640 1 -- 172.21.15.1:0/2066130423 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad00c3300 con 0x7f6ad00c5d10 2024-03-06T07:24:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.282+0000 7f6ad4992640 1 -- 172.21.15.1:0/2066130423 >> v1:172.21.15.98:6789/0 conn(0x7f6ad00c5d10 legacy=0x7f6ad0156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.282+0000 7f6ad4992640 1 -- 172.21.15.1:0/2066130423 >> v1:172.21.15.1:6789/0 conn(0x7f6ad0152690 legacy=0x7f6ad0155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.282+0000 7f6ad4992640 1 -- 172.21.15.1:0/2066130423 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad00c3b60 con 0x7f6ad00c6730 2024-03-06T07:24:38.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.283+0000 7f6ad4992640 1 --2- 172.21.15.1:0/2066130423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ad00c6730 0x7f6ad00c57d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00097a0 tx=0x7f6ac002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a72eb7663b5cee29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.283+0000 7f6ac7fff640 1 -- 172.21.15.1:0/2066130423 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0035020 con 0x7f6ad00c6730 2024-03-06T07:24:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.283+0000 7f6ac7fff640 1 -- 172.21.15.1:0/2066130423 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac0031830 con 0x7f6ad00c6730 2024-03-06T07:24:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.283+0000 7f6ac7fff640 1 -- 172.21.15.1:0/2066130423 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0031c00 con 0x7f6ad00c6730 2024-03-06T07:24:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.284+0000 7f6ad73f7640 1 -- 172.21.15.1:0/2066130423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ad00c6730 msgr2=0x7f6ad00c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.284+0000 7f6ad73f7640 1 --2- 172.21.15.1:0/2066130423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ad00c6730 0x7f6ad00c57d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00097a0 tx=0x7f6ac002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.284+0000 7f6ad73f7640 1 -- 172.21.15.1:0/2066130423 shutdown_connections 2024-03-06T07:24:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.284+0000 7f6ad73f7640 1 --2- 172.21.15.1:0/2066130423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ad00c6730 0x7f6ad00c57d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.284+0000 7f6ad73f7640 1 -- 172.21.15.1:0/2066130423 >> 172.21.15.1:0/2066130423 conn(0x7f6ad006b410 msgr2=0x7f6ad006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.284+0000 7f6ad73f7640 1 -- 172.21.15.1:0/2066130423 shutdown_connections 2024-03-06T07:24:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.284+0000 7f6ad73f7640 1 -- 172.21.15.1:0/2066130423 wait complete. 2024-03-06T07:24:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.285+0000 7f6ad73f7640 1 Processor -- start 2024-03-06T07:24:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.285+0000 7f6ad73f7640 1 -- start start 2024-03-06T07:24:38.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad73f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ad00c5d10 0x7f6ad0146c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad73f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad00c6730 0x7f6ad00acfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad73f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ad0152690 0x7f6ad006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad73f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad0152bf0 con 0x7f6ad00c6730 2024-03-06T07:24:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad73f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad0152a70 con 0x7f6ad0152690 2024-03-06T07:24:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad73f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad01573e0 con 0x7f6ad00c5d10 2024-03-06T07:24:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad4992640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad00c6730 0x7f6ad00acfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad4992640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad00c6730 0x7f6ad00acfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58016/0 (socket says 172.21.15.1:58016) 2024-03-06T07:24:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad4992640 1 -- 172.21.15.1:0/836231817 learned_addr learned my addr 172.21.15.1:0/836231817 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad5193640 1 --2- 172.21.15.1:0/836231817 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ad00c5d10 0x7f6ad0146c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad5994640 1 --2- 172.21.15.1:0/836231817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ad0152690 0x7f6ad006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad4992640 1 -- 172.21.15.1:0/836231817 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ad00c5d10 msgr2=0x7f6ad0146c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad4992640 1 --2- 172.21.15.1:0/836231817 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ad00c5d10 0x7f6ad0146c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad4992640 1 -- 172.21.15.1:0/836231817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ad0152690 msgr2=0x7f6ad006d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad4992640 1 --2- 172.21.15.1:0/836231817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ad0152690 0x7f6ad006d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad4992640 1 -- 172.21.15.1:0/836231817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad01ee410 con 0x7f6ad00c6730 2024-03-06T07:24:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad4992640 1 --2- 172.21.15.1:0/836231817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad00c6730 0x7f6ad00acfb0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0009890 tx=0x7f6ac00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ac5ffb640 1 -- 172.21.15.1:0/836231817 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0035040 con 0x7f6ad00c6730 2024-03-06T07:24:38.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ac5ffb640 1 -- 172.21.15.1:0/836231817 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac003b920 con 0x7f6ad00c6730 2024-03-06T07:24:38.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ad73f7640 1 -- 172.21.15.1:0/836231817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad01ee6f0 con 0x7f6ad00c6730 2024-03-06T07:24:38.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.286+0000 7f6ac5ffb640 1 -- 172.21.15.1:0/836231817 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac003bd10 con 0x7f6ad00c6730 2024-03-06T07:24:38.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.287+0000 7f6ad73f7640 1 -- 172.21.15.1:0/836231817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad01eecb0 con 0x7f6ad00c6730 2024-03-06T07:24:38.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.287+0000 7f6ad73f7640 1 -- 172.21.15.1:0/836231817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad0067fd0 con 0x7f6ad00c6730 2024-03-06T07:24:38.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.288+0000 7f6ac5ffb640 1 -- 172.21.15.1:0/836231817 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ac0038020 con 0x7f6ad00c6730 2024-03-06T07:24:38.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.288+0000 7f6ac5ffb640 1 --2- 172.21.15.1:0/836231817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ab406b4d0 0x7f6ab406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.288+0000 7f6ac5ffb640 1 -- 172.21.15.1:0/836231817 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f6ac00ac3b0 con 0x7f6ad00c6730 2024-03-06T07:24:38.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.289+0000 7f6ad5193640 1 --2- 172.21.15.1:0/836231817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ab406b4d0 0x7f6ab406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.289+0000 7f6ad5193640 1 --2- 172.21.15.1:0/836231817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ab406b4d0 0x7f6ab406d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f6acc000f60 tx=0x7f6acc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.293+0000 7f6ac5ffb640 1 -- 172.21.15.1:0/836231817 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ac007a9f0 con 0x7f6ad00c6730 2024-03-06T07:24:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.513+0000 7f6ad73f7640 1 -- 172.21.15.1:0/836231817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ad00c2c40 con 0x7f6ad00c6730 2024-03-06T07:24:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.513+0000 7f6ac5ffb640 1 -- 172.21.15.1:0/836231817 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f6ac007ea90 con 0x7f6ad00c6730 2024-03-06T07:24:38.515 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:38.515 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":98,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1741783790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1741783790},{"type":"v1","addr":"172.21.15.98:6836","nonce":1741783790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7662,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3096543492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3096543492},{"type":"v1","addr":"172.21.15.1:6833","nonce":3096543492}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-06T07:24:10.220198+0000","modified":"2024-03-06T07:24:30.609444+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6122":{"gid":6122,"name":"b","rank":0,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6837/2632566151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2632566151},{"type":"v1","addr":"172.21.15.98:6837","nonce":2632566151}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6670":{"gid":6670,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.1:6835/2244230066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2244230066},{"type":"v1","addr":"172.21.15.1:6835","nonce":2244230066}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6122,"qdb_cluster":[6122]},"id":8}]} 2024-03-06T07:24:38.516 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 98 2024-03-06T07:24:38.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.515+0000 7f6ad73f7640 1 -- 172.21.15.1:0/836231817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ab406b4d0 msgr2=0x7f6ab406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.515+0000 7f6ad73f7640 1 --2- 172.21.15.1:0/836231817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ab406b4d0 0x7f6ab406d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f6acc000f60 tx=0x7f6acc009040 comp rx=0 tx=0).stop 2024-03-06T07:24:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.515+0000 7f6ad73f7640 1 -- 172.21.15.1:0/836231817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad00c6730 msgr2=0x7f6ad00acfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.515+0000 7f6ad73f7640 1 --2- 172.21.15.1:0/836231817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad00c6730 0x7f6ad00acfb0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0009890 tx=0x7f6ac00046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.515+0000 7f6ad73f7640 1 -- 172.21.15.1:0/836231817 shutdown_connections 2024-03-06T07:24:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.515+0000 7f6ad73f7640 1 --2- 172.21.15.1:0/836231817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ab406b4d0 0x7f6ab406d990 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.515+0000 7f6ad73f7640 1 --2- 172.21.15.1:0/836231817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ad0152690 0x7f6ad006d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.515+0000 7f6ad73f7640 1 --2- 172.21.15.1:0/836231817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad00c6730 0x7f6ad00acfb0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.515+0000 7f6ad73f7640 1 --2- 172.21.15.1:0/836231817 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ad00c5d10 0x7f6ad0146c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.515+0000 7f6ad73f7640 1 -- 172.21.15.1:0/836231817 >> 172.21.15.1:0/836231817 conn(0x7f6ad006b410 msgr2=0x7f6ad0151e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.515+0000 7f6ad73f7640 1 -- 172.21.15.1:0/836231817 shutdown_connections 2024-03-06T07:24:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.515+0000 7f6ad73f7640 1 -- 172.21.15.1:0/836231817 wait complete. 2024-03-06T07:24:38.528 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.627+0000 7f5af79d0640 1 Processor -- start 2024-03-06T07:24:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.627+0000 7f5af79d0640 1 -- start start 2024-03-06T07:24:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.628+0000 7f5af79d0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5af00c1390 0x7f5af00c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.628+0000 7f5af79d0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5af01552e0 0x7f5af0155700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.628+0000 7f5af79d0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af01588e0 con 0x7f5af0152730 2024-03-06T07:24:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.628+0000 7f5af79d0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af00c1d60 con 0x7f5af01552e0 2024-03-06T07:24:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.628+0000 7f5af79d0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af0158bc0 con 0x7f5af00c1390 2024-03-06T07:24:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.628+0000 7f5af5f6d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5af01552e0 0x7f5af0155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.628+0000 7f5af5f6d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5af01552e0 0x7f5af0155700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58030/0 (socket says 172.21.15.1:58030) 2024-03-06T07:24:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.628+0000 7f5af5f6d640 1 -- 172.21.15.1:0/3402066557 learned_addr learned my addr 172.21.15.1:0/3402066557 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.628+0000 7f5af4f6b640 1 --2- 172.21.15.1:0/3402066557 >> v2:172.21.15.98:3300/0 conn(0x7f5af00c1390 0x7f5af00c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.628+0000 7f5af5f6d640 1 -- 172.21.15.1:0/3402066557 >> v2:172.21.15.98:3300/0 conn(0x7f5af00c1390 msgr2=0x7f5af00c1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.628+0000 7f5af5f6d640 1 --2- 172.21.15.1:0/3402066557 >> v2:172.21.15.98:3300/0 conn(0x7f5af00c1390 0x7f5af00c1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.628+0000 7f5af5f6d640 1 -- 172.21.15.1:0/3402066557 >> v1:172.21.15.98:6789/0 conn(0x7f5af0152730 legacy=0x7f5af00c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.628+0000 7f5af5f6d640 1 -- 172.21.15.1:0/3402066557 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af0158ea0 con 0x7f5af01552e0 2024-03-06T07:24:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.629+0000 7f5af5f6d640 1 --2- 172.21.15.1:0/3402066557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af01552e0 0x7f5af0155700 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f5af008ca60 tx=0x7f5ae802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770da215e3e3deb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.629+0000 7f5ae7fff640 1 -- 172.21.15.1:0/3402066557 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae802c9f0 con 0x7f5af01552e0 2024-03-06T07:24:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.629+0000 7f5ae7fff640 1 -- 172.21.15.1:0/3402066557 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae802cb90 con 0x7f5af01552e0 2024-03-06T07:24:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.629+0000 7f5ae7fff640 1 -- 172.21.15.1:0/3402066557 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae80343f0 con 0x7f5af01552e0 2024-03-06T07:24:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.630+0000 7f5af79d0640 1 -- 172.21.15.1:0/3402066557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af01552e0 msgr2=0x7f5af0155700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.630+0000 7f5af79d0640 1 --2- 172.21.15.1:0/3402066557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af01552e0 0x7f5af0155700 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f5af008ca60 tx=0x7f5ae802be50 comp rx=0 tx=0).stop 2024-03-06T07:24:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.630+0000 7f5af79d0640 1 -- 172.21.15.1:0/3402066557 shutdown_connections 2024-03-06T07:24:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.630+0000 7f5af79d0640 1 --2- 172.21.15.1:0/3402066557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af01552e0 0x7f5af0155700 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.630+0000 7f5af79d0640 1 --2- 172.21.15.1:0/3402066557 >> v2:172.21.15.98:3300/0 conn(0x7f5af00c1390 0x7f5af00c1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.630+0000 7f5af79d0640 1 -- 172.21.15.1:0/3402066557 >> 172.21.15.1:0/3402066557 conn(0x7f5af0066d00 msgr2=0x7f5af0067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.630+0000 7f5af79d0640 1 -- 172.21.15.1:0/3402066557 shutdown_connections 2024-03-06T07:24:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.630+0000 7f5af79d0640 1 -- 172.21.15.1:0/3402066557 wait complete. 2024-03-06T07:24:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.631+0000 7f5af79d0640 1 Processor -- start 2024-03-06T07:24:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.631+0000 7f5af79d0640 1 -- start start 2024-03-06T07:24:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.631+0000 7f5af79d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5af00c1390 0x7f5af0157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.631+0000 7f5af79d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5af0152730 0x7f5af01ec080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.631+0000 7f5af79d0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af01552e0 0x7f5af0140170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.631+0000 7f5af79d0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af0152b10 con 0x7f5af01552e0 2024-03-06T07:24:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.631+0000 7f5af79d0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af0159460 con 0x7f5af0152730 2024-03-06T07:24:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.631+0000 7f5af79d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af00c71a0 con 0x7f5af00c1390 2024-03-06T07:24:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.632+0000 7f5af5f6d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af01552e0 0x7f5af0140170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.632+0000 7f5af5f6d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af01552e0 0x7f5af0140170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58046/0 (socket says 172.21.15.1:58046) 2024-03-06T07:24:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.632+0000 7f5af5f6d640 1 -- 172.21.15.1:0/756897511 learned_addr learned my addr 172.21.15.1:0/756897511 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.632+0000 7f5af4f6b640 1 --2- 172.21.15.1:0/756897511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5af0152730 0x7f5af01ec080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.632+0000 7f5af576c640 1 --2- 172.21.15.1:0/756897511 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5af00c1390 0x7f5af0157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.632+0000 7f5af5f6d640 1 -- 172.21.15.1:0/756897511 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5af00c1390 msgr2=0x7f5af0157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.632+0000 7f5af5f6d640 1 --2- 172.21.15.1:0/756897511 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5af00c1390 0x7f5af0157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.632+0000 7f5af5f6d640 1 -- 172.21.15.1:0/756897511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5af0152730 msgr2=0x7f5af01ec080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.632+0000 7f5af5f6d640 1 --2- 172.21.15.1:0/756897511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5af0152730 0x7f5af01ec080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.632+0000 7f5af5f6d640 1 -- 172.21.15.1:0/756897511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af00c9bd0 con 0x7f5af01552e0 2024-03-06T07:24:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.632+0000 7f5af5f6d640 1 --2- 172.21.15.1:0/756897511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af01552e0 0x7f5af0140170 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8009f40 tx=0x7f5ae8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.632+0000 7f5ae5ffb640 1 -- 172.21.15.1:0/756897511 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8044020 con 0x7f5af01552e0 2024-03-06T07:24:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.632+0000 7f5ae5ffb640 1 -- 172.21.15.1:0/756897511 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae8040470 con 0x7f5af01552e0 2024-03-06T07:24:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.632+0000 7f5af79d0640 1 -- 172.21.15.1:0/756897511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af01eede0 con 0x7f5af01552e0 2024-03-06T07:24:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.632+0000 7f5ae5ffb640 1 -- 172.21.15.1:0/756897511 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae802c9f0 con 0x7f5af01552e0 2024-03-06T07:24:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.632+0000 7f5af79d0640 1 -- 172.21.15.1:0/756897511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af01eb090 con 0x7f5af01552e0 2024-03-06T07:24:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.633+0000 7f5af79d0640 1 -- 172.21.15.1:0/756897511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af0067fd0 con 0x7f5af01552e0 2024-03-06T07:24:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.634+0000 7f5ae5ffb640 1 -- 172.21.15.1:0/756897511 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ae8047020 con 0x7f5af01552e0 2024-03-06T07:24:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.634+0000 7f5ae5ffb640 1 --2- 172.21.15.1:0/756897511 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ad406b500 0x7f5ad406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.634+0000 7f5ae5ffb640 1 -- 172.21.15.1:0/756897511 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5ae80ac9f0 con 0x7f5af01552e0 2024-03-06T07:24:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.634+0000 7f5af576c640 1 --2- 172.21.15.1:0/756897511 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ad406b500 0x7f5ad406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.635+0000 7f5af576c640 1 --2- 172.21.15.1:0/756897511 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ad406b500 0x7f5ad406d9c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0016e0 tx=0x7f5aec009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.638+0000 7f5ae5ffb640 1 -- 172.21.15.1:0/756897511 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ae80b3080 con 0x7f5af01552e0 2024-03-06T07:24:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.821+0000 7f5af79d0640 1 -- 172.21.15.1:0/756897511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5af00c6870 con 0x7f5af01552e0 2024-03-06T07:24:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.822+0000 7f5ae5ffb640 1 -- 172.21.15.1:0/756897511 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16959 (secure 0 0 0) 0x7f5ae804e030 con 0x7f5af01552e0 2024-03-06T07:24:38.824 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:38.824 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":109,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:24:21.903206+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:08.484521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.184053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/2433923518":"2024-03-07T07:24:21.903185+0000","172.21.15.1:6832/2433923518":"2024-03-07T07:24:21.903185+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.824+0000 7f5af79d0640 1 -- 172.21.15.1:0/756897511 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ad406b500 msgr2=0x7f5ad406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.824+0000 7f5af79d0640 1 --2- 172.21.15.1:0/756897511 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ad406b500 0x7f5ad406d9c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0016e0 tx=0x7f5aec009040 comp rx=0 tx=0).stop 2024-03-06T07:24:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.824+0000 7f5af79d0640 1 -- 172.21.15.1:0/756897511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af01552e0 msgr2=0x7f5af0140170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.824+0000 7f5af79d0640 1 --2- 172.21.15.1:0/756897511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af01552e0 0x7f5af0140170 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8009f40 tx=0x7f5ae8004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.824+0000 7f5af79d0640 1 -- 172.21.15.1:0/756897511 shutdown_connections 2024-03-06T07:24:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.824+0000 7f5af79d0640 1 --2- 172.21.15.1:0/756897511 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ad406b500 0x7f5ad406d9c0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.824+0000 7f5af79d0640 1 --2- 172.21.15.1:0/756897511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af01552e0 0x7f5af0140170 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.824+0000 7f5af79d0640 1 --2- 172.21.15.1:0/756897511 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5af0152730 0x7f5af01ec080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.824+0000 7f5af79d0640 1 --2- 172.21.15.1:0/756897511 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5af00c1390 0x7f5af0157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.824+0000 7f5af79d0640 1 -- 172.21.15.1:0/756897511 >> 172.21.15.1:0/756897511 conn(0x7f5af0066d00 msgr2=0x7f5af00c6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.824+0000 7f5af79d0640 1 -- 172.21.15.1:0/756897511 shutdown_connections 2024-03-06T07:24:38.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.824+0000 7f5af79d0640 1 -- 172.21.15.1:0/756897511 wait complete. 2024-03-06T07:24:38.837 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:24:38.838 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:38.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.937+0000 7ff1fa6b8640 1 Processor -- start 2024-03-06T07:24:38.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.937+0000 7ff1fa6b8640 1 -- start start 2024-03-06T07:24:38.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.938+0000 7ff1fa6b8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1f4152650 0x7ff1f4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.938+0000 7ff1fa6b8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff1f40bf9d0 0x7ff1f40c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.938+0000 7ff1fa6b8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f40c2fc0 con 0x7ff1f40c6b10 2024-03-06T07:24:38.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.938+0000 7ff1fa6b8640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1f4155800 con 0x7ff1f40bf9d0 2024-03-06T07:24:38.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.938+0000 7ff1fa6b8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f4158da0 con 0x7ff1f4152650 2024-03-06T07:24:38.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.938+0000 7ff1f8c55640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff1f40bf9d0 0x7ff1f40c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.938+0000 7ff1f37fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff1f40c6b10 0x7ff1f4158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59124/0 (socket says 172.21.15.1:59124) 2024-03-06T07:24:38.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.938+0000 7ff1f3fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1f4152650 0x7ff1f4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.938+0000 7ff1f37fe640 1 -- 172.21.15.1:0/2683809370 learned_addr learned my addr 172.21.15.1:0/2683809370 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:38.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.939+0000 7ff1f3fff640 1 -- 172.21.15.1:0/2683809370 >> v2:172.21.15.1:3300/0 conn(0x7ff1f40bf9d0 msgr2=0x7ff1f40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:38.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.939+0000 7ff1f3fff640 1 --2- 172.21.15.1:0/2683809370 >> v2:172.21.15.1:3300/0 conn(0x7ff1f40bf9d0 0x7ff1f40c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.939+0000 7ff1f3fff640 1 -- 172.21.15.1:0/2683809370 >> v1:172.21.15.1:6789/0 conn(0x7ff1f40c6b10 legacy=0x7ff1f4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.939+0000 7ff1f3fff640 1 -- 172.21.15.1:0/2683809370 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f40c6690 con 0x7ff1f4152650 2024-03-06T07:24:38.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.939+0000 7ff1f8c55640 1 --2- 172.21.15.1:0/2683809370 >> v2:172.21.15.1:3300/0 conn(0x7ff1f40bf9d0 0x7ff1f40c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:38.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.939+0000 7ff1f3fff640 1 --2- 172.21.15.1:0/2683809370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1f4152650 0x7ff1f4155c90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff1e40097a0 tx=0x7ff1e402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d712cce34a4b115d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.940+0000 7ff1f2ffd640 1 -- 172.21.15.1:0/2683809370 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4031780 con 0x7ff1f4152650 2024-03-06T07:24:38.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.940+0000 7ff1f2ffd640 1 -- 172.21.15.1:0/2683809370 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e4037070 con 0x7ff1f4152650 2024-03-06T07:24:38.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.940+0000 7ff1f2ffd640 1 -- 172.21.15.1:0/2683809370 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4031b40 con 0x7ff1f4152650 2024-03-06T07:24:38.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.940+0000 7ff1fa6b8640 1 -- 172.21.15.1:0/2683809370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1f4152650 msgr2=0x7ff1f4155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.940+0000 7ff1fa6b8640 1 --2- 172.21.15.1:0/2683809370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1f4152650 0x7ff1f4155c90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff1e40097a0 tx=0x7ff1e402b820 comp rx=0 tx=0).stop 2024-03-06T07:24:38.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.940+0000 7ff1fa6b8640 1 -- 172.21.15.1:0/2683809370 shutdown_connections 2024-03-06T07:24:38.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.940+0000 7ff1fa6b8640 1 --2- 172.21.15.1:0/2683809370 >> v2:172.21.15.1:3300/0 conn(0x7ff1f40bf9d0 0x7ff1f40c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.940+0000 7ff1fa6b8640 1 --2- 172.21.15.1:0/2683809370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1f4152650 0x7ff1f4155c90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.940+0000 7ff1fa6b8640 1 -- 172.21.15.1:0/2683809370 >> 172.21.15.1:0/2683809370 conn(0x7ff1f4066ee0 msgr2=0x7ff1f40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.940+0000 7ff1fa6b8640 1 -- 172.21.15.1:0/2683809370 shutdown_connections 2024-03-06T07:24:38.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.940+0000 7ff1fa6b8640 1 -- 172.21.15.1:0/2683809370 wait complete. 2024-03-06T07:24:38.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.941+0000 7ff1fa6b8640 1 Processor -- start 2024-03-06T07:24:38.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.941+0000 7ff1fa6b8640 1 -- start start 2024-03-06T07:24:38.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.942+0000 7ff1fa6b8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1f40bf9d0 0x7ff1f4158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.942+0000 7ff1fa6b8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1f4152650 0x7ff1f40c5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.942+0000 7ff1fa6b8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1f41eab90 0x7ff1f41ee780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.942+0000 7ff1fa6b8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f41551b0 con 0x7ff1f40bf9d0 2024-03-06T07:24:38.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.942+0000 7ff1fa6b8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f40c6fa0 con 0x7ff1f41eab90 2024-03-06T07:24:38.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.942+0000 7ff1fa6b8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f4155490 con 0x7ff1f4152650 2024-03-06T07:24:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.942+0000 7ff1f3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1f40bf9d0 0x7ff1f4158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.942+0000 7ff1f3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1f40bf9d0 0x7ff1f4158680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58058/0 (socket says 172.21.15.1:58058) 2024-03-06T07:24:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.942+0000 7ff1f3fff640 1 -- 172.21.15.1:0/847868018 learned_addr learned my addr 172.21.15.1:0/847868018 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.942+0000 7ff1f8c55640 1 --2- 172.21.15.1:0/847868018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1f41eab90 0x7ff1f41ee780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.942+0000 7ff1f37fe640 1 --2- 172.21.15.1:0/847868018 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1f4152650 0x7ff1f40c5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.942+0000 7ff1f3fff640 1 -- 172.21.15.1:0/847868018 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1f4152650 msgr2=0x7ff1f40c5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.942+0000 7ff1f3fff640 1 --2- 172.21.15.1:0/847868018 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1f4152650 0x7ff1f40c5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.942+0000 7ff1f3fff640 1 -- 172.21.15.1:0/847868018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1f41eab90 msgr2=0x7ff1f41ee780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.942+0000 7ff1f3fff640 1 --2- 172.21.15.1:0/847868018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1f41eab90 0x7ff1f41ee780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.942+0000 7ff1f3fff640 1 -- 172.21.15.1:0/847868018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f41ee2a0 con 0x7ff1f40bf9d0 2024-03-06T07:24:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.942+0000 7ff1f8c55640 1 --2- 172.21.15.1:0/847868018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1f41eab90 0x7ff1f41ee780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.943+0000 7ff1f3fff640 1 --2- 172.21.15.1:0/847868018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1f40bf9d0 0x7ff1f4158680 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4034040 tx=0x7ff1e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.943+0000 7ff1f0ff9640 1 -- 172.21.15.1:0/847868018 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4032610 con 0x7ff1f40bf9d0 2024-03-06T07:24:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.943+0000 7ff1f0ff9640 1 -- 172.21.15.1:0/847868018 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e40327b0 con 0x7ff1f40bf9d0 2024-03-06T07:24:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.943+0000 7ff1f0ff9640 1 -- 172.21.15.1:0/847868018 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4032ba0 con 0x7ff1f40bf9d0 2024-03-06T07:24:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.943+0000 7ff1fa6b8640 1 -- 172.21.15.1:0/847868018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f4154a70 con 0x7ff1f40bf9d0 2024-03-06T07:24:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.943+0000 7ff1fa6b8640 1 -- 172.21.15.1:0/847868018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f40c60d0 con 0x7ff1f40bf9d0 2024-03-06T07:24:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.944+0000 7ff1fa6b8640 1 -- 172.21.15.1:0/847868018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f406cf30 con 0x7ff1f40bf9d0 2024-03-06T07:24:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.944+0000 7ff1f0ff9640 1 -- 172.21.15.1:0/847868018 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1e404e440 con 0x7ff1f40bf9d0 2024-03-06T07:24:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.945+0000 7ff1f0ff9640 1 --2- 172.21.15.1:0/847868018 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1e006b500 0x7ff1e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.945+0000 7ff1f0ff9640 1 -- 172.21.15.1:0/847868018 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff1e40ac0e0 con 0x7ff1f40bf9d0 2024-03-06T07:24:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.945+0000 7ff1f37fe640 1 --2- 172.21.15.1:0/847868018 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1e006b500 0x7ff1e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.945+0000 7ff1f37fe640 1 --2- 172.21.15.1:0/847868018 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1e006b500 0x7ff1e006d9c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff1f40c54c0 tx=0x7ff1e8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:38.949+0000 7ff1f0ff9640 1 -- 172.21.15.1:0/847868018 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1f406cf30 con 0x7ff1f40bf9d0 2024-03-06T07:24:39.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.167+0000 7ff1fa6b8640 1 -- 172.21.15.1:0/847868018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1f4067fd0 con 0x7ff1f40bf9d0 2024-03-06T07:24:39.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.167+0000 7ff1f0ff9640 1 -- 172.21.15.1:0/847868018 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4978 (secure 0 0 0) 0x7ff1e4040070 con 0x7ff1f40bf9d0 2024-03-06T07:24:39.170 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:39.170 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":98,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1741783790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1741783790},{"type":"v1","addr":"172.21.15.98:6836","nonce":1741783790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7662,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3096543492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3096543492},{"type":"v1","addr":"172.21.15.1:6833","nonce":3096543492}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-06T07:24:10.220198+0000","modified":"2024-03-06T07:24:30.609444+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6122":{"gid":6122,"name":"b","rank":0,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6837/2632566151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2632566151},{"type":"v1","addr":"172.21.15.98:6837","nonce":2632566151}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6670":{"gid":6670,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.1:6835/2244230066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2244230066},{"type":"v1","addr":"172.21.15.1:6835","nonce":2244230066}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6122,"qdb_cluster":[6122]},"id":8}]} 2024-03-06T07:24:39.170 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 98 2024-03-06T07:24:39.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.169+0000 7ff1fa6b8640 1 -- 172.21.15.1:0/847868018 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1e006b500 msgr2=0x7ff1e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.169+0000 7ff1fa6b8640 1 --2- 172.21.15.1:0/847868018 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1e006b500 0x7ff1e006d9c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff1f40c54c0 tx=0x7ff1e8009040 comp rx=0 tx=0).stop 2024-03-06T07:24:39.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.169+0000 7ff1fa6b8640 1 -- 172.21.15.1:0/847868018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1f40bf9d0 msgr2=0x7ff1f4158680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.169+0000 7ff1fa6b8640 1 --2- 172.21.15.1:0/847868018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1f40bf9d0 0x7ff1f4158680 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4034040 tx=0x7ff1e40046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.169+0000 7ff1fa6b8640 1 -- 172.21.15.1:0/847868018 shutdown_connections 2024-03-06T07:24:39.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.169+0000 7ff1fa6b8640 1 --2- 172.21.15.1:0/847868018 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1e006b500 0x7ff1e006d9c0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.169+0000 7ff1fa6b8640 1 --2- 172.21.15.1:0/847868018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1f41eab90 0x7ff1f41ee780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.169+0000 7ff1fa6b8640 1 --2- 172.21.15.1:0/847868018 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1f4152650 0x7ff1f40c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.169+0000 7ff1fa6b8640 1 --2- 172.21.15.1:0/847868018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1f40bf9d0 0x7ff1f4158680 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.169+0000 7ff1fa6b8640 1 -- 172.21.15.1:0/847868018 >> 172.21.15.1:0/847868018 conn(0x7ff1f4066ee0 msgr2=0x7ff1f40bff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.169+0000 7ff1fa6b8640 1 -- 172.21.15.1:0/847868018 shutdown_connections 2024-03-06T07:24:39.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.169+0000 7ff1fa6b8640 1 -- 172.21.15.1:0/847868018 wait complete. 2024-03-06T07:24:39.182 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:39.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.281+0000 7fa76691e640 1 Processor -- start 2024-03-06T07:24:39.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.281+0000 7fa76691e640 1 -- start start 2024-03-06T07:24:39.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.282+0000 7fa76691e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa760159140 0x7fa7600c4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.282+0000 7fa76691e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7600c7510 0x7fa76015fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.282+0000 7fa76691e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa760072d00 con 0x7fa7600c69b0 2024-03-06T07:24:39.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.282+0000 7fa76691e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7600c0e80 con 0x7fa760159140 2024-03-06T07:24:39.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.282+0000 7fa76691e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7600c13b0 con 0x7fa7600c7510 2024-03-06T07:24:39.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.282+0000 7fa75f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa7600c69b0 0x7fa7600c2b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59140/0 (socket says 172.21.15.1:59140) 2024-03-06T07:24:39.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.282+0000 7fa75ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa760159140 0x7fa7600c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.282+0000 7fa75f7fe640 1 -- 172.21.15.1:0/2605303868 learned_addr learned my addr 172.21.15.1:0/2605303868 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:39.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.282+0000 7fa764ebb640 1 --2- 172.21.15.1:0/2605303868 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7600c7510 0x7fa76015fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.282+0000 7fa75effd640 1 -- 172.21.15.1:0/2605303868 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684643651 0 0) 0x7fa760072d00 con 0x7fa7600c69b0 2024-03-06T07:24:39.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.283+0000 7fa75effd640 1 -- 172.21.15.1:0/2605303868 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa748003520 con 0x7fa7600c69b0 2024-03-06T07:24:39.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.283+0000 7fa75ffff640 1 -- 172.21.15.1:0/2605303868 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7600c7510 msgr2=0x7fa76015fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:39.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.283+0000 7fa75ffff640 1 --2- 172.21.15.1:0/2605303868 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7600c7510 0x7fa76015fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.283+0000 7fa75ffff640 1 -- 172.21.15.1:0/2605303868 >> v1:172.21.15.1:6789/0 conn(0x7fa7600c69b0 legacy=0x7fa7600c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.283+0000 7fa75ffff640 1 -- 172.21.15.1:0/2605303868 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa760073180 con 0x7fa760159140 2024-03-06T07:24:39.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.283+0000 7fa764ebb640 1 --2- 172.21.15.1:0/2605303868 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7600c7510 0x7fa76015fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:39.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.283+0000 7fa75ffff640 1 --2- 172.21.15.1:0/2605303868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa760159140 0x7fa7600c4490 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fa7500097a0 tx=0x7fa75002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8358a4b4541b888 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.283+0000 7fa75effd640 1 -- 172.21.15.1:0/2605303868 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa750031780 con 0x7fa760159140 2024-03-06T07:24:39.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.284+0000 7fa75effd640 1 -- 172.21.15.1:0/2605303868 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa750037070 con 0x7fa760159140 2024-03-06T07:24:39.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.284+0000 7fa75effd640 1 -- 172.21.15.1:0/2605303868 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa750031b20 con 0x7fa760159140 2024-03-06T07:24:39.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.284+0000 7fa76691e640 1 -- 172.21.15.1:0/2605303868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa760159140 msgr2=0x7fa7600c4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.284+0000 7fa76691e640 1 --2- 172.21.15.1:0/2605303868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa760159140 0x7fa7600c4490 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fa7500097a0 tx=0x7fa75002b820 comp rx=0 tx=0).stop 2024-03-06T07:24:39.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.284+0000 7fa76691e640 1 -- 172.21.15.1:0/2605303868 shutdown_connections 2024-03-06T07:24:39.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.284+0000 7fa76691e640 1 --2- 172.21.15.1:0/2605303868 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7600c7510 0x7fa76015fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.284+0000 7fa76691e640 1 --2- 172.21.15.1:0/2605303868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa760159140 0x7fa7600c4490 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.284+0000 7fa76691e640 1 -- 172.21.15.1:0/2605303868 >> 172.21.15.1:0/2605303868 conn(0x7fa7600c7a50 msgr2=0x7fa7600c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.284+0000 7fa76691e640 1 -- 172.21.15.1:0/2605303868 shutdown_connections 2024-03-06T07:24:39.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.284+0000 7fa76691e640 1 -- 172.21.15.1:0/2605303868 wait complete. 2024-03-06T07:24:39.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.285+0000 7fa76691e640 1 Processor -- start 2024-03-06T07:24:39.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.285+0000 7fa76691e640 1 -- start start 2024-03-06T07:24:39.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.286+0000 7fa76691e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7600c69b0 0x7fa7601eaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.286+0000 7fa76691e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7600c7510 0x7fa7600c3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.286+0000 7fa76691e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa760159140 0x7fa7601edb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.286+0000 7fa76691e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7600c10d0 con 0x7fa7600c69b0 2024-03-06T07:24:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.286+0000 7fa76691e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7600c6520 con 0x7fa7600c7510 2024-03-06T07:24:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.286+0000 7fa76691e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa760159520 con 0x7fa760159140 2024-03-06T07:24:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.286+0000 7fa75ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7600c69b0 0x7fa7601eaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.286+0000 7fa75ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7600c69b0 0x7fa7601eaf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58076/0 (socket says 172.21.15.1:58076) 2024-03-06T07:24:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.286+0000 7fa75ffff640 1 -- 172.21.15.1:0/1087531170 learned_addr learned my addr 172.21.15.1:0/1087531170 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.286+0000 7fa764ebb640 1 --2- 172.21.15.1:0/1087531170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa760159140 0x7fa7601edb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.286+0000 7fa75f7fe640 1 --2- 172.21.15.1:0/1087531170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7600c7510 0x7fa7600c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.286+0000 7fa75ffff640 1 -- 172.21.15.1:0/1087531170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa760159140 msgr2=0x7fa7601edb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.286+0000 7fa75ffff640 1 --2- 172.21.15.1:0/1087531170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa760159140 0x7fa7601edb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.286+0000 7fa75ffff640 1 -- 172.21.15.1:0/1087531170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7600c7510 msgr2=0x7fa7600c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.286+0000 7fa75ffff640 1 --2- 172.21.15.1:0/1087531170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7600c7510 0x7fa7600c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.286+0000 7fa75ffff640 1 -- 172.21.15.1:0/1087531170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76009a1b0 con 0x7fa7600c69b0 2024-03-06T07:24:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.287+0000 7fa75ffff640 1 --2- 172.21.15.1:0/1087531170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7600c69b0 0x7fa7601eaf70 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fa750000c00 tx=0x7fa7500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.287+0000 7fa75cff9640 1 -- 172.21.15.1:0/1087531170 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa750032610 con 0x7fa7600c69b0 2024-03-06T07:24:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.287+0000 7fa75cff9640 1 -- 172.21.15.1:0/1087531170 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7500327b0 con 0x7fa7600c69b0 2024-03-06T07:24:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.287+0000 7fa76691e640 1 -- 172.21.15.1:0/1087531170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7601eb4b0 con 0x7fa7600c69b0 2024-03-06T07:24:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.287+0000 7fa75cff9640 1 -- 172.21.15.1:0/1087531170 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa750032b80 con 0x7fa7600c69b0 2024-03-06T07:24:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.287+0000 7fa76691e640 1 -- 172.21.15.1:0/1087531170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa760099ed0 con 0x7fa7600c69b0 2024-03-06T07:24:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.288+0000 7fa76691e640 1 -- 172.21.15.1:0/1087531170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7600686e0 con 0x7fa7600c69b0 2024-03-06T07:24:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.288+0000 7fa75cff9640 1 -- 172.21.15.1:0/1087531170 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa75004c060 con 0x7fa7600c69b0 2024-03-06T07:24:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.289+0000 7fa75cff9640 1 --2- 172.21.15.1:0/1087531170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa74c06b4d0 0x7fa74c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.289+0000 7fa75cff9640 1 -- 172.21.15.1:0/1087531170 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa7500abcb0 con 0x7fa7600c69b0 2024-03-06T07:24:39.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.289+0000 7fa75f7fe640 1 --2- 172.21.15.1:0/1087531170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa74c06b4d0 0x7fa74c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.289+0000 7fa75f7fe640 1 --2- 172.21.15.1:0/1087531170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa74c06b4d0 0x7fa74c06d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa7600a7d70 tx=0x7fa754009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.293+0000 7fa75cff9640 1 -- 172.21.15.1:0/1087531170 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa750040070 con 0x7fa7600c69b0 2024-03-06T07:24:39.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.510+0000 7fa76691e640 1 -- 172.21.15.1:0/1087531170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa760072f60 con 0x7fa7600c69b0 2024-03-06T07:24:39.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.510+0000 7fa75cff9640 1 -- 172.21.15.1:0/1087531170 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4978 (secure 0 0 0) 0x7fa75002f050 con 0x7fa7600c69b0 2024-03-06T07:24:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:39.512 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":98,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1741783790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1741783790},{"type":"v1","addr":"172.21.15.98:6836","nonce":1741783790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7662,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3096543492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3096543492},{"type":"v1","addr":"172.21.15.1:6833","nonce":3096543492}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-06T07:24:10.220198+0000","modified":"2024-03-06T07:24:30.609444+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6122":{"gid":6122,"name":"b","rank":0,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.98:6837/2632566151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2632566151},{"type":"v1","addr":"172.21.15.98:6837","nonce":2632566151}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6670":{"gid":6670,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.1:6835/2244230066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2244230066},{"type":"v1","addr":"172.21.15.1:6835","nonce":2244230066}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6122,"qdb_cluster":[6122]},"id":8}]} 2024-03-06T07:24:39.513 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 98 2024-03-06T07:24:39.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.512+0000 7fa76691e640 1 -- 172.21.15.1:0/1087531170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa74c06b4d0 msgr2=0x7fa74c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.512+0000 7fa76691e640 1 --2- 172.21.15.1:0/1087531170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa74c06b4d0 0x7fa74c06d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa7600a7d70 tx=0x7fa754009040 comp rx=0 tx=0).stop 2024-03-06T07:24:39.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.512+0000 7fa76691e640 1 -- 172.21.15.1:0/1087531170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7600c69b0 msgr2=0x7fa7601eaf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.512+0000 7fa76691e640 1 --2- 172.21.15.1:0/1087531170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7600c69b0 0x7fa7601eaf70 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fa750000c00 tx=0x7fa7500046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.512+0000 7fa76691e640 1 -- 172.21.15.1:0/1087531170 shutdown_connections 2024-03-06T07:24:39.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.512+0000 7fa76691e640 1 --2- 172.21.15.1:0/1087531170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa74c06b4d0 0x7fa74c06d990 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.512+0000 7fa76691e640 1 --2- 172.21.15.1:0/1087531170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa760159140 0x7fa7601edb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.512+0000 7fa76691e640 1 --2- 172.21.15.1:0/1087531170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7600c7510 0x7fa7600c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.512+0000 7fa76691e640 1 --2- 172.21.15.1:0/1087531170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7600c69b0 0x7fa7601eaf70 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.512+0000 7fa76691e640 1 -- 172.21.15.1:0/1087531170 >> 172.21.15.1:0/1087531170 conn(0x7fa7600c7a50 msgr2=0x7fa7600aa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.512+0000 7fa76691e640 1 -- 172.21.15.1:0/1087531170 shutdown_connections 2024-03-06T07:24:39.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.512+0000 7fa76691e640 1 -- 172.21.15.1:0/1087531170 wait complete. 2024-03-06T07:24:39.525 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:39.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.624+0000 7fcf3755e640 1 Processor -- start 2024-03-06T07:24:39.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.625+0000 7fcf3755e640 1 -- start start 2024-03-06T07:24:39.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.625+0000 7fcf3755e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcf30152650 0x7fcf30155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.625+0000 7fcf3755e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf300c6b10 0x7fcf30158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.625+0000 7fcf3755e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf300c2fc0 con 0x7fcf300bf9d0 2024-03-06T07:24:39.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.625+0000 7fcf3755e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf30155800 con 0x7fcf30152650 2024-03-06T07:24:39.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.625+0000 7fcf3755e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf300c6690 con 0x7fcf300c6b10 2024-03-06T07:24:39.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.625+0000 7fcf352fa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcf30152650 0x7fcf30155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.625+0000 7fcf35afb640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcf300bf9d0 0x7fcf300c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59148/0 (socket says 172.21.15.1:59148) 2024-03-06T07:24:39.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.625+0000 7fcf35afb640 1 -- 172.21.15.1:0/409200513 learned_addr learned my addr 172.21.15.1:0/409200513 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:39.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.625+0000 7fcf34af9640 1 --2- 172.21.15.1:0/409200513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf300c6b10 0x7fcf30158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.626+0000 7fcf352fa640 1 -- 172.21.15.1:0/409200513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf300c6b10 msgr2=0x7fcf30158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:39.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.626+0000 7fcf352fa640 1 --2- 172.21.15.1:0/409200513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf300c6b10 0x7fcf30158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.626+0000 7fcf352fa640 1 -- 172.21.15.1:0/409200513 >> v1:172.21.15.1:6789/0 conn(0x7fcf300bf9d0 legacy=0x7fcf300c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.626+0000 7fcf27fff640 1 -- 172.21.15.1:0/409200513 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662609567 0 0) 0x7fcf300c2fc0 con 0x7fcf300bf9d0 2024-03-06T07:24:39.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.626+0000 7fcf352fa640 1 -- 172.21.15.1:0/409200513 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf300c7280 con 0x7fcf30152650 2024-03-06T07:24:39.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.626+0000 7fcf352fa640 1 --2- 172.21.15.1:0/409200513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf30152650 0x7fcf30155c90 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c0097a0 tx=0x7fcf2c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20df3175b70b33f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.627+0000 7fcf27fff640 1 -- 172.21.15.1:0/409200513 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c0315a0 con 0x7fcf30152650 2024-03-06T07:24:39.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.627+0000 7fcf27fff640 1 -- 172.21.15.1:0/409200513 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2c037070 con 0x7fcf30152650 2024-03-06T07:24:39.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.627+0000 7fcf27fff640 1 -- 172.21.15.1:0/409200513 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c031960 con 0x7fcf30152650 2024-03-06T07:24:39.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.627+0000 7fcf3755e640 1 -- 172.21.15.1:0/409200513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf30152650 msgr2=0x7fcf30155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.627+0000 7fcf3755e640 1 --2- 172.21.15.1:0/409200513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf30152650 0x7fcf30155c90 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c0097a0 tx=0x7fcf2c02b840 comp rx=0 tx=0).stop 2024-03-06T07:24:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.627+0000 7fcf3755e640 1 -- 172.21.15.1:0/409200513 shutdown_connections 2024-03-06T07:24:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.627+0000 7fcf3755e640 1 --2- 172.21.15.1:0/409200513 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf300c6b10 0x7fcf30158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.627+0000 7fcf3755e640 1 --2- 172.21.15.1:0/409200513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf30152650 0x7fcf30155c90 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.627+0000 7fcf3755e640 1 -- 172.21.15.1:0/409200513 >> 172.21.15.1:0/409200513 conn(0x7fcf30066ee0 msgr2=0x7fcf300672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.627+0000 7fcf3755e640 1 -- 172.21.15.1:0/409200513 shutdown_connections 2024-03-06T07:24:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.627+0000 7fcf3755e640 1 -- 172.21.15.1:0/409200513 wait complete. 2024-03-06T07:24:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.628+0000 7fcf3755e640 1 Processor -- start 2024-03-06T07:24:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.628+0000 7fcf3755e640 1 -- start start 2024-03-06T07:24:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.629+0000 7fcf3755e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf300bf9d0 0x7fcf30156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.629+0000 7fcf3755e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf30152650 0x7fcf3014db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.629+0000 7fcf3755e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf301eace0 0x7fcf301eb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.629+0000 7fcf3755e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf30155490 con 0x7fcf301eace0 2024-03-06T07:24:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.629+0000 7fcf3755e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf301551b0 con 0x7fcf30152650 2024-03-06T07:24:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.629+0000 7fcf3755e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf30152a30 con 0x7fcf300bf9d0 2024-03-06T07:24:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.629+0000 7fcf35afb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf301eace0 0x7fcf301eb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.629+0000 7fcf35afb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf301eace0 0x7fcf301eb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58092/0 (socket says 172.21.15.1:58092) 2024-03-06T07:24:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.629+0000 7fcf35afb640 1 -- 172.21.15.1:0/2482661488 learned_addr learned my addr 172.21.15.1:0/2482661488 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.629+0000 7fcf352fa640 1 --2- 172.21.15.1:0/2482661488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf300bf9d0 0x7fcf30156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.629+0000 7fcf34af9640 1 --2- 172.21.15.1:0/2482661488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf30152650 0x7fcf3014db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.629+0000 7fcf35afb640 1 -- 172.21.15.1:0/2482661488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf300bf9d0 msgr2=0x7fcf30156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.629+0000 7fcf35afb640 1 --2- 172.21.15.1:0/2482661488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf300bf9d0 0x7fcf30156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.629+0000 7fcf35afb640 1 -- 172.21.15.1:0/2482661488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf30152650 msgr2=0x7fcf3014db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.629+0000 7fcf35afb640 1 --2- 172.21.15.1:0/2482661488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf30152650 0x7fcf3014db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.629+0000 7fcf35afb640 1 -- 172.21.15.1:0/2482661488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf30158f40 con 0x7fcf301eace0 2024-03-06T07:24:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.630+0000 7fcf35afb640 1 --2- 172.21.15.1:0/2482661488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf301eace0 0x7fcf301eb190 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800c700 tx=0x7fcf2800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.630+0000 7fcf25ffb640 1 -- 172.21.15.1:0/2482661488 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28018070 con 0x7fcf301eace0 2024-03-06T07:24:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.630+0000 7fcf25ffb640 1 -- 172.21.15.1:0/2482661488 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2800ce10 con 0x7fcf301eace0 2024-03-06T07:24:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.630+0000 7fcf3755e640 1 -- 172.21.15.1:0/2482661488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf30159220 con 0x7fcf301eace0 2024-03-06T07:24:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.630+0000 7fcf25ffb640 1 -- 172.21.15.1:0/2482661488 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2800a860 con 0x7fcf301eace0 2024-03-06T07:24:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.630+0000 7fcf3755e640 1 -- 172.21.15.1:0/2482661488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf30157f20 con 0x7fcf301eace0 2024-03-06T07:24:39.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.631+0000 7fcf3755e640 1 -- 172.21.15.1:0/2482661488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf300c2f50 con 0x7fcf301eace0 2024-03-06T07:24:39.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.631+0000 7fcf25ffb640 1 -- 172.21.15.1:0/2482661488 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf2801b070 con 0x7fcf301eace0 2024-03-06T07:24:39.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.632+0000 7fcf25ffb640 1 --2- 172.21.15.1:0/2482661488 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf1406b4d0 0x7fcf1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.632+0000 7fcf25ffb640 1 -- 172.21.15.1:0/2482661488 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fcf280898a0 con 0x7fcf301eace0 2024-03-06T07:24:39.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.632+0000 7fcf352fa640 1 --2- 172.21.15.1:0/2482661488 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf1406b4d0 0x7fcf1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.632+0000 7fcf352fa640 1 --2- 172.21.15.1:0/2482661488 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf1406b4d0 0x7fcf1406d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c036000 tx=0x7fcf2c03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.636+0000 7fcf25ffb640 1 -- 172.21.15.1:0/2482661488 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf300c2f50 con 0x7fcf301eace0 2024-03-06T07:24:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.821+0000 7fcf3755e640 1 -- 172.21.15.1:0/2482661488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf30067fd0 con 0x7fcf301eace0 2024-03-06T07:24:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.823+0000 7fcf25ffb640 1 -- 172.21.15.1:0/2482661488 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16959 (secure 0 0 0) 0x7fcf28057f80 con 0x7fcf301eace0 2024-03-06T07:24:39.825 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:39.825 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":109,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:24:21.903206+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:08.484521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.184053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/2433923518":"2024-03-07T07:24:21.903185+0000","172.21.15.1:6832/2433923518":"2024-03-07T07:24:21.903185+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:39.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.825+0000 7fcf3755e640 1 -- 172.21.15.1:0/2482661488 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf1406b4d0 msgr2=0x7fcf1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.825+0000 7fcf3755e640 1 --2- 172.21.15.1:0/2482661488 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf1406b4d0 0x7fcf1406d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c036000 tx=0x7fcf2c03d040 comp rx=0 tx=0).stop 2024-03-06T07:24:39.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.825+0000 7fcf3755e640 1 -- 172.21.15.1:0/2482661488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf301eace0 msgr2=0x7fcf301eb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.825+0000 7fcf3755e640 1 --2- 172.21.15.1:0/2482661488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf301eace0 0x7fcf301eb190 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800c700 tx=0x7fcf2800cbc0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.825+0000 7fcf3755e640 1 -- 172.21.15.1:0/2482661488 shutdown_connections 2024-03-06T07:24:39.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.825+0000 7fcf3755e640 1 --2- 172.21.15.1:0/2482661488 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf1406b4d0 0x7fcf1406d990 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.825+0000 7fcf3755e640 1 --2- 172.21.15.1:0/2482661488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf301eace0 0x7fcf301eb190 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.825+0000 7fcf3755e640 1 --2- 172.21.15.1:0/2482661488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf30152650 0x7fcf3014db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.825+0000 7fcf3755e640 1 --2- 172.21.15.1:0/2482661488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf300bf9d0 0x7fcf30156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.825+0000 7fcf3755e640 1 -- 172.21.15.1:0/2482661488 >> 172.21.15.1:0/2482661488 conn(0x7fcf30066ee0 msgr2=0x7fcf300c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.825+0000 7fcf3755e640 1 -- 172.21.15.1:0/2482661488 shutdown_connections 2024-03-06T07:24:39.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.825+0000 7fcf3755e640 1 -- 172.21.15.1:0/2482661488 wait complete. 2024-03-06T07:24:39.838 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:24:39.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.938+0000 7fc1e62d4640 1 Processor -- start 2024-03-06T07:24:39.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.938+0000 7fc1e62d4640 1 -- start start 2024-03-06T07:24:39.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.939+0000 7fc1e62d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1e0152650 0x7fc1e0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.939+0000 7fc1e62d4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc1e00bf9d0 0x7fc1e00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.939+0000 7fc1e62d4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e00c2fc0 con 0x7fc1e00c6b10 2024-03-06T07:24:39.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.939+0000 7fc1e62d4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e0155800 con 0x7fc1e00bf9d0 2024-03-06T07:24:39.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.939+0000 7fc1e62d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e0158da0 con 0x7fc1e0152650 2024-03-06T07:24:39.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.939+0000 7fc1e4871640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc1e00bf9d0 0x7fc1e00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.939+0000 7fc1dffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1e0152650 0x7fc1e0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.939+0000 7fc1df7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc1e00c6b10 0x7fc1e0158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:47454/0 (socket says 172.21.15.1:47454) 2024-03-06T07:24:39.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.939+0000 7fc1df7fe640 1 -- 172.21.15.1:0/462621613 learned_addr learned my addr 172.21.15.1:0/462621613 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:39.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.940+0000 7fc1e4871640 1 -- 172.21.15.1:0/462621613 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1e0152650 msgr2=0x7fc1e0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:39.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.940+0000 7fc1e4871640 1 --2- 172.21.15.1:0/462621613 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1e0152650 0x7fc1e0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.940+0000 7fc1e4871640 1 -- 172.21.15.1:0/462621613 >> v1:172.21.15.98:6789/0 conn(0x7fc1e00c6b10 legacy=0x7fc1e0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.940+0000 7fc1deffd640 1 -- 172.21.15.1:0/462621613 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961305822 0 0) 0x7fc1e00c2fc0 con 0x7fc1e00c6b10 2024-03-06T07:24:39.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.940+0000 7fc1e4871640 1 -- 172.21.15.1:0/462621613 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e00c6690 con 0x7fc1e00bf9d0 2024-03-06T07:24:39.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.940+0000 7fc1e4871640 1 --2- 172.21.15.1:0/462621613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1e00bf9d0 0x7fc1e00c1460 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8009d90 tx=0x7fc1d802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53ef8808d5db4a4c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.941+0000 7fc1deffd640 1 -- 172.21.15.1:0/462621613 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8039070 con 0x7fc1e00bf9d0 2024-03-06T07:24:39.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.941+0000 7fc1deffd640 1 -- 172.21.15.1:0/462621613 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d802cbc0 con 0x7fc1e00bf9d0 2024-03-06T07:24:39.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.941+0000 7fc1deffd640 1 -- 172.21.15.1:0/462621613 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8034440 con 0x7fc1e00bf9d0 2024-03-06T07:24:39.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.941+0000 7fc1e62d4640 1 -- 172.21.15.1:0/462621613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1e00bf9d0 msgr2=0x7fc1e00c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.941+0000 7fc1e62d4640 1 --2- 172.21.15.1:0/462621613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1e00bf9d0 0x7fc1e00c1460 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8009d90 tx=0x7fc1d802be10 comp rx=0 tx=0).stop 2024-03-06T07:24:39.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.941+0000 7fc1e62d4640 1 -- 172.21.15.1:0/462621613 shutdown_connections 2024-03-06T07:24:39.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.941+0000 7fc1e62d4640 1 --2- 172.21.15.1:0/462621613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1e00bf9d0 0x7fc1e00c1460 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.941+0000 7fc1e62d4640 1 --2- 172.21.15.1:0/462621613 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1e0152650 0x7fc1e0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.941+0000 7fc1e62d4640 1 -- 172.21.15.1:0/462621613 >> 172.21.15.1:0/462621613 conn(0x7fc1e0066ee0 msgr2=0x7fc1e00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.941+0000 7fc1e62d4640 1 -- 172.21.15.1:0/462621613 shutdown_connections 2024-03-06T07:24:39.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.941+0000 7fc1e62d4640 1 -- 172.21.15.1:0/462621613 wait complete. 2024-03-06T07:24:39.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.942+0000 7fc1e62d4640 1 Processor -- start 2024-03-06T07:24:39.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.943+0000 7fc1e62d4640 1 -- start start 2024-03-06T07:24:39.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.943+0000 7fc1e62d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1e00bf9d0 0x7fc1e01e1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.943+0000 7fc1e62d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1e0152650 0x7fc1e014c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.943+0000 7fc1e62d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1e014d670 0x7fc1e014c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.943+0000 7fc1e62d4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e01551b0 con 0x7fc1e014d670 2024-03-06T07:24:39.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.943+0000 7fc1e62d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e00c6fa0 con 0x7fc1e00bf9d0 2024-03-06T07:24:39.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.943+0000 7fc1e62d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e0155490 con 0x7fc1e0152650 2024-03-06T07:24:39.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.943+0000 7fc1e4871640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1e014d670 0x7fc1e014c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.943+0000 7fc1e4871640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1e014d670 0x7fc1e014c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58102/0 (socket says 172.21.15.1:58102) 2024-03-06T07:24:39.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.943+0000 7fc1e4871640 1 -- 172.21.15.1:0/3031428814 learned_addr learned my addr 172.21.15.1:0/3031428814 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:39.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.943+0000 7fc1dffff640 1 --2- 172.21.15.1:0/3031428814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1e00bf9d0 0x7fc1e01e1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.943+0000 7fc1df7fe640 1 --2- 172.21.15.1:0/3031428814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1e0152650 0x7fc1e014c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.944+0000 7fc1df7fe640 1 -- 172.21.15.1:0/3031428814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1e00bf9d0 msgr2=0x7fc1e01e1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.944+0000 7fc1df7fe640 1 --2- 172.21.15.1:0/3031428814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1e00bf9d0 0x7fc1e01e1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.944+0000 7fc1df7fe640 1 -- 172.21.15.1:0/3031428814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1e014d670 msgr2=0x7fc1e014c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.944+0000 7fc1df7fe640 1 --2- 172.21.15.1:0/3031428814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1e014d670 0x7fc1e014c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.944+0000 7fc1df7fe640 1 -- 172.21.15.1:0/3031428814 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e0154ed0 con 0x7fc1e0152650 2024-03-06T07:24:39.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.944+0000 7fc1e4871640 1 --2- 172.21.15.1:0/3031428814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1e014d670 0x7fc1e014c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:39.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.944+0000 7fc1dffff640 1 --2- 172.21.15.1:0/3031428814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1e00bf9d0 0x7fc1e01e1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:39.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.944+0000 7fc1df7fe640 1 --2- 172.21.15.1:0/3031428814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1e0152650 0x7fc1e014c420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc1d400c450 tx=0x7fc1d400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.944+0000 7fc1dcff9640 1 -- 172.21.15.1:0/3031428814 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4018070 con 0x7fc1e0152650 2024-03-06T07:24:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.944+0000 7fc1dcff9640 1 -- 172.21.15.1:0/3031428814 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d40074e0 con 0x7fc1e0152650 2024-03-06T07:24:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.944+0000 7fc1dcff9640 1 -- 172.21.15.1:0/3031428814 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4004a20 con 0x7fc1e0152650 2024-03-06T07:24:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.944+0000 7fc1e62d4640 1 -- 172.21.15.1:0/3031428814 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e014d420 con 0x7fc1e0152650 2024-03-06T07:24:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.944+0000 7fc1e62d4640 1 -- 172.21.15.1:0/3031428814 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e0154910 con 0x7fc1e0152650 2024-03-06T07:24:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.945+0000 7fc1e62d4640 1 -- 172.21.15.1:0/3031428814 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e006b540 con 0x7fc1e0152650 2024-03-06T07:24:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.946+0000 7fc1dcff9640 1 -- 172.21.15.1:0/3031428814 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1d401b070 con 0x7fc1e0152650 2024-03-06T07:24:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.946+0000 7fc1dcff9640 1 --2- 172.21.15.1:0/3031428814 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc1cc06b4d0 0x7fc1cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.946+0000 7fc1dcff9640 1 -- 172.21.15.1:0/3031428814 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc1d40895e0 con 0x7fc1e0152650 2024-03-06T07:24:39.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.946+0000 7fc1dffff640 1 --2- 172.21.15.1:0/3031428814 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc1cc06b4d0 0x7fc1cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.947+0000 7fc1dffff640 1 --2- 172.21.15.1:0/3031428814 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc1cc06b4d0 0x7fc1cc06d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc1e00ca6e0 tx=0x7fc1d0009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:39.951+0000 7fc1dcff9640 1 -- 172.21.15.1:0/3031428814 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1e006b540 con 0x7fc1e0152650 2024-03-06T07:24:40.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.174+0000 7fc1e62d4640 1 -- 172.21.15.1:0/3031428814 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc1e00b06f0 con 0x7fc1e0152650 2024-03-06T07:24:40.177 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:24:40.175+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:24:40.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.179+0000 7fc1dcff9640 1 -- 172.21.15.1:0/3031428814 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v99) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc1d4057c30 con 0x7fc1e0152650 2024-03-06T07:24:40.181 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:24:40.183 INFO:tasks.ceph.mds.b.smithi098.stderr: -599> 2024-03-06T07:24:26.143+0000 7f9608b23640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.182+0000 7fc1e62d4640 1 -- 172.21.15.1:0/3031428814 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc1cc06b4d0 msgr2=0x7fc1cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.182+0000 7fc1e62d4640 1 --2- 172.21.15.1:0/3031428814 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc1cc06b4d0 0x7fc1cc06d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc1e00ca6e0 tx=0x7fc1d0009190 comp rx=0 tx=0).stop 2024-03-06T07:24:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.182+0000 7fc1e62d4640 1 -- 172.21.15.1:0/3031428814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1e0152650 msgr2=0x7fc1e014c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.182+0000 7fc1e62d4640 1 --2- 172.21.15.1:0/3031428814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1e0152650 0x7fc1e014c420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc1d400c450 tx=0x7fc1d400c910 comp rx=0 tx=0).stop 2024-03-06T07:24:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.182+0000 7fc1e62d4640 1 -- 172.21.15.1:0/3031428814 shutdown_connections 2024-03-06T07:24:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.182+0000 7fc1e62d4640 1 --2- 172.21.15.1:0/3031428814 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc1cc06b4d0 0x7fc1cc06d990 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.182+0000 7fc1e62d4640 1 --2- 172.21.15.1:0/3031428814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1e014d670 0x7fc1e014c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.182+0000 7fc1e62d4640 1 --2- 172.21.15.1:0/3031428814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc1e0152650 0x7fc1e014c420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.182+0000 7fc1e62d4640 1 --2- 172.21.15.1:0/3031428814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1e00bf9d0 0x7fc1e01e1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.182+0000 7fc1e62d4640 1 -- 172.21.15.1:0/3031428814 >> 172.21.15.1:0/3031428814 conn(0x7fc1e0066ee0 msgr2=0x7fc1e014e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:40.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.182+0000 7fc1e62d4640 1 -- 172.21.15.1:0/3031428814 shutdown_connections 2024-03-06T07:24:40.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.182+0000 7fc1e62d4640 1 -- 172.21.15.1:0/3031428814 wait complete. 2024-03-06T07:24:40.195 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:24:40.205 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:24:40.206 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:24:40.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.297+0000 7fc64d30c640 1 Processor -- start 2024-03-06T07:24:40.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.297+0000 7fc64d30c640 1 -- start start 2024-03-06T07:24:40.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.298+0000 7fc64d30c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc648150520 0x7fc6481543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.298+0000 7fc64d30c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc648150900 con 0x7fc6480c32f0 2024-03-06T07:24:40.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.298+0000 7fc64d30c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc648071790 con 0x7fc648070c80 2024-03-06T07:24:40.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.298+0000 7fc64d30c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc648153ed0 con 0x7fc648150520 2024-03-06T07:24:40.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.298+0000 7fc6467fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc6480c32f0 0x7fc6481537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59154/0 (socket says 172.21.15.1:59154) 2024-03-06T07:24:40.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.298+0000 7fc6467fc640 1 -- 172.21.15.1:0/1863067013 learned_addr learned my addr 172.21.15.1:0/1863067013 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:40.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.298+0000 7fc646ffd640 1 --2- 172.21.15.1:0/1863067013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc648150520 0x7fc6481543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.298+0000 7fc645ffb640 1 -- 172.21.15.1:0/1863067013 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389257632 0 0) 0x7fc648150900 con 0x7fc6480c32f0 2024-03-06T07:24:40.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.298+0000 7fc645ffb640 1 -- 172.21.15.1:0/1863067013 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc634003520 con 0x7fc6480c32f0 2024-03-06T07:24:40.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.298+0000 7fc645ffb640 1 -- 172.21.15.1:0/1863067013 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763941181 0 0) 0x7fc648071790 con 0x7fc648070c80 2024-03-06T07:24:40.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.298+0000 7fc645ffb640 1 -- 172.21.15.1:0/1863067013 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc648150900 con 0x7fc648070c80 2024-03-06T07:24:40.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.298+0000 7fc646ffd640 1 -- 172.21.15.1:0/1863067013 >> v1:172.21.15.98:6789/0 conn(0x7fc648070c80 legacy=0x7fc64815d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.298+0000 7fc646ffd640 1 -- 172.21.15.1:0/1863067013 >> v1:172.21.15.1:6789/0 conn(0x7fc6480c32f0 legacy=0x7fc6481537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.299+0000 7fc646ffd640 1 -- 172.21.15.1:0/1863067013 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc648157050 con 0x7fc648150520 2024-03-06T07:24:40.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.299+0000 7fc646ffd640 1 --2- 172.21.15.1:0/1863067013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc648150520 0x7fc6481543a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc6400097d0 tx=0x7fc64002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=758a2f55b02c71ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.299+0000 7fc645ffb640 1 -- 172.21.15.1:0/1863067013 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc640035020 con 0x7fc648150520 2024-03-06T07:24:40.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.299+0000 7fc645ffb640 1 -- 172.21.15.1:0/1863067013 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc640031890 con 0x7fc648150520 2024-03-06T07:24:40.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.299+0000 7fc645ffb640 1 -- 172.21.15.1:0/1863067013 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc640031c60 con 0x7fc648150520 2024-03-06T07:24:40.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.300+0000 7fc64d30c640 1 -- 172.21.15.1:0/1863067013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc648150520 msgr2=0x7fc6481543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.300+0000 7fc64d30c640 1 --2- 172.21.15.1:0/1863067013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc648150520 0x7fc6481543a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc6400097d0 tx=0x7fc64002b930 comp rx=0 tx=0).stop 2024-03-06T07:24:40.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.300+0000 7fc64d30c640 1 -- 172.21.15.1:0/1863067013 shutdown_connections 2024-03-06T07:24:40.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.300+0000 7fc64d30c640 1 --2- 172.21.15.1:0/1863067013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc648150520 0x7fc6481543a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.300+0000 7fc64d30c640 1 -- 172.21.15.1:0/1863067013 >> 172.21.15.1:0/1863067013 conn(0x7fc6480c88f0 msgr2=0x7fc6480c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:40.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.300+0000 7fc64d30c640 1 -- 172.21.15.1:0/1863067013 shutdown_connections 2024-03-06T07:24:40.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.300+0000 7fc64d30c640 1 -- 172.21.15.1:0/1863067013 wait complete. 2024-03-06T07:24:40.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.301+0000 7fc64d30c640 1 Processor -- start 2024-03-06T07:24:40.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.301+0000 7fc64d30c640 1 -- start start 2024-03-06T07:24:40.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc64d30c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc648070c80 0x7fc64814a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc64d30c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6480c32f0 0x7fc6480c4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc64d30c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc648150520 0x7fc6480c2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc64d30c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc648157330 con 0x7fc648070c80 2024-03-06T07:24:40.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc64d30c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6481532a0 con 0x7fc648150520 2024-03-06T07:24:40.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc64d30c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6480c7ec0 con 0x7fc6480c32f0 2024-03-06T07:24:40.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc646ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc648070c80 0x7fc64814a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc646ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc648070c80 0x7fc64814a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58108/0 (socket says 172.21.15.1:58108) 2024-03-06T07:24:40.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc646ffd640 1 -- 172.21.15.1:0/777361587 learned_addr learned my addr 172.21.15.1:0/777361587 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:40.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc6467fc640 1 --2- 172.21.15.1:0/777361587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6480c32f0 0x7fc6480c4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc6477fe640 1 --2- 172.21.15.1:0/777361587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc648150520 0x7fc6480c2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc646ffd640 1 -- 172.21.15.1:0/777361587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6480c32f0 msgr2=0x7fc6480c4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc646ffd640 1 --2- 172.21.15.1:0/777361587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6480c32f0 0x7fc6480c4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc646ffd640 1 -- 172.21.15.1:0/777361587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc648150520 msgr2=0x7fc6480c2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc646ffd640 1 --2- 172.21.15.1:0/777361587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc648150520 0x7fc6480c2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc646ffd640 1 -- 172.21.15.1:0/777361587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6481eecf0 con 0x7fc648070c80 2024-03-06T07:24:40.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc6467fc640 1 --2- 172.21.15.1:0/777361587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6480c32f0 0x7fc6480c4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:40.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc6477fe640 1 --2- 172.21.15.1:0/777361587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc648150520 0x7fc6480c2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:40.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.302+0000 7fc646ffd640 1 --2- 172.21.15.1:0/777361587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc648070c80 0x7fc64814a2e0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fc640000c00 tx=0x7fc6400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.303+0000 7fc63f7fe640 1 -- 172.21.15.1:0/777361587 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc640035040 con 0x7fc648070c80 2024-03-06T07:24:40.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.303+0000 7fc63f7fe640 1 -- 172.21.15.1:0/777361587 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc640030d20 con 0x7fc648070c80 2024-03-06T07:24:40.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.303+0000 7fc64d30c640 1 -- 172.21.15.1:0/777361587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6480ab980 con 0x7fc648070c80 2024-03-06T07:24:40.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.303+0000 7fc63f7fe640 1 -- 172.21.15.1:0/777361587 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6400456e0 con 0x7fc648070c80 2024-03-06T07:24:40.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.303+0000 7fc64d30c640 1 -- 172.21.15.1:0/777361587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6480ad3c0 con 0x7fc648070c80 2024-03-06T07:24:40.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.304+0000 7fc64d30c640 1 -- 172.21.15.1:0/777361587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6480ad1f0 con 0x7fc648070c80 2024-03-06T07:24:40.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.304+0000 7fc63f7fe640 1 -- 172.21.15.1:0/777361587 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc640038020 con 0x7fc648070c80 2024-03-06T07:24:40.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.305+0000 7fc63f7fe640 1 --2- 172.21.15.1:0/777361587 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc62806b4d0 0x7fc62806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.305+0000 7fc63f7fe640 1 -- 172.21.15.1:0/777361587 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fc6400ac730 con 0x7fc648070c80 2024-03-06T07:24:40.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.305+0000 7fc6467fc640 1 --2- 172.21.15.1:0/777361587 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc62806b4d0 0x7fc62806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.305+0000 7fc6467fc640 1 --2- 172.21.15.1:0/777361587 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc62806b4d0 0x7fc62806d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc6481529c0 tx=0x7fc630009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.309+0000 7fc63f7fe640 1 -- 172.21.15.1:0/777361587 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc64002d040 con 0x7fc648070c80 2024-03-06T07:24:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:40.528+0000 7fc64d30c640 1 -- 172.21.15.1:0/777361587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc6480686e0 con 0x7fc648070c80 2024-03-06T07:24:41.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.181+0000 7fc63f7fe640 1 -- 172.21.15.1:0/777361587 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v100) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc64007b380 con 0x7fc648070c80 2024-03-06T07:24:41.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.184+0000 7fc64d30c640 1 -- 172.21.15.1:0/777361587 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc62806b4d0 msgr2=0x7fc62806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.184+0000 7fc64d30c640 1 --2- 172.21.15.1:0/777361587 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc62806b4d0 0x7fc62806d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc6481529c0 tx=0x7fc630009040 comp rx=0 tx=0).stop 2024-03-06T07:24:41.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.184+0000 7fc64d30c640 1 -- 172.21.15.1:0/777361587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc648070c80 msgr2=0x7fc64814a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.184+0000 7fc64d30c640 1 --2- 172.21.15.1:0/777361587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc648070c80 0x7fc64814a2e0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fc640000c00 tx=0x7fc6400046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.184+0000 7fc64d30c640 1 -- 172.21.15.1:0/777361587 shutdown_connections 2024-03-06T07:24:41.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.184+0000 7fc64d30c640 1 --2- 172.21.15.1:0/777361587 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc62806b4d0 0x7fc62806d990 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.184+0000 7fc64d30c640 1 --2- 172.21.15.1:0/777361587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc648150520 0x7fc6480c2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.184+0000 7fc64d30c640 1 --2- 172.21.15.1:0/777361587 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6480c32f0 0x7fc6480c4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.184+0000 7fc64d30c640 1 --2- 172.21.15.1:0/777361587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc648070c80 0x7fc64814a2e0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.184+0000 7fc64d30c640 1 -- 172.21.15.1:0/777361587 >> 172.21.15.1:0/777361587 conn(0x7fc6480c88f0 msgr2=0x7fc6480c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:41.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.184+0000 7fc64d30c640 1 -- 172.21.15.1:0/777361587 shutdown_connections 2024-03-06T07:24:41.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.184+0000 7fc64d30c640 1 -- 172.21.15.1:0/777361587 wait complete. 2024-03-06T07:24:41.200 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:24:41.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.351+0000 7fc7d57f4640 1 Processor -- start 2024-03-06T07:24:41.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.352+0000 7fc7d57f4640 1 -- start start 2024-03-06T07:24:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.352+0000 7fc7d57f4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7d006c3f0 0x7fc7d0158800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.352+0000 7fc7d57f4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc7d0158d40 0x7fc7d006f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.352+0000 7fc7d57f4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d00a9790 con 0x7fc7d0152c00 2024-03-06T07:24:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.352+0000 7fc7d57f4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d00c3990 con 0x7fc7d0158d40 2024-03-06T07:24:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.352+0000 7fc7d57f4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d00a9560 con 0x7fc7d006c3f0 2024-03-06T07:24:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.352+0000 7fc7ceffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc7d0152c00 0x7fc7d0072e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59162/0 (socket says 172.21.15.1:59162) 2024-03-06T07:24:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.352+0000 7fc7ceffd640 1 -- 172.21.15.1:0/763587591 learned_addr learned my addr 172.21.15.1:0/763587591 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cdffb640 1 -- 172.21.15.1:0/763587591 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208807387 0 0) 0x7fc7d00a9790 con 0x7fc7d0152c00 2024-03-06T07:24:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cdffb640 1 -- 172.21.15.1:0/763587591 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b8003520 con 0x7fc7d0152c00 2024-03-06T07:24:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7ce7fc640 1 --2- 172.21.15.1:0/763587591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7d006c3f0 0x7fc7d0158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cf7fe640 1 --2- 172.21.15.1:0/763587591 >> v2:172.21.15.98:3300/0 conn(0x7fc7d0158d40 0x7fc7d006f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cdffb640 1 -- 172.21.15.1:0/763587591 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1292923832 0 0) 0x7fc7b8003520 con 0x7fc7d0152c00 2024-03-06T07:24:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cdffb640 1 -- 172.21.15.1:0/763587591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7d006c3f0 msgr2=0x7fc7d0158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cdffb640 1 --2- 172.21.15.1:0/763587591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7d006c3f0 0x7fc7d0158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cdffb640 1 -- 172.21.15.1:0/763587591 >> v2:172.21.15.98:3300/0 conn(0x7fc7d0158d40 msgr2=0x7fc7d006f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cdffb640 1 --2- 172.21.15.1:0/763587591 >> v2:172.21.15.98:3300/0 conn(0x7fc7d0158d40 0x7fc7d006f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cdffb640 1 -- 172.21.15.1:0/763587591 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d00a9a70 con 0x7fc7d0152c00 2024-03-06T07:24:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cdffb640 1 -- 172.21.15.1:0/763587591 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fc7c0001670 con 0x7fc7d0152c00 2024-03-06T07:24:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cdffb640 1 -- 172.21.15.1:0/763587591 >> v1:172.21.15.1:6789/0 conn(0x7fc7d0152c00 legacy=0x7fc7d0072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cdffb640 1 --2- 172.21.15.1:0/763587591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7d006c3f0 0x7fc7b80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cdffb640 1 --2- 172.21.15.1:0/763587591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b8003f30 0x7fc7b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cdffb640 1 --2- 172.21.15.1:0/763587591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7b8004d40 0x7fc7b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cdffb640 1 -- 172.21.15.1:0/763587591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d00c3990 con 0x7fc7b8003f30 2024-03-06T07:24:41.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cdffb640 1 -- 172.21.15.1:0/763587591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b800b9c0 con 0x7fc7b8004d40 2024-03-06T07:24:41.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cdffb640 1 -- 172.21.15.1:0/763587591 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b800bb90 con 0x7fc7d006c3f0 2024-03-06T07:24:41.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cf7fe640 1 --2- 172.21.15.1:0/763587591 >> v2:172.21.15.98:3300/0 conn(0x7fc7d0158d40 0x7fc7d006f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:41.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7ce7fc640 1 --2- 172.21.15.1:0/763587591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b8003f30 0x7fc7b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7ceffd640 1 --2- 172.21.15.1:0/763587591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7d006c3f0 0x7fc7b80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cf7fe640 1 --2- 172.21.15.1:0/763587591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7b8004d40 0x7fc7b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7ce7fc640 1 -- 172.21.15.1:0/763587591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7d006c3f0 msgr2=0x7fc7b80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:41.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7ce7fc640 1 --2- 172.21.15.1:0/763587591 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7d006c3f0 0x7fc7b80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7ce7fc640 1 -- 172.21.15.1:0/763587591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7b8004d40 msgr2=0x7fc7b80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:41.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7ce7fc640 1 --2- 172.21.15.1:0/763587591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7b8004d40 0x7fc7b80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7ce7fc640 1 -- 172.21.15.1:0/763587591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b8003520 con 0x7fc7b8003f30 2024-03-06T07:24:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.353+0000 7fc7cf7fe640 1 --2- 172.21.15.1:0/763587591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7b8004d40 0x7fc7b80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.354+0000 7fc7ce7fc640 1 --2- 172.21.15.1:0/763587591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b8003f30 0x7fc7b8004800 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009ac0 tx=0x7fc7b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a7255fa070d1b3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.354+0000 7fc7cdffb640 1 -- 172.21.15.1:0/763587591 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4039070 con 0x7fc7b8003f30 2024-03-06T07:24:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.354+0000 7fc7cdffb640 1 -- 172.21.15.1:0/763587591 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c402c6b0 con 0x7fc7b8003f30 2024-03-06T07:24:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.354+0000 7fc7cdffb640 1 -- 172.21.15.1:0/763587591 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4009690 con 0x7fc7b8003f30 2024-03-06T07:24:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.355+0000 7fc7d57f4640 1 -- 172.21.15.1:0/763587591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b8003f30 msgr2=0x7fc7b8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.355+0000 7fc7d57f4640 1 --2- 172.21.15.1:0/763587591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b8003f30 0x7fc7b8004800 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009ac0 tx=0x7fc7b8005830 comp rx=0 tx=0).stop 2024-03-06T07:24:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.355+0000 7fc7cf7fe640 1 -- 172.21.15.1:0/763587591 reap_dead start 2024-03-06T07:24:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.355+0000 7fc7d57f4640 1 -- 172.21.15.1:0/763587591 shutdown_connections 2024-03-06T07:24:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.355+0000 7fc7d57f4640 1 -- 172.21.15.1:0/763587591 >> 172.21.15.1:0/763587591 conn(0x7fc7d00c76b0 msgr2=0x7fc7d00c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.355+0000 7fc7d57f4640 1 -- 172.21.15.1:0/763587591 shutdown_connections 2024-03-06T07:24:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.355+0000 7fc7d57f4640 1 -- 172.21.15.1:0/763587591 wait complete. 2024-03-06T07:24:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.356+0000 7fc7d57f4640 1 Processor -- start 2024-03-06T07:24:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.356+0000 7fc7d57f4640 1 -- start start 2024-03-06T07:24:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.357+0000 7fc7d57f4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b8003f30 0x7fc7d00c0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.357+0000 7fc7d57f4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7d00c5d40 0x7fc7d01edec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.357+0000 7fc7d57f4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7d01ee400 0x7fc7d00c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.357+0000 7fc7d57f4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d006bf60 con 0x7fc7b8003f30 2024-03-06T07:24:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.357+0000 7fc7d57f4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d00aa030 con 0x7fc7d01ee400 2024-03-06T07:24:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.357+0000 7fc7d57f4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d00724a0 con 0x7fc7d00c5d40 2024-03-06T07:24:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.357+0000 7fc7ceffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b8003f30 0x7fc7d00c0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.357+0000 7fc7ceffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b8003f30 0x7fc7d00c0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58130/0 (socket says 172.21.15.1:58130) 2024-03-06T07:24:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.357+0000 7fc7ceffd640 1 -- 172.21.15.1:0/1930190669 learned_addr learned my addr 172.21.15.1:0/1930190669 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.357+0000 7fc7ce7fc640 1 --2- 172.21.15.1:0/1930190669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7d00c5d40 0x7fc7d01edec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.357+0000 7fc7cf7fe640 1 --2- 172.21.15.1:0/1930190669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7d01ee400 0x7fc7d00c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.357+0000 7fc7ceffd640 1 -- 172.21.15.1:0/1930190669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7d00c5d40 msgr2=0x7fc7d01edec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.357+0000 7fc7ceffd640 1 --2- 172.21.15.1:0/1930190669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7d00c5d40 0x7fc7d01edec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.357+0000 7fc7ceffd640 1 -- 172.21.15.1:0/1930190669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7d01ee400 msgr2=0x7fc7d00c3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.357+0000 7fc7ceffd640 1 --2- 172.21.15.1:0/1930190669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7d01ee400 0x7fc7d00c3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.357+0000 7fc7ceffd640 1 -- 172.21.15.1:0/1930190669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d01e09d0 con 0x7fc7b8003f30 2024-03-06T07:24:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.357+0000 7fc7ceffd640 1 --2- 172.21.15.1:0/1930190669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b8003f30 0x7fc7d00c0960 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00072f0 tx=0x7fc7c00077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.358+0000 7fc7b77fe640 1 -- 172.21.15.1:0/1930190669 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c001a070 con 0x7fc7b8003f30 2024-03-06T07:24:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.358+0000 7fc7b77fe640 1 -- 172.21.15.1:0/1930190669 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c000f9c0 con 0x7fc7b8003f30 2024-03-06T07:24:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.358+0000 7fc7d57f4640 1 -- 172.21.15.1:0/1930190669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d0154f90 con 0x7fc7b8003f30 2024-03-06T07:24:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.358+0000 7fc7b77fe640 1 -- 172.21.15.1:0/1930190669 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c000fdb0 con 0x7fc7b8003f30 2024-03-06T07:24:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.358+0000 7fc7d57f4640 1 -- 172.21.15.1:0/1930190669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d00c0ea0 con 0x7fc7b8003f30 2024-03-06T07:24:41.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.359+0000 7fc7d57f4640 1 -- 172.21.15.1:0/1930190669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d00680b0 con 0x7fc7b8003f30 2024-03-06T07:24:41.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.359+0000 7fc7b77fe640 1 -- 172.21.15.1:0/1930190669 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7c001d360 con 0x7fc7b8003f30 2024-03-06T07:24:41.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.360+0000 7fc7b77fe640 1 --2- 172.21.15.1:0/1930190669 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc7bc06b500 0x7fc7bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.360+0000 7fc7b77fe640 1 -- 172.21.15.1:0/1930190669 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fc7c0089d70 con 0x7fc7b8003f30 2024-03-06T07:24:41.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.360+0000 7fc7ce7fc640 1 --2- 172.21.15.1:0/1930190669 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc7bc06b500 0x7fc7bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.360+0000 7fc7ce7fc640 1 --2- 172.21.15.1:0/1930190669 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc7bc06b500 0x7fc7bc06d9c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4036000 tx=0x7fc7c4030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.364+0000 7fc7b77fe640 1 -- 172.21.15.1:0/1930190669 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7c0058a40 con 0x7fc7b8003f30 2024-03-06T07:24:41.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:41.548+0000 7fc7d57f4640 1 -- 172.21.15.1:0/1930190669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc7d00c3b30 con 0x7fc7b8003f30 2024-03-06T07:24:42.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.183+0000 7fc7b77fe640 1 -- 172.21.15.1:0/1930190669 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v111) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc7c0058340 con 0x7fc7b8003f30 2024-03-06T07:24:42.185 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:24:42.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.186+0000 7fc7d57f4640 1 -- 172.21.15.1:0/1930190669 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc7bc06b500 msgr2=0x7fc7bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.186+0000 7fc7d57f4640 1 --2- 172.21.15.1:0/1930190669 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc7bc06b500 0x7fc7bc06d9c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4036000 tx=0x7fc7c4030000 comp rx=0 tx=0).stop 2024-03-06T07:24:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.186+0000 7fc7d57f4640 1 -- 172.21.15.1:0/1930190669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b8003f30 msgr2=0x7fc7d00c0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.186+0000 7fc7d57f4640 1 --2- 172.21.15.1:0/1930190669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b8003f30 0x7fc7d00c0960 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00072f0 tx=0x7fc7c00077b0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.187+0000 7fc7d57f4640 1 -- 172.21.15.1:0/1930190669 shutdown_connections 2024-03-06T07:24:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.187+0000 7fc7d57f4640 1 --2- 172.21.15.1:0/1930190669 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc7bc06b500 0x7fc7bc06d9c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.187+0000 7fc7d57f4640 1 --2- 172.21.15.1:0/1930190669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7d01ee400 0x7fc7d00c3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.187+0000 7fc7d57f4640 1 --2- 172.21.15.1:0/1930190669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7d00c5d40 0x7fc7d01edec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.187+0000 7fc7d57f4640 1 --2- 172.21.15.1:0/1930190669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b8003f30 0x7fc7d00c0960 secure :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00072f0 tx=0x7fc7c00077b0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.187+0000 7fc7d57f4640 1 -- 172.21.15.1:0/1930190669 >> 172.21.15.1:0/1930190669 conn(0x7fc7d00c76b0 msgr2=0x7fc7d01512c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.190+0000 7fc7d57f4640 1 -- 172.21.15.1:0/1930190669 shutdown_connections 2024-03-06T07:24:42.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.190+0000 7fc7d57f4640 1 -- 172.21.15.1:0/1930190669 wait complete. 2024-03-06T07:24:42.207 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:24:42.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.318+0000 7f269aac9640 1 Processor -- start 2024-03-06T07:24:42.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.318+0000 7f269aac9640 1 -- start start 2024-03-06T07:24:42.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.319+0000 7f269aac9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2694152780 0x7f26940c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.319+0000 7f269aac9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940c69b0 con 0x7f26940c1010 2024-03-06T07:24:42.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.319+0000 7f269aac9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2694155370 con 0x7f2694155e50 2024-03-06T07:24:42.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.319+0000 7f269aac9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26940c6c90 con 0x7f2694152780 2024-03-06T07:24:42.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.319+0000 7f2698865640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2694152780 0x7f26940c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.319+0000 7f2693fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f26940c1010 0x7f26940c6fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59172/0 (socket says 172.21.15.1:59172) 2024-03-06T07:24:42.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.319+0000 7f2693fff640 1 -- 172.21.15.1:0/3979312205 learned_addr learned my addr 172.21.15.1:0/3979312205 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:42.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.319+0000 7f2698865640 1 -- 172.21.15.1:0/3979312205 >> v1:172.21.15.98:6789/0 conn(0x7f2694155e50 legacy=0x7f26940cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.319+0000 7f2698865640 1 -- 172.21.15.1:0/3979312205 >> v1:172.21.15.1:6789/0 conn(0x7f26940c1010 legacy=0x7f26940c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.319+0000 7f2698865640 1 -- 172.21.15.1:0/3979312205 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2694155650 con 0x7f2694152780 2024-03-06T07:24:42.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.320+0000 7f2698865640 1 --2- 172.21.15.1:0/3979312205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2694152780 0x7f26940c6000 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f26880097a0 tx=0x7f268802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b032c2aceb9211 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.320+0000 7f26937fe640 1 -- 172.21.15.1:0/3979312205 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26880315a0 con 0x7f2694152780 2024-03-06T07:24:42.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.320+0000 7f26937fe640 1 -- 172.21.15.1:0/3979312205 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2688037070 con 0x7f2694152780 2024-03-06T07:24:42.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.320+0000 7f26937fe640 1 -- 172.21.15.1:0/3979312205 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2688031960 con 0x7f2694152780 2024-03-06T07:24:42.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.321+0000 7f269aac9640 1 -- 172.21.15.1:0/3979312205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2694152780 msgr2=0x7f26940c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.321+0000 7f269aac9640 1 --2- 172.21.15.1:0/3979312205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2694152780 0x7f26940c6000 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f26880097a0 tx=0x7f268802b840 comp rx=0 tx=0).stop 2024-03-06T07:24:42.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.321+0000 7f269aac9640 1 -- 172.21.15.1:0/3979312205 shutdown_connections 2024-03-06T07:24:42.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.321+0000 7f269aac9640 1 --2- 172.21.15.1:0/3979312205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2694152780 0x7f26940c6000 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.321+0000 7f269aac9640 1 -- 172.21.15.1:0/3979312205 >> 172.21.15.1:0/3979312205 conn(0x7f269408b720 msgr2=0x7f269406d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.321+0000 7f269aac9640 1 -- 172.21.15.1:0/3979312205 shutdown_connections 2024-03-06T07:24:42.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.321+0000 7f269aac9640 1 -- 172.21.15.1:0/3979312205 wait complete. 2024-03-06T07:24:42.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.322+0000 7f269aac9640 1 Processor -- start 2024-03-06T07:24:42.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.322+0000 7f269aac9640 1 -- start start 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f269aac9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26940c1010 0x7f26941ee970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f269aac9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2694152780 0x7f2694157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f269aac9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2694155e50 0x7f26941565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f269aac9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2694152b60 con 0x7f26940c1010 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f269aac9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26940c76d0 con 0x7f2694152780 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f269aac9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26940cb210 con 0x7f2694155e50 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f2698865640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26940c1010 0x7f26941ee970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f2698865640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26940c1010 0x7f26941ee970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58140/0 (socket says 172.21.15.1:58140) 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f2698865640 1 -- 172.21.15.1:0/3187758790 learned_addr learned my addr 172.21.15.1:0/3187758790 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f2693fff640 1 --2- 172.21.15.1:0/3187758790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2694152780 0x7f2694157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f2699066640 1 --2- 172.21.15.1:0/3187758790 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2694155e50 0x7f26941565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f2698865640 1 -- 172.21.15.1:0/3187758790 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2694155e50 msgr2=0x7f26941565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f2698865640 1 --2- 172.21.15.1:0/3187758790 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2694155e50 0x7f26941565c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f2698865640 1 -- 172.21.15.1:0/3187758790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2694152780 msgr2=0x7f2694157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f2698865640 1 --2- 172.21.15.1:0/3187758790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2694152780 0x7f2694157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f2698865640 1 -- 172.21.15.1:0/3187758790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2694154600 con 0x7f26940c1010 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f2699066640 1 --2- 172.21.15.1:0/3187758790 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2694155e50 0x7f26941565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f2693fff640 1 --2- 172.21.15.1:0/3187758790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2694152780 0x7f2694157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f2698865640 1 --2- 172.21.15.1:0/3187758790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26940c1010 0x7f26941ee970 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f26880098c0 tx=0x7f26880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f26917fa640 1 -- 172.21.15.1:0/3187758790 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2688032840 con 0x7f26940c1010 2024-03-06T07:24:42.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f26917fa640 1 -- 172.21.15.1:0/3187758790 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2688032a00 con 0x7f26940c1010 2024-03-06T07:24:42.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f26917fa640 1 -- 172.21.15.1:0/3187758790 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2688032df0 con 0x7f26940c1010 2024-03-06T07:24:42.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f269aac9640 1 -- 172.21.15.1:0/3187758790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26941548e0 con 0x7f26940c1010 2024-03-06T07:24:42.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.323+0000 7f269aac9640 1 -- 172.21.15.1:0/3187758790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26941ee450 con 0x7f26940c1010 2024-03-06T07:24:42.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.324+0000 7f269aac9640 1 -- 172.21.15.1:0/3187758790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26940680b0 con 0x7f26940c1010 2024-03-06T07:24:42.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.325+0000 7f26917fa640 1 -- 172.21.15.1:0/3187758790 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f268804c020 con 0x7f26940c1010 2024-03-06T07:24:42.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.325+0000 7f26917fa640 1 --2- 172.21.15.1:0/3187758790 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f267806b500 0x7f267806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.325+0000 7f26917fa640 1 -- 172.21.15.1:0/3187758790 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5835+0+0 (secure 0 0 0) 0x7f268802f020 con 0x7f26940c1010 2024-03-06T07:24:42.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.326+0000 7f2693fff640 1 --2- 172.21.15.1:0/3187758790 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f267806b500 0x7f267806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.326+0000 7f2693fff640 1 --2- 172.21.15.1:0/3187758790 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f267806b500 0x7f267806d9c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f269414b390 tx=0x7f2680009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.330+0000 7f26917fa640 1 -- 172.21.15.1:0/3187758790 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2688040070 con 0x7f26940c1010 2024-03-06T07:24:42.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:42.511+0000 7f269aac9640 1 -- 172.21.15.1:0/3187758790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f26940c5de0 con 0x7f26940c1010 2024-03-06T07:24:43.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.187+0000 7f26917fa640 1 -- 172.21.15.1:0/3187758790 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v112) v1 ==== 169+0+0 (secure 0 0 0) 0x7f268807acf0 con 0x7f26940c1010 2024-03-06T07:24:43.189 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-06T07:24:43.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.193+0000 7f269aac9640 1 -- 172.21.15.1:0/3187758790 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f267806b500 msgr2=0x7f267806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.193+0000 7f269aac9640 1 --2- 172.21.15.1:0/3187758790 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f267806b500 0x7f267806d9c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f269414b390 tx=0x7f2680009040 comp rx=0 tx=0).stop 2024-03-06T07:24:43.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.193+0000 7f269aac9640 1 -- 172.21.15.1:0/3187758790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26940c1010 msgr2=0x7f26941ee970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.193+0000 7f269aac9640 1 --2- 172.21.15.1:0/3187758790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26940c1010 0x7f26941ee970 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f26880098c0 tx=0x7f26880046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.193+0000 7f269aac9640 1 -- 172.21.15.1:0/3187758790 shutdown_connections 2024-03-06T07:24:43.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.193+0000 7f269aac9640 1 --2- 172.21.15.1:0/3187758790 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f267806b500 0x7f267806d9c0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.193+0000 7f269aac9640 1 --2- 172.21.15.1:0/3187758790 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2694155e50 0x7f26941565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.193+0000 7f269aac9640 1 --2- 172.21.15.1:0/3187758790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2694152780 0x7f2694157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.193+0000 7f269aac9640 1 --2- 172.21.15.1:0/3187758790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26940c1010 0x7f26941ee970 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.193+0000 7f269aac9640 1 -- 172.21.15.1:0/3187758790 >> 172.21.15.1:0/3187758790 conn(0x7f269408b720 msgr2=0x7f26940c78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.193+0000 7f269aac9640 1 -- 172.21.15.1:0/3187758790 shutdown_connections 2024-03-06T07:24:43.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.194+0000 7f269aac9640 1 -- 172.21.15.1:0/3187758790 wait complete. 2024-03-06T07:24:43.210 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:24:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.314+0000 7fb7f4cd4640 1 Processor -- start 2024-03-06T07:24:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.314+0000 7fb7f4cd4640 1 -- start start 2024-03-06T07:24:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.315+0000 7fb7f4cd4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb7f01505b0 0x7fb7f00c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.315+0000 7fb7f4cd4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7f00c16e0 0x7fb7f0156df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.315+0000 7fb7f4cd4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f006f5c0 con 0x7fb7f00c1260 2024-03-06T07:24:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.315+0000 7fb7f4cd4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f0152f80 con 0x7fb7f01505b0 2024-03-06T07:24:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.315+0000 7fb7f4cd4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f0150990 con 0x7fb7f00c16e0 2024-03-06T07:24:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.315+0000 7fb7edd9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb7f00c1260 0x7fb7f01566e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59178/0 (socket says 172.21.15.1:59178) 2024-03-06T07:24:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.315+0000 7fb7edd9b640 1 -- 172.21.15.1:0/3754315784 learned_addr learned my addr 172.21.15.1:0/3754315784 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.315+0000 7fb7ed59a640 1 -- 172.21.15.1:0/3754315784 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9045532 0 0) 0x7fb7f006f5c0 con 0x7fb7f00c1260 2024-03-06T07:24:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.315+0000 7fb7eed9d640 1 --2- 172.21.15.1:0/3754315784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7f00c16e0 0x7fb7f0156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.315+0000 7fb7ee59c640 1 --2- 172.21.15.1:0/3754315784 >> v2:172.21.15.98:3300/0 conn(0x7fb7f01505b0 0x7fb7f00c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.315+0000 7fb7ed59a640 1 -- 172.21.15.1:0/3754315784 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7dc003520 con 0x7fb7f00c1260 2024-03-06T07:24:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.315+0000 7fb7ed59a640 1 -- 172.21.15.1:0/3754315784 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1335187564 0 0) 0x7fb7dc003520 con 0x7fb7f00c1260 2024-03-06T07:24:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ed59a640 1 -- 172.21.15.1:0/3754315784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7f00c16e0 msgr2=0x7fb7f0156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ed59a640 1 --2- 172.21.15.1:0/3754315784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7f00c16e0 0x7fb7f0156df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ed59a640 1 -- 172.21.15.1:0/3754315784 >> v2:172.21.15.98:3300/0 conn(0x7fb7f01505b0 msgr2=0x7fb7f00c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ed59a640 1 --2- 172.21.15.1:0/3754315784 >> v2:172.21.15.98:3300/0 conn(0x7fb7f01505b0 0x7fb7f00c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ed59a640 1 -- 172.21.15.1:0/3754315784 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f0153240 con 0x7fb7f00c1260 2024-03-06T07:24:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ed59a640 1 -- 172.21.15.1:0/3754315784 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fb7d8001670 con 0x7fb7f00c1260 2024-03-06T07:24:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ed59a640 1 -- 172.21.15.1:0/3754315784 >> v1:172.21.15.1:6789/0 conn(0x7fb7f00c1260 legacy=0x7fb7f01566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ed59a640 1 --2- 172.21.15.1:0/3754315784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7dc0039b0 0x7fb7dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ed59a640 1 --2- 172.21.15.1:0/3754315784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7dc004800 0x7fb7dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ed59a640 1 --2- 172.21.15.1:0/3754315784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7f00c16e0 0x7fb7dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ed59a640 1 -- 172.21.15.1:0/3754315784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f0152f80 con 0x7fb7dc0039b0 2024-03-06T07:24:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ed59a640 1 -- 172.21.15.1:0/3754315784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f0150990 con 0x7fb7dc004800 2024-03-06T07:24:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ed59a640 1 -- 172.21.15.1:0/3754315784 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc00b9c0 con 0x7fb7f00c16e0 2024-03-06T07:24:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ee59c640 1 --2- 172.21.15.1:0/3754315784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7dc0039b0 0x7fb7dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7edd9b640 1 --2- 172.21.15.1:0/3754315784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7dc004800 0x7fb7dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7eed9d640 1 --2- 172.21.15.1:0/3754315784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7f00c16e0 0x7fb7dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ee59c640 1 -- 172.21.15.1:0/3754315784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7f00c16e0 msgr2=0x7fb7dc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ee59c640 1 --2- 172.21.15.1:0/3754315784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7f00c16e0 0x7fb7dc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ee59c640 1 -- 172.21.15.1:0/3754315784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7dc004800 msgr2=0x7fb7dc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ee59c640 1 --2- 172.21.15.1:0/3754315784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7dc004800 0x7fb7dc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7ee59c640 1 -- 172.21.15.1:0/3754315784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc003520 con 0x7fb7dc0039b0 2024-03-06T07:24:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.316+0000 7fb7eed9d640 1 --2- 172.21.15.1:0/3754315784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7f00c16e0 0x7fb7dc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.317+0000 7fb7ee59c640 1 --2- 172.21.15.1:0/3754315784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7dc0039b0 0x7fb7dc003dd0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800e170 tx=0x7fb7dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=746d809130735e3b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.317+0000 7fb7ed59a640 1 -- 172.21.15.1:0/3754315784 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8039070 con 0x7fb7dc0039b0 2024-03-06T07:24:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.317+0000 7fb7ed59a640 1 -- 172.21.15.1:0/3754315784 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e802cc20 con 0x7fb7dc0039b0 2024-03-06T07:24:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.317+0000 7fb7ed59a640 1 -- 172.21.15.1:0/3754315784 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8009c10 con 0x7fb7dc0039b0 2024-03-06T07:24:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.317+0000 7fb7f4cd4640 1 -- 172.21.15.1:0/3754315784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7dc0039b0 msgr2=0x7fb7dc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.317+0000 7fb7f4cd4640 1 --2- 172.21.15.1:0/3754315784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7dc0039b0 0x7fb7dc003dd0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800e170 tx=0x7fb7dc005830 comp rx=0 tx=0).stop 2024-03-06T07:24:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.317+0000 7fb7eed9d640 1 -- 172.21.15.1:0/3754315784 reap_dead start 2024-03-06T07:24:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.318+0000 7fb7f4cd4640 1 -- 172.21.15.1:0/3754315784 shutdown_connections 2024-03-06T07:24:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.318+0000 7fb7f4cd4640 1 -- 172.21.15.1:0/3754315784 >> 172.21.15.1:0/3754315784 conn(0x7fb7f00c6860 msgr2=0x7fb7f00c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.318+0000 7fb7f4cd4640 1 -- 172.21.15.1:0/3754315784 shutdown_connections 2024-03-06T07:24:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.318+0000 7fb7f4cd4640 1 -- 172.21.15.1:0/3754315784 wait complete. 2024-03-06T07:24:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.319+0000 7fb7f4cd4640 1 Processor -- start 2024-03-06T07:24:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.319+0000 7fb7f4cd4640 1 -- start start 2024-03-06T07:24:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.319+0000 7fb7f4cd4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7dc0039b0 0x7fb7f01e8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7f4cd4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7f01555f0 0x7fb7f0147800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7f4cd4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f0147d40 0x7fb7f014b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7f4cd4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f0153ae0 con 0x7fb7f01555f0 2024-03-06T07:24:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7f4cd4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f0153800 con 0x7fb7f0147d40 2024-03-06T07:24:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7f4cd4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f00c4ca0 con 0x7fb7dc0039b0 2024-03-06T07:24:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7edd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7f01555f0 0x7fb7f0147800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7edd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7f01555f0 0x7fb7f0147800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58162/0 (socket says 172.21.15.1:58162) 2024-03-06T07:24:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7edd9b640 1 -- 172.21.15.1:0/2844133372 learned_addr learned my addr 172.21.15.1:0/2844133372 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7eed9d640 1 --2- 172.21.15.1:0/2844133372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f0147d40 0x7fb7f014b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7ee59c640 1 --2- 172.21.15.1:0/2844133372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7dc0039b0 0x7fb7f01e8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7edd9b640 1 -- 172.21.15.1:0/2844133372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7dc0039b0 msgr2=0x7fb7f01e8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7edd9b640 1 --2- 172.21.15.1:0/2844133372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7dc0039b0 0x7fb7f01e8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7edd9b640 1 -- 172.21.15.1:0/2844133372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f0147d40 msgr2=0x7fb7f014b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7edd9b640 1 --2- 172.21.15.1:0/2844133372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f0147d40 0x7fb7f014b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7edd9b640 1 -- 172.21.15.1:0/2844133372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f014a830 con 0x7fb7f01555f0 2024-03-06T07:24:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7ee59c640 1 --2- 172.21.15.1:0/2844133372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7dc0039b0 0x7fb7f01e8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7edd9b640 1 --2- 172.21.15.1:0/2844133372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7f01555f0 0x7fb7f0147800 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8007370 tx=0x7fb7d8007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7e6ffd640 1 -- 172.21.15.1:0/2844133372 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d800f960 con 0x7fb7f01555f0 2024-03-06T07:24:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7e6ffd640 1 -- 172.21.15.1:0/2844133372 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d8018070 con 0x7fb7f01555f0 2024-03-06T07:24:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7f4cd4640 1 -- 172.21.15.1:0/2844133372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f00a89d0 con 0x7fb7f01555f0 2024-03-06T07:24:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7e6ffd640 1 -- 172.21.15.1:0/2844133372 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d800fd40 con 0x7fb7f01555f0 2024-03-06T07:24:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.320+0000 7fb7f4cd4640 1 -- 172.21.15.1:0/2844133372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f014aaf0 con 0x7fb7f01555f0 2024-03-06T07:24:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.321+0000 7fb7f4cd4640 1 -- 172.21.15.1:0/2844133372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f0067fd0 con 0x7fb7f01555f0 2024-03-06T07:24:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.322+0000 7fb7e6ffd640 1 -- 172.21.15.1:0/2844133372 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7d801d460 con 0x7fb7f01555f0 2024-03-06T07:24:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.322+0000 7fb7e6ffd640 1 --2- 172.21.15.1:0/2844133372 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7d006b4d0 0x7fb7d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.322+0000 7fb7e6ffd640 1 -- 172.21.15.1:0/2844133372 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fb7d808a100 con 0x7fb7f01555f0 2024-03-06T07:24:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.322+0000 7fb7ee59c640 1 --2- 172.21.15.1:0/2844133372 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7d006b4d0 0x7fb7d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.323+0000 7fb7ee59c640 1 --2- 172.21.15.1:0/2844133372 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7d006b4d0 0x7fb7d006d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8036000 tx=0x7fb7e802e000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.326+0000 7fb7e6ffd640 1 -- 172.21.15.1:0/2844133372 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7d8059130 con 0x7fb7f01555f0 2024-03-06T07:24:43.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.510+0000 7fb7f4cd4640 1 -- 172.21.15.1:0/2844133372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb7f01563b0 con 0x7fb7f01555f0 2024-03-06T07:24:43.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.513+0000 7fb7e6ffd640 1 -- 172.21.15.1:0/2844133372 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb7d8058a30 con 0x7fb7f01555f0 2024-03-06T07:24:43.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.515+0000 7fb7f4cd4640 1 -- 172.21.15.1:0/2844133372 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7d006b4d0 msgr2=0x7fb7d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.515+0000 7fb7f4cd4640 1 --2- 172.21.15.1:0/2844133372 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7d006b4d0 0x7fb7d006d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8036000 tx=0x7fb7e802e000 comp rx=0 tx=0).stop 2024-03-06T07:24:43.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.515+0000 7fb7f4cd4640 1 -- 172.21.15.1:0/2844133372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7f01555f0 msgr2=0x7fb7f0147800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.515+0000 7fb7f4cd4640 1 --2- 172.21.15.1:0/2844133372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7f01555f0 0x7fb7f0147800 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8007370 tx=0x7fb7d8007830 comp rx=0 tx=0).stop 2024-03-06T07:24:43.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.516+0000 7fb7f4cd4640 1 -- 172.21.15.1:0/2844133372 shutdown_connections 2024-03-06T07:24:43.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.516+0000 7fb7f4cd4640 1 --2- 172.21.15.1:0/2844133372 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7d006b4d0 0x7fb7d006d990 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.516+0000 7fb7f4cd4640 1 --2- 172.21.15.1:0/2844133372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f0147d40 0x7fb7f014b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.516+0000 7fb7f4cd4640 1 --2- 172.21.15.1:0/2844133372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7f01555f0 0x7fb7f0147800 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.516+0000 7fb7f4cd4640 1 --2- 172.21.15.1:0/2844133372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7dc0039b0 0x7fb7f01e8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.516+0000 7fb7f4cd4640 1 -- 172.21.15.1:0/2844133372 >> 172.21.15.1:0/2844133372 conn(0x7fb7f00c6860 msgr2=0x7fb7f014a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.516+0000 7fb7f4cd4640 1 -- 172.21.15.1:0/2844133372 shutdown_connections 2024-03-06T07:24:43.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.516+0000 7fb7f4cd4640 1 -- 172.21.15.1:0/2844133372 wait complete. 2024-03-06T07:24:43.530 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-03-06T07:24:43.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.631+0000 7f0990c4e640 1 Processor -- start 2024-03-06T07:24:43.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.631+0000 7f0990c4e640 1 -- start start 2024-03-06T07:24:43.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.632+0000 7f0990c4e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f098c0c5850 0x7f098c0c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.632+0000 7f0990c4e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f098c155dc0 con 0x7f098c155220 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.632+0000 7f0990c4e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f098c158910 con 0x7f098c152920 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.632+0000 7f0990c4e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098c152d00 con 0x7f098c0c5850 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.632+0000 7f098ad9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f098c155220 0x7f098c1556b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59186/0 (socket says 172.21.15.1:59186) 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.632+0000 7f098ad9d640 1 -- 172.21.15.1:0/3357185783 learned_addr learned my addr 172.21.15.1:0/3357185783 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.632+0000 7f098959a640 1 -- 172.21.15.1:0/3357185783 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369433043 0 0) 0x7f098c155dc0 con 0x7f098c155220 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.632+0000 7f0989d9b640 1 --2- 172.21.15.1:0/3357185783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f098c0c5850 0x7f098c0c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.632+0000 7f098959a640 1 -- 172.21.15.1:0/3357185783 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0978003520 con 0x7f098c155220 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098959a640 1 -- 172.21.15.1:0/3357185783 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 360355010 0 0) 0x7f0978003520 con 0x7f098c155220 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098959a640 1 -- 172.21.15.1:0/3357185783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f098c0c5850 msgr2=0x7f098c0c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098959a640 1 --2- 172.21.15.1:0/3357185783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f098c0c5850 0x7f098c0c5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098959a640 1 -- 172.21.15.1:0/3357185783 >> v1:172.21.15.98:6789/0 conn(0x7f098c152920 legacy=0x7f098c06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098959a640 1 -- 172.21.15.1:0/3357185783 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098c158bf0 con 0x7f098c155220 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098959a640 1 -- 172.21.15.1:0/3357185783 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f097c0019f0 con 0x7f098c155220 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098959a640 1 -- 172.21.15.1:0/3357185783 >> v1:172.21.15.1:6789/0 conn(0x7f098c155220 legacy=0x7f098c1556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098959a640 1 --2- 172.21.15.1:0/3357185783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09780039e0 0x7f0978003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098959a640 1 --2- 172.21.15.1:0/3357185783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0978004800 0x7f0978004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098959a640 1 --2- 172.21.15.1:0/3357185783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f098c0c5850 0x7f0978005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098959a640 1 -- 172.21.15.1:0/3357185783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098c152d00 con 0x7f09780039e0 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098959a640 1 -- 172.21.15.1:0/3357185783 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097800bb50 con 0x7f0978004800 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098959a640 1 -- 172.21.15.1:0/3357185783 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097800bd50 con 0x7f098c0c5850 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098a59c640 1 --2- 172.21.15.1:0/3357185783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09780039e0 0x7f0978003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098ad9d640 1 --2- 172.21.15.1:0/3357185783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f098c0c5850 0x7f0978005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f0989d9b640 1 --2- 172.21.15.1:0/3357185783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0978004800 0x7f0978004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098a59c640 1 -- 172.21.15.1:0/3357185783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f098c0c5850 msgr2=0x7f0978005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098a59c640 1 --2- 172.21.15.1:0/3357185783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f098c0c5850 0x7f0978005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098a59c640 1 -- 172.21.15.1:0/3357185783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0978004800 msgr2=0x7f0978004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098a59c640 1 --2- 172.21.15.1:0/3357185783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0978004800 0x7f0978004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.633+0000 7f098a59c640 1 -- 172.21.15.1:0/3357185783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0978003520 con 0x7f09780039e0 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.634+0000 7f098a59c640 1 --2- 172.21.15.1:0/3357185783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09780039e0 0x7f0978003e00 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f0984001750 tx=0x7f0978005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec39f459f1061a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.634+0000 7f098959a640 1 -- 172.21.15.1:0/3357185783 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0984001370 con 0x7f09780039e0 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.634+0000 7f098959a640 1 -- 172.21.15.1:0/3357185783 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09840052d0 con 0x7f09780039e0 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.634+0000 7f098959a640 1 -- 172.21.15.1:0/3357185783 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0984003830 con 0x7f09780039e0 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.635+0000 7f0990c4e640 1 -- 172.21.15.1:0/3357185783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09780039e0 msgr2=0x7f0978003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.635+0000 7f0990c4e640 1 --2- 172.21.15.1:0/3357185783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09780039e0 0x7f0978003e00 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f0984001750 tx=0x7f0978005830 comp rx=0 tx=0).stop 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.635+0000 7f098ad9d640 1 -- 172.21.15.1:0/3357185783 reap_dead start 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.635+0000 7f0990c4e640 1 -- 172.21.15.1:0/3357185783 shutdown_connections 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.635+0000 7f0990c4e640 1 -- 172.21.15.1:0/3357185783 >> 172.21.15.1:0/3357185783 conn(0x7f098c0a9700 msgr2=0x7f098c0a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.635+0000 7f0990c4e640 1 -- 172.21.15.1:0/3357185783 shutdown_connections 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.635+0000 7f0990c4e640 1 -- 172.21.15.1:0/3357185783 wait complete. 2024-03-06T07:24:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.636+0000 7f0990c4e640 1 Processor -- start 2024-03-06T07:24:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.636+0000 7f0990c4e640 1 -- start start 2024-03-06T07:24:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0990c4e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09780039e0 0x7f098c14b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0990c4e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f098c0acf80 0x7f098c148d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0990c4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f098c151030 0x7f098c149260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0990c4e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098c159240 con 0x7f098c0acf80 2024-03-06T07:24:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0990c4e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098c1596d0 con 0x7f098c151030 2024-03-06T07:24:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0990c4e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098c0c6380 con 0x7f09780039e0 2024-03-06T07:24:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0989d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f098c0acf80 0x7f098c148d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0989d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f098c0acf80 0x7f098c148d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58166/0 (socket says 172.21.15.1:58166) 2024-03-06T07:24:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0989d9b640 1 -- 172.21.15.1:0/2501895841 learned_addr learned my addr 172.21.15.1:0/2501895841 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0989d9b640 1 -- 172.21.15.1:0/2501895841 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09780039e0 msgr2=0x7f098c14b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0989d9b640 1 --2- 172.21.15.1:0/2501895841 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09780039e0 0x7f098c14b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0989d9b640 1 -- 172.21.15.1:0/2501895841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f098c151030 msgr2=0x7f098c149260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0989d9b640 1 --2- 172.21.15.1:0/2501895841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f098c151030 0x7f098c149260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0989d9b640 1 -- 172.21.15.1:0/2501895841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098c0a9350 con 0x7f098c0acf80 2024-03-06T07:24:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0989d9b640 1 --2- 172.21.15.1:0/2501895841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f098c0acf80 0x7f098c148d20 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f0974011940 tx=0x7f0974011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0982ffd640 1 -- 172.21.15.1:0/2501895841 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097400fa10 con 0x7f098c0acf80 2024-03-06T07:24:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0982ffd640 1 -- 172.21.15.1:0/2501895841 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097400fbb0 con 0x7f098c0acf80 2024-03-06T07:24:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0990c4e640 1 -- 172.21.15.1:0/2501895841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f098c0aa9c0 con 0x7f098c0acf80 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0982ffd640 1 -- 172.21.15.1:0/2501895841 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0974014440 con 0x7f098c0acf80 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.637+0000 7f0990c4e640 1 -- 172.21.15.1:0/2501895841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f098c14b770 con 0x7f098c0acf80 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.638+0000 7f0990c4e640 1 -- 172.21.15.1:0/2501895841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f098c1ea830 con 0x7f098c0acf80 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.639+0000 7f0982ffd640 1 -- 172.21.15.1:0/2501895841 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0974012040 con 0x7f098c0acf80 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.639+0000 7f0982ffd640 1 --2- 172.21.15.1:0/2501895841 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f096c06b500 0x7f096c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.639+0000 7f0982ffd640 1 -- 172.21.15.1:0/2501895841 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f097408a470 con 0x7f098c0acf80 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.639+0000 7f098a59c640 1 --2- 172.21.15.1:0/2501895841 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f096c06b500 0x7f096c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.640+0000 7f098a59c640 1 --2- 172.21.15.1:0/2501895841 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f096c06b500 0x7f096c06d9c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0984001750 tx=0x7f098402f000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.643+0000 7f0982ffd640 1 -- 172.21.15.1:0/2501895841 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0974019030 con 0x7f098c0acf80 2024-03-06T07:24:43.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.823+0000 7f0990c4e640 1 -- 172.21.15.1:0/2501895841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7f098c06bdf0 con 0x7f098c0acf80 2024-03-06T07:24:43.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.826+0000 7f0982ffd640 1 -- 172.21.15.1:0/2501895841 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v39) v1 ==== 96+0+0 (secure 0 0 0) 0x7f09740594a0 con 0x7f098c0acf80 2024-03-06T07:24:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.828+0000 7f0990c4e640 1 -- 172.21.15.1:0/2501895841 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f096c06b500 msgr2=0x7f096c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.828+0000 7f0990c4e640 1 --2- 172.21.15.1:0/2501895841 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f096c06b500 0x7f096c06d9c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0984001750 tx=0x7f098402f000 comp rx=0 tx=0).stop 2024-03-06T07:24:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.828+0000 7f0990c4e640 1 -- 172.21.15.1:0/2501895841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f098c0acf80 msgr2=0x7f098c148d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.828+0000 7f0990c4e640 1 --2- 172.21.15.1:0/2501895841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f098c0acf80 0x7f098c148d20 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f0974011940 tx=0x7f0974011e00 comp rx=0 tx=0).stop 2024-03-06T07:24:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.828+0000 7f0990c4e640 1 -- 172.21.15.1:0/2501895841 shutdown_connections 2024-03-06T07:24:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.828+0000 7f0990c4e640 1 --2- 172.21.15.1:0/2501895841 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f096c06b500 0x7f096c06d9c0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.828+0000 7f0990c4e640 1 --2- 172.21.15.1:0/2501895841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f098c151030 0x7f098c149260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.828+0000 7f0990c4e640 1 --2- 172.21.15.1:0/2501895841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f098c0acf80 0x7f098c148d20 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.828+0000 7f0990c4e640 1 --2- 172.21.15.1:0/2501895841 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09780039e0 0x7f098c14b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.828+0000 7f0990c4e640 1 -- 172.21.15.1:0/2501895841 >> 172.21.15.1:0/2501895841 conn(0x7f098c0a9700 msgr2=0x7f098c154740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.828+0000 7f0990c4e640 1 -- 172.21.15.1:0/2501895841 shutdown_connections 2024-03-06T07:24:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.828+0000 7f0990c4e640 1 -- 172.21.15.1:0/2501895841 wait complete. 2024-03-06T07:24:43.841 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.a mds_join_fs 2024-03-06T07:24:43.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.942+0000 7fe6c715c640 1 Processor -- start 2024-03-06T07:24:43.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.942+0000 7fe6c715c640 1 -- start start 2024-03-06T07:24:43.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6c715c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6c00c6730 0x7fe6c00c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6c715c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c00c3300 con 0x7fe6c00c5d10 2024-03-06T07:24:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6c715c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c00c35a0 con 0x7fe6c0152690 2024-03-06T07:24:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6c715c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c00c3880 con 0x7fe6c00c6730 2024-03-06T07:24:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6c56f9640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe6c00c5d10 0x7fe6c0156cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59200/0 (socket says 172.21.15.1:59200) 2024-03-06T07:24:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6c56f9640 1 -- 172.21.15.1:0/3603507066 learned_addr learned my addr 172.21.15.1:0/3603507066 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bffff640 1 --2- 172.21.15.1:0/3603507066 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6c00c6730 0x7fe6c00c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bf7fe640 1 -- 172.21.15.1:0/3603507066 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761481630 0 0) 0x7fe6c00c3300 con 0x7fe6c00c5d10 2024-03-06T07:24:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bf7fe640 1 -- 172.21.15.1:0/3603507066 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6b0003520 con 0x7fe6c00c5d10 2024-03-06T07:24:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bf7fe640 1 -- 172.21.15.1:0/3603507066 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360328784 0 0) 0x7fe6c00c35a0 con 0x7fe6c0152690 2024-03-06T07:24:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bf7fe640 1 -- 172.21.15.1:0/3603507066 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c00c3300 con 0x7fe6c0152690 2024-03-06T07:24:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bf7fe640 1 -- 172.21.15.1:0/3603507066 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 532395051 0 0) 0x7fe6b0003520 con 0x7fe6c00c5d10 2024-03-06T07:24:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bf7fe640 1 -- 172.21.15.1:0/3603507066 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6c00c6730 msgr2=0x7fe6c00c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bf7fe640 1 --2- 172.21.15.1:0/3603507066 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6c00c6730 0x7fe6c00c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bf7fe640 1 -- 172.21.15.1:0/3603507066 >> v1:172.21.15.98:6789/0 conn(0x7fe6c0152690 legacy=0x7fe6c0155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bf7fe640 1 -- 172.21.15.1:0/3603507066 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c00c3b60 con 0x7fe6c00c5d10 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bf7fe640 1 -- 172.21.15.1:0/3603507066 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fe6b80019f0 con 0x7fe6c00c5d10 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bf7fe640 1 -- 172.21.15.1:0/3603507066 >> v1:172.21.15.1:6789/0 conn(0x7fe6c00c5d10 legacy=0x7fe6c0156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bf7fe640 1 --2- 172.21.15.1:0/3603507066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6b0003b70 0x7fe6b0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bf7fe640 1 --2- 172.21.15.1:0/3603507066 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6c00c6730 0x7fe6b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bf7fe640 1 --2- 172.21.15.1:0/3603507066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b0004d40 0x7fe6b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bf7fe640 1 -- 172.21.15.1:0/3603507066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c00c3880 con 0x7fe6b0003b70 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bf7fe640 1 -- 172.21.15.1:0/3603507066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b000bb50 con 0x7fe6b0004d40 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bf7fe640 1 -- 172.21.15.1:0/3603507066 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b000bd20 con 0x7fe6c00c6730 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6c4ef8640 1 --2- 172.21.15.1:0/3603507066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6b0003b70 0x7fe6b0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6c56f9640 1 --2- 172.21.15.1:0/3603507066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b0004d40 0x7fe6b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.943+0000 7fe6bffff640 1 --2- 172.21.15.1:0/3603507066 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6c00c6730 0x7fe6b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.944+0000 7fe6c4ef8640 1 -- 172.21.15.1:0/3603507066 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6c00c6730 msgr2=0x7fe6b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.944+0000 7fe6c4ef8640 1 --2- 172.21.15.1:0/3603507066 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6c00c6730 0x7fe6b0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.944+0000 7fe6c4ef8640 1 -- 172.21.15.1:0/3603507066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b0004d40 msgr2=0x7fe6b00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.944+0000 7fe6c4ef8640 1 --2- 172.21.15.1:0/3603507066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b0004d40 0x7fe6b00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.944+0000 7fe6c4ef8640 1 -- 172.21.15.1:0/3603507066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b0003520 con 0x7fe6b0003b70 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.944+0000 7fe6c4ef8640 1 --2- 172.21.15.1:0/3603507066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6b0003b70 0x7fe6b0003f70 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4001900 tx=0x7fe6b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fb6987f1ad2bc22 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.945+0000 7fe6bf7fe640 1 -- 172.21.15.1:0/3603507066 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4037070 con 0x7fe6b0003b70 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.945+0000 7fe6bf7fe640 1 -- 172.21.15.1:0/3603507066 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b40052d0 con 0x7fe6b0003b70 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.945+0000 7fe6bf7fe640 1 -- 172.21.15.1:0/3603507066 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b40014f0 con 0x7fe6b0003b70 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.945+0000 7fe6c715c640 1 -- 172.21.15.1:0/3603507066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6b0003b70 msgr2=0x7fe6b0003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.945+0000 7fe6c715c640 1 --2- 172.21.15.1:0/3603507066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6b0003b70 0x7fe6b0003f70 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4001900 tx=0x7fe6b0005830 comp rx=0 tx=0).stop 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.945+0000 7fe6c56f9640 1 -- 172.21.15.1:0/3603507066 reap_dead start 2024-03-06T07:24:43.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.945+0000 7fe6c715c640 1 -- 172.21.15.1:0/3603507066 shutdown_connections 2024-03-06T07:24:43.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.945+0000 7fe6c715c640 1 -- 172.21.15.1:0/3603507066 >> 172.21.15.1:0/3603507066 conn(0x7fe6c006b410 msgr2=0x7fe6c006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.945+0000 7fe6c715c640 1 -- 172.21.15.1:0/3603507066 shutdown_connections 2024-03-06T07:24:43.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.945+0000 7fe6c715c640 1 -- 172.21.15.1:0/3603507066 wait complete. 2024-03-06T07:24:43.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.946+0000 7fe6c715c640 1 Processor -- start 2024-03-06T07:24:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.946+0000 7fe6c715c640 1 -- start start 2024-03-06T07:24:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.947+0000 7fe6c715c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b0003b70 0x7fe6c0151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.947+0000 7fe6c715c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6c0155440 0x7fe6c0153370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.947+0000 7fe6c715c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6c01558c0 0x7fe6c01538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.947+0000 7fe6c715c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c0152bf0 con 0x7fe6c01558c0 2024-03-06T07:24:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.947+0000 7fe6c715c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c0152a70 con 0x7fe6b0003b70 2024-03-06T07:24:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.947+0000 7fe6c715c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c00c57d0 con 0x7fe6c0155440 2024-03-06T07:24:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.947+0000 7fe6c56f9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6c01558c0 0x7fe6c01538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.947+0000 7fe6c56f9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6c01558c0 0x7fe6c01538b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58184/0 (socket says 172.21.15.1:58184) 2024-03-06T07:24:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.947+0000 7fe6c56f9640 1 -- 172.21.15.1:0/222000275 learned_addr learned my addr 172.21.15.1:0/222000275 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.948+0000 7fe6c56f9640 1 -- 172.21.15.1:0/222000275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6c0155440 msgr2=0x7fe6c0153370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.948+0000 7fe6c56f9640 1 --2- 172.21.15.1:0/222000275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6c0155440 0x7fe6c0153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.948+0000 7fe6c56f9640 1 -- 172.21.15.1:0/222000275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b0003b70 msgr2=0x7fe6c0151030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.948+0000 7fe6c56f9640 1 --2- 172.21.15.1:0/222000275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b0003b70 0x7fe6c0151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.948+0000 7fe6c56f9640 1 -- 172.21.15.1:0/222000275 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c00ac130 con 0x7fe6c01558c0 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.948+0000 7fe6c56f9640 1 --2- 172.21.15.1:0/222000275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6c01558c0 0x7fe6c01538b0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fe6c00c2c10 tx=0x7fe6b800a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.948+0000 7fe6bd7fa640 1 -- 172.21.15.1:0/222000275 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8017020 con 0x7fe6c01558c0 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.948+0000 7fe6bd7fa640 1 -- 172.21.15.1:0/222000275 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b800ab80 con 0x7fe6c01558c0 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.948+0000 7fe6bd7fa640 1 -- 172.21.15.1:0/222000275 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b80133f0 con 0x7fe6c01558c0 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.948+0000 7fe6c715c640 1 -- 172.21.15.1:0/222000275 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c00ac410 con 0x7fe6c01558c0 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.948+0000 7fe6c715c640 1 -- 172.21.15.1:0/222000275 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c00ad0f0 con 0x7fe6c01558c0 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.949+0000 7fe6c715c640 1 -- 172.21.15.1:0/222000275 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c00acc00 con 0x7fe6c01558c0 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.949+0000 7fe6bd7fa640 1 -- 172.21.15.1:0/222000275 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe6b801a020 con 0x7fe6c01558c0 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.950+0000 7fe6bd7fa640 1 --2- 172.21.15.1:0/222000275 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe6a406b500 0x7fe6a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.950+0000 7fe6bd7fa640 1 -- 172.21.15.1:0/222000275 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fe6b808a720 con 0x7fe6c01558c0 2024-03-06T07:24:43.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.950+0000 7fe6c4ef8640 1 --2- 172.21.15.1:0/222000275 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe6a406b500 0x7fe6a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.951+0000 7fe6c4ef8640 1 --2- 172.21.15.1:0/222000275 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe6a406b500 0x7fe6a406d9c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4034000 tx=0x7fe6b4009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:43.954+0000 7fe6bd7fa640 1 -- 172.21.15.1:0/222000275 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6b8021020 con 0x7fe6c01558c0 2024-03-06T07:24:44.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.138+0000 7fe6c715c640 1 -- 172.21.15.1:0/222000275 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"} v 0) v1 -- 0x7fe6c0067fd0 con 0x7fe6c01558c0 2024-03-06T07:24:44.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.141+0000 7fe6bd7fa640 1 -- 172.21.15.1:0/222000275 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"}]=0 v40) v1 ==== 96+0+0 (secure 0 0 0) 0x7fe6b8059750 con 0x7fe6c01558c0 2024-03-06T07:24:44.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.143+0000 7fe6c715c640 1 -- 172.21.15.1:0/222000275 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe6a406b500 msgr2=0x7fe6a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.143+0000 7fe6c715c640 1 --2- 172.21.15.1:0/222000275 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe6a406b500 0x7fe6a406d9c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4034000 tx=0x7fe6b4009030 comp rx=0 tx=0).stop 2024-03-06T07:24:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.143+0000 7fe6c715c640 1 -- 172.21.15.1:0/222000275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6c01558c0 msgr2=0x7fe6c01538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.143+0000 7fe6c715c640 1 --2- 172.21.15.1:0/222000275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6c01558c0 0x7fe6c01538b0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fe6c00c2c10 tx=0x7fe6b800a600 comp rx=0 tx=0).stop 2024-03-06T07:24:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.143+0000 7fe6c715c640 1 -- 172.21.15.1:0/222000275 shutdown_connections 2024-03-06T07:24:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.143+0000 7fe6c715c640 1 --2- 172.21.15.1:0/222000275 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe6a406b500 0x7fe6a406d9c0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.143+0000 7fe6c715c640 1 --2- 172.21.15.1:0/222000275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6c01558c0 0x7fe6c01538b0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.143+0000 7fe6c715c640 1 --2- 172.21.15.1:0/222000275 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6c0155440 0x7fe6c0153370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.143+0000 7fe6c715c640 1 --2- 172.21.15.1:0/222000275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b0003b70 0x7fe6c0151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.143+0000 7fe6c715c640 1 -- 172.21.15.1:0/222000275 >> 172.21.15.1:0/222000275 conn(0x7fe6c006b410 msgr2=0x7fe6c0071160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.143+0000 7fe6c715c640 1 -- 172.21.15.1:0/222000275 shutdown_connections 2024-03-06T07:24:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.143+0000 7fe6c715c640 1 -- 172.21.15.1:0/222000275 wait complete. 2024-03-06T07:24:44.156 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay' 2024-03-06T07:24:44.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.256+0000 7fee2850d640 1 Processor -- start 2024-03-06T07:24:44.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.256+0000 7fee2850d640 1 -- start start 2024-03-06T07:24:44.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.257+0000 7fee2850d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fee20152650 0x7fee20155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.257+0000 7fee2850d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee200c6b10 0x7fee20158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.257+0000 7fee2850d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee200c2fc0 con 0x7fee200bf9d0 2024-03-06T07:24:44.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.257+0000 7fee2850d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee20155800 con 0x7fee20152650 2024-03-06T07:24:44.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.257+0000 7fee2850d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee200c6690 con 0x7fee200c6b10 2024-03-06T07:24:44.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.257+0000 7fee262a9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fee20152650 0x7fee20155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.257+0000 7fee262a9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fee20152650 0x7fee20155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58200/0 (socket says 172.21.15.1:58200) 2024-03-06T07:24:44.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.257+0000 7fee262a9640 1 -- 172.21.15.1:0/2149381026 learned_addr learned my addr 172.21.15.1:0/2149381026 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:44.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.257+0000 7fee262a9640 1 -- 172.21.15.1:0/2149381026 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee200c6b10 msgr2=0x7fee20158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.257+0000 7fee25aa8640 1 --2- 172.21.15.1:0/2149381026 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee200c6b10 0x7fee20158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.258+0000 7fee262a9640 1 --2- 172.21.15.1:0/2149381026 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee200c6b10 0x7fee20158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.258+0000 7fee262a9640 1 -- 172.21.15.1:0/2149381026 >> v1:172.21.15.98:6789/0 conn(0x7fee200bf9d0 legacy=0x7fee200c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.258+0000 7fee262a9640 1 -- 172.21.15.1:0/2149381026 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee200c7280 con 0x7fee20152650 2024-03-06T07:24:44.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.258+0000 7fee25aa8640 1 --2- 172.21.15.1:0/2149381026 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee200c6b10 0x7fee20158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:44.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.258+0000 7fee262a9640 1 --2- 172.21.15.1:0/2149381026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee20152650 0x7fee20155c90 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fee10009e50 tx=0x7fee1002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb68bd58c52e1427 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.258+0000 7fee252a7640 1 -- 172.21.15.1:0/2149381026 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1002c9e0 con 0x7fee20152650 2024-03-06T07:24:44.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.258+0000 7fee252a7640 1 -- 172.21.15.1:0/2149381026 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1002cb80 con 0x7fee20152650 2024-03-06T07:24:44.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.258+0000 7fee252a7640 1 -- 172.21.15.1:0/2149381026 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee100343f0 con 0x7fee20152650 2024-03-06T07:24:44.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.259+0000 7fee2850d640 1 -- 172.21.15.1:0/2149381026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee20152650 msgr2=0x7fee20155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.259+0000 7fee2850d640 1 --2- 172.21.15.1:0/2149381026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee20152650 0x7fee20155c90 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fee10009e50 tx=0x7fee1002be40 comp rx=0 tx=0).stop 2024-03-06T07:24:44.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.259+0000 7fee2850d640 1 -- 172.21.15.1:0/2149381026 shutdown_connections 2024-03-06T07:24:44.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.259+0000 7fee2850d640 1 --2- 172.21.15.1:0/2149381026 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee200c6b10 0x7fee20158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.259+0000 7fee2850d640 1 --2- 172.21.15.1:0/2149381026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee20152650 0x7fee20155c90 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.259+0000 7fee2850d640 1 -- 172.21.15.1:0/2149381026 >> 172.21.15.1:0/2149381026 conn(0x7fee20066ee0 msgr2=0x7fee200672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.259+0000 7fee2850d640 1 -- 172.21.15.1:0/2149381026 shutdown_connections 2024-03-06T07:24:44.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.259+0000 7fee2850d640 1 -- 172.21.15.1:0/2149381026 wait complete. 2024-03-06T07:24:44.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.260+0000 7fee2850d640 1 Processor -- start 2024-03-06T07:24:44.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.260+0000 7fee2850d640 1 -- start start 2024-03-06T07:24:44.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee2850d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee200bf9d0 0x7fee2014b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee2850d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee20152650 0x7fee200c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee2850d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee2014c410 0x7fee201eb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee2850d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee20155490 con 0x7fee2014c410 2024-03-06T07:24:44.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee2850d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee201551b0 con 0x7fee200bf9d0 2024-03-06T07:24:44.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee2850d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee20152a30 con 0x7fee20152650 2024-03-06T07:24:44.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee26aaa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee2014c410 0x7fee201eb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee26aaa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee2014c410 0x7fee201eb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58216/0 (socket says 172.21.15.1:58216) 2024-03-06T07:24:44.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee26aaa640 1 -- 172.21.15.1:0/3014648553 learned_addr learned my addr 172.21.15.1:0/3014648553 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:44.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee25aa8640 1 --2- 172.21.15.1:0/3014648553 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee20152650 0x7fee200c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee262a9640 1 --2- 172.21.15.1:0/3014648553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee200bf9d0 0x7fee2014b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee26aaa640 1 -- 172.21.15.1:0/3014648553 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee20152650 msgr2=0x7fee200c54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee26aaa640 1 --2- 172.21.15.1:0/3014648553 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee20152650 0x7fee200c54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee26aaa640 1 -- 172.21.15.1:0/3014648553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee200bf9d0 msgr2=0x7fee2014b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee26aaa640 1 --2- 172.21.15.1:0/3014648553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee200bf9d0 0x7fee2014b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee26aaa640 1 -- 172.21.15.1:0/3014648553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee20154e50 con 0x7fee2014c410 2024-03-06T07:24:44.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee262a9640 1 --2- 172.21.15.1:0/3014648553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee200bf9d0 0x7fee2014b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:44.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.261+0000 7fee26aaa640 1 --2- 172.21.15.1:0/3014648553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee2014c410 0x7fee201eb5e0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fee1800b440 tx=0x7fee1800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.262+0000 7fee1effd640 1 -- 172.21.15.1:0/3014648553 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee18015020 con 0x7fee2014c410 2024-03-06T07:24:44.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.262+0000 7fee1effd640 1 -- 172.21.15.1:0/3014648553 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee18011830 con 0x7fee2014c410 2024-03-06T07:24:44.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.262+0000 7fee2850d640 1 -- 172.21.15.1:0/3014648553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee20149f40 con 0x7fee2014c410 2024-03-06T07:24:44.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.262+0000 7fee1effd640 1 -- 172.21.15.1:0/3014648553 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee18011c20 con 0x7fee2014c410 2024-03-06T07:24:44.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.262+0000 7fee2850d640 1 -- 172.21.15.1:0/3014648553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee20154ae0 con 0x7fee2014c410 2024-03-06T07:24:44.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.263+0000 7fee2850d640 1 -- 172.21.15.1:0/3014648553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee201eaf50 con 0x7fee2014c410 2024-03-06T07:24:44.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.263+0000 7fee1effd640 1 -- 172.21.15.1:0/3014648553 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee18018020 con 0x7fee2014c410 2024-03-06T07:24:44.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.264+0000 7fee1effd640 1 --2- 172.21.15.1:0/3014648553 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee0c06b500 0x7fee0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.264+0000 7fee1effd640 1 -- 172.21.15.1:0/3014648553 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fee18089c50 con 0x7fee2014c410 2024-03-06T07:24:44.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.264+0000 7fee262a9640 1 --2- 172.21.15.1:0/3014648553 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee0c06b500 0x7fee0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.264+0000 7fee262a9640 1 --2- 172.21.15.1:0/3014648553 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee0c06b500 0x7fee0c06d9c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fee10035000 tx=0x7fee10030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.268+0000 7fee1effd640 1 -- 172.21.15.1:0/3014648553 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee201eaf50 con 0x7fee2014c410 2024-03-06T07:24:44.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:44.522+0000 7fee2850d640 1 -- 172.21.15.1:0/3014648553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]} v 0) v1 -- 0x7fee200672c0 con 0x7fee2014c410 2024-03-06T07:24:45.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.193+0000 7fee1effd640 1 -- 172.21.15.1:0/3014648553 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]}]=0 v264) v1 ==== 153+0+0 (secure 0 0 0) 0x7fee1808f020 con 0x7fee2014c410 2024-03-06T07:24:45.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.195+0000 7fee2850d640 1 -- 172.21.15.1:0/3014648553 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee0c06b500 msgr2=0x7fee0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.195+0000 7fee2850d640 1 --2- 172.21.15.1:0/3014648553 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee0c06b500 0x7fee0c06d9c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fee10035000 tx=0x7fee10030000 comp rx=0 tx=0).stop 2024-03-06T07:24:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.195+0000 7fee2850d640 1 -- 172.21.15.1:0/3014648553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee2014c410 msgr2=0x7fee201eb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.195+0000 7fee2850d640 1 --2- 172.21.15.1:0/3014648553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee2014c410 0x7fee201eb5e0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fee1800b440 tx=0x7fee1800b900 comp rx=0 tx=0).stop 2024-03-06T07:24:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.195+0000 7fee2850d640 1 -- 172.21.15.1:0/3014648553 shutdown_connections 2024-03-06T07:24:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.195+0000 7fee2850d640 1 --2- 172.21.15.1:0/3014648553 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee0c06b500 0x7fee0c06d9c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.195+0000 7fee2850d640 1 --2- 172.21.15.1:0/3014648553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee2014c410 0x7fee201eb5e0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.195+0000 7fee2850d640 1 --2- 172.21.15.1:0/3014648553 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee20152650 0x7fee200c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.195+0000 7fee2850d640 1 --2- 172.21.15.1:0/3014648553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee200bf9d0 0x7fee2014b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.195+0000 7fee2850d640 1 -- 172.21.15.1:0/3014648553 >> 172.21.15.1:0/3014648553 conn(0x7fee20066ee0 msgr2=0x7fee200c4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.195+0000 7fee2850d640 1 -- 172.21.15.1:0/3014648553 shutdown_connections 2024-03-06T07:24:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.195+0000 7fee2850d640 1 -- 172.21.15.1:0/3014648553 wait complete. 2024-03-06T07:24:45.210 INFO:tasks.cephfs_test_runner:That a standby-replay daemon with weak affinity is replaced by a stronger one. ... ok 2024-03-06T07:24:45.210 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_steady (tasks.cephfs.test_failover.TestClusterAffinity) That a sole MDS with mds_join_fs set will come back as active eventually even after failover. 2024-03-06T07:24:45.210 INFO:tasks.cephfs_test_runner:test_join_fs_steady (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-06T07:24:45.211 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:45.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.315+0000 7f744fb3a640 1 Processor -- start 2024-03-06T07:24:45.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.315+0000 7f744fb3a640 1 -- start start 2024-03-06T07:24:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.315+0000 7f744fb3a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7448152920 0x7f744806aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.315+0000 7f744fb3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f744806b530 0x7f74480c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.315+0000 7f744fb3a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7448158910 con 0x7f74480c5d90 2024-03-06T07:24:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.315+0000 7f744fb3a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7448152d00 con 0x7f7448152920 2024-03-06T07:24:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.315+0000 7f744fb3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74481596d0 con 0x7f744806b530 2024-03-06T07:24:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.316+0000 7f744e0d7640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f74480c5d90 0x7f7448155220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54420/0 (socket says 172.21.15.1:54420) 2024-03-06T07:24:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.316+0000 7f744e0d7640 1 -- 172.21.15.1:0/1405136971 learned_addr learned my addr 172.21.15.1:0/1405136971 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.316+0000 7f744d8d6640 1 --2- 172.21.15.1:0/1405136971 >> v2:172.21.15.98:3300/0 conn(0x7f7448152920 0x7f744806aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.316+0000 7f744d0d5640 1 --2- 172.21.15.1:0/1405136971 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f744806b530 0x7f74480c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.316+0000 7f744c8d4640 1 -- 172.21.15.1:0/1405136971 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699181698 0 0) 0x7f7448158910 con 0x7f74480c5d90 2024-03-06T07:24:45.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.316+0000 7f744c8d4640 1 -- 172.21.15.1:0/1405136971 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7430003520 con 0x7f74480c5d90 2024-03-06T07:24:45.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.316+0000 7f744d8d6640 1 -- 172.21.15.1:0/1405136971 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f744806b530 msgr2=0x7f74480c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:45.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.316+0000 7f744d8d6640 1 --2- 172.21.15.1:0/1405136971 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f744806b530 0x7f74480c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.316+0000 7f744c8d4640 1 -- 172.21.15.1:0/1405136971 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1948794504 0 0) 0x7f7430003520 con 0x7f74480c5d90 2024-03-06T07:24:45.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.316+0000 7f744d8d6640 1 -- 172.21.15.1:0/1405136971 >> v1:172.21.15.1:6789/0 conn(0x7f74480c5d90 legacy=0x7f7448155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.316+0000 7f744d8d6640 1 -- 172.21.15.1:0/1405136971 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7448158bf0 con 0x7f7448152920 2024-03-06T07:24:45.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.317+0000 7f744d8d6640 1 --2- 172.21.15.1:0/1405136971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7448152920 0x7f744806aff0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f74400097d0 tx=0x7f744002b6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5c1dfed9fa500f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.317+0000 7f744c8d4640 1 -- 172.21.15.1:0/1405136971 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7440037070 con 0x7f7448152920 2024-03-06T07:24:45.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.317+0000 7f744c8d4640 1 -- 172.21.15.1:0/1405136971 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7440030510 con 0x7f7448152920 2024-03-06T07:24:45.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.317+0000 7f744c8d4640 1 -- 172.21.15.1:0/1405136971 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74380019f0 con 0x7f7448152920 2024-03-06T07:24:45.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.318+0000 7f744fb3a640 1 -- 172.21.15.1:0/1405136971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7448152920 msgr2=0x7f744806aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.318+0000 7f744fb3a640 1 --2- 172.21.15.1:0/1405136971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7448152920 0x7f744806aff0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f74400097d0 tx=0x7f744002b6e0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.318+0000 7f744fb3a640 1 -- 172.21.15.1:0/1405136971 shutdown_connections 2024-03-06T07:24:45.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.318+0000 7f744fb3a640 1 --2- 172.21.15.1:0/1405136971 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f744806b530 0x7f74480c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.318+0000 7f744fb3a640 1 --2- 172.21.15.1:0/1405136971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7448152920 0x7f744806aff0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.318+0000 7f744fb3a640 1 -- 172.21.15.1:0/1405136971 >> 172.21.15.1:0/1405136971 conn(0x7f74480a9700 msgr2=0x7f74480a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.318+0000 7f744fb3a640 1 -- 172.21.15.1:0/1405136971 shutdown_connections 2024-03-06T07:24:45.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.318+0000 7f744fb3a640 1 -- 172.21.15.1:0/1405136971 wait complete. 2024-03-06T07:24:45.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.319+0000 7f744fb3a640 1 Processor -- start 2024-03-06T07:24:45.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.319+0000 7f744fb3a640 1 -- start start 2024-03-06T07:24:45.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.319+0000 7f744fb3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f744806b530 0x7f7448154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.319+0000 7f744fb3a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74480c5d90 0x7f74481eb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.319+0000 7f744fb3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7448152920 0x7f74481eb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.319+0000 7f744fb3a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74480c6380 con 0x7f74480c5d90 2024-03-06T07:24:45.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.319+0000 7f744fb3a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7448159240 con 0x7f744806b530 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.319+0000 7f744fb3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7448155930 con 0x7f7448152920 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.320+0000 7f744d0d5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74480c5d90 0x7f74481eb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.320+0000 7f744d0d5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74480c5d90 0x7f74481eb4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33428/0 (socket says 172.21.15.1:33428) 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.320+0000 7f744d0d5640 1 -- 172.21.15.1:0/2457912809 learned_addr learned my addr 172.21.15.1:0/2457912809 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.320+0000 7f744d8d6640 1 --2- 172.21.15.1:0/2457912809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f744806b530 0x7f7448154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.320+0000 7f744e0d7640 1 --2- 172.21.15.1:0/2457912809 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7448152920 0x7f74481eb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.320+0000 7f744d0d5640 1 -- 172.21.15.1:0/2457912809 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7448152920 msgr2=0x7f74481eb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.320+0000 7f744d0d5640 1 --2- 172.21.15.1:0/2457912809 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7448152920 0x7f74481eb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.320+0000 7f744d0d5640 1 -- 172.21.15.1:0/2457912809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f744806b530 msgr2=0x7f7448154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.320+0000 7f744d0d5640 1 --2- 172.21.15.1:0/2457912809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f744806b530 0x7f7448154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.320+0000 7f744d0d5640 1 -- 172.21.15.1:0/2457912809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74481ebf30 con 0x7f74480c5d90 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.320+0000 7f744d0d5640 1 --2- 172.21.15.1:0/2457912809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74480c5d90 0x7f74481eb4b0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f743c00e980 tx=0x7f743c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.320+0000 7f74467fc640 1 -- 172.21.15.1:0/2457912809 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c017070 con 0x7f74480c5d90 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.320+0000 7f74467fc640 1 -- 172.21.15.1:0/2457912809 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743c00ca00 con 0x7f74480c5d90 2024-03-06T07:24:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.320+0000 7f744fb3a640 1 -- 172.21.15.1:0/2457912809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74481eafb0 con 0x7f74480c5d90 2024-03-06T07:24:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.320+0000 7f74467fc640 1 -- 172.21.15.1:0/2457912809 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c00cdd0 con 0x7f74480c5d90 2024-03-06T07:24:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.320+0000 7f744fb3a640 1 -- 172.21.15.1:0/2457912809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74481eec90 con 0x7f74480c5d90 2024-03-06T07:24:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.321+0000 7f744fb3a640 1 -- 172.21.15.1:0/2457912809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74480aa4c0 con 0x7f74480c5d90 2024-03-06T07:24:45.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.322+0000 7f74467fc640 1 -- 172.21.15.1:0/2457912809 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f743c01a070 con 0x7f74480c5d90 2024-03-06T07:24:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.322+0000 7f74467fc640 1 --2- 172.21.15.1:0/2457912809 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f743406b4d0 0x7f743406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.322+0000 7f74467fc640 1 -- 172.21.15.1:0/2457912809 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f743c08a000 con 0x7f74480c5d90 2024-03-06T07:24:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.323+0000 7f744d8d6640 1 --2- 172.21.15.1:0/2457912809 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f743406b4d0 0x7f743406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.323+0000 7f744d8d6640 1 --2- 172.21.15.1:0/2457912809 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f743406b4d0 0x7f743406d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7440034000 tx=0x7f7440008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.326+0000 7f74467fc640 1 -- 172.21.15.1:0/2457912809 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74480aa4c0 con 0x7f74480c5d90 2024-03-06T07:24:45.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.509+0000 7f744fb3a640 1 -- 172.21.15.1:0/2457912809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f744806bdf0 con 0x7f74480c5d90 2024-03-06T07:24:45.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.510+0000 7f74467fc640 1 -- 172.21.15.1:0/2457912809 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+13311 (secure 0 0 0) 0x7f74481ebf30 con 0x7f74480c5d90 2024-03-06T07:24:45.512 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:45.512 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":112,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:24:43.184538+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6837/2632566151":"2024-03-07T07:24:40.176718+0000","172.21.15.98:6835/2632566151":"2024-03-07T07:24:40.176718+0000","172.21.15.1:6833/2433923518":"2024-03-07T07:24:21.903185+0000","172.21.15.1:6832/2433923518":"2024-03-07T07:24:21.903185+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:45.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.512+0000 7f744fb3a640 1 -- 172.21.15.1:0/2457912809 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f743406b4d0 msgr2=0x7f743406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.512+0000 7f744fb3a640 1 --2- 172.21.15.1:0/2457912809 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f743406b4d0 0x7f743406d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7440034000 tx=0x7f7440008030 comp rx=0 tx=0).stop 2024-03-06T07:24:45.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.512+0000 7f744fb3a640 1 -- 172.21.15.1:0/2457912809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74480c5d90 msgr2=0x7f74481eb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.512+0000 7f744fb3a640 1 --2- 172.21.15.1:0/2457912809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74480c5d90 0x7f74481eb4b0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f743c00e980 tx=0x7f743c00ee40 comp rx=0 tx=0).stop 2024-03-06T07:24:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.512+0000 7f744fb3a640 1 -- 172.21.15.1:0/2457912809 shutdown_connections 2024-03-06T07:24:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.512+0000 7f744fb3a640 1 --2- 172.21.15.1:0/2457912809 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f743406b4d0 0x7f743406d990 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.512+0000 7f744fb3a640 1 --2- 172.21.15.1:0/2457912809 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7448152920 0x7f74481eb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.512+0000 7f744fb3a640 1 --2- 172.21.15.1:0/2457912809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74480c5d90 0x7f74481eb4b0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.512+0000 7f744fb3a640 1 --2- 172.21.15.1:0/2457912809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f744806b530 0x7f7448154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.512+0000 7f744fb3a640 1 -- 172.21.15.1:0/2457912809 >> 172.21.15.1:0/2457912809 conn(0x7f74480a9700 msgr2=0x7f74481517b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.512+0000 7f744fb3a640 1 -- 172.21.15.1:0/2457912809 shutdown_connections 2024-03-06T07:24:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.512+0000 7f744fb3a640 1 -- 172.21.15.1:0/2457912809 wait complete. 2024-03-06T07:24:45.525 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:45.525 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:45.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.627+0000 7fd9ae881640 1 Processor -- start 2024-03-06T07:24:45.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.627+0000 7fd9ae881640 1 -- start start 2024-03-06T07:24:45.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.628+0000 7fd9ae881640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a8152690 0x7fd9a8155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.628+0000 7fd9ae881640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd9a8156cd0 0x7fd9a8157160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.628+0000 7fd9ae881640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a81561e0 con 0x7fd9a80c6730 2024-03-06T07:24:45.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.628+0000 7fd9ae881640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a80c3300 con 0x7fd9a8156cd0 2024-03-06T07:24:45.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.628+0000 7fd9ae881640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a80c35a0 con 0x7fd9a8152690 2024-03-06T07:24:45.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.628+0000 7fd9a77fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd9a80c6730 0x7fd9a80c57d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54426/0 (socket says 172.21.15.1:54426) 2024-03-06T07:24:45.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.628+0000 7fd9a77fe640 1 -- 172.21.15.1:0/3082996690 learned_addr learned my addr 172.21.15.1:0/3082996690 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:45.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.628+0000 7fd9a7fff640 1 --2- 172.21.15.1:0/3082996690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a8152690 0x7fd9a8155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.628+0000 7fd9ace1e640 1 --2- 172.21.15.1:0/3082996690 >> v2:172.21.15.98:3300/0 conn(0x7fd9a8156cd0 0x7fd9a8157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.628+0000 7fd9a6ffd640 1 -- 172.21.15.1:0/3082996690 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501893957 0 0) 0x7fd99c001110 con 0x7fd9a80c6730 2024-03-06T07:24:45.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.628+0000 7fd9a6ffd640 1 -- 172.21.15.1:0/3082996690 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd990003520 con 0x7fd9a80c6730 2024-03-06T07:24:45.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.628+0000 7fd9a6ffd640 1 -- 172.21.15.1:0/3082996690 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2540626205 0 0) 0x7fd990003520 con 0x7fd9a80c6730 2024-03-06T07:24:45.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.628+0000 7fd9a7fff640 1 -- 172.21.15.1:0/3082996690 >> v2:172.21.15.98:3300/0 conn(0x7fd9a8156cd0 msgr2=0x7fd9a8157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:45.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.628+0000 7fd9a7fff640 1 --2- 172.21.15.1:0/3082996690 >> v2:172.21.15.98:3300/0 conn(0x7fd9a8156cd0 0x7fd9a8157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.628+0000 7fd9a7fff640 1 -- 172.21.15.1:0/3082996690 >> v1:172.21.15.1:6789/0 conn(0x7fd9a80c6730 legacy=0x7fd9a80c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.628+0000 7fd9a7fff640 1 -- 172.21.15.1:0/3082996690 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a80c3880 con 0x7fd9a8152690 2024-03-06T07:24:45.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.628+0000 7fd9ace1e640 1 --2- 172.21.15.1:0/3082996690 >> v2:172.21.15.98:3300/0 conn(0x7fd9a8156cd0 0x7fd9a8157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:45.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.629+0000 7fd9a7fff640 1 --2- 172.21.15.1:0/3082996690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a8152690 0x7fd9a8155ca0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd9980097a0 tx=0x7fd99802b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b81c52db36990ffe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.629+0000 7fd9a6ffd640 1 -- 172.21.15.1:0/3082996690 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd998037070 con 0x7fd9a8152690 2024-03-06T07:24:45.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.629+0000 7fd9a6ffd640 1 -- 172.21.15.1:0/3082996690 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd998030510 con 0x7fd9a8152690 2024-03-06T07:24:45.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.629+0000 7fd9a6ffd640 1 -- 172.21.15.1:0/3082996690 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c001840 con 0x7fd9a8152690 2024-03-06T07:24:45.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.630+0000 7fd9ae881640 1 -- 172.21.15.1:0/3082996690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a8152690 msgr2=0x7fd9a8155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.630+0000 7fd9ae881640 1 --2- 172.21.15.1:0/3082996690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a8152690 0x7fd9a8155ca0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd9980097a0 tx=0x7fd99802b710 comp rx=0 tx=0).stop 2024-03-06T07:24:45.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.630+0000 7fd9ae881640 1 -- 172.21.15.1:0/3082996690 shutdown_connections 2024-03-06T07:24:45.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.630+0000 7fd9ae881640 1 --2- 172.21.15.1:0/3082996690 >> v2:172.21.15.98:3300/0 conn(0x7fd9a8156cd0 0x7fd9a8157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.630+0000 7fd9ae881640 1 --2- 172.21.15.1:0/3082996690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a8152690 0x7fd9a8155ca0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.630+0000 7fd9ae881640 1 -- 172.21.15.1:0/3082996690 >> 172.21.15.1:0/3082996690 conn(0x7fd9a806b410 msgr2=0x7fd9a806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.630+0000 7fd9ae881640 1 -- 172.21.15.1:0/3082996690 shutdown_connections 2024-03-06T07:24:45.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.630+0000 7fd9ae881640 1 -- 172.21.15.1:0/3082996690 wait complete. 2024-03-06T07:24:45.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.631+0000 7fd9ae881640 1 Processor -- start 2024-03-06T07:24:45.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.631+0000 7fd9ae881640 1 -- start start 2024-03-06T07:24:45.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.632+0000 7fd9ae881640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a80c6730 0x7fd9a81eed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.632+0000 7fd9ae881640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9a8152690 0x7fd9a80c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.632+0000 7fd9ae881640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a8156cd0 0x7fd9a81eac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.632+0000 7fd9ae881640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a80c5ee0 con 0x7fd9a80c6730 2024-03-06T07:24:45.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.632+0000 7fd9ae881640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a81567b0 con 0x7fd9a8152690 2024-03-06T07:24:45.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.632+0000 7fd9ae881640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a80c6060 con 0x7fd9a8156cd0 2024-03-06T07:24:45.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.632+0000 7fd9a7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a80c6730 0x7fd9a81eed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.632+0000 7fd9a7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a80c6730 0x7fd9a81eed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33440/0 (socket says 172.21.15.1:33440) 2024-03-06T07:24:45.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.632+0000 7fd9a7fff640 1 -- 172.21.15.1:0/3764090465 learned_addr learned my addr 172.21.15.1:0/3764090465 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:45.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.632+0000 7fd9a7fff640 1 -- 172.21.15.1:0/3764090465 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a8156cd0 msgr2=0x7fd9a81eac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.632+0000 7fd9a77fe640 1 --2- 172.21.15.1:0/3764090465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9a8152690 0x7fd9a80c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.632+0000 7fd9a7fff640 1 --2- 172.21.15.1:0/3764090465 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a8156cd0 0x7fd9a81eac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.632+0000 7fd9a7fff640 1 -- 172.21.15.1:0/3764090465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9a8152690 msgr2=0x7fd9a80c83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.632+0000 7fd9a7fff640 1 --2- 172.21.15.1:0/3764090465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9a8152690 0x7fd9a80c83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.632+0000 7fd9a7fff640 1 -- 172.21.15.1:0/3764090465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a80ac410 con 0x7fd9a80c6730 2024-03-06T07:24:45.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.632+0000 7fd9a7fff640 1 --2- 172.21.15.1:0/3764090465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a80c6730 0x7fd9a81eed40 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fd99802b750 tx=0x7fd9980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.633+0000 7fd9a4ff9640 1 -- 172.21.15.1:0/3764090465 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd998041020 con 0x7fd9a80c6730 2024-03-06T07:24:45.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.633+0000 7fd9a4ff9640 1 -- 172.21.15.1:0/3764090465 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99803c870 con 0x7fd9a80c6730 2024-03-06T07:24:45.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.633+0000 7fd9a4ff9640 1 -- 172.21.15.1:0/3764090465 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd998037040 con 0x7fd9a80c6730 2024-03-06T07:24:45.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.633+0000 7fd9ae881640 1 -- 172.21.15.1:0/3764090465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a80713c0 con 0x7fd9a80c6730 2024-03-06T07:24:45.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.633+0000 7fd9ae881640 1 -- 172.21.15.1:0/3764090465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a81ee260 con 0x7fd9a80c6730 2024-03-06T07:24:45.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.634+0000 7fd9ae881640 1 -- 172.21.15.1:0/3764090465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a8154420 con 0x7fd9a80c6730 2024-03-06T07:24:45.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.634+0000 7fd9a4ff9640 1 -- 172.21.15.1:0/3764090465 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd998044020 con 0x7fd9a80c6730 2024-03-06T07:24:45.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.635+0000 7fd9a4ff9640 1 --2- 172.21.15.1:0/3764090465 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd99406b4d0 0x7fd99406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.635+0000 7fd9a4ff9640 1 -- 172.21.15.1:0/3764090465 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fd9980aba50 con 0x7fd9a80c6730 2024-03-06T07:24:45.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.635+0000 7fd9a77fe640 1 --2- 172.21.15.1:0/3764090465 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd99406b4d0 0x7fd99406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.635+0000 7fd9a77fe640 1 --2- 172.21.15.1:0/3764090465 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd99406b4d0 0x7fd99406d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd9a806d2a0 tx=0x7fd99c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.639+0000 7fd9a4ff9640 1 -- 172.21.15.1:0/3764090465 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9a8154420 con 0x7fd9a80c6730 2024-03-06T07:24:45.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.818+0000 7fd9ae881640 1 -- 172.21.15.1:0/3764090465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd9a8067fd0 con 0x7fd9a80c6730 2024-03-06T07:24:45.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.818+0000 7fd9a4ff9640 1 -- 172.21.15.1:0/3764090465 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd99804b030 con 0x7fd9a80c6730 2024-03-06T07:24:45.820 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:24:45.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.820+0000 7fd9ae881640 1 -- 172.21.15.1:0/3764090465 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd99406b4d0 msgr2=0x7fd99406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.820+0000 7fd9ae881640 1 --2- 172.21.15.1:0/3764090465 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd99406b4d0 0x7fd99406d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd9a806d2a0 tx=0x7fd99c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:45.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.820+0000 7fd9ae881640 1 -- 172.21.15.1:0/3764090465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a80c6730 msgr2=0x7fd9a81eed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.820+0000 7fd9ae881640 1 --2- 172.21.15.1:0/3764090465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a80c6730 0x7fd9a81eed40 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fd99802b750 tx=0x7fd9980046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.820+0000 7fd9ae881640 1 -- 172.21.15.1:0/3764090465 shutdown_connections 2024-03-06T07:24:45.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.820+0000 7fd9ae881640 1 --2- 172.21.15.1:0/3764090465 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd99406b4d0 0x7fd99406d990 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.820+0000 7fd9ae881640 1 --2- 172.21.15.1:0/3764090465 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9a8156cd0 0x7fd9a81eac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.820+0000 7fd9ae881640 1 --2- 172.21.15.1:0/3764090465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9a8152690 0x7fd9a80c83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.820+0000 7fd9ae881640 1 --2- 172.21.15.1:0/3764090465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9a80c6730 0x7fd9a81eed40 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.820+0000 7fd9ae881640 1 -- 172.21.15.1:0/3764090465 >> 172.21.15.1:0/3764090465 conn(0x7fd9a806b410 msgr2=0x7fd9a80a9e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.820+0000 7fd9ae881640 1 -- 172.21.15.1:0/3764090465 shutdown_connections 2024-03-06T07:24:45.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.820+0000 7fd9ae881640 1 -- 172.21.15.1:0/3764090465 wait complete. 2024-03-06T07:24:45.833 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady' 2024-03-06T07:24:45.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.932+0000 7f9052234640 1 Processor -- start 2024-03-06T07:24:45.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.932+0000 7f9052234640 1 -- start start 2024-03-06T07:24:45.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.933+0000 7f9052234640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f904c152820 0x7f904c0c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.933+0000 7f9052234640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c0c7180 con 0x7f904c0af6f0 2024-03-06T07:24:45.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.933+0000 7f9052234640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c155460 con 0x7f904c0c0670 2024-03-06T07:24:45.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.933+0000 7f9052234640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f904c0c6bc0 con 0x7f904c152820 2024-03-06T07:24:45.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.933+0000 7f904bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f904c0af6f0 0x7f904c15fb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54442/0 (socket says 172.21.15.1:54442) 2024-03-06T07:24:45.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.933+0000 7f904bfff640 1 -- 172.21.15.1:0/917637336 learned_addr learned my addr 172.21.15.1:0/917637336 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:45.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.933+0000 7f904a7fc640 1 -- 172.21.15.1:0/917637336 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236491568 0 0) 0x7f904c0c7180 con 0x7f904c0af6f0 2024-03-06T07:24:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.933+0000 7f904b7fe640 1 --2- 172.21.15.1:0/917637336 >> v2:172.21.15.98:3300/0 conn(0x7f904c152820 0x7f904c0c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.933+0000 7f904a7fc640 1 -- 172.21.15.1:0/917637336 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9034003520 con 0x7f904c0af6f0 2024-03-06T07:24:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904a7fc640 1 -- 172.21.15.1:0/917637336 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2547069705 0 0) 0x7f9034003520 con 0x7f904c0af6f0 2024-03-06T07:24:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904a7fc640 1 -- 172.21.15.1:0/917637336 >> v2:172.21.15.98:3300/0 conn(0x7f904c152820 msgr2=0x7f904c0c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904a7fc640 1 --2- 172.21.15.1:0/917637336 >> v2:172.21.15.98:3300/0 conn(0x7f904c152820 0x7f904c0c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904a7fc640 1 -- 172.21.15.1:0/917637336 >> v1:172.21.15.98:6789/0 conn(0x7f904c0c0670 legacy=0x7f904c0c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904a7fc640 1 -- 172.21.15.1:0/917637336 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c0c6ea0 con 0x7f904c0af6f0 2024-03-06T07:24:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904a7fc640 1 -- 172.21.15.1:0/917637336 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f90440019f0 con 0x7f904c0af6f0 2024-03-06T07:24:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904a7fc640 1 -- 172.21.15.1:0/917637336 >> v1:172.21.15.1:6789/0 conn(0x7f904c0af6f0 legacy=0x7f904c15fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904a7fc640 1 --2- 172.21.15.1:0/917637336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90340039b0 0x7f9034003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904a7fc640 1 --2- 172.21.15.1:0/917637336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9034004800 0x7f9034004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904a7fc640 1 --2- 172.21.15.1:0/917637336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90340051c0 0x7f903400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904a7fc640 1 -- 172.21.15.1:0/917637336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c0c6bc0 con 0x7f90340051c0 2024-03-06T07:24:45.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904a7fc640 1 -- 172.21.15.1:0/917637336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903400c060 con 0x7f9034004800 2024-03-06T07:24:45.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904a7fc640 1 -- 172.21.15.1:0/917637336 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903400c260 con 0x7f90340039b0 2024-03-06T07:24:45.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904bfff640 1 --2- 172.21.15.1:0/917637336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90340051c0 0x7f903400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904b7fe640 1 --2- 172.21.15.1:0/917637336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90340039b0 0x7f9034003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904affd640 1 --2- 172.21.15.1:0/917637336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9034004800 0x7f9034004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904bfff640 1 -- 172.21.15.1:0/917637336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90340039b0 msgr2=0x7f9034003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:45.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904bfff640 1 --2- 172.21.15.1:0/917637336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90340039b0 0x7f9034003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904bfff640 1 -- 172.21.15.1:0/917637336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9034004800 msgr2=0x7f9034004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:45.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904bfff640 1 --2- 172.21.15.1:0/917637336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9034004800 0x7f9034004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904bfff640 1 -- 172.21.15.1:0/917637336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9034003520 con 0x7f90340051c0 2024-03-06T07:24:45.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.934+0000 7f904affd640 1 --2- 172.21.15.1:0/917637336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9034004800 0x7f9034004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:45.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.935+0000 7f904bfff640 1 -- 172.21.15.1:0/917637336 reap_dead start 2024-03-06T07:24:45.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.935+0000 7f904bfff640 1 --2- 172.21.15.1:0/917637336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90340051c0 0x7f903400b9c0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f904400a840 tx=0x7f9034005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd1e22b9b6b05a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.935+0000 7f904a7fc640 1 -- 172.21.15.1:0/917637336 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904400a460 con 0x7f90340051c0 2024-03-06T07:24:45.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.935+0000 7f904a7fc640 1 -- 172.21.15.1:0/917637336 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9044034650 con 0x7f90340051c0 2024-03-06T07:24:45.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.935+0000 7f904a7fc640 1 -- 172.21.15.1:0/917637336 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9044034a40 con 0x7f90340051c0 2024-03-06T07:24:45.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.936+0000 7f9052234640 1 -- 172.21.15.1:0/917637336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90340051c0 msgr2=0x7f903400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.936+0000 7f9052234640 1 --2- 172.21.15.1:0/917637336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90340051c0 0x7f903400b9c0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f904400a840 tx=0x7f9034005830 comp rx=0 tx=0).stop 2024-03-06T07:24:45.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.936+0000 7f9052234640 1 -- 172.21.15.1:0/917637336 shutdown_connections 2024-03-06T07:24:45.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.936+0000 7f9052234640 1 --2- 172.21.15.1:0/917637336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90340051c0 0x7f903400b9c0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.936+0000 7f9052234640 1 -- 172.21.15.1:0/917637336 >> 172.21.15.1:0/917637336 conn(0x7f904c072720 msgr2=0x7f904c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.936+0000 7f9052234640 1 -- 172.21.15.1:0/917637336 shutdown_connections 2024-03-06T07:24:45.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.936+0000 7f9052234640 1 -- 172.21.15.1:0/917637336 wait complete. 2024-03-06T07:24:45.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.937+0000 7f9052234640 1 Processor -- start 2024-03-06T07:24:45.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.937+0000 7f9052234640 1 -- start start 2024-03-06T07:24:45.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f9052234640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90340051c0 0x7f904c0aa820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f9052234640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f904c14d650 0x7f904c0c3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f9052234640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f904c14caa0 0x7f904c14cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f9052234640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c152d80 con 0x7f904c14caa0 2024-03-06T07:24:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f9052234640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c152c00 con 0x7f904c14d650 2024-03-06T07:24:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f9052234640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f904c0afc40 con 0x7f90340051c0 2024-03-06T07:24:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f904bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f904c14caa0 0x7f904c14cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f904bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f904c14caa0 0x7f904c14cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33454/0 (socket says 172.21.15.1:33454) 2024-03-06T07:24:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f904bfff640 1 -- 172.21.15.1:0/819153406 learned_addr learned my addr 172.21.15.1:0/819153406 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f904b7fe640 1 --2- 172.21.15.1:0/819153406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90340051c0 0x7f904c0aa820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f904affd640 1 --2- 172.21.15.1:0/819153406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f904c14d650 0x7f904c0c3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f904bfff640 1 -- 172.21.15.1:0/819153406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90340051c0 msgr2=0x7f904c0aa820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f904bfff640 1 --2- 172.21.15.1:0/819153406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90340051c0 0x7f904c0aa820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f904bfff640 1 -- 172.21.15.1:0/819153406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f904c14d650 msgr2=0x7f904c0c3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f904bfff640 1 --2- 172.21.15.1:0/819153406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f904c14d650 0x7f904c0c3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f904bfff640 1 -- 172.21.15.1:0/819153406 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c14d460 con 0x7f904c14caa0 2024-03-06T07:24:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f904b7fe640 1 --2- 172.21.15.1:0/819153406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90340051c0 0x7f904c0aa820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f904bfff640 1 --2- 172.21.15.1:0/819153406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f904c14caa0 0x7f904c14cf20 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f904400a930 tx=0x7f904c1559e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f9033fff640 1 -- 172.21.15.1:0/819153406 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9044039040 con 0x7f904c14caa0 2024-03-06T07:24:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f9033fff640 1 -- 172.21.15.1:0/819153406 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9044043070 con 0x7f904c14caa0 2024-03-06T07:24:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f9052234640 1 -- 172.21.15.1:0/819153406 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f904c1eee50 con 0x7f904c14caa0 2024-03-06T07:24:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f9033fff640 1 -- 172.21.15.1:0/819153406 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9044004b60 con 0x7f904c14caa0 2024-03-06T07:24:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.938+0000 7f9052234640 1 -- 172.21.15.1:0/819153406 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f904c1592c0 con 0x7f904c14caa0 2024-03-06T07:24:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.939+0000 7f9052234640 1 -- 172.21.15.1:0/819153406 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f904c1ef370 con 0x7f904c14caa0 2024-03-06T07:24:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.940+0000 7f9033fff640 1 -- 172.21.15.1:0/819153406 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9044004d00 con 0x7f904c14caa0 2024-03-06T07:24:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.940+0000 7f9033fff640 1 --2- 172.21.15.1:0/819153406 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f903806b500 0x7f903806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.940+0000 7f9033fff640 1 -- 172.21.15.1:0/819153406 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f90440abfa0 con 0x7f904c14caa0 2024-03-06T07:24:45.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.940+0000 7f904b7fe640 1 --2- 172.21.15.1:0/819153406 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f903806b500 0x7f903806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.941+0000 7f904b7fe640 1 --2- 172.21.15.1:0/819153406 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f903806b500 0x7f903806d9c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f903c009b40 tx=0x7f903c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:45.945+0000 7f9033fff640 1 -- 172.21.15.1:0/819153406 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f904c1ef370 con 0x7f904c14caa0 2024-03-06T07:24:46.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:46.195+0000 7f9052234640 1 -- 172.21.15.1:0/819153406 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]} v 0) v1 -- 0x7f904c0686e0 con 0x7f904c14caa0 2024-03-06T07:24:47.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.198+0000 7f9033fff640 1 -- 172.21.15.1:0/819153406 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]}]=0 v266) v1 ==== 148+0+0 (secure 0 0 0) 0x7f904407af40 con 0x7f904c14caa0 2024-03-06T07:24:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.201+0000 7f9052234640 1 -- 172.21.15.1:0/819153406 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f903806b500 msgr2=0x7f903806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.201+0000 7f9052234640 1 --2- 172.21.15.1:0/819153406 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f903806b500 0x7f903806d9c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f903c009b40 tx=0x7f903c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.201+0000 7f9052234640 1 -- 172.21.15.1:0/819153406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f904c14caa0 msgr2=0x7f904c14cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.201+0000 7f9052234640 1 --2- 172.21.15.1:0/819153406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f904c14caa0 0x7f904c14cf20 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f904400a930 tx=0x7f904c1559e0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.201+0000 7f9052234640 1 -- 172.21.15.1:0/819153406 shutdown_connections 2024-03-06T07:24:47.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.201+0000 7f9052234640 1 --2- 172.21.15.1:0/819153406 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f903806b500 0x7f903806d9c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.201+0000 7f9052234640 1 --2- 172.21.15.1:0/819153406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f904c14caa0 0x7f904c14cf20 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.201+0000 7f9052234640 1 --2- 172.21.15.1:0/819153406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f904c14d650 0x7f904c0c3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.201+0000 7f9052234640 1 --2- 172.21.15.1:0/819153406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90340051c0 0x7f904c0aa820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.201+0000 7f9052234640 1 -- 172.21.15.1:0/819153406 >> 172.21.15.1:0/819153406 conn(0x7f904c072720 msgr2=0x7f904c06d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.201+0000 7f9052234640 1 -- 172.21.15.1:0/819153406 shutdown_connections 2024-03-06T07:24:47.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.201+0000 7f9052234640 1 -- 172.21.15.1:0/819153406 wait complete. 2024-03-06T07:24:47.216 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:24:47.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.324+0000 7f34d7478640 1 Processor -- start 2024-03-06T07:24:47.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.325+0000 7f34d7478640 1 -- start start 2024-03-06T07:24:47.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.325+0000 7f34d7478640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f34d00c0d00 0x7f34d0151ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.325+0000 7f34d7478640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34d006c650 0x7f34d0152530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.325+0000 7f34d7478640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d00a8c40 con 0x7f34d00c3050 2024-03-06T07:24:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.325+0000 7f34d7478640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d00a8f20 con 0x7f34d00c0d00 2024-03-06T07:24:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.325+0000 7f34d7478640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d00a9200 con 0x7f34d006c650 2024-03-06T07:24:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34d5214640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f34d00c3050 0x7f34d00c05f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54454/0 (socket says 172.21.15.1:54454) 2024-03-06T07:24:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34d5214640 1 -- 172.21.15.1:0/197982127 learned_addr learned my addr 172.21.15.1:0/197982127 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34d5a15640 1 --2- 172.21.15.1:0/197982127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34d006c650 0x7f34d0152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34d4a13640 1 --2- 172.21.15.1:0/197982127 >> v2:172.21.15.98:3300/0 conn(0x7f34d00c0d00 0x7f34d0151ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34c7fff640 1 -- 172.21.15.1:0/197982127 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327298946 0 0) 0x7f34d00a8c40 con 0x7f34d00c3050 2024-03-06T07:24:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34c7fff640 1 -- 172.21.15.1:0/197982127 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34bc003520 con 0x7f34d00c3050 2024-03-06T07:24:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34c7fff640 1 -- 172.21.15.1:0/197982127 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4005052602 0 0) 0x7f34bc003520 con 0x7f34d00c3050 2024-03-06T07:24:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34c7fff640 1 -- 172.21.15.1:0/197982127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34d006c650 msgr2=0x7f34d0152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34c7fff640 1 --2- 172.21.15.1:0/197982127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34d006c650 0x7f34d0152530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34c7fff640 1 -- 172.21.15.1:0/197982127 >> v2:172.21.15.98:3300/0 conn(0x7f34d00c0d00 msgr2=0x7f34d0151ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34c7fff640 1 --2- 172.21.15.1:0/197982127 >> v2:172.21.15.98:3300/0 conn(0x7f34d00c0d00 0x7f34d0151ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34c7fff640 1 -- 172.21.15.1:0/197982127 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d00a94e0 con 0x7f34d00c3050 2024-03-06T07:24:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34c7fff640 1 -- 172.21.15.1:0/197982127 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f34cc001670 con 0x7f34d00c3050 2024-03-06T07:24:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34c7fff640 1 -- 172.21.15.1:0/197982127 >> v1:172.21.15.1:6789/0 conn(0x7f34d00c3050 legacy=0x7f34d00c05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34c7fff640 1 --2- 172.21.15.1:0/197982127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34bc0039b0 0x7f34bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34c7fff640 1 --2- 172.21.15.1:0/197982127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34bc004800 0x7f34bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34c7fff640 1 --2- 172.21.15.1:0/197982127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34bc0051c0 0x7f34bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34c7fff640 1 -- 172.21.15.1:0/197982127 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d00a8f20 con 0x7f34bc004800 2024-03-06T07:24:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34c7fff640 1 -- 172.21.15.1:0/197982127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d00a9200 con 0x7f34bc0039b0 2024-03-06T07:24:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34c7fff640 1 -- 172.21.15.1:0/197982127 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc00bf30 con 0x7f34bc0051c0 2024-03-06T07:24:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34d5a15640 1 --2- 172.21.15.1:0/197982127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34d006c650 0x7f34d0152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.326+0000 7f34d4a13640 1 --2- 172.21.15.1:0/197982127 >> v2:172.21.15.98:3300/0 conn(0x7f34d00c0d00 0x7f34d0151ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.327+0000 7f34d4a13640 1 --2- 172.21.15.1:0/197982127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34bc004800 0x7f34bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.327+0000 7f34d5a15640 1 --2- 172.21.15.1:0/197982127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34bc0051c0 0x7f34bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.327+0000 7f34d5214640 1 --2- 172.21.15.1:0/197982127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34bc0039b0 0x7f34bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.327+0000 7f34d4a13640 1 -- 172.21.15.1:0/197982127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34bc0051c0 msgr2=0x7f34bc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.327+0000 7f34d4a13640 1 --2- 172.21.15.1:0/197982127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34bc0051c0 0x7f34bc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.327+0000 7f34d4a13640 1 -- 172.21.15.1:0/197982127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34bc0039b0 msgr2=0x7f34bc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.327+0000 7f34d4a13640 1 --2- 172.21.15.1:0/197982127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34bc0039b0 0x7f34bc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.327+0000 7f34d4a13640 1 -- 172.21.15.1:0/197982127 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34bc003520 con 0x7f34bc004800 2024-03-06T07:24:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.328+0000 7f34d4a13640 1 --2- 172.21.15.1:0/197982127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34bc004800 0x7f34bc004c80 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009c70 tx=0x7f34bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6317d5e9d78f49a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.328+0000 7f34c7fff640 1 -- 172.21.15.1:0/197982127 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c0009840 con 0x7f34bc004800 2024-03-06T07:24:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.328+0000 7f34c7fff640 1 -- 172.21.15.1:0/197982127 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c0004020 con 0x7f34bc004800 2024-03-06T07:24:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.328+0000 7f34c7fff640 1 -- 172.21.15.1:0/197982127 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c0004410 con 0x7f34bc004800 2024-03-06T07:24:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.328+0000 7f34d7478640 1 -- 172.21.15.1:0/197982127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34bc004800 msgr2=0x7f34bc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.328+0000 7f34d7478640 1 --2- 172.21.15.1:0/197982127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34bc004800 0x7f34bc004c80 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009c70 tx=0x7f34bc005830 comp rx=0 tx=0).stop 2024-03-06T07:24:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.328+0000 7f34d5a15640 1 -- 172.21.15.1:0/197982127 reap_dead start 2024-03-06T07:24:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.328+0000 7f34d7478640 1 -- 172.21.15.1:0/197982127 shutdown_connections 2024-03-06T07:24:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.328+0000 7f34d7478640 1 -- 172.21.15.1:0/197982127 >> 172.21.15.1:0/197982127 conn(0x7f34d00c3cc0 msgr2=0x7f34d00c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.328+0000 7f34d7478640 1 -- 172.21.15.1:0/197982127 shutdown_connections 2024-03-06T07:24:47.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.328+0000 7f34d7478640 1 -- 172.21.15.1:0/197982127 wait complete. 2024-03-06T07:24:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.329+0000 7f34d7478640 1 Processor -- start 2024-03-06T07:24:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.329+0000 7f34d7478640 1 -- start start 2024-03-06T07:24:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.330+0000 7f34d7478640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34bc004800 0x7f34d0149050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.330+0000 7f34d7478640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34d0149710 0x7f34d0150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.330+0000 7f34d7478640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34d006f000 0x7f34d01485c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.330+0000 7f34d7478640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d00a9ce0 con 0x7f34bc004800 2024-03-06T07:24:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.330+0000 7f34d7478640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d00c3980 con 0x7f34d0149710 2024-03-06T07:24:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.330+0000 7f34d7478640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d01506d0 con 0x7f34d006f000 2024-03-06T07:24:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.330+0000 7f34d5214640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34bc004800 0x7f34d0149050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.330+0000 7f34d5214640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34bc004800 0x7f34d0149050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33472/0 (socket says 172.21.15.1:33472) 2024-03-06T07:24:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.330+0000 7f34d5214640 1 -- 172.21.15.1:0/2897991278 learned_addr learned my addr 172.21.15.1:0/2897991278 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.330+0000 7f34d5214640 1 -- 172.21.15.1:0/2897991278 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34d006f000 msgr2=0x7f34d01485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.330+0000 7f34d5214640 1 --2- 172.21.15.1:0/2897991278 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34d006f000 0x7f34d01485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.330+0000 7f34d5214640 1 -- 172.21.15.1:0/2897991278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34d0149710 msgr2=0x7f34d0150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.330+0000 7f34d5214640 1 --2- 172.21.15.1:0/2897991278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34d0149710 0x7f34d0150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.331+0000 7f34d5214640 1 -- 172.21.15.1:0/2897991278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d00c4fd0 con 0x7f34bc004800 2024-03-06T07:24:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.331+0000 7f34d5214640 1 --2- 172.21.15.1:0/2897991278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34bc004800 0x7f34d0149050 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f34cc007fd0 tx=0x7f34cc007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.331+0000 7f34c5ffb640 1 -- 172.21.15.1:0/2897991278 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc00f730 con 0x7f34bc004800 2024-03-06T07:24:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.331+0000 7f34c5ffb640 1 -- 172.21.15.1:0/2897991278 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34cc018070 con 0x7f34bc004800 2024-03-06T07:24:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.331+0000 7f34d7478640 1 -- 172.21.15.1:0/2897991278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d01516a0 con 0x7f34bc004800 2024-03-06T07:24:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.331+0000 7f34c5ffb640 1 -- 172.21.15.1:0/2897991278 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc00fb50 con 0x7f34bc004800 2024-03-06T07:24:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.331+0000 7f34d7478640 1 -- 172.21.15.1:0/2897991278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d0147dc0 con 0x7f34bc004800 2024-03-06T07:24:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.332+0000 7f34d7478640 1 -- 172.21.15.1:0/2897991278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d0067fd0 con 0x7f34bc004800 2024-03-06T07:24:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.332+0000 7f34c5ffb640 1 -- 172.21.15.1:0/2897991278 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f34cc00fcf0 con 0x7f34bc004800 2024-03-06T07:24:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.333+0000 7f34c5ffb640 1 --2- 172.21.15.1:0/2897991278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34b406b4d0 0x7f34b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.333+0000 7f34c5ffb640 1 -- 172.21.15.1:0/2897991278 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f34cc08a030 con 0x7f34bc004800 2024-03-06T07:24:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.333+0000 7f34d4a13640 1 --2- 172.21.15.1:0/2897991278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34b406b4d0 0x7f34b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.334+0000 7f34d4a13640 1 --2- 172.21.15.1:0/2897991278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34b406b4d0 0x7f34b406d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f34c0035000 tx=0x7f34c002e000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.337+0000 7f34c5ffb640 1 -- 172.21.15.1:0/2897991278 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f34cc059060 con 0x7f34bc004800 2024-03-06T07:24:47.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.520+0000 7f34d7478640 1 -- 172.21.15.1:0/2897991278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f34d00a9e60 con 0x7f34bc004800 2024-03-06T07:24:47.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.523+0000 7f34c5ffb640 1 -- 172.21.15.1:0/2897991278 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v41)=0 v41) v1 ==== 122+0+0 (secure 0 0 0) 0x7f34cc058960 con 0x7f34bc004800 2024-03-06T07:24:47.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.526+0000 7f34d7478640 1 -- 172.21.15.1:0/2897991278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34b406b4d0 msgr2=0x7f34b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.526+0000 7f34d7478640 1 --2- 172.21.15.1:0/2897991278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34b406b4d0 0x7f34b406d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f34c0035000 tx=0x7f34c002e000 comp rx=0 tx=0).stop 2024-03-06T07:24:47.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.526+0000 7f34d7478640 1 -- 172.21.15.1:0/2897991278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34bc004800 msgr2=0x7f34d0149050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.526+0000 7f34d7478640 1 --2- 172.21.15.1:0/2897991278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34bc004800 0x7f34d0149050 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f34cc007fd0 tx=0x7f34cc007430 comp rx=0 tx=0).stop 2024-03-06T07:24:47.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.526+0000 7f34d7478640 1 -- 172.21.15.1:0/2897991278 shutdown_connections 2024-03-06T07:24:47.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.526+0000 7f34d7478640 1 --2- 172.21.15.1:0/2897991278 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34b406b4d0 0x7f34b406d990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.526+0000 7f34d7478640 1 --2- 172.21.15.1:0/2897991278 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34d006f000 0x7f34d01485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.526+0000 7f34d7478640 1 --2- 172.21.15.1:0/2897991278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34d0149710 0x7f34d0150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.526+0000 7f34d7478640 1 --2- 172.21.15.1:0/2897991278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34bc004800 0x7f34d0149050 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.526+0000 7f34d7478640 1 -- 172.21.15.1:0/2897991278 >> 172.21.15.1:0/2897991278 conn(0x7f34d00c3cc0 msgr2=0x7f34d00c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.526+0000 7f34d7478640 1 -- 172.21.15.1:0/2897991278 shutdown_connections 2024-03-06T07:24:47.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.526+0000 7f34d7478640 1 -- 172.21.15.1:0/2897991278 wait complete. 2024-03-06T07:24:47.538 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:47.538 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:47.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:47.627 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:47.627 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:24:47.627 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:47.642 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:24:47.642 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:24:47.643 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:47.659 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:24:47.660 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:24:47.660 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.781+0000 7f544569a640 1 Processor -- start 2024-03-06T07:24:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.781+0000 7f544569a640 1 -- start start 2024-03-06T07:24:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.782+0000 7f544569a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f544006fee0 0x7f54400c3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.782+0000 7f544569a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54400a5f30 0x7f54400718b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.782+0000 7f544569a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54400702c0 con 0x7f5440073080 2024-03-06T07:24:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.782+0000 7f544569a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54400a66b0 con 0x7f54400a5f30 2024-03-06T07:24:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.782+0000 7f544569a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54400c16d0 con 0x7f544006fee0 2024-03-06T07:24:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.782+0000 7f543e7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54400a5f30 0x7f54400718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.782+0000 7f543f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5440073080 0x7f5440071df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54466/0 (socket says 172.21.15.1:54466) 2024-03-06T07:24:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.782+0000 7f543f7fe640 1 -- 172.21.15.1:0/4256107819 learned_addr learned my addr 172.21.15.1:0/4256107819 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.782+0000 7f543effd640 1 --2- 172.21.15.1:0/4256107819 >> v2:172.21.15.98:3300/0 conn(0x7f544006fee0 0x7f54400c3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.782+0000 7f543e7fc640 1 -- 172.21.15.1:0/4256107819 >> v2:172.21.15.98:3300/0 conn(0x7f544006fee0 msgr2=0x7f54400c3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.782+0000 7f543e7fc640 1 --2- 172.21.15.1:0/4256107819 >> v2:172.21.15.98:3300/0 conn(0x7f544006fee0 0x7f54400c3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.782+0000 7f543e7fc640 1 -- 172.21.15.1:0/4256107819 >> v1:172.21.15.1:6789/0 conn(0x7f5440073080 legacy=0x7f5440071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.782+0000 7f543dffb640 1 -- 172.21.15.1:0/4256107819 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227703565 0 0) 0x7f54400702c0 con 0x7f5440073080 2024-03-06T07:24:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.782+0000 7f543e7fc640 1 -- 172.21.15.1:0/4256107819 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54400c1990 con 0x7f54400a5f30 2024-03-06T07:24:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.783+0000 7f543e7fc640 1 --2- 172.21.15.1:0/4256107819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54400a5f30 0x7f54400718b0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f544006bed0 tx=0x7f542802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf035e5ff04a4cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.783+0000 7f543dffb640 1 -- 172.21.15.1:0/4256107819 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5428035020 con 0x7f54400a5f30 2024-03-06T07:24:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.783+0000 7f543dffb640 1 -- 172.21.15.1:0/4256107819 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5428031830 con 0x7f54400a5f30 2024-03-06T07:24:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.783+0000 7f543dffb640 1 -- 172.21.15.1:0/4256107819 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5428031c00 con 0x7f54400a5f30 2024-03-06T07:24:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.784+0000 7f544569a640 1 -- 172.21.15.1:0/4256107819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54400a5f30 msgr2=0x7f54400718b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.784+0000 7f544569a640 1 --2- 172.21.15.1:0/4256107819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54400a5f30 0x7f54400718b0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f544006bed0 tx=0x7f542802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.784+0000 7f544569a640 1 -- 172.21.15.1:0/4256107819 shutdown_connections 2024-03-06T07:24:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.784+0000 7f544569a640 1 --2- 172.21.15.1:0/4256107819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54400a5f30 0x7f54400718b0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.784+0000 7f544569a640 1 --2- 172.21.15.1:0/4256107819 >> v2:172.21.15.98:3300/0 conn(0x7f544006fee0 0x7f54400c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.784+0000 7f544569a640 1 -- 172.21.15.1:0/4256107819 >> 172.21.15.1:0/4256107819 conn(0x7f54400c6f40 msgr2=0x7f54400c7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.784+0000 7f544569a640 1 -- 172.21.15.1:0/4256107819 shutdown_connections 2024-03-06T07:24:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.784+0000 7f544569a640 1 -- 172.21.15.1:0/4256107819 wait complete. 2024-03-06T07:24:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.785+0000 7f544569a640 1 Processor -- start 2024-03-06T07:24:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.785+0000 7f544569a640 1 -- start start 2024-03-06T07:24:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f544569a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f544006fee0 0x7f5440155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f544569a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5440073080 0x7f54401eeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f544569a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54400a5f30 0x7f544014b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f544569a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5440153940 con 0x7f544006fee0 2024-03-06T07:24:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f544569a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54400c3250 con 0x7f5440073080 2024-03-06T07:24:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f544569a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54400c3dd0 con 0x7f54400a5f30 2024-03-06T07:24:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f543effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f544006fee0 0x7f5440155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f543effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f544006fee0 0x7f5440155000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33476/0 (socket says 172.21.15.1:33476) 2024-03-06T07:24:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f543effd640 1 -- 172.21.15.1:0/714439792 learned_addr learned my addr 172.21.15.1:0/714439792 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f543e7fc640 1 --2- 172.21.15.1:0/714439792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5440073080 0x7f54401eeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f543f7fe640 1 --2- 172.21.15.1:0/714439792 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54400a5f30 0x7f544014b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f543effd640 1 -- 172.21.15.1:0/714439792 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54400a5f30 msgr2=0x7f544014b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f543effd640 1 --2- 172.21.15.1:0/714439792 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54400a5f30 0x7f544014b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f543effd640 1 -- 172.21.15.1:0/714439792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5440073080 msgr2=0x7f54401eeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f543effd640 1 --2- 172.21.15.1:0/714439792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5440073080 0x7f54401eeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f543effd640 1 -- 172.21.15.1:0/714439792 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5440156bb0 con 0x7f544006fee0 2024-03-06T07:24:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f543effd640 1 --2- 172.21.15.1:0/714439792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f544006fee0 0x7f5440155000 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f543800d800 tx=0x7f543800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f54377fe640 1 -- 172.21.15.1:0/714439792 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543800b6c0 con 0x7f544006fee0 2024-03-06T07:24:47.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.786+0000 7f54377fe640 1 -- 172.21.15.1:0/714439792 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5438019070 con 0x7f544006fee0 2024-03-06T07:24:47.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.787+0000 7f544569a640 1 -- 172.21.15.1:0/714439792 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5440156e90 con 0x7f544006fee0 2024-03-06T07:24:47.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.787+0000 7f54377fe640 1 -- 172.21.15.1:0/714439792 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543800ba80 con 0x7f544006fee0 2024-03-06T07:24:47.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.787+0000 7f544569a640 1 -- 172.21.15.1:0/714439792 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54401ee320 con 0x7f544006fee0 2024-03-06T07:24:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.788+0000 7f544569a640 1 -- 172.21.15.1:0/714439792 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5440067fd0 con 0x7f544006fee0 2024-03-06T07:24:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.788+0000 7f54377fe640 1 -- 172.21.15.1:0/714439792 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f543800bc20 con 0x7f544006fee0 2024-03-06T07:24:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.788+0000 7f54377fe640 1 --2- 172.21.15.1:0/714439792 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f542006b4d0 0x7f542006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.789+0000 7f54377fe640 1 -- 172.21.15.1:0/714439792 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f543808a500 con 0x7f544006fee0 2024-03-06T07:24:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.789+0000 7f543e7fc640 1 --2- 172.21.15.1:0/714439792 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f542006b4d0 0x7f542006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.789+0000 7f543e7fc640 1 --2- 172.21.15.1:0/714439792 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f542006b4d0 0x7f542006d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5428036000 tx=0x7f5428008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:47.793+0000 7f54377fe640 1 -- 172.21.15.1:0/714439792 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5438059530 con 0x7f544006fee0 2024-03-06T07:24:48.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.011+0000 7f544569a640 1 -- 172.21.15.1:0/714439792 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5440072e60 con 0x7f544006fee0 2024-03-06T07:24:48.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.011+0000 7f54377fe640 1 -- 172.21.15.1:0/714439792 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3696 (secure 0 0 0) 0x7f5438058e30 con 0x7f544006fee0 2024-03-06T07:24:48.014 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:48.014 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":100,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1741783790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1741783790},{"type":"v1","addr":"172.21.15.98:6836","nonce":1741783790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7009,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1351517387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1351517387},{"type":"v1","addr":"172.21.15.1:6835","nonce":1351517387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":7012,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1857080322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1857080322},{"type":"v1","addr":"172.21.15.98:6837","nonce":1857080322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":7662,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3096543492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3096543492},{"type":"v1","addr":"172.21.15.1:6833","nonce":3096543492}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[]} 2024-03-06T07:24:48.014 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 100 2024-03-06T07:24:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.013+0000 7f544569a640 1 -- 172.21.15.1:0/714439792 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f542006b4d0 msgr2=0x7f542006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.013+0000 7f544569a640 1 --2- 172.21.15.1:0/714439792 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f542006b4d0 0x7f542006d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5428036000 tx=0x7f5428008030 comp rx=0 tx=0).stop 2024-03-06T07:24:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.013+0000 7f544569a640 1 -- 172.21.15.1:0/714439792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f544006fee0 msgr2=0x7f5440155000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.013+0000 7f544569a640 1 --2- 172.21.15.1:0/714439792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f544006fee0 0x7f5440155000 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f543800d800 tx=0x7f543800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.013+0000 7f544569a640 1 -- 172.21.15.1:0/714439792 shutdown_connections 2024-03-06T07:24:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.013+0000 7f544569a640 1 --2- 172.21.15.1:0/714439792 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f542006b4d0 0x7f542006d990 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.013+0000 7f544569a640 1 --2- 172.21.15.1:0/714439792 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54400a5f30 0x7f544014b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.013+0000 7f544569a640 1 --2- 172.21.15.1:0/714439792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5440073080 0x7f54401eeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.013+0000 7f544569a640 1 --2- 172.21.15.1:0/714439792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f544006fee0 0x7f5440155000 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.013+0000 7f544569a640 1 -- 172.21.15.1:0/714439792 >> 172.21.15.1:0/714439792 conn(0x7f54400c6f40 msgr2=0x7f544006f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.013+0000 7f544569a640 1 -- 172.21.15.1:0/714439792 shutdown_connections 2024-03-06T07:24:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.013+0000 7f544569a640 1 -- 172.21.15.1:0/714439792 wait complete. 2024-03-06T07:24:48.026 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:24:48.026 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:24:48.027 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:24:48.027 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:48.027 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:24:48.027 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:24:48.027 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:24:48.027 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:48.028 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:24:48.028 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:24:48.028 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:24:48.028 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:48.028 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:24:48.028 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:24:48.028 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:24:48.029 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:48.036 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:24:48.036 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:24:48.037 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:24:48.037 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:24:48.039 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:24:48.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:24:48.040 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:24:48.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:24:48.079 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:24:48.079 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:48.080 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:24:48.080 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:48.082 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:24:48.082 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:48.083 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:24:48.083 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:48.083 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:24:48.155 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:24:48.156 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:24:48.164 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:24:48.164 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:24:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.187+0000 7f71e9d3e640 1 Processor -- start 2024-03-06T07:24:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.187+0000 7f71e9d3e640 1 -- start start 2024-03-06T07:24:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.188+0000 7f71e9d3e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f71e4072bd0 0x7f71e40c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.188+0000 7f71e9d3e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e4151dd0 con 0x7f71e4152b70 2024-03-06T07:24:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.188+0000 7f71e9d3e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e4156190 con 0x7f71e40c2b30 2024-03-06T07:24:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.188+0000 7f71e9d3e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e40c3f40 con 0x7f71e4072bd0 2024-03-06T07:24:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.188+0000 7f71e2ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f71e4072bd0 0x7f71e40c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.188+0000 7f71e3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f71e4152b70 0x7f71e40c3830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54476/0 (socket says 172.21.15.1:54476) 2024-03-06T07:24:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.188+0000 7f71e3fff640 1 -- 172.21.15.1:0/281199828 learned_addr learned my addr 172.21.15.1:0/281199828 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.188+0000 7f71e2ffd640 1 -- 172.21.15.1:0/281199828 >> v1:172.21.15.98:6789/0 conn(0x7f71e40c2b30 legacy=0x7f71e40724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.188+0000 7f71e2ffd640 1 -- 172.21.15.1:0/281199828 >> v1:172.21.15.1:6789/0 conn(0x7f71e4152b70 legacy=0x7f71e40c3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.188+0000 7f71e27fc640 1 -- 172.21.15.1:0/281199828 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966834153 0 0) 0x7f71e4151dd0 con 0x7f71e4152b70 2024-03-06T07:24:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.188+0000 7f71e2ffd640 1 -- 172.21.15.1:0/281199828 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e4152090 con 0x7f71e4072bd0 2024-03-06T07:24:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.189+0000 7f71e2ffd640 1 --2- 172.21.15.1:0/281199828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e4072bd0 0x7f71e40c32f0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f71d80097a0 tx=0x7f71d802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31dc6afcb476d8ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.189+0000 7f71e27fc640 1 -- 172.21.15.1:0/281199828 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d8034070 con 0x7f71e4072bd0 2024-03-06T07:24:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.189+0000 7f71e27fc640 1 -- 172.21.15.1:0/281199828 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d8030610 con 0x7f71e4072bd0 2024-03-06T07:24:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.189+0000 7f71e27fc640 1 -- 172.21.15.1:0/281199828 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d80309e0 con 0x7f71e4072bd0 2024-03-06T07:24:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.190+0000 7f71e9d3e640 1 -- 172.21.15.1:0/281199828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e4072bd0 msgr2=0x7f71e40c32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.190+0000 7f71e9d3e640 1 --2- 172.21.15.1:0/281199828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e4072bd0 0x7f71e40c32f0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f71d80097a0 tx=0x7f71d802b960 comp rx=0 tx=0).stop 2024-03-06T07:24:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.190+0000 7f71e9d3e640 1 -- 172.21.15.1:0/281199828 shutdown_connections 2024-03-06T07:24:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.190+0000 7f71e9d3e640 1 --2- 172.21.15.1:0/281199828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e4072bd0 0x7f71e40c32f0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.190+0000 7f71e9d3e640 1 -- 172.21.15.1:0/281199828 >> 172.21.15.1:0/281199828 conn(0x7f71e40c6860 msgr2=0x7f71e40c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.190+0000 7f71e9d3e640 1 -- 172.21.15.1:0/281199828 shutdown_connections 2024-03-06T07:24:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.190+0000 7f71e9d3e640 1 -- 172.21.15.1:0/281199828 wait complete. 2024-03-06T07:24:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.191+0000 7f71e9d3e640 1 Processor -- start 2024-03-06T07:24:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.191+0000 7f71e9d3e640 1 -- start start 2024-03-06T07:24:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.192+0000 7f71e9d3e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e4072bd0 0x7f71e41db6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.192+0000 7f71e9d3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71e40c2b30 0x7f71e4144e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.192+0000 7f71e9d3e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71e4152b70 0x7f71e41480c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.192+0000 7f71e9d3e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e4155d80 con 0x7f71e4072bd0 2024-03-06T07:24:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.192+0000 7f71e9d3e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e41526e0 con 0x7f71e4152b70 2024-03-06T07:24:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.192+0000 7f71e9d3e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e4155f00 con 0x7f71e40c2b30 2024-03-06T07:24:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.192+0000 7f71e37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e4072bd0 0x7f71e41db6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.192+0000 7f71e37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e4072bd0 0x7f71e41db6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33538/0 (socket says 172.21.15.1:33538) 2024-03-06T07:24:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.192+0000 7f71e37fe640 1 -- 172.21.15.1:0/3992389687 learned_addr learned my addr 172.21.15.1:0/3992389687 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.192+0000 7f71e2ffd640 1 --2- 172.21.15.1:0/3992389687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71e40c2b30 0x7f71e4144e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.192+0000 7f71e3fff640 1 --2- 172.21.15.1:0/3992389687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71e4152b70 0x7f71e41480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.192+0000 7f71e37fe640 1 -- 172.21.15.1:0/3992389687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71e40c2b30 msgr2=0x7f71e4144e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.192+0000 7f71e37fe640 1 --2- 172.21.15.1:0/3992389687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71e40c2b30 0x7f71e4144e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.192+0000 7f71e37fe640 1 -- 172.21.15.1:0/3992389687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71e4152b70 msgr2=0x7f71e41480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.192+0000 7f71e37fe640 1 --2- 172.21.15.1:0/3992389687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71e4152b70 0x7f71e41480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.192+0000 7f71e37fe640 1 -- 172.21.15.1:0/3992389687 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e41e9a40 con 0x7f71e4072bd0 2024-03-06T07:24:48.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.193+0000 7f71e37fe640 1 --2- 172.21.15.1:0/3992389687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e4072bd0 0x7f71e41db6a0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f71d400c480 tx=0x7f71d400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.193+0000 7f71cbfff640 1 -- 172.21.15.1:0/3992389687 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d40052d0 con 0x7f71e4072bd0 2024-03-06T07:24:48.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.193+0000 7f71cbfff640 1 -- 172.21.15.1:0/3992389687 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d4005470 con 0x7f71e4072bd0 2024-03-06T07:24:48.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.193+0000 7f71e9d3e640 1 -- 172.21.15.1:0/3992389687 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e41475e0 con 0x7f71e4072bd0 2024-03-06T07:24:48.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.193+0000 7f71cbfff640 1 -- 172.21.15.1:0/3992389687 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d4012870 con 0x7f71e4072bd0 2024-03-06T07:24:48.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.193+0000 7f71e9d3e640 1 -- 172.21.15.1:0/3992389687 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e41dbbe0 con 0x7f71e4072bd0 2024-03-06T07:24:48.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.194+0000 7f71e9d3e640 1 -- 172.21.15.1:0/3992389687 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e4067fd0 con 0x7f71e4072bd0 2024-03-06T07:24:48.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.194+0000 7f71cbfff640 1 -- 172.21.15.1:0/3992389687 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71d4012a10 con 0x7f71e4072bd0 2024-03-06T07:24:48.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.194+0000 7f71cbfff640 1 --2- 172.21.15.1:0/3992389687 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f71d006b500 0x7f71d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.195+0000 7f71cbfff640 1 -- 172.21.15.1:0/3992389687 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f71d4089ba0 con 0x7f71e4072bd0 2024-03-06T07:24:48.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.195+0000 7f71e2ffd640 1 --2- 172.21.15.1:0/3992389687 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f71d006b500 0x7f71d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.195+0000 7f71e2ffd640 1 --2- 172.21.15.1:0/3992389687 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f71d006b500 0x7f71d006d9c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f71d8036000 tx=0x7f71d8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.199+0000 7f71cbfff640 1 -- 172.21.15.1:0/3992389687 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f71d4058580 con 0x7f71e4072bd0 2024-03-06T07:24:48.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:48.396+0000 7f71e9d3e640 1 -- 172.21.15.1:0/3992389687 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f71e4071790 con 0x7f71e4072bd0 2024-03-06T07:24:49.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.219+0000 7f71cbfff640 1 -- 172.21.15.1:0/3992389687 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v117) v1 ==== 97+0+0 (secure 0 0 0) 0x7f71d405c620 con 0x7f71e4072bd0 2024-03-06T07:24:49.221 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-06T07:24:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.222+0000 7f71e9d3e640 1 -- 172.21.15.1:0/3992389687 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f71d006b500 msgr2=0x7f71d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.222+0000 7f71e9d3e640 1 --2- 172.21.15.1:0/3992389687 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f71d006b500 0x7f71d006d9c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f71d8036000 tx=0x7f71d8008030 comp rx=0 tx=0).stop 2024-03-06T07:24:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.222+0000 7f71e9d3e640 1 -- 172.21.15.1:0/3992389687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e4072bd0 msgr2=0x7f71e41db6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.222+0000 7f71e9d3e640 1 --2- 172.21.15.1:0/3992389687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e4072bd0 0x7f71e41db6a0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f71d400c480 tx=0x7f71d400c940 comp rx=0 tx=0).stop 2024-03-06T07:24:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.222+0000 7f71e9d3e640 1 -- 172.21.15.1:0/3992389687 shutdown_connections 2024-03-06T07:24:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.222+0000 7f71e9d3e640 1 --2- 172.21.15.1:0/3992389687 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f71d006b500 0x7f71d006d9c0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.222+0000 7f71e9d3e640 1 --2- 172.21.15.1:0/3992389687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71e4152b70 0x7f71e41480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.222+0000 7f71e9d3e640 1 --2- 172.21.15.1:0/3992389687 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71e40c2b30 0x7f71e4144e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.222+0000 7f71e9d3e640 1 --2- 172.21.15.1:0/3992389687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e4072bd0 0x7f71e41db6a0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.222+0000 7f71e9d3e640 1 -- 172.21.15.1:0/3992389687 >> 172.21.15.1:0/3992389687 conn(0x7f71e40c6860 msgr2=0x7f71e4154e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.223+0000 7f71e9d3e640 1 -- 172.21.15.1:0/3992389687 shutdown_connections 2024-03-06T07:24:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.223+0000 7f71e9d3e640 1 -- 172.21.15.1:0/3992389687 wait complete. 2024-03-06T07:24:49.238 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:24:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.343+0000 7f1fefb3a640 1 Processor -- start 2024-03-06T07:24:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.343+0000 7f1fefb3a640 1 -- start start 2024-03-06T07:24:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.344+0000 7f1fefb3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fe8152820 0x7f1fe80c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.344+0000 7f1fefb3a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1fe80af6f0 0x7f1fe815fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.344+0000 7f1fefb3a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe80c7180 con 0x7f1fe80c0670 2024-03-06T07:24:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.344+0000 7f1fefb3a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe8155460 con 0x7f1fe80af6f0 2024-03-06T07:24:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.344+0000 7f1fefb3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe80c6bc0 con 0x7f1fe8152820 2024-03-06T07:24:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.344+0000 7f1fee0d7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1fe80af6f0 0x7f1fe815fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.344+0000 7f1fee0d7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1fe80af6f0 0x7f1fe815fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33546/0 (socket says 172.21.15.1:33546) 2024-03-06T07:24:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.344+0000 7f1fee0d7640 1 -- 172.21.15.1:0/2527472641 learned_addr learned my addr 172.21.15.1:0/2527472641 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.345+0000 7f1fee0d7640 1 -- 172.21.15.1:0/2527472641 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fe8152820 msgr2=0x7f1fe80c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.345+0000 7f1fed8d6640 1 --2- 172.21.15.1:0/2527472641 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fe8152820 0x7f1fe80c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.345+0000 7f1fee0d7640 1 --2- 172.21.15.1:0/2527472641 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fe8152820 0x7f1fe80c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.345+0000 7f1fee0d7640 1 -- 172.21.15.1:0/2527472641 >> v1:172.21.15.98:6789/0 conn(0x7f1fe80c0670 legacy=0x7f1fe80c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.345+0000 7f1fee0d7640 1 -- 172.21.15.1:0/2527472641 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe80c6ea0 con 0x7f1fe80af6f0 2024-03-06T07:24:49.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.345+0000 7f1fed8d6640 1 --2- 172.21.15.1:0/2527472641 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fe8152820 0x7f1fe80c0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:49.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.345+0000 7f1fee0d7640 1 --2- 172.21.15.1:0/2527472641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe80af6f0 0x7f1fe815fb60 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8009e50 tx=0x7f1fd802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=981a60fb774322c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.346+0000 7f1fec8d4640 1 -- 172.21.15.1:0/2527472641 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd802ca70 con 0x7f1fe80af6f0 2024-03-06T07:24:49.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.346+0000 7f1fec8d4640 1 -- 172.21.15.1:0/2527472641 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd802cc10 con 0x7f1fe80af6f0 2024-03-06T07:24:49.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.346+0000 7f1fec8d4640 1 -- 172.21.15.1:0/2527472641 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd80343f0 con 0x7f1fe80af6f0 2024-03-06T07:24:49.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.346+0000 7f1fefb3a640 1 -- 172.21.15.1:0/2527472641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe80af6f0 msgr2=0x7f1fe815fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.346+0000 7f1fefb3a640 1 --2- 172.21.15.1:0/2527472641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe80af6f0 0x7f1fe815fb60 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8009e50 tx=0x7f1fd802bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.346+0000 7f1fefb3a640 1 -- 172.21.15.1:0/2527472641 shutdown_connections 2024-03-06T07:24:49.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.346+0000 7f1fefb3a640 1 --2- 172.21.15.1:0/2527472641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe80af6f0 0x7f1fe815fb60 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.346+0000 7f1fefb3a640 1 --2- 172.21.15.1:0/2527472641 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fe8152820 0x7f1fe80c0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.346+0000 7f1fefb3a640 1 -- 172.21.15.1:0/2527472641 >> 172.21.15.1:0/2527472641 conn(0x7f1fe8072720 msgr2=0x7f1fe8072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.346+0000 7f1fefb3a640 1 -- 172.21.15.1:0/2527472641 shutdown_connections 2024-03-06T07:24:49.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.346+0000 7f1fefb3a640 1 -- 172.21.15.1:0/2527472641 wait complete. 2024-03-06T07:24:49.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.347+0000 7f1fefb3a640 1 Processor -- start 2024-03-06T07:24:49.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.347+0000 7f1fefb3a640 1 -- start start 2024-03-06T07:24:49.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.348+0000 7f1fefb3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe80af6f0 0x7f1fe80c3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.348+0000 7f1fefb3a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe80c0670 0x7f1fe8153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.348+0000 7f1fefb3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fe8152820 0x7f1fe80c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.348+0000 7f1fefb3a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe80c7c80 con 0x7f1fe80c0670 2024-03-06T07:24:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.348+0000 7f1fefb3a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe80c7b00 con 0x7f1fe80af6f0 2024-03-06T07:24:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.348+0000 7f1fefb3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe8152c00 con 0x7f1fe8152820 2024-03-06T07:24:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.348+0000 7f1fed0d5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe80c0670 0x7f1fe8153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.348+0000 7f1fed0d5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe80c0670 0x7f1fe8153e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33550/0 (socket says 172.21.15.1:33550) 2024-03-06T07:24:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.348+0000 7f1fed0d5640 1 -- 172.21.15.1:0/4256543736 learned_addr learned my addr 172.21.15.1:0/4256543736 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.348+0000 7f1fee0d7640 1 --2- 172.21.15.1:0/4256543736 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fe8152820 0x7f1fe80c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.348+0000 7f1fed8d6640 1 --2- 172.21.15.1:0/4256543736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe80af6f0 0x7f1fe80c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.349+0000 7f1fed0d5640 1 -- 172.21.15.1:0/4256543736 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fe8152820 msgr2=0x7f1fe80c42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.349+0000 7f1fed0d5640 1 --2- 172.21.15.1:0/4256543736 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fe8152820 0x7f1fe80c42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.349+0000 7f1fed0d5640 1 -- 172.21.15.1:0/4256543736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe80af6f0 msgr2=0x7f1fe80c3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.349+0000 7f1fed0d5640 1 --2- 172.21.15.1:0/4256543736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe80af6f0 0x7f1fe80c3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.349+0000 7f1fed0d5640 1 -- 172.21.15.1:0/4256543736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe80c35a0 con 0x7f1fe80c0670 2024-03-06T07:24:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.349+0000 7f1fed8d6640 1 --2- 172.21.15.1:0/4256543736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe80af6f0 0x7f1fe80c3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.349+0000 7f1fee0d7640 1 --2- 172.21.15.1:0/4256543736 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fe8152820 0x7f1fe80c42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.349+0000 7f1fed0d5640 1 --2- 172.21.15.1:0/4256543736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe80c0670 0x7f1fe8153e10 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00c400 tx=0x7f1fdc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.349+0000 7f1fe67fc640 1 -- 172.21.15.1:0/4256543736 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc011790 con 0x7f1fe80c0670 2024-03-06T07:24:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.349+0000 7f1fe67fc640 1 -- 172.21.15.1:0/4256543736 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc017070 con 0x7f1fe80c0670 2024-03-06T07:24:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.349+0000 7f1fefb3a640 1 -- 172.21.15.1:0/4256543736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe8158b40 con 0x7f1fe80c0670 2024-03-06T07:24:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.349+0000 7f1fe67fc640 1 -- 172.21.15.1:0/4256543736 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc011bb0 con 0x7f1fe80c0670 2024-03-06T07:24:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.349+0000 7f1fefb3a640 1 -- 172.21.15.1:0/4256543736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe8157120 con 0x7f1fe80c0670 2024-03-06T07:24:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.350+0000 7f1fefb3a640 1 -- 172.21.15.1:0/4256543736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe80686e0 con 0x7f1fe80c0670 2024-03-06T07:24:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.350+0000 7f1fe67fc640 1 -- 172.21.15.1:0/4256543736 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fdc01c460 con 0x7f1fe80c0670 2024-03-06T07:24:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.351+0000 7f1fe67fc640 1 --2- 172.21.15.1:0/4256543736 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1fd406b4d0 0x7f1fd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.351+0000 7f1fe67fc640 1 -- 172.21.15.1:0/4256543736 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f1fdc089ef0 con 0x7f1fe80c0670 2024-03-06T07:24:49.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.351+0000 7f1fed8d6640 1 --2- 172.21.15.1:0/4256543736 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1fd406b4d0 0x7f1fd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.351+0000 7f1fed8d6640 1 --2- 172.21.15.1:0/4256543736 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1fd406b4d0 0x7f1fd406d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0006fd0 tx=0x7f1fe0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.355+0000 7f1fe67fc640 1 -- 172.21.15.1:0/4256543736 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1fdc058f30 con 0x7f1fe80c0670 2024-03-06T07:24:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.606+0000 7f1fefb3a640 1 -- 172.21.15.1:0/4256543736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f1fe80c5b40 con 0x7f1fe80c0670 2024-03-06T07:24:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.606+0000 7f1fe67fc640 1 -- 172.21.15.1:0/4256543736 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f1fdc058830 con 0x7f1fe80c0670 2024-03-06T07:24:49.608 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlnxyWARAAKIPXU0IafNQ1RzBPX+3DYw==", 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:49.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:49.610 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-06T07:24:49.610 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl2fXxDRAAuLqH+HpKcN1JTts9ZtDTXA==", 2024-03-06T07:24:49.610 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:49.610 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:24:49.610 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:24:49.610 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:24:49.610 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:49.610 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:49.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:49.610 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-06T07:24:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl+IzqCBAAmgLkarFGt9Ihs/8kJYicBQ==", 2024-03-06T07:24:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:24:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:24:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:24:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-06T07:24:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlipdRFRAAA34vrsTEoX3L1CKb1YAV9g==", 2024-03-06T07:24:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:24:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:24:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:24:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-06T07:24:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAGGehlQK39LxAAOpPIZvl3FqQwCgyo3yoOGQ==", 2024-03-06T07:24:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:24:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-06T07:24:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAJGehlTxwDLhAAMwN9E0+3Jh5XNXnLo+fw3g==", 2024-03-06T07:24:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAMGehl4qrlLBAAm0VWQRNvSQIpDx5xf4Fyig==", 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-06T07:24:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAPGehllO5HKxAAozm6ycG+fRxatoZTogeJTA==", 2024-03-06T07:24:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:24:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-06T07:24:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAVGehlLtLBMhAAUrV6q6ZCQluIcoo/S/NdbA==", 2024-03-06T07:24:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:24:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-06T07:24:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAYGehlHp59MRAAxxgpSPTEZkYKjl3fr0xeeQ==", 2024-03-06T07:24:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:24:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-06T07:24:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAbGehlTnXcMBAAFOBqcnh5ubAz+o6AGjccWQ==", 2024-03-06T07:24:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:24:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-06T07:24:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAeGehl1qH1LhAAaSl7Aj7qlN6d4xt5SKAmcg==", 2024-03-06T07:24:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:24:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-06T07:24:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng==", 2024-03-06T07:24:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:24:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-06T07:24:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-06T07:24:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlMyv7HBAAi7vt7aZWEaZuMpFrB5ofZw==", 2024-03-06T07:24:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:24:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-06T07:24:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-06T07:24:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-06T07:24:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-06T07:24:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlBS/vIBAAN86N2K4ze5PJB8kR869Lfg==", 2024-03-06T07:24:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:24:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:24:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:24:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-06T07:24:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlJ/jPORAAhLk5MTGafsGKhZSYhvhBZg==", 2024-03-06T07:24:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:24:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:24:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-06T07:24:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlAl8rNhAAvdU0Qvo/FJYnxc4/H8MIqQ==", 2024-03-06T07:24:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:24:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:24:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:24:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.620 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:49.620 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-06T07:24:49.620 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:24:49.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.608+0000 7f1fefb3a640 1 -- 172.21.15.1:0/4256543736 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1fd406b4d0 msgr2=0x7f1fd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.608+0000 7f1fefb3a640 1 --2- 172.21.15.1:0/4256543736 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1fd406b4d0 0x7f1fd406d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0006fd0 tx=0x7f1fe0008040 comp rx=0 tx=0).stop 2024-03-06T07:24:49.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.608+0000 7f1fefb3a640 1 -- 172.21.15.1:0/4256543736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe80c0670 msgr2=0x7f1fe8153e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.608+0000 7f1fefb3a640 1 --2- 172.21.15.1:0/4256543736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe80c0670 0x7f1fe8153e10 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00c400 tx=0x7f1fdc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.608+0000 7f1fefb3a640 1 -- 172.21.15.1:0/4256543736 shutdown_connections 2024-03-06T07:24:49.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.608+0000 7f1fefb3a640 1 --2- 172.21.15.1:0/4256543736 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1fd406b4d0 0x7f1fd406d990 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.608+0000 7f1fefb3a640 1 --2- 172.21.15.1:0/4256543736 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fe8152820 0x7f1fe80c42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.608+0000 7f1fefb3a640 1 --2- 172.21.15.1:0/4256543736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe80c0670 0x7f1fe8153e10 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.608+0000 7f1fefb3a640 1 --2- 172.21.15.1:0/4256543736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe80af6f0 0x7f1fe80c3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.608+0000 7f1fefb3a640 1 -- 172.21.15.1:0/4256543736 >> 172.21.15.1:0/4256543736 conn(0x7f1fe8072720 msgr2=0x7f1fe80c5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.609+0000 7f1fefb3a640 1 -- 172.21.15.1:0/4256543736 shutdown_connections 2024-03-06T07:24:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.609+0000 7f1fefb3a640 1 -- 172.21.15.1:0/4256543736 wait complete. 2024-03-06T07:24:49.622 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:49.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.723+0000 7fb6feafa640 1 Processor -- start 2024-03-06T07:24:49.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.723+0000 7fb6feafa640 1 -- start start 2024-03-06T07:24:49.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.723+0000 7fb6feafa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb6f8152920 0x7fb6f806aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.723+0000 7fb6feafa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb6f806b530 0x7fb6f80c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.723+0000 7fb6feafa640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f8158910 con 0x7fb6f80c5d90 2024-03-06T07:24:49.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.723+0000 7fb6feafa640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f8152d00 con 0x7fb6f806b530 2024-03-06T07:24:49.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.723+0000 7fb6feafa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f81596d0 con 0x7fb6f8152920 2024-03-06T07:24:49.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.724+0000 7fb6f7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb6f806b530 0x7fb6f80c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.724+0000 7fb6fd097640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb6f80c5d90 0x7fb6f8155220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54488/0 (socket says 172.21.15.1:54488) 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.724+0000 7fb6fd097640 1 -- 172.21.15.1:0/90735781 learned_addr learned my addr 172.21.15.1:0/90735781 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.724+0000 7fb6fc896640 1 --2- 172.21.15.1:0/90735781 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb6f8152920 0x7fb6f806aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.724+0000 7fb6f7fff640 1 -- 172.21.15.1:0/90735781 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb6f8152920 msgr2=0x7fb6f806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.724+0000 7fb6f7fff640 1 --2- 172.21.15.1:0/90735781 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb6f8152920 0x7fb6f806aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.724+0000 7fb6f7fff640 1 -- 172.21.15.1:0/90735781 >> v1:172.21.15.1:6789/0 conn(0x7fb6f80c5d90 legacy=0x7fb6f8155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.724+0000 7fb6f7fff640 1 -- 172.21.15.1:0/90735781 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f8158bf0 con 0x7fb6f806b530 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.724+0000 7fb6f77fe640 1 -- 172.21.15.1:0/90735781 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230122572 0 0) 0x7fb6f8158910 con 0x7fb6f80c5d90 2024-03-06T07:24:49.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.725+0000 7fb6f7fff640 1 --2- 172.21.15.1:0/90735781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f806b530 0x7fb6f80c5850 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fb6e40097d0 tx=0x7fb6e402b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e617ec81c0f46016 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.725+0000 7fb6f77fe640 1 -- 172.21.15.1:0/90735781 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e4031770 con 0x7fb6f806b530 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.725+0000 7fb6f77fe640 1 -- 172.21.15.1:0/90735781 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e4037070 con 0x7fb6f806b530 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.725+0000 7fb6f77fe640 1 -- 172.21.15.1:0/90735781 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e4031b30 con 0x7fb6f806b530 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.725+0000 7fb6feafa640 1 -- 172.21.15.1:0/90735781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f806b530 msgr2=0x7fb6f80c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.725+0000 7fb6feafa640 1 --2- 172.21.15.1:0/90735781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f806b530 0x7fb6f80c5850 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fb6e40097d0 tx=0x7fb6e402b810 comp rx=0 tx=0).stop 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.725+0000 7fb6feafa640 1 -- 172.21.15.1:0/90735781 shutdown_connections 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.726+0000 7fb6feafa640 1 --2- 172.21.15.1:0/90735781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f806b530 0x7fb6f80c5850 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.726+0000 7fb6feafa640 1 --2- 172.21.15.1:0/90735781 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb6f8152920 0x7fb6f806aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.726+0000 7fb6feafa640 1 -- 172.21.15.1:0/90735781 >> 172.21.15.1:0/90735781 conn(0x7fb6f80a9700 msgr2=0x7fb6f80a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.726+0000 7fb6feafa640 1 -- 172.21.15.1:0/90735781 shutdown_connections 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.726+0000 7fb6feafa640 1 -- 172.21.15.1:0/90735781 wait complete. 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.727+0000 7fb6feafa640 1 Processor -- start 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.727+0000 7fb6feafa640 1 -- start start 2024-03-06T07:24:49.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.727+0000 7fb6feafa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f806b530 0x7fb6f81540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.727+0000 7fb6feafa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb6f80c5d90 0x7fb6f8154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.727+0000 7fb6feafa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f8152920 0x7fb6f81469a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.727+0000 7fb6feafa640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f80c6380 con 0x7fb6f8152920 2024-03-06T07:24:49.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.727+0000 7fb6feafa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f8159240 con 0x7fb6f806b530 2024-03-06T07:24:49.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.727+0000 7fb6feafa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f8155930 con 0x7fb6f80c5d90 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6fd097640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f8152920 0x7fb6f81469a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6fd097640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f8152920 0x7fb6f81469a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33556/0 (socket says 172.21.15.1:33556) 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6fd097640 1 -- 172.21.15.1:0/460106507 learned_addr learned my addr 172.21.15.1:0/460106507 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6fc896640 1 --2- 172.21.15.1:0/460106507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f806b530 0x7fb6f81540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6f7fff640 1 --2- 172.21.15.1:0/460106507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb6f80c5d90 0x7fb6f8154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6fd097640 1 -- 172.21.15.1:0/460106507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb6f80c5d90 msgr2=0x7fb6f8154600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6fd097640 1 --2- 172.21.15.1:0/460106507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb6f80c5d90 0x7fb6f8154600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6fd097640 1 -- 172.21.15.1:0/460106507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f806b530 msgr2=0x7fb6f81540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6fd097640 1 --2- 172.21.15.1:0/460106507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f806b530 0x7fb6f81540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6fd097640 1 -- 172.21.15.1:0/460106507 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f806fc10 con 0x7fb6f8152920 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6f7fff640 1 --2- 172.21.15.1:0/460106507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb6f80c5d90 0x7fb6f8154600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6fc896640 1 --2- 172.21.15.1:0/460106507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f806b530 0x7fb6f81540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6fd097640 1 --2- 172.21.15.1:0/460106507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f8152920 0x7fb6f81469a0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000c7e0 tx=0x7fb6f000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6f57fa640 1 -- 172.21.15.1:0/460106507 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0018070 con 0x7fb6f8152920 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6f57fa640 1 -- 172.21.15.1:0/460106507 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f000a640 con 0x7fb6f8152920 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6feafa640 1 -- 172.21.15.1:0/460106507 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f806fef0 con 0x7fb6f8152920 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6f57fa640 1 -- 172.21.15.1:0/460106507 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f000aa30 con 0x7fb6f8152920 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.728+0000 7fb6feafa640 1 -- 172.21.15.1:0/460106507 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f806e4b0 con 0x7fb6f8152920 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.729+0000 7fb6feafa640 1 -- 172.21.15.1:0/460106507 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f8067fd0 con 0x7fb6f8152920 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.730+0000 7fb6f57fa640 1 -- 172.21.15.1:0/460106507 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb6f000abd0 con 0x7fb6f8152920 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.730+0000 7fb6f57fa640 1 --2- 172.21.15.1:0/460106507 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb6dc06b4d0 0x7fb6dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.730+0000 7fb6f57fa640 1 -- 172.21.15.1:0/460106507 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb6f0089730 con 0x7fb6f8152920 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.730+0000 7fb6fc896640 1 --2- 172.21.15.1:0/460106507 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb6dc06b4d0 0x7fb6dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.731+0000 7fb6fc896640 1 --2- 172.21.15.1:0/460106507 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb6dc06b4d0 0x7fb6dc06d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec006fd0 tx=0x7fb6ec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.735+0000 7fb6f57fa640 1 -- 172.21.15.1:0/460106507 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb6f0058780 con 0x7fb6f8152920 2024-03-06T07:24:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.915+0000 7fb6feafa640 1 -- 172.21.15.1:0/460106507 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6f806bdf0 con 0x7fb6f8152920 2024-03-06T07:24:49.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.915+0000 7fb6f57fa640 1 -- 172.21.15.1:0/460106507 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+13054 (secure 0 0 0) 0x7fb6f0058080 con 0x7fb6f8152920 2024-03-06T07:24:49.917 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:49.918 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":117,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:24:49.218581+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:49.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.917+0000 7fb6feafa640 1 -- 172.21.15.1:0/460106507 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb6dc06b4d0 msgr2=0x7fb6dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.917+0000 7fb6feafa640 1 --2- 172.21.15.1:0/460106507 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb6dc06b4d0 0x7fb6dc06d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec006fd0 tx=0x7fb6ec008040 comp rx=0 tx=0).stop 2024-03-06T07:24:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.917+0000 7fb6feafa640 1 -- 172.21.15.1:0/460106507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f8152920 msgr2=0x7fb6f81469a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.917+0000 7fb6feafa640 1 --2- 172.21.15.1:0/460106507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f8152920 0x7fb6f81469a0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000c7e0 tx=0x7fb6f000cca0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.917+0000 7fb6feafa640 1 -- 172.21.15.1:0/460106507 shutdown_connections 2024-03-06T07:24:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.917+0000 7fb6feafa640 1 --2- 172.21.15.1:0/460106507 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb6dc06b4d0 0x7fb6dc06d990 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.917+0000 7fb6feafa640 1 --2- 172.21.15.1:0/460106507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f8152920 0x7fb6f81469a0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.917+0000 7fb6feafa640 1 --2- 172.21.15.1:0/460106507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb6f80c5d90 0x7fb6f8154600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.917+0000 7fb6feafa640 1 --2- 172.21.15.1:0/460106507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f806b530 0x7fb6f81540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.917+0000 7fb6feafa640 1 -- 172.21.15.1:0/460106507 >> 172.21.15.1:0/460106507 conn(0x7fb6f80a9700 msgr2=0x7fb6f814aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.917+0000 7fb6feafa640 1 -- 172.21.15.1:0/460106507 shutdown_connections 2024-03-06T07:24:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:49.917+0000 7fb6feafa640 1 -- 172.21.15.1:0/460106507 wait complete. 2024-03-06T07:24:49.930 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:49.931 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:50.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.033+0000 7f964a97a640 1 Processor -- start 2024-03-06T07:24:50.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.033+0000 7f964a97a640 1 -- start start 2024-03-06T07:24:50.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.034+0000 7f964a97a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9644152690 0x7f9644155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.034+0000 7f964a97a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9644156cd0 0x7f9644157160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.034+0000 7f964a97a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96441561e0 con 0x7f96440c6730 2024-03-06T07:24:50.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.034+0000 7f964a97a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96440c3300 con 0x7f9644156cd0 2024-03-06T07:24:50.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.034+0000 7f964a97a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96440c35a0 con 0x7f9644152690 2024-03-06T07:24:50.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.034+0000 7f9648f17640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9644156cd0 0x7f9644157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.034+0000 7f9643fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9644152690 0x7f9644155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.034+0000 7f9648f17640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9644156cd0 0x7f9644157160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:42468/0 (socket says 172.21.15.1:42468) 2024-03-06T07:24:50.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.034+0000 7f9643fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9644152690 0x7f9644155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:38390/0 (socket says 172.21.15.1:38390) 2024-03-06T07:24:50.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.034+0000 7f9648f17640 1 -- 172.21.15.1:0/2871491217 learned_addr learned my addr 172.21.15.1:0/2871491217 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:50.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.034+0000 7f9648f17640 1 -- 172.21.15.1:0/2871491217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9644152690 msgr2=0x7f9644155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.034+0000 7f9648f17640 1 --2- 172.21.15.1:0/2871491217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9644152690 0x7f9644155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.034+0000 7f9648f17640 1 -- 172.21.15.1:0/2871491217 >> v1:172.21.15.98:6789/0 conn(0x7f96440c6730 legacy=0x7f96440c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.034+0000 7f9648f17640 1 -- 172.21.15.1:0/2871491217 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96440c3880 con 0x7f9644156cd0 2024-03-06T07:24:50.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.034+0000 7f9643fff640 1 --2- 172.21.15.1:0/2871491217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9644152690 0x7f9644155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:50.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.035+0000 7f9648f17640 1 --2- 172.21.15.1:0/2871491217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9644156cd0 0x7f9644157160 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f963c009e50 tx=0x7f963c02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48656ac3d2f437de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.035+0000 7f9642ffd640 1 -- 172.21.15.1:0/2871491217 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963c02ca70 con 0x7f9644156cd0 2024-03-06T07:24:50.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.035+0000 7f9642ffd640 1 -- 172.21.15.1:0/2871491217 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f963c02cc10 con 0x7f9644156cd0 2024-03-06T07:24:50.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.035+0000 7f9642ffd640 1 -- 172.21.15.1:0/2871491217 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963c0343f0 con 0x7f9644156cd0 2024-03-06T07:24:50.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.036+0000 7f964a97a640 1 -- 172.21.15.1:0/2871491217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9644156cd0 msgr2=0x7f9644157160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.036+0000 7f964a97a640 1 --2- 172.21.15.1:0/2871491217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9644156cd0 0x7f9644157160 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f963c009e50 tx=0x7f963c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.036+0000 7f964a97a640 1 -- 172.21.15.1:0/2871491217 shutdown_connections 2024-03-06T07:24:50.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.036+0000 7f964a97a640 1 --2- 172.21.15.1:0/2871491217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9644156cd0 0x7f9644157160 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.036+0000 7f964a97a640 1 --2- 172.21.15.1:0/2871491217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9644152690 0x7f9644155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.036+0000 7f964a97a640 1 -- 172.21.15.1:0/2871491217 >> 172.21.15.1:0/2871491217 conn(0x7f964406b410 msgr2=0x7f964406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.036+0000 7f964a97a640 1 -- 172.21.15.1:0/2871491217 shutdown_connections 2024-03-06T07:24:50.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.036+0000 7f964a97a640 1 -- 172.21.15.1:0/2871491217 wait complete. 2024-03-06T07:24:50.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.037+0000 7f964a97a640 1 Processor -- start 2024-03-06T07:24:50.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.037+0000 7f964a97a640 1 -- start start 2024-03-06T07:24:50.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.038+0000 7f964a97a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96440c6730 0x7f96441eba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.038+0000 7f964a97a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9644152690 0x7f96441eef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.038+0000 7f964a97a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9644156cd0 0x7f96440c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.038+0000 7f964a97a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96440c5ee0 con 0x7f9644152690 2024-03-06T07:24:50.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.038+0000 7f964a97a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96441567b0 con 0x7f96440c6730 2024-03-06T07:24:50.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.038+0000 7f964a97a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96440c6060 con 0x7f9644156cd0 2024-03-06T07:24:50.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.038+0000 7f96437fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9644152690 0x7f96441eef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.038+0000 7f96437fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9644152690 0x7f96441eef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33560/0 (socket says 172.21.15.1:33560) 2024-03-06T07:24:50.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.038+0000 7f96437fe640 1 -- 172.21.15.1:0/3533555175 learned_addr learned my addr 172.21.15.1:0/3533555175 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:50.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.038+0000 7f9643fff640 1 --2- 172.21.15.1:0/3533555175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96440c6730 0x7f96441eba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.038+0000 7f9648f17640 1 --2- 172.21.15.1:0/3533555175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9644156cd0 0x7f96440c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.038+0000 7f96437fe640 1 -- 172.21.15.1:0/3533555175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9644156cd0 msgr2=0x7f96440c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.038+0000 7f96437fe640 1 --2- 172.21.15.1:0/3533555175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9644156cd0 0x7f96440c7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.038+0000 7f96437fe640 1 -- 172.21.15.1:0/3533555175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96440c6730 msgr2=0x7f96441eba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.038+0000 7f96437fe640 1 --2- 172.21.15.1:0/3533555175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96440c6730 0x7f96441eba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.038+0000 7f96437fe640 1 -- 172.21.15.1:0/3533555175 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96441eafb0 con 0x7f9644152690 2024-03-06T07:24:50.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.038+0000 7f9643fff640 1 --2- 172.21.15.1:0/3533555175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96440c6730 0x7f96441eba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:50.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.039+0000 7f96437fe640 1 --2- 172.21.15.1:0/3533555175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9644152690 0x7f96441eef30 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f963400c3c0 tx=0x7f963400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.039+0000 7f9640ff9640 1 -- 172.21.15.1:0/3533555175 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96340052d0 con 0x7f9644152690 2024-03-06T07:24:50.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.039+0000 7f9640ff9640 1 -- 172.21.15.1:0/3533555175 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9634005490 con 0x7f9644152690 2024-03-06T07:24:50.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.039+0000 7f964a97a640 1 -- 172.21.15.1:0/3533555175 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964414f990 con 0x7f9644152690 2024-03-06T07:24:50.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.039+0000 7f9640ff9640 1 -- 172.21.15.1:0/3533555175 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9634007260 con 0x7f9644152690 2024-03-06T07:24:50.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.039+0000 7f964a97a640 1 -- 172.21.15.1:0/3533555175 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f964414ff50 con 0x7f9644152690 2024-03-06T07:24:50.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.040+0000 7f964a97a640 1 -- 172.21.15.1:0/3533555175 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9644067fd0 con 0x7f9644152690 2024-03-06T07:24:50.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.040+0000 7f9640ff9640 1 -- 172.21.15.1:0/3533555175 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9634007400 con 0x7f9644152690 2024-03-06T07:24:50.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.041+0000 7f9640ff9640 1 --2- 172.21.15.1:0/3533555175 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f962406b4d0 0x7f962406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.041+0000 7f9640ff9640 1 -- 172.21.15.1:0/3533555175 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f96340894e0 con 0x7f9644152690 2024-03-06T07:24:50.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.041+0000 7f9643fff640 1 --2- 172.21.15.1:0/3533555175 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f962406b4d0 0x7f962406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.042+0000 7f9643fff640 1 --2- 172.21.15.1:0/3533555175 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f962406b4d0 0x7f962406d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f962c009a20 tx=0x7f962c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.045+0000 7f9640ff9640 1 -- 172.21.15.1:0/3533555175 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96441561e0 con 0x7f9644152690 2024-03-06T07:24:50.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.223+0000 7f964a97a640 1 -- 172.21.15.1:0/3533555175 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f96440ad5d0 con 0x7f9644152690 2024-03-06T07:24:50.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.224+0000 7f9640ff9640 1 -- 172.21.15.1:0/3533555175 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9634058510 con 0x7f9644152690 2024-03-06T07:24:50.226 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:24:50.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.226+0000 7f964a97a640 1 -- 172.21.15.1:0/3533555175 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f962406b4d0 msgr2=0x7f962406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.226+0000 7f964a97a640 1 --2- 172.21.15.1:0/3533555175 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f962406b4d0 0x7f962406d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f962c009a20 tx=0x7f962c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:50.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.226+0000 7f964a97a640 1 -- 172.21.15.1:0/3533555175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9644152690 msgr2=0x7f96441eef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.226+0000 7f964a97a640 1 --2- 172.21.15.1:0/3533555175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9644152690 0x7f96441eef30 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f963400c3c0 tx=0x7f963400c880 comp rx=0 tx=0).stop 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.226+0000 7f964a97a640 1 -- 172.21.15.1:0/3533555175 shutdown_connections 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.226+0000 7f964a97a640 1 --2- 172.21.15.1:0/3533555175 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f962406b4d0 0x7f962406d990 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.226+0000 7f964a97a640 1 --2- 172.21.15.1:0/3533555175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9644156cd0 0x7f96440c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.226+0000 7f964a97a640 1 --2- 172.21.15.1:0/3533555175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9644152690 0x7f96441eef30 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.226+0000 7f964a97a640 1 --2- 172.21.15.1:0/3533555175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96440c6730 0x7f96441eba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.226+0000 7f964a97a640 1 -- 172.21.15.1:0/3533555175 >> 172.21.15.1:0/3533555175 conn(0x7f964406b410 msgr2=0x7f96440c57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.226+0000 7f964a97a640 1 -- 172.21.15.1:0/3533555175 shutdown_connections 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.226+0000 7f964a97a640 1 -- 172.21.15.1:0/3533555175 wait complete. 2024-03-06T07:24:50.239 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:24:50.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.338+0000 7f8971fa0640 1 Processor -- start 2024-03-06T07:24:50.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.338+0000 7f8971fa0640 1 -- start start 2024-03-06T07:24:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.339+0000 7f8971fa0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f896c0c49b0 0x7f896c0c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.339+0000 7f8971fa0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f896c0c7e60 0x7f896c0c3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.339+0000 7f8971fa0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f896c1532a0 con 0x7f896c0c8a70 2024-03-06T07:24:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.339+0000 7f8971fa0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f896c0c8ef0 con 0x7f896c0c7e60 2024-03-06T07:24:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.339+0000 7f8971fa0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f896c071530 con 0x7f896c0c49b0 2024-03-06T07:24:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.339+0000 7f896bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f896c0c8a70 0x7f896c0c7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54494/0 (socket says 172.21.15.1:54494) 2024-03-06T07:24:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.339+0000 7f896bfff640 1 -- 172.21.15.1:0/3706345918 learned_addr learned my addr 172.21.15.1:0/3706345918 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.339+0000 7f897179f640 1 --2- 172.21.15.1:0/3706345918 >> v2:172.21.15.98:3300/0 conn(0x7f896c0c7e60 0x7f896c0c3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.339+0000 7f8970f9e640 1 --2- 172.21.15.1:0/3706345918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f896c0c49b0 0x7f896c0c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.339+0000 7f896b7fe640 1 -- 172.21.15.1:0/3706345918 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372660390 0 0) 0x7f896c1532a0 con 0x7f896c0c8a70 2024-03-06T07:24:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f896b7fe640 1 -- 172.21.15.1:0/3706345918 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8954003520 con 0x7f896c0c8a70 2024-03-06T07:24:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f896b7fe640 1 -- 172.21.15.1:0/3706345918 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2443203991 0 0) 0x7f8954003520 con 0x7f896c0c8a70 2024-03-06T07:24:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f896b7fe640 1 -- 172.21.15.1:0/3706345918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f896c0c49b0 msgr2=0x7f896c0c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f896b7fe640 1 --2- 172.21.15.1:0/3706345918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f896c0c49b0 0x7f896c0c8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f896b7fe640 1 -- 172.21.15.1:0/3706345918 >> v2:172.21.15.98:3300/0 conn(0x7f896c0c7e60 msgr2=0x7f896c0c3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f896b7fe640 1 --2- 172.21.15.1:0/3706345918 >> v2:172.21.15.98:3300/0 conn(0x7f896c0c7e60 0x7f896c0c3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f896b7fe640 1 -- 172.21.15.1:0/3706345918 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896c153580 con 0x7f896c0c8a70 2024-03-06T07:24:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f896b7fe640 1 -- 172.21.15.1:0/3706345918 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f8960001670 con 0x7f896c0c8a70 2024-03-06T07:24:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f896b7fe640 1 -- 172.21.15.1:0/3706345918 >> v1:172.21.15.1:6789/0 conn(0x7f896c0c8a70 legacy=0x7f896c0c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f896b7fe640 1 --2- 172.21.15.1:0/3706345918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89540039e0 0x7f8954003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f896b7fe640 1 --2- 172.21.15.1:0/3706345918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f896c0c49b0 0x7f8954004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f896b7fe640 1 --2- 172.21.15.1:0/3706345918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8954004d40 0x7f89540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f896b7fe640 1 -- 172.21.15.1:0/3706345918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896c0c8ef0 con 0x7f8954004d40 2024-03-06T07:24:50.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f896b7fe640 1 -- 172.21.15.1:0/3706345918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895400b9c0 con 0x7f89540039e0 2024-03-06T07:24:50.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f896b7fe640 1 -- 172.21.15.1:0/3706345918 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f895400bb70 con 0x7f896c0c49b0 2024-03-06T07:24:50.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f897179f640 1 --2- 172.21.15.1:0/3706345918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8954004d40 0x7f89540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f8970f9e640 1 --2- 172.21.15.1:0/3706345918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89540039e0 0x7f8954003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f896bfff640 1 --2- 172.21.15.1:0/3706345918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f896c0c49b0 0x7f8954004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f897179f640 1 -- 172.21.15.1:0/3706345918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f896c0c49b0 msgr2=0x7f8954004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f897179f640 1 --2- 172.21.15.1:0/3706345918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f896c0c49b0 0x7f8954004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f897179f640 1 -- 172.21.15.1:0/3706345918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89540039e0 msgr2=0x7f8954003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f897179f640 1 --2- 172.21.15.1:0/3706345918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89540039e0 0x7f8954003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.340+0000 7f897179f640 1 -- 172.21.15.1:0/3706345918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8954003520 con 0x7f8954004d40 2024-03-06T07:24:50.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.341+0000 7f897179f640 1 --2- 172.21.15.1:0/3706345918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8954004d40 0x7f89540051c0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f895c009ac0 tx=0x7f8954005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c27fba4f15459fd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.341+0000 7f896b7fe640 1 -- 172.21.15.1:0/3706345918 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895c039070 con 0x7f8954004d40 2024-03-06T07:24:50.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.341+0000 7f896b7fe640 1 -- 172.21.15.1:0/3706345918 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f895c02c710 con 0x7f8954004d40 2024-03-06T07:24:50.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.341+0000 7f896b7fe640 1 -- 172.21.15.1:0/3706345918 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895c009690 con 0x7f8954004d40 2024-03-06T07:24:50.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.342+0000 7f8971fa0640 1 -- 172.21.15.1:0/3706345918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8954004d40 msgr2=0x7f89540051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.342+0000 7f8971fa0640 1 --2- 172.21.15.1:0/3706345918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8954004d40 0x7f89540051c0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f895c009ac0 tx=0x7f8954005830 comp rx=0 tx=0).stop 2024-03-06T07:24:50.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.342+0000 7f897179f640 1 -- 172.21.15.1:0/3706345918 reap_dead start 2024-03-06T07:24:50.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.342+0000 7f8971fa0640 1 -- 172.21.15.1:0/3706345918 shutdown_connections 2024-03-06T07:24:50.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.342+0000 7f8971fa0640 1 -- 172.21.15.1:0/3706345918 >> 172.21.15.1:0/3706345918 conn(0x7f896c06b620 msgr2=0x7f896c0aa4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.342+0000 7f8971fa0640 1 -- 172.21.15.1:0/3706345918 shutdown_connections 2024-03-06T07:24:50.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.342+0000 7f8971fa0640 1 -- 172.21.15.1:0/3706345918 wait complete. 2024-03-06T07:24:50.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.343+0000 7f8971fa0640 1 Processor -- start 2024-03-06T07:24:50.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.343+0000 7f8971fa0640 1 -- start start 2024-03-06T07:24:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f8971fa0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f896c1568e0 0x7f896c0c1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f8971fa0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f896c06ded0 0x7f896c1eda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f8971fa0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f896c1eb150 0x7f896c1de560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f8971fa0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896c153b40 con 0x7f896c1eb150 2024-03-06T07:24:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f8971fa0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896c150960 con 0x7f896c1568e0 2024-03-06T07:24:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f8971fa0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f896c152f90 con 0x7f896c06ded0 2024-03-06T07:24:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f897179f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f896c1eb150 0x7f896c1de560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f897179f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f896c1eb150 0x7f896c1de560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33568/0 (socket says 172.21.15.1:33568) 2024-03-06T07:24:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f897179f640 1 -- 172.21.15.1:0/531285200 learned_addr learned my addr 172.21.15.1:0/531285200 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f8970f9e640 1 --2- 172.21.15.1:0/531285200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f896c1568e0 0x7f896c0c1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f896bfff640 1 --2- 172.21.15.1:0/531285200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f896c06ded0 0x7f896c1eda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f897179f640 1 -- 172.21.15.1:0/531285200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f896c06ded0 msgr2=0x7f896c1eda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f897179f640 1 --2- 172.21.15.1:0/531285200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f896c06ded0 0x7f896c1eda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f897179f640 1 -- 172.21.15.1:0/531285200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f896c1568e0 msgr2=0x7f896c0c1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f897179f640 1 --2- 172.21.15.1:0/531285200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f896c1568e0 0x7f896c0c1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f897179f640 1 -- 172.21.15.1:0/531285200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896c1ea690 con 0x7f896c1eb150 2024-03-06T07:24:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f8970f9e640 1 --2- 172.21.15.1:0/531285200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f896c1568e0 0x7f896c0c1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f897179f640 1 --2- 172.21.15.1:0/531285200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f896c1eb150 0x7f896c1de560 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f895c009be0 tx=0x7f895c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f89697fa640 1 -- 172.21.15.1:0/531285200 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895c039040 con 0x7f896c1eb150 2024-03-06T07:24:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f89697fa640 1 -- 172.21.15.1:0/531285200 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f895c043070 con 0x7f896c1eb150 2024-03-06T07:24:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f8971fa0640 1 -- 172.21.15.1:0/531285200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f896c1546f0 con 0x7f896c1eb150 2024-03-06T07:24:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.344+0000 7f89697fa640 1 -- 172.21.15.1:0/531285200 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895c004470 con 0x7f896c1eb150 2024-03-06T07:24:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.345+0000 7f8971fa0640 1 -- 172.21.15.1:0/531285200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f896c1eac30 con 0x7f896c1eb150 2024-03-06T07:24:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.345+0000 7f8971fa0640 1 -- 172.21.15.1:0/531285200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f896c1eae30 con 0x7f896c1eb150 2024-03-06T07:24:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.346+0000 7f89697fa640 1 -- 172.21.15.1:0/531285200 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f895c003610 con 0x7f896c1eb150 2024-03-06T07:24:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.346+0000 7f89697fa640 1 --2- 172.21.15.1:0/531285200 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f895806b4d0 0x7f895806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.347+0000 7f89697fa640 1 -- 172.21.15.1:0/531285200 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f895c0ab1b0 con 0x7f896c1eb150 2024-03-06T07:24:50.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.347+0000 7f8970f9e640 1 --2- 172.21.15.1:0/531285200 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f895806b4d0 0x7f895806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.347+0000 7f8970f9e640 1 --2- 172.21.15.1:0/531285200 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f895806b4d0 0x7f895806d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f896c0c78c0 tx=0x7f896400b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.351+0000 7f89697fa640 1 -- 172.21.15.1:0/531285200 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f896c1eae30 con 0x7f896c1eb150 2024-03-06T07:24:50.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.532+0000 7f8971fa0640 1 -- 172.21.15.1:0/531285200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f896c1ea4a0 con 0x7f896c1eb150 2024-03-06T07:24:50.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.532+0000 7f89697fa640 1 -- 172.21.15.1:0/531285200 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v117) v1 ==== 84+0+65 (secure 0 0 0) 0x7f895c04c020 con 0x7f896c1eb150 2024-03-06T07:24:50.534 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:50.535 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-06T07:24:50.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:24:50.535 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-06T07:24:50.535 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-06T07:24:50.535 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:24:50.535 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.534+0000 7f8971fa0640 1 -- 172.21.15.1:0/531285200 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f895806b4d0 msgr2=0x7f895806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.534+0000 7f8971fa0640 1 --2- 172.21.15.1:0/531285200 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f895806b4d0 0x7f895806d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f896c0c78c0 tx=0x7f896400b040 comp rx=0 tx=0).stop 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.534+0000 7f8971fa0640 1 -- 172.21.15.1:0/531285200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f896c1eb150 msgr2=0x7f896c1de560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.534+0000 7f8971fa0640 1 --2- 172.21.15.1:0/531285200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f896c1eb150 0x7f896c1de560 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f895c009be0 tx=0x7f895c004730 comp rx=0 tx=0).stop 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.535+0000 7f8971fa0640 1 -- 172.21.15.1:0/531285200 shutdown_connections 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.535+0000 7f8971fa0640 1 --2- 172.21.15.1:0/531285200 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f895806b4d0 0x7f895806d990 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.535+0000 7f8971fa0640 1 --2- 172.21.15.1:0/531285200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f896c1eb150 0x7f896c1de560 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.535+0000 7f8971fa0640 1 --2- 172.21.15.1:0/531285200 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f896c06ded0 0x7f896c1eda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.535+0000 7f8971fa0640 1 --2- 172.21.15.1:0/531285200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f896c1568e0 0x7f896c0c1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.535+0000 7f8971fa0640 1 -- 172.21.15.1:0/531285200 >> 172.21.15.1:0/531285200 conn(0x7f896c06b620 msgr2=0x7f896c0aa490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.535+0000 7f8971fa0640 1 -- 172.21.15.1:0/531285200 shutdown_connections 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.535+0000 7f8971fa0640 1 -- 172.21.15.1:0/531285200 wait complete. 2024-03-06T07:24:50.547 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:24:50.548 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:24:50.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.651+0000 7f8f1bc78640 1 Processor -- start 2024-03-06T07:24:50.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.651+0000 7f8f1bc78640 1 -- start start 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.652+0000 7f8f1bc78640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f1406f100 0x7f8f1406f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.652+0000 7f8f1bc78640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f140a9790 con 0x7f8f1406c3f0 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.652+0000 7f8f1bc78640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f140a9a70 con 0x7f8f14152c00 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.652+0000 7f8f1bc78640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f140c3990 con 0x7f8f1406f100 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.652+0000 7f8f19213640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8f1406c3f0 0x7f8f14158800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54496/0 (socket says 172.21.15.1:54496) 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.652+0000 7f8f19213640 1 -- 172.21.15.1:0/1324324195 learned_addr learned my addr 172.21.15.1:0/1324324195 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.652+0000 7f8f18a12640 1 -- 172.21.15.1:0/1324324195 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094486884 0 0) 0x7f8f140a9790 con 0x7f8f1406c3f0 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.652+0000 7f8f18a12640 1 -- 172.21.15.1:0/1324324195 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f00003520 con 0x7f8f1406c3f0 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.652+0000 7f8f1a215640 1 --2- 172.21.15.1:0/1324324195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f1406f100 0x7f8f1406f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.652+0000 7f8f18a12640 1 -- 172.21.15.1:0/1324324195 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2041491284 0 0) 0x7f8f00003520 con 0x7f8f1406c3f0 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.652+0000 7f8f18a12640 1 -- 172.21.15.1:0/1324324195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f1406f100 msgr2=0x7f8f1406f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.652+0000 7f8f18a12640 1 --2- 172.21.15.1:0/1324324195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f1406f100 0x7f8f1406f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.652+0000 7f8f18a12640 1 -- 172.21.15.1:0/1324324195 >> v1:172.21.15.98:6789/0 conn(0x7f8f14152c00 legacy=0x7f8f14072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.653+0000 7f8f18a12640 1 -- 172.21.15.1:0/1324324195 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f140a9d50 con 0x7f8f1406c3f0 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.653+0000 7f8f18a12640 1 -- 172.21.15.1:0/1324324195 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f8f04001670 con 0x7f8f1406c3f0 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.653+0000 7f8f18a12640 1 -- 172.21.15.1:0/1324324195 >> v1:172.21.15.1:6789/0 conn(0x7f8f1406c3f0 legacy=0x7f8f14158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.653+0000 7f8f18a12640 1 --2- 172.21.15.1:0/1324324195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f1406f100 0x7f8f000039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.653+0000 7f8f18a12640 1 --2- 172.21.15.1:0/1324324195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f00003f20 0x7f8f00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.653+0000 7f8f18a12640 1 --2- 172.21.15.1:0/1324324195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f00004d40 0x7f8f000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.653+0000 7f8f18a12640 1 -- 172.21.15.1:0/1324324195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f140c3990 con 0x7f8f00003f20 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.653+0000 7f8f18a12640 1 -- 172.21.15.1:0/1324324195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0000ba20 con 0x7f8f00004d40 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.653+0000 7f8f18a12640 1 -- 172.21.15.1:0/1324324195 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0000bc20 con 0x7f8f1406f100 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.653+0000 7f8f19213640 1 --2- 172.21.15.1:0/1324324195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f00003f20 0x7f8f00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.653+0000 7f8f1a215640 1 --2- 172.21.15.1:0/1324324195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f00004d40 0x7f8f000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.653+0000 7f8f19a14640 1 --2- 172.21.15.1:0/1324324195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f1406f100 0x7f8f000039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.653+0000 7f8f19213640 1 -- 172.21.15.1:0/1324324195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f1406f100 msgr2=0x7f8f000039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.653+0000 7f8f19213640 1 --2- 172.21.15.1:0/1324324195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f1406f100 0x7f8f000039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.653+0000 7f8f19213640 1 -- 172.21.15.1:0/1324324195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f00004d40 msgr2=0x7f8f000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.653+0000 7f8f19213640 1 --2- 172.21.15.1:0/1324324195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f00004d40 0x7f8f000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.653+0000 7f8f19213640 1 -- 172.21.15.1:0/1324324195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f00003520 con 0x7f8f00003f20 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.654+0000 7f8f19213640 1 --2- 172.21.15.1:0/1324324195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f00003f20 0x7f8f00004800 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400c240 tx=0x7f8f00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73250085384211 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.654+0000 7f8f18a12640 1 -- 172.21.15.1:0/1324324195 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04007d30 con 0x7f8f00003f20 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.654+0000 7f8f18a12640 1 -- 172.21.15.1:0/1324324195 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f04030070 con 0x7f8f00003f20 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.654+0000 7f8f18a12640 1 -- 172.21.15.1:0/1324324195 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04005d70 con 0x7f8f00003f20 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.654+0000 7f8f1bc78640 1 -- 172.21.15.1:0/1324324195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f00003f20 msgr2=0x7f8f00004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.654+0000 7f8f1bc78640 1 --2- 172.21.15.1:0/1324324195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f00003f20 0x7f8f00004800 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400c240 tx=0x7f8f00005830 comp rx=0 tx=0).stop 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.654+0000 7f8f1a215640 1 -- 172.21.15.1:0/1324324195 reap_dead start 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.654+0000 7f8f1bc78640 1 -- 172.21.15.1:0/1324324195 shutdown_connections 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.654+0000 7f8f1bc78640 1 -- 172.21.15.1:0/1324324195 >> 172.21.15.1:0/1324324195 conn(0x7f8f140c76b0 msgr2=0x7f8f140c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.655+0000 7f8f1bc78640 1 -- 172.21.15.1:0/1324324195 shutdown_connections 2024-03-06T07:24:50.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.655+0000 7f8f1bc78640 1 -- 172.21.15.1:0/1324324195 wait complete. 2024-03-06T07:24:50.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.656+0000 7f8f1bc78640 1 Processor -- start 2024-03-06T07:24:50.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.656+0000 7f8f1bc78640 1 -- start start 2024-03-06T07:24:50.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.656+0000 7f8f1bc78640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f00003f20 0x7f8f1414ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.656+0000 7f8f1bc78640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f14156600 0x7f8f140c0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.656+0000 7f8f1bc78640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f140c0c60 0x7f8f141eae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.656+0000 7f8f1bc78640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f140aa030 con 0x7f8f140c0c60 2024-03-06T07:24:50.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.656+0000 7f8f1bc78640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1406bf60 con 0x7f8f14156600 2024-03-06T07:24:50.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.656+0000 7f8f1bc78640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f140724a0 con 0x7f8f00003f20 2024-03-06T07:24:50.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.656+0000 7f8f1a215640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f140c0c60 0x7f8f141eae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.656+0000 7f8f1a215640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f140c0c60 0x7f8f141eae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33590/0 (socket says 172.21.15.1:33590) 2024-03-06T07:24:50.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.656+0000 7f8f1a215640 1 -- 172.21.15.1:0/543753843 learned_addr learned my addr 172.21.15.1:0/543753843 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:50.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.657+0000 7f8f19213640 1 --2- 172.21.15.1:0/543753843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f14156600 0x7f8f140c0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.657+0000 7f8f19a14640 1 --2- 172.21.15.1:0/543753843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f00003f20 0x7f8f1414ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.657+0000 7f8f1a215640 1 -- 172.21.15.1:0/543753843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f00003f20 msgr2=0x7f8f1414ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.657+0000 7f8f1a215640 1 --2- 172.21.15.1:0/543753843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f00003f20 0x7f8f1414ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.657+0000 7f8f1a215640 1 -- 172.21.15.1:0/543753843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f14156600 msgr2=0x7f8f140c0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.657+0000 7f8f1a215640 1 --2- 172.21.15.1:0/543753843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f14156600 0x7f8f140c0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.657+0000 7f8f1a215640 1 -- 172.21.15.1:0/543753843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f140ac4b0 con 0x7f8f140c0c60 2024-03-06T07:24:50.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.657+0000 7f8f19213640 1 --2- 172.21.15.1:0/543753843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f14156600 0x7f8f140c0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:50.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.657+0000 7f8f19a14640 1 --2- 172.21.15.1:0/543753843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f00003f20 0x7f8f1414ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:50.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.657+0000 7f8f1a215640 1 --2- 172.21.15.1:0/543753843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f140c0c60 0x7f8f141eae60 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c011bf0 tx=0x7f8f0c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.657+0000 7f8f0a7fc640 1 -- 172.21.15.1:0/543753843 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c00fdb0 con 0x7f8f140c0c60 2024-03-06T07:24:50.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.657+0000 7f8f0a7fc640 1 -- 172.21.15.1:0/543753843 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0c015430 con 0x7f8f140c0c60 2024-03-06T07:24:50.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.657+0000 7f8f0a7fc640 1 -- 172.21.15.1:0/543753843 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c015820 con 0x7f8f140c0c60 2024-03-06T07:24:50.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.657+0000 7f8f1bc78640 1 -- 172.21.15.1:0/543753843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f141df4f0 con 0x7f8f140c0c60 2024-03-06T07:24:50.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.657+0000 7f8f1bc78640 1 -- 172.21.15.1:0/543753843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f141ee9b0 con 0x7f8f140c0c60 2024-03-06T07:24:50.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.658+0000 7f8f1bc78640 1 -- 172.21.15.1:0/543753843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f140680b0 con 0x7f8f140c0c60 2024-03-06T07:24:50.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.658+0000 7f8f0a7fc640 1 -- 172.21.15.1:0/543753843 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f0c0159c0 con 0x7f8f140c0c60 2024-03-06T07:24:50.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.659+0000 7f8f0a7fc640 1 --2- 172.21.15.1:0/543753843 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ef806b500 0x7f8ef806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.659+0000 7f8f0a7fc640 1 -- 172.21.15.1:0/543753843 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8f0c08ae10 con 0x7f8f140c0c60 2024-03-06T07:24:50.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.659+0000 7f8f19a14640 1 --2- 172.21.15.1:0/543753843 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ef806b500 0x7f8ef806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.660+0000 7f8f19a14640 1 --2- 172.21.15.1:0/543753843 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ef806b500 0x7f8ef806d9c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8f10003700 tx=0x7f8f10009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.663+0000 7f8f0a7fc640 1 -- 172.21.15.1:0/543753843 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f0c019030 con 0x7f8f140c0c60 2024-03-06T07:24:50.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:50.845+0000 7f8f1bc78640 1 -- 172.21.15.1:0/543753843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8f140c3b30 con 0x7f8f140c0c60 2024-03-06T07:24:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.225+0000 7f8f0a7fc640 1 -- 172.21.15.1:0/543753843 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v118) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8f0c010040 con 0x7f8f140c0c60 2024-03-06T07:24:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-06T07:24:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.227+0000 7f8f1bc78640 1 -- 172.21.15.1:0/543753843 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ef806b500 msgr2=0x7f8ef806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.227+0000 7f8f1bc78640 1 --2- 172.21.15.1:0/543753843 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ef806b500 0x7f8ef806d9c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8f10003700 tx=0x7f8f10009040 comp rx=0 tx=0).stop 2024-03-06T07:24:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.227+0000 7f8f1bc78640 1 -- 172.21.15.1:0/543753843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f140c0c60 msgr2=0x7f8f141eae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.227+0000 7f8f1bc78640 1 --2- 172.21.15.1:0/543753843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f140c0c60 0x7f8f141eae60 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c011bf0 tx=0x7f8f0c00f580 comp rx=0 tx=0).stop 2024-03-06T07:24:51.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.227+0000 7f8f1bc78640 1 -- 172.21.15.1:0/543753843 shutdown_connections 2024-03-06T07:24:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.227+0000 7f8f1bc78640 1 --2- 172.21.15.1:0/543753843 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ef806b500 0x7f8ef806d9c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.227+0000 7f8f1bc78640 1 --2- 172.21.15.1:0/543753843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f140c0c60 0x7f8f141eae60 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.227+0000 7f8f1bc78640 1 --2- 172.21.15.1:0/543753843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f14156600 0x7f8f140c0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.227+0000 7f8f1bc78640 1 --2- 172.21.15.1:0/543753843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f00003f20 0x7f8f1414ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.227+0000 7f8f1bc78640 1 -- 172.21.15.1:0/543753843 >> 172.21.15.1:0/543753843 conn(0x7f8f140c76b0 msgr2=0x7f8f14153aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.227+0000 7f8f1bc78640 1 -- 172.21.15.1:0/543753843 shutdown_connections 2024-03-06T07:24:51.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.227+0000 7f8f1bc78640 1 -- 172.21.15.1:0/543753843 wait complete. 2024-03-06T07:24:51.243 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:24:51.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.350+0000 7fafd86fb640 1 Processor -- start 2024-03-06T07:24:51.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.350+0000 7fafd86fb640 1 -- start start 2024-03-06T07:24:51.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.351+0000 7fafd86fb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fafd00c6b10 0x7fafd0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.351+0000 7fafd86fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafd00bf9d0 0x7fafd00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.351+0000 7fafd86fb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd00c2fc0 con 0x7fafd0152650 2024-03-06T07:24:51.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.351+0000 7fafd86fb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafd0155800 con 0x7fafd00c6b10 2024-03-06T07:24:51.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.351+0000 7fafd86fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd0158da0 con 0x7fafd00bf9d0 2024-03-06T07:24:51.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.351+0000 7fafd5c96640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fafd00c6b10 0x7fafd0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.351+0000 7fafd5c96640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fafd00c6b10 0x7fafd0155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33598/0 (socket says 172.21.15.1:33598) 2024-03-06T07:24:51.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.351+0000 7fafd5c96640 1 -- 172.21.15.1:0/67393669 learned_addr learned my addr 172.21.15.1:0/67393669 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:51.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.352+0000 7fafd6c98640 1 --2- 172.21.15.1:0/67393669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafd00bf9d0 0x7fafd00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.352+0000 7fafd5c96640 1 -- 172.21.15.1:0/67393669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafd00bf9d0 msgr2=0x7fafd00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.352+0000 7fafd5c96640 1 --2- 172.21.15.1:0/67393669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafd00bf9d0 0x7fafd00c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.352+0000 7fafd5c96640 1 -- 172.21.15.1:0/67393669 >> v1:172.21.15.98:6789/0 conn(0x7fafd0152650 legacy=0x7fafd0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.352+0000 7fafd5c96640 1 -- 172.21.15.1:0/67393669 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd00c6690 con 0x7fafd00c6b10 2024-03-06T07:24:51.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.352+0000 7fafd5c96640 1 --2- 172.21.15.1:0/67393669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafd00c6b10 0x7fafd0155c90 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fafd00cb8e0 tx=0x7fafc002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60abcf345cb32c6f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.353+0000 7fafd5495640 1 -- 172.21.15.1:0/67393669 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc00315a0 con 0x7fafd00c6b10 2024-03-06T07:24:51.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.353+0000 7fafd5495640 1 -- 172.21.15.1:0/67393669 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc0037070 con 0x7fafd00c6b10 2024-03-06T07:24:51.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.353+0000 7fafd5495640 1 -- 172.21.15.1:0/67393669 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc0031960 con 0x7fafd00c6b10 2024-03-06T07:24:51.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.353+0000 7fafd86fb640 1 -- 172.21.15.1:0/67393669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafd00c6b10 msgr2=0x7fafd0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.353+0000 7fafd86fb640 1 --2- 172.21.15.1:0/67393669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafd00c6b10 0x7fafd0155c90 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fafd00cb8e0 tx=0x7fafc002b840 comp rx=0 tx=0).stop 2024-03-06T07:24:51.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.353+0000 7fafd86fb640 1 -- 172.21.15.1:0/67393669 shutdown_connections 2024-03-06T07:24:51.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.353+0000 7fafd86fb640 1 --2- 172.21.15.1:0/67393669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafd00bf9d0 0x7fafd00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.353+0000 7fafd86fb640 1 --2- 172.21.15.1:0/67393669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafd00c6b10 0x7fafd0155c90 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.353+0000 7fafd86fb640 1 -- 172.21.15.1:0/67393669 >> 172.21.15.1:0/67393669 conn(0x7fafd0066ee0 msgr2=0x7fafd00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.353+0000 7fafd86fb640 1 -- 172.21.15.1:0/67393669 shutdown_connections 2024-03-06T07:24:51.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.353+0000 7fafd86fb640 1 -- 172.21.15.1:0/67393669 wait complete. 2024-03-06T07:24:51.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.354+0000 7fafd86fb640 1 Processor -- start 2024-03-06T07:24:51.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.354+0000 7fafd86fb640 1 -- start start 2024-03-06T07:24:51.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.355+0000 7fafd86fb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd00bf9d0 0x7fafd01572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.355+0000 7fafd86fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafd0152650 0x7fafd01509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.355+0000 7fafd86fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafd01519e0 0x7fafd00c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.355+0000 7fafd86fb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd01551b0 con 0x7fafd01519e0 2024-03-06T07:24:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.355+0000 7fafd86fb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd00c6fa0 con 0x7fafd00bf9d0 2024-03-06T07:24:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.355+0000 7fafd86fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd0155490 con 0x7fafd0152650 2024-03-06T07:24:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.355+0000 7fafd6c98640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafd01519e0 0x7fafd00c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.355+0000 7fafd6c98640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafd01519e0 0x7fafd00c87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33608/0 (socket says 172.21.15.1:33608) 2024-03-06T07:24:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.355+0000 7fafd6c98640 1 -- 172.21.15.1:0/3103153584 learned_addr learned my addr 172.21.15.1:0/3103153584 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.355+0000 7fafd6497640 1 --2- 172.21.15.1:0/3103153584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd00bf9d0 0x7fafd01572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.355+0000 7fafd5c96640 1 --2- 172.21.15.1:0/3103153584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafd0152650 0x7fafd01509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.355+0000 7fafd6c98640 1 -- 172.21.15.1:0/3103153584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafd0152650 msgr2=0x7fafd01509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.355+0000 7fafd6c98640 1 --2- 172.21.15.1:0/3103153584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafd0152650 0x7fafd01509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.355+0000 7fafd6c98640 1 -- 172.21.15.1:0/3103153584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd00bf9d0 msgr2=0x7fafd01572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.355+0000 7fafd6c98640 1 --2- 172.21.15.1:0/3103153584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd00bf9d0 0x7fafd01572f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.355+0000 7fafd6c98640 1 -- 172.21.15.1:0/3103153584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd01eb5c0 con 0x7fafd01519e0 2024-03-06T07:24:51.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.356+0000 7fafd6c98640 1 --2- 172.21.15.1:0/3103153584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafd01519e0 0x7fafd00c87d0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fafc800cc60 tx=0x7fafc800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.356+0000 7fafc6ffd640 1 -- 172.21.15.1:0/3103153584 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc8003990 con 0x7fafd01519e0 2024-03-06T07:24:51.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.356+0000 7fafc6ffd640 1 -- 172.21.15.1:0/3103153584 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc8003b30 con 0x7fafd01519e0 2024-03-06T07:24:51.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.356+0000 7fafc6ffd640 1 -- 172.21.15.1:0/3103153584 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc8004020 con 0x7fafd01519e0 2024-03-06T07:24:51.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.356+0000 7fafd86fb640 1 -- 172.21.15.1:0/3103153584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd0156530 con 0x7fafd01519e0 2024-03-06T07:24:51.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.356+0000 7fafd86fb640 1 -- 172.21.15.1:0/3103153584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd0156af0 con 0x7fafd01519e0 2024-03-06T07:24:51.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.357+0000 7fafd86fb640 1 -- 172.21.15.1:0/3103153584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd00677f0 con 0x7fafd01519e0 2024-03-06T07:24:51.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.357+0000 7fafc6ffd640 1 -- 172.21.15.1:0/3103153584 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fafc80041c0 con 0x7fafd01519e0 2024-03-06T07:24:51.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.358+0000 7fafc6ffd640 1 --2- 172.21.15.1:0/3103153584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fafb406b4d0 0x7fafb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.358+0000 7fafc6ffd640 1 -- 172.21.15.1:0/3103153584 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fafc808a540 con 0x7fafd01519e0 2024-03-06T07:24:51.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.358+0000 7fafd6497640 1 --2- 172.21.15.1:0/3103153584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fafb406b4d0 0x7fafb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.358+0000 7fafd6497640 1 --2- 172.21.15.1:0/3103153584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fafb406b4d0 0x7fafb406d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fafd014f7b0 tx=0x7fafcc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.362+0000 7fafc6ffd640 1 -- 172.21.15.1:0/3103153584 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fafd00677f0 con 0x7fafd01519e0 2024-03-06T07:24:51.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:51.544+0000 7fafd86fb640 1 -- 172.21.15.1:0/3103153584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fafd0067fd0 con 0x7fafd01519e0 2024-03-06T07:24:52.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.232+0000 7fafc6ffd640 1 -- 172.21.15.1:0/3103153584 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v119) v1 ==== 170+0+0 (secure 0 0 0) 0x7fafc8014030 con 0x7fafd01519e0 2024-03-06T07:24:52.234 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-06T07:24:52.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.235+0000 7fafd86fb640 1 -- 172.21.15.1:0/3103153584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fafb406b4d0 msgr2=0x7fafb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.235+0000 7fafd86fb640 1 --2- 172.21.15.1:0/3103153584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fafb406b4d0 0x7fafb406d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fafd014f7b0 tx=0x7fafcc009040 comp rx=0 tx=0).stop 2024-03-06T07:24:52.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.235+0000 7fafd86fb640 1 -- 172.21.15.1:0/3103153584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafd01519e0 msgr2=0x7fafd00c87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.235+0000 7fafd86fb640 1 --2- 172.21.15.1:0/3103153584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafd01519e0 0x7fafd00c87d0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fafc800cc60 tx=0x7fafc800a5c0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.235+0000 7fafd86fb640 1 -- 172.21.15.1:0/3103153584 shutdown_connections 2024-03-06T07:24:52.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.235+0000 7fafd86fb640 1 --2- 172.21.15.1:0/3103153584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fafb406b4d0 0x7fafb406d990 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.235+0000 7fafd86fb640 1 --2- 172.21.15.1:0/3103153584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafd01519e0 0x7fafd00c87d0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.235+0000 7fafd86fb640 1 --2- 172.21.15.1:0/3103153584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafd0152650 0x7fafd01509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.235+0000 7fafd86fb640 1 --2- 172.21.15.1:0/3103153584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd00bf9d0 0x7fafd01572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.235+0000 7fafd86fb640 1 -- 172.21.15.1:0/3103153584 >> 172.21.15.1:0/3103153584 conn(0x7fafd0066ee0 msgr2=0x7fafd00c3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.235+0000 7fafd86fb640 1 -- 172.21.15.1:0/3103153584 shutdown_connections 2024-03-06T07:24:52.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.235+0000 7fafd86fb640 1 -- 172.21.15.1:0/3103153584 wait complete. 2024-03-06T07:24:52.261 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:24:52.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.365+0000 7f68a88f4640 1 Processor -- start 2024-03-06T07:24:52.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.365+0000 7f68a88f4640 1 -- start start 2024-03-06T07:24:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.366+0000 7f68a88f4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f68a4152650 0x7f68a4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.366+0000 7f68a88f4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f68a40bf9d0 0x7f68a40c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.366+0000 7f68a88f4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a40c2fc0 con 0x7f68a40c6b10 2024-03-06T07:24:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.366+0000 7f68a88f4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a4155800 con 0x7f68a4152650 2024-03-06T07:24:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.366+0000 7f68a88f4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a4158da0 con 0x7f68a40bf9d0 2024-03-06T07:24:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.366+0000 7f68a259c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f68a4152650 0x7f68a4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.366+0000 7f68a259c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f68a4152650 0x7f68a4155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33610/0 (socket says 172.21.15.1:33610) 2024-03-06T07:24:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.366+0000 7f68a259c640 1 -- 172.21.15.1:0/3144081251 learned_addr learned my addr 172.21.15.1:0/3144081251 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.367+0000 7f68a2d9d640 1 --2- 172.21.15.1:0/3144081251 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f68a40bf9d0 0x7f68a40c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.367+0000 7f68a259c640 1 -- 172.21.15.1:0/3144081251 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f68a40bf9d0 msgr2=0x7f68a40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.367+0000 7f68a259c640 1 --2- 172.21.15.1:0/3144081251 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f68a40bf9d0 0x7f68a40c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.367+0000 7f68a259c640 1 -- 172.21.15.1:0/3144081251 >> v1:172.21.15.98:6789/0 conn(0x7f68a40c6b10 legacy=0x7f68a4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.367+0000 7f68a259c640 1 -- 172.21.15.1:0/3144081251 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a40c6690 con 0x7f68a4152650 2024-03-06T07:24:52.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.367+0000 7f68a259c640 1 --2- 172.21.15.1:0/3144081251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a4152650 0x7f68a4155c90 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f6894009870 tx=0x7f689402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d19a9b5ab5a31a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.368+0000 7f68a159a640 1 -- 172.21.15.1:0/3144081251 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6894035020 con 0x7f68a4152650 2024-03-06T07:24:52.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.368+0000 7f68a159a640 1 -- 172.21.15.1:0/3144081251 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6894031640 con 0x7f68a4152650 2024-03-06T07:24:52.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.368+0000 7f68a159a640 1 -- 172.21.15.1:0/3144081251 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689402d040 con 0x7f68a4152650 2024-03-06T07:24:52.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.368+0000 7f68a88f4640 1 -- 172.21.15.1:0/3144081251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a4152650 msgr2=0x7f68a4155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.368+0000 7f68a88f4640 1 --2- 172.21.15.1:0/3144081251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a4152650 0x7f68a4155c90 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f6894009870 tx=0x7f689402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.368+0000 7f68a88f4640 1 -- 172.21.15.1:0/3144081251 shutdown_connections 2024-03-06T07:24:52.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.368+0000 7f68a88f4640 1 --2- 172.21.15.1:0/3144081251 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f68a40bf9d0 0x7f68a40c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.368+0000 7f68a88f4640 1 --2- 172.21.15.1:0/3144081251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a4152650 0x7f68a4155c90 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.368+0000 7f68a88f4640 1 -- 172.21.15.1:0/3144081251 >> 172.21.15.1:0/3144081251 conn(0x7f68a4066ee0 msgr2=0x7f68a40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.368+0000 7f68a88f4640 1 -- 172.21.15.1:0/3144081251 shutdown_connections 2024-03-06T07:24:52.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.368+0000 7f68a88f4640 1 -- 172.21.15.1:0/3144081251 wait complete. 2024-03-06T07:24:52.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.369+0000 7f68a88f4640 1 Processor -- start 2024-03-06T07:24:52.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.369+0000 7f68a88f4640 1 -- start start 2024-03-06T07:24:52.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.370+0000 7f68a88f4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68a40bf9d0 0x7f68a4158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.370+0000 7f68a88f4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f68a4152650 0x7f68a4156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.370+0000 7f68a88f4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a4150940 0x7f68a414b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.370+0000 7f68a88f4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a41551b0 con 0x7f68a4150940 2024-03-06T07:24:52.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.370+0000 7f68a88f4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a40c6fa0 con 0x7f68a40bf9d0 2024-03-06T07:24:52.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.370+0000 7f68a88f4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a4155490 con 0x7f68a4152650 2024-03-06T07:24:52.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.370+0000 7f68a2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a4150940 0x7f68a414b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.370+0000 7f68a2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a4150940 0x7f68a414b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33618/0 (socket says 172.21.15.1:33618) 2024-03-06T07:24:52.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.370+0000 7f68a2d9d640 1 -- 172.21.15.1:0/3011635733 learned_addr learned my addr 172.21.15.1:0/3011635733 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:52.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.371+0000 7f68a1d9b640 1 --2- 172.21.15.1:0/3011635733 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f68a4152650 0x7f68a4156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.371+0000 7f68a259c640 1 --2- 172.21.15.1:0/3011635733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68a40bf9d0 0x7f68a4158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.371+0000 7f68a2d9d640 1 -- 172.21.15.1:0/3011635733 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f68a4152650 msgr2=0x7f68a4156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.371+0000 7f68a2d9d640 1 --2- 172.21.15.1:0/3011635733 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f68a4152650 0x7f68a4156b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.371+0000 7f68a2d9d640 1 -- 172.21.15.1:0/3011635733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68a40bf9d0 msgr2=0x7f68a4158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.371+0000 7f68a2d9d640 1 --2- 172.21.15.1:0/3011635733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68a40bf9d0 0x7f68a4158800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.371+0000 7f68a2d9d640 1 -- 172.21.15.1:0/3011635733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a41eed40 con 0x7f68a4150940 2024-03-06T07:24:52.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.371+0000 7f68a2d9d640 1 --2- 172.21.15.1:0/3011635733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a4150940 0x7f68a414b090 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f689c007e60 tx=0x7f689c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.371+0000 7f6892ffd640 1 -- 172.21.15.1:0/3011635733 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c014420 con 0x7f68a4150940 2024-03-06T07:24:52.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.371+0000 7f6892ffd640 1 -- 172.21.15.1:0/3011635733 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689c019070 con 0x7f68a4150940 2024-03-06T07:24:52.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.371+0000 7f68a88f4640 1 -- 172.21.15.1:0/3011635733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a41ef020 con 0x7f68a4150940 2024-03-06T07:24:52.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.371+0000 7f6892ffd640 1 -- 172.21.15.1:0/3011635733 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c014820 con 0x7f68a4150940 2024-03-06T07:24:52.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.371+0000 7f68a88f4640 1 -- 172.21.15.1:0/3011635733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a414bbe0 con 0x7f68a4150940 2024-03-06T07:24:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.372+0000 7f68a88f4640 1 -- 172.21.15.1:0/3011635733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a4067fd0 con 0x7f68a4150940 2024-03-06T07:24:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.373+0000 7f6892ffd640 1 -- 172.21.15.1:0/3011635733 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f689c0149c0 con 0x7f68a4150940 2024-03-06T07:24:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.373+0000 7f6892ffd640 1 --2- 172.21.15.1:0/3011635733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f688406b4d0 0x7f688406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.373+0000 7f6892ffd640 1 -- 172.21.15.1:0/3011635733 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f689c08ab60 con 0x7f68a4150940 2024-03-06T07:24:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.373+0000 7f68a259c640 1 --2- 172.21.15.1:0/3011635733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f688406b4d0 0x7f688406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.374+0000 7f68a259c640 1 --2- 172.21.15.1:0/3011635733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f688406b4d0 0x7f688406d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f6894036000 tx=0x7f6894008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.378+0000 7f6892ffd640 1 -- 172.21.15.1:0/3011635733 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f689c0598b0 con 0x7f68a4150940 2024-03-06T07:24:52.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:52.596+0000 7f68a88f4640 1 -- 172.21.15.1:0/3011635733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f68a40b06f0 con 0x7f68a4150940 2024-03-06T07:24:53.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.241+0000 7f6892ffd640 1 -- 172.21.15.1:0/3011635733 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 19 and data pool 20 v102) v1 ==== 174+0+0 (secure 0 0 0) 0x7f689c0591b0 con 0x7f68a4150940 2024-03-06T07:24:53.252 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 19 and data pool 20 2024-03-06T07:24:53.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.256+0000 7f68a88f4640 1 -- 172.21.15.1:0/3011635733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f688406b4d0 msgr2=0x7f688406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.256+0000 7f68a88f4640 1 --2- 172.21.15.1:0/3011635733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f688406b4d0 0x7f688406d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f6894036000 tx=0x7f6894008030 comp rx=0 tx=0).stop 2024-03-06T07:24:53.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.256+0000 7f68a88f4640 1 -- 172.21.15.1:0/3011635733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a4150940 msgr2=0x7f68a414b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.256+0000 7f68a88f4640 1 --2- 172.21.15.1:0/3011635733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a4150940 0x7f68a414b090 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f689c007e60 tx=0x7f689c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.259+0000 7f68a88f4640 1 -- 172.21.15.1:0/3011635733 shutdown_connections 2024-03-06T07:24:53.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.259+0000 7f68a88f4640 1 --2- 172.21.15.1:0/3011635733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f688406b4d0 0x7f688406d990 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.259+0000 7f68a88f4640 1 --2- 172.21.15.1:0/3011635733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a4150940 0x7f68a414b090 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.259+0000 7f68a88f4640 1 --2- 172.21.15.1:0/3011635733 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f68a4152650 0x7f68a4156b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.259+0000 7f68a88f4640 1 --2- 172.21.15.1:0/3011635733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68a40bf9d0 0x7f68a4158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.259+0000 7f68a88f4640 1 -- 172.21.15.1:0/3011635733 >> 172.21.15.1:0/3011635733 conn(0x7f68a4066ee0 msgr2=0x7f68a40bfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.260+0000 7f68a88f4640 1 -- 172.21.15.1:0/3011635733 shutdown_connections 2024-03-06T07:24:53.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.261+0000 7f68a88f4640 1 -- 172.21.15.1:0/3011635733 wait complete. 2024-03-06T07:24:53.279 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:53.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.382+0000 7f97934df640 1 Processor -- start 2024-03-06T07:24:53.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.382+0000 7f97934df640 1 -- start start 2024-03-06T07:24:53.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.383+0000 7f97934df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f978c0c2b30 0x7f978c0724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.383+0000 7f97934df640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f978c152b70 0x7f978c072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.383+0000 7f97934df640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978c151dd0 con 0x7f978c0a5fa0 2024-03-06T07:24:53.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.383+0000 7f97934df640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978c1526e0 con 0x7f978c152b70 2024-03-06T07:24:53.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.383+0000 7f97934df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c156190 con 0x7f978c0c2b30 2024-03-06T07:24:53.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.383+0000 7f979127b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f978c0c2b30 0x7f978c0724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.383+0000 7f9790a7a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f978c152b70 0x7f978c072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.383+0000 7f9790a7a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f978c152b70 0x7f978c072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:42532/0 (socket says 172.21.15.1:42532) 2024-03-06T07:24:53.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.383+0000 7f9790a7a640 1 -- 172.21.15.1:0/465632951 learned_addr learned my addr 172.21.15.1:0/465632951 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:53.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.383+0000 7f979127b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f978c0c2b30 0x7f978c0724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:38472/0 (socket says 172.21.15.1:38472) 2024-03-06T07:24:53.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.384+0000 7f9790a7a640 1 -- 172.21.15.1:0/465632951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f978c0c2b30 msgr2=0x7f978c0724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.384+0000 7f9790a7a640 1 --2- 172.21.15.1:0/465632951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f978c0c2b30 0x7f978c0724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.384+0000 7f9790a7a640 1 -- 172.21.15.1:0/465632951 >> v1:172.21.15.98:6789/0 conn(0x7f978c0a5fa0 legacy=0x7f978c0c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.384+0000 7f9790a7a640 1 -- 172.21.15.1:0/465632951 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c152090 con 0x7f978c152b70 2024-03-06T07:24:53.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.384+0000 7f979127b640 1 --2- 172.21.15.1:0/465632951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f978c0c2b30 0x7f978c0724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:53.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.385+0000 7f9790a7a640 1 --2- 172.21.15.1:0/465632951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f978c152b70 0x7f978c072a00 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9780009870 tx=0x7f978002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=408e120b5c4e20fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.385+0000 7f978bfff640 1 -- 172.21.15.1:0/465632951 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9780034070 con 0x7f978c152b70 2024-03-06T07:24:53.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.385+0000 7f978bfff640 1 -- 172.21.15.1:0/465632951 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9780030840 con 0x7f978c152b70 2024-03-06T07:24:53.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.385+0000 7f978bfff640 1 -- 172.21.15.1:0/465632951 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9780030c10 con 0x7f978c152b70 2024-03-06T07:24:53.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.385+0000 7f97934df640 1 -- 172.21.15.1:0/465632951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f978c152b70 msgr2=0x7f978c072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.385+0000 7f97934df640 1 --2- 172.21.15.1:0/465632951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f978c152b70 0x7f978c072a00 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9780009870 tx=0x7f978002b980 comp rx=0 tx=0).stop 2024-03-06T07:24:53.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.385+0000 7f97934df640 1 -- 172.21.15.1:0/465632951 shutdown_connections 2024-03-06T07:24:53.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.385+0000 7f97934df640 1 --2- 172.21.15.1:0/465632951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f978c152b70 0x7f978c072a00 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.385+0000 7f97934df640 1 --2- 172.21.15.1:0/465632951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f978c0c2b30 0x7f978c0724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.385+0000 7f97934df640 1 -- 172.21.15.1:0/465632951 >> 172.21.15.1:0/465632951 conn(0x7f978c0c6860 msgr2=0x7f978c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.385+0000 7f97934df640 1 -- 172.21.15.1:0/465632951 shutdown_connections 2024-03-06T07:24:53.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.385+0000 7f97934df640 1 -- 172.21.15.1:0/465632951 wait complete. 2024-03-06T07:24:53.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.386+0000 7f97934df640 1 Processor -- start 2024-03-06T07:24:53.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.386+0000 7f97934df640 1 -- start start 2024-03-06T07:24:53.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.387+0000 7f97934df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f978c0a5fa0 0x7f978c0ac560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.387+0000 7f97934df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f978c0c2b30 0x7f978c1dd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.387+0000 7f97934df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f978c152b70 0x7f978c0bedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.387+0000 7f97934df640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c155f00 con 0x7f978c0c2b30 2024-03-06T07:24:53.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.387+0000 7f97934df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c155d80 con 0x7f978c152b70 2024-03-06T07:24:53.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.387+0000 7f97934df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0c3a00 con 0x7f978c0a5fa0 2024-03-06T07:24:53.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.387+0000 7f9790a7a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f978c0c2b30 0x7f978c1dd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.387+0000 7f9790a7a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f978c0c2b30 0x7f978c1dd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33628/0 (socket says 172.21.15.1:33628) 2024-03-06T07:24:53.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.387+0000 7f9790a7a640 1 -- 172.21.15.1:0/2548961103 learned_addr learned my addr 172.21.15.1:0/2548961103 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:53.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.387+0000 7f979127b640 1 --2- 172.21.15.1:0/2548961103 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f978c0a5fa0 0x7f978c0ac560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.387+0000 7f9791a7c640 1 --2- 172.21.15.1:0/2548961103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f978c152b70 0x7f978c0bedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.388+0000 7f9790a7a640 1 -- 172.21.15.1:0/2548961103 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f978c0a5fa0 msgr2=0x7f978c0ac560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.388+0000 7f9790a7a640 1 --2- 172.21.15.1:0/2548961103 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f978c0a5fa0 0x7f978c0ac560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.388+0000 7f9790a7a640 1 -- 172.21.15.1:0/2548961103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f978c152b70 msgr2=0x7f978c0bedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.388+0000 7f9790a7a640 1 --2- 172.21.15.1:0/2548961103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f978c152b70 0x7f978c0bedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.388+0000 7f9790a7a640 1 -- 172.21.15.1:0/2548961103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c1ed210 con 0x7f978c0c2b30 2024-03-06T07:24:53.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.388+0000 7f9790a7a640 1 --2- 172.21.15.1:0/2548961103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f978c0c2b30 0x7f978c1dd640 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f9780000c00 tx=0x7f97800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.388+0000 7f9789ffb640 1 -- 172.21.15.1:0/2548961103 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9780034040 con 0x7f978c0c2b30 2024-03-06T07:24:53.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.388+0000 7f9789ffb640 1 -- 172.21.15.1:0/2548961103 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9780039d90 con 0x7f978c0c2b30 2024-03-06T07:24:53.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.388+0000 7f97934df640 1 -- 172.21.15.1:0/2548961103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c0be8e0 con 0x7f978c0c2b30 2024-03-06T07:24:53.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.388+0000 7f9789ffb640 1 -- 172.21.15.1:0/2548961103 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9780043630 con 0x7f978c0c2b30 2024-03-06T07:24:53.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.388+0000 7f97934df640 1 -- 172.21.15.1:0/2548961103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c0a6fd0 con 0x7f978c0c2b30 2024-03-06T07:24:53.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.389+0000 7f97934df640 1 -- 172.21.15.1:0/2548961103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978c0beab0 con 0x7f978c0c2b30 2024-03-06T07:24:53.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.390+0000 7f9789ffb640 1 -- 172.21.15.1:0/2548961103 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f978004a060 con 0x7f978c0c2b30 2024-03-06T07:24:53.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.390+0000 7f9789ffb640 1 --2- 172.21.15.1:0/2548961103 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f977806b4d0 0x7f977806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.390+0000 7f9789ffb640 1 -- 172.21.15.1:0/2548961103 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f97800ac140 con 0x7f978c0c2b30 2024-03-06T07:24:53.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.390+0000 7f979127b640 1 --2- 172.21.15.1:0/2548961103 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f977806b4d0 0x7f977806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.391+0000 7f979127b640 1 --2- 172.21.15.1:0/2548961103 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f977806b4d0 0x7f977806d990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9784009560 tx=0x7f9784009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.394+0000 7f9789ffb640 1 -- 172.21.15.1:0/2548961103 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f978003e070 con 0x7f978c0c2b30 2024-03-06T07:24:53.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.581+0000 7f97934df640 1 -- 172.21.15.1:0/2548961103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f978c071790 con 0x7f978c0c2b30 2024-03-06T07:24:53.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.582+0000 7f9789ffb640 1 -- 172.21.15.1:0/2548961103 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16793 (secure 0 0 0) 0x7f978007a870 con 0x7f978c0c2b30 2024-03-06T07:24:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:53.584 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":120,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:24:52.597984+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:50.865151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:51.564469+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:53.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.584+0000 7f97934df640 1 -- 172.21.15.1:0/2548961103 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f977806b4d0 msgr2=0x7f977806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.584+0000 7f97934df640 1 --2- 172.21.15.1:0/2548961103 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f977806b4d0 0x7f977806d990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9784009560 tx=0x7f9784009190 comp rx=0 tx=0).stop 2024-03-06T07:24:53.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.584+0000 7f97934df640 1 -- 172.21.15.1:0/2548961103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f978c0c2b30 msgr2=0x7f978c1dd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.584+0000 7f97934df640 1 --2- 172.21.15.1:0/2548961103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f978c0c2b30 0x7f978c1dd640 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f9780000c00 tx=0x7f97800046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.584+0000 7f97934df640 1 -- 172.21.15.1:0/2548961103 shutdown_connections 2024-03-06T07:24:53.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.584+0000 7f97934df640 1 --2- 172.21.15.1:0/2548961103 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f977806b4d0 0x7f977806d990 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.584+0000 7f97934df640 1 --2- 172.21.15.1:0/2548961103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f978c152b70 0x7f978c0bedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.584+0000 7f97934df640 1 --2- 172.21.15.1:0/2548961103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f978c0c2b30 0x7f978c1dd640 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.584+0000 7f97934df640 1 --2- 172.21.15.1:0/2548961103 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f978c0a5fa0 0x7f978c0ac560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.584+0000 7f97934df640 1 -- 172.21.15.1:0/2548961103 >> 172.21.15.1:0/2548961103 conn(0x7f978c0c6860 msgr2=0x7f978c0c6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.584+0000 7f97934df640 1 -- 172.21.15.1:0/2548961103 shutdown_connections 2024-03-06T07:24:53.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.584+0000 7f97934df640 1 -- 172.21.15.1:0/2548961103 wait complete. 2024-03-06T07:24:53.597 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:53.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.697+0000 7f59293e2640 1 Processor -- start 2024-03-06T07:24:53.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.697+0000 7f59293e2640 1 -- start start 2024-03-06T07:24:53.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59293e2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f59240c90f0 0x7f59241556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59293e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5924155bf0 0x7f592415fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59293e2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5924155200 con 0x7f59240c8d10 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59293e2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59240c33e0 con 0x7f59240c90f0 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59293e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59240c36c0 con 0x7f5924155bf0 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f5922ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f59240c8d10 0x7f59241590f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54504/0 (socket says 172.21.15.1:54504) 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f5922ffd640 1 -- 172.21.15.1:0/551935979 learned_addr learned my addr 172.21.15.1:0/551935979 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59227fc640 1 -- 172.21.15.1:0/551935979 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977854224 0 0) 0x7f5924155200 con 0x7f59240c8d10 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59227fc640 1 -- 172.21.15.1:0/551935979 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5910003520 con 0x7f59240c8d10 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f591bfff640 1 --2- 172.21.15.1:0/551935979 >> v2:172.21.15.98:3300/0 conn(0x7f59240c90f0 0x7f59241556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59237fe640 1 --2- 172.21.15.1:0/551935979 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5924155bf0 0x7f592415fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59227fc640 1 -- 172.21.15.1:0/551935979 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3742236928 0 0) 0x7f5910003520 con 0x7f59240c8d10 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59227fc640 1 -- 172.21.15.1:0/551935979 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5924155bf0 msgr2=0x7f592415fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59227fc640 1 --2- 172.21.15.1:0/551935979 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5924155bf0 0x7f592415fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59227fc640 1 -- 172.21.15.1:0/551935979 >> v2:172.21.15.98:3300/0 conn(0x7f59240c90f0 msgr2=0x7f59241556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59227fc640 1 --2- 172.21.15.1:0/551935979 >> v2:172.21.15.98:3300/0 conn(0x7f59240c90f0 0x7f59241556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59227fc640 1 -- 172.21.15.1:0/551935979 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59240c9550 con 0x7f59240c8d10 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59227fc640 1 -- 172.21.15.1:0/551935979 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f5914001670 con 0x7f59240c8d10 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59227fc640 1 -- 172.21.15.1:0/551935979 >> v1:172.21.15.1:6789/0 conn(0x7f59240c8d10 legacy=0x7f59241590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59227fc640 1 --2- 172.21.15.1:0/551935979 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5924155bf0 0x7f59100039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59227fc640 1 --2- 172.21.15.1:0/551935979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5910003f30 0x7f5910004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59227fc640 1 --2- 172.21.15.1:0/551935979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5910004d40 0x7f59100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59227fc640 1 -- 172.21.15.1:0/551935979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59240c33e0 con 0x7f5910004d40 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59227fc640 1 -- 172.21.15.1:0/551935979 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59240c36c0 con 0x7f5910003f30 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59227fc640 1 -- 172.21.15.1:0/551935979 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591000b9c0 con 0x7f5924155bf0 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.698+0000 7f59237fe640 1 --2- 172.21.15.1:0/551935979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5910004d40 0x7f59100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.699+0000 7f591bfff640 1 --2- 172.21.15.1:0/551935979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5910003f30 0x7f5910004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.699+0000 7f5922ffd640 1 --2- 172.21.15.1:0/551935979 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5924155bf0 0x7f59100039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.699+0000 7f59237fe640 1 -- 172.21.15.1:0/551935979 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5924155bf0 msgr2=0x7f59100039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.699+0000 7f59237fe640 1 --2- 172.21.15.1:0/551935979 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5924155bf0 0x7f59100039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.699+0000 7f59237fe640 1 -- 172.21.15.1:0/551935979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5910003f30 msgr2=0x7f5910004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.699+0000 7f59237fe640 1 --2- 172.21.15.1:0/551935979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5910003f30 0x7f5910004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.699+0000 7f59237fe640 1 -- 172.21.15.1:0/551935979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5910003520 con 0x7f5910004d40 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.700+0000 7f59237fe640 1 --2- 172.21.15.1:0/551935979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5910004d40 0x7f59100051c0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f591c009fc0 tx=0x7f5910005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a64bade6cbdfefe5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.700+0000 7f59227fc640 1 -- 172.21.15.1:0/551935979 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c039070 con 0x7f5910004d40 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.700+0000 7f59227fc640 1 -- 172.21.15.1:0/551935979 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591c02cc00 con 0x7f5910004d40 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.700+0000 7f59227fc640 1 -- 172.21.15.1:0/551935979 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c009c10 con 0x7f5910004d40 2024-03-06T07:24:53.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.700+0000 7f59293e2640 1 -- 172.21.15.1:0/551935979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5910004d40 msgr2=0x7f59100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.700+0000 7f59293e2640 1 --2- 172.21.15.1:0/551935979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5910004d40 0x7f59100051c0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f591c009fc0 tx=0x7f5910005830 comp rx=0 tx=0).stop 2024-03-06T07:24:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.700+0000 7f59237fe640 1 -- 172.21.15.1:0/551935979 reap_dead start 2024-03-06T07:24:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.700+0000 7f59293e2640 1 -- 172.21.15.1:0/551935979 shutdown_connections 2024-03-06T07:24:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.700+0000 7f59293e2640 1 -- 172.21.15.1:0/551935979 >> 172.21.15.1:0/551935979 conn(0x7f592406b3e0 msgr2=0x7f592406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.700+0000 7f59293e2640 1 -- 172.21.15.1:0/551935979 shutdown_connections 2024-03-06T07:24:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.700+0000 7f59293e2640 1 -- 172.21.15.1:0/551935979 wait complete. 2024-03-06T07:24:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.701+0000 7f59293e2640 1 Processor -- start 2024-03-06T07:24:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.701+0000 7f59293e2640 1 -- start start 2024-03-06T07:24:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.702+0000 7f59293e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5910004d40 0x7f59240a9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.702+0000 7f59293e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f592414d290 0x7f59240a9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.702+0000 7f59293e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59240ab720 0x7f59240abbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.702+0000 7f59293e2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59240c7560 con 0x7f5910004d40 2024-03-06T07:24:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.702+0000 7f59293e2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5924156060 con 0x7f59240ab720 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.702+0000 7f59293e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59241529e0 con 0x7f592414d290 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.702+0000 7f5922ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5910004d40 0x7f59240a9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.702+0000 7f5922ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5910004d40 0x7f59240a9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33646/0 (socket says 172.21.15.1:33646) 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.702+0000 7f5922ffd640 1 -- 172.21.15.1:0/2213712877 learned_addr learned my addr 172.21.15.1:0/2213712877 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.702+0000 7f5922ffd640 1 -- 172.21.15.1:0/2213712877 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f592414d290 msgr2=0x7f59240a9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.702+0000 7f5922ffd640 1 --2- 172.21.15.1:0/2213712877 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f592414d290 0x7f59240a9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.702+0000 7f5922ffd640 1 -- 172.21.15.1:0/2213712877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59240ab720 msgr2=0x7f59240abbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.702+0000 7f5922ffd640 1 --2- 172.21.15.1:0/2213712877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59240ab720 0x7f59240abbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.702+0000 7f5922ffd640 1 -- 172.21.15.1:0/2213712877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592414d770 con 0x7f5910004d40 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.703+0000 7f5922ffd640 1 --2- 172.21.15.1:0/2213712877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5910004d40 0x7f59240a9500 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f5914007fd0 tx=0x7f59140077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.703+0000 7f591b7fe640 1 -- 172.21.15.1:0/2213712877 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591400f940 con 0x7f5910004d40 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.703+0000 7f591b7fe640 1 -- 172.21.15.1:0/2213712877 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5914018070 con 0x7f5910004d40 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.703+0000 7f59293e2640 1 -- 172.21.15.1:0/2213712877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59240ad5b0 con 0x7f5910004d40 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.703+0000 7f591b7fe640 1 -- 172.21.15.1:0/2213712877 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591400fce0 con 0x7f5910004d40 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.703+0000 7f59293e2640 1 -- 172.21.15.1:0/2213712877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59240ac110 con 0x7f5910004d40 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.704+0000 7f59293e2640 1 -- 172.21.15.1:0/2213712877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5924067fd0 con 0x7f5910004d40 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.704+0000 7f591b7fe640 1 -- 172.21.15.1:0/2213712877 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f591401d460 con 0x7f5910004d40 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.705+0000 7f591b7fe640 1 --2- 172.21.15.1:0/2213712877 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f590406b500 0x7f590406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.705+0000 7f591b7fe640 1 -- 172.21.15.1:0/2213712877 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f591408a4d0 con 0x7f5910004d40 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.705+0000 7f591bfff640 1 --2- 172.21.15.1:0/2213712877 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f590406b500 0x7f590406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.705+0000 7f591bfff640 1 --2- 172.21.15.1:0/2213712877 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f590406b500 0x7f590406d9c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f590c006b10 tx=0x7f590c006aa0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.709+0000 7f591b7fe640 1 -- 172.21.15.1:0/2213712877 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5914059250 con 0x7f5910004d40 2024-03-06T07:24:53.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.890+0000 7f59293e2640 1 -- 172.21.15.1:0/2213712877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59240ad780 con 0x7f5910004d40 2024-03-06T07:24:53.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.891+0000 7f591b7fe640 1 -- 172.21.15.1:0/2213712877 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16793 (secure 0 0 0) 0x7f5914058b50 con 0x7f5910004d40 2024-03-06T07:24:53.893 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:53.894 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":120,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:24:52.597984+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:50.865151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:51.564469+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:53.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.893+0000 7f59293e2640 1 -- 172.21.15.1:0/2213712877 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f590406b500 msgr2=0x7f590406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.893+0000 7f59293e2640 1 --2- 172.21.15.1:0/2213712877 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f590406b500 0x7f590406d9c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f590c006b10 tx=0x7f590c006aa0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.893+0000 7f59293e2640 1 -- 172.21.15.1:0/2213712877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5910004d40 msgr2=0x7f59240a9500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.893+0000 7f59293e2640 1 --2- 172.21.15.1:0/2213712877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5910004d40 0x7f59240a9500 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f5914007fd0 tx=0x7f59140077a0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.893+0000 7f59293e2640 1 -- 172.21.15.1:0/2213712877 shutdown_connections 2024-03-06T07:24:53.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.893+0000 7f59293e2640 1 --2- 172.21.15.1:0/2213712877 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f590406b500 0x7f590406d9c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.893+0000 7f59293e2640 1 --2- 172.21.15.1:0/2213712877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59240ab720 0x7f59240abbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.893+0000 7f59293e2640 1 --2- 172.21.15.1:0/2213712877 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f592414d290 0x7f59240a9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.893+0000 7f59293e2640 1 --2- 172.21.15.1:0/2213712877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5910004d40 0x7f59240a9500 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.893+0000 7f59293e2640 1 -- 172.21.15.1:0/2213712877 >> 172.21.15.1:0/2213712877 conn(0x7f592406b3e0 msgr2=0x7f5924152ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.893+0000 7f59293e2640 1 -- 172.21.15.1:0/2213712877 shutdown_connections 2024-03-06T07:24:53.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:53.893+0000 7f59293e2640 1 -- 172.21.15.1:0/2213712877 wait complete. 2024-03-06T07:24:53.906 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:24:54.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.009+0000 7f3f898d4640 1 Processor -- start 2024-03-06T07:24:54.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.009+0000 7f3f898d4640 1 -- start start 2024-03-06T07:24:54.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.009+0000 7f3f898d4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3f840c2b30 0x7f3f840724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.010+0000 7f3f898d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f84152b70 0x7f3f84072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.010+0000 7f3f898d4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f84151dd0 con 0x7f3f840a5fa0 2024-03-06T07:24:54.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.010+0000 7f3f898d4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f841526e0 con 0x7f3f840c2b30 2024-03-06T07:24:54.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.010+0000 7f3f898d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f84156190 con 0x7f3f84152b70 2024-03-06T07:24:54.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.010+0000 7f3f7bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f84152b70 0x7f3f84072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.010+0000 7f3f82ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3f840c2b30 0x7f3f840724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.010+0000 7f3f7bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f84152b70 0x7f3f84072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:38516/0 (socket says 172.21.15.1:38516) 2024-03-06T07:24:54.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.010+0000 7f3f82ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3f840c2b30 0x7f3f840724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:42574/0 (socket says 172.21.15.1:42574) 2024-03-06T07:24:54.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.010+0000 7f3f7bfff640 1 -- 172.21.15.1:0/2075227424 learned_addr learned my addr 172.21.15.1:0/2075227424 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:54.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.010+0000 7f3f7bfff640 1 -- 172.21.15.1:0/2075227424 >> v2:172.21.15.98:3300/0 conn(0x7f3f840c2b30 msgr2=0x7f3f840724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.010+0000 7f3f7bfff640 1 --2- 172.21.15.1:0/2075227424 >> v2:172.21.15.98:3300/0 conn(0x7f3f840c2b30 0x7f3f840724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.010+0000 7f3f7bfff640 1 -- 172.21.15.1:0/2075227424 >> v1:172.21.15.98:6789/0 conn(0x7f3f840a5fa0 legacy=0x7f3f840c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.010+0000 7f3f7bfff640 1 -- 172.21.15.1:0/2075227424 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f84152090 con 0x7f3f84152b70 2024-03-06T07:24:54.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.011+0000 7f3f7bfff640 1 --2- 172.21.15.1:0/2075227424 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f84152b70 0x7f3f84072a00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c009870 tx=0x7f3f6c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2068e6200c1589cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.011+0000 7f3f827fc640 1 -- 172.21.15.1:0/2075227424 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c034070 con 0x7f3f84152b70 2024-03-06T07:24:54.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.011+0000 7f3f827fc640 1 -- 172.21.15.1:0/2075227424 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6c030840 con 0x7f3f84152b70 2024-03-06T07:24:54.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.011+0000 7f3f827fc640 1 -- 172.21.15.1:0/2075227424 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c030c10 con 0x7f3f84152b70 2024-03-06T07:24:54.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.012+0000 7f3f898d4640 1 -- 172.21.15.1:0/2075227424 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f84152b70 msgr2=0x7f3f84072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.012+0000 7f3f898d4640 1 --2- 172.21.15.1:0/2075227424 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f84152b70 0x7f3f84072a00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c009870 tx=0x7f3f6c02b980 comp rx=0 tx=0).stop 2024-03-06T07:24:54.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.012+0000 7f3f898d4640 1 -- 172.21.15.1:0/2075227424 shutdown_connections 2024-03-06T07:24:54.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.012+0000 7f3f898d4640 1 --2- 172.21.15.1:0/2075227424 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f84152b70 0x7f3f84072a00 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.012+0000 7f3f898d4640 1 --2- 172.21.15.1:0/2075227424 >> v2:172.21.15.98:3300/0 conn(0x7f3f840c2b30 0x7f3f840724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.012+0000 7f3f898d4640 1 -- 172.21.15.1:0/2075227424 >> 172.21.15.1:0/2075227424 conn(0x7f3f840c6860 msgr2=0x7f3f840c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:54.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.012+0000 7f3f898d4640 1 -- 172.21.15.1:0/2075227424 shutdown_connections 2024-03-06T07:24:54.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.012+0000 7f3f898d4640 1 -- 172.21.15.1:0/2075227424 wait complete. 2024-03-06T07:24:54.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.013+0000 7f3f898d4640 1 Processor -- start 2024-03-06T07:24:54.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.013+0000 7f3f898d4640 1 -- start start 2024-03-06T07:24:54.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.014+0000 7f3f898d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f840a5fa0 0x7f3f841e8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.014+0000 7f3f898d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f840c2b30 0x7f3f841ded50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.014+0000 7f3f898d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f84152b70 0x7f3f8414bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.014+0000 7f3f898d4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f84155f00 con 0x7f3f84152b70 2024-03-06T07:24:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.014+0000 7f3f898d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f84155d80 con 0x7f3f840c2b30 2024-03-06T07:24:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.014+0000 7f3f898d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f840c3a00 con 0x7f3f840a5fa0 2024-03-06T07:24:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.014+0000 7f3f837fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f84152b70 0x7f3f8414bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.014+0000 7f3f837fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f84152b70 0x7f3f8414bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33662/0 (socket says 172.21.15.1:33662) 2024-03-06T07:24:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.014+0000 7f3f837fe640 1 -- 172.21.15.1:0/11615741 learned_addr learned my addr 172.21.15.1:0/11615741 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.014+0000 7f3f82ffd640 1 --2- 172.21.15.1:0/11615741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f840a5fa0 0x7f3f841e8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.014+0000 7f3f7bfff640 1 --2- 172.21.15.1:0/11615741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f840c2b30 0x7f3f841ded50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.014+0000 7f3f837fe640 1 -- 172.21.15.1:0/11615741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f840a5fa0 msgr2=0x7f3f841e8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.014+0000 7f3f837fe640 1 --2- 172.21.15.1:0/11615741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f840a5fa0 0x7f3f841e8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.014+0000 7f3f837fe640 1 -- 172.21.15.1:0/11615741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f840c2b30 msgr2=0x7f3f841ded50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.014+0000 7f3f837fe640 1 --2- 172.21.15.1:0/11615741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f840c2b30 0x7f3f841ded50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.014+0000 7f3f837fe640 1 -- 172.21.15.1:0/11615741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f840ac690 con 0x7f3f84152b70 2024-03-06T07:24:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.014+0000 7f3f837fe640 1 --2- 172.21.15.1:0/11615741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f84152b70 0x7f3f8414bb60 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c00c7a0 tx=0x7f3f7c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.015+0000 7f3f80ff9640 1 -- 172.21.15.1:0/11615741 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c003b70 con 0x7f3f84152b70 2024-03-06T07:24:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.015+0000 7f3f80ff9640 1 -- 172.21.15.1:0/11615741 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7c003d30 con 0x7f3f84152b70 2024-03-06T07:24:54.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.015+0000 7f3f898d4640 1 -- 172.21.15.1:0/11615741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f840ac970 con 0x7f3f84152b70 2024-03-06T07:24:54.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.015+0000 7f3f80ff9640 1 -- 172.21.15.1:0/11615741 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c00aa90 con 0x7f3f84152b70 2024-03-06T07:24:54.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.015+0000 7f3f898d4640 1 -- 172.21.15.1:0/11615741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f841dd690 con 0x7f3f84152b70 2024-03-06T07:24:54.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.016+0000 7f3f898d4640 1 -- 172.21.15.1:0/11615741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f84067fd0 con 0x7f3f84152b70 2024-03-06T07:24:54.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.016+0000 7f3f80ff9640 1 -- 172.21.15.1:0/11615741 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f7c00ac30 con 0x7f3f84152b70 2024-03-06T07:24:54.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.016+0000 7f3f80ff9640 1 --2- 172.21.15.1:0/11615741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3f6406b4d0 0x7f3f6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.017+0000 7f3f80ff9640 1 -- 172.21.15.1:0/11615741 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3f7c08a100 con 0x7f3f84152b70 2024-03-06T07:24:54.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.017+0000 7f3f82ffd640 1 --2- 172.21.15.1:0/11615741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3f6406b4d0 0x7f3f6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.017+0000 7f3f82ffd640 1 --2- 172.21.15.1:0/11615741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3f6406b4d0 0x7f3f6406d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3f74009b40 tx=0x7f3f74009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.021+0000 7f3f80ff9640 1 -- 172.21.15.1:0/11615741 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f7c058830 con 0x7f3f84152b70 2024-03-06T07:24:54.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:54.243+0000 7f3f898d4640 1 -- 172.21.15.1:0/11615741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3f84071790 con 0x7f3f84152b70 2024-03-06T07:24:55.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.250+0000 7f3f80ff9640 1 -- 172.21.15.1:0/11615741 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v105) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3f7c05c8d0 con 0x7f3f84152b70 2024-03-06T07:24:55.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.253+0000 7f3f898d4640 1 -- 172.21.15.1:0/11615741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3f6406b4d0 msgr2=0x7f3f6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.253+0000 7f3f898d4640 1 --2- 172.21.15.1:0/11615741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3f6406b4d0 0x7f3f6406d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3f74009b40 tx=0x7f3f74009190 comp rx=0 tx=0).stop 2024-03-06T07:24:55.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.253+0000 7f3f898d4640 1 -- 172.21.15.1:0/11615741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f84152b70 msgr2=0x7f3f8414bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.253+0000 7f3f898d4640 1 --2- 172.21.15.1:0/11615741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f84152b70 0x7f3f8414bb60 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c00c7a0 tx=0x7f3f7c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:24:55.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.253+0000 7f3f898d4640 1 -- 172.21.15.1:0/11615741 shutdown_connections 2024-03-06T07:24:55.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.253+0000 7f3f898d4640 1 --2- 172.21.15.1:0/11615741 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3f6406b4d0 0x7f3f6406d990 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.253+0000 7f3f898d4640 1 --2- 172.21.15.1:0/11615741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f84152b70 0x7f3f8414bb60 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.253+0000 7f3f898d4640 1 --2- 172.21.15.1:0/11615741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f840c2b30 0x7f3f841ded50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.253+0000 7f3f898d4640 1 --2- 172.21.15.1:0/11615741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f840a5fa0 0x7f3f841e8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.253+0000 7f3f898d4640 1 -- 172.21.15.1:0/11615741 >> 172.21.15.1:0/11615741 conn(0x7f3f840c6860 msgr2=0x7f3f840c4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.253+0000 7f3f898d4640 1 -- 172.21.15.1:0/11615741 shutdown_connections 2024-03-06T07:24:55.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.253+0000 7f3f898d4640 1 -- 172.21.15.1:0/11615741 wait complete. 2024-03-06T07:24:55.266 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:24:55.266 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:24:55.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.385+0000 7f2b49f20640 1 Processor -- start 2024-03-06T07:24:55.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.385+0000 7f2b49f20640 1 -- start start 2024-03-06T07:24:55.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.386+0000 7f2b49f20640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2b440c4dd0 0x7f2b440c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.386+0000 7f2b49f20640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b440c79c0 0x7f2b44152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.386+0000 7f2b49f20640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2b440c5d80 0x7f2b441533e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.386+0000 7f2b49f20640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b44156490 con 0x7f2b440c4dd0 2024-03-06T07:24:55.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.386+0000 7f2b49f20640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b44157010 con 0x7f2b440c5d80 2024-03-06T07:24:55.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.386+0000 7f2b49f20640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b441566e0 con 0x7f2b440c79c0 2024-03-06T07:24:55.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.386+0000 7f2b437fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2b440c4dd0 0x7f2b440c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.386+0000 7f2b437fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2b440c4dd0 0x7f2b440c5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36822/0 (socket says 172.21.15.1:36822) 2024-03-06T07:24:55.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.386+0000 7f2b437fe640 1 -- 172.21.15.1:0/1772990509 learned_addr learned my addr 172.21.15.1:0/1772990509 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:55.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.387+0000 7f2b43fff640 1 --2- 172.21.15.1:0/1772990509 >> v2:172.21.15.98:3300/0 conn(0x7f2b440c5d80 0x7f2b441533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.387+0000 7f2b42ffd640 1 --2- 172.21.15.1:0/1772990509 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b440c79c0 0x7f2b44152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.387+0000 7f2b437fe640 1 -- 172.21.15.1:0/1772990509 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b440c79c0 msgr2=0x7f2b44152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.387+0000 7f2b437fe640 1 --2- 172.21.15.1:0/1772990509 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b440c79c0 0x7f2b44152ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.387+0000 7f2b437fe640 1 -- 172.21.15.1:0/1772990509 >> v2:172.21.15.98:3300/0 conn(0x7f2b440c5d80 msgr2=0x7f2b441533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.387+0000 7f2b437fe640 1 --2- 172.21.15.1:0/1772990509 >> v2:172.21.15.98:3300/0 conn(0x7f2b440c5d80 0x7f2b441533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.387+0000 7f2b437fe640 1 -- 172.21.15.1:0/1772990509 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b441569c0 con 0x7f2b440c4dd0 2024-03-06T07:24:55.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.388+0000 7f2b437fe640 1 --2- 172.21.15.1:0/1772990509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b440c4dd0 0x7f2b440c5840 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f2b440c6730 tx=0x7f2b3402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88a68e8abb55f6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.388+0000 7f2b427fc640 1 -- 172.21.15.1:0/1772990509 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34038070 con 0x7f2b440c4dd0 2024-03-06T07:24:55.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.388+0000 7f2b427fc640 1 -- 172.21.15.1:0/1772990509 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3402f050 con 0x7f2b440c4dd0 2024-03-06T07:24:55.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.388+0000 7f2b427fc640 1 -- 172.21.15.1:0/1772990509 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34031970 con 0x7f2b440c4dd0 2024-03-06T07:24:55.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.388+0000 7f2b49f20640 1 -- 172.21.15.1:0/1772990509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b440c4dd0 msgr2=0x7f2b440c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.388+0000 7f2b49f20640 1 --2- 172.21.15.1:0/1772990509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b440c4dd0 0x7f2b440c5840 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f2b440c6730 tx=0x7f2b3402b7f0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.388+0000 7f2b49f20640 1 -- 172.21.15.1:0/1772990509 shutdown_connections 2024-03-06T07:24:55.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.388+0000 7f2b49f20640 1 --2- 172.21.15.1:0/1772990509 >> v2:172.21.15.98:3300/0 conn(0x7f2b440c5d80 0x7f2b441533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.388+0000 7f2b49f20640 1 --2- 172.21.15.1:0/1772990509 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b440c79c0 0x7f2b44152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.388+0000 7f2b49f20640 1 --2- 172.21.15.1:0/1772990509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b440c4dd0 0x7f2b440c5840 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.388+0000 7f2b49f20640 1 -- 172.21.15.1:0/1772990509 >> 172.21.15.1:0/1772990509 conn(0x7f2b4406b410 msgr2=0x7f2b4406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.388+0000 7f2b49f20640 1 -- 172.21.15.1:0/1772990509 shutdown_connections 2024-03-06T07:24:55.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.388+0000 7f2b49f20640 1 -- 172.21.15.1:0/1772990509 wait complete. 2024-03-06T07:24:55.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.390+0000 7f2b49f20640 1 Processor -- start 2024-03-06T07:24:55.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.390+0000 7f2b49f20640 1 -- start start 2024-03-06T07:24:55.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.391+0000 7f2b49f20640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b440c4dd0 0x7f2b44151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.391+0000 7f2b49f20640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b440c5d80 0x7f2b441ebdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.391+0000 7f2b49f20640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b440adb40 0x7f2b441ec2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.391+0000 7f2b49f20640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b44071770 con 0x7f2b440c4dd0 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.391+0000 7f2b49f20640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b440715f0 con 0x7f2b440adb40 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.391+0000 7f2b49f20640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b44153920 con 0x7f2b440c5d80 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.391+0000 7f2b437fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b440c4dd0 0x7f2b44151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.391+0000 7f2b437fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b440c4dd0 0x7f2b44151e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36826/0 (socket says 172.21.15.1:36826) 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.391+0000 7f2b437fe640 1 -- 172.21.15.1:0/3247948123 learned_addr learned my addr 172.21.15.1:0/3247948123 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.392+0000 7f2b42ffd640 1 --2- 172.21.15.1:0/3247948123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b440c5d80 0x7f2b441ebdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.392+0000 7f2b43fff640 1 --2- 172.21.15.1:0/3247948123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b440adb40 0x7f2b441ec2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.392+0000 7f2b437fe640 1 -- 172.21.15.1:0/3247948123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b440c5d80 msgr2=0x7f2b441ebdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.392+0000 7f2b437fe640 1 --2- 172.21.15.1:0/3247948123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b440c5d80 0x7f2b441ebdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.392+0000 7f2b437fe640 1 -- 172.21.15.1:0/3247948123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b440adb40 msgr2=0x7f2b441ec2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.392+0000 7f2b437fe640 1 --2- 172.21.15.1:0/3247948123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b440adb40 0x7f2b441ec2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.392+0000 7f2b437fe640 1 -- 172.21.15.1:0/3247948123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b44152760 con 0x7f2b440c4dd0 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.392+0000 7f2b437fe640 1 --2- 172.21.15.1:0/3247948123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b440c4dd0 0x7f2b44151e80 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009960 tx=0x7f2b340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.392+0000 7f2b3bfff640 1 -- 172.21.15.1:0/3247948123 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34042020 con 0x7f2b440c4dd0 2024-03-06T07:24:55.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.392+0000 7f2b3bfff640 1 -- 172.21.15.1:0/3247948123 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3403d890 con 0x7f2b440c4dd0 2024-03-06T07:24:55.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.392+0000 7f2b49f20640 1 -- 172.21.15.1:0/3247948123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b441dd040 con 0x7f2b440c4dd0 2024-03-06T07:24:55.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.392+0000 7f2b3bfff640 1 -- 172.21.15.1:0/3247948123 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34038040 con 0x7f2b440c4dd0 2024-03-06T07:24:55.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.392+0000 7f2b49f20640 1 -- 172.21.15.1:0/3247948123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b441dd8c0 con 0x7f2b440c4dd0 2024-03-06T07:24:55.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.394+0000 7f2b49f20640 1 -- 172.21.15.1:0/3247948123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b44067fd0 con 0x7f2b440c4dd0 2024-03-06T07:24:55.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.394+0000 7f2b3bfff640 1 -- 172.21.15.1:0/3247948123 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b34045020 con 0x7f2b440c4dd0 2024-03-06T07:24:55.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.394+0000 7f2b3bfff640 1 --2- 172.21.15.1:0/3247948123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b2406b500 0x7f2b2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.394+0000 7f2b3bfff640 1 -- 172.21.15.1:0/3247948123 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2b340abbc0 con 0x7f2b440c4dd0 2024-03-06T07:24:55.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.394+0000 7f2b42ffd640 1 --2- 172.21.15.1:0/3247948123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b2406b500 0x7f2b2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.395+0000 7f2b42ffd640 1 --2- 172.21.15.1:0/3247948123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b2406b500 0x7f2b2406d9c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f2b440c6580 tx=0x7f2b2c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.399+0000 7f2b3bfff640 1 -- 172.21.15.1:0/3247948123 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b3404c030 con 0x7f2b440c4dd0 2024-03-06T07:24:55.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:55.620+0000 7f2b49f20640 1 -- 172.21.15.1:0/3247948123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2b441dd210 con 0x7f2b440c4dd0 2024-03-06T07:24:56.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.254+0000 7f2b3bfff640 1 -- 172.21.15.1:0/3247948123 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v106) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2b3407a2f0 con 0x7f2b440c4dd0 2024-03-06T07:24:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.257+0000 7f2b49f20640 1 -- 172.21.15.1:0/3247948123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b2406b500 msgr2=0x7f2b2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.257+0000 7f2b49f20640 1 --2- 172.21.15.1:0/3247948123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b2406b500 0x7f2b2406d9c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f2b440c6580 tx=0x7f2b2c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.257+0000 7f2b49f20640 1 -- 172.21.15.1:0/3247948123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b440c4dd0 msgr2=0x7f2b44151e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.257+0000 7f2b49f20640 1 --2- 172.21.15.1:0/3247948123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b440c4dd0 0x7f2b44151e80 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009960 tx=0x7f2b340046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.257+0000 7f2b49f20640 1 -- 172.21.15.1:0/3247948123 shutdown_connections 2024-03-06T07:24:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.257+0000 7f2b49f20640 1 --2- 172.21.15.1:0/3247948123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b2406b500 0x7f2b2406d9c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.257+0000 7f2b49f20640 1 --2- 172.21.15.1:0/3247948123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b440adb40 0x7f2b441ec2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.257+0000 7f2b49f20640 1 --2- 172.21.15.1:0/3247948123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b440c5d80 0x7f2b441ebdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.257+0000 7f2b49f20640 1 --2- 172.21.15.1:0/3247948123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b440c4dd0 0x7f2b44151e80 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.257+0000 7f2b49f20640 1 -- 172.21.15.1:0/3247948123 >> 172.21.15.1:0/3247948123 conn(0x7f2b4406b410 msgr2=0x7f2b4406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.257+0000 7f2b49f20640 1 -- 172.21.15.1:0/3247948123 shutdown_connections 2024-03-06T07:24:56.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.257+0000 7f2b49f20640 1 -- 172.21.15.1:0/3247948123 wait complete. 2024-03-06T07:24:56.269 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:24:56.270 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:56.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.369+0000 7fcf56615640 1 Processor -- start 2024-03-06T07:24:56.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.369+0000 7fcf56615640 1 -- start start 2024-03-06T07:24:56.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.370+0000 7fcf56615640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcf50152650 0x7fcf50155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.370+0000 7fcf56615640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcf500c6b10 0x7fcf50158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.370+0000 7fcf56615640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf500c2fc0 con 0x7fcf500bf9d0 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.370+0000 7fcf56615640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf50155800 con 0x7fcf500c6b10 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.370+0000 7fcf56615640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf500c6690 con 0x7fcf50152650 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.370+0000 7fcf4f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcf500c6b10 0x7fcf50158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.370+0000 7fcf54bb2640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcf500bf9d0 0x7fcf500c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41604/0 (socket says 172.21.15.1:41604) 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.370+0000 7fcf54bb2640 1 -- 172.21.15.1:0/2338886486 learned_addr learned my addr 172.21.15.1:0/2338886486 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.371+0000 7fcf4ffff640 1 --2- 172.21.15.1:0/2338886486 >> v2:172.21.15.98:3300/0 conn(0x7fcf50152650 0x7fcf50155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.371+0000 7fcf4f7fe640 1 -- 172.21.15.1:0/2338886486 >> v2:172.21.15.98:3300/0 conn(0x7fcf50152650 msgr2=0x7fcf50155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.371+0000 7fcf4f7fe640 1 --2- 172.21.15.1:0/2338886486 >> v2:172.21.15.98:3300/0 conn(0x7fcf50152650 0x7fcf50155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.371+0000 7fcf4f7fe640 1 -- 172.21.15.1:0/2338886486 >> v1:172.21.15.1:6789/0 conn(0x7fcf500bf9d0 legacy=0x7fcf500c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.371+0000 7fcf4effd640 1 -- 172.21.15.1:0/2338886486 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604476907 0 0) 0x7fcf500c2fc0 con 0x7fcf500bf9d0 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.371+0000 7fcf4f7fe640 1 -- 172.21.15.1:0/2338886486 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf500c7280 con 0x7fcf500c6b10 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.371+0000 7fcf4ffff640 1 --2- 172.21.15.1:0/2338886486 >> v2:172.21.15.98:3300/0 conn(0x7fcf50152650 0x7fcf50155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:56.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.372+0000 7fcf4f7fe640 1 --2- 172.21.15.1:0/2338886486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf500c6b10 0x7fcf50158690 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fcf500cac90 tx=0x7fcf4402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881e00e13321bb34 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.372+0000 7fcf4effd640 1 -- 172.21.15.1:0/2338886486 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf440315a0 con 0x7fcf500c6b10 2024-03-06T07:24:56.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.372+0000 7fcf4effd640 1 -- 172.21.15.1:0/2338886486 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf44037070 con 0x7fcf500c6b10 2024-03-06T07:24:56.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.372+0000 7fcf4effd640 1 -- 172.21.15.1:0/2338886486 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf44031960 con 0x7fcf500c6b10 2024-03-06T07:24:56.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.372+0000 7fcf56615640 1 -- 172.21.15.1:0/2338886486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf500c6b10 msgr2=0x7fcf50158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.372+0000 7fcf56615640 1 --2- 172.21.15.1:0/2338886486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf500c6b10 0x7fcf50158690 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fcf500cac90 tx=0x7fcf4402b840 comp rx=0 tx=0).stop 2024-03-06T07:24:56.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.372+0000 7fcf56615640 1 -- 172.21.15.1:0/2338886486 shutdown_connections 2024-03-06T07:24:56.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.372+0000 7fcf56615640 1 --2- 172.21.15.1:0/2338886486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf500c6b10 0x7fcf50158690 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.372+0000 7fcf56615640 1 --2- 172.21.15.1:0/2338886486 >> v2:172.21.15.98:3300/0 conn(0x7fcf50152650 0x7fcf50155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.372+0000 7fcf56615640 1 -- 172.21.15.1:0/2338886486 >> 172.21.15.1:0/2338886486 conn(0x7fcf50066ee0 msgr2=0x7fcf500672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.372+0000 7fcf56615640 1 -- 172.21.15.1:0/2338886486 shutdown_connections 2024-03-06T07:24:56.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.372+0000 7fcf56615640 1 -- 172.21.15.1:0/2338886486 wait complete. 2024-03-06T07:24:56.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.374+0000 7fcf56615640 1 Processor -- start 2024-03-06T07:24:56.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.374+0000 7fcf56615640 1 -- start start 2024-03-06T07:24:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.375+0000 7fcf56615640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf500bf9d0 0x7fcf5014da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.375+0000 7fcf56615640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf50152650 0x7fcf501eb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.375+0000 7fcf56615640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf50145d40 0x7fcf501eb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.375+0000 7fcf56615640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf50155490 con 0x7fcf50145d40 2024-03-06T07:24:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.375+0000 7fcf56615640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf501551b0 con 0x7fcf500bf9d0 2024-03-06T07:24:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.375+0000 7fcf56615640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf50152a30 con 0x7fcf50152650 2024-03-06T07:24:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.375+0000 7fcf54bb2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf50145d40 0x7fcf501eb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.375+0000 7fcf54bb2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf50145d40 0x7fcf501eb910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36832/0 (socket says 172.21.15.1:36832) 2024-03-06T07:24:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.375+0000 7fcf54bb2640 1 -- 172.21.15.1:0/1593333903 learned_addr learned my addr 172.21.15.1:0/1593333903 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.376+0000 7fcf4ffff640 1 --2- 172.21.15.1:0/1593333903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf500bf9d0 0x7fcf5014da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.376+0000 7fcf54bb2640 1 -- 172.21.15.1:0/1593333903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf50152650 msgr2=0x7fcf501eb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.376+0000 7fcf54bb2640 1 --2- 172.21.15.1:0/1593333903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf50152650 0x7fcf501eb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.376+0000 7fcf54bb2640 1 -- 172.21.15.1:0/1593333903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf500bf9d0 msgr2=0x7fcf5014da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.376+0000 7fcf54bb2640 1 --2- 172.21.15.1:0/1593333903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf500bf9d0 0x7fcf5014da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.376+0000 7fcf54bb2640 1 -- 172.21.15.1:0/1593333903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf500ca720 con 0x7fcf50145d40 2024-03-06T07:24:56.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.376+0000 7fcf54bb2640 1 --2- 172.21.15.1:0/1593333903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf50145d40 0x7fcf501eb910 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fcf4800c700 tx=0x7fcf4800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.376+0000 7fcf4cff9640 1 -- 172.21.15.1:0/1593333903 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf48018070 con 0x7fcf50145d40 2024-03-06T07:24:56.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.376+0000 7fcf4cff9640 1 -- 172.21.15.1:0/1593333903 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf4800ce10 con 0x7fcf50145d40 2024-03-06T07:24:56.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.376+0000 7fcf56615640 1 -- 172.21.15.1:0/1593333903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf500ca440 con 0x7fcf50145d40 2024-03-06T07:24:56.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.376+0000 7fcf4cff9640 1 -- 172.21.15.1:0/1593333903 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4800a860 con 0x7fcf50145d40 2024-03-06T07:24:56.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.376+0000 7fcf56615640 1 -- 172.21.15.1:0/1593333903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf501eaeb0 con 0x7fcf50145d40 2024-03-06T07:24:56.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.377+0000 7fcf56615640 1 -- 172.21.15.1:0/1593333903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf500cb510 con 0x7fcf50145d40 2024-03-06T07:24:56.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.378+0000 7fcf4cff9640 1 -- 172.21.15.1:0/1593333903 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf4801b070 con 0x7fcf50145d40 2024-03-06T07:24:56.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.378+0000 7fcf4cff9640 1 --2- 172.21.15.1:0/1593333903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf3c06b500 0x7fcf3c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.378+0000 7fcf4cff9640 1 -- 172.21.15.1:0/1593333903 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcf480898b0 con 0x7fcf50145d40 2024-03-06T07:24:56.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.378+0000 7fcf4ffff640 1 --2- 172.21.15.1:0/1593333903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf3c06b500 0x7fcf3c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.379+0000 7fcf4ffff640 1 --2- 172.21.15.1:0/1593333903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf3c06b500 0x7fcf3c06d9c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fcf4000d8b0 tx=0x7fcf40008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.384+0000 7fcf4cff9640 1 -- 172.21.15.1:0/1593333903 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf48058630 con 0x7fcf50145d40 2024-03-06T07:24:56.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.619+0000 7fcf56615640 1 -- 172.21.15.1:0/1593333903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf500b06f0 con 0x7fcf50145d40 2024-03-06T07:24:56.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.620+0000 7fcf4cff9640 1 -- 172.21.15.1:0/1593333903 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4976 (secure 0 0 0) 0x7fcf48057f30 con 0x7fcf50145d40 2024-03-06T07:24:56.622 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:56.622 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":106,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6446,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4151599120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4151599120},{"type":"v1","addr":"172.21.15.1:6835","nonce":4151599120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/155600632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":155600632},{"type":"v1","addr":"172.21.15.98:6837","nonce":155600632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7084,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1420541659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1420541659},{"type":"v1","addr":"172.21.15.98:6836","nonce":1420541659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.598303+0000","modified":"2024-03-06T07:24:55.621912+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7863},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7863":{"gid":7863,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1515309674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1515309674},{"type":"v1","addr":"172.21.15.1:6834","nonce":1515309674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7863,"qdb_cluster":[7863]},"id":9}]} 2024-03-06T07:24:56.622 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 106 2024-03-06T07:24:56.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.621+0000 7fcf56615640 1 -- 172.21.15.1:0/1593333903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf3c06b500 msgr2=0x7fcf3c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.621+0000 7fcf56615640 1 --2- 172.21.15.1:0/1593333903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf3c06b500 0x7fcf3c06d9c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fcf4000d8b0 tx=0x7fcf40008040 comp rx=0 tx=0).stop 2024-03-06T07:24:56.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.621+0000 7fcf56615640 1 -- 172.21.15.1:0/1593333903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf50145d40 msgr2=0x7fcf501eb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.621+0000 7fcf56615640 1 --2- 172.21.15.1:0/1593333903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf50145d40 0x7fcf501eb910 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fcf4800c700 tx=0x7fcf4800cbc0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.622+0000 7fcf56615640 1 -- 172.21.15.1:0/1593333903 shutdown_connections 2024-03-06T07:24:56.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.622+0000 7fcf56615640 1 --2- 172.21.15.1:0/1593333903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcf3c06b500 0x7fcf3c06d9c0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.622+0000 7fcf56615640 1 --2- 172.21.15.1:0/1593333903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf50145d40 0x7fcf501eb910 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.622+0000 7fcf56615640 1 --2- 172.21.15.1:0/1593333903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcf50152650 0x7fcf501eb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.622+0000 7fcf56615640 1 --2- 172.21.15.1:0/1593333903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf500bf9d0 0x7fcf5014da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.622+0000 7fcf56615640 1 -- 172.21.15.1:0/1593333903 >> 172.21.15.1:0/1593333903 conn(0x7fcf50066ee0 msgr2=0x7fcf5014e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.622+0000 7fcf56615640 1 -- 172.21.15.1:0/1593333903 shutdown_connections 2024-03-06T07:24:56.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.622+0000 7fcf56615640 1 -- 172.21.15.1:0/1593333903 wait complete. 2024-03-06T07:24:56.635 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:56.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.734+0000 7f396a49d640 1 Processor -- start 2024-03-06T07:24:56.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.734+0000 7f396a49d640 1 -- start start 2024-03-06T07:24:56.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.735+0000 7f396a49d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39640bf9d0 0x7f3964155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.735+0000 7f396a49d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39640c2fc0 con 0x7f3964152650 2024-03-06T07:24:56.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.735+0000 7f396a49d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3964158da0 con 0x7f39640c6b10 2024-03-06T07:24:56.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.735+0000 7f396a49d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3964155800 con 0x7f39640bf9d0 2024-03-06T07:24:56.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.735+0000 7f3963fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3964152650 0x7f3964158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41620/0 (socket says 172.21.15.1:41620) 2024-03-06T07:24:56.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.735+0000 7f3963fff640 1 -- 172.21.15.1:0/1575701595 learned_addr learned my addr 172.21.15.1:0/1575701595 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:56.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.735+0000 7f3968a3a640 1 --2- 172.21.15.1:0/1575701595 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39640bf9d0 0x7f3964155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.735+0000 7f3962ffd640 1 -- 172.21.15.1:0/1575701595 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704831440 0 0) 0x7f39640c2fc0 con 0x7f3964152650 2024-03-06T07:24:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.735+0000 7f3962ffd640 1 -- 172.21.15.1:0/1575701595 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f394c003520 con 0x7f3964152650 2024-03-06T07:24:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.735+0000 7f3962ffd640 1 -- 172.21.15.1:0/1575701595 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864387055 0 0) 0x7f3964158da0 con 0x7f39640c6b10 2024-03-06T07:24:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.735+0000 7f3962ffd640 1 -- 172.21.15.1:0/1575701595 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39640c2fc0 con 0x7f39640c6b10 2024-03-06T07:24:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.735+0000 7f3962ffd640 1 -- 172.21.15.1:0/1575701595 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3653614077 0 0) 0x7f394c003520 con 0x7f3964152650 2024-03-06T07:24:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.735+0000 7f3962ffd640 1 -- 172.21.15.1:0/1575701595 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39640bf9d0 msgr2=0x7f3964155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.735+0000 7f3962ffd640 1 --2- 172.21.15.1:0/1575701595 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39640bf9d0 0x7f3964155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f3962ffd640 1 -- 172.21.15.1:0/1575701595 >> v1:172.21.15.98:6789/0 conn(0x7f39640c6b10 legacy=0x7f39640c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f3962ffd640 1 -- 172.21.15.1:0/1575701595 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39640c6690 con 0x7f3964152650 2024-03-06T07:24:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f3962ffd640 1 -- 172.21.15.1:0/1575701595 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f3954001690 con 0x7f3964152650 2024-03-06T07:24:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f3962ffd640 1 -- 172.21.15.1:0/1575701595 >> v1:172.21.15.1:6789/0 conn(0x7f3964152650 legacy=0x7f3964158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f3962ffd640 1 --2- 172.21.15.1:0/1575701595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f394c003b70 0x7f394c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f3962ffd640 1 --2- 172.21.15.1:0/1575701595 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39640bf9d0 0x7f394c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f3962ffd640 1 --2- 172.21.15.1:0/1575701595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f394c004d40 0x7f394c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f3962ffd640 1 -- 172.21.15.1:0/1575701595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3964155800 con 0x7f394c004d40 2024-03-06T07:24:56.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f3962ffd640 1 -- 172.21.15.1:0/1575701595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c00baf0 con 0x7f394c003b70 2024-03-06T07:24:56.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f3962ffd640 1 -- 172.21.15.1:0/1575701595 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394c00bcf0 con 0x7f39640bf9d0 2024-03-06T07:24:56.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f3968a3a640 1 --2- 172.21.15.1:0/1575701595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f394c004d40 0x7f394c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f39637fe640 1 --2- 172.21.15.1:0/1575701595 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39640bf9d0 0x7f394c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f3963fff640 1 --2- 172.21.15.1:0/1575701595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f394c003b70 0x7f394c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f3968a3a640 1 -- 172.21.15.1:0/1575701595 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39640bf9d0 msgr2=0x7f394c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f3968a3a640 1 --2- 172.21.15.1:0/1575701595 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39640bf9d0 0x7f394c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f3968a3a640 1 -- 172.21.15.1:0/1575701595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f394c003b70 msgr2=0x7f394c003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f3968a3a640 1 --2- 172.21.15.1:0/1575701595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f394c003b70 0x7f394c003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.736+0000 7f3968a3a640 1 -- 172.21.15.1:0/1575701595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394c003520 con 0x7f394c004d40 2024-03-06T07:24:56.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.737+0000 7f3968a3a640 1 --2- 172.21.15.1:0/1575701595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f394c004d40 0x7f394c0051c0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f395c009fc0 tx=0x7f394c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2650c58ee871f2f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.737+0000 7f3962ffd640 1 -- 172.21.15.1:0/1575701595 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c039070 con 0x7f394c004d40 2024-03-06T07:24:56.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.737+0000 7f3962ffd640 1 -- 172.21.15.1:0/1575701595 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395c02cc20 con 0x7f394c004d40 2024-03-06T07:24:56.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.737+0000 7f3962ffd640 1 -- 172.21.15.1:0/1575701595 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c009c10 con 0x7f394c004d40 2024-03-06T07:24:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.738+0000 7f396a49d640 1 -- 172.21.15.1:0/1575701595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f394c004d40 msgr2=0x7f394c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.738+0000 7f396a49d640 1 --2- 172.21.15.1:0/1575701595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f394c004d40 0x7f394c0051c0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f395c009fc0 tx=0x7f394c005830 comp rx=0 tx=0).stop 2024-03-06T07:24:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.738+0000 7f3968a3a640 1 -- 172.21.15.1:0/1575701595 reap_dead start 2024-03-06T07:24:56.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.738+0000 7f396a49d640 1 -- 172.21.15.1:0/1575701595 shutdown_connections 2024-03-06T07:24:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.738+0000 7f396a49d640 1 -- 172.21.15.1:0/1575701595 >> 172.21.15.1:0/1575701595 conn(0x7f3964066ee0 msgr2=0x7f39640672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.738+0000 7f396a49d640 1 -- 172.21.15.1:0/1575701595 shutdown_connections 2024-03-06T07:24:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.738+0000 7f396a49d640 1 -- 172.21.15.1:0/1575701595 wait complete. 2024-03-06T07:24:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.739+0000 7f396a49d640 1 Processor -- start 2024-03-06T07:24:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.740+0000 7f396a49d640 1 -- start start 2024-03-06T07:24:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.741+0000 7f396a49d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f396414fb10 0x7f39641518d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.741+0000 7f396a49d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f396414ba40 0x7f396414fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.741+0000 7f396a49d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39641eb160 0x7f39641eed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.741+0000 7f396a49d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39641551b0 con 0x7f396414fb10 2024-03-06T07:24:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.741+0000 7f396a49d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39640c6fa0 con 0x7f396414ba40 2024-03-06T07:24:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.741+0000 7f396a49d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3964155490 con 0x7f39641eb160 2024-03-06T07:24:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.741+0000 7f3963fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f396414fb10 0x7f39641518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.741+0000 7f3963fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f396414fb10 0x7f39641518d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36852/0 (socket says 172.21.15.1:36852) 2024-03-06T07:24:56.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.741+0000 7f3963fff640 1 -- 172.21.15.1:0/3998597629 learned_addr learned my addr 172.21.15.1:0/3998597629 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.741+0000 7f3963fff640 1 -- 172.21.15.1:0/3998597629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39641eb160 msgr2=0x7f39641eed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.741+0000 7f3963fff640 1 --2- 172.21.15.1:0/3998597629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39641eb160 0x7f39641eed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.741+0000 7f3963fff640 1 -- 172.21.15.1:0/3998597629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f396414ba40 msgr2=0x7f396414fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.741+0000 7f3963fff640 1 --2- 172.21.15.1:0/3998597629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f396414ba40 0x7f396414fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.741+0000 7f3963fff640 1 -- 172.21.15.1:0/3998597629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39641e2690 con 0x7f396414fb10 2024-03-06T07:24:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.741+0000 7f3963fff640 1 --2- 172.21.15.1:0/3998597629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f396414fb10 0x7f39641518d0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f3954004f70 tx=0x7f39540074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.741+0000 7f3960ff9640 1 -- 172.21.15.1:0/3998597629 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3954007740 con 0x7f396414fb10 2024-03-06T07:24:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.742+0000 7f3960ff9640 1 -- 172.21.15.1:0/3998597629 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3954018070 con 0x7f396414fb10 2024-03-06T07:24:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.742+0000 7f396a49d640 1 -- 172.21.15.1:0/3998597629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39641e1df0 con 0x7f396414fb10 2024-03-06T07:24:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.742+0000 7f3960ff9640 1 -- 172.21.15.1:0/3998597629 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395400fb00 con 0x7f396414fb10 2024-03-06T07:24:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.742+0000 7f396a49d640 1 -- 172.21.15.1:0/3998597629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39641e23b0 con 0x7f396414fb10 2024-03-06T07:24:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.743+0000 7f396a49d640 1 -- 172.21.15.1:0/3998597629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396414f1b0 con 0x7f396414fb10 2024-03-06T07:24:56.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.743+0000 7f3960ff9640 1 -- 172.21.15.1:0/3998597629 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f395400fca0 con 0x7f396414fb10 2024-03-06T07:24:56.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.744+0000 7f3960ff9640 1 --2- 172.21.15.1:0/3998597629 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f395006b6a0 0x7f395006db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.744+0000 7f3960ff9640 1 -- 172.21.15.1:0/3998597629 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f395408a300 con 0x7f396414fb10 2024-03-06T07:24:56.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.744+0000 7f39637fe640 1 --2- 172.21.15.1:0/3998597629 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f395006b6a0 0x7f395006db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.745+0000 7f39637fe640 1 --2- 172.21.15.1:0/3998597629 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f395006b6a0 0x7f395006db60 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f396414dcf0 tx=0x7f3958009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:56.748+0000 7f3960ff9640 1 -- 172.21.15.1:0/3998597629 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f396414f1b0 con 0x7f396414fb10 2024-03-06T07:24:57.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.037+0000 7f396a49d640 1 -- 172.21.15.1:0/3998597629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3964067fd0 con 0x7f396414fb10 2024-03-06T07:24:57.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.039+0000 7f3960ff9640 1 -- 172.21.15.1:0/3998597629 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+16780 (secure 0 0 0) 0x7f39540589a0 con 0x7f396414fb10 2024-03-06T07:24:57.041 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:57.041 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":121,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:24:54.242199+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:50.865151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:51.564469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:57.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.041+0000 7f396a49d640 1 -- 172.21.15.1:0/3998597629 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f395006b6a0 msgr2=0x7f395006db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.041+0000 7f396a49d640 1 --2- 172.21.15.1:0/3998597629 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f395006b6a0 0x7f395006db60 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f396414dcf0 tx=0x7f3958009040 comp rx=0 tx=0).stop 2024-03-06T07:24:57.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.041+0000 7f396a49d640 1 -- 172.21.15.1:0/3998597629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f396414fb10 msgr2=0x7f39641518d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.041+0000 7f396a49d640 1 --2- 172.21.15.1:0/3998597629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f396414fb10 0x7f39641518d0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f3954004f70 tx=0x7f39540074f0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.041+0000 7f396a49d640 1 -- 172.21.15.1:0/3998597629 shutdown_connections 2024-03-06T07:24:57.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.041+0000 7f396a49d640 1 --2- 172.21.15.1:0/3998597629 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f395006b6a0 0x7f395006db60 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.041+0000 7f396a49d640 1 --2- 172.21.15.1:0/3998597629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39641eb160 0x7f39641eed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.041+0000 7f396a49d640 1 --2- 172.21.15.1:0/3998597629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f396414ba40 0x7f396414fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.041+0000 7f396a49d640 1 --2- 172.21.15.1:0/3998597629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f396414fb10 0x7f39641518d0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.041+0000 7f396a49d640 1 -- 172.21.15.1:0/3998597629 >> 172.21.15.1:0/3998597629 conn(0x7f3964066ee0 msgr2=0x7f39640c0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.041+0000 7f396a49d640 1 -- 172.21.15.1:0/3998597629 shutdown_connections 2024-03-06T07:24:57.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.041+0000 7f396a49d640 1 -- 172.21.15.1:0/3998597629 wait complete. 2024-03-06T07:24:57.054 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:24:57.054 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:57.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.156+0000 7fecc1434640 1 Processor -- start 2024-03-06T07:24:57.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.156+0000 7fecc1434640 1 -- start start 2024-03-06T07:24:57.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.157+0000 7fecc1434640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fecbc0bf9d0 0x7fecbc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.157+0000 7fecc1434640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecbc0c2fc0 con 0x7fecbc152650 2024-03-06T07:24:57.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.157+0000 7fecc1434640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecbc158da0 con 0x7fecbc0c6b10 2024-03-06T07:24:57.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.157+0000 7fecc1434640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecbc155800 con 0x7fecbc0bf9d0 2024-03-06T07:24:57.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.157+0000 7fecbb7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fecbc0bf9d0 0x7fecbc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.157+0000 7fecbaffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fecbc152650 0x7fecbc158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41636/0 (socket says 172.21.15.1:41636) 2024-03-06T07:24:57.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.157+0000 7fecbaffd640 1 -- 172.21.15.1:0/1502031949 learned_addr learned my addr 172.21.15.1:0/1502031949 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:57.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.158+0000 7fecbb7fe640 1 -- 172.21.15.1:0/1502031949 >> v1:172.21.15.98:6789/0 conn(0x7fecbc0c6b10 legacy=0x7fecbc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.158+0000 7fecbb7fe640 1 -- 172.21.15.1:0/1502031949 >> v1:172.21.15.1:6789/0 conn(0x7fecbc152650 legacy=0x7fecbc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.158+0000 7fecbb7fe640 1 -- 172.21.15.1:0/1502031949 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecbc0c6690 con 0x7fecbc0bf9d0 2024-03-06T07:24:57.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.158+0000 7fecbb7fe640 1 --2- 172.21.15.1:0/1502031949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecbc0bf9d0 0x7fecbc155c90 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fecb4009d90 tx=0x7fecb402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92c40720bc7b1ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.158+0000 7fecb9ffb640 1 -- 172.21.15.1:0/1502031949 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb4039070 con 0x7fecbc0bf9d0 2024-03-06T07:24:57.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.158+0000 7fecb9ffb640 1 -- 172.21.15.1:0/1502031949 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb402cc60 con 0x7fecbc0bf9d0 2024-03-06T07:24:57.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.159+0000 7fecb9ffb640 1 -- 172.21.15.1:0/1502031949 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb4034660 con 0x7fecbc0bf9d0 2024-03-06T07:24:57.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.159+0000 7fecc1434640 1 -- 172.21.15.1:0/1502031949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecbc0bf9d0 msgr2=0x7fecbc155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.159+0000 7fecc1434640 1 --2- 172.21.15.1:0/1502031949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecbc0bf9d0 0x7fecbc155c90 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fecb4009d90 tx=0x7fecb402beb0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.159+0000 7fecc1434640 1 -- 172.21.15.1:0/1502031949 shutdown_connections 2024-03-06T07:24:57.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.159+0000 7fecc1434640 1 --2- 172.21.15.1:0/1502031949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecbc0bf9d0 0x7fecbc155c90 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.159+0000 7fecc1434640 1 -- 172.21.15.1:0/1502031949 >> 172.21.15.1:0/1502031949 conn(0x7fecbc066ee0 msgr2=0x7fecbc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.159+0000 7fecc1434640 1 -- 172.21.15.1:0/1502031949 shutdown_connections 2024-03-06T07:24:57.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.159+0000 7fecc1434640 1 -- 172.21.15.1:0/1502031949 wait complete. 2024-03-06T07:24:57.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.160+0000 7fecc1434640 1 Processor -- start 2024-03-06T07:24:57.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.161+0000 7fecc1434640 1 -- start start 2024-03-06T07:24:57.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.161+0000 7fecc1434640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fecbc0bf9d0 0x7fecbc154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.161+0000 7fecc1434640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecbc152650 0x7fecbc1515a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.161+0000 7fecc1434640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecbc151ae0 0x7fecbc14b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.161+0000 7fecc1434640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecbc1551b0 con 0x7fecbc151ae0 2024-03-06T07:24:57.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.161+0000 7fecc1434640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecbc0c6fa0 con 0x7fecbc152650 2024-03-06T07:24:57.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.161+0000 7fecc1434640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecbc155490 con 0x7fecbc0bf9d0 2024-03-06T07:24:57.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.161+0000 7fecbb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecbc151ae0 0x7fecbc14b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.161+0000 7fecbb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecbc151ae0 0x7fecbc14b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36872/0 (socket says 172.21.15.1:36872) 2024-03-06T07:24:57.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.161+0000 7fecbb7fe640 1 -- 172.21.15.1:0/414258634 learned_addr learned my addr 172.21.15.1:0/414258634 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:57.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.162+0000 7fecbaffd640 1 --2- 172.21.15.1:0/414258634 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fecbc0bf9d0 0x7fecbc154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.162+0000 7fecba7fc640 1 --2- 172.21.15.1:0/414258634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecbc152650 0x7fecbc1515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.162+0000 7fecbb7fe640 1 -- 172.21.15.1:0/414258634 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fecbc0bf9d0 msgr2=0x7fecbc154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.162+0000 7fecbb7fe640 1 --2- 172.21.15.1:0/414258634 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fecbc0bf9d0 0x7fecbc154700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.162+0000 7fecbb7fe640 1 -- 172.21.15.1:0/414258634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecbc152650 msgr2=0x7fecbc1515a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.162+0000 7fecbb7fe640 1 --2- 172.21.15.1:0/414258634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecbc152650 0x7fecbc1515a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.162+0000 7fecbb7fe640 1 -- 172.21.15.1:0/414258634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecbc152f10 con 0x7fecbc151ae0 2024-03-06T07:24:57.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.162+0000 7fecba7fc640 1 --2- 172.21.15.1:0/414258634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecbc152650 0x7fecbc1515a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:57.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.162+0000 7fecbaffd640 1 --2- 172.21.15.1:0/414258634 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fecbc0bf9d0 0x7fecbc154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:57.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.162+0000 7fecbb7fe640 1 --2- 172.21.15.1:0/414258634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecbc151ae0 0x7fecbc14b170 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fecb4009d90 tx=0x7fecb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.162+0000 7feca37fe640 1 -- 172.21.15.1:0/414258634 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb4039050 con 0x7fecbc151ae0 2024-03-06T07:24:57.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.162+0000 7feca37fe640 1 -- 172.21.15.1:0/414258634 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb4045070 con 0x7fecbc151ae0 2024-03-06T07:24:57.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.162+0000 7fecc1434640 1 -- 172.21.15.1:0/414258634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecbc1531f0 con 0x7fecbc151ae0 2024-03-06T07:24:57.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.162+0000 7feca37fe640 1 -- 172.21.15.1:0/414258634 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb403e830 con 0x7fecbc151ae0 2024-03-06T07:24:57.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.162+0000 7fecc1434640 1 -- 172.21.15.1:0/414258634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecbc1537b0 con 0x7fecbc151ae0 2024-03-06T07:24:57.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.163+0000 7fecc1434640 1 -- 172.21.15.1:0/414258634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecbc067fd0 con 0x7fecbc151ae0 2024-03-06T07:24:57.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.163+0000 7feca37fe640 1 -- 172.21.15.1:0/414258634 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fecb403e9d0 con 0x7fecbc151ae0 2024-03-06T07:24:57.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.164+0000 7feca37fe640 1 --2- 172.21.15.1:0/414258634 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feca806b4d0 0x7feca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.164+0000 7feca37fe640 1 -- 172.21.15.1:0/414258634 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fecb40ad320 con 0x7fecbc151ae0 2024-03-06T07:24:57.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.164+0000 7fecbaffd640 1 --2- 172.21.15.1:0/414258634 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feca806b4d0 0x7feca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.164+0000 7fecbaffd640 1 --2- 172.21.15.1:0/414258634 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feca806b4d0 0x7feca806d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fecb0005880 tx=0x7fecb0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.168+0000 7feca37fe640 1 -- 172.21.15.1:0/414258634 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fecb407c1c0 con 0x7fecbc151ae0 2024-03-06T07:24:57.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.349+0000 7fecc1434640 1 -- 172.21.15.1:0/414258634 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fecbc0b01a0 con 0x7feca806b4d0 2024-03-06T07:24:57.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.356+0000 7feca37fe640 1 -- 172.21.15.1:0/414258634 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252560 (secure 0 0 0) 0x7fecbc0b01a0 con 0x7feca806b4d0 2024-03-06T07:24:57.361 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:57.363 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:24:57.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.359+0000 7fecc1434640 1 -- 172.21.15.1:0/414258634 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feca806b4d0 msgr2=0x7feca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.359+0000 7fecc1434640 1 --2- 172.21.15.1:0/414258634 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feca806b4d0 0x7feca806d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fecb0005880 tx=0x7fecb0009040 comp rx=0 tx=0).stop 2024-03-06T07:24:57.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.359+0000 7fecc1434640 1 -- 172.21.15.1:0/414258634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecbc151ae0 msgr2=0x7fecbc14b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.359+0000 7fecc1434640 1 --2- 172.21.15.1:0/414258634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecbc151ae0 0x7fecbc14b170 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fecb4009d90 tx=0x7fecb4004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:57.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.359+0000 7fecc1434640 1 -- 172.21.15.1:0/414258634 shutdown_connections 2024-03-06T07:24:57.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.359+0000 7fecc1434640 1 --2- 172.21.15.1:0/414258634 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feca806b4d0 0x7feca806d990 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.359+0000 7fecc1434640 1 --2- 172.21.15.1:0/414258634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecbc151ae0 0x7fecbc14b170 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.359+0000 7fecc1434640 1 --2- 172.21.15.1:0/414258634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecbc152650 0x7fecbc1515a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.359+0000 7fecc1434640 1 --2- 172.21.15.1:0/414258634 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fecbc0bf9d0 0x7fecbc154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.359+0000 7fecc1434640 1 -- 172.21.15.1:0/414258634 >> 172.21.15.1:0/414258634 conn(0x7fecbc066ee0 msgr2=0x7fecbc0c59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.359+0000 7fecc1434640 1 -- 172.21.15.1:0/414258634 shutdown_connections 2024-03-06T07:24:57.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.359+0000 7fecc1434640 1 -- 172.21.15.1:0/414258634 wait complete. 2024-03-06T07:24:57.375 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-03-06T07:24:56.752831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261432,"kb_used_data":11972,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713288,"statfs":{"total":773094113280,"available":772826406912,"internally_reserved":0,"allocated":12259328,"data_stored":6998384,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.521375"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439736+0000","last_change":"2024-03-06T07:24:53.248053+0000","last_active":"2024-03-06T07:24:54.439736+0000","last_peered":"2024-03-06T07:24:54.439736+0000","last_clean":"2024-03-06T07:24:54.439736+0000","last_became_active":"2024-03-06T07:24:53.247769+0000","last_became_peered":"2024-03-06T07:24:53.247769+0000","last_unstale":"2024-03-06T07:24:54.439736+0000","last_undegraded":"2024-03-06T07:24:54.439736+0000","last_fullsized":"2024-03-06T07:24:54.439736+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:56.349947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256418+0000","last_change":"2024-03-06T07:24:53.258292+0000","last_active":"2024-03-06T07:24:54.256418+0000","last_peered":"2024-03-06T07:24:54.256418+0000","last_clean":"2024-03-06T07:24:54.256418+0000","last_became_active":"2024-03-06T07:24:53.257461+0000","last_became_peered":"2024-03-06T07:24:53.257461+0000","last_unstale":"2024-03-06T07:24:54.256418+0000","last_undegraded":"2024-03-06T07:24:54.256418+0000","last_fullsized":"2024-03-06T07:24:54.256418+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:03.358652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439938+0000","last_change":"2024-03-06T07:24:53.254981+0000","last_active":"2024-03-06T07:24:54.439938+0000","last_peered":"2024-03-06T07:24:54.439938+0000","last_clean":"2024-03-06T07:24:54.439938+0000","last_became_active":"2024-03-06T07:24:53.254714+0000","last_became_peered":"2024-03-06T07:24:53.254714+0000","last_unstale":"2024-03-06T07:24:54.439938+0000","last_undegraded":"2024-03-06T07:24:54.439938+0000","last_fullsized":"2024-03-06T07:24:54.439938+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:54.825505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256238+0000","last_change":"2024-03-06T07:24:53.252463+0000","last_active":"2024-03-06T07:24:54.256238+0000","last_peered":"2024-03-06T07:24:54.256238+0000","last_clean":"2024-03-06T07:24:54.256238+0000","last_became_active":"2024-03-06T07:24:53.252183+0000","last_became_peered":"2024-03-06T07:24:53.252183+0000","last_unstale":"2024-03-06T07:24:54.256238+0000","last_undegraded":"2024-03-06T07:24:54.256238+0000","last_fullsized":"2024-03-06T07:24:54.256238+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:23.122619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439024+0000","last_change":"2024-03-06T07:24:53.259191+0000","last_active":"2024-03-06T07:24:54.439024+0000","last_peered":"2024-03-06T07:24:54.439024+0000","last_clean":"2024-03-06T07:24:54.439024+0000","last_became_active":"2024-03-06T07:24:53.258904+0000","last_became_peered":"2024-03-06T07:24:53.258904+0000","last_unstale":"2024-03-06T07:24:54.439024+0000","last_undegraded":"2024-03-06T07:24:54.439024+0000","last_fullsized":"2024-03-06T07:24:54.439024+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:54.330802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249639+0000","last_change":"2024-03-06T07:24:53.255509+0000","last_active":"2024-03-06T07:24:54.249639+0000","last_peered":"2024-03-06T07:24:54.249639+0000","last_clean":"2024-03-06T07:24:54.249639+0000","last_became_active":"2024-03-06T07:24:53.255228+0000","last_became_peered":"2024-03-06T07:24:53.255228+0000","last_unstale":"2024-03-06T07:24:54.249639+0000","last_undegraded":"2024-03-06T07:24:54.249639+0000","last_fullsized":"2024-03-06T07:24:54.249639+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:28.333906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250340+0000","last_change":"2024-03-06T07:24:53.254560+0000","last_active":"2024-03-06T07:24:54.250340+0000","last_peered":"2024-03-06T07:24:54.250340+0000","last_clean":"2024-03-06T07:24:54.250340+0000","last_became_active":"2024-03-06T07:24:53.250079+0000","last_became_peered":"2024-03-06T07:24:53.250079+0000","last_unstale":"2024-03-06T07:24:54.250340+0000","last_undegraded":"2024-03-06T07:24:54.250340+0000","last_fullsized":"2024-03-06T07:24:54.250340+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:16.327805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.253727+0000","last_change":"2024-03-06T07:24:53.248358+0000","last_active":"2024-03-06T07:24:54.253727+0000","last_peered":"2024-03-06T07:24:54.253727+0000","last_clean":"2024-03-06T07:24:54.253727+0000","last_became_active":"2024-03-06T07:24:53.248089+0000","last_became_peered":"2024-03-06T07:24:53.248089+0000","last_unstale":"2024-03-06T07:24:54.253727+0000","last_undegraded":"2024-03-06T07:24:54.253727+0000","last_fullsized":"2024-03-06T07:24:54.253727+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:27.838125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249902+0000","last_change":"2024-03-06T07:24:53.253735+0000","last_active":"2024-03-06T07:24:54.249902+0000","last_peered":"2024-03-06T07:24:54.249902+0000","last_clean":"2024-03-06T07:24:54.249902+0000","last_became_active":"2024-03-06T07:24:53.253463+0000","last_became_peered":"2024-03-06T07:24:53.253463+0000","last_unstale":"2024-03-06T07:24:54.249902+0000","last_undegraded":"2024-03-06T07:24:54.249902+0000","last_fullsized":"2024-03-06T07:24:54.249902+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:51.891213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.248811+0000","last_change":"2024-03-06T07:24:53.247511+0000","last_active":"2024-03-06T07:24:54.248811+0000","last_peered":"2024-03-06T07:24:54.248811+0000","last_clean":"2024-03-06T07:24:54.248811+0000","last_became_active":"2024-03-06T07:24:53.247223+0000","last_became_peered":"2024-03-06T07:24:53.247223+0000","last_unstale":"2024-03-06T07:24:54.248811+0000","last_undegraded":"2024-03-06T07:24:54.248811+0000","last_fullsized":"2024-03-06T07:24:54.248811+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:30.974392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256225+0000","last_change":"2024-03-06T07:24:53.253451+0000","last_active":"2024-03-06T07:24:54.256225+0000","last_peered":"2024-03-06T07:24:54.256225+0000","last_clean":"2024-03-06T07:24:54.256225+0000","last_became_active":"2024-03-06T07:24:53.253198+0000","last_became_peered":"2024-03-06T07:24:53.253198+0000","last_unstale":"2024-03-06T07:24:54.256225+0000","last_undegraded":"2024-03-06T07:24:54.256225+0000","last_fullsized":"2024-03-06T07:24:54.256225+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:43.213777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439903+0000","last_change":"2024-03-06T07:24:53.255928+0000","last_active":"2024-03-06T07:24:54.439903+0000","last_peered":"2024-03-06T07:24:54.439903+0000","last_clean":"2024-03-06T07:24:54.439903+0000","last_became_active":"2024-03-06T07:24:53.255604+0000","last_became_peered":"2024-03-06T07:24:53.255604+0000","last_unstale":"2024-03-06T07:24:54.439903+0000","last_undegraded":"2024-03-06T07:24:54.439903+0000","last_fullsized":"2024-03-06T07:24:54.439903+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:31.410528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249817+0000","last_change":"2024-03-06T07:24:53.255551+0000","last_active":"2024-03-06T07:24:54.249817+0000","last_peered":"2024-03-06T07:24:54.249817+0000","last_clean":"2024-03-06T07:24:54.249817+0000","last_became_active":"2024-03-06T07:24:53.255255+0000","last_became_peered":"2024-03-06T07:24:53.255255+0000","last_unstale":"2024-03-06T07:24:54.249817+0000","last_undegraded":"2024-03-06T07:24:54.249817+0000","last_fullsized":"2024-03-06T07:24:54.249817+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:36.778107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439307+0000","last_change":"2024-03-06T07:24:53.245269+0000","last_active":"2024-03-06T07:24:54.439307+0000","last_peered":"2024-03-06T07:24:54.439307+0000","last_clean":"2024-03-06T07:24:54.439307+0000","last_became_active":"2024-03-06T07:24:53.244958+0000","last_became_peered":"2024-03-06T07:24:53.244958+0000","last_unstale":"2024-03-06T07:24:54.439307+0000","last_undegraded":"2024-03-06T07:24:54.439307+0000","last_fullsized":"2024-03-06T07:24:54.439307+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:10.294078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439122+0000","last_change":"2024-03-06T07:24:53.246040+0000","last_active":"2024-03-06T07:24:54.439122+0000","last_peered":"2024-03-06T07:24:54.439122+0000","last_clean":"2024-03-06T07:24:54.439122+0000","last_became_active":"2024-03-06T07:24:53.245749+0000","last_became_peered":"2024-03-06T07:24:53.245749+0000","last_unstale":"2024-03-06T07:24:54.439122+0000","last_undegraded":"2024-03-06T07:24:54.439122+0000","last_fullsized":"2024-03-06T07:24:54.439122+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:52.353345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249503+0000","last_change":"2024-03-06T07:24:53.258563+0000","last_active":"2024-03-06T07:24:54.249503+0000","last_peered":"2024-03-06T07:24:54.249503+0000","last_clean":"2024-03-06T07:24:54.249503+0000","last_became_active":"2024-03-06T07:24:53.257821+0000","last_became_peered":"2024-03-06T07:24:53.257821+0000","last_unstale":"2024-03-06T07:24:54.249503+0000","last_undegraded":"2024-03-06T07:24:54.249503+0000","last_fullsized":"2024-03-06T07:24:54.249503+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:05.761133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249996+0000","last_change":"2024-03-06T07:24:53.256941+0000","last_active":"2024-03-06T07:24:54.249996+0000","last_peered":"2024-03-06T07:24:54.249996+0000","last_clean":"2024-03-06T07:24:54.249996+0000","last_became_active":"2024-03-06T07:24:53.256676+0000","last_became_peered":"2024-03-06T07:24:53.256676+0000","last_unstale":"2024-03-06T07:24:54.249996+0000","last_undegraded":"2024-03-06T07:24:54.249996+0000","last_fullsized":"2024-03-06T07:24:54.249996+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:23.378702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439390+0000","last_change":"2024-03-06T07:24:53.260235+0000","last_active":"2024-03-06T07:24:54.439390+0000","last_peered":"2024-03-06T07:24:54.439390+0000","last_clean":"2024-03-06T07:24:54.439390+0000","last_became_active":"2024-03-06T07:24:53.259779+0000","last_became_peered":"2024-03-06T07:24:53.259779+0000","last_unstale":"2024-03-06T07:24:54.439390+0000","last_undegraded":"2024-03-06T07:24:54.439390+0000","last_fullsized":"2024-03-06T07:24:54.439390+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:29.942697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250588+0000","last_change":"2024-03-06T07:24:53.257044+0000","last_active":"2024-03-06T07:24:54.250588+0000","last_peered":"2024-03-06T07:24:54.250588+0000","last_clean":"2024-03-06T07:24:54.250588+0000","last_became_active":"2024-03-06T07:24:53.256788+0000","last_became_peered":"2024-03-06T07:24:53.256788+0000","last_unstale":"2024-03-06T07:24:54.250588+0000","last_undegraded":"2024-03-06T07:24:54.250588+0000","last_fullsized":"2024-03-06T07:24:54.250588+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:30.909753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439660+0000","last_change":"2024-03-06T07:24:53.258248+0000","last_active":"2024-03-06T07:24:54.439660+0000","last_peered":"2024-03-06T07:24:54.439660+0000","last_clean":"2024-03-06T07:24:54.439660+0000","last_became_active":"2024-03-06T07:24:53.257968+0000","last_became_peered":"2024-03-06T07:24:53.257968+0000","last_unstale":"2024-03-06T07:24:54.439660+0000","last_undegraded":"2024-03-06T07:24:54.439660+0000","last_fullsized":"2024-03-06T07:24:54.439660+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:09.808543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256195+0000","last_change":"2024-03-06T07:24:53.263730+0000","last_active":"2024-03-06T07:24:54.256195+0000","last_peered":"2024-03-06T07:24:54.256195+0000","last_clean":"2024-03-06T07:24:54.256195+0000","last_became_active":"2024-03-06T07:24:53.263451+0000","last_became_peered":"2024-03-06T07:24:53.263451+0000","last_unstale":"2024-03-06T07:24:54.256195+0000","last_undegraded":"2024-03-06T07:24:54.256195+0000","last_fullsized":"2024-03-06T07:24:54.256195+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:44.388371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439787+0000","last_change":"2024-03-06T07:24:53.256663+0000","last_active":"2024-03-06T07:24:54.439787+0000","last_peered":"2024-03-06T07:24:54.439787+0000","last_clean":"2024-03-06T07:24:54.439787+0000","last_became_active":"2024-03-06T07:24:53.256366+0000","last_became_peered":"2024-03-06T07:24:53.256366+0000","last_unstale":"2024-03-06T07:24:54.439787+0000","last_undegraded":"2024-03-06T07:24:54.439787+0000","last_fullsized":"2024-03-06T07:24:54.439787+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:54.593816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256136+0000","last_change":"2024-03-06T07:24:53.251100+0000","last_active":"2024-03-06T07:24:54.256136+0000","last_peered":"2024-03-06T07:24:54.256136+0000","last_clean":"2024-03-06T07:24:54.256136+0000","last_became_active":"2024-03-06T07:24:53.250823+0000","last_became_peered":"2024-03-06T07:24:53.250823+0000","last_unstale":"2024-03-06T07:24:54.256136+0000","last_undegraded":"2024-03-06T07:24:54.256136+0000","last_fullsized":"2024-03-06T07:24:54.256136+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:27.838125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249584+0000","last_change":"2024-03-06T07:24:53.254207+0000","last_active":"2024-03-06T07:24:54.249584+0000","last_peered":"2024-03-06T07:24:54.249584+0000","last_clean":"2024-03-06T07:24:54.249584+0000","last_became_active":"2024-03-06T07:24:53.253888+0000","last_became_peered":"2024-03-06T07:24:53.253888+0000","last_unstale":"2024-03-06T07:24:54.249584+0000","last_undegraded":"2024-03-06T07:24:54.249584+0000","last_fullsized":"2024-03-06T07:24:54.249584+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:07.871828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.440109+0000","last_change":"2024-03-06T07:24:53.259683+0000","last_active":"2024-03-06T07:24:54.440109+0000","last_peered":"2024-03-06T07:24:54.440109+0000","last_clean":"2024-03-06T07:24:54.440109+0000","last_became_active":"2024-03-06T07:24:53.259095+0000","last_became_peered":"2024-03-06T07:24:53.259095+0000","last_unstale":"2024-03-06T07:24:54.440109+0000","last_undegraded":"2024-03-06T07:24:54.440109+0000","last_fullsized":"2024-03-06T07:24:54.440109+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:57.193091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439960+0000","last_change":"2024-03-06T07:24:53.253741+0000","last_active":"2024-03-06T07:24:54.439960+0000","last_peered":"2024-03-06T07:24:54.439960+0000","last_clean":"2024-03-06T07:24:54.439960+0000","last_became_active":"2024-03-06T07:24:53.253472+0000","last_became_peered":"2024-03-06T07:24:53.253472+0000","last_unstale":"2024-03-06T07:24:54.439960+0000","last_undegraded":"2024-03-06T07:24:54.439960+0000","last_fullsized":"2024-03-06T07:24:54.439960+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:48.653846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250728+0000","last_change":"2024-03-06T07:24:53.248452+0000","last_active":"2024-03-06T07:24:54.250728+0000","last_peered":"2024-03-06T07:24:54.250728+0000","last_clean":"2024-03-06T07:24:54.250728+0000","last_became_active":"2024-03-06T07:24:53.248158+0000","last_became_peered":"2024-03-06T07:24:53.248158+0000","last_unstale":"2024-03-06T07:24:54.250728+0000","last_undegraded":"2024-03-06T07:24:54.250728+0000","last_fullsized":"2024-03-06T07:24:54.250728+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:42.285105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.248688+0000","last_change":"2024-03-06T07:24:53.246872+0000","last_active":"2024-03-06T07:24:54.248688+0000","last_peered":"2024-03-06T07:24:54.248688+0000","last_clean":"2024-03-06T07:24:54.248688+0000","last_became_active":"2024-03-06T07:24:53.246585+0000","last_became_peered":"2024-03-06T07:24:53.246585+0000","last_unstale":"2024-03-06T07:24:54.248688+0000","last_undegraded":"2024-03-06T07:24:54.248688+0000","last_fullsized":"2024-03-06T07:24:54.248688+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:52.290888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439926+0000","last_change":"2024-03-06T07:24:53.261003+0000","last_active":"2024-03-06T07:24:54.439926+0000","last_peered":"2024-03-06T07:24:54.439926+0000","last_clean":"2024-03-06T07:24:54.439926+0000","last_became_active":"2024-03-06T07:24:53.260752+0000","last_became_peered":"2024-03-06T07:24:53.260752+0000","last_unstale":"2024-03-06T07:24:54.439926+0000","last_undegraded":"2024-03-06T07:24:54.439926+0000","last_fullsized":"2024-03-06T07:24:54.439926+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:43.309242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439798+0000","last_change":"2024-03-06T07:24:53.260357+0000","last_active":"2024-03-06T07:24:54.439798+0000","last_peered":"2024-03-06T07:24:54.439798+0000","last_clean":"2024-03-06T07:24:54.439798+0000","last_became_active":"2024-03-06T07:24:53.259659+0000","last_became_peered":"2024-03-06T07:24:53.259659+0000","last_unstale":"2024-03-06T07:24:54.439798+0000","last_undegraded":"2024-03-06T07:24:54.439798+0000","last_fullsized":"2024-03-06T07:24:54.439798+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:46.394428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.253794+0000","last_change":"2024-03-06T07:24:53.252892+0000","last_active":"2024-03-06T07:24:54.253794+0000","last_peered":"2024-03-06T07:24:54.253794+0000","last_clean":"2024-03-06T07:24:54.253794+0000","last_became_active":"2024-03-06T07:24:53.252633+0000","last_became_peered":"2024-03-06T07:24:53.252633+0000","last_unstale":"2024-03-06T07:24:54.253794+0000","last_undegraded":"2024-03-06T07:24:54.253794+0000","last_fullsized":"2024-03-06T07:24:54.253794+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:43.213777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.248415+0000","last_change":"2024-03-06T07:24:53.249410+0000","last_active":"2024-03-06T07:24:54.248415+0000","last_peered":"2024-03-06T07:24:54.248415+0000","last_clean":"2024-03-06T07:24:54.248415+0000","last_became_active":"2024-03-06T07:24:53.249136+0000","last_became_peered":"2024-03-06T07:24:53.249136+0000","last_unstale":"2024-03-06T07:24:54.248415+0000","last_undegraded":"2024-03-06T07:24:54.248415+0000","last_fullsized":"2024-03-06T07:24:54.248415+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:12.785622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.253735+0000","last_change":"2024-03-06T07:24:53.244091+0000","last_active":"2024-03-06T07:24:54.253735+0000","last_peered":"2024-03-06T07:24:54.253735+0000","last_clean":"2024-03-06T07:24:54.253735+0000","last_became_active":"2024-03-06T07:24:53.243786+0000","last_became_peered":"2024-03-06T07:24:53.243786+0000","last_unstale":"2024-03-06T07:24:54.253735+0000","last_undegraded":"2024-03-06T07:24:54.253735+0000","last_fullsized":"2024-03-06T07:24:54.253735+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:44.388371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"120'4","reported_seq":18,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.440149+0000","last_change":"2024-03-06T07:24:52.247044+0000","last_active":"2024-03-06T07:24:54.440149+0000","last_peered":"2024-03-06T07:24:54.440149+0000","last_clean":"2024-03-06T07:24:54.440149+0000","last_became_active":"2024-03-06T07:24:52.246760+0000","last_became_peered":"2024-03-06T07:24:52.246760+0000","last_unstale":"2024-03-06T07:24:54.440149+0000","last_undegraded":"2024-03-06T07:24:54.440149+0000","last_fullsized":"2024-03-06T07:24:54.440149+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:04.476430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439298+0000","last_change":"2024-03-06T07:24:53.256811+0000","last_active":"2024-03-06T07:24:54.439298+0000","last_peered":"2024-03-06T07:24:54.439298+0000","last_clean":"2024-03-06T07:24:54.439298+0000","last_became_active":"2024-03-06T07:24:53.256524+0000","last_became_peered":"2024-03-06T07:24:53.256524+0000","last_unstale":"2024-03-06T07:24:54.439298+0000","last_undegraded":"2024-03-06T07:24:54.439298+0000","last_fullsized":"2024-03-06T07:24:54.439298+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:03.848378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439594+0000","last_change":"2024-03-06T07:24:52.248683+0000","last_active":"2024-03-06T07:24:54.439594+0000","last_peered":"2024-03-06T07:24:54.439594+0000","last_clean":"2024-03-06T07:24:54.439594+0000","last_became_active":"2024-03-06T07:24:52.248404+0000","last_became_peered":"2024-03-06T07:24:52.248404+0000","last_unstale":"2024-03-06T07:24:54.439594+0000","last_undegraded":"2024-03-06T07:24:54.439594+0000","last_fullsized":"2024-03-06T07:24:54.439594+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:15.323244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256238+0000","last_change":"2024-03-06T07:24:53.250936+0000","last_active":"2024-03-06T07:24:54.256238+0000","last_peered":"2024-03-06T07:24:54.256238+0000","last_clean":"2024-03-06T07:24:54.256238+0000","last_became_active":"2024-03-06T07:24:53.250685+0000","last_became_peered":"2024-03-06T07:24:53.250685+0000","last_unstale":"2024-03-06T07:24:54.256238+0000","last_undegraded":"2024-03-06T07:24:54.256238+0000","last_fullsized":"2024-03-06T07:24:54.256238+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:32.092076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250803+0000","last_change":"2024-03-06T07:24:52.249483+0000","last_active":"2024-03-06T07:24:54.250803+0000","last_peered":"2024-03-06T07:24:54.250803+0000","last_clean":"2024-03-06T07:24:54.250803+0000","last_became_active":"2024-03-06T07:24:52.249176+0000","last_became_peered":"2024-03-06T07:24:52.249176+0000","last_unstale":"2024-03-06T07:24:54.250803+0000","last_undegraded":"2024-03-06T07:24:54.250803+0000","last_fullsized":"2024-03-06T07:24:54.250803+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:02.354091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439552+0000","last_change":"2024-03-06T07:24:53.253820+0000","last_active":"2024-03-06T07:24:54.439552+0000","last_peered":"2024-03-06T07:24:54.439552+0000","last_clean":"2024-03-06T07:24:54.439552+0000","last_became_active":"2024-03-06T07:24:53.253522+0000","last_became_peered":"2024-03-06T07:24:53.253522+0000","last_unstale":"2024-03-06T07:24:54.439552+0000","last_undegraded":"2024-03-06T07:24:54.439552+0000","last_fullsized":"2024-03-06T07:24:54.439552+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:59.346066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250733+0000","last_change":"2024-03-06T07:24:52.250493+0000","last_active":"2024-03-06T07:24:54.250733+0000","last_peered":"2024-03-06T07:24:54.250733+0000","last_clean":"2024-03-06T07:24:54.250733+0000","last_became_active":"2024-03-06T07:24:52.250205+0000","last_became_peered":"2024-03-06T07:24:52.250205+0000","last_unstale":"2024-03-06T07:24:54.250733+0000","last_undegraded":"2024-03-06T07:24:54.250733+0000","last_fullsized":"2024-03-06T07:24:54.250733+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.501831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.253991+0000","last_change":"2024-03-06T07:24:53.253795+0000","last_active":"2024-03-06T07:24:54.253991+0000","last_peered":"2024-03-06T07:24:54.253991+0000","last_clean":"2024-03-06T07:24:54.253991+0000","last_became_active":"2024-03-06T07:24:53.253513+0000","last_became_peered":"2024-03-06T07:24:53.253513+0000","last_unstale":"2024-03-06T07:24:54.253991+0000","last_undegraded":"2024-03-06T07:24:54.253991+0000","last_fullsized":"2024-03-06T07:24:54.253991+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:23.122619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.253883+0000","last_change":"2024-03-06T07:24:52.249840+0000","last_active":"2024-03-06T07:24:54.253883+0000","last_peered":"2024-03-06T07:24:54.253883+0000","last_clean":"2024-03-06T07:24:54.253883+0000","last_became_active":"2024-03-06T07:24:52.249544+0000","last_became_peered":"2024-03-06T07:24:52.249544+0000","last_unstale":"2024-03-06T07:24:54.253883+0000","last_undegraded":"2024-03-06T07:24:54.253883+0000","last_fullsized":"2024-03-06T07:24:54.253883+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:28.907321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.253989+0000","last_change":"2024-03-06T07:24:53.247716+0000","last_active":"2024-03-06T07:24:54.253989+0000","last_peered":"2024-03-06T07:24:54.253989+0000","last_clean":"2024-03-06T07:24:54.253989+0000","last_became_active":"2024-03-06T07:24:53.247457+0000","last_became_peered":"2024-03-06T07:24:53.247457+0000","last_unstale":"2024-03-06T07:24:54.253989+0000","last_undegraded":"2024-03-06T07:24:54.253989+0000","last_fullsized":"2024-03-06T07:24:54.253989+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:32.092076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.438940+0000","last_change":"2024-03-06T07:24:52.252509+0000","last_active":"2024-03-06T07:24:54.438940+0000","last_peered":"2024-03-06T07:24:54.438940+0000","last_clean":"2024-03-06T07:24:54.438940+0000","last_became_active":"2024-03-06T07:24:52.252136+0000","last_became_peered":"2024-03-06T07:24:52.252136+0000","last_unstale":"2024-03-06T07:24:54.438940+0000","last_undegraded":"2024-03-06T07:24:54.438940+0000","last_fullsized":"2024-03-06T07:24:54.438940+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:57.778524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439487+0000","last_change":"2024-03-06T07:24:53.258341+0000","last_active":"2024-03-06T07:24:54.439487+0000","last_peered":"2024-03-06T07:24:54.439487+0000","last_clean":"2024-03-06T07:24:54.439487+0000","last_became_active":"2024-03-06T07:24:53.258058+0000","last_became_peered":"2024-03-06T07:24:53.258058+0000","last_unstale":"2024-03-06T07:24:54.439487+0000","last_undegraded":"2024-03-06T07:24:54.439487+0000","last_fullsized":"2024-03-06T07:24:54.439487+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:20.737007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249488+0000","last_change":"2024-03-06T07:24:52.254290+0000","last_active":"2024-03-06T07:24:54.249488+0000","last_peered":"2024-03-06T07:24:54.249488+0000","last_clean":"2024-03-06T07:24:54.249488+0000","last_became_active":"2024-03-06T07:24:52.253973+0000","last_became_peered":"2024-03-06T07:24:52.253973+0000","last_unstale":"2024-03-06T07:24:54.249488+0000","last_undegraded":"2024-03-06T07:24:54.249488+0000","last_fullsized":"2024-03-06T07:24:54.249488+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:42.304681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250674+0000","last_change":"2024-03-06T07:24:53.247738+0000","last_active":"2024-03-06T07:24:54.250674+0000","last_peered":"2024-03-06T07:24:54.250674+0000","last_clean":"2024-03-06T07:24:54.250674+0000","last_became_active":"2024-03-06T07:24:53.247476+0000","last_became_peered":"2024-03-06T07:24:53.247476+0000","last_unstale":"2024-03-06T07:24:54.250674+0000","last_undegraded":"2024-03-06T07:24:54.250674+0000","last_fullsized":"2024-03-06T07:24:54.250674+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:10.368888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439600+0000","last_change":"2024-03-06T07:24:52.250234+0000","last_active":"2024-03-06T07:24:54.439600+0000","last_peered":"2024-03-06T07:24:54.439600+0000","last_clean":"2024-03-06T07:24:54.439600+0000","last_became_active":"2024-03-06T07:24:52.249932+0000","last_became_peered":"2024-03-06T07:24:52.249932+0000","last_unstale":"2024-03-06T07:24:54.439600+0000","last_undegraded":"2024-03-06T07:24:54.439600+0000","last_fullsized":"2024-03-06T07:24:54.439600+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:17.992143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.438987+0000","last_change":"2024-03-06T07:24:53.256255+0000","last_active":"2024-03-06T07:24:54.438987+0000","last_peered":"2024-03-06T07:24:54.438987+0000","last_clean":"2024-03-06T07:24:54.438987+0000","last_became_active":"2024-03-06T07:24:53.254484+0000","last_became_peered":"2024-03-06T07:24:53.254484+0000","last_unstale":"2024-03-06T07:24:54.438987+0000","last_undegraded":"2024-03-06T07:24:54.438987+0000","last_fullsized":"2024-03-06T07:24:54.438987+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:39.350175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256446+0000","last_change":"2024-03-06T07:24:52.250483+0000","last_active":"2024-03-06T07:24:54.256446+0000","last_peered":"2024-03-06T07:24:54.256446+0000","last_clean":"2024-03-06T07:24:54.256446+0000","last_became_active":"2024-03-06T07:24:52.250201+0000","last_became_peered":"2024-03-06T07:24:52.250201+0000","last_unstale":"2024-03-06T07:24:54.256446+0000","last_undegraded":"2024-03-06T07:24:54.256446+0000","last_fullsized":"2024-03-06T07:24:54.256446+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:45.262772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249775+0000","last_change":"2024-03-06T07:24:53.248967+0000","last_active":"2024-03-06T07:24:54.249775+0000","last_peered":"2024-03-06T07:24:54.249775+0000","last_clean":"2024-03-06T07:24:54.249775+0000","last_became_active":"2024-03-06T07:24:53.248669+0000","last_became_peered":"2024-03-06T07:24:53.248669+0000","last_unstale":"2024-03-06T07:24:54.249775+0000","last_undegraded":"2024-03-06T07:24:54.249775+0000","last_fullsized":"2024-03-06T07:24:54.249775+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:00.496495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439552+0000","last_change":"2024-03-06T07:24:52.251851+0000","last_active":"2024-03-06T07:24:54.439552+0000","last_peered":"2024-03-06T07:24:54.439552+0000","last_clean":"2024-03-06T07:24:54.439552+0000","last_became_active":"2024-03-06T07:24:52.251549+0000","last_became_peered":"2024-03-06T07:24:52.251549+0000","last_unstale":"2024-03-06T07:24:54.439552+0000","last_undegraded":"2024-03-06T07:24:54.439552+0000","last_fullsized":"2024-03-06T07:24:54.439552+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:22.655955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439789+0000","last_change":"2024-03-06T07:24:53.255935+0000","last_active":"2024-03-06T07:24:54.439789+0000","last_peered":"2024-03-06T07:24:54.439789+0000","last_clean":"2024-03-06T07:24:54.439789+0000","last_became_active":"2024-03-06T07:24:53.255457+0000","last_became_peered":"2024-03-06T07:24:53.255457+0000","last_unstale":"2024-03-06T07:24:54.439789+0000","last_undegraded":"2024-03-06T07:24:54.439789+0000","last_fullsized":"2024-03-06T07:24:54.439789+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:41.534030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"120'3","reported_seq":17,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250865+0000","last_change":"2024-03-06T07:24:52.245152+0000","last_active":"2024-03-06T07:24:54.250865+0000","last_peered":"2024-03-06T07:24:54.250865+0000","last_clean":"2024-03-06T07:24:54.250865+0000","last_became_active":"2024-03-06T07:24:52.244695+0000","last_became_peered":"2024-03-06T07:24:52.244695+0000","last_unstale":"2024-03-06T07:24:54.250865+0000","last_undegraded":"2024-03-06T07:24:54.250865+0000","last_fullsized":"2024-03-06T07:24:54.250865+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:29.124565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439995+0000","last_change":"2024-03-06T07:24:53.249958+0000","last_active":"2024-03-06T07:24:54.439995+0000","last_peered":"2024-03-06T07:24:54.439995+0000","last_clean":"2024-03-06T07:24:54.439995+0000","last_became_active":"2024-03-06T07:24:53.249636+0000","last_became_peered":"2024-03-06T07:24:53.249636+0000","last_unstale":"2024-03-06T07:24:54.439995+0000","last_undegraded":"2024-03-06T07:24:54.439995+0000","last_fullsized":"2024-03-06T07:24:54.439995+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:47.432979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"120'6","reported_seq":20,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250846+0000","last_change":"2024-03-06T07:24:52.250220+0000","last_active":"2024-03-06T07:24:54.250846+0000","last_peered":"2024-03-06T07:24:54.250846+0000","last_clean":"2024-03-06T07:24:54.250846+0000","last_became_active":"2024-03-06T07:24:52.249566+0000","last_became_peered":"2024-03-06T07:24:52.249566+0000","last_unstale":"2024-03-06T07:24:54.250846+0000","last_undegraded":"2024-03-06T07:24:54.250846+0000","last_fullsized":"2024-03-06T07:24:54.250846+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:21.406441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.248748+0000","last_change":"2024-03-06T07:24:53.247650+0000","last_active":"2024-03-06T07:24:54.248748+0000","last_peered":"2024-03-06T07:24:54.248748+0000","last_clean":"2024-03-06T07:24:54.248748+0000","last_became_active":"2024-03-06T07:24:53.247367+0000","last_became_peered":"2024-03-06T07:24:53.247367+0000","last_unstale":"2024-03-06T07:24:54.248748+0000","last_undegraded":"2024-03-06T07:24:54.248748+0000","last_fullsized":"2024-03-06T07:24:54.248748+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:43.938094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439586+0000","last_change":"2024-03-06T07:24:52.251172+0000","last_active":"2024-03-06T07:24:54.439586+0000","last_peered":"2024-03-06T07:24:54.439586+0000","last_clean":"2024-03-06T07:24:54.439586+0000","last_became_active":"2024-03-06T07:24:52.250886+0000","last_became_peered":"2024-03-06T07:24:52.250886+0000","last_unstale":"2024-03-06T07:24:54.439586+0000","last_undegraded":"2024-03-06T07:24:54.439586+0000","last_fullsized":"2024-03-06T07:24:54.439586+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:44.062114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.251037+0000","last_change":"2024-03-06T07:24:53.252070+0000","last_active":"2024-03-06T07:24:54.251037+0000","last_peered":"2024-03-06T07:24:54.251037+0000","last_clean":"2024-03-06T07:24:54.251037+0000","last_became_active":"2024-03-06T07:24:53.251684+0000","last_became_peered":"2024-03-06T07:24:53.251684+0000","last_unstale":"2024-03-06T07:24:54.251037+0000","last_undegraded":"2024-03-06T07:24:54.251037+0000","last_fullsized":"2024-03-06T07:24:54.251037+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:33.368117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439606+0000","last_change":"2024-03-06T07:24:52.243716+0000","last_active":"2024-03-06T07:24:54.439606+0000","last_peered":"2024-03-06T07:24:54.439606+0000","last_clean":"2024-03-06T07:24:54.439606+0000","last_became_active":"2024-03-06T07:24:52.243443+0000","last_became_peered":"2024-03-06T07:24:52.243443+0000","last_unstale":"2024-03-06T07:24:54.439606+0000","last_undegraded":"2024-03-06T07:24:54.439606+0000","last_fullsized":"2024-03-06T07:24:54.439606+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:36.682515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439706+0000","last_change":"2024-03-06T07:24:53.265368+0000","last_active":"2024-03-06T07:24:54.439706+0000","last_peered":"2024-03-06T07:24:54.439706+0000","last_clean":"2024-03-06T07:24:54.439706+0000","last_became_active":"2024-03-06T07:24:53.265105+0000","last_became_peered":"2024-03-06T07:24:53.265105+0000","last_unstale":"2024-03-06T07:24:54.439706+0000","last_undegraded":"2024-03-06T07:24:54.439706+0000","last_fullsized":"2024-03-06T07:24:54.439706+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:51.633397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"120'8","reported_seq":22,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249614+0000","last_change":"2024-03-06T07:24:52.254923+0000","last_active":"2024-03-06T07:24:54.249614+0000","last_peered":"2024-03-06T07:24:54.249614+0000","last_clean":"2024-03-06T07:24:54.249614+0000","last_became_active":"2024-03-06T07:24:52.254647+0000","last_became_peered":"2024-03-06T07:24:52.254647+0000","last_unstale":"2024-03-06T07:24:54.249614+0000","last_undegraded":"2024-03-06T07:24:54.249614+0000","last_fullsized":"2024-03-06T07:24:54.249614+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:45.389867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439586+0000","last_change":"2024-03-06T07:24:53.256886+0000","last_active":"2024-03-06T07:24:54.439586+0000","last_peered":"2024-03-06T07:24:54.439586+0000","last_clean":"2024-03-06T07:24:54.439586+0000","last_became_active":"2024-03-06T07:24:53.256158+0000","last_became_peered":"2024-03-06T07:24:53.256158+0000","last_unstale":"2024-03-06T07:24:54.439586+0000","last_undegraded":"2024-03-06T07:24:54.439586+0000","last_fullsized":"2024-03-06T07:24:54.439586+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:25.241414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256481+0000","last_change":"2024-03-06T07:24:52.248044+0000","last_active":"2024-03-06T07:24:54.256481+0000","last_peered":"2024-03-06T07:24:54.256481+0000","last_clean":"2024-03-06T07:24:54.256481+0000","last_became_active":"2024-03-06T07:24:52.247701+0000","last_became_peered":"2024-03-06T07:24:52.247701+0000","last_unstale":"2024-03-06T07:24:54.256481+0000","last_undegraded":"2024-03-06T07:24:54.256481+0000","last_fullsized":"2024-03-06T07:24:54.256481+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:28.907321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250596+0000","last_change":"2024-03-06T07:24:53.254344+0000","last_active":"2024-03-06T07:24:54.250596+0000","last_peered":"2024-03-06T07:24:54.250596+0000","last_clean":"2024-03-06T07:24:54.250596+0000","last_became_active":"2024-03-06T07:24:53.253871+0000","last_became_peered":"2024-03-06T07:24:53.253871+0000","last_unstale":"2024-03-06T07:24:54.250596+0000","last_undegraded":"2024-03-06T07:24:54.250596+0000","last_fullsized":"2024-03-06T07:24:54.250596+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:09.269921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439966+0000","last_change":"2024-03-06T07:24:52.247010+0000","last_active":"2024-03-06T07:24:54.439966+0000","last_peered":"2024-03-06T07:24:54.439966+0000","last_clean":"2024-03-06T07:24:54.439966+0000","last_became_active":"2024-03-06T07:24:52.246730+0000","last_became_peered":"2024-03-06T07:24:52.246730+0000","last_unstale":"2024-03-06T07:24:54.439966+0000","last_undegraded":"2024-03-06T07:24:54.439966+0000","last_fullsized":"2024-03-06T07:24:54.439966+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:49.426429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.251040+0000","last_change":"2024-03-06T07:24:53.248482+0000","last_active":"2024-03-06T07:24:54.251040+0000","last_peered":"2024-03-06T07:24:54.251040+0000","last_clean":"2024-03-06T07:24:54.251040+0000","last_became_active":"2024-03-06T07:24:53.248185+0000","last_became_peered":"2024-03-06T07:24:53.248185+0000","last_unstale":"2024-03-06T07:24:54.251040+0000","last_undegraded":"2024-03-06T07:24:54.251040+0000","last_fullsized":"2024-03-06T07:24:54.251040+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:57.903166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250066+0000","last_change":"2024-03-06T07:24:52.253907+0000","last_active":"2024-03-06T07:24:54.250066+0000","last_peered":"2024-03-06T07:24:54.250066+0000","last_clean":"2024-03-06T07:24:54.250066+0000","last_became_active":"2024-03-06T07:24:52.253561+0000","last_became_peered":"2024-03-06T07:24:52.253561+0000","last_unstale":"2024-03-06T07:24:54.250066+0000","last_undegraded":"2024-03-06T07:24:54.250066+0000","last_fullsized":"2024-03-06T07:24:54.250066+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:53.820944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.248679+0000","last_change":"2024-03-06T07:24:53.253080+0000","last_active":"2024-03-06T07:24:54.248679+0000","last_peered":"2024-03-06T07:24:54.248679+0000","last_clean":"2024-03-06T07:24:54.248679+0000","last_became_active":"2024-03-06T07:24:53.252576+0000","last_became_peered":"2024-03-06T07:24:53.252576+0000","last_unstale":"2024-03-06T07:24:54.248679+0000","last_undegraded":"2024-03-06T07:24:54.248679+0000","last_fullsized":"2024-03-06T07:24:54.248679+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:37.547736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439471+0000","last_change":"2024-03-06T07:24:52.254150+0000","last_active":"2024-03-06T07:24:54.439471+0000","last_peered":"2024-03-06T07:24:54.439471+0000","last_clean":"2024-03-06T07:24:54.439471+0000","last_became_active":"2024-03-06T07:24:52.253882+0000","last_became_peered":"2024-03-06T07:24:52.253882+0000","last_unstale":"2024-03-06T07:24:54.439471+0000","last_undegraded":"2024-03-06T07:24:54.439471+0000","last_fullsized":"2024-03-06T07:24:54.439471+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:29.905192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439379+0000","last_change":"2024-03-06T07:24:53.264528+0000","last_active":"2024-03-06T07:24:54.439379+0000","last_peered":"2024-03-06T07:24:54.439379+0000","last_clean":"2024-03-06T07:24:54.439379+0000","last_became_active":"2024-03-06T07:24:53.264085+0000","last_became_peered":"2024-03-06T07:24:53.264085+0000","last_unstale":"2024-03-06T07:24:54.439379+0000","last_undegraded":"2024-03-06T07:24:54.439379+0000","last_fullsized":"2024-03-06T07:24:54.439379+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:12.103222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.253910+0000","last_change":"2024-03-06T07:24:52.253449+0000","last_active":"2024-03-06T07:24:54.253910+0000","last_peered":"2024-03-06T07:24:54.253910+0000","last_clean":"2024-03-06T07:24:54.253910+0000","last_became_active":"2024-03-06T07:24:52.253160+0000","last_became_peered":"2024-03-06T07:24:52.253160+0000","last_unstale":"2024-03-06T07:24:54.253910+0000","last_undegraded":"2024-03-06T07:24:54.253910+0000","last_fullsized":"2024-03-06T07:24:54.253910+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:24.236853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439443+0000","last_change":"2024-03-06T07:24:53.246000+0000","last_active":"2024-03-06T07:24:54.439443+0000","last_peered":"2024-03-06T07:24:54.439443+0000","last_clean":"2024-03-06T07:24:54.439443+0000","last_became_active":"2024-03-06T07:24:53.245699+0000","last_became_peered":"2024-03-06T07:24:53.245699+0000","last_unstale":"2024-03-06T07:24:54.439443+0000","last_undegraded":"2024-03-06T07:24:54.439443+0000","last_fullsized":"2024-03-06T07:24:54.439443+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:00.657287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439399+0000","last_change":"2024-03-06T07:24:52.249983+0000","last_active":"2024-03-06T07:24:54.439399+0000","last_peered":"2024-03-06T07:24:54.439399+0000","last_clean":"2024-03-06T07:24:54.439399+0000","last_became_active":"2024-03-06T07:24:52.249713+0000","last_became_peered":"2024-03-06T07:24:52.249713+0000","last_unstale":"2024-03-06T07:24:54.439399+0000","last_undegraded":"2024-03-06T07:24:54.439399+0000","last_fullsized":"2024-03-06T07:24:54.439399+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:37.394132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439583+0000","last_change":"2024-03-06T07:24:53.259552+0000","last_active":"2024-03-06T07:24:54.439583+0000","last_peered":"2024-03-06T07:24:54.439583+0000","last_clean":"2024-03-06T07:24:54.439583+0000","last_became_active":"2024-03-06T07:24:53.259269+0000","last_became_peered":"2024-03-06T07:24:53.259269+0000","last_unstale":"2024-03-06T07:24:54.439583+0000","last_undegraded":"2024-03-06T07:24:54.439583+0000","last_fullsized":"2024-03-06T07:24:54.439583+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:03.090749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250694+0000","last_change":"2024-03-06T07:24:52.245263+0000","last_active":"2024-03-06T07:24:54.250694+0000","last_peered":"2024-03-06T07:24:54.250694+0000","last_clean":"2024-03-06T07:24:54.250694+0000","last_became_active":"2024-03-06T07:24:52.244989+0000","last_became_peered":"2024-03-06T07:24:52.244989+0000","last_unstale":"2024-03-06T07:24:54.250694+0000","last_undegraded":"2024-03-06T07:24:54.250694+0000","last_fullsized":"2024-03-06T07:24:54.250694+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:22.118058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256412+0000","last_change":"2024-03-06T07:24:53.251996+0000","last_active":"2024-03-06T07:24:54.256412+0000","last_peered":"2024-03-06T07:24:54.256412+0000","last_clean":"2024-03-06T07:24:54.256412+0000","last_became_active":"2024-03-06T07:24:53.251734+0000","last_became_peered":"2024-03-06T07:24:53.251734+0000","last_unstale":"2024-03-06T07:24:54.256412+0000","last_undegraded":"2024-03-06T07:24:54.256412+0000","last_fullsized":"2024-03-06T07:24:54.256412+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:52.506392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249695+0000","last_change":"2024-03-06T07:24:52.248581+0000","last_active":"2024-03-06T07:24:54.249695+0000","last_peered":"2024-03-06T07:24:54.249695+0000","last_clean":"2024-03-06T07:24:54.249695+0000","last_became_active":"2024-03-06T07:24:52.248317+0000","last_became_peered":"2024-03-06T07:24:52.248317+0000","last_unstale":"2024-03-06T07:24:54.249695+0000","last_undegraded":"2024-03-06T07:24:54.249695+0000","last_fullsized":"2024-03-06T07:24:54.249695+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:40.529469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439147+0000","last_change":"2024-03-06T07:24:53.256765+0000","last_active":"2024-03-06T07:24:54.439147+0000","last_peered":"2024-03-06T07:24:54.439147+0000","last_clean":"2024-03-06T07:24:54.439147+0000","last_became_active":"2024-03-06T07:24:53.256476+0000","last_became_peered":"2024-03-06T07:24:53.256476+0000","last_unstale":"2024-03-06T07:24:54.439147+0000","last_undegraded":"2024-03-06T07:24:54.439147+0000","last_fullsized":"2024-03-06T07:24:54.439147+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:28.574847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439740+0000","last_change":"2024-03-06T07:24:52.248393+0000","last_active":"2024-03-06T07:24:54.439740+0000","last_peered":"2024-03-06T07:24:54.439740+0000","last_clean":"2024-03-06T07:24:54.439740+0000","last_became_active":"2024-03-06T07:24:52.248126+0000","last_became_peered":"2024-03-06T07:24:52.248126+0000","last_unstale":"2024-03-06T07:24:54.439740+0000","last_undegraded":"2024-03-06T07:24:54.439740+0000","last_fullsized":"2024-03-06T07:24:54.439740+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:38.691189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439598+0000","last_change":"2024-03-06T07:24:53.248008+0000","last_active":"2024-03-06T07:24:54.439598+0000","last_peered":"2024-03-06T07:24:54.439598+0000","last_clean":"2024-03-06T07:24:54.439598+0000","last_became_active":"2024-03-06T07:24:53.247727+0000","last_became_peered":"2024-03-06T07:24:53.247727+0000","last_unstale":"2024-03-06T07:24:54.439598+0000","last_undegraded":"2024-03-06T07:24:54.439598+0000","last_fullsized":"2024-03-06T07:24:54.439598+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:29.911882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"121'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.255540+0000","last_change":"2024-03-06T07:24:52.252134+0000","last_active":"2024-03-06T07:24:54.255540+0000","last_peered":"2024-03-06T07:24:54.255540+0000","last_clean":"2024-03-06T07:24:54.255540+0000","last_became_active":"2024-03-06T07:24:52.251774+0000","last_became_peered":"2024-03-06T07:24:52.251774+0000","last_unstale":"2024-03-06T07:24:54.255540+0000","last_undegraded":"2024-03-06T07:24:54.255540+0000","last_fullsized":"2024-03-06T07:24:54.255540+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:45.389867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250675+0000","last_change":"2024-03-06T07:24:53.253111+0000","last_active":"2024-03-06T07:24:54.250675+0000","last_peered":"2024-03-06T07:24:54.250675+0000","last_clean":"2024-03-06T07:24:54.250675+0000","last_became_active":"2024-03-06T07:24:53.252833+0000","last_became_peered":"2024-03-06T07:24:53.252833+0000","last_unstale":"2024-03-06T07:24:54.250675+0000","last_undegraded":"2024-03-06T07:24:54.250675+0000","last_fullsized":"2024-03-06T07:24:54.250675+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:08.595386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.254075+0000","last_change":"2024-03-06T07:24:52.250376+0000","last_active":"2024-03-06T07:24:54.254075+0000","last_peered":"2024-03-06T07:24:54.254075+0000","last_clean":"2024-03-06T07:24:54.254075+0000","last_became_active":"2024-03-06T07:24:52.250122+0000","last_became_peered":"2024-03-06T07:24:52.250122+0000","last_unstale":"2024-03-06T07:24:54.254075+0000","last_undegraded":"2024-03-06T07:24:54.254075+0000","last_fullsized":"2024-03-06T07:24:54.254075+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:40.529469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.248787+0000","last_change":"2024-03-06T07:24:53.246460+0000","last_active":"2024-03-06T07:24:54.248787+0000","last_peered":"2024-03-06T07:24:54.248787+0000","last_clean":"2024-03-06T07:24:54.248787+0000","last_became_active":"2024-03-06T07:24:53.246157+0000","last_became_peered":"2024-03-06T07:24:53.246157+0000","last_unstale":"2024-03-06T07:24:54.248787+0000","last_undegraded":"2024-03-06T07:24:54.248787+0000","last_fullsized":"2024-03-06T07:24:54.248787+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:01.649291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250030+0000","last_change":"2024-03-06T07:24:52.253206+0000","last_active":"2024-03-06T07:24:54.250030+0000","last_peered":"2024-03-06T07:24:54.250030+0000","last_clean":"2024-03-06T07:24:54.250030+0000","last_became_active":"2024-03-06T07:24:52.252885+0000","last_became_peered":"2024-03-06T07:24:52.252885+0000","last_unstale":"2024-03-06T07:24:54.250030+0000","last_undegraded":"2024-03-06T07:24:54.250030+0000","last_fullsized":"2024-03-06T07:24:54.250030+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:47.649285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249941+0000","last_change":"2024-03-06T07:24:53.251552+0000","last_active":"2024-03-06T07:24:54.249941+0000","last_peered":"2024-03-06T07:24:54.249941+0000","last_clean":"2024-03-06T07:24:54.249941+0000","last_became_active":"2024-03-06T07:24:53.251275+0000","last_became_peered":"2024-03-06T07:24:53.251275+0000","last_unstale":"2024-03-06T07:24:54.249941+0000","last_undegraded":"2024-03-06T07:24:54.249941+0000","last_fullsized":"2024-03-06T07:24:54.249941+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:58.783086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256234+0000","last_change":"2024-03-06T07:24:52.246330+0000","last_active":"2024-03-06T07:24:54.256234+0000","last_peered":"2024-03-06T07:24:54.256234+0000","last_clean":"2024-03-06T07:24:54.256234+0000","last_became_active":"2024-03-06T07:24:52.246067+0000","last_became_peered":"2024-03-06T07:24:52.246067+0000","last_unstale":"2024-03-06T07:24:54.256234+0000","last_undegraded":"2024-03-06T07:24:54.256234+0000","last_fullsized":"2024-03-06T07:24:54.256234+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:37.112908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250075+0000","last_change":"2024-03-06T07:24:53.252967+0000","last_active":"2024-03-06T07:24:54.250075+0000","last_peered":"2024-03-06T07:24:54.250075+0000","last_clean":"2024-03-06T07:24:54.250075+0000","last_became_active":"2024-03-06T07:24:53.252675+0000","last_became_peered":"2024-03-06T07:24:53.252675+0000","last_unstale":"2024-03-06T07:24:54.250075+0000","last_undegraded":"2024-03-06T07:24:54.250075+0000","last_fullsized":"2024-03-06T07:24:54.250075+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:16.327805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250975+0000","last_change":"2024-03-06T07:24:52.251188+0000","last_active":"2024-03-06T07:24:54.250975+0000","last_peered":"2024-03-06T07:24:54.250975+0000","last_clean":"2024-03-06T07:24:54.250975+0000","last_became_active":"2024-03-06T07:24:52.250925+0000","last_became_peered":"2024-03-06T07:24:52.250925+0000","last_unstale":"2024-03-06T07:24:54.250975+0000","last_undegraded":"2024-03-06T07:24:54.250975+0000","last_fullsized":"2024-03-06T07:24:54.250975+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:42.209216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249709+0000","last_change":"2024-03-06T07:24:53.252096+0000","last_active":"2024-03-06T07:24:54.249709+0000","last_peered":"2024-03-06T07:24:54.249709+0000","last_clean":"2024-03-06T07:24:54.249709+0000","last_became_active":"2024-03-06T07:24:53.251873+0000","last_became_peered":"2024-03-06T07:24:53.251873+0000","last_unstale":"2024-03-06T07:24:54.249709+0000","last_undegraded":"2024-03-06T07:24:54.249709+0000","last_fullsized":"2024-03-06T07:24:54.249709+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:06.485394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249677+0000","last_change":"2024-03-06T07:24:52.248523+0000","last_active":"2024-03-06T07:24:54.249677+0000","last_peered":"2024-03-06T07:24:54.249677+0000","last_clean":"2024-03-06T07:24:54.249677+0000","last_became_active":"2024-03-06T07:24:52.248248+0000","last_became_peered":"2024-03-06T07:24:52.248248+0000","last_unstale":"2024-03-06T07:24:54.249677+0000","last_undegraded":"2024-03-06T07:24:54.249677+0000","last_fullsized":"2024-03-06T07:24:54.249677+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:46.428418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250819+0000","last_change":"2024-03-06T07:24:53.253425+0000","last_active":"2024-03-06T07:24:54.250819+0000","last_peered":"2024-03-06T07:24:54.250819+0000","last_clean":"2024-03-06T07:24:54.250819+0000","last_became_active":"2024-03-06T07:24:53.252966+0000","last_became_peered":"2024-03-06T07:24:53.252966+0000","last_unstale":"2024-03-06T07:24:54.250819+0000","last_undegraded":"2024-03-06T07:24:54.250819+0000","last_fullsized":"2024-03-06T07:24:54.250819+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:04.537106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439091+0000","last_change":"2024-03-06T07:24:52.253847+0000","last_active":"2024-03-06T07:24:54.439091+0000","last_peered":"2024-03-06T07:24:54.439091+0000","last_clean":"2024-03-06T07:24:54.439091+0000","last_became_active":"2024-03-06T07:24:52.253549+0000","last_became_peered":"2024-03-06T07:24:52.253549+0000","last_unstale":"2024-03-06T07:24:54.439091+0000","last_undegraded":"2024-03-06T07:24:54.439091+0000","last_fullsized":"2024-03-06T07:24:54.439091+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:22.374141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.440119+0000","last_change":"2024-03-06T07:24:53.256391+0000","last_active":"2024-03-06T07:24:54.440119+0000","last_peered":"2024-03-06T07:24:54.440119+0000","last_clean":"2024-03-06T07:24:54.440119+0000","last_became_active":"2024-03-06T07:24:53.256049+0000","last_became_peered":"2024-03-06T07:24:53.256049+0000","last_unstale":"2024-03-06T07:24:54.440119+0000","last_undegraded":"2024-03-06T07:24:54.440119+0000","last_fullsized":"2024-03-06T07:24:54.440119+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:27.838125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.251120+0000","last_change":"2024-03-06T07:24:52.246324+0000","last_active":"2024-03-06T07:24:54.251120+0000","last_peered":"2024-03-06T07:24:54.251120+0000","last_clean":"2024-03-06T07:24:54.251120+0000","last_became_active":"2024-03-06T07:24:52.245836+0000","last_became_peered":"2024-03-06T07:24:52.245836+0000","last_unstale":"2024-03-06T07:24:54.251120+0000","last_undegraded":"2024-03-06T07:24:54.251120+0000","last_fullsized":"2024-03-06T07:24:54.251120+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:31.087515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":364,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250162+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:24:54.250162+0000","last_peered":"2024-03-06T07:24:54.250162+0000","last_clean":"2024-03-06T07:24:54.250162+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:24:54.250162+0000","last_undegraded":"2024-03-06T07:24:54.250162+0000","last_fullsized":"2024-03-06T07:24:54.250162+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32728,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339112,"statfs":{"total":96636764160,"available":96603250688,"internally_reserved":0,"allocated":1585152,"data_stored":1316164,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32656,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339184,"statfs":{"total":96636764160,"available":96603324416,"internally_reserved":0,"allocated":1511424,"data_stored":724766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32652,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339188,"statfs":{"total":96636764160,"available":96603328512,"internally_reserved":0,"allocated":1507328,"data_stored":724116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1515520,"data_stored":730786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32680,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339160,"statfs":{"total":96636764160,"available":96603299840,"internally_reserved":0,"allocated":1531904,"data_stored":732468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1511424,"data_stored":724710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32652,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339188,"statfs":{"total":96636764160,"available":96603328512,"internally_reserved":0,"allocated":1507328,"data_stored":724164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32736,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339104,"statfs":{"total":96636764160,"available":96603242496,"internally_reserved":0,"allocated":1589248,"data_stored":1321210,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:57.380 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:57.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.515+0000 7f3e41d0c640 1 Processor -- start 2024-03-06T07:24:57.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.515+0000 7f3e41d0c640 1 -- start start 2024-03-06T07:24:57.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.516+0000 7f3e41d0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e3c152650 0x7f3e3c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.516+0000 7f3e41d0c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3e3c0c6b10 0x7f3e3c158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.516+0000 7f3e41d0c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e3c0c2fc0 con 0x7f3e3c0bf9d0 2024-03-06T07:24:57.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.516+0000 7f3e41d0c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e3c155800 con 0x7f3e3c0c6b10 2024-03-06T07:24:57.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.516+0000 7f3e41d0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c0c6690 con 0x7f3e3c152650 2024-03-06T07:24:57.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.516+0000 7f3e3affd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3e3c0c6b10 0x7f3e3c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.516+0000 7f3e3bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3e3c0bf9d0 0x7f3e3c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41638/0 (socket says 172.21.15.1:41638) 2024-03-06T07:24:57.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.516+0000 7f3e3bfff640 1 -- 172.21.15.1:0/3469148857 learned_addr learned my addr 172.21.15.1:0/3469148857 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:57.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.516+0000 7f3e3affd640 1 -- 172.21.15.1:0/3469148857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e3c152650 msgr2=0x7f3e3c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.516+0000 7f3e3affd640 1 --2- 172.21.15.1:0/3469148857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e3c152650 0x7f3e3c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.516+0000 7f3e3affd640 1 -- 172.21.15.1:0/3469148857 >> v1:172.21.15.1:6789/0 conn(0x7f3e3c0bf9d0 legacy=0x7f3e3c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.516+0000 7f3e3affd640 1 -- 172.21.15.1:0/3469148857 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c0c7280 con 0x7f3e3c0c6b10 2024-03-06T07:24:57.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.516+0000 7f3e3a7fc640 1 -- 172.21.15.1:0/3469148857 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841877335 0 0) 0x7f3e3c0c2fc0 con 0x7f3e3c0bf9d0 2024-03-06T07:24:57.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.517+0000 7f3e3affd640 1 --2- 172.21.15.1:0/3469148857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e3c0c6b10 0x7f3e3c158690 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f3e240097a0 tx=0x7f3e2402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae9af8efc68396 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.517+0000 7f3e3a7fc640 1 -- 172.21.15.1:0/3469148857 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e240315a0 con 0x7f3e3c0c6b10 2024-03-06T07:24:57.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.517+0000 7f3e3a7fc640 1 -- 172.21.15.1:0/3469148857 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e24037070 con 0x7f3e3c0c6b10 2024-03-06T07:24:57.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.517+0000 7f3e3a7fc640 1 -- 172.21.15.1:0/3469148857 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e24031960 con 0x7f3e3c0c6b10 2024-03-06T07:24:57.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.518+0000 7f3e41d0c640 1 -- 172.21.15.1:0/3469148857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e3c0c6b10 msgr2=0x7f3e3c158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.518+0000 7f3e41d0c640 1 --2- 172.21.15.1:0/3469148857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e3c0c6b10 0x7f3e3c158690 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f3e240097a0 tx=0x7f3e2402b840 comp rx=0 tx=0).stop 2024-03-06T07:24:57.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.518+0000 7f3e41d0c640 1 -- 172.21.15.1:0/3469148857 shutdown_connections 2024-03-06T07:24:57.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.518+0000 7f3e41d0c640 1 --2- 172.21.15.1:0/3469148857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e3c0c6b10 0x7f3e3c158690 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.518+0000 7f3e41d0c640 1 --2- 172.21.15.1:0/3469148857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e3c152650 0x7f3e3c155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.518+0000 7f3e41d0c640 1 -- 172.21.15.1:0/3469148857 >> 172.21.15.1:0/3469148857 conn(0x7f3e3c066ee0 msgr2=0x7f3e3c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.518+0000 7f3e41d0c640 1 -- 172.21.15.1:0/3469148857 shutdown_connections 2024-03-06T07:24:57.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.518+0000 7f3e41d0c640 1 -- 172.21.15.1:0/3469148857 wait complete. 2024-03-06T07:24:57.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.519+0000 7f3e41d0c640 1 Processor -- start 2024-03-06T07:24:57.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.519+0000 7f3e41d0c640 1 -- start start 2024-03-06T07:24:57.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.520+0000 7f3e41d0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e3c0bf9d0 0x7f3e3c1eb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.520+0000 7f3e41d0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e3c152650 0x7f3e3c1541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.520+0000 7f3e41d0c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e3c156490 0x7f3e3c14c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.520+0000 7f3e41d0c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c155490 con 0x7f3e3c156490 2024-03-06T07:24:57.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.520+0000 7f3e41d0c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c1551b0 con 0x7f3e3c152650 2024-03-06T07:24:57.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.520+0000 7f3e41d0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c152a30 con 0x7f3e3c0bf9d0 2024-03-06T07:24:57.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.520+0000 7f3e3bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e3c156490 0x7f3e3c14c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.520+0000 7f3e3bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e3c156490 0x7f3e3c14c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36894/0 (socket says 172.21.15.1:36894) 2024-03-06T07:24:57.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.520+0000 7f3e3bfff640 1 -- 172.21.15.1:0/3321682610 learned_addr learned my addr 172.21.15.1:0/3321682610 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:57.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.520+0000 7f3e3affd640 1 --2- 172.21.15.1:0/3321682610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e3c152650 0x7f3e3c1541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.520+0000 7f3e3b7fe640 1 --2- 172.21.15.1:0/3321682610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e3c0bf9d0 0x7f3e3c1eb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.520+0000 7f3e3bfff640 1 -- 172.21.15.1:0/3321682610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e3c0bf9d0 msgr2=0x7f3e3c1eb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.520+0000 7f3e3bfff640 1 --2- 172.21.15.1:0/3321682610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e3c0bf9d0 0x7f3e3c1eb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.520+0000 7f3e3bfff640 1 -- 172.21.15.1:0/3321682610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e3c152650 msgr2=0x7f3e3c1541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.520+0000 7f3e3bfff640 1 --2- 172.21.15.1:0/3321682610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e3c152650 0x7f3e3c1541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.520+0000 7f3e3bfff640 1 -- 172.21.15.1:0/3321682610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c1ebd20 con 0x7f3e3c156490 2024-03-06T07:24:57.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.520+0000 7f3e3b7fe640 1 --2- 172.21.15.1:0/3321682610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e3c0bf9d0 0x7f3e3c1eb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:57.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.521+0000 7f3e3bfff640 1 --2- 172.21.15.1:0/3321682610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e3c156490 0x7f3e3c14c7e0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f3e3400c700 tx=0x7f3e3400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.521+0000 7f3e33fff640 1 -- 172.21.15.1:0/3321682610 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e34018070 con 0x7f3e3c156490 2024-03-06T07:24:57.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.521+0000 7f3e33fff640 1 -- 172.21.15.1:0/3321682610 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3400ce10 con 0x7f3e3c156490 2024-03-06T07:24:57.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.521+0000 7f3e41d0c640 1 -- 172.21.15.1:0/3321682610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3c14d500 con 0x7f3e3c156490 2024-03-06T07:24:57.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.521+0000 7f3e33fff640 1 -- 172.21.15.1:0/3321682610 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3400a860 con 0x7f3e3c156490 2024-03-06T07:24:57.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.521+0000 7f3e41d0c640 1 -- 172.21.15.1:0/3321682610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3c14dac0 con 0x7f3e3c156490 2024-03-06T07:24:57.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.522+0000 7f3e41d0c640 1 -- 172.21.15.1:0/3321682610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3c067fd0 con 0x7f3e3c156490 2024-03-06T07:24:57.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.523+0000 7f3e33fff640 1 -- 172.21.15.1:0/3321682610 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e3401b070 con 0x7f3e3c156490 2024-03-06T07:24:57.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.523+0000 7f3e33fff640 1 --2- 172.21.15.1:0/3321682610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3e1c06b4d0 0x7f3e1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.523+0000 7f3e33fff640 1 -- 172.21.15.1:0/3321682610 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3e340898e0 con 0x7f3e3c156490 2024-03-06T07:24:57.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.523+0000 7f3e3b7fe640 1 --2- 172.21.15.1:0/3321682610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3e1c06b4d0 0x7f3e1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.524+0000 7f3e3b7fe640 1 --2- 172.21.15.1:0/3321682610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3e1c06b4d0 0x7f3e1c06d990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c009930 tx=0x7f3e2c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.527+0000 7f3e33fff640 1 -- 172.21.15.1:0/3321682610 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3e34057f80 con 0x7f3e3c156490 2024-03-06T07:24:57.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.708+0000 7f3e41d0c640 1 -- 172.21.15.1:0/3321682610 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3e3c0b06f0 con 0x7f3e1c06b4d0 2024-03-06T07:24:57.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.714+0000 7f3e33fff640 1 -- 172.21.15.1:0/3321682610 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252560 (secure 0 0 0) 0x7f3e3c0b06f0 con 0x7f3e1c06b4d0 2024-03-06T07:24:57.719 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:57.720 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:24:57.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.717+0000 7f3e41d0c640 1 -- 172.21.15.1:0/3321682610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3e1c06b4d0 msgr2=0x7f3e1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.717+0000 7f3e41d0c640 1 --2- 172.21.15.1:0/3321682610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3e1c06b4d0 0x7f3e1c06d990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c009930 tx=0x7f3e2c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:57.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.717+0000 7f3e41d0c640 1 -- 172.21.15.1:0/3321682610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e3c156490 msgr2=0x7f3e3c14c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.717+0000 7f3e41d0c640 1 --2- 172.21.15.1:0/3321682610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e3c156490 0x7f3e3c14c7e0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f3e3400c700 tx=0x7f3e3400cbc0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.717+0000 7f3e41d0c640 1 -- 172.21.15.1:0/3321682610 shutdown_connections 2024-03-06T07:24:57.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.717+0000 7f3e41d0c640 1 --2- 172.21.15.1:0/3321682610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3e1c06b4d0 0x7f3e1c06d990 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.717+0000 7f3e41d0c640 1 --2- 172.21.15.1:0/3321682610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e3c156490 0x7f3e3c14c7e0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.717+0000 7f3e41d0c640 1 --2- 172.21.15.1:0/3321682610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e3c152650 0x7f3e3c1541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.717+0000 7f3e41d0c640 1 --2- 172.21.15.1:0/3321682610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e3c0bf9d0 0x7f3e3c1eb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.717+0000 7f3e41d0c640 1 -- 172.21.15.1:0/3321682610 >> 172.21.15.1:0/3321682610 conn(0x7f3e3c066ee0 msgr2=0x7f3e3c0c5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.717+0000 7f3e41d0c640 1 -- 172.21.15.1:0/3321682610 shutdown_connections 2024-03-06T07:24:57.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.717+0000 7f3e41d0c640 1 -- 172.21.15.1:0/3321682610 wait complete. 2024-03-06T07:24:57.730 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-03-06T07:24:56.752831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261432,"kb_used_data":11972,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713288,"statfs":{"total":773094113280,"available":772826406912,"internally_reserved":0,"allocated":12259328,"data_stored":6998384,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.521375"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439736+0000","last_change":"2024-03-06T07:24:53.248053+0000","last_active":"2024-03-06T07:24:54.439736+0000","last_peered":"2024-03-06T07:24:54.439736+0000","last_clean":"2024-03-06T07:24:54.439736+0000","last_became_active":"2024-03-06T07:24:53.247769+0000","last_became_peered":"2024-03-06T07:24:53.247769+0000","last_unstale":"2024-03-06T07:24:54.439736+0000","last_undegraded":"2024-03-06T07:24:54.439736+0000","last_fullsized":"2024-03-06T07:24:54.439736+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:56.349947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256418+0000","last_change":"2024-03-06T07:24:53.258292+0000","last_active":"2024-03-06T07:24:54.256418+0000","last_peered":"2024-03-06T07:24:54.256418+0000","last_clean":"2024-03-06T07:24:54.256418+0000","last_became_active":"2024-03-06T07:24:53.257461+0000","last_became_peered":"2024-03-06T07:24:53.257461+0000","last_unstale":"2024-03-06T07:24:54.256418+0000","last_undegraded":"2024-03-06T07:24:54.256418+0000","last_fullsized":"2024-03-06T07:24:54.256418+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:03.358652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439938+0000","last_change":"2024-03-06T07:24:53.254981+0000","last_active":"2024-03-06T07:24:54.439938+0000","last_peered":"2024-03-06T07:24:54.439938+0000","last_clean":"2024-03-06T07:24:54.439938+0000","last_became_active":"2024-03-06T07:24:53.254714+0000","last_became_peered":"2024-03-06T07:24:53.254714+0000","last_unstale":"2024-03-06T07:24:54.439938+0000","last_undegraded":"2024-03-06T07:24:54.439938+0000","last_fullsized":"2024-03-06T07:24:54.439938+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:54.825505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256238+0000","last_change":"2024-03-06T07:24:53.252463+0000","last_active":"2024-03-06T07:24:54.256238+0000","last_peered":"2024-03-06T07:24:54.256238+0000","last_clean":"2024-03-06T07:24:54.256238+0000","last_became_active":"2024-03-06T07:24:53.252183+0000","last_became_peered":"2024-03-06T07:24:53.252183+0000","last_unstale":"2024-03-06T07:24:54.256238+0000","last_undegraded":"2024-03-06T07:24:54.256238+0000","last_fullsized":"2024-03-06T07:24:54.256238+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:23.122619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439024+0000","last_change":"2024-03-06T07:24:53.259191+0000","last_active":"2024-03-06T07:24:54.439024+0000","last_peered":"2024-03-06T07:24:54.439024+0000","last_clean":"2024-03-06T07:24:54.439024+0000","last_became_active":"2024-03-06T07:24:53.258904+0000","last_became_peered":"2024-03-06T07:24:53.258904+0000","last_unstale":"2024-03-06T07:24:54.439024+0000","last_undegraded":"2024-03-06T07:24:54.439024+0000","last_fullsized":"2024-03-06T07:24:54.439024+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:54.330802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249639+0000","last_change":"2024-03-06T07:24:53.255509+0000","last_active":"2024-03-06T07:24:54.249639+0000","last_peered":"2024-03-06T07:24:54.249639+0000","last_clean":"2024-03-06T07:24:54.249639+0000","last_became_active":"2024-03-06T07:24:53.255228+0000","last_became_peered":"2024-03-06T07:24:53.255228+0000","last_unstale":"2024-03-06T07:24:54.249639+0000","last_undegraded":"2024-03-06T07:24:54.249639+0000","last_fullsized":"2024-03-06T07:24:54.249639+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:28.333906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250340+0000","last_change":"2024-03-06T07:24:53.254560+0000","last_active":"2024-03-06T07:24:54.250340+0000","last_peered":"2024-03-06T07:24:54.250340+0000","last_clean":"2024-03-06T07:24:54.250340+0000","last_became_active":"2024-03-06T07:24:53.250079+0000","last_became_peered":"2024-03-06T07:24:53.250079+0000","last_unstale":"2024-03-06T07:24:54.250340+0000","last_undegraded":"2024-03-06T07:24:54.250340+0000","last_fullsized":"2024-03-06T07:24:54.250340+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:16.327805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.253727+0000","last_change":"2024-03-06T07:24:53.248358+0000","last_active":"2024-03-06T07:24:54.253727+0000","last_peered":"2024-03-06T07:24:54.253727+0000","last_clean":"2024-03-06T07:24:54.253727+0000","last_became_active":"2024-03-06T07:24:53.248089+0000","last_became_peered":"2024-03-06T07:24:53.248089+0000","last_unstale":"2024-03-06T07:24:54.253727+0000","last_undegraded":"2024-03-06T07:24:54.253727+0000","last_fullsized":"2024-03-06T07:24:54.253727+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:27.838125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249902+0000","last_change":"2024-03-06T07:24:53.253735+0000","last_active":"2024-03-06T07:24:54.249902+0000","last_peered":"2024-03-06T07:24:54.249902+0000","last_clean":"2024-03-06T07:24:54.249902+0000","last_became_active":"2024-03-06T07:24:53.253463+0000","last_became_peered":"2024-03-06T07:24:53.253463+0000","last_unstale":"2024-03-06T07:24:54.249902+0000","last_undegraded":"2024-03-06T07:24:54.249902+0000","last_fullsized":"2024-03-06T07:24:54.249902+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:51.891213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.248811+0000","last_change":"2024-03-06T07:24:53.247511+0000","last_active":"2024-03-06T07:24:54.248811+0000","last_peered":"2024-03-06T07:24:54.248811+0000","last_clean":"2024-03-06T07:24:54.248811+0000","last_became_active":"2024-03-06T07:24:53.247223+0000","last_became_peered":"2024-03-06T07:24:53.247223+0000","last_unstale":"2024-03-06T07:24:54.248811+0000","last_undegraded":"2024-03-06T07:24:54.248811+0000","last_fullsized":"2024-03-06T07:24:54.248811+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:30.974392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256225+0000","last_change":"2024-03-06T07:24:53.253451+0000","last_active":"2024-03-06T07:24:54.256225+0000","last_peered":"2024-03-06T07:24:54.256225+0000","last_clean":"2024-03-06T07:24:54.256225+0000","last_became_active":"2024-03-06T07:24:53.253198+0000","last_became_peered":"2024-03-06T07:24:53.253198+0000","last_unstale":"2024-03-06T07:24:54.256225+0000","last_undegraded":"2024-03-06T07:24:54.256225+0000","last_fullsized":"2024-03-06T07:24:54.256225+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:43.213777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439903+0000","last_change":"2024-03-06T07:24:53.255928+0000","last_active":"2024-03-06T07:24:54.439903+0000","last_peered":"2024-03-06T07:24:54.439903+0000","last_clean":"2024-03-06T07:24:54.439903+0000","last_became_active":"2024-03-06T07:24:53.255604+0000","last_became_peered":"2024-03-06T07:24:53.255604+0000","last_unstale":"2024-03-06T07:24:54.439903+0000","last_undegraded":"2024-03-06T07:24:54.439903+0000","last_fullsized":"2024-03-06T07:24:54.439903+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:31.410528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249817+0000","last_change":"2024-03-06T07:24:53.255551+0000","last_active":"2024-03-06T07:24:54.249817+0000","last_peered":"2024-03-06T07:24:54.249817+0000","last_clean":"2024-03-06T07:24:54.249817+0000","last_became_active":"2024-03-06T07:24:53.255255+0000","last_became_peered":"2024-03-06T07:24:53.255255+0000","last_unstale":"2024-03-06T07:24:54.249817+0000","last_undegraded":"2024-03-06T07:24:54.249817+0000","last_fullsized":"2024-03-06T07:24:54.249817+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:36.778107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439307+0000","last_change":"2024-03-06T07:24:53.245269+0000","last_active":"2024-03-06T07:24:54.439307+0000","last_peered":"2024-03-06T07:24:54.439307+0000","last_clean":"2024-03-06T07:24:54.439307+0000","last_became_active":"2024-03-06T07:24:53.244958+0000","last_became_peered":"2024-03-06T07:24:53.244958+0000","last_unstale":"2024-03-06T07:24:54.439307+0000","last_undegraded":"2024-03-06T07:24:54.439307+0000","last_fullsized":"2024-03-06T07:24:54.439307+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:10.294078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439122+0000","last_change":"2024-03-06T07:24:53.246040+0000","last_active":"2024-03-06T07:24:54.439122+0000","last_peered":"2024-03-06T07:24:54.439122+0000","last_clean":"2024-03-06T07:24:54.439122+0000","last_became_active":"2024-03-06T07:24:53.245749+0000","last_became_peered":"2024-03-06T07:24:53.245749+0000","last_unstale":"2024-03-06T07:24:54.439122+0000","last_undegraded":"2024-03-06T07:24:54.439122+0000","last_fullsized":"2024-03-06T07:24:54.439122+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:52.353345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249503+0000","last_change":"2024-03-06T07:24:53.258563+0000","last_active":"2024-03-06T07:24:54.249503+0000","last_peered":"2024-03-06T07:24:54.249503+0000","last_clean":"2024-03-06T07:24:54.249503+0000","last_became_active":"2024-03-06T07:24:53.257821+0000","last_became_peered":"2024-03-06T07:24:53.257821+0000","last_unstale":"2024-03-06T07:24:54.249503+0000","last_undegraded":"2024-03-06T07:24:54.249503+0000","last_fullsized":"2024-03-06T07:24:54.249503+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:05.761133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249996+0000","last_change":"2024-03-06T07:24:53.256941+0000","last_active":"2024-03-06T07:24:54.249996+0000","last_peered":"2024-03-06T07:24:54.249996+0000","last_clean":"2024-03-06T07:24:54.249996+0000","last_became_active":"2024-03-06T07:24:53.256676+0000","last_became_peered":"2024-03-06T07:24:53.256676+0000","last_unstale":"2024-03-06T07:24:54.249996+0000","last_undegraded":"2024-03-06T07:24:54.249996+0000","last_fullsized":"2024-03-06T07:24:54.249996+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:23.378702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439390+0000","last_change":"2024-03-06T07:24:53.260235+0000","last_active":"2024-03-06T07:24:54.439390+0000","last_peered":"2024-03-06T07:24:54.439390+0000","last_clean":"2024-03-06T07:24:54.439390+0000","last_became_active":"2024-03-06T07:24:53.259779+0000","last_became_peered":"2024-03-06T07:24:53.259779+0000","last_unstale":"2024-03-06T07:24:54.439390+0000","last_undegraded":"2024-03-06T07:24:54.439390+0000","last_fullsized":"2024-03-06T07:24:54.439390+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:29.942697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250588+0000","last_change":"2024-03-06T07:24:53.257044+0000","last_active":"2024-03-06T07:24:54.250588+0000","last_peered":"2024-03-06T07:24:54.250588+0000","last_clean":"2024-03-06T07:24:54.250588+0000","last_became_active":"2024-03-06T07:24:53.256788+0000","last_became_peered":"2024-03-06T07:24:53.256788+0000","last_unstale":"2024-03-06T07:24:54.250588+0000","last_undegraded":"2024-03-06T07:24:54.250588+0000","last_fullsized":"2024-03-06T07:24:54.250588+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:30.909753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439660+0000","last_change":"2024-03-06T07:24:53.258248+0000","last_active":"2024-03-06T07:24:54.439660+0000","last_peered":"2024-03-06T07:24:54.439660+0000","last_clean":"2024-03-06T07:24:54.439660+0000","last_became_active":"2024-03-06T07:24:53.257968+0000","last_became_peered":"2024-03-06T07:24:53.257968+0000","last_unstale":"2024-03-06T07:24:54.439660+0000","last_undegraded":"2024-03-06T07:24:54.439660+0000","last_fullsized":"2024-03-06T07:24:54.439660+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:09.808543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256195+0000","last_change":"2024-03-06T07:24:53.263730+0000","last_active":"2024-03-06T07:24:54.256195+0000","last_peered":"2024-03-06T07:24:54.256195+0000","last_clean":"2024-03-06T07:24:54.256195+0000","last_became_active":"2024-03-06T07:24:53.263451+0000","last_became_peered":"2024-03-06T07:24:53.263451+0000","last_unstale":"2024-03-06T07:24:54.256195+0000","last_undegraded":"2024-03-06T07:24:54.256195+0000","last_fullsized":"2024-03-06T07:24:54.256195+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:44.388371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439787+0000","last_change":"2024-03-06T07:24:53.256663+0000","last_active":"2024-03-06T07:24:54.439787+0000","last_peered":"2024-03-06T07:24:54.439787+0000","last_clean":"2024-03-06T07:24:54.439787+0000","last_became_active":"2024-03-06T07:24:53.256366+0000","last_became_peered":"2024-03-06T07:24:53.256366+0000","last_unstale":"2024-03-06T07:24:54.439787+0000","last_undegraded":"2024-03-06T07:24:54.439787+0000","last_fullsized":"2024-03-06T07:24:54.439787+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:54.593816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256136+0000","last_change":"2024-03-06T07:24:53.251100+0000","last_active":"2024-03-06T07:24:54.256136+0000","last_peered":"2024-03-06T07:24:54.256136+0000","last_clean":"2024-03-06T07:24:54.256136+0000","last_became_active":"2024-03-06T07:24:53.250823+0000","last_became_peered":"2024-03-06T07:24:53.250823+0000","last_unstale":"2024-03-06T07:24:54.256136+0000","last_undegraded":"2024-03-06T07:24:54.256136+0000","last_fullsized":"2024-03-06T07:24:54.256136+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:27.838125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249584+0000","last_change":"2024-03-06T07:24:53.254207+0000","last_active":"2024-03-06T07:24:54.249584+0000","last_peered":"2024-03-06T07:24:54.249584+0000","last_clean":"2024-03-06T07:24:54.249584+0000","last_became_active":"2024-03-06T07:24:53.253888+0000","last_became_peered":"2024-03-06T07:24:53.253888+0000","last_unstale":"2024-03-06T07:24:54.249584+0000","last_undegraded":"2024-03-06T07:24:54.249584+0000","last_fullsized":"2024-03-06T07:24:54.249584+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:07.871828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.440109+0000","last_change":"2024-03-06T07:24:53.259683+0000","last_active":"2024-03-06T07:24:54.440109+0000","last_peered":"2024-03-06T07:24:54.440109+0000","last_clean":"2024-03-06T07:24:54.440109+0000","last_became_active":"2024-03-06T07:24:53.259095+0000","last_became_peered":"2024-03-06T07:24:53.259095+0000","last_unstale":"2024-03-06T07:24:54.440109+0000","last_undegraded":"2024-03-06T07:24:54.440109+0000","last_fullsized":"2024-03-06T07:24:54.440109+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:57.193091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439960+0000","last_change":"2024-03-06T07:24:53.253741+0000","last_active":"2024-03-06T07:24:54.439960+0000","last_peered":"2024-03-06T07:24:54.439960+0000","last_clean":"2024-03-06T07:24:54.439960+0000","last_became_active":"2024-03-06T07:24:53.253472+0000","last_became_peered":"2024-03-06T07:24:53.253472+0000","last_unstale":"2024-03-06T07:24:54.439960+0000","last_undegraded":"2024-03-06T07:24:54.439960+0000","last_fullsized":"2024-03-06T07:24:54.439960+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:48.653846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250728+0000","last_change":"2024-03-06T07:24:53.248452+0000","last_active":"2024-03-06T07:24:54.250728+0000","last_peered":"2024-03-06T07:24:54.250728+0000","last_clean":"2024-03-06T07:24:54.250728+0000","last_became_active":"2024-03-06T07:24:53.248158+0000","last_became_peered":"2024-03-06T07:24:53.248158+0000","last_unstale":"2024-03-06T07:24:54.250728+0000","last_undegraded":"2024-03-06T07:24:54.250728+0000","last_fullsized":"2024-03-06T07:24:54.250728+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:42.285105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.248688+0000","last_change":"2024-03-06T07:24:53.246872+0000","last_active":"2024-03-06T07:24:54.248688+0000","last_peered":"2024-03-06T07:24:54.248688+0000","last_clean":"2024-03-06T07:24:54.248688+0000","last_became_active":"2024-03-06T07:24:53.246585+0000","last_became_peered":"2024-03-06T07:24:53.246585+0000","last_unstale":"2024-03-06T07:24:54.248688+0000","last_undegraded":"2024-03-06T07:24:54.248688+0000","last_fullsized":"2024-03-06T07:24:54.248688+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:52.290888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439926+0000","last_change":"2024-03-06T07:24:53.261003+0000","last_active":"2024-03-06T07:24:54.439926+0000","last_peered":"2024-03-06T07:24:54.439926+0000","last_clean":"2024-03-06T07:24:54.439926+0000","last_became_active":"2024-03-06T07:24:53.260752+0000","last_became_peered":"2024-03-06T07:24:53.260752+0000","last_unstale":"2024-03-06T07:24:54.439926+0000","last_undegraded":"2024-03-06T07:24:54.439926+0000","last_fullsized":"2024-03-06T07:24:54.439926+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:43.309242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439798+0000","last_change":"2024-03-06T07:24:53.260357+0000","last_active":"2024-03-06T07:24:54.439798+0000","last_peered":"2024-03-06T07:24:54.439798+0000","last_clean":"2024-03-06T07:24:54.439798+0000","last_became_active":"2024-03-06T07:24:53.259659+0000","last_became_peered":"2024-03-06T07:24:53.259659+0000","last_unstale":"2024-03-06T07:24:54.439798+0000","last_undegraded":"2024-03-06T07:24:54.439798+0000","last_fullsized":"2024-03-06T07:24:54.439798+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:46.394428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.253794+0000","last_change":"2024-03-06T07:24:53.252892+0000","last_active":"2024-03-06T07:24:54.253794+0000","last_peered":"2024-03-06T07:24:54.253794+0000","last_clean":"2024-03-06T07:24:54.253794+0000","last_became_active":"2024-03-06T07:24:53.252633+0000","last_became_peered":"2024-03-06T07:24:53.252633+0000","last_unstale":"2024-03-06T07:24:54.253794+0000","last_undegraded":"2024-03-06T07:24:54.253794+0000","last_fullsized":"2024-03-06T07:24:54.253794+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:43.213777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.248415+0000","last_change":"2024-03-06T07:24:53.249410+0000","last_active":"2024-03-06T07:24:54.248415+0000","last_peered":"2024-03-06T07:24:54.248415+0000","last_clean":"2024-03-06T07:24:54.248415+0000","last_became_active":"2024-03-06T07:24:53.249136+0000","last_became_peered":"2024-03-06T07:24:53.249136+0000","last_unstale":"2024-03-06T07:24:54.248415+0000","last_undegraded":"2024-03-06T07:24:54.248415+0000","last_fullsized":"2024-03-06T07:24:54.248415+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:12.785622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.253735+0000","last_change":"2024-03-06T07:24:53.244091+0000","last_active":"2024-03-06T07:24:54.253735+0000","last_peered":"2024-03-06T07:24:54.253735+0000","last_clean":"2024-03-06T07:24:54.253735+0000","last_became_active":"2024-03-06T07:24:53.243786+0000","last_became_peered":"2024-03-06T07:24:53.243786+0000","last_unstale":"2024-03-06T07:24:54.253735+0000","last_undegraded":"2024-03-06T07:24:54.253735+0000","last_fullsized":"2024-03-06T07:24:54.253735+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:44.388371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"120'4","reported_seq":18,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.440149+0000","last_change":"2024-03-06T07:24:52.247044+0000","last_active":"2024-03-06T07:24:54.440149+0000","last_peered":"2024-03-06T07:24:54.440149+0000","last_clean":"2024-03-06T07:24:54.440149+0000","last_became_active":"2024-03-06T07:24:52.246760+0000","last_became_peered":"2024-03-06T07:24:52.246760+0000","last_unstale":"2024-03-06T07:24:54.440149+0000","last_undegraded":"2024-03-06T07:24:54.440149+0000","last_fullsized":"2024-03-06T07:24:54.440149+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:04.476430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439298+0000","last_change":"2024-03-06T07:24:53.256811+0000","last_active":"2024-03-06T07:24:54.439298+0000","last_peered":"2024-03-06T07:24:54.439298+0000","last_clean":"2024-03-06T07:24:54.439298+0000","last_became_active":"2024-03-06T07:24:53.256524+0000","last_became_peered":"2024-03-06T07:24:53.256524+0000","last_unstale":"2024-03-06T07:24:54.439298+0000","last_undegraded":"2024-03-06T07:24:54.439298+0000","last_fullsized":"2024-03-06T07:24:54.439298+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:03.848378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439594+0000","last_change":"2024-03-06T07:24:52.248683+0000","last_active":"2024-03-06T07:24:54.439594+0000","last_peered":"2024-03-06T07:24:54.439594+0000","last_clean":"2024-03-06T07:24:54.439594+0000","last_became_active":"2024-03-06T07:24:52.248404+0000","last_became_peered":"2024-03-06T07:24:52.248404+0000","last_unstale":"2024-03-06T07:24:54.439594+0000","last_undegraded":"2024-03-06T07:24:54.439594+0000","last_fullsized":"2024-03-06T07:24:54.439594+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:15.323244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256238+0000","last_change":"2024-03-06T07:24:53.250936+0000","last_active":"2024-03-06T07:24:54.256238+0000","last_peered":"2024-03-06T07:24:54.256238+0000","last_clean":"2024-03-06T07:24:54.256238+0000","last_became_active":"2024-03-06T07:24:53.250685+0000","last_became_peered":"2024-03-06T07:24:53.250685+0000","last_unstale":"2024-03-06T07:24:54.256238+0000","last_undegraded":"2024-03-06T07:24:54.256238+0000","last_fullsized":"2024-03-06T07:24:54.256238+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:32.092076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250803+0000","last_change":"2024-03-06T07:24:52.249483+0000","last_active":"2024-03-06T07:24:54.250803+0000","last_peered":"2024-03-06T07:24:54.250803+0000","last_clean":"2024-03-06T07:24:54.250803+0000","last_became_active":"2024-03-06T07:24:52.249176+0000","last_became_peered":"2024-03-06T07:24:52.249176+0000","last_unstale":"2024-03-06T07:24:54.250803+0000","last_undegraded":"2024-03-06T07:24:54.250803+0000","last_fullsized":"2024-03-06T07:24:54.250803+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:02.354091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439552+0000","last_change":"2024-03-06T07:24:53.253820+0000","last_active":"2024-03-06T07:24:54.439552+0000","last_peered":"2024-03-06T07:24:54.439552+0000","last_clean":"2024-03-06T07:24:54.439552+0000","last_became_active":"2024-03-06T07:24:53.253522+0000","last_became_peered":"2024-03-06T07:24:53.253522+0000","last_unstale":"2024-03-06T07:24:54.439552+0000","last_undegraded":"2024-03-06T07:24:54.439552+0000","last_fullsized":"2024-03-06T07:24:54.439552+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:59.346066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250733+0000","last_change":"2024-03-06T07:24:52.250493+0000","last_active":"2024-03-06T07:24:54.250733+0000","last_peered":"2024-03-06T07:24:54.250733+0000","last_clean":"2024-03-06T07:24:54.250733+0000","last_became_active":"2024-03-06T07:24:52.250205+0000","last_became_peered":"2024-03-06T07:24:52.250205+0000","last_unstale":"2024-03-06T07:24:54.250733+0000","last_undegraded":"2024-03-06T07:24:54.250733+0000","last_fullsized":"2024-03-06T07:24:54.250733+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.501831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.253991+0000","last_change":"2024-03-06T07:24:53.253795+0000","last_active":"2024-03-06T07:24:54.253991+0000","last_peered":"2024-03-06T07:24:54.253991+0000","last_clean":"2024-03-06T07:24:54.253991+0000","last_became_active":"2024-03-06T07:24:53.253513+0000","last_became_peered":"2024-03-06T07:24:53.253513+0000","last_unstale":"2024-03-06T07:24:54.253991+0000","last_undegraded":"2024-03-06T07:24:54.253991+0000","last_fullsized":"2024-03-06T07:24:54.253991+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:23.122619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.253883+0000","last_change":"2024-03-06T07:24:52.249840+0000","last_active":"2024-03-06T07:24:54.253883+0000","last_peered":"2024-03-06T07:24:54.253883+0000","last_clean":"2024-03-06T07:24:54.253883+0000","last_became_active":"2024-03-06T07:24:52.249544+0000","last_became_peered":"2024-03-06T07:24:52.249544+0000","last_unstale":"2024-03-06T07:24:54.253883+0000","last_undegraded":"2024-03-06T07:24:54.253883+0000","last_fullsized":"2024-03-06T07:24:54.253883+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:28.907321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.253989+0000","last_change":"2024-03-06T07:24:53.247716+0000","last_active":"2024-03-06T07:24:54.253989+0000","last_peered":"2024-03-06T07:24:54.253989+0000","last_clean":"2024-03-06T07:24:54.253989+0000","last_became_active":"2024-03-06T07:24:53.247457+0000","last_became_peered":"2024-03-06T07:24:53.247457+0000","last_unstale":"2024-03-06T07:24:54.253989+0000","last_undegraded":"2024-03-06T07:24:54.253989+0000","last_fullsized":"2024-03-06T07:24:54.253989+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:32.092076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.438940+0000","last_change":"2024-03-06T07:24:52.252509+0000","last_active":"2024-03-06T07:24:54.438940+0000","last_peered":"2024-03-06T07:24:54.438940+0000","last_clean":"2024-03-06T07:24:54.438940+0000","last_became_active":"2024-03-06T07:24:52.252136+0000","last_became_peered":"2024-03-06T07:24:52.252136+0000","last_unstale":"2024-03-06T07:24:54.438940+0000","last_undegraded":"2024-03-06T07:24:54.438940+0000","last_fullsized":"2024-03-06T07:24:54.438940+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:57.778524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439487+0000","last_change":"2024-03-06T07:24:53.258341+0000","last_active":"2024-03-06T07:24:54.439487+0000","last_peered":"2024-03-06T07:24:54.439487+0000","last_clean":"2024-03-06T07:24:54.439487+0000","last_became_active":"2024-03-06T07:24:53.258058+0000","last_became_peered":"2024-03-06T07:24:53.258058+0000","last_unstale":"2024-03-06T07:24:54.439487+0000","last_undegraded":"2024-03-06T07:24:54.439487+0000","last_fullsized":"2024-03-06T07:24:54.439487+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:20.737007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249488+0000","last_change":"2024-03-06T07:24:52.254290+0000","last_active":"2024-03-06T07:24:54.249488+0000","last_peered":"2024-03-06T07:24:54.249488+0000","last_clean":"2024-03-06T07:24:54.249488+0000","last_became_active":"2024-03-06T07:24:52.253973+0000","last_became_peered":"2024-03-06T07:24:52.253973+0000","last_unstale":"2024-03-06T07:24:54.249488+0000","last_undegraded":"2024-03-06T07:24:54.249488+0000","last_fullsized":"2024-03-06T07:24:54.249488+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:42.304681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250674+0000","last_change":"2024-03-06T07:24:53.247738+0000","last_active":"2024-03-06T07:24:54.250674+0000","last_peered":"2024-03-06T07:24:54.250674+0000","last_clean":"2024-03-06T07:24:54.250674+0000","last_became_active":"2024-03-06T07:24:53.247476+0000","last_became_peered":"2024-03-06T07:24:53.247476+0000","last_unstale":"2024-03-06T07:24:54.250674+0000","last_undegraded":"2024-03-06T07:24:54.250674+0000","last_fullsized":"2024-03-06T07:24:54.250674+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:10.368888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439600+0000","last_change":"2024-03-06T07:24:52.250234+0000","last_active":"2024-03-06T07:24:54.439600+0000","last_peered":"2024-03-06T07:24:54.439600+0000","last_clean":"2024-03-06T07:24:54.439600+0000","last_became_active":"2024-03-06T07:24:52.249932+0000","last_became_peered":"2024-03-06T07:24:52.249932+0000","last_unstale":"2024-03-06T07:24:54.439600+0000","last_undegraded":"2024-03-06T07:24:54.439600+0000","last_fullsized":"2024-03-06T07:24:54.439600+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:17.992143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.438987+0000","last_change":"2024-03-06T07:24:53.256255+0000","last_active":"2024-03-06T07:24:54.438987+0000","last_peered":"2024-03-06T07:24:54.438987+0000","last_clean":"2024-03-06T07:24:54.438987+0000","last_became_active":"2024-03-06T07:24:53.254484+0000","last_became_peered":"2024-03-06T07:24:53.254484+0000","last_unstale":"2024-03-06T07:24:54.438987+0000","last_undegraded":"2024-03-06T07:24:54.438987+0000","last_fullsized":"2024-03-06T07:24:54.438987+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:39.350175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256446+0000","last_change":"2024-03-06T07:24:52.250483+0000","last_active":"2024-03-06T07:24:54.256446+0000","last_peered":"2024-03-06T07:24:54.256446+0000","last_clean":"2024-03-06T07:24:54.256446+0000","last_became_active":"2024-03-06T07:24:52.250201+0000","last_became_peered":"2024-03-06T07:24:52.250201+0000","last_unstale":"2024-03-06T07:24:54.256446+0000","last_undegraded":"2024-03-06T07:24:54.256446+0000","last_fullsized":"2024-03-06T07:24:54.256446+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:45.262772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249775+0000","last_change":"2024-03-06T07:24:53.248967+0000","last_active":"2024-03-06T07:24:54.249775+0000","last_peered":"2024-03-06T07:24:54.249775+0000","last_clean":"2024-03-06T07:24:54.249775+0000","last_became_active":"2024-03-06T07:24:53.248669+0000","last_became_peered":"2024-03-06T07:24:53.248669+0000","last_unstale":"2024-03-06T07:24:54.249775+0000","last_undegraded":"2024-03-06T07:24:54.249775+0000","last_fullsized":"2024-03-06T07:24:54.249775+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:00.496495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439552+0000","last_change":"2024-03-06T07:24:52.251851+0000","last_active":"2024-03-06T07:24:54.439552+0000","last_peered":"2024-03-06T07:24:54.439552+0000","last_clean":"2024-03-06T07:24:54.439552+0000","last_became_active":"2024-03-06T07:24:52.251549+0000","last_became_peered":"2024-03-06T07:24:52.251549+0000","last_unstale":"2024-03-06T07:24:54.439552+0000","last_undegraded":"2024-03-06T07:24:54.439552+0000","last_fullsized":"2024-03-06T07:24:54.439552+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:22.655955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439789+0000","last_change":"2024-03-06T07:24:53.255935+0000","last_active":"2024-03-06T07:24:54.439789+0000","last_peered":"2024-03-06T07:24:54.439789+0000","last_clean":"2024-03-06T07:24:54.439789+0000","last_became_active":"2024-03-06T07:24:53.255457+0000","last_became_peered":"2024-03-06T07:24:53.255457+0000","last_unstale":"2024-03-06T07:24:54.439789+0000","last_undegraded":"2024-03-06T07:24:54.439789+0000","last_fullsized":"2024-03-06T07:24:54.439789+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:41.534030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"120'3","reported_seq":17,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250865+0000","last_change":"2024-03-06T07:24:52.245152+0000","last_active":"2024-03-06T07:24:54.250865+0000","last_peered":"2024-03-06T07:24:54.250865+0000","last_clean":"2024-03-06T07:24:54.250865+0000","last_became_active":"2024-03-06T07:24:52.244695+0000","last_became_peered":"2024-03-06T07:24:52.244695+0000","last_unstale":"2024-03-06T07:24:54.250865+0000","last_undegraded":"2024-03-06T07:24:54.250865+0000","last_fullsized":"2024-03-06T07:24:54.250865+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:29.124565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439995+0000","last_change":"2024-03-06T07:24:53.249958+0000","last_active":"2024-03-06T07:24:54.439995+0000","last_peered":"2024-03-06T07:24:54.439995+0000","last_clean":"2024-03-06T07:24:54.439995+0000","last_became_active":"2024-03-06T07:24:53.249636+0000","last_became_peered":"2024-03-06T07:24:53.249636+0000","last_unstale":"2024-03-06T07:24:54.439995+0000","last_undegraded":"2024-03-06T07:24:54.439995+0000","last_fullsized":"2024-03-06T07:24:54.439995+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:47.432979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"120'6","reported_seq":20,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250846+0000","last_change":"2024-03-06T07:24:52.250220+0000","last_active":"2024-03-06T07:24:54.250846+0000","last_peered":"2024-03-06T07:24:54.250846+0000","last_clean":"2024-03-06T07:24:54.250846+0000","last_became_active":"2024-03-06T07:24:52.249566+0000","last_became_peered":"2024-03-06T07:24:52.249566+0000","last_unstale":"2024-03-06T07:24:54.250846+0000","last_undegraded":"2024-03-06T07:24:54.250846+0000","last_fullsized":"2024-03-06T07:24:54.250846+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:21.406441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.248748+0000","last_change":"2024-03-06T07:24:53.247650+0000","last_active":"2024-03-06T07:24:54.248748+0000","last_peered":"2024-03-06T07:24:54.248748+0000","last_clean":"2024-03-06T07:24:54.248748+0000","last_became_active":"2024-03-06T07:24:53.247367+0000","last_became_peered":"2024-03-06T07:24:53.247367+0000","last_unstale":"2024-03-06T07:24:54.248748+0000","last_undegraded":"2024-03-06T07:24:54.248748+0000","last_fullsized":"2024-03-06T07:24:54.248748+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:43.938094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439586+0000","last_change":"2024-03-06T07:24:52.251172+0000","last_active":"2024-03-06T07:24:54.439586+0000","last_peered":"2024-03-06T07:24:54.439586+0000","last_clean":"2024-03-06T07:24:54.439586+0000","last_became_active":"2024-03-06T07:24:52.250886+0000","last_became_peered":"2024-03-06T07:24:52.250886+0000","last_unstale":"2024-03-06T07:24:54.439586+0000","last_undegraded":"2024-03-06T07:24:54.439586+0000","last_fullsized":"2024-03-06T07:24:54.439586+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:44.062114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.251037+0000","last_change":"2024-03-06T07:24:53.252070+0000","last_active":"2024-03-06T07:24:54.251037+0000","last_peered":"2024-03-06T07:24:54.251037+0000","last_clean":"2024-03-06T07:24:54.251037+0000","last_became_active":"2024-03-06T07:24:53.251684+0000","last_became_peered":"2024-03-06T07:24:53.251684+0000","last_unstale":"2024-03-06T07:24:54.251037+0000","last_undegraded":"2024-03-06T07:24:54.251037+0000","last_fullsized":"2024-03-06T07:24:54.251037+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:33.368117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439606+0000","last_change":"2024-03-06T07:24:52.243716+0000","last_active":"2024-03-06T07:24:54.439606+0000","last_peered":"2024-03-06T07:24:54.439606+0000","last_clean":"2024-03-06T07:24:54.439606+0000","last_became_active":"2024-03-06T07:24:52.243443+0000","last_became_peered":"2024-03-06T07:24:52.243443+0000","last_unstale":"2024-03-06T07:24:54.439606+0000","last_undegraded":"2024-03-06T07:24:54.439606+0000","last_fullsized":"2024-03-06T07:24:54.439606+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:36.682515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439706+0000","last_change":"2024-03-06T07:24:53.265368+0000","last_active":"2024-03-06T07:24:54.439706+0000","last_peered":"2024-03-06T07:24:54.439706+0000","last_clean":"2024-03-06T07:24:54.439706+0000","last_became_active":"2024-03-06T07:24:53.265105+0000","last_became_peered":"2024-03-06T07:24:53.265105+0000","last_unstale":"2024-03-06T07:24:54.439706+0000","last_undegraded":"2024-03-06T07:24:54.439706+0000","last_fullsized":"2024-03-06T07:24:54.439706+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:51.633397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"120'8","reported_seq":22,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249614+0000","last_change":"2024-03-06T07:24:52.254923+0000","last_active":"2024-03-06T07:24:54.249614+0000","last_peered":"2024-03-06T07:24:54.249614+0000","last_clean":"2024-03-06T07:24:54.249614+0000","last_became_active":"2024-03-06T07:24:52.254647+0000","last_became_peered":"2024-03-06T07:24:52.254647+0000","last_unstale":"2024-03-06T07:24:54.249614+0000","last_undegraded":"2024-03-06T07:24:54.249614+0000","last_fullsized":"2024-03-06T07:24:54.249614+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:45.389867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439586+0000","last_change":"2024-03-06T07:24:53.256886+0000","last_active":"2024-03-06T07:24:54.439586+0000","last_peered":"2024-03-06T07:24:54.439586+0000","last_clean":"2024-03-06T07:24:54.439586+0000","last_became_active":"2024-03-06T07:24:53.256158+0000","last_became_peered":"2024-03-06T07:24:53.256158+0000","last_unstale":"2024-03-06T07:24:54.439586+0000","last_undegraded":"2024-03-06T07:24:54.439586+0000","last_fullsized":"2024-03-06T07:24:54.439586+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:25.241414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256481+0000","last_change":"2024-03-06T07:24:52.248044+0000","last_active":"2024-03-06T07:24:54.256481+0000","last_peered":"2024-03-06T07:24:54.256481+0000","last_clean":"2024-03-06T07:24:54.256481+0000","last_became_active":"2024-03-06T07:24:52.247701+0000","last_became_peered":"2024-03-06T07:24:52.247701+0000","last_unstale":"2024-03-06T07:24:54.256481+0000","last_undegraded":"2024-03-06T07:24:54.256481+0000","last_fullsized":"2024-03-06T07:24:54.256481+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:28.907321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250596+0000","last_change":"2024-03-06T07:24:53.254344+0000","last_active":"2024-03-06T07:24:54.250596+0000","last_peered":"2024-03-06T07:24:54.250596+0000","last_clean":"2024-03-06T07:24:54.250596+0000","last_became_active":"2024-03-06T07:24:53.253871+0000","last_became_peered":"2024-03-06T07:24:53.253871+0000","last_unstale":"2024-03-06T07:24:54.250596+0000","last_undegraded":"2024-03-06T07:24:54.250596+0000","last_fullsized":"2024-03-06T07:24:54.250596+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:09.269921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439966+0000","last_change":"2024-03-06T07:24:52.247010+0000","last_active":"2024-03-06T07:24:54.439966+0000","last_peered":"2024-03-06T07:24:54.439966+0000","last_clean":"2024-03-06T07:24:54.439966+0000","last_became_active":"2024-03-06T07:24:52.246730+0000","last_became_peered":"2024-03-06T07:24:52.246730+0000","last_unstale":"2024-03-06T07:24:54.439966+0000","last_undegraded":"2024-03-06T07:24:54.439966+0000","last_fullsized":"2024-03-06T07:24:54.439966+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:49.426429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.251040+0000","last_change":"2024-03-06T07:24:53.248482+0000","last_active":"2024-03-06T07:24:54.251040+0000","last_peered":"2024-03-06T07:24:54.251040+0000","last_clean":"2024-03-06T07:24:54.251040+0000","last_became_active":"2024-03-06T07:24:53.248185+0000","last_became_peered":"2024-03-06T07:24:53.248185+0000","last_unstale":"2024-03-06T07:24:54.251040+0000","last_undegraded":"2024-03-06T07:24:54.251040+0000","last_fullsized":"2024-03-06T07:24:54.251040+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:57.903166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250066+0000","last_change":"2024-03-06T07:24:52.253907+0000","last_active":"2024-03-06T07:24:54.250066+0000","last_peered":"2024-03-06T07:24:54.250066+0000","last_clean":"2024-03-06T07:24:54.250066+0000","last_became_active":"2024-03-06T07:24:52.253561+0000","last_became_peered":"2024-03-06T07:24:52.253561+0000","last_unstale":"2024-03-06T07:24:54.250066+0000","last_undegraded":"2024-03-06T07:24:54.250066+0000","last_fullsized":"2024-03-06T07:24:54.250066+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:53.820944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.248679+0000","last_change":"2024-03-06T07:24:53.253080+0000","last_active":"2024-03-06T07:24:54.248679+0000","last_peered":"2024-03-06T07:24:54.248679+0000","last_clean":"2024-03-06T07:24:54.248679+0000","last_became_active":"2024-03-06T07:24:53.252576+0000","last_became_peered":"2024-03-06T07:24:53.252576+0000","last_unstale":"2024-03-06T07:24:54.248679+0000","last_undegraded":"2024-03-06T07:24:54.248679+0000","last_fullsized":"2024-03-06T07:24:54.248679+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:37.547736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439471+0000","last_change":"2024-03-06T07:24:52.254150+0000","last_active":"2024-03-06T07:24:54.439471+0000","last_peered":"2024-03-06T07:24:54.439471+0000","last_clean":"2024-03-06T07:24:54.439471+0000","last_became_active":"2024-03-06T07:24:52.253882+0000","last_became_peered":"2024-03-06T07:24:52.253882+0000","last_unstale":"2024-03-06T07:24:54.439471+0000","last_undegraded":"2024-03-06T07:24:54.439471+0000","last_fullsized":"2024-03-06T07:24:54.439471+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:29.905192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439379+0000","last_change":"2024-03-06T07:24:53.264528+0000","last_active":"2024-03-06T07:24:54.439379+0000","last_peered":"2024-03-06T07:24:54.439379+0000","last_clean":"2024-03-06T07:24:54.439379+0000","last_became_active":"2024-03-06T07:24:53.264085+0000","last_became_peered":"2024-03-06T07:24:53.264085+0000","last_unstale":"2024-03-06T07:24:54.439379+0000","last_undegraded":"2024-03-06T07:24:54.439379+0000","last_fullsized":"2024-03-06T07:24:54.439379+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:12.103222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.253910+0000","last_change":"2024-03-06T07:24:52.253449+0000","last_active":"2024-03-06T07:24:54.253910+0000","last_peered":"2024-03-06T07:24:54.253910+0000","last_clean":"2024-03-06T07:24:54.253910+0000","last_became_active":"2024-03-06T07:24:52.253160+0000","last_became_peered":"2024-03-06T07:24:52.253160+0000","last_unstale":"2024-03-06T07:24:54.253910+0000","last_undegraded":"2024-03-06T07:24:54.253910+0000","last_fullsized":"2024-03-06T07:24:54.253910+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:24.236853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439443+0000","last_change":"2024-03-06T07:24:53.246000+0000","last_active":"2024-03-06T07:24:54.439443+0000","last_peered":"2024-03-06T07:24:54.439443+0000","last_clean":"2024-03-06T07:24:54.439443+0000","last_became_active":"2024-03-06T07:24:53.245699+0000","last_became_peered":"2024-03-06T07:24:53.245699+0000","last_unstale":"2024-03-06T07:24:54.439443+0000","last_undegraded":"2024-03-06T07:24:54.439443+0000","last_fullsized":"2024-03-06T07:24:54.439443+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:00.657287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439399+0000","last_change":"2024-03-06T07:24:52.249983+0000","last_active":"2024-03-06T07:24:54.439399+0000","last_peered":"2024-03-06T07:24:54.439399+0000","last_clean":"2024-03-06T07:24:54.439399+0000","last_became_active":"2024-03-06T07:24:52.249713+0000","last_became_peered":"2024-03-06T07:24:52.249713+0000","last_unstale":"2024-03-06T07:24:54.439399+0000","last_undegraded":"2024-03-06T07:24:54.439399+0000","last_fullsized":"2024-03-06T07:24:54.439399+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:37.394132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439583+0000","last_change":"2024-03-06T07:24:53.259552+0000","last_active":"2024-03-06T07:24:54.439583+0000","last_peered":"2024-03-06T07:24:54.439583+0000","last_clean":"2024-03-06T07:24:54.439583+0000","last_became_active":"2024-03-06T07:24:53.259269+0000","last_became_peered":"2024-03-06T07:24:53.259269+0000","last_unstale":"2024-03-06T07:24:54.439583+0000","last_undegraded":"2024-03-06T07:24:54.439583+0000","last_fullsized":"2024-03-06T07:24:54.439583+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:03.090749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250694+0000","last_change":"2024-03-06T07:24:52.245263+0000","last_active":"2024-03-06T07:24:54.250694+0000","last_peered":"2024-03-06T07:24:54.250694+0000","last_clean":"2024-03-06T07:24:54.250694+0000","last_became_active":"2024-03-06T07:24:52.244989+0000","last_became_peered":"2024-03-06T07:24:52.244989+0000","last_unstale":"2024-03-06T07:24:54.250694+0000","last_undegraded":"2024-03-06T07:24:54.250694+0000","last_fullsized":"2024-03-06T07:24:54.250694+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:22.118058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256412+0000","last_change":"2024-03-06T07:24:53.251996+0000","last_active":"2024-03-06T07:24:54.256412+0000","last_peered":"2024-03-06T07:24:54.256412+0000","last_clean":"2024-03-06T07:24:54.256412+0000","last_became_active":"2024-03-06T07:24:53.251734+0000","last_became_peered":"2024-03-06T07:24:53.251734+0000","last_unstale":"2024-03-06T07:24:54.256412+0000","last_undegraded":"2024-03-06T07:24:54.256412+0000","last_fullsized":"2024-03-06T07:24:54.256412+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:52.506392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249695+0000","last_change":"2024-03-06T07:24:52.248581+0000","last_active":"2024-03-06T07:24:54.249695+0000","last_peered":"2024-03-06T07:24:54.249695+0000","last_clean":"2024-03-06T07:24:54.249695+0000","last_became_active":"2024-03-06T07:24:52.248317+0000","last_became_peered":"2024-03-06T07:24:52.248317+0000","last_unstale":"2024-03-06T07:24:54.249695+0000","last_undegraded":"2024-03-06T07:24:54.249695+0000","last_fullsized":"2024-03-06T07:24:54.249695+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:40.529469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439147+0000","last_change":"2024-03-06T07:24:53.256765+0000","last_active":"2024-03-06T07:24:54.439147+0000","last_peered":"2024-03-06T07:24:54.439147+0000","last_clean":"2024-03-06T07:24:54.439147+0000","last_became_active":"2024-03-06T07:24:53.256476+0000","last_became_peered":"2024-03-06T07:24:53.256476+0000","last_unstale":"2024-03-06T07:24:54.439147+0000","last_undegraded":"2024-03-06T07:24:54.439147+0000","last_fullsized":"2024-03-06T07:24:54.439147+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:28.574847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439740+0000","last_change":"2024-03-06T07:24:52.248393+0000","last_active":"2024-03-06T07:24:54.439740+0000","last_peered":"2024-03-06T07:24:54.439740+0000","last_clean":"2024-03-06T07:24:54.439740+0000","last_became_active":"2024-03-06T07:24:52.248126+0000","last_became_peered":"2024-03-06T07:24:52.248126+0000","last_unstale":"2024-03-06T07:24:54.439740+0000","last_undegraded":"2024-03-06T07:24:54.439740+0000","last_fullsized":"2024-03-06T07:24:54.439740+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:38.691189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439598+0000","last_change":"2024-03-06T07:24:53.248008+0000","last_active":"2024-03-06T07:24:54.439598+0000","last_peered":"2024-03-06T07:24:54.439598+0000","last_clean":"2024-03-06T07:24:54.439598+0000","last_became_active":"2024-03-06T07:24:53.247727+0000","last_became_peered":"2024-03-06T07:24:53.247727+0000","last_unstale":"2024-03-06T07:24:54.439598+0000","last_undegraded":"2024-03-06T07:24:54.439598+0000","last_fullsized":"2024-03-06T07:24:54.439598+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:29.911882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"121'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.255540+0000","last_change":"2024-03-06T07:24:52.252134+0000","last_active":"2024-03-06T07:24:54.255540+0000","last_peered":"2024-03-06T07:24:54.255540+0000","last_clean":"2024-03-06T07:24:54.255540+0000","last_became_active":"2024-03-06T07:24:52.251774+0000","last_became_peered":"2024-03-06T07:24:52.251774+0000","last_unstale":"2024-03-06T07:24:54.255540+0000","last_undegraded":"2024-03-06T07:24:54.255540+0000","last_fullsized":"2024-03-06T07:24:54.255540+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:45.389867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250675+0000","last_change":"2024-03-06T07:24:53.253111+0000","last_active":"2024-03-06T07:24:54.250675+0000","last_peered":"2024-03-06T07:24:54.250675+0000","last_clean":"2024-03-06T07:24:54.250675+0000","last_became_active":"2024-03-06T07:24:53.252833+0000","last_became_peered":"2024-03-06T07:24:53.252833+0000","last_unstale":"2024-03-06T07:24:54.250675+0000","last_undegraded":"2024-03-06T07:24:54.250675+0000","last_fullsized":"2024-03-06T07:24:54.250675+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:08.595386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.254075+0000","last_change":"2024-03-06T07:24:52.250376+0000","last_active":"2024-03-06T07:24:54.254075+0000","last_peered":"2024-03-06T07:24:54.254075+0000","last_clean":"2024-03-06T07:24:54.254075+0000","last_became_active":"2024-03-06T07:24:52.250122+0000","last_became_peered":"2024-03-06T07:24:52.250122+0000","last_unstale":"2024-03-06T07:24:54.254075+0000","last_undegraded":"2024-03-06T07:24:54.254075+0000","last_fullsized":"2024-03-06T07:24:54.254075+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:40.529469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.248787+0000","last_change":"2024-03-06T07:24:53.246460+0000","last_active":"2024-03-06T07:24:54.248787+0000","last_peered":"2024-03-06T07:24:54.248787+0000","last_clean":"2024-03-06T07:24:54.248787+0000","last_became_active":"2024-03-06T07:24:53.246157+0000","last_became_peered":"2024-03-06T07:24:53.246157+0000","last_unstale":"2024-03-06T07:24:54.248787+0000","last_undegraded":"2024-03-06T07:24:54.248787+0000","last_fullsized":"2024-03-06T07:24:54.248787+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:01.649291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250030+0000","last_change":"2024-03-06T07:24:52.253206+0000","last_active":"2024-03-06T07:24:54.250030+0000","last_peered":"2024-03-06T07:24:54.250030+0000","last_clean":"2024-03-06T07:24:54.250030+0000","last_became_active":"2024-03-06T07:24:52.252885+0000","last_became_peered":"2024-03-06T07:24:52.252885+0000","last_unstale":"2024-03-06T07:24:54.250030+0000","last_undegraded":"2024-03-06T07:24:54.250030+0000","last_fullsized":"2024-03-06T07:24:54.250030+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:47.649285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249941+0000","last_change":"2024-03-06T07:24:53.251552+0000","last_active":"2024-03-06T07:24:54.249941+0000","last_peered":"2024-03-06T07:24:54.249941+0000","last_clean":"2024-03-06T07:24:54.249941+0000","last_became_active":"2024-03-06T07:24:53.251275+0000","last_became_peered":"2024-03-06T07:24:53.251275+0000","last_unstale":"2024-03-06T07:24:54.249941+0000","last_undegraded":"2024-03-06T07:24:54.249941+0000","last_fullsized":"2024-03-06T07:24:54.249941+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:58.783086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.256234+0000","last_change":"2024-03-06T07:24:52.246330+0000","last_active":"2024-03-06T07:24:54.256234+0000","last_peered":"2024-03-06T07:24:54.256234+0000","last_clean":"2024-03-06T07:24:54.256234+0000","last_became_active":"2024-03-06T07:24:52.246067+0000","last_became_peered":"2024-03-06T07:24:52.246067+0000","last_unstale":"2024-03-06T07:24:54.256234+0000","last_undegraded":"2024-03-06T07:24:54.256234+0000","last_fullsized":"2024-03-06T07:24:54.256234+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:37.112908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250075+0000","last_change":"2024-03-06T07:24:53.252967+0000","last_active":"2024-03-06T07:24:54.250075+0000","last_peered":"2024-03-06T07:24:54.250075+0000","last_clean":"2024-03-06T07:24:54.250075+0000","last_became_active":"2024-03-06T07:24:53.252675+0000","last_became_peered":"2024-03-06T07:24:53.252675+0000","last_unstale":"2024-03-06T07:24:54.250075+0000","last_undegraded":"2024-03-06T07:24:54.250075+0000","last_fullsized":"2024-03-06T07:24:54.250075+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:16.327805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250975+0000","last_change":"2024-03-06T07:24:52.251188+0000","last_active":"2024-03-06T07:24:54.250975+0000","last_peered":"2024-03-06T07:24:54.250975+0000","last_clean":"2024-03-06T07:24:54.250975+0000","last_became_active":"2024-03-06T07:24:52.250925+0000","last_became_peered":"2024-03-06T07:24:52.250925+0000","last_unstale":"2024-03-06T07:24:54.250975+0000","last_undegraded":"2024-03-06T07:24:54.250975+0000","last_fullsized":"2024-03-06T07:24:54.250975+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:42.209216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249709+0000","last_change":"2024-03-06T07:24:53.252096+0000","last_active":"2024-03-06T07:24:54.249709+0000","last_peered":"2024-03-06T07:24:54.249709+0000","last_clean":"2024-03-06T07:24:54.249709+0000","last_became_active":"2024-03-06T07:24:53.251873+0000","last_became_peered":"2024-03-06T07:24:53.251873+0000","last_unstale":"2024-03-06T07:24:54.249709+0000","last_undegraded":"2024-03-06T07:24:54.249709+0000","last_fullsized":"2024-03-06T07:24:54.249709+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:06.485394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.249677+0000","last_change":"2024-03-06T07:24:52.248523+0000","last_active":"2024-03-06T07:24:54.249677+0000","last_peered":"2024-03-06T07:24:54.249677+0000","last_clean":"2024-03-06T07:24:54.249677+0000","last_became_active":"2024-03-06T07:24:52.248248+0000","last_became_peered":"2024-03-06T07:24:52.248248+0000","last_unstale":"2024-03-06T07:24:54.249677+0000","last_undegraded":"2024-03-06T07:24:54.249677+0000","last_fullsized":"2024-03-06T07:24:54.249677+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:46.428418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250819+0000","last_change":"2024-03-06T07:24:53.253425+0000","last_active":"2024-03-06T07:24:54.250819+0000","last_peered":"2024-03-06T07:24:54.250819+0000","last_clean":"2024-03-06T07:24:54.250819+0000","last_became_active":"2024-03-06T07:24:53.252966+0000","last_became_peered":"2024-03-06T07:24:53.252966+0000","last_unstale":"2024-03-06T07:24:54.250819+0000","last_undegraded":"2024-03-06T07:24:54.250819+0000","last_fullsized":"2024-03-06T07:24:54.250819+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:04.537106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.439091+0000","last_change":"2024-03-06T07:24:52.253847+0000","last_active":"2024-03-06T07:24:54.439091+0000","last_peered":"2024-03-06T07:24:54.439091+0000","last_clean":"2024-03-06T07:24:54.439091+0000","last_became_active":"2024-03-06T07:24:52.253549+0000","last_became_peered":"2024-03-06T07:24:52.253549+0000","last_unstale":"2024-03-06T07:24:54.439091+0000","last_undegraded":"2024-03-06T07:24:54.439091+0000","last_fullsized":"2024-03-06T07:24:54.439091+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:22.374141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.440119+0000","last_change":"2024-03-06T07:24:53.256391+0000","last_active":"2024-03-06T07:24:54.440119+0000","last_peered":"2024-03-06T07:24:54.440119+0000","last_clean":"2024-03-06T07:24:54.440119+0000","last_became_active":"2024-03-06T07:24:53.256049+0000","last_became_peered":"2024-03-06T07:24:53.256049+0000","last_unstale":"2024-03-06T07:24:54.440119+0000","last_undegraded":"2024-03-06T07:24:54.440119+0000","last_fullsized":"2024-03-06T07:24:54.440119+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:52.226230+0000","last_clean_scrub_stamp":"2024-03-06T07:24:52.226230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:27.838125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.251120+0000","last_change":"2024-03-06T07:24:52.246324+0000","last_active":"2024-03-06T07:24:54.251120+0000","last_peered":"2024-03-06T07:24:54.251120+0000","last_clean":"2024-03-06T07:24:54.251120+0000","last_became_active":"2024-03-06T07:24:52.245836+0000","last_became_peered":"2024-03-06T07:24:52.245836+0000","last_unstale":"2024-03-06T07:24:54.251120+0000","last_undegraded":"2024-03-06T07:24:54.251120+0000","last_fullsized":"2024-03-06T07:24:54.251120+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.221669+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.221669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:31.087515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":364,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.250162+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:24:54.250162+0000","last_peered":"2024-03-06T07:24:54.250162+0000","last_clean":"2024-03-06T07:24:54.250162+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:24:54.250162+0000","last_undegraded":"2024-03-06T07:24:54.250162+0000","last_fullsized":"2024-03-06T07:24:54.250162+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32728,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339112,"statfs":{"total":96636764160,"available":96603250688,"internally_reserved":0,"allocated":1585152,"data_stored":1316164,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32656,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339184,"statfs":{"total":96636764160,"available":96603324416,"internally_reserved":0,"allocated":1511424,"data_stored":724766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32652,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339188,"statfs":{"total":96636764160,"available":96603328512,"internally_reserved":0,"allocated":1507328,"data_stored":724116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1515520,"data_stored":730786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32680,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339160,"statfs":{"total":96636764160,"available":96603299840,"internally_reserved":0,"allocated":1531904,"data_stored":732468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1511424,"data_stored":724710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32652,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339188,"statfs":{"total":96636764160,"available":96603328512,"internally_reserved":0,"allocated":1507328,"data_stored":724164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32736,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339104,"statfs":{"total":96636764160,"available":96603242496,"internally_reserved":0,"allocated":1589248,"data_stored":1321210,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:57.734 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:24:57.734 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:24:57.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.877+0000 7f5296a0e640 1 Processor -- start 2024-03-06T07:24:57.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.877+0000 7f5296a0e640 1 -- start start 2024-03-06T07:24:57.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.878+0000 7f5296a0e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5290152650 0x7f5290155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.878+0000 7f5296a0e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f52900c6b10 0x7f5290158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.878+0000 7f5296a0e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f52900bf9d0 0x7f52900c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.878+0000 7f5296a0e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5290155800 con 0x7f52900bf9d0 2024-03-06T07:24:57.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.878+0000 7f5296a0e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52900c2fc0 con 0x7f52900c6b10 2024-03-06T07:24:57.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.878+0000 7f5296a0e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52900c6690 con 0x7f5290152650 2024-03-06T07:24:57.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.878+0000 7f5294fab640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f52900bf9d0 0x7f52900c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.878+0000 7f5294fab640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f52900bf9d0 0x7f52900c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36908/0 (socket says 172.21.15.1:36908) 2024-03-06T07:24:57.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.878+0000 7f5294fab640 1 -- 172.21.15.1:0/2134957502 learned_addr learned my addr 172.21.15.1:0/2134957502 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:57.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.878+0000 7f5294fab640 1 -- 172.21.15.1:0/2134957502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5290152650 msgr2=0x7f5290155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.878+0000 7f528ffff640 1 --2- 172.21.15.1:0/2134957502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5290152650 0x7f5290155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.878+0000 7f528f7fe640 1 --2- 172.21.15.1:0/2134957502 >> v2:172.21.15.98:3300/0 conn(0x7f52900c6b10 0x7f5290158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.879+0000 7f5294fab640 1 --2- 172.21.15.1:0/2134957502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5290152650 0x7f5290155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.879+0000 7f5294fab640 1 -- 172.21.15.1:0/2134957502 >> v2:172.21.15.98:3300/0 conn(0x7f52900c6b10 msgr2=0x7f5290158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.879+0000 7f5294fab640 1 --2- 172.21.15.1:0/2134957502 >> v2:172.21.15.98:3300/0 conn(0x7f52900c6b10 0x7f5290158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.879+0000 7f5294fab640 1 -- 172.21.15.1:0/2134957502 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52900c7280 con 0x7f52900bf9d0 2024-03-06T07:24:57.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.879+0000 7f528f7fe640 1 --2- 172.21.15.1:0/2134957502 >> v2:172.21.15.98:3300/0 conn(0x7f52900c6b10 0x7f5290158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:57.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.879+0000 7f528ffff640 1 --2- 172.21.15.1:0/2134957502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5290152650 0x7f5290155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:57.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.880+0000 7f5294fab640 1 --2- 172.21.15.1:0/2134957502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52900bf9d0 0x7f52900c1460 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f529008c500 tx=0x7f528802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9c771e6c2f3f41f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.880+0000 7f528effd640 1 -- 172.21.15.1:0/2134957502 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528802c9e0 con 0x7f52900bf9d0 2024-03-06T07:24:57.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.880+0000 7f528effd640 1 -- 172.21.15.1:0/2134957502 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528802cb80 con 0x7f52900bf9d0 2024-03-06T07:24:57.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.880+0000 7f528effd640 1 -- 172.21.15.1:0/2134957502 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52880343f0 con 0x7f52900bf9d0 2024-03-06T07:24:57.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.880+0000 7f5296a0e640 1 -- 172.21.15.1:0/2134957502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52900bf9d0 msgr2=0x7f52900c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.880+0000 7f5296a0e640 1 --2- 172.21.15.1:0/2134957502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52900bf9d0 0x7f52900c1460 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f529008c500 tx=0x7f528802be40 comp rx=0 tx=0).stop 2024-03-06T07:24:57.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.880+0000 7f5296a0e640 1 -- 172.21.15.1:0/2134957502 shutdown_connections 2024-03-06T07:24:57.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.880+0000 7f5296a0e640 1 --2- 172.21.15.1:0/2134957502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52900bf9d0 0x7f52900c1460 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.880+0000 7f5296a0e640 1 --2- 172.21.15.1:0/2134957502 >> v2:172.21.15.98:3300/0 conn(0x7f52900c6b10 0x7f5290158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.880+0000 7f5296a0e640 1 --2- 172.21.15.1:0/2134957502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5290152650 0x7f5290155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.880+0000 7f5296a0e640 1 -- 172.21.15.1:0/2134957502 >> 172.21.15.1:0/2134957502 conn(0x7f5290066ee0 msgr2=0x7f52900672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.880+0000 7f5296a0e640 1 -- 172.21.15.1:0/2134957502 shutdown_connections 2024-03-06T07:24:57.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.880+0000 7f5296a0e640 1 -- 172.21.15.1:0/2134957502 wait complete. 2024-03-06T07:24:57.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.881+0000 7f5296a0e640 1 Processor -- start 2024-03-06T07:24:57.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.882+0000 7f5296a0e640 1 -- start start 2024-03-06T07:24:57.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.882+0000 7f5296a0e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52900bf9d0 0x7f5290145240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.882+0000 7f5296a0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5290152650 0x7f5290147450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.882+0000 7f5296a0e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290144d10 0x7f52900c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.882+0000 7f5296a0e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5290155490 con 0x7f5290144d10 2024-03-06T07:24:57.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.882+0000 7f5296a0e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52901551b0 con 0x7f5290152650 2024-03-06T07:24:57.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.882+0000 7f5296a0e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5290152a30 con 0x7f52900bf9d0 2024-03-06T07:24:57.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.882+0000 7f5294fab640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290144d10 0x7f52900c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.882+0000 7f5294fab640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290144d10 0x7f52900c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36920/0 (socket says 172.21.15.1:36920) 2024-03-06T07:24:57.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.882+0000 7f5294fab640 1 -- 172.21.15.1:0/72070201 learned_addr learned my addr 172.21.15.1:0/72070201 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:57.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.883+0000 7f528f7fe640 1 --2- 172.21.15.1:0/72070201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5290152650 0x7f5290147450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.883+0000 7f528ffff640 1 --2- 172.21.15.1:0/72070201 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52900bf9d0 0x7f5290145240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.883+0000 7f5294fab640 1 -- 172.21.15.1:0/72070201 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52900bf9d0 msgr2=0x7f5290145240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.883+0000 7f5294fab640 1 --2- 172.21.15.1:0/72070201 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52900bf9d0 0x7f5290145240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.883+0000 7f5294fab640 1 -- 172.21.15.1:0/72070201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5290152650 msgr2=0x7f5290147450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.883+0000 7f5294fab640 1 --2- 172.21.15.1:0/72070201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5290152650 0x7f5290147450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.883+0000 7f5294fab640 1 -- 172.21.15.1:0/72070201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52900c3ee0 con 0x7f5290144d10 2024-03-06T07:24:57.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.883+0000 7f528f7fe640 1 --2- 172.21.15.1:0/72070201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5290152650 0x7f5290147450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:57.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.883+0000 7f5294fab640 1 --2- 172.21.15.1:0/72070201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290144d10 0x7f52900c9270 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f5288009f40 tx=0x7f5288004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.883+0000 7f528cff9640 1 -- 172.21.15.1:0/72070201 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5288043020 con 0x7f5290144d10 2024-03-06T07:24:57.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.883+0000 7f528cff9640 1 -- 172.21.15.1:0/72070201 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528803e450 con 0x7f5290144d10 2024-03-06T07:24:57.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.883+0000 7f528cff9640 1 -- 172.21.15.1:0/72070201 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5288009510 con 0x7f5290144d10 2024-03-06T07:24:57.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.883+0000 7f5296a0e640 1 -- 172.21.15.1:0/72070201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52900c41c0 con 0x7f5290144d10 2024-03-06T07:24:57.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.883+0000 7f5296a0e640 1 -- 172.21.15.1:0/72070201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52900ca480 con 0x7f5290144d10 2024-03-06T07:24:57.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.884+0000 7f5296a0e640 1 -- 172.21.15.1:0/72070201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5290067fd0 con 0x7f5290144d10 2024-03-06T07:24:57.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.884+0000 7f528cff9640 1 -- 172.21.15.1:0/72070201 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5288046020 con 0x7f5290144d10 2024-03-06T07:24:57.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.885+0000 7f528cff9640 1 --2- 172.21.15.1:0/72070201 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f527006b500 0x7f527006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.885+0000 7f528cff9640 1 -- 172.21.15.1:0/72070201 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f52880acf80 con 0x7f5290144d10 2024-03-06T07:24:57.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.885+0000 7f528ffff640 1 --2- 172.21.15.1:0/72070201 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f527006b500 0x7f527006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.885+0000 7f528ffff640 1 --2- 172.21.15.1:0/72070201 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f527006b500 0x7f527006d9c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f528000a990 tx=0x7f5280008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:57.889+0000 7f528cff9640 1 -- 172.21.15.1:0/72070201 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f528804d020 con 0x7f5290144d10 2024-03-06T07:24:58.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.140+0000 7f5296a0e640 1 -- 172.21.15.1:0/72070201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f52900ca650 con 0x7f5290144d10 2024-03-06T07:24:58.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.142+0000 7f528cff9640 1 -- 172.21.15.1:0/72070201 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f528807bc70 con 0x7f5290144d10 2024-03-06T07:24:58.145 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-06T07:24:58.145 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng== 2024-03-06T07:24:58.145 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-06T07:24:58.145 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-06T07:24:58.145 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:58.145 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-06T07:24:58.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.145+0000 7f5296a0e640 1 -- 172.21.15.1:0/72070201 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f527006b500 msgr2=0x7f527006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.145+0000 7f5296a0e640 1 --2- 172.21.15.1:0/72070201 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f527006b500 0x7f527006d9c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f528000a990 tx=0x7f5280008040 comp rx=0 tx=0).stop 2024-03-06T07:24:58.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.145+0000 7f5296a0e640 1 -- 172.21.15.1:0/72070201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290144d10 msgr2=0x7f52900c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.145+0000 7f5296a0e640 1 --2- 172.21.15.1:0/72070201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290144d10 0x7f52900c9270 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f5288009f40 tx=0x7f5288004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:58.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.145+0000 7f5296a0e640 1 -- 172.21.15.1:0/72070201 shutdown_connections 2024-03-06T07:24:58.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.145+0000 7f5296a0e640 1 --2- 172.21.15.1:0/72070201 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f527006b500 0x7f527006d9c0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.145+0000 7f5296a0e640 1 --2- 172.21.15.1:0/72070201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290144d10 0x7f52900c9270 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.145+0000 7f5296a0e640 1 --2- 172.21.15.1:0/72070201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5290152650 0x7f5290147450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.145+0000 7f5296a0e640 1 --2- 172.21.15.1:0/72070201 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52900bf9d0 0x7f5290145240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.145+0000 7f5296a0e640 1 -- 172.21.15.1:0/72070201 >> 172.21.15.1:0/72070201 conn(0x7f5290066ee0 msgr2=0x7f529014fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:58.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.145+0000 7f5296a0e640 1 -- 172.21.15.1:0/72070201 shutdown_connections 2024-03-06T07:24:58.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.145+0000 7f5296a0e640 1 -- 172.21.15.1:0/72070201 wait complete. 2024-03-06T07:24:58.158 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:58.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.257+0000 7f55c433e640 1 Processor -- start 2024-03-06T07:24:58.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.257+0000 7f55c433e640 1 -- start start 2024-03-06T07:24:58.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.258+0000 7f55c433e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55bc0c2b30 0x7f55bc0724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.258+0000 7f55c433e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f55bc152b70 0x7f55bc072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.258+0000 7f55c433e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55bc151dd0 con 0x7f55bc0a5fa0 2024-03-06T07:24:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.258+0000 7f55c433e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55bc1526e0 con 0x7f55bc152b70 2024-03-06T07:24:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.258+0000 7f55c433e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc156190 con 0x7f55bc0c2b30 2024-03-06T07:24:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.258+0000 7f55c28db640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f55bc0a5fa0 0x7f55bc0c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41642/0 (socket says 172.21.15.1:41642) 2024-03-06T07:24:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.258+0000 7f55c28db640 1 -- 172.21.15.1:0/1204503663 learned_addr learned my addr 172.21.15.1:0/1204503663 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.258+0000 7f55c20da640 1 --2- 172.21.15.1:0/1204503663 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55bc0c2b30 0x7f55bc0724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.258+0000 7f55c18d9640 1 --2- 172.21.15.1:0/1204503663 >> v2:172.21.15.98:3300/0 conn(0x7f55bc152b70 0x7f55bc072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.258+0000 7f55c10d8640 1 -- 172.21.15.1:0/1204503663 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053149402 0 0) 0x7f55bc151dd0 con 0x7f55bc0a5fa0 2024-03-06T07:24:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.258+0000 7f55c10d8640 1 -- 172.21.15.1:0/1204503663 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55a8003520 con 0x7f55bc0a5fa0 2024-03-06T07:24:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.258+0000 7f55c10d8640 1 -- 172.21.15.1:0/1204503663 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3898092859 0 0) 0x7f55a8003520 con 0x7f55bc0a5fa0 2024-03-06T07:24:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.258+0000 7f55c10d8640 1 -- 172.21.15.1:0/1204503663 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55bc0c2b30 msgr2=0x7f55bc0724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.258+0000 7f55c10d8640 1 --2- 172.21.15.1:0/1204503663 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55bc0c2b30 0x7f55bc0724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.258+0000 7f55c10d8640 1 -- 172.21.15.1:0/1204503663 >> v2:172.21.15.98:3300/0 conn(0x7f55bc152b70 msgr2=0x7f55bc072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.258+0000 7f55c10d8640 1 --2- 172.21.15.1:0/1204503663 >> v2:172.21.15.98:3300/0 conn(0x7f55bc152b70 0x7f55bc072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.258+0000 7f55c10d8640 1 -- 172.21.15.1:0/1204503663 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55bc152090 con 0x7f55bc0a5fa0 2024-03-06T07:24:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.259+0000 7f55c10d8640 1 -- 172.21.15.1:0/1204503663 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f55b40019f0 con 0x7f55bc0a5fa0 2024-03-06T07:24:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.259+0000 7f55c10d8640 1 -- 172.21.15.1:0/1204503663 >> v1:172.21.15.1:6789/0 conn(0x7f55bc0a5fa0 legacy=0x7f55bc0c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.259+0000 7f55c10d8640 1 --2- 172.21.15.1:0/1204503663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a80039b0 0x7f55a8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.259+0000 7f55c10d8640 1 --2- 172.21.15.1:0/1204503663 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55bc0c2b30 0x7f55a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.259+0000 7f55c10d8640 1 --2- 172.21.15.1:0/1204503663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55a8004d40 0x7f55a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.259+0000 7f55c10d8640 1 -- 172.21.15.1:0/1204503663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc1526e0 con 0x7f55a8004d40 2024-03-06T07:24:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.259+0000 7f55c10d8640 1 -- 172.21.15.1:0/1204503663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc156190 con 0x7f55a80039b0 2024-03-06T07:24:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.259+0000 7f55c10d8640 1 -- 172.21.15.1:0/1204503663 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a800b9c0 con 0x7f55bc0c2b30 2024-03-06T07:24:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.259+0000 7f55c18d9640 1 --2- 172.21.15.1:0/1204503663 >> v2:172.21.15.98:3300/0 conn(0x7f55bc152b70 0x7f55bc072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.259+0000 7f55c28db640 1 --2- 172.21.15.1:0/1204503663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55a8004d40 0x7f55a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.259+0000 7f55c20da640 1 --2- 172.21.15.1:0/1204503663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a80039b0 0x7f55a8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.259+0000 7f55c18d9640 1 --2- 172.21.15.1:0/1204503663 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55bc0c2b30 0x7f55a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.259+0000 7f55c28db640 1 -- 172.21.15.1:0/1204503663 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55bc0c2b30 msgr2=0x7f55a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.259+0000 7f55c28db640 1 --2- 172.21.15.1:0/1204503663 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55bc0c2b30 0x7f55a8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.259+0000 7f55c28db640 1 -- 172.21.15.1:0/1204503663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a80039b0 msgr2=0x7f55a8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.259+0000 7f55c28db640 1 --2- 172.21.15.1:0/1204503663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a80039b0 0x7f55a8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.259+0000 7f55c28db640 1 -- 172.21.15.1:0/1204503663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a8003520 con 0x7f55a8004d40 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.260+0000 7f55c28db640 1 --2- 172.21.15.1:0/1204503663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55a8004d40 0x7f55a80051c0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f55b400a8a0 tx=0x7f55a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf2ef8ba4cde26b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.260+0000 7f55c10d8640 1 -- 172.21.15.1:0/1204503663 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b400a4c0 con 0x7f55a8004d40 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.260+0000 7f55c10d8640 1 -- 172.21.15.1:0/1204503663 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b4034650 con 0x7f55a8004d40 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.260+0000 7f55c10d8640 1 -- 172.21.15.1:0/1204503663 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b4034a40 con 0x7f55a8004d40 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.261+0000 7f55c433e640 1 -- 172.21.15.1:0/1204503663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55a8004d40 msgr2=0x7f55a80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.261+0000 7f55c433e640 1 --2- 172.21.15.1:0/1204503663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55a8004d40 0x7f55a80051c0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f55b400a8a0 tx=0x7f55a8005830 comp rx=0 tx=0).stop 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.261+0000 7f55c28db640 1 -- 172.21.15.1:0/1204503663 reap_dead start 2024-03-06T07:24:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.261+0000 7f55c433e640 1 -- 172.21.15.1:0/1204503663 shutdown_connections 2024-03-06T07:24:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.261+0000 7f55c433e640 1 -- 172.21.15.1:0/1204503663 >> 172.21.15.1:0/1204503663 conn(0x7f55bc0c6860 msgr2=0x7f55bc0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.261+0000 7f55c433e640 1 -- 172.21.15.1:0/1204503663 shutdown_connections 2024-03-06T07:24:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.261+0000 7f55c433e640 1 -- 172.21.15.1:0/1204503663 wait complete. 2024-03-06T07:24:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.262+0000 7f55c433e640 1 Processor -- start 2024-03-06T07:24:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.262+0000 7f55c433e640 1 -- start start 2024-03-06T07:24:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.262+0000 7f55c433e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a8004d40 0x7f55bc144e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.262+0000 7f55c433e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55bc151610 0x7f55bc0bead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.262+0000 7f55c433e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55bc0bf010 0x7f55bc150490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55c433e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc155f00 con 0x7f55bc151610 2024-03-06T07:24:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55c433e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc155d80 con 0x7f55a8004d40 2024-03-06T07:24:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55c433e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc0724c0 con 0x7f55bc0bf010 2024-03-06T07:24:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55c18d9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55bc151610 0x7f55bc0bead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55c18d9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55bc151610 0x7f55bc0bead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36938/0 (socket says 172.21.15.1:36938) 2024-03-06T07:24:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55c18d9640 1 -- 172.21.15.1:0/1836102983 learned_addr learned my addr 172.21.15.1:0/1836102983 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55c20da640 1 --2- 172.21.15.1:0/1836102983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a8004d40 0x7f55bc144e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55c28db640 1 --2- 172.21.15.1:0/1836102983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55bc0bf010 0x7f55bc150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55c18d9640 1 -- 172.21.15.1:0/1836102983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55bc0bf010 msgr2=0x7f55bc150490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55c18d9640 1 --2- 172.21.15.1:0/1836102983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55bc0bf010 0x7f55bc150490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55c18d9640 1 -- 172.21.15.1:0/1836102983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a8004d40 msgr2=0x7f55bc144e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55c18d9640 1 --2- 172.21.15.1:0/1836102983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a8004d40 0x7f55bc144e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55c18d9640 1 -- 172.21.15.1:0/1836102983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55bc1453d0 con 0x7f55bc151610 2024-03-06T07:24:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55c28db640 1 --2- 172.21.15.1:0/1836102983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55bc0bf010 0x7f55bc150490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55c18d9640 1 --2- 172.21.15.1:0/1836102983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55bc151610 0x7f55bc0bead0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f55ac011960 tx=0x7f55ac011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55b2ffd640 1 -- 172.21.15.1:0/1836102983 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac00fb70 con 0x7f55bc151610 2024-03-06T07:24:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55b2ffd640 1 -- 172.21.15.1:0/1836102983 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac011e90 con 0x7f55bc151610 2024-03-06T07:24:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55b2ffd640 1 -- 172.21.15.1:0/1836102983 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac015460 con 0x7f55bc151610 2024-03-06T07:24:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.263+0000 7f55c433e640 1 -- 172.21.15.1:0/1836102983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55bc0bf610 con 0x7f55bc151610 2024-03-06T07:24:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.264+0000 7f55c433e640 1 -- 172.21.15.1:0/1836102983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55bc0ac980 con 0x7f55bc151610 2024-03-06T07:24:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.264+0000 7f55c433e640 1 -- 172.21.15.1:0/1836102983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55bc067fd0 con 0x7f55bc151610 2024-03-06T07:24:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.265+0000 7f55b2ffd640 1 -- 172.21.15.1:0/1836102983 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55ac015600 con 0x7f55bc151610 2024-03-06T07:24:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.265+0000 7f55b2ffd640 1 --2- 172.21.15.1:0/1836102983 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55a006b4d0 0x7f55a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.265+0000 7f55b2ffd640 1 -- 172.21.15.1:0/1836102983 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f55ac08ad70 con 0x7f55bc151610 2024-03-06T07:24:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.265+0000 7f55c20da640 1 --2- 172.21.15.1:0/1836102983 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55a006b4d0 0x7f55a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.266+0000 7f55c20da640 1 --2- 172.21.15.1:0/1836102983 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55a006b4d0 0x7f55a006d990 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009bc0 tx=0x7f55b8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.269+0000 7f55b2ffd640 1 -- 172.21.15.1:0/1836102983 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f55bc1453d0 con 0x7f55bc151610 2024-03-06T07:24:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.488+0000 7f55c433e640 1 -- 172.21.15.1:0/1836102983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55bc071790 con 0x7f55bc151610 2024-03-06T07:24:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.488+0000 7f55b2ffd640 1 -- 172.21.15.1:0/1836102983 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4976 (secure 0 0 0) 0x7f55ac019030 con 0x7f55bc151610 2024-03-06T07:24:58.490 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:58.490 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":106,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6446,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4151599120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4151599120},{"type":"v1","addr":"172.21.15.1:6835","nonce":4151599120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/155600632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":155600632},{"type":"v1","addr":"172.21.15.98:6837","nonce":155600632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7084,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1420541659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1420541659},{"type":"v1","addr":"172.21.15.98:6836","nonce":1420541659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.598303+0000","modified":"2024-03-06T07:24:55.621912+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7863},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7863":{"gid":7863,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1515309674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1515309674},{"type":"v1","addr":"172.21.15.1:6834","nonce":1515309674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7863,"qdb_cluster":[7863]},"id":9}]} 2024-03-06T07:24:58.491 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 106 2024-03-06T07:24:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.490+0000 7f55c433e640 1 -- 172.21.15.1:0/1836102983 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55a006b4d0 msgr2=0x7f55a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.490+0000 7f55c433e640 1 --2- 172.21.15.1:0/1836102983 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55a006b4d0 0x7f55a006d990 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009bc0 tx=0x7f55b8009190 comp rx=0 tx=0).stop 2024-03-06T07:24:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.490+0000 7f55c433e640 1 -- 172.21.15.1:0/1836102983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55bc151610 msgr2=0x7f55bc0bead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.490+0000 7f55c433e640 1 --2- 172.21.15.1:0/1836102983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55bc151610 0x7f55bc0bead0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f55ac011960 tx=0x7f55ac011d60 comp rx=0 tx=0).stop 2024-03-06T07:24:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.490+0000 7f55c433e640 1 -- 172.21.15.1:0/1836102983 shutdown_connections 2024-03-06T07:24:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.490+0000 7f55c433e640 1 --2- 172.21.15.1:0/1836102983 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f55a006b4d0 0x7f55a006d990 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.490+0000 7f55c433e640 1 --2- 172.21.15.1:0/1836102983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55bc0bf010 0x7f55bc150490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.490+0000 7f55c433e640 1 --2- 172.21.15.1:0/1836102983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55bc151610 0x7f55bc0bead0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.490+0000 7f55c433e640 1 --2- 172.21.15.1:0/1836102983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a8004d40 0x7f55bc144e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.490+0000 7f55c433e640 1 -- 172.21.15.1:0/1836102983 >> 172.21.15.1:0/1836102983 conn(0x7f55bc0c6860 msgr2=0x7f55bc0c2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.491+0000 7f55c433e640 1 -- 172.21.15.1:0/1836102983 shutdown_connections 2024-03-06T07:24:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:58.491+0000 7f55c433e640 1 -- 172.21.15.1:0/1836102983 wait complete. 2024-03-06T07:24:58.530 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-06T07:24:58.528+0000 7f0046de1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:58.906 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:52.598303+0000', 'modified': '2024-03-06T07:24:55.621912+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7863}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7863': {'gid': 7863, 'name': 'c', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6834/1515309674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1515309674}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1515309674}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7863, 'qdb_cluster': [7863]} 2024-03-06T07:24:58.906 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:24:58.906 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:24:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.007+0000 7f4a7ebf7640 1 Processor -- start 2024-03-06T07:24:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.007+0000 7f4a7ebf7640 1 -- start start 2024-03-06T07:24:59.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.008+0000 7f4a7ebf7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4a78152650 0x7f4a78155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.008+0000 7f4a7ebf7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a780bf9d0 0x7f4a780c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.008+0000 7f4a7ebf7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a780c2fc0 con 0x7f4a780c6b10 2024-03-06T07:24:59.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.008+0000 7f4a7ebf7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a78155800 con 0x7f4a78152650 2024-03-06T07:24:59.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.008+0000 7f4a7ebf7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a78158da0 con 0x7f4a780bf9d0 2024-03-06T07:24:59.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.008+0000 7f4a7c993640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4a78152650 0x7f4a78155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.008+0000 7f4a77fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4a780c6b10 0x7f4a78158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41654/0 (socket says 172.21.15.1:41654) 2024-03-06T07:24:59.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.008+0000 7f4a77fff640 1 -- 172.21.15.1:0/1352024321 learned_addr learned my addr 172.21.15.1:0/1352024321 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:59.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.008+0000 7f4a7d194640 1 --2- 172.21.15.1:0/1352024321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a780bf9d0 0x7f4a780c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.008+0000 7f4a7c993640 1 -- 172.21.15.1:0/1352024321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a780bf9d0 msgr2=0x7f4a780c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.008+0000 7f4a7c993640 1 --2- 172.21.15.1:0/1352024321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a780bf9d0 0x7f4a780c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.008+0000 7f4a7c993640 1 -- 172.21.15.1:0/1352024321 >> v1:172.21.15.1:6789/0 conn(0x7f4a780c6b10 legacy=0x7f4a78158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.008+0000 7f4a777fe640 1 -- 172.21.15.1:0/1352024321 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661647231 0 0) 0x7f4a780c2fc0 con 0x7f4a780c6b10 2024-03-06T07:24:59.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.008+0000 7f4a7c993640 1 -- 172.21.15.1:0/1352024321 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a780c6690 con 0x7f4a78152650 2024-03-06T07:24:59.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.008+0000 7f4a7d194640 1 --2- 172.21.15.1:0/1352024321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a780bf9d0 0x7f4a780c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:59.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.009+0000 7f4a7c993640 1 --2- 172.21.15.1:0/1352024321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78152650 0x7f4a78155c90 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0097a0 tx=0x7f4a6c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f4c487cf54cba6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.009+0000 7f4a777fe640 1 -- 172.21.15.1:0/1352024321 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c0315a0 con 0x7f4a78152650 2024-03-06T07:24:59.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.009+0000 7f4a777fe640 1 -- 172.21.15.1:0/1352024321 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6c037070 con 0x7f4a78152650 2024-03-06T07:24:59.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.009+0000 7f4a777fe640 1 -- 172.21.15.1:0/1352024321 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c031960 con 0x7f4a78152650 2024-03-06T07:24:59.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.010+0000 7f4a7ebf7640 1 -- 172.21.15.1:0/1352024321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78152650 msgr2=0x7f4a78155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.010+0000 7f4a7ebf7640 1 --2- 172.21.15.1:0/1352024321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78152650 0x7f4a78155c90 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0097a0 tx=0x7f4a6c02b840 comp rx=0 tx=0).stop 2024-03-06T07:24:59.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.010+0000 7f4a7ebf7640 1 -- 172.21.15.1:0/1352024321 shutdown_connections 2024-03-06T07:24:59.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.010+0000 7f4a7ebf7640 1 --2- 172.21.15.1:0/1352024321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a780bf9d0 0x7f4a780c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.010+0000 7f4a7ebf7640 1 --2- 172.21.15.1:0/1352024321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a78152650 0x7f4a78155c90 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.010+0000 7f4a7ebf7640 1 -- 172.21.15.1:0/1352024321 >> 172.21.15.1:0/1352024321 conn(0x7f4a78066ee0 msgr2=0x7f4a780672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.010+0000 7f4a7ebf7640 1 -- 172.21.15.1:0/1352024321 shutdown_connections 2024-03-06T07:24:59.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.010+0000 7f4a7ebf7640 1 -- 172.21.15.1:0/1352024321 wait complete. 2024-03-06T07:24:59.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.011+0000 7f4a7ebf7640 1 Processor -- start 2024-03-06T07:24:59.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.011+0000 7f4a7ebf7640 1 -- start start 2024-03-06T07:24:59.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.012+0000 7f4a7ebf7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a780bf9d0 0x7f4a781e18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.012+0000 7f4a7ebf7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a78152650 0x7f4a781df4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.012+0000 7f4a7ebf7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a781e09a0 0x7f4a781eddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.012+0000 7f4a7ebf7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a781551b0 con 0x7f4a781e09a0 2024-03-06T07:24:59.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.012+0000 7f4a7ebf7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a780c6fa0 con 0x7f4a78152650 2024-03-06T07:24:59.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.012+0000 7f4a7ebf7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a78155490 con 0x7f4a780bf9d0 2024-03-06T07:24:59.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.012+0000 7f4a7d194640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a781e09a0 0x7f4a781eddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.012+0000 7f4a7d194640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a781e09a0 0x7f4a781eddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36968/0 (socket says 172.21.15.1:36968) 2024-03-06T07:24:59.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.012+0000 7f4a7d194640 1 -- 172.21.15.1:0/897455029 learned_addr learned my addr 172.21.15.1:0/897455029 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:59.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.012+0000 7f4a7c993640 1 --2- 172.21.15.1:0/897455029 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a780bf9d0 0x7f4a781e18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.012+0000 7f4a77fff640 1 --2- 172.21.15.1:0/897455029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a78152650 0x7f4a781df4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.012+0000 7f4a7d194640 1 -- 172.21.15.1:0/897455029 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a780bf9d0 msgr2=0x7f4a781e18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.012+0000 7f4a7d194640 1 --2- 172.21.15.1:0/897455029 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a780bf9d0 0x7f4a781e18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.012+0000 7f4a7d194640 1 -- 172.21.15.1:0/897455029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a78152650 msgr2=0x7f4a781df4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.012+0000 7f4a7d194640 1 --2- 172.21.15.1:0/897455029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a78152650 0x7f4a781df4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.012+0000 7f4a7d194640 1 -- 172.21.15.1:0/897455029 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a781de950 con 0x7f4a781e09a0 2024-03-06T07:24:59.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.013+0000 7f4a7d194640 1 --2- 172.21.15.1:0/897455029 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a781e09a0 0x7f4a781eddf0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f4a7000cd80 tx=0x7f4a7000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.013+0000 7f4a757fa640 1 -- 172.21.15.1:0/897455029 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7000f680 con 0x7f4a781e09a0 2024-03-06T07:24:59.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.013+0000 7f4a757fa640 1 -- 172.21.15.1:0/897455029 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7000f840 con 0x7f4a781e09a0 2024-03-06T07:24:59.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.013+0000 7f4a7ebf7640 1 -- 172.21.15.1:0/897455029 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a781dec30 con 0x7f4a781e09a0 2024-03-06T07:24:59.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.013+0000 7f4a757fa640 1 -- 172.21.15.1:0/897455029 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7000fc30 con 0x7f4a781e09a0 2024-03-06T07:24:59.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.013+0000 7f4a7ebf7640 1 -- 172.21.15.1:0/897455029 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a781df9e0 con 0x7f4a781e09a0 2024-03-06T07:24:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.014+0000 7f4a757fa640 1 -- 172.21.15.1:0/897455029 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a700213f0 con 0x7f4a781e09a0 2024-03-06T07:24:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.015+0000 7f4a757fa640 1 --2- 172.21.15.1:0/897455029 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a5c06b4d0 0x7f4a5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.015+0000 7f4a757fa640 1 -- 172.21.15.1:0/897455029 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4a7008b250 con 0x7f4a781e09a0 2024-03-06T07:24:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.015+0000 7f4a7c993640 1 --2- 172.21.15.1:0/897455029 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a5c06b4d0 0x7f4a5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.016+0000 7f4a7c993640 1 --2- 172.21.15.1:0/897455029 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a5c06b4d0 0x7f4a5c06d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c036000 tx=0x7f4a6c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.016+0000 7f4a817e9740 1 Processor -- start 2024-03-06T07:24:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.016+0000 7f4a817e9740 1 -- start start 2024-03-06T07:24:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.016+0000 7f4a817e9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c8760a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.016+0000 7f4a817e9740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c8763330 0x5578c8760f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.016+0000 7f4a817e9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c87614b0 0x5578c875d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.016+0000 7f4a817e9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578c86c76b0 con 0x5578c8762f50 2024-03-06T07:24:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.016+0000 7f4a817e9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578c8670a60 con 0x5578c87614b0 2024-03-06T07:24:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.016+0000 7f4a817e9740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5578c86c3750 con 0x5578c8763330 2024-03-06T07:24:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.016+0000 7f4a7c993640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c8760a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.016+0000 7f4a7c993640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c8760a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36976/0 (socket says 172.21.15.1:36976) 2024-03-06T07:24:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.016+0000 7f4a7c993640 1 -- 172.21.15.1:0/844721681 learned_addr learned my addr 172.21.15.1:0/844721681 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.016+0000 7f4a77fff640 1 --2- 172.21.15.1:0/844721681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c8763330 0x5578c8760f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.017+0000 7f4a7d194640 1 --2- 172.21.15.1:0/844721681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c87614b0 0x5578c875d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.017+0000 7f4a7c993640 1 -- 172.21.15.1:0/844721681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c8763330 msgr2=0x5578c8760f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.017+0000 7f4a7c993640 1 --2- 172.21.15.1:0/844721681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c8763330 0x5578c8760f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.017+0000 7f4a7c993640 1 -- 172.21.15.1:0/844721681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c87614b0 msgr2=0x5578c875d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.017+0000 7f4a7c993640 1 --2- 172.21.15.1:0/844721681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c87614b0 0x5578c875d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.017+0000 7f4a7c993640 1 -- 172.21.15.1:0/844721681 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578c868ef30 con 0x5578c8762f50 2024-03-06T07:24:59.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.017+0000 7f4a7d194640 1 --2- 172.21.15.1:0/844721681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c87614b0 0x5578c875d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:59.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.017+0000 7f4a77fff640 1 --2- 172.21.15.1:0/844721681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c8763330 0x5578c8760f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:59.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.017+0000 7f4a7c993640 1 --2- 172.21.15.1:0/844721681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c8760a30 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c02b880 tx=0x7f4a6c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c7add0afaa1db3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.017+0000 7f4a627fc640 1 -- 172.21.15.1:0/844721681 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c032b80 con 0x5578c8762f50 2024-03-06T07:24:59.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.017+0000 7f4a627fc640 1 -- 172.21.15.1:0/844721681 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6c037040 con 0x5578c8762f50 2024-03-06T07:24:59.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.017+0000 7f4a627fc640 1 -- 172.21.15.1:0/844721681 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c051430 con 0x5578c8762f50 2024-03-06T07:24:59.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.018+0000 7f4a817e9740 1 -- 172.21.15.1:0/844721681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 msgr2=0x5578c8760a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.018+0000 7f4a817e9740 1 --2- 172.21.15.1:0/844721681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c8760a30 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c02b880 tx=0x7f4a6c030000 comp rx=0 tx=0).stop 2024-03-06T07:24:59.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.018+0000 7f4a817e9740 1 -- 172.21.15.1:0/844721681 shutdown_connections 2024-03-06T07:24:59.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.018+0000 7f4a817e9740 1 --2- 172.21.15.1:0/844721681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c87614b0 0x5578c875d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.018+0000 7f4a817e9740 1 --2- 172.21.15.1:0/844721681 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c8763330 0x5578c8760f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.018+0000 7f4a817e9740 1 --2- 172.21.15.1:0/844721681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c8760a30 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.018+0000 7f4a817e9740 1 -- 172.21.15.1:0/844721681 >> 172.21.15.1:0/844721681 conn(0x5578c875f7d0 msgr2=0x5578c87843c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.018+0000 7f4a817e9740 1 -- 172.21.15.1:0/844721681 shutdown_connections 2024-03-06T07:24:59.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.018+0000 7f4a817e9740 1 -- 172.21.15.1:0/844721681 wait complete. 2024-03-06T07:24:59.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.018+0000 7f4a817e9740 1 -- start start 2024-03-06T07:24:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.018+0000 7f4a817e9740 1 Processor -- start 2024-03-06T07:24:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.019+0000 7f4a817e9740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c8762f50 0x5578c879f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.019+0000 7f4a817e9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8763330 0x5578c87b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.019+0000 7f4a817e9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c878ac10 0x5578c879ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.019+0000 7f4a817e9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578c8785230 con 0x5578c8763330 2024-03-06T07:24:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.019+0000 7f4a817e9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578c8683ea0 con 0x5578c878ac10 2024-03-06T07:24:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.019+0000 7f4a817e9740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5578c8784c70 con 0x5578c8762f50 2024-03-06T07:24:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.019+0000 7f4a817e9740 10 client.0 _ll_register_callbacks cb 0x5578c8683870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.019+0000 7f4a77fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8763330 0x5578c87b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.019+0000 7f4a77fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8763330 0x5578c87b36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36982/0 (socket says 172.21.15.1:36982) 2024-03-06T07:24:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.019+0000 7f4a77fff640 1 -- 172.21.15.1:0/1513326338 learned_addr learned my addr 172.21.15.1:0/1513326338 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.019+0000 7f4a7c993640 1 --2- 172.21.15.1:0/1513326338 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c8762f50 0x5578c879f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.019+0000 7f4a7d194640 1 --2- 172.21.15.1:0/1513326338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c878ac10 0x5578c879ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a77fff640 1 -- 172.21.15.1:0/1513326338 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c8762f50 msgr2=0x5578c879f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a77fff640 1 --2- 172.21.15.1:0/1513326338 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c8762f50 0x5578c879f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a77fff640 1 -- 172.21.15.1:0/1513326338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c878ac10 msgr2=0x5578c879ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a77fff640 1 --2- 172.21.15.1:0/1513326338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c878ac10 0x5578c879ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a77fff640 1 -- 172.21.15.1:0/1513326338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578c8684270 con 0x5578c8763330 2024-03-06T07:24:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a77fff640 1 --2- 172.21.15.1:0/1513326338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8763330 0x5578c87b36a0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f4a780c5280 tx=0x7f4a6400f660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f21c0c7e582eda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a577fe640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:24:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a577fe640 1 -- 172.21.15.1:0/1513326338 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a640058b0 con 0x5578c8763330 2024-03-06T07:24:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a577fe640 1 -- 172.21.15.1:0/1513326338 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4a64005a50 con 0x5578c8763330 2024-03-06T07:24:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a817e9740 1 -- 172.21.15.1:0/1513326338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4a6c037040 con 0x5578c8763330 2024-03-06T07:24:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a577fe640 1 -- 172.21.15.1:0/1513326338 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a64020050 con 0x5578c8763330 2024-03-06T07:24:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a817e9740 10 client.8010 fetch_fsmap learned FSMap version 106 2024-03-06T07:24:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a817e9740 1 -- 172.21.15.1:0/1513326338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=2+}) v3 -- 0x5578c83dc2d0 con 0x5578c8763330 2024-03-06T07:24:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a577fe640 1 -- 172.21.15.1:0/1513326338 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6c037040 con 0x5578c8763330 2024-03-06T07:24:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a577fe640 1 -- 172.21.15.1:0/1513326338 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 106) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4a64023020 con 0x5578c8763330 2024-03-06T07:24:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a817e9740 10 client.8010 fetch_fsmap finished waiting for FSMap version 106 2024-03-06T07:24:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a817e9740 10 client.8010 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:24:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a817e9740 20 client.8010 populate_metadata read hostname 'smithi001' 2024-03-06T07:24:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a817e9740 1 --2- 172.21.15.1:0/1513326338 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x5578c87b0220 0x5578c87be510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a817e9740 4 client.8010 mds_command: new command op to 7863 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:24:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.020+0000 7f4a817e9740 1 -- 172.21.15.1:0/1513326338 --> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5578c86d4bf0 con 0x5578c87b0220 2024-03-06T07:24:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.021+0000 7f4a7c993640 1 --2- 172.21.15.1:0/1513326338 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x5578c87b0220 0x5578c87be510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.021+0000 7f4a7c993640 1 --2- 172.21.15.1:0/1513326338 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x5578c87b0220 0x5578c87be510 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42392e30e72b760b server_cookie=cc746cd72d5cdd5b in_seq=0 out_seq=0 2024-03-06T07:24:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.021+0000 7f4a577fe640 10 client.8010 ms_handle_connect on v2:172.21.15.1:6832/1515309674 2024-03-06T07:24:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.021+0000 7f4a577fe640 1 -- 172.21.15.1:0/1513326338 <== mds.0 v2:172.21.15.1:6832/1515309674 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5578c86d4bf0 con 0x5578c87b0220 2024-03-06T07:24:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.021+0000 7f4a577fe640 10 client.8010 handle_command_reply: tid=0 2024-03-06T07:24:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.021+0000 7f4a817e9740 1 client.8010 shutdown 2024-03-06T07:24:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.022+0000 7f4a817e9740 1 -- 172.21.15.1:0/1513326338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8763330 msgr2=0x5578c87b36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.022+0000 7f4a817e9740 1 --2- 172.21.15.1:0/1513326338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8763330 0x5578c87b36a0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f4a780c5280 tx=0x7f4a6400f660 comp rx=0 tx=0).stop 2024-03-06T07:24:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.022+0000 7f4a817e9740 1 -- 172.21.15.1:0/1513326338 shutdown_connections 2024-03-06T07:24:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.022+0000 7f4a817e9740 1 --2- 172.21.15.1:0/1513326338 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x5578c87b0220 0x5578c87be510 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.022+0000 7f4a817e9740 1 --2- 172.21.15.1:0/1513326338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c878ac10 0x5578c879ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.022+0000 7f4a817e9740 1 --2- 172.21.15.1:0/1513326338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8763330 0x5578c87b36a0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.022+0000 7f4a817e9740 1 --2- 172.21.15.1:0/1513326338 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c8762f50 0x5578c879f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.022+0000 7f4a577fe640 0 client.8010 ms_handle_reset on v2:172.21.15.1:6832/1515309674 2024-03-06T07:24:59.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.022+0000 7f4a817e9740 1 -- 172.21.15.1:0/1513326338 >> 172.21.15.1:0/1513326338 conn(0x5578c87614b0 msgr2=0x5578c87840f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.022+0000 7f4a817e9740 1 -- 172.21.15.1:0/1513326338 shutdown_connections 2024-03-06T07:24:59.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.022+0000 7f4a817e9740 1 -- 172.21.15.1:0/1513326338 wait complete. 2024-03-06T07:24:59.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.022+0000 7f4a817e9740 20 client.8010 trim_cache size 0 max 16384 2024-03-06T07:24:59.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.039+0000 7f4a817e9740 1 Processor -- start 2024-03-06T07:24:59.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.039+0000 7f4a817e9740 1 -- start start 2024-03-06T07:24:59.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.039+0000 7f4a817e9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c875e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.039+0000 7f4a817e9740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c8763330 0x5578c87605a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.039+0000 7f4a817e9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c87a1dc0 0x5578c8760ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.039+0000 7f4a817e9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a781f6ba0 con 0x5578c8762f50 2024-03-06T07:24:59.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.039+0000 7f4a817e9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7814d4b0 con 0x5578c87a1dc0 2024-03-06T07:24:59.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.039+0000 7f4a817e9740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a781f65d0 con 0x5578c8763330 2024-03-06T07:24:59.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.039+0000 7f4a7c993640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c875e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.039+0000 7f4a7c993640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c875e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36988/0 (socket says 172.21.15.1:36988) 2024-03-06T07:24:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.039+0000 7f4a7c993640 1 -- 172.21.15.1:0/3028535755 learned_addr learned my addr 172.21.15.1:0/3028535755 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.039+0000 7f4a7d194640 1 --2- 172.21.15.1:0/3028535755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c87a1dc0 0x5578c8760ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.039+0000 7f4a7c993640 1 -- 172.21.15.1:0/3028535755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c8763330 msgr2=0x5578c87605a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.039+0000 7f4a7c993640 1 --2- 172.21.15.1:0/3028535755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c8763330 0x5578c87605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.040+0000 7f4a7c993640 1 -- 172.21.15.1:0/3028535755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c87a1dc0 msgr2=0x5578c8760ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.040+0000 7f4a7c993640 1 --2- 172.21.15.1:0/3028535755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c87a1dc0 0x5578c8760ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.040+0000 7f4a7c993640 1 -- 172.21.15.1:0/3028535755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578c87a1ae0 con 0x5578c8762f50 2024-03-06T07:24:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.040+0000 7f4a7d194640 1 --2- 172.21.15.1:0/3028535755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c87a1dc0 0x5578c8760ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.040+0000 7f4a7c993640 1 --2- 172.21.15.1:0/3028535755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c875e7b0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c02be50 tx=0x7f4a6c009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b2266983297e24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.040+0000 7f4a61ffb640 1 -- 172.21.15.1:0/3028535755 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c031e50 con 0x5578c8762f50 2024-03-06T07:24:59.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.040+0000 7f4a61ffb640 1 -- 172.21.15.1:0/3028535755 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6c037040 con 0x5578c8762f50 2024-03-06T07:24:59.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.040+0000 7f4a61ffb640 1 -- 172.21.15.1:0/3028535755 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c032ba0 con 0x5578c8762f50 2024-03-06T07:24:59.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.040+0000 7f4a817e9740 1 -- 172.21.15.1:0/3028535755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 msgr2=0x5578c875e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.040+0000 7f4a817e9740 1 --2- 172.21.15.1:0/3028535755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c875e7b0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c02be50 tx=0x7f4a6c009190 comp rx=0 tx=0).stop 2024-03-06T07:24:59.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.040+0000 7f4a817e9740 1 -- 172.21.15.1:0/3028535755 shutdown_connections 2024-03-06T07:24:59.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.040+0000 7f4a817e9740 1 --2- 172.21.15.1:0/3028535755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c87a1dc0 0x5578c8760ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.040+0000 7f4a817e9740 1 --2- 172.21.15.1:0/3028535755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c8763330 0x5578c87605a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.040+0000 7f4a817e9740 1 --2- 172.21.15.1:0/3028535755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c875e7b0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.040+0000 7f4a817e9740 1 -- 172.21.15.1:0/3028535755 >> 172.21.15.1:0/3028535755 conn(0x5578c87614b0 msgr2=0x5578c879b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.040+0000 7f4a817e9740 1 -- 172.21.15.1:0/3028535755 shutdown_connections 2024-03-06T07:24:59.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.040+0000 7f4a817e9740 1 -- 172.21.15.1:0/3028535755 wait complete. 2024-03-06T07:24:59.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.041+0000 7f4a817e9740 1 -- start start 2024-03-06T07:24:59.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.041+0000 7f4a817e9740 1 Processor -- start 2024-03-06T07:24:59.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a817e9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c87ac330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a817e9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c8763330 0x5578c875c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a817e9740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c87a1dc0 0x5578c880d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a817e9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578c8785890 con 0x5578c8762f50 2024-03-06T07:24:59.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a817e9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578c8789560 con 0x5578c8763330 2024-03-06T07:24:59.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a817e9740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5578c87b3c80 con 0x5578c87a1dc0 2024-03-06T07:24:59.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a817e9740 10 client.0 _ll_register_callbacks cb 0x7f4a64001a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:59.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a7c993640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c87ac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a7c993640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c87ac330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36994/0 (socket says 172.21.15.1:36994) 2024-03-06T07:24:59.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a7c993640 1 -- 172.21.15.1:0/3536295594 learned_addr learned my addr 172.21.15.1:0/3536295594 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:59.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a77fff640 1 --2- 172.21.15.1:0/3536295594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c8763330 0x5578c875c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a7d194640 1 --2- 172.21.15.1:0/3536295594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c87a1dc0 0x5578c880d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a7c993640 1 -- 172.21.15.1:0/3536295594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c87a1dc0 msgr2=0x5578c880d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a7c993640 1 --2- 172.21.15.1:0/3536295594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c87a1dc0 0x5578c880d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a7c993640 1 -- 172.21.15.1:0/3536295594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c8763330 msgr2=0x5578c875c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a7c993640 1 --2- 172.21.15.1:0/3536295594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c8763330 0x5578c875c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a7c993640 1 -- 172.21.15.1:0/3536295594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578c84a66b0 con 0x5578c8762f50 2024-03-06T07:24:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a77fff640 1 --2- 172.21.15.1:0/3536295594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c8763330 0x5578c875c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a7c993640 1 --2- 172.21.15.1:0/3536295594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c87ac330 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00b0b0 tx=0x7f4a6c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2babef311b19e742 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.042+0000 7f4a577fe640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:24:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.043+0000 7f4a577fe640 1 -- 172.21.15.1:0/3536295594 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c032ba0 con 0x5578c8762f50 2024-03-06T07:24:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.043+0000 7f4a577fe640 1 -- 172.21.15.1:0/3536295594 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4a6c032d40 con 0x5578c8762f50 2024-03-06T07:24:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.043+0000 7f4a817e9740 1 -- 172.21.15.1:0/3536295594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4a6c037040 con 0x5578c8762f50 2024-03-06T07:24:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.043+0000 7f4a577fe640 1 -- 172.21.15.1:0/3536295594 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a6c06e050 con 0x5578c8762f50 2024-03-06T07:24:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.043+0000 7f4a817e9740 10 client.8016 fetch_fsmap learned FSMap version 106 2024-03-06T07:24:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.043+0000 7f4a817e9740 1 -- 172.21.15.1:0/3536295594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=2+}) v3 -- 0x5578c8813ef0 con 0x5578c8762f50 2024-03-06T07:24:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.043+0000 7f4a577fe640 1 -- 172.21.15.1:0/3536295594 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6c037040 con 0x5578c8762f50 2024-03-06T07:24:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.043+0000 7f4a577fe640 1 -- 172.21.15.1:0/3536295594 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 106) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4a6c071020 con 0x5578c8762f50 2024-03-06T07:24:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.043+0000 7f4a817e9740 10 client.8016 fetch_fsmap finished waiting for FSMap version 106 2024-03-06T07:24:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.043+0000 7f4a817e9740 10 client.8016 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:24:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.043+0000 7f4a817e9740 20 client.8016 populate_metadata read hostname 'smithi001' 2024-03-06T07:24:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.043+0000 7f4a817e9740 1 --2- 172.21.15.1:0/3536295594 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x5578c880d5e0 0x5578c879db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.043+0000 7f4a817e9740 4 client.8016 mds_command: new command op to 7863 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:24:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.043+0000 7f4a817e9740 1 -- 172.21.15.1:0/3536295594 --> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5578c86cef40 con 0x5578c880d5e0 2024-03-06T07:24:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.043+0000 7f4a77fff640 1 --2- 172.21.15.1:0/3536295594 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x5578c880d5e0 0x5578c879db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.043+0000 7f4a577fe640 10 client.8016 ms_handle_connect on v2:172.21.15.1:6832/1515309674 2024-03-06T07:24:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.043+0000 7f4a77fff640 1 --2- 172.21.15.1:0/3536295594 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x5578c880d5e0 0x5578c879db40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3906d7defca37dd server_cookie=fff9cf35d3426050 in_seq=0 out_seq=0 2024-03-06T07:24:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a577fe640 1 -- 172.21.15.1:0/3536295594 <== mds.0 v2:172.21.15.1:6832/1515309674 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x5578c86cef40 con 0x5578c880d5e0 2024-03-06T07:24:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a577fe640 10 client.8016 handle_command_reply: tid=0 2024-03-06T07:24:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a817e9740 1 client.8016 shutdown 2024-03-06T07:24:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a817e9740 1 -- 172.21.15.1:0/3536295594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 msgr2=0x5578c87ac330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a817e9740 1 --2- 172.21.15.1:0/3536295594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c87ac330 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00b0b0 tx=0x7f4a6c02e000 comp rx=0 tx=0).stop 2024-03-06T07:24:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a817e9740 1 -- 172.21.15.1:0/3536295594 shutdown_connections 2024-03-06T07:24:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a817e9740 1 --2- 172.21.15.1:0/3536295594 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x5578c880d5e0 0x5578c879db40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a817e9740 1 --2- 172.21.15.1:0/3536295594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5578c87a1dc0 0x5578c880d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a817e9740 1 --2- 172.21.15.1:0/3536295594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5578c8763330 0x5578c875c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a577fe640 0 client.8016 ms_handle_reset on v2:172.21.15.1:6832/1515309674 2024-03-06T07:24:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a817e9740 1 --2- 172.21.15.1:0/3536295594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5578c8762f50 0x5578c87ac330 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a817e9740 1 -- 172.21.15.1:0/3536295594 >> 172.21.15.1:0/3536295594 conn(0x5578c87614b0 msgr2=0x5578c87b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a817e9740 1 -- 172.21.15.1:0/3536295594 shutdown_connections 2024-03-06T07:24:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a817e9740 1 -- 172.21.15.1:0/3536295594 wait complete. 2024-03-06T07:24:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a817e9740 20 client.8016 trim_cache size 0 max 16384 2024-03-06T07:24:59.054 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:24:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:24:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:24:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 7863, 2024-03-06T07:24:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:24:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:24:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:24:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 5.7978204140000003, 2024-03-06T07:24:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 106, 2024-03-06T07:24:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 121, 2024-03-06T07:24:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 121, 2024-03-06T07:24:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 10.888957438 2024-03-06T07:24:59.055 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:24:59.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a7ebf7640 1 -- 172.21.15.1:0/897455029 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a5c06b4d0 msgr2=0x7f4a5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a7ebf7640 1 --2- 172.21.15.1:0/897455029 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a5c06b4d0 0x7f4a5c06d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c036000 tx=0x7f4a6c008030 comp rx=0 tx=0).stop 2024-03-06T07:24:59.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a7ebf7640 1 -- 172.21.15.1:0/897455029 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a781e09a0 msgr2=0x7f4a781eddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.044+0000 7f4a7ebf7640 1 --2- 172.21.15.1:0/897455029 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a781e09a0 0x7f4a781eddf0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f4a7000cd80 tx=0x7f4a7000a650 comp rx=0 tx=0).stop 2024-03-06T07:24:59.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.045+0000 7f4a7ebf7640 1 -- 172.21.15.1:0/897455029 shutdown_connections 2024-03-06T07:24:59.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.045+0000 7f4a7ebf7640 1 --2- 172.21.15.1:0/897455029 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a5c06b4d0 0x7f4a5c06d990 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.045+0000 7f4a7ebf7640 1 --2- 172.21.15.1:0/897455029 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a781e09a0 0x7f4a781eddf0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.045+0000 7f4a7ebf7640 1 --2- 172.21.15.1:0/897455029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a78152650 0x7f4a781df4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.045+0000 7f4a7ebf7640 1 --2- 172.21.15.1:0/897455029 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a780bf9d0 0x7f4a781e18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.045+0000 7f4a7ebf7640 1 -- 172.21.15.1:0/897455029 >> 172.21.15.1:0/897455029 conn(0x7f4a78066ee0 msgr2=0x7f4a780672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.045+0000 7f4a7ebf7640 1 -- 172.21.15.1:0/897455029 shutdown_connections 2024-03-06T07:24:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.045+0000 7f4a7ebf7640 1 -- 172.21.15.1:0/897455029 wait complete. 2024-03-06T07:24:59.058 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:59.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.157+0000 7f5ed61a0640 1 Processor -- start 2024-03-06T07:24:59.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.158+0000 7f5ed61a0640 1 -- start start 2024-03-06T07:24:59.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.159+0000 7f5ed61a0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5ed00c2390 0x7f5ed00c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.159+0000 7f5ed61a0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5ed00c46f0 0x7f5ed014ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.159+0000 7f5ed61a0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed0157340 con 0x7f5ed0150510 2024-03-06T07:24:59.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.159+0000 7f5ed61a0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed0156580 con 0x7f5ed00c46f0 2024-03-06T07:24:59.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.159+0000 7f5ed61a0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed0156860 con 0x7f5ed00c2390 2024-03-06T07:24:59.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.159+0000 7f5ed499d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5ed00c46f0 0x7f5ed014ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.159+0000 7f5ed599f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5ed0150510 0x7f5ed0152df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41660/0 (socket says 172.21.15.1:41660) 2024-03-06T07:24:59.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.159+0000 7f5ed599f640 1 -- 172.21.15.1:0/3917430576 learned_addr learned my addr 172.21.15.1:0/3917430576 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:59.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.159+0000 7f5ed519e640 1 --2- 172.21.15.1:0/3917430576 >> v2:172.21.15.98:3300/0 conn(0x7f5ed00c2390 0x7f5ed00c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.159+0000 7f5ed499d640 1 -- 172.21.15.1:0/3917430576 >> v2:172.21.15.98:3300/0 conn(0x7f5ed00c2390 msgr2=0x7f5ed00c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.159+0000 7f5ed499d640 1 --2- 172.21.15.1:0/3917430576 >> v2:172.21.15.98:3300/0 conn(0x7f5ed00c2390 0x7f5ed00c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.159+0000 7f5ed499d640 1 -- 172.21.15.1:0/3917430576 >> v1:172.21.15.1:6789/0 conn(0x7f5ed0150510 legacy=0x7f5ed0152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.159+0000 7f5ed499d640 1 -- 172.21.15.1:0/3917430576 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed0156b40 con 0x7f5ed00c46f0 2024-03-06T07:24:59.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.159+0000 7f5ec3fff640 1 -- 172.21.15.1:0/3917430576 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517676723 0 0) 0x7f5ed0157340 con 0x7f5ed0150510 2024-03-06T07:24:59.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.160+0000 7f5ed499d640 1 --2- 172.21.15.1:0/3917430576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed00c46f0 0x7f5ed014ffd0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f5ed006f1e0 tx=0x7f5ec402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2322481032c9abf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.160+0000 7f5ec3fff640 1 -- 172.21.15.1:0/3917430576 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec40315a0 con 0x7f5ed00c46f0 2024-03-06T07:24:59.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.160+0000 7f5ec3fff640 1 -- 172.21.15.1:0/3917430576 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec4037070 con 0x7f5ed00c46f0 2024-03-06T07:24:59.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.160+0000 7f5ec3fff640 1 -- 172.21.15.1:0/3917430576 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec4031960 con 0x7f5ed00c46f0 2024-03-06T07:24:59.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.161+0000 7f5ed61a0640 1 -- 172.21.15.1:0/3917430576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed00c46f0 msgr2=0x7f5ed014ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.161+0000 7f5ed61a0640 1 --2- 172.21.15.1:0/3917430576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed00c46f0 0x7f5ed014ffd0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f5ed006f1e0 tx=0x7f5ec402b840 comp rx=0 tx=0).stop 2024-03-06T07:24:59.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.161+0000 7f5ed61a0640 1 -- 172.21.15.1:0/3917430576 shutdown_connections 2024-03-06T07:24:59.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.161+0000 7f5ed61a0640 1 --2- 172.21.15.1:0/3917430576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed00c46f0 0x7f5ed014ffd0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.161+0000 7f5ed61a0640 1 --2- 172.21.15.1:0/3917430576 >> v2:172.21.15.98:3300/0 conn(0x7f5ed00c2390 0x7f5ed00c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.161+0000 7f5ed61a0640 1 -- 172.21.15.1:0/3917430576 >> 172.21.15.1:0/3917430576 conn(0x7f5ed006eb50 msgr2=0x7f5ed006c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.161+0000 7f5ed61a0640 1 -- 172.21.15.1:0/3917430576 shutdown_connections 2024-03-06T07:24:59.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.161+0000 7f5ed61a0640 1 -- 172.21.15.1:0/3917430576 wait complete. 2024-03-06T07:24:59.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.162+0000 7f5ed61a0640 1 Processor -- start 2024-03-06T07:24:59.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.162+0000 7f5ed61a0640 1 -- start start 2024-03-06T07:24:59.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed61a0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ed00c2390 0x7f5ed01557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed61a0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed00c46f0 0x7f5ed0145b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed61a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ed0150510 0x7f5ed0146090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed61a0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed0070430 con 0x7f5ed00c46f0 2024-03-06T07:24:59.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed61a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed00702b0 con 0x7f5ed0150510 2024-03-06T07:24:59.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed61a0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed00c2770 con 0x7f5ed00c2390 2024-03-06T07:24:59.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed499d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed00c46f0 0x7f5ed0145b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed499d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed00c46f0 0x7f5ed0145b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37008/0 (socket says 172.21.15.1:37008) 2024-03-06T07:24:59.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed499d640 1 -- 172.21.15.1:0/3432567650 learned_addr learned my addr 172.21.15.1:0/3432567650 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:59.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed599f640 1 --2- 172.21.15.1:0/3432567650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ed0150510 0x7f5ed0146090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed519e640 1 --2- 172.21.15.1:0/3432567650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ed00c2390 0x7f5ed01557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed499d640 1 -- 172.21.15.1:0/3432567650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ed00c2390 msgr2=0x7f5ed01557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed499d640 1 --2- 172.21.15.1:0/3432567650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ed00c2390 0x7f5ed01557e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed499d640 1 -- 172.21.15.1:0/3432567650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ed0150510 msgr2=0x7f5ed0146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed499d640 1 --2- 172.21.15.1:0/3432567650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ed0150510 0x7f5ed0146090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed499d640 1 -- 172.21.15.1:0/3432567650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed00a9d30 con 0x7f5ed00c46f0 2024-03-06T07:24:59.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed499d640 1 --2- 172.21.15.1:0/3432567650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed00c46f0 0x7f5ed0145b50 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4009890 tx=0x7f5ec40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ec1ffb640 1 -- 172.21.15.1:0/3432567650 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec4032840 con 0x7f5ed00c46f0 2024-03-06T07:24:59.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ec1ffb640 1 -- 172.21.15.1:0/3432567650 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec4032a00 con 0x7f5ed00c46f0 2024-03-06T07:24:59.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ed61a0640 1 -- 172.21.15.1:0/3432567650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed00aa010 con 0x7f5ed00c46f0 2024-03-06T07:24:59.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.163+0000 7f5ec1ffb640 1 -- 172.21.15.1:0/3432567650 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec4032df0 con 0x7f5ed00c46f0 2024-03-06T07:24:59.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.164+0000 7f5ed61a0640 1 -- 172.21.15.1:0/3432567650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed0154cc0 con 0x7f5ed00c46f0 2024-03-06T07:24:59.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.165+0000 7f5ed61a0640 1 -- 172.21.15.1:0/3432567650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed00686e0 con 0x7f5ed00c46f0 2024-03-06T07:24:59.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.165+0000 7f5ec1ffb640 1 -- 172.21.15.1:0/3432567650 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ec404c020 con 0x7f5ed00c46f0 2024-03-06T07:24:59.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.165+0000 7f5ec1ffb640 1 --2- 172.21.15.1:0/3432567650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eb406b4d0 0x7f5eb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.165+0000 7f5ec1ffb640 1 -- 172.21.15.1:0/3432567650 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5ec402f020 con 0x7f5ed00c46f0 2024-03-06T07:24:59.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.166+0000 7f5ed519e640 1 --2- 172.21.15.1:0/3432567650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eb406b4d0 0x7f5eb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.166+0000 7f5ed519e640 1 --2- 172.21.15.1:0/3432567650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eb406b4d0 0x7f5eb406d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc006fd0 tx=0x7f5ecc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.170+0000 7f5ec1ffb640 1 -- 172.21.15.1:0/3432567650 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ec4040070 con 0x7f5ed00c46f0 2024-03-06T07:24:59.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.417+0000 7f5ed61a0640 1 -- 172.21.15.1:0/3432567650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ed006f460 con 0x7f5ed00c46f0 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.418+0000 7f5ec1ffb640 1 -- 172.21.15.1:0/3432567650 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4976 (secure 0 0 0) 0x7f5ec407a750 con 0x7f5ed00c46f0 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":106,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6446,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4151599120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4151599120},{"type":"v1","addr":"172.21.15.1:6835","nonce":4151599120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/155600632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":155600632},{"type":"v1","addr":"172.21.15.98:6837","nonce":155600632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7084,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1420541659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1420541659},{"type":"v1","addr":"172.21.15.98:6836","nonce":1420541659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.598303+0000","modified":"2024-03-06T07:24:55.621912+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7863},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7863":{"gid":7863,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1515309674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1515309674},{"type":"v1","addr":"172.21.15.1:6834","nonce":1515309674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7863,"qdb_cluster":[7863]},"id":9}]} 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 106 2024-03-06T07:24:59.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.420+0000 7f5ed61a0640 1 -- 172.21.15.1:0/3432567650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eb406b4d0 msgr2=0x7f5eb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.420+0000 7f5ed61a0640 1 --2- 172.21.15.1:0/3432567650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eb406b4d0 0x7f5eb406d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc006fd0 tx=0x7f5ecc008040 comp rx=0 tx=0).stop 2024-03-06T07:24:59.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.420+0000 7f5ed61a0640 1 -- 172.21.15.1:0/3432567650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed00c46f0 msgr2=0x7f5ed0145b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.420+0000 7f5ed61a0640 1 --2- 172.21.15.1:0/3432567650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed00c46f0 0x7f5ed0145b50 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4009890 tx=0x7f5ec40046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.420+0000 7f5ed61a0640 1 -- 172.21.15.1:0/3432567650 shutdown_connections 2024-03-06T07:24:59.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.420+0000 7f5ed61a0640 1 --2- 172.21.15.1:0/3432567650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eb406b4d0 0x7f5eb406d990 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.420+0000 7f5ed61a0640 1 --2- 172.21.15.1:0/3432567650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ed0150510 0x7f5ed0146090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.420+0000 7f5ed61a0640 1 --2- 172.21.15.1:0/3432567650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed00c46f0 0x7f5ed0145b50 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.420+0000 7f5ed61a0640 1 --2- 172.21.15.1:0/3432567650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ed00c2390 0x7f5ed01557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.420+0000 7f5ed61a0640 1 -- 172.21.15.1:0/3432567650 >> 172.21.15.1:0/3432567650 conn(0x7f5ed006eb50 msgr2=0x7f5ed006f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.420+0000 7f5ed61a0640 1 -- 172.21.15.1:0/3432567650 shutdown_connections 2024-03-06T07:24:59.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.420+0000 7f5ed61a0640 1 -- 172.21.15.1:0/3432567650 wait complete. 2024-03-06T07:24:59.433 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 106, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:24:52.598303+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 106, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7863": { "addr": "172.21.15.1:6834/1515309674", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 1515309674, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 1515309674, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7863, "incarnation": 102, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-03-06T07:24:55.621912+0000", "qdb_cluster": [ 7863 ], "qdb_leader": 7863, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7863 } } } ], "standbys": [ { "addr": "172.21.15.1:6835/4151599120", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 4151599120, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 4151599120, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6446, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6837/155600632", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 155600632, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 155600632, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6458, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6836/1420541659", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 1420541659, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 1420541659, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7084, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:24:59.434 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:24:59.434 INFO:tasks.cephfs.test_failover:current = [{'gid': 6446, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/4151599120', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 4151599120}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 4151599120}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}, {'gid': 7084, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/1420541659', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 1420541659}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1420541659}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}, {'gid': 7863, 'name': 'c', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6834/1515309674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1515309674}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1515309674}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6458, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/155600632', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 155600632}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 155600632}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}] 2024-03-06T07:24:59.434 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs 2024-03-06T07:24:59.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.537+0000 7fb150cb8640 1 Processor -- start 2024-03-06T07:24:59.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.537+0000 7fb150cb8640 1 -- start start 2024-03-06T07:24:59.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.537+0000 7fb150cb8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb14c152650 0x7fb14c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.537+0000 7fb150cb8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb14c0c6b10 0x7fb14c158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.537+0000 7fb150cb8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb14c0c2fc0 con 0x7fb14c0bf9d0 2024-03-06T07:24:59.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.537+0000 7fb150cb8640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb14c155800 con 0x7fb14c0c6b10 2024-03-06T07:24:59.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.537+0000 7fb150cb8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c0c6690 con 0x7fb14c152650 2024-03-06T07:24:59.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb14ad9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb14c0bf9d0 0x7fb14c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41666/0 (socket says 172.21.15.1:41666) 2024-03-06T07:24:59.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb14ad9d640 1 -- 172.21.15.1:0/1040268523 learned_addr learned my addr 172.21.15.1:0/1040268523 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:24:59.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb141d9b640 1 --2- 172.21.15.1:0/1040268523 >> v2:172.21.15.98:3300/0 conn(0x7fb14c0c6b10 0x7fb14c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb14a59c640 1 --2- 172.21.15.1:0/1040268523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb14c152650 0x7fb14c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb149d9b640 1 -- 172.21.15.1:0/1040268523 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840181955 0 0) 0x7fb14c0c2fc0 con 0x7fb14c0bf9d0 2024-03-06T07:24:59.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb149d9b640 1 -- 172.21.15.1:0/1040268523 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb138003520 con 0x7fb14c0bf9d0 2024-03-06T07:24:59.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb149d9b640 1 -- 172.21.15.1:0/1040268523 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 165500821 0 0) 0x7fb138003520 con 0x7fb14c0bf9d0 2024-03-06T07:24:59.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb149d9b640 1 -- 172.21.15.1:0/1040268523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb14c152650 msgr2=0x7fb14c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb149d9b640 1 --2- 172.21.15.1:0/1040268523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb14c152650 0x7fb14c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb149d9b640 1 -- 172.21.15.1:0/1040268523 >> v2:172.21.15.98:3300/0 conn(0x7fb14c0c6b10 msgr2=0x7fb14c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb149d9b640 1 --2- 172.21.15.1:0/1040268523 >> v2:172.21.15.98:3300/0 conn(0x7fb14c0c6b10 0x7fb14c158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb149d9b640 1 -- 172.21.15.1:0/1040268523 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb14c0c7280 con 0x7fb14c0bf9d0 2024-03-06T07:24:59.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb149d9b640 1 -- 172.21.15.1:0/1040268523 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fb1440019f0 con 0x7fb14c0bf9d0 2024-03-06T07:24:59.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb149d9b640 1 -- 172.21.15.1:0/1040268523 >> v1:172.21.15.1:6789/0 conn(0x7fb14c0bf9d0 legacy=0x7fb14c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb149d9b640 1 --2- 172.21.15.1:0/1040268523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1380039b0 0x7fb138003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb149d9b640 1 --2- 172.21.15.1:0/1040268523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb138004800 0x7fb138004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb149d9b640 1 --2- 172.21.15.1:0/1040268523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb14c152650 0x7fb1380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb149d9b640 1 -- 172.21.15.1:0/1040268523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c155800 con 0x7fb1380039b0 2024-03-06T07:24:59.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb149d9b640 1 -- 172.21.15.1:0/1040268523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c0c6690 con 0x7fb138004800 2024-03-06T07:24:59.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb149d9b640 1 -- 172.21.15.1:0/1040268523 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13800b9c0 con 0x7fb14c152650 2024-03-06T07:24:59.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb14a59c640 1 --2- 172.21.15.1:0/1040268523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1380039b0 0x7fb138003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb141d9b640 1 --2- 172.21.15.1:0/1040268523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb138004800 0x7fb138004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.538+0000 7fb14ad9d640 1 --2- 172.21.15.1:0/1040268523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb14c152650 0x7fb1380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.539+0000 7fb14a59c640 1 -- 172.21.15.1:0/1040268523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb14c152650 msgr2=0x7fb1380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.539+0000 7fb14a59c640 1 --2- 172.21.15.1:0/1040268523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb14c152650 0x7fb1380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.539+0000 7fb14a59c640 1 -- 172.21.15.1:0/1040268523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb138004800 msgr2=0x7fb138004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.539+0000 7fb14a59c640 1 --2- 172.21.15.1:0/1040268523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb138004800 0x7fb138004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.539+0000 7fb14a59c640 1 -- 172.21.15.1:0/1040268523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb138003520 con 0x7fb1380039b0 2024-03-06T07:24:59.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.539+0000 7fb14a59c640 1 --2- 172.21.15.1:0/1040268523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1380039b0 0x7fb138003dd0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fb13c009af0 tx=0x7fb138005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d75637815daa99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.540+0000 7fb149d9b640 1 -- 172.21.15.1:0/1040268523 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c039070 con 0x7fb1380039b0 2024-03-06T07:24:59.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.540+0000 7fb149d9b640 1 -- 172.21.15.1:0/1040268523 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13c02c6b0 con 0x7fb1380039b0 2024-03-06T07:24:59.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.540+0000 7fb149d9b640 1 -- 172.21.15.1:0/1040268523 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c009690 con 0x7fb1380039b0 2024-03-06T07:24:59.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.540+0000 7fb150cb8640 1 -- 172.21.15.1:0/1040268523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1380039b0 msgr2=0x7fb138003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.540+0000 7fb150cb8640 1 --2- 172.21.15.1:0/1040268523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1380039b0 0x7fb138003dd0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fb13c009af0 tx=0x7fb138005830 comp rx=0 tx=0).stop 2024-03-06T07:24:59.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.540+0000 7fb14ad9d640 1 -- 172.21.15.1:0/1040268523 reap_dead start 2024-03-06T07:24:59.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.540+0000 7fb150cb8640 1 -- 172.21.15.1:0/1040268523 shutdown_connections 2024-03-06T07:24:59.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.540+0000 7fb150cb8640 1 -- 172.21.15.1:0/1040268523 >> 172.21.15.1:0/1040268523 conn(0x7fb14c066ee0 msgr2=0x7fb14c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.540+0000 7fb150cb8640 1 -- 172.21.15.1:0/1040268523 shutdown_connections 2024-03-06T07:24:59.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.540+0000 7fb150cb8640 1 -- 172.21.15.1:0/1040268523 wait complete. 2024-03-06T07:24:59.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.542+0000 7fb150cb8640 1 Processor -- start 2024-03-06T07:24:59.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.542+0000 7fb150cb8640 1 -- start start 2024-03-06T07:24:59.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.542+0000 7fb150cb8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1380039b0 0x7fb14c156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.542+0000 7fb150cb8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb14c156690 0x7fb14c1573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.542+0000 7fb150cb8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb14c144190 0x7fb14c144610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.542+0000 7fb150cb8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c155490 con 0x7fb1380039b0 2024-03-06T07:24:59.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.542+0000 7fb150cb8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c1551b0 con 0x7fb14c144190 2024-03-06T07:24:59.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.542+0000 7fb150cb8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c152a30 con 0x7fb14c156690 2024-03-06T07:24:59.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.543+0000 7fb14a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1380039b0 0x7fb14c156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.543+0000 7fb14a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1380039b0 0x7fb14c156e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37038/0 (socket says 172.21.15.1:37038) 2024-03-06T07:24:59.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.543+0000 7fb14a59c640 1 -- 172.21.15.1:0/1959131773 learned_addr learned my addr 172.21.15.1:0/1959131773 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.543+0000 7fb14ad9d640 1 --2- 172.21.15.1:0/1959131773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb14c144190 0x7fb14c144610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.543+0000 7fb141d9b640 1 --2- 172.21.15.1:0/1959131773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb14c156690 0x7fb14c1573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.543+0000 7fb14a59c640 1 -- 172.21.15.1:0/1959131773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb14c156690 msgr2=0x7fb14c1573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.543+0000 7fb14a59c640 1 --2- 172.21.15.1:0/1959131773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb14c156690 0x7fb14c1573b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.543+0000 7fb14a59c640 1 -- 172.21.15.1:0/1959131773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb14c144190 msgr2=0x7fb14c144610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.543+0000 7fb14a59c640 1 --2- 172.21.15.1:0/1959131773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb14c144190 0x7fb14c144610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.543+0000 7fb14a59c640 1 -- 172.21.15.1:0/1959131773 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb14c1578f0 con 0x7fb1380039b0 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.543+0000 7fb14a59c640 1 --2- 172.21.15.1:0/1959131773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1380039b0 0x7fb14c156e70 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fb13c037040 tx=0x7fb13c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.543+0000 7fb1437fe640 1 -- 172.21.15.1:0/1959131773 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c042020 con 0x7fb1380039b0 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.543+0000 7fb1437fe640 1 -- 172.21.15.1:0/1959131773 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13c03a040 con 0x7fb1380039b0 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.543+0000 7fb1437fe640 1 -- 172.21.15.1:0/1959131773 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c039070 con 0x7fb1380039b0 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.543+0000 7fb150cb8640 1 -- 172.21.15.1:0/1959131773 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb14c1eefa0 con 0x7fb1380039b0 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.543+0000 7fb150cb8640 1 -- 172.21.15.1:0/1959131773 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb14c158df0 con 0x7fb1380039b0 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.545+0000 7fb150cb8640 1 -- 172.21.15.1:0/1959131773 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb14c067fd0 con 0x7fb1380039b0 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.545+0000 7fb1437fe640 1 -- 172.21.15.1:0/1959131773 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb13c045020 con 0x7fb1380039b0 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.545+0000 7fb1437fe640 1 --2- 172.21.15.1:0/1959131773 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb12c06b4d0 0x7fb12c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.545+0000 7fb1437fe640 1 -- 172.21.15.1:0/1959131773 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb13c0ab650 con 0x7fb1380039b0 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.546+0000 7fb141d9b640 1 --2- 172.21.15.1:0/1959131773 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb12c06b4d0 0x7fb12c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.546+0000 7fb141d9b640 1 --2- 172.21.15.1:0/1959131773 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb12c06b4d0 0x7fb12c06d990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb14c0c5ac0 tx=0x7fb134009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.550+0000 7fb1437fe640 1 -- 172.21.15.1:0/1959131773 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb13c04c070 con 0x7fb1380039b0 2024-03-06T07:24:59.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.778+0000 7fb150cb8640 1 -- 172.21.15.1:0/1959131773 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fb14c151e70 con 0x7fb1380039b0 2024-03-06T07:24:59.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.781+0000 7fb1437fe640 1 -- 172.21.15.1:0/1959131773 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v42)=0 v42) v1 ==== 116+0+0 (secure 0 0 0) 0x7fb13c07a490 con 0x7fb1380039b0 2024-03-06T07:24:59.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.783+0000 7fb150cb8640 1 -- 172.21.15.1:0/1959131773 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb12c06b4d0 msgr2=0x7fb12c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.783+0000 7fb150cb8640 1 --2- 172.21.15.1:0/1959131773 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb12c06b4d0 0x7fb12c06d990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb14c0c5ac0 tx=0x7fb134009190 comp rx=0 tx=0).stop 2024-03-06T07:24:59.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.784+0000 7fb150cb8640 1 -- 172.21.15.1:0/1959131773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1380039b0 msgr2=0x7fb14c156e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.784+0000 7fb150cb8640 1 --2- 172.21.15.1:0/1959131773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1380039b0 0x7fb14c156e70 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fb13c037040 tx=0x7fb13c004730 comp rx=0 tx=0).stop 2024-03-06T07:24:59.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.784+0000 7fb150cb8640 1 -- 172.21.15.1:0/1959131773 shutdown_connections 2024-03-06T07:24:59.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.784+0000 7fb150cb8640 1 --2- 172.21.15.1:0/1959131773 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb12c06b4d0 0x7fb12c06d990 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.784+0000 7fb150cb8640 1 --2- 172.21.15.1:0/1959131773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb14c144190 0x7fb14c144610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.784+0000 7fb150cb8640 1 --2- 172.21.15.1:0/1959131773 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb14c156690 0x7fb14c1573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.784+0000 7fb150cb8640 1 --2- 172.21.15.1:0/1959131773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1380039b0 0x7fb14c156e70 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.784+0000 7fb150cb8640 1 -- 172.21.15.1:0/1959131773 >> 172.21.15.1:0/1959131773 conn(0x7fb14c066ee0 msgr2=0x7fb14c0c0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.784+0000 7fb150cb8640 1 -- 172.21.15.1:0/1959131773 shutdown_connections 2024-03-06T07:24:59.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.784+0000 7fb150cb8640 1 -- 172.21.15.1:0/1959131773 wait complete. 2024-03-06T07:24:59.797 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:59.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.896+0000 7f8a84803640 1 Processor -- start 2024-03-06T07:24:59.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.896+0000 7f8a84803640 1 -- start start 2024-03-06T07:24:59.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.897+0000 7f8a84803640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8a800c8c40 0x7f8a800c7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.897+0000 7f8a84803640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a800c7c90 0x7f8a800c3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.897+0000 7f8a84803640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a801532a0 con 0x7f8a800c49b0 2024-03-06T07:24:59.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.897+0000 7f8a84803640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a80071530 con 0x7f8a800c7c90 2024-03-06T07:24:59.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.897+0000 7f8a84803640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a80150960 con 0x7f8a800c8c40 2024-03-06T07:24:59.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.897+0000 7f8a7f59e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a800c7c90 0x7f8a800c3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.897+0000 7f8a7f59e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a800c7c90 0x7f8a800c3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37042/0 (socket says 172.21.15.1:37042) 2024-03-06T07:24:59.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.897+0000 7f8a7f59e640 1 -- 172.21.15.1:0/2178055883 learned_addr learned my addr 172.21.15.1:0/2178055883 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:59.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.897+0000 7f8a7e59c640 1 --2- 172.21.15.1:0/2178055883 >> v2:172.21.15.98:3300/0 conn(0x7f8a800c8c40 0x7f8a800c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.897+0000 7f8a7f59e640 1 -- 172.21.15.1:0/2178055883 >> v2:172.21.15.98:3300/0 conn(0x7f8a800c8c40 msgr2=0x7f8a800c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.897+0000 7f8a7f59e640 1 --2- 172.21.15.1:0/2178055883 >> v2:172.21.15.98:3300/0 conn(0x7f8a800c8c40 0x7f8a800c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.897+0000 7f8a7f59e640 1 -- 172.21.15.1:0/2178055883 >> v1:172.21.15.98:6789/0 conn(0x7f8a800c49b0 legacy=0x7f8a800c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.897+0000 7f8a7f59e640 1 -- 172.21.15.1:0/2178055883 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a80153580 con 0x7f8a800c7c90 2024-03-06T07:24:59.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.899+0000 7f8a7f59e640 1 --2- 172.21.15.1:0/2178055883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a800c7c90 0x7f8a800c3100 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f8a70009e50 tx=0x7f8a7002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d158c91172917ac9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.899+0000 7f8a7dd9b640 1 -- 172.21.15.1:0/2178055883 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7002ca80 con 0x7f8a800c7c90 2024-03-06T07:24:59.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.899+0000 7f8a7dd9b640 1 -- 172.21.15.1:0/2178055883 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7002cc20 con 0x7f8a800c7c90 2024-03-06T07:24:59.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.899+0000 7f8a7dd9b640 1 -- 172.21.15.1:0/2178055883 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a700343f0 con 0x7f8a800c7c90 2024-03-06T07:24:59.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.899+0000 7f8a84803640 1 -- 172.21.15.1:0/2178055883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a800c7c90 msgr2=0x7f8a800c3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.899+0000 7f8a84803640 1 --2- 172.21.15.1:0/2178055883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a800c7c90 0x7f8a800c3100 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f8a70009e50 tx=0x7f8a7002bee0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.899+0000 7f8a84803640 1 -- 172.21.15.1:0/2178055883 shutdown_connections 2024-03-06T07:24:59.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.899+0000 7f8a84803640 1 --2- 172.21.15.1:0/2178055883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a800c7c90 0x7f8a800c3100 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.899+0000 7f8a84803640 1 --2- 172.21.15.1:0/2178055883 >> v2:172.21.15.98:3300/0 conn(0x7f8a800c8c40 0x7f8a800c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.899+0000 7f8a84803640 1 -- 172.21.15.1:0/2178055883 >> 172.21.15.1:0/2178055883 conn(0x7f8a8006b620 msgr2=0x7f8a800aa4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.900+0000 7f8a84803640 1 -- 172.21.15.1:0/2178055883 shutdown_connections 2024-03-06T07:24:59.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.900+0000 7f8a84803640 1 -- 172.21.15.1:0/2178055883 wait complete. 2024-03-06T07:24:59.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.901+0000 7f8a84803640 1 Processor -- start 2024-03-06T07:24:59.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.901+0000 7f8a84803640 1 -- start start 2024-03-06T07:24:59.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.902+0000 7f8a84803640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a800c49b0 0x7f8a801568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.902+0000 7f8a84803640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a800c7c90 0x7f8a80156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.902+0000 7f8a84803640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a800c8c40 0x7f8a800c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.902+0000 7f8a84803640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a80152f90 con 0x7f8a800c8c40 2024-03-06T07:24:59.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.902+0000 7f8a84803640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a80153b40 con 0x7f8a800c49b0 2024-03-06T07:24:59.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.902+0000 7f8a84803640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800c8120 con 0x7f8a800c7c90 2024-03-06T07:24:59.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.902+0000 7f8a7f59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a800c8c40 0x7f8a800c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.902+0000 7f8a7f59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a800c8c40 0x7f8a800c1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37054/0 (socket says 172.21.15.1:37054) 2024-03-06T07:24:59.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.902+0000 7f8a7f59e640 1 -- 172.21.15.1:0/4126054942 learned_addr learned my addr 172.21.15.1:0/4126054942 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:24:59.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.902+0000 7f8a7ed9d640 1 --2- 172.21.15.1:0/4126054942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a800c49b0 0x7f8a801568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.902+0000 7f8a7e59c640 1 --2- 172.21.15.1:0/4126054942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a800c7c90 0x7f8a80156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.902+0000 7f8a7f59e640 1 -- 172.21.15.1:0/4126054942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a800c7c90 msgr2=0x7f8a80156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.902+0000 7f8a7f59e640 1 --2- 172.21.15.1:0/4126054942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a800c7c90 0x7f8a80156df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.902+0000 7f8a7f59e640 1 -- 172.21.15.1:0/4126054942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a800c49b0 msgr2=0x7f8a801568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.902+0000 7f8a7f59e640 1 --2- 172.21.15.1:0/4126054942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a800c49b0 0x7f8a801568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.902+0000 7f8a7f59e640 1 -- 172.21.15.1:0/4126054942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a801e9050 con 0x7f8a800c8c40 2024-03-06T07:24:59.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.902+0000 7f8a7f59e640 1 --2- 172.21.15.1:0/4126054942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a800c8c40 0x7f8a800c1f80 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f8a70009f70 tx=0x7f8a70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.903+0000 7f8a677fe640 1 -- 172.21.15.1:0/4126054942 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70043020 con 0x7f8a800c8c40 2024-03-06T07:24:59.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.903+0000 7f8a677fe640 1 -- 172.21.15.1:0/4126054942 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7003e450 con 0x7f8a800c8c40 2024-03-06T07:24:59.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.903+0000 7f8a84803640 1 -- 172.21.15.1:0/4126054942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a800bd720 con 0x7f8a800c8c40 2024-03-06T07:24:59.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.903+0000 7f8a677fe640 1 -- 172.21.15.1:0/4126054942 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70009510 con 0x7f8a800c8c40 2024-03-06T07:24:59.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.903+0000 7f8a84803640 1 -- 172.21.15.1:0/4126054942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a800bdce0 con 0x7f8a800c8c40 2024-03-06T07:24:59.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.904+0000 7f8a84803640 1 -- 172.21.15.1:0/4126054942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a800686e0 con 0x7f8a800c8c40 2024-03-06T07:24:59.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.905+0000 7f8a677fe640 1 -- 172.21.15.1:0/4126054942 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a70046020 con 0x7f8a800c8c40 2024-03-06T07:24:59.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.905+0000 7f8a677fe640 1 --2- 172.21.15.1:0/4126054942 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a6c06b4d0 0x7f8a6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.905+0000 7f8a677fe640 1 -- 172.21.15.1:0/4126054942 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8a700ac960 con 0x7f8a800c8c40 2024-03-06T07:24:59.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.905+0000 7f8a7ed9d640 1 --2- 172.21.15.1:0/4126054942 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a6c06b4d0 0x7f8a6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.906+0000 7f8a7ed9d640 1 --2- 172.21.15.1:0/4126054942 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a6c06b4d0 0x7f8a6c06d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8a78001780 tx=0x7f8a78009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:24:59.909+0000 7f8a677fe640 1 -- 172.21.15.1:0/4126054942 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a7004d030 con 0x7f8a800c8c40 2024-03-06T07:25:00.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.127+0000 7f8a84803640 1 -- 172.21.15.1:0/4126054942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a80151a40 con 0x7f8a800c8c40 2024-03-06T07:25:00.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.128+0000 7f8a677fe640 1 -- 172.21.15.1:0/4126054942 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4976 (secure 0 0 0) 0x7f8a7007b090 con 0x7f8a800c8c40 2024-03-06T07:25:00.374 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:00.374 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":106,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6446,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4151599120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4151599120},{"type":"v1","addr":"172.21.15.1:6835","nonce":4151599120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/155600632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":155600632},{"type":"v1","addr":"172.21.15.98:6837","nonce":155600632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7084,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1420541659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1420541659},{"type":"v1","addr":"172.21.15.98:6836","nonce":1420541659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.598303+0000","modified":"2024-03-06T07:24:55.621912+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7863},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7863":{"gid":7863,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1515309674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1515309674},{"type":"v1","addr":"172.21.15.1:6834","nonce":1515309674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7863,"qdb_cluster":[7863]},"id":9}]} 2024-03-06T07:25:00.376 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 106 2024-03-06T07:25:00.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.129+0000 7f8a84803640 1 -- 172.21.15.1:0/4126054942 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a6c06b4d0 msgr2=0x7f8a6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.129+0000 7f8a84803640 1 --2- 172.21.15.1:0/4126054942 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a6c06b4d0 0x7f8a6c06d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8a78001780 tx=0x7f8a78009040 comp rx=0 tx=0).stop 2024-03-06T07:25:00.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.129+0000 7f8a84803640 1 -- 172.21.15.1:0/4126054942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a800c8c40 msgr2=0x7f8a800c1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.129+0000 7f8a84803640 1 --2- 172.21.15.1:0/4126054942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a800c8c40 0x7f8a800c1f80 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f8a70009f70 tx=0x7f8a70004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:00.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.129+0000 7f8a84803640 1 -- 172.21.15.1:0/4126054942 shutdown_connections 2024-03-06T07:25:00.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.129+0000 7f8a84803640 1 --2- 172.21.15.1:0/4126054942 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a6c06b4d0 0x7f8a6c06d990 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.129+0000 7f8a84803640 1 --2- 172.21.15.1:0/4126054942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a800c8c40 0x7f8a800c1f80 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.129+0000 7f8a84803640 1 --2- 172.21.15.1:0/4126054942 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a800c7c90 0x7f8a80156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.129+0000 7f8a84803640 1 --2- 172.21.15.1:0/4126054942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a800c49b0 0x7f8a801568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.129+0000 7f8a84803640 1 -- 172.21.15.1:0/4126054942 >> 172.21.15.1:0/4126054942 conn(0x7f8a8006b620 msgr2=0x7f8a800aa490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.129+0000 7f8a84803640 1 -- 172.21.15.1:0/4126054942 shutdown_connections 2024-03-06T07:25:00.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.130+0000 7f8a84803640 1 -- 172.21.15.1:0/4126054942 wait complete. 2024-03-06T07:25:00.781 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:52.598303+0000', 'modified': '2024-03-06T07:24:55.621912+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7863}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7863': {'gid': 7863, 'name': 'c', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6834/1515309674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1515309674}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1515309674}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7863, 'qdb_cluster': [7863]} 2024-03-06T07:25:00.781 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:00.781 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:25:00.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.882+0000 7f1c5d3e9640 1 Processor -- start 2024-03-06T07:25:00.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.882+0000 7f1c5d3e9640 1 -- start start 2024-03-06T07:25:00.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.883+0000 7f1c5d3e9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1c58152820 0x7f1c580c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.883+0000 7f1c5d3e9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c580c7180 con 0x7f1c580c0670 2024-03-06T07:25:00.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.883+0000 7f1c5d3e9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c58155460 con 0x7f1c580af6f0 2024-03-06T07:25:00.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.883+0000 7f1c5d3e9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c580c6bc0 con 0x7f1c58152820 2024-03-06T07:25:00.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.883+0000 7f1c567fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1c580c0670 0x7f1c580c73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41668/0 (socket says 172.21.15.1:41668) 2024-03-06T07:25:00.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.883+0000 7f1c567fc640 1 -- 172.21.15.1:0/3708344000 learned_addr learned my addr 172.21.15.1:0/3708344000 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:00.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.883+0000 7f1c56ffd640 1 --2- 172.21.15.1:0/3708344000 >> v2:172.21.15.98:3300/0 conn(0x7f1c58152820 0x7f1c580c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.883+0000 7f1c55ffb640 1 -- 172.21.15.1:0/3708344000 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538860004 0 0) 0x7f1c580c7180 con 0x7f1c580c0670 2024-03-06T07:25:00.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.883+0000 7f1c55ffb640 1 -- 172.21.15.1:0/3708344000 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c40003520 con 0x7f1c580c0670 2024-03-06T07:25:00.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.883+0000 7f1c55ffb640 1 -- 172.21.15.1:0/3708344000 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3181015504 0 0) 0x7f1c40003520 con 0x7f1c580c0670 2024-03-06T07:25:00.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.883+0000 7f1c55ffb640 1 -- 172.21.15.1:0/3708344000 >> v2:172.21.15.98:3300/0 conn(0x7f1c58152820 msgr2=0x7f1c580c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.883+0000 7f1c55ffb640 1 --2- 172.21.15.1:0/3708344000 >> v2:172.21.15.98:3300/0 conn(0x7f1c58152820 0x7f1c580c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.883+0000 7f1c55ffb640 1 -- 172.21.15.1:0/3708344000 >> v1:172.21.15.98:6789/0 conn(0x7f1c580af6f0 legacy=0x7f1c5815fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.883+0000 7f1c55ffb640 1 -- 172.21.15.1:0/3708344000 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c580c6ea0 con 0x7f1c580c0670 2024-03-06T07:25:00.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c55ffb640 1 -- 172.21.15.1:0/3708344000 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f1c4c001670 con 0x7f1c580c0670 2024-03-06T07:25:00.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c55ffb640 1 -- 172.21.15.1:0/3708344000 >> v1:172.21.15.1:6789/0 conn(0x7f1c580c0670 legacy=0x7f1c580c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c55ffb640 1 --2- 172.21.15.1:0/3708344000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c400039b0 0x7f1c40003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c55ffb640 1 --2- 172.21.15.1:0/3708344000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c40004800 0x7f1c40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c55ffb640 1 --2- 172.21.15.1:0/3708344000 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c400051c0 0x7f1c4000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c55ffb640 1 -- 172.21.15.1:0/3708344000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c580c6bc0 con 0x7f1c40004800 2024-03-06T07:25:00.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c55ffb640 1 -- 172.21.15.1:0/3708344000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4000c060 con 0x7f1c400039b0 2024-03-06T07:25:00.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c55ffb640 1 -- 172.21.15.1:0/3708344000 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4000c260 con 0x7f1c400051c0 2024-03-06T07:25:00.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c567fc640 1 --2- 172.21.15.1:0/3708344000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c40004800 0x7f1c40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c56ffd640 1 --2- 172.21.15.1:0/3708344000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c400039b0 0x7f1c40003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c577fe640 1 --2- 172.21.15.1:0/3708344000 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c400051c0 0x7f1c4000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c567fc640 1 -- 172.21.15.1:0/3708344000 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c400051c0 msgr2=0x7f1c4000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c567fc640 1 --2- 172.21.15.1:0/3708344000 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c400051c0 0x7f1c4000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c567fc640 1 -- 172.21.15.1:0/3708344000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c400039b0 msgr2=0x7f1c40003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c567fc640 1 --2- 172.21.15.1:0/3708344000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c400039b0 0x7f1c40003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c567fc640 1 -- 172.21.15.1:0/3708344000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c40003520 con 0x7f1c40004800 2024-03-06T07:25:00.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c577fe640 1 --2- 172.21.15.1:0/3708344000 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c400051c0 0x7f1c4000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:00.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.884+0000 7f1c577fe640 1 -- 172.21.15.1:0/3708344000 reap_dead start 2024-03-06T07:25:00.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.885+0000 7f1c567fc640 1 --2- 172.21.15.1:0/3708344000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c40004800 0x7f1c40004c80 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c00c210 tx=0x7f1c40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1220d7fd8c057c06 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.885+0000 7f1c55ffb640 1 -- 172.21.15.1:0/3708344000 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c007cf0 con 0x7f1c40004800 2024-03-06T07:25:00.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.885+0000 7f1c55ffb640 1 -- 172.21.15.1:0/3708344000 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4c030070 con 0x7f1c40004800 2024-03-06T07:25:00.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.885+0000 7f1c55ffb640 1 -- 172.21.15.1:0/3708344000 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c005d30 con 0x7f1c40004800 2024-03-06T07:25:00.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.885+0000 7f1c5d3e9640 1 -- 172.21.15.1:0/3708344000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c40004800 msgr2=0x7f1c40004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.885+0000 7f1c5d3e9640 1 --2- 172.21.15.1:0/3708344000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c40004800 0x7f1c40004c80 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c00c210 tx=0x7f1c40005830 comp rx=0 tx=0).stop 2024-03-06T07:25:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.885+0000 7f1c5d3e9640 1 -- 172.21.15.1:0/3708344000 shutdown_connections 2024-03-06T07:25:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.885+0000 7f1c5d3e9640 1 --2- 172.21.15.1:0/3708344000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c40004800 0x7f1c40004c80 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.885+0000 7f1c5d3e9640 1 -- 172.21.15.1:0/3708344000 >> 172.21.15.1:0/3708344000 conn(0x7f1c58072720 msgr2=0x7f1c58072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.886+0000 7f1c5d3e9640 1 -- 172.21.15.1:0/3708344000 shutdown_connections 2024-03-06T07:25:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.886+0000 7f1c5d3e9640 1 -- 172.21.15.1:0/3708344000 wait complete. 2024-03-06T07:25:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.887+0000 7f1c5d3e9640 1 Processor -- start 2024-03-06T07:25:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.887+0000 7f1c5d3e9640 1 -- start start 2024-03-06T07:25:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.888+0000 7f1c5d3e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c40004800 0x7f1c58159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.888+0000 7f1c5d3e9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c58156d20 0x7f1c581549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.888+0000 7f1c5d3e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c5814ce10 0x7f1c581ead90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.888+0000 7f1c5d3e9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c58152d80 con 0x7f1c58156d20 2024-03-06T07:25:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.888+0000 7f1c5d3e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c58152c00 con 0x7f1c40004800 2024-03-06T07:25:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.888+0000 7f1c5d3e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c580afc40 con 0x7f1c5814ce10 2024-03-06T07:25:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.888+0000 7f1c567fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c58156d20 0x7f1c581549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.888+0000 7f1c567fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c58156d20 0x7f1c581549c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37076/0 (socket says 172.21.15.1:37076) 2024-03-06T07:25:00.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.888+0000 7f1c567fc640 1 -- 172.21.15.1:0/4048384035 learned_addr learned my addr 172.21.15.1:0/4048384035 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:00.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.888+0000 7f1c577fe640 1 --2- 172.21.15.1:0/4048384035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c5814ce10 0x7f1c581ead90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.888+0000 7f1c56ffd640 1 --2- 172.21.15.1:0/4048384035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c40004800 0x7f1c58159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.888+0000 7f1c567fc640 1 -- 172.21.15.1:0/4048384035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c5814ce10 msgr2=0x7f1c581ead90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.888+0000 7f1c567fc640 1 --2- 172.21.15.1:0/4048384035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c5814ce10 0x7f1c581ead90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.888+0000 7f1c567fc640 1 -- 172.21.15.1:0/4048384035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c40004800 msgr2=0x7f1c58159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.888+0000 7f1c567fc640 1 --2- 172.21.15.1:0/4048384035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c40004800 0x7f1c58159210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.888+0000 7f1c567fc640 1 -- 172.21.15.1:0/4048384035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c580c4940 con 0x7f1c58156d20 2024-03-06T07:25:00.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.889+0000 7f1c567fc640 1 --2- 172.21.15.1:0/4048384035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c58156d20 0x7f1c581549c0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c00c330 tx=0x7f1c581559e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.889+0000 7f1c3f7fe640 1 -- 172.21.15.1:0/4048384035 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c045070 con 0x7f1c58156d20 2024-03-06T07:25:00.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.889+0000 7f1c3f7fe640 1 -- 172.21.15.1:0/4048384035 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4c004bf0 con 0x7f1c58156d20 2024-03-06T07:25:00.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.889+0000 7f1c5d3e9640 1 -- 172.21.15.1:0/4048384035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c581ee600 con 0x7f1c58156d20 2024-03-06T07:25:00.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.889+0000 7f1c3f7fe640 1 -- 172.21.15.1:0/4048384035 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c007cf0 con 0x7f1c58156d20 2024-03-06T07:25:00.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.889+0000 7f1c5d3e9640 1 -- 172.21.15.1:0/4048384035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c581ee8a0 con 0x7f1c58156d20 2024-03-06T07:25:00.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.890+0000 7f1c3f7fe640 1 -- 172.21.15.1:0/4048384035 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c4c007070 con 0x7f1c58156d20 2024-03-06T07:25:00.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.891+0000 7f1c3f7fe640 1 --2- 172.21.15.1:0/4048384035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c4406b500 0x7f1c4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.891+0000 7f1c3f7fe640 1 -- 172.21.15.1:0/4048384035 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1c4c0ab3c0 con 0x7f1c58156d20 2024-03-06T07:25:00.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.891+0000 7f1c56ffd640 1 --2- 172.21.15.1:0/4048384035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c4406b500 0x7f1c4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.891+0000 7f1c56ffd640 1 --2- 172.21.15.1:0/4048384035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c4406b500 0x7f1c4406d9c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1c48009a60 tx=0x7f1c48008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.891+0000 7f1c5ffd5740 1 Processor -- start 2024-03-06T07:25:00.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.892+0000 7f1c5ffd5740 1 -- start start 2024-03-06T07:25:00.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.892+0000 7f1c5ffd5740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748644f50 0x55e748642a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.892+0000 7f1c5ffd5740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748645330 0x55e748642f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.892+0000 7f1c5ffd5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7486434b0 0x55e74863f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.892+0000 7f1c5ffd5740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7485a96b0 con 0x55e748645330 2024-03-06T07:25:00.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.892+0000 7f1c5ffd5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e748552a60 con 0x55e7486434b0 2024-03-06T07:25:00.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.892+0000 7f1c5ffd5740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e7485a5750 con 0x55e748644f50 2024-03-06T07:25:00.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.892+0000 7f1c567fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748645330 0x55e748642f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.892+0000 7f1c567fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748645330 0x55e748642f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37090/0 (socket says 172.21.15.1:37090) 2024-03-06T07:25:00.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.892+0000 7f1c567fc640 1 -- 172.21.15.1:0/377567902 learned_addr learned my addr 172.21.15.1:0/377567902 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:00.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.893+0000 7f1c56ffd640 1 --2- 172.21.15.1:0/377567902 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748644f50 0x55e748642a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.893+0000 7f1c577fe640 1 --2- 172.21.15.1:0/377567902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7486434b0 0x55e74863f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.893+0000 7f1c567fc640 1 -- 172.21.15.1:0/377567902 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748644f50 msgr2=0x55e748642a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.893+0000 7f1c567fc640 1 --2- 172.21.15.1:0/377567902 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748644f50 0x55e748642a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.893+0000 7f1c567fc640 1 -- 172.21.15.1:0/377567902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7486434b0 msgr2=0x55e74863f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.893+0000 7f1c567fc640 1 --2- 172.21.15.1:0/377567902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7486434b0 0x55e74863f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.893+0000 7f1c567fc640 1 -- 172.21.15.1:0/377567902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e748570f30 con 0x55e748645330 2024-03-06T07:25:00.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.893+0000 7f1c567fc640 1 --2- 172.21.15.1:0/377567902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748645330 0x55e748642f70 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c081d00 tx=0x7f1c4c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7c5c3eb52962a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.893+0000 7f1c3cff9640 1 -- 172.21.15.1:0/377567902 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c07dc70 con 0x55e748645330 2024-03-06T07:25:00.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.893+0000 7f1c3cff9640 1 -- 172.21.15.1:0/377567902 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4c07ae50 con 0x55e748645330 2024-03-06T07:25:00.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.893+0000 7f1c3cff9640 1 -- 172.21.15.1:0/377567902 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c082230 con 0x55e748645330 2024-03-06T07:25:00.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.894+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/377567902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748645330 msgr2=0x55e748642f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.894+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/377567902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748645330 0x55e748642f70 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c081d00 tx=0x7f1c4c000ed0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.894+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/377567902 shutdown_connections 2024-03-06T07:25:00.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.894+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/377567902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7486434b0 0x55e74863f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.894+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/377567902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748645330 0x55e748642f70 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.894+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/377567902 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748644f50 0x55e748642a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.894+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/377567902 >> 172.21.15.1:0/377567902 conn(0x55e7486417d0 msgr2=0x55e7486663c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.894+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/377567902 shutdown_connections 2024-03-06T07:25:00.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.894+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/377567902 wait complete. 2024-03-06T07:25:00.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.895+0000 7f1c5ffd5740 1 -- start start 2024-03-06T07:25:00.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.895+0000 7f1c5ffd5740 1 Processor -- start 2024-03-06T07:25:00.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.895+0000 7f1c5ffd5740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748644f50 0x55e74866a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.895+0000 7f1c5ffd5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e748645330 0x55e7486657b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.895+0000 7f1c5ffd5740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748691480 0x55e7486949f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.895+0000 7f1c5ffd5740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e748552a60 con 0x55e748644f50 2024-03-06T07:25:00.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.895+0000 7f1c5ffd5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e748565ea0 con 0x55e748645330 2024-03-06T07:25:00.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.895+0000 7f1c5ffd5740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e7485a96b0 con 0x55e748691480 2024-03-06T07:25:00.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.895+0000 7f1c5ffd5740 10 client.0 _ll_register_callbacks cb 0x55e748565870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:00.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.895+0000 7f1c56ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748644f50 0x55e74866a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.895+0000 7f1c56ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748644f50 0x55e74866a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37094/0 (socket says 172.21.15.1:37094) 2024-03-06T07:25:00.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.895+0000 7f1c56ffd640 1 -- 172.21.15.1:0/946535281 learned_addr learned my addr 172.21.15.1:0/946535281 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:00.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.896+0000 7f1c567fc640 1 --2- 172.21.15.1:0/946535281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e748645330 0x55e7486657b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.896+0000 7f1c577fe640 1 --2- 172.21.15.1:0/946535281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748691480 0x55e7486949f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.896+0000 7f1c56ffd640 1 -- 172.21.15.1:0/946535281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748691480 msgr2=0x55e7486949f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.896+0000 7f1c56ffd640 1 --2- 172.21.15.1:0/946535281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748691480 0x55e7486949f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.896+0000 7f1c56ffd640 1 -- 172.21.15.1:0/946535281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e748645330 msgr2=0x55e7486657b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.896+0000 7f1c56ffd640 1 --2- 172.21.15.1:0/946535281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e748645330 0x55e7486657b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.896+0000 7f1c56ffd640 1 -- 172.21.15.1:0/946535281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e748566270 con 0x55e748644f50 2024-03-06T07:25:00.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.896+0000 7f1c56ffd640 1 --2- 172.21.15.1:0/946535281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748644f50 0x55e74866a200 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f1c48016810 tx=0x7f1c48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0afef887a718deb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.896+0000 7f1c35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:25:00.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.896+0000 7f1c35ffb640 1 -- 172.21.15.1:0/946535281 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48016d90 con 0x55e748644f50 2024-03-06T07:25:00.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.896+0000 7f1c35ffb640 1 -- 172.21.15.1:0/946535281 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1c48004030 con 0x55e748644f50 2024-03-06T07:25:00.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.896+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/946535281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1c4c07ae50 con 0x55e748644f50 2024-03-06T07:25:00.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.896+0000 7f1c35ffb640 1 -- 172.21.15.1:0/946535281 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c48033050 con 0x55e748644f50 2024-03-06T07:25:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.896+0000 7f1c5ffd5740 10 client.8049 fetch_fsmap learned FSMap version 106 2024-03-06T07:25:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.896+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/946535281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=2+}) v3 -- 0x55e7482be2d0 con 0x55e748644f50 2024-03-06T07:25:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.896+0000 7f1c35ffb640 1 -- 172.21.15.1:0/946535281 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4c07ae50 con 0x55e748644f50 2024-03-06T07:25:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.897+0000 7f1c35ffb640 1 -- 172.21.15.1:0/946535281 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 106) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f1c48036020 con 0x55e748644f50 2024-03-06T07:25:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.897+0000 7f1c5ffd5740 10 client.8049 fetch_fsmap finished waiting for FSMap version 106 2024-03-06T07:25:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.897+0000 7f1c5ffd5740 10 client.8049 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:25:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.897+0000 7f1c5ffd5740 20 client.8049 populate_metadata read hostname 'smithi001' 2024-03-06T07:25:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.897+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/946535281 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x55e7486620f0 0x55e7486624d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.897+0000 7f1c5ffd5740 4 client.8049 mds_command: new command op to 7863 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.897+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/946535281 --> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e7485b6bf0 con 0x55e7486620f0 2024-03-06T07:25:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.897+0000 7f1c567fc640 1 --2- 172.21.15.1:0/946535281 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x55e7486620f0 0x55e7486624d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.897+0000 7f1c35ffb640 10 client.8049 ms_handle_connect on v2:172.21.15.1:6832/1515309674 2024-03-06T07:25:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.897+0000 7f1c567fc640 1 --2- 172.21.15.1:0/946535281 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x55e7486620f0 0x55e7486624d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5788c855cc0f4b3f server_cookie=c6efcfa73897d7d2 in_seq=0 out_seq=0 2024-03-06T07:25:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.898+0000 7f1c35ffb640 1 -- 172.21.15.1:0/946535281 <== mds.0 v2:172.21.15.1:6832/1515309674 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e7485b6bf0 con 0x55e7486620f0 2024-03-06T07:25:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.898+0000 7f1c35ffb640 10 client.8049 handle_command_reply: tid=0 2024-03-06T07:25:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.898+0000 7f1c5ffd5740 1 client.8049 shutdown 2024-03-06T07:25:00.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.898+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/946535281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748644f50 msgr2=0x55e74866a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.898+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/946535281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748644f50 0x55e74866a200 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f1c48016810 tx=0x7f1c48004730 comp rx=0 tx=0).stop 2024-03-06T07:25:00.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.898+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/946535281 shutdown_connections 2024-03-06T07:25:00.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.898+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/946535281 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x55e7486620f0 0x55e7486624d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.898+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/946535281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748691480 0x55e7486949f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.898+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/946535281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e748645330 0x55e7486657b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.898+0000 7f1c35ffb640 0 client.8049 ms_handle_reset on v2:172.21.15.1:6832/1515309674 2024-03-06T07:25:00.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.898+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/946535281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748644f50 0x55e74866a200 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.898+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/946535281 >> 172.21.15.1:0/946535281 conn(0x55e7486434b0 msgr2=0x55e7486438e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.898+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/946535281 shutdown_connections 2024-03-06T07:25:00.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.898+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/946535281 wait complete. 2024-03-06T07:25:00.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.898+0000 7f1c5ffd5740 20 client.8049 trim_cache size 0 max 16384 2024-03-06T07:25:00.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.915+0000 7f1c5ffd5740 1 Processor -- start 2024-03-06T07:25:00.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.915+0000 7f1c5ffd5740 1 -- start start 2024-03-06T07:25:00.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.915+0000 7f1c5ffd5740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748644f50 0x55e74863e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.915+0000 7f1c5ffd5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e748645330 0x55e74863f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.915+0000 7f1c5ffd5740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748691480 0x55e74863fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.915+0000 7f1c5ffd5740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c581f76e0 con 0x55e748644f50 2024-03-06T07:25:00.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.915+0000 7f1c5ffd5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c581f6ae0 con 0x55e748645330 2024-03-06T07:25:00.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.915+0000 7f1c5ffd5740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c581f70e0 con 0x55e748691480 2024-03-06T07:25:00.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.916+0000 7f1c56ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748644f50 0x55e74863e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.916+0000 7f1c56ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748644f50 0x55e74863e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37102/0 (socket says 172.21.15.1:37102) 2024-03-06T07:25:00.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.916+0000 7f1c56ffd640 1 -- 172.21.15.1:0/3510902720 learned_addr learned my addr 172.21.15.1:0/3510902720 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:00.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.916+0000 7f1c577fe640 1 --2- 172.21.15.1:0/3510902720 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748691480 0x55e74863fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.916+0000 7f1c567fc640 1 --2- 172.21.15.1:0/3510902720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e748645330 0x55e74863f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.916+0000 7f1c56ffd640 1 -- 172.21.15.1:0/3510902720 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748691480 msgr2=0x55e74863fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.916+0000 7f1c56ffd640 1 --2- 172.21.15.1:0/3510902720 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748691480 0x55e74863fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.916+0000 7f1c56ffd640 1 -- 172.21.15.1:0/3510902720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e748645330 msgr2=0x55e74863f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.916+0000 7f1c56ffd640 1 --2- 172.21.15.1:0/3510902720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e748645330 0x55e74863f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.916+0000 7f1c56ffd640 1 -- 172.21.15.1:0/3510902720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7483886b0 con 0x55e748644f50 2024-03-06T07:25:00.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.916+0000 7f1c56ffd640 1 --2- 172.21.15.1:0/3510902720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748644f50 0x55e74863e700 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f1c48013ad0 tx=0x7f1c48025000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23f52a3c597b9186 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.916+0000 7f1c37fff640 1 -- 172.21.15.1:0/3510902720 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48015790 con 0x55e748644f50 2024-03-06T07:25:00.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.916+0000 7f1c37fff640 1 -- 172.21.15.1:0/3510902720 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4801c030 con 0x55e748644f50 2024-03-06T07:25:00.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.916+0000 7f1c37fff640 1 -- 172.21.15.1:0/3510902720 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48015b30 con 0x55e748644f50 2024-03-06T07:25:00.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.917+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/3510902720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748644f50 msgr2=0x55e74863e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.917+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/3510902720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748644f50 0x55e74863e700 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f1c48013ad0 tx=0x7f1c48025000 comp rx=0 tx=0).stop 2024-03-06T07:25:00.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.917+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/3510902720 shutdown_connections 2024-03-06T07:25:00.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.917+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/3510902720 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748691480 0x55e74863fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.917+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/3510902720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e748645330 0x55e74863f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.917+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/3510902720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748644f50 0x55e74863e700 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.917+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/3510902720 >> 172.21.15.1:0/3510902720 conn(0x55e7486434b0 msgr2=0x55e74863ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.917+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/3510902720 shutdown_connections 2024-03-06T07:25:00.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.917+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/3510902720 wait complete. 2024-03-06T07:25:00.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.918+0000 7f1c5ffd5740 1 -- start start 2024-03-06T07:25:00.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.918+0000 7f1c5ffd5740 1 Processor -- start 2024-03-06T07:25:00.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c5ffd5740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748644f50 0x55e748668450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c5ffd5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e748645330 0x55e7486f4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c5ffd5740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748691480 0x55e7486ee4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c5ffd5740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e748691d50 con 0x55e748691480 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c5ffd5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e748668df0 con 0x55e748645330 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c5ffd5740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e7486954d0 con 0x55e748644f50 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c5ffd5740 10 client.0 _ll_register_callbacks cb 0x7f1c48016460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c577fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748691480 0x55e7486ee4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c577fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748691480 0x55e7486ee4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37116/0 (socket says 172.21.15.1:37116) 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c577fe640 1 -- 172.21.15.1:0/1447838976 learned_addr learned my addr 172.21.15.1:0/1447838976 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c577fe640 1 -- 172.21.15.1:0/1447838976 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748644f50 msgr2=0x55e748668450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c577fe640 1 --2- 172.21.15.1:0/1447838976 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748644f50 0x55e748668450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c577fe640 1 -- 172.21.15.1:0/1447838976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e748645330 msgr2=0x55e7486f4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c577fe640 1 --2- 172.21.15.1:0/1447838976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e748645330 0x55e7486f4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c577fe640 1 -- 172.21.15.1:0/1447838976 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e7486f12f0 con 0x55e748691480 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c577fe640 1 --2- 172.21.15.1:0/1447838976 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748691480 0x55e7486ee4b0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f1c580c5b10 tx=0x7f1c50017800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ca1dd265b3228d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:25:00.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c35ffb640 1 -- 172.21.15.1:0/1447838976 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c50006b50 con 0x55e748691480 2024-03-06T07:25:00.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.919+0000 7f1c35ffb640 1 -- 172.21.15.1:0/1447838976 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1c50006cf0 con 0x55e748691480 2024-03-06T07:25:00.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.920+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/1447838976 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1c4801c030 con 0x55e748691480 2024-03-06T07:25:00.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.920+0000 7f1c35ffb640 1 -- 172.21.15.1:0/1447838976 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c50026050 con 0x55e748691480 2024-03-06T07:25:00.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.920+0000 7f1c5ffd5740 10 client.8055 fetch_fsmap learned FSMap version 106 2024-03-06T07:25:00.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.920+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/1447838976 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=2+}) v3 -- 0x55e7486f5660 con 0x55e748691480 2024-03-06T07:25:00.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.920+0000 7f1c35ffb640 1 -- 172.21.15.1:0/1447838976 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4801c030 con 0x55e748691480 2024-03-06T07:25:00.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.920+0000 7f1c35ffb640 1 -- 172.21.15.1:0/1447838976 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 106) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f1c50029020 con 0x55e748691480 2024-03-06T07:25:00.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.920+0000 7f1c5ffd5740 10 client.8055 fetch_fsmap finished waiting for FSMap version 106 2024-03-06T07:25:00.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.920+0000 7f1c5ffd5740 10 client.8055 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:25:00.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.920+0000 7f1c5ffd5740 20 client.8055 populate_metadata read hostname 'smithi001' 2024-03-06T07:25:00.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.920+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/1447838976 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x55e748684650 0x55e748684a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.920+0000 7f1c5ffd5740 4 client.8055 mds_command: new command op to 7863 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:00.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.920+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/1447838976 --> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e7485b0f40 con 0x55e748684650 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.920+0000 7f1c56ffd640 1 --2- 172.21.15.1:0/1447838976 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x55e748684650 0x55e748684a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.920+0000 7f1c35ffb640 10 client.8055 ms_handle_connect on v2:172.21.15.1:6832/1515309674 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.920+0000 7f1c56ffd640 1 --2- 172.21.15.1:0/1447838976 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x55e748684650 0x55e748684a30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ff84272c7c470b0 server_cookie=4b13ec09e58cac80 in_seq=0 out_seq=0 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c35ffb640 1 -- 172.21.15.1:0/1447838976 <== mds.0 v2:172.21.15.1:6832/1515309674 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55e7485b0f40 con 0x55e748684650 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c35ffb640 10 client.8055 handle_command_reply: tid=0 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5ffd5740 1 client.8055 shutdown 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/1447838976 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748691480 msgr2=0x55e7486ee4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/1447838976 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748691480 0x55e7486ee4b0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f1c580c5b10 tx=0x7f1c50017800 comp rx=0 tx=0).stop 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/1447838976 shutdown_connections 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/1447838976 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x55e748684650 0x55e748684a30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/1447838976 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e748691480 0x55e7486ee4b0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/1447838976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e748645330 0x55e7486f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c35ffb640 0 client.8055 ms_handle_reset on v2:172.21.15.1:6832/1515309674 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5ffd5740 1 --2- 172.21.15.1:0/1447838976 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e748644f50 0x55e748668450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/1447838976 >> 172.21.15.1:0/1447838976 conn(0x55e7486434b0 msgr2=0x55e74863e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/1447838976 shutdown_connections 2024-03-06T07:25:00.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5ffd5740 1 -- 172.21.15.1:0/1447838976 wait complete. 2024-03-06T07:25:00.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5ffd5740 20 client.8055 trim_cache size 0 max 16384 2024-03-06T07:25:00.928 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:25:00.928 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:25:00.928 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:25:00.928 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 7863, 2024-03-06T07:25:00.928 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:25:00.929 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:25:00.929 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:25:00.929 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 7.6747777519999998, 2024-03-06T07:25:00.929 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 106, 2024-03-06T07:25:00.929 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 121, 2024-03-06T07:25:00.929 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 121, 2024-03-06T07:25:00.929 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 12.765917878 2024-03-06T07:25:00.929 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:25:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5d3e9640 1 -- 172.21.15.1:0/4048384035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c4406b500 msgr2=0x7f1c4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5d3e9640 1 --2- 172.21.15.1:0/4048384035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c4406b500 0x7f1c4406d9c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1c48009a60 tx=0x7f1c48008040 comp rx=0 tx=0).stop 2024-03-06T07:25:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5d3e9640 1 -- 172.21.15.1:0/4048384035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c58156d20 msgr2=0x7f1c581549c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5d3e9640 1 --2- 172.21.15.1:0/4048384035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c58156d20 0x7f1c581549c0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c00c330 tx=0x7f1c581559e0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5d3e9640 1 -- 172.21.15.1:0/4048384035 shutdown_connections 2024-03-06T07:25:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5d3e9640 1 --2- 172.21.15.1:0/4048384035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c4406b500 0x7f1c4406d9c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5d3e9640 1 --2- 172.21.15.1:0/4048384035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c5814ce10 0x7f1c581ead90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5d3e9640 1 --2- 172.21.15.1:0/4048384035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c58156d20 0x7f1c581549c0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5d3e9640 1 --2- 172.21.15.1:0/4048384035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c40004800 0x7f1c58159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.921+0000 7f1c5d3e9640 1 -- 172.21.15.1:0/4048384035 >> 172.21.15.1:0/4048384035 conn(0x7f1c58072720 msgr2=0x7f1c58151500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.922+0000 7f1c5d3e9640 1 -- 172.21.15.1:0/4048384035 shutdown_connections 2024-03-06T07:25:00.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:00.922+0000 7f1c5d3e9640 1 -- 172.21.15.1:0/4048384035 wait complete. 2024-03-06T07:25:00.934 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 106, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:24:52.598303+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 106, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7863": { "addr": "172.21.15.1:6834/1515309674", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 1515309674, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 1515309674, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7863, "incarnation": 102, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-03-06T07:24:55.621912+0000", "qdb_cluster": [ 7863 ], "qdb_leader": 7863, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7863 } } } ], "standbys": [ { "addr": "172.21.15.1:6835/4151599120", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 4151599120, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 4151599120, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6446, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6837/155600632", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 155600632, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 155600632, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6458, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6836/1420541659", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 1420541659, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 1420541659, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7084, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:25:00.935 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 9, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:25:00.935 INFO:tasks.cephfs.test_failover:current = [{'gid': 6446, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/4151599120', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 4151599120}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 4151599120}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}, {'gid': 7084, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/1420541659', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 1420541659}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1420541659}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}, {'gid': 7863, 'name': 'c', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6834/1515309674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1515309674}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1515309674}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6458, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/155600632', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 155600632}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 155600632}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}] 2024-03-06T07:25:00.935 DEBUG:tasks.cephfs.test_failover:9 != -1 2024-03-06T07:25:00.935 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:25:01.647 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:01.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:05.937 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.036+0000 7fdf7fdb6640 1 Processor -- start 2024-03-06T07:25:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.036+0000 7fdf7fdb6640 1 -- start start 2024-03-06T07:25:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.037+0000 7fdf7fdb6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf7806e670 0x7fdf7806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.037+0000 7fdf7fdb6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdf780a8520 0x7fdf780c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.037+0000 7fdf7fdb6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdf78150290 0x7fdf78150720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.037+0000 7fdf7fdb6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf780c2ca0 con 0x7fdf78150290 2024-03-06T07:25:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.037+0000 7fdf7fdb6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf7806f060 con 0x7fdf780a8520 2024-03-06T07:25:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.037+0000 7fdf7fdb6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf780c26e0 con 0x7fdf7806e670 2024-03-06T07:25:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.037+0000 7fdf7e353640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdf78150290 0x7fdf78150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.037+0000 7fdf7e353640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdf78150290 0x7fdf78150720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59524/0 (socket says 172.21.15.1:59524) 2024-03-06T07:25:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.037+0000 7fdf7e353640 1 -- 172.21.15.1:0/3880965959 learned_addr learned my addr 172.21.15.1:0/3880965959 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.037+0000 7fdf7d351640 1 --2- 172.21.15.1:0/3880965959 >> v2:172.21.15.98:3300/0 conn(0x7fdf780a8520 0x7fdf780c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.037+0000 7fdf7db52640 1 --2- 172.21.15.1:0/3880965959 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf7806e670 0x7fdf7806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.037+0000 7fdf7e353640 1 -- 172.21.15.1:0/3880965959 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf7806e670 msgr2=0x7fdf7806ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.037+0000 7fdf7e353640 1 --2- 172.21.15.1:0/3880965959 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf7806e670 0x7fdf7806ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.037+0000 7fdf7e353640 1 -- 172.21.15.1:0/3880965959 >> v2:172.21.15.98:3300/0 conn(0x7fdf780a8520 msgr2=0x7fdf780c1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.037+0000 7fdf7e353640 1 --2- 172.21.15.1:0/3880965959 >> v2:172.21.15.98:3300/0 conn(0x7fdf780a8520 0x7fdf780c1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.037+0000 7fdf7e353640 1 -- 172.21.15.1:0/3880965959 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf780c29c0 con 0x7fdf78150290 2024-03-06T07:25:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.038+0000 7fdf7db52640 1 --2- 172.21.15.1:0/3880965959 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf7806e670 0x7fdf7806ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.038+0000 7fdf7d351640 1 --2- 172.21.15.1:0/3880965959 >> v2:172.21.15.98:3300/0 conn(0x7fdf780a8520 0x7fdf780c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.038+0000 7fdf7e353640 1 --2- 172.21.15.1:0/3880965959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf78150290 0x7fdf78150720 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fdf7806ca70 tx=0x7fdf7002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed298b07d02b0db8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.038+0000 7fdf7cb50640 1 -- 172.21.15.1:0/3880965959 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf700317d0 con 0x7fdf78150290 2024-03-06T07:25:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.038+0000 7fdf7cb50640 1 -- 172.21.15.1:0/3880965959 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf70037070 con 0x7fdf78150290 2024-03-06T07:25:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.038+0000 7fdf7cb50640 1 -- 172.21.15.1:0/3880965959 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf70031b90 con 0x7fdf78150290 2024-03-06T07:25:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.039+0000 7fdf7fdb6640 1 -- 172.21.15.1:0/3880965959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf78150290 msgr2=0x7fdf78150720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.039+0000 7fdf7fdb6640 1 --2- 172.21.15.1:0/3880965959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf78150290 0x7fdf78150720 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fdf7806ca70 tx=0x7fdf7002b870 comp rx=0 tx=0).stop 2024-03-06T07:25:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.039+0000 7fdf7fdb6640 1 -- 172.21.15.1:0/3880965959 shutdown_connections 2024-03-06T07:25:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.039+0000 7fdf7fdb6640 1 --2- 172.21.15.1:0/3880965959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf78150290 0x7fdf78150720 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.039+0000 7fdf7fdb6640 1 --2- 172.21.15.1:0/3880965959 >> v2:172.21.15.98:3300/0 conn(0x7fdf780a8520 0x7fdf780c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.039+0000 7fdf7fdb6640 1 --2- 172.21.15.1:0/3880965959 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf7806e670 0x7fdf7806ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.039+0000 7fdf7fdb6640 1 -- 172.21.15.1:0/3880965959 >> 172.21.15.1:0/3880965959 conn(0x7fdf780c3cc0 msgr2=0x7fdf780c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.039+0000 7fdf7fdb6640 1 -- 172.21.15.1:0/3880965959 shutdown_connections 2024-03-06T07:25:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.039+0000 7fdf7fdb6640 1 -- 172.21.15.1:0/3880965959 wait complete. 2024-03-06T07:25:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.040+0000 7fdf7fdb6640 1 Processor -- start 2024-03-06T07:25:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.040+0000 7fdf7fdb6640 1 -- start start 2024-03-06T07:25:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7fdb6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7806e670 0x7fdf781e9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7fdb6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf780a8520 0x7fdf781e90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7fdb6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf78150290 0x7fdf7814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7fdb6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf78153620 con 0x7fdf7806e670 2024-03-06T07:25:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7fdb6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf781534a0 con 0x7fdf780a8520 2024-03-06T07:25:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7fdb6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf781537a0 con 0x7fdf78150290 2024-03-06T07:25:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7db52640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7806e670 0x7fdf781e9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7db52640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7806e670 0x7fdf781e9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59530/0 (socket says 172.21.15.1:59530) 2024-03-06T07:25:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7db52640 1 -- 172.21.15.1:0/1693217753 learned_addr learned my addr 172.21.15.1:0/1693217753 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7d351640 1 --2- 172.21.15.1:0/1693217753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf780a8520 0x7fdf781e90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7e353640 1 --2- 172.21.15.1:0/1693217753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf78150290 0x7fdf7814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7db52640 1 -- 172.21.15.1:0/1693217753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf78150290 msgr2=0x7fdf7814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7db52640 1 --2- 172.21.15.1:0/1693217753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf78150290 0x7fdf7814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7db52640 1 -- 172.21.15.1:0/1693217753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf780a8520 msgr2=0x7fdf781e90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7db52640 1 --2- 172.21.15.1:0/1693217753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf780a8520 0x7fdf781e90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7db52640 1 -- 172.21.15.1:0/1693217753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf78152210 con 0x7fdf7806e670 2024-03-06T07:25:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7d351640 1 --2- 172.21.15.1:0/1693217753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf780a8520 0x7fdf781e90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7e353640 1 --2- 172.21.15.1:0/1693217753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf78150290 0x7fdf7814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.041+0000 7fdf7db52640 1 --2- 172.21.15.1:0/1693217753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7806e670 0x7fdf781e9ad0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800cee0 tx=0x7fdf6800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.042+0000 7fdf767fc640 1 -- 172.21.15.1:0/1693217753 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6801a070 con 0x7fdf7806e670 2024-03-06T07:25:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.042+0000 7fdf767fc640 1 -- 172.21.15.1:0/1693217753 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6800d840 con 0x7fdf7806e670 2024-03-06T07:25:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.042+0000 7fdf7fdb6640 1 -- 172.21.15.1:0/1693217753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf781524f0 con 0x7fdf7806e670 2024-03-06T07:25:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.042+0000 7fdf767fc640 1 -- 172.21.15.1:0/1693217753 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6800dc10 con 0x7fdf7806e670 2024-03-06T07:25:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.042+0000 7fdf7fdb6640 1 -- 172.21.15.1:0/1693217753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf781510e0 con 0x7fdf7806e670 2024-03-06T07:25:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.043+0000 7fdf7fdb6640 1 -- 172.21.15.1:0/1693217753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf78067570 con 0x7fdf7806e670 2024-03-06T07:25:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.043+0000 7fdf767fc640 1 -- 172.21.15.1:0/1693217753 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf6801d070 con 0x7fdf7806e670 2024-03-06T07:25:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.043+0000 7fdf767fc640 1 --2- 172.21.15.1:0/1693217753 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf6406b4d0 0x7fdf6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.044+0000 7fdf767fc640 1 -- 172.21.15.1:0/1693217753 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdf6808aa00 con 0x7fdf7806e670 2024-03-06T07:25:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.044+0000 7fdf7d351640 1 --2- 172.21.15.1:0/1693217753 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf6406b4d0 0x7fdf6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.044+0000 7fdf7d351640 1 --2- 172.21.15.1:0/1693217753 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf6406b4d0 0x7fdf6406d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c006fd0 tx=0x7fdf6c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.048+0000 7fdf767fc640 1 -- 172.21.15.1:0/1693217753 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf68059130 con 0x7fdf7806e670 2024-03-06T07:25:06.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.267+0000 7fdf7fdb6640 1 -- 172.21.15.1:0/1693217753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf781506a0 con 0x7fdf7806e670 2024-03-06T07:25:06.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.267+0000 7fdf767fc640 1 -- 172.21.15.1:0/1693217753 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4975 (secure 0 0 0) 0x7fdf6805d1d0 con 0x7fdf7806e670 2024-03-06T07:25:06.269 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:06.269 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":107,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6446,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4151599120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4151599120},{"type":"v1","addr":"172.21.15.1:6835","nonce":4151599120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/155600632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":155600632},{"type":"v1","addr":"172.21.15.98:6837","nonce":155600632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7084,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1420541659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1420541659},{"type":"v1","addr":"172.21.15.98:6836","nonce":1420541659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.598303+0000","modified":"2024-03-06T07:25:01.273589+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7863},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7863":{"gid":7863,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":5,"addr":"172.21.15.1:6834/1515309674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1515309674},{"type":"v1","addr":"172.21.15.1:6834","nonce":1515309674}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7863,"qdb_cluster":[7863]},"id":9}]} 2024-03-06T07:25:06.270 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 107 2024-03-06T07:25:06.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.269+0000 7fdf7fdb6640 1 -- 172.21.15.1:0/1693217753 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf6406b4d0 msgr2=0x7fdf6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.269+0000 7fdf7fdb6640 1 --2- 172.21.15.1:0/1693217753 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf6406b4d0 0x7fdf6406d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c006fd0 tx=0x7fdf6c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:06.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.269+0000 7fdf7fdb6640 1 -- 172.21.15.1:0/1693217753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7806e670 msgr2=0x7fdf781e9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.269+0000 7fdf7fdb6640 1 --2- 172.21.15.1:0/1693217753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7806e670 0x7fdf781e9ad0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800cee0 tx=0x7fdf6800a710 comp rx=0 tx=0).stop 2024-03-06T07:25:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.269+0000 7fdf7fdb6640 1 -- 172.21.15.1:0/1693217753 shutdown_connections 2024-03-06T07:25:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.269+0000 7fdf7fdb6640 1 --2- 172.21.15.1:0/1693217753 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf6406b4d0 0x7fdf6406d990 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.269+0000 7fdf7fdb6640 1 --2- 172.21.15.1:0/1693217753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf78150290 0x7fdf7814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.269+0000 7fdf7fdb6640 1 --2- 172.21.15.1:0/1693217753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf780a8520 0x7fdf781e90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.269+0000 7fdf7fdb6640 1 --2- 172.21.15.1:0/1693217753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7806e670 0x7fdf781e9ad0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.269+0000 7fdf7fdb6640 1 -- 172.21.15.1:0/1693217753 >> 172.21.15.1:0/1693217753 conn(0x7fdf780c3cc0 msgr2=0x7fdf780a8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.269+0000 7fdf7fdb6640 1 -- 172.21.15.1:0/1693217753 shutdown_connections 2024-03-06T07:25:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.269+0000 7fdf7fdb6640 1 -- 172.21.15.1:0/1693217753 wait complete. 2024-03-06T07:25:06.684 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:52.598303+0000', 'modified': '2024-03-06T07:25:01.273589+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7863}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7863': {'gid': 7863, 'name': 'c', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.1:6834/1515309674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1515309674}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1515309674}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7863, 'qdb_cluster': [7863]} 2024-03-06T07:25:06.684 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:06.685 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:25:06.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.785+0000 7f23059f6640 1 Processor -- start 2024-03-06T07:25:06.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.785+0000 7f23059f6640 1 -- start start 2024-03-06T07:25:06.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.786+0000 7f23059f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2300152650 0x7f2300155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.787+0000 7f23059f6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23000bf9d0 0x7f23000c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.787+0000 7f23059f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23000c2fc0 con 0x7f23000c6b10 2024-03-06T07:25:06.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.787+0000 7f23059f6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2300155800 con 0x7f23000bf9d0 2024-03-06T07:25:06.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.787+0000 7f23059f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2300158da0 con 0x7f2300152650 2024-03-06T07:25:06.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.787+0000 7f22ff7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23000bf9d0 0x7f23000c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.787+0000 7f22ff7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23000bf9d0 0x7f23000c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59532/0 (socket says 172.21.15.1:59532) 2024-03-06T07:25:06.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.787+0000 7f22ff7fe640 1 -- 172.21.15.1:0/3701908849 learned_addr learned my addr 172.21.15.1:0/3701908849 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:06.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.787+0000 7f22ff7fe640 1 -- 172.21.15.1:0/3701908849 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2300152650 msgr2=0x7f2300155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.787+0000 7f22feffd640 1 --2- 172.21.15.1:0/3701908849 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2300152650 0x7f2300155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.787+0000 7f22ff7fe640 1 --2- 172.21.15.1:0/3701908849 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2300152650 0x7f2300155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.787+0000 7f22ff7fe640 1 -- 172.21.15.1:0/3701908849 >> v1:172.21.15.98:6789/0 conn(0x7f23000c6b10 legacy=0x7f2300158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.787+0000 7f22ff7fe640 1 -- 172.21.15.1:0/3701908849 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23000c6690 con 0x7f23000bf9d0 2024-03-06T07:25:06.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.787+0000 7f22feffd640 1 --2- 172.21.15.1:0/3701908849 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2300152650 0x7f2300155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:06.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.788+0000 7f22ff7fe640 1 --2- 172.21.15.1:0/3701908849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23000bf9d0 0x7f23000c1460 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009e50 tx=0x7f22f802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae314841923b274d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.788+0000 7f22fdffb640 1 -- 172.21.15.1:0/3701908849 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f802c9e0 con 0x7f23000bf9d0 2024-03-06T07:25:06.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.788+0000 7f22fdffb640 1 -- 172.21.15.1:0/3701908849 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f802cb80 con 0x7f23000bf9d0 2024-03-06T07:25:06.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.788+0000 7f22fdffb640 1 -- 172.21.15.1:0/3701908849 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f80343f0 con 0x7f23000bf9d0 2024-03-06T07:25:06.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.788+0000 7f23059f6640 1 -- 172.21.15.1:0/3701908849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23000bf9d0 msgr2=0x7f23000c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.788+0000 7f23059f6640 1 --2- 172.21.15.1:0/3701908849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23000bf9d0 0x7f23000c1460 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009e50 tx=0x7f22f802be40 comp rx=0 tx=0).stop 2024-03-06T07:25:06.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.789+0000 7f23059f6640 1 -- 172.21.15.1:0/3701908849 shutdown_connections 2024-03-06T07:25:06.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.789+0000 7f23059f6640 1 --2- 172.21.15.1:0/3701908849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23000bf9d0 0x7f23000c1460 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.789+0000 7f23059f6640 1 --2- 172.21.15.1:0/3701908849 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2300152650 0x7f2300155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.789+0000 7f23059f6640 1 -- 172.21.15.1:0/3701908849 >> 172.21.15.1:0/3701908849 conn(0x7f2300066ee0 msgr2=0x7f23000672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.789+0000 7f23059f6640 1 -- 172.21.15.1:0/3701908849 shutdown_connections 2024-03-06T07:25:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.789+0000 7f23059f6640 1 -- 172.21.15.1:0/3701908849 wait complete. 2024-03-06T07:25:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.790+0000 7f23059f6640 1 Processor -- start 2024-03-06T07:25:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.790+0000 7f23059f6640 1 -- start start 2024-03-06T07:25:06.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.790+0000 7f23059f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23000bf9d0 0x7f23001ee780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.790+0000 7f23059f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2300152650 0x7f23000c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.790+0000 7f23059f6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23000c3e30 0x7f23000c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.790+0000 7f23059f6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23001551b0 con 0x7f23000c3e30 2024-03-06T07:25:06.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.790+0000 7f23059f6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23000c6fa0 con 0x7f2300152650 2024-03-06T07:25:06.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.790+0000 7f23059f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2300155490 con 0x7f23000bf9d0 2024-03-06T07:25:06.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.791+0000 7f22ff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23000c3e30 0x7f23000c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.791+0000 7f22ff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23000c3e30 0x7f23000c9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59536/0 (socket says 172.21.15.1:59536) 2024-03-06T07:25:06.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.791+0000 7f22ff7fe640 1 -- 172.21.15.1:0/3356234172 learned_addr learned my addr 172.21.15.1:0/3356234172 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:06.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.791+0000 7f22fe7fc640 1 --2- 172.21.15.1:0/3356234172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2300152650 0x7f23000c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.791+0000 7f22feffd640 1 --2- 172.21.15.1:0/3356234172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23000bf9d0 0x7f23001ee780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.791+0000 7f22ff7fe640 1 -- 172.21.15.1:0/3356234172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23000bf9d0 msgr2=0x7f23001ee780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.791+0000 7f22ff7fe640 1 --2- 172.21.15.1:0/3356234172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23000bf9d0 0x7f23001ee780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.791+0000 7f22ff7fe640 1 -- 172.21.15.1:0/3356234172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2300152650 msgr2=0x7f23000c9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.791+0000 7f22ff7fe640 1 --2- 172.21.15.1:0/3356234172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2300152650 0x7f23000c9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.791+0000 7f22ff7fe640 1 -- 172.21.15.1:0/3356234172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23001ef200 con 0x7f23000c3e30 2024-03-06T07:25:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.791+0000 7f22fe7fc640 1 --2- 172.21.15.1:0/3356234172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2300152650 0x7f23000c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.791+0000 7f22ff7fe640 1 --2- 172.21.15.1:0/3356234172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23000c3e30 0x7f23000c9760 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009e50 tx=0x7f22f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.791+0000 7f22e7fff640 1 -- 172.21.15.1:0/3356234172 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f8043020 con 0x7f23000c3e30 2024-03-06T07:25:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.791+0000 7f22e7fff640 1 -- 172.21.15.1:0/3356234172 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f803e450 con 0x7f23000c3e30 2024-03-06T07:25:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.791+0000 7f23059f6640 1 -- 172.21.15.1:0/3356234172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23000ca200 con 0x7f23000c3e30 2024-03-06T07:25:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.791+0000 7f22e7fff640 1 -- 172.21.15.1:0/3356234172 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f803e860 con 0x7f23000c3e30 2024-03-06T07:25:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.791+0000 7f23059f6640 1 -- 172.21.15.1:0/3356234172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2300157310 con 0x7f23000c3e30 2024-03-06T07:25:06.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.793+0000 7f22e7fff640 1 -- 172.21.15.1:0/3356234172 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f22f8046020 con 0x7f23000c3e30 2024-03-06T07:25:06.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.794+0000 7f22e7fff640 1 --2- 172.21.15.1:0/3356234172 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f22ec06b4d0 0x7f22ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.794+0000 7f22e7fff640 1 -- 172.21.15.1:0/3356234172 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f22f80ace00 con 0x7f23000c3e30 2024-03-06T07:25:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.794+0000 7f22feffd640 1 --2- 172.21.15.1:0/3356234172 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f22ec06b4d0 0x7f22ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.794+0000 7f22feffd640 1 --2- 172.21.15.1:0/3356234172 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f22ec06b4d0 0x7f22ec06d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f22f000a990 tx=0x7f22f0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.795+0000 7f23085d9740 1 Processor -- start 2024-03-06T07:25:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.795+0000 7f23085d9740 1 -- start start 2024-03-06T07:25:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.795+0000 7f23085d9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8adf50 0x55de4b8aba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.795+0000 7f23085d9740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8ae330 0x55de4b8abf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f23085d9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8ac4b0 0x55de4b8a8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f23085d9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de4b8126b0 con 0x55de4b8adf50 2024-03-06T07:25:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f23085d9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de4b7bba60 con 0x55de4b8ac4b0 2024-03-06T07:25:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f23085d9740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de4b80e750 con 0x55de4b8ae330 2024-03-06T07:25:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f22feffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8adf50 0x55de4b8aba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f22feffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8adf50 0x55de4b8aba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59550/0 (socket says 172.21.15.1:59550) 2024-03-06T07:25:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f22feffd640 1 -- 172.21.15.1:0/3333964107 learned_addr learned my addr 172.21.15.1:0/3333964107 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f22fe7fc640 1 --2- 172.21.15.1:0/3333964107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8ae330 0x55de4b8abf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f22ff7fe640 1 --2- 172.21.15.1:0/3333964107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8ac4b0 0x55de4b8a8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f22feffd640 1 -- 172.21.15.1:0/3333964107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8ae330 msgr2=0x55de4b8abf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f22feffd640 1 --2- 172.21.15.1:0/3333964107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8ae330 0x55de4b8abf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f22feffd640 1 -- 172.21.15.1:0/3333964107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8ac4b0 msgr2=0x55de4b8a8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f22feffd640 1 --2- 172.21.15.1:0/3333964107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8ac4b0 0x55de4b8a8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f22feffd640 1 -- 172.21.15.1:0/3333964107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de4b7d9f30 con 0x55de4b8adf50 2024-03-06T07:25:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f22feffd640 1 --2- 172.21.15.1:0/3333964107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8adf50 0x55de4b8aba30 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f22f000fa50 tx=0x7f22f0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8790f36e5c7d53e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f22e57fa640 1 -- 172.21.15.1:0/3333964107 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f001b050 con 0x55de4b8adf50 2024-03-06T07:25:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f22e57fa640 1 -- 172.21.15.1:0/3333964107 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f0025070 con 0x55de4b8adf50 2024-03-06T07:25:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.796+0000 7f22e57fa640 1 -- 172.21.15.1:0/3333964107 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f0014de0 con 0x55de4b8adf50 2024-03-06T07:25:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.797+0000 7f23085d9740 1 -- 172.21.15.1:0/3333964107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8adf50 msgr2=0x55de4b8aba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.797+0000 7f23085d9740 1 --2- 172.21.15.1:0/3333964107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8adf50 0x55de4b8aba30 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f22f000fa50 tx=0x7f22f0012000 comp rx=0 tx=0).stop 2024-03-06T07:25:06.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.797+0000 7f23085d9740 1 -- 172.21.15.1:0/3333964107 shutdown_connections 2024-03-06T07:25:06.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.797+0000 7f23085d9740 1 --2- 172.21.15.1:0/3333964107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8ac4b0 0x55de4b8a8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.797+0000 7f23085d9740 1 --2- 172.21.15.1:0/3333964107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8ae330 0x55de4b8abf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.797+0000 7f23085d9740 1 --2- 172.21.15.1:0/3333964107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8adf50 0x55de4b8aba30 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.797+0000 7f23085d9740 1 -- 172.21.15.1:0/3333964107 >> 172.21.15.1:0/3333964107 conn(0x55de4b8aa7d0 msgr2=0x55de4b8cf3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.797+0000 7f23085d9740 1 -- 172.21.15.1:0/3333964107 shutdown_connections 2024-03-06T07:25:06.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.797+0000 7f23085d9740 1 -- 172.21.15.1:0/3333964107 wait complete. 2024-03-06T07:25:06.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.798+0000 7f23085d9740 1 -- start start 2024-03-06T07:25:06.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.798+0000 7f23085d9740 1 Processor -- start 2024-03-06T07:25:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.798+0000 7f23085d9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8adf50 0x55de4b8e7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.798+0000 7f23085d9740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8ae330 0x55de4b8ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.798+0000 7f23085d9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8d0600 0x55de4b8d0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.798+0000 7f23085d9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de4b8d0230 con 0x55de4b8adf50 2024-03-06T07:25:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.798+0000 7f23085d9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de4b7ceea0 con 0x55de4b8d0600 2024-03-06T07:25:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.798+0000 7f23085d9740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de4b8cfc70 con 0x55de4b8ae330 2024-03-06T07:25:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.798+0000 7f23085d9740 10 client.0 _ll_register_callbacks cb 0x55de4b7ce870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.798+0000 7f22feffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8adf50 0x55de4b8e7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f22feffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8adf50 0x55de4b8e7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59556/0 (socket says 172.21.15.1:59556) 2024-03-06T07:25:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f22feffd640 1 -- 172.21.15.1:0/4224473964 learned_addr learned my addr 172.21.15.1:0/4224473964 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f22ff7fe640 1 --2- 172.21.15.1:0/4224473964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8d0600 0x55de4b8d0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f22fe7fc640 1 --2- 172.21.15.1:0/4224473964 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8ae330 0x55de4b8ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f22feffd640 1 -- 172.21.15.1:0/4224473964 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8ae330 msgr2=0x55de4b8ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f22feffd640 1 --2- 172.21.15.1:0/4224473964 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8ae330 0x55de4b8ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f22feffd640 1 -- 172.21.15.1:0/4224473964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8d0600 msgr2=0x55de4b8d0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f22feffd640 1 --2- 172.21.15.1:0/4224473964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8d0600 0x55de4b8d0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f22feffd640 1 -- 172.21.15.1:0/4224473964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de4b7cf270 con 0x55de4b8adf50 2024-03-06T07:25:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f22feffd640 1 --2- 172.21.15.1:0/4224473964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8adf50 0x55de4b8e7640 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f22f0008010 tx=0x7f22f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b115d7395dfe15 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f22de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:25:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f22de7fc640 1 -- 172.21.15.1:0/4224473964 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f0020a90 con 0x55de4b8adf50 2024-03-06T07:25:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f22de7fc640 1 -- 172.21.15.1:0/4224473964 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f22f0034cb0 con 0x55de4b8adf50 2024-03-06T07:25:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f23085d9740 1 -- 172.21.15.1:0/4224473964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f22f0025070 con 0x55de4b8adf50 2024-03-06T07:25:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f22de7fc640 1 -- 172.21.15.1:0/4224473964 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22f0020c30 con 0x55de4b8adf50 2024-03-06T07:25:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f23085d9740 10 client.8073 fetch_fsmap learned FSMap version 107 2024-03-06T07:25:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f23085d9740 1 -- 172.21.15.1:0/4224473964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) v3 -- 0x55de4b5272d0 con 0x55de4b8adf50 2024-03-06T07:25:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.799+0000 7f22de7fc640 1 -- 172.21.15.1:0/4224473964 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f0025070 con 0x55de4b8adf50 2024-03-06T07:25:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.800+0000 7f22de7fc640 1 -- 172.21.15.1:0/4224473964 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 107) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f22f0033470 con 0x55de4b8adf50 2024-03-06T07:25:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.800+0000 7f23085d9740 10 client.8073 fetch_fsmap finished waiting for FSMap version 107 2024-03-06T07:25:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.800+0000 7f23085d9740 10 client.8073 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:25:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.800+0000 7f23085d9740 20 client.8073 populate_metadata read hostname 'smithi001' 2024-03-06T07:25:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.800+0000 7f23085d9740 1 --2- 172.21.15.1:0/4224473964 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x55de4b8fb130 0x55de4b9092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.800+0000 7f23085d9740 4 client.8073 mds_command: new command op to 7863 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.800+0000 7f23085d9740 1 -- 172.21.15.1:0/4224473964 --> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55de4b81fbf0 con 0x55de4b8fb130 2024-03-06T07:25:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.800+0000 7f22fe7fc640 1 --2- 172.21.15.1:0/4224473964 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x55de4b8fb130 0x55de4b9092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.800+0000 7f22de7fc640 10 client.8073 ms_handle_connect on v2:172.21.15.1:6832/1515309674 2024-03-06T07:25:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.800+0000 7f22fe7fc640 1 --2- 172.21.15.1:0/4224473964 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x55de4b8fb130 0x55de4b9092e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ad1ff1ed6414b2f server_cookie=7bc9d30f8f59b833 in_seq=0 out_seq=0 2024-03-06T07:25:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.801+0000 7f22de7fc640 1 -- 172.21.15.1:0/4224473964 <== mds.0 v2:172.21.15.1:6832/1515309674 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55de4b81fbf0 con 0x55de4b8fb130 2024-03-06T07:25:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.801+0000 7f22de7fc640 10 client.8073 handle_command_reply: tid=0 2024-03-06T07:25:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.801+0000 7f23085d9740 1 client.8073 shutdown 2024-03-06T07:25:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.801+0000 7f23085d9740 1 -- 172.21.15.1:0/4224473964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8adf50 msgr2=0x55de4b8e7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.801+0000 7f23085d9740 1 --2- 172.21.15.1:0/4224473964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8adf50 0x55de4b8e7640 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f22f0008010 tx=0x7f22f0004730 comp rx=0 tx=0).stop 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.801+0000 7f23085d9740 1 -- 172.21.15.1:0/4224473964 shutdown_connections 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.801+0000 7f23085d9740 1 --2- 172.21.15.1:0/4224473964 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x55de4b8fb130 0x55de4b9092e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.801+0000 7f23085d9740 1 --2- 172.21.15.1:0/4224473964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8d0600 0x55de4b8d0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.801+0000 7f23085d9740 1 --2- 172.21.15.1:0/4224473964 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8ae330 0x55de4b8ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.801+0000 7f23085d9740 1 --2- 172.21.15.1:0/4224473964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8adf50 0x55de4b8e7640 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.801+0000 7f23085d9740 1 -- 172.21.15.1:0/4224473964 >> 172.21.15.1:0/4224473964 conn(0x55de4b8ac4b0 msgr2=0x55de4b8eab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.801+0000 7f22de7fc640 0 client.8073 ms_handle_reset on v2:172.21.15.1:6832/1515309674 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.801+0000 7f23085d9740 1 -- 172.21.15.1:0/4224473964 shutdown_connections 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.801+0000 7f23085d9740 1 -- 172.21.15.1:0/4224473964 wait complete. 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.801+0000 7f23085d9740 20 client.8073 trim_cache size 0 max 16384 2024-03-06T07:25:06.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.818+0000 7f23085d9740 1 Processor -- start 2024-03-06T07:25:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.818+0000 7f23085d9740 1 -- start start 2024-03-06T07:25:06.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.818+0000 7f23085d9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8adf50 0x55de4b8f7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.818+0000 7f23085d9740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8ae330 0x55de4b8ed600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.818+0000 7f23085d9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8d0600 0x55de4b8ab790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.818+0000 7f23085d9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23001f6de0 con 0x55de4b8d0600 2024-03-06T07:25:06.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.818+0000 7f23085d9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23001eb8b0 con 0x55de4b8adf50 2024-03-06T07:25:06.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.818+0000 7f23085d9740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23001f6a50 con 0x55de4b8ae330 2024-03-06T07:25:06.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.819+0000 7f22ff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8d0600 0x55de4b8ab790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.819+0000 7f22ff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8d0600 0x55de4b8ab790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59572/0 (socket says 172.21.15.1:59572) 2024-03-06T07:25:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.819+0000 7f22ff7fe640 1 -- 172.21.15.1:0/3399604148 learned_addr learned my addr 172.21.15.1:0/3399604148 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.819+0000 7f22feffd640 1 --2- 172.21.15.1:0/3399604148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8adf50 0x55de4b8f7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.819+0000 7f22fe7fc640 1 --2- 172.21.15.1:0/3399604148 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8ae330 0x55de4b8ed600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.819+0000 7f22ff7fe640 1 -- 172.21.15.1:0/3399604148 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8ae330 msgr2=0x55de4b8ed600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.819+0000 7f22ff7fe640 1 --2- 172.21.15.1:0/3399604148 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8ae330 0x55de4b8ed600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.819+0000 7f22ff7fe640 1 -- 172.21.15.1:0/3399604148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8adf50 msgr2=0x55de4b8f7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.819+0000 7f22ff7fe640 1 --2- 172.21.15.1:0/3399604148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8adf50 0x55de4b8f7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.819+0000 7f22ff7fe640 1 -- 172.21.15.1:0/3399604148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de4b5f16b0 con 0x55de4b8d0600 2024-03-06T07:25:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.819+0000 7f22feffd640 1 --2- 172.21.15.1:0/3399604148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8adf50 0x55de4b8f7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.819+0000 7f22fe7fc640 1 --2- 172.21.15.1:0/3399604148 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8ae330 0x55de4b8ed600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.819+0000 7f22ff7fe640 1 --2- 172.21.15.1:0/3399604148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8d0600 0x55de4b8ab790 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f22f802c450 tx=0x7f22f802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49acf0ebcb4778be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.819+0000 7f22e4ff9640 1 -- 172.21.15.1:0/3399604148 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f807f6f0 con 0x55de4b8d0600 2024-03-06T07:25:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.819+0000 7f22e4ff9640 1 -- 172.21.15.1:0/3399604148 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f8083cb0 con 0x55de4b8d0600 2024-03-06T07:25:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.819+0000 7f22e4ff9640 1 -- 172.21.15.1:0/3399604148 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f8077c10 con 0x55de4b8d0600 2024-03-06T07:25:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.820+0000 7f23085d9740 1 -- 172.21.15.1:0/3399604148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8d0600 msgr2=0x55de4b8ab790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.820+0000 7f23085d9740 1 --2- 172.21.15.1:0/3399604148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8d0600 0x55de4b8ab790 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f22f802c450 tx=0x7f22f802be40 comp rx=0 tx=0).stop 2024-03-06T07:25:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.820+0000 7f23085d9740 1 -- 172.21.15.1:0/3399604148 shutdown_connections 2024-03-06T07:25:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.820+0000 7f23085d9740 1 --2- 172.21.15.1:0/3399604148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8d0600 0x55de4b8ab790 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.820+0000 7f23085d9740 1 --2- 172.21.15.1:0/3399604148 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8ae330 0x55de4b8ed600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.820+0000 7f23085d9740 1 --2- 172.21.15.1:0/3399604148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8adf50 0x55de4b8f7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.820+0000 7f23085d9740 1 -- 172.21.15.1:0/3399604148 >> 172.21.15.1:0/3399604148 conn(0x55de4b8ac4b0 msgr2=0x55de4b90e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.820+0000 7f23085d9740 1 -- 172.21.15.1:0/3399604148 shutdown_connections 2024-03-06T07:25:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.820+0000 7f23085d9740 1 -- 172.21.15.1:0/3399604148 wait complete. 2024-03-06T07:25:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.820+0000 7f23085d9740 1 -- start start 2024-03-06T07:25:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.820+0000 7f23085d9740 1 Processor -- start 2024-03-06T07:25:06.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.821+0000 7f23085d9740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8adf50 0x55de4b8eda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.821+0000 7f23085d9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8ae330 0x55de4b8a7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.821+0000 7f23085d9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8d0600 0x55de4b956fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.821+0000 7f23085d9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de4b8fe560 con 0x55de4b8d0600 2024-03-06T07:25:06.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.821+0000 7f23085d9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de4b8fdd60 con 0x55de4b8ae330 2024-03-06T07:25:06.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.821+0000 7f23085d9740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de4b8cf270 con 0x55de4b8adf50 2024-03-06T07:25:06.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.821+0000 7f23085d9740 10 client.0 _ll_register_callbacks cb 0x7f22f000f950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:06.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.821+0000 7f22ff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8d0600 0x55de4b956fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.821+0000 7f22ff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8d0600 0x55de4b956fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59580/0 (socket says 172.21.15.1:59580) 2024-03-06T07:25:06.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.821+0000 7f22ff7fe640 1 -- 172.21.15.1:0/3493117437 learned_addr learned my addr 172.21.15.1:0/3493117437 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:06.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.821+0000 7f22feffd640 1 --2- 172.21.15.1:0/3493117437 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8adf50 0x55de4b8eda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.821+0000 7f22fe7fc640 1 --2- 172.21.15.1:0/3493117437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8ae330 0x55de4b8a7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f22ff7fe640 1 -- 172.21.15.1:0/3493117437 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8adf50 msgr2=0x55de4b8eda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f22ff7fe640 1 --2- 172.21.15.1:0/3493117437 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8adf50 0x55de4b8eda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f22ff7fe640 1 -- 172.21.15.1:0/3493117437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8ae330 msgr2=0x55de4b8a7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f22ff7fe640 1 --2- 172.21.15.1:0/3493117437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8ae330 0x55de4b8a7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f22ff7fe640 1 -- 172.21.15.1:0/3493117437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de4b8ebf60 con 0x55de4b8d0600 2024-03-06T07:25:06.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f22ff7fe640 1 --2- 172.21.15.1:0/3493117437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8d0600 0x55de4b956fc0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f22f8091040 tx=0x7f22f8009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f93f44b16eaa37 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f22de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:25:06.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f22de7fc640 1 -- 172.21.15.1:0/3493117437 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f807f6f0 con 0x55de4b8d0600 2024-03-06T07:25:06.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f22de7fc640 1 -- 172.21.15.1:0/3493117437 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f22f8077c10 con 0x55de4b8d0600 2024-03-06T07:25:06.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f23085d9740 1 -- 172.21.15.1:0/3493117437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f22f8083cb0 con 0x55de4b8d0600 2024-03-06T07:25:06.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f22de7fc640 1 -- 172.21.15.1:0/3493117437 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22f808f1a0 con 0x55de4b8d0600 2024-03-06T07:25:06.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f23085d9740 10 client.8079 fetch_fsmap learned FSMap version 107 2024-03-06T07:25:06.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f23085d9740 1 -- 172.21.15.1:0/3493117437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) v3 -- 0x55de4b95d230 con 0x55de4b8d0600 2024-03-06T07:25:06.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f22de7fc640 1 -- 172.21.15.1:0/3493117437 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f8083cb0 con 0x55de4b8d0600 2024-03-06T07:25:06.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f22de7fc640 1 -- 172.21.15.1:0/3493117437 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 107) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f22f8083820 con 0x55de4b8d0600 2024-03-06T07:25:06.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f23085d9740 10 client.8079 fetch_fsmap finished waiting for FSMap version 107 2024-03-06T07:25:06.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f23085d9740 10 client.8079 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:25:06.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f23085d9740 20 client.8079 populate_metadata read hostname 'smithi001' 2024-03-06T07:25:06.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f23085d9740 1 --2- 172.21.15.1:0/3493117437 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x55de4b957500 0x55de4b8cb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f23085d9740 4 client.8079 mds_command: new command op to 7863 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:06.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f23085d9740 1 -- 172.21.15.1:0/3493117437 --> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55de4b819f40 con 0x55de4b957500 2024-03-06T07:25:06.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.822+0000 7f22feffd640 1 --2- 172.21.15.1:0/3493117437 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x55de4b957500 0x55de4b8cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.823+0000 7f22de7fc640 10 client.8079 ms_handle_connect on v2:172.21.15.1:6832/1515309674 2024-03-06T07:25:06.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.823+0000 7f22feffd640 1 --2- 172.21.15.1:0/3493117437 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x55de4b957500 0x55de4b8cb940 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a20bb48654c57424 server_cookie=874b02407dd7fe77 in_seq=0 out_seq=0 2024-03-06T07:25:06.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.823+0000 7f22de7fc640 1 -- 172.21.15.1:0/3493117437 <== mds.0 v2:172.21.15.1:6832/1515309674 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55de4b819f40 con 0x55de4b957500 2024-03-06T07:25:06.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.823+0000 7f22de7fc640 10 client.8079 handle_command_reply: tid=0 2024-03-06T07:25:06.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.823+0000 7f23085d9740 1 client.8079 shutdown 2024-03-06T07:25:06.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.823+0000 7f23085d9740 1 -- 172.21.15.1:0/3493117437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8d0600 msgr2=0x55de4b956fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.823+0000 7f23085d9740 1 --2- 172.21.15.1:0/3493117437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8d0600 0x55de4b956fc0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f22f8091040 tx=0x7f22f8009810 comp rx=0 tx=0).stop 2024-03-06T07:25:06.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.823+0000 7f23085d9740 1 -- 172.21.15.1:0/3493117437 shutdown_connections 2024-03-06T07:25:06.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.823+0000 7f23085d9740 1 --2- 172.21.15.1:0/3493117437 >> [v2:172.21.15.1:6832/1515309674,v1:172.21.15.1:6834/1515309674] conn(0x55de4b957500 0x55de4b8cb940 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.823+0000 7f23085d9740 1 --2- 172.21.15.1:0/3493117437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de4b8d0600 0x55de4b956fc0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.823+0000 7f23085d9740 1 --2- 172.21.15.1:0/3493117437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55de4b8ae330 0x55de4b8a7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.823+0000 7f23085d9740 1 --2- 172.21.15.1:0/3493117437 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55de4b8adf50 0x55de4b8eda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.823+0000 7f22de7fc640 0 client.8079 ms_handle_reset on v2:172.21.15.1:6832/1515309674 2024-03-06T07:25:06.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.823+0000 7f23085d9740 1 -- 172.21.15.1:0/3493117437 >> 172.21.15.1:0/3493117437 conn(0x55de4b8ac4b0 msgr2=0x55de4b90e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.823+0000 7f23085d9740 1 -- 172.21.15.1:0/3493117437 shutdown_connections 2024-03-06T07:25:06.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.823+0000 7f23085d9740 1 -- 172.21.15.1:0/3493117437 wait complete. 2024-03-06T07:25:06.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.823+0000 7f23085d9740 20 client.8079 trim_cache size 0 max 16384 2024-03-06T07:25:06.832 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:25:06.832 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:25:06.833 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:25:06.833 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 7863, 2024-03-06T07:25:06.833 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:25:06.833 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:25:06.833 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:25:06.833 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 13.577150757, 2024-03-06T07:25:06.833 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 107, 2024-03-06T07:25:06.833 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 121, 2024-03-06T07:25:06.833 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 121, 2024-03-06T07:25:06.833 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 18.668289876999999 2024-03-06T07:25:06.833 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:25:06.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.824+0000 7f23059f6640 1 -- 172.21.15.1:0/3356234172 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f22ec06b4d0 msgr2=0x7f22ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.824+0000 7f23059f6640 1 --2- 172.21.15.1:0/3356234172 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f22ec06b4d0 0x7f22ec06d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f22f000a990 tx=0x7f22f0008040 comp rx=0 tx=0).stop 2024-03-06T07:25:06.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.824+0000 7f23059f6640 1 -- 172.21.15.1:0/3356234172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23000c3e30 msgr2=0x7f23000c9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.824+0000 7f23059f6640 1 --2- 172.21.15.1:0/3356234172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23000c3e30 0x7f23000c9760 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009e50 tx=0x7f22f8004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:06.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.824+0000 7f23059f6640 1 -- 172.21.15.1:0/3356234172 shutdown_connections 2024-03-06T07:25:06.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.824+0000 7f23059f6640 1 --2- 172.21.15.1:0/3356234172 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f22ec06b4d0 0x7f22ec06d990 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.824+0000 7f23059f6640 1 --2- 172.21.15.1:0/3356234172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23000c3e30 0x7f23000c9760 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.824+0000 7f23059f6640 1 --2- 172.21.15.1:0/3356234172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2300152650 0x7f23000c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.824+0000 7f23059f6640 1 --2- 172.21.15.1:0/3356234172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23000bf9d0 0x7f23001ee780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.824+0000 7f23059f6640 1 -- 172.21.15.1:0/3356234172 >> 172.21.15.1:0/3356234172 conn(0x7f2300066ee0 msgr2=0x7f23000c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.824+0000 7f23059f6640 1 -- 172.21.15.1:0/3356234172 shutdown_connections 2024-03-06T07:25:06.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.824+0000 7f23059f6640 1 -- 172.21.15.1:0/3356234172 wait complete. 2024-03-06T07:25:06.836 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 107, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:24:52.598303+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 107, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7863": { "addr": "172.21.15.1:6834/1515309674", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 1515309674, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 1515309674, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7863, "incarnation": 102, "join_fscid": 9, "name": "c", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-03-06T07:25:01.273589+0000", "qdb_cluster": [ 7863 ], "qdb_leader": 7863, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7863 } } } ], "standbys": [ { "addr": "172.21.15.1:6835/4151599120", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 4151599120, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 4151599120, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6446, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6837/155600632", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 155600632, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 155600632, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6458, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6836/1420541659", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 1420541659, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 1420541659, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7084, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:25:06.837 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 9, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:25:06.838 INFO:tasks.cephfs.test_failover:current = [{'gid': 6446, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/4151599120', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 4151599120}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 4151599120}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}, {'gid': 7084, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/1420541659', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 1420541659}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1420541659}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}, {'gid': 7863, 'name': 'c', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.1:6834/1515309674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1515309674}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1515309674}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6458, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/155600632', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 155600632}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 155600632}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}] 2024-03-06T07:25:06.838 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:25:06.838 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 9:0 2024-03-06T07:25:06.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.939+0000 7fac3d6fb640 1 Processor -- start 2024-03-06T07:25:06.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.939+0000 7fac3d6fb640 1 -- start start 2024-03-06T07:25:06.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.940+0000 7fac3d6fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac38152690 0x7fac38155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.940+0000 7fac3d6fb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fac380c6730 0x7fac380c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.940+0000 7fac3d6fb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac381561e0 con 0x7fac380c5d10 2024-03-06T07:25:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.940+0000 7fac3d6fb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac380c3300 con 0x7fac380c6730 2024-03-06T07:25:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.940+0000 7fac3d6fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac380c35a0 con 0x7fac38152690 2024-03-06T07:25:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.940+0000 7fac377fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fac380c5d10 0x7fac38156cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56766/0 (socket says 172.21.15.1:56766) 2024-03-06T07:25:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.940+0000 7fac377fe640 1 -- 172.21.15.1:0/880035613 learned_addr learned my addr 172.21.15.1:0/880035613 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.940+0000 7fac367fc640 1 --2- 172.21.15.1:0/880035613 >> v2:172.21.15.98:3300/0 conn(0x7fac380c6730 0x7fac380c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.940+0000 7fac36ffd640 1 --2- 172.21.15.1:0/880035613 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac38152690 0x7fac38155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.940+0000 7fac35ffb640 1 -- 172.21.15.1:0/880035613 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568729637 0 0) 0x7fac30001490 con 0x7fac380c5d10 2024-03-06T07:25:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.940+0000 7fac35ffb640 1 -- 172.21.15.1:0/880035613 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac24003520 con 0x7fac380c5d10 2024-03-06T07:25:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.940+0000 7fac36ffd640 1 -- 172.21.15.1:0/880035613 >> v2:172.21.15.98:3300/0 conn(0x7fac380c6730 msgr2=0x7fac380c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.940+0000 7fac36ffd640 1 --2- 172.21.15.1:0/880035613 >> v2:172.21.15.98:3300/0 conn(0x7fac380c6730 0x7fac380c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.940+0000 7fac36ffd640 1 -- 172.21.15.1:0/880035613 >> v1:172.21.15.1:6789/0 conn(0x7fac380c5d10 legacy=0x7fac38156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.940+0000 7fac35ffb640 1 -- 172.21.15.1:0/880035613 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2845188010 0 0) 0x7fac24003520 con 0x7fac380c5d10 2024-03-06T07:25:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.940+0000 7fac36ffd640 1 -- 172.21.15.1:0/880035613 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac380c3880 con 0x7fac38152690 2024-03-06T07:25:06.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.942+0000 7fac36ffd640 1 --2- 172.21.15.1:0/880035613 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac38152690 0x7fac38155ca0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fac280097a0 tx=0x7fac2802b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a73a24f117b8b963 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.942+0000 7fac35ffb640 1 -- 172.21.15.1:0/880035613 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac28037070 con 0x7fac38152690 2024-03-06T07:25:06.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.942+0000 7fac35ffb640 1 -- 172.21.15.1:0/880035613 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac28030510 con 0x7fac38152690 2024-03-06T07:25:06.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.942+0000 7fac35ffb640 1 -- 172.21.15.1:0/880035613 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30001bc0 con 0x7fac38152690 2024-03-06T07:25:06.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.942+0000 7fac3d6fb640 1 -- 172.21.15.1:0/880035613 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac38152690 msgr2=0x7fac38155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.942+0000 7fac3d6fb640 1 --2- 172.21.15.1:0/880035613 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac38152690 0x7fac38155ca0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fac280097a0 tx=0x7fac2802b710 comp rx=0 tx=0).stop 2024-03-06T07:25:06.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.942+0000 7fac3d6fb640 1 -- 172.21.15.1:0/880035613 shutdown_connections 2024-03-06T07:25:06.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.942+0000 7fac3d6fb640 1 --2- 172.21.15.1:0/880035613 >> v2:172.21.15.98:3300/0 conn(0x7fac380c6730 0x7fac380c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.942+0000 7fac3d6fb640 1 --2- 172.21.15.1:0/880035613 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac38152690 0x7fac38155ca0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.942+0000 7fac3d6fb640 1 -- 172.21.15.1:0/880035613 >> 172.21.15.1:0/880035613 conn(0x7fac3806b410 msgr2=0x7fac3806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.942+0000 7fac3d6fb640 1 -- 172.21.15.1:0/880035613 shutdown_connections 2024-03-06T07:25:06.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.942+0000 7fac3d6fb640 1 -- 172.21.15.1:0/880035613 wait complete. 2024-03-06T07:25:06.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.944+0000 7fac3d6fb640 1 Processor -- start 2024-03-06T07:25:06.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.944+0000 7fac3d6fb640 1 -- start start 2024-03-06T07:25:06.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.944+0000 7fac3d6fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac380c5d10 0x7fac380c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.944+0000 7fac3d6fb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac380c6730 0x7fac381de880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.944+0000 7fac3d6fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac38152690 0x7fac381dedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.944+0000 7fac3d6fb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac38152a70 con 0x7fac380c5d10 2024-03-06T07:25:06.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.944+0000 7fac3d6fb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac381567b0 con 0x7fac380c6730 2024-03-06T07:25:06.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.944+0000 7fac3d6fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac38152bf0 con 0x7fac38152690 2024-03-06T07:25:06.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.944+0000 7fac36ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac380c5d10 0x7fac380c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.944+0000 7fac36ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac380c5d10 0x7fac380c8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59582/0 (socket says 172.21.15.1:59582) 2024-03-06T07:25:06.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.944+0000 7fac36ffd640 1 -- 172.21.15.1:0/2846476823 learned_addr learned my addr 172.21.15.1:0/2846476823 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:06.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.945+0000 7fac377fe640 1 --2- 172.21.15.1:0/2846476823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac38152690 0x7fac381dedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.945+0000 7fac367fc640 1 --2- 172.21.15.1:0/2846476823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac380c6730 0x7fac381de880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.945+0000 7fac36ffd640 1 -- 172.21.15.1:0/2846476823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac38152690 msgr2=0x7fac381dedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.945+0000 7fac36ffd640 1 --2- 172.21.15.1:0/2846476823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac38152690 0x7fac381dedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.945+0000 7fac36ffd640 1 -- 172.21.15.1:0/2846476823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac380c6730 msgr2=0x7fac381de880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.945+0000 7fac36ffd640 1 --2- 172.21.15.1:0/2846476823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac380c6730 0x7fac381de880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.945+0000 7fac36ffd640 1 -- 172.21.15.1:0/2846476823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac38151e40 con 0x7fac380c5d10 2024-03-06T07:25:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.945+0000 7fac36ffd640 1 --2- 172.21.15.1:0/2846476823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac380c5d10 0x7fac380c8110 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fac2802b750 tx=0x7fac280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.945+0000 7fac2f7fe640 1 -- 172.21.15.1:0/2846476823 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac28040020 con 0x7fac380c5d10 2024-03-06T07:25:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.945+0000 7fac2f7fe640 1 -- 172.21.15.1:0/2846476823 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac28038040 con 0x7fac380c5d10 2024-03-06T07:25:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.945+0000 7fac3d6fb640 1 -- 172.21.15.1:0/2846476823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac38152e90 con 0x7fac380c5d10 2024-03-06T07:25:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.945+0000 7fac2f7fe640 1 -- 172.21.15.1:0/2846476823 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac28037070 con 0x7fac380c5d10 2024-03-06T07:25:06.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.945+0000 7fac3d6fb640 1 -- 172.21.15.1:0/2846476823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac38153450 con 0x7fac380c5d10 2024-03-06T07:25:06.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.946+0000 7fac3d6fb640 1 -- 172.21.15.1:0/2846476823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac38067fd0 con 0x7fac380c5d10 2024-03-06T07:25:06.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.947+0000 7fac2f7fe640 1 -- 172.21.15.1:0/2846476823 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac28043020 con 0x7fac380c5d10 2024-03-06T07:25:06.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.947+0000 7fac2f7fe640 1 --2- 172.21.15.1:0/2846476823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac1806b4d0 0x7fac1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.947+0000 7fac2f7fe640 1 -- 172.21.15.1:0/2846476823 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fac280ab6d0 con 0x7fac380c5d10 2024-03-06T07:25:06.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.947+0000 7fac367fc640 1 --2- 172.21.15.1:0/2846476823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac1806b4d0 0x7fac1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.948+0000 7fac367fc640 1 --2- 172.21.15.1:0/2846476823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac1806b4d0 0x7fac1806d990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fac381eefc0 tx=0x7fac20009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:06.952+0000 7fac2f7fe640 1 -- 172.21.15.1:0/2846476823 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac2804a030 con 0x7fac380c5d10 2024-03-06T07:25:07.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.206+0000 7fac3d6fb640 1 -- 172.21.15.1:0/2846476823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "9:0"} v 0) v1 -- 0x7fac38071190 con 0x7fac380c5d10 2024-03-06T07:25:07.208 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:25:07.206+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:25:07.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.207+0000 7fac2f7fe640 1 -- 172.21.15.1:0/2846476823 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "9:0"}]=0 failed mds gid 7863 v108) v1 ==== 97+0+0 (secure 0 0 0) 0x7fac28079e00 con 0x7fac380c5d10 2024-03-06T07:25:07.209 INFO:teuthology.orchestra.run.smithi001.stderr:failed mds gid 7863 2024-03-06T07:25:07.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.210+0000 7fac3d6fb640 1 -- 172.21.15.1:0/2846476823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac1806b4d0 msgr2=0x7fac1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.210+0000 7fac3d6fb640 1 --2- 172.21.15.1:0/2846476823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac1806b4d0 0x7fac1806d990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fac381eefc0 tx=0x7fac20009190 comp rx=0 tx=0).stop 2024-03-06T07:25:07.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.210+0000 7fac3d6fb640 1 -- 172.21.15.1:0/2846476823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac380c5d10 msgr2=0x7fac380c8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.210+0000 7fac3d6fb640 1 --2- 172.21.15.1:0/2846476823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac380c5d10 0x7fac380c8110 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fac2802b750 tx=0x7fac280046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.210+0000 7fac3d6fb640 1 -- 172.21.15.1:0/2846476823 shutdown_connections 2024-03-06T07:25:07.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.210+0000 7fac3d6fb640 1 --2- 172.21.15.1:0/2846476823 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fac1806b4d0 0x7fac1806d990 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.210+0000 7fac3d6fb640 1 --2- 172.21.15.1:0/2846476823 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac38152690 0x7fac381dedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.210+0000 7fac3d6fb640 1 --2- 172.21.15.1:0/2846476823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac380c6730 0x7fac381de880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.211+0000 7fac3d6fb640 1 --2- 172.21.15.1:0/2846476823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac380c5d10 0x7fac380c8110 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.211+0000 7fac3d6fb640 1 -- 172.21.15.1:0/2846476823 >> 172.21.15.1:0/2846476823 conn(0x7fac3806b410 msgr2=0x7fac380a9e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.211+0000 7fac3d6fb640 1 -- 172.21.15.1:0/2846476823 shutdown_connections 2024-03-06T07:25:07.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.211+0000 7fac3d6fb640 1 -- 172.21.15.1:0/2846476823 wait complete. 2024-03-06T07:25:07.214 INFO:tasks.ceph.mds.c.smithi001.stderr: -387> 2024-03-06T07:24:58.528+0000 7f0046de1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:07.225 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:07.239 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:25:07.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.327+0000 7f10b93b6640 1 Processor -- start 2024-03-06T07:25:07.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.327+0000 7f10b93b6640 1 -- start start 2024-03-06T07:25:07.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.328+0000 7f10b93b6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f10b40c3260 0x7f10b40c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.328+0000 7f10b93b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10b4155b70 0x7f10b4159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.328+0000 7f10b93b6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b40c2b90 con 0x7f10b40c8cd0 2024-03-06T07:25:07.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.328+0000 7f10b93b6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b4156020 con 0x7f10b40c3260 2024-03-06T07:25:07.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.328+0000 7f10b93b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b40c37b0 con 0x7f10b4155b70 2024-03-06T07:25:07.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.328+0000 7f10b27fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f10b40c3260 0x7f10b40c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.328+0000 7f10b27fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f10b40c3260 0x7f10b40c90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59604/0 (socket says 172.21.15.1:59604) 2024-03-06T07:25:07.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.328+0000 7f10b27fc640 1 -- 172.21.15.1:0/580952227 learned_addr learned my addr 172.21.15.1:0/580952227 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:07.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.328+0000 7f10b37fe640 1 --2- 172.21.15.1:0/580952227 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10b4155b70 0x7f10b4159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.328+0000 7f10b27fc640 1 -- 172.21.15.1:0/580952227 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10b4155b70 msgr2=0x7f10b4159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.328+0000 7f10b27fc640 1 --2- 172.21.15.1:0/580952227 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10b4155b70 0x7f10b4159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.328+0000 7f10b27fc640 1 -- 172.21.15.1:0/580952227 >> v1:172.21.15.98:6789/0 conn(0x7f10b40c8cd0 legacy=0x7f10b40c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.328+0000 7f10b27fc640 1 -- 172.21.15.1:0/580952227 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b40c3a90 con 0x7f10b40c3260 2024-03-06T07:25:07.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.329+0000 7f10b27fc640 1 --2- 172.21.15.1:0/580952227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b40c3260 0x7f10b40c90b0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f109c009870 tx=0x7f109c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b2c09da03c61699 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.329+0000 7f10b1ffb640 1 -- 172.21.15.1:0/580952227 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c035020 con 0x7f10b40c3260 2024-03-06T07:25:07.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.329+0000 7f10b1ffb640 1 -- 172.21.15.1:0/580952227 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109c031640 con 0x7f10b40c3260 2024-03-06T07:25:07.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.329+0000 7f10b1ffb640 1 -- 172.21.15.1:0/580952227 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c02d040 con 0x7f10b40c3260 2024-03-06T07:25:07.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.330+0000 7f10b93b6640 1 -- 172.21.15.1:0/580952227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b40c3260 msgr2=0x7f10b40c90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.330+0000 7f10b93b6640 1 --2- 172.21.15.1:0/580952227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b40c3260 0x7f10b40c90b0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f109c009870 tx=0x7f109c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.330+0000 7f10b93b6640 1 -- 172.21.15.1:0/580952227 shutdown_connections 2024-03-06T07:25:07.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.330+0000 7f10b93b6640 1 --2- 172.21.15.1:0/580952227 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10b4155b70 0x7f10b4159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.330+0000 7f10b93b6640 1 --2- 172.21.15.1:0/580952227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b40c3260 0x7f10b40c90b0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.330+0000 7f10b93b6640 1 -- 172.21.15.1:0/580952227 >> 172.21.15.1:0/580952227 conn(0x7f10b406b3e0 msgr2=0x7f10b406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.330+0000 7f10b93b6640 1 -- 172.21.15.1:0/580952227 shutdown_connections 2024-03-06T07:25:07.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.330+0000 7f10b93b6640 1 -- 172.21.15.1:0/580952227 wait complete. 2024-03-06T07:25:07.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.331+0000 7f10b93b6640 1 Processor -- start 2024-03-06T07:25:07.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.331+0000 7f10b93b6640 1 -- start start 2024-03-06T07:25:07.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b93b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10b40c3260 0x7f10b4070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b93b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b40c8cd0 0x7f10b4154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b93b6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b4155b70 0x7f10b41ee360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b93b6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b4159550 con 0x7f10b4155b70 2024-03-06T07:25:07.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b93b6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b4158af0 con 0x7f10b40c8cd0 2024-03-06T07:25:07.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b93b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b40c8560 con 0x7f10b40c3260 2024-03-06T07:25:07.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b4155b70 0x7f10b41ee360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b4155b70 0x7f10b41ee360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59618/0 (socket says 172.21.15.1:59618) 2024-03-06T07:25:07.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b37fe640 1 -- 172.21.15.1:0/228071057 learned_addr learned my addr 172.21.15.1:0/228071057 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:07.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b2ffd640 1 --2- 172.21.15.1:0/228071057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10b40c3260 0x7f10b4070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b27fc640 1 --2- 172.21.15.1:0/228071057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b40c8cd0 0x7f10b4154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b37fe640 1 -- 172.21.15.1:0/228071057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10b40c3260 msgr2=0x7f10b4070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b37fe640 1 --2- 172.21.15.1:0/228071057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10b40c3260 0x7f10b4070ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b37fe640 1 -- 172.21.15.1:0/228071057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b40c8cd0 msgr2=0x7f10b4154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b37fe640 1 --2- 172.21.15.1:0/228071057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b40c8cd0 0x7f10b4154bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b37fe640 1 -- 172.21.15.1:0/228071057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b4071430 con 0x7f10b4155b70 2024-03-06T07:25:07.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b27fc640 1 --2- 172.21.15.1:0/228071057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b40c8cd0 0x7f10b4154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:07.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b37fe640 1 --2- 172.21.15.1:0/228071057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b4155b70 0x7f10b41ee360 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f10a4007e40 tx=0x7f10a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10ab7fe640 1 -- 172.21.15.1:0/228071057 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4014620 con 0x7f10b4155b70 2024-03-06T07:25:07.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10ab7fe640 1 -- 172.21.15.1:0/228071057 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a4019070 con 0x7f10b4155b70 2024-03-06T07:25:07.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b93b6640 1 -- 172.21.15.1:0/228071057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b40716d0 con 0x7f10b4155b70 2024-03-06T07:25:07.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10ab7fe640 1 -- 172.21.15.1:0/228071057 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4014a20 con 0x7f10b4155b70 2024-03-06T07:25:07.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.332+0000 7f10b93b6640 1 -- 172.21.15.1:0/228071057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b41eb420 con 0x7f10b4155b70 2024-03-06T07:25:07.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.333+0000 7f10b93b6640 1 -- 172.21.15.1:0/228071057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b4067fd0 con 0x7f10b4155b70 2024-03-06T07:25:07.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.334+0000 7f10ab7fe640 1 -- 172.21.15.1:0/228071057 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10a4014bc0 con 0x7f10b4155b70 2024-03-06T07:25:07.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.334+0000 7f10ab7fe640 1 --2- 172.21.15.1:0/228071057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f109406b4d0 0x7f109406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.334+0000 7f10ab7fe640 1 -- 172.21.15.1:0/228071057 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f10a408ac90 con 0x7f10b4155b70 2024-03-06T07:25:07.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.334+0000 7f10b2ffd640 1 --2- 172.21.15.1:0/228071057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f109406b4d0 0x7f109406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.335+0000 7f10b2ffd640 1 --2- 172.21.15.1:0/228071057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f109406b4d0 0x7f109406d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f10ac007fd0 tx=0x7f10ac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.339+0000 7f10ab7fe640 1 -- 172.21.15.1:0/228071057 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f10a40599c0 con 0x7f10b4155b70 2024-03-06T07:25:07.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.556+0000 7f10b93b6640 1 -- 172.21.15.1:0/228071057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10b40c2d80 con 0x7f10b4155b70 2024-03-06T07:25:07.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.557+0000 7f10ab7fe640 1 -- 172.21.15.1:0/228071057 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+4182 (secure 0 0 0) 0x7f10a40592c0 con 0x7f10b4155b70 2024-03-06T07:25:07.559 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:07.559 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":109,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6446,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4151599120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4151599120},{"type":"v1","addr":"172.21.15.1:6835","nonce":4151599120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/155600632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":155600632},{"type":"v1","addr":"172.21.15.98:6837","nonce":155600632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.598303+0000","modified":"2024-03-06T07:25:07.211867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7084":{"gid":7084,"name":"b","rank":0,"incarnation":109,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6836/1420541659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1420541659},{"type":"v1","addr":"172.21.15.98:6836","nonce":1420541659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":9}]} 2024-03-06T07:25:07.559 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 109 2024-03-06T07:25:07.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.559+0000 7f10b93b6640 1 -- 172.21.15.1:0/228071057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f109406b4d0 msgr2=0x7f109406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.559+0000 7f10b93b6640 1 --2- 172.21.15.1:0/228071057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f109406b4d0 0x7f109406d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f10ac007fd0 tx=0x7f10ac009040 comp rx=0 tx=0).stop 2024-03-06T07:25:07.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.559+0000 7f10b93b6640 1 -- 172.21.15.1:0/228071057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b4155b70 msgr2=0x7f10b41ee360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.559+0000 7f10b93b6640 1 --2- 172.21.15.1:0/228071057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b4155b70 0x7f10b41ee360 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f10a4007e40 tx=0x7f10a400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.559+0000 7f10b93b6640 1 -- 172.21.15.1:0/228071057 shutdown_connections 2024-03-06T07:25:07.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.559+0000 7f10b93b6640 1 --2- 172.21.15.1:0/228071057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f109406b4d0 0x7f109406d990 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.559+0000 7f10b93b6640 1 --2- 172.21.15.1:0/228071057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b4155b70 0x7f10b41ee360 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.559+0000 7f10b93b6640 1 --2- 172.21.15.1:0/228071057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b40c8cd0 0x7f10b4154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.559+0000 7f10b93b6640 1 --2- 172.21.15.1:0/228071057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10b40c3260 0x7f10b4070ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.559+0000 7f10b93b6640 1 -- 172.21.15.1:0/228071057 >> 172.21.15.1:0/228071057 conn(0x7f10b406b3e0 msgr2=0x7f10b4156d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.559+0000 7f10b93b6640 1 -- 172.21.15.1:0/228071057 shutdown_connections 2024-03-06T07:25:07.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:07.559+0000 7f10b93b6640 1 -- 172.21.15.1:0/228071057 wait complete. 2024-03-06T07:25:07.974 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 109, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:52.598303+0000', 'modified': '2024-03-06T07:25:07.211867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 122, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7084}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7084': {'gid': 7084, 'name': 'b', 'rank': 0, 'incarnation': 109, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.98:6836/1420541659', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 1420541659}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1420541659}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:07.974 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7084:up:replay 2024-03-06T07:25:08.975 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.087+0000 7fc6430b5640 1 Processor -- start 2024-03-06T07:25:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.087+0000 7fc6430b5640 1 -- start start 2024-03-06T07:25:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.088+0000 7fc6430b5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc63c0c3eb0 0x7fc63c0c7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.088+0000 7fc6430b5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc63c0c7940 0x7fc63c15fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.088+0000 7fc6430b5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc63c0c71c0 con 0x7fc63c06d060 2024-03-06T07:25:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.088+0000 7fc6430b5640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc63c0c59b0 con 0x7fc63c0c3eb0 2024-03-06T07:25:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.088+0000 7fc6430b5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc63c0c4c00 con 0x7fc63c0c7940 2024-03-06T07:25:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.088+0000 7fc640e51640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc63c0c3eb0 0x7fc63c0c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.088+0000 7fc640e51640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc63c0c3eb0 0x7fc63c0c7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59624/0 (socket says 172.21.15.1:59624) 2024-03-06T07:25:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.088+0000 7fc640e51640 1 -- 172.21.15.1:0/469370930 learned_addr learned my addr 172.21.15.1:0/469370930 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.088+0000 7fc641652640 1 --2- 172.21.15.1:0/469370930 >> v2:172.21.15.98:3300/0 conn(0x7fc63c0c7940 0x7fc63c15fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.089+0000 7fc640e51640 1 -- 172.21.15.1:0/469370930 >> v2:172.21.15.98:3300/0 conn(0x7fc63c0c7940 msgr2=0x7fc63c15fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.089+0000 7fc640e51640 1 --2- 172.21.15.1:0/469370930 >> v2:172.21.15.98:3300/0 conn(0x7fc63c0c7940 0x7fc63c15fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.089+0000 7fc640e51640 1 -- 172.21.15.1:0/469370930 >> v1:172.21.15.98:6789/0 conn(0x7fc63c06d060 legacy=0x7fc63c1558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.089+0000 7fc640e51640 1 -- 172.21.15.1:0/469370930 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63c0c4e50 con 0x7fc63c0c3eb0 2024-03-06T07:25:09.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.089+0000 7fc640e51640 1 --2- 172.21.15.1:0/469370930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c0c3eb0 0x7fc63c0c7400 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fc63c0c1cb0 tx=0x7fc63402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74ca3e3f408793d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.090+0000 7fc63b7fe640 1 -- 172.21.15.1:0/469370930 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63402c960 con 0x7fc63c0c3eb0 2024-03-06T07:25:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.090+0000 7fc63b7fe640 1 -- 172.21.15.1:0/469370930 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63402cb00 con 0x7fc63c0c3eb0 2024-03-06T07:25:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.090+0000 7fc63b7fe640 1 -- 172.21.15.1:0/469370930 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6340343f0 con 0x7fc63c0c3eb0 2024-03-06T07:25:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.090+0000 7fc6430b5640 1 -- 172.21.15.1:0/469370930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c0c3eb0 msgr2=0x7fc63c0c7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.090+0000 7fc6430b5640 1 --2- 172.21.15.1:0/469370930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c0c3eb0 0x7fc63c0c7400 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fc63c0c1cb0 tx=0x7fc63402bdc0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.090+0000 7fc6430b5640 1 -- 172.21.15.1:0/469370930 shutdown_connections 2024-03-06T07:25:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.090+0000 7fc6430b5640 1 --2- 172.21.15.1:0/469370930 >> v2:172.21.15.98:3300/0 conn(0x7fc63c0c7940 0x7fc63c15fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.090+0000 7fc6430b5640 1 --2- 172.21.15.1:0/469370930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c0c3eb0 0x7fc63c0c7400 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.090+0000 7fc6430b5640 1 -- 172.21.15.1:0/469370930 >> 172.21.15.1:0/469370930 conn(0x7fc63c066d00 msgr2=0x7fc63c067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.090+0000 7fc6430b5640 1 -- 172.21.15.1:0/469370930 shutdown_connections 2024-03-06T07:25:09.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.090+0000 7fc6430b5640 1 -- 172.21.15.1:0/469370930 wait complete. 2024-03-06T07:25:09.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.091+0000 7fc6430b5640 1 Processor -- start 2024-03-06T07:25:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.091+0000 7fc6430b5640 1 -- start start 2024-03-06T07:25:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.092+0000 7fc6430b5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63c06d060 0x7fc63c147530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.092+0000 7fc6430b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc63c0c3eb0 0x7fc63c0c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.092+0000 7fc6430b5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c0c7940 0x7fc63c157620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.092+0000 7fc6430b5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c0c4290 con 0x7fc63c0c7940 2024-03-06T07:25:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.092+0000 7fc6430b5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c155420 con 0x7fc63c0c3eb0 2024-03-06T07:25:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.092+0000 7fc6430b5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c1eecb0 con 0x7fc63c06d060 2024-03-06T07:25:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.092+0000 7fc641652640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c0c7940 0x7fc63c157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.092+0000 7fc641652640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c0c7940 0x7fc63c157620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59632/0 (socket says 172.21.15.1:59632) 2024-03-06T07:25:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.092+0000 7fc641652640 1 -- 172.21.15.1:0/554051770 learned_addr learned my addr 172.21.15.1:0/554051770 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.093+0000 7fc63bfff640 1 --2- 172.21.15.1:0/554051770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc63c0c3eb0 0x7fc63c0c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.093+0000 7fc640e51640 1 --2- 172.21.15.1:0/554051770 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63c06d060 0x7fc63c147530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.093+0000 7fc641652640 1 -- 172.21.15.1:0/554051770 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63c06d060 msgr2=0x7fc63c147530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.093+0000 7fc641652640 1 --2- 172.21.15.1:0/554051770 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63c06d060 0x7fc63c147530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.093+0000 7fc641652640 1 -- 172.21.15.1:0/554051770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc63c0c3eb0 msgr2=0x7fc63c0c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.093+0000 7fc641652640 1 --2- 172.21.15.1:0/554051770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc63c0c3eb0 0x7fc63c0c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.093+0000 7fc641652640 1 -- 172.21.15.1:0/554051770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63c147250 con 0x7fc63c0c7940 2024-03-06T07:25:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.093+0000 7fc640e51640 1 --2- 172.21.15.1:0/554051770 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63c06d060 0x7fc63c147530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.093+0000 7fc63bfff640 1 --2- 172.21.15.1:0/554051770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc63c0c3eb0 0x7fc63c0c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.093+0000 7fc641652640 1 --2- 172.21.15.1:0/554051770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c0c7940 0x7fc63c157620 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00d800 tx=0x7fc62c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.093+0000 7fc6397fa640 1 -- 172.21.15.1:0/554051770 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c00b6c0 con 0x7fc63c0c7940 2024-03-06T07:25:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.093+0000 7fc6397fa640 1 -- 172.21.15.1:0/554051770 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62c019070 con 0x7fc63c0c7940 2024-03-06T07:25:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.093+0000 7fc6397fa640 1 -- 172.21.15.1:0/554051770 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c00bac0 con 0x7fc63c0c7940 2024-03-06T07:25:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.093+0000 7fc6430b5640 1 -- 172.21.15.1:0/554051770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc63c1eef70 con 0x7fc63c0c7940 2024-03-06T07:25:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.093+0000 7fc6430b5640 1 -- 172.21.15.1:0/554051770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc63c0c9ff0 con 0x7fc63c0c7940 2024-03-06T07:25:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.094+0000 7fc6430b5640 1 -- 172.21.15.1:0/554051770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc63c0c3a10 con 0x7fc63c0c7940 2024-03-06T07:25:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.095+0000 7fc6397fa640 1 -- 172.21.15.1:0/554051770 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc62c00bc60 con 0x7fc63c0c7940 2024-03-06T07:25:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.095+0000 7fc6397fa640 1 --2- 172.21.15.1:0/554051770 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc62806b4d0 0x7fc62806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.095+0000 7fc6397fa640 1 -- 172.21.15.1:0/554051770 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc62c08a7e0 con 0x7fc63c0c7940 2024-03-06T07:25:09.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.095+0000 7fc640e51640 1 --2- 172.21.15.1:0/554051770 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc62806b4d0 0x7fc62806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.096+0000 7fc640e51640 1 --2- 172.21.15.1:0/554051770 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc62806b4d0 0x7fc62806d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fc634035000 tx=0x7fc634030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.100+0000 7fc6397fa640 1 -- 172.21.15.1:0/554051770 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc63c0c3a10 con 0x7fc63c0c7940 2024-03-06T07:25:09.303 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:25:09.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.327+0000 7fc6430b5640 1 -- 172.21.15.1:0/554051770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc63c154730 con 0x7fc63c0c7940 2024-03-06T07:25:09.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.327+0000 7fc6397fa640 1 -- 172.21.15.1:0/554051770 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4178 (secure 0 0 0) 0x7fc63c154730 con 0x7fc63c0c7940 2024-03-06T07:25:09.329 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:09.329 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":112,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6446,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4151599120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4151599120},{"type":"v1","addr":"172.21.15.1:6835","nonce":4151599120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/155600632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":155600632},{"type":"v1","addr":"172.21.15.98:6837","nonce":155600632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.598303+0000","modified":"2024-03-06T07:25:09.267117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8091":{"gid":8091,"name":"c","rank":0,"incarnation":112,"state":"up:replay","state_seq":1,"addr":"172.21.15.1:6834/1747678472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1747678472},{"type":"v1","addr":"172.21.15.1:6834","nonce":1747678472}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":9}]} 2024-03-06T07:25:09.330 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 112 2024-03-06T07:25:09.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.329+0000 7fc6430b5640 1 -- 172.21.15.1:0/554051770 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc62806b4d0 msgr2=0x7fc62806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.329+0000 7fc6430b5640 1 --2- 172.21.15.1:0/554051770 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc62806b4d0 0x7fc62806d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fc634035000 tx=0x7fc634030000 comp rx=0 tx=0).stop 2024-03-06T07:25:09.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.329+0000 7fc6430b5640 1 -- 172.21.15.1:0/554051770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c0c7940 msgr2=0x7fc63c157620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.329+0000 7fc6430b5640 1 --2- 172.21.15.1:0/554051770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c0c7940 0x7fc63c157620 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00d800 tx=0x7fc62c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.329+0000 7fc6430b5640 1 -- 172.21.15.1:0/554051770 shutdown_connections 2024-03-06T07:25:09.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.329+0000 7fc6430b5640 1 --2- 172.21.15.1:0/554051770 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc62806b4d0 0x7fc62806d990 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.330+0000 7fc6430b5640 1 --2- 172.21.15.1:0/554051770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63c0c7940 0x7fc63c157620 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.330+0000 7fc6430b5640 1 --2- 172.21.15.1:0/554051770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc63c0c3eb0 0x7fc63c0c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.330+0000 7fc6430b5640 1 --2- 172.21.15.1:0/554051770 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc63c06d060 0x7fc63c147530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.330+0000 7fc6430b5640 1 -- 172.21.15.1:0/554051770 >> 172.21.15.1:0/554051770 conn(0x7fc63c066d00 msgr2=0x7fc63c0c26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.330+0000 7fc6430b5640 1 -- 172.21.15.1:0/554051770 shutdown_connections 2024-03-06T07:25:09.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:09.330+0000 7fc6430b5640 1 -- 172.21.15.1:0/554051770 wait complete. 2024-03-06T07:25:09.744 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 112, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:52.598303+0000', 'modified': '2024-03-06T07:25:09.267117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 123, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8091}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8091': {'gid': 8091, 'name': 'c', 'rank': 0, 'incarnation': 112, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.1:6834/1747678472', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1747678472}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1747678472}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:09.744 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8091:up:replay 2024-03-06T07:25:10.745 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:10.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.857+0000 7f62b160e640 1 Processor -- start 2024-03-06T07:25:10.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.857+0000 7f62b160e640 1 -- start start 2024-03-06T07:25:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.858+0000 7f62b160e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f62ac0c2b30 0x7f62ac0724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.858+0000 7f62b160e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62ac152b70 0x7f62ac072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.858+0000 7f62b160e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62ac151dd0 con 0x7f62ac0a5fa0 2024-03-06T07:25:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.858+0000 7f62b160e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62ac1526e0 con 0x7f62ac0c2b30 2024-03-06T07:25:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.858+0000 7f62b160e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac156190 con 0x7f62ac152b70 2024-03-06T07:25:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.858+0000 7f62ab7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f62ac0a5fa0 0x7f62ac0c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56776/0 (socket says 172.21.15.1:56776) 2024-03-06T07:25:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.858+0000 7f62ab7fe640 1 -- 172.21.15.1:0/1778167251 learned_addr learned my addr 172.21.15.1:0/1778167251 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.858+0000 7f62aaffd640 1 --2- 172.21.15.1:0/1778167251 >> v2:172.21.15.98:3300/0 conn(0x7f62ac0c2b30 0x7f62ac0724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.858+0000 7f62aa7fc640 1 --2- 172.21.15.1:0/1778167251 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62ac152b70 0x7f62ac072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.858+0000 7f62a9ffb640 1 -- 172.21.15.1:0/1778167251 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262273704 0 0) 0x7f62ac151dd0 con 0x7f62ac0a5fa0 2024-03-06T07:25:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.858+0000 7f62a9ffb640 1 -- 172.21.15.1:0/1778167251 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6294003520 con 0x7f62ac0a5fa0 2024-03-06T07:25:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62a9ffb640 1 -- 172.21.15.1:0/1778167251 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2625969539 0 0) 0x7f6294003520 con 0x7f62ac0a5fa0 2024-03-06T07:25:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62a9ffb640 1 -- 172.21.15.1:0/1778167251 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62ac152b70 msgr2=0x7f62ac072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62a9ffb640 1 --2- 172.21.15.1:0/1778167251 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62ac152b70 0x7f62ac072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62a9ffb640 1 -- 172.21.15.1:0/1778167251 >> v2:172.21.15.98:3300/0 conn(0x7f62ac0c2b30 msgr2=0x7f62ac0724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62a9ffb640 1 --2- 172.21.15.1:0/1778167251 >> v2:172.21.15.98:3300/0 conn(0x7f62ac0c2b30 0x7f62ac0724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62a9ffb640 1 -- 172.21.15.1:0/1778167251 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac152090 con 0x7f62ac0a5fa0 2024-03-06T07:25:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62a9ffb640 1 -- 172.21.15.1:0/1778167251 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f62a4001670 con 0x7f62ac0a5fa0 2024-03-06T07:25:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62a9ffb640 1 -- 172.21.15.1:0/1778167251 >> v1:172.21.15.1:6789/0 conn(0x7f62ac0a5fa0 legacy=0x7f62ac0c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62a9ffb640 1 --2- 172.21.15.1:0/1778167251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62940039b0 0x7f6294003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62a9ffb640 1 --2- 172.21.15.1:0/1778167251 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62ac152b70 0x7f6294004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62a9ffb640 1 --2- 172.21.15.1:0/1778167251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6294004d40 0x7f62940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62a9ffb640 1 -- 172.21.15.1:0/1778167251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac1526e0 con 0x7f62940039b0 2024-03-06T07:25:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62a9ffb640 1 -- 172.21.15.1:0/1778167251 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac156190 con 0x7f6294004d40 2024-03-06T07:25:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62a9ffb640 1 -- 172.21.15.1:0/1778167251 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629400b9c0 con 0x7f62ac152b70 2024-03-06T07:25:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62aaffd640 1 --2- 172.21.15.1:0/1778167251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62940039b0 0x7f6294003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62aa7fc640 1 --2- 172.21.15.1:0/1778167251 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62ac152b70 0x7f6294004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62ab7fe640 1 --2- 172.21.15.1:0/1778167251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6294004d40 0x7f62940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62aaffd640 1 -- 172.21.15.1:0/1778167251 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62ac152b70 msgr2=0x7f6294004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62aaffd640 1 --2- 172.21.15.1:0/1778167251 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62ac152b70 0x7f6294004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62aaffd640 1 -- 172.21.15.1:0/1778167251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6294004d40 msgr2=0x7f62940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62aaffd640 1 --2- 172.21.15.1:0/1778167251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6294004d40 0x7f62940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62aaffd640 1 -- 172.21.15.1:0/1778167251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6294003520 con 0x7f62940039b0 2024-03-06T07:25:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.859+0000 7f62ab7fe640 1 --2- 172.21.15.1:0/1778167251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6294004d40 0x7f62940051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.860+0000 7f62aaffd640 1 --2- 172.21.15.1:0/1778167251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62940039b0 0x7f6294003dd0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f629c009fc0 tx=0x7f6294005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d18cc116198179cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.860+0000 7f62a9ffb640 1 -- 172.21.15.1:0/1778167251 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c039070 con 0x7f62940039b0 2024-03-06T07:25:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.860+0000 7f62a9ffb640 1 -- 172.21.15.1:0/1778167251 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629c02cc60 con 0x7f62940039b0 2024-03-06T07:25:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.860+0000 7f62a9ffb640 1 -- 172.21.15.1:0/1778167251 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c009c10 con 0x7f62940039b0 2024-03-06T07:25:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.860+0000 7f62b160e640 1 -- 172.21.15.1:0/1778167251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62940039b0 msgr2=0x7f6294003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.860+0000 7f62b160e640 1 --2- 172.21.15.1:0/1778167251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62940039b0 0x7f6294003dd0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f629c009fc0 tx=0x7f6294005830 comp rx=0 tx=0).stop 2024-03-06T07:25:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.860+0000 7f62ab7fe640 1 -- 172.21.15.1:0/1778167251 reap_dead start 2024-03-06T07:25:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.861+0000 7f62b160e640 1 -- 172.21.15.1:0/1778167251 shutdown_connections 2024-03-06T07:25:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.861+0000 7f62b160e640 1 -- 172.21.15.1:0/1778167251 >> 172.21.15.1:0/1778167251 conn(0x7f62ac0c6860 msgr2=0x7f62ac0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.861+0000 7f62b160e640 1 -- 172.21.15.1:0/1778167251 shutdown_connections 2024-03-06T07:25:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.861+0000 7f62b160e640 1 -- 172.21.15.1:0/1778167251 wait complete. 2024-03-06T07:25:10.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.862+0000 7f62b160e640 1 Processor -- start 2024-03-06T07:25:10.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.862+0000 7f62b160e640 1 -- start start 2024-03-06T07:25:10.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62b160e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62940039b0 0x7f62ac0bf960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62b160e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62ac0bdd20 0x7f62ac0bfea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62b160e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62ac1ec4c0 0x7f62ac1ec940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62b160e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac155f00 con 0x7f62ac0bdd20 2024-03-06T07:25:10.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62b160e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac155d80 con 0x7f62940039b0 2024-03-06T07:25:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62b160e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac072a00 con 0x7f62ac1ec4c0 2024-03-06T07:25:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62aa7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62ac0bdd20 0x7f62ac0bfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62aa7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62ac0bdd20 0x7f62ac0bfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59646/0 (socket says 172.21.15.1:59646) 2024-03-06T07:25:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62aa7fc640 1 -- 172.21.15.1:0/2308286344 learned_addr learned my addr 172.21.15.1:0/2308286344 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62aaffd640 1 --2- 172.21.15.1:0/2308286344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62940039b0 0x7f62ac0bf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62ab7fe640 1 --2- 172.21.15.1:0/2308286344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62ac1ec4c0 0x7f62ac1ec940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62aa7fc640 1 -- 172.21.15.1:0/2308286344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62ac1ec4c0 msgr2=0x7f62ac1ec940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62aa7fc640 1 --2- 172.21.15.1:0/2308286344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62ac1ec4c0 0x7f62ac1ec940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62aa7fc640 1 -- 172.21.15.1:0/2308286344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62940039b0 msgr2=0x7f62ac0bf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62aa7fc640 1 --2- 172.21.15.1:0/2308286344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62940039b0 0x7f62ac0bf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62aa7fc640 1 -- 172.21.15.1:0/2308286344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac0bb920 con 0x7f62ac0bdd20 2024-03-06T07:25:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62aa7fc640 1 --2- 172.21.15.1:0/2308286344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62ac0bdd20 0x7f62ac0bfea0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f62a0011a00 tx=0x7f62a0011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62937fe640 1 -- 172.21.15.1:0/2308286344 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a000fb50 con 0x7f62ac0bdd20 2024-03-06T07:25:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62937fe640 1 -- 172.21.15.1:0/2308286344 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a000fcf0 con 0x7f62ac0bdd20 2024-03-06T07:25:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62b160e640 1 -- 172.21.15.1:0/2308286344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ac0c03e0 con 0x7f62ac0bdd20 2024-03-06T07:25:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.863+0000 7f62937fe640 1 -- 172.21.15.1:0/2308286344 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a000fea0 con 0x7f62ac0bdd20 2024-03-06T07:25:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.864+0000 7f62b160e640 1 -- 172.21.15.1:0/2308286344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ac14c3a0 con 0x7f62ac0bdd20 2024-03-06T07:25:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.865+0000 7f62b160e640 1 -- 172.21.15.1:0/2308286344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ac067fd0 con 0x7f62ac0bdd20 2024-03-06T07:25:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.865+0000 7f62937fe640 1 -- 172.21.15.1:0/2308286344 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f62a0012040 con 0x7f62ac0bdd20 2024-03-06T07:25:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.865+0000 7f62937fe640 1 --2- 172.21.15.1:0/2308286344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f629806b500 0x7f629806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.866+0000 7f62937fe640 1 -- 172.21.15.1:0/2308286344 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f62a008aad0 con 0x7f62ac0bdd20 2024-03-06T07:25:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.866+0000 7f62aaffd640 1 --2- 172.21.15.1:0/2308286344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f629806b500 0x7f629806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.866+0000 7f62aaffd640 1 --2- 172.21.15.1:0/2308286344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f629806b500 0x7f629806d9c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f629c036000 tx=0x7f629c02e000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:10.870+0000 7f62937fe640 1 -- 172.21.15.1:0/2308286344 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f62a0019030 con 0x7f62ac0bdd20 2024-03-06T07:25:11.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:11.090+0000 7f62b160e640 1 -- 172.21.15.1:0/2308286344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62ac071790 con 0x7f62ac0bdd20 2024-03-06T07:25:11.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:11.090+0000 7f62937fe640 1 -- 172.21.15.1:0/2308286344 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+4973 (secure 0 0 0) 0x7f62a00597a0 con 0x7f62ac0bdd20 2024-03-06T07:25:11.092 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:11.092 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":113,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6446,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4151599120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4151599120},{"type":"v1","addr":"172.21.15.1:6835","nonce":4151599120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/155600632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":155600632},{"type":"v1","addr":"172.21.15.98:6837","nonce":155600632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6587,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1600553134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1600553134},{"type":"v1","addr":"172.21.15.98:6836","nonce":1600553134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.598303+0000","modified":"2024-03-06T07:25:09.282652+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8091":{"gid":8091,"name":"c","rank":0,"incarnation":112,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.1:6834/1747678472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1747678472},{"type":"v1","addr":"172.21.15.1:6834","nonce":1747678472}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":9}]} 2024-03-06T07:25:11.093 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 113 2024-03-06T07:25:11.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:11.092+0000 7f62b160e640 1 -- 172.21.15.1:0/2308286344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f629806b500 msgr2=0x7f629806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:11.092+0000 7f62b160e640 1 --2- 172.21.15.1:0/2308286344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f629806b500 0x7f629806d9c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f629c036000 tx=0x7f629c02e000 comp rx=0 tx=0).stop 2024-03-06T07:25:11.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:11.092+0000 7f62b160e640 1 -- 172.21.15.1:0/2308286344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62ac0bdd20 msgr2=0x7f62ac0bfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:11.092+0000 7f62b160e640 1 --2- 172.21.15.1:0/2308286344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62ac0bdd20 0x7f62ac0bfea0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f62a0011a00 tx=0x7f62a0011ec0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:11.092+0000 7f62b160e640 1 -- 172.21.15.1:0/2308286344 shutdown_connections 2024-03-06T07:25:11.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:11.092+0000 7f62b160e640 1 --2- 172.21.15.1:0/2308286344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f629806b500 0x7f629806d9c0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:11.092+0000 7f62b160e640 1 --2- 172.21.15.1:0/2308286344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62ac1ec4c0 0x7f62ac1ec940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:11.092+0000 7f62b160e640 1 --2- 172.21.15.1:0/2308286344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62ac0bdd20 0x7f62ac0bfea0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:11.092+0000 7f62b160e640 1 --2- 172.21.15.1:0/2308286344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62940039b0 0x7f62ac0bf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:11.092+0000 7f62b160e640 1 -- 172.21.15.1:0/2308286344 >> 172.21.15.1:0/2308286344 conn(0x7f62ac0c6860 msgr2=0x7f62ac14e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:11.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:11.092+0000 7f62b160e640 1 -- 172.21.15.1:0/2308286344 shutdown_connections 2024-03-06T07:25:11.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:11.092+0000 7f62b160e640 1 -- 172.21.15.1:0/2308286344 wait complete. 2024-03-06T07:25:11.507 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 113, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:52.598303+0000', 'modified': '2024-03-06T07:25:09.282652+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 123, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8091}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8091': {'gid': 8091, 'name': 'c', 'rank': 0, 'incarnation': 112, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.1:6834/1747678472', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1747678472}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1747678472}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:11.508 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8091:up:reconnect 2024-03-06T07:25:12.509 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:12.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.609+0000 7f43e9135640 1 Processor -- start 2024-03-06T07:25:12.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.609+0000 7f43e9135640 1 -- start start 2024-03-06T07:25:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.610+0000 7f43e9135640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43e4152650 0x7f43e4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.610+0000 7f43e9135640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f43e40bf9d0 0x7f43e40c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.610+0000 7f43e9135640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e40c2fc0 con 0x7f43e40c6b10 2024-03-06T07:25:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.610+0000 7f43e9135640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e4155800 con 0x7f43e40bf9d0 2024-03-06T07:25:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.610+0000 7f43e9135640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e4158da0 con 0x7f43e4152650 2024-03-06T07:25:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.610+0000 7f43e359e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f43e40bf9d0 0x7f43e40c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.610+0000 7f43e259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f43e40c6b10 0x7f43e4158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56780/0 (socket says 172.21.15.1:56780) 2024-03-06T07:25:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.610+0000 7f43e259c640 1 -- 172.21.15.1:0/256377023 learned_addr learned my addr 172.21.15.1:0/256377023 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.610+0000 7f43e2d9d640 1 --2- 172.21.15.1:0/256377023 >> v2:172.21.15.98:3300/0 conn(0x7f43e4152650 0x7f43e4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.610+0000 7f43e359e640 1 -- 172.21.15.1:0/256377023 >> v2:172.21.15.98:3300/0 conn(0x7f43e4152650 msgr2=0x7f43e4155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.610+0000 7f43e359e640 1 --2- 172.21.15.1:0/256377023 >> v2:172.21.15.98:3300/0 conn(0x7f43e4152650 0x7f43e4155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.610+0000 7f43e359e640 1 -- 172.21.15.1:0/256377023 >> v1:172.21.15.1:6789/0 conn(0x7f43e40c6b10 legacy=0x7f43e4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.610+0000 7f43e359e640 1 -- 172.21.15.1:0/256377023 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e40c6690 con 0x7f43e40bf9d0 2024-03-06T07:25:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.610+0000 7f43e1d9b640 1 -- 172.21.15.1:0/256377023 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984627323 0 0) 0x7f43e40c2fc0 con 0x7f43e40c6b10 2024-03-06T07:25:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.610+0000 7f43e2d9d640 1 --2- 172.21.15.1:0/256377023 >> v2:172.21.15.98:3300/0 conn(0x7f43e4152650 0x7f43e4155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.611+0000 7f43e359e640 1 --2- 172.21.15.1:0/256377023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e40bf9d0 0x7f43e40c1460 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f43e40cac90 tx=0x7f43d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4398fe8ed4b4ef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.611+0000 7f43e1d9b640 1 -- 172.21.15.1:0/256377023 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d40315a0 con 0x7f43e40bf9d0 2024-03-06T07:25:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.611+0000 7f43e1d9b640 1 -- 172.21.15.1:0/256377023 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d4037070 con 0x7f43e40bf9d0 2024-03-06T07:25:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.611+0000 7f43e1d9b640 1 -- 172.21.15.1:0/256377023 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d4031960 con 0x7f43e40bf9d0 2024-03-06T07:25:12.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.612+0000 7f43e9135640 1 -- 172.21.15.1:0/256377023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e40bf9d0 msgr2=0x7f43e40c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.612+0000 7f43e9135640 1 --2- 172.21.15.1:0/256377023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e40bf9d0 0x7f43e40c1460 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f43e40cac90 tx=0x7f43d402b840 comp rx=0 tx=0).stop 2024-03-06T07:25:12.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.612+0000 7f43e9135640 1 -- 172.21.15.1:0/256377023 shutdown_connections 2024-03-06T07:25:12.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.612+0000 7f43e9135640 1 --2- 172.21.15.1:0/256377023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e40bf9d0 0x7f43e40c1460 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.612+0000 7f43e9135640 1 --2- 172.21.15.1:0/256377023 >> v2:172.21.15.98:3300/0 conn(0x7f43e4152650 0x7f43e4155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.612+0000 7f43e9135640 1 -- 172.21.15.1:0/256377023 >> 172.21.15.1:0/256377023 conn(0x7f43e4066ee0 msgr2=0x7f43e40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:12.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.612+0000 7f43e9135640 1 -- 172.21.15.1:0/256377023 shutdown_connections 2024-03-06T07:25:12.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.612+0000 7f43e9135640 1 -- 172.21.15.1:0/256377023 wait complete. 2024-03-06T07:25:12.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.613+0000 7f43e9135640 1 Processor -- start 2024-03-06T07:25:12.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.613+0000 7f43e9135640 1 -- start start 2024-03-06T07:25:12.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e9135640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43e40bf9d0 0x7f43e4156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e9135640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e4152650 0x7f43e41eb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e9135640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43e4154cc0 0x7f43e40c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e9135640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e41551b0 con 0x7f43e4152650 2024-03-06T07:25:12.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e9135640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e40c6fa0 con 0x7f43e40bf9d0 2024-03-06T07:25:12.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e9135640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e4155490 con 0x7f43e4154cc0 2024-03-06T07:25:12.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e4152650 0x7f43e41eb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e4152650 0x7f43e41eb5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59678/0 (socket says 172.21.15.1:59678) 2024-03-06T07:25:12.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e259c640 1 -- 172.21.15.1:0/3293566709 learned_addr learned my addr 172.21.15.1:0/3293566709 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:12.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e359e640 1 --2- 172.21.15.1:0/3293566709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43e4154cc0 0x7f43e40c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e2d9d640 1 --2- 172.21.15.1:0/3293566709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43e40bf9d0 0x7f43e4156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e259c640 1 -- 172.21.15.1:0/3293566709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43e4154cc0 msgr2=0x7f43e40c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e259c640 1 --2- 172.21.15.1:0/3293566709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43e4154cc0 0x7f43e40c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e259c640 1 -- 172.21.15.1:0/3293566709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43e40bf9d0 msgr2=0x7f43e4156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e259c640 1 --2- 172.21.15.1:0/3293566709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43e40bf9d0 0x7f43e4156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e259c640 1 -- 172.21.15.1:0/3293566709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e4154080 con 0x7f43e4152650 2024-03-06T07:25:12.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e2d9d640 1 --2- 172.21.15.1:0/3293566709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43e40bf9d0 0x7f43e4156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:12.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.614+0000 7f43e259c640 1 --2- 172.21.15.1:0/3293566709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e4152650 0x7f43e41eb5c0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00c4d0 tx=0x7f43cc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:12.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.615+0000 7f43db7fe640 1 -- 172.21.15.1:0/3293566709 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc018070 con 0x7f43e4152650 2024-03-06T07:25:12.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.615+0000 7f43db7fe640 1 -- 172.21.15.1:0/3293566709 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43cc007500 con 0x7f43e4152650 2024-03-06T07:25:12.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.615+0000 7f43e9135640 1 -- 172.21.15.1:0/3293566709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e4154360 con 0x7f43e4152650 2024-03-06T07:25:12.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.615+0000 7f43db7fe640 1 -- 172.21.15.1:0/3293566709 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc004a20 con 0x7f43e4152650 2024-03-06T07:25:12.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.615+0000 7f43e9135640 1 -- 172.21.15.1:0/3293566709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e41530a0 con 0x7f43e4152650 2024-03-06T07:25:12.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.616+0000 7f43e9135640 1 -- 172.21.15.1:0/3293566709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e41532a0 con 0x7f43e4152650 2024-03-06T07:25:12.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.616+0000 7f43db7fe640 1 -- 172.21.15.1:0/3293566709 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43cc004bc0 con 0x7f43e4152650 2024-03-06T07:25:12.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.617+0000 7f43db7fe640 1 --2- 172.21.15.1:0/3293566709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43c406b4d0 0x7f43c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.617+0000 7f43db7fe640 1 -- 172.21.15.1:0/3293566709 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f43cc0897b0 con 0x7f43e4152650 2024-03-06T07:25:12.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.617+0000 7f43e2d9d640 1 --2- 172.21.15.1:0/3293566709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43c406b4d0 0x7f43c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.617+0000 7f43e2d9d640 1 --2- 172.21.15.1:0/3293566709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43c406b4d0 0x7f43c406d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f43dc009540 tx=0x7f43dc00b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:12.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.621+0000 7f43db7fe640 1 -- 172.21.15.1:0/3293566709 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43e41532a0 con 0x7f43e4152650 2024-03-06T07:25:12.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.891+0000 7f43e9135640 1 -- 172.21.15.1:0/3293566709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43e4067fd0 con 0x7f43e4152650 2024-03-06T07:25:12.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.892+0000 7f43db7fe640 1 -- 172.21.15.1:0/3293566709 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+4977 (secure 0 0 0) 0x7f43cc058400 con 0x7f43e4152650 2024-03-06T07:25:12.894 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:12.894 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":115,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6446,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4151599120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4151599120},{"type":"v1","addr":"172.21.15.1:6835","nonce":4151599120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/155600632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":155600632},{"type":"v1","addr":"172.21.15.98:6837","nonce":155600632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6587,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1600553134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1600553134},{"type":"v1","addr":"172.21.15.98:6836","nonce":1600553134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.598303+0000","modified":"2024-03-06T07:25:12.276319+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8091":{"gid":8091,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":4,"addr":"172.21.15.1:6834/1747678472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1747678472},{"type":"v1","addr":"172.21.15.1:6834","nonce":1747678472}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8091,"qdb_cluster":[8091]},"id":9}]} 2024-03-06T07:25:12.894 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 115 2024-03-06T07:25:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.894+0000 7f43e9135640 1 -- 172.21.15.1:0/3293566709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43c406b4d0 msgr2=0x7f43c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.894+0000 7f43e9135640 1 --2- 172.21.15.1:0/3293566709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43c406b4d0 0x7f43c406d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f43dc009540 tx=0x7f43dc00b040 comp rx=0 tx=0).stop 2024-03-06T07:25:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.894+0000 7f43e9135640 1 -- 172.21.15.1:0/3293566709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e4152650 msgr2=0x7f43e41eb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.894+0000 7f43e9135640 1 --2- 172.21.15.1:0/3293566709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e4152650 0x7f43e41eb5c0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00c4d0 tx=0x7f43cc00c990 comp rx=0 tx=0).stop 2024-03-06T07:25:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.894+0000 7f43e9135640 1 -- 172.21.15.1:0/3293566709 shutdown_connections 2024-03-06T07:25:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.894+0000 7f43e9135640 1 --2- 172.21.15.1:0/3293566709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43c406b4d0 0x7f43c406d990 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.894+0000 7f43e9135640 1 --2- 172.21.15.1:0/3293566709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43e4154cc0 0x7f43e40c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.894+0000 7f43e9135640 1 --2- 172.21.15.1:0/3293566709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e4152650 0x7f43e41eb5c0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.894+0000 7f43e9135640 1 --2- 172.21.15.1:0/3293566709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43e40bf9d0 0x7f43e4156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.894+0000 7f43e9135640 1 -- 172.21.15.1:0/3293566709 >> 172.21.15.1:0/3293566709 conn(0x7f43e4066ee0 msgr2=0x7f43e40c5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:12.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.894+0000 7f43e9135640 1 -- 172.21.15.1:0/3293566709 shutdown_connections 2024-03-06T07:25:12.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:12.894+0000 7f43e9135640 1 -- 172.21.15.1:0/3293566709 wait complete. 2024-03-06T07:25:13.224 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-06T07:25:13.222+0000 7f9ada22e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:13.309 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:52.598303+0000', 'modified': '2024-03-06T07:25:12.276319+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 123, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8091}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8091': {'gid': 8091, 'name': 'c', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.1:6834/1747678472', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1747678472}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1747678472}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8091, 'qdb_cluster': [8091]} 2024-03-06T07:25:13.310 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:13.310 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:25:13.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.410+0000 7f336a855640 1 Processor -- start 2024-03-06T07:25:13.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.410+0000 7f336a855640 1 -- start start 2024-03-06T07:25:13.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.411+0000 7f336a855640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f33640c3ba0 0x7f3364156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.411+0000 7f336a855640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3364157130 con 0x7f33640a7280 2024-03-06T07:25:13.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.411+0000 7f336a855640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3364152fb0 con 0x7f33640c4ef0 2024-03-06T07:25:13.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.411+0000 7f336a855640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3364153290 con 0x7f33640c3ba0 2024-03-06T07:25:13.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.411+0000 7f3363fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f33640a7280 0x7f3364072d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56786/0 (socket says 172.21.15.1:56786) 2024-03-06T07:25:13.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.411+0000 7f3363fff640 1 -- 172.21.15.1:0/2787352769 learned_addr learned my addr 172.21.15.1:0/2787352769 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:13.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.411+0000 7f3362ffd640 1 -- 172.21.15.1:0/2787352769 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611219521 0 0) 0x7f3364157130 con 0x7f33640a7280 2024-03-06T07:25:13.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3362ffd640 1 -- 172.21.15.1:0/2787352769 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f334c003520 con 0x7f33640a7280 2024-03-06T07:25:13.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3368df2640 1 --2- 172.21.15.1:0/2787352769 >> v2:172.21.15.98:3300/0 conn(0x7f33640c3ba0 0x7f3364156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3362ffd640 1 -- 172.21.15.1:0/2787352769 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3510173887 0 0) 0x7f334c003520 con 0x7f33640a7280 2024-03-06T07:25:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3362ffd640 1 -- 172.21.15.1:0/2787352769 >> v2:172.21.15.98:3300/0 conn(0x7f33640c3ba0 msgr2=0x7f3364156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3362ffd640 1 --2- 172.21.15.1:0/2787352769 >> v2:172.21.15.98:3300/0 conn(0x7f33640c3ba0 0x7f3364156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3362ffd640 1 -- 172.21.15.1:0/2787352769 >> v1:172.21.15.98:6789/0 conn(0x7f33640c4ef0 legacy=0x7f33641564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3362ffd640 1 -- 172.21.15.1:0/2787352769 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3364153570 con 0x7f33640a7280 2024-03-06T07:25:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3362ffd640 1 -- 172.21.15.1:0/2787352769 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f3358001670 con 0x7f33640a7280 2024-03-06T07:25:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3362ffd640 1 -- 172.21.15.1:0/2787352769 >> v1:172.21.15.1:6789/0 conn(0x7f33640a7280 legacy=0x7f3364072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3362ffd640 1 --2- 172.21.15.1:0/2787352769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0039b0 0x7f334c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3362ffd640 1 --2- 172.21.15.1:0/2787352769 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f334c004800 0x7f334c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3362ffd640 1 --2- 172.21.15.1:0/2787352769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f334c0051c0 0x7f334c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3362ffd640 1 -- 172.21.15.1:0/2787352769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3364153290 con 0x7f334c0039b0 2024-03-06T07:25:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3362ffd640 1 -- 172.21.15.1:0/2787352769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c00c160 con 0x7f334c0051c0 2024-03-06T07:25:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3362ffd640 1 -- 172.21.15.1:0/2787352769 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334c00c360 con 0x7f334c004800 2024-03-06T07:25:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3363fff640 1 --2- 172.21.15.1:0/2787352769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0039b0 0x7f334c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3368df2640 1 --2- 172.21.15.1:0/2787352769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f334c0051c0 0x7f334c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f33637fe640 1 --2- 172.21.15.1:0/2787352769 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f334c004800 0x7f334c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3363fff640 1 -- 172.21.15.1:0/2787352769 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f334c004800 msgr2=0x7f334c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3363fff640 1 --2- 172.21.15.1:0/2787352769 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f334c004800 0x7f334c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3363fff640 1 -- 172.21.15.1:0/2787352769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f334c0051c0 msgr2=0x7f334c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3363fff640 1 --2- 172.21.15.1:0/2787352769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f334c0051c0 0x7f334c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3363fff640 1 -- 172.21.15.1:0/2787352769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c003520 con 0x7f334c0039b0 2024-03-06T07:25:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.412+0000 7f3368df2640 1 -- 172.21.15.1:0/2787352769 reap_dead start 2024-03-06T07:25:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.413+0000 7f3363fff640 1 --2- 172.21.15.1:0/2787352769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0039b0 0x7f334c003dd0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f335800c240 tx=0x7f334c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3466eb4750265238 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.413+0000 7f3362ffd640 1 -- 172.21.15.1:0/2787352769 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358007cf0 con 0x7f334c0039b0 2024-03-06T07:25:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.413+0000 7f3362ffd640 1 -- 172.21.15.1:0/2787352769 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3358030070 con 0x7f334c0039b0 2024-03-06T07:25:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.413+0000 7f3362ffd640 1 -- 172.21.15.1:0/2787352769 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358005d10 con 0x7f334c0039b0 2024-03-06T07:25:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.414+0000 7f336a855640 1 -- 172.21.15.1:0/2787352769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0039b0 msgr2=0x7f334c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.414+0000 7f336a855640 1 --2- 172.21.15.1:0/2787352769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0039b0 0x7f334c003dd0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f335800c240 tx=0x7f334c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.414+0000 7f336a855640 1 -- 172.21.15.1:0/2787352769 shutdown_connections 2024-03-06T07:25:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.414+0000 7f336a855640 1 --2- 172.21.15.1:0/2787352769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0039b0 0x7f334c003dd0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.414+0000 7f336a855640 1 -- 172.21.15.1:0/2787352769 >> 172.21.15.1:0/2787352769 conn(0x7f33640c7a50 msgr2=0x7f33640c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.414+0000 7f336a855640 1 -- 172.21.15.1:0/2787352769 shutdown_connections 2024-03-06T07:25:13.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.414+0000 7f336a855640 1 -- 172.21.15.1:0/2787352769 wait complete. 2024-03-06T07:25:13.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.415+0000 7f336a855640 1 Processor -- start 2024-03-06T07:25:13.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.415+0000 7f336a855640 1 -- start start 2024-03-06T07:25:13.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f336a855640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0039b0 0x7f3364152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f336a855640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3364070600 0x7f33640ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f336a855640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3364154230 0x7f33640ac6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f336a855640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33640c24e0 con 0x7f334c0039b0 2024-03-06T07:25:13.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f336a855640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3364153b30 con 0x7f3364070600 2024-03-06T07:25:13.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f336a855640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33640c2660 con 0x7f3364154230 2024-03-06T07:25:13.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f3363fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0039b0 0x7f3364152100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f3363fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0039b0 0x7f3364152100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59702/0 (socket says 172.21.15.1:59702) 2024-03-06T07:25:13.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f3363fff640 1 -- 172.21.15.1:0/3345416803 learned_addr learned my addr 172.21.15.1:0/3345416803 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:13.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f3368df2640 1 --2- 172.21.15.1:0/3345416803 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3364154230 0x7f33640ac6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f33637fe640 1 --2- 172.21.15.1:0/3345416803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3364070600 0x7f33640ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f3363fff640 1 -- 172.21.15.1:0/3345416803 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3364154230 msgr2=0x7f33640ac6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f3363fff640 1 --2- 172.21.15.1:0/3345416803 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3364154230 0x7f33640ac6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f3363fff640 1 -- 172.21.15.1:0/3345416803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3364070600 msgr2=0x7f33640ac1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f3363fff640 1 --2- 172.21.15.1:0/3345416803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3364070600 0x7f33640ac1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f3363fff640 1 -- 172.21.15.1:0/3345416803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336414f580 con 0x7f334c0039b0 2024-03-06T07:25:13.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f3368df2640 1 --2- 172.21.15.1:0/3345416803 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3364154230 0x7f33640ac6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:13.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.416+0000 7f3363fff640 1 --2- 172.21.15.1:0/3345416803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0039b0 0x7f3364152100 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f335800c360 tx=0x7f3364152ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.417+0000 7f3360ff9640 1 -- 172.21.15.1:0/3345416803 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335802de60 con 0x7f334c0039b0 2024-03-06T07:25:13.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.417+0000 7f3360ff9640 1 -- 172.21.15.1:0/3345416803 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33580049f0 con 0x7f334c0039b0 2024-03-06T07:25:13.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.417+0000 7f3360ff9640 1 -- 172.21.15.1:0/3345416803 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358004da0 con 0x7f334c0039b0 2024-03-06T07:25:13.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.417+0000 7f336a855640 1 -- 172.21.15.1:0/3345416803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33640aa910 con 0x7f334c0039b0 2024-03-06T07:25:13.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.417+0000 7f336a855640 1 -- 172.21.15.1:0/3345416803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3364152640 con 0x7f334c0039b0 2024-03-06T07:25:13.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.419+0000 7f3360ff9640 1 -- 172.21.15.1:0/3345416803 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3358007070 con 0x7f334c0039b0 2024-03-06T07:25:13.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.419+0000 7f3360ff9640 1 --2- 172.21.15.1:0/3345416803 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f335006b4d0 0x7f335006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.419+0000 7f3360ff9640 1 -- 172.21.15.1:0/3345416803 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f33580ab960 con 0x7f334c0039b0 2024-03-06T07:25:13.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.419+0000 7f33637fe640 1 --2- 172.21.15.1:0/3345416803 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f335006b4d0 0x7f335006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.420+0000 7f336d43f740 1 Processor -- start 2024-03-06T07:25:13.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.420+0000 7f336d43f740 1 -- start start 2024-03-06T07:25:13.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.420+0000 7f33637fe640 1 --2- 172.21.15.1:0/3345416803 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f335006b4d0 0x7f335006d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3354000c00 tx=0x7f3354009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.420+0000 7f336d43f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bcf50 0x5583a75baa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.420+0000 7f336d43f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bd330 0x5583a75baf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.420+0000 7f336d43f740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75bb4b0 0x5583a75b7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.420+0000 7f336d43f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583a75216b0 con 0x5583a75bd330 2024-03-06T07:25:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.420+0000 7f336d43f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583a74caa60 con 0x5583a75bcf50 2024-03-06T07:25:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.420+0000 7f336d43f740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5583a751d750 con 0x5583a75bb4b0 2024-03-06T07:25:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.420+0000 7f33637fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bd330 0x5583a75baf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.420+0000 7f33637fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bd330 0x5583a75baf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59714/0 (socket says 172.21.15.1:59714) 2024-03-06T07:25:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.420+0000 7f33637fe640 1 -- 172.21.15.1:0/3751377810 learned_addr learned my addr 172.21.15.1:0/3751377810 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.421+0000 7f3363fff640 1 --2- 172.21.15.1:0/3751377810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bcf50 0x5583a75baa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.421+0000 7f3368df2640 1 --2- 172.21.15.1:0/3751377810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75bb4b0 0x5583a75b7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.421+0000 7f33637fe640 1 -- 172.21.15.1:0/3751377810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75bb4b0 msgr2=0x5583a75b7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.421+0000 7f33637fe640 1 --2- 172.21.15.1:0/3751377810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75bb4b0 0x5583a75b7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.421+0000 7f33637fe640 1 -- 172.21.15.1:0/3751377810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bcf50 msgr2=0x5583a75baa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.421+0000 7f33637fe640 1 --2- 172.21.15.1:0/3751377810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bcf50 0x5583a75baa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.421+0000 7f33637fe640 1 -- 172.21.15.1:0/3751377810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583a74e8f30 con 0x5583a75bd330 2024-03-06T07:25:13.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.421+0000 7f33637fe640 1 --2- 172.21.15.1:0/3751377810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bd330 0x5583a75baf70 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f33540014d0 tx=0x7f3354013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a61d615b0e18e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.421+0000 7f3349ffb640 1 -- 172.21.15.1:0/3751377810 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354027070 con 0x5583a75bd330 2024-03-06T07:25:13.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.421+0000 7f3349ffb640 1 -- 172.21.15.1:0/3751377810 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33540146c0 con 0x5583a75bd330 2024-03-06T07:25:13.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.421+0000 7f3349ffb640 1 -- 172.21.15.1:0/3751377810 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354014a70 con 0x5583a75bd330 2024-03-06T07:25:13.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.422+0000 7f336d43f740 1 -- 172.21.15.1:0/3751377810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bd330 msgr2=0x5583a75baf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.422+0000 7f336d43f740 1 --2- 172.21.15.1:0/3751377810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bd330 0x5583a75baf70 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f33540014d0 tx=0x7f3354013000 comp rx=0 tx=0).stop 2024-03-06T07:25:13.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.422+0000 7f336d43f740 1 -- 172.21.15.1:0/3751377810 shutdown_connections 2024-03-06T07:25:13.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.422+0000 7f336d43f740 1 --2- 172.21.15.1:0/3751377810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75bb4b0 0x5583a75b7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.422+0000 7f336d43f740 1 --2- 172.21.15.1:0/3751377810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bd330 0x5583a75baf70 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.422+0000 7f336d43f740 1 --2- 172.21.15.1:0/3751377810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bcf50 0x5583a75baa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.422+0000 7f336d43f740 1 -- 172.21.15.1:0/3751377810 >> 172.21.15.1:0/3751377810 conn(0x5583a75b97d0 msgr2=0x5583a75de3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.422+0000 7f336d43f740 1 -- 172.21.15.1:0/3751377810 shutdown_connections 2024-03-06T07:25:13.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.422+0000 7f336d43f740 1 -- 172.21.15.1:0/3751377810 wait complete. 2024-03-06T07:25:13.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.423+0000 7f336d43f740 1 -- start start 2024-03-06T07:25:13.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.423+0000 7f336d43f740 1 Processor -- start 2024-03-06T07:25:13.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.424+0000 7f336d43f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bcf50 0x5583a75da480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.424+0000 7f336d43f740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75bd330 0x5583a75dd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.424+0000 7f336d43f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75e1d80 0x5583a75e0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.424+0000 7f336d43f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583a75df230 con 0x5583a75e1d80 2024-03-06T07:25:13.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.424+0000 7f336d43f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583a74ddea0 con 0x5583a75bcf50 2024-03-06T07:25:13.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.424+0000 7f336d43f740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5583a75dec70 con 0x5583a75bd330 2024-03-06T07:25:13.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.424+0000 7f336d43f740 10 client.0 _ll_register_callbacks cb 0x5583a74dd870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:13.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.424+0000 7f3368df2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75e1d80 0x5583a75e0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.424+0000 7f3368df2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75e1d80 0x5583a75e0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59728/0 (socket says 172.21.15.1:59728) 2024-03-06T07:25:13.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.424+0000 7f3368df2640 1 -- 172.21.15.1:0/3140853383 learned_addr learned my addr 172.21.15.1:0/3140853383 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:13.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.424+0000 7f33637fe640 1 --2- 172.21.15.1:0/3140853383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75bd330 0x5583a75dd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.424+0000 7f3363fff640 1 --2- 172.21.15.1:0/3140853383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bcf50 0x5583a75da480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f3368df2640 1 -- 172.21.15.1:0/3140853383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75bd330 msgr2=0x5583a75dd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f3368df2640 1 --2- 172.21.15.1:0/3140853383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75bd330 0x5583a75dd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f3368df2640 1 -- 172.21.15.1:0/3140853383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bcf50 msgr2=0x5583a75da480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f3368df2640 1 --2- 172.21.15.1:0/3140853383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bcf50 0x5583a75da480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f3368df2640 1 -- 172.21.15.1:0/3140853383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5583a74de270 con 0x5583a75e1d80 2024-03-06T07:25:13.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f3368df2640 1 --2- 172.21.15.1:0/3140853383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75e1d80 0x5583a75e0a60 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f33640c52d0 tx=0x7f335c013680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3759b87569542e8a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f3342ffd640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:25:13.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f3342ffd640 1 -- 172.21.15.1:0/3140853383 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335c015ea0 con 0x5583a75e1d80 2024-03-06T07:25:13.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f3342ffd640 1 -- 172.21.15.1:0/3140853383 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f335c003990 con 0x5583a75e1d80 2024-03-06T07:25:13.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f336d43f740 1 -- 172.21.15.1:0/3140853383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33540146c0 con 0x5583a75e1d80 2024-03-06T07:25:13.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f3342ffd640 1 -- 172.21.15.1:0/3140853383 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) v2 ==== 24+0+0 (secure 0 0 0) 0x7f335c028050 con 0x5583a75e1d80 2024-03-06T07:25:13.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f336d43f740 10 client.8133 fetch_fsmap learned FSMap version 115 2024-03-06T07:25:13.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f336d43f740 1 -- 172.21.15.1:0/3140853383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=2+}) v3 -- 0x5583a72362d0 con 0x5583a75e1d80 2024-03-06T07:25:13.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f3342ffd640 1 -- 172.21.15.1:0/3140853383 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33540146c0 con 0x5583a75e1d80 2024-03-06T07:25:13.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f3342ffd640 1 -- 172.21.15.1:0/3140853383 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 115) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f335c02b020 con 0x5583a75e1d80 2024-03-06T07:25:13.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f336d43f740 10 client.8133 fetch_fsmap finished waiting for FSMap version 115 2024-03-06T07:25:13.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f336d43f740 10 client.8133 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:25:13.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f336d43f740 20 client.8133 populate_metadata read hostname 'smithi001' 2024-03-06T07:25:13.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f336d43f740 1 --2- 172.21.15.1:0/3140853383 >> [v2:172.21.15.1:6832/1747678472,v1:172.21.15.1:6834/1747678472] conn(0x5583a75e01b0 0x5583a75e0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f336d43f740 4 client.8133 mds_command: new command op to 8091 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:13.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.425+0000 7f336d43f740 1 -- 172.21.15.1:0/3140853383 --> [v2:172.21.15.1:6832/1747678472,v1:172.21.15.1:6834/1747678472] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5583a752ebf0 con 0x5583a75e01b0 2024-03-06T07:25:13.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.426+0000 7f3363fff640 1 --2- 172.21.15.1:0/3140853383 >> [v2:172.21.15.1:6832/1747678472,v1:172.21.15.1:6834/1747678472] conn(0x5583a75e01b0 0x5583a75e0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.426+0000 7f3342ffd640 10 client.8133 ms_handle_connect on v2:172.21.15.1:6832/1747678472 2024-03-06T07:25:13.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.426+0000 7f3363fff640 1 --2- 172.21.15.1:0/3140853383 >> [v2:172.21.15.1:6832/1747678472,v1:172.21.15.1:6834/1747678472] conn(0x5583a75e01b0 0x5583a75e0fa0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88a70ac67acf0898 server_cookie=84d177239485ee8b in_seq=0 out_seq=0 2024-03-06T07:25:13.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.427+0000 7f3342ffd640 1 -- 172.21.15.1:0/3140853383 <== mds.0 v2:172.21.15.1:6832/1747678472 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5583a752ebf0 con 0x5583a75e01b0 2024-03-06T07:25:13.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.427+0000 7f3342ffd640 10 client.8133 handle_command_reply: tid=0 2024-03-06T07:25:13.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.427+0000 7f336d43f740 1 client.8133 shutdown 2024-03-06T07:25:13.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.427+0000 7f336d43f740 1 -- 172.21.15.1:0/3140853383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75e1d80 msgr2=0x5583a75e0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.427+0000 7f336d43f740 1 --2- 172.21.15.1:0/3140853383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75e1d80 0x5583a75e0a60 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f33640c52d0 tx=0x7f335c013680 comp rx=0 tx=0).stop 2024-03-06T07:25:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.427+0000 7f336d43f740 1 -- 172.21.15.1:0/3140853383 shutdown_connections 2024-03-06T07:25:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.427+0000 7f336d43f740 1 --2- 172.21.15.1:0/3140853383 >> [v2:172.21.15.1:6832/1747678472,v1:172.21.15.1:6834/1747678472] conn(0x5583a75e01b0 0x5583a75e0fa0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.427+0000 7f336d43f740 1 --2- 172.21.15.1:0/3140853383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75e1d80 0x5583a75e0a60 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.427+0000 7f336d43f740 1 --2- 172.21.15.1:0/3140853383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75bd330 0x5583a75dd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.427+0000 7f3342ffd640 0 client.8133 ms_handle_reset on v2:172.21.15.1:6832/1747678472 2024-03-06T07:25:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.427+0000 7f336d43f740 1 --2- 172.21.15.1:0/3140853383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bcf50 0x5583a75da480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.427+0000 7f336d43f740 1 -- 172.21.15.1:0/3140853383 >> 172.21.15.1:0/3140853383 conn(0x5583a75bb4b0 msgr2=0x5583a75bc380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.427+0000 7f336d43f740 1 -- 172.21.15.1:0/3140853383 shutdown_connections 2024-03-06T07:25:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.427+0000 7f336d43f740 1 -- 172.21.15.1:0/3140853383 wait complete. 2024-03-06T07:25:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.427+0000 7f336d43f740 20 client.8133 trim_cache size 0 max 16384 2024-03-06T07:25:13.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.460+0000 7f336d43f740 1 Processor -- start 2024-03-06T07:25:13.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.460+0000 7f336d43f740 1 -- start start 2024-03-06T07:25:13.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f336d43f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bcf50 0x5583a75e0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f336d43f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bd330 0x5583a75bb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f336d43f740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75e1d80 0x5583a75ba5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f336d43f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33641f77b0 con 0x5583a75bd330 2024-03-06T07:25:13.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f336d43f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336406f4a0 con 0x5583a75bcf50 2024-03-06T07:25:13.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f336d43f740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33641f71b0 con 0x5583a75e1d80 2024-03-06T07:25:13.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f33637fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bd330 0x5583a75bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f33637fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bd330 0x5583a75bb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59738/0 (socket says 172.21.15.1:59738) 2024-03-06T07:25:13.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f33637fe640 1 -- 172.21.15.1:0/2972616704 learned_addr learned my addr 172.21.15.1:0/2972616704 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:13.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f3363fff640 1 --2- 172.21.15.1:0/2972616704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bcf50 0x5583a75e0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f3368df2640 1 --2- 172.21.15.1:0/2972616704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75e1d80 0x5583a75ba5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f33637fe640 1 -- 172.21.15.1:0/2972616704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75e1d80 msgr2=0x5583a75ba5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f33637fe640 1 --2- 172.21.15.1:0/2972616704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75e1d80 0x5583a75ba5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f33637fe640 1 -- 172.21.15.1:0/2972616704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bcf50 msgr2=0x5583a75e0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f33637fe640 1 --2- 172.21.15.1:0/2972616704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bcf50 0x5583a75e0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f33637fe640 1 -- 172.21.15.1:0/2972616704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583a73006b0 con 0x5583a75bd330 2024-03-06T07:25:13.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f3368df2640 1 --2- 172.21.15.1:0/2972616704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75e1d80 0x5583a75ba5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:13.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.461+0000 7f3363fff640 1 --2- 172.21.15.1:0/2972616704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bcf50 0x5583a75e0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:13.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.462+0000 7f33637fe640 1 --2- 172.21.15.1:0/2972616704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bd330 0x5583a75bb890 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f3354001ca0 tx=0x7f3354014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b1835b1b87c0616 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.462+0000 7f33497fa640 1 -- 172.21.15.1:0/2972616704 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354027070 con 0x5583a75bd330 2024-03-06T07:25:13.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.462+0000 7f33497fa640 1 -- 172.21.15.1:0/2972616704 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3354033070 con 0x5583a75bd330 2024-03-06T07:25:13.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.462+0000 7f33497fa640 1 -- 172.21.15.1:0/2972616704 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354014b00 con 0x5583a75bd330 2024-03-06T07:25:13.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.462+0000 7f336d43f740 1 -- 172.21.15.1:0/2972616704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bd330 msgr2=0x5583a75bb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.462+0000 7f336d43f740 1 --2- 172.21.15.1:0/2972616704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bd330 0x5583a75bb890 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f3354001ca0 tx=0x7f3354014000 comp rx=0 tx=0).stop 2024-03-06T07:25:13.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.462+0000 7f336d43f740 1 -- 172.21.15.1:0/2972616704 shutdown_connections 2024-03-06T07:25:13.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.462+0000 7f336d43f740 1 --2- 172.21.15.1:0/2972616704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75e1d80 0x5583a75ba5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.462+0000 7f336d43f740 1 --2- 172.21.15.1:0/2972616704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bd330 0x5583a75bb890 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.462+0000 7f336d43f740 1 --2- 172.21.15.1:0/2972616704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bcf50 0x5583a75e0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.462+0000 7f336d43f740 1 -- 172.21.15.1:0/2972616704 >> 172.21.15.1:0/2972616704 conn(0x5583a75bb4b0 msgr2=0x5583a75bc380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.462+0000 7f336d43f740 1 -- 172.21.15.1:0/2972616704 shutdown_connections 2024-03-06T07:25:13.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.462+0000 7f336d43f740 1 -- 172.21.15.1:0/2972616704 wait complete. 2024-03-06T07:25:13.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.463+0000 7f336d43f740 1 -- start start 2024-03-06T07:25:13.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.463+0000 7f336d43f740 1 Processor -- start 2024-03-06T07:25:13.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.464+0000 7f336d43f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bcf50 0x5583a75dcbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.464+0000 7f336d43f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bd330 0x5583a76094e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.464+0000 7f336d43f740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75e1d80 0x5583a7605fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.464+0000 7f336d43f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583a75e1560 con 0x5583a75bcf50 2024-03-06T07:25:13.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.464+0000 7f336d43f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583a760d6e0 con 0x5583a75bd330 2024-03-06T07:25:13.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.464+0000 7f336d43f740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5583a75df600 con 0x5583a75e1d80 2024-03-06T07:25:13.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.464+0000 7f336d43f740 10 client.0 _ll_register_callbacks cb 0x7f335c0156d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:13.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.464+0000 7f3363fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bcf50 0x5583a75dcbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.464+0000 7f3363fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bcf50 0x5583a75dcbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59742/0 (socket says 172.21.15.1:59742) 2024-03-06T07:25:13.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.464+0000 7f3363fff640 1 -- 172.21.15.1:0/785627757 learned_addr learned my addr 172.21.15.1:0/785627757 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:13.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.464+0000 7f3368df2640 1 --2- 172.21.15.1:0/785627757 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75e1d80 0x5583a7605fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.464+0000 7f33637fe640 1 --2- 172.21.15.1:0/785627757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bd330 0x5583a76094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f3363fff640 1 -- 172.21.15.1:0/785627757 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75e1d80 msgr2=0x5583a7605fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f3363fff640 1 --2- 172.21.15.1:0/785627757 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75e1d80 0x5583a7605fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f3363fff640 1 -- 172.21.15.1:0/785627757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bd330 msgr2=0x5583a76094e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f3363fff640 1 --2- 172.21.15.1:0/785627757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bd330 0x5583a76094e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f3363fff640 1 -- 172.21.15.1:0/785627757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5583a75b7750 con 0x5583a75bcf50 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f33637fe640 1 --2- 172.21.15.1:0/785627757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bd330 0x5583a76094e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f3363fff640 1 --2- 172.21.15.1:0/785627757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bcf50 0x5583a75dcbb0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f3358079d30 tx=0x7f33580ab040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4948f29f8374aaf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f3342ffd640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f3342ffd640 1 -- 172.21.15.1:0/785627757 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358079480 con 0x5583a75bcf50 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f3342ffd640 1 -- 172.21.15.1:0/785627757 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f3358079620 con 0x5583a75bcf50 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f336d43f740 1 -- 172.21.15.1:0/785627757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3354033070 con 0x5583a75bcf50 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f3342ffd640 1 -- 172.21.15.1:0/785627757 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3358007360 con 0x5583a75bcf50 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f336d43f740 10 client.8139 fetch_fsmap learned FSMap version 115 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f336d43f740 1 -- 172.21.15.1:0/785627757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=2+}) v3 -- 0x5583a7609a20 con 0x5583a75bcf50 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f3342ffd640 1 -- 172.21.15.1:0/785627757 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3358098e50 con 0x5583a75bcf50 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f3342ffd640 1 -- 172.21.15.1:0/785627757 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 115) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f3358049070 con 0x5583a75bcf50 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f336d43f740 10 client.8139 fetch_fsmap finished waiting for FSMap version 115 2024-03-06T07:25:13.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f336d43f740 10 client.8139 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:25:13.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f336d43f740 20 client.8139 populate_metadata read hostname 'smithi001' 2024-03-06T07:25:13.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f336d43f740 1 --2- 172.21.15.1:0/785627757 >> [v2:172.21.15.1:6832/1747678472,v1:172.21.15.1:6834/1747678472] conn(0x5583a76068c0 0x5583a7666b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f336d43f740 4 client.8139 mds_command: new command op to 8091 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:13.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.465+0000 7f336d43f740 1 -- 172.21.15.1:0/785627757 --> [v2:172.21.15.1:6832/1747678472,v1:172.21.15.1:6834/1747678472] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5583a7528f40 con 0x5583a76068c0 2024-03-06T07:25:13.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.466+0000 7f33637fe640 1 --2- 172.21.15.1:0/785627757 >> [v2:172.21.15.1:6832/1747678472,v1:172.21.15.1:6834/1747678472] conn(0x5583a76068c0 0x5583a7666b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.466+0000 7f3342ffd640 10 client.8139 ms_handle_connect on v2:172.21.15.1:6832/1747678472 2024-03-06T07:25:13.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.466+0000 7f33637fe640 1 --2- 172.21.15.1:0/785627757 >> [v2:172.21.15.1:6832/1747678472,v1:172.21.15.1:6834/1747678472] conn(0x5583a76068c0 0x5583a7666b90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=658aaf2bd1bb9579 server_cookie=82d8e43bcf7ebd75 in_seq=0 out_seq=0 2024-03-06T07:25:13.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.466+0000 7f3342ffd640 1 -- 172.21.15.1:0/785627757 <== mds.0 v2:172.21.15.1:6832/1747678472 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x5583a7528f40 con 0x5583a76068c0 2024-03-06T07:25:13.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.466+0000 7f3342ffd640 10 client.8139 handle_command_reply: tid=0 2024-03-06T07:25:13.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.466+0000 7f336d43f740 1 client.8139 shutdown 2024-03-06T07:25:13.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.466+0000 7f336d43f740 1 -- 172.21.15.1:0/785627757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bcf50 msgr2=0x5583a75dcbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.466+0000 7f336d43f740 1 --2- 172.21.15.1:0/785627757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bcf50 0x5583a75dcbb0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f3358079d30 tx=0x7f33580ab040 comp rx=0 tx=0).stop 2024-03-06T07:25:13.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.466+0000 7f336d43f740 1 -- 172.21.15.1:0/785627757 shutdown_connections 2024-03-06T07:25:13.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.466+0000 7f336d43f740 1 --2- 172.21.15.1:0/785627757 >> [v2:172.21.15.1:6832/1747678472,v1:172.21.15.1:6834/1747678472] conn(0x5583a76068c0 0x5583a7666b90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.466+0000 7f336d43f740 1 --2- 172.21.15.1:0/785627757 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583a75e1d80 0x5583a7605fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.466+0000 7f336d43f740 1 --2- 172.21.15.1:0/785627757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583a75bd330 0x5583a76094e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.466+0000 7f3342ffd640 0 client.8139 ms_handle_reset on v2:172.21.15.1:6832/1747678472 2024-03-06T07:25:13.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.466+0000 7f336d43f740 1 --2- 172.21.15.1:0/785627757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583a75bcf50 0x5583a75dcbb0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.466+0000 7f336d43f740 1 -- 172.21.15.1:0/785627757 >> 172.21.15.1:0/785627757 conn(0x5583a75bb4b0 msgr2=0x5583a75bb890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.466+0000 7f336d43f740 1 -- 172.21.15.1:0/785627757 shutdown_connections 2024-03-06T07:25:13.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.466+0000 7f336d43f740 1 -- 172.21.15.1:0/785627757 wait complete. 2024-03-06T07:25:13.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.467+0000 7f336d43f740 20 client.8139 trim_cache size 0 max 16384 2024-03-06T07:25:13.477 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:25:13.477 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:25:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:25:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 8091, 2024-03-06T07:25:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:25:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:25:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:25:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 4.1986149160000004, 2024-03-06T07:25:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 115, 2024-03-06T07:25:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 123, 2024-03-06T07:25:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 123, 2024-03-06T07:25:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 6.2276956920000002 2024-03-06T07:25:13.478 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:25:13.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.467+0000 7f336a855640 1 -- 172.21.15.1:0/3345416803 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f335006b4d0 msgr2=0x7f335006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.467+0000 7f336a855640 1 --2- 172.21.15.1:0/3345416803 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f335006b4d0 0x7f335006d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3354000c00 tx=0x7f3354009040 comp rx=0 tx=0).stop 2024-03-06T07:25:13.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.467+0000 7f336a855640 1 -- 172.21.15.1:0/3345416803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0039b0 msgr2=0x7f3364152100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.467+0000 7f336a855640 1 --2- 172.21.15.1:0/3345416803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0039b0 0x7f3364152100 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f335800c360 tx=0x7f3364152ea0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.467+0000 7f336a855640 1 -- 172.21.15.1:0/3345416803 shutdown_connections 2024-03-06T07:25:13.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.467+0000 7f336a855640 1 --2- 172.21.15.1:0/3345416803 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f335006b4d0 0x7f335006d990 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.467+0000 7f336a855640 1 --2- 172.21.15.1:0/3345416803 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3364154230 0x7f33640ac6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.467+0000 7f336a855640 1 --2- 172.21.15.1:0/3345416803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3364070600 0x7f33640ac1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.467+0000 7f336a855640 1 --2- 172.21.15.1:0/3345416803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0039b0 0x7f3364152100 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.467+0000 7f336a855640 1 -- 172.21.15.1:0/3345416803 >> 172.21.15.1:0/3345416803 conn(0x7f33640c7a50 msgr2=0x7f33640c0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.467+0000 7f336a855640 1 -- 172.21.15.1:0/3345416803 shutdown_connections 2024-03-06T07:25:13.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.467+0000 7f336a855640 1 -- 172.21.15.1:0/3345416803 wait complete. 2024-03-06T07:25:13.481 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 115, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:24:52.598303+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 115, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8091": { "addr": "172.21.15.1:6834/1747678472", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 1747678472, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 1747678472, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8091, "incarnation": 112, "join_fscid": 9, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 123, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-03-06T07:25:12.276319+0000", "qdb_cluster": [ 8091 ], "qdb_leader": 8091, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8091 } } } ], "standbys": [ { "addr": "172.21.15.1:6835/4151599120", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 4151599120, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 4151599120, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6446, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6837/155600632", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 155600632, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 155600632, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6458, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6836/1600553134", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 1600553134, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 1600553134, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 113, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6587, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:25:13.482 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 9, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:25:13.482 INFO:tasks.cephfs.test_failover:current = [{'gid': 6446, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/4151599120', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 4151599120}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 4151599120}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}, {'gid': 6587, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/1600553134', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 1600553134}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1600553134}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 113}, {'gid': 8091, 'name': 'c', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.1:6834/1747678472', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1747678472}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1747678472}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6458, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/155600632', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 155600632}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 155600632}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}] 2024-03-06T07:25:13.482 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-06T07:25:13.482 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:13.482 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:13.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:13.568 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:13.569 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:25:13.569 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:13.584 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:25:13.584 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:25:13.585 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:13.602 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:25:13.602 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:13.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.761+0000 7f8bb6b1e640 1 Processor -- start 2024-03-06T07:25:13.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.761+0000 7f8bb6b1e640 1 -- start start 2024-03-06T07:25:13.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.762+0000 7f8bb6b1e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8bb0152650 0x7f8bb0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.762+0000 7f8bb6b1e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bb00bf9d0 0x7f8bb00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.762+0000 7f8bb6b1e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb00c2fc0 con 0x7f8bb00c6b10 2024-03-06T07:25:13.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.762+0000 7f8bb6b1e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bb0155800 con 0x7f8bb0152650 2024-03-06T07:25:13.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.762+0000 7f8bb6b1e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb0158da0 con 0x7f8bb00bf9d0 2024-03-06T07:25:13.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.763+0000 7f8bb50bb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bb00bf9d0 0x7f8bb00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.763+0000 7f8bb48ba640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8bb0152650 0x7f8bb0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.763+0000 7f8bb48ba640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8bb0152650 0x7f8bb0155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:37694/0 (socket says 172.21.15.1:37694) 2024-03-06T07:25:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.763+0000 7f8bb50bb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bb00bf9d0 0x7f8bb00c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:50756/0 (socket says 172.21.15.1:50756) 2024-03-06T07:25:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.763+0000 7f8bb48ba640 1 -- 172.21.15.1:0/2885491138 learned_addr learned my addr 172.21.15.1:0/2885491138 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.763+0000 7f8bb50bb640 1 -- 172.21.15.1:0/2885491138 >> v2:172.21.15.98:3300/0 conn(0x7f8bb0152650 msgr2=0x7f8bb0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.763+0000 7f8bb50bb640 1 --2- 172.21.15.1:0/2885491138 >> v2:172.21.15.98:3300/0 conn(0x7f8bb0152650 0x7f8bb0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.763+0000 7f8bb50bb640 1 -- 172.21.15.1:0/2885491138 >> v1:172.21.15.98:6789/0 conn(0x7f8bb00c6b10 legacy=0x7f8bb0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.763+0000 7f8bb50bb640 1 -- 172.21.15.1:0/2885491138 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb00c6690 con 0x7f8bb00bf9d0 2024-03-06T07:25:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.763+0000 7f8baf7fe640 1 -- 172.21.15.1:0/2885491138 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554623188 0 0) 0x7f8bb00c2fc0 con 0x7f8bb00c6b10 2024-03-06T07:25:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.763+0000 7f8bb48ba640 1 --2- 172.21.15.1:0/2885491138 >> v2:172.21.15.98:3300/0 conn(0x7f8bb0152650 0x7f8bb0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:13.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.764+0000 7f8bb50bb640 1 --2- 172.21.15.1:0/2885491138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bb00bf9d0 0x7f8bb00c1460 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009d90 tx=0x7f8ba802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c21ef946768637d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.764+0000 7f8baf7fe640 1 -- 172.21.15.1:0/2885491138 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba8039070 con 0x7f8bb00bf9d0 2024-03-06T07:25:13.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.764+0000 7f8baf7fe640 1 -- 172.21.15.1:0/2885491138 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba802cbc0 con 0x7f8bb00bf9d0 2024-03-06T07:25:13.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.764+0000 7f8baf7fe640 1 -- 172.21.15.1:0/2885491138 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba8034440 con 0x7f8bb00bf9d0 2024-03-06T07:25:13.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.765+0000 7f8bb6b1e640 1 -- 172.21.15.1:0/2885491138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bb00bf9d0 msgr2=0x7f8bb00c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.765+0000 7f8bb6b1e640 1 --2- 172.21.15.1:0/2885491138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bb00bf9d0 0x7f8bb00c1460 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009d90 tx=0x7f8ba802be10 comp rx=0 tx=0).stop 2024-03-06T07:25:13.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.765+0000 7f8bb6b1e640 1 -- 172.21.15.1:0/2885491138 shutdown_connections 2024-03-06T07:25:13.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.765+0000 7f8bb6b1e640 1 --2- 172.21.15.1:0/2885491138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bb00bf9d0 0x7f8bb00c1460 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.765+0000 7f8bb6b1e640 1 --2- 172.21.15.1:0/2885491138 >> v2:172.21.15.98:3300/0 conn(0x7f8bb0152650 0x7f8bb0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.765+0000 7f8bb6b1e640 1 -- 172.21.15.1:0/2885491138 >> 172.21.15.1:0/2885491138 conn(0x7f8bb0066ee0 msgr2=0x7f8bb00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.765+0000 7f8bb6b1e640 1 -- 172.21.15.1:0/2885491138 shutdown_connections 2024-03-06T07:25:13.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.765+0000 7f8bb6b1e640 1 -- 172.21.15.1:0/2885491138 wait complete. 2024-03-06T07:25:13.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.767+0000 7f8bb6b1e640 1 Processor -- start 2024-03-06T07:25:13.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.767+0000 7f8bb6b1e640 1 -- start start 2024-03-06T07:25:13.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8bb6b1e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bb00bf9d0 0x7f8bb01eaad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8bb6b1e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bb0152650 0x7f8bb0150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8bb6b1e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bb014bc70 0x7f8bb0153080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8bb6b1e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb01551b0 con 0x7f8bb014bc70 2024-03-06T07:25:13.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8bb6b1e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb00c6fa0 con 0x7f8bb00bf9d0 2024-03-06T07:25:13.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8bb6b1e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb0155490 con 0x7f8bb0152650 2024-03-06T07:25:13.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8bb50bb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bb014bc70 0x7f8bb0153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8bb50bb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bb014bc70 0x7f8bb0153080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59746/0 (socket says 172.21.15.1:59746) 2024-03-06T07:25:13.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8bb50bb640 1 -- 172.21.15.1:0/3763652180 learned_addr learned my addr 172.21.15.1:0/3763652180 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:13.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8bb48ba640 1 --2- 172.21.15.1:0/3763652180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bb00bf9d0 0x7f8bb01eaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8baffff640 1 --2- 172.21.15.1:0/3763652180 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bb0152650 0x7f8bb0150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8bb50bb640 1 -- 172.21.15.1:0/3763652180 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bb0152650 msgr2=0x7f8bb0150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8bb50bb640 1 --2- 172.21.15.1:0/3763652180 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bb0152650 0x7f8bb0150fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8bb50bb640 1 -- 172.21.15.1:0/3763652180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bb00bf9d0 msgr2=0x7f8bb01eaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8bb50bb640 1 --2- 172.21.15.1:0/3763652180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bb00bf9d0 0x7f8bb01eaad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8bb50bb640 1 -- 172.21.15.1:0/3763652180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb01eb1d0 con 0x7f8bb014bc70 2024-03-06T07:25:13.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8bb48ba640 1 --2- 172.21.15.1:0/3763652180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bb00bf9d0 0x7f8bb01eaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:13.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.768+0000 7f8baffff640 1 --2- 172.21.15.1:0/3763652180 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bb0152650 0x7f8bb0150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:13.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.769+0000 7f8bb50bb640 1 --2- 172.21.15.1:0/3763652180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bb014bc70 0x7f8bb0153080 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009d90 tx=0x7f8ba8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.769+0000 7f8bad7fa640 1 -- 172.21.15.1:0/3763652180 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba8039050 con 0x7f8bb014bc70 2024-03-06T07:25:13.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.769+0000 7f8bad7fa640 1 -- 172.21.15.1:0/3763652180 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba8045070 con 0x7f8bb014bc70 2024-03-06T07:25:13.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.769+0000 7f8bb6b1e640 1 -- 172.21.15.1:0/3763652180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb01eb4b0 con 0x7f8bb014bc70 2024-03-06T07:25:13.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.769+0000 7f8bad7fa640 1 -- 172.21.15.1:0/3763652180 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba8009450 con 0x7f8bb014bc70 2024-03-06T07:25:13.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.769+0000 7f8bb6b1e640 1 -- 172.21.15.1:0/3763652180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb0152b60 con 0x7f8bb014bc70 2024-03-06T07:25:13.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.770+0000 7f8bb6b1e640 1 -- 172.21.15.1:0/3763652180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb0152d60 con 0x7f8bb014bc70 2024-03-06T07:25:13.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.770+0000 7f8bad7fa640 1 -- 172.21.15.1:0/3763652180 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ba803e850 con 0x7f8bb014bc70 2024-03-06T07:25:13.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.771+0000 7f8bad7fa640 1 --2- 172.21.15.1:0/3763652180 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8b9406b4d0 0x7f8b9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.771+0000 7f8bad7fa640 1 -- 172.21.15.1:0/3763652180 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f8ba80acfe0 con 0x7f8bb014bc70 2024-03-06T07:25:13.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.771+0000 7f8bb48ba640 1 --2- 172.21.15.1:0/3763652180 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8b9406b4d0 0x7f8b9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.772+0000 7f8bb48ba640 1 --2- 172.21.15.1:0/3763652180 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8b9406b4d0 0x7f8b9406d990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4009630 tx=0x7f8ba4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.775+0000 7f8bad7fa640 1 -- 172.21.15.1:0/3763652180 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8bb0152d60 con 0x7f8bb014bc70 2024-03-06T07:25:13.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.992+0000 7f8bb6b1e640 1 -- 172.21.15.1:0/3763652180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bb0067fd0 con 0x7f8bb014bc70 2024-03-06T07:25:13.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.993+0000 7f8bad7fa640 1 -- 172.21.15.1:0/3763652180 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+4977 (secure 0 0 0) 0x7f8ba8032e20 con 0x7f8bb014bc70 2024-03-06T07:25:13.995 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:13.995 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":115,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6446,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4151599120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4151599120},{"type":"v1","addr":"172.21.15.1:6835","nonce":4151599120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/155600632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":155600632},{"type":"v1","addr":"172.21.15.98:6837","nonce":155600632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6587,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1600553134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1600553134},{"type":"v1","addr":"172.21.15.98:6836","nonce":1600553134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.598303+0000","modified":"2024-03-06T07:25:12.276319+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8091":{"gid":8091,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":4,"addr":"172.21.15.1:6834/1747678472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1747678472},{"type":"v1","addr":"172.21.15.1:6834","nonce":1747678472}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8091,"qdb_cluster":[8091]},"id":9}]} 2024-03-06T07:25:13.995 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 115 2024-03-06T07:25:13.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.995+0000 7f8bb6b1e640 1 -- 172.21.15.1:0/3763652180 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8b9406b4d0 msgr2=0x7f8b9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.995+0000 7f8bb6b1e640 1 --2- 172.21.15.1:0/3763652180 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8b9406b4d0 0x7f8b9406d990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4009630 tx=0x7f8ba4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:13.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.995+0000 7f8bb6b1e640 1 -- 172.21.15.1:0/3763652180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bb014bc70 msgr2=0x7f8bb0153080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.995+0000 7f8bb6b1e640 1 --2- 172.21.15.1:0/3763652180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bb014bc70 0x7f8bb0153080 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009d90 tx=0x7f8ba8004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:13.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.995+0000 7f8bb6b1e640 1 -- 172.21.15.1:0/3763652180 shutdown_connections 2024-03-06T07:25:13.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.995+0000 7f8bb6b1e640 1 --2- 172.21.15.1:0/3763652180 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8b9406b4d0 0x7f8b9406d990 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.995+0000 7f8bb6b1e640 1 --2- 172.21.15.1:0/3763652180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bb014bc70 0x7f8bb0153080 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.995+0000 7f8bb6b1e640 1 --2- 172.21.15.1:0/3763652180 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bb0152650 0x7f8bb0150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.995+0000 7f8bb6b1e640 1 --2- 172.21.15.1:0/3763652180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bb00bf9d0 0x7f8bb01eaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.995+0000 7f8bb6b1e640 1 -- 172.21.15.1:0/3763652180 >> 172.21.15.1:0/3763652180 conn(0x7f8bb0066ee0 msgr2=0x7f8bb00c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.995+0000 7f8bb6b1e640 1 -- 172.21.15.1:0/3763652180 shutdown_connections 2024-03-06T07:25:13.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:13.995+0000 7f8bb6b1e640 1 -- 172.21.15.1:0/3763652180 wait complete. 2024-03-06T07:25:14.008 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:14.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.107+0000 7f70e204e640 1 Processor -- start 2024-03-06T07:25:14.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.107+0000 7f70e204e640 1 -- start start 2024-03-06T07:25:14.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.108+0000 7f70e204e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc0c6b10 0x7f70dc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.108+0000 7f70e204e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70dc0c2fc0 con 0x7f70dc152650 2024-03-06T07:25:14.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.108+0000 7f70e204e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70dc158da0 con 0x7f70dc0bf9d0 2024-03-06T07:25:14.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.108+0000 7f70e204e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc155800 con 0x7f70dc0c6b10 2024-03-06T07:25:14.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.108+0000 7f70db7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f70dc152650 0x7f70dc158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56794/0 (socket says 172.21.15.1:56794) 2024-03-06T07:25:14.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.108+0000 7f70db7fe640 1 -- 172.21.15.1:0/2226343596 learned_addr learned my addr 172.21.15.1:0/2226343596 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:14.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.108+0000 7f70daffd640 1 --2- 172.21.15.1:0/2226343596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc0c6b10 0x7f70dc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.108+0000 7f70da7fc640 1 -- 172.21.15.1:0/2226343596 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251894522 0 0) 0x7f70dc0c2fc0 con 0x7f70dc152650 2024-03-06T07:25:14.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.108+0000 7f70da7fc640 1 -- 172.21.15.1:0/2226343596 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70c8003520 con 0x7f70dc152650 2024-03-06T07:25:14.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.108+0000 7f70da7fc640 1 -- 172.21.15.1:0/2226343596 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1165470778 0 0) 0x7f70c8003520 con 0x7f70dc152650 2024-03-06T07:25:14.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.108+0000 7f70da7fc640 1 -- 172.21.15.1:0/2226343596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc0c6b10 msgr2=0x7f70dc155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:14.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.108+0000 7f70da7fc640 1 --2- 172.21.15.1:0/2226343596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc0c6b10 0x7f70dc155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.108+0000 7f70da7fc640 1 -- 172.21.15.1:0/2226343596 >> v1:172.21.15.98:6789/0 conn(0x7f70dc0bf9d0 legacy=0x7f70dc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.108+0000 7f70da7fc640 1 -- 172.21.15.1:0/2226343596 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc0c6690 con 0x7f70dc152650 2024-03-06T07:25:14.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.109+0000 7f70da7fc640 1 -- 172.21.15.1:0/2226343596 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f70c4001670 con 0x7f70dc152650 2024-03-06T07:25:14.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.109+0000 7f70da7fc640 1 -- 172.21.15.1:0/2226343596 >> v1:172.21.15.1:6789/0 conn(0x7f70dc152650 legacy=0x7f70dc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.109+0000 7f70da7fc640 1 --2- 172.21.15.1:0/2226343596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70c80039e0 0x7f70c8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.109+0000 7f70da7fc640 1 --2- 172.21.15.1:0/2226343596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70c8004300 0x7f70c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.109+0000 7f70da7fc640 1 --2- 172.21.15.1:0/2226343596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70c8004f90 0x7f70c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.109+0000 7f70da7fc640 1 -- 172.21.15.1:0/2226343596 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc155800 con 0x7f70c8004300 2024-03-06T07:25:14.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.109+0000 7f70da7fc640 1 -- 172.21.15.1:0/2226343596 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c800bf00 con 0x7f70c8004f90 2024-03-06T07:25:14.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.109+0000 7f70da7fc640 1 -- 172.21.15.1:0/2226343596 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c800c0b0 con 0x7f70c80039e0 2024-03-06T07:25:14.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.109+0000 7f70daffd640 1 --2- 172.21.15.1:0/2226343596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70c8004300 0x7f70c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.109+0000 7f70db7fe640 1 --2- 172.21.15.1:0/2226343596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70c80039e0 0x7f70c8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.109+0000 7f70dbfff640 1 --2- 172.21.15.1:0/2226343596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70c8004f90 0x7f70c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.109+0000 7f70daffd640 1 -- 172.21.15.1:0/2226343596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70c80039e0 msgr2=0x7f70c8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:14.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.109+0000 7f70daffd640 1 --2- 172.21.15.1:0/2226343596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70c80039e0 0x7f70c8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.109+0000 7f70daffd640 1 -- 172.21.15.1:0/2226343596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70c8004f90 msgr2=0x7f70c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.109+0000 7f70daffd640 1 --2- 172.21.15.1:0/2226343596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70c8004f90 0x7f70c800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.109+0000 7f70daffd640 1 -- 172.21.15.1:0/2226343596 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c8003520 con 0x7f70c8004300 2024-03-06T07:25:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.109+0000 7f70db7fe640 1 --2- 172.21.15.1:0/2226343596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70c80039e0 0x7f70c8003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.110+0000 7f70daffd640 1 --2- 172.21.15.1:0/2226343596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70c8004300 0x7f70c8004800 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f70cc009ac0 tx=0x7f70c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e20d098103a57c4c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.110+0000 7f70da7fc640 1 -- 172.21.15.1:0/2226343596 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc039070 con 0x7f70c8004300 2024-03-06T07:25:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.110+0000 7f70da7fc640 1 -- 172.21.15.1:0/2226343596 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70cc02c6b0 con 0x7f70c8004300 2024-03-06T07:25:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.110+0000 7f70da7fc640 1 -- 172.21.15.1:0/2226343596 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc009690 con 0x7f70c8004300 2024-03-06T07:25:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.110+0000 7f70e204e640 1 -- 172.21.15.1:0/2226343596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70c8004300 msgr2=0x7f70c8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.110+0000 7f70e204e640 1 --2- 172.21.15.1:0/2226343596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70c8004300 0x7f70c8004800 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f70cc009ac0 tx=0x7f70c8005830 comp rx=0 tx=0).stop 2024-03-06T07:25:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.110+0000 7f70dbfff640 1 -- 172.21.15.1:0/2226343596 reap_dead start 2024-03-06T07:25:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.110+0000 7f70e204e640 1 -- 172.21.15.1:0/2226343596 shutdown_connections 2024-03-06T07:25:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.110+0000 7f70e204e640 1 -- 172.21.15.1:0/2226343596 >> 172.21.15.1:0/2226343596 conn(0x7f70dc066ee0 msgr2=0x7f70dc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.111+0000 7f70e204e640 1 -- 172.21.15.1:0/2226343596 shutdown_connections 2024-03-06T07:25:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.111+0000 7f70e204e640 1 -- 172.21.15.1:0/2226343596 wait complete. 2024-03-06T07:25:14.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.112+0000 7f70e204e640 1 Processor -- start 2024-03-06T07:25:14.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.112+0000 7f70e204e640 1 -- start start 2024-03-06T07:25:14.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.112+0000 7f70e204e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70c8004300 0x7f70dc1eedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.112+0000 7f70e204e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc1eb800 0x7f70dc153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.112+0000 7f70e204e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70dc151130 0x7f70dc148240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.112+0000 7f70e204e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc1551b0 con 0x7f70dc151130 2024-03-06T07:25:14.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.112+0000 7f70e204e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc0c6fa0 con 0x7f70c8004300 2024-03-06T07:25:14.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.112+0000 7f70e204e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc155490 con 0x7f70dc1eb800 2024-03-06T07:25:14.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.113+0000 7f70dbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70dc151130 0x7f70dc148240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.113+0000 7f70dbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70dc151130 0x7f70dc148240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59758/0 (socket says 172.21.15.1:59758) 2024-03-06T07:25:14.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.113+0000 7f70dbfff640 1 -- 172.21.15.1:0/2533356391 learned_addr learned my addr 172.21.15.1:0/2533356391 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:14.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.113+0000 7f70daffd640 1 --2- 172.21.15.1:0/2533356391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc1eb800 0x7f70dc153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.113+0000 7f70db7fe640 1 --2- 172.21.15.1:0/2533356391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70c8004300 0x7f70dc1eedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.113+0000 7f70dbfff640 1 -- 172.21.15.1:0/2533356391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc1eb800 msgr2=0x7f70dc153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.113+0000 7f70dbfff640 1 --2- 172.21.15.1:0/2533356391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc1eb800 0x7f70dc153220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.113+0000 7f70dbfff640 1 -- 172.21.15.1:0/2533356391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70c8004300 msgr2=0x7f70dc1eedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.113+0000 7f70dbfff640 1 --2- 172.21.15.1:0/2533356391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70c8004300 0x7f70dc1eedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.113+0000 7f70dbfff640 1 -- 172.21.15.1:0/2533356391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc1eb020 con 0x7f70dc151130 2024-03-06T07:25:14.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.113+0000 7f70dbfff640 1 --2- 172.21.15.1:0/2533356391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70dc151130 0x7f70dc148240 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f70dc08c760 tx=0x7f70d400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.113+0000 7f70d3fff640 1 -- 172.21.15.1:0/2533356391 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d4003970 con 0x7f70dc151130 2024-03-06T07:25:14.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.113+0000 7f70d3fff640 1 -- 172.21.15.1:0/2533356391 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d4018070 con 0x7f70dc151130 2024-03-06T07:25:14.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.113+0000 7f70e204e640 1 -- 172.21.15.1:0/2533356391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70dc1581b0 con 0x7f70dc151130 2024-03-06T07:25:14.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.113+0000 7f70d3fff640 1 -- 172.21.15.1:0/2533356391 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d4003d50 con 0x7f70dc151130 2024-03-06T07:25:14.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.113+0000 7f70e204e640 1 -- 172.21.15.1:0/2533356391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70dc153760 con 0x7f70dc151130 2024-03-06T07:25:14.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.114+0000 7f70e204e640 1 -- 172.21.15.1:0/2533356391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70dc1eae30 con 0x7f70dc151130 2024-03-06T07:25:14.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.115+0000 7f70d3fff640 1 -- 172.21.15.1:0/2533356391 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70d40050d0 con 0x7f70dc151130 2024-03-06T07:25:14.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.115+0000 7f70d3fff640 1 --2- 172.21.15.1:0/2533356391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f70bc06b4d0 0x7f70bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.116+0000 7f70d3fff640 1 -- 172.21.15.1:0/2533356391 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f70d4089fd0 con 0x7f70dc151130 2024-03-06T07:25:14.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.116+0000 7f70db7fe640 1 --2- 172.21.15.1:0/2533356391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f70bc06b4d0 0x7f70bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.116+0000 7f70db7fe640 1 --2- 172.21.15.1:0/2533356391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f70bc06b4d0 0x7f70bc06d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f70c4005b70 tx=0x7f70c4007040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.120+0000 7f70d3fff640 1 -- 172.21.15.1:0/2533356391 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f70d40585d0 con 0x7f70dc151130 2024-03-06T07:25:14.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.301+0000 7f70e204e640 1 -- 172.21.15.1:0/2533356391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70dc067fd0 con 0x7f70dc151130 2024-03-06T07:25:14.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.302+0000 7f70d3fff640 1 -- 172.21.15.1:0/2533356391 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) v1 ==== 74+0+17037 (secure 0 0 0) 0x7f70d405c670 con 0x7f70dc151130 2024-03-06T07:25:14.304 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:14.305 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":123,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:25:09.267130+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:50.865151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:51.564469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6836/1420541659":"2024-03-07T07:25:09.267103+0000","172.21.15.98:6835/1420541659":"2024-03-07T07:25:09.267103+0000","172.21.15.1:6834/1515309674":"2024-03-07T07:25:07.207409+0000","172.21.15.1:6832/1515309674":"2024-03-07T07:25:07.207409+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.305+0000 7f70e204e640 1 -- 172.21.15.1:0/2533356391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f70bc06b4d0 msgr2=0x7f70bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.305+0000 7f70e204e640 1 --2- 172.21.15.1:0/2533356391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f70bc06b4d0 0x7f70bc06d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f70c4005b70 tx=0x7f70c4007040 comp rx=0 tx=0).stop 2024-03-06T07:25:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.305+0000 7f70e204e640 1 -- 172.21.15.1:0/2533356391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70dc151130 msgr2=0x7f70dc148240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.305+0000 7f70e204e640 1 --2- 172.21.15.1:0/2533356391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70dc151130 0x7f70dc148240 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f70dc08c760 tx=0x7f70d400eb90 comp rx=0 tx=0).stop 2024-03-06T07:25:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.305+0000 7f70e204e640 1 -- 172.21.15.1:0/2533356391 shutdown_connections 2024-03-06T07:25:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.305+0000 7f70e204e640 1 --2- 172.21.15.1:0/2533356391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f70bc06b4d0 0x7f70bc06d990 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.305+0000 7f70e204e640 1 --2- 172.21.15.1:0/2533356391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70dc151130 0x7f70dc148240 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.305+0000 7f70e204e640 1 --2- 172.21.15.1:0/2533356391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70dc1eb800 0x7f70dc153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.305+0000 7f70e204e640 1 --2- 172.21.15.1:0/2533356391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70c8004300 0x7f70dc1eedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.305+0000 7f70e204e640 1 -- 172.21.15.1:0/2533356391 >> 172.21.15.1:0/2533356391 conn(0x7f70dc066ee0 msgr2=0x7f70dc0cb130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.305+0000 7f70e204e640 1 -- 172.21.15.1:0/2533356391 shutdown_connections 2024-03-06T07:25:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.305+0000 7f70e204e640 1 -- 172.21.15.1:0/2533356391 wait complete. 2024-03-06T07:25:14.319 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:50.865151+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:51.564469+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '121', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:14.320 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.422+0000 7fe745155640 1 Processor -- start 2024-03-06T07:25:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.422+0000 7fe745155640 1 -- start start 2024-03-06T07:25:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.423+0000 7fe745155640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe740152840 0x7fe7400c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.423+0000 7fe745155640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7400c60b0 0x7fe7401553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.423+0000 7fe745155640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7400c5e40 con 0x7fe7400c3970 2024-03-06T07:25:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.423+0000 7fe745155640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7400c67f0 con 0x7fe740152840 2024-03-06T07:25:14.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.423+0000 7fe745155640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7400c6a40 con 0x7fe7400c60b0 2024-03-06T07:25:14.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.424+0000 7fe73e59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe740152840 0x7fe7400c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.424+0000 7fe73f59e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7400c60b0 0x7fe7401553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.424+0000 7fe73ed9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe7400c3970 0x7fe740152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:57938/0 (socket says 172.21.15.1:57938) 2024-03-06T07:25:14.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.424+0000 7fe73ed9d640 1 -- 172.21.15.1:0/4082168147 learned_addr learned my addr 172.21.15.1:0/4082168147 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:14.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.424+0000 7fe73e59c640 1 -- 172.21.15.1:0/4082168147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7400c60b0 msgr2=0x7fe7401553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:14.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.424+0000 7fe73e59c640 1 --2- 172.21.15.1:0/4082168147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7400c60b0 0x7fe7401553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.424+0000 7fe73e59c640 1 -- 172.21.15.1:0/4082168147 >> v1:172.21.15.98:6789/0 conn(0x7fe7400c3970 legacy=0x7fe740152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.424+0000 7fe73dd9b640 1 -- 172.21.15.1:0/4082168147 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809352792 0 0) 0x7fe7400c5e40 con 0x7fe7400c3970 2024-03-06T07:25:14.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.424+0000 7fe73e59c640 1 -- 172.21.15.1:0/4082168147 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7400c6d20 con 0x7fe740152840 2024-03-06T07:25:14.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.424+0000 7fe73f59e640 1 --2- 172.21.15.1:0/4082168147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7400c60b0 0x7fe7401553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:14.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.425+0000 7fe73e59c640 1 --2- 172.21.15.1:0/4082168147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe740152840 0x7fe7400c8170 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe7280097a0 tx=0x7fe72802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55f236c84a0dce37 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.425+0000 7fe73dd9b640 1 -- 172.21.15.1:0/4082168147 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe728035020 con 0x7fe740152840 2024-03-06T07:25:14.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.425+0000 7fe73dd9b640 1 -- 172.21.15.1:0/4082168147 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe728031810 con 0x7fe740152840 2024-03-06T07:25:14.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.425+0000 7fe73dd9b640 1 -- 172.21.15.1:0/4082168147 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe728031be0 con 0x7fe740152840 2024-03-06T07:25:14.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.426+0000 7fe745155640 1 -- 172.21.15.1:0/4082168147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe740152840 msgr2=0x7fe7400c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.426+0000 7fe745155640 1 --2- 172.21.15.1:0/4082168147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe740152840 0x7fe7400c8170 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe7280097a0 tx=0x7fe72802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.426+0000 7fe745155640 1 -- 172.21.15.1:0/4082168147 shutdown_connections 2024-03-06T07:25:14.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.426+0000 7fe745155640 1 --2- 172.21.15.1:0/4082168147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7400c60b0 0x7fe7401553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.426+0000 7fe745155640 1 --2- 172.21.15.1:0/4082168147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe740152840 0x7fe7400c8170 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.426+0000 7fe745155640 1 -- 172.21.15.1:0/4082168147 >> 172.21.15.1:0/4082168147 conn(0x7fe7400c88f0 msgr2=0x7fe7400c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.426+0000 7fe745155640 1 -- 172.21.15.1:0/4082168147 shutdown_connections 2024-03-06T07:25:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.426+0000 7fe745155640 1 -- 172.21.15.1:0/4082168147 wait complete. 2024-03-06T07:25:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.428+0000 7fe745155640 1 Processor -- start 2024-03-06T07:25:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.428+0000 7fe745155640 1 -- start start 2024-03-06T07:25:14.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.429+0000 7fe745155640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7400c3970 0x7fe7400aa0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.429+0000 7fe745155640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7400c60b0 0x7fe7400ac4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.429+0000 7fe745155640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe740152840 0x7fe74014d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.429+0000 7fe745155640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7400c75c0 con 0x7fe7400c3970 2024-03-06T07:25:14.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.429+0000 7fe745155640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7400c72e0 con 0x7fe740152840 2024-03-06T07:25:14.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.429+0000 7fe745155640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7400c3d50 con 0x7fe7400c60b0 2024-03-06T07:25:14.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.429+0000 7fe73ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7400c3970 0x7fe7400aa0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.429+0000 7fe73ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7400c3970 0x7fe7400aa0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34786/0 (socket says 172.21.15.1:34786) 2024-03-06T07:25:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.429+0000 7fe73ed9d640 1 -- 172.21.15.1:0/725066376 learned_addr learned my addr 172.21.15.1:0/725066376 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.429+0000 7fe73e59c640 1 --2- 172.21.15.1:0/725066376 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7400c60b0 0x7fe7400ac4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.429+0000 7fe73f59e640 1 --2- 172.21.15.1:0/725066376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe740152840 0x7fe74014d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.429+0000 7fe73ed9d640 1 -- 172.21.15.1:0/725066376 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7400c60b0 msgr2=0x7fe7400ac4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.429+0000 7fe73ed9d640 1 --2- 172.21.15.1:0/725066376 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7400c60b0 0x7fe7400ac4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.429+0000 7fe73ed9d640 1 -- 172.21.15.1:0/725066376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe740152840 msgr2=0x7fe74014d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.429+0000 7fe73ed9d640 1 --2- 172.21.15.1:0/725066376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe740152840 0x7fe74014d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.429+0000 7fe73ed9d640 1 -- 172.21.15.1:0/725066376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7400aca30 con 0x7fe7400c3970 2024-03-06T07:25:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.429+0000 7fe73ed9d640 1 --2- 172.21.15.1:0/725066376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7400c3970 0x7fe7400aa0f0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fe73000c890 tx=0x7fe73000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.430+0000 7fe7377fe640 1 -- 172.21.15.1:0/725066376 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe730018070 con 0x7fe7400c3970 2024-03-06T07:25:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.430+0000 7fe7377fe640 1 -- 172.21.15.1:0/725066376 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73000a640 con 0x7fe7400c3970 2024-03-06T07:25:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.430+0000 7fe745155640 1 -- 172.21.15.1:0/725066376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7400a98d0 con 0x7fe7400c3970 2024-03-06T07:25:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.430+0000 7fe7377fe640 1 -- 172.21.15.1:0/725066376 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73000aa30 con 0x7fe7400c3970 2024-03-06T07:25:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.430+0000 7fe745155640 1 -- 172.21.15.1:0/725066376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7401eb3e0 con 0x7fe7400c3970 2024-03-06T07:25:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.431+0000 7fe745155640 1 -- 172.21.15.1:0/725066376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7400686e0 con 0x7fe7400c3970 2024-03-06T07:25:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.431+0000 7fe7377fe640 1 -- 172.21.15.1:0/725066376 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe73001b070 con 0x7fe7400c3970 2024-03-06T07:25:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.432+0000 7fe7377fe640 1 --2- 172.21.15.1:0/725066376 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe72006b4d0 0x7fe72006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.432+0000 7fe7377fe640 1 -- 172.21.15.1:0/725066376 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fe730089a50 con 0x7fe7400c3970 2024-03-06T07:25:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.432+0000 7fe73e59c640 1 --2- 172.21.15.1:0/725066376 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe72006b4d0 0x7fe72006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.433+0000 7fe73e59c640 1 --2- 172.21.15.1:0/725066376 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe72006b4d0 0x7fe72006d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe728036000 tx=0x7fe72803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.436+0000 7fe7377fe640 1 -- 172.21.15.1:0/725066376 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe730058720 con 0x7fe7400c3970 2024-03-06T07:25:14.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.638+0000 7fe745155640 1 -- 172.21.15.1:0/725066376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe7400c5bb0 con 0x7fe7400c3970 2024-03-06T07:25:14.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.639+0000 7fe7377fe640 1 -- 172.21.15.1:0/725066376 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v123) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe730058020 con 0x7fe7400c3970 2024-03-06T07:25:14.641 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:25:14.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.641+0000 7fe745155640 1 -- 172.21.15.1:0/725066376 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe72006b4d0 msgr2=0x7fe72006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.641+0000 7fe745155640 1 --2- 172.21.15.1:0/725066376 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe72006b4d0 0x7fe72006d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe728036000 tx=0x7fe72803c040 comp rx=0 tx=0).stop 2024-03-06T07:25:14.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.641+0000 7fe745155640 1 -- 172.21.15.1:0/725066376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7400c3970 msgr2=0x7fe7400aa0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.641+0000 7fe745155640 1 --2- 172.21.15.1:0/725066376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7400c3970 0x7fe7400aa0f0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fe73000c890 tx=0x7fe73000cd50 comp rx=0 tx=0).stop 2024-03-06T07:25:14.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.641+0000 7fe745155640 1 -- 172.21.15.1:0/725066376 shutdown_connections 2024-03-06T07:25:14.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.641+0000 7fe745155640 1 --2- 172.21.15.1:0/725066376 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe72006b4d0 0x7fe72006d990 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.641+0000 7fe745155640 1 --2- 172.21.15.1:0/725066376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe740152840 0x7fe74014d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.641+0000 7fe745155640 1 --2- 172.21.15.1:0/725066376 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7400c60b0 0x7fe7400ac4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.641+0000 7fe745155640 1 --2- 172.21.15.1:0/725066376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7400c3970 0x7fe7400aa0f0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.641+0000 7fe745155640 1 -- 172.21.15.1:0/725066376 >> 172.21.15.1:0/725066376 conn(0x7fe7400c88f0 msgr2=0x7fe740151650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:14.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.641+0000 7fe745155640 1 -- 172.21.15.1:0/725066376 shutdown_connections 2024-03-06T07:25:14.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.641+0000 7fe745155640 1 -- 172.21.15.1:0/725066376 wait complete. 2024-03-06T07:25:14.657 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:14.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.775+0000 7fc61d3e2640 1 Processor -- start 2024-03-06T07:25:14.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.775+0000 7fc61d3e2640 1 -- start start 2024-03-06T07:25:14.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.776+0000 7fc61d3e2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc618152650 0x7fc618155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.776+0000 7fc61d3e2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc6180bf9d0 0x7fc6180c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.776+0000 7fc61d3e2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6180c2fc0 con 0x7fc6180c6b10 2024-03-06T07:25:14.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.776+0000 7fc61d3e2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc618155800 con 0x7fc6180bf9d0 2024-03-06T07:25:14.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.776+0000 7fc61d3e2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc618158da0 con 0x7fc618152650 2024-03-06T07:25:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.776+0000 7fc6177fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc6180bf9d0 0x7fc6180c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.776+0000 7fc6177fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc6180bf9d0 0x7fc6180c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34794/0 (socket says 172.21.15.1:34794) 2024-03-06T07:25:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.776+0000 7fc6177fe640 1 -- 172.21.15.1:0/337002422 learned_addr learned my addr 172.21.15.1:0/337002422 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.776+0000 7fc6177fe640 1 -- 172.21.15.1:0/337002422 >> v2:172.21.15.98:3300/0 conn(0x7fc618152650 msgr2=0x7fc618155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.776+0000 7fc6177fe640 1 --2- 172.21.15.1:0/337002422 >> v2:172.21.15.98:3300/0 conn(0x7fc618152650 0x7fc618155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.776+0000 7fc6177fe640 1 -- 172.21.15.1:0/337002422 >> v1:172.21.15.98:6789/0 conn(0x7fc6180c6b10 legacy=0x7fc618158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.776+0000 7fc6177fe640 1 -- 172.21.15.1:0/337002422 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6180c6690 con 0x7fc6180bf9d0 2024-03-06T07:25:14.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.777+0000 7fc6177fe640 1 --2- 172.21.15.1:0/337002422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6180bf9d0 0x7fc6180c1460 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fc6180cac90 tx=0x7fc61002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ad3b4009650f5a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.777+0000 7fc615ffb640 1 -- 172.21.15.1:0/337002422 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61002c9f0 con 0x7fc6180bf9d0 2024-03-06T07:25:14.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.777+0000 7fc615ffb640 1 -- 172.21.15.1:0/337002422 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc61002cb90 con 0x7fc6180bf9d0 2024-03-06T07:25:14.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.777+0000 7fc615ffb640 1 -- 172.21.15.1:0/337002422 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6100343f0 con 0x7fc6180bf9d0 2024-03-06T07:25:14.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.778+0000 7fc61d3e2640 1 -- 172.21.15.1:0/337002422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6180bf9d0 msgr2=0x7fc6180c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.778+0000 7fc61d3e2640 1 --2- 172.21.15.1:0/337002422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6180bf9d0 0x7fc6180c1460 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fc6180cac90 tx=0x7fc61002be50 comp rx=0 tx=0).stop 2024-03-06T07:25:14.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.778+0000 7fc61d3e2640 1 -- 172.21.15.1:0/337002422 shutdown_connections 2024-03-06T07:25:14.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.778+0000 7fc61d3e2640 1 --2- 172.21.15.1:0/337002422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6180bf9d0 0x7fc6180c1460 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.778+0000 7fc61d3e2640 1 --2- 172.21.15.1:0/337002422 >> v2:172.21.15.98:3300/0 conn(0x7fc618152650 0x7fc618155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.778+0000 7fc61d3e2640 1 -- 172.21.15.1:0/337002422 >> 172.21.15.1:0/337002422 conn(0x7fc618066ee0 msgr2=0x7fc6180672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:14.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.778+0000 7fc61d3e2640 1 -- 172.21.15.1:0/337002422 shutdown_connections 2024-03-06T07:25:14.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.778+0000 7fc61d3e2640 1 -- 172.21.15.1:0/337002422 wait complete. 2024-03-06T07:25:14.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.780+0000 7fc61d3e2640 1 Processor -- start 2024-03-06T07:25:14.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.780+0000 7fc61d3e2640 1 -- start start 2024-03-06T07:25:14.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc61d3e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6180bf9d0 0x7fc618153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc61d3e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc618152650 0x7fc61814b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc61d3e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc61814e580 0x7fc6181ee680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc61d3e2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6181551b0 con 0x7fc61814e580 2024-03-06T07:25:14.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc61d3e2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6180c6fa0 con 0x7fc618152650 2024-03-06T07:25:14.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc61d3e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc618155490 con 0x7fc6180bf9d0 2024-03-06T07:25:14.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc6177fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc61814e580 0x7fc6181ee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc6177fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc61814e580 0x7fc6181ee680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34804/0 (socket says 172.21.15.1:34804) 2024-03-06T07:25:14.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc6177fe640 1 -- 172.21.15.1:0/262389480 learned_addr learned my addr 172.21.15.1:0/262389480 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:14.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc6167fc640 1 --2- 172.21.15.1:0/262389480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc618152650 0x7fc61814b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc616ffd640 1 --2- 172.21.15.1:0/262389480 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6180bf9d0 0x7fc618153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc6177fe640 1 -- 172.21.15.1:0/262389480 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6180bf9d0 msgr2=0x7fc618153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc6177fe640 1 --2- 172.21.15.1:0/262389480 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6180bf9d0 0x7fc618153bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc6177fe640 1 -- 172.21.15.1:0/262389480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc618152650 msgr2=0x7fc61814b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc6177fe640 1 --2- 172.21.15.1:0/262389480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc618152650 0x7fc61814b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc6177fe640 1 -- 172.21.15.1:0/262389480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61814d5d0 con 0x7fc61814e580 2024-03-06T07:25:14.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc6177fe640 1 --2- 172.21.15.1:0/262389480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc61814e580 0x7fc6181ee680 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fc610009f40 tx=0x7fc610004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc5ff7fe640 1 -- 172.21.15.1:0/262389480 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc610043020 con 0x7fc61814e580 2024-03-06T07:25:14.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.781+0000 7fc5ff7fe640 1 -- 172.21.15.1:0/262389480 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc61003e450 con 0x7fc61814e580 2024-03-06T07:25:14.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.782+0000 7fc61d3e2640 1 -- 172.21.15.1:0/262389480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc61814d8b0 con 0x7fc61814e580 2024-03-06T07:25:14.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.782+0000 7fc5ff7fe640 1 -- 172.21.15.1:0/262389480 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc610009510 con 0x7fc61814e580 2024-03-06T07:25:14.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.782+0000 7fc61d3e2640 1 -- 172.21.15.1:0/262389480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61814ea00 con 0x7fc61814e580 2024-03-06T07:25:14.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.783+0000 7fc61d3e2640 1 -- 172.21.15.1:0/262389480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc618067fd0 con 0x7fc61814e580 2024-03-06T07:25:14.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.783+0000 7fc5ff7fe640 1 -- 172.21.15.1:0/262389480 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc610046020 con 0x7fc61814e580 2024-03-06T07:25:14.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.783+0000 7fc5ff7fe640 1 --2- 172.21.15.1:0/262389480 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc60406b4d0 0x7fc60406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.784+0000 7fc5ff7fe640 1 -- 172.21.15.1:0/262389480 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fc6100acbe0 con 0x7fc61814e580 2024-03-06T07:25:14.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.784+0000 7fc616ffd640 1 --2- 172.21.15.1:0/262389480 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc60406b4d0 0x7fc60406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.784+0000 7fc616ffd640 1 --2- 172.21.15.1:0/262389480 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc60406b4d0 0x7fc60406d990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc61814b2e0 tx=0x7fc60c009110 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.792+0000 7fc5ff7fe640 1 -- 172.21.15.1:0/262389480 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc61004d030 con 0x7fc61814e580 2024-03-06T07:25:14.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.973+0000 7fc61d3e2640 1 -- 172.21.15.1:0/262389480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc618156040 con 0x7fc61814e580 2024-03-06T07:25:14.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.973+0000 7fc5ff7fe640 1 -- 172.21.15.1:0/262389480 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v123) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc61007b8b0 con 0x7fc61814e580 2024-03-06T07:25:14.975 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:25:14.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.975+0000 7fc61d3e2640 1 -- 172.21.15.1:0/262389480 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc60406b4d0 msgr2=0x7fc60406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.975+0000 7fc61d3e2640 1 --2- 172.21.15.1:0/262389480 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc60406b4d0 0x7fc60406d990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc61814b2e0 tx=0x7fc60c009110 comp rx=0 tx=0).stop 2024-03-06T07:25:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.975+0000 7fc61d3e2640 1 -- 172.21.15.1:0/262389480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc61814e580 msgr2=0x7fc6181ee680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.975+0000 7fc61d3e2640 1 --2- 172.21.15.1:0/262389480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc61814e580 0x7fc6181ee680 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fc610009f40 tx=0x7fc610004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.975+0000 7fc61d3e2640 1 -- 172.21.15.1:0/262389480 shutdown_connections 2024-03-06T07:25:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.975+0000 7fc61d3e2640 1 --2- 172.21.15.1:0/262389480 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc60406b4d0 0x7fc60406d990 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.975+0000 7fc61d3e2640 1 --2- 172.21.15.1:0/262389480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc61814e580 0x7fc6181ee680 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.975+0000 7fc61d3e2640 1 --2- 172.21.15.1:0/262389480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc618152650 0x7fc61814b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.975+0000 7fc61d3e2640 1 --2- 172.21.15.1:0/262389480 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6180bf9d0 0x7fc618153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.975+0000 7fc61d3e2640 1 -- 172.21.15.1:0/262389480 >> 172.21.15.1:0/262389480 conn(0x7fc618066ee0 msgr2=0x7fc6180bfdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.975+0000 7fc61d3e2640 1 -- 172.21.15.1:0/262389480 shutdown_connections 2024-03-06T07:25:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:14.975+0000 7fc61d3e2640 1 -- 172.21.15.1:0/262389480 wait complete. 2024-03-06T07:25:14.988 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:15.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.090+0000 7fc037ec9640 1 Processor -- start 2024-03-06T07:25:15.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.090+0000 7fc037ec9640 1 -- start start 2024-03-06T07:25:15.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.091+0000 7fc037ec9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc0300c6b10 0x7fc030155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.091+0000 7fc037ec9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc0300bf9d0 0x7fc0300c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.091+0000 7fc037ec9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0300c2fc0 con 0x7fc030152650 2024-03-06T07:25:15.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.091+0000 7fc037ec9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc030155800 con 0x7fc0300bf9d0 2024-03-06T07:25:15.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.091+0000 7fc037ec9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc030158da0 con 0x7fc0300c6b10 2024-03-06T07:25:15.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.091+0000 7fc036466640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc0300bf9d0 0x7fc0300c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.091+0000 7fc035c65640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc030152650 0x7fc030158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39380/0 (socket says 172.21.15.1:39380) 2024-03-06T07:25:15.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.091+0000 7fc035c65640 1 -- 172.21.15.1:0/859354013 learned_addr learned my addr 172.21.15.1:0/859354013 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:15.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.091+0000 7fc035464640 1 --2- 172.21.15.1:0/859354013 >> v2:172.21.15.98:3300/0 conn(0x7fc0300c6b10 0x7fc030155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.091+0000 7fc036466640 1 -- 172.21.15.1:0/859354013 >> v2:172.21.15.98:3300/0 conn(0x7fc0300c6b10 msgr2=0x7fc030155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.091+0000 7fc036466640 1 --2- 172.21.15.1:0/859354013 >> v2:172.21.15.98:3300/0 conn(0x7fc0300c6b10 0x7fc030155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.091+0000 7fc036466640 1 -- 172.21.15.1:0/859354013 >> v1:172.21.15.1:6789/0 conn(0x7fc030152650 legacy=0x7fc030158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.091+0000 7fc036466640 1 -- 172.21.15.1:0/859354013 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0300c6690 con 0x7fc0300bf9d0 2024-03-06T07:25:15.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.091+0000 7fc034c63640 1 -- 172.21.15.1:0/859354013 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353495716 0 0) 0x7fc0300c2fc0 con 0x7fc030152650 2024-03-06T07:25:15.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.091+0000 7fc035464640 1 --2- 172.21.15.1:0/859354013 >> v2:172.21.15.98:3300/0 conn(0x7fc0300c6b10 0x7fc030155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:15.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.092+0000 7fc036466640 1 --2- 172.21.15.1:0/859354013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0300bf9d0 0x7fc0300c1460 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fc0300cb8e0 tx=0x7fc02802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202d73ac8f5559ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.092+0000 7fc034c63640 1 -- 172.21.15.1:0/859354013 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc028039070 con 0x7fc0300bf9d0 2024-03-06T07:25:15.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.092+0000 7fc034c63640 1 -- 172.21.15.1:0/859354013 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02802cbf0 con 0x7fc0300bf9d0 2024-03-06T07:25:15.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.092+0000 7fc034c63640 1 -- 172.21.15.1:0/859354013 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc028034440 con 0x7fc0300bf9d0 2024-03-06T07:25:15.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.093+0000 7fc037ec9640 1 -- 172.21.15.1:0/859354013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0300bf9d0 msgr2=0x7fc0300c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.093+0000 7fc037ec9640 1 --2- 172.21.15.1:0/859354013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0300bf9d0 0x7fc0300c1460 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fc0300cb8e0 tx=0x7fc02802be40 comp rx=0 tx=0).stop 2024-03-06T07:25:15.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.093+0000 7fc037ec9640 1 -- 172.21.15.1:0/859354013 shutdown_connections 2024-03-06T07:25:15.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.093+0000 7fc037ec9640 1 --2- 172.21.15.1:0/859354013 >> v2:172.21.15.98:3300/0 conn(0x7fc0300c6b10 0x7fc030155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.093+0000 7fc037ec9640 1 --2- 172.21.15.1:0/859354013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0300bf9d0 0x7fc0300c1460 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.093+0000 7fc037ec9640 1 -- 172.21.15.1:0/859354013 >> 172.21.15.1:0/859354013 conn(0x7fc030066ee0 msgr2=0x7fc0300672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.093+0000 7fc037ec9640 1 -- 172.21.15.1:0/859354013 shutdown_connections 2024-03-06T07:25:15.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.093+0000 7fc037ec9640 1 -- 172.21.15.1:0/859354013 wait complete. 2024-03-06T07:25:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.094+0000 7fc037ec9640 1 Processor -- start 2024-03-06T07:25:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.094+0000 7fc037ec9640 1 -- start start 2024-03-06T07:25:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.094+0000 7fc037ec9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0300bf9d0 0x7fc0301eb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc037ec9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc030152650 0x7fc0300c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc037ec9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc03014b4e0 0x7fc030152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc037ec9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0301551b0 con 0x7fc0300bf9d0 2024-03-06T07:25:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc037ec9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0300c6fa0 con 0x7fc030152650 2024-03-06T07:25:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc037ec9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc030155490 con 0x7fc03014b4e0 2024-03-06T07:25:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc035c65640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0300bf9d0 0x7fc0301eb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc035c65640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0300bf9d0 0x7fc0301eb450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34808/0 (socket says 172.21.15.1:34808) 2024-03-06T07:25:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc035c65640 1 -- 172.21.15.1:0/3448093831 learned_addr learned my addr 172.21.15.1:0/3448093831 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc035464640 1 --2- 172.21.15.1:0/3448093831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc030152650 0x7fc0300c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc036466640 1 --2- 172.21.15.1:0/3448093831 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc03014b4e0 0x7fc030152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc035c65640 1 -- 172.21.15.1:0/3448093831 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc03014b4e0 msgr2=0x7fc030152a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc035c65640 1 --2- 172.21.15.1:0/3448093831 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc03014b4e0 0x7fc030152a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc035c65640 1 -- 172.21.15.1:0/3448093831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc030152650 msgr2=0x7fc0300c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc035c65640 1 --2- 172.21.15.1:0/3448093831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc030152650 0x7fc0300c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc035c65640 1 -- 172.21.15.1:0/3448093831 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc030153c70 con 0x7fc0300bf9d0 2024-03-06T07:25:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc035c65640 1 --2- 172.21.15.1:0/3448093831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0300bf9d0 0x7fc0301eb450 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fc02c00c4d0 tx=0x7fc02c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc0267fc640 1 -- 172.21.15.1:0/3448093831 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c018070 con 0x7fc0300bf9d0 2024-03-06T07:25:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc0267fc640 1 -- 172.21.15.1:0/3448093831 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02c007500 con 0x7fc0300bf9d0 2024-03-06T07:25:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc037ec9640 1 -- 172.21.15.1:0/3448093831 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc030154c20 con 0x7fc0300bf9d0 2024-03-06T07:25:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc0267fc640 1 -- 172.21.15.1:0/3448093831 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c004a20 con 0x7fc0300bf9d0 2024-03-06T07:25:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.095+0000 7fc037ec9640 1 -- 172.21.15.1:0/3448093831 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc030154f00 con 0x7fc0300bf9d0 2024-03-06T07:25:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.097+0000 7fc037ec9640 1 -- 172.21.15.1:0/3448093831 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc030067fd0 con 0x7fc0300bf9d0 2024-03-06T07:25:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.097+0000 7fc0267fc640 1 -- 172.21.15.1:0/3448093831 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc02c01b070 con 0x7fc0300bf9d0 2024-03-06T07:25:15.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.097+0000 7fc0267fc640 1 --2- 172.21.15.1:0/3448093831 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc01406b4d0 0x7fc01406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.097+0000 7fc0267fc640 1 -- 172.21.15.1:0/3448093831 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fc02c0896f0 con 0x7fc0300bf9d0 2024-03-06T07:25:15.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.097+0000 7fc035464640 1 --2- 172.21.15.1:0/3448093831 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc01406b4d0 0x7fc01406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.098+0000 7fc035464640 1 --2- 172.21.15.1:0/3448093831 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc01406b4d0 0x7fc01406d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc0301e2200 tx=0x7fc020008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.102+0000 7fc0267fc640 1 -- 172.21.15.1:0/3448093831 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc02c0583c0 con 0x7fc0300bf9d0 2024-03-06T07:25:15.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.284+0000 7fc037ec9640 1 -- 172.21.15.1:0/3448093831 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc0300b06f0 con 0x7fc0300bf9d0 2024-03-06T07:25:15.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.286+0000 7fc0267fc640 1 -- 172.21.15.1:0/3448093831 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v123) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc02c057cc0 con 0x7fc0300bf9d0 2024-03-06T07:25:15.288 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:25:15.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.288+0000 7fc037ec9640 1 -- 172.21.15.1:0/3448093831 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc01406b4d0 msgr2=0x7fc01406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.288+0000 7fc037ec9640 1 --2- 172.21.15.1:0/3448093831 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc01406b4d0 0x7fc01406d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc0301e2200 tx=0x7fc020008040 comp rx=0 tx=0).stop 2024-03-06T07:25:15.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.288+0000 7fc037ec9640 1 -- 172.21.15.1:0/3448093831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0300bf9d0 msgr2=0x7fc0301eb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.288+0000 7fc037ec9640 1 --2- 172.21.15.1:0/3448093831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0300bf9d0 0x7fc0301eb450 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fc02c00c4d0 tx=0x7fc02c00c990 comp rx=0 tx=0).stop 2024-03-06T07:25:15.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.288+0000 7fc037ec9640 1 -- 172.21.15.1:0/3448093831 shutdown_connections 2024-03-06T07:25:15.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.288+0000 7fc037ec9640 1 --2- 172.21.15.1:0/3448093831 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc01406b4d0 0x7fc01406d990 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.288+0000 7fc037ec9640 1 --2- 172.21.15.1:0/3448093831 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc03014b4e0 0x7fc030152a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.288+0000 7fc037ec9640 1 --2- 172.21.15.1:0/3448093831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc030152650 0x7fc0300c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.288+0000 7fc037ec9640 1 --2- 172.21.15.1:0/3448093831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0300bf9d0 0x7fc0301eb450 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.288+0000 7fc037ec9640 1 -- 172.21.15.1:0/3448093831 >> 172.21.15.1:0/3448093831 conn(0x7fc030066ee0 msgr2=0x7fc0300672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.288+0000 7fc037ec9640 1 -- 172.21.15.1:0/3448093831 shutdown_connections 2024-03-06T07:25:15.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.288+0000 7fc037ec9640 1 -- 172.21.15.1:0/3448093831 wait complete. 2024-03-06T07:25:15.301 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:15.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.400+0000 7f59dfc37640 1 Processor -- start 2024-03-06T07:25:15.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.401+0000 7f59dfc37640 1 -- start start 2024-03-06T07:25:15.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.401+0000 7f59dfc37640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f59d8152650 0x7f59d8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.401+0000 7f59dfc37640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59d80c6b10 0x7f59d8158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.401+0000 7f59dfc37640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d80c2fc0 con 0x7f59d80bf9d0 2024-03-06T07:25:15.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.401+0000 7f59dfc37640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d8155800 con 0x7f59d8152650 2024-03-06T07:25:15.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.401+0000 7f59dfc37640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80c6690 con 0x7f59d80c6b10 2024-03-06T07:25:15.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.401+0000 7f59dd9d3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f59d8152650 0x7f59d8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.401+0000 7f59de1d4640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f59d80bf9d0 0x7f59d80c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39394/0 (socket says 172.21.15.1:39394) 2024-03-06T07:25:15.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.401+0000 7f59de1d4640 1 -- 172.21.15.1:0/3174926861 learned_addr learned my addr 172.21.15.1:0/3174926861 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:15.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.402+0000 7f59dd1d2640 1 --2- 172.21.15.1:0/3174926861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59d80c6b10 0x7f59d8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.402+0000 7f59dc9d1640 1 -- 172.21.15.1:0/3174926861 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782992911 0 0) 0x7f59d80c2fc0 con 0x7f59d80bf9d0 2024-03-06T07:25:15.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.402+0000 7f59dc9d1640 1 -- 172.21.15.1:0/3174926861 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59c4003520 con 0x7f59d80bf9d0 2024-03-06T07:25:15.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.402+0000 7f59dd9d3640 1 -- 172.21.15.1:0/3174926861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59d80c6b10 msgr2=0x7f59d8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.402+0000 7f59dd9d3640 1 --2- 172.21.15.1:0/3174926861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59d80c6b10 0x7f59d8158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.402+0000 7f59dd9d3640 1 -- 172.21.15.1:0/3174926861 >> v1:172.21.15.1:6789/0 conn(0x7f59d80bf9d0 legacy=0x7f59d80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.402+0000 7f59dd9d3640 1 -- 172.21.15.1:0/3174926861 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d80c7280 con 0x7f59d8152650 2024-03-06T07:25:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.403+0000 7f59dd9d3640 1 --2- 172.21.15.1:0/3174926861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59d8152650 0x7f59d8155c90 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f59cc0097a0 tx=0x7f59cc02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf7665418e34ee4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.403+0000 7f59dc9d1640 1 -- 172.21.15.1:0/3174926861 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc031780 con 0x7f59d8152650 2024-03-06T07:25:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.403+0000 7f59dc9d1640 1 -- 172.21.15.1:0/3174926861 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59cc037070 con 0x7f59d8152650 2024-03-06T07:25:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.403+0000 7f59dc9d1640 1 -- 172.21.15.1:0/3174926861 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc031b40 con 0x7f59d8152650 2024-03-06T07:25:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.403+0000 7f59dfc37640 1 -- 172.21.15.1:0/3174926861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59d8152650 msgr2=0x7f59d8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.403+0000 7f59dfc37640 1 --2- 172.21.15.1:0/3174926861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59d8152650 0x7f59d8155c90 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f59cc0097a0 tx=0x7f59cc02b820 comp rx=0 tx=0).stop 2024-03-06T07:25:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.403+0000 7f59dfc37640 1 -- 172.21.15.1:0/3174926861 shutdown_connections 2024-03-06T07:25:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.403+0000 7f59dfc37640 1 --2- 172.21.15.1:0/3174926861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59d80c6b10 0x7f59d8158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.403+0000 7f59dfc37640 1 --2- 172.21.15.1:0/3174926861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59d8152650 0x7f59d8155c90 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.403+0000 7f59dfc37640 1 -- 172.21.15.1:0/3174926861 >> 172.21.15.1:0/3174926861 conn(0x7f59d8066ee0 msgr2=0x7f59d80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.403+0000 7f59dfc37640 1 -- 172.21.15.1:0/3174926861 shutdown_connections 2024-03-06T07:25:15.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.403+0000 7f59dfc37640 1 -- 172.21.15.1:0/3174926861 wait complete. 2024-03-06T07:25:15.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.404+0000 7f59dfc37640 1 Processor -- start 2024-03-06T07:25:15.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.404+0000 7f59dfc37640 1 -- start start 2024-03-06T07:25:15.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.405+0000 7f59dfc37640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59d80bf9d0 0x7f59d81df0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.405+0000 7f59dfc37640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59d8152650 0x7f59d8153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.405+0000 7f59dfc37640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59d81531b0 0x7f59d8144300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.405+0000 7f59dfc37640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d8155490 con 0x7f59d80bf9d0 2024-03-06T07:25:15.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.405+0000 7f59dfc37640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d81551b0 con 0x7f59d8152650 2024-03-06T07:25:15.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.405+0000 7f59dfc37640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d8152a30 con 0x7f59d81531b0 2024-03-06T07:25:15.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.405+0000 7f59dd9d3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59d80bf9d0 0x7f59d81df0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.405+0000 7f59dd9d3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59d80bf9d0 0x7f59d81df0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34824/0 (socket says 172.21.15.1:34824) 2024-03-06T07:25:15.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.405+0000 7f59dd9d3640 1 -- 172.21.15.1:0/605528295 learned_addr learned my addr 172.21.15.1:0/605528295 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:15.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.406+0000 7f59dd9d3640 1 -- 172.21.15.1:0/605528295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59d81531b0 msgr2=0x7f59d8144300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.406+0000 7f59de1d4640 1 --2- 172.21.15.1:0/605528295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59d81531b0 0x7f59d8144300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.406+0000 7f59dd1d2640 1 --2- 172.21.15.1:0/605528295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59d8152650 0x7f59d8153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.406+0000 7f59dd9d3640 1 --2- 172.21.15.1:0/605528295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59d81531b0 0x7f59d8144300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.406+0000 7f59dd9d3640 1 -- 172.21.15.1:0/605528295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59d8152650 msgr2=0x7f59d8153670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.406+0000 7f59dd9d3640 1 --2- 172.21.15.1:0/605528295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59d8152650 0x7f59d8153670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.406+0000 7f59dd9d3640 1 -- 172.21.15.1:0/605528295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d81e0600 con 0x7f59d80bf9d0 2024-03-06T07:25:15.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.406+0000 7f59dd9d3640 1 --2- 172.21.15.1:0/605528295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59d80bf9d0 0x7f59d81df0d0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f59cc0098c0 tx=0x7f59cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.406+0000 7f59ca7fc640 1 -- 172.21.15.1:0/605528295 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc032610 con 0x7f59d80bf9d0 2024-03-06T07:25:15.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.406+0000 7f59ca7fc640 1 -- 172.21.15.1:0/605528295 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59cc0327d0 con 0x7f59d80bf9d0 2024-03-06T07:25:15.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.406+0000 7f59ca7fc640 1 -- 172.21.15.1:0/605528295 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc032bc0 con 0x7f59d80bf9d0 2024-03-06T07:25:15.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.406+0000 7f59dfc37640 1 -- 172.21.15.1:0/605528295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d81e1230 con 0x7f59d80bf9d0 2024-03-06T07:25:15.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.406+0000 7f59dfc37640 1 -- 172.21.15.1:0/605528295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d80c9280 con 0x7f59d80bf9d0 2024-03-06T07:25:15.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.407+0000 7f59dfc37640 1 -- 172.21.15.1:0/605528295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d8067fd0 con 0x7f59d80bf9d0 2024-03-06T07:25:15.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.407+0000 7f59ca7fc640 1 -- 172.21.15.1:0/605528295 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f59cc04e440 con 0x7f59d80bf9d0 2024-03-06T07:25:15.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.408+0000 7f59ca7fc640 1 --2- 172.21.15.1:0/605528295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f59bc06b500 0x7f59bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.408+0000 7f59ca7fc640 1 -- 172.21.15.1:0/605528295 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f59cc0ac040 con 0x7f59d80bf9d0 2024-03-06T07:25:15.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.408+0000 7f59dd1d2640 1 --2- 172.21.15.1:0/605528295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f59bc06b500 0x7f59bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.409+0000 7f59dd1d2640 1 --2- 172.21.15.1:0/605528295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f59bc06b500 0x7f59bc06d9c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f59d806ced0 tx=0x7f59d4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.416+0000 7f59ca7fc640 1 -- 172.21.15.1:0/605528295 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f59cc040070 con 0x7f59d80bf9d0 2024-03-06T07:25:15.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.635+0000 7f59dfc37640 1 -- 172.21.15.1:0/605528295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59d814ef50 con 0x7f59d80bf9d0 2024-03-06T07:25:15.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.635+0000 7f59ca7fc640 1 -- 172.21.15.1:0/605528295 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+4977 (secure 0 0 0) 0x7f59cc02f050 con 0x7f59d80bf9d0 2024-03-06T07:25:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:15.638 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":115,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6446,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4151599120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4151599120},{"type":"v1","addr":"172.21.15.1:6835","nonce":4151599120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/155600632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":155600632},{"type":"v1","addr":"172.21.15.98:6837","nonce":155600632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6587,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1600553134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1600553134},{"type":"v1","addr":"172.21.15.98:6836","nonce":1600553134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.598303+0000","modified":"2024-03-06T07:25:12.276319+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8091":{"gid":8091,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":4,"addr":"172.21.15.1:6834/1747678472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1747678472},{"type":"v1","addr":"172.21.15.1:6834","nonce":1747678472}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8091,"qdb_cluster":[8091]},"id":9}]} 2024-03-06T07:25:15.638 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 115 2024-03-06T07:25:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.637+0000 7f59dfc37640 1 -- 172.21.15.1:0/605528295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f59bc06b500 msgr2=0x7f59bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.637+0000 7f59dfc37640 1 --2- 172.21.15.1:0/605528295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f59bc06b500 0x7f59bc06d9c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f59d806ced0 tx=0x7f59d4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.637+0000 7f59dfc37640 1 -- 172.21.15.1:0/605528295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59d80bf9d0 msgr2=0x7f59d81df0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.637+0000 7f59dfc37640 1 --2- 172.21.15.1:0/605528295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59d80bf9d0 0x7f59d81df0d0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f59cc0098c0 tx=0x7f59cc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.637+0000 7f59dfc37640 1 -- 172.21.15.1:0/605528295 shutdown_connections 2024-03-06T07:25:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.637+0000 7f59dfc37640 1 --2- 172.21.15.1:0/605528295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f59bc06b500 0x7f59bc06d9c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.637+0000 7f59dfc37640 1 --2- 172.21.15.1:0/605528295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59d81531b0 0x7f59d8144300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.637+0000 7f59dfc37640 1 --2- 172.21.15.1:0/605528295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59d8152650 0x7f59d8153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.637+0000 7f59dfc37640 1 --2- 172.21.15.1:0/605528295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59d80bf9d0 0x7f59d81df0d0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.637+0000 7f59dfc37640 1 -- 172.21.15.1:0/605528295 >> 172.21.15.1:0/605528295 conn(0x7f59d8066ee0 msgr2=0x7f59d80c5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.637+0000 7f59dfc37640 1 -- 172.21.15.1:0/605528295 shutdown_connections 2024-03-06T07:25:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.637+0000 7f59dfc37640 1 -- 172.21.15.1:0/605528295 wait complete. 2024-03-06T07:25:15.650 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:15.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.759+0000 7f83ee01c640 1 Processor -- start 2024-03-06T07:25:15.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.759+0000 7f83ee01c640 1 -- start start 2024-03-06T07:25:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.760+0000 7f83ee01c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f83e81513c0 0x7f83e80726a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.760+0000 7f83ee01c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e8151830 con 0x7f83e8072f20 2024-03-06T07:25:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.760+0000 7f83ee01c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e80a6140 con 0x7f83e8150fe0 2024-03-06T07:25:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.760+0000 7f83ee01c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e81526b0 con 0x7f83e81513c0 2024-03-06T07:25:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.760+0000 7f83e6ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f83e8072f20 0x7f83e8071f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39400/0 (socket says 172.21.15.1:39400) 2024-03-06T07:25:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.760+0000 7f83e6ffd640 1 -- 172.21.15.1:0/512266172 learned_addr learned my addr 172.21.15.1:0/512266172 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.760+0000 7f83e7fff640 1 --2- 172.21.15.1:0/512266172 >> v2:172.21.15.98:3300/0 conn(0x7f83e81513c0 0x7f83e80726a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.760+0000 7f83e67fc640 1 -- 172.21.15.1:0/512266172 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883771521 0 0) 0x7f83e8151830 con 0x7f83e8072f20 2024-03-06T07:25:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.760+0000 7f83e67fc640 1 -- 172.21.15.1:0/512266172 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83d0003520 con 0x7f83e8072f20 2024-03-06T07:25:15.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e67fc640 1 -- 172.21.15.1:0/512266172 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411881773 0 0) 0x7f83e80a6140 con 0x7f83e8150fe0 2024-03-06T07:25:15.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e67fc640 1 -- 172.21.15.1:0/512266172 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83e8151830 con 0x7f83e8150fe0 2024-03-06T07:25:15.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e67fc640 1 -- 172.21.15.1:0/512266172 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1565338147 0 0) 0x7f83d0003520 con 0x7f83e8072f20 2024-03-06T07:25:15.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e67fc640 1 -- 172.21.15.1:0/512266172 >> v2:172.21.15.98:3300/0 conn(0x7f83e81513c0 msgr2=0x7f83e80726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e67fc640 1 --2- 172.21.15.1:0/512266172 >> v2:172.21.15.98:3300/0 conn(0x7f83e81513c0 0x7f83e80726a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e67fc640 1 -- 172.21.15.1:0/512266172 >> v1:172.21.15.98:6789/0 conn(0x7f83e8150fe0 legacy=0x7f83e806edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e67fc640 1 -- 172.21.15.1:0/512266172 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e8151aa0 con 0x7f83e8072f20 2024-03-06T07:25:15.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e67fc640 1 -- 172.21.15.1:0/512266172 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f83dc001670 con 0x7f83e8072f20 2024-03-06T07:25:15.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e67fc640 1 -- 172.21.15.1:0/512266172 >> v1:172.21.15.1:6789/0 conn(0x7f83e8072f20 legacy=0x7f83e8071f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e67fc640 1 --2- 172.21.15.1:0/512266172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83d0003b40 0x7f83d0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e67fc640 1 --2- 172.21.15.1:0/512266172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83d0004800 0x7f83d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e67fc640 1 --2- 172.21.15.1:0/512266172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d00051c0 0x7f83d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e67fc640 1 -- 172.21.15.1:0/512266172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d000c0c0 con 0x7f83d0004800 2024-03-06T07:25:15.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e67fc640 1 -- 172.21.15.1:0/512266172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d000c270 con 0x7f83d00051c0 2024-03-06T07:25:15.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e67fc640 1 -- 172.21.15.1:0/512266172 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83d000d580 con 0x7f83d0003b40 2024-03-06T07:25:15.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e6ffd640 1 --2- 172.21.15.1:0/512266172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83d0004800 0x7f83d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e77fe640 1 --2- 172.21.15.1:0/512266172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83d0003b40 0x7f83d0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e7fff640 1 --2- 172.21.15.1:0/512266172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d00051c0 0x7f83d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e6ffd640 1 -- 172.21.15.1:0/512266172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83d0003b40 msgr2=0x7f83d0003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e6ffd640 1 --2- 172.21.15.1:0/512266172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83d0003b40 0x7f83d0003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e6ffd640 1 -- 172.21.15.1:0/512266172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d00051c0 msgr2=0x7f83d000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e6ffd640 1 --2- 172.21.15.1:0/512266172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d00051c0 0x7f83d000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e7fff640 1 -- 172.21.15.1:0/512266172 reap_dead start 2024-03-06T07:25:15.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e6ffd640 1 -- 172.21.15.1:0/512266172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d0003520 con 0x7f83d0004800 2024-03-06T07:25:15.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.761+0000 7f83e77fe640 1 --2- 172.21.15.1:0/512266172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83d0003b40 0x7f83d0003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:15.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.762+0000 7f83e6ffd640 1 --2- 172.21.15.1:0/512266172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83d0004800 0x7f83d0004c80 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f83dc00c190 tx=0x7f83d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d92c1d0c59a13a5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.762+0000 7f83e67fc640 1 -- 172.21.15.1:0/512266172 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83dc007ce0 con 0x7f83d0004800 2024-03-06T07:25:15.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.763+0000 7f83e67fc640 1 -- 172.21.15.1:0/512266172 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83dc030070 con 0x7f83d0004800 2024-03-06T07:25:15.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.763+0000 7f83e67fc640 1 -- 172.21.15.1:0/512266172 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83dc005d90 con 0x7f83d0004800 2024-03-06T07:25:15.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.763+0000 7f83ee01c640 1 -- 172.21.15.1:0/512266172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83d0004800 msgr2=0x7f83d0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.763+0000 7f83ee01c640 1 --2- 172.21.15.1:0/512266172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83d0004800 0x7f83d0004c80 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f83dc00c190 tx=0x7f83d0005830 comp rx=0 tx=0).stop 2024-03-06T07:25:15.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.763+0000 7f83ee01c640 1 -- 172.21.15.1:0/512266172 shutdown_connections 2024-03-06T07:25:15.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.763+0000 7f83ee01c640 1 --2- 172.21.15.1:0/512266172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83d0004800 0x7f83d0004c80 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.763+0000 7f83ee01c640 1 -- 172.21.15.1:0/512266172 >> 172.21.15.1:0/512266172 conn(0x7f83e80c6860 msgr2=0x7f83e80c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.763+0000 7f83ee01c640 1 -- 172.21.15.1:0/512266172 shutdown_connections 2024-03-06T07:25:15.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.763+0000 7f83ee01c640 1 -- 172.21.15.1:0/512266172 wait complete. 2024-03-06T07:25:15.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.765+0000 7f83ee01c640 1 Processor -- start 2024-03-06T07:25:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.765+0000 7f83ee01c640 1 -- start start 2024-03-06T07:25:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.766+0000 7f83ee01c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83d0004800 0x7f83e81eacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.766+0000 7f83ee01c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83e814b120 0x7f83e81e7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.766+0000 7f83ee01c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83e8153d10 0x7f83e8146c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.766+0000 7f83ee01c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e8152340 con 0x7f83e8153d10 2024-03-06T07:25:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.766+0000 7f83ee01c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e8152060 con 0x7f83e814b120 2024-03-06T07:25:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.766+0000 7f83ee01c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e806edb0 con 0x7f83d0004800 2024-03-06T07:25:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.766+0000 7f83e7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83e8153d10 0x7f83e8146c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.766+0000 7f83e7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83e8153d10 0x7f83e8146c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34840/0 (socket says 172.21.15.1:34840) 2024-03-06T07:25:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.766+0000 7f83e7fff640 1 -- 172.21.15.1:0/832414250 learned_addr learned my addr 172.21.15.1:0/832414250 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.766+0000 7f83e77fe640 1 --2- 172.21.15.1:0/832414250 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83d0004800 0x7f83e81eacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.766+0000 7f83e6ffd640 1 --2- 172.21.15.1:0/832414250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83e814b120 0x7f83e81e7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.767+0000 7f83e7fff640 1 -- 172.21.15.1:0/832414250 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83d0004800 msgr2=0x7f83e81eacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.767+0000 7f83e7fff640 1 --2- 172.21.15.1:0/832414250 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83d0004800 0x7f83e81eacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.767+0000 7f83e7fff640 1 -- 172.21.15.1:0/832414250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83e814b120 msgr2=0x7f83e81e7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.767+0000 7f83e7fff640 1 --2- 172.21.15.1:0/832414250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83e814b120 0x7f83e81e7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.767+0000 7f83e7fff640 1 -- 172.21.15.1:0/832414250 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e81e7ab0 con 0x7f83e8153d10 2024-03-06T07:25:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.767+0000 7f83e77fe640 1 --2- 172.21.15.1:0/832414250 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83d0004800 0x7f83e81eacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.767+0000 7f83e6ffd640 1 --2- 172.21.15.1:0/832414250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83e814b120 0x7f83e81e7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.767+0000 7f83e7fff640 1 --2- 172.21.15.1:0/832414250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83e8153d10 0x7f83e8146c70 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f83e806eb30 tx=0x7f83e00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.767+0000 7f83cffff640 1 -- 172.21.15.1:0/832414250 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e0011db0 con 0x7f83e8153d10 2024-03-06T07:25:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.767+0000 7f83cffff640 1 -- 172.21.15.1:0/832414250 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e001b070 con 0x7f83e8153d10 2024-03-06T07:25:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.767+0000 7f83ee01c640 1 -- 172.21.15.1:0/832414250 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e814b650 con 0x7f83e8153d10 2024-03-06T07:25:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.767+0000 7f83cffff640 1 -- 172.21.15.1:0/832414250 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e0003990 con 0x7f83e8153d10 2024-03-06T07:25:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.767+0000 7f83ee01c640 1 -- 172.21.15.1:0/832414250 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e81ea4c0 con 0x7f83e8153d10 2024-03-06T07:25:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.768+0000 7f83ee01c640 1 -- 172.21.15.1:0/832414250 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e8067fd0 con 0x7f83e8153d10 2024-03-06T07:25:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.768+0000 7f83cffff640 1 -- 172.21.15.1:0/832414250 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83e0005d10 con 0x7f83e8153d10 2024-03-06T07:25:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.769+0000 7f83cffff640 1 --2- 172.21.15.1:0/832414250 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f83d406b4d0 0x7f83d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.769+0000 7f83cffff640 1 -- 172.21.15.1:0/832414250 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f83e008a970 con 0x7f83e8153d10 2024-03-06T07:25:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.769+0000 7f83e77fe640 1 --2- 172.21.15.1:0/832414250 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f83d406b4d0 0x7f83d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.769+0000 7f83e77fe640 1 --2- 172.21.15.1:0/832414250 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f83d406b4d0 0x7f83d406d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f83e80505a0 tx=0x7f83d8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.773+0000 7f83cffff640 1 -- 172.21.15.1:0/832414250 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83e0025020 con 0x7f83e8153d10 2024-03-06T07:25:15.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.954+0000 7f83ee01c640 1 -- 172.21.15.1:0/832414250 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83e80a60f0 con 0x7f83e8153d10 2024-03-06T07:25:15.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.955+0000 7f83cffff640 1 -- 172.21.15.1:0/832414250 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) v1 ==== 74+0+17037 (secure 0 0 0) 0x7f83e0014350 con 0x7f83e8153d10 2024-03-06T07:25:15.957 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:15.957 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":123,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:25:09.267130+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:50.865151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:51.564469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6836/1420541659":"2024-03-07T07:25:09.267103+0000","172.21.15.98:6835/1420541659":"2024-03-07T07:25:09.267103+0000","172.21.15.1:6834/1515309674":"2024-03-07T07:25:07.207409+0000","172.21.15.1:6832/1515309674":"2024-03-07T07:25:07.207409+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:15.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.957+0000 7f83ee01c640 1 -- 172.21.15.1:0/832414250 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f83d406b4d0 msgr2=0x7f83d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.957+0000 7f83ee01c640 1 --2- 172.21.15.1:0/832414250 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f83d406b4d0 0x7f83d406d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f83e80505a0 tx=0x7f83d8009040 comp rx=0 tx=0).stop 2024-03-06T07:25:15.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.957+0000 7f83ee01c640 1 -- 172.21.15.1:0/832414250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83e8153d10 msgr2=0x7f83e8146c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.957+0000 7f83ee01c640 1 --2- 172.21.15.1:0/832414250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83e8153d10 0x7f83e8146c70 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f83e806eb30 tx=0x7f83e00115a0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.957+0000 7f83ee01c640 1 -- 172.21.15.1:0/832414250 shutdown_connections 2024-03-06T07:25:15.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.957+0000 7f83ee01c640 1 --2- 172.21.15.1:0/832414250 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f83d406b4d0 0x7f83d406d990 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.957+0000 7f83ee01c640 1 --2- 172.21.15.1:0/832414250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83e8153d10 0x7f83e8146c70 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.957+0000 7f83ee01c640 1 --2- 172.21.15.1:0/832414250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83e814b120 0x7f83e81e7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.957+0000 7f83ee01c640 1 --2- 172.21.15.1:0/832414250 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83d0004800 0x7f83e81eacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.957+0000 7f83ee01c640 1 -- 172.21.15.1:0/832414250 >> 172.21.15.1:0/832414250 conn(0x7f83e80c6860 msgr2=0x7f83e80c6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.958+0000 7f83ee01c640 1 -- 172.21.15.1:0/832414250 shutdown_connections 2024-03-06T07:25:15.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:15.958+0000 7f83ee01c640 1 -- 172.21.15.1:0/832414250 wait complete. 2024-03-06T07:25:15.971 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:25:15.971 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:16.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.070+0000 7f8d770a4640 1 Processor -- start 2024-03-06T07:25:16.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.070+0000 7f8d770a4640 1 -- start start 2024-03-06T07:25:16.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.071+0000 7f8d770a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d700c2b30 0x7f8d700724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.071+0000 7f8d770a4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8d70152b70 0x7f8d70072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.071+0000 7f8d770a4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d70151dd0 con 0x7f8d700a5fa0 2024-03-06T07:25:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.071+0000 7f8d770a4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d701526e0 con 0x7f8d70152b70 2024-03-06T07:25:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.071+0000 7f8d770a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d70156190 con 0x7f8d700c2b30 2024-03-06T07:25:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.071+0000 7f8d74e40640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d700c2b30 0x7f8d700724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.071+0000 7f8d6ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8d70152b70 0x7f8d70072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.071+0000 7f8d6ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8d70152b70 0x7f8d70072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:53942/0 (socket says 172.21.15.1:53942) 2024-03-06T07:25:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.071+0000 7f8d74e40640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d700c2b30 0x7f8d700724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:35056/0 (socket says 172.21.15.1:35056) 2024-03-06T07:25:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.071+0000 7f8d75641640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8d700a5fa0 0x7f8d700c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:57966/0 (socket says 172.21.15.1:57966) 2024-03-06T07:25:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.071+0000 7f8d6ffff640 1 -- 172.21.15.1:0/2231298433 learned_addr learned my addr 172.21.15.1:0/2231298433 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.071+0000 7f8d6ffff640 1 -- 172.21.15.1:0/2231298433 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d700c2b30 msgr2=0x7f8d700724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.071+0000 7f8d6ffff640 1 --2- 172.21.15.1:0/2231298433 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d700c2b30 0x7f8d700724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.071+0000 7f8d6ffff640 1 -- 172.21.15.1:0/2231298433 >> v1:172.21.15.98:6789/0 conn(0x7f8d700a5fa0 legacy=0x7f8d700c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.071+0000 7f8d6f7fe640 1 -- 172.21.15.1:0/2231298433 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782923719 0 0) 0x7f8d70151dd0 con 0x7f8d700a5fa0 2024-03-06T07:25:16.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.071+0000 7f8d6ffff640 1 -- 172.21.15.1:0/2231298433 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d70152090 con 0x7f8d70152b70 2024-03-06T07:25:16.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.072+0000 7f8d6ffff640 1 --2- 172.21.15.1:0/2231298433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d70152b70 0x7f8d70072a00 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c009870 tx=0x7f8d5c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5154781d3be32b0d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.072+0000 7f8d6f7fe640 1 -- 172.21.15.1:0/2231298433 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c034070 con 0x7f8d70152b70 2024-03-06T07:25:16.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.072+0000 7f8d6f7fe640 1 -- 172.21.15.1:0/2231298433 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5c030840 con 0x7f8d70152b70 2024-03-06T07:25:16.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.072+0000 7f8d6f7fe640 1 -- 172.21.15.1:0/2231298433 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c030c10 con 0x7f8d70152b70 2024-03-06T07:25:16.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.073+0000 7f8d770a4640 1 -- 172.21.15.1:0/2231298433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d70152b70 msgr2=0x7f8d70072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.073+0000 7f8d770a4640 1 --2- 172.21.15.1:0/2231298433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d70152b70 0x7f8d70072a00 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c009870 tx=0x7f8d5c02b980 comp rx=0 tx=0).stop 2024-03-06T07:25:16.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.073+0000 7f8d770a4640 1 -- 172.21.15.1:0/2231298433 shutdown_connections 2024-03-06T07:25:16.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.073+0000 7f8d770a4640 1 --2- 172.21.15.1:0/2231298433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d70152b70 0x7f8d70072a00 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.073+0000 7f8d770a4640 1 --2- 172.21.15.1:0/2231298433 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d700c2b30 0x7f8d700724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.073+0000 7f8d770a4640 1 -- 172.21.15.1:0/2231298433 >> 172.21.15.1:0/2231298433 conn(0x7f8d700c6860 msgr2=0x7f8d700c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:16.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.073+0000 7f8d770a4640 1 -- 172.21.15.1:0/2231298433 shutdown_connections 2024-03-06T07:25:16.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.073+0000 7f8d770a4640 1 -- 172.21.15.1:0/2231298433 wait complete. 2024-03-06T07:25:16.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.074+0000 7f8d770a4640 1 Processor -- start 2024-03-06T07:25:16.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.074+0000 7f8d770a4640 1 -- start start 2024-03-06T07:25:16.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d770a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d700a5fa0 0x7f8d7013fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d770a4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d700c2b30 0x7f8d700bfc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d770a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d70152b70 0x7f8d701eaf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d770a4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d70155f00 con 0x7f8d700c2b30 2024-03-06T07:25:16.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d770a4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d70155d80 con 0x7f8d700a5fa0 2024-03-06T07:25:16.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d770a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d700c3a00 con 0x7f8d70152b70 2024-03-06T07:25:16.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d6ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d700c2b30 0x7f8d700bfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d6ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d700c2b30 0x7f8d700bfc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34846/0 (socket says 172.21.15.1:34846) 2024-03-06T07:25:16.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d6ffff640 1 -- 172.21.15.1:0/3965555830 learned_addr learned my addr 172.21.15.1:0/3965555830 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:16.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d75641640 1 --2- 172.21.15.1:0/3965555830 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d70152b70 0x7f8d701eaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d74e40640 1 --2- 172.21.15.1:0/3965555830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d700a5fa0 0x7f8d7013fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d6ffff640 1 -- 172.21.15.1:0/3965555830 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d70152b70 msgr2=0x7f8d701eaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d6ffff640 1 --2- 172.21.15.1:0/3965555830 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d70152b70 0x7f8d701eaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d6ffff640 1 -- 172.21.15.1:0/3965555830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d700a5fa0 msgr2=0x7f8d7013fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d6ffff640 1 --2- 172.21.15.1:0/3965555830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d700a5fa0 0x7f8d7013fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d6ffff640 1 -- 172.21.15.1:0/3965555830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7014c410 con 0x7f8d700c2b30 2024-03-06T07:25:16.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d74e40640 1 --2- 172.21.15.1:0/3965555830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d700a5fa0 0x7f8d7013fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:16.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.075+0000 7f8d6ffff640 1 --2- 172.21.15.1:0/3965555830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d700c2b30 0x7f8d700bfc90 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c000c00 tx=0x7f8d5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.076+0000 7f8d6d7fa640 1 -- 172.21.15.1:0/3965555830 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c034040 con 0x7f8d700c2b30 2024-03-06T07:25:16.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.076+0000 7f8d6d7fa640 1 -- 172.21.15.1:0/3965555830 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5c03ab90 con 0x7f8d700c2b30 2024-03-06T07:25:16.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.076+0000 7f8d770a4640 1 -- 172.21.15.1:0/3965555830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7014c130 con 0x7f8d700c2b30 2024-03-06T07:25:16.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.076+0000 7f8d6d7fa640 1 -- 172.21.15.1:0/3965555830 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c004030 con 0x7f8d700c2b30 2024-03-06T07:25:16.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.076+0000 7f8d770a4640 1 -- 172.21.15.1:0/3965555830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d701ef230 con 0x7f8d700c2b30 2024-03-06T07:25:16.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.077+0000 7f8d770a4640 1 -- 172.21.15.1:0/3965555830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d70067fd0 con 0x7f8d700c2b30 2024-03-06T07:25:16.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.077+0000 7f8d6d7fa640 1 -- 172.21.15.1:0/3965555830 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d5c04b060 con 0x7f8d700c2b30 2024-03-06T07:25:16.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.078+0000 7f8d6d7fa640 1 --2- 172.21.15.1:0/3965555830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d5406b4d0 0x7f8d5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.078+0000 7f8d6d7fa640 1 -- 172.21.15.1:0/3965555830 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f8d5c0ac110 con 0x7f8d700c2b30 2024-03-06T07:25:16.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.078+0000 7f8d74e40640 1 --2- 172.21.15.1:0/3965555830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d5406b4d0 0x7f8d5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.078+0000 7f8d74e40640 1 --2- 172.21.15.1:0/3965555830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d5406b4d0 0x7f8d5406d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f8d701eee80 tx=0x7f8d64008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.086+0000 7f8d6d7fa640 1 -- 172.21.15.1:0/3965555830 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d5c03f070 con 0x7f8d700c2b30 2024-03-06T07:25:16.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.307+0000 7f8d770a4640 1 -- 172.21.15.1:0/3965555830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d700a5d10 con 0x7f8d700c2b30 2024-03-06T07:25:16.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.307+0000 7f8d6d7fa640 1 -- 172.21.15.1:0/3965555830 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+4977 (secure 0 0 0) 0x7f8d5c07af00 con 0x7f8d700c2b30 2024-03-06T07:25:16.310 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:16.310 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":115,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6446,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4151599120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4151599120},{"type":"v1","addr":"172.21.15.1:6835","nonce":4151599120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/155600632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":155600632},{"type":"v1","addr":"172.21.15.98:6837","nonce":155600632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6587,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1600553134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1600553134},{"type":"v1","addr":"172.21.15.98:6836","nonce":1600553134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.598303+0000","modified":"2024-03-06T07:25:12.276319+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8091":{"gid":8091,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":4,"addr":"172.21.15.1:6834/1747678472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1747678472},{"type":"v1","addr":"172.21.15.1:6834","nonce":1747678472}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8091,"qdb_cluster":[8091]},"id":9}]} 2024-03-06T07:25:16.310 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 115 2024-03-06T07:25:16.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.310+0000 7f8d770a4640 1 -- 172.21.15.1:0/3965555830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d5406b4d0 msgr2=0x7f8d5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.310+0000 7f8d770a4640 1 --2- 172.21.15.1:0/3965555830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d5406b4d0 0x7f8d5406d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f8d701eee80 tx=0x7f8d64008040 comp rx=0 tx=0).stop 2024-03-06T07:25:16.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.310+0000 7f8d770a4640 1 -- 172.21.15.1:0/3965555830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d700c2b30 msgr2=0x7f8d700bfc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.310+0000 7f8d770a4640 1 --2- 172.21.15.1:0/3965555830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d700c2b30 0x7f8d700bfc90 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c000c00 tx=0x7f8d5c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.310+0000 7f8d770a4640 1 -- 172.21.15.1:0/3965555830 shutdown_connections 2024-03-06T07:25:16.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.310+0000 7f8d770a4640 1 --2- 172.21.15.1:0/3965555830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d5406b4d0 0x7f8d5406d990 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.310+0000 7f8d770a4640 1 --2- 172.21.15.1:0/3965555830 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d70152b70 0x7f8d701eaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.310+0000 7f8d770a4640 1 --2- 172.21.15.1:0/3965555830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d700c2b30 0x7f8d700bfc90 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.310+0000 7f8d770a4640 1 --2- 172.21.15.1:0/3965555830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d700a5fa0 0x7f8d7013fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.310+0000 7f8d770a4640 1 -- 172.21.15.1:0/3965555830 >> 172.21.15.1:0/3965555830 conn(0x7f8d700c6860 msgr2=0x7f8d70072810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:16.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.310+0000 7f8d770a4640 1 -- 172.21.15.1:0/3965555830 shutdown_connections 2024-03-06T07:25:16.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.310+0000 7f8d770a4640 1 -- 172.21.15.1:0/3965555830 wait complete. 2024-03-06T07:25:16.323 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:16.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.422+0000 7fc476325640 1 Processor -- start 2024-03-06T07:25:16.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.423+0000 7fc476325640 1 -- start start 2024-03-06T07:25:16.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.423+0000 7fc476325640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc4700c6b10 0x7fc470155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.423+0000 7fc476325640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4700bf9d0 0x7fc4700c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.423+0000 7fc476325640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4700c2fc0 con 0x7fc470152650 2024-03-06T07:25:16.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.423+0000 7fc476325640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc470155800 con 0x7fc4700c6b10 2024-03-06T07:25:16.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.423+0000 7fc476325640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc470158da0 con 0x7fc4700bf9d0 2024-03-06T07:25:16.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.423+0000 7fc46f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc4700c6b10 0x7fc470155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.423+0000 7fc46ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc470152650 0x7fc470158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39404/0 (socket says 172.21.15.1:39404) 2024-03-06T07:25:16.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.423+0000 7fc46ffff640 1 -- 172.21.15.1:0/759734761 learned_addr learned my addr 172.21.15.1:0/759734761 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:16.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.424+0000 7fc4748c2640 1 --2- 172.21.15.1:0/759734761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4700bf9d0 0x7fc4700c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.424+0000 7fc46f7fe640 1 -- 172.21.15.1:0/759734761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4700bf9d0 msgr2=0x7fc4700c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:16.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.424+0000 7fc46f7fe640 1 --2- 172.21.15.1:0/759734761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4700bf9d0 0x7fc4700c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.424+0000 7fc46f7fe640 1 -- 172.21.15.1:0/759734761 >> v1:172.21.15.1:6789/0 conn(0x7fc470152650 legacy=0x7fc470158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.424+0000 7fc46f7fe640 1 -- 172.21.15.1:0/759734761 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4700c6690 con 0x7fc4700c6b10 2024-03-06T07:25:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.424+0000 7fc46f7fe640 1 --2- 172.21.15.1:0/759734761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4700c6b10 0x7fc470155c90 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fc4580097a0 tx=0x7fc45802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793c95c18c271309 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.425+0000 7fc46effd640 1 -- 172.21.15.1:0/759734761 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4580315a0 con 0x7fc4700c6b10 2024-03-06T07:25:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.425+0000 7fc46effd640 1 -- 172.21.15.1:0/759734761 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc458037070 con 0x7fc4700c6b10 2024-03-06T07:25:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.425+0000 7fc46effd640 1 -- 172.21.15.1:0/759734761 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458031960 con 0x7fc4700c6b10 2024-03-06T07:25:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.425+0000 7fc476325640 1 -- 172.21.15.1:0/759734761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4700c6b10 msgr2=0x7fc470155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.425+0000 7fc476325640 1 --2- 172.21.15.1:0/759734761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4700c6b10 0x7fc470155c90 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fc4580097a0 tx=0x7fc45802b840 comp rx=0 tx=0).stop 2024-03-06T07:25:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.425+0000 7fc476325640 1 -- 172.21.15.1:0/759734761 shutdown_connections 2024-03-06T07:25:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.425+0000 7fc476325640 1 --2- 172.21.15.1:0/759734761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4700bf9d0 0x7fc4700c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.425+0000 7fc476325640 1 --2- 172.21.15.1:0/759734761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4700c6b10 0x7fc470155c90 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.425+0000 7fc476325640 1 -- 172.21.15.1:0/759734761 >> 172.21.15.1:0/759734761 conn(0x7fc470066ee0 msgr2=0x7fc4700672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:16.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.425+0000 7fc476325640 1 -- 172.21.15.1:0/759734761 shutdown_connections 2024-03-06T07:25:16.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.425+0000 7fc476325640 1 -- 172.21.15.1:0/759734761 wait complete. 2024-03-06T07:25:16.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.426+0000 7fc476325640 1 Processor -- start 2024-03-06T07:25:16.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.427+0000 7fc476325640 1 -- start start 2024-03-06T07:25:16.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.427+0000 7fc476325640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4700bf9d0 0x7fc4701e1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.427+0000 7fc476325640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc470152650 0x7fc47014b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.427+0000 7fc476325640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470154c30 0x7fc47014b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.427+0000 7fc476325640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4701551b0 con 0x7fc470152650 2024-03-06T07:25:16.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.427+0000 7fc476325640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4700c6fa0 con 0x7fc470154c30 2024-03-06T07:25:16.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.427+0000 7fc476325640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc470155490 con 0x7fc4700bf9d0 2024-03-06T07:25:16.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.427+0000 7fc46f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc470152650 0x7fc47014b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.427+0000 7fc46f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc470152650 0x7fc47014b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34874/0 (socket says 172.21.15.1:34874) 2024-03-06T07:25:16.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.427+0000 7fc46f7fe640 1 -- 172.21.15.1:0/3438884965 learned_addr learned my addr 172.21.15.1:0/3438884965 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:16.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.427+0000 7fc4748c2640 1 --2- 172.21.15.1:0/3438884965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470154c30 0x7fc47014b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.427+0000 7fc46ffff640 1 --2- 172.21.15.1:0/3438884965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4700bf9d0 0x7fc4701e1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.428+0000 7fc46f7fe640 1 -- 172.21.15.1:0/3438884965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4700bf9d0 msgr2=0x7fc4701e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.428+0000 7fc46f7fe640 1 --2- 172.21.15.1:0/3438884965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4700bf9d0 0x7fc4701e1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.428+0000 7fc46f7fe640 1 -- 172.21.15.1:0/3438884965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470154c30 msgr2=0x7fc47014b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.428+0000 7fc46f7fe640 1 --2- 172.21.15.1:0/3438884965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470154c30 0x7fc47014b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.428+0000 7fc46f7fe640 1 -- 172.21.15.1:0/3438884965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc470154240 con 0x7fc470152650 2024-03-06T07:25:16.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.428+0000 7fc4748c2640 1 --2- 172.21.15.1:0/3438884965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470154c30 0x7fc47014b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:16.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.428+0000 7fc46f7fe640 1 --2- 172.21.15.1:0/3438884965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc470152650 0x7fc47014b0c0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fc458034040 tx=0x7fc4580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.428+0000 7fc46cff9640 1 -- 172.21.15.1:0/3438884965 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458032840 con 0x7fc470152650 2024-03-06T07:25:16.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.428+0000 7fc46cff9640 1 -- 172.21.15.1:0/3438884965 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc458032a00 con 0x7fc470152650 2024-03-06T07:25:16.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.428+0000 7fc476325640 1 -- 172.21.15.1:0/3438884965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc470156630 con 0x7fc470152650 2024-03-06T07:25:16.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.428+0000 7fc46cff9640 1 -- 172.21.15.1:0/3438884965 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458032df0 con 0x7fc470152650 2024-03-06T07:25:16.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.428+0000 7fc476325640 1 -- 172.21.15.1:0/3438884965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4701e24f0 con 0x7fc470152650 2024-03-06T07:25:16.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.429+0000 7fc476325640 1 -- 172.21.15.1:0/3438884965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc470067fd0 con 0x7fc470152650 2024-03-06T07:25:16.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.430+0000 7fc46cff9640 1 -- 172.21.15.1:0/3438884965 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc45804e430 con 0x7fc470152650 2024-03-06T07:25:16.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.430+0000 7fc46cff9640 1 --2- 172.21.15.1:0/3438884965 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc45006b4d0 0x7fc45006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.430+0000 7fc46cff9640 1 -- 172.21.15.1:0/3438884965 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fc45802f030 con 0x7fc470152650 2024-03-06T07:25:16.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.430+0000 7fc46ffff640 1 --2- 172.21.15.1:0/3438884965 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc45006b4d0 0x7fc45006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.431+0000 7fc46ffff640 1 --2- 172.21.15.1:0/3438884965 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc45006b4d0 0x7fc45006d990 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc460000f60 tx=0x7fc460009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.434+0000 7fc46cff9640 1 -- 172.21.15.1:0/3438884965 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc458040070 con 0x7fc470152650 2024-03-06T07:25:16.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.670+0000 7fc476325640 1 -- 172.21.15.1:0/3438884965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc470156800 con 0x7fc470152650 2024-03-06T07:25:16.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.670+0000 7fc46cff9640 1 -- 172.21.15.1:0/3438884965 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+4977 (secure 0 0 0) 0x7fc45807afd0 con 0x7fc470152650 2024-03-06T07:25:16.673 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:16.673 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":115,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6446,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4151599120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4151599120},{"type":"v1","addr":"172.21.15.1:6835","nonce":4151599120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/155600632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":155600632},{"type":"v1","addr":"172.21.15.98:6837","nonce":155600632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6587,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1600553134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1600553134},{"type":"v1","addr":"172.21.15.98:6836","nonce":1600553134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.598303+0000","modified":"2024-03-06T07:25:12.276319+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8091":{"gid":8091,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":4,"addr":"172.21.15.1:6834/1747678472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1747678472},{"type":"v1","addr":"172.21.15.1:6834","nonce":1747678472}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8091,"qdb_cluster":[8091]},"id":9}]} 2024-03-06T07:25:16.673 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 115 2024-03-06T07:25:16.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.672+0000 7fc476325640 1 -- 172.21.15.1:0/3438884965 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc45006b4d0 msgr2=0x7fc45006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.672+0000 7fc476325640 1 --2- 172.21.15.1:0/3438884965 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc45006b4d0 0x7fc45006d990 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc460000f60 tx=0x7fc460009040 comp rx=0 tx=0).stop 2024-03-06T07:25:16.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.672+0000 7fc476325640 1 -- 172.21.15.1:0/3438884965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc470152650 msgr2=0x7fc47014b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.672+0000 7fc476325640 1 --2- 172.21.15.1:0/3438884965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc470152650 0x7fc47014b0c0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fc458034040 tx=0x7fc4580046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.672+0000 7fc476325640 1 -- 172.21.15.1:0/3438884965 shutdown_connections 2024-03-06T07:25:16.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.672+0000 7fc476325640 1 --2- 172.21.15.1:0/3438884965 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc45006b4d0 0x7fc45006d990 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.672+0000 7fc476325640 1 --2- 172.21.15.1:0/3438884965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470154c30 0x7fc47014b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.672+0000 7fc476325640 1 --2- 172.21.15.1:0/3438884965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc470152650 0x7fc47014b0c0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.672+0000 7fc476325640 1 --2- 172.21.15.1:0/3438884965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4700bf9d0 0x7fc4701e1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.672+0000 7fc476325640 1 -- 172.21.15.1:0/3438884965 >> 172.21.15.1:0/3438884965 conn(0x7fc470066ee0 msgr2=0x7fc4700c3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:16.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.673+0000 7fc476325640 1 -- 172.21.15.1:0/3438884965 shutdown_connections 2024-03-06T07:25:16.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.673+0000 7fc476325640 1 -- 172.21.15.1:0/3438884965 wait complete. 2024-03-06T07:25:16.685 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:16.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.784+0000 7f911eab5640 1 Processor -- start 2024-03-06T07:25:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.784+0000 7f911eab5640 1 -- start start 2024-03-06T07:25:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.785+0000 7f911eab5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f91180c6b10 0x7f9118155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.785+0000 7f911eab5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91180bf9d0 0x7f91180c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.785+0000 7f911eab5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91180c2fc0 con 0x7f9118152650 2024-03-06T07:25:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.785+0000 7f911eab5640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9118155800 con 0x7f91180c6b10 2024-03-06T07:25:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.785+0000 7f911eab5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9118158da0 con 0x7f91180bf9d0 2024-03-06T07:25:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.785+0000 7f9117fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f91180c6b10 0x7f9118155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.785+0000 7f911c851640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9118152650 0x7f9118158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39406/0 (socket says 172.21.15.1:39406) 2024-03-06T07:25:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.785+0000 7f911c851640 1 -- 172.21.15.1:0/1083974411 learned_addr learned my addr 172.21.15.1:0/1083974411 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.785+0000 7f911d052640 1 --2- 172.21.15.1:0/1083974411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91180bf9d0 0x7f91180c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.785+0000 7f9117fff640 1 -- 172.21.15.1:0/1083974411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91180bf9d0 msgr2=0x7f91180c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:16.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.785+0000 7f9117fff640 1 --2- 172.21.15.1:0/1083974411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91180bf9d0 0x7f91180c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.785+0000 7f9117fff640 1 -- 172.21.15.1:0/1083974411 >> v1:172.21.15.1:6789/0 conn(0x7f9118152650 legacy=0x7f9118158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.785+0000 7f9117fff640 1 -- 172.21.15.1:0/1083974411 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91180c6690 con 0x7f91180c6b10 2024-03-06T07:25:16.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.785+0000 7f91177fe640 1 -- 172.21.15.1:0/1083974411 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662024140 0 0) 0x7f91180c2fc0 con 0x7f9118152650 2024-03-06T07:25:16.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.786+0000 7f9117fff640 1 --2- 172.21.15.1:0/1083974411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91180c6b10 0x7f9118155c90 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f91040097a0 tx=0x7f910402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d86de964d2125c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.786+0000 7f91177fe640 1 -- 172.21.15.1:0/1083974411 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91040315a0 con 0x7f91180c6b10 2024-03-06T07:25:16.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.786+0000 7f91177fe640 1 -- 172.21.15.1:0/1083974411 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9104037070 con 0x7f91180c6b10 2024-03-06T07:25:16.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.786+0000 7f91177fe640 1 -- 172.21.15.1:0/1083974411 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9104031960 con 0x7f91180c6b10 2024-03-06T07:25:16.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.787+0000 7f911eab5640 1 -- 172.21.15.1:0/1083974411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91180c6b10 msgr2=0x7f9118155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.787+0000 7f911eab5640 1 --2- 172.21.15.1:0/1083974411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91180c6b10 0x7f9118155c90 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f91040097a0 tx=0x7f910402b840 comp rx=0 tx=0).stop 2024-03-06T07:25:16.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.787+0000 7f911eab5640 1 -- 172.21.15.1:0/1083974411 shutdown_connections 2024-03-06T07:25:16.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.787+0000 7f911eab5640 1 --2- 172.21.15.1:0/1083974411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91180bf9d0 0x7f91180c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.787+0000 7f911eab5640 1 --2- 172.21.15.1:0/1083974411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91180c6b10 0x7f9118155c90 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.787+0000 7f911eab5640 1 -- 172.21.15.1:0/1083974411 >> 172.21.15.1:0/1083974411 conn(0x7f9118066ee0 msgr2=0x7f91180672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:16.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.787+0000 7f911eab5640 1 -- 172.21.15.1:0/1083974411 shutdown_connections 2024-03-06T07:25:16.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.787+0000 7f911eab5640 1 -- 172.21.15.1:0/1083974411 wait complete. 2024-03-06T07:25:16.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.788+0000 7f911eab5640 1 Processor -- start 2024-03-06T07:25:16.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.788+0000 7f911eab5640 1 -- start start 2024-03-06T07:25:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.789+0000 7f911eab5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91180bf9d0 0x7f91181eedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.789+0000 7f911eab5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9118152650 0x7f91180c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.789+0000 7f911eab5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9118151190 0x7f91181569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.789+0000 7f911eab5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91181551b0 con 0x7f9118151190 2024-03-06T07:25:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.789+0000 7f911eab5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91180c6fa0 con 0x7f91180bf9d0 2024-03-06T07:25:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.789+0000 7f911eab5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9118155490 con 0x7f9118152650 2024-03-06T07:25:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.789+0000 7f911d052640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9118151190 0x7f91181569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.789+0000 7f911d052640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9118151190 0x7f91181569a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34884/0 (socket says 172.21.15.1:34884) 2024-03-06T07:25:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.789+0000 7f911d052640 1 -- 172.21.15.1:0/4112886404 learned_addr learned my addr 172.21.15.1:0/4112886404 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.789+0000 7f9117fff640 1 --2- 172.21.15.1:0/4112886404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9118152650 0x7f91180c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.790+0000 7f911c851640 1 --2- 172.21.15.1:0/4112886404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91180bf9d0 0x7f91181eedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.790+0000 7f911d052640 1 -- 172.21.15.1:0/4112886404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9118152650 msgr2=0x7f91180c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.790+0000 7f911d052640 1 --2- 172.21.15.1:0/4112886404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9118152650 0x7f91180c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.790+0000 7f911d052640 1 -- 172.21.15.1:0/4112886404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91180bf9d0 msgr2=0x7f91181eedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.790+0000 7f911d052640 1 --2- 172.21.15.1:0/4112886404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91180bf9d0 0x7f91181eedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.790+0000 7f911d052640 1 -- 172.21.15.1:0/4112886404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91180ca800 con 0x7f9118151190 2024-03-06T07:25:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.790+0000 7f9117fff640 1 --2- 172.21.15.1:0/4112886404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9118152650 0x7f91180c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.790+0000 7f911d052640 1 --2- 172.21.15.1:0/4112886404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9118151190 0x7f91181569a0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f9110007e60 tx=0x7f911000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.790+0000 7f91157fa640 1 -- 172.21.15.1:0/4112886404 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9110014420 con 0x7f9118151190 2024-03-06T07:25:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.790+0000 7f91157fa640 1 -- 172.21.15.1:0/4112886404 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9110019070 con 0x7f9118151190 2024-03-06T07:25:16.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.790+0000 7f911eab5640 1 -- 172.21.15.1:0/4112886404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9118151650 con 0x7f9118151190 2024-03-06T07:25:16.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.790+0000 7f91157fa640 1 -- 172.21.15.1:0/4112886404 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9110014820 con 0x7f9118151190 2024-03-06T07:25:16.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.790+0000 7f911eab5640 1 -- 172.21.15.1:0/4112886404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91181eb2b0 con 0x7f9118151190 2024-03-06T07:25:16.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.791+0000 7f911eab5640 1 -- 172.21.15.1:0/4112886404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9118067fd0 con 0x7f9118151190 2024-03-06T07:25:16.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.792+0000 7f91157fa640 1 -- 172.21.15.1:0/4112886404 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91100149c0 con 0x7f9118151190 2024-03-06T07:25:16.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.792+0000 7f91157fa640 1 --2- 172.21.15.1:0/4112886404 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90fc06b500 0x7f90fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.792+0000 7f91157fa640 1 -- 172.21.15.1:0/4112886404 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f911008ad10 con 0x7f9118151190 2024-03-06T07:25:16.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.793+0000 7f911c851640 1 --2- 172.21.15.1:0/4112886404 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90fc06b500 0x7f90fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.793+0000 7f911c851640 1 --2- 172.21.15.1:0/4112886404 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90fc06b500 0x7f90fc06d9c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f910c000f60 tx=0x7f910c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.797+0000 7f91157fa640 1 -- 172.21.15.1:0/4112886404 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9110090020 con 0x7f9118151190 2024-03-06T07:25:16.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.979+0000 7f911eab5640 1 -- 172.21.15.1:0/4112886404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91180b06f0 con 0x7f9118151190 2024-03-06T07:25:16.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.980+0000 7f91157fa640 1 -- 172.21.15.1:0/4112886404 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) v1 ==== 74+0+17037 (secure 0 0 0) 0x7f91100598b0 con 0x7f9118151190 2024-03-06T07:25:16.983 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:16.983 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":123,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:25:09.267130+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:50.865151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:51.564469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6836/1420541659":"2024-03-07T07:25:09.267103+0000","172.21.15.98:6835/1420541659":"2024-03-07T07:25:09.267103+0000","172.21.15.1:6834/1515309674":"2024-03-07T07:25:07.207409+0000","172.21.15.1:6832/1515309674":"2024-03-07T07:25:07.207409+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.983+0000 7f911eab5640 1 -- 172.21.15.1:0/4112886404 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90fc06b500 msgr2=0x7f90fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.983+0000 7f911eab5640 1 --2- 172.21.15.1:0/4112886404 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90fc06b500 0x7f90fc06d9c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f910c000f60 tx=0x7f910c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.983+0000 7f911eab5640 1 -- 172.21.15.1:0/4112886404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9118151190 msgr2=0x7f91181569a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.983+0000 7f911eab5640 1 --2- 172.21.15.1:0/4112886404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9118151190 0x7f91181569a0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f9110007e60 tx=0x7f911000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.983+0000 7f911eab5640 1 -- 172.21.15.1:0/4112886404 shutdown_connections 2024-03-06T07:25:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.983+0000 7f911eab5640 1 --2- 172.21.15.1:0/4112886404 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90fc06b500 0x7f90fc06d9c0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.983+0000 7f911eab5640 1 --2- 172.21.15.1:0/4112886404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9118151190 0x7f91181569a0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.983+0000 7f911eab5640 1 --2- 172.21.15.1:0/4112886404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9118152650 0x7f91180c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.983+0000 7f911eab5640 1 --2- 172.21.15.1:0/4112886404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91180bf9d0 0x7f91181eedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.983+0000 7f911eab5640 1 -- 172.21.15.1:0/4112886404 >> 172.21.15.1:0/4112886404 conn(0x7f9118066ee0 msgr2=0x7f91180c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.983+0000 7f911eab5640 1 -- 172.21.15.1:0/4112886404 shutdown_connections 2024-03-06T07:25:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:16.983+0000 7f911eab5640 1 -- 172.21.15.1:0/4112886404 wait complete. 2024-03-06T07:25:16.996 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:25:17.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.096+0000 7f6f5cb55640 1 Processor -- start 2024-03-06T07:25:17.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.096+0000 7f6f5cb55640 1 -- start start 2024-03-06T07:25:17.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.097+0000 7f6f5cb55640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f580c2b30 0x7f6f580724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f5cb55640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6f580a5fa0 0x7f6f58072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f5cb55640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f58151dd0 con 0x7f6f58152b70 2024-03-06T07:25:17.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f5cb55640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f581526e0 con 0x7f6f580a5fa0 2024-03-06T07:25:17.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f5cb55640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f58156190 con 0x7f6f580c2b30 2024-03-06T07:25:17.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f4ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6f58152b70 0x7f6f580c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39422/0 (socket says 172.21.15.1:39422) 2024-03-06T07:25:17.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f4ffff640 1 -- 172.21.15.1:0/1693600776 learned_addr learned my addr 172.21.15.1:0/1693600776 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:17.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f5659c640 1 --2- 172.21.15.1:0/1693600776 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f580c2b30 0x7f6f580724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f56d9d640 1 --2- 172.21.15.1:0/1693600776 >> v2:172.21.15.98:3300/0 conn(0x7f6f580a5fa0 0x7f6f58072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f55d9b640 1 -- 172.21.15.1:0/1693600776 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123577726 0 0) 0x7f6f58151dd0 con 0x7f6f58152b70 2024-03-06T07:25:17.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f55d9b640 1 -- 172.21.15.1:0/1693600776 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f44003520 con 0x7f6f58152b70 2024-03-06T07:25:17.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f55d9b640 1 -- 172.21.15.1:0/1693600776 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1343099768 0 0) 0x7f6f44003520 con 0x7f6f58152b70 2024-03-06T07:25:17.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f55d9b640 1 -- 172.21.15.1:0/1693600776 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f580c2b30 msgr2=0x7f6f580724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:17.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f55d9b640 1 --2- 172.21.15.1:0/1693600776 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f580c2b30 0x7f6f580724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f55d9b640 1 -- 172.21.15.1:0/1693600776 >> v2:172.21.15.98:3300/0 conn(0x7f6f580a5fa0 msgr2=0x7f6f58072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:17.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f55d9b640 1 --2- 172.21.15.1:0/1693600776 >> v2:172.21.15.98:3300/0 conn(0x7f6f580a5fa0 0x7f6f58072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f55d9b640 1 -- 172.21.15.1:0/1693600776 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f58152090 con 0x7f6f58152b70 2024-03-06T07:25:17.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f55d9b640 1 -- 172.21.15.1:0/1693600776 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f6f40001670 con 0x7f6f58152b70 2024-03-06T07:25:17.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f55d9b640 1 -- 172.21.15.1:0/1693600776 >> v1:172.21.15.1:6789/0 conn(0x7f6f58152b70 legacy=0x7f6f580c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f55d9b640 1 --2- 172.21.15.1:0/1693600776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f440039b0 0x7f6f44003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f55d9b640 1 --2- 172.21.15.1:0/1693600776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f44004800 0x7f6f44004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f55d9b640 1 --2- 172.21.15.1:0/1693600776 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f580c2b30 0x7f6f440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f55d9b640 1 -- 172.21.15.1:0/1693600776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f581526e0 con 0x7f6f44004800 2024-03-06T07:25:17.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f55d9b640 1 -- 172.21.15.1:0/1693600776 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f58156190 con 0x7f6f440039b0 2024-03-06T07:25:17.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.098+0000 7f6f55d9b640 1 -- 172.21.15.1:0/1693600776 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4400dbd0 con 0x7f6f580c2b30 2024-03-06T07:25:17.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.099+0000 7f6f4ffff640 1 --2- 172.21.15.1:0/1693600776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f44004800 0x7f6f44004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.099+0000 7f6f5659c640 1 --2- 172.21.15.1:0/1693600776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f440039b0 0x7f6f44003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.099+0000 7f6f56d9d640 1 --2- 172.21.15.1:0/1693600776 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f580c2b30 0x7f6f440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.099+0000 7f6f4ffff640 1 -- 172.21.15.1:0/1693600776 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f580c2b30 msgr2=0x7f6f440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:17.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.099+0000 7f6f4ffff640 1 --2- 172.21.15.1:0/1693600776 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f580c2b30 0x7f6f440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.099+0000 7f6f4ffff640 1 -- 172.21.15.1:0/1693600776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f440039b0 msgr2=0x7f6f44003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:17.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.099+0000 7f6f4ffff640 1 --2- 172.21.15.1:0/1693600776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f440039b0 0x7f6f44003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.099+0000 7f6f4ffff640 1 -- 172.21.15.1:0/1693600776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f44003520 con 0x7f6f44004800 2024-03-06T07:25:17.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.100+0000 7f6f4ffff640 1 --2- 172.21.15.1:0/1693600776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f44004800 0x7f6f44004c80 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000c210 tx=0x7f6f44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19fffdbe82d830e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.100+0000 7f6f55d9b640 1 -- 172.21.15.1:0/1693600776 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40007d50 con 0x7f6f44004800 2024-03-06T07:25:17.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.100+0000 7f6f55d9b640 1 -- 172.21.15.1:0/1693600776 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f40030070 con 0x7f6f44004800 2024-03-06T07:25:17.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.100+0000 7f6f55d9b640 1 -- 172.21.15.1:0/1693600776 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4002cda0 con 0x7f6f44004800 2024-03-06T07:25:17.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.100+0000 7f6f5cb55640 1 -- 172.21.15.1:0/1693600776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f44004800 msgr2=0x7f6f44004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.100+0000 7f6f5cb55640 1 --2- 172.21.15.1:0/1693600776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f44004800 0x7f6f44004c80 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000c210 tx=0x7f6f44005830 comp rx=0 tx=0).stop 2024-03-06T07:25:17.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.100+0000 7f6f56d9d640 1 -- 172.21.15.1:0/1693600776 reap_dead start 2024-03-06T07:25:17.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.100+0000 7f6f5cb55640 1 -- 172.21.15.1:0/1693600776 shutdown_connections 2024-03-06T07:25:17.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.100+0000 7f6f5cb55640 1 -- 172.21.15.1:0/1693600776 >> 172.21.15.1:0/1693600776 conn(0x7f6f580c6860 msgr2=0x7f6f580c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:17.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.101+0000 7f6f5cb55640 1 -- 172.21.15.1:0/1693600776 shutdown_connections 2024-03-06T07:25:17.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.101+0000 7f6f5cb55640 1 -- 172.21.15.1:0/1693600776 wait complete. 2024-03-06T07:25:17.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.102+0000 7f6f5cb55640 1 Processor -- start 2024-03-06T07:25:17.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.102+0000 7f6f5cb55640 1 -- start start 2024-03-06T07:25:17.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.102+0000 7f6f5cb55640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f44004800 0x7f6f5806eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.102+0000 7f6f5cb55640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f58149b90 0x7f6f58149080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.102+0000 7f6f5cb55640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f5814a0c0 0x7f6f581495c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.102+0000 7f6f5cb55640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f58155f00 con 0x7f6f44004800 2024-03-06T07:25:17.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.102+0000 7f6f5cb55640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f58155d80 con 0x7f6f58149b90 2024-03-06T07:25:17.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.102+0000 7f6f5cb55640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f580724c0 con 0x7f6f5814a0c0 2024-03-06T07:25:17.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.102+0000 7f6f5659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f44004800 0x7f6f5806eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.103+0000 7f6f5659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f44004800 0x7f6f5806eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34896/0 (socket says 172.21.15.1:34896) 2024-03-06T07:25:17.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.103+0000 7f6f5659c640 1 -- 172.21.15.1:0/1090981089 learned_addr learned my addr 172.21.15.1:0/1090981089 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:17.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.103+0000 7f6f56d9d640 1 --2- 172.21.15.1:0/1090981089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f5814a0c0 0x7f6f581495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.103+0000 7f6f4ffff640 1 --2- 172.21.15.1:0/1090981089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f58149b90 0x7f6f58149080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.103+0000 7f6f5659c640 1 -- 172.21.15.1:0/1090981089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f5814a0c0 msgr2=0x7f6f581495c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.103+0000 7f6f5659c640 1 --2- 172.21.15.1:0/1090981089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f5814a0c0 0x7f6f581495c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.103+0000 7f6f5659c640 1 -- 172.21.15.1:0/1090981089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f58149b90 msgr2=0x7f6f58149080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.103+0000 7f6f5659c640 1 --2- 172.21.15.1:0/1090981089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f58149b90 0x7f6f58149080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.103+0000 7f6f5659c640 1 -- 172.21.15.1:0/1090981089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f580bd960 con 0x7f6f44004800 2024-03-06T07:25:17.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.103+0000 7f6f5659c640 1 --2- 172.21.15.1:0/1090981089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f44004800 0x7f6f5806eb60 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f6f48011910 tx=0x7f6f48011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.103+0000 7f6f4effd640 1 -- 172.21.15.1:0/1090981089 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4800f9d0 con 0x7f6f44004800 2024-03-06T07:25:17.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.103+0000 7f6f4effd640 1 -- 172.21.15.1:0/1090981089 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4800fb70 con 0x7f6f44004800 2024-03-06T07:25:17.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.103+0000 7f6f5cb55640 1 -- 172.21.15.1:0/1090981089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f580bd330 con 0x7f6f44004800 2024-03-06T07:25:17.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.103+0000 7f6f4effd640 1 -- 172.21.15.1:0/1090981089 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48014440 con 0x7f6f44004800 2024-03-06T07:25:17.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.103+0000 7f6f5cb55640 1 -- 172.21.15.1:0/1090981089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f580bd5f0 con 0x7f6f44004800 2024-03-06T07:25:17.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.104+0000 7f6f5cb55640 1 -- 172.21.15.1:0/1090981089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f58067fd0 con 0x7f6f44004800 2024-03-06T07:25:17.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.105+0000 7f6f4effd640 1 -- 172.21.15.1:0/1090981089 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f48012040 con 0x7f6f44004800 2024-03-06T07:25:17.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.105+0000 7f6f4effd640 1 --2- 172.21.15.1:0/1090981089 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f3806b500 0x7f6f3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.105+0000 7f6f4effd640 1 -- 172.21.15.1:0/1090981089 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f6f4808a710 con 0x7f6f44004800 2024-03-06T07:25:17.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.106+0000 7f6f4ffff640 1 --2- 172.21.15.1:0/1090981089 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f3806b500 0x7f6f3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.106+0000 7f6f4ffff640 1 --2- 172.21.15.1:0/1090981089 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f3806b500 0x7f6f3806d9c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6f40034000 tx=0x7f6f40007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.112+0000 7f6f4effd640 1 -- 172.21.15.1:0/1090981089 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f48019030 con 0x7f6f44004800 2024-03-06T07:25:17.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.355+0000 7f6f5cb55640 1 -- 172.21.15.1:0/1090981089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6f58071790 con 0x7f6f44004800 2024-03-06T07:25:17.357 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:25:17.356+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:25:17.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.357+0000 7f6f4effd640 1 -- 172.21.15.1:0/1090981089 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v116) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6f48088e20 con 0x7f6f44004800 2024-03-06T07:25:17.359 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:25:17.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.360+0000 7f6f5cb55640 1 -- 172.21.15.1:0/1090981089 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f3806b500 msgr2=0x7f6f3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.360+0000 7f6f5cb55640 1 --2- 172.21.15.1:0/1090981089 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f3806b500 0x7f6f3806d9c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6f40034000 tx=0x7f6f40007000 comp rx=0 tx=0).stop 2024-03-06T07:25:17.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.360+0000 7f6f5cb55640 1 -- 172.21.15.1:0/1090981089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f44004800 msgr2=0x7f6f5806eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.360+0000 7f6f5cb55640 1 --2- 172.21.15.1:0/1090981089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f44004800 0x7f6f5806eb60 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f6f48011910 tx=0x7f6f48011dd0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.360+0000 7f6f5cb55640 1 -- 172.21.15.1:0/1090981089 shutdown_connections 2024-03-06T07:25:17.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.360+0000 7f6f5cb55640 1 --2- 172.21.15.1:0/1090981089 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6f3806b500 0x7f6f3806d9c0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.360+0000 7f6f5cb55640 1 --2- 172.21.15.1:0/1090981089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6f5814a0c0 0x7f6f581495c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.360+0000 7f6f5cb55640 1 --2- 172.21.15.1:0/1090981089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f58149b90 0x7f6f58149080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.360+0000 7f6f5cb55640 1 --2- 172.21.15.1:0/1090981089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f44004800 0x7f6f5806eb60 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.360+0000 7f6f5cb55640 1 -- 172.21.15.1:0/1090981089 >> 172.21.15.1:0/1090981089 conn(0x7f6f580c6860 msgr2=0x7f6f580c5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:17.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.360+0000 7f6f5cb55640 1 -- 172.21.15.1:0/1090981089 shutdown_connections 2024-03-06T07:25:17.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.360+0000 7f6f5cb55640 1 -- 172.21.15.1:0/1090981089 wait complete. 2024-03-06T07:25:17.363 INFO:tasks.ceph.mds.c.smithi001.stderr: -200> 2024-03-06T07:25:13.222+0000 7f9ada22e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:17.373 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:25:17.387 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:25:17.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.475+0000 7fc64f125640 1 Processor -- start 2024-03-06T07:25:17.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.475+0000 7fc64f125640 1 -- start start 2024-03-06T07:25:17.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.475+0000 7fc64f125640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc6480c6730 0x7fc6480c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.475+0000 7fc64f125640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6480c3300 con 0x7fc648152690 2024-03-06T07:25:17.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.475+0000 7fc64f125640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6480c35a0 con 0x7fc6480c5d10 2024-03-06T07:25:17.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc64f125640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6480c3880 con 0x7fc6480c6730 2024-03-06T07:25:17.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc64cec1640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc648152690 0x7fc648155ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39436/0 (socket says 172.21.15.1:39436) 2024-03-06T07:25:17.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc64cec1640 1 -- 172.21.15.1:0/3302633391 learned_addr learned my addr 172.21.15.1:0/3302633391 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:17.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63ffff640 1 --2- 172.21.15.1:0/3302633391 >> v2:172.21.15.98:3300/0 conn(0x7fc6480c6730 0x7fc6480c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63f7fe640 1 -- 172.21.15.1:0/3302633391 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137590147 0 0) 0x7fc6480c3300 con 0x7fc648152690 2024-03-06T07:25:17.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63f7fe640 1 -- 172.21.15.1:0/3302633391 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc630003520 con 0x7fc648152690 2024-03-06T07:25:17.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63f7fe640 1 -- 172.21.15.1:0/3302633391 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3091843769 0 0) 0x7fc630003520 con 0x7fc648152690 2024-03-06T07:25:17.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63f7fe640 1 -- 172.21.15.1:0/3302633391 >> v2:172.21.15.98:3300/0 conn(0x7fc6480c6730 msgr2=0x7fc6480c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:17.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63f7fe640 1 --2- 172.21.15.1:0/3302633391 >> v2:172.21.15.98:3300/0 conn(0x7fc6480c6730 0x7fc6480c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63f7fe640 1 -- 172.21.15.1:0/3302633391 >> v1:172.21.15.98:6789/0 conn(0x7fc6480c5d10 legacy=0x7fc648156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63f7fe640 1 -- 172.21.15.1:0/3302633391 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6480c3b60 con 0x7fc648152690 2024-03-06T07:25:17.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63f7fe640 1 -- 172.21.15.1:0/3302633391 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fc644001690 con 0x7fc648152690 2024-03-06T07:25:17.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63ffff640 1 --2- 172.21.15.1:0/3302633391 >> v2:172.21.15.98:3300/0 conn(0x7fc6480c6730 0x7fc6480c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:17.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63f7fe640 1 -- 172.21.15.1:0/3302633391 >> v1:172.21.15.1:6789/0 conn(0x7fc648152690 legacy=0x7fc648155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63f7fe640 1 --2- 172.21.15.1:0/3302633391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6300039b0 0x7fc630003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63f7fe640 1 --2- 172.21.15.1:0/3302633391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc630004800 0x7fc630004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63f7fe640 1 --2- 172.21.15.1:0/3302633391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300051c0 0x7fc63000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63f7fe640 1 -- 172.21.15.1:0/3302633391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6480c3880 con 0x7fc6300051c0 2024-03-06T07:25:17.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63f7fe640 1 -- 172.21.15.1:0/3302633391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63000c160 con 0x7fc6300039b0 2024-03-06T07:25:17.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63f7fe640 1 -- 172.21.15.1:0/3302633391 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63000c360 con 0x7fc630004800 2024-03-06T07:25:17.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc64d6c2640 1 --2- 172.21.15.1:0/3302633391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300051c0 0x7fc63000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc63ffff640 1 --2- 172.21.15.1:0/3302633391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc630004800 0x7fc630004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.476+0000 7fc64cec1640 1 --2- 172.21.15.1:0/3302633391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6300039b0 0x7fc630003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.477+0000 7fc64d6c2640 1 -- 172.21.15.1:0/3302633391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc630004800 msgr2=0x7fc630004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:17.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.477+0000 7fc64d6c2640 1 --2- 172.21.15.1:0/3302633391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc630004800 0x7fc630004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.477+0000 7fc64d6c2640 1 -- 172.21.15.1:0/3302633391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6300039b0 msgr2=0x7fc630003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.477+0000 7fc64d6c2640 1 --2- 172.21.15.1:0/3302633391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6300039b0 0x7fc630003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.477+0000 7fc64d6c2640 1 -- 172.21.15.1:0/3302633391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc630003520 con 0x7fc6300051c0 2024-03-06T07:25:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.477+0000 7fc64d6c2640 1 -- 172.21.15.1:0/3302633391 reap_dead start 2024-03-06T07:25:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.478+0000 7fc64d6c2640 1 --2- 172.21.15.1:0/3302633391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300051c0 0x7fc63000b9c0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fc640001e50 tx=0x7fc630005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7614c2f541c5bcd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.478+0000 7fc63f7fe640 1 -- 172.21.15.1:0/3302633391 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc640001a70 con 0x7fc6300051c0 2024-03-06T07:25:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.478+0000 7fc63f7fe640 1 -- 172.21.15.1:0/3302633391 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc640003bb0 con 0x7fc6300051c0 2024-03-06T07:25:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.478+0000 7fc63f7fe640 1 -- 172.21.15.1:0/3302633391 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6400050d0 con 0x7fc6300051c0 2024-03-06T07:25:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.478+0000 7fc64f125640 1 -- 172.21.15.1:0/3302633391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300051c0 msgr2=0x7fc63000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.478+0000 7fc64f125640 1 --2- 172.21.15.1:0/3302633391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300051c0 0x7fc63000b9c0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fc640001e50 tx=0x7fc630005830 comp rx=0 tx=0).stop 2024-03-06T07:25:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.478+0000 7fc64f125640 1 -- 172.21.15.1:0/3302633391 shutdown_connections 2024-03-06T07:25:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.478+0000 7fc64f125640 1 --2- 172.21.15.1:0/3302633391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300051c0 0x7fc63000b9c0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.478+0000 7fc64f125640 1 -- 172.21.15.1:0/3302633391 >> 172.21.15.1:0/3302633391 conn(0x7fc64806b410 msgr2=0x7fc64806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.478+0000 7fc64f125640 1 -- 172.21.15.1:0/3302633391 shutdown_connections 2024-03-06T07:25:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.478+0000 7fc64f125640 1 -- 172.21.15.1:0/3302633391 wait complete. 2024-03-06T07:25:17.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.479+0000 7fc64f125640 1 Processor -- start 2024-03-06T07:25:17.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.479+0000 7fc64f125640 1 -- start start 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.480+0000 7fc64f125640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6300051c0 0x7fc6481eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.480+0000 7fc64f125640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc648099cc0 0x7fc648150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.480+0000 7fc64f125640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc648150bb0 0x7fc6480ac910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.480+0000 7fc64f125640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc648152bf0 con 0x7fc648150bb0 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.480+0000 7fc64f125640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc648152a70 con 0x7fc648099cc0 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.480+0000 7fc64f125640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6480c57d0 con 0x7fc6300051c0 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.480+0000 7fc64d6c2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc648150bb0 0x7fc6480ac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.480+0000 7fc64d6c2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc648150bb0 0x7fc6480ac910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34926/0 (socket says 172.21.15.1:34926) 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.480+0000 7fc64d6c2640 1 -- 172.21.15.1:0/401518477 learned_addr learned my addr 172.21.15.1:0/401518477 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.480+0000 7fc64cec1640 1 --2- 172.21.15.1:0/401518477 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6300051c0 0x7fc6481eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.480+0000 7fc63ffff640 1 --2- 172.21.15.1:0/401518477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc648099cc0 0x7fc648150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.481+0000 7fc64d6c2640 1 -- 172.21.15.1:0/401518477 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6300051c0 msgr2=0x7fc6481eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.481+0000 7fc64d6c2640 1 --2- 172.21.15.1:0/401518477 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6300051c0 0x7fc6481eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.481+0000 7fc64d6c2640 1 -- 172.21.15.1:0/401518477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc648099cc0 msgr2=0x7fc648150670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.481+0000 7fc64d6c2640 1 --2- 172.21.15.1:0/401518477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc648099cc0 0x7fc648150670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.481+0000 7fc64d6c2640 1 -- 172.21.15.1:0/401518477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc648154810 con 0x7fc648150bb0 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.481+0000 7fc64cec1640 1 --2- 172.21.15.1:0/401518477 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6300051c0 0x7fc6481eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.481+0000 7fc63ffff640 1 --2- 172.21.15.1:0/401518477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc648099cc0 0x7fc648150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.481+0000 7fc64d6c2640 1 --2- 172.21.15.1:0/401518477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc648150bb0 0x7fc6480ac910 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fc640005040 tx=0x7fc6400014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.481+0000 7fc63d7fa640 1 -- 172.21.15.1:0/401518477 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc640006d60 con 0x7fc648150bb0 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.481+0000 7fc63d7fa640 1 -- 172.21.15.1:0/401518477 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc640007030 con 0x7fc648150bb0 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.481+0000 7fc64f125640 1 -- 172.21.15.1:0/401518477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc648154af0 con 0x7fc648150bb0 2024-03-06T07:25:17.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.481+0000 7fc63d7fa640 1 -- 172.21.15.1:0/401518477 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6400073e0 con 0x7fc648150bb0 2024-03-06T07:25:17.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.481+0000 7fc64f125640 1 -- 172.21.15.1:0/401518477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc648151030 con 0x7fc648150bb0 2024-03-06T07:25:17.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.482+0000 7fc64f125640 1 -- 172.21.15.1:0/401518477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc648067fd0 con 0x7fc648150bb0 2024-03-06T07:25:17.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.482+0000 7fc63d7fa640 1 -- 172.21.15.1:0/401518477 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc640007580 con 0x7fc648150bb0 2024-03-06T07:25:17.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.483+0000 7fc63d7fa640 1 --2- 172.21.15.1:0/401518477 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc63406b500 0x7fc63406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.483+0000 7fc63d7fa640 1 -- 172.21.15.1:0/401518477 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6359+0+0 (secure 0 0 0) 0x7fc6400ac120 con 0x7fc648150bb0 2024-03-06T07:25:17.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.483+0000 7fc64cec1640 1 --2- 172.21.15.1:0/401518477 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc63406b500 0x7fc63406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.483+0000 7fc64cec1640 1 --2- 172.21.15.1:0/401518477 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc63406b500 0x7fc63406d9c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc6480c34d0 tx=0x7fc644009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.487+0000 7fc63d7fa640 1 -- 172.21.15.1:0/401518477 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc640033030 con 0x7fc648150bb0 2024-03-06T07:25:17.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:17.711+0000 7fc64f125640 1 -- 172.21.15.1:0/401518477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc648070010 con 0x7fc648150bb0 2024-03-06T07:25:18.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.358+0000 7fc63d7fa640 1 -- 172.21.15.1:0/401518477 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v117) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc64007ad10 con 0x7fc648150bb0 2024-03-06T07:25:18.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.361+0000 7fc64f125640 1 -- 172.21.15.1:0/401518477 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc63406b500 msgr2=0x7fc63406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.361+0000 7fc64f125640 1 --2- 172.21.15.1:0/401518477 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc63406b500 0x7fc63406d9c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc6480c34d0 tx=0x7fc644009040 comp rx=0 tx=0).stop 2024-03-06T07:25:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.361+0000 7fc64f125640 1 -- 172.21.15.1:0/401518477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc648150bb0 msgr2=0x7fc6480ac910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.361+0000 7fc64f125640 1 --2- 172.21.15.1:0/401518477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc648150bb0 0x7fc6480ac910 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fc640005040 tx=0x7fc6400014a0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.361+0000 7fc64f125640 1 -- 172.21.15.1:0/401518477 shutdown_connections 2024-03-06T07:25:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.361+0000 7fc64f125640 1 --2- 172.21.15.1:0/401518477 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc63406b500 0x7fc63406d9c0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.361+0000 7fc64f125640 1 --2- 172.21.15.1:0/401518477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc648150bb0 0x7fc6480ac910 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.361+0000 7fc64f125640 1 --2- 172.21.15.1:0/401518477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc648099cc0 0x7fc648150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.361+0000 7fc64f125640 1 --2- 172.21.15.1:0/401518477 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6300051c0 0x7fc6481eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.361+0000 7fc64f125640 1 -- 172.21.15.1:0/401518477 >> 172.21.15.1:0/401518477 conn(0x7fc64806b410 msgr2=0x7fc64806f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.361+0000 7fc64f125640 1 -- 172.21.15.1:0/401518477 shutdown_connections 2024-03-06T07:25:18.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.361+0000 7fc64f125640 1 -- 172.21.15.1:0/401518477 wait complete. 2024-03-06T07:25:18.377 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:25:18.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.528+0000 7f56d1a2a640 1 Processor -- start 2024-03-06T07:25:18.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.528+0000 7f56d1a2a640 1 -- start start 2024-03-06T07:25:18.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.529+0000 7f56d1a2a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f56cc0c6800 0x7f56cc156a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.529+0000 7f56d1a2a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56cc156f90 0x7f56cc15fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.529+0000 7f56d1a2a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56cc158560 con 0x7f56cc158180 2024-03-06T07:25:18.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.529+0000 7f56d1a2a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56cc156210 con 0x7f56cc0c6800 2024-03-06T07:25:18.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.529+0000 7f56d1a2a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0c3ea0 con 0x7f56cc156f90 2024-03-06T07:25:18.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.529+0000 7f56caffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f56cc158180 0x7f56cc0c58e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39448/0 (socket says 172.21.15.1:39448) 2024-03-06T07:25:18.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.529+0000 7f56caffd640 1 -- 172.21.15.1:0/2567243065 learned_addr learned my addr 172.21.15.1:0/2567243065 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:18.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.529+0000 7f56cb7fe640 1 --2- 172.21.15.1:0/2567243065 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56cc156f90 0x7f56cc15fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.529+0000 7f56ca7fc640 1 --2- 172.21.15.1:0/2567243065 >> v2:172.21.15.98:3300/0 conn(0x7f56cc0c6800 0x7f56cc156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.529+0000 7f56c9ffb640 1 -- 172.21.15.1:0/2567243065 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908880057 0 0) 0x7f56cc158560 con 0x7f56cc158180 2024-03-06T07:25:18.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.529+0000 7f56c9ffb640 1 -- 172.21.15.1:0/2567243065 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56b4003520 con 0x7f56cc158180 2024-03-06T07:25:18.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.529+0000 7f56cb7fe640 1 -- 172.21.15.1:0/2567243065 >> v2:172.21.15.98:3300/0 conn(0x7f56cc0c6800 msgr2=0x7f56cc156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:18.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.529+0000 7f56cb7fe640 1 --2- 172.21.15.1:0/2567243065 >> v2:172.21.15.98:3300/0 conn(0x7f56cc0c6800 0x7f56cc156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.529+0000 7f56cb7fe640 1 -- 172.21.15.1:0/2567243065 >> v1:172.21.15.1:6789/0 conn(0x7f56cc158180 legacy=0x7f56cc0c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.529+0000 7f56cb7fe640 1 -- 172.21.15.1:0/2567243065 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc155600 con 0x7f56cc156f90 2024-03-06T07:25:18.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.530+0000 7f56cb7fe640 1 --2- 172.21.15.1:0/2567243065 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56cc156f90 0x7f56cc15fb70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f56c4009d90 tx=0x7f56c402bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=293ade150aaaa55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.530+0000 7f56c9ffb640 1 -- 172.21.15.1:0/2567243065 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4039070 con 0x7f56cc156f90 2024-03-06T07:25:18.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.530+0000 7f56c9ffb640 1 -- 172.21.15.1:0/2567243065 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c402cb30 con 0x7f56cc156f90 2024-03-06T07:25:18.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.530+0000 7f56c9ffb640 1 -- 172.21.15.1:0/2567243065 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4034440 con 0x7f56cc156f90 2024-03-06T07:25:18.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.531+0000 7f56d1a2a640 1 -- 172.21.15.1:0/2567243065 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56cc156f90 msgr2=0x7f56cc15fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.531+0000 7f56d1a2a640 1 --2- 172.21.15.1:0/2567243065 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56cc156f90 0x7f56cc15fb70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f56c4009d90 tx=0x7f56c402bd80 comp rx=0 tx=0).stop 2024-03-06T07:25:18.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.531+0000 7f56d1a2a640 1 -- 172.21.15.1:0/2567243065 shutdown_connections 2024-03-06T07:25:18.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.531+0000 7f56d1a2a640 1 --2- 172.21.15.1:0/2567243065 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56cc156f90 0x7f56cc15fb70 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.531+0000 7f56d1a2a640 1 --2- 172.21.15.1:0/2567243065 >> v2:172.21.15.98:3300/0 conn(0x7f56cc0c6800 0x7f56cc156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.531+0000 7f56d1a2a640 1 -- 172.21.15.1:0/2567243065 >> 172.21.15.1:0/2567243065 conn(0x7f56cc06b410 msgr2=0x7f56cc06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:18.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.531+0000 7f56d1a2a640 1 -- 172.21.15.1:0/2567243065 shutdown_connections 2024-03-06T07:25:18.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.531+0000 7f56d1a2a640 1 -- 172.21.15.1:0/2567243065 wait complete. 2024-03-06T07:25:18.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.532+0000 7f56d1a2a640 1 Processor -- start 2024-03-06T07:25:18.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.532+0000 7f56d1a2a640 1 -- start start 2024-03-06T07:25:18.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56d1a2a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56cc0c6800 0x7f56cc1589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56d1a2a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56cc156f90 0x7f56cc14b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56d1a2a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56cc158180 0x7f56cc14ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56d1a2a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc155ea0 con 0x7f56cc156f90 2024-03-06T07:25:18.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56d1a2a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc155bc0 con 0x7f56cc158180 2024-03-06T07:25:18.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56d1a2a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0c5ff0 con 0x7f56cc0c6800 2024-03-06T07:25:18.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56ca7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56cc156f90 0x7f56cc14b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56ca7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56cc156f90 0x7f56cc14b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34940/0 (socket says 172.21.15.1:34940) 2024-03-06T07:25:18.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56ca7fc640 1 -- 172.21.15.1:0/2084414394 learned_addr learned my addr 172.21.15.1:0/2084414394 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:18.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56caffd640 1 --2- 172.21.15.1:0/2084414394 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56cc0c6800 0x7f56cc1589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56cb7fe640 1 --2- 172.21.15.1:0/2084414394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56cc158180 0x7f56cc14ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56ca7fc640 1 -- 172.21.15.1:0/2084414394 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56cc0c6800 msgr2=0x7f56cc1589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56ca7fc640 1 --2- 172.21.15.1:0/2084414394 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56cc0c6800 0x7f56cc1589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56ca7fc640 1 -- 172.21.15.1:0/2084414394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56cc158180 msgr2=0x7f56cc14ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56ca7fc640 1 --2- 172.21.15.1:0/2084414394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56cc158180 0x7f56cc14ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56ca7fc640 1 -- 172.21.15.1:0/2084414394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc1eb3f0 con 0x7f56cc156f90 2024-03-06T07:25:18.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56cb7fe640 1 --2- 172.21.15.1:0/2084414394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56cc158180 0x7f56cc14ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:18.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.533+0000 7f56ca7fc640 1 --2- 172.21.15.1:0/2084414394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56cc156f90 0x7f56cc14b500 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f56c000e8a0 tx=0x7f56c000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.534+0000 7f56b3fff640 1 -- 172.21.15.1:0/2084414394 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c0017070 con 0x7f56cc156f90 2024-03-06T07:25:18.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.534+0000 7f56b3fff640 1 -- 172.21.15.1:0/2084414394 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c000c940 con 0x7f56cc156f90 2024-03-06T07:25:18.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.534+0000 7f56d1a2a640 1 -- 172.21.15.1:0/2084414394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56cc0a8ec0 con 0x7f56cc156f90 2024-03-06T07:25:18.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.534+0000 7f56b3fff640 1 -- 172.21.15.1:0/2084414394 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c000cd30 con 0x7f56cc156f90 2024-03-06T07:25:18.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.534+0000 7f56d1a2a640 1 -- 172.21.15.1:0/2084414394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56cc0a9480 con 0x7f56cc156f90 2024-03-06T07:25:18.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.535+0000 7f56d1a2a640 1 -- 172.21.15.1:0/2084414394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56cc0680b0 con 0x7f56cc156f90 2024-03-06T07:25:18.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.535+0000 7f56b3fff640 1 -- 172.21.15.1:0/2084414394 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56c001a070 con 0x7f56cc156f90 2024-03-06T07:25:18.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.535+0000 7f56b3fff640 1 --2- 172.21.15.1:0/2084414394 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f56b806b4d0 0x7f56b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.535+0000 7f56b3fff640 1 -- 172.21.15.1:0/2084414394 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6359+0+0 (secure 0 0 0) 0x7f56c008a630 con 0x7f56cc156f90 2024-03-06T07:25:18.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.536+0000 7f56caffd640 1 --2- 172.21.15.1:0/2084414394 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f56b806b4d0 0x7f56b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.536+0000 7f56caffd640 1 --2- 172.21.15.1:0/2084414394 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f56b806b4d0 0x7f56b806d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f56bc005880 tx=0x7f56bc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.540+0000 7f56b3fff640 1 -- 172.21.15.1:0/2084414394 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56c00592b0 con 0x7f56cc156f90 2024-03-06T07:25:18.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:18.723+0000 7f56d1a2a640 1 -- 172.21.15.1:0/2084414394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f56cc0c2df0 con 0x7f56cc156f90 2024-03-06T07:25:19.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.366+0000 7f56b3fff640 1 -- 172.21.15.1:0/2084414394 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v125) v1 ==== 181+0+0 (secure 0 0 0) 0x7f56c0058bb0 con 0x7f56cc156f90 2024-03-06T07:25:19.368 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:25:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.369+0000 7f56d1a2a640 1 -- 172.21.15.1:0/2084414394 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f56b806b4d0 msgr2=0x7f56b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.369+0000 7f56d1a2a640 1 --2- 172.21.15.1:0/2084414394 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f56b806b4d0 0x7f56b806d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f56bc005880 tx=0x7f56bc009040 comp rx=0 tx=0).stop 2024-03-06T07:25:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.369+0000 7f56d1a2a640 1 -- 172.21.15.1:0/2084414394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56cc156f90 msgr2=0x7f56cc14b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.369+0000 7f56d1a2a640 1 --2- 172.21.15.1:0/2084414394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56cc156f90 0x7f56cc14b500 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f56c000e8a0 tx=0x7f56c000ed60 comp rx=0 tx=0).stop 2024-03-06T07:25:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.369+0000 7f56d1a2a640 1 -- 172.21.15.1:0/2084414394 shutdown_connections 2024-03-06T07:25:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.369+0000 7f56d1a2a640 1 --2- 172.21.15.1:0/2084414394 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f56b806b4d0 0x7f56b806d990 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.369+0000 7f56d1a2a640 1 --2- 172.21.15.1:0/2084414394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56cc158180 0x7f56cc14ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.369+0000 7f56d1a2a640 1 --2- 172.21.15.1:0/2084414394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56cc156f90 0x7f56cc14b500 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.369+0000 7f56d1a2a640 1 --2- 172.21.15.1:0/2084414394 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56cc0c6800 0x7f56cc1589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.369+0000 7f56d1a2a640 1 -- 172.21.15.1:0/2084414394 >> 172.21.15.1:0/2084414394 conn(0x7f56cc06b410 msgr2=0x7f56cc0c58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.370+0000 7f56d1a2a640 1 -- 172.21.15.1:0/2084414394 shutdown_connections 2024-03-06T07:25:19.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.370+0000 7f56d1a2a640 1 -- 172.21.15.1:0/2084414394 wait complete. 2024-03-06T07:25:19.386 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:25:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.492+0000 7f5e64c9d640 1 Processor -- start 2024-03-06T07:25:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.492+0000 7f5e64c9d640 1 -- start start 2024-03-06T07:25:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.493+0000 7f5e64c9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5e60150780 0x7f5e6006fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.493+0000 7f5e64c9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5e600c4ce0 0x7f5e600c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.493+0000 7f5e64c9d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e60156740 con 0x7f5e6006ff50 2024-03-06T07:25:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.493+0000 7f5e64c9d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e60156a20 con 0x7f5e600c4ce0 2024-03-06T07:25:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.493+0000 7f5e64c9d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e60156d00 con 0x7f5e60150780 2024-03-06T07:25:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.493+0000 7f5e5dd9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5e600c4ce0 0x7f5e600c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.493+0000 7f5e5dd9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5e600c4ce0 0x7f5e600c68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34946/0 (socket says 172.21.15.1:34946) 2024-03-06T07:25:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.493+0000 7f5e5dd9b640 1 -- 172.21.15.1:0/3530866357 learned_addr learned my addr 172.21.15.1:0/3530866357 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.493+0000 7f5e5dd9b640 1 -- 172.21.15.1:0/3530866357 >> v2:172.21.15.98:3300/0 conn(0x7f5e60150780 msgr2=0x7f5e6006fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.493+0000 7f5e5dd9b640 1 --2- 172.21.15.1:0/3530866357 >> v2:172.21.15.98:3300/0 conn(0x7f5e60150780 0x7f5e6006fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.493+0000 7f5e5dd9b640 1 -- 172.21.15.1:0/3530866357 >> v1:172.21.15.98:6789/0 conn(0x7f5e6006ff50 legacy=0x7f5e600c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.493+0000 7f5e5dd9b640 1 -- 172.21.15.1:0/3530866357 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e60156fe0 con 0x7f5e600c4ce0 2024-03-06T07:25:19.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.494+0000 7f5e5dd9b640 1 --2- 172.21.15.1:0/3530866357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e600c4ce0 0x7f5e600c68f0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f5e600c8440 tx=0x7f5e5402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=231b4ab69d1fa390 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:19.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.494+0000 7f5e5d59a640 1 -- 172.21.15.1:0/3530866357 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5402e070 con 0x7f5e600c4ce0 2024-03-06T07:25:19.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.494+0000 7f5e5d59a640 1 -- 172.21.15.1:0/3530866357 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e54037070 con 0x7f5e600c4ce0 2024-03-06T07:25:19.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.494+0000 7f5e5d59a640 1 -- 172.21.15.1:0/3530866357 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54031a00 con 0x7f5e600c4ce0 2024-03-06T07:25:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.495+0000 7f5e64c9d640 1 -- 172.21.15.1:0/3530866357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e600c4ce0 msgr2=0x7f5e600c68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.495+0000 7f5e64c9d640 1 --2- 172.21.15.1:0/3530866357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e600c4ce0 0x7f5e600c68f0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f5e600c8440 tx=0x7f5e5402b880 comp rx=0 tx=0).stop 2024-03-06T07:25:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.495+0000 7f5e64c9d640 1 -- 172.21.15.1:0/3530866357 shutdown_connections 2024-03-06T07:25:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.495+0000 7f5e64c9d640 1 --2- 172.21.15.1:0/3530866357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e600c4ce0 0x7f5e600c68f0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.495+0000 7f5e64c9d640 1 --2- 172.21.15.1:0/3530866357 >> v2:172.21.15.98:3300/0 conn(0x7f5e60150780 0x7f5e6006fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.495+0000 7f5e64c9d640 1 -- 172.21.15.1:0/3530866357 >> 172.21.15.1:0/3530866357 conn(0x7f5e600aa6a0 msgr2=0x7f5e600aaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.495+0000 7f5e64c9d640 1 -- 172.21.15.1:0/3530866357 shutdown_connections 2024-03-06T07:25:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.495+0000 7f5e64c9d640 1 -- 172.21.15.1:0/3530866357 wait complete. 2024-03-06T07:25:19.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.496+0000 7f5e64c9d640 1 Processor -- start 2024-03-06T07:25:19.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.496+0000 7f5e64c9d640 1 -- start start 2024-03-06T07:25:19.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.497+0000 7f5e64c9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e6006ff50 0x7f5e60148d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.497+0000 7f5e64c9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e600c4ce0 0x7f5e600c1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.497+0000 7f5e64c9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e60150780 0x7f5e60152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.497+0000 7f5e64c9d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e600ab270 con 0x7f5e600c4ce0 2024-03-06T07:25:19.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.497+0000 7f5e64c9d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e600c7540 con 0x7f5e60150780 2024-03-06T07:25:19.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.497+0000 7f5e64c9d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e600c84b0 con 0x7f5e6006ff50 2024-03-06T07:25:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.497+0000 7f5e5dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e600c4ce0 0x7f5e600c1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.497+0000 7f5e5dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e600c4ce0 0x7f5e600c1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34954/0 (socket says 172.21.15.1:34954) 2024-03-06T07:25:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.497+0000 7f5e5dd9b640 1 -- 172.21.15.1:0/197908523 learned_addr learned my addr 172.21.15.1:0/197908523 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.497+0000 7f5e5dd9b640 1 -- 172.21.15.1:0/197908523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e6006ff50 msgr2=0x7f5e60148d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.497+0000 7f5e5dd9b640 1 --2- 172.21.15.1:0/197908523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e6006ff50 0x7f5e60148d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.497+0000 7f5e5dd9b640 1 -- 172.21.15.1:0/197908523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e60150780 msgr2=0x7f5e60152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.497+0000 7f5e5dd9b640 1 --2- 172.21.15.1:0/197908523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e60150780 0x7f5e60152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.497+0000 7f5e5dd9b640 1 -- 172.21.15.1:0/197908523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e600c2e10 con 0x7f5e600c4ce0 2024-03-06T07:25:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.498+0000 7f5e5dd9b640 1 --2- 172.21.15.1:0/197908523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e600c4ce0 0x7f5e600c1d20 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f5e54000c00 tx=0x7f5e540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.498+0000 7f5e4effd640 1 -- 172.21.15.1:0/197908523 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54032880 con 0x7f5e600c4ce0 2024-03-06T07:25:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.498+0000 7f5e4effd640 1 -- 172.21.15.1:0/197908523 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e54032a20 con 0x7f5e600c4ce0 2024-03-06T07:25:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.498+0000 7f5e64c9d640 1 -- 172.21.15.1:0/197908523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e601eb690 con 0x7f5e600c4ce0 2024-03-06T07:25:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.498+0000 7f5e4effd640 1 -- 172.21.15.1:0/197908523 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54032e10 con 0x7f5e600c4ce0 2024-03-06T07:25:19.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.498+0000 7f5e64c9d640 1 -- 172.21.15.1:0/197908523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e60149260 con 0x7f5e600c4ce0 2024-03-06T07:25:19.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.499+0000 7f5e64c9d640 1 -- 172.21.15.1:0/197908523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e600686e0 con 0x7f5e600c4ce0 2024-03-06T07:25:19.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.499+0000 7f5e4effd640 1 -- 172.21.15.1:0/197908523 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e5404c060 con 0x7f5e600c4ce0 2024-03-06T07:25:19.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.500+0000 7f5e4effd640 1 --2- 172.21.15.1:0/197908523 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e4006b4d0 0x7f5e4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.500+0000 7f5e4effd640 1 -- 172.21.15.1:0/197908523 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5921+0+0 (secure 0 0 0) 0x7f5e540abcd0 con 0x7f5e600c4ce0 2024-03-06T07:25:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.500+0000 7f5e5e59c640 1 --2- 172.21.15.1:0/197908523 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e4006b4d0 0x7f5e4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.500+0000 7f5e5e59c640 1 --2- 172.21.15.1:0/197908523 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e4006b4d0 0x7f5e4006d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f5e6006d9f0 tx=0x7f5e50006bf0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:19.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.504+0000 7f5e4effd640 1 -- 172.21.15.1:0/197908523 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e54040070 con 0x7f5e600c4ce0 2024-03-06T07:25:19.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:19.686+0000 7f5e64c9d640 1 -- 172.21.15.1:0/197908523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5e600c33c0 con 0x7f5e600c4ce0 2024-03-06T07:25:20.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.373+0000 7f5e4effd640 1 -- 172.21.15.1:0/197908523 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v126) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5e5407a410 con 0x7f5e600c4ce0 2024-03-06T07:25:20.375 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-06T07:25:20.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.377+0000 7f5e64c9d640 1 -- 172.21.15.1:0/197908523 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e4006b4d0 msgr2=0x7f5e4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.377+0000 7f5e64c9d640 1 --2- 172.21.15.1:0/197908523 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e4006b4d0 0x7f5e4006d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f5e6006d9f0 tx=0x7f5e50006bf0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.377+0000 7f5e64c9d640 1 -- 172.21.15.1:0/197908523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e600c4ce0 msgr2=0x7f5e600c1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.377+0000 7f5e64c9d640 1 --2- 172.21.15.1:0/197908523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e600c4ce0 0x7f5e600c1d20 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f5e54000c00 tx=0x7f5e540046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.377+0000 7f5e64c9d640 1 -- 172.21.15.1:0/197908523 shutdown_connections 2024-03-06T07:25:20.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.377+0000 7f5e64c9d640 1 --2- 172.21.15.1:0/197908523 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e4006b4d0 0x7f5e4006d990 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.377+0000 7f5e64c9d640 1 --2- 172.21.15.1:0/197908523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e60150780 0x7f5e60152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.377+0000 7f5e64c9d640 1 --2- 172.21.15.1:0/197908523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e600c4ce0 0x7f5e600c1d20 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.377+0000 7f5e64c9d640 1 --2- 172.21.15.1:0/197908523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e6006ff50 0x7f5e60148d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.377+0000 7f5e64c9d640 1 -- 172.21.15.1:0/197908523 >> 172.21.15.1:0/197908523 conn(0x7f5e600aa6a0 msgr2=0x7f5e600a8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.377+0000 7f5e64c9d640 1 -- 172.21.15.1:0/197908523 shutdown_connections 2024-03-06T07:25:20.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.377+0000 7f5e64c9d640 1 -- 172.21.15.1:0/197908523 wait complete. 2024-03-06T07:25:20.394 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:25:20.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.502+0000 7f902277a640 1 Processor -- start 2024-03-06T07:25:20.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.502+0000 7f902277a640 1 -- start start 2024-03-06T07:25:20.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.503+0000 7f902277a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f901c14cdc0 0x7f901c152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.503+0000 7f902277a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901c156500 0x7f901c0c2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.503+0000 7f902277a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f901c0c2890 0x7f901c06b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.503+0000 7f902277a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f901c0c2dd0 con 0x7f901c0c2890 2024-03-06T07:25:20.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.503+0000 7f902277a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f901c1569e0 con 0x7f901c14cdc0 2024-03-06T07:25:20.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.503+0000 7f902277a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901c156cc0 con 0x7f901c156500 2024-03-06T07:25:20.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.503+0000 7f9020d17640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f901c0c2890 0x7f901c06b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.503+0000 7f9020d17640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f901c0c2890 0x7f901c06b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34962/0 (socket says 172.21.15.1:34962) 2024-03-06T07:25:20.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.503+0000 7f9020d17640 1 -- 172.21.15.1:0/1110828284 learned_addr learned my addr 172.21.15.1:0/1110828284 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:20.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.504+0000 7f9020d17640 1 -- 172.21.15.1:0/1110828284 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901c156500 msgr2=0x7f901c0c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.504+0000 7f901b7fe640 1 --2- 172.21.15.1:0/1110828284 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901c156500 0x7f901c0c2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.504+0000 7f901bfff640 1 --2- 172.21.15.1:0/1110828284 >> v2:172.21.15.98:3300/0 conn(0x7f901c14cdc0 0x7f901c152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.504+0000 7f9020d17640 1 --2- 172.21.15.1:0/1110828284 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901c156500 0x7f901c0c2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.504+0000 7f9020d17640 1 -- 172.21.15.1:0/1110828284 >> v2:172.21.15.98:3300/0 conn(0x7f901c14cdc0 msgr2=0x7f901c152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.504+0000 7f9020d17640 1 --2- 172.21.15.1:0/1110828284 >> v2:172.21.15.98:3300/0 conn(0x7f901c14cdc0 0x7f901c152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.504+0000 7f9020d17640 1 -- 172.21.15.1:0/1110828284 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901c156fa0 con 0x7f901c0c2890 2024-03-06T07:25:20.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.504+0000 7f901bfff640 1 --2- 172.21.15.1:0/1110828284 >> v2:172.21.15.98:3300/0 conn(0x7f901c14cdc0 0x7f901c152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:20.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.504+0000 7f901b7fe640 1 --2- 172.21.15.1:0/1110828284 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901c156500 0x7f901c0c2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:20.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.504+0000 7f9020d17640 1 --2- 172.21.15.1:0/1110828284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c0c2890 0x7f901c06b510 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f901c071cf0 tx=0x7f901402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5236ff93e8aff7b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.505+0000 7f901affd640 1 -- 172.21.15.1:0/1110828284 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901402ca70 con 0x7f901c0c2890 2024-03-06T07:25:20.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.505+0000 7f901affd640 1 -- 172.21.15.1:0/1110828284 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901402cc10 con 0x7f901c0c2890 2024-03-06T07:25:20.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.505+0000 7f901affd640 1 -- 172.21.15.1:0/1110828284 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90140343f0 con 0x7f901c0c2890 2024-03-06T07:25:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.505+0000 7f902277a640 1 -- 172.21.15.1:0/1110828284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c0c2890 msgr2=0x7f901c06b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.505+0000 7f902277a640 1 --2- 172.21.15.1:0/1110828284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c0c2890 0x7f901c06b510 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f901c071cf0 tx=0x7f901402bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.505+0000 7f902277a640 1 -- 172.21.15.1:0/1110828284 shutdown_connections 2024-03-06T07:25:20.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.505+0000 7f902277a640 1 --2- 172.21.15.1:0/1110828284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c0c2890 0x7f901c06b510 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.505+0000 7f902277a640 1 --2- 172.21.15.1:0/1110828284 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901c156500 0x7f901c0c2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.505+0000 7f902277a640 1 --2- 172.21.15.1:0/1110828284 >> v2:172.21.15.98:3300/0 conn(0x7f901c14cdc0 0x7f901c152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.505+0000 7f902277a640 1 -- 172.21.15.1:0/1110828284 >> 172.21.15.1:0/1110828284 conn(0x7f901c0a9700 msgr2=0x7f901c0a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.505+0000 7f902277a640 1 -- 172.21.15.1:0/1110828284 shutdown_connections 2024-03-06T07:25:20.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.505+0000 7f902277a640 1 -- 172.21.15.1:0/1110828284 wait complete. 2024-03-06T07:25:20.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.506+0000 7f902277a640 1 Processor -- start 2024-03-06T07:25:20.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.507+0000 7f902277a640 1 -- start start 2024-03-06T07:25:20.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.507+0000 7f902277a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f901c0c2890 0x7f901c072810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.507+0000 7f902277a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c14cdc0 0x7f901c14e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.507+0000 7f902277a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901c156500 0x7f901c14edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.507+0000 7f902277a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c0c65e0 con 0x7f901c14cdc0 2024-03-06T07:25:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.507+0000 7f902277a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c0c5a70 con 0x7f901c0c2890 2024-03-06T07:25:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.507+0000 7f902277a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901c14d1a0 con 0x7f901c156500 2024-03-06T07:25:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.507+0000 7f901b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c14cdc0 0x7f901c14e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.507+0000 7f901b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c14cdc0 0x7f901c14e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34976/0 (socket says 172.21.15.1:34976) 2024-03-06T07:25:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.507+0000 7f901b7fe640 1 -- 172.21.15.1:0/1839752169 learned_addr learned my addr 172.21.15.1:0/1839752169 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.507+0000 7f901bfff640 1 --2- 172.21.15.1:0/1839752169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f901c0c2890 0x7f901c072810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.507+0000 7f9020d17640 1 --2- 172.21.15.1:0/1839752169 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901c156500 0x7f901c14edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.508+0000 7f901b7fe640 1 -- 172.21.15.1:0/1839752169 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901c156500 msgr2=0x7f901c14edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.508+0000 7f901b7fe640 1 --2- 172.21.15.1:0/1839752169 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901c156500 0x7f901c14edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.508+0000 7f901b7fe640 1 -- 172.21.15.1:0/1839752169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f901c0c2890 msgr2=0x7f901c072810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.508+0000 7f901b7fe640 1 --2- 172.21.15.1:0/1839752169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f901c0c2890 0x7f901c072810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.508+0000 7f901b7fe640 1 -- 172.21.15.1:0/1839752169 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901c150860 con 0x7f901c14cdc0 2024-03-06T07:25:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.508+0000 7f901bfff640 1 --2- 172.21.15.1:0/1839752169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f901c0c2890 0x7f901c072810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.508+0000 7f9020d17640 1 --2- 172.21.15.1:0/1839752169 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901c156500 0x7f901c14edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.508+0000 7f901b7fe640 1 --2- 172.21.15.1:0/1839752169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c14cdc0 0x7f901c14e880 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f901000c820 tx=0x7f901000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.508+0000 7f9018ff9640 1 -- 172.21.15.1:0/1839752169 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9010015070 con 0x7f901c14cdc0 2024-03-06T07:25:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.508+0000 7f9018ff9640 1 -- 172.21.15.1:0/1839752169 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9010007a30 con 0x7f901c14cdc0 2024-03-06T07:25:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.508+0000 7f902277a640 1 -- 172.21.15.1:0/1839752169 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f901c0aa7c0 con 0x7f901c14cdc0 2024-03-06T07:25:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.508+0000 7f9018ff9640 1 -- 172.21.15.1:0/1839752169 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9010007e00 con 0x7f901c14cdc0 2024-03-06T07:25:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.508+0000 7f902277a640 1 -- 172.21.15.1:0/1839752169 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901c1ec930 con 0x7f901c14cdc0 2024-03-06T07:25:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.509+0000 7f902277a640 1 -- 172.21.15.1:0/1839752169 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901c1ec720 con 0x7f901c14cdc0 2024-03-06T07:25:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.509+0000 7f9018ff9640 1 -- 172.21.15.1:0/1839752169 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9010018070 con 0x7f901c14cdc0 2024-03-06T07:25:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.510+0000 7f9018ff9640 1 --2- 172.21.15.1:0/1839752169 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f900806b4d0 0x7f900806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.510+0000 7f9018ff9640 1 -- 172.21.15.1:0/1839752169 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f901008a080 con 0x7f901c14cdc0 2024-03-06T07:25:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.510+0000 7f901bfff640 1 --2- 172.21.15.1:0/1839752169 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f900806b4d0 0x7f900806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.511+0000 7f901bfff640 1 --2- 172.21.15.1:0/1839752169 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f900806b4d0 0x7f900806d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f900c006b10 tx=0x7f900c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.514+0000 7f9018ff9640 1 -- 172.21.15.1:0/1839752169 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9010058fc0 con 0x7f901c14cdc0 2024-03-06T07:25:20.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.696+0000 7f902277a640 1 -- 172.21.15.1:0/1839752169 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f901c14e5e0 con 0x7f901c14cdc0 2024-03-06T07:25:20.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.699+0000 7f9018ff9640 1 -- 172.21.15.1:0/1839752169 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7f90100588c0 con 0x7f901c14cdc0 2024-03-06T07:25:20.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.701+0000 7f902277a640 1 -- 172.21.15.1:0/1839752169 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f900806b4d0 msgr2=0x7f900806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.701+0000 7f902277a640 1 --2- 172.21.15.1:0/1839752169 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f900806b4d0 0x7f900806d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f900c006b10 tx=0x7f900c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:20.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.701+0000 7f902277a640 1 -- 172.21.15.1:0/1839752169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c14cdc0 msgr2=0x7f901c14e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.701+0000 7f902277a640 1 --2- 172.21.15.1:0/1839752169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c14cdc0 0x7f901c14e880 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f901000c820 tx=0x7f901000cc20 comp rx=0 tx=0).stop 2024-03-06T07:25:20.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.701+0000 7f902277a640 1 -- 172.21.15.1:0/1839752169 shutdown_connections 2024-03-06T07:25:20.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.701+0000 7f902277a640 1 --2- 172.21.15.1:0/1839752169 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f900806b4d0 0x7f900806d990 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.701+0000 7f902277a640 1 --2- 172.21.15.1:0/1839752169 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f901c156500 0x7f901c14edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.701+0000 7f902277a640 1 --2- 172.21.15.1:0/1839752169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c14cdc0 0x7f901c14e880 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.701+0000 7f902277a640 1 --2- 172.21.15.1:0/1839752169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f901c0c2890 0x7f901c072810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.701+0000 7f902277a640 1 -- 172.21.15.1:0/1839752169 >> 172.21.15.1:0/1839752169 conn(0x7f901c0a9700 msgr2=0x7f901c0a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.701+0000 7f902277a640 1 -- 172.21.15.1:0/1839752169 shutdown_connections 2024-03-06T07:25:20.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.701+0000 7f902277a640 1 -- 172.21.15.1:0/1839752169 wait complete. 2024-03-06T07:25:20.714 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-03-06T07:25:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.820+0000 7f1154b77640 1 Processor -- start 2024-03-06T07:25:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.820+0000 7f1154b77640 1 -- start start 2024-03-06T07:25:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.821+0000 7f1154b77640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f115006bc20 0x7f1150152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.821+0000 7f1154b77640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11501569e0 con 0x7f1150156500 2024-03-06T07:25:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.821+0000 7f1154b77640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1150156cc0 con 0x7f115014cdc0 2024-03-06T07:25:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.821+0000 7f1154b77640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11500c2dd0 con 0x7f115006bc20 2024-03-06T07:25:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.821+0000 7f114dd9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1150156500 0x7f115006b510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39462/0 (socket says 172.21.15.1:39462) 2024-03-06T07:25:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.821+0000 7f114dd9b640 1 -- 172.21.15.1:0/4150507526 learned_addr learned my addr 172.21.15.1:0/4150507526 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114d59a640 1 -- 172.21.15.1:0/4150507526 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387662022 0 0) 0x7f11501569e0 con 0x7f1150156500 2024-03-06T07:25:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114d59a640 1 -- 172.21.15.1:0/4150507526 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113c003520 con 0x7f1150156500 2024-03-06T07:25:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114ed9d640 1 --2- 172.21.15.1:0/4150507526 >> v2:172.21.15.98:3300/0 conn(0x7f115006bc20 0x7f1150152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114d59a640 1 -- 172.21.15.1:0/4150507526 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3272250376 0 0) 0x7f113c003520 con 0x7f1150156500 2024-03-06T07:25:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114d59a640 1 -- 172.21.15.1:0/4150507526 >> v2:172.21.15.98:3300/0 conn(0x7f115006bc20 msgr2=0x7f1150152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114d59a640 1 --2- 172.21.15.1:0/4150507526 >> v2:172.21.15.98:3300/0 conn(0x7f115006bc20 0x7f1150152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114d59a640 1 -- 172.21.15.1:0/4150507526 >> v1:172.21.15.98:6789/0 conn(0x7f115014cdc0 legacy=0x7f11500c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114d59a640 1 -- 172.21.15.1:0/4150507526 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1150156fa0 con 0x7f1150156500 2024-03-06T07:25:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114d59a640 1 -- 172.21.15.1:0/4150507526 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f1140001670 con 0x7f1150156500 2024-03-06T07:25:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114d59a640 1 -- 172.21.15.1:0/4150507526 >> v1:172.21.15.1:6789/0 conn(0x7f1150156500 legacy=0x7f115006b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114d59a640 1 --2- 172.21.15.1:0/4150507526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f113c0039b0 0x7f113c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114d59a640 1 --2- 172.21.15.1:0/4150507526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f113c004800 0x7f113c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114d59a640 1 --2- 172.21.15.1:0/4150507526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f113c0051c0 0x7f113c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114d59a640 1 -- 172.21.15.1:0/4150507526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c00c130 con 0x7f113c004800 2024-03-06T07:25:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114d59a640 1 -- 172.21.15.1:0/4150507526 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c00c2e0 con 0x7f113c0039b0 2024-03-06T07:25:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114d59a640 1 -- 172.21.15.1:0/4150507526 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113c00c4e0 con 0x7f113c0051c0 2024-03-06T07:25:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114dd9b640 1 --2- 172.21.15.1:0/4150507526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f113c004800 0x7f113c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114e59c640 1 --2- 172.21.15.1:0/4150507526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f113c0039b0 0x7f113c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114ed9d640 1 --2- 172.21.15.1:0/4150507526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f113c0051c0 0x7f113c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114dd9b640 1 -- 172.21.15.1:0/4150507526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f113c0051c0 msgr2=0x7f113c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114dd9b640 1 --2- 172.21.15.1:0/4150507526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f113c0051c0 0x7f113c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114dd9b640 1 -- 172.21.15.1:0/4150507526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f113c0039b0 msgr2=0x7f113c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114dd9b640 1 --2- 172.21.15.1:0/4150507526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f113c0039b0 0x7f113c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114dd9b640 1 -- 172.21.15.1:0/4150507526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c003520 con 0x7f113c004800 2024-03-06T07:25:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.822+0000 7f114ed9d640 1 -- 172.21.15.1:0/4150507526 reap_dead start 2024-03-06T07:25:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.823+0000 7f114dd9b640 1 --2- 172.21.15.1:0/4150507526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f113c004800 0x7f113c004c80 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f114000c240 tx=0x7f113c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4878e0a94de8bcc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.823+0000 7f114d59a640 1 -- 172.21.15.1:0/4150507526 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1140007d10 con 0x7f113c004800 2024-03-06T07:25:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.823+0000 7f114d59a640 1 -- 172.21.15.1:0/4150507526 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1140030070 con 0x7f113c004800 2024-03-06T07:25:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.823+0000 7f114d59a640 1 -- 172.21.15.1:0/4150507526 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1140005d90 con 0x7f113c004800 2024-03-06T07:25:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.824+0000 7f1154b77640 1 -- 172.21.15.1:0/4150507526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f113c004800 msgr2=0x7f113c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.824+0000 7f1154b77640 1 --2- 172.21.15.1:0/4150507526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f113c004800 0x7f113c004c80 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f114000c240 tx=0x7f113c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.824+0000 7f1154b77640 1 -- 172.21.15.1:0/4150507526 shutdown_connections 2024-03-06T07:25:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.824+0000 7f1154b77640 1 --2- 172.21.15.1:0/4150507526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f113c004800 0x7f113c004c80 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.824+0000 7f1154b77640 1 -- 172.21.15.1:0/4150507526 >> 172.21.15.1:0/4150507526 conn(0x7f11500a9700 msgr2=0x7f11500a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.824+0000 7f1154b77640 1 -- 172.21.15.1:0/4150507526 shutdown_connections 2024-03-06T07:25:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.824+0000 7f1154b77640 1 -- 172.21.15.1:0/4150507526 wait complete. 2024-03-06T07:25:20.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.825+0000 7f1154b77640 1 Processor -- start 2024-03-06T07:25:20.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.825+0000 7f1154b77640 1 -- start start 2024-03-06T07:25:20.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.826+0000 7f1154b77640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f113c004800 0x7f11501eacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.826+0000 7f1154b77640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115014fcb0 0x7f115014e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.826+0000 7f1154b77640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11500c0ff0 0x7f11501eea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.826+0000 7f1154b77640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11500c65e0 con 0x7f11500c0ff0 2024-03-06T07:25:20.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.826+0000 7f1154b77640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11500c5a70 con 0x7f113c004800 2024-03-06T07:25:20.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.826+0000 7f1154b77640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115014d1a0 con 0x7f115014fcb0 2024-03-06T07:25:20.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.826+0000 7f114ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11500c0ff0 0x7f11501eea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.826+0000 7f114ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11500c0ff0 0x7f11501eea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35000/0 (socket says 172.21.15.1:35000) 2024-03-06T07:25:20.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.826+0000 7f114ed9d640 1 -- 172.21.15.1:0/931404389 learned_addr learned my addr 172.21.15.1:0/931404389 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:20.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.826+0000 7f114dd9b640 1 --2- 172.21.15.1:0/931404389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115014fcb0 0x7f115014e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.826+0000 7f114e59c640 1 --2- 172.21.15.1:0/931404389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f113c004800 0x7f11501eacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.826+0000 7f114ed9d640 1 -- 172.21.15.1:0/931404389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115014fcb0 msgr2=0x7f115014e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.826+0000 7f114ed9d640 1 --2- 172.21.15.1:0/931404389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115014fcb0 0x7f115014e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.826+0000 7f114ed9d640 1 -- 172.21.15.1:0/931404389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f113c004800 msgr2=0x7f11501eacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.826+0000 7f114ed9d640 1 --2- 172.21.15.1:0/931404389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f113c004800 0x7f11501eacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.826+0000 7f114ed9d640 1 -- 172.21.15.1:0/931404389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11501ee230 con 0x7f11500c0ff0 2024-03-06T07:25:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.826+0000 7f114ed9d640 1 --2- 172.21.15.1:0/931404389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11500c0ff0 0x7f11501eea10 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f1150150a50 tx=0x7f114800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.827+0000 7f1146ffd640 1 -- 172.21.15.1:0/931404389 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114800fd80 con 0x7f11500c0ff0 2024-03-06T07:25:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.827+0000 7f1146ffd640 1 -- 172.21.15.1:0/931404389 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114801d070 con 0x7f11500c0ff0 2024-03-06T07:25:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.827+0000 7f1154b77640 1 -- 172.21.15.1:0/931404389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11500c4f80 con 0x7f11500c0ff0 2024-03-06T07:25:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.827+0000 7f1146ffd640 1 -- 172.21.15.1:0/931404389 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148015670 con 0x7f11500c0ff0 2024-03-06T07:25:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.827+0000 7f1154b77640 1 -- 172.21.15.1:0/931404389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1150150470 con 0x7f11500c0ff0 2024-03-06T07:25:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.828+0000 7f1154b77640 1 -- 172.21.15.1:0/931404389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1150067fd0 con 0x7f11500c0ff0 2024-03-06T07:25:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.828+0000 7f1146ffd640 1 -- 172.21.15.1:0/931404389 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1148015810 con 0x7f11500c0ff0 2024-03-06T07:25:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.828+0000 7f1146ffd640 1 --2- 172.21.15.1:0/931404389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f113006b500 0x7f113006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.829+0000 7f1146ffd640 1 -- 172.21.15.1:0/931404389 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f114808ad50 con 0x7f11500c0ff0 2024-03-06T07:25:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.829+0000 7f114e59c640 1 --2- 172.21.15.1:0/931404389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f113006b500 0x7f113006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.829+0000 7f114e59c640 1 --2- 172.21.15.1:0/931404389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f113006b500 0x7f113006d9c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1138000c00 tx=0x7f1138009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:20.833+0000 7f1146ffd640 1 -- 172.21.15.1:0/931404389 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1148027020 con 0x7f11500c0ff0 2024-03-06T07:25:21.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.014+0000 7f1154b77640 1 -- 172.21.15.1:0/931404389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7f1150153e00 con 0x7f11500c0ff0 2024-03-06T07:25:21.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.017+0000 7f1146ffd640 1 -- 172.21.15.1:0/931404389 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v44) v1 ==== 96+0+0 (secure 0 0 0) 0x7f1148010040 con 0x7f11500c0ff0 2024-03-06T07:25:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.019+0000 7f1154b77640 1 -- 172.21.15.1:0/931404389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f113006b500 msgr2=0x7f113006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.019+0000 7f1154b77640 1 --2- 172.21.15.1:0/931404389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f113006b500 0x7f113006d9c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1138000c00 tx=0x7f1138009040 comp rx=0 tx=0).stop 2024-03-06T07:25:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.019+0000 7f1154b77640 1 -- 172.21.15.1:0/931404389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11500c0ff0 msgr2=0x7f11501eea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.019+0000 7f1154b77640 1 --2- 172.21.15.1:0/931404389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11500c0ff0 0x7f11501eea10 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f1150150a50 tx=0x7f114800f580 comp rx=0 tx=0).stop 2024-03-06T07:25:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.019+0000 7f1154b77640 1 -- 172.21.15.1:0/931404389 shutdown_connections 2024-03-06T07:25:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.019+0000 7f1154b77640 1 --2- 172.21.15.1:0/931404389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f113006b500 0x7f113006d9c0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.019+0000 7f1154b77640 1 --2- 172.21.15.1:0/931404389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11500c0ff0 0x7f11501eea10 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.019+0000 7f1154b77640 1 --2- 172.21.15.1:0/931404389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f115014fcb0 0x7f115014e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.019+0000 7f1154b77640 1 --2- 172.21.15.1:0/931404389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f113c004800 0x7f11501eacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.020+0000 7f1154b77640 1 -- 172.21.15.1:0/931404389 >> 172.21.15.1:0/931404389 conn(0x7f11500a9700 msgr2=0x7f115006b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:21.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.020+0000 7f1154b77640 1 -- 172.21.15.1:0/931404389 shutdown_connections 2024-03-06T07:25:21.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.020+0000 7f1154b77640 1 -- 172.21.15.1:0/931404389 wait complete. 2024-03-06T07:25:21.032 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady' 2024-03-06T07:25:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.132+0000 7f45b1f93640 1 Processor -- start 2024-03-06T07:25:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.132+0000 7f45b1f93640 1 -- start start 2024-03-06T07:25:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.132+0000 7f45b1f93640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f45ac0c1390 0x7f45ac0c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.132+0000 7f45b1f93640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f45ac1552e0 0x7f45ac155700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.132+0000 7f45b1f93640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ac1588e0 con 0x7f45ac152730 2024-03-06T07:25:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.132+0000 7f45b1f93640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45ac0c1d60 con 0x7f45ac0c1390 2024-03-06T07:25:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.132+0000 7f45b1f93640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45ac158bc0 con 0x7f45ac1552e0 2024-03-06T07:25:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.133+0000 7f45aaffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f45ac0c1390 0x7f45ac0c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.133+0000 7f45aaffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f45ac0c1390 0x7f45ac0c1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35010/0 (socket says 172.21.15.1:35010) 2024-03-06T07:25:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.133+0000 7f45aaffd640 1 -- 172.21.15.1:0/2641116165 learned_addr learned my addr 172.21.15.1:0/2641116165 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.133+0000 7f45abfff640 1 --2- 172.21.15.1:0/2641116165 >> v2:172.21.15.98:3300/0 conn(0x7f45ac1552e0 0x7f45ac155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.133+0000 7f45aaffd640 1 -- 172.21.15.1:0/2641116165 >> v2:172.21.15.98:3300/0 conn(0x7f45ac1552e0 msgr2=0x7f45ac155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.133+0000 7f45aaffd640 1 --2- 172.21.15.1:0/2641116165 >> v2:172.21.15.98:3300/0 conn(0x7f45ac1552e0 0x7f45ac155700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.133+0000 7f45aaffd640 1 -- 172.21.15.1:0/2641116165 >> v1:172.21.15.98:6789/0 conn(0x7f45ac152730 legacy=0x7f45ac0c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.133+0000 7f45aaffd640 1 -- 172.21.15.1:0/2641116165 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac158ea0 con 0x7f45ac0c1390 2024-03-06T07:25:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.134+0000 7f45aaffd640 1 --2- 172.21.15.1:0/2641116165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45ac0c1390 0x7f45ac0c1820 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f45ac08c2e0 tx=0x7f459c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c80e5049281250 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.134+0000 7f45aa7fc640 1 -- 172.21.15.1:0/2641116165 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c038070 con 0x7f45ac0c1390 2024-03-06T07:25:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.134+0000 7f45aa7fc640 1 -- 172.21.15.1:0/2641116165 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459c02f050 con 0x7f45ac0c1390 2024-03-06T07:25:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.134+0000 7f45aa7fc640 1 -- 172.21.15.1:0/2641116165 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c031970 con 0x7f45ac0c1390 2024-03-06T07:25:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.134+0000 7f45b1f93640 1 -- 172.21.15.1:0/2641116165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45ac0c1390 msgr2=0x7f45ac0c1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.134+0000 7f45b1f93640 1 --2- 172.21.15.1:0/2641116165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45ac0c1390 0x7f45ac0c1820 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f45ac08c2e0 tx=0x7f459c02b7f0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.134+0000 7f45b1f93640 1 -- 172.21.15.1:0/2641116165 shutdown_connections 2024-03-06T07:25:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.134+0000 7f45b1f93640 1 --2- 172.21.15.1:0/2641116165 >> v2:172.21.15.98:3300/0 conn(0x7f45ac1552e0 0x7f45ac155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.134+0000 7f45b1f93640 1 --2- 172.21.15.1:0/2641116165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45ac0c1390 0x7f45ac0c1820 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.134+0000 7f45b1f93640 1 -- 172.21.15.1:0/2641116165 >> 172.21.15.1:0/2641116165 conn(0x7f45ac066d00 msgr2=0x7f45ac067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.134+0000 7f45b1f93640 1 -- 172.21.15.1:0/2641116165 shutdown_connections 2024-03-06T07:25:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.134+0000 7f45b1f93640 1 -- 172.21.15.1:0/2641116165 wait complete. 2024-03-06T07:25:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.135+0000 7f45b1f93640 1 Processor -- start 2024-03-06T07:25:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.136+0000 7f45b1f93640 1 -- start start 2024-03-06T07:25:21.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.136+0000 7f45b1f93640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45ac0c1390 0x7f45ac1eed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.136+0000 7f45b1f93640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ac152730 0x7f45ac150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.136+0000 7f45b1f93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45ac1552e0 0x7f45ac1574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.136+0000 7f45b1f93640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac152b10 con 0x7f45ac1552e0 2024-03-06T07:25:21.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.136+0000 7f45b1f93640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac159460 con 0x7f45ac152730 2024-03-06T07:25:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.136+0000 7f45b1f93640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac0c71a0 con 0x7f45ac0c1390 2024-03-06T07:25:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.136+0000 7f45abfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45ac1552e0 0x7f45ac1574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.136+0000 7f45abfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45ac1552e0 0x7f45ac1574b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35012/0 (socket says 172.21.15.1:35012) 2024-03-06T07:25:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.136+0000 7f45abfff640 1 -- 172.21.15.1:0/3837369147 learned_addr learned my addr 172.21.15.1:0/3837369147 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.137+0000 7f45abfff640 1 -- 172.21.15.1:0/3837369147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45ac0c1390 msgr2=0x7f45ac1eed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.137+0000 7f45ab7fe640 1 --2- 172.21.15.1:0/3837369147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45ac0c1390 0x7f45ac1eed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.137+0000 7f45aaffd640 1 --2- 172.21.15.1:0/3837369147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ac152730 0x7f45ac150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.137+0000 7f45abfff640 1 --2- 172.21.15.1:0/3837369147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45ac0c1390 0x7f45ac1eed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.137+0000 7f45abfff640 1 -- 172.21.15.1:0/3837369147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ac152730 msgr2=0x7f45ac150800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.137+0000 7f45abfff640 1 --2- 172.21.15.1:0/3837369147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ac152730 0x7f45ac150800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.137+0000 7f45abfff640 1 -- 172.21.15.1:0/3837369147 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac150ee0 con 0x7f45ac1552e0 2024-03-06T07:25:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.137+0000 7f45abfff640 1 --2- 172.21.15.1:0/3837369147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45ac1552e0 0x7f45ac1574b0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f45a4007d90 tx=0x7f45a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.137+0000 7f4593fff640 1 -- 172.21.15.1:0/3837369147 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4014420 con 0x7f45ac1552e0 2024-03-06T07:25:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.137+0000 7f4593fff640 1 -- 172.21.15.1:0/3837369147 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a4019070 con 0x7f45ac1552e0 2024-03-06T07:25:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.137+0000 7f45b1f93640 1 -- 172.21.15.1:0/3837369147 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ac1ee290 con 0x7f45ac1552e0 2024-03-06T07:25:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.137+0000 7f4593fff640 1 -- 172.21.15.1:0/3837369147 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4014820 con 0x7f45ac1552e0 2024-03-06T07:25:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.137+0000 7f45b1f93640 1 -- 172.21.15.1:0/3837369147 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ac1579f0 con 0x7f45ac1552e0 2024-03-06T07:25:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.138+0000 7f45b1f93640 1 -- 172.21.15.1:0/3837369147 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ac067fd0 con 0x7f45ac1552e0 2024-03-06T07:25:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.139+0000 7f4593fff640 1 -- 172.21.15.1:0/3837369147 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45a40149c0 con 0x7f45ac1552e0 2024-03-06T07:25:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.139+0000 7f4593fff640 1 --2- 172.21.15.1:0/3837369147 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f459806b500 0x7f459806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.139+0000 7f4593fff640 1 -- 172.21.15.1:0/3837369147 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f45a408a970 con 0x7f45ac1552e0 2024-03-06T07:25:21.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.139+0000 7f45ab7fe640 1 --2- 172.21.15.1:0/3837369147 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f459806b500 0x7f459806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.140+0000 7f45ab7fe640 1 --2- 172.21.15.1:0/3837369147 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f459806b500 0x7f459806d9c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f45a000a750 tx=0x7f45a0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:21.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.143+0000 7f4593fff640 1 -- 172.21.15.1:0/3837369147 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45a40598b0 con 0x7f45ac1552e0 2024-03-06T07:25:21.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:21.395+0000 7f45b1f93640 1 -- 172.21.15.1:0/3837369147 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]} v 0) v1 -- 0x7f45ac0c6870 con 0x7f45ac1552e0 2024-03-06T07:25:22.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.376+0000 7f4593fff640 1 -- 172.21.15.1:0/3837369147 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]}]=0 v298) v1 ==== 145+0+0 (secure 0 0 0) 0x7f45a40591b0 con 0x7f45ac1552e0 2024-03-06T07:25:22.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.379+0000 7f45b1f93640 1 -- 172.21.15.1:0/3837369147 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f459806b500 msgr2=0x7f459806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.379+0000 7f45b1f93640 1 --2- 172.21.15.1:0/3837369147 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f459806b500 0x7f459806d9c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f45a000a750 tx=0x7f45a0008040 comp rx=0 tx=0).stop 2024-03-06T07:25:22.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.379+0000 7f45b1f93640 1 -- 172.21.15.1:0/3837369147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45ac1552e0 msgr2=0x7f45ac1574b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.379+0000 7f45b1f93640 1 --2- 172.21.15.1:0/3837369147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45ac1552e0 0x7f45ac1574b0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f45a4007d90 tx=0x7f45a400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.379+0000 7f45b1f93640 1 -- 172.21.15.1:0/3837369147 shutdown_connections 2024-03-06T07:25:22.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.379+0000 7f45b1f93640 1 --2- 172.21.15.1:0/3837369147 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f459806b500 0x7f459806d9c0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.379+0000 7f45b1f93640 1 --2- 172.21.15.1:0/3837369147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45ac1552e0 0x7f45ac1574b0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.379+0000 7f45b1f93640 1 --2- 172.21.15.1:0/3837369147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ac152730 0x7f45ac150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.379+0000 7f45b1f93640 1 --2- 172.21.15.1:0/3837369147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45ac0c1390 0x7f45ac1eed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.379+0000 7f45b1f93640 1 -- 172.21.15.1:0/3837369147 >> 172.21.15.1:0/3837369147 conn(0x7f45ac066d00 msgr2=0x7f45ac0ca820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:22.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.379+0000 7f45b1f93640 1 -- 172.21.15.1:0/3837369147 shutdown_connections 2024-03-06T07:25:22.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.379+0000 7f45b1f93640 1 -- 172.21.15.1:0/3837369147 wait complete. 2024-03-06T07:25:22.394 INFO:tasks.cephfs_test_runner:That a sole MDS with mds_join_fs set will come back as active eventually even after failover. ... ok 2024-03-06T07:25:22.394 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_unset (tasks.cephfs.test_failover.TestClusterAffinity) That unsetting mds_join_fs will cause failover if another high-affinity standby exists. 2024-03-06T07:25:22.395 INFO:tasks.cephfs_test_runner:test_join_fs_unset (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-06T07:25:22.395 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.502+0000 7fb356fb5640 1 Processor -- start 2024-03-06T07:25:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.502+0000 7fb356fb5640 1 -- start start 2024-03-06T07:25:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb356fb5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb350150fe0 0x7fb3501513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb356fb5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3500a6140 con 0x7fb350072f20 2024-03-06T07:25:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb356fb5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb350151aa0 con 0x7fb350071f90 2024-03-06T07:25:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb356fb5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3501526b0 con 0x7fb350150fe0 2024-03-06T07:25:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb34ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb350072f20 0x7fb35006edb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39466/0 (socket says 172.21.15.1:39466) 2024-03-06T07:25:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb34ffff640 1 -- 172.21.15.1:0/1692968874 learned_addr learned my addr 172.21.15.1:0/1692968874 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb34f7fe640 1 -- 172.21.15.1:0/1692968874 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169418863 0 0) 0x7fb3500a6140 con 0x7fb350072f20 2024-03-06T07:25:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb34f7fe640 1 -- 172.21.15.1:0/1692968874 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb340003440 con 0x7fb350072f20 2024-03-06T07:25:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb354d51640 1 --2- 172.21.15.1:0/1692968874 >> v2:172.21.15.98:3300/0 conn(0x7fb350150fe0 0x7fb3501513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb34f7fe640 1 -- 172.21.15.1:0/1692968874 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 586256928 0 0) 0x7fb340003440 con 0x7fb350072f20 2024-03-06T07:25:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb34f7fe640 1 -- 172.21.15.1:0/1692968874 >> v2:172.21.15.98:3300/0 conn(0x7fb350150fe0 msgr2=0x7fb3501513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb34f7fe640 1 --2- 172.21.15.1:0/1692968874 >> v2:172.21.15.98:3300/0 conn(0x7fb350150fe0 0x7fb3501513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb34f7fe640 1 -- 172.21.15.1:0/1692968874 >> v1:172.21.15.98:6789/0 conn(0x7fb350071f90 legacy=0x7fb350072400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb34f7fe640 1 -- 172.21.15.1:0/1692968874 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb350151d80 con 0x7fb350072f20 2024-03-06T07:25:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb34f7fe640 1 -- 172.21.15.1:0/1692968874 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fb33c001670 con 0x7fb350072f20 2024-03-06T07:25:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb34f7fe640 1 -- 172.21.15.1:0/1692968874 >> v1:172.21.15.1:6789/0 conn(0x7fb350072f20 legacy=0x7fb35006edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb34f7fe640 1 --2- 172.21.15.1:0/1692968874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb340003aa0 0x7fb340003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb34f7fe640 1 --2- 172.21.15.1:0/1692968874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb340008850 0x7fb34000acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb34f7fe640 1 --2- 172.21.15.1:0/1692968874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb34000b560 0x7fb34000da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb34f7fe640 1 -- 172.21.15.1:0/1692968874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34000e2d0 con 0x7fb340008850 2024-03-06T07:25:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb34f7fe640 1 -- 172.21.15.1:0/1692968874 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34000e480 con 0x7fb34000b560 2024-03-06T07:25:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.503+0000 7fb34f7fe640 1 -- 172.21.15.1:0/1692968874 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34000e680 con 0x7fb340003aa0 2024-03-06T07:25:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.504+0000 7fb34ffff640 1 --2- 172.21.15.1:0/1692968874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb340008850 0x7fb34000acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.504+0000 7fb354d51640 1 --2- 172.21.15.1:0/1692968874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb340003aa0 0x7fb340003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.504+0000 7fb355552640 1 --2- 172.21.15.1:0/1692968874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb34000b560 0x7fb34000da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.504+0000 7fb34ffff640 1 -- 172.21.15.1:0/1692968874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb340003aa0 msgr2=0x7fb340003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.504+0000 7fb34ffff640 1 --2- 172.21.15.1:0/1692968874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb340003aa0 0x7fb340003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.504+0000 7fb34ffff640 1 -- 172.21.15.1:0/1692968874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb34000b560 msgr2=0x7fb34000da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.504+0000 7fb34ffff640 1 --2- 172.21.15.1:0/1692968874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb34000b560 0x7fb34000da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.504+0000 7fb34ffff640 1 -- 172.21.15.1:0/1692968874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb340003440 con 0x7fb340008850 2024-03-06T07:25:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.504+0000 7fb355552640 1 --2- 172.21.15.1:0/1692968874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb34000b560 0x7fb34000da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.504+0000 7fb355552640 1 -- 172.21.15.1:0/1692968874 reap_dead start 2024-03-06T07:25:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.505+0000 7fb34ffff640 1 --2- 172.21.15.1:0/1692968874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb340008850 0x7fb34000acc0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fb33c00c240 tx=0x7fb340003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd8b2951009a220 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.505+0000 7fb34f7fe640 1 -- 172.21.15.1:0/1692968874 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c007d10 con 0x7fb340008850 2024-03-06T07:25:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.505+0000 7fb34f7fe640 1 -- 172.21.15.1:0/1692968874 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33c030070 con 0x7fb340008850 2024-03-06T07:25:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.505+0000 7fb34f7fe640 1 -- 172.21.15.1:0/1692968874 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c02cdc0 con 0x7fb340008850 2024-03-06T07:25:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.505+0000 7fb356fb5640 1 -- 172.21.15.1:0/1692968874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb340008850 msgr2=0x7fb34000acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.505+0000 7fb356fb5640 1 --2- 172.21.15.1:0/1692968874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb340008850 0x7fb34000acc0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fb33c00c240 tx=0x7fb340003730 comp rx=0 tx=0).stop 2024-03-06T07:25:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.505+0000 7fb356fb5640 1 -- 172.21.15.1:0/1692968874 shutdown_connections 2024-03-06T07:25:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.505+0000 7fb356fb5640 1 --2- 172.21.15.1:0/1692968874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb340008850 0x7fb34000acc0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.505+0000 7fb356fb5640 1 -- 172.21.15.1:0/1692968874 >> 172.21.15.1:0/1692968874 conn(0x7fb3500c6860 msgr2=0x7fb3500c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.505+0000 7fb356fb5640 1 -- 172.21.15.1:0/1692968874 shutdown_connections 2024-03-06T07:25:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.505+0000 7fb356fb5640 1 -- 172.21.15.1:0/1692968874 wait complete. 2024-03-06T07:25:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.507+0000 7fb356fb5640 1 Processor -- start 2024-03-06T07:25:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.507+0000 7fb356fb5640 1 -- start start 2024-03-06T07:25:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.507+0000 7fb356fb5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb340008850 0x7fb3500a7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.507+0000 7fb356fb5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3500a6f30 0x7fb3500a73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.507+0000 7fb356fb5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3501eb140 0x7fb3501eb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.507+0000 7fb356fb5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3501513c0 con 0x7fb3500a6f30 2024-03-06T07:25:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.507+0000 7fb356fb5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb350152340 con 0x7fb3501eb140 2024-03-06T07:25:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.507+0000 7fb356fb5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb350151540 con 0x7fb340008850 2024-03-06T07:25:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.508+0000 7fb34ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3500a6f30 0x7fb3500a73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.508+0000 7fb34ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3500a6f30 0x7fb3500a73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35028/0 (socket says 172.21.15.1:35028) 2024-03-06T07:25:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.508+0000 7fb34ffff640 1 -- 172.21.15.1:0/2134001684 learned_addr learned my addr 172.21.15.1:0/2134001684 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.508+0000 7fb354d51640 1 --2- 172.21.15.1:0/2134001684 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb340008850 0x7fb3500a7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.508+0000 7fb355552640 1 --2- 172.21.15.1:0/2134001684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3501eb140 0x7fb3501eb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.508+0000 7fb34ffff640 1 -- 172.21.15.1:0/2134001684 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb340008850 msgr2=0x7fb3500a7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.508+0000 7fb34ffff640 1 --2- 172.21.15.1:0/2134001684 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb340008850 0x7fb3500a7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.508+0000 7fb34ffff640 1 -- 172.21.15.1:0/2134001684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3501eb140 msgr2=0x7fb3501eb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.508+0000 7fb34ffff640 1 --2- 172.21.15.1:0/2134001684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3501eb140 0x7fb3501eb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.508+0000 7fb34ffff640 1 -- 172.21.15.1:0/2134001684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3501ee240 con 0x7fb3500a6f30 2024-03-06T07:25:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.508+0000 7fb34ffff640 1 --2- 172.21.15.1:0/2134001684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3500a6f30 0x7fb3500a73b0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fb33c001970 tx=0x7fb3500c2b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.508+0000 7fb34d7fa640 1 -- 172.21.15.1:0/2134001684 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c045070 con 0x7fb3500a6f30 2024-03-06T07:25:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.508+0000 7fb34d7fa640 1 -- 172.21.15.1:0/2134001684 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33c0049f0 con 0x7fb3500a6f30 2024-03-06T07:25:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.508+0000 7fb356fb5640 1 -- 172.21.15.1:0/2134001684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3500ab950 con 0x7fb3500a6f30 2024-03-06T07:25:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.508+0000 7fb34d7fa640 1 -- 172.21.15.1:0/2134001684 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c007d10 con 0x7fb3500a6f30 2024-03-06T07:25:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.508+0000 7fb356fb5640 1 -- 172.21.15.1:0/2134001684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3500abf10 con 0x7fb3500a6f30 2024-03-06T07:25:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.509+0000 7fb356fb5640 1 -- 172.21.15.1:0/2134001684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb350067fd0 con 0x7fb3500a6f30 2024-03-06T07:25:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.510+0000 7fb34d7fa640 1 -- 172.21.15.1:0/2134001684 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb33c007070 con 0x7fb3500a6f30 2024-03-06T07:25:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.510+0000 7fb34d7fa640 1 --2- 172.21.15.1:0/2134001684 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb33406b500 0x7fb33406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.510+0000 7fb34d7fa640 1 -- 172.21.15.1:0/2134001684 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fb33c0aacc0 con 0x7fb3500a6f30 2024-03-06T07:25:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.511+0000 7fb354d51640 1 --2- 172.21.15.1:0/2134001684 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb33406b500 0x7fb33406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.511+0000 7fb354d51640 1 --2- 172.21.15.1:0/2134001684 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb33406b500 0x7fb33406d9c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fb3501e8950 tx=0x7fb344009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.515+0000 7fb34d7fa640 1 -- 172.21.15.1:0/2134001684 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb33c079d50 con 0x7fb3500a6f30 2024-03-06T07:25:22.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.697+0000 7fb356fb5640 1 -- 172.21.15.1:0/2134001684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb35006eb90 con 0x7fb3500a6f30 2024-03-06T07:25:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.698+0000 7fb34d7fa640 1 -- 172.21.15.1:0/2134001684 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+13439 (secure 0 0 0) 0x7fb33c079650 con 0x7fb3500a6f30 2024-03-06T07:25:22.700 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:22.700 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":126,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:25:20.367498+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6834/1747678472":"2024-03-07T07:25:17.356815+0000","172.21.15.1:6832/1747678472":"2024-03-07T07:25:17.356815+0000","172.21.15.98:6836/1420541659":"2024-03-07T07:25:09.267103+0000","172.21.15.98:6835/1420541659":"2024-03-07T07:25:09.267103+0000","172.21.15.1:6834/1515309674":"2024-03-07T07:25:07.207409+0000","172.21.15.1:6832/1515309674":"2024-03-07T07:25:07.207409+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.700+0000 7fb356fb5640 1 -- 172.21.15.1:0/2134001684 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb33406b500 msgr2=0x7fb33406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.700+0000 7fb356fb5640 1 --2- 172.21.15.1:0/2134001684 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb33406b500 0x7fb33406d9c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fb3501e8950 tx=0x7fb344009190 comp rx=0 tx=0).stop 2024-03-06T07:25:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.700+0000 7fb356fb5640 1 -- 172.21.15.1:0/2134001684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3500a6f30 msgr2=0x7fb3500a73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.700+0000 7fb356fb5640 1 --2- 172.21.15.1:0/2134001684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3500a6f30 0x7fb3500a73b0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fb33c001970 tx=0x7fb3500c2b20 comp rx=0 tx=0).stop 2024-03-06T07:25:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.700+0000 7fb356fb5640 1 -- 172.21.15.1:0/2134001684 shutdown_connections 2024-03-06T07:25:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.700+0000 7fb356fb5640 1 --2- 172.21.15.1:0/2134001684 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb33406b500 0x7fb33406d9c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.700+0000 7fb356fb5640 1 --2- 172.21.15.1:0/2134001684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3501eb140 0x7fb3501eb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.700+0000 7fb356fb5640 1 --2- 172.21.15.1:0/2134001684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3500a6f30 0x7fb3500a73b0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.700+0000 7fb356fb5640 1 --2- 172.21.15.1:0/2134001684 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb340008850 0x7fb3500a7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.700+0000 7fb356fb5640 1 -- 172.21.15.1:0/2134001684 >> 172.21.15.1:0/2134001684 conn(0x7fb3500c6860 msgr2=0x7fb35014f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.700+0000 7fb356fb5640 1 -- 172.21.15.1:0/2134001684 shutdown_connections 2024-03-06T07:25:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.700+0000 7fb356fb5640 1 -- 172.21.15.1:0/2134001684 wait complete. 2024-03-06T07:25:22.713 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:22.713 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:22.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.817+0000 7f88607f7640 1 Processor -- start 2024-03-06T07:25:22.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.817+0000 7f88607f7640 1 -- start start 2024-03-06T07:25:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.818+0000 7f88607f7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f88580c8cd0 0x7f88580c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.818+0000 7f88607f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8858155b70 0x7f8858159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.818+0000 7f88607f7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88580c2b90 con 0x7f88580c3260 2024-03-06T07:25:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.818+0000 7f88607f7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8858156020 con 0x7f88580c8cd0 2024-03-06T07:25:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.818+0000 7f88607f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88580c37b0 con 0x7f8858155b70 2024-03-06T07:25:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.818+0000 7f885ed94640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8858155b70 0x7f8858159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.818+0000 7f885ed94640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8858155b70 0x7f8858159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:35252/0 (socket says 172.21.15.1:35252) 2024-03-06T07:25:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.818+0000 7f885ed94640 1 -- 172.21.15.1:0/2477468538 learned_addr learned my addr 172.21.15.1:0/2477468538 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:22.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.818+0000 7f885e593640 1 --2- 172.21.15.1:0/2477468538 >> v2:172.21.15.98:3300/0 conn(0x7f88580c8cd0 0x7f88580c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.818+0000 7f885ed94640 1 -- 172.21.15.1:0/2477468538 >> v2:172.21.15.98:3300/0 conn(0x7f88580c8cd0 msgr2=0x7f88580c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:22.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.818+0000 7f885ed94640 1 --2- 172.21.15.1:0/2477468538 >> v2:172.21.15.98:3300/0 conn(0x7f88580c8cd0 0x7f88580c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.818+0000 7f885ed94640 1 -- 172.21.15.1:0/2477468538 >> v1:172.21.15.98:6789/0 conn(0x7f88580c3260 legacy=0x7f88580c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.818+0000 7f885ed94640 1 -- 172.21.15.1:0/2477468538 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88580c3a90 con 0x7f8858155b70 2024-03-06T07:25:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.819+0000 7f885ed94640 1 --2- 172.21.15.1:0/2477468538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8858155b70 0x7f8858159010 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8850009e50 tx=0x7f885002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3abf8f3c01abdae7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.819+0000 7f885d591640 1 -- 172.21.15.1:0/2477468538 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885002ca70 con 0x7f8858155b70 2024-03-06T07:25:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.819+0000 7f885d591640 1 -- 172.21.15.1:0/2477468538 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f885002cc10 con 0x7f8858155b70 2024-03-06T07:25:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.819+0000 7f885d591640 1 -- 172.21.15.1:0/2477468538 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88500343f0 con 0x7f8858155b70 2024-03-06T07:25:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.820+0000 7f88607f7640 1 -- 172.21.15.1:0/2477468538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8858155b70 msgr2=0x7f8858159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.820+0000 7f88607f7640 1 --2- 172.21.15.1:0/2477468538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8858155b70 0x7f8858159010 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8850009e50 tx=0x7f885002bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.820+0000 7f88607f7640 1 -- 172.21.15.1:0/2477468538 shutdown_connections 2024-03-06T07:25:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.820+0000 7f88607f7640 1 --2- 172.21.15.1:0/2477468538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8858155b70 0x7f8858159010 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.820+0000 7f88607f7640 1 --2- 172.21.15.1:0/2477468538 >> v2:172.21.15.98:3300/0 conn(0x7f88580c8cd0 0x7f88580c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.820+0000 7f88607f7640 1 -- 172.21.15.1:0/2477468538 >> 172.21.15.1:0/2477468538 conn(0x7f885806b3e0 msgr2=0x7f885806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.820+0000 7f88607f7640 1 -- 172.21.15.1:0/2477468538 shutdown_connections 2024-03-06T07:25:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.820+0000 7f88607f7640 1 -- 172.21.15.1:0/2477468538 wait complete. 2024-03-06T07:25:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.821+0000 7f88607f7640 1 Processor -- start 2024-03-06T07:25:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.821+0000 7f88607f7640 1 -- start start 2024-03-06T07:25:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f88607f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88580c3260 0x7f88581ee300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f88607f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88580c8cd0 0x7f88581ee840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f88607f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8858155b70 0x7f885814e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f88607f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8858159550 con 0x7f88580c3260 2024-03-06T07:25:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f88607f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8858158af0 con 0x7f88580c8cd0 2024-03-06T07:25:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f88607f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88580c8560 con 0x7f8858155b70 2024-03-06T07:25:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f885e593640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88580c3260 0x7f88581ee300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f885e593640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88580c3260 0x7f88581ee300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35030/0 (socket says 172.21.15.1:35030) 2024-03-06T07:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f885e593640 1 -- 172.21.15.1:0/1179973737 learned_addr learned my addr 172.21.15.1:0/1179973737 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f885ed94640 1 --2- 172.21.15.1:0/1179973737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8858155b70 0x7f885814e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f885e593640 1 -- 172.21.15.1:0/1179973737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8858155b70 msgr2=0x7f885814e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f885dd92640 1 --2- 172.21.15.1:0/1179973737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88580c8cd0 0x7f88581ee840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f885e593640 1 --2- 172.21.15.1:0/1179973737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8858155b70 0x7f885814e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f885e593640 1 -- 172.21.15.1:0/1179973737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88580c8cd0 msgr2=0x7f88581ee840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f885e593640 1 --2- 172.21.15.1:0/1179973737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88580c8cd0 0x7f88581ee840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f885e593640 1 -- 172.21.15.1:0/1179973737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88580a8d50 con 0x7f88580c3260 2024-03-06T07:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f885ed94640 1 --2- 172.21.15.1:0/1179973737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8858155b70 0x7f885814e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f885dd92640 1 --2- 172.21.15.1:0/1179973737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88580c8cd0 0x7f88581ee840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f885e593640 1 --2- 172.21.15.1:0/1179973737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88580c3260 0x7f88581ee300 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f885400e9e0 tx=0x7f885400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f884effd640 1 -- 172.21.15.1:0/1179973737 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8854017070 con 0x7f88580c3260 2024-03-06T07:25:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.822+0000 7f884effd640 1 -- 172.21.15.1:0/1179973737 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f885400c9d0 con 0x7f88580c3260 2024-03-06T07:25:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.823+0000 7f88607f7640 1 -- 172.21.15.1:0/1179973737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88580a8790 con 0x7f88580c3260 2024-03-06T07:25:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.823+0000 7f884effd640 1 -- 172.21.15.1:0/1179973737 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885400cdc0 con 0x7f88580c3260 2024-03-06T07:25:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.823+0000 7f88607f7640 1 -- 172.21.15.1:0/1179973737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88581ec130 con 0x7f88580c3260 2024-03-06T07:25:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.823+0000 7f88607f7640 1 -- 172.21.15.1:0/1179973737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8858067fd0 con 0x7f88580c3260 2024-03-06T07:25:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.824+0000 7f884effd640 1 -- 172.21.15.1:0/1179973737 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f885401a070 con 0x7f88580c3260 2024-03-06T07:25:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.824+0000 7f884effd640 1 --2- 172.21.15.1:0/1179973737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f883c06b4d0 0x7f883c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.824+0000 7f884effd640 1 -- 172.21.15.1:0/1179973737 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f885408a0f0 con 0x7f88580c3260 2024-03-06T07:25:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.825+0000 7f885dd92640 1 --2- 172.21.15.1:0/1179973737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f883c06b4d0 0x7f883c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.825+0000 7f885dd92640 1 --2- 172.21.15.1:0/1179973737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f883c06b4d0 0x7f883c06d990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f88580ab680 tx=0x7f8848009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:22.829+0000 7f884effd640 1 -- 172.21.15.1:0/1179973737 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88540590c0 con 0x7f88580c3260 2024-03-06T07:25:23.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.009+0000 7f88607f7640 1 -- 172.21.15.1:0/1179973737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f88580c2d80 con 0x7f88580c3260 2024-03-06T07:25:23.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.009+0000 7f884effd640 1 -- 172.21.15.1:0/1179973737 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) v1 ==== 93+0+10 (secure 0 0 0) 0x7f88540589c0 con 0x7f88580c3260 2024-03-06T07:25:23.012 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:25:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.011+0000 7f88607f7640 1 -- 172.21.15.1:0/1179973737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f883c06b4d0 msgr2=0x7f883c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.011+0000 7f88607f7640 1 --2- 172.21.15.1:0/1179973737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f883c06b4d0 0x7f883c06d990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f88580ab680 tx=0x7f8848009040 comp rx=0 tx=0).stop 2024-03-06T07:25:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.011+0000 7f88607f7640 1 -- 172.21.15.1:0/1179973737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88580c3260 msgr2=0x7f88581ee300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.011+0000 7f88607f7640 1 --2- 172.21.15.1:0/1179973737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88580c3260 0x7f88581ee300 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f885400e9e0 tx=0x7f885400eea0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.011+0000 7f88607f7640 1 -- 172.21.15.1:0/1179973737 shutdown_connections 2024-03-06T07:25:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.011+0000 7f88607f7640 1 --2- 172.21.15.1:0/1179973737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f883c06b4d0 0x7f883c06d990 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.011+0000 7f88607f7640 1 --2- 172.21.15.1:0/1179973737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8858155b70 0x7f885814e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.011+0000 7f88607f7640 1 --2- 172.21.15.1:0/1179973737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88580c8cd0 0x7f88581ee840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.011+0000 7f88607f7640 1 --2- 172.21.15.1:0/1179973737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88580c3260 0x7f88581ee300 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.011+0000 7f88607f7640 1 -- 172.21.15.1:0/1179973737 >> 172.21.15.1:0/1179973737 conn(0x7f885806b3e0 msgr2=0x7f885806c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.011+0000 7f88607f7640 1 -- 172.21.15.1:0/1179973737 shutdown_connections 2024-03-06T07:25:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.011+0000 7f88607f7640 1 -- 172.21.15.1:0/1179973737 wait complete. 2024-03-06T07:25:23.024 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset' 2024-03-06T07:25:23.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.124+0000 7f76281ac640 1 Processor -- start 2024-03-06T07:25:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.124+0000 7f76281ac640 1 -- start start 2024-03-06T07:25:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.125+0000 7f76281ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76200a8520 0x7f762006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.125+0000 7f76281ac640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f76200c1bc0 0x7f7620153290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.125+0000 7f76281ac640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f762006f060 con 0x7f762006e670 2024-03-06T07:25:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.125+0000 7f76281ac640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76200c2ca0 con 0x7f76200c1bc0 2024-03-06T07:25:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.125+0000 7f76281ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76200c26e0 con 0x7f76200a8520 2024-03-06T07:25:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.125+0000 7f7626749640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f76200c1bc0 0x7f7620153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.125+0000 7f7626749640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f76200c1bc0 0x7f7620153290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35032/0 (socket says 172.21.15.1:35032) 2024-03-06T07:25:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.125+0000 7f7626749640 1 -- 172.21.15.1:0/4268202213 learned_addr learned my addr 172.21.15.1:0/4268202213 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.125+0000 7f7625747640 1 --2- 172.21.15.1:0/4268202213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76200a8520 0x7f762006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.125+0000 7f7626749640 1 -- 172.21.15.1:0/4268202213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76200a8520 msgr2=0x7f762006ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.125+0000 7f7626749640 1 --2- 172.21.15.1:0/4268202213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76200a8520 0x7f762006ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.125+0000 7f7626749640 1 -- 172.21.15.1:0/4268202213 >> v1:172.21.15.98:6789/0 conn(0x7f762006e670 legacy=0x7f7620150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.125+0000 7f7626749640 1 -- 172.21.15.1:0/4268202213 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76200c29c0 con 0x7f76200c1bc0 2024-03-06T07:25:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.126+0000 7f7626749640 1 --2- 172.21.15.1:0/4268202213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76200c1bc0 0x7f7620153290 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f7618009e50 tx=0x7f761802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=545679e82162522b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.126+0000 7f7624f46640 1 -- 172.21.15.1:0/4268202213 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761802c9e0 con 0x7f76200c1bc0 2024-03-06T07:25:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.126+0000 7f7624f46640 1 -- 172.21.15.1:0/4268202213 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761802cb80 con 0x7f76200c1bc0 2024-03-06T07:25:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.126+0000 7f7624f46640 1 -- 172.21.15.1:0/4268202213 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76180343f0 con 0x7f76200c1bc0 2024-03-06T07:25:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.127+0000 7f76281ac640 1 -- 172.21.15.1:0/4268202213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76200c1bc0 msgr2=0x7f7620153290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.127+0000 7f76281ac640 1 --2- 172.21.15.1:0/4268202213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76200c1bc0 0x7f7620153290 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f7618009e50 tx=0x7f761802be40 comp rx=0 tx=0).stop 2024-03-06T07:25:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.127+0000 7f76281ac640 1 -- 172.21.15.1:0/4268202213 shutdown_connections 2024-03-06T07:25:23.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.127+0000 7f76281ac640 1 --2- 172.21.15.1:0/4268202213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76200c1bc0 0x7f7620153290 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.127+0000 7f76281ac640 1 --2- 172.21.15.1:0/4268202213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76200a8520 0x7f762006ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.127+0000 7f76281ac640 1 -- 172.21.15.1:0/4268202213 >> 172.21.15.1:0/4268202213 conn(0x7f76200c3cc0 msgr2=0x7f76200c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.127+0000 7f76281ac640 1 -- 172.21.15.1:0/4268202213 shutdown_connections 2024-03-06T07:25:23.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.127+0000 7f76281ac640 1 -- 172.21.15.1:0/4268202213 wait complete. 2024-03-06T07:25:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.128+0000 7f76281ac640 1 Processor -- start 2024-03-06T07:25:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.128+0000 7f76281ac640 1 -- start start 2024-03-06T07:25:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.128+0000 7f76281ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f762006e670 0x7f76200c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.128+0000 7f76281ac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76200a8520 0x7f76200c4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.128+0000 7f76281ac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76200c1bc0 0x7f76200ab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.128+0000 7f76281ac640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7620153950 con 0x7f76200a8520 2024-03-06T07:25:23.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.128+0000 7f76281ac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76201537d0 con 0x7f76200c1bc0 2024-03-06T07:25:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.128+0000 7f76281ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7620153ad0 con 0x7f762006e670 2024-03-06T07:25:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.129+0000 7f7625747640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76200a8520 0x7f76200c4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.129+0000 7f7625747640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76200a8520 0x7f76200c4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35038/0 (socket says 172.21.15.1:35038) 2024-03-06T07:25:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.129+0000 7f7625747640 1 -- 172.21.15.1:0/649777573 learned_addr learned my addr 172.21.15.1:0/649777573 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.129+0000 7f7625f48640 1 --2- 172.21.15.1:0/649777573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f762006e670 0x7f76200c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.129+0000 7f7626749640 1 --2- 172.21.15.1:0/649777573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76200c1bc0 0x7f76200ab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.129+0000 7f7625747640 1 -- 172.21.15.1:0/649777573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f762006e670 msgr2=0x7f76200c6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.129+0000 7f7625747640 1 --2- 172.21.15.1:0/649777573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f762006e670 0x7f76200c6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.129+0000 7f7625747640 1 -- 172.21.15.1:0/649777573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76200c1bc0 msgr2=0x7f76200ab6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.129+0000 7f7625747640 1 --2- 172.21.15.1:0/649777573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76200c1bc0 0x7f76200ab6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.129+0000 7f7625747640 1 -- 172.21.15.1:0/649777573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76200c52f0 con 0x7f76200a8520 2024-03-06T07:25:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.129+0000 7f7626749640 1 --2- 172.21.15.1:0/649777573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76200c1bc0 0x7f76200ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.129+0000 7f7625747640 1 --2- 172.21.15.1:0/649777573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76200a8520 0x7f76200c4db0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f761000c7f0 tx=0x7f761000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.129+0000 7f76167fc640 1 -- 172.21.15.1:0/649777573 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7610015070 con 0x7f76200a8520 2024-03-06T07:25:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.129+0000 7f76167fc640 1 -- 172.21.15.1:0/649777573 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7610004220 con 0x7f76200a8520 2024-03-06T07:25:23.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.129+0000 7f76281ac640 1 -- 172.21.15.1:0/649777573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762014f7c0 con 0x7f76200a8520 2024-03-06T07:25:23.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.129+0000 7f76167fc640 1 -- 172.21.15.1:0/649777573 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7610007620 con 0x7f76200a8520 2024-03-06T07:25:23.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.129+0000 7f76281ac640 1 -- 172.21.15.1:0/649777573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7620150e70 con 0x7f76200a8520 2024-03-06T07:25:23.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.130+0000 7f76281ac640 1 -- 172.21.15.1:0/649777573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f762014fce0 con 0x7f76200a8520 2024-03-06T07:25:23.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.131+0000 7f76167fc640 1 -- 172.21.15.1:0/649777573 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f76100079d0 con 0x7f76200a8520 2024-03-06T07:25:23.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.131+0000 7f76167fc640 1 --2- 172.21.15.1:0/649777573 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f760406b4d0 0x7f760406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.131+0000 7f76167fc640 1 -- 172.21.15.1:0/649777573 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f761008a420 con 0x7f76200a8520 2024-03-06T07:25:23.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.131+0000 7f7625f48640 1 --2- 172.21.15.1:0/649777573 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f760406b4d0 0x7f760406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.132+0000 7f7625f48640 1 --2- 172.21.15.1:0/649777573 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f760406b4d0 0x7f760406d990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f76200c0f40 tx=0x7f761c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.135+0000 7f76167fc640 1 -- 172.21.15.1:0/649777573 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7610058d20 con 0x7f76200a8520 2024-03-06T07:25:23.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:23.387+0000 7f76281ac640 1 -- 172.21.15.1:0/649777573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]} v 0) v1 -- 0x7f7620067fd0 con 0x7f76200a8520 2024-03-06T07:25:24.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.383+0000 7f76167fc640 1 -- 172.21.15.1:0/649777573 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]}]=0 v300) v1 ==== 147+0+0 (secure 0 0 0) 0x7f761005cdc0 con 0x7f76200a8520 2024-03-06T07:25:24.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.386+0000 7f76281ac640 1 -- 172.21.15.1:0/649777573 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f760406b4d0 msgr2=0x7f760406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.386+0000 7f76281ac640 1 --2- 172.21.15.1:0/649777573 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f760406b4d0 0x7f760406d990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f76200c0f40 tx=0x7f761c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:24.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.386+0000 7f76281ac640 1 -- 172.21.15.1:0/649777573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76200a8520 msgr2=0x7f76200c4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.386+0000 7f76281ac640 1 --2- 172.21.15.1:0/649777573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76200a8520 0x7f76200c4db0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f761000c7f0 tx=0x7f761000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.386+0000 7f76281ac640 1 -- 172.21.15.1:0/649777573 shutdown_connections 2024-03-06T07:25:24.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.386+0000 7f76281ac640 1 --2- 172.21.15.1:0/649777573 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f760406b4d0 0x7f760406d990 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.386+0000 7f76281ac640 1 --2- 172.21.15.1:0/649777573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76200c1bc0 0x7f76200ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.386+0000 7f76281ac640 1 --2- 172.21.15.1:0/649777573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76200a8520 0x7f76200c4db0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.386+0000 7f76281ac640 1 --2- 172.21.15.1:0/649777573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f762006e670 0x7f76200c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.386+0000 7f76281ac640 1 -- 172.21.15.1:0/649777573 >> 172.21.15.1:0/649777573 conn(0x7f76200c3cc0 msgr2=0x7f762006f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.386+0000 7f76281ac640 1 -- 172.21.15.1:0/649777573 shutdown_connections 2024-03-06T07:25:24.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.386+0000 7f76281ac640 1 -- 172.21.15.1:0/649777573 wait complete. 2024-03-06T07:25:24.401 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:25:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.510+0000 7f4752702640 1 Processor -- start 2024-03-06T07:25:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.510+0000 7f4752702640 1 -- start start 2024-03-06T07:25:24.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.510+0000 7f4752702640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f474c0c3260 0x7f474c0c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.511+0000 7f4752702640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f474c155b70 0x7f474c159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.511+0000 7f4752702640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474c0c2b90 con 0x7f474c0c8cd0 2024-03-06T07:25:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.511+0000 7f4752702640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f474c156020 con 0x7f474c0c3260 2024-03-06T07:25:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.511+0000 7f4752702640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474c0c37b0 con 0x7f474c155b70 2024-03-06T07:25:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.511+0000 7f474b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f474c0c3260 0x7f474c0c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.511+0000 7f474b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f474c0c3260 0x7f474c0c90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52984/0 (socket says 172.21.15.1:52984) 2024-03-06T07:25:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.511+0000 7f474b7fe640 1 -- 172.21.15.1:0/3582968628 learned_addr learned my addr 172.21.15.1:0/3582968628 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.511+0000 7f4750c9f640 1 --2- 172.21.15.1:0/3582968628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f474c155b70 0x7f474c159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.511+0000 7f474b7fe640 1 -- 172.21.15.1:0/3582968628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f474c155b70 msgr2=0x7f474c159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.511+0000 7f474b7fe640 1 --2- 172.21.15.1:0/3582968628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f474c155b70 0x7f474c159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.511+0000 7f474b7fe640 1 -- 172.21.15.1:0/3582968628 >> v1:172.21.15.98:6789/0 conn(0x7f474c0c8cd0 legacy=0x7f474c0c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.511+0000 7f474b7fe640 1 -- 172.21.15.1:0/3582968628 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c0c3a90 con 0x7f474c0c3260 2024-03-06T07:25:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.511+0000 7f4750c9f640 1 --2- 172.21.15.1:0/3582968628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f474c155b70 0x7f474c159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.512+0000 7f474b7fe640 1 --2- 172.21.15.1:0/3582968628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474c0c3260 0x7f474c0c90b0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f473c009870 tx=0x7f473c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cbaa500d0649b01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.512+0000 7f474affd640 1 -- 172.21.15.1:0/3582968628 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c0317c0 con 0x7f474c0c3260 2024-03-06T07:25:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.512+0000 7f474affd640 1 -- 172.21.15.1:0/3582968628 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473c037070 con 0x7f474c0c3260 2024-03-06T07:25:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.512+0000 7f474affd640 1 -- 172.21.15.1:0/3582968628 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c031b80 con 0x7f474c0c3260 2024-03-06T07:25:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.512+0000 7f4752702640 1 -- 172.21.15.1:0/3582968628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474c0c3260 msgr2=0x7f474c0c90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.512+0000 7f4752702640 1 --2- 172.21.15.1:0/3582968628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474c0c3260 0x7f474c0c90b0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f473c009870 tx=0x7f473c02b860 comp rx=0 tx=0).stop 2024-03-06T07:25:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.512+0000 7f4752702640 1 -- 172.21.15.1:0/3582968628 shutdown_connections 2024-03-06T07:25:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.512+0000 7f4752702640 1 --2- 172.21.15.1:0/3582968628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f474c155b70 0x7f474c159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.512+0000 7f4752702640 1 --2- 172.21.15.1:0/3582968628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474c0c3260 0x7f474c0c90b0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.512+0000 7f4752702640 1 -- 172.21.15.1:0/3582968628 >> 172.21.15.1:0/3582968628 conn(0x7f474c06b3e0 msgr2=0x7f474c06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.512+0000 7f4752702640 1 -- 172.21.15.1:0/3582968628 shutdown_connections 2024-03-06T07:25:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.512+0000 7f4752702640 1 -- 172.21.15.1:0/3582968628 wait complete. 2024-03-06T07:25:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.514+0000 7f4752702640 1 Processor -- start 2024-03-06T07:25:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.514+0000 7f4752702640 1 -- start start 2024-03-06T07:25:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.514+0000 7f4752702640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f474c0c3260 0x7f474c1eb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.514+0000 7f4752702640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474c0c8cd0 0x7f474c0ad2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.514+0000 7f4752702640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f474c155b70 0x7f474c1dfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.514+0000 7f4752702640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c159550 con 0x7f474c0c8cd0 2024-03-06T07:25:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.514+0000 7f4752702640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c158af0 con 0x7f474c0c3260 2024-03-06T07:25:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.514+0000 7f4752702640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474c0c8560 con 0x7f474c155b70 2024-03-06T07:25:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.515+0000 7f474b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474c0c8cd0 0x7f474c0ad2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.515+0000 7f474b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474c0c8cd0 0x7f474c0ad2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52998/0 (socket says 172.21.15.1:52998) 2024-03-06T07:25:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.515+0000 7f474b7fe640 1 -- 172.21.15.1:0/1137723048 learned_addr learned my addr 172.21.15.1:0/1137723048 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.515+0000 7f4750c9f640 1 --2- 172.21.15.1:0/1137723048 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f474c155b70 0x7f474c1dfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.515+0000 7f474bfff640 1 --2- 172.21.15.1:0/1137723048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f474c0c3260 0x7f474c1eb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.515+0000 7f474b7fe640 1 -- 172.21.15.1:0/1137723048 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f474c155b70 msgr2=0x7f474c1dfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.515+0000 7f474b7fe640 1 --2- 172.21.15.1:0/1137723048 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f474c155b70 0x7f474c1dfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.515+0000 7f474b7fe640 1 -- 172.21.15.1:0/1137723048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f474c0c3260 msgr2=0x7f474c1eb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.515+0000 7f474b7fe640 1 --2- 172.21.15.1:0/1137723048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f474c0c3260 0x7f474c1eb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.515+0000 7f474b7fe640 1 -- 172.21.15.1:0/1137723048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c0a84c0 con 0x7f474c0c8cd0 2024-03-06T07:25:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.515+0000 7f474bfff640 1 --2- 172.21.15.1:0/1137723048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f474c0c3260 0x7f474c1eb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.515+0000 7f474b7fe640 1 --2- 172.21.15.1:0/1137723048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474c0c8cd0 0x7f474c0ad2b0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f473c000c00 tx=0x7f473c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.515+0000 7f4748ff9640 1 -- 172.21.15.1:0/1137723048 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c032860 con 0x7f474c0c8cd0 2024-03-06T07:25:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.515+0000 7f4748ff9640 1 -- 172.21.15.1:0/1137723048 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473c032a00 con 0x7f474c0c8cd0 2024-03-06T07:25:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.515+0000 7f4752702640 1 -- 172.21.15.1:0/1137723048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f474c1e11c0 con 0x7f474c0c8cd0 2024-03-06T07:25:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.515+0000 7f4748ff9640 1 -- 172.21.15.1:0/1137723048 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c032df0 con 0x7f474c0c8cd0 2024-03-06T07:25:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.515+0000 7f4752702640 1 -- 172.21.15.1:0/1137723048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f474c1e14a0 con 0x7f474c0c8cd0 2024-03-06T07:25:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.516+0000 7f4752702640 1 -- 172.21.15.1:0/1137723048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f474c067fd0 con 0x7f474c0c8cd0 2024-03-06T07:25:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.517+0000 7f4748ff9640 1 -- 172.21.15.1:0/1137723048 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f473c04c060 con 0x7f474c0c8cd0 2024-03-06T07:25:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.517+0000 7f4748ff9640 1 --2- 172.21.15.1:0/1137723048 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f473806b500 0x7f473806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.517+0000 7f4748ff9640 1 -- 172.21.15.1:0/1137723048 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f473c0ab920 con 0x7f474c0c8cd0 2024-03-06T07:25:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.518+0000 7f474bfff640 1 --2- 172.21.15.1:0/1137723048 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f473806b500 0x7f473806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.518+0000 7f474bfff640 1 --2- 172.21.15.1:0/1137723048 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f473806b500 0x7f473806d9c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f474c0c6340 tx=0x7f4740009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.522+0000 7f4748ff9640 1 -- 172.21.15.1:0/1137723048 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f473c040070 con 0x7f474c0c8cd0 2024-03-06T07:25:24.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.702+0000 7f4752702640 1 -- 172.21.15.1:0/1137723048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f474c0c2d80 con 0x7f474c0c8cd0 2024-03-06T07:25:24.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.706+0000 7f4748ff9640 1 -- 172.21.15.1:0/1137723048 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v45)=0 v45) v1 ==== 122+0+0 (secure 0 0 0) 0x7f473c07aa10 con 0x7f474c0c8cd0 2024-03-06T07:25:24.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.708+0000 7f4752702640 1 -- 172.21.15.1:0/1137723048 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f473806b500 msgr2=0x7f473806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.708+0000 7f4752702640 1 --2- 172.21.15.1:0/1137723048 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f473806b500 0x7f473806d9c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f474c0c6340 tx=0x7f4740009040 comp rx=0 tx=0).stop 2024-03-06T07:25:24.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.708+0000 7f4752702640 1 -- 172.21.15.1:0/1137723048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474c0c8cd0 msgr2=0x7f474c0ad2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.708+0000 7f4752702640 1 --2- 172.21.15.1:0/1137723048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474c0c8cd0 0x7f474c0ad2b0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f473c000c00 tx=0x7f473c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.708+0000 7f4752702640 1 -- 172.21.15.1:0/1137723048 shutdown_connections 2024-03-06T07:25:24.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.708+0000 7f4752702640 1 --2- 172.21.15.1:0/1137723048 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f473806b500 0x7f473806d9c0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.708+0000 7f4752702640 1 --2- 172.21.15.1:0/1137723048 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f474c155b70 0x7f474c1dfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.708+0000 7f4752702640 1 --2- 172.21.15.1:0/1137723048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474c0c8cd0 0x7f474c0ad2b0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.708+0000 7f4752702640 1 --2- 172.21.15.1:0/1137723048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f474c0c3260 0x7f474c1eb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.708+0000 7f4752702640 1 -- 172.21.15.1:0/1137723048 >> 172.21.15.1:0/1137723048 conn(0x7f474c06b3e0 msgr2=0x7f474c06c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.708+0000 7f4752702640 1 -- 172.21.15.1:0/1137723048 shutdown_connections 2024-03-06T07:25:24.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.708+0000 7f4752702640 1 -- 172.21.15.1:0/1137723048 wait complete. 2024-03-06T07:25:24.721 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:24.721 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:24.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:24.818 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:24.818 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:25:24.818 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:24.833 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:25:24.833 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:25:24.833 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:24.850 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 none /run/credentials/systemd-tmpfiles-clean.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 2024-03-06T07:25:24.850 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:25:24.850 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:24.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.972+0000 7f727bbc7640 1 Processor -- start 2024-03-06T07:25:24.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.972+0000 7f727bbc7640 1 -- start start 2024-03-06T07:25:24.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.973+0000 7f727bbc7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f72740c2b30 0x7f72740724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.973+0000 7f727bbc7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f72740a5fa0 0x7f7274072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.973+0000 7f727bbc7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7274151dd0 con 0x7f7274152b70 2024-03-06T07:25:24.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.973+0000 7f727bbc7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72741526e0 con 0x7f72740a5fa0 2024-03-06T07:25:24.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.973+0000 7f727bbc7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7274156190 con 0x7f72740c2b30 2024-03-06T07:25:24.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.973+0000 7f727a164640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f72740a5fa0 0x7f7274072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.973+0000 7f7279162640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7274152b70 0x7f72740c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32938/0 (socket says 172.21.15.1:32938) 2024-03-06T07:25:24.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.973+0000 7f7279162640 1 -- 172.21.15.1:0/3637244551 learned_addr learned my addr 172.21.15.1:0/3637244551 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:24.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.973+0000 7f727a164640 1 -- 172.21.15.1:0/3637244551 >> v2:172.21.15.98:3300/0 conn(0x7f72740c2b30 msgr2=0x7f72740724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.973+0000 7f727a164640 1 --2- 172.21.15.1:0/3637244551 >> v2:172.21.15.98:3300/0 conn(0x7f72740c2b30 0x7f72740724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.973+0000 7f727a164640 1 -- 172.21.15.1:0/3637244551 >> v1:172.21.15.1:6789/0 conn(0x7f7274152b70 legacy=0x7f72740c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.973+0000 7f727a164640 1 -- 172.21.15.1:0/3637244551 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7274152090 con 0x7f72740a5fa0 2024-03-06T07:25:24.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.974+0000 7f727a164640 1 --2- 172.21.15.1:0/3637244551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72740a5fa0 0x7f7274072a00 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f72740c6570 tx=0x7f727002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9be9ff4bcfb405 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.974+0000 7f7278961640 1 -- 172.21.15.1:0/3637244551 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7270039070 con 0x7f72740a5fa0 2024-03-06T07:25:24.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.974+0000 7f7278961640 1 -- 172.21.15.1:0/3637244551 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727002cc80 con 0x7f72740a5fa0 2024-03-06T07:25:24.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.974+0000 7f7278961640 1 -- 172.21.15.1:0/3637244551 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7270034660 con 0x7f72740a5fa0 2024-03-06T07:25:24.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.975+0000 7f727bbc7640 1 -- 172.21.15.1:0/3637244551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72740a5fa0 msgr2=0x7f7274072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.975+0000 7f727bbc7640 1 --2- 172.21.15.1:0/3637244551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72740a5fa0 0x7f7274072a00 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f72740c6570 tx=0x7f727002bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.975+0000 7f727bbc7640 1 -- 172.21.15.1:0/3637244551 shutdown_connections 2024-03-06T07:25:24.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.975+0000 7f727bbc7640 1 --2- 172.21.15.1:0/3637244551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72740a5fa0 0x7f7274072a00 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.975+0000 7f727bbc7640 1 --2- 172.21.15.1:0/3637244551 >> v2:172.21.15.98:3300/0 conn(0x7f72740c2b30 0x7f72740724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.975+0000 7f727bbc7640 1 -- 172.21.15.1:0/3637244551 >> 172.21.15.1:0/3637244551 conn(0x7f72740c6860 msgr2=0x7f72740c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.975+0000 7f727bbc7640 1 -- 172.21.15.1:0/3637244551 shutdown_connections 2024-03-06T07:25:24.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.975+0000 7f727bbc7640 1 -- 172.21.15.1:0/3637244551 wait complete. 2024-03-06T07:25:24.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.976+0000 7f727bbc7640 1 Processor -- start 2024-03-06T07:25:24.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.976+0000 7f727bbc7640 1 -- start start 2024-03-06T07:25:24.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f727bbc7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72740a5fa0 0x7f727414aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f727bbc7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72740c2b30 0x7f7274149c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f727bbc7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7274152b70 0x7f72741447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f727bbc7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7274155f00 con 0x7f72740c2b30 2024-03-06T07:25:24.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f727bbc7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7274155d80 con 0x7f7274152b70 2024-03-06T07:25:24.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f727bbc7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72740c3a00 con 0x7f72740a5fa0 2024-03-06T07:25:24.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f7279162640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72740c2b30 0x7f7274149c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f7279162640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72740c2b30 0x7f7274149c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53006/0 (socket says 172.21.15.1:53006) 2024-03-06T07:25:24.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f7279162640 1 -- 172.21.15.1:0/2753094346 learned_addr learned my addr 172.21.15.1:0/2753094346 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:24.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f7279162640 1 -- 172.21.15.1:0/2753094346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72740a5fa0 msgr2=0x7f727414aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f7279963640 1 --2- 172.21.15.1:0/2753094346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72740a5fa0 0x7f727414aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f727a164640 1 --2- 172.21.15.1:0/2753094346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7274152b70 0x7f72741447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f7279162640 1 --2- 172.21.15.1:0/2753094346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72740a5fa0 0x7f727414aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f7279162640 1 -- 172.21.15.1:0/2753094346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7274152b70 msgr2=0x7f72741447b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f7279162640 1 --2- 172.21.15.1:0/2753094346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7274152b70 0x7f72741447b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f7279162640 1 -- 172.21.15.1:0/2753094346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7274154c80 con 0x7f72740c2b30 2024-03-06T07:25:24.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f7279963640 1 --2- 172.21.15.1:0/2753094346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72740a5fa0 0x7f727414aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:24.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f7279162640 1 --2- 172.21.15.1:0/2753094346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72740c2b30 0x7f7274149c50 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f726400c3c0 tx=0x7f726400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.977+0000 7f726a7fc640 1 -- 172.21.15.1:0/2753094346 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7264017070 con 0x7f72740c2b30 2024-03-06T07:25:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.978+0000 7f726a7fc640 1 -- 172.21.15.1:0/2753094346 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7264005ce0 con 0x7f72740c2b30 2024-03-06T07:25:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.978+0000 7f727bbc7640 1 -- 172.21.15.1:0/2753094346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72741543e0 con 0x7f72740c2b30 2024-03-06T07:25:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.978+0000 7f726a7fc640 1 -- 172.21.15.1:0/2753094346 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72640072f0 con 0x7f72740c2b30 2024-03-06T07:25:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.978+0000 7f727bbc7640 1 -- 172.21.15.1:0/2753094346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72741549a0 con 0x7f72740c2b30 2024-03-06T07:25:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.978+0000 7f727bbc7640 1 -- 172.21.15.1:0/2753094346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7274067fd0 con 0x7f72740c2b30 2024-03-06T07:25:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.979+0000 7f726a7fc640 1 -- 172.21.15.1:0/2753094346 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f726401a070 con 0x7f72740c2b30 2024-03-06T07:25:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.979+0000 7f726a7fc640 1 --2- 172.21.15.1:0/2753094346 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f725806b4d0 0x7f725806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.979+0000 7f726a7fc640 1 -- 172.21.15.1:0/2753094346 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f7264089450 con 0x7f72740c2b30 2024-03-06T07:25:24.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.980+0000 7f7279963640 1 --2- 172.21.15.1:0/2753094346 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f725806b4d0 0x7f725806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.980+0000 7f7279963640 1 --2- 172.21.15.1:0/2753094346 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f725806b4d0 0x7f725806d990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f726c006fd0 tx=0x7f726c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:24.984+0000 7f726a7fc640 1 -- 172.21.15.1:0/2753094346 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7264058420 con 0x7f72740c2b30 2024-03-06T07:25:25.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.204+0000 7f727bbc7640 1 -- 172.21.15.1:0/2753094346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72741462c0 con 0x7f72740c2b30 2024-03-06T07:25:25.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.204+0000 7f726a7fc640 1 -- 172.21.15.1:0/2753094346 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3695 (secure 0 0 0) 0x7f7264057d20 con 0x7f72740c2b30 2024-03-06T07:25:25.206 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:25.206 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":117,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6446,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4151599120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":4151599120},{"type":"v1","addr":"172.21.15.1:6835","nonce":4151599120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/155600632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":155600632},{"type":"v1","addr":"172.21.15.98:6837","nonce":155600632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":6587,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1600553134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1600553134},{"type":"v1","addr":"172.21.15.98:6836","nonce":1600553134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":8214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3919152259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3919152259},{"type":"v1","addr":"172.21.15.1:6834","nonce":3919152259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":117}],"filesystems":[]} 2024-03-06T07:25:25.207 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 117 2024-03-06T07:25:25.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.206+0000 7f727bbc7640 1 -- 172.21.15.1:0/2753094346 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f725806b4d0 msgr2=0x7f725806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.206+0000 7f727bbc7640 1 --2- 172.21.15.1:0/2753094346 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f725806b4d0 0x7f725806d990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f726c006fd0 tx=0x7f726c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.206+0000 7f727bbc7640 1 -- 172.21.15.1:0/2753094346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72740c2b30 msgr2=0x7f7274149c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.206+0000 7f727bbc7640 1 --2- 172.21.15.1:0/2753094346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72740c2b30 0x7f7274149c50 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f726400c3c0 tx=0x7f726400c880 comp rx=0 tx=0).stop 2024-03-06T07:25:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.206+0000 7f727bbc7640 1 -- 172.21.15.1:0/2753094346 shutdown_connections 2024-03-06T07:25:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.206+0000 7f727bbc7640 1 --2- 172.21.15.1:0/2753094346 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f725806b4d0 0x7f725806d990 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.206+0000 7f727bbc7640 1 --2- 172.21.15.1:0/2753094346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7274152b70 0x7f72741447b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.206+0000 7f727bbc7640 1 --2- 172.21.15.1:0/2753094346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72740c2b30 0x7f7274149c50 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.206+0000 7f727bbc7640 1 --2- 172.21.15.1:0/2753094346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72740a5fa0 0x7f727414aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.206+0000 7f727bbc7640 1 -- 172.21.15.1:0/2753094346 >> 172.21.15.1:0/2753094346 conn(0x7f72740c6860 msgr2=0x7f727406e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.206+0000 7f727bbc7640 1 -- 172.21.15.1:0/2753094346 shutdown_connections 2024-03-06T07:25:25.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.206+0000 7f727bbc7640 1 -- 172.21.15.1:0/2753094346 wait complete. 2024-03-06T07:25:25.219 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:25:25.219 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:25:25.220 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:25:25.220 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:25.220 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:25:25.220 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:25:25.220 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:25:25.220 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:25.220 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:25:25.221 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:25:25.221 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:25:25.221 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:25.221 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:25:25.221 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:25:25.221 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:25:25.221 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:25.229 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:25:25.229 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:25:25.230 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:25:25.230 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:25:25.231 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:25:25.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:25:25.232 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:25:25.232 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:25:25.271 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:25:25.271 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:25.272 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:25:25.272 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:25.274 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:25:25.274 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:25.275 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:25:25.275 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:25.275 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:25:25.349 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:25:25.349 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:25:25.352 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:25:25.352 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:25:25.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.379+0000 7fbe6f28d640 1 Processor -- start 2024-03-06T07:25:25.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.379+0000 7fbe6f28d640 1 -- start start 2024-03-06T07:25:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.379+0000 7fbe6f28d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbe680c1260 0x7fbe680c16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.379+0000 7fbe6f28d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe681566e0 0x7fbe68156b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.379+0000 7fbe6f28d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6806f5c0 con 0x7fbe681505b0 2024-03-06T07:25:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.379+0000 7fbe6f28d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe680c4e70 con 0x7fbe680c1260 2024-03-06T07:25:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.379+0000 7fbe6f28d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68152f80 con 0x7fbe681566e0 2024-03-06T07:25:25.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.380+0000 7fbe6c828640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbe680c1260 0x7fbe680c16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.380+0000 7fbe6d82a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe681566e0 0x7fbe68156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.380+0000 7fbe6d029640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbe681505b0 0x7fbe680c4760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:47960/0 (socket says 172.21.15.1:47960) 2024-03-06T07:25:25.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.380+0000 7fbe6d029640 1 -- 172.21.15.1:0/1375925710 learned_addr learned my addr 172.21.15.1:0/1375925710 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:25.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.380+0000 7fbe5ffff640 1 -- 172.21.15.1:0/1375925710 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547224676 0 0) 0x7fbe6806f5c0 con 0x7fbe681505b0 2024-03-06T07:25:25.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.380+0000 7fbe6d82a640 1 -- 172.21.15.1:0/1375925710 >> v2:172.21.15.98:3300/0 conn(0x7fbe680c1260 msgr2=0x7fbe680c16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:25.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.380+0000 7fbe6d82a640 1 --2- 172.21.15.1:0/1375925710 >> v2:172.21.15.98:3300/0 conn(0x7fbe680c1260 0x7fbe680c16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.380+0000 7fbe6d82a640 1 -- 172.21.15.1:0/1375925710 >> v1:172.21.15.98:6789/0 conn(0x7fbe681505b0 legacy=0x7fbe680c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.380+0000 7fbe6d82a640 1 -- 172.21.15.1:0/1375925710 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe68150990 con 0x7fbe681566e0 2024-03-06T07:25:25.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.381+0000 7fbe6d82a640 1 --2- 172.21.15.1:0/1375925710 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe681566e0 0x7fbe68156b70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbe60009d90 tx=0x7fbe6002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cec5c7b6fa61c47c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.381+0000 7fbe5ffff640 1 -- 172.21.15.1:0/1375925710 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe60039070 con 0x7fbe681566e0 2024-03-06T07:25:25.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.381+0000 7fbe5ffff640 1 -- 172.21.15.1:0/1375925710 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6002cbc0 con 0x7fbe681566e0 2024-03-06T07:25:25.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.381+0000 7fbe5ffff640 1 -- 172.21.15.1:0/1375925710 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe60034440 con 0x7fbe681566e0 2024-03-06T07:25:25.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.381+0000 7fbe6f28d640 1 -- 172.21.15.1:0/1375925710 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe681566e0 msgr2=0x7fbe68156b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.382+0000 7fbe6f28d640 1 --2- 172.21.15.1:0/1375925710 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe681566e0 0x7fbe68156b70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbe60009d90 tx=0x7fbe6002be10 comp rx=0 tx=0).stop 2024-03-06T07:25:25.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.382+0000 7fbe6f28d640 1 -- 172.21.15.1:0/1375925710 shutdown_connections 2024-03-06T07:25:25.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.382+0000 7fbe6f28d640 1 --2- 172.21.15.1:0/1375925710 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe681566e0 0x7fbe68156b70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.382+0000 7fbe6f28d640 1 --2- 172.21.15.1:0/1375925710 >> v2:172.21.15.98:3300/0 conn(0x7fbe680c1260 0x7fbe680c16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.382+0000 7fbe6f28d640 1 -- 172.21.15.1:0/1375925710 >> 172.21.15.1:0/1375925710 conn(0x7fbe680c6860 msgr2=0x7fbe680c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:25.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.382+0000 7fbe6f28d640 1 -- 172.21.15.1:0/1375925710 shutdown_connections 2024-03-06T07:25:25.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.382+0000 7fbe6f28d640 1 -- 172.21.15.1:0/1375925710 wait complete. 2024-03-06T07:25:25.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.383+0000 7fbe6f28d640 1 Processor -- start 2024-03-06T07:25:25.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.383+0000 7fbe6f28d640 1 -- start start 2024-03-06T07:25:25.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.383+0000 7fbe6f28d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe680c1260 0x7fbe681472e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.383+0000 7fbe6f28d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe681505b0 0x7fbe681dc850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.383+0000 7fbe6f28d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe681566e0 0x7fbe68154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.383+0000 7fbe6f28d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68153800 con 0x7fbe681566e0 2024-03-06T07:25:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.383+0000 7fbe6f28d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68153520 con 0x7fbe680c1260 2024-03-06T07:25:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.383+0000 7fbe6f28d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68153ae0 con 0x7fbe681505b0 2024-03-06T07:25:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.383+0000 7fbe6d82a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe681566e0 0x7fbe68154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.384+0000 7fbe6c828640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe681505b0 0x7fbe681dc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.384+0000 7fbe6d82a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe681566e0 0x7fbe68154af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53064/0 (socket says 172.21.15.1:53064) 2024-03-06T07:25:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.384+0000 7fbe6d82a640 1 -- 172.21.15.1:0/1809266456 learned_addr learned my addr 172.21.15.1:0/1809266456 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.384+0000 7fbe6d029640 1 --2- 172.21.15.1:0/1809266456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe680c1260 0x7fbe681472e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.384+0000 7fbe6d82a640 1 -- 172.21.15.1:0/1809266456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe681505b0 msgr2=0x7fbe681dc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.384+0000 7fbe6d82a640 1 --2- 172.21.15.1:0/1809266456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe681505b0 0x7fbe681dc850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.384+0000 7fbe6d82a640 1 -- 172.21.15.1:0/1809266456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe680c1260 msgr2=0x7fbe681472e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.384+0000 7fbe6d82a640 1 --2- 172.21.15.1:0/1809266456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe680c1260 0x7fbe681472e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.384+0000 7fbe6d82a640 1 -- 172.21.15.1:0/1809266456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe681ddfc0 con 0x7fbe681566e0 2024-03-06T07:25:25.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.384+0000 7fbe6c828640 1 --2- 172.21.15.1:0/1809266456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe681505b0 0x7fbe681dc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:25.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.384+0000 7fbe6d82a640 1 --2- 172.21.15.1:0/1809266456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe681566e0 0x7fbe68154af0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fbe60009d90 tx=0x7fbe60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.387+0000 7fbe5dffb640 1 -- 172.21.15.1:0/1809266456 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe60039050 con 0x7fbe681566e0 2024-03-06T07:25:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.387+0000 7fbe5dffb640 1 -- 172.21.15.1:0/1809266456 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe60045070 con 0x7fbe681566e0 2024-03-06T07:25:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.387+0000 7fbe5dffb640 1 -- 172.21.15.1:0/1809266456 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe60009450 con 0x7fbe681566e0 2024-03-06T07:25:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.387+0000 7fbe6f28d640 1 -- 172.21.15.1:0/1809266456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe68147000 con 0x7fbe681566e0 2024-03-06T07:25:25.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.387+0000 7fbe6f28d640 1 -- 172.21.15.1:0/1809266456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe681e98b0 con 0x7fbe681566e0 2024-03-06T07:25:25.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.388+0000 7fbe6f28d640 1 -- 172.21.15.1:0/1809266456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe68154620 con 0x7fbe681566e0 2024-03-06T07:25:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.391+0000 7fbe5dffb640 1 -- 172.21.15.1:0/1809266456 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe6003e850 con 0x7fbe681566e0 2024-03-06T07:25:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.392+0000 7fbe5dffb640 1 --2- 172.21.15.1:0/1809266456 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe4c06b500 0x7fbe4c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.392+0000 7fbe5dffb640 1 -- 172.21.15.1:0/1809266456 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fbe600ad850 con 0x7fbe681566e0 2024-03-06T07:25:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.392+0000 7fbe6d029640 1 --2- 172.21.15.1:0/1809266456 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe4c06b500 0x7fbe4c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.393+0000 7fbe6d029640 1 --2- 172.21.15.1:0/1809266456 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe4c06b500 0x7fbe4c06d9c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fbe64000f60 tx=0x7fbe64007b70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.395+0000 7fbe5dffb640 1 -- 172.21.15.1:0/1809266456 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe6007c8a0 con 0x7fbe681566e0 2024-03-06T07:25:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:25.590+0000 7fbe6f28d640 1 -- 172.21.15.1:0/1809266456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fbe6814fd60 con 0x7fbe681566e0 2024-03-06T07:25:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.401+0000 7fbe5dffb640 1 -- 172.21.15.1:0/1809266456 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v131) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbe6007c1a0 con 0x7fbe681566e0 2024-03-06T07:25:26.403 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-06T07:25:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.404+0000 7fbe6f28d640 1 -- 172.21.15.1:0/1809266456 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe4c06b500 msgr2=0x7fbe4c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.404+0000 7fbe6f28d640 1 --2- 172.21.15.1:0/1809266456 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe4c06b500 0x7fbe4c06d9c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fbe64000f60 tx=0x7fbe64007b70 comp rx=0 tx=0).stop 2024-03-06T07:25:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.404+0000 7fbe6f28d640 1 -- 172.21.15.1:0/1809266456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe681566e0 msgr2=0x7fbe68154af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.404+0000 7fbe6f28d640 1 --2- 172.21.15.1:0/1809266456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe681566e0 0x7fbe68154af0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fbe60009d90 tx=0x7fbe60004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.404+0000 7fbe6f28d640 1 -- 172.21.15.1:0/1809266456 shutdown_connections 2024-03-06T07:25:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.405+0000 7fbe6f28d640 1 --2- 172.21.15.1:0/1809266456 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe4c06b500 0x7fbe4c06d9c0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.405+0000 7fbe6f28d640 1 --2- 172.21.15.1:0/1809266456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe681566e0 0x7fbe68154af0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.405+0000 7fbe6f28d640 1 --2- 172.21.15.1:0/1809266456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe681505b0 0x7fbe681dc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.405+0000 7fbe6f28d640 1 --2- 172.21.15.1:0/1809266456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe680c1260 0x7fbe681472e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.405+0000 7fbe6f28d640 1 -- 172.21.15.1:0/1809266456 >> 172.21.15.1:0/1809266456 conn(0x7fbe680c6860 msgr2=0x7fbe680c58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.405+0000 7fbe6f28d640 1 -- 172.21.15.1:0/1809266456 shutdown_connections 2024-03-06T07:25:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.405+0000 7fbe6f28d640 1 -- 172.21.15.1:0/1809266456 wait complete. 2024-03-06T07:25:26.420 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:25:26.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.526+0000 7fa2cf0ed640 1 Processor -- start 2024-03-06T07:25:26.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.526+0000 7fa2cf0ed640 1 -- start start 2024-03-06T07:25:26.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2cf0ed640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2c8149a90 0x7fa2c80c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2cf0ed640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa2c80c1260 0x7fa2c80c16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2cf0ed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c81565a0 0x7fa2c8156a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2cf0ed640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c81530a0 con 0x7fa2c8149a90 2024-03-06T07:25:26.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2cf0ed640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c8153380 con 0x7fa2c80c1260 2024-03-06T07:25:26.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2cf0ed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c8153660 con 0x7fa2c81565a0 2024-03-06T07:25:26.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2cce89640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2c8149a90 0x7fa2c80c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2cce89640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2c8149a90 0x7fa2c80c4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53078/0 (socket says 172.21.15.1:53078) 2024-03-06T07:25:26.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2cce89640 1 -- 172.21.15.1:0/3910194169 learned_addr learned my addr 172.21.15.1:0/3910194169 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:26.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2cd68a640 1 --2- 172.21.15.1:0/3910194169 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c81565a0 0x7fa2c8156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2c7fff640 1 --2- 172.21.15.1:0/3910194169 >> v2:172.21.15.98:3300/0 conn(0x7fa2c80c1260 0x7fa2c80c16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2cce89640 1 -- 172.21.15.1:0/3910194169 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c81565a0 msgr2=0x7fa2c8156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:26.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2cce89640 1 --2- 172.21.15.1:0/3910194169 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c81565a0 0x7fa2c8156a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2cce89640 1 -- 172.21.15.1:0/3910194169 >> v2:172.21.15.98:3300/0 conn(0x7fa2c80c1260 msgr2=0x7fa2c80c16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:26.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2cce89640 1 --2- 172.21.15.1:0/3910194169 >> v2:172.21.15.98:3300/0 conn(0x7fa2c80c1260 0x7fa2c80c16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2cce89640 1 -- 172.21.15.1:0/3910194169 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c8153940 con 0x7fa2c8149a90 2024-03-06T07:25:26.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2cd68a640 1 --2- 172.21.15.1:0/3910194169 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c81565a0 0x7fa2c8156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:26.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.527+0000 7fa2c7fff640 1 --2- 172.21.15.1:0/3910194169 >> v2:172.21.15.98:3300/0 conn(0x7fa2c80c1260 0x7fa2c80c16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:26.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.528+0000 7fa2cce89640 1 --2- 172.21.15.1:0/3910194169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c8149a90 0x7fa2c80c4760 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fa2c80a6520 tx=0x7fa2bc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b6e4afbfe6ef62 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.528+0000 7fa2c77fe640 1 -- 172.21.15.1:0/3910194169 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc02e070 con 0x7fa2c8149a90 2024-03-06T07:25:26.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.528+0000 7fa2c77fe640 1 -- 172.21.15.1:0/3910194169 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2bc037070 con 0x7fa2c8149a90 2024-03-06T07:25:26.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.528+0000 7fa2c77fe640 1 -- 172.21.15.1:0/3910194169 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc031a00 con 0x7fa2c8149a90 2024-03-06T07:25:26.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.528+0000 7fa2cf0ed640 1 -- 172.21.15.1:0/3910194169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c8149a90 msgr2=0x7fa2c80c4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.529+0000 7fa2cf0ed640 1 --2- 172.21.15.1:0/3910194169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c8149a90 0x7fa2c80c4760 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fa2c80a6520 tx=0x7fa2bc02b880 comp rx=0 tx=0).stop 2024-03-06T07:25:26.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.529+0000 7fa2cf0ed640 1 -- 172.21.15.1:0/3910194169 shutdown_connections 2024-03-06T07:25:26.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.529+0000 7fa2cf0ed640 1 --2- 172.21.15.1:0/3910194169 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c81565a0 0x7fa2c8156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.529+0000 7fa2cf0ed640 1 --2- 172.21.15.1:0/3910194169 >> v2:172.21.15.98:3300/0 conn(0x7fa2c80c1260 0x7fa2c80c16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.529+0000 7fa2cf0ed640 1 --2- 172.21.15.1:0/3910194169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c8149a90 0x7fa2c80c4760 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.529+0000 7fa2cf0ed640 1 -- 172.21.15.1:0/3910194169 >> 172.21.15.1:0/3910194169 conn(0x7fa2c80c6860 msgr2=0x7fa2c80c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.529+0000 7fa2cf0ed640 1 -- 172.21.15.1:0/3910194169 shutdown_connections 2024-03-06T07:25:26.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.529+0000 7fa2cf0ed640 1 -- 172.21.15.1:0/3910194169 wait complete. 2024-03-06T07:25:26.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.530+0000 7fa2cf0ed640 1 Processor -- start 2024-03-06T07:25:26.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.530+0000 7fa2cf0ed640 1 -- start start 2024-03-06T07:25:26.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cf0ed640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c80c1260 0x7fa2c814cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cf0ed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c8149a90 0x7fa2c80abd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cf0ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c81565a0 0x7fa2c81e78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cf0ed640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c806f4a0 con 0x7fa2c80c1260 2024-03-06T07:25:26.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cf0ed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c80c4ca0 con 0x7fa2c81565a0 2024-03-06T07:25:26.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cf0ed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c806f620 con 0x7fa2c8149a90 2024-03-06T07:25:26.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cce89640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c80c1260 0x7fa2c814cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cce89640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c80c1260 0x7fa2c814cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53086/0 (socket says 172.21.15.1:53086) 2024-03-06T07:25:26.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cce89640 1 -- 172.21.15.1:0/2216705179 learned_addr learned my addr 172.21.15.1:0/2216705179 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:26.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2c7fff640 1 --2- 172.21.15.1:0/2216705179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c8149a90 0x7fa2c80abd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cd68a640 1 --2- 172.21.15.1:0/2216705179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c81565a0 0x7fa2c81e78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cce89640 1 -- 172.21.15.1:0/2216705179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c8149a90 msgr2=0x7fa2c80abd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cce89640 1 --2- 172.21.15.1:0/2216705179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c8149a90 0x7fa2c80abd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cce89640 1 -- 172.21.15.1:0/2216705179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c81565a0 msgr2=0x7fa2c81e78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cce89640 1 --2- 172.21.15.1:0/2216705179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c81565a0 0x7fa2c81e78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cce89640 1 -- 172.21.15.1:0/2216705179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c8073420 con 0x7fa2c80c1260 2024-03-06T07:25:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cd68a640 1 --2- 172.21.15.1:0/2216705179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c81565a0 0x7fa2c81e78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cce89640 1 --2- 172.21.15.1:0/2216705179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c80c1260 0x7fa2c814cd60 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc000c00 tx=0x7fa2bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2c57fa640 1 -- 172.21.15.1:0/2216705179 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc032880 con 0x7fa2c80c1260 2024-03-06T07:25:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2c57fa640 1 -- 172.21.15.1:0/2216705179 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2bc032a20 con 0x7fa2c80c1260 2024-03-06T07:25:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2c57fa640 1 -- 172.21.15.1:0/2216705179 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc032e10 con 0x7fa2c80c1260 2024-03-06T07:25:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cf0ed640 1 -- 172.21.15.1:0/2216705179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c8072090 con 0x7fa2c80c1260 2024-03-06T07:25:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.531+0000 7fa2cf0ed640 1 -- 172.21.15.1:0/2216705179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c8072650 con 0x7fa2c80c1260 2024-03-06T07:25:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.533+0000 7fa2cf0ed640 1 -- 172.21.15.1:0/2216705179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c8067fd0 con 0x7fa2c80c1260 2024-03-06T07:25:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.533+0000 7fa2c57fa640 1 -- 172.21.15.1:0/2216705179 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2bc04c060 con 0x7fa2c80c1260 2024-03-06T07:25:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.533+0000 7fa2c57fa640 1 --2- 172.21.15.1:0/2216705179 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2ac06b4d0 0x7fa2ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.534+0000 7fa2c57fa640 1 -- 172.21.15.1:0/2216705179 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fa2bc0abc00 con 0x7fa2c80c1260 2024-03-06T07:25:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.534+0000 7fa2c7fff640 1 --2- 172.21.15.1:0/2216705179 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2ac06b4d0 0x7fa2ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.534+0000 7fa2c7fff640 1 --2- 172.21.15.1:0/2216705179 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2ac06b4d0 0x7fa2ac06d990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa2c814d2a0 tx=0x7fa2b4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.538+0000 7fa2c57fa640 1 -- 172.21.15.1:0/2216705179 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2bc0b2080 con 0x7fa2c80c1260 2024-03-06T07:25:26.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.786+0000 7fa2cf0ed640 1 -- 172.21.15.1:0/2216705179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa2c80c6360 con 0x7fa2c80c1260 2024-03-06T07:25:26.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.786+0000 7fa2c57fa640 1 -- 172.21.15.1:0/2216705179 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+4724 (secure 0 0 0) 0x7fa2bc040070 con 0x7fa2c80c1260 2024-03-06T07:25:26.789 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:26.789 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:25:26.789 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-06T07:25:26.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:26.789 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-06T07:25:26.789 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlnxyWARAAKIPXU0IafNQ1RzBPX+3DYw==", 2024-03-06T07:25:26.789 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:25:26.789 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:25:26.789 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:25:26.789 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:25:26.789 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:25:26.789 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:25:26.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:26.790 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-06T07:25:26.790 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl2fXxDRAAuLqH+HpKcN1JTts9ZtDTXA==", 2024-03-06T07:25:26.790 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:25:26.790 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:25:26.790 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:25:26.790 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:25:26.790 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:25:26.790 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:25:26.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:26.790 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-06T07:25:26.790 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl+IzqCBAAmgLkarFGt9Ihs/8kJYicBQ==", 2024-03-06T07:25:26.790 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:25:26.791 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:25:26.791 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:25:26.791 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:25:26.791 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:25:26.791 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.791 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:25:26.791 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:26.791 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-06T07:25:26.791 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlipdRFRAAA34vrsTEoX3L1CKb1YAV9g==", 2024-03-06T07:25:26.791 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:25:26.791 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:25:26.791 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:25:26.791 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:25:26.791 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:25:26.791 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:25:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-06T07:25:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAGGehlQK39LxAAOpPIZvl3FqQwCgyo3yoOGQ==", 2024-03-06T07:25:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:25:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:25:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:25:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:25:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-06T07:25:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAJGehlTxwDLhAAMwN9E0+3Jh5XNXnLo+fw3g==", 2024-03-06T07:25:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:25:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:26.793 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:25:26.793 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:25:26.793 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:25:26.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:26.793 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-06T07:25:26.793 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAMGehl4qrlLBAAm0VWQRNvSQIpDx5xf4Fyig==", 2024-03-06T07:25:26.793 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:25:26.793 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:26.793 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:25:26.793 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:25:26.793 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:25:26.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:26.793 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-06T07:25:26.794 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAPGehllO5HKxAAozm6ycG+fRxatoZTogeJTA==", 2024-03-06T07:25:26.794 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:25:26.794 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:26.794 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:25:26.794 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:25:26.794 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:25:26.794 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:26.794 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-06T07:25:26.794 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAVGehlLtLBMhAAUrV6q6ZCQluIcoo/S/NdbA==", 2024-03-06T07:25:26.794 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:25:26.794 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:26.794 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:25:26.794 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:25:26.794 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:25:26.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:26.795 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-06T07:25:26.795 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAYGehlHp59MRAAxxgpSPTEZkYKjl3fr0xeeQ==", 2024-03-06T07:25:26.795 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:25:26.795 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:26.795 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:25:26.795 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:25:26.795 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:25:26.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:26.795 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-06T07:25:26.795 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAbGehlTnXcMBAAFOBqcnh5ubAz+o6AGjccWQ==", 2024-03-06T07:25:26.795 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:25:26.795 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:26.796 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:25:26.796 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:25:26.796 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:25:26.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:26.796 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-06T07:25:26.796 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAeGehl1qH1LhAAaSl7Aj7qlN6d4xt5SKAmcg==", 2024-03-06T07:25:26.796 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:25:26.796 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:26.796 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:25:26.796 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:25:26.796 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:25:26.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:26.796 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-06T07:25:26.796 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng==", 2024-03-06T07:25:26.797 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:25:26.797 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:25:26.797 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-06T07:25:26.797 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:26.797 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:25:26.797 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:26.797 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-06T07:25:26.797 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlMyv7HBAAi7vt7aZWEaZuMpFrB5ofZw==", 2024-03-06T07:25:26.797 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:25:26.797 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:25:26.797 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-06T07:25:26.797 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-06T07:25:26.797 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-06T07:25:26.797 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:25:26.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:26.798 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-06T07:25:26.798 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlBS/vIBAAN86N2K4ze5PJB8kR869Lfg==", 2024-03-06T07:25:26.798 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:25:26.798 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:25:26.798 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:25:26.798 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:25:26.798 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:25:26.798 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:25:26.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:26.798 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-06T07:25:26.798 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlJ/jPORAAhLk5MTGafsGKhZSYhvhBZg==", 2024-03-06T07:25:26.798 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:25:26.798 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:25:26.799 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:26.799 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:25:26.799 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:25:26.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:26.799 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-06T07:25:26.799 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlAl8rNhAAvdU0Qvo/FJYnxc4/H8MIqQ==", 2024-03-06T07:25:26.799 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:25:26.799 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:25:26.799 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:26.799 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:25:26.799 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.799 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:26.799 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-06T07:25:26.799 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:25:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.789+0000 7fa2cf0ed640 1 -- 172.21.15.1:0/2216705179 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2ac06b4d0 msgr2=0x7fa2ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.789+0000 7fa2cf0ed640 1 --2- 172.21.15.1:0/2216705179 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2ac06b4d0 0x7fa2ac06d990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa2c814d2a0 tx=0x7fa2b4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.789+0000 7fa2cf0ed640 1 -- 172.21.15.1:0/2216705179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c80c1260 msgr2=0x7fa2c814cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.789+0000 7fa2cf0ed640 1 --2- 172.21.15.1:0/2216705179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c80c1260 0x7fa2c814cd60 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc000c00 tx=0x7fa2bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.789+0000 7fa2cf0ed640 1 -- 172.21.15.1:0/2216705179 shutdown_connections 2024-03-06T07:25:26.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.789+0000 7fa2cf0ed640 1 --2- 172.21.15.1:0/2216705179 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2ac06b4d0 0x7fa2ac06d990 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.789+0000 7fa2cf0ed640 1 --2- 172.21.15.1:0/2216705179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c81565a0 0x7fa2c81e78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.789+0000 7fa2cf0ed640 1 --2- 172.21.15.1:0/2216705179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c8149a90 0x7fa2c80abd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.789+0000 7fa2cf0ed640 1 --2- 172.21.15.1:0/2216705179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c80c1260 0x7fa2c814cd60 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.789+0000 7fa2cf0ed640 1 -- 172.21.15.1:0/2216705179 >> 172.21.15.1:0/2216705179 conn(0x7fa2c80c6860 msgr2=0x7fa2c80c6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.789+0000 7fa2cf0ed640 1 -- 172.21.15.1:0/2216705179 shutdown_connections 2024-03-06T07:25:26.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.789+0000 7fa2cf0ed640 1 -- 172.21.15.1:0/2216705179 wait complete. 2024-03-06T07:25:26.802 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:26.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.902+0000 7f129424e640 1 Processor -- start 2024-03-06T07:25:26.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.902+0000 7f129424e640 1 -- start start 2024-03-06T07:25:26.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f129424e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f128c072f20 0x7f128c1513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f129424e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c0a6140 con 0x7f128c150fe0 2024-03-06T07:25:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f129424e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c151aa0 con 0x7f128c071f90 2024-03-06T07:25:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f129424e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f128c1526b0 con 0x7f128c072f20 2024-03-06T07:25:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f1291fea640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f128c150fe0 0x7f128c06edb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32952/0 (socket says 172.21.15.1:32952) 2024-03-06T07:25:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f1291fea640 1 -- 172.21.15.1:0/2690760913 learned_addr learned my addr 172.21.15.1:0/2690760913 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f12917e9640 1 --2- 172.21.15.1:0/2690760913 >> v2:172.21.15.98:3300/0 conn(0x7f128c072f20 0x7f128c1513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f1290fe8640 1 -- 172.21.15.1:0/2690760913 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188943788 0 0) 0x7f128c0a6140 con 0x7f128c150fe0 2024-03-06T07:25:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f1290fe8640 1 -- 172.21.15.1:0/2690760913 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1278003520 con 0x7f128c150fe0 2024-03-06T07:25:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f1290fe8640 1 -- 172.21.15.1:0/2690760913 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2922021342 0 0) 0x7f1278003520 con 0x7f128c150fe0 2024-03-06T07:25:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f1290fe8640 1 -- 172.21.15.1:0/2690760913 >> v2:172.21.15.98:3300/0 conn(0x7f128c072f20 msgr2=0x7f128c1513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f1290fe8640 1 --2- 172.21.15.1:0/2690760913 >> v2:172.21.15.98:3300/0 conn(0x7f128c072f20 0x7f128c1513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f1290fe8640 1 -- 172.21.15.1:0/2690760913 >> v1:172.21.15.98:6789/0 conn(0x7f128c071f90 legacy=0x7f128c072400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f1290fe8640 1 -- 172.21.15.1:0/2690760913 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c151d80 con 0x7f128c150fe0 2024-03-06T07:25:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f1290fe8640 1 -- 172.21.15.1:0/2690760913 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f1288001670 con 0x7f128c150fe0 2024-03-06T07:25:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f1290fe8640 1 -- 172.21.15.1:0/2690760913 >> v1:172.21.15.1:6789/0 conn(0x7f128c150fe0 legacy=0x7f128c06edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f1290fe8640 1 --2- 172.21.15.1:0/2690760913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12780039b0 0x7f1278003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f1290fe8640 1 --2- 172.21.15.1:0/2690760913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12780043b0 0x7f1278004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f1290fe8640 1 --2- 172.21.15.1:0/2690760913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1278005120 0x7f127800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.903+0000 7f1290fe8640 1 -- 172.21.15.1:0/2690760913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127800c030 con 0x7f12780039b0 2024-03-06T07:25:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.904+0000 7f1290fe8640 1 -- 172.21.15.1:0/2690760913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127800c1e0 con 0x7f12780043b0 2024-03-06T07:25:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.904+0000 7f1290fe8640 1 -- 172.21.15.1:0/2690760913 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f127800c3e0 con 0x7f1278005120 2024-03-06T07:25:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.904+0000 7f1291fea640 1 --2- 172.21.15.1:0/2690760913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12780039b0 0x7f1278003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.904+0000 7f12917e9640 1 --2- 172.21.15.1:0/2690760913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12780043b0 0x7f1278004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.904+0000 7f12927eb640 1 --2- 172.21.15.1:0/2690760913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1278005120 0x7f127800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.904+0000 7f1291fea640 1 -- 172.21.15.1:0/2690760913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1278005120 msgr2=0x7f127800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.904+0000 7f1291fea640 1 --2- 172.21.15.1:0/2690760913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1278005120 0x7f127800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.904+0000 7f1291fea640 1 -- 172.21.15.1:0/2690760913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12780043b0 msgr2=0x7f1278004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:26.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.904+0000 7f1291fea640 1 --2- 172.21.15.1:0/2690760913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12780043b0 0x7f1278004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.904+0000 7f1291fea640 1 -- 172.21.15.1:0/2690760913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1278003520 con 0x7f12780039b0 2024-03-06T07:25:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.904+0000 7f12927eb640 1 --2- 172.21.15.1:0/2690760913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1278005120 0x7f127800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.904+0000 7f12927eb640 1 -- 172.21.15.1:0/2690760913 reap_dead start 2024-03-06T07:25:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.905+0000 7f1291fea640 1 --2- 172.21.15.1:0/2690760913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12780039b0 0x7f1278003d90 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f128800d1f0 tx=0x7f1278005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5691cd9ba66799 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.905+0000 7f1290fe8640 1 -- 172.21.15.1:0/2690760913 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128800c040 con 0x7f12780039b0 2024-03-06T07:25:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.905+0000 7f1290fe8640 1 -- 172.21.15.1:0/2690760913 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1288031070 con 0x7f12780039b0 2024-03-06T07:25:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.905+0000 7f1290fe8640 1 -- 172.21.15.1:0/2690760913 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1288005b10 con 0x7f12780039b0 2024-03-06T07:25:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.905+0000 7f129424e640 1 -- 172.21.15.1:0/2690760913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12780039b0 msgr2=0x7f1278003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.905+0000 7f129424e640 1 --2- 172.21.15.1:0/2690760913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12780039b0 0x7f1278003d90 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f128800d1f0 tx=0x7f1278005830 comp rx=0 tx=0).stop 2024-03-06T07:25:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.905+0000 7f129424e640 1 -- 172.21.15.1:0/2690760913 shutdown_connections 2024-03-06T07:25:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.905+0000 7f129424e640 1 --2- 172.21.15.1:0/2690760913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12780039b0 0x7f1278003d90 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.905+0000 7f129424e640 1 -- 172.21.15.1:0/2690760913 >> 172.21.15.1:0/2690760913 conn(0x7f128c0c6860 msgr2=0x7f128c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.905+0000 7f129424e640 1 -- 172.21.15.1:0/2690760913 shutdown_connections 2024-03-06T07:25:26.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.905+0000 7f129424e640 1 -- 172.21.15.1:0/2690760913 wait complete. 2024-03-06T07:25:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.907+0000 7f129424e640 1 Processor -- start 2024-03-06T07:25:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.907+0000 7f129424e640 1 -- start start 2024-03-06T07:25:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.907+0000 7f129424e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12780039b0 0x7f128c1e9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.907+0000 7f129424e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f128c0bd440 0x7f128c14f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.907+0000 7f129424e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f128c1e9c70 0x7f128c1ea120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.907+0000 7f129424e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128c1513c0 con 0x7f128c1e9c70 2024-03-06T07:25:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.907+0000 7f129424e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128c152340 con 0x7f12780039b0 2024-03-06T07:25:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.907+0000 7f129424e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f128c151540 con 0x7f128c0bd440 2024-03-06T07:25:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.907+0000 7f12927eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f128c1e9c70 0x7f128c1ea120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.907+0000 7f12927eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f128c1e9c70 0x7f128c1ea120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53110/0 (socket says 172.21.15.1:53110) 2024-03-06T07:25:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.907+0000 7f12927eb640 1 -- 172.21.15.1:0/2326873351 learned_addr learned my addr 172.21.15.1:0/2326873351 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.908+0000 7f1291fea640 1 --2- 172.21.15.1:0/2326873351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12780039b0 0x7f128c1e9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.908+0000 7f12917e9640 1 --2- 172.21.15.1:0/2326873351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f128c0bd440 0x7f128c14f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.908+0000 7f12927eb640 1 -- 172.21.15.1:0/2326873351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f128c0bd440 msgr2=0x7f128c14f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.908+0000 7f12927eb640 1 --2- 172.21.15.1:0/2326873351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f128c0bd440 0x7f128c14f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.908+0000 7f12927eb640 1 -- 172.21.15.1:0/2326873351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12780039b0 msgr2=0x7f128c1e9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.908+0000 7f12927eb640 1 --2- 172.21.15.1:0/2326873351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12780039b0 0x7f128c1e9280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.908+0000 7f12927eb640 1 -- 172.21.15.1:0/2326873351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c1ea660 con 0x7f128c1e9c70 2024-03-06T07:25:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.908+0000 7f12927eb640 1 --2- 172.21.15.1:0/2326873351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f128c1e9c70 0x7f128c1ea120 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f128c06fa30 tx=0x7f128400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.908+0000 7f12827fc640 1 -- 172.21.15.1:0/2326873351 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128400ee50 con 0x7f128c1e9c70 2024-03-06T07:25:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.908+0000 7f12827fc640 1 -- 172.21.15.1:0/2326873351 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1284018070 con 0x7f128c1e9c70 2024-03-06T07:25:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.908+0000 7f129424e640 1 -- 172.21.15.1:0/2326873351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f128c0bba20 con 0x7f128c1e9c70 2024-03-06T07:25:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.908+0000 7f12827fc640 1 -- 172.21.15.1:0/2326873351 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1284003dc0 con 0x7f128c1e9c70 2024-03-06T07:25:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.908+0000 7f129424e640 1 -- 172.21.15.1:0/2326873351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f128c0bbfe0 con 0x7f128c1e9c70 2024-03-06T07:25:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.909+0000 7f129424e640 1 -- 172.21.15.1:0/2326873351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f128c067fd0 con 0x7f128c1e9c70 2024-03-06T07:25:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.910+0000 7f12827fc640 1 -- 172.21.15.1:0/2326873351 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f12840050d0 con 0x7f128c1e9c70 2024-03-06T07:25:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.910+0000 7f12827fc640 1 --2- 172.21.15.1:0/2326873351 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f127006b500 0x7f127006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.910+0000 7f12827fc640 1 -- 172.21.15.1:0/2326873351 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f1284089d60 con 0x7f128c1e9c70 2024-03-06T07:25:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.910+0000 7f1291fea640 1 --2- 172.21.15.1:0/2326873351 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f127006b500 0x7f127006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.911+0000 7f1291fea640 1 --2- 172.21.15.1:0/2326873351 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f127006b500 0x7f127006d9c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f1288035000 tx=0x7f128803a040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:26.914+0000 7f12827fc640 1 -- 172.21.15.1:0/2326873351 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f128c1ea660 con 0x7f128c1e9c70 2024-03-06T07:25:27.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.097+0000 7f129424e640 1 -- 172.21.15.1:0/2326873351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f128c06eb90 con 0x7f128c1e9c70 2024-03-06T07:25:27.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.098+0000 7f12827fc640 1 -- 172.21.15.1:0/2326873351 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+13054 (secure 0 0 0) 0x7f1284058e30 con 0x7f128c1e9c70 2024-03-06T07:25:27.100 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:27.100 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":131,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:25:26.400846+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:27.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.100+0000 7f129424e640 1 -- 172.21.15.1:0/2326873351 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f127006b500 msgr2=0x7f127006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.100+0000 7f129424e640 1 --2- 172.21.15.1:0/2326873351 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f127006b500 0x7f127006d9c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f1288035000 tx=0x7f128803a040 comp rx=0 tx=0).stop 2024-03-06T07:25:27.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.100+0000 7f129424e640 1 -- 172.21.15.1:0/2326873351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f128c1e9c70 msgr2=0x7f128c1ea120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.100+0000 7f129424e640 1 --2- 172.21.15.1:0/2326873351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f128c1e9c70 0x7f128c1ea120 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f128c06fa30 tx=0x7f128400ec00 comp rx=0 tx=0).stop 2024-03-06T07:25:27.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.100+0000 7f129424e640 1 -- 172.21.15.1:0/2326873351 shutdown_connections 2024-03-06T07:25:27.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.100+0000 7f129424e640 1 --2- 172.21.15.1:0/2326873351 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f127006b500 0x7f127006d9c0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.100+0000 7f129424e640 1 --2- 172.21.15.1:0/2326873351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f128c1e9c70 0x7f128c1ea120 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.100+0000 7f129424e640 1 --2- 172.21.15.1:0/2326873351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f128c0bd440 0x7f128c14f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.100+0000 7f129424e640 1 --2- 172.21.15.1:0/2326873351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12780039b0 0x7f128c1e9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.100+0000 7f129424e640 1 -- 172.21.15.1:0/2326873351 >> 172.21.15.1:0/2326873351 conn(0x7f128c0c6860 msgr2=0x7f128c0c59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.100+0000 7f129424e640 1 -- 172.21.15.1:0/2326873351 shutdown_connections 2024-03-06T07:25:27.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.100+0000 7f129424e640 1 -- 172.21.15.1:0/2326873351 wait complete. 2024-03-06T07:25:27.113 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:27.113 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:27.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.215+0000 7f5c2d719640 1 Processor -- start 2024-03-06T07:25:27.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.215+0000 7f5c2d719640 1 -- start start 2024-03-06T07:25:27.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c2d719640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5c28152690 0x7f5c28155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c2d719640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c280c6730 0x7f5c280c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c2d719640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c281561e0 con 0x7f5c280c5d10 2024-03-06T07:25:27.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c2d719640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c280c3300 con 0x7f5c28152690 2024-03-06T07:25:27.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c2d719640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c280c35a0 con 0x7f5c280c6730 2024-03-06T07:25:27.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c277fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5c280c5d10 0x7f5c28156cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32956/0 (socket says 172.21.15.1:32956) 2024-03-06T07:25:27.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c277fe640 1 -- 172.21.15.1:0/2090826952 learned_addr learned my addr 172.21.15.1:0/2090826952 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:27.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c26ffd640 1 --2- 172.21.15.1:0/2090826952 >> v2:172.21.15.98:3300/0 conn(0x7f5c28152690 0x7f5c28155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c25ffb640 1 -- 172.21.15.1:0/2090826952 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070684126 0 0) 0x7f5c18001490 con 0x7f5c280c5d10 2024-03-06T07:25:27.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c25ffb640 1 -- 172.21.15.1:0/2090826952 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c14003520 con 0x7f5c280c5d10 2024-03-06T07:25:27.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c267fc640 1 --2- 172.21.15.1:0/2090826952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c280c6730 0x7f5c280c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c25ffb640 1 -- 172.21.15.1:0/2090826952 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3089617176 0 0) 0x7f5c14003520 con 0x7f5c280c5d10 2024-03-06T07:25:27.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c25ffb640 1 -- 172.21.15.1:0/2090826952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c280c6730 msgr2=0x7f5c280c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c25ffb640 1 --2- 172.21.15.1:0/2090826952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c280c6730 0x7f5c280c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c25ffb640 1 -- 172.21.15.1:0/2090826952 >> v2:172.21.15.98:3300/0 conn(0x7f5c28152690 msgr2=0x7f5c28155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c25ffb640 1 --2- 172.21.15.1:0/2090826952 >> v2:172.21.15.98:3300/0 conn(0x7f5c28152690 0x7f5c28155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c25ffb640 1 -- 172.21.15.1:0/2090826952 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c280c3880 con 0x7f5c280c5d10 2024-03-06T07:25:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c25ffb640 1 -- 172.21.15.1:0/2090826952 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f5c18001bc0 con 0x7f5c280c5d10 2024-03-06T07:25:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c25ffb640 1 -- 172.21.15.1:0/2090826952 >> v1:172.21.15.1:6789/0 conn(0x7f5c280c5d10 legacy=0x7f5c28156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c25ffb640 1 --2- 172.21.15.1:0/2090826952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c140039b0 0x7f5c14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c25ffb640 1 --2- 172.21.15.1:0/2090826952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c280c6730 0x7f5c14004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c25ffb640 1 --2- 172.21.15.1:0/2090826952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c14004d40 0x7f5c140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c25ffb640 1 -- 172.21.15.1:0/2090826952 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c280c3300 con 0x7f5c140039b0 2024-03-06T07:25:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c25ffb640 1 -- 172.21.15.1:0/2090826952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c280c35a0 con 0x7f5c14004d40 2024-03-06T07:25:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.216+0000 7f5c25ffb640 1 -- 172.21.15.1:0/2090826952 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1400b9c0 con 0x7f5c280c6730 2024-03-06T07:25:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.217+0000 7f5c26ffd640 1 --2- 172.21.15.1:0/2090826952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c140039b0 0x7f5c14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.217+0000 7f5c267fc640 1 --2- 172.21.15.1:0/2090826952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c280c6730 0x7f5c14004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.217+0000 7f5c277fe640 1 --2- 172.21.15.1:0/2090826952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c14004d40 0x7f5c140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.217+0000 7f5c26ffd640 1 -- 172.21.15.1:0/2090826952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c280c6730 msgr2=0x7f5c14004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.217+0000 7f5c26ffd640 1 --2- 172.21.15.1:0/2090826952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c280c6730 0x7f5c14004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.217+0000 7f5c26ffd640 1 -- 172.21.15.1:0/2090826952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c14004d40 msgr2=0x7f5c140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.217+0000 7f5c26ffd640 1 --2- 172.21.15.1:0/2090826952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c14004d40 0x7f5c140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.217+0000 7f5c26ffd640 1 -- 172.21.15.1:0/2090826952 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c14003520 con 0x7f5c140039b0 2024-03-06T07:25:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.217+0000 7f5c277fe640 1 --2- 172.21.15.1:0/2090826952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c14004d40 0x7f5c140051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.218+0000 7f5c26ffd640 1 --2- 172.21.15.1:0/2090826952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c140039b0 0x7f5c14003dd0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f5c20009ac0 tx=0x7f5c14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a6734403193e1f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.218+0000 7f5c25ffb640 1 -- 172.21.15.1:0/2090826952 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c20039070 con 0x7f5c140039b0 2024-03-06T07:25:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.218+0000 7f5c25ffb640 1 -- 172.21.15.1:0/2090826952 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2002c710 con 0x7f5c140039b0 2024-03-06T07:25:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.218+0000 7f5c25ffb640 1 -- 172.21.15.1:0/2090826952 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c20009690 con 0x7f5c140039b0 2024-03-06T07:25:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.218+0000 7f5c2d719640 1 -- 172.21.15.1:0/2090826952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c140039b0 msgr2=0x7f5c14003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.218+0000 7f5c2d719640 1 --2- 172.21.15.1:0/2090826952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c140039b0 0x7f5c14003dd0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f5c20009ac0 tx=0x7f5c14005830 comp rx=0 tx=0).stop 2024-03-06T07:25:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.218+0000 7f5c277fe640 1 -- 172.21.15.1:0/2090826952 reap_dead start 2024-03-06T07:25:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.218+0000 7f5c2d719640 1 -- 172.21.15.1:0/2090826952 shutdown_connections 2024-03-06T07:25:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.218+0000 7f5c2d719640 1 -- 172.21.15.1:0/2090826952 >> 172.21.15.1:0/2090826952 conn(0x7f5c2806b410 msgr2=0x7f5c2806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.218+0000 7f5c2d719640 1 -- 172.21.15.1:0/2090826952 shutdown_connections 2024-03-06T07:25:27.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.218+0000 7f5c2d719640 1 -- 172.21.15.1:0/2090826952 wait complete. 2024-03-06T07:25:27.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.220+0000 7f5c2d719640 1 Processor -- start 2024-03-06T07:25:27.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.220+0000 7f5c2d719640 1 -- start start 2024-03-06T07:25:27.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.220+0000 7f5c2d719640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c140039b0 0x7f5c28154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.220+0000 7f5c2d719640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c2814f540 0x7f5c281506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.220+0000 7f5c2d719640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c281e1520 0x7f5c281e19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.220+0000 7f5c2d719640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c28152a70 con 0x7f5c2814f540 2024-03-06T07:25:27.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.220+0000 7f5c2d719640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c281567b0 con 0x7f5c281e1520 2024-03-06T07:25:27.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.220+0000 7f5c2d719640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c28152bf0 con 0x7f5c140039b0 2024-03-06T07:25:27.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.220+0000 7f5c267fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c2814f540 0x7f5c281506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.220+0000 7f5c267fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c2814f540 0x7f5c281506c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53140/0 (socket says 172.21.15.1:53140) 2024-03-06T07:25:27.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.220+0000 7f5c267fc640 1 -- 172.21.15.1:0/2022913845 learned_addr learned my addr 172.21.15.1:0/2022913845 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:27.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.220+0000 7f5c26ffd640 1 --2- 172.21.15.1:0/2022913845 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c140039b0 0x7f5c28154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.220+0000 7f5c277fe640 1 --2- 172.21.15.1:0/2022913845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c281e1520 0x7f5c281e19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.221+0000 7f5c267fc640 1 -- 172.21.15.1:0/2022913845 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c140039b0 msgr2=0x7f5c28154e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.221+0000 7f5c267fc640 1 --2- 172.21.15.1:0/2022913845 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c140039b0 0x7f5c28154e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.221+0000 7f5c267fc640 1 -- 172.21.15.1:0/2022913845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c281e1520 msgr2=0x7f5c281e19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.221+0000 7f5c267fc640 1 --2- 172.21.15.1:0/2022913845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c281e1520 0x7f5c281e19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.221+0000 7f5c267fc640 1 -- 172.21.15.1:0/2022913845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c281ef6b0 con 0x7f5c2814f540 2024-03-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.221+0000 7f5c26ffd640 1 --2- 172.21.15.1:0/2022913845 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c140039b0 0x7f5c28154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.221+0000 7f5c277fe640 1 --2- 172.21.15.1:0/2022913845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c281e1520 0x7f5c281e19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.221+0000 7f5c267fc640 1 --2- 172.21.15.1:0/2022913845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c2814f540 0x7f5c281506c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f5c10011a00 tx=0x7f5c10011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.221+0000 7f5c1f7fe640 1 -- 172.21.15.1:0/2022913845 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1000fa90 con 0x7f5c2814f540 2024-03-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.221+0000 7f5c1f7fe640 1 -- 172.21.15.1:0/2022913845 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1000fc30 con 0x7f5c2814f540 2024-03-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.221+0000 7f5c2d719640 1 -- 172.21.15.1:0/2022913845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c28154370 con 0x7f5c2814f540 2024-03-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.221+0000 7f5c1f7fe640 1 -- 172.21.15.1:0/2022913845 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c10014440 con 0x7f5c2814f540 2024-03-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.221+0000 7f5c2d719640 1 -- 172.21.15.1:0/2022913845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c28154610 con 0x7f5c2814f540 2024-03-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.222+0000 7f5c2d719640 1 -- 172.21.15.1:0/2022913845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c281ef160 con 0x7f5c2814f540 2024-03-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.222+0000 7f5c1f7fe640 1 -- 172.21.15.1:0/2022913845 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c10012040 con 0x7f5c2814f540 2024-03-06T07:25:27.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.223+0000 7f5c1f7fe640 1 --2- 172.21.15.1:0/2022913845 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0806b500 0x7f5c0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.223+0000 7f5c1f7fe640 1 -- 172.21.15.1:0/2022913845 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5c1008a530 con 0x7f5c2814f540 2024-03-06T07:25:27.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.223+0000 7f5c26ffd640 1 --2- 172.21.15.1:0/2022913845 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0806b500 0x7f5c0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.223+0000 7f5c26ffd640 1 --2- 172.21.15.1:0/2022913845 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0806b500 0x7f5c0806d9c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f5c20036000 tx=0x7f5c20030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.227+0000 7f5c1f7fe640 1 -- 172.21.15.1:0/2022913845 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c10019030 con 0x7f5c2814f540 2024-03-06T07:25:27.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.406+0000 7f5c2d719640 1 -- 172.21.15.1:0/2022913845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5c280c5750 con 0x7f5c2814f540 2024-03-06T07:25:27.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.407+0000 7f5c1f7fe640 1 -- 172.21.15.1:0/2022913845 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v131) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5c10059600 con 0x7f5c2814f540 2024-03-06T07:25:27.409 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:25:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.409+0000 7f5c2d719640 1 -- 172.21.15.1:0/2022913845 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0806b500 msgr2=0x7f5c0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.409+0000 7f5c2d719640 1 --2- 172.21.15.1:0/2022913845 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0806b500 0x7f5c0806d9c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f5c20036000 tx=0x7f5c20030000 comp rx=0 tx=0).stop 2024-03-06T07:25:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.409+0000 7f5c2d719640 1 -- 172.21.15.1:0/2022913845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c2814f540 msgr2=0x7f5c281506c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.409+0000 7f5c2d719640 1 --2- 172.21.15.1:0/2022913845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c2814f540 0x7f5c281506c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f5c10011a00 tx=0x7f5c10011ec0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.409+0000 7f5c2d719640 1 -- 172.21.15.1:0/2022913845 shutdown_connections 2024-03-06T07:25:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.409+0000 7f5c2d719640 1 --2- 172.21.15.1:0/2022913845 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c0806b500 0x7f5c0806d9c0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.409+0000 7f5c2d719640 1 --2- 172.21.15.1:0/2022913845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c281e1520 0x7f5c281e19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.409+0000 7f5c2d719640 1 --2- 172.21.15.1:0/2022913845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c2814f540 0x7f5c281506c0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.409+0000 7f5c2d719640 1 --2- 172.21.15.1:0/2022913845 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c140039b0 0x7f5c28154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.409+0000 7f5c2d719640 1 -- 172.21.15.1:0/2022913845 >> 172.21.15.1:0/2022913845 conn(0x7f5c2806b410 msgr2=0x7f5c28157390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.409+0000 7f5c2d719640 1 -- 172.21.15.1:0/2022913845 shutdown_connections 2024-03-06T07:25:27.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.409+0000 7f5c2d719640 1 -- 172.21.15.1:0/2022913845 wait complete. 2024-03-06T07:25:27.422 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:25:27.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.521+0000 7f8ce6b3e640 1 Processor -- start 2024-03-06T07:25:27.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.521+0000 7f8ce6b3e640 1 -- start start 2024-03-06T07:25:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.522+0000 7f8ce6b3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ce0152650 0x7f8ce0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.522+0000 7f8ce6b3e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8ce00c6b10 0x7f8ce0158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.522+0000 7f8ce6b3e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce00c2fc0 con 0x7f8ce00bf9d0 2024-03-06T07:25:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.522+0000 7f8ce6b3e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce0155800 con 0x7f8ce00c6b10 2024-03-06T07:25:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.522+0000 7f8ce6b3e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce00c6690 con 0x7f8ce0152650 2024-03-06T07:25:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.522+0000 7f8cdffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8ce00c6b10 0x7f8ce0158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.522+0000 7f8cdffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8ce00c6b10 0x7f8ce0158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53156/0 (socket says 172.21.15.1:53156) 2024-03-06T07:25:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.522+0000 7f8cdffff640 1 -- 172.21.15.1:0/3765851812 learned_addr learned my addr 172.21.15.1:0/3765851812 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.523+0000 7f8cdffff640 1 -- 172.21.15.1:0/3765851812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ce0152650 msgr2=0x7f8ce0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.523+0000 7f8cdffff640 1 --2- 172.21.15.1:0/3765851812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ce0152650 0x7f8ce0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.523+0000 7f8cdffff640 1 -- 172.21.15.1:0/3765851812 >> v1:172.21.15.98:6789/0 conn(0x7f8ce00bf9d0 legacy=0x7f8ce00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.523+0000 7f8cdffff640 1 -- 172.21.15.1:0/3765851812 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce00c7280 con 0x7f8ce00c6b10 2024-03-06T07:25:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.523+0000 7f8cdffff640 1 --2- 172.21.15.1:0/3765851812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce00c6b10 0x7f8ce0158690 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4009870 tx=0x7f8cd402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=907db1b99649abe0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.523+0000 7f8cdf7fe640 1 -- 172.21.15.1:0/3765851812 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd40317c0 con 0x7f8ce00c6b10 2024-03-06T07:25:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.523+0000 7f8cdf7fe640 1 -- 172.21.15.1:0/3765851812 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd4037070 con 0x7f8ce00c6b10 2024-03-06T07:25:27.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.523+0000 7f8cdf7fe640 1 -- 172.21.15.1:0/3765851812 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd4031b80 con 0x7f8ce00c6b10 2024-03-06T07:25:27.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.524+0000 7f8ce6b3e640 1 -- 172.21.15.1:0/3765851812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce00c6b10 msgr2=0x7f8ce0158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.524+0000 7f8ce6b3e640 1 --2- 172.21.15.1:0/3765851812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce00c6b10 0x7f8ce0158690 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4009870 tx=0x7f8cd402b860 comp rx=0 tx=0).stop 2024-03-06T07:25:27.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.524+0000 7f8ce6b3e640 1 -- 172.21.15.1:0/3765851812 shutdown_connections 2024-03-06T07:25:27.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.524+0000 7f8ce6b3e640 1 --2- 172.21.15.1:0/3765851812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce00c6b10 0x7f8ce0158690 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.524+0000 7f8ce6b3e640 1 --2- 172.21.15.1:0/3765851812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ce0152650 0x7f8ce0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.524+0000 7f8ce6b3e640 1 -- 172.21.15.1:0/3765851812 >> 172.21.15.1:0/3765851812 conn(0x7f8ce0066ee0 msgr2=0x7f8ce00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.524+0000 7f8ce6b3e640 1 -- 172.21.15.1:0/3765851812 shutdown_connections 2024-03-06T07:25:27.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.524+0000 7f8ce6b3e640 1 -- 172.21.15.1:0/3765851812 wait complete. 2024-03-06T07:25:27.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.525+0000 7f8ce6b3e640 1 Processor -- start 2024-03-06T07:25:27.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.525+0000 7f8ce6b3e640 1 -- start start 2024-03-06T07:25:27.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8ce6b3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ce00bf9d0 0x7f8ce00c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8ce6b3e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce0152650 0x7f8ce0148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8ce6b3e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ce00c9d80 0x7f8ce0149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8ce6b3e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce0155490 con 0x7f8ce0152650 2024-03-06T07:25:27.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8ce6b3e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce01551b0 con 0x7f8ce00c9d80 2024-03-06T07:25:27.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8ce6b3e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce0152a30 con 0x7f8ce00bf9d0 2024-03-06T07:25:27.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8cdffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce0152650 0x7f8ce0148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8cdffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce0152650 0x7f8ce0148e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53162/0 (socket says 172.21.15.1:53162) 2024-03-06T07:25:27.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8cdffff640 1 -- 172.21.15.1:0/3078098165 learned_addr learned my addr 172.21.15.1:0/3078098165 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:27.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8ce48da640 1 --2- 172.21.15.1:0/3078098165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ce00bf9d0 0x7f8ce00c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8ce50db640 1 --2- 172.21.15.1:0/3078098165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ce00c9d80 0x7f8ce0149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8cdffff640 1 -- 172.21.15.1:0/3078098165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ce00bf9d0 msgr2=0x7f8ce00c54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8cdffff640 1 --2- 172.21.15.1:0/3078098165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ce00bf9d0 0x7f8ce00c54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8cdffff640 1 -- 172.21.15.1:0/3078098165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ce00c9d80 msgr2=0x7f8ce0149340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8cdffff640 1 --2- 172.21.15.1:0/3078098165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ce00c9d80 0x7f8ce0149340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8cdffff640 1 -- 172.21.15.1:0/3078098165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce0149880 con 0x7f8ce0152650 2024-03-06T07:25:27.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8ce48da640 1 --2- 172.21.15.1:0/3078098165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ce00bf9d0 0x7f8ce00c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:27.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.526+0000 7f8ce50db640 1 --2- 172.21.15.1:0/3078098165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ce00c9d80 0x7f8ce0149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:27.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.527+0000 7f8cdffff640 1 --2- 172.21.15.1:0/3078098165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce0152650 0x7f8ce0148e00 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4009870 tx=0x7f8cd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.527+0000 7f8cdd7fa640 1 -- 172.21.15.1:0/3078098165 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd4032860 con 0x7f8ce0152650 2024-03-06T07:25:27.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.527+0000 7f8cdd7fa640 1 -- 172.21.15.1:0/3078098165 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd4032a00 con 0x7f8ce0152650 2024-03-06T07:25:27.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.527+0000 7f8ce6b3e640 1 -- 172.21.15.1:0/3078098165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce01ee460 con 0x7f8ce0152650 2024-03-06T07:25:27.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.527+0000 7f8cdd7fa640 1 -- 172.21.15.1:0/3078098165 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd4032df0 con 0x7f8ce0152650 2024-03-06T07:25:27.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.527+0000 7f8ce6b3e640 1 -- 172.21.15.1:0/3078098165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce01eea20 con 0x7f8ce0152650 2024-03-06T07:25:27.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.528+0000 7f8ce6b3e640 1 -- 172.21.15.1:0/3078098165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce0067fd0 con 0x7f8ce0152650 2024-03-06T07:25:27.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.528+0000 7f8cdd7fa640 1 -- 172.21.15.1:0/3078098165 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8cd404c060 con 0x7f8ce0152650 2024-03-06T07:25:27.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.529+0000 7f8cdd7fa640 1 --2- 172.21.15.1:0/3078098165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ccc06b500 0x7f8ccc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.529+0000 7f8cdd7fa640 1 -- 172.21.15.1:0/3078098165 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8cd40ab830 con 0x7f8ce0152650 2024-03-06T07:25:27.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.529+0000 7f8ce48da640 1 --2- 172.21.15.1:0/3078098165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ccc06b500 0x7f8ccc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.529+0000 7f8ce48da640 1 --2- 172.21.15.1:0/3078098165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ccc06b500 0x7f8ccc06d9c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8007eb0 tx=0x7f8cd80073c0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.533+0000 7f8cdd7fa640 1 -- 172.21.15.1:0/3078098165 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8cd4040070 con 0x7f8ce0152650 2024-03-06T07:25:27.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.715+0000 7f8ce6b3e640 1 -- 172.21.15.1:0/3078098165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8ce00b06f0 con 0x7f8ce0152650 2024-03-06T07:25:27.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.716+0000 7f8cdd7fa640 1 -- 172.21.15.1:0/3078098165 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v131) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8cd407a900 con 0x7f8ce0152650 2024-03-06T07:25:27.718 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:27.718 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-06T07:25:27.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:25:27.718 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-06T07:25:27.718 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-06T07:25:27.718 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:25:27.718 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-06T07:25:27.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.717+0000 7f8ce6b3e640 1 -- 172.21.15.1:0/3078098165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ccc06b500 msgr2=0x7f8ccc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.717+0000 7f8ce6b3e640 1 --2- 172.21.15.1:0/3078098165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ccc06b500 0x7f8ccc06d9c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8007eb0 tx=0x7f8cd80073c0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.717+0000 7f8ce6b3e640 1 -- 172.21.15.1:0/3078098165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce0152650 msgr2=0x7f8ce0148e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.717+0000 7f8ce6b3e640 1 --2- 172.21.15.1:0/3078098165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce0152650 0x7f8ce0148e00 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4009870 tx=0x7f8cd40046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.718+0000 7f8ce6b3e640 1 -- 172.21.15.1:0/3078098165 shutdown_connections 2024-03-06T07:25:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.718+0000 7f8ce6b3e640 1 --2- 172.21.15.1:0/3078098165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8ccc06b500 0x7f8ccc06d9c0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.718+0000 7f8ce6b3e640 1 --2- 172.21.15.1:0/3078098165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ce00c9d80 0x7f8ce0149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.718+0000 7f8ce6b3e640 1 --2- 172.21.15.1:0/3078098165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce0152650 0x7f8ce0148e00 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.718+0000 7f8ce6b3e640 1 --2- 172.21.15.1:0/3078098165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ce00bf9d0 0x7f8ce00c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.718+0000 7f8ce6b3e640 1 -- 172.21.15.1:0/3078098165 >> 172.21.15.1:0/3078098165 conn(0x7f8ce0066ee0 msgr2=0x7f8ce00672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.718+0000 7f8ce6b3e640 1 -- 172.21.15.1:0/3078098165 shutdown_connections 2024-03-06T07:25:27.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.718+0000 7f8ce6b3e640 1 -- 172.21.15.1:0/3078098165 wait complete. 2024-03-06T07:25:27.730 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:25:27.731 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:25:27.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.834+0000 7f46be4cf640 1 Processor -- start 2024-03-06T07:25:27.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.834+0000 7f46be4cf640 1 -- start start 2024-03-06T07:25:27.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.834+0000 7f46be4cf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46b80c1060 0x7f46b8155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.834+0000 7f46be4cf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f46b80c58c0 0x7f46b80a6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.834+0000 7f46be4cf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b8156170 con 0x7f46b80a7270 2024-03-06T07:25:27.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.834+0000 7f46be4cf640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46b80c6f00 con 0x7f46b80c58c0 2024-03-06T07:25:27.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.834+0000 7f46be4cf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b80c71a0 con 0x7f46b80c1060 2024-03-06T07:25:27.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.834+0000 7f46bca6c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f46b80a7270 0x7f46b814ee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32964/0 (socket says 172.21.15.1:32964) 2024-03-06T07:25:27.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.834+0000 7f46bca6c640 1 -- 172.21.15.1:0/2765540941 learned_addr learned my addr 172.21.15.1:0/2765540941 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:27.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b6ffd640 1 -- 172.21.15.1:0/2765540941 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742128144 0 0) 0x7f46b8156170 con 0x7f46b80a7270 2024-03-06T07:25:27.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b6ffd640 1 -- 172.21.15.1:0/2765540941 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a0003520 con 0x7f46b80a7270 2024-03-06T07:25:27.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b7fff640 1 --2- 172.21.15.1:0/2765540941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46b80c1060 0x7f46b8155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b77fe640 1 --2- 172.21.15.1:0/2765540941 >> v2:172.21.15.98:3300/0 conn(0x7f46b80c58c0 0x7f46b80a6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b6ffd640 1 -- 172.21.15.1:0/2765540941 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3072237674 0 0) 0x7f46a0003520 con 0x7f46b80a7270 2024-03-06T07:25:27.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b6ffd640 1 -- 172.21.15.1:0/2765540941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46b80c1060 msgr2=0x7f46b8155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b6ffd640 1 --2- 172.21.15.1:0/2765540941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46b80c1060 0x7f46b8155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b6ffd640 1 -- 172.21.15.1:0/2765540941 >> v2:172.21.15.98:3300/0 conn(0x7f46b80c58c0 msgr2=0x7f46b80a6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b6ffd640 1 --2- 172.21.15.1:0/2765540941 >> v2:172.21.15.98:3300/0 conn(0x7f46b80c58c0 0x7f46b80a6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b6ffd640 1 -- 172.21.15.1:0/2765540941 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b80c7480 con 0x7f46b80a7270 2024-03-06T07:25:27.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b6ffd640 1 -- 172.21.15.1:0/2765540941 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f46b00019f0 con 0x7f46b80a7270 2024-03-06T07:25:27.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b6ffd640 1 -- 172.21.15.1:0/2765540941 >> v1:172.21.15.1:6789/0 conn(0x7f46b80a7270 legacy=0x7f46b814ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b6ffd640 1 --2- 172.21.15.1:0/2765540941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a00039b0 0x7f46a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b6ffd640 1 --2- 172.21.15.1:0/2765540941 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46a0004800 0x7f46a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b6ffd640 1 --2- 172.21.15.1:0/2765540941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46b80c1060 0x7f46a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b6ffd640 1 -- 172.21.15.1:0/2765540941 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b80c6f00 con 0x7f46a0004800 2024-03-06T07:25:27.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b6ffd640 1 -- 172.21.15.1:0/2765540941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b80c71a0 con 0x7f46a00039b0 2024-03-06T07:25:27.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b6ffd640 1 -- 172.21.15.1:0/2765540941 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a000b9c0 con 0x7f46b80c1060 2024-03-06T07:25:27.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b77fe640 1 --2- 172.21.15.1:0/2765540941 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46a0004800 0x7f46a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b7fff640 1 --2- 172.21.15.1:0/2765540941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a00039b0 0x7f46a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46bca6c640 1 --2- 172.21.15.1:0/2765540941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46b80c1060 0x7f46a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b77fe640 1 -- 172.21.15.1:0/2765540941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46b80c1060 msgr2=0x7f46a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b77fe640 1 --2- 172.21.15.1:0/2765540941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46b80c1060 0x7f46a00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b77fe640 1 -- 172.21.15.1:0/2765540941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a00039b0 msgr2=0x7f46a0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b77fe640 1 --2- 172.21.15.1:0/2765540941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a00039b0 0x7f46a0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.835+0000 7f46b77fe640 1 -- 172.21.15.1:0/2765540941 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a0003520 con 0x7f46a0004800 2024-03-06T07:25:27.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.836+0000 7f46b77fe640 1 --2- 172.21.15.1:0/2765540941 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46a0004800 0x7f46a0004c80 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f46ac009af0 tx=0x7f46a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe59f3f38dde93a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.836+0000 7f46b6ffd640 1 -- 172.21.15.1:0/2765540941 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac039070 con 0x7f46a0004800 2024-03-06T07:25:27.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.836+0000 7f46b6ffd640 1 -- 172.21.15.1:0/2765540941 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ac02c710 con 0x7f46a0004800 2024-03-06T07:25:27.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.836+0000 7f46b6ffd640 1 -- 172.21.15.1:0/2765540941 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac009690 con 0x7f46a0004800 2024-03-06T07:25:27.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.837+0000 7f46be4cf640 1 -- 172.21.15.1:0/2765540941 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46a0004800 msgr2=0x7f46a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.837+0000 7f46be4cf640 1 --2- 172.21.15.1:0/2765540941 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46a0004800 0x7f46a0004c80 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f46ac009af0 tx=0x7f46a0005830 comp rx=0 tx=0).stop 2024-03-06T07:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.837+0000 7f46bca6c640 1 -- 172.21.15.1:0/2765540941 reap_dead start 2024-03-06T07:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.837+0000 7f46be4cf640 1 -- 172.21.15.1:0/2765540941 shutdown_connections 2024-03-06T07:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.837+0000 7f46be4cf640 1 -- 172.21.15.1:0/2765540941 >> 172.21.15.1:0/2765540941 conn(0x7f46b80c44c0 msgr2=0x7f46b80c48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.837+0000 7f46be4cf640 1 -- 172.21.15.1:0/2765540941 shutdown_connections 2024-03-06T07:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.837+0000 7f46be4cf640 1 -- 172.21.15.1:0/2765540941 wait complete. 2024-03-06T07:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.838+0000 7f46be4cf640 1 Processor -- start 2024-03-06T07:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.838+0000 7f46be4cf640 1 -- start start 2024-03-06T07:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46be4cf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46a0004800 0x7f46b8159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46be4cf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46b80ace90 0x7f46b8149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46be4cf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46b8152990 0x7f46b814a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46be4cf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b80c5d40 con 0x7f46b8152990 2024-03-06T07:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46be4cf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b8155710 con 0x7f46b80ace90 2024-03-06T07:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46be4cf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b80c5ec0 con 0x7f46a0004800 2024-03-06T07:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46bca6c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46b8152990 0x7f46b814a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46bca6c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46b8152990 0x7f46b814a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53182/0 (socket says 172.21.15.1:53182) 2024-03-06T07:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46bca6c640 1 -- 172.21.15.1:0/551151117 learned_addr learned my addr 172.21.15.1:0/551151117 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46b7fff640 1 --2- 172.21.15.1:0/551151117 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46a0004800 0x7f46b8159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46b77fe640 1 --2- 172.21.15.1:0/551151117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46b80ace90 0x7f46b8149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46bca6c640 1 -- 172.21.15.1:0/551151117 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46a0004800 msgr2=0x7f46b8159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46bca6c640 1 --2- 172.21.15.1:0/551151117 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46a0004800 0x7f46b8159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46bca6c640 1 -- 172.21.15.1:0/551151117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46b80ace90 msgr2=0x7f46b8149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46bca6c640 1 --2- 172.21.15.1:0/551151117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46b80ace90 0x7f46b8149c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46bca6c640 1 -- 172.21.15.1:0/551151117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b8152440 con 0x7f46b8152990 2024-03-06T07:25:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46b77fe640 1 --2- 172.21.15.1:0/551151117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46b80ace90 0x7f46b8149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.839+0000 7f46bca6c640 1 --2- 172.21.15.1:0/551151117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46b8152990 0x7f46b814a160 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f46b80c6a10 tx=0x7f46b000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.840+0000 7f46b4ff9640 1 -- 172.21.15.1:0/551151117 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b000ec10 con 0x7f46b8152990 2024-03-06T07:25:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.840+0000 7f46b4ff9640 1 -- 172.21.15.1:0/551151117 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b000edd0 con 0x7f46b8152990 2024-03-06T07:25:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.840+0000 7f46b4ff9640 1 -- 172.21.15.1:0/551151117 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0014670 con 0x7f46b8152990 2024-03-06T07:25:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.840+0000 7f46be4cf640 1 -- 172.21.15.1:0/551151117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b8152700 con 0x7f46b8152990 2024-03-06T07:25:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.840+0000 7f46be4cf640 1 -- 172.21.15.1:0/551151117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b81eb0b0 con 0x7f46b8152990 2024-03-06T07:25:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.841+0000 7f46be4cf640 1 -- 172.21.15.1:0/551151117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b80680b0 con 0x7f46b8152990 2024-03-06T07:25:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.841+0000 7f46b4ff9640 1 -- 172.21.15.1:0/551151117 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46b0014810 con 0x7f46b8152990 2024-03-06T07:25:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.841+0000 7f46b4ff9640 1 --2- 172.21.15.1:0/551151117 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f46a406b500 0x7f46a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.841+0000 7f46b4ff9640 1 -- 172.21.15.1:0/551151117 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f46b0089d60 con 0x7f46b8152990 2024-03-06T07:25:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.842+0000 7f46b7fff640 1 --2- 172.21.15.1:0/551151117 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f46a406b500 0x7f46a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.842+0000 7f46b7fff640 1 --2- 172.21.15.1:0/551151117 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f46a406b500 0x7f46a406d9c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f46a8009ae0 tx=0x7f46a8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:27.846+0000 7f46b4ff9640 1 -- 172.21.15.1:0/551151117 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46b0058e30 con 0x7f46b8152990 2024-03-06T07:25:28.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.026+0000 7f46be4cf640 1 -- 172.21.15.1:0/551151117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f46b80c6ce0 con 0x7f46b8152990 2024-03-06T07:25:28.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.414+0000 7f46b4ff9640 1 -- 172.21.15.1:0/551151117 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v132) v1 ==== 138+0+0 (secure 0 0 0) 0x7f46b0058730 con 0x7f46b8152990 2024-03-06T07:25:28.416 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-06T07:25:28.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.416+0000 7f46be4cf640 1 -- 172.21.15.1:0/551151117 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f46a406b500 msgr2=0x7f46a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.416+0000 7f46be4cf640 1 --2- 172.21.15.1:0/551151117 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f46a406b500 0x7f46a406d9c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f46a8009ae0 tx=0x7f46a8009190 comp rx=0 tx=0).stop 2024-03-06T07:25:28.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.416+0000 7f46be4cf640 1 -- 172.21.15.1:0/551151117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46b8152990 msgr2=0x7f46b814a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.416+0000 7f46be4cf640 1 --2- 172.21.15.1:0/551151117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46b8152990 0x7f46b814a160 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f46b80c6a10 tx=0x7f46b000e530 comp rx=0 tx=0).stop 2024-03-06T07:25:28.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.416+0000 7f46be4cf640 1 -- 172.21.15.1:0/551151117 shutdown_connections 2024-03-06T07:25:28.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.416+0000 7f46be4cf640 1 --2- 172.21.15.1:0/551151117 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f46a406b500 0x7f46a406d9c0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.416+0000 7f46be4cf640 1 --2- 172.21.15.1:0/551151117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46b8152990 0x7f46b814a160 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.416+0000 7f46be4cf640 1 --2- 172.21.15.1:0/551151117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46b80ace90 0x7f46b8149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.416+0000 7f46be4cf640 1 --2- 172.21.15.1:0/551151117 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46a0004800 0x7f46b8159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.416+0000 7f46be4cf640 1 -- 172.21.15.1:0/551151117 >> 172.21.15.1:0/551151117 conn(0x7f46b80c44c0 msgr2=0x7f46b80c6280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.416+0000 7f46be4cf640 1 -- 172.21.15.1:0/551151117 shutdown_connections 2024-03-06T07:25:28.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.416+0000 7f46be4cf640 1 -- 172.21.15.1:0/551151117 wait complete. 2024-03-06T07:25:28.433 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:25:28.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.541+0000 7f32a7925640 1 Processor -- start 2024-03-06T07:25:28.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.541+0000 7f32a7925640 1 -- start start 2024-03-06T07:25:28.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.542+0000 7f32a7925640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32a00c1390 0x7f32a00c17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.542+0000 7f32a7925640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32a01552e0 0x7f32a0155750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.542+0000 7f32a7925640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a01588e0 con 0x7f32a0152730 2024-03-06T07:25:28.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.542+0000 7f32a7925640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a00c1d30 con 0x7f32a01552e0 2024-03-06T07:25:28.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.542+0000 7f32a7925640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a0158bc0 con 0x7f32a00c1390 2024-03-06T07:25:28.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.542+0000 7f32a5ec2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32a01552e0 0x7f32a0155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.542+0000 7f32a5ec2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32a01552e0 0x7f32a0155750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53190/0 (socket says 172.21.15.1:53190) 2024-03-06T07:25:28.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.542+0000 7f32a5ec2640 1 -- 172.21.15.1:0/2449660752 learned_addr learned my addr 172.21.15.1:0/2449660752 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:28.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.542+0000 7f32a4ec0640 1 --2- 172.21.15.1:0/2449660752 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32a00c1390 0x7f32a00c17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.542+0000 7f32a5ec2640 1 -- 172.21.15.1:0/2449660752 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32a00c1390 msgr2=0x7f32a00c17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:28.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.542+0000 7f32a5ec2640 1 --2- 172.21.15.1:0/2449660752 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32a00c1390 0x7f32a00c17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.542+0000 7f32a5ec2640 1 -- 172.21.15.1:0/2449660752 >> v1:172.21.15.98:6789/0 conn(0x7f32a0152730 legacy=0x7f32a00c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:28.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.542+0000 7f32a5ec2640 1 -- 172.21.15.1:0/2449660752 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a0158ea0 con 0x7f32a01552e0 2024-03-06T07:25:28.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.542+0000 7f32a4ec0640 1 --2- 172.21.15.1:0/2449660752 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32a00c1390 0x7f32a00c17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:28.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.543+0000 7f32a5ec2640 1 --2- 172.21.15.1:0/2449660752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a01552e0 0x7f32a0155750 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f32a006c070 tx=0x7f329802bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38c89e69134399d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.543+0000 7f3297fff640 1 -- 172.21.15.1:0/2449660752 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329802c910 con 0x7f32a01552e0 2024-03-06T07:25:28.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.543+0000 7f3297fff640 1 -- 172.21.15.1:0/2449660752 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329802cab0 con 0x7f32a01552e0 2024-03-06T07:25:28.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.543+0000 7f3297fff640 1 -- 172.21.15.1:0/2449660752 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32980343f0 con 0x7f32a01552e0 2024-03-06T07:25:28.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.543+0000 7f32a7925640 1 -- 172.21.15.1:0/2449660752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a01552e0 msgr2=0x7f32a0155750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.543+0000 7f32a7925640 1 --2- 172.21.15.1:0/2449660752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a01552e0 0x7f32a0155750 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f32a006c070 tx=0x7f329802bd70 comp rx=0 tx=0).stop 2024-03-06T07:25:28.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.543+0000 7f32a7925640 1 -- 172.21.15.1:0/2449660752 shutdown_connections 2024-03-06T07:25:28.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.543+0000 7f32a7925640 1 --2- 172.21.15.1:0/2449660752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a01552e0 0x7f32a0155750 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.543+0000 7f32a7925640 1 --2- 172.21.15.1:0/2449660752 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32a00c1390 0x7f32a00c17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.543+0000 7f32a7925640 1 -- 172.21.15.1:0/2449660752 >> 172.21.15.1:0/2449660752 conn(0x7f32a0066d00 msgr2=0x7f32a0067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.544+0000 7f32a7925640 1 -- 172.21.15.1:0/2449660752 shutdown_connections 2024-03-06T07:25:28.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.544+0000 7f32a7925640 1 -- 172.21.15.1:0/2449660752 wait complete. 2024-03-06T07:25:28.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.545+0000 7f32a7925640 1 Processor -- start 2024-03-06T07:25:28.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.545+0000 7f32a7925640 1 -- start start 2024-03-06T07:25:28.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.545+0000 7f32a7925640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a00c1390 0x7f32a01eeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.545+0000 7f32a7925640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32a0152730 0x7f32a00c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.545+0000 7f32a7925640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a01552e0 0x7f32a014acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.545+0000 7f32a7925640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a0152b10 con 0x7f32a01552e0 2024-03-06T07:25:28.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.545+0000 7f32a7925640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a0159460 con 0x7f32a00c1390 2024-03-06T07:25:28.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.545+0000 7f32a7925640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a00c71a0 con 0x7f32a0152730 2024-03-06T07:25:28.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.545+0000 7f32a5ec2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a01552e0 0x7f32a014acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.545+0000 7f32a5ec2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a01552e0 0x7f32a014acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53196/0 (socket says 172.21.15.1:53196) 2024-03-06T07:25:28.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.545+0000 7f32a5ec2640 1 -- 172.21.15.1:0/2732193508 learned_addr learned my addr 172.21.15.1:0/2732193508 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:28.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.546+0000 7f32a4ec0640 1 --2- 172.21.15.1:0/2732193508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32a0152730 0x7f32a00c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.546+0000 7f32a56c1640 1 --2- 172.21.15.1:0/2732193508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a00c1390 0x7f32a01eeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.546+0000 7f32a5ec2640 1 -- 172.21.15.1:0/2732193508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32a0152730 msgr2=0x7f32a00c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.546+0000 7f32a5ec2640 1 --2- 172.21.15.1:0/2732193508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32a0152730 0x7f32a00c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.546+0000 7f32a5ec2640 1 -- 172.21.15.1:0/2732193508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a00c1390 msgr2=0x7f32a01eeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.546+0000 7f32a5ec2640 1 --2- 172.21.15.1:0/2732193508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a00c1390 0x7f32a01eeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.546+0000 7f32a5ec2640 1 -- 172.21.15.1:0/2732193508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a0153170 con 0x7f32a01552e0 2024-03-06T07:25:28.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.546+0000 7f32a4ec0640 1 --2- 172.21.15.1:0/2732193508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32a0152730 0x7f32a00c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:28.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.546+0000 7f32a56c1640 1 --2- 172.21.15.1:0/2732193508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a00c1390 0x7f32a01eeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:28.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.546+0000 7f32a5ec2640 1 --2- 172.21.15.1:0/2732193508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a01552e0 0x7f32a014acb0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f3298009f40 tx=0x7f3298004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.546+0000 7f3295ffb640 1 -- 172.21.15.1:0/2732193508 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3298043020 con 0x7f32a01552e0 2024-03-06T07:25:28.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.546+0000 7f3295ffb640 1 -- 172.21.15.1:0/2732193508 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329803e450 con 0x7f32a01552e0 2024-03-06T07:25:28.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.546+0000 7f3295ffb640 1 -- 172.21.15.1:0/2732193508 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3298009510 con 0x7f32a01552e0 2024-03-06T07:25:28.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.546+0000 7f32a7925640 1 -- 172.21.15.1:0/2732193508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a0157e00 con 0x7f32a01552e0 2024-03-06T07:25:28.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.546+0000 7f32a7925640 1 -- 172.21.15.1:0/2732193508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a01583c0 con 0x7f32a01552e0 2024-03-06T07:25:28.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.547+0000 7f32a7925640 1 -- 172.21.15.1:0/2732193508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a0067fd0 con 0x7f32a01552e0 2024-03-06T07:25:28.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.548+0000 7f3295ffb640 1 -- 172.21.15.1:0/2732193508 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3298046020 con 0x7f32a01552e0 2024-03-06T07:25:28.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.548+0000 7f3295ffb640 1 --2- 172.21.15.1:0/2732193508 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f328406b500 0x7f328406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.548+0000 7f3295ffb640 1 -- 172.21.15.1:0/2732193508 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f32980acc60 con 0x7f32a01552e0 2024-03-06T07:25:28.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.549+0000 7f32a56c1640 1 --2- 172.21.15.1:0/2732193508 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f328406b500 0x7f328406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.549+0000 7f32a56c1640 1 --2- 172.21.15.1:0/2732193508 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f328406b500 0x7f328406d9c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f329c006fd0 tx=0x7f329c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.553+0000 7f3295ffb640 1 -- 172.21.15.1:0/2732193508 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f329804d020 con 0x7f32a01552e0 2024-03-06T07:25:28.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:28.733+0000 7f32a7925640 1 -- 172.21.15.1:0/2732193508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f32a00c6870 con 0x7f32a01552e0 2024-03-06T07:25:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.418+0000 7f3295ffb640 1 -- 172.21.15.1:0/2732193508 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v133) v1 ==== 170+0+0 (secure 0 0 0) 0x7f329807bce0 con 0x7f32a01552e0 2024-03-06T07:25:29.421 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-06T07:25:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.424+0000 7f32a7925640 1 -- 172.21.15.1:0/2732193508 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f328406b500 msgr2=0x7f328406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.424+0000 7f32a7925640 1 --2- 172.21.15.1:0/2732193508 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f328406b500 0x7f328406d9c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f329c006fd0 tx=0x7f329c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.424+0000 7f32a7925640 1 -- 172.21.15.1:0/2732193508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a01552e0 msgr2=0x7f32a014acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.424+0000 7f32a7925640 1 --2- 172.21.15.1:0/2732193508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a01552e0 0x7f32a014acb0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f3298009f40 tx=0x7f3298004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.424+0000 7f32a7925640 1 -- 172.21.15.1:0/2732193508 shutdown_connections 2024-03-06T07:25:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.424+0000 7f32a7925640 1 --2- 172.21.15.1:0/2732193508 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f328406b500 0x7f328406d9c0 secure :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f329c006fd0 tx=0x7f329c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.424+0000 7f32a7925640 1 --2- 172.21.15.1:0/2732193508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a01552e0 0x7f32a014acb0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.424+0000 7f32a7925640 1 --2- 172.21.15.1:0/2732193508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32a0152730 0x7f32a00c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.425+0000 7f32a7925640 1 --2- 172.21.15.1:0/2732193508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a00c1390 0x7f32a01eeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.425+0000 7f32a7925640 1 -- 172.21.15.1:0/2732193508 >> 172.21.15.1:0/2732193508 conn(0x7f32a0066d00 msgr2=0x7f32a00b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.426+0000 7f32a7925640 1 -- 172.21.15.1:0/2732193508 shutdown_connections 2024-03-06T07:25:29.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.428+0000 7f32a7925640 1 -- 172.21.15.1:0/2732193508 wait complete. 2024-03-06T07:25:29.451 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:25:29.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.558+0000 7fa6d48df640 1 Processor -- start 2024-03-06T07:25:29.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.558+0000 7fa6d48df640 1 -- start start 2024-03-06T07:25:29.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.559+0000 7fa6d48df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6d0152650 0x7fa6d0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.559+0000 7fa6d48df640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa6d00c6b10 0x7fa6d0158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.559+0000 7fa6d48df640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa6d00bf9d0 0x7fa6d00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.559+0000 7fa6d48df640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d0155800 con 0x7fa6d00bf9d0 2024-03-06T07:25:29.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.559+0000 7fa6d48df640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d00c2fc0 con 0x7fa6d00c6b10 2024-03-06T07:25:29.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.559+0000 7fa6d48df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d00c6690 con 0x7fa6d0152650 2024-03-06T07:25:29.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.559+0000 7fa6ced9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa6d00bf9d0 0x7fa6d00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.559+0000 7fa6ced9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa6d00bf9d0 0x7fa6d00c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53206/0 (socket says 172.21.15.1:53206) 2024-03-06T07:25:29.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.559+0000 7fa6ced9d640 1 -- 172.21.15.1:0/213666703 learned_addr learned my addr 172.21.15.1:0/213666703 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:29.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.560+0000 7fa6cdd9b640 1 --2- 172.21.15.1:0/213666703 >> v2:172.21.15.98:3300/0 conn(0x7fa6d00c6b10 0x7fa6d0158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.560+0000 7fa6ce59c640 1 --2- 172.21.15.1:0/213666703 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6d0152650 0x7fa6d0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.560+0000 7fa6ced9d640 1 -- 172.21.15.1:0/213666703 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6d0152650 msgr2=0x7fa6d0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.560+0000 7fa6ced9d640 1 --2- 172.21.15.1:0/213666703 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6d0152650 0x7fa6d0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.560+0000 7fa6ced9d640 1 -- 172.21.15.1:0/213666703 >> v2:172.21.15.98:3300/0 conn(0x7fa6d00c6b10 msgr2=0x7fa6d0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.560+0000 7fa6ced9d640 1 --2- 172.21.15.1:0/213666703 >> v2:172.21.15.98:3300/0 conn(0x7fa6d00c6b10 0x7fa6d0158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.560+0000 7fa6ced9d640 1 -- 172.21.15.1:0/213666703 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d00c7280 con 0x7fa6d00bf9d0 2024-03-06T07:25:29.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.560+0000 7fa6cdd9b640 1 --2- 172.21.15.1:0/213666703 >> v2:172.21.15.98:3300/0 conn(0x7fa6d00c6b10 0x7fa6d0158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:29.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.560+0000 7fa6ce59c640 1 --2- 172.21.15.1:0/213666703 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6d0152650 0x7fa6d0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:29.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.560+0000 7fa6ced9d640 1 --2- 172.21.15.1:0/213666703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6d00bf9d0 0x7fa6d00c1460 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fa6d008c500 tx=0x7fa6c002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8d4a1a23c40d9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.560+0000 7fa6cd59a640 1 -- 172.21.15.1:0/213666703 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c002c9e0 con 0x7fa6d00bf9d0 2024-03-06T07:25:29.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.561+0000 7fa6cd59a640 1 -- 172.21.15.1:0/213666703 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c002cb80 con 0x7fa6d00bf9d0 2024-03-06T07:25:29.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.561+0000 7fa6cd59a640 1 -- 172.21.15.1:0/213666703 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c00343f0 con 0x7fa6d00bf9d0 2024-03-06T07:25:29.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.561+0000 7fa6d48df640 1 -- 172.21.15.1:0/213666703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6d00bf9d0 msgr2=0x7fa6d00c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.561+0000 7fa6d48df640 1 --2- 172.21.15.1:0/213666703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6d00bf9d0 0x7fa6d00c1460 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fa6d008c500 tx=0x7fa6c002be40 comp rx=0 tx=0).stop 2024-03-06T07:25:29.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.561+0000 7fa6d48df640 1 -- 172.21.15.1:0/213666703 shutdown_connections 2024-03-06T07:25:29.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.561+0000 7fa6d48df640 1 --2- 172.21.15.1:0/213666703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6d00bf9d0 0x7fa6d00c1460 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.561+0000 7fa6d48df640 1 --2- 172.21.15.1:0/213666703 >> v2:172.21.15.98:3300/0 conn(0x7fa6d00c6b10 0x7fa6d0158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.561+0000 7fa6d48df640 1 --2- 172.21.15.1:0/213666703 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6d0152650 0x7fa6d0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.561+0000 7fa6d48df640 1 -- 172.21.15.1:0/213666703 >> 172.21.15.1:0/213666703 conn(0x7fa6d0066ee0 msgr2=0x7fa6d00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.561+0000 7fa6d48df640 1 -- 172.21.15.1:0/213666703 shutdown_connections 2024-03-06T07:25:29.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.561+0000 7fa6d48df640 1 -- 172.21.15.1:0/213666703 wait complete. 2024-03-06T07:25:29.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.562+0000 7fa6d48df640 1 Processor -- start 2024-03-06T07:25:29.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.563+0000 7fa6d48df640 1 -- start start 2024-03-06T07:25:29.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.563+0000 7fa6d48df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6d00bf9d0 0x7fa6d0153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.563+0000 7fa6d48df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6d0152650 0x7fa6d00c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.563+0000 7fa6d48df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d014dd30 0x7fa6d014c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.563+0000 7fa6d48df640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d0155490 con 0x7fa6d00bf9d0 2024-03-06T07:25:29.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.563+0000 7fa6d48df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d01551b0 con 0x7fa6d014dd30 2024-03-06T07:25:29.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.563+0000 7fa6d48df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d0152a30 con 0x7fa6d0152650 2024-03-06T07:25:29.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.563+0000 7fa6ce59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6d00bf9d0 0x7fa6d0153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.563+0000 7fa6ce59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6d00bf9d0 0x7fa6d0153c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53208/0 (socket says 172.21.15.1:53208) 2024-03-06T07:25:29.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.563+0000 7fa6ce59c640 1 -- 172.21.15.1:0/921342520 learned_addr learned my addr 172.21.15.1:0/921342520 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:29.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.563+0000 7fa6cdd9b640 1 --2- 172.21.15.1:0/921342520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6d0152650 0x7fa6d00c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.563+0000 7fa6ced9d640 1 --2- 172.21.15.1:0/921342520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d014dd30 0x7fa6d014c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.564+0000 7fa6ce59c640 1 -- 172.21.15.1:0/921342520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6d0152650 msgr2=0x7fa6d00c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.564+0000 7fa6ce59c640 1 --2- 172.21.15.1:0/921342520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6d0152650 0x7fa6d00c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.564+0000 7fa6ce59c640 1 -- 172.21.15.1:0/921342520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d014dd30 msgr2=0x7fa6d014c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.564+0000 7fa6ce59c640 1 --2- 172.21.15.1:0/921342520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d014dd30 0x7fa6d014c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.564+0000 7fa6ce59c640 1 -- 172.21.15.1:0/921342520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d014d350 con 0x7fa6d00bf9d0 2024-03-06T07:25:29.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.564+0000 7fa6ce59c640 1 --2- 172.21.15.1:0/921342520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6d00bf9d0 0x7fa6d0153c70 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800c820 tx=0x7fa6c800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.564+0000 7fa6c6ffd640 1 -- 172.21.15.1:0/921342520 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8015070 con 0x7fa6d00bf9d0 2024-03-06T07:25:29.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.564+0000 7fa6c6ffd640 1 -- 172.21.15.1:0/921342520 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c8007a30 con 0x7fa6d00bf9d0 2024-03-06T07:25:29.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.564+0000 7fa6d48df640 1 -- 172.21.15.1:0/921342520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d01541b0 con 0x7fa6d00bf9d0 2024-03-06T07:25:29.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.564+0000 7fa6c6ffd640 1 -- 172.21.15.1:0/921342520 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8007e00 con 0x7fa6d00bf9d0 2024-03-06T07:25:29.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.564+0000 7fa6d48df640 1 -- 172.21.15.1:0/921342520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d014d070 con 0x7fa6d00bf9d0 2024-03-06T07:25:29.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.565+0000 7fa6d48df640 1 -- 172.21.15.1:0/921342520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d014d920 con 0x7fa6d00bf9d0 2024-03-06T07:25:29.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.565+0000 7fa6c6ffd640 1 -- 172.21.15.1:0/921342520 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6c8018070 con 0x7fa6d00bf9d0 2024-03-06T07:25:29.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.566+0000 7fa6c6ffd640 1 --2- 172.21.15.1:0/921342520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa6b006b4d0 0x7fa6b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.566+0000 7fa6c6ffd640 1 -- 172.21.15.1:0/921342520 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fa6c808a1e0 con 0x7fa6d00bf9d0 2024-03-06T07:25:29.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.566+0000 7fa6cdd9b640 1 --2- 172.21.15.1:0/921342520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa6b006b4d0 0x7fa6b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.566+0000 7fa6cdd9b640 1 --2- 172.21.15.1:0/921342520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa6b006b4d0 0x7fa6b006d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0159750 tx=0x7fa6b8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.570+0000 7fa6c6ffd640 1 -- 172.21.15.1:0/921342520 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6c8058fc0 con 0x7fa6d00bf9d0 2024-03-06T07:25:29.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:29.791+0000 7fa6d48df640 1 -- 172.21.15.1:0/921342520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa6d0067fd0 con 0x7fa6d00bf9d0 2024-03-06T07:25:30.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.420+0000 7fa6c6ffd640 1 -- 172.21.15.1:0/921342520 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 21 and data pool 22 v119) v1 ==== 174+0+0 (secure 0 0 0) 0x7fa6c80588c0 con 0x7fa6d00bf9d0 2024-03-06T07:25:30.422 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 21 and data pool 22 2024-03-06T07:25:30.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.424+0000 7fa6d48df640 1 -- 172.21.15.1:0/921342520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa6b006b4d0 msgr2=0x7fa6b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.424+0000 7fa6d48df640 1 --2- 172.21.15.1:0/921342520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa6b006b4d0 0x7fa6b006d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0159750 tx=0x7fa6b8008040 comp rx=0 tx=0).stop 2024-03-06T07:25:30.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.424+0000 7fa6d48df640 1 -- 172.21.15.1:0/921342520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6d00bf9d0 msgr2=0x7fa6d0153c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.424+0000 7fa6d48df640 1 --2- 172.21.15.1:0/921342520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6d00bf9d0 0x7fa6d0153c70 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800c820 tx=0x7fa6c800cc20 comp rx=0 tx=0).stop 2024-03-06T07:25:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.425+0000 7fa6d48df640 1 -- 172.21.15.1:0/921342520 shutdown_connections 2024-03-06T07:25:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.425+0000 7fa6d48df640 1 --2- 172.21.15.1:0/921342520 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa6b006b4d0 0x7fa6b006d990 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.425+0000 7fa6d48df640 1 --2- 172.21.15.1:0/921342520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d014dd30 0x7fa6d014c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.425+0000 7fa6d48df640 1 --2- 172.21.15.1:0/921342520 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6d0152650 0x7fa6d00c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.425+0000 7fa6d48df640 1 --2- 172.21.15.1:0/921342520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6d00bf9d0 0x7fa6d0153c70 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.425+0000 7fa6d48df640 1 -- 172.21.15.1:0/921342520 >> 172.21.15.1:0/921342520 conn(0x7fa6d0066ee0 msgr2=0x7fa6d00c07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.426+0000 7fa6d48df640 1 -- 172.21.15.1:0/921342520 shutdown_connections 2024-03-06T07:25:30.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.426+0000 7fa6d48df640 1 -- 172.21.15.1:0/921342520 wait complete. 2024-03-06T07:25:30.455 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:30.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.558+0000 7fce0450c640 1 Processor -- start 2024-03-06T07:25:30.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.558+0000 7fce0450c640 1 -- start start 2024-03-06T07:25:30.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.559+0000 7fce0450c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcdfc1513c0 0x7fcdfc0726a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.559+0000 7fce0450c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdfc151830 con 0x7fcdfc150fe0 2024-03-06T07:25:30.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.559+0000 7fce0450c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdfc0a6140 con 0x7fcdfc072f20 2024-03-06T07:25:30.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.559+0000 7fce0450c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdfc1526b0 con 0x7fcdfc1513c0 2024-03-06T07:25:30.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.559+0000 7fce022a8640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcdfc150fe0 0x7fcdfc06edb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32968/0 (socket says 172.21.15.1:32968) 2024-03-06T07:25:30.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.559+0000 7fce022a8640 1 -- 172.21.15.1:0/2385170888 learned_addr learned my addr 172.21.15.1:0/2385170888 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:30.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce012a6640 1 -- 172.21.15.1:0/2385170888 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431207210 0 0) 0x7fcdfc151830 con 0x7fcdfc150fe0 2024-03-06T07:25:30.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce02aa9640 1 --2- 172.21.15.1:0/2385170888 >> v2:172.21.15.98:3300/0 conn(0x7fcdfc1513c0 0x7fcdfc0726a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce012a6640 1 -- 172.21.15.1:0/2385170888 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde8003520 con 0x7fcdfc150fe0 2024-03-06T07:25:30.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce012a6640 1 -- 172.21.15.1:0/2385170888 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282197204 0 0) 0x7fcdfc0a6140 con 0x7fcdfc072f20 2024-03-06T07:25:30.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce012a6640 1 -- 172.21.15.1:0/2385170888 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdfc151830 con 0x7fcdfc072f20 2024-03-06T07:25:30.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce012a6640 1 -- 172.21.15.1:0/2385170888 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3463562651 0 0) 0x7fcde8003520 con 0x7fcdfc150fe0 2024-03-06T07:25:30.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce012a6640 1 -- 172.21.15.1:0/2385170888 >> v2:172.21.15.98:3300/0 conn(0x7fcdfc1513c0 msgr2=0x7fcdfc0726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce012a6640 1 --2- 172.21.15.1:0/2385170888 >> v2:172.21.15.98:3300/0 conn(0x7fcdfc1513c0 0x7fcdfc0726a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce012a6640 1 -- 172.21.15.1:0/2385170888 >> v1:172.21.15.98:6789/0 conn(0x7fcdfc072f20 legacy=0x7fcdfc071f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce012a6640 1 -- 172.21.15.1:0/2385170888 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc151aa0 con 0x7fcdfc150fe0 2024-03-06T07:25:30.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce012a6640 1 -- 172.21.15.1:0/2385170888 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fcdf8001670 con 0x7fcdfc150fe0 2024-03-06T07:25:30.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce012a6640 1 -- 172.21.15.1:0/2385170888 >> v1:172.21.15.1:6789/0 conn(0x7fcdfc150fe0 legacy=0x7fcdfc06edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce012a6640 1 --2- 172.21.15.1:0/2385170888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcde8003b90 0x7fcde8003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce012a6640 1 --2- 172.21.15.1:0/2385170888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcde8004800 0x7fcde8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce012a6640 1 --2- 172.21.15.1:0/2385170888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcde8005210 0x7fcde800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce012a6640 1 -- 172.21.15.1:0/2385170888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde800c110 con 0x7fcde8005210 2024-03-06T07:25:30.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce012a6640 1 -- 172.21.15.1:0/2385170888 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde800c2c0 con 0x7fcde8003b90 2024-03-06T07:25:30.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce012a6640 1 -- 172.21.15.1:0/2385170888 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde800c4c0 con 0x7fcde8004800 2024-03-06T07:25:30.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce02aa9640 1 --2- 172.21.15.1:0/2385170888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcde8005210 0x7fcde800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce022a8640 1 --2- 172.21.15.1:0/2385170888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcde8003b90 0x7fcde8003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.560+0000 7fce01aa7640 1 --2- 172.21.15.1:0/2385170888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcde8004800 0x7fcde8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.561+0000 7fce02aa9640 1 -- 172.21.15.1:0/2385170888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcde8004800 msgr2=0x7fcde8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.561+0000 7fce02aa9640 1 --2- 172.21.15.1:0/2385170888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcde8004800 0x7fcde8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.561+0000 7fce02aa9640 1 -- 172.21.15.1:0/2385170888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcde8003b90 msgr2=0x7fcde8003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.561+0000 7fce02aa9640 1 --2- 172.21.15.1:0/2385170888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcde8003b90 0x7fcde8003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.561+0000 7fce02aa9640 1 -- 172.21.15.1:0/2385170888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde8003520 con 0x7fcde8005210 2024-03-06T07:25:30.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.561+0000 7fce02aa9640 1 -- 172.21.15.1:0/2385170888 reap_dead start 2024-03-06T07:25:30.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.561+0000 7fce02aa9640 1 --2- 172.21.15.1:0/2385170888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcde8005210 0x7fcde800b9c0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fcdf400f270 tx=0x7fcde8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=557ae7d69a3f69 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.561+0000 7fce012a6640 1 -- 172.21.15.1:0/2385170888 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf400ad20 con 0x7fcde8005210 2024-03-06T07:25:30.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.561+0000 7fce012a6640 1 -- 172.21.15.1:0/2385170888 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf402db30 con 0x7fcde8005210 2024-03-06T07:25:30.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.561+0000 7fce012a6640 1 -- 172.21.15.1:0/2385170888 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4003990 con 0x7fcde8005210 2024-03-06T07:25:30.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.562+0000 7fce0450c640 1 -- 172.21.15.1:0/2385170888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcde8005210 msgr2=0x7fcde800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.562+0000 7fce0450c640 1 --2- 172.21.15.1:0/2385170888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcde8005210 0x7fcde800b9c0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fcdf400f270 tx=0x7fcde8005830 comp rx=0 tx=0).stop 2024-03-06T07:25:30.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.562+0000 7fce0450c640 1 -- 172.21.15.1:0/2385170888 shutdown_connections 2024-03-06T07:25:30.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.562+0000 7fce0450c640 1 --2- 172.21.15.1:0/2385170888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcde8005210 0x7fcde800b9c0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.562+0000 7fce0450c640 1 -- 172.21.15.1:0/2385170888 >> 172.21.15.1:0/2385170888 conn(0x7fcdfc0c6860 msgr2=0x7fcdfc0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.562+0000 7fce0450c640 1 -- 172.21.15.1:0/2385170888 shutdown_connections 2024-03-06T07:25:30.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.562+0000 7fce0450c640 1 -- 172.21.15.1:0/2385170888 wait complete. 2024-03-06T07:25:30.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.563+0000 7fce0450c640 1 Processor -- start 2024-03-06T07:25:30.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.563+0000 7fce0450c640 1 -- start start 2024-03-06T07:25:30.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce0450c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcde8005210 0x7fcdfc1485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce0450c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdfc1e04d0 0x7fcdfc1dd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce0450c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdfc1dd8f0 0x7fcdfc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce0450c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc152340 con 0x7fcdfc1dd8f0 2024-03-06T07:25:30.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce0450c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc152060 con 0x7fcde8005210 2024-03-06T07:25:30.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce0450c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc06edb0 con 0x7fcdfc1e04d0 2024-03-06T07:25:30.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce02aa9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdfc1dd8f0 0x7fcdfc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce02aa9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdfc1dd8f0 0x7fcdfc0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53232/0 (socket says 172.21.15.1:53232) 2024-03-06T07:25:30.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce02aa9640 1 -- 172.21.15.1:0/2256226957 learned_addr learned my addr 172.21.15.1:0/2256226957 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce01aa7640 1 --2- 172.21.15.1:0/2256226957 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdfc1e04d0 0x7fcdfc1dd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce022a8640 1 --2- 172.21.15.1:0/2256226957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcde8005210 0x7fcdfc1485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce02aa9640 1 -- 172.21.15.1:0/2256226957 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdfc1e04d0 msgr2=0x7fcdfc1dd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce02aa9640 1 --2- 172.21.15.1:0/2256226957 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdfc1e04d0 0x7fcdfc1dd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce02aa9640 1 -- 172.21.15.1:0/2256226957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcde8005210 msgr2=0x7fcdfc1485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce02aa9640 1 --2- 172.21.15.1:0/2256226957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcde8005210 0x7fcdfc1485e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce02aa9640 1 -- 172.21.15.1:0/2256226957 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc1dfe40 con 0x7fcdfc1dd8f0 2024-03-06T07:25:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce022a8640 1 --2- 172.21.15.1:0/2256226957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcde8005210 0x7fcdfc1485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.564+0000 7fce02aa9640 1 --2- 172.21.15.1:0/2256226957 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdfc1dd8f0 0x7fcdfc0bfec0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4008cb0 tx=0x7fcdf400a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.565+0000 7fcdf2ffd640 1 -- 172.21.15.1:0/2256226957 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf400ad20 con 0x7fcdfc1dd8f0 2024-03-06T07:25:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.565+0000 7fcdf2ffd640 1 -- 172.21.15.1:0/2256226957 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf4003b90 con 0x7fcdfc1dd8f0 2024-03-06T07:25:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.565+0000 7fce0450c640 1 -- 172.21.15.1:0/2256226957 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdfc0ac7d0 con 0x7fcdfc1dd8f0 2024-03-06T07:25:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.565+0000 7fcdf2ffd640 1 -- 172.21.15.1:0/2256226957 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4004020 con 0x7fcdfc1dd8f0 2024-03-06T07:25:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.565+0000 7fce0450c640 1 -- 172.21.15.1:0/2256226957 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdfc1eaf90 con 0x7fcdfc1dd8f0 2024-03-06T07:25:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.566+0000 7fce0450c640 1 -- 172.21.15.1:0/2256226957 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdfc1eaac0 con 0x7fcdfc1dd8f0 2024-03-06T07:25:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.566+0000 7fcdf2ffd640 1 -- 172.21.15.1:0/2256226957 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcdf4005d10 con 0x7fcdfc1dd8f0 2024-03-06T07:25:30.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.566+0000 7fcdf2ffd640 1 --2- 172.21.15.1:0/2256226957 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcde006b500 0x7fcde006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.567+0000 7fcdf2ffd640 1 -- 172.21.15.1:0/2256226957 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcdf40abad0 con 0x7fcdfc1dd8f0 2024-03-06T07:25:30.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.567+0000 7fce022a8640 1 --2- 172.21.15.1:0/2256226957 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcde006b500 0x7fcde006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.567+0000 7fce022a8640 1 --2- 172.21.15.1:0/2256226957 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcde006b500 0x7fcde006d9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8000c00 tx=0x7fcdf8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.571+0000 7fcdf2ffd640 1 -- 172.21.15.1:0/2256226957 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcdf407a7c0 con 0x7fcdfc1dd8f0 2024-03-06T07:25:30.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.752+0000 7fce0450c640 1 -- 172.21.15.1:0/2256226957 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcdfc067fd0 con 0x7fcdfc1dd8f0 2024-03-06T07:25:30.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.753+0000 7fcdf2ffd640 1 -- 172.21.15.1:0/2256226957 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+16785 (secure 0 0 0) 0x7fcdf407a0c0 con 0x7fcdfc1dd8f0 2024-03-06T07:25:30.755 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:30.755 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":134,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:25:29.792522+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:28.046435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:28.753643+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:30.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.755+0000 7fce0450c640 1 -- 172.21.15.1:0/2256226957 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcde006b500 msgr2=0x7fcde006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.755+0000 7fce0450c640 1 --2- 172.21.15.1:0/2256226957 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcde006b500 0x7fcde006d9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8000c00 tx=0x7fcdf8009040 comp rx=0 tx=0).stop 2024-03-06T07:25:30.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.755+0000 7fce0450c640 1 -- 172.21.15.1:0/2256226957 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdfc1dd8f0 msgr2=0x7fcdfc0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.755+0000 7fce0450c640 1 --2- 172.21.15.1:0/2256226957 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdfc1dd8f0 0x7fcdfc0bfec0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4008cb0 tx=0x7fcdf400a860 comp rx=0 tx=0).stop 2024-03-06T07:25:30.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.755+0000 7fce0450c640 1 -- 172.21.15.1:0/2256226957 shutdown_connections 2024-03-06T07:25:30.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.755+0000 7fce0450c640 1 --2- 172.21.15.1:0/2256226957 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcde006b500 0x7fcde006d9c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.755+0000 7fce0450c640 1 --2- 172.21.15.1:0/2256226957 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdfc1dd8f0 0x7fcdfc0bfec0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.755+0000 7fce0450c640 1 --2- 172.21.15.1:0/2256226957 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdfc1e04d0 0x7fcdfc1dd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.755+0000 7fce0450c640 1 --2- 172.21.15.1:0/2256226957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcde8005210 0x7fcdfc1485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.755+0000 7fce0450c640 1 -- 172.21.15.1:0/2256226957 >> 172.21.15.1:0/2256226957 conn(0x7fcdfc0c6860 msgr2=0x7fcdfc0c4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.755+0000 7fce0450c640 1 -- 172.21.15.1:0/2256226957 shutdown_connections 2024-03-06T07:25:30.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.755+0000 7fce0450c640 1 -- 172.21.15.1:0/2256226957 wait complete. 2024-03-06T07:25:30.767 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:30.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.866+0000 7f3108d25640 1 Processor -- start 2024-03-06T07:25:30.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.866+0000 7f3108d25640 1 -- start start 2024-03-06T07:25:30.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.867+0000 7f3108d25640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3104152650 0x7f3104155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.867+0000 7f3108d25640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f31040bf9d0 0x7f31040c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.867+0000 7f3108d25640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31040c2fc0 con 0x7f31040c6b10 2024-03-06T07:25:30.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.867+0000 7f3108d25640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3104155800 con 0x7f31040bf9d0 2024-03-06T07:25:30.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.867+0000 7f3108d25640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3104158da0 con 0x7f3104152650 2024-03-06T07:25:30.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.867+0000 7f3102d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f31040bf9d0 0x7f31040c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.867+0000 7f30fbfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f31040c6b10 0x7f3104158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32976/0 (socket says 172.21.15.1:32976) 2024-03-06T07:25:30.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.867+0000 7f30fbfff640 1 -- 172.21.15.1:0/278557915 learned_addr learned my addr 172.21.15.1:0/278557915 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:30.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.868+0000 7f310259c640 1 --2- 172.21.15.1:0/278557915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3104152650 0x7f3104155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.868+0000 7f3102d9d640 1 -- 172.21.15.1:0/278557915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3104152650 msgr2=0x7f3104155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.868+0000 7f3102d9d640 1 --2- 172.21.15.1:0/278557915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3104152650 0x7f3104155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.868+0000 7f3102d9d640 1 -- 172.21.15.1:0/278557915 >> v1:172.21.15.1:6789/0 conn(0x7f31040c6b10 legacy=0x7f3104158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.868+0000 7f3101d9b640 1 -- 172.21.15.1:0/278557915 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142516828 0 0) 0x7f31040c2fc0 con 0x7f31040c6b10 2024-03-06T07:25:30.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.868+0000 7f3102d9d640 1 -- 172.21.15.1:0/278557915 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31040c6690 con 0x7f31040bf9d0 2024-03-06T07:25:30.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.868+0000 7f310259c640 1 --2- 172.21.15.1:0/278557915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3104152650 0x7f3104155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:30.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.868+0000 7f3102d9d640 1 --2- 172.21.15.1:0/278557915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31040bf9d0 0x7f31040c1460 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009d90 tx=0x7f30fc02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77df9c9613c5c890 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.869+0000 7f3101d9b640 1 -- 172.21.15.1:0/278557915 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc039070 con 0x7f31040bf9d0 2024-03-06T07:25:30.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.869+0000 7f3101d9b640 1 -- 172.21.15.1:0/278557915 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30fc02cbc0 con 0x7f31040bf9d0 2024-03-06T07:25:30.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.869+0000 7f3101d9b640 1 -- 172.21.15.1:0/278557915 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc034440 con 0x7f31040bf9d0 2024-03-06T07:25:30.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.869+0000 7f3108d25640 1 -- 172.21.15.1:0/278557915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31040bf9d0 msgr2=0x7f31040c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.869+0000 7f3108d25640 1 --2- 172.21.15.1:0/278557915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31040bf9d0 0x7f31040c1460 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009d90 tx=0x7f30fc02be10 comp rx=0 tx=0).stop 2024-03-06T07:25:30.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.869+0000 7f3108d25640 1 -- 172.21.15.1:0/278557915 shutdown_connections 2024-03-06T07:25:30.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.869+0000 7f3108d25640 1 --2- 172.21.15.1:0/278557915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31040bf9d0 0x7f31040c1460 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.869+0000 7f3108d25640 1 --2- 172.21.15.1:0/278557915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3104152650 0x7f3104155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.869+0000 7f3108d25640 1 -- 172.21.15.1:0/278557915 >> 172.21.15.1:0/278557915 conn(0x7f3104066ee0 msgr2=0x7f31040672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.869+0000 7f3108d25640 1 -- 172.21.15.1:0/278557915 shutdown_connections 2024-03-06T07:25:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.869+0000 7f3108d25640 1 -- 172.21.15.1:0/278557915 wait complete. 2024-03-06T07:25:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.870+0000 7f3108d25640 1 Processor -- start 2024-03-06T07:25:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.870+0000 7f3108d25640 1 -- start start 2024-03-06T07:25:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.871+0000 7f3108d25640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31040bf9d0 0x7f31041540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.871+0000 7f3108d25640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3104152650 0x7f310414e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.871+0000 7f3108d25640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31040b0240 0x7f310414e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.871+0000 7f3108d25640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31041551b0 con 0x7f31040b0240 2024-03-06T07:25:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.871+0000 7f3108d25640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31040c6fa0 con 0x7f3104152650 2024-03-06T07:25:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.871+0000 7f3108d25640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3104155490 con 0x7f31040bf9d0 2024-03-06T07:25:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.871+0000 7f3102d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31040b0240 0x7f310414e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.871+0000 7f3102d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31040b0240 0x7f310414e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53246/0 (socket says 172.21.15.1:53246) 2024-03-06T07:25:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.871+0000 7f3102d9d640 1 -- 172.21.15.1:0/720534611 learned_addr learned my addr 172.21.15.1:0/720534611 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.871+0000 7f310259c640 1 --2- 172.21.15.1:0/720534611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31040bf9d0 0x7f31041540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.871+0000 7f30fbfff640 1 --2- 172.21.15.1:0/720534611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3104152650 0x7f310414e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.871+0000 7f3102d9d640 1 -- 172.21.15.1:0/720534611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31040bf9d0 msgr2=0x7f31041540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.872+0000 7f3102d9d640 1 --2- 172.21.15.1:0/720534611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31040bf9d0 0x7f31041540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.872+0000 7f3102d9d640 1 -- 172.21.15.1:0/720534611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3104152650 msgr2=0x7f310414e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.872+0000 7f3102d9d640 1 --2- 172.21.15.1:0/720534611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3104152650 0x7f310414e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.872+0000 7f3102d9d640 1 -- 172.21.15.1:0/720534611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31041e0e60 con 0x7f31040b0240 2024-03-06T07:25:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.872+0000 7f3102d9d640 1 --2- 172.21.15.1:0/720534611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31040b0240 0x7f310414e7e0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009eb0 tx=0x7f30fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.872+0000 7f30faffd640 1 -- 172.21.15.1:0/720534611 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc039050 con 0x7f31040b0240 2024-03-06T07:25:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.872+0000 7f30faffd640 1 -- 172.21.15.1:0/720534611 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30fc045070 con 0x7f31040b0240 2024-03-06T07:25:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.872+0000 7f3108d25640 1 -- 172.21.15.1:0/720534611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31040afe90 con 0x7f31040b0240 2024-03-06T07:25:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.872+0000 7f30faffd640 1 -- 172.21.15.1:0/720534611 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc03e830 con 0x7f31040b0240 2024-03-06T07:25:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.872+0000 7f3108d25640 1 -- 172.21.15.1:0/720534611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3104154620 con 0x7f31040b0240 2024-03-06T07:25:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.873+0000 7f3108d25640 1 -- 172.21.15.1:0/720534611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3104067fd0 con 0x7f31040b0240 2024-03-06T07:25:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.873+0000 7f30faffd640 1 -- 172.21.15.1:0/720534611 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f30fc03e9d0 con 0x7f31040b0240 2024-03-06T07:25:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.874+0000 7f30faffd640 1 --2- 172.21.15.1:0/720534611 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30e406b4d0 0x7f30e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.874+0000 7f30faffd640 1 -- 172.21.15.1:0/720534611 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f30fc0ad1b0 con 0x7f31040b0240 2024-03-06T07:25:30.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.874+0000 7f310259c640 1 --2- 172.21.15.1:0/720534611 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30e406b4d0 0x7f30e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.874+0000 7f310259c640 1 --2- 172.21.15.1:0/720534611 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30e406b4d0 0x7f30e406d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f30f400a990 tx=0x7f30f4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:30.878+0000 7f30faffd640 1 -- 172.21.15.1:0/720534611 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30fc07bea0 con 0x7f31040b0240 2024-03-06T07:25:31.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.060+0000 7f3108d25640 1 -- 172.21.15.1:0/720534611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31040b06f0 con 0x7f31040b0240 2024-03-06T07:25:31.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.061+0000 7f30faffd640 1 -- 172.21.15.1:0/720534611 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+16785 (secure 0 0 0) 0x7f30fc07b7a0 con 0x7f31040b0240 2024-03-06T07:25:31.064 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:31.064 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":134,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:25:29.792522+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:28.046435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:28.753643+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:31.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.063+0000 7f3108d25640 1 -- 172.21.15.1:0/720534611 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30e406b4d0 msgr2=0x7f30e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.063+0000 7f3108d25640 1 --2- 172.21.15.1:0/720534611 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30e406b4d0 0x7f30e406d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f30f400a990 tx=0x7f30f4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:31.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.063+0000 7f3108d25640 1 -- 172.21.15.1:0/720534611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31040b0240 msgr2=0x7f310414e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.063+0000 7f3108d25640 1 --2- 172.21.15.1:0/720534611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31040b0240 0x7f310414e7e0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009eb0 tx=0x7f30fc004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:31.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.064+0000 7f3108d25640 1 -- 172.21.15.1:0/720534611 shutdown_connections 2024-03-06T07:25:31.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.064+0000 7f3108d25640 1 --2- 172.21.15.1:0/720534611 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30e406b4d0 0x7f30e406d990 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.064+0000 7f3108d25640 1 --2- 172.21.15.1:0/720534611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31040b0240 0x7f310414e7e0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.064+0000 7f3108d25640 1 --2- 172.21.15.1:0/720534611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3104152650 0x7f310414e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.064+0000 7f3108d25640 1 --2- 172.21.15.1:0/720534611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31040bf9d0 0x7f31041540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.064+0000 7f3108d25640 1 -- 172.21.15.1:0/720534611 >> 172.21.15.1:0/720534611 conn(0x7f3104066ee0 msgr2=0x7f31040c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.064+0000 7f3108d25640 1 -- 172.21.15.1:0/720534611 shutdown_connections 2024-03-06T07:25:31.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.064+0000 7f3108d25640 1 -- 172.21.15.1:0/720534611 wait complete. 2024-03-06T07:25:31.077 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:25:31.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.179+0000 7fd4289f7640 1 Processor -- start 2024-03-06T07:25:31.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.179+0000 7fd4289f7640 1 -- start start 2024-03-06T07:25:31.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.180+0000 7fd4289f7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd4240afc40 0x7fd424160b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.180+0000 7fd4289f7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4240c7180 con 0x7fd4240af6f0 2024-03-06T07:25:31.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.180+0000 7fd4289f7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd424155460 con 0x7fd424152820 2024-03-06T07:25:31.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.180+0000 7fd4289f7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4240c6bc0 con 0x7fd4240afc40 2024-03-06T07:25:31.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.180+0000 7fd421d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd4240af6f0 0x7fd4240c73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32982/0 (socket says 172.21.15.1:32982) 2024-03-06T07:25:31.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.180+0000 7fd421d9b640 1 -- 172.21.15.1:0/3144751784 learned_addr learned my addr 172.21.15.1:0/3144751784 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:31.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.180+0000 7fd42159a640 1 -- 172.21.15.1:0/3144751784 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356121400 0 0) 0x7fd4240c7180 con 0x7fd4240af6f0 2024-03-06T07:25:31.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.180+0000 7fd422d9d640 1 --2- 172.21.15.1:0/3144751784 >> v2:172.21.15.98:3300/0 conn(0x7fd4240afc40 0x7fd424160b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.180+0000 7fd42159a640 1 -- 172.21.15.1:0/3144751784 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd410003520 con 0x7fd4240af6f0 2024-03-06T07:25:31.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42159a640 1 -- 172.21.15.1:0/3144751784 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2558183105 0 0) 0x7fd410003520 con 0x7fd4240af6f0 2024-03-06T07:25:31.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42159a640 1 -- 172.21.15.1:0/3144751784 >> v2:172.21.15.98:3300/0 conn(0x7fd4240afc40 msgr2=0x7fd424160b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42159a640 1 --2- 172.21.15.1:0/3144751784 >> v2:172.21.15.98:3300/0 conn(0x7fd4240afc40 0x7fd424160b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42159a640 1 -- 172.21.15.1:0/3144751784 >> v1:172.21.15.98:6789/0 conn(0x7fd424152820 legacy=0x7fd4240c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42159a640 1 -- 172.21.15.1:0/3144751784 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4240c6ea0 con 0x7fd4240af6f0 2024-03-06T07:25:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42159a640 1 -- 172.21.15.1:0/3144751784 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fd40c001670 con 0x7fd4240af6f0 2024-03-06T07:25:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42159a640 1 -- 172.21.15.1:0/3144751784 >> v1:172.21.15.1:6789/0 conn(0x7fd4240af6f0 legacy=0x7fd4240c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42159a640 1 --2- 172.21.15.1:0/3144751784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4100039b0 0x7fd410003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42159a640 1 --2- 172.21.15.1:0/3144751784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd410004800 0x7fd410004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42159a640 1 --2- 172.21.15.1:0/3144751784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4100051c0 0x7fd41000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42159a640 1 -- 172.21.15.1:0/3144751784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4240c6bc0 con 0x7fd4100039b0 2024-03-06T07:25:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42159a640 1 -- 172.21.15.1:0/3144751784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41000c060 con 0x7fd4100051c0 2024-03-06T07:25:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42159a640 1 -- 172.21.15.1:0/3144751784 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd41000c260 con 0x7fd410004800 2024-03-06T07:25:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42259c640 1 --2- 172.21.15.1:0/3144751784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4100039b0 0x7fd410003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd421d9b640 1 --2- 172.21.15.1:0/3144751784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd410004800 0x7fd410004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd422d9d640 1 --2- 172.21.15.1:0/3144751784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4100051c0 0x7fd41000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42259c640 1 -- 172.21.15.1:0/3144751784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd410004800 msgr2=0x7fd410004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42259c640 1 --2- 172.21.15.1:0/3144751784 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd410004800 0x7fd410004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42259c640 1 -- 172.21.15.1:0/3144751784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4100051c0 msgr2=0x7fd41000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42259c640 1 --2- 172.21.15.1:0/3144751784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4100051c0 0x7fd41000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd42259c640 1 -- 172.21.15.1:0/3144751784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd410003520 con 0x7fd4100039b0 2024-03-06T07:25:31.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd422d9d640 1 --2- 172.21.15.1:0/3144751784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4100051c0 0x7fd41000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:31.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.181+0000 7fd422d9d640 1 -- 172.21.15.1:0/3144751784 reap_dead start 2024-03-06T07:25:31.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.182+0000 7fd42259c640 1 --2- 172.21.15.1:0/3144751784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4100039b0 0x7fd410003dd0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fd414001e50 tx=0x7fd410005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d1e4862e9f53e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.182+0000 7fd42159a640 1 -- 172.21.15.1:0/3144751784 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd414001a70 con 0x7fd4100039b0 2024-03-06T07:25:31.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.182+0000 7fd42159a640 1 -- 172.21.15.1:0/3144751784 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd414003b70 con 0x7fd4100039b0 2024-03-06T07:25:31.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.182+0000 7fd42159a640 1 -- 172.21.15.1:0/3144751784 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4140050d0 con 0x7fd4100039b0 2024-03-06T07:25:31.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.183+0000 7fd4289f7640 1 -- 172.21.15.1:0/3144751784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4100039b0 msgr2=0x7fd410003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.183+0000 7fd4289f7640 1 --2- 172.21.15.1:0/3144751784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4100039b0 0x7fd410003dd0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fd414001e50 tx=0x7fd410005830 comp rx=0 tx=0).stop 2024-03-06T07:25:31.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.183+0000 7fd4289f7640 1 -- 172.21.15.1:0/3144751784 shutdown_connections 2024-03-06T07:25:31.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.183+0000 7fd4289f7640 1 --2- 172.21.15.1:0/3144751784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4100039b0 0x7fd410003dd0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.183+0000 7fd4289f7640 1 -- 172.21.15.1:0/3144751784 >> 172.21.15.1:0/3144751784 conn(0x7fd424072720 msgr2=0x7fd424072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.183+0000 7fd4289f7640 1 -- 172.21.15.1:0/3144751784 shutdown_connections 2024-03-06T07:25:31.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.183+0000 7fd4289f7640 1 -- 172.21.15.1:0/3144751784 wait complete. 2024-03-06T07:25:31.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.184+0000 7fd4289f7640 1 Processor -- start 2024-03-06T07:25:31.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.184+0000 7fd4289f7640 1 -- start start 2024-03-06T07:25:31.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd4289f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4100039b0 0x7fd424154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd4289f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd424156570 0x7fd424157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd4289f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4241ee250 0x7fd4241ee6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd4289f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd424152d80 con 0x7fd4241ee250 2024-03-06T07:25:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd4289f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd424152c00 con 0x7fd424156570 2024-03-06T07:25:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd4289f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd424155f00 con 0x7fd4100039b0 2024-03-06T07:25:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd422d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4241ee250 0x7fd4241ee6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd422d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4241ee250 0x7fd4241ee6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53274/0 (socket says 172.21.15.1:53274) 2024-03-06T07:25:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd422d9d640 1 -- 172.21.15.1:0/3518964860 learned_addr learned my addr 172.21.15.1:0/3518964860 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd42259c640 1 --2- 172.21.15.1:0/3518964860 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4100039b0 0x7fd424154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd421d9b640 1 --2- 172.21.15.1:0/3518964860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd424156570 0x7fd424157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd422d9d640 1 -- 172.21.15.1:0/3518964860 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4100039b0 msgr2=0x7fd424154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd422d9d640 1 --2- 172.21.15.1:0/3518964860 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4100039b0 0x7fd424154520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd422d9d640 1 -- 172.21.15.1:0/3518964860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd424156570 msgr2=0x7fd424157960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd422d9d640 1 --2- 172.21.15.1:0/3518964860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd424156570 0x7fd424157960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd422d9d640 1 -- 172.21.15.1:0/3518964860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd424156ae0 con 0x7fd4241ee250 2024-03-06T07:25:31.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd421d9b640 1 --2- 172.21.15.1:0/3518964860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd424156570 0x7fd424157960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:31.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd422d9d640 1 --2- 172.21.15.1:0/3518964860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4241ee250 0x7fd4241ee6d0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fd41c011490 tx=0x7fd41c011950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd41affd640 1 -- 172.21.15.1:0/3518964860 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41c019020 con 0x7fd4241ee250 2024-03-06T07:25:31.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd41affd640 1 -- 172.21.15.1:0/3518964860 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd41c004440 con 0x7fd4241ee250 2024-03-06T07:25:31.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd4289f7640 1 -- 172.21.15.1:0/3518964860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd424153a40 con 0x7fd4241ee250 2024-03-06T07:25:31.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd41affd640 1 -- 172.21.15.1:0/3518964860 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41c003850 con 0x7fd4241ee250 2024-03-06T07:25:31.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.185+0000 7fd4289f7640 1 -- 172.21.15.1:0/3518964860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4241e85f0 con 0x7fd4241ee250 2024-03-06T07:25:31.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.187+0000 7fd4289f7640 1 -- 172.21.15.1:0/3518964860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4240686e0 con 0x7fd4241ee250 2024-03-06T07:25:31.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.187+0000 7fd41affd640 1 -- 172.21.15.1:0/3518964860 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd41c01c020 con 0x7fd4241ee250 2024-03-06T07:25:31.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.187+0000 7fd41affd640 1 --2- 172.21.15.1:0/3518964860 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd40406b500 0x7fd40406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.187+0000 7fd41affd640 1 -- 172.21.15.1:0/3518964860 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd41c08a300 con 0x7fd4241ee250 2024-03-06T07:25:31.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.188+0000 7fd42259c640 1 --2- 172.21.15.1:0/3518964860 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd40406b500 0x7fd40406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.188+0000 7fd42259c640 1 --2- 172.21.15.1:0/3518964860 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd40406b500 0x7fd40406d9c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd414034000 tx=0x7fd41403b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.192+0000 7fd41affd640 1 -- 172.21.15.1:0/3518964860 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd41c023030 con 0x7fd4241ee250 2024-03-06T07:25:31.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.414+0000 7fd4289f7640 1 -- 172.21.15.1:0/3518964860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd4240c5b40 con 0x7fd4241ee250 2024-03-06T07:25:31.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.432+0000 7fd41affd640 1 -- 172.21.15.1:0/3518964860 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v120) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd41c059080 con 0x7fd4241ee250 2024-03-06T07:25:31.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.435+0000 7fd4289f7640 1 -- 172.21.15.1:0/3518964860 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd40406b500 msgr2=0x7fd40406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.435+0000 7fd4289f7640 1 --2- 172.21.15.1:0/3518964860 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd40406b500 0x7fd40406d9c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd414034000 tx=0x7fd41403b040 comp rx=0 tx=0).stop 2024-03-06T07:25:31.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.435+0000 7fd4289f7640 1 -- 172.21.15.1:0/3518964860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4241ee250 msgr2=0x7fd4241ee6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.435+0000 7fd4289f7640 1 --2- 172.21.15.1:0/3518964860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4241ee250 0x7fd4241ee6d0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fd41c011490 tx=0x7fd41c011950 comp rx=0 tx=0).stop 2024-03-06T07:25:31.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.435+0000 7fd4289f7640 1 -- 172.21.15.1:0/3518964860 shutdown_connections 2024-03-06T07:25:31.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.435+0000 7fd4289f7640 1 --2- 172.21.15.1:0/3518964860 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd40406b500 0x7fd40406d9c0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.435+0000 7fd4289f7640 1 --2- 172.21.15.1:0/3518964860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4241ee250 0x7fd4241ee6d0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.435+0000 7fd4289f7640 1 --2- 172.21.15.1:0/3518964860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd424156570 0x7fd424157960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.435+0000 7fd4289f7640 1 --2- 172.21.15.1:0/3518964860 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4100039b0 0x7fd424154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.435+0000 7fd4289f7640 1 -- 172.21.15.1:0/3518964860 >> 172.21.15.1:0/3518964860 conn(0x7fd424072720 msgr2=0x7fd4240ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.435+0000 7fd4289f7640 1 -- 172.21.15.1:0/3518964860 shutdown_connections 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.435+0000 7fd4289f7640 1 -- 172.21.15.1:0/3518964860 wait complete. 2024-03-06T07:25:31.447 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:25:31.448 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:25:31.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.550+0000 7f2746df7640 1 Processor -- start 2024-03-06T07:25:31.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.550+0000 7f2746df7640 1 -- start start 2024-03-06T07:25:31.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.551+0000 7f2746df7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2740152650 0x7f2740155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.551+0000 7f2746df7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27400c2fc0 con 0x7f27400c6b10 2024-03-06T07:25:31.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.551+0000 7f2746df7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2740158da0 con 0x7f27400bf9d0 2024-03-06T07:25:31.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.551+0000 7f2746df7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2740155800 con 0x7f2740152650 2024-03-06T07:25:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.551+0000 7f2744b93640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2740152650 0x7f2740155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.551+0000 7f273ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f27400c6b10 0x7f2740158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32992/0 (socket says 172.21.15.1:32992) 2024-03-06T07:25:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.551+0000 7f273ffff640 1 -- 172.21.15.1:0/1529434879 learned_addr learned my addr 172.21.15.1:0/1529434879 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.551+0000 7f2744b93640 1 -- 172.21.15.1:0/1529434879 >> v1:172.21.15.98:6789/0 conn(0x7f27400bf9d0 legacy=0x7f27400c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.551+0000 7f2744b93640 1 -- 172.21.15.1:0/1529434879 >> v1:172.21.15.1:6789/0 conn(0x7f27400c6b10 legacy=0x7f2740158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.551+0000 7f2744b93640 1 -- 172.21.15.1:0/1529434879 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27400c6690 con 0x7f2740152650 2024-03-06T07:25:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.551+0000 7f273f7fe640 1 -- 172.21.15.1:0/1529434879 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290685152 0 0) 0x7f27400c2fc0 con 0x7f27400c6b10 2024-03-06T07:25:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.552+0000 7f2744b93640 1 --2- 172.21.15.1:0/1529434879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2740152650 0x7f2740155c90 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f2738009d90 tx=0x7f273802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8515d43923ead788 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.553+0000 7f273f7fe640 1 -- 172.21.15.1:0/1529434879 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738039070 con 0x7f2740152650 2024-03-06T07:25:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.553+0000 7f273f7fe640 1 -- 172.21.15.1:0/1529434879 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273802cbd0 con 0x7f2740152650 2024-03-06T07:25:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.553+0000 7f273f7fe640 1 -- 172.21.15.1:0/1529434879 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738034440 con 0x7f2740152650 2024-03-06T07:25:31.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.553+0000 7f2746df7640 1 -- 172.21.15.1:0/1529434879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2740152650 msgr2=0x7f2740155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.553+0000 7f2746df7640 1 --2- 172.21.15.1:0/1529434879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2740152650 0x7f2740155c90 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f2738009d90 tx=0x7f273802be20 comp rx=0 tx=0).stop 2024-03-06T07:25:31.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.553+0000 7f2746df7640 1 -- 172.21.15.1:0/1529434879 shutdown_connections 2024-03-06T07:25:31.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.553+0000 7f2746df7640 1 --2- 172.21.15.1:0/1529434879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2740152650 0x7f2740155c90 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.553+0000 7f2746df7640 1 -- 172.21.15.1:0/1529434879 >> 172.21.15.1:0/1529434879 conn(0x7f2740066ee0 msgr2=0x7f27400672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.553+0000 7f2746df7640 1 -- 172.21.15.1:0/1529434879 shutdown_connections 2024-03-06T07:25:31.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.553+0000 7f2746df7640 1 -- 172.21.15.1:0/1529434879 wait complete. 2024-03-06T07:25:31.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.554+0000 7f2746df7640 1 Processor -- start 2024-03-06T07:25:31.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.554+0000 7f2746df7640 1 -- start start 2024-03-06T07:25:31.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.555+0000 7f2746df7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27400bf9d0 0x7f27401508f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.555+0000 7f2746df7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2740152650 0x7f27401eb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.555+0000 7f2746df7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f274014ba40 0x7f27401eb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.555+0000 7f2746df7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27401551b0 con 0x7f274014ba40 2024-03-06T07:25:31.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.555+0000 7f2746df7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27400c6fa0 con 0x7f27400bf9d0 2024-03-06T07:25:31.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.555+0000 7f2746df7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2740155490 con 0x7f2740152650 2024-03-06T07:25:31.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.555+0000 7f2745394640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f274014ba40 0x7f27401eb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.555+0000 7f2745394640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f274014ba40 0x7f27401eb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53280/0 (socket says 172.21.15.1:53280) 2024-03-06T07:25:31.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.555+0000 7f2745394640 1 -- 172.21.15.1:0/502252199 learned_addr learned my addr 172.21.15.1:0/502252199 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:31.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.555+0000 7f273ffff640 1 --2- 172.21.15.1:0/502252199 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2740152650 0x7f27401eb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.555+0000 7f2744b93640 1 --2- 172.21.15.1:0/502252199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27400bf9d0 0x7f27401508f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.556+0000 7f2745394640 1 -- 172.21.15.1:0/502252199 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2740152650 msgr2=0x7f27401eb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.556+0000 7f2745394640 1 --2- 172.21.15.1:0/502252199 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2740152650 0x7f27401eb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.556+0000 7f2745394640 1 -- 172.21.15.1:0/502252199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27400bf9d0 msgr2=0x7f27401508f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.556+0000 7f2745394640 1 --2- 172.21.15.1:0/502252199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27400bf9d0 0x7f27401508f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.556+0000 7f2745394640 1 -- 172.21.15.1:0/502252199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2740153470 con 0x7f274014ba40 2024-03-06T07:25:31.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.556+0000 7f2744b93640 1 --2- 172.21.15.1:0/502252199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27400bf9d0 0x7f27401508f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:31.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.556+0000 7f2745394640 1 --2- 172.21.15.1:0/502252199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f274014ba40 0x7f27401eb760 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f272800b440 tx=0x7f272800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.556+0000 7f273d7fa640 1 -- 172.21.15.1:0/502252199 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2728015020 con 0x7f274014ba40 2024-03-06T07:25:31.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.556+0000 7f273d7fa640 1 -- 172.21.15.1:0/502252199 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2728011830 con 0x7f274014ba40 2024-03-06T07:25:31.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.556+0000 7f273d7fa640 1 -- 172.21.15.1:0/502252199 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2728011c20 con 0x7f274014ba40 2024-03-06T07:25:31.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.556+0000 7f2746df7640 1 -- 172.21.15.1:0/502252199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2740154130 con 0x7f274014ba40 2024-03-06T07:25:31.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.556+0000 7f2746df7640 1 -- 172.21.15.1:0/502252199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27401546f0 con 0x7f274014ba40 2024-03-06T07:25:31.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.557+0000 7f2746df7640 1 -- 172.21.15.1:0/502252199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27400c2f50 con 0x7f274014ba40 2024-03-06T07:25:31.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.557+0000 7f273d7fa640 1 -- 172.21.15.1:0/502252199 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2728018020 con 0x7f274014ba40 2024-03-06T07:25:31.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.558+0000 7f273d7fa640 1 --2- 172.21.15.1:0/502252199 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f272006b4d0 0x7f272006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.558+0000 7f273d7fa640 1 -- 172.21.15.1:0/502252199 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2728089f30 con 0x7f274014ba40 2024-03-06T07:25:31.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.558+0000 7f2744b93640 1 --2- 172.21.15.1:0/502252199 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f272006b4d0 0x7f272006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.558+0000 7f2744b93640 1 --2- 172.21.15.1:0/502252199 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f272006b4d0 0x7f272006d990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2738036000 tx=0x7f2738030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.562+0000 7f273d7fa640 1 -- 172.21.15.1:0/502252199 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f27400c2f50 con 0x7f274014ba40 2024-03-06T07:25:31.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:31.780+0000 7f2746df7640 1 -- 172.21.15.1:0/502252199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2740067fd0 con 0x7f274014ba40 2024-03-06T07:25:31.788 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:31.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:32.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.442+0000 7f273d7fa640 1 -- 172.21.15.1:0/502252199 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v121) v1 ==== 122+0+0 (secure 0 0 0) 0x7f272801f030 con 0x7f274014ba40 2024-03-06T07:25:32.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.444+0000 7f2746df7640 1 -- 172.21.15.1:0/502252199 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f272006b4d0 msgr2=0x7f272006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.444+0000 7f2746df7640 1 --2- 172.21.15.1:0/502252199 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f272006b4d0 0x7f272006d990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2738036000 tx=0x7f2738030000 comp rx=0 tx=0).stop 2024-03-06T07:25:32.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.444+0000 7f2746df7640 1 -- 172.21.15.1:0/502252199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f274014ba40 msgr2=0x7f27401eb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.444+0000 7f2746df7640 1 --2- 172.21.15.1:0/502252199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f274014ba40 0x7f27401eb760 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f272800b440 tx=0x7f272800b900 comp rx=0 tx=0).stop 2024-03-06T07:25:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.444+0000 7f2746df7640 1 -- 172.21.15.1:0/502252199 shutdown_connections 2024-03-06T07:25:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.444+0000 7f2746df7640 1 --2- 172.21.15.1:0/502252199 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f272006b4d0 0x7f272006d990 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.444+0000 7f2746df7640 1 --2- 172.21.15.1:0/502252199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f274014ba40 0x7f27401eb760 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.444+0000 7f2746df7640 1 --2- 172.21.15.1:0/502252199 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2740152650 0x7f27401eb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.444+0000 7f2746df7640 1 --2- 172.21.15.1:0/502252199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27400bf9d0 0x7f27401508f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.444+0000 7f2746df7640 1 -- 172.21.15.1:0/502252199 >> 172.21.15.1:0/502252199 conn(0x7f2740066ee0 msgr2=0x7f27400c39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.444+0000 7f2746df7640 1 -- 172.21.15.1:0/502252199 shutdown_connections 2024-03-06T07:25:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.444+0000 7f2746df7640 1 -- 172.21.15.1:0/502252199 wait complete. 2024-03-06T07:25:32.457 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:25:32.457 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:32.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.557+0000 7f3a8637a640 1 Processor -- start 2024-03-06T07:25:32.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.557+0000 7f3a8637a640 1 -- start start 2024-03-06T07:25:32.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.557+0000 7f3a8637a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a80152820 0x7f3a800c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.557+0000 7f3a8637a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3a800af6f0 0x7f3a8015fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.557+0000 7f3a8637a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a800c7180 con 0x7f3a800c0670 2024-03-06T07:25:32.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a8637a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a80155460 con 0x7f3a800af6f0 2024-03-06T07:25:32.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a8637a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a800c6bc0 con 0x7f3a80152820 2024-03-06T07:25:32.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3a800c0670 0x7f3a800c73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32998/0 (socket says 172.21.15.1:32998) 2024-03-06T07:25:32.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7f7fe640 1 -- 172.21.15.1:0/540993452 learned_addr learned my addr 172.21.15.1:0/540993452 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:32.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7ffff640 1 --2- 172.21.15.1:0/540993452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a80152820 0x7f3a800c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a84917640 1 --2- 172.21.15.1:0/540993452 >> v2:172.21.15.98:3300/0 conn(0x7f3a800af6f0 0x7f3a8015fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7effd640 1 -- 172.21.15.1:0/540993452 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395607488 0 0) 0x7f3a800c7180 con 0x7f3a800c0670 2024-03-06T07:25:32.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7effd640 1 -- 172.21.15.1:0/540993452 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a68003520 con 0x7f3a800c0670 2024-03-06T07:25:32.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7effd640 1 -- 172.21.15.1:0/540993452 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1176165425 0 0) 0x7f3a68003520 con 0x7f3a800c0670 2024-03-06T07:25:32.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7effd640 1 -- 172.21.15.1:0/540993452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a80152820 msgr2=0x7f3a800c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7effd640 1 --2- 172.21.15.1:0/540993452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a80152820 0x7f3a800c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7effd640 1 -- 172.21.15.1:0/540993452 >> v2:172.21.15.98:3300/0 conn(0x7f3a800af6f0 msgr2=0x7f3a8015fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7effd640 1 --2- 172.21.15.1:0/540993452 >> v2:172.21.15.98:3300/0 conn(0x7f3a800af6f0 0x7f3a8015fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7effd640 1 -- 172.21.15.1:0/540993452 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a800c6ea0 con 0x7f3a800c0670 2024-03-06T07:25:32.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7effd640 1 -- 172.21.15.1:0/540993452 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f3a74001670 con 0x7f3a800c0670 2024-03-06T07:25:32.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a84917640 1 --2- 172.21.15.1:0/540993452 >> v2:172.21.15.98:3300/0 conn(0x7f3a800af6f0 0x7f3a8015fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:32.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7effd640 1 -- 172.21.15.1:0/540993452 >> v1:172.21.15.1:6789/0 conn(0x7f3a800c0670 legacy=0x7f3a800c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7effd640 1 --2- 172.21.15.1:0/540993452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a680039b0 0x7f3a68003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7effd640 1 --2- 172.21.15.1:0/540993452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a80152820 0x7f3a68004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7effd640 1 --2- 172.21.15.1:0/540993452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a68004d40 0x7f3a680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7effd640 1 -- 172.21.15.1:0/540993452 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a80155460 con 0x7f3a68004d40 2024-03-06T07:25:32.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7effd640 1 -- 172.21.15.1:0/540993452 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a800c6bc0 con 0x7f3a680039b0 2024-03-06T07:25:32.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a7effd640 1 -- 172.21.15.1:0/540993452 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6800b9c0 con 0x7f3a80152820 2024-03-06T07:25:32.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.558+0000 7f3a84917640 1 --2- 172.21.15.1:0/540993452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a68004d40 0x7f3a680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.559+0000 7f3a7ffff640 1 --2- 172.21.15.1:0/540993452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a680039b0 0x7f3a68003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.559+0000 7f3a7f7fe640 1 --2- 172.21.15.1:0/540993452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a80152820 0x7f3a68004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.559+0000 7f3a84917640 1 -- 172.21.15.1:0/540993452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a80152820 msgr2=0x7f3a68004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.559+0000 7f3a84917640 1 --2- 172.21.15.1:0/540993452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a80152820 0x7f3a68004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.559+0000 7f3a84917640 1 -- 172.21.15.1:0/540993452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a680039b0 msgr2=0x7f3a68003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.559+0000 7f3a84917640 1 --2- 172.21.15.1:0/540993452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a680039b0 0x7f3a68003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.559+0000 7f3a84917640 1 -- 172.21.15.1:0/540993452 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a68003520 con 0x7f3a68004d40 2024-03-06T07:25:32.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.559+0000 7f3a7ffff640 1 --2- 172.21.15.1:0/540993452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a680039b0 0x7f3a68003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:32.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.560+0000 7f3a84917640 1 --2- 172.21.15.1:0/540993452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a68004d40 0x7f3a680051c0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f3a7800e320 tx=0x7f3a68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92d7281a8be18a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.560+0000 7f3a7effd640 1 -- 172.21.15.1:0/540993452 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a78009de0 con 0x7f3a68004d40 2024-03-06T07:25:32.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.560+0000 7f3a7effd640 1 -- 172.21.15.1:0/540993452 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7802cde0 con 0x7f3a68004d40 2024-03-06T07:25:32.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.560+0000 7f3a7effd640 1 -- 172.21.15.1:0/540993452 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a78003b90 con 0x7f3a68004d40 2024-03-06T07:25:32.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.561+0000 7f3a8637a640 1 -- 172.21.15.1:0/540993452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a68004d40 msgr2=0x7f3a680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.561+0000 7f3a8637a640 1 --2- 172.21.15.1:0/540993452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a68004d40 0x7f3a680051c0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f3a7800e320 tx=0x7f3a68005830 comp rx=0 tx=0).stop 2024-03-06T07:25:32.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.561+0000 7f3a84917640 1 -- 172.21.15.1:0/540993452 reap_dead start 2024-03-06T07:25:32.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.561+0000 7f3a8637a640 1 -- 172.21.15.1:0/540993452 shutdown_connections 2024-03-06T07:25:32.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.561+0000 7f3a8637a640 1 -- 172.21.15.1:0/540993452 >> 172.21.15.1:0/540993452 conn(0x7f3a80072720 msgr2=0x7f3a80072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.561+0000 7f3a8637a640 1 -- 172.21.15.1:0/540993452 shutdown_connections 2024-03-06T07:25:32.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.561+0000 7f3a8637a640 1 -- 172.21.15.1:0/540993452 wait complete. 2024-03-06T07:25:32.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.562+0000 7f3a8637a640 1 Processor -- start 2024-03-06T07:25:32.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.562+0000 7f3a8637a640 1 -- start start 2024-03-06T07:25:32.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.562+0000 7f3a8637a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a801e2210 0x7f3a8014a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a8637a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a8014b3d0 0x7f3a8014abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a8637a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a801edf00 0x7f3a801ee380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a8637a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a800c7c80 con 0x7f3a8014b3d0 2024-03-06T07:25:32.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a8637a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a800c7b00 con 0x7f3a801e2210 2024-03-06T07:25:32.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a8637a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a80152c00 con 0x7f3a801edf00 2024-03-06T07:25:32.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a7f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a8014b3d0 0x7f3a8014abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a7f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a8014b3d0 0x7f3a8014abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53292/0 (socket says 172.21.15.1:53292) 2024-03-06T07:25:32.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a7f7fe640 1 -- 172.21.15.1:0/2250154899 learned_addr learned my addr 172.21.15.1:0/2250154899 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:32.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a84917640 1 --2- 172.21.15.1:0/2250154899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a801edf00 0x7f3a801ee380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a7ffff640 1 --2- 172.21.15.1:0/2250154899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a801e2210 0x7f3a8014a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a7f7fe640 1 -- 172.21.15.1:0/2250154899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a801edf00 msgr2=0x7f3a801ee380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a7f7fe640 1 --2- 172.21.15.1:0/2250154899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a801edf00 0x7f3a801ee380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a7f7fe640 1 -- 172.21.15.1:0/2250154899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a801e2210 msgr2=0x7f3a8014a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a7f7fe640 1 --2- 172.21.15.1:0/2250154899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a801e2210 0x7f3a8014a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a7f7fe640 1 -- 172.21.15.1:0/2250154899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a80154a40 con 0x7f3a8014b3d0 2024-03-06T07:25:32.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a7f7fe640 1 --2- 172.21.15.1:0/2250154899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a8014b3d0 0x7f3a8014abe0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f3a74007290 tx=0x7f3a74007690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a7cff9640 1 -- 172.21.15.1:0/2250154899 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7400f960 con 0x7f3a8014b3d0 2024-03-06T07:25:32.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a7cff9640 1 -- 172.21.15.1:0/2250154899 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a74018070 con 0x7f3a8014b3d0 2024-03-06T07:25:32.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a8637a640 1 -- 172.21.15.1:0/2250154899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a8014d600 con 0x7f3a8014b3d0 2024-03-06T07:25:32.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a7cff9640 1 -- 172.21.15.1:0/2250154899 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7400fd40 con 0x7f3a8014b3d0 2024-03-06T07:25:32.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.563+0000 7f3a8637a640 1 -- 172.21.15.1:0/2250154899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a8014dbc0 con 0x7f3a8014b3d0 2024-03-06T07:25:32.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.564+0000 7f3a8637a640 1 -- 172.21.15.1:0/2250154899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a800686e0 con 0x7f3a8014b3d0 2024-03-06T07:25:32.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.565+0000 7f3a7cff9640 1 -- 172.21.15.1:0/2250154899 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a7401d460 con 0x7f3a8014b3d0 2024-03-06T07:25:32.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.565+0000 7f3a7cff9640 1 --2- 172.21.15.1:0/2250154899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a6c06b500 0x7f3a6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.566+0000 7f3a7cff9640 1 -- 172.21.15.1:0/2250154899 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3a7408a460 con 0x7f3a8014b3d0 2024-03-06T07:25:32.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.566+0000 7f3a7ffff640 1 --2- 172.21.15.1:0/2250154899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a6c06b500 0x7f3a6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.566+0000 7f3a7ffff640 1 --2- 172.21.15.1:0/2250154899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a6c06b500 0x7f3a6c06d9c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009ce0 tx=0x7f3a70009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.570+0000 7f3a7cff9640 1 -- 172.21.15.1:0/2250154899 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a74059150 con 0x7f3a8014b3d0 2024-03-06T07:25:32.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.793+0000 7f3a8637a640 1 -- 172.21.15.1:0/2250154899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a800c0cc0 con 0x7f3a8014b3d0 2024-03-06T07:25:32.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.793+0000 7f3a7cff9640 1 -- 172.21.15.1:0/2250154899 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+4981 (secure 0 0 0) 0x7f3a74058a50 con 0x7f3a8014b3d0 2024-03-06T07:25:32.796 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:32.796 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2848968195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2848968195},{"type":"v1","addr":"172.21.15.1:6835","nonce":2848968195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7387,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1671554096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1671554096},{"type":"v1","addr":"172.21.15.98:6836","nonce":1671554096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2739580331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2739580331},{"type":"v1","addr":"172.21.15.98:6837","nonce":2739580331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:29.792840+0000","modified":"2024-03-06T07:25:31.781726+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"a","rank":0,"incarnation":119,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1706279305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1706279305},{"type":"v1","addr":"172.21.15.1:6834","nonce":1706279305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8301,"qdb_cluster":[8301]},"id":10}]} 2024-03-06T07:25:32.796 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 121 2024-03-06T07:25:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.795+0000 7f3a8637a640 1 -- 172.21.15.1:0/2250154899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a6c06b500 msgr2=0x7f3a6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.795+0000 7f3a8637a640 1 --2- 172.21.15.1:0/2250154899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a6c06b500 0x7f3a6c06d9c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009ce0 tx=0x7f3a70009190 comp rx=0 tx=0).stop 2024-03-06T07:25:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.795+0000 7f3a8637a640 1 -- 172.21.15.1:0/2250154899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a8014b3d0 msgr2=0x7f3a8014abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.795+0000 7f3a8637a640 1 --2- 172.21.15.1:0/2250154899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a8014b3d0 0x7f3a8014abe0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f3a74007290 tx=0x7f3a74007690 comp rx=0 tx=0).stop 2024-03-06T07:25:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.795+0000 7f3a8637a640 1 -- 172.21.15.1:0/2250154899 shutdown_connections 2024-03-06T07:25:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.795+0000 7f3a8637a640 1 --2- 172.21.15.1:0/2250154899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a6c06b500 0x7f3a6c06d9c0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.795+0000 7f3a8637a640 1 --2- 172.21.15.1:0/2250154899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a801edf00 0x7f3a801ee380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.795+0000 7f3a8637a640 1 --2- 172.21.15.1:0/2250154899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a8014b3d0 0x7f3a8014abe0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.795+0000 7f3a8637a640 1 --2- 172.21.15.1:0/2250154899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a801e2210 0x7f3a8014a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.795+0000 7f3a8637a640 1 -- 172.21.15.1:0/2250154899 >> 172.21.15.1:0/2250154899 conn(0x7f3a80072720 msgr2=0x7f3a80150030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.796+0000 7f3a8637a640 1 -- 172.21.15.1:0/2250154899 shutdown_connections 2024-03-06T07:25:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.796+0000 7f3a8637a640 1 -- 172.21.15.1:0/2250154899 wait complete. 2024-03-06T07:25:32.808 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:32.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.907+0000 7fbe53325640 1 Processor -- start 2024-03-06T07:25:32.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.907+0000 7fbe53325640 1 -- start start 2024-03-06T07:25:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.908+0000 7fbe53325640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbe4c150520 0x7fbe4c1543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.908+0000 7fbe53325640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe4c070c80 0x7fbe4c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.908+0000 7fbe53325640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c150900 con 0x7fbe4c0c32f0 2024-03-06T07:25:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.908+0000 7fbe53325640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe4c153ed0 con 0x7fbe4c150520 2024-03-06T07:25:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.908+0000 7fbe53325640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c071790 con 0x7fbe4c070c80 2024-03-06T07:25:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.908+0000 7fbe518c2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe4c070c80 0x7fbe4c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.908+0000 7fbe510c1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbe4c150520 0x7fbe4c1543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.909+0000 7fbe510c1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbe4c150520 0x7fbe4c1543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:43734/0 (socket says 172.21.15.1:43734) 2024-03-06T07:25:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.909+0000 7fbe508c0640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbe4c0c32f0 0x7fbe4c1537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:48058/0 (socket says 172.21.15.1:48058) 2024-03-06T07:25:32.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.909+0000 7fbe510c1640 1 -- 172.21.15.1:0/656222818 learned_addr learned my addr 172.21.15.1:0/656222818 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:32.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.909+0000 7fbe510c1640 1 -- 172.21.15.1:0/656222818 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe4c070c80 msgr2=0x7fbe4c0710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.909+0000 7fbe510c1640 1 --2- 172.21.15.1:0/656222818 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe4c070c80 0x7fbe4c0710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.909+0000 7fbe510c1640 1 -- 172.21.15.1:0/656222818 >> v1:172.21.15.98:6789/0 conn(0x7fbe4c0c32f0 legacy=0x7fbe4c1537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.909+0000 7fbe3ffff640 1 -- 172.21.15.1:0/656222818 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852836093 0 0) 0x7fbe4c150900 con 0x7fbe4c0c32f0 2024-03-06T07:25:32.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.909+0000 7fbe510c1640 1 -- 172.21.15.1:0/656222818 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4c157050 con 0x7fbe4c150520 2024-03-06T07:25:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.910+0000 7fbe510c1640 1 --2- 172.21.15.1:0/656222818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe4c150520 0x7fbe4c1543a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fbe48009870 tx=0x7fbe4802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c38ed5d11eb71a7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.910+0000 7fbe3ffff640 1 -- 172.21.15.1:0/656222818 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe48034070 con 0x7fbe4c150520 2024-03-06T07:25:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.910+0000 7fbe3ffff640 1 -- 172.21.15.1:0/656222818 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe48030840 con 0x7fbe4c150520 2024-03-06T07:25:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.910+0000 7fbe3ffff640 1 -- 172.21.15.1:0/656222818 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe48030c10 con 0x7fbe4c150520 2024-03-06T07:25:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.911+0000 7fbe53325640 1 -- 172.21.15.1:0/656222818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe4c150520 msgr2=0x7fbe4c1543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.911+0000 7fbe53325640 1 --2- 172.21.15.1:0/656222818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe4c150520 0x7fbe4c1543a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fbe48009870 tx=0x7fbe4802b980 comp rx=0 tx=0).stop 2024-03-06T07:25:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.911+0000 7fbe53325640 1 -- 172.21.15.1:0/656222818 shutdown_connections 2024-03-06T07:25:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.911+0000 7fbe53325640 1 --2- 172.21.15.1:0/656222818 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe4c070c80 0x7fbe4c0710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.911+0000 7fbe53325640 1 --2- 172.21.15.1:0/656222818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe4c150520 0x7fbe4c1543a0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.911+0000 7fbe53325640 1 -- 172.21.15.1:0/656222818 >> 172.21.15.1:0/656222818 conn(0x7fbe4c0c88f0 msgr2=0x7fbe4c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.911+0000 7fbe53325640 1 -- 172.21.15.1:0/656222818 shutdown_connections 2024-03-06T07:25:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.911+0000 7fbe53325640 1 -- 172.21.15.1:0/656222818 wait complete. 2024-03-06T07:25:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.912+0000 7fbe53325640 1 Processor -- start 2024-03-06T07:25:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.912+0000 7fbe53325640 1 -- start start 2024-03-06T07:25:32.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe53325640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe4c070c80 0x7fbe4c1ea080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe53325640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe4c0c32f0 0x7fbe4c0aae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe53325640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe4c150520 0x7fbe4c155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe53325640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c157330 con 0x7fbe4c0c32f0 2024-03-06T07:25:32.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe53325640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c1532a0 con 0x7fbe4c070c80 2024-03-06T07:25:32.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe53325640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c0c7ec0 con 0x7fbe4c150520 2024-03-06T07:25:32.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe508c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe4c0c32f0 0x7fbe4c0aae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe508c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe4c0c32f0 0x7fbe4c0aae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53296/0 (socket says 172.21.15.1:53296) 2024-03-06T07:25:32.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe508c0640 1 -- 172.21.15.1:0/1105876999 learned_addr learned my addr 172.21.15.1:0/1105876999 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:32.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe510c1640 1 --2- 172.21.15.1:0/1105876999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe4c070c80 0x7fbe4c1ea080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe518c2640 1 --2- 172.21.15.1:0/1105876999 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe4c150520 0x7fbe4c155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe508c0640 1 -- 172.21.15.1:0/1105876999 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe4c150520 msgr2=0x7fbe4c155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe508c0640 1 --2- 172.21.15.1:0/1105876999 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe4c150520 0x7fbe4c155b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe508c0640 1 -- 172.21.15.1:0/1105876999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe4c070c80 msgr2=0x7fbe4c1ea080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe508c0640 1 --2- 172.21.15.1:0/1105876999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe4c070c80 0x7fbe4c1ea080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe508c0640 1 -- 172.21.15.1:0/1105876999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4c154af0 con 0x7fbe4c0c32f0 2024-03-06T07:25:32.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe508c0640 1 --2- 172.21.15.1:0/1105876999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe4c0c32f0 0x7fbe4c0aae80 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fbe4000c500 tx=0x7fbe4000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe3dffb640 1 -- 172.21.15.1:0/1105876999 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe40018070 con 0x7fbe4c0c32f0 2024-03-06T07:25:32.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe3dffb640 1 -- 172.21.15.1:0/1105876999 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe40007580 con 0x7fbe4c0c32f0 2024-03-06T07:25:32.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe53325640 1 -- 172.21.15.1:0/1105876999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe4c154dd0 con 0x7fbe4c0c32f0 2024-03-06T07:25:32.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.914+0000 7fbe3dffb640 1 -- 172.21.15.1:0/1105876999 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe40004a20 con 0x7fbe4c0c32f0 2024-03-06T07:25:32.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.915+0000 7fbe53325640 1 -- 172.21.15.1:0/1105876999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4c155390 con 0x7fbe4c0c32f0 2024-03-06T07:25:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.916+0000 7fbe53325640 1 -- 172.21.15.1:0/1105876999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe4c0686e0 con 0x7fbe4c0c32f0 2024-03-06T07:25:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.916+0000 7fbe3dffb640 1 -- 172.21.15.1:0/1105876999 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe4001b070 con 0x7fbe4c0c32f0 2024-03-06T07:25:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.916+0000 7fbe3dffb640 1 --2- 172.21.15.1:0/1105876999 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe3006b4d0 0x7fbe3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.916+0000 7fbe3dffb640 1 -- 172.21.15.1:0/1105876999 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbe40089570 con 0x7fbe4c0c32f0 2024-03-06T07:25:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.917+0000 7fbe510c1640 1 --2- 172.21.15.1:0/1105876999 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe3006b4d0 0x7fbe3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.917+0000 7fbe510c1640 1 --2- 172.21.15.1:0/1105876999 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe3006b4d0 0x7fbe3006d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fbe48036000 tx=0x7fbe4803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:32.925+0000 7fbe3dffb640 1 -- 172.21.15.1:0/1105876999 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe40057ca0 con 0x7fbe4c0c32f0 2024-03-06T07:25:33.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.107+0000 7fbe53325640 1 -- 172.21.15.1:0/1105876999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe4c14fd40 con 0x7fbe4c0c32f0 2024-03-06T07:25:33.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.108+0000 7fbe3dffb640 1 -- 172.21.15.1:0/1105876999 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+16772 (secure 0 0 0) 0x7fbe4005bd40 con 0x7fbe4c0c32f0 2024-03-06T07:25:33.110 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:33.110 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":135,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:25:31.429480+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:28.046435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:28.753643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.109+0000 7fbe53325640 1 -- 172.21.15.1:0/1105876999 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe3006b4d0 msgr2=0x7fbe3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.109+0000 7fbe53325640 1 --2- 172.21.15.1:0/1105876999 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe3006b4d0 0x7fbe3006d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fbe48036000 tx=0x7fbe4803c040 comp rx=0 tx=0).stop 2024-03-06T07:25:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.110+0000 7fbe53325640 1 -- 172.21.15.1:0/1105876999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe4c0c32f0 msgr2=0x7fbe4c0aae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.110+0000 7fbe53325640 1 --2- 172.21.15.1:0/1105876999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe4c0c32f0 0x7fbe4c0aae80 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fbe4000c500 tx=0x7fbe4000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.110+0000 7fbe53325640 1 -- 172.21.15.1:0/1105876999 shutdown_connections 2024-03-06T07:25:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.110+0000 7fbe53325640 1 --2- 172.21.15.1:0/1105876999 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbe3006b4d0 0x7fbe3006d990 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.110+0000 7fbe53325640 1 --2- 172.21.15.1:0/1105876999 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbe4c150520 0x7fbe4c155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.110+0000 7fbe53325640 1 --2- 172.21.15.1:0/1105876999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe4c0c32f0 0x7fbe4c0aae80 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.110+0000 7fbe53325640 1 --2- 172.21.15.1:0/1105876999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe4c070c80 0x7fbe4c1ea080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.110+0000 7fbe53325640 1 -- 172.21.15.1:0/1105876999 >> 172.21.15.1:0/1105876999 conn(0x7fbe4c0c88f0 msgr2=0x7fbe4c14b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.110+0000 7fbe53325640 1 -- 172.21.15.1:0/1105876999 shutdown_connections 2024-03-06T07:25:33.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.110+0000 7fbe53325640 1 -- 172.21.15.1:0/1105876999 wait complete. 2024-03-06T07:25:33.122 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:25:33.123 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:33.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.238+0000 7f899b15e640 1 Processor -- start 2024-03-06T07:25:33.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.238+0000 7f899b15e640 1 -- start start 2024-03-06T07:25:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.239+0000 7f899b15e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8994150520 0x7f89941543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.239+0000 7f899b15e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8994150900 con 0x7f89940c32f0 2024-03-06T07:25:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.239+0000 7f899b15e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8994071790 con 0x7f8994070c80 2024-03-06T07:25:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.239+0000 7f899b15e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8994153ed0 con 0x7f8994150520 2024-03-06T07:25:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.239+0000 7f8993fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f89940c32f0 0x7f89941537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33002/0 (socket says 172.21.15.1:33002) 2024-03-06T07:25:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.239+0000 7f8993fff640 1 -- 172.21.15.1:0/2572686897 learned_addr learned my addr 172.21.15.1:0/2572686897 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.239+0000 7f89937fe640 1 -- 172.21.15.1:0/2572686897 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039893405 0 0) 0x7f8994150900 con 0x7f89940c32f0 2024-03-06T07:25:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.239+0000 7f89937fe640 1 -- 172.21.15.1:0/2572686897 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8984003520 con 0x7f89940c32f0 2024-03-06T07:25:33.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.239+0000 7f8998efa640 1 --2- 172.21.15.1:0/2572686897 >> v2:172.21.15.98:3300/0 conn(0x7f8994150520 0x7f89941543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f89937fe640 1 -- 172.21.15.1:0/2572686897 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4254907283 0 0) 0x7f8984003520 con 0x7f89940c32f0 2024-03-06T07:25:33.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f89937fe640 1 -- 172.21.15.1:0/2572686897 >> v2:172.21.15.98:3300/0 conn(0x7f8994150520 msgr2=0x7f89941543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:33.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f89937fe640 1 --2- 172.21.15.1:0/2572686897 >> v2:172.21.15.98:3300/0 conn(0x7f8994150520 0x7f89941543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f89937fe640 1 -- 172.21.15.1:0/2572686897 >> v1:172.21.15.98:6789/0 conn(0x7f8994070c80 legacy=0x7f899415d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f89937fe640 1 -- 172.21.15.1:0/2572686897 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8994157050 con 0x7f89940c32f0 2024-03-06T07:25:33.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f89937fe640 1 -- 172.21.15.1:0/2572686897 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f8980001670 con 0x7f89940c32f0 2024-03-06T07:25:33.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f89937fe640 1 -- 172.21.15.1:0/2572686897 >> v1:172.21.15.1:6789/0 conn(0x7f89940c32f0 legacy=0x7f89941537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f89937fe640 1 --2- 172.21.15.1:0/2572686897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89840039b0 0x7f8984003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f89937fe640 1 --2- 172.21.15.1:0/2572686897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89840043b0 0x7f8984004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f89937fe640 1 --2- 172.21.15.1:0/2572686897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8984005080 0x7f898400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f89937fe640 1 -- 172.21.15.1:0/2572686897 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8994153ed0 con 0x7f89840043b0 2024-03-06T07:25:33.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f89937fe640 1 -- 172.21.15.1:0/2572686897 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898400bfe0 con 0x7f8984005080 2024-03-06T07:25:33.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f89937fe640 1 -- 172.21.15.1:0/2572686897 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898400c1e0 con 0x7f89840039b0 2024-03-06T07:25:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f8993fff640 1 --2- 172.21.15.1:0/2572686897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89840043b0 0x7f8984004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f8998efa640 1 --2- 172.21.15.1:0/2572686897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89840039b0 0x7f8984003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f89996fb640 1 --2- 172.21.15.1:0/2572686897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8984005080 0x7f898400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f8993fff640 1 -- 172.21.15.1:0/2572686897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89840039b0 msgr2=0x7f8984003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f8993fff640 1 --2- 172.21.15.1:0/2572686897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89840039b0 0x7f8984003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f8993fff640 1 -- 172.21.15.1:0/2572686897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8984005080 msgr2=0x7f898400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f8993fff640 1 --2- 172.21.15.1:0/2572686897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8984005080 0x7f898400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f8993fff640 1 -- 172.21.15.1:0/2572686897 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8984003520 con 0x7f89840043b0 2024-03-06T07:25:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f8998efa640 1 --2- 172.21.15.1:0/2572686897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89840039b0 0x7f8984003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f89996fb640 1 --2- 172.21.15.1:0/2572686897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8984005080 0x7f898400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.240+0000 7f89996fb640 1 -- 172.21.15.1:0/2572686897 reap_dead start 2024-03-06T07:25:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.241+0000 7f8993fff640 1 --2- 172.21.15.1:0/2572686897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89840043b0 0x7f8984004880 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f898000c240 tx=0x7f8984005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882162996f1d640d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.241+0000 7f89937fe640 1 -- 172.21.15.1:0/2572686897 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8980007d30 con 0x7f89840043b0 2024-03-06T07:25:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.241+0000 7f89937fe640 1 -- 172.21.15.1:0/2572686897 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8980030070 con 0x7f89840043b0 2024-03-06T07:25:33.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.241+0000 7f89937fe640 1 -- 172.21.15.1:0/2572686897 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898002cd80 con 0x7f89840043b0 2024-03-06T07:25:33.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.242+0000 7f899b15e640 1 -- 172.21.15.1:0/2572686897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89840043b0 msgr2=0x7f8984004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.242+0000 7f899b15e640 1 --2- 172.21.15.1:0/2572686897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89840043b0 0x7f8984004880 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f898000c240 tx=0x7f8984005830 comp rx=0 tx=0).stop 2024-03-06T07:25:33.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.242+0000 7f899b15e640 1 -- 172.21.15.1:0/2572686897 shutdown_connections 2024-03-06T07:25:33.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.242+0000 7f899b15e640 1 --2- 172.21.15.1:0/2572686897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89840043b0 0x7f8984004880 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.242+0000 7f899b15e640 1 -- 172.21.15.1:0/2572686897 >> 172.21.15.1:0/2572686897 conn(0x7f89940c88f0 msgr2=0x7f89940c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.242+0000 7f899b15e640 1 -- 172.21.15.1:0/2572686897 shutdown_connections 2024-03-06T07:25:33.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.242+0000 7f899b15e640 1 -- 172.21.15.1:0/2572686897 wait complete. 2024-03-06T07:25:33.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.243+0000 7f899b15e640 1 Processor -- start 2024-03-06T07:25:33.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.243+0000 7f899b15e640 1 -- start start 2024-03-06T07:25:33.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f899b15e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89840043b0 0x7f8994143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f899b15e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89940ab730 0x7f89940c4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f899b15e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89940c45a0 0x7f89941ec1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f899b15e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8994157330 con 0x7f89840043b0 2024-03-06T07:25:33.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f899b15e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89941532a0 con 0x7f89940c45a0 2024-03-06T07:25:33.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f899b15e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89940c7ec0 con 0x7f89940ab730 2024-03-06T07:25:33.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f8998efa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89840043b0 0x7f8994143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f8998efa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89840043b0 0x7f8994143570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53314/0 (socket says 172.21.15.1:53314) 2024-03-06T07:25:33.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f8998efa640 1 -- 172.21.15.1:0/961663142 learned_addr learned my addr 172.21.15.1:0/961663142 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:33.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f8993fff640 1 --2- 172.21.15.1:0/961663142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89940ab730 0x7f89940c4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f89996fb640 1 --2- 172.21.15.1:0/961663142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89940c45a0 0x7f89941ec1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f8998efa640 1 -- 172.21.15.1:0/961663142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89940ab730 msgr2=0x7f89940c4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f8998efa640 1 --2- 172.21.15.1:0/961663142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89940ab730 0x7f89940c4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f8998efa640 1 -- 172.21.15.1:0/961663142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89940c45a0 msgr2=0x7f89941ec1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f8998efa640 1 --2- 172.21.15.1:0/961663142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89940c45a0 0x7f89941ec1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f8998efa640 1 -- 172.21.15.1:0/961663142 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89940abed0 con 0x7f89840043b0 2024-03-06T07:25:33.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f8998efa640 1 --2- 172.21.15.1:0/961663142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89840043b0 0x7f8994143570 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f89880114e0 tx=0x7f89880118e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f89917fa640 1 -- 172.21.15.1:0/961663142 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8988019070 con 0x7f89840043b0 2024-03-06T07:25:33.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f89917fa640 1 -- 172.21.15.1:0/961663142 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8988015870 con 0x7f89840043b0 2024-03-06T07:25:33.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f899b15e640 1 -- 172.21.15.1:0/961663142 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89940c4e60 con 0x7f89840043b0 2024-03-06T07:25:33.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.244+0000 7f89917fa640 1 -- 172.21.15.1:0/961663142 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8988015c40 con 0x7f89840043b0 2024-03-06T07:25:33.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.245+0000 7f899b15e640 1 -- 172.21.15.1:0/961663142 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8994143ab0 con 0x7f89840043b0 2024-03-06T07:25:33.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.246+0000 7f899b15e640 1 -- 172.21.15.1:0/961663142 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f899406e900 con 0x7f89840043b0 2024-03-06T07:25:33.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.246+0000 7f89917fa640 1 -- 172.21.15.1:0/961663142 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f898801c070 con 0x7f89840043b0 2024-03-06T07:25:33.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.247+0000 7f89917fa640 1 --2- 172.21.15.1:0/961663142 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f897806b4d0 0x7f897806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.247+0000 7f89917fa640 1 -- 172.21.15.1:0/961663142 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f898808a820 con 0x7f89840043b0 2024-03-06T07:25:33.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.247+0000 7f8993fff640 1 --2- 172.21.15.1:0/961663142 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f897806b4d0 0x7f897806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.248+0000 7f8993fff640 1 --2- 172.21.15.1:0/961663142 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f897806b4d0 0x7f897806d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8980034000 tx=0x7f8980009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.251+0000 7f89917fa640 1 -- 172.21.15.1:0/961663142 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f899406e900 con 0x7f89840043b0 2024-03-06T07:25:33.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.428+0000 7f899b15e640 1 -- 172.21.15.1:0/961663142 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f89941564e0 con 0x7f897806b4d0 2024-03-06T07:25:33.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.434+0000 7f89917fa640 1 -- 172.21.15.1:0/961663142 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247165 (secure 0 0 0) 0x7f89941564e0 con 0x7f897806b4d0 2024-03-06T07:25:33.438 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:25:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.437+0000 7f899b15e640 1 -- 172.21.15.1:0/961663142 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f897806b4d0 msgr2=0x7f897806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.437+0000 7f899b15e640 1 --2- 172.21.15.1:0/961663142 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f897806b4d0 0x7f897806d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8980034000 tx=0x7f8980009030 comp rx=0 tx=0).stop 2024-03-06T07:25:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.437+0000 7f899b15e640 1 -- 172.21.15.1:0/961663142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89840043b0 msgr2=0x7f8994143570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.437+0000 7f899b15e640 1 --2- 172.21.15.1:0/961663142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89840043b0 0x7f8994143570 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f89880114e0 tx=0x7f89880118e0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.437+0000 7f899b15e640 1 -- 172.21.15.1:0/961663142 shutdown_connections 2024-03-06T07:25:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.437+0000 7f899b15e640 1 --2- 172.21.15.1:0/961663142 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f897806b4d0 0x7f897806d990 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.437+0000 7f899b15e640 1 --2- 172.21.15.1:0/961663142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89940c45a0 0x7f89941ec1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.437+0000 7f899b15e640 1 --2- 172.21.15.1:0/961663142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89940ab730 0x7f89940c4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.437+0000 7f899b15e640 1 --2- 172.21.15.1:0/961663142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89840043b0 0x7f8994143570 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.437+0000 7f899b15e640 1 -- 172.21.15.1:0/961663142 >> 172.21.15.1:0/961663142 conn(0x7f89940c88f0 msgr2=0x7f899414f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.437+0000 7f899b15e640 1 -- 172.21.15.1:0/961663142 shutdown_connections 2024-03-06T07:25:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.437+0000 7f899b15e640 1 -- 172.21.15.1:0/961663142 wait complete. 2024-03-06T07:25:33.450 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-03-06T07:25:32.761549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262588,"kb_used_data":13112,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712132,"statfs":{"total":773094113280,"available":772825223168,"internally_reserved":0,"allocated":13426688,"data_stored":7576910,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.340873"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429803+0000","last_change":"2024-03-06T07:25:29.429803+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429803+0000","last_undegraded":"2024-03-06T07:25:29.429803+0000","last_fullsized":"2024-03-06T07:25:29.429803+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429415+0000","last_change":"2024-03-06T07:25:29.429415+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429415+0000","last_undegraded":"2024-03-06T07:25:29.429415+0000","last_fullsized":"2024-03-06T07:25:29.429415+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.432950+0000","last_change":"2024-03-06T07:25:29.432950+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.432950+0000","last_undegraded":"2024-03-06T07:25:29.432950+0000","last_fullsized":"2024-03-06T07:25:29.432950+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.434735+0000","last_change":"2024-03-06T07:25:29.434735+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.434735+0000","last_undegraded":"2024-03-06T07:25:29.434735+0000","last_fullsized":"2024-03-06T07:25:29.434735+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423617+0000","last_change":"2024-03-06T07:25:29.423617+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423617+0000","last_undegraded":"2024-03-06T07:25:29.423617+0000","last_fullsized":"2024-03-06T07:25:29.423617+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423430+0000","last_change":"2024-03-06T07:25:29.423430+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423430+0000","last_undegraded":"2024-03-06T07:25:29.423430+0000","last_fullsized":"2024-03-06T07:25:29.423430+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430543+0000","last_change":"2024-03-06T07:25:29.430543+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430543+0000","last_undegraded":"2024-03-06T07:25:29.430543+0000","last_fullsized":"2024-03-06T07:25:29.430543+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428243+0000","last_change":"2024-03-06T07:25:29.428243+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428243+0000","last_undegraded":"2024-03-06T07:25:29.428243+0000","last_fullsized":"2024-03-06T07:25:29.428243+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423943+0000","last_change":"2024-03-06T07:25:29.423943+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423943+0000","last_undegraded":"2024-03-06T07:25:29.423943+0000","last_fullsized":"2024-03-06T07:25:29.423943+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423699+0000","last_change":"2024-03-06T07:25:29.423699+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423699+0000","last_undegraded":"2024-03-06T07:25:29.423699+0000","last_fullsized":"2024-03-06T07:25:29.423699+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427685+0000","last_change":"2024-03-06T07:25:29.427685+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427685+0000","last_undegraded":"2024-03-06T07:25:29.427685+0000","last_fullsized":"2024-03-06T07:25:29.427685+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.433682+0000","last_change":"2024-03-06T07:25:29.433682+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.433682+0000","last_undegraded":"2024-03-06T07:25:29.433682+0000","last_fullsized":"2024-03-06T07:25:29.433682+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.424516+0000","last_change":"2024-03-06T07:25:29.424516+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.424516+0000","last_undegraded":"2024-03-06T07:25:29.424516+0000","last_fullsized":"2024-03-06T07:25:29.424516+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.433176+0000","last_change":"2024-03-06T07:25:29.433176+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.433176+0000","last_undegraded":"2024-03-06T07:25:29.433176+0000","last_fullsized":"2024-03-06T07:25:29.433176+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.432452+0000","last_change":"2024-03-06T07:25:29.432452+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.432452+0000","last_undegraded":"2024-03-06T07:25:29.432452+0000","last_fullsized":"2024-03-06T07:25:29.432452+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423245+0000","last_change":"2024-03-06T07:25:29.423245+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423245+0000","last_undegraded":"2024-03-06T07:25:29.423245+0000","last_fullsized":"2024-03-06T07:25:29.423245+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.434177+0000","last_change":"2024-03-06T07:25:29.434177+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.434177+0000","last_undegraded":"2024-03-06T07:25:29.434177+0000","last_fullsized":"2024-03-06T07:25:29.434177+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428279+0000","last_change":"2024-03-06T07:25:29.428279+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428279+0000","last_undegraded":"2024-03-06T07:25:29.428279+0000","last_fullsized":"2024-03-06T07:25:29.428279+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429606+0000","last_change":"2024-03-06T07:25:29.429606+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429606+0000","last_undegraded":"2024-03-06T07:25:29.429606+0000","last_fullsized":"2024-03-06T07:25:29.429606+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427157+0000","last_change":"2024-03-06T07:25:29.427157+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427157+0000","last_undegraded":"2024-03-06T07:25:29.427157+0000","last_fullsized":"2024-03-06T07:25:29.427157+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.424094+0000","last_change":"2024-03-06T07:25:29.424094+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.424094+0000","last_undegraded":"2024-03-06T07:25:29.424094+0000","last_fullsized":"2024-03-06T07:25:29.424094+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427504+0000","last_change":"2024-03-06T07:25:29.427504+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427504+0000","last_undegraded":"2024-03-06T07:25:29.427504+0000","last_fullsized":"2024-03-06T07:25:29.427504+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.426134+0000","last_change":"2024-03-06T07:25:29.426134+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.426134+0000","last_undegraded":"2024-03-06T07:25:29.426134+0000","last_fullsized":"2024-03-06T07:25:29.426134+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.426814+0000","last_change":"2024-03-06T07:25:29.426814+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.426814+0000","last_undegraded":"2024-03-06T07:25:29.426814+0000","last_fullsized":"2024-03-06T07:25:29.426814+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429388+0000","last_change":"2024-03-06T07:25:29.429388+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429388+0000","last_undegraded":"2024-03-06T07:25:29.429388+0000","last_fullsized":"2024-03-06T07:25:29.429388+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431459+0000","last_change":"2024-03-06T07:25:29.431459+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431459+0000","last_undegraded":"2024-03-06T07:25:29.431459+0000","last_fullsized":"2024-03-06T07:25:29.431459+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.422691+0000","last_change":"2024-03-06T07:25:29.422691+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.422691+0000","last_undegraded":"2024-03-06T07:25:29.422691+0000","last_fullsized":"2024-03-06T07:25:29.422691+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427342+0000","last_change":"2024-03-06T07:25:29.427342+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427342+0000","last_undegraded":"2024-03-06T07:25:29.427342+0000","last_fullsized":"2024-03-06T07:25:29.427342+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.426341+0000","last_change":"2024-03-06T07:25:29.426341+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.426341+0000","last_undegraded":"2024-03-06T07:25:29.426341+0000","last_fullsized":"2024-03-06T07:25:29.426341+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429494+0000","last_change":"2024-03-06T07:25:29.429494+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429494+0000","last_undegraded":"2024-03-06T07:25:29.429494+0000","last_fullsized":"2024-03-06T07:25:29.429494+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.424039+0000","last_change":"2024-03-06T07:25:29.424039+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.424039+0000","last_undegraded":"2024-03-06T07:25:29.424039+0000","last_fullsized":"2024-03-06T07:25:29.424039+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427794+0000","last_change":"2024-03-06T07:25:29.427794+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427794+0000","last_undegraded":"2024-03-06T07:25:29.427794+0000","last_fullsized":"2024-03-06T07:25:29.427794+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430468+0000","last_change":"2024-03-06T07:25:29.430468+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430468+0000","last_undegraded":"2024-03-06T07:25:29.430468+0000","last_fullsized":"2024-03-06T07:25:29.430468+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.429642+0000","last_change":"2024-03-06T07:25:29.427051+0000","last_active":"2024-03-06T07:25:29.429642+0000","last_peered":"2024-03-06T07:25:29.429642+0000","last_clean":"2024-03-06T07:25:29.429642+0000","last_became_active":"2024-03-06T07:25:29.426750+0000","last_became_peered":"2024-03-06T07:25:29.426750+0000","last_unstale":"2024-03-06T07:25:29.429642+0000","last_undegraded":"2024-03-06T07:25:29.429642+0000","last_fullsized":"2024-03-06T07:25:29.429642+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:20.119206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430796+0000","last_change":"2024-03-06T07:25:29.430796+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430796+0000","last_undegraded":"2024-03-06T07:25:29.430796+0000","last_fullsized":"2024-03-06T07:25:29.430796+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.444195+0000","last_change":"2024-03-06T07:25:29.441160+0000","last_active":"2024-03-06T07:25:29.444195+0000","last_peered":"2024-03-06T07:25:29.444195+0000","last_clean":"2024-03-06T07:25:29.444195+0000","last_became_active":"2024-03-06T07:25:29.440834+0000","last_became_peered":"2024-03-06T07:25:29.440834+0000","last_unstale":"2024-03-06T07:25:29.444195+0000","last_undegraded":"2024-03-06T07:25:29.444195+0000","last_fullsized":"2024-03-06T07:25:29.444195+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:13.644859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.424888+0000","last_change":"2024-03-06T07:25:29.424888+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.424888+0000","last_undegraded":"2024-03-06T07:25:29.424888+0000","last_fullsized":"2024-03-06T07:25:29.424888+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.446817+0000","last_change":"2024-03-06T07:25:29.443849+0000","last_active":"2024-03-06T07:25:29.446817+0000","last_peered":"2024-03-06T07:25:29.446817+0000","last_clean":"2024-03-06T07:25:29.446817+0000","last_became_active":"2024-03-06T07:25:29.443606+0000","last_became_peered":"2024-03-06T07:25:29.443606+0000","last_unstale":"2024-03-06T07:25:29.446817+0000","last_undegraded":"2024-03-06T07:25:29.446817+0000","last_fullsized":"2024-03-06T07:25:29.446817+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:04.414856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427180+0000","last_change":"2024-03-06T07:25:29.427180+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427180+0000","last_undegraded":"2024-03-06T07:25:29.427180+0000","last_fullsized":"2024-03-06T07:25:29.427180+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442922+0000","last_change":"2024-03-06T07:25:29.442089+0000","last_active":"2024-03-06T07:25:29.442922+0000","last_peered":"2024-03-06T07:25:29.442922+0000","last_clean":"2024-03-06T07:25:29.442922+0000","last_became_active":"2024-03-06T07:25:29.441679+0000","last_became_peered":"2024-03-06T07:25:29.441679+0000","last_unstale":"2024-03-06T07:25:29.442922+0000","last_undegraded":"2024-03-06T07:25:29.442922+0000","last_fullsized":"2024-03-06T07:25:29.442922+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:48.284334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.426438+0000","last_change":"2024-03-06T07:25:29.426438+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.426438+0000","last_undegraded":"2024-03-06T07:25:29.426438+0000","last_fullsized":"2024-03-06T07:25:29.426438+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.444420+0000","last_change":"2024-03-06T07:25:29.438025+0000","last_active":"2024-03-06T07:25:29.444420+0000","last_peered":"2024-03-06T07:25:29.444420+0000","last_clean":"2024-03-06T07:25:29.444420+0000","last_became_active":"2024-03-06T07:25:29.437765+0000","last_became_peered":"2024-03-06T07:25:29.437765+0000","last_unstale":"2024-03-06T07:25:29.444420+0000","last_undegraded":"2024-03-06T07:25:29.444420+0000","last_fullsized":"2024-03-06T07:25:29.444420+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:28.072325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431875+0000","last_change":"2024-03-06T07:25:29.431875+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431875+0000","last_undegraded":"2024-03-06T07:25:29.431875+0000","last_fullsized":"2024-03-06T07:25:29.431875+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.439676+0000","last_change":"2024-03-06T07:25:29.437797+0000","last_active":"2024-03-06T07:25:29.439676+0000","last_peered":"2024-03-06T07:25:29.439676+0000","last_clean":"2024-03-06T07:25:29.439676+0000","last_became_active":"2024-03-06T07:25:29.437461+0000","last_became_peered":"2024-03-06T07:25:29.437461+0000","last_unstale":"2024-03-06T07:25:29.439676+0000","last_undegraded":"2024-03-06T07:25:29.439676+0000","last_fullsized":"2024-03-06T07:25:29.439676+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:07.155504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431792+0000","last_change":"2024-03-06T07:25:29.431792+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431792+0000","last_undegraded":"2024-03-06T07:25:29.431792+0000","last_fullsized":"2024-03-06T07:25:29.431792+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442539+0000","last_change":"2024-03-06T07:25:29.437803+0000","last_active":"2024-03-06T07:25:29.442539+0000","last_peered":"2024-03-06T07:25:29.442539+0000","last_clean":"2024-03-06T07:25:29.442539+0000","last_became_active":"2024-03-06T07:25:29.437527+0000","last_became_peered":"2024-03-06T07:25:29.437527+0000","last_unstale":"2024-03-06T07:25:29.442539+0000","last_undegraded":"2024-03-06T07:25:29.442539+0000","last_fullsized":"2024-03-06T07:25:29.442539+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:40.718218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428427+0000","last_change":"2024-03-06T07:25:29.428427+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428427+0000","last_undegraded":"2024-03-06T07:25:29.428427+0000","last_fullsized":"2024-03-06T07:25:29.428427+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.434590+0000","last_change":"2024-03-06T07:25:29.433974+0000","last_active":"2024-03-06T07:25:29.434590+0000","last_peered":"2024-03-06T07:25:29.434590+0000","last_clean":"2024-03-06T07:25:29.434590+0000","last_became_active":"2024-03-06T07:25:29.433432+0000","last_became_peered":"2024-03-06T07:25:29.433432+0000","last_unstale":"2024-03-06T07:25:29.434590+0000","last_undegraded":"2024-03-06T07:25:29.434590+0000","last_fullsized":"2024-03-06T07:25:29.434590+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:37.830403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427340+0000","last_change":"2024-03-06T07:25:29.427340+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427340+0000","last_undegraded":"2024-03-06T07:25:29.427340+0000","last_fullsized":"2024-03-06T07:25:29.427340+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.439779+0000","last_change":"2024-03-06T07:25:29.435113+0000","last_active":"2024-03-06T07:25:29.439779+0000","last_peered":"2024-03-06T07:25:29.439779+0000","last_clean":"2024-03-06T07:25:29.439779+0000","last_became_active":"2024-03-06T07:25:29.434807+0000","last_became_peered":"2024-03-06T07:25:29.434807+0000","last_unstale":"2024-03-06T07:25:29.439779+0000","last_undegraded":"2024-03-06T07:25:29.439779+0000","last_fullsized":"2024-03-06T07:25:29.439779+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:57.999261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427245+0000","last_change":"2024-03-06T07:25:29.427245+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427245+0000","last_undegraded":"2024-03-06T07:25:29.427245+0000","last_fullsized":"2024-03-06T07:25:29.427245+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442775+0000","last_change":"2024-03-06T07:25:29.437148+0000","last_active":"2024-03-06T07:25:29.442775+0000","last_peered":"2024-03-06T07:25:29.442775+0000","last_clean":"2024-03-06T07:25:29.442775+0000","last_became_active":"2024-03-06T07:25:29.436865+0000","last_became_peered":"2024-03-06T07:25:29.436865+0000","last_unstale":"2024-03-06T07:25:29.442775+0000","last_undegraded":"2024-03-06T07:25:29.442775+0000","last_fullsized":"2024-03-06T07:25:29.442775+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:44.776498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427865+0000","last_change":"2024-03-06T07:25:29.427865+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427865+0000","last_undegraded":"2024-03-06T07:25:29.427865+0000","last_fullsized":"2024-03-06T07:25:29.427865+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.434630+0000","last_change":"2024-03-06T07:25:29.431284+0000","last_active":"2024-03-06T07:25:29.434630+0000","last_peered":"2024-03-06T07:25:29.434630+0000","last_clean":"2024-03-06T07:25:29.434630+0000","last_became_active":"2024-03-06T07:25:29.431000+0000","last_became_peered":"2024-03-06T07:25:29.431000+0000","last_unstale":"2024-03-06T07:25:29.434630+0000","last_undegraded":"2024-03-06T07:25:29.434630+0000","last_fullsized":"2024-03-06T07:25:29.434630+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:29.131604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427343+0000","last_change":"2024-03-06T07:25:29.427343+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427343+0000","last_undegraded":"2024-03-06T07:25:29.427343+0000","last_fullsized":"2024-03-06T07:25:29.427343+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.428420+0000","last_change":"2024-03-06T07:25:29.426864+0000","last_active":"2024-03-06T07:25:29.428420+0000","last_peered":"2024-03-06T07:25:29.428420+0000","last_clean":"2024-03-06T07:25:29.428420+0000","last_became_active":"2024-03-06T07:25:29.426563+0000","last_became_peered":"2024-03-06T07:25:29.426563+0000","last_unstale":"2024-03-06T07:25:29.428420+0000","last_undegraded":"2024-03-06T07:25:29.428420+0000","last_fullsized":"2024-03-06T07:25:29.428420+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:28.472000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.432056+0000","last_change":"2024-03-06T07:25:29.432056+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.432056+0000","last_undegraded":"2024-03-06T07:25:29.432056+0000","last_fullsized":"2024-03-06T07:25:29.432056+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.446869+0000","last_change":"2024-03-06T07:25:29.442654+0000","last_active":"2024-03-06T07:25:29.446869+0000","last_peered":"2024-03-06T07:25:29.446869+0000","last_clean":"2024-03-06T07:25:29.446869+0000","last_became_active":"2024-03-06T07:25:29.442384+0000","last_became_peered":"2024-03-06T07:25:29.442384+0000","last_unstale":"2024-03-06T07:25:29.446869+0000","last_undegraded":"2024-03-06T07:25:29.446869+0000","last_fullsized":"2024-03-06T07:25:29.446869+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:26:04.515018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.422909+0000","last_change":"2024-03-06T07:25:29.422909+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.422909+0000","last_undegraded":"2024-03-06T07:25:29.422909+0000","last_fullsized":"2024-03-06T07:25:29.422909+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.444356+0000","last_change":"2024-03-06T07:25:29.437290+0000","last_active":"2024-03-06T07:25:29.444356+0000","last_peered":"2024-03-06T07:25:29.444356+0000","last_clean":"2024-03-06T07:25:29.444356+0000","last_became_active":"2024-03-06T07:25:29.437012+0000","last_became_peered":"2024-03-06T07:25:29.437012+0000","last_unstale":"2024-03-06T07:25:29.444356+0000","last_undegraded":"2024-03-06T07:25:29.444356+0000","last_fullsized":"2024-03-06T07:25:29.444356+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:44.052940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430432+0000","last_change":"2024-03-06T07:25:29.430432+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430432+0000","last_undegraded":"2024-03-06T07:25:29.430432+0000","last_fullsized":"2024-03-06T07:25:29.430432+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.449085+0000","last_change":"2024-03-06T07:25:29.444312+0000","last_active":"2024-03-06T07:25:29.449085+0000","last_peered":"2024-03-06T07:25:29.449085+0000","last_clean":"2024-03-06T07:25:29.449085+0000","last_became_active":"2024-03-06T07:25:29.444031+0000","last_became_peered":"2024-03-06T07:25:29.444031+0000","last_unstale":"2024-03-06T07:25:29.449085+0000","last_undegraded":"2024-03-06T07:25:29.449085+0000","last_fullsized":"2024-03-06T07:25:29.449085+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:43.317473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428129+0000","last_change":"2024-03-06T07:25:29.428129+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428129+0000","last_undegraded":"2024-03-06T07:25:29.428129+0000","last_fullsized":"2024-03-06T07:25:29.428129+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.443032+0000","last_change":"2024-03-06T07:25:29.442625+0000","last_active":"2024-03-06T07:25:29.443032+0000","last_peered":"2024-03-06T07:25:29.443032+0000","last_clean":"2024-03-06T07:25:29.443032+0000","last_became_active":"2024-03-06T07:25:29.442320+0000","last_became_peered":"2024-03-06T07:25:29.442320+0000","last_unstale":"2024-03-06T07:25:29.443032+0000","last_undegraded":"2024-03-06T07:25:29.443032+0000","last_fullsized":"2024-03-06T07:25:29.443032+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:36.851563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428953+0000","last_change":"2024-03-06T07:25:29.428953+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428953+0000","last_undegraded":"2024-03-06T07:25:29.428953+0000","last_fullsized":"2024-03-06T07:25:29.428953+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.441944+0000","last_change":"2024-03-06T07:25:29.437681+0000","last_active":"2024-03-06T07:25:29.441944+0000","last_peered":"2024-03-06T07:25:29.441944+0000","last_clean":"2024-03-06T07:25:29.441944+0000","last_became_active":"2024-03-06T07:25:29.437415+0000","last_became_peered":"2024-03-06T07:25:29.437415+0000","last_unstale":"2024-03-06T07:25:29.441944+0000","last_undegraded":"2024-03-06T07:25:29.441944+0000","last_fullsized":"2024-03-06T07:25:29.441944+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:59.804700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431074+0000","last_change":"2024-03-06T07:25:29.431074+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431074+0000","last_undegraded":"2024-03-06T07:25:29.431074+0000","last_fullsized":"2024-03-06T07:25:29.431074+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442850+0000","last_change":"2024-03-06T07:25:29.442155+0000","last_active":"2024-03-06T07:25:29.442850+0000","last_peered":"2024-03-06T07:25:29.442850+0000","last_clean":"2024-03-06T07:25:29.442850+0000","last_became_active":"2024-03-06T07:25:29.441793+0000","last_became_peered":"2024-03-06T07:25:29.441793+0000","last_unstale":"2024-03-06T07:25:29.442850+0000","last_undegraded":"2024-03-06T07:25:29.442850+0000","last_fullsized":"2024-03-06T07:25:29.442850+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:04.755959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428958+0000","last_change":"2024-03-06T07:25:29.428958+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428958+0000","last_undegraded":"2024-03-06T07:25:29.428958+0000","last_fullsized":"2024-03-06T07:25:29.428958+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.446861+0000","last_change":"2024-03-06T07:25:29.442853+0000","last_active":"2024-03-06T07:25:29.446861+0000","last_peered":"2024-03-06T07:25:29.446861+0000","last_clean":"2024-03-06T07:25:29.446861+0000","last_became_active":"2024-03-06T07:25:29.442618+0000","last_became_peered":"2024-03-06T07:25:29.442618+0000","last_unstale":"2024-03-06T07:25:29.446861+0000","last_undegraded":"2024-03-06T07:25:29.446861+0000","last_fullsized":"2024-03-06T07:25:29.446861+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:42.666506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427796+0000","last_change":"2024-03-06T07:25:29.427796+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427796+0000","last_undegraded":"2024-03-06T07:25:29.427796+0000","last_fullsized":"2024-03-06T07:25:29.427796+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.434976+0000","last_change":"2024-03-06T07:25:29.433350+0000","last_active":"2024-03-06T07:25:29.434976+0000","last_peered":"2024-03-06T07:25:29.434976+0000","last_clean":"2024-03-06T07:25:29.434976+0000","last_became_active":"2024-03-06T07:25:29.433064+0000","last_became_peered":"2024-03-06T07:25:29.433064+0000","last_unstale":"2024-03-06T07:25:29.434976+0000","last_undegraded":"2024-03-06T07:25:29.434976+0000","last_fullsized":"2024-03-06T07:25:29.434976+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:09.022615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429265+0000","last_change":"2024-03-06T07:25:29.429265+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429265+0000","last_undegraded":"2024-03-06T07:25:29.429265+0000","last_fullsized":"2024-03-06T07:25:29.429265+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.444452+0000","last_change":"2024-03-06T07:25:29.438191+0000","last_active":"2024-03-06T07:25:29.444452+0000","last_peered":"2024-03-06T07:25:29.444452+0000","last_clean":"2024-03-06T07:25:29.444452+0000","last_became_active":"2024-03-06T07:25:29.437589+0000","last_became_peered":"2024-03-06T07:25:29.437589+0000","last_unstale":"2024-03-06T07:25:29.444452+0000","last_undegraded":"2024-03-06T07:25:29.444452+0000","last_fullsized":"2024-03-06T07:25:29.444452+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:34.964198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428529+0000","last_change":"2024-03-06T07:25:29.428529+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428529+0000","last_undegraded":"2024-03-06T07:25:29.428529+0000","last_fullsized":"2024-03-06T07:25:29.428529+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.434947+0000","last_change":"2024-03-06T07:25:29.432629+0000","last_active":"2024-03-06T07:25:29.434947+0000","last_peered":"2024-03-06T07:25:29.434947+0000","last_clean":"2024-03-06T07:25:29.434947+0000","last_became_active":"2024-03-06T07:25:29.432348+0000","last_became_peered":"2024-03-06T07:25:29.432348+0000","last_unstale":"2024-03-06T07:25:29.434947+0000","last_undegraded":"2024-03-06T07:25:29.434947+0000","last_fullsized":"2024-03-06T07:25:29.434947+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:43.071021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430039+0000","last_change":"2024-03-06T07:25:29.430039+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430039+0000","last_undegraded":"2024-03-06T07:25:29.430039+0000","last_fullsized":"2024-03-06T07:25:29.430039+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.444491+0000","last_change":"2024-03-06T07:25:29.439680+0000","last_active":"2024-03-06T07:25:29.444491+0000","last_peered":"2024-03-06T07:25:29.444491+0000","last_clean":"2024-03-06T07:25:29.444491+0000","last_became_active":"2024-03-06T07:25:29.439401+0000","last_became_peered":"2024-03-06T07:25:29.439401+0000","last_unstale":"2024-03-06T07:25:29.444491+0000","last_undegraded":"2024-03-06T07:25:29.444491+0000","last_fullsized":"2024-03-06T07:25:29.444491+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:59.559815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.424099+0000","last_change":"2024-03-06T07:25:29.424099+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.424099+0000","last_undegraded":"2024-03-06T07:25:29.424099+0000","last_fullsized":"2024-03-06T07:25:29.424099+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.448969+0000","last_change":"2024-03-06T07:25:29.445124+0000","last_active":"2024-03-06T07:25:29.448969+0000","last_peered":"2024-03-06T07:25:29.448969+0000","last_clean":"2024-03-06T07:25:29.448969+0000","last_became_active":"2024-03-06T07:25:29.444870+0000","last_became_peered":"2024-03-06T07:25:29.444870+0000","last_unstale":"2024-03-06T07:25:29.448969+0000","last_undegraded":"2024-03-06T07:25:29.448969+0000","last_fullsized":"2024-03-06T07:25:29.448969+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:36.677607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.433944+0000","last_change":"2024-03-06T07:25:29.433944+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.433944+0000","last_undegraded":"2024-03-06T07:25:29.433944+0000","last_fullsized":"2024-03-06T07:25:29.433944+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.449041+0000","last_change":"2024-03-06T07:25:29.442634+0000","last_active":"2024-03-06T07:25:29.449041+0000","last_peered":"2024-03-06T07:25:29.449041+0000","last_clean":"2024-03-06T07:25:29.449041+0000","last_became_active":"2024-03-06T07:25:29.442362+0000","last_became_peered":"2024-03-06T07:25:29.442362+0000","last_unstale":"2024-03-06T07:25:29.449041+0000","last_undegraded":"2024-03-06T07:25:29.449041+0000","last_fullsized":"2024-03-06T07:25:29.449041+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:09.549229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430682+0000","last_change":"2024-03-06T07:25:29.430682+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430682+0000","last_undegraded":"2024-03-06T07:25:29.430682+0000","last_fullsized":"2024-03-06T07:25:29.430682+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.441524+0000","last_change":"2024-03-06T07:25:29.440980+0000","last_active":"2024-03-06T07:25:29.441524+0000","last_peered":"2024-03-06T07:25:29.441524+0000","last_clean":"2024-03-06T07:25:29.441524+0000","last_became_active":"2024-03-06T07:25:29.440645+0000","last_became_peered":"2024-03-06T07:25:29.440645+0000","last_unstale":"2024-03-06T07:25:29.441524+0000","last_undegraded":"2024-03-06T07:25:29.441524+0000","last_fullsized":"2024-03-06T07:25:29.441524+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:30.511914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423291+0000","last_change":"2024-03-06T07:25:29.423291+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423291+0000","last_undegraded":"2024-03-06T07:25:29.423291+0000","last_fullsized":"2024-03-06T07:25:29.423291+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.439976+0000","last_change":"2024-03-06T07:25:29.439438+0000","last_active":"2024-03-06T07:25:29.439976+0000","last_peered":"2024-03-06T07:25:29.439976+0000","last_clean":"2024-03-06T07:25:29.439976+0000","last_became_active":"2024-03-06T07:25:29.439129+0000","last_became_peered":"2024-03-06T07:25:29.439129+0000","last_unstale":"2024-03-06T07:25:29.439976+0000","last_undegraded":"2024-03-06T07:25:29.439976+0000","last_fullsized":"2024-03-06T07:25:29.439976+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:37.162898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.425093+0000","last_change":"2024-03-06T07:25:29.425093+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.425093+0000","last_undegraded":"2024-03-06T07:25:29.425093+0000","last_fullsized":"2024-03-06T07:25:29.425093+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442801+0000","last_change":"2024-03-06T07:25:29.439263+0000","last_active":"2024-03-06T07:25:29.442801+0000","last_peered":"2024-03-06T07:25:29.442801+0000","last_clean":"2024-03-06T07:25:29.442801+0000","last_became_active":"2024-03-06T07:25:29.438961+0000","last_became_peered":"2024-03-06T07:25:29.438961+0000","last_unstale":"2024-03-06T07:25:29.442801+0000","last_undegraded":"2024-03-06T07:25:29.442801+0000","last_fullsized":"2024-03-06T07:25:29.442801+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:34.084278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431354+0000","last_change":"2024-03-06T07:25:29.431354+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431354+0000","last_undegraded":"2024-03-06T07:25:29.431354+0000","last_fullsized":"2024-03-06T07:25:29.431354+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442824+0000","last_change":"2024-03-06T07:25:29.442215+0000","last_active":"2024-03-06T07:25:29.442824+0000","last_peered":"2024-03-06T07:25:29.442824+0000","last_clean":"2024-03-06T07:25:29.442824+0000","last_became_active":"2024-03-06T07:25:29.441739+0000","last_became_peered":"2024-03-06T07:25:29.441739+0000","last_unstale":"2024-03-06T07:25:29.442824+0000","last_undegraded":"2024-03-06T07:25:29.442824+0000","last_fullsized":"2024-03-06T07:25:29.442824+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:21.032029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431388+0000","last_change":"2024-03-06T07:25:29.431388+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431388+0000","last_undegraded":"2024-03-06T07:25:29.431388+0000","last_fullsized":"2024-03-06T07:25:29.431388+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442516+0000","last_change":"2024-03-06T07:25:29.437548+0000","last_active":"2024-03-06T07:25:29.442516+0000","last_peered":"2024-03-06T07:25:29.442516+0000","last_clean":"2024-03-06T07:25:29.442516+0000","last_became_active":"2024-03-06T07:25:29.437281+0000","last_became_peered":"2024-03-06T07:25:29.437281+0000","last_unstale":"2024-03-06T07:25:29.442516+0000","last_undegraded":"2024-03-06T07:25:29.442516+0000","last_fullsized":"2024-03-06T07:25:29.442516+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:18.466217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.426909+0000","last_change":"2024-03-06T07:25:29.426909+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.426909+0000","last_undegraded":"2024-03-06T07:25:29.426909+0000","last_fullsized":"2024-03-06T07:25:29.426909+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442173+0000","last_change":"2024-03-06T07:25:29.437129+0000","last_active":"2024-03-06T07:25:29.442173+0000","last_peered":"2024-03-06T07:25:29.442173+0000","last_clean":"2024-03-06T07:25:29.442173+0000","last_became_active":"2024-03-06T07:25:29.436825+0000","last_became_peered":"2024-03-06T07:25:29.436825+0000","last_unstale":"2024-03-06T07:25:29.442173+0000","last_undegraded":"2024-03-06T07:25:29.442173+0000","last_fullsized":"2024-03-06T07:25:29.442173+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:28.307200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.425972+0000","last_change":"2024-03-06T07:25:29.425972+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.425972+0000","last_undegraded":"2024-03-06T07:25:29.425972+0000","last_fullsized":"2024-03-06T07:25:29.425972+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.441092+0000","last_change":"2024-03-06T07:25:29.440579+0000","last_active":"2024-03-06T07:25:29.441092+0000","last_peered":"2024-03-06T07:25:29.441092+0000","last_clean":"2024-03-06T07:25:29.441092+0000","last_became_active":"2024-03-06T07:25:29.440247+0000","last_became_peered":"2024-03-06T07:25:29.440247+0000","last_unstale":"2024-03-06T07:25:29.441092+0000","last_undegraded":"2024-03-06T07:25:29.441092+0000","last_fullsized":"2024-03-06T07:25:29.441092+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:13.223054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":385,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.428367+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:25:29.428367+0000","last_peered":"2024-03-06T07:25:29.428367+0000","last_clean":"2024-03-06T07:25:29.428367+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:25:29.428367+0000","last_undegraded":"2024-03-06T07:25:29.428367+0000","last_fullsized":"2024-03-06T07:25:29.428367+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32868,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338972,"statfs":{"total":96636764160,"available":96603107328,"internally_reserved":0,"allocated":1724416,"data_stored":1388223,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32812,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339028,"statfs":{"total":96636764160,"available":96603164672,"internally_reserved":0,"allocated":1671168,"data_stored":804522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574913,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32808,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339032,"statfs":{"total":96636764160,"available":96603168768,"internally_reserved":0,"allocated":1658880,"data_stored":797855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32808,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339032,"statfs":{"total":96636764160,"available":96603168768,"internally_reserved":0,"allocated":1658880,"data_stored":797855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32808,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339032,"statfs":{"total":96636764160,"available":96603168768,"internally_reserved":0,"allocated":1658880,"data_stored":797855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574913,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32800,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339040,"statfs":{"total":96636764160,"available":96603176960,"internally_reserved":0,"allocated":1658880,"data_stored":797855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32812,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339028,"statfs":{"total":96636764160,"available":96603164672,"internally_reserved":0,"allocated":1671168,"data_stored":804522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574914,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32872,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338968,"statfs":{"total":96636764160,"available":96603103232,"internally_reserved":0,"allocated":1724416,"data_stored":1388223,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:33.454 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:33.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.589+0000 7f32390c9640 1 Processor -- start 2024-03-06T07:25:33.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.589+0000 7f32390c9640 1 -- start start 2024-03-06T07:25:33.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.590+0000 7f32390c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3234072bd0 0x7f32340c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.590+0000 7f32390c9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3234152b70 0x7f32340c3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.590+0000 7f32390c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3234151dd0 con 0x7f32340c2b30 2024-03-06T07:25:33.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.590+0000 7f32390c9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32341526e0 con 0x7f3234152b70 2024-03-06T07:25:33.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.590+0000 7f32390c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3234156190 con 0x7f3234072bd0 2024-03-06T07:25:33.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.590+0000 7f323259c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3234072bd0 0x7f32340c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.590+0000 7f323359e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3234152b70 0x7f32340c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.590+0000 7f323259c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3234072bd0 0x7f32340c32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:47886/0 (socket says 172.21.15.1:47886) 2024-03-06T07:25:33.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.590+0000 7f323259c640 1 -- 172.21.15.1:0/3761813889 learned_addr learned my addr 172.21.15.1:0/3761813889 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:33.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.590+0000 7f323359e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3234152b70 0x7f32340c3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:43788/0 (socket says 172.21.15.1:43788) 2024-03-06T07:25:33.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.590+0000 7f3232d9d640 1 --1- 172.21.15.1:0/3761813889 >> v1:172.21.15.98:6789/0 conn(0x7f32340c2b30 0x7f32340724c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:48076/0 (socket says 172.21.15.1:48076) 2024-03-06T07:25:33.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.591+0000 7f323259c640 1 -- 172.21.15.1:0/3761813889 >> v2:172.21.15.98:3300/0 conn(0x7f3234152b70 msgr2=0x7f32340c3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:33.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.591+0000 7f323259c640 1 --2- 172.21.15.1:0/3761813889 >> v2:172.21.15.98:3300/0 conn(0x7f3234152b70 0x7f32340c3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.591+0000 7f323259c640 1 -- 172.21.15.1:0/3761813889 >> v1:172.21.15.98:6789/0 conn(0x7f32340c2b30 legacy=0x7f32340724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.591+0000 7f323259c640 1 -- 172.21.15.1:0/3761813889 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3234152090 con 0x7f3234072bd0 2024-03-06T07:25:33.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.591+0000 7f323259c640 1 --2- 172.21.15.1:0/3761813889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3234072bd0 0x7f32340c32f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3224009870 tx=0x7f322402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6ce7c1e01360360 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.592+0000 7f3231d9b640 1 -- 172.21.15.1:0/3761813889 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3224034070 con 0x7f3234072bd0 2024-03-06T07:25:33.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.592+0000 7f3231d9b640 1 -- 172.21.15.1:0/3761813889 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3224030840 con 0x7f3234072bd0 2024-03-06T07:25:33.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.592+0000 7f3231d9b640 1 -- 172.21.15.1:0/3761813889 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3224030c10 con 0x7f3234072bd0 2024-03-06T07:25:33.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.592+0000 7f32390c9640 1 -- 172.21.15.1:0/3761813889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3234072bd0 msgr2=0x7f32340c32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.592+0000 7f32390c9640 1 --2- 172.21.15.1:0/3761813889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3234072bd0 0x7f32340c32f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3224009870 tx=0x7f322402b980 comp rx=0 tx=0).stop 2024-03-06T07:25:33.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.592+0000 7f32390c9640 1 -- 172.21.15.1:0/3761813889 shutdown_connections 2024-03-06T07:25:33.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.592+0000 7f32390c9640 1 --2- 172.21.15.1:0/3761813889 >> v2:172.21.15.98:3300/0 conn(0x7f3234152b70 0x7f32340c3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.592+0000 7f32390c9640 1 --2- 172.21.15.1:0/3761813889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3234072bd0 0x7f32340c32f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.592+0000 7f32390c9640 1 -- 172.21.15.1:0/3761813889 >> 172.21.15.1:0/3761813889 conn(0x7f32340c6860 msgr2=0x7f32340c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.592+0000 7f32390c9640 1 -- 172.21.15.1:0/3761813889 shutdown_connections 2024-03-06T07:25:33.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.592+0000 7f32390c9640 1 -- 172.21.15.1:0/3761813889 wait complete. 2024-03-06T07:25:33.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.593+0000 7f32390c9640 1 Processor -- start 2024-03-06T07:25:33.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.594+0000 7f32390c9640 1 -- start start 2024-03-06T07:25:33.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.594+0000 7f32390c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3234072bd0 0x7f32340c46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.594+0000 7f32390c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32340c2b30 0x7f3234149920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.594+0000 7f32390c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3234152b70 0x7f3234149e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.594+0000 7f32390c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32340c3ef0 con 0x7f3234152b70 2024-03-06T07:25:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.594+0000 7f32390c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32340c3d70 con 0x7f3234072bd0 2024-03-06T07:25:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.594+0000 7f32390c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3234155d80 con 0x7f32340c2b30 2024-03-06T07:25:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.594+0000 7f323359e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3234152b70 0x7f3234149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.594+0000 7f323359e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3234152b70 0x7f3234149e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53320/0 (socket says 172.21.15.1:53320) 2024-03-06T07:25:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.594+0000 7f323359e640 1 -- 172.21.15.1:0/3319818191 learned_addr learned my addr 172.21.15.1:0/3319818191 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.595+0000 7f323259c640 1 --2- 172.21.15.1:0/3319818191 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32340c2b30 0x7f3234149920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.595+0000 7f3232d9d640 1 --2- 172.21.15.1:0/3319818191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3234072bd0 0x7f32340c46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.595+0000 7f323359e640 1 -- 172.21.15.1:0/3319818191 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32340c2b30 msgr2=0x7f3234149920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.595+0000 7f323359e640 1 --2- 172.21.15.1:0/3319818191 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32340c2b30 0x7f3234149920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.595+0000 7f323359e640 1 -- 172.21.15.1:0/3319818191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3234072bd0 msgr2=0x7f32340c46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.595+0000 7f323359e640 1 --2- 172.21.15.1:0/3319818191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3234072bd0 0x7f32340c46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.595+0000 7f323359e640 1 -- 172.21.15.1:0/3319818191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32340bc720 con 0x7f3234152b70 2024-03-06T07:25:33.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.595+0000 7f323359e640 1 --2- 172.21.15.1:0/3319818191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3234152b70 0x7f3234149e60 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f322c00efc0 tx=0x7f322c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.595+0000 7f322b7fe640 1 -- 172.21.15.1:0/3319818191 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c018020 con 0x7f3234152b70 2024-03-06T07:25:33.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.595+0000 7f322b7fe640 1 -- 172.21.15.1:0/3319818191 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c014480 con 0x7f3234152b70 2024-03-06T07:25:33.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.595+0000 7f322b7fe640 1 -- 172.21.15.1:0/3319818191 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c009e60 con 0x7f3234152b70 2024-03-06T07:25:33.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.595+0000 7f32390c9640 1 -- 172.21.15.1:0/3319818191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32340bdc50 con 0x7f3234152b70 2024-03-06T07:25:33.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.595+0000 7f32390c9640 1 -- 172.21.15.1:0/3319818191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32340be210 con 0x7f3234152b70 2024-03-06T07:25:33.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.596+0000 7f32390c9640 1 -- 172.21.15.1:0/3319818191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3234067fd0 con 0x7f3234152b70 2024-03-06T07:25:33.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.597+0000 7f322b7fe640 1 -- 172.21.15.1:0/3319818191 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f322c01b020 con 0x7f3234152b70 2024-03-06T07:25:33.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.597+0000 7f322b7fe640 1 --2- 172.21.15.1:0/3319818191 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f321406b4d0 0x7f321406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.598+0000 7f322b7fe640 1 -- 172.21.15.1:0/3319818191 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f322c08b090 con 0x7f3234152b70 2024-03-06T07:25:33.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.598+0000 7f3232d9d640 1 --2- 172.21.15.1:0/3319818191 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f321406b4d0 0x7f321406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.598+0000 7f3232d9d640 1 --2- 172.21.15.1:0/3319818191 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f321406b4d0 0x7f321406d990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f321c007fd0 tx=0x7f321c007e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.602+0000 7f322b7fe640 1 -- 172.21.15.1:0/3319818191 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f322c022020 con 0x7f3234152b70 2024-03-06T07:25:33.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.782+0000 7f32390c9640 1 -- 172.21.15.1:0/3319818191 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3234071790 con 0x7f321406b4d0 2024-03-06T07:25:33.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.788+0000 7f322b7fe640 1 -- 172.21.15.1:0/3319818191 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247165 (secure 0 0 0) 0x7f3234071790 con 0x7f321406b4d0 2024-03-06T07:25:33.792 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:33.793 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:25:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.791+0000 7f32390c9640 1 -- 172.21.15.1:0/3319818191 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f321406b4d0 msgr2=0x7f321406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.791+0000 7f32390c9640 1 --2- 172.21.15.1:0/3319818191 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f321406b4d0 0x7f321406d990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f321c007fd0 tx=0x7f321c007e80 comp rx=0 tx=0).stop 2024-03-06T07:25:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.791+0000 7f32390c9640 1 -- 172.21.15.1:0/3319818191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3234152b70 msgr2=0x7f3234149e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.791+0000 7f32390c9640 1 --2- 172.21.15.1:0/3319818191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3234152b70 0x7f3234149e60 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f322c00efc0 tx=0x7f322c00c490 comp rx=0 tx=0).stop 2024-03-06T07:25:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.791+0000 7f32390c9640 1 -- 172.21.15.1:0/3319818191 shutdown_connections 2024-03-06T07:25:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.791+0000 7f32390c9640 1 --2- 172.21.15.1:0/3319818191 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f321406b4d0 0x7f321406d990 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.791+0000 7f32390c9640 1 --2- 172.21.15.1:0/3319818191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3234152b70 0x7f3234149e60 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.791+0000 7f32390c9640 1 --2- 172.21.15.1:0/3319818191 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32340c2b30 0x7f3234149920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.791+0000 7f32390c9640 1 --2- 172.21.15.1:0/3319818191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3234072bd0 0x7f32340c46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.791+0000 7f32390c9640 1 -- 172.21.15.1:0/3319818191 >> 172.21.15.1:0/3319818191 conn(0x7f32340c6860 msgr2=0x7f323414fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.791+0000 7f32390c9640 1 -- 172.21.15.1:0/3319818191 shutdown_connections 2024-03-06T07:25:33.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.791+0000 7f32390c9640 1 -- 172.21.15.1:0/3319818191 wait complete. 2024-03-06T07:25:33.804 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-03-06T07:25:32.761549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262588,"kb_used_data":13112,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712132,"statfs":{"total":773094113280,"available":772825223168,"internally_reserved":0,"allocated":13426688,"data_stored":7576910,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.340873"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429803+0000","last_change":"2024-03-06T07:25:29.429803+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429803+0000","last_undegraded":"2024-03-06T07:25:29.429803+0000","last_fullsized":"2024-03-06T07:25:29.429803+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429415+0000","last_change":"2024-03-06T07:25:29.429415+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429415+0000","last_undegraded":"2024-03-06T07:25:29.429415+0000","last_fullsized":"2024-03-06T07:25:29.429415+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.432950+0000","last_change":"2024-03-06T07:25:29.432950+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.432950+0000","last_undegraded":"2024-03-06T07:25:29.432950+0000","last_fullsized":"2024-03-06T07:25:29.432950+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.434735+0000","last_change":"2024-03-06T07:25:29.434735+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.434735+0000","last_undegraded":"2024-03-06T07:25:29.434735+0000","last_fullsized":"2024-03-06T07:25:29.434735+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423617+0000","last_change":"2024-03-06T07:25:29.423617+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423617+0000","last_undegraded":"2024-03-06T07:25:29.423617+0000","last_fullsized":"2024-03-06T07:25:29.423617+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423430+0000","last_change":"2024-03-06T07:25:29.423430+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423430+0000","last_undegraded":"2024-03-06T07:25:29.423430+0000","last_fullsized":"2024-03-06T07:25:29.423430+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430543+0000","last_change":"2024-03-06T07:25:29.430543+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430543+0000","last_undegraded":"2024-03-06T07:25:29.430543+0000","last_fullsized":"2024-03-06T07:25:29.430543+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428243+0000","last_change":"2024-03-06T07:25:29.428243+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428243+0000","last_undegraded":"2024-03-06T07:25:29.428243+0000","last_fullsized":"2024-03-06T07:25:29.428243+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423943+0000","last_change":"2024-03-06T07:25:29.423943+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423943+0000","last_undegraded":"2024-03-06T07:25:29.423943+0000","last_fullsized":"2024-03-06T07:25:29.423943+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423699+0000","last_change":"2024-03-06T07:25:29.423699+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423699+0000","last_undegraded":"2024-03-06T07:25:29.423699+0000","last_fullsized":"2024-03-06T07:25:29.423699+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427685+0000","last_change":"2024-03-06T07:25:29.427685+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427685+0000","last_undegraded":"2024-03-06T07:25:29.427685+0000","last_fullsized":"2024-03-06T07:25:29.427685+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.433682+0000","last_change":"2024-03-06T07:25:29.433682+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.433682+0000","last_undegraded":"2024-03-06T07:25:29.433682+0000","last_fullsized":"2024-03-06T07:25:29.433682+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.424516+0000","last_change":"2024-03-06T07:25:29.424516+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.424516+0000","last_undegraded":"2024-03-06T07:25:29.424516+0000","last_fullsized":"2024-03-06T07:25:29.424516+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.433176+0000","last_change":"2024-03-06T07:25:29.433176+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.433176+0000","last_undegraded":"2024-03-06T07:25:29.433176+0000","last_fullsized":"2024-03-06T07:25:29.433176+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.432452+0000","last_change":"2024-03-06T07:25:29.432452+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.432452+0000","last_undegraded":"2024-03-06T07:25:29.432452+0000","last_fullsized":"2024-03-06T07:25:29.432452+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423245+0000","last_change":"2024-03-06T07:25:29.423245+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423245+0000","last_undegraded":"2024-03-06T07:25:29.423245+0000","last_fullsized":"2024-03-06T07:25:29.423245+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.434177+0000","last_change":"2024-03-06T07:25:29.434177+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.434177+0000","last_undegraded":"2024-03-06T07:25:29.434177+0000","last_fullsized":"2024-03-06T07:25:29.434177+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428279+0000","last_change":"2024-03-06T07:25:29.428279+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428279+0000","last_undegraded":"2024-03-06T07:25:29.428279+0000","last_fullsized":"2024-03-06T07:25:29.428279+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429606+0000","last_change":"2024-03-06T07:25:29.429606+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429606+0000","last_undegraded":"2024-03-06T07:25:29.429606+0000","last_fullsized":"2024-03-06T07:25:29.429606+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427157+0000","last_change":"2024-03-06T07:25:29.427157+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427157+0000","last_undegraded":"2024-03-06T07:25:29.427157+0000","last_fullsized":"2024-03-06T07:25:29.427157+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.424094+0000","last_change":"2024-03-06T07:25:29.424094+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.424094+0000","last_undegraded":"2024-03-06T07:25:29.424094+0000","last_fullsized":"2024-03-06T07:25:29.424094+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427504+0000","last_change":"2024-03-06T07:25:29.427504+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427504+0000","last_undegraded":"2024-03-06T07:25:29.427504+0000","last_fullsized":"2024-03-06T07:25:29.427504+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.426134+0000","last_change":"2024-03-06T07:25:29.426134+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.426134+0000","last_undegraded":"2024-03-06T07:25:29.426134+0000","last_fullsized":"2024-03-06T07:25:29.426134+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.426814+0000","last_change":"2024-03-06T07:25:29.426814+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.426814+0000","last_undegraded":"2024-03-06T07:25:29.426814+0000","last_fullsized":"2024-03-06T07:25:29.426814+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429388+0000","last_change":"2024-03-06T07:25:29.429388+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429388+0000","last_undegraded":"2024-03-06T07:25:29.429388+0000","last_fullsized":"2024-03-06T07:25:29.429388+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431459+0000","last_change":"2024-03-06T07:25:29.431459+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431459+0000","last_undegraded":"2024-03-06T07:25:29.431459+0000","last_fullsized":"2024-03-06T07:25:29.431459+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.422691+0000","last_change":"2024-03-06T07:25:29.422691+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.422691+0000","last_undegraded":"2024-03-06T07:25:29.422691+0000","last_fullsized":"2024-03-06T07:25:29.422691+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427342+0000","last_change":"2024-03-06T07:25:29.427342+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427342+0000","last_undegraded":"2024-03-06T07:25:29.427342+0000","last_fullsized":"2024-03-06T07:25:29.427342+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.426341+0000","last_change":"2024-03-06T07:25:29.426341+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.426341+0000","last_undegraded":"2024-03-06T07:25:29.426341+0000","last_fullsized":"2024-03-06T07:25:29.426341+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429494+0000","last_change":"2024-03-06T07:25:29.429494+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429494+0000","last_undegraded":"2024-03-06T07:25:29.429494+0000","last_fullsized":"2024-03-06T07:25:29.429494+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.424039+0000","last_change":"2024-03-06T07:25:29.424039+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.424039+0000","last_undegraded":"2024-03-06T07:25:29.424039+0000","last_fullsized":"2024-03-06T07:25:29.424039+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427794+0000","last_change":"2024-03-06T07:25:29.427794+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427794+0000","last_undegraded":"2024-03-06T07:25:29.427794+0000","last_fullsized":"2024-03-06T07:25:29.427794+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430468+0000","last_change":"2024-03-06T07:25:29.430468+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430468+0000","last_undegraded":"2024-03-06T07:25:29.430468+0000","last_fullsized":"2024-03-06T07:25:29.430468+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.429642+0000","last_change":"2024-03-06T07:25:29.427051+0000","last_active":"2024-03-06T07:25:29.429642+0000","last_peered":"2024-03-06T07:25:29.429642+0000","last_clean":"2024-03-06T07:25:29.429642+0000","last_became_active":"2024-03-06T07:25:29.426750+0000","last_became_peered":"2024-03-06T07:25:29.426750+0000","last_unstale":"2024-03-06T07:25:29.429642+0000","last_undegraded":"2024-03-06T07:25:29.429642+0000","last_fullsized":"2024-03-06T07:25:29.429642+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:20.119206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430796+0000","last_change":"2024-03-06T07:25:29.430796+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430796+0000","last_undegraded":"2024-03-06T07:25:29.430796+0000","last_fullsized":"2024-03-06T07:25:29.430796+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.444195+0000","last_change":"2024-03-06T07:25:29.441160+0000","last_active":"2024-03-06T07:25:29.444195+0000","last_peered":"2024-03-06T07:25:29.444195+0000","last_clean":"2024-03-06T07:25:29.444195+0000","last_became_active":"2024-03-06T07:25:29.440834+0000","last_became_peered":"2024-03-06T07:25:29.440834+0000","last_unstale":"2024-03-06T07:25:29.444195+0000","last_undegraded":"2024-03-06T07:25:29.444195+0000","last_fullsized":"2024-03-06T07:25:29.444195+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:13.644859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.424888+0000","last_change":"2024-03-06T07:25:29.424888+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.424888+0000","last_undegraded":"2024-03-06T07:25:29.424888+0000","last_fullsized":"2024-03-06T07:25:29.424888+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.446817+0000","last_change":"2024-03-06T07:25:29.443849+0000","last_active":"2024-03-06T07:25:29.446817+0000","last_peered":"2024-03-06T07:25:29.446817+0000","last_clean":"2024-03-06T07:25:29.446817+0000","last_became_active":"2024-03-06T07:25:29.443606+0000","last_became_peered":"2024-03-06T07:25:29.443606+0000","last_unstale":"2024-03-06T07:25:29.446817+0000","last_undegraded":"2024-03-06T07:25:29.446817+0000","last_fullsized":"2024-03-06T07:25:29.446817+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:04.414856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427180+0000","last_change":"2024-03-06T07:25:29.427180+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427180+0000","last_undegraded":"2024-03-06T07:25:29.427180+0000","last_fullsized":"2024-03-06T07:25:29.427180+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442922+0000","last_change":"2024-03-06T07:25:29.442089+0000","last_active":"2024-03-06T07:25:29.442922+0000","last_peered":"2024-03-06T07:25:29.442922+0000","last_clean":"2024-03-06T07:25:29.442922+0000","last_became_active":"2024-03-06T07:25:29.441679+0000","last_became_peered":"2024-03-06T07:25:29.441679+0000","last_unstale":"2024-03-06T07:25:29.442922+0000","last_undegraded":"2024-03-06T07:25:29.442922+0000","last_fullsized":"2024-03-06T07:25:29.442922+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:48.284334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.426438+0000","last_change":"2024-03-06T07:25:29.426438+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.426438+0000","last_undegraded":"2024-03-06T07:25:29.426438+0000","last_fullsized":"2024-03-06T07:25:29.426438+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.444420+0000","last_change":"2024-03-06T07:25:29.438025+0000","last_active":"2024-03-06T07:25:29.444420+0000","last_peered":"2024-03-06T07:25:29.444420+0000","last_clean":"2024-03-06T07:25:29.444420+0000","last_became_active":"2024-03-06T07:25:29.437765+0000","last_became_peered":"2024-03-06T07:25:29.437765+0000","last_unstale":"2024-03-06T07:25:29.444420+0000","last_undegraded":"2024-03-06T07:25:29.444420+0000","last_fullsized":"2024-03-06T07:25:29.444420+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:28.072325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431875+0000","last_change":"2024-03-06T07:25:29.431875+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431875+0000","last_undegraded":"2024-03-06T07:25:29.431875+0000","last_fullsized":"2024-03-06T07:25:29.431875+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.439676+0000","last_change":"2024-03-06T07:25:29.437797+0000","last_active":"2024-03-06T07:25:29.439676+0000","last_peered":"2024-03-06T07:25:29.439676+0000","last_clean":"2024-03-06T07:25:29.439676+0000","last_became_active":"2024-03-06T07:25:29.437461+0000","last_became_peered":"2024-03-06T07:25:29.437461+0000","last_unstale":"2024-03-06T07:25:29.439676+0000","last_undegraded":"2024-03-06T07:25:29.439676+0000","last_fullsized":"2024-03-06T07:25:29.439676+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:07.155504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431792+0000","last_change":"2024-03-06T07:25:29.431792+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431792+0000","last_undegraded":"2024-03-06T07:25:29.431792+0000","last_fullsized":"2024-03-06T07:25:29.431792+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442539+0000","last_change":"2024-03-06T07:25:29.437803+0000","last_active":"2024-03-06T07:25:29.442539+0000","last_peered":"2024-03-06T07:25:29.442539+0000","last_clean":"2024-03-06T07:25:29.442539+0000","last_became_active":"2024-03-06T07:25:29.437527+0000","last_became_peered":"2024-03-06T07:25:29.437527+0000","last_unstale":"2024-03-06T07:25:29.442539+0000","last_undegraded":"2024-03-06T07:25:29.442539+0000","last_fullsized":"2024-03-06T07:25:29.442539+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:40.718218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428427+0000","last_change":"2024-03-06T07:25:29.428427+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428427+0000","last_undegraded":"2024-03-06T07:25:29.428427+0000","last_fullsized":"2024-03-06T07:25:29.428427+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.434590+0000","last_change":"2024-03-06T07:25:29.433974+0000","last_active":"2024-03-06T07:25:29.434590+0000","last_peered":"2024-03-06T07:25:29.434590+0000","last_clean":"2024-03-06T07:25:29.434590+0000","last_became_active":"2024-03-06T07:25:29.433432+0000","last_became_peered":"2024-03-06T07:25:29.433432+0000","last_unstale":"2024-03-06T07:25:29.434590+0000","last_undegraded":"2024-03-06T07:25:29.434590+0000","last_fullsized":"2024-03-06T07:25:29.434590+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:37.830403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427340+0000","last_change":"2024-03-06T07:25:29.427340+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427340+0000","last_undegraded":"2024-03-06T07:25:29.427340+0000","last_fullsized":"2024-03-06T07:25:29.427340+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.439779+0000","last_change":"2024-03-06T07:25:29.435113+0000","last_active":"2024-03-06T07:25:29.439779+0000","last_peered":"2024-03-06T07:25:29.439779+0000","last_clean":"2024-03-06T07:25:29.439779+0000","last_became_active":"2024-03-06T07:25:29.434807+0000","last_became_peered":"2024-03-06T07:25:29.434807+0000","last_unstale":"2024-03-06T07:25:29.439779+0000","last_undegraded":"2024-03-06T07:25:29.439779+0000","last_fullsized":"2024-03-06T07:25:29.439779+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:57.999261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427245+0000","last_change":"2024-03-06T07:25:29.427245+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427245+0000","last_undegraded":"2024-03-06T07:25:29.427245+0000","last_fullsized":"2024-03-06T07:25:29.427245+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442775+0000","last_change":"2024-03-06T07:25:29.437148+0000","last_active":"2024-03-06T07:25:29.442775+0000","last_peered":"2024-03-06T07:25:29.442775+0000","last_clean":"2024-03-06T07:25:29.442775+0000","last_became_active":"2024-03-06T07:25:29.436865+0000","last_became_peered":"2024-03-06T07:25:29.436865+0000","last_unstale":"2024-03-06T07:25:29.442775+0000","last_undegraded":"2024-03-06T07:25:29.442775+0000","last_fullsized":"2024-03-06T07:25:29.442775+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:44.776498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427865+0000","last_change":"2024-03-06T07:25:29.427865+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427865+0000","last_undegraded":"2024-03-06T07:25:29.427865+0000","last_fullsized":"2024-03-06T07:25:29.427865+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.434630+0000","last_change":"2024-03-06T07:25:29.431284+0000","last_active":"2024-03-06T07:25:29.434630+0000","last_peered":"2024-03-06T07:25:29.434630+0000","last_clean":"2024-03-06T07:25:29.434630+0000","last_became_active":"2024-03-06T07:25:29.431000+0000","last_became_peered":"2024-03-06T07:25:29.431000+0000","last_unstale":"2024-03-06T07:25:29.434630+0000","last_undegraded":"2024-03-06T07:25:29.434630+0000","last_fullsized":"2024-03-06T07:25:29.434630+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:29.131604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427343+0000","last_change":"2024-03-06T07:25:29.427343+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427343+0000","last_undegraded":"2024-03-06T07:25:29.427343+0000","last_fullsized":"2024-03-06T07:25:29.427343+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.428420+0000","last_change":"2024-03-06T07:25:29.426864+0000","last_active":"2024-03-06T07:25:29.428420+0000","last_peered":"2024-03-06T07:25:29.428420+0000","last_clean":"2024-03-06T07:25:29.428420+0000","last_became_active":"2024-03-06T07:25:29.426563+0000","last_became_peered":"2024-03-06T07:25:29.426563+0000","last_unstale":"2024-03-06T07:25:29.428420+0000","last_undegraded":"2024-03-06T07:25:29.428420+0000","last_fullsized":"2024-03-06T07:25:29.428420+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:28.472000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.432056+0000","last_change":"2024-03-06T07:25:29.432056+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.432056+0000","last_undegraded":"2024-03-06T07:25:29.432056+0000","last_fullsized":"2024-03-06T07:25:29.432056+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.446869+0000","last_change":"2024-03-06T07:25:29.442654+0000","last_active":"2024-03-06T07:25:29.446869+0000","last_peered":"2024-03-06T07:25:29.446869+0000","last_clean":"2024-03-06T07:25:29.446869+0000","last_became_active":"2024-03-06T07:25:29.442384+0000","last_became_peered":"2024-03-06T07:25:29.442384+0000","last_unstale":"2024-03-06T07:25:29.446869+0000","last_undegraded":"2024-03-06T07:25:29.446869+0000","last_fullsized":"2024-03-06T07:25:29.446869+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:26:04.515018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.422909+0000","last_change":"2024-03-06T07:25:29.422909+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.422909+0000","last_undegraded":"2024-03-06T07:25:29.422909+0000","last_fullsized":"2024-03-06T07:25:29.422909+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.444356+0000","last_change":"2024-03-06T07:25:29.437290+0000","last_active":"2024-03-06T07:25:29.444356+0000","last_peered":"2024-03-06T07:25:29.444356+0000","last_clean":"2024-03-06T07:25:29.444356+0000","last_became_active":"2024-03-06T07:25:29.437012+0000","last_became_peered":"2024-03-06T07:25:29.437012+0000","last_unstale":"2024-03-06T07:25:29.444356+0000","last_undegraded":"2024-03-06T07:25:29.444356+0000","last_fullsized":"2024-03-06T07:25:29.444356+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:44.052940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430432+0000","last_change":"2024-03-06T07:25:29.430432+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430432+0000","last_undegraded":"2024-03-06T07:25:29.430432+0000","last_fullsized":"2024-03-06T07:25:29.430432+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.449085+0000","last_change":"2024-03-06T07:25:29.444312+0000","last_active":"2024-03-06T07:25:29.449085+0000","last_peered":"2024-03-06T07:25:29.449085+0000","last_clean":"2024-03-06T07:25:29.449085+0000","last_became_active":"2024-03-06T07:25:29.444031+0000","last_became_peered":"2024-03-06T07:25:29.444031+0000","last_unstale":"2024-03-06T07:25:29.449085+0000","last_undegraded":"2024-03-06T07:25:29.449085+0000","last_fullsized":"2024-03-06T07:25:29.449085+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:43.317473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428129+0000","last_change":"2024-03-06T07:25:29.428129+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428129+0000","last_undegraded":"2024-03-06T07:25:29.428129+0000","last_fullsized":"2024-03-06T07:25:29.428129+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.443032+0000","last_change":"2024-03-06T07:25:29.442625+0000","last_active":"2024-03-06T07:25:29.443032+0000","last_peered":"2024-03-06T07:25:29.443032+0000","last_clean":"2024-03-06T07:25:29.443032+0000","last_became_active":"2024-03-06T07:25:29.442320+0000","last_became_peered":"2024-03-06T07:25:29.442320+0000","last_unstale":"2024-03-06T07:25:29.443032+0000","last_undegraded":"2024-03-06T07:25:29.443032+0000","last_fullsized":"2024-03-06T07:25:29.443032+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:36.851563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428953+0000","last_change":"2024-03-06T07:25:29.428953+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428953+0000","last_undegraded":"2024-03-06T07:25:29.428953+0000","last_fullsized":"2024-03-06T07:25:29.428953+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.441944+0000","last_change":"2024-03-06T07:25:29.437681+0000","last_active":"2024-03-06T07:25:29.441944+0000","last_peered":"2024-03-06T07:25:29.441944+0000","last_clean":"2024-03-06T07:25:29.441944+0000","last_became_active":"2024-03-06T07:25:29.437415+0000","last_became_peered":"2024-03-06T07:25:29.437415+0000","last_unstale":"2024-03-06T07:25:29.441944+0000","last_undegraded":"2024-03-06T07:25:29.441944+0000","last_fullsized":"2024-03-06T07:25:29.441944+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:59.804700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431074+0000","last_change":"2024-03-06T07:25:29.431074+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431074+0000","last_undegraded":"2024-03-06T07:25:29.431074+0000","last_fullsized":"2024-03-06T07:25:29.431074+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442850+0000","last_change":"2024-03-06T07:25:29.442155+0000","last_active":"2024-03-06T07:25:29.442850+0000","last_peered":"2024-03-06T07:25:29.442850+0000","last_clean":"2024-03-06T07:25:29.442850+0000","last_became_active":"2024-03-06T07:25:29.441793+0000","last_became_peered":"2024-03-06T07:25:29.441793+0000","last_unstale":"2024-03-06T07:25:29.442850+0000","last_undegraded":"2024-03-06T07:25:29.442850+0000","last_fullsized":"2024-03-06T07:25:29.442850+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:04.755959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428958+0000","last_change":"2024-03-06T07:25:29.428958+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428958+0000","last_undegraded":"2024-03-06T07:25:29.428958+0000","last_fullsized":"2024-03-06T07:25:29.428958+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.446861+0000","last_change":"2024-03-06T07:25:29.442853+0000","last_active":"2024-03-06T07:25:29.446861+0000","last_peered":"2024-03-06T07:25:29.446861+0000","last_clean":"2024-03-06T07:25:29.446861+0000","last_became_active":"2024-03-06T07:25:29.442618+0000","last_became_peered":"2024-03-06T07:25:29.442618+0000","last_unstale":"2024-03-06T07:25:29.446861+0000","last_undegraded":"2024-03-06T07:25:29.446861+0000","last_fullsized":"2024-03-06T07:25:29.446861+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:42.666506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427796+0000","last_change":"2024-03-06T07:25:29.427796+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427796+0000","last_undegraded":"2024-03-06T07:25:29.427796+0000","last_fullsized":"2024-03-06T07:25:29.427796+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.434976+0000","last_change":"2024-03-06T07:25:29.433350+0000","last_active":"2024-03-06T07:25:29.434976+0000","last_peered":"2024-03-06T07:25:29.434976+0000","last_clean":"2024-03-06T07:25:29.434976+0000","last_became_active":"2024-03-06T07:25:29.433064+0000","last_became_peered":"2024-03-06T07:25:29.433064+0000","last_unstale":"2024-03-06T07:25:29.434976+0000","last_undegraded":"2024-03-06T07:25:29.434976+0000","last_fullsized":"2024-03-06T07:25:29.434976+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:09.022615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429265+0000","last_change":"2024-03-06T07:25:29.429265+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429265+0000","last_undegraded":"2024-03-06T07:25:29.429265+0000","last_fullsized":"2024-03-06T07:25:29.429265+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.444452+0000","last_change":"2024-03-06T07:25:29.438191+0000","last_active":"2024-03-06T07:25:29.444452+0000","last_peered":"2024-03-06T07:25:29.444452+0000","last_clean":"2024-03-06T07:25:29.444452+0000","last_became_active":"2024-03-06T07:25:29.437589+0000","last_became_peered":"2024-03-06T07:25:29.437589+0000","last_unstale":"2024-03-06T07:25:29.444452+0000","last_undegraded":"2024-03-06T07:25:29.444452+0000","last_fullsized":"2024-03-06T07:25:29.444452+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:34.964198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428529+0000","last_change":"2024-03-06T07:25:29.428529+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428529+0000","last_undegraded":"2024-03-06T07:25:29.428529+0000","last_fullsized":"2024-03-06T07:25:29.428529+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.434947+0000","last_change":"2024-03-06T07:25:29.432629+0000","last_active":"2024-03-06T07:25:29.434947+0000","last_peered":"2024-03-06T07:25:29.434947+0000","last_clean":"2024-03-06T07:25:29.434947+0000","last_became_active":"2024-03-06T07:25:29.432348+0000","last_became_peered":"2024-03-06T07:25:29.432348+0000","last_unstale":"2024-03-06T07:25:29.434947+0000","last_undegraded":"2024-03-06T07:25:29.434947+0000","last_fullsized":"2024-03-06T07:25:29.434947+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:43.071021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430039+0000","last_change":"2024-03-06T07:25:29.430039+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430039+0000","last_undegraded":"2024-03-06T07:25:29.430039+0000","last_fullsized":"2024-03-06T07:25:29.430039+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.444491+0000","last_change":"2024-03-06T07:25:29.439680+0000","last_active":"2024-03-06T07:25:29.444491+0000","last_peered":"2024-03-06T07:25:29.444491+0000","last_clean":"2024-03-06T07:25:29.444491+0000","last_became_active":"2024-03-06T07:25:29.439401+0000","last_became_peered":"2024-03-06T07:25:29.439401+0000","last_unstale":"2024-03-06T07:25:29.444491+0000","last_undegraded":"2024-03-06T07:25:29.444491+0000","last_fullsized":"2024-03-06T07:25:29.444491+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:59.559815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.424099+0000","last_change":"2024-03-06T07:25:29.424099+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.424099+0000","last_undegraded":"2024-03-06T07:25:29.424099+0000","last_fullsized":"2024-03-06T07:25:29.424099+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.448969+0000","last_change":"2024-03-06T07:25:29.445124+0000","last_active":"2024-03-06T07:25:29.448969+0000","last_peered":"2024-03-06T07:25:29.448969+0000","last_clean":"2024-03-06T07:25:29.448969+0000","last_became_active":"2024-03-06T07:25:29.444870+0000","last_became_peered":"2024-03-06T07:25:29.444870+0000","last_unstale":"2024-03-06T07:25:29.448969+0000","last_undegraded":"2024-03-06T07:25:29.448969+0000","last_fullsized":"2024-03-06T07:25:29.448969+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:36.677607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.433944+0000","last_change":"2024-03-06T07:25:29.433944+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.433944+0000","last_undegraded":"2024-03-06T07:25:29.433944+0000","last_fullsized":"2024-03-06T07:25:29.433944+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.449041+0000","last_change":"2024-03-06T07:25:29.442634+0000","last_active":"2024-03-06T07:25:29.449041+0000","last_peered":"2024-03-06T07:25:29.449041+0000","last_clean":"2024-03-06T07:25:29.449041+0000","last_became_active":"2024-03-06T07:25:29.442362+0000","last_became_peered":"2024-03-06T07:25:29.442362+0000","last_unstale":"2024-03-06T07:25:29.449041+0000","last_undegraded":"2024-03-06T07:25:29.449041+0000","last_fullsized":"2024-03-06T07:25:29.449041+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:09.549229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430682+0000","last_change":"2024-03-06T07:25:29.430682+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430682+0000","last_undegraded":"2024-03-06T07:25:29.430682+0000","last_fullsized":"2024-03-06T07:25:29.430682+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.441524+0000","last_change":"2024-03-06T07:25:29.440980+0000","last_active":"2024-03-06T07:25:29.441524+0000","last_peered":"2024-03-06T07:25:29.441524+0000","last_clean":"2024-03-06T07:25:29.441524+0000","last_became_active":"2024-03-06T07:25:29.440645+0000","last_became_peered":"2024-03-06T07:25:29.440645+0000","last_unstale":"2024-03-06T07:25:29.441524+0000","last_undegraded":"2024-03-06T07:25:29.441524+0000","last_fullsized":"2024-03-06T07:25:29.441524+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:30.511914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423291+0000","last_change":"2024-03-06T07:25:29.423291+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423291+0000","last_undegraded":"2024-03-06T07:25:29.423291+0000","last_fullsized":"2024-03-06T07:25:29.423291+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.439976+0000","last_change":"2024-03-06T07:25:29.439438+0000","last_active":"2024-03-06T07:25:29.439976+0000","last_peered":"2024-03-06T07:25:29.439976+0000","last_clean":"2024-03-06T07:25:29.439976+0000","last_became_active":"2024-03-06T07:25:29.439129+0000","last_became_peered":"2024-03-06T07:25:29.439129+0000","last_unstale":"2024-03-06T07:25:29.439976+0000","last_undegraded":"2024-03-06T07:25:29.439976+0000","last_fullsized":"2024-03-06T07:25:29.439976+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:37.162898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.425093+0000","last_change":"2024-03-06T07:25:29.425093+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.425093+0000","last_undegraded":"2024-03-06T07:25:29.425093+0000","last_fullsized":"2024-03-06T07:25:29.425093+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442801+0000","last_change":"2024-03-06T07:25:29.439263+0000","last_active":"2024-03-06T07:25:29.442801+0000","last_peered":"2024-03-06T07:25:29.442801+0000","last_clean":"2024-03-06T07:25:29.442801+0000","last_became_active":"2024-03-06T07:25:29.438961+0000","last_became_peered":"2024-03-06T07:25:29.438961+0000","last_unstale":"2024-03-06T07:25:29.442801+0000","last_undegraded":"2024-03-06T07:25:29.442801+0000","last_fullsized":"2024-03-06T07:25:29.442801+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:34.084278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431354+0000","last_change":"2024-03-06T07:25:29.431354+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431354+0000","last_undegraded":"2024-03-06T07:25:29.431354+0000","last_fullsized":"2024-03-06T07:25:29.431354+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442824+0000","last_change":"2024-03-06T07:25:29.442215+0000","last_active":"2024-03-06T07:25:29.442824+0000","last_peered":"2024-03-06T07:25:29.442824+0000","last_clean":"2024-03-06T07:25:29.442824+0000","last_became_active":"2024-03-06T07:25:29.441739+0000","last_became_peered":"2024-03-06T07:25:29.441739+0000","last_unstale":"2024-03-06T07:25:29.442824+0000","last_undegraded":"2024-03-06T07:25:29.442824+0000","last_fullsized":"2024-03-06T07:25:29.442824+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:21.032029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431388+0000","last_change":"2024-03-06T07:25:29.431388+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431388+0000","last_undegraded":"2024-03-06T07:25:29.431388+0000","last_fullsized":"2024-03-06T07:25:29.431388+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442516+0000","last_change":"2024-03-06T07:25:29.437548+0000","last_active":"2024-03-06T07:25:29.442516+0000","last_peered":"2024-03-06T07:25:29.442516+0000","last_clean":"2024-03-06T07:25:29.442516+0000","last_became_active":"2024-03-06T07:25:29.437281+0000","last_became_peered":"2024-03-06T07:25:29.437281+0000","last_unstale":"2024-03-06T07:25:29.442516+0000","last_undegraded":"2024-03-06T07:25:29.442516+0000","last_fullsized":"2024-03-06T07:25:29.442516+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:18.466217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.426909+0000","last_change":"2024-03-06T07:25:29.426909+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.426909+0000","last_undegraded":"2024-03-06T07:25:29.426909+0000","last_fullsized":"2024-03-06T07:25:29.426909+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442173+0000","last_change":"2024-03-06T07:25:29.437129+0000","last_active":"2024-03-06T07:25:29.442173+0000","last_peered":"2024-03-06T07:25:29.442173+0000","last_clean":"2024-03-06T07:25:29.442173+0000","last_became_active":"2024-03-06T07:25:29.436825+0000","last_became_peered":"2024-03-06T07:25:29.436825+0000","last_unstale":"2024-03-06T07:25:29.442173+0000","last_undegraded":"2024-03-06T07:25:29.442173+0000","last_fullsized":"2024-03-06T07:25:29.442173+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:28.307200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.425972+0000","last_change":"2024-03-06T07:25:29.425972+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.425972+0000","last_undegraded":"2024-03-06T07:25:29.425972+0000","last_fullsized":"2024-03-06T07:25:29.425972+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.441092+0000","last_change":"2024-03-06T07:25:29.440579+0000","last_active":"2024-03-06T07:25:29.441092+0000","last_peered":"2024-03-06T07:25:29.441092+0000","last_clean":"2024-03-06T07:25:29.441092+0000","last_became_active":"2024-03-06T07:25:29.440247+0000","last_became_peered":"2024-03-06T07:25:29.440247+0000","last_unstale":"2024-03-06T07:25:29.441092+0000","last_undegraded":"2024-03-06T07:25:29.441092+0000","last_fullsized":"2024-03-06T07:25:29.441092+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:13.223054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":385,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.428367+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:25:29.428367+0000","last_peered":"2024-03-06T07:25:29.428367+0000","last_clean":"2024-03-06T07:25:29.428367+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:25:29.428367+0000","last_undegraded":"2024-03-06T07:25:29.428367+0000","last_fullsized":"2024-03-06T07:25:29.428367+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32868,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338972,"statfs":{"total":96636764160,"available":96603107328,"internally_reserved":0,"allocated":1724416,"data_stored":1388223,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32812,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339028,"statfs":{"total":96636764160,"available":96603164672,"internally_reserved":0,"allocated":1671168,"data_stored":804522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574913,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32808,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339032,"statfs":{"total":96636764160,"available":96603168768,"internally_reserved":0,"allocated":1658880,"data_stored":797855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32808,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339032,"statfs":{"total":96636764160,"available":96603168768,"internally_reserved":0,"allocated":1658880,"data_stored":797855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32808,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339032,"statfs":{"total":96636764160,"available":96603168768,"internally_reserved":0,"allocated":1658880,"data_stored":797855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574913,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32800,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339040,"statfs":{"total":96636764160,"available":96603176960,"internally_reserved":0,"allocated":1658880,"data_stored":797855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32812,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339028,"statfs":{"total":96636764160,"available":96603164672,"internally_reserved":0,"allocated":1671168,"data_stored":804522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574914,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32872,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338968,"statfs":{"total":96636764160,"available":96603103232,"internally_reserved":0,"allocated":1724416,"data_stored":1388223,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:33.807 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.942+0000 7fc3bee39640 1 Processor -- start 2024-03-06T07:25:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.942+0000 7fc3bee39640 1 -- start start 2024-03-06T07:25:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.943+0000 7fc3bee39640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc3b80c3970 0x7fc3b80c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.943+0000 7fc3bee39640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3b80c60b0 0x7fc3b8152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.943+0000 7fc3bee39640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b80c5e40 con 0x7fc3b8152670 2024-03-06T07:25:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.943+0000 7fc3bee39640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b80c67f0 con 0x7fc3b80c3970 2024-03-06T07:25:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.943+0000 7fc3bee39640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b80c6a40 con 0x7fc3b80c60b0 2024-03-06T07:25:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.943+0000 7fc3bcbd5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc3b80c3970 0x7fc3b80c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.943+0000 7fc3bd3d6640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc3b8152670 0x7fc3b81553d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33004/0 (socket says 172.21.15.1:33004) 2024-03-06T07:25:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.943+0000 7fc3bd3d6640 1 -- 172.21.15.1:0/3687336366 learned_addr learned my addr 172.21.15.1:0/3687336366 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.944+0000 7fc3b7fff640 1 --2- 172.21.15.1:0/3687336366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3b80c60b0 0x7fc3b8152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.944+0000 7fc3bcbd5640 1 -- 172.21.15.1:0/3687336366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3b80c60b0 msgr2=0x7fc3b8152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.944+0000 7fc3bcbd5640 1 --2- 172.21.15.1:0/3687336366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3b80c60b0 0x7fc3b8152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.944+0000 7fc3bcbd5640 1 -- 172.21.15.1:0/3687336366 >> v1:172.21.15.1:6789/0 conn(0x7fc3b8152670 legacy=0x7fc3b81553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.944+0000 7fc3bcbd5640 1 -- 172.21.15.1:0/3687336366 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b80c6d20 con 0x7fc3b80c3970 2024-03-06T07:25:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.944+0000 7fc3b77fe640 1 -- 172.21.15.1:0/3687336366 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825756446 0 0) 0x7fc3b80c5e40 con 0x7fc3b8152670 2024-03-06T07:25:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.944+0000 7fc3bcbd5640 1 --2- 172.21.15.1:0/3687336366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3b80c3970 0x7fc3b80c8170 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac0097a0 tx=0x7fc3ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6ea723778638d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.945+0000 7fc3b77fe640 1 -- 172.21.15.1:0/3687336366 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac035020 con 0x7fc3b80c3970 2024-03-06T07:25:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.945+0000 7fc3b77fe640 1 -- 172.21.15.1:0/3687336366 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3ac031830 con 0x7fc3b80c3970 2024-03-06T07:25:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.945+0000 7fc3b77fe640 1 -- 172.21.15.1:0/3687336366 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac031c00 con 0x7fc3b80c3970 2024-03-06T07:25:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.945+0000 7fc3bee39640 1 -- 172.21.15.1:0/3687336366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3b80c3970 msgr2=0x7fc3b80c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.945+0000 7fc3bee39640 1 --2- 172.21.15.1:0/3687336366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3b80c3970 0x7fc3b80c8170 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac0097a0 tx=0x7fc3ac02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.945+0000 7fc3bee39640 1 -- 172.21.15.1:0/3687336366 shutdown_connections 2024-03-06T07:25:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.945+0000 7fc3bee39640 1 --2- 172.21.15.1:0/3687336366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3b80c60b0 0x7fc3b8152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.945+0000 7fc3bee39640 1 --2- 172.21.15.1:0/3687336366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3b80c3970 0x7fc3b80c8170 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.945+0000 7fc3bee39640 1 -- 172.21.15.1:0/3687336366 >> 172.21.15.1:0/3687336366 conn(0x7fc3b80c88f0 msgr2=0x7fc3b80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.945+0000 7fc3bee39640 1 -- 172.21.15.1:0/3687336366 shutdown_connections 2024-03-06T07:25:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.945+0000 7fc3bee39640 1 -- 172.21.15.1:0/3687336366 wait complete. 2024-03-06T07:25:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.946+0000 7fc3bee39640 1 Processor -- start 2024-03-06T07:25:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.946+0000 7fc3bee39640 1 -- start start 2024-03-06T07:25:33.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.947+0000 7fc3bee39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3b80c3970 0x7fc3b80acc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.947+0000 7fc3bee39640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3b80c60b0 0x7fc3b80a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.947+0000 7fc3bee39640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3b8152670 0x7fc3b8155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.947+0000 7fc3bee39640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b80c75c0 con 0x7fc3b8152670 2024-03-06T07:25:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.947+0000 7fc3bee39640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b80c72e0 con 0x7fc3b80c3970 2024-03-06T07:25:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.947+0000 7fc3bee39640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b80c3d50 con 0x7fc3b80c60b0 2024-03-06T07:25:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.947+0000 7fc3bd3d6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3b8152670 0x7fc3b8155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.947+0000 7fc3bd3d6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3b8152670 0x7fc3b8155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53338/0 (socket says 172.21.15.1:53338) 2024-03-06T07:25:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.947+0000 7fc3bd3d6640 1 -- 172.21.15.1:0/3449024097 learned_addr learned my addr 172.21.15.1:0/3449024097 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.948+0000 7fc3b7fff640 1 --2- 172.21.15.1:0/3449024097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3b80c60b0 0x7fc3b80a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.948+0000 7fc3bcbd5640 1 --2- 172.21.15.1:0/3449024097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3b80c3970 0x7fc3b80acc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.948+0000 7fc3bd3d6640 1 -- 172.21.15.1:0/3449024097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3b80c60b0 msgr2=0x7fc3b80a97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.948+0000 7fc3bd3d6640 1 --2- 172.21.15.1:0/3449024097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3b80c60b0 0x7fc3b80a97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.948+0000 7fc3bd3d6640 1 -- 172.21.15.1:0/3449024097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3b80c3970 msgr2=0x7fc3b80acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.948+0000 7fc3bd3d6640 1 --2- 172.21.15.1:0/3449024097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3b80c3970 0x7fc3b80acc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.948+0000 7fc3bd3d6640 1 -- 172.21.15.1:0/3449024097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b80ad170 con 0x7fc3b8152670 2024-03-06T07:25:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.948+0000 7fc3bd3d6640 1 --2- 172.21.15.1:0/3449024097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3b8152670 0x7fc3b8155e70 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fc3b000c730 tx=0x7fc3b000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.948+0000 7fc3b57fa640 1 -- 172.21.15.1:0/3449024097 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0018070 con 0x7fc3b8152670 2024-03-06T07:25:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.948+0000 7fc3b57fa640 1 -- 172.21.15.1:0/3449024097 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b000ce40 con 0x7fc3b8152670 2024-03-06T07:25:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.948+0000 7fc3bee39640 1 -- 172.21.15.1:0/3449024097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b80ab280 con 0x7fc3b8152670 2024-03-06T07:25:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.948+0000 7fc3b57fa640 1 -- 172.21.15.1:0/3449024097 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b000a860 con 0x7fc3b8152670 2024-03-06T07:25:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.948+0000 7fc3bee39640 1 -- 172.21.15.1:0/3449024097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b81eafb0 con 0x7fc3b8152670 2024-03-06T07:25:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.949+0000 7fc3bee39640 1 -- 172.21.15.1:0/3449024097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b80686e0 con 0x7fc3b8152670 2024-03-06T07:25:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.950+0000 7fc3b57fa640 1 -- 172.21.15.1:0/3449024097 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3b001b070 con 0x7fc3b8152670 2024-03-06T07:25:33.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.950+0000 7fc3b57fa640 1 --2- 172.21.15.1:0/3449024097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc39c06b500 0x7fc39c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.950+0000 7fc3b57fa640 1 -- 172.21.15.1:0/3449024097 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc3b00898b0 con 0x7fc3b8152670 2024-03-06T07:25:33.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.950+0000 7fc3bcbd5640 1 --2- 172.21.15.1:0/3449024097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc39c06b500 0x7fc39c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.951+0000 7fc3bcbd5640 1 --2- 172.21.15.1:0/3449024097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc39c06b500 0x7fc39c06d9c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac036000 tx=0x7fc3ac008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:33.954+0000 7fc3b57fa640 1 -- 172.21.15.1:0/3449024097 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3b0058630 con 0x7fc3b8152670 2024-03-06T07:25:34.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.251+0000 7fc3bee39640 1 -- 172.21.15.1:0/3449024097 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc3b81ead60 con 0x7fc39c06b500 2024-03-06T07:25:34.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.257+0000 7fc3b57fa640 1 -- 172.21.15.1:0/3449024097 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247165 (secure 0 0 0) 0x7fc3ac085590 con 0x7fc39c06b500 2024-03-06T07:25:34.262 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:25:34.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.260+0000 7fc3bee39640 1 -- 172.21.15.1:0/3449024097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc39c06b500 msgr2=0x7fc39c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.260+0000 7fc3bee39640 1 --2- 172.21.15.1:0/3449024097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc39c06b500 0x7fc39c06d9c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac036000 tx=0x7fc3ac008030 comp rx=0 tx=0).stop 2024-03-06T07:25:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.260+0000 7fc3bee39640 1 -- 172.21.15.1:0/3449024097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3b8152670 msgr2=0x7fc3b8155e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.260+0000 7fc3bee39640 1 --2- 172.21.15.1:0/3449024097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3b8152670 0x7fc3b8155e70 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fc3b000c730 tx=0x7fc3b000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.260+0000 7fc3bee39640 1 -- 172.21.15.1:0/3449024097 shutdown_connections 2024-03-06T07:25:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.260+0000 7fc3bee39640 1 --2- 172.21.15.1:0/3449024097 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc39c06b500 0x7fc39c06d9c0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.260+0000 7fc3bee39640 1 --2- 172.21.15.1:0/3449024097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3b8152670 0x7fc3b8155e70 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.260+0000 7fc3bee39640 1 --2- 172.21.15.1:0/3449024097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3b80c60b0 0x7fc3b80a97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.260+0000 7fc3bee39640 1 --2- 172.21.15.1:0/3449024097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3b80c3970 0x7fc3b80acc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.260+0000 7fc3bee39640 1 -- 172.21.15.1:0/3449024097 >> 172.21.15.1:0/3449024097 conn(0x7fc3b80c88f0 msgr2=0x7fc3b8154af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.260+0000 7fc3bee39640 1 -- 172.21.15.1:0/3449024097 shutdown_connections 2024-03-06T07:25:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.260+0000 7fc3bee39640 1 -- 172.21.15.1:0/3449024097 wait complete. 2024-03-06T07:25:34.273 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-03-06T07:25:32.761549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262588,"kb_used_data":13112,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712132,"statfs":{"total":773094113280,"available":772825223168,"internally_reserved":0,"allocated":13426688,"data_stored":7576910,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.340873"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429803+0000","last_change":"2024-03-06T07:25:29.429803+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429803+0000","last_undegraded":"2024-03-06T07:25:29.429803+0000","last_fullsized":"2024-03-06T07:25:29.429803+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429415+0000","last_change":"2024-03-06T07:25:29.429415+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429415+0000","last_undegraded":"2024-03-06T07:25:29.429415+0000","last_fullsized":"2024-03-06T07:25:29.429415+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.432950+0000","last_change":"2024-03-06T07:25:29.432950+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.432950+0000","last_undegraded":"2024-03-06T07:25:29.432950+0000","last_fullsized":"2024-03-06T07:25:29.432950+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.434735+0000","last_change":"2024-03-06T07:25:29.434735+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.434735+0000","last_undegraded":"2024-03-06T07:25:29.434735+0000","last_fullsized":"2024-03-06T07:25:29.434735+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423617+0000","last_change":"2024-03-06T07:25:29.423617+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423617+0000","last_undegraded":"2024-03-06T07:25:29.423617+0000","last_fullsized":"2024-03-06T07:25:29.423617+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423430+0000","last_change":"2024-03-06T07:25:29.423430+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423430+0000","last_undegraded":"2024-03-06T07:25:29.423430+0000","last_fullsized":"2024-03-06T07:25:29.423430+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430543+0000","last_change":"2024-03-06T07:25:29.430543+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430543+0000","last_undegraded":"2024-03-06T07:25:29.430543+0000","last_fullsized":"2024-03-06T07:25:29.430543+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428243+0000","last_change":"2024-03-06T07:25:29.428243+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428243+0000","last_undegraded":"2024-03-06T07:25:29.428243+0000","last_fullsized":"2024-03-06T07:25:29.428243+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423943+0000","last_change":"2024-03-06T07:25:29.423943+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423943+0000","last_undegraded":"2024-03-06T07:25:29.423943+0000","last_fullsized":"2024-03-06T07:25:29.423943+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423699+0000","last_change":"2024-03-06T07:25:29.423699+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423699+0000","last_undegraded":"2024-03-06T07:25:29.423699+0000","last_fullsized":"2024-03-06T07:25:29.423699+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427685+0000","last_change":"2024-03-06T07:25:29.427685+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427685+0000","last_undegraded":"2024-03-06T07:25:29.427685+0000","last_fullsized":"2024-03-06T07:25:29.427685+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.433682+0000","last_change":"2024-03-06T07:25:29.433682+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.433682+0000","last_undegraded":"2024-03-06T07:25:29.433682+0000","last_fullsized":"2024-03-06T07:25:29.433682+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.424516+0000","last_change":"2024-03-06T07:25:29.424516+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.424516+0000","last_undegraded":"2024-03-06T07:25:29.424516+0000","last_fullsized":"2024-03-06T07:25:29.424516+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.433176+0000","last_change":"2024-03-06T07:25:29.433176+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.433176+0000","last_undegraded":"2024-03-06T07:25:29.433176+0000","last_fullsized":"2024-03-06T07:25:29.433176+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.432452+0000","last_change":"2024-03-06T07:25:29.432452+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.432452+0000","last_undegraded":"2024-03-06T07:25:29.432452+0000","last_fullsized":"2024-03-06T07:25:29.432452+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423245+0000","last_change":"2024-03-06T07:25:29.423245+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423245+0000","last_undegraded":"2024-03-06T07:25:29.423245+0000","last_fullsized":"2024-03-06T07:25:29.423245+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.434177+0000","last_change":"2024-03-06T07:25:29.434177+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.434177+0000","last_undegraded":"2024-03-06T07:25:29.434177+0000","last_fullsized":"2024-03-06T07:25:29.434177+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428279+0000","last_change":"2024-03-06T07:25:29.428279+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428279+0000","last_undegraded":"2024-03-06T07:25:29.428279+0000","last_fullsized":"2024-03-06T07:25:29.428279+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429606+0000","last_change":"2024-03-06T07:25:29.429606+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429606+0000","last_undegraded":"2024-03-06T07:25:29.429606+0000","last_fullsized":"2024-03-06T07:25:29.429606+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427157+0000","last_change":"2024-03-06T07:25:29.427157+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427157+0000","last_undegraded":"2024-03-06T07:25:29.427157+0000","last_fullsized":"2024-03-06T07:25:29.427157+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.424094+0000","last_change":"2024-03-06T07:25:29.424094+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.424094+0000","last_undegraded":"2024-03-06T07:25:29.424094+0000","last_fullsized":"2024-03-06T07:25:29.424094+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427504+0000","last_change":"2024-03-06T07:25:29.427504+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427504+0000","last_undegraded":"2024-03-06T07:25:29.427504+0000","last_fullsized":"2024-03-06T07:25:29.427504+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.426134+0000","last_change":"2024-03-06T07:25:29.426134+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.426134+0000","last_undegraded":"2024-03-06T07:25:29.426134+0000","last_fullsized":"2024-03-06T07:25:29.426134+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.426814+0000","last_change":"2024-03-06T07:25:29.426814+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.426814+0000","last_undegraded":"2024-03-06T07:25:29.426814+0000","last_fullsized":"2024-03-06T07:25:29.426814+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429388+0000","last_change":"2024-03-06T07:25:29.429388+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429388+0000","last_undegraded":"2024-03-06T07:25:29.429388+0000","last_fullsized":"2024-03-06T07:25:29.429388+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431459+0000","last_change":"2024-03-06T07:25:29.431459+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431459+0000","last_undegraded":"2024-03-06T07:25:29.431459+0000","last_fullsized":"2024-03-06T07:25:29.431459+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.422691+0000","last_change":"2024-03-06T07:25:29.422691+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.422691+0000","last_undegraded":"2024-03-06T07:25:29.422691+0000","last_fullsized":"2024-03-06T07:25:29.422691+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427342+0000","last_change":"2024-03-06T07:25:29.427342+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427342+0000","last_undegraded":"2024-03-06T07:25:29.427342+0000","last_fullsized":"2024-03-06T07:25:29.427342+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.426341+0000","last_change":"2024-03-06T07:25:29.426341+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.426341+0000","last_undegraded":"2024-03-06T07:25:29.426341+0000","last_fullsized":"2024-03-06T07:25:29.426341+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429494+0000","last_change":"2024-03-06T07:25:29.429494+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429494+0000","last_undegraded":"2024-03-06T07:25:29.429494+0000","last_fullsized":"2024-03-06T07:25:29.429494+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.424039+0000","last_change":"2024-03-06T07:25:29.424039+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.424039+0000","last_undegraded":"2024-03-06T07:25:29.424039+0000","last_fullsized":"2024-03-06T07:25:29.424039+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427794+0000","last_change":"2024-03-06T07:25:29.427794+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427794+0000","last_undegraded":"2024-03-06T07:25:29.427794+0000","last_fullsized":"2024-03-06T07:25:29.427794+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430468+0000","last_change":"2024-03-06T07:25:29.430468+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430468+0000","last_undegraded":"2024-03-06T07:25:29.430468+0000","last_fullsized":"2024-03-06T07:25:29.430468+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.429642+0000","last_change":"2024-03-06T07:25:29.427051+0000","last_active":"2024-03-06T07:25:29.429642+0000","last_peered":"2024-03-06T07:25:29.429642+0000","last_clean":"2024-03-06T07:25:29.429642+0000","last_became_active":"2024-03-06T07:25:29.426750+0000","last_became_peered":"2024-03-06T07:25:29.426750+0000","last_unstale":"2024-03-06T07:25:29.429642+0000","last_undegraded":"2024-03-06T07:25:29.429642+0000","last_fullsized":"2024-03-06T07:25:29.429642+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:20.119206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430796+0000","last_change":"2024-03-06T07:25:29.430796+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430796+0000","last_undegraded":"2024-03-06T07:25:29.430796+0000","last_fullsized":"2024-03-06T07:25:29.430796+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.444195+0000","last_change":"2024-03-06T07:25:29.441160+0000","last_active":"2024-03-06T07:25:29.444195+0000","last_peered":"2024-03-06T07:25:29.444195+0000","last_clean":"2024-03-06T07:25:29.444195+0000","last_became_active":"2024-03-06T07:25:29.440834+0000","last_became_peered":"2024-03-06T07:25:29.440834+0000","last_unstale":"2024-03-06T07:25:29.444195+0000","last_undegraded":"2024-03-06T07:25:29.444195+0000","last_fullsized":"2024-03-06T07:25:29.444195+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:13.644859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.424888+0000","last_change":"2024-03-06T07:25:29.424888+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.424888+0000","last_undegraded":"2024-03-06T07:25:29.424888+0000","last_fullsized":"2024-03-06T07:25:29.424888+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.446817+0000","last_change":"2024-03-06T07:25:29.443849+0000","last_active":"2024-03-06T07:25:29.446817+0000","last_peered":"2024-03-06T07:25:29.446817+0000","last_clean":"2024-03-06T07:25:29.446817+0000","last_became_active":"2024-03-06T07:25:29.443606+0000","last_became_peered":"2024-03-06T07:25:29.443606+0000","last_unstale":"2024-03-06T07:25:29.446817+0000","last_undegraded":"2024-03-06T07:25:29.446817+0000","last_fullsized":"2024-03-06T07:25:29.446817+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:04.414856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427180+0000","last_change":"2024-03-06T07:25:29.427180+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427180+0000","last_undegraded":"2024-03-06T07:25:29.427180+0000","last_fullsized":"2024-03-06T07:25:29.427180+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442922+0000","last_change":"2024-03-06T07:25:29.442089+0000","last_active":"2024-03-06T07:25:29.442922+0000","last_peered":"2024-03-06T07:25:29.442922+0000","last_clean":"2024-03-06T07:25:29.442922+0000","last_became_active":"2024-03-06T07:25:29.441679+0000","last_became_peered":"2024-03-06T07:25:29.441679+0000","last_unstale":"2024-03-06T07:25:29.442922+0000","last_undegraded":"2024-03-06T07:25:29.442922+0000","last_fullsized":"2024-03-06T07:25:29.442922+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:48.284334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.426438+0000","last_change":"2024-03-06T07:25:29.426438+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.426438+0000","last_undegraded":"2024-03-06T07:25:29.426438+0000","last_fullsized":"2024-03-06T07:25:29.426438+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.444420+0000","last_change":"2024-03-06T07:25:29.438025+0000","last_active":"2024-03-06T07:25:29.444420+0000","last_peered":"2024-03-06T07:25:29.444420+0000","last_clean":"2024-03-06T07:25:29.444420+0000","last_became_active":"2024-03-06T07:25:29.437765+0000","last_became_peered":"2024-03-06T07:25:29.437765+0000","last_unstale":"2024-03-06T07:25:29.444420+0000","last_undegraded":"2024-03-06T07:25:29.444420+0000","last_fullsized":"2024-03-06T07:25:29.444420+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:28.072325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431875+0000","last_change":"2024-03-06T07:25:29.431875+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431875+0000","last_undegraded":"2024-03-06T07:25:29.431875+0000","last_fullsized":"2024-03-06T07:25:29.431875+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.439676+0000","last_change":"2024-03-06T07:25:29.437797+0000","last_active":"2024-03-06T07:25:29.439676+0000","last_peered":"2024-03-06T07:25:29.439676+0000","last_clean":"2024-03-06T07:25:29.439676+0000","last_became_active":"2024-03-06T07:25:29.437461+0000","last_became_peered":"2024-03-06T07:25:29.437461+0000","last_unstale":"2024-03-06T07:25:29.439676+0000","last_undegraded":"2024-03-06T07:25:29.439676+0000","last_fullsized":"2024-03-06T07:25:29.439676+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:07.155504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431792+0000","last_change":"2024-03-06T07:25:29.431792+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431792+0000","last_undegraded":"2024-03-06T07:25:29.431792+0000","last_fullsized":"2024-03-06T07:25:29.431792+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442539+0000","last_change":"2024-03-06T07:25:29.437803+0000","last_active":"2024-03-06T07:25:29.442539+0000","last_peered":"2024-03-06T07:25:29.442539+0000","last_clean":"2024-03-06T07:25:29.442539+0000","last_became_active":"2024-03-06T07:25:29.437527+0000","last_became_peered":"2024-03-06T07:25:29.437527+0000","last_unstale":"2024-03-06T07:25:29.442539+0000","last_undegraded":"2024-03-06T07:25:29.442539+0000","last_fullsized":"2024-03-06T07:25:29.442539+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:40.718218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428427+0000","last_change":"2024-03-06T07:25:29.428427+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428427+0000","last_undegraded":"2024-03-06T07:25:29.428427+0000","last_fullsized":"2024-03-06T07:25:29.428427+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.434590+0000","last_change":"2024-03-06T07:25:29.433974+0000","last_active":"2024-03-06T07:25:29.434590+0000","last_peered":"2024-03-06T07:25:29.434590+0000","last_clean":"2024-03-06T07:25:29.434590+0000","last_became_active":"2024-03-06T07:25:29.433432+0000","last_became_peered":"2024-03-06T07:25:29.433432+0000","last_unstale":"2024-03-06T07:25:29.434590+0000","last_undegraded":"2024-03-06T07:25:29.434590+0000","last_fullsized":"2024-03-06T07:25:29.434590+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:37.830403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427340+0000","last_change":"2024-03-06T07:25:29.427340+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427340+0000","last_undegraded":"2024-03-06T07:25:29.427340+0000","last_fullsized":"2024-03-06T07:25:29.427340+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.439779+0000","last_change":"2024-03-06T07:25:29.435113+0000","last_active":"2024-03-06T07:25:29.439779+0000","last_peered":"2024-03-06T07:25:29.439779+0000","last_clean":"2024-03-06T07:25:29.439779+0000","last_became_active":"2024-03-06T07:25:29.434807+0000","last_became_peered":"2024-03-06T07:25:29.434807+0000","last_unstale":"2024-03-06T07:25:29.439779+0000","last_undegraded":"2024-03-06T07:25:29.439779+0000","last_fullsized":"2024-03-06T07:25:29.439779+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:57.999261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427245+0000","last_change":"2024-03-06T07:25:29.427245+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427245+0000","last_undegraded":"2024-03-06T07:25:29.427245+0000","last_fullsized":"2024-03-06T07:25:29.427245+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442775+0000","last_change":"2024-03-06T07:25:29.437148+0000","last_active":"2024-03-06T07:25:29.442775+0000","last_peered":"2024-03-06T07:25:29.442775+0000","last_clean":"2024-03-06T07:25:29.442775+0000","last_became_active":"2024-03-06T07:25:29.436865+0000","last_became_peered":"2024-03-06T07:25:29.436865+0000","last_unstale":"2024-03-06T07:25:29.442775+0000","last_undegraded":"2024-03-06T07:25:29.442775+0000","last_fullsized":"2024-03-06T07:25:29.442775+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:44.776498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427865+0000","last_change":"2024-03-06T07:25:29.427865+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427865+0000","last_undegraded":"2024-03-06T07:25:29.427865+0000","last_fullsized":"2024-03-06T07:25:29.427865+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.434630+0000","last_change":"2024-03-06T07:25:29.431284+0000","last_active":"2024-03-06T07:25:29.434630+0000","last_peered":"2024-03-06T07:25:29.434630+0000","last_clean":"2024-03-06T07:25:29.434630+0000","last_became_active":"2024-03-06T07:25:29.431000+0000","last_became_peered":"2024-03-06T07:25:29.431000+0000","last_unstale":"2024-03-06T07:25:29.434630+0000","last_undegraded":"2024-03-06T07:25:29.434630+0000","last_fullsized":"2024-03-06T07:25:29.434630+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:29.131604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427343+0000","last_change":"2024-03-06T07:25:29.427343+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427343+0000","last_undegraded":"2024-03-06T07:25:29.427343+0000","last_fullsized":"2024-03-06T07:25:29.427343+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.428420+0000","last_change":"2024-03-06T07:25:29.426864+0000","last_active":"2024-03-06T07:25:29.428420+0000","last_peered":"2024-03-06T07:25:29.428420+0000","last_clean":"2024-03-06T07:25:29.428420+0000","last_became_active":"2024-03-06T07:25:29.426563+0000","last_became_peered":"2024-03-06T07:25:29.426563+0000","last_unstale":"2024-03-06T07:25:29.428420+0000","last_undegraded":"2024-03-06T07:25:29.428420+0000","last_fullsized":"2024-03-06T07:25:29.428420+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:28.472000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.432056+0000","last_change":"2024-03-06T07:25:29.432056+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.432056+0000","last_undegraded":"2024-03-06T07:25:29.432056+0000","last_fullsized":"2024-03-06T07:25:29.432056+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.446869+0000","last_change":"2024-03-06T07:25:29.442654+0000","last_active":"2024-03-06T07:25:29.446869+0000","last_peered":"2024-03-06T07:25:29.446869+0000","last_clean":"2024-03-06T07:25:29.446869+0000","last_became_active":"2024-03-06T07:25:29.442384+0000","last_became_peered":"2024-03-06T07:25:29.442384+0000","last_unstale":"2024-03-06T07:25:29.446869+0000","last_undegraded":"2024-03-06T07:25:29.446869+0000","last_fullsized":"2024-03-06T07:25:29.446869+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:26:04.515018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.422909+0000","last_change":"2024-03-06T07:25:29.422909+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.422909+0000","last_undegraded":"2024-03-06T07:25:29.422909+0000","last_fullsized":"2024-03-06T07:25:29.422909+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.444356+0000","last_change":"2024-03-06T07:25:29.437290+0000","last_active":"2024-03-06T07:25:29.444356+0000","last_peered":"2024-03-06T07:25:29.444356+0000","last_clean":"2024-03-06T07:25:29.444356+0000","last_became_active":"2024-03-06T07:25:29.437012+0000","last_became_peered":"2024-03-06T07:25:29.437012+0000","last_unstale":"2024-03-06T07:25:29.444356+0000","last_undegraded":"2024-03-06T07:25:29.444356+0000","last_fullsized":"2024-03-06T07:25:29.444356+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:44.052940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430432+0000","last_change":"2024-03-06T07:25:29.430432+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430432+0000","last_undegraded":"2024-03-06T07:25:29.430432+0000","last_fullsized":"2024-03-06T07:25:29.430432+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.449085+0000","last_change":"2024-03-06T07:25:29.444312+0000","last_active":"2024-03-06T07:25:29.449085+0000","last_peered":"2024-03-06T07:25:29.449085+0000","last_clean":"2024-03-06T07:25:29.449085+0000","last_became_active":"2024-03-06T07:25:29.444031+0000","last_became_peered":"2024-03-06T07:25:29.444031+0000","last_unstale":"2024-03-06T07:25:29.449085+0000","last_undegraded":"2024-03-06T07:25:29.449085+0000","last_fullsized":"2024-03-06T07:25:29.449085+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:43.317473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428129+0000","last_change":"2024-03-06T07:25:29.428129+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428129+0000","last_undegraded":"2024-03-06T07:25:29.428129+0000","last_fullsized":"2024-03-06T07:25:29.428129+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.443032+0000","last_change":"2024-03-06T07:25:29.442625+0000","last_active":"2024-03-06T07:25:29.443032+0000","last_peered":"2024-03-06T07:25:29.443032+0000","last_clean":"2024-03-06T07:25:29.443032+0000","last_became_active":"2024-03-06T07:25:29.442320+0000","last_became_peered":"2024-03-06T07:25:29.442320+0000","last_unstale":"2024-03-06T07:25:29.443032+0000","last_undegraded":"2024-03-06T07:25:29.443032+0000","last_fullsized":"2024-03-06T07:25:29.443032+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:36.851563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428953+0000","last_change":"2024-03-06T07:25:29.428953+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428953+0000","last_undegraded":"2024-03-06T07:25:29.428953+0000","last_fullsized":"2024-03-06T07:25:29.428953+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.441944+0000","last_change":"2024-03-06T07:25:29.437681+0000","last_active":"2024-03-06T07:25:29.441944+0000","last_peered":"2024-03-06T07:25:29.441944+0000","last_clean":"2024-03-06T07:25:29.441944+0000","last_became_active":"2024-03-06T07:25:29.437415+0000","last_became_peered":"2024-03-06T07:25:29.437415+0000","last_unstale":"2024-03-06T07:25:29.441944+0000","last_undegraded":"2024-03-06T07:25:29.441944+0000","last_fullsized":"2024-03-06T07:25:29.441944+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:59.804700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431074+0000","last_change":"2024-03-06T07:25:29.431074+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431074+0000","last_undegraded":"2024-03-06T07:25:29.431074+0000","last_fullsized":"2024-03-06T07:25:29.431074+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442850+0000","last_change":"2024-03-06T07:25:29.442155+0000","last_active":"2024-03-06T07:25:29.442850+0000","last_peered":"2024-03-06T07:25:29.442850+0000","last_clean":"2024-03-06T07:25:29.442850+0000","last_became_active":"2024-03-06T07:25:29.441793+0000","last_became_peered":"2024-03-06T07:25:29.441793+0000","last_unstale":"2024-03-06T07:25:29.442850+0000","last_undegraded":"2024-03-06T07:25:29.442850+0000","last_fullsized":"2024-03-06T07:25:29.442850+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:04.755959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428958+0000","last_change":"2024-03-06T07:25:29.428958+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428958+0000","last_undegraded":"2024-03-06T07:25:29.428958+0000","last_fullsized":"2024-03-06T07:25:29.428958+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.446861+0000","last_change":"2024-03-06T07:25:29.442853+0000","last_active":"2024-03-06T07:25:29.446861+0000","last_peered":"2024-03-06T07:25:29.446861+0000","last_clean":"2024-03-06T07:25:29.446861+0000","last_became_active":"2024-03-06T07:25:29.442618+0000","last_became_peered":"2024-03-06T07:25:29.442618+0000","last_unstale":"2024-03-06T07:25:29.446861+0000","last_undegraded":"2024-03-06T07:25:29.446861+0000","last_fullsized":"2024-03-06T07:25:29.446861+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:42.666506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.427796+0000","last_change":"2024-03-06T07:25:29.427796+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.427796+0000","last_undegraded":"2024-03-06T07:25:29.427796+0000","last_fullsized":"2024-03-06T07:25:29.427796+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.434976+0000","last_change":"2024-03-06T07:25:29.433350+0000","last_active":"2024-03-06T07:25:29.434976+0000","last_peered":"2024-03-06T07:25:29.434976+0000","last_clean":"2024-03-06T07:25:29.434976+0000","last_became_active":"2024-03-06T07:25:29.433064+0000","last_became_peered":"2024-03-06T07:25:29.433064+0000","last_unstale":"2024-03-06T07:25:29.434976+0000","last_undegraded":"2024-03-06T07:25:29.434976+0000","last_fullsized":"2024-03-06T07:25:29.434976+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:09.022615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.429265+0000","last_change":"2024-03-06T07:25:29.429265+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.429265+0000","last_undegraded":"2024-03-06T07:25:29.429265+0000","last_fullsized":"2024-03-06T07:25:29.429265+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.444452+0000","last_change":"2024-03-06T07:25:29.438191+0000","last_active":"2024-03-06T07:25:29.444452+0000","last_peered":"2024-03-06T07:25:29.444452+0000","last_clean":"2024-03-06T07:25:29.444452+0000","last_became_active":"2024-03-06T07:25:29.437589+0000","last_became_peered":"2024-03-06T07:25:29.437589+0000","last_unstale":"2024-03-06T07:25:29.444452+0000","last_undegraded":"2024-03-06T07:25:29.444452+0000","last_fullsized":"2024-03-06T07:25:29.444452+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:34.964198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.428529+0000","last_change":"2024-03-06T07:25:29.428529+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.428529+0000","last_undegraded":"2024-03-06T07:25:29.428529+0000","last_fullsized":"2024-03-06T07:25:29.428529+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.434947+0000","last_change":"2024-03-06T07:25:29.432629+0000","last_active":"2024-03-06T07:25:29.434947+0000","last_peered":"2024-03-06T07:25:29.434947+0000","last_clean":"2024-03-06T07:25:29.434947+0000","last_became_active":"2024-03-06T07:25:29.432348+0000","last_became_peered":"2024-03-06T07:25:29.432348+0000","last_unstale":"2024-03-06T07:25:29.434947+0000","last_undegraded":"2024-03-06T07:25:29.434947+0000","last_fullsized":"2024-03-06T07:25:29.434947+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:43.071021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430039+0000","last_change":"2024-03-06T07:25:29.430039+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430039+0000","last_undegraded":"2024-03-06T07:25:29.430039+0000","last_fullsized":"2024-03-06T07:25:29.430039+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.444491+0000","last_change":"2024-03-06T07:25:29.439680+0000","last_active":"2024-03-06T07:25:29.444491+0000","last_peered":"2024-03-06T07:25:29.444491+0000","last_clean":"2024-03-06T07:25:29.444491+0000","last_became_active":"2024-03-06T07:25:29.439401+0000","last_became_peered":"2024-03-06T07:25:29.439401+0000","last_unstale":"2024-03-06T07:25:29.444491+0000","last_undegraded":"2024-03-06T07:25:29.444491+0000","last_fullsized":"2024-03-06T07:25:29.444491+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:59.559815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.424099+0000","last_change":"2024-03-06T07:25:29.424099+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.424099+0000","last_undegraded":"2024-03-06T07:25:29.424099+0000","last_fullsized":"2024-03-06T07:25:29.424099+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.448969+0000","last_change":"2024-03-06T07:25:29.445124+0000","last_active":"2024-03-06T07:25:29.448969+0000","last_peered":"2024-03-06T07:25:29.448969+0000","last_clean":"2024-03-06T07:25:29.448969+0000","last_became_active":"2024-03-06T07:25:29.444870+0000","last_became_peered":"2024-03-06T07:25:29.444870+0000","last_unstale":"2024-03-06T07:25:29.448969+0000","last_undegraded":"2024-03-06T07:25:29.448969+0000","last_fullsized":"2024-03-06T07:25:29.448969+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:36.677607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.433944+0000","last_change":"2024-03-06T07:25:29.433944+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.433944+0000","last_undegraded":"2024-03-06T07:25:29.433944+0000","last_fullsized":"2024-03-06T07:25:29.433944+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.449041+0000","last_change":"2024-03-06T07:25:29.442634+0000","last_active":"2024-03-06T07:25:29.449041+0000","last_peered":"2024-03-06T07:25:29.449041+0000","last_clean":"2024-03-06T07:25:29.449041+0000","last_became_active":"2024-03-06T07:25:29.442362+0000","last_became_peered":"2024-03-06T07:25:29.442362+0000","last_unstale":"2024-03-06T07:25:29.449041+0000","last_undegraded":"2024-03-06T07:25:29.449041+0000","last_fullsized":"2024-03-06T07:25:29.449041+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:09.549229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.430682+0000","last_change":"2024-03-06T07:25:29.430682+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.430682+0000","last_undegraded":"2024-03-06T07:25:29.430682+0000","last_fullsized":"2024-03-06T07:25:29.430682+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.441524+0000","last_change":"2024-03-06T07:25:29.440980+0000","last_active":"2024-03-06T07:25:29.441524+0000","last_peered":"2024-03-06T07:25:29.441524+0000","last_clean":"2024-03-06T07:25:29.441524+0000","last_became_active":"2024-03-06T07:25:29.440645+0000","last_became_peered":"2024-03-06T07:25:29.440645+0000","last_unstale":"2024-03-06T07:25:29.441524+0000","last_undegraded":"2024-03-06T07:25:29.441524+0000","last_fullsized":"2024-03-06T07:25:29.441524+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:30.511914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.423291+0000","last_change":"2024-03-06T07:25:29.423291+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.423291+0000","last_undegraded":"2024-03-06T07:25:29.423291+0000","last_fullsized":"2024-03-06T07:25:29.423291+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.439976+0000","last_change":"2024-03-06T07:25:29.439438+0000","last_active":"2024-03-06T07:25:29.439976+0000","last_peered":"2024-03-06T07:25:29.439976+0000","last_clean":"2024-03-06T07:25:29.439976+0000","last_became_active":"2024-03-06T07:25:29.439129+0000","last_became_peered":"2024-03-06T07:25:29.439129+0000","last_unstale":"2024-03-06T07:25:29.439976+0000","last_undegraded":"2024-03-06T07:25:29.439976+0000","last_fullsized":"2024-03-06T07:25:29.439976+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:37.162898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.425093+0000","last_change":"2024-03-06T07:25:29.425093+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.425093+0000","last_undegraded":"2024-03-06T07:25:29.425093+0000","last_fullsized":"2024-03-06T07:25:29.425093+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442801+0000","last_change":"2024-03-06T07:25:29.439263+0000","last_active":"2024-03-06T07:25:29.442801+0000","last_peered":"2024-03-06T07:25:29.442801+0000","last_clean":"2024-03-06T07:25:29.442801+0000","last_became_active":"2024-03-06T07:25:29.438961+0000","last_became_peered":"2024-03-06T07:25:29.438961+0000","last_unstale":"2024-03-06T07:25:29.442801+0000","last_undegraded":"2024-03-06T07:25:29.442801+0000","last_fullsized":"2024-03-06T07:25:29.442801+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:34.084278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431354+0000","last_change":"2024-03-06T07:25:29.431354+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431354+0000","last_undegraded":"2024-03-06T07:25:29.431354+0000","last_fullsized":"2024-03-06T07:25:29.431354+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442824+0000","last_change":"2024-03-06T07:25:29.442215+0000","last_active":"2024-03-06T07:25:29.442824+0000","last_peered":"2024-03-06T07:25:29.442824+0000","last_clean":"2024-03-06T07:25:29.442824+0000","last_became_active":"2024-03-06T07:25:29.441739+0000","last_became_peered":"2024-03-06T07:25:29.441739+0000","last_unstale":"2024-03-06T07:25:29.442824+0000","last_undegraded":"2024-03-06T07:25:29.442824+0000","last_fullsized":"2024-03-06T07:25:29.442824+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:21.032029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.431388+0000","last_change":"2024-03-06T07:25:29.431388+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.431388+0000","last_undegraded":"2024-03-06T07:25:29.431388+0000","last_fullsized":"2024-03-06T07:25:29.431388+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442516+0000","last_change":"2024-03-06T07:25:29.437548+0000","last_active":"2024-03-06T07:25:29.442516+0000","last_peered":"2024-03-06T07:25:29.442516+0000","last_clean":"2024-03-06T07:25:29.442516+0000","last_became_active":"2024-03-06T07:25:29.437281+0000","last_became_peered":"2024-03-06T07:25:29.437281+0000","last_unstale":"2024-03-06T07:25:29.442516+0000","last_undegraded":"2024-03-06T07:25:29.442516+0000","last_fullsized":"2024-03-06T07:25:29.442516+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:18.466217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.426909+0000","last_change":"2024-03-06T07:25:29.426909+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.426909+0000","last_undegraded":"2024-03-06T07:25:29.426909+0000","last_fullsized":"2024-03-06T07:25:29.426909+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.442173+0000","last_change":"2024-03-06T07:25:29.437129+0000","last_active":"2024-03-06T07:25:29.442173+0000","last_peered":"2024-03-06T07:25:29.442173+0000","last_clean":"2024-03-06T07:25:29.442173+0000","last_became_active":"2024-03-06T07:25:29.436825+0000","last_became_peered":"2024-03-06T07:25:29.436825+0000","last_unstale":"2024-03-06T07:25:29.442173+0000","last_undegraded":"2024-03-06T07:25:29.442173+0000","last_fullsized":"2024-03-06T07:25:29.442173+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:28.307200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-03-06T07:25:29.425972+0000","last_change":"2024-03-06T07:25:29.425972+0000","last_active":"2024-03-06T07:25:29.415355+0000","last_peered":"2024-03-06T07:25:29.415355+0000","last_clean":"2024-03-06T07:25:29.415355+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:29.425972+0000","last_undegraded":"2024-03-06T07:25:29.425972+0000","last_fullsized":"2024-03-06T07:25:29.425972+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":9,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.441092+0000","last_change":"2024-03-06T07:25:29.440579+0000","last_active":"2024-03-06T07:25:29.441092+0000","last_peered":"2024-03-06T07:25:29.441092+0000","last_clean":"2024-03-06T07:25:29.441092+0000","last_became_active":"2024-03-06T07:25:29.440247+0000","last_became_peered":"2024-03-06T07:25:29.440247+0000","last_unstale":"2024-03-06T07:25:29.441092+0000","last_undegraded":"2024-03-06T07:25:29.441092+0000","last_fullsized":"2024-03-06T07:25:29.441092+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:13.223054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":385,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-06T07:25:29.428367+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:25:29.428367+0000","last_peered":"2024-03-06T07:25:29.428367+0000","last_clean":"2024-03-06T07:25:29.428367+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:25:29.428367+0000","last_undegraded":"2024-03-06T07:25:29.428367+0000","last_fullsized":"2024-03-06T07:25:29.428367+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32868,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338972,"statfs":{"total":96636764160,"available":96603107328,"internally_reserved":0,"allocated":1724416,"data_stored":1388223,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32812,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339028,"statfs":{"total":96636764160,"available":96603164672,"internally_reserved":0,"allocated":1671168,"data_stored":804522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574913,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32808,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339032,"statfs":{"total":96636764160,"available":96603168768,"internally_reserved":0,"allocated":1658880,"data_stored":797855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32808,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339032,"statfs":{"total":96636764160,"available":96603168768,"internally_reserved":0,"allocated":1658880,"data_stored":797855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32808,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339032,"statfs":{"total":96636764160,"available":96603168768,"internally_reserved":0,"allocated":1658880,"data_stored":797855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574913,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32800,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339040,"statfs":{"total":96636764160,"available":96603176960,"internally_reserved":0,"allocated":1658880,"data_stored":797855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32812,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339028,"statfs":{"total":96636764160,"available":96603164672,"internally_reserved":0,"allocated":1671168,"data_stored":804522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574914,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32872,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338968,"statfs":{"total":96636764160,"available":96603103232,"internally_reserved":0,"allocated":1724416,"data_stored":1388223,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:34.277 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.3f is not active+clean 2024-03-06T07:25:34.277 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.429803+0000', 'last_change': '2024-03-06T07:25:29.429803+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.429803+0000', 'last_undegraded': '2024-03-06T07:25:29.429803+0000', 'last_fullsized': '2024-03-06T07:25:29.429803+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:34.277 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.3e is not active+clean 2024-03-06T07:25:34.277 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.429415+0000', 'last_change': '2024-03-06T07:25:29.429415+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.429415+0000', 'last_undegraded': '2024-03-06T07:25:29.429415+0000', 'last_fullsized': '2024-03-06T07:25:29.429415+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:34.277 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.3d is not active+clean 2024-03-06T07:25:34.277 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.432950+0000', 'last_change': '2024-03-06T07:25:29.432950+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.432950+0000', 'last_undegraded': '2024-03-06T07:25:29.432950+0000', 'last_fullsized': '2024-03-06T07:25:29.432950+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:34.277 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.3c is not active+clean 2024-03-06T07:25:34.277 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.434735+0000', 'last_change': '2024-03-06T07:25:29.434735+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.434735+0000', 'last_undegraded': '2024-03-06T07:25:29.434735+0000', 'last_fullsized': '2024-03-06T07:25:29.434735+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:34.277 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.3b is not active+clean 2024-03-06T07:25:34.277 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.423617+0000', 'last_change': '2024-03-06T07:25:29.423617+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.423617+0000', 'last_undegraded': '2024-03-06T07:25:29.423617+0000', 'last_fullsized': '2024-03-06T07:25:29.423617+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:34.277 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.3a is not active+clean 2024-03-06T07:25:34.278 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.423430+0000', 'last_change': '2024-03-06T07:25:29.423430+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.423430+0000', 'last_undegraded': '2024-03-06T07:25:29.423430+0000', 'last_fullsized': '2024-03-06T07:25:29.423430+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:34.278 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.39 is not active+clean 2024-03-06T07:25:34.278 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.430543+0000', 'last_change': '2024-03-06T07:25:29.430543+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.430543+0000', 'last_undegraded': '2024-03-06T07:25:29.430543+0000', 'last_fullsized': '2024-03-06T07:25:29.430543+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:34.278 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.38 is not active+clean 2024-03-06T07:25:34.278 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.428243+0000', 'last_change': '2024-03-06T07:25:29.428243+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.428243+0000', 'last_undegraded': '2024-03-06T07:25:29.428243+0000', 'last_fullsized': '2024-03-06T07:25:29.428243+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:34.278 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.37 is not active+clean 2024-03-06T07:25:34.278 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.423943+0000', 'last_change': '2024-03-06T07:25:29.423943+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.423943+0000', 'last_undegraded': '2024-03-06T07:25:29.423943+0000', 'last_fullsized': '2024-03-06T07:25:29.423943+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:34.278 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.36 is not active+clean 2024-03-06T07:25:34.278 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.423699+0000', 'last_change': '2024-03-06T07:25:29.423699+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.423699+0000', 'last_undegraded': '2024-03-06T07:25:29.423699+0000', 'last_fullsized': '2024-03-06T07:25:29.423699+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:34.278 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.35 is not active+clean 2024-03-06T07:25:34.278 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.427685+0000', 'last_change': '2024-03-06T07:25:29.427685+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.427685+0000', 'last_undegraded': '2024-03-06T07:25:29.427685+0000', 'last_fullsized': '2024-03-06T07:25:29.427685+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:34.279 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.34 is not active+clean 2024-03-06T07:25:34.279 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.433682+0000', 'last_change': '2024-03-06T07:25:29.433682+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.433682+0000', 'last_undegraded': '2024-03-06T07:25:29.433682+0000', 'last_fullsized': '2024-03-06T07:25:29.433682+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:34.279 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.33 is not active+clean 2024-03-06T07:25:34.279 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.424516+0000', 'last_change': '2024-03-06T07:25:29.424516+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.424516+0000', 'last_undegraded': '2024-03-06T07:25:29.424516+0000', 'last_fullsized': '2024-03-06T07:25:29.424516+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:34.279 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.32 is not active+clean 2024-03-06T07:25:34.279 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.433176+0000', 'last_change': '2024-03-06T07:25:29.433176+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.433176+0000', 'last_undegraded': '2024-03-06T07:25:29.433176+0000', 'last_fullsized': '2024-03-06T07:25:29.433176+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:34.279 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.31 is not active+clean 2024-03-06T07:25:34.279 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.432452+0000', 'last_change': '2024-03-06T07:25:29.432452+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.432452+0000', 'last_undegraded': '2024-03-06T07:25:29.432452+0000', 'last_fullsized': '2024-03-06T07:25:29.432452+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:34.279 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.30 is not active+clean 2024-03-06T07:25:34.279 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.423245+0000', 'last_change': '2024-03-06T07:25:29.423245+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.423245+0000', 'last_undegraded': '2024-03-06T07:25:29.423245+0000', 'last_fullsized': '2024-03-06T07:25:29.423245+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:34.279 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.2f is not active+clean 2024-03-06T07:25:34.280 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.434177+0000', 'last_change': '2024-03-06T07:25:29.434177+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.434177+0000', 'last_undegraded': '2024-03-06T07:25:29.434177+0000', 'last_fullsized': '2024-03-06T07:25:29.434177+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:34.280 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.2e is not active+clean 2024-03-06T07:25:34.280 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.428279+0000', 'last_change': '2024-03-06T07:25:29.428279+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.428279+0000', 'last_undegraded': '2024-03-06T07:25:29.428279+0000', 'last_fullsized': '2024-03-06T07:25:29.428279+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:34.280 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.2d is not active+clean 2024-03-06T07:25:34.280 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.429606+0000', 'last_change': '2024-03-06T07:25:29.429606+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.429606+0000', 'last_undegraded': '2024-03-06T07:25:29.429606+0000', 'last_fullsized': '2024-03-06T07:25:29.429606+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:34.280 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.2c is not active+clean 2024-03-06T07:25:34.280 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.427157+0000', 'last_change': '2024-03-06T07:25:29.427157+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.427157+0000', 'last_undegraded': '2024-03-06T07:25:29.427157+0000', 'last_fullsized': '2024-03-06T07:25:29.427157+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:34.280 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.2b is not active+clean 2024-03-06T07:25:34.280 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.424094+0000', 'last_change': '2024-03-06T07:25:29.424094+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.424094+0000', 'last_undegraded': '2024-03-06T07:25:29.424094+0000', 'last_fullsized': '2024-03-06T07:25:29.424094+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:34.280 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.2a is not active+clean 2024-03-06T07:25:34.281 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.427504+0000', 'last_change': '2024-03-06T07:25:29.427504+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.427504+0000', 'last_undegraded': '2024-03-06T07:25:29.427504+0000', 'last_fullsized': '2024-03-06T07:25:29.427504+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:34.281 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.29 is not active+clean 2024-03-06T07:25:34.281 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.426134+0000', 'last_change': '2024-03-06T07:25:29.426134+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.426134+0000', 'last_undegraded': '2024-03-06T07:25:29.426134+0000', 'last_fullsized': '2024-03-06T07:25:29.426134+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:34.281 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.28 is not active+clean 2024-03-06T07:25:34.281 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.426814+0000', 'last_change': '2024-03-06T07:25:29.426814+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.426814+0000', 'last_undegraded': '2024-03-06T07:25:29.426814+0000', 'last_fullsized': '2024-03-06T07:25:29.426814+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:34.281 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.27 is not active+clean 2024-03-06T07:25:34.281 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.429388+0000', 'last_change': '2024-03-06T07:25:29.429388+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.429388+0000', 'last_undegraded': '2024-03-06T07:25:29.429388+0000', 'last_fullsized': '2024-03-06T07:25:29.429388+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:34.281 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.26 is not active+clean 2024-03-06T07:25:34.281 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.431459+0000', 'last_change': '2024-03-06T07:25:29.431459+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.431459+0000', 'last_undegraded': '2024-03-06T07:25:29.431459+0000', 'last_fullsized': '2024-03-06T07:25:29.431459+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:34.281 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.25 is not active+clean 2024-03-06T07:25:34.281 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.422691+0000', 'last_change': '2024-03-06T07:25:29.422691+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.422691+0000', 'last_undegraded': '2024-03-06T07:25:29.422691+0000', 'last_fullsized': '2024-03-06T07:25:29.422691+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:34.282 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.24 is not active+clean 2024-03-06T07:25:34.282 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.427342+0000', 'last_change': '2024-03-06T07:25:29.427342+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.427342+0000', 'last_undegraded': '2024-03-06T07:25:29.427342+0000', 'last_fullsized': '2024-03-06T07:25:29.427342+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:34.282 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.23 is not active+clean 2024-03-06T07:25:34.282 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.426341+0000', 'last_change': '2024-03-06T07:25:29.426341+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.426341+0000', 'last_undegraded': '2024-03-06T07:25:29.426341+0000', 'last_fullsized': '2024-03-06T07:25:29.426341+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:34.282 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.22 is not active+clean 2024-03-06T07:25:34.282 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.429494+0000', 'last_change': '2024-03-06T07:25:29.429494+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.429494+0000', 'last_undegraded': '2024-03-06T07:25:29.429494+0000', 'last_fullsized': '2024-03-06T07:25:29.429494+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:34.282 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.21 is not active+clean 2024-03-06T07:25:34.282 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.424039+0000', 'last_change': '2024-03-06T07:25:29.424039+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.424039+0000', 'last_undegraded': '2024-03-06T07:25:29.424039+0000', 'last_fullsized': '2024-03-06T07:25:29.424039+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:34.282 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.20 is not active+clean 2024-03-06T07:25:34.282 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.427794+0000', 'last_change': '2024-03-06T07:25:29.427794+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.427794+0000', 'last_undegraded': '2024-03-06T07:25:29.427794+0000', 'last_fullsized': '2024-03-06T07:25:29.427794+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:34.282 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.1c is not active+clean 2024-03-06T07:25:34.282 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.430468+0000', 'last_change': '2024-03-06T07:25:29.430468+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.430468+0000', 'last_undegraded': '2024-03-06T07:25:29.430468+0000', 'last_fullsized': '2024-03-06T07:25:29.430468+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:34.283 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.1d is not active+clean 2024-03-06T07:25:34.283 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.430796+0000', 'last_change': '2024-03-06T07:25:29.430796+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.430796+0000', 'last_undegraded': '2024-03-06T07:25:29.430796+0000', 'last_fullsized': '2024-03-06T07:25:29.430796+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:34.283 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.1e is not active+clean 2024-03-06T07:25:34.283 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.424888+0000', 'last_change': '2024-03-06T07:25:29.424888+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.424888+0000', 'last_undegraded': '2024-03-06T07:25:29.424888+0000', 'last_fullsized': '2024-03-06T07:25:29.424888+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:34.283 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.1f is not active+clean 2024-03-06T07:25:34.283 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.427180+0000', 'last_change': '2024-03-06T07:25:29.427180+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.427180+0000', 'last_undegraded': '2024-03-06T07:25:29.427180+0000', 'last_fullsized': '2024-03-06T07:25:29.427180+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:34.283 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.18 is not active+clean 2024-03-06T07:25:34.283 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.426438+0000', 'last_change': '2024-03-06T07:25:29.426438+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.426438+0000', 'last_undegraded': '2024-03-06T07:25:29.426438+0000', 'last_fullsized': '2024-03-06T07:25:29.426438+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:34.283 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.19 is not active+clean 2024-03-06T07:25:34.283 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.431875+0000', 'last_change': '2024-03-06T07:25:29.431875+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.431875+0000', 'last_undegraded': '2024-03-06T07:25:29.431875+0000', 'last_fullsized': '2024-03-06T07:25:29.431875+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:34.283 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.1a is not active+clean 2024-03-06T07:25:34.284 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.431792+0000', 'last_change': '2024-03-06T07:25:29.431792+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.431792+0000', 'last_undegraded': '2024-03-06T07:25:29.431792+0000', 'last_fullsized': '2024-03-06T07:25:29.431792+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:34.284 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.1b is not active+clean 2024-03-06T07:25:34.284 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.428427+0000', 'last_change': '2024-03-06T07:25:29.428427+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.428427+0000', 'last_undegraded': '2024-03-06T07:25:29.428427+0000', 'last_fullsized': '2024-03-06T07:25:29.428427+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:34.284 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.14 is not active+clean 2024-03-06T07:25:34.284 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.427340+0000', 'last_change': '2024-03-06T07:25:29.427340+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.427340+0000', 'last_undegraded': '2024-03-06T07:25:29.427340+0000', 'last_fullsized': '2024-03-06T07:25:29.427340+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:34.284 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.15 is not active+clean 2024-03-06T07:25:34.284 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.427245+0000', 'last_change': '2024-03-06T07:25:29.427245+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.427245+0000', 'last_undegraded': '2024-03-06T07:25:29.427245+0000', 'last_fullsized': '2024-03-06T07:25:29.427245+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:34.284 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.16 is not active+clean 2024-03-06T07:25:34.284 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.427865+0000', 'last_change': '2024-03-06T07:25:29.427865+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.427865+0000', 'last_undegraded': '2024-03-06T07:25:29.427865+0000', 'last_fullsized': '2024-03-06T07:25:29.427865+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:34.284 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.10 is not active+clean 2024-03-06T07:25:34.284 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.427343+0000', 'last_change': '2024-03-06T07:25:29.427343+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.427343+0000', 'last_undegraded': '2024-03-06T07:25:29.427343+0000', 'last_fullsized': '2024-03-06T07:25:29.427343+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:34.285 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.11 is not active+clean 2024-03-06T07:25:34.285 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.432056+0000', 'last_change': '2024-03-06T07:25:29.432056+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.432056+0000', 'last_undegraded': '2024-03-06T07:25:29.432056+0000', 'last_fullsized': '2024-03-06T07:25:29.432056+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:34.285 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.12 is not active+clean 2024-03-06T07:25:34.285 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.422909+0000', 'last_change': '2024-03-06T07:25:29.422909+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.422909+0000', 'last_undegraded': '2024-03-06T07:25:29.422909+0000', 'last_fullsized': '2024-03-06T07:25:29.422909+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:34.285 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.13 is not active+clean 2024-03-06T07:25:34.285 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.430432+0000', 'last_change': '2024-03-06T07:25:29.430432+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.430432+0000', 'last_undegraded': '2024-03-06T07:25:29.430432+0000', 'last_fullsized': '2024-03-06T07:25:29.430432+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:34.285 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.c is not active+clean 2024-03-06T07:25:34.285 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.428129+0000', 'last_change': '2024-03-06T07:25:29.428129+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.428129+0000', 'last_undegraded': '2024-03-06T07:25:29.428129+0000', 'last_fullsized': '2024-03-06T07:25:29.428129+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:34.285 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.d is not active+clean 2024-03-06T07:25:34.285 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.428953+0000', 'last_change': '2024-03-06T07:25:29.428953+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.428953+0000', 'last_undegraded': '2024-03-06T07:25:29.428953+0000', 'last_fullsized': '2024-03-06T07:25:29.428953+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:34.285 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.e is not active+clean 2024-03-06T07:25:34.285 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.431074+0000', 'last_change': '2024-03-06T07:25:29.431074+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.431074+0000', 'last_undegraded': '2024-03-06T07:25:29.431074+0000', 'last_fullsized': '2024-03-06T07:25:29.431074+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:34.286 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.f is not active+clean 2024-03-06T07:25:34.286 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.428958+0000', 'last_change': '2024-03-06T07:25:29.428958+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.428958+0000', 'last_undegraded': '2024-03-06T07:25:29.428958+0000', 'last_fullsized': '2024-03-06T07:25:29.428958+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:34.286 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.8 is not active+clean 2024-03-06T07:25:34.286 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.427796+0000', 'last_change': '2024-03-06T07:25:29.427796+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.427796+0000', 'last_undegraded': '2024-03-06T07:25:29.427796+0000', 'last_fullsized': '2024-03-06T07:25:29.427796+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:34.286 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.9 is not active+clean 2024-03-06T07:25:34.286 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.429265+0000', 'last_change': '2024-03-06T07:25:29.429265+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.429265+0000', 'last_undegraded': '2024-03-06T07:25:29.429265+0000', 'last_fullsized': '2024-03-06T07:25:29.429265+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:34.286 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.a is not active+clean 2024-03-06T07:25:34.286 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.428529+0000', 'last_change': '2024-03-06T07:25:29.428529+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.428529+0000', 'last_undegraded': '2024-03-06T07:25:29.428529+0000', 'last_fullsized': '2024-03-06T07:25:29.428529+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:34.286 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.b is not active+clean 2024-03-06T07:25:34.286 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.430039+0000', 'last_change': '2024-03-06T07:25:29.430039+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.430039+0000', 'last_undegraded': '2024-03-06T07:25:29.430039+0000', 'last_fullsized': '2024-03-06T07:25:29.430039+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:34.286 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.4 is not active+clean 2024-03-06T07:25:34.287 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.424099+0000', 'last_change': '2024-03-06T07:25:29.424099+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.424099+0000', 'last_undegraded': '2024-03-06T07:25:29.424099+0000', 'last_fullsized': '2024-03-06T07:25:29.424099+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:34.287 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.5 is not active+clean 2024-03-06T07:25:34.287 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.433944+0000', 'last_change': '2024-03-06T07:25:29.433944+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.433944+0000', 'last_undegraded': '2024-03-06T07:25:29.433944+0000', 'last_fullsized': '2024-03-06T07:25:29.433944+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:34.287 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.6 is not active+clean 2024-03-06T07:25:34.287 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.430682+0000', 'last_change': '2024-03-06T07:25:29.430682+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.430682+0000', 'last_undegraded': '2024-03-06T07:25:29.430682+0000', 'last_fullsized': '2024-03-06T07:25:29.430682+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:34.287 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.7 is not active+clean 2024-03-06T07:25:34.287 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.423291+0000', 'last_change': '2024-03-06T07:25:29.423291+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.423291+0000', 'last_undegraded': '2024-03-06T07:25:29.423291+0000', 'last_fullsized': '2024-03-06T07:25:29.423291+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:34.287 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.0 is not active+clean 2024-03-06T07:25:34.287 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.425093+0000', 'last_change': '2024-03-06T07:25:29.425093+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.425093+0000', 'last_undegraded': '2024-03-06T07:25:29.425093+0000', 'last_fullsized': '2024-03-06T07:25:29.425093+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:34.287 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.1 is not active+clean 2024-03-06T07:25:34.287 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.431354+0000', 'last_change': '2024-03-06T07:25:29.431354+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.431354+0000', 'last_undegraded': '2024-03-06T07:25:29.431354+0000', 'last_fullsized': '2024-03-06T07:25:29.431354+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:34.288 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.2 is not active+clean 2024-03-06T07:25:34.288 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.431388+0000', 'last_change': '2024-03-06T07:25:29.431388+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.431388+0000', 'last_undegraded': '2024-03-06T07:25:29.431388+0000', 'last_fullsized': '2024-03-06T07:25:29.431388+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:34.288 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.3 is not active+clean 2024-03-06T07:25:34.288 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.426909+0000', 'last_change': '2024-03-06T07:25:29.426909+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.426909+0000', 'last_undegraded': '2024-03-06T07:25:29.426909+0000', 'last_fullsized': '2024-03-06T07:25:29.426909+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:34.288 INFO:tasks.cephfs.filesystem.ceph_manager:PG 22.17 is not active+clean 2024-03-06T07:25:34.288 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '22.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:29.425972+0000', 'last_change': '2024-03-06T07:25:29.425972+0000', 'last_active': '2024-03-06T07:25:29.415355+0000', 'last_peered': '2024-03-06T07:25:29.415355+0000', 'last_clean': '2024-03-06T07:25:29.415355+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:29.425972+0000', 'last_undegraded': '2024-03-06T07:25:29.425972+0000', 'last_fullsized': '2024-03-06T07:25:29.425972+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:29.415355+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:34.288 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:25:34.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.413+0000 7f1fa3e64640 1 Processor -- start 2024-03-06T07:25:34.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.413+0000 7f1fa3e64640 1 -- start start 2024-03-06T07:25:34.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.415+0000 7f1fa3e64640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1f9c14ebe0 0x7f1f9c0c3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.415+0000 7f1fa3e64640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1f9c06b700 0x7f1f9c0c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.415+0000 7f1fa3e64640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f9c06c720 con 0x7f1f9c0c15f0 2024-03-06T07:25:34.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.415+0000 7f1fa3e64640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f9c151e20 con 0x7f1f9c06b700 2024-03-06T07:25:34.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.415+0000 7f1fa3e64640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f9c1517d0 con 0x7f1f9c14ebe0 2024-03-06T07:25:34.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.415+0000 7f1fa2401640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1f9c06b700 0x7f1f9c0c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.415+0000 7f1fa2401640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1f9c06b700 0x7f1f9c0c3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48090/0 (socket says 172.21.15.1:48090) 2024-03-06T07:25:34.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.415+0000 7f1fa2401640 1 -- 172.21.15.1:0/58044748 learned_addr learned my addr 172.21.15.1:0/58044748 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:34.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.415+0000 7f1fa1c00640 1 --2- 172.21.15.1:0/58044748 >> v2:172.21.15.98:3300/0 conn(0x7f1f9c14ebe0 0x7f1f9c0c3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.415+0000 7f1fa2401640 1 -- 172.21.15.1:0/58044748 >> v2:172.21.15.98:3300/0 conn(0x7f1f9c14ebe0 msgr2=0x7f1f9c0c3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.415+0000 7f1fa2401640 1 --2- 172.21.15.1:0/58044748 >> v2:172.21.15.98:3300/0 conn(0x7f1f9c14ebe0 0x7f1f9c0c3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.415+0000 7f1fa2401640 1 -- 172.21.15.1:0/58044748 >> v1:172.21.15.98:6789/0 conn(0x7f1f9c0c15f0 legacy=0x7f1f9c06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.415+0000 7f1fa2401640 1 -- 172.21.15.1:0/58044748 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9c151ab0 con 0x7f1f9c06b700 2024-03-06T07:25:34.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.416+0000 7f1fa2401640 1 --2- 172.21.15.1:0/58044748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f9c06b700 0x7f1f9c0c3690 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0c3c20 tx=0x7f1f9402d090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abbac942b8fdf210 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.416+0000 7f1fa0bfe640 1 -- 172.21.15.1:0/58044748 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9402da30 con 0x7f1f9c06b700 2024-03-06T07:25:34.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.416+0000 7f1fa0bfe640 1 -- 172.21.15.1:0/58044748 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9402dbd0 con 0x7f1f9c06b700 2024-03-06T07:25:34.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.416+0000 7f1fa0bfe640 1 -- 172.21.15.1:0/58044748 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f94008500 con 0x7f1f9c06b700 2024-03-06T07:25:34.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.417+0000 7f1fa3e64640 1 -- 172.21.15.1:0/58044748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f9c06b700 msgr2=0x7f1f9c0c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.417+0000 7f1fa3e64640 1 --2- 172.21.15.1:0/58044748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f9c06b700 0x7f1f9c0c3690 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0c3c20 tx=0x7f1f9402d090 comp rx=0 tx=0).stop 2024-03-06T07:25:34.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.417+0000 7f1fa3e64640 1 -- 172.21.15.1:0/58044748 shutdown_connections 2024-03-06T07:25:34.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.417+0000 7f1fa3e64640 1 --2- 172.21.15.1:0/58044748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f9c06b700 0x7f1f9c0c3690 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.417+0000 7f1fa3e64640 1 --2- 172.21.15.1:0/58044748 >> v2:172.21.15.98:3300/0 conn(0x7f1f9c14ebe0 0x7f1f9c0c3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.417+0000 7f1fa3e64640 1 -- 172.21.15.1:0/58044748 >> 172.21.15.1:0/58044748 conn(0x7f1f9c071cf0 msgr2=0x7f1f9c0720d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.417+0000 7f1fa3e64640 1 -- 172.21.15.1:0/58044748 shutdown_connections 2024-03-06T07:25:34.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.417+0000 7f1fa3e64640 1 -- 172.21.15.1:0/58044748 wait complete. 2024-03-06T07:25:34.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.418+0000 7f1fa3e64640 1 Processor -- start 2024-03-06T07:25:34.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.418+0000 7f1fa3e64640 1 -- start start 2024-03-06T07:25:34.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa3e64640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f9c06b700 0x7f1f9c1ec840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa3e64640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f9c0c15f0 0x7f1f9c1504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa3e64640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1f9c14ebe0 0x7f1f9c1e8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa3e64640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c1522b0 con 0x7f1f9c0c15f0 2024-03-06T07:25:34.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa3e64640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c0c2be0 con 0x7f1f9c06b700 2024-03-06T07:25:34.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa3e64640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c1ecd80 con 0x7f1f9c14ebe0 2024-03-06T07:25:34.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa13ff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f9c0c15f0 0x7f1f9c1504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa13ff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f9c0c15f0 0x7f1f9c1504e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48104/0 (socket says 172.21.15.1:48104) 2024-03-06T07:25:34.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa13ff640 1 -- 172.21.15.1:0/1085506388 learned_addr learned my addr 172.21.15.1:0/1085506388 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:34.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa1c00640 1 --2- 172.21.15.1:0/1085506388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f9c06b700 0x7f1f9c1ec840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa2401640 1 --2- 172.21.15.1:0/1085506388 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1f9c14ebe0 0x7f1f9c1e8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa13ff640 1 -- 172.21.15.1:0/1085506388 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1f9c14ebe0 msgr2=0x7f1f9c1e8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa13ff640 1 --2- 172.21.15.1:0/1085506388 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1f9c14ebe0 0x7f1f9c1e8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa13ff640 1 -- 172.21.15.1:0/1085506388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f9c06b700 msgr2=0x7f1f9c1ec840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa13ff640 1 --2- 172.21.15.1:0/1085506388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f9c06b700 0x7f1f9c1ec840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa13ff640 1 -- 172.21.15.1:0/1085506388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9c0c67d0 con 0x7f1f9c0c15f0 2024-03-06T07:25:34.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa2401640 1 --2- 172.21.15.1:0/1085506388 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1f9c14ebe0 0x7f1f9c1e8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:34.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.419+0000 7f1fa13ff640 1 --2- 172.21.15.1:0/1085506388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f9c0c15f0 0x7f1f9c1504e0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c00c3c0 tx=0x7f1f8c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.420+0000 7f1f927fc640 1 -- 172.21.15.1:0/1085506388 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c0052d0 con 0x7f1f9c0c15f0 2024-03-06T07:25:34.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.420+0000 7f1f927fc640 1 -- 172.21.15.1:0/1085506388 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c005490 con 0x7f1f9c0c15f0 2024-03-06T07:25:34.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.420+0000 7f1fa3e64640 1 -- 172.21.15.1:0/1085506388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f9c0c5f30 con 0x7f1f9c0c15f0 2024-03-06T07:25:34.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.420+0000 7f1f927fc640 1 -- 172.21.15.1:0/1085506388 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c007260 con 0x7f1f9c0c15f0 2024-03-06T07:25:34.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.420+0000 7f1fa3e64640 1 -- 172.21.15.1:0/1085506388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9c0c64f0 con 0x7f1f9c0c15f0 2024-03-06T07:25:34.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.421+0000 7f1fa3e64640 1 -- 172.21.15.1:0/1085506388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f9c0686e0 con 0x7f1f9c0c15f0 2024-03-06T07:25:34.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.421+0000 7f1f927fc640 1 -- 172.21.15.1:0/1085506388 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f8c007400 con 0x7f1f9c0c15f0 2024-03-06T07:25:34.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.422+0000 7f1f927fc640 1 --2- 172.21.15.1:0/1085506388 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1f8006b4d0 0x7f1f8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.422+0000 7f1f927fc640 1 -- 172.21.15.1:0/1085506388 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1f8c0897e0 con 0x7f1f9c0c15f0 2024-03-06T07:25:34.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.422+0000 7f1fa1c00640 1 --2- 172.21.15.1:0/1085506388 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1f8006b4d0 0x7f1f8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.422+0000 7f1fa1c00640 1 --2- 172.21.15.1:0/1085506388 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1f8006b4d0 0x7f1f8006d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1f98009ad0 tx=0x7f1f98008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.430+0000 7f1f927fc640 1 -- 172.21.15.1:0/1085506388 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f8c058560 con 0x7f1f9c0c15f0 2024-03-06T07:25:34.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.681+0000 7f1fa3e64640 1 -- 172.21.15.1:0/1085506388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f1f9c152430 con 0x7f1f9c0c15f0 2024-03-06T07:25:34.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.681+0000 7f1f927fc640 1 -- 172.21.15.1:0/1085506388 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1329 (secure 0 0 0) 0x7f1f8c057e60 con 0x7f1f9c0c15f0 2024-03-06T07:25:34.683 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:34.684 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":330,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":135,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709614,"num_in_osds":8,"osd_in_since":1709709606,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":268890112,"bytes_avail":772825223168,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":121,"id":10,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":10,"rank":0,"name":"a","status":"up:active","gid":8301}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-03-06T07:24:40.749477+0000","services":{"mds":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:25:34.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.683+0000 7f1fa3e64640 1 -- 172.21.15.1:0/1085506388 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1f8006b4d0 msgr2=0x7f1f8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.683+0000 7f1fa3e64640 1 --2- 172.21.15.1:0/1085506388 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1f8006b4d0 0x7f1f8006d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1f98009ad0 tx=0x7f1f98008040 comp rx=0 tx=0).stop 2024-03-06T07:25:34.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.683+0000 7f1fa3e64640 1 -- 172.21.15.1:0/1085506388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f9c0c15f0 msgr2=0x7f1f9c1504e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.683+0000 7f1fa3e64640 1 --2- 172.21.15.1:0/1085506388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f9c0c15f0 0x7f1f9c1504e0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c00c3c0 tx=0x7f1f8c00c880 comp rx=0 tx=0).stop 2024-03-06T07:25:34.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.683+0000 7f1fa3e64640 1 -- 172.21.15.1:0/1085506388 shutdown_connections 2024-03-06T07:25:34.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.683+0000 7f1fa3e64640 1 --2- 172.21.15.1:0/1085506388 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1f8006b4d0 0x7f1f8006d990 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.683+0000 7f1fa3e64640 1 --2- 172.21.15.1:0/1085506388 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1f9c14ebe0 0x7f1f9c1e8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.683+0000 7f1fa3e64640 1 --2- 172.21.15.1:0/1085506388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f9c0c15f0 0x7f1f9c1504e0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.683+0000 7f1fa3e64640 1 --2- 172.21.15.1:0/1085506388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f9c06b700 0x7f1f9c1ec840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.683+0000 7f1fa3e64640 1 -- 172.21.15.1:0/1085506388 >> 172.21.15.1:0/1085506388 conn(0x7f1f9c071cf0 msgr2=0x7f1f9c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.683+0000 7f1fa3e64640 1 -- 172.21.15.1:0/1085506388 shutdown_connections 2024-03-06T07:25:34.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.683+0000 7f1fa3e64640 1 -- 172.21.15.1:0/1085506388 wait complete. 2024-03-06T07:25:34.697 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:25:34.697 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.796+0000 7fa6ea6c2640 1 Processor -- start 2024-03-06T07:25:34.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.796+0000 7fa6ea6c2640 1 -- start start 2024-03-06T07:25:34.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.797+0000 7fa6ea6c2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6e40c6b10 0x7fa6e4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.797+0000 7fa6ea6c2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e40c2fc0 con 0x7fa6e40bf9d0 2024-03-06T07:25:34.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.797+0000 7fa6ea6c2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e4158da0 con 0x7fa6e4152650 2024-03-06T07:25:34.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.797+0000 7fa6ea6c2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e4155800 con 0x7fa6e40c6b10 2024-03-06T07:25:34.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.797+0000 7fa6e8c5f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa6e40bf9d0 0x7fa6e40c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39950/0 (socket says 172.21.15.1:39950) 2024-03-06T07:25:34.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.797+0000 7fa6e8c5f640 1 -- 172.21.15.1:0/3516241620 learned_addr learned my addr 172.21.15.1:0/3516241620 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:34.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.797+0000 7fa6e37fe640 1 --2- 172.21.15.1:0/3516241620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6e40c6b10 0x7fa6e4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e2ffd640 1 -- 172.21.15.1:0/3516241620 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753786964 0 0) 0x7fa6e40c2fc0 con 0x7fa6e40bf9d0 2024-03-06T07:25:34.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e2ffd640 1 -- 172.21.15.1:0/3516241620 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6cc003520 con 0x7fa6e40bf9d0 2024-03-06T07:25:34.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e2ffd640 1 -- 172.21.15.1:0/3516241620 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647012174 0 0) 0x7fa6e4158da0 con 0x7fa6e4152650 2024-03-06T07:25:34.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e2ffd640 1 -- 172.21.15.1:0/3516241620 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6e40c2fc0 con 0x7fa6e4152650 2024-03-06T07:25:34.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e2ffd640 1 -- 172.21.15.1:0/3516241620 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3245851762 0 0) 0x7fa6cc003520 con 0x7fa6e40bf9d0 2024-03-06T07:25:34.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e2ffd640 1 -- 172.21.15.1:0/3516241620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6e40c6b10 msgr2=0x7fa6e4155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e2ffd640 1 --2- 172.21.15.1:0/3516241620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6e40c6b10 0x7fa6e4155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e2ffd640 1 -- 172.21.15.1:0/3516241620 >> v1:172.21.15.98:6789/0 conn(0x7fa6e4152650 legacy=0x7fa6e4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e2ffd640 1 -- 172.21.15.1:0/3516241620 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e40c6690 con 0x7fa6e40bf9d0 2024-03-06T07:25:34.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e2ffd640 1 -- 172.21.15.1:0/3516241620 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fa6dc0019f0 con 0x7fa6e40bf9d0 2024-03-06T07:25:34.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e2ffd640 1 -- 172.21.15.1:0/3516241620 >> v1:172.21.15.1:6789/0 conn(0x7fa6e40bf9d0 legacy=0x7fa6e40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e2ffd640 1 --2- 172.21.15.1:0/3516241620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6cc003b70 0x7fa6cc003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e2ffd640 1 --2- 172.21.15.1:0/3516241620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6cc004800 0x7fa6cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e2ffd640 1 --2- 172.21.15.1:0/3516241620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6cc0051c0 0x7fa6cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e2ffd640 1 -- 172.21.15.1:0/3516241620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e4155800 con 0x7fa6cc0051c0 2024-03-06T07:25:34.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e2ffd640 1 -- 172.21.15.1:0/3516241620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc00bf30 con 0x7fa6cc004800 2024-03-06T07:25:34.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e2ffd640 1 -- 172.21.15.1:0/3516241620 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc00c130 con 0x7fa6cc003b70 2024-03-06T07:25:34.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e8c5f640 1 --2- 172.21.15.1:0/3516241620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6cc0051c0 0x7fa6cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e3fff640 1 --2- 172.21.15.1:0/3516241620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6cc003b70 0x7fa6cc003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e37fe640 1 --2- 172.21.15.1:0/3516241620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6cc004800 0x7fa6cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e8c5f640 1 -- 172.21.15.1:0/3516241620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6cc003b70 msgr2=0x7fa6cc003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e8c5f640 1 --2- 172.21.15.1:0/3516241620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6cc003b70 0x7fa6cc003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e8c5f640 1 -- 172.21.15.1:0/3516241620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6cc004800 msgr2=0x7fa6cc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e8c5f640 1 --2- 172.21.15.1:0/3516241620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6cc004800 0x7fa6cc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.798+0000 7fa6e8c5f640 1 -- 172.21.15.1:0/3516241620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc003520 con 0x7fa6cc0051c0 2024-03-06T07:25:34.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.799+0000 7fa6e3fff640 1 --2- 172.21.15.1:0/3516241620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6cc003b70 0x7fa6cc003f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:34.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.800+0000 7fa6e8c5f640 1 --2- 172.21.15.1:0/3516241620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6cc0051c0 0x7fa6cc00b9c0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00a7c0 tx=0x7fa6cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a985549e3ada0abf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.800+0000 7fa6e2ffd640 1 -- 172.21.15.1:0/3516241620 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc00a410 con 0x7fa6cc0051c0 2024-03-06T07:25:34.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.800+0000 7fa6e2ffd640 1 -- 172.21.15.1:0/3516241620 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6dc034650 con 0x7fa6cc0051c0 2024-03-06T07:25:34.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.800+0000 7fa6e2ffd640 1 -- 172.21.15.1:0/3516241620 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc034a80 con 0x7fa6cc0051c0 2024-03-06T07:25:34.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.800+0000 7fa6ea6c2640 1 -- 172.21.15.1:0/3516241620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6cc0051c0 msgr2=0x7fa6cc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.800+0000 7fa6ea6c2640 1 --2- 172.21.15.1:0/3516241620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6cc0051c0 0x7fa6cc00b9c0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00a7c0 tx=0x7fa6cc005830 comp rx=0 tx=0).stop 2024-03-06T07:25:34.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.800+0000 7fa6e8c5f640 1 -- 172.21.15.1:0/3516241620 reap_dead start 2024-03-06T07:25:34.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.800+0000 7fa6ea6c2640 1 -- 172.21.15.1:0/3516241620 shutdown_connections 2024-03-06T07:25:34.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.800+0000 7fa6ea6c2640 1 -- 172.21.15.1:0/3516241620 >> 172.21.15.1:0/3516241620 conn(0x7fa6e4066ee0 msgr2=0x7fa6e40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.800+0000 7fa6ea6c2640 1 -- 172.21.15.1:0/3516241620 shutdown_connections 2024-03-06T07:25:34.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.800+0000 7fa6ea6c2640 1 -- 172.21.15.1:0/3516241620 wait complete. 2024-03-06T07:25:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.801+0000 7fa6ea6c2640 1 Processor -- start 2024-03-06T07:25:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.802+0000 7fa6ea6c2640 1 -- start start 2024-03-06T07:25:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.802+0000 7fa6ea6c2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6e41509b0 0x7fa6e41eb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.802+0000 7fa6ea6c2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6e414d8e0 0x7fa6e41eec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.802+0000 7fa6ea6c2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e41de700 0x7fa6e414cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.802+0000 7fa6ea6c2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e41551b0 con 0x7fa6e414d8e0 2024-03-06T07:25:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.802+0000 7fa6ea6c2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e40c6fa0 con 0x7fa6e41de700 2024-03-06T07:25:34.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.802+0000 7fa6ea6c2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e4155490 con 0x7fa6e41509b0 2024-03-06T07:25:34.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.802+0000 7fa6e37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6e414d8e0 0x7fa6e41eec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.802+0000 7fa6e37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6e414d8e0 0x7fa6e41eec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48124/0 (socket says 172.21.15.1:48124) 2024-03-06T07:25:34.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.802+0000 7fa6e37fe640 1 -- 172.21.15.1:0/3472934351 learned_addr learned my addr 172.21.15.1:0/3472934351 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:34.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.803+0000 7fa6e8c5f640 1 --2- 172.21.15.1:0/3472934351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e41de700 0x7fa6e414cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.803+0000 7fa6e37fe640 1 -- 172.21.15.1:0/3472934351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6e41509b0 msgr2=0x7fa6e41eb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.803+0000 7fa6e3fff640 1 --2- 172.21.15.1:0/3472934351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6e41509b0 0x7fa6e41eb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.803+0000 7fa6e37fe640 1 --2- 172.21.15.1:0/3472934351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6e41509b0 0x7fa6e41eb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.803+0000 7fa6e37fe640 1 -- 172.21.15.1:0/3472934351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e41de700 msgr2=0x7fa6e414cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.803+0000 7fa6e37fe640 1 --2- 172.21.15.1:0/3472934351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e41de700 0x7fa6e414cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.803+0000 7fa6e37fe640 1 -- 172.21.15.1:0/3472934351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e41eadf0 con 0x7fa6e414d8e0 2024-03-06T07:25:34.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.803+0000 7fa6e8c5f640 1 --2- 172.21.15.1:0/3472934351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e41de700 0x7fa6e414cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:34.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.803+0000 7fa6e37fe640 1 --2- 172.21.15.1:0/3472934351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6e414d8e0 0x7fa6e41eec30 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fa6d80119a0 tx=0x7fa6d8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.803+0000 7fa6e0ff9640 1 -- 172.21.15.1:0/3472934351 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d800fa90 con 0x7fa6e414d8e0 2024-03-06T07:25:34.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.803+0000 7fa6e0ff9640 1 -- 172.21.15.1:0/3472934351 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d800fc30 con 0x7fa6e414d8e0 2024-03-06T07:25:34.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.803+0000 7fa6ea6c2640 1 -- 172.21.15.1:0/3472934351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e4151b00 con 0x7fa6e414d8e0 2024-03-06T07:25:34.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.803+0000 7fa6e0ff9640 1 -- 172.21.15.1:0/3472934351 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8014440 con 0x7fa6e414d8e0 2024-03-06T07:25:34.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.803+0000 7fa6ea6c2640 1 -- 172.21.15.1:0/3472934351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e4151de0 con 0x7fa6e414d8e0 2024-03-06T07:25:34.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.804+0000 7fa6ea6c2640 1 -- 172.21.15.1:0/3472934351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e40c0570 con 0x7fa6e414d8e0 2024-03-06T07:25:34.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.805+0000 7fa6e0ff9640 1 -- 172.21.15.1:0/3472934351 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6d8012040 con 0x7fa6e414d8e0 2024-03-06T07:25:34.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.805+0000 7fa6e0ff9640 1 --2- 172.21.15.1:0/3472934351 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa6d006b500 0x7fa6d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.805+0000 7fa6e0ff9640 1 -- 172.21.15.1:0/3472934351 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa6d808a970 con 0x7fa6e414d8e0 2024-03-06T07:25:34.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.805+0000 7fa6e3fff640 1 --2- 172.21.15.1:0/3472934351 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa6d006b500 0x7fa6d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.806+0000 7fa6e3fff640 1 --2- 172.21.15.1:0/3472934351 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa6d006b500 0x7fa6d006d9c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4005410 tx=0x7fa6d4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:34.809+0000 7fa6e0ff9640 1 -- 172.21.15.1:0/3472934351 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6e40c0570 con 0x7fa6e414d8e0 2024-03-06T07:25:35.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:35.142+0000 7fa6ea6c2640 1 -- 172.21.15.1:0/3472934351 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa6e41518b0 con 0x7fa6d006b500 2024-03-06T07:25:35.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:35.148+0000 7fa6e0ff9640 1 -- 172.21.15.1:0/3472934351 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252563 (secure 0 0 0) 0x7fa6e41518b0 con 0x7fa6d006b500 2024-03-06T07:25:35.153 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:35.154 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:25:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:35.152+0000 7fa6ea6c2640 1 -- 172.21.15.1:0/3472934351 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa6d006b500 msgr2=0x7fa6d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:35.152+0000 7fa6ea6c2640 1 --2- 172.21.15.1:0/3472934351 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa6d006b500 0x7fa6d006d9c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4005410 tx=0x7fa6d4009040 comp rx=0 tx=0).stop 2024-03-06T07:25:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:35.152+0000 7fa6ea6c2640 1 -- 172.21.15.1:0/3472934351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6e414d8e0 msgr2=0x7fa6e41eec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:35.152+0000 7fa6ea6c2640 1 --2- 172.21.15.1:0/3472934351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6e414d8e0 0x7fa6e41eec30 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fa6d80119a0 tx=0x7fa6d8011e60 comp rx=0 tx=0).stop 2024-03-06T07:25:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:35.152+0000 7fa6ea6c2640 1 -- 172.21.15.1:0/3472934351 shutdown_connections 2024-03-06T07:25:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:35.152+0000 7fa6ea6c2640 1 --2- 172.21.15.1:0/3472934351 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa6d006b500 0x7fa6d006d9c0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:35.152+0000 7fa6ea6c2640 1 --2- 172.21.15.1:0/3472934351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e41de700 0x7fa6e414cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:35.152+0000 7fa6ea6c2640 1 --2- 172.21.15.1:0/3472934351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6e414d8e0 0x7fa6e41eec30 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:35.152+0000 7fa6ea6c2640 1 --2- 172.21.15.1:0/3472934351 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6e41509b0 0x7fa6e41eb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:35.152+0000 7fa6ea6c2640 1 -- 172.21.15.1:0/3472934351 >> 172.21.15.1:0/3472934351 conn(0x7fa6e4066ee0 msgr2=0x7fa6e40c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:35.152+0000 7fa6ea6c2640 1 -- 172.21.15.1:0/3472934351 shutdown_connections 2024-03-06T07:25:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:35.152+0000 7fa6ea6c2640 1 -- 172.21.15.1:0/3472934351 wait complete. 2024-03-06T07:25:35.168 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-03-06T07:25:34.762089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262900,"kb_used_data":13440,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754711820,"statfs":{"total":773094113280,"available":772824903680,"internally_reserved":0,"allocated":13762560,"data_stored":7732604,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001307"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830221+0000","last_change":"2024-03-06T07:25:30.444293+0000","last_active":"2024-03-06T07:25:31.830221+0000","last_peered":"2024-03-06T07:25:31.830221+0000","last_clean":"2024-03-06T07:25:31.830221+0000","last_became_active":"2024-03-06T07:25:30.444021+0000","last_became_peered":"2024-03-06T07:25:30.444021+0000","last_unstale":"2024-03-06T07:25:31.830221+0000","last_undegraded":"2024-03-06T07:25:31.830221+0000","last_fullsized":"2024-03-06T07:25:31.830221+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:29.480013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830228+0000","last_change":"2024-03-06T07:25:30.447395+0000","last_active":"2024-03-06T07:25:31.830228+0000","last_peered":"2024-03-06T07:25:31.830228+0000","last_clean":"2024-03-06T07:25:31.830228+0000","last_became_active":"2024-03-06T07:25:30.447149+0000","last_became_peered":"2024-03-06T07:25:30.447149+0000","last_unstale":"2024-03-06T07:25:31.830228+0000","last_undegraded":"2024-03-06T07:25:31.830228+0000","last_fullsized":"2024-03-06T07:25:31.830228+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:38.838416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830485+0000","last_change":"2024-03-06T07:25:30.441647+0000","last_active":"2024-03-06T07:25:31.830485+0000","last_peered":"2024-03-06T07:25:31.830485+0000","last_clean":"2024-03-06T07:25:31.830485+0000","last_became_active":"2024-03-06T07:25:30.441370+0000","last_became_peered":"2024-03-06T07:25:30.441370+0000","last_unstale":"2024-03-06T07:25:31.830485+0000","last_undegraded":"2024-03-06T07:25:31.830485+0000","last_fullsized":"2024-03-06T07:25:31.830485+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:09.291931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830275+0000","last_change":"2024-03-06T07:25:30.446880+0000","last_active":"2024-03-06T07:25:31.830275+0000","last_peered":"2024-03-06T07:25:31.830275+0000","last_clean":"2024-03-06T07:25:31.830275+0000","last_became_active":"2024-03-06T07:25:30.446552+0000","last_became_peered":"2024-03-06T07:25:30.446552+0000","last_unstale":"2024-03-06T07:25:31.830275+0000","last_undegraded":"2024-03-06T07:25:31.830275+0000","last_fullsized":"2024-03-06T07:25:31.830275+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:05.763972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829929+0000","last_change":"2024-03-06T07:25:30.450863+0000","last_active":"2024-03-06T07:25:31.829929+0000","last_peered":"2024-03-06T07:25:31.829929+0000","last_clean":"2024-03-06T07:25:31.829929+0000","last_became_active":"2024-03-06T07:25:30.450369+0000","last_became_peered":"2024-03-06T07:25:30.450369+0000","last_unstale":"2024-03-06T07:25:31.829929+0000","last_undegraded":"2024-03-06T07:25:31.829929+0000","last_fullsized":"2024-03-06T07:25:31.829929+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:21.975453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829665+0000","last_change":"2024-03-06T07:25:30.451578+0000","last_active":"2024-03-06T07:25:31.829665+0000","last_peered":"2024-03-06T07:25:31.829665+0000","last_clean":"2024-03-06T07:25:31.829665+0000","last_became_active":"2024-03-06T07:25:30.450761+0000","last_became_peered":"2024-03-06T07:25:30.450761+0000","last_unstale":"2024-03-06T07:25:31.829665+0000","last_undegraded":"2024-03-06T07:25:31.829665+0000","last_fullsized":"2024-03-06T07:25:31.829665+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:28.467671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830685+0000","last_change":"2024-03-06T07:25:30.440574+0000","last_active":"2024-03-06T07:25:31.830685+0000","last_peered":"2024-03-06T07:25:31.830685+0000","last_clean":"2024-03-06T07:25:31.830685+0000","last_became_active":"2024-03-06T07:25:30.440238+0000","last_became_peered":"2024-03-06T07:25:30.440238+0000","last_unstale":"2024-03-06T07:25:31.830685+0000","last_undegraded":"2024-03-06T07:25:31.830685+0000","last_fullsized":"2024-03-06T07:25:31.830685+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:56.981074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.436430+0000","last_change":"2024-03-06T07:25:30.445595+0000","last_active":"2024-03-06T07:25:31.436430+0000","last_peered":"2024-03-06T07:25:31.436430+0000","last_clean":"2024-03-06T07:25:31.436430+0000","last_became_active":"2024-03-06T07:25:30.445322+0000","last_became_peered":"2024-03-06T07:25:30.445322+0000","last_unstale":"2024-03-06T07:25:31.436430+0000","last_undegraded":"2024-03-06T07:25:31.436430+0000","last_fullsized":"2024-03-06T07:25:31.436430+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:40.279874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830129+0000","last_change":"2024-03-06T07:25:30.451652+0000","last_active":"2024-03-06T07:25:31.830129+0000","last_peered":"2024-03-06T07:25:31.830129+0000","last_clean":"2024-03-06T07:25:31.830129+0000","last_became_active":"2024-03-06T07:25:30.451246+0000","last_became_peered":"2024-03-06T07:25:30.451246+0000","last_unstale":"2024-03-06T07:25:31.830129+0000","last_undegraded":"2024-03-06T07:25:31.830129+0000","last_fullsized":"2024-03-06T07:25:31.830129+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:05.119163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830154+0000","last_change":"2024-03-06T07:25:30.451819+0000","last_active":"2024-03-06T07:25:31.830154+0000","last_peered":"2024-03-06T07:25:31.830154+0000","last_clean":"2024-03-06T07:25:31.830154+0000","last_became_active":"2024-03-06T07:25:30.451524+0000","last_became_peered":"2024-03-06T07:25:30.451524+0000","last_unstale":"2024-03-06T07:25:31.830154+0000","last_undegraded":"2024-03-06T07:25:31.830154+0000","last_fullsized":"2024-03-06T07:25:31.830154+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:57.273782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435458+0000","last_change":"2024-03-06T07:25:30.440174+0000","last_active":"2024-03-06T07:25:31.435458+0000","last_peered":"2024-03-06T07:25:31.435458+0000","last_clean":"2024-03-06T07:25:31.435458+0000","last_became_active":"2024-03-06T07:25:30.439024+0000","last_became_peered":"2024-03-06T07:25:30.439024+0000","last_unstale":"2024-03-06T07:25:31.435458+0000","last_undegraded":"2024-03-06T07:25:31.435458+0000","last_fullsized":"2024-03-06T07:25:31.435458+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:20.498367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830345+0000","last_change":"2024-03-06T07:25:30.451990+0000","last_active":"2024-03-06T07:25:31.830345+0000","last_peered":"2024-03-06T07:25:31.830345+0000","last_clean":"2024-03-06T07:25:31.830345+0000","last_became_active":"2024-03-06T07:25:30.451743+0000","last_became_peered":"2024-03-06T07:25:30.451743+0000","last_unstale":"2024-03-06T07:25:31.830345+0000","last_undegraded":"2024-03-06T07:25:31.830345+0000","last_fullsized":"2024-03-06T07:25:31.830345+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:49.292347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829779+0000","last_change":"2024-03-06T07:25:30.451900+0000","last_active":"2024-03-06T07:25:31.829779+0000","last_peered":"2024-03-06T07:25:31.829779+0000","last_clean":"2024-03-06T07:25:31.829779+0000","last_became_active":"2024-03-06T07:25:30.451603+0000","last_became_peered":"2024-03-06T07:25:30.451603+0000","last_unstale":"2024-03-06T07:25:31.829779+0000","last_undegraded":"2024-03-06T07:25:31.829779+0000","last_fullsized":"2024-03-06T07:25:31.829779+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:52.847907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829938+0000","last_change":"2024-03-06T07:25:30.453541+0000","last_active":"2024-03-06T07:25:31.829938+0000","last_peered":"2024-03-06T07:25:31.829938+0000","last_clean":"2024-03-06T07:25:31.829938+0000","last_became_active":"2024-03-06T07:25:30.453278+0000","last_became_peered":"2024-03-06T07:25:30.453278+0000","last_unstale":"2024-03-06T07:25:31.829938+0000","last_undegraded":"2024-03-06T07:25:31.829938+0000","last_fullsized":"2024-03-06T07:25:31.829938+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:10.557242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829336+0000","last_change":"2024-03-06T07:25:30.446787+0000","last_active":"2024-03-06T07:25:31.829336+0000","last_peered":"2024-03-06T07:25:31.829336+0000","last_clean":"2024-03-06T07:25:31.829336+0000","last_became_active":"2024-03-06T07:25:30.446548+0000","last_became_peered":"2024-03-06T07:25:30.446548+0000","last_unstale":"2024-03-06T07:25:31.829336+0000","last_undegraded":"2024-03-06T07:25:31.829336+0000","last_fullsized":"2024-03-06T07:25:31.829336+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:14.736861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829302+0000","last_change":"2024-03-06T07:25:30.442224+0000","last_active":"2024-03-06T07:25:31.829302+0000","last_peered":"2024-03-06T07:25:31.829302+0000","last_clean":"2024-03-06T07:25:31.829302+0000","last_became_active":"2024-03-06T07:25:30.441929+0000","last_became_peered":"2024-03-06T07:25:30.441929+0000","last_unstale":"2024-03-06T07:25:31.829302+0000","last_undegraded":"2024-03-06T07:25:31.829302+0000","last_fullsized":"2024-03-06T07:25:31.829302+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:16.840393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830677+0000","last_change":"2024-03-06T07:25:30.453093+0000","last_active":"2024-03-06T07:25:31.830677+0000","last_peered":"2024-03-06T07:25:31.830677+0000","last_clean":"2024-03-06T07:25:31.830677+0000","last_became_active":"2024-03-06T07:25:30.452702+0000","last_became_peered":"2024-03-06T07:25:30.452702+0000","last_unstale":"2024-03-06T07:25:31.830677+0000","last_undegraded":"2024-03-06T07:25:31.830677+0000","last_fullsized":"2024-03-06T07:25:31.830677+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:07.131822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435755+0000","last_change":"2024-03-06T07:25:30.440348+0000","last_active":"2024-03-06T07:25:31.435755+0000","last_peered":"2024-03-06T07:25:31.435755+0000","last_clean":"2024-03-06T07:25:31.435755+0000","last_became_active":"2024-03-06T07:25:30.440088+0000","last_became_peered":"2024-03-06T07:25:30.440088+0000","last_unstale":"2024-03-06T07:25:31.435755+0000","last_undegraded":"2024-03-06T07:25:31.435755+0000","last_fullsized":"2024-03-06T07:25:31.435755+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:05.027250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437971+0000","last_change":"2024-03-06T07:25:30.430262+0000","last_active":"2024-03-06T07:25:31.437971+0000","last_peered":"2024-03-06T07:25:31.437971+0000","last_clean":"2024-03-06T07:25:31.437971+0000","last_became_active":"2024-03-06T07:25:30.429985+0000","last_became_peered":"2024-03-06T07:25:30.429985+0000","last_unstale":"2024-03-06T07:25:31.437971+0000","last_undegraded":"2024-03-06T07:25:31.437971+0000","last_fullsized":"2024-03-06T07:25:31.437971+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:24.202062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829536+0000","last_change":"2024-03-06T07:25:30.437975+0000","last_active":"2024-03-06T07:25:31.829536+0000","last_peered":"2024-03-06T07:25:31.829536+0000","last_clean":"2024-03-06T07:25:31.829536+0000","last_became_active":"2024-03-06T07:25:30.437651+0000","last_became_peered":"2024-03-06T07:25:30.437651+0000","last_unstale":"2024-03-06T07:25:31.829536+0000","last_undegraded":"2024-03-06T07:25:31.829536+0000","last_fullsized":"2024-03-06T07:25:31.829536+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:16.840393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829810+0000","last_change":"2024-03-06T07:25:30.451180+0000","last_active":"2024-03-06T07:25:31.829810+0000","last_peered":"2024-03-06T07:25:31.829810+0000","last_clean":"2024-03-06T07:25:31.829810+0000","last_became_active":"2024-03-06T07:25:30.450588+0000","last_became_peered":"2024-03-06T07:25:30.450588+0000","last_unstale":"2024-03-06T07:25:31.829810+0000","last_undegraded":"2024-03-06T07:25:31.829810+0000","last_fullsized":"2024-03-06T07:25:31.829810+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:02.529236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829905+0000","last_change":"2024-03-06T07:25:30.438232+0000","last_active":"2024-03-06T07:25:31.829905+0000","last_peered":"2024-03-06T07:25:31.829905+0000","last_clean":"2024-03-06T07:25:31.829905+0000","last_became_active":"2024-03-06T07:25:30.437844+0000","last_became_peered":"2024-03-06T07:25:30.437844+0000","last_unstale":"2024-03-06T07:25:31.829905+0000","last_undegraded":"2024-03-06T07:25:31.829905+0000","last_fullsized":"2024-03-06T07:25:31.829905+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:22.923158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437441+0000","last_change":"2024-03-06T07:25:30.443679+0000","last_active":"2024-03-06T07:25:31.437441+0000","last_peered":"2024-03-06T07:25:31.437441+0000","last_clean":"2024-03-06T07:25:31.437441+0000","last_became_active":"2024-03-06T07:25:30.442945+0000","last_became_peered":"2024-03-06T07:25:30.442945+0000","last_unstale":"2024-03-06T07:25:31.437441+0000","last_undegraded":"2024-03-06T07:25:31.437441+0000","last_fullsized":"2024-03-06T07:25:31.437441+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:29.315213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435471+0000","last_change":"2024-03-06T07:25:30.436669+0000","last_active":"2024-03-06T07:25:31.435471+0000","last_peered":"2024-03-06T07:25:31.435471+0000","last_clean":"2024-03-06T07:25:31.435471+0000","last_became_active":"2024-03-06T07:25:30.436393+0000","last_became_peered":"2024-03-06T07:25:30.436393+0000","last_unstale":"2024-03-06T07:25:31.435471+0000","last_undegraded":"2024-03-06T07:25:31.435471+0000","last_fullsized":"2024-03-06T07:25:31.435471+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:32.014630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830361+0000","last_change":"2024-03-06T07:25:30.438072+0000","last_active":"2024-03-06T07:25:31.830361+0000","last_peered":"2024-03-06T07:25:31.830361+0000","last_clean":"2024-03-06T07:25:31.830361+0000","last_became_active":"2024-03-06T07:25:30.437799+0000","last_became_peered":"2024-03-06T07:25:30.437799+0000","last_unstale":"2024-03-06T07:25:31.830361+0000","last_undegraded":"2024-03-06T07:25:31.830361+0000","last_fullsized":"2024-03-06T07:25:31.830361+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:38.838416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830061+0000","last_change":"2024-03-06T07:25:30.445237+0000","last_active":"2024-03-06T07:25:31.830061+0000","last_peered":"2024-03-06T07:25:31.830061+0000","last_clean":"2024-03-06T07:25:31.830061+0000","last_became_active":"2024-03-06T07:25:30.444941+0000","last_became_peered":"2024-03-06T07:25:30.444941+0000","last_unstale":"2024-03-06T07:25:31.830061+0000","last_undegraded":"2024-03-06T07:25:31.830061+0000","last_fullsized":"2024-03-06T07:25:31.830061+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:14.736861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830083+0000","last_change":"2024-03-06T07:25:30.450156+0000","last_active":"2024-03-06T07:25:31.830083+0000","last_peered":"2024-03-06T07:25:31.830083+0000","last_clean":"2024-03-06T07:25:31.830083+0000","last_became_active":"2024-03-06T07:25:30.449834+0000","last_became_peered":"2024-03-06T07:25:30.449834+0000","last_unstale":"2024-03-06T07:25:31.830083+0000","last_undegraded":"2024-03-06T07:25:31.830083+0000","last_fullsized":"2024-03-06T07:25:31.830083+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:22.040091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435534+0000","last_change":"2024-03-06T07:25:30.437638+0000","last_active":"2024-03-06T07:25:31.435534+0000","last_peered":"2024-03-06T07:25:31.435534+0000","last_clean":"2024-03-06T07:25:31.435534+0000","last_became_active":"2024-03-06T07:25:30.437378+0000","last_became_peered":"2024-03-06T07:25:30.437378+0000","last_unstale":"2024-03-06T07:25:31.435534+0000","last_undegraded":"2024-03-06T07:25:31.435534+0000","last_fullsized":"2024-03-06T07:25:31.435534+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:25.842972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437077+0000","last_change":"2024-03-06T07:25:30.451891+0000","last_active":"2024-03-06T07:25:31.437077+0000","last_peered":"2024-03-06T07:25:31.437077+0000","last_clean":"2024-03-06T07:25:31.437077+0000","last_became_active":"2024-03-06T07:25:30.451607+0000","last_became_peered":"2024-03-06T07:25:30.451607+0000","last_unstale":"2024-03-06T07:25:31.437077+0000","last_undegraded":"2024-03-06T07:25:31.437077+0000","last_fullsized":"2024-03-06T07:25:31.437077+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:59.007274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435415+0000","last_change":"2024-03-06T07:25:30.443060+0000","last_active":"2024-03-06T07:25:31.435415+0000","last_peered":"2024-03-06T07:25:31.435415+0000","last_clean":"2024-03-06T07:25:31.435415+0000","last_became_active":"2024-03-06T07:25:30.442727+0000","last_became_peered":"2024-03-06T07:25:30.442727+0000","last_unstale":"2024-03-06T07:25:31.435415+0000","last_undegraded":"2024-03-06T07:25:31.435415+0000","last_fullsized":"2024-03-06T07:25:31.435415+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:18.723155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829504+0000","last_change":"2024-03-06T07:25:30.443492+0000","last_active":"2024-03-06T07:25:31.829504+0000","last_peered":"2024-03-06T07:25:31.829504+0000","last_clean":"2024-03-06T07:25:31.829504+0000","last_became_active":"2024-03-06T07:25:30.443221+0000","last_became_peered":"2024-03-06T07:25:30.443221+0000","last_unstale":"2024-03-06T07:25:31.829504+0000","last_undegraded":"2024-03-06T07:25:31.829504+0000","last_fullsized":"2024-03-06T07:25:31.829504+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:16.872734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.436436+0000","last_change":"2024-03-06T07:25:30.438762+0000","last_active":"2024-03-06T07:25:31.436436+0000","last_peered":"2024-03-06T07:25:31.436436+0000","last_clean":"2024-03-06T07:25:31.436436+0000","last_became_active":"2024-03-06T07:25:30.438487+0000","last_became_peered":"2024-03-06T07:25:30.438487+0000","last_unstale":"2024-03-06T07:25:31.436436+0000","last_undegraded":"2024-03-06T07:25:31.436436+0000","last_fullsized":"2024-03-06T07:25:31.436436+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:57.926133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830441+0000","last_change":"2024-03-06T07:25:30.440740+0000","last_active":"2024-03-06T07:25:31.830441+0000","last_peered":"2024-03-06T07:25:31.830441+0000","last_clean":"2024-03-06T07:25:31.830441+0000","last_became_active":"2024-03-06T07:25:30.440480+0000","last_became_peered":"2024-03-06T07:25:30.440480+0000","last_unstale":"2024-03-06T07:25:31.830441+0000","last_undegraded":"2024-03-06T07:25:31.830441+0000","last_fullsized":"2024-03-06T07:25:31.830441+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:28.229786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"134'4","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830217+0000","last_change":"2024-03-06T07:25:29.427051+0000","last_active":"2024-03-06T07:25:31.830217+0000","last_peered":"2024-03-06T07:25:31.830217+0000","last_clean":"2024-03-06T07:25:31.830217+0000","last_became_active":"2024-03-06T07:25:29.426750+0000","last_became_peered":"2024-03-06T07:25:29.426750+0000","last_unstale":"2024-03-06T07:25:31.830217+0000","last_undegraded":"2024-03-06T07:25:31.830217+0000","last_fullsized":"2024-03-06T07:25:31.830217+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:31.306148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830517+0000","last_change":"2024-03-06T07:25:30.445523+0000","last_active":"2024-03-06T07:25:31.830517+0000","last_peered":"2024-03-06T07:25:31.830517+0000","last_clean":"2024-03-06T07:25:31.830517+0000","last_became_active":"2024-03-06T07:25:30.445218+0000","last_became_peered":"2024-03-06T07:25:30.445218+0000","last_unstale":"2024-03-06T07:25:31.830517+0000","last_undegraded":"2024-03-06T07:25:31.830517+0000","last_fullsized":"2024-03-06T07:25:31.830517+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:30.209609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437800+0000","last_change":"2024-03-06T07:25:29.441160+0000","last_active":"2024-03-06T07:25:31.437800+0000","last_peered":"2024-03-06T07:25:31.437800+0000","last_clean":"2024-03-06T07:25:31.437800+0000","last_became_active":"2024-03-06T07:25:29.440834+0000","last_became_peered":"2024-03-06T07:25:29.440834+0000","last_unstale":"2024-03-06T07:25:31.437800+0000","last_undegraded":"2024-03-06T07:25:31.437800+0000","last_fullsized":"2024-03-06T07:25:31.437800+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:48.966734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437881+0000","last_change":"2024-03-06T07:25:30.445335+0000","last_active":"2024-03-06T07:25:31.437881+0000","last_peered":"2024-03-06T07:25:31.437881+0000","last_clean":"2024-03-06T07:25:31.437881+0000","last_became_active":"2024-03-06T07:25:30.444886+0000","last_became_peered":"2024-03-06T07:25:30.444886+0000","last_unstale":"2024-03-06T07:25:31.437881+0000","last_undegraded":"2024-03-06T07:25:31.437881+0000","last_fullsized":"2024-03-06T07:25:31.437881+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:00.812713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.438089+0000","last_change":"2024-03-06T07:25:29.443849+0000","last_active":"2024-03-06T07:25:31.438089+0000","last_peered":"2024-03-06T07:25:31.438089+0000","last_clean":"2024-03-06T07:25:31.438089+0000","last_became_active":"2024-03-06T07:25:29.443606+0000","last_became_peered":"2024-03-06T07:25:29.443606+0000","last_unstale":"2024-03-06T07:25:31.438089+0000","last_undegraded":"2024-03-06T07:25:31.438089+0000","last_fullsized":"2024-03-06T07:25:31.438089+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:13.728848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.438070+0000","last_change":"2024-03-06T07:25:30.442447+0000","last_active":"2024-03-06T07:25:31.438070+0000","last_peered":"2024-03-06T07:25:31.438070+0000","last_clean":"2024-03-06T07:25:31.438070+0000","last_became_active":"2024-03-06T07:25:30.442174+0000","last_became_peered":"2024-03-06T07:25:30.442174+0000","last_unstale":"2024-03-06T07:25:31.438070+0000","last_undegraded":"2024-03-06T07:25:31.438070+0000","last_fullsized":"2024-03-06T07:25:31.438070+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:32.314161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830439+0000","last_change":"2024-03-06T07:25:29.442089+0000","last_active":"2024-03-06T07:25:31.830439+0000","last_peered":"2024-03-06T07:25:31.830439+0000","last_clean":"2024-03-06T07:25:31.830439+0000","last_became_active":"2024-03-06T07:25:29.441679+0000","last_became_peered":"2024-03-06T07:25:29.441679+0000","last_unstale":"2024-03-06T07:25:31.830439+0000","last_undegraded":"2024-03-06T07:25:31.830439+0000","last_fullsized":"2024-03-06T07:25:31.830439+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:39.271861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829386+0000","last_change":"2024-03-06T07:25:30.430202+0000","last_active":"2024-03-06T07:25:31.829386+0000","last_peered":"2024-03-06T07:25:31.829386+0000","last_clean":"2024-03-06T07:25:31.829386+0000","last_became_active":"2024-03-06T07:25:30.429898+0000","last_became_peered":"2024-03-06T07:25:30.429898+0000","last_unstale":"2024-03-06T07:25:31.829386+0000","last_undegraded":"2024-03-06T07:25:31.829386+0000","last_fullsized":"2024-03-06T07:25:31.829386+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:43.230154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437180+0000","last_change":"2024-03-06T07:25:29.438025+0000","last_active":"2024-03-06T07:25:31.437180+0000","last_peered":"2024-03-06T07:25:31.437180+0000","last_clean":"2024-03-06T07:25:31.437180+0000","last_became_active":"2024-03-06T07:25:29.437765+0000","last_became_peered":"2024-03-06T07:25:29.437765+0000","last_unstale":"2024-03-06T07:25:31.437180+0000","last_undegraded":"2024-03-06T07:25:31.437180+0000","last_fullsized":"2024-03-06T07:25:31.437180+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:45.451033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829850+0000","last_change":"2024-03-06T07:25:30.450582+0000","last_active":"2024-03-06T07:25:31.829850+0000","last_peered":"2024-03-06T07:25:31.829850+0000","last_clean":"2024-03-06T07:25:31.829850+0000","last_became_active":"2024-03-06T07:25:30.450290+0000","last_became_peered":"2024-03-06T07:25:30.450290+0000","last_unstale":"2024-03-06T07:25:31.829850+0000","last_undegraded":"2024-03-06T07:25:31.829850+0000","last_fullsized":"2024-03-06T07:25:31.829850+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:35.092291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829762+0000","last_change":"2024-03-06T07:25:29.437797+0000","last_active":"2024-03-06T07:25:31.829762+0000","last_peered":"2024-03-06T07:25:31.829762+0000","last_clean":"2024-03-06T07:25:31.829762+0000","last_became_active":"2024-03-06T07:25:29.437461+0000","last_became_peered":"2024-03-06T07:25:29.437461+0000","last_unstale":"2024-03-06T07:25:31.829762+0000","last_undegraded":"2024-03-06T07:25:31.829762+0000","last_fullsized":"2024-03-06T07:25:31.829762+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:27.221773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830295+0000","last_change":"2024-03-06T07:25:30.443360+0000","last_active":"2024-03-06T07:25:31.830295+0000","last_peered":"2024-03-06T07:25:31.830295+0000","last_clean":"2024-03-06T07:25:31.830295+0000","last_became_active":"2024-03-06T07:25:30.443092+0000","last_became_peered":"2024-03-06T07:25:30.443092+0000","last_unstale":"2024-03-06T07:25:31.830295+0000","last_undegraded":"2024-03-06T07:25:31.830295+0000","last_fullsized":"2024-03-06T07:25:31.830295+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:38.170911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830650+0000","last_change":"2024-03-06T07:25:29.437803+0000","last_active":"2024-03-06T07:25:31.830650+0000","last_peered":"2024-03-06T07:25:31.830650+0000","last_clean":"2024-03-06T07:25:31.830650+0000","last_became_active":"2024-03-06T07:25:29.437527+0000","last_became_peered":"2024-03-06T07:25:29.437527+0000","last_unstale":"2024-03-06T07:25:31.830650+0000","last_undegraded":"2024-03-06T07:25:31.830650+0000","last_fullsized":"2024-03-06T07:25:31.830650+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:30.511914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829939+0000","last_change":"2024-03-06T07:25:30.448317+0000","last_active":"2024-03-06T07:25:31.829939+0000","last_peered":"2024-03-06T07:25:31.829939+0000","last_clean":"2024-03-06T07:25:31.829939+0000","last_became_active":"2024-03-06T07:25:30.447324+0000","last_became_peered":"2024-03-06T07:25:30.447324+0000","last_unstale":"2024-03-06T07:25:31.829939+0000","last_undegraded":"2024-03-06T07:25:31.829939+0000","last_fullsized":"2024-03-06T07:25:31.829939+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:49.974747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829310+0000","last_change":"2024-03-06T07:25:29.433974+0000","last_active":"2024-03-06T07:25:31.829310+0000","last_peered":"2024-03-06T07:25:31.829310+0000","last_clean":"2024-03-06T07:25:31.829310+0000","last_became_active":"2024-03-06T07:25:29.433432+0000","last_became_peered":"2024-03-06T07:25:29.433432+0000","last_unstale":"2024-03-06T07:25:31.829310+0000","last_undegraded":"2024-03-06T07:25:31.829310+0000","last_fullsized":"2024-03-06T07:25:31.829310+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:23.524639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.436866+0000","last_change":"2024-03-06T07:25:30.431306+0000","last_active":"2024-03-06T07:25:31.436866+0000","last_peered":"2024-03-06T07:25:31.436866+0000","last_clean":"2024-03-06T07:25:31.436866+0000","last_became_active":"2024-03-06T07:25:30.431027+0000","last_became_peered":"2024-03-06T07:25:30.431027+0000","last_unstale":"2024-03-06T07:25:31.436866+0000","last_undegraded":"2024-03-06T07:25:31.436866+0000","last_fullsized":"2024-03-06T07:25:31.436866+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:51.412317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.438191+0000","last_change":"2024-03-06T07:25:29.435113+0000","last_active":"2024-03-06T07:25:31.438191+0000","last_peered":"2024-03-06T07:25:31.438191+0000","last_clean":"2024-03-06T07:25:31.438191+0000","last_became_active":"2024-03-06T07:25:29.434807+0000","last_became_peered":"2024-03-06T07:25:29.434807+0000","last_unstale":"2024-03-06T07:25:31.438191+0000","last_undegraded":"2024-03-06T07:25:31.438191+0000","last_fullsized":"2024-03-06T07:25:31.438191+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:21.915144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435615+0000","last_change":"2024-03-06T07:25:30.438598+0000","last_active":"2024-03-06T07:25:31.435615+0000","last_peered":"2024-03-06T07:25:31.435615+0000","last_clean":"2024-03-06T07:25:31.435615+0000","last_became_active":"2024-03-06T07:25:30.438347+0000","last_became_peered":"2024-03-06T07:25:30.438347+0000","last_unstale":"2024-03-06T07:25:31.435615+0000","last_undegraded":"2024-03-06T07:25:31.435615+0000","last_fullsized":"2024-03-06T07:25:31.435615+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:24.622105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830547+0000","last_change":"2024-03-06T07:25:29.437148+0000","last_active":"2024-03-06T07:25:31.830547+0000","last_peered":"2024-03-06T07:25:31.830547+0000","last_clean":"2024-03-06T07:25:31.830547+0000","last_became_active":"2024-03-06T07:25:29.436865+0000","last_became_peered":"2024-03-06T07:25:29.436865+0000","last_unstale":"2024-03-06T07:25:31.830547+0000","last_undegraded":"2024-03-06T07:25:31.830547+0000","last_fullsized":"2024-03-06T07:25:31.830547+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:06.123809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435752+0000","last_change":"2024-03-06T07:25:30.439628+0000","last_active":"2024-03-06T07:25:31.435752+0000","last_peered":"2024-03-06T07:25:31.435752+0000","last_clean":"2024-03-06T07:25:31.435752+0000","last_became_active":"2024-03-06T07:25:30.439344+0000","last_became_peered":"2024-03-06T07:25:30.439344+0000","last_unstale":"2024-03-06T07:25:31.435752+0000","last_undegraded":"2024-03-06T07:25:31.435752+0000","last_fullsized":"2024-03-06T07:25:31.435752+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:34.382216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"134'3","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435585+0000","last_change":"2024-03-06T07:25:29.431284+0000","last_active":"2024-03-06T07:25:31.435585+0000","last_peered":"2024-03-06T07:25:31.435585+0000","last_clean":"2024-03-06T07:25:31.435585+0000","last_became_active":"2024-03-06T07:25:29.431000+0000","last_became_peered":"2024-03-06T07:25:29.431000+0000","last_unstale":"2024-03-06T07:25:31.435585+0000","last_undegraded":"2024-03-06T07:25:31.435585+0000","last_fullsized":"2024-03-06T07:25:31.435585+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:15.007937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.436622+0000","last_change":"2024-03-06T07:25:30.433959+0000","last_active":"2024-03-06T07:25:31.436622+0000","last_peered":"2024-03-06T07:25:31.436622+0000","last_clean":"2024-03-06T07:25:31.436622+0000","last_became_active":"2024-03-06T07:25:30.433362+0000","last_became_peered":"2024-03-06T07:25:30.433362+0000","last_unstale":"2024-03-06T07:25:31.436622+0000","last_undegraded":"2024-03-06T07:25:31.436622+0000","last_fullsized":"2024-03-06T07:25:31.436622+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:28.822522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829902+0000","last_change":"2024-03-06T07:25:29.426864+0000","last_active":"2024-03-06T07:25:31.829902+0000","last_peered":"2024-03-06T07:25:31.829902+0000","last_clean":"2024-03-06T07:25:31.829902+0000","last_became_active":"2024-03-06T07:25:29.426563+0000","last_became_peered":"2024-03-06T07:25:29.426563+0000","last_unstale":"2024-03-06T07:25:31.829902+0000","last_undegraded":"2024-03-06T07:25:31.829902+0000","last_fullsized":"2024-03-06T07:25:31.829902+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:29.201595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830071+0000","last_change":"2024-03-06T07:25:30.446147+0000","last_active":"2024-03-06T07:25:31.830071+0000","last_peered":"2024-03-06T07:25:31.830071+0000","last_clean":"2024-03-06T07:25:31.830071+0000","last_became_active":"2024-03-06T07:25:30.445879+0000","last_became_peered":"2024-03-06T07:25:30.445879+0000","last_unstale":"2024-03-06T07:25:31.830071+0000","last_undegraded":"2024-03-06T07:25:31.830071+0000","last_fullsized":"2024-03-06T07:25:31.830071+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:29.480013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830072+0000","last_change":"2024-03-06T07:25:29.442654+0000","last_active":"2024-03-06T07:25:31.830072+0000","last_peered":"2024-03-06T07:25:31.830072+0000","last_clean":"2024-03-06T07:25:31.830072+0000","last_became_active":"2024-03-06T07:25:29.442384+0000","last_became_peered":"2024-03-06T07:25:29.442384+0000","last_unstale":"2024-03-06T07:25:31.830072+0000","last_undegraded":"2024-03-06T07:25:31.830072+0000","last_fullsized":"2024-03-06T07:25:31.830072+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:59.559815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829955+0000","last_change":"2024-03-06T07:25:30.451587+0000","last_active":"2024-03-06T07:25:31.829955+0000","last_peered":"2024-03-06T07:25:31.829955+0000","last_clean":"2024-03-06T07:25:31.829955+0000","last_became_active":"2024-03-06T07:25:30.451185+0000","last_became_peered":"2024-03-06T07:25:30.451185+0000","last_unstale":"2024-03-06T07:25:31.829955+0000","last_undegraded":"2024-03-06T07:25:31.829955+0000","last_fullsized":"2024-03-06T07:25:31.829955+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:13.407299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830119+0000","last_change":"2024-03-06T07:25:29.437290+0000","last_active":"2024-03-06T07:25:31.830119+0000","last_peered":"2024-03-06T07:25:31.830119+0000","last_clean":"2024-03-06T07:25:31.830119+0000","last_became_active":"2024-03-06T07:25:29.437012+0000","last_became_peered":"2024-03-06T07:25:29.437012+0000","last_unstale":"2024-03-06T07:25:31.830119+0000","last_undegraded":"2024-03-06T07:25:31.830119+0000","last_fullsized":"2024-03-06T07:25:31.830119+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:12.959219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830071+0000","last_change":"2024-03-06T07:25:30.453579+0000","last_active":"2024-03-06T07:25:31.830071+0000","last_peered":"2024-03-06T07:25:31.830071+0000","last_clean":"2024-03-06T07:25:31.830071+0000","last_became_active":"2024-03-06T07:25:30.453302+0000","last_became_peered":"2024-03-06T07:25:30.453302+0000","last_unstale":"2024-03-06T07:25:31.830071+0000","last_undegraded":"2024-03-06T07:25:31.830071+0000","last_fullsized":"2024-03-06T07:25:31.830071+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:56.981074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"134'8","reported_seq":22,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829704+0000","last_change":"2024-03-06T07:25:29.444312+0000","last_active":"2024-03-06T07:25:31.829704+0000","last_peered":"2024-03-06T07:25:31.829704+0000","last_clean":"2024-03-06T07:25:31.829704+0000","last_became_active":"2024-03-06T07:25:29.444031+0000","last_became_peered":"2024-03-06T07:25:29.444031+0000","last_unstale":"2024-03-06T07:25:31.829704+0000","last_undegraded":"2024-03-06T07:25:31.829704+0000","last_fullsized":"2024-03-06T07:25:31.829704+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:04.755959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437353+0000","last_change":"2024-03-06T07:25:30.435897+0000","last_active":"2024-03-06T07:25:31.437353+0000","last_peered":"2024-03-06T07:25:31.437353+0000","last_clean":"2024-03-06T07:25:31.437353+0000","last_became_active":"2024-03-06T07:25:30.435607+0000","last_became_peered":"2024-03-06T07:25:30.435607+0000","last_unstale":"2024-03-06T07:25:31.437353+0000","last_undegraded":"2024-03-06T07:25:31.437353+0000","last_fullsized":"2024-03-06T07:25:31.437353+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:30.209609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435694+0000","last_change":"2024-03-06T07:25:29.442625+0000","last_active":"2024-03-06T07:25:31.435694+0000","last_peered":"2024-03-06T07:25:31.435694+0000","last_clean":"2024-03-06T07:25:31.435694+0000","last_became_active":"2024-03-06T07:25:29.442320+0000","last_became_peered":"2024-03-06T07:25:29.442320+0000","last_unstale":"2024-03-06T07:25:31.435694+0000","last_undegraded":"2024-03-06T07:25:31.435694+0000","last_fullsized":"2024-03-06T07:25:31.435694+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:13.813181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.438105+0000","last_change":"2024-03-06T07:25:30.430934+0000","last_active":"2024-03-06T07:25:31.438105+0000","last_peered":"2024-03-06T07:25:31.438105+0000","last_clean":"2024-03-06T07:25:31.438105+0000","last_became_active":"2024-03-06T07:25:30.430670+0000","last_became_peered":"2024-03-06T07:25:30.430670+0000","last_unstale":"2024-03-06T07:25:31.438105+0000","last_undegraded":"2024-03-06T07:25:31.438105+0000","last_fullsized":"2024-03-06T07:25:31.438105+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:45.720290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437918+0000","last_change":"2024-03-06T07:25:29.437681+0000","last_active":"2024-03-06T07:25:31.437918+0000","last_peered":"2024-03-06T07:25:31.437918+0000","last_clean":"2024-03-06T07:25:31.437918+0000","last_became_active":"2024-03-06T07:25:29.437415+0000","last_became_peered":"2024-03-06T07:25:29.437415+0000","last_unstale":"2024-03-06T07:25:31.437918+0000","last_undegraded":"2024-03-06T07:25:31.437918+0000","last_fullsized":"2024-03-06T07:25:31.437918+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:28.472000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830297+0000","last_change":"2024-03-06T07:25:30.452441+0000","last_active":"2024-03-06T07:25:31.830297+0000","last_peered":"2024-03-06T07:25:31.830297+0000","last_clean":"2024-03-06T07:25:31.830297+0000","last_became_active":"2024-03-06T07:25:30.451744+0000","last_became_peered":"2024-03-06T07:25:30.451744+0000","last_unstale":"2024-03-06T07:25:31.830297+0000","last_undegraded":"2024-03-06T07:25:31.830297+0000","last_fullsized":"2024-03-06T07:25:31.830297+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:21.127220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830572+0000","last_change":"2024-03-06T07:25:29.442155+0000","last_active":"2024-03-06T07:25:31.830572+0000","last_peered":"2024-03-06T07:25:31.830572+0000","last_clean":"2024-03-06T07:25:31.830572+0000","last_became_active":"2024-03-06T07:25:29.441793+0000","last_became_peered":"2024-03-06T07:25:29.441793+0000","last_unstale":"2024-03-06T07:25:31.830572+0000","last_undegraded":"2024-03-06T07:25:31.830572+0000","last_fullsized":"2024-03-06T07:25:31.830572+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:56.918119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830504+0000","last_change":"2024-03-06T07:25:30.447996+0000","last_active":"2024-03-06T07:25:31.830504+0000","last_peered":"2024-03-06T07:25:31.830504+0000","last_clean":"2024-03-06T07:25:31.830504+0000","last_became_active":"2024-03-06T07:25:30.447719+0000","last_became_peered":"2024-03-06T07:25:30.447719+0000","last_unstale":"2024-03-06T07:25:31.830504+0000","last_undegraded":"2024-03-06T07:25:31.830504+0000","last_fullsized":"2024-03-06T07:25:31.830504+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:38.170911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830365+0000","last_change":"2024-03-06T07:25:29.442853+0000","last_active":"2024-03-06T07:25:31.830365+0000","last_peered":"2024-03-06T07:25:31.830365+0000","last_clean":"2024-03-06T07:25:31.830365+0000","last_became_active":"2024-03-06T07:25:29.442618+0000","last_became_peered":"2024-03-06T07:25:29.442618+0000","last_unstale":"2024-03-06T07:25:31.830365+0000","last_undegraded":"2024-03-06T07:25:31.830365+0000","last_fullsized":"2024-03-06T07:25:31.830365+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:28.072325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437470+0000","last_change":"2024-03-06T07:25:30.431551+0000","last_active":"2024-03-06T07:25:31.437470+0000","last_peered":"2024-03-06T07:25:31.437470+0000","last_clean":"2024-03-06T07:25:31.437470+0000","last_became_active":"2024-03-06T07:25:30.431272+0000","last_became_peered":"2024-03-06T07:25:30.431272+0000","last_unstale":"2024-03-06T07:25:31.437470+0000","last_undegraded":"2024-03-06T07:25:31.437470+0000","last_fullsized":"2024-03-06T07:25:31.437470+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:09.371674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829493+0000","last_change":"2024-03-06T07:25:29.433350+0000","last_active":"2024-03-06T07:25:31.829493+0000","last_peered":"2024-03-06T07:25:31.829493+0000","last_clean":"2024-03-06T07:25:31.829493+0000","last_became_active":"2024-03-06T07:25:29.433064+0000","last_became_peered":"2024-03-06T07:25:29.433064+0000","last_unstale":"2024-03-06T07:25:31.829493+0000","last_undegraded":"2024-03-06T07:25:31.829493+0000","last_fullsized":"2024-03-06T07:25:31.829493+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:26.604928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830217+0000","last_change":"2024-03-06T07:25:30.447999+0000","last_active":"2024-03-06T07:25:31.830217+0000","last_peered":"2024-03-06T07:25:31.830217+0000","last_clean":"2024-03-06T07:25:31.830217+0000","last_became_active":"2024-03-06T07:25:30.447751+0000","last_became_peered":"2024-03-06T07:25:30.447751+0000","last_unstale":"2024-03-06T07:25:31.830217+0000","last_undegraded":"2024-03-06T07:25:31.830217+0000","last_fullsized":"2024-03-06T07:25:31.830217+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:21.127220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437639+0000","last_change":"2024-03-06T07:25:29.438191+0000","last_active":"2024-03-06T07:25:31.437639+0000","last_peered":"2024-03-06T07:25:31.437639+0000","last_clean":"2024-03-06T07:25:31.437639+0000","last_became_active":"2024-03-06T07:25:29.437589+0000","last_became_peered":"2024-03-06T07:25:29.437589+0000","last_unstale":"2024-03-06T07:25:31.437639+0000","last_undegraded":"2024-03-06T07:25:31.437639+0000","last_fullsized":"2024-03-06T07:25:31.437639+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:07.090865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437341+0000","last_change":"2024-03-06T07:25:30.436064+0000","last_active":"2024-03-06T07:25:31.437341+0000","last_peered":"2024-03-06T07:25:31.437341+0000","last_clean":"2024-03-06T07:25:31.437341+0000","last_became_active":"2024-03-06T07:25:30.435766+0000","last_became_peered":"2024-03-06T07:25:30.435766+0000","last_unstale":"2024-03-06T07:25:31.437341+0000","last_undegraded":"2024-03-06T07:25:31.437341+0000","last_fullsized":"2024-03-06T07:25:31.437341+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:09.291931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829433+0000","last_change":"2024-03-06T07:25:29.432629+0000","last_active":"2024-03-06T07:25:31.829433+0000","last_peered":"2024-03-06T07:25:31.829433+0000","last_clean":"2024-03-06T07:25:31.829433+0000","last_became_active":"2024-03-06T07:25:29.432348+0000","last_became_peered":"2024-03-06T07:25:29.432348+0000","last_unstale":"2024-03-06T07:25:31.829433+0000","last_undegraded":"2024-03-06T07:25:31.829433+0000","last_fullsized":"2024-03-06T07:25:31.829433+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:53.935560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830418+0000","last_change":"2024-03-06T07:25:30.445533+0000","last_active":"2024-03-06T07:25:31.830418+0000","last_peered":"2024-03-06T07:25:31.830418+0000","last_clean":"2024-03-06T07:25:31.830418+0000","last_became_active":"2024-03-06T07:25:30.445277+0000","last_became_peered":"2024-03-06T07:25:30.445277+0000","last_unstale":"2024-03-06T07:25:31.830418+0000","last_undegraded":"2024-03-06T07:25:31.830418+0000","last_fullsized":"2024-03-06T07:25:31.830418+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:24.532653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.436896+0000","last_change":"2024-03-06T07:25:29.439680+0000","last_active":"2024-03-06T07:25:31.436896+0000","last_peered":"2024-03-06T07:25:31.436896+0000","last_clean":"2024-03-06T07:25:31.436896+0000","last_became_active":"2024-03-06T07:25:29.439401+0000","last_became_peered":"2024-03-06T07:25:29.439401+0000","last_unstale":"2024-03-06T07:25:31.436896+0000","last_undegraded":"2024-03-06T07:25:31.436896+0000","last_fullsized":"2024-03-06T07:25:31.436896+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:46.550000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830065+0000","last_change":"2024-03-06T07:25:30.439905+0000","last_active":"2024-03-06T07:25:31.830065+0000","last_peered":"2024-03-06T07:25:31.830065+0000","last_clean":"2024-03-06T07:25:31.830065+0000","last_became_active":"2024-03-06T07:25:30.439622+0000","last_became_peered":"2024-03-06T07:25:30.439622+0000","last_unstale":"2024-03-06T07:25:31.830065+0000","last_undegraded":"2024-03-06T07:25:31.830065+0000","last_fullsized":"2024-03-06T07:25:31.830065+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:47.850721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830771+0000","last_change":"2024-03-06T07:25:29.445124+0000","last_active":"2024-03-06T07:25:31.830771+0000","last_peered":"2024-03-06T07:25:31.830771+0000","last_clean":"2024-03-06T07:25:31.830771+0000","last_became_active":"2024-03-06T07:25:29.444870+0000","last_became_peered":"2024-03-06T07:25:29.444870+0000","last_unstale":"2024-03-06T07:25:31.830771+0000","last_undegraded":"2024-03-06T07:25:31.830771+0000","last_fullsized":"2024-03-06T07:25:31.830771+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:34.964198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830173+0000","last_change":"2024-03-06T07:25:30.451056+0000","last_active":"2024-03-06T07:25:31.830173+0000","last_peered":"2024-03-06T07:25:31.830173+0000","last_clean":"2024-03-06T07:25:31.830173+0000","last_became_active":"2024-03-06T07:25:30.450757+0000","last_became_peered":"2024-03-06T07:25:30.450757+0000","last_unstale":"2024-03-06T07:25:31.830173+0000","last_undegraded":"2024-03-06T07:25:31.830173+0000","last_fullsized":"2024-03-06T07:25:31.830173+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:31.519928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"135'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830684+0000","last_change":"2024-03-06T07:25:29.442634+0000","last_active":"2024-03-06T07:25:31.830684+0000","last_peered":"2024-03-06T07:25:31.830684+0000","last_clean":"2024-03-06T07:25:31.830684+0000","last_became_active":"2024-03-06T07:25:29.442362+0000","last_became_peered":"2024-03-06T07:25:29.442362+0000","last_unstale":"2024-03-06T07:25:31.830684+0000","last_undegraded":"2024-03-06T07:25:31.830684+0000","last_fullsized":"2024-03-06T07:25:31.830684+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:56.918119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830451+0000","last_change":"2024-03-06T07:25:30.444465+0000","last_active":"2024-03-06T07:25:31.830451+0000","last_peered":"2024-03-06T07:25:31.830451+0000","last_clean":"2024-03-06T07:25:31.830451+0000","last_became_active":"2024-03-06T07:25:30.444195+0000","last_became_peered":"2024-03-06T07:25:30.444195+0000","last_unstale":"2024-03-06T07:25:31.830451+0000","last_undegraded":"2024-03-06T07:25:31.830451+0000","last_fullsized":"2024-03-06T07:25:31.830451+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:08.163517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830594+0000","last_change":"2024-03-06T07:25:29.440980+0000","last_active":"2024-03-06T07:25:31.830594+0000","last_peered":"2024-03-06T07:25:31.830594+0000","last_clean":"2024-03-06T07:25:31.830594+0000","last_became_active":"2024-03-06T07:25:29.440645+0000","last_became_peered":"2024-03-06T07:25:29.440645+0000","last_unstale":"2024-03-06T07:25:31.830594+0000","last_undegraded":"2024-03-06T07:25:31.830594+0000","last_fullsized":"2024-03-06T07:25:31.830594+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:45.989655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830154+0000","last_change":"2024-03-06T07:25:30.450300+0000","last_active":"2024-03-06T07:25:31.830154+0000","last_peered":"2024-03-06T07:25:31.830154+0000","last_clean":"2024-03-06T07:25:31.830154+0000","last_became_active":"2024-03-06T07:25:30.449929+0000","last_became_peered":"2024-03-06T07:25:30.449929+0000","last_unstale":"2024-03-06T07:25:31.830154+0000","last_undegraded":"2024-03-06T07:25:31.830154+0000","last_fullsized":"2024-03-06T07:25:31.830154+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:00.788232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830164+0000","last_change":"2024-03-06T07:25:29.439438+0000","last_active":"2024-03-06T07:25:31.830164+0000","last_peered":"2024-03-06T07:25:31.830164+0000","last_clean":"2024-03-06T07:25:31.830164+0000","last_became_active":"2024-03-06T07:25:29.439129+0000","last_became_peered":"2024-03-06T07:25:29.439129+0000","last_unstale":"2024-03-06T07:25:31.830164+0000","last_undegraded":"2024-03-06T07:25:31.830164+0000","last_fullsized":"2024-03-06T07:25:31.830164+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:55.973061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.436940+0000","last_change":"2024-03-06T07:25:30.437101+0000","last_active":"2024-03-06T07:25:31.436940+0000","last_peered":"2024-03-06T07:25:31.436940+0000","last_clean":"2024-03-06T07:25:31.436940+0000","last_became_active":"2024-03-06T07:25:30.436831+0000","last_became_peered":"2024-03-06T07:25:30.436831+0000","last_unstale":"2024-03-06T07:25:31.436940+0000","last_undegraded":"2024-03-06T07:25:31.436940+0000","last_fullsized":"2024-03-06T07:25:31.436940+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:47.277810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830054+0000","last_change":"2024-03-06T07:25:29.439263+0000","last_active":"2024-03-06T07:25:31.830054+0000","last_peered":"2024-03-06T07:25:31.830054+0000","last_clean":"2024-03-06T07:25:31.830054+0000","last_became_active":"2024-03-06T07:25:29.438961+0000","last_became_peered":"2024-03-06T07:25:29.438961+0000","last_unstale":"2024-03-06T07:25:31.830054+0000","last_undegraded":"2024-03-06T07:25:31.830054+0000","last_fullsized":"2024-03-06T07:25:31.830054+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:27.814509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830003+0000","last_change":"2024-03-06T07:25:30.453797+0000","last_active":"2024-03-06T07:25:31.830003+0000","last_peered":"2024-03-06T07:25:31.830003+0000","last_clean":"2024-03-06T07:25:31.830003+0000","last_became_active":"2024-03-06T07:25:30.453331+0000","last_became_peered":"2024-03-06T07:25:30.453331+0000","last_unstale":"2024-03-06T07:25:31.830003+0000","last_undegraded":"2024-03-06T07:25:31.830003+0000","last_fullsized":"2024-03-06T07:25:31.830003+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:44.325487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435373+0000","last_change":"2024-03-06T07:25:29.442215+0000","last_active":"2024-03-06T07:25:31.435373+0000","last_peered":"2024-03-06T07:25:31.435373+0000","last_clean":"2024-03-06T07:25:31.435373+0000","last_became_active":"2024-03-06T07:25:29.441739+0000","last_became_peered":"2024-03-06T07:25:29.441739+0000","last_unstale":"2024-03-06T07:25:31.435373+0000","last_undegraded":"2024-03-06T07:25:31.435373+0000","last_fullsized":"2024-03-06T07:25:31.435373+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:26.693912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830296+0000","last_change":"2024-03-06T07:25:30.438800+0000","last_active":"2024-03-06T07:25:31.830296+0000","last_peered":"2024-03-06T07:25:31.830296+0000","last_clean":"2024-03-06T07:25:31.830296+0000","last_became_active":"2024-03-06T07:25:30.438535+0000","last_became_peered":"2024-03-06T07:25:30.438535+0000","last_unstale":"2024-03-06T07:25:31.830296+0000","last_undegraded":"2024-03-06T07:25:31.830296+0000","last_fullsized":"2024-03-06T07:25:31.830296+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:08.163517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830753+0000","last_change":"2024-03-06T07:25:29.437548+0000","last_active":"2024-03-06T07:25:31.830753+0000","last_peered":"2024-03-06T07:25:31.830753+0000","last_clean":"2024-03-06T07:25:31.830753+0000","last_became_active":"2024-03-06T07:25:29.437281+0000","last_became_peered":"2024-03-06T07:25:29.437281+0000","last_unstale":"2024-03-06T07:25:31.830753+0000","last_undegraded":"2024-03-06T07:25:31.830753+0000","last_fullsized":"2024-03-06T07:25:31.830753+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:48.284334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.436712+0000","last_change":"2024-03-06T07:25:30.452462+0000","last_active":"2024-03-06T07:25:31.436712+0000","last_peered":"2024-03-06T07:25:31.436712+0000","last_clean":"2024-03-06T07:25:31.436712+0000","last_became_active":"2024-03-06T07:25:30.452136+0000","last_became_peered":"2024-03-06T07:25:30.452136+0000","last_unstale":"2024-03-06T07:25:31.436712+0000","last_undegraded":"2024-03-06T07:25:31.436712+0000","last_fullsized":"2024-03-06T07:25:31.436712+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:46.997668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437478+0000","last_change":"2024-03-06T07:25:29.437129+0000","last_active":"2024-03-06T07:25:31.437478+0000","last_peered":"2024-03-06T07:25:31.437478+0000","last_clean":"2024-03-06T07:25:31.437478+0000","last_became_active":"2024-03-06T07:25:29.436825+0000","last_became_peered":"2024-03-06T07:25:29.436825+0000","last_unstale":"2024-03-06T07:25:31.437478+0000","last_undegraded":"2024-03-06T07:25:31.437478+0000","last_fullsized":"2024-03-06T07:25:31.437478+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:14.063844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.438367+0000","last_change":"2024-03-06T07:25:30.436987+0000","last_active":"2024-03-06T07:25:31.438367+0000","last_peered":"2024-03-06T07:25:31.438367+0000","last_clean":"2024-03-06T07:25:31.438367+0000","last_became_active":"2024-03-06T07:25:30.436716+0000","last_became_peered":"2024-03-06T07:25:30.436716+0000","last_unstale":"2024-03-06T07:25:31.438367+0000","last_undegraded":"2024-03-06T07:25:31.438367+0000","last_fullsized":"2024-03-06T07:25:31.438367+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:14.652872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"134'6","reported_seq":20,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829603+0000","last_change":"2024-03-06T07:25:29.440579+0000","last_active":"2024-03-06T07:25:31.829603+0000","last_peered":"2024-03-06T07:25:31.829603+0000","last_clean":"2024-03-06T07:25:31.829603+0000","last_became_active":"2024-03-06T07:25:29.440247+0000","last_became_peered":"2024-03-06T07:25:29.440247+0000","last_unstale":"2024-03-06T07:25:31.829603+0000","last_undegraded":"2024-03-06T07:25:31.829603+0000","last_fullsized":"2024-03-06T07:25:31.829603+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:12.630221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":389,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437588+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:25:31.437588+0000","last_peered":"2024-03-06T07:25:31.437588+0000","last_clean":"2024-03-06T07:25:31.437588+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:25:31.437588+0000","last_undegraded":"2024-03-06T07:25:31.437588+0000","last_fullsized":"2024-03-06T07:25:31.437588+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574915,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32912,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338928,"statfs":{"total":96636764160,"available":96603062272,"internally_reserved":0,"allocated":1765376,"data_stored":1408771,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32840,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339000,"statfs":{"total":96636764160,"available":96603136000,"internally_reserved":0,"allocated":1699840,"data_stored":818451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32836,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339004,"statfs":{"total":96636764160,"available":96603140096,"internally_reserved":0,"allocated":1695744,"data_stored":818381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574915,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1708032,"data_stored":819993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574914,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":818997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1703936,"data_stored":818493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32840,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339000,"statfs":{"total":96636764160,"available":96603136000,"internally_reserved":0,"allocated":1699840,"data_stored":818451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1785856,"data_stored":1411067,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:35.715 INFO:tasks.ceph.mds.a.smithi001.stderr:2024-03-06T07:25:35.713+0000 7f85ca137640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:38.172 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.272+0000 7ff4e07c7640 1 Processor -- start 2024-03-06T07:25:38.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.272+0000 7ff4e07c7640 1 -- start start 2024-03-06T07:25:38.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.273+0000 7ff4e07c7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff4d80bf9d0 0x7ff4d8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.273+0000 7ff4e07c7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d80c2fc0 con 0x7ff4d8152650 2024-03-06T07:25:38.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.273+0000 7ff4e07c7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d8158da0 con 0x7ff4d80c6b10 2024-03-06T07:25:38.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.273+0000 7ff4e07c7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d8155800 con 0x7ff4d80bf9d0 2024-03-06T07:25:38.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.273+0000 7ff4de563640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff4d8152650 0x7ff4d8158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39958/0 (socket says 172.21.15.1:39958) 2024-03-06T07:25:38.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.273+0000 7ff4de563640 1 -- 172.21.15.1:0/2400274422 learned_addr learned my addr 172.21.15.1:0/2400274422 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:38.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4dd561640 1 -- 172.21.15.1:0/2400274422 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512078982 0 0) 0x7ff4d80c2fc0 con 0x7ff4d8152650 2024-03-06T07:25:38.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4dd561640 1 -- 172.21.15.1:0/2400274422 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c4003520 con 0x7ff4d8152650 2024-03-06T07:25:38.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4ded64640 1 --2- 172.21.15.1:0/2400274422 >> v2:172.21.15.98:3300/0 conn(0x7ff4d80bf9d0 0x7ff4d8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4dd561640 1 -- 172.21.15.1:0/2400274422 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2846248011 0 0) 0x7ff4c4003520 con 0x7ff4d8152650 2024-03-06T07:25:38.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4dd561640 1 -- 172.21.15.1:0/2400274422 >> v2:172.21.15.98:3300/0 conn(0x7ff4d80bf9d0 msgr2=0x7ff4d8155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4dd561640 1 --2- 172.21.15.1:0/2400274422 >> v2:172.21.15.98:3300/0 conn(0x7ff4d80bf9d0 0x7ff4d8155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4dd561640 1 -- 172.21.15.1:0/2400274422 >> v1:172.21.15.98:6789/0 conn(0x7ff4d80c6b10 legacy=0x7ff4d80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4dd561640 1 -- 172.21.15.1:0/2400274422 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d80c6690 con 0x7ff4d8152650 2024-03-06T07:25:38.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4dd561640 1 -- 172.21.15.1:0/2400274422 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7ff4d0001670 con 0x7ff4d8152650 2024-03-06T07:25:38.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4dd561640 1 -- 172.21.15.1:0/2400274422 >> v1:172.21.15.1:6789/0 conn(0x7ff4d8152650 legacy=0x7ff4d8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4dd561640 1 --2- 172.21.15.1:0/2400274422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c40039b0 0x7ff4c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4dd561640 1 --2- 172.21.15.1:0/2400274422 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c4004800 0x7ff4c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4dd561640 1 --2- 172.21.15.1:0/2400274422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c40051c0 0x7ff4c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4dd561640 1 -- 172.21.15.1:0/2400274422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d8155800 con 0x7ff4c40039b0 2024-03-06T07:25:38.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4dd561640 1 -- 172.21.15.1:0/2400274422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c400c160 con 0x7ff4c40051c0 2024-03-06T07:25:38.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4dd561640 1 -- 172.21.15.1:0/2400274422 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c400c360 con 0x7ff4c4004800 2024-03-06T07:25:38.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4de563640 1 --2- 172.21.15.1:0/2400274422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c40039b0 0x7ff4c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4ddd62640 1 --2- 172.21.15.1:0/2400274422 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c4004800 0x7ff4c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4ded64640 1 --2- 172.21.15.1:0/2400274422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c40051c0 0x7ff4c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4de563640 1 -- 172.21.15.1:0/2400274422 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c4004800 msgr2=0x7ff4c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4de563640 1 --2- 172.21.15.1:0/2400274422 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c4004800 0x7ff4c4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4de563640 1 -- 172.21.15.1:0/2400274422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c40051c0 msgr2=0x7ff4c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.274+0000 7ff4de563640 1 --2- 172.21.15.1:0/2400274422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c40051c0 0x7ff4c400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.275+0000 7ff4de563640 1 -- 172.21.15.1:0/2400274422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c4003520 con 0x7ff4c40039b0 2024-03-06T07:25:38.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.275+0000 7ff4ded64640 1 --2- 172.21.15.1:0/2400274422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c40051c0 0x7ff4c400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:38.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.275+0000 7ff4ded64640 1 -- 172.21.15.1:0/2400274422 reap_dead start 2024-03-06T07:25:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.275+0000 7ff4de563640 1 --2- 172.21.15.1:0/2400274422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c40039b0 0x7ff4c4003dd0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000c240 tx=0x7ff4c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab959a21273327c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.275+0000 7ff4dd561640 1 -- 172.21.15.1:0/2400274422 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0007cf0 con 0x7ff4c40039b0 2024-03-06T07:25:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.275+0000 7ff4dd561640 1 -- 172.21.15.1:0/2400274422 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d0030070 con 0x7ff4c40039b0 2024-03-06T07:25:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.275+0000 7ff4dd561640 1 -- 172.21.15.1:0/2400274422 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0005d70 con 0x7ff4c40039b0 2024-03-06T07:25:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.276+0000 7ff4e07c7640 1 -- 172.21.15.1:0/2400274422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c40039b0 msgr2=0x7ff4c4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.276+0000 7ff4e07c7640 1 --2- 172.21.15.1:0/2400274422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c40039b0 0x7ff4c4003dd0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000c240 tx=0x7ff4c4005830 comp rx=0 tx=0).stop 2024-03-06T07:25:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.276+0000 7ff4e07c7640 1 -- 172.21.15.1:0/2400274422 shutdown_connections 2024-03-06T07:25:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.276+0000 7ff4e07c7640 1 --2- 172.21.15.1:0/2400274422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c40039b0 0x7ff4c4003dd0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.276+0000 7ff4e07c7640 1 -- 172.21.15.1:0/2400274422 >> 172.21.15.1:0/2400274422 conn(0x7ff4d8066ee0 msgr2=0x7ff4d80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.276+0000 7ff4e07c7640 1 -- 172.21.15.1:0/2400274422 shutdown_connections 2024-03-06T07:25:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.276+0000 7ff4e07c7640 1 -- 172.21.15.1:0/2400274422 wait complete. 2024-03-06T07:25:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.277+0000 7ff4e07c7640 1 Processor -- start 2024-03-06T07:25:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.277+0000 7ff4e07c7640 1 -- start start 2024-03-06T07:25:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.278+0000 7ff4e07c7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c40039b0 0x7ff4d814c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.278+0000 7ff4e07c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4d81ee290 0x7ff4d8156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.278+0000 7ff4e07c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4d81ee710 0x7ff4d81573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.278+0000 7ff4e07c7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d81551b0 con 0x7ff4d81ee290 2024-03-06T07:25:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.278+0000 7ff4e07c7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d80c6fa0 con 0x7ff4d81ee710 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.278+0000 7ff4e07c7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d8155490 con 0x7ff4c40039b0 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.278+0000 7ff4ddd62640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4d81ee290 0x7ff4d8156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.278+0000 7ff4ddd62640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4d81ee290 0x7ff4d8156ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48152/0 (socket says 172.21.15.1:48152) 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.278+0000 7ff4ddd62640 1 -- 172.21.15.1:0/1460608013 learned_addr learned my addr 172.21.15.1:0/1460608013 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.278+0000 7ff4de563640 1 --2- 172.21.15.1:0/1460608013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c40039b0 0x7ff4d814c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.278+0000 7ff4ded64640 1 --2- 172.21.15.1:0/1460608013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4d81ee710 0x7ff4d81573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.278+0000 7ff4ddd62640 1 -- 172.21.15.1:0/1460608013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c40039b0 msgr2=0x7ff4d814c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.278+0000 7ff4ddd62640 1 --2- 172.21.15.1:0/1460608013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c40039b0 0x7ff4d814c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.278+0000 7ff4ddd62640 1 -- 172.21.15.1:0/1460608013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4d81ee710 msgr2=0x7ff4d81573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.278+0000 7ff4ddd62640 1 --2- 172.21.15.1:0/1460608013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4d81ee710 0x7ff4d81573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.278+0000 7ff4ddd62640 1 -- 172.21.15.1:0/1460608013 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d8151ce0 con 0x7ff4d81ee290 2024-03-06T07:25:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.278+0000 7ff4ded64640 1 --2- 172.21.15.1:0/1460608013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4d81ee710 0x7ff4d81573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.279+0000 7ff4ddd62640 1 --2- 172.21.15.1:0/1460608013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4d81ee290 0x7ff4d8156ea0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7ff4c800f820 tx=0x7ff4c800fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.279+0000 7ff4ceffd640 1 -- 172.21.15.1:0/1460608013 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c80050d0 con 0x7ff4d81ee290 2024-03-06T07:25:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.279+0000 7ff4ceffd640 1 -- 172.21.15.1:0/1460608013 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c8018070 con 0x7ff4d81ee290 2024-03-06T07:25:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.279+0000 7ff4e07c7640 1 -- 172.21.15.1:0/1460608013 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d814c300 con 0x7ff4d81ee290 2024-03-06T07:25:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.279+0000 7ff4ceffd640 1 -- 172.21.15.1:0/1460608013 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8005490 con 0x7ff4d81ee290 2024-03-06T07:25:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.279+0000 7ff4e07c7640 1 -- 172.21.15.1:0/1460608013 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d81eee90 con 0x7ff4d81ee290 2024-03-06T07:25:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.280+0000 7ff4e07c7640 1 -- 172.21.15.1:0/1460608013 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d8067fd0 con 0x7ff4d81ee290 2024-03-06T07:25:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.281+0000 7ff4ceffd640 1 -- 172.21.15.1:0/1460608013 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4c800d730 con 0x7ff4d81ee290 2024-03-06T07:25:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.281+0000 7ff4ceffd640 1 --2- 172.21.15.1:0/1460608013 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4bc06b500 0x7ff4bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.281+0000 7ff4ceffd640 1 -- 172.21.15.1:0/1460608013 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff4c808a040 con 0x7ff4d81ee290 2024-03-06T07:25:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.281+0000 7ff4de563640 1 --2- 172.21.15.1:0/1460608013 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4bc06b500 0x7ff4bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.282+0000 7ff4de563640 1 --2- 172.21.15.1:0/1460608013 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4bc06b500 0x7ff4bc06d9c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0034000 tx=0x7ff4d0007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.285+0000 7ff4ceffd640 1 -- 172.21.15.1:0/1460608013 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4d8151ce0 con 0x7ff4d81ee290 2024-03-06T07:25:38.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.470+0000 7ff4e07c7640 1 -- 172.21.15.1:0/1460608013 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4d80b06f0 con 0x7ff4bc06b500 2024-03-06T07:25:38.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.477+0000 7ff4ceffd640 1 -- 172.21.15.1:0/1460608013 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252563 (secure 0 0 0) 0x7ff4d80b06f0 con 0x7ff4bc06b500 2024-03-06T07:25:38.482 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:38.483 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:25:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.480+0000 7ff4e07c7640 1 -- 172.21.15.1:0/1460608013 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4bc06b500 msgr2=0x7ff4bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.480+0000 7ff4e07c7640 1 --2- 172.21.15.1:0/1460608013 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4bc06b500 0x7ff4bc06d9c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0034000 tx=0x7ff4d0007000 comp rx=0 tx=0).stop 2024-03-06T07:25:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.480+0000 7ff4e07c7640 1 -- 172.21.15.1:0/1460608013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4d81ee290 msgr2=0x7ff4d8156ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.480+0000 7ff4e07c7640 1 --2- 172.21.15.1:0/1460608013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4d81ee290 0x7ff4d8156ea0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7ff4c800f820 tx=0x7ff4c800fce0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.480+0000 7ff4e07c7640 1 -- 172.21.15.1:0/1460608013 shutdown_connections 2024-03-06T07:25:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.480+0000 7ff4e07c7640 1 --2- 172.21.15.1:0/1460608013 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff4bc06b500 0x7ff4bc06d9c0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.480+0000 7ff4e07c7640 1 --2- 172.21.15.1:0/1460608013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4d81ee710 0x7ff4d81573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.480+0000 7ff4e07c7640 1 --2- 172.21.15.1:0/1460608013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4d81ee290 0x7ff4d8156ea0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.480+0000 7ff4e07c7640 1 --2- 172.21.15.1:0/1460608013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c40039b0 0x7ff4d814c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.480+0000 7ff4e07c7640 1 -- 172.21.15.1:0/1460608013 >> 172.21.15.1:0/1460608013 conn(0x7ff4d8066ee0 msgr2=0x7ff4d80c52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.480+0000 7ff4e07c7640 1 -- 172.21.15.1:0/1460608013 shutdown_connections 2024-03-06T07:25:38.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.480+0000 7ff4e07c7640 1 -- 172.21.15.1:0/1460608013 wait complete. 2024-03-06T07:25:38.493 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-03-06T07:25:36.762764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262900,"kb_used_data":13440,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754711820,"statfs":{"total":773094113280,"available":772824903680,"internally_reserved":0,"allocated":13762560,"data_stored":7732604,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.341687"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830221+0000","last_change":"2024-03-06T07:25:30.444293+0000","last_active":"2024-03-06T07:25:31.830221+0000","last_peered":"2024-03-06T07:25:31.830221+0000","last_clean":"2024-03-06T07:25:31.830221+0000","last_became_active":"2024-03-06T07:25:30.444021+0000","last_became_peered":"2024-03-06T07:25:30.444021+0000","last_unstale":"2024-03-06T07:25:31.830221+0000","last_undegraded":"2024-03-06T07:25:31.830221+0000","last_fullsized":"2024-03-06T07:25:31.830221+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:29.480013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830228+0000","last_change":"2024-03-06T07:25:30.447395+0000","last_active":"2024-03-06T07:25:31.830228+0000","last_peered":"2024-03-06T07:25:31.830228+0000","last_clean":"2024-03-06T07:25:31.830228+0000","last_became_active":"2024-03-06T07:25:30.447149+0000","last_became_peered":"2024-03-06T07:25:30.447149+0000","last_unstale":"2024-03-06T07:25:31.830228+0000","last_undegraded":"2024-03-06T07:25:31.830228+0000","last_fullsized":"2024-03-06T07:25:31.830228+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:38.838416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830485+0000","last_change":"2024-03-06T07:25:30.441647+0000","last_active":"2024-03-06T07:25:31.830485+0000","last_peered":"2024-03-06T07:25:31.830485+0000","last_clean":"2024-03-06T07:25:31.830485+0000","last_became_active":"2024-03-06T07:25:30.441370+0000","last_became_peered":"2024-03-06T07:25:30.441370+0000","last_unstale":"2024-03-06T07:25:31.830485+0000","last_undegraded":"2024-03-06T07:25:31.830485+0000","last_fullsized":"2024-03-06T07:25:31.830485+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:09.291931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830275+0000","last_change":"2024-03-06T07:25:30.446880+0000","last_active":"2024-03-06T07:25:31.830275+0000","last_peered":"2024-03-06T07:25:31.830275+0000","last_clean":"2024-03-06T07:25:31.830275+0000","last_became_active":"2024-03-06T07:25:30.446552+0000","last_became_peered":"2024-03-06T07:25:30.446552+0000","last_unstale":"2024-03-06T07:25:31.830275+0000","last_undegraded":"2024-03-06T07:25:31.830275+0000","last_fullsized":"2024-03-06T07:25:31.830275+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:05.763972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829929+0000","last_change":"2024-03-06T07:25:30.450863+0000","last_active":"2024-03-06T07:25:31.829929+0000","last_peered":"2024-03-06T07:25:31.829929+0000","last_clean":"2024-03-06T07:25:31.829929+0000","last_became_active":"2024-03-06T07:25:30.450369+0000","last_became_peered":"2024-03-06T07:25:30.450369+0000","last_unstale":"2024-03-06T07:25:31.829929+0000","last_undegraded":"2024-03-06T07:25:31.829929+0000","last_fullsized":"2024-03-06T07:25:31.829929+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:21.975453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829665+0000","last_change":"2024-03-06T07:25:30.451578+0000","last_active":"2024-03-06T07:25:31.829665+0000","last_peered":"2024-03-06T07:25:31.829665+0000","last_clean":"2024-03-06T07:25:31.829665+0000","last_became_active":"2024-03-06T07:25:30.450761+0000","last_became_peered":"2024-03-06T07:25:30.450761+0000","last_unstale":"2024-03-06T07:25:31.829665+0000","last_undegraded":"2024-03-06T07:25:31.829665+0000","last_fullsized":"2024-03-06T07:25:31.829665+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:28.467671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830685+0000","last_change":"2024-03-06T07:25:30.440574+0000","last_active":"2024-03-06T07:25:31.830685+0000","last_peered":"2024-03-06T07:25:31.830685+0000","last_clean":"2024-03-06T07:25:31.830685+0000","last_became_active":"2024-03-06T07:25:30.440238+0000","last_became_peered":"2024-03-06T07:25:30.440238+0000","last_unstale":"2024-03-06T07:25:31.830685+0000","last_undegraded":"2024-03-06T07:25:31.830685+0000","last_fullsized":"2024-03-06T07:25:31.830685+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:56.981074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.436430+0000","last_change":"2024-03-06T07:25:30.445595+0000","last_active":"2024-03-06T07:25:31.436430+0000","last_peered":"2024-03-06T07:25:31.436430+0000","last_clean":"2024-03-06T07:25:31.436430+0000","last_became_active":"2024-03-06T07:25:30.445322+0000","last_became_peered":"2024-03-06T07:25:30.445322+0000","last_unstale":"2024-03-06T07:25:31.436430+0000","last_undegraded":"2024-03-06T07:25:31.436430+0000","last_fullsized":"2024-03-06T07:25:31.436430+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:40.279874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830129+0000","last_change":"2024-03-06T07:25:30.451652+0000","last_active":"2024-03-06T07:25:31.830129+0000","last_peered":"2024-03-06T07:25:31.830129+0000","last_clean":"2024-03-06T07:25:31.830129+0000","last_became_active":"2024-03-06T07:25:30.451246+0000","last_became_peered":"2024-03-06T07:25:30.451246+0000","last_unstale":"2024-03-06T07:25:31.830129+0000","last_undegraded":"2024-03-06T07:25:31.830129+0000","last_fullsized":"2024-03-06T07:25:31.830129+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:05.119163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830154+0000","last_change":"2024-03-06T07:25:30.451819+0000","last_active":"2024-03-06T07:25:31.830154+0000","last_peered":"2024-03-06T07:25:31.830154+0000","last_clean":"2024-03-06T07:25:31.830154+0000","last_became_active":"2024-03-06T07:25:30.451524+0000","last_became_peered":"2024-03-06T07:25:30.451524+0000","last_unstale":"2024-03-06T07:25:31.830154+0000","last_undegraded":"2024-03-06T07:25:31.830154+0000","last_fullsized":"2024-03-06T07:25:31.830154+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:57.273782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435458+0000","last_change":"2024-03-06T07:25:30.440174+0000","last_active":"2024-03-06T07:25:31.435458+0000","last_peered":"2024-03-06T07:25:31.435458+0000","last_clean":"2024-03-06T07:25:31.435458+0000","last_became_active":"2024-03-06T07:25:30.439024+0000","last_became_peered":"2024-03-06T07:25:30.439024+0000","last_unstale":"2024-03-06T07:25:31.435458+0000","last_undegraded":"2024-03-06T07:25:31.435458+0000","last_fullsized":"2024-03-06T07:25:31.435458+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:20.498367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830345+0000","last_change":"2024-03-06T07:25:30.451990+0000","last_active":"2024-03-06T07:25:31.830345+0000","last_peered":"2024-03-06T07:25:31.830345+0000","last_clean":"2024-03-06T07:25:31.830345+0000","last_became_active":"2024-03-06T07:25:30.451743+0000","last_became_peered":"2024-03-06T07:25:30.451743+0000","last_unstale":"2024-03-06T07:25:31.830345+0000","last_undegraded":"2024-03-06T07:25:31.830345+0000","last_fullsized":"2024-03-06T07:25:31.830345+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:49.292347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829779+0000","last_change":"2024-03-06T07:25:30.451900+0000","last_active":"2024-03-06T07:25:31.829779+0000","last_peered":"2024-03-06T07:25:31.829779+0000","last_clean":"2024-03-06T07:25:31.829779+0000","last_became_active":"2024-03-06T07:25:30.451603+0000","last_became_peered":"2024-03-06T07:25:30.451603+0000","last_unstale":"2024-03-06T07:25:31.829779+0000","last_undegraded":"2024-03-06T07:25:31.829779+0000","last_fullsized":"2024-03-06T07:25:31.829779+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:52.847907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829938+0000","last_change":"2024-03-06T07:25:30.453541+0000","last_active":"2024-03-06T07:25:31.829938+0000","last_peered":"2024-03-06T07:25:31.829938+0000","last_clean":"2024-03-06T07:25:31.829938+0000","last_became_active":"2024-03-06T07:25:30.453278+0000","last_became_peered":"2024-03-06T07:25:30.453278+0000","last_unstale":"2024-03-06T07:25:31.829938+0000","last_undegraded":"2024-03-06T07:25:31.829938+0000","last_fullsized":"2024-03-06T07:25:31.829938+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:10.557242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829336+0000","last_change":"2024-03-06T07:25:30.446787+0000","last_active":"2024-03-06T07:25:31.829336+0000","last_peered":"2024-03-06T07:25:31.829336+0000","last_clean":"2024-03-06T07:25:31.829336+0000","last_became_active":"2024-03-06T07:25:30.446548+0000","last_became_peered":"2024-03-06T07:25:30.446548+0000","last_unstale":"2024-03-06T07:25:31.829336+0000","last_undegraded":"2024-03-06T07:25:31.829336+0000","last_fullsized":"2024-03-06T07:25:31.829336+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:14.736861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829302+0000","last_change":"2024-03-06T07:25:30.442224+0000","last_active":"2024-03-06T07:25:31.829302+0000","last_peered":"2024-03-06T07:25:31.829302+0000","last_clean":"2024-03-06T07:25:31.829302+0000","last_became_active":"2024-03-06T07:25:30.441929+0000","last_became_peered":"2024-03-06T07:25:30.441929+0000","last_unstale":"2024-03-06T07:25:31.829302+0000","last_undegraded":"2024-03-06T07:25:31.829302+0000","last_fullsized":"2024-03-06T07:25:31.829302+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:16.840393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830677+0000","last_change":"2024-03-06T07:25:30.453093+0000","last_active":"2024-03-06T07:25:31.830677+0000","last_peered":"2024-03-06T07:25:31.830677+0000","last_clean":"2024-03-06T07:25:31.830677+0000","last_became_active":"2024-03-06T07:25:30.452702+0000","last_became_peered":"2024-03-06T07:25:30.452702+0000","last_unstale":"2024-03-06T07:25:31.830677+0000","last_undegraded":"2024-03-06T07:25:31.830677+0000","last_fullsized":"2024-03-06T07:25:31.830677+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:07.131822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435755+0000","last_change":"2024-03-06T07:25:30.440348+0000","last_active":"2024-03-06T07:25:31.435755+0000","last_peered":"2024-03-06T07:25:31.435755+0000","last_clean":"2024-03-06T07:25:31.435755+0000","last_became_active":"2024-03-06T07:25:30.440088+0000","last_became_peered":"2024-03-06T07:25:30.440088+0000","last_unstale":"2024-03-06T07:25:31.435755+0000","last_undegraded":"2024-03-06T07:25:31.435755+0000","last_fullsized":"2024-03-06T07:25:31.435755+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:05.027250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437971+0000","last_change":"2024-03-06T07:25:30.430262+0000","last_active":"2024-03-06T07:25:31.437971+0000","last_peered":"2024-03-06T07:25:31.437971+0000","last_clean":"2024-03-06T07:25:31.437971+0000","last_became_active":"2024-03-06T07:25:30.429985+0000","last_became_peered":"2024-03-06T07:25:30.429985+0000","last_unstale":"2024-03-06T07:25:31.437971+0000","last_undegraded":"2024-03-06T07:25:31.437971+0000","last_fullsized":"2024-03-06T07:25:31.437971+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:24.202062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829536+0000","last_change":"2024-03-06T07:25:30.437975+0000","last_active":"2024-03-06T07:25:31.829536+0000","last_peered":"2024-03-06T07:25:31.829536+0000","last_clean":"2024-03-06T07:25:31.829536+0000","last_became_active":"2024-03-06T07:25:30.437651+0000","last_became_peered":"2024-03-06T07:25:30.437651+0000","last_unstale":"2024-03-06T07:25:31.829536+0000","last_undegraded":"2024-03-06T07:25:31.829536+0000","last_fullsized":"2024-03-06T07:25:31.829536+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:16.840393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829810+0000","last_change":"2024-03-06T07:25:30.451180+0000","last_active":"2024-03-06T07:25:31.829810+0000","last_peered":"2024-03-06T07:25:31.829810+0000","last_clean":"2024-03-06T07:25:31.829810+0000","last_became_active":"2024-03-06T07:25:30.450588+0000","last_became_peered":"2024-03-06T07:25:30.450588+0000","last_unstale":"2024-03-06T07:25:31.829810+0000","last_undegraded":"2024-03-06T07:25:31.829810+0000","last_fullsized":"2024-03-06T07:25:31.829810+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:02.529236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829905+0000","last_change":"2024-03-06T07:25:30.438232+0000","last_active":"2024-03-06T07:25:31.829905+0000","last_peered":"2024-03-06T07:25:31.829905+0000","last_clean":"2024-03-06T07:25:31.829905+0000","last_became_active":"2024-03-06T07:25:30.437844+0000","last_became_peered":"2024-03-06T07:25:30.437844+0000","last_unstale":"2024-03-06T07:25:31.829905+0000","last_undegraded":"2024-03-06T07:25:31.829905+0000","last_fullsized":"2024-03-06T07:25:31.829905+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:22.923158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437441+0000","last_change":"2024-03-06T07:25:30.443679+0000","last_active":"2024-03-06T07:25:31.437441+0000","last_peered":"2024-03-06T07:25:31.437441+0000","last_clean":"2024-03-06T07:25:31.437441+0000","last_became_active":"2024-03-06T07:25:30.442945+0000","last_became_peered":"2024-03-06T07:25:30.442945+0000","last_unstale":"2024-03-06T07:25:31.437441+0000","last_undegraded":"2024-03-06T07:25:31.437441+0000","last_fullsized":"2024-03-06T07:25:31.437441+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:29.315213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435471+0000","last_change":"2024-03-06T07:25:30.436669+0000","last_active":"2024-03-06T07:25:31.435471+0000","last_peered":"2024-03-06T07:25:31.435471+0000","last_clean":"2024-03-06T07:25:31.435471+0000","last_became_active":"2024-03-06T07:25:30.436393+0000","last_became_peered":"2024-03-06T07:25:30.436393+0000","last_unstale":"2024-03-06T07:25:31.435471+0000","last_undegraded":"2024-03-06T07:25:31.435471+0000","last_fullsized":"2024-03-06T07:25:31.435471+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:32.014630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830361+0000","last_change":"2024-03-06T07:25:30.438072+0000","last_active":"2024-03-06T07:25:31.830361+0000","last_peered":"2024-03-06T07:25:31.830361+0000","last_clean":"2024-03-06T07:25:31.830361+0000","last_became_active":"2024-03-06T07:25:30.437799+0000","last_became_peered":"2024-03-06T07:25:30.437799+0000","last_unstale":"2024-03-06T07:25:31.830361+0000","last_undegraded":"2024-03-06T07:25:31.830361+0000","last_fullsized":"2024-03-06T07:25:31.830361+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:38.838416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830061+0000","last_change":"2024-03-06T07:25:30.445237+0000","last_active":"2024-03-06T07:25:31.830061+0000","last_peered":"2024-03-06T07:25:31.830061+0000","last_clean":"2024-03-06T07:25:31.830061+0000","last_became_active":"2024-03-06T07:25:30.444941+0000","last_became_peered":"2024-03-06T07:25:30.444941+0000","last_unstale":"2024-03-06T07:25:31.830061+0000","last_undegraded":"2024-03-06T07:25:31.830061+0000","last_fullsized":"2024-03-06T07:25:31.830061+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:14.736861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830083+0000","last_change":"2024-03-06T07:25:30.450156+0000","last_active":"2024-03-06T07:25:31.830083+0000","last_peered":"2024-03-06T07:25:31.830083+0000","last_clean":"2024-03-06T07:25:31.830083+0000","last_became_active":"2024-03-06T07:25:30.449834+0000","last_became_peered":"2024-03-06T07:25:30.449834+0000","last_unstale":"2024-03-06T07:25:31.830083+0000","last_undegraded":"2024-03-06T07:25:31.830083+0000","last_fullsized":"2024-03-06T07:25:31.830083+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:22.040091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435534+0000","last_change":"2024-03-06T07:25:30.437638+0000","last_active":"2024-03-06T07:25:31.435534+0000","last_peered":"2024-03-06T07:25:31.435534+0000","last_clean":"2024-03-06T07:25:31.435534+0000","last_became_active":"2024-03-06T07:25:30.437378+0000","last_became_peered":"2024-03-06T07:25:30.437378+0000","last_unstale":"2024-03-06T07:25:31.435534+0000","last_undegraded":"2024-03-06T07:25:31.435534+0000","last_fullsized":"2024-03-06T07:25:31.435534+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:25.842972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437077+0000","last_change":"2024-03-06T07:25:30.451891+0000","last_active":"2024-03-06T07:25:31.437077+0000","last_peered":"2024-03-06T07:25:31.437077+0000","last_clean":"2024-03-06T07:25:31.437077+0000","last_became_active":"2024-03-06T07:25:30.451607+0000","last_became_peered":"2024-03-06T07:25:30.451607+0000","last_unstale":"2024-03-06T07:25:31.437077+0000","last_undegraded":"2024-03-06T07:25:31.437077+0000","last_fullsized":"2024-03-06T07:25:31.437077+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:59.007274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435415+0000","last_change":"2024-03-06T07:25:30.443060+0000","last_active":"2024-03-06T07:25:31.435415+0000","last_peered":"2024-03-06T07:25:31.435415+0000","last_clean":"2024-03-06T07:25:31.435415+0000","last_became_active":"2024-03-06T07:25:30.442727+0000","last_became_peered":"2024-03-06T07:25:30.442727+0000","last_unstale":"2024-03-06T07:25:31.435415+0000","last_undegraded":"2024-03-06T07:25:31.435415+0000","last_fullsized":"2024-03-06T07:25:31.435415+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:18.723155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829504+0000","last_change":"2024-03-06T07:25:30.443492+0000","last_active":"2024-03-06T07:25:31.829504+0000","last_peered":"2024-03-06T07:25:31.829504+0000","last_clean":"2024-03-06T07:25:31.829504+0000","last_became_active":"2024-03-06T07:25:30.443221+0000","last_became_peered":"2024-03-06T07:25:30.443221+0000","last_unstale":"2024-03-06T07:25:31.829504+0000","last_undegraded":"2024-03-06T07:25:31.829504+0000","last_fullsized":"2024-03-06T07:25:31.829504+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:16.872734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.436436+0000","last_change":"2024-03-06T07:25:30.438762+0000","last_active":"2024-03-06T07:25:31.436436+0000","last_peered":"2024-03-06T07:25:31.436436+0000","last_clean":"2024-03-06T07:25:31.436436+0000","last_became_active":"2024-03-06T07:25:30.438487+0000","last_became_peered":"2024-03-06T07:25:30.438487+0000","last_unstale":"2024-03-06T07:25:31.436436+0000","last_undegraded":"2024-03-06T07:25:31.436436+0000","last_fullsized":"2024-03-06T07:25:31.436436+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:57.926133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830441+0000","last_change":"2024-03-06T07:25:30.440740+0000","last_active":"2024-03-06T07:25:31.830441+0000","last_peered":"2024-03-06T07:25:31.830441+0000","last_clean":"2024-03-06T07:25:31.830441+0000","last_became_active":"2024-03-06T07:25:30.440480+0000","last_became_peered":"2024-03-06T07:25:30.440480+0000","last_unstale":"2024-03-06T07:25:31.830441+0000","last_undegraded":"2024-03-06T07:25:31.830441+0000","last_fullsized":"2024-03-06T07:25:31.830441+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:28.229786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"134'4","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830217+0000","last_change":"2024-03-06T07:25:29.427051+0000","last_active":"2024-03-06T07:25:31.830217+0000","last_peered":"2024-03-06T07:25:31.830217+0000","last_clean":"2024-03-06T07:25:31.830217+0000","last_became_active":"2024-03-06T07:25:29.426750+0000","last_became_peered":"2024-03-06T07:25:29.426750+0000","last_unstale":"2024-03-06T07:25:31.830217+0000","last_undegraded":"2024-03-06T07:25:31.830217+0000","last_fullsized":"2024-03-06T07:25:31.830217+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:31.306148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830517+0000","last_change":"2024-03-06T07:25:30.445523+0000","last_active":"2024-03-06T07:25:31.830517+0000","last_peered":"2024-03-06T07:25:31.830517+0000","last_clean":"2024-03-06T07:25:31.830517+0000","last_became_active":"2024-03-06T07:25:30.445218+0000","last_became_peered":"2024-03-06T07:25:30.445218+0000","last_unstale":"2024-03-06T07:25:31.830517+0000","last_undegraded":"2024-03-06T07:25:31.830517+0000","last_fullsized":"2024-03-06T07:25:31.830517+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:30.209609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437800+0000","last_change":"2024-03-06T07:25:29.441160+0000","last_active":"2024-03-06T07:25:31.437800+0000","last_peered":"2024-03-06T07:25:31.437800+0000","last_clean":"2024-03-06T07:25:31.437800+0000","last_became_active":"2024-03-06T07:25:29.440834+0000","last_became_peered":"2024-03-06T07:25:29.440834+0000","last_unstale":"2024-03-06T07:25:31.437800+0000","last_undegraded":"2024-03-06T07:25:31.437800+0000","last_fullsized":"2024-03-06T07:25:31.437800+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:48.966734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437881+0000","last_change":"2024-03-06T07:25:30.445335+0000","last_active":"2024-03-06T07:25:31.437881+0000","last_peered":"2024-03-06T07:25:31.437881+0000","last_clean":"2024-03-06T07:25:31.437881+0000","last_became_active":"2024-03-06T07:25:30.444886+0000","last_became_peered":"2024-03-06T07:25:30.444886+0000","last_unstale":"2024-03-06T07:25:31.437881+0000","last_undegraded":"2024-03-06T07:25:31.437881+0000","last_fullsized":"2024-03-06T07:25:31.437881+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:00.812713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.438089+0000","last_change":"2024-03-06T07:25:29.443849+0000","last_active":"2024-03-06T07:25:31.438089+0000","last_peered":"2024-03-06T07:25:31.438089+0000","last_clean":"2024-03-06T07:25:31.438089+0000","last_became_active":"2024-03-06T07:25:29.443606+0000","last_became_peered":"2024-03-06T07:25:29.443606+0000","last_unstale":"2024-03-06T07:25:31.438089+0000","last_undegraded":"2024-03-06T07:25:31.438089+0000","last_fullsized":"2024-03-06T07:25:31.438089+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:13.728848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.438070+0000","last_change":"2024-03-06T07:25:30.442447+0000","last_active":"2024-03-06T07:25:31.438070+0000","last_peered":"2024-03-06T07:25:31.438070+0000","last_clean":"2024-03-06T07:25:31.438070+0000","last_became_active":"2024-03-06T07:25:30.442174+0000","last_became_peered":"2024-03-06T07:25:30.442174+0000","last_unstale":"2024-03-06T07:25:31.438070+0000","last_undegraded":"2024-03-06T07:25:31.438070+0000","last_fullsized":"2024-03-06T07:25:31.438070+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:32.314161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830439+0000","last_change":"2024-03-06T07:25:29.442089+0000","last_active":"2024-03-06T07:25:31.830439+0000","last_peered":"2024-03-06T07:25:31.830439+0000","last_clean":"2024-03-06T07:25:31.830439+0000","last_became_active":"2024-03-06T07:25:29.441679+0000","last_became_peered":"2024-03-06T07:25:29.441679+0000","last_unstale":"2024-03-06T07:25:31.830439+0000","last_undegraded":"2024-03-06T07:25:31.830439+0000","last_fullsized":"2024-03-06T07:25:31.830439+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:39.271861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829386+0000","last_change":"2024-03-06T07:25:30.430202+0000","last_active":"2024-03-06T07:25:31.829386+0000","last_peered":"2024-03-06T07:25:31.829386+0000","last_clean":"2024-03-06T07:25:31.829386+0000","last_became_active":"2024-03-06T07:25:30.429898+0000","last_became_peered":"2024-03-06T07:25:30.429898+0000","last_unstale":"2024-03-06T07:25:31.829386+0000","last_undegraded":"2024-03-06T07:25:31.829386+0000","last_fullsized":"2024-03-06T07:25:31.829386+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:43.230154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437180+0000","last_change":"2024-03-06T07:25:29.438025+0000","last_active":"2024-03-06T07:25:31.437180+0000","last_peered":"2024-03-06T07:25:31.437180+0000","last_clean":"2024-03-06T07:25:31.437180+0000","last_became_active":"2024-03-06T07:25:29.437765+0000","last_became_peered":"2024-03-06T07:25:29.437765+0000","last_unstale":"2024-03-06T07:25:31.437180+0000","last_undegraded":"2024-03-06T07:25:31.437180+0000","last_fullsized":"2024-03-06T07:25:31.437180+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:45.451033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829850+0000","last_change":"2024-03-06T07:25:30.450582+0000","last_active":"2024-03-06T07:25:31.829850+0000","last_peered":"2024-03-06T07:25:31.829850+0000","last_clean":"2024-03-06T07:25:31.829850+0000","last_became_active":"2024-03-06T07:25:30.450290+0000","last_became_peered":"2024-03-06T07:25:30.450290+0000","last_unstale":"2024-03-06T07:25:31.829850+0000","last_undegraded":"2024-03-06T07:25:31.829850+0000","last_fullsized":"2024-03-06T07:25:31.829850+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:35.092291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829762+0000","last_change":"2024-03-06T07:25:29.437797+0000","last_active":"2024-03-06T07:25:31.829762+0000","last_peered":"2024-03-06T07:25:31.829762+0000","last_clean":"2024-03-06T07:25:31.829762+0000","last_became_active":"2024-03-06T07:25:29.437461+0000","last_became_peered":"2024-03-06T07:25:29.437461+0000","last_unstale":"2024-03-06T07:25:31.829762+0000","last_undegraded":"2024-03-06T07:25:31.829762+0000","last_fullsized":"2024-03-06T07:25:31.829762+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:27.221773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830295+0000","last_change":"2024-03-06T07:25:30.443360+0000","last_active":"2024-03-06T07:25:31.830295+0000","last_peered":"2024-03-06T07:25:31.830295+0000","last_clean":"2024-03-06T07:25:31.830295+0000","last_became_active":"2024-03-06T07:25:30.443092+0000","last_became_peered":"2024-03-06T07:25:30.443092+0000","last_unstale":"2024-03-06T07:25:31.830295+0000","last_undegraded":"2024-03-06T07:25:31.830295+0000","last_fullsized":"2024-03-06T07:25:31.830295+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:38.170911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830650+0000","last_change":"2024-03-06T07:25:29.437803+0000","last_active":"2024-03-06T07:25:31.830650+0000","last_peered":"2024-03-06T07:25:31.830650+0000","last_clean":"2024-03-06T07:25:31.830650+0000","last_became_active":"2024-03-06T07:25:29.437527+0000","last_became_peered":"2024-03-06T07:25:29.437527+0000","last_unstale":"2024-03-06T07:25:31.830650+0000","last_undegraded":"2024-03-06T07:25:31.830650+0000","last_fullsized":"2024-03-06T07:25:31.830650+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:30.511914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829939+0000","last_change":"2024-03-06T07:25:30.448317+0000","last_active":"2024-03-06T07:25:31.829939+0000","last_peered":"2024-03-06T07:25:31.829939+0000","last_clean":"2024-03-06T07:25:31.829939+0000","last_became_active":"2024-03-06T07:25:30.447324+0000","last_became_peered":"2024-03-06T07:25:30.447324+0000","last_unstale":"2024-03-06T07:25:31.829939+0000","last_undegraded":"2024-03-06T07:25:31.829939+0000","last_fullsized":"2024-03-06T07:25:31.829939+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:49.974747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829310+0000","last_change":"2024-03-06T07:25:29.433974+0000","last_active":"2024-03-06T07:25:31.829310+0000","last_peered":"2024-03-06T07:25:31.829310+0000","last_clean":"2024-03-06T07:25:31.829310+0000","last_became_active":"2024-03-06T07:25:29.433432+0000","last_became_peered":"2024-03-06T07:25:29.433432+0000","last_unstale":"2024-03-06T07:25:31.829310+0000","last_undegraded":"2024-03-06T07:25:31.829310+0000","last_fullsized":"2024-03-06T07:25:31.829310+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:23.524639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.436866+0000","last_change":"2024-03-06T07:25:30.431306+0000","last_active":"2024-03-06T07:25:31.436866+0000","last_peered":"2024-03-06T07:25:31.436866+0000","last_clean":"2024-03-06T07:25:31.436866+0000","last_became_active":"2024-03-06T07:25:30.431027+0000","last_became_peered":"2024-03-06T07:25:30.431027+0000","last_unstale":"2024-03-06T07:25:31.436866+0000","last_undegraded":"2024-03-06T07:25:31.436866+0000","last_fullsized":"2024-03-06T07:25:31.436866+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:51.412317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.438191+0000","last_change":"2024-03-06T07:25:29.435113+0000","last_active":"2024-03-06T07:25:31.438191+0000","last_peered":"2024-03-06T07:25:31.438191+0000","last_clean":"2024-03-06T07:25:31.438191+0000","last_became_active":"2024-03-06T07:25:29.434807+0000","last_became_peered":"2024-03-06T07:25:29.434807+0000","last_unstale":"2024-03-06T07:25:31.438191+0000","last_undegraded":"2024-03-06T07:25:31.438191+0000","last_fullsized":"2024-03-06T07:25:31.438191+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:21.915144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435615+0000","last_change":"2024-03-06T07:25:30.438598+0000","last_active":"2024-03-06T07:25:31.435615+0000","last_peered":"2024-03-06T07:25:31.435615+0000","last_clean":"2024-03-06T07:25:31.435615+0000","last_became_active":"2024-03-06T07:25:30.438347+0000","last_became_peered":"2024-03-06T07:25:30.438347+0000","last_unstale":"2024-03-06T07:25:31.435615+0000","last_undegraded":"2024-03-06T07:25:31.435615+0000","last_fullsized":"2024-03-06T07:25:31.435615+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:24.622105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830547+0000","last_change":"2024-03-06T07:25:29.437148+0000","last_active":"2024-03-06T07:25:31.830547+0000","last_peered":"2024-03-06T07:25:31.830547+0000","last_clean":"2024-03-06T07:25:31.830547+0000","last_became_active":"2024-03-06T07:25:29.436865+0000","last_became_peered":"2024-03-06T07:25:29.436865+0000","last_unstale":"2024-03-06T07:25:31.830547+0000","last_undegraded":"2024-03-06T07:25:31.830547+0000","last_fullsized":"2024-03-06T07:25:31.830547+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:06.123809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435752+0000","last_change":"2024-03-06T07:25:30.439628+0000","last_active":"2024-03-06T07:25:31.435752+0000","last_peered":"2024-03-06T07:25:31.435752+0000","last_clean":"2024-03-06T07:25:31.435752+0000","last_became_active":"2024-03-06T07:25:30.439344+0000","last_became_peered":"2024-03-06T07:25:30.439344+0000","last_unstale":"2024-03-06T07:25:31.435752+0000","last_undegraded":"2024-03-06T07:25:31.435752+0000","last_fullsized":"2024-03-06T07:25:31.435752+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:34.382216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"134'3","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435585+0000","last_change":"2024-03-06T07:25:29.431284+0000","last_active":"2024-03-06T07:25:31.435585+0000","last_peered":"2024-03-06T07:25:31.435585+0000","last_clean":"2024-03-06T07:25:31.435585+0000","last_became_active":"2024-03-06T07:25:29.431000+0000","last_became_peered":"2024-03-06T07:25:29.431000+0000","last_unstale":"2024-03-06T07:25:31.435585+0000","last_undegraded":"2024-03-06T07:25:31.435585+0000","last_fullsized":"2024-03-06T07:25:31.435585+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:15.007937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.436622+0000","last_change":"2024-03-06T07:25:30.433959+0000","last_active":"2024-03-06T07:25:31.436622+0000","last_peered":"2024-03-06T07:25:31.436622+0000","last_clean":"2024-03-06T07:25:31.436622+0000","last_became_active":"2024-03-06T07:25:30.433362+0000","last_became_peered":"2024-03-06T07:25:30.433362+0000","last_unstale":"2024-03-06T07:25:31.436622+0000","last_undegraded":"2024-03-06T07:25:31.436622+0000","last_fullsized":"2024-03-06T07:25:31.436622+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:28.822522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829902+0000","last_change":"2024-03-06T07:25:29.426864+0000","last_active":"2024-03-06T07:25:31.829902+0000","last_peered":"2024-03-06T07:25:31.829902+0000","last_clean":"2024-03-06T07:25:31.829902+0000","last_became_active":"2024-03-06T07:25:29.426563+0000","last_became_peered":"2024-03-06T07:25:29.426563+0000","last_unstale":"2024-03-06T07:25:31.829902+0000","last_undegraded":"2024-03-06T07:25:31.829902+0000","last_fullsized":"2024-03-06T07:25:31.829902+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:29.201595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830071+0000","last_change":"2024-03-06T07:25:30.446147+0000","last_active":"2024-03-06T07:25:31.830071+0000","last_peered":"2024-03-06T07:25:31.830071+0000","last_clean":"2024-03-06T07:25:31.830071+0000","last_became_active":"2024-03-06T07:25:30.445879+0000","last_became_peered":"2024-03-06T07:25:30.445879+0000","last_unstale":"2024-03-06T07:25:31.830071+0000","last_undegraded":"2024-03-06T07:25:31.830071+0000","last_fullsized":"2024-03-06T07:25:31.830071+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:29.480013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830072+0000","last_change":"2024-03-06T07:25:29.442654+0000","last_active":"2024-03-06T07:25:31.830072+0000","last_peered":"2024-03-06T07:25:31.830072+0000","last_clean":"2024-03-06T07:25:31.830072+0000","last_became_active":"2024-03-06T07:25:29.442384+0000","last_became_peered":"2024-03-06T07:25:29.442384+0000","last_unstale":"2024-03-06T07:25:31.830072+0000","last_undegraded":"2024-03-06T07:25:31.830072+0000","last_fullsized":"2024-03-06T07:25:31.830072+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:59.559815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829955+0000","last_change":"2024-03-06T07:25:30.451587+0000","last_active":"2024-03-06T07:25:31.829955+0000","last_peered":"2024-03-06T07:25:31.829955+0000","last_clean":"2024-03-06T07:25:31.829955+0000","last_became_active":"2024-03-06T07:25:30.451185+0000","last_became_peered":"2024-03-06T07:25:30.451185+0000","last_unstale":"2024-03-06T07:25:31.829955+0000","last_undegraded":"2024-03-06T07:25:31.829955+0000","last_fullsized":"2024-03-06T07:25:31.829955+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:13.407299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830119+0000","last_change":"2024-03-06T07:25:29.437290+0000","last_active":"2024-03-06T07:25:31.830119+0000","last_peered":"2024-03-06T07:25:31.830119+0000","last_clean":"2024-03-06T07:25:31.830119+0000","last_became_active":"2024-03-06T07:25:29.437012+0000","last_became_peered":"2024-03-06T07:25:29.437012+0000","last_unstale":"2024-03-06T07:25:31.830119+0000","last_undegraded":"2024-03-06T07:25:31.830119+0000","last_fullsized":"2024-03-06T07:25:31.830119+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:12.959219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830071+0000","last_change":"2024-03-06T07:25:30.453579+0000","last_active":"2024-03-06T07:25:31.830071+0000","last_peered":"2024-03-06T07:25:31.830071+0000","last_clean":"2024-03-06T07:25:31.830071+0000","last_became_active":"2024-03-06T07:25:30.453302+0000","last_became_peered":"2024-03-06T07:25:30.453302+0000","last_unstale":"2024-03-06T07:25:31.830071+0000","last_undegraded":"2024-03-06T07:25:31.830071+0000","last_fullsized":"2024-03-06T07:25:31.830071+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:56.981074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"134'8","reported_seq":22,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829704+0000","last_change":"2024-03-06T07:25:29.444312+0000","last_active":"2024-03-06T07:25:31.829704+0000","last_peered":"2024-03-06T07:25:31.829704+0000","last_clean":"2024-03-06T07:25:31.829704+0000","last_became_active":"2024-03-06T07:25:29.444031+0000","last_became_peered":"2024-03-06T07:25:29.444031+0000","last_unstale":"2024-03-06T07:25:31.829704+0000","last_undegraded":"2024-03-06T07:25:31.829704+0000","last_fullsized":"2024-03-06T07:25:31.829704+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:04.755959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437353+0000","last_change":"2024-03-06T07:25:30.435897+0000","last_active":"2024-03-06T07:25:31.437353+0000","last_peered":"2024-03-06T07:25:31.437353+0000","last_clean":"2024-03-06T07:25:31.437353+0000","last_became_active":"2024-03-06T07:25:30.435607+0000","last_became_peered":"2024-03-06T07:25:30.435607+0000","last_unstale":"2024-03-06T07:25:31.437353+0000","last_undegraded":"2024-03-06T07:25:31.437353+0000","last_fullsized":"2024-03-06T07:25:31.437353+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:30.209609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435694+0000","last_change":"2024-03-06T07:25:29.442625+0000","last_active":"2024-03-06T07:25:31.435694+0000","last_peered":"2024-03-06T07:25:31.435694+0000","last_clean":"2024-03-06T07:25:31.435694+0000","last_became_active":"2024-03-06T07:25:29.442320+0000","last_became_peered":"2024-03-06T07:25:29.442320+0000","last_unstale":"2024-03-06T07:25:31.435694+0000","last_undegraded":"2024-03-06T07:25:31.435694+0000","last_fullsized":"2024-03-06T07:25:31.435694+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:13.813181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.438105+0000","last_change":"2024-03-06T07:25:30.430934+0000","last_active":"2024-03-06T07:25:31.438105+0000","last_peered":"2024-03-06T07:25:31.438105+0000","last_clean":"2024-03-06T07:25:31.438105+0000","last_became_active":"2024-03-06T07:25:30.430670+0000","last_became_peered":"2024-03-06T07:25:30.430670+0000","last_unstale":"2024-03-06T07:25:31.438105+0000","last_undegraded":"2024-03-06T07:25:31.438105+0000","last_fullsized":"2024-03-06T07:25:31.438105+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:45.720290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437918+0000","last_change":"2024-03-06T07:25:29.437681+0000","last_active":"2024-03-06T07:25:31.437918+0000","last_peered":"2024-03-06T07:25:31.437918+0000","last_clean":"2024-03-06T07:25:31.437918+0000","last_became_active":"2024-03-06T07:25:29.437415+0000","last_became_peered":"2024-03-06T07:25:29.437415+0000","last_unstale":"2024-03-06T07:25:31.437918+0000","last_undegraded":"2024-03-06T07:25:31.437918+0000","last_fullsized":"2024-03-06T07:25:31.437918+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:28.472000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830297+0000","last_change":"2024-03-06T07:25:30.452441+0000","last_active":"2024-03-06T07:25:31.830297+0000","last_peered":"2024-03-06T07:25:31.830297+0000","last_clean":"2024-03-06T07:25:31.830297+0000","last_became_active":"2024-03-06T07:25:30.451744+0000","last_became_peered":"2024-03-06T07:25:30.451744+0000","last_unstale":"2024-03-06T07:25:31.830297+0000","last_undegraded":"2024-03-06T07:25:31.830297+0000","last_fullsized":"2024-03-06T07:25:31.830297+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:21.127220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830572+0000","last_change":"2024-03-06T07:25:29.442155+0000","last_active":"2024-03-06T07:25:31.830572+0000","last_peered":"2024-03-06T07:25:31.830572+0000","last_clean":"2024-03-06T07:25:31.830572+0000","last_became_active":"2024-03-06T07:25:29.441793+0000","last_became_peered":"2024-03-06T07:25:29.441793+0000","last_unstale":"2024-03-06T07:25:31.830572+0000","last_undegraded":"2024-03-06T07:25:31.830572+0000","last_fullsized":"2024-03-06T07:25:31.830572+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:56.918119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830504+0000","last_change":"2024-03-06T07:25:30.447996+0000","last_active":"2024-03-06T07:25:31.830504+0000","last_peered":"2024-03-06T07:25:31.830504+0000","last_clean":"2024-03-06T07:25:31.830504+0000","last_became_active":"2024-03-06T07:25:30.447719+0000","last_became_peered":"2024-03-06T07:25:30.447719+0000","last_unstale":"2024-03-06T07:25:31.830504+0000","last_undegraded":"2024-03-06T07:25:31.830504+0000","last_fullsized":"2024-03-06T07:25:31.830504+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:38.170911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830365+0000","last_change":"2024-03-06T07:25:29.442853+0000","last_active":"2024-03-06T07:25:31.830365+0000","last_peered":"2024-03-06T07:25:31.830365+0000","last_clean":"2024-03-06T07:25:31.830365+0000","last_became_active":"2024-03-06T07:25:29.442618+0000","last_became_peered":"2024-03-06T07:25:29.442618+0000","last_unstale":"2024-03-06T07:25:31.830365+0000","last_undegraded":"2024-03-06T07:25:31.830365+0000","last_fullsized":"2024-03-06T07:25:31.830365+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:28.072325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437470+0000","last_change":"2024-03-06T07:25:30.431551+0000","last_active":"2024-03-06T07:25:31.437470+0000","last_peered":"2024-03-06T07:25:31.437470+0000","last_clean":"2024-03-06T07:25:31.437470+0000","last_became_active":"2024-03-06T07:25:30.431272+0000","last_became_peered":"2024-03-06T07:25:30.431272+0000","last_unstale":"2024-03-06T07:25:31.437470+0000","last_undegraded":"2024-03-06T07:25:31.437470+0000","last_fullsized":"2024-03-06T07:25:31.437470+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:09.371674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829493+0000","last_change":"2024-03-06T07:25:29.433350+0000","last_active":"2024-03-06T07:25:31.829493+0000","last_peered":"2024-03-06T07:25:31.829493+0000","last_clean":"2024-03-06T07:25:31.829493+0000","last_became_active":"2024-03-06T07:25:29.433064+0000","last_became_peered":"2024-03-06T07:25:29.433064+0000","last_unstale":"2024-03-06T07:25:31.829493+0000","last_undegraded":"2024-03-06T07:25:31.829493+0000","last_fullsized":"2024-03-06T07:25:31.829493+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:26.604928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830217+0000","last_change":"2024-03-06T07:25:30.447999+0000","last_active":"2024-03-06T07:25:31.830217+0000","last_peered":"2024-03-06T07:25:31.830217+0000","last_clean":"2024-03-06T07:25:31.830217+0000","last_became_active":"2024-03-06T07:25:30.447751+0000","last_became_peered":"2024-03-06T07:25:30.447751+0000","last_unstale":"2024-03-06T07:25:31.830217+0000","last_undegraded":"2024-03-06T07:25:31.830217+0000","last_fullsized":"2024-03-06T07:25:31.830217+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:21.127220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437639+0000","last_change":"2024-03-06T07:25:29.438191+0000","last_active":"2024-03-06T07:25:31.437639+0000","last_peered":"2024-03-06T07:25:31.437639+0000","last_clean":"2024-03-06T07:25:31.437639+0000","last_became_active":"2024-03-06T07:25:29.437589+0000","last_became_peered":"2024-03-06T07:25:29.437589+0000","last_unstale":"2024-03-06T07:25:31.437639+0000","last_undegraded":"2024-03-06T07:25:31.437639+0000","last_fullsized":"2024-03-06T07:25:31.437639+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:07.090865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437341+0000","last_change":"2024-03-06T07:25:30.436064+0000","last_active":"2024-03-06T07:25:31.437341+0000","last_peered":"2024-03-06T07:25:31.437341+0000","last_clean":"2024-03-06T07:25:31.437341+0000","last_became_active":"2024-03-06T07:25:30.435766+0000","last_became_peered":"2024-03-06T07:25:30.435766+0000","last_unstale":"2024-03-06T07:25:31.437341+0000","last_undegraded":"2024-03-06T07:25:31.437341+0000","last_fullsized":"2024-03-06T07:25:31.437341+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:09.291931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829433+0000","last_change":"2024-03-06T07:25:29.432629+0000","last_active":"2024-03-06T07:25:31.829433+0000","last_peered":"2024-03-06T07:25:31.829433+0000","last_clean":"2024-03-06T07:25:31.829433+0000","last_became_active":"2024-03-06T07:25:29.432348+0000","last_became_peered":"2024-03-06T07:25:29.432348+0000","last_unstale":"2024-03-06T07:25:31.829433+0000","last_undegraded":"2024-03-06T07:25:31.829433+0000","last_fullsized":"2024-03-06T07:25:31.829433+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:53.935560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830418+0000","last_change":"2024-03-06T07:25:30.445533+0000","last_active":"2024-03-06T07:25:31.830418+0000","last_peered":"2024-03-06T07:25:31.830418+0000","last_clean":"2024-03-06T07:25:31.830418+0000","last_became_active":"2024-03-06T07:25:30.445277+0000","last_became_peered":"2024-03-06T07:25:30.445277+0000","last_unstale":"2024-03-06T07:25:31.830418+0000","last_undegraded":"2024-03-06T07:25:31.830418+0000","last_fullsized":"2024-03-06T07:25:31.830418+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:24.532653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.436896+0000","last_change":"2024-03-06T07:25:29.439680+0000","last_active":"2024-03-06T07:25:31.436896+0000","last_peered":"2024-03-06T07:25:31.436896+0000","last_clean":"2024-03-06T07:25:31.436896+0000","last_became_active":"2024-03-06T07:25:29.439401+0000","last_became_peered":"2024-03-06T07:25:29.439401+0000","last_unstale":"2024-03-06T07:25:31.436896+0000","last_undegraded":"2024-03-06T07:25:31.436896+0000","last_fullsized":"2024-03-06T07:25:31.436896+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:46.550000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830065+0000","last_change":"2024-03-06T07:25:30.439905+0000","last_active":"2024-03-06T07:25:31.830065+0000","last_peered":"2024-03-06T07:25:31.830065+0000","last_clean":"2024-03-06T07:25:31.830065+0000","last_became_active":"2024-03-06T07:25:30.439622+0000","last_became_peered":"2024-03-06T07:25:30.439622+0000","last_unstale":"2024-03-06T07:25:31.830065+0000","last_undegraded":"2024-03-06T07:25:31.830065+0000","last_fullsized":"2024-03-06T07:25:31.830065+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:47.850721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830771+0000","last_change":"2024-03-06T07:25:29.445124+0000","last_active":"2024-03-06T07:25:31.830771+0000","last_peered":"2024-03-06T07:25:31.830771+0000","last_clean":"2024-03-06T07:25:31.830771+0000","last_became_active":"2024-03-06T07:25:29.444870+0000","last_became_peered":"2024-03-06T07:25:29.444870+0000","last_unstale":"2024-03-06T07:25:31.830771+0000","last_undegraded":"2024-03-06T07:25:31.830771+0000","last_fullsized":"2024-03-06T07:25:31.830771+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:34.964198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830173+0000","last_change":"2024-03-06T07:25:30.451056+0000","last_active":"2024-03-06T07:25:31.830173+0000","last_peered":"2024-03-06T07:25:31.830173+0000","last_clean":"2024-03-06T07:25:31.830173+0000","last_became_active":"2024-03-06T07:25:30.450757+0000","last_became_peered":"2024-03-06T07:25:30.450757+0000","last_unstale":"2024-03-06T07:25:31.830173+0000","last_undegraded":"2024-03-06T07:25:31.830173+0000","last_fullsized":"2024-03-06T07:25:31.830173+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:31.519928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"135'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830684+0000","last_change":"2024-03-06T07:25:29.442634+0000","last_active":"2024-03-06T07:25:31.830684+0000","last_peered":"2024-03-06T07:25:31.830684+0000","last_clean":"2024-03-06T07:25:31.830684+0000","last_became_active":"2024-03-06T07:25:29.442362+0000","last_became_peered":"2024-03-06T07:25:29.442362+0000","last_unstale":"2024-03-06T07:25:31.830684+0000","last_undegraded":"2024-03-06T07:25:31.830684+0000","last_fullsized":"2024-03-06T07:25:31.830684+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:56.918119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830451+0000","last_change":"2024-03-06T07:25:30.444465+0000","last_active":"2024-03-06T07:25:31.830451+0000","last_peered":"2024-03-06T07:25:31.830451+0000","last_clean":"2024-03-06T07:25:31.830451+0000","last_became_active":"2024-03-06T07:25:30.444195+0000","last_became_peered":"2024-03-06T07:25:30.444195+0000","last_unstale":"2024-03-06T07:25:31.830451+0000","last_undegraded":"2024-03-06T07:25:31.830451+0000","last_fullsized":"2024-03-06T07:25:31.830451+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:08.163517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830594+0000","last_change":"2024-03-06T07:25:29.440980+0000","last_active":"2024-03-06T07:25:31.830594+0000","last_peered":"2024-03-06T07:25:31.830594+0000","last_clean":"2024-03-06T07:25:31.830594+0000","last_became_active":"2024-03-06T07:25:29.440645+0000","last_became_peered":"2024-03-06T07:25:29.440645+0000","last_unstale":"2024-03-06T07:25:31.830594+0000","last_undegraded":"2024-03-06T07:25:31.830594+0000","last_fullsized":"2024-03-06T07:25:31.830594+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:45.989655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830154+0000","last_change":"2024-03-06T07:25:30.450300+0000","last_active":"2024-03-06T07:25:31.830154+0000","last_peered":"2024-03-06T07:25:31.830154+0000","last_clean":"2024-03-06T07:25:31.830154+0000","last_became_active":"2024-03-06T07:25:30.449929+0000","last_became_peered":"2024-03-06T07:25:30.449929+0000","last_unstale":"2024-03-06T07:25:31.830154+0000","last_undegraded":"2024-03-06T07:25:31.830154+0000","last_fullsized":"2024-03-06T07:25:31.830154+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:00.788232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830164+0000","last_change":"2024-03-06T07:25:29.439438+0000","last_active":"2024-03-06T07:25:31.830164+0000","last_peered":"2024-03-06T07:25:31.830164+0000","last_clean":"2024-03-06T07:25:31.830164+0000","last_became_active":"2024-03-06T07:25:29.439129+0000","last_became_peered":"2024-03-06T07:25:29.439129+0000","last_unstale":"2024-03-06T07:25:31.830164+0000","last_undegraded":"2024-03-06T07:25:31.830164+0000","last_fullsized":"2024-03-06T07:25:31.830164+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:55.973061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.436940+0000","last_change":"2024-03-06T07:25:30.437101+0000","last_active":"2024-03-06T07:25:31.436940+0000","last_peered":"2024-03-06T07:25:31.436940+0000","last_clean":"2024-03-06T07:25:31.436940+0000","last_became_active":"2024-03-06T07:25:30.436831+0000","last_became_peered":"2024-03-06T07:25:30.436831+0000","last_unstale":"2024-03-06T07:25:31.436940+0000","last_undegraded":"2024-03-06T07:25:31.436940+0000","last_fullsized":"2024-03-06T07:25:31.436940+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:47.277810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830054+0000","last_change":"2024-03-06T07:25:29.439263+0000","last_active":"2024-03-06T07:25:31.830054+0000","last_peered":"2024-03-06T07:25:31.830054+0000","last_clean":"2024-03-06T07:25:31.830054+0000","last_became_active":"2024-03-06T07:25:29.438961+0000","last_became_peered":"2024-03-06T07:25:29.438961+0000","last_unstale":"2024-03-06T07:25:31.830054+0000","last_undegraded":"2024-03-06T07:25:31.830054+0000","last_fullsized":"2024-03-06T07:25:31.830054+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:27.814509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830003+0000","last_change":"2024-03-06T07:25:30.453797+0000","last_active":"2024-03-06T07:25:31.830003+0000","last_peered":"2024-03-06T07:25:31.830003+0000","last_clean":"2024-03-06T07:25:31.830003+0000","last_became_active":"2024-03-06T07:25:30.453331+0000","last_became_peered":"2024-03-06T07:25:30.453331+0000","last_unstale":"2024-03-06T07:25:31.830003+0000","last_undegraded":"2024-03-06T07:25:31.830003+0000","last_fullsized":"2024-03-06T07:25:31.830003+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:44.325487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.435373+0000","last_change":"2024-03-06T07:25:29.442215+0000","last_active":"2024-03-06T07:25:31.435373+0000","last_peered":"2024-03-06T07:25:31.435373+0000","last_clean":"2024-03-06T07:25:31.435373+0000","last_became_active":"2024-03-06T07:25:29.441739+0000","last_became_peered":"2024-03-06T07:25:29.441739+0000","last_unstale":"2024-03-06T07:25:31.435373+0000","last_undegraded":"2024-03-06T07:25:31.435373+0000","last_fullsized":"2024-03-06T07:25:31.435373+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:26.693912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830296+0000","last_change":"2024-03-06T07:25:30.438800+0000","last_active":"2024-03-06T07:25:31.830296+0000","last_peered":"2024-03-06T07:25:31.830296+0000","last_clean":"2024-03-06T07:25:31.830296+0000","last_became_active":"2024-03-06T07:25:30.438535+0000","last_became_peered":"2024-03-06T07:25:30.438535+0000","last_unstale":"2024-03-06T07:25:31.830296+0000","last_undegraded":"2024-03-06T07:25:31.830296+0000","last_fullsized":"2024-03-06T07:25:31.830296+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:08.163517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.830753+0000","last_change":"2024-03-06T07:25:29.437548+0000","last_active":"2024-03-06T07:25:31.830753+0000","last_peered":"2024-03-06T07:25:31.830753+0000","last_clean":"2024-03-06T07:25:31.830753+0000","last_became_active":"2024-03-06T07:25:29.437281+0000","last_became_peered":"2024-03-06T07:25:29.437281+0000","last_unstale":"2024-03-06T07:25:31.830753+0000","last_undegraded":"2024-03-06T07:25:31.830753+0000","last_fullsized":"2024-03-06T07:25:31.830753+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:48.284334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.436712+0000","last_change":"2024-03-06T07:25:30.452462+0000","last_active":"2024-03-06T07:25:31.436712+0000","last_peered":"2024-03-06T07:25:31.436712+0000","last_clean":"2024-03-06T07:25:31.436712+0000","last_became_active":"2024-03-06T07:25:30.452136+0000","last_became_peered":"2024-03-06T07:25:30.452136+0000","last_unstale":"2024-03-06T07:25:31.436712+0000","last_undegraded":"2024-03-06T07:25:31.436712+0000","last_fullsized":"2024-03-06T07:25:31.436712+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:46.997668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437478+0000","last_change":"2024-03-06T07:25:29.437129+0000","last_active":"2024-03-06T07:25:31.437478+0000","last_peered":"2024-03-06T07:25:31.437478+0000","last_clean":"2024-03-06T07:25:31.437478+0000","last_became_active":"2024-03-06T07:25:29.436825+0000","last_became_peered":"2024-03-06T07:25:29.436825+0000","last_unstale":"2024-03-06T07:25:31.437478+0000","last_undegraded":"2024-03-06T07:25:31.437478+0000","last_fullsized":"2024-03-06T07:25:31.437478+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:14.063844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.438367+0000","last_change":"2024-03-06T07:25:30.436987+0000","last_active":"2024-03-06T07:25:31.438367+0000","last_peered":"2024-03-06T07:25:31.438367+0000","last_clean":"2024-03-06T07:25:31.438367+0000","last_became_active":"2024-03-06T07:25:30.436716+0000","last_became_peered":"2024-03-06T07:25:30.436716+0000","last_unstale":"2024-03-06T07:25:31.438367+0000","last_undegraded":"2024-03-06T07:25:31.438367+0000","last_fullsized":"2024-03-06T07:25:31.438367+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:29.415355+0000","last_clean_scrub_stamp":"2024-03-06T07:25:29.415355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:14.652872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"134'6","reported_seq":20,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.829603+0000","last_change":"2024-03-06T07:25:29.440579+0000","last_active":"2024-03-06T07:25:31.829603+0000","last_peered":"2024-03-06T07:25:31.829603+0000","last_clean":"2024-03-06T07:25:31.829603+0000","last_became_active":"2024-03-06T07:25:29.440247+0000","last_became_peered":"2024-03-06T07:25:29.440247+0000","last_unstale":"2024-03-06T07:25:31.829603+0000","last_undegraded":"2024-03-06T07:25:31.829603+0000","last_fullsized":"2024-03-06T07:25:31.829603+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:28.407342+0000","last_clean_scrub_stamp":"2024-03-06T07:25:28.407342+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:12.630221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":389,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:25:31.437588+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:25:31.437588+0000","last_peered":"2024-03-06T07:25:31.437588+0000","last_clean":"2024-03-06T07:25:31.437588+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:25:31.437588+0000","last_undegraded":"2024-03-06T07:25:31.437588+0000","last_fullsized":"2024-03-06T07:25:31.437588+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574915,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32912,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338928,"statfs":{"total":96636764160,"available":96603062272,"internally_reserved":0,"allocated":1765376,"data_stored":1408771,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32840,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339000,"statfs":{"total":96636764160,"available":96603136000,"internally_reserved":0,"allocated":1699840,"data_stored":818451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32836,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339004,"statfs":{"total":96636764160,"available":96603140096,"internally_reserved":0,"allocated":1695744,"data_stored":818381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574915,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1708032,"data_stored":819993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574914,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":818997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1703936,"data_stored":818493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32840,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339000,"statfs":{"total":96636764160,"available":96603136000,"internally_reserved":0,"allocated":1699840,"data_stored":818451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1785856,"data_stored":1411067,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:38.497 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:25:38.497 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:25:38.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.639+0000 7f08d3793640 1 Processor -- start 2024-03-06T07:25:38.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.639+0000 7f08d3793640 1 -- start start 2024-03-06T07:25:38.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.640+0000 7f08d3793640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f08cc152650 0x7f08cc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.640+0000 7f08d3793640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08cc0c2fc0 con 0x7f08cc0bf9d0 2024-03-06T07:25:38.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.640+0000 7f08d3793640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08cc158da0 con 0x7f08cc0c6b10 2024-03-06T07:25:38.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.640+0000 7f08d3793640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08cc155800 con 0x7f08cc152650 2024-03-06T07:25:38.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.640+0000 7f08d152f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f08cc152650 0x7f08cc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.640+0000 7f08d1d30640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f08cc0bf9d0 0x7f08cc0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39964/0 (socket says 172.21.15.1:39964) 2024-03-06T07:25:38.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.640+0000 7f08d1d30640 1 -- 172.21.15.1:0/337758867 learned_addr learned my addr 172.21.15.1:0/337758867 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:38.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.641+0000 7f08d152f640 1 -- 172.21.15.1:0/337758867 >> v1:172.21.15.98:6789/0 conn(0x7f08cc0c6b10 legacy=0x7f08cc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.641+0000 7f08d152f640 1 -- 172.21.15.1:0/337758867 >> v1:172.21.15.1:6789/0 conn(0x7f08cc0bf9d0 legacy=0x7f08cc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.641+0000 7f08d152f640 1 -- 172.21.15.1:0/337758867 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08cc0c6690 con 0x7f08cc152650 2024-03-06T07:25:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.641+0000 7f08d152f640 1 --2- 172.21.15.1:0/337758867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08cc152650 0x7f08cc155c90 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f08c80097a0 tx=0x7f08c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f9232ca5041a641 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.642+0000 7f08c3fff640 1 -- 172.21.15.1:0/337758867 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c8035020 con 0x7f08cc152650 2024-03-06T07:25:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.642+0000 7f08c3fff640 1 -- 172.21.15.1:0/337758867 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c8031830 con 0x7f08cc152650 2024-03-06T07:25:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.642+0000 7f08c3fff640 1 -- 172.21.15.1:0/337758867 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c8031c00 con 0x7f08cc152650 2024-03-06T07:25:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.642+0000 7f08d3793640 1 -- 172.21.15.1:0/337758867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08cc152650 msgr2=0x7f08cc155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.642+0000 7f08d3793640 1 --2- 172.21.15.1:0/337758867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08cc152650 0x7f08cc155c90 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f08c80097a0 tx=0x7f08c802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.642+0000 7f08d3793640 1 -- 172.21.15.1:0/337758867 shutdown_connections 2024-03-06T07:25:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.642+0000 7f08d3793640 1 --2- 172.21.15.1:0/337758867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08cc152650 0x7f08cc155c90 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.642+0000 7f08d3793640 1 -- 172.21.15.1:0/337758867 >> 172.21.15.1:0/337758867 conn(0x7f08cc066ee0 msgr2=0x7f08cc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.642+0000 7f08d3793640 1 -- 172.21.15.1:0/337758867 shutdown_connections 2024-03-06T07:25:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.642+0000 7f08d3793640 1 -- 172.21.15.1:0/337758867 wait complete. 2024-03-06T07:25:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.644+0000 7f08d3793640 1 Processor -- start 2024-03-06T07:25:38.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.644+0000 7f08d3793640 1 -- start start 2024-03-06T07:25:38.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.645+0000 7f08d3793640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08cc0bf9d0 0x7f08cc1eb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.645+0000 7f08d3793640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08cc152650 0x7f08cc1eb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.645+0000 7f08d3793640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08cc146700 0x7f08cc150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.645+0000 7f08d3793640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc1551b0 con 0x7f08cc152650 2024-03-06T07:25:38.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.645+0000 7f08d3793640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc0c6fa0 con 0x7f08cc0bf9d0 2024-03-06T07:25:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.645+0000 7f08d3793640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc155490 con 0x7f08cc146700 2024-03-06T07:25:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.645+0000 7f08d0d2e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08cc152650 0x7f08cc1eb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.645+0000 7f08d0d2e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08cc152650 0x7f08cc1eb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48168/0 (socket says 172.21.15.1:48168) 2024-03-06T07:25:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.645+0000 7f08d0d2e640 1 -- 172.21.15.1:0/3462667140 learned_addr learned my addr 172.21.15.1:0/3462667140 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.645+0000 7f08d152f640 1 --2- 172.21.15.1:0/3462667140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08cc0bf9d0 0x7f08cc1eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.645+0000 7f08d0d2e640 1 -- 172.21.15.1:0/3462667140 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08cc146700 msgr2=0x7f08cc150270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.645+0000 7f08d1d30640 1 --2- 172.21.15.1:0/3462667140 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08cc146700 0x7f08cc150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.645+0000 7f08d0d2e640 1 --2- 172.21.15.1:0/3462667140 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08cc146700 0x7f08cc150270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.645+0000 7f08d0d2e640 1 -- 172.21.15.1:0/3462667140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08cc0bf9d0 msgr2=0x7f08cc1eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.645+0000 7f08d0d2e640 1 --2- 172.21.15.1:0/3462667140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08cc0bf9d0 0x7f08cc1eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.645+0000 7f08d0d2e640 1 -- 172.21.15.1:0/3462667140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08cc146490 con 0x7f08cc152650 2024-03-06T07:25:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.646+0000 7f08d0d2e640 1 --2- 172.21.15.1:0/3462667140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08cc152650 0x7f08cc1eb750 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00c400 tx=0x7f08bc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.646+0000 7f08c1ffb640 1 -- 172.21.15.1:0/3462667140 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08bc0052d0 con 0x7f08cc152650 2024-03-06T07:25:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.646+0000 7f08c1ffb640 1 -- 172.21.15.1:0/3462667140 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08bc005490 con 0x7f08cc152650 2024-03-06T07:25:38.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.646+0000 7f08d3793640 1 -- 172.21.15.1:0/3462667140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08cc0c9430 con 0x7f08cc152650 2024-03-06T07:25:38.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.646+0000 7f08c1ffb640 1 -- 172.21.15.1:0/3462667140 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08bc007260 con 0x7f08cc152650 2024-03-06T07:25:38.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.646+0000 7f08d3793640 1 -- 172.21.15.1:0/3462667140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08cc1eacf0 con 0x7f08cc152650 2024-03-06T07:25:38.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.647+0000 7f08d3793640 1 -- 172.21.15.1:0/3462667140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08cc1eaef0 con 0x7f08cc152650 2024-03-06T07:25:38.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.647+0000 7f08c1ffb640 1 -- 172.21.15.1:0/3462667140 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f08bc007400 con 0x7f08cc152650 2024-03-06T07:25:38.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.648+0000 7f08c1ffb640 1 --2- 172.21.15.1:0/3462667140 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f08b006b4d0 0x7f08b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.648+0000 7f08c1ffb640 1 -- 172.21.15.1:0/3462667140 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f08bc089880 con 0x7f08cc152650 2024-03-06T07:25:38.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.648+0000 7f08d152f640 1 --2- 172.21.15.1:0/3462667140 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f08b006b4d0 0x7f08b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.649+0000 7f08d152f640 1 --2- 172.21.15.1:0/3462667140 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f08b006b4d0 0x7f08b006d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f08c8036000 tx=0x7f08c8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.656+0000 7f08c1ffb640 1 -- 172.21.15.1:0/3462667140 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08cc1eaef0 con 0x7f08cc152650 2024-03-06T07:25:38.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.904+0000 7f08d3793640 1 -- 172.21.15.1:0/3462667140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f08cc067fd0 con 0x7f08cc152650 2024-03-06T07:25:38.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.906+0000 7f08c1ffb640 1 -- 172.21.15.1:0/3462667140 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7f08bc058600 con 0x7f08cc152650 2024-03-06T07:25:38.908 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-06T07:25:38.908 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng== 2024-03-06T07:25:38.909 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-06T07:25:38.909 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-06T07:25:38.909 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-06T07:25:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.908+0000 7f08d3793640 1 -- 172.21.15.1:0/3462667140 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f08b006b4d0 msgr2=0x7f08b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.908+0000 7f08d3793640 1 --2- 172.21.15.1:0/3462667140 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f08b006b4d0 0x7f08b006d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f08c8036000 tx=0x7f08c8008030 comp rx=0 tx=0).stop 2024-03-06T07:25:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.908+0000 7f08d3793640 1 -- 172.21.15.1:0/3462667140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08cc152650 msgr2=0x7f08cc1eb750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.908+0000 7f08d3793640 1 --2- 172.21.15.1:0/3462667140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08cc152650 0x7f08cc1eb750 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00c400 tx=0x7f08bc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.908+0000 7f08d3793640 1 -- 172.21.15.1:0/3462667140 shutdown_connections 2024-03-06T07:25:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.908+0000 7f08d3793640 1 --2- 172.21.15.1:0/3462667140 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f08b006b4d0 0x7f08b006d990 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.908+0000 7f08d3793640 1 --2- 172.21.15.1:0/3462667140 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08cc146700 0x7f08cc150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.908+0000 7f08d3793640 1 --2- 172.21.15.1:0/3462667140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08cc152650 0x7f08cc1eb750 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.908+0000 7f08d3793640 1 --2- 172.21.15.1:0/3462667140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08cc0bf9d0 0x7f08cc1eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.908+0000 7f08d3793640 1 -- 172.21.15.1:0/3462667140 >> 172.21.15.1:0/3462667140 conn(0x7f08cc066ee0 msgr2=0x7f08cc14e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.909+0000 7f08d3793640 1 -- 172.21.15.1:0/3462667140 shutdown_connections 2024-03-06T07:25:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:38.909+0000 7f08d3793640 1 -- 172.21.15.1:0/3462667140 wait complete. 2024-03-06T07:25:38.922 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:39.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.021+0000 7f16a169d640 1 Processor -- start 2024-03-06T07:25:39.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.021+0000 7f16a169d640 1 -- start start 2024-03-06T07:25:39.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f16a169d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f169c0bf9d0 0x7f169c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f16a169d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169c0c2fc0 con 0x7f169c0c6b10 2024-03-06T07:25:39.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f16a169d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169c158da0 con 0x7f169c152650 2024-03-06T07:25:39.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f16a169d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f169c155800 con 0x7f169c0bf9d0 2024-03-06T07:25:39.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f169a7fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f169c0c6b10 0x7f169c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39980/0 (socket says 172.21.15.1:39980) 2024-03-06T07:25:39.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f169a7fc640 1 -- 172.21.15.1:0/1533116178 learned_addr learned my addr 172.21.15.1:0/1533116178 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:39.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f169b7fe640 1 --2- 172.21.15.1:0/1533116178 >> v2:172.21.15.98:3300/0 conn(0x7f169c0bf9d0 0x7f169c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f1699ffb640 1 -- 172.21.15.1:0/1533116178 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562226319 0 0) 0x7f169c0c2fc0 con 0x7f169c0c6b10 2024-03-06T07:25:39.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f1699ffb640 1 -- 172.21.15.1:0/1533116178 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1684003520 con 0x7f169c0c6b10 2024-03-06T07:25:39.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f1699ffb640 1 -- 172.21.15.1:0/1533116178 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 723713304 0 0) 0x7f1684003520 con 0x7f169c0c6b10 2024-03-06T07:25:39.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f1699ffb640 1 -- 172.21.15.1:0/1533116178 >> v2:172.21.15.98:3300/0 conn(0x7f169c0bf9d0 msgr2=0x7f169c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f1699ffb640 1 --2- 172.21.15.1:0/1533116178 >> v2:172.21.15.98:3300/0 conn(0x7f169c0bf9d0 0x7f169c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f1699ffb640 1 -- 172.21.15.1:0/1533116178 >> v1:172.21.15.98:6789/0 conn(0x7f169c152650 legacy=0x7f169c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f1699ffb640 1 -- 172.21.15.1:0/1533116178 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c0c6690 con 0x7f169c0c6b10 2024-03-06T07:25:39.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f1699ffb640 1 -- 172.21.15.1:0/1533116178 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f1690001670 con 0x7f169c0c6b10 2024-03-06T07:25:39.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f1699ffb640 1 -- 172.21.15.1:0/1533116178 >> v1:172.21.15.1:6789/0 conn(0x7f169c0c6b10 legacy=0x7f169c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f1699ffb640 1 --2- 172.21.15.1:0/1533116178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16840039b0 0x7f1684003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.022+0000 7f1699ffb640 1 --2- 172.21.15.1:0/1533116178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1684004800 0x7f1684004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.023+0000 7f1699ffb640 1 --2- 172.21.15.1:0/1533116178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16840051c0 0x7f168400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.023+0000 7f1699ffb640 1 -- 172.21.15.1:0/1533116178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c155800 con 0x7f16840039b0 2024-03-06T07:25:39.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.023+0000 7f1699ffb640 1 -- 172.21.15.1:0/1533116178 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168400c160 con 0x7f16840051c0 2024-03-06T07:25:39.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.023+0000 7f1699ffb640 1 -- 172.21.15.1:0/1533116178 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168400c360 con 0x7f1684004800 2024-03-06T07:25:39.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.023+0000 7f169affd640 1 --2- 172.21.15.1:0/1533116178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16840039b0 0x7f1684003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.023+0000 7f169b7fe640 1 --2- 172.21.15.1:0/1533116178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16840051c0 0x7f168400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.023+0000 7f169a7fc640 1 --2- 172.21.15.1:0/1533116178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1684004800 0x7f1684004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.023+0000 7f169affd640 1 -- 172.21.15.1:0/1533116178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1684004800 msgr2=0x7f1684004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.023+0000 7f169affd640 1 --2- 172.21.15.1:0/1533116178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1684004800 0x7f1684004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.023+0000 7f169affd640 1 -- 172.21.15.1:0/1533116178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16840051c0 msgr2=0x7f168400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.023+0000 7f169affd640 1 --2- 172.21.15.1:0/1533116178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16840051c0 0x7f168400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.023+0000 7f169affd640 1 -- 172.21.15.1:0/1533116178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1684003520 con 0x7f16840039b0 2024-03-06T07:25:39.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.023+0000 7f169b7fe640 1 -- 172.21.15.1:0/1533116178 reap_dead start 2024-03-06T07:25:39.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.023+0000 7f169a7fc640 1 --2- 172.21.15.1:0/1533116178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1684004800 0x7f1684004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:39.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.024+0000 7f169affd640 1 --2- 172.21.15.1:0/1533116178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16840039b0 0x7f1684003dd0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f1694001e50 tx=0x7f1684005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fcd19d5c583b4c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.024+0000 7f1699ffb640 1 -- 172.21.15.1:0/1533116178 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694001a70 con 0x7f16840039b0 2024-03-06T07:25:39.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.024+0000 7f1699ffb640 1 -- 172.21.15.1:0/1533116178 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1694003b70 con 0x7f16840039b0 2024-03-06T07:25:39.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.024+0000 7f1699ffb640 1 -- 172.21.15.1:0/1533116178 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16940050d0 con 0x7f16840039b0 2024-03-06T07:25:39.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.025+0000 7f16a169d640 1 -- 172.21.15.1:0/1533116178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16840039b0 msgr2=0x7f1684003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.025+0000 7f16a169d640 1 --2- 172.21.15.1:0/1533116178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16840039b0 0x7f1684003dd0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f1694001e50 tx=0x7f1684005830 comp rx=0 tx=0).stop 2024-03-06T07:25:39.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.025+0000 7f16a169d640 1 -- 172.21.15.1:0/1533116178 shutdown_connections 2024-03-06T07:25:39.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.025+0000 7f16a169d640 1 --2- 172.21.15.1:0/1533116178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16840039b0 0x7f1684003dd0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.025+0000 7f16a169d640 1 -- 172.21.15.1:0/1533116178 >> 172.21.15.1:0/1533116178 conn(0x7f169c066ee0 msgr2=0x7f169c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.025+0000 7f16a169d640 1 -- 172.21.15.1:0/1533116178 shutdown_connections 2024-03-06T07:25:39.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.025+0000 7f16a169d640 1 -- 172.21.15.1:0/1533116178 wait complete. 2024-03-06T07:25:39.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.026+0000 7f16a169d640 1 Processor -- start 2024-03-06T07:25:39.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.026+0000 7f16a169d640 1 -- start start 2024-03-06T07:25:39.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.027+0000 7f16a169d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16840039b0 0x7f169c158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.027+0000 7f16a169d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c14b370 0x7f169c1ee290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.027+0000 7f16a169d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f169c1ee7d0 0x7f169c150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.027+0000 7f16a169d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c1551b0 con 0x7f169c1ee7d0 2024-03-06T07:25:39.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.027+0000 7f16a169d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0c6fa0 con 0x7f16840039b0 2024-03-06T07:25:39.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.027+0000 7f16a169d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169c155490 con 0x7f169c14b370 2024-03-06T07:25:39.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.027+0000 7f169b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f169c1ee7d0 0x7f169c150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.027+0000 7f169b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f169c1ee7d0 0x7f169c150320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48176/0 (socket says 172.21.15.1:48176) 2024-03-06T07:25:39.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.027+0000 7f169b7fe640 1 -- 172.21.15.1:0/3998789149 learned_addr learned my addr 172.21.15.1:0/3998789149 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:39.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.027+0000 7f169a7fc640 1 --2- 172.21.15.1:0/3998789149 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c14b370 0x7f169c1ee290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.027+0000 7f169affd640 1 --2- 172.21.15.1:0/3998789149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16840039b0 0x7f169c158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.027+0000 7f169b7fe640 1 -- 172.21.15.1:0/3998789149 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c14b370 msgr2=0x7f169c1ee290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.027+0000 7f169b7fe640 1 --2- 172.21.15.1:0/3998789149 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c14b370 0x7f169c1ee290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.027+0000 7f169b7fe640 1 -- 172.21.15.1:0/3998789149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16840039b0 msgr2=0x7f169c158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.027+0000 7f169b7fe640 1 --2- 172.21.15.1:0/3998789149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16840039b0 0x7f169c158090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.027+0000 7f169b7fe640 1 -- 172.21.15.1:0/3998789149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c1585d0 con 0x7f169c1ee7d0 2024-03-06T07:25:39.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.027+0000 7f169b7fe640 1 --2- 172.21.15.1:0/3998789149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f169c1ee7d0 0x7f169c150320 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f169c08c760 tx=0x7f168c011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.028+0000 7f16837fe640 1 -- 172.21.15.1:0/3998789149 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c019070 con 0x7f169c1ee7d0 2024-03-06T07:25:39.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.028+0000 7f16837fe640 1 -- 172.21.15.1:0/3998789149 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168c004240 con 0x7f169c1ee7d0 2024-03-06T07:25:39.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.028+0000 7f16837fe640 1 -- 172.21.15.1:0/3998789149 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c00f710 con 0x7f169c1ee7d0 2024-03-06T07:25:39.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.028+0000 7f16a169d640 1 -- 172.21.15.1:0/3998789149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169c1eec80 con 0x7f169c1ee7d0 2024-03-06T07:25:39.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.028+0000 7f16a169d640 1 -- 172.21.15.1:0/3998789149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f169c158a00 con 0x7f169c1ee7d0 2024-03-06T07:25:39.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.029+0000 7f16a169d640 1 -- 172.21.15.1:0/3998789149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169c0b06f0 con 0x7f169c1ee7d0 2024-03-06T07:25:39.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.029+0000 7f16837fe640 1 -- 172.21.15.1:0/3998789149 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f168c01c070 con 0x7f169c1ee7d0 2024-03-06T07:25:39.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.030+0000 7f16837fe640 1 --2- 172.21.15.1:0/3998789149 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f168806b4d0 0x7f168806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.030+0000 7f16837fe640 1 -- 172.21.15.1:0/3998789149 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f168c08a3c0 con 0x7f169c1ee7d0 2024-03-06T07:25:39.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.030+0000 7f169affd640 1 --2- 172.21.15.1:0/3998789149 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f168806b4d0 0x7f168806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.030+0000 7f169affd640 1 --2- 172.21.15.1:0/3998789149 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f168806b4d0 0x7f168806d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f1694034000 tx=0x7f169403b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.034+0000 7f16837fe640 1 -- 172.21.15.1:0/3998789149 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f169c1585d0 con 0x7f169c1ee7d0 2024-03-06T07:25:39.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.251+0000 7f16a169d640 1 -- 172.21.15.1:0/3998789149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f169c155d30 con 0x7f169c1ee7d0 2024-03-06T07:25:39.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.251+0000 7f16837fe640 1 -- 172.21.15.1:0/3998789149 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+4981 (secure 0 0 0) 0x7f168c058af0 con 0x7f169c1ee7d0 2024-03-06T07:25:39.253 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:39.253 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2848968195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2848968195},{"type":"v1","addr":"172.21.15.1:6835","nonce":2848968195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7387,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1671554096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1671554096},{"type":"v1","addr":"172.21.15.98:6836","nonce":1671554096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2739580331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2739580331},{"type":"v1","addr":"172.21.15.98:6837","nonce":2739580331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:29.792840+0000","modified":"2024-03-06T07:25:31.781726+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"a","rank":0,"incarnation":119,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1706279305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1706279305},{"type":"v1","addr":"172.21.15.1:6834","nonce":1706279305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8301,"qdb_cluster":[8301]},"id":10}]} 2024-03-06T07:25:39.254 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 121 2024-03-06T07:25:39.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.253+0000 7f16a169d640 1 -- 172.21.15.1:0/3998789149 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f168806b4d0 msgr2=0x7f168806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.253+0000 7f16a169d640 1 --2- 172.21.15.1:0/3998789149 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f168806b4d0 0x7f168806d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f1694034000 tx=0x7f169403b040 comp rx=0 tx=0).stop 2024-03-06T07:25:39.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.253+0000 7f16a169d640 1 -- 172.21.15.1:0/3998789149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f169c1ee7d0 msgr2=0x7f169c150320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.253+0000 7f16a169d640 1 --2- 172.21.15.1:0/3998789149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f169c1ee7d0 0x7f169c150320 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f169c08c760 tx=0x7f168c011cf0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.253+0000 7f16a169d640 1 -- 172.21.15.1:0/3998789149 shutdown_connections 2024-03-06T07:25:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.253+0000 7f16a169d640 1 --2- 172.21.15.1:0/3998789149 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f168806b4d0 0x7f168806d990 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.253+0000 7f16a169d640 1 --2- 172.21.15.1:0/3998789149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f169c1ee7d0 0x7f169c150320 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.253+0000 7f16a169d640 1 --2- 172.21.15.1:0/3998789149 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f169c14b370 0x7f169c1ee290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.253+0000 7f16a169d640 1 --2- 172.21.15.1:0/3998789149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16840039b0 0x7f169c158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.253+0000 7f16a169d640 1 -- 172.21.15.1:0/3998789149 >> 172.21.15.1:0/3998789149 conn(0x7f169c066ee0 msgr2=0x7f169c0c0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.253+0000 7f16a169d640 1 -- 172.21.15.1:0/3998789149 shutdown_connections 2024-03-06T07:25:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.253+0000 7f16a169d640 1 -- 172.21.15.1:0/3998789149 wait complete. 2024-03-06T07:25:39.668 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:29.792840+0000', 'modified': '2024-03-06T07:25:31.781726+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8301}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8301': {'gid': 8301, 'name': 'a', 'rank': 0, 'incarnation': 119, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6834/1706279305', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1706279305}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1706279305}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8301, 'qdb_cluster': [8301]} 2024-03-06T07:25:39.669 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:39.669 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:25:39.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.770+0000 7ff7acc1c640 1 Processor -- start 2024-03-06T07:25:39.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.770+0000 7ff7acc1c640 1 -- start start 2024-03-06T07:25:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.770+0000 7ff7acc1c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff7a80bf9d0 0x7ff7a8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.771+0000 7ff7acc1c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a80c2fc0 con 0x7ff7a8152650 2024-03-06T07:25:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.771+0000 7ff7acc1c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a8158da0 con 0x7ff7a80c6b10 2024-03-06T07:25:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.771+0000 7ff7acc1c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7a8155800 con 0x7ff7a80bf9d0 2024-03-06T07:25:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.771+0000 7ff7a6d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff7a80bf9d0 0x7ff7a8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.771+0000 7ff7a659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff7a8152650 0x7ff7a8158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39984/0 (socket says 172.21.15.1:39984) 2024-03-06T07:25:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.771+0000 7ff7a6d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff7a80bf9d0 0x7ff7a8155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48184/0 (socket says 172.21.15.1:48184) 2024-03-06T07:25:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.771+0000 7ff7a659c640 1 -- 172.21.15.1:0/1845494214 learned_addr learned my addr 172.21.15.1:0/1845494214 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.771+0000 7ff7a6d9d640 1 -- 172.21.15.1:0/1845494214 >> v1:172.21.15.98:6789/0 conn(0x7ff7a80c6b10 legacy=0x7ff7a80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.771+0000 7ff7a6d9d640 1 -- 172.21.15.1:0/1845494214 >> v1:172.21.15.1:6789/0 conn(0x7ff7a8152650 legacy=0x7ff7a8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.771+0000 7ff7a6d9d640 1 -- 172.21.15.1:0/1845494214 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a80c6690 con 0x7ff7a80bf9d0 2024-03-06T07:25:39.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.771+0000 7ff7a559a640 1 -- 172.21.15.1:0/1845494214 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3312675098 0 0) 0x7ff7a80c2fc0 con 0x7ff7a8152650 2024-03-06T07:25:39.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.772+0000 7ff7a6d9d640 1 --2- 172.21.15.1:0/1845494214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a80bf9d0 0x7ff7a8155c90 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7ff7a00097a0 tx=0x7ff7a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e01dd2588b45cf1b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.772+0000 7ff7a559a640 1 -- 172.21.15.1:0/1845494214 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0035020 con 0x7ff7a80bf9d0 2024-03-06T07:25:39.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.772+0000 7ff7a559a640 1 -- 172.21.15.1:0/1845494214 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a0031830 con 0x7ff7a80bf9d0 2024-03-06T07:25:39.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.772+0000 7ff7a559a640 1 -- 172.21.15.1:0/1845494214 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0031c00 con 0x7ff7a80bf9d0 2024-03-06T07:25:39.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.773+0000 7ff7acc1c640 1 -- 172.21.15.1:0/1845494214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a80bf9d0 msgr2=0x7ff7a8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.773+0000 7ff7acc1c640 1 --2- 172.21.15.1:0/1845494214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a80bf9d0 0x7ff7a8155c90 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7ff7a00097a0 tx=0x7ff7a002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.773+0000 7ff7acc1c640 1 -- 172.21.15.1:0/1845494214 shutdown_connections 2024-03-06T07:25:39.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.773+0000 7ff7acc1c640 1 --2- 172.21.15.1:0/1845494214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a80bf9d0 0x7ff7a8155c90 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.773+0000 7ff7acc1c640 1 -- 172.21.15.1:0/1845494214 >> 172.21.15.1:0/1845494214 conn(0x7ff7a8066ee0 msgr2=0x7ff7a80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.773+0000 7ff7acc1c640 1 -- 172.21.15.1:0/1845494214 shutdown_connections 2024-03-06T07:25:39.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.773+0000 7ff7acc1c640 1 -- 172.21.15.1:0/1845494214 wait complete. 2024-03-06T07:25:39.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.775+0000 7ff7acc1c640 1 Processor -- start 2024-03-06T07:25:39.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.775+0000 7ff7acc1c640 1 -- start start 2024-03-06T07:25:39.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7acc1c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7a80bf9d0 0x7ff7a81ee390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7acc1c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a8152650 0x7ff7a80c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7acc1c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7a81ee8d0 0x7ff7a80c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7acc1c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a81551b0 con 0x7ff7a8152650 2024-03-06T07:25:39.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7acc1c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a80c6fa0 con 0x7ff7a81ee8d0 2024-03-06T07:25:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7acc1c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a8155490 con 0x7ff7a80bf9d0 2024-03-06T07:25:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7a5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a8152650 0x7ff7a80c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7a5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a8152650 0x7ff7a80c9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48194/0 (socket says 172.21.15.1:48194) 2024-03-06T07:25:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7a5d9b640 1 -- 172.21.15.1:0/1336779321 learned_addr learned my addr 172.21.15.1:0/1336779321 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7a659c640 1 --2- 172.21.15.1:0/1336779321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7a80bf9d0 0x7ff7a81ee390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7a6d9d640 1 --2- 172.21.15.1:0/1336779321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7a81ee8d0 0x7ff7a80c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7a5d9b640 1 -- 172.21.15.1:0/1336779321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7a80bf9d0 msgr2=0x7ff7a81ee390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7a5d9b640 1 --2- 172.21.15.1:0/1336779321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7a80bf9d0 0x7ff7a81ee390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7a5d9b640 1 -- 172.21.15.1:0/1336779321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7a81ee8d0 msgr2=0x7ff7a80c9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7a5d9b640 1 --2- 172.21.15.1:0/1336779321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7a81ee8d0 0x7ff7a80c9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7a5d9b640 1 -- 172.21.15.1:0/1336779321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a81eed80 con 0x7ff7a8152650 2024-03-06T07:25:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7a659c640 1 --2- 172.21.15.1:0/1336779321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7a80bf9d0 0x7ff7a81ee390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7a6d9d640 1 --2- 172.21.15.1:0/1336779321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7a81ee8d0 0x7ff7a80c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.776+0000 7ff7a5d9b640 1 --2- 172.21.15.1:0/1336779321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a8152650 0x7ff7a80c9670 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7ff79000c450 tx=0x7ff79000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.777+0000 7ff79effd640 1 -- 172.21.15.1:0/1336779321 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7900052d0 con 0x7ff7a8152650 2024-03-06T07:25:39.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.777+0000 7ff79effd640 1 -- 172.21.15.1:0/1336779321 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff790005470 con 0x7ff7a8152650 2024-03-06T07:25:39.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.777+0000 7ff7acc1c640 1 -- 172.21.15.1:0/1336779321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a8153f60 con 0x7ff7a8152650 2024-03-06T07:25:39.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.777+0000 7ff79effd640 1 -- 172.21.15.1:0/1336779321 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff790012870 con 0x7ff7a8152650 2024-03-06T07:25:39.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.777+0000 7ff7acc1c640 1 -- 172.21.15.1:0/1336779321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a8153230 con 0x7ff7a8152650 2024-03-06T07:25:39.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.779+0000 7ff79effd640 1 -- 172.21.15.1:0/1336779321 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff790012a10 con 0x7ff7a8152650 2024-03-06T07:25:39.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.779+0000 7ff79effd640 1 --2- 172.21.15.1:0/1336779321 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff78806b4d0 0x7ff78806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.779+0000 7ff79effd640 1 -- 172.21.15.1:0/1336779321 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff790089f50 con 0x7ff7a8152650 2024-03-06T07:25:39.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.780+0000 7ff7a659c640 1 --2- 172.21.15.1:0/1336779321 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff78806b4d0 0x7ff78806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.780+0000 7ff7af808740 1 Processor -- start 2024-03-06T07:25:39.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.780+0000 7ff7af808740 1 -- start start 2024-03-06T07:25:39.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.780+0000 7ff7a659c640 1 --2- 172.21.15.1:0/1336779321 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff78806b4d0 0x7ff78806d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff798007fd0 tx=0x7ff798005b70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.781+0000 7ff7af808740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a00cf50 0x561d6a00aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.781+0000 7ff7af808740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a00af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.781+0000 7ff7af808740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00b4b0 0x561d6a007680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.781+0000 7ff7af808740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d69f716b0 con 0x561d6a00d330 2024-03-06T07:25:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.781+0000 7ff7af808740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d69f1aa60 con 0x561d6a00cf50 2024-03-06T07:25:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.781+0000 7ff7af808740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x561d69f6d750 con 0x561d6a00b4b0 2024-03-06T07:25:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.781+0000 7ff7a5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a00af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.781+0000 7ff7a5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a00af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48202/0 (socket says 172.21.15.1:48202) 2024-03-06T07:25:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.781+0000 7ff7a5d9b640 1 -- 172.21.15.1:0/966341496 learned_addr learned my addr 172.21.15.1:0/966341496 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.781+0000 7ff7a6d9d640 1 --2- 172.21.15.1:0/966341496 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00b4b0 0x561d6a007680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.781+0000 7ff7a659c640 1 --2- 172.21.15.1:0/966341496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a00cf50 0x561d6a00aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.781+0000 7ff7a5d9b640 1 -- 172.21.15.1:0/966341496 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00b4b0 msgr2=0x561d6a007680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.781+0000 7ff7a5d9b640 1 --2- 172.21.15.1:0/966341496 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00b4b0 0x561d6a007680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.782+0000 7ff7a5d9b640 1 -- 172.21.15.1:0/966341496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a00cf50 msgr2=0x561d6a00aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.782+0000 7ff7a5d9b640 1 --2- 172.21.15.1:0/966341496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a00cf50 0x561d6a00aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.782+0000 7ff7a5d9b640 1 -- 172.21.15.1:0/966341496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d69f38f30 con 0x561d6a00d330 2024-03-06T07:25:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.782+0000 7ff7a659c640 1 --2- 172.21.15.1:0/966341496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a00cf50 0x561d6a00aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.782+0000 7ff7a5d9b640 1 --2- 172.21.15.1:0/966341496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a00af70 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7ff790000f30 tx=0x7ff790058e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d06784ff39b0ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.782+0000 7ff787fff640 1 -- 172.21.15.1:0/966341496 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff790060ce0 con 0x561d6a00d330 2024-03-06T07:25:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.782+0000 7ff787fff640 1 -- 172.21.15.1:0/966341496 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7900568d0 con 0x561d6a00d330 2024-03-06T07:25:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.782+0000 7ff787fff640 1 -- 172.21.15.1:0/966341496 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff790054c40 con 0x561d6a00d330 2024-03-06T07:25:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.782+0000 7ff7af808740 1 -- 172.21.15.1:0/966341496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 msgr2=0x561d6a00af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.782+0000 7ff7af808740 1 --2- 172.21.15.1:0/966341496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a00af70 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7ff790000f30 tx=0x7ff790058e10 comp rx=0 tx=0).stop 2024-03-06T07:25:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.782+0000 7ff7af808740 1 -- 172.21.15.1:0/966341496 shutdown_connections 2024-03-06T07:25:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.782+0000 7ff7af808740 1 --2- 172.21.15.1:0/966341496 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00b4b0 0x561d6a007680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.782+0000 7ff7af808740 1 --2- 172.21.15.1:0/966341496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a00af70 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.782+0000 7ff7af808740 1 --2- 172.21.15.1:0/966341496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a00cf50 0x561d6a00aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.783+0000 7ff7af808740 1 -- 172.21.15.1:0/966341496 >> 172.21.15.1:0/966341496 conn(0x561d6a0097d0 msgr2=0x561d6a02e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.783+0000 7ff7af808740 1 -- 172.21.15.1:0/966341496 shutdown_connections 2024-03-06T07:25:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.783+0000 7ff7af808740 1 -- 172.21.15.1:0/966341496 wait complete. 2024-03-06T07:25:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.783+0000 7ff7af808740 1 -- start start 2024-03-06T07:25:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.784+0000 7ff7af808740 1 Processor -- start 2024-03-06T07:25:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7af808740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a00cf50 0x561d6a02dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7af808740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00d330 0x561d6a031f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7af808740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a02f970 0x561d6a02fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7af808740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d6a02f230 con 0x561d6a02f970 2024-03-06T07:25:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7af808740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d69f2dea0 con 0x561d6a00cf50 2024-03-06T07:25:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7af808740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x561d6a02ec70 con 0x561d6a00d330 2024-03-06T07:25:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7af808740 10 client.0 _ll_register_callbacks cb 0x561d69f2d870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7a6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a02f970 0x561d6a02fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7a6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a02f970 0x561d6a02fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48218/0 (socket says 172.21.15.1:48218) 2024-03-06T07:25:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7a6d9d640 1 -- 172.21.15.1:0/1477244429 learned_addr learned my addr 172.21.15.1:0/1477244429 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7a659c640 1 --2- 172.21.15.1:0/1477244429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a00cf50 0x561d6a02dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7a5d9b640 1 --2- 172.21.15.1:0/1477244429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00d330 0x561d6a031f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7a6d9d640 1 -- 172.21.15.1:0/1477244429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00d330 msgr2=0x561d6a031f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7a6d9d640 1 --2- 172.21.15.1:0/1477244429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00d330 0x561d6a031f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7a6d9d640 1 -- 172.21.15.1:0/1477244429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a00cf50 msgr2=0x561d6a02dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7a6d9d640 1 --2- 172.21.15.1:0/1477244429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a00cf50 0x561d6a02dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7a6d9d640 1 -- 172.21.15.1:0/1477244429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d69f2e270 con 0x561d6a02f970 2024-03-06T07:25:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.785+0000 7ff7a659c640 1 --2- 172.21.15.1:0/1477244429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a00cf50 0x561d6a02dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.786+0000 7ff7a6d9d640 1 --2- 172.21.15.1:0/1477244429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a02f970 0x561d6a02fdf0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7ff7a00098c0 tx=0x7ff7a002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce5254e5c54b5f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.786+0000 7ff7857fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:25:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.786+0000 7ff7857fa640 1 -- 172.21.15.1:0/1477244429 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a004b060 con 0x561d6a02f970 2024-03-06T07:25:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.786+0000 7ff7857fa640 1 -- 172.21.15.1:0/1477244429 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff7a0031b00 con 0x561d6a02f970 2024-03-06T07:25:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.786+0000 7ff7af808740 1 -- 172.21.15.1:0/1477244429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff7900568d0 con 0x561d6a02f970 2024-03-06T07:25:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.786+0000 7ff7857fa640 1 -- 172.21.15.1:0/1477244429 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7a0051050 con 0x561d6a02f970 2024-03-06T07:25:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.786+0000 7ff7af808740 10 client.8460 fetch_fsmap learned FSMap version 121 2024-03-06T07:25:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.786+0000 7ff7af808740 1 -- 172.21.15.1:0/1477244429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=2+}) v3 -- 0x561d69c862d0 con 0x561d6a02f970 2024-03-06T07:25:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.786+0000 7ff7857fa640 1 -- 172.21.15.1:0/1477244429 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7900568d0 con 0x561d6a02f970 2024-03-06T07:25:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.786+0000 7ff7857fa640 1 -- 172.21.15.1:0/1477244429 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 121) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff7a0054020 con 0x561d6a02f970 2024-03-06T07:25:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.786+0000 7ff7af808740 10 client.8460 fetch_fsmap finished waiting for FSMap version 121 2024-03-06T07:25:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.786+0000 7ff7af808740 10 client.8460 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:25:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.786+0000 7ff7af808740 20 client.8460 populate_metadata read hostname 'smithi001' 2024-03-06T07:25:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.786+0000 7ff7af808740 1 --2- 172.21.15.1:0/1477244429 >> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] conn(0x561d6a05d7e0 0x561d6a068570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.786+0000 7ff7af808740 4 client.8460 mds_command: new command op to 8301 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.786+0000 7ff7af808740 1 -- 172.21.15.1:0/1477244429 --> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561d69f7ebf0 con 0x561d6a05d7e0 2024-03-06T07:25:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.786+0000 7ff7a659c640 1 --2- 172.21.15.1:0/1477244429 >> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] conn(0x561d6a05d7e0 0x561d6a068570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.787+0000 7ff7a659c640 1 --2- 172.21.15.1:0/1477244429 >> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] conn(0x561d6a05d7e0 0x561d6a068570 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed529391db6623c6 server_cookie=d7a4bccd223f2997 in_seq=0 out_seq=0 2024-03-06T07:25:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.787+0000 7ff7857fa640 10 client.8460 ms_handle_connect on v2:172.21.15.1:6832/1706279305 2024-03-06T07:25:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.788+0000 7ff7857fa640 1 -- 172.21.15.1:0/1477244429 <== mds.0 v2:172.21.15.1:6832/1706279305 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561d69f7ebf0 con 0x561d6a05d7e0 2024-03-06T07:25:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.788+0000 7ff7857fa640 10 client.8460 handle_command_reply: tid=0 2024-03-06T07:25:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.788+0000 7ff7af808740 1 client.8460 shutdown 2024-03-06T07:25:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.788+0000 7ff7af808740 1 -- 172.21.15.1:0/1477244429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a02f970 msgr2=0x561d6a02fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.788+0000 7ff7af808740 1 --2- 172.21.15.1:0/1477244429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a02f970 0x561d6a02fdf0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7ff7a00098c0 tx=0x7ff7a002e000 comp rx=0 tx=0).stop 2024-03-06T07:25:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.788+0000 7ff7af808740 1 -- 172.21.15.1:0/1477244429 shutdown_connections 2024-03-06T07:25:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.788+0000 7ff7af808740 1 --2- 172.21.15.1:0/1477244429 >> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] conn(0x561d6a05d7e0 0x561d6a068570 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.788+0000 7ff7af808740 1 --2- 172.21.15.1:0/1477244429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a02f970 0x561d6a02fdf0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.788+0000 7ff7af808740 1 --2- 172.21.15.1:0/1477244429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00d330 0x561d6a031f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.788+0000 7ff7857fa640 0 client.8460 ms_handle_reset on v2:172.21.15.1:6832/1706279305 2024-03-06T07:25:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.788+0000 7ff7af808740 1 --2- 172.21.15.1:0/1477244429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a00cf50 0x561d6a02dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.788+0000 7ff7af808740 1 -- 172.21.15.1:0/1477244429 >> 172.21.15.1:0/1477244429 conn(0x561d6a00b4b0 msgr2=0x561d6a00c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.788+0000 7ff7af808740 1 -- 172.21.15.1:0/1477244429 shutdown_connections 2024-03-06T07:25:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.788+0000 7ff7af808740 1 -- 172.21.15.1:0/1477244429 wait complete. 2024-03-06T07:25:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.788+0000 7ff7af808740 20 client.8460 trim_cache size 0 max 16384 2024-03-06T07:25:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.805+0000 7ff7af808740 1 Processor -- start 2024-03-06T07:25:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.805+0000 7ff7af808740 1 -- start start 2024-03-06T07:25:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.805+0000 7ff7af808740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00cf50 0x561d6a06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.805+0000 7ff7af808740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a00a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.805+0000 7ff7af808740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a02f970 0x561d6a02cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.805+0000 7ff7af808740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a81f7c20 con 0x561d6a00d330 2024-03-06T07:25:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.805+0000 7ff7af808740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a81f6d90 con 0x561d6a02f970 2024-03-06T07:25:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.805+0000 7ff7af808740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a81eb5e0 con 0x561d6a00cf50 2024-03-06T07:25:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.806+0000 7ff7a5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a00a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.806+0000 7ff7a5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a00a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48226/0 (socket says 172.21.15.1:48226) 2024-03-06T07:25:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.806+0000 7ff7a5d9b640 1 -- 172.21.15.1:0/3009232456 learned_addr learned my addr 172.21.15.1:0/3009232456 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.806+0000 7ff7a659c640 1 --2- 172.21.15.1:0/3009232456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00cf50 0x561d6a06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.806+0000 7ff7a6d9d640 1 --2- 172.21.15.1:0/3009232456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a02f970 0x561d6a02cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.806+0000 7ff7a5d9b640 1 -- 172.21.15.1:0/3009232456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00cf50 msgr2=0x561d6a06d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.806+0000 7ff7a5d9b640 1 --2- 172.21.15.1:0/3009232456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00cf50 0x561d6a06d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.806+0000 7ff7a5d9b640 1 -- 172.21.15.1:0/3009232456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a02f970 msgr2=0x561d6a02cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.806+0000 7ff7a5d9b640 1 --2- 172.21.15.1:0/3009232456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a02f970 0x561d6a02cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.806+0000 7ff7a5d9b640 1 -- 172.21.15.1:0/3009232456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d69d506b0 con 0x561d6a00d330 2024-03-06T07:25:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.806+0000 7ff7a6d9d640 1 --2- 172.21.15.1:0/3009232456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a02f970 0x561d6a02cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:39.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.806+0000 7ff7a5d9b640 1 --2- 172.21.15.1:0/3009232456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a00a5a0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7ff79000ce10 tx=0x7ff79005b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f2cb01e810c3e99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.806+0000 7ff7877fe640 1 -- 172.21.15.1:0/3009232456 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff790086070 con 0x561d6a00d330 2024-03-06T07:25:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.806+0000 7ff7877fe640 1 -- 172.21.15.1:0/3009232456 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff790060ce0 con 0x561d6a00d330 2024-03-06T07:25:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.806+0000 7ff7877fe640 1 -- 172.21.15.1:0/3009232456 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff790058110 con 0x561d6a00d330 2024-03-06T07:25:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.807+0000 7ff7af808740 1 -- 172.21.15.1:0/3009232456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 msgr2=0x561d6a00a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.807+0000 7ff7af808740 1 --2- 172.21.15.1:0/3009232456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a00a5a0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7ff79000ce10 tx=0x7ff79005b000 comp rx=0 tx=0).stop 2024-03-06T07:25:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.807+0000 7ff7af808740 1 -- 172.21.15.1:0/3009232456 shutdown_connections 2024-03-06T07:25:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.807+0000 7ff7af808740 1 --2- 172.21.15.1:0/3009232456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a02f970 0x561d6a02cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.807+0000 7ff7af808740 1 --2- 172.21.15.1:0/3009232456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a00a5a0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.807+0000 7ff7af808740 1 --2- 172.21.15.1:0/3009232456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00cf50 0x561d6a06d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.807+0000 7ff7af808740 1 -- 172.21.15.1:0/3009232456 >> 172.21.15.1:0/3009232456 conn(0x561d6a00b4b0 msgr2=0x561d6a04ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.807+0000 7ff7af808740 1 -- 172.21.15.1:0/3009232456 shutdown_connections 2024-03-06T07:25:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.807+0000 7ff7af808740 1 -- 172.21.15.1:0/3009232456 wait complete. 2024-03-06T07:25:39.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.807+0000 7ff7af808740 1 -- start start 2024-03-06T07:25:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.807+0000 7ff7af808740 1 Processor -- start 2024-03-06T07:25:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.808+0000 7ff7af808740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00cf50 0x561d6a04cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.808+0000 7ff7af808740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a0b67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.808+0000 7ff7af808740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a02f970 0x561d6a0b6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.808+0000 7ff7af808740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d6a042ee0 con 0x561d6a00d330 2024-03-06T07:25:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.808+0000 7ff7af808740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a81f6870 con 0x561d6a02f970 2024-03-06T07:25:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.808+0000 7ff7af808740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x561d6a05a640 con 0x561d6a00cf50 2024-03-06T07:25:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.808+0000 7ff7af808740 10 client.0 _ll_register_callbacks cb 0x7ff7a00041a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.808+0000 7ff7a5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a0b67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.808+0000 7ff7a5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a0b67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48240/0 (socket says 172.21.15.1:48240) 2024-03-06T07:25:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.808+0000 7ff7a5d9b640 1 -- 172.21.15.1:0/1926629217 learned_addr learned my addr 172.21.15.1:0/1926629217 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7a659c640 1 --2- 172.21.15.1:0/1926629217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00cf50 0x561d6a04cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7a5d9b640 1 -- 172.21.15.1:0/1926629217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00cf50 msgr2=0x561d6a04cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7a6d9d640 1 --2- 172.21.15.1:0/1926629217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a02f970 0x561d6a0b6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7a5d9b640 1 --2- 172.21.15.1:0/1926629217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00cf50 0x561d6a04cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7a5d9b640 1 -- 172.21.15.1:0/1926629217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a02f970 msgr2=0x561d6a0b6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7a5d9b640 1 --2- 172.21.15.1:0/1926629217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a02f970 0x561d6a0b6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7a5d9b640 1 -- 172.21.15.1:0/1926629217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d6a02d670 con 0x561d6a00d330 2024-03-06T07:25:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7a5d9b640 1 --2- 172.21.15.1:0/1926629217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a0b67c0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7ff790007fd0 tx=0x7ff790001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be036e67ae7ff229 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7857fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:25:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7857fa640 1 -- 172.21.15.1:0/1926629217 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff790077060 con 0x561d6a00d330 2024-03-06T07:25:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7857fa640 1 -- 172.21.15.1:0/1926629217 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff790054c40 con 0x561d6a00d330 2024-03-06T07:25:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7af808740 1 -- 172.21.15.1:0/1926629217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff790060ce0 con 0x561d6a00d330 2024-03-06T07:25:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7857fa640 1 -- 172.21.15.1:0/1926629217 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff790086070 con 0x561d6a00d330 2024-03-06T07:25:39.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7af808740 10 client.8466 fetch_fsmap learned FSMap version 121 2024-03-06T07:25:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7af808740 1 -- 172.21.15.1:0/1926629217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=2+}) v3 -- 0x561d6a040e90 con 0x561d6a00d330 2024-03-06T07:25:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7857fa640 1 -- 172.21.15.1:0/1926629217 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff790060ce0 con 0x561d6a00d330 2024-03-06T07:25:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7857fa640 1 -- 172.21.15.1:0/1926629217 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 121) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff790078ce0 con 0x561d6a00d330 2024-03-06T07:25:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7af808740 10 client.8466 fetch_fsmap finished waiting for FSMap version 121 2024-03-06T07:25:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7af808740 10 client.8466 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:25:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7af808740 20 client.8466 populate_metadata read hostname 'smithi001' 2024-03-06T07:25:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7af808740 1 --2- 172.21.15.1:0/1926629217 >> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] conn(0x561d6a0bcba0 0x561d6a03ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7af808740 4 client.8466 mds_command: new command op to 8301 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.809+0000 7ff7af808740 1 -- 172.21.15.1:0/1926629217 --> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561d6a04bb00 con 0x561d6a0bcba0 2024-03-06T07:25:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7a659c640 1 --2- 172.21.15.1:0/1926629217 >> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] conn(0x561d6a0bcba0 0x561d6a03ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7857fa640 10 client.8466 ms_handle_connect on v2:172.21.15.1:6832/1706279305 2024-03-06T07:25:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7a659c640 1 --2- 172.21.15.1:0/1926629217 >> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] conn(0x561d6a0bcba0 0x561d6a03ea90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ac7e0a3a5bc4f89 server_cookie=c6ebfac0418a1808 in_seq=0 out_seq=0 2024-03-06T07:25:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7857fa640 1 -- 172.21.15.1:0/1926629217 <== mds.0 v2:172.21.15.1:6832/1706279305 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x561d6a04bb00 con 0x561d6a0bcba0 2024-03-06T07:25:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7857fa640 10 client.8466 handle_command_reply: tid=0 2024-03-06T07:25:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7af808740 1 client.8466 shutdown 2024-03-06T07:25:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7af808740 1 -- 172.21.15.1:0/1926629217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 msgr2=0x561d6a0b67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7af808740 1 --2- 172.21.15.1:0/1926629217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a0b67c0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7ff790007fd0 tx=0x7ff790001120 comp rx=0 tx=0).stop 2024-03-06T07:25:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7af808740 1 -- 172.21.15.1:0/1926629217 shutdown_connections 2024-03-06T07:25:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7af808740 1 --2- 172.21.15.1:0/1926629217 >> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] conn(0x561d6a0bcba0 0x561d6a03ea90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7af808740 1 --2- 172.21.15.1:0/1926629217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561d6a02f970 0x561d6a0b6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7af808740 1 --2- 172.21.15.1:0/1926629217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561d6a00d330 0x561d6a0b67c0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7857fa640 0 client.8466 ms_handle_reset on v2:172.21.15.1:6832/1706279305 2024-03-06T07:25:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7af808740 1 --2- 172.21.15.1:0/1926629217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x561d6a00cf50 0x561d6a04cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7af808740 1 -- 172.21.15.1:0/1926629217 >> 172.21.15.1:0/1926629217 conn(0x561d6a00b4b0 msgr2=0x561d6a0072e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7af808740 1 -- 172.21.15.1:0/1926629217 shutdown_connections 2024-03-06T07:25:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7af808740 1 -- 172.21.15.1:0/1926629217 wait complete. 2024-03-06T07:25:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.810+0000 7ff7af808740 20 client.8466 trim_cache size 0 max 16384 2024-03-06T07:25:39.819 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:25:39.819 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:25:39.819 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:25:39.819 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 8301, 2024-03-06T07:25:39.819 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:25:39.819 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:25:39.819 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:25:39.819 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 9.3892918390000002, 2024-03-06T07:25:39.819 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 121, 2024-03-06T07:25:39.819 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 135, 2024-03-06T07:25:39.819 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 135, 2024-03-06T07:25:39.819 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 14.461387799000001 2024-03-06T07:25:39.819 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:25:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.811+0000 7ff7acc1c640 1 -- 172.21.15.1:0/1336779321 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff78806b4d0 msgr2=0x7ff78806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.811+0000 7ff7acc1c640 1 --2- 172.21.15.1:0/1336779321 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff78806b4d0 0x7ff78806d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff798007fd0 tx=0x7ff798005b70 comp rx=0 tx=0).stop 2024-03-06T07:25:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.811+0000 7ff7acc1c640 1 -- 172.21.15.1:0/1336779321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a8152650 msgr2=0x7ff7a80c9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.811+0000 7ff7acc1c640 1 --2- 172.21.15.1:0/1336779321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a8152650 0x7ff7a80c9670 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7ff79000c450 tx=0x7ff79000c910 comp rx=0 tx=0).stop 2024-03-06T07:25:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.811+0000 7ff7acc1c640 1 -- 172.21.15.1:0/1336779321 shutdown_connections 2024-03-06T07:25:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.811+0000 7ff7acc1c640 1 --2- 172.21.15.1:0/1336779321 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff78806b4d0 0x7ff78806d990 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.811+0000 7ff7acc1c640 1 --2- 172.21.15.1:0/1336779321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7a81ee8d0 0x7ff7a80c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.811+0000 7ff7acc1c640 1 --2- 172.21.15.1:0/1336779321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a8152650 0x7ff7a80c9670 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.811+0000 7ff7acc1c640 1 --2- 172.21.15.1:0/1336779321 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7a80bf9d0 0x7ff7a81ee390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.811+0000 7ff7acc1c640 1 -- 172.21.15.1:0/1336779321 >> 172.21.15.1:0/1336779321 conn(0x7ff7a8066ee0 msgr2=0x7ff7a80c5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.811+0000 7ff7acc1c640 1 -- 172.21.15.1:0/1336779321 shutdown_connections 2024-03-06T07:25:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.811+0000 7ff7acc1c640 1 -- 172.21.15.1:0/1336779321 wait complete. 2024-03-06T07:25:39.824 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:39.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.923+0000 7f78cfe81640 1 Processor -- start 2024-03-06T07:25:39.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.923+0000 7f78cfe81640 1 -- start start 2024-03-06T07:25:39.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.924+0000 7f78cfe81640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f78c8152650 0x7f78c8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.924+0000 7f78cfe81640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f78c80bf9d0 0x7f78c80c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.924+0000 7f78cfe81640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c80c2fc0 con 0x7f78c80c6b10 2024-03-06T07:25:39.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.924+0000 7f78cfe81640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c8155800 con 0x7f78c8152650 2024-03-06T07:25:39.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.924+0000 7f78cfe81640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c8158da0 con 0x7f78c80bf9d0 2024-03-06T07:25:39.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.924+0000 7f78cdc1d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f78c8152650 0x7f78c8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.924+0000 7f78cdc1d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f78c8152650 0x7f78c8155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48242/0 (socket says 172.21.15.1:48242) 2024-03-06T07:25:39.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.924+0000 7f78cdc1d640 1 -- 172.21.15.1:0/3969405806 learned_addr learned my addr 172.21.15.1:0/3969405806 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.924+0000 7f78cdc1d640 1 -- 172.21.15.1:0/3969405806 >> v2:172.21.15.98:3300/0 conn(0x7f78c80bf9d0 msgr2=0x7f78c80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.924+0000 7f78cdc1d640 1 --2- 172.21.15.1:0/3969405806 >> v2:172.21.15.98:3300/0 conn(0x7f78c80bf9d0 0x7f78c80c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.924+0000 7f78cdc1d640 1 -- 172.21.15.1:0/3969405806 >> v1:172.21.15.98:6789/0 conn(0x7f78c80c6b10 legacy=0x7f78c8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.924+0000 7f78cdc1d640 1 -- 172.21.15.1:0/3969405806 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c80c6690 con 0x7f78c8152650 2024-03-06T07:25:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.925+0000 7f78cdc1d640 1 --2- 172.21.15.1:0/3969405806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78c8152650 0x7f78c8155c90 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f78c81597b0 tx=0x7f78c002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42841252cfb89ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.925+0000 7f78ccc1b640 1 -- 172.21.15.1:0/3969405806 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c002e070 con 0x7f78c8152650 2024-03-06T07:25:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.925+0000 7f78ccc1b640 1 -- 172.21.15.1:0/3969405806 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c0037070 con 0x7f78c8152650 2024-03-06T07:25:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.925+0000 7f78ccc1b640 1 -- 172.21.15.1:0/3969405806 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c0031a00 con 0x7f78c8152650 2024-03-06T07:25:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.926+0000 7f78cfe81640 1 -- 172.21.15.1:0/3969405806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78c8152650 msgr2=0x7f78c8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.926+0000 7f78cfe81640 1 --2- 172.21.15.1:0/3969405806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78c8152650 0x7f78c8155c90 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f78c81597b0 tx=0x7f78c002b880 comp rx=0 tx=0).stop 2024-03-06T07:25:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.926+0000 7f78cfe81640 1 -- 172.21.15.1:0/3969405806 shutdown_connections 2024-03-06T07:25:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.926+0000 7f78cfe81640 1 --2- 172.21.15.1:0/3969405806 >> v2:172.21.15.98:3300/0 conn(0x7f78c80bf9d0 0x7f78c80c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.926+0000 7f78cfe81640 1 --2- 172.21.15.1:0/3969405806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78c8152650 0x7f78c8155c90 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.926+0000 7f78cfe81640 1 -- 172.21.15.1:0/3969405806 >> 172.21.15.1:0/3969405806 conn(0x7f78c8066ee0 msgr2=0x7f78c80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.926+0000 7f78cfe81640 1 -- 172.21.15.1:0/3969405806 shutdown_connections 2024-03-06T07:25:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.926+0000 7f78cfe81640 1 -- 172.21.15.1:0/3969405806 wait complete. 2024-03-06T07:25:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.927+0000 7f78cfe81640 1 Processor -- start 2024-03-06T07:25:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.927+0000 7f78cfe81640 1 -- start start 2024-03-06T07:25:39.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.928+0000 7f78cfe81640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78c80bf9d0 0x7f78c80c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.928+0000 7f78cfe81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78c8152650 0x7f78c8154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.928+0000 7f78cfe81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78c8148300 0x7f78c81487b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.928+0000 7f78cfe81640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c81551b0 con 0x7f78c80bf9d0 2024-03-06T07:25:39.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.928+0000 7f78cfe81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c80c6fa0 con 0x7f78c8148300 2024-03-06T07:25:39.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.928+0000 7f78cfe81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c8155490 con 0x7f78c8152650 2024-03-06T07:25:39.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.928+0000 7f78cdc1d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78c80bf9d0 0x7f78c80c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.928+0000 7f78cdc1d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78c80bf9d0 0x7f78c80c9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48256/0 (socket says 172.21.15.1:48256) 2024-03-06T07:25:39.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.928+0000 7f78cdc1d640 1 -- 172.21.15.1:0/1880724092 learned_addr learned my addr 172.21.15.1:0/1880724092 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:39.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.928+0000 7f78cd41c640 1 --2- 172.21.15.1:0/1880724092 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78c8152650 0x7f78c8154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.928+0000 7f78ce41e640 1 --2- 172.21.15.1:0/1880724092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78c8148300 0x7f78c81487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.928+0000 7f78cdc1d640 1 -- 172.21.15.1:0/1880724092 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78c8152650 msgr2=0x7f78c8154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.928+0000 7f78cdc1d640 1 --2- 172.21.15.1:0/1880724092 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78c8152650 0x7f78c8154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.928+0000 7f78cdc1d640 1 -- 172.21.15.1:0/1880724092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78c8148300 msgr2=0x7f78c81487b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.928+0000 7f78cdc1d640 1 --2- 172.21.15.1:0/1880724092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78c8148300 0x7f78c81487b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.928+0000 7f78cdc1d640 1 -- 172.21.15.1:0/1880724092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c814ff20 con 0x7f78c80bf9d0 2024-03-06T07:25:39.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.929+0000 7f78cdc1d640 1 --2- 172.21.15.1:0/1880724092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78c80bf9d0 0x7f78c80c9880 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f78c002b8c0 tx=0x7f78c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.929+0000 7f78c67fc640 1 -- 172.21.15.1:0/1880724092 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c0032880 con 0x7f78c80bf9d0 2024-03-06T07:25:39.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.929+0000 7f78c67fc640 1 -- 172.21.15.1:0/1880724092 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c0032a20 con 0x7f78c80bf9d0 2024-03-06T07:25:39.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.929+0000 7f78cfe81640 1 -- 172.21.15.1:0/1880724092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c81eb6c0 con 0x7f78c80bf9d0 2024-03-06T07:25:39.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.929+0000 7f78c67fc640 1 -- 172.21.15.1:0/1880724092 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c0032e10 con 0x7f78c80bf9d0 2024-03-06T07:25:39.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.929+0000 7f78cfe81640 1 -- 172.21.15.1:0/1880724092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c81ebc80 con 0x7f78c80bf9d0 2024-03-06T07:25:39.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.930+0000 7f78cfe81640 1 -- 172.21.15.1:0/1880724092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c8067fd0 con 0x7f78c80bf9d0 2024-03-06T07:25:39.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.931+0000 7f78c67fc640 1 -- 172.21.15.1:0/1880724092 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f78c00453f0 con 0x7f78c80bf9d0 2024-03-06T07:25:39.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.931+0000 7f78c67fc640 1 --2- 172.21.15.1:0/1880724092 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f78b406b500 0x7f78b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.931+0000 7f78c67fc640 1 -- 172.21.15.1:0/1880724092 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f78c00ac3c0 con 0x7f78c80bf9d0 2024-03-06T07:25:39.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.931+0000 7f78cd41c640 1 --2- 172.21.15.1:0/1880724092 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f78b406b500 0x7f78b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.932+0000 7f78cd41c640 1 --2- 172.21.15.1:0/1880724092 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f78b406b500 0x7f78b406d9c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f78c80b05f0 tx=0x7f78bc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:39.936+0000 7f78c67fc640 1 -- 172.21.15.1:0/1880724092 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78c0040070 con 0x7f78c80bf9d0 2024-03-06T07:25:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.152+0000 7f78cfe81640 1 -- 172.21.15.1:0/1880724092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78c8151e00 con 0x7f78c80bf9d0 2024-03-06T07:25:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.153+0000 7f78c67fc640 1 -- 172.21.15.1:0/1880724092 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+4981 (secure 0 0 0) 0x7f78c007b1d0 con 0x7f78c80bf9d0 2024-03-06T07:25:40.155 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:40.155 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2848968195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2848968195},{"type":"v1","addr":"172.21.15.1:6835","nonce":2848968195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7387,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1671554096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1671554096},{"type":"v1","addr":"172.21.15.98:6836","nonce":1671554096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2739580331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2739580331},{"type":"v1","addr":"172.21.15.98:6837","nonce":2739580331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:29.792840+0000","modified":"2024-03-06T07:25:31.781726+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"a","rank":0,"incarnation":119,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1706279305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1706279305},{"type":"v1","addr":"172.21.15.1:6834","nonce":1706279305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8301,"qdb_cluster":[8301]},"id":10}]} 2024-03-06T07:25:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 121 2024-03-06T07:25:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.155+0000 7f78cfe81640 1 -- 172.21.15.1:0/1880724092 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f78b406b500 msgr2=0x7f78b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.155+0000 7f78cfe81640 1 --2- 172.21.15.1:0/1880724092 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f78b406b500 0x7f78b406d9c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f78c80b05f0 tx=0x7f78bc009040 comp rx=0 tx=0).stop 2024-03-06T07:25:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.155+0000 7f78cfe81640 1 -- 172.21.15.1:0/1880724092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78c80bf9d0 msgr2=0x7f78c80c9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.155+0000 7f78cfe81640 1 --2- 172.21.15.1:0/1880724092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78c80bf9d0 0x7f78c80c9880 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f78c002b8c0 tx=0x7f78c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.155+0000 7f78cfe81640 1 -- 172.21.15.1:0/1880724092 shutdown_connections 2024-03-06T07:25:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.155+0000 7f78cfe81640 1 --2- 172.21.15.1:0/1880724092 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f78b406b500 0x7f78b406d9c0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.155+0000 7f78cfe81640 1 --2- 172.21.15.1:0/1880724092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78c8148300 0x7f78c81487b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.155+0000 7f78cfe81640 1 --2- 172.21.15.1:0/1880724092 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78c8152650 0x7f78c8154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.155+0000 7f78cfe81640 1 --2- 172.21.15.1:0/1880724092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78c80bf9d0 0x7f78c80c9880 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.155+0000 7f78cfe81640 1 -- 172.21.15.1:0/1880724092 >> 172.21.15.1:0/1880724092 conn(0x7f78c8066ee0 msgr2=0x7f78c814b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.155+0000 7f78cfe81640 1 -- 172.21.15.1:0/1880724092 shutdown_connections 2024-03-06T07:25:40.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.155+0000 7f78cfe81640 1 -- 172.21.15.1:0/1880724092 wait complete. 2024-03-06T07:25:40.168 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 121, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:25:29.792840+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 121, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8301": { "addr": "172.21.15.1:6834/1706279305", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 1706279305, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 1706279305, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8301, "incarnation": 119, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-03-06T07:25:31.781726+0000", "qdb_cluster": [ 8301 ], "qdb_leader": 8301, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8301 } } } ], "standbys": [ { "addr": "172.21.15.1:6835/2848968195", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 2848968195, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 2848968195, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 118, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6686, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6836/1671554096", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 1671554096, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 1671554096, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 118, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7387, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6837/2739580331", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 2739580331, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2739580331, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 118, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7390, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:25:40.168 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:25:40.169 INFO:tasks.cephfs.test_failover:current = [{'gid': 8301, 'name': 'a', 'rank': 0, 'incarnation': 119, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6834/1706279305', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1706279305}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1706279305}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7390, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/2739580331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2739580331}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2739580331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 118}, {'gid': 6686, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/2848968195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 2848968195}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2848968195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 118}, {'gid': 7387, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/1671554096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 1671554096}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1671554096}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 118}] 2024-03-06T07:25:40.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs 2024-03-06T07:25:40.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.271+0000 7f547ab02640 1 Processor -- start 2024-03-06T07:25:40.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.271+0000 7f547ab02640 1 -- start start 2024-03-06T07:25:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.271+0000 7f547ab02640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54740c4560 0x7f5474155c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.271+0000 7f547ab02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54740c39c0 0x7f54741561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.272+0000 7f547ab02640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54740c7420 con 0x7f54740c4940 2024-03-06T07:25:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.272+0000 7f547ab02640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54741566f0 con 0x7f54740c4560 2024-03-06T07:25:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.272+0000 7f547ab02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54740ab760 con 0x7f54740c39c0 2024-03-06T07:25:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.272+0000 7f547889e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54740c4560 0x7f5474155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.272+0000 7f547889e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54740c4560 0x7f5474155c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48270/0 (socket says 172.21.15.1:48270) 2024-03-06T07:25:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.272+0000 7f547889e640 1 -- 172.21.15.1:0/1850915199 learned_addr learned my addr 172.21.15.1:0/1850915199 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.272+0000 7f5473fff640 1 --2- 172.21.15.1:0/1850915199 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54740c39c0 0x7f54741561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.272+0000 7f547889e640 1 -- 172.21.15.1:0/1850915199 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54740c39c0 msgr2=0x7f54741561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.272+0000 7f547889e640 1 --2- 172.21.15.1:0/1850915199 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54740c39c0 0x7f54741561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.272+0000 7f547889e640 1 -- 172.21.15.1:0/1850915199 >> v1:172.21.15.98:6789/0 conn(0x7f54740c4940 legacy=0x7f54740ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.272+0000 7f547889e640 1 -- 172.21.15.1:0/1850915199 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54740c76c0 con 0x7f54740c4560 2024-03-06T07:25:40.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.273+0000 7f547889e640 1 --2- 172.21.15.1:0/1850915199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54740c4560 0x7f5474155c70 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f546800bb10 tx=0x7f546802e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa0f417816ffdfb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.273+0000 7f54737fe640 1 -- 172.21.15.1:0/1850915199 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546803b070 con 0x7f54740c4560 2024-03-06T07:25:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.273+0000 7f54737fe640 1 -- 172.21.15.1:0/1850915199 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5468036440 con 0x7f54740c4560 2024-03-06T07:25:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.273+0000 7f54737fe640 1 -- 172.21.15.1:0/1850915199 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5468036810 con 0x7f54740c4560 2024-03-06T07:25:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.273+0000 7f547ab02640 1 -- 172.21.15.1:0/1850915199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54740c4560 msgr2=0x7f5474155c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.273+0000 7f547ab02640 1 --2- 172.21.15.1:0/1850915199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54740c4560 0x7f5474155c70 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f546800bb10 tx=0x7f546802e0c0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.273+0000 7f547ab02640 1 -- 172.21.15.1:0/1850915199 shutdown_connections 2024-03-06T07:25:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.273+0000 7f547ab02640 1 --2- 172.21.15.1:0/1850915199 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54740c39c0 0x7f54741561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.273+0000 7f547ab02640 1 --2- 172.21.15.1:0/1850915199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54740c4560 0x7f5474155c70 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.273+0000 7f547ab02640 1 -- 172.21.15.1:0/1850915199 >> 172.21.15.1:0/1850915199 conn(0x7f54740c7a50 msgr2=0x7f54740c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.273+0000 7f547ab02640 1 -- 172.21.15.1:0/1850915199 shutdown_connections 2024-03-06T07:25:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.273+0000 7f547ab02640 1 -- 172.21.15.1:0/1850915199 wait complete. 2024-03-06T07:25:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.274+0000 7f547ab02640 1 Processor -- start 2024-03-06T07:25:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.275+0000 7f547ab02640 1 -- start start 2024-03-06T07:25:40.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.275+0000 7f547ab02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54740c39c0 0x7f54741eb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.275+0000 7f547ab02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54740c4560 0x7f547414f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.275+0000 7f547ab02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54740c4940 0x7f54741ee100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.275+0000 7f547ab02640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547406e6d0 con 0x7f54740c4560 2024-03-06T07:25:40.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.275+0000 7f547ab02640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54740abd00 con 0x7f54740c39c0 2024-03-06T07:25:40.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.275+0000 7f547ab02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547406e850 con 0x7f54740c4940 2024-03-06T07:25:40.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.275+0000 7f5473fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54740c4560 0x7f547414f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.275+0000 7f5473fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54740c4560 0x7f547414f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48286/0 (socket says 172.21.15.1:48286) 2024-03-06T07:25:40.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.275+0000 7f5473fff640 1 -- 172.21.15.1:0/1637342702 learned_addr learned my addr 172.21.15.1:0/1637342702 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:40.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.276+0000 7f547909f640 1 --2- 172.21.15.1:0/1637342702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54740c4940 0x7f54741ee100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.276+0000 7f547889e640 1 --2- 172.21.15.1:0/1637342702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54740c39c0 0x7f54741eb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.276+0000 7f5473fff640 1 -- 172.21.15.1:0/1637342702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54740c4940 msgr2=0x7f54741ee100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.276+0000 7f5473fff640 1 --2- 172.21.15.1:0/1637342702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54740c4940 0x7f54741ee100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.276+0000 7f5473fff640 1 -- 172.21.15.1:0/1637342702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54740c39c0 msgr2=0x7f54741eb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.276+0000 7f5473fff640 1 --2- 172.21.15.1:0/1637342702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54740c39c0 0x7f54741eb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.276+0000 7f5473fff640 1 -- 172.21.15.1:0/1637342702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547409c790 con 0x7f54740c4560 2024-03-06T07:25:40.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.276+0000 7f547909f640 1 --2- 172.21.15.1:0/1637342702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54740c4940 0x7f54741ee100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:40.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.276+0000 7f5473fff640 1 --2- 172.21.15.1:0/1637342702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54740c4560 0x7f547414f330 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f546000d800 tx=0x7f546000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.276+0000 7f54717fa640 1 -- 172.21.15.1:0/1637342702 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546000b6c0 con 0x7f54740c4560 2024-03-06T07:25:40.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.276+0000 7f54717fa640 1 -- 172.21.15.1:0/1637342702 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5460019070 con 0x7f54740c4560 2024-03-06T07:25:40.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.276+0000 7f547ab02640 1 -- 172.21.15.1:0/1637342702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5474070480 con 0x7f54740c4560 2024-03-06T07:25:40.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.276+0000 7f54717fa640 1 -- 172.21.15.1:0/1637342702 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546000ba80 con 0x7f54740c4560 2024-03-06T07:25:40.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.276+0000 7f547ab02640 1 -- 172.21.15.1:0/1637342702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54741eae60 con 0x7f54740c4560 2024-03-06T07:25:40.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.277+0000 7f547ab02640 1 -- 172.21.15.1:0/1637342702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54740686e0 con 0x7f54740c4560 2024-03-06T07:25:40.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.278+0000 7f54717fa640 1 -- 172.21.15.1:0/1637342702 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f546000bc20 con 0x7f54740c4560 2024-03-06T07:25:40.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.278+0000 7f54717fa640 1 --2- 172.21.15.1:0/1637342702 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f545806b500 0x7f545806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.278+0000 7f54717fa640 1 -- 172.21.15.1:0/1637342702 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f546008a790 con 0x7f54740c4560 2024-03-06T07:25:40.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.278+0000 7f547889e640 1 --2- 172.21.15.1:0/1637342702 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f545806b500 0x7f545806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.279+0000 7f547889e640 1 --2- 172.21.15.1:0/1637342702 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f545806b500 0x7f545806d9c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f5468038000 tx=0x7f5468008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.282+0000 7f54717fa640 1 -- 172.21.15.1:0/1637342702 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5460059490 con 0x7f54740c4560 2024-03-06T07:25:40.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.463+0000 7f547ab02640 1 -- 172.21.15.1:0/1637342702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f54740a6e50 con 0x7f54740c4560 2024-03-06T07:25:40.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.466+0000 7f54717fa640 1 -- 172.21.15.1:0/1637342702 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v46)=0 v46) v1 ==== 116+0+0 (secure 0 0 0) 0x7f5460058d90 con 0x7f54740c4560 2024-03-06T07:25:40.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.468+0000 7f547ab02640 1 -- 172.21.15.1:0/1637342702 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f545806b500 msgr2=0x7f545806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.468+0000 7f547ab02640 1 --2- 172.21.15.1:0/1637342702 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f545806b500 0x7f545806d9c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f5468038000 tx=0x7f5468008040 comp rx=0 tx=0).stop 2024-03-06T07:25:40.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.468+0000 7f547ab02640 1 -- 172.21.15.1:0/1637342702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54740c4560 msgr2=0x7f547414f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.468+0000 7f547ab02640 1 --2- 172.21.15.1:0/1637342702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54740c4560 0x7f547414f330 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f546000d800 tx=0x7f546000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.469+0000 7f547ab02640 1 -- 172.21.15.1:0/1637342702 shutdown_connections 2024-03-06T07:25:40.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.469+0000 7f547ab02640 1 --2- 172.21.15.1:0/1637342702 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f545806b500 0x7f545806d9c0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.469+0000 7f547ab02640 1 --2- 172.21.15.1:0/1637342702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54740c4940 0x7f54741ee100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.469+0000 7f547ab02640 1 --2- 172.21.15.1:0/1637342702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54740c4560 0x7f547414f330 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.469+0000 7f547ab02640 1 --2- 172.21.15.1:0/1637342702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54740c39c0 0x7f54741eb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.469+0000 7f547ab02640 1 -- 172.21.15.1:0/1637342702 >> 172.21.15.1:0/1637342702 conn(0x7f54740c7a50 msgr2=0x7f54740a9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.469+0000 7f547ab02640 1 -- 172.21.15.1:0/1637342702 shutdown_connections 2024-03-06T07:25:40.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.469+0000 7f547ab02640 1 -- 172.21.15.1:0/1637342702 wait complete. 2024-03-06T07:25:40.481 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs 2024-03-06T07:25:40.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.583+0000 7f865cf8c640 1 Processor -- start 2024-03-06T07:25:40.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.583+0000 7f865cf8c640 1 -- start start 2024-03-06T07:25:40.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.584+0000 7f865cf8c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8658152690 0x7f8658155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.584+0000 7f865cf8c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86580c6730 0x7f86580c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.584+0000 7f865cf8c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86581561e0 con 0x7f86580c5d10 2024-03-06T07:25:40.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.584+0000 7f865cf8c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86580c3300 con 0x7f8658152690 2024-03-06T07:25:40.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.584+0000 7f865cf8c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86580c35a0 con 0x7f86580c6730 2024-03-06T07:25:40.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.584+0000 7f865659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8658152690 0x7f8658155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.584+0000 7f865659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8658152690 0x7f8658155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48290/0 (socket says 172.21.15.1:48290) 2024-03-06T07:25:40.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.584+0000 7f865659c640 1 -- 172.21.15.1:0/258613740 learned_addr learned my addr 172.21.15.1:0/258613740 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:40.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.584+0000 7f8655d9b640 1 --2- 172.21.15.1:0/258613740 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86580c6730 0x7f86580c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.584+0000 7f865659c640 1 -- 172.21.15.1:0/258613740 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86580c6730 msgr2=0x7f86580c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.584+0000 7f865659c640 1 --2- 172.21.15.1:0/258613740 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86580c6730 0x7f86580c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.584+0000 7f865659c640 1 -- 172.21.15.1:0/258613740 >> v1:172.21.15.98:6789/0 conn(0x7f86580c5d10 legacy=0x7f8658156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.584+0000 7f865659c640 1 -- 172.21.15.1:0/258613740 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86580c3880 con 0x7f8658152690 2024-03-06T07:25:40.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.585+0000 7f865659c640 1 --2- 172.21.15.1:0/258613740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8658152690 0x7f8658155ca0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f8640009870 tx=0x7f864002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6412f892a2b20ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.585+0000 7f865559a640 1 -- 172.21.15.1:0/258613740 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86400317c0 con 0x7f8658152690 2024-03-06T07:25:40.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.585+0000 7f865559a640 1 -- 172.21.15.1:0/258613740 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8640037070 con 0x7f8658152690 2024-03-06T07:25:40.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.585+0000 7f865559a640 1 -- 172.21.15.1:0/258613740 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640031b80 con 0x7f8658152690 2024-03-06T07:25:40.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.586+0000 7f865cf8c640 1 -- 172.21.15.1:0/258613740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8658152690 msgr2=0x7f8658155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.586+0000 7f865cf8c640 1 --2- 172.21.15.1:0/258613740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8658152690 0x7f8658155ca0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f8640009870 tx=0x7f864002b860 comp rx=0 tx=0).stop 2024-03-06T07:25:40.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.586+0000 7f865cf8c640 1 -- 172.21.15.1:0/258613740 shutdown_connections 2024-03-06T07:25:40.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.586+0000 7f865cf8c640 1 --2- 172.21.15.1:0/258613740 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86580c6730 0x7f86580c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.586+0000 7f865cf8c640 1 --2- 172.21.15.1:0/258613740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8658152690 0x7f8658155ca0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.586+0000 7f865cf8c640 1 -- 172.21.15.1:0/258613740 >> 172.21.15.1:0/258613740 conn(0x7f865806b410 msgr2=0x7f865806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.586+0000 7f865cf8c640 1 -- 172.21.15.1:0/258613740 shutdown_connections 2024-03-06T07:25:40.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.586+0000 7f865cf8c640 1 -- 172.21.15.1:0/258613740 wait complete. 2024-03-06T07:25:40.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.587+0000 7f865cf8c640 1 Processor -- start 2024-03-06T07:25:40.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.587+0000 7f865cf8c640 1 -- start start 2024-03-06T07:25:40.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.588+0000 7f865cf8c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86580c5d10 0x7f86580ad970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.588+0000 7f865cf8c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86580c6730 0x7f865814fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.588+0000 7f865cf8c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8658152690 0x7f865806c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.588+0000 7f865cf8c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8658152a70 con 0x7f8658152690 2024-03-06T07:25:40.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.588+0000 7f865cf8c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86581567b0 con 0x7f86580c6730 2024-03-06T07:25:40.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.588+0000 7f865cf8c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8658152bf0 con 0x7f86580c5d10 2024-03-06T07:25:40.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.588+0000 7f8656d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8658152690 0x7f865806c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.588+0000 7f8656d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8658152690 0x7f865806c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48306/0 (socket says 172.21.15.1:48306) 2024-03-06T07:25:40.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.588+0000 7f8656d9d640 1 -- 172.21.15.1:0/212704577 learned_addr learned my addr 172.21.15.1:0/212704577 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:40.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.588+0000 7f865659c640 1 --2- 172.21.15.1:0/212704577 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86580c5d10 0x7f86580ad970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.588+0000 7f8655d9b640 1 --2- 172.21.15.1:0/212704577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86580c6730 0x7f865814fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.588+0000 7f8656d9d640 1 -- 172.21.15.1:0/212704577 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86580c5d10 msgr2=0x7f86580ad970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.588+0000 7f8656d9d640 1 --2- 172.21.15.1:0/212704577 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86580c5d10 0x7f86580ad970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.588+0000 7f8656d9d640 1 -- 172.21.15.1:0/212704577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86580c6730 msgr2=0x7f865814fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.588+0000 7f8656d9d640 1 --2- 172.21.15.1:0/212704577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86580c6730 0x7f865814fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.588+0000 7f8656d9d640 1 -- 172.21.15.1:0/212704577 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86581eb7e0 con 0x7f8658152690 2024-03-06T07:25:40.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.589+0000 7f8656d9d640 1 --2- 172.21.15.1:0/212704577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8658152690 0x7f865806c100 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f865000b990 tx=0x7f865000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.589+0000 7f864effd640 1 -- 172.21.15.1:0/212704577 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8650019070 con 0x7f8658152690 2024-03-06T07:25:40.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.589+0000 7f864effd640 1 -- 172.21.15.1:0/212704577 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865000cba0 con 0x7f8658152690 2024-03-06T07:25:40.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.589+0000 7f865cf8c640 1 -- 172.21.15.1:0/212704577 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f865814a590 con 0x7f8658152690 2024-03-06T07:25:40.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.589+0000 7f864effd640 1 -- 172.21.15.1:0/212704577 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8650014440 con 0x7f8658152690 2024-03-06T07:25:40.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.589+0000 7f865cf8c640 1 -- 172.21.15.1:0/212704577 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8658150060 con 0x7f8658152690 2024-03-06T07:25:40.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.590+0000 7f865cf8c640 1 -- 172.21.15.1:0/212704577 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8658067fd0 con 0x7f8658152690 2024-03-06T07:25:40.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.590+0000 7f864effd640 1 -- 172.21.15.1:0/212704577 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f865001c070 con 0x7f8658152690 2024-03-06T07:25:40.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.590+0000 7f864effd640 1 --2- 172.21.15.1:0/212704577 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f863806b4d0 0x7f863806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.591+0000 7f864effd640 1 -- 172.21.15.1:0/212704577 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f865008a0c0 con 0x7f8658152690 2024-03-06T07:25:40.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.591+0000 7f865659c640 1 --2- 172.21.15.1:0/212704577 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f863806b4d0 0x7f863806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.591+0000 7f865659c640 1 --2- 172.21.15.1:0/212704577 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f863806b4d0 0x7f863806d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8640036000 tx=0x7f864003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.597+0000 7f864effd640 1 -- 172.21.15.1:0/212704577 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8650058da0 con 0x7f8658152690 2024-03-06T07:25:40.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.781+0000 7f865cf8c640 1 -- 172.21.15.1:0/212704577 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f8658148440 con 0x7f8658152690 2024-03-06T07:25:40.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.784+0000 7f864effd640 1 -- 172.21.15.1:0/212704577 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v47)=0 v47) v1 ==== 116+0+0 (secure 0 0 0) 0x7f86500586a0 con 0x7f8658152690 2024-03-06T07:25:40.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.786+0000 7f865cf8c640 1 -- 172.21.15.1:0/212704577 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f863806b4d0 msgr2=0x7f863806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.786+0000 7f865cf8c640 1 --2- 172.21.15.1:0/212704577 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f863806b4d0 0x7f863806d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8640036000 tx=0x7f864003d040 comp rx=0 tx=0).stop 2024-03-06T07:25:40.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.786+0000 7f865cf8c640 1 -- 172.21.15.1:0/212704577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8658152690 msgr2=0x7f865806c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.786+0000 7f865cf8c640 1 --2- 172.21.15.1:0/212704577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8658152690 0x7f865806c100 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f865000b990 tx=0x7f865000be50 comp rx=0 tx=0).stop 2024-03-06T07:25:40.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.786+0000 7f865cf8c640 1 -- 172.21.15.1:0/212704577 shutdown_connections 2024-03-06T07:25:40.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.786+0000 7f865cf8c640 1 --2- 172.21.15.1:0/212704577 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f863806b4d0 0x7f863806d990 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.786+0000 7f865cf8c640 1 --2- 172.21.15.1:0/212704577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8658152690 0x7f865806c100 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.786+0000 7f865cf8c640 1 --2- 172.21.15.1:0/212704577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86580c6730 0x7f865814fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.786+0000 7f865cf8c640 1 --2- 172.21.15.1:0/212704577 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86580c5d10 0x7f86580ad970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.786+0000 7f865cf8c640 1 -- 172.21.15.1:0/212704577 >> 172.21.15.1:0/212704577 conn(0x7f865806b410 msgr2=0x7f8658150dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.786+0000 7f865cf8c640 1 -- 172.21.15.1:0/212704577 shutdown_connections 2024-03-06T07:25:40.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.786+0000 7f865cf8c640 1 -- 172.21.15.1:0/212704577 wait complete. 2024-03-06T07:25:40.799 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:40.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.898+0000 7ff46e1e2640 1 Processor -- start 2024-03-06T07:25:40.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.898+0000 7ff46e1e2640 1 -- start start 2024-03-06T07:25:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.899+0000 7ff46e1e2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff468152650 0x7ff468155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.899+0000 7ff46e1e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4680bf9d0 0x7ff4680c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.899+0000 7ff46e1e2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4680c2fc0 con 0x7ff4680c6b10 2024-03-06T07:25:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.899+0000 7ff46e1e2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff468155800 con 0x7ff468152650 2024-03-06T07:25:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.899+0000 7ff46e1e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff468158da0 con 0x7ff4680bf9d0 2024-03-06T07:25:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.899+0000 7ff4677fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff468152650 0x7ff468155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.899+0000 7ff466ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff4680c6b10 0x7ff468158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39992/0 (socket says 172.21.15.1:39992) 2024-03-06T07:25:40.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.899+0000 7ff466ffd640 1 -- 172.21.15.1:0/4024121623 learned_addr learned my addr 172.21.15.1:0/4024121623 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.899+0000 7ff467fff640 1 --2- 172.21.15.1:0/4024121623 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4680bf9d0 0x7ff4680c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.899+0000 7ff4677fe640 1 -- 172.21.15.1:0/4024121623 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4680bf9d0 msgr2=0x7ff4680c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.899+0000 7ff4677fe640 1 --2- 172.21.15.1:0/4024121623 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4680bf9d0 0x7ff4680c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.899+0000 7ff4677fe640 1 -- 172.21.15.1:0/4024121623 >> v1:172.21.15.1:6789/0 conn(0x7ff4680c6b10 legacy=0x7ff468158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.899+0000 7ff4677fe640 1 -- 172.21.15.1:0/4024121623 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4680c6690 con 0x7ff468152650 2024-03-06T07:25:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.899+0000 7ff4667fc640 1 -- 172.21.15.1:0/4024121623 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042474695 0 0) 0x7ff4680c2fc0 con 0x7ff4680c6b10 2024-03-06T07:25:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.899+0000 7ff467fff640 1 --2- 172.21.15.1:0/4024121623 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4680bf9d0 0x7ff4680c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.900+0000 7ff4677fe640 1 --2- 172.21.15.1:0/4024121623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff468152650 0x7ff468155c90 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7ff4580097a0 tx=0x7ff45802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af83149ccf987e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.900+0000 7ff4667fc640 1 -- 172.21.15.1:0/4024121623 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4580315a0 con 0x7ff468152650 2024-03-06T07:25:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.900+0000 7ff4667fc640 1 -- 172.21.15.1:0/4024121623 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff458037070 con 0x7ff468152650 2024-03-06T07:25:40.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.900+0000 7ff4667fc640 1 -- 172.21.15.1:0/4024121623 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff458031960 con 0x7ff468152650 2024-03-06T07:25:40.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.901+0000 7ff46e1e2640 1 -- 172.21.15.1:0/4024121623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff468152650 msgr2=0x7ff468155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.901+0000 7ff46e1e2640 1 --2- 172.21.15.1:0/4024121623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff468152650 0x7ff468155c90 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7ff4580097a0 tx=0x7ff45802b840 comp rx=0 tx=0).stop 2024-03-06T07:25:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.901+0000 7ff46e1e2640 1 -- 172.21.15.1:0/4024121623 shutdown_connections 2024-03-06T07:25:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.901+0000 7ff46e1e2640 1 --2- 172.21.15.1:0/4024121623 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4680bf9d0 0x7ff4680c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.901+0000 7ff46e1e2640 1 --2- 172.21.15.1:0/4024121623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff468152650 0x7ff468155c90 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.901+0000 7ff46e1e2640 1 -- 172.21.15.1:0/4024121623 >> 172.21.15.1:0/4024121623 conn(0x7ff468066ee0 msgr2=0x7ff4680672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.901+0000 7ff46e1e2640 1 -- 172.21.15.1:0/4024121623 shutdown_connections 2024-03-06T07:25:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.901+0000 7ff46e1e2640 1 -- 172.21.15.1:0/4024121623 wait complete. 2024-03-06T07:25:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.902+0000 7ff46e1e2640 1 Processor -- start 2024-03-06T07:25:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.902+0000 7ff46e1e2640 1 -- start start 2024-03-06T07:25:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.903+0000 7ff46e1e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4680bf9d0 0x7ff4681ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.903+0000 7ff46e1e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff468152650 0x7ff46814fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.903+0000 7ff46e1e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4680c5a00 0x7ff4681e0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.903+0000 7ff46e1e2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4681551b0 con 0x7ff4680bf9d0 2024-03-06T07:25:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.903+0000 7ff46e1e2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4680c6fa0 con 0x7ff468152650 2024-03-06T07:25:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.903+0000 7ff46e1e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff468155490 con 0x7ff4680c5a00 2024-03-06T07:25:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.903+0000 7ff4677fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4680bf9d0 0x7ff4681ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.903+0000 7ff4677fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4680bf9d0 0x7ff4681ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48318/0 (socket says 172.21.15.1:48318) 2024-03-06T07:25:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.903+0000 7ff4677fe640 1 -- 172.21.15.1:0/2597763260 learned_addr learned my addr 172.21.15.1:0/2597763260 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.903+0000 7ff4677fe640 1 -- 172.21.15.1:0/2597763260 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4680c5a00 msgr2=0x7ff4681e0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.903+0000 7ff4677fe640 1 --2- 172.21.15.1:0/2597763260 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4680c5a00 0x7ff4681e0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.903+0000 7ff4677fe640 1 -- 172.21.15.1:0/2597763260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff468152650 msgr2=0x7ff46814fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.903+0000 7ff4677fe640 1 --2- 172.21.15.1:0/2597763260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff468152650 0x7ff46814fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.903+0000 7ff4677fe640 1 -- 172.21.15.1:0/2597763260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff468156490 con 0x7ff4680bf9d0 2024-03-06T07:25:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.903+0000 7ff4677fe640 1 --2- 172.21.15.1:0/2597763260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4680bf9d0 0x7ff4681ee7e0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7ff4580097a0 tx=0x7ff4580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.903+0000 7ff44ffff640 1 -- 172.21.15.1:0/2597763260 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff458004230 con 0x7ff4680bf9d0 2024-03-06T07:25:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.904+0000 7ff44ffff640 1 -- 172.21.15.1:0/2597763260 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4580043d0 con 0x7ff4680bf9d0 2024-03-06T07:25:40.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.904+0000 7ff46e1e2640 1 -- 172.21.15.1:0/2597763260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff468156770 con 0x7ff4680bf9d0 2024-03-06T07:25:40.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.904+0000 7ff44ffff640 1 -- 172.21.15.1:0/2597763260 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff458003850 con 0x7ff4680bf9d0 2024-03-06T07:25:40.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.904+0000 7ff46e1e2640 1 -- 172.21.15.1:0/2597763260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4681e1460 con 0x7ff4680bf9d0 2024-03-06T07:25:40.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.905+0000 7ff46e1e2640 1 -- 172.21.15.1:0/2597763260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff468067fd0 con 0x7ff4680bf9d0 2024-03-06T07:25:40.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.905+0000 7ff44ffff640 1 -- 172.21.15.1:0/2597763260 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4580039f0 con 0x7ff4680bf9d0 2024-03-06T07:25:40.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.905+0000 7ff44ffff640 1 --2- 172.21.15.1:0/2597763260 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff45406b4d0 0x7ff45406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.905+0000 7ff44ffff640 1 -- 172.21.15.1:0/2597763260 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff45802f030 con 0x7ff4680bf9d0 2024-03-06T07:25:40.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.906+0000 7ff466ffd640 1 --2- 172.21.15.1:0/2597763260 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff45406b4d0 0x7ff45406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.906+0000 7ff466ffd640 1 --2- 172.21.15.1:0/2597763260 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff45406b4d0 0x7ff45406d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff45c005880 tx=0x7ff45c007950 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:40.913+0000 7ff44ffff640 1 -- 172.21.15.1:0/2597763260 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff458041070 con 0x7ff4680bf9d0 2024-03-06T07:25:41.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.147+0000 7ff46e1e2640 1 -- 172.21.15.1:0/2597763260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff468156940 con 0x7ff4680bf9d0 2024-03-06T07:25:41.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.148+0000 7ff44ffff640 1 -- 172.21.15.1:0/2597763260 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+4981 (secure 0 0 0) 0x7ff45807b220 con 0x7ff4680bf9d0 2024-03-06T07:25:41.150 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:41.150 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2848968195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2848968195},{"type":"v1","addr":"172.21.15.1:6835","nonce":2848968195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7387,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1671554096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1671554096},{"type":"v1","addr":"172.21.15.98:6836","nonce":1671554096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2739580331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2739580331},{"type":"v1","addr":"172.21.15.98:6837","nonce":2739580331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:29.792840+0000","modified":"2024-03-06T07:25:31.781726+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"a","rank":0,"incarnation":119,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1706279305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1706279305},{"type":"v1","addr":"172.21.15.1:6834","nonce":1706279305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8301,"qdb_cluster":[8301]},"id":10}]} 2024-03-06T07:25:41.150 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 121 2024-03-06T07:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.150+0000 7ff46e1e2640 1 -- 172.21.15.1:0/2597763260 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff45406b4d0 msgr2=0x7ff45406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.150+0000 7ff46e1e2640 1 --2- 172.21.15.1:0/2597763260 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff45406b4d0 0x7ff45406d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff45c005880 tx=0x7ff45c007950 comp rx=0 tx=0).stop 2024-03-06T07:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.150+0000 7ff46e1e2640 1 -- 172.21.15.1:0/2597763260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4680bf9d0 msgr2=0x7ff4681ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.150+0000 7ff46e1e2640 1 --2- 172.21.15.1:0/2597763260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4680bf9d0 0x7ff4681ee7e0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7ff4580097a0 tx=0x7ff4580046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.150+0000 7ff46e1e2640 1 -- 172.21.15.1:0/2597763260 shutdown_connections 2024-03-06T07:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.151+0000 7ff46e1e2640 1 --2- 172.21.15.1:0/2597763260 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff45406b4d0 0x7ff45406d990 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.151+0000 7ff46e1e2640 1 --2- 172.21.15.1:0/2597763260 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4680c5a00 0x7ff4681e0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.151+0000 7ff46e1e2640 1 --2- 172.21.15.1:0/2597763260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff468152650 0x7ff46814fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.151+0000 7ff46e1e2640 1 --2- 172.21.15.1:0/2597763260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4680bf9d0 0x7ff4681ee7e0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.151+0000 7ff46e1e2640 1 -- 172.21.15.1:0/2597763260 >> 172.21.15.1:0/2597763260 conn(0x7ff468066ee0 msgr2=0x7ff4680c3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.151+0000 7ff46e1e2640 1 -- 172.21.15.1:0/2597763260 shutdown_connections 2024-03-06T07:25:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.151+0000 7ff46e1e2640 1 -- 172.21.15.1:0/2597763260 wait complete. 2024-03-06T07:25:41.570 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:29.792840+0000', 'modified': '2024-03-06T07:25:31.781726+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8301}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8301': {'gid': 8301, 'name': 'a', 'rank': 0, 'incarnation': 119, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6834/1706279305', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1706279305}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1706279305}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8301, 'qdb_cluster': [8301]} 2024-03-06T07:25:41.570 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:41.570 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:25:41.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.671+0000 7f646eac3640 1 Processor -- start 2024-03-06T07:25:41.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.671+0000 7f646eac3640 1 -- start start 2024-03-06T07:25:41.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f646eac3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f64680bf9d0 0x7f6468155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f646eac3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64680c2fc0 con 0x7f64680c6b10 2024-03-06T07:25:41.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f646eac3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6468158da0 con 0x7f6468152650 2024-03-06T07:25:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f646eac3640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6468155800 con 0x7f64680bf9d0 2024-03-06T07:25:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f6467fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f64680c6b10 0x7f64680c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39998/0 (socket says 172.21.15.1:39998) 2024-03-06T07:25:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f6467fff640 1 -- 172.21.15.1:0/2133784879 learned_addr learned my addr 172.21.15.1:0/2133784879 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f64677fe640 1 -- 172.21.15.1:0/2133784879 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624481369 0 0) 0x7f64680c2fc0 con 0x7f64680c6b10 2024-03-06T07:25:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f64677fe640 1 -- 172.21.15.1:0/2133784879 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6458003520 con 0x7f64680c6b10 2024-03-06T07:25:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f646d060640 1 --2- 172.21.15.1:0/2133784879 >> v2:172.21.15.98:3300/0 conn(0x7f64680bf9d0 0x7f6468155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f64677fe640 1 -- 172.21.15.1:0/2133784879 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 787343302 0 0) 0x7f6458003520 con 0x7f64680c6b10 2024-03-06T07:25:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f64677fe640 1 -- 172.21.15.1:0/2133784879 >> v2:172.21.15.98:3300/0 conn(0x7f64680bf9d0 msgr2=0x7f6468155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f64677fe640 1 --2- 172.21.15.1:0/2133784879 >> v2:172.21.15.98:3300/0 conn(0x7f64680bf9d0 0x7f6468155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f64677fe640 1 -- 172.21.15.1:0/2133784879 >> v1:172.21.15.98:6789/0 conn(0x7f6468152650 legacy=0x7f6468158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f64677fe640 1 -- 172.21.15.1:0/2133784879 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64680c6690 con 0x7f64680c6b10 2024-03-06T07:25:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f64677fe640 1 -- 172.21.15.1:0/2133784879 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f6454001670 con 0x7f64680c6b10 2024-03-06T07:25:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f64677fe640 1 -- 172.21.15.1:0/2133784879 >> v1:172.21.15.1:6789/0 conn(0x7f64680c6b10 legacy=0x7f64680c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f64677fe640 1 --2- 172.21.15.1:0/2133784879 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6458003a00 0x7f6458003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f64677fe640 1 --2- 172.21.15.1:0/2133784879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6458004800 0x7f6458004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f64677fe640 1 --2- 172.21.15.1:0/2133784879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6458005210 0x7f645800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f64677fe640 1 -- 172.21.15.1:0/2133784879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6468155800 con 0x7f6458004800 2024-03-06T07:25:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f64677fe640 1 -- 172.21.15.1:0/2133784879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645800c160 con 0x7f6458005210 2024-03-06T07:25:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.672+0000 7f64677fe640 1 -- 172.21.15.1:0/2133784879 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f645800c360 con 0x7f6458003a00 2024-03-06T07:25:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.673+0000 7f6467fff640 1 --2- 172.21.15.1:0/2133784879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6458004800 0x7f6458004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.673+0000 7f646c85f640 1 --2- 172.21.15.1:0/2133784879 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6458003a00 0x7f6458003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.673+0000 7f646d060640 1 --2- 172.21.15.1:0/2133784879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6458005210 0x7f645800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.673+0000 7f6467fff640 1 -- 172.21.15.1:0/2133784879 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6458003a00 msgr2=0x7f6458003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.673+0000 7f6467fff640 1 --2- 172.21.15.1:0/2133784879 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6458003a00 0x7f6458003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.673+0000 7f6467fff640 1 -- 172.21.15.1:0/2133784879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6458005210 msgr2=0x7f645800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.673+0000 7f6467fff640 1 --2- 172.21.15.1:0/2133784879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6458005210 0x7f645800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.673+0000 7f6467fff640 1 -- 172.21.15.1:0/2133784879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6458003520 con 0x7f6458004800 2024-03-06T07:25:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.673+0000 7f646d060640 1 -- 172.21.15.1:0/2133784879 reap_dead start 2024-03-06T07:25:41.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.674+0000 7f6467fff640 1 --2- 172.21.15.1:0/2133784879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6458004800 0x7f6458004c80 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f6454007fd0 tx=0x7f6458005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d917d1f0f15471d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.674+0000 7f64677fe640 1 -- 172.21.15.1:0/2133784879 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6454007c20 con 0x7f6458004800 2024-03-06T07:25:41.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.674+0000 7f64677fe640 1 -- 172.21.15.1:0/2133784879 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6454030070 con 0x7f6458004800 2024-03-06T07:25:41.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.674+0000 7f64677fe640 1 -- 172.21.15.1:0/2133784879 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6454005d30 con 0x7f6458004800 2024-03-06T07:25:41.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.675+0000 7f646eac3640 1 -- 172.21.15.1:0/2133784879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6458004800 msgr2=0x7f6458004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.675+0000 7f646eac3640 1 --2- 172.21.15.1:0/2133784879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6458004800 0x7f6458004c80 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f6454007fd0 tx=0x7f6458005830 comp rx=0 tx=0).stop 2024-03-06T07:25:41.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.675+0000 7f646eac3640 1 -- 172.21.15.1:0/2133784879 shutdown_connections 2024-03-06T07:25:41.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.675+0000 7f646eac3640 1 --2- 172.21.15.1:0/2133784879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6458004800 0x7f6458004c80 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.675+0000 7f646eac3640 1 -- 172.21.15.1:0/2133784879 >> 172.21.15.1:0/2133784879 conn(0x7f6468066ee0 msgr2=0x7f64680672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.675+0000 7f646eac3640 1 -- 172.21.15.1:0/2133784879 shutdown_connections 2024-03-06T07:25:41.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.675+0000 7f646eac3640 1 -- 172.21.15.1:0/2133784879 wait complete. 2024-03-06T07:25:41.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.676+0000 7f646eac3640 1 Processor -- start 2024-03-06T07:25:41.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.676+0000 7f646eac3640 1 -- start start 2024-03-06T07:25:41.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f646eac3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6458004800 0x7f6468157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f646eac3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64681588c0 0x7f64681ee1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f646eac3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64681ee6f0 0x7f646814a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f646eac3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64681551b0 con 0x7f64681588c0 2024-03-06T07:25:41.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f646eac3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64680c6fa0 con 0x7f6458004800 2024-03-06T07:25:41.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f646eac3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6468155490 con 0x7f64681ee6f0 2024-03-06T07:25:41.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f6467fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64681588c0 0x7f64681ee1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f6467fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64681588c0 0x7f64681ee1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48340/0 (socket says 172.21.15.1:48340) 2024-03-06T07:25:41.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f6467fff640 1 -- 172.21.15.1:0/1412196329 learned_addr learned my addr 172.21.15.1:0/1412196329 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:41.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f6467fff640 1 -- 172.21.15.1:0/1412196329 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64681ee6f0 msgr2=0x7f646814a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f6467fff640 1 --2- 172.21.15.1:0/1412196329 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64681ee6f0 0x7f646814a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f6467fff640 1 -- 172.21.15.1:0/1412196329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6458004800 msgr2=0x7f6468157da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f6467fff640 1 --2- 172.21.15.1:0/1412196329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6458004800 0x7f6468157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f6467fff640 1 -- 172.21.15.1:0/1412196329 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64681572d0 con 0x7f64681588c0 2024-03-06T07:25:41.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f6467fff640 1 --2- 172.21.15.1:0/1412196329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64681588c0 0x7f64681ee1b0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f6454000c00 tx=0x7f64680bfe40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f64657fa640 1 -- 172.21.15.1:0/1412196329 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6454046070 con 0x7f64681588c0 2024-03-06T07:25:41.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f64657fa640 1 -- 172.21.15.1:0/1412196329 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6454007290 con 0x7f64681588c0 2024-03-06T07:25:41.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f646eac3640 1 -- 172.21.15.1:0/1412196329 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f646814b480 con 0x7f64681588c0 2024-03-06T07:25:41.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f64657fa640 1 -- 172.21.15.1:0/1412196329 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6454001b60 con 0x7f64681588c0 2024-03-06T07:25:41.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.677+0000 7f646eac3640 1 -- 172.21.15.1:0/1412196329 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6468156db0 con 0x7f64681588c0 2024-03-06T07:25:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.680+0000 7f64657fa640 1 -- 172.21.15.1:0/1412196329 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6454049070 con 0x7f64681588c0 2024-03-06T07:25:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.680+0000 7f64657fa640 1 --2- 172.21.15.1:0/1412196329 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f644c06b4d0 0x7f644c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.680+0000 7f64657fa640 1 -- 172.21.15.1:0/1412196329 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f64540ab490 con 0x7f64681588c0 2024-03-06T07:25:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.680+0000 7f6471785740 1 Processor -- start 2024-03-06T07:25:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.680+0000 7f6471785740 1 -- start start 2024-03-06T07:25:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.680+0000 7f646c85f640 1 --2- 172.21.15.1:0/1412196329 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f644c06b4d0 0x7f644c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f6471785740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb22f50 0x55d56fb20a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f6471785740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb20f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f6471785740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb214b0 0x55d56fb1d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f6471785740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d56fa876b0 con 0x55d56fb23330 2024-03-06T07:25:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f6471785740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d56fa30a60 con 0x55d56fb214b0 2024-03-06T07:25:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f6471785740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d56fa83750 con 0x55d56fb22f50 2024-03-06T07:25:41.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f646c85f640 1 --2- 172.21.15.1:0/1412196329 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f644c06b4d0 0x7f644c06d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f6468155d40 tx=0x7f645c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f6467fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb20f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f6467fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb20f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48354/0 (socket says 172.21.15.1:48354) 2024-03-06T07:25:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f6467fff640 1 -- 172.21.15.1:0/3726285202 learned_addr learned my addr 172.21.15.1:0/3726285202 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f646c85f640 1 --2- 172.21.15.1:0/3726285202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb22f50 0x55d56fb20a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f646d060640 1 --2- 172.21.15.1:0/3726285202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb214b0 0x55d56fb1d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f6467fff640 1 -- 172.21.15.1:0/3726285202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb22f50 msgr2=0x55d56fb20a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f6467fff640 1 --2- 172.21.15.1:0/3726285202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb22f50 0x55d56fb20a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f6467fff640 1 -- 172.21.15.1:0/3726285202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb214b0 msgr2=0x55d56fb1d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f6467fff640 1 --2- 172.21.15.1:0/3726285202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb214b0 0x55d56fb1d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f6467fff640 1 -- 172.21.15.1:0/3726285202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d56fa4ef30 con 0x55d56fb23330 2024-03-06T07:25:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f646c85f640 1 --2- 172.21.15.1:0/3726285202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb22f50 0x55d56fb20a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f646d060640 1 --2- 172.21.15.1:0/3726285202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb214b0 0x55d56fb1d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.681+0000 7f6467fff640 1 --2- 172.21.15.1:0/3726285202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb20f70 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f64540a9860 tx=0x7f645407a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c291d3d0de5cb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.682+0000 7f64527fc640 1 -- 172.21.15.1:0/3726285202 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6454082220 con 0x55d56fb23330 2024-03-06T07:25:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.682+0000 7f64527fc640 1 -- 172.21.15.1:0/3726285202 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6454030040 con 0x55d56fb23330 2024-03-06T07:25:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.682+0000 7f64527fc640 1 -- 172.21.15.1:0/3726285202 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6454077e10 con 0x55d56fb23330 2024-03-06T07:25:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.682+0000 7f6471785740 1 -- 172.21.15.1:0/3726285202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 msgr2=0x55d56fb20f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.682+0000 7f6471785740 1 --2- 172.21.15.1:0/3726285202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb20f70 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f64540a9860 tx=0x7f645407a2d0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.682+0000 7f6471785740 1 -- 172.21.15.1:0/3726285202 shutdown_connections 2024-03-06T07:25:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.682+0000 7f6471785740 1 --2- 172.21.15.1:0/3726285202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb214b0 0x55d56fb1d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.682+0000 7f6471785740 1 --2- 172.21.15.1:0/3726285202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb20f70 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.682+0000 7f6471785740 1 --2- 172.21.15.1:0/3726285202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb22f50 0x55d56fb20a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.682+0000 7f6471785740 1 -- 172.21.15.1:0/3726285202 >> 172.21.15.1:0/3726285202 conn(0x55d56fb1f7d0 msgr2=0x55d56fb443c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.682+0000 7f6471785740 1 -- 172.21.15.1:0/3726285202 shutdown_connections 2024-03-06T07:25:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.682+0000 7f6471785740 1 -- 172.21.15.1:0/3726285202 wait complete. 2024-03-06T07:25:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.683+0000 7f6471785740 1 -- start start 2024-03-06T07:25:41.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.683+0000 7f6471785740 1 Processor -- start 2024-03-06T07:25:41.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.684+0000 7f6471785740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb22f50 0x55d56fb5c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.684+0000 7f6471785740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb6fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.684+0000 7f6471785740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb45f30 0x55d56fb463b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.684+0000 7f6471785740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d56fa30a60 con 0x55d56fb23330 2024-03-06T07:25:41.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.684+0000 7f6471785740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d56fa43ea0 con 0x55d56fb22f50 2024-03-06T07:25:41.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.684+0000 7f6471785740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d56fa876b0 con 0x55d56fb45f30 2024-03-06T07:25:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f6471785740 10 client.0 _ll_register_callbacks cb 0x55d56fa43870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f6467fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb6fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f6467fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb6fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48358/0 (socket says 172.21.15.1:48358) 2024-03-06T07:25:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f6467fff640 1 -- 172.21.15.1:0/3346014285 learned_addr learned my addr 172.21.15.1:0/3346014285 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f646d060640 1 --2- 172.21.15.1:0/3346014285 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb45f30 0x55d56fb463b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f646c85f640 1 --2- 172.21.15.1:0/3346014285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb22f50 0x55d56fb5c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f6467fff640 1 -- 172.21.15.1:0/3346014285 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb45f30 msgr2=0x55d56fb463b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f6467fff640 1 --2- 172.21.15.1:0/3346014285 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb45f30 0x55d56fb463b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f6467fff640 1 -- 172.21.15.1:0/3346014285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb22f50 msgr2=0x55d56fb5c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f6467fff640 1 --2- 172.21.15.1:0/3346014285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb22f50 0x55d56fb5c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f6467fff640 1 -- 172.21.15.1:0/3346014285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d56fa44270 con 0x55d56fb23330 2024-03-06T07:25:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f646c85f640 1 --2- 172.21.15.1:0/3346014285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb22f50 0x55d56fb5c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f6467fff640 1 --2- 172.21.15.1:0/3346014285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb6fe70 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f645407a3c0 tx=0x7f6454001820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c991afafd106f67a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f64477fe640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:25:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f64477fe640 1 -- 172.21.15.1:0/3346014285 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6454082220 con 0x55d56fb23330 2024-03-06T07:25:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f64477fe640 1 -- 172.21.15.1:0/3346014285 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6454076180 con 0x55d56fb23330 2024-03-06T07:25:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f6471785740 1 -- 172.21.15.1:0/3346014285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6454030040 con 0x55d56fb23330 2024-03-06T07:25:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f64477fe640 1 -- 172.21.15.1:0/3346014285 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6454097050 con 0x55d56fb23330 2024-03-06T07:25:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f6471785740 10 client.8505 fetch_fsmap learned FSMap version 121 2024-03-06T07:25:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f6471785740 1 -- 172.21.15.1:0/3346014285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=2+}) v3 -- 0x55d56f79c2d0 con 0x55d56fb23330 2024-03-06T07:25:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.685+0000 7f64477fe640 1 -- 172.21.15.1:0/3346014285 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6454030040 con 0x55d56fb23330 2024-03-06T07:25:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.686+0000 7f64477fe640 1 -- 172.21.15.1:0/3346014285 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 121) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f64540ab070 con 0x55d56fb23330 2024-03-06T07:25:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.686+0000 7f6471785740 10 client.8505 fetch_fsmap finished waiting for FSMap version 121 2024-03-06T07:25:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.686+0000 7f6471785740 10 client.8505 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:25:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.686+0000 7f6471785740 20 client.8505 populate_metadata read hostname 'smithi001' 2024-03-06T07:25:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.686+0000 7f6471785740 1 --2- 172.21.15.1:0/3346014285 >> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] conn(0x55d56fb72e00 0x55d56fb731e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.686+0000 7f6471785740 4 client.8505 mds_command: new command op to 8301 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.686+0000 7f6471785740 1 -- 172.21.15.1:0/3346014285 --> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d56fa94bf0 con 0x55d56fb72e00 2024-03-06T07:25:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.686+0000 7f646c85f640 1 --2- 172.21.15.1:0/3346014285 >> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] conn(0x55d56fb72e00 0x55d56fb731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.686+0000 7f646c85f640 1 --2- 172.21.15.1:0/3346014285 >> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] conn(0x55d56fb72e00 0x55d56fb731e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff11fd01efaf005a server_cookie=23a55b0feb65e8d7 in_seq=0 out_seq=0 2024-03-06T07:25:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.686+0000 7f64477fe640 10 client.8505 ms_handle_connect on v2:172.21.15.1:6832/1706279305 2024-03-06T07:25:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.687+0000 7f64477fe640 1 -- 172.21.15.1:0/3346014285 <== mds.0 v2:172.21.15.1:6832/1706279305 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d56fa94bf0 con 0x55d56fb72e00 2024-03-06T07:25:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.687+0000 7f64477fe640 10 client.8505 handle_command_reply: tid=0 2024-03-06T07:25:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.687+0000 7f6471785740 1 client.8505 shutdown 2024-03-06T07:25:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.687+0000 7f6471785740 1 -- 172.21.15.1:0/3346014285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 msgr2=0x55d56fb6fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.687+0000 7f6471785740 1 --2- 172.21.15.1:0/3346014285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb6fe70 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f645407a3c0 tx=0x7f6454001820 comp rx=0 tx=0).stop 2024-03-06T07:25:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.687+0000 7f6471785740 1 -- 172.21.15.1:0/3346014285 shutdown_connections 2024-03-06T07:25:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.687+0000 7f6471785740 1 --2- 172.21.15.1:0/3346014285 >> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] conn(0x55d56fb72e00 0x55d56fb731e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.687+0000 7f6471785740 1 --2- 172.21.15.1:0/3346014285 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb45f30 0x55d56fb463b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.687+0000 7f6471785740 1 --2- 172.21.15.1:0/3346014285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb6fe70 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.687+0000 7f6471785740 1 --2- 172.21.15.1:0/3346014285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb22f50 0x55d56fb5c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.687+0000 7f6471785740 1 -- 172.21.15.1:0/3346014285 >> 172.21.15.1:0/3346014285 conn(0x55d56fb214b0 msgr2=0x55d56fb44c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.687+0000 7f64477fe640 0 client.8505 ms_handle_reset on v2:172.21.15.1:6832/1706279305 2024-03-06T07:25:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.687+0000 7f6471785740 1 -- 172.21.15.1:0/3346014285 shutdown_connections 2024-03-06T07:25:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.687+0000 7f6471785740 1 -- 172.21.15.1:0/3346014285 wait complete. 2024-03-06T07:25:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.687+0000 7f6471785740 20 client.8505 trim_cache size 0 max 16384 2024-03-06T07:25:41.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.704+0000 7f6471785740 1 Processor -- start 2024-03-06T07:25:41.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.704+0000 7f6471785740 1 -- start start 2024-03-06T07:25:41.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.704+0000 7f6471785740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb22f50 0x55d56fb442e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.704+0000 7f6471785740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb20980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.704+0000 7f6471785740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb45f30 0x55d56fb62dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.704+0000 7f6471785740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64681f74f0 con 0x55d56fb23330 2024-03-06T07:25:41.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.704+0000 7f6471785740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64680c93d0 con 0x55d56fb22f50 2024-03-06T07:25:41.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.704+0000 7f6471785740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64681f6f20 con 0x55d56fb45f30 2024-03-06T07:25:41.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.704+0000 7f6467fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb20980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.705+0000 7f6467fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb20980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48370/0 (socket says 172.21.15.1:48370) 2024-03-06T07:25:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.705+0000 7f6467fff640 1 -- 172.21.15.1:0/758011913 learned_addr learned my addr 172.21.15.1:0/758011913 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.705+0000 7f646d060640 1 --2- 172.21.15.1:0/758011913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb45f30 0x55d56fb62dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.705+0000 7f646c85f640 1 --2- 172.21.15.1:0/758011913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb22f50 0x55d56fb442e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.705+0000 7f6467fff640 1 -- 172.21.15.1:0/758011913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb45f30 msgr2=0x55d56fb62dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.705+0000 7f6467fff640 1 --2- 172.21.15.1:0/758011913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb45f30 0x55d56fb62dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.705+0000 7f6467fff640 1 -- 172.21.15.1:0/758011913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb22f50 msgr2=0x55d56fb442e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.705+0000 7f6467fff640 1 --2- 172.21.15.1:0/758011913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb22f50 0x55d56fb442e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.705+0000 7f6467fff640 1 -- 172.21.15.1:0/758011913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d56f8666b0 con 0x55d56fb23330 2024-03-06T07:25:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.705+0000 7f646d060640 1 --2- 172.21.15.1:0/758011913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb45f30 0x55d56fb62dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.705+0000 7f646c85f640 1 --2- 172.21.15.1:0/758011913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb22f50 0x55d56fb442e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.705+0000 7f6467fff640 1 --2- 172.21.15.1:0/758011913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb20980 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f6454081cf0 tx=0x7f6454090000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7606be02912ff588 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.705+0000 7f6451ffb640 1 -- 172.21.15.1:0/758011913 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64540ab070 con 0x55d56fb23330 2024-03-06T07:25:41.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.705+0000 7f6451ffb640 1 -- 172.21.15.1:0/758011913 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6454030070 con 0x55d56fb23330 2024-03-06T07:25:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.705+0000 7f6451ffb640 1 -- 172.21.15.1:0/758011913 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6454078f60 con 0x55d56fb23330 2024-03-06T07:25:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.706+0000 7f6471785740 1 -- 172.21.15.1:0/758011913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 msgr2=0x55d56fb20980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.706+0000 7f6471785740 1 --2- 172.21.15.1:0/758011913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb20980 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f6454081cf0 tx=0x7f6454090000 comp rx=0 tx=0).stop 2024-03-06T07:25:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.706+0000 7f6471785740 1 -- 172.21.15.1:0/758011913 shutdown_connections 2024-03-06T07:25:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.706+0000 7f6471785740 1 --2- 172.21.15.1:0/758011913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb45f30 0x55d56fb62dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.706+0000 7f6471785740 1 --2- 172.21.15.1:0/758011913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb23330 0x55d56fb20980 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.706+0000 7f6471785740 1 --2- 172.21.15.1:0/758011913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb22f50 0x55d56fb442e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.706+0000 7f6471785740 1 -- 172.21.15.1:0/758011913 >> 172.21.15.1:0/758011913 conn(0x55d56fb214b0 msgr2=0x55d56fb6f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.706+0000 7f6471785740 1 -- 172.21.15.1:0/758011913 shutdown_connections 2024-03-06T07:25:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.706+0000 7f6471785740 1 -- 172.21.15.1:0/758011913 wait complete. 2024-03-06T07:25:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.706+0000 7f6471785740 1 -- start start 2024-03-06T07:25:41.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.706+0000 7f6471785740 1 Processor -- start 2024-03-06T07:25:41.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.707+0000 7f6471785740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb22f50 0x55d56fb48e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.707+0000 7f6471785740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb23330 0x55d56fb1d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.707+0000 7f6471785740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb45f30 0x55d56fbcb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.707+0000 7f6471785740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d56fa30a60 con 0x55d56fb45f30 2024-03-06T07:25:41.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.707+0000 7f6471785740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d56fa876b0 con 0x55d56fb23330 2024-03-06T07:25:41.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.707+0000 7f6471785740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d56fa43ea0 con 0x55d56fb22f50 2024-03-06T07:25:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.707+0000 7f6471785740 10 client.0 _ll_register_callbacks cb 0x7f6454079470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.707+0000 7f646d060640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb45f30 0x55d56fbcb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.707+0000 7f646d060640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb45f30 0x55d56fbcb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48386/0 (socket says 172.21.15.1:48386) 2024-03-06T07:25:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.707+0000 7f646d060640 1 -- 172.21.15.1:0/4234863286 learned_addr learned my addr 172.21.15.1:0/4234863286 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f646c85f640 1 --2- 172.21.15.1:0/4234863286 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb22f50 0x55d56fb48e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f6467fff640 1 --2- 172.21.15.1:0/4234863286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb23330 0x55d56fb1d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f646d060640 1 -- 172.21.15.1:0/4234863286 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb22f50 msgr2=0x55d56fb48e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f646d060640 1 --2- 172.21.15.1:0/4234863286 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb22f50 0x55d56fb48e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f646d060640 1 -- 172.21.15.1:0/4234863286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb23330 msgr2=0x55d56fb1d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f646d060640 1 --2- 172.21.15.1:0/4234863286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb23330 0x55d56fb1d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f646d060640 1 -- 172.21.15.1:0/4234863286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d56fb61550 con 0x55d56fb45f30 2024-03-06T07:25:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f6467fff640 1 --2- 172.21.15.1:0/4234863286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb23330 0x55d56fb1d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f646d060640 1 --2- 172.21.15.1:0/4234863286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb45f30 0x55d56fbcb1b0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f646808c760 tx=0x7f64600046a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ac71fb422caf1fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f64477fe640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:25:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f64477fe640 1 -- 172.21.15.1:0/4234863286 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6460018af0 con 0x55d56fb45f30 2024-03-06T07:25:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f64477fe640 1 -- 172.21.15.1:0/4234863286 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6460018c90 con 0x55d56fb45f30 2024-03-06T07:25:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f6471785740 1 -- 172.21.15.1:0/4234863286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6454030070 con 0x55d56fb45f30 2024-03-06T07:25:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f64477fe640 1 -- 172.21.15.1:0/4234863286 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6460029050 con 0x55d56fb45f30 2024-03-06T07:25:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f6471785740 10 client.8511 fetch_fsmap learned FSMap version 121 2024-03-06T07:25:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f6471785740 1 -- 172.21.15.1:0/4234863286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=2+}) v3 -- 0x55d56fb47510 con 0x55d56fb45f30 2024-03-06T07:25:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f64477fe640 1 -- 172.21.15.1:0/4234863286 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6454030070 con 0x55d56fb45f30 2024-03-06T07:25:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f64477fe640 1 -- 172.21.15.1:0/4234863286 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 121) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f646002c020 con 0x55d56fb45f30 2024-03-06T07:25:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f6471785740 10 client.8511 fetch_fsmap finished waiting for FSMap version 121 2024-03-06T07:25:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f6471785740 10 client.8511 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:25:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f6471785740 20 client.8511 populate_metadata read hostname 'smithi001' 2024-03-06T07:25:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f6471785740 1 --2- 172.21.15.1:0/4234863286 >> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] conn(0x55d56fbd13e0 0x55d56fbd17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f6471785740 4 client.8511 mds_command: new command op to 8301 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.708+0000 7f6471785740 1 -- 172.21.15.1:0/4234863286 --> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d56fa8ef40 con 0x55d56fbd13e0 2024-03-06T07:25:41.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f646c85f640 1 --2- 172.21.15.1:0/4234863286 >> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] conn(0x55d56fbd13e0 0x55d56fbd17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f64477fe640 10 client.8511 ms_handle_connect on v2:172.21.15.1:6832/1706279305 2024-03-06T07:25:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f646c85f640 1 --2- 172.21.15.1:0/4234863286 >> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] conn(0x55d56fbd13e0 0x55d56fbd17c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d58874788d279fc server_cookie=f413f1cc4283571d in_seq=0 out_seq=0 2024-03-06T07:25:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f64477fe640 1 -- 172.21.15.1:0/4234863286 <== mds.0 v2:172.21.15.1:6832/1706279305 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55d56fa8ef40 con 0x55d56fbd13e0 2024-03-06T07:25:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f64477fe640 10 client.8511 handle_command_reply: tid=0 2024-03-06T07:25:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f6471785740 1 client.8511 shutdown 2024-03-06T07:25:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f6471785740 1 -- 172.21.15.1:0/4234863286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb45f30 msgr2=0x55d56fbcb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f6471785740 1 --2- 172.21.15.1:0/4234863286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb45f30 0x55d56fbcb1b0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f646808c760 tx=0x7f64600046a0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f6471785740 1 -- 172.21.15.1:0/4234863286 shutdown_connections 2024-03-06T07:25:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f6471785740 1 --2- 172.21.15.1:0/4234863286 >> [v2:172.21.15.1:6832/1706279305,v1:172.21.15.1:6834/1706279305] conn(0x55d56fbd13e0 0x55d56fbd17c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f6471785740 1 --2- 172.21.15.1:0/4234863286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d56fb45f30 0x55d56fbcb1b0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f6471785740 1 --2- 172.21.15.1:0/4234863286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d56fb23330 0x55d56fb1d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f64477fe640 0 client.8511 ms_handle_reset on v2:172.21.15.1:6832/1706279305 2024-03-06T07:25:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f6471785740 1 --2- 172.21.15.1:0/4234863286 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55d56fb22f50 0x55d56fb48e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f6471785740 1 -- 172.21.15.1:0/4234863286 >> 172.21.15.1:0/4234863286 conn(0x55d56fb214b0 msgr2=0x55d56fb1c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f6471785740 1 -- 172.21.15.1:0/4234863286 shutdown_connections 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f6471785740 1 -- 172.21.15.1:0/4234863286 wait complete. 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.709+0000 7f6471785740 20 client.8511 trim_cache size 0 max 16384 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 8301, 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 11.288348223, 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 121, 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 135, 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 135, 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 16.360440777000001 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.710+0000 7f646eac3640 1 -- 172.21.15.1:0/1412196329 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f644c06b4d0 msgr2=0x7f644c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.710+0000 7f646eac3640 1 --2- 172.21.15.1:0/1412196329 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f644c06b4d0 0x7f644c06d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f6468155d40 tx=0x7f645c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.710+0000 7f646eac3640 1 -- 172.21.15.1:0/1412196329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64681588c0 msgr2=0x7f64681ee1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.710+0000 7f646eac3640 1 --2- 172.21.15.1:0/1412196329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64681588c0 0x7f64681ee1b0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f6454000c00 tx=0x7f64680bfe40 comp rx=0 tx=0).stop 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.710+0000 7f646eac3640 1 -- 172.21.15.1:0/1412196329 shutdown_connections 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.710+0000 7f646eac3640 1 --2- 172.21.15.1:0/1412196329 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f644c06b4d0 0x7f644c06d990 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.710+0000 7f646eac3640 1 --2- 172.21.15.1:0/1412196329 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64681ee6f0 0x7f646814a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.710+0000 7f646eac3640 1 --2- 172.21.15.1:0/1412196329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64681588c0 0x7f64681ee1b0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.710+0000 7f646eac3640 1 --2- 172.21.15.1:0/1412196329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6458004800 0x7f6468157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.710+0000 7f646eac3640 1 -- 172.21.15.1:0/1412196329 >> 172.21.15.1:0/1412196329 conn(0x7f6468066ee0 msgr2=0x7f64680bff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.710+0000 7f646eac3640 1 -- 172.21.15.1:0/1412196329 shutdown_connections 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:41.710+0000 7f646eac3640 1 -- 172.21.15.1:0/1412196329 wait complete. 2024-03-06T07:25:41.723 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 121, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:25:29.792840+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 121, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8301": { "addr": "172.21.15.1:6834/1706279305", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 1706279305, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 1706279305, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8301, "incarnation": 119, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-03-06T07:25:31.781726+0000", "qdb_cluster": [ 8301 ], "qdb_leader": 8301, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8301 } } } ], "standbys": [ { "addr": "172.21.15.1:6835/2848968195", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 2848968195, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 2848968195, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 118, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6686, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6836/1671554096", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 1671554096, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 1671554096, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 118, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7387, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6837/2739580331", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 2739580331, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2739580331, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 118, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7390, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:25:41.724 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 10, 'name': 'c'}, {'join_fscid': 10, 'name': 'd'}] 2024-03-06T07:25:41.724 INFO:tasks.cephfs.test_failover:current = [{'gid': 8301, 'name': 'a', 'rank': 0, 'incarnation': 119, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6834/1706279305', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1706279305}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1706279305}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7390, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/2739580331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2739580331}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2739580331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 118}, {'gid': 6686, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/2848968195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 2848968195}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2848968195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 118}, {'gid': 7387, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/1671554096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 1671554096}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1671554096}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 118}] 2024-03-06T07:25:41.724 DEBUG:tasks.cephfs.test_failover:10 != -1 2024-03-06T07:25:41.724 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:25:41.794 INFO:tasks.ceph.mds.a.smithi001.stderr: -264> 2024-03-06T07:25:35.713+0000 7f85ca137640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:41.819 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:25:45.717 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-06T07:25:45.715+0000 7f4ef00d3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:46.726 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:46.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.829+0000 7fb357876640 1 Processor -- start 2024-03-06T07:25:46.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.829+0000 7fb357876640 1 -- start start 2024-03-06T07:25:46.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.830+0000 7fb357876640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb350152650 0x7fb350155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.830+0000 7fb357876640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb3500c6b10 0x7fb350158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.830+0000 7fb357876640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3500c2fc0 con 0x7fb3500bf9d0 2024-03-06T07:25:46.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.830+0000 7fb357876640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb350155800 con 0x7fb3500c6b10 2024-03-06T07:25:46.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.830+0000 7fb357876640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3500c6690 con 0x7fb350152650 2024-03-06T07:25:46.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.830+0000 7fb355e13640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb3500bf9d0 0x7fb3500c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54988/0 (socket says 172.21.15.1:54988) 2024-03-06T07:25:46.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.830+0000 7fb355e13640 1 -- 172.21.15.1:0/1743689906 learned_addr learned my addr 172.21.15.1:0/1743689906 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:46.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb347fff640 1 -- 172.21.15.1:0/1743689906 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270687801 0 0) 0x7fb3500c2fc0 con 0x7fb3500bf9d0 2024-03-06T07:25:46.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb347fff640 1 -- 172.21.15.1:0/1743689906 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb33c003520 con 0x7fb3500bf9d0 2024-03-06T07:25:46.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb354e11640 1 --2- 172.21.15.1:0/1743689906 >> v2:172.21.15.98:3300/0 conn(0x7fb3500c6b10 0x7fb350158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb355612640 1 --2- 172.21.15.1:0/1743689906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb350152650 0x7fb350155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb347fff640 1 -- 172.21.15.1:0/1743689906 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2741825888 0 0) 0x7fb33c003520 con 0x7fb3500bf9d0 2024-03-06T07:25:46.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb347fff640 1 -- 172.21.15.1:0/1743689906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb350152650 msgr2=0x7fb350155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:46.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb347fff640 1 --2- 172.21.15.1:0/1743689906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb350152650 0x7fb350155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb347fff640 1 -- 172.21.15.1:0/1743689906 >> v2:172.21.15.98:3300/0 conn(0x7fb3500c6b10 msgr2=0x7fb350158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:46.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb347fff640 1 --2- 172.21.15.1:0/1743689906 >> v2:172.21.15.98:3300/0 conn(0x7fb3500c6b10 0x7fb350158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb347fff640 1 -- 172.21.15.1:0/1743689906 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3500c7280 con 0x7fb3500bf9d0 2024-03-06T07:25:46.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb347fff640 1 -- 172.21.15.1:0/1743689906 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fb3480019f0 con 0x7fb3500bf9d0 2024-03-06T07:25:46.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb347fff640 1 -- 172.21.15.1:0/1743689906 >> v1:172.21.15.1:6789/0 conn(0x7fb3500bf9d0 legacy=0x7fb3500c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb347fff640 1 --2- 172.21.15.1:0/1743689906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb33c0039b0 0x7fb33c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb347fff640 1 --2- 172.21.15.1:0/1743689906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb33c004800 0x7fb33c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb347fff640 1 --2- 172.21.15.1:0/1743689906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb350152650 0x7fb33c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb347fff640 1 -- 172.21.15.1:0/1743689906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb350155800 con 0x7fb33c004800 2024-03-06T07:25:46.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb347fff640 1 -- 172.21.15.1:0/1743689906 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3500c6690 con 0x7fb33c0039b0 2024-03-06T07:25:46.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb347fff640 1 -- 172.21.15.1:0/1743689906 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c00b9c0 con 0x7fb350152650 2024-03-06T07:25:46.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb354e11640 1 --2- 172.21.15.1:0/1743689906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb33c004800 0x7fb33c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb355612640 1 --2- 172.21.15.1:0/1743689906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb33c0039b0 0x7fb33c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb355e13640 1 --2- 172.21.15.1:0/1743689906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb350152650 0x7fb33c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb354e11640 1 -- 172.21.15.1:0/1743689906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb350152650 msgr2=0x7fb33c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:46.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb354e11640 1 --2- 172.21.15.1:0/1743689906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb350152650 0x7fb33c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb354e11640 1 -- 172.21.15.1:0/1743689906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb33c0039b0 msgr2=0x7fb33c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:46.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.831+0000 7fb354e11640 1 --2- 172.21.15.1:0/1743689906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb33c0039b0 0x7fb33c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.832+0000 7fb354e11640 1 -- 172.21.15.1:0/1743689906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c003520 con 0x7fb33c004800 2024-03-06T07:25:46.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.833+0000 7fb354e11640 1 --2- 172.21.15.1:0/1743689906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb33c004800 0x7fb33c004c80 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fb340009af0 tx=0x7fb33c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7211448bcf4b5ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.833+0000 7fb347fff640 1 -- 172.21.15.1:0/1743689906 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb340039070 con 0x7fb33c004800 2024-03-06T07:25:46.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.833+0000 7fb347fff640 1 -- 172.21.15.1:0/1743689906 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34002c710 con 0x7fb33c004800 2024-03-06T07:25:46.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.833+0000 7fb347fff640 1 -- 172.21.15.1:0/1743689906 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb340009690 con 0x7fb33c004800 2024-03-06T07:25:46.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.833+0000 7fb357876640 1 -- 172.21.15.1:0/1743689906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb33c004800 msgr2=0x7fb33c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.833+0000 7fb357876640 1 --2- 172.21.15.1:0/1743689906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb33c004800 0x7fb33c004c80 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fb340009af0 tx=0x7fb33c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:46.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.833+0000 7fb355e13640 1 -- 172.21.15.1:0/1743689906 reap_dead start 2024-03-06T07:25:46.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.833+0000 7fb357876640 1 -- 172.21.15.1:0/1743689906 shutdown_connections 2024-03-06T07:25:46.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.833+0000 7fb357876640 1 -- 172.21.15.1:0/1743689906 >> 172.21.15.1:0/1743689906 conn(0x7fb350066ee0 msgr2=0x7fb3500672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:46.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.833+0000 7fb357876640 1 -- 172.21.15.1:0/1743689906 shutdown_connections 2024-03-06T07:25:46.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.833+0000 7fb357876640 1 -- 172.21.15.1:0/1743689906 wait complete. 2024-03-06T07:25:46.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.834+0000 7fb357876640 1 Processor -- start 2024-03-06T07:25:46.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.835+0000 7fb357876640 1 -- start start 2024-03-06T07:25:46.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.835+0000 7fb357876640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb33c004800 0x7fb3500c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.835+0000 7fb357876640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3501ee150 0x7fb3500c9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.835+0000 7fb357876640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb35014aa60 0x7fb3501483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.835+0000 7fb357876640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb350155490 con 0x7fb33c004800 2024-03-06T07:25:46.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.835+0000 7fb357876640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3501551b0 con 0x7fb3501ee150 2024-03-06T07:25:46.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.835+0000 7fb357876640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb350152a30 con 0x7fb35014aa60 2024-03-06T07:25:46.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.835+0000 7fb355612640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb33c004800 0x7fb3500c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.835+0000 7fb355612640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb33c004800 0x7fb3500c9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36440/0 (socket says 172.21.15.1:36440) 2024-03-06T07:25:46.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.835+0000 7fb355612640 1 -- 172.21.15.1:0/4074372127 learned_addr learned my addr 172.21.15.1:0/4074372127 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:46.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.835+0000 7fb354e11640 1 --2- 172.21.15.1:0/4074372127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3501ee150 0x7fb3500c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.835+0000 7fb355e13640 1 --2- 172.21.15.1:0/4074372127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb35014aa60 0x7fb3501483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.836+0000 7fb355612640 1 -- 172.21.15.1:0/4074372127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb35014aa60 msgr2=0x7fb3501483e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.836+0000 7fb355612640 1 --2- 172.21.15.1:0/4074372127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb35014aa60 0x7fb3501483e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.836+0000 7fb355612640 1 -- 172.21.15.1:0/4074372127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3501ee150 msgr2=0x7fb3500c9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.836+0000 7fb355612640 1 --2- 172.21.15.1:0/4074372127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3501ee150 0x7fb3500c9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.836+0000 7fb355612640 1 -- 172.21.15.1:0/4074372127 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35014b6e0 con 0x7fb33c004800 2024-03-06T07:25:46.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.836+0000 7fb354e11640 1 --2- 172.21.15.1:0/4074372127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3501ee150 0x7fb3500c9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:46.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.836+0000 7fb355612640 1 --2- 172.21.15.1:0/4074372127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb33c004800 0x7fb3500c9a60 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fb34c011940 tx=0x7fb34c011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.836+0000 7fb345ffb640 1 -- 172.21.15.1:0/4074372127 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c00fa90 con 0x7fb33c004800 2024-03-06T07:25:46.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.836+0000 7fb345ffb640 1 -- 172.21.15.1:0/4074372127 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34c00fc30 con 0x7fb33c004800 2024-03-06T07:25:46.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.836+0000 7fb357876640 1 -- 172.21.15.1:0/4074372127 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3500ca0d0 con 0x7fb33c004800 2024-03-06T07:25:46.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.836+0000 7fb345ffb640 1 -- 172.21.15.1:0/4074372127 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c014440 con 0x7fb33c004800 2024-03-06T07:25:46.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.836+0000 7fb357876640 1 -- 172.21.15.1:0/4074372127 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3500c8900 con 0x7fb33c004800 2024-03-06T07:25:46.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.837+0000 7fb357876640 1 -- 172.21.15.1:0/4074372127 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb350067fd0 con 0x7fb33c004800 2024-03-06T07:25:46.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.838+0000 7fb345ffb640 1 -- 172.21.15.1:0/4074372127 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb34c012040 con 0x7fb33c004800 2024-03-06T07:25:46.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.838+0000 7fb345ffb640 1 --2- 172.21.15.1:0/4074372127 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb33406b4d0 0x7fb33406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.838+0000 7fb345ffb640 1 -- 172.21.15.1:0/4074372127 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb34c08a820 con 0x7fb33c004800 2024-03-06T07:25:46.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.838+0000 7fb354e11640 1 --2- 172.21.15.1:0/4074372127 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb33406b4d0 0x7fb33406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.839+0000 7fb354e11640 1 --2- 172.21.15.1:0/4074372127 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb33406b4d0 0x7fb33406d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb340036000 tx=0x7fb340030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:46.842+0000 7fb345ffb640 1 -- 172.21.15.1:0/4074372127 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb35014b6e0 con 0x7fb33c004800 2024-03-06T07:25:47.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.088+0000 7fb357876640 1 -- 172.21.15.1:0/4074372127 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3500b06f0 con 0x7fb33c004800 2024-03-06T07:25:47.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.088+0000 7fb345ffb640 1 -- 172.21.15.1:0/4074372127 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+4983 (secure 0 0 0) 0x7fb34c019030 con 0x7fb33c004800 2024-03-06T07:25:47.091 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:47.091 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":126,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7387,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.98:6836/1671554096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1671554096},{"type":"v1","addr":"172.21.15.98:6836","nonce":1671554096}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122},{"gid":7390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2739580331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2739580331},{"type":"v1","addr":"172.21.15.98:6837","nonce":2739580331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":8517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3374845636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3374845636},{"type":"v1","addr":"172.21.15.1:6834","nonce":3374845636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:29.792840+0000","modified":"2024-03-06T07:25:44.791512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6686":{"gid":6686,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":8,"addr":"172.21.15.1:6835/2848968195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2848968195},{"type":"v1","addr":"172.21.15.1:6835","nonce":2848968195}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6686,"qdb_cluster":[6686]},"id":10}]} 2024-03-06T07:25:47.091 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 126 2024-03-06T07:25:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.090+0000 7fb357876640 1 -- 172.21.15.1:0/4074372127 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb33406b4d0 msgr2=0x7fb33406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.090+0000 7fb357876640 1 --2- 172.21.15.1:0/4074372127 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb33406b4d0 0x7fb33406d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb340036000 tx=0x7fb340030000 comp rx=0 tx=0).stop 2024-03-06T07:25:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.090+0000 7fb357876640 1 -- 172.21.15.1:0/4074372127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb33c004800 msgr2=0x7fb3500c9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.090+0000 7fb357876640 1 --2- 172.21.15.1:0/4074372127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb33c004800 0x7fb3500c9a60 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fb34c011940 tx=0x7fb34c011e00 comp rx=0 tx=0).stop 2024-03-06T07:25:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.090+0000 7fb357876640 1 -- 172.21.15.1:0/4074372127 shutdown_connections 2024-03-06T07:25:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.090+0000 7fb357876640 1 --2- 172.21.15.1:0/4074372127 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb33406b4d0 0x7fb33406d990 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.090+0000 7fb357876640 1 --2- 172.21.15.1:0/4074372127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb35014aa60 0x7fb3501483e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.090+0000 7fb357876640 1 --2- 172.21.15.1:0/4074372127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3501ee150 0x7fb3500c9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.090+0000 7fb357876640 1 --2- 172.21.15.1:0/4074372127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb33c004800 0x7fb3500c9a60 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.090+0000 7fb357876640 1 -- 172.21.15.1:0/4074372127 >> 172.21.15.1:0/4074372127 conn(0x7fb350066ee0 msgr2=0x7fb3500672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.090+0000 7fb357876640 1 -- 172.21.15.1:0/4074372127 shutdown_connections 2024-03-06T07:25:47.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.090+0000 7fb357876640 1 -- 172.21.15.1:0/4074372127 wait complete. 2024-03-06T07:25:47.505 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 126, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:29.792840+0000', 'modified': '2024-03-06T07:25:44.791512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 136, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6686}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6686': {'gid': 6686, 'name': 'c', 'rank': 0, 'incarnation': 123, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.1:6835/2848968195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 2848968195}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2848968195}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6686, 'qdb_cluster': [6686]} 2024-03-06T07:25:47.506 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:47.506 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:25:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.606+0000 7f20e51d0640 1 Processor -- start 2024-03-06T07:25:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.606+0000 7f20e51d0640 1 -- start start 2024-03-06T07:25:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.607+0000 7f20e51d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20e006f640 0x7f20e00c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.607+0000 7f20e51d0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20e00c14c0 0x7f20e01558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.607+0000 7f20e51d0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e00c31f0 con 0x7f20e00c05f0 2024-03-06T07:25:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.607+0000 7f20e51d0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20e006fe90 con 0x7f20e00c14c0 2024-03-06T07:25:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.607+0000 7f20e51d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e00c2fc0 con 0x7f20e006f640 2024-03-06T07:25:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.607+0000 7f20ded9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f20e00c05f0 0x7f20e01551b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54990/0 (socket says 172.21.15.1:54990) 2024-03-06T07:25:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.607+0000 7f20ded9d640 1 -- 172.21.15.1:0/2373094499 learned_addr learned my addr 172.21.15.1:0/2373094499 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.607+0000 7f20d7fff640 1 --2- 172.21.15.1:0/2373094499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20e006f640 0x7f20e00c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.607+0000 7f20df59e640 1 --2- 172.21.15.1:0/2373094499 >> v2:172.21.15.98:3300/0 conn(0x7f20e00c14c0 0x7f20e01558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.607+0000 7f20de59c640 1 -- 172.21.15.1:0/2373094499 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745723604 0 0) 0x7f20e00c31f0 con 0x7f20e00c05f0 2024-03-06T07:25:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.607+0000 7f20de59c640 1 -- 172.21.15.1:0/2373094499 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20cc003520 con 0x7f20e00c05f0 2024-03-06T07:25:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.607+0000 7f20d7fff640 1 -- 172.21.15.1:0/2373094499 >> v2:172.21.15.98:3300/0 conn(0x7f20e00c14c0 msgr2=0x7f20e01558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.607+0000 7f20d7fff640 1 --2- 172.21.15.1:0/2373094499 >> v2:172.21.15.98:3300/0 conn(0x7f20e00c14c0 0x7f20e01558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.607+0000 7f20d7fff640 1 -- 172.21.15.1:0/2373094499 >> v1:172.21.15.1:6789/0 conn(0x7f20e00c05f0 legacy=0x7f20e01551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.607+0000 7f20d7fff640 1 -- 172.21.15.1:0/2373094499 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e0155e00 con 0x7f20e006f640 2024-03-06T07:25:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.608+0000 7f20d7fff640 1 --2- 172.21.15.1:0/2373094499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20e006f640 0x7f20e00c09d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f20c80097a0 tx=0x7f20c802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99a29ccc9f40a365 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.608+0000 7f20de59c640 1 -- 172.21.15.1:0/2373094499 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c8035020 con 0x7f20e006f640 2024-03-06T07:25:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.608+0000 7f20de59c640 1 -- 172.21.15.1:0/2373094499 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c8031830 con 0x7f20e006f640 2024-03-06T07:25:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.608+0000 7f20de59c640 1 -- 172.21.15.1:0/2373094499 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c8031be0 con 0x7f20e006f640 2024-03-06T07:25:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.609+0000 7f20e51d0640 1 -- 172.21.15.1:0/2373094499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20e006f640 msgr2=0x7f20e00c09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.609+0000 7f20e51d0640 1 --2- 172.21.15.1:0/2373094499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20e006f640 0x7f20e00c09d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f20c80097a0 tx=0x7f20c802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.609+0000 7f20e51d0640 1 -- 172.21.15.1:0/2373094499 shutdown_connections 2024-03-06T07:25:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.609+0000 7f20e51d0640 1 --2- 172.21.15.1:0/2373094499 >> v2:172.21.15.98:3300/0 conn(0x7f20e00c14c0 0x7f20e01558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.609+0000 7f20e51d0640 1 --2- 172.21.15.1:0/2373094499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20e006f640 0x7f20e00c09d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.609+0000 7f20e51d0640 1 -- 172.21.15.1:0/2373094499 >> 172.21.15.1:0/2373094499 conn(0x7f20e00c3cc0 msgr2=0x7f20e00c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.609+0000 7f20e51d0640 1 -- 172.21.15.1:0/2373094499 shutdown_connections 2024-03-06T07:25:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.609+0000 7f20e51d0640 1 -- 172.21.15.1:0/2373094499 wait complete. 2024-03-06T07:25:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.610+0000 7f20e51d0640 1 Processor -- start 2024-03-06T07:25:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.610+0000 7f20e51d0640 1 -- start start 2024-03-06T07:25:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.611+0000 7f20e51d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20e006f640 0x7f20e00c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.611+0000 7f20e51d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20e00c05f0 0x7f20e0153270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.611+0000 7f20e51d0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20e00c14c0 0x7f20e01eaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.611+0000 7f20e51d0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e00c3770 con 0x7f20e00c14c0 2024-03-06T07:25:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.611+0000 7f20e51d0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e00c3490 con 0x7f20e00c05f0 2024-03-06T07:25:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.611+0000 7f20e51d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e00c1040 con 0x7f20e006f640 2024-03-06T07:25:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.611+0000 7f20df59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20e00c14c0 0x7f20e01eaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.611+0000 7f20df59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20e00c14c0 0x7f20e01eaca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36454/0 (socket says 172.21.15.1:36454) 2024-03-06T07:25:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.611+0000 7f20df59e640 1 -- 172.21.15.1:0/2586638498 learned_addr learned my addr 172.21.15.1:0/2586638498 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.611+0000 7f20ded9d640 1 --2- 172.21.15.1:0/2586638498 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20e006f640 0x7f20e00c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.611+0000 7f20d7fff640 1 --2- 172.21.15.1:0/2586638498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20e00c05f0 0x7f20e0153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.611+0000 7f20df59e640 1 -- 172.21.15.1:0/2586638498 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20e006f640 msgr2=0x7f20e00c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.611+0000 7f20df59e640 1 --2- 172.21.15.1:0/2586638498 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20e006f640 0x7f20e00c67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.611+0000 7f20df59e640 1 -- 172.21.15.1:0/2586638498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20e00c05f0 msgr2=0x7f20e0153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.611+0000 7f20df59e640 1 --2- 172.21.15.1:0/2586638498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20e00c05f0 0x7f20e0153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.611+0000 7f20df59e640 1 -- 172.21.15.1:0/2586638498 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e00c6d00 con 0x7f20e00c14c0 2024-03-06T07:25:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.612+0000 7f20df59e640 1 --2- 172.21.15.1:0/2586638498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20e00c14c0 0x7f20e01eaca0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f20d8009d30 tx=0x7f20d800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.612+0000 7f20d77fe640 1 -- 172.21.15.1:0/2586638498 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d8018020 con 0x7f20e00c14c0 2024-03-06T07:25:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.612+0000 7f20d77fe640 1 -- 172.21.15.1:0/2586638498 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d8014480 con 0x7f20e00c14c0 2024-03-06T07:25:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.612+0000 7f20e51d0640 1 -- 172.21.15.1:0/2586638498 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e00c5940 con 0x7f20e00c14c0 2024-03-06T07:25:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.612+0000 7f20d77fe640 1 -- 172.21.15.1:0/2586638498 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d8014870 con 0x7f20e00c14c0 2024-03-06T07:25:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.612+0000 7f20e51d0640 1 -- 172.21.15.1:0/2586638498 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e01537b0 con 0x7f20e00c14c0 2024-03-06T07:25:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.614+0000 7f20d77fe640 1 -- 172.21.15.1:0/2586638498 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f20d801b020 con 0x7f20e00c14c0 2024-03-06T07:25:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.614+0000 7f20d77fe640 1 --2- 172.21.15.1:0/2586638498 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20c006b500 0x7f20c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.614+0000 7f20d77fe640 1 -- 172.21.15.1:0/2586638498 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f20d808acf0 con 0x7f20e00c14c0 2024-03-06T07:25:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.614+0000 7f20ded9d640 1 --2- 172.21.15.1:0/2586638498 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20c006b500 0x7f20c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.615+0000 7f20e7db8740 1 Processor -- start 2024-03-06T07:25:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.615+0000 7f20ded9d640 1 --2- 172.21.15.1:0/2586638498 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20c006b500 0x7f20c006d9c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f20d0005880 tx=0x7f20d0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.615+0000 7f20e7db8740 1 -- start start 2024-03-06T07:25:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.616+0000 7f20e7db8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559504914f50 0x559504912a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.616+0000 7f20e7db8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504915330 0x559504912f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.616+0000 7f20e7db8740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5595049134b0 0x55950490f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.616+0000 7f20e7db8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595048796b0 con 0x559504915330 2024-03-06T07:25:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.616+0000 7f20e7db8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559504822a60 con 0x559504914f50 2024-03-06T07:25:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.616+0000 7f20e7db8740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x559504875750 con 0x5595049134b0 2024-03-06T07:25:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.616+0000 7f20d7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504915330 0x559504912f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.616+0000 7f20d7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504915330 0x559504912f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36464/0 (socket says 172.21.15.1:36464) 2024-03-06T07:25:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.616+0000 7f20d7fff640 1 -- 172.21.15.1:0/3407059393 learned_addr learned my addr 172.21.15.1:0/3407059393 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.616+0000 7f20df59e640 1 --2- 172.21.15.1:0/3407059393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5595049134b0 0x55950490f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.616+0000 7f20ded9d640 1 --2- 172.21.15.1:0/3407059393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559504914f50 0x559504912a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.617+0000 7f20d7fff640 1 -- 172.21.15.1:0/3407059393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5595049134b0 msgr2=0x55950490f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.617+0000 7f20d7fff640 1 --2- 172.21.15.1:0/3407059393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5595049134b0 0x55950490f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.617+0000 7f20d7fff640 1 -- 172.21.15.1:0/3407059393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559504914f50 msgr2=0x559504912a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.617+0000 7f20d7fff640 1 --2- 172.21.15.1:0/3407059393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559504914f50 0x559504912a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.617+0000 7f20d7fff640 1 -- 172.21.15.1:0/3407059393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559504840f30 con 0x559504915330 2024-03-06T07:25:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.617+0000 7f20ded9d640 1 --2- 172.21.15.1:0/3407059393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559504914f50 0x559504912a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.617+0000 7f20d7fff640 1 --2- 172.21.15.1:0/3407059393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504915330 0x559504912f70 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f20c80097a0 tx=0x7f20c8030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ff46e7bf16b065 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.617+0000 7f20d4ff9640 1 -- 172.21.15.1:0/3407059393 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c8035040 con 0x559504915330 2024-03-06T07:25:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.617+0000 7f20d4ff9640 1 -- 172.21.15.1:0/3407059393 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c8031630 con 0x559504915330 2024-03-06T07:25:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.617+0000 7f20d4ff9640 1 -- 172.21.15.1:0/3407059393 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c80319e0 con 0x559504915330 2024-03-06T07:25:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.617+0000 7f20e7db8740 1 -- 172.21.15.1:0/3407059393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504915330 msgr2=0x559504912f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.617+0000 7f20e7db8740 1 --2- 172.21.15.1:0/3407059393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504915330 0x559504912f70 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f20c80097a0 tx=0x7f20c8030000 comp rx=0 tx=0).stop 2024-03-06T07:25:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.618+0000 7f20e7db8740 1 -- 172.21.15.1:0/3407059393 shutdown_connections 2024-03-06T07:25:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.618+0000 7f20e7db8740 1 --2- 172.21.15.1:0/3407059393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5595049134b0 0x55950490f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.618+0000 7f20e7db8740 1 --2- 172.21.15.1:0/3407059393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504915330 0x559504912f70 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.618+0000 7f20e7db8740 1 --2- 172.21.15.1:0/3407059393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559504914f50 0x559504912a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.618+0000 7f20e7db8740 1 -- 172.21.15.1:0/3407059393 >> 172.21.15.1:0/3407059393 conn(0x5595049117d0 msgr2=0x5595049363c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.618+0000 7f20e7db8740 1 -- 172.21.15.1:0/3407059393 shutdown_connections 2024-03-06T07:25:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.618+0000 7f20e7db8740 1 -- 172.21.15.1:0/3407059393 wait complete. 2024-03-06T07:25:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.618+0000 7f20e7db8740 1 -- start start 2024-03-06T07:25:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.618+0000 7f20e7db8740 1 Processor -- start 2024-03-06T07:25:47.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.619+0000 7f20e7db8740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x559504914f50 0x5595049357b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.619+0000 7f20e7db8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559504915330 0x559504938050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.619+0000 7f20e7db8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55950493ba70 0x559504932060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.619+0000 7f20e7db8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x559504937230 con 0x55950493ba70 2024-03-06T07:25:47.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.619+0000 7f20e7db8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559504835ea0 con 0x559504915330 2024-03-06T07:25:47.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.619+0000 7f20e7db8740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x559504936c70 con 0x559504914f50 2024-03-06T07:25:47.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.619+0000 7f20e7db8740 10 client.0 _ll_register_callbacks cb 0x559504835870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:47.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.619+0000 7f20df59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55950493ba70 0x559504932060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.619+0000 7f20df59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55950493ba70 0x559504932060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36468/0 (socket says 172.21.15.1:36468) 2024-03-06T07:25:47.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.619+0000 7f20df59e640 1 -- 172.21.15.1:0/254612903 learned_addr learned my addr 172.21.15.1:0/254612903 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:47.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.619+0000 7f20d7fff640 1 --2- 172.21.15.1:0/254612903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559504915330 0x559504938050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.619+0000 7f20ded9d640 1 --2- 172.21.15.1:0/254612903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x559504914f50 0x5595049357b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.619+0000 7f20df59e640 1 -- 172.21.15.1:0/254612903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x559504914f50 msgr2=0x5595049357b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.619+0000 7f20df59e640 1 --2- 172.21.15.1:0/254612903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x559504914f50 0x5595049357b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.619+0000 7f20df59e640 1 -- 172.21.15.1:0/254612903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559504915330 msgr2=0x559504938050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.619+0000 7f20df59e640 1 --2- 172.21.15.1:0/254612903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559504915330 0x559504938050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.619+0000 7f20df59e640 1 -- 172.21.15.1:0/254612903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559504836270 con 0x55950493ba70 2024-03-06T07:25:47.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.620+0000 7f20df59e640 1 --2- 172.21.15.1:0/254612903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55950493ba70 0x559504932060 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f20d80890d0 tx=0x7f20d80593d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83e03f05df4b3bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.620+0000 7f20bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:25:47.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.620+0000 7f20bdffb640 1 -- 172.21.15.1:0/254612903 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d8087060 con 0x55950493ba70 2024-03-06T07:25:47.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.620+0000 7f20bdffb640 1 -- 172.21.15.1:0/254612903 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f20d8058820 con 0x55950493ba70 2024-03-06T07:25:47.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.620+0000 7f20e7db8740 1 -- 172.21.15.1:0/254612903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f20c8031630 con 0x55950493ba70 2024-03-06T07:25:47.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.620+0000 7f20bdffb640 1 -- 172.21.15.1:0/254612903 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=126) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20d8058a20 con 0x55950493ba70 2024-03-06T07:25:47.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.620+0000 7f20e7db8740 10 client.8535 fetch_fsmap learned FSMap version 126 2024-03-06T07:25:47.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.620+0000 7f20e7db8740 1 -- 172.21.15.1:0/254612903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=126,monmap=2+}) v3 -- 0x55950458e2d0 con 0x55950493ba70 2024-03-06T07:25:47.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.620+0000 7f20bdffb640 1 -- 172.21.15.1:0/254612903 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c8031630 con 0x55950493ba70 2024-03-06T07:25:47.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.620+0000 7f20bdffb640 1 -- 172.21.15.1:0/254612903 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 126) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f20d805f400 con 0x55950493ba70 2024-03-06T07:25:47.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.620+0000 7f20e7db8740 10 client.8535 fetch_fsmap finished waiting for FSMap version 126 2024-03-06T07:25:47.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.620+0000 7f20e7db8740 10 client.8535 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:25:47.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.620+0000 7f20e7db8740 20 client.8535 populate_metadata read hostname 'smithi001' 2024-03-06T07:25:47.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.620+0000 7f20e7db8740 1 --2- 172.21.15.1:0/254612903 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x559504965480 0x5595049702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.620+0000 7f20e7db8740 4 client.8535 mds_command: new command op to 6686 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:47.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.620+0000 7f20e7db8740 1 -- 172.21.15.1:0/254612903 --> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559504886bf0 con 0x559504965480 2024-03-06T07:25:47.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.620+0000 7f20ded9d640 1 --2- 172.21.15.1:0/254612903 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x559504965480 0x5595049702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.621+0000 7f20bdffb640 10 client.8535 ms_handle_connect on v2:172.21.15.1:6833/2848968195 2024-03-06T07:25:47.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.621+0000 7f20ded9d640 1 --2- 172.21.15.1:0/254612903 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x559504965480 0x5595049702c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=234f7af8e0e75cd9 server_cookie=159eda7109b34f7e in_seq=0 out_seq=0 2024-03-06T07:25:47.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.622+0000 7f20bdffb640 1 -- 172.21.15.1:0/254612903 <== mds.0 v2:172.21.15.1:6833/2848968195 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559504886bf0 con 0x559504965480 2024-03-06T07:25:47.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.622+0000 7f20bdffb640 10 client.8535 handle_command_reply: tid=0 2024-03-06T07:25:47.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.622+0000 7f20e7db8740 1 client.8535 shutdown 2024-03-06T07:25:47.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.622+0000 7f20e7db8740 1 -- 172.21.15.1:0/254612903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55950493ba70 msgr2=0x559504932060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.622+0000 7f20e7db8740 1 --2- 172.21.15.1:0/254612903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55950493ba70 0x559504932060 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f20d80890d0 tx=0x7f20d80593d0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.622+0000 7f20e7db8740 1 -- 172.21.15.1:0/254612903 shutdown_connections 2024-03-06T07:25:47.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.622+0000 7f20e7db8740 1 --2- 172.21.15.1:0/254612903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55950493ba70 0x559504932060 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.622+0000 7f20e7db8740 1 --2- 172.21.15.1:0/254612903 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x559504965480 0x5595049702c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.622+0000 7f20e7db8740 1 --2- 172.21.15.1:0/254612903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559504915330 0x559504938050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.622+0000 7f20e7db8740 1 --2- 172.21.15.1:0/254612903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x559504914f50 0x5595049357b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.622+0000 7f20bdffb640 0 client.8535 ms_handle_reset on v2:172.21.15.1:6833/2848968195 2024-03-06T07:25:47.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.622+0000 7f20e7db8740 1 -- 172.21.15.1:0/254612903 >> 172.21.15.1:0/254612903 conn(0x5595049134b0 msgr2=0x5595049138e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.622+0000 7f20e7db8740 1 -- 172.21.15.1:0/254612903 shutdown_connections 2024-03-06T07:25:47.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.622+0000 7f20e7db8740 1 -- 172.21.15.1:0/254612903 wait complete. 2024-03-06T07:25:47.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.622+0000 7f20e7db8740 20 client.8535 trim_cache size 0 max 16384 2024-03-06T07:25:47.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.639+0000 7f20e7db8740 1 Processor -- start 2024-03-06T07:25:47.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.639+0000 7f20e7db8740 1 -- start start 2024-03-06T07:25:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.640+0000 7f20e7db8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559504914f50 0x559504934940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.640+0000 7f20e7db8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504915330 0x5595049125a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.640+0000 7f20e7db8740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55950493ba70 0x559504912ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.640+0000 7f20e7db8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e01f6af0 con 0x559504915330 2024-03-06T07:25:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.640+0000 7f20e7db8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e01402a0 con 0x559504914f50 2024-03-06T07:25:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.640+0000 7f20e7db8740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e01eed40 con 0x55950493ba70 2024-03-06T07:25:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.640+0000 7f20d7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504915330 0x5595049125a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.640+0000 7f20d7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504915330 0x5595049125a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36482/0 (socket says 172.21.15.1:36482) 2024-03-06T07:25:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.640+0000 7f20d7fff640 1 -- 172.21.15.1:0/1730613793 learned_addr learned my addr 172.21.15.1:0/1730613793 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.640+0000 7f20df59e640 1 --2- 172.21.15.1:0/1730613793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55950493ba70 0x559504912ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.640+0000 7f20d7fff640 1 -- 172.21.15.1:0/1730613793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55950493ba70 msgr2=0x559504912ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.640+0000 7f20d7fff640 1 --2- 172.21.15.1:0/1730613793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55950493ba70 0x559504912ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.640+0000 7f20d7fff640 1 -- 172.21.15.1:0/1730613793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559504914f50 msgr2=0x559504934940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.640+0000 7f20d7fff640 1 --2- 172.21.15.1:0/1730613793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559504914f50 0x559504934940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.640+0000 7f20d7fff640 1 -- 172.21.15.1:0/1730613793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595046586b0 con 0x559504915330 2024-03-06T07:25:47.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.640+0000 7f20df59e640 1 --2- 172.21.15.1:0/1730613793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55950493ba70 0x559504912ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:47.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.641+0000 7f20d7fff640 1 --2- 172.21.15.1:0/1730613793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504915330 0x5595049125a0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f20c80098c0 tx=0x7f20c8031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1173aa0b3b756aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.641+0000 7f20bffff640 1 -- 172.21.15.1:0/1730613793 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c8035040 con 0x559504915330 2024-03-06T07:25:47.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.641+0000 7f20bffff640 1 -- 172.21.15.1:0/1730613793 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c8006a20 con 0x559504915330 2024-03-06T07:25:47.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.641+0000 7f20bffff640 1 -- 172.21.15.1:0/1730613793 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c8031630 con 0x559504915330 2024-03-06T07:25:47.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.641+0000 7f20e7db8740 1 -- 172.21.15.1:0/1730613793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504915330 msgr2=0x5595049125a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.641+0000 7f20e7db8740 1 --2- 172.21.15.1:0/1730613793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504915330 0x5595049125a0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f20c80098c0 tx=0x7f20c8031000 comp rx=0 tx=0).stop 2024-03-06T07:25:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.641+0000 7f20e7db8740 1 -- 172.21.15.1:0/1730613793 shutdown_connections 2024-03-06T07:25:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.641+0000 7f20e7db8740 1 --2- 172.21.15.1:0/1730613793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55950493ba70 0x559504912ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.641+0000 7f20e7db8740 1 --2- 172.21.15.1:0/1730613793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504915330 0x5595049125a0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.641+0000 7f20e7db8740 1 --2- 172.21.15.1:0/1730613793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559504914f50 0x559504934940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.641+0000 7f20e7db8740 1 -- 172.21.15.1:0/1730613793 >> 172.21.15.1:0/1730613793 conn(0x5595049134b0 msgr2=0x55950494e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.641+0000 7f20e7db8740 1 -- 172.21.15.1:0/1730613793 shutdown_connections 2024-03-06T07:25:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.641+0000 7f20e7db8740 1 -- 172.21.15.1:0/1730613793 wait complete. 2024-03-06T07:25:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.642+0000 7f20e7db8740 1 -- start start 2024-03-06T07:25:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.642+0000 7f20e7db8740 1 Processor -- start 2024-03-06T07:25:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.642+0000 7f20e7db8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504914f50 0x55950494d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.642+0000 7f20e7db8740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x559504915330 0x559504974c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20e7db8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950493ba70 0x55950495e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20e7db8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x559504937b90 con 0x559504914f50 2024-03-06T07:25:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20e7db8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559504965300 con 0x55950493ba70 2024-03-06T07:25:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20e7db8740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x559504936340 con 0x559504915330 2024-03-06T07:25:47.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20e7db8740 10 client.0 _ll_register_callbacks cb 0x7f20d8058f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:47.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20ded9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504914f50 0x55950494d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20ded9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504914f50 0x55950494d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36488/0 (socket says 172.21.15.1:36488) 2024-03-06T07:25:47.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20ded9d640 1 -- 172.21.15.1:0/212343873 learned_addr learned my addr 172.21.15.1:0/212343873 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:47.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20df59e640 1 --2- 172.21.15.1:0/212343873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950493ba70 0x55950495e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20d7fff640 1 --2- 172.21.15.1:0/212343873 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x559504915330 0x559504974c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20ded9d640 1 -- 172.21.15.1:0/212343873 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x559504915330 msgr2=0x559504974c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20ded9d640 1 --2- 172.21.15.1:0/212343873 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x559504915330 0x559504974c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20ded9d640 1 -- 172.21.15.1:0/212343873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950493ba70 msgr2=0x55950495e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20ded9d640 1 --2- 172.21.15.1:0/212343873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950493ba70 0x55950495e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20ded9d640 1 -- 172.21.15.1:0/212343873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55950495ddc0 con 0x559504914f50 2024-03-06T07:25:47.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20ded9d640 1 --2- 172.21.15.1:0/212343873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504914f50 0x55950494d250 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x55950495f2b0 tx=0x7f20d0007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b864d0e1640a9004 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:25:47.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20bdffb640 1 -- 172.21.15.1:0/212343873 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d0026050 con 0x559504914f50 2024-03-06T07:25:47.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20bdffb640 1 -- 172.21.15.1:0/212343873 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f20d0010040 con 0x559504914f50 2024-03-06T07:25:47.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.643+0000 7f20e7db8740 1 -- 172.21.15.1:0/212343873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f20c8006a20 con 0x559504914f50 2024-03-06T07:25:47.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.644+0000 7f20bdffb640 1 -- 172.21.15.1:0/212343873 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=126) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20d0035050 con 0x559504914f50 2024-03-06T07:25:47.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.644+0000 7f20e7db8740 10 client.8541 fetch_fsmap learned FSMap version 126 2024-03-06T07:25:47.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.644+0000 7f20e7db8740 1 -- 172.21.15.1:0/212343873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=126,monmap=2+}) v3 -- 0x55950493b200 con 0x559504914f50 2024-03-06T07:25:47.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.644+0000 7f20bdffb640 1 -- 172.21.15.1:0/212343873 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c8006a20 con 0x559504914f50 2024-03-06T07:25:47.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.644+0000 7f20bdffb640 1 -- 172.21.15.1:0/212343873 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 126) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f20d0038020 con 0x559504914f50 2024-03-06T07:25:47.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.644+0000 7f20e7db8740 10 client.8541 fetch_fsmap finished waiting for FSMap version 126 2024-03-06T07:25:47.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.644+0000 7f20e7db8740 10 client.8541 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:25:47.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.644+0000 7f20e7db8740 20 client.8541 populate_metadata read hostname 'smithi001' 2024-03-06T07:25:47.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.644+0000 7f20e7db8740 1 --2- 172.21.15.1:0/212343873 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x5595049bece0 0x5595049bf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.644+0000 7f20e7db8740 4 client.8541 mds_command: new command op to 6686 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:47.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.644+0000 7f20e7db8740 1 -- 172.21.15.1:0/212343873 --> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55950493cdd0 con 0x5595049bece0 2024-03-06T07:25:47.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.644+0000 7f20d7fff640 1 --2- 172.21.15.1:0/212343873 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x5595049bece0 0x5595049bf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.644+0000 7f20bdffb640 10 client.8541 ms_handle_connect on v2:172.21.15.1:6833/2848968195 2024-03-06T07:25:47.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.644+0000 7f20d7fff640 1 --2- 172.21.15.1:0/212343873 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x5595049bece0 0x5595049bf0c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e160b8a7a49a33c server_cookie=55e416568b5e1734 in_seq=0 out_seq=0 2024-03-06T07:25:47.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.644+0000 7f20bdffb640 1 -- 172.21.15.1:0/212343873 <== mds.0 v2:172.21.15.1:6833/2848968195 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55950493cdd0 con 0x5595049bece0 2024-03-06T07:25:47.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.644+0000 7f20bdffb640 10 client.8541 handle_command_reply: tid=0 2024-03-06T07:25:47.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.644+0000 7f20e7db8740 1 client.8541 shutdown 2024-03-06T07:25:47.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e7db8740 1 -- 172.21.15.1:0/212343873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504914f50 msgr2=0x55950494d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e7db8740 1 --2- 172.21.15.1:0/212343873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504914f50 0x55950494d250 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x55950495f2b0 tx=0x7f20d0007d10 comp rx=0 tx=0).stop 2024-03-06T07:25:47.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e7db8740 1 -- 172.21.15.1:0/212343873 shutdown_connections 2024-03-06T07:25:47.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e7db8740 1 --2- 172.21.15.1:0/212343873 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x5595049bece0 0x5595049bf0c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e7db8740 1 --2- 172.21.15.1:0/212343873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950493ba70 0x55950495e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e7db8740 1 --2- 172.21.15.1:0/212343873 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x559504915330 0x559504974c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20bdffb640 0 client.8541 ms_handle_reset on v2:172.21.15.1:6833/2848968195 2024-03-06T07:25:47.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e7db8740 1 --2- 172.21.15.1:0/212343873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559504914f50 0x55950494d250 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e7db8740 1 -- 172.21.15.1:0/212343873 >> 172.21.15.1:0/212343873 conn(0x5595049134b0 msgr2=0x55950494e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e7db8740 1 -- 172.21.15.1:0/212343873 shutdown_connections 2024-03-06T07:25:47.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e7db8740 1 -- 172.21.15.1:0/212343873 wait complete. 2024-03-06T07:25:47.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e7db8740 20 client.8541 trim_cache size 0 max 16384 2024-03-06T07:25:47.653 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:25:47.653 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:25:47.653 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:25:47.653 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 6686, 2024-03-06T07:25:47.653 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:25:47.653 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:25:47.653 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:25:47.653 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 5.8561369010000002, 2024-03-06T07:25:47.653 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 126, 2024-03-06T07:25:47.654 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 136, 2024-03-06T07:25:47.654 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 136, 2024-03-06T07:25:47.654 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 22.295841981999999 2024-03-06T07:25:47.654 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:25:47.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e51d0640 1 -- 172.21.15.1:0/2586638498 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20c006b500 msgr2=0x7f20c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e51d0640 1 --2- 172.21.15.1:0/2586638498 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20c006b500 0x7f20c006d9c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f20d0005880 tx=0x7f20d0009040 comp rx=0 tx=0).stop 2024-03-06T07:25:47.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e51d0640 1 -- 172.21.15.1:0/2586638498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20e00c14c0 msgr2=0x7f20e01eaca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e51d0640 1 --2- 172.21.15.1:0/2586638498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20e00c14c0 0x7f20e01eaca0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f20d8009d30 tx=0x7f20d800c490 comp rx=0 tx=0).stop 2024-03-06T07:25:47.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e51d0640 1 -- 172.21.15.1:0/2586638498 shutdown_connections 2024-03-06T07:25:47.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e51d0640 1 --2- 172.21.15.1:0/2586638498 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f20c006b500 0x7f20c006d9c0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e51d0640 1 --2- 172.21.15.1:0/2586638498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20e00c14c0 0x7f20e01eaca0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e51d0640 1 --2- 172.21.15.1:0/2586638498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20e00c05f0 0x7f20e0153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e51d0640 1 --2- 172.21.15.1:0/2586638498 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20e006f640 0x7f20e00c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e51d0640 1 -- 172.21.15.1:0/2586638498 >> 172.21.15.1:0/2586638498 conn(0x7f20e00c3cc0 msgr2=0x7f20e0149c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e51d0640 1 -- 172.21.15.1:0/2586638498 shutdown_connections 2024-03-06T07:25:47.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:47.645+0000 7f20e51d0640 1 -- 172.21.15.1:0/2586638498 wait complete. 2024-03-06T07:25:47.658 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 126, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:25:29.792840+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 126, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6686": { "addr": "172.21.15.1:6835/2848968195", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 2848968195, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 2848968195, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6686, "incarnation": 123, "join_fscid": 10, "name": "c", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 136, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-03-06T07:25:44.791512+0000", "qdb_cluster": [ 6686 ], "qdb_leader": 6686, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6686 } } } ], "standbys": [ { "addr": "172.21.15.98:6836/1671554096", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 1671554096, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 1671554096, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 122, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7387, "incarnation": 0, "join_fscid": 10, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 5 }, { "addr": "172.21.15.98:6837/2739580331", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 2739580331, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2739580331, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 118, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7390, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6834/3374845636", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 3374845636, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 3374845636, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 124, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8517, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:25:47.659 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 10, 'name': 'c'}, {'join_fscid': 10, 'name': 'd'}] 2024-03-06T07:25:47.659 INFO:tasks.cephfs.test_failover:current = [{'gid': 8517, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6834/3374845636', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 3374845636}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 3374845636}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 124}, {'gid': 7390, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/2739580331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2739580331}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2739580331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 118}, {'gid': 6686, 'name': 'c', 'rank': 0, 'incarnation': 123, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.1:6835/2848968195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 2848968195}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2848968195}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7387, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 5, 'addr': '172.21.15.98:6836/1671554096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 1671554096}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1671554096}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 122}] 2024-03-06T07:25:47.659 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:25:52.660 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:52.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.760+0000 7f62d1e39640 1 Processor -- start 2024-03-06T07:25:52.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.760+0000 7f62d1e39640 1 -- start start 2024-03-06T07:25:52.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.760+0000 7f62d1e39640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f62cc0c05f0 0x7f62cc0c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.760+0000 7f62d1e39640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc0c31f0 con 0x7f62cc0c14c0 2024-03-06T07:25:52.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.760+0000 7f62d1e39640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc06fbb0 con 0x7f62cc06f640 2024-03-06T07:25:52.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.760+0000 7f62d1e39640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62cc06fe90 con 0x7f62cc0c05f0 2024-03-06T07:25:52.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.761+0000 7f62cb7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f62cc0c05f0 0x7f62cc0c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.761+0000 7f62cbfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f62cc0c14c0 0x7f62cc1558c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54998/0 (socket says 172.21.15.1:54998) 2024-03-06T07:25:52.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.761+0000 7f62cbfff640 1 -- 172.21.15.1:0/534390725 learned_addr learned my addr 172.21.15.1:0/534390725 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:52.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.761+0000 7f62cb7fe640 1 -- 172.21.15.1:0/534390725 >> v1:172.21.15.98:6789/0 conn(0x7f62cc06f640 legacy=0x7f62cc1551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.761+0000 7f62cb7fe640 1 -- 172.21.15.1:0/534390725 >> v1:172.21.15.1:6789/0 conn(0x7f62cc0c14c0 legacy=0x7f62cc1558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.761+0000 7f62cb7fe640 1 -- 172.21.15.1:0/534390725 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc0c3490 con 0x7f62cc0c05f0 2024-03-06T07:25:52.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.762+0000 7f62cb7fe640 1 --2- 172.21.15.1:0/534390725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62cc0c05f0 0x7f62cc0c09d0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f62bc0097a0 tx=0x7f62bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae454566d28c84b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.762+0000 7f62caffd640 1 -- 172.21.15.1:0/534390725 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc035020 con 0x7f62cc0c05f0 2024-03-06T07:25:52.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.762+0000 7f62caffd640 1 -- 172.21.15.1:0/534390725 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62bc031830 con 0x7f62cc0c05f0 2024-03-06T07:25:52.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.762+0000 7f62caffd640 1 -- 172.21.15.1:0/534390725 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc031c00 con 0x7f62cc0c05f0 2024-03-06T07:25:52.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.762+0000 7f62d1e39640 1 -- 172.21.15.1:0/534390725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62cc0c05f0 msgr2=0x7f62cc0c09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.762+0000 7f62d1e39640 1 --2- 172.21.15.1:0/534390725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62cc0c05f0 0x7f62cc0c09d0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f62bc0097a0 tx=0x7f62bc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.763+0000 7f62d1e39640 1 -- 172.21.15.1:0/534390725 shutdown_connections 2024-03-06T07:25:52.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.763+0000 7f62d1e39640 1 --2- 172.21.15.1:0/534390725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62cc0c05f0 0x7f62cc0c09d0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.763+0000 7f62d1e39640 1 -- 172.21.15.1:0/534390725 >> 172.21.15.1:0/534390725 conn(0x7f62cc0c3cc0 msgr2=0x7f62cc0c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.763+0000 7f62d1e39640 1 -- 172.21.15.1:0/534390725 shutdown_connections 2024-03-06T07:25:52.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.763+0000 7f62d1e39640 1 -- 172.21.15.1:0/534390725 wait complete. 2024-03-06T07:25:52.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.764+0000 7f62d1e39640 1 Processor -- start 2024-03-06T07:25:52.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.764+0000 7f62d1e39640 1 -- start start 2024-03-06T07:25:52.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.764+0000 7f62d1e39640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62cc06f640 0x7f62cc0c4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.764+0000 7f62d1e39640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62cc0c05f0 0x7f62cc1427b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62d1e39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62cc0c14c0 0x7f62cc1ee620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62d1e39640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62cc0c1040 con 0x7f62cc0c05f0 2024-03-06T07:25:52.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62d1e39640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62cc14f460 con 0x7f62cc0c14c0 2024-03-06T07:25:52.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62d1e39640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62cc1609d0 con 0x7f62cc06f640 2024-03-06T07:25:52.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62c2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62cc0c05f0 0x7f62cc1427b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62c2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62cc0c05f0 0x7f62cc1427b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36508/0 (socket says 172.21.15.1:36508) 2024-03-06T07:25:52.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62c2ffd640 1 -- 172.21.15.1:0/351405190 learned_addr learned my addr 172.21.15.1:0/351405190 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:52.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62cbfff640 1 --2- 172.21.15.1:0/351405190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62cc0c14c0 0x7f62cc1ee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62cb7fe640 1 --2- 172.21.15.1:0/351405190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62cc06f640 0x7f62cc0c4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62c2ffd640 1 -- 172.21.15.1:0/351405190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62cc06f640 msgr2=0x7f62cc0c4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62c2ffd640 1 --2- 172.21.15.1:0/351405190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62cc06f640 0x7f62cc0c4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62c2ffd640 1 -- 172.21.15.1:0/351405190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62cc0c14c0 msgr2=0x7f62cc1ee620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62c2ffd640 1 --2- 172.21.15.1:0/351405190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62cc0c14c0 0x7f62cc1ee620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62c2ffd640 1 -- 172.21.15.1:0/351405190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc157e00 con 0x7f62cc0c05f0 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62cb7fe640 1 --2- 172.21.15.1:0/351405190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62cc06f640 0x7f62cc0c4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62cbfff640 1 --2- 172.21.15.1:0/351405190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62cc0c14c0 0x7f62cc1ee620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62c2ffd640 1 --2- 172.21.15.1:0/351405190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62cc0c05f0 0x7f62cc1427b0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f62b400c460 tx=0x7f62b400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62c8ff9640 1 -- 172.21.15.1:0/351405190 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b4016020 con 0x7f62cc0c05f0 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62c8ff9640 1 -- 172.21.15.1:0/351405190 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b4012870 con 0x7f62cc0c05f0 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62d1e39640 1 -- 172.21.15.1:0/351405190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62cc157340 con 0x7f62cc0c05f0 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62c8ff9640 1 -- 172.21.15.1:0/351405190 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b4012c60 con 0x7f62cc0c05f0 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.765+0000 7f62d1e39640 1 -- 172.21.15.1:0/351405190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62cc157060 con 0x7f62cc0c05f0 2024-03-06T07:25:52.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.766+0000 7f62d1e39640 1 -- 172.21.15.1:0/351405190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62cc0c2ae0 con 0x7f62cc0c05f0 2024-03-06T07:25:52.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.767+0000 7f62c8ff9640 1 -- 172.21.15.1:0/351405190 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f62b4019020 con 0x7f62cc0c05f0 2024-03-06T07:25:52.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.767+0000 7f62c8ff9640 1 --2- 172.21.15.1:0/351405190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f62ac06b500 0x7f62ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.768+0000 7f62c8ff9640 1 -- 172.21.15.1:0/351405190 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f62b4089e40 con 0x7f62cc0c05f0 2024-03-06T07:25:52.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.768+0000 7f62cb7fe640 1 --2- 172.21.15.1:0/351405190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f62ac06b500 0x7f62ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.768+0000 7f62cb7fe640 1 --2- 172.21.15.1:0/351405190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f62ac06b500 0x7f62ac06d9c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f62bc036000 tx=0x7f62bc008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.772+0000 7f62c8ff9640 1 -- 172.21.15.1:0/351405190 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f62cc0c2ae0 con 0x7f62cc0c05f0 2024-03-06T07:25:52.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.990+0000 7f62d1e39640 1 -- 172.21.15.1:0/351405190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62cc067fd0 con 0x7f62cc0c05f0 2024-03-06T07:25:52.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.990+0000 7f62c8ff9640 1 -- 172.21.15.1:0/351405190 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+4983 (secure 0 0 0) 0x7f62b4020070 con 0x7f62cc0c05f0 2024-03-06T07:25:52.992 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:52.992 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":126,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7387,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.98:6836/1671554096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1671554096},{"type":"v1","addr":"172.21.15.98:6836","nonce":1671554096}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122},{"gid":7390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2739580331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2739580331},{"type":"v1","addr":"172.21.15.98:6837","nonce":2739580331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":8517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3374845636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3374845636},{"type":"v1","addr":"172.21.15.1:6834","nonce":3374845636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:29.792840+0000","modified":"2024-03-06T07:25:44.791512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6686":{"gid":6686,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":8,"addr":"172.21.15.1:6835/2848968195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2848968195},{"type":"v1","addr":"172.21.15.1:6835","nonce":2848968195}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6686,"qdb_cluster":[6686]},"id":10}]} 2024-03-06T07:25:52.992 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 126 2024-03-06T07:25:52.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.992+0000 7f62d1e39640 1 -- 172.21.15.1:0/351405190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f62ac06b500 msgr2=0x7f62ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.992+0000 7f62d1e39640 1 --2- 172.21.15.1:0/351405190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f62ac06b500 0x7f62ac06d9c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f62bc036000 tx=0x7f62bc008030 comp rx=0 tx=0).stop 2024-03-06T07:25:52.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.992+0000 7f62d1e39640 1 -- 172.21.15.1:0/351405190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62cc0c05f0 msgr2=0x7f62cc1427b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.992+0000 7f62d1e39640 1 --2- 172.21.15.1:0/351405190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62cc0c05f0 0x7f62cc1427b0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f62b400c460 tx=0x7f62b400c920 comp rx=0 tx=0).stop 2024-03-06T07:25:52.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.992+0000 7f62d1e39640 1 -- 172.21.15.1:0/351405190 shutdown_connections 2024-03-06T07:25:52.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.992+0000 7f62d1e39640 1 --2- 172.21.15.1:0/351405190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f62ac06b500 0x7f62ac06d9c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.992+0000 7f62d1e39640 1 --2- 172.21.15.1:0/351405190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62cc0c14c0 0x7f62cc1ee620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.992+0000 7f62d1e39640 1 --2- 172.21.15.1:0/351405190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62cc0c05f0 0x7f62cc1427b0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.992+0000 7f62d1e39640 1 --2- 172.21.15.1:0/351405190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f62cc06f640 0x7f62cc0c4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.992+0000 7f62d1e39640 1 -- 172.21.15.1:0/351405190 >> 172.21.15.1:0/351405190 conn(0x7f62cc0c3cc0 msgr2=0x7f62cc1551b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.992+0000 7f62d1e39640 1 -- 172.21.15.1:0/351405190 shutdown_connections 2024-03-06T07:25:52.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:52.992+0000 7f62d1e39640 1 -- 172.21.15.1:0/351405190 wait complete. 2024-03-06T07:25:53.407 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 126, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:29.792840+0000', 'modified': '2024-03-06T07:25:44.791512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 136, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6686}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6686': {'gid': 6686, 'name': 'c', 'rank': 0, 'incarnation': 123, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.1:6835/2848968195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 2848968195}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2848968195}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6686, 'qdb_cluster': [6686]} 2024-03-06T07:25:53.407 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:53.408 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:25:53.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.510+0000 7fd5657bc640 1 Processor -- start 2024-03-06T07:25:53.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.510+0000 7fd5657bc640 1 -- start start 2024-03-06T07:25:53.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.511+0000 7fd5657bc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd560152730 0x7fd5600c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.511+0000 7fd5657bc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5601588e0 con 0x7fd5600c1aa0 2024-03-06T07:25:53.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.511+0000 7fd5657bc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd560158bc0 con 0x7fd5600c6fd0 2024-03-06T07:25:53.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.511+0000 7fd5657bc640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd560158ea0 con 0x7fd560152730 2024-03-06T07:25:53.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.511+0000 7fd55effd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd560152730 0x7fd5600c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.511+0000 7fd55f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd5600c1aa0 0x7fd5601552e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55010/0 (socket says 172.21.15.1:55010) 2024-03-06T07:25:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.511+0000 7fd55f7fe640 1 -- 172.21.15.1:0/4062742419 learned_addr learned my addr 172.21.15.1:0/4062742419 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.511+0000 7fd55e7fc640 1 -- 172.21.15.1:0/4062742419 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841806955 0 0) 0x7fd5601588e0 con 0x7fd5600c1aa0 2024-03-06T07:25:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.511+0000 7fd55e7fc640 1 -- 172.21.15.1:0/4062742419 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd54c003520 con 0x7fd5600c1aa0 2024-03-06T07:25:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.511+0000 7fd55effd640 1 -- 172.21.15.1:0/4062742419 >> v1:172.21.15.98:6789/0 conn(0x7fd5600c6fd0 legacy=0x7fd5600c1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.511+0000 7fd55effd640 1 -- 172.21.15.1:0/4062742419 >> v1:172.21.15.1:6789/0 conn(0x7fd5600c1aa0 legacy=0x7fd5601552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.511+0000 7fd55effd640 1 -- 172.21.15.1:0/4062742419 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd560159180 con 0x7fd560152730 2024-03-06T07:25:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.512+0000 7fd55effd640 1 --2- 172.21.15.1:0/4062742419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd560152730 0x7fd5600c6a90 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fd5500097a0 tx=0x7fd55002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a12369e5336f7dac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.512+0000 7fd55e7fc640 1 -- 172.21.15.1:0/4062742419 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550038070 con 0x7fd560152730 2024-03-06T07:25:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.512+0000 7fd55e7fc640 1 -- 172.21.15.1:0/4062742419 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd550031700 con 0x7fd560152730 2024-03-06T07:25:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.512+0000 7fd55e7fc640 1 -- 172.21.15.1:0/4062742419 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550031ad0 con 0x7fd560152730 2024-03-06T07:25:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.513+0000 7fd5657bc640 1 -- 172.21.15.1:0/4062742419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd560152730 msgr2=0x7fd5600c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.513+0000 7fd5657bc640 1 --2- 172.21.15.1:0/4062742419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd560152730 0x7fd5600c6a90 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fd5500097a0 tx=0x7fd55002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.513+0000 7fd5657bc640 1 -- 172.21.15.1:0/4062742419 shutdown_connections 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.513+0000 7fd5657bc640 1 --2- 172.21.15.1:0/4062742419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd560152730 0x7fd5600c6a90 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.513+0000 7fd5657bc640 1 -- 172.21.15.1:0/4062742419 >> 172.21.15.1:0/4062742419 conn(0x7fd560066d00 msgr2=0x7fd560067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.513+0000 7fd5657bc640 1 -- 172.21.15.1:0/4062742419 shutdown_connections 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.513+0000 7fd5657bc640 1 -- 172.21.15.1:0/4062742419 wait complete. 2024-03-06T07:25:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.514+0000 7fd5657bc640 1 Processor -- start 2024-03-06T07:25:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.514+0000 7fd5657bc640 1 -- start start 2024-03-06T07:25:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.514+0000 7fd5657bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5600c1aa0 0x7fd5600c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.514+0000 7fd5657bc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5600c6fd0 0x7fd560142000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.514+0000 7fd5657bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd560152730 0x7fd560142540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd5657bc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5601559f0 con 0x7fd5600c6fd0 2024-03-06T07:25:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd5657bc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd560152b10 con 0x7fd5600c1aa0 2024-03-06T07:25:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd5657bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd560155b70 con 0x7fd560152730 2024-03-06T07:25:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd557fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5600c6fd0 0x7fd560142000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd557fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5600c6fd0 0x7fd560142000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36538/0 (socket says 172.21.15.1:36538) 2024-03-06T07:25:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd557fff640 1 -- 172.21.15.1:0/412886255 learned_addr learned my addr 172.21.15.1:0/412886255 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd55f7fe640 1 --2- 172.21.15.1:0/412886255 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd560152730 0x7fd560142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd55effd640 1 --2- 172.21.15.1:0/412886255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5600c1aa0 0x7fd5600c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd557fff640 1 -- 172.21.15.1:0/412886255 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd560152730 msgr2=0x7fd560142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd557fff640 1 --2- 172.21.15.1:0/412886255 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd560152730 0x7fd560142540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd557fff640 1 -- 172.21.15.1:0/412886255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5600c1aa0 msgr2=0x7fd5600c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd557fff640 1 --2- 172.21.15.1:0/412886255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5600c1aa0 0x7fd5600c9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd557fff640 1 -- 172.21.15.1:0/412886255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd560142a80 con 0x7fd5600c6fd0 2024-03-06T07:25:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd557fff640 1 --2- 172.21.15.1:0/412886255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5600c6fd0 0x7fd560142000 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fd54800c460 tx=0x7fd54800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd5577fe640 1 -- 172.21.15.1:0/412886255 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd548011790 con 0x7fd5600c6fd0 2024-03-06T07:25:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd5577fe640 1 -- 172.21.15.1:0/412886255 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd548017070 con 0x7fd5600c6fd0 2024-03-06T07:25:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd5657bc640 1 -- 172.21.15.1:0/412886255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5601ea900 con 0x7fd5600c6fd0 2024-03-06T07:25:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd5577fe640 1 -- 172.21.15.1:0/412886255 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd548011bb0 con 0x7fd5600c6fd0 2024-03-06T07:25:53.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.515+0000 7fd5657bc640 1 -- 172.21.15.1:0/412886255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5601eaec0 con 0x7fd5600c6fd0 2024-03-06T07:25:53.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.517+0000 7fd5577fe640 1 -- 172.21.15.1:0/412886255 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd54801c460 con 0x7fd5600c6fd0 2024-03-06T07:25:53.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.517+0000 7fd5577fe640 1 --2- 172.21.15.1:0/412886255 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd54006b4d0 0x7fd54006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.517+0000 7fd5577fe640 1 -- 172.21.15.1:0/412886255 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd54808a0b0 con 0x7fd5600c6fd0 2024-03-06T07:25:53.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.518+0000 7fd55effd640 1 --2- 172.21.15.1:0/412886255 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd54006b4d0 0x7fd54006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.518+0000 7fd55effd640 1 --2- 172.21.15.1:0/412886255 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd54006b4d0 0x7fd54006d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd550035000 tx=0x7fd550008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.518+0000 7fd5683a1740 1 Processor -- start 2024-03-06T07:25:53.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.518+0000 7fd5683a1740 1 -- start start 2024-03-06T07:25:53.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.519+0000 7fd5683a1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9e9ff50 0x55ddd9e9da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.519+0000 7fd5683a1740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9ea0330 0x55ddd9e9df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.519+0000 7fd5683a1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9e9e4b0 0x55ddd9e9a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.519+0000 7fd5683a1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddd9e046b0 con 0x55ddd9e9ff50 2024-03-06T07:25:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.519+0000 7fd5683a1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddd9dada60 con 0x55ddd9e9e4b0 2024-03-06T07:25:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.519+0000 7fd5683a1740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ddd9e00750 con 0x55ddd9ea0330 2024-03-06T07:25:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.519+0000 7fd55effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9e9ff50 0x55ddd9e9da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.519+0000 7fd55effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9e9ff50 0x55ddd9e9da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36548/0 (socket says 172.21.15.1:36548) 2024-03-06T07:25:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.519+0000 7fd55effd640 1 -- 172.21.15.1:0/1048093927 learned_addr learned my addr 172.21.15.1:0/1048093927 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.519+0000 7fd55f7fe640 1 --2- 172.21.15.1:0/1048093927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9e9e4b0 0x55ddd9e9a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.519+0000 7fd557fff640 1 --2- 172.21.15.1:0/1048093927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9ea0330 0x55ddd9e9df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.519+0000 7fd55effd640 1 -- 172.21.15.1:0/1048093927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9ea0330 msgr2=0x55ddd9e9df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.519+0000 7fd55effd640 1 --2- 172.21.15.1:0/1048093927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9ea0330 0x55ddd9e9df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.519+0000 7fd55effd640 1 -- 172.21.15.1:0/1048093927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9e9e4b0 msgr2=0x55ddd9e9a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.519+0000 7fd55effd640 1 --2- 172.21.15.1:0/1048093927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9e9e4b0 0x55ddd9e9a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.519+0000 7fd55effd640 1 -- 172.21.15.1:0/1048093927 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ddd9dcbf30 con 0x55ddd9e9ff50 2024-03-06T07:25:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.519+0000 7fd55f7fe640 1 --2- 172.21.15.1:0/1048093927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9e9e4b0 0x55ddd9e9a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.520+0000 7fd55effd640 1 --2- 172.21.15.1:0/1048093927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9e9ff50 0x55ddd9e9da30 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fd550009890 tx=0x7fd550030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68576edb67021139 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.520+0000 7fd554ff9640 1 -- 172.21.15.1:0/1048093927 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550043070 con 0x55ddd9e9ff50 2024-03-06T07:25:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.520+0000 7fd554ff9640 1 -- 172.21.15.1:0/1048093927 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd550053480 con 0x55ddd9e9ff50 2024-03-06T07:25:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.520+0000 7fd554ff9640 1 -- 172.21.15.1:0/1048093927 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550038040 con 0x55ddd9e9ff50 2024-03-06T07:25:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.520+0000 7fd5683a1740 1 -- 172.21.15.1:0/1048093927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9e9ff50 msgr2=0x55ddd9e9da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.520+0000 7fd5683a1740 1 --2- 172.21.15.1:0/1048093927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9e9ff50 0x55ddd9e9da30 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fd550009890 tx=0x7fd550030000 comp rx=0 tx=0).stop 2024-03-06T07:25:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.520+0000 7fd5683a1740 1 -- 172.21.15.1:0/1048093927 shutdown_connections 2024-03-06T07:25:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.520+0000 7fd5683a1740 1 --2- 172.21.15.1:0/1048093927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9e9e4b0 0x55ddd9e9a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.520+0000 7fd5683a1740 1 --2- 172.21.15.1:0/1048093927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9ea0330 0x55ddd9e9df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.520+0000 7fd5683a1740 1 --2- 172.21.15.1:0/1048093927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9e9ff50 0x55ddd9e9da30 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.520+0000 7fd5683a1740 1 -- 172.21.15.1:0/1048093927 >> 172.21.15.1:0/1048093927 conn(0x55ddd9e9c7d0 msgr2=0x55ddd9ec13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.521+0000 7fd5683a1740 1 -- 172.21.15.1:0/1048093927 shutdown_connections 2024-03-06T07:25:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.521+0000 7fd5683a1740 1 -- 172.21.15.1:0/1048093927 wait complete. 2024-03-06T07:25:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.521+0000 7fd5683a1740 1 -- start start 2024-03-06T07:25:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.521+0000 7fd5683a1740 1 Processor -- start 2024-03-06T07:25:53.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.522+0000 7fd5683a1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9e9ff50 0x55ddd9ec3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.522+0000 7fd5683a1740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9ea0330 0x55ddd9ec65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.522+0000 7fd5683a1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9ec5a30 0x55ddd9eefe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.522+0000 7fd5683a1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddd9ec2230 con 0x55ddd9e9ff50 2024-03-06T07:25:53.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.522+0000 7fd5683a1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddd9dc0ea0 con 0x55ddd9ec5a30 2024-03-06T07:25:53.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.522+0000 7fd5683a1740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ddd9ec1c70 con 0x55ddd9ea0330 2024-03-06T07:25:53.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.522+0000 7fd55effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9e9ff50 0x55ddd9ec3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.522+0000 7fd5683a1740 10 client.0 _ll_register_callbacks cb 0x55ddd9dc0870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:53.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.522+0000 7fd55effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9e9ff50 0x55ddd9ec3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36552/0 (socket says 172.21.15.1:36552) 2024-03-06T07:25:53.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.522+0000 7fd55effd640 1 -- 172.21.15.1:0/1767417965 learned_addr learned my addr 172.21.15.1:0/1767417965 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:53.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.522+0000 7fd557fff640 1 --2- 172.21.15.1:0/1767417965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9ea0330 0x55ddd9ec65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.522+0000 7fd55f7fe640 1 --2- 172.21.15.1:0/1767417965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9ec5a30 0x55ddd9eefe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.522+0000 7fd55effd640 1 -- 172.21.15.1:0/1767417965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9ea0330 msgr2=0x55ddd9ec65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.522+0000 7fd55effd640 1 --2- 172.21.15.1:0/1767417965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9ea0330 0x55ddd9ec65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.522+0000 7fd55effd640 1 -- 172.21.15.1:0/1767417965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9ec5a30 msgr2=0x55ddd9eefe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.522+0000 7fd55effd640 1 --2- 172.21.15.1:0/1767417965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9ec5a30 0x55ddd9eefe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.522+0000 7fd55effd640 1 -- 172.21.15.1:0/1767417965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ddd9dc1270 con 0x55ddd9e9ff50 2024-03-06T07:25:53.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.523+0000 7fd55effd640 1 --2- 172.21.15.1:0/1767417965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9e9ff50 0x55ddd9ec3be0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fd550036040 tx=0x7fd550004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c4601c95f5203f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.523+0000 7fd53dffb640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:25:53.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.523+0000 7fd53dffb640 1 -- 172.21.15.1:0/1767417965 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550043040 con 0x55ddd9e9ff50 2024-03-06T07:25:53.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.523+0000 7fd53dffb640 1 -- 172.21.15.1:0/1767417965 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd550053d70 con 0x55ddd9e9ff50 2024-03-06T07:25:53.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.523+0000 7fd5683a1740 1 -- 172.21.15.1:0/1767417965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd550053480 con 0x55ddd9e9ff50 2024-03-06T07:25:53.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.523+0000 7fd53dffb640 1 -- 172.21.15.1:0/1767417965 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=126) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd550038040 con 0x55ddd9e9ff50 2024-03-06T07:25:53.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.523+0000 7fd5683a1740 10 client.8565 fetch_fsmap learned FSMap version 126 2024-03-06T07:25:53.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.523+0000 7fd5683a1740 1 -- 172.21.15.1:0/1767417965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=126,monmap=2+}) v3 -- 0x55ddd9b192d0 con 0x55ddd9e9ff50 2024-03-06T07:25:53.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.523+0000 7fd53dffb640 1 -- 172.21.15.1:0/1767417965 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd550053480 con 0x55ddd9e9ff50 2024-03-06T07:25:53.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.523+0000 7fd53dffb640 1 -- 172.21.15.1:0/1767417965 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 126) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd550067080 con 0x55ddd9e9ff50 2024-03-06T07:25:53.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.523+0000 7fd5683a1740 10 client.8565 fetch_fsmap finished waiting for FSMap version 126 2024-03-06T07:25:53.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.523+0000 7fd5683a1740 10 client.8565 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:25:53.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.523+0000 7fd5683a1740 20 client.8565 populate_metadata read hostname 'smithi001' 2024-03-06T07:25:53.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.523+0000 7fd5683a1740 1 --2- 172.21.15.1:0/1767417965 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x55ddd9ef04e0 0x55ddd9ef08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.523+0000 7fd5683a1740 4 client.8565 mds_command: new command op to 6686 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:53.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.523+0000 7fd5683a1740 1 -- 172.21.15.1:0/1767417965 --> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ddd9e11bf0 con 0x55ddd9ef04e0 2024-03-06T07:25:53.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.523+0000 7fd557fff640 1 --2- 172.21.15.1:0/1767417965 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x55ddd9ef04e0 0x55ddd9ef08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.524+0000 7fd53dffb640 10 client.8565 ms_handle_connect on v2:172.21.15.1:6833/2848968195 2024-03-06T07:25:53.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.524+0000 7fd557fff640 1 --2- 172.21.15.1:0/1767417965 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x55ddd9ef04e0 0x55ddd9ef08e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7874088fb1eded6d server_cookie=378437d9b4cd6f43 in_seq=0 out_seq=0 2024-03-06T07:25:53.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.524+0000 7fd53dffb640 1 -- 172.21.15.1:0/1767417965 <== mds.0 v2:172.21.15.1:6833/2848968195 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ddd9e11bf0 con 0x55ddd9ef04e0 2024-03-06T07:25:53.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.524+0000 7fd53dffb640 10 client.8565 handle_command_reply: tid=0 2024-03-06T07:25:53.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.524+0000 7fd5683a1740 1 client.8565 shutdown 2024-03-06T07:25:53.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.524+0000 7fd5683a1740 1 -- 172.21.15.1:0/1767417965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9e9ff50 msgr2=0x55ddd9ec3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.524+0000 7fd5683a1740 1 --2- 172.21.15.1:0/1767417965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9e9ff50 0x55ddd9ec3be0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fd550036040 tx=0x7fd550004730 comp rx=0 tx=0).stop 2024-03-06T07:25:53.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.524+0000 7fd5683a1740 1 -- 172.21.15.1:0/1767417965 shutdown_connections 2024-03-06T07:25:53.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.524+0000 7fd5683a1740 1 --2- 172.21.15.1:0/1767417965 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x55ddd9ef04e0 0x55ddd9ef08e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.524+0000 7fd53dffb640 0 client.8565 ms_handle_reset on v2:172.21.15.1:6833/2848968195 2024-03-06T07:25:53.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.524+0000 7fd5683a1740 1 --2- 172.21.15.1:0/1767417965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9ec5a30 0x55ddd9eefe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.524+0000 7fd5683a1740 1 --2- 172.21.15.1:0/1767417965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9ea0330 0x55ddd9ec65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.524+0000 7fd5683a1740 1 --2- 172.21.15.1:0/1767417965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9e9ff50 0x55ddd9ec3be0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.524+0000 7fd5683a1740 1 -- 172.21.15.1:0/1767417965 >> 172.21.15.1:0/1767417965 conn(0x55ddd9e9e4b0 msgr2=0x55ddd9edcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.525+0000 7fd5683a1740 1 -- 172.21.15.1:0/1767417965 shutdown_connections 2024-03-06T07:25:53.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.525+0000 7fd5683a1740 1 -- 172.21.15.1:0/1767417965 wait complete. 2024-03-06T07:25:53.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.525+0000 7fd5683a1740 20 client.8565 trim_cache size 0 max 16384 2024-03-06T07:25:53.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.541+0000 7fd5683a1740 1 Processor -- start 2024-03-06T07:25:53.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.541+0000 7fd5683a1740 1 -- start start 2024-03-06T07:25:53.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.542+0000 7fd5683a1740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9e9ff50 0x55ddd9e9b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.542+0000 7fd5683a1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9ea0330 0x55ddd9ee8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.542+0000 7fd5683a1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9ec5a30 0x55ddd9ee8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.542+0000 7fd5683a1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5601f6ed0 con 0x55ddd9ea0330 2024-03-06T07:25:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.542+0000 7fd5683a1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5601f62d0 con 0x55ddd9ec5a30 2024-03-06T07:25:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.542+0000 7fd5683a1740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5601f68d0 con 0x55ddd9e9ff50 2024-03-06T07:25:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.542+0000 7fd557fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9ea0330 0x55ddd9ee8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.542+0000 7fd557fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9ea0330 0x55ddd9ee8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36556/0 (socket says 172.21.15.1:36556) 2024-03-06T07:25:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.542+0000 7fd557fff640 1 -- 172.21.15.1:0/2976206864 learned_addr learned my addr 172.21.15.1:0/2976206864 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.542+0000 7fd557fff640 1 -- 172.21.15.1:0/2976206864 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9e9ff50 msgr2=0x55ddd9e9b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.542+0000 7fd557fff640 1 --2- 172.21.15.1:0/2976206864 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9e9ff50 0x55ddd9e9b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.542+0000 7fd557fff640 1 -- 172.21.15.1:0/2976206864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9ec5a30 msgr2=0x55ddd9ee8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.542+0000 7fd55f7fe640 1 --2- 172.21.15.1:0/2976206864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9ec5a30 0x55ddd9ee8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.542+0000 7fd557fff640 1 --2- 172.21.15.1:0/2976206864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9ec5a30 0x55ddd9ee8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.542+0000 7fd557fff640 1 -- 172.21.15.1:0/2976206864 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ddd9ec6940 con 0x55ddd9ea0330 2024-03-06T07:25:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.543+0000 7fd557fff640 1 --2- 172.21.15.1:0/2976206864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9ea0330 0x55ddd9ee8870 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fd548058680 tx=0x7fd5480882c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc2489ec5563164 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.543+0000 7fd53ffff640 1 -- 172.21.15.1:0/2976206864 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd548085030 con 0x55ddd9ea0330 2024-03-06T07:25:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.543+0000 7fd53ffff640 1 -- 172.21.15.1:0/2976206864 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd548056790 con 0x55ddd9ea0330 2024-03-06T07:25:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.543+0000 7fd53ffff640 1 -- 172.21.15.1:0/2976206864 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5480569f0 con 0x55ddd9ea0330 2024-03-06T07:25:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.543+0000 7fd5683a1740 1 -- 172.21.15.1:0/2976206864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9ea0330 msgr2=0x55ddd9ee8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.543+0000 7fd5683a1740 1 --2- 172.21.15.1:0/2976206864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9ea0330 0x55ddd9ee8870 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fd548058680 tx=0x7fd5480882c0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.543+0000 7fd5683a1740 1 -- 172.21.15.1:0/2976206864 shutdown_connections 2024-03-06T07:25:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.543+0000 7fd5683a1740 1 --2- 172.21.15.1:0/2976206864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9ec5a30 0x55ddd9ee8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.543+0000 7fd5683a1740 1 --2- 172.21.15.1:0/2976206864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9ea0330 0x55ddd9ee8870 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.543+0000 7fd5683a1740 1 --2- 172.21.15.1:0/2976206864 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9e9ff50 0x55ddd9e9b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.543+0000 7fd5683a1740 1 -- 172.21.15.1:0/2976206864 >> 172.21.15.1:0/2976206864 conn(0x55ddd9e9e4b0 msgr2=0x55ddd9ea0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.543+0000 7fd5683a1740 1 -- 172.21.15.1:0/2976206864 shutdown_connections 2024-03-06T07:25:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.543+0000 7fd5683a1740 1 -- 172.21.15.1:0/2976206864 wait complete. 2024-03-06T07:25:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.544+0000 7fd5683a1740 1 -- start start 2024-03-06T07:25:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.544+0000 7fd5683a1740 1 Processor -- start 2024-03-06T07:25:53.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd5683a1740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9e9ff50 0x55ddd9ee8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd5683a1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9ea0330 0x55ddd9f4c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd5683a1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9ec5a30 0x55ddd9f45c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd5683a1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddd9ec2bc0 con 0x55ddd9ec5a30 2024-03-06T07:25:53.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd5683a1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddd9ef0360 con 0x55ddd9ea0330 2024-03-06T07:25:53.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd5683a1740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ddd9ec14a0 con 0x55ddd9e9ff50 2024-03-06T07:25:53.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd5683a1740 10 client.0 _ll_register_callbacks cb 0x7fd550053630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:53.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd55f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9ec5a30 0x55ddd9f45c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd55f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9ec5a30 0x55ddd9f45c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36568/0 (socket says 172.21.15.1:36568) 2024-03-06T07:25:53.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd55f7fe640 1 -- 172.21.15.1:0/85847859 learned_addr learned my addr 172.21.15.1:0/85847859 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:53.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd557fff640 1 --2- 172.21.15.1:0/85847859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9ea0330 0x55ddd9f4c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd55effd640 1 --2- 172.21.15.1:0/85847859 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9e9ff50 0x55ddd9ee8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd55f7fe640 1 -- 172.21.15.1:0/85847859 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9e9ff50 msgr2=0x55ddd9ee8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd55f7fe640 1 --2- 172.21.15.1:0/85847859 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9e9ff50 0x55ddd9ee8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd55f7fe640 1 -- 172.21.15.1:0/85847859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9ea0330 msgr2=0x55ddd9f4c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd55f7fe640 1 --2- 172.21.15.1:0/85847859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9ea0330 0x55ddd9f4c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd55f7fe640 1 -- 172.21.15.1:0/85847859 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ddd9be36b0 con 0x55ddd9ec5a30 2024-03-06T07:25:53.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd55f7fe640 1 --2- 172.21.15.1:0/85847859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9ec5a30 0x55ddd9f45c40 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fd5580012e0 tx=0x7fd5580074d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6764b41e6b7e324 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd53dffb640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:25:53.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd53dffb640 1 -- 172.21.15.1:0/85847859 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55801b020 con 0x55ddd9ec5a30 2024-03-06T07:25:53.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd53dffb640 1 -- 172.21.15.1:0/85847859 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd558017670 con 0x55ddd9ec5a30 2024-03-06T07:25:53.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd5683a1740 1 -- 172.21.15.1:0/85847859 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd548056790 con 0x55ddd9ec5a30 2024-03-06T07:25:53.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd53dffb640 1 -- 172.21.15.1:0/85847859 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=126) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd558025050 con 0x55ddd9ec5a30 2024-03-06T07:25:53.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd5683a1740 10 client.8571 fetch_fsmap learned FSMap version 126 2024-03-06T07:25:53.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.545+0000 7fd5683a1740 1 -- 172.21.15.1:0/85847859 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=126,monmap=2+}) v3 -- 0x55ddd9ec7900 con 0x55ddd9ec5a30 2024-03-06T07:25:53.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.546+0000 7fd53dffb640 1 -- 172.21.15.1:0/85847859 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd548056790 con 0x55ddd9ec5a30 2024-03-06T07:25:53.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.546+0000 7fd53dffb640 1 -- 172.21.15.1:0/85847859 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 126) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd558028020 con 0x55ddd9ec5a30 2024-03-06T07:25:53.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.546+0000 7fd5683a1740 10 client.8571 fetch_fsmap finished waiting for FSMap version 126 2024-03-06T07:25:53.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.546+0000 7fd5683a1740 10 client.8571 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:25:53.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.546+0000 7fd5683a1740 20 client.8571 populate_metadata read hostname 'smithi001' 2024-03-06T07:25:53.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.546+0000 7fd5683a1740 1 --2- 172.21.15.1:0/85847859 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x55ddd9ebf9b0 0x55ddd9ebfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.546+0000 7fd5683a1740 4 client.8571 mds_command: new command op to 6686 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.546+0000 7fd5683a1740 1 -- 172.21.15.1:0/85847859 --> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ddd9e0bf40 con 0x55ddd9ebf9b0 2024-03-06T07:25:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.546+0000 7fd55effd640 1 --2- 172.21.15.1:0/85847859 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x55ddd9ebf9b0 0x55ddd9ebfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.546+0000 7fd53dffb640 10 client.8571 ms_handle_connect on v2:172.21.15.1:6833/2848968195 2024-03-06T07:25:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.546+0000 7fd55effd640 1 --2- 172.21.15.1:0/85847859 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x55ddd9ebf9b0 0x55ddd9ebfd90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3d7c115642e9661 server_cookie=9c1a508216d078e4 in_seq=0 out_seq=0 2024-03-06T07:25:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.546+0000 7fd53dffb640 1 -- 172.21.15.1:0/85847859 <== mds.0 v2:172.21.15.1:6833/2848968195 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x55ddd9e0bf40 con 0x55ddd9ebf9b0 2024-03-06T07:25:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.546+0000 7fd53dffb640 10 client.8571 handle_command_reply: tid=0 2024-03-06T07:25:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.546+0000 7fd5683a1740 1 client.8571 shutdown 2024-03-06T07:25:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.546+0000 7fd5683a1740 1 -- 172.21.15.1:0/85847859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9ec5a30 msgr2=0x55ddd9f45c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.546+0000 7fd5683a1740 1 --2- 172.21.15.1:0/85847859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9ec5a30 0x55ddd9f45c40 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fd5580012e0 tx=0x7fd5580074d0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.546+0000 7fd5683a1740 1 -- 172.21.15.1:0/85847859 shutdown_connections 2024-03-06T07:25:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5683a1740 1 --2- 172.21.15.1:0/85847859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddd9ec5a30 0x55ddd9f45c40 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5683a1740 1 --2- 172.21.15.1:0/85847859 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x55ddd9ebf9b0 0x55ddd9ebfd90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5683a1740 1 --2- 172.21.15.1:0/85847859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ddd9ea0330 0x55ddd9f4c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5683a1740 1 --2- 172.21.15.1:0/85847859 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ddd9e9ff50 0x55ddd9ee8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd53dffb640 0 client.8571 ms_handle_reset on v2:172.21.15.1:6833/2848968195 2024-03-06T07:25:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5683a1740 1 -- 172.21.15.1:0/85847859 >> 172.21.15.1:0/85847859 conn(0x55ddd9e9e4b0 msgr2=0x55ddd9ec3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5683a1740 1 -- 172.21.15.1:0/85847859 shutdown_connections 2024-03-06T07:25:53.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5683a1740 1 -- 172.21.15.1:0/85847859 wait complete. 2024-03-06T07:25:53.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5683a1740 20 client.8571 trim_cache size 0 max 16384 2024-03-06T07:25:53.554 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:25:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:25:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:25:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 6686, 2024-03-06T07:25:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:25:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:25:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:25:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 11.758017848, 2024-03-06T07:25:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 126, 2024-03-06T07:25:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 136, 2024-03-06T07:25:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 136, 2024-03-06T07:25:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 28.19772369 2024-03-06T07:25:53.555 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:25:53.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5657bc640 1 -- 172.21.15.1:0/412886255 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd54006b4d0 msgr2=0x7fd54006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5657bc640 1 --2- 172.21.15.1:0/412886255 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd54006b4d0 0x7fd54006d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd550035000 tx=0x7fd550008030 comp rx=0 tx=0).stop 2024-03-06T07:25:53.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5657bc640 1 -- 172.21.15.1:0/412886255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5600c6fd0 msgr2=0x7fd560142000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5657bc640 1 --2- 172.21.15.1:0/412886255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5600c6fd0 0x7fd560142000 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fd54800c460 tx=0x7fd54800c920 comp rx=0 tx=0).stop 2024-03-06T07:25:53.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5657bc640 1 -- 172.21.15.1:0/412886255 shutdown_connections 2024-03-06T07:25:53.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5657bc640 1 --2- 172.21.15.1:0/412886255 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd54006b4d0 0x7fd54006d990 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5657bc640 1 --2- 172.21.15.1:0/412886255 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd560152730 0x7fd560142540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5657bc640 1 --2- 172.21.15.1:0/412886255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5600c6fd0 0x7fd560142000 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5657bc640 1 --2- 172.21.15.1:0/412886255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5600c1aa0 0x7fd5600c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5657bc640 1 -- 172.21.15.1:0/412886255 >> 172.21.15.1:0/412886255 conn(0x7fd560066d00 msgr2=0x7fd5600c7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5657bc640 1 -- 172.21.15.1:0/412886255 shutdown_connections 2024-03-06T07:25:53.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.547+0000 7fd5657bc640 1 -- 172.21.15.1:0/412886255 wait complete. 2024-03-06T07:25:53.560 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-03-06T07:25:53.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.661+0000 7fdb34c71640 1 Processor -- start 2024-03-06T07:25:53.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.661+0000 7fdb34c71640 1 -- start start 2024-03-06T07:25:53.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb34c71640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdb3006f6d0 0x7fdb300a9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb34c71640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb30152b00 con 0x7fdb300c14f0 2024-03-06T07:25:53.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb34c71640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3006fdb0 con 0x7fdb300a9760 2024-03-06T07:25:53.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb34c71640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb301551f0 con 0x7fdb3006f6d0 2024-03-06T07:25:53.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2dd9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdb300c14f0 0x7fdb3015da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55026/0 (socket says 172.21.15.1:55026) 2024-03-06T07:25:53.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2dd9b640 1 -- 172.21.15.1:0/1845847156 learned_addr learned my addr 172.21.15.1:0/1845847156 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:25:53.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2ed9d640 1 --2- 172.21.15.1:0/1845847156 >> v2:172.21.15.98:3300/0 conn(0x7fdb3006f6d0 0x7fdb300a9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2d59a640 1 -- 172.21.15.1:0/1845847156 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420477522 0 0) 0x7fdb30152b00 con 0x7fdb300c14f0 2024-03-06T07:25:53.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2d59a640 1 -- 172.21.15.1:0/1845847156 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb18003520 con 0x7fdb300c14f0 2024-03-06T07:25:53.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2d59a640 1 -- 172.21.15.1:0/1845847156 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052173959 0 0) 0x7fdb3006fdb0 con 0x7fdb300a9760 2024-03-06T07:25:53.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2d59a640 1 -- 172.21.15.1:0/1845847156 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb30152b00 con 0x7fdb300a9760 2024-03-06T07:25:53.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2d59a640 1 -- 172.21.15.1:0/1845847156 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 527893135 0 0) 0x7fdb18003520 con 0x7fdb300c14f0 2024-03-06T07:25:53.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2d59a640 1 -- 172.21.15.1:0/1845847156 >> v2:172.21.15.98:3300/0 conn(0x7fdb3006f6d0 msgr2=0x7fdb300a9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2d59a640 1 --2- 172.21.15.1:0/1845847156 >> v2:172.21.15.98:3300/0 conn(0x7fdb3006f6d0 0x7fdb300a9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2d59a640 1 -- 172.21.15.1:0/1845847156 >> v1:172.21.15.98:6789/0 conn(0x7fdb300a9760 legacy=0x7fdb30155680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2d59a640 1 -- 172.21.15.1:0/1845847156 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb300c0720 con 0x7fdb300c14f0 2024-03-06T07:25:53.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2d59a640 1 -- 172.21.15.1:0/1845847156 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fdb24001670 con 0x7fdb300c14f0 2024-03-06T07:25:53.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2ed9d640 1 --2- 172.21.15.1:0/1845847156 >> v2:172.21.15.98:3300/0 conn(0x7fdb3006f6d0 0x7fdb300a9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:53.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2d59a640 1 -- 172.21.15.1:0/1845847156 >> v1:172.21.15.1:6789/0 conn(0x7fdb300c14f0 legacy=0x7fdb3015da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2d59a640 1 --2- 172.21.15.1:0/1845847156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb18003b40 0x7fdb18003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2d59a640 1 --2- 172.21.15.1:0/1845847156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb18004800 0x7fdb18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2d59a640 1 --2- 172.21.15.1:0/1845847156 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb18005230 0x7fdb1800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2d59a640 1 -- 172.21.15.1:0/1845847156 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb301551f0 con 0x7fdb18003b40 2024-03-06T07:25:53.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2d59a640 1 -- 172.21.15.1:0/1845847156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1800c190 con 0x7fdb18004800 2024-03-06T07:25:53.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.663+0000 7fdb2d59a640 1 -- 172.21.15.1:0/1845847156 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1800c390 con 0x7fdb18005230 2024-03-06T07:25:53.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.664+0000 7fdb2e59c640 1 --2- 172.21.15.1:0/1845847156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb18003b40 0x7fdb18003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.664+0000 7fdb2dd9b640 1 --2- 172.21.15.1:0/1845847156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb18004800 0x7fdb18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.664+0000 7fdb2ed9d640 1 --2- 172.21.15.1:0/1845847156 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb18005230 0x7fdb1800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.664+0000 7fdb2e59c640 1 -- 172.21.15.1:0/1845847156 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb18005230 msgr2=0x7fdb1800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.664+0000 7fdb2e59c640 1 --2- 172.21.15.1:0/1845847156 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb18005230 0x7fdb1800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.664+0000 7fdb2e59c640 1 -- 172.21.15.1:0/1845847156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb18004800 msgr2=0x7fdb18004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.664+0000 7fdb2e59c640 1 --2- 172.21.15.1:0/1845847156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb18004800 0x7fdb18004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.664+0000 7fdb2e59c640 1 -- 172.21.15.1:0/1845847156 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb18003520 con 0x7fdb18003b40 2024-03-06T07:25:53.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.664+0000 7fdb2ed9d640 1 --2- 172.21.15.1:0/1845847156 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb18005230 0x7fdb1800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:53.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.664+0000 7fdb2ed9d640 1 -- 172.21.15.1:0/1845847156 reap_dead start 2024-03-06T07:25:53.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.665+0000 7fdb2e59c640 1 --2- 172.21.15.1:0/1845847156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb18003b40 0x7fdb18003f40 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fdb200018d0 tx=0x7fdb18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e912b327f25a55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.665+0000 7fdb2d59a640 1 -- 172.21.15.1:0/1845847156 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb200014f0 con 0x7fdb18003b40 2024-03-06T07:25:53.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.665+0000 7fdb2d59a640 1 -- 172.21.15.1:0/1845847156 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2002f050 con 0x7fdb18003b40 2024-03-06T07:25:53.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.665+0000 7fdb2d59a640 1 -- 172.21.15.1:0/1845847156 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb200035f0 con 0x7fdb18003b40 2024-03-06T07:25:53.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.665+0000 7fdb34c71640 1 -- 172.21.15.1:0/1845847156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb18003b40 msgr2=0x7fdb18003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.665+0000 7fdb34c71640 1 --2- 172.21.15.1:0/1845847156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb18003b40 0x7fdb18003f40 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fdb200018d0 tx=0x7fdb18005830 comp rx=0 tx=0).stop 2024-03-06T07:25:53.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.665+0000 7fdb34c71640 1 -- 172.21.15.1:0/1845847156 shutdown_connections 2024-03-06T07:25:53.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.665+0000 7fdb34c71640 1 --2- 172.21.15.1:0/1845847156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb18003b40 0x7fdb18003f40 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.665+0000 7fdb34c71640 1 -- 172.21.15.1:0/1845847156 >> 172.21.15.1:0/1845847156 conn(0x7fdb300c3cc0 msgr2=0x7fdb300c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.665+0000 7fdb34c71640 1 -- 172.21.15.1:0/1845847156 shutdown_connections 2024-03-06T07:25:53.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.665+0000 7fdb34c71640 1 -- 172.21.15.1:0/1845847156 wait complete. 2024-03-06T07:25:53.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.667+0000 7fdb34c71640 1 Processor -- start 2024-03-06T07:25:53.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.667+0000 7fdb34c71640 1 -- start start 2024-03-06T07:25:53.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.667+0000 7fdb34c71640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb18003b40 0x7fdb300c5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.667+0000 7fdb34c71640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb300c6740 0x7fdb301eb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.667+0000 7fdb34c71640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb30150b20 0x7fdb300aa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.667+0000 7fdb34c71640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb300c29e0 con 0x7fdb18003b40 2024-03-06T07:25:53.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.667+0000 7fdb34c71640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb300a9520 con 0x7fdb30150b20 2024-03-06T07:25:53.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.667+0000 7fdb34c71640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb300c0ce0 con 0x7fdb300c6740 2024-03-06T07:25:53.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.667+0000 7fdb2e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb18003b40 0x7fdb300c5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.667+0000 7fdb2e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb18003b40 0x7fdb300c5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36586/0 (socket says 172.21.15.1:36586) 2024-03-06T07:25:53.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.667+0000 7fdb2e59c640 1 -- 172.21.15.1:0/3742910472 learned_addr learned my addr 172.21.15.1:0/3742910472 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:53.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.668+0000 7fdb2ed9d640 1 --2- 172.21.15.1:0/3742910472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb30150b20 0x7fdb300aa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.668+0000 7fdb2dd9b640 1 --2- 172.21.15.1:0/3742910472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb300c6740 0x7fdb301eb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.668+0000 7fdb2e59c640 1 -- 172.21.15.1:0/3742910472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb300c6740 msgr2=0x7fdb301eb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.668+0000 7fdb2e59c640 1 --2- 172.21.15.1:0/3742910472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb300c6740 0x7fdb301eb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.668+0000 7fdb2e59c640 1 -- 172.21.15.1:0/3742910472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb30150b20 msgr2=0x7fdb300aa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.668+0000 7fdb2e59c640 1 --2- 172.21.15.1:0/3742910472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb30150b20 0x7fdb300aa2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.668+0000 7fdb2e59c640 1 -- 172.21.15.1:0/3742910472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb301eb780 con 0x7fdb18003b40 2024-03-06T07:25:53.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.668+0000 7fdb2e59c640 1 --2- 172.21.15.1:0/3742910472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb18003b40 0x7fdb300c5930 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fdb20035040 tx=0x7fdb200012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.668+0000 7fdb1effd640 1 -- 172.21.15.1:0/3742910472 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb20042070 con 0x7fdb18003b40 2024-03-06T07:25:53.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.668+0000 7fdb1effd640 1 -- 172.21.15.1:0/3742910472 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb20007040 con 0x7fdb18003b40 2024-03-06T07:25:53.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.668+0000 7fdb34c71640 1 -- 172.21.15.1:0/3742910472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb300c6c40 con 0x7fdb18003b40 2024-03-06T07:25:53.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.668+0000 7fdb1effd640 1 -- 172.21.15.1:0/3742910472 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb200014f0 con 0x7fdb18003b40 2024-03-06T07:25:53.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.668+0000 7fdb34c71640 1 -- 172.21.15.1:0/3742910472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb300c5130 con 0x7fdb18003b40 2024-03-06T07:25:53.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.669+0000 7fdb34c71640 1 -- 172.21.15.1:0/3742910472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb300686e0 con 0x7fdb18003b40 2024-03-06T07:25:53.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.670+0000 7fdb1effd640 1 -- 172.21.15.1:0/3742910472 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb2003c040 con 0x7fdb18003b40 2024-03-06T07:25:53.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.670+0000 7fdb1effd640 1 --2- 172.21.15.1:0/3742910472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb1006b4d0 0x7fdb1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.670+0000 7fdb1effd640 1 -- 172.21.15.1:0/3742910472 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fdb200abb30 con 0x7fdb18003b40 2024-03-06T07:25:53.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.670+0000 7fdb2dd9b640 1 --2- 172.21.15.1:0/3742910472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb1006b4d0 0x7fdb1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.671+0000 7fdb2dd9b640 1 --2- 172.21.15.1:0/3742910472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb1006b4d0 0x7fdb1006d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fdb240011e0 tx=0x7fdb24009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.674+0000 7fdb1effd640 1 -- 172.21.15.1:0/3742910472 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb301eb780 con 0x7fdb18003b40 2024-03-06T07:25:53.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.854+0000 7fdb34c71640 1 -- 172.21.15.1:0/3742910472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7fdb300c12d0 con 0x7fdb18003b40 2024-03-06T07:25:53.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.857+0000 7fdb1effd640 1 -- 172.21.15.1:0/3742910472 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v48) v1 ==== 96+0+0 (secure 0 0 0) 0x7fdb2007a860 con 0x7fdb18003b40 2024-03-06T07:25:53.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.859+0000 7fdb34c71640 1 -- 172.21.15.1:0/3742910472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb1006b4d0 msgr2=0x7fdb1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.859+0000 7fdb34c71640 1 --2- 172.21.15.1:0/3742910472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb1006b4d0 0x7fdb1006d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fdb240011e0 tx=0x7fdb24009040 comp rx=0 tx=0).stop 2024-03-06T07:25:53.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.859+0000 7fdb34c71640 1 -- 172.21.15.1:0/3742910472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb18003b40 msgr2=0x7fdb300c5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.859+0000 7fdb34c71640 1 --2- 172.21.15.1:0/3742910472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb18003b40 0x7fdb300c5930 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fdb20035040 tx=0x7fdb200012d0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.859+0000 7fdb34c71640 1 -- 172.21.15.1:0/3742910472 shutdown_connections 2024-03-06T07:25:53.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.859+0000 7fdb34c71640 1 --2- 172.21.15.1:0/3742910472 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb1006b4d0 0x7fdb1006d990 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.859+0000 7fdb34c71640 1 --2- 172.21.15.1:0/3742910472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb30150b20 0x7fdb300aa2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.859+0000 7fdb34c71640 1 --2- 172.21.15.1:0/3742910472 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb300c6740 0x7fdb301eb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.859+0000 7fdb34c71640 1 --2- 172.21.15.1:0/3742910472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb18003b40 0x7fdb300c5930 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.859+0000 7fdb34c71640 1 -- 172.21.15.1:0/3742910472 >> 172.21.15.1:0/3742910472 conn(0x7fdb300c3cc0 msgr2=0x7fdb30153120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.859+0000 7fdb34c71640 1 -- 172.21.15.1:0/3742910472 shutdown_connections 2024-03-06T07:25:53.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.859+0000 7fdb34c71640 1 -- 172.21.15.1:0/3742910472 wait complete. 2024-03-06T07:25:53.871 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:53.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.973+0000 7f1bcd9d7640 1 Processor -- start 2024-03-06T07:25:53.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.973+0000 7f1bcd9d7640 1 -- start start 2024-03-06T07:25:53.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.974+0000 7f1bcd9d7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1bc8152650 0x7f1bc8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.974+0000 7f1bcd9d7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bc80c6b10 0x7f1bc8158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.974+0000 7f1bcd9d7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1bc80bf9d0 0x7f1bc80c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.974+0000 7f1bcd9d7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bc8155800 con 0x7f1bc80bf9d0 2024-03-06T07:25:53.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.974+0000 7f1bcd9d7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bc80c2fc0 con 0x7f1bc8152650 2024-03-06T07:25:53.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.974+0000 7f1bcd9d7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc80c6690 con 0x7f1bc80c6b10 2024-03-06T07:25:53.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.974+0000 7f1bc77fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1bc80bf9d0 0x7f1bc80c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.974+0000 7f1bc77fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1bc80bf9d0 0x7f1bc80c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36588/0 (socket says 172.21.15.1:36588) 2024-03-06T07:25:53.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.974+0000 7f1bc77fe640 1 -- 172.21.15.1:0/68660435 learned_addr learned my addr 172.21.15.1:0/68660435 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:53.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.974+0000 7f1bc67fc640 1 --2- 172.21.15.1:0/68660435 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bc80c6b10 0x7f1bc8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.974+0000 7f1bc77fe640 1 -- 172.21.15.1:0/68660435 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bc80c6b10 msgr2=0x7f1bc8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.974+0000 7f1bc6ffd640 1 --2- 172.21.15.1:0/68660435 >> v2:172.21.15.98:3300/0 conn(0x7f1bc8152650 0x7f1bc8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.974+0000 7f1bc77fe640 1 --2- 172.21.15.1:0/68660435 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bc80c6b10 0x7f1bc8158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.974+0000 7f1bc77fe640 1 -- 172.21.15.1:0/68660435 >> v2:172.21.15.98:3300/0 conn(0x7f1bc8152650 msgr2=0x7f1bc8155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.974+0000 7f1bc77fe640 1 --2- 172.21.15.1:0/68660435 >> v2:172.21.15.98:3300/0 conn(0x7f1bc8152650 0x7f1bc8155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.974+0000 7f1bc77fe640 1 -- 172.21.15.1:0/68660435 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc80c7280 con 0x7f1bc80bf9d0 2024-03-06T07:25:53.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.975+0000 7f1bc77fe640 1 --2- 172.21.15.1:0/68660435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bc80bf9d0 0x7f1bc80c1460 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f1bc80cac90 tx=0x7f1bc002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20da253e201e22eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.975+0000 7f1bc5ffb640 1 -- 172.21.15.1:0/68660435 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc002c9e0 con 0x7f1bc80bf9d0 2024-03-06T07:25:53.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.975+0000 7f1bc5ffb640 1 -- 172.21.15.1:0/68660435 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc002cb80 con 0x7f1bc80bf9d0 2024-03-06T07:25:53.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.975+0000 7f1bc5ffb640 1 -- 172.21.15.1:0/68660435 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc00343f0 con 0x7f1bc80bf9d0 2024-03-06T07:25:53.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.975+0000 7f1bcd9d7640 1 -- 172.21.15.1:0/68660435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bc80bf9d0 msgr2=0x7f1bc80c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.975+0000 7f1bcd9d7640 1 --2- 172.21.15.1:0/68660435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bc80bf9d0 0x7f1bc80c1460 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f1bc80cac90 tx=0x7f1bc002be40 comp rx=0 tx=0).stop 2024-03-06T07:25:53.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.975+0000 7f1bcd9d7640 1 -- 172.21.15.1:0/68660435 shutdown_connections 2024-03-06T07:25:53.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.975+0000 7f1bcd9d7640 1 --2- 172.21.15.1:0/68660435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bc80bf9d0 0x7f1bc80c1460 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.975+0000 7f1bcd9d7640 1 --2- 172.21.15.1:0/68660435 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bc80c6b10 0x7f1bc8158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.975+0000 7f1bcd9d7640 1 --2- 172.21.15.1:0/68660435 >> v2:172.21.15.98:3300/0 conn(0x7f1bc8152650 0x7f1bc8155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.975+0000 7f1bcd9d7640 1 -- 172.21.15.1:0/68660435 >> 172.21.15.1:0/68660435 conn(0x7f1bc8066ee0 msgr2=0x7f1bc80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.976+0000 7f1bcd9d7640 1 -- 172.21.15.1:0/68660435 shutdown_connections 2024-03-06T07:25:53.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.976+0000 7f1bcd9d7640 1 -- 172.21.15.1:0/68660435 wait complete. 2024-03-06T07:25:53.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.977+0000 7f1bcd9d7640 1 Processor -- start 2024-03-06T07:25:53.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.977+0000 7f1bcd9d7640 1 -- start start 2024-03-06T07:25:53.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.978+0000 7f1bcd9d7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bc80bf9d0 0x7f1bc814c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.978+0000 7f1bcd9d7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bc8152650 0x7f1bc80c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.978+0000 7f1bcd9d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc80c9df0 0x7f1bc8153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.978+0000 7f1bcd9d7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc8155490 con 0x7f1bc8152650 2024-03-06T07:25:53.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.978+0000 7f1bcd9d7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc81551b0 con 0x7f1bc80c9df0 2024-03-06T07:25:53.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.978+0000 7f1bcd9d7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc8152a30 con 0x7f1bc80bf9d0 2024-03-06T07:25:53.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.979+0000 7f1bc67fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bc8152650 0x7f1bc80c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.979+0000 7f1bc67fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bc8152650 0x7f1bc80c98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36592/0 (socket says 172.21.15.1:36592) 2024-03-06T07:25:53.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.979+0000 7f1bc67fc640 1 -- 172.21.15.1:0/3337560934 learned_addr learned my addr 172.21.15.1:0/3337560934 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:53.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.979+0000 7f1bc77fe640 1 --2- 172.21.15.1:0/3337560934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc80c9df0 0x7f1bc8153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.979+0000 7f1bc6ffd640 1 --2- 172.21.15.1:0/3337560934 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bc80bf9d0 0x7f1bc814c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.979+0000 7f1bc67fc640 1 -- 172.21.15.1:0/3337560934 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bc80bf9d0 msgr2=0x7f1bc814c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.979+0000 7f1bc67fc640 1 --2- 172.21.15.1:0/3337560934 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bc80bf9d0 0x7f1bc814c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.979+0000 7f1bc67fc640 1 -- 172.21.15.1:0/3337560934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc80c9df0 msgr2=0x7f1bc8153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.979+0000 7f1bc67fc640 1 --2- 172.21.15.1:0/3337560934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc80c9df0 0x7f1bc8153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.979+0000 7f1bc67fc640 1 -- 172.21.15.1:0/3337560934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc81ebba0 con 0x7f1bc8152650 2024-03-06T07:25:53.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.979+0000 7f1bc67fc640 1 --2- 172.21.15.1:0/3337560934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bc8152650 0x7f1bc80c98b0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00c570 tx=0x7f1bbc00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.979+0000 7f1baffff640 1 -- 172.21.15.1:0/3337560934 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc015070 con 0x7f1bc8152650 2024-03-06T07:25:53.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.979+0000 7f1baffff640 1 -- 172.21.15.1:0/3337560934 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bbc011800 con 0x7f1bc8152650 2024-03-06T07:25:53.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.979+0000 7f1bcd9d7640 1 -- 172.21.15.1:0/3337560934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc80c8dd0 con 0x7f1bc8152650 2024-03-06T07:25:53.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.979+0000 7f1baffff640 1 -- 172.21.15.1:0/3337560934 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc011bf0 con 0x7f1bc8152650 2024-03-06T07:25:53.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.979+0000 7f1bcd9d7640 1 -- 172.21.15.1:0/3337560934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc80c9390 con 0x7f1bc8152650 2024-03-06T07:25:53.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.980+0000 7f1bcd9d7640 1 -- 172.21.15.1:0/3337560934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc8067fd0 con 0x7f1bc8152650 2024-03-06T07:25:53.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.981+0000 7f1baffff640 1 -- 172.21.15.1:0/3337560934 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1bbc018070 con 0x7f1bc8152650 2024-03-06T07:25:53.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.982+0000 7f1baffff640 1 --2- 172.21.15.1:0/3337560934 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1bb406b4d0 0x7f1bb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.982+0000 7f1baffff640 1 -- 172.21.15.1:0/3337560934 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1bbc08a1b0 con 0x7f1bc8152650 2024-03-06T07:25:53.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.982+0000 7f1bc6ffd640 1 --2- 172.21.15.1:0/3337560934 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1bb406b4d0 0x7f1bb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.982+0000 7f1bc6ffd640 1 --2- 172.21.15.1:0/3337560934 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1bb406b4d0 0x7f1bb406d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1bb800a970 tx=0x7f1bb8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:53.986+0000 7f1baffff640 1 -- 172.21.15.1:0/3337560934 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1bbc058e40 con 0x7f1bc8152650 2024-03-06T07:25:54.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.224+0000 7f1bcd9d7640 1 -- 172.21.15.1:0/3337560934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bc80c48b0 con 0x7f1bc8152650 2024-03-06T07:25:54.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.224+0000 7f1baffff640 1 -- 172.21.15.1:0/3337560934 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+4983 (secure 0 0 0) 0x7f1bbc058740 con 0x7f1bc8152650 2024-03-06T07:25:54.226 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:25:54.227 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":126,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7387,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.98:6836/1671554096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1671554096},{"type":"v1","addr":"172.21.15.98:6836","nonce":1671554096}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122},{"gid":7390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2739580331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2739580331},{"type":"v1","addr":"172.21.15.98:6837","nonce":2739580331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":8517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3374845636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3374845636},{"type":"v1","addr":"172.21.15.1:6834","nonce":3374845636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:29.792840+0000","modified":"2024-03-06T07:25:44.791512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6686":{"gid":6686,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":8,"addr":"172.21.15.1:6835/2848968195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2848968195},{"type":"v1","addr":"172.21.15.1:6835","nonce":2848968195}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6686,"qdb_cluster":[6686]},"id":10}]} 2024-03-06T07:25:54.227 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 126 2024-03-06T07:25:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.226+0000 7f1bcd9d7640 1 -- 172.21.15.1:0/3337560934 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1bb406b4d0 msgr2=0x7f1bb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.226+0000 7f1bcd9d7640 1 --2- 172.21.15.1:0/3337560934 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1bb406b4d0 0x7f1bb406d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1bb800a970 tx=0x7f1bb8008040 comp rx=0 tx=0).stop 2024-03-06T07:25:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.226+0000 7f1bcd9d7640 1 -- 172.21.15.1:0/3337560934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bc8152650 msgr2=0x7f1bc80c98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.226+0000 7f1bcd9d7640 1 --2- 172.21.15.1:0/3337560934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bc8152650 0x7f1bc80c98b0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00c570 tx=0x7f1bbc00ca30 comp rx=0 tx=0).stop 2024-03-06T07:25:54.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.226+0000 7f1bcd9d7640 1 -- 172.21.15.1:0/3337560934 shutdown_connections 2024-03-06T07:25:54.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.226+0000 7f1bcd9d7640 1 --2- 172.21.15.1:0/3337560934 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1bb406b4d0 0x7f1bb406d990 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.226+0000 7f1bcd9d7640 1 --2- 172.21.15.1:0/3337560934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bc80c9df0 0x7f1bc8153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.226+0000 7f1bcd9d7640 1 --2- 172.21.15.1:0/3337560934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bc8152650 0x7f1bc80c98b0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.226+0000 7f1bcd9d7640 1 --2- 172.21.15.1:0/3337560934 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bc80bf9d0 0x7f1bc814c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.226+0000 7f1bcd9d7640 1 -- 172.21.15.1:0/3337560934 >> 172.21.15.1:0/3337560934 conn(0x7f1bc8066ee0 msgr2=0x7f1bc80672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.226+0000 7f1bcd9d7640 1 -- 172.21.15.1:0/3337560934 shutdown_connections 2024-03-06T07:25:54.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.226+0000 7f1bcd9d7640 1 -- 172.21.15.1:0/3337560934 wait complete. 2024-03-06T07:25:54.641 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 126, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:29.792840+0000', 'modified': '2024-03-06T07:25:44.791512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 136, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6686}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6686': {'gid': 6686, 'name': 'c', 'rank': 0, 'incarnation': 123, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.1:6835/2848968195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 2848968195}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2848968195}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6686, 'qdb_cluster': [6686]} 2024-03-06T07:25:54.642 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:54.642 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:25:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.743+0000 7f33e86f4640 1 Processor -- start 2024-03-06T07:25:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.743+0000 7f33e86f4640 1 -- start start 2024-03-06T07:25:54.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.744+0000 7f33e86f4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33e00c2b30 0x7f33e00724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.744+0000 7f33e86f4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f33e0152b70 0x7f33e0072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.744+0000 7f33e86f4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e0151dd0 con 0x7f33e00a5fa0 2024-03-06T07:25:54.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.744+0000 7f33e86f4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33e01526e0 con 0x7f33e0152b70 2024-03-06T07:25:54.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.744+0000 7f33e86f4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e0156190 con 0x7f33e00c2b30 2024-03-06T07:25:54.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.744+0000 7f33e5c8f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f33e0152b70 0x7f33e0072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.744+0000 7f33e5c8f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f33e0152b70 0x7f33e0072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42378/0 (socket says 172.21.15.1:42378) 2024-03-06T07:25:54.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.744+0000 7f33e5c8f640 1 -- 172.21.15.1:0/2494239996 learned_addr learned my addr 172.21.15.1:0/2494239996 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:54.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.744+0000 7f33e6490640 1 --2- 172.21.15.1:0/2494239996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33e00c2b30 0x7f33e00724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.744+0000 7f33e5c8f640 1 -- 172.21.15.1:0/2494239996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33e00c2b30 msgr2=0x7f33e00724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.744+0000 7f33e5c8f640 1 --2- 172.21.15.1:0/2494239996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33e00c2b30 0x7f33e00724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.744+0000 7f33e5c8f640 1 -- 172.21.15.1:0/2494239996 >> v1:172.21.15.98:6789/0 conn(0x7f33e00a5fa0 legacy=0x7f33e00c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.744+0000 7f33e5c8f640 1 -- 172.21.15.1:0/2494239996 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e0152090 con 0x7f33e0152b70 2024-03-06T07:25:54.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.745+0000 7f33e5c8f640 1 --2- 172.21.15.1:0/2494239996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33e0152b70 0x7f33e0072a00 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f33dc009870 tx=0x7f33dc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613c3db9ef01701a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.745+0000 7f33e548e640 1 -- 172.21.15.1:0/2494239996 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc035020 con 0x7f33e0152b70 2024-03-06T07:25:54.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.745+0000 7f33e548e640 1 -- 172.21.15.1:0/2494239996 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33dc031640 con 0x7f33e0152b70 2024-03-06T07:25:54.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.745+0000 7f33e548e640 1 -- 172.21.15.1:0/2494239996 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc02d040 con 0x7f33e0152b70 2024-03-06T07:25:54.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.746+0000 7f33e86f4640 1 -- 172.21.15.1:0/2494239996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33e0152b70 msgr2=0x7f33e0072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.746+0000 7f33e86f4640 1 --2- 172.21.15.1:0/2494239996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33e0152b70 0x7f33e0072a00 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f33dc009870 tx=0x7f33dc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.746+0000 7f33e86f4640 1 -- 172.21.15.1:0/2494239996 shutdown_connections 2024-03-06T07:25:54.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.746+0000 7f33e86f4640 1 --2- 172.21.15.1:0/2494239996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33e0152b70 0x7f33e0072a00 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.746+0000 7f33e86f4640 1 --2- 172.21.15.1:0/2494239996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33e00c2b30 0x7f33e00724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.746+0000 7f33e86f4640 1 -- 172.21.15.1:0/2494239996 >> 172.21.15.1:0/2494239996 conn(0x7f33e00c6860 msgr2=0x7f33e00c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.746+0000 7f33e86f4640 1 -- 172.21.15.1:0/2494239996 shutdown_connections 2024-03-06T07:25:54.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.746+0000 7f33e86f4640 1 -- 172.21.15.1:0/2494239996 wait complete. 2024-03-06T07:25:54.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.748+0000 7f33e86f4640 1 Processor -- start 2024-03-06T07:25:54.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.748+0000 7f33e86f4640 1 -- start start 2024-03-06T07:25:54.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.749+0000 7f33e86f4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33e00a5fa0 0x7f33e00c4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.749+0000 7f33e86f4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33e00c2b30 0x7f33e014e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.749+0000 7f33e86f4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33e0152b70 0x7f33e014ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.749+0000 7f33e86f4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e0155f00 con 0x7f33e00a5fa0 2024-03-06T07:25:54.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.749+0000 7f33e86f4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e0155d80 con 0x7f33e0152b70 2024-03-06T07:25:54.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.749+0000 7f33e86f4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e00c3a00 con 0x7f33e00c2b30 2024-03-06T07:25:54.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.749+0000 7f33e6490640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33e00a5fa0 0x7f33e00c4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.749+0000 7f33e6490640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33e00a5fa0 0x7f33e00c4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42388/0 (socket says 172.21.15.1:42388) 2024-03-06T07:25:54.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.749+0000 7f33e6490640 1 -- 172.21.15.1:0/495733456 learned_addr learned my addr 172.21.15.1:0/495733456 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:54.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.749+0000 7f33e6490640 1 -- 172.21.15.1:0/495733456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33e00c2b30 msgr2=0x7f33e014e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.749+0000 7f33e6c91640 1 --2- 172.21.15.1:0/495733456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33e0152b70 0x7f33e014ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.749+0000 7f33e6490640 1 --2- 172.21.15.1:0/495733456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33e00c2b30 0x7f33e014e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.749+0000 7f33e6490640 1 -- 172.21.15.1:0/495733456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33e0152b70 msgr2=0x7f33e014ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.749+0000 7f33e6490640 1 --2- 172.21.15.1:0/495733456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33e0152b70 0x7f33e014ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.749+0000 7f33e6490640 1 -- 172.21.15.1:0/495733456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e00bb910 con 0x7f33e00a5fa0 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.749+0000 7f33e6490640 1 --2- 172.21.15.1:0/495733456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33e00a5fa0 0x7f33e00c4dc0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f33d000d800 tx=0x7f33d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.750+0000 7f33d6ffd640 1 -- 172.21.15.1:0/495733456 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d000b6c0 con 0x7f33e00a5fa0 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.750+0000 7f33d6ffd640 1 -- 172.21.15.1:0/495733456 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d0019070 con 0x7f33e00a5fa0 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.750+0000 7f33e86f4640 1 -- 172.21.15.1:0/495733456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e00bcb10 con 0x7f33e00a5fa0 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.750+0000 7f33d6ffd640 1 -- 172.21.15.1:0/495733456 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d000ba80 con 0x7f33e00a5fa0 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.750+0000 7f33e86f4640 1 -- 172.21.15.1:0/495733456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e014dfc0 con 0x7f33e00a5fa0 2024-03-06T07:25:54.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.751+0000 7f33d6ffd640 1 -- 172.21.15.1:0/495733456 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33d000bc20 con 0x7f33e00a5fa0 2024-03-06T07:25:54.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.751+0000 7f33d6ffd640 1 --2- 172.21.15.1:0/495733456 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f33c406b500 0x7f33c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.752+0000 7f33d6ffd640 1 -- 172.21.15.1:0/495733456 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f33d008a7d0 con 0x7f33e00a5fa0 2024-03-06T07:25:54.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.752+0000 7f33e5c8f640 1 --2- 172.21.15.1:0/495733456 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f33c406b500 0x7f33c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.752+0000 7f33e5c8f640 1 --2- 172.21.15.1:0/495733456 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f33c406b500 0x7f33c406d9c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f33dc036000 tx=0x7f33dc030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.752+0000 7f33eb3b8740 1 Processor -- start 2024-03-06T07:25:54.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.753+0000 7f33eb3b8740 1 -- start start 2024-03-06T07:25:54.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.753+0000 7f33eb3b8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 0x55a260990a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.753+0000 7f33eb3b8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a260990f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.753+0000 7f33eb3b8740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609914b0 0x55a26098d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.753+0000 7f33eb3b8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2608f76b0 con 0x55a260993330 2024-03-06T07:25:54.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.753+0000 7f33eb3b8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2608a0a60 con 0x55a260992f50 2024-03-06T07:25:54.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.753+0000 7f33eb3b8740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a2608f3750 con 0x55a2609914b0 2024-03-06T07:25:54.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.753+0000 7f33e5c8f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a260990f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.753+0000 7f33e5c8f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a260990f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42400/0 (socket says 172.21.15.1:42400) 2024-03-06T07:25:54.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.753+0000 7f33e5c8f640 1 -- 172.21.15.1:0/3887696765 learned_addr learned my addr 172.21.15.1:0/3887696765 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:54.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.753+0000 7f33e6490640 1 --2- 172.21.15.1:0/3887696765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 0x55a260990a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.753+0000 7f33e6c91640 1 --2- 172.21.15.1:0/3887696765 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609914b0 0x55a26098d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.754+0000 7f33e5c8f640 1 -- 172.21.15.1:0/3887696765 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609914b0 msgr2=0x55a26098d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.754+0000 7f33e5c8f640 1 --2- 172.21.15.1:0/3887696765 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609914b0 0x55a26098d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.754+0000 7f33e5c8f640 1 -- 172.21.15.1:0/3887696765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 msgr2=0x55a260990a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.754+0000 7f33e5c8f640 1 --2- 172.21.15.1:0/3887696765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 0x55a260990a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.754+0000 7f33e5c8f640 1 -- 172.21.15.1:0/3887696765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2608bef30 con 0x55a260993330 2024-03-06T07:25:54.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.754+0000 7f33e5c8f640 1 --2- 172.21.15.1:0/3887696765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a260990f70 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f33dc0090b0 tx=0x7f33dc031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b54b561bf4fff34 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.754+0000 7f33c3fff640 1 -- 172.21.15.1:0/3887696765 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc040070 con 0x55a260993330 2024-03-06T07:25:54.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.754+0000 7f33c3fff640 1 -- 172.21.15.1:0/3887696765 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33dc03a620 con 0x55a260993330 2024-03-06T07:25:54.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.754+0000 7f33c3fff640 1 -- 172.21.15.1:0/3887696765 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc035040 con 0x55a260993330 2024-03-06T07:25:54.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.755+0000 7f33eb3b8740 1 -- 172.21.15.1:0/3887696765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 msgr2=0x55a260990f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.755+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/3887696765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a260990f70 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f33dc0090b0 tx=0x7f33dc031000 comp rx=0 tx=0).stop 2024-03-06T07:25:54.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.755+0000 7f33eb3b8740 1 -- 172.21.15.1:0/3887696765 shutdown_connections 2024-03-06T07:25:54.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.755+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/3887696765 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609914b0 0x55a26098d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.755+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/3887696765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a260990f70 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.755+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/3887696765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 0x55a260990a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.755+0000 7f33eb3b8740 1 -- 172.21.15.1:0/3887696765 >> 172.21.15.1:0/3887696765 conn(0x55a26098f7d0 msgr2=0x55a2609b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.755+0000 7f33eb3b8740 1 -- 172.21.15.1:0/3887696765 shutdown_connections 2024-03-06T07:25:54.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.755+0000 7f33eb3b8740 1 -- 172.21.15.1:0/3887696765 wait complete. 2024-03-06T07:25:54.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.755+0000 7f33eb3b8740 1 -- start start 2024-03-06T07:25:54.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.755+0000 7f33eb3b8740 1 Processor -- start 2024-03-06T07:25:54.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.756+0000 7f33eb3b8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 0x55a2609b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.756+0000 7f33eb3b8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609b5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.756+0000 7f33eb3b8740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609b76e0 0x55a2609b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.756+0000 7f33eb3b8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2609b5230 con 0x55a260993330 2024-03-06T07:25:54.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.756+0000 7f33eb3b8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2608b3ea0 con 0x55a260992f50 2024-03-06T07:25:54.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.756+0000 7f33eb3b8740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a2609b4c70 con 0x55a2609b76e0 2024-03-06T07:25:54.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.756+0000 7f33eb3b8740 10 client.0 _ll_register_callbacks cb 0x55a2608b3870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:54.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.756+0000 7f33e5c8f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609b5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.756+0000 7f33e5c8f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609b5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42414/0 (socket says 172.21.15.1:42414) 2024-03-06T07:25:54.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.756+0000 7f33e5c8f640 1 -- 172.21.15.1:0/913058772 learned_addr learned my addr 172.21.15.1:0/913058772 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:54.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.756+0000 7f33e6490640 1 --2- 172.21.15.1:0/913058772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 0x55a2609b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.756+0000 7f33e6c91640 1 --2- 172.21.15.1:0/913058772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609b76e0 0x55a2609b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33e5c8f640 1 -- 172.21.15.1:0/913058772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609b76e0 msgr2=0x55a2609b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33e5c8f640 1 --2- 172.21.15.1:0/913058772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609b76e0 0x55a2609b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33e5c8f640 1 -- 172.21.15.1:0/913058772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 msgr2=0x55a2609b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33e5c8f640 1 --2- 172.21.15.1:0/913058772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 0x55a2609b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33e5c8f640 1 -- 172.21.15.1:0/913058772 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a2608b4270 con 0x55a260993330 2024-03-06T07:25:54.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33e5c8f640 1 --2- 172.21.15.1:0/913058772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609b5600 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f33dc02bf00 tx=0x7f33dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c7582901c0787 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:25:54.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33c17fa640 1 -- 172.21.15.1:0/913058772 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc040040 con 0x55a260993330 2024-03-06T07:25:54.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33c17fa640 1 -- 172.21.15.1:0/913058772 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f33dc057650 con 0x55a260993330 2024-03-06T07:25:54.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33eb3b8740 1 -- 172.21.15.1:0/913058772 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33dc03a620 con 0x55a260993330 2024-03-06T07:25:54.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33c17fa640 1 -- 172.21.15.1:0/913058772 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=126) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33dc05f050 con 0x55a260993330 2024-03-06T07:25:54.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33eb3b8740 10 client.8595 fetch_fsmap learned FSMap version 126 2024-03-06T07:25:54.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33eb3b8740 1 -- 172.21.15.1:0/913058772 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=126,monmap=2+}) v3 -- 0x55a26060c2d0 con 0x55a260993330 2024-03-06T07:25:54.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33c17fa640 1 -- 172.21.15.1:0/913058772 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33dc03a620 con 0x55a260993330 2024-03-06T07:25:54.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33c17fa640 1 -- 172.21.15.1:0/913058772 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 126) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f33dc062020 con 0x55a260993330 2024-03-06T07:25:54.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33eb3b8740 10 client.8595 fetch_fsmap finished waiting for FSMap version 126 2024-03-06T07:25:54.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33eb3b8740 10 client.8595 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:25:54.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33eb3b8740 20 client.8595 populate_metadata read hostname 'smithi001' 2024-03-06T07:25:54.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/913058772 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x55a2609e7680 0x55a2609e7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33eb3b8740 4 client.8595 mds_command: new command op to 6686 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:54.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.757+0000 7f33eb3b8740 1 -- 172.21.15.1:0/913058772 --> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a260904bf0 con 0x55a2609e7680 2024-03-06T07:25:54.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.758+0000 7f33e6490640 1 --2- 172.21.15.1:0/913058772 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x55a2609e7680 0x55a2609e7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.758+0000 7f33c17fa640 10 client.8595 ms_handle_connect on v2:172.21.15.1:6833/2848968195 2024-03-06T07:25:54.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.758+0000 7f33e6490640 1 --2- 172.21.15.1:0/913058772 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x55a2609e7680 0x55a2609e7a60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33bdc5ee1124b92b server_cookie=c62606603c44537d in_seq=0 out_seq=0 2024-03-06T07:25:54.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.758+0000 7f33c17fa640 1 -- 172.21.15.1:0/913058772 <== mds.0 v2:172.21.15.1:6833/2848968195 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a260904bf0 con 0x55a2609e7680 2024-03-06T07:25:54.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.758+0000 7f33c17fa640 10 client.8595 handle_command_reply: tid=0 2024-03-06T07:25:54.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.759+0000 7f33eb3b8740 1 client.8595 shutdown 2024-03-06T07:25:54.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.759+0000 7f33eb3b8740 1 -- 172.21.15.1:0/913058772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 msgr2=0x55a2609b5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.759+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/913058772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609b5600 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f33dc02bf00 tx=0x7f33dc004730 comp rx=0 tx=0).stop 2024-03-06T07:25:54.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.759+0000 7f33eb3b8740 1 -- 172.21.15.1:0/913058772 shutdown_connections 2024-03-06T07:25:54.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.759+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/913058772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609b76e0 0x55a2609b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.759+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/913058772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609b5600 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.759+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/913058772 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x55a2609e7680 0x55a2609e7a60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.759+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/913058772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 0x55a2609b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.759+0000 7f33c17fa640 0 client.8595 ms_handle_reset on v2:172.21.15.1:6833/2848968195 2024-03-06T07:25:54.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.759+0000 7f33eb3b8740 1 -- 172.21.15.1:0/913058772 >> 172.21.15.1:0/913058772 conn(0x55a2609914b0 msgr2=0x55a2609918e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.759+0000 7f33eb3b8740 1 -- 172.21.15.1:0/913058772 shutdown_connections 2024-03-06T07:25:54.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.759+0000 7f33eb3b8740 1 -- 172.21.15.1:0/913058772 wait complete. 2024-03-06T07:25:54.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.759+0000 7f33eb3b8740 20 client.8595 trim_cache size 0 max 16384 2024-03-06T07:25:54.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.792+0000 7f33eb3b8740 1 Processor -- start 2024-03-06T07:25:54.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.792+0000 7f33eb3b8740 1 -- start start 2024-03-06T07:25:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.793+0000 7f33eb3b8740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609914b0 0x55a26098c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.793+0000 7f33eb3b8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 0x55a2609b1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.793+0000 7f33eb3b8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609b1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.793+0000 7f33eb3b8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e015ea40 con 0x55a260993330 2024-03-06T07:25:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.793+0000 7f33eb3b8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e0147960 con 0x55a260992f50 2024-03-06T07:25:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.793+0000 7f33eb3b8740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e014c7f0 con 0x55a2609914b0 2024-03-06T07:25:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.793+0000 7f33e6c91640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.793+0000 7f33e6c91640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609b1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42422/0 (socket says 172.21.15.1:42422) 2024-03-06T07:25:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.793+0000 7f33e6c91640 1 -- 172.21.15.1:0/1765306233 learned_addr learned my addr 172.21.15.1:0/1765306233 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.793+0000 7f33e6490640 1 --2- 172.21.15.1:0/1765306233 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609914b0 0x55a26098c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.793+0000 7f33e5c8f640 1 --2- 172.21.15.1:0/1765306233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 0x55a2609b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.793+0000 7f33e6c91640 1 -- 172.21.15.1:0/1765306233 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609914b0 msgr2=0x55a26098c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.793+0000 7f33e6c91640 1 --2- 172.21.15.1:0/1765306233 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609914b0 0x55a26098c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.794+0000 7f33e6c91640 1 -- 172.21.15.1:0/1765306233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 msgr2=0x55a2609b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.794+0000 7f33e6c91640 1 --2- 172.21.15.1:0/1765306233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 0x55a2609b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.794+0000 7f33e6c91640 1 -- 172.21.15.1:0/1765306233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2606d66b0 con 0x55a260993330 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.794+0000 7f33e5c8f640 1 --2- 172.21.15.1:0/1765306233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 0x55a2609b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.794+0000 7f33e6c91640 1 --2- 172.21.15.1:0/1765306233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609b1850 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f33d80056e0 tx=0x7f33d8013700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5cd70e70e2b8f09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.794+0000 7f33c37fe640 1 -- 172.21.15.1:0/1765306233 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d801d430 con 0x55a260993330 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.794+0000 7f33c37fe640 1 -- 172.21.15.1:0/1765306233 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d801d5d0 con 0x55a260993330 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.794+0000 7f33c37fe640 1 -- 172.21.15.1:0/1765306233 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d801d980 con 0x55a260993330 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.794+0000 7f33eb3b8740 1 -- 172.21.15.1:0/1765306233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 msgr2=0x55a2609b1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.794+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/1765306233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609b1850 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f33d80056e0 tx=0x7f33d8013700 comp rx=0 tx=0).stop 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.794+0000 7f33eb3b8740 1 -- 172.21.15.1:0/1765306233 shutdown_connections 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.794+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/1765306233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609b1850 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.794+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/1765306233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 0x55a2609b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.794+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/1765306233 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609914b0 0x55a26098c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.794+0000 7f33eb3b8740 1 -- 172.21.15.1:0/1765306233 >> 172.21.15.1:0/1765306233 conn(0x55a26098ee70 msgr2=0x55a2609cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.794+0000 7f33eb3b8740 1 -- 172.21.15.1:0/1765306233 shutdown_connections 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.794+0000 7f33eb3b8740 1 -- 172.21.15.1:0/1765306233 wait complete. 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.795+0000 7f33eb3b8740 1 -- start start 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.795+0000 7f33eb3b8740 1 Processor -- start 2024-03-06T07:25:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.796+0000 7f33eb3b8740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609914b0 0x55a2609b1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.796+0000 7f33eb3b8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 0x55a2609aed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.796+0000 7f33eb3b8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609cb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.796+0000 7f33eb3b8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2609b5230 con 0x55a260993330 2024-03-06T07:25:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.796+0000 7f33eb3b8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2609b7430 con 0x55a260992f50 2024-03-06T07:25:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.796+0000 7f33eb3b8740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a2609bdc10 con 0x55a2609914b0 2024-03-06T07:25:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.796+0000 7f33eb3b8740 10 client.0 _ll_register_callbacks cb 0x7f33d8003c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.796+0000 7f33e6c91640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.796+0000 7f33e6c91640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609cb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42424/0 (socket says 172.21.15.1:42424) 2024-03-06T07:25:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.796+0000 7f33e6c91640 1 -- 172.21.15.1:0/3633648490 learned_addr learned my addr 172.21.15.1:0/3633648490 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.796+0000 7f33e6490640 1 --2- 172.21.15.1:0/3633648490 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609914b0 0x55a2609b1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.796+0000 7f33e5c8f640 1 --2- 172.21.15.1:0/3633648490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 0x55a2609aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.796+0000 7f33e6c91640 1 -- 172.21.15.1:0/3633648490 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609914b0 msgr2=0x55a2609b1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.796+0000 7f33e6c91640 1 --2- 172.21.15.1:0/3633648490 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609914b0 0x55a2609b1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.796+0000 7f33e6c91640 1 -- 172.21.15.1:0/3633648490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 msgr2=0x55a2609aed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.796+0000 7f33e6c91640 1 --2- 172.21.15.1:0/3633648490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 0x55a2609aed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.796+0000 7f33e6c91640 1 -- 172.21.15.1:0/3633648490 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a2609943e0 con 0x55a260993330 2024-03-06T07:25:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.797+0000 7f33e6c91640 1 --2- 172.21.15.1:0/3633648490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609cb550 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f33d801dc50 tx=0x7f33d8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f878274a4eda4c0a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.797+0000 7f33c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:25:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.797+0000 7f33c17fa640 1 -- 172.21.15.1:0/3633648490 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d802e060 con 0x55a260993330 2024-03-06T07:25:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.797+0000 7f33c17fa640 1 -- 172.21.15.1:0/3633648490 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f33d8015620 con 0x55a260993330 2024-03-06T07:25:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.797+0000 7f33eb3b8740 1 -- 172.21.15.1:0/3633648490 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33d801d5d0 con 0x55a260993330 2024-03-06T07:25:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.797+0000 7f33c17fa640 1 -- 172.21.15.1:0/3633648490 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=126) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33d801d430 con 0x55a260993330 2024-03-06T07:25:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.797+0000 7f33eb3b8740 10 client.8601 fetch_fsmap learned FSMap version 126 2024-03-06T07:25:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.797+0000 7f33eb3b8740 1 -- 172.21.15.1:0/3633648490 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=126,monmap=2+}) v3 -- 0x55a2609e1680 con 0x55a260993330 2024-03-06T07:25:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.797+0000 7f33c17fa640 1 -- 172.21.15.1:0/3633648490 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d801d5d0 con 0x55a260993330 2024-03-06T07:25:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.797+0000 7f33c17fa640 1 -- 172.21.15.1:0/3633648490 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 126) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f33d8037080 con 0x55a260993330 2024-03-06T07:25:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.797+0000 7f33eb3b8740 10 client.8601 fetch_fsmap finished waiting for FSMap version 126 2024-03-06T07:25:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.797+0000 7f33eb3b8740 10 client.8601 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:25:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.797+0000 7f33eb3b8740 20 client.8601 populate_metadata read hostname 'smithi001' 2024-03-06T07:25:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.797+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/3633648490 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x55a2609b76e0 0x55a260a463b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.797+0000 7f33eb3b8740 4 client.8601 mds_command: new command op to 6686 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.797+0000 7f33eb3b8740 1 -- 172.21.15.1:0/3633648490 --> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a2608fef40 con 0x55a2609b76e0 2024-03-06T07:25:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.797+0000 7f33e6490640 1 --2- 172.21.15.1:0/3633648490 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x55a2609b76e0 0x55a260a463b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.798+0000 7f33c17fa640 10 client.8601 ms_handle_connect on v2:172.21.15.1:6833/2848968195 2024-03-06T07:25:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.798+0000 7f33e6490640 1 --2- 172.21.15.1:0/3633648490 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x55a2609b76e0 0x55a260a463b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb9535f493c0c56d server_cookie=ca7322536fdc06ca in_seq=0 out_seq=0 2024-03-06T07:25:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.798+0000 7f33c17fa640 1 -- 172.21.15.1:0/3633648490 <== mds.0 v2:172.21.15.1:6833/2848968195 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55a2608fef40 con 0x55a2609b76e0 2024-03-06T07:25:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.798+0000 7f33c17fa640 10 client.8601 handle_command_reply: tid=0 2024-03-06T07:25:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.798+0000 7f33eb3b8740 1 client.8601 shutdown 2024-03-06T07:25:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.798+0000 7f33eb3b8740 1 -- 172.21.15.1:0/3633648490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 msgr2=0x55a2609cb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.798+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/3633648490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609cb550 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f33d801dc50 tx=0x7f33d8016000 comp rx=0 tx=0).stop 2024-03-06T07:25:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.798+0000 7f33eb3b8740 1 -- 172.21.15.1:0/3633648490 shutdown_connections 2024-03-06T07:25:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.798+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/3633648490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a260993330 0x55a2609cb550 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.798+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/3633648490 >> [v2:172.21.15.1:6833/2848968195,v1:172.21.15.1:6835/2848968195] conn(0x55a2609b76e0 0x55a260a463b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.798+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/3633648490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a260992f50 0x55a2609aed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.798+0000 7f33eb3b8740 1 --2- 172.21.15.1:0/3633648490 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a2609914b0 0x55a2609b1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.798+0000 7f33c17fa640 0 client.8601 ms_handle_reset on v2:172.21.15.1:6833/2848968195 2024-03-06T07:25:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.798+0000 7f33eb3b8740 1 -- 172.21.15.1:0/3633648490 >> 172.21.15.1:0/3633648490 conn(0x55a26098ee70 msgr2=0x55a26098c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.798+0000 7f33eb3b8740 1 -- 172.21.15.1:0/3633648490 shutdown_connections 2024-03-06T07:25:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.798+0000 7f33eb3b8740 1 -- 172.21.15.1:0/3633648490 wait complete. 2024-03-06T07:25:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.798+0000 7f33eb3b8740 20 client.8601 trim_cache size 0 max 16384 2024-03-06T07:25:54.807 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:25:54.807 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:25:54.807 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:25:54.807 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 6686, 2024-03-06T07:25:54.807 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:25:54.807 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:25:54.807 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:25:54.807 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 13.009527657, 2024-03-06T07:25:54.807 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 126, 2024-03-06T07:25:54.807 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 136, 2024-03-06T07:25:54.807 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 136, 2024-03-06T07:25:54.807 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 29.449233181 2024-03-06T07:25:54.807 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:25:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.799+0000 7f33e86f4640 1 -- 172.21.15.1:0/495733456 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f33c406b500 msgr2=0x7f33c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.799+0000 7f33e86f4640 1 --2- 172.21.15.1:0/495733456 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f33c406b500 0x7f33c406d9c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f33dc036000 tx=0x7f33dc030000 comp rx=0 tx=0).stop 2024-03-06T07:25:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.799+0000 7f33e86f4640 1 -- 172.21.15.1:0/495733456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33e00a5fa0 msgr2=0x7f33e00c4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.799+0000 7f33e86f4640 1 --2- 172.21.15.1:0/495733456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33e00a5fa0 0x7f33e00c4dc0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f33d000d800 tx=0x7f33d000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.799+0000 7f33e86f4640 1 -- 172.21.15.1:0/495733456 shutdown_connections 2024-03-06T07:25:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.799+0000 7f33e86f4640 1 --2- 172.21.15.1:0/495733456 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f33c406b500 0x7f33c406d9c0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.799+0000 7f33e86f4640 1 --2- 172.21.15.1:0/495733456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33e0152b70 0x7f33e014ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.799+0000 7f33e86f4640 1 --2- 172.21.15.1:0/495733456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33e00c2b30 0x7f33e014e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.799+0000 7f33e86f4640 1 --2- 172.21.15.1:0/495733456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33e00a5fa0 0x7f33e00c4dc0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.799+0000 7f33e86f4640 1 -- 172.21.15.1:0/495733456 >> 172.21.15.1:0/495733456 conn(0x7f33e00c6860 msgr2=0x7f33e014fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.799+0000 7f33e86f4640 1 -- 172.21.15.1:0/495733456 shutdown_connections 2024-03-06T07:25:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:54.799+0000 7f33e86f4640 1 -- 172.21.15.1:0/495733456 wait complete. 2024-03-06T07:25:54.811 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 126, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:25:29.792840+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 126, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6686": { "addr": "172.21.15.1:6835/2848968195", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 2848968195, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 2848968195, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6686, "incarnation": 123, "join_fscid": 10, "name": "c", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 136, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-03-06T07:25:44.791512+0000", "qdb_cluster": [ 6686 ], "qdb_leader": 6686, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6686 } } } ], "standbys": [ { "addr": "172.21.15.98:6836/1671554096", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 1671554096, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 1671554096, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 122, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7387, "incarnation": 0, "join_fscid": 10, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 5 }, { "addr": "172.21.15.98:6837/2739580331", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 2739580331, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2739580331, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 118, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7390, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6834/3374845636", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 3374845636, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 3374845636, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 124, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8517, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:25:54.812 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 10, 'name': 'd', 'state': 'up:active'}] 2024-03-06T07:25:54.812 INFO:tasks.cephfs.test_failover:current = [{'gid': 8517, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6834/3374845636', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 3374845636}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 3374845636}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 124}, {'gid': 7390, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/2739580331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2739580331}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2739580331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 118}, {'gid': 6686, 'name': 'c', 'rank': 0, 'incarnation': 123, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.1:6835/2848968195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 2848968195}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2848968195}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7387, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 5, 'addr': '172.21.15.98:6836/1671554096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 1671554096}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1671554096}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 122}] 2024-03-06T07:25:54.813 DEBUG:tasks.cephfs.test_failover:-1 != 10 2024-03-06T07:25:54.813 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:25:55.871 INFO:tasks.ceph.mds.c.smithi001.stderr: -476> 2024-03-06T07:25:45.715+0000 7f4ef00d3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:55.895 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:25:59.813 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:59.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.926+0000 7f243a28d640 1 Processor -- start 2024-03-06T07:25:59.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.926+0000 7f243a28d640 1 -- start start 2024-03-06T07:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.927+0000 7f243a28d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2434152920 0x7f243406aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.927+0000 7f243a28d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f243406b530 0x7f24340c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.927+0000 7f243a28d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2434158910 con 0x7f24340c5d90 2024-03-06T07:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.927+0000 7f243a28d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2434152d00 con 0x7f243406b530 2024-03-06T07:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.927+0000 7f243a28d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24341596d0 con 0x7f2434152920 2024-03-06T07:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.927+0000 7f24337fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f243406b530 0x7f24340c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.927+0000 7f24337fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f243406b530 0x7f24340c5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42450/0 (socket says 172.21.15.1:42450) 2024-03-06T07:25:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.927+0000 7f24337fe640 1 -- 172.21.15.1:0/3201002177 learned_addr learned my addr 172.21.15.1:0/3201002177 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.927+0000 7f2433fff640 1 --2- 172.21.15.1:0/3201002177 >> v2:172.21.15.98:3300/0 conn(0x7f2434152920 0x7f243406aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.927+0000 7f24337fe640 1 -- 172.21.15.1:0/3201002177 >> v2:172.21.15.98:3300/0 conn(0x7f2434152920 msgr2=0x7f243406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.927+0000 7f24337fe640 1 --2- 172.21.15.1:0/3201002177 >> v2:172.21.15.98:3300/0 conn(0x7f2434152920 0x7f243406aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.927+0000 7f24337fe640 1 -- 172.21.15.1:0/3201002177 >> v1:172.21.15.98:6789/0 conn(0x7f24340c5d90 legacy=0x7f2434155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.927+0000 7f24337fe640 1 -- 172.21.15.1:0/3201002177 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2434158bf0 con 0x7f243406b530 2024-03-06T07:25:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.928+0000 7f24337fe640 1 --2- 172.21.15.1:0/3201002177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f243406b530 0x7f24340c5850 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f243406bd50 tx=0x7f242402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1103e79deadf49d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.928+0000 7f2432ffd640 1 -- 172.21.15.1:0/3201002177 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2424035020 con 0x7f243406b530 2024-03-06T07:25:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.928+0000 7f2432ffd640 1 -- 172.21.15.1:0/3201002177 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2424031870 con 0x7f243406b530 2024-03-06T07:25:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.928+0000 7f2432ffd640 1 -- 172.21.15.1:0/3201002177 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2424031c40 con 0x7f243406b530 2024-03-06T07:25:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.929+0000 7f243a28d640 1 -- 172.21.15.1:0/3201002177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f243406b530 msgr2=0x7f24340c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.929+0000 7f243a28d640 1 --2- 172.21.15.1:0/3201002177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f243406b530 0x7f24340c5850 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f243406bd50 tx=0x7f242402b910 comp rx=0 tx=0).stop 2024-03-06T07:25:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.929+0000 7f243a28d640 1 -- 172.21.15.1:0/3201002177 shutdown_connections 2024-03-06T07:25:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.929+0000 7f243a28d640 1 --2- 172.21.15.1:0/3201002177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f243406b530 0x7f24340c5850 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.929+0000 7f243a28d640 1 --2- 172.21.15.1:0/3201002177 >> v2:172.21.15.98:3300/0 conn(0x7f2434152920 0x7f243406aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.929+0000 7f243a28d640 1 -- 172.21.15.1:0/3201002177 >> 172.21.15.1:0/3201002177 conn(0x7f24340a9700 msgr2=0x7f24340a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.929+0000 7f243a28d640 1 -- 172.21.15.1:0/3201002177 shutdown_connections 2024-03-06T07:25:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.929+0000 7f243a28d640 1 -- 172.21.15.1:0/3201002177 wait complete. 2024-03-06T07:25:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.930+0000 7f243a28d640 1 Processor -- start 2024-03-06T07:25:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.930+0000 7f243a28d640 1 -- start start 2024-03-06T07:25:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.931+0000 7f243a28d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f243406b530 0x7f24340ad3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.931+0000 7f243a28d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24340c5d90 0x7f24341eb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.931+0000 7f243a28d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2434152920 0x7f24340a8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.931+0000 7f243a28d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24340c6380 con 0x7f24340c5d90 2024-03-06T07:25:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.931+0000 7f243a28d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2434159240 con 0x7f2434152920 2024-03-06T07:25:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.931+0000 7f243a28d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2434155930 con 0x7f243406b530 2024-03-06T07:25:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.931+0000 7f24337fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24340c5d90 0x7f24341eb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.931+0000 7f24337fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24340c5d90 0x7f24341eb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42462/0 (socket says 172.21.15.1:42462) 2024-03-06T07:25:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.931+0000 7f24337fe640 1 -- 172.21.15.1:0/3791287565 learned_addr learned my addr 172.21.15.1:0/3791287565 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:25:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.931+0000 7f2433fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f243406b530 0x7f24340ad3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.931+0000 7f243882a640 1 --2- 172.21.15.1:0/3791287565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2434152920 0x7f24340a8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.932+0000 7f24337fe640 1 -- 172.21.15.1:0/3791287565 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f243406b530 msgr2=0x7f24340ad3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.932+0000 7f24337fe640 1 --2- 172.21.15.1:0/3791287565 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f243406b530 0x7f24340ad3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.932+0000 7f24337fe640 1 -- 172.21.15.1:0/3791287565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2434152920 msgr2=0x7f24340a8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.932+0000 7f24337fe640 1 --2- 172.21.15.1:0/3791287565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2434152920 0x7f24340a8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.932+0000 7f24337fe640 1 -- 172.21.15.1:0/3791287565 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2434153a70 con 0x7f24340c5d90 2024-03-06T07:25:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.932+0000 7f24337fe640 1 --2- 172.21.15.1:0/3791287565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24340c5d90 0x7f24341eb5f0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f2424000c00 tx=0x7f24240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.932+0000 7f2430ff9640 1 -- 172.21.15.1:0/3791287565 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2424040070 con 0x7f24340c5d90 2024-03-06T07:25:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.932+0000 7f2430ff9640 1 -- 172.21.15.1:0/3791287565 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2424030d00 con 0x7f24340c5d90 2024-03-06T07:25:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.932+0000 7f2430ff9640 1 -- 172.21.15.1:0/3791287565 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2424035040 con 0x7f24340c5d90 2024-03-06T07:25:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.932+0000 7f243a28d640 1 -- 172.21.15.1:0/3791287565 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2434154890 con 0x7f24340c5d90 2024-03-06T07:25:59.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.932+0000 7f243a28d640 1 -- 172.21.15.1:0/3791287565 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24340a9440 con 0x7f24340c5d90 2024-03-06T07:25:59.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.933+0000 7f243a28d640 1 -- 172.21.15.1:0/3791287565 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2434067fd0 con 0x7f24340c5d90 2024-03-06T07:25:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.933+0000 7f2430ff9640 1 -- 172.21.15.1:0/3791287565 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f242404b060 con 0x7f24340c5d90 2024-03-06T07:25:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.934+0000 7f2430ff9640 1 --2- 172.21.15.1:0/3791287565 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f242006b4d0 0x7f242006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.934+0000 7f2430ff9640 1 -- 172.21.15.1:0/3791287565 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f24240abf70 con 0x7f24340c5d90 2024-03-06T07:25:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.934+0000 7f2433fff640 1 --2- 172.21.15.1:0/3791287565 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f242006b4d0 0x7f242006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.935+0000 7f2433fff640 1 --2- 172.21.15.1:0/3791287565 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f242006b4d0 0x7f242006d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f24340ad9e0 tx=0x7f2428009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:25:59.938+0000 7f2430ff9640 1 -- 172.21.15.1:0/3791287565 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f242407a5f0 con 0x7f24340c5d90 2024-03-06T07:26:00.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.156+0000 7f243a28d640 1 -- 172.21.15.1:0/3791287565 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f243406bdf0 con 0x7f24340c5d90 2024-03-06T07:26:00.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.156+0000 7f2430ff9640 1 -- 172.21.15.1:0/3791287565 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+4984 (secure 0 0 0) 0x7f242407e690 con 0x7f24340c5d90 2024-03-06T07:26:00.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:00.159 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":131,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2739580331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2739580331},{"type":"v1","addr":"172.21.15.98:6837","nonce":2739580331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":8517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3374845636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3374845636},{"type":"v1","addr":"172.21.15.1:6834","nonce":3374845636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":8607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1602915919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1602915919},{"type":"v1","addr":"172.21.15.1:6835","nonce":1602915919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:29.792840+0000","modified":"2024-03-06T07:25:58.868171+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7387},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.98:6836/1671554096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1671554096},{"type":"v1","addr":"172.21.15.98:6836","nonce":1671554096}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7387,"qdb_cluster":[7387]},"id":10}]} 2024-03-06T07:26:00.159 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 131 2024-03-06T07:26:00.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.158+0000 7f243a28d640 1 -- 172.21.15.1:0/3791287565 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f242006b4d0 msgr2=0x7f242006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.158+0000 7f243a28d640 1 --2- 172.21.15.1:0/3791287565 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f242006b4d0 0x7f242006d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f24340ad9e0 tx=0x7f2428009190 comp rx=0 tx=0).stop 2024-03-06T07:26:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.158+0000 7f243a28d640 1 -- 172.21.15.1:0/3791287565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24340c5d90 msgr2=0x7f24341eb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.158+0000 7f243a28d640 1 --2- 172.21.15.1:0/3791287565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24340c5d90 0x7f24341eb5f0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f2424000c00 tx=0x7f24240046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.158+0000 7f243a28d640 1 -- 172.21.15.1:0/3791287565 shutdown_connections 2024-03-06T07:26:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.158+0000 7f243a28d640 1 --2- 172.21.15.1:0/3791287565 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f242006b4d0 0x7f242006d990 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.158+0000 7f243a28d640 1 --2- 172.21.15.1:0/3791287565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2434152920 0x7f24340a8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.158+0000 7f243a28d640 1 --2- 172.21.15.1:0/3791287565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24340c5d90 0x7f24341eb5f0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.158+0000 7f243a28d640 1 --2- 172.21.15.1:0/3791287565 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f243406b530 0x7f24340ad3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.158+0000 7f243a28d640 1 -- 172.21.15.1:0/3791287565 >> 172.21.15.1:0/3791287565 conn(0x7f24340a9700 msgr2=0x7f24340a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.158+0000 7f243a28d640 1 -- 172.21.15.1:0/3791287565 shutdown_connections 2024-03-06T07:26:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.158+0000 7f243a28d640 1 -- 172.21.15.1:0/3791287565 wait complete. 2024-03-06T07:26:00.573 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 131, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:29.792840+0000', 'modified': '2024-03-06T07:25:58.868171+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 137, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7387}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7387': {'gid': 7387, 'name': 'd', 'rank': 0, 'incarnation': 128, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.98:6836/1671554096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 1671554096}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1671554096}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7387, 'qdb_cluster': [7387]} 2024-03-06T07:26:00.574 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:26:00.574 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:26:00.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.677+0000 7f8420ef4640 1 Processor -- start 2024-03-06T07:26:00.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.677+0000 7f8420ef4640 1 -- start start 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.678+0000 7f8420ef4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f841c14cdc0 0x7f841c152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.678+0000 7f8420ef4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f841c06b510 0x7f841c06b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.678+0000 7f8420ef4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f841c1569e0 con 0x7f841c156500 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.678+0000 7f8420ef4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f841c0c2dd0 con 0x7f841c14cdc0 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.678+0000 7f8420ef4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f841c156cc0 con 0x7f841c06b510 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.678+0000 7f8419d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f841c156500 0x7f841c0c2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38284/0 (socket says 172.21.15.1:38284) 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.678+0000 7f8419d9b640 1 -- 172.21.15.1:0/458230893 learned_addr learned my addr 172.21.15.1:0/458230893 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.678+0000 7f841a59c640 1 --2- 172.21.15.1:0/458230893 >> v2:172.21.15.98:3300/0 conn(0x7f841c14cdc0 0x7f841c152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.678+0000 7f841ad9d640 1 --2- 172.21.15.1:0/458230893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f841c06b510 0x7f841c06b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.678+0000 7f841959a640 1 -- 172.21.15.1:0/458230893 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1200030500 0 0) 0x7f841c1569e0 con 0x7f841c156500 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.678+0000 7f841959a640 1 -- 172.21.15.1:0/458230893 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8404003520 con 0x7f841c156500 2024-03-06T07:26:00.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.678+0000 7f841a59c640 1 -- 172.21.15.1:0/458230893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f841c06b510 msgr2=0x7f841c06b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.678+0000 7f841a59c640 1 --2- 172.21.15.1:0/458230893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f841c06b510 0x7f841c06b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.678+0000 7f841a59c640 1 -- 172.21.15.1:0/458230893 >> v1:172.21.15.1:6789/0 conn(0x7f841c156500 legacy=0x7f841c0c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.678+0000 7f841a59c640 1 -- 172.21.15.1:0/458230893 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841c156fa0 con 0x7f841c14cdc0 2024-03-06T07:26:00.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.678+0000 7f841ad9d640 1 --2- 172.21.15.1:0/458230893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f841c06b510 0x7f841c06b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:00.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.679+0000 7f841a59c640 1 --2- 172.21.15.1:0/458230893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f841c14cdc0 0x7f841c152f10 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f84140097a0 tx=0x7f841402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcbc01c7434468f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.679+0000 7f841959a640 1 -- 172.21.15.1:0/458230893 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8414035020 con 0x7f841c14cdc0 2024-03-06T07:26:00.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.679+0000 7f841959a640 1 -- 172.21.15.1:0/458230893 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8414031830 con 0x7f841c14cdc0 2024-03-06T07:26:00.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.679+0000 7f841959a640 1 -- 172.21.15.1:0/458230893 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8414031c00 con 0x7f841c14cdc0 2024-03-06T07:26:00.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.680+0000 7f8420ef4640 1 -- 172.21.15.1:0/458230893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f841c14cdc0 msgr2=0x7f841c152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.680+0000 7f8420ef4640 1 --2- 172.21.15.1:0/458230893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f841c14cdc0 0x7f841c152f10 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f84140097a0 tx=0x7f841402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.680+0000 7f8420ef4640 1 -- 172.21.15.1:0/458230893 shutdown_connections 2024-03-06T07:26:00.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.680+0000 7f8420ef4640 1 --2- 172.21.15.1:0/458230893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f841c06b510 0x7f841c06b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.680+0000 7f8420ef4640 1 --2- 172.21.15.1:0/458230893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f841c14cdc0 0x7f841c152f10 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.680+0000 7f8420ef4640 1 -- 172.21.15.1:0/458230893 >> 172.21.15.1:0/458230893 conn(0x7f841c0a9700 msgr2=0x7f841c0a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.680+0000 7f8420ef4640 1 -- 172.21.15.1:0/458230893 shutdown_connections 2024-03-06T07:26:00.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.680+0000 7f8420ef4640 1 -- 172.21.15.1:0/458230893 wait complete. 2024-03-06T07:26:00.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.681+0000 7f8420ef4640 1 Processor -- start 2024-03-06T07:26:00.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.681+0000 7f8420ef4640 1 -- start start 2024-03-06T07:26:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.682+0000 7f8420ef4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f841c06b510 0x7f841c0a8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.682+0000 7f8420ef4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f841c14cdc0 0x7f841c1dda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.682+0000 7f8420ef4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f841c156500 0x7f841c1ddfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.682+0000 7f8420ef4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c06bea0 con 0x7f841c06b510 2024-03-06T07:26:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.682+0000 7f8420ef4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c0c5a70 con 0x7f841c156500 2024-03-06T07:26:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.682+0000 7f8420ef4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f841c0c65e0 con 0x7f841c14cdc0 2024-03-06T07:26:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.682+0000 7f841a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f841c06b510 0x7f841c0a8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.682+0000 7f841a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f841c06b510 0x7f841c0a8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42470/0 (socket says 172.21.15.1:42470) 2024-03-06T07:26:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.682+0000 7f841a59c640 1 -- 172.21.15.1:0/2784610109 learned_addr learned my addr 172.21.15.1:0/2784610109 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.682+0000 7f8419d9b640 1 --2- 172.21.15.1:0/2784610109 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f841c14cdc0 0x7f841c1dda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.682+0000 7f841ad9d640 1 --2- 172.21.15.1:0/2784610109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f841c156500 0x7f841c1ddfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.682+0000 7f841a59c640 1 -- 172.21.15.1:0/2784610109 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f841c14cdc0 msgr2=0x7f841c1dda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.682+0000 7f841a59c640 1 --2- 172.21.15.1:0/2784610109 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f841c14cdc0 0x7f841c1dda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.682+0000 7f841a59c640 1 -- 172.21.15.1:0/2784610109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f841c156500 msgr2=0x7f841c1ddfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.682+0000 7f841a59c640 1 --2- 172.21.15.1:0/2784610109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f841c156500 0x7f841c1ddfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.682+0000 7f841a59c640 1 -- 172.21.15.1:0/2784610109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841c1defa0 con 0x7f841c06b510 2024-03-06T07:26:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.683+0000 7f841a59c640 1 --2- 172.21.15.1:0/2784610109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f841c06b510 0x7f841c0a8050 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f84140098c0 tx=0x7f84140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.683+0000 7f840affd640 1 -- 172.21.15.1:0/2784610109 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8414035040 con 0x7f841c06b510 2024-03-06T07:26:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.683+0000 7f840affd640 1 -- 172.21.15.1:0/2784610109 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841403b860 con 0x7f841c06b510 2024-03-06T07:26:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.683+0000 7f840affd640 1 -- 172.21.15.1:0/2784610109 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841403bc50 con 0x7f841c06b510 2024-03-06T07:26:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.683+0000 7f8420ef4640 1 -- 172.21.15.1:0/2784610109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f841c1df280 con 0x7f841c06b510 2024-03-06T07:26:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.683+0000 7f8420ef4640 1 -- 172.21.15.1:0/2784610109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f841c0a8590 con 0x7f841c06b510 2024-03-06T07:26:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.684+0000 7f840affd640 1 -- 172.21.15.1:0/2784610109 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8414038020 con 0x7f841c06b510 2024-03-06T07:26:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.685+0000 7f840affd640 1 --2- 172.21.15.1:0/2784610109 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f83fc06b4d0 0x7f83fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.685+0000 7f840affd640 1 -- 172.21.15.1:0/2784610109 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f84140ac2f0 con 0x7f841c06b510 2024-03-06T07:26:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.685+0000 7f8419d9b640 1 --2- 172.21.15.1:0/2784610109 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f83fc06b4d0 0x7f83fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.685+0000 7f8419d9b640 1 --2- 172.21.15.1:0/2784610109 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f83fc06b4d0 0x7f83fc06d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f841c0c4a00 tx=0x7f8410009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.686+0000 7f8423bba740 1 Processor -- start 2024-03-06T07:26:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.686+0000 7f8423bba740 1 -- start start 2024-03-06T07:26:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.686+0000 7f8423bba740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0caca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.686+0000 7f8423bba740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caf330 0x5583e0cacf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.686+0000 7f8423bba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0cad4b0 0x5583e0ca9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.686+0000 7f8423bba740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e0c136b0 con 0x5583e0caef50 2024-03-06T07:26:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.686+0000 7f8423bba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e0bbca60 con 0x5583e0cad4b0 2024-03-06T07:26:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.686+0000 7f8423bba740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5583e0c0f750 con 0x5583e0caf330 2024-03-06T07:26:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.687+0000 7f841a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0caca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.687+0000 7f841a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0caca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42486/0 (socket says 172.21.15.1:42486) 2024-03-06T07:26:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.687+0000 7f841a59c640 1 -- 172.21.15.1:0/2724266849 learned_addr learned my addr 172.21.15.1:0/2724266849 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.687+0000 7f8419d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caf330 0x5583e0cacf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.687+0000 7f841ad9d640 1 --2- 172.21.15.1:0/2724266849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0cad4b0 0x5583e0ca9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.687+0000 7f841a59c640 1 -- 172.21.15.1:0/2724266849 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caf330 msgr2=0x5583e0cacf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.687+0000 7f841a59c640 1 --2- 172.21.15.1:0/2724266849 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caf330 0x5583e0cacf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.687+0000 7f841a59c640 1 -- 172.21.15.1:0/2724266849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0cad4b0 msgr2=0x5583e0ca9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.687+0000 7f841a59c640 1 --2- 172.21.15.1:0/2724266849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0cad4b0 0x5583e0ca9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.687+0000 7f841a59c640 1 -- 172.21.15.1:0/2724266849 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583e0bdaf30 con 0x5583e0caef50 2024-03-06T07:26:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.687+0000 7f8419d9b640 1 --2- 172.21.15.1:0/2724266849 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caf330 0x5583e0cacf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.687+0000 7f841a59c640 1 --2- 172.21.15.1:0/2724266849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0caca30 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f8414082aa0 tx=0x7f8414009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae8388fdc13cff1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.687+0000 7f83fbfff640 1 -- 172.21.15.1:0/2724266849 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841407a8c0 con 0x5583e0caef50 2024-03-06T07:26:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.687+0000 7f83fbfff640 1 -- 172.21.15.1:0/2724266849 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841407ea10 con 0x5583e0caef50 2024-03-06T07:26:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.687+0000 7f83fbfff640 1 -- 172.21.15.1:0/2724266849 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8414082fd0 con 0x5583e0caef50 2024-03-06T07:26:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.688+0000 7f8423bba740 1 -- 172.21.15.1:0/2724266849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 msgr2=0x5583e0caca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.688+0000 7f8423bba740 1 --2- 172.21.15.1:0/2724266849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0caca30 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f8414082aa0 tx=0x7f8414009190 comp rx=0 tx=0).stop 2024-03-06T07:26:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.688+0000 7f8423bba740 1 -- 172.21.15.1:0/2724266849 shutdown_connections 2024-03-06T07:26:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.688+0000 7f8423bba740 1 --2- 172.21.15.1:0/2724266849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0cad4b0 0x5583e0ca9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.688+0000 7f8423bba740 1 --2- 172.21.15.1:0/2724266849 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caf330 0x5583e0cacf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.688+0000 7f8423bba740 1 --2- 172.21.15.1:0/2724266849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0caca30 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.688+0000 7f8423bba740 1 -- 172.21.15.1:0/2724266849 >> 172.21.15.1:0/2724266849 conn(0x5583e0cab7d0 msgr2=0x5583e0cd03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.688+0000 7f8423bba740 1 -- 172.21.15.1:0/2724266849 shutdown_connections 2024-03-06T07:26:00.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.688+0000 7f8423bba740 1 -- 172.21.15.1:0/2724266849 wait complete. 2024-03-06T07:26:00.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.688+0000 7f8423bba740 1 -- start start 2024-03-06T07:26:00.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.689+0000 7f8423bba740 1 Processor -- start 2024-03-06T07:26:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.689+0000 7f8423bba740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0ce44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.689+0000 7f8423bba740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caf330 0x5583e0ce3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.689+0000 7f8423bba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0ce5870 0x5583e0ce8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.689+0000 7f8423bba740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e0c136b0 con 0x5583e0caef50 2024-03-06T07:26:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.689+0000 7f8423bba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e0bcfea0 con 0x5583e0ce5870 2024-03-06T07:26:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.689+0000 7f8423bba740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5583e0cd1230 con 0x5583e0caf330 2024-03-06T07:26:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.689+0000 7f8423bba740 10 client.0 _ll_register_callbacks cb 0x5583e0bcf870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.689+0000 7f841a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0ce44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.689+0000 7f841a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0ce44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42498/0 (socket says 172.21.15.1:42498) 2024-03-06T07:26:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f841a59c640 1 -- 172.21.15.1:0/298608375 learned_addr learned my addr 172.21.15.1:0/298608375 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f8419d9b640 1 --2- 172.21.15.1:0/298608375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caf330 0x5583e0ce3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f841ad9d640 1 --2- 172.21.15.1:0/298608375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0ce5870 0x5583e0ce8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f841a59c640 1 -- 172.21.15.1:0/298608375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caf330 msgr2=0x5583e0ce3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f841a59c640 1 --2- 172.21.15.1:0/298608375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caf330 0x5583e0ce3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f841a59c640 1 -- 172.21.15.1:0/298608375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0ce5870 msgr2=0x5583e0ce8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f841a59c640 1 --2- 172.21.15.1:0/298608375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0ce5870 0x5583e0ce8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f841a59c640 1 -- 172.21.15.1:0/298608375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5583e0bd0270 con 0x5583e0caef50 2024-03-06T07:26:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f8419d9b640 1 --2- 172.21.15.1:0/298608375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caf330 0x5583e0ce3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f841a59c640 1 --2- 172.21.15.1:0/298608375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0ce44e0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f841402b910 tx=0x7f8414031f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f154c2be261be2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f83f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:26:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f83f97fa640 1 -- 172.21.15.1:0/298608375 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8414097060 con 0x5583e0caef50 2024-03-06T07:26:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f83f97fa640 1 -- 172.21.15.1:0/298608375 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f8414078bc0 con 0x5583e0caef50 2024-03-06T07:26:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f8423bba740 1 -- 172.21.15.1:0/298608375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f841407ea10 con 0x5583e0caef50 2024-03-06T07:26:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f83f97fa640 1 -- 172.21.15.1:0/298608375 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=131) v2 ==== 24+0+0 (secure 0 0 0) 0x7f841407a8c0 con 0x5583e0caef50 2024-03-06T07:26:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f8423bba740 10 client.8625 fetch_fsmap learned FSMap version 131 2024-03-06T07:26:00.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f8423bba740 1 -- 172.21.15.1:0/298608375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=131,monmap=2+}) v3 -- 0x5583e09282d0 con 0x5583e0caef50 2024-03-06T07:26:00.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.690+0000 7f83f97fa640 1 -- 172.21.15.1:0/298608375 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841407ea10 con 0x5583e0caef50 2024-03-06T07:26:00.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.691+0000 7f83f97fa640 1 -- 172.21.15.1:0/298608375 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 131) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8414099350 con 0x5583e0caef50 2024-03-06T07:26:00.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.691+0000 7f8423bba740 10 client.8625 fetch_fsmap finished waiting for FSMap version 131 2024-03-06T07:26:00.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.691+0000 7f8423bba740 10 client.8625 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:26:00.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.691+0000 7f8423bba740 20 client.8625 populate_metadata read hostname 'smithi001' 2024-03-06T07:26:00.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.691+0000 7f8423bba740 1 --2- 172.21.15.1:0/298608375 >> [v2:172.21.15.98:6835/1671554096,v1:172.21.15.98:6836/1671554096] conn(0x5583e0cff080 0x5583e0cff460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.691+0000 7f8423bba740 4 client.8625 mds_command: new command op to 7387 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:00.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.691+0000 7f8423bba740 1 -- 172.21.15.1:0/298608375 --> [v2:172.21.15.98:6835/1671554096,v1:172.21.15.98:6836/1671554096] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5583e0c20bf0 con 0x5583e0cff080 2024-03-06T07:26:00.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.691+0000 7f8419d9b640 1 --2- 172.21.15.1:0/298608375 >> [v2:172.21.15.98:6835/1671554096,v1:172.21.15.98:6836/1671554096] conn(0x5583e0cff080 0x5583e0cff460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.691+0000 7f83f97fa640 10 client.8625 ms_handle_connect on v2:172.21.15.98:6835/1671554096 2024-03-06T07:26:00.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.691+0000 7f8419d9b640 1 --2- 172.21.15.1:0/298608375 >> [v2:172.21.15.98:6835/1671554096,v1:172.21.15.98:6836/1671554096] conn(0x5583e0cff080 0x5583e0cff460 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdbd576ebc7be2e7 server_cookie=2385e34ac8037ac5 in_seq=0 out_seq=0 2024-03-06T07:26:00.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.692+0000 7f83f97fa640 1 -- 172.21.15.1:0/298608375 <== mds.0 v2:172.21.15.98:6835/1671554096 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5583e0c20bf0 con 0x5583e0cff080 2024-03-06T07:26:00.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.692+0000 7f83f97fa640 10 client.8625 handle_command_reply: tid=0 2024-03-06T07:26:00.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.692+0000 7f8423bba740 1 client.8625 shutdown 2024-03-06T07:26:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.692+0000 7f8423bba740 1 -- 172.21.15.1:0/298608375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 msgr2=0x5583e0ce44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.692+0000 7f8423bba740 1 --2- 172.21.15.1:0/298608375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0ce44e0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f841402b910 tx=0x7f8414031f90 comp rx=0 tx=0).stop 2024-03-06T07:26:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.692+0000 7f8423bba740 1 -- 172.21.15.1:0/298608375 shutdown_connections 2024-03-06T07:26:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.692+0000 7f8423bba740 1 --2- 172.21.15.1:0/298608375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0ce5870 0x5583e0ce8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.692+0000 7f8423bba740 1 --2- 172.21.15.1:0/298608375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caf330 0x5583e0ce3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.692+0000 7f8423bba740 1 --2- 172.21.15.1:0/298608375 >> [v2:172.21.15.98:6835/1671554096,v1:172.21.15.98:6836/1671554096] conn(0x5583e0cff080 0x5583e0cff460 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.692+0000 7f8423bba740 1 --2- 172.21.15.1:0/298608375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0ce44e0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.692+0000 7f83f97fa640 0 client.8625 ms_handle_reset on v2:172.21.15.98:6835/1671554096 2024-03-06T07:26:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.692+0000 7f8423bba740 1 -- 172.21.15.1:0/298608375 >> 172.21.15.1:0/298608375 conn(0x5583e0cad4b0 msgr2=0x5583e0cadb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.692+0000 7f8423bba740 1 -- 172.21.15.1:0/298608375 shutdown_connections 2024-03-06T07:26:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.692+0000 7f8423bba740 1 -- 172.21.15.1:0/298608375 wait complete. 2024-03-06T07:26:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.692+0000 7f8423bba740 20 client.8625 trim_cache size 0 max 16384 2024-03-06T07:26:00.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.709+0000 7f8423bba740 1 Processor -- start 2024-03-06T07:26:00.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.709+0000 7f8423bba740 1 -- start start 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f8423bba740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0caa7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f8423bba740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caf330 0x5583e0cff460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f8423bba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0ce5870 0x5583e0cff9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f8423bba740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c1f4aa0 con 0x5583e0caef50 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f8423bba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c1ecde0 con 0x5583e0ce5870 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f8423bba740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f841c14bc70 con 0x5583e0caf330 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f841a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0caa7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f841a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0caa7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42508/0 (socket says 172.21.15.1:42508) 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f841a59c640 1 -- 172.21.15.1:0/3964162656 learned_addr learned my addr 172.21.15.1:0/3964162656 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f8419d9b640 1 --2- 172.21.15.1:0/3964162656 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caf330 0x5583e0cff460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f841ad9d640 1 --2- 172.21.15.1:0/3964162656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0ce5870 0x5583e0cff9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f841a59c640 1 -- 172.21.15.1:0/3964162656 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caf330 msgr2=0x5583e0cff460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f841a59c640 1 --2- 172.21.15.1:0/3964162656 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caf330 0x5583e0cff460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f841a59c640 1 -- 172.21.15.1:0/3964162656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0ce5870 msgr2=0x5583e0cff9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f841a59c640 1 --2- 172.21.15.1:0/3964162656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0ce5870 0x5583e0cff9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f841a59c640 1 -- 172.21.15.1:0/3964162656 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583e09f26b0 con 0x5583e0caef50 2024-03-06T07:26:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f841a59c640 1 --2- 172.21.15.1:0/3964162656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0caa7b0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f841402be60 tx=0x7f84140a5070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82a26aeadfe3a38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f83fb7fe640 1 -- 172.21.15.1:0/3964162656 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841407ea10 con 0x5583e0caef50 2024-03-06T07:26:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f83fb7fe640 1 -- 172.21.15.1:0/3964162656 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84140a7030 con 0x5583e0caef50 2024-03-06T07:26:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.710+0000 7f83fb7fe640 1 -- 172.21.15.1:0/3964162656 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8414078bc0 con 0x5583e0caef50 2024-03-06T07:26:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.711+0000 7f8423bba740 1 -- 172.21.15.1:0/3964162656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 msgr2=0x5583e0caa7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.711+0000 7f8423bba740 1 --2- 172.21.15.1:0/3964162656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0caa7b0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f841402be60 tx=0x7f84140a5070 comp rx=0 tx=0).stop 2024-03-06T07:26:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.711+0000 7f8423bba740 1 -- 172.21.15.1:0/3964162656 shutdown_connections 2024-03-06T07:26:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.711+0000 7f8423bba740 1 --2- 172.21.15.1:0/3964162656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0ce5870 0x5583e0cff9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.711+0000 7f8423bba740 1 --2- 172.21.15.1:0/3964162656 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caf330 0x5583e0cff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.711+0000 7f8423bba740 1 --2- 172.21.15.1:0/3964162656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caef50 0x5583e0caa7b0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.711+0000 7f8423bba740 1 -- 172.21.15.1:0/3964162656 >> 172.21.15.1:0/3964162656 conn(0x5583e0cad4b0 msgr2=0x5583e0ce8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.711+0000 7f8423bba740 1 -- 172.21.15.1:0/3964162656 shutdown_connections 2024-03-06T07:26:00.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.711+0000 7f8423bba740 1 -- 172.21.15.1:0/3964162656 wait complete. 2024-03-06T07:26:00.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.711+0000 7f8423bba740 1 -- start start 2024-03-06T07:26:00.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.712+0000 7f8423bba740 1 Processor -- start 2024-03-06T07:26:00.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.712+0000 7f8423bba740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caef50 0x5583e0cff860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.712+0000 7f8423bba740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caf330 0x5583e0cf7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.712+0000 7f8423bba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0ce5870 0x5583e0ce0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.712+0000 7f8423bba740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e0bcfea0 con 0x5583e0caf330 2024-03-06T07:26:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.712+0000 7f8423bba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e0ce8ca0 con 0x5583e0ce5870 2024-03-06T07:26:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.712+0000 7f8423bba740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5583e0c136b0 con 0x5583e0caef50 2024-03-06T07:26:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.712+0000 7f8423bba740 10 client.0 _ll_register_callbacks cb 0x7f8414006a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.712+0000 7f8419d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caf330 0x5583e0cf7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.712+0000 7f8419d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caf330 0x5583e0cf7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42520/0 (socket says 172.21.15.1:42520) 2024-03-06T07:26:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.712+0000 7f8419d9b640 1 -- 172.21.15.1:0/3502892474 learned_addr learned my addr 172.21.15.1:0/3502892474 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.712+0000 7f841a59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caef50 0x5583e0cff860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.713+0000 7f841ad9d640 1 --2- 172.21.15.1:0/3502892474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0ce5870 0x5583e0ce0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.713+0000 7f8419d9b640 1 -- 172.21.15.1:0/3502892474 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caef50 msgr2=0x5583e0cff860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.713+0000 7f8419d9b640 1 --2- 172.21.15.1:0/3502892474 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caef50 0x5583e0cff860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.713+0000 7f8419d9b640 1 -- 172.21.15.1:0/3502892474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0ce5870 msgr2=0x5583e0ce0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.713+0000 7f8419d9b640 1 --2- 172.21.15.1:0/3502892474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0ce5870 0x5583e0ce0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.713+0000 7f8419d9b640 1 -- 172.21.15.1:0/3502892474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5583e0cacf00 con 0x5583e0caf330 2024-03-06T07:26:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.713+0000 7f841a59c640 1 --2- 172.21.15.1:0/3502892474 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caef50 0x5583e0cff860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.713+0000 7f8419d9b640 1 --2- 172.21.15.1:0/3502892474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caf330 0x5583e0cf7a20 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f841c0ab230 tx=0x7f84100010a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10a8b25da409ba44 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.713+0000 7f83f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:26:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.713+0000 7f83f97fa640 1 -- 172.21.15.1:0/3502892474 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410028b50 con 0x5583e0caf330 2024-03-06T07:26:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.713+0000 7f83f97fa640 1 -- 172.21.15.1:0/3502892474 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f8410010040 con 0x5583e0caf330 2024-03-06T07:26:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.713+0000 7f8423bba740 1 -- 172.21.15.1:0/3502892474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f84140a7030 con 0x5583e0caf330 2024-03-06T07:26:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.713+0000 7f83f97fa640 1 -- 172.21.15.1:0/3502892474 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=131) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8410039050 con 0x5583e0caf330 2024-03-06T07:26:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.713+0000 7f8423bba740 10 client.8631 fetch_fsmap learned FSMap version 131 2024-03-06T07:26:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.713+0000 7f8423bba740 1 -- 172.21.15.1:0/3502892474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=131,monmap=2+}) v3 -- 0x5583e0cd3ae0 con 0x5583e0caf330 2024-03-06T07:26:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.713+0000 7f83f97fa640 1 -- 172.21.15.1:0/3502892474 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84140a7030 con 0x5583e0caf330 2024-03-06T07:26:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f83f97fa640 1 -- 172.21.15.1:0/3502892474 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 131) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f841002f020 con 0x5583e0caf330 2024-03-06T07:26:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f8423bba740 10 client.8631 fetch_fsmap finished waiting for FSMap version 131 2024-03-06T07:26:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f8423bba740 10 client.8631 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:26:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f8423bba740 20 client.8631 populate_metadata read hostname 'smithi001' 2024-03-06T07:26:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f8423bba740 1 --2- 172.21.15.1:0/3502892474 >> [v2:172.21.15.98:6835/1671554096,v1:172.21.15.98:6836/1671554096] conn(0x5583e0d5b9d0 0x5583e0d5bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f8423bba740 4 client.8631 mds_command: new command op to 7387 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f8423bba740 1 -- 172.21.15.1:0/3502892474 --> [v2:172.21.15.98:6835/1671554096,v1:172.21.15.98:6836/1671554096] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5583e0c1af40 con 0x5583e0d5b9d0 2024-03-06T07:26:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f841a59c640 1 --2- 172.21.15.1:0/3502892474 >> [v2:172.21.15.98:6835/1671554096,v1:172.21.15.98:6836/1671554096] conn(0x5583e0d5b9d0 0x5583e0d5bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f83f97fa640 10 client.8631 ms_handle_connect on v2:172.21.15.98:6835/1671554096 2024-03-06T07:26:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f841a59c640 1 --2- 172.21.15.1:0/3502892474 >> [v2:172.21.15.98:6835/1671554096,v1:172.21.15.98:6836/1671554096] conn(0x5583e0d5b9d0 0x5583e0d5bdb0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ce7a227e2f412f server_cookie=370448da188558bf in_seq=0 out_seq=0 2024-03-06T07:26:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f83f97fa640 1 -- 172.21.15.1:0/3502892474 <== mds.0 v2:172.21.15.98:6835/1671554096 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x5583e0c1af40 con 0x5583e0d5b9d0 2024-03-06T07:26:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f83f97fa640 10 client.8631 handle_command_reply: tid=0 2024-03-06T07:26:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f8423bba740 1 client.8631 shutdown 2024-03-06T07:26:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f8423bba740 1 -- 172.21.15.1:0/3502892474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caf330 msgr2=0x5583e0cf7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f8423bba740 1 --2- 172.21.15.1:0/3502892474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caf330 0x5583e0cf7a20 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f841c0ab230 tx=0x7f84100010a0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f8423bba740 1 -- 172.21.15.1:0/3502892474 shutdown_connections 2024-03-06T07:26:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f8423bba740 1 --2- 172.21.15.1:0/3502892474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5583e0ce5870 0x5583e0ce0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f8423bba740 1 --2- 172.21.15.1:0/3502892474 >> [v2:172.21.15.98:6835/1671554096,v1:172.21.15.98:6836/1671554096] conn(0x5583e0d5b9d0 0x5583e0d5bdb0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f8423bba740 1 --2- 172.21.15.1:0/3502892474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5583e0caf330 0x5583e0cf7a20 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f8423bba740 1 --2- 172.21.15.1:0/3502892474 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5583e0caef50 0x5583e0cff860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f83f97fa640 0 client.8631 ms_handle_reset on v2:172.21.15.98:6835/1671554096 2024-03-06T07:26:00.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.714+0000 7f8423bba740 1 -- 172.21.15.1:0/3502892474 >> 172.21.15.1:0/3502892474 conn(0x5583e0cad4b0 msgr2=0x5583e0caa790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.715+0000 7f8423bba740 1 -- 172.21.15.1:0/3502892474 shutdown_connections 2024-03-06T07:26:00.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.715+0000 7f8423bba740 1 -- 172.21.15.1:0/3502892474 wait complete. 2024-03-06T07:26:00.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.715+0000 7f8423bba740 20 client.8631 trim_cache size 0 max 16384 2024-03-06T07:26:00.723 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:26:00.723 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:26:00.723 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:26:00.723 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 7387, 2024-03-06T07:26:00.723 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:26:00.723 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:26:00.724 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:26:00.724 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 4.8506273359999996, 2024-03-06T07:26:00.724 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 131, 2024-03-06T07:26:00.724 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 137, 2024-03-06T07:26:00.724 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 137, 2024-03-06T07:26:00.724 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 35.362534613000001 2024-03-06T07:26:00.724 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:26:00.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.715+0000 7f8420ef4640 1 -- 172.21.15.1:0/2784610109 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f83fc06b4d0 msgr2=0x7f83fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.715+0000 7f8420ef4640 1 --2- 172.21.15.1:0/2784610109 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f83fc06b4d0 0x7f83fc06d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f841c0c4a00 tx=0x7f8410009040 comp rx=0 tx=0).stop 2024-03-06T07:26:00.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.715+0000 7f8420ef4640 1 -- 172.21.15.1:0/2784610109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f841c06b510 msgr2=0x7f841c0a8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.715+0000 7f8420ef4640 1 --2- 172.21.15.1:0/2784610109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f841c06b510 0x7f841c0a8050 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f84140098c0 tx=0x7f84140046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.715+0000 7f8420ef4640 1 -- 172.21.15.1:0/2784610109 shutdown_connections 2024-03-06T07:26:00.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.715+0000 7f8420ef4640 1 --2- 172.21.15.1:0/2784610109 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f83fc06b4d0 0x7f83fc06d990 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.715+0000 7f8420ef4640 1 --2- 172.21.15.1:0/2784610109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f841c156500 0x7f841c1ddfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.715+0000 7f8420ef4640 1 --2- 172.21.15.1:0/2784610109 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f841c14cdc0 0x7f841c1dda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.715+0000 7f8420ef4640 1 --2- 172.21.15.1:0/2784610109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f841c06b510 0x7f841c0a8050 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.715+0000 7f8420ef4640 1 -- 172.21.15.1:0/2784610109 >> 172.21.15.1:0/2784610109 conn(0x7f841c0a9700 msgr2=0x7f841c154a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.715+0000 7f8420ef4640 1 -- 172.21.15.1:0/2784610109 shutdown_connections 2024-03-06T07:26:00.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.715+0000 7f8420ef4640 1 -- 172.21.15.1:0/2784610109 wait complete. 2024-03-06T07:26:00.726 INFO:tasks.ceph.mds.d.smithi098.stderr:2024-03-06T07:26:00.715+0000 7fc40a923640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:00.728 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 131, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:25:29.792840+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 131, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7387": { "addr": "172.21.15.98:6836/1671554096", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 1671554096, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 1671554096, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7387, "incarnation": 128, "join_fscid": 10, "name": "d", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 137, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-03-06T07:25:58.868171+0000", "qdb_cluster": [ 7387 ], "qdb_leader": 7387, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7387 } } } ], "standbys": [ { "addr": "172.21.15.98:6837/2739580331", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 2739580331, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2739580331, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 118, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7390, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6834/3374845636", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 3374845636, "type": "v2" }, { "addr": "172.21.15.1:6834", "nonce": 3374845636, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 124, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8517, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6835/1602915919", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 1602915919, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 1602915919, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 129, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8607, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:26:00.729 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 10, 'name': 'd', 'state': 'up:active'}] 2024-03-06T07:26:00.729 INFO:tasks.cephfs.test_failover:current = [{'gid': 8517, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6834/3374845636', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 3374845636}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 3374845636}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 124}, {'gid': 7390, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/2739580331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2739580331}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2739580331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 118}, {'gid': 8607, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/1602915919', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1602915919}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1602915919}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 129}, {'gid': 7387, 'name': 'd', 'rank': 0, 'incarnation': 128, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.98:6836/1671554096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 1671554096}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 1671554096}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-03-06T07:26:00.729 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:26:00.729 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:00.729 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:00.781 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:00.826 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:00.826 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:26:00.826 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:00.841 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:26:00.841 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:26:00.841 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:00.855 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:26:00.855 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:00.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.979+0000 7f9e257d7640 1 Processor -- start 2024-03-06T07:26:00.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.979+0000 7f9e257d7640 1 -- start start 2024-03-06T07:26:00.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.980+0000 7f9e257d7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9e200c6b10 0x7f9e20155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.980+0000 7f9e257d7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e200c2fc0 con 0x7f9e20152650 2024-03-06T07:26:00.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.980+0000 7f9e257d7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e20158da0 con 0x7f9e200bf9d0 2024-03-06T07:26:00.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.980+0000 7f9e257d7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e20155800 con 0x7f9e200c6b10 2024-03-06T07:26:00.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.980+0000 7f9e1effd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9e20152650 0x7f9e20158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38290/0 (socket says 172.21.15.1:38290) 2024-03-06T07:26:00.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.980+0000 7f9e1effd640 1 -- 172.21.15.1:0/3651353405 learned_addr learned my addr 172.21.15.1:0/3651353405 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:00.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.980+0000 7f9e17fff640 1 --2- 172.21.15.1:0/3651353405 >> v2:172.21.15.98:3300/0 conn(0x7f9e200c6b10 0x7f9e20155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.981+0000 7f9e1e7fc640 1 -- 172.21.15.1:0/3651353405 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298539340 0 0) 0x7f9e200c2fc0 con 0x7f9e20152650 2024-03-06T07:26:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.981+0000 7f9e1e7fc640 1 -- 172.21.15.1:0/3651353405 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e0c003520 con 0x7f9e20152650 2024-03-06T07:26:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.981+0000 7f9e1e7fc640 1 -- 172.21.15.1:0/3651353405 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231275621 0 0) 0x7f9e20158da0 con 0x7f9e200bf9d0 2024-03-06T07:26:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.981+0000 7f9e1e7fc640 1 -- 172.21.15.1:0/3651353405 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e200c2fc0 con 0x7f9e200bf9d0 2024-03-06T07:26:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.981+0000 7f9e17fff640 1 -- 172.21.15.1:0/3651353405 >> v1:172.21.15.98:6789/0 conn(0x7f9e200bf9d0 legacy=0x7f9e200c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.981+0000 7f9e17fff640 1 -- 172.21.15.1:0/3651353405 >> v1:172.21.15.1:6789/0 conn(0x7f9e20152650 legacy=0x7f9e20158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.981+0000 7f9e17fff640 1 -- 172.21.15.1:0/3651353405 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e200c6690 con 0x7f9e200c6b10 2024-03-06T07:26:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.982+0000 7f9e17fff640 1 --2- 172.21.15.1:0/3651353405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e200c6b10 0x7f9e20155c90 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9e080097a0 tx=0x7f9e0802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9bc8ed0ad990720 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.982+0000 7f9e1e7fc640 1 -- 172.21.15.1:0/3651353405 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e08035020 con 0x7f9e200c6b10 2024-03-06T07:26:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.982+0000 7f9e1e7fc640 1 -- 172.21.15.1:0/3651353405 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e08031830 con 0x7f9e200c6b10 2024-03-06T07:26:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.982+0000 7f9e1e7fc640 1 -- 172.21.15.1:0/3651353405 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e08031be0 con 0x7f9e200c6b10 2024-03-06T07:26:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.982+0000 7f9e257d7640 1 -- 172.21.15.1:0/3651353405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e200c6b10 msgr2=0x7f9e20155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.982+0000 7f9e257d7640 1 --2- 172.21.15.1:0/3651353405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e200c6b10 0x7f9e20155c90 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9e080097a0 tx=0x7f9e0802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.982+0000 7f9e257d7640 1 -- 172.21.15.1:0/3651353405 shutdown_connections 2024-03-06T07:26:00.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.982+0000 7f9e257d7640 1 --2- 172.21.15.1:0/3651353405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e200c6b10 0x7f9e20155c90 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.982+0000 7f9e257d7640 1 -- 172.21.15.1:0/3651353405 >> 172.21.15.1:0/3651353405 conn(0x7f9e20066ee0 msgr2=0x7f9e200672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.982+0000 7f9e257d7640 1 -- 172.21.15.1:0/3651353405 shutdown_connections 2024-03-06T07:26:00.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.982+0000 7f9e257d7640 1 -- 172.21.15.1:0/3651353405 wait complete. 2024-03-06T07:26:00.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.983+0000 7f9e257d7640 1 Processor -- start 2024-03-06T07:26:00.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.984+0000 7f9e257d7640 1 -- start start 2024-03-06T07:26:00.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.984+0000 7f9e257d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e200bf9d0 0x7f9e200c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.984+0000 7f9e257d7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e20152650 0x7f9e2013d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.984+0000 7f9e257d7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e201ee620 0x7f9e2013d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.984+0000 7f9e257d7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e201551b0 con 0x7f9e201ee620 2024-03-06T07:26:00.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.984+0000 7f9e257d7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e200c6fa0 con 0x7f9e200bf9d0 2024-03-06T07:26:00.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.984+0000 7f9e257d7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e20155490 con 0x7f9e20152650 2024-03-06T07:26:00.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.984+0000 7f9e1f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e201ee620 0x7f9e2013d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.984+0000 7f9e1f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e201ee620 0x7f9e2013d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42534/0 (socket says 172.21.15.1:42534) 2024-03-06T07:26:00.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.984+0000 7f9e1f7fe640 1 -- 172.21.15.1:0/571871733 learned_addr learned my addr 172.21.15.1:0/571871733 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:00.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.984+0000 7f9e1effd640 1 --2- 172.21.15.1:0/571871733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e200bf9d0 0x7f9e200c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.984+0000 7f9e17fff640 1 --2- 172.21.15.1:0/571871733 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e20152650 0x7f9e2013d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.985+0000 7f9e1f7fe640 1 -- 172.21.15.1:0/571871733 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e20152650 msgr2=0x7f9e2013d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.985+0000 7f9e1f7fe640 1 --2- 172.21.15.1:0/571871733 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e20152650 0x7f9e2013d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.985+0000 7f9e1f7fe640 1 -- 172.21.15.1:0/571871733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e200bf9d0 msgr2=0x7f9e200c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.985+0000 7f9e1f7fe640 1 --2- 172.21.15.1:0/571871733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e200bf9d0 0x7f9e200c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.985+0000 7f9e1f7fe640 1 -- 172.21.15.1:0/571871733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e201eb0b0 con 0x7f9e201ee620 2024-03-06T07:26:00.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.985+0000 7f9e1f7fe640 1 --2- 172.21.15.1:0/571871733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e201ee620 0x7f9e2013d6d0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800c6e0 tx=0x7f9e1800cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.985+0000 7f9e177fe640 1 -- 172.21.15.1:0/571871733 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18003b70 con 0x7f9e201ee620 2024-03-06T07:26:00.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.985+0000 7f9e177fe640 1 -- 172.21.15.1:0/571871733 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e18003d10 con 0x7f9e201ee620 2024-03-06T07:26:00.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.985+0000 7f9e257d7640 1 -- 172.21.15.1:0/571871733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e201eeb20 con 0x7f9e201ee620 2024-03-06T07:26:00.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.985+0000 7f9e177fe640 1 -- 172.21.15.1:0/571871733 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e180052d0 con 0x7f9e201ee620 2024-03-06T07:26:00.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.985+0000 7f9e257d7640 1 -- 172.21.15.1:0/571871733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e201ef0e0 con 0x7f9e201ee620 2024-03-06T07:26:00.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.986+0000 7f9e257d7640 1 -- 172.21.15.1:0/571871733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e20067fd0 con 0x7f9e201ee620 2024-03-06T07:26:00.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.986+0000 7f9e177fe640 1 -- 172.21.15.1:0/571871733 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e18005470 con 0x7f9e201ee620 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.987+0000 7f9e177fe640 1 --2- 172.21.15.1:0/571871733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e0006b500 0x7f9e0006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.987+0000 7f9e177fe640 1 -- 172.21.15.1:0/571871733 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f9e1808a0c0 con 0x7f9e201ee620 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.987+0000 7f9e1effd640 1 --2- 172.21.15.1:0/571871733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e0006b500 0x7f9e0006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.987+0000 7f9e1effd640 1 --2- 172.21.15.1:0/571871733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e0006b500 0x7f9e0006d9c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9e100011e0 tx=0x7f9e10007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:00.991+0000 7f9e177fe640 1 -- 172.21.15.1:0/571871733 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e18058740 con 0x7f9e201ee620 2024-03-06T07:26:01.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.210+0000 7f9e257d7640 1 -- 172.21.15.1:0/571871733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e2006b500 con 0x7f9e201ee620 2024-03-06T07:26:01.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.211+0000 7f9e177fe640 1 -- 172.21.15.1:0/571871733 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+4984 (secure 0 0 0) 0x7f9e1805c7e0 con 0x7f9e201ee620 2024-03-06T07:26:01.213 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:01.213 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":131,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2739580331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2739580331},{"type":"v1","addr":"172.21.15.98:6837","nonce":2739580331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":8517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3374845636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3374845636},{"type":"v1","addr":"172.21.15.1:6834","nonce":3374845636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":8607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1602915919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1602915919},{"type":"v1","addr":"172.21.15.1:6835","nonce":1602915919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:29.792840+0000","modified":"2024-03-06T07:25:58.868171+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7387},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.98:6836/1671554096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1671554096},{"type":"v1","addr":"172.21.15.98:6836","nonce":1671554096}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7387,"qdb_cluster":[7387]},"id":10}]} 2024-03-06T07:26:01.214 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 131 2024-03-06T07:26:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.213+0000 7f9e257d7640 1 -- 172.21.15.1:0/571871733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e0006b500 msgr2=0x7f9e0006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.213+0000 7f9e257d7640 1 --2- 172.21.15.1:0/571871733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e0006b500 0x7f9e0006d9c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9e100011e0 tx=0x7f9e10007d30 comp rx=0 tx=0).stop 2024-03-06T07:26:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.213+0000 7f9e257d7640 1 -- 172.21.15.1:0/571871733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e201ee620 msgr2=0x7f9e2013d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.213+0000 7f9e257d7640 1 --2- 172.21.15.1:0/571871733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e201ee620 0x7f9e2013d6d0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800c6e0 tx=0x7f9e1800cba0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.213+0000 7f9e257d7640 1 -- 172.21.15.1:0/571871733 shutdown_connections 2024-03-06T07:26:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.213+0000 7f9e257d7640 1 --2- 172.21.15.1:0/571871733 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e0006b500 0x7f9e0006d9c0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.213+0000 7f9e257d7640 1 --2- 172.21.15.1:0/571871733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e201ee620 0x7f9e2013d6d0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.213+0000 7f9e257d7640 1 --2- 172.21.15.1:0/571871733 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e20152650 0x7f9e2013d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.213+0000 7f9e257d7640 1 --2- 172.21.15.1:0/571871733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e200bf9d0 0x7f9e200c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.213+0000 7f9e257d7640 1 -- 172.21.15.1:0/571871733 >> 172.21.15.1:0/571871733 conn(0x7f9e20066ee0 msgr2=0x7f9e200c3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.213+0000 7f9e257d7640 1 -- 172.21.15.1:0/571871733 shutdown_connections 2024-03-06T07:26:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.213+0000 7f9e257d7640 1 -- 172.21.15.1:0/571871733 wait complete. 2024-03-06T07:26:01.226 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:01.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.326+0000 7f47dd84e640 1 Processor -- start 2024-03-06T07:26:01.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.326+0000 7f47dd84e640 1 -- start start 2024-03-06T07:26:01.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.327+0000 7f47dd84e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f47d80c5850 0x7f47d80c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.327+0000 7f47dd84e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d8155220 0x7f47d81556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.327+0000 7f47dd84e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d8158910 con 0x7f47d8152920 2024-03-06T07:26:01.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.327+0000 7f47dd84e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47d8152d00 con 0x7f47d80c5850 2024-03-06T07:26:01.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.327+0000 7f47dd84e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d81596d0 con 0x7f47d8155220 2024-03-06T07:26:01.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.327+0000 7f47d6ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f47d8152920 0x7f47d806aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38304/0 (socket says 172.21.15.1:38304) 2024-03-06T07:26:01.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.327+0000 7f47d6ffd640 1 -- 172.21.15.1:0/543543491 learned_addr learned my addr 172.21.15.1:0/543543491 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:01.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.327+0000 7f47d77fe640 1 --2- 172.21.15.1:0/543543491 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d8155220 0x7f47d81556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.327+0000 7f47d67fc640 1 --2- 172.21.15.1:0/543543491 >> v2:172.21.15.98:3300/0 conn(0x7f47d80c5850 0x7f47d80c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.327+0000 7f47d5ffb640 1 -- 172.21.15.1:0/543543491 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680345501 0 0) 0x7f47d8158910 con 0x7f47d8152920 2024-03-06T07:26:01.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.327+0000 7f47d5ffb640 1 -- 172.21.15.1:0/543543491 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c0003520 con 0x7f47d8152920 2024-03-06T07:26:01.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.327+0000 7f47d77fe640 1 -- 172.21.15.1:0/543543491 >> v2:172.21.15.98:3300/0 conn(0x7f47d80c5850 msgr2=0x7f47d80c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.327+0000 7f47d77fe640 1 --2- 172.21.15.1:0/543543491 >> v2:172.21.15.98:3300/0 conn(0x7f47d80c5850 0x7f47d80c5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.327+0000 7f47d77fe640 1 -- 172.21.15.1:0/543543491 >> v1:172.21.15.1:6789/0 conn(0x7f47d8152920 legacy=0x7f47d806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.327+0000 7f47d77fe640 1 -- 172.21.15.1:0/543543491 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d8158bf0 con 0x7f47d8155220 2024-03-06T07:26:01.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.328+0000 7f47d77fe640 1 --2- 172.21.15.1:0/543543491 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d8155220 0x7f47d81556b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f47d0009d90 tx=0x7f47d002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=200f36198d15540e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.328+0000 7f47d5ffb640 1 -- 172.21.15.1:0/543543491 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d0039070 con 0x7f47d8155220 2024-03-06T07:26:01.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.328+0000 7f47d5ffb640 1 -- 172.21.15.1:0/543543491 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d002cbc0 con 0x7f47d8155220 2024-03-06T07:26:01.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.328+0000 7f47d5ffb640 1 -- 172.21.15.1:0/543543491 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d0034440 con 0x7f47d8155220 2024-03-06T07:26:01.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.329+0000 7f47dd84e640 1 -- 172.21.15.1:0/543543491 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d8155220 msgr2=0x7f47d81556b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.329+0000 7f47dd84e640 1 --2- 172.21.15.1:0/543543491 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d8155220 0x7f47d81556b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f47d0009d90 tx=0x7f47d002be10 comp rx=0 tx=0).stop 2024-03-06T07:26:01.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.329+0000 7f47dd84e640 1 -- 172.21.15.1:0/543543491 shutdown_connections 2024-03-06T07:26:01.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.329+0000 7f47dd84e640 1 --2- 172.21.15.1:0/543543491 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d8155220 0x7f47d81556b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.329+0000 7f47dd84e640 1 --2- 172.21.15.1:0/543543491 >> v2:172.21.15.98:3300/0 conn(0x7f47d80c5850 0x7f47d80c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.329+0000 7f47dd84e640 1 -- 172.21.15.1:0/543543491 >> 172.21.15.1:0/543543491 conn(0x7f47d80a9700 msgr2=0x7f47d80a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.329+0000 7f47dd84e640 1 -- 172.21.15.1:0/543543491 shutdown_connections 2024-03-06T07:26:01.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.329+0000 7f47dd84e640 1 -- 172.21.15.1:0/543543491 wait complete. 2024-03-06T07:26:01.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.330+0000 7f47dd84e640 1 Processor -- start 2024-03-06T07:26:01.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.330+0000 7f47dd84e640 1 -- start start 2024-03-06T07:26:01.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47dd84e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d80c5850 0x7f47d8149dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47dd84e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d8152920 0x7f47d80ac790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47dd84e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47d8155220 0x7f47d80accd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47dd84e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d80c6380 con 0x7f47d8155220 2024-03-06T07:26:01.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47dd84e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d8159240 con 0x7f47d8152920 2024-03-06T07:26:01.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47dd84e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d806b700 con 0x7f47d80c5850 2024-03-06T07:26:01.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47d77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47d8155220 0x7f47d80accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47d77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47d8155220 0x7f47d80accd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42538/0 (socket says 172.21.15.1:42538) 2024-03-06T07:26:01.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47d77fe640 1 -- 172.21.15.1:0/1968883540 learned_addr learned my addr 172.21.15.1:0/1968883540 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:01.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47d6ffd640 1 --2- 172.21.15.1:0/1968883540 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d80c5850 0x7f47d8149dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47d67fc640 1 --2- 172.21.15.1:0/1968883540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d8152920 0x7f47d80ac790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47d77fe640 1 -- 172.21.15.1:0/1968883540 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d80c5850 msgr2=0x7f47d8149dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47d77fe640 1 --2- 172.21.15.1:0/1968883540 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d80c5850 0x7f47d8149dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47d77fe640 1 -- 172.21.15.1:0/1968883540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d8152920 msgr2=0x7f47d80ac790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47d77fe640 1 --2- 172.21.15.1:0/1968883540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d8152920 0x7f47d80ac790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47d77fe640 1 -- 172.21.15.1:0/1968883540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d8158970 con 0x7f47d8155220 2024-03-06T07:26:01.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47d77fe640 1 --2- 172.21.15.1:0/1968883540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47d8155220 0x7f47d80accd0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f47d0009d90 tx=0x7f47d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47bffff640 1 -- 172.21.15.1:0/1968883540 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d0039050 con 0x7f47d8155220 2024-03-06T07:26:01.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47bffff640 1 -- 172.21.15.1:0/1968883540 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d003a040 con 0x7f47d8155220 2024-03-06T07:26:01.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47dd84e640 1 -- 172.21.15.1:0/1968883540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d81eb050 con 0x7f47d8155220 2024-03-06T07:26:01.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47bffff640 1 -- 172.21.15.1:0/1968883540 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d0009450 con 0x7f47d8155220 2024-03-06T07:26:01.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.331+0000 7f47dd84e640 1 -- 172.21.15.1:0/1968883540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d8070530 con 0x7f47d8155220 2024-03-06T07:26:01.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.332+0000 7f47dd84e640 1 -- 172.21.15.1:0/1968883540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d8067fd0 con 0x7f47d8155220 2024-03-06T07:26:01.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.333+0000 7f47bffff640 1 -- 172.21.15.1:0/1968883540 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f47d0047020 con 0x7f47d8155220 2024-03-06T07:26:01.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.333+0000 7f47bffff640 1 --2- 172.21.15.1:0/1968883540 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47c406b4d0 0x7f47c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.333+0000 7f47bffff640 1 -- 172.21.15.1:0/1968883540 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f47d00ac1d0 con 0x7f47d8155220 2024-03-06T07:26:01.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.333+0000 7f47d6ffd640 1 --2- 172.21.15.1:0/1968883540 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47c406b4d0 0x7f47c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.334+0000 7f47d6ffd640 1 --2- 172.21.15.1:0/1968883540 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47c406b4d0 0x7f47c406d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f47c8005880 tx=0x7f47c8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.338+0000 7f47bffff640 1 -- 172.21.15.1:0/1968883540 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f47d004e020 con 0x7f47d8155220 2024-03-06T07:26:01.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.522+0000 7f47dd84e640 1 -- 172.21.15.1:0/1968883540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47d806bdf0 con 0x7f47d8155220 2024-03-06T07:26:01.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.523+0000 7f47bffff640 1 -- 172.21.15.1:0/1968883540 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+17027 (secure 0 0 0) 0x7f47d007a910 con 0x7f47d8155220 2024-03-06T07:26:01.525 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:01.525 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":137,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:25:55.860450+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:28.046435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:28.753643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6835/2848968195":"2024-03-07T07:25:55.860421+0000","172.21.15.1:6833/2848968195":"2024-03-07T07:25:55.860421+0000","172.21.15.1:6834/1706279305":"2024-03-07T07:25:41.786393+0000","172.21.15.1:6832/1706279305":"2024-03-07T07:25:41.786393+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:01.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.525+0000 7f47dd84e640 1 -- 172.21.15.1:0/1968883540 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47c406b4d0 msgr2=0x7f47c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.525+0000 7f47dd84e640 1 --2- 172.21.15.1:0/1968883540 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47c406b4d0 0x7f47c406d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f47c8005880 tx=0x7f47c8009040 comp rx=0 tx=0).stop 2024-03-06T07:26:01.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.525+0000 7f47dd84e640 1 -- 172.21.15.1:0/1968883540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47d8155220 msgr2=0x7f47d80accd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.525+0000 7f47dd84e640 1 --2- 172.21.15.1:0/1968883540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47d8155220 0x7f47d80accd0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f47d0009d90 tx=0x7f47d0004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:01.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.525+0000 7f47dd84e640 1 -- 172.21.15.1:0/1968883540 shutdown_connections 2024-03-06T07:26:01.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.525+0000 7f47dd84e640 1 --2- 172.21.15.1:0/1968883540 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47c406b4d0 0x7f47c406d990 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.525+0000 7f47dd84e640 1 --2- 172.21.15.1:0/1968883540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47d8155220 0x7f47d80accd0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.525+0000 7f47dd84e640 1 --2- 172.21.15.1:0/1968883540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d8152920 0x7f47d80ac790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.525+0000 7f47dd84e640 1 --2- 172.21.15.1:0/1968883540 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d80c5850 0x7f47d8149dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.525+0000 7f47dd84e640 1 -- 172.21.15.1:0/1968883540 >> 172.21.15.1:0/1968883540 conn(0x7f47d80a9700 msgr2=0x7f47d8155bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.525+0000 7f47dd84e640 1 -- 172.21.15.1:0/1968883540 shutdown_connections 2024-03-06T07:26:01.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.525+0000 7f47dd84e640 1 -- 172.21.15.1:0/1968883540 wait complete. 2024-03-06T07:26:01.539 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:28.046435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '134', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:28.753643+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '135', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:01.581 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:01.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.683+0000 7f5b24781640 1 Processor -- start 2024-03-06T07:26:01.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.683+0000 7f5b24781640 1 -- start start 2024-03-06T07:26:01.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.684+0000 7f5b24781640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5b1c156cd0 0x7f5b1c157120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.684+0000 7f5b24781640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c0c3300 con 0x7f5b1c152690 2024-03-06T07:26:01.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.684+0000 7f5b24781640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c0c35a0 con 0x7f5b1c0c6730 2024-03-06T07:26:01.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.684+0000 7f5b24781640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b1c0c3880 con 0x7f5b1c156cd0 2024-03-06T07:26:01.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.684+0000 7f5b22d1e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5b1c156cd0 0x7f5b1c157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.685+0000 7f5b2251d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5b1c152690 0x7f5b1c155ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38318/0 (socket says 172.21.15.1:38318) 2024-03-06T07:26:01.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.685+0000 7f5b2251d640 1 -- 172.21.15.1:0/1338175284 learned_addr learned my addr 172.21.15.1:0/1338175284 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:01.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.685+0000 7f5b22d1e640 1 -- 172.21.15.1:0/1338175284 >> v1:172.21.15.98:6789/0 conn(0x7f5b1c0c6730 legacy=0x7f5b1c0c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.685+0000 7f5b22d1e640 1 -- 172.21.15.1:0/1338175284 >> v1:172.21.15.1:6789/0 conn(0x7f5b1c152690 legacy=0x7f5b1c155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.685+0000 7f5b22d1e640 1 -- 172.21.15.1:0/1338175284 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c0c3b60 con 0x7f5b1c156cd0 2024-03-06T07:26:01.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.686+0000 7f5b22d1e640 1 --2- 172.21.15.1:0/1338175284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b1c156cd0 0x7f5b1c157120 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f5b14009d90 tx=0x7f5b1402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd1f6735bce746cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.686+0000 7f5b2151b640 1 -- 172.21.15.1:0/1338175284 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14039070 con 0x7f5b1c156cd0 2024-03-06T07:26:01.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.686+0000 7f5b2151b640 1 -- 172.21.15.1:0/1338175284 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1402cc60 con 0x7f5b1c156cd0 2024-03-06T07:26:01.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.686+0000 7f5b2151b640 1 -- 172.21.15.1:0/1338175284 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14034660 con 0x7f5b1c156cd0 2024-03-06T07:26:01.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.686+0000 7f5b24781640 1 -- 172.21.15.1:0/1338175284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b1c156cd0 msgr2=0x7f5b1c157120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.686+0000 7f5b24781640 1 --2- 172.21.15.1:0/1338175284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b1c156cd0 0x7f5b1c157120 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f5b14009d90 tx=0x7f5b1402beb0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.686+0000 7f5b24781640 1 -- 172.21.15.1:0/1338175284 shutdown_connections 2024-03-06T07:26:01.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.686+0000 7f5b24781640 1 --2- 172.21.15.1:0/1338175284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b1c156cd0 0x7f5b1c157120 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.686+0000 7f5b24781640 1 -- 172.21.15.1:0/1338175284 >> 172.21.15.1:0/1338175284 conn(0x7f5b1c06b410 msgr2=0x7f5b1c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.686+0000 7f5b24781640 1 -- 172.21.15.1:0/1338175284 shutdown_connections 2024-03-06T07:26:01.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.686+0000 7f5b24781640 1 -- 172.21.15.1:0/1338175284 wait complete. 2024-03-06T07:26:01.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.687+0000 7f5b24781640 1 Processor -- start 2024-03-06T07:26:01.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.687+0000 7f5b24781640 1 -- start start 2024-03-06T07:26:01.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.688+0000 7f5b24781640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b1c0c6730 0x7f5b1c0c6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.688+0000 7f5b24781640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b1c152690 0x7f5b1c0ac430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.688+0000 7f5b24781640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b1c156cd0 0x7f5b1c1deaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.688+0000 7f5b24781640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c0c6060 con 0x7f5b1c0c6730 2024-03-06T07:26:01.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.688+0000 7f5b24781640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c0c5ee0 con 0x7f5b1c156cd0 2024-03-06T07:26:01.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.688+0000 7f5b24781640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c152a70 con 0x7f5b1c152690 2024-03-06T07:26:01.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.688+0000 7f5b2251d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b1c0c6730 0x7f5b1c0c6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.688+0000 7f5b2251d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b1c0c6730 0x7f5b1c0c6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42562/0 (socket says 172.21.15.1:42562) 2024-03-06T07:26:01.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.688+0000 7f5b2251d640 1 -- 172.21.15.1:0/2237196035 learned_addr learned my addr 172.21.15.1:0/2237196035 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:01.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.688+0000 7f5b21d1c640 1 --2- 172.21.15.1:0/2237196035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b1c152690 0x7f5b1c0ac430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.688+0000 7f5b22d1e640 1 --2- 172.21.15.1:0/2237196035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b1c156cd0 0x7f5b1c1deaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.688+0000 7f5b2251d640 1 -- 172.21.15.1:0/2237196035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b1c152690 msgr2=0x7f5b1c0ac430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.688+0000 7f5b2251d640 1 --2- 172.21.15.1:0/2237196035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b1c152690 0x7f5b1c0ac430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.688+0000 7f5b2251d640 1 -- 172.21.15.1:0/2237196035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b1c156cd0 msgr2=0x7f5b1c1deaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.688+0000 7f5b2251d640 1 --2- 172.21.15.1:0/2237196035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b1c156cd0 0x7f5b1c1deaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.688+0000 7f5b2251d640 1 -- 172.21.15.1:0/2237196035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c0ad680 con 0x7f5b1c0c6730 2024-03-06T07:26:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.689+0000 7f5b21d1c640 1 --2- 172.21.15.1:0/2237196035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b1c152690 0x7f5b1c0ac430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.689+0000 7f5b22d1e640 1 --2- 172.21.15.1:0/2237196035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b1c156cd0 0x7f5b1c1deaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.689+0000 7f5b2251d640 1 --2- 172.21.15.1:0/2237196035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b1c0c6730 0x7f5b1c0c6bf0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800c400 tx=0x7f5b1800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.689+0000 7f5b12ffd640 1 -- 172.21.15.1:0/2237196035 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18017070 con 0x7f5b1c0c6730 2024-03-06T07:26:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.689+0000 7f5b12ffd640 1 -- 172.21.15.1:0/2237196035 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b18005ce0 con 0x7f5b1c0c6730 2024-03-06T07:26:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.689+0000 7f5b24781640 1 -- 172.21.15.1:0/2237196035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1c0ab950 con 0x7f5b1c0c6730 2024-03-06T07:26:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.689+0000 7f5b12ffd640 1 -- 172.21.15.1:0/2237196035 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18007310 con 0x7f5b1c0c6730 2024-03-06T07:26:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.689+0000 7f5b24781640 1 -- 172.21.15.1:0/2237196035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1c0abf10 con 0x7f5b1c0c6730 2024-03-06T07:26:01.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.690+0000 7f5b24781640 1 -- 172.21.15.1:0/2237196035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1c067fd0 con 0x7f5b1c0c6730 2024-03-06T07:26:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.690+0000 7f5b12ffd640 1 -- 172.21.15.1:0/2237196035 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b1801a070 con 0x7f5b1c0c6730 2024-03-06T07:26:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.691+0000 7f5b12ffd640 1 --2- 172.21.15.1:0/2237196035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b0006b4d0 0x7f5b0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.691+0000 7f5b12ffd640 1 -- 172.21.15.1:0/2237196035 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f5b18089750 con 0x7f5b1c0c6730 2024-03-06T07:26:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.691+0000 7f5b21d1c640 1 --2- 172.21.15.1:0/2237196035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b0006b4d0 0x7f5b0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.691+0000 7f5b21d1c640 1 --2- 172.21.15.1:0/2237196035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b0006b4d0 0x7f5b0006d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0a91a0 tx=0x7f5b0c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.695+0000 7f5b12ffd640 1 -- 172.21.15.1:0/2237196035 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b18058420 con 0x7f5b1c0c6730 2024-03-06T07:26:01.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.875+0000 7f5b24781640 1 -- 172.21.15.1:0/2237196035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5b1c0c2c40 con 0x7f5b1c0c6730 2024-03-06T07:26:01.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.875+0000 7f5b12ffd640 1 -- 172.21.15.1:0/2237196035 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v137) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5b18057d20 con 0x7f5b1c0c6730 2024-03-06T07:26:01.877 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:26:01.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.877+0000 7f5b24781640 1 -- 172.21.15.1:0/2237196035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b0006b4d0 msgr2=0x7f5b0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.877+0000 7f5b24781640 1 --2- 172.21.15.1:0/2237196035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b0006b4d0 0x7f5b0006d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0a91a0 tx=0x7f5b0c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:01.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.877+0000 7f5b24781640 1 -- 172.21.15.1:0/2237196035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b1c0c6730 msgr2=0x7f5b1c0c6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.877+0000 7f5b24781640 1 --2- 172.21.15.1:0/2237196035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b1c0c6730 0x7f5b1c0c6bf0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800c400 tx=0x7f5b1800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.877+0000 7f5b24781640 1 -- 172.21.15.1:0/2237196035 shutdown_connections 2024-03-06T07:26:01.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.877+0000 7f5b24781640 1 --2- 172.21.15.1:0/2237196035 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b0006b4d0 0x7f5b0006d990 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.877+0000 7f5b24781640 1 --2- 172.21.15.1:0/2237196035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b1c156cd0 0x7f5b1c1deaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.877+0000 7f5b24781640 1 --2- 172.21.15.1:0/2237196035 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b1c152690 0x7f5b1c0ac430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.877+0000 7f5b24781640 1 --2- 172.21.15.1:0/2237196035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b1c0c6730 0x7f5b1c0c6bf0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.877+0000 7f5b24781640 1 -- 172.21.15.1:0/2237196035 >> 172.21.15.1:0/2237196035 conn(0x7f5b1c06b410 msgr2=0x7f5b1c06fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.877+0000 7f5b24781640 1 -- 172.21.15.1:0/2237196035 shutdown_connections 2024-03-06T07:26:01.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.877+0000 7f5b24781640 1 -- 172.21.15.1:0/2237196035 wait complete. 2024-03-06T07:26:01.890 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:01.928 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:01.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.992+0000 7f6051e8a640 1 Processor -- start 2024-03-06T07:26:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.992+0000 7f6051e8a640 1 -- start start 2024-03-06T07:26:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.993+0000 7f6051e8a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604c0c69b0 0x7f604c0c2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.993+0000 7f6051e8a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c072d00 con 0x7f604c159140 2024-03-06T07:26:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.993+0000 7f6051e8a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c073180 con 0x7f604c0c7510 2024-03-06T07:26:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.993+0000 7f6051e8a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604c0c0e80 con 0x7f604c0c69b0 2024-03-06T07:26:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.993+0000 7f604b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f604c159140 0x7f604c0c4490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38328/0 (socket says 172.21.15.1:38328) 2024-03-06T07:26:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.993+0000 7f604b7fe640 1 -- 172.21.15.1:0/1297276241 learned_addr learned my addr 172.21.15.1:0/1297276241 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.993+0000 7f604affd640 1 --2- 172.21.15.1:0/1297276241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604c0c69b0 0x7f604c0c2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.993+0000 7f604a7fc640 1 -- 172.21.15.1:0/1297276241 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221686237 0 0) 0x7f604c072d00 con 0x7f604c159140 2024-03-06T07:26:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.994+0000 7f604a7fc640 1 -- 172.21.15.1:0/1297276241 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6034003520 con 0x7f604c159140 2024-03-06T07:26:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.994+0000 7f604a7fc640 1 -- 172.21.15.1:0/1297276241 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351824353 0 0) 0x7f604c073180 con 0x7f604c0c7510 2024-03-06T07:26:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.994+0000 7f604a7fc640 1 -- 172.21.15.1:0/1297276241 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f604c072d00 con 0x7f604c0c7510 2024-03-06T07:26:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.994+0000 7f604affd640 1 -- 172.21.15.1:0/1297276241 >> v1:172.21.15.98:6789/0 conn(0x7f604c0c7510 legacy=0x7f604c15fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.994+0000 7f604affd640 1 -- 172.21.15.1:0/1297276241 >> v1:172.21.15.1:6789/0 conn(0x7f604c159140 legacy=0x7f604c0c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.994+0000 7f604affd640 1 -- 172.21.15.1:0/1297276241 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c073460 con 0x7f604c0c69b0 2024-03-06T07:26:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.994+0000 7f604affd640 1 --2- 172.21.15.1:0/1297276241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604c0c69b0 0x7f604c0c2b30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f60400097d0 tx=0x7f604002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4307418f96a6ad55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.995+0000 7f604a7fc640 1 -- 172.21.15.1:0/1297276241 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040035020 con 0x7f604c0c69b0 2024-03-06T07:26:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.995+0000 7f604a7fc640 1 -- 172.21.15.1:0/1297276241 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6040031800 con 0x7f604c0c69b0 2024-03-06T07:26:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.995+0000 7f604a7fc640 1 -- 172.21.15.1:0/1297276241 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040031bd0 con 0x7f604c0c69b0 2024-03-06T07:26:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.995+0000 7f6051e8a640 1 -- 172.21.15.1:0/1297276241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604c0c69b0 msgr2=0x7f604c0c2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.995+0000 7f6051e8a640 1 --2- 172.21.15.1:0/1297276241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604c0c69b0 0x7f604c0c2b30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f60400097d0 tx=0x7f604002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.995+0000 7f6051e8a640 1 -- 172.21.15.1:0/1297276241 shutdown_connections 2024-03-06T07:26:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.995+0000 7f6051e8a640 1 --2- 172.21.15.1:0/1297276241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604c0c69b0 0x7f604c0c2b30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.995+0000 7f6051e8a640 1 -- 172.21.15.1:0/1297276241 >> 172.21.15.1:0/1297276241 conn(0x7f604c0c7a50 msgr2=0x7f604c0c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.995+0000 7f6051e8a640 1 -- 172.21.15.1:0/1297276241 shutdown_connections 2024-03-06T07:26:01.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.995+0000 7f6051e8a640 1 -- 172.21.15.1:0/1297276241 wait complete. 2024-03-06T07:26:01.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.996+0000 7f6051e8a640 1 Processor -- start 2024-03-06T07:26:02.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.997+0000 7f6051e8a640 1 -- start start 2024-03-06T07:26:02.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.997+0000 7f6051e8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f604c0c69b0 0x7f604c1ee4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.997+0000 7f6051e8a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604c0c7510 0x7f604c0ac530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.997+0000 7f6051e8a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f604c159140 0x7f604c0aca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.997+0000 7f6051e8a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c0c6520 con 0x7f604c159140 2024-03-06T07:26:02.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.997+0000 7f6051e8a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c0c13b0 con 0x7f604c0c69b0 2024-03-06T07:26:02.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.997+0000 7f6051e8a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604c0c4ba0 con 0x7f604c0c7510 2024-03-06T07:26:02.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.997+0000 7f604bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f604c159140 0x7f604c0aca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.997+0000 7f604bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f604c159140 0x7f604c0aca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42574/0 (socket says 172.21.15.1:42574) 2024-03-06T07:26:02.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.997+0000 7f604bfff640 1 -- 172.21.15.1:0/1171572446 learned_addr learned my addr 172.21.15.1:0/1171572446 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:02.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.997+0000 7f604affd640 1 --2- 172.21.15.1:0/1171572446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604c0c7510 0x7f604c0ac530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.997+0000 7f604b7fe640 1 --2- 172.21.15.1:0/1171572446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f604c0c69b0 0x7f604c1ee4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.998+0000 7f604bfff640 1 -- 172.21.15.1:0/1171572446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604c0c7510 msgr2=0x7f604c0ac530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.998+0000 7f604bfff640 1 --2- 172.21.15.1:0/1171572446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604c0c7510 0x7f604c0ac530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.998+0000 7f604bfff640 1 -- 172.21.15.1:0/1171572446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f604c0c69b0 msgr2=0x7f604c1ee4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.998+0000 7f604bfff640 1 --2- 172.21.15.1:0/1171572446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f604c0c69b0 0x7f604c1ee4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.998+0000 7f604bfff640 1 -- 172.21.15.1:0/1171572446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c1eedc0 con 0x7f604c159140 2024-03-06T07:26:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.998+0000 7f604bfff640 1 --2- 172.21.15.1:0/1171572446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f604c159140 0x7f604c0aca70 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f604400c7a0 tx=0x7f604400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.998+0000 7f6033fff640 1 -- 172.21.15.1:0/1171572446 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6044003b70 con 0x7f604c159140 2024-03-06T07:26:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.998+0000 7f6033fff640 1 -- 172.21.15.1:0/1171572446 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6044003d10 con 0x7f604c159140 2024-03-06T07:26:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.998+0000 7f6033fff640 1 -- 172.21.15.1:0/1171572446 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604400aa90 con 0x7f604c159140 2024-03-06T07:26:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.998+0000 7f6051e8a640 1 -- 172.21.15.1:0/1171572446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604c1ef0a0 con 0x7f604c159140 2024-03-06T07:26:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.998+0000 7f6051e8a640 1 -- 172.21.15.1:0/1171572446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f604c1501c0 con 0x7f604c159140 2024-03-06T07:26:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.999+0000 7f6051e8a640 1 -- 172.21.15.1:0/1171572446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604c0686e0 con 0x7f604c159140 2024-03-06T07:26:02.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:01.999+0000 7f6033fff640 1 -- 172.21.15.1:0/1171572446 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f604400ac30 con 0x7f604c159140 2024-03-06T07:26:02.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.000+0000 7f6033fff640 1 --2- 172.21.15.1:0/1171572446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f603806b500 0x7f603806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.000+0000 7f6033fff640 1 -- 172.21.15.1:0/1171572446 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f604408a250 con 0x7f604c159140 2024-03-06T07:26:02.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.000+0000 7f604b7fe640 1 --2- 172.21.15.1:0/1171572446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f603806b500 0x7f603806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.000+0000 7f604b7fe640 1 --2- 172.21.15.1:0/1171572446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f603806b500 0x7f603806d9c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f603c005880 tx=0x7f603c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.004+0000 7f6033fff640 1 -- 172.21.15.1:0/1171572446 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6044058830 con 0x7f604c159140 2024-03-06T07:26:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.184+0000 7f6051e8a640 1 -- 172.21.15.1:0/1171572446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f604c072f60 con 0x7f604c159140 2024-03-06T07:26:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.184+0000 7f6033fff640 1 -- 172.21.15.1:0/1171572446 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v137) v1 ==== 104+0+11 (secure 0 0 0) 0x7f604405c8d0 con 0x7f604c159140 2024-03-06T07:26:02.186 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:26:02.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.186+0000 7f6051e8a640 1 -- 172.21.15.1:0/1171572446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f603806b500 msgr2=0x7f603806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.186+0000 7f6051e8a640 1 --2- 172.21.15.1:0/1171572446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f603806b500 0x7f603806d9c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f603c005880 tx=0x7f603c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:02.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.186+0000 7f6051e8a640 1 -- 172.21.15.1:0/1171572446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f604c159140 msgr2=0x7f604c0aca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.186+0000 7f6051e8a640 1 --2- 172.21.15.1:0/1171572446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f604c159140 0x7f604c0aca70 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f604400c7a0 tx=0x7f604400cc60 comp rx=0 tx=0).stop 2024-03-06T07:26:02.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.186+0000 7f6051e8a640 1 -- 172.21.15.1:0/1171572446 shutdown_connections 2024-03-06T07:26:02.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.186+0000 7f6051e8a640 1 --2- 172.21.15.1:0/1171572446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f603806b500 0x7f603806d9c0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.186+0000 7f6051e8a640 1 --2- 172.21.15.1:0/1171572446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f604c159140 0x7f604c0aca70 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.186+0000 7f6051e8a640 1 --2- 172.21.15.1:0/1171572446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604c0c7510 0x7f604c0ac530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.186+0000 7f6051e8a640 1 --2- 172.21.15.1:0/1171572446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f604c0c69b0 0x7f604c1ee4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.186+0000 7f6051e8a640 1 -- 172.21.15.1:0/1171572446 >> 172.21.15.1:0/1171572446 conn(0x7f604c0c7a50 msgr2=0x7f604c0c4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.186+0000 7f6051e8a640 1 -- 172.21.15.1:0/1171572446 shutdown_connections 2024-03-06T07:26:02.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.186+0000 7f6051e8a640 1 -- 172.21.15.1:0/1171572446 wait complete. 2024-03-06T07:26:02.199 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:02.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.340+0000 7f3c4b56c640 1 Processor -- start 2024-03-06T07:26:02.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.340+0000 7f3c4b56c640 1 -- start start 2024-03-06T07:26:02.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.341+0000 7f3c4b56c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3c440c2b30 0x7f3c440724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.341+0000 7f3c4b56c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c44151dd0 con 0x7f3c44152b70 2024-03-06T07:26:02.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.341+0000 7f3c4b56c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c44156190 con 0x7f3c440a5fa0 2024-03-06T07:26:02.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.341+0000 7f3c4b56c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c441526e0 con 0x7f3c440c2b30 2024-03-06T07:26:02.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.341+0000 7f3c49308640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3c440c2b30 0x7f3c440724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.341+0000 7f3c48b07640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3c44152b70 0x7f3c440c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38334/0 (socket says 172.21.15.1:38334) 2024-03-06T07:26:02.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.341+0000 7f3c48b07640 1 -- 172.21.15.1:0/753417041 learned_addr learned my addr 172.21.15.1:0/753417041 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:02.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.341+0000 7f3c49308640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3c440c2b30 0x7f3c440724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42586/0 (socket says 172.21.15.1:42586) 2024-03-06T07:26:02.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.341+0000 7f3c49308640 1 -- 172.21.15.1:0/753417041 >> v1:172.21.15.98:6789/0 conn(0x7f3c440a5fa0 legacy=0x7f3c4415c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.341+0000 7f3c49308640 1 -- 172.21.15.1:0/753417041 >> v1:172.21.15.1:6789/0 conn(0x7f3c44152b70 legacy=0x7f3c440c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.341+0000 7f3c43fff640 1 -- 172.21.15.1:0/753417041 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305709101 0 0) 0x7f3c44151dd0 con 0x7f3c44152b70 2024-03-06T07:26:02.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.341+0000 7f3c49308640 1 -- 172.21.15.1:0/753417041 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c44152090 con 0x7f3c440c2b30 2024-03-06T07:26:02.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.342+0000 7f3c49308640 1 --2- 172.21.15.1:0/753417041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c440c2b30 0x7f3c440724c0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c00ba40 tx=0x7f3c3c02e130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dff4fec51b486e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.342+0000 7f3c43fff640 1 -- 172.21.15.1:0/753417041 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c03b070 con 0x7f3c440c2b30 2024-03-06T07:26:02.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.342+0000 7f3c43fff640 1 -- 172.21.15.1:0/753417041 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3c036440 con 0x7f3c440c2b30 2024-03-06T07:26:02.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.342+0000 7f3c43fff640 1 -- 172.21.15.1:0/753417041 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c036810 con 0x7f3c440c2b30 2024-03-06T07:26:02.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.343+0000 7f3c4b56c640 1 -- 172.21.15.1:0/753417041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c440c2b30 msgr2=0x7f3c440724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.343+0000 7f3c4b56c640 1 --2- 172.21.15.1:0/753417041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c440c2b30 0x7f3c440724c0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c00ba40 tx=0x7f3c3c02e130 comp rx=0 tx=0).stop 2024-03-06T07:26:02.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.343+0000 7f3c4b56c640 1 -- 172.21.15.1:0/753417041 shutdown_connections 2024-03-06T07:26:02.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.343+0000 7f3c4b56c640 1 --2- 172.21.15.1:0/753417041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c440c2b30 0x7f3c440724c0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.343+0000 7f3c4b56c640 1 -- 172.21.15.1:0/753417041 >> 172.21.15.1:0/753417041 conn(0x7f3c440c6860 msgr2=0x7f3c440c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.343+0000 7f3c4b56c640 1 -- 172.21.15.1:0/753417041 shutdown_connections 2024-03-06T07:26:02.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.343+0000 7f3c4b56c640 1 -- 172.21.15.1:0/753417041 wait complete. 2024-03-06T07:26:02.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.344+0000 7f3c4b56c640 1 Processor -- start 2024-03-06T07:26:02.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.344+0000 7f3c4b56c640 1 -- start start 2024-03-06T07:26:02.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c4b56c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c440a5fa0 0x7f3c441e6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c4b56c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c440c2b30 0x7f3c44153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c4b56c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c44152b70 0x7f3c441499d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c4b56c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c44155f00 con 0x7f3c44152b70 2024-03-06T07:26:02.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c4b56c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c44155d80 con 0x7f3c440a5fa0 2024-03-06T07:26:02.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c4b56c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c44072a00 con 0x7f3c440c2b30 2024-03-06T07:26:02.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c49b09640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c44152b70 0x7f3c441499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c49308640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c440a5fa0 0x7f3c441e6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c49b09640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c44152b70 0x7f3c441499d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42600/0 (socket says 172.21.15.1:42600) 2024-03-06T07:26:02.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c49b09640 1 -- 172.21.15.1:0/454376749 learned_addr learned my addr 172.21.15.1:0/454376749 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:02.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c48b07640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c440c2b30 0x7f3c44153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c49308640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c440a5fa0 0x7f3c441e6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:49208/0 (socket says 172.21.15.1:49208) 2024-03-06T07:26:02.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c49b09640 1 -- 172.21.15.1:0/454376749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c440c2b30 msgr2=0x7f3c44153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c49b09640 1 --2- 172.21.15.1:0/454376749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c440c2b30 0x7f3c44153430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c49b09640 1 -- 172.21.15.1:0/454376749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c440a5fa0 msgr2=0x7f3c441e6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c49b09640 1 --2- 172.21.15.1:0/454376749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c440a5fa0 0x7f3c441e6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c49b09640 1 -- 172.21.15.1:0/454376749 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c441db4d0 con 0x7f3c44152b70 2024-03-06T07:26:02.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.345+0000 7f3c49b09640 1 --2- 172.21.15.1:0/454376749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c44152b70 0x7f3c441499d0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400c7d0 tx=0x7f3c3400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.346+0000 7f3c41ffb640 1 -- 172.21.15.1:0/454376749 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c34003b70 con 0x7f3c44152b70 2024-03-06T07:26:02.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.346+0000 7f3c41ffb640 1 -- 172.21.15.1:0/454376749 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c34003d30 con 0x7f3c44152b70 2024-03-06T07:26:02.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.346+0000 7f3c4b56c640 1 -- 172.21.15.1:0/454376749 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c441db7b0 con 0x7f3c44152b70 2024-03-06T07:26:02.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.346+0000 7f3c41ffb640 1 -- 172.21.15.1:0/454376749 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3400aa90 con 0x7f3c44152b70 2024-03-06T07:26:02.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.346+0000 7f3c4b56c640 1 -- 172.21.15.1:0/454376749 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c441dbd70 con 0x7f3c44152b70 2024-03-06T07:26:02.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.347+0000 7f3c4b56c640 1 -- 172.21.15.1:0/454376749 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c44067fd0 con 0x7f3c44152b70 2024-03-06T07:26:02.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.347+0000 7f3c41ffb640 1 -- 172.21.15.1:0/454376749 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c3400ac30 con 0x7f3c44152b70 2024-03-06T07:26:02.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.347+0000 7f3c41ffb640 1 --2- 172.21.15.1:0/454376749 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c3006b4d0 0x7f3c3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.348+0000 7f3c41ffb640 1 -- 172.21.15.1:0/454376749 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f3c3408a220 con 0x7f3c44152b70 2024-03-06T07:26:02.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.348+0000 7f3c49308640 1 --2- 172.21.15.1:0/454376749 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c3006b4d0 0x7f3c3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.348+0000 7f3c49308640 1 --2- 172.21.15.1:0/454376749 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c3006b4d0 0x7f3c3006d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c038000 tx=0x7f3c3c032000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.352+0000 7f3c41ffb640 1 -- 172.21.15.1:0/454376749 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c34058800 con 0x7f3c44152b70 2024-03-06T07:26:02.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.535+0000 7f3c4b56c640 1 -- 172.21.15.1:0/454376749 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3c440c09d0 con 0x7f3c44152b70 2024-03-06T07:26:02.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.535+0000 7f3c41ffb640 1 -- 172.21.15.1:0/454376749 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v137) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3c3405c8a0 con 0x7f3c44152b70 2024-03-06T07:26:02.537 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:26:02.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.537+0000 7f3c4b56c640 1 -- 172.21.15.1:0/454376749 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c3006b4d0 msgr2=0x7f3c3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.537+0000 7f3c4b56c640 1 --2- 172.21.15.1:0/454376749 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c3006b4d0 0x7f3c3006d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c038000 tx=0x7f3c3c032000 comp rx=0 tx=0).stop 2024-03-06T07:26:02.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.537+0000 7f3c4b56c640 1 -- 172.21.15.1:0/454376749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c44152b70 msgr2=0x7f3c441499d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.537+0000 7f3c4b56c640 1 --2- 172.21.15.1:0/454376749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c44152b70 0x7f3c441499d0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400c7d0 tx=0x7f3c3400cc90 comp rx=0 tx=0).stop 2024-03-06T07:26:02.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.537+0000 7f3c4b56c640 1 -- 172.21.15.1:0/454376749 shutdown_connections 2024-03-06T07:26:02.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.537+0000 7f3c4b56c640 1 --2- 172.21.15.1:0/454376749 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c3006b4d0 0x7f3c3006d990 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.537+0000 7f3c4b56c640 1 --2- 172.21.15.1:0/454376749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c44152b70 0x7f3c441499d0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.537+0000 7f3c4b56c640 1 --2- 172.21.15.1:0/454376749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c440c2b30 0x7f3c44153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.537+0000 7f3c4b56c640 1 --2- 172.21.15.1:0/454376749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c440a5fa0 0x7f3c441e6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.537+0000 7f3c4b56c640 1 -- 172.21.15.1:0/454376749 >> 172.21.15.1:0/454376749 conn(0x7f3c440c6860 msgr2=0x7f3c4414f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.537+0000 7f3c4b56c640 1 -- 172.21.15.1:0/454376749 shutdown_connections 2024-03-06T07:26:02.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.537+0000 7f3c4b56c640 1 -- 172.21.15.1:0/454376749 wait complete. 2024-03-06T07:26:02.550 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:02.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.649+0000 7fa4cbc78640 1 Processor -- start 2024-03-06T07:26:02.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.649+0000 7fa4cbc78640 1 -- start start 2024-03-06T07:26:02.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.650+0000 7fa4cbc78640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa4c40c1260 0x7fa4c40c16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.650+0000 7fa4cbc78640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4c41565a0 0x7fa4c4156a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.650+0000 7fa4cbc78640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c41530a0 con 0x7fa4c4149a90 2024-03-06T07:26:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.650+0000 7fa4cbc78640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c4153380 con 0x7fa4c40c1260 2024-03-06T07:26:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.650+0000 7fa4cbc78640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c4153660 con 0x7fa4c41565a0 2024-03-06T07:26:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.650+0000 7fa4c9213640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa4c40c1260 0x7fa4c40c16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.650+0000 7fa4c9a14640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa4c4149a90 0x7fa4c40c4760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38342/0 (socket says 172.21.15.1:38342) 2024-03-06T07:26:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.650+0000 7fa4c9213640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa4c40c1260 0x7fa4c40c16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42616/0 (socket says 172.21.15.1:42616) 2024-03-06T07:26:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.650+0000 7fa4c9a14640 1 -- 172.21.15.1:0/2298804438 learned_addr learned my addr 172.21.15.1:0/2298804438 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.650+0000 7fa4ca215640 1 --2- 172.21.15.1:0/2298804438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4c41565a0 0x7fa4c4156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.650+0000 7fa4ca215640 1 -- 172.21.15.1:0/2298804438 >> v2:172.21.15.1:3300/0 conn(0x7fa4c40c1260 msgr2=0x7fa4c40c16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.650+0000 7fa4ca215640 1 --2- 172.21.15.1:0/2298804438 >> v2:172.21.15.1:3300/0 conn(0x7fa4c40c1260 0x7fa4c40c16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.650+0000 7fa4ca215640 1 -- 172.21.15.1:0/2298804438 >> v1:172.21.15.1:6789/0 conn(0x7fa4c4149a90 legacy=0x7fa4c40c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.650+0000 7fa4c8a12640 1 -- 172.21.15.1:0/2298804438 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739454184 0 0) 0x7fa4c41530a0 con 0x7fa4c4149a90 2024-03-06T07:26:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.650+0000 7fa4ca215640 1 -- 172.21.15.1:0/2298804438 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c4153940 con 0x7fa4c41565a0 2024-03-06T07:26:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.650+0000 7fa4c9213640 1 --2- 172.21.15.1:0/2298804438 >> v2:172.21.15.1:3300/0 conn(0x7fa4c40c1260 0x7fa4c40c16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:02.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.651+0000 7fa4ca215640 1 --2- 172.21.15.1:0/2298804438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4c41565a0 0x7fa4c4156a10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4009d90 tx=0x7fa4b402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f9828211e8530a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.651+0000 7fa4c8a12640 1 -- 172.21.15.1:0/2298804438 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4039070 con 0x7fa4c41565a0 2024-03-06T07:26:02.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.651+0000 7fa4c8a12640 1 -- 172.21.15.1:0/2298804438 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b402cbc0 con 0x7fa4c41565a0 2024-03-06T07:26:02.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.651+0000 7fa4c8a12640 1 -- 172.21.15.1:0/2298804438 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4034440 con 0x7fa4c41565a0 2024-03-06T07:26:02.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.652+0000 7fa4cbc78640 1 -- 172.21.15.1:0/2298804438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4c41565a0 msgr2=0x7fa4c4156a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.652+0000 7fa4cbc78640 1 --2- 172.21.15.1:0/2298804438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4c41565a0 0x7fa4c4156a10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4009d90 tx=0x7fa4b402be10 comp rx=0 tx=0).stop 2024-03-06T07:26:02.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.652+0000 7fa4cbc78640 1 -- 172.21.15.1:0/2298804438 shutdown_connections 2024-03-06T07:26:02.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.652+0000 7fa4cbc78640 1 --2- 172.21.15.1:0/2298804438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4c41565a0 0x7fa4c4156a10 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.652+0000 7fa4cbc78640 1 --2- 172.21.15.1:0/2298804438 >> v2:172.21.15.1:3300/0 conn(0x7fa4c40c1260 0x7fa4c40c16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.652+0000 7fa4cbc78640 1 -- 172.21.15.1:0/2298804438 >> 172.21.15.1:0/2298804438 conn(0x7fa4c40c6860 msgr2=0x7fa4c40c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.652+0000 7fa4cbc78640 1 -- 172.21.15.1:0/2298804438 shutdown_connections 2024-03-06T07:26:02.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.652+0000 7fa4cbc78640 1 -- 172.21.15.1:0/2298804438 wait complete. 2024-03-06T07:26:02.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.653+0000 7fa4cbc78640 1 Processor -- start 2024-03-06T07:26:02.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.653+0000 7fa4cbc78640 1 -- start start 2024-03-06T07:26:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4cbc78640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c40c1260 0x7fa4c41502f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4cbc78640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4c4149a90 0x7fa4c414c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4cbc78640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c41565a0 0x7fa4c4145c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4cbc78640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c406f620 con 0x7fa4c41565a0 2024-03-06T07:26:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4cbc78640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c406f4a0 con 0x7fa4c40c1260 2024-03-06T07:26:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4cbc78640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c4156f50 con 0x7fa4c4149a90 2024-03-06T07:26:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4ca215640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c41565a0 0x7fa4c4145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4ca215640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c41565a0 0x7fa4c4145c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42618/0 (socket says 172.21.15.1:42618) 2024-03-06T07:26:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4ca215640 1 -- 172.21.15.1:0/3849548442 learned_addr learned my addr 172.21.15.1:0/3849548442 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4c9a14640 1 --2- 172.21.15.1:0/3849548442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c40c1260 0x7fa4c41502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4c9213640 1 --2- 172.21.15.1:0/3849548442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4c4149a90 0x7fa4c414c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4ca215640 1 -- 172.21.15.1:0/3849548442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4c4149a90 msgr2=0x7fa4c414c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4ca215640 1 --2- 172.21.15.1:0/3849548442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4c4149a90 0x7fa4c414c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4ca215640 1 -- 172.21.15.1:0/3849548442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c40c1260 msgr2=0x7fa4c41502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4ca215640 1 --2- 172.21.15.1:0/3849548442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c40c1260 0x7fa4c41502f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4ca215640 1 -- 172.21.15.1:0/3849548442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c40bc010 con 0x7fa4c41565a0 2024-03-06T07:26:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4ca215640 1 --2- 172.21.15.1:0/3849548442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c41565a0 0x7fa4c4145c30 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4037040 tx=0x7fa4b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4c27fc640 1 -- 172.21.15.1:0/3849548442 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4044020 con 0x7fa4c41565a0 2024-03-06T07:26:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.654+0000 7fa4c27fc640 1 -- 172.21.15.1:0/3849548442 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b403a040 con 0x7fa4c41565a0 2024-03-06T07:26:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.655+0000 7fa4cbc78640 1 -- 172.21.15.1:0/3849548442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c40bc2f0 con 0x7fa4c41565a0 2024-03-06T07:26:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.655+0000 7fa4c27fc640 1 -- 172.21.15.1:0/3849548442 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4039050 con 0x7fa4c41565a0 2024-03-06T07:26:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.655+0000 7fa4cbc78640 1 -- 172.21.15.1:0/3849548442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c414eda0 con 0x7fa4c41565a0 2024-03-06T07:26:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.655+0000 7fa4cbc78640 1 -- 172.21.15.1:0/3849548442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c4067fd0 con 0x7fa4c41565a0 2024-03-06T07:26:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.656+0000 7fa4c27fc640 1 -- 172.21.15.1:0/3849548442 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa4b4003bb0 con 0x7fa4c41565a0 2024-03-06T07:26:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.656+0000 7fa4c27fc640 1 --2- 172.21.15.1:0/3849548442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa4b006b4d0 0x7fa4b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.656+0000 7fa4c27fc640 1 -- 172.21.15.1:0/3849548442 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fa4b40ac9a0 con 0x7fa4c41565a0 2024-03-06T07:26:02.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.656+0000 7fa4c9a14640 1 --2- 172.21.15.1:0/3849548442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa4b006b4d0 0x7fa4b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.657+0000 7fa4c9a14640 1 --2- 172.21.15.1:0/3849548442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa4b006b4d0 0x7fa4b006d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc000f60 tx=0x7fa4bc007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.661+0000 7fa4c27fc640 1 -- 172.21.15.1:0/3849548442 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa4b404e060 con 0x7fa4c41565a0 2024-03-06T07:26:02.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.878+0000 7fa4cbc78640 1 -- 172.21.15.1:0/3849548442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4c40c5c60 con 0x7fa4c41565a0 2024-03-06T07:26:02.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.879+0000 7fa4c27fc640 1 -- 172.21.15.1:0/3849548442 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+4984 (secure 0 0 0) 0x7fa4b407af80 con 0x7fa4c41565a0 2024-03-06T07:26:02.881 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:02.881 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":131,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2739580331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2739580331},{"type":"v1","addr":"172.21.15.98:6837","nonce":2739580331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":8517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3374845636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3374845636},{"type":"v1","addr":"172.21.15.1:6834","nonce":3374845636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":8607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1602915919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1602915919},{"type":"v1","addr":"172.21.15.1:6835","nonce":1602915919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:29.792840+0000","modified":"2024-03-06T07:25:58.868171+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7387},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.98:6836/1671554096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1671554096},{"type":"v1","addr":"172.21.15.98:6836","nonce":1671554096}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7387,"qdb_cluster":[7387]},"id":10}]} 2024-03-06T07:26:02.881 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 131 2024-03-06T07:26:02.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.881+0000 7fa4cbc78640 1 -- 172.21.15.1:0/3849548442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa4b006b4d0 msgr2=0x7fa4b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.881+0000 7fa4cbc78640 1 --2- 172.21.15.1:0/3849548442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa4b006b4d0 0x7fa4b006d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc000f60 tx=0x7fa4bc007d50 comp rx=0 tx=0).stop 2024-03-06T07:26:02.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.881+0000 7fa4cbc78640 1 -- 172.21.15.1:0/3849548442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c41565a0 msgr2=0x7fa4c4145c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.881+0000 7fa4cbc78640 1 --2- 172.21.15.1:0/3849548442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c41565a0 0x7fa4c4145c30 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4037040 tx=0x7fa4b4004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.881+0000 7fa4cbc78640 1 -- 172.21.15.1:0/3849548442 shutdown_connections 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.881+0000 7fa4cbc78640 1 --2- 172.21.15.1:0/3849548442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa4b006b4d0 0x7fa4b006d990 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.881+0000 7fa4cbc78640 1 --2- 172.21.15.1:0/3849548442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c41565a0 0x7fa4c4145c30 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.881+0000 7fa4cbc78640 1 --2- 172.21.15.1:0/3849548442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4c4149a90 0x7fa4c414c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.881+0000 7fa4cbc78640 1 --2- 172.21.15.1:0/3849548442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c40c1260 0x7fa4c41502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.881+0000 7fa4cbc78640 1 -- 172.21.15.1:0/3849548442 >> 172.21.15.1:0/3849548442 conn(0x7fa4c40c6860 msgr2=0x7fa4c40c2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.881+0000 7fa4cbc78640 1 -- 172.21.15.1:0/3849548442 shutdown_connections 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.881+0000 7fa4cbc78640 1 -- 172.21.15.1:0/3849548442 wait complete. 2024-03-06T07:26:02.894 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.993+0000 7f8099fac640 1 Processor -- start 2024-03-06T07:26:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.993+0000 7f8099fac640 1 -- start start 2024-03-06T07:26:02.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.994+0000 7f8099fac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8094152650 0x7f8094155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.994+0000 7f8099fac640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80940c2fc0 con 0x7f80940bf9d0 2024-03-06T07:26:02.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.994+0000 7f8099fac640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8094158da0 con 0x7f80940c6b10 2024-03-06T07:26:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.994+0000 7f8099fac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8094155800 con 0x7f8094152650 2024-03-06T07:26:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.994+0000 7f8093fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f80940bf9d0 0x7f80940c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38350/0 (socket says 172.21.15.1:38350) 2024-03-06T07:26:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.994+0000 7f8093fff640 1 -- 172.21.15.1:0/1327540897 learned_addr learned my addr 172.21.15.1:0/1327540897 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.994+0000 7f80937fe640 1 --2- 172.21.15.1:0/1327540897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8094152650 0x7f8094155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.995+0000 7f80937fe640 1 -- 172.21.15.1:0/1327540897 >> v1:172.21.15.98:6789/0 conn(0x7f80940c6b10 legacy=0x7f8094158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.995+0000 7f80937fe640 1 -- 172.21.15.1:0/1327540897 >> v1:172.21.15.1:6789/0 conn(0x7f80940bf9d0 legacy=0x7f80940c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.995+0000 7f80927fc640 1 -- 172.21.15.1:0/1327540897 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915210129 0 0) 0x7f80940c2fc0 con 0x7f80940bf9d0 2024-03-06T07:26:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.995+0000 7f80937fe640 1 -- 172.21.15.1:0/1327540897 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80940c6690 con 0x7f8094152650 2024-03-06T07:26:02.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.995+0000 7f80937fe640 1 --2- 172.21.15.1:0/1327540897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8094152650 0x7f8094155c90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8084009750 tx=0x7f808402b680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abab94e5eccd79c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.995+0000 7f80927fc640 1 -- 172.21.15.1:0/1327540897 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8084037070 con 0x7f8094152650 2024-03-06T07:26:02.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.995+0000 7f80927fc640 1 -- 172.21.15.1:0/1327540897 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8084030710 con 0x7f8094152650 2024-03-06T07:26:02.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.995+0000 7f80927fc640 1 -- 172.21.15.1:0/1327540897 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8084030ae0 con 0x7f8094152650 2024-03-06T07:26:02.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.996+0000 7f8099fac640 1 -- 172.21.15.1:0/1327540897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8094152650 msgr2=0x7f8094155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.996+0000 7f8099fac640 1 --2- 172.21.15.1:0/1327540897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8094152650 0x7f8094155c90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8084009750 tx=0x7f808402b680 comp rx=0 tx=0).stop 2024-03-06T07:26:02.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.996+0000 7f8099fac640 1 -- 172.21.15.1:0/1327540897 shutdown_connections 2024-03-06T07:26:02.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.996+0000 7f8099fac640 1 --2- 172.21.15.1:0/1327540897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8094152650 0x7f8094155c90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.996+0000 7f8099fac640 1 -- 172.21.15.1:0/1327540897 >> 172.21.15.1:0/1327540897 conn(0x7f8094066ee0 msgr2=0x7f80940672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.996+0000 7f8099fac640 1 -- 172.21.15.1:0/1327540897 shutdown_connections 2024-03-06T07:26:02.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.996+0000 7f8099fac640 1 -- 172.21.15.1:0/1327540897 wait complete. 2024-03-06T07:26:03.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.997+0000 7f8099fac640 1 Processor -- start 2024-03-06T07:26:03.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.997+0000 7f8099fac640 1 -- start start 2024-03-06T07:26:03.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.998+0000 7f8099fac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80940bf9d0 0x7f809414e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.998+0000 7f8099fac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8094152650 0x7f809414a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.998+0000 7f8099fac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80940c87d0 0x7f809414af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.998+0000 7f8099fac640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80941551b0 con 0x7f80940bf9d0 2024-03-06T07:26:03.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.998+0000 7f8099fac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80940c6fa0 con 0x7f80940c87d0 2024-03-06T07:26:03.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.998+0000 7f8099fac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8094155490 con 0x7f8094152650 2024-03-06T07:26:03.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.998+0000 7f80937fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80940bf9d0 0x7f809414e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.998+0000 7f80937fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80940bf9d0 0x7f809414e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42624/0 (socket says 172.21.15.1:42624) 2024-03-06T07:26:03.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.998+0000 7f80937fe640 1 -- 172.21.15.1:0/1917938855 learned_addr learned my addr 172.21.15.1:0/1917938855 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:03.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.998+0000 7f8092ffd640 1 --2- 172.21.15.1:0/1917938855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8094152650 0x7f809414a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.998+0000 7f8093fff640 1 --2- 172.21.15.1:0/1917938855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80940c87d0 0x7f809414af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.998+0000 7f80937fe640 1 -- 172.21.15.1:0/1917938855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8094152650 msgr2=0x7f809414a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.998+0000 7f80937fe640 1 --2- 172.21.15.1:0/1917938855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8094152650 0x7f809414a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.998+0000 7f80937fe640 1 -- 172.21.15.1:0/1917938855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80940c87d0 msgr2=0x7f809414af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.998+0000 7f80937fe640 1 --2- 172.21.15.1:0/1917938855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80940c87d0 0x7f809414af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.998+0000 7f80937fe640 1 -- 172.21.15.1:0/1917938855 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8094158970 con 0x7f80940bf9d0 2024-03-06T07:26:03.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.998+0000 7f80937fe640 1 --2- 172.21.15.1:0/1917938855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80940bf9d0 0x7f809414e460 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f8084009870 tx=0x7f80840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.999+0000 7f807bfff640 1 -- 172.21.15.1:0/1917938855 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8084042070 con 0x7f80940bf9d0 2024-03-06T07:26:03.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.999+0000 7f807bfff640 1 -- 172.21.15.1:0/1917938855 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80840329c0 con 0x7f80940bf9d0 2024-03-06T07:26:03.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.999+0000 7f8099fac640 1 -- 172.21.15.1:0/1917938855 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f809414b7a0 con 0x7f80940bf9d0 2024-03-06T07:26:03.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.999+0000 7f807bfff640 1 -- 172.21.15.1:0/1917938855 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8084037040 con 0x7f80940bf9d0 2024-03-06T07:26:03.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:02.999+0000 7f8099fac640 1 -- 172.21.15.1:0/1917938855 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f809414e9a0 con 0x7f80940bf9d0 2024-03-06T07:26:03.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.000+0000 7f8099fac640 1 -- 172.21.15.1:0/1917938855 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8094067fd0 con 0x7f80940bf9d0 2024-03-06T07:26:03.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.000+0000 7f807bfff640 1 -- 172.21.15.1:0/1917938855 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8084045070 con 0x7f80940bf9d0 2024-03-06T07:26:03.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.000+0000 7f807bfff640 1 --2- 172.21.15.1:0/1917938855 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f808006b4d0 0x7f808006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.001+0000 7f807bfff640 1 -- 172.21.15.1:0/1917938855 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f80840ac310 con 0x7f80940bf9d0 2024-03-06T07:26:03.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.001+0000 7f8092ffd640 1 --2- 172.21.15.1:0/1917938855 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f808006b4d0 0x7f808006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.001+0000 7f8092ffd640 1 --2- 172.21.15.1:0/1917938855 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f808006b4d0 0x7f808006d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8088000f60 tx=0x7f8088009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.005+0000 7f807bfff640 1 -- 172.21.15.1:0/1917938855 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f808407aab0 con 0x7f80940bf9d0 2024-03-06T07:26:03.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.185+0000 7f8099fac640 1 -- 172.21.15.1:0/1917938855 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80940c38c0 con 0x7f80940bf9d0 2024-03-06T07:26:03.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.187+0000 7f807bfff640 1 -- 172.21.15.1:0/1917938855 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+17027 (secure 0 0 0) 0x7f808407eb50 con 0x7f80940bf9d0 2024-03-06T07:26:03.189 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:03.189 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":137,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:25:55.860450+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:28.046435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:28.753643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6835/2848968195":"2024-03-07T07:25:55.860421+0000","172.21.15.1:6833/2848968195":"2024-03-07T07:25:55.860421+0000","172.21.15.1:6834/1706279305":"2024-03-07T07:25:41.786393+0000","172.21.15.1:6832/1706279305":"2024-03-07T07:25:41.786393+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:03.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.189+0000 7f8099fac640 1 -- 172.21.15.1:0/1917938855 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f808006b4d0 msgr2=0x7f808006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.189+0000 7f8099fac640 1 --2- 172.21.15.1:0/1917938855 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f808006b4d0 0x7f808006d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8088000f60 tx=0x7f8088009040 comp rx=0 tx=0).stop 2024-03-06T07:26:03.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.189+0000 7f8099fac640 1 -- 172.21.15.1:0/1917938855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80940bf9d0 msgr2=0x7f809414e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.189+0000 7f8099fac640 1 --2- 172.21.15.1:0/1917938855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80940bf9d0 0x7f809414e460 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f8084009870 tx=0x7f80840046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.189+0000 7f8099fac640 1 -- 172.21.15.1:0/1917938855 shutdown_connections 2024-03-06T07:26:03.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.189+0000 7f8099fac640 1 --2- 172.21.15.1:0/1917938855 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f808006b4d0 0x7f808006d990 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.189+0000 7f8099fac640 1 --2- 172.21.15.1:0/1917938855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80940c87d0 0x7f809414af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.189+0000 7f8099fac640 1 --2- 172.21.15.1:0/1917938855 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8094152650 0x7f809414a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.189+0000 7f8099fac640 1 --2- 172.21.15.1:0/1917938855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80940bf9d0 0x7f809414e460 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.189+0000 7f8099fac640 1 -- 172.21.15.1:0/1917938855 >> 172.21.15.1:0/1917938855 conn(0x7f8094066ee0 msgr2=0x7f80940c0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.189+0000 7f8099fac640 1 -- 172.21.15.1:0/1917938855 shutdown_connections 2024-03-06T07:26:03.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.189+0000 7f8099fac640 1 -- 172.21.15.1:0/1917938855 wait complete. 2024-03-06T07:26:03.202 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:26:03.202 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:03.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.300+0000 7f03c47a0640 1 Processor -- start 2024-03-06T07:26:03.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.300+0000 7f03c47a0640 1 -- start start 2024-03-06T07:26:03.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.301+0000 7f03c47a0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f03bc152650 0x7f03bc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.301+0000 7f03c47a0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03bc0c6b10 0x7f03bc158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.301+0000 7f03c47a0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03bc0c2fc0 con 0x7f03bc0bf9d0 2024-03-06T07:26:03.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.301+0000 7f03c47a0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03bc155800 con 0x7f03bc0c6b10 2024-03-06T07:26:03.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.301+0000 7f03c47a0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03bc0c6690 con 0x7f03bc152650 2024-03-06T07:26:03.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.301+0000 7f03c1d3b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03bc0c6b10 0x7f03bc158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.301+0000 7f03c2d3d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f03bc0bf9d0 0x7f03bc0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38356/0 (socket says 172.21.15.1:38356) 2024-03-06T07:26:03.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.302+0000 7f03c2d3d640 1 -- 172.21.15.1:0/3220086136 learned_addr learned my addr 172.21.15.1:0/3220086136 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:03.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.302+0000 7f03c253c640 1 --2- 172.21.15.1:0/3220086136 >> v2:172.21.15.98:3300/0 conn(0x7f03bc152650 0x7f03bc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.302+0000 7f03c1d3b640 1 -- 172.21.15.1:0/3220086136 >> v2:172.21.15.98:3300/0 conn(0x7f03bc152650 msgr2=0x7f03bc155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.302+0000 7f03c1d3b640 1 --2- 172.21.15.1:0/3220086136 >> v2:172.21.15.98:3300/0 conn(0x7f03bc152650 0x7f03bc155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.302+0000 7f03c1d3b640 1 -- 172.21.15.1:0/3220086136 >> v1:172.21.15.1:6789/0 conn(0x7f03bc0bf9d0 legacy=0x7f03bc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.302+0000 7f03c1d3b640 1 -- 172.21.15.1:0/3220086136 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03bc0c7280 con 0x7f03bc0c6b10 2024-03-06T07:26:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.302+0000 7f03c153a640 1 -- 172.21.15.1:0/3220086136 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167519709 0 0) 0x7f03bc0c2fc0 con 0x7f03bc0bf9d0 2024-03-06T07:26:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.303+0000 7f03c1d3b640 1 --2- 172.21.15.1:0/3220086136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc0c6b10 0x7f03bc158690 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f03bc0cac90 tx=0x7f03ac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ab8134e8b176f4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.303+0000 7f03c153a640 1 -- 172.21.15.1:0/3220086136 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac0315a0 con 0x7f03bc0c6b10 2024-03-06T07:26:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.303+0000 7f03c153a640 1 -- 172.21.15.1:0/3220086136 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ac037070 con 0x7f03bc0c6b10 2024-03-06T07:26:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.303+0000 7f03c153a640 1 -- 172.21.15.1:0/3220086136 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac031960 con 0x7f03bc0c6b10 2024-03-06T07:26:03.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.303+0000 7f03c47a0640 1 -- 172.21.15.1:0/3220086136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc0c6b10 msgr2=0x7f03bc158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.303+0000 7f03c47a0640 1 --2- 172.21.15.1:0/3220086136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc0c6b10 0x7f03bc158690 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f03bc0cac90 tx=0x7f03ac02b840 comp rx=0 tx=0).stop 2024-03-06T07:26:03.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.303+0000 7f03c47a0640 1 -- 172.21.15.1:0/3220086136 shutdown_connections 2024-03-06T07:26:03.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.303+0000 7f03c47a0640 1 --2- 172.21.15.1:0/3220086136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc0c6b10 0x7f03bc158690 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.303+0000 7f03c47a0640 1 --2- 172.21.15.1:0/3220086136 >> v2:172.21.15.98:3300/0 conn(0x7f03bc152650 0x7f03bc155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.303+0000 7f03c47a0640 1 -- 172.21.15.1:0/3220086136 >> 172.21.15.1:0/3220086136 conn(0x7f03bc066ee0 msgr2=0x7f03bc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.303+0000 7f03c47a0640 1 -- 172.21.15.1:0/3220086136 shutdown_connections 2024-03-06T07:26:03.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.303+0000 7f03c47a0640 1 -- 172.21.15.1:0/3220086136 wait complete. 2024-03-06T07:26:03.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.304+0000 7f03c47a0640 1 Processor -- start 2024-03-06T07:26:03.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.304+0000 7f03c47a0640 1 -- start start 2024-03-06T07:26:03.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.305+0000 7f03c47a0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03bc0bf9d0 0x7f03bc158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.305+0000 7f03c47a0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc152650 0x7f03bc154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.305+0000 7f03c47a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03bc1443c0 0x7f03bc144850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.305+0000 7f03c47a0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc155490 con 0x7f03bc152650 2024-03-06T07:26:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.305+0000 7f03c47a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc1551b0 con 0x7f03bc1443c0 2024-03-06T07:26:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.305+0000 7f03c47a0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc152a30 con 0x7f03bc0bf9d0 2024-03-06T07:26:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.305+0000 7f03c1d3b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc152650 0x7f03bc154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.305+0000 7f03c1d3b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc152650 0x7f03bc154780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42642/0 (socket says 172.21.15.1:42642) 2024-03-06T07:26:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.305+0000 7f03c1d3b640 1 -- 172.21.15.1:0/45577374 learned_addr learned my addr 172.21.15.1:0/45577374 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.305+0000 7f03c253c640 1 --2- 172.21.15.1:0/45577374 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03bc0bf9d0 0x7f03bc158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.305+0000 7f03c2d3d640 1 --2- 172.21.15.1:0/45577374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03bc1443c0 0x7f03bc144850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.305+0000 7f03c1d3b640 1 -- 172.21.15.1:0/45577374 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03bc0bf9d0 msgr2=0x7f03bc158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.305+0000 7f03c1d3b640 1 --2- 172.21.15.1:0/45577374 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03bc0bf9d0 0x7f03bc158c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.305+0000 7f03c1d3b640 1 -- 172.21.15.1:0/45577374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03bc1443c0 msgr2=0x7f03bc144850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.305+0000 7f03c1d3b640 1 --2- 172.21.15.1:0/45577374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03bc1443c0 0x7f03bc144850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.305+0000 7f03c1d3b640 1 -- 172.21.15.1:0/45577374 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03bc14edc0 con 0x7f03bc152650 2024-03-06T07:26:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.306+0000 7f03c1d3b640 1 --2- 172.21.15.1:0/45577374 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc152650 0x7f03bc154780 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f03ac000c00 tx=0x7f03ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.306+0000 7f03b2ffd640 1 -- 172.21.15.1:0/45577374 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac032630 con 0x7f03bc152650 2024-03-06T07:26:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.306+0000 7f03b2ffd640 1 -- 172.21.15.1:0/45577374 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ac0327f0 con 0x7f03bc152650 2024-03-06T07:26:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.306+0000 7f03c47a0640 1 -- 172.21.15.1:0/45577374 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03bc1eeaf0 con 0x7f03bc152650 2024-03-06T07:26:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.306+0000 7f03b2ffd640 1 -- 172.21.15.1:0/45577374 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac032be0 con 0x7f03bc152650 2024-03-06T07:26:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.306+0000 7f03c47a0640 1 -- 172.21.15.1:0/45577374 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03bc1eedd0 con 0x7f03bc152650 2024-03-06T07:26:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.307+0000 7f03c47a0640 1 -- 172.21.15.1:0/45577374 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03bc067fd0 con 0x7f03bc152650 2024-03-06T07:26:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.307+0000 7f03b2ffd640 1 -- 172.21.15.1:0/45577374 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03ac04c020 con 0x7f03bc152650 2024-03-06T07:26:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.307+0000 7f03b2ffd640 1 --2- 172.21.15.1:0/45577374 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f03a006b500 0x7f03a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.308+0000 7f03b2ffd640 1 -- 172.21.15.1:0/45577374 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f03ac02f020 con 0x7f03bc152650 2024-03-06T07:26:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.308+0000 7f03c253c640 1 --2- 172.21.15.1:0/45577374 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f03a006b500 0x7f03a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.308+0000 7f03c253c640 1 --2- 172.21.15.1:0/45577374 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f03a006b500 0x7f03a006d9c0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f03bc0b0430 tx=0x7f03b8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.312+0000 7f03b2ffd640 1 -- 172.21.15.1:0/45577374 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f03ac040070 con 0x7f03bc152650 2024-03-06T07:26:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.530+0000 7f03c47a0640 1 -- 172.21.15.1:0/45577374 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03bc0b06f0 con 0x7f03bc152650 2024-03-06T07:26:03.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.530+0000 7f03b2ffd640 1 -- 172.21.15.1:0/45577374 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+4984 (secure 0 0 0) 0x7f03ac07a680 con 0x7f03bc152650 2024-03-06T07:26:03.532 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:03.532 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":131,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2739580331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2739580331},{"type":"v1","addr":"172.21.15.98:6837","nonce":2739580331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":8517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3374845636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3374845636},{"type":"v1","addr":"172.21.15.1:6834","nonce":3374845636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":8607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1602915919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1602915919},{"type":"v1","addr":"172.21.15.1:6835","nonce":1602915919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:29.792840+0000","modified":"2024-03-06T07:25:58.868171+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7387},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.98:6836/1671554096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1671554096},{"type":"v1","addr":"172.21.15.98:6836","nonce":1671554096}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7387,"qdb_cluster":[7387]},"id":10}]} 2024-03-06T07:26:03.533 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 131 2024-03-06T07:26:03.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.532+0000 7f03c47a0640 1 -- 172.21.15.1:0/45577374 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f03a006b500 msgr2=0x7f03a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.532+0000 7f03c47a0640 1 --2- 172.21.15.1:0/45577374 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f03a006b500 0x7f03a006d9c0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f03bc0b0430 tx=0x7f03b8009190 comp rx=0 tx=0).stop 2024-03-06T07:26:03.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.532+0000 7f03c47a0640 1 -- 172.21.15.1:0/45577374 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc152650 msgr2=0x7f03bc154780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.532+0000 7f03c47a0640 1 --2- 172.21.15.1:0/45577374 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc152650 0x7f03bc154780 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f03ac000c00 tx=0x7f03ac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.532+0000 7f03c47a0640 1 -- 172.21.15.1:0/45577374 shutdown_connections 2024-03-06T07:26:03.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.532+0000 7f03c47a0640 1 --2- 172.21.15.1:0/45577374 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f03a006b500 0x7f03a006d9c0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.532+0000 7f03c47a0640 1 --2- 172.21.15.1:0/45577374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03bc1443c0 0x7f03bc144850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.532+0000 7f03c47a0640 1 --2- 172.21.15.1:0/45577374 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc152650 0x7f03bc154780 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.532+0000 7f03c47a0640 1 --2- 172.21.15.1:0/45577374 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03bc0bf9d0 0x7f03bc158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.532+0000 7f03c47a0640 1 -- 172.21.15.1:0/45577374 >> 172.21.15.1:0/45577374 conn(0x7f03bc066ee0 msgr2=0x7f03bc1506a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.532+0000 7f03c47a0640 1 -- 172.21.15.1:0/45577374 shutdown_connections 2024-03-06T07:26:03.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.532+0000 7f03c47a0640 1 -- 172.21.15.1:0/45577374 wait complete. 2024-03-06T07:26:03.545 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:03.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.644+0000 7f07f653a640 1 Processor -- start 2024-03-06T07:26:03.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.644+0000 7f07f653a640 1 -- start start 2024-03-06T07:26:03.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.645+0000 7f07f653a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f07f0152730 0x7f07f00c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.645+0000 7f07f653a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f01588e0 con 0x7f07f00c6fd0 2024-03-06T07:26:03.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.645+0000 7f07f653a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f0158bc0 con 0x7f07f00c1aa0 2024-03-06T07:26:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.645+0000 7f07f653a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f0158ea0 con 0x7f07f0152730 2024-03-06T07:26:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.645+0000 7f07effff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f07f0152730 0x7f07f00c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.645+0000 7f07ef7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f07f00c6fd0 0x7f07f00c1390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38368/0 (socket says 172.21.15.1:38368) 2024-03-06T07:26:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.645+0000 7f07effff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f07f0152730 0x7f07f00c6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42656/0 (socket says 172.21.15.1:42656) 2024-03-06T07:26:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.645+0000 7f07ef7fe640 1 -- 172.21.15.1:0/3583073708 learned_addr learned my addr 172.21.15.1:0/3583073708 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.645+0000 7f07effff640 1 -- 172.21.15.1:0/3583073708 >> v1:172.21.15.98:6789/0 conn(0x7f07f00c1aa0 legacy=0x7f07f01552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.645+0000 7f07effff640 1 -- 172.21.15.1:0/3583073708 >> v1:172.21.15.1:6789/0 conn(0x7f07f00c6fd0 legacy=0x7f07f00c1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.645+0000 7f07effff640 1 -- 172.21.15.1:0/3583073708 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f0159180 con 0x7f07f0152730 2024-03-06T07:26:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.646+0000 7f07effff640 1 --2- 172.21.15.1:0/3583073708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f0152730 0x7f07f00c6a90 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f07e40097a0 tx=0x7f07e402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfbb2cf3f41f5083 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.646+0000 7f07eeffd640 1 -- 172.21.15.1:0/3583073708 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e40315a0 con 0x7f07f0152730 2024-03-06T07:26:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.646+0000 7f07eeffd640 1 -- 172.21.15.1:0/3583073708 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e4037070 con 0x7f07f0152730 2024-03-06T07:26:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.646+0000 7f07eeffd640 1 -- 172.21.15.1:0/3583073708 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e4031960 con 0x7f07f0152730 2024-03-06T07:26:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.647+0000 7f07f653a640 1 -- 172.21.15.1:0/3583073708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f0152730 msgr2=0x7f07f00c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.647+0000 7f07f653a640 1 --2- 172.21.15.1:0/3583073708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f0152730 0x7f07f00c6a90 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f07e40097a0 tx=0x7f07e402b840 comp rx=0 tx=0).stop 2024-03-06T07:26:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.647+0000 7f07f653a640 1 -- 172.21.15.1:0/3583073708 shutdown_connections 2024-03-06T07:26:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.647+0000 7f07f653a640 1 --2- 172.21.15.1:0/3583073708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f0152730 0x7f07f00c6a90 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.647+0000 7f07f653a640 1 -- 172.21.15.1:0/3583073708 >> 172.21.15.1:0/3583073708 conn(0x7f07f0066d00 msgr2=0x7f07f0067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.647+0000 7f07f653a640 1 -- 172.21.15.1:0/3583073708 shutdown_connections 2024-03-06T07:26:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.647+0000 7f07f653a640 1 -- 172.21.15.1:0/3583073708 wait complete. 2024-03-06T07:26:03.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.648+0000 7f07f653a640 1 Processor -- start 2024-03-06T07:26:03.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.648+0000 7f07f653a640 1 -- start start 2024-03-06T07:26:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07f653a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07f00c1aa0 0x7f07f01eb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07f653a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07f00c6fd0 0x7f07f0151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07f653a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f0152730 0x7f07f00c06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07f653a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f01559f0 con 0x7f07f0152730 2024-03-06T07:26:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07f653a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f0152b10 con 0x7f07f00c1aa0 2024-03-06T07:26:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07f653a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f0155b70 con 0x7f07f00c6fd0 2024-03-06T07:26:03.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07f4ad7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f0152730 0x7f07f00c06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07f4ad7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f0152730 0x7f07f00c06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42668/0 (socket says 172.21.15.1:42668) 2024-03-06T07:26:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07f4ad7640 1 -- 172.21.15.1:0/1273886442 learned_addr learned my addr 172.21.15.1:0/1273886442 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07effff640 1 --2- 172.21.15.1:0/1273886442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07f00c1aa0 0x7f07f01eb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07ef7fe640 1 --2- 172.21.15.1:0/1273886442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07f00c6fd0 0x7f07f0151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07f4ad7640 1 -- 172.21.15.1:0/1273886442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07f00c6fd0 msgr2=0x7f07f0151170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07f4ad7640 1 --2- 172.21.15.1:0/1273886442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07f00c6fd0 0x7f07f0151170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07f4ad7640 1 -- 172.21.15.1:0/1273886442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07f00c1aa0 msgr2=0x7f07f01eb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07f4ad7640 1 --2- 172.21.15.1:0/1273886442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07f00c1aa0 0x7f07f01eb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07f4ad7640 1 -- 172.21.15.1:0/1273886442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f00b0700 con 0x7f07f0152730 2024-03-06T07:26:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07f4ad7640 1 --2- 172.21.15.1:0/1273886442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f0152730 0x7f07f00c06f0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f07e800c760 tx=0x7f07e800cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07ecff9640 1 -- 172.21.15.1:0/1273886442 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e8003b70 con 0x7f07f0152730 2024-03-06T07:26:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07ecff9640 1 -- 172.21.15.1:0/1273886442 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e8003d10 con 0x7f07f0152730 2024-03-06T07:26:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07f653a640 1 -- 172.21.15.1:0/1273886442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f01eebc0 con 0x7f07f0152730 2024-03-06T07:26:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07ecff9640 1 -- 172.21.15.1:0/1273886442 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e8005310 con 0x7f07f0152730 2024-03-06T07:26:03.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.649+0000 7f07f653a640 1 -- 172.21.15.1:0/1273886442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f01eaf00 con 0x7f07f0152730 2024-03-06T07:26:03.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.650+0000 7f07f653a640 1 -- 172.21.15.1:0/1273886442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f0067fd0 con 0x7f07f0152730 2024-03-06T07:26:03.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.651+0000 7f07ecff9640 1 -- 172.21.15.1:0/1273886442 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f07e80054b0 con 0x7f07f0152730 2024-03-06T07:26:03.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.651+0000 7f07ecff9640 1 --2- 172.21.15.1:0/1273886442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f07dc06b4d0 0x7f07dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.651+0000 7f07ecff9640 1 -- 172.21.15.1:0/1273886442 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f07e808a140 con 0x7f07f0152730 2024-03-06T07:26:03.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.651+0000 7f07effff640 1 --2- 172.21.15.1:0/1273886442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f07dc06b4d0 0x7f07dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.652+0000 7f07effff640 1 --2- 172.21.15.1:0/1273886442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f07dc06b4d0 0x7f07dc06d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f07e4036000 tx=0x7f07e403d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.655+0000 7f07ecff9640 1 -- 172.21.15.1:0/1273886442 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f07f00b0700 con 0x7f07f0152730 2024-03-06T07:26:03.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.878+0000 7f07f653a640 1 -- 172.21.15.1:0/1273886442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07f00c6870 con 0x7f07f0152730 2024-03-06T07:26:03.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.878+0000 7f07ecff9640 1 -- 172.21.15.1:0/1273886442 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+4984 (secure 0 0 0) 0x7f07e8058720 con 0x7f07f0152730 2024-03-06T07:26:03.881 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:03.881 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":131,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2739580331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2739580331},{"type":"v1","addr":"172.21.15.98:6837","nonce":2739580331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":8517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3374845636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3374845636},{"type":"v1","addr":"172.21.15.1:6834","nonce":3374845636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":8607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1602915919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1602915919},{"type":"v1","addr":"172.21.15.1:6835","nonce":1602915919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:29.792840+0000","modified":"2024-03-06T07:25:58.868171+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7387},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.98:6836/1671554096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1671554096},{"type":"v1","addr":"172.21.15.98:6836","nonce":1671554096}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7387,"qdb_cluster":[7387]},"id":10}]} 2024-03-06T07:26:03.881 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 131 2024-03-06T07:26:03.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.880+0000 7f07f653a640 1 -- 172.21.15.1:0/1273886442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f07dc06b4d0 msgr2=0x7f07dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.880+0000 7f07f653a640 1 --2- 172.21.15.1:0/1273886442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f07dc06b4d0 0x7f07dc06d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f07e4036000 tx=0x7f07e403d040 comp rx=0 tx=0).stop 2024-03-06T07:26:03.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.880+0000 7f07f653a640 1 -- 172.21.15.1:0/1273886442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f0152730 msgr2=0x7f07f00c06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.880+0000 7f07f653a640 1 --2- 172.21.15.1:0/1273886442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f0152730 0x7f07f00c06f0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f07e800c760 tx=0x7f07e800cb60 comp rx=0 tx=0).stop 2024-03-06T07:26:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.880+0000 7f07f653a640 1 -- 172.21.15.1:0/1273886442 shutdown_connections 2024-03-06T07:26:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.880+0000 7f07f653a640 1 --2- 172.21.15.1:0/1273886442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f07dc06b4d0 0x7f07dc06d990 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.880+0000 7f07f653a640 1 --2- 172.21.15.1:0/1273886442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f0152730 0x7f07f00c06f0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.880+0000 7f07f653a640 1 --2- 172.21.15.1:0/1273886442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f07f00c6fd0 0x7f07f0151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.880+0000 7f07f653a640 1 --2- 172.21.15.1:0/1273886442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07f00c1aa0 0x7f07f01eb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.880+0000 7f07f653a640 1 -- 172.21.15.1:0/1273886442 >> 172.21.15.1:0/1273886442 conn(0x7f07f0066d00 msgr2=0x7f07f0155280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.881+0000 7f07f653a640 1 -- 172.21.15.1:0/1273886442 shutdown_connections 2024-03-06T07:26:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.881+0000 7f07f653a640 1 -- 172.21.15.1:0/1273886442 wait complete. 2024-03-06T07:26:03.893 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:03.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.992+0000 7febcadd0640 1 Processor -- start 2024-03-06T07:26:03.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.992+0000 7febcadd0640 1 -- start start 2024-03-06T07:26:03.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.993+0000 7febcadd0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7febc406d060 0x7febc41558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.993+0000 7febcadd0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc40c71c0 con 0x7febc40c3eb0 2024-03-06T07:26:03.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.993+0000 7febcadd0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc40c4e50 con 0x7febc40af640 2024-03-06T07:26:03.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.993+0000 7febcadd0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febc40c59b0 con 0x7febc406d060 2024-03-06T07:26:03.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.993+0000 7febc3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7febc406d060 0x7febc41558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.993+0000 7febc8b6c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7febc40c3eb0 0x7febc40c7400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38382/0 (socket says 172.21.15.1:38382) 2024-03-06T07:26:03.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.993+0000 7febc8b6c640 1 -- 172.21.15.1:0/1508521455 learned_addr learned my addr 172.21.15.1:0/1508521455 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:03.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.994+0000 7febc3fff640 1 -- 172.21.15.1:0/1508521455 >> v1:172.21.15.98:6789/0 conn(0x7febc40af640 legacy=0x7febc415fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.994+0000 7febc3fff640 1 -- 172.21.15.1:0/1508521455 >> v1:172.21.15.1:6789/0 conn(0x7febc40c3eb0 legacy=0x7febc40c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.994+0000 7febc3fff640 1 -- 172.21.15.1:0/1508521455 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc40c6ee0 con 0x7febc406d060 2024-03-06T07:26:03.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.994+0000 7febc3fff640 1 --2- 172.21.15.1:0/1508521455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febc406d060 0x7febc41558b0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7febb80097a0 tx=0x7febb802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf500fd5e54b331 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.994+0000 7febc37fe640 1 -- 172.21.15.1:0/1508521455 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb80315a0 con 0x7febc406d060 2024-03-06T07:26:03.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.994+0000 7febc37fe640 1 -- 172.21.15.1:0/1508521455 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb8037070 con 0x7febc406d060 2024-03-06T07:26:03.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.994+0000 7febc37fe640 1 -- 172.21.15.1:0/1508521455 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb8031960 con 0x7febc406d060 2024-03-06T07:26:03.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.995+0000 7febcadd0640 1 -- 172.21.15.1:0/1508521455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febc406d060 msgr2=0x7febc41558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.995+0000 7febcadd0640 1 --2- 172.21.15.1:0/1508521455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febc406d060 0x7febc41558b0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7febb80097a0 tx=0x7febb802b840 comp rx=0 tx=0).stop 2024-03-06T07:26:03.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.995+0000 7febcadd0640 1 -- 172.21.15.1:0/1508521455 shutdown_connections 2024-03-06T07:26:03.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.995+0000 7febcadd0640 1 --2- 172.21.15.1:0/1508521455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febc406d060 0x7febc41558b0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.995+0000 7febcadd0640 1 -- 172.21.15.1:0/1508521455 >> 172.21.15.1:0/1508521455 conn(0x7febc4066d00 msgr2=0x7febc4067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.995+0000 7febcadd0640 1 -- 172.21.15.1:0/1508521455 shutdown_connections 2024-03-06T07:26:03.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.995+0000 7febcadd0640 1 -- 172.21.15.1:0/1508521455 wait complete. 2024-03-06T07:26:04.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.996+0000 7febcadd0640 1 Processor -- start 2024-03-06T07:26:04.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.996+0000 7febcadd0640 1 -- start start 2024-03-06T07:26:04.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.997+0000 7febcadd0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febc406d060 0x7febc4149f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.997+0000 7febcadd0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febc40af640 0x7febc4153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.997+0000 7febcadd0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febc40c3eb0 0x7febc41ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.997+0000 7febcadd0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc4155420 con 0x7febc40c3eb0 2024-03-06T07:26:04.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.997+0000 7febcadd0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc40c4c00 con 0x7febc406d060 2024-03-06T07:26:04.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.997+0000 7febcadd0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc40c4290 con 0x7febc40af640 2024-03-06T07:26:04.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.997+0000 7febc936d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febc40c3eb0 0x7febc41ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.997+0000 7febc936d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febc40c3eb0 0x7febc41ee590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42684/0 (socket says 172.21.15.1:42684) 2024-03-06T07:26:04.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.997+0000 7febc936d640 1 -- 172.21.15.1:0/3767025767 learned_addr learned my addr 172.21.15.1:0/3767025767 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.997+0000 7febc8b6c640 1 --2- 172.21.15.1:0/3767025767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febc406d060 0x7febc4149f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.997+0000 7febc3fff640 1 --2- 172.21.15.1:0/3767025767 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febc40af640 0x7febc4153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.997+0000 7febc936d640 1 -- 172.21.15.1:0/3767025767 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febc40af640 msgr2=0x7febc4153bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.997+0000 7febc936d640 1 --2- 172.21.15.1:0/3767025767 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febc40af640 0x7febc4153bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.997+0000 7febc936d640 1 -- 172.21.15.1:0/3767025767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febc406d060 msgr2=0x7febc4149f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.997+0000 7febc936d640 1 --2- 172.21.15.1:0/3767025767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febc406d060 0x7febc4149f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.997+0000 7febc936d640 1 -- 172.21.15.1:0/3767025767 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc4148c50 con 0x7febc40c3eb0 2024-03-06T07:26:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.998+0000 7febc936d640 1 --2- 172.21.15.1:0/3767025767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febc40c3eb0 0x7febc41ee590 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7febb400c390 tx=0x7febb400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.998+0000 7febc17fa640 1 -- 172.21.15.1:0/3767025767 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb4017070 con 0x7febc40c3eb0 2024-03-06T07:26:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.998+0000 7febc17fa640 1 -- 172.21.15.1:0/3767025767 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb40052d0 con 0x7febc40c3eb0 2024-03-06T07:26:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.998+0000 7febcadd0640 1 -- 172.21.15.1:0/3767025767 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc41ee0b0 con 0x7febc40c3eb0 2024-03-06T07:26:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.998+0000 7febc17fa640 1 -- 172.21.15.1:0/3767025767 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb4003810 con 0x7febc40c3eb0 2024-03-06T07:26:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.998+0000 7febcadd0640 1 -- 172.21.15.1:0/3767025767 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febc4150740 con 0x7febc40c3eb0 2024-03-06T07:26:04.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.999+0000 7febcadd0640 1 -- 172.21.15.1:0/3767025767 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc41ee280 con 0x7febc40c3eb0 2024-03-06T07:26:04.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:03.999+0000 7febc17fa640 1 -- 172.21.15.1:0/3767025767 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7febb401a070 con 0x7febc40c3eb0 2024-03-06T07:26:04.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.000+0000 7febc17fa640 1 --2- 172.21.15.1:0/3767025767 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febb006b4d0 0x7febb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.000+0000 7febc17fa640 1 -- 172.21.15.1:0/3767025767 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7febb4089650 con 0x7febc40c3eb0 2024-03-06T07:26:04.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.000+0000 7febc8b6c640 1 --2- 172.21.15.1:0/3767025767 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febb006b4d0 0x7febb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.000+0000 7febc8b6c640 1 --2- 172.21.15.1:0/3767025767 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febb006b4d0 0x7febb006d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7febbc005c00 tx=0x7febbc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.004+0000 7febc17fa640 1 -- 172.21.15.1:0/3767025767 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7febb4057cd0 con 0x7febc40c3eb0 2024-03-06T07:26:04.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.187+0000 7febcadd0640 1 -- 172.21.15.1:0/3767025767 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febc4067fd0 con 0x7febc40c3eb0 2024-03-06T07:26:04.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.188+0000 7febc17fa640 1 -- 172.21.15.1:0/3767025767 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+17027 (secure 0 0 0) 0x7febb405bd70 con 0x7febc40c3eb0 2024-03-06T07:26:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:04.190 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":137,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:25:55.860450+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:28.046435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:28.753643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6835/2848968195":"2024-03-07T07:25:55.860421+0000","172.21.15.1:6833/2848968195":"2024-03-07T07:25:55.860421+0000","172.21.15.1:6834/1706279305":"2024-03-07T07:25:41.786393+0000","172.21.15.1:6832/1706279305":"2024-03-07T07:25:41.786393+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:04.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.190+0000 7febcadd0640 1 -- 172.21.15.1:0/3767025767 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febb006b4d0 msgr2=0x7febb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.190+0000 7febcadd0640 1 --2- 172.21.15.1:0/3767025767 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febb006b4d0 0x7febb006d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7febbc005c00 tx=0x7febbc009040 comp rx=0 tx=0).stop 2024-03-06T07:26:04.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.190+0000 7febcadd0640 1 -- 172.21.15.1:0/3767025767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febc40c3eb0 msgr2=0x7febc41ee590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.190+0000 7febcadd0640 1 --2- 172.21.15.1:0/3767025767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febc40c3eb0 0x7febc41ee590 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7febb400c390 tx=0x7febb400c850 comp rx=0 tx=0).stop 2024-03-06T07:26:04.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.190+0000 7febcadd0640 1 -- 172.21.15.1:0/3767025767 shutdown_connections 2024-03-06T07:26:04.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.190+0000 7febcadd0640 1 --2- 172.21.15.1:0/3767025767 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febb006b4d0 0x7febb006d990 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.190+0000 7febcadd0640 1 --2- 172.21.15.1:0/3767025767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febc40c3eb0 0x7febc41ee590 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.190+0000 7febcadd0640 1 --2- 172.21.15.1:0/3767025767 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febc40af640 0x7febc4153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.190+0000 7febcadd0640 1 --2- 172.21.15.1:0/3767025767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febc406d060 0x7febc4149f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.190+0000 7febcadd0640 1 -- 172.21.15.1:0/3767025767 >> 172.21.15.1:0/3767025767 conn(0x7febc4066d00 msgr2=0x7febc4152c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.190+0000 7febcadd0640 1 -- 172.21.15.1:0/3767025767 shutdown_connections 2024-03-06T07:26:04.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.190+0000 7febcadd0640 1 -- 172.21.15.1:0/3767025767 wait complete. 2024-03-06T07:26:04.203 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:26:04.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.303+0000 7f82e7f1e640 1 Processor -- start 2024-03-06T07:26:04.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.303+0000 7f82e7f1e640 1 -- start start 2024-03-06T07:26:04.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.304+0000 7f82e7f1e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82e00bf9d0 0x7f82e0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.304+0000 7f82e7f1e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e00c2fc0 con 0x7f82e00c6b10 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.304+0000 7f82e7f1e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e0158da0 con 0x7f82e0152650 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.304+0000 7f82e7f1e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e0155800 con 0x7f82e00bf9d0 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.304+0000 7f82e64bb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82e00bf9d0 0x7f82e0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.304+0000 7f82e54b9640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f82e00c6b10 0x7f82e00c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38390/0 (socket says 172.21.15.1:38390) 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.304+0000 7f82e64bb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82e00bf9d0 0x7f82e0155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42700/0 (socket says 172.21.15.1:42700) 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.304+0000 7f82e54b9640 1 -- 172.21.15.1:0/3032833874 learned_addr learned my addr 172.21.15.1:0/3032833874 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.305+0000 7f82e64bb640 1 -- 172.21.15.1:0/3032833874 >> v1:172.21.15.98:6789/0 conn(0x7f82e0152650 legacy=0x7f82e0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.305+0000 7f82e64bb640 1 -- 172.21.15.1:0/3032833874 >> v1:172.21.15.1:6789/0 conn(0x7f82e00c6b10 legacy=0x7f82e00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.305+0000 7f82e4cb8640 1 -- 172.21.15.1:0/3032833874 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076594556 0 0) 0x7f82e0158da0 con 0x7f82e0152650 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.305+0000 7f82e64bb640 1 -- 172.21.15.1:0/3032833874 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e00c6690 con 0x7f82e00bf9d0 2024-03-06T07:26:04.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.305+0000 7f82e64bb640 1 --2- 172.21.15.1:0/3032833874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00bf9d0 0x7f82e0155c90 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f82d8009d90 tx=0x7f82d802bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c34a8ecb9ab9119 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.306+0000 7f82e4cb8640 1 -- 172.21.15.1:0/3032833874 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d8039070 con 0x7f82e00bf9d0 2024-03-06T07:26:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.306+0000 7f82e4cb8640 1 -- 172.21.15.1:0/3032833874 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d802ca20 con 0x7f82e00bf9d0 2024-03-06T07:26:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.306+0000 7f82e4cb8640 1 -- 172.21.15.1:0/3032833874 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d802cdf0 con 0x7f82e00bf9d0 2024-03-06T07:26:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.306+0000 7f82e7f1e640 1 -- 172.21.15.1:0/3032833874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00bf9d0 msgr2=0x7f82e0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.306+0000 7f82e7f1e640 1 --2- 172.21.15.1:0/3032833874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00bf9d0 0x7f82e0155c90 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f82d8009d90 tx=0x7f82d802bcf0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.306+0000 7f82e7f1e640 1 -- 172.21.15.1:0/3032833874 shutdown_connections 2024-03-06T07:26:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.306+0000 7f82e7f1e640 1 --2- 172.21.15.1:0/3032833874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00bf9d0 0x7f82e0155c90 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.306+0000 7f82e7f1e640 1 -- 172.21.15.1:0/3032833874 >> 172.21.15.1:0/3032833874 conn(0x7f82e0066ee0 msgr2=0x7f82e00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.306+0000 7f82e7f1e640 1 -- 172.21.15.1:0/3032833874 shutdown_connections 2024-03-06T07:26:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.306+0000 7f82e7f1e640 1 -- 172.21.15.1:0/3032833874 wait complete. 2024-03-06T07:26:04.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.307+0000 7f82e7f1e640 1 Processor -- start 2024-03-06T07:26:04.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.307+0000 7f82e7f1e640 1 -- start start 2024-03-06T07:26:04.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.308+0000 7f82e7f1e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00bf9d0 0x7f82e01ee290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.308+0000 7f82e7f1e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82e0152650 0x7f82e0152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.308+0000 7f82e7f1e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82e0153420 0x7f82e014a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.308+0000 7f82e7f1e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e01551b0 con 0x7f82e00bf9d0 2024-03-06T07:26:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.308+0000 7f82e7f1e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e00c6fa0 con 0x7f82e0152650 2024-03-06T07:26:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.308+0000 7f82e7f1e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e0155490 con 0x7f82e0153420 2024-03-06T07:26:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.308+0000 7f82e5cba640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00bf9d0 0x7f82e01ee290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.308+0000 7f82e5cba640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00bf9d0 0x7f82e01ee290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42710/0 (socket says 172.21.15.1:42710) 2024-03-06T07:26:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.308+0000 7f82e5cba640 1 -- 172.21.15.1:0/1828917877 learned_addr learned my addr 172.21.15.1:0/1828917877 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.308+0000 7f82e64bb640 1 --2- 172.21.15.1:0/1828917877 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82e0153420 0x7f82e014a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.308+0000 7f82e54b9640 1 --2- 172.21.15.1:0/1828917877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82e0152650 0x7f82e0152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.308+0000 7f82e5cba640 1 -- 172.21.15.1:0/1828917877 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82e0153420 msgr2=0x7f82e014a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.308+0000 7f82e5cba640 1 --2- 172.21.15.1:0/1828917877 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82e0153420 0x7f82e014a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.308+0000 7f82e5cba640 1 -- 172.21.15.1:0/1828917877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82e0152650 msgr2=0x7f82e0152a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.308+0000 7f82e5cba640 1 --2- 172.21.15.1:0/1828917877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82e0152650 0x7f82e0152a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.308+0000 7f82e5cba640 1 -- 172.21.15.1:0/1828917877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e01587d0 con 0x7f82e00bf9d0 2024-03-06T07:26:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.309+0000 7f82e5cba640 1 --2- 172.21.15.1:0/1828917877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00bf9d0 0x7f82e01ee290 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00c4e0 tx=0x7f82dc00c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.309+0000 7f82d67fc640 1 -- 172.21.15.1:0/1828917877 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc017070 con 0x7f82e00bf9d0 2024-03-06T07:26:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.309+0000 7f82d67fc640 1 -- 172.21.15.1:0/1828917877 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82dc0035f0 con 0x7f82e00bf9d0 2024-03-06T07:26:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.309+0000 7f82e7f1e640 1 -- 172.21.15.1:0/1828917877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e01577d0 con 0x7f82e00bf9d0 2024-03-06T07:26:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.309+0000 7f82d67fc640 1 -- 172.21.15.1:0/1828917877 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc0039e0 con 0x7f82e00bf9d0 2024-03-06T07:26:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.309+0000 7f82e7f1e640 1 -- 172.21.15.1:0/1828917877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e01ee7d0 con 0x7f82e00bf9d0 2024-03-06T07:26:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.310+0000 7f82e7f1e640 1 -- 172.21.15.1:0/1828917877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e0067fd0 con 0x7f82e00bf9d0 2024-03-06T07:26:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.310+0000 7f82d67fc640 1 -- 172.21.15.1:0/1828917877 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82dc01a070 con 0x7f82e00bf9d0 2024-03-06T07:26:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.311+0000 7f82d67fc640 1 --2- 172.21.15.1:0/1828917877 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f82c406b500 0x7f82c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.311+0000 7f82d67fc640 1 -- 172.21.15.1:0/1828917877 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f82dc089660 con 0x7f82e00bf9d0 2024-03-06T07:26:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.311+0000 7f82e54b9640 1 --2- 172.21.15.1:0/1828917877 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f82c406b500 0x7f82c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.311+0000 7f82e54b9640 1 --2- 172.21.15.1:0/1828917877 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f82c406b500 0x7f82c406d9c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f82e0152060 tx=0x7f82d0007e20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.315+0000 7f82d67fc640 1 -- 172.21.15.1:0/1828917877 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82dc057c50 con 0x7f82e00bf9d0 2024-03-06T07:26:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.533+0000 7f82e7f1e640 1 -- 172.21.15.1:0/1828917877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f82e00b06f0 con 0x7f82e00bf9d0 2024-03-06T07:26:04.535 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:26:04.534+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:04.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.534+0000 7f82d67fc640 1 -- 172.21.15.1:0/1828917877 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v132) v1 ==== 161+0+0 (secure 0 0 0) 0x7f82dc05bcf0 con 0x7f82e00bf9d0 2024-03-06T07:26:04.537 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:26:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.537+0000 7f82e7f1e640 1 -- 172.21.15.1:0/1828917877 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f82c406b500 msgr2=0x7f82c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.537+0000 7f82e7f1e640 1 --2- 172.21.15.1:0/1828917877 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f82c406b500 0x7f82c406d9c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f82e0152060 tx=0x7f82d0007e20 comp rx=0 tx=0).stop 2024-03-06T07:26:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.537+0000 7f82e7f1e640 1 -- 172.21.15.1:0/1828917877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00bf9d0 msgr2=0x7f82e01ee290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.537+0000 7f82e7f1e640 1 --2- 172.21.15.1:0/1828917877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00bf9d0 0x7f82e01ee290 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00c4e0 tx=0x7f82dc00c9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.537+0000 7f82e7f1e640 1 -- 172.21.15.1:0/1828917877 shutdown_connections 2024-03-06T07:26:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.537+0000 7f82e7f1e640 1 --2- 172.21.15.1:0/1828917877 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f82c406b500 0x7f82c406d9c0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.537+0000 7f82e7f1e640 1 --2- 172.21.15.1:0/1828917877 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82e0153420 0x7f82e014a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.537+0000 7f82e7f1e640 1 --2- 172.21.15.1:0/1828917877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82e0152650 0x7f82e0152a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.537+0000 7f82e7f1e640 1 --2- 172.21.15.1:0/1828917877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00bf9d0 0x7f82e01ee290 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.537+0000 7f82e7f1e640 1 -- 172.21.15.1:0/1828917877 >> 172.21.15.1:0/1828917877 conn(0x7f82e0066ee0 msgr2=0x7f82e00caee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.537+0000 7f82e7f1e640 1 -- 172.21.15.1:0/1828917877 shutdown_connections 2024-03-06T07:26:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.537+0000 7f82e7f1e640 1 -- 172.21.15.1:0/1828917877 wait complete. 2024-03-06T07:26:04.543 INFO:tasks.ceph.mds.d.smithi098.stderr: -122> 2024-03-06T07:26:00.715+0000 7fc40a923640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:04.550 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:26:04.566 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:26:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.652+0000 7f72a78c9640 1 Processor -- start 2024-03-06T07:26:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.652+0000 7f72a78c9640 1 -- start start 2024-03-06T07:26:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.653+0000 7f72a78c9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f72a006d060 0x7f72a01558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.653+0000 7f72a78c9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f72a00af640 0x7f72a0155df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.653+0000 7f72a78c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a00c71c0 con 0x7f72a00c3eb0 2024-03-06T07:26:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.653+0000 7f72a78c9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a00c59b0 con 0x7f72a006d060 2024-03-06T07:26:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.653+0000 7f72a78c9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a00c4c00 con 0x7f72a00af640 2024-03-06T07:26:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.653+0000 7f72a4e64640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f72a006d060 0x7f72a01558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.653+0000 7f72a4e64640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f72a006d060 0x7f72a01558b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33212/0 (socket says 172.21.15.1:33212) 2024-03-06T07:26:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.653+0000 7f72a4e64640 1 -- 172.21.15.1:0/1335411997 learned_addr learned my addr 172.21.15.1:0/1335411997 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.653+0000 7f72a5e66640 1 --2- 172.21.15.1:0/1335411997 >> v2:172.21.15.98:3300/0 conn(0x7f72a00af640 0x7f72a0155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.653+0000 7f72a4e64640 1 -- 172.21.15.1:0/1335411997 >> v2:172.21.15.98:3300/0 conn(0x7f72a00af640 msgr2=0x7f72a0155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.653+0000 7f72a4e64640 1 --2- 172.21.15.1:0/1335411997 >> v2:172.21.15.98:3300/0 conn(0x7f72a00af640 0x7f72a0155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.653+0000 7f72a4e64640 1 -- 172.21.15.1:0/1335411997 >> v1:172.21.15.98:6789/0 conn(0x7f72a00c3eb0 legacy=0x7f72a00c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.653+0000 7f72a4e64640 1 -- 172.21.15.1:0/1335411997 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a00c4e50 con 0x7f72a006d060 2024-03-06T07:26:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.654+0000 7f72a4e64640 1 --2- 172.21.15.1:0/1335411997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72a006d060 0x7f72a01558b0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f72a00c1cb0 tx=0x7f729002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8abe0e915c9ad17 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.654+0000 7f7297fff640 1 -- 172.21.15.1:0/1335411997 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729002e070 con 0x7f72a006d060 2024-03-06T07:26:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.654+0000 7f7297fff640 1 -- 172.21.15.1:0/1335411997 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7290037070 con 0x7f72a006d060 2024-03-06T07:26:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.654+0000 7f7297fff640 1 -- 172.21.15.1:0/1335411997 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7290031a00 con 0x7f72a006d060 2024-03-06T07:26:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.655+0000 7f72a78c9640 1 -- 172.21.15.1:0/1335411997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72a006d060 msgr2=0x7f72a01558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.655+0000 7f72a78c9640 1 --2- 172.21.15.1:0/1335411997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72a006d060 0x7f72a01558b0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f72a00c1cb0 tx=0x7f729002b880 comp rx=0 tx=0).stop 2024-03-06T07:26:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.655+0000 7f72a78c9640 1 -- 172.21.15.1:0/1335411997 shutdown_connections 2024-03-06T07:26:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.655+0000 7f72a78c9640 1 --2- 172.21.15.1:0/1335411997 >> v2:172.21.15.98:3300/0 conn(0x7f72a00af640 0x7f72a0155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.655+0000 7f72a78c9640 1 --2- 172.21.15.1:0/1335411997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72a006d060 0x7f72a01558b0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.655+0000 7f72a78c9640 1 -- 172.21.15.1:0/1335411997 >> 172.21.15.1:0/1335411997 conn(0x7f72a0066d00 msgr2=0x7f72a0067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.655+0000 7f72a78c9640 1 -- 172.21.15.1:0/1335411997 shutdown_connections 2024-03-06T07:26:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.655+0000 7f72a78c9640 1 -- 172.21.15.1:0/1335411997 wait complete. 2024-03-06T07:26:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.656+0000 7f72a78c9640 1 Processor -- start 2024-03-06T07:26:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.656+0000 7f72a78c9640 1 -- start start 2024-03-06T07:26:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a78c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72a006d060 0x7f72a0157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a78c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72a00af640 0x7f72a00aa230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a78c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72a00c3eb0 0x7f72a01eba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a78c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a00c7b10 con 0x7f72a006d060 2024-03-06T07:26:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a78c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a0155420 con 0x7f72a00af640 2024-03-06T07:26:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a78c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a00c7c90 con 0x7f72a00c3eb0 2024-03-06T07:26:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a5665640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72a006d060 0x7f72a0157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a5665640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72a006d060 0x7f72a0157a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33228/0 (socket says 172.21.15.1:33228) 2024-03-06T07:26:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a5665640 1 -- 172.21.15.1:0/2407513074 learned_addr learned my addr 172.21.15.1:0/2407513074 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a5e66640 1 --2- 172.21.15.1:0/2407513074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72a00c3eb0 0x7f72a01eba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a4e64640 1 --2- 172.21.15.1:0/2407513074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72a00af640 0x7f72a00aa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a5665640 1 -- 172.21.15.1:0/2407513074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72a00c3eb0 msgr2=0x7f72a01eba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a5665640 1 --2- 172.21.15.1:0/2407513074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72a00c3eb0 0x7f72a01eba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a5665640 1 -- 172.21.15.1:0/2407513074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72a00af640 msgr2=0x7f72a00aa230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a5665640 1 --2- 172.21.15.1:0/2407513074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72a00af640 0x7f72a00aa230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a5665640 1 -- 172.21.15.1:0/2407513074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a0158030 con 0x7f72a006d060 2024-03-06T07:26:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a5e66640 1 --2- 172.21.15.1:0/2407513074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72a00c3eb0 0x7f72a01eba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a5665640 1 --2- 172.21.15.1:0/2407513074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72a006d060 0x7f72a0157a50 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f729800c790 tx=0x7f729800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f7295ffb640 1 -- 172.21.15.1:0/2407513074 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7298016020 con 0x7f72a006d060 2024-03-06T07:26:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f7295ffb640 1 -- 172.21.15.1:0/2407513074 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7298006d50 con 0x7f72a006d060 2024-03-06T07:26:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f72a78c9640 1 -- 172.21.15.1:0/2407513074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a0158310 con 0x7f72a006d060 2024-03-06T07:26:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.657+0000 7f7295ffb640 1 -- 172.21.15.1:0/2407513074 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7298003bb0 con 0x7f72a006d060 2024-03-06T07:26:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.658+0000 7f72a78c9640 1 -- 172.21.15.1:0/2407513074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a01ee9b0 con 0x7f72a006d060 2024-03-06T07:26:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.659+0000 7f72a78c9640 1 -- 172.21.15.1:0/2407513074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a00c0c60 con 0x7f72a006d060 2024-03-06T07:26:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.659+0000 7f7295ffb640 1 -- 172.21.15.1:0/2407513074 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f729800a850 con 0x7f72a006d060 2024-03-06T07:26:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.659+0000 7f7295ffb640 1 --2- 172.21.15.1:0/2407513074 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f728406b500 0x7f728406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.659+0000 7f7295ffb640 1 -- 172.21.15.1:0/2407513074 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6359+0+0 (secure 0 0 0) 0x7f729808a120 con 0x7f72a006d060 2024-03-06T07:26:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.659+0000 7f72a4e64640 1 --2- 172.21.15.1:0/2407513074 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f728406b500 0x7f728406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.660+0000 7f72a4e64640 1 --2- 172.21.15.1:0/2407513074 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f728406b500 0x7f728406d9c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7290036000 tx=0x7f729003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.664+0000 7f7295ffb640 1 -- 172.21.15.1:0/2407513074 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f72a00c0c60 con 0x7f72a006d060 2024-03-06T07:26:04.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:04.883+0000 7f72a78c9640 1 -- 172.21.15.1:0/2407513074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f72a0067fd0 con 0x7f72a006d060 2024-03-06T07:26:05.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.541+0000 7f7295ffb640 1 -- 172.21.15.1:0/2407513074 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v133) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7298020020 con 0x7f72a006d060 2024-03-06T07:26:05.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.544+0000 7f72a78c9640 1 -- 172.21.15.1:0/2407513074 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f728406b500 msgr2=0x7f728406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.544+0000 7f72a78c9640 1 --2- 172.21.15.1:0/2407513074 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f728406b500 0x7f728406d9c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7290036000 tx=0x7f729003d040 comp rx=0 tx=0).stop 2024-03-06T07:26:05.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.544+0000 7f72a78c9640 1 -- 172.21.15.1:0/2407513074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72a006d060 msgr2=0x7f72a0157a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.544+0000 7f72a78c9640 1 --2- 172.21.15.1:0/2407513074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72a006d060 0x7f72a0157a50 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f729800c790 tx=0x7f729800cc50 comp rx=0 tx=0).stop 2024-03-06T07:26:05.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.544+0000 7f72a78c9640 1 -- 172.21.15.1:0/2407513074 shutdown_connections 2024-03-06T07:26:05.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.544+0000 7f72a78c9640 1 --2- 172.21.15.1:0/2407513074 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f728406b500 0x7f728406d9c0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.544+0000 7f72a78c9640 1 --2- 172.21.15.1:0/2407513074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72a00c3eb0 0x7f72a01eba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.544+0000 7f72a78c9640 1 --2- 172.21.15.1:0/2407513074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72a00af640 0x7f72a00aa230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.544+0000 7f72a78c9640 1 --2- 172.21.15.1:0/2407513074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72a006d060 0x7f72a0157a50 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.544+0000 7f72a78c9640 1 -- 172.21.15.1:0/2407513074 >> 172.21.15.1:0/2407513074 conn(0x7f72a0066d00 msgr2=0x7f72a00c7390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.544+0000 7f72a78c9640 1 -- 172.21.15.1:0/2407513074 shutdown_connections 2024-03-06T07:26:05.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.544+0000 7f72a78c9640 1 -- 172.21.15.1:0/2407513074 wait complete. 2024-03-06T07:26:05.559 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:26:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.669+0000 7fd0e330f640 1 Processor -- start 2024-03-06T07:26:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.669+0000 7fd0e330f640 1 -- start start 2024-03-06T07:26:05.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.669+0000 7fd0e330f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0dc0c5dd0 0x7fd0dc153b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.669+0000 7fd0e330f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd0dc0c1e90 0x7fd0dc0c1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.669+0000 7fd0e330f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd0dc0c1880 0x7fd0dc0c4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.669+0000 7fd0e330f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0dc1540c0 con 0x7fd0dc0c1e90 2024-03-06T07:26:05.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.669+0000 7fd0e330f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0dc06dab0 con 0x7fd0dc0c1880 2024-03-06T07:26:05.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.670+0000 7fd0e330f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc0c2db0 con 0x7fd0dc0c5dd0 2024-03-06T07:26:05.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.670+0000 7fd0e08aa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd0dc0c1e90 0x7fd0dc0c1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.670+0000 7fd0e08aa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd0dc0c1e90 0x7fd0dc0c1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33232/0 (socket says 172.21.15.1:33232) 2024-03-06T07:26:05.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.670+0000 7fd0e08aa640 1 -- 172.21.15.1:0/407114272 learned_addr learned my addr 172.21.15.1:0/407114272 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:05.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.670+0000 7fd0e10ab640 1 --2- 172.21.15.1:0/407114272 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0dc0c5dd0 0x7fd0dc153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.670+0000 7fd0e08aa640 1 -- 172.21.15.1:0/407114272 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0dc0c5dd0 msgr2=0x7fd0dc153b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.670+0000 7fd0e08aa640 1 --2- 172.21.15.1:0/407114272 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0dc0c5dd0 0x7fd0dc153b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.670+0000 7fd0e08aa640 1 -- 172.21.15.1:0/407114272 >> v2:172.21.15.98:3300/0 conn(0x7fd0dc0c1880 msgr2=0x7fd0dc0c4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.670+0000 7fd0e18ac640 1 --2- 172.21.15.1:0/407114272 >> v2:172.21.15.98:3300/0 conn(0x7fd0dc0c1880 0x7fd0dc0c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.670+0000 7fd0e08aa640 1 --2- 172.21.15.1:0/407114272 >> v2:172.21.15.98:3300/0 conn(0x7fd0dc0c1880 0x7fd0dc0c4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.670+0000 7fd0e08aa640 1 -- 172.21.15.1:0/407114272 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc0c3090 con 0x7fd0dc0c1e90 2024-03-06T07:26:05.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.670+0000 7fd0e18ac640 1 --2- 172.21.15.1:0/407114272 >> v2:172.21.15.98:3300/0 conn(0x7fd0dc0c1880 0x7fd0dc0c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:05.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.671+0000 7fd0e08aa640 1 --2- 172.21.15.1:0/407114272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0dc0c1e90 0x7fd0dc0c1340 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc0c27e0 tx=0x7fd0d002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7a7e1e63f23add4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.671+0000 7fd0dbfff640 1 -- 172.21.15.1:0/407114272 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d002e070 con 0x7fd0dc0c1e90 2024-03-06T07:26:05.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.671+0000 7fd0dbfff640 1 -- 172.21.15.1:0/407114272 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d0037070 con 0x7fd0dc0c1e90 2024-03-06T07:26:05.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.671+0000 7fd0dbfff640 1 -- 172.21.15.1:0/407114272 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0031a00 con 0x7fd0dc0c1e90 2024-03-06T07:26:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.671+0000 7fd0e330f640 1 -- 172.21.15.1:0/407114272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0dc0c1e90 msgr2=0x7fd0dc0c1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.671+0000 7fd0e330f640 1 --2- 172.21.15.1:0/407114272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0dc0c1e90 0x7fd0dc0c1340 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc0c27e0 tx=0x7fd0d002b880 comp rx=0 tx=0).stop 2024-03-06T07:26:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.671+0000 7fd0e330f640 1 -- 172.21.15.1:0/407114272 shutdown_connections 2024-03-06T07:26:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.671+0000 7fd0e330f640 1 --2- 172.21.15.1:0/407114272 >> v2:172.21.15.98:3300/0 conn(0x7fd0dc0c1880 0x7fd0dc0c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.671+0000 7fd0e330f640 1 --2- 172.21.15.1:0/407114272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0dc0c1e90 0x7fd0dc0c1340 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.671+0000 7fd0e330f640 1 --2- 172.21.15.1:0/407114272 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0dc0c5dd0 0x7fd0dc153b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.671+0000 7fd0e330f640 1 -- 172.21.15.1:0/407114272 >> 172.21.15.1:0/407114272 conn(0x7fd0dc0a8e10 msgr2=0x7fd0dc0c3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.672+0000 7fd0e330f640 1 -- 172.21.15.1:0/407114272 shutdown_connections 2024-03-06T07:26:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.672+0000 7fd0e330f640 1 -- 172.21.15.1:0/407114272 wait complete. 2024-03-06T07:26:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.673+0000 7fd0e330f640 1 Processor -- start 2024-03-06T07:26:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.673+0000 7fd0e330f640 1 -- start start 2024-03-06T07:26:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.673+0000 7fd0e330f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0dc0c1880 0x7fd0dc0bffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.673+0000 7fd0e330f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0dc0c1e90 0x7fd0dc14cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.673+0000 7fd0e330f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0dc0c5dd0 0x7fd0dc14d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.673+0000 7fd0e330f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc0c61b0 con 0x7fd0dc0c1e90 2024-03-06T07:26:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.673+0000 7fd0e330f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc06e050 con 0x7fd0dc0c1880 2024-03-06T07:26:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.673+0000 7fd0e330f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc0c4e80 con 0x7fd0dc0c5dd0 2024-03-06T07:26:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.674+0000 7fd0e08aa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0dc0c1e90 0x7fd0dc14cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.674+0000 7fd0e08aa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0dc0c1e90 0x7fd0dc14cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33244/0 (socket says 172.21.15.1:33244) 2024-03-06T07:26:05.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.674+0000 7fd0e08aa640 1 -- 172.21.15.1:0/329736989 learned_addr learned my addr 172.21.15.1:0/329736989 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:05.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.674+0000 7fd0e18ac640 1 --2- 172.21.15.1:0/329736989 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0dc0c5dd0 0x7fd0dc14d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.674+0000 7fd0e10ab640 1 --2- 172.21.15.1:0/329736989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0dc0c1880 0x7fd0dc0bffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.674+0000 7fd0e08aa640 1 -- 172.21.15.1:0/329736989 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0dc0c5dd0 msgr2=0x7fd0dc14d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.674+0000 7fd0e08aa640 1 --2- 172.21.15.1:0/329736989 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0dc0c5dd0 0x7fd0dc14d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.674+0000 7fd0e08aa640 1 -- 172.21.15.1:0/329736989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0dc0c1880 msgr2=0x7fd0dc0bffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.674+0000 7fd0e08aa640 1 --2- 172.21.15.1:0/329736989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0dc0c1880 0x7fd0dc0bffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.674+0000 7fd0e08aa640 1 -- 172.21.15.1:0/329736989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc0ac900 con 0x7fd0dc0c1e90 2024-03-06T07:26:05.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.674+0000 7fd0e08aa640 1 --2- 172.21.15.1:0/329736989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0dc0c1e90 0x7fd0dc14cae0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fd0d002b8c0 tx=0x7fd0d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.674+0000 7fd0d9ffb640 1 -- 172.21.15.1:0/329736989 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0032880 con 0x7fd0dc0c1e90 2024-03-06T07:26:05.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.674+0000 7fd0d9ffb640 1 -- 172.21.15.1:0/329736989 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d0032a20 con 0x7fd0dc0c1e90 2024-03-06T07:26:05.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.674+0000 7fd0e330f640 1 -- 172.21.15.1:0/329736989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0dc0a5820 con 0x7fd0dc0c1e90 2024-03-06T07:26:05.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.674+0000 7fd0d9ffb640 1 -- 172.21.15.1:0/329736989 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0032e10 con 0x7fd0dc0c1e90 2024-03-06T07:26:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.674+0000 7fd0e330f640 1 -- 172.21.15.1:0/329736989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0dc1ee8e0 con 0x7fd0dc0c1e90 2024-03-06T07:26:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.675+0000 7fd0e330f640 1 -- 172.21.15.1:0/329736989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0dc0680b0 con 0x7fd0dc0c1e90 2024-03-06T07:26:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.676+0000 7fd0d9ffb640 1 -- 172.21.15.1:0/329736989 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0d00453f0 con 0x7fd0dc0c1e90 2024-03-06T07:26:05.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.676+0000 7fd0d9ffb640 1 --2- 172.21.15.1:0/329736989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd0c806b500 0x7fd0c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.677+0000 7fd0d9ffb640 1 -- 172.21.15.1:0/329736989 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6359+0+0 (secure 0 0 0) 0x7fd0d00ac310 con 0x7fd0dc0c1e90 2024-03-06T07:26:05.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.677+0000 7fd0e10ab640 1 --2- 172.21.15.1:0/329736989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd0c806b500 0x7fd0c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.677+0000 7fd0e10ab640 1 --2- 172.21.15.1:0/329736989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd0c806b500 0x7fd0c806d9c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc009a60 tx=0x7fd0cc003990 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.681+0000 7fd0d9ffb640 1 -- 172.21.15.1:0/329736989 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0dc0ac900 con 0x7fd0dc0c1e90 2024-03-06T07:26:05.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:05.862+0000 7fd0e330f640 1 -- 172.21.15.1:0/329736989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd0dc153ea0 con 0x7fd0dc0c1e90 2024-03-06T07:26:06.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.542+0000 7fd0d9ffb640 1 -- 172.21.15.1:0/329736989 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v139) v1 ==== 181+0+0 (secure 0 0 0) 0x7fd0d0040070 con 0x7fd0dc0c1e90 2024-03-06T07:26:06.545 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:26:06.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.547+0000 7fd0e330f640 1 -- 172.21.15.1:0/329736989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd0c806b500 msgr2=0x7fd0c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.547+0000 7fd0e330f640 1 --2- 172.21.15.1:0/329736989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd0c806b500 0x7fd0c806d9c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc009a60 tx=0x7fd0cc003990 comp rx=0 tx=0).stop 2024-03-06T07:26:06.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.547+0000 7fd0e330f640 1 -- 172.21.15.1:0/329736989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0dc0c1e90 msgr2=0x7fd0dc14cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.547+0000 7fd0e330f640 1 --2- 172.21.15.1:0/329736989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0dc0c1e90 0x7fd0dc14cae0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fd0d002b8c0 tx=0x7fd0d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.547+0000 7fd0e330f640 1 -- 172.21.15.1:0/329736989 shutdown_connections 2024-03-06T07:26:06.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.547+0000 7fd0e330f640 1 --2- 172.21.15.1:0/329736989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd0c806b500 0x7fd0c806d9c0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.547+0000 7fd0e330f640 1 --2- 172.21.15.1:0/329736989 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0dc0c5dd0 0x7fd0dc14d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.547+0000 7fd0e330f640 1 --2- 172.21.15.1:0/329736989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0dc0c1e90 0x7fd0dc14cae0 secure :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fd0d002b8c0 tx=0x7fd0d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.547+0000 7fd0e330f640 1 --2- 172.21.15.1:0/329736989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0dc0c1880 0x7fd0dc0bffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.547+0000 7fd0e330f640 1 -- 172.21.15.1:0/329736989 >> 172.21.15.1:0/329736989 conn(0x7fd0dc0a8e10 msgr2=0x7fd0dc150f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.548+0000 7fd0e330f640 1 -- 172.21.15.1:0/329736989 shutdown_connections 2024-03-06T07:26:06.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.548+0000 7fd0e330f640 1 -- 172.21.15.1:0/329736989 wait complete. 2024-03-06T07:26:06.567 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:26:06.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.676+0000 7fb740237640 1 Processor -- start 2024-03-06T07:26:06.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.676+0000 7fb740237640 1 -- start start 2024-03-06T07:26:06.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb740237640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb73806f100 0x7fb73806f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb740237640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7380a9790 con 0x7fb73806c3f0 2024-03-06T07:26:06.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb740237640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7380a9a70 con 0x7fb738152c00 2024-03-06T07:26:06.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb740237640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380c3990 con 0x7fb73806f100 2024-03-06T07:26:06.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73d7d2640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb73806c3f0 0x7fb738158800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36802/0 (socket says 172.21.15.1:36802) 2024-03-06T07:26:06.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73d7d2640 1 -- 172.21.15.1:0/667833674 learned_addr learned my addr 172.21.15.1:0/667833674 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:06.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73e7d4640 1 --2- 172.21.15.1:0/667833674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb73806f100 0x7fb73806f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73cfd1640 1 -- 172.21.15.1:0/667833674 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047954563 0 0) 0x7fb7380a9790 con 0x7fb73806c3f0 2024-03-06T07:26:06.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73cfd1640 1 -- 172.21.15.1:0/667833674 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb724003520 con 0x7fb73806c3f0 2024-03-06T07:26:06.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73cfd1640 1 -- 172.21.15.1:0/667833674 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3718699880 0 0) 0x7fb724003520 con 0x7fb73806c3f0 2024-03-06T07:26:06.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73cfd1640 1 -- 172.21.15.1:0/667833674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb73806f100 msgr2=0x7fb73806f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73cfd1640 1 --2- 172.21.15.1:0/667833674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb73806f100 0x7fb73806f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73cfd1640 1 -- 172.21.15.1:0/667833674 >> v1:172.21.15.98:6789/0 conn(0x7fb738152c00 legacy=0x7fb738072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73cfd1640 1 -- 172.21.15.1:0/667833674 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7380a9d50 con 0x7fb73806c3f0 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73cfd1640 1 -- 172.21.15.1:0/667833674 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fb728001670 con 0x7fb73806c3f0 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73cfd1640 1 -- 172.21.15.1:0/667833674 >> v1:172.21.15.1:6789/0 conn(0x7fb73806c3f0 legacy=0x7fb738158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73cfd1640 1 --2- 172.21.15.1:0/667833674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb73806f100 0x7fb7240039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73cfd1640 1 --2- 172.21.15.1:0/667833674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb724003f20 0x7fb724004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73cfd1640 1 --2- 172.21.15.1:0/667833674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb724004d40 0x7fb7240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73cfd1640 1 -- 172.21.15.1:0/667833674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380c3990 con 0x7fb724003f20 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73cfd1640 1 -- 172.21.15.1:0/667833674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72400ba20 con 0x7fb724004d40 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.677+0000 7fb73cfd1640 1 -- 172.21.15.1:0/667833674 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72400bc20 con 0x7fb73806f100 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.678+0000 7fb73d7d2640 1 --2- 172.21.15.1:0/667833674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb724003f20 0x7fb724004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.678+0000 7fb73dfd3640 1 --2- 172.21.15.1:0/667833674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb73806f100 0x7fb7240039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.678+0000 7fb73e7d4640 1 --2- 172.21.15.1:0/667833674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb724004d40 0x7fb7240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.678+0000 7fb73d7d2640 1 -- 172.21.15.1:0/667833674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb73806f100 msgr2=0x7fb7240039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.678+0000 7fb73d7d2640 1 --2- 172.21.15.1:0/667833674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb73806f100 0x7fb7240039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.678+0000 7fb73d7d2640 1 -- 172.21.15.1:0/667833674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb724004d40 msgr2=0x7fb7240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.678+0000 7fb73d7d2640 1 --2- 172.21.15.1:0/667833674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb724004d40 0x7fb7240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.678+0000 7fb73d7d2640 1 -- 172.21.15.1:0/667833674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb724003520 con 0x7fb724003f20 2024-03-06T07:26:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.679+0000 7fb73d7d2640 1 --2- 172.21.15.1:0/667833674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb724003f20 0x7fb724004800 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fb72800c210 tx=0x7fb724005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5ee3f330138258c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.679+0000 7fb73cfd1640 1 -- 172.21.15.1:0/667833674 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb728007ce0 con 0x7fb724003f20 2024-03-06T07:26:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.679+0000 7fb73cfd1640 1 -- 172.21.15.1:0/667833674 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb728030070 con 0x7fb724003f20 2024-03-06T07:26:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.679+0000 7fb73cfd1640 1 -- 172.21.15.1:0/667833674 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb728005d70 con 0x7fb724003f20 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.679+0000 7fb740237640 1 -- 172.21.15.1:0/667833674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb724003f20 msgr2=0x7fb724004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.679+0000 7fb740237640 1 --2- 172.21.15.1:0/667833674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb724003f20 0x7fb724004800 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fb72800c210 tx=0x7fb724005830 comp rx=0 tx=0).stop 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.679+0000 7fb73e7d4640 1 -- 172.21.15.1:0/667833674 reap_dead start 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.679+0000 7fb740237640 1 -- 172.21.15.1:0/667833674 shutdown_connections 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.679+0000 7fb740237640 1 -- 172.21.15.1:0/667833674 >> 172.21.15.1:0/667833674 conn(0x7fb7380c76b0 msgr2=0x7fb7380c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.679+0000 7fb740237640 1 -- 172.21.15.1:0/667833674 shutdown_connections 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.680+0000 7fb740237640 1 -- 172.21.15.1:0/667833674 wait complete. 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.681+0000 7fb740237640 1 Processor -- start 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.681+0000 7fb740237640 1 -- start start 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.681+0000 7fb740237640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb724003f20 0x7fb73814ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.681+0000 7fb740237640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb738156600 0x7fb7380c0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.681+0000 7fb740237640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7380c0c60 0x7fb7381eae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.681+0000 7fb740237640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380aa030 con 0x7fb7380c0c60 2024-03-06T07:26:06.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.681+0000 7fb740237640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73806bf60 con 0x7fb724003f20 2024-03-06T07:26:06.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.681+0000 7fb740237640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380724a0 con 0x7fb738156600 2024-03-06T07:26:06.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.681+0000 7fb73e7d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7380c0c60 0x7fb7381eae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.681+0000 7fb73e7d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7380c0c60 0x7fb7381eae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33260/0 (socket says 172.21.15.1:33260) 2024-03-06T07:26:06.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.681+0000 7fb73e7d4640 1 -- 172.21.15.1:0/3993051632 learned_addr learned my addr 172.21.15.1:0/3993051632 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:06.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.681+0000 7fb73dfd3640 1 --2- 172.21.15.1:0/3993051632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb724003f20 0x7fb73814ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.681+0000 7fb73d7d2640 1 --2- 172.21.15.1:0/3993051632 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb738156600 0x7fb7380c0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.682+0000 7fb73e7d4640 1 -- 172.21.15.1:0/3993051632 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb738156600 msgr2=0x7fb7380c0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.682+0000 7fb73e7d4640 1 --2- 172.21.15.1:0/3993051632 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb738156600 0x7fb7380c0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.682+0000 7fb73e7d4640 1 -- 172.21.15.1:0/3993051632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb724003f20 msgr2=0x7fb73814ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.682+0000 7fb73e7d4640 1 --2- 172.21.15.1:0/3993051632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb724003f20 0x7fb73814ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.682+0000 7fb73e7d4640 1 -- 172.21.15.1:0/3993051632 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7380ac4b0 con 0x7fb7380c0c60 2024-03-06T07:26:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.682+0000 7fb73e7d4640 1 --2- 172.21.15.1:0/3993051632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7380c0c60 0x7fb7381eae60 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fb730011bf0 tx=0x7fb73000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.682+0000 7fb72e7fc640 1 -- 172.21.15.1:0/3993051632 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73000fdb0 con 0x7fb7380c0c60 2024-03-06T07:26:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.682+0000 7fb72e7fc640 1 -- 172.21.15.1:0/3993051632 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb730015430 con 0x7fb7380c0c60 2024-03-06T07:26:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.682+0000 7fb740237640 1 -- 172.21.15.1:0/3993051632 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7381df4f0 con 0x7fb7380c0c60 2024-03-06T07:26:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.682+0000 7fb72e7fc640 1 -- 172.21.15.1:0/3993051632 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb730015820 con 0x7fb7380c0c60 2024-03-06T07:26:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.682+0000 7fb740237640 1 -- 172.21.15.1:0/3993051632 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7381ee9b0 con 0x7fb7380c0c60 2024-03-06T07:26:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.683+0000 7fb740237640 1 -- 172.21.15.1:0/3993051632 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7380680b0 con 0x7fb7380c0c60 2024-03-06T07:26:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.684+0000 7fb72e7fc640 1 -- 172.21.15.1:0/3993051632 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7300159c0 con 0x7fb7380c0c60 2024-03-06T07:26:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.684+0000 7fb72e7fc640 1 --2- 172.21.15.1:0/3993051632 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb71c06b500 0x7fb71c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.684+0000 7fb72e7fc640 1 -- 172.21.15.1:0/3993051632 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5921+0+0 (secure 0 0 0) 0x7fb73008af60 con 0x7fb7380c0c60 2024-03-06T07:26:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.684+0000 7fb73dfd3640 1 --2- 172.21.15.1:0/3993051632 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb71c06b500 0x7fb71c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.685+0000 7fb73dfd3640 1 --2- 172.21.15.1:0/3993051632 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb71c06b500 0x7fb71c06d9c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb734001910 tx=0x7fb734009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.688+0000 7fb72e7fc640 1 -- 172.21.15.1:0/3993051632 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb730019030 con 0x7fb7380c0c60 2024-03-06T07:26:06.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:06.873+0000 7fb740237640 1 -- 172.21.15.1:0/3993051632 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb7380c3b30 con 0x7fb7380c0c60 2024-03-06T07:26:07.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.557+0000 7fb72e7fc640 1 -- 172.21.15.1:0/3993051632 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v140) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb730010040 con 0x7fb7380c0c60 2024-03-06T07:26:07.560 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-06T07:26:07.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.561+0000 7fb740237640 1 -- 172.21.15.1:0/3993051632 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb71c06b500 msgr2=0x7fb71c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.561+0000 7fb740237640 1 --2- 172.21.15.1:0/3993051632 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb71c06b500 0x7fb71c06d9c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb734001910 tx=0x7fb734009040 comp rx=0 tx=0).stop 2024-03-06T07:26:07.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.561+0000 7fb740237640 1 -- 172.21.15.1:0/3993051632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7380c0c60 msgr2=0x7fb7381eae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.561+0000 7fb740237640 1 --2- 172.21.15.1:0/3993051632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7380c0c60 0x7fb7381eae60 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fb730011bf0 tx=0x7fb73000f580 comp rx=0 tx=0).stop 2024-03-06T07:26:07.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.561+0000 7fb740237640 1 -- 172.21.15.1:0/3993051632 shutdown_connections 2024-03-06T07:26:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.561+0000 7fb740237640 1 --2- 172.21.15.1:0/3993051632 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb71c06b500 0x7fb71c06d9c0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.561+0000 7fb740237640 1 --2- 172.21.15.1:0/3993051632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7380c0c60 0x7fb7381eae60 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.561+0000 7fb740237640 1 --2- 172.21.15.1:0/3993051632 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb738156600 0x7fb7380c0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.561+0000 7fb740237640 1 --2- 172.21.15.1:0/3993051632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb724003f20 0x7fb73814ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.561+0000 7fb740237640 1 -- 172.21.15.1:0/3993051632 >> 172.21.15.1:0/3993051632 conn(0x7fb7380c76b0 msgr2=0x7fb738153aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.561+0000 7fb740237640 1 -- 172.21.15.1:0/3993051632 shutdown_connections 2024-03-06T07:26:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.561+0000 7fb740237640 1 -- 172.21.15.1:0/3993051632 wait complete. 2024-03-06T07:26:07.579 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:26:07.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.685+0000 7fa6b317a640 1 Processor -- start 2024-03-06T07:26:07.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.686+0000 7fa6b317a640 1 -- start start 2024-03-06T07:26:07.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.686+0000 7fa6b317a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6ac152650 0x7fa6ac155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.686+0000 7fa6b317a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa6ac0bf9d0 0x7fa6ac0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.686+0000 7fa6b317a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ac0c2fc0 con 0x7fa6ac0c6b10 2024-03-06T07:26:07.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.686+0000 7fa6b317a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6ac155800 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.686+0000 7fa6b317a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac158da0 con 0x7fa6ac152650 2024-03-06T07:26:07.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.686+0000 7fa6b1717640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa6ac0bf9d0 0x7fa6ac0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.686+0000 7fa6abfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa6ac0c6b10 0x7fa6ac158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36810/0 (socket says 172.21.15.1:36810) 2024-03-06T07:26:07.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.686+0000 7fa6abfff640 1 -- 172.21.15.1:0/2768509482 learned_addr learned my addr 172.21.15.1:0/2768509482 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:07.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.687+0000 7fa6b0f16640 1 --2- 172.21.15.1:0/2768509482 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6ac152650 0x7fa6ac155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.687+0000 7fa6b1717640 1 -- 172.21.15.1:0/2768509482 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6ac152650 msgr2=0x7fa6ac155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.687+0000 7fa6b1717640 1 --2- 172.21.15.1:0/2768509482 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6ac152650 0x7fa6ac155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.687+0000 7fa6b1717640 1 -- 172.21.15.1:0/2768509482 >> v1:172.21.15.1:6789/0 conn(0x7fa6ac0c6b10 legacy=0x7fa6ac158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.687+0000 7fa6b1717640 1 -- 172.21.15.1:0/2768509482 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac0c6690 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.687+0000 7fa6ab7fe640 1 -- 172.21.15.1:0/2768509482 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741470899 0 0) 0x7fa6ac0c2fc0 con 0x7fa6ac0c6b10 2024-03-06T07:26:07.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.688+0000 7fa6b1717640 1 --2- 172.21.15.1:0/2768509482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac0bf9d0 0x7fa6ac0c1460 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4009d90 tx=0x7fa6a402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d1c0757a1f7220 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.688+0000 7fa6ab7fe640 1 -- 172.21.15.1:0/2768509482 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4039070 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.688+0000 7fa6ab7fe640 1 -- 172.21.15.1:0/2768509482 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a402cbc0 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.688+0000 7fa6ab7fe640 1 -- 172.21.15.1:0/2768509482 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4034440 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.688+0000 7fa6b317a640 1 -- 172.21.15.1:0/2768509482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac0bf9d0 msgr2=0x7fa6ac0c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.688+0000 7fa6b317a640 1 --2- 172.21.15.1:0/2768509482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac0bf9d0 0x7fa6ac0c1460 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4009d90 tx=0x7fa6a402be10 comp rx=0 tx=0).stop 2024-03-06T07:26:07.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.688+0000 7fa6b317a640 1 -- 172.21.15.1:0/2768509482 shutdown_connections 2024-03-06T07:26:07.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.688+0000 7fa6b317a640 1 --2- 172.21.15.1:0/2768509482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac0bf9d0 0x7fa6ac0c1460 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.688+0000 7fa6b317a640 1 --2- 172.21.15.1:0/2768509482 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6ac152650 0x7fa6ac155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.688+0000 7fa6b317a640 1 -- 172.21.15.1:0/2768509482 >> 172.21.15.1:0/2768509482 conn(0x7fa6ac066ee0 msgr2=0x7fa6ac0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.688+0000 7fa6b317a640 1 -- 172.21.15.1:0/2768509482 shutdown_connections 2024-03-06T07:26:07.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.688+0000 7fa6b317a640 1 -- 172.21.15.1:0/2768509482 wait complete. 2024-03-06T07:26:07.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.689+0000 7fa6b317a640 1 Processor -- start 2024-03-06T07:26:07.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.689+0000 7fa6b317a640 1 -- start start 2024-03-06T07:26:07.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.690+0000 7fa6b317a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac0bf9d0 0x7fa6ac0c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.690+0000 7fa6b317a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6ac152650 0x7fa6ac1eafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.690+0000 7fa6b317a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6ac1edfb0 0x7fa6ac1eb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.690+0000 7fa6b317a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac1551b0 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.690+0000 7fa6b317a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac0c6fa0 con 0x7fa6ac152650 2024-03-06T07:26:07.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.690+0000 7fa6b317a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac155490 con 0x7fa6ac1edfb0 2024-03-06T07:26:07.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.690+0000 7fa6b0f16640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac0bf9d0 0x7fa6ac0c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.690+0000 7fa6b0f16640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac0bf9d0 0x7fa6ac0c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33270/0 (socket says 172.21.15.1:33270) 2024-03-06T07:26:07.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.690+0000 7fa6b0f16640 1 -- 172.21.15.1:0/3415704437 learned_addr learned my addr 172.21.15.1:0/3415704437 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:07.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.691+0000 7fa6b1717640 1 --2- 172.21.15.1:0/3415704437 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6ac1edfb0 0x7fa6ac1eb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.691+0000 7fa6abfff640 1 --2- 172.21.15.1:0/3415704437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6ac152650 0x7fa6ac1eafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.691+0000 7fa6b0f16640 1 -- 172.21.15.1:0/3415704437 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6ac1edfb0 msgr2=0x7fa6ac1eb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.691+0000 7fa6b0f16640 1 --2- 172.21.15.1:0/3415704437 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6ac1edfb0 0x7fa6ac1eb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.691+0000 7fa6b0f16640 1 -- 172.21.15.1:0/3415704437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6ac152650 msgr2=0x7fa6ac1eafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.691+0000 7fa6b0f16640 1 --2- 172.21.15.1:0/3415704437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6ac152650 0x7fa6ac1eafd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.691+0000 7fa6b0f16640 1 -- 172.21.15.1:0/3415704437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac1ee520 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.691+0000 7fa6b0f16640 1 --2- 172.21.15.1:0/3415704437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac0bf9d0 0x7fa6ac0c9a90 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000c7f0 tx=0x7fa6a000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.691+0000 7fa6a97fa640 1 -- 172.21.15.1:0/3415704437 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0015070 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.691+0000 7fa6a97fa640 1 -- 172.21.15.1:0/3415704437 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a0004240 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.691+0000 7fa6b317a640 1 -- 172.21.15.1:0/3415704437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ac1ee800 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.691+0000 7fa6a97fa640 1 -- 172.21.15.1:0/3415704437 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0007620 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.691+0000 7fa6b317a640 1 -- 172.21.15.1:0/3415704437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ac1eedc0 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.692+0000 7fa6b317a640 1 -- 172.21.15.1:0/3415704437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ac0c6280 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.693+0000 7fa6a97fa640 1 -- 172.21.15.1:0/3415704437 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6a00077c0 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.693+0000 7fa6a97fa640 1 --2- 172.21.15.1:0/3415704437 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa69006b4d0 0x7fa69006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.693+0000 7fa6a97fa640 1 -- 172.21.15.1:0/3415704437 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fa6a008a390 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.693+0000 7fa6abfff640 1 --2- 172.21.15.1:0/3415704437 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa69006b4d0 0x7fa69006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.694+0000 7fa6abfff640 1 --2- 172.21.15.1:0/3415704437 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa69006b4d0 0x7fa69006d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac149420 tx=0x7fa698009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.697+0000 7fa6a97fa640 1 -- 172.21.15.1:0/3415704437 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6ac0c6280 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.879+0000 7fa6b317a640 1 -- 172.21.15.1:0/3415704437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa6ac0b06f0 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.882+0000 7fa6a97fa640 1 -- 172.21.15.1:0/3415704437 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v49) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa6a00592c0 con 0x7fa6ac0bf9d0 2024-03-06T07:26:07.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.884+0000 7fa6b317a640 1 -- 172.21.15.1:0/3415704437 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa69006b4d0 msgr2=0x7fa69006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.884+0000 7fa6b317a640 1 --2- 172.21.15.1:0/3415704437 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa69006b4d0 0x7fa69006d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac149420 tx=0x7fa698009040 comp rx=0 tx=0).stop 2024-03-06T07:26:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.884+0000 7fa6b317a640 1 -- 172.21.15.1:0/3415704437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac0bf9d0 msgr2=0x7fa6ac0c9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.884+0000 7fa6b317a640 1 --2- 172.21.15.1:0/3415704437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac0bf9d0 0x7fa6ac0c9a90 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000c7f0 tx=0x7fa6a000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.884+0000 7fa6b317a640 1 -- 172.21.15.1:0/3415704437 shutdown_connections 2024-03-06T07:26:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.884+0000 7fa6b317a640 1 --2- 172.21.15.1:0/3415704437 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa69006b4d0 0x7fa69006d990 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.884+0000 7fa6b317a640 1 --2- 172.21.15.1:0/3415704437 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6ac1edfb0 0x7fa6ac1eb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.884+0000 7fa6b317a640 1 --2- 172.21.15.1:0/3415704437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6ac152650 0x7fa6ac1eafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.884+0000 7fa6b317a640 1 --2- 172.21.15.1:0/3415704437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac0bf9d0 0x7fa6ac0c9a90 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.884+0000 7fa6b317a640 1 -- 172.21.15.1:0/3415704437 >> 172.21.15.1:0/3415704437 conn(0x7fa6ac066ee0 msgr2=0x7fa6ac0c3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.885+0000 7fa6b317a640 1 -- 172.21.15.1:0/3415704437 shutdown_connections 2024-03-06T07:26:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.885+0000 7fa6b317a640 1 -- 172.21.15.1:0/3415704437 wait complete. 2024-03-06T07:26:07.897 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-03-06T07:26:08.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.998+0000 7f72226fb640 1 Processor -- start 2024-03-06T07:26:08.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.998+0000 7f72226fb640 1 -- start start 2024-03-06T07:26:08.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.999+0000 7f72226fb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f721c0c5cb0 0x7f721c155930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.999+0000 7f72226fb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f721c158910 con 0x7f721c0c5850 2024-03-06T07:26:08.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.999+0000 7f72226fb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f721c158bf0 con 0x7f721c152920 2024-03-06T07:26:08.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.999+0000 7f72226fb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f721c152d00 con 0x7f721c0c5cb0 2024-03-06T07:26:08.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.999+0000 7f7220c98640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f721c0c5cb0 0x7f721c155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.999+0000 7f7220c98640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f721c0c5cb0 0x7f721c155930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33276/0 (socket says 172.21.15.1:33276) 2024-03-06T07:26:08.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.999+0000 7f7220c98640 1 -- 172.21.15.1:0/235773255 learned_addr learned my addr 172.21.15.1:0/235773255 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:08.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.999+0000 7f721b7fe640 1 --1- 172.21.15.1:0/235773255 >> v1:172.21.15.1:6789/0 conn(0x7f721c0c5850 0x7f721c155220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36816/0 (socket says 172.21.15.1:36816) 2024-03-06T07:26:08.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.999+0000 7f7220c98640 1 -- 172.21.15.1:0/235773255 >> v1:172.21.15.98:6789/0 conn(0x7f721c152920 legacy=0x7f721c06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.999+0000 7f7220c98640 1 -- 172.21.15.1:0/235773255 >> v1:172.21.15.1:6789/0 conn(0x7f721c0c5850 legacy=0x7f721c155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:07.999+0000 7f7220c98640 1 -- 172.21.15.1:0/235773255 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c158ed0 con 0x7f721c0c5cb0 2024-03-06T07:26:08.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.000+0000 7f7220c98640 1 --2- 172.21.15.1:0/235773255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c0c5cb0 0x7f721c155930 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f7214009e50 tx=0x7f721402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d85e944cf13db3b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.000+0000 7f721affd640 1 -- 172.21.15.1:0/235773255 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721402cb00 con 0x7f721c0c5cb0 2024-03-06T07:26:08.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.000+0000 7f721affd640 1 -- 172.21.15.1:0/235773255 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f721402cca0 con 0x7f721c0c5cb0 2024-03-06T07:26:08.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.000+0000 7f721affd640 1 -- 172.21.15.1:0/235773255 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7214034610 con 0x7f721c0c5cb0 2024-03-06T07:26:08.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.001+0000 7f72226fb640 1 -- 172.21.15.1:0/235773255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c0c5cb0 msgr2=0x7f721c155930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.001+0000 7f72226fb640 1 --2- 172.21.15.1:0/235773255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c0c5cb0 0x7f721c155930 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f7214009e50 tx=0x7f721402bf60 comp rx=0 tx=0).stop 2024-03-06T07:26:08.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.001+0000 7f72226fb640 1 -- 172.21.15.1:0/235773255 shutdown_connections 2024-03-06T07:26:08.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.001+0000 7f72226fb640 1 --2- 172.21.15.1:0/235773255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c0c5cb0 0x7f721c155930 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.001+0000 7f72226fb640 1 -- 172.21.15.1:0/235773255 >> 172.21.15.1:0/235773255 conn(0x7f721c0a9700 msgr2=0x7f721c0a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.001+0000 7f72226fb640 1 -- 172.21.15.1:0/235773255 shutdown_connections 2024-03-06T07:26:08.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.001+0000 7f72226fb640 1 -- 172.21.15.1:0/235773255 wait complete. 2024-03-06T07:26:08.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.002+0000 7f72226fb640 1 Processor -- start 2024-03-06T07:26:08.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.002+0000 7f72226fb640 1 -- start start 2024-03-06T07:26:08.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f72226fb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f721c0c5850 0x7f721c14ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f72226fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c0c5cb0 0x7f721c1edf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f72226fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f721c152920 0x7f721c0a8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f72226fb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c159240 con 0x7f721c0c5cb0 2024-03-06T07:26:08.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f72226fb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c1596d0 con 0x7f721c0c5850 2024-03-06T07:26:08.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f72226fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721c06b700 con 0x7f721c152920 2024-03-06T07:26:08.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f721b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c0c5cb0 0x7f721c1edf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f721b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c0c5cb0 0x7f721c1edf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33286/0 (socket says 172.21.15.1:33286) 2024-03-06T07:26:08.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f721b7fe640 1 -- 172.21.15.1:0/124959341 learned_addr learned my addr 172.21.15.1:0/124959341 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:08.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f7220c98640 1 --2- 172.21.15.1:0/124959341 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f721c152920 0x7f721c0a8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f721bfff640 1 --2- 172.21.15.1:0/124959341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f721c0c5850 0x7f721c14ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f721b7fe640 1 -- 172.21.15.1:0/124959341 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f721c152920 msgr2=0x7f721c0a8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f721b7fe640 1 --2- 172.21.15.1:0/124959341 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f721c152920 0x7f721c0a8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f721b7fe640 1 -- 172.21.15.1:0/124959341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f721c0c5850 msgr2=0x7f721c14ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f721b7fe640 1 --2- 172.21.15.1:0/124959341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f721c0c5850 0x7f721c14ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f721b7fe640 1 -- 172.21.15.1:0/124959341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c14ef70 con 0x7f721c0c5cb0 2024-03-06T07:26:08.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f721b7fe640 1 --2- 172.21.15.1:0/124959341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c0c5cb0 0x7f721c1edf00 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f721000c500 tx=0x7f721000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f7218ff9640 1 -- 172.21.15.1:0/124959341 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7210018070 con 0x7f721c0c5cb0 2024-03-06T07:26:08.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f7218ff9640 1 -- 172.21.15.1:0/124959341 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7210007500 con 0x7f721c0c5cb0 2024-03-06T07:26:08.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f72226fb640 1 -- 172.21.15.1:0/124959341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721c0a9320 con 0x7f721c0c5cb0 2024-03-06T07:26:08.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f7218ff9640 1 -- 172.21.15.1:0/124959341 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72100049e0 con 0x7f721c0c5cb0 2024-03-06T07:26:08.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.003+0000 7f72226fb640 1 -- 172.21.15.1:0/124959341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721c158130 con 0x7f721c0c5cb0 2024-03-06T07:26:08.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.004+0000 7f72226fb640 1 -- 172.21.15.1:0/124959341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721c0c0df0 con 0x7f721c0c5cb0 2024-03-06T07:26:08.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.005+0000 7f7218ff9640 1 -- 172.21.15.1:0/124959341 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7210004b80 con 0x7f721c0c5cb0 2024-03-06T07:26:08.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.005+0000 7f7218ff9640 1 --2- 172.21.15.1:0/124959341 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f720806b4d0 0x7f720806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.005+0000 7f7218ff9640 1 -- 172.21.15.1:0/124959341 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f72100893a0 con 0x7f721c0c5cb0 2024-03-06T07:26:08.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.005+0000 7f721bfff640 1 --2- 172.21.15.1:0/124959341 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f720806b4d0 0x7f720806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.006+0000 7f721bfff640 1 --2- 172.21.15.1:0/124959341 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f720806b4d0 0x7f720806d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f720c007fd0 tx=0x7f720c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.009+0000 7f7218ff9640 1 -- 172.21.15.1:0/124959341 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f721c0c0df0 con 0x7f721c0c5cb0 2024-03-06T07:26:08.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.189+0000 7f72226fb640 1 -- 172.21.15.1:0/124959341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7f721c06bdf0 con 0x7f721c0c5cb0 2024-03-06T07:26:08.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.190+0000 7f7218ff9640 1 -- 172.21.15.1:0/124959341 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v49) v1 ==== 96+0+0 (secure 0 0 0) 0x7f7210057c10 con 0x7f721c0c5cb0 2024-03-06T07:26:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.191+0000 7f72226fb640 1 -- 172.21.15.1:0/124959341 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f720806b4d0 msgr2=0x7f720806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.191+0000 7f72226fb640 1 --2- 172.21.15.1:0/124959341 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f720806b4d0 0x7f720806d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f720c007fd0 tx=0x7f720c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.191+0000 7f72226fb640 1 -- 172.21.15.1:0/124959341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c0c5cb0 msgr2=0x7f721c1edf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.191+0000 7f72226fb640 1 --2- 172.21.15.1:0/124959341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c0c5cb0 0x7f721c1edf00 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f721000c500 tx=0x7f721000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.191+0000 7f72226fb640 1 -- 172.21.15.1:0/124959341 shutdown_connections 2024-03-06T07:26:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.191+0000 7f72226fb640 1 --2- 172.21.15.1:0/124959341 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f720806b4d0 0x7f720806d990 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.191+0000 7f72226fb640 1 --2- 172.21.15.1:0/124959341 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f721c152920 0x7f721c0a8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.191+0000 7f72226fb640 1 --2- 172.21.15.1:0/124959341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c0c5cb0 0x7f721c1edf00 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.191+0000 7f72226fb640 1 --2- 172.21.15.1:0/124959341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f721c0c5850 0x7f721c14ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.191+0000 7f72226fb640 1 -- 172.21.15.1:0/124959341 >> 172.21.15.1:0/124959341 conn(0x7f721c0a9700 msgr2=0x7f721c0c3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.192+0000 7f72226fb640 1 -- 172.21.15.1:0/124959341 shutdown_connections 2024-03-06T07:26:08.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.192+0000 7f72226fb640 1 -- 172.21.15.1:0/124959341 wait complete. 2024-03-06T07:26:08.204 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-03-06T07:26:08.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.305+0000 7fe9aab0c640 1 Processor -- start 2024-03-06T07:26:08.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.305+0000 7fe9aab0c640 1 -- start start 2024-03-06T07:26:08.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.305+0000 7fe9aab0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9a4072f20 0x7fe9a41513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.305+0000 7fe9aab0c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a40a6140 con 0x7fe9a4071f90 2024-03-06T07:26:08.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.305+0000 7fe9aab0c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a4151aa0 con 0x7fe9a4150fe0 2024-03-06T07:26:08.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9aab0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a41526b0 con 0x7fe9a4072f20 2024-03-06T07:26:08.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a90a9640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe9a4071f90 0x7fe9a4072420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36832/0 (socket says 172.21.15.1:36832) 2024-03-06T07:26:08.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a90a9640 1 -- 172.21.15.1:0/28306406 learned_addr learned my addr 172.21.15.1:0/28306406 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:08.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a3fff640 1 --2- 172.21.15.1:0/28306406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9a4072f20 0x7fe9a41513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a37fe640 1 -- 172.21.15.1:0/28306406 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920696803 0 0) 0x7fe9a40a6140 con 0x7fe9a4071f90 2024-03-06T07:26:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a37fe640 1 -- 172.21.15.1:0/28306406 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe98c003520 con 0x7fe9a4071f90 2024-03-06T07:26:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a37fe640 1 -- 172.21.15.1:0/28306406 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3313622451 0 0) 0x7fe98c003520 con 0x7fe9a4071f90 2024-03-06T07:26:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a37fe640 1 -- 172.21.15.1:0/28306406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9a4072f20 msgr2=0x7fe9a41513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a37fe640 1 --2- 172.21.15.1:0/28306406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9a4072f20 0x7fe9a41513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a37fe640 1 -- 172.21.15.1:0/28306406 >> v1:172.21.15.98:6789/0 conn(0x7fe9a4150fe0 legacy=0x7fe9a406edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a37fe640 1 -- 172.21.15.1:0/28306406 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a4151d80 con 0x7fe9a4071f90 2024-03-06T07:26:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a37fe640 1 -- 172.21.15.1:0/28306406 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fe994001670 con 0x7fe9a4071f90 2024-03-06T07:26:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a37fe640 1 -- 172.21.15.1:0/28306406 >> v1:172.21.15.1:6789/0 conn(0x7fe9a4071f90 legacy=0x7fe9a4072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a37fe640 1 --2- 172.21.15.1:0/28306406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c0039e0 0x7fe98c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a37fe640 1 --2- 172.21.15.1:0/28306406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9a4072f20 0x7fe98c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a37fe640 1 --2- 172.21.15.1:0/28306406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe98c004d40 0x7fe98c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a37fe640 1 -- 172.21.15.1:0/28306406 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe98c00ba70 con 0x7fe98c0039e0 2024-03-06T07:26:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a37fe640 1 -- 172.21.15.1:0/28306406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe98c00bbf0 con 0x7fe98c004d40 2024-03-06T07:26:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a37fe640 1 -- 172.21.15.1:0/28306406 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe98c00bdf0 con 0x7fe9a4072f20 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a88a8640 1 --2- 172.21.15.1:0/28306406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c0039e0 0x7fe98c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a3fff640 1 --2- 172.21.15.1:0/28306406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9a4072f20 0x7fe98c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.306+0000 7fe9a90a9640 1 --2- 172.21.15.1:0/28306406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe98c004d40 0x7fe98c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.307+0000 7fe9a88a8640 1 -- 172.21.15.1:0/28306406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9a4072f20 msgr2=0x7fe98c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.307+0000 7fe9a88a8640 1 --2- 172.21.15.1:0/28306406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9a4072f20 0x7fe98c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.307+0000 7fe9a88a8640 1 -- 172.21.15.1:0/28306406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe98c004d40 msgr2=0x7fe98c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.307+0000 7fe9a88a8640 1 --2- 172.21.15.1:0/28306406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe98c004d40 0x7fe98c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.307+0000 7fe9a88a8640 1 -- 172.21.15.1:0/28306406 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe98c003520 con 0x7fe98c0039e0 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.307+0000 7fe9a3fff640 1 --2- 172.21.15.1:0/28306406 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9a4072f20 0x7fe98c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.307+0000 7fe9a88a8640 1 --2- 172.21.15.1:0/28306406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c0039e0 0x7fe98c003dc0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fe99c001f60 tx=0x7fe98c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=541f7076e7a4c449 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.308+0000 7fe9a37fe640 1 -- 172.21.15.1:0/28306406 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c039070 con 0x7fe98c0039e0 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.308+0000 7fe9a37fe640 1 -- 172.21.15.1:0/28306406 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99c02c730 con 0x7fe98c0039e0 2024-03-06T07:26:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.308+0000 7fe9a37fe640 1 -- 172.21.15.1:0/28306406 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c001b30 con 0x7fe98c0039e0 2024-03-06T07:26:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.308+0000 7fe9aab0c640 1 -- 172.21.15.1:0/28306406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c0039e0 msgr2=0x7fe98c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.308+0000 7fe9aab0c640 1 --2- 172.21.15.1:0/28306406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c0039e0 0x7fe98c003dc0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fe99c001f60 tx=0x7fe98c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.308+0000 7fe9a90a9640 1 -- 172.21.15.1:0/28306406 reap_dead start 2024-03-06T07:26:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.308+0000 7fe9aab0c640 1 -- 172.21.15.1:0/28306406 shutdown_connections 2024-03-06T07:26:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.308+0000 7fe9aab0c640 1 -- 172.21.15.1:0/28306406 >> 172.21.15.1:0/28306406 conn(0x7fe9a40c6860 msgr2=0x7fe9a40c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.308+0000 7fe9aab0c640 1 -- 172.21.15.1:0/28306406 shutdown_connections 2024-03-06T07:26:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.308+0000 7fe9aab0c640 1 -- 172.21.15.1:0/28306406 wait complete. 2024-03-06T07:26:08.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.309+0000 7fe9aab0c640 1 Processor -- start 2024-03-06T07:26:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.309+0000 7fe9aab0c640 1 -- start start 2024-03-06T07:26:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.310+0000 7fe9aab0c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c0039e0 0x7fe9a4146e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.310+0000 7fe9aab0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9a4148020 0x7fe9a4145dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.310+0000 7fe9aab0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9a41e90a0 0x7fe9a4146300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.310+0000 7fe9aab0c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a406f4c0 con 0x7fe98c0039e0 2024-03-06T07:26:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.310+0000 7fe9aab0c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a4152340 con 0x7fe9a41e90a0 2024-03-06T07:26:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.310+0000 7fe9aab0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a406f640 con 0x7fe9a4148020 2024-03-06T07:26:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.310+0000 7fe9a88a8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c0039e0 0x7fe9a4146e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.310+0000 7fe9a88a8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c0039e0 0x7fe9a4146e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33292/0 (socket says 172.21.15.1:33292) 2024-03-06T07:26:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.310+0000 7fe9a88a8640 1 -- 172.21.15.1:0/2583303072 learned_addr learned my addr 172.21.15.1:0/2583303072 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.310+0000 7fe9a90a9640 1 --2- 172.21.15.1:0/2583303072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9a41e90a0 0x7fe9a4146300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.310+0000 7fe9a88a8640 1 -- 172.21.15.1:0/2583303072 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9a4148020 msgr2=0x7fe9a4145dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.310+0000 7fe9a3fff640 1 --2- 172.21.15.1:0/2583303072 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9a4148020 0x7fe9a4145dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.310+0000 7fe9a88a8640 1 --2- 172.21.15.1:0/2583303072 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9a4148020 0x7fe9a4145dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.310+0000 7fe9a88a8640 1 -- 172.21.15.1:0/2583303072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9a41e90a0 msgr2=0x7fe9a4146300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.310+0000 7fe9a88a8640 1 --2- 172.21.15.1:0/2583303072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9a41e90a0 0x7fe9a4146300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.310+0000 7fe9a88a8640 1 -- 172.21.15.1:0/2583303072 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a4148da0 con 0x7fe98c0039e0 2024-03-06T07:26:08.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.310+0000 7fe9a3fff640 1 --2- 172.21.15.1:0/2583303072 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9a4148020 0x7fe9a4145dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:08.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.311+0000 7fe9a88a8640 1 --2- 172.21.15.1:0/2583303072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c0039e0 0x7fe9a4146e20 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fe99c02c180 tx=0x7fe99c0015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.311+0000 7fe9a17fa640 1 -- 172.21.15.1:0/2583303072 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c039040 con 0x7fe98c0039e0 2024-03-06T07:26:08.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.311+0000 7fe9a17fa640 1 -- 172.21.15.1:0/2583303072 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99c003c30 con 0x7fe98c0039e0 2024-03-06T07:26:08.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.311+0000 7fe9a17fa640 1 -- 172.21.15.1:0/2583303072 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c005070 con 0x7fe98c0039e0 2024-03-06T07:26:08.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.311+0000 7fe9aab0c640 1 -- 172.21.15.1:0/2583303072 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a41487e0 con 0x7fe98c0039e0 2024-03-06T07:26:08.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.311+0000 7fe9aab0c640 1 -- 172.21.15.1:0/2583303072 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a4147b00 con 0x7fe98c0039e0 2024-03-06T07:26:08.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.312+0000 7fe9aab0c640 1 -- 172.21.15.1:0/2583303072 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a4072bc0 con 0x7fe98c0039e0 2024-03-06T07:26:08.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.312+0000 7fe9a17fa640 1 -- 172.21.15.1:0/2583303072 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe99c005210 con 0x7fe98c0039e0 2024-03-06T07:26:08.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.313+0000 7fe9a17fa640 1 --2- 172.21.15.1:0/2583303072 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe99006b4d0 0x7fe99006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.313+0000 7fe9a17fa640 1 -- 172.21.15.1:0/2583303072 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fe99c0ac030 con 0x7fe98c0039e0 2024-03-06T07:26:08.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.313+0000 7fe9a3fff640 1 --2- 172.21.15.1:0/2583303072 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe99006b4d0 0x7fe99006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.314+0000 7fe9a3fff640 1 --2- 172.21.15.1:0/2583303072 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe99006b4d0 0x7fe99006d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe998006d40 tx=0x7fe998008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.317+0000 7fe9a17fa640 1 -- 172.21.15.1:0/2583303072 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe9a4072bc0 con 0x7fe98c0039e0 2024-03-06T07:26:08.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.498+0000 7fe9aab0c640 1 -- 172.21.15.1:0/2583303072 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7fe9a40c7630 con 0x7fe98c0039e0 2024-03-06T07:26:08.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.501+0000 7fe9a17fa640 1 -- 172.21.15.1:0/2583303072 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v50) v1 ==== 96+0+0 (secure 0 0 0) 0x7fe9a40c7630 con 0x7fe98c0039e0 2024-03-06T07:26:08.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.503+0000 7fe9aab0c640 1 -- 172.21.15.1:0/2583303072 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe99006b4d0 msgr2=0x7fe99006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.503+0000 7fe9aab0c640 1 --2- 172.21.15.1:0/2583303072 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe99006b4d0 0x7fe99006d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe998006d40 tx=0x7fe998008040 comp rx=0 tx=0).stop 2024-03-06T07:26:08.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.503+0000 7fe9aab0c640 1 -- 172.21.15.1:0/2583303072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c0039e0 msgr2=0x7fe9a4146e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.503+0000 7fe9aab0c640 1 --2- 172.21.15.1:0/2583303072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c0039e0 0x7fe9a4146e20 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fe99c02c180 tx=0x7fe99c0015b0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.504+0000 7fe9aab0c640 1 -- 172.21.15.1:0/2583303072 shutdown_connections 2024-03-06T07:26:08.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.504+0000 7fe9aab0c640 1 --2- 172.21.15.1:0/2583303072 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe99006b4d0 0x7fe99006d990 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.504+0000 7fe9aab0c640 1 --2- 172.21.15.1:0/2583303072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9a41e90a0 0x7fe9a4146300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.504+0000 7fe9aab0c640 1 --2- 172.21.15.1:0/2583303072 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9a4148020 0x7fe9a4145dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.504+0000 7fe9aab0c640 1 --2- 172.21.15.1:0/2583303072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c0039e0 0x7fe9a4146e20 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.504+0000 7fe9aab0c640 1 -- 172.21.15.1:0/2583303072 >> 172.21.15.1:0/2583303072 conn(0x7fe9a40c6860 msgr2=0x7fe9a414dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.504+0000 7fe9aab0c640 1 -- 172.21.15.1:0/2583303072 shutdown_connections 2024-03-06T07:26:08.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.504+0000 7fe9aab0c640 1 -- 172.21.15.1:0/2583303072 wait complete. 2024-03-06T07:26:08.516 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset' 2024-03-06T07:26:08.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.617+0000 7f20b1e15640 1 Processor -- start 2024-03-06T07:26:08.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.617+0000 7f20b1e15640 1 -- start start 2024-03-06T07:26:08.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.618+0000 7f20b1e15640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f20ac0c2b30 0x7f20ac0724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.618+0000 7f20b1e15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20ac0a5fa0 0x7f20ac072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.618+0000 7f20b1e15640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ac151dd0 con 0x7f20ac152b70 2024-03-06T07:26:08.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.618+0000 7f20b1e15640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20ac1526e0 con 0x7f20ac0c2b30 2024-03-06T07:26:08.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.618+0000 7f20b1e15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac156190 con 0x7f20ac0a5fa0 2024-03-06T07:26:08.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.618+0000 7f20ab7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f20ac0c2b30 0x7f20ac0724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.618+0000 7f20aaffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f20ac152b70 0x7f20ac0c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36846/0 (socket says 172.21.15.1:36846) 2024-03-06T07:26:08.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.618+0000 7f20aaffd640 1 -- 172.21.15.1:0/33326891 learned_addr learned my addr 172.21.15.1:0/33326891 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:08.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.618+0000 7f20abfff640 1 --2- 172.21.15.1:0/33326891 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20ac0a5fa0 0x7f20ac072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.618+0000 7f20ab7fe640 1 -- 172.21.15.1:0/33326891 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20ac0a5fa0 msgr2=0x7f20ac072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.618+0000 7f20ab7fe640 1 --2- 172.21.15.1:0/33326891 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20ac0a5fa0 0x7f20ac072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.618+0000 7f20ab7fe640 1 -- 172.21.15.1:0/33326891 >> v1:172.21.15.1:6789/0 conn(0x7f20ac152b70 legacy=0x7f20ac0c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.618+0000 7f20ab7fe640 1 -- 172.21.15.1:0/33326891 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ac152090 con 0x7f20ac0c2b30 2024-03-06T07:26:08.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.618+0000 7f20aa7fc640 1 -- 172.21.15.1:0/33326891 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521715163 0 0) 0x7f20ac151dd0 con 0x7f20ac152b70 2024-03-06T07:26:08.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.619+0000 7f20ab7fe640 1 --2- 172.21.15.1:0/33326891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ac0c2b30 0x7f20ac0724c0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f209c0097a0 tx=0x7f209c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3e129d4d7b1a5f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.619+0000 7f20aa7fc640 1 -- 172.21.15.1:0/33326891 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c035020 con 0x7f20ac0c2b30 2024-03-06T07:26:08.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.619+0000 7f20aa7fc640 1 -- 172.21.15.1:0/33326891 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209c031830 con 0x7f20ac0c2b30 2024-03-06T07:26:08.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.619+0000 7f20aa7fc640 1 -- 172.21.15.1:0/33326891 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c031c00 con 0x7f20ac0c2b30 2024-03-06T07:26:08.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.620+0000 7f20b1e15640 1 -- 172.21.15.1:0/33326891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ac0c2b30 msgr2=0x7f20ac0724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.620+0000 7f20b1e15640 1 --2- 172.21.15.1:0/33326891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ac0c2b30 0x7f20ac0724c0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f209c0097a0 tx=0x7f209c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.620+0000 7f20b1e15640 1 -- 172.21.15.1:0/33326891 shutdown_connections 2024-03-06T07:26:08.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.620+0000 7f20b1e15640 1 --2- 172.21.15.1:0/33326891 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20ac0a5fa0 0x7f20ac072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.620+0000 7f20b1e15640 1 --2- 172.21.15.1:0/33326891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ac0c2b30 0x7f20ac0724c0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.620+0000 7f20b1e15640 1 -- 172.21.15.1:0/33326891 >> 172.21.15.1:0/33326891 conn(0x7f20ac0c6860 msgr2=0x7f20ac0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.620+0000 7f20b1e15640 1 -- 172.21.15.1:0/33326891 shutdown_connections 2024-03-06T07:26:08.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.620+0000 7f20b1e15640 1 -- 172.21.15.1:0/33326891 wait complete. 2024-03-06T07:26:08.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.621+0000 7f20b1e15640 1 Processor -- start 2024-03-06T07:26:08.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.621+0000 7f20b1e15640 1 -- start start 2024-03-06T07:26:08.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20b1e15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20ac0a5fa0 0x7f20ac14a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20b1e15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20ac0c2b30 0x7f20ac06eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20b1e15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ac152b70 0x7f20ac1eabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20b1e15640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac155f00 con 0x7f20ac152b70 2024-03-06T07:26:08.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20b1e15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac155d80 con 0x7f20ac0c2b30 2024-03-06T07:26:08.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20b1e15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac0c3a00 con 0x7f20ac0a5fa0 2024-03-06T07:26:08.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20abfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ac152b70 0x7f20ac1eabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20abfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ac152b70 0x7f20ac1eabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33308/0 (socket says 172.21.15.1:33308) 2024-03-06T07:26:08.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20abfff640 1 -- 172.21.15.1:0/3652017377 learned_addr learned my addr 172.21.15.1:0/3652017377 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:08.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20ab7fe640 1 --2- 172.21.15.1:0/3652017377 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20ac0a5fa0 0x7f20ac14a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20aaffd640 1 --2- 172.21.15.1:0/3652017377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20ac0c2b30 0x7f20ac06eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20abfff640 1 -- 172.21.15.1:0/3652017377 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20ac0a5fa0 msgr2=0x7f20ac14a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20abfff640 1 --2- 172.21.15.1:0/3652017377 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20ac0a5fa0 0x7f20ac14a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20abfff640 1 -- 172.21.15.1:0/3652017377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20ac0c2b30 msgr2=0x7f20ac06eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20abfff640 1 --2- 172.21.15.1:0/3652017377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20ac0c2b30 0x7f20ac06eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20abfff640 1 -- 172.21.15.1:0/3652017377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ac1ec980 con 0x7f20ac152b70 2024-03-06T07:26:08.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20abfff640 1 --2- 172.21.15.1:0/3652017377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ac152b70 0x7f20ac1eabd0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f20a4007e40 tx=0x7f20a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f2093fff640 1 -- 172.21.15.1:0/3652017377 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4014620 con 0x7f20ac152b70 2024-03-06T07:26:08.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f2093fff640 1 -- 172.21.15.1:0/3652017377 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a4019070 con 0x7f20ac152b70 2024-03-06T07:26:08.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f20b1e15640 1 -- 172.21.15.1:0/3652017377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ac1ecc60 con 0x7f20ac152b70 2024-03-06T07:26:08.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.622+0000 7f2093fff640 1 -- 172.21.15.1:0/3652017377 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4014a20 con 0x7f20ac152b70 2024-03-06T07:26:08.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.623+0000 7f20b1e15640 1 -- 172.21.15.1:0/3652017377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ac1ed220 con 0x7f20ac152b70 2024-03-06T07:26:08.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.624+0000 7f20b1e15640 1 -- 172.21.15.1:0/3652017377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ac067fd0 con 0x7f20ac152b70 2024-03-06T07:26:08.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.624+0000 7f2093fff640 1 -- 172.21.15.1:0/3652017377 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f20a4014bc0 con 0x7f20ac152b70 2024-03-06T07:26:08.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.624+0000 7f2093fff640 1 --2- 172.21.15.1:0/3652017377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f209806b4d0 0x7f209806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.624+0000 7f2093fff640 1 -- 172.21.15.1:0/3652017377 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f20a408a910 con 0x7f20ac152b70 2024-03-06T07:26:08.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.625+0000 7f20ab7fe640 1 --2- 172.21.15.1:0/3652017377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f209806b4d0 0x7f209806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.625+0000 7f20ab7fe640 1 --2- 172.21.15.1:0/3652017377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f209806b4d0 0x7f209806d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f209c036000 tx=0x7f209c03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.629+0000 7f2093fff640 1 -- 172.21.15.1:0/3652017377 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f20a40591f0 con 0x7f20ac152b70 2024-03-06T07:26:08.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:08.880+0000 7f20b1e15640 1 -- 172.21.15.1:0/3652017377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]} v 0) v1 -- 0x7f20ac071790 con 0x7f20ac152b70 2024-03-06T07:26:09.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.563+0000 7f2093fff640 1 -- 172.21.15.1:0/3652017377 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]}]=0 v339) v1 ==== 144+0+0 (secure 0 0 0) 0x7f20a405f070 con 0x7f20ac152b70 2024-03-06T07:26:09.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.567+0000 7f20b1e15640 1 -- 172.21.15.1:0/3652017377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f209806b4d0 msgr2=0x7f209806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.567+0000 7f20b1e15640 1 --2- 172.21.15.1:0/3652017377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f209806b4d0 0x7f209806d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f209c036000 tx=0x7f209c03d040 comp rx=0 tx=0).stop 2024-03-06T07:26:09.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.567+0000 7f20b1e15640 1 -- 172.21.15.1:0/3652017377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ac152b70 msgr2=0x7f20ac1eabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.567+0000 7f20b1e15640 1 --2- 172.21.15.1:0/3652017377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ac152b70 0x7f20ac1eabd0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f20a4007e40 tx=0x7f20a400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.567+0000 7f20b1e15640 1 -- 172.21.15.1:0/3652017377 shutdown_connections 2024-03-06T07:26:09.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.567+0000 7f20b1e15640 1 --2- 172.21.15.1:0/3652017377 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f209806b4d0 0x7f209806d990 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.567+0000 7f20b1e15640 1 --2- 172.21.15.1:0/3652017377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ac152b70 0x7f20ac1eabd0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.567+0000 7f20b1e15640 1 --2- 172.21.15.1:0/3652017377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20ac0c2b30 0x7f20ac06eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.567+0000 7f20b1e15640 1 --2- 172.21.15.1:0/3652017377 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20ac0a5fa0 0x7f20ac14a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.567+0000 7f20b1e15640 1 -- 172.21.15.1:0/3652017377 >> 172.21.15.1:0/3652017377 conn(0x7f20ac0c6860 msgr2=0x7f20ac072410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.567+0000 7f20b1e15640 1 -- 172.21.15.1:0/3652017377 shutdown_connections 2024-03-06T07:26:09.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.567+0000 7f20b1e15640 1 -- 172.21.15.1:0/3652017377 wait complete. 2024-03-06T07:26:09.583 INFO:tasks.cephfs_test_runner:That unsetting mds_join_fs will cause failover if another high-affinity standby exists. ... ok 2024-03-06T07:26:09.583 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_vanilla (tasks.cephfs.test_failover.TestClusterAffinity) That a vanilla standby is preferred over others with mds_join_fs set to another fs. 2024-03-06T07:26:09.583 INFO:tasks.cephfs_test_runner:test_join_fs_vanilla (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-06T07:26:09.583 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:09.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.690+0000 7f853313e640 1 Processor -- start 2024-03-06T07:26:09.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.690+0000 7f853313e640 1 -- start start 2024-03-06T07:26:09.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f853313e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f852c149a90 0x7f852c0c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f853313e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f852c0c1260 0x7f852c0c16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f853313e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f852c1530a0 con 0x7f852c1565a0 2024-03-06T07:26:09.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f853313e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f852c153380 con 0x7f852c149a90 2024-03-06T07:26:09.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f853313e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852c153660 con 0x7f852c0c1260 2024-03-06T07:26:09.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85316db640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f852c1565a0 0x7f852c156a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36858/0 (socket says 172.21.15.1:36858) 2024-03-06T07:26:09.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85316db640 1 -- 172.21.15.1:0/4225421910 learned_addr learned my addr 172.21.15.1:0/4225421910 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:09.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f8523fff640 1 --2- 172.21.15.1:0/4225421910 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f852c0c1260 0x7f852c0c16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f8530eda640 1 --2- 172.21.15.1:0/4225421910 >> v2:172.21.15.98:3300/0 conn(0x7f852c149a90 0x7f852c0c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85237fe640 1 -- 172.21.15.1:0/4225421910 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904476842 0 0) 0x7f852c1530a0 con 0x7f852c1565a0 2024-03-06T07:26:09.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85237fe640 1 -- 172.21.15.1:0/4225421910 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8514003520 con 0x7f852c1565a0 2024-03-06T07:26:09.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85237fe640 1 -- 172.21.15.1:0/4225421910 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1971959886 0 0) 0x7f8514003520 con 0x7f852c1565a0 2024-03-06T07:26:09.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85237fe640 1 -- 172.21.15.1:0/4225421910 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f852c0c1260 msgr2=0x7f852c0c16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85237fe640 1 --2- 172.21.15.1:0/4225421910 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f852c0c1260 0x7f852c0c16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85237fe640 1 -- 172.21.15.1:0/4225421910 >> v2:172.21.15.98:3300/0 conn(0x7f852c149a90 msgr2=0x7f852c0c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85237fe640 1 --2- 172.21.15.1:0/4225421910 >> v2:172.21.15.98:3300/0 conn(0x7f852c149a90 0x7f852c0c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85237fe640 1 -- 172.21.15.1:0/4225421910 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c153940 con 0x7f852c1565a0 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85237fe640 1 -- 172.21.15.1:0/4225421910 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f85240019f0 con 0x7f852c1565a0 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85237fe640 1 -- 172.21.15.1:0/4225421910 >> v1:172.21.15.1:6789/0 conn(0x7f852c1565a0 legacy=0x7f852c156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85237fe640 1 --2- 172.21.15.1:0/4225421910 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f852c0c1260 0x7f85140039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85237fe640 1 --2- 172.21.15.1:0/4225421910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8514003f30 0x7f8514004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85237fe640 1 --2- 172.21.15.1:0/4225421910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8514004d40 0x7f85140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85237fe640 1 -- 172.21.15.1:0/4225421910 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c153380 con 0x7f8514004d40 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85237fe640 1 -- 172.21.15.1:0/4225421910 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c153660 con 0x7f8514003f30 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.691+0000 7f85237fe640 1 -- 172.21.15.1:0/4225421910 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f851400ba10 con 0x7f852c0c1260 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.692+0000 7f8530eda640 1 --2- 172.21.15.1:0/4225421910 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f852c0c1260 0x7f85140039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.692+0000 7f85316db640 1 --2- 172.21.15.1:0/4225421910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8514004d40 0x7f85140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.692+0000 7f8523fff640 1 --2- 172.21.15.1:0/4225421910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8514003f30 0x7f8514004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.692+0000 7f85316db640 1 -- 172.21.15.1:0/4225421910 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f852c0c1260 msgr2=0x7f85140039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.692+0000 7f85316db640 1 --2- 172.21.15.1:0/4225421910 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f852c0c1260 0x7f85140039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.692+0000 7f85316db640 1 -- 172.21.15.1:0/4225421910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8514003f30 msgr2=0x7f8514004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.692+0000 7f85316db640 1 --2- 172.21.15.1:0/4225421910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8514003f30 0x7f8514004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.692+0000 7f85316db640 1 -- 172.21.15.1:0/4225421910 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8514003520 con 0x7f8514004d40 2024-03-06T07:26:09.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.692+0000 7f8530eda640 1 --2- 172.21.15.1:0/4225421910 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f852c0c1260 0x7f85140039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:09.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.693+0000 7f85316db640 1 --2- 172.21.15.1:0/4225421910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8514004d40 0x7f85140051c0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f852400a890 tx=0x7f8514005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5855008707d89d3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.693+0000 7f85237fe640 1 -- 172.21.15.1:0/4225421910 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852400a4e0 con 0x7f8514004d40 2024-03-06T07:26:09.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.693+0000 7f85237fe640 1 -- 172.21.15.1:0/4225421910 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8524034650 con 0x7f8514004d40 2024-03-06T07:26:09.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.693+0000 7f85237fe640 1 -- 172.21.15.1:0/4225421910 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8524034a40 con 0x7f8514004d40 2024-03-06T07:26:09.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.693+0000 7f853313e640 1 -- 172.21.15.1:0/4225421910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8514004d40 msgr2=0x7f85140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.693+0000 7f853313e640 1 --2- 172.21.15.1:0/4225421910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8514004d40 0x7f85140051c0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f852400a890 tx=0x7f8514005830 comp rx=0 tx=0).stop 2024-03-06T07:26:09.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.693+0000 7f85316db640 1 -- 172.21.15.1:0/4225421910 reap_dead start 2024-03-06T07:26:09.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.693+0000 7f853313e640 1 -- 172.21.15.1:0/4225421910 shutdown_connections 2024-03-06T07:26:09.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.693+0000 7f853313e640 1 -- 172.21.15.1:0/4225421910 >> 172.21.15.1:0/4225421910 conn(0x7f852c0c6860 msgr2=0x7f852c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.693+0000 7f853313e640 1 -- 172.21.15.1:0/4225421910 shutdown_connections 2024-03-06T07:26:09.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.693+0000 7f853313e640 1 -- 172.21.15.1:0/4225421910 wait complete. 2024-03-06T07:26:09.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.694+0000 7f853313e640 1 Processor -- start 2024-03-06T07:26:09.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.694+0000 7f853313e640 1 -- start start 2024-03-06T07:26:09.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.695+0000 7f853313e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f852c1ec110 0x7f852c1ec4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.695+0000 7f853313e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f852c150290 0x7f852c1ea360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.695+0000 7f853313e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f852c0ac3e0 0x7f852c1ea8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.695+0000 7f853313e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c157140 con 0x7f852c150290 2024-03-06T07:26:09.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.695+0000 7f853313e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c0c4ca0 con 0x7f852c1ec110 2024-03-06T07:26:09.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.695+0000 7f853313e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852c1e9ed0 con 0x7f852c0ac3e0 2024-03-06T07:26:09.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.695+0000 7f8523fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f852c150290 0x7f852c1ea360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.695+0000 7f8523fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f852c150290 0x7f852c1ea360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33340/0 (socket says 172.21.15.1:33340) 2024-03-06T07:26:09.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.695+0000 7f8523fff640 1 -- 172.21.15.1:0/867248705 learned_addr learned my addr 172.21.15.1:0/867248705 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:09.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.696+0000 7f8530eda640 1 --2- 172.21.15.1:0/867248705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f852c1ec110 0x7f852c1ec4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.696+0000 7f85316db640 1 --2- 172.21.15.1:0/867248705 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f852c0ac3e0 0x7f852c1ea8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.696+0000 7f8523fff640 1 -- 172.21.15.1:0/867248705 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f852c0ac3e0 msgr2=0x7f852c1ea8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.696+0000 7f8523fff640 1 --2- 172.21.15.1:0/867248705 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f852c0ac3e0 0x7f852c1ea8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.696+0000 7f8523fff640 1 -- 172.21.15.1:0/867248705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f852c1ec110 msgr2=0x7f852c1ec4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.696+0000 7f8523fff640 1 --2- 172.21.15.1:0/867248705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f852c1ec110 0x7f852c1ec4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.696+0000 7f8523fff640 1 -- 172.21.15.1:0/867248705 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c1ecc00 con 0x7f852c150290 2024-03-06T07:26:09.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.696+0000 7f85316db640 1 --2- 172.21.15.1:0/867248705 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f852c0ac3e0 0x7f852c1ea8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:09.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.696+0000 7f8523fff640 1 --2- 172.21.15.1:0/867248705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f852c150290 0x7f852c1ea360 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f851c011910 tx=0x7f851c011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.696+0000 7f85217fa640 1 -- 172.21.15.1:0/867248705 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c00fa30 con 0x7f852c150290 2024-03-06T07:26:09.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.696+0000 7f85217fa640 1 -- 172.21.15.1:0/867248705 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f851c00fbd0 con 0x7f852c150290 2024-03-06T07:26:09.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.696+0000 7f853313e640 1 -- 172.21.15.1:0/867248705 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f852c0c0cf0 con 0x7f852c150290 2024-03-06T07:26:09.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.696+0000 7f85217fa640 1 -- 172.21.15.1:0/867248705 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c014440 con 0x7f852c150290 2024-03-06T07:26:09.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.696+0000 7f853313e640 1 -- 172.21.15.1:0/867248705 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f852c1eade0 con 0x7f852c150290 2024-03-06T07:26:09.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.697+0000 7f853313e640 1 -- 172.21.15.1:0/867248705 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f852c0c0ae0 con 0x7f852c150290 2024-03-06T07:26:09.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.698+0000 7f85217fa640 1 -- 172.21.15.1:0/867248705 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f851c012040 con 0x7f852c150290 2024-03-06T07:26:09.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.698+0000 7f85217fa640 1 --2- 172.21.15.1:0/867248705 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f851806b500 0x7f851806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.698+0000 7f85217fa640 1 -- 172.21.15.1:0/867248705 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f851c08a5f0 con 0x7f852c150290 2024-03-06T07:26:09.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.698+0000 7f8530eda640 1 --2- 172.21.15.1:0/867248705 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f851806b500 0x7f851806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.699+0000 7f8530eda640 1 --2- 172.21.15.1:0/867248705 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f851806b500 0x7f851806d9c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f8528009de0 tx=0x7f8528009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.704+0000 7f85217fa640 1 -- 172.21.15.1:0/867248705 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f851c019030 con 0x7f852c150290 2024-03-06T07:26:09.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.888+0000 7f853313e640 1 -- 172.21.15.1:0/867248705 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f852c067fd0 con 0x7f852c150290 2024-03-06T07:26:09.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.889+0000 7f85217fa640 1 -- 172.21.15.1:0/867248705 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+13439 (secure 0 0 0) 0x7f851c0595c0 con 0x7f852c150290 2024-03-06T07:26:09.891 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:09.891 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":140,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:26:07.549989+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6836/1671554096":"2024-03-07T07:26:04.534764+0000","172.21.15.98:6835/1671554096":"2024-03-07T07:26:04.534764+0000","172.21.15.1:6835/2848968195":"2024-03-07T07:25:55.860421+0000","172.21.15.1:6833/2848968195":"2024-03-07T07:25:55.860421+0000","172.21.15.1:6834/1706279305":"2024-03-07T07:25:41.786393+0000","172.21.15.1:6832/1706279305":"2024-03-07T07:25:41.786393+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:09.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.891+0000 7f853313e640 1 -- 172.21.15.1:0/867248705 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f851806b500 msgr2=0x7f851806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.891+0000 7f853313e640 1 --2- 172.21.15.1:0/867248705 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f851806b500 0x7f851806d9c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f8528009de0 tx=0x7f8528009190 comp rx=0 tx=0).stop 2024-03-06T07:26:09.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.891+0000 7f853313e640 1 -- 172.21.15.1:0/867248705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f852c150290 msgr2=0x7f852c1ea360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.891+0000 7f853313e640 1 --2- 172.21.15.1:0/867248705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f852c150290 0x7f852c1ea360 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f851c011910 tx=0x7f851c011d10 comp rx=0 tx=0).stop 2024-03-06T07:26:09.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.891+0000 7f853313e640 1 -- 172.21.15.1:0/867248705 shutdown_connections 2024-03-06T07:26:09.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.891+0000 7f853313e640 1 --2- 172.21.15.1:0/867248705 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f851806b500 0x7f851806d9c0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.891+0000 7f853313e640 1 --2- 172.21.15.1:0/867248705 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f852c0ac3e0 0x7f852c1ea8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.891+0000 7f853313e640 1 --2- 172.21.15.1:0/867248705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f852c150290 0x7f852c1ea360 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.891+0000 7f853313e640 1 --2- 172.21.15.1:0/867248705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f852c1ec110 0x7f852c1ec4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.891+0000 7f853313e640 1 -- 172.21.15.1:0/867248705 >> 172.21.15.1:0/867248705 conn(0x7f852c0c6860 msgr2=0x7f852c0c5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.891+0000 7f853313e640 1 -- 172.21.15.1:0/867248705 shutdown_connections 2024-03-06T07:26:09.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:09.891+0000 7f853313e640 1 -- 172.21.15.1:0/867248705 wait complete. 2024-03-06T07:26:09.904 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:09.904 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:10.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.005+0000 7fd7c6193640 1 Processor -- start 2024-03-06T07:26:10.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.005+0000 7fd7c6193640 1 -- start start 2024-03-06T07:26:10.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.006+0000 7fd7c6193640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7c0152650 0x7fd7c0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.006+0000 7fd7c6193640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd7c00c6b10 0x7fd7c0158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.006+0000 7fd7c6193640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c00c2fc0 con 0x7fd7c00bf9d0 2024-03-06T07:26:10.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.006+0000 7fd7c6193640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c0155800 con 0x7fd7c0152650 2024-03-06T07:26:10.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.006+0000 7fd7c6193640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c00c6690 con 0x7fd7c00c6b10 2024-03-06T07:26:10.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.006+0000 7fd7bf7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7c0152650 0x7fd7c0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.006+0000 7fd7bffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd7c00bf9d0 0x7fd7c00c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36872/0 (socket says 172.21.15.1:36872) 2024-03-06T07:26:10.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.006+0000 7fd7bffff640 1 -- 172.21.15.1:0/3972803534 learned_addr learned my addr 172.21.15.1:0/3972803534 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:10.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.007+0000 7fd7beffd640 1 --2- 172.21.15.1:0/3972803534 >> v2:172.21.15.98:3300/0 conn(0x7fd7c00c6b10 0x7fd7c0158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.007+0000 7fd7bf7fe640 1 -- 172.21.15.1:0/3972803534 >> v2:172.21.15.98:3300/0 conn(0x7fd7c00c6b10 msgr2=0x7fd7c0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.007+0000 7fd7bf7fe640 1 --2- 172.21.15.1:0/3972803534 >> v2:172.21.15.98:3300/0 conn(0x7fd7c00c6b10 0x7fd7c0158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.007+0000 7fd7bf7fe640 1 -- 172.21.15.1:0/3972803534 >> v1:172.21.15.1:6789/0 conn(0x7fd7c00bf9d0 legacy=0x7fd7c00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.007+0000 7fd7bf7fe640 1 -- 172.21.15.1:0/3972803534 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c00c7280 con 0x7fd7c0152650 2024-03-06T07:26:10.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.007+0000 7fd7be7fc640 1 -- 172.21.15.1:0/3972803534 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787787960 0 0) 0x7fd7c00c2fc0 con 0x7fd7c00bf9d0 2024-03-06T07:26:10.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.007+0000 7fd7bf7fe640 1 --2- 172.21.15.1:0/3972803534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c0152650 0x7fd7c0155c90 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fd7c00cb8e0 tx=0x7fd7b002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eabb1c4919934b03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.008+0000 7fd7be7fc640 1 -- 172.21.15.1:0/3972803534 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b00315a0 con 0x7fd7c0152650 2024-03-06T07:26:10.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.008+0000 7fd7be7fc640 1 -- 172.21.15.1:0/3972803534 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b0037070 con 0x7fd7c0152650 2024-03-06T07:26:10.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.008+0000 7fd7be7fc640 1 -- 172.21.15.1:0/3972803534 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0031960 con 0x7fd7c0152650 2024-03-06T07:26:10.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.008+0000 7fd7c6193640 1 -- 172.21.15.1:0/3972803534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c0152650 msgr2=0x7fd7c0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.008+0000 7fd7c6193640 1 --2- 172.21.15.1:0/3972803534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c0152650 0x7fd7c0155c90 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fd7c00cb8e0 tx=0x7fd7b002b840 comp rx=0 tx=0).stop 2024-03-06T07:26:10.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.008+0000 7fd7c6193640 1 -- 172.21.15.1:0/3972803534 shutdown_connections 2024-03-06T07:26:10.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.008+0000 7fd7c6193640 1 --2- 172.21.15.1:0/3972803534 >> v2:172.21.15.98:3300/0 conn(0x7fd7c00c6b10 0x7fd7c0158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.008+0000 7fd7c6193640 1 --2- 172.21.15.1:0/3972803534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c0152650 0x7fd7c0155c90 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.008+0000 7fd7c6193640 1 -- 172.21.15.1:0/3972803534 >> 172.21.15.1:0/3972803534 conn(0x7fd7c0066ee0 msgr2=0x7fd7c00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.008+0000 7fd7c6193640 1 -- 172.21.15.1:0/3972803534 shutdown_connections 2024-03-06T07:26:10.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.008+0000 7fd7c6193640 1 -- 172.21.15.1:0/3972803534 wait complete. 2024-03-06T07:26:10.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.009+0000 7fd7c6193640 1 Processor -- start 2024-03-06T07:26:10.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.009+0000 7fd7c6193640 1 -- start start 2024-03-06T07:26:10.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7c6193640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c00bf9d0 0x7fd7c0153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7c6193640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7c0152650 0x7fd7c01deaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7c6193640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7c0154410 0x7fd7c01defe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7c6193640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c0155490 con 0x7fd7c00bf9d0 2024-03-06T07:26:10.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7c6193640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c01551b0 con 0x7fd7c0154410 2024-03-06T07:26:10.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7c6193640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c0152a30 con 0x7fd7c0152650 2024-03-06T07:26:10.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7bf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c00bf9d0 0x7fd7c0153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7bf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c00bf9d0 0x7fd7c0153b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33366/0 (socket says 172.21.15.1:33366) 2024-03-06T07:26:10.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7bf7fe640 1 -- 172.21.15.1:0/126316445 learned_addr learned my addr 172.21.15.1:0/126316445 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:10.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7bffff640 1 --2- 172.21.15.1:0/126316445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7c0154410 0x7fd7c01defe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7beffd640 1 --2- 172.21.15.1:0/126316445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7c0152650 0x7fd7c01deaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7bf7fe640 1 -- 172.21.15.1:0/126316445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7c0152650 msgr2=0x7fd7c01deaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7bf7fe640 1 --2- 172.21.15.1:0/126316445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7c0152650 0x7fd7c01deaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7bf7fe640 1 -- 172.21.15.1:0/126316445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7c0154410 msgr2=0x7fd7c01defe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7bf7fe640 1 --2- 172.21.15.1:0/126316445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7c0154410 0x7fd7c01defe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7bf7fe640 1 -- 172.21.15.1:0/126316445 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c0154c60 con 0x7fd7c00bf9d0 2024-03-06T07:26:10.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7bffff640 1 --2- 172.21.15.1:0/126316445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7c0154410 0x7fd7c01defe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:10.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.010+0000 7fd7beffd640 1 --2- 172.21.15.1:0/126316445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7c0152650 0x7fd7c01deaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:10.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.011+0000 7fd7bf7fe640 1 --2- 172.21.15.1:0/126316445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c00bf9d0 0x7fd7c0153b90 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0000c00 tx=0x7fd7b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.011+0000 7fd7a7fff640 1 -- 172.21.15.1:0/126316445 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0032840 con 0x7fd7c00bf9d0 2024-03-06T07:26:10.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.011+0000 7fd7a7fff640 1 -- 172.21.15.1:0/126316445 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b0032a00 con 0x7fd7c00bf9d0 2024-03-06T07:26:10.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.011+0000 7fd7a7fff640 1 -- 172.21.15.1:0/126316445 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0032df0 con 0x7fd7c00bf9d0 2024-03-06T07:26:10.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.011+0000 7fd7c6193640 1 -- 172.21.15.1:0/126316445 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c01eb8b0 con 0x7fd7c00bf9d0 2024-03-06T07:26:10.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.011+0000 7fd7c6193640 1 -- 172.21.15.1:0/126316445 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c0154960 con 0x7fd7c00bf9d0 2024-03-06T07:26:10.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.012+0000 7fd7c6193640 1 -- 172.21.15.1:0/126316445 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c01de7b0 con 0x7fd7c00bf9d0 2024-03-06T07:26:10.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.012+0000 7fd7a7fff640 1 -- 172.21.15.1:0/126316445 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd7b004c020 con 0x7fd7c00bf9d0 2024-03-06T07:26:10.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.012+0000 7fd7a7fff640 1 --2- 172.21.15.1:0/126316445 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd7ac06b4d0 0x7fd7ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.013+0000 7fd7a7fff640 1 -- 172.21.15.1:0/126316445 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fd7b002f020 con 0x7fd7c00bf9d0 2024-03-06T07:26:10.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.013+0000 7fd7beffd640 1 --2- 172.21.15.1:0/126316445 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd7ac06b4d0 0x7fd7ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.013+0000 7fd7beffd640 1 --2- 172.21.15.1:0/126316445 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd7ac06b4d0 0x7fd7ac06d990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fd7c00c2430 tx=0x7fd7b4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.017+0000 7fd7a7fff640 1 -- 172.21.15.1:0/126316445 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd7b0040070 con 0x7fd7c00bf9d0 2024-03-06T07:26:10.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.200+0000 7fd7c6193640 1 -- 172.21.15.1:0/126316445 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd7c0067fd0 con 0x7fd7c00bf9d0 2024-03-06T07:26:10.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.200+0000 7fd7a7fff640 1 -- 172.21.15.1:0/126316445 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v140) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd7b007b100 con 0x7fd7c00bf9d0 2024-03-06T07:26:10.202 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:26:10.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.202+0000 7fd7c6193640 1 -- 172.21.15.1:0/126316445 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd7ac06b4d0 msgr2=0x7fd7ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.202+0000 7fd7c6193640 1 --2- 172.21.15.1:0/126316445 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd7ac06b4d0 0x7fd7ac06d990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fd7c00c2430 tx=0x7fd7b4008040 comp rx=0 tx=0).stop 2024-03-06T07:26:10.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.202+0000 7fd7c6193640 1 -- 172.21.15.1:0/126316445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c00bf9d0 msgr2=0x7fd7c0153b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.202+0000 7fd7c6193640 1 --2- 172.21.15.1:0/126316445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c00bf9d0 0x7fd7c0153b90 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0000c00 tx=0x7fd7b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.202+0000 7fd7c6193640 1 -- 172.21.15.1:0/126316445 shutdown_connections 2024-03-06T07:26:10.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.202+0000 7fd7c6193640 1 --2- 172.21.15.1:0/126316445 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd7ac06b4d0 0x7fd7ac06d990 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.202+0000 7fd7c6193640 1 --2- 172.21.15.1:0/126316445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7c0154410 0x7fd7c01defe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.202+0000 7fd7c6193640 1 --2- 172.21.15.1:0/126316445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7c0152650 0x7fd7c01deaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.202+0000 7fd7c6193640 1 --2- 172.21.15.1:0/126316445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c00bf9d0 0x7fd7c0153b90 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.202+0000 7fd7c6193640 1 -- 172.21.15.1:0/126316445 >> 172.21.15.1:0/126316445 conn(0x7fd7c0066ee0 msgr2=0x7fd7c014e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.202+0000 7fd7c6193640 1 -- 172.21.15.1:0/126316445 shutdown_connections 2024-03-06T07:26:10.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.202+0000 7fd7c6193640 1 -- 172.21.15.1:0/126316445 wait complete. 2024-03-06T07:26:10.215 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla' 2024-03-06T07:26:10.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.314+0000 7f32c30df640 1 Processor -- start 2024-03-06T07:26:10.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.314+0000 7f32c30df640 1 -- start start 2024-03-06T07:26:10.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.315+0000 7f32c30df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32bc06e670 0x7f32bc06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.315+0000 7f32c30df640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f32bc0a8520 0x7f32bc0c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.315+0000 7f32c30df640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32bc06f060 con 0x7f32bc150290 2024-03-06T07:26:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.315+0000 7f32c30df640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32bc0c2ca0 con 0x7f32bc0a8520 2024-03-06T07:26:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.315+0000 7f32c30df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc0c26e0 con 0x7f32bc06e670 2024-03-06T07:26:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.315+0000 7f32c0e7b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32bc06e670 0x7f32bc06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.315+0000 7f32bbfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f32bc0a8520 0x7f32bc0c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.315+0000 7f32c0e7b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32bc06e670 0x7f32bc06ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:45424/0 (socket says 172.21.15.1:45424) 2024-03-06T07:26:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.315+0000 7f32c0e7b640 1 -- 172.21.15.1:0/1554728848 learned_addr learned my addr 172.21.15.1:0/1554728848 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.316+0000 7f32c0e7b640 1 -- 172.21.15.1:0/1554728848 >> v2:172.21.15.98:3300/0 conn(0x7f32bc0a8520 msgr2=0x7f32bc0c1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.316+0000 7f32c0e7b640 1 --2- 172.21.15.1:0/1554728848 >> v2:172.21.15.98:3300/0 conn(0x7f32bc0a8520 0x7f32bc0c1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.316+0000 7f32c0e7b640 1 -- 172.21.15.1:0/1554728848 >> v1:172.21.15.98:6789/0 conn(0x7f32bc150290 legacy=0x7f32bc153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.316+0000 7f32c0e7b640 1 -- 172.21.15.1:0/1554728848 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32bc0c29c0 con 0x7f32bc06e670 2024-03-06T07:26:10.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.317+0000 7f32c0e7b640 1 --2- 172.21.15.1:0/1554728848 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32bc06e670 0x7f32bc06ea50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f32b0009870 tx=0x7f32b002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b34eb9a31dc2d73b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.317+0000 7f32bb7fe640 1 -- 172.21.15.1:0/1554728848 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b0035020 con 0x7f32bc06e670 2024-03-06T07:26:10.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.317+0000 7f32bb7fe640 1 -- 172.21.15.1:0/1554728848 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b0031640 con 0x7f32bc06e670 2024-03-06T07:26:10.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.317+0000 7f32bb7fe640 1 -- 172.21.15.1:0/1554728848 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b002d040 con 0x7f32bc06e670 2024-03-06T07:26:10.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.317+0000 7f32c30df640 1 -- 172.21.15.1:0/1554728848 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32bc06e670 msgr2=0x7f32bc06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.317+0000 7f32c30df640 1 --2- 172.21.15.1:0/1554728848 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32bc06e670 0x7f32bc06ea50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f32b0009870 tx=0x7f32b002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.317+0000 7f32c30df640 1 -- 172.21.15.1:0/1554728848 shutdown_connections 2024-03-06T07:26:10.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.317+0000 7f32c30df640 1 --2- 172.21.15.1:0/1554728848 >> v2:172.21.15.98:3300/0 conn(0x7f32bc0a8520 0x7f32bc0c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.317+0000 7f32c30df640 1 --2- 172.21.15.1:0/1554728848 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32bc06e670 0x7f32bc06ea50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.317+0000 7f32c30df640 1 -- 172.21.15.1:0/1554728848 >> 172.21.15.1:0/1554728848 conn(0x7f32bc0c3cc0 msgr2=0x7f32bc0c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.317+0000 7f32c30df640 1 -- 172.21.15.1:0/1554728848 shutdown_connections 2024-03-06T07:26:10.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.317+0000 7f32c30df640 1 -- 172.21.15.1:0/1554728848 wait complete. 2024-03-06T07:26:10.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.318+0000 7f32c30df640 1 Processor -- start 2024-03-06T07:26:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.318+0000 7f32c30df640 1 -- start start 2024-03-06T07:26:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.319+0000 7f32c30df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32bc06e670 0x7f32bc155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.319+0000 7f32c30df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0a8520 0x7f32bc1ebd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.319+0000 7f32c30df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32bc150290 0x7f32bc1525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.319+0000 7f32c30df640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc150af0 con 0x7f32bc0a8520 2024-03-06T07:26:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.319+0000 7f32c30df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc150970 con 0x7f32bc150290 2024-03-06T07:26:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.319+0000 7f32c30df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc1539a0 con 0x7f32bc06e670 2024-03-06T07:26:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.319+0000 7f32bbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0a8520 0x7f32bc1ebd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.319+0000 7f32bbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0a8520 0x7f32bc1ebd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33372/0 (socket says 172.21.15.1:33372) 2024-03-06T07:26:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.319+0000 7f32bbfff640 1 -- 172.21.15.1:0/1781800653 learned_addr learned my addr 172.21.15.1:0/1781800653 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.320+0000 7f32c0e7b640 1 --2- 172.21.15.1:0/1781800653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32bc06e670 0x7f32bc155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.320+0000 7f32c167c640 1 --2- 172.21.15.1:0/1781800653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32bc150290 0x7f32bc1525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.320+0000 7f32bbfff640 1 -- 172.21.15.1:0/1781800653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32bc06e670 msgr2=0x7f32bc155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.320+0000 7f32bbfff640 1 --2- 172.21.15.1:0/1781800653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32bc06e670 0x7f32bc155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.320+0000 7f32bbfff640 1 -- 172.21.15.1:0/1781800653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32bc150290 msgr2=0x7f32bc1525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.320+0000 7f32bbfff640 1 --2- 172.21.15.1:0/1781800653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32bc150290 0x7f32bc1525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.320+0000 7f32bbfff640 1 -- 172.21.15.1:0/1781800653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32bc146440 con 0x7f32bc0a8520 2024-03-06T07:26:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.320+0000 7f32bbfff640 1 --2- 172.21.15.1:0/1781800653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0a8520 0x7f32bc1ebd40 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f32a800e9b0 tx=0x7f32a800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.320+0000 7f32b97fa640 1 -- 172.21.15.1:0/1781800653 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a8017070 con 0x7f32bc0a8520 2024-03-06T07:26:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.320+0000 7f32b97fa640 1 -- 172.21.15.1:0/1781800653 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a800c9a0 con 0x7f32bc0a8520 2024-03-06T07:26:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.320+0000 7f32c30df640 1 -- 172.21.15.1:0/1781800653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32bc156400 con 0x7f32bc0a8520 2024-03-06T07:26:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.320+0000 7f32b97fa640 1 -- 172.21.15.1:0/1781800653 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a800cd90 con 0x7f32bc0a8520 2024-03-06T07:26:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.320+0000 7f32c30df640 1 -- 172.21.15.1:0/1781800653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32bc1569c0 con 0x7f32bc0a8520 2024-03-06T07:26:10.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.321+0000 7f32c30df640 1 -- 172.21.15.1:0/1781800653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32bc067fd0 con 0x7f32bc0a8520 2024-03-06T07:26:10.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.322+0000 7f32b97fa640 1 -- 172.21.15.1:0/1781800653 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f32a801a070 con 0x7f32bc0a8520 2024-03-06T07:26:10.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.322+0000 7f32b97fa640 1 --2- 172.21.15.1:0/1781800653 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32a006b4d0 0x7f32a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.322+0000 7f32b97fa640 1 -- 172.21.15.1:0/1781800653 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f32a808a440 con 0x7f32bc0a8520 2024-03-06T07:26:10.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.322+0000 7f32c0e7b640 1 --2- 172.21.15.1:0/1781800653 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32a006b4d0 0x7f32a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.323+0000 7f32c0e7b640 1 --2- 172.21.15.1:0/1781800653 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32a006b4d0 0x7f32a006d990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f32b0036000 tx=0x7f32b003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.326+0000 7f32b97fa640 1 -- 172.21.15.1:0/1781800653 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32a8059410 con 0x7f32bc0a8520 2024-03-06T07:26:10.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:10.581+0000 7f32c30df640 1 -- 172.21.15.1:0/1781800653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]} v 0) v1 -- 0x7f32bc067570 con 0x7f32bc0a8520 2024-03-06T07:26:11.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.569+0000 7f32b97fa640 1 -- 172.21.15.1:0/1781800653 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]}]=0 v341) v1 ==== 149+0+0 (secure 0 0 0) 0x7f32a8058d10 con 0x7f32bc0a8520 2024-03-06T07:26:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.572+0000 7f32c30df640 1 -- 172.21.15.1:0/1781800653 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32a006b4d0 msgr2=0x7f32a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.572+0000 7f32c30df640 1 --2- 172.21.15.1:0/1781800653 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32a006b4d0 0x7f32a006d990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f32b0036000 tx=0x7f32b003c040 comp rx=0 tx=0).stop 2024-03-06T07:26:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.572+0000 7f32c30df640 1 -- 172.21.15.1:0/1781800653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0a8520 msgr2=0x7f32bc1ebd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.572+0000 7f32c30df640 1 --2- 172.21.15.1:0/1781800653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0a8520 0x7f32bc1ebd40 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f32a800e9b0 tx=0x7f32a800ee70 comp rx=0 tx=0).stop 2024-03-06T07:26:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.572+0000 7f32c30df640 1 -- 172.21.15.1:0/1781800653 shutdown_connections 2024-03-06T07:26:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.572+0000 7f32c30df640 1 --2- 172.21.15.1:0/1781800653 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32a006b4d0 0x7f32a006d990 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.572+0000 7f32c30df640 1 --2- 172.21.15.1:0/1781800653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32bc150290 0x7f32bc1525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.572+0000 7f32c30df640 1 --2- 172.21.15.1:0/1781800653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0a8520 0x7f32bc1ebd40 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.572+0000 7f32c30df640 1 --2- 172.21.15.1:0/1781800653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32bc06e670 0x7f32bc155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.573+0000 7f32c30df640 1 -- 172.21.15.1:0/1781800653 >> 172.21.15.1:0/1781800653 conn(0x7f32bc0c3cc0 msgr2=0x7f32bc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.573+0000 7f32c30df640 1 -- 172.21.15.1:0/1781800653 shutdown_connections 2024-03-06T07:26:11.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.573+0000 7f32c30df640 1 -- 172.21.15.1:0/1781800653 wait complete. 2024-03-06T07:26:11.589 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:26:11.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.700+0000 7fcbe058a640 1 Processor -- start 2024-03-06T07:26:11.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.700+0000 7fcbe058a640 1 -- start start 2024-03-06T07:26:11.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.701+0000 7fcbe058a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcbd80c32f0 0x7fcbd81543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.701+0000 7fcbe058a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcbd8070c80 0x7fcbd80710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.701+0000 7fcbe058a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd8150900 con 0x7fcbd8150520 2024-03-06T07:26:11.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.701+0000 7fcbe058a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd8153ed0 con 0x7fcbd80c32f0 2024-03-06T07:26:11.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.701+0000 7fcbe058a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd8071790 con 0x7fcbd8070c80 2024-03-06T07:26:11.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.701+0000 7fcbde326640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcbd8150520 0x7fcbd81537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36878/0 (socket says 172.21.15.1:36878) 2024-03-06T07:26:11.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.701+0000 7fcbde326640 1 -- 172.21.15.1:0/2637210440 learned_addr learned my addr 172.21.15.1:0/2637210440 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:11.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.701+0000 7fcbddb25640 1 --2- 172.21.15.1:0/2637210440 >> v2:172.21.15.1:3300/0 conn(0x7fcbd80c32f0 0x7fcbd81543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.702+0000 7fcbddb25640 1 -- 172.21.15.1:0/2637210440 >> v2:172.21.15.98:3300/0 conn(0x7fcbd8070c80 msgr2=0x7fcbd80710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.702+0000 7fcbddb25640 1 --2- 172.21.15.1:0/2637210440 >> v2:172.21.15.98:3300/0 conn(0x7fcbd8070c80 0x7fcbd80710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.702+0000 7fcbddb25640 1 -- 172.21.15.1:0/2637210440 >> v1:172.21.15.1:6789/0 conn(0x7fcbd8150520 legacy=0x7fcbd81537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.702+0000 7fcbdd324640 1 -- 172.21.15.1:0/2637210440 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641914453 0 0) 0x7fcbd8150900 con 0x7fcbd8150520 2024-03-06T07:26:11.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.702+0000 7fcbddb25640 1 -- 172.21.15.1:0/2637210440 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd8157050 con 0x7fcbd80c32f0 2024-03-06T07:26:11.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.702+0000 7fcbddb25640 1 --2- 172.21.15.1:0/2637210440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbd80c32f0 0x7fcbd81543a0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fcbd80c3190 tx=0x7fcbcc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41bd28464c4ac043 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.703+0000 7fcbdd324640 1 -- 172.21.15.1:0/2637210440 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc035020 con 0x7fcbd80c32f0 2024-03-06T07:26:11.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.703+0000 7fcbdd324640 1 -- 172.21.15.1:0/2637210440 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbcc031830 con 0x7fcbd80c32f0 2024-03-06T07:26:11.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.703+0000 7fcbdd324640 1 -- 172.21.15.1:0/2637210440 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc031c00 con 0x7fcbd80c32f0 2024-03-06T07:26:11.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.703+0000 7fcbe058a640 1 -- 172.21.15.1:0/2637210440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbd80c32f0 msgr2=0x7fcbd81543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.703+0000 7fcbe058a640 1 --2- 172.21.15.1:0/2637210440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbd80c32f0 0x7fcbd81543a0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fcbd80c3190 tx=0x7fcbcc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.703+0000 7fcbe058a640 1 -- 172.21.15.1:0/2637210440 shutdown_connections 2024-03-06T07:26:11.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.703+0000 7fcbe058a640 1 --2- 172.21.15.1:0/2637210440 >> v2:172.21.15.98:3300/0 conn(0x7fcbd8070c80 0x7fcbd80710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.703+0000 7fcbe058a640 1 --2- 172.21.15.1:0/2637210440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbd80c32f0 0x7fcbd81543a0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.703+0000 7fcbe058a640 1 -- 172.21.15.1:0/2637210440 >> 172.21.15.1:0/2637210440 conn(0x7fcbd80c88f0 msgr2=0x7fcbd80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.703+0000 7fcbe058a640 1 -- 172.21.15.1:0/2637210440 shutdown_connections 2024-03-06T07:26:11.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.703+0000 7fcbe058a640 1 -- 172.21.15.1:0/2637210440 wait complete. 2024-03-06T07:26:11.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.704+0000 7fcbe058a640 1 Processor -- start 2024-03-06T07:26:11.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.704+0000 7fcbe058a640 1 -- start start 2024-03-06T07:26:11.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.705+0000 7fcbe058a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbd8070c80 0x7fcbd80acb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.705+0000 7fcbe058a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbd80c32f0 0x7fcbd8146170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.705+0000 7fcbe058a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbd8150520 0x7fcbd81466b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.705+0000 7fcbe058a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd8157330 con 0x7fcbd80c32f0 2024-03-06T07:26:11.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.705+0000 7fcbe058a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd81532a0 con 0x7fcbd8070c80 2024-03-06T07:26:11.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.705+0000 7fcbe058a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd80c7ec0 con 0x7fcbd8150520 2024-03-06T07:26:11.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.705+0000 7fcbddb25640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbd80c32f0 0x7fcbd8146170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.705+0000 7fcbddb25640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbd80c32f0 0x7fcbd8146170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33396/0 (socket says 172.21.15.1:33396) 2024-03-06T07:26:11.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.705+0000 7fcbddb25640 1 -- 172.21.15.1:0/3508625002 learned_addr learned my addr 172.21.15.1:0/3508625002 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:11.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.705+0000 7fcbddb25640 1 -- 172.21.15.1:0/3508625002 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbd8150520 msgr2=0x7fcbd81466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.705+0000 7fcbddb25640 1 --2- 172.21.15.1:0/3508625002 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbd8150520 0x7fcbd81466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.705+0000 7fcbde326640 1 --2- 172.21.15.1:0/3508625002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbd8070c80 0x7fcbd80acb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.705+0000 7fcbddb25640 1 -- 172.21.15.1:0/3508625002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbd8070c80 msgr2=0x7fcbd80acb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.705+0000 7fcbddb25640 1 --2- 172.21.15.1:0/3508625002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbd8070c80 0x7fcbd80acb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.705+0000 7fcbddb25640 1 -- 172.21.15.1:0/3508625002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd8149c80 con 0x7fcbd80c32f0 2024-03-06T07:26:11.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.706+0000 7fcbddb25640 1 --2- 172.21.15.1:0/3508625002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbd80c32f0 0x7fcbd8146170 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc009890 tx=0x7fcbcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.706+0000 7fcbd6ffd640 1 -- 172.21.15.1:0/3508625002 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc004230 con 0x7fcbd80c32f0 2024-03-06T07:26:11.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.706+0000 7fcbd6ffd640 1 -- 172.21.15.1:0/3508625002 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbcc0043d0 con 0x7fcbd80c32f0 2024-03-06T07:26:11.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.706+0000 7fcbe058a640 1 -- 172.21.15.1:0/3508625002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd81475e0 con 0x7fcbd80c32f0 2024-03-06T07:26:11.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.706+0000 7fcbd6ffd640 1 -- 172.21.15.1:0/3508625002 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc03cad0 con 0x7fcbd80c32f0 2024-03-06T07:26:11.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.706+0000 7fcbe058a640 1 -- 172.21.15.1:0/3508625002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd806e0b0 con 0x7fcbd80c32f0 2024-03-06T07:26:11.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.707+0000 7fcbe058a640 1 -- 172.21.15.1:0/3508625002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd806e900 con 0x7fcbd80c32f0 2024-03-06T07:26:11.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.707+0000 7fcbd6ffd640 1 -- 172.21.15.1:0/3508625002 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcbcc038020 con 0x7fcbd80c32f0 2024-03-06T07:26:11.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.708+0000 7fcbd6ffd640 1 --2- 172.21.15.1:0/3508625002 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcbc406b4d0 0x7fcbc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.708+0000 7fcbd6ffd640 1 -- 172.21.15.1:0/3508625002 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fcbcc0ac2c0 con 0x7fcbd80c32f0 2024-03-06T07:26:11.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.708+0000 7fcbde326640 1 --2- 172.21.15.1:0/3508625002 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcbc406b4d0 0x7fcbc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.708+0000 7fcbde326640 1 --2- 172.21.15.1:0/3508625002 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcbc406b4d0 0x7fcbc406d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0000f60 tx=0x7fcbd0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.712+0000 7fcbd6ffd640 1 -- 172.21.15.1:0/3508625002 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcbd806e900 con 0x7fcbd80c32f0 2024-03-06T07:26:11.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.893+0000 7fcbe058a640 1 -- 172.21.15.1:0/3508625002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcbd80686e0 con 0x7fcbd80c32f0 2024-03-06T07:26:11.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.896+0000 7fcbd6ffd640 1 -- 172.21.15.1:0/3508625002 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v51)=0 v51) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcbcc041070 con 0x7fcbd80c32f0 2024-03-06T07:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.898+0000 7fcbe058a640 1 -- 172.21.15.1:0/3508625002 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcbc406b4d0 msgr2=0x7fcbc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.898+0000 7fcbe058a640 1 --2- 172.21.15.1:0/3508625002 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcbc406b4d0 0x7fcbc406d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0000f60 tx=0x7fcbd0009040 comp rx=0 tx=0).stop 2024-03-06T07:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.898+0000 7fcbe058a640 1 -- 172.21.15.1:0/3508625002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbd80c32f0 msgr2=0x7fcbd8146170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.898+0000 7fcbe058a640 1 --2- 172.21.15.1:0/3508625002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbd80c32f0 0x7fcbd8146170 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc009890 tx=0x7fcbcc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.898+0000 7fcbe058a640 1 -- 172.21.15.1:0/3508625002 shutdown_connections 2024-03-06T07:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.898+0000 7fcbe058a640 1 --2- 172.21.15.1:0/3508625002 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcbc406b4d0 0x7fcbc406d990 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.898+0000 7fcbe058a640 1 --2- 172.21.15.1:0/3508625002 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbd8150520 0x7fcbd81466b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.898+0000 7fcbe058a640 1 --2- 172.21.15.1:0/3508625002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbd80c32f0 0x7fcbd8146170 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.898+0000 7fcbe058a640 1 --2- 172.21.15.1:0/3508625002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbd8070c80 0x7fcbd80acb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.898+0000 7fcbe058a640 1 -- 172.21.15.1:0/3508625002 >> 172.21.15.1:0/3508625002 conn(0x7fcbd80c88f0 msgr2=0x7fcbd80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.898+0000 7fcbe058a640 1 -- 172.21.15.1:0/3508625002 shutdown_connections 2024-03-06T07:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:11.898+0000 7fcbe058a640 1 -- 172.21.15.1:0/3508625002 wait complete. 2024-03-06T07:26:11.911 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:11.911 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:11.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:12.000 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:12.000 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:26:12.000 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:12.014 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:26:12.015 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:26:12.015 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:12.032 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:26:12.032 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:26:12.032 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:12.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.154+0000 7fe445f16640 1 Processor -- start 2024-03-06T07:26:12.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.154+0000 7fe445f16640 1 -- start start 2024-03-06T07:26:12.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.154+0000 7fe445f16640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe440152690 0x7fe440155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.154+0000 7fe445f16640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe440156cd0 0x7fe440157160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.154+0000 7fe445f16640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4401561e0 con 0x7fe4400c6730 2024-03-06T07:26:12.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.154+0000 7fe445f16640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4400c3300 con 0x7fe440152690 2024-03-06T07:26:12.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.154+0000 7fe445f16640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4400c35a0 con 0x7fe440156cd0 2024-03-06T07:26:12.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43effd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe4400c6730 0x7fe4400c57d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36884/0 (socket says 172.21.15.1:36884) 2024-03-06T07:26:12.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43effd640 1 -- 172.21.15.1:0/1711112246 learned_addr learned my addr 172.21.15.1:0/1711112246 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:12.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43ffff640 1 --2- 172.21.15.1:0/1711112246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe440156cd0 0x7fe440157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43f7fe640 1 --2- 172.21.15.1:0/1711112246 >> v2:172.21.15.98:3300/0 conn(0x7fe440152690 0x7fe440155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43e7fc640 1 -- 172.21.15.1:0/1711112246 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716285106 0 0) 0x7fe434001110 con 0x7fe4400c6730 2024-03-06T07:26:12.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43e7fc640 1 -- 172.21.15.1:0/1711112246 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe428003520 con 0x7fe4400c6730 2024-03-06T07:26:12.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43e7fc640 1 -- 172.21.15.1:0/1711112246 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1109203046 0 0) 0x7fe428003520 con 0x7fe4400c6730 2024-03-06T07:26:12.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43e7fc640 1 -- 172.21.15.1:0/1711112246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe440156cd0 msgr2=0x7fe440157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43e7fc640 1 --2- 172.21.15.1:0/1711112246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe440156cd0 0x7fe440157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43e7fc640 1 -- 172.21.15.1:0/1711112246 >> v2:172.21.15.98:3300/0 conn(0x7fe440152690 msgr2=0x7fe440155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43e7fc640 1 --2- 172.21.15.1:0/1711112246 >> v2:172.21.15.98:3300/0 conn(0x7fe440152690 0x7fe440155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43e7fc640 1 -- 172.21.15.1:0/1711112246 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4400c3880 con 0x7fe4400c6730 2024-03-06T07:26:12.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43e7fc640 1 -- 172.21.15.1:0/1711112246 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fe434001840 con 0x7fe4400c6730 2024-03-06T07:26:12.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43e7fc640 1 -- 172.21.15.1:0/1711112246 >> v1:172.21.15.1:6789/0 conn(0x7fe4400c6730 legacy=0x7fe4400c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43e7fc640 1 --2- 172.21.15.1:0/1711112246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe440156cd0 0x7fe4280039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43e7fc640 1 --2- 172.21.15.1:0/1711112246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe428003f30 0x7fe428004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43e7fc640 1 --2- 172.21.15.1:0/1711112246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe428004d40 0x7fe4280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43e7fc640 1 -- 172.21.15.1:0/1711112246 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4400c3300 con 0x7fe428004d40 2024-03-06T07:26:12.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43e7fc640 1 -- 172.21.15.1:0/1711112246 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4400c35a0 con 0x7fe428003f30 2024-03-06T07:26:12.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43e7fc640 1 -- 172.21.15.1:0/1711112246 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe42800ba10 con 0x7fe440156cd0 2024-03-06T07:26:12.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43ffff640 1 --2- 172.21.15.1:0/1711112246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe428004d40 0x7fe4280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43f7fe640 1 --2- 172.21.15.1:0/1711112246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe440156cd0 0x7fe4280039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.155+0000 7fe43effd640 1 --2- 172.21.15.1:0/1711112246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe428003f30 0x7fe428004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.156+0000 7fe43ffff640 1 -- 172.21.15.1:0/1711112246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe440156cd0 msgr2=0x7fe4280039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.156+0000 7fe43ffff640 1 --2- 172.21.15.1:0/1711112246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe440156cd0 0x7fe4280039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.156+0000 7fe43ffff640 1 -- 172.21.15.1:0/1711112246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe428003f30 msgr2=0x7fe428004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.156+0000 7fe43ffff640 1 --2- 172.21.15.1:0/1711112246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe428003f30 0x7fe428004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.156+0000 7fe43ffff640 1 -- 172.21.15.1:0/1711112246 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe428003520 con 0x7fe428004d40 2024-03-06T07:26:12.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.156+0000 7fe43ffff640 1 --2- 172.21.15.1:0/1711112246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe428004d40 0x7fe4280051c0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fe43800e170 tx=0x7fe428005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69c82a1c01a25de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.156+0000 7fe43e7fc640 1 -- 172.21.15.1:0/1711112246 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe438039070 con 0x7fe428004d40 2024-03-06T07:26:12.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.156+0000 7fe43e7fc640 1 -- 172.21.15.1:0/1711112246 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe43802cbf0 con 0x7fe428004d40 2024-03-06T07:26:12.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.156+0000 7fe43e7fc640 1 -- 172.21.15.1:0/1711112246 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe438009c10 con 0x7fe428004d40 2024-03-06T07:26:12.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.157+0000 7fe445f16640 1 -- 172.21.15.1:0/1711112246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe428004d40 msgr2=0x7fe4280051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.157+0000 7fe445f16640 1 --2- 172.21.15.1:0/1711112246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe428004d40 0x7fe4280051c0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fe43800e170 tx=0x7fe428005830 comp rx=0 tx=0).stop 2024-03-06T07:26:12.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.157+0000 7fe43ffff640 1 -- 172.21.15.1:0/1711112246 reap_dead start 2024-03-06T07:26:12.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.157+0000 7fe445f16640 1 -- 172.21.15.1:0/1711112246 shutdown_connections 2024-03-06T07:26:12.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.157+0000 7fe445f16640 1 -- 172.21.15.1:0/1711112246 >> 172.21.15.1:0/1711112246 conn(0x7fe44006b410 msgr2=0x7fe44006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.157+0000 7fe445f16640 1 -- 172.21.15.1:0/1711112246 shutdown_connections 2024-03-06T07:26:12.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.157+0000 7fe445f16640 1 -- 172.21.15.1:0/1711112246 wait complete. 2024-03-06T07:26:12.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.158+0000 7fe445f16640 1 Processor -- start 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.158+0000 7fe445f16640 1 -- start start 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.159+0000 7fe445f16640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe428004d40 0x7fe4400ac740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.159+0000 7fe445f16640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4400ad880 0x7fe44006d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.159+0000 7fe445f16640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4401e2310 0x7fe4401ee440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.159+0000 7fe445f16640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4400c5ee0 con 0x7fe4400ad880 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.159+0000 7fe445f16640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4401567b0 con 0x7fe428004d40 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.159+0000 7fe445f16640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4400c6060 con 0x7fe4401e2310 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.159+0000 7fe43effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4400ad880 0x7fe44006d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.159+0000 7fe43effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4400ad880 0x7fe44006d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33408/0 (socket says 172.21.15.1:33408) 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.159+0000 7fe43effd640 1 -- 172.21.15.1:0/1372324548 learned_addr learned my addr 172.21.15.1:0/1372324548 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.159+0000 7fe43f7fe640 1 --2- 172.21.15.1:0/1372324548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe428004d40 0x7fe4400ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.159+0000 7fe43ffff640 1 --2- 172.21.15.1:0/1372324548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4401e2310 0x7fe4401ee440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.159+0000 7fe43effd640 1 -- 172.21.15.1:0/1372324548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4401e2310 msgr2=0x7fe4401ee440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.159+0000 7fe43effd640 1 --2- 172.21.15.1:0/1372324548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4401e2310 0x7fe4401ee440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.159+0000 7fe43effd640 1 -- 172.21.15.1:0/1372324548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe428004d40 msgr2=0x7fe4400ac740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.159+0000 7fe43effd640 1 --2- 172.21.15.1:0/1372324548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe428004d40 0x7fe4400ac740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.159+0000 7fe43effd640 1 -- 172.21.15.1:0/1372324548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4400c7e40 con 0x7fe4400ad880 2024-03-06T07:26:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.160+0000 7fe43effd640 1 --2- 172.21.15.1:0/1372324548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4400ad880 0x7fe44006d4f0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fe434000c00 tx=0x7fe434007660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.160+0000 7fe427fff640 1 -- 172.21.15.1:0/1372324548 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe434007790 con 0x7fe4400ad880 2024-03-06T07:26:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.160+0000 7fe427fff640 1 -- 172.21.15.1:0/1372324548 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe434018070 con 0x7fe4400ad880 2024-03-06T07:26:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.160+0000 7fe445f16640 1 -- 172.21.15.1:0/1372324548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4400c8b40 con 0x7fe4400ad880 2024-03-06T07:26:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.160+0000 7fe427fff640 1 -- 172.21.15.1:0/1372324548 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43400fc00 con 0x7fe4400ad880 2024-03-06T07:26:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.160+0000 7fe445f16640 1 -- 172.21.15.1:0/1372324548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4400c7b60 con 0x7fe4400ad880 2024-03-06T07:26:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.161+0000 7fe445f16640 1 -- 172.21.15.1:0/1372324548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4400c2eb0 con 0x7fe4400ad880 2024-03-06T07:26:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.161+0000 7fe427fff640 1 -- 172.21.15.1:0/1372324548 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe43401d460 con 0x7fe4400ad880 2024-03-06T07:26:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.162+0000 7fe427fff640 1 --2- 172.21.15.1:0/1372324548 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe42c06b4d0 0x7fe42c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.162+0000 7fe427fff640 1 -- 172.21.15.1:0/1372324548 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fe43408a120 con 0x7fe4400ad880 2024-03-06T07:26:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.162+0000 7fe43f7fe640 1 --2- 172.21.15.1:0/1372324548 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe42c06b4d0 0x7fe42c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.162+0000 7fe43f7fe640 1 --2- 172.21.15.1:0/1372324548 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe42c06b4d0 0x7fe42c06d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fe430009c40 tx=0x7fe430009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.166+0000 7fe427fff640 1 -- 172.21.15.1:0/1372324548 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe4400c2eb0 con 0x7fe4400ad880 2024-03-06T07:26:12.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.384+0000 7fe445f16640 1 -- 172.21.15.1:0/1372324548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe440067fd0 con 0x7fe4400ad880 2024-03-06T07:26:12.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.384+0000 7fe427fff640 1 -- 172.21.15.1:0/1372324548 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+3698 (secure 0 0 0) 0x7fe4401561e0 con 0x7fe4400ad880 2024-03-06T07:26:12.386 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:12.386 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":133,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2739580331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2739580331},{"type":"v1","addr":"172.21.15.98:6837","nonce":2739580331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7702,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2944361744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2944361744},{"type":"v1","addr":"172.21.15.98:6836","nonce":2944361744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":8517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/3374845636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3374845636},{"type":"v1","addr":"172.21.15.1:6834","nonce":3374845636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":8607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1602915919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1602915919},{"type":"v1","addr":"172.21.15.1:6835","nonce":1602915919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129}],"filesystems":[]} 2024-03-06T07:26:12.387 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 133 2024-03-06T07:26:12.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.386+0000 7fe445f16640 1 -- 172.21.15.1:0/1372324548 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe42c06b4d0 msgr2=0x7fe42c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.386+0000 7fe445f16640 1 --2- 172.21.15.1:0/1372324548 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe42c06b4d0 0x7fe42c06d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fe430009c40 tx=0x7fe430009190 comp rx=0 tx=0).stop 2024-03-06T07:26:12.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.386+0000 7fe445f16640 1 -- 172.21.15.1:0/1372324548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4400ad880 msgr2=0x7fe44006d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.386+0000 7fe445f16640 1 --2- 172.21.15.1:0/1372324548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4400ad880 0x7fe44006d4f0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fe434000c00 tx=0x7fe434007660 comp rx=0 tx=0).stop 2024-03-06T07:26:12.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.386+0000 7fe445f16640 1 -- 172.21.15.1:0/1372324548 shutdown_connections 2024-03-06T07:26:12.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.386+0000 7fe445f16640 1 --2- 172.21.15.1:0/1372324548 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe42c06b4d0 0x7fe42c06d990 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.386+0000 7fe445f16640 1 --2- 172.21.15.1:0/1372324548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4401e2310 0x7fe4401ee440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.386+0000 7fe445f16640 1 --2- 172.21.15.1:0/1372324548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4400ad880 0x7fe44006d4f0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.386+0000 7fe445f16640 1 --2- 172.21.15.1:0/1372324548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe428004d40 0x7fe4400ac740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.386+0000 7fe445f16640 1 -- 172.21.15.1:0/1372324548 >> 172.21.15.1:0/1372324548 conn(0x7fe44006b410 msgr2=0x7fe44006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.386+0000 7fe445f16640 1 -- 172.21.15.1:0/1372324548 shutdown_connections 2024-03-06T07:26:12.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.386+0000 7fe445f16640 1 -- 172.21.15.1:0/1372324548 wait complete. 2024-03-06T07:26:12.399 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:26:12.399 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:26:12.400 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:26:12.400 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:12.400 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:26:12.400 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:26:12.400 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:26:12.400 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:12.400 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:26:12.400 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:26:12.401 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:26:12.401 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:12.401 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:26:12.401 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:26:12.401 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:26:12.401 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:12.409 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:26:12.409 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:26:12.410 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:26:12.410 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:26:12.411 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:26:12.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:26:12.412 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:26:12.412 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:26:12.451 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:26:12.451 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:12.452 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:26:12.452 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:12.454 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:26:12.454 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:12.455 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:26:12.455 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:12.455 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:26:12.527 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:26:12.528 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:26:12.536 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:26:12.537 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:26:12.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.560+0000 7f5f42247640 1 Processor -- start 2024-03-06T07:26:12.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.560+0000 7f5f42247640 1 -- start start 2024-03-06T07:26:12.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.561+0000 7f5f42247640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5f3c0c32f0 0x7f5f3c1543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.561+0000 7f5f42247640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3c150900 con 0x7f5f3c150520 2024-03-06T07:26:12.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.561+0000 7f5f42247640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3c071790 con 0x7f5f3c070c80 2024-03-06T07:26:12.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.561+0000 7f5f42247640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f3c153ed0 con 0x7f5f3c0c32f0 2024-03-06T07:26:12.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.561+0000 7f5f3b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5f3c150520 0x7f5f3c1537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36904/0 (socket says 172.21.15.1:36904) 2024-03-06T07:26:12.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.561+0000 7f5f3b7fe640 1 -- 172.21.15.1:0/908994319 learned_addr learned my addr 172.21.15.1:0/908994319 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:12.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.561+0000 7f5f3a7fc640 1 -- 172.21.15.1:0/908994319 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935204298 0 0) 0x7f5f3c150900 con 0x7f5f3c150520 2024-03-06T07:26:12.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.561+0000 7f5f3affd640 1 --2- 172.21.15.1:0/908994319 >> v2:172.21.15.98:3300/0 conn(0x7f5f3c0c32f0 0x7f5f3c1543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.561+0000 7f5f3a7fc640 1 -- 172.21.15.1:0/908994319 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f28003520 con 0x7f5f3c150520 2024-03-06T07:26:12.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3a7fc640 1 -- 172.21.15.1:0/908994319 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3161031089 0 0) 0x7f5f28003520 con 0x7f5f3c150520 2024-03-06T07:26:12.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3a7fc640 1 -- 172.21.15.1:0/908994319 >> v2:172.21.15.98:3300/0 conn(0x7f5f3c0c32f0 msgr2=0x7f5f3c1543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3a7fc640 1 --2- 172.21.15.1:0/908994319 >> v2:172.21.15.98:3300/0 conn(0x7f5f3c0c32f0 0x7f5f3c1543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3a7fc640 1 -- 172.21.15.1:0/908994319 >> v1:172.21.15.98:6789/0 conn(0x7f5f3c070c80 legacy=0x7f5f3c15d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3a7fc640 1 -- 172.21.15.1:0/908994319 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3c157050 con 0x7f5f3c150520 2024-03-06T07:26:12.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3a7fc640 1 -- 172.21.15.1:0/908994319 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f5f2c001670 con 0x7f5f3c150520 2024-03-06T07:26:12.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3a7fc640 1 -- 172.21.15.1:0/908994319 >> v1:172.21.15.1:6789/0 conn(0x7f5f3c150520 legacy=0x7f5f3c1537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3a7fc640 1 --2- 172.21.15.1:0/908994319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f28003a00 0x7f5f28003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3a7fc640 1 --2- 172.21.15.1:0/908994319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f28004800 0x7f5f28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3a7fc640 1 --2- 172.21.15.1:0/908994319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f280051c0 0x7f5f2800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3a7fc640 1 -- 172.21.15.1:0/908994319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c153ed0 con 0x7f5f280051c0 2024-03-06T07:26:12.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3a7fc640 1 -- 172.21.15.1:0/908994319 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2800c0c0 con 0x7f5f28004800 2024-03-06T07:26:12.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3a7fc640 1 -- 172.21.15.1:0/908994319 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2800c2c0 con 0x7f5f28003a00 2024-03-06T07:26:12.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3bfff640 1 --2- 172.21.15.1:0/908994319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f280051c0 0x7f5f2800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3affd640 1 --2- 172.21.15.1:0/908994319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f28004800 0x7f5f28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3b7fe640 1 --2- 172.21.15.1:0/908994319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f28003a00 0x7f5f28003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3bfff640 1 -- 172.21.15.1:0/908994319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f28003a00 msgr2=0x7f5f28003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3bfff640 1 --2- 172.21.15.1:0/908994319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f28003a00 0x7f5f28003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3bfff640 1 -- 172.21.15.1:0/908994319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f28004800 msgr2=0x7f5f28004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3bfff640 1 --2- 172.21.15.1:0/908994319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f28004800 0x7f5f28004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.562+0000 7f5f3bfff640 1 -- 172.21.15.1:0/908994319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f28003520 con 0x7f5f280051c0 2024-03-06T07:26:12.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.563+0000 7f5f3bfff640 1 -- 172.21.15.1:0/908994319 reap_dead start 2024-03-06T07:26:12.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.563+0000 7f5f3bfff640 1 --2- 172.21.15.1:0/908994319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f280051c0 0x7f5f2800b9c0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f5f34001e50 tx=0x7f5f28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=526687e4c92ed251 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.563+0000 7f5f3a7fc640 1 -- 172.21.15.1:0/908994319 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f34001a70 con 0x7f5f280051c0 2024-03-06T07:26:12.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.563+0000 7f5f3a7fc640 1 -- 172.21.15.1:0/908994319 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f34003bb0 con 0x7f5f280051c0 2024-03-06T07:26:12.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.563+0000 7f5f3a7fc640 1 -- 172.21.15.1:0/908994319 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f340050d0 con 0x7f5f280051c0 2024-03-06T07:26:12.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.563+0000 7f5f42247640 1 -- 172.21.15.1:0/908994319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f280051c0 msgr2=0x7f5f2800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.563+0000 7f5f42247640 1 --2- 172.21.15.1:0/908994319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f280051c0 0x7f5f2800b9c0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f5f34001e50 tx=0x7f5f28005830 comp rx=0 tx=0).stop 2024-03-06T07:26:12.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.564+0000 7f5f42247640 1 -- 172.21.15.1:0/908994319 shutdown_connections 2024-03-06T07:26:12.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.564+0000 7f5f42247640 1 --2- 172.21.15.1:0/908994319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f280051c0 0x7f5f2800b9c0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.564+0000 7f5f42247640 1 -- 172.21.15.1:0/908994319 >> 172.21.15.1:0/908994319 conn(0x7f5f3c0c88f0 msgr2=0x7f5f3c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.564+0000 7f5f42247640 1 -- 172.21.15.1:0/908994319 shutdown_connections 2024-03-06T07:26:12.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.564+0000 7f5f42247640 1 -- 172.21.15.1:0/908994319 wait complete. 2024-03-06T07:26:12.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.565+0000 7f5f42247640 1 Processor -- start 2024-03-06T07:26:12.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.565+0000 7f5f42247640 1 -- start start 2024-03-06T07:26:12.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f42247640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f280051c0 0x7f5f3c14f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f42247640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3c1ec750 0x7f5f3c147660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f42247640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f3c147ba0 0x7f5f3c145800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f42247640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c157330 con 0x7f5f280051c0 2024-03-06T07:26:12.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f42247640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c1532a0 con 0x7f5f3c1ec750 2024-03-06T07:26:12.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f42247640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c0c7ec0 con 0x7f5f3c147ba0 2024-03-06T07:26:12.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f3b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f280051c0 0x7f5f3c14f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f3b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f280051c0 0x7f5f3c14f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33464/0 (socket says 172.21.15.1:33464) 2024-03-06T07:26:12.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f3b7fe640 1 -- 172.21.15.1:0/1280405947 learned_addr learned my addr 172.21.15.1:0/1280405947 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:12.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f3b7fe640 1 -- 172.21.15.1:0/1280405947 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f3c147ba0 msgr2=0x7f5f3c145800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f3b7fe640 1 --2- 172.21.15.1:0/1280405947 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f3c147ba0 0x7f5f3c145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f3b7fe640 1 -- 172.21.15.1:0/1280405947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3c1ec750 msgr2=0x7f5f3c147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f3b7fe640 1 --2- 172.21.15.1:0/1280405947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3c1ec750 0x7f5f3c147660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f3b7fe640 1 -- 172.21.15.1:0/1280405947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3c1dc5b0 con 0x7f5f280051c0 2024-03-06T07:26:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f3b7fe640 1 --2- 172.21.15.1:0/1280405947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f280051c0 0x7f5f3c14f000 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c007290 tx=0x7f5f2c007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f33fff640 1 -- 172.21.15.1:0/1280405947 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c01a070 con 0x7f5f280051c0 2024-03-06T07:26:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f33fff640 1 -- 172.21.15.1:0/1280405947 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2c00f940 con 0x7f5f280051c0 2024-03-06T07:26:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f33fff640 1 -- 172.21.15.1:0/1280405947 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c00fd10 con 0x7f5f280051c0 2024-03-06T07:26:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f42247640 1 -- 172.21.15.1:0/1280405947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f3c1dc820 con 0x7f5f280051c0 2024-03-06T07:26:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.566+0000 7f5f42247640 1 -- 172.21.15.1:0/1280405947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f3c14f540 con 0x7f5f280051c0 2024-03-06T07:26:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.567+0000 7f5f42247640 1 -- 172.21.15.1:0/1280405947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3c0686e0 con 0x7f5f280051c0 2024-03-06T07:26:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.568+0000 7f5f33fff640 1 -- 172.21.15.1:0/1280405947 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f2c01d070 con 0x7f5f280051c0 2024-03-06T07:26:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.568+0000 7f5f33fff640 1 --2- 172.21.15.1:0/1280405947 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5f1c06b4d0 0x7f5f1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.568+0000 7f5f33fff640 1 -- 172.21.15.1:0/1280405947 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f5f2c089ac0 con 0x7f5f280051c0 2024-03-06T07:26:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.569+0000 7f5f3affd640 1 --2- 172.21.15.1:0/1280405947 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5f1c06b4d0 0x7f5f1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.569+0000 7f5f3affd640 1 --2- 172.21.15.1:0/1280405947 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5f1c06b4d0 0x7f5f1c06d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c152440 tx=0x7f5f24009320 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.572+0000 7f5f33fff640 1 -- 172.21.15.1:0/1280405947 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f2c058a90 con 0x7f5f280051c0 2024-03-06T07:26:12.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:12.768+0000 7f5f42247640 1 -- 172.21.15.1:0/1280405947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5f3c0c7b10 con 0x7f5f280051c0 2024-03-06T07:26:13.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.593+0000 7f5f33fff640 1 -- 172.21.15.1:0/1280405947 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v145) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5f2c058390 con 0x7f5f280051c0 2024-03-06T07:26:13.595 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-06T07:26:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.596+0000 7f5f42247640 1 -- 172.21.15.1:0/1280405947 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5f1c06b4d0 msgr2=0x7f5f1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.596+0000 7f5f42247640 1 --2- 172.21.15.1:0/1280405947 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5f1c06b4d0 0x7f5f1c06d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c152440 tx=0x7f5f24009320 comp rx=0 tx=0).stop 2024-03-06T07:26:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.596+0000 7f5f42247640 1 -- 172.21.15.1:0/1280405947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f280051c0 msgr2=0x7f5f3c14f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.596+0000 7f5f42247640 1 --2- 172.21.15.1:0/1280405947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f280051c0 0x7f5f3c14f000 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c007290 tx=0x7f5f2c007750 comp rx=0 tx=0).stop 2024-03-06T07:26:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.596+0000 7f5f42247640 1 -- 172.21.15.1:0/1280405947 shutdown_connections 2024-03-06T07:26:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.596+0000 7f5f42247640 1 --2- 172.21.15.1:0/1280405947 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5f1c06b4d0 0x7f5f1c06d990 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.596+0000 7f5f42247640 1 --2- 172.21.15.1:0/1280405947 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f3c147ba0 0x7f5f3c145800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.596+0000 7f5f42247640 1 --2- 172.21.15.1:0/1280405947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3c1ec750 0x7f5f3c147660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.596+0000 7f5f42247640 1 --2- 172.21.15.1:0/1280405947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f280051c0 0x7f5f3c14f000 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.596+0000 7f5f42247640 1 -- 172.21.15.1:0/1280405947 >> 172.21.15.1:0/1280405947 conn(0x7f5f3c0c88f0 msgr2=0x7f5f3c1525a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.596+0000 7f5f42247640 1 -- 172.21.15.1:0/1280405947 shutdown_connections 2024-03-06T07:26:13.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.596+0000 7f5f42247640 1 -- 172.21.15.1:0/1280405947 wait complete. 2024-03-06T07:26:13.612 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:26:13.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.717+0000 7f145835e640 1 Processor -- start 2024-03-06T07:26:13.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.717+0000 7f145835e640 1 -- start start 2024-03-06T07:26:13.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.718+0000 7f145835e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14500c6730 0x7f14500c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.718+0000 7f145835e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f14500c5d10 0x7f1450156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.718+0000 7f145835e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14500c3300 con 0x7f1450152690 2024-03-06T07:26:13.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.718+0000 7f145835e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14500c35a0 con 0x7f14500c5d10 2024-03-06T07:26:13.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.718+0000 7f145835e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14500c3880 con 0x7f14500c6730 2024-03-06T07:26:13.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.718+0000 7f14568fb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f14500c5d10 0x7f1450156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.718+0000 7f14568fb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f14500c5d10 0x7f1450156cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33478/0 (socket says 172.21.15.1:33478) 2024-03-06T07:26:13.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.718+0000 7f14568fb640 1 -- 172.21.15.1:0/978535894 learned_addr learned my addr 172.21.15.1:0/978535894 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:13.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.718+0000 7f14558f9640 1 --2- 172.21.15.1:0/978535894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14500c6730 0x7f14500c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.718+0000 7f14568fb640 1 -- 172.21.15.1:0/978535894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14500c6730 msgr2=0x7f14500c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.718+0000 7f14568fb640 1 --2- 172.21.15.1:0/978535894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14500c6730 0x7f14500c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.718+0000 7f14568fb640 1 -- 172.21.15.1:0/978535894 >> v1:172.21.15.98:6789/0 conn(0x7f1450152690 legacy=0x7f1450155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.718+0000 7f14568fb640 1 -- 172.21.15.1:0/978535894 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14500c3b60 con 0x7f14500c5d10 2024-03-06T07:26:13.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.718+0000 7f14558f9640 1 --2- 172.21.15.1:0/978535894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14500c6730 0x7f14500c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:13.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.719+0000 7f14568fb640 1 --2- 172.21.15.1:0/978535894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14500c5d10 0x7f1450156cd0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f1440009e50 tx=0x7f144002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a4469c059ee90e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.719+0000 7f14550f8640 1 -- 172.21.15.1:0/978535894 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144002c9e0 con 0x7f14500c5d10 2024-03-06T07:26:13.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.719+0000 7f14550f8640 1 -- 172.21.15.1:0/978535894 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144002cb80 con 0x7f14500c5d10 2024-03-06T07:26:13.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.719+0000 7f14550f8640 1 -- 172.21.15.1:0/978535894 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14400343f0 con 0x7f14500c5d10 2024-03-06T07:26:13.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.720+0000 7f145835e640 1 -- 172.21.15.1:0/978535894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14500c5d10 msgr2=0x7f1450156cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.720+0000 7f145835e640 1 --2- 172.21.15.1:0/978535894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14500c5d10 0x7f1450156cd0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f1440009e50 tx=0x7f144002be40 comp rx=0 tx=0).stop 2024-03-06T07:26:13.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.720+0000 7f145835e640 1 -- 172.21.15.1:0/978535894 shutdown_connections 2024-03-06T07:26:13.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.720+0000 7f145835e640 1 --2- 172.21.15.1:0/978535894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14500c5d10 0x7f1450156cd0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.720+0000 7f145835e640 1 --2- 172.21.15.1:0/978535894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14500c6730 0x7f14500c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.720+0000 7f145835e640 1 -- 172.21.15.1:0/978535894 >> 172.21.15.1:0/978535894 conn(0x7f145006b410 msgr2=0x7f145006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.720+0000 7f145835e640 1 -- 172.21.15.1:0/978535894 shutdown_connections 2024-03-06T07:26:13.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.720+0000 7f145835e640 1 -- 172.21.15.1:0/978535894 wait complete. 2024-03-06T07:26:13.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.721+0000 7f145835e640 1 Processor -- start 2024-03-06T07:26:13.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.721+0000 7f145835e640 1 -- start start 2024-03-06T07:26:13.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.722+0000 7f145835e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14500c5d10 0x7f14500ac030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.722+0000 7f145835e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14500c6730 0x7f14500c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.722+0000 7f145835e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1450152690 0x7f145014c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.722+0000 7f145835e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1450152bf0 con 0x7f1450152690 2024-03-06T07:26:13.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.722+0000 7f145835e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1450152a70 con 0x7f14500c5d10 2024-03-06T07:26:13.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.722+0000 7f145835e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1450157210 con 0x7f14500c6730 2024-03-06T07:26:13.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.722+0000 7f14568fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1450152690 0x7f145014c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.722+0000 7f14568fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1450152690 0x7f145014c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33482/0 (socket says 172.21.15.1:33482) 2024-03-06T07:26:13.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.722+0000 7f14568fb640 1 -- 172.21.15.1:0/2350609066 learned_addr learned my addr 172.21.15.1:0/2350609066 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:13.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.722+0000 7f14558f9640 1 --2- 172.21.15.1:0/2350609066 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14500c6730 0x7f14500c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.722+0000 7f14560fa640 1 --2- 172.21.15.1:0/2350609066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14500c5d10 0x7f14500ac030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.722+0000 7f14568fb640 1 -- 172.21.15.1:0/2350609066 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14500c6730 msgr2=0x7f14500c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.722+0000 7f14568fb640 1 --2- 172.21.15.1:0/2350609066 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14500c6730 0x7f14500c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.722+0000 7f14568fb640 1 -- 172.21.15.1:0/2350609066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14500c5d10 msgr2=0x7f14500ac030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.722+0000 7f14568fb640 1 --2- 172.21.15.1:0/2350609066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14500c5d10 0x7f14500ac030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.722+0000 7f14568fb640 1 -- 172.21.15.1:0/2350609066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14500ad520 con 0x7f1450152690 2024-03-06T07:26:13.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.722+0000 7f14560fa640 1 --2- 172.21.15.1:0/2350609066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14500c5d10 0x7f14500ac030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:13.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.723+0000 7f14568fb640 1 --2- 172.21.15.1:0/2350609066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1450152690 0x7f145014c9d0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f1440036040 tx=0x7f1440004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.723+0000 7f144effd640 1 -- 172.21.15.1:0/2350609066 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1440043020 con 0x7f1450152690 2024-03-06T07:26:13.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.723+0000 7f144effd640 1 -- 172.21.15.1:0/2350609066 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144003e450 con 0x7f1450152690 2024-03-06T07:26:13.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.723+0000 7f145835e640 1 -- 172.21.15.1:0/2350609066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14500ad800 con 0x7f1450152690 2024-03-06T07:26:13.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.723+0000 7f144effd640 1 -- 172.21.15.1:0/2350609066 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144003e840 con 0x7f1450152690 2024-03-06T07:26:13.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.723+0000 7f145835e640 1 -- 172.21.15.1:0/2350609066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14500a9680 con 0x7f1450152690 2024-03-06T07:26:13.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.724+0000 7f145835e640 1 -- 172.21.15.1:0/2350609066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1450157e10 con 0x7f1450152690 2024-03-06T07:26:13.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.724+0000 7f144effd640 1 -- 172.21.15.1:0/2350609066 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1440046020 con 0x7f1450152690 2024-03-06T07:26:13.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.725+0000 7f144effd640 1 --2- 172.21.15.1:0/2350609066 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f143c06b4d0 0x7f143c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.725+0000 7f144effd640 1 -- 172.21.15.1:0/2350609066 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f14400ac9c0 con 0x7f1450152690 2024-03-06T07:26:13.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.725+0000 7f14560fa640 1 --2- 172.21.15.1:0/2350609066 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f143c06b4d0 0x7f143c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.725+0000 7f14560fa640 1 --2- 172.21.15.1:0/2350609066 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f143c06b4d0 0x7f143c06d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f1448003eb0 tx=0x7f1448009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.729+0000 7f144effd640 1 -- 172.21.15.1:0/2350609066 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1450157e10 con 0x7f1450152690 2024-03-06T07:26:13.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.977+0000 7f145835e640 1 -- 172.21.15.1:0/2350609066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f1450067fd0 con 0x7f1450152690 2024-03-06T07:26:13.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.977+0000 7f144effd640 1 -- 172.21.15.1:0/2350609066 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f14400aee20 con 0x7f1450152690 2024-03-06T07:26:13.979 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:13.979 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:26:13.979 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-06T07:26:13.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:13.979 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-06T07:26:13.980 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlnxyWARAAKIPXU0IafNQ1RzBPX+3DYw==", 2024-03-06T07:26:13.980 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:26:13.980 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:26:13.980 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:26:13.980 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:26:13.980 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:26:13.980 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:13.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:13.980 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-06T07:26:13.980 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl2fXxDRAAuLqH+HpKcN1JTts9ZtDTXA==", 2024-03-06T07:26:13.980 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:26:13.980 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:26:13.980 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:26:13.981 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:26:13.981 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:26:13.981 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:13.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:13.981 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-06T07:26:13.981 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl+IzqCBAAmgLkarFGt9Ihs/8kJYicBQ==", 2024-03-06T07:26:13.981 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:26:13.981 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:26:13.981 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:26:13.981 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:26:13.981 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:26:13.981 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:13.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:13.981 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-06T07:26:13.982 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlipdRFRAAA34vrsTEoX3L1CKb1YAV9g==", 2024-03-06T07:26:13.982 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:26:13.982 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:26:13.982 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:26:13.982 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:26:13.982 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:26:13.982 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:13.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:13.982 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-06T07:26:13.982 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAGGehlQK39LxAAOpPIZvl3FqQwCgyo3yoOGQ==", 2024-03-06T07:26:13.982 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:26:13.982 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:13.982 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:26:13.982 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:26:13.983 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:13.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:13.983 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-06T07:26:13.983 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAJGehlTxwDLhAAMwN9E0+3Jh5XNXnLo+fw3g==", 2024-03-06T07:26:13.983 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:26:13.983 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:13.983 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:26:13.983 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:26:13.983 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:13.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:13.983 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-06T07:26:13.983 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAMGehl4qrlLBAAm0VWQRNvSQIpDx5xf4Fyig==", 2024-03-06T07:26:13.983 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:26:13.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:13.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:26:13.984 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:26:13.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:13.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:13.984 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-06T07:26:13.984 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAPGehllO5HKxAAozm6ycG+fRxatoZTogeJTA==", 2024-03-06T07:26:13.984 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:26:13.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:13.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:26:13.984 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:26:13.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:13.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:13.985 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-06T07:26:13.985 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAVGehlLtLBMhAAUrV6q6ZCQluIcoo/S/NdbA==", 2024-03-06T07:26:13.985 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:26:13.985 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:13.985 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:26:13.985 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:26:13.985 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:13.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:13.986 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-06T07:26:13.986 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAYGehlHp59MRAAxxgpSPTEZkYKjl3fr0xeeQ==", 2024-03-06T07:26:13.986 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:26:13.986 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:13.986 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:26:13.986 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:26:13.986 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:13.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:13.986 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-06T07:26:13.986 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAbGehlTnXcMBAAFOBqcnh5ubAz+o6AGjccWQ==", 2024-03-06T07:26:13.986 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:26:13.986 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:13.986 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:26:13.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:26:13.987 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:13.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:13.987 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-06T07:26:13.987 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAeGehl1qH1LhAAaSl7Aj7qlN6d4xt5SKAmcg==", 2024-03-06T07:26:13.987 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:26:13.987 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:13.987 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:26:13.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:26:13.987 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:13.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:13.987 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-06T07:26:13.987 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng==", 2024-03-06T07:26:13.988 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:26:13.988 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:26:13.988 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-06T07:26:13.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:13.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:13.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:13.988 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-06T07:26:13.988 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlMyv7HBAAi7vt7aZWEaZuMpFrB5ofZw==", 2024-03-06T07:26:13.988 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:26:13.988 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:26:13.988 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-06T07:26:13.988 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-06T07:26:13.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-06T07:26:13.989 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:13.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:13.989 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-06T07:26:13.989 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlBS/vIBAAN86N2K4ze5PJB8kR869Lfg==", 2024-03-06T07:26:13.989 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:26:13.989 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:26:13.989 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:26:13.989 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:26:13.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:26:13.989 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:13.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:13.989 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-06T07:26:13.989 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlJ/jPORAAhLk5MTGafsGKhZSYhvhBZg==", 2024-03-06T07:26:13.990 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:26:13.990 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:26:13.990 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:13.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:26:13.990 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:13.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:13.990 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-06T07:26:13.990 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlAl8rNhAAvdU0Qvo/FJYnxc4/H8MIqQ==", 2024-03-06T07:26:13.990 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:26:13.990 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:26:13.990 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:13.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:26:13.990 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.990 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:13.991 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-06T07:26:13.991 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:26:13.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.979+0000 7f145835e640 1 -- 172.21.15.1:0/2350609066 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f143c06b4d0 msgr2=0x7f143c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.979+0000 7f145835e640 1 --2- 172.21.15.1:0/2350609066 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f143c06b4d0 0x7f143c06d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f1448003eb0 tx=0x7f1448009040 comp rx=0 tx=0).stop 2024-03-06T07:26:13.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.979+0000 7f145835e640 1 -- 172.21.15.1:0/2350609066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1450152690 msgr2=0x7f145014c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.979+0000 7f145835e640 1 --2- 172.21.15.1:0/2350609066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1450152690 0x7f145014c9d0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f1440036040 tx=0x7f1440004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:13.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.979+0000 7f145835e640 1 -- 172.21.15.1:0/2350609066 shutdown_connections 2024-03-06T07:26:13.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.979+0000 7f145835e640 1 --2- 172.21.15.1:0/2350609066 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f143c06b4d0 0x7f143c06d990 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.979+0000 7f145835e640 1 --2- 172.21.15.1:0/2350609066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1450152690 0x7f145014c9d0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.979+0000 7f145835e640 1 --2- 172.21.15.1:0/2350609066 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14500c6730 0x7f14500c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.979+0000 7f145835e640 1 --2- 172.21.15.1:0/2350609066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14500c5d10 0x7f14500ac030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.979+0000 7f145835e640 1 -- 172.21.15.1:0/2350609066 >> 172.21.15.1:0/2350609066 conn(0x7f145006b410 msgr2=0x7f1450151f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.979+0000 7f145835e640 1 -- 172.21.15.1:0/2350609066 shutdown_connections 2024-03-06T07:26:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:13.979+0000 7f145835e640 1 -- 172.21.15.1:0/2350609066 wait complete. 2024-03-06T07:26:13.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:14.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.092+0000 7f26badbd640 1 Processor -- start 2024-03-06T07:26:14.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.092+0000 7f26badbd640 1 -- start start 2024-03-06T07:26:14.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.093+0000 7f26badbd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f26b4072bd0 0x7f26b40c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.093+0000 7f26badbd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26b4152b70 0x7f26b40c3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.093+0000 7f26badbd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b4151dd0 con 0x7f26b40c2b30 2024-03-06T07:26:14.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.093+0000 7f26badbd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b41526e0 con 0x7f26b4072bd0 2024-03-06T07:26:14.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.093+0000 7f26badbd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b4156190 con 0x7f26b4152b70 2024-03-06T07:26:14.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.094+0000 7f26b3fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f26b4072bd0 0x7f26b40c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.094+0000 7f26b935a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26b4152b70 0x7f26b40c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.094+0000 7f26b8b59640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f26b40c2b30 0x7f26b40724c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:44162/0 (socket says 172.21.15.1:44162) 2024-03-06T07:26:14.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.094+0000 7f26b935a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26b4152b70 0x7f26b40c3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:45532/0 (socket says 172.21.15.1:45532) 2024-03-06T07:26:14.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.094+0000 7f26b3fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f26b4072bd0 0x7f26b40c32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:42060/0 (socket says 172.21.15.1:42060) 2024-03-06T07:26:14.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.094+0000 7f26b8b59640 1 -- 172.21.15.1:0/3136224128 learned_addr learned my addr 172.21.15.1:0/3136224128 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:14.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.094+0000 7f26b37fe640 1 -- 172.21.15.1:0/3136224128 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153976699 0 0) 0x7f26b4151dd0 con 0x7f26b40c2b30 2024-03-06T07:26:14.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.094+0000 7f26b37fe640 1 -- 172.21.15.1:0/3136224128 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26a4005730 con 0x7f26b40c2b30 2024-03-06T07:26:14.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.094+0000 7f26b935a640 1 -- 172.21.15.1:0/3136224128 >> v2:172.21.15.98:3300/0 conn(0x7f26b4072bd0 msgr2=0x7f26b40c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.094+0000 7f26b935a640 1 --2- 172.21.15.1:0/3136224128 >> v2:172.21.15.98:3300/0 conn(0x7f26b4072bd0 0x7f26b40c32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.094+0000 7f26b935a640 1 -- 172.21.15.1:0/3136224128 >> v1:172.21.15.98:6789/0 conn(0x7f26b40c2b30 legacy=0x7f26b40724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.094+0000 7f26b935a640 1 -- 172.21.15.1:0/3136224128 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b4152090 con 0x7f26b4152b70 2024-03-06T07:26:14.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.094+0000 7f26b3fff640 1 --2- 172.21.15.1:0/3136224128 >> v2:172.21.15.98:3300/0 conn(0x7f26b4072bd0 0x7f26b40c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:14.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.095+0000 7f26b935a640 1 --2- 172.21.15.1:0/3136224128 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26b4152b70 0x7f26b40c3830 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f26b4152860 tx=0x7f26ac02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdb3cf17d8f5f4b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.095+0000 7f26b37fe640 1 -- 172.21.15.1:0/3136224128 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac039070 con 0x7f26b4152b70 2024-03-06T07:26:14.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.095+0000 7f26b37fe640 1 -- 172.21.15.1:0/3136224128 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ac02cc40 con 0x7f26b4152b70 2024-03-06T07:26:14.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.095+0000 7f26b37fe640 1 -- 172.21.15.1:0/3136224128 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac034440 con 0x7f26b4152b70 2024-03-06T07:26:14.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.095+0000 7f26badbd640 1 -- 172.21.15.1:0/3136224128 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26b4152b70 msgr2=0x7f26b40c3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.095+0000 7f26badbd640 1 --2- 172.21.15.1:0/3136224128 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26b4152b70 0x7f26b40c3830 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f26b4152860 tx=0x7f26ac02be90 comp rx=0 tx=0).stop 2024-03-06T07:26:14.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.095+0000 7f26badbd640 1 -- 172.21.15.1:0/3136224128 shutdown_connections 2024-03-06T07:26:14.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.095+0000 7f26badbd640 1 --2- 172.21.15.1:0/3136224128 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26b4152b70 0x7f26b40c3830 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.095+0000 7f26badbd640 1 --2- 172.21.15.1:0/3136224128 >> v2:172.21.15.98:3300/0 conn(0x7f26b4072bd0 0x7f26b40c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.096+0000 7f26badbd640 1 -- 172.21.15.1:0/3136224128 >> 172.21.15.1:0/3136224128 conn(0x7f26b40c6860 msgr2=0x7f26b40c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.096+0000 7f26badbd640 1 -- 172.21.15.1:0/3136224128 shutdown_connections 2024-03-06T07:26:14.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.096+0000 7f26badbd640 1 -- 172.21.15.1:0/3136224128 wait complete. 2024-03-06T07:26:14.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.097+0000 7f26badbd640 1 Processor -- start 2024-03-06T07:26:14.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.097+0000 7f26badbd640 1 -- start start 2024-03-06T07:26:14.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.097+0000 7f26badbd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26b4072bd0 0x7f26b41533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.097+0000 7f26badbd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26b40c2b30 0x7f26b40bdb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.097+0000 7f26badbd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b4152b70 0x7f26b40be070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.097+0000 7f26badbd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b40c3ef0 con 0x7f26b4152b70 2024-03-06T07:26:14.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.097+0000 7f26badbd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b40c3d70 con 0x7f26b40c2b30 2024-03-06T07:26:14.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.098+0000 7f26badbd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b4155d80 con 0x7f26b4072bd0 2024-03-06T07:26:14.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.098+0000 7f26b935a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b4152b70 0x7f26b40be070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.098+0000 7f26b935a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b4152b70 0x7f26b40be070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33492/0 (socket says 172.21.15.1:33492) 2024-03-06T07:26:14.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.098+0000 7f26b935a640 1 -- 172.21.15.1:0/289486432 learned_addr learned my addr 172.21.15.1:0/289486432 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:14.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.098+0000 7f26b8b59640 1 --2- 172.21.15.1:0/289486432 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26b4072bd0 0x7f26b41533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.098+0000 7f26b3fff640 1 --2- 172.21.15.1:0/289486432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26b40c2b30 0x7f26b40bdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.098+0000 7f26b935a640 1 -- 172.21.15.1:0/289486432 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26b4072bd0 msgr2=0x7f26b41533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.098+0000 7f26b935a640 1 --2- 172.21.15.1:0/289486432 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26b4072bd0 0x7f26b41533a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.098+0000 7f26b935a640 1 -- 172.21.15.1:0/289486432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26b40c2b30 msgr2=0x7f26b40bdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.098+0000 7f26b935a640 1 --2- 172.21.15.1:0/289486432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26b40c2b30 0x7f26b40bdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.098+0000 7f26b935a640 1 -- 172.21.15.1:0/289486432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b40a71c0 con 0x7f26b4152b70 2024-03-06T07:26:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.098+0000 7f26b935a640 1 --2- 172.21.15.1:0/289486432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b4152b70 0x7f26b40be070 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f26ac009e80 tx=0x7f26ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.098+0000 7f26b17fa640 1 -- 172.21.15.1:0/289486432 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac042020 con 0x7f26b4152b70 2024-03-06T07:26:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.098+0000 7f26b17fa640 1 -- 172.21.15.1:0/289486432 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ac03a040 con 0x7f26b4152b70 2024-03-06T07:26:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.098+0000 7f26badbd640 1 -- 172.21.15.1:0/289486432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b40c0350 con 0x7f26b4152b70 2024-03-06T07:26:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.098+0000 7f26b17fa640 1 -- 172.21.15.1:0/289486432 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac039070 con 0x7f26b4152b70 2024-03-06T07:26:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.098+0000 7f26badbd640 1 -- 172.21.15.1:0/289486432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b4154710 con 0x7f26b4152b70 2024-03-06T07:26:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.099+0000 7f26badbd640 1 -- 172.21.15.1:0/289486432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b4067fd0 con 0x7f26b4152b70 2024-03-06T07:26:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.100+0000 7f26b17fa640 1 -- 172.21.15.1:0/289486432 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26ac045020 con 0x7f26b4152b70 2024-03-06T07:26:14.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.100+0000 7f26b17fa640 1 --2- 172.21.15.1:0/289486432 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f269806b4d0 0x7f269806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.100+0000 7f26b17fa640 1 -- 172.21.15.1:0/289486432 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f26ac0abd50 con 0x7f26b4152b70 2024-03-06T07:26:14.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.100+0000 7f26b8b59640 1 --2- 172.21.15.1:0/289486432 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f269806b4d0 0x7f269806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.101+0000 7f26b8b59640 1 --2- 172.21.15.1:0/289486432 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f269806b4d0 0x7f269806d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f26a8005880 tx=0x7f26a8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.104+0000 7f26b17fa640 1 -- 172.21.15.1:0/289486432 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26ac04c070 con 0x7f26b4152b70 2024-03-06T07:26:14.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.289+0000 7f26badbd640 1 -- 172.21.15.1:0/289486432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26b40bd890 con 0x7f26b4152b70 2024-03-06T07:26:14.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.289+0000 7f26b17fa640 1 -- 172.21.15.1:0/289486432 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+13054 (secure 0 0 0) 0x7f26ac07adc0 con 0x7f26b4152b70 2024-03-06T07:26:14.291 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:14.291 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":145,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:26:13.587511+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:14.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.292+0000 7f26badbd640 1 -- 172.21.15.1:0/289486432 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f269806b4d0 msgr2=0x7f269806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.292+0000 7f26badbd640 1 --2- 172.21.15.1:0/289486432 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f269806b4d0 0x7f269806d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f26a8005880 tx=0x7f26a8009040 comp rx=0 tx=0).stop 2024-03-06T07:26:14.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.292+0000 7f26badbd640 1 -- 172.21.15.1:0/289486432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b4152b70 msgr2=0x7f26b40be070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.292+0000 7f26badbd640 1 --2- 172.21.15.1:0/289486432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b4152b70 0x7f26b40be070 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f26ac009e80 tx=0x7f26ac004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:14.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.292+0000 7f26badbd640 1 -- 172.21.15.1:0/289486432 shutdown_connections 2024-03-06T07:26:14.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.292+0000 7f26badbd640 1 --2- 172.21.15.1:0/289486432 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f269806b4d0 0x7f269806d990 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.292+0000 7f26badbd640 1 --2- 172.21.15.1:0/289486432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b4152b70 0x7f26b40be070 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.292+0000 7f26badbd640 1 --2- 172.21.15.1:0/289486432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26b40c2b30 0x7f26b40bdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.292+0000 7f26badbd640 1 --2- 172.21.15.1:0/289486432 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26b4072bd0 0x7f26b41533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.292+0000 7f26badbd640 1 -- 172.21.15.1:0/289486432 >> 172.21.15.1:0/289486432 conn(0x7f26b40c6860 msgr2=0x7f26b4150610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.292+0000 7f26badbd640 1 -- 172.21.15.1:0/289486432 shutdown_connections 2024-03-06T07:26:14.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.292+0000 7f26badbd640 1 -- 172.21.15.1:0/289486432 wait complete. 2024-03-06T07:26:14.305 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:14.305 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:14.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.407+0000 7f415d9e9640 1 Processor -- start 2024-03-06T07:26:14.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.407+0000 7f415d9e9640 1 -- start start 2024-03-06T07:26:14.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.408+0000 7f415d9e9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4158152730 0x7f41580c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.408+0000 7f415d9e9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f41580c6fd0 0x7f41580c1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.408+0000 7f415d9e9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41580c1cf0 con 0x7f41580c18d0 2024-03-06T07:26:14.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.408+0000 7f415d9e9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41581588e0 con 0x7f41580c6fd0 2024-03-06T07:26:14.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.408+0000 7f415d9e9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4158158bc0 con 0x7f4158152730 2024-03-06T07:26:14.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.408+0000 7f41567fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f41580c6fd0 0x7f41580c1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.408+0000 7f41577fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f41580c18d0 0x7f41581552e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52646/0 (socket says 172.21.15.1:52646) 2024-03-06T07:26:14.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.408+0000 7f41567fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f41580c6fd0 0x7f41580c1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44480/0 (socket says 172.21.15.1:44480) 2024-03-06T07:26:14.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.408+0000 7f41577fe640 1 -- 172.21.15.1:0/1796468187 learned_addr learned my addr 172.21.15.1:0/1796468187 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:14.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.408+0000 7f4156ffd640 1 --2- 172.21.15.1:0/1796468187 >> v2:172.21.15.98:3300/0 conn(0x7f4158152730 0x7f41580c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.409+0000 7f41567fc640 1 -- 172.21.15.1:0/1796468187 >> v2:172.21.15.98:3300/0 conn(0x7f4158152730 msgr2=0x7f41580c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.409+0000 7f41567fc640 1 --2- 172.21.15.1:0/1796468187 >> v2:172.21.15.98:3300/0 conn(0x7f4158152730 0x7f41580c6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.409+0000 7f41567fc640 1 -- 172.21.15.1:0/1796468187 >> v1:172.21.15.1:6789/0 conn(0x7f41580c18d0 legacy=0x7f41581552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.409+0000 7f41567fc640 1 -- 172.21.15.1:0/1796468187 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4158158ea0 con 0x7f41580c6fd0 2024-03-06T07:26:14.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.409+0000 7f4155ffb640 1 -- 172.21.15.1:0/1796468187 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428871456 0 0) 0x7f41580c1cf0 con 0x7f41580c18d0 2024-03-06T07:26:14.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.409+0000 7f4156ffd640 1 --2- 172.21.15.1:0/1796468187 >> v2:172.21.15.98:3300/0 conn(0x7f4158152730 0x7f41580c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:14.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.409+0000 7f41567fc640 1 --2- 172.21.15.1:0/1796468187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41580c6fd0 0x7f41580c1390 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f415808c220 tx=0x7f414002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b466c4b53c608334 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.409+0000 7f4155ffb640 1 -- 172.21.15.1:0/1796468187 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4140038070 con 0x7f41580c6fd0 2024-03-06T07:26:14.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.410+0000 7f4155ffb640 1 -- 172.21.15.1:0/1796468187 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4140031710 con 0x7f41580c6fd0 2024-03-06T07:26:14.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.410+0000 7f4155ffb640 1 -- 172.21.15.1:0/1796468187 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4140031ae0 con 0x7f41580c6fd0 2024-03-06T07:26:14.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.410+0000 7f415d9e9640 1 -- 172.21.15.1:0/1796468187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41580c6fd0 msgr2=0x7f41580c1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.410+0000 7f415d9e9640 1 --2- 172.21.15.1:0/1796468187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41580c6fd0 0x7f41580c1390 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f415808c220 tx=0x7f414002b7b0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.410+0000 7f415d9e9640 1 -- 172.21.15.1:0/1796468187 shutdown_connections 2024-03-06T07:26:14.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.410+0000 7f415d9e9640 1 --2- 172.21.15.1:0/1796468187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41580c6fd0 0x7f41580c1390 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.410+0000 7f415d9e9640 1 --2- 172.21.15.1:0/1796468187 >> v2:172.21.15.98:3300/0 conn(0x7f4158152730 0x7f41580c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.410+0000 7f415d9e9640 1 -- 172.21.15.1:0/1796468187 >> 172.21.15.1:0/1796468187 conn(0x7f4158066d00 msgr2=0x7f4158067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.410+0000 7f415d9e9640 1 -- 172.21.15.1:0/1796468187 shutdown_connections 2024-03-06T07:26:14.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.410+0000 7f415d9e9640 1 -- 172.21.15.1:0/1796468187 wait complete. 2024-03-06T07:26:14.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.411+0000 7f415d9e9640 1 Processor -- start 2024-03-06T07:26:14.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.411+0000 7f415d9e9640 1 -- start start 2024-03-06T07:26:14.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.412+0000 7f415d9e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41580c18d0 0x7f41581ea430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.412+0000 7f415d9e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41580c6fd0 0x7f4158157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.412+0000 7f415d9e9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4158152730 0x7f41580af270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.412+0000 7f415d9e9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4158152b10 con 0x7f4158152730 2024-03-06T07:26:14.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.412+0000 7f415d9e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4158159460 con 0x7f41580c6fd0 2024-03-06T07:26:14.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.412+0000 7f415d9e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41581559f0 con 0x7f41580c18d0 2024-03-06T07:26:14.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.412+0000 7f41577fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4158152730 0x7f41580af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.412+0000 7f41577fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4158152730 0x7f41580af270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44484/0 (socket says 172.21.15.1:44484) 2024-03-06T07:26:14.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.412+0000 7f41577fe640 1 -- 172.21.15.1:0/3688582929 learned_addr learned my addr 172.21.15.1:0/3688582929 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:14.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.412+0000 7f41567fc640 1 --2- 172.21.15.1:0/3688582929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41580c6fd0 0x7f4158157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.412+0000 7f4156ffd640 1 --2- 172.21.15.1:0/3688582929 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41580c18d0 0x7f41581ea430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.412+0000 7f41577fe640 1 -- 172.21.15.1:0/3688582929 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41580c18d0 msgr2=0x7f41581ea430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.412+0000 7f41577fe640 1 --2- 172.21.15.1:0/3688582929 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41580c18d0 0x7f41581ea430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.412+0000 7f41577fe640 1 -- 172.21.15.1:0/3688582929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41580c6fd0 msgr2=0x7f4158157530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.412+0000 7f41577fe640 1 --2- 172.21.15.1:0/3688582929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41580c6fd0 0x7f4158157530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.412+0000 7f41577fe640 1 -- 172.21.15.1:0/3688582929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41581eae20 con 0x7f4158152730 2024-03-06T07:26:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.413+0000 7f41577fe640 1 --2- 172.21.15.1:0/3688582929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4158152730 0x7f41580af270 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f414800c800 tx=0x7f414800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.413+0000 7f414ffff640 1 -- 172.21.15.1:0/3688582929 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4148018070 con 0x7f4158152730 2024-03-06T07:26:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.413+0000 7f414ffff640 1 -- 172.21.15.1:0/3688582929 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4148004d80 con 0x7f4158152730 2024-03-06T07:26:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.413+0000 7f415d9e9640 1 -- 172.21.15.1:0/3688582929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41580b03b0 con 0x7f4158152730 2024-03-06T07:26:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.413+0000 7f414ffff640 1 -- 172.21.15.1:0/3688582929 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414800a880 con 0x7f4158152730 2024-03-06T07:26:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.413+0000 7f415d9e9640 1 -- 172.21.15.1:0/3688582929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41580b0970 con 0x7f4158152730 2024-03-06T07:26:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.414+0000 7f415d9e9640 1 -- 172.21.15.1:0/3688582929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4158067fd0 con 0x7f4158152730 2024-03-06T07:26:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.414+0000 7f414ffff640 1 -- 172.21.15.1:0/3688582929 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f414801b070 con 0x7f4158152730 2024-03-06T07:26:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.415+0000 7f414ffff640 1 --2- 172.21.15.1:0/3688582929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f413806b4d0 0x7f413806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.415+0000 7f414ffff640 1 -- 172.21.15.1:0/3688582929 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f4148089500 con 0x7f4158152730 2024-03-06T07:26:14.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.415+0000 7f4156ffd640 1 --2- 172.21.15.1:0/3688582929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f413806b4d0 0x7f413806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.416+0000 7f4156ffd640 1 --2- 172.21.15.1:0/3688582929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f413806b4d0 0x7f413806d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4150006fd0 tx=0x7f4150008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.419+0000 7f414ffff640 1 -- 172.21.15.1:0/3688582929 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4148058670 con 0x7f4158152730 2024-03-06T07:26:14.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.599+0000 7f415d9e9640 1 -- 172.21.15.1:0/3688582929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f41580aa920 con 0x7f4158152730 2024-03-06T07:26:14.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.599+0000 7f414ffff640 1 -- 172.21.15.1:0/3688582929 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v145) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4148057f70 con 0x7f4158152730 2024-03-06T07:26:14.601 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:26:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.602+0000 7f415d9e9640 1 -- 172.21.15.1:0/3688582929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f413806b4d0 msgr2=0x7f413806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.602+0000 7f415d9e9640 1 --2- 172.21.15.1:0/3688582929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f413806b4d0 0x7f413806d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4150006fd0 tx=0x7f4150008040 comp rx=0 tx=0).stop 2024-03-06T07:26:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.602+0000 7f415d9e9640 1 -- 172.21.15.1:0/3688582929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4158152730 msgr2=0x7f41580af270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.602+0000 7f415d9e9640 1 --2- 172.21.15.1:0/3688582929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4158152730 0x7f41580af270 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f414800c800 tx=0x7f414800ccc0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.602+0000 7f415d9e9640 1 -- 172.21.15.1:0/3688582929 shutdown_connections 2024-03-06T07:26:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.602+0000 7f415d9e9640 1 --2- 172.21.15.1:0/3688582929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f413806b4d0 0x7f413806d990 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.602+0000 7f415d9e9640 1 --2- 172.21.15.1:0/3688582929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4158152730 0x7f41580af270 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.602+0000 7f415d9e9640 1 --2- 172.21.15.1:0/3688582929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41580c6fd0 0x7f4158157530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.602+0000 7f415d9e9640 1 --2- 172.21.15.1:0/3688582929 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41580c18d0 0x7f41581ea430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.602+0000 7f415d9e9640 1 -- 172.21.15.1:0/3688582929 >> 172.21.15.1:0/3688582929 conn(0x7f4158066d00 msgr2=0x7f4158151940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.602+0000 7f415d9e9640 1 -- 172.21.15.1:0/3688582929 shutdown_connections 2024-03-06T07:26:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.602+0000 7f415d9e9640 1 -- 172.21.15.1:0/3688582929 wait complete. 2024-03-06T07:26:14.615 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:26:14.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.714+0000 7f2fdf3e9640 1 Processor -- start 2024-03-06T07:26:14.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.714+0000 7f2fdf3e9640 1 -- start start 2024-03-06T07:26:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.715+0000 7f2fdf3e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd80c3970 0x7f2fd80c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.715+0000 7f2fdf3e9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2fd8152840 0x7f2fd81553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.715+0000 7f2fdf3e9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd80c5e40 con 0x7f2fd80c60b0 2024-03-06T07:26:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.715+0000 7f2fdf3e9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd80c67f0 con 0x7f2fd8152840 2024-03-06T07:26:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.715+0000 7f2fdf3e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd80c6a40 con 0x7f2fd80c3970 2024-03-06T07:26:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.715+0000 7f2fdc984640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2fd80c60b0 0x7f2fd8152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52660/0 (socket says 172.21.15.1:52660) 2024-03-06T07:26:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.715+0000 7f2fdc984640 1 -- 172.21.15.1:0/705360945 learned_addr learned my addr 172.21.15.1:0/705360945 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.715+0000 7f2fdd185640 1 --2- 172.21.15.1:0/705360945 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd80c3970 0x7f2fd80c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.715+0000 7f2fdd986640 1 --2- 172.21.15.1:0/705360945 >> v2:172.21.15.98:3300/0 conn(0x7f2fd8152840 0x7f2fd81553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.715+0000 7f2fd7fff640 1 -- 172.21.15.1:0/705360945 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750125827 0 0) 0x7f2fd80c5e40 con 0x7f2fd80c60b0 2024-03-06T07:26:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.715+0000 7f2fd7fff640 1 -- 172.21.15.1:0/705360945 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc0003520 con 0x7f2fd80c60b0 2024-03-06T07:26:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.716+0000 7f2fdd185640 1 -- 172.21.15.1:0/705360945 >> v2:172.21.15.98:3300/0 conn(0x7f2fd8152840 msgr2=0x7f2fd81553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.716+0000 7f2fdd185640 1 --2- 172.21.15.1:0/705360945 >> v2:172.21.15.98:3300/0 conn(0x7f2fd8152840 0x7f2fd81553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.716+0000 7f2fdd185640 1 -- 172.21.15.1:0/705360945 >> v1:172.21.15.1:6789/0 conn(0x7f2fd80c60b0 legacy=0x7f2fd8152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.716+0000 7f2fdd185640 1 -- 172.21.15.1:0/705360945 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd80c6d20 con 0x7f2fd80c3970 2024-03-06T07:26:14.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.716+0000 7f2fdd185640 1 --2- 172.21.15.1:0/705360945 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd80c3970 0x7f2fd80c8170 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00097a0 tx=0x7f2fd002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e95712d301d12b7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.717+0000 7f2fd7fff640 1 -- 172.21.15.1:0/705360945 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0035020 con 0x7f2fd80c3970 2024-03-06T07:26:14.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.717+0000 7f2fd7fff640 1 -- 172.21.15.1:0/705360945 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd0031830 con 0x7f2fd80c3970 2024-03-06T07:26:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.717+0000 7f2fd7fff640 1 -- 172.21.15.1:0/705360945 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0031c00 con 0x7f2fd80c3970 2024-03-06T07:26:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.717+0000 7f2fdf3e9640 1 -- 172.21.15.1:0/705360945 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd80c3970 msgr2=0x7f2fd80c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.717+0000 7f2fdf3e9640 1 --2- 172.21.15.1:0/705360945 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd80c3970 0x7f2fd80c8170 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00097a0 tx=0x7f2fd002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.717+0000 7f2fdf3e9640 1 -- 172.21.15.1:0/705360945 shutdown_connections 2024-03-06T07:26:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.717+0000 7f2fdf3e9640 1 --2- 172.21.15.1:0/705360945 >> v2:172.21.15.98:3300/0 conn(0x7f2fd8152840 0x7f2fd81553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.717+0000 7f2fdf3e9640 1 --2- 172.21.15.1:0/705360945 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd80c3970 0x7f2fd80c8170 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.717+0000 7f2fdf3e9640 1 -- 172.21.15.1:0/705360945 >> 172.21.15.1:0/705360945 conn(0x7f2fd80c88f0 msgr2=0x7f2fd80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.717+0000 7f2fdf3e9640 1 -- 172.21.15.1:0/705360945 shutdown_connections 2024-03-06T07:26:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.717+0000 7f2fdf3e9640 1 -- 172.21.15.1:0/705360945 wait complete. 2024-03-06T07:26:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.718+0000 7f2fdf3e9640 1 Processor -- start 2024-03-06T07:26:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.718+0000 7f2fdf3e9640 1 -- start start 2024-03-06T07:26:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.719+0000 7f2fdf3e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd80c3970 0x7f2fd81ea440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.719+0000 7f2fdf3e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fd80c60b0 0x7f2fd81e7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.719+0000 7f2fdf3e9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd8152840 0x7f2fd81ee360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.719+0000 7f2fdf3e9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd80c75c0 con 0x7f2fd8152840 2024-03-06T07:26:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.719+0000 7f2fdf3e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd80c72e0 con 0x7f2fd80c60b0 2024-03-06T07:26:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.719+0000 7f2fdf3e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd80c3d50 con 0x7f2fd80c3970 2024-03-06T07:26:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.719+0000 7f2fdd986640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd8152840 0x7f2fd81ee360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.719+0000 7f2fdd986640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd8152840 0x7f2fd81ee360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44496/0 (socket says 172.21.15.1:44496) 2024-03-06T07:26:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.719+0000 7f2fdd986640 1 -- 172.21.15.1:0/2813354359 learned_addr learned my addr 172.21.15.1:0/2813354359 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.719+0000 7f2fdd185640 1 --2- 172.21.15.1:0/2813354359 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd80c3970 0x7f2fd81ea440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.719+0000 7f2fdd986640 1 -- 172.21.15.1:0/2813354359 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd80c3970 msgr2=0x7f2fd81ea440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.719+0000 7f2fdd986640 1 --2- 172.21.15.1:0/2813354359 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd80c3970 0x7f2fd81ea440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.719+0000 7f2fdd986640 1 -- 172.21.15.1:0/2813354359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fd80c60b0 msgr2=0x7f2fd81e7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.719+0000 7f2fdd986640 1 --2- 172.21.15.1:0/2813354359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fd80c60b0 0x7f2fd81e7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.719+0000 7f2fdd986640 1 -- 172.21.15.1:0/2813354359 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd80ac6a0 con 0x7f2fd8152840 2024-03-06T07:26:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.720+0000 7f2fdd986640 1 --2- 172.21.15.1:0/2813354359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd8152840 0x7f2fd81ee360 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800efc0 tx=0x7f2fc800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.720+0000 7f2fd5ffb640 1 -- 172.21.15.1:0/2813354359 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8018020 con 0x7f2fd8152840 2024-03-06T07:26:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.720+0000 7f2fd5ffb640 1 -- 172.21.15.1:0/2813354359 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc8014480 con 0x7f2fd8152840 2024-03-06T07:26:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.720+0000 7f2fdf3e9640 1 -- 172.21.15.1:0/2813354359 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd81e8080 con 0x7f2fd8152840 2024-03-06T07:26:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.720+0000 7f2fd5ffb640 1 -- 172.21.15.1:0/2813354359 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8009e60 con 0x7f2fd8152840 2024-03-06T07:26:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.720+0000 7f2fdf3e9640 1 -- 172.21.15.1:0/2813354359 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd8071050 con 0x7f2fd8152840 2024-03-06T07:26:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.721+0000 7f2fdf3e9640 1 -- 172.21.15.1:0/2813354359 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd80686e0 con 0x7f2fd8152840 2024-03-06T07:26:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.721+0000 7f2fd5ffb640 1 -- 172.21.15.1:0/2813354359 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fc8005d10 con 0x7f2fd8152840 2024-03-06T07:26:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.722+0000 7f2fd5ffb640 1 --2- 172.21.15.1:0/2813354359 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fc406b4d0 0x7f2fc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.722+0000 7f2fd5ffb640 1 -- 172.21.15.1:0/2813354359 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f2fc808acb0 con 0x7f2fd8152840 2024-03-06T07:26:14.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.722+0000 7f2fdd185640 1 --2- 172.21.15.1:0/2813354359 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fc406b4d0 0x7f2fc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.722+0000 7f2fdd185640 1 --2- 172.21.15.1:0/2813354359 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fc406b4d0 0x7f2fc406d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0036000 tx=0x7f2fd0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.726+0000 7f2fd5ffb640 1 -- 172.21.15.1:0/2813354359 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fc8022030 con 0x7f2fd8152840 2024-03-06T07:26:14.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.912+0000 7f2fdf3e9640 1 -- 172.21.15.1:0/2813354359 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2fd80c21c0 con 0x7f2fd8152840 2024-03-06T07:26:14.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.913+0000 7f2fd5ffb640 1 -- 172.21.15.1:0/2813354359 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v145) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2fc8010040 con 0x7f2fd8152840 2024-03-06T07:26:14.915 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:14.915 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-06T07:26:14.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:14.915 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-06T07:26:14.915 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-06T07:26:14.915 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:14.915 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-06T07:26:14.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.915+0000 7f2fdf3e9640 1 -- 172.21.15.1:0/2813354359 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fc406b4d0 msgr2=0x7f2fc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.915+0000 7f2fdf3e9640 1 --2- 172.21.15.1:0/2813354359 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fc406b4d0 0x7f2fc406d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0036000 tx=0x7f2fd0008030 comp rx=0 tx=0).stop 2024-03-06T07:26:14.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.915+0000 7f2fdf3e9640 1 -- 172.21.15.1:0/2813354359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd8152840 msgr2=0x7f2fd81ee360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.915+0000 7f2fdf3e9640 1 --2- 172.21.15.1:0/2813354359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd8152840 0x7f2fd81ee360 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800efc0 tx=0x7f2fc800c490 comp rx=0 tx=0).stop 2024-03-06T07:26:14.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.915+0000 7f2fdf3e9640 1 -- 172.21.15.1:0/2813354359 shutdown_connections 2024-03-06T07:26:14.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.915+0000 7f2fdf3e9640 1 --2- 172.21.15.1:0/2813354359 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fc406b4d0 0x7f2fc406d990 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.915+0000 7f2fdf3e9640 1 --2- 172.21.15.1:0/2813354359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd8152840 0x7f2fd81ee360 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.915+0000 7f2fdf3e9640 1 --2- 172.21.15.1:0/2813354359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fd80c60b0 0x7f2fd81e7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.915+0000 7f2fdf3e9640 1 --2- 172.21.15.1:0/2813354359 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd80c3970 0x7f2fd81ea440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.915+0000 7f2fdf3e9640 1 -- 172.21.15.1:0/2813354359 >> 172.21.15.1:0/2813354359 conn(0x7f2fd80c88f0 msgr2=0x7f2fd80c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.915+0000 7f2fdf3e9640 1 -- 172.21.15.1:0/2813354359 shutdown_connections 2024-03-06T07:26:14.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:14.915+0000 7f2fdf3e9640 1 -- 172.21.15.1:0/2813354359 wait complete. 2024-03-06T07:26:14.928 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:26:14.928 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:26:15.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.031+0000 7f294f2af640 1 Processor -- start 2024-03-06T07:26:15.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.031+0000 7f294f2af640 1 -- start start 2024-03-06T07:26:15.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.032+0000 7f294f2af640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f294814d180 0x7f29480c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.032+0000 7f294f2af640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29480c5d90 0x7f2948156c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.032+0000 7f294f2af640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2948153f60 con 0x7f29480c5910 2024-03-06T07:26:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.032+0000 7f294f2af640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29480c80a0 con 0x7f29480c5d90 2024-03-06T07:26:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.032+0000 7f294f2af640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29481531a0 con 0x7f294814d180 2024-03-06T07:26:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.032+0000 7f294d84c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29480c5d90 0x7f2948156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.032+0000 7f294d84c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29480c5d90 0x7f2948156c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44498/0 (socket says 172.21.15.1:44498) 2024-03-06T07:26:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.032+0000 7f294d84c640 1 -- 172.21.15.1:0/399642783 learned_addr learned my addr 172.21.15.1:0/399642783 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.032+0000 7f294d04b640 1 --2- 172.21.15.1:0/399642783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f294814d180 0x7f29480c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.032+0000 7f294d84c640 1 -- 172.21.15.1:0/399642783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f294814d180 msgr2=0x7f29480c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.032+0000 7f294d84c640 1 --2- 172.21.15.1:0/399642783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f294814d180 0x7f29480c7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.032+0000 7f294d84c640 1 -- 172.21.15.1:0/399642783 >> v1:172.21.15.98:6789/0 conn(0x7f29480c5910 legacy=0x7f2948156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.032+0000 7f294d84c640 1 -- 172.21.15.1:0/399642783 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2948153480 con 0x7f29480c5d90 2024-03-06T07:26:15.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.033+0000 7f294d84c640 1 --2- 172.21.15.1:0/399642783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29480c5d90 0x7f2948156c90 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f2938009e50 tx=0x7f293802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2057db074e7a897f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.033+0000 7f2947fff640 1 -- 172.21.15.1:0/399642783 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293802c9e0 con 0x7f29480c5d90 2024-03-06T07:26:15.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.033+0000 7f2947fff640 1 -- 172.21.15.1:0/399642783 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293802cb80 con 0x7f29480c5d90 2024-03-06T07:26:15.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.033+0000 7f2947fff640 1 -- 172.21.15.1:0/399642783 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29380343f0 con 0x7f29480c5d90 2024-03-06T07:26:15.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.034+0000 7f294f2af640 1 -- 172.21.15.1:0/399642783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29480c5d90 msgr2=0x7f2948156c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.034+0000 7f294f2af640 1 --2- 172.21.15.1:0/399642783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29480c5d90 0x7f2948156c90 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f2938009e50 tx=0x7f293802be40 comp rx=0 tx=0).stop 2024-03-06T07:26:15.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.034+0000 7f294f2af640 1 -- 172.21.15.1:0/399642783 shutdown_connections 2024-03-06T07:26:15.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.034+0000 7f294f2af640 1 --2- 172.21.15.1:0/399642783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29480c5d90 0x7f2948156c90 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.034+0000 7f294f2af640 1 --2- 172.21.15.1:0/399642783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f294814d180 0x7f29480c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.034+0000 7f294f2af640 1 -- 172.21.15.1:0/399642783 >> 172.21.15.1:0/399642783 conn(0x7f294806b410 msgr2=0x7f294806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:15.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.034+0000 7f294f2af640 1 -- 172.21.15.1:0/399642783 shutdown_connections 2024-03-06T07:26:15.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.034+0000 7f294f2af640 1 -- 172.21.15.1:0/399642783 wait complete. 2024-03-06T07:26:15.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.035+0000 7f294f2af640 1 Processor -- start 2024-03-06T07:26:15.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.035+0000 7f294f2af640 1 -- start start 2024-03-06T07:26:15.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294f2af640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29480c5910 0x7f2948154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294f2af640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29480c5d90 0x7f29481ec400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294f2af640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f294814d180 0x7f29481ec940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294f2af640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29480c76e0 con 0x7f29480c5910 2024-03-06T07:26:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294f2af640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2948153a40 con 0x7f29480c5d90 2024-03-06T07:26:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294f2af640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2948071820 con 0x7f294814d180 2024-03-06T07:26:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294d04b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29480c5910 0x7f2948154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294d04b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29480c5910 0x7f2948154740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44508/0 (socket says 172.21.15.1:44508) 2024-03-06T07:26:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294d04b640 1 -- 172.21.15.1:0/3977825436 learned_addr learned my addr 172.21.15.1:0/3977825436 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294c84a640 1 --2- 172.21.15.1:0/3977825436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29480c5d90 0x7f29481ec400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294d84c640 1 --2- 172.21.15.1:0/3977825436 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f294814d180 0x7f29481ec940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294d04b640 1 -- 172.21.15.1:0/3977825436 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f294814d180 msgr2=0x7f29481ec940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294d04b640 1 --2- 172.21.15.1:0/3977825436 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f294814d180 0x7f29481ec940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294d04b640 1 -- 172.21.15.1:0/3977825436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29480c5d90 msgr2=0x7f29481ec400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294d04b640 1 --2- 172.21.15.1:0/3977825436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29480c5d90 0x7f29481ec400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294d04b640 1 -- 172.21.15.1:0/3977825436 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2948155d90 con 0x7f29480c5910 2024-03-06T07:26:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294d04b640 1 --2- 172.21.15.1:0/3977825436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29480c5910 0x7f2948154740 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f294000d5f0 tx=0x7f294000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f2945ffb640 1 -- 172.21.15.1:0/3977825436 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29400125f0 con 0x7f29480c5910 2024-03-06T07:26:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f2945ffb640 1 -- 172.21.15.1:0/3977825436 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2940019070 con 0x7f29480c5910 2024-03-06T07:26:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f294f2af640 1 -- 172.21.15.1:0/3977825436 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2948156070 con 0x7f29480c5910 2024-03-06T07:26:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.036+0000 7f2945ffb640 1 -- 172.21.15.1:0/3977825436 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29400129d0 con 0x7f29480c5910 2024-03-06T07:26:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.037+0000 7f294f2af640 1 -- 172.21.15.1:0/3977825436 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29480abbb0 con 0x7f29480c5910 2024-03-06T07:26:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.037+0000 7f294f2af640 1 -- 172.21.15.1:0/3977825436 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29480680b0 con 0x7f29480c5910 2024-03-06T07:26:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.038+0000 7f2945ffb640 1 -- 172.21.15.1:0/3977825436 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2940012b70 con 0x7f29480c5910 2024-03-06T07:26:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.038+0000 7f2945ffb640 1 --2- 172.21.15.1:0/3977825436 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f293406b4d0 0x7f293406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.038+0000 7f2945ffb640 1 -- 172.21.15.1:0/3977825436 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f294008a380 con 0x7f29480c5910 2024-03-06T07:26:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.038+0000 7f294c84a640 1 --2- 172.21.15.1:0/3977825436 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f293406b4d0 0x7f293406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.039+0000 7f294c84a640 1 --2- 172.21.15.1:0/3977825436 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f293406b4d0 0x7f293406d990 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f293c006fd0 tx=0x7f293c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.042+0000 7f2945ffb640 1 -- 172.21.15.1:0/3977825436 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2940059450 con 0x7f29480c5910 2024-03-06T07:26:15.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.225+0000 7f294f2af640 1 -- 172.21.15.1:0/3977825436 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f29480ade90 con 0x7f29480c5910 2024-03-06T07:26:15.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.598+0000 7f2945ffb640 1 -- 172.21.15.1:0/3977825436 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v146) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2940058d50 con 0x7f29480c5910 2024-03-06T07:26:15.601 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-06T07:26:15.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.603+0000 7f294f2af640 1 -- 172.21.15.1:0/3977825436 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f293406b4d0 msgr2=0x7f293406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.603+0000 7f294f2af640 1 --2- 172.21.15.1:0/3977825436 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f293406b4d0 0x7f293406d990 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f293c006fd0 tx=0x7f293c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:15.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.603+0000 7f294f2af640 1 -- 172.21.15.1:0/3977825436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29480c5910 msgr2=0x7f2948154740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.603+0000 7f294f2af640 1 --2- 172.21.15.1:0/3977825436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29480c5910 0x7f2948154740 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f294000d5f0 tx=0x7f294000dab0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.605+0000 7f294f2af640 1 -- 172.21.15.1:0/3977825436 shutdown_connections 2024-03-06T07:26:15.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.605+0000 7f294f2af640 1 --2- 172.21.15.1:0/3977825436 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f293406b4d0 0x7f293406d990 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.605+0000 7f294f2af640 1 --2- 172.21.15.1:0/3977825436 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f294814d180 0x7f29481ec940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.605+0000 7f294f2af640 1 --2- 172.21.15.1:0/3977825436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29480c5d90 0x7f29481ec400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.605+0000 7f294f2af640 1 --2- 172.21.15.1:0/3977825436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29480c5910 0x7f2948154740 secure :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f294000d5f0 tx=0x7f294000dab0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.605+0000 7f294f2af640 1 -- 172.21.15.1:0/3977825436 >> 172.21.15.1:0/3977825436 conn(0x7f294806b410 msgr2=0x7f29480a8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:15.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.605+0000 7f294f2af640 1 -- 172.21.15.1:0/3977825436 shutdown_connections 2024-03-06T07:26:15.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.606+0000 7f294f2af640 1 -- 172.21.15.1:0/3977825436 wait complete. 2024-03-06T07:26:15.620 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:26:15.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.727+0000 7f890e17a640 1 Processor -- start 2024-03-06T07:26:15.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.727+0000 7f890e17a640 1 -- start start 2024-03-06T07:26:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.728+0000 7f890e17a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89080c2b30 0x7f89080724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.728+0000 7f890e17a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f89080a5fa0 0x7f8908072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.728+0000 7f890e17a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8908151dd0 con 0x7f8908152b70 2024-03-06T07:26:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.728+0000 7f890e17a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89081526e0 con 0x7f89080a5fa0 2024-03-06T07:26:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.728+0000 7f890e17a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8908156190 con 0x7f89080c2b30 2024-03-06T07:26:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.728+0000 7f8906ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8908152b70 0x7f89080c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52664/0 (socket says 172.21.15.1:52664) 2024-03-06T07:26:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.728+0000 7f8906ffd640 1 -- 172.21.15.1:0/669265688 learned_addr learned my addr 172.21.15.1:0/669265688 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.728+0000 7f8907fff640 1 --2- 172.21.15.1:0/669265688 >> v2:172.21.15.1:3300/0 conn(0x7f89080a5fa0 0x7f8908072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.728+0000 7f89077fe640 1 --2- 172.21.15.1:0/669265688 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89080c2b30 0x7f89080724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.728+0000 7f89067fc640 1 -- 172.21.15.1:0/669265688 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272756423 0 0) 0x7f8908151dd0 con 0x7f8908152b70 2024-03-06T07:26:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.728+0000 7f89067fc640 1 -- 172.21.15.1:0/669265688 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88f0003520 con 0x7f8908152b70 2024-03-06T07:26:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.728+0000 7f8907fff640 1 -- 172.21.15.1:0/669265688 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89080c2b30 msgr2=0x7f89080724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.728+0000 7f8907fff640 1 --2- 172.21.15.1:0/669265688 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89080c2b30 0x7f89080724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.728+0000 7f8907fff640 1 -- 172.21.15.1:0/669265688 >> v1:172.21.15.1:6789/0 conn(0x7f8908152b70 legacy=0x7f89080c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.728+0000 7f8907fff640 1 -- 172.21.15.1:0/669265688 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8908152090 con 0x7f89080a5fa0 2024-03-06T07:26:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.729+0000 7f8907fff640 1 --2- 172.21.15.1:0/669265688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080a5fa0 0x7f8908072a00 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f8900009d90 tx=0x7f890002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4a1de77abd7d75 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.729+0000 7f89067fc640 1 -- 172.21.15.1:0/669265688 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8900039070 con 0x7f89080a5fa0 2024-03-06T07:26:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.729+0000 7f89067fc640 1 -- 172.21.15.1:0/669265688 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890002cc50 con 0x7f89080a5fa0 2024-03-06T07:26:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.729+0000 7f89067fc640 1 -- 172.21.15.1:0/669265688 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8900034440 con 0x7f89080a5fa0 2024-03-06T07:26:15.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.730+0000 7f890e17a640 1 -- 172.21.15.1:0/669265688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080a5fa0 msgr2=0x7f8908072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.730+0000 7f890e17a640 1 --2- 172.21.15.1:0/669265688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080a5fa0 0x7f8908072a00 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f8900009d90 tx=0x7f890002bea0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.730+0000 7f890e17a640 1 -- 172.21.15.1:0/669265688 shutdown_connections 2024-03-06T07:26:15.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.730+0000 7f890e17a640 1 --2- 172.21.15.1:0/669265688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080a5fa0 0x7f8908072a00 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.730+0000 7f890e17a640 1 --2- 172.21.15.1:0/669265688 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89080c2b30 0x7f89080724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.730+0000 7f890e17a640 1 -- 172.21.15.1:0/669265688 >> 172.21.15.1:0/669265688 conn(0x7f89080c6860 msgr2=0x7f89080c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:15.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.730+0000 7f890e17a640 1 -- 172.21.15.1:0/669265688 shutdown_connections 2024-03-06T07:26:15.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.730+0000 7f890e17a640 1 -- 172.21.15.1:0/669265688 wait complete. 2024-03-06T07:26:15.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.731+0000 7f890e17a640 1 Processor -- start 2024-03-06T07:26:15.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.731+0000 7f890e17a640 1 -- start start 2024-03-06T07:26:15.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f890e17a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89080a5fa0 0x7f89081e7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f890e17a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080c2b30 0x7f8908150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f890e17a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8908152b70 0x7f89080bf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f890e17a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8908155f00 con 0x7f89080c2b30 2024-03-06T07:26:15.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f890e17a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8908155d80 con 0x7f8908152b70 2024-03-06T07:26:15.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f890e17a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89080c3a00 con 0x7f89080a5fa0 2024-03-06T07:26:15.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f8906ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080c2b30 0x7f8908150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f8906ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080c2b30 0x7f8908150310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44532/0 (socket says 172.21.15.1:44532) 2024-03-06T07:26:15.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f8906ffd640 1 -- 172.21.15.1:0/3789322277 learned_addr learned my addr 172.21.15.1:0/3789322277 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:15.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f89077fe640 1 --2- 172.21.15.1:0/3789322277 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89080a5fa0 0x7f89081e7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f8907fff640 1 --2- 172.21.15.1:0/3789322277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8908152b70 0x7f89080bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f8906ffd640 1 -- 172.21.15.1:0/3789322277 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89080a5fa0 msgr2=0x7f89081e7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f8906ffd640 1 --2- 172.21.15.1:0/3789322277 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89080a5fa0 0x7f89081e7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f8906ffd640 1 -- 172.21.15.1:0/3789322277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8908152b70 msgr2=0x7f89080bf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f8906ffd640 1 --2- 172.21.15.1:0/3789322277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8908152b70 0x7f89080bf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f8906ffd640 1 -- 172.21.15.1:0/3789322277 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890814b510 con 0x7f89080c2b30 2024-03-06T07:26:15.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f8906ffd640 1 --2- 172.21.15.1:0/3789322277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080c2b30 0x7f8908150310 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f88fc00c390 tx=0x7f88fc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f88effff640 1 -- 172.21.15.1:0/3789322277 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc017070 con 0x7f89080c2b30 2024-03-06T07:26:15.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f88effff640 1 -- 172.21.15.1:0/3789322277 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88fc005ce0 con 0x7f89080c2b30 2024-03-06T07:26:15.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f890e17a640 1 -- 172.21.15.1:0/3789322277 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89081e8590 con 0x7f89080c2b30 2024-03-06T07:26:15.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f88effff640 1 -- 172.21.15.1:0/3789322277 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc0072f0 con 0x7f89080c2b30 2024-03-06T07:26:15.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.732+0000 7f890e17a640 1 -- 172.21.15.1:0/3789322277 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89081e8b50 con 0x7f89080c2b30 2024-03-06T07:26:15.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.734+0000 7f890e17a640 1 -- 172.21.15.1:0/3789322277 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8908067fd0 con 0x7f89080c2b30 2024-03-06T07:26:15.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.734+0000 7f88effff640 1 -- 172.21.15.1:0/3789322277 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88fc01a070 con 0x7f89080c2b30 2024-03-06T07:26:15.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.734+0000 7f88effff640 1 --2- 172.21.15.1:0/3789322277 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88f406b4d0 0x7f88f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.734+0000 7f88effff640 1 -- 172.21.15.1:0/3789322277 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f88fc0895d0 con 0x7f89080c2b30 2024-03-06T07:26:15.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.735+0000 7f89077fe640 1 --2- 172.21.15.1:0/3789322277 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88f406b4d0 0x7f88f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.735+0000 7f89077fe640 1 --2- 172.21.15.1:0/3789322277 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88f406b4d0 0x7f88f406d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f88f8006fd0 tx=0x7f88f8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.739+0000 7f88effff640 1 -- 172.21.15.1:0/3789322277 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f890814b510 con 0x7f89080c2b30 2024-03-06T07:26:15.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:15.922+0000 7f890e17a640 1 -- 172.21.15.1:0/3789322277 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f89080c08d0 con 0x7f89080c2b30 2024-03-06T07:26:16.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.601+0000 7f88effff640 1 -- 172.21.15.1:0/3789322277 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v147) v1 ==== 170+0+0 (secure 0 0 0) 0x7f88fc0584a0 con 0x7f89080c2b30 2024-03-06T07:26:16.603 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-06T07:26:16.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.604+0000 7f890e17a640 1 -- 172.21.15.1:0/3789322277 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88f406b4d0 msgr2=0x7f88f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.604+0000 7f890e17a640 1 --2- 172.21.15.1:0/3789322277 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88f406b4d0 0x7f88f406d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f88f8006fd0 tx=0x7f88f8008040 comp rx=0 tx=0).stop 2024-03-06T07:26:16.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.604+0000 7f890e17a640 1 -- 172.21.15.1:0/3789322277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080c2b30 msgr2=0x7f8908150310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.604+0000 7f890e17a640 1 --2- 172.21.15.1:0/3789322277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080c2b30 0x7f8908150310 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f88fc00c390 tx=0x7f88fc00c850 comp rx=0 tx=0).stop 2024-03-06T07:26:16.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.604+0000 7f890e17a640 1 -- 172.21.15.1:0/3789322277 shutdown_connections 2024-03-06T07:26:16.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.604+0000 7f890e17a640 1 --2- 172.21.15.1:0/3789322277 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88f406b4d0 0x7f88f406d990 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.604+0000 7f890e17a640 1 --2- 172.21.15.1:0/3789322277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8908152b70 0x7f89080bf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.604+0000 7f890e17a640 1 --2- 172.21.15.1:0/3789322277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080c2b30 0x7f8908150310 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.604+0000 7f890e17a640 1 --2- 172.21.15.1:0/3789322277 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89080a5fa0 0x7f89081e7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.604+0000 7f890e17a640 1 -- 172.21.15.1:0/3789322277 >> 172.21.15.1:0/3789322277 conn(0x7f89080c6860 msgr2=0x7f8908072410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.604+0000 7f890e17a640 1 -- 172.21.15.1:0/3789322277 shutdown_connections 2024-03-06T07:26:16.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.605+0000 7f890e17a640 1 -- 172.21.15.1:0/3789322277 wait complete. 2024-03-06T07:26:16.638 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:26:16.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.743+0000 7f8dadeb5640 1 Processor -- start 2024-03-06T07:26:16.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.743+0000 7f8dadeb5640 1 -- start start 2024-03-06T07:26:16.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.743+0000 7f8dadeb5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8da80c6730 0x7f8da80c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.743+0000 7f8dadeb5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8da80c5d10 0x7f8da8156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.743+0000 7f8dadeb5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da80c3300 con 0x7f8da8152690 2024-03-06T07:26:16.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.743+0000 7f8dadeb5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8da80c35a0 con 0x7f8da80c5d10 2024-03-06T07:26:16.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.743+0000 7f8dadeb5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da80c3880 con 0x7f8da80c6730 2024-03-06T07:26:16.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.744+0000 7f8da6ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8da80c6730 0x7f8da80c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.744+0000 7f8da7fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8da80c5d10 0x7f8da8156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.744+0000 7f8da6ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8da80c6730 0x7f8da80c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:48024/0 (socket says 172.21.15.1:48024) 2024-03-06T07:26:16.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.744+0000 7f8da6ffd640 1 -- 172.21.15.1:0/2719240734 learned_addr learned my addr 172.21.15.1:0/2719240734 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:16.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.744+0000 7f8da77fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8da8152690 0x7f8da8155ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:56364/0 (socket says 172.21.15.1:56364) 2024-03-06T07:26:16.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.744+0000 7f8da6ffd640 1 -- 172.21.15.1:0/2719240734 >> v2:172.21.15.98:3300/0 conn(0x7f8da80c5d10 msgr2=0x7f8da8156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.744+0000 7f8da6ffd640 1 --2- 172.21.15.1:0/2719240734 >> v2:172.21.15.98:3300/0 conn(0x7f8da80c5d10 0x7f8da8156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.744+0000 7f8da6ffd640 1 -- 172.21.15.1:0/2719240734 >> v1:172.21.15.98:6789/0 conn(0x7f8da8152690 legacy=0x7f8da8155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.744+0000 7f8da6ffd640 1 -- 172.21.15.1:0/2719240734 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da80c3b60 con 0x7f8da80c6730 2024-03-06T07:26:16.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.745+0000 7f8da6ffd640 1 --2- 172.21.15.1:0/2719240734 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8da80c6730 0x7f8da80c57d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c009870 tx=0x7f8d9c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1014d9e75188c5a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.745+0000 7f8da67fc640 1 -- 172.21.15.1:0/2719240734 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c035020 con 0x7f8da80c6730 2024-03-06T07:26:16.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.745+0000 7f8da67fc640 1 -- 172.21.15.1:0/2719240734 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d9c031640 con 0x7f8da80c6730 2024-03-06T07:26:16.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.745+0000 7f8da67fc640 1 -- 172.21.15.1:0/2719240734 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c02d040 con 0x7f8da80c6730 2024-03-06T07:26:16.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.746+0000 7f8dadeb5640 1 -- 172.21.15.1:0/2719240734 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8da80c6730 msgr2=0x7f8da80c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.746+0000 7f8dadeb5640 1 --2- 172.21.15.1:0/2719240734 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8da80c6730 0x7f8da80c57d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c009870 tx=0x7f8d9c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.746+0000 7f8dadeb5640 1 -- 172.21.15.1:0/2719240734 shutdown_connections 2024-03-06T07:26:16.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.746+0000 7f8dadeb5640 1 --2- 172.21.15.1:0/2719240734 >> v2:172.21.15.98:3300/0 conn(0x7f8da80c5d10 0x7f8da8156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.746+0000 7f8dadeb5640 1 --2- 172.21.15.1:0/2719240734 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8da80c6730 0x7f8da80c57d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.746+0000 7f8dadeb5640 1 -- 172.21.15.1:0/2719240734 >> 172.21.15.1:0/2719240734 conn(0x7f8da806b410 msgr2=0x7f8da806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.746+0000 7f8dadeb5640 1 -- 172.21.15.1:0/2719240734 shutdown_connections 2024-03-06T07:26:16.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.746+0000 7f8dadeb5640 1 -- 172.21.15.1:0/2719240734 wait complete. 2024-03-06T07:26:16.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.747+0000 7f8dadeb5640 1 Processor -- start 2024-03-06T07:26:16.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.747+0000 7f8dadeb5640 1 -- start start 2024-03-06T07:26:16.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8dadeb5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da80c5d10 0x7f8da81e1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8dadeb5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8da80c6730 0x7f8da81540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8dadeb5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8da8152690 0x7f8da80ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8dadeb5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da8152bf0 con 0x7f8da80c5d10 2024-03-06T07:26:16.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8dadeb5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da8152a70 con 0x7f8da8152690 2024-03-06T07:26:16.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8dadeb5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da8157210 con 0x7f8da80c6730 2024-03-06T07:26:16.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8da77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da80c5d10 0x7f8da81e1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8da77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da80c5d10 0x7f8da81e1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44534/0 (socket says 172.21.15.1:44534) 2024-03-06T07:26:16.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8da77fe640 1 -- 172.21.15.1:0/3496683415 learned_addr learned my addr 172.21.15.1:0/3496683415 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:16.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8da6ffd640 1 --2- 172.21.15.1:0/3496683415 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8da80c6730 0x7f8da81540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8da7fff640 1 --2- 172.21.15.1:0/3496683415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8da8152690 0x7f8da80ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8da77fe640 1 -- 172.21.15.1:0/3496683415 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8da80c6730 msgr2=0x7f8da81540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8da77fe640 1 --2- 172.21.15.1:0/3496683415 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8da80c6730 0x7f8da81540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8da77fe640 1 -- 172.21.15.1:0/3496683415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8da8152690 msgr2=0x7f8da80ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8da77fe640 1 --2- 172.21.15.1:0/3496683415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8da8152690 0x7f8da80ace80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8da77fe640 1 -- 172.21.15.1:0/3496683415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da806d250 con 0x7f8da80c5d10 2024-03-06T07:26:16.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8da6ffd640 1 --2- 172.21.15.1:0/3496683415 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8da80c6730 0x7f8da81540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:16.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8da77fe640 1 --2- 172.21.15.1:0/3496683415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da80c5d10 0x7f8da81e1450 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f8d9800c430 tx=0x7f8d9800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8d8ffff640 1 -- 172.21.15.1:0/3496683415 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d980074e0 con 0x7f8da80c5d10 2024-03-06T07:26:16.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8d8ffff640 1 -- 172.21.15.1:0/3496683415 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d98007680 con 0x7f8da80c5d10 2024-03-06T07:26:16.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.748+0000 7f8dadeb5640 1 -- 172.21.15.1:0/3496683415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da81eede0 con 0x7f8da80c5d10 2024-03-06T07:26:16.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.749+0000 7f8d8ffff640 1 -- 172.21.15.1:0/3496683415 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d98004a00 con 0x7f8da80c5d10 2024-03-06T07:26:16.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.749+0000 7f8dadeb5640 1 -- 172.21.15.1:0/3496683415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da8155120 con 0x7f8da80c5d10 2024-03-06T07:26:16.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.749+0000 7f8dadeb5640 1 -- 172.21.15.1:0/3496683415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da8067fd0 con 0x7f8da80c5d10 2024-03-06T07:26:16.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.750+0000 7f8d8ffff640 1 -- 172.21.15.1:0/3496683415 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d9801a020 con 0x7f8da80c5d10 2024-03-06T07:26:16.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.750+0000 7f8d8ffff640 1 --2- 172.21.15.1:0/3496683415 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d9406b500 0x7f8d9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.751+0000 7f8d8ffff640 1 -- 172.21.15.1:0/3496683415 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f8d98089950 con 0x7f8da80c5d10 2024-03-06T07:26:16.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.751+0000 7f8da6ffd640 1 --2- 172.21.15.1:0/3496683415 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d9406b500 0x7f8d9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.751+0000 7f8da6ffd640 1 --2- 172.21.15.1:0/3496683415 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d9406b500 0x7f8d9406d9c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c036000 tx=0x7f8d9c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.755+0000 7f8d8ffff640 1 -- 172.21.15.1:0/3496683415 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d980580e0 con 0x7f8da80c5d10 2024-03-06T07:26:16.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:16.975+0000 7f8dadeb5640 1 -- 172.21.15.1:0/3496683415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f8da80c2c40 con 0x7f8da80c5d10 2024-03-06T07:26:17.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.626+0000 7f8d8ffff640 1 -- 172.21.15.1:0/3496683415 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 23 and data pool 24 v135) v1 ==== 174+0+0 (secure 0 0 0) 0x7f8d9805c180 con 0x7f8da80c5d10 2024-03-06T07:26:17.631 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 23 and data pool 24 2024-03-06T07:26:17.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.638+0000 7f8dadeb5640 1 -- 172.21.15.1:0/3496683415 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d9406b500 msgr2=0x7f8d9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.638+0000 7f8dadeb5640 1 --2- 172.21.15.1:0/3496683415 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d9406b500 0x7f8d9406d9c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c036000 tx=0x7f8d9c03c040 comp rx=0 tx=0).stop 2024-03-06T07:26:17.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.638+0000 7f8dadeb5640 1 -- 172.21.15.1:0/3496683415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da80c5d10 msgr2=0x7f8da81e1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.638+0000 7f8dadeb5640 1 --2- 172.21.15.1:0/3496683415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da80c5d10 0x7f8da81e1450 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f8d9800c430 tx=0x7f8d9800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.638+0000 7f8dadeb5640 1 -- 172.21.15.1:0/3496683415 shutdown_connections 2024-03-06T07:26:17.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.638+0000 7f8dadeb5640 1 --2- 172.21.15.1:0/3496683415 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d9406b500 0x7f8d9406d9c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.638+0000 7f8dadeb5640 1 --2- 172.21.15.1:0/3496683415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8da8152690 0x7f8da80ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.638+0000 7f8dadeb5640 1 --2- 172.21.15.1:0/3496683415 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8da80c6730 0x7f8da81540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.638+0000 7f8dadeb5640 1 --2- 172.21.15.1:0/3496683415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da80c5d10 0x7f8da81e1450 secure :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f8d9800c430 tx=0x7f8d9800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.638+0000 7f8dadeb5640 1 -- 172.21.15.1:0/3496683415 >> 172.21.15.1:0/3496683415 conn(0x7f8da806b410 msgr2=0x7f8da806d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.639+0000 7f8dadeb5640 1 -- 172.21.15.1:0/3496683415 shutdown_connections 2024-03-06T07:26:17.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.639+0000 7f8dadeb5640 1 -- 172.21.15.1:0/3496683415 wait complete. 2024-03-06T07:26:17.660 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:17.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.763+0000 7f1e36efb640 1 Processor -- start 2024-03-06T07:26:17.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.764+0000 7f1e36efb640 1 -- start start 2024-03-06T07:26:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.764+0000 7f1e36efb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1e300af6f0 0x7f1e300c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.764+0000 7f1e36efb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1e300c7930 0x7f1e3015fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.764+0000 7f1e36efb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e300c7180 con 0x7f1e30152820 2024-03-06T07:26:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.764+0000 7f1e36efb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e30155460 con 0x7f1e300c7930 2024-03-06T07:26:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.764+0000 7f1e36efb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e300c6bc0 con 0x7f1e300af6f0 2024-03-06T07:26:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.764+0000 7f1e35498640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1e300c7930 0x7f1e3015fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.764+0000 7f1e35498640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1e300c7930 0x7f1e3015fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44548/0 (socket says 172.21.15.1:44548) 2024-03-06T07:26:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.764+0000 7f1e34c97640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1e30152820 0x7f1e300c0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52676/0 (socket says 172.21.15.1:52676) 2024-03-06T07:26:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.764+0000 7f1e35498640 1 -- 172.21.15.1:0/1142859465 learned_addr learned my addr 172.21.15.1:0/1142859465 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.765+0000 7f1e2ffff640 1 --2- 172.21.15.1:0/1142859465 >> v2:172.21.15.98:3300/0 conn(0x7f1e300af6f0 0x7f1e300c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.765+0000 7f1e35498640 1 -- 172.21.15.1:0/1142859465 >> v2:172.21.15.98:3300/0 conn(0x7f1e300af6f0 msgr2=0x7f1e300c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.765+0000 7f1e35498640 1 --2- 172.21.15.1:0/1142859465 >> v2:172.21.15.98:3300/0 conn(0x7f1e300af6f0 0x7f1e300c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.765+0000 7f1e35498640 1 -- 172.21.15.1:0/1142859465 >> v1:172.21.15.1:6789/0 conn(0x7f1e30152820 legacy=0x7f1e300c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.765+0000 7f1e35498640 1 -- 172.21.15.1:0/1142859465 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e300c6ea0 con 0x7f1e300c7930 2024-03-06T07:26:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.765+0000 7f1e35498640 1 --2- 172.21.15.1:0/1142859465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300c7930 0x7f1e3015fb60 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f1e3006cca0 tx=0x7f1e2802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ca64461fd0e9b6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.766+0000 7f1e2f7fe640 1 -- 172.21.15.1:0/1142859465 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2802cb20 con 0x7f1e300c7930 2024-03-06T07:26:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.766+0000 7f1e2f7fe640 1 -- 172.21.15.1:0/1142859465 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2802ccc0 con 0x7f1e300c7930 2024-03-06T07:26:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.766+0000 7f1e2f7fe640 1 -- 172.21.15.1:0/1142859465 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e28034610 con 0x7f1e300c7930 2024-03-06T07:26:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.766+0000 7f1e36efb640 1 -- 172.21.15.1:0/1142859465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300c7930 msgr2=0x7f1e3015fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.766+0000 7f1e36efb640 1 --2- 172.21.15.1:0/1142859465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300c7930 0x7f1e3015fb60 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f1e3006cca0 tx=0x7f1e2802bf80 comp rx=0 tx=0).stop 2024-03-06T07:26:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.766+0000 7f1e36efb640 1 -- 172.21.15.1:0/1142859465 shutdown_connections 2024-03-06T07:26:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.766+0000 7f1e36efb640 1 --2- 172.21.15.1:0/1142859465 >> v2:172.21.15.98:3300/0 conn(0x7f1e300af6f0 0x7f1e300c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.766+0000 7f1e36efb640 1 --2- 172.21.15.1:0/1142859465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300c7930 0x7f1e3015fb60 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.766+0000 7f1e36efb640 1 -- 172.21.15.1:0/1142859465 >> 172.21.15.1:0/1142859465 conn(0x7f1e30072720 msgr2=0x7f1e30072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.766+0000 7f1e36efb640 1 -- 172.21.15.1:0/1142859465 shutdown_connections 2024-03-06T07:26:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.766+0000 7f1e36efb640 1 -- 172.21.15.1:0/1142859465 wait complete. 2024-03-06T07:26:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.767+0000 7f1e36efb640 1 Processor -- start 2024-03-06T07:26:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.767+0000 7f1e36efb640 1 -- start start 2024-03-06T07:26:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.768+0000 7f1e36efb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e300af6f0 0x7f1e30158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.768+0000 7f1e36efb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300c7930 0x7f1e301570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.768+0000 7f1e36efb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e30152820 0x7f1e301e7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.768+0000 7f1e36efb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e30152c00 con 0x7f1e300c7930 2024-03-06T07:26:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.768+0000 7f1e36efb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e300c0840 con 0x7f1e300af6f0 2024-03-06T07:26:17.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.768+0000 7f1e36efb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e30152d80 con 0x7f1e30152820 2024-03-06T07:26:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.768+0000 7f1e2ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300c7930 0x7f1e301570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.768+0000 7f1e2ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300c7930 0x7f1e301570d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44564/0 (socket says 172.21.15.1:44564) 2024-03-06T07:26:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.768+0000 7f1e2ffff640 1 -- 172.21.15.1:0/1115323510 learned_addr learned my addr 172.21.15.1:0/1115323510 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.768+0000 7f1e34c97640 1 --2- 172.21.15.1:0/1115323510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e300af6f0 0x7f1e30158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.768+0000 7f1e35498640 1 --2- 172.21.15.1:0/1115323510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e30152820 0x7f1e301e7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.768+0000 7f1e2ffff640 1 -- 172.21.15.1:0/1115323510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e30152820 msgr2=0x7f1e301e7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.768+0000 7f1e2ffff640 1 --2- 172.21.15.1:0/1115323510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e30152820 0x7f1e301e7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.768+0000 7f1e2ffff640 1 -- 172.21.15.1:0/1115323510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e300af6f0 msgr2=0x7f1e30158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.768+0000 7f1e2ffff640 1 --2- 172.21.15.1:0/1115323510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e300af6f0 0x7f1e30158ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.768+0000 7f1e2ffff640 1 -- 172.21.15.1:0/1115323510 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e301edcd0 con 0x7f1e300c7930 2024-03-06T07:26:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.769+0000 7f1e35498640 1 --2- 172.21.15.1:0/1115323510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e30152820 0x7f1e301e7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.769+0000 7f1e2ffff640 1 --2- 172.21.15.1:0/1115323510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300c7930 0x7f1e301570d0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f1e1800d800 tx=0x7f1e1800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.769+0000 7f1e2d7fa640 1 -- 172.21.15.1:0/1115323510 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1800b6c0 con 0x7f1e300c7930 2024-03-06T07:26:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.769+0000 7f1e2d7fa640 1 -- 172.21.15.1:0/1115323510 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e18019070 con 0x7f1e300c7930 2024-03-06T07:26:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.769+0000 7f1e2d7fa640 1 -- 172.21.15.1:0/1115323510 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1800ba80 con 0x7f1e300c7930 2024-03-06T07:26:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.769+0000 7f1e36efb640 1 -- 172.21.15.1:0/1115323510 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e301595f0 con 0x7f1e300c7930 2024-03-06T07:26:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.769+0000 7f1e36efb640 1 -- 172.21.15.1:0/1115323510 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e300c3c90 con 0x7f1e300c7930 2024-03-06T07:26:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.770+0000 7f1e36efb640 1 -- 172.21.15.1:0/1115323510 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e300686e0 con 0x7f1e300c7930 2024-03-06T07:26:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.770+0000 7f1e2d7fa640 1 -- 172.21.15.1:0/1115323510 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e1800bc20 con 0x7f1e300c7930 2024-03-06T07:26:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.771+0000 7f1e2d7fa640 1 --2- 172.21.15.1:0/1115323510 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e1006b4d0 0x7f1e1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.771+0000 7f1e2d7fa640 1 -- 172.21.15.1:0/1115323510 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1e1808a850 con 0x7f1e300c7930 2024-03-06T07:26:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.771+0000 7f1e34c97640 1 --2- 172.21.15.1:0/1115323510 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e1006b4d0 0x7f1e1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.771+0000 7f1e34c97640 1 --2- 172.21.15.1:0/1115323510 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e1006b4d0 0x7f1e1006d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1e20000f60 tx=0x7f1e20007ce0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.775+0000 7f1e2d7fa640 1 -- 172.21.15.1:0/1115323510 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e18059530 con 0x7f1e300c7930 2024-03-06T07:26:17.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.960+0000 7f1e36efb640 1 -- 172.21.15.1:0/1115323510 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e300c5b40 con 0x7f1e300c7930 2024-03-06T07:26:17.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.961+0000 7f1e2d7fa640 1 -- 172.21.15.1:0/1115323510 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16777 (secure 0 0 0) 0x7f1e18058e30 con 0x7f1e300c7930 2024-03-06T07:26:17.963 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:17.963 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":148,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:26:16.976813+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:15.245426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:15.942343+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:17.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.963+0000 7f1e36efb640 1 -- 172.21.15.1:0/1115323510 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e1006b4d0 msgr2=0x7f1e1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.963+0000 7f1e36efb640 1 --2- 172.21.15.1:0/1115323510 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e1006b4d0 0x7f1e1006d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1e20000f60 tx=0x7f1e20007ce0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.963+0000 7f1e36efb640 1 -- 172.21.15.1:0/1115323510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300c7930 msgr2=0x7f1e301570d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.963+0000 7f1e36efb640 1 --2- 172.21.15.1:0/1115323510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300c7930 0x7f1e301570d0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f1e1800d800 tx=0x7f1e1800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.963+0000 7f1e36efb640 1 -- 172.21.15.1:0/1115323510 shutdown_connections 2024-03-06T07:26:17.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.963+0000 7f1e36efb640 1 --2- 172.21.15.1:0/1115323510 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e1006b4d0 0x7f1e1006d990 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.963+0000 7f1e36efb640 1 --2- 172.21.15.1:0/1115323510 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e30152820 0x7f1e301e7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.963+0000 7f1e36efb640 1 --2- 172.21.15.1:0/1115323510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300c7930 0x7f1e301570d0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.963+0000 7f1e36efb640 1 --2- 172.21.15.1:0/1115323510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e300af6f0 0x7f1e30158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.963+0000 7f1e36efb640 1 -- 172.21.15.1:0/1115323510 >> 172.21.15.1:0/1115323510 conn(0x7f1e30072720 msgr2=0x7f1e3006cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.964+0000 7f1e36efb640 1 -- 172.21.15.1:0/1115323510 shutdown_connections 2024-03-06T07:26:17.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:17.964+0000 7f1e36efb640 1 -- 172.21.15.1:0/1115323510 wait complete. 2024-03-06T07:26:17.977 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:18.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.075+0000 7fa1fc73a640 1 Processor -- start 2024-03-06T07:26:18.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.076+0000 7fa1fc73a640 1 -- start start 2024-03-06T07:26:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.076+0000 7fa1fc73a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa1f4152920 0x7fa1f406aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.076+0000 7fa1fc73a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1f406b530 0x7fa1f40c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.076+0000 7fa1fc73a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f4158910 con 0x7fa1f40c5d90 2024-03-06T07:26:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.076+0000 7fa1fc73a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1f4152d00 con 0x7fa1f4152920 2024-03-06T07:26:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.076+0000 7fa1fc73a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f41596d0 con 0x7fa1f406b530 2024-03-06T07:26:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.077+0000 7fa1fa4d6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa1f4152920 0x7fa1f406aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.077+0000 7fa1facd7640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa1f40c5d90 0x7fa1f4155220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52680/0 (socket says 172.21.15.1:52680) 2024-03-06T07:26:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.077+0000 7fa1facd7640 1 -- 172.21.15.1:0/1943277818 learned_addr learned my addr 172.21.15.1:0/1943277818 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.077+0000 7fa1f9cd5640 1 --2- 172.21.15.1:0/1943277818 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1f406b530 0x7fa1f40c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.077+0000 7fa1fa4d6640 1 -- 172.21.15.1:0/1943277818 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1f406b530 msgr2=0x7fa1f40c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.077+0000 7fa1fa4d6640 1 --2- 172.21.15.1:0/1943277818 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1f406b530 0x7fa1f40c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.077+0000 7fa1fa4d6640 1 -- 172.21.15.1:0/1943277818 >> v1:172.21.15.1:6789/0 conn(0x7fa1f40c5d90 legacy=0x7fa1f4155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.077+0000 7fa1f94d4640 1 -- 172.21.15.1:0/1943277818 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856485417 0 0) 0x7fa1f4158910 con 0x7fa1f40c5d90 2024-03-06T07:26:18.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.077+0000 7fa1fa4d6640 1 -- 172.21.15.1:0/1943277818 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f4158bf0 con 0x7fa1f4152920 2024-03-06T07:26:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.078+0000 7fa1fa4d6640 1 --2- 172.21.15.1:0/1943277818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1f4152920 0x7fa1f406aff0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40097d0 tx=0x7fa1e402b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d81a59bbf620a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.078+0000 7fa1f94d4640 1 -- 172.21.15.1:0/1943277818 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4031770 con 0x7fa1f4152920 2024-03-06T07:26:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.078+0000 7fa1f94d4640 1 -- 172.21.15.1:0/1943277818 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e4037070 con 0x7fa1f4152920 2024-03-06T07:26:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.078+0000 7fa1f94d4640 1 -- 172.21.15.1:0/1943277818 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4031b30 con 0x7fa1f4152920 2024-03-06T07:26:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.078+0000 7fa1fc73a640 1 -- 172.21.15.1:0/1943277818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1f4152920 msgr2=0x7fa1f406aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.078+0000 7fa1fc73a640 1 --2- 172.21.15.1:0/1943277818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1f4152920 0x7fa1f406aff0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40097d0 tx=0x7fa1e402b810 comp rx=0 tx=0).stop 2024-03-06T07:26:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.078+0000 7fa1fc73a640 1 -- 172.21.15.1:0/1943277818 shutdown_connections 2024-03-06T07:26:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.078+0000 7fa1fc73a640 1 --2- 172.21.15.1:0/1943277818 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1f406b530 0x7fa1f40c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.078+0000 7fa1fc73a640 1 --2- 172.21.15.1:0/1943277818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1f4152920 0x7fa1f406aff0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.078+0000 7fa1fc73a640 1 -- 172.21.15.1:0/1943277818 >> 172.21.15.1:0/1943277818 conn(0x7fa1f40a9700 msgr2=0x7fa1f40a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.078+0000 7fa1fc73a640 1 -- 172.21.15.1:0/1943277818 shutdown_connections 2024-03-06T07:26:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.078+0000 7fa1fc73a640 1 -- 172.21.15.1:0/1943277818 wait complete. 2024-03-06T07:26:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.079+0000 7fa1fc73a640 1 Processor -- start 2024-03-06T07:26:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.079+0000 7fa1fc73a640 1 -- start start 2024-03-06T07:26:18.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.080+0000 7fa1fc73a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1f406b530 0x7fa1f41eaa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.080+0000 7fa1fc73a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1f40c5d90 0x7fa1f40aa7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.080+0000 7fa1fc73a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1f4152920 0x7fa1f406f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.080+0000 7fa1fc73a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f40c6380 con 0x7fa1f406b530 2024-03-06T07:26:18.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.080+0000 7fa1fc73a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f4159240 con 0x7fa1f4152920 2024-03-06T07:26:18.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.080+0000 7fa1fc73a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f4155930 con 0x7fa1f40c5d90 2024-03-06T07:26:18.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.080+0000 7fa1fa4d6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1f406b530 0x7fa1f41eaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.080+0000 7fa1fa4d6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1f406b530 0x7fa1f41eaa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44596/0 (socket says 172.21.15.1:44596) 2024-03-06T07:26:18.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.080+0000 7fa1fa4d6640 1 -- 172.21.15.1:0/2209214835 learned_addr learned my addr 172.21.15.1:0/2209214835 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:18.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.080+0000 7fa1f9cd5640 1 --2- 172.21.15.1:0/2209214835 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1f40c5d90 0x7fa1f40aa7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.080+0000 7fa1facd7640 1 --2- 172.21.15.1:0/2209214835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1f4152920 0x7fa1f406f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.081+0000 7fa1fa4d6640 1 -- 172.21.15.1:0/2209214835 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1f40c5d90 msgr2=0x7fa1f40aa7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.081+0000 7fa1fa4d6640 1 --2- 172.21.15.1:0/2209214835 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1f40c5d90 0x7fa1f40aa7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.081+0000 7fa1fa4d6640 1 -- 172.21.15.1:0/2209214835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1f4152920 msgr2=0x7fa1f406f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.081+0000 7fa1fa4d6640 1 --2- 172.21.15.1:0/2209214835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1f4152920 0x7fa1f406f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.081+0000 7fa1fa4d6640 1 -- 172.21.15.1:0/2209214835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f4158910 con 0x7fa1f406b530 2024-03-06T07:26:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.081+0000 7fa1facd7640 1 --2- 172.21.15.1:0/2209214835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1f4152920 0x7fa1f406f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.081+0000 7fa1fa4d6640 1 --2- 172.21.15.1:0/2209214835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1f406b530 0x7fa1f41eaa60 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4034040 tx=0x7fa1e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.081+0000 7fa1eaffd640 1 -- 172.21.15.1:0/2209214835 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4032600 con 0x7fa1f406b530 2024-03-06T07:26:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.081+0000 7fa1eaffd640 1 -- 172.21.15.1:0/2209214835 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e40327c0 con 0x7fa1f406b530 2024-03-06T07:26:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.081+0000 7fa1eaffd640 1 -- 172.21.15.1:0/2209214835 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4032bb0 con 0x7fa1f406b530 2024-03-06T07:26:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.081+0000 7fa1fc73a640 1 -- 172.21.15.1:0/2209214835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f40a72a0 con 0x7fa1f406b530 2024-03-06T07:26:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.081+0000 7fa1fc73a640 1 -- 172.21.15.1:0/2209214835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f40a7820 con 0x7fa1f406b530 2024-03-06T07:26:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.082+0000 7fa1fc73a640 1 -- 172.21.15.1:0/2209214835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f4067fd0 con 0x7fa1f406b530 2024-03-06T07:26:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.082+0000 7fa1eaffd640 1 -- 172.21.15.1:0/2209214835 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa1e404c060 con 0x7fa1f406b530 2024-03-06T07:26:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.083+0000 7fa1eaffd640 1 --2- 172.21.15.1:0/2209214835 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa1d806b4d0 0x7fa1d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.083+0000 7fa1eaffd640 1 -- 172.21.15.1:0/2209214835 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa1e40ab7f0 con 0x7fa1f406b530 2024-03-06T07:26:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.083+0000 7fa1f9cd5640 1 --2- 172.21.15.1:0/2209214835 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa1d806b4d0 0x7fa1d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.083+0000 7fa1f9cd5640 1 --2- 172.21.15.1:0/2209214835 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa1d806b4d0 0x7fa1d806d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa1f40a96d0 tx=0x7fa1f0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.087+0000 7fa1eaffd640 1 -- 172.21.15.1:0/2209214835 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa1e4040070 con 0x7fa1f406b530 2024-03-06T07:26:18.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.267+0000 7fa1fc73a640 1 -- 172.21.15.1:0/2209214835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1f406bdf0 con 0x7fa1f406b530 2024-03-06T07:26:18.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.269+0000 7fa1eaffd640 1 -- 172.21.15.1:0/2209214835 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16777 (secure 0 0 0) 0x7fa1e407a570 con 0x7fa1f406b530 2024-03-06T07:26:18.271 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:18.271 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":148,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:26:16.976813+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:15.245426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:15.942343+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.270+0000 7fa1fc73a640 1 -- 172.21.15.1:0/2209214835 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa1d806b4d0 msgr2=0x7fa1d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.270+0000 7fa1fc73a640 1 --2- 172.21.15.1:0/2209214835 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa1d806b4d0 0x7fa1d806d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa1f40a96d0 tx=0x7fa1f0008040 comp rx=0 tx=0).stop 2024-03-06T07:26:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.270+0000 7fa1fc73a640 1 -- 172.21.15.1:0/2209214835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1f406b530 msgr2=0x7fa1f41eaa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.270+0000 7fa1fc73a640 1 --2- 172.21.15.1:0/2209214835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1f406b530 0x7fa1f41eaa60 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4034040 tx=0x7fa1e40046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.271+0000 7fa1fc73a640 1 -- 172.21.15.1:0/2209214835 shutdown_connections 2024-03-06T07:26:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.271+0000 7fa1fc73a640 1 --2- 172.21.15.1:0/2209214835 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa1d806b4d0 0x7fa1d806d990 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.271+0000 7fa1fc73a640 1 --2- 172.21.15.1:0/2209214835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1f4152920 0x7fa1f406f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.271+0000 7fa1fc73a640 1 --2- 172.21.15.1:0/2209214835 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1f40c5d90 0x7fa1f40aa7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.271+0000 7fa1fc73a640 1 --2- 172.21.15.1:0/2209214835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1f406b530 0x7fa1f41eaa60 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.271+0000 7fa1fc73a640 1 -- 172.21.15.1:0/2209214835 >> 172.21.15.1:0/2209214835 conn(0x7fa1f40a9700 msgr2=0x7fa1f41512c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.271+0000 7fa1fc73a640 1 -- 172.21.15.1:0/2209214835 shutdown_connections 2024-03-06T07:26:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.271+0000 7fa1fc73a640 1 -- 172.21.15.1:0/2209214835 wait complete. 2024-03-06T07:26:18.284 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:26:18.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.385+0000 7f0fec90c640 1 Processor -- start 2024-03-06T07:26:18.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.385+0000 7f0fec90c640 1 -- start start 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.386+0000 7f0fec90c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0fe8152650 0x7f0fe8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.386+0000 7f0fec90c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0fe80bf9d0 0x7f0fe80c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.386+0000 7f0fec90c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe80c2fc0 con 0x7f0fe80c6b10 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.386+0000 7f0fec90c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe8155800 con 0x7f0fe8152650 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.386+0000 7f0fec90c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe8158da0 con 0x7f0fe80bf9d0 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.386+0000 7f0fe659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0fe8152650 0x7f0fe8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.386+0000 7f0fe659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0fe8152650 0x7f0fe8155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44610/0 (socket says 172.21.15.1:44610) 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.386+0000 7f0fe659c640 1 -- 172.21.15.1:0/3975870006 learned_addr learned my addr 172.21.15.1:0/3975870006 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.386+0000 7f0fe6d9d640 1 --2- 172.21.15.1:0/3975870006 >> v2:172.21.15.98:3300/0 conn(0x7f0fe80bf9d0 0x7f0fe80c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.387+0000 7f0fe659c640 1 -- 172.21.15.1:0/3975870006 >> v2:172.21.15.98:3300/0 conn(0x7f0fe80bf9d0 msgr2=0x7f0fe80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.387+0000 7f0fe659c640 1 --2- 172.21.15.1:0/3975870006 >> v2:172.21.15.98:3300/0 conn(0x7f0fe80bf9d0 0x7f0fe80c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.387+0000 7f0fe659c640 1 -- 172.21.15.1:0/3975870006 >> v1:172.21.15.98:6789/0 conn(0x7f0fe80c6b10 legacy=0x7f0fe8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.387+0000 7f0fe659c640 1 -- 172.21.15.1:0/3975870006 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe80c6690 con 0x7f0fe8152650 2024-03-06T07:26:18.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.387+0000 7f0fe659c640 1 --2- 172.21.15.1:0/3975870006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fe8152650 0x7f0fe8155c90 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f0fe81597b0 tx=0x7f0fe002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beb0c9b19653343f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.388+0000 7f0fe559a640 1 -- 172.21.15.1:0/3975870006 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0035020 con 0x7f0fe8152650 2024-03-06T07:26:18.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.388+0000 7f0fe559a640 1 -- 172.21.15.1:0/3975870006 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe0031870 con 0x7f0fe8152650 2024-03-06T07:26:18.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.388+0000 7f0fe559a640 1 -- 172.21.15.1:0/3975870006 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0031c40 con 0x7f0fe8152650 2024-03-06T07:26:18.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.388+0000 7f0fec90c640 1 -- 172.21.15.1:0/3975870006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fe8152650 msgr2=0x7f0fe8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.388+0000 7f0fec90c640 1 --2- 172.21.15.1:0/3975870006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fe8152650 0x7f0fe8155c90 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f0fe81597b0 tx=0x7f0fe002b910 comp rx=0 tx=0).stop 2024-03-06T07:26:18.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.388+0000 7f0fec90c640 1 -- 172.21.15.1:0/3975870006 shutdown_connections 2024-03-06T07:26:18.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.388+0000 7f0fec90c640 1 --2- 172.21.15.1:0/3975870006 >> v2:172.21.15.98:3300/0 conn(0x7f0fe80bf9d0 0x7f0fe80c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.388+0000 7f0fec90c640 1 --2- 172.21.15.1:0/3975870006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fe8152650 0x7f0fe8155c90 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.388+0000 7f0fec90c640 1 -- 172.21.15.1:0/3975870006 >> 172.21.15.1:0/3975870006 conn(0x7f0fe8066ee0 msgr2=0x7f0fe80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.388+0000 7f0fec90c640 1 -- 172.21.15.1:0/3975870006 shutdown_connections 2024-03-06T07:26:18.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.388+0000 7f0fec90c640 1 -- 172.21.15.1:0/3975870006 wait complete. 2024-03-06T07:26:18.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.389+0000 7f0fec90c640 1 Processor -- start 2024-03-06T07:26:18.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.389+0000 7f0fec90c640 1 -- start start 2024-03-06T07:26:18.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.390+0000 7f0fec90c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fe80bf9d0 0x7f0fe8158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.390+0000 7f0fec90c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fe8152650 0x7f0fe8153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.390+0000 7f0fec90c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fe8156b10 0x7f0fe8156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.390+0000 7f0fec90c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe81551b0 con 0x7f0fe8152650 2024-03-06T07:26:18.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.390+0000 7f0fec90c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe80c6fa0 con 0x7f0fe8156b10 2024-03-06T07:26:18.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.390+0000 7f0fec90c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8155490 con 0x7f0fe80bf9d0 2024-03-06T07:26:18.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.390+0000 7f0fe5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fe8152650 0x7f0fe8153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.390+0000 7f0fe5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fe8152650 0x7f0fe8153650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44624/0 (socket says 172.21.15.1:44624) 2024-03-06T07:26:18.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.390+0000 7f0fe5d9b640 1 -- 172.21.15.1:0/1454177151 learned_addr learned my addr 172.21.15.1:0/1454177151 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:18.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.390+0000 7f0fe659c640 1 --2- 172.21.15.1:0/1454177151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fe80bf9d0 0x7f0fe8158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.391+0000 7f0fe6d9d640 1 --2- 172.21.15.1:0/1454177151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fe8156b10 0x7f0fe8156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.391+0000 7f0fe5d9b640 1 -- 172.21.15.1:0/1454177151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fe80bf9d0 msgr2=0x7f0fe8158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.391+0000 7f0fe5d9b640 1 --2- 172.21.15.1:0/1454177151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fe80bf9d0 0x7f0fe8158830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.391+0000 7f0fe5d9b640 1 -- 172.21.15.1:0/1454177151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fe8156b10 msgr2=0x7f0fe8156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.391+0000 7f0fe5d9b640 1 --2- 172.21.15.1:0/1454177151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fe8156b10 0x7f0fe8156fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.391+0000 7f0fe5d9b640 1 -- 172.21.15.1:0/1454177151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe8153e90 con 0x7f0fe8152650 2024-03-06T07:26:18.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.391+0000 7f0fe659c640 1 --2- 172.21.15.1:0/1454177151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fe80bf9d0 0x7f0fe8158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:18.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.391+0000 7f0fe5d9b640 1 --2- 172.21.15.1:0/1454177151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fe8152650 0x7f0fe8153650 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f0fd000c400 tx=0x7f0fd000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.391+0000 7f0fdeffd640 1 -- 172.21.15.1:0/1454177151 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd00052d0 con 0x7f0fe8152650 2024-03-06T07:26:18.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.391+0000 7f0fdeffd640 1 -- 172.21.15.1:0/1454177151 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd0005490 con 0x7f0fe8152650 2024-03-06T07:26:18.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.391+0000 7f0fec90c640 1 -- 172.21.15.1:0/1454177151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe80c87d0 con 0x7f0fe8152650 2024-03-06T07:26:18.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.391+0000 7f0fdeffd640 1 -- 172.21.15.1:0/1454177151 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0007260 con 0x7f0fe8152650 2024-03-06T07:26:18.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.391+0000 7f0fec90c640 1 -- 172.21.15.1:0/1454177151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe81eb950 con 0x7f0fe8152650 2024-03-06T07:26:18.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.392+0000 7f0fec90c640 1 -- 172.21.15.1:0/1454177151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe8067fd0 con 0x7f0fe8152650 2024-03-06T07:26:18.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.392+0000 7f0fdeffd640 1 -- 172.21.15.1:0/1454177151 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0fd0007400 con 0x7f0fe8152650 2024-03-06T07:26:18.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.393+0000 7f0fdeffd640 1 --2- 172.21.15.1:0/1454177151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fc806b4d0 0x7f0fc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.393+0000 7f0fdeffd640 1 -- 172.21.15.1:0/1454177151 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0fd0089880 con 0x7f0fe8152650 2024-03-06T07:26:18.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.393+0000 7f0fe659c640 1 --2- 172.21.15.1:0/1454177151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fc806b4d0 0x7f0fc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.394+0000 7f0fe659c640 1 --2- 172.21.15.1:0/1454177151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fc806b4d0 0x7f0fc806d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0036000 tx=0x7f0fe0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.397+0000 7f0fdeffd640 1 -- 172.21.15.1:0/1454177151 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0fd0058600 con 0x7f0fe8152650 2024-03-06T07:26:18.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.620+0000 7f0fec90c640 1 -- 172.21.15.1:0/1454177151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0fe80b06f0 con 0x7f0fe8152650 2024-03-06T07:26:18.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.622+0000 7f0fdeffd640 1 -- 172.21.15.1:0/1454177151 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v136) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0fd0057f00 con 0x7f0fe8152650 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.624+0000 7f0fec90c640 1 -- 172.21.15.1:0/1454177151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fc806b4d0 msgr2=0x7f0fc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.624+0000 7f0fec90c640 1 --2- 172.21.15.1:0/1454177151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fc806b4d0 0x7f0fc806d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0036000 tx=0x7f0fe0008030 comp rx=0 tx=0).stop 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.624+0000 7f0fec90c640 1 -- 172.21.15.1:0/1454177151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fe8152650 msgr2=0x7f0fe8153650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.624+0000 7f0fec90c640 1 --2- 172.21.15.1:0/1454177151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fe8152650 0x7f0fe8153650 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f0fd000c400 tx=0x7f0fd000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.624+0000 7f0fec90c640 1 -- 172.21.15.1:0/1454177151 shutdown_connections 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.624+0000 7f0fec90c640 1 --2- 172.21.15.1:0/1454177151 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0fc806b4d0 0x7f0fc806d990 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.624+0000 7f0fec90c640 1 --2- 172.21.15.1:0/1454177151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fe8156b10 0x7f0fe8156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.624+0000 7f0fec90c640 1 --2- 172.21.15.1:0/1454177151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fe8152650 0x7f0fe8153650 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.624+0000 7f0fec90c640 1 --2- 172.21.15.1:0/1454177151 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fe80bf9d0 0x7f0fe8158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.624+0000 7f0fec90c640 1 -- 172.21.15.1:0/1454177151 >> 172.21.15.1:0/1454177151 conn(0x7f0fe8066ee0 msgr2=0x7f0fe80c3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.624+0000 7f0fec90c640 1 -- 172.21.15.1:0/1454177151 shutdown_connections 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.624+0000 7f0fec90c640 1 -- 172.21.15.1:0/1454177151 wait complete. 2024-03-06T07:26:18.639 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:26:18.639 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:26:18.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.741+0000 7f4132a2a640 1 Processor -- start 2024-03-06T07:26:18.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.741+0000 7f4132a2a640 1 -- start start 2024-03-06T07:26:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.742+0000 7f4132a2a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f412c0c3970 0x7f412c0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.742+0000 7f4132a2a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f412c0c60b0 0x7f412c152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.742+0000 7f4132a2a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f412c0c5e40 con 0x7f412c152670 2024-03-06T07:26:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.742+0000 7f4132a2a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f412c152ad0 con 0x7f412c0c60b0 2024-03-06T07:26:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.742+0000 7f4132a2a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412c0c67f0 con 0x7f412c0c3970 2024-03-06T07:26:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.742+0000 7f4130fc7640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f412c152670 0x7f412c1553d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52688/0 (socket says 172.21.15.1:52688) 2024-03-06T07:26:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.742+0000 7f4130fc7640 1 -- 172.21.15.1:0/1625035289 learned_addr learned my addr 172.21.15.1:0/1625035289 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412affd640 1 -- 172.21.15.1:0/1625035289 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475201452 0 0) 0x7f412c0c5e40 con 0x7f412c152670 2024-03-06T07:26:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412bfff640 1 --2- 172.21.15.1:0/1625035289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f412c0c3970 0x7f412c0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412affd640 1 -- 172.21.15.1:0/1625035289 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4114003520 con 0x7f412c152670 2024-03-06T07:26:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412b7fe640 1 --2- 172.21.15.1:0/1625035289 >> v2:172.21.15.98:3300/0 conn(0x7f412c0c60b0 0x7f412c152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412affd640 1 -- 172.21.15.1:0/1625035289 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3039212686 0 0) 0x7f4114003520 con 0x7f412c152670 2024-03-06T07:26:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412affd640 1 -- 172.21.15.1:0/1625035289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f412c0c3970 msgr2=0x7f412c0c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412affd640 1 --2- 172.21.15.1:0/1625035289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f412c0c3970 0x7f412c0c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412affd640 1 -- 172.21.15.1:0/1625035289 >> v2:172.21.15.98:3300/0 conn(0x7f412c0c60b0 msgr2=0x7f412c152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412affd640 1 --2- 172.21.15.1:0/1625035289 >> v2:172.21.15.98:3300/0 conn(0x7f412c0c60b0 0x7f412c152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412affd640 1 -- 172.21.15.1:0/1625035289 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412c0c6a40 con 0x7f412c152670 2024-03-06T07:26:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412affd640 1 -- 172.21.15.1:0/1625035289 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f41240019f0 con 0x7f412c152670 2024-03-06T07:26:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412affd640 1 -- 172.21.15.1:0/1625035289 >> v1:172.21.15.1:6789/0 conn(0x7f412c152670 legacy=0x7f412c1553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412affd640 1 --2- 172.21.15.1:0/1625035289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41140039b0 0x7f4114003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412affd640 1 --2- 172.21.15.1:0/1625035289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4114004800 0x7f4114004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412affd640 1 --2- 172.21.15.1:0/1625035289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f412c0c3970 0x7f41140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412affd640 1 -- 172.21.15.1:0/1625035289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c152ad0 con 0x7f41140039b0 2024-03-06T07:26:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412affd640 1 -- 172.21.15.1:0/1625035289 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c0c67f0 con 0x7f4114004800 2024-03-06T07:26:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412affd640 1 -- 172.21.15.1:0/1625035289 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411400ba70 con 0x7f412c0c3970 2024-03-06T07:26:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412b7fe640 1 --2- 172.21.15.1:0/1625035289 >> v2:172.21.15.98:3300/0 conn(0x7f412c0c60b0 0x7f412c152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412bfff640 1 --2- 172.21.15.1:0/1625035289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41140039b0 0x7f4114003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f4130fc7640 1 --2- 172.21.15.1:0/1625035289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f412c0c3970 0x7f41140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412b7fe640 1 --2- 172.21.15.1:0/1625035289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4114004800 0x7f4114004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412bfff640 1 -- 172.21.15.1:0/1625035289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f412c0c3970 msgr2=0x7f41140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412bfff640 1 --2- 172.21.15.1:0/1625035289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f412c0c3970 0x7f41140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412bfff640 1 -- 172.21.15.1:0/1625035289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4114004800 msgr2=0x7f4114004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412bfff640 1 --2- 172.21.15.1:0/1625035289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4114004800 0x7f4114004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412bfff640 1 -- 172.21.15.1:0/1625035289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4114003520 con 0x7f41140039b0 2024-03-06T07:26:18.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f412b7fe640 1 --2- 172.21.15.1:0/1625035289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4114004800 0x7f4114004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:18.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.743+0000 7f4130fc7640 1 --2- 172.21.15.1:0/1625035289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f412c0c3970 0x7f41140051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:18.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.744+0000 7f412bfff640 1 --2- 172.21.15.1:0/1625035289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41140039b0 0x7f4114003dd0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f411c00ba00 tx=0x7f4114005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb44fb650ed439a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.744+0000 7f412affd640 1 -- 172.21.15.1:0/1625035289 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c036020 con 0x7f41140039b0 2024-03-06T07:26:18.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.744+0000 7f412affd640 1 -- 172.21.15.1:0/1625035289 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411c0325f0 con 0x7f41140039b0 2024-03-06T07:26:18.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.744+0000 7f412affd640 1 -- 172.21.15.1:0/1625035289 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c00b600 con 0x7f41140039b0 2024-03-06T07:26:18.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.745+0000 7f4132a2a640 1 -- 172.21.15.1:0/1625035289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41140039b0 msgr2=0x7f4114003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.745+0000 7f4132a2a640 1 --2- 172.21.15.1:0/1625035289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41140039b0 0x7f4114003dd0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f411c00ba00 tx=0x7f4114005830 comp rx=0 tx=0).stop 2024-03-06T07:26:18.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.745+0000 7f4130fc7640 1 -- 172.21.15.1:0/1625035289 reap_dead start 2024-03-06T07:26:18.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.745+0000 7f4132a2a640 1 -- 172.21.15.1:0/1625035289 shutdown_connections 2024-03-06T07:26:18.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.745+0000 7f4132a2a640 1 -- 172.21.15.1:0/1625035289 >> 172.21.15.1:0/1625035289 conn(0x7f412c0c88f0 msgr2=0x7f412c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.745+0000 7f4132a2a640 1 -- 172.21.15.1:0/1625035289 shutdown_connections 2024-03-06T07:26:18.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.745+0000 7f4132a2a640 1 -- 172.21.15.1:0/1625035289 wait complete. 2024-03-06T07:26:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.746+0000 7f4132a2a640 1 Processor -- start 2024-03-06T07:26:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.747+0000 7f4132a2a640 1 -- start start 2024-03-06T07:26:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f4132a2a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41140039b0 0x7f412c0707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f4132a2a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f412c0ada30 0x7f412c0ac320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f4132a2a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f412c14f4b0 0x7f412c0ac860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f4132a2a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c0c72e0 con 0x7f412c0ada30 2024-03-06T07:26:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f4132a2a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c0c7000 con 0x7f412c14f4b0 2024-03-06T07:26:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f4132a2a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412c0c75c0 con 0x7f41140039b0 2024-03-06T07:26:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f412b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f412c0ada30 0x7f412c0ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f412b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f412c0ada30 0x7f412c0ac320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44632/0 (socket says 172.21.15.1:44632) 2024-03-06T07:26:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f412b7fe640 1 -- 172.21.15.1:0/1790800972 learned_addr learned my addr 172.21.15.1:0/1790800972 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f412bfff640 1 --2- 172.21.15.1:0/1790800972 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41140039b0 0x7f412c0707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f4130fc7640 1 --2- 172.21.15.1:0/1790800972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f412c14f4b0 0x7f412c0ac860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f412b7fe640 1 -- 172.21.15.1:0/1790800972 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41140039b0 msgr2=0x7f412c0707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f412b7fe640 1 --2- 172.21.15.1:0/1790800972 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41140039b0 0x7f412c0707a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f412b7fe640 1 -- 172.21.15.1:0/1790800972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f412c14f4b0 msgr2=0x7f412c0ac860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f412b7fe640 1 --2- 172.21.15.1:0/1790800972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f412c14f4b0 0x7f412c0ac860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f412b7fe640 1 -- 172.21.15.1:0/1790800972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412c1eb360 con 0x7f412c0ada30 2024-03-06T07:26:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f4130fc7640 1 --2- 172.21.15.1:0/1790800972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f412c14f4b0 0x7f412c0ac860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.748+0000 7f412b7fe640 1 --2- 172.21.15.1:0/1790800972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f412c0ada30 0x7f412c0ac320 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f4120010860 tx=0x7f4120010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.749+0000 7f4128ff9640 1 -- 172.21.15.1:0/1790800972 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412001a020 con 0x7f412c0ada30 2024-03-06T07:26:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.749+0000 7f4128ff9640 1 -- 172.21.15.1:0/1790800972 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412000eb60 con 0x7f412c0ada30 2024-03-06T07:26:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.749+0000 7f4132a2a640 1 -- 172.21.15.1:0/1790800972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f412c1eb640 con 0x7f412c0ada30 2024-03-06T07:26:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.749+0000 7f4128ff9640 1 -- 172.21.15.1:0/1790800972 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412000f040 con 0x7f412c0ada30 2024-03-06T07:26:18.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.749+0000 7f4132a2a640 1 -- 172.21.15.1:0/1790800972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412c070ce0 con 0x7f412c0ada30 2024-03-06T07:26:18.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.750+0000 7f4132a2a640 1 -- 172.21.15.1:0/1790800972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f412c0686e0 con 0x7f412c0ada30 2024-03-06T07:26:18.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.751+0000 7f4128ff9640 1 -- 172.21.15.1:0/1790800972 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f412001d020 con 0x7f412c0ada30 2024-03-06T07:26:18.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.751+0000 7f4128ff9640 1 --2- 172.21.15.1:0/1790800972 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f411806b500 0x7f411806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.751+0000 7f4128ff9640 1 -- 172.21.15.1:0/1790800972 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f412008aaf0 con 0x7f412c0ada30 2024-03-06T07:26:18.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.751+0000 7f412bfff640 1 --2- 172.21.15.1:0/1790800972 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f411806b500 0x7f411806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.752+0000 7f412bfff640 1 --2- 172.21.15.1:0/1790800972 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f411806b500 0x7f411806d9c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f41140057e0 tx=0x7f411c03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.756+0000 7f4128ff9640 1 -- 172.21.15.1:0/1790800972 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4120024020 con 0x7f412c0ada30 2024-03-06T07:26:18.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:18.974+0000 7f4132a2a640 1 -- 172.21.15.1:0/1790800972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f412c0adea0 con 0x7f412c0ada30 2024-03-06T07:26:19.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.634+0000 7f4128ff9640 1 -- 172.21.15.1:0/1790800972 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v137) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4120059870 con 0x7f412c0ada30 2024-03-06T07:26:19.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.636+0000 7f4132a2a640 1 -- 172.21.15.1:0/1790800972 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f411806b500 msgr2=0x7f411806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.636+0000 7f4132a2a640 1 --2- 172.21.15.1:0/1790800972 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f411806b500 0x7f411806d9c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f41140057e0 tx=0x7f411c03d040 comp rx=0 tx=0).stop 2024-03-06T07:26:19.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.636+0000 7f4132a2a640 1 -- 172.21.15.1:0/1790800972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f412c0ada30 msgr2=0x7f412c0ac320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.636+0000 7f4132a2a640 1 --2- 172.21.15.1:0/1790800972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f412c0ada30 0x7f412c0ac320 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f4120010860 tx=0x7f4120010d20 comp rx=0 tx=0).stop 2024-03-06T07:26:19.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.636+0000 7f4132a2a640 1 -- 172.21.15.1:0/1790800972 shutdown_connections 2024-03-06T07:26:19.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.636+0000 7f4132a2a640 1 --2- 172.21.15.1:0/1790800972 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f411806b500 0x7f411806d9c0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.636+0000 7f4132a2a640 1 --2- 172.21.15.1:0/1790800972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f412c14f4b0 0x7f412c0ac860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.636+0000 7f4132a2a640 1 --2- 172.21.15.1:0/1790800972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f412c0ada30 0x7f412c0ac320 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.636+0000 7f4132a2a640 1 --2- 172.21.15.1:0/1790800972 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41140039b0 0x7f412c0707a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.636+0000 7f4132a2a640 1 -- 172.21.15.1:0/1790800972 >> 172.21.15.1:0/1790800972 conn(0x7f412c0c88f0 msgr2=0x7f412c06cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.636+0000 7f4132a2a640 1 -- 172.21.15.1:0/1790800972 shutdown_connections 2024-03-06T07:26:19.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.637+0000 7f4132a2a640 1 -- 172.21.15.1:0/1790800972 wait complete. 2024-03-06T07:26:19.649 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:26:19.650 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:19.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.749+0000 7f7405120640 1 Processor -- start 2024-03-06T07:26:19.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.749+0000 7f7405120640 1 -- start start 2024-03-06T07:26:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.750+0000 7f7405120640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7400152840 0x7f74000c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.750+0000 7f7405120640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74000c5e40 con 0x7f74000c60b0 2024-03-06T07:26:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.750+0000 7f7405120640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74000c6a40 con 0x7f74000c3970 2024-03-06T07:26:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.750+0000 7f7405120640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74000c67f0 con 0x7f7400152840 2024-03-06T07:26:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.750+0000 7f73ff59e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f74000c60b0 0x7f74001553d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52696/0 (socket says 172.21.15.1:52696) 2024-03-06T07:26:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.750+0000 7f73ff59e640 1 -- 172.21.15.1:0/4093961264 learned_addr learned my addr 172.21.15.1:0/4093961264 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:19.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fe59c640 1 --2- 172.21.15.1:0/4093961264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7400152840 0x7f74000c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fdd9b640 1 -- 172.21.15.1:0/4093961264 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137246037 0 0) 0x7f74000c5e40 con 0x7f74000c60b0 2024-03-06T07:26:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fdd9b640 1 -- 172.21.15.1:0/4093961264 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e8003520 con 0x7f74000c60b0 2024-03-06T07:26:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fdd9b640 1 -- 172.21.15.1:0/4093961264 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1848582239 0 0) 0x7f73e8003520 con 0x7f74000c60b0 2024-03-06T07:26:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fdd9b640 1 -- 172.21.15.1:0/4093961264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7400152840 msgr2=0x7f74000c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fdd9b640 1 --2- 172.21.15.1:0/4093961264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7400152840 0x7f74000c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fdd9b640 1 -- 172.21.15.1:0/4093961264 >> v1:172.21.15.98:6789/0 conn(0x7f74000c3970 legacy=0x7f7400152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fdd9b640 1 -- 172.21.15.1:0/4093961264 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74000c6d20 con 0x7f74000c60b0 2024-03-06T07:26:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fdd9b640 1 -- 172.21.15.1:0/4093961264 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f73f80019f0 con 0x7f74000c60b0 2024-03-06T07:26:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fdd9b640 1 -- 172.21.15.1:0/4093961264 >> v1:172.21.15.1:6789/0 conn(0x7f74000c60b0 legacy=0x7f74001553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fdd9b640 1 --2- 172.21.15.1:0/4093961264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73e80039e0 0x7f73e8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fdd9b640 1 --2- 172.21.15.1:0/4093961264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73e8004800 0x7f73e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fdd9b640 1 --2- 172.21.15.1:0/4093961264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7400152840 0x7f73e8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fdd9b640 1 -- 172.21.15.1:0/4093961264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74000c67f0 con 0x7f73e80039e0 2024-03-06T07:26:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fdd9b640 1 -- 172.21.15.1:0/4093961264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e800baf0 con 0x7f73e8004800 2024-03-06T07:26:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fdd9b640 1 -- 172.21.15.1:0/4093961264 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e800bcf0 con 0x7f7400152840 2024-03-06T07:26:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fed9d640 1 --2- 172.21.15.1:0/4093961264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73e80039e0 0x7f73e8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fe59c640 1 --2- 172.21.15.1:0/4093961264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73e8004800 0x7f73e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73ff59e640 1 --2- 172.21.15.1:0/4093961264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7400152840 0x7f73e8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fed9d640 1 -- 172.21.15.1:0/4093961264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7400152840 msgr2=0x7f73e8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fed9d640 1 --2- 172.21.15.1:0/4093961264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7400152840 0x7f73e8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fed9d640 1 -- 172.21.15.1:0/4093961264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73e8004800 msgr2=0x7f73e8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fed9d640 1 --2- 172.21.15.1:0/4093961264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73e8004800 0x7f73e8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.751+0000 7f73fed9d640 1 -- 172.21.15.1:0/4093961264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e8003520 con 0x7f73e80039e0 2024-03-06T07:26:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.752+0000 7f73fed9d640 1 --2- 172.21.15.1:0/4093961264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73e80039e0 0x7f73e8003e00 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f73f0001750 tx=0x7f73e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b8dd26ca9ae0f6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.752+0000 7f73fdd9b640 1 -- 172.21.15.1:0/4093961264 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f0001370 con 0x7f73e80039e0 2024-03-06T07:26:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.752+0000 7f73fdd9b640 1 -- 172.21.15.1:0/4093961264 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f00052d0 con 0x7f73e80039e0 2024-03-06T07:26:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.752+0000 7f73fdd9b640 1 -- 172.21.15.1:0/4093961264 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f0003830 con 0x7f73e80039e0 2024-03-06T07:26:19.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.753+0000 7f7405120640 1 -- 172.21.15.1:0/4093961264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73e80039e0 msgr2=0x7f73e8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.753+0000 7f7405120640 1 --2- 172.21.15.1:0/4093961264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73e80039e0 0x7f73e8003e00 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f73f0001750 tx=0x7f73e8005830 comp rx=0 tx=0).stop 2024-03-06T07:26:19.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.753+0000 7f73ff59e640 1 -- 172.21.15.1:0/4093961264 reap_dead start 2024-03-06T07:26:19.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.753+0000 7f7405120640 1 -- 172.21.15.1:0/4093961264 shutdown_connections 2024-03-06T07:26:19.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.753+0000 7f7405120640 1 -- 172.21.15.1:0/4093961264 >> 172.21.15.1:0/4093961264 conn(0x7f74000c88f0 msgr2=0x7f74000c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.753+0000 7f7405120640 1 -- 172.21.15.1:0/4093961264 shutdown_connections 2024-03-06T07:26:19.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.753+0000 7f7405120640 1 -- 172.21.15.1:0/4093961264 wait complete. 2024-03-06T07:26:19.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.755+0000 7f7405120640 1 Processor -- start 2024-03-06T07:26:19.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.755+0000 7f7405120640 1 -- start start 2024-03-06T07:26:19.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.756+0000 7f7405120640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73e80039e0 0x7f7400070c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.756+0000 7f7405120640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f740014d310 0x7f74000ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.756+0000 7f7405120640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f740006fd80 0x7f7400070230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.756+0000 7f7405120640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74000c75c0 con 0x7f740006fd80 2024-03-06T07:26:19.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.756+0000 7f7405120640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74000c72e0 con 0x7f73e80039e0 2024-03-06T07:26:19.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.756+0000 7f7405120640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74000c3d50 con 0x7f740014d310 2024-03-06T07:26:19.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.756+0000 7f73ff59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f740006fd80 0x7f7400070230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.756+0000 7f73ff59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f740006fd80 0x7f7400070230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44642/0 (socket says 172.21.15.1:44642) 2024-03-06T07:26:19.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.756+0000 7f73ff59e640 1 -- 172.21.15.1:0/3445907359 learned_addr learned my addr 172.21.15.1:0/3445907359 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:19.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.757+0000 7f73fed9d640 1 --2- 172.21.15.1:0/3445907359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73e80039e0 0x7f7400070c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.757+0000 7f73ff59e640 1 -- 172.21.15.1:0/3445907359 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f740014d310 msgr2=0x7f74000ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.757+0000 7f73fe59c640 1 --2- 172.21.15.1:0/3445907359 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f740014d310 0x7f74000ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.757+0000 7f73ff59e640 1 --2- 172.21.15.1:0/3445907359 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f740014d310 0x7f74000ad2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.757+0000 7f73ff59e640 1 -- 172.21.15.1:0/3445907359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73e80039e0 msgr2=0x7f7400070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.757+0000 7f73ff59e640 1 --2- 172.21.15.1:0/3445907359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73e80039e0 0x7f7400070c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.757+0000 7f73ff59e640 1 -- 172.21.15.1:0/3445907359 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740014c6b0 con 0x7f740006fd80 2024-03-06T07:26:19.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.757+0000 7f73fed9d640 1 --2- 172.21.15.1:0/3445907359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73e80039e0 0x7f7400070c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:19.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.757+0000 7f73ff59e640 1 --2- 172.21.15.1:0/3445907359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f740006fd80 0x7f7400070230 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f73f8000f80 tx=0x7f73f8007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.757+0000 7f73e77fe640 1 -- 172.21.15.1:0/3445907359 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f8017020 con 0x7f740006fd80 2024-03-06T07:26:19.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.757+0000 7f73e77fe640 1 -- 172.21.15.1:0/3445907359 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f800ab80 con 0x7f740006fd80 2024-03-06T07:26:19.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.757+0000 7f7405120640 1 -- 172.21.15.1:0/3445907359 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f740014c990 con 0x7f740006fd80 2024-03-06T07:26:19.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.757+0000 7f73e77fe640 1 -- 172.21.15.1:0/3445907359 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f80133f0 con 0x7f740006fd80 2024-03-06T07:26:19.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.757+0000 7f7405120640 1 -- 172.21.15.1:0/3445907359 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f740006f580 con 0x7f740006fd80 2024-03-06T07:26:19.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.758+0000 7f7405120640 1 -- 172.21.15.1:0/3445907359 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74000c7f30 con 0x7f740006fd80 2024-03-06T07:26:19.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.759+0000 7f73e77fe640 1 -- 172.21.15.1:0/3445907359 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73f801a020 con 0x7f740006fd80 2024-03-06T07:26:19.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.759+0000 7f73e77fe640 1 --2- 172.21.15.1:0/3445907359 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73ec06b500 0x7f73ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.759+0000 7f73e77fe640 1 -- 172.21.15.1:0/3445907359 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f73f808aa80 con 0x7f740006fd80 2024-03-06T07:26:19.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.759+0000 7f73fed9d640 1 --2- 172.21.15.1:0/3445907359 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73ec06b500 0x7f73ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.760+0000 7f73fed9d640 1 --2- 172.21.15.1:0/3445907359 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73ec06b500 0x7f73ec06d9c0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f73f0033000 tx=0x7f73f0009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.764+0000 7f73e77fe640 1 -- 172.21.15.1:0/3445907359 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74000c7f30 con 0x7f740006fd80 2024-03-06T07:26:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.986+0000 7f7405120640 1 -- 172.21.15.1:0/3445907359 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74000adea0 con 0x7f740006fd80 2024-03-06T07:26:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.987+0000 7f73e77fe640 1 -- 172.21.15.1:0/3445907359 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+4975 (secure 0 0 0) 0x7f73f8021020 con 0x7f740006fd80 2024-03-06T07:26:19.989 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:19.989 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":137,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6989,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2360179300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2360179300},{"type":"v1","addr":"172.21.15.98:6837","nonce":2360179300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":7774,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3360241462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3360241462},{"type":"v1","addr":"172.21.15.98:6836","nonce":3360241462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":8820,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/1089548586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1089548586},{"type":"v1","addr":"172.21.15.1:6833","nonce":1089548586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:16.977128+0000","modified":"2024-03-06T07:26:18.975595+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8823},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8823":{"gid":8823,"name":"a","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/67091521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":67091521},{"type":"v1","addr":"172.21.15.1:6835","nonce":67091521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8823,"qdb_cluster":[8823]},"id":11}]} 2024-03-06T07:26:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 137 2024-03-06T07:26:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.989+0000 7f7405120640 1 -- 172.21.15.1:0/3445907359 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73ec06b500 msgr2=0x7f73ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.989+0000 7f7405120640 1 --2- 172.21.15.1:0/3445907359 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73ec06b500 0x7f73ec06d9c0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f73f0033000 tx=0x7f73f0009030 comp rx=0 tx=0).stop 2024-03-06T07:26:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.989+0000 7f7405120640 1 -- 172.21.15.1:0/3445907359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f740006fd80 msgr2=0x7f7400070230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.989+0000 7f7405120640 1 --2- 172.21.15.1:0/3445907359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f740006fd80 0x7f7400070230 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f73f8000f80 tx=0x7f73f8007b20 comp rx=0 tx=0).stop 2024-03-06T07:26:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.989+0000 7f7405120640 1 -- 172.21.15.1:0/3445907359 shutdown_connections 2024-03-06T07:26:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.989+0000 7f7405120640 1 --2- 172.21.15.1:0/3445907359 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f73ec06b500 0x7f73ec06d9c0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.989+0000 7f7405120640 1 --2- 172.21.15.1:0/3445907359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f740006fd80 0x7f7400070230 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.989+0000 7f7405120640 1 --2- 172.21.15.1:0/3445907359 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f740014d310 0x7f74000ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.989+0000 7f7405120640 1 --2- 172.21.15.1:0/3445907359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73e80039e0 0x7f7400070c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.989+0000 7f7405120640 1 -- 172.21.15.1:0/3445907359 >> 172.21.15.1:0/3445907359 conn(0x7f74000c88f0 msgr2=0x7f740006caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.989+0000 7f7405120640 1 -- 172.21.15.1:0/3445907359 shutdown_connections 2024-03-06T07:26:19.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:19.989+0000 7f7405120640 1 -- 172.21.15.1:0/3445907359 wait complete. 2024-03-06T07:26:20.002 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:20.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.102+0000 7f30d9abb640 1 Processor -- start 2024-03-06T07:26:20.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.102+0000 7f30d9abb640 1 -- start start 2024-03-06T07:26:20.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.103+0000 7f30d9abb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f30d4152840 0x7f30d40c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.103+0000 7f30d9abb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d40c60b0 0x7f30d41553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.103+0000 7f30d9abb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d40c5e40 con 0x7f30d40c3970 2024-03-06T07:26:20.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.103+0000 7f30d9abb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30d40c67f0 con 0x7f30d4152840 2024-03-06T07:26:20.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.103+0000 7f30d9abb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d40c6a40 con 0x7f30d40c60b0 2024-03-06T07:26:20.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.103+0000 7f30d2ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f30d4152840 0x7f30d40c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.103+0000 7f30d2ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f30d4152840 0x7f30d40c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44658/0 (socket says 172.21.15.1:44658) 2024-03-06T07:26:20.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.103+0000 7f30d2ffd640 1 -- 172.21.15.1:0/1443736082 learned_addr learned my addr 172.21.15.1:0/1443736082 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:20.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.103+0000 7f30d3fff640 1 --2- 172.21.15.1:0/1443736082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d40c60b0 0x7f30d41553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.103+0000 7f30d2ffd640 1 -- 172.21.15.1:0/1443736082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d40c60b0 msgr2=0x7f30d41553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.103+0000 7f30d2ffd640 1 --2- 172.21.15.1:0/1443736082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d40c60b0 0x7f30d41553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.103+0000 7f30d2ffd640 1 -- 172.21.15.1:0/1443736082 >> v1:172.21.15.98:6789/0 conn(0x7f30d40c3970 legacy=0x7f30d4152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.103+0000 7f30d2ffd640 1 -- 172.21.15.1:0/1443736082 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d40c6d20 con 0x7f30d4152840 2024-03-06T07:26:20.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.104+0000 7f30d2ffd640 1 --2- 172.21.15.1:0/1443736082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d4152840 0x7f30d40c8170 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f30c4009870 tx=0x7f30c402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bbe794d7d231d9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.104+0000 7f30d27fc640 1 -- 172.21.15.1:0/1443736082 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c4034070 con 0x7f30d4152840 2024-03-06T07:26:20.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.104+0000 7f30d27fc640 1 -- 172.21.15.1:0/1443736082 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c4030840 con 0x7f30d4152840 2024-03-06T07:26:20.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.104+0000 7f30d27fc640 1 -- 172.21.15.1:0/1443736082 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c4030c10 con 0x7f30d4152840 2024-03-06T07:26:20.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.105+0000 7f30d9abb640 1 -- 172.21.15.1:0/1443736082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d4152840 msgr2=0x7f30d40c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.105+0000 7f30d9abb640 1 --2- 172.21.15.1:0/1443736082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d4152840 0x7f30d40c8170 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f30c4009870 tx=0x7f30c402b980 comp rx=0 tx=0).stop 2024-03-06T07:26:20.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.105+0000 7f30d9abb640 1 -- 172.21.15.1:0/1443736082 shutdown_connections 2024-03-06T07:26:20.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.105+0000 7f30d9abb640 1 --2- 172.21.15.1:0/1443736082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d40c60b0 0x7f30d41553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.105+0000 7f30d9abb640 1 --2- 172.21.15.1:0/1443736082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d4152840 0x7f30d40c8170 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.105+0000 7f30d9abb640 1 -- 172.21.15.1:0/1443736082 >> 172.21.15.1:0/1443736082 conn(0x7f30d40c88f0 msgr2=0x7f30d40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.105+0000 7f30d9abb640 1 -- 172.21.15.1:0/1443736082 shutdown_connections 2024-03-06T07:26:20.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.105+0000 7f30d9abb640 1 -- 172.21.15.1:0/1443736082 wait complete. 2024-03-06T07:26:20.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.106+0000 7f30d9abb640 1 Processor -- start 2024-03-06T07:26:20.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.106+0000 7f30d9abb640 1 -- start start 2024-03-06T07:26:20.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.106+0000 7f30d9abb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30d40c3970 0x7f30d4150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.106+0000 7f30d9abb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d40c60b0 0x7f30d4156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30d9abb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d4152840 0x7f30d41572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30d9abb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d40c75c0 con 0x7f30d40c60b0 2024-03-06T07:26:20.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30d9abb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d40c72e0 con 0x7f30d40c3970 2024-03-06T07:26:20.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30d9abb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d40c3d50 con 0x7f30d4152840 2024-03-06T07:26:20.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30d2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d40c60b0 0x7f30d4156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30d2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d40c60b0 0x7f30d4156d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44660/0 (socket says 172.21.15.1:44660) 2024-03-06T07:26:20.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30d2ffd640 1 -- 172.21.15.1:0/1869541240 learned_addr learned my addr 172.21.15.1:0/1869541240 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:20.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30d37fe640 1 --2- 172.21.15.1:0/1869541240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30d40c3970 0x7f30d4150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30d3fff640 1 --2- 172.21.15.1:0/1869541240 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d4152840 0x7f30d41572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30d2ffd640 1 -- 172.21.15.1:0/1869541240 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d4152840 msgr2=0x7f30d41572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30d2ffd640 1 --2- 172.21.15.1:0/1869541240 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d4152840 0x7f30d41572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30d2ffd640 1 -- 172.21.15.1:0/1869541240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30d40c3970 msgr2=0x7f30d4150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30d2ffd640 1 --2- 172.21.15.1:0/1869541240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30d40c3970 0x7f30d4150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30d2ffd640 1 -- 172.21.15.1:0/1869541240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d4070c50 con 0x7f30d40c60b0 2024-03-06T07:26:20.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30d2ffd640 1 --2- 172.21.15.1:0/1869541240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d40c60b0 0x7f30d4156d90 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f30c4000c00 tx=0x7f30c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30bbfff640 1 -- 172.21.15.1:0/1869541240 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c4034040 con 0x7f30d40c60b0 2024-03-06T07:26:20.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30bbfff640 1 -- 172.21.15.1:0/1869541240 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c4039d90 con 0x7f30d40c60b0 2024-03-06T07:26:20.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30bbfff640 1 -- 172.21.15.1:0/1869541240 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c4043610 con 0x7f30d40c60b0 2024-03-06T07:26:20.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.107+0000 7f30d9abb640 1 -- 172.21.15.1:0/1869541240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d4070f30 con 0x7f30d40c60b0 2024-03-06T07:26:20.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.108+0000 7f30d9abb640 1 -- 172.21.15.1:0/1869541240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d4157d50 con 0x7f30d40c60b0 2024-03-06T07:26:20.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.109+0000 7f30d9abb640 1 -- 172.21.15.1:0/1869541240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d40686e0 con 0x7f30d40c60b0 2024-03-06T07:26:20.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.109+0000 7f30bbfff640 1 -- 172.21.15.1:0/1869541240 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f30c404a060 con 0x7f30d40c60b0 2024-03-06T07:26:20.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.110+0000 7f30bbfff640 1 --2- 172.21.15.1:0/1869541240 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30c006b4d0 0x7f30c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.110+0000 7f30bbfff640 1 -- 172.21.15.1:0/1869541240 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f30c40ac190 con 0x7f30d40c60b0 2024-03-06T07:26:20.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.110+0000 7f30d37fe640 1 --2- 172.21.15.1:0/1869541240 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30c006b4d0 0x7f30c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.111+0000 7f30d37fe640 1 --2- 172.21.15.1:0/1869541240 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30c006b4d0 0x7f30c006d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f30c8001780 tx=0x7f30c8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.115+0000 7f30bbfff640 1 -- 172.21.15.1:0/1869541240 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30c403e070 con 0x7f30d40c60b0 2024-03-06T07:26:20.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.297+0000 7f30d9abb640 1 -- 172.21.15.1:0/1869541240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30d4158f50 con 0x7f30d40c60b0 2024-03-06T07:26:20.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.298+0000 7f30bbfff640 1 -- 172.21.15.1:0/1869541240 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+16764 (secure 0 0 0) 0x7f30c407a8c0 con 0x7f30d40c60b0 2024-03-06T07:26:20.300 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:20.301 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":149,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:26:18.622325+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:15.245426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:15.942343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:20.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.301+0000 7f30d9abb640 1 -- 172.21.15.1:0/1869541240 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30c006b4d0 msgr2=0x7f30c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.301+0000 7f30d9abb640 1 --2- 172.21.15.1:0/1869541240 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30c006b4d0 0x7f30c006d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f30c8001780 tx=0x7f30c8009040 comp rx=0 tx=0).stop 2024-03-06T07:26:20.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.301+0000 7f30d9abb640 1 -- 172.21.15.1:0/1869541240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d40c60b0 msgr2=0x7f30d4156d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.301+0000 7f30d9abb640 1 --2- 172.21.15.1:0/1869541240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d40c60b0 0x7f30d4156d90 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f30c4000c00 tx=0x7f30c40046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.301+0000 7f30d9abb640 1 -- 172.21.15.1:0/1869541240 shutdown_connections 2024-03-06T07:26:20.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.301+0000 7f30d9abb640 1 --2- 172.21.15.1:0/1869541240 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f30c006b4d0 0x7f30c006d990 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.301+0000 7f30d9abb640 1 --2- 172.21.15.1:0/1869541240 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30d4152840 0x7f30d41572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.301+0000 7f30d9abb640 1 --2- 172.21.15.1:0/1869541240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30d40c60b0 0x7f30d4156d90 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.301+0000 7f30d9abb640 1 --2- 172.21.15.1:0/1869541240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30d40c3970 0x7f30d4150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.301+0000 7f30d9abb640 1 -- 172.21.15.1:0/1869541240 >> 172.21.15.1:0/1869541240 conn(0x7f30d40c88f0 msgr2=0x7f30d40c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.301+0000 7f30d9abb640 1 -- 172.21.15.1:0/1869541240 shutdown_connections 2024-03-06T07:26:20.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.301+0000 7f30d9abb640 1 -- 172.21.15.1:0/1869541240 wait complete. 2024-03-06T07:26:20.317 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:26:20.318 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:20.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.417+0000 7f031c51e640 1 Processor -- start 2024-03-06T07:26:20.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.417+0000 7f031c51e640 1 -- start start 2024-03-06T07:26:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.418+0000 7f031c51e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03140c6b10 0x7f0314155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.418+0000 7f031c51e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03140c2fc0 con 0x7f0314152650 2024-03-06T07:26:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.418+0000 7f031c51e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0314158da0 con 0x7f03140bf9d0 2024-03-06T07:26:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.418+0000 7f031c51e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0314155800 con 0x7f03140c6b10 2024-03-06T07:26:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.418+0000 7f031a2ba640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0314152650 0x7f0314158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52702/0 (socket says 172.21.15.1:52702) 2024-03-06T07:26:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.418+0000 7f031a2ba640 1 -- 172.21.15.1:0/2374741414 learned_addr learned my addr 172.21.15.1:0/2374741414 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f0319ab9640 1 --2- 172.21.15.1:0/2374741414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03140c6b10 0x7f0314155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f03192b8640 1 -- 172.21.15.1:0/2374741414 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233454231 0 0) 0x7f03140c2fc0 con 0x7f0314152650 2024-03-06T07:26:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f03192b8640 1 -- 172.21.15.1:0/2374741414 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0300003520 con 0x7f0314152650 2024-03-06T07:26:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f03192b8640 1 -- 172.21.15.1:0/2374741414 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060638325 0 0) 0x7f0314158da0 con 0x7f03140bf9d0 2024-03-06T07:26:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f03192b8640 1 -- 172.21.15.1:0/2374741414 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03140c2fc0 con 0x7f03140bf9d0 2024-03-06T07:26:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f03192b8640 1 -- 172.21.15.1:0/2374741414 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 650765177 0 0) 0x7f0300003520 con 0x7f0314152650 2024-03-06T07:26:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f03192b8640 1 -- 172.21.15.1:0/2374741414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03140c6b10 msgr2=0x7f0314155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f03192b8640 1 --2- 172.21.15.1:0/2374741414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03140c6b10 0x7f0314155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f03192b8640 1 -- 172.21.15.1:0/2374741414 >> v1:172.21.15.98:6789/0 conn(0x7f03140bf9d0 legacy=0x7f03140c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f03192b8640 1 -- 172.21.15.1:0/2374741414 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03140c6690 con 0x7f0314152650 2024-03-06T07:26:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f03192b8640 1 -- 172.21.15.1:0/2374741414 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f0304001670 con 0x7f0314152650 2024-03-06T07:26:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f03192b8640 1 -- 172.21.15.1:0/2374741414 >> v1:172.21.15.1:6789/0 conn(0x7f0314152650 legacy=0x7f0314158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f03192b8640 1 --2- 172.21.15.1:0/2374741414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0300003b70 0x7f0300003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f03192b8640 1 --2- 172.21.15.1:0/2374741414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0300004800 0x7f0300004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f03192b8640 1 --2- 172.21.15.1:0/2374741414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03000051c0 0x7f030000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f03192b8640 1 -- 172.21.15.1:0/2374741414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0314155800 con 0x7f0300003b70 2024-03-06T07:26:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f03192b8640 1 -- 172.21.15.1:0/2374741414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030000c030 con 0x7f03000051c0 2024-03-06T07:26:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f03192b8640 1 -- 172.21.15.1:0/2374741414 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030000c230 con 0x7f0300004800 2024-03-06T07:26:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f0319ab9640 1 --2- 172.21.15.1:0/2374741414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03140c6b10 0x7f0314155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f031a2ba640 1 --2- 172.21.15.1:0/2374741414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0300003b70 0x7f0300003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f031aabb640 1 --2- 172.21.15.1:0/2374741414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03000051c0 0x7f030000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f0319ab9640 1 --2- 172.21.15.1:0/2374741414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0300004800 0x7f0300004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f031a2ba640 1 -- 172.21.15.1:0/2374741414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0300004800 msgr2=0x7f0300004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f031a2ba640 1 --2- 172.21.15.1:0/2374741414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0300004800 0x7f0300004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f031a2ba640 1 -- 172.21.15.1:0/2374741414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03000051c0 msgr2=0x7f030000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f031a2ba640 1 --2- 172.21.15.1:0/2374741414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03000051c0 0x7f030000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f031a2ba640 1 -- 172.21.15.1:0/2374741414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0300003520 con 0x7f0300003b70 2024-03-06T07:26:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.419+0000 7f031aabb640 1 --2- 172.21.15.1:0/2374741414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03000051c0 0x7f030000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.420+0000 7f031a2ba640 1 --2- 172.21.15.1:0/2374741414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0300003b70 0x7f0300003f70 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f030400c210 tx=0x7f0300005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56cd423fa03d047d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.420+0000 7f03192b8640 1 -- 172.21.15.1:0/2374741414 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0304007ce0 con 0x7f0300003b70 2024-03-06T07:26:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.420+0000 7f03192b8640 1 -- 172.21.15.1:0/2374741414 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0304030070 con 0x7f0300003b70 2024-03-06T07:26:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.420+0000 7f03192b8640 1 -- 172.21.15.1:0/2374741414 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0304005d50 con 0x7f0300003b70 2024-03-06T07:26:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.421+0000 7f031c51e640 1 -- 172.21.15.1:0/2374741414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0300003b70 msgr2=0x7f0300003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.421+0000 7f031c51e640 1 --2- 172.21.15.1:0/2374741414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0300003b70 0x7f0300003f70 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f030400c210 tx=0x7f0300005830 comp rx=0 tx=0).stop 2024-03-06T07:26:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.421+0000 7f031aabb640 1 -- 172.21.15.1:0/2374741414 reap_dead start 2024-03-06T07:26:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.421+0000 7f031c51e640 1 -- 172.21.15.1:0/2374741414 shutdown_connections 2024-03-06T07:26:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.421+0000 7f031c51e640 1 -- 172.21.15.1:0/2374741414 >> 172.21.15.1:0/2374741414 conn(0x7f0314066ee0 msgr2=0x7f03140672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.421+0000 7f031c51e640 1 -- 172.21.15.1:0/2374741414 shutdown_connections 2024-03-06T07:26:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.421+0000 7f031c51e640 1 -- 172.21.15.1:0/2374741414 wait complete. 2024-03-06T07:26:20.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.422+0000 7f031c51e640 1 Processor -- start 2024-03-06T07:26:20.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.422+0000 7f031c51e640 1 -- start start 2024-03-06T07:26:20.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.423+0000 7f031c51e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0300003b70 0x7f03140c9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.423+0000 7f031c51e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0314158500 0x7f0314150fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.423+0000 7f031c51e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f031414b270 0x7f031414b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.423+0000 7f031c51e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03141551b0 con 0x7f0314158500 2024-03-06T07:26:20.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.423+0000 7f031c51e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03140c6fa0 con 0x7f0300003b70 2024-03-06T07:26:20.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.423+0000 7f031c51e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0314155490 con 0x7f031414b270 2024-03-06T07:26:20.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.423+0000 7f0319ab9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0314158500 0x7f0314150fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.423+0000 7f0319ab9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0314158500 0x7f0314150fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44668/0 (socket says 172.21.15.1:44668) 2024-03-06T07:26:20.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.423+0000 7f0319ab9640 1 -- 172.21.15.1:0/4222213846 learned_addr learned my addr 172.21.15.1:0/4222213846 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:20.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.423+0000 7f031a2ba640 1 --2- 172.21.15.1:0/4222213846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0300003b70 0x7f03140c9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.423+0000 7f031aabb640 1 --2- 172.21.15.1:0/4222213846 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f031414b270 0x7f031414b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.423+0000 7f0319ab9640 1 -- 172.21.15.1:0/4222213846 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f031414b270 msgr2=0x7f031414b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.423+0000 7f0319ab9640 1 --2- 172.21.15.1:0/4222213846 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f031414b270 0x7f031414b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.423+0000 7f0319ab9640 1 -- 172.21.15.1:0/4222213846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0300003b70 msgr2=0x7f03140c9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.423+0000 7f0319ab9640 1 --2- 172.21.15.1:0/4222213846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0300003b70 0x7f03140c9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.423+0000 7f0319ab9640 1 -- 172.21.15.1:0/4222213846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0314152ce0 con 0x7f0314158500 2024-03-06T07:26:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.423+0000 7f0319ab9640 1 --2- 172.21.15.1:0/4222213846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0314158500 0x7f0314150fc0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f03100115d0 tx=0x7f0310011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.424+0000 7f030affd640 1 -- 172.21.15.1:0/4222213846 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031000f8f0 con 0x7f0314158500 2024-03-06T07:26:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.424+0000 7f030affd640 1 -- 172.21.15.1:0/4222213846 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f031000fa90 con 0x7f0314158500 2024-03-06T07:26:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.424+0000 7f031c51e640 1 -- 172.21.15.1:0/4222213846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0314152f80 con 0x7f0314158500 2024-03-06T07:26:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.424+0000 7f030affd640 1 -- 172.21.15.1:0/4222213846 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031000fe80 con 0x7f0314158500 2024-03-06T07:26:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.424+0000 7f031c51e640 1 -- 172.21.15.1:0/4222213846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0314153540 con 0x7f0314158500 2024-03-06T07:26:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.425+0000 7f031c51e640 1 -- 172.21.15.1:0/4222213846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0314067fd0 con 0x7f0314158500 2024-03-06T07:26:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.425+0000 7f030affd640 1 -- 172.21.15.1:0/4222213846 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0310015460 con 0x7f0314158500 2024-03-06T07:26:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.426+0000 7f030affd640 1 --2- 172.21.15.1:0/4222213846 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02f806b4d0 0x7f02f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.426+0000 7f030affd640 1 -- 172.21.15.1:0/4222213846 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f031008ac40 con 0x7f0314158500 2024-03-06T07:26:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.426+0000 7f031a2ba640 1 --2- 172.21.15.1:0/4222213846 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02f806b4d0 0x7f02f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.426+0000 7f031a2ba640 1 --2- 172.21.15.1:0/4222213846 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02f806b4d0 0x7f02f806d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0304034000 tx=0x7f0304007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.430+0000 7f030affd640 1 -- 172.21.15.1:0/4222213846 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0310019030 con 0x7f0314158500 2024-03-06T07:26:20.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.612+0000 7f031c51e640 1 -- 172.21.15.1:0/4222213846 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f03140b06f0 con 0x7f02f806b4d0 2024-03-06T07:26:20.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.618+0000 7f030affd640 1 -- 172.21.15.1:0/4222213846 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237318 (secure 0 0 0) 0x7f030407ef20 con 0x7f02f806b4d0 2024-03-06T07:26:20.623 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:20.624 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:26:20.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.620+0000 7f031c51e640 1 -- 172.21.15.1:0/4222213846 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02f806b4d0 msgr2=0x7f02f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.620+0000 7f031c51e640 1 --2- 172.21.15.1:0/4222213846 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02f806b4d0 0x7f02f806d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0304034000 tx=0x7f0304007000 comp rx=0 tx=0).stop 2024-03-06T07:26:20.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.621+0000 7f031c51e640 1 -- 172.21.15.1:0/4222213846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0314158500 msgr2=0x7f0314150fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.621+0000 7f031c51e640 1 --2- 172.21.15.1:0/4222213846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0314158500 0x7f0314150fc0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f03100115d0 tx=0x7f0310011a90 comp rx=0 tx=0).stop 2024-03-06T07:26:20.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.621+0000 7f031c51e640 1 -- 172.21.15.1:0/4222213846 shutdown_connections 2024-03-06T07:26:20.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.621+0000 7f031c51e640 1 --2- 172.21.15.1:0/4222213846 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02f806b4d0 0x7f02f806d990 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.621+0000 7f031c51e640 1 --2- 172.21.15.1:0/4222213846 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f031414b270 0x7f031414b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.621+0000 7f031c51e640 1 --2- 172.21.15.1:0/4222213846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0314158500 0x7f0314150fc0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.621+0000 7f031c51e640 1 --2- 172.21.15.1:0/4222213846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0300003b70 0x7f03140c9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.621+0000 7f031c51e640 1 -- 172.21.15.1:0/4222213846 >> 172.21.15.1:0/4222213846 conn(0x7f0314066ee0 msgr2=0x7f03140c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.621+0000 7f031c51e640 1 -- 172.21.15.1:0/4222213846 shutdown_connections 2024-03-06T07:26:20.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.621+0000 7f031c51e640 1 -- 172.21.15.1:0/4222213846 wait complete. 2024-03-06T07:26:20.634 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-03-06T07:26:18.772238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":263876,"kb_used_data":14400,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710844,"statfs":{"total":773094113280,"available":772823904256,"internally_reserved":0,"allocated":14745600,"data_stored":8214664,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000922"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.599836+0000","last_change":"2024-03-06T07:26:16.599836+0000","last_active":"2024-03-06T07:26:16.599836+0000","last_peered":"2024-03-06T07:26:16.599836+0000","last_clean":"2024-03-06T07:26:16.599836+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.599836+0000","last_undegraded":"2024-03-06T07:26:16.599836+0000","last_fullsized":"2024-03-06T07:26:16.599836+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:15.595702+0000","last_change":"2024-03-06T07:26:15.595702+0000","last_active":"2024-03-06T07:26:15.595702+0000","last_peered":"2024-03-06T07:26:15.595702+0000","last_clean":"2024-03-06T07:26:15.595702+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.595702+0000","last_undegraded":"2024-03-06T07:26:15.595702+0000","last_fullsized":"2024-03-06T07:26:15.595702+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":408,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:26:13.903593+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:26:13.903593+0000","last_peered":"2024-03-06T07:26:13.903593+0000","last_clean":"2024-03-06T07:26:13.903593+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:26:13.903593+0000","last_undegraded":"2024-03-06T07:26:13.903593+0000","last_fullsized":"2024-03-06T07:26:13.903593+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1892352,"data_stored":1469609,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32964,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338876,"statfs":{"total":96636764160,"available":96603009024,"internally_reserved":0,"allocated":1826816,"data_stored":879241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":879241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32964,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338876,"statfs":{"total":96636764160,"available":96603009024,"internally_reserved":0,"allocated":1826816,"data_stored":879241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":879241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32976,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1826816,"data_stored":879241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32964,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338876,"statfs":{"total":96636764160,"available":96603009024,"internally_reserved":0,"allocated":1826816,"data_stored":879241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33036,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338804,"statfs":{"total":96636764160,"available":96602935296,"internally_reserved":0,"allocated":1892352,"data_stored":1469609,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:20.638 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:20.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.773+0000 7f4743e9d640 1 Processor -- start 2024-03-06T07:26:20.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.773+0000 7f4743e9d640 1 -- start start 2024-03-06T07:26:20.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.775+0000 7f4743e9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f473c152650 0x7f473c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.775+0000 7f4743e9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f473c0c6b10 0x7f473c158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.775+0000 7f4743e9d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f473c0c2fc0 con 0x7f473c0bf9d0 2024-03-06T07:26:20.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.775+0000 7f4743e9d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f473c155800 con 0x7f473c152650 2024-03-06T07:26:20.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.775+0000 7f4743e9d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f473c0c6690 con 0x7f473c0c6b10 2024-03-06T07:26:20.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.775+0000 7f4741c39640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f473c152650 0x7f473c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.775+0000 7f474243a640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f473c0bf9d0 0x7f473c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52718/0 (socket says 172.21.15.1:52718) 2024-03-06T07:26:20.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.775+0000 7f474243a640 1 -- 172.21.15.1:0/1444141998 learned_addr learned my addr 172.21.15.1:0/1444141998 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:20.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.775+0000 7f4741c39640 1 -- 172.21.15.1:0/1444141998 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f473c0c6b10 msgr2=0x7f473c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.775+0000 7f4741438640 1 --2- 172.21.15.1:0/1444141998 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f473c0c6b10 0x7f473c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.775+0000 7f4740c37640 1 -- 172.21.15.1:0/1444141998 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382328149 0 0) 0x7f473c0c2fc0 con 0x7f473c0bf9d0 2024-03-06T07:26:20.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.775+0000 7f4741c39640 1 --2- 172.21.15.1:0/1444141998 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f473c0c6b10 0x7f473c158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.775+0000 7f4741c39640 1 -- 172.21.15.1:0/1444141998 >> v1:172.21.15.1:6789/0 conn(0x7f473c0bf9d0 legacy=0x7f473c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.775+0000 7f4741c39640 1 -- 172.21.15.1:0/1444141998 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473c0c7280 con 0x7f473c152650 2024-03-06T07:26:20.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.775+0000 7f4741438640 1 --2- 172.21.15.1:0/1444141998 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f473c0c6b10 0x7f473c158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:20.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.776+0000 7f4741c39640 1 --2- 172.21.15.1:0/1444141998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f473c152650 0x7f473c155c90 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f47300097a0 tx=0x7f473002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc46ff0541095d00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.776+0000 7f4740c37640 1 -- 172.21.15.1:0/1444141998 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4730031780 con 0x7f473c152650 2024-03-06T07:26:20.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.776+0000 7f4740c37640 1 -- 172.21.15.1:0/1444141998 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4730037070 con 0x7f473c152650 2024-03-06T07:26:20.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.776+0000 7f4740c37640 1 -- 172.21.15.1:0/1444141998 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4730031b40 con 0x7f473c152650 2024-03-06T07:26:20.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.777+0000 7f4743e9d640 1 -- 172.21.15.1:0/1444141998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f473c152650 msgr2=0x7f473c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.777+0000 7f4743e9d640 1 --2- 172.21.15.1:0/1444141998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f473c152650 0x7f473c155c90 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f47300097a0 tx=0x7f473002b820 comp rx=0 tx=0).stop 2024-03-06T07:26:20.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.777+0000 7f4743e9d640 1 -- 172.21.15.1:0/1444141998 shutdown_connections 2024-03-06T07:26:20.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.777+0000 7f4743e9d640 1 --2- 172.21.15.1:0/1444141998 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f473c0c6b10 0x7f473c158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.777+0000 7f4743e9d640 1 --2- 172.21.15.1:0/1444141998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f473c152650 0x7f473c155c90 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.777+0000 7f4743e9d640 1 -- 172.21.15.1:0/1444141998 >> 172.21.15.1:0/1444141998 conn(0x7f473c066ee0 msgr2=0x7f473c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.777+0000 7f4743e9d640 1 -- 172.21.15.1:0/1444141998 shutdown_connections 2024-03-06T07:26:20.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.777+0000 7f4743e9d640 1 -- 172.21.15.1:0/1444141998 wait complete. 2024-03-06T07:26:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.778+0000 7f4743e9d640 1 Processor -- start 2024-03-06T07:26:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.778+0000 7f4743e9d640 1 -- start start 2024-03-06T07:26:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4743e9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f473c0bf9d0 0x7f473c0c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4743e9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f473c152650 0x7f473c144dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4743e9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f473c1493c0 0x7f473c145300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4743e9d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c155490 con 0x7f473c0bf9d0 2024-03-06T07:26:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4743e9d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c1551b0 con 0x7f473c1493c0 2024-03-06T07:26:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4743e9d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f473c152a30 con 0x7f473c152650 2024-03-06T07:26:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4741c39640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f473c0bf9d0 0x7f473c0c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4741c39640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f473c0bf9d0 0x7f473c0c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44684/0 (socket says 172.21.15.1:44684) 2024-03-06T07:26:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4741c39640 1 -- 172.21.15.1:0/2444085098 learned_addr learned my addr 172.21.15.1:0/2444085098 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4741c39640 1 -- 172.21.15.1:0/2444085098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f473c152650 msgr2=0x7f473c144dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4741438640 1 --2- 172.21.15.1:0/2444085098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f473c152650 0x7f473c144dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4741c39640 1 --2- 172.21.15.1:0/2444085098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f473c152650 0x7f473c144dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f474243a640 1 --2- 172.21.15.1:0/2444085098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f473c1493c0 0x7f473c145300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4741c39640 1 -- 172.21.15.1:0/2444085098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f473c1493c0 msgr2=0x7f473c145300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4741c39640 1 --2- 172.21.15.1:0/2444085098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f473c1493c0 0x7f473c145300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4741c39640 1 -- 172.21.15.1:0/2444085098 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473c1ebac0 con 0x7f473c0bf9d0 2024-03-06T07:26:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4741c39640 1 --2- 172.21.15.1:0/2444085098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f473c0bf9d0 0x7f473c0c9a90 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f4730034040 tx=0x7f47300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f472e7fc640 1 -- 172.21.15.1:0/2444085098 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4730004230 con 0x7f473c0bf9d0 2024-03-06T07:26:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f472e7fc640 1 -- 172.21.15.1:0/2444085098 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47300043f0 con 0x7f473c0bf9d0 2024-03-06T07:26:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4743e9d640 1 -- 172.21.15.1:0/2444085098 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f473c14d8c0 con 0x7f473c0bf9d0 2024-03-06T07:26:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f472e7fc640 1 -- 172.21.15.1:0/2444085098 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4730032860 con 0x7f473c0bf9d0 2024-03-06T07:26:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.780+0000 7f4743e9d640 1 -- 172.21.15.1:0/2444085098 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f473c0c9fd0 con 0x7f473c0bf9d0 2024-03-06T07:26:20.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.781+0000 7f4743e9d640 1 -- 172.21.15.1:0/2444085098 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f473c0c85a0 con 0x7f473c0bf9d0 2024-03-06T07:26:20.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.782+0000 7f472e7fc640 1 -- 172.21.15.1:0/2444085098 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4730032a00 con 0x7f473c0bf9d0 2024-03-06T07:26:20.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.783+0000 7f472e7fc640 1 --2- 172.21.15.1:0/2444085098 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f472006b500 0x7f472006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.783+0000 7f472e7fc640 1 -- 172.21.15.1:0/2444085098 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f47300ac4f0 con 0x7f473c0bf9d0 2024-03-06T07:26:20.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.783+0000 7f4741438640 1 --2- 172.21.15.1:0/2444085098 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f472006b500 0x7f472006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.783+0000 7f4741438640 1 --2- 172.21.15.1:0/2444085098 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f472006b500 0x7f472006d9c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f473c0c5fa0 tx=0x7f4738008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.786+0000 7f472e7fc640 1 -- 172.21.15.1:0/2444085098 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f473c0c85a0 con 0x7f473c0bf9d0 2024-03-06T07:26:20.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.963+0000 7f4743e9d640 1 -- 172.21.15.1:0/2444085098 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f473c0c04a0 con 0x7f472006b500 2024-03-06T07:26:20.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.969+0000 7f472e7fc640 1 -- 172.21.15.1:0/2444085098 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252556 (secure 0 0 0) 0x7f473c0c04a0 con 0x7f472006b500 2024-03-06T07:26:20.974 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:20.975 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:26:20.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.972+0000 7f4743e9d640 1 -- 172.21.15.1:0/2444085098 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f472006b500 msgr2=0x7f472006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.972+0000 7f4743e9d640 1 --2- 172.21.15.1:0/2444085098 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f472006b500 0x7f472006d9c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f473c0c5fa0 tx=0x7f4738008040 comp rx=0 tx=0).stop 2024-03-06T07:26:20.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.972+0000 7f4743e9d640 1 -- 172.21.15.1:0/2444085098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f473c0bf9d0 msgr2=0x7f473c0c9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.972+0000 7f4743e9d640 1 --2- 172.21.15.1:0/2444085098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f473c0bf9d0 0x7f473c0c9a90 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f4730034040 tx=0x7f47300046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.972+0000 7f4743e9d640 1 -- 172.21.15.1:0/2444085098 shutdown_connections 2024-03-06T07:26:20.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.972+0000 7f4743e9d640 1 --2- 172.21.15.1:0/2444085098 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f472006b500 0x7f472006d9c0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.972+0000 7f4743e9d640 1 --2- 172.21.15.1:0/2444085098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f473c1493c0 0x7f473c145300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.972+0000 7f4743e9d640 1 --2- 172.21.15.1:0/2444085098 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f473c152650 0x7f473c144dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.972+0000 7f4743e9d640 1 --2- 172.21.15.1:0/2444085098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f473c0bf9d0 0x7f473c0c9a90 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.972+0000 7f4743e9d640 1 -- 172.21.15.1:0/2444085098 >> 172.21.15.1:0/2444085098 conn(0x7f473c066ee0 msgr2=0x7f473c0c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.972+0000 7f4743e9d640 1 -- 172.21.15.1:0/2444085098 shutdown_connections 2024-03-06T07:26:20.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:20.972+0000 7f4743e9d640 1 -- 172.21.15.1:0/2444085098 wait complete. 2024-03-06T07:26:20.985 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-03-06T07:26:20.772624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264272,"kb_used_data":14812,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710448,"statfs":{"total":773094113280,"available":772823498752,"internally_reserved":0,"allocated":15167488,"data_stored":8413224,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.171867"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.404613+0000","last_change":"2024-03-06T07:26:17.632610+0000","last_active":"2024-03-06T07:26:19.404613+0000","last_peered":"2024-03-06T07:26:19.404613+0000","last_clean":"2024-03-06T07:26:19.404613+0000","last_became_active":"2024-03-06T07:26:17.630763+0000","last_became_peered":"2024-03-06T07:26:17.630763+0000","last_unstale":"2024-03-06T07:26:19.404613+0000","last_undegraded":"2024-03-06T07:26:19.404613+0000","last_fullsized":"2024-03-06T07:26:19.404613+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:35.035202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.351533+0000","last_change":"2024-03-06T07:26:17.632425+0000","last_active":"2024-03-06T07:26:19.351533+0000","last_peered":"2024-03-06T07:26:19.351533+0000","last_clean":"2024-03-06T07:26:19.351533+0000","last_became_active":"2024-03-06T07:26:17.632153+0000","last_became_peered":"2024-03-06T07:26:17.632153+0000","last_unstale":"2024-03-06T07:26:19.351533+0000","last_undegraded":"2024-03-06T07:26:19.351533+0000","last_fullsized":"2024-03-06T07:26:19.351533+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:01.829378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403839+0000","last_change":"2024-03-06T07:26:17.630353+0000","last_active":"2024-03-06T07:26:19.403839+0000","last_peered":"2024-03-06T07:26:19.403839+0000","last_clean":"2024-03-06T07:26:19.403839+0000","last_became_active":"2024-03-06T07:26:17.630080+0000","last_became_peered":"2024-03-06T07:26:17.630080+0000","last_unstale":"2024-03-06T07:26:19.403839+0000","last_undegraded":"2024-03-06T07:26:19.403839+0000","last_fullsized":"2024-03-06T07:26:19.403839+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:49.713717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403907+0000","last_change":"2024-03-06T07:26:17.630434+0000","last_active":"2024-03-06T07:26:19.403907+0000","last_peered":"2024-03-06T07:26:19.403907+0000","last_clean":"2024-03-06T07:26:19.403907+0000","last_became_active":"2024-03-06T07:26:17.630139+0000","last_became_peered":"2024-03-06T07:26:17.630139+0000","last_unstale":"2024-03-06T07:26:19.403907+0000","last_undegraded":"2024-03-06T07:26:19.403907+0000","last_fullsized":"2024-03-06T07:26:19.403907+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:09.159934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403134+0000","last_change":"2024-03-06T07:26:17.632665+0000","last_active":"2024-03-06T07:26:19.403134+0000","last_peered":"2024-03-06T07:26:19.403134+0000","last_clean":"2024-03-06T07:26:19.403134+0000","last_became_active":"2024-03-06T07:26:17.632377+0000","last_became_peered":"2024-03-06T07:26:17.632377+0000","last_unstale":"2024-03-06T07:26:19.403134+0000","last_undegraded":"2024-03-06T07:26:19.403134+0000","last_fullsized":"2024-03-06T07:26:19.403134+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:23.225363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.351042+0000","last_change":"2024-03-06T07:26:17.633911+0000","last_active":"2024-03-06T07:26:19.351042+0000","last_peered":"2024-03-06T07:26:19.351042+0000","last_clean":"2024-03-06T07:26:19.351042+0000","last_became_active":"2024-03-06T07:26:17.633638+0000","last_became_peered":"2024-03-06T07:26:17.633638+0000","last_unstale":"2024-03-06T07:26:19.351042+0000","last_undegraded":"2024-03-06T07:26:19.351042+0000","last_fullsized":"2024-03-06T07:26:19.351042+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:01.350963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631208+0000","last_change":"2024-03-06T07:26:17.629894+0000","last_active":"2024-03-06T07:26:18.631208+0000","last_peered":"2024-03-06T07:26:18.631208+0000","last_clean":"2024-03-06T07:26:18.631208+0000","last_became_active":"2024-03-06T07:26:17.629625+0000","last_became_peered":"2024-03-06T07:26:17.629625+0000","last_unstale":"2024-03-06T07:26:18.631208+0000","last_undegraded":"2024-03-06T07:26:18.631208+0000","last_fullsized":"2024-03-06T07:26:18.631208+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:46.687284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631781+0000","last_change":"2024-03-06T07:26:17.632711+0000","last_active":"2024-03-06T07:26:18.631781+0000","last_peered":"2024-03-06T07:26:18.631781+0000","last_clean":"2024-03-06T07:26:18.631781+0000","last_became_active":"2024-03-06T07:26:17.632173+0000","last_became_peered":"2024-03-06T07:26:17.632173+0000","last_unstale":"2024-03-06T07:26:18.631781+0000","last_undegraded":"2024-03-06T07:26:18.631781+0000","last_fullsized":"2024-03-06T07:26:18.631781+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:25.595909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631526+0000","last_change":"2024-03-06T07:26:17.638531+0000","last_active":"2024-03-06T07:26:18.631526+0000","last_peered":"2024-03-06T07:26:18.631526+0000","last_clean":"2024-03-06T07:26:18.631526+0000","last_became_active":"2024-03-06T07:26:17.638254+0000","last_became_peered":"2024-03-06T07:26:17.638254+0000","last_unstale":"2024-03-06T07:26:18.631526+0000","last_undegraded":"2024-03-06T07:26:18.631526+0000","last_fullsized":"2024-03-06T07:26:18.631526+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:01.350963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.632221+0000","last_change":"2024-03-06T07:26:17.632455+0000","last_active":"2024-03-06T07:26:18.632221+0000","last_peered":"2024-03-06T07:26:18.632221+0000","last_clean":"2024-03-06T07:26:18.632221+0000","last_became_active":"2024-03-06T07:26:17.632195+0000","last_became_peered":"2024-03-06T07:26:17.632195+0000","last_unstale":"2024-03-06T07:26:18.632221+0000","last_undegraded":"2024-03-06T07:26:18.632221+0000","last_fullsized":"2024-03-06T07:26:18.632221+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:14.404340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.404508+0000","last_change":"2024-03-06T07:26:17.633984+0000","last_active":"2024-03-06T07:26:19.404508+0000","last_peered":"2024-03-06T07:26:19.404508+0000","last_clean":"2024-03-06T07:26:19.404508+0000","last_became_active":"2024-03-06T07:26:17.633666+0000","last_became_peered":"2024-03-06T07:26:17.633666+0000","last_unstale":"2024-03-06T07:26:19.404508+0000","last_undegraded":"2024-03-06T07:26:19.404508+0000","last_fullsized":"2024-03-06T07:26:19.404508+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:43.884264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.351240+0000","last_change":"2024-03-06T07:26:17.634798+0000","last_active":"2024-03-06T07:26:19.351240+0000","last_peered":"2024-03-06T07:26:19.351240+0000","last_clean":"2024-03-06T07:26:19.351240+0000","last_became_active":"2024-03-06T07:26:17.634537+0000","last_became_peered":"2024-03-06T07:26:17.634537+0000","last_unstale":"2024-03-06T07:26:19.351240+0000","last_undegraded":"2024-03-06T07:26:19.351240+0000","last_fullsized":"2024-03-06T07:26:19.351240+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:25.595909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631653+0000","last_change":"2024-03-06T07:26:17.640814+0000","last_active":"2024-03-06T07:26:18.631653+0000","last_peered":"2024-03-06T07:26:18.631653+0000","last_clean":"2024-03-06T07:26:18.631653+0000","last_became_active":"2024-03-06T07:26:17.640444+0000","last_became_peered":"2024-03-06T07:26:17.640444+0000","last_unstale":"2024-03-06T07:26:18.631653+0000","last_undegraded":"2024-03-06T07:26:18.631653+0000","last_fullsized":"2024-03-06T07:26:18.631653+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:35.920913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.402944+0000","last_change":"2024-03-06T07:26:17.631892+0000","last_active":"2024-03-06T07:26:19.402944+0000","last_peered":"2024-03-06T07:26:19.402944+0000","last_clean":"2024-03-06T07:26:19.402944+0000","last_became_active":"2024-03-06T07:26:17.631498+0000","last_became_peered":"2024-03-06T07:26:17.631498+0000","last_unstale":"2024-03-06T07:26:19.402944+0000","last_undegraded":"2024-03-06T07:26:19.402944+0000","last_fullsized":"2024-03-06T07:26:19.402944+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:09.253874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.404410+0000","last_change":"2024-03-06T07:26:17.634220+0000","last_active":"2024-03-06T07:26:19.404410+0000","last_peered":"2024-03-06T07:26:19.404410+0000","last_clean":"2024-03-06T07:26:19.404410+0000","last_became_active":"2024-03-06T07:26:17.633922+0000","last_became_peered":"2024-03-06T07:26:17.633922+0000","last_unstale":"2024-03-06T07:26:19.404410+0000","last_undegraded":"2024-03-06T07:26:19.404410+0000","last_fullsized":"2024-03-06T07:26:19.404410+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:51.900557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403430+0000","last_change":"2024-03-06T07:26:17.635856+0000","last_active":"2024-03-06T07:26:19.403430+0000","last_peered":"2024-03-06T07:26:19.403430+0000","last_clean":"2024-03-06T07:26:19.403430+0000","last_became_active":"2024-03-06T07:26:17.631353+0000","last_became_peered":"2024-03-06T07:26:17.631353+0000","last_unstale":"2024-03-06T07:26:19.403430+0000","last_undegraded":"2024-03-06T07:26:19.403430+0000","last_fullsized":"2024-03-06T07:26:19.403430+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:15.652152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631670+0000","last_change":"2024-03-06T07:26:17.628960+0000","last_active":"2024-03-06T07:26:18.631670+0000","last_peered":"2024-03-06T07:26:18.631670+0000","last_clean":"2024-03-06T07:26:18.631670+0000","last_became_active":"2024-03-06T07:26:17.628681+0000","last_became_peered":"2024-03-06T07:26:17.628681+0000","last_unstale":"2024-03-06T07:26:18.631670+0000","last_undegraded":"2024-03-06T07:26:18.631670+0000","last_fullsized":"2024-03-06T07:26:18.631670+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:40.950597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403938+0000","last_change":"2024-03-06T07:26:17.629582+0000","last_active":"2024-03-06T07:26:19.403938+0000","last_peered":"2024-03-06T07:26:19.403938+0000","last_clean":"2024-03-06T07:26:19.403938+0000","last_became_active":"2024-03-06T07:26:17.629246+0000","last_became_peered":"2024-03-06T07:26:17.629246+0000","last_unstale":"2024-03-06T07:26:19.403938+0000","last_undegraded":"2024-03-06T07:26:19.403938+0000","last_fullsized":"2024-03-06T07:26:19.403938+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:56.556155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.404706+0000","last_change":"2024-03-06T07:26:17.632386+0000","last_active":"2024-03-06T07:26:19.404706+0000","last_peered":"2024-03-06T07:26:19.404706+0000","last_clean":"2024-03-06T07:26:19.404706+0000","last_became_active":"2024-03-06T07:26:17.631867+0000","last_became_peered":"2024-03-06T07:26:17.631867+0000","last_unstale":"2024-03-06T07:26:19.404706+0000","last_undegraded":"2024-03-06T07:26:19.404706+0000","last_fullsized":"2024-03-06T07:26:19.404706+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:46.687284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.630449+0000","last_change":"2024-03-06T07:26:17.622563+0000","last_active":"2024-03-06T07:26:18.630449+0000","last_peered":"2024-03-06T07:26:18.630449+0000","last_clean":"2024-03-06T07:26:18.630449+0000","last_became_active":"2024-03-06T07:26:17.622293+0000","last_became_peered":"2024-03-06T07:26:17.622293+0000","last_unstale":"2024-03-06T07:26:18.630449+0000","last_undegraded":"2024-03-06T07:26:18.630449+0000","last_fullsized":"2024-03-06T07:26:18.630449+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:34.462291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631785+0000","last_change":"2024-03-06T07:26:17.637385+0000","last_active":"2024-03-06T07:26:18.631785+0000","last_peered":"2024-03-06T07:26:18.631785+0000","last_clean":"2024-03-06T07:26:18.631785+0000","last_became_active":"2024-03-06T07:26:17.637063+0000","last_became_peered":"2024-03-06T07:26:17.637063+0000","last_unstale":"2024-03-06T07:26:18.631785+0000","last_undegraded":"2024-03-06T07:26:18.631785+0000","last_fullsized":"2024-03-06T07:26:18.631785+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:49.333296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.351127+0000","last_change":"2024-03-06T07:26:17.632278+0000","last_active":"2024-03-06T07:26:19.351127+0000","last_peered":"2024-03-06T07:26:19.351127+0000","last_clean":"2024-03-06T07:26:19.351127+0000","last_became_active":"2024-03-06T07:26:17.631791+0000","last_became_peered":"2024-03-06T07:26:17.631791+0000","last_unstale":"2024-03-06T07:26:19.351127+0000","last_undegraded":"2024-03-06T07:26:19.351127+0000","last_fullsized":"2024-03-06T07:26:19.351127+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:40.950597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.402822+0000","last_change":"2024-03-06T07:26:17.627902+0000","last_active":"2024-03-06T07:26:19.402822+0000","last_peered":"2024-03-06T07:26:19.402822+0000","last_clean":"2024-03-06T07:26:19.402822+0000","last_became_active":"2024-03-06T07:26:17.627643+0000","last_became_peered":"2024-03-06T07:26:17.627643+0000","last_unstale":"2024-03-06T07:26:19.402822+0000","last_undegraded":"2024-03-06T07:26:19.402822+0000","last_fullsized":"2024-03-06T07:26:19.402822+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:30.600647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403588+0000","last_change":"2024-03-06T07:26:17.631421+0000","last_active":"2024-03-06T07:26:19.403588+0000","last_peered":"2024-03-06T07:26:19.403588+0000","last_clean":"2024-03-06T07:26:19.403588+0000","last_became_active":"2024-03-06T07:26:17.631022+0000","last_became_peered":"2024-03-06T07:26:17.631022+0000","last_unstale":"2024-03-06T07:26:19.403588+0000","last_undegraded":"2024-03-06T07:26:19.403588+0000","last_fullsized":"2024-03-06T07:26:19.403588+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:40.032388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.632459+0000","last_change":"2024-03-06T07:26:17.637044+0000","last_active":"2024-03-06T07:26:18.632459+0000","last_peered":"2024-03-06T07:26:18.632459+0000","last_clean":"2024-03-06T07:26:18.632459+0000","last_became_active":"2024-03-06T07:26:17.636771+0000","last_became_peered":"2024-03-06T07:26:17.636771+0000","last_unstale":"2024-03-06T07:26:18.632459+0000","last_undegraded":"2024-03-06T07:26:18.632459+0000","last_fullsized":"2024-03-06T07:26:18.632459+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:29.354563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.629942+0000","last_change":"2024-03-06T07:26:17.636369+0000","last_active":"2024-03-06T07:26:18.629942+0000","last_peered":"2024-03-06T07:26:18.629942+0000","last_clean":"2024-03-06T07:26:18.629942+0000","last_became_active":"2024-03-06T07:26:17.636090+0000","last_became_peered":"2024-03-06T07:26:17.636090+0000","last_unstale":"2024-03-06T07:26:18.629942+0000","last_undegraded":"2024-03-06T07:26:18.629942+0000","last_fullsized":"2024-03-06T07:26:18.629942+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:45.110614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.351385+0000","last_change":"2024-03-06T07:26:17.633006+0000","last_active":"2024-03-06T07:26:19.351385+0000","last_peered":"2024-03-06T07:26:19.351385+0000","last_clean":"2024-03-06T07:26:19.351385+0000","last_became_active":"2024-03-06T07:26:17.632523+0000","last_became_peered":"2024-03-06T07:26:17.632523+0000","last_unstale":"2024-03-06T07:26:19.351385+0000","last_undegraded":"2024-03-06T07:26:19.351385+0000","last_fullsized":"2024-03-06T07:26:19.351385+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:41.655929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.632293+0000","last_change":"2024-03-06T07:26:17.638448+0000","last_active":"2024-03-06T07:26:18.632293+0000","last_peered":"2024-03-06T07:26:18.632293+0000","last_clean":"2024-03-06T07:26:18.632293+0000","last_became_active":"2024-03-06T07:26:17.638188+0000","last_became_peered":"2024-03-06T07:26:17.638188+0000","last_unstale":"2024-03-06T07:26:18.632293+0000","last_undegraded":"2024-03-06T07:26:18.632293+0000","last_fullsized":"2024-03-06T07:26:18.632293+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:15.455961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.351314+0000","last_change":"2024-03-06T07:26:17.634053+0000","last_active":"2024-03-06T07:26:19.351314+0000","last_peered":"2024-03-06T07:26:19.351314+0000","last_clean":"2024-03-06T07:26:19.351314+0000","last_became_active":"2024-03-06T07:26:17.633451+0000","last_became_peered":"2024-03-06T07:26:17.633451+0000","last_unstale":"2024-03-06T07:26:19.351314+0000","last_undegraded":"2024-03-06T07:26:19.351314+0000","last_fullsized":"2024-03-06T07:26:19.351314+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:25.173456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403886+0000","last_change":"2024-03-06T07:26:17.627891+0000","last_active":"2024-03-06T07:26:19.403886+0000","last_peered":"2024-03-06T07:26:19.403886+0000","last_clean":"2024-03-06T07:26:19.403886+0000","last_became_active":"2024-03-06T07:26:17.627624+0000","last_became_peered":"2024-03-06T07:26:17.627624+0000","last_unstale":"2024-03-06T07:26:19.403886+0000","last_undegraded":"2024-03-06T07:26:19.403886+0000","last_fullsized":"2024-03-06T07:26:19.403886+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:56.476412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.404554+0000","last_change":"2024-03-06T07:26:17.635467+0000","last_active":"2024-03-06T07:26:19.404554+0000","last_peered":"2024-03-06T07:26:19.404554+0000","last_clean":"2024-03-06T07:26:19.404554+0000","last_became_active":"2024-03-06T07:26:17.635125+0000","last_became_peered":"2024-03-06T07:26:17.635125+0000","last_unstale":"2024-03-06T07:26:19.404554+0000","last_undegraded":"2024-03-06T07:26:19.404554+0000","last_fullsized":"2024-03-06T07:26:19.404554+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:39.829929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403896+0000","last_change":"2024-03-06T07:26:17.638944+0000","last_active":"2024-03-06T07:26:19.403896+0000","last_peered":"2024-03-06T07:26:19.403896+0000","last_clean":"2024-03-06T07:26:19.403896+0000","last_became_active":"2024-03-06T07:26:17.638649+0000","last_became_peered":"2024-03-06T07:26:17.638649+0000","last_unstale":"2024-03-06T07:26:19.403896+0000","last_undegraded":"2024-03-06T07:26:19.403896+0000","last_fullsized":"2024-03-06T07:26:19.403896+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:17.394090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.351151+0000","last_change":"2024-03-06T07:26:17.635968+0000","last_active":"2024-03-06T07:26:19.351151+0000","last_peered":"2024-03-06T07:26:19.351151+0000","last_clean":"2024-03-06T07:26:19.351151+0000","last_became_active":"2024-03-06T07:26:17.635095+0000","last_became_peered":"2024-03-06T07:26:17.635095+0000","last_unstale":"2024-03-06T07:26:19.351151+0000","last_undegraded":"2024-03-06T07:26:19.351151+0000","last_fullsized":"2024-03-06T07:26:19.351151+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:26.879181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"148'4","reported_seq":18,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.404725+0000","last_change":"2024-03-06T07:26:16.626240+0000","last_active":"2024-03-06T07:26:19.404725+0000","last_peered":"2024-03-06T07:26:19.404725+0000","last_clean":"2024-03-06T07:26:19.404725+0000","last_became_active":"2024-03-06T07:26:16.626011+0000","last_became_peered":"2024-03-06T07:26:16.626011+0000","last_unstale":"2024-03-06T07:26:19.404725+0000","last_undegraded":"2024-03-06T07:26:19.404725+0000","last_fullsized":"2024-03-06T07:26:19.404725+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:30.259380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.630969+0000","last_change":"2024-03-06T07:26:17.627512+0000","last_active":"2024-03-06T07:26:18.630969+0000","last_peered":"2024-03-06T07:26:18.630969+0000","last_clean":"2024-03-06T07:26:18.630969+0000","last_became_active":"2024-03-06T07:26:17.627235+0000","last_became_peered":"2024-03-06T07:26:17.627235+0000","last_unstale":"2024-03-06T07:26:18.630969+0000","last_undegraded":"2024-03-06T07:26:18.630969+0000","last_fullsized":"2024-03-06T07:26:18.630969+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:19.245848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"148'1","reported_seq":15,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403171+0000","last_change":"2024-03-06T07:26:16.625703+0000","last_active":"2024-03-06T07:26:19.403171+0000","last_peered":"2024-03-06T07:26:19.403171+0000","last_clean":"2024-03-06T07:26:19.403171+0000","last_became_active":"2024-03-06T07:26:16.625426+0000","last_became_peered":"2024-03-06T07:26:16.625426+0000","last_unstale":"2024-03-06T07:26:19.403171+0000","last_undegraded":"2024-03-06T07:26:19.403171+0000","last_fullsized":"2024-03-06T07:26:19.403171+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:39.946462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403000+0000","last_change":"2024-03-06T07:26:17.632107+0000","last_active":"2024-03-06T07:26:19.403000+0000","last_peered":"2024-03-06T07:26:19.403000+0000","last_clean":"2024-03-06T07:26:19.403000+0000","last_became_active":"2024-03-06T07:26:17.631481+0000","last_became_peered":"2024-03-06T07:26:17.631481+0000","last_unstale":"2024-03-06T07:26:19.403000+0000","last_undegraded":"2024-03-06T07:26:19.403000+0000","last_fullsized":"2024-03-06T07:26:19.403000+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:20.662554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"148'2","reported_seq":16,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.404718+0000","last_change":"2024-03-06T07:26:16.620199+0000","last_active":"2024-03-06T07:26:19.404718+0000","last_peered":"2024-03-06T07:26:19.404718+0000","last_clean":"2024-03-06T07:26:19.404718+0000","last_became_active":"2024-03-06T07:26:16.619877+0000","last_became_peered":"2024-03-06T07:26:16.619877+0000","last_unstale":"2024-03-06T07:26:19.404718+0000","last_undegraded":"2024-03-06T07:26:19.404718+0000","last_fullsized":"2024-03-06T07:26:19.404718+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:14.648017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.351326+0000","last_change":"2024-03-06T07:26:17.634295+0000","last_active":"2024-03-06T07:26:19.351326+0000","last_peered":"2024-03-06T07:26:19.351326+0000","last_clean":"2024-03-06T07:26:19.351326+0000","last_became_active":"2024-03-06T07:26:17.633744+0000","last_became_peered":"2024-03-06T07:26:17.633744+0000","last_unstale":"2024-03-06T07:26:19.351326+0000","last_undegraded":"2024-03-06T07:26:19.351326+0000","last_fullsized":"2024-03-06T07:26:19.351326+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:14.404340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"148'2","reported_seq":16,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.630450+0000","last_change":"2024-03-06T07:26:16.616715+0000","last_active":"2024-03-06T07:26:18.630450+0000","last_peered":"2024-03-06T07:26:18.630450+0000","last_clean":"2024-03-06T07:26:18.630450+0000","last_became_active":"2024-03-06T07:26:16.616466+0000","last_became_peered":"2024-03-06T07:26:16.616466+0000","last_unstale":"2024-03-06T07:26:18.630450+0000","last_undegraded":"2024-03-06T07:26:18.630450+0000","last_fullsized":"2024-03-06T07:26:18.630450+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:54.279225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631775+0000","last_change":"2024-03-06T07:26:17.631966+0000","last_active":"2024-03-06T07:26:18.631775+0000","last_peered":"2024-03-06T07:26:18.631775+0000","last_clean":"2024-03-06T07:26:18.631775+0000","last_became_active":"2024-03-06T07:26:17.631673+0000","last_became_peered":"2024-03-06T07:26:17.631673+0000","last_unstale":"2024-03-06T07:26:18.631775+0000","last_undegraded":"2024-03-06T07:26:18.631775+0000","last_fullsized":"2024-03-06T07:26:18.631775+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:26.879181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"148'1","reported_seq":15,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631974+0000","last_change":"2024-03-06T07:26:16.619602+0000","last_active":"2024-03-06T07:26:18.631974+0000","last_peered":"2024-03-06T07:26:18.631974+0000","last_clean":"2024-03-06T07:26:18.631974+0000","last_became_active":"2024-03-06T07:26:16.619282+0000","last_became_peered":"2024-03-06T07:26:16.619282+0000","last_unstale":"2024-03-06T07:26:18.631974+0000","last_undegraded":"2024-03-06T07:26:18.631974+0000","last_fullsized":"2024-03-06T07:26:18.631974+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:59.587645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.630534+0000","last_change":"2024-03-06T07:26:17.628669+0000","last_active":"2024-03-06T07:26:18.630534+0000","last_peered":"2024-03-06T07:26:18.630534+0000","last_clean":"2024-03-06T07:26:18.630534+0000","last_became_active":"2024-03-06T07:26:17.628371+0000","last_became_peered":"2024-03-06T07:26:17.628371+0000","last_unstale":"2024-03-06T07:26:18.630534+0000","last_undegraded":"2024-03-06T07:26:18.630534+0000","last_fullsized":"2024-03-06T07:26:18.630534+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:16.499694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"148'1","reported_seq":15,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403061+0000","last_change":"2024-03-06T07:26:16.615912+0000","last_active":"2024-03-06T07:26:19.403061+0000","last_peered":"2024-03-06T07:26:19.403061+0000","last_clean":"2024-03-06T07:26:19.403061+0000","last_became_active":"2024-03-06T07:26:16.615625+0000","last_became_peered":"2024-03-06T07:26:16.615625+0000","last_unstale":"2024-03-06T07:26:19.403061+0000","last_undegraded":"2024-03-06T07:26:19.403061+0000","last_fullsized":"2024-03-06T07:26:19.403061+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:24.591774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.630083+0000","last_change":"2024-03-06T07:26:17.626866+0000","last_active":"2024-03-06T07:26:18.630083+0000","last_peered":"2024-03-06T07:26:18.630083+0000","last_clean":"2024-03-06T07:26:18.630083+0000","last_became_active":"2024-03-06T07:26:17.626586+0000","last_became_peered":"2024-03-06T07:26:17.626586+0000","last_unstale":"2024-03-06T07:26:18.630083+0000","last_undegraded":"2024-03-06T07:26:18.630083+0000","last_fullsized":"2024-03-06T07:26:18.630083+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:34.182149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.629792+0000","last_change":"2024-03-06T07:26:16.622797+0000","last_active":"2024-03-06T07:26:18.629792+0000","last_peered":"2024-03-06T07:26:18.629792+0000","last_clean":"2024-03-06T07:26:18.629792+0000","last_became_active":"2024-03-06T07:26:16.622534+0000","last_became_peered":"2024-03-06T07:26:16.622534+0000","last_unstale":"2024-03-06T07:26:18.629792+0000","last_undegraded":"2024-03-06T07:26:18.629792+0000","last_fullsized":"2024-03-06T07:26:18.629792+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:39.697277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.404629+0000","last_change":"2024-03-06T07:26:17.632829+0000","last_active":"2024-03-06T07:26:19.404629+0000","last_peered":"2024-03-06T07:26:19.404629+0000","last_clean":"2024-03-06T07:26:19.404629+0000","last_became_active":"2024-03-06T07:26:17.631051+0000","last_became_peered":"2024-03-06T07:26:17.631051+0000","last_unstale":"2024-03-06T07:26:19.404629+0000","last_undegraded":"2024-03-06T07:26:19.404629+0000","last_fullsized":"2024-03-06T07:26:19.404629+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:09.224572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631866+0000","last_change":"2024-03-06T07:26:16.618575+0000","last_active":"2024-03-06T07:26:18.631866+0000","last_peered":"2024-03-06T07:26:18.631866+0000","last_clean":"2024-03-06T07:26:18.631866+0000","last_became_active":"2024-03-06T07:26:16.618143+0000","last_became_peered":"2024-03-06T07:26:16.618143+0000","last_unstale":"2024-03-06T07:26:18.631866+0000","last_undegraded":"2024-03-06T07:26:18.631866+0000","last_fullsized":"2024-03-06T07:26:18.631866+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:48.709582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.351057+0000","last_change":"2024-03-06T07:26:17.636599+0000","last_active":"2024-03-06T07:26:19.351057+0000","last_peered":"2024-03-06T07:26:19.351057+0000","last_clean":"2024-03-06T07:26:19.351057+0000","last_became_active":"2024-03-06T07:26:17.635451+0000","last_became_peered":"2024-03-06T07:26:17.635451+0000","last_unstale":"2024-03-06T07:26:19.351057+0000","last_undegraded":"2024-03-06T07:26:19.351057+0000","last_fullsized":"2024-03-06T07:26:19.351057+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:27.398343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.632422+0000","last_change":"2024-03-06T07:26:16.627460+0000","last_active":"2024-03-06T07:26:18.632422+0000","last_peered":"2024-03-06T07:26:18.632422+0000","last_clean":"2024-03-06T07:26:18.632422+0000","last_became_active":"2024-03-06T07:26:16.626345+0000","last_became_peered":"2024-03-06T07:26:16.626345+0000","last_unstale":"2024-03-06T07:26:18.632422+0000","last_undegraded":"2024-03-06T07:26:18.632422+0000","last_fullsized":"2024-03-06T07:26:18.632422+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:24.257795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631208+0000","last_change":"2024-03-06T07:26:17.634986+0000","last_active":"2024-03-06T07:26:18.631208+0000","last_peered":"2024-03-06T07:26:18.631208+0000","last_clean":"2024-03-06T07:26:18.631208+0000","last_became_active":"2024-03-06T07:26:17.634680+0000","last_became_peered":"2024-03-06T07:26:17.634680+0000","last_unstale":"2024-03-06T07:26:18.631208+0000","last_undegraded":"2024-03-06T07:26:18.631208+0000","last_fullsized":"2024-03-06T07:26:18.631208+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:48.372843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"148'3","reported_seq":17,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403954+0000","last_change":"2024-03-06T07:26:16.623947+0000","last_active":"2024-03-06T07:26:19.403954+0000","last_peered":"2024-03-06T07:26:19.403954+0000","last_clean":"2024-03-06T07:26:19.403954+0000","last_became_active":"2024-03-06T07:26:16.623662+0000","last_became_peered":"2024-03-06T07:26:16.623662+0000","last_unstale":"2024-03-06T07:26:19.403954+0000","last_undegraded":"2024-03-06T07:26:19.403954+0000","last_fullsized":"2024-03-06T07:26:19.403954+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:00.411414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.630475+0000","last_change":"2024-03-06T07:26:17.626440+0000","last_active":"2024-03-06T07:26:18.630475+0000","last_peered":"2024-03-06T07:26:18.630475+0000","last_clean":"2024-03-06T07:26:18.630475+0000","last_became_active":"2024-03-06T07:26:17.626149+0000","last_became_peered":"2024-03-06T07:26:17.626149+0000","last_unstale":"2024-03-06T07:26:18.630475+0000","last_undegraded":"2024-03-06T07:26:18.630475+0000","last_fullsized":"2024-03-06T07:26:18.630475+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:46.191755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"148'6","reported_seq":20,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.632371+0000","last_change":"2024-03-06T07:26:16.627536+0000","last_active":"2024-03-06T07:26:18.632371+0000","last_peered":"2024-03-06T07:26:18.632371+0000","last_clean":"2024-03-06T07:26:18.632371+0000","last_became_active":"2024-03-06T07:26:16.627229+0000","last_became_peered":"2024-03-06T07:26:16.627229+0000","last_unstale":"2024-03-06T07:26:18.632371+0000","last_undegraded":"2024-03-06T07:26:18.632371+0000","last_fullsized":"2024-03-06T07:26:18.632371+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:00.825243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403793+0000","last_change":"2024-03-06T07:26:17.638945+0000","last_active":"2024-03-06T07:26:19.403793+0000","last_peered":"2024-03-06T07:26:19.403793+0000","last_clean":"2024-03-06T07:26:19.403793+0000","last_became_active":"2024-03-06T07:26:17.638664+0000","last_became_peered":"2024-03-06T07:26:17.638664+0000","last_unstale":"2024-03-06T07:26:19.403793+0000","last_undegraded":"2024-03-06T07:26:19.403793+0000","last_fullsized":"2024-03-06T07:26:19.403793+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:15.414267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"148'2","reported_seq":16,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403158+0000","last_change":"2024-03-06T07:26:16.625884+0000","last_active":"2024-03-06T07:26:19.403158+0000","last_peered":"2024-03-06T07:26:19.403158+0000","last_clean":"2024-03-06T07:26:19.403158+0000","last_became_active":"2024-03-06T07:26:16.625605+0000","last_became_peered":"2024-03-06T07:26:16.625605+0000","last_unstale":"2024-03-06T07:26:19.403158+0000","last_undegraded":"2024-03-06T07:26:19.403158+0000","last_fullsized":"2024-03-06T07:26:19.403158+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:13.400206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631316+0000","last_change":"2024-03-06T07:26:17.631376+0000","last_active":"2024-03-06T07:26:18.631316+0000","last_peered":"2024-03-06T07:26:18.631316+0000","last_clean":"2024-03-06T07:26:18.631316+0000","last_became_active":"2024-03-06T07:26:17.631068+0000","last_became_peered":"2024-03-06T07:26:17.631068+0000","last_unstale":"2024-03-06T07:26:18.631316+0000","last_undegraded":"2024-03-06T07:26:18.631316+0000","last_fullsized":"2024-03-06T07:26:18.631316+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:46.783070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631354+0000","last_change":"2024-03-06T07:26:16.625352+0000","last_active":"2024-03-06T07:26:18.631354+0000","last_peered":"2024-03-06T07:26:18.631354+0000","last_clean":"2024-03-06T07:26:18.631354+0000","last_became_active":"2024-03-06T07:26:16.624893+0000","last_became_peered":"2024-03-06T07:26:16.624893+0000","last_unstale":"2024-03-06T07:26:18.631354+0000","last_undegraded":"2024-03-06T07:26:18.631354+0000","last_fullsized":"2024-03-06T07:26:18.631354+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:29.596513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.404720+0000","last_change":"2024-03-06T07:26:17.639031+0000","last_active":"2024-03-06T07:26:19.404720+0000","last_peered":"2024-03-06T07:26:19.404720+0000","last_clean":"2024-03-06T07:26:19.404720+0000","last_became_active":"2024-03-06T07:26:17.638716+0000","last_became_peered":"2024-03-06T07:26:17.638716+0000","last_unstale":"2024-03-06T07:26:19.404720+0000","last_undegraded":"2024-03-06T07:26:19.404720+0000","last_fullsized":"2024-03-06T07:26:19.404720+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:46.783070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.404697+0000","last_change":"2024-03-06T07:26:16.620528+0000","last_active":"2024-03-06T07:26:19.404697+0000","last_peered":"2024-03-06T07:26:19.404697+0000","last_clean":"2024-03-06T07:26:19.404697+0000","last_became_active":"2024-03-06T07:26:16.620245+0000","last_became_peered":"2024-03-06T07:26:16.620245+0000","last_unstale":"2024-03-06T07:26:19.404697+0000","last_undegraded":"2024-03-06T07:26:19.404697+0000","last_fullsized":"2024-03-06T07:26:19.404697+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:39.028254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.630005+0000","last_change":"2024-03-06T07:26:17.641468+0000","last_active":"2024-03-06T07:26:18.630005+0000","last_peered":"2024-03-06T07:26:18.630005+0000","last_clean":"2024-03-06T07:26:18.630005+0000","last_became_active":"2024-03-06T07:26:17.641155+0000","last_became_peered":"2024-03-06T07:26:17.641155+0000","last_unstale":"2024-03-06T07:26:18.630005+0000","last_undegraded":"2024-03-06T07:26:18.630005+0000","last_fullsized":"2024-03-06T07:26:18.630005+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:27.464355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"148'8","reported_seq":22,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631970+0000","last_change":"2024-03-06T07:26:16.620548+0000","last_active":"2024-03-06T07:26:18.631970+0000","last_peered":"2024-03-06T07:26:18.631970+0000","last_clean":"2024-03-06T07:26:18.631970+0000","last_became_active":"2024-03-06T07:26:16.620289+0000","last_became_peered":"2024-03-06T07:26:16.620289+0000","last_unstale":"2024-03-06T07:26:18.631970+0000","last_undegraded":"2024-03-06T07:26:18.631970+0000","last_fullsized":"2024-03-06T07:26:18.631970+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:08.155799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631908+0000","last_change":"2024-03-06T07:26:17.629187+0000","last_active":"2024-03-06T07:26:18.631908+0000","last_peered":"2024-03-06T07:26:18.631908+0000","last_clean":"2024-03-06T07:26:18.631908+0000","last_became_active":"2024-03-06T07:26:17.628903+0000","last_became_peered":"2024-03-06T07:26:17.628903+0000","last_unstale":"2024-03-06T07:26:18.631908+0000","last_undegraded":"2024-03-06T07:26:18.631908+0000","last_fullsized":"2024-03-06T07:26:18.631908+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:25.173456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"148'1","reported_seq":15,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.632518+0000","last_change":"2024-03-06T07:26:16.617108+0000","last_active":"2024-03-06T07:26:18.632518+0000","last_peered":"2024-03-06T07:26:18.632518+0000","last_clean":"2024-03-06T07:26:18.632518+0000","last_became_active":"2024-03-06T07:26:16.616776+0000","last_became_peered":"2024-03-06T07:26:16.616776+0000","last_unstale":"2024-03-06T07:26:18.632518+0000","last_undegraded":"2024-03-06T07:26:18.632518+0000","last_fullsized":"2024-03-06T07:26:18.632518+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:01.223054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.404472+0000","last_change":"2024-03-06T07:26:17.634792+0000","last_active":"2024-03-06T07:26:19.404472+0000","last_peered":"2024-03-06T07:26:19.404472+0000","last_clean":"2024-03-06T07:26:19.404472+0000","last_became_active":"2024-03-06T07:26:17.634367+0000","last_became_peered":"2024-03-06T07:26:17.634367+0000","last_unstale":"2024-03-06T07:26:19.404472+0000","last_undegraded":"2024-03-06T07:26:19.404472+0000","last_fullsized":"2024-03-06T07:26:19.404472+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:11.386543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"148'2","reported_seq":16,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.632212+0000","last_change":"2024-03-06T07:26:16.619012+0000","last_active":"2024-03-06T07:26:18.632212+0000","last_peered":"2024-03-06T07:26:18.632212+0000","last_clean":"2024-03-06T07:26:18.632212+0000","last_became_active":"2024-03-06T07:26:16.617718+0000","last_became_peered":"2024-03-06T07:26:16.617718+0000","last_unstale":"2024-03-06T07:26:18.632212+0000","last_undegraded":"2024-03-06T07:26:18.632212+0000","last_fullsized":"2024-03-06T07:26:18.632212+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:08.220437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403805+0000","last_change":"2024-03-06T07:26:17.633394+0000","last_active":"2024-03-06T07:26:19.403805+0000","last_peered":"2024-03-06T07:26:19.403805+0000","last_clean":"2024-03-06T07:26:19.403805+0000","last_became_active":"2024-03-06T07:26:17.632438+0000","last_became_peered":"2024-03-06T07:26:17.632438+0000","last_unstale":"2024-03-06T07:26:19.403805+0000","last_undegraded":"2024-03-06T07:26:19.403805+0000","last_fullsized":"2024-03-06T07:26:19.403805+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:12.024002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"148'1","reported_seq":15,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403178+0000","last_change":"2024-03-06T07:26:16.628646+0000","last_active":"2024-03-06T07:26:19.403178+0000","last_peered":"2024-03-06T07:26:19.403178+0000","last_clean":"2024-03-06T07:26:19.403178+0000","last_became_active":"2024-03-06T07:26:16.627947+0000","last_became_peered":"2024-03-06T07:26:16.627947+0000","last_unstale":"2024-03-06T07:26:19.403178+0000","last_undegraded":"2024-03-06T07:26:19.403178+0000","last_fullsized":"2024-03-06T07:26:19.403178+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:25.875046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.404181+0000","last_change":"2024-03-06T07:26:17.634254+0000","last_active":"2024-03-06T07:26:19.404181+0000","last_peered":"2024-03-06T07:26:19.404181+0000","last_clean":"2024-03-06T07:26:19.404181+0000","last_became_active":"2024-03-06T07:26:17.633961+0000","last_became_peered":"2024-03-06T07:26:17.633961+0000","last_unstale":"2024-03-06T07:26:19.404181+0000","last_undegraded":"2024-03-06T07:26:19.404181+0000","last_fullsized":"2024-03-06T07:26:19.404181+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:59.873611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631830+0000","last_change":"2024-03-06T07:26:16.619532+0000","last_active":"2024-03-06T07:26:18.631830+0000","last_peered":"2024-03-06T07:26:18.631830+0000","last_clean":"2024-03-06T07:26:18.631830+0000","last_became_active":"2024-03-06T07:26:16.619261+0000","last_became_peered":"2024-03-06T07:26:16.619261+0000","last_unstale":"2024-03-06T07:26:18.631830+0000","last_undegraded":"2024-03-06T07:26:18.631830+0000","last_fullsized":"2024-03-06T07:26:18.631830+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:34.031068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.632406+0000","last_change":"2024-03-06T07:26:17.637879+0000","last_active":"2024-03-06T07:26:18.632406+0000","last_peered":"2024-03-06T07:26:18.632406+0000","last_clean":"2024-03-06T07:26:18.632406+0000","last_became_active":"2024-03-06T07:26:17.637094+0000","last_became_peered":"2024-03-06T07:26:17.637094+0000","last_unstale":"2024-03-06T07:26:18.632406+0000","last_undegraded":"2024-03-06T07:26:18.632406+0000","last_fullsized":"2024-03-06T07:26:18.632406+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:34.084422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"148'2","reported_seq":16,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403210+0000","last_change":"2024-03-06T07:26:16.622856+0000","last_active":"2024-03-06T07:26:19.403210+0000","last_peered":"2024-03-06T07:26:19.403210+0000","last_clean":"2024-03-06T07:26:19.403210+0000","last_became_active":"2024-03-06T07:26:16.622421+0000","last_became_peered":"2024-03-06T07:26:16.622421+0000","last_unstale":"2024-03-06T07:26:19.403210+0000","last_undegraded":"2024-03-06T07:26:19.403210+0000","last_fullsized":"2024-03-06T07:26:19.403210+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:40.651795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403140+0000","last_change":"2024-03-06T07:26:17.629188+0000","last_active":"2024-03-06T07:26:19.403140+0000","last_peered":"2024-03-06T07:26:19.403140+0000","last_clean":"2024-03-06T07:26:19.403140+0000","last_became_active":"2024-03-06T07:26:17.628881+0000","last_became_peered":"2024-03-06T07:26:17.628881+0000","last_unstale":"2024-03-06T07:26:19.403140+0000","last_undegraded":"2024-03-06T07:26:19.403140+0000","last_fullsized":"2024-03-06T07:26:19.403140+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:44.957682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403926+0000","last_change":"2024-03-06T07:26:16.625755+0000","last_active":"2024-03-06T07:26:19.403926+0000","last_peered":"2024-03-06T07:26:19.403926+0000","last_clean":"2024-03-06T07:26:19.403926+0000","last_became_active":"2024-03-06T07:26:16.625489+0000","last_became_peered":"2024-03-06T07:26:16.625489+0000","last_unstale":"2024-03-06T07:26:19.403926+0000","last_undegraded":"2024-03-06T07:26:19.403926+0000","last_fullsized":"2024-03-06T07:26:19.403926+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:03.053080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403178+0000","last_change":"2024-03-06T07:26:17.629894+0000","last_active":"2024-03-06T07:26:19.403178+0000","last_peered":"2024-03-06T07:26:19.403178+0000","last_clean":"2024-03-06T07:26:19.403178+0000","last_became_active":"2024-03-06T07:26:17.629624+0000","last_became_peered":"2024-03-06T07:26:17.629624+0000","last_unstale":"2024-03-06T07:26:19.403178+0000","last_undegraded":"2024-03-06T07:26:19.403178+0000","last_fullsized":"2024-03-06T07:26:19.403178+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:48.520981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.630390+0000","last_change":"2024-03-06T07:26:16.618167+0000","last_active":"2024-03-06T07:26:18.630390+0000","last_peered":"2024-03-06T07:26:18.630390+0000","last_clean":"2024-03-06T07:26:18.630390+0000","last_became_active":"2024-03-06T07:26:16.617893+0000","last_became_peered":"2024-03-06T07:26:16.617893+0000","last_unstale":"2024-03-06T07:26:18.630390+0000","last_undegraded":"2024-03-06T07:26:18.630390+0000","last_fullsized":"2024-03-06T07:26:18.630390+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:33.738360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.351561+0000","last_change":"2024-03-06T07:26:17.632248+0000","last_active":"2024-03-06T07:26:19.351561+0000","last_peered":"2024-03-06T07:26:19.351561+0000","last_clean":"2024-03-06T07:26:19.351561+0000","last_became_active":"2024-03-06T07:26:17.631912+0000","last_became_peered":"2024-03-06T07:26:17.631912+0000","last_unstale":"2024-03-06T07:26:19.351561+0000","last_undegraded":"2024-03-06T07:26:19.351561+0000","last_fullsized":"2024-03-06T07:26:19.351561+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:29.113308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.351315+0000","last_change":"2024-03-06T07:26:16.616949+0000","last_active":"2024-03-06T07:26:19.351315+0000","last_peered":"2024-03-06T07:26:19.351315+0000","last_clean":"2024-03-06T07:26:19.351315+0000","last_became_active":"2024-03-06T07:26:16.616670+0000","last_became_peered":"2024-03-06T07:26:16.616670+0000","last_unstale":"2024-03-06T07:26:19.351315+0000","last_undegraded":"2024-03-06T07:26:19.351315+0000","last_fullsized":"2024-03-06T07:26:19.351315+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:59.587645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.404102+0000","last_change":"2024-03-06T07:26:17.628994+0000","last_active":"2024-03-06T07:26:19.404102+0000","last_peered":"2024-03-06T07:26:19.404102+0000","last_clean":"2024-03-06T07:26:19.404102+0000","last_became_active":"2024-03-06T07:26:17.628722+0000","last_became_peered":"2024-03-06T07:26:17.628722+0000","last_unstale":"2024-03-06T07:26:19.404102+0000","last_undegraded":"2024-03-06T07:26:19.404102+0000","last_fullsized":"2024-03-06T07:26:19.404102+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:43.090011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"148'2","reported_seq":16,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403287+0000","last_change":"2024-03-06T07:26:16.620011+0000","last_active":"2024-03-06T07:26:19.403287+0000","last_peered":"2024-03-06T07:26:19.403287+0000","last_clean":"2024-03-06T07:26:19.403287+0000","last_became_active":"2024-03-06T07:26:16.619736+0000","last_became_peered":"2024-03-06T07:26:16.619736+0000","last_unstale":"2024-03-06T07:26:19.403287+0000","last_undegraded":"2024-03-06T07:26:19.403287+0000","last_fullsized":"2024-03-06T07:26:19.403287+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:24.169321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631341+0000","last_change":"2024-03-06T07:26:17.628485+0000","last_active":"2024-03-06T07:26:18.631341+0000","last_peered":"2024-03-06T07:26:18.631341+0000","last_clean":"2024-03-06T07:26:18.631341+0000","last_became_active":"2024-03-06T07:26:17.628204+0000","last_became_peered":"2024-03-06T07:26:17.628204+0000","last_unstale":"2024-03-06T07:26:18.631341+0000","last_undegraded":"2024-03-06T07:26:18.631341+0000","last_fullsized":"2024-03-06T07:26:18.631341+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:56.134884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"149'1","reported_seq":15,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.633504+0000","last_change":"2024-03-06T07:26:16.627066+0000","last_active":"2024-03-06T07:26:18.633504+0000","last_peered":"2024-03-06T07:26:18.633504+0000","last_clean":"2024-03-06T07:26:18.633504+0000","last_became_active":"2024-03-06T07:26:16.626418+0000","last_became_peered":"2024-03-06T07:26:16.626418+0000","last_unstale":"2024-03-06T07:26:18.633504+0000","last_undegraded":"2024-03-06T07:26:18.633504+0000","last_fullsized":"2024-03-06T07:26:18.633504+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:37.036435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.630109+0000","last_change":"2024-03-06T07:26:17.626185+0000","last_active":"2024-03-06T07:26:18.630109+0000","last_peered":"2024-03-06T07:26:18.630109+0000","last_clean":"2024-03-06T07:26:18.630109+0000","last_became_active":"2024-03-06T07:26:17.625895+0000","last_became_peered":"2024-03-06T07:26:17.625895+0000","last_unstale":"2024-03-06T07:26:18.630109+0000","last_undegraded":"2024-03-06T07:26:18.630109+0000","last_fullsized":"2024-03-06T07:26:18.630109+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:16.007003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.632035+0000","last_change":"2024-03-06T07:26:16.625471+0000","last_active":"2024-03-06T07:26:18.632035+0000","last_peered":"2024-03-06T07:26:18.632035+0000","last_clean":"2024-03-06T07:26:18.632035+0000","last_became_active":"2024-03-06T07:26:16.625198+0000","last_became_peered":"2024-03-06T07:26:16.625198+0000","last_unstale":"2024-03-06T07:26:18.632035+0000","last_undegraded":"2024-03-06T07:26:18.632035+0000","last_fullsized":"2024-03-06T07:26:18.632035+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:37.984120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631569+0000","last_change":"2024-03-06T07:26:17.630631+0000","last_active":"2024-03-06T07:26:18.631569+0000","last_peered":"2024-03-06T07:26:18.631569+0000","last_clean":"2024-03-06T07:26:18.631569+0000","last_became_active":"2024-03-06T07:26:17.630365+0000","last_became_peered":"2024-03-06T07:26:17.630365+0000","last_unstale":"2024-03-06T07:26:18.631569+0000","last_undegraded":"2024-03-06T07:26:18.631569+0000","last_fullsized":"2024-03-06T07:26:18.631569+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:27.398343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.630530+0000","last_change":"2024-03-06T07:26:16.611334+0000","last_active":"2024-03-06T07:26:18.630530+0000","last_peered":"2024-03-06T07:26:18.630530+0000","last_clean":"2024-03-06T07:26:18.630530+0000","last_became_active":"2024-03-06T07:26:16.610646+0000","last_became_peered":"2024-03-06T07:26:16.610646+0000","last_unstale":"2024-03-06T07:26:18.630530+0000","last_undegraded":"2024-03-06T07:26:18.630530+0000","last_fullsized":"2024-03-06T07:26:18.630530+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:32.639393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.404107+0000","last_change":"2024-03-06T07:26:17.632379+0000","last_active":"2024-03-06T07:26:19.404107+0000","last_peered":"2024-03-06T07:26:19.404107+0000","last_clean":"2024-03-06T07:26:19.404107+0000","last_became_active":"2024-03-06T07:26:17.631899+0000","last_became_peered":"2024-03-06T07:26:17.631899+0000","last_unstale":"2024-03-06T07:26:19.404107+0000","last_undegraded":"2024-03-06T07:26:19.404107+0000","last_fullsized":"2024-03-06T07:26:19.404107+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:00.591780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.403731+0000","last_change":"2024-03-06T07:26:16.623898+0000","last_active":"2024-03-06T07:26:19.403731+0000","last_peered":"2024-03-06T07:26:19.403731+0000","last_clean":"2024-03-06T07:26:19.403731+0000","last_became_active":"2024-03-06T07:26:16.623602+0000","last_became_peered":"2024-03-06T07:26:16.623602+0000","last_unstale":"2024-03-06T07:26:19.403731+0000","last_undegraded":"2024-03-06T07:26:19.403731+0000","last_fullsized":"2024-03-06T07:26:19.403731+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:56.210974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.351471+0000","last_change":"2024-03-06T07:26:17.632856+0000","last_active":"2024-03-06T07:26:19.351471+0000","last_peered":"2024-03-06T07:26:19.351471+0000","last_clean":"2024-03-06T07:26:19.351471+0000","last_became_active":"2024-03-06T07:26:17.632027+0000","last_became_peered":"2024-03-06T07:26:17.632027+0000","last_unstale":"2024-03-06T07:26:19.351471+0000","last_undegraded":"2024-03-06T07:26:19.351471+0000","last_fullsized":"2024-03-06T07:26:19.351471+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:38.988254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.631438+0000","last_change":"2024-03-06T07:26:16.627698+0000","last_active":"2024-03-06T07:26:18.631438+0000","last_peered":"2024-03-06T07:26:18.631438+0000","last_clean":"2024-03-06T07:26:18.631438+0000","last_became_active":"2024-03-06T07:26:16.627318+0000","last_became_peered":"2024-03-06T07:26:16.627318+0000","last_unstale":"2024-03-06T07:26:18.631438+0000","last_undegraded":"2024-03-06T07:26:18.631438+0000","last_fullsized":"2024-03-06T07:26:18.631438+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:08.249739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.632198+0000","last_change":"2024-03-06T07:26:17.637676+0000","last_active":"2024-03-06T07:26:18.632198+0000","last_peered":"2024-03-06T07:26:18.632198+0000","last_clean":"2024-03-06T07:26:18.632198+0000","last_became_active":"2024-03-06T07:26:17.637375+0000","last_became_peered":"2024-03-06T07:26:17.637375+0000","last_unstale":"2024-03-06T07:26:18.632198+0000","last_undegraded":"2024-03-06T07:26:18.632198+0000","last_fullsized":"2024-03-06T07:26:18.632198+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:18.556057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.404788+0000","last_change":"2024-03-06T07:26:16.620202+0000","last_active":"2024-03-06T07:26:19.404788+0000","last_peered":"2024-03-06T07:26:19.404788+0000","last_clean":"2024-03-06T07:26:19.404788+0000","last_became_active":"2024-03-06T07:26:16.619897+0000","last_became_peered":"2024-03-06T07:26:16.619897+0000","last_unstale":"2024-03-06T07:26:19.404788+0000","last_undegraded":"2024-03-06T07:26:19.404788+0000","last_fullsized":"2024-03-06T07:26:19.404788+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:48.709582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.630158+0000","last_change":"2024-03-06T07:26:17.625716+0000","last_active":"2024-03-06T07:26:18.630158+0000","last_peered":"2024-03-06T07:26:18.630158+0000","last_clean":"2024-03-06T07:26:18.630158+0000","last_became_active":"2024-03-06T07:26:17.625429+0000","last_became_peered":"2024-03-06T07:26:17.625429+0000","last_unstale":"2024-03-06T07:26:18.630158+0000","last_undegraded":"2024-03-06T07:26:18.630158+0000","last_fullsized":"2024-03-06T07:26:18.630158+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:38.596798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.402744+0000","last_change":"2024-03-06T07:26:16.620839+0000","last_active":"2024-03-06T07:26:19.402744+0000","last_peered":"2024-03-06T07:26:19.402744+0000","last_clean":"2024-03-06T07:26:19.402744+0000","last_became_active":"2024-03-06T07:26:16.620561+0000","last_became_peered":"2024-03-06T07:26:16.620561+0000","last_unstale":"2024-03-06T07:26:19.402744+0000","last_undegraded":"2024-03-06T07:26:19.402744+0000","last_fullsized":"2024-03-06T07:26:19.402744+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:26.394208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.351049+0000","last_change":"2024-03-06T07:26:17.632976+0000","last_active":"2024-03-06T07:26:19.351049+0000","last_peered":"2024-03-06T07:26:19.351049+0000","last_clean":"2024-03-06T07:26:19.351049+0000","last_became_active":"2024-03-06T07:26:17.632721+0000","last_became_peered":"2024-03-06T07:26:17.632721+0000","last_unstale":"2024-03-06T07:26:19.351049+0000","last_undegraded":"2024-03-06T07:26:19.351049+0000","last_fullsized":"2024-03-06T07:26:19.351049+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:56.134884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.632290+0000","last_change":"2024-03-06T07:26:16.626430+0000","last_active":"2024-03-06T07:26:18.632290+0000","last_peered":"2024-03-06T07:26:18.632290+0000","last_clean":"2024-03-06T07:26:18.632290+0000","last_became_active":"2024-03-06T07:26:16.625971+0000","last_became_peered":"2024-03-06T07:26:16.625971+0000","last_unstale":"2024-03-06T07:26:18.632290+0000","last_undegraded":"2024-03-06T07:26:18.632290+0000","last_fullsized":"2024-03-06T07:26:18.632290+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:28.109173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":417,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.351408+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:26:19.351408+0000","last_peered":"2024-03-06T07:26:19.351408+0000","last_clean":"2024-03-06T07:26:19.351408+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:26:19.351408+0000","last_undegraded":"2024-03-06T07:26:19.351408+0000","last_fullsized":"2024-03-06T07:26:19.351408+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574924,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33064,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338776,"statfs":{"total":96636764160,"available":96602906624,"internally_reserved":0,"allocated":1929216,"data_stored":1489644,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33016,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1875968,"data_stored":905968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574923,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1884160,"data_stored":906092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574924,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33016,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1880064,"data_stored":900040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574923,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33036,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338804,"statfs":{"total":96636764160,"available":96602935296,"internally_reserved":0,"allocated":1896448,"data_stored":908184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574923,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33016,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1875968,"data_stored":905968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1888256,"data_stored":907580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574924,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33072,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338768,"statfs":{"total":96636764160,"available":96602898432,"internally_reserved":0,"allocated":1937408,"data_stored":1489748,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:20.988 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:26:20.988 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:26:21.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.130+0000 7f01871d7640 1 Processor -- start 2024-03-06T07:26:21.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.130+0000 7f01871d7640 1 -- start start 2024-03-06T07:26:21.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.131+0000 7f01871d7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f01800c6b10 0x7f0180155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.131+0000 7f01871d7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01800c2fc0 con 0x7f0180152650 2024-03-06T07:26:21.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.131+0000 7f01871d7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0180158da0 con 0x7f01800bf9d0 2024-03-06T07:26:21.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.131+0000 7f01871d7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0180155800 con 0x7f01800c6b10 2024-03-06T07:26:21.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.131+0000 7f0184f73640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0180152650 0x7f0180158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52730/0 (socket says 172.21.15.1:52730) 2024-03-06T07:26:21.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.131+0000 7f0184f73640 1 -- 172.21.15.1:0/2790487619 learned_addr learned my addr 172.21.15.1:0/2790487619 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017ffff640 1 --2- 172.21.15.1:0/2790487619 >> v2:172.21.15.98:3300/0 conn(0x7f01800c6b10 0x7f0180155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017f7fe640 1 -- 172.21.15.1:0/2790487619 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469010235 0 0) 0x7f01800c2fc0 con 0x7f0180152650 2024-03-06T07:26:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017f7fe640 1 -- 172.21.15.1:0/2790487619 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0168003520 con 0x7f0180152650 2024-03-06T07:26:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017f7fe640 1 -- 172.21.15.1:0/2790487619 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 877417821 0 0) 0x7f0168003520 con 0x7f0180152650 2024-03-06T07:26:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017f7fe640 1 -- 172.21.15.1:0/2790487619 >> v2:172.21.15.98:3300/0 conn(0x7f01800c6b10 msgr2=0x7f0180155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017f7fe640 1 --2- 172.21.15.1:0/2790487619 >> v2:172.21.15.98:3300/0 conn(0x7f01800c6b10 0x7f0180155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017f7fe640 1 -- 172.21.15.1:0/2790487619 >> v1:172.21.15.98:6789/0 conn(0x7f01800bf9d0 legacy=0x7f01800c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017f7fe640 1 -- 172.21.15.1:0/2790487619 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01800c6690 con 0x7f0180152650 2024-03-06T07:26:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017f7fe640 1 -- 172.21.15.1:0/2790487619 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f01780019f0 con 0x7f0180152650 2024-03-06T07:26:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017f7fe640 1 -- 172.21.15.1:0/2790487619 >> v1:172.21.15.1:6789/0 conn(0x7f0180152650 legacy=0x7f0180158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017f7fe640 1 --2- 172.21.15.1:0/2790487619 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01680039b0 0x7f0168003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017f7fe640 1 --2- 172.21.15.1:0/2790487619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0168004800 0x7f0168004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017f7fe640 1 --2- 172.21.15.1:0/2790487619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01680051c0 0x7f016800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017f7fe640 1 -- 172.21.15.1:0/2790487619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0180155800 con 0x7f0168004800 2024-03-06T07:26:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017f7fe640 1 -- 172.21.15.1:0/2790487619 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016800c160 con 0x7f01680051c0 2024-03-06T07:26:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017f7fe640 1 -- 172.21.15.1:0/2790487619 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016800c360 con 0x7f01680039b0 2024-03-06T07:26:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017ffff640 1 --2- 172.21.15.1:0/2790487619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0168004800 0x7f0168004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f0184f73640 1 --2- 172.21.15.1:0/2790487619 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01680039b0 0x7f0168003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f0185774640 1 --2- 172.21.15.1:0/2790487619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01680051c0 0x7f016800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017ffff640 1 -- 172.21.15.1:0/2790487619 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01680039b0 msgr2=0x7f0168003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017ffff640 1 --2- 172.21.15.1:0/2790487619 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01680039b0 0x7f0168003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017ffff640 1 -- 172.21.15.1:0/2790487619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01680051c0 msgr2=0x7f016800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017ffff640 1 --2- 172.21.15.1:0/2790487619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01680051c0 0x7f016800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f017ffff640 1 -- 172.21.15.1:0/2790487619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0168003520 con 0x7f0168004800 2024-03-06T07:26:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.132+0000 7f0185774640 1 -- 172.21.15.1:0/2790487619 reap_dead start 2024-03-06T07:26:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.133+0000 7f017ffff640 1 --2- 172.21.15.1:0/2790487619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0168004800 0x7f0168004c80 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f0174009ac0 tx=0x7f0168005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a76a3685bec1fb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.133+0000 7f017f7fe640 1 -- 172.21.15.1:0/2790487619 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0174039070 con 0x7f0168004800 2024-03-06T07:26:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.133+0000 7f017f7fe640 1 -- 172.21.15.1:0/2790487619 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f017402c6b0 con 0x7f0168004800 2024-03-06T07:26:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.133+0000 7f017f7fe640 1 -- 172.21.15.1:0/2790487619 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0174009690 con 0x7f0168004800 2024-03-06T07:26:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.134+0000 7f01871d7640 1 -- 172.21.15.1:0/2790487619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0168004800 msgr2=0x7f0168004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.134+0000 7f01871d7640 1 --2- 172.21.15.1:0/2790487619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0168004800 0x7f0168004c80 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f0174009ac0 tx=0x7f0168005830 comp rx=0 tx=0).stop 2024-03-06T07:26:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.134+0000 7f01871d7640 1 -- 172.21.15.1:0/2790487619 shutdown_connections 2024-03-06T07:26:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.134+0000 7f01871d7640 1 --2- 172.21.15.1:0/2790487619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0168004800 0x7f0168004c80 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.134+0000 7f01871d7640 1 -- 172.21.15.1:0/2790487619 >> 172.21.15.1:0/2790487619 conn(0x7f0180066ee0 msgr2=0x7f01800672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.134+0000 7f01871d7640 1 -- 172.21.15.1:0/2790487619 shutdown_connections 2024-03-06T07:26:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.134+0000 7f01871d7640 1 -- 172.21.15.1:0/2790487619 wait complete. 2024-03-06T07:26:21.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.135+0000 7f01871d7640 1 Processor -- start 2024-03-06T07:26:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.135+0000 7f01871d7640 1 -- start start 2024-03-06T07:26:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.136+0000 7f01871d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0168004800 0x7f01801eeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.136+0000 7f01871d7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f018014ba40 0x7f0180153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.136+0000 7f01871d7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f018014cde0 0x7f01801eb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.136+0000 7f01871d7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01801551b0 con 0x7f018014ba40 2024-03-06T07:26:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.136+0000 7f01871d7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01800c6fa0 con 0x7f0168004800 2024-03-06T07:26:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.136+0000 7f01871d7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0180155490 con 0x7f018014cde0 2024-03-06T07:26:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.136+0000 7f017ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f018014ba40 0x7f0180153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.136+0000 7f017ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f018014ba40 0x7f0180153b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44704/0 (socket says 172.21.15.1:44704) 2024-03-06T07:26:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.136+0000 7f017ffff640 1 -- 172.21.15.1:0/2315247538 learned_addr learned my addr 172.21.15.1:0/2315247538 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.136+0000 7f0185774640 1 --2- 172.21.15.1:0/2315247538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f018014cde0 0x7f01801eb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.136+0000 7f0184f73640 1 --2- 172.21.15.1:0/2315247538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0168004800 0x7f01801eeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.136+0000 7f017ffff640 1 -- 172.21.15.1:0/2315247538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f018014cde0 msgr2=0x7f01801eb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.136+0000 7f017ffff640 1 --2- 172.21.15.1:0/2315247538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f018014cde0 0x7f01801eb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.136+0000 7f017ffff640 1 -- 172.21.15.1:0/2315247538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0168004800 msgr2=0x7f01801eeed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.136+0000 7f017ffff640 1 --2- 172.21.15.1:0/2315247538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0168004800 0x7f01801eeed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.136+0000 7f017ffff640 1 -- 172.21.15.1:0/2315247538 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01801519d0 con 0x7f018014ba40 2024-03-06T07:26:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.137+0000 7f017ffff640 1 --2- 172.21.15.1:0/2315247538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f018014ba40 0x7f0180153b50 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f0174000c00 tx=0x7f0174004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.137+0000 7f017d7fa640 1 -- 172.21.15.1:0/2315247538 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0174042020 con 0x7f018014ba40 2024-03-06T07:26:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.137+0000 7f017d7fa640 1 -- 172.21.15.1:0/2315247538 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f017403a040 con 0x7f018014ba40 2024-03-06T07:26:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.137+0000 7f017d7fa640 1 -- 172.21.15.1:0/2315247538 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0174039070 con 0x7f018014ba40 2024-03-06T07:26:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.137+0000 7f01871d7640 1 -- 172.21.15.1:0/2315247538 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01801eb170 con 0x7f018014ba40 2024-03-06T07:26:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.137+0000 7f01871d7640 1 -- 172.21.15.1:0/2315247538 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0180154090 con 0x7f018014ba40 2024-03-06T07:26:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.138+0000 7f01871d7640 1 -- 172.21.15.1:0/2315247538 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01801eaf80 con 0x7f018014ba40 2024-03-06T07:26:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.138+0000 7f017d7fa640 1 -- 172.21.15.1:0/2315247538 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0174045020 con 0x7f018014ba40 2024-03-06T07:26:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.139+0000 7f017d7fa640 1 --2- 172.21.15.1:0/2315247538 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f016c06b4d0 0x7f016c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.139+0000 7f017d7fa640 1 -- 172.21.15.1:0/2315247538 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f01740ab770 con 0x7f018014ba40 2024-03-06T07:26:21.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.139+0000 7f0184f73640 1 --2- 172.21.15.1:0/2315247538 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f016c06b4d0 0x7f016c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.139+0000 7f0184f73640 1 --2- 172.21.15.1:0/2315247538 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f016c06b4d0 0x7f016c06d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0178004c20 tx=0x7f0178006040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.143+0000 7f017d7fa640 1 -- 172.21.15.1:0/2315247538 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f017404c070 con 0x7f018014ba40 2024-03-06T07:26:21.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.393+0000 7f01871d7640 1 -- 172.21.15.1:0/2315247538 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f0180067fd0 con 0x7f018014ba40 2024-03-06T07:26:21.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.396+0000 7f017d7fa640 1 -- 172.21.15.1:0/2315247538 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+147 (secure 0 0 0) 0x7f017407a4f0 con 0x7f018014ba40 2024-03-06T07:26:21.398 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-06T07:26:21.398 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng== 2024-03-06T07:26:21.398 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-06T07:26:21.398 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-06T07:26:21.398 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:21.398 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-06T07:26:21.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.398+0000 7f01871d7640 1 -- 172.21.15.1:0/2315247538 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f016c06b4d0 msgr2=0x7f016c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.398+0000 7f01871d7640 1 --2- 172.21.15.1:0/2315247538 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f016c06b4d0 0x7f016c06d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0178004c20 tx=0x7f0178006040 comp rx=0 tx=0).stop 2024-03-06T07:26:21.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.398+0000 7f01871d7640 1 -- 172.21.15.1:0/2315247538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f018014ba40 msgr2=0x7f0180153b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.398+0000 7f01871d7640 1 --2- 172.21.15.1:0/2315247538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f018014ba40 0x7f0180153b50 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f0174000c00 tx=0x7f0174004730 comp rx=0 tx=0).stop 2024-03-06T07:26:21.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.398+0000 7f01871d7640 1 -- 172.21.15.1:0/2315247538 shutdown_connections 2024-03-06T07:26:21.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.398+0000 7f01871d7640 1 --2- 172.21.15.1:0/2315247538 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f016c06b4d0 0x7f016c06d990 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.398+0000 7f01871d7640 1 --2- 172.21.15.1:0/2315247538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f018014cde0 0x7f01801eb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.398+0000 7f01871d7640 1 --2- 172.21.15.1:0/2315247538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f018014ba40 0x7f0180153b50 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.398+0000 7f01871d7640 1 --2- 172.21.15.1:0/2315247538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0168004800 0x7f01801eeed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.398+0000 7f01871d7640 1 -- 172.21.15.1:0/2315247538 >> 172.21.15.1:0/2315247538 conn(0x7f0180066ee0 msgr2=0x7f01800c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.398+0000 7f01871d7640 1 -- 172.21.15.1:0/2315247538 shutdown_connections 2024-03-06T07:26:21.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.398+0000 7f01871d7640 1 -- 172.21.15.1:0/2315247538 wait complete. 2024-03-06T07:26:21.414 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:21.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.513+0000 7fb7212af640 1 Processor -- start 2024-03-06T07:26:21.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.513+0000 7fb7212af640 1 -- start start 2024-03-06T07:26:21.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.514+0000 7fb7212af640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c152730 0x7fb71c0c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.514+0000 7fb7212af640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb71c0c6fd0 0x7fb71c0c1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.514+0000 7fb7212af640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb71c1588e0 con 0x7fb71c0c18d0 2024-03-06T07:26:21.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.514+0000 7fb7212af640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb71c0c1d20 con 0x7fb71c0c6fd0 2024-03-06T07:26:21.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.514+0000 7fb7212af640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c158bc0 con 0x7fb71c152730 2024-03-06T07:26:21.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.514+0000 7fb71affd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c152730 0x7fb71c0c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.514+0000 7fb71a7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb71c0c6fd0 0x7fb71c0c1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.514+0000 7fb71affd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c152730 0x7fb71c0c6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:48230/0 (socket says 172.21.15.1:48230) 2024-03-06T07:26:21.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.514+0000 7fb71affd640 1 -- 172.21.15.1:0/3725830816 learned_addr learned my addr 172.21.15.1:0/3725830816 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:21.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.514+0000 7fb71a7fc640 1 --2- 172.21.15.1:0/3725830816 >> v2:172.21.15.98:3300/0 conn(0x7fb71c0c6fd0 0x7fb71c0c1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:47516/0 (socket says 172.21.15.1:47516) 2024-03-06T07:26:21.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.515+0000 7fb71affd640 1 -- 172.21.15.1:0/3725830816 >> v2:172.21.15.98:3300/0 conn(0x7fb71c0c6fd0 msgr2=0x7fb71c0c1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.515+0000 7fb719ffb640 1 -- 172.21.15.1:0/3725830816 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176521810 0 0) 0x7fb71c1588e0 con 0x7fb71c0c18d0 2024-03-06T07:26:21.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.515+0000 7fb71affd640 1 --2- 172.21.15.1:0/3725830816 >> v2:172.21.15.98:3300/0 conn(0x7fb71c0c6fd0 0x7fb71c0c1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.515+0000 7fb71affd640 1 -- 172.21.15.1:0/3725830816 >> v1:172.21.15.98:6789/0 conn(0x7fb71c0c18d0 legacy=0x7fb71c1552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.515+0000 7fb71affd640 1 -- 172.21.15.1:0/3725830816 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71c158ea0 con 0x7fb71c152730 2024-03-06T07:26:21.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.515+0000 7fb71a7fc640 1 --2- 172.21.15.1:0/3725830816 >> v2:172.21.15.98:3300/0 conn(0x7fb71c0c6fd0 0x7fb71c0c1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:21.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.515+0000 7fb71affd640 1 --2- 172.21.15.1:0/3725830816 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c152730 0x7fb71c0c6a90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb714009870 tx=0x7fb71402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3883718d99521b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.515+0000 7fb719ffb640 1 -- 172.21.15.1:0/3725830816 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7140317c0 con 0x7fb71c152730 2024-03-06T07:26:21.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.516+0000 7fb719ffb640 1 -- 172.21.15.1:0/3725830816 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb714037070 con 0x7fb71c152730 2024-03-06T07:26:21.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.516+0000 7fb719ffb640 1 -- 172.21.15.1:0/3725830816 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb714031b80 con 0x7fb71c152730 2024-03-06T07:26:21.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.516+0000 7fb7212af640 1 -- 172.21.15.1:0/3725830816 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c152730 msgr2=0x7fb71c0c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.516+0000 7fb7212af640 1 --2- 172.21.15.1:0/3725830816 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c152730 0x7fb71c0c6a90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb714009870 tx=0x7fb71402b860 comp rx=0 tx=0).stop 2024-03-06T07:26:21.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.516+0000 7fb7212af640 1 -- 172.21.15.1:0/3725830816 shutdown_connections 2024-03-06T07:26:21.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.516+0000 7fb7212af640 1 --2- 172.21.15.1:0/3725830816 >> v2:172.21.15.98:3300/0 conn(0x7fb71c0c6fd0 0x7fb71c0c1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.516+0000 7fb7212af640 1 --2- 172.21.15.1:0/3725830816 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c152730 0x7fb71c0c6a90 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.516+0000 7fb7212af640 1 -- 172.21.15.1:0/3725830816 >> 172.21.15.1:0/3725830816 conn(0x7fb71c066d00 msgr2=0x7fb71c067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.516+0000 7fb7212af640 1 -- 172.21.15.1:0/3725830816 shutdown_connections 2024-03-06T07:26:21.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.516+0000 7fb7212af640 1 -- 172.21.15.1:0/3725830816 wait complete. 2024-03-06T07:26:21.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.518+0000 7fb7212af640 1 Processor -- start 2024-03-06T07:26:21.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.518+0000 7fb7212af640 1 -- start start 2024-03-06T07:26:21.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.519+0000 7fb7212af640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb71c0c18d0 0x7fb71c153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.519+0000 7fb7212af640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c0c6fd0 0x7fb71c14cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.519+0000 7fb7212af640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c152730 0x7fb71c14d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.519+0000 7fb7212af640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c152b10 con 0x7fb71c152730 2024-03-06T07:26:21.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.519+0000 7fb7212af640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c159460 con 0x7fb71c0c18d0 2024-03-06T07:26:21.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.519+0000 7fb7212af640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c1559f0 con 0x7fb71c0c6fd0 2024-03-06T07:26:21.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.519+0000 7fb71b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c152730 0x7fb71c14d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.519+0000 7fb71b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c152730 0x7fb71c14d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44718/0 (socket says 172.21.15.1:44718) 2024-03-06T07:26:21.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.519+0000 7fb71b7fe640 1 -- 172.21.15.1:0/1362002057 learned_addr learned my addr 172.21.15.1:0/1362002057 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:21.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.520+0000 7fb71a7fc640 1 --2- 172.21.15.1:0/1362002057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c0c6fd0 0x7fb71c14cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.520+0000 7fb71affd640 1 --2- 172.21.15.1:0/1362002057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb71c0c18d0 0x7fb71c153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.520+0000 7fb71b7fe640 1 -- 172.21.15.1:0/1362002057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c0c6fd0 msgr2=0x7fb71c14cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.520+0000 7fb71b7fe640 1 --2- 172.21.15.1:0/1362002057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c0c6fd0 0x7fb71c14cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.520+0000 7fb71b7fe640 1 -- 172.21.15.1:0/1362002057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb71c0c18d0 msgr2=0x7fb71c153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.520+0000 7fb71b7fe640 1 --2- 172.21.15.1:0/1362002057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb71c0c18d0 0x7fb71c153e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.520+0000 7fb71b7fe640 1 -- 172.21.15.1:0/1362002057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71c1df250 con 0x7fb71c152730 2024-03-06T07:26:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.520+0000 7fb71affd640 1 --2- 172.21.15.1:0/1362002057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb71c0c18d0 0x7fb71c153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.520+0000 7fb71a7fc640 1 --2- 172.21.15.1:0/1362002057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c0c6fd0 0x7fb71c14cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.520+0000 7fb71b7fe640 1 --2- 172.21.15.1:0/1362002057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c152730 0x7fb71c14d460 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fb70c00c750 tx=0x7fb70c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.520+0000 7fb7037fe640 1 -- 172.21.15.1:0/1362002057 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c003970 con 0x7fb71c152730 2024-03-06T07:26:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.520+0000 7fb7037fe640 1 -- 172.21.15.1:0/1362002057 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70c003b10 con 0x7fb71c152730 2024-03-06T07:26:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.520+0000 7fb7212af640 1 -- 172.21.15.1:0/1362002057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb71c0c8a70 con 0x7fb71c152730 2024-03-06T07:26:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.520+0000 7fb7037fe640 1 -- 172.21.15.1:0/1362002057 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c00a850 con 0x7fb71c152730 2024-03-06T07:26:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.520+0000 7fb7212af640 1 -- 172.21.15.1:0/1362002057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb71c1de900 con 0x7fb71c152730 2024-03-06T07:26:21.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.521+0000 7fb7212af640 1 -- 172.21.15.1:0/1362002057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb71c0b0870 con 0x7fb71c152730 2024-03-06T07:26:21.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.522+0000 7fb7037fe640 1 -- 172.21.15.1:0/1362002057 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb70c00a9f0 con 0x7fb71c152730 2024-03-06T07:26:21.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.522+0000 7fb7037fe640 1 --2- 172.21.15.1:0/1362002057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb70806b4d0 0x7fb70806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.523+0000 7fb7037fe640 1 -- 172.21.15.1:0/1362002057 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb70c08a1b0 con 0x7fb71c152730 2024-03-06T07:26:21.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.523+0000 7fb71affd640 1 --2- 172.21.15.1:0/1362002057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb70806b4d0 0x7fb70806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.523+0000 7fb71affd640 1 --2- 172.21.15.1:0/1362002057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb70806b4d0 0x7fb70806d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb714036000 tx=0x7fb714008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.531+0000 7fb7037fe640 1 -- 172.21.15.1:0/1362002057 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb71c0b0870 con 0x7fb71c152730 2024-03-06T07:26:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.763+0000 7fb7212af640 1 -- 172.21.15.1:0/1362002057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb71c067fd0 con 0x7fb71c152730 2024-03-06T07:26:21.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.764+0000 7fb7037fe640 1 -- 172.21.15.1:0/1362002057 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+4975 (secure 0 0 0) 0x7fb70c058f30 con 0x7fb71c152730 2024-03-06T07:26:21.766 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:21.766 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":137,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6989,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2360179300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2360179300},{"type":"v1","addr":"172.21.15.98:6837","nonce":2360179300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":7774,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3360241462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3360241462},{"type":"v1","addr":"172.21.15.98:6836","nonce":3360241462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":8820,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/1089548586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1089548586},{"type":"v1","addr":"172.21.15.1:6833","nonce":1089548586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:16.977128+0000","modified":"2024-03-06T07:26:18.975595+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8823},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8823":{"gid":8823,"name":"a","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/67091521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":67091521},{"type":"v1","addr":"172.21.15.1:6835","nonce":67091521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8823,"qdb_cluster":[8823]},"id":11}]} 2024-03-06T07:26:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 137 2024-03-06T07:26:21.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.766+0000 7fb7212af640 1 -- 172.21.15.1:0/1362002057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb70806b4d0 msgr2=0x7fb70806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.766+0000 7fb7212af640 1 --2- 172.21.15.1:0/1362002057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb70806b4d0 0x7fb70806d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb714036000 tx=0x7fb714008030 comp rx=0 tx=0).stop 2024-03-06T07:26:21.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.766+0000 7fb7212af640 1 -- 172.21.15.1:0/1362002057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c152730 msgr2=0x7fb71c14d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.766+0000 7fb7212af640 1 --2- 172.21.15.1:0/1362002057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c152730 0x7fb71c14d460 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fb70c00c750 tx=0x7fb70c00cc10 comp rx=0 tx=0).stop 2024-03-06T07:26:21.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.766+0000 7fb7212af640 1 -- 172.21.15.1:0/1362002057 shutdown_connections 2024-03-06T07:26:21.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.766+0000 7fb7212af640 1 --2- 172.21.15.1:0/1362002057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb70806b4d0 0x7fb70806d990 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.766+0000 7fb7212af640 1 --2- 172.21.15.1:0/1362002057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb71c152730 0x7fb71c14d460 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.766+0000 7fb7212af640 1 --2- 172.21.15.1:0/1362002057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb71c0c6fd0 0x7fb71c14cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.766+0000 7fb7212af640 1 --2- 172.21.15.1:0/1362002057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb71c0c18d0 0x7fb71c153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.766+0000 7fb7212af640 1 -- 172.21.15.1:0/1362002057 >> 172.21.15.1:0/1362002057 conn(0x7fb71c066d00 msgr2=0x7fb71c074450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.766+0000 7fb7212af640 1 -- 172.21.15.1:0/1362002057 shutdown_connections 2024-03-06T07:26:21.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:21.766+0000 7fb7212af640 1 -- 172.21.15.1:0/1362002057 wait complete. 2024-03-06T07:26:22.182 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 137, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:16.977128+0000', 'modified': '2024-03-06T07:26:18.975595+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8823}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8823': {'gid': 8823, 'name': 'a', 'rank': 0, 'incarnation': 135, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/67091521', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 67091521}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 67091521}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8823, 'qdb_cluster': [8823]} 2024-03-06T07:26:22.182 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:26:22.182 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:26:22.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.283+0000 7f778553e640 1 Processor -- start 2024-03-06T07:26:22.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.284+0000 7f778553e640 1 -- start start 2024-03-06T07:26:22.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.285+0000 7f778553e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7780152650 0x7f7780155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.285+0000 7f778553e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f77800bf9d0 0x7f77800c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.285+0000 7f778553e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77800c2fc0 con 0x7f77800c6b10 2024-03-06T07:26:22.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.285+0000 7f778553e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7780155800 con 0x7f77800bf9d0 2024-03-06T07:26:22.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.285+0000 7f778553e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7780158da0 con 0x7f7780152650 2024-03-06T07:26:22.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.285+0000 7f777f7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f77800bf9d0 0x7f77800c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.285+0000 7f777effd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7780152650 0x7f7780155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.285+0000 7f777f7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f77800bf9d0 0x7f77800c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:47542/0 (socket says 172.21.15.1:47542) 2024-03-06T07:26:22.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.285+0000 7f777f7fe640 1 -- 172.21.15.1:0/3903119498 learned_addr learned my addr 172.21.15.1:0/3903119498 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:22.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.285+0000 7f777effd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7780152650 0x7f7780155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:48248/0 (socket says 172.21.15.1:48248) 2024-03-06T07:26:22.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.286+0000 7f777f7fe640 1 -- 172.21.15.1:0/3903119498 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7780152650 msgr2=0x7f7780155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.286+0000 7f777f7fe640 1 --2- 172.21.15.1:0/3903119498 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7780152650 0x7f7780155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.286+0000 7f777f7fe640 1 -- 172.21.15.1:0/3903119498 >> v1:172.21.15.98:6789/0 conn(0x7f77800c6b10 legacy=0x7f7780158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.286+0000 7f777f7fe640 1 -- 172.21.15.1:0/3903119498 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77800c6690 con 0x7f77800bf9d0 2024-03-06T07:26:22.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.286+0000 7f777effd640 1 --2- 172.21.15.1:0/3903119498 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7780152650 0x7f7780155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:22.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.286+0000 7f777f7fe640 1 --2- 172.21.15.1:0/3903119498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77800bf9d0 0x7f77800c1460 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7778009e50 tx=0x7f777802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39b7f594d1d8fa93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.287+0000 7f777dffb640 1 -- 172.21.15.1:0/3903119498 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777802ca70 con 0x7f77800bf9d0 2024-03-06T07:26:22.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.287+0000 7f777dffb640 1 -- 172.21.15.1:0/3903119498 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777802cc10 con 0x7f77800bf9d0 2024-03-06T07:26:22.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.287+0000 7f777dffb640 1 -- 172.21.15.1:0/3903119498 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77780343f0 con 0x7f77800bf9d0 2024-03-06T07:26:22.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.287+0000 7f778553e640 1 -- 172.21.15.1:0/3903119498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77800bf9d0 msgr2=0x7f77800c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.287+0000 7f778553e640 1 --2- 172.21.15.1:0/3903119498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77800bf9d0 0x7f77800c1460 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7778009e50 tx=0x7f777802bed0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.287+0000 7f778553e640 1 -- 172.21.15.1:0/3903119498 shutdown_connections 2024-03-06T07:26:22.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.287+0000 7f778553e640 1 --2- 172.21.15.1:0/3903119498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77800bf9d0 0x7f77800c1460 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.287+0000 7f778553e640 1 --2- 172.21.15.1:0/3903119498 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7780152650 0x7f7780155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.287+0000 7f778553e640 1 -- 172.21.15.1:0/3903119498 >> 172.21.15.1:0/3903119498 conn(0x7f7780066ee0 msgr2=0x7f77800672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.287+0000 7f778553e640 1 -- 172.21.15.1:0/3903119498 shutdown_connections 2024-03-06T07:26:22.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.287+0000 7f778553e640 1 -- 172.21.15.1:0/3903119498 wait complete. 2024-03-06T07:26:22.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.288+0000 7f778553e640 1 Processor -- start 2024-03-06T07:26:22.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.288+0000 7f778553e640 1 -- start start 2024-03-06T07:26:22.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f778553e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77800bf9d0 0x7f7780150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f778553e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7780152650 0x7f7780154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f778553e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f778014b470 0x7f778014a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f778553e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77801551b0 con 0x7f77800bf9d0 2024-03-06T07:26:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f778553e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77800c6fa0 con 0x7f7780152650 2024-03-06T07:26:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f778553e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7780155490 con 0x7f778014b470 2024-03-06T07:26:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f777effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77800bf9d0 0x7f7780150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f777effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77800bf9d0 0x7f7780150e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44722/0 (socket says 172.21.15.1:44722) 2024-03-06T07:26:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f777effd640 1 -- 172.21.15.1:0/1733371012 learned_addr learned my addr 172.21.15.1:0/1733371012 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f777f7fe640 1 --2- 172.21.15.1:0/1733371012 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f778014b470 0x7f778014a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f777e7fc640 1 --2- 172.21.15.1:0/1733371012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7780152650 0x7f7780154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f777effd640 1 -- 172.21.15.1:0/1733371012 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f778014b470 msgr2=0x7f778014a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f777effd640 1 --2- 172.21.15.1:0/1733371012 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f778014b470 0x7f778014a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f777effd640 1 -- 172.21.15.1:0/1733371012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7780152650 msgr2=0x7f7780154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f777effd640 1 --2- 172.21.15.1:0/1733371012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7780152650 0x7f7780154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f777effd640 1 -- 172.21.15.1:0/1733371012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77800c9b20 con 0x7f77800bf9d0 2024-03-06T07:26:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f777f7fe640 1 --2- 172.21.15.1:0/1733371012 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f778014b470 0x7f778014a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:22.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.289+0000 7f777e7fc640 1 --2- 172.21.15.1:0/1733371012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7780152650 0x7f7780154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:22.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.290+0000 7f777effd640 1 --2- 172.21.15.1:0/1733371012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77800bf9d0 0x7f7780150e90 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f777000ebe0 tx=0x7f777000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.290+0000 7f77677fe640 1 -- 172.21.15.1:0/1733371012 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7770017070 con 0x7f77800bf9d0 2024-03-06T07:26:22.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.290+0000 7f77677fe640 1 -- 172.21.15.1:0/1733371012 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777000cc30 con 0x7f77800bf9d0 2024-03-06T07:26:22.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.290+0000 7f778553e640 1 -- 172.21.15.1:0/1733371012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77800c9dc0 con 0x7f77800bf9d0 2024-03-06T07:26:22.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.290+0000 7f77677fe640 1 -- 172.21.15.1:0/1733371012 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7770013470 con 0x7f77800bf9d0 2024-03-06T07:26:22.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.290+0000 7f778553e640 1 -- 172.21.15.1:0/1733371012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77800ca380 con 0x7f77800bf9d0 2024-03-06T07:26:22.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.291+0000 7f77677fe640 1 -- 172.21.15.1:0/1733371012 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f777001a070 con 0x7f77800bf9d0 2024-03-06T07:26:22.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.292+0000 7f77677fe640 1 --2- 172.21.15.1:0/1733371012 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f776c06b4d0 0x7f776c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.292+0000 7f77677fe640 1 -- 172.21.15.1:0/1733371012 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f777008a5f0 con 0x7f77800bf9d0 2024-03-06T07:26:22.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.292+0000 7f777e7fc640 1 --2- 172.21.15.1:0/1733371012 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f776c06b4d0 0x7f776c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.293+0000 7f777e7fc640 1 --2- 172.21.15.1:0/1733371012 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f776c06b4d0 0x7f776c06d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f77801516f0 tx=0x7f7774009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.293+0000 7f7788206740 1 Processor -- start 2024-03-06T07:26:22.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.293+0000 7f7788206740 1 -- start start 2024-03-06T07:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.293+0000 7f7788206740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d224f50 0x555e6d222a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.293+0000 7f7788206740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d222f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.293+0000 7f7788206740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d2234b0 0x555e6d21f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.294+0000 7f7788206740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e6d1896b0 con 0x555e6d225330 2024-03-06T07:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.294+0000 7f7788206740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e6d132a60 con 0x555e6d224f50 2024-03-06T07:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.294+0000 7f7788206740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x555e6d185750 con 0x555e6d2234b0 2024-03-06T07:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.294+0000 7f777e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d222f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.294+0000 7f777e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d222f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44738/0 (socket says 172.21.15.1:44738) 2024-03-06T07:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.294+0000 7f777e7fc640 1 -- 172.21.15.1:0/810281149 learned_addr learned my addr 172.21.15.1:0/810281149 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.294+0000 7f777f7fe640 1 --2- 172.21.15.1:0/810281149 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d2234b0 0x555e6d21f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.294+0000 7f777effd640 1 --2- 172.21.15.1:0/810281149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d224f50 0x555e6d222a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.294+0000 7f777e7fc640 1 -- 172.21.15.1:0/810281149 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d2234b0 msgr2=0x555e6d21f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.294+0000 7f777e7fc640 1 --2- 172.21.15.1:0/810281149 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d2234b0 0x555e6d21f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.294+0000 7f777e7fc640 1 -- 172.21.15.1:0/810281149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d224f50 msgr2=0x555e6d222a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.294+0000 7f777e7fc640 1 --2- 172.21.15.1:0/810281149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d224f50 0x555e6d222a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.294+0000 7f777e7fc640 1 -- 172.21.15.1:0/810281149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e6d150f30 con 0x555e6d225330 2024-03-06T07:26:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.294+0000 7f777e7fc640 1 --2- 172.21.15.1:0/810281149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d222f70 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f777400d800 tx=0x7f7774010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4eb46d9b147d661 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.294+0000 7f7764ff9640 1 -- 172.21.15.1:0/810281149 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7774024070 con 0x555e6d225330 2024-03-06T07:26:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.294+0000 7f7764ff9640 1 -- 172.21.15.1:0/810281149 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7774014ba0 con 0x555e6d225330 2024-03-06T07:26:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.294+0000 7f7764ff9640 1 -- 172.21.15.1:0/810281149 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7774007050 con 0x555e6d225330 2024-03-06T07:26:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.295+0000 7f7788206740 1 -- 172.21.15.1:0/810281149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 msgr2=0x555e6d222f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.295+0000 7f7788206740 1 --2- 172.21.15.1:0/810281149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d222f70 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f777400d800 tx=0x7f7774010000 comp rx=0 tx=0).stop 2024-03-06T07:26:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.295+0000 7f7788206740 1 -- 172.21.15.1:0/810281149 shutdown_connections 2024-03-06T07:26:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.295+0000 7f7788206740 1 --2- 172.21.15.1:0/810281149 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d2234b0 0x555e6d21f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.295+0000 7f7788206740 1 --2- 172.21.15.1:0/810281149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d222f70 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.295+0000 7f7788206740 1 --2- 172.21.15.1:0/810281149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d224f50 0x555e6d222a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.295+0000 7f7788206740 1 -- 172.21.15.1:0/810281149 >> 172.21.15.1:0/810281149 conn(0x555e6d2217d0 msgr2=0x555e6d2463c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.295+0000 7f7788206740 1 -- 172.21.15.1:0/810281149 shutdown_connections 2024-03-06T07:26:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.295+0000 7f7788206740 1 -- 172.21.15.1:0/810281149 wait complete. 2024-03-06T07:26:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.296+0000 7f7788206740 1 -- start start 2024-03-06T07:26:22.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.296+0000 7f7788206740 1 Processor -- start 2024-03-06T07:26:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.296+0000 7f7788206740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d224f50 0x555e6d2457b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.296+0000 7f7788206740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d225330 0x555e6d24b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.296+0000 7f7788206740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d274ad0 0x555e6d241ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.296+0000 7f7788206740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e6d247230 con 0x555e6d224f50 2024-03-06T07:26:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.296+0000 7f7788206740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e6d145ea0 con 0x555e6d274ad0 2024-03-06T07:26:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.296+0000 7f7788206740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x555e6d246c70 con 0x555e6d225330 2024-03-06T07:26:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.296+0000 7f777effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d224f50 0x555e6d2457b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.296+0000 7f7788206740 10 client.0 _ll_register_callbacks cb 0x555e6d145870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f777effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d224f50 0x555e6d2457b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44740/0 (socket says 172.21.15.1:44740) 2024-03-06T07:26:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f777effd640 1 -- 172.21.15.1:0/460321259 learned_addr learned my addr 172.21.15.1:0/460321259 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f777e7fc640 1 --2- 172.21.15.1:0/460321259 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d225330 0x555e6d24b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f777f7fe640 1 --2- 172.21.15.1:0/460321259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d274ad0 0x555e6d241ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f777effd640 1 -- 172.21.15.1:0/460321259 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d225330 msgr2=0x555e6d24b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f777effd640 1 --2- 172.21.15.1:0/460321259 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d225330 0x555e6d24b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f777effd640 1 -- 172.21.15.1:0/460321259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d274ad0 msgr2=0x555e6d241ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f777effd640 1 --2- 172.21.15.1:0/460321259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d274ad0 0x555e6d241ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f777effd640 1 -- 172.21.15.1:0/460321259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e6d146270 con 0x555e6d224f50 2024-03-06T07:26:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f777f7fe640 1 --2- 172.21.15.1:0/460321259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d274ad0 0x555e6d241ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f777effd640 1 --2- 172.21.15.1:0/460321259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d224f50 0x555e6d2457b0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f777000f010 tx=0x7f777005f0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9560c336b1dff7ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f775dffb640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:26:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f775dffb640 1 -- 172.21.15.1:0/460321259 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7770070020 con 0x555e6d224f50 2024-03-06T07:26:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f775dffb640 1 -- 172.21.15.1:0/460321259 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f77700582c0 con 0x555e6d224f50 2024-03-06T07:26:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f7788206740 1 -- 172.21.15.1:0/460321259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7774014ba0 con 0x555e6d224f50 2024-03-06T07:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f775dffb640 1 -- 172.21.15.1:0/460321259 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=137) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7770075050 con 0x555e6d224f50 2024-03-06T07:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f7788206740 10 client.8964 fetch_fsmap learned FSMap version 137 2024-03-06T07:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f7788206740 1 -- 172.21.15.1:0/460321259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=137,monmap=2+}) v3 -- 0x555e6ce9e2d0 con 0x555e6d224f50 2024-03-06T07:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f775dffb640 1 -- 172.21.15.1:0/460321259 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7774014ba0 con 0x555e6d224f50 2024-03-06T07:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f775dffb640 1 -- 172.21.15.1:0/460321259 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 137) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f777005dce0 con 0x555e6d224f50 2024-03-06T07:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.297+0000 7f7788206740 10 client.8964 fetch_fsmap finished waiting for FSMap version 137 2024-03-06T07:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.298+0000 7f7788206740 10 client.8964 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.298+0000 7f7788206740 20 client.8964 populate_metadata read hostname 'smithi001' 2024-03-06T07:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.298+0000 7f7788206740 1 --2- 172.21.15.1:0/460321259 >> [v2:172.21.15.1:6834/67091521,v1:172.21.15.1:6835/67091521] conn(0x555e6d248000 0x555e6d280110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.298+0000 7f7788206740 4 client.8964 mds_command: new command op to 8823 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.298+0000 7f7788206740 1 -- 172.21.15.1:0/460321259 --> [v2:172.21.15.1:6834/67091521,v1:172.21.15.1:6835/67091521] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e6d196bf0 con 0x555e6d248000 2024-03-06T07:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.298+0000 7f777e7fc640 1 --2- 172.21.15.1:0/460321259 >> [v2:172.21.15.1:6834/67091521,v1:172.21.15.1:6835/67091521] conn(0x555e6d248000 0x555e6d280110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.298+0000 7f775dffb640 10 client.8964 ms_handle_connect on v2:172.21.15.1:6834/67091521 2024-03-06T07:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.298+0000 7f777e7fc640 1 --2- 172.21.15.1:0/460321259 >> [v2:172.21.15.1:6834/67091521,v1:172.21.15.1:6835/67091521] conn(0x555e6d248000 0x555e6d280110 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=806760d484710a2e server_cookie=46cc1c5be98495d0 in_seq=0 out_seq=0 2024-03-06T07:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.299+0000 7f775dffb640 1 -- 172.21.15.1:0/460321259 <== mds.0 v2:172.21.15.1:6834/67091521 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555e6d196bf0 con 0x555e6d248000 2024-03-06T07:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.299+0000 7f775dffb640 10 client.8964 handle_command_reply: tid=0 2024-03-06T07:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.299+0000 7f7788206740 1 client.8964 shutdown 2024-03-06T07:26:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.299+0000 7f7788206740 1 -- 172.21.15.1:0/460321259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d224f50 msgr2=0x555e6d2457b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.299+0000 7f7788206740 1 --2- 172.21.15.1:0/460321259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d224f50 0x555e6d2457b0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f777000f010 tx=0x7f777005f0b0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.299+0000 7f7788206740 1 -- 172.21.15.1:0/460321259 shutdown_connections 2024-03-06T07:26:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.299+0000 7f7788206740 1 --2- 172.21.15.1:0/460321259 >> [v2:172.21.15.1:6834/67091521,v1:172.21.15.1:6835/67091521] conn(0x555e6d248000 0x555e6d280110 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.299+0000 7f7788206740 1 --2- 172.21.15.1:0/460321259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d274ad0 0x555e6d241ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.299+0000 7f7788206740 1 --2- 172.21.15.1:0/460321259 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d225330 0x555e6d24b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.299+0000 7f7788206740 1 --2- 172.21.15.1:0/460321259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d224f50 0x555e6d2457b0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.299+0000 7f775dffb640 0 client.8964 ms_handle_reset on v2:172.21.15.1:6834/67091521 2024-03-06T07:26:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.299+0000 7f7788206740 1 -- 172.21.15.1:0/460321259 >> 172.21.15.1:0/460321259 conn(0x555e6d2234b0 msgr2=0x555e6d224380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.299+0000 7f7788206740 1 -- 172.21.15.1:0/460321259 shutdown_connections 2024-03-06T07:26:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.299+0000 7f7788206740 1 -- 172.21.15.1:0/460321259 wait complete. 2024-03-06T07:26:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.299+0000 7f7788206740 20 client.8964 trim_cache size 0 max 16384 2024-03-06T07:26:22.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.316+0000 7f7788206740 1 Processor -- start 2024-03-06T07:26:22.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.316+0000 7f7788206740 1 -- start start 2024-03-06T07:26:22.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.316+0000 7f7788206740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d224f50 0x555e6d24b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.316+0000 7f7788206740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d223890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.316+0000 7f7788206740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d274ad0 0x555e6d21f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.316+0000 7f7788206740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77801f6ec0 con 0x555e6d225330 2024-03-06T07:26:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.316+0000 7f7788206740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77801eed90 con 0x555e6d274ad0 2024-03-06T07:26:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.316+0000 7f7788206740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77801f68c0 con 0x555e6d224f50 2024-03-06T07:26:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.316+0000 7f777e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d223890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.316+0000 7f777e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d223890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44750/0 (socket says 172.21.15.1:44750) 2024-03-06T07:26:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.316+0000 7f777e7fc640 1 -- 172.21.15.1:0/3096438918 learned_addr learned my addr 172.21.15.1:0/3096438918 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f777e7fc640 1 -- 172.21.15.1:0/3096438918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d224f50 msgr2=0x555e6d24b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f777e7fc640 1 --2- 172.21.15.1:0/3096438918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d224f50 0x555e6d24b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f777e7fc640 1 -- 172.21.15.1:0/3096438918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d274ad0 msgr2=0x555e6d21f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f777e7fc640 1 --2- 172.21.15.1:0/3096438918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d274ad0 0x555e6d21f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f777e7fc640 1 -- 172.21.15.1:0/3096438918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e6cf686b0 con 0x555e6d225330 2024-03-06T07:26:22.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f777e7fc640 1 --2- 172.21.15.1:0/3096438918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d223890 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f777400d7d0 tx=0x7f7774012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a69163fe8c38b3e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f775ffff640 1 -- 172.21.15.1:0/3096438918 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7774024040 con 0x555e6d225330 2024-03-06T07:26:22.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f775ffff640 1 -- 172.21.15.1:0/3096438918 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7774025040 con 0x555e6d225330 2024-03-06T07:26:22.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f775ffff640 1 -- 172.21.15.1:0/3096438918 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7774030030 con 0x555e6d225330 2024-03-06T07:26:22.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f7788206740 1 -- 172.21.15.1:0/3096438918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 msgr2=0x555e6d223890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f7788206740 1 --2- 172.21.15.1:0/3096438918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d223890 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f777400d7d0 tx=0x7f7774012000 comp rx=0 tx=0).stop 2024-03-06T07:26:22.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f7788206740 1 -- 172.21.15.1:0/3096438918 shutdown_connections 2024-03-06T07:26:22.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f7788206740 1 --2- 172.21.15.1:0/3096438918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d274ad0 0x555e6d21f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f7788206740 1 --2- 172.21.15.1:0/3096438918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d223890 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f7788206740 1 --2- 172.21.15.1:0/3096438918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d224f50 0x555e6d24b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f7788206740 1 -- 172.21.15.1:0/3096438918 >> 172.21.15.1:0/3096438918 conn(0x555e6d2234b0 msgr2=0x555e6d21e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f7788206740 1 -- 172.21.15.1:0/3096438918 shutdown_connections 2024-03-06T07:26:22.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.317+0000 7f7788206740 1 -- 172.21.15.1:0/3096438918 wait complete. 2024-03-06T07:26:22.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.318+0000 7f7788206740 1 -- start start 2024-03-06T07:26:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.318+0000 7f7788206740 1 Processor -- start 2024-03-06T07:26:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f7788206740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d224f50 0x555e6d21f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f7788206740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d2d1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f7788206740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d274ad0 0x555e6d2d45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f7788206740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e6d249b40 con 0x555e6d225330 2024-03-06T07:26:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f7788206740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e6d2428d0 con 0x555e6d274ad0 2024-03-06T07:26:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f7788206740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x555e6d2753f0 con 0x555e6d224f50 2024-03-06T07:26:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f7788206740 10 client.0 _ll_register_callbacks cb 0x7f7770055000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f777e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d2d1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f777e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d2d1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44752/0 (socket says 172.21.15.1:44752) 2024-03-06T07:26:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f777e7fc640 1 -- 172.21.15.1:0/3790694187 learned_addr learned my addr 172.21.15.1:0/3790694187 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f777f7fe640 1 --2- 172.21.15.1:0/3790694187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d274ad0 0x555e6d2d45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f777effd640 1 --2- 172.21.15.1:0/3790694187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d224f50 0x555e6d21f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f777e7fc640 1 -- 172.21.15.1:0/3790694187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d224f50 msgr2=0x555e6d21f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f777e7fc640 1 --2- 172.21.15.1:0/3790694187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d224f50 0x555e6d21f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f777e7fc640 1 -- 172.21.15.1:0/3790694187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d274ad0 msgr2=0x555e6d2d45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f777e7fc640 1 --2- 172.21.15.1:0/3790694187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d274ad0 0x555e6d2d45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f777e7fc640 1 -- 172.21.15.1:0/3790694187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e6d243f00 con 0x555e6d225330 2024-03-06T07:26:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f777effd640 1 --2- 172.21.15.1:0/3790694187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d224f50 0x555e6d21f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f777e7fc640 1 --2- 172.21.15.1:0/3790694187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d2d1050 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x555e6d2645d0 tx=0x7f7774000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aacb58376cf0f1d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f775dffb640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:26:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f775dffb640 1 -- 172.21.15.1:0/3790694187 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7774024040 con 0x555e6d225330 2024-03-06T07:26:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f775dffb640 1 -- 172.21.15.1:0/3790694187 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f77740148a0 con 0x555e6d225330 2024-03-06T07:26:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.319+0000 7f7788206740 1 -- 172.21.15.1:0/3790694187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7774025040 con 0x555e6d225330 2024-03-06T07:26:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.320+0000 7f775dffb640 1 -- 172.21.15.1:0/3790694187 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=137) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7774042050 con 0x555e6d225330 2024-03-06T07:26:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.320+0000 7f7788206740 10 client.8970 fetch_fsmap learned FSMap version 137 2024-03-06T07:26:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.320+0000 7f7788206740 1 -- 172.21.15.1:0/3790694187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=137,monmap=2+}) v3 -- 0x555e6d25d1b0 con 0x555e6d225330 2024-03-06T07:26:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.320+0000 7f775dffb640 1 -- 172.21.15.1:0/3790694187 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7774025040 con 0x555e6d225330 2024-03-06T07:26:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.320+0000 7f775dffb640 1 -- 172.21.15.1:0/3790694187 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 137) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f7774045020 con 0x555e6d225330 2024-03-06T07:26:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.320+0000 7f7788206740 10 client.8970 fetch_fsmap finished waiting for FSMap version 137 2024-03-06T07:26:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.320+0000 7f7788206740 10 client.8970 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.320+0000 7f7788206740 20 client.8970 populate_metadata read hostname 'smithi001' 2024-03-06T07:26:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.320+0000 7f7788206740 1 --2- 172.21.15.1:0/3790694187 >> [v2:172.21.15.1:6834/67091521,v1:172.21.15.1:6835/67091521] conn(0x555e6d2d4b70 0x555e6d2d4f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.320+0000 7f7788206740 4 client.8970 mds_command: new command op to 8823 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.320+0000 7f7788206740 1 -- 172.21.15.1:0/3790694187 --> [v2:172.21.15.1:6834/67091521,v1:172.21.15.1:6835/67091521] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555e6d190f40 con 0x555e6d2d4b70 2024-03-06T07:26:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.320+0000 7f777effd640 1 --2- 172.21.15.1:0/3790694187 >> [v2:172.21.15.1:6834/67091521,v1:172.21.15.1:6835/67091521] conn(0x555e6d2d4b70 0x555e6d2d4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.320+0000 7f775dffb640 10 client.8970 ms_handle_connect on v2:172.21.15.1:6834/67091521 2024-03-06T07:26:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.320+0000 7f777effd640 1 --2- 172.21.15.1:0/3790694187 >> [v2:172.21.15.1:6834/67091521,v1:172.21.15.1:6835/67091521] conn(0x555e6d2d4b70 0x555e6d2d4f50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e79e21bedf17a12 server_cookie=22c3e05ec86fddc6 in_seq=0 out_seq=0 2024-03-06T07:26:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.320+0000 7f775dffb640 1 -- 172.21.15.1:0/3790694187 <== mds.0 v2:172.21.15.1:6834/67091521 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x555e6d190f40 con 0x555e6d2d4b70 2024-03-06T07:26:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.320+0000 7f775dffb640 10 client.8970 handle_command_reply: tid=0 2024-03-06T07:26:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.320+0000 7f7788206740 1 client.8970 shutdown 2024-03-06T07:26:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f7788206740 1 -- 172.21.15.1:0/3790694187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 msgr2=0x555e6d2d1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f7788206740 1 --2- 172.21.15.1:0/3790694187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d2d1050 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x555e6d2645d0 tx=0x7f7774000bc0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f7788206740 1 -- 172.21.15.1:0/3790694187 shutdown_connections 2024-03-06T07:26:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f7788206740 1 --2- 172.21.15.1:0/3790694187 >> [v2:172.21.15.1:6834/67091521,v1:172.21.15.1:6835/67091521] conn(0x555e6d2d4b70 0x555e6d2d4f50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f7788206740 1 --2- 172.21.15.1:0/3790694187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555e6d274ad0 0x555e6d2d45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f7788206740 1 --2- 172.21.15.1:0/3790694187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555e6d225330 0x555e6d2d1050 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f7788206740 1 --2- 172.21.15.1:0/3790694187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555e6d224f50 0x555e6d21f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f7788206740 1 -- 172.21.15.1:0/3790694187 >> 172.21.15.1:0/3790694187 conn(0x555e6d2234b0 msgr2=0x555e6d223890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f775dffb640 0 client.8970 ms_handle_reset on v2:172.21.15.1:6834/67091521 2024-03-06T07:26:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f7788206740 1 -- 172.21.15.1:0/3790694187 shutdown_connections 2024-03-06T07:26:22.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f7788206740 1 -- 172.21.15.1:0/3790694187 wait complete. 2024-03-06T07:26:22.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f7788206740 20 client.8970 trim_cache size 0 max 16384 2024-03-06T07:26:22.329 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:26:22.329 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:26:22.329 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:26:22.329 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 8823, 2024-03-06T07:26:22.329 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:26:22.329 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:26:22.329 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:26:22.329 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 4.6980821070000003, 2024-03-06T07:26:22.329 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 137, 2024-03-06T07:26:22.330 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 149, 2024-03-06T07:26:22.330 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 149, 2024-03-06T07:26:22.330 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 9.7929306740000008 2024-03-06T07:26:22.330 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:26:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f778553e640 1 -- 172.21.15.1:0/1733371012 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f776c06b4d0 msgr2=0x7f776c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f778553e640 1 --2- 172.21.15.1:0/1733371012 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f776c06b4d0 0x7f776c06d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f77801516f0 tx=0x7f7774009040 comp rx=0 tx=0).stop 2024-03-06T07:26:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f778553e640 1 -- 172.21.15.1:0/1733371012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77800bf9d0 msgr2=0x7f7780150e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f778553e640 1 --2- 172.21.15.1:0/1733371012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77800bf9d0 0x7f7780150e90 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f777000ebe0 tx=0x7f777000c520 comp rx=0 tx=0).stop 2024-03-06T07:26:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f778553e640 1 -- 172.21.15.1:0/1733371012 shutdown_connections 2024-03-06T07:26:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f778553e640 1 --2- 172.21.15.1:0/1733371012 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f776c06b4d0 0x7f776c06d990 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f778553e640 1 --2- 172.21.15.1:0/1733371012 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f778014b470 0x7f778014a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f778553e640 1 --2- 172.21.15.1:0/1733371012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7780152650 0x7f7780154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f778553e640 1 --2- 172.21.15.1:0/1733371012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77800bf9d0 0x7f7780150e90 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f778553e640 1 -- 172.21.15.1:0/1733371012 >> 172.21.15.1:0/1733371012 conn(0x7f7780066ee0 msgr2=0x7f77800c3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f778553e640 1 -- 172.21.15.1:0/1733371012 shutdown_connections 2024-03-06T07:26:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.321+0000 7f778553e640 1 -- 172.21.15.1:0/1733371012 wait complete. 2024-03-06T07:26:22.334 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:22.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.434+0000 7f539076b640 1 Processor -- start 2024-03-06T07:26:22.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.434+0000 7f539076b640 1 -- start start 2024-03-06T07:26:22.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.434+0000 7f539076b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5388152650 0x7f5388155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.434+0000 7f539076b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f53880c6b10 0x7f5388158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.435+0000 7f539076b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53880c2fc0 con 0x7f53880bf9d0 2024-03-06T07:26:22.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.435+0000 7f539076b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5388155800 con 0x7f5388152650 2024-03-06T07:26:22.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.435+0000 7f539076b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53880c6690 con 0x7f53880c6b10 2024-03-06T07:26:22.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.435+0000 7f538e507640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5388152650 0x7f5388155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.435+0000 7f538e507640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5388152650 0x7f5388155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44766/0 (socket says 172.21.15.1:44766) 2024-03-06T07:26:22.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.435+0000 7f538e507640 1 -- 172.21.15.1:0/3443458849 learned_addr learned my addr 172.21.15.1:0/3443458849 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:22.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.435+0000 7f538dd06640 1 --2- 172.21.15.1:0/3443458849 >> v2:172.21.15.98:3300/0 conn(0x7f53880c6b10 0x7f5388158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.435+0000 7f538e507640 1 -- 172.21.15.1:0/3443458849 >> v2:172.21.15.98:3300/0 conn(0x7f53880c6b10 msgr2=0x7f5388158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.435+0000 7f538e507640 1 --2- 172.21.15.1:0/3443458849 >> v2:172.21.15.98:3300/0 conn(0x7f53880c6b10 0x7f5388158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.435+0000 7f538e507640 1 -- 172.21.15.1:0/3443458849 >> v1:172.21.15.98:6789/0 conn(0x7f53880bf9d0 legacy=0x7f53880c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.435+0000 7f538e507640 1 -- 172.21.15.1:0/3443458849 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53880c7280 con 0x7f5388152650 2024-03-06T07:26:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.436+0000 7f538e507640 1 --2- 172.21.15.1:0/3443458849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5388152650 0x7f5388155c90 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f53880cb8e0 tx=0x7f538402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58deb8a528df7219 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.436+0000 7f538d505640 1 -- 172.21.15.1:0/3443458849 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538402ca80 con 0x7f5388152650 2024-03-06T07:26:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.436+0000 7f538d505640 1 -- 172.21.15.1:0/3443458849 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538402cc20 con 0x7f5388152650 2024-03-06T07:26:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.436+0000 7f538d505640 1 -- 172.21.15.1:0/3443458849 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53840343f0 con 0x7f5388152650 2024-03-06T07:26:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.436+0000 7f539076b640 1 -- 172.21.15.1:0/3443458849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5388152650 msgr2=0x7f5388155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.436+0000 7f539076b640 1 --2- 172.21.15.1:0/3443458849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5388152650 0x7f5388155c90 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f53880cb8e0 tx=0x7f538402bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.437+0000 7f539076b640 1 -- 172.21.15.1:0/3443458849 shutdown_connections 2024-03-06T07:26:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.437+0000 7f539076b640 1 --2- 172.21.15.1:0/3443458849 >> v2:172.21.15.98:3300/0 conn(0x7f53880c6b10 0x7f5388158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.437+0000 7f539076b640 1 --2- 172.21.15.1:0/3443458849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5388152650 0x7f5388155c90 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.437+0000 7f539076b640 1 -- 172.21.15.1:0/3443458849 >> 172.21.15.1:0/3443458849 conn(0x7f5388066ee0 msgr2=0x7f53880672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.437+0000 7f539076b640 1 -- 172.21.15.1:0/3443458849 shutdown_connections 2024-03-06T07:26:22.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.437+0000 7f539076b640 1 -- 172.21.15.1:0/3443458849 wait complete. 2024-03-06T07:26:22.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.438+0000 7f539076b640 1 Processor -- start 2024-03-06T07:26:22.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.438+0000 7f539076b640 1 -- start start 2024-03-06T07:26:22.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.438+0000 7f539076b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53880bf9d0 0x7f53880c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.438+0000 7f539076b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5388152650 0x7f53880c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.438+0000 7f539076b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53881e22f0 0x7f53881ee6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.438+0000 7f539076b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5388155490 con 0x7f53880bf9d0 2024-03-06T07:26:22.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.438+0000 7f539076b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53881551b0 con 0x7f5388152650 2024-03-06T07:26:22.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.438+0000 7f539076b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5388152a30 con 0x7f53881e22f0 2024-03-06T07:26:22.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.438+0000 7f538e507640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53880bf9d0 0x7f53880c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.438+0000 7f538e507640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53880bf9d0 0x7f53880c95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44768/0 (socket says 172.21.15.1:44768) 2024-03-06T07:26:22.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.438+0000 7f538e507640 1 -- 172.21.15.1:0/1090771478 learned_addr learned my addr 172.21.15.1:0/1090771478 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:22.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.439+0000 7f538ed08640 1 --2- 172.21.15.1:0/1090771478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53881e22f0 0x7f53881ee6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.439+0000 7f538dd06640 1 --2- 172.21.15.1:0/1090771478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5388152650 0x7f53880c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.439+0000 7f538e507640 1 -- 172.21.15.1:0/1090771478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53881e22f0 msgr2=0x7f53881ee6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.439+0000 7f538e507640 1 --2- 172.21.15.1:0/1090771478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53881e22f0 0x7f53881ee6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.439+0000 7f538e507640 1 -- 172.21.15.1:0/1090771478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5388152650 msgr2=0x7f53880c9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.439+0000 7f538e507640 1 --2- 172.21.15.1:0/1090771478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5388152650 0x7f53880c9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.439+0000 7f538e507640 1 -- 172.21.15.1:0/1090771478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5388154200 con 0x7f53880bf9d0 2024-03-06T07:26:22.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.439+0000 7f538e507640 1 --2- 172.21.15.1:0/1090771478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53880bf9d0 0x7f53880c95b0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f5384036040 tx=0x7f5384004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.439+0000 7f537effd640 1 -- 172.21.15.1:0/1090771478 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384043020 con 0x7f53880bf9d0 2024-03-06T07:26:22.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.439+0000 7f537effd640 1 -- 172.21.15.1:0/1090771478 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538403e450 con 0x7f53880bf9d0 2024-03-06T07:26:22.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.439+0000 7f539076b640 1 -- 172.21.15.1:0/1090771478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5388153c40 con 0x7f53880bf9d0 2024-03-06T07:26:22.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.439+0000 7f537effd640 1 -- 172.21.15.1:0/1090771478 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384009510 con 0x7f53880bf9d0 2024-03-06T07:26:22.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.439+0000 7f539076b640 1 -- 172.21.15.1:0/1090771478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53880c8db0 con 0x7f53880bf9d0 2024-03-06T07:26:22.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.440+0000 7f539076b640 1 -- 172.21.15.1:0/1090771478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53880b06f0 con 0x7f53880bf9d0 2024-03-06T07:26:22.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.440+0000 7f537effd640 1 -- 172.21.15.1:0/1090771478 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5384046020 con 0x7f53880bf9d0 2024-03-06T07:26:22.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.441+0000 7f537effd640 1 --2- 172.21.15.1:0/1090771478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f536c06b4d0 0x7f536c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.441+0000 7f537effd640 1 -- 172.21.15.1:0/1090771478 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f53840acb90 con 0x7f53880bf9d0 2024-03-06T07:26:22.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.441+0000 7f538dd06640 1 --2- 172.21.15.1:0/1090771478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f536c06b4d0 0x7f536c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.442+0000 7f538dd06640 1 --2- 172.21.15.1:0/1090771478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f536c06b4d0 0x7f536c06d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5378006fd0 tx=0x7f5378008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.445+0000 7f537effd640 1 -- 172.21.15.1:0/1090771478 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f538404d030 con 0x7f53880bf9d0 2024-03-06T07:26:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.626+0000 7f539076b640 1 -- 172.21.15.1:0/1090771478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53880676e0 con 0x7f53880bf9d0 2024-03-06T07:26:22.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.627+0000 7f537effd640 1 -- 172.21.15.1:0/1090771478 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+16764 (secure 0 0 0) 0x7f538407b990 con 0x7f53880bf9d0 2024-03-06T07:26:22.629 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:22.629 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":149,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:26:18.622325+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:15.245426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:15.942343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.629+0000 7f539076b640 1 -- 172.21.15.1:0/1090771478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f536c06b4d0 msgr2=0x7f536c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.629+0000 7f539076b640 1 --2- 172.21.15.1:0/1090771478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f536c06b4d0 0x7f536c06d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5378006fd0 tx=0x7f5378008040 comp rx=0 tx=0).stop 2024-03-06T07:26:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.629+0000 7f539076b640 1 -- 172.21.15.1:0/1090771478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53880bf9d0 msgr2=0x7f53880c95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.629+0000 7f539076b640 1 --2- 172.21.15.1:0/1090771478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53880bf9d0 0x7f53880c95b0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f5384036040 tx=0x7f5384004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.629+0000 7f539076b640 1 -- 172.21.15.1:0/1090771478 shutdown_connections 2024-03-06T07:26:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.629+0000 7f539076b640 1 --2- 172.21.15.1:0/1090771478 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f536c06b4d0 0x7f536c06d990 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.629+0000 7f539076b640 1 --2- 172.21.15.1:0/1090771478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53881e22f0 0x7f53881ee6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.629+0000 7f539076b640 1 --2- 172.21.15.1:0/1090771478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5388152650 0x7f53880c9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.629+0000 7f539076b640 1 --2- 172.21.15.1:0/1090771478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53880bf9d0 0x7f53880c95b0 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.629+0000 7f539076b640 1 -- 172.21.15.1:0/1090771478 >> 172.21.15.1:0/1090771478 conn(0x7f5388066ee0 msgr2=0x7f538814fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.629+0000 7f539076b640 1 -- 172.21.15.1:0/1090771478 shutdown_connections 2024-03-06T07:26:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.629+0000 7f539076b640 1 -- 172.21.15.1:0/1090771478 wait complete. 2024-03-06T07:26:22.642 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:26:15.245426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:26:15.942343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '149', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:22.643 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:22.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.745+0000 7fd2d0f32640 1 Processor -- start 2024-03-06T07:26:22.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.745+0000 7fd2d0f32640 1 -- start start 2024-03-06T07:26:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.745+0000 7fd2d0f32640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2cc150520 0x7fd2cc1543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.745+0000 7fd2d0f32640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd2cc070c80 0x7fd2cc0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.745+0000 7fd2d0f32640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2cc150900 con 0x7fd2cc0c32f0 2024-03-06T07:26:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.746+0000 7fd2d0f32640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2cc153ed0 con 0x7fd2cc070c80 2024-03-06T07:26:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.746+0000 7fd2d0f32640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc071790 con 0x7fd2cc150520 2024-03-06T07:26:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.746+0000 7fd2cad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd2cc070c80 0x7fd2cc0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.746+0000 7fd2cad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd2cc070c80 0x7fd2cc0710f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44780/0 (socket says 172.21.15.1:44780) 2024-03-06T07:26:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.746+0000 7fd2cad9d640 1 -- 172.21.15.1:0/2449014737 learned_addr learned my addr 172.21.15.1:0/2449014737 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.746+0000 7fd2ca59c640 1 --2- 172.21.15.1:0/2449014737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2cc150520 0x7fd2cc1543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.746+0000 7fd2cad9d640 1 -- 172.21.15.1:0/2449014737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2cc150520 msgr2=0x7fd2cc1543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.746+0000 7fd2cad9d640 1 --2- 172.21.15.1:0/2449014737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2cc150520 0x7fd2cc1543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.746+0000 7fd2cad9d640 1 -- 172.21.15.1:0/2449014737 >> v1:172.21.15.98:6789/0 conn(0x7fd2cc0c32f0 legacy=0x7fd2cc1537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.746+0000 7fd2cad9d640 1 -- 172.21.15.1:0/2449014737 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2cc157050 con 0x7fd2cc070c80 2024-03-06T07:26:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.746+0000 7fd2ca59c640 1 --2- 172.21.15.1:0/2449014737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2cc150520 0x7fd2cc1543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.747+0000 7fd2cad9d640 1 --2- 172.21.15.1:0/2449014737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2cc070c80 0x7fd2cc0710f0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4009e50 tx=0x7fd2c402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef88cbea3e0dad2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.747+0000 7fd2c959a640 1 -- 172.21.15.1:0/2449014737 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c402cb00 con 0x7fd2cc070c80 2024-03-06T07:26:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.747+0000 7fd2c959a640 1 -- 172.21.15.1:0/2449014737 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c402cca0 con 0x7fd2cc070c80 2024-03-06T07:26:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.747+0000 7fd2c959a640 1 -- 172.21.15.1:0/2449014737 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4034610 con 0x7fd2cc070c80 2024-03-06T07:26:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.747+0000 7fd2d0f32640 1 -- 172.21.15.1:0/2449014737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2cc070c80 msgr2=0x7fd2cc0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.747+0000 7fd2d0f32640 1 --2- 172.21.15.1:0/2449014737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2cc070c80 0x7fd2cc0710f0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4009e50 tx=0x7fd2c402bf60 comp rx=0 tx=0).stop 2024-03-06T07:26:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.748+0000 7fd2d0f32640 1 -- 172.21.15.1:0/2449014737 shutdown_connections 2024-03-06T07:26:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.748+0000 7fd2d0f32640 1 --2- 172.21.15.1:0/2449014737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2cc070c80 0x7fd2cc0710f0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.748+0000 7fd2d0f32640 1 --2- 172.21.15.1:0/2449014737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2cc150520 0x7fd2cc1543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.748+0000 7fd2d0f32640 1 -- 172.21.15.1:0/2449014737 >> 172.21.15.1:0/2449014737 conn(0x7fd2cc0c88f0 msgr2=0x7fd2cc0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.748+0000 7fd2d0f32640 1 -- 172.21.15.1:0/2449014737 shutdown_connections 2024-03-06T07:26:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.748+0000 7fd2d0f32640 1 -- 172.21.15.1:0/2449014737 wait complete. 2024-03-06T07:26:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.749+0000 7fd2d0f32640 1 Processor -- start 2024-03-06T07:26:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.749+0000 7fd2d0f32640 1 -- start start 2024-03-06T07:26:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.749+0000 7fd2d0f32640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2cc070c80 0x7fd2cc145cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.749+0000 7fd2d0f32640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2cc0c32f0 0x7fd2cc1559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.749+0000 7fd2d0f32640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2cc150520 0x7fd2cc0a88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2d0f32640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc157330 con 0x7fd2cc150520 2024-03-06T07:26:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2d0f32640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc1532a0 con 0x7fd2cc070c80 2024-03-06T07:26:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2d0f32640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc0c7ec0 con 0x7fd2cc0c32f0 2024-03-06T07:26:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2cad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2cc150520 0x7fd2cc0a88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2cad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2cc150520 0x7fd2cc0a88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44788/0 (socket says 172.21.15.1:44788) 2024-03-06T07:26:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2cad9d640 1 -- 172.21.15.1:0/761151627 learned_addr learned my addr 172.21.15.1:0/761151627 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2ca59c640 1 --2- 172.21.15.1:0/761151627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2cc070c80 0x7fd2cc145cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2c9d9b640 1 --2- 172.21.15.1:0/761151627 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2cc0c32f0 0x7fd2cc1559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2cad9d640 1 -- 172.21.15.1:0/761151627 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2cc0c32f0 msgr2=0x7fd2cc1559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2cad9d640 1 --2- 172.21.15.1:0/761151627 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2cc0c32f0 0x7fd2cc1559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2cad9d640 1 -- 172.21.15.1:0/761151627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2cc070c80 msgr2=0x7fd2cc145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2cad9d640 1 --2- 172.21.15.1:0/761151627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2cc070c80 0x7fd2cc145cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2cad9d640 1 -- 172.21.15.1:0/761151627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2cc0ac950 con 0x7fd2cc150520 2024-03-06T07:26:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2cad9d640 1 --2- 172.21.15.1:0/761151627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2cc150520 0x7fd2cc0a88f0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4009f70 tx=0x7fd2c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2baffd640 1 -- 172.21.15.1:0/761151627 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4043020 con 0x7fd2cc150520 2024-03-06T07:26:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2baffd640 1 -- 172.21.15.1:0/761151627 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c402ce50 con 0x7fd2cc150520 2024-03-06T07:26:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2d0f32640 1 -- 172.21.15.1:0/761151627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2cc1ecce0 con 0x7fd2cc150520 2024-03-06T07:26:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2baffd640 1 -- 172.21.15.1:0/761151627 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4009510 con 0x7fd2cc150520 2024-03-06T07:26:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.750+0000 7fd2d0f32640 1 -- 172.21.15.1:0/761151627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2cc0acef0 con 0x7fd2cc150520 2024-03-06T07:26:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.751+0000 7fd2d0f32640 1 -- 172.21.15.1:0/761151627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2cc06e900 con 0x7fd2cc150520 2024-03-06T07:26:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.752+0000 7fd2baffd640 1 -- 172.21.15.1:0/761151627 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2c4046020 con 0x7fd2cc150520 2024-03-06T07:26:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.753+0000 7fd2baffd640 1 --2- 172.21.15.1:0/761151627 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd2ac06b500 0x7fd2ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.753+0000 7fd2baffd640 1 -- 172.21.15.1:0/761151627 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd2c40ac940 con 0x7fd2cc150520 2024-03-06T07:26:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.753+0000 7fd2ca59c640 1 --2- 172.21.15.1:0/761151627 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd2ac06b500 0x7fd2ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.754+0000 7fd2ca59c640 1 --2- 172.21.15.1:0/761151627 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd2ac06b500 0x7fd2ac06d9c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc006fd0 tx=0x7fd2bc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.757+0000 7fd2baffd640 1 -- 172.21.15.1:0/761151627 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2cc06e900 con 0x7fd2cc150520 2024-03-06T07:26:22.903 INFO:tasks.ceph.mds.a.smithi001.stderr:2024-03-06T07:26:22.901+0000 7fade67ea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:22.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.956+0000 7fd2d0f32640 1 -- 172.21.15.1:0/761151627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd2cc0c7b10 con 0x7fd2cc150520 2024-03-06T07:26:22.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.956+0000 7fd2baffd640 1 -- 172.21.15.1:0/761151627 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v149) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd2c404d030 con 0x7fd2cc150520 2024-03-06T07:26:22.958 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:26:22.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.958+0000 7fd2d0f32640 1 -- 172.21.15.1:0/761151627 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd2ac06b500 msgr2=0x7fd2ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.958+0000 7fd2d0f32640 1 --2- 172.21.15.1:0/761151627 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd2ac06b500 0x7fd2ac06d9c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc006fd0 tx=0x7fd2bc008040 comp rx=0 tx=0).stop 2024-03-06T07:26:22.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.958+0000 7fd2d0f32640 1 -- 172.21.15.1:0/761151627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2cc150520 msgr2=0x7fd2cc0a88f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.958+0000 7fd2d0f32640 1 --2- 172.21.15.1:0/761151627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2cc150520 0x7fd2cc0a88f0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4009f70 tx=0x7fd2c4004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:22.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.958+0000 7fd2d0f32640 1 -- 172.21.15.1:0/761151627 shutdown_connections 2024-03-06T07:26:22.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.958+0000 7fd2d0f32640 1 --2- 172.21.15.1:0/761151627 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd2ac06b500 0x7fd2ac06d9c0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.958+0000 7fd2d0f32640 1 --2- 172.21.15.1:0/761151627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2cc150520 0x7fd2cc0a88f0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.958+0000 7fd2d0f32640 1 --2- 172.21.15.1:0/761151627 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2cc0c32f0 0x7fd2cc1559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.958+0000 7fd2d0f32640 1 --2- 172.21.15.1:0/761151627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2cc070c80 0x7fd2cc145cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.958+0000 7fd2d0f32640 1 -- 172.21.15.1:0/761151627 >> 172.21.15.1:0/761151627 conn(0x7fd2cc0c88f0 msgr2=0x7fd2cc071060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.958+0000 7fd2d0f32640 1 -- 172.21.15.1:0/761151627 shutdown_connections 2024-03-06T07:26:22.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:22.958+0000 7fd2d0f32640 1 -- 172.21.15.1:0/761151627 wait complete. 2024-03-06T07:26:22.971 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:23.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.073+0000 7f1189b16640 1 Processor -- start 2024-03-06T07:26:23.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.073+0000 7f1189b16640 1 -- start start 2024-03-06T07:26:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.074+0000 7f1189b16640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1184152650 0x7f1184155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.074+0000 7f1189b16640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f11840bf9d0 0x7f11840c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.074+0000 7f1189b16640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11840c2fc0 con 0x7f11840c6b10 2024-03-06T07:26:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.074+0000 7f1189b16640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1184155800 con 0x7f1184152650 2024-03-06T07:26:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.074+0000 7f1189b16640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1184158da0 con 0x7f11840bf9d0 2024-03-06T07:26:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.074+0000 7f11837fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1184152650 0x7f1184155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.074+0000 7f11837fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1184152650 0x7f1184155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44796/0 (socket says 172.21.15.1:44796) 2024-03-06T07:26:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.074+0000 7f11837fe640 1 -- 172.21.15.1:0/2746171827 learned_addr learned my addr 172.21.15.1:0/2746171827 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.074+0000 7f1182ffd640 1 --1- 172.21.15.1:0/2746171827 >> v1:172.21.15.1:6789/0 conn(0x7f11840c6b10 0x7f1184158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52732/0 (socket says 172.21.15.1:52732) 2024-03-06T07:26:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.074+0000 7f1183fff640 1 --2- 172.21.15.1:0/2746171827 >> v2:172.21.15.98:3300/0 conn(0x7f11840bf9d0 0x7f11840c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.074+0000 7f11837fe640 1 -- 172.21.15.1:0/2746171827 >> v2:172.21.15.98:3300/0 conn(0x7f11840bf9d0 msgr2=0x7f11840c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.074+0000 7f11837fe640 1 --2- 172.21.15.1:0/2746171827 >> v2:172.21.15.98:3300/0 conn(0x7f11840bf9d0 0x7f11840c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.074+0000 7f11837fe640 1 -- 172.21.15.1:0/2746171827 >> v1:172.21.15.1:6789/0 conn(0x7f11840c6b10 legacy=0x7f1184158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.074+0000 7f11827fc640 1 -- 172.21.15.1:0/2746171827 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248685650 0 0) 0x7f11840c2fc0 con 0x7f11840c6b10 2024-03-06T07:26:23.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.074+0000 7f11837fe640 1 -- 172.21.15.1:0/2746171827 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11840c6690 con 0x7f1184152650 2024-03-06T07:26:23.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.075+0000 7f11837fe640 1 --2- 172.21.15.1:0/2746171827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1184152650 0x7f1184155c90 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f11841597b0 tx=0x7f117402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ee30f9e7ec86e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.076+0000 7f11827fc640 1 -- 172.21.15.1:0/2746171827 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174035020 con 0x7f1184152650 2024-03-06T07:26:23.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.076+0000 7f11827fc640 1 -- 172.21.15.1:0/2746171827 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1174031870 con 0x7f1184152650 2024-03-06T07:26:23.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.076+0000 7f11827fc640 1 -- 172.21.15.1:0/2746171827 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174031c40 con 0x7f1184152650 2024-03-06T07:26:23.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.076+0000 7f1189b16640 1 -- 172.21.15.1:0/2746171827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1184152650 msgr2=0x7f1184155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.076+0000 7f1189b16640 1 --2- 172.21.15.1:0/2746171827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1184152650 0x7f1184155c90 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f11841597b0 tx=0x7f117402b910 comp rx=0 tx=0).stop 2024-03-06T07:26:23.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.076+0000 7f1189b16640 1 -- 172.21.15.1:0/2746171827 shutdown_connections 2024-03-06T07:26:23.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.076+0000 7f1189b16640 1 --2- 172.21.15.1:0/2746171827 >> v2:172.21.15.98:3300/0 conn(0x7f11840bf9d0 0x7f11840c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.076+0000 7f1189b16640 1 --2- 172.21.15.1:0/2746171827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1184152650 0x7f1184155c90 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.076+0000 7f1189b16640 1 -- 172.21.15.1:0/2746171827 >> 172.21.15.1:0/2746171827 conn(0x7f1184066ee0 msgr2=0x7f11840672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.076+0000 7f1189b16640 1 -- 172.21.15.1:0/2746171827 shutdown_connections 2024-03-06T07:26:23.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.076+0000 7f1189b16640 1 -- 172.21.15.1:0/2746171827 wait complete. 2024-03-06T07:26:23.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.078+0000 7f1189b16640 1 Processor -- start 2024-03-06T07:26:23.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.078+0000 7f1189b16640 1 -- start start 2024-03-06T07:26:23.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f1189b16640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11840bf9d0 0x7f11841e1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f1189b16640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1184152650 0x7f1184150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f1189b16640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f118414d770 0x7f11841ee230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f1189b16640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11841551b0 con 0x7f11840bf9d0 2024-03-06T07:26:23.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f1189b16640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11840c6fa0 con 0x7f118414d770 2024-03-06T07:26:23.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f1189b16640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1184155490 con 0x7f1184152650 2024-03-06T07:26:23.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f11837fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11840bf9d0 0x7f11841e1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f11837fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11840bf9d0 0x7f11841e1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44802/0 (socket says 172.21.15.1:44802) 2024-03-06T07:26:23.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f11837fe640 1 -- 172.21.15.1:0/3456645774 learned_addr learned my addr 172.21.15.1:0/3456645774 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:23.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f1183fff640 1 --2- 172.21.15.1:0/3456645774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f118414d770 0x7f11841ee230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f1182ffd640 1 --2- 172.21.15.1:0/3456645774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1184152650 0x7f1184150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f11837fe640 1 -- 172.21.15.1:0/3456645774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1184152650 msgr2=0x7f1184150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f11837fe640 1 --2- 172.21.15.1:0/3456645774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1184152650 0x7f1184150fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f11837fe640 1 -- 172.21.15.1:0/3456645774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f118414d770 msgr2=0x7f11841ee230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f11837fe640 1 --2- 172.21.15.1:0/3456645774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f118414d770 0x7f11841ee230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f11837fe640 1 -- 172.21.15.1:0/3456645774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1184154890 con 0x7f11840bf9d0 2024-03-06T07:26:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f1182ffd640 1 --2- 172.21.15.1:0/3456645774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1184152650 0x7f1184150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f1183fff640 1 --2- 172.21.15.1:0/3456645774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f118414d770 0x7f11841ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f11837fe640 1 --2- 172.21.15.1:0/3456645774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11840bf9d0 0x7f11841e1e90 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f1174000c00 tx=0x7f11740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f116bfff640 1 -- 172.21.15.1:0/3456645774 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174041070 con 0x7f11840bf9d0 2024-03-06T07:26:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f116bfff640 1 -- 172.21.15.1:0/3456645774 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1174004030 con 0x7f11840bf9d0 2024-03-06T07:26:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f1189b16640 1 -- 172.21.15.1:0/3456645774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1184154b00 con 0x7f11840bf9d0 2024-03-06T07:26:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f116bfff640 1 -- 172.21.15.1:0/3456645774 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174035040 con 0x7f11840bf9d0 2024-03-06T07:26:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.079+0000 7f1189b16640 1 -- 172.21.15.1:0/3456645774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11841514e0 con 0x7f11840bf9d0 2024-03-06T07:26:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.081+0000 7f1189b16640 1 -- 172.21.15.1:0/3456645774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1184067fd0 con 0x7f11840bf9d0 2024-03-06T07:26:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.081+0000 7f116bfff640 1 -- 172.21.15.1:0/3456645774 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1174038020 con 0x7f11840bf9d0 2024-03-06T07:26:23.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.081+0000 7f116bfff640 1 --2- 172.21.15.1:0/3456645774 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f117006b500 0x7f117006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.081+0000 7f116bfff640 1 -- 172.21.15.1:0/3456645774 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f11740ac3c0 con 0x7f11840bf9d0 2024-03-06T07:26:23.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.082+0000 7f1182ffd640 1 --2- 172.21.15.1:0/3456645774 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f117006b500 0x7f117006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.082+0000 7f1182ffd640 1 --2- 172.21.15.1:0/3456645774 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f117006b500 0x7f117006d9c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f11840c0d20 tx=0x7f1178009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.090+0000 7f116bfff640 1 -- 172.21.15.1:0/3456645774 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1174043350 con 0x7f11840bf9d0 2024-03-06T07:26:23.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.268+0000 7f1189b16640 1 -- 172.21.15.1:0/3456645774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f11840b06f0 con 0x7f11840bf9d0 2024-03-06T07:26:23.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.269+0000 7f116bfff640 1 -- 172.21.15.1:0/3456645774 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v149) v1 ==== 104+0+11 (secure 0 0 0) 0x7f117407b140 con 0x7f11840bf9d0 2024-03-06T07:26:23.271 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:26:23.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.271+0000 7f1189b16640 1 -- 172.21.15.1:0/3456645774 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f117006b500 msgr2=0x7f117006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.271+0000 7f1189b16640 1 --2- 172.21.15.1:0/3456645774 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f117006b500 0x7f117006d9c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f11840c0d20 tx=0x7f1178009040 comp rx=0 tx=0).stop 2024-03-06T07:26:23.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.271+0000 7f1189b16640 1 -- 172.21.15.1:0/3456645774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11840bf9d0 msgr2=0x7f11841e1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.271+0000 7f1189b16640 1 --2- 172.21.15.1:0/3456645774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11840bf9d0 0x7f11841e1e90 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f1174000c00 tx=0x7f11740046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.271+0000 7f1189b16640 1 -- 172.21.15.1:0/3456645774 shutdown_connections 2024-03-06T07:26:23.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.271+0000 7f1189b16640 1 --2- 172.21.15.1:0/3456645774 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f117006b500 0x7f117006d9c0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.271+0000 7f1189b16640 1 --2- 172.21.15.1:0/3456645774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f118414d770 0x7f11841ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.271+0000 7f1189b16640 1 --2- 172.21.15.1:0/3456645774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1184152650 0x7f1184150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.271+0000 7f1189b16640 1 --2- 172.21.15.1:0/3456645774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11840bf9d0 0x7f11841e1e90 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.271+0000 7f1189b16640 1 -- 172.21.15.1:0/3456645774 >> 172.21.15.1:0/3456645774 conn(0x7f1184066ee0 msgr2=0x7f11840c5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.271+0000 7f1189b16640 1 -- 172.21.15.1:0/3456645774 shutdown_connections 2024-03-06T07:26:23.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.271+0000 7f1189b16640 1 -- 172.21.15.1:0/3456645774 wait complete. 2024-03-06T07:26:23.284 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:23.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.391+0000 7f1898cc9640 1 Processor -- start 2024-03-06T07:26:23.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.391+0000 7f1898cc9640 1 -- start start 2024-03-06T07:26:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f1898cc9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18940c6b10 0x7f1894155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f1898cc9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f18940bf9d0 0x7f18940c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f1898cc9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18940c2fc0 con 0x7f1894152650 2024-03-06T07:26:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f1898cc9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1894155800 con 0x7f18940bf9d0 2024-03-06T07:26:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f1898cc9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1894158da0 con 0x7f18940c6b10 2024-03-06T07:26:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f189259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1894152650 0x7f1894158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52734/0 (socket says 172.21.15.1:52734) 2024-03-06T07:26:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f189259c640 1 -- 172.21.15.1:0/158925076 learned_addr learned my addr 172.21.15.1:0/158925076 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f1892d9d640 1 --2- 172.21.15.1:0/158925076 >> v2:172.21.15.98:3300/0 conn(0x7f18940bf9d0 0x7f18940c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f1891d9b640 1 --2- 172.21.15.1:0/158925076 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18940c6b10 0x7f1894155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f189159a640 1 -- 172.21.15.1:0/158925076 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929145059 0 0) 0x7f18940c2fc0 con 0x7f1894152650 2024-03-06T07:26:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f189159a640 1 -- 172.21.15.1:0/158925076 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f187c003520 con 0x7f1894152650 2024-03-06T07:26:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f189159a640 1 -- 172.21.15.1:0/158925076 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 492937586 0 0) 0x7f187c003520 con 0x7f1894152650 2024-03-06T07:26:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f189159a640 1 -- 172.21.15.1:0/158925076 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18940c6b10 msgr2=0x7f1894155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f189159a640 1 --2- 172.21.15.1:0/158925076 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18940c6b10 0x7f1894155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f189159a640 1 -- 172.21.15.1:0/158925076 >> v2:172.21.15.98:3300/0 conn(0x7f18940bf9d0 msgr2=0x7f18940c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f189159a640 1 --2- 172.21.15.1:0/158925076 >> v2:172.21.15.98:3300/0 conn(0x7f18940bf9d0 0x7f18940c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f189159a640 1 -- 172.21.15.1:0/158925076 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18940c6690 con 0x7f1894152650 2024-03-06T07:26:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f189159a640 1 -- 172.21.15.1:0/158925076 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f1884001670 con 0x7f1894152650 2024-03-06T07:26:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.392+0000 7f189159a640 1 -- 172.21.15.1:0/158925076 >> v1:172.21.15.1:6789/0 conn(0x7f1894152650 legacy=0x7f1894158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.393+0000 7f189159a640 1 --2- 172.21.15.1:0/158925076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f187c0039b0 0x7f187c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.393+0000 7f189159a640 1 --2- 172.21.15.1:0/158925076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f187c004800 0x7f187c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.393+0000 7f189159a640 1 --2- 172.21.15.1:0/158925076 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f187c0051c0 0x7f187c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.393+0000 7f189159a640 1 -- 172.21.15.1:0/158925076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1894155800 con 0x7f187c004800 2024-03-06T07:26:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.393+0000 7f189159a640 1 -- 172.21.15.1:0/158925076 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1894158da0 con 0x7f187c0039b0 2024-03-06T07:26:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.393+0000 7f189159a640 1 -- 172.21.15.1:0/158925076 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f187c00bf00 con 0x7f187c0051c0 2024-03-06T07:26:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.393+0000 7f1891d9b640 1 --2- 172.21.15.1:0/158925076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f187c004800 0x7f187c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.393+0000 7f1892d9d640 1 --2- 172.21.15.1:0/158925076 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f187c0051c0 0x7f187c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.393+0000 7f189259c640 1 --2- 172.21.15.1:0/158925076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f187c0039b0 0x7f187c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.393+0000 7f1891d9b640 1 -- 172.21.15.1:0/158925076 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f187c0051c0 msgr2=0x7f187c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.393+0000 7f1891d9b640 1 --2- 172.21.15.1:0/158925076 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f187c0051c0 0x7f187c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.393+0000 7f1891d9b640 1 -- 172.21.15.1:0/158925076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f187c0039b0 msgr2=0x7f187c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.393+0000 7f1891d9b640 1 --2- 172.21.15.1:0/158925076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f187c0039b0 0x7f187c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.393+0000 7f1891d9b640 1 -- 172.21.15.1:0/158925076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187c003520 con 0x7f187c004800 2024-03-06T07:26:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.393+0000 7f189259c640 1 --2- 172.21.15.1:0/158925076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f187c0039b0 0x7f187c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.394+0000 7f1891d9b640 1 --2- 172.21.15.1:0/158925076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f187c004800 0x7f187c004c80 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f1888009ac0 tx=0x7f187c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53fb3fc1d84210d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.394+0000 7f189159a640 1 -- 172.21.15.1:0/158925076 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1888039070 con 0x7f187c004800 2024-03-06T07:26:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.394+0000 7f189159a640 1 -- 172.21.15.1:0/158925076 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188802c6b0 con 0x7f187c004800 2024-03-06T07:26:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.394+0000 7f189159a640 1 -- 172.21.15.1:0/158925076 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1888009690 con 0x7f187c004800 2024-03-06T07:26:23.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.394+0000 7f1898cc9640 1 -- 172.21.15.1:0/158925076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f187c004800 msgr2=0x7f187c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.394+0000 7f1898cc9640 1 --2- 172.21.15.1:0/158925076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f187c004800 0x7f187c004c80 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f1888009ac0 tx=0x7f187c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:23.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.394+0000 7f1892d9d640 1 -- 172.21.15.1:0/158925076 reap_dead start 2024-03-06T07:26:23.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.395+0000 7f1898cc9640 1 -- 172.21.15.1:0/158925076 shutdown_connections 2024-03-06T07:26:23.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.395+0000 7f1898cc9640 1 -- 172.21.15.1:0/158925076 >> 172.21.15.1:0/158925076 conn(0x7f1894066ee0 msgr2=0x7f18940672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.395+0000 7f1898cc9640 1 -- 172.21.15.1:0/158925076 shutdown_connections 2024-03-06T07:26:23.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.395+0000 7f1898cc9640 1 -- 172.21.15.1:0/158925076 wait complete. 2024-03-06T07:26:23.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.396+0000 7f1898cc9640 1 Processor -- start 2024-03-06T07:26:23.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.396+0000 7f1898cc9640 1 -- start start 2024-03-06T07:26:23.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.396+0000 7f1898cc9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f187c004800 0x7f18941501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.396+0000 7f1898cc9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1894151b40 0x7f189414a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.396+0000 7f1898cc9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f189414a810 0x7f1894152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.396+0000 7f1898cc9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18941551b0 con 0x7f189414a810 2024-03-06T07:26:23.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.396+0000 7f1898cc9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18940c6fa0 con 0x7f1894151b40 2024-03-06T07:26:23.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.396+0000 7f1898cc9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1894155490 con 0x7f187c004800 2024-03-06T07:26:23.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.397+0000 7f1892d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f189414a810 0x7f1894152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.397+0000 7f1892d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f189414a810 0x7f1894152ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44816/0 (socket says 172.21.15.1:44816) 2024-03-06T07:26:23.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.397+0000 7f1892d9d640 1 -- 172.21.15.1:0/1502802446 learned_addr learned my addr 172.21.15.1:0/1502802446 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:23.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.397+0000 7f1891d9b640 1 --2- 172.21.15.1:0/1502802446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1894151b40 0x7f189414a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.397+0000 7f189259c640 1 --2- 172.21.15.1:0/1502802446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f187c004800 0x7f18941501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.397+0000 7f1892d9d640 1 -- 172.21.15.1:0/1502802446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f187c004800 msgr2=0x7f18941501a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.397+0000 7f1892d9d640 1 --2- 172.21.15.1:0/1502802446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f187c004800 0x7f18941501a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.397+0000 7f1892d9d640 1 -- 172.21.15.1:0/1502802446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1894151b40 msgr2=0x7f189414a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.397+0000 7f1892d9d640 1 --2- 172.21.15.1:0/1502802446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1894151b40 0x7f189414a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.397+0000 7f1892d9d640 1 -- 172.21.15.1:0/1502802446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18940c03d0 con 0x7f189414a810 2024-03-06T07:26:23.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.397+0000 7f1892d9d640 1 --2- 172.21.15.1:0/1502802446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f189414a810 0x7f1894152ff0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f189408c760 tx=0x7f188c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.397+0000 7f1882ffd640 1 -- 172.21.15.1:0/1502802446 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c00fde0 con 0x7f189414a810 2024-03-06T07:26:23.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.397+0000 7f1882ffd640 1 -- 172.21.15.1:0/1502802446 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188c01d070 con 0x7f189414a810 2024-03-06T07:26:23.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.397+0000 7f1898cc9640 1 -- 172.21.15.1:0/1502802446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1894156e30 con 0x7f189414a810 2024-03-06T07:26:23.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.397+0000 7f1882ffd640 1 -- 172.21.15.1:0/1502802446 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c015670 con 0x7f189414a810 2024-03-06T07:26:23.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.397+0000 7f1898cc9640 1 -- 172.21.15.1:0/1502802446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18941570f0 con 0x7f189414a810 2024-03-06T07:26:23.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.398+0000 7f1898cc9640 1 -- 172.21.15.1:0/1502802446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1894067fd0 con 0x7f189414a810 2024-03-06T07:26:23.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.399+0000 7f1882ffd640 1 -- 172.21.15.1:0/1502802446 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f188c015810 con 0x7f189414a810 2024-03-06T07:26:23.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.399+0000 7f1882ffd640 1 --2- 172.21.15.1:0/1502802446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f187406b500 0x7f187406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.399+0000 7f1882ffd640 1 -- 172.21.15.1:0/1502802446 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f188c08abc0 con 0x7f189414a810 2024-03-06T07:26:23.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.400+0000 7f189259c640 1 --2- 172.21.15.1:0/1502802446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f187406b500 0x7f187406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.400+0000 7f189259c640 1 --2- 172.21.15.1:0/1502802446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f187406b500 0x7f187406d9c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f1884001eb0 tx=0x7f1884007040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.404+0000 7f1882ffd640 1 -- 172.21.15.1:0/1502802446 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f18940c03d0 con 0x7f189414a810 2024-03-06T07:26:23.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.629+0000 7f1898cc9640 1 -- 172.21.15.1:0/1502802446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f18940c68f0 con 0x7f189414a810 2024-03-06T07:26:23.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.630+0000 7f1882ffd640 1 -- 172.21.15.1:0/1502802446 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v149) v1 ==== 100+0+11 (secure 0 0 0) 0x7f188c010040 con 0x7f189414a810 2024-03-06T07:26:23.632 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:26:23.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.631+0000 7f1898cc9640 1 -- 172.21.15.1:0/1502802446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f187406b500 msgr2=0x7f187406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.631+0000 7f1898cc9640 1 --2- 172.21.15.1:0/1502802446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f187406b500 0x7f187406d9c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f1884001eb0 tx=0x7f1884007040 comp rx=0 tx=0).stop 2024-03-06T07:26:23.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.631+0000 7f1898cc9640 1 -- 172.21.15.1:0/1502802446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f189414a810 msgr2=0x7f1894152ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.631+0000 7f1898cc9640 1 --2- 172.21.15.1:0/1502802446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f189414a810 0x7f1894152ff0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f189408c760 tx=0x7f188c00f580 comp rx=0 tx=0).stop 2024-03-06T07:26:23.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.631+0000 7f1898cc9640 1 -- 172.21.15.1:0/1502802446 shutdown_connections 2024-03-06T07:26:23.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.631+0000 7f1898cc9640 1 --2- 172.21.15.1:0/1502802446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f187406b500 0x7f187406d9c0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.631+0000 7f1898cc9640 1 --2- 172.21.15.1:0/1502802446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f189414a810 0x7f1894152ff0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.631+0000 7f1898cc9640 1 --2- 172.21.15.1:0/1502802446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1894151b40 0x7f189414a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.631+0000 7f1898cc9640 1 --2- 172.21.15.1:0/1502802446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f187c004800 0x7f18941501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.631+0000 7f1898cc9640 1 -- 172.21.15.1:0/1502802446 >> 172.21.15.1:0/1502802446 conn(0x7f1894066ee0 msgr2=0x7f18940c0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.631+0000 7f1898cc9640 1 -- 172.21.15.1:0/1502802446 shutdown_connections 2024-03-06T07:26:23.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.631+0000 7f1898cc9640 1 -- 172.21.15.1:0/1502802446 wait complete. 2024-03-06T07:26:23.644 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:26:23.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.744+0000 7ff573afb640 1 Processor -- start 2024-03-06T07:26:23.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.744+0000 7ff573afb640 1 -- start start 2024-03-06T07:26:23.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.745+0000 7ff573afb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff56c0c6b10 0x7ff56c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.745+0000 7ff573afb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56c0c2fc0 con 0x7ff56c0bf9d0 2024-03-06T07:26:23.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.745+0000 7ff573afb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56c158da0 con 0x7ff56c152650 2024-03-06T07:26:23.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.745+0000 7ff573afb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff56c155800 con 0x7ff56c0c6b10 2024-03-06T07:26:23.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.745+0000 7ff571096640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff56c0c6b10 0x7ff56c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.745+0000 7ff572098640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff56c0bf9d0 0x7ff56c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52748/0 (socket says 172.21.15.1:52748) 2024-03-06T07:26:23.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.745+0000 7ff571096640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff56c0c6b10 0x7ff56c155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44822/0 (socket says 172.21.15.1:44822) 2024-03-06T07:26:23.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.745+0000 7ff572098640 1 -- 172.21.15.1:0/1596114285 learned_addr learned my addr 172.21.15.1:0/1596114285 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:23.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.746+0000 7ff571096640 1 -- 172.21.15.1:0/1596114285 >> v1:172.21.15.98:6789/0 conn(0x7ff56c152650 legacy=0x7ff56c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.746+0000 7ff571096640 1 -- 172.21.15.1:0/1596114285 >> v1:172.21.15.1:6789/0 conn(0x7ff56c0bf9d0 legacy=0x7ff56c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.746+0000 7ff571096640 1 -- 172.21.15.1:0/1596114285 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c0c6690 con 0x7ff56c0c6b10 2024-03-06T07:26:23.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.746+0000 7ff571096640 1 --2- 172.21.15.1:0/1596114285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c0c6b10 0x7ff56c155c90 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7ff55c0097a0 tx=0x7ff55c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a32e760df2157e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.746+0000 7ff570895640 1 -- 172.21.15.1:0/1596114285 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c035020 con 0x7ff56c0c6b10 2024-03-06T07:26:23.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.746+0000 7ff570895640 1 -- 172.21.15.1:0/1596114285 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55c031830 con 0x7ff56c0c6b10 2024-03-06T07:26:23.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.746+0000 7ff570895640 1 -- 172.21.15.1:0/1596114285 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c031c00 con 0x7ff56c0c6b10 2024-03-06T07:26:23.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.747+0000 7ff573afb640 1 -- 172.21.15.1:0/1596114285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c0c6b10 msgr2=0x7ff56c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.747+0000 7ff573afb640 1 --2- 172.21.15.1:0/1596114285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c0c6b10 0x7ff56c155c90 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7ff55c0097a0 tx=0x7ff55c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.747+0000 7ff573afb640 1 -- 172.21.15.1:0/1596114285 shutdown_connections 2024-03-06T07:26:23.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.747+0000 7ff573afb640 1 --2- 172.21.15.1:0/1596114285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c0c6b10 0x7ff56c155c90 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.747+0000 7ff573afb640 1 -- 172.21.15.1:0/1596114285 >> 172.21.15.1:0/1596114285 conn(0x7ff56c066ee0 msgr2=0x7ff56c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.747+0000 7ff573afb640 1 -- 172.21.15.1:0/1596114285 shutdown_connections 2024-03-06T07:26:23.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.747+0000 7ff573afb640 1 -- 172.21.15.1:0/1596114285 wait complete. 2024-03-06T07:26:23.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.748+0000 7ff573afb640 1 Processor -- start 2024-03-06T07:26:23.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.748+0000 7ff573afb640 1 -- start start 2024-03-06T07:26:23.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff573afb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff56c0bf9d0 0x7ff56c14fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff573afb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff56c152650 0x7ff56c157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff573afb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c156830 0x7ff56c14b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff573afb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c1551b0 con 0x7ff56c156830 2024-03-06T07:26:23.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff573afb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c0c6fa0 con 0x7ff56c0bf9d0 2024-03-06T07:26:23.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff573afb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c155490 con 0x7ff56c152650 2024-03-06T07:26:23.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff572098640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c156830 0x7ff56c14b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff572098640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c156830 0x7ff56c14b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44832/0 (socket says 172.21.15.1:44832) 2024-03-06T07:26:23.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff572098640 1 -- 172.21.15.1:0/46099754 learned_addr learned my addr 172.21.15.1:0/46099754 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:23.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff571096640 1 --2- 172.21.15.1:0/46099754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff56c152650 0x7ff56c157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff571897640 1 --2- 172.21.15.1:0/46099754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff56c0bf9d0 0x7ff56c14fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff572098640 1 -- 172.21.15.1:0/46099754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff56c152650 msgr2=0x7ff56c157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff572098640 1 --2- 172.21.15.1:0/46099754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff56c152650 0x7ff56c157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff572098640 1 -- 172.21.15.1:0/46099754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff56c0bf9d0 msgr2=0x7ff56c14fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff572098640 1 --2- 172.21.15.1:0/46099754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff56c0bf9d0 0x7ff56c14fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff572098640 1 -- 172.21.15.1:0/46099754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c158610 con 0x7ff56c156830 2024-03-06T07:26:23.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff571897640 1 --2- 172.21.15.1:0/46099754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff56c0bf9d0 0x7ff56c14fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:23.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff571096640 1 --2- 172.21.15.1:0/46099754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff56c152650 0x7ff56c157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:23.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.749+0000 7ff572098640 1 --2- 172.21.15.1:0/46099754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c156830 0x7ff56c14b2c0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7ff56400c370 tx=0x7ff56400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.750+0000 7ff5627fc640 1 -- 172.21.15.1:0/46099754 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff564017070 con 0x7ff56c156830 2024-03-06T07:26:23.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.750+0000 7ff5627fc640 1 -- 172.21.15.1:0/46099754 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff564005ce0 con 0x7ff56c156830 2024-03-06T07:26:23.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.750+0000 7ff573afb640 1 -- 172.21.15.1:0/46099754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56c1eb9c0 con 0x7ff56c156830 2024-03-06T07:26:23.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.750+0000 7ff5627fc640 1 -- 172.21.15.1:0/46099754 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5640072d0 con 0x7ff56c156830 2024-03-06T07:26:23.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.750+0000 7ff573afb640 1 -- 172.21.15.1:0/46099754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56c1ebc60 con 0x7ff56c156830 2024-03-06T07:26:23.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.751+0000 7ff573afb640 1 -- 172.21.15.1:0/46099754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff56c067fd0 con 0x7ff56c156830 2024-03-06T07:26:23.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.751+0000 7ff5627fc640 1 -- 172.21.15.1:0/46099754 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff56401a070 con 0x7ff56c156830 2024-03-06T07:26:23.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.752+0000 7ff5627fc640 1 --2- 172.21.15.1:0/46099754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff55006b4d0 0x7ff55006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.752+0000 7ff5627fc640 1 -- 172.21.15.1:0/46099754 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff564089530 con 0x7ff56c156830 2024-03-06T07:26:23.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.752+0000 7ff571897640 1 --2- 172.21.15.1:0/46099754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff55006b4d0 0x7ff55006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.753+0000 7ff571897640 1 --2- 172.21.15.1:0/46099754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff55006b4d0 0x7ff55006d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7ff568001e00 tx=0x7ff568009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.756+0000 7ff5627fc640 1 -- 172.21.15.1:0/46099754 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff56c158610 con 0x7ff56c156830 2024-03-06T07:26:23.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.971+0000 7ff573afb640 1 -- 172.21.15.1:0/46099754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff56c0b06f0 con 0x7ff56c156830 2024-03-06T07:26:23.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.971+0000 7ff5627fc640 1 -- 172.21.15.1:0/46099754 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v149) v1 ==== 84+0+209 (secure 0 0 0) 0x7ff564057c60 con 0x7ff56c156830 2024-03-06T07:26:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:23.974 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-06T07:26:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-06T07:26:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-06T07:26:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 23, 2024-03-06T07:26:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": "cephfs_metadata" 2024-03-06T07:26:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:26:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:26:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 24, 2024-03-06T07:26:23.975 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": "cephfs_data" 2024-03-06T07:26:23.975 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:26:23.975 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-06T07:26:23.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.973+0000 7ff573afb640 1 -- 172.21.15.1:0/46099754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff55006b4d0 msgr2=0x7ff55006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.973+0000 7ff573afb640 1 --2- 172.21.15.1:0/46099754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff55006b4d0 0x7ff55006d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7ff568001e00 tx=0x7ff568009040 comp rx=0 tx=0).stop 2024-03-06T07:26:23.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.973+0000 7ff573afb640 1 -- 172.21.15.1:0/46099754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c156830 msgr2=0x7ff56c14b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.973+0000 7ff573afb640 1 --2- 172.21.15.1:0/46099754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c156830 0x7ff56c14b2c0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7ff56400c370 tx=0x7ff56400c830 comp rx=0 tx=0).stop 2024-03-06T07:26:23.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.973+0000 7ff573afb640 1 -- 172.21.15.1:0/46099754 shutdown_connections 2024-03-06T07:26:23.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.973+0000 7ff573afb640 1 --2- 172.21.15.1:0/46099754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff55006b4d0 0x7ff55006d990 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.973+0000 7ff573afb640 1 --2- 172.21.15.1:0/46099754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c156830 0x7ff56c14b2c0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.973+0000 7ff573afb640 1 --2- 172.21.15.1:0/46099754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff56c152650 0x7ff56c157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.973+0000 7ff573afb640 1 --2- 172.21.15.1:0/46099754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff56c0bf9d0 0x7ff56c14fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.973+0000 7ff573afb640 1 -- 172.21.15.1:0/46099754 >> 172.21.15.1:0/46099754 conn(0x7ff56c066ee0 msgr2=0x7ff56c0672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.973+0000 7ff573afb640 1 -- 172.21.15.1:0/46099754 shutdown_connections 2024-03-06T07:26:23.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:23.973+0000 7ff573afb640 1 -- 172.21.15.1:0/46099754 wait complete. 2024-03-06T07:26:23.986 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs2' 2024-03-06T07:26:23.986 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_metadata --pg_num_min 64 2024-03-06T07:26:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.090+0000 7f2fd594b640 1 Processor -- start 2024-03-06T07:26:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.090+0000 7f2fd594b640 1 -- start start 2024-03-06T07:26:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.090+0000 7f2fd594b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd0152780 0x7f2fd00c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.090+0000 7f2fd594b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2fd0155e50 0x7f2fd00cab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.090+0000 7f2fd594b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd00c69b0 con 0x7f2fd00c1010 2024-03-06T07:26:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.090+0000 7f2fd594b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd00c6c90 con 0x7f2fd0155e50 2024-03-06T07:26:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.090+0000 7f2fd594b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd0155370 con 0x7f2fd0152780 2024-03-06T07:26:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.090+0000 7f2fcf7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2fd0155e50 0x7f2fd00cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.090+0000 7f2fcf7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2fd0155e50 0x7f2fd00cab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44836/0 (socket says 172.21.15.1:44836) 2024-03-06T07:26:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.090+0000 7f2fcf7fe640 1 -- 172.21.15.1:0/2910653013 learned_addr learned my addr 172.21.15.1:0/2910653013 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.091+0000 7f2fceffd640 1 --2- 172.21.15.1:0/2910653013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd0152780 0x7f2fd00c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.091+0000 7f2fcf7fe640 1 -- 172.21.15.1:0/2910653013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd0152780 msgr2=0x7f2fd00c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.091+0000 7f2fcf7fe640 1 --2- 172.21.15.1:0/2910653013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd0152780 0x7f2fd00c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.091+0000 7f2fcf7fe640 1 -- 172.21.15.1:0/2910653013 >> v1:172.21.15.98:6789/0 conn(0x7f2fd00c1010 legacy=0x7f2fd00c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.091+0000 7f2fcf7fe640 1 -- 172.21.15.1:0/2910653013 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd0155650 con 0x7f2fd0155e50 2024-03-06T07:26:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.091+0000 7f2fceffd640 1 --2- 172.21.15.1:0/2910653013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd0152780 0x7f2fd00c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.091+0000 7f2fcf7fe640 1 --2- 172.21.15.1:0/2910653013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd0155e50 0x7f2fd00cab00 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8009e50 tx=0x7f2fc802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c97cdc0322b08108 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.092+0000 7f2fce7fc640 1 -- 172.21.15.1:0/2910653013 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc802ca70 con 0x7f2fd0155e50 2024-03-06T07:26:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.092+0000 7f2fce7fc640 1 -- 172.21.15.1:0/2910653013 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc802cc10 con 0x7f2fd0155e50 2024-03-06T07:26:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.092+0000 7f2fce7fc640 1 -- 172.21.15.1:0/2910653013 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc80343f0 con 0x7f2fd0155e50 2024-03-06T07:26:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.092+0000 7f2fd594b640 1 -- 172.21.15.1:0/2910653013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd0155e50 msgr2=0x7f2fd00cab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.092+0000 7f2fd594b640 1 --2- 172.21.15.1:0/2910653013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd0155e50 0x7f2fd00cab00 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8009e50 tx=0x7f2fc802bed0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.092+0000 7f2fd594b640 1 -- 172.21.15.1:0/2910653013 shutdown_connections 2024-03-06T07:26:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.092+0000 7f2fd594b640 1 --2- 172.21.15.1:0/2910653013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd0155e50 0x7f2fd00cab00 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.092+0000 7f2fd594b640 1 --2- 172.21.15.1:0/2910653013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd0152780 0x7f2fd00c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.092+0000 7f2fd594b640 1 -- 172.21.15.1:0/2910653013 >> 172.21.15.1:0/2910653013 conn(0x7f2fd008b720 msgr2=0x7f2fd006d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.092+0000 7f2fd594b640 1 -- 172.21.15.1:0/2910653013 shutdown_connections 2024-03-06T07:26:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.092+0000 7f2fd594b640 1 -- 172.21.15.1:0/2910653013 wait complete. 2024-03-06T07:26:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.093+0000 7f2fd594b640 1 Processor -- start 2024-03-06T07:26:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.093+0000 7f2fd594b640 1 -- start start 2024-03-06T07:26:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.094+0000 7f2fd594b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd00c1010 0x7f2fd0150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.094+0000 7f2fd594b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fd0152780 0x7f2fd01e7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.094+0000 7f2fd594b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd0155e50 0x7f2fd01eddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.094+0000 7f2fd594b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd0152b60 con 0x7f2fd0155e50 2024-03-06T07:26:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.094+0000 7f2fd594b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd00c76d0 con 0x7f2fd0152780 2024-03-06T07:26:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.094+0000 7f2fd594b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd00c6540 con 0x7f2fd00c1010 2024-03-06T07:26:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.094+0000 7f2fcf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd0155e50 0x7f2fd01eddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.094+0000 7f2fcf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd0155e50 0x7f2fd01eddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44842/0 (socket says 172.21.15.1:44842) 2024-03-06T07:26:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.094+0000 7f2fcf7fe640 1 -- 172.21.15.1:0/1189202935 learned_addr learned my addr 172.21.15.1:0/1189202935 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.094+0000 7f2fc7fff640 1 --2- 172.21.15.1:0/1189202935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fd0152780 0x7f2fd01e7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.094+0000 7f2fceffd640 1 --2- 172.21.15.1:0/1189202935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd00c1010 0x7f2fd0150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.094+0000 7f2fcf7fe640 1 -- 172.21.15.1:0/1189202935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd00c1010 msgr2=0x7f2fd0150660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.094+0000 7f2fcf7fe640 1 --2- 172.21.15.1:0/1189202935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd00c1010 0x7f2fd0150660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.095+0000 7f2fcf7fe640 1 -- 172.21.15.1:0/1189202935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fd0152780 msgr2=0x7f2fd01e7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.095+0000 7f2fcf7fe640 1 --2- 172.21.15.1:0/1189202935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fd0152780 0x7f2fd01e7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.095+0000 7f2fcf7fe640 1 -- 172.21.15.1:0/1189202935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd00c9c60 con 0x7f2fd0155e50 2024-03-06T07:26:24.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.095+0000 7f2fcf7fe640 1 --2- 172.21.15.1:0/1189202935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd0155e50 0x7f2fd01eddc0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8009e50 tx=0x7f2fc8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.095+0000 7f2fccff9640 1 -- 172.21.15.1:0/1189202935 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8044020 con 0x7f2fd0155e50 2024-03-06T07:26:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.095+0000 7f2fccff9640 1 -- 172.21.15.1:0/1189202935 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc8040670 con 0x7f2fd0155e50 2024-03-06T07:26:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.095+0000 7f2fd594b640 1 -- 172.21.15.1:0/1189202935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd01563e0 con 0x7f2fd0155e50 2024-03-06T07:26:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.095+0000 7f2fccff9640 1 -- 172.21.15.1:0/1189202935 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8009510 con 0x7f2fd0155e50 2024-03-06T07:26:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.095+0000 7f2fd594b640 1 -- 172.21.15.1:0/1189202935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd014bd10 con 0x7f2fd0155e50 2024-03-06T07:26:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.096+0000 7f2fd594b640 1 -- 172.21.15.1:0/1189202935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd00680b0 con 0x7f2fd0155e50 2024-03-06T07:26:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.096+0000 7f2fccff9640 1 -- 172.21.15.1:0/1189202935 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fc8047020 con 0x7f2fd0155e50 2024-03-06T07:26:24.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.097+0000 7f2fccff9640 1 --2- 172.21.15.1:0/1189202935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fb006b4d0 0x7f2fb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.097+0000 7f2fccff9640 1 -- 172.21.15.1:0/1189202935 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2fc80acc50 con 0x7f2fd0155e50 2024-03-06T07:26:24.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.097+0000 7f2fceffd640 1 --2- 172.21.15.1:0/1189202935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fb006b4d0 0x7f2fb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.097+0000 7f2fceffd640 1 --2- 172.21.15.1:0/1189202935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fb006b4d0 0x7f2fb006d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0006fd0 tx=0x7f2fc0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.101+0000 7f2fccff9640 1 -- 172.21.15.1:0/1189202935 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fc80b2020 con 0x7f2fd0155e50 2024-03-06T07:26:24.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.284+0000 7f2fd594b640 1 -- 172.21.15.1:0/1189202935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2fd00c5de0 con 0x7f2fd0155e50 2024-03-06T07:26:24.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.655+0000 7f2fccff9640 1 -- 172.21.15.1:0/1189202935 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64}]=0 pool 'cephfs2_metadata' created v150) v1 ==== 140+0+0 (secure 0 0 0) 0x7f2fc804e030 con 0x7f2fd0155e50 2024-03-06T07:26:24.657 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs2_metadata' created 2024-03-06T07:26:24.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.658+0000 7f2fd594b640 1 -- 172.21.15.1:0/1189202935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fb006b4d0 msgr2=0x7f2fb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.658+0000 7f2fd594b640 1 --2- 172.21.15.1:0/1189202935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fb006b4d0 0x7f2fb006d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0006fd0 tx=0x7f2fc0008040 comp rx=0 tx=0).stop 2024-03-06T07:26:24.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.658+0000 7f2fd594b640 1 -- 172.21.15.1:0/1189202935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd0155e50 msgr2=0x7f2fd01eddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.658+0000 7f2fd594b640 1 --2- 172.21.15.1:0/1189202935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd0155e50 0x7f2fd01eddc0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8009e50 tx=0x7f2fc8004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:24.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.658+0000 7f2fd594b640 1 -- 172.21.15.1:0/1189202935 shutdown_connections 2024-03-06T07:26:24.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.658+0000 7f2fd594b640 1 --2- 172.21.15.1:0/1189202935 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fb006b4d0 0x7f2fb006d990 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.658+0000 7f2fd594b640 1 --2- 172.21.15.1:0/1189202935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd0155e50 0x7f2fd01eddc0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.658+0000 7f2fd594b640 1 --2- 172.21.15.1:0/1189202935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fd0152780 0x7f2fd01e7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.658+0000 7f2fd594b640 1 --2- 172.21.15.1:0/1189202935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fd00c1010 0x7f2fd0150660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.658+0000 7f2fd594b640 1 -- 172.21.15.1:0/1189202935 >> 172.21.15.1:0/1189202935 conn(0x7f2fd008b720 msgr2=0x7f2fd00c0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.659+0000 7f2fd594b640 1 -- 172.21.15.1:0/1189202935 shutdown_connections 2024-03-06T07:26:24.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.659+0000 7f2fd594b640 1 -- 172.21.15.1:0/1189202935 wait complete. 2024-03-06T07:26:24.676 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:26:24.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.783+0000 7f0ec435e640 1 Processor -- start 2024-03-06T07:26:24.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.783+0000 7f0ec435e640 1 -- start start 2024-03-06T07:26:24.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.784+0000 7f0ec435e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ebc0bf9d0 0x7f0ebc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.784+0000 7f0ec435e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc0c2fc0 con 0x7f0ebc152650 2024-03-06T07:26:24.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.784+0000 7f0ec435e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc158da0 con 0x7f0ebc0c6b10 2024-03-06T07:26:24.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.784+0000 7f0ec435e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ebc155800 con 0x7f0ebc0bf9d0 2024-03-06T07:26:24.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.784+0000 7f0ec28fb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ebc0bf9d0 0x7f0ebc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.784+0000 7f0ec28fb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ebc0bf9d0 0x7f0ebc155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58476/0 (socket says 172.21.15.1:58476) 2024-03-06T07:26:24.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.784+0000 7f0ec28fb640 1 -- 172.21.15.1:0/50021108 learned_addr learned my addr 172.21.15.1:0/50021108 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:24.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.785+0000 7f0ec28fb640 1 -- 172.21.15.1:0/50021108 >> v1:172.21.15.98:6789/0 conn(0x7f0ebc0c6b10 legacy=0x7f0ebc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.785+0000 7f0ec28fb640 1 -- 172.21.15.1:0/50021108 >> v1:172.21.15.1:6789/0 conn(0x7f0ebc152650 legacy=0x7f0ebc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.785+0000 7f0ec10f8640 1 -- 172.21.15.1:0/50021108 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157185344 0 0) 0x7f0ebc0c2fc0 con 0x7f0ebc152650 2024-03-06T07:26:24.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.785+0000 7f0ec28fb640 1 -- 172.21.15.1:0/50021108 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ebc0c6690 con 0x7f0ebc0bf9d0 2024-03-06T07:26:24.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.786+0000 7f0ec28fb640 1 --2- 172.21.15.1:0/50021108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ebc0bf9d0 0x7f0ebc155c90 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4009e50 tx=0x7f0eb402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4f53a3fabcd2cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.786+0000 7f0ec10f8640 1 -- 172.21.15.1:0/50021108 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb402cb20 con 0x7f0ebc0bf9d0 2024-03-06T07:26:24.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.786+0000 7f0ec10f8640 1 -- 172.21.15.1:0/50021108 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb402ccc0 con 0x7f0ebc0bf9d0 2024-03-06T07:26:24.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.786+0000 7f0ec10f8640 1 -- 172.21.15.1:0/50021108 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4034610 con 0x7f0ebc0bf9d0 2024-03-06T07:26:24.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.786+0000 7f0ec435e640 1 -- 172.21.15.1:0/50021108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ebc0bf9d0 msgr2=0x7f0ebc155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.786+0000 7f0ec435e640 1 --2- 172.21.15.1:0/50021108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ebc0bf9d0 0x7f0ebc155c90 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4009e50 tx=0x7f0eb402bf80 comp rx=0 tx=0).stop 2024-03-06T07:26:24.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.786+0000 7f0ec435e640 1 -- 172.21.15.1:0/50021108 shutdown_connections 2024-03-06T07:26:24.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.786+0000 7f0ec435e640 1 --2- 172.21.15.1:0/50021108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ebc0bf9d0 0x7f0ebc155c90 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.786+0000 7f0ec435e640 1 -- 172.21.15.1:0/50021108 >> 172.21.15.1:0/50021108 conn(0x7f0ebc066ee0 msgr2=0x7f0ebc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.786+0000 7f0ec435e640 1 -- 172.21.15.1:0/50021108 shutdown_connections 2024-03-06T07:26:24.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.786+0000 7f0ec435e640 1 -- 172.21.15.1:0/50021108 wait complete. 2024-03-06T07:26:24.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.787+0000 7f0ec435e640 1 Processor -- start 2024-03-06T07:26:24.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.787+0000 7f0ec435e640 1 -- start start 2024-03-06T07:26:24.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec435e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ebc0bf9d0 0x7f0ebc154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec435e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ebc152650 0x7f0ebc153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec435e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ebc0c87d0 0x7f0ebc1505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec435e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc1551b0 con 0x7f0ebc152650 2024-03-06T07:26:24.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec435e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc0c6fa0 con 0x7f0ebc0c87d0 2024-03-06T07:26:24.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec435e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc155490 con 0x7f0ebc0bf9d0 2024-03-06T07:26:24.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec18f9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ebc152650 0x7f0ebc153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec18f9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ebc152650 0x7f0ebc153160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58482/0 (socket says 172.21.15.1:58482) 2024-03-06T07:26:24.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec18f9640 1 -- 172.21.15.1:0/1821963876 learned_addr learned my addr 172.21.15.1:0/1821963876 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:24.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec28fb640 1 --2- 172.21.15.1:0/1821963876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ebc0c87d0 0x7f0ebc1505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec20fa640 1 --2- 172.21.15.1:0/1821963876 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ebc0bf9d0 0x7f0ebc154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec18f9640 1 -- 172.21.15.1:0/1821963876 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ebc0bf9d0 msgr2=0x7f0ebc154c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec18f9640 1 --2- 172.21.15.1:0/1821963876 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ebc0bf9d0 0x7f0ebc154c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec18f9640 1 -- 172.21.15.1:0/1821963876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ebc0c87d0 msgr2=0x7f0ebc1505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec18f9640 1 --2- 172.21.15.1:0/1821963876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ebc0c87d0 0x7f0ebc1505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec18f9640 1 -- 172.21.15.1:0/1821963876 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ebc14ddb0 con 0x7f0ebc152650 2024-03-06T07:26:24.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec28fb640 1 --2- 172.21.15.1:0/1821963876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ebc0c87d0 0x7f0ebc1505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:24.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.788+0000 7f0ec20fa640 1 --2- 172.21.15.1:0/1821963876 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ebc0bf9d0 0x7f0ebc154c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:24.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.789+0000 7f0ec18f9640 1 --2- 172.21.15.1:0/1821963876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ebc152650 0x7f0ebc153160 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00c450 tx=0x7f0eac00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.789+0000 7f0eb2ffd640 1 -- 172.21.15.1:0/1821963876 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac017070 con 0x7f0ebc152650 2024-03-06T07:26:24.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.789+0000 7f0eb2ffd640 1 -- 172.21.15.1:0/1821963876 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eac0052d0 con 0x7f0ebc152650 2024-03-06T07:26:24.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.789+0000 7f0ec435e640 1 -- 172.21.15.1:0/1821963876 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ebc1ef180 con 0x7f0ebc152650 2024-03-06T07:26:24.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.789+0000 7f0eb2ffd640 1 -- 172.21.15.1:0/1821963876 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac003830 con 0x7f0ebc152650 2024-03-06T07:26:24.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.789+0000 7f0ec435e640 1 -- 172.21.15.1:0/1821963876 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ebc14da40 con 0x7f0ebc152650 2024-03-06T07:26:24.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.790+0000 7f0ec435e640 1 -- 172.21.15.1:0/1821963876 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ebc0c4fe0 con 0x7f0ebc152650 2024-03-06T07:26:24.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.791+0000 7f0eb2ffd640 1 -- 172.21.15.1:0/1821963876 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0eac01a070 con 0x7f0ebc152650 2024-03-06T07:26:24.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.791+0000 7f0eb2ffd640 1 --2- 172.21.15.1:0/1821963876 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ea006b500 0x7f0ea006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.791+0000 7f0eb2ffd640 1 -- 172.21.15.1:0/1821963876 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6472+0+0 (secure 0 0 0) 0x7f0eac089800 con 0x7f0ebc152650 2024-03-06T07:26:24.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.792+0000 7f0ec20fa640 1 --2- 172.21.15.1:0/1821963876 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ea006b500 0x7f0ea006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.792+0000 7f0ec20fa640 1 --2- 172.21.15.1:0/1821963876 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ea006b500 0x7f0ea006d9c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8000f60 tx=0x7f0eb8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:24.796+0000 7f0eb2ffd640 1 -- 172.21.15.1:0/1821963876 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ebc0c4fe0 con 0x7f0ebc152650 2024-03-06T07:26:25.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.066+0000 7f0ec435e640 1 -- 172.21.15.1:0/1821963876 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0ebc067fd0 con 0x7f0ebc152650 2024-03-06T07:26:25.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.657+0000 7f0eb2ffd640 1 -- 172.21.15.1:0/1821963876 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs2_data' created v151) v1 ==== 172+0+0 (secure 0 0 0) 0x7f0eac058410 con 0x7f0ebc152650 2024-03-06T07:26:25.659 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs2_data' created 2024-03-06T07:26:25.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.660+0000 7f0ec435e640 1 -- 172.21.15.1:0/1821963876 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ea006b500 msgr2=0x7f0ea006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.660+0000 7f0ec435e640 1 --2- 172.21.15.1:0/1821963876 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ea006b500 0x7f0ea006d9c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8000f60 tx=0x7f0eb8009040 comp rx=0 tx=0).stop 2024-03-06T07:26:25.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.660+0000 7f0ec435e640 1 -- 172.21.15.1:0/1821963876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ebc152650 msgr2=0x7f0ebc153160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.660+0000 7f0ec435e640 1 --2- 172.21.15.1:0/1821963876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ebc152650 0x7f0ebc153160 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00c450 tx=0x7f0eac00c910 comp rx=0 tx=0).stop 2024-03-06T07:26:25.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.660+0000 7f0ec435e640 1 -- 172.21.15.1:0/1821963876 shutdown_connections 2024-03-06T07:26:25.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.660+0000 7f0ec435e640 1 --2- 172.21.15.1:0/1821963876 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0ea006b500 0x7f0ea006d9c0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.660+0000 7f0ec435e640 1 --2- 172.21.15.1:0/1821963876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ebc0c87d0 0x7f0ebc1505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.660+0000 7f0ec435e640 1 --2- 172.21.15.1:0/1821963876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ebc152650 0x7f0ebc153160 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.660+0000 7f0ec435e640 1 --2- 172.21.15.1:0/1821963876 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0ebc0bf9d0 0x7f0ebc154c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.660+0000 7f0ec435e640 1 -- 172.21.15.1:0/1821963876 >> 172.21.15.1:0/1821963876 conn(0x7f0ebc066ee0 msgr2=0x7f0ebc0c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.662+0000 7f0ec435e640 1 -- 172.21.15.1:0/1821963876 shutdown_connections 2024-03-06T07:26:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.662+0000 7f0ec435e640 1 -- 172.21.15.1:0/1821963876 wait complete. 2024-03-06T07:26:25.688 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs2 cephfs2_metadata cephfs2_data 2024-03-06T07:26:25.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.790+0000 7f6269a2a640 1 Processor -- start 2024-03-06T07:26:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.791+0000 7f6269a2a640 1 -- start start 2024-03-06T07:26:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.791+0000 7f6269a2a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f62640c7560 0x7f6264155860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.791+0000 7f6269a2a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6264155080 con 0x7f6264152890 2024-03-06T07:26:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.791+0000 7f6269a2a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62640c35b0 con 0x7f6264155da0 2024-03-06T07:26:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.791+0000 7f6269a2a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62640c3890 con 0x7f62640c7560 2024-03-06T07:26:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.791+0000 7f625a7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f62640c7560 0x7f6264155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.791+0000 7f625a7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f62640c7560 0x7f6264155860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58484/0 (socket says 172.21.15.1:58484) 2024-03-06T07:26:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.791+0000 7f625a7fc640 1 -- 172.21.15.1:0/2624199049 learned_addr learned my addr 172.21.15.1:0/2624199049 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.791+0000 7f6262ffd640 1 --1- 172.21.15.1:0/2624199049 >> v1:172.21.15.1:6789/0 conn(0x7f6264152890 0x7f6264159030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60304/0 (socket says 172.21.15.1:60304) 2024-03-06T07:26:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.792+0000 7f625a7fc640 1 -- 172.21.15.1:0/2624199049 >> v1:172.21.15.98:6789/0 conn(0x7f6264155da0 legacy=0x7f62640c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.792+0000 7f625a7fc640 1 -- 172.21.15.1:0/2624199049 >> v1:172.21.15.1:6789/0 conn(0x7f6264152890 legacy=0x7f6264159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.792+0000 7f625a7fc640 1 -- 172.21.15.1:0/2624199049 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62640c3b70 con 0x7f62640c7560 2024-03-06T07:26:25.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.792+0000 7f625a7fc640 1 --2- 172.21.15.1:0/2624199049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62640c7560 0x7f6264155860 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f624c009870 tx=0x7f624c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0ab583bffd1eba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.793+0000 7f62627fc640 1 -- 172.21.15.1:0/2624199049 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c035020 con 0x7f62640c7560 2024-03-06T07:26:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.793+0000 7f62627fc640 1 -- 172.21.15.1:0/2624199049 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624c031870 con 0x7f62640c7560 2024-03-06T07:26:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.793+0000 7f62627fc640 1 -- 172.21.15.1:0/2624199049 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c031c40 con 0x7f62640c7560 2024-03-06T07:26:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.793+0000 7f6269a2a640 1 -- 172.21.15.1:0/2624199049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62640c7560 msgr2=0x7f6264155860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.793+0000 7f6269a2a640 1 --2- 172.21.15.1:0/2624199049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62640c7560 0x7f6264155860 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f624c009870 tx=0x7f624c02b910 comp rx=0 tx=0).stop 2024-03-06T07:26:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.793+0000 7f6269a2a640 1 -- 172.21.15.1:0/2624199049 shutdown_connections 2024-03-06T07:26:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.793+0000 7f6269a2a640 1 --2- 172.21.15.1:0/2624199049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62640c7560 0x7f6264155860 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.793+0000 7f6269a2a640 1 -- 172.21.15.1:0/2624199049 >> 172.21.15.1:0/2624199049 conn(0x7f626406b3e0 msgr2=0x7f626406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.793+0000 7f6269a2a640 1 -- 172.21.15.1:0/2624199049 shutdown_connections 2024-03-06T07:26:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.793+0000 7f6269a2a640 1 -- 172.21.15.1:0/2624199049 wait complete. 2024-03-06T07:26:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.794+0000 7f6269a2a640 1 Processor -- start 2024-03-06T07:26:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.794+0000 7f6269a2a640 1 -- start start 2024-03-06T07:26:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.795+0000 7f6269a2a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62640c7560 0x7f62641eb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.795+0000 7f6269a2a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6264152890 0x7f62640a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.795+0000 7f6269a2a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6264099d40 0x7f626406c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.795+0000 7f6269a2a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62640c9400 con 0x7f62640c7560 2024-03-06T07:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.795+0000 7f6269a2a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62640c3260 con 0x7f6264099d40 2024-03-06T07:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.795+0000 7f6269a2a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62640c9580 con 0x7f6264152890 2024-03-06T07:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.795+0000 7f6262ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62640c7560 0x7f62641eb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.795+0000 7f6262ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62640c7560 0x7f62641eb320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58490/0 (socket says 172.21.15.1:58490) 2024-03-06T07:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.795+0000 7f6262ffd640 1 -- 172.21.15.1:0/741032789 learned_addr learned my addr 172.21.15.1:0/741032789 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.795+0000 7f62637fe640 1 --2- 172.21.15.1:0/741032789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6264099d40 0x7f626406c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.795+0000 7f625a7fc640 1 --2- 172.21.15.1:0/741032789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6264152890 0x7f62640a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.796+0000 7f6262ffd640 1 -- 172.21.15.1:0/741032789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6264152890 msgr2=0x7f62640a8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.796+0000 7f6262ffd640 1 --2- 172.21.15.1:0/741032789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6264152890 0x7f62640a8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.796+0000 7f6262ffd640 1 -- 172.21.15.1:0/741032789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6264099d40 msgr2=0x7f626406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.796+0000 7f6262ffd640 1 --2- 172.21.15.1:0/741032789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6264099d40 0x7f626406c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.796+0000 7f6262ffd640 1 -- 172.21.15.1:0/741032789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62641eb860 con 0x7f62640c7560 2024-03-06T07:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.796+0000 7f625a7fc640 1 --2- 172.21.15.1:0/741032789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6264152890 0x7f62640a8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.796+0000 7f6262ffd640 1 --2- 172.21.15.1:0/741032789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62640c7560 0x7f62641eb320 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f625400c800 tx=0x7f625400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.796+0000 7f6260ff9640 1 -- 172.21.15.1:0/741032789 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6254017070 con 0x7f62640c7560 2024-03-06T07:26:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.796+0000 7f6260ff9640 1 -- 172.21.15.1:0/741032789 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625400a640 con 0x7f62640c7560 2024-03-06T07:26:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.796+0000 7f6269a2a640 1 -- 172.21.15.1:0/741032789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62640a9b70 con 0x7f62640c7560 2024-03-06T07:26:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.796+0000 7f6260ff9640 1 -- 172.21.15.1:0/741032789 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625400aa30 con 0x7f62640c7560 2024-03-06T07:26:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.796+0000 7f6269a2a640 1 -- 172.21.15.1:0/741032789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62640aa130 con 0x7f62640c7560 2024-03-06T07:26:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.797+0000 7f6269a2a640 1 -- 172.21.15.1:0/741032789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62640aa370 con 0x7f62640c7560 2024-03-06T07:26:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.797+0000 7f6260ff9640 1 -- 172.21.15.1:0/741032789 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f625401a070 con 0x7f62640c7560 2024-03-06T07:26:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.798+0000 7f6260ff9640 1 --2- 172.21.15.1:0/741032789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f624406b4d0 0x7f624406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.798+0000 7f6260ff9640 1 -- 172.21.15.1:0/741032789 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6855+0+0 (secure 0 0 0) 0x7f6254089bf0 con 0x7f62640c7560 2024-03-06T07:26:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.798+0000 7f625a7fc640 1 --2- 172.21.15.1:0/741032789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f624406b4d0 0x7f624406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.799+0000 7f625a7fc640 1 --2- 172.21.15.1:0/741032789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f624406b4d0 0x7f624406d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f624c036000 tx=0x7f624c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:25.803+0000 7f6260ff9640 1 -- 172.21.15.1:0/741032789 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f62640aa370 con 0x7f62640c7560 2024-03-06T07:26:26.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.022+0000 7f6269a2a640 1 -- 172.21.15.1:0/741032789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"} v 0) v1 -- 0x7f62640c2d80 con 0x7f62640c7560 2024-03-06T07:26:26.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.690+0000 7f6260ff9640 1 -- 172.21.15.1:0/741032789 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"}]=0 new fs with metadata pool 25 and data pool 26 v138) v1 ==== 177+0+0 (secure 0 0 0) 0x7f6254058030 con 0x7f62640c7560 2024-03-06T07:26:26.692 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 25 and data pool 26 2024-03-06T07:26:26.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.698+0000 7f6269a2a640 1 -- 172.21.15.1:0/741032789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f624406b4d0 msgr2=0x7f624406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.698+0000 7f6269a2a640 1 --2- 172.21.15.1:0/741032789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f624406b4d0 0x7f624406d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f624c036000 tx=0x7f624c03c040 comp rx=0 tx=0).stop 2024-03-06T07:26:26.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.698+0000 7f6269a2a640 1 -- 172.21.15.1:0/741032789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62640c7560 msgr2=0x7f62641eb320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.698+0000 7f6269a2a640 1 --2- 172.21.15.1:0/741032789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62640c7560 0x7f62641eb320 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f625400c800 tx=0x7f625400ccc0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.698+0000 7f6269a2a640 1 -- 172.21.15.1:0/741032789 shutdown_connections 2024-03-06T07:26:26.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.698+0000 7f6269a2a640 1 --2- 172.21.15.1:0/741032789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f624406b4d0 0x7f624406d990 secure :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f624c036000 tx=0x7f624c03c040 comp rx=0 tx=0).stop 2024-03-06T07:26:26.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.698+0000 7f6269a2a640 1 --2- 172.21.15.1:0/741032789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6264099d40 0x7f626406c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.698+0000 7f6269a2a640 1 --2- 172.21.15.1:0/741032789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6264152890 0x7f62640a8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.698+0000 7f6269a2a640 1 --2- 172.21.15.1:0/741032789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62640c7560 0x7f62641eb320 secure :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f625400c800 tx=0x7f625400ccc0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.698+0000 7f6269a2a640 1 -- 172.21.15.1:0/741032789 >> 172.21.15.1:0/741032789 conn(0x7f626406b3e0 msgr2=0x7f62641577f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.699+0000 7f6269a2a640 1 -- 172.21.15.1:0/741032789 shutdown_connections 2024-03-06T07:26:26.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.699+0000 7f6269a2a640 1 -- 172.21.15.1:0/741032789 wait complete. 2024-03-06T07:26:26.721 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:26.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.820+0000 7fa0d7dc7640 1 Processor -- start 2024-03-06T07:26:26.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.820+0000 7fa0d7dc7640 1 -- start start 2024-03-06T07:26:26.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.821+0000 7fa0d7dc7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa0d01543a0 0x7fa0d015e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.821+0000 7fa0d7dc7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d0150900 con 0x7fa0d0150520 2024-03-06T07:26:26.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.821+0000 7fa0d7dc7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d0071790 con 0x7fa0d00c32f0 2024-03-06T07:26:26.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.821+0000 7fa0d7dc7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0d0153ed0 con 0x7fa0d01543a0 2024-03-06T07:26:26.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.821+0000 7fa0d6364640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa0d01543a0 0x7fa0d015e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.821+0000 7fa0d5b63640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa0d0150520 0x7fa0d01537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60306/0 (socket says 172.21.15.1:60306) 2024-03-06T07:26:26.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.821+0000 7fa0d5b63640 1 -- 172.21.15.1:0/1387493792 learned_addr learned my addr 172.21.15.1:0/1387493792 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:26.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.821+0000 7fa0d6364640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa0d01543a0 0x7fa0d015e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58498/0 (socket says 172.21.15.1:58498) 2024-03-06T07:26:26.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.821+0000 7fa0d6364640 1 -- 172.21.15.1:0/1387493792 >> v1:172.21.15.98:6789/0 conn(0x7fa0d00c32f0 legacy=0x7fa0d0070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.821+0000 7fa0d6364640 1 -- 172.21.15.1:0/1387493792 >> v1:172.21.15.1:6789/0 conn(0x7fa0d0150520 legacy=0x7fa0d01537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.821+0000 7fa0d6364640 1 -- 172.21.15.1:0/1387493792 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d0157050 con 0x7fa0d01543a0 2024-03-06T07:26:26.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.821+0000 7fa0d4b61640 1 -- 172.21.15.1:0/1387493792 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055519687 0 0) 0x7fa0d0150900 con 0x7fa0d0150520 2024-03-06T07:26:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.822+0000 7fa0d6364640 1 --2- 172.21.15.1:0/1387493792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d01543a0 0x7fa0d015e840 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8009d90 tx=0x7fa0c802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7bcc13433708056 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.822+0000 7fa0d4b61640 1 -- 172.21.15.1:0/1387493792 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8039070 con 0x7fa0d01543a0 2024-03-06T07:26:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.822+0000 7fa0d4b61640 1 -- 172.21.15.1:0/1387493792 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c802cc60 con 0x7fa0d01543a0 2024-03-06T07:26:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.822+0000 7fa0d4b61640 1 -- 172.21.15.1:0/1387493792 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8034660 con 0x7fa0d01543a0 2024-03-06T07:26:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.822+0000 7fa0d7dc7640 1 -- 172.21.15.1:0/1387493792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d01543a0 msgr2=0x7fa0d015e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.822+0000 7fa0d7dc7640 1 --2- 172.21.15.1:0/1387493792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d01543a0 0x7fa0d015e840 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8009d90 tx=0x7fa0c802beb0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.823+0000 7fa0d7dc7640 1 -- 172.21.15.1:0/1387493792 shutdown_connections 2024-03-06T07:26:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.823+0000 7fa0d7dc7640 1 --2- 172.21.15.1:0/1387493792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d01543a0 0x7fa0d015e840 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.823+0000 7fa0d7dc7640 1 -- 172.21.15.1:0/1387493792 >> 172.21.15.1:0/1387493792 conn(0x7fa0d00c88f0 msgr2=0x7fa0d00c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.823+0000 7fa0d7dc7640 1 -- 172.21.15.1:0/1387493792 shutdown_connections 2024-03-06T07:26:26.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.823+0000 7fa0d7dc7640 1 -- 172.21.15.1:0/1387493792 wait complete. 2024-03-06T07:26:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.824+0000 7fa0d7dc7640 1 Processor -- start 2024-03-06T07:26:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.824+0000 7fa0d7dc7640 1 -- start start 2024-03-06T07:26:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.824+0000 7fa0d7dc7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0d00c32f0 0x7fa0d0143620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.824+0000 7fa0d7dc7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d0150520 0x7fa0d00c42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.824+0000 7fa0d7dc7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d01543a0 0x7fa0d01ebfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.824+0000 7fa0d7dc7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d01532a0 con 0x7fa0d01543a0 2024-03-06T07:26:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.824+0000 7fa0d7dc7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d01547e0 con 0x7fa0d0150520 2024-03-06T07:26:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.824+0000 7fa0d7dc7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d0157330 con 0x7fa0d00c32f0 2024-03-06T07:26:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.824+0000 7fa0d6364640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d01543a0 0x7fa0d01ebfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.824+0000 7fa0d6364640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d01543a0 0x7fa0d01ebfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58510/0 (socket says 172.21.15.1:58510) 2024-03-06T07:26:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.824+0000 7fa0d6364640 1 -- 172.21.15.1:0/3132171820 learned_addr learned my addr 172.21.15.1:0/3132171820 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:26.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.825+0000 7fa0d5362640 1 --2- 172.21.15.1:0/3132171820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d0150520 0x7fa0d00c42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.825+0000 7fa0d5b63640 1 --2- 172.21.15.1:0/3132171820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0d00c32f0 0x7fa0d0143620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.825+0000 7fa0d6364640 1 -- 172.21.15.1:0/3132171820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0d00c32f0 msgr2=0x7fa0d0143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.825+0000 7fa0d6364640 1 --2- 172.21.15.1:0/3132171820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0d00c32f0 0x7fa0d0143620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.825+0000 7fa0d6364640 1 -- 172.21.15.1:0/3132171820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d0150520 msgr2=0x7fa0d00c42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.825+0000 7fa0d6364640 1 --2- 172.21.15.1:0/3132171820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d0150520 0x7fa0d00c42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.825+0000 7fa0d6364640 1 -- 172.21.15.1:0/3132171820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d0148ed0 con 0x7fa0d01543a0 2024-03-06T07:26:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.825+0000 7fa0d5362640 1 --2- 172.21.15.1:0/3132171820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d0150520 0x7fa0d00c42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.825+0000 7fa0d6364640 1 --2- 172.21.15.1:0/3132171820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d01543a0 0x7fa0d01ebfc0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8037040 tx=0x7fa0c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.825+0000 7fa0c67fc640 1 -- 172.21.15.1:0/3132171820 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8044030 con 0x7fa0d01543a0 2024-03-06T07:26:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.825+0000 7fa0c67fc640 1 -- 172.21.15.1:0/3132171820 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c802ca60 con 0x7fa0d01543a0 2024-03-06T07:26:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.825+0000 7fa0d7dc7640 1 -- 172.21.15.1:0/3132171820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d0147a30 con 0x7fa0d01543a0 2024-03-06T07:26:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.825+0000 7fa0c67fc640 1 -- 172.21.15.1:0/3132171820 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8039040 con 0x7fa0d01543a0 2024-03-06T07:26:26.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.825+0000 7fa0d7dc7640 1 -- 172.21.15.1:0/3132171820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d0147470 con 0x7fa0d01543a0 2024-03-06T07:26:26.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.826+0000 7fa0d7dc7640 1 -- 172.21.15.1:0/3132171820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d00686e0 con 0x7fa0d01543a0 2024-03-06T07:26:26.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.827+0000 7fa0c67fc640 1 -- 172.21.15.1:0/3132171820 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0c8047070 con 0x7fa0d01543a0 2024-03-06T07:26:26.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.827+0000 7fa0c67fc640 1 --2- 172.21.15.1:0/3132171820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0b406b4d0 0x7fa0b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.828+0000 7fa0c67fc640 1 -- 172.21.15.1:0/3132171820 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fa0c80ad020 con 0x7fa0d01543a0 2024-03-06T07:26:26.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.828+0000 7fa0d5b63640 1 --2- 172.21.15.1:0/3132171820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0b406b4d0 0x7fa0b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.828+0000 7fa0d5b63640 1 --2- 172.21.15.1:0/3132171820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0b406b4d0 0x7fa0b406d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc000f60 tx=0x7fa0cc007b70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:26.832+0000 7fa0c67fc640 1 -- 172.21.15.1:0/3132171820 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0c804e070 con 0x7fa0d01543a0 2024-03-06T07:26:27.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.013+0000 7fa0d7dc7640 1 -- 172.21.15.1:0/3132171820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0d00c7b10 con 0x7fa0d01543a0 2024-03-06T07:26:27.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.015+0000 7fa0c67fc640 1 -- 172.21.15.1:0/3132171820 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+20495 (secure 0 0 0) 0x7fa0c807b9b0 con 0x7fa0d01543a0 2024-03-06T07:26:27.017 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:27.017 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":152,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:26:26.023754+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:15.245426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:15.942343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:26:24.304587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:26:25.085741+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:27.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.016+0000 7fa0d7dc7640 1 -- 172.21.15.1:0/3132171820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0b406b4d0 msgr2=0x7fa0b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.016+0000 7fa0d7dc7640 1 --2- 172.21.15.1:0/3132171820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0b406b4d0 0x7fa0b406d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc000f60 tx=0x7fa0cc007b70 comp rx=0 tx=0).stop 2024-03-06T07:26:27.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.016+0000 7fa0d7dc7640 1 -- 172.21.15.1:0/3132171820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d01543a0 msgr2=0x7fa0d01ebfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.016+0000 7fa0d7dc7640 1 --2- 172.21.15.1:0/3132171820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d01543a0 0x7fa0d01ebfc0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8037040 tx=0x7fa0c8004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:27.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.017+0000 7fa0d7dc7640 1 -- 172.21.15.1:0/3132171820 shutdown_connections 2024-03-06T07:26:27.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.017+0000 7fa0d7dc7640 1 --2- 172.21.15.1:0/3132171820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0b406b4d0 0x7fa0b406d990 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.017+0000 7fa0d7dc7640 1 --2- 172.21.15.1:0/3132171820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d01543a0 0x7fa0d01ebfc0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.017+0000 7fa0d7dc7640 1 --2- 172.21.15.1:0/3132171820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d0150520 0x7fa0d00c42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.017+0000 7fa0d7dc7640 1 --2- 172.21.15.1:0/3132171820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0d00c32f0 0x7fa0d0143620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.017+0000 7fa0d7dc7640 1 -- 172.21.15.1:0/3132171820 >> 172.21.15.1:0/3132171820 conn(0x7fa0d00c88f0 msgr2=0x7fa0d00c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.017+0000 7fa0d7dc7640 1 -- 172.21.15.1:0/3132171820 shutdown_connections 2024-03-06T07:26:27.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.017+0000 7fa0d7dc7640 1 -- 172.21.15.1:0/3132171820 wait complete. 2024-03-06T07:26:27.030 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:27.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.129+0000 7f597b179640 1 Processor -- start 2024-03-06T07:26:27.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.129+0000 7f597b179640 1 -- start start 2024-03-06T07:26:27.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.129+0000 7f597b179640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5974152690 0x7f5974155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f597b179640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59741561e0 con 0x7f59740c6730 2024-03-06T07:26:27.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f597b179640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59740c3300 con 0x7f5974156cd0 2024-03-06T07:26:27.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f597b179640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59740c35a0 con 0x7f5974152690 2024-03-06T07:26:27.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f5973fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f59740c6730 0x7f59740c57d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60316/0 (socket says 172.21.15.1:60316) 2024-03-06T07:26:27.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f5973fff640 1 -- 172.21.15.1:0/428435605 learned_addr learned my addr 172.21.15.1:0/428435605 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:27.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f5978f15640 1 --2- 172.21.15.1:0/428435605 >> v2:172.21.15.98:3300/0 conn(0x7f5974152690 0x7f5974155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f59737fe640 1 -- 172.21.15.1:0/428435605 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491121339 0 0) 0x7f5968001110 con 0x7f59740c6730 2024-03-06T07:26:27.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f59737fe640 1 -- 172.21.15.1:0/428435605 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f595c003520 con 0x7f59740c6730 2024-03-06T07:26:27.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f59737fe640 1 -- 172.21.15.1:0/428435605 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266996609 0 0) 0x7f59740c3300 con 0x7f5974156cd0 2024-03-06T07:26:27.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f59737fe640 1 -- 172.21.15.1:0/428435605 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5968001110 con 0x7f5974156cd0 2024-03-06T07:26:27.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f59737fe640 1 -- 172.21.15.1:0/428435605 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2949046881 0 0) 0x7f595c003520 con 0x7f59740c6730 2024-03-06T07:26:27.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f59737fe640 1 -- 172.21.15.1:0/428435605 >> v2:172.21.15.98:3300/0 conn(0x7f5974152690 msgr2=0x7f5974155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f59737fe640 1 --2- 172.21.15.1:0/428435605 >> v2:172.21.15.98:3300/0 conn(0x7f5974152690 0x7f5974155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f59737fe640 1 -- 172.21.15.1:0/428435605 >> v1:172.21.15.98:6789/0 conn(0x7f5974156cd0 legacy=0x7f5974157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f59737fe640 1 -- 172.21.15.1:0/428435605 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59740c3880 con 0x7f59740c6730 2024-03-06T07:26:27.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f5978f15640 1 --2- 172.21.15.1:0/428435605 >> v2:172.21.15.98:3300/0 conn(0x7f5974152690 0x7f5974155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:27.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f59737fe640 1 -- 172.21.15.1:0/428435605 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f5968001840 con 0x7f59740c6730 2024-03-06T07:26:27.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f59737fe640 1 -- 172.21.15.1:0/428435605 >> v1:172.21.15.1:6789/0 conn(0x7f59740c6730 legacy=0x7f59740c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f59737fe640 1 --2- 172.21.15.1:0/428435605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f595c003b40 0x7f595c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f59737fe640 1 --2- 172.21.15.1:0/428435605 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f595c004800 0x7f595c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f59737fe640 1 --2- 172.21.15.1:0/428435605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f595c0051c0 0x7f595c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f59737fe640 1 -- 172.21.15.1:0/428435605 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59740c35a0 con 0x7f595c0051c0 2024-03-06T07:26:27.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f59737fe640 1 -- 172.21.15.1:0/428435605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595c00c0f0 con 0x7f595c003b40 2024-03-06T07:26:27.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.130+0000 7f59737fe640 1 -- 172.21.15.1:0/428435605 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f595c00c2f0 con 0x7f595c004800 2024-03-06T07:26:27.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.131+0000 7f5979716640 1 --2- 172.21.15.1:0/428435605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f595c0051c0 0x7f595c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.131+0000 7f5978f15640 1 --2- 172.21.15.1:0/428435605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f595c003b40 0x7f595c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.131+0000 7f5973fff640 1 --2- 172.21.15.1:0/428435605 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f595c004800 0x7f595c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.131+0000 7f5979716640 1 -- 172.21.15.1:0/428435605 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f595c004800 msgr2=0x7f595c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.131+0000 7f5979716640 1 --2- 172.21.15.1:0/428435605 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f595c004800 0x7f595c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.131+0000 7f5979716640 1 -- 172.21.15.1:0/428435605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f595c003b40 msgr2=0x7f595c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.131+0000 7f5979716640 1 --2- 172.21.15.1:0/428435605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f595c003b40 0x7f595c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.131+0000 7f5979716640 1 -- 172.21.15.1:0/428435605 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595c003520 con 0x7f595c0051c0 2024-03-06T07:26:27.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.131+0000 7f5979716640 1 -- 172.21.15.1:0/428435605 reap_dead start 2024-03-06T07:26:27.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.132+0000 7f5979716640 1 --2- 172.21.15.1:0/428435605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f595c0051c0 0x7f595c00b9c0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f59640018d0 tx=0x7f595c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33494a8b5493f75 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.132+0000 7f59737fe640 1 -- 172.21.15.1:0/428435605 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59640014f0 con 0x7f595c0051c0 2024-03-06T07:26:27.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.132+0000 7f59737fe640 1 -- 172.21.15.1:0/428435605 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59640052d0 con 0x7f595c0051c0 2024-03-06T07:26:27.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.132+0000 7f59737fe640 1 -- 172.21.15.1:0/428435605 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59640037f0 con 0x7f595c0051c0 2024-03-06T07:26:27.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.132+0000 7f597b179640 1 -- 172.21.15.1:0/428435605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f595c0051c0 msgr2=0x7f595c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.132+0000 7f597b179640 1 --2- 172.21.15.1:0/428435605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f595c0051c0 0x7f595c00b9c0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f59640018d0 tx=0x7f595c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:27.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.132+0000 7f597b179640 1 -- 172.21.15.1:0/428435605 shutdown_connections 2024-03-06T07:26:27.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.132+0000 7f597b179640 1 --2- 172.21.15.1:0/428435605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f595c0051c0 0x7f595c00b9c0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.132+0000 7f597b179640 1 -- 172.21.15.1:0/428435605 >> 172.21.15.1:0/428435605 conn(0x7f597406b410 msgr2=0x7f597406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.133+0000 7f597b179640 1 -- 172.21.15.1:0/428435605 shutdown_connections 2024-03-06T07:26:27.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.133+0000 7f597b179640 1 -- 172.21.15.1:0/428435605 wait complete. 2024-03-06T07:26:27.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.134+0000 7f597b179640 1 Processor -- start 2024-03-06T07:26:27.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.134+0000 7f597b179640 1 -- start start 2024-03-06T07:26:27.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.135+0000 7f597b179640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f595c0051c0 0x7f5974150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.135+0000 7f597b179640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59740a7c80 0x7f59740a7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.135+0000 7f597b179640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5974154ce0 0x7f59741edfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.135+0000 7f597b179640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5974152a70 con 0x7f595c0051c0 2024-03-06T07:26:27.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.135+0000 7f597b179640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59741567b0 con 0x7f5974154ce0 2024-03-06T07:26:27.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.135+0000 7f597b179640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5974152bf0 con 0x7f59740a7c80 2024-03-06T07:26:27.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.136+0000 7f5978f15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f595c0051c0 0x7f5974150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.136+0000 7f5978f15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f595c0051c0 0x7f5974150ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58538/0 (socket says 172.21.15.1:58538) 2024-03-06T07:26:27.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.136+0000 7f5978f15640 1 -- 172.21.15.1:0/767057503 learned_addr learned my addr 172.21.15.1:0/767057503 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:27.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.136+0000 7f5973fff640 1 --2- 172.21.15.1:0/767057503 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59740a7c80 0x7f59740a7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.136+0000 7f5979716640 1 --2- 172.21.15.1:0/767057503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5974154ce0 0x7f59741edfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.136+0000 7f5978f15640 1 -- 172.21.15.1:0/767057503 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59740a7c80 msgr2=0x7f59740a7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.136+0000 7f5978f15640 1 --2- 172.21.15.1:0/767057503 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59740a7c80 0x7f59740a7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.136+0000 7f5978f15640 1 -- 172.21.15.1:0/767057503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5974154ce0 msgr2=0x7f59741edfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.136+0000 7f5978f15640 1 --2- 172.21.15.1:0/767057503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5974154ce0 0x7f59741edfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.136+0000 7f5978f15640 1 -- 172.21.15.1:0/767057503 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59741eb100 con 0x7f595c0051c0 2024-03-06T07:26:27.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.136+0000 7f5978f15640 1 --2- 172.21.15.1:0/767057503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f595c0051c0 0x7f5974150ce0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f596c00f490 tx=0x7f596c00f950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.136+0000 7f59717fa640 1 -- 172.21.15.1:0/767057503 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c003b90 con 0x7f595c0051c0 2024-03-06T07:26:27.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.136+0000 7f59717fa640 1 -- 172.21.15.1:0/767057503 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596c003d50 con 0x7f595c0051c0 2024-03-06T07:26:27.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.136+0000 7f597b179640 1 -- 172.21.15.1:0/767057503 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5974154730 con 0x7f595c0051c0 2024-03-06T07:26:27.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.136+0000 7f59717fa640 1 -- 172.21.15.1:0/767057503 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c004270 con 0x7f595c0051c0 2024-03-06T07:26:27.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.136+0000 7f597b179640 1 -- 172.21.15.1:0/767057503 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5974151220 con 0x7f595c0051c0 2024-03-06T07:26:27.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.137+0000 7f597b179640 1 -- 172.21.15.1:0/767057503 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5974067fd0 con 0x7f595c0051c0 2024-03-06T07:26:27.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.138+0000 7f59717fa640 1 -- 172.21.15.1:0/767057503 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f596c004410 con 0x7f595c0051c0 2024-03-06T07:26:27.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.138+0000 7f59717fa640 1 --2- 172.21.15.1:0/767057503 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f596006b500 0x7f596006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.139+0000 7f59717fa640 1 -- 172.21.15.1:0/767057503 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f596c08ba20 con 0x7f595c0051c0 2024-03-06T07:26:27.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.139+0000 7f5973fff640 1 --2- 172.21.15.1:0/767057503 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f596006b500 0x7f596006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.139+0000 7f5973fff640 1 --2- 172.21.15.1:0/767057503 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f596006b500 0x7f596006d9c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f59741e8420 tx=0x7f5968007a20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.143+0000 7f59717fa640 1 -- 172.21.15.1:0/767057503 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f596c019070 con 0x7f595c0051c0 2024-03-06T07:26:27.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.325+0000 7f597b179640 1 -- 172.21.15.1:0/767057503 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59740c2c40 con 0x7f595c0051c0 2024-03-06T07:26:27.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.327+0000 7f59717fa640 1 -- 172.21.15.1:0/767057503 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+20495 (secure 0 0 0) 0x7f596c010040 con 0x7f595c0051c0 2024-03-06T07:26:27.329 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:27.329 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":152,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:26:26.023754+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:15.245426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:15.942343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:26:24.304587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:26:25.085741+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:27.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.329+0000 7f597b179640 1 -- 172.21.15.1:0/767057503 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f596006b500 msgr2=0x7f596006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.329+0000 7f597b179640 1 --2- 172.21.15.1:0/767057503 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f596006b500 0x7f596006d9c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f59741e8420 tx=0x7f5968007a20 comp rx=0 tx=0).stop 2024-03-06T07:26:27.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.329+0000 7f597b179640 1 -- 172.21.15.1:0/767057503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f595c0051c0 msgr2=0x7f5974150ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.329+0000 7f597b179640 1 --2- 172.21.15.1:0/767057503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f595c0051c0 0x7f5974150ce0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f596c00f490 tx=0x7f596c00f950 comp rx=0 tx=0).stop 2024-03-06T07:26:27.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.329+0000 7f597b179640 1 -- 172.21.15.1:0/767057503 shutdown_connections 2024-03-06T07:26:27.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.329+0000 7f597b179640 1 --2- 172.21.15.1:0/767057503 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f596006b500 0x7f596006d9c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.329+0000 7f597b179640 1 --2- 172.21.15.1:0/767057503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5974154ce0 0x7f59741edfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.329+0000 7f597b179640 1 --2- 172.21.15.1:0/767057503 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59740a7c80 0x7f59740a7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.329+0000 7f597b179640 1 --2- 172.21.15.1:0/767057503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f595c0051c0 0x7f5974150ce0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.329+0000 7f597b179640 1 -- 172.21.15.1:0/767057503 >> 172.21.15.1:0/767057503 conn(0x7f597406b410 msgr2=0x7f597406f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.329+0000 7f597b179640 1 -- 172.21.15.1:0/767057503 shutdown_connections 2024-03-06T07:26:27.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.329+0000 7f597b179640 1 -- 172.21.15.1:0/767057503 wait complete. 2024-03-06T07:26:27.342 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 standby_count_wanted 0 2024-03-06T07:26:27.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.445+0000 7f33c6d72640 1 Processor -- start 2024-03-06T07:26:27.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.445+0000 7f33c6d72640 1 -- start start 2024-03-06T07:26:27.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.445+0000 7f33c6d72640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f33c0152690 0x7f33c0155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.445+0000 7f33c6d72640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33c00c6730 0x7f33c00c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.446+0000 7f33c6d72640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c01561e0 con 0x7f33c00c5d10 2024-03-06T07:26:27.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.446+0000 7f33c6d72640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c00c3300 con 0x7f33c0152690 2024-03-06T07:26:27.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.446+0000 7f33c6d72640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c00c35a0 con 0x7f33c00c6730 2024-03-06T07:26:27.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.446+0000 7f33c4b0e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f33c0152690 0x7f33c0155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.446+0000 7f33c4b0e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f33c0152690 0x7f33c0155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58546/0 (socket says 172.21.15.1:58546) 2024-03-06T07:26:27.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.446+0000 7f33c4b0e640 1 -- 172.21.15.1:0/2022054584 learned_addr learned my addr 172.21.15.1:0/2022054584 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:27.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.446+0000 7f33bffff640 1 --2- 172.21.15.1:0/2022054584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33c00c6730 0x7f33c00c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.446+0000 7f33c4b0e640 1 -- 172.21.15.1:0/2022054584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33c00c6730 msgr2=0x7f33c00c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.446+0000 7f33c4b0e640 1 --2- 172.21.15.1:0/2022054584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33c00c6730 0x7f33c00c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.446+0000 7f33c4b0e640 1 -- 172.21.15.1:0/2022054584 >> v1:172.21.15.98:6789/0 conn(0x7f33c00c5d10 legacy=0x7f33c0156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.446+0000 7f33c4b0e640 1 -- 172.21.15.1:0/2022054584 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c00c3880 con 0x7f33c0152690 2024-03-06T07:26:27.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.446+0000 7f33bffff640 1 --2- 172.21.15.1:0/2022054584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33c00c6730 0x7f33c00c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:27.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.447+0000 7f33c4b0e640 1 --2- 172.21.15.1:0/2022054584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33c0152690 0x7f33c0155ca0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f33b4009870 tx=0x7f33b402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b6ffd04bcfb6d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.447+0000 7f33bf7fe640 1 -- 172.21.15.1:0/2022054584 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b4035020 con 0x7f33c0152690 2024-03-06T07:26:27.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.447+0000 7f33bf7fe640 1 -- 172.21.15.1:0/2022054584 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b4031640 con 0x7f33c0152690 2024-03-06T07:26:27.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.447+0000 7f33bf7fe640 1 -- 172.21.15.1:0/2022054584 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b402d040 con 0x7f33c0152690 2024-03-06T07:26:27.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.447+0000 7f33c6d72640 1 -- 172.21.15.1:0/2022054584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33c0152690 msgr2=0x7f33c0155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.447+0000 7f33c6d72640 1 --2- 172.21.15.1:0/2022054584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33c0152690 0x7f33c0155ca0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f33b4009870 tx=0x7f33b402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.447+0000 7f33c6d72640 1 -- 172.21.15.1:0/2022054584 shutdown_connections 2024-03-06T07:26:27.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.447+0000 7f33c6d72640 1 --2- 172.21.15.1:0/2022054584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33c00c6730 0x7f33c00c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.447+0000 7f33c6d72640 1 --2- 172.21.15.1:0/2022054584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33c0152690 0x7f33c0155ca0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.447+0000 7f33c6d72640 1 -- 172.21.15.1:0/2022054584 >> 172.21.15.1:0/2022054584 conn(0x7f33c006b410 msgr2=0x7f33c006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.448+0000 7f33c6d72640 1 -- 172.21.15.1:0/2022054584 shutdown_connections 2024-03-06T07:26:27.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.448+0000 7f33c6d72640 1 -- 172.21.15.1:0/2022054584 wait complete. 2024-03-06T07:26:27.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.449+0000 7f33c6d72640 1 Processor -- start 2024-03-06T07:26:27.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.449+0000 7f33c6d72640 1 -- start start 2024-03-06T07:26:27.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.449+0000 7f33c6d72640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33c00c5d10 0x7f33c01465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.449+0000 7f33c6d72640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33c00c6730 0x7f33c006d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.449+0000 7f33c6d72640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33c0152690 0x7f33c00ac220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.449+0000 7f33c6d72640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c0152a70 con 0x7f33c00c5d10 2024-03-06T07:26:27.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.449+0000 7f33c6d72640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c01567b0 con 0x7f33c00c6730 2024-03-06T07:26:27.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.449+0000 7f33c6d72640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c0152bf0 con 0x7f33c0152690 2024-03-06T07:26:27.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.449+0000 7f33c4b0e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33c00c5d10 0x7f33c01465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.450+0000 7f33c4b0e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33c00c5d10 0x7f33c01465d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58550/0 (socket says 172.21.15.1:58550) 2024-03-06T07:26:27.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.450+0000 7f33c4b0e640 1 -- 172.21.15.1:0/370329081 learned_addr learned my addr 172.21.15.1:0/370329081 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:27.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.450+0000 7f33c530f640 1 --2- 172.21.15.1:0/370329081 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33c0152690 0x7f33c00ac220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.450+0000 7f33bffff640 1 --2- 172.21.15.1:0/370329081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33c00c6730 0x7f33c006d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.450+0000 7f33c4b0e640 1 -- 172.21.15.1:0/370329081 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33c0152690 msgr2=0x7f33c00ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.450+0000 7f33c4b0e640 1 --2- 172.21.15.1:0/370329081 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33c0152690 0x7f33c00ac220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.450+0000 7f33c4b0e640 1 -- 172.21.15.1:0/370329081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33c00c6730 msgr2=0x7f33c006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.450+0000 7f33c4b0e640 1 --2- 172.21.15.1:0/370329081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33c00c6730 0x7f33c006d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.450+0000 7f33c4b0e640 1 -- 172.21.15.1:0/370329081 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c00c7250 con 0x7f33c00c5d10 2024-03-06T07:26:27.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.450+0000 7f33c4b0e640 1 --2- 172.21.15.1:0/370329081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33c00c5d10 0x7f33c01465d0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f33b402b930 tx=0x7f33b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.450+0000 7f33bd7fa640 1 -- 172.21.15.1:0/370329081 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b4040070 con 0x7f33c00c5d10 2024-03-06T07:26:27.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.450+0000 7f33bd7fa640 1 -- 172.21.15.1:0/370329081 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b403e070 con 0x7f33c00c5d10 2024-03-06T07:26:27.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.450+0000 7f33c6d72640 1 -- 172.21.15.1:0/370329081 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c0146e40 con 0x7f33c00c5d10 2024-03-06T07:26:27.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.450+0000 7f33bd7fa640 1 -- 172.21.15.1:0/370329081 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b4035040 con 0x7f33c00c5d10 2024-03-06T07:26:27.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.450+0000 7f33c6d72640 1 -- 172.21.15.1:0/370329081 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c00c6f70 con 0x7f33c00c5d10 2024-03-06T07:26:27.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.451+0000 7f33c6d72640 1 -- 172.21.15.1:0/370329081 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c0067fd0 con 0x7f33c00c5d10 2024-03-06T07:26:27.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.452+0000 7f33bd7fa640 1 -- 172.21.15.1:0/370329081 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33b4038020 con 0x7f33c00c5d10 2024-03-06T07:26:27.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.452+0000 7f33bd7fa640 1 --2- 172.21.15.1:0/370329081 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f33a406b4d0 0x7f33a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.452+0000 7f33bd7fa640 1 -- 172.21.15.1:0/370329081 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f33b40ac0c0 con 0x7f33c00c5d10 2024-03-06T07:26:27.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.452+0000 7f33bffff640 1 --2- 172.21.15.1:0/370329081 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f33a406b4d0 0x7f33a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.453+0000 7f33bffff640 1 --2- 172.21.15.1:0/370329081 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f33a406b4d0 0x7f33a406d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f33c0157c30 tx=0x7f33ac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.460+0000 7f33bd7fa640 1 -- 172.21.15.1:0/370329081 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33b407aa60 con 0x7f33c00c5d10 2024-03-06T07:26:27.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.685+0000 7f33c6d72640 1 -- 172.21.15.1:0/370329081 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f33c00c2c40 con 0x7f33c00c5d10 2024-03-06T07:26:27.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.694+0000 7f33bd7fa640 1 -- 172.21.15.1:0/370329081 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"}]=0 v139) v1 ==== 119+0+0 (secure 0 0 0) 0x7f33b4043e40 con 0x7f33c00c5d10 2024-03-06T07:26:27.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.697+0000 7f33c6d72640 1 -- 172.21.15.1:0/370329081 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f33a406b4d0 msgr2=0x7f33a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.697+0000 7f33c6d72640 1 --2- 172.21.15.1:0/370329081 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f33a406b4d0 0x7f33a406d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f33c0157c30 tx=0x7f33ac008040 comp rx=0 tx=0).stop 2024-03-06T07:26:27.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.697+0000 7f33c6d72640 1 -- 172.21.15.1:0/370329081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33c00c5d10 msgr2=0x7f33c01465d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.697+0000 7f33c6d72640 1 --2- 172.21.15.1:0/370329081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33c00c5d10 0x7f33c01465d0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f33b402b930 tx=0x7f33b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.697+0000 7f33c6d72640 1 -- 172.21.15.1:0/370329081 shutdown_connections 2024-03-06T07:26:27.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.697+0000 7f33c6d72640 1 --2- 172.21.15.1:0/370329081 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f33a406b4d0 0x7f33a406d990 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.697+0000 7f33c6d72640 1 --2- 172.21.15.1:0/370329081 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33c0152690 0x7f33c00ac220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.697+0000 7f33c6d72640 1 --2- 172.21.15.1:0/370329081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33c00c6730 0x7f33c006d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.697+0000 7f33c6d72640 1 --2- 172.21.15.1:0/370329081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33c00c5d10 0x7f33c01465d0 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.697+0000 7f33c6d72640 1 -- 172.21.15.1:0/370329081 >> 172.21.15.1:0/370329081 conn(0x7f33c006b410 msgr2=0x7f33c0155250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.697+0000 7f33c6d72640 1 -- 172.21.15.1:0/370329081 shutdown_connections 2024-03-06T07:26:27.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.697+0000 7f33c6d72640 1 -- 172.21.15.1:0/370329081 wait complete. 2024-03-06T07:26:27.712 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:26:27.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 allow_standby_replay false 2024-03-06T07:26:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.814+0000 7fde7aa15640 1 Processor -- start 2024-03-06T07:26:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.814+0000 7fde7aa15640 1 -- start start 2024-03-06T07:26:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.815+0000 7fde7aa15640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fde74152730 0x7fde740c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.815+0000 7fde7aa15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde740c6fd0 0x7fde740c1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.815+0000 7fde7aa15640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde741588e0 con 0x7fde740c18d0 2024-03-06T07:26:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.815+0000 7fde7aa15640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde740c1d20 con 0x7fde74152730 2024-03-06T07:26:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.815+0000 7fde7aa15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde74158bc0 con 0x7fde740c6fd0 2024-03-06T07:26:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.815+0000 7fde6bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde740c6fd0 0x7fde740c1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.815+0000 7fde73fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fde74152730 0x7fde740c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.815+0000 7fde6bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde740c6fd0 0x7fde740c1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:55840/0 (socket says 172.21.15.1:55840) 2024-03-06T07:26:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.815+0000 7fde6bfff640 1 -- 172.21.15.1:0/3768675690 learned_addr learned my addr 172.21.15.1:0/3768675690 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.816+0000 7fde6bfff640 1 -- 172.21.15.1:0/3768675690 >> v2:172.21.15.98:3300/0 conn(0x7fde74152730 msgr2=0x7fde740c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.816+0000 7fde6bfff640 1 --2- 172.21.15.1:0/3768675690 >> v2:172.21.15.98:3300/0 conn(0x7fde74152730 0x7fde740c6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.816+0000 7fde6bfff640 1 -- 172.21.15.1:0/3768675690 >> v1:172.21.15.98:6789/0 conn(0x7fde740c18d0 legacy=0x7fde741552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.816+0000 7fde6bfff640 1 -- 172.21.15.1:0/3768675690 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde74158ea0 con 0x7fde740c6fd0 2024-03-06T07:26:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.816+0000 7fde6bfff640 1 --2- 172.21.15.1:0/3768675690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde740c6fd0 0x7fde740c1390 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fde5c009870 tx=0x7fde5c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bde4988ede30ce6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.816+0000 7fde737fe640 1 -- 172.21.15.1:0/3768675690 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c0317c0 con 0x7fde740c6fd0 2024-03-06T07:26:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.817+0000 7fde737fe640 1 -- 172.21.15.1:0/3768675690 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5c037070 con 0x7fde740c6fd0 2024-03-06T07:26:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.817+0000 7fde737fe640 1 -- 172.21.15.1:0/3768675690 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c031b80 con 0x7fde740c6fd0 2024-03-06T07:26:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.817+0000 7fde7aa15640 1 -- 172.21.15.1:0/3768675690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde740c6fd0 msgr2=0x7fde740c1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.817+0000 7fde7aa15640 1 --2- 172.21.15.1:0/3768675690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde740c6fd0 0x7fde740c1390 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fde5c009870 tx=0x7fde5c02b860 comp rx=0 tx=0).stop 2024-03-06T07:26:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.817+0000 7fde7aa15640 1 -- 172.21.15.1:0/3768675690 shutdown_connections 2024-03-06T07:26:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.817+0000 7fde7aa15640 1 --2- 172.21.15.1:0/3768675690 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde740c6fd0 0x7fde740c1390 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.817+0000 7fde7aa15640 1 --2- 172.21.15.1:0/3768675690 >> v2:172.21.15.98:3300/0 conn(0x7fde74152730 0x7fde740c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.817+0000 7fde7aa15640 1 -- 172.21.15.1:0/3768675690 >> 172.21.15.1:0/3768675690 conn(0x7fde74066d00 msgr2=0x7fde74067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.817+0000 7fde7aa15640 1 -- 172.21.15.1:0/3768675690 shutdown_connections 2024-03-06T07:26:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.817+0000 7fde7aa15640 1 -- 172.21.15.1:0/3768675690 wait complete. 2024-03-06T07:26:27.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.818+0000 7fde7aa15640 1 Processor -- start 2024-03-06T07:26:27.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.818+0000 7fde7aa15640 1 -- start start 2024-03-06T07:26:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.819+0000 7fde7aa15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde740c18d0 0x7fde741edfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.819+0000 7fde7aa15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde740c6fd0 0x7fde741df520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.819+0000 7fde7aa15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde74152730 0x7fde740c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.819+0000 7fde7aa15640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde74152b10 con 0x7fde74152730 2024-03-06T07:26:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.819+0000 7fde7aa15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde74159460 con 0x7fde740c18d0 2024-03-06T07:26:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.819+0000 7fde7aa15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde741559f0 con 0x7fde740c6fd0 2024-03-06T07:26:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.819+0000 7fde78fb2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde74152730 0x7fde740c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.819+0000 7fde78fb2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde74152730 0x7fde740c8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58558/0 (socket says 172.21.15.1:58558) 2024-03-06T07:26:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.819+0000 7fde78fb2640 1 -- 172.21.15.1:0/999436626 learned_addr learned my addr 172.21.15.1:0/999436626 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.819+0000 7fde6bfff640 1 --2- 172.21.15.1:0/999436626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde740c6fd0 0x7fde741df520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.819+0000 7fde73fff640 1 --2- 172.21.15.1:0/999436626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde740c18d0 0x7fde741edfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.819+0000 7fde78fb2640 1 -- 172.21.15.1:0/999436626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde740c6fd0 msgr2=0x7fde741df520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.819+0000 7fde78fb2640 1 --2- 172.21.15.1:0/999436626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde740c6fd0 0x7fde741df520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.819+0000 7fde78fb2640 1 -- 172.21.15.1:0/999436626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde740c18d0 msgr2=0x7fde741edfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.819+0000 7fde78fb2640 1 --2- 172.21.15.1:0/999436626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde740c18d0 0x7fde741edfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.819+0000 7fde78fb2640 1 -- 172.21.15.1:0/999436626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde741de870 con 0x7fde74152730 2024-03-06T07:26:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.820+0000 7fde78fb2640 1 --2- 172.21.15.1:0/999436626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde74152730 0x7fde740c8810 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00c6b0 tx=0x7fde6c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.820+0000 7fde717fa640 1 -- 172.21.15.1:0/999436626 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c003b90 con 0x7fde74152730 2024-03-06T07:26:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.820+0000 7fde717fa640 1 -- 172.21.15.1:0/999436626 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6c003d30 con 0x7fde74152730 2024-03-06T07:26:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.820+0000 7fde717fa640 1 -- 172.21.15.1:0/999436626 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c005310 con 0x7fde74152730 2024-03-06T07:26:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.820+0000 7fde7aa15640 1 -- 172.21.15.1:0/999436626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7414ea70 con 0x7fde74152730 2024-03-06T07:26:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.820+0000 7fde7aa15640 1 -- 172.21.15.1:0/999436626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde740ca300 con 0x7fde74152730 2024-03-06T07:26:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.821+0000 7fde7aa15640 1 -- 172.21.15.1:0/999436626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde74067fd0 con 0x7fde74152730 2024-03-06T07:26:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.821+0000 7fde717fa640 1 -- 172.21.15.1:0/999436626 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde6c0054b0 con 0x7fde74152730 2024-03-06T07:26:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.822+0000 7fde717fa640 1 --2- 172.21.15.1:0/999436626 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fde5406b500 0x7fde5406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.822+0000 7fde717fa640 1 -- 172.21.15.1:0/999436626 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fde6c08a4f0 con 0x7fde74152730 2024-03-06T07:26:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.822+0000 7fde73fff640 1 --2- 172.21.15.1:0/999436626 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fde5406b500 0x7fde5406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.823+0000 7fde73fff640 1 --2- 172.21.15.1:0/999436626 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fde5406b500 0x7fde5406d9c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fde64009a30 tx=0x7fde64008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:27.826+0000 7fde717fa640 1 -- 172.21.15.1:0/999436626 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde6c058e90 con 0x7fde74152730 2024-03-06T07:26:27.903 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-06T07:26:27.902+0000 7f3676d68640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:28.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.059+0000 7fde7aa15640 1 -- 172.21.15.1:0/999436626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fde740c6870 con 0x7fde74152730 2024-03-06T07:26:28.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.695+0000 7fde717fa640 1 -- 172.21.15.1:0/999436626 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"}]=0 v140) v1 ==== 123+0+0 (secure 0 0 0) 0x7fde6c058790 con 0x7fde74152730 2024-03-06T07:26:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.698+0000 7fde7aa15640 1 -- 172.21.15.1:0/999436626 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fde5406b500 msgr2=0x7fde5406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.698+0000 7fde7aa15640 1 --2- 172.21.15.1:0/999436626 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fde5406b500 0x7fde5406d9c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fde64009a30 tx=0x7fde64008040 comp rx=0 tx=0).stop 2024-03-06T07:26:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.698+0000 7fde7aa15640 1 -- 172.21.15.1:0/999436626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde74152730 msgr2=0x7fde740c8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.698+0000 7fde7aa15640 1 --2- 172.21.15.1:0/999436626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde74152730 0x7fde740c8810 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00c6b0 tx=0x7fde6c00cb70 comp rx=0 tx=0).stop 2024-03-06T07:26:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.698+0000 7fde7aa15640 1 -- 172.21.15.1:0/999436626 shutdown_connections 2024-03-06T07:26:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.698+0000 7fde7aa15640 1 --2- 172.21.15.1:0/999436626 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fde5406b500 0x7fde5406d9c0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.698+0000 7fde7aa15640 1 --2- 172.21.15.1:0/999436626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde74152730 0x7fde740c8810 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.698+0000 7fde7aa15640 1 --2- 172.21.15.1:0/999436626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde740c6fd0 0x7fde741df520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.698+0000 7fde7aa15640 1 --2- 172.21.15.1:0/999436626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde740c18d0 0x7fde741edfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.698+0000 7fde7aa15640 1 -- 172.21.15.1:0/999436626 >> 172.21.15.1:0/999436626 conn(0x7fde74066d00 msgr2=0x7fde740c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.698+0000 7fde7aa15640 1 -- 172.21.15.1:0/999436626 shutdown_connections 2024-03-06T07:26:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.698+0000 7fde7aa15640 1 -- 172.21.15.1:0/999436626 wait complete. 2024-03-06T07:26:28.712 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs2' 2024-03-06T07:26:28.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:28.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.815+0000 7ff8745b6640 1 Processor -- start 2024-03-06T07:26:28.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.815+0000 7ff8745b6640 1 -- start start 2024-03-06T07:26:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.815+0000 7ff8745b6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff86c0c3970 0x7ff86c0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.815+0000 7ff8745b6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86c0c5e40 con 0x7ff86c0c60b0 2024-03-06T07:26:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.815+0000 7ff8745b6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86c0c6a40 con 0x7ff86c152840 2024-03-06T07:26:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.815+0000 7ff8745b6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff86c0c67f0 con 0x7ff86c0c3970 2024-03-06T07:26:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.815+0000 7ff872352640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff86c0c3970 0x7ff86c0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.815+0000 7ff871b51640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff86c0c60b0 0x7ff86c152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60326/0 (socket says 172.21.15.1:60326) 2024-03-06T07:26:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.816+0000 7ff872352640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff86c0c3970 0x7ff86c0c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58560/0 (socket says 172.21.15.1:58560) 2024-03-06T07:26:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.816+0000 7ff871b51640 1 -- 172.21.15.1:0/776601859 learned_addr learned my addr 172.21.15.1:0/776601859 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.816+0000 7ff872352640 1 -- 172.21.15.1:0/776601859 >> v1:172.21.15.98:6789/0 conn(0x7ff86c152840 legacy=0x7ff86c1553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.816+0000 7ff872352640 1 -- 172.21.15.1:0/776601859 >> v1:172.21.15.1:6789/0 conn(0x7ff86c0c60b0 legacy=0x7ff86c152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.816+0000 7ff872352640 1 -- 172.21.15.1:0/776601859 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c0c6d20 con 0x7ff86c0c3970 2024-03-06T07:26:28.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.817+0000 7ff872352640 1 --2- 172.21.15.1:0/776601859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff86c0c3970 0x7ff86c0c8170 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7ff8680097a0 tx=0x7ff86802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66f840c14ce72ca7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.817+0000 7ff871350640 1 -- 172.21.15.1:0/776601859 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868034070 con 0x7ff86c0c3970 2024-03-06T07:26:28.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.817+0000 7ff871350640 1 -- 172.21.15.1:0/776601859 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff868030610 con 0x7ff86c0c3970 2024-03-06T07:26:28.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.817+0000 7ff871350640 1 -- 172.21.15.1:0/776601859 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8680309e0 con 0x7ff86c0c3970 2024-03-06T07:26:28.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.817+0000 7ff8745b6640 1 -- 172.21.15.1:0/776601859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff86c0c3970 msgr2=0x7ff86c0c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.817+0000 7ff8745b6640 1 --2- 172.21.15.1:0/776601859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff86c0c3970 0x7ff86c0c8170 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7ff8680097a0 tx=0x7ff86802b960 comp rx=0 tx=0).stop 2024-03-06T07:26:28.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.817+0000 7ff8745b6640 1 -- 172.21.15.1:0/776601859 shutdown_connections 2024-03-06T07:26:28.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.817+0000 7ff8745b6640 1 --2- 172.21.15.1:0/776601859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff86c0c3970 0x7ff86c0c8170 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.817+0000 7ff8745b6640 1 -- 172.21.15.1:0/776601859 >> 172.21.15.1:0/776601859 conn(0x7ff86c0c88f0 msgr2=0x7ff86c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:28.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.817+0000 7ff8745b6640 1 -- 172.21.15.1:0/776601859 shutdown_connections 2024-03-06T07:26:28.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.817+0000 7ff8745b6640 1 -- 172.21.15.1:0/776601859 wait complete. 2024-03-06T07:26:28.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.818+0000 7ff8745b6640 1 Processor -- start 2024-03-06T07:26:28.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.819+0000 7ff8745b6640 1 -- start start 2024-03-06T07:26:28.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.819+0000 7ff8745b6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff86c0c3970 0x7ff86c0a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.819+0000 7ff8745b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff86c0c60b0 0x7ff86c1ee420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.819+0000 7ff8745b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff86c152840 0x7ff86c14cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.819+0000 7ff8745b6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c0c75c0 con 0x7ff86c0c3970 2024-03-06T07:26:28.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.819+0000 7ff8745b6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c0c72e0 con 0x7ff86c152840 2024-03-06T07:26:28.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.819+0000 7ff8745b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c0c3d50 con 0x7ff86c0c60b0 2024-03-06T07:26:28.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.819+0000 7ff872352640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff86c0c3970 0x7ff86c0a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.819+0000 7ff872352640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff86c0c3970 0x7ff86c0a97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58564/0 (socket says 172.21.15.1:58564) 2024-03-06T07:26:28.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.819+0000 7ff872352640 1 -- 172.21.15.1:0/137627036 learned_addr learned my addr 172.21.15.1:0/137627036 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:28.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.820+0000 7ff871b51640 1 --2- 172.21.15.1:0/137627036 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff86c0c60b0 0x7ff86c1ee420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.820+0000 7ff872b53640 1 --2- 172.21.15.1:0/137627036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff86c152840 0x7ff86c14cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.820+0000 7ff872352640 1 -- 172.21.15.1:0/137627036 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff86c0c60b0 msgr2=0x7ff86c1ee420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.820+0000 7ff872352640 1 --2- 172.21.15.1:0/137627036 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff86c0c60b0 0x7ff86c1ee420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.820+0000 7ff872352640 1 -- 172.21.15.1:0/137627036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff86c152840 msgr2=0x7ff86c14cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.820+0000 7ff872352640 1 --2- 172.21.15.1:0/137627036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff86c152840 0x7ff86c14cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.820+0000 7ff872352640 1 -- 172.21.15.1:0/137627036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c0ac410 con 0x7ff86c0c3970 2024-03-06T07:26:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.820+0000 7ff872b53640 1 --2- 172.21.15.1:0/137627036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff86c152840 0x7ff86c14cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.820+0000 7ff871b51640 1 --2- 172.21.15.1:0/137627036 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff86c0c60b0 0x7ff86c1ee420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.820+0000 7ff872352640 1 --2- 172.21.15.1:0/137627036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff86c0c3970 0x7ff86c0a97d0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7ff8680098c0 tx=0x7ff8680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.820+0000 7ff862ffd640 1 -- 172.21.15.1:0/137627036 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868034040 con 0x7ff86c0c3970 2024-03-06T07:26:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.820+0000 7ff862ffd640 1 -- 172.21.15.1:0/137627036 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff868039a10 con 0x7ff86c0c3970 2024-03-06T07:26:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.820+0000 7ff8745b6640 1 -- 172.21.15.1:0/137627036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff86c070cc0 con 0x7ff86c0c3970 2024-03-06T07:26:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.820+0000 7ff862ffd640 1 -- 172.21.15.1:0/137627036 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868039de0 con 0x7ff86c0c3970 2024-03-06T07:26:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.820+0000 7ff8745b6640 1 -- 172.21.15.1:0/137627036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86c071280 con 0x7ff86c0c3970 2024-03-06T07:26:28.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.821+0000 7ff8745b6640 1 -- 172.21.15.1:0/137627036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff86c0686e0 con 0x7ff86c0c3970 2024-03-06T07:26:28.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.822+0000 7ff862ffd640 1 -- 172.21.15.1:0/137627036 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff868037070 con 0x7ff86c0c3970 2024-03-06T07:26:28.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.822+0000 7ff862ffd640 1 --2- 172.21.15.1:0/137627036 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff85006b500 0x7ff85006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.822+0000 7ff862ffd640 1 -- 172.21.15.1:0/137627036 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7ff8680ad190 con 0x7ff86c0c3970 2024-03-06T07:26:28.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.822+0000 7ff871b51640 1 --2- 172.21.15.1:0/137627036 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff85006b500 0x7ff85006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.823+0000 7ff871b51640 1 --2- 172.21.15.1:0/137627036 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff85006b500 0x7ff85006d9c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff86c1eb870 tx=0x7ff85c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:28.826+0000 7ff862ffd640 1 -- 172.21.15.1:0/137627036 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff86804a360 con 0x7ff86c0c3970 2024-03-06T07:26:29.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.046+0000 7ff8745b6640 1 -- 172.21.15.1:0/137627036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff86c151fa0 con 0x7ff86c0c3970 2024-03-06T07:26:29.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.046+0000 7ff862ffd640 1 -- 172.21.15.1:0/137627036 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+6260 (secure 0 0 0) 0x7ff86807bca0 con 0x7ff86c0c3970 2024-03-06T07:26:29.048 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:29.048 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":140,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6989,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2360179300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2360179300},{"type":"v1","addr":"172.21.15.98:6837","nonce":2360179300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":7774,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3360241462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3360241462},{"type":"v1","addr":"172.21.15.98:6836","nonce":3360241462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:16.977128+0000","modified":"2024-03-06T07:26:18.975595+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8823},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8823":{"gid":8823,"name":"a","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/67091521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":67091521},{"type":"v1","addr":"172.21.15.1:6835","nonce":67091521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8823,"qdb_cluster":[8823]},"id":11},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:26.024256+0000","modified":"2024-03-06T07:26:28.060419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8820},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8820":{"gid":8820,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":5,"addr":"172.21.15.1:6833/1089548586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1089548586},{"type":"v1","addr":"172.21.15.1:6833","nonce":1089548586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8820,"qdb_cluster":[8820]},"id":12}]} 2024-03-06T07:26:29.049 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 140 2024-03-06T07:26:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.048+0000 7ff8745b6640 1 -- 172.21.15.1:0/137627036 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff85006b500 msgr2=0x7ff85006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.048+0000 7ff8745b6640 1 --2- 172.21.15.1:0/137627036 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff85006b500 0x7ff85006d9c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff86c1eb870 tx=0x7ff85c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.048+0000 7ff8745b6640 1 -- 172.21.15.1:0/137627036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff86c0c3970 msgr2=0x7ff86c0a97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.048+0000 7ff8745b6640 1 --2- 172.21.15.1:0/137627036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff86c0c3970 0x7ff86c0a97d0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7ff8680098c0 tx=0x7ff8680046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.048+0000 7ff8745b6640 1 -- 172.21.15.1:0/137627036 shutdown_connections 2024-03-06T07:26:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.048+0000 7ff8745b6640 1 --2- 172.21.15.1:0/137627036 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff85006b500 0x7ff85006d9c0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.048+0000 7ff8745b6640 1 --2- 172.21.15.1:0/137627036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff86c152840 0x7ff86c14cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.048+0000 7ff8745b6640 1 --2- 172.21.15.1:0/137627036 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff86c0c60b0 0x7ff86c1ee420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.048+0000 7ff8745b6640 1 --2- 172.21.15.1:0/137627036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff86c0c3970 0x7ff86c0a97d0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.048+0000 7ff8745b6640 1 -- 172.21.15.1:0/137627036 >> 172.21.15.1:0/137627036 conn(0x7ff86c0c88f0 msgr2=0x7ff86c1553d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.048+0000 7ff8745b6640 1 -- 172.21.15.1:0/137627036 shutdown_connections 2024-03-06T07:26:29.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.048+0000 7ff8745b6640 1 -- 172.21.15.1:0/137627036 wait complete. 2024-03-06T07:26:29.061 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:29.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.161+0000 7f53e9f8a640 1 Processor -- start 2024-03-06T07:26:29.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.161+0000 7f53e9f8a640 1 -- start start 2024-03-06T07:26:29.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.162+0000 7f53e9f8a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f53e4152650 0x7f53e4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.162+0000 7f53e9f8a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e40c2fc0 con 0x7f53e40bf9d0 2024-03-06T07:26:29.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.162+0000 7f53e9f8a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e4158da0 con 0x7f53e40c6b10 2024-03-06T07:26:29.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.162+0000 7f53e9f8a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e4155800 con 0x7f53e4152650 2024-03-06T07:26:29.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.162+0000 7f53e37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f53e4152650 0x7f53e4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.162+0000 7f53e37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f53e4152650 0x7f53e4155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58574/0 (socket says 172.21.15.1:58574) 2024-03-06T07:26:29.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.162+0000 7f53e37fe640 1 -- 172.21.15.1:0/513130979 learned_addr learned my addr 172.21.15.1:0/513130979 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:29.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.162+0000 7f53e3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f53e40bf9d0 0x7f53e40c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60332/0 (socket says 172.21.15.1:60332) 2024-03-06T07:26:29.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.162+0000 7f53e37fe640 1 -- 172.21.15.1:0/513130979 >> v1:172.21.15.98:6789/0 conn(0x7f53e40c6b10 legacy=0x7f53e4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.162+0000 7f53e37fe640 1 -- 172.21.15.1:0/513130979 >> v1:172.21.15.1:6789/0 conn(0x7f53e40bf9d0 legacy=0x7f53e40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.162+0000 7f53e27fc640 1 -- 172.21.15.1:0/513130979 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244510702 0 0) 0x7f53e40c2fc0 con 0x7f53e40bf9d0 2024-03-06T07:26:29.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.162+0000 7f53e37fe640 1 -- 172.21.15.1:0/513130979 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e40c6690 con 0x7f53e4152650 2024-03-06T07:26:29.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.163+0000 7f53e37fe640 1 --2- 172.21.15.1:0/513130979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e4152650 0x7f53e4155c90 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f53d4009870 tx=0x7f53d402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e28b933a68b2f747 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.163+0000 7f53e27fc640 1 -- 172.21.15.1:0/513130979 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d4034070 con 0x7f53e4152650 2024-03-06T07:26:29.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.163+0000 7f53e27fc640 1 -- 172.21.15.1:0/513130979 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d4030860 con 0x7f53e4152650 2024-03-06T07:26:29.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.163+0000 7f53e27fc640 1 -- 172.21.15.1:0/513130979 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d4030c30 con 0x7f53e4152650 2024-03-06T07:26:29.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.164+0000 7f53e9f8a640 1 -- 172.21.15.1:0/513130979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e4152650 msgr2=0x7f53e4155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.164+0000 7f53e9f8a640 1 --2- 172.21.15.1:0/513130979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e4152650 0x7f53e4155c90 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f53d4009870 tx=0x7f53d402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.164+0000 7f53e9f8a640 1 -- 172.21.15.1:0/513130979 shutdown_connections 2024-03-06T07:26:29.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.164+0000 7f53e9f8a640 1 --2- 172.21.15.1:0/513130979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e4152650 0x7f53e4155c90 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.164+0000 7f53e9f8a640 1 -- 172.21.15.1:0/513130979 >> 172.21.15.1:0/513130979 conn(0x7f53e4066ee0 msgr2=0x7f53e40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.164+0000 7f53e9f8a640 1 -- 172.21.15.1:0/513130979 shutdown_connections 2024-03-06T07:26:29.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.164+0000 7f53e9f8a640 1 -- 172.21.15.1:0/513130979 wait complete. 2024-03-06T07:26:29.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.165+0000 7f53e9f8a640 1 Processor -- start 2024-03-06T07:26:29.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.165+0000 7f53e9f8a640 1 -- start start 2024-03-06T07:26:29.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.166+0000 7f53e9f8a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53e40bf9d0 0x7f53e40c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.166+0000 7f53e9f8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53e4152650 0x7f53e41e1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.166+0000 7f53e9f8a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e4154c90 0x7f53e41e1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.166+0000 7f53e9f8a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e41551b0 con 0x7f53e4154c90 2024-03-06T07:26:29.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.166+0000 7f53e9f8a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e40c6fa0 con 0x7f53e4152650 2024-03-06T07:26:29.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.166+0000 7f53e9f8a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e4155490 con 0x7f53e40bf9d0 2024-03-06T07:26:29.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.166+0000 7f53e3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e4154c90 0x7f53e41e1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.166+0000 7f53e3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e4154c90 0x7f53e41e1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58586/0 (socket says 172.21.15.1:58586) 2024-03-06T07:26:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.166+0000 7f53e3fff640 1 -- 172.21.15.1:0/2586294420 learned_addr learned my addr 172.21.15.1:0/2586294420 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.166+0000 7f53e37fe640 1 --2- 172.21.15.1:0/2586294420 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53e40bf9d0 0x7f53e40c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.166+0000 7f53e2ffd640 1 --2- 172.21.15.1:0/2586294420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53e4152650 0x7f53e41e1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.166+0000 7f53e3fff640 1 -- 172.21.15.1:0/2586294420 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53e40bf9d0 msgr2=0x7f53e40c9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.166+0000 7f53e3fff640 1 --2- 172.21.15.1:0/2586294420 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53e40bf9d0 0x7f53e40c9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.166+0000 7f53e3fff640 1 -- 172.21.15.1:0/2586294420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53e4152650 msgr2=0x7f53e41e1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.166+0000 7f53e3fff640 1 --2- 172.21.15.1:0/2586294420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53e4152650 0x7f53e41e1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.166+0000 7f53e3fff640 1 -- 172.21.15.1:0/2586294420 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e4154740 con 0x7f53e4154c90 2024-03-06T07:26:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.167+0000 7f53e3fff640 1 --2- 172.21.15.1:0/2586294420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e4154c90 0x7f53e41e1fd0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f53dc00c700 tx=0x7f53dc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.167+0000 7f53cbfff640 1 -- 172.21.15.1:0/2586294420 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53dc018070 con 0x7f53e4154c90 2024-03-06T07:26:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.167+0000 7f53cbfff640 1 -- 172.21.15.1:0/2586294420 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53dc00ce10 con 0x7f53e4154c90 2024-03-06T07:26:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.167+0000 7f53e9f8a640 1 -- 172.21.15.1:0/2586294420 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e4154a20 con 0x7f53e4154c90 2024-03-06T07:26:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.167+0000 7f53cbfff640 1 -- 172.21.15.1:0/2586294420 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53dc00a860 con 0x7f53e4154c90 2024-03-06T07:26:29.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.167+0000 7f53e9f8a640 1 -- 172.21.15.1:0/2586294420 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e414b290 con 0x7f53e4154c90 2024-03-06T07:26:29.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.168+0000 7f53e9f8a640 1 -- 172.21.15.1:0/2586294420 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e4153ca0 con 0x7f53e4154c90 2024-03-06T07:26:29.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.168+0000 7f53cbfff640 1 -- 172.21.15.1:0/2586294420 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53dc01b070 con 0x7f53e4154c90 2024-03-06T07:26:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.169+0000 7f53cbfff640 1 --2- 172.21.15.1:0/2586294420 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f53d006b500 0x7f53d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.169+0000 7f53cbfff640 1 -- 172.21.15.1:0/2586294420 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f53dc089b50 con 0x7f53e4154c90 2024-03-06T07:26:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.169+0000 7f53e37fe640 1 --2- 172.21.15.1:0/2586294420 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f53d006b500 0x7f53d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.171+0000 7f53e37fe640 1 --2- 172.21.15.1:0/2586294420 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f53d006b500 0x7f53d006d9c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f53d4036000 tx=0x7f53d4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.174+0000 7f53cbfff640 1 -- 172.21.15.1:0/2586294420 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53e4153ca0 con 0x7f53e4154c90 2024-03-06T07:26:29.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.359+0000 7f53e9f8a640 1 -- 172.21.15.1:0/2586294420 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53e40b06f0 con 0x7f53e4154c90 2024-03-06T07:26:29.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.361+0000 7f53cbfff640 1 -- 172.21.15.1:0/2586294420 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+20482 (secure 0 0 0) 0x7f53dc057f30 con 0x7f53e4154c90 2024-03-06T07:26:29.362 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:29.362 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":153,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:26:27.676586+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:15.245426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:15.942343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:26:24.304587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:26:25.085741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:29.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.363+0000 7f53e9f8a640 1 -- 172.21.15.1:0/2586294420 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f53d006b500 msgr2=0x7f53d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.363+0000 7f53e9f8a640 1 --2- 172.21.15.1:0/2586294420 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f53d006b500 0x7f53d006d9c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f53d4036000 tx=0x7f53d4008030 comp rx=0 tx=0).stop 2024-03-06T07:26:29.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.363+0000 7f53e9f8a640 1 -- 172.21.15.1:0/2586294420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e4154c90 msgr2=0x7f53e41e1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.363+0000 7f53e9f8a640 1 --2- 172.21.15.1:0/2586294420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e4154c90 0x7f53e41e1fd0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f53dc00c700 tx=0x7f53dc00cbc0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.363+0000 7f53e9f8a640 1 -- 172.21.15.1:0/2586294420 shutdown_connections 2024-03-06T07:26:29.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.363+0000 7f53e9f8a640 1 --2- 172.21.15.1:0/2586294420 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f53d006b500 0x7f53d006d9c0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.363+0000 7f53e9f8a640 1 --2- 172.21.15.1:0/2586294420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e4154c90 0x7f53e41e1fd0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.363+0000 7f53e9f8a640 1 --2- 172.21.15.1:0/2586294420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53e4152650 0x7f53e41e1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.363+0000 7f53e9f8a640 1 --2- 172.21.15.1:0/2586294420 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53e40bf9d0 0x7f53e40c9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.363+0000 7f53e9f8a640 1 -- 172.21.15.1:0/2586294420 >> 172.21.15.1:0/2586294420 conn(0x7f53e4066ee0 msgr2=0x7f53e40c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.364+0000 7f53e9f8a640 1 -- 172.21.15.1:0/2586294420 shutdown_connections 2024-03-06T07:26:29.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.364+0000 7f53e9f8a640 1 -- 172.21.15.1:0/2586294420 wait complete. 2024-03-06T07:26:29.376 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:26:29.376 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:29.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.476+0000 7fabc87f6640 1 Processor -- start 2024-03-06T07:26:29.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.476+0000 7fabc87f6640 1 -- start start 2024-03-06T07:26:29.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.477+0000 7fabc87f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabc0152650 0x7fabc0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.477+0000 7fabc87f6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fabc00c6b10 0x7fabc0158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.477+0000 7fabc87f6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fabc00bf9d0 0x7fabc00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.477+0000 7fabc87f6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc0155800 con 0x7fabc00c6b10 2024-03-06T07:26:29.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.477+0000 7fabc87f6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc00c2fc0 con 0x7fabc00bf9d0 2024-03-06T07:26:29.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.477+0000 7fabc87f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc00c6690 con 0x7fabc0152650 2024-03-06T07:26:29.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.477+0000 7fabc5d91640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fabc00c6b10 0x7fabc0158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.477+0000 7fabc5d91640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fabc00c6b10 0x7fabc0158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58594/0 (socket says 172.21.15.1:58594) 2024-03-06T07:26:29.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.477+0000 7fabc5d91640 1 -- 172.21.15.1:0/2250545755 learned_addr learned my addr 172.21.15.1:0/2250545755 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:29.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.478+0000 7fabc6592640 1 --2- 172.21.15.1:0/2250545755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabc0152650 0x7fabc0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.478+0000 7fabc6d93640 1 --2- 172.21.15.1:0/2250545755 >> v2:172.21.15.98:3300/0 conn(0x7fabc00bf9d0 0x7fabc00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.478+0000 7fabc5d91640 1 -- 172.21.15.1:0/2250545755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabc0152650 msgr2=0x7fabc0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:29.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.478+0000 7fabc5d91640 1 --2- 172.21.15.1:0/2250545755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabc0152650 0x7fabc0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.478+0000 7fabc5d91640 1 -- 172.21.15.1:0/2250545755 >> v2:172.21.15.98:3300/0 conn(0x7fabc00bf9d0 msgr2=0x7fabc00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:29.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.478+0000 7fabc5d91640 1 --2- 172.21.15.1:0/2250545755 >> v2:172.21.15.98:3300/0 conn(0x7fabc00bf9d0 0x7fabc00c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.478+0000 7fabc5d91640 1 -- 172.21.15.1:0/2250545755 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc00c7280 con 0x7fabc00c6b10 2024-03-06T07:26:29.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.478+0000 7fabc5d91640 1 --2- 172.21.15.1:0/2250545755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc00c6b10 0x7fabc0158690 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fabc01597b0 tx=0x7fabb002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd1e615dd6a75a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.479+0000 7fabc5590640 1 -- 172.21.15.1:0/2250545755 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb002e070 con 0x7fabc00c6b10 2024-03-06T07:26:29.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.479+0000 7fabc5590640 1 -- 172.21.15.1:0/2250545755 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb0037070 con 0x7fabc00c6b10 2024-03-06T07:26:29.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.479+0000 7fabc5590640 1 -- 172.21.15.1:0/2250545755 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb0031a00 con 0x7fabc00c6b10 2024-03-06T07:26:29.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.479+0000 7fabc87f6640 1 -- 172.21.15.1:0/2250545755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc00c6b10 msgr2=0x7fabc0158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.479+0000 7fabc87f6640 1 --2- 172.21.15.1:0/2250545755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc00c6b10 0x7fabc0158690 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fabc01597b0 tx=0x7fabb002b880 comp rx=0 tx=0).stop 2024-03-06T07:26:29.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.479+0000 7fabc87f6640 1 -- 172.21.15.1:0/2250545755 shutdown_connections 2024-03-06T07:26:29.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.479+0000 7fabc87f6640 1 --2- 172.21.15.1:0/2250545755 >> v2:172.21.15.98:3300/0 conn(0x7fabc00bf9d0 0x7fabc00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.479+0000 7fabc87f6640 1 --2- 172.21.15.1:0/2250545755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc00c6b10 0x7fabc0158690 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.479+0000 7fabc87f6640 1 --2- 172.21.15.1:0/2250545755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabc0152650 0x7fabc0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.479+0000 7fabc87f6640 1 -- 172.21.15.1:0/2250545755 >> 172.21.15.1:0/2250545755 conn(0x7fabc0066ee0 msgr2=0x7fabc00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.479+0000 7fabc87f6640 1 -- 172.21.15.1:0/2250545755 shutdown_connections 2024-03-06T07:26:29.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.479+0000 7fabc87f6640 1 -- 172.21.15.1:0/2250545755 wait complete. 2024-03-06T07:26:29.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.481+0000 7fabc87f6640 1 Processor -- start 2024-03-06T07:26:29.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.481+0000 7fabc87f6640 1 -- start start 2024-03-06T07:26:29.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.481+0000 7fabc87f6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc00bf9d0 0x7fabc01eba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.481+0000 7fabc87f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabc0152650 0x7fabc01de970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.481+0000 7fabc87f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabc0158770 0x7fabc0158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.481+0000 7fabc87f6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc0155490 con 0x7fabc00bf9d0 2024-03-06T07:26:29.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.481+0000 7fabc87f6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc01551b0 con 0x7fabc0152650 2024-03-06T07:26:29.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.481+0000 7fabc87f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc0152a30 con 0x7fabc0158770 2024-03-06T07:26:29.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.482+0000 7fabc6592640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc00bf9d0 0x7fabc01eba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.482+0000 7fabc6592640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc00bf9d0 0x7fabc01eba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58608/0 (socket says 172.21.15.1:58608) 2024-03-06T07:26:29.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.482+0000 7fabc6592640 1 -- 172.21.15.1:0/1223047913 learned_addr learned my addr 172.21.15.1:0/1223047913 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:29.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.482+0000 7fabc6d93640 1 --2- 172.21.15.1:0/1223047913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabc0158770 0x7fabc0158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.482+0000 7fabc5d91640 1 --2- 172.21.15.1:0/1223047913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabc0152650 0x7fabc01de970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.482+0000 7fabc6592640 1 -- 172.21.15.1:0/1223047913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabc0158770 msgr2=0x7fabc0158c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.482+0000 7fabc6592640 1 --2- 172.21.15.1:0/1223047913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabc0158770 0x7fabc0158c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.482+0000 7fabc6592640 1 -- 172.21.15.1:0/1223047913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabc0152650 msgr2=0x7fabc01de970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.482+0000 7fabc6592640 1 --2- 172.21.15.1:0/1223047913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabc0152650 0x7fabc01de970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.482+0000 7fabc6592640 1 -- 172.21.15.1:0/1223047913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc0159140 con 0x7fabc00bf9d0 2024-03-06T07:26:29.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.482+0000 7fabc6592640 1 --2- 172.21.15.1:0/1223047913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc00bf9d0 0x7fabc01eba00 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00d830 tx=0x7fabbc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.482+0000 7fabb6ffd640 1 -- 172.21.15.1:0/1223047913 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc00b660 con 0x7fabc00bf9d0 2024-03-06T07:26:29.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.482+0000 7fabb6ffd640 1 -- 172.21.15.1:0/1223047913 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabbc019070 con 0x7fabc00bf9d0 2024-03-06T07:26:29.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.482+0000 7fabb6ffd640 1 -- 172.21.15.1:0/1223047913 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc00ba20 con 0x7fabc00bf9d0 2024-03-06T07:26:29.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.482+0000 7fabc87f6640 1 -- 172.21.15.1:0/1223047913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc01df580 con 0x7fabc00bf9d0 2024-03-06T07:26:29.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.482+0000 7fabc87f6640 1 -- 172.21.15.1:0/1223047913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc01dfb40 con 0x7fabc00bf9d0 2024-03-06T07:26:29.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.484+0000 7fabc87f6640 1 -- 172.21.15.1:0/1223047913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc0067fd0 con 0x7fabc00bf9d0 2024-03-06T07:26:29.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.484+0000 7fabb6ffd640 1 -- 172.21.15.1:0/1223047913 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fabbc01e460 con 0x7fabc00bf9d0 2024-03-06T07:26:29.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.485+0000 7fabb6ffd640 1 --2- 172.21.15.1:0/1223047913 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faba406b4d0 0x7faba406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.485+0000 7fabb6ffd640 1 -- 172.21.15.1:0/1223047913 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fabbc08ab50 con 0x7fabc00bf9d0 2024-03-06T07:26:29.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.485+0000 7fabc5d91640 1 --2- 172.21.15.1:0/1223047913 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faba406b4d0 0x7faba406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.486+0000 7fabc5d91640 1 --2- 172.21.15.1:0/1223047913 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faba406b4d0 0x7faba406d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fabb0036000 tx=0x7fabb003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.489+0000 7fabb6ffd640 1 -- 172.21.15.1:0/1223047913 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fabbc058f30 con 0x7fabc00bf9d0 2024-03-06T07:26:29.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.676+0000 7fabc87f6640 1 -- 172.21.15.1:0/1223047913 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fabc00b06f0 con 0x7faba406b4d0 2024-03-06T07:26:29.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.687+0000 7fabb6ffd640 1 -- 172.21.15.1:0/1223047913 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+479322 (secure 0 0 0) 0x7fabc00b06f0 con 0x7faba406b4d0 2024-03-06T07:26:29.693 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:29.698 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:26:29.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.691+0000 7fabc87f6640 1 -- 172.21.15.1:0/1223047913 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faba406b4d0 msgr2=0x7faba406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.691+0000 7fabc87f6640 1 --2- 172.21.15.1:0/1223047913 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faba406b4d0 0x7faba406d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fabb0036000 tx=0x7fabb003d040 comp rx=0 tx=0).stop 2024-03-06T07:26:29.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.692+0000 7fabc87f6640 1 -- 172.21.15.1:0/1223047913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc00bf9d0 msgr2=0x7fabc01eba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.692+0000 7fabc87f6640 1 --2- 172.21.15.1:0/1223047913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc00bf9d0 0x7fabc01eba00 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00d830 tx=0x7fabbc00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.692+0000 7fabc87f6640 1 -- 172.21.15.1:0/1223047913 shutdown_connections 2024-03-06T07:26:29.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.692+0000 7fabc87f6640 1 --2- 172.21.15.1:0/1223047913 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faba406b4d0 0x7faba406d990 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.692+0000 7fabc87f6640 1 --2- 172.21.15.1:0/1223047913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabc0158770 0x7fabc0158c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.692+0000 7fabc87f6640 1 --2- 172.21.15.1:0/1223047913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabc0152650 0x7fabc01de970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.692+0000 7fabc87f6640 1 --2- 172.21.15.1:0/1223047913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc00bf9d0 0x7fabc01eba00 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.692+0000 7fabc87f6640 1 -- 172.21.15.1:0/1223047913 >> 172.21.15.1:0/1223047913 conn(0x7fabc0066ee0 msgr2=0x7fabc00672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.692+0000 7fabc87f6640 1 -- 172.21.15.1:0/1223047913 shutdown_connections 2024-03-06T07:26:29.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.692+0000 7fabc87f6640 1 -- 172.21.15.1:0/1223047913 wait complete. 2024-03-06T07:26:29.704 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-03-06T07:26:28.774882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264320,"kb_used_data":14848,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710400,"statfs":{"total":773094113280,"available":772823449600,"internally_reserved":0,"allocated":15204352,"data_stored":8433300,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.115308"},"pg_stats":[{"pgid":"26.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3f","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663770+0000","last_change":"2024-03-06T07:26:17.632610+0000","last_active":"2024-03-06T07:26:24.663770+0000","last_peered":"2024-03-06T07:26:24.663770+0000","last_clean":"2024-03-06T07:26:24.663770+0000","last_became_active":"2024-03-06T07:26:17.630763+0000","last_became_peered":"2024-03-06T07:26:17.630763+0000","last_unstale":"2024-03-06T07:26:24.663770+0000","last_undegraded":"2024-03-06T07:26:24.663770+0000","last_fullsized":"2024-03-06T07:26:24.663770+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:35.035202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664635+0000","last_change":"2024-03-06T07:26:17.632425+0000","last_active":"2024-03-06T07:26:24.664635+0000","last_peered":"2024-03-06T07:26:24.664635+0000","last_clean":"2024-03-06T07:26:24.664635+0000","last_became_active":"2024-03-06T07:26:17.632153+0000","last_became_peered":"2024-03-06T07:26:17.632153+0000","last_unstale":"2024-03-06T07:26:24.664635+0000","last_undegraded":"2024-03-06T07:26:24.664635+0000","last_fullsized":"2024-03-06T07:26:24.664635+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:01.829378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663942+0000","last_change":"2024-03-06T07:26:17.630353+0000","last_active":"2024-03-06T07:26:24.663942+0000","last_peered":"2024-03-06T07:26:24.663942+0000","last_clean":"2024-03-06T07:26:24.663942+0000","last_became_active":"2024-03-06T07:26:17.630080+0000","last_became_peered":"2024-03-06T07:26:17.630080+0000","last_unstale":"2024-03-06T07:26:24.663942+0000","last_undegraded":"2024-03-06T07:26:24.663942+0000","last_fullsized":"2024-03-06T07:26:24.663942+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:49.713717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663552+0000","last_change":"2024-03-06T07:26:17.630434+0000","last_active":"2024-03-06T07:26:24.663552+0000","last_peered":"2024-03-06T07:26:24.663552+0000","last_clean":"2024-03-06T07:26:24.663552+0000","last_became_active":"2024-03-06T07:26:17.630139+0000","last_became_peered":"2024-03-06T07:26:17.630139+0000","last_unstale":"2024-03-06T07:26:24.663552+0000","last_undegraded":"2024-03-06T07:26:24.663552+0000","last_fullsized":"2024-03-06T07:26:24.663552+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:09.159934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668267+0000","last_change":"2024-03-06T07:26:17.632665+0000","last_active":"2024-03-06T07:26:24.668267+0000","last_peered":"2024-03-06T07:26:24.668267+0000","last_clean":"2024-03-06T07:26:24.668267+0000","last_became_active":"2024-03-06T07:26:17.632377+0000","last_became_peered":"2024-03-06T07:26:17.632377+0000","last_unstale":"2024-03-06T07:26:24.668267+0000","last_undegraded":"2024-03-06T07:26:24.668267+0000","last_fullsized":"2024-03-06T07:26:24.668267+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:23.225363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665614+0000","last_change":"2024-03-06T07:26:17.633911+0000","last_active":"2024-03-06T07:26:24.665614+0000","last_peered":"2024-03-06T07:26:24.665614+0000","last_clean":"2024-03-06T07:26:24.665614+0000","last_became_active":"2024-03-06T07:26:17.633638+0000","last_became_peered":"2024-03-06T07:26:17.633638+0000","last_unstale":"2024-03-06T07:26:24.665614+0000","last_undegraded":"2024-03-06T07:26:24.665614+0000","last_fullsized":"2024-03-06T07:26:24.665614+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:01.350963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664718+0000","last_change":"2024-03-06T07:26:17.629894+0000","last_active":"2024-03-06T07:26:24.664718+0000","last_peered":"2024-03-06T07:26:24.664718+0000","last_clean":"2024-03-06T07:26:24.664718+0000","last_became_active":"2024-03-06T07:26:17.629625+0000","last_became_peered":"2024-03-06T07:26:17.629625+0000","last_unstale":"2024-03-06T07:26:24.664718+0000","last_undegraded":"2024-03-06T07:26:24.664718+0000","last_fullsized":"2024-03-06T07:26:24.664718+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:46.687284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664295+0000","last_change":"2024-03-06T07:26:17.632711+0000","last_active":"2024-03-06T07:26:24.664295+0000","last_peered":"2024-03-06T07:26:24.664295+0000","last_clean":"2024-03-06T07:26:24.664295+0000","last_became_active":"2024-03-06T07:26:17.632173+0000","last_became_peered":"2024-03-06T07:26:17.632173+0000","last_unstale":"2024-03-06T07:26:24.664295+0000","last_undegraded":"2024-03-06T07:26:24.664295+0000","last_fullsized":"2024-03-06T07:26:24.664295+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:25.595909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665506+0000","last_change":"2024-03-06T07:26:17.638531+0000","last_active":"2024-03-06T07:26:24.665506+0000","last_peered":"2024-03-06T07:26:24.665506+0000","last_clean":"2024-03-06T07:26:24.665506+0000","last_became_active":"2024-03-06T07:26:17.638254+0000","last_became_peered":"2024-03-06T07:26:17.638254+0000","last_unstale":"2024-03-06T07:26:24.665506+0000","last_undegraded":"2024-03-06T07:26:24.665506+0000","last_fullsized":"2024-03-06T07:26:24.665506+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:01.350963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665485+0000","last_change":"2024-03-06T07:26:17.632455+0000","last_active":"2024-03-06T07:26:24.665485+0000","last_peered":"2024-03-06T07:26:24.665485+0000","last_clean":"2024-03-06T07:26:24.665485+0000","last_became_active":"2024-03-06T07:26:17.632195+0000","last_became_peered":"2024-03-06T07:26:17.632195+0000","last_unstale":"2024-03-06T07:26:24.665485+0000","last_undegraded":"2024-03-06T07:26:24.665485+0000","last_fullsized":"2024-03-06T07:26:24.665485+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:14.404340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667172+0000","last_change":"2024-03-06T07:26:17.633984+0000","last_active":"2024-03-06T07:26:24.667172+0000","last_peered":"2024-03-06T07:26:24.667172+0000","last_clean":"2024-03-06T07:26:24.667172+0000","last_became_active":"2024-03-06T07:26:17.633666+0000","last_became_peered":"2024-03-06T07:26:17.633666+0000","last_unstale":"2024-03-06T07:26:24.667172+0000","last_undegraded":"2024-03-06T07:26:24.667172+0000","last_fullsized":"2024-03-06T07:26:24.667172+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:43.884264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664561+0000","last_change":"2024-03-06T07:26:17.634798+0000","last_active":"2024-03-06T07:26:24.664561+0000","last_peered":"2024-03-06T07:26:24.664561+0000","last_clean":"2024-03-06T07:26:24.664561+0000","last_became_active":"2024-03-06T07:26:17.634537+0000","last_became_peered":"2024-03-06T07:26:17.634537+0000","last_unstale":"2024-03-06T07:26:24.664561+0000","last_undegraded":"2024-03-06T07:26:24.664561+0000","last_fullsized":"2024-03-06T07:26:24.664561+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:25.595909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662407+0000","last_change":"2024-03-06T07:26:17.640814+0000","last_active":"2024-03-06T07:26:24.662407+0000","last_peered":"2024-03-06T07:26:24.662407+0000","last_clean":"2024-03-06T07:26:24.662407+0000","last_became_active":"2024-03-06T07:26:17.640444+0000","last_became_peered":"2024-03-06T07:26:17.640444+0000","last_unstale":"2024-03-06T07:26:24.662407+0000","last_undegraded":"2024-03-06T07:26:24.662407+0000","last_fullsized":"2024-03-06T07:26:24.662407+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:35.920913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667754+0000","last_change":"2024-03-06T07:26:17.631892+0000","last_active":"2024-03-06T07:26:24.667754+0000","last_peered":"2024-03-06T07:26:24.667754+0000","last_clean":"2024-03-06T07:26:24.667754+0000","last_became_active":"2024-03-06T07:26:17.631498+0000","last_became_peered":"2024-03-06T07:26:17.631498+0000","last_unstale":"2024-03-06T07:26:24.667754+0000","last_undegraded":"2024-03-06T07:26:24.667754+0000","last_fullsized":"2024-03-06T07:26:24.667754+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:09.253874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666379+0000","last_change":"2024-03-06T07:26:17.634220+0000","last_active":"2024-03-06T07:26:24.666379+0000","last_peered":"2024-03-06T07:26:24.666379+0000","last_clean":"2024-03-06T07:26:24.666379+0000","last_became_active":"2024-03-06T07:26:17.633922+0000","last_became_peered":"2024-03-06T07:26:17.633922+0000","last_unstale":"2024-03-06T07:26:24.666379+0000","last_undegraded":"2024-03-06T07:26:24.666379+0000","last_fullsized":"2024-03-06T07:26:24.666379+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:51.900557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662529+0000","last_change":"2024-03-06T07:26:17.635856+0000","last_active":"2024-03-06T07:26:24.662529+0000","last_peered":"2024-03-06T07:26:24.662529+0000","last_clean":"2024-03-06T07:26:24.662529+0000","last_became_active":"2024-03-06T07:26:17.631353+0000","last_became_peered":"2024-03-06T07:26:17.631353+0000","last_unstale":"2024-03-06T07:26:24.662529+0000","last_undegraded":"2024-03-06T07:26:24.662529+0000","last_fullsized":"2024-03-06T07:26:24.662529+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:15.652152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665594+0000","last_change":"2024-03-06T07:26:17.628960+0000","last_active":"2024-03-06T07:26:24.665594+0000","last_peered":"2024-03-06T07:26:24.665594+0000","last_clean":"2024-03-06T07:26:24.665594+0000","last_became_active":"2024-03-06T07:26:17.628681+0000","last_became_peered":"2024-03-06T07:26:17.628681+0000","last_unstale":"2024-03-06T07:26:24.665594+0000","last_undegraded":"2024-03-06T07:26:24.665594+0000","last_fullsized":"2024-03-06T07:26:24.665594+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:40.950597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666241+0000","last_change":"2024-03-06T07:26:17.629582+0000","last_active":"2024-03-06T07:26:24.666241+0000","last_peered":"2024-03-06T07:26:24.666241+0000","last_clean":"2024-03-06T07:26:24.666241+0000","last_became_active":"2024-03-06T07:26:17.629246+0000","last_became_peered":"2024-03-06T07:26:17.629246+0000","last_unstale":"2024-03-06T07:26:24.666241+0000","last_undegraded":"2024-03-06T07:26:24.666241+0000","last_fullsized":"2024-03-06T07:26:24.666241+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:56.556155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667465+0000","last_change":"2024-03-06T07:26:17.632386+0000","last_active":"2024-03-06T07:26:24.667465+0000","last_peered":"2024-03-06T07:26:24.667465+0000","last_clean":"2024-03-06T07:26:24.667465+0000","last_became_active":"2024-03-06T07:26:17.631867+0000","last_became_peered":"2024-03-06T07:26:17.631867+0000","last_unstale":"2024-03-06T07:26:24.667465+0000","last_undegraded":"2024-03-06T07:26:24.667465+0000","last_fullsized":"2024-03-06T07:26:24.667465+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:46.687284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658784+0000","last_change":"2024-03-06T07:26:17.622563+0000","last_active":"2024-03-06T07:26:24.658784+0000","last_peered":"2024-03-06T07:26:24.658784+0000","last_clean":"2024-03-06T07:26:24.658784+0000","last_became_active":"2024-03-06T07:26:17.622293+0000","last_became_peered":"2024-03-06T07:26:17.622293+0000","last_unstale":"2024-03-06T07:26:24.658784+0000","last_undegraded":"2024-03-06T07:26:24.658784+0000","last_fullsized":"2024-03-06T07:26:24.658784+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:34.462291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662501+0000","last_change":"2024-03-06T07:26:17.637385+0000","last_active":"2024-03-06T07:26:24.662501+0000","last_peered":"2024-03-06T07:26:24.662501+0000","last_clean":"2024-03-06T07:26:24.662501+0000","last_became_active":"2024-03-06T07:26:17.637063+0000","last_became_peered":"2024-03-06T07:26:17.637063+0000","last_unstale":"2024-03-06T07:26:24.662501+0000","last_undegraded":"2024-03-06T07:26:24.662501+0000","last_fullsized":"2024-03-06T07:26:24.662501+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:49.333296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665712+0000","last_change":"2024-03-06T07:26:17.632278+0000","last_active":"2024-03-06T07:26:24.665712+0000","last_peered":"2024-03-06T07:26:24.665712+0000","last_clean":"2024-03-06T07:26:24.665712+0000","last_became_active":"2024-03-06T07:26:17.631791+0000","last_became_peered":"2024-03-06T07:26:17.631791+0000","last_unstale":"2024-03-06T07:26:24.665712+0000","last_undegraded":"2024-03-06T07:26:24.665712+0000","last_fullsized":"2024-03-06T07:26:24.665712+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:40.950597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667556+0000","last_change":"2024-03-06T07:26:17.627902+0000","last_active":"2024-03-06T07:26:24.667556+0000","last_peered":"2024-03-06T07:26:24.667556+0000","last_clean":"2024-03-06T07:26:24.667556+0000","last_became_active":"2024-03-06T07:26:17.627643+0000","last_became_peered":"2024-03-06T07:26:17.627643+0000","last_unstale":"2024-03-06T07:26:24.667556+0000","last_undegraded":"2024-03-06T07:26:24.667556+0000","last_fullsized":"2024-03-06T07:26:24.667556+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:30.600647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662946+0000","last_change":"2024-03-06T07:26:17.631421+0000","last_active":"2024-03-06T07:26:24.662946+0000","last_peered":"2024-03-06T07:26:24.662946+0000","last_clean":"2024-03-06T07:26:24.662946+0000","last_became_active":"2024-03-06T07:26:17.631022+0000","last_became_peered":"2024-03-06T07:26:17.631022+0000","last_unstale":"2024-03-06T07:26:24.662946+0000","last_undegraded":"2024-03-06T07:26:24.662946+0000","last_fullsized":"2024-03-06T07:26:24.662946+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:40.032388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662940+0000","last_change":"2024-03-06T07:26:17.637044+0000","last_active":"2024-03-06T07:26:24.662940+0000","last_peered":"2024-03-06T07:26:24.662940+0000","last_clean":"2024-03-06T07:26:24.662940+0000","last_became_active":"2024-03-06T07:26:17.636771+0000","last_became_peered":"2024-03-06T07:26:17.636771+0000","last_unstale":"2024-03-06T07:26:24.662940+0000","last_undegraded":"2024-03-06T07:26:24.662940+0000","last_fullsized":"2024-03-06T07:26:24.662940+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:29.354563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658522+0000","last_change":"2024-03-06T07:26:17.636369+0000","last_active":"2024-03-06T07:26:24.658522+0000","last_peered":"2024-03-06T07:26:24.658522+0000","last_clean":"2024-03-06T07:26:24.658522+0000","last_became_active":"2024-03-06T07:26:17.636090+0000","last_became_peered":"2024-03-06T07:26:17.636090+0000","last_unstale":"2024-03-06T07:26:24.658522+0000","last_undegraded":"2024-03-06T07:26:24.658522+0000","last_fullsized":"2024-03-06T07:26:24.658522+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:45.110614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664860+0000","last_change":"2024-03-06T07:26:17.633006+0000","last_active":"2024-03-06T07:26:24.664860+0000","last_peered":"2024-03-06T07:26:24.664860+0000","last_clean":"2024-03-06T07:26:24.664860+0000","last_became_active":"2024-03-06T07:26:17.632523+0000","last_became_peered":"2024-03-06T07:26:17.632523+0000","last_unstale":"2024-03-06T07:26:24.664860+0000","last_undegraded":"2024-03-06T07:26:24.664860+0000","last_fullsized":"2024-03-06T07:26:24.664860+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:41.655929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662348+0000","last_change":"2024-03-06T07:26:17.638448+0000","last_active":"2024-03-06T07:26:24.662348+0000","last_peered":"2024-03-06T07:26:24.662348+0000","last_clean":"2024-03-06T07:26:24.662348+0000","last_became_active":"2024-03-06T07:26:17.638188+0000","last_became_peered":"2024-03-06T07:26:17.638188+0000","last_unstale":"2024-03-06T07:26:24.662348+0000","last_undegraded":"2024-03-06T07:26:24.662348+0000","last_fullsized":"2024-03-06T07:26:24.662348+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:15.455961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666402+0000","last_change":"2024-03-06T07:26:17.634053+0000","last_active":"2024-03-06T07:26:24.666402+0000","last_peered":"2024-03-06T07:26:24.666402+0000","last_clean":"2024-03-06T07:26:24.666402+0000","last_became_active":"2024-03-06T07:26:17.633451+0000","last_became_peered":"2024-03-06T07:26:17.633451+0000","last_unstale":"2024-03-06T07:26:24.666402+0000","last_undegraded":"2024-03-06T07:26:24.666402+0000","last_fullsized":"2024-03-06T07:26:24.666402+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:25.173456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665705+0000","last_change":"2024-03-06T07:26:17.627891+0000","last_active":"2024-03-06T07:26:24.665705+0000","last_peered":"2024-03-06T07:26:24.665705+0000","last_clean":"2024-03-06T07:26:24.665705+0000","last_became_active":"2024-03-06T07:26:17.627624+0000","last_became_peered":"2024-03-06T07:26:17.627624+0000","last_unstale":"2024-03-06T07:26:24.665705+0000","last_undegraded":"2024-03-06T07:26:24.665705+0000","last_fullsized":"2024-03-06T07:26:24.665705+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:56.476412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666557+0000","last_change":"2024-03-06T07:26:17.635467+0000","last_active":"2024-03-06T07:26:24.666557+0000","last_peered":"2024-03-06T07:26:24.666557+0000","last_clean":"2024-03-06T07:26:24.666557+0000","last_became_active":"2024-03-06T07:26:17.635125+0000","last_became_peered":"2024-03-06T07:26:17.635125+0000","last_unstale":"2024-03-06T07:26:24.666557+0000","last_undegraded":"2024-03-06T07:26:24.666557+0000","last_fullsized":"2024-03-06T07:26:24.666557+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:39.829929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665165+0000","last_change":"2024-03-06T07:26:17.638944+0000","last_active":"2024-03-06T07:26:24.665165+0000","last_peered":"2024-03-06T07:26:24.665165+0000","last_clean":"2024-03-06T07:26:24.665165+0000","last_became_active":"2024-03-06T07:26:17.638649+0000","last_became_peered":"2024-03-06T07:26:17.638649+0000","last_unstale":"2024-03-06T07:26:24.665165+0000","last_undegraded":"2024-03-06T07:26:24.665165+0000","last_fullsized":"2024-03-06T07:26:24.665165+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:17.394090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665307+0000","last_change":"2024-03-06T07:26:24.665307+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665307+0000","last_undegraded":"2024-03-06T07:26:24.665307+0000","last_fullsized":"2024-03-06T07:26:24.665307+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664442+0000","last_change":"2024-03-06T07:26:17.635968+0000","last_active":"2024-03-06T07:26:24.664442+0000","last_peered":"2024-03-06T07:26:24.664442+0000","last_clean":"2024-03-06T07:26:24.664442+0000","last_became_active":"2024-03-06T07:26:17.635095+0000","last_became_peered":"2024-03-06T07:26:17.635095+0000","last_unstale":"2024-03-06T07:26:24.664442+0000","last_undegraded":"2024-03-06T07:26:24.664442+0000","last_fullsized":"2024-03-06T07:26:24.664442+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:26.879181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"148'4","reported_seq":20,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663882+0000","last_change":"2024-03-06T07:26:16.626240+0000","last_active":"2024-03-06T07:26:24.663882+0000","last_peered":"2024-03-06T07:26:24.663882+0000","last_clean":"2024-03-06T07:26:24.663882+0000","last_became_active":"2024-03-06T07:26:16.626011+0000","last_became_peered":"2024-03-06T07:26:16.626011+0000","last_unstale":"2024-03-06T07:26:24.663882+0000","last_undegraded":"2024-03-06T07:26:24.663882+0000","last_fullsized":"2024-03-06T07:26:24.663882+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:30.259380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665984+0000","last_change":"2024-03-06T07:26:24.665984+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665984+0000","last_undegraded":"2024-03-06T07:26:24.665984+0000","last_fullsized":"2024-03-06T07:26:24.665984+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662038+0000","last_change":"2024-03-06T07:26:17.627512+0000","last_active":"2024-03-06T07:26:24.662038+0000","last_peered":"2024-03-06T07:26:24.662038+0000","last_clean":"2024-03-06T07:26:24.662038+0000","last_became_active":"2024-03-06T07:26:17.627235+0000","last_became_peered":"2024-03-06T07:26:17.627235+0000","last_unstale":"2024-03-06T07:26:24.662038+0000","last_undegraded":"2024-03-06T07:26:24.662038+0000","last_fullsized":"2024-03-06T07:26:24.662038+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:19.245848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"148'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668199+0000","last_change":"2024-03-06T07:26:16.625703+0000","last_active":"2024-03-06T07:26:24.668199+0000","last_peered":"2024-03-06T07:26:24.668199+0000","last_clean":"2024-03-06T07:26:24.668199+0000","last_became_active":"2024-03-06T07:26:16.625426+0000","last_became_peered":"2024-03-06T07:26:16.625426+0000","last_unstale":"2024-03-06T07:26:24.668199+0000","last_undegraded":"2024-03-06T07:26:24.668199+0000","last_fullsized":"2024-03-06T07:26:24.668199+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:39.946462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.662178+0000","last_change":"2024-03-06T07:26:24.662178+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.662178+0000","last_undegraded":"2024-03-06T07:26:24.662178+0000","last_fullsized":"2024-03-06T07:26:24.662178+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667846+0000","last_change":"2024-03-06T07:26:17.632107+0000","last_active":"2024-03-06T07:26:24.667846+0000","last_peered":"2024-03-06T07:26:24.667846+0000","last_clean":"2024-03-06T07:26:24.667846+0000","last_became_active":"2024-03-06T07:26:17.631481+0000","last_became_peered":"2024-03-06T07:26:17.631481+0000","last_unstale":"2024-03-06T07:26:24.667846+0000","last_undegraded":"2024-03-06T07:26:24.667846+0000","last_fullsized":"2024-03-06T07:26:24.667846+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:20.662554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667577+0000","last_change":"2024-03-06T07:26:16.620199+0000","last_active":"2024-03-06T07:26:24.667577+0000","last_peered":"2024-03-06T07:26:24.667577+0000","last_clean":"2024-03-06T07:26:24.667577+0000","last_became_active":"2024-03-06T07:26:16.619877+0000","last_became_peered":"2024-03-06T07:26:16.619877+0000","last_unstale":"2024-03-06T07:26:24.667577+0000","last_undegraded":"2024-03-06T07:26:24.667577+0000","last_fullsized":"2024-03-06T07:26:24.667577+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:14.648017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.663658+0000","last_change":"2024-03-06T07:26:24.663658+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.663658+0000","last_undegraded":"2024-03-06T07:26:24.663658+0000","last_fullsized":"2024-03-06T07:26:24.663658+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666492+0000","last_change":"2024-03-06T07:26:17.634295+0000","last_active":"2024-03-06T07:26:24.666492+0000","last_peered":"2024-03-06T07:26:24.666492+0000","last_clean":"2024-03-06T07:26:24.666492+0000","last_became_active":"2024-03-06T07:26:17.633744+0000","last_became_peered":"2024-03-06T07:26:17.633744+0000","last_unstale":"2024-03-06T07:26:24.666492+0000","last_undegraded":"2024-03-06T07:26:24.666492+0000","last_fullsized":"2024-03-06T07:26:24.666492+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:14.404340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658789+0000","last_change":"2024-03-06T07:26:16.616715+0000","last_active":"2024-03-06T07:26:24.658789+0000","last_peered":"2024-03-06T07:26:24.658789+0000","last_clean":"2024-03-06T07:26:24.658789+0000","last_became_active":"2024-03-06T07:26:16.616466+0000","last_became_peered":"2024-03-06T07:26:16.616466+0000","last_unstale":"2024-03-06T07:26:24.658789+0000","last_undegraded":"2024-03-06T07:26:24.658789+0000","last_fullsized":"2024-03-06T07:26:24.658789+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:54.279225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665856+0000","last_change":"2024-03-06T07:26:24.665856+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665856+0000","last_undegraded":"2024-03-06T07:26:24.665856+0000","last_fullsized":"2024-03-06T07:26:24.665856+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665092+0000","last_change":"2024-03-06T07:26:17.631966+0000","last_active":"2024-03-06T07:26:24.665092+0000","last_peered":"2024-03-06T07:26:24.665092+0000","last_clean":"2024-03-06T07:26:24.665092+0000","last_became_active":"2024-03-06T07:26:17.631673+0000","last_became_peered":"2024-03-06T07:26:17.631673+0000","last_unstale":"2024-03-06T07:26:24.665092+0000","last_undegraded":"2024-03-06T07:26:24.665092+0000","last_fullsized":"2024-03-06T07:26:24.665092+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:26.879181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"148'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664861+0000","last_change":"2024-03-06T07:26:16.619602+0000","last_active":"2024-03-06T07:26:24.664861+0000","last_peered":"2024-03-06T07:26:24.664861+0000","last_clean":"2024-03-06T07:26:24.664861+0000","last_became_active":"2024-03-06T07:26:16.619282+0000","last_became_peered":"2024-03-06T07:26:16.619282+0000","last_unstale":"2024-03-06T07:26:24.664861+0000","last_undegraded":"2024-03-06T07:26:24.664861+0000","last_fullsized":"2024-03-06T07:26:24.664861+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:59.587645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.662223+0000","last_change":"2024-03-06T07:26:24.662223+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.662223+0000","last_undegraded":"2024-03-06T07:26:24.662223+0000","last_fullsized":"2024-03-06T07:26:24.662223+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658427+0000","last_change":"2024-03-06T07:26:17.628669+0000","last_active":"2024-03-06T07:26:24.658427+0000","last_peered":"2024-03-06T07:26:24.658427+0000","last_clean":"2024-03-06T07:26:24.658427+0000","last_became_active":"2024-03-06T07:26:17.628371+0000","last_became_peered":"2024-03-06T07:26:17.628371+0000","last_unstale":"2024-03-06T07:26:24.658427+0000","last_undegraded":"2024-03-06T07:26:24.658427+0000","last_fullsized":"2024-03-06T07:26:24.658427+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:16.499694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"148'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668006+0000","last_change":"2024-03-06T07:26:16.615912+0000","last_active":"2024-03-06T07:26:24.668006+0000","last_peered":"2024-03-06T07:26:24.668006+0000","last_clean":"2024-03-06T07:26:24.668006+0000","last_became_active":"2024-03-06T07:26:16.615625+0000","last_became_peered":"2024-03-06T07:26:16.615625+0000","last_unstale":"2024-03-06T07:26:24.668006+0000","last_undegraded":"2024-03-06T07:26:24.668006+0000","last_fullsized":"2024-03-06T07:26:24.668006+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:24.591774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661284+0000","last_change":"2024-03-06T07:26:24.661284+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661284+0000","last_undegraded":"2024-03-06T07:26:24.661284+0000","last_fullsized":"2024-03-06T07:26:24.661284+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658604+0000","last_change":"2024-03-06T07:26:17.626866+0000","last_active":"2024-03-06T07:26:24.658604+0000","last_peered":"2024-03-06T07:26:24.658604+0000","last_clean":"2024-03-06T07:26:24.658604+0000","last_became_active":"2024-03-06T07:26:17.626586+0000","last_became_peered":"2024-03-06T07:26:17.626586+0000","last_unstale":"2024-03-06T07:26:24.658604+0000","last_undegraded":"2024-03-06T07:26:24.658604+0000","last_fullsized":"2024-03-06T07:26:24.658604+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:34.182149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658031+0000","last_change":"2024-03-06T07:26:16.622797+0000","last_active":"2024-03-06T07:26:24.658031+0000","last_peered":"2024-03-06T07:26:24.658031+0000","last_clean":"2024-03-06T07:26:24.658031+0000","last_became_active":"2024-03-06T07:26:16.622534+0000","last_became_peered":"2024-03-06T07:26:16.622534+0000","last_unstale":"2024-03-06T07:26:24.658031+0000","last_undegraded":"2024-03-06T07:26:24.658031+0000","last_fullsized":"2024-03-06T07:26:24.658031+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:39.697277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.663648+0000","last_change":"2024-03-06T07:26:24.663648+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.663648+0000","last_undegraded":"2024-03-06T07:26:24.663648+0000","last_fullsized":"2024-03-06T07:26:24.663648+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663807+0000","last_change":"2024-03-06T07:26:17.632829+0000","last_active":"2024-03-06T07:26:24.663807+0000","last_peered":"2024-03-06T07:26:24.663807+0000","last_clean":"2024-03-06T07:26:24.663807+0000","last_became_active":"2024-03-06T07:26:17.631051+0000","last_became_peered":"2024-03-06T07:26:17.631051+0000","last_unstale":"2024-03-06T07:26:24.663807+0000","last_undegraded":"2024-03-06T07:26:24.663807+0000","last_fullsized":"2024-03-06T07:26:24.663807+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:09.224572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664647+0000","last_change":"2024-03-06T07:26:16.618575+0000","last_active":"2024-03-06T07:26:24.664647+0000","last_peered":"2024-03-06T07:26:24.664647+0000","last_clean":"2024-03-06T07:26:24.664647+0000","last_became_active":"2024-03-06T07:26:16.618143+0000","last_became_peered":"2024-03-06T07:26:16.618143+0000","last_unstale":"2024-03-06T07:26:24.664647+0000","last_undegraded":"2024-03-06T07:26:24.664647+0000","last_fullsized":"2024-03-06T07:26:24.664647+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:48.709582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661586+0000","last_change":"2024-03-06T07:26:24.661586+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661586+0000","last_undegraded":"2024-03-06T07:26:24.661586+0000","last_fullsized":"2024-03-06T07:26:24.661586+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664401+0000","last_change":"2024-03-06T07:26:17.636599+0000","last_active":"2024-03-06T07:26:24.664401+0000","last_peered":"2024-03-06T07:26:24.664401+0000","last_clean":"2024-03-06T07:26:24.664401+0000","last_became_active":"2024-03-06T07:26:17.635451+0000","last_became_peered":"2024-03-06T07:26:17.635451+0000","last_unstale":"2024-03-06T07:26:24.664401+0000","last_undegraded":"2024-03-06T07:26:24.664401+0000","last_fullsized":"2024-03-06T07:26:24.664401+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:27.398343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662893+0000","last_change":"2024-03-06T07:26:16.627460+0000","last_active":"2024-03-06T07:26:24.662893+0000","last_peered":"2024-03-06T07:26:24.662893+0000","last_clean":"2024-03-06T07:26:24.662893+0000","last_became_active":"2024-03-06T07:26:16.626345+0000","last_became_peered":"2024-03-06T07:26:16.626345+0000","last_unstale":"2024-03-06T07:26:24.662893+0000","last_undegraded":"2024-03-06T07:26:24.662893+0000","last_fullsized":"2024-03-06T07:26:24.662893+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:24.257795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661596+0000","last_change":"2024-03-06T07:26:24.661596+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661596+0000","last_undegraded":"2024-03-06T07:26:24.661596+0000","last_fullsized":"2024-03-06T07:26:24.661596+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662215+0000","last_change":"2024-03-06T07:26:17.634986+0000","last_active":"2024-03-06T07:26:24.662215+0000","last_peered":"2024-03-06T07:26:24.662215+0000","last_clean":"2024-03-06T07:26:24.662215+0000","last_became_active":"2024-03-06T07:26:17.634680+0000","last_became_peered":"2024-03-06T07:26:17.634680+0000","last_unstale":"2024-03-06T07:26:24.662215+0000","last_undegraded":"2024-03-06T07:26:24.662215+0000","last_fullsized":"2024-03-06T07:26:24.662215+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:48.372843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"148'3","reported_seq":19,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664051+0000","last_change":"2024-03-06T07:26:16.623947+0000","last_active":"2024-03-06T07:26:24.664051+0000","last_peered":"2024-03-06T07:26:24.664051+0000","last_clean":"2024-03-06T07:26:24.664051+0000","last_became_active":"2024-03-06T07:26:16.623662+0000","last_became_peered":"2024-03-06T07:26:16.623662+0000","last_unstale":"2024-03-06T07:26:24.664051+0000","last_undegraded":"2024-03-06T07:26:24.664051+0000","last_fullsized":"2024-03-06T07:26:24.664051+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:00.411414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.663853+0000","last_change":"2024-03-06T07:26:24.663853+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.663853+0000","last_undegraded":"2024-03-06T07:26:24.663853+0000","last_fullsized":"2024-03-06T07:26:24.663853+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658162+0000","last_change":"2024-03-06T07:26:17.626440+0000","last_active":"2024-03-06T07:26:24.658162+0000","last_peered":"2024-03-06T07:26:24.658162+0000","last_clean":"2024-03-06T07:26:24.658162+0000","last_became_active":"2024-03-06T07:26:17.626149+0000","last_became_peered":"2024-03-06T07:26:17.626149+0000","last_unstale":"2024-03-06T07:26:24.658162+0000","last_undegraded":"2024-03-06T07:26:24.658162+0000","last_fullsized":"2024-03-06T07:26:24.658162+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:46.191755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"148'6","reported_seq":22,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662564+0000","last_change":"2024-03-06T07:26:16.627536+0000","last_active":"2024-03-06T07:26:24.662564+0000","last_peered":"2024-03-06T07:26:24.662564+0000","last_clean":"2024-03-06T07:26:24.662564+0000","last_became_active":"2024-03-06T07:26:16.627229+0000","last_became_peered":"2024-03-06T07:26:16.627229+0000","last_unstale":"2024-03-06T07:26:24.662564+0000","last_undegraded":"2024-03-06T07:26:24.662564+0000","last_fullsized":"2024-03-06T07:26:24.662564+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:00.825243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665684+0000","last_change":"2024-03-06T07:26:24.665684+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665684+0000","last_undegraded":"2024-03-06T07:26:24.665684+0000","last_fullsized":"2024-03-06T07:26:24.665684+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665948+0000","last_change":"2024-03-06T07:26:17.638945+0000","last_active":"2024-03-06T07:26:24.665948+0000","last_peered":"2024-03-06T07:26:24.665948+0000","last_clean":"2024-03-06T07:26:24.665948+0000","last_became_active":"2024-03-06T07:26:17.638664+0000","last_became_peered":"2024-03-06T07:26:17.638664+0000","last_unstale":"2024-03-06T07:26:24.665948+0000","last_undegraded":"2024-03-06T07:26:24.665948+0000","last_fullsized":"2024-03-06T07:26:24.665948+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:15.414267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668359+0000","last_change":"2024-03-06T07:26:16.625884+0000","last_active":"2024-03-06T07:26:24.668359+0000","last_peered":"2024-03-06T07:26:24.668359+0000","last_clean":"2024-03-06T07:26:24.668359+0000","last_became_active":"2024-03-06T07:26:16.625605+0000","last_became_peered":"2024-03-06T07:26:16.625605+0000","last_unstale":"2024-03-06T07:26:24.668359+0000","last_undegraded":"2024-03-06T07:26:24.668359+0000","last_fullsized":"2024-03-06T07:26:24.668359+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:13.400206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661600+0000","last_change":"2024-03-06T07:26:24.661600+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661600+0000","last_undegraded":"2024-03-06T07:26:24.661600+0000","last_fullsized":"2024-03-06T07:26:24.661600+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664941+0000","last_change":"2024-03-06T07:26:17.631376+0000","last_active":"2024-03-06T07:26:24.664941+0000","last_peered":"2024-03-06T07:26:24.664941+0000","last_clean":"2024-03-06T07:26:24.664941+0000","last_became_active":"2024-03-06T07:26:17.631068+0000","last_became_peered":"2024-03-06T07:26:17.631068+0000","last_unstale":"2024-03-06T07:26:24.664941+0000","last_undegraded":"2024-03-06T07:26:24.664941+0000","last_fullsized":"2024-03-06T07:26:24.664941+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:46.783070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662415+0000","last_change":"2024-03-06T07:26:16.625352+0000","last_active":"2024-03-06T07:26:24.662415+0000","last_peered":"2024-03-06T07:26:24.662415+0000","last_clean":"2024-03-06T07:26:24.662415+0000","last_became_active":"2024-03-06T07:26:16.624893+0000","last_became_peered":"2024-03-06T07:26:16.624893+0000","last_unstale":"2024-03-06T07:26:24.662415+0000","last_undegraded":"2024-03-06T07:26:24.662415+0000","last_fullsized":"2024-03-06T07:26:24.662415+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:29.596513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.667139+0000","last_change":"2024-03-06T07:26:24.667139+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.667139+0000","last_undegraded":"2024-03-06T07:26:24.667139+0000","last_fullsized":"2024-03-06T07:26:24.667139+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667199+0000","last_change":"2024-03-06T07:26:17.639031+0000","last_active":"2024-03-06T07:26:24.667199+0000","last_peered":"2024-03-06T07:26:24.667199+0000","last_clean":"2024-03-06T07:26:24.667199+0000","last_became_active":"2024-03-06T07:26:17.638716+0000","last_became_peered":"2024-03-06T07:26:17.638716+0000","last_unstale":"2024-03-06T07:26:24.667199+0000","last_undegraded":"2024-03-06T07:26:24.667199+0000","last_fullsized":"2024-03-06T07:26:24.667199+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:46.783070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666494+0000","last_change":"2024-03-06T07:26:16.620528+0000","last_active":"2024-03-06T07:26:24.666494+0000","last_peered":"2024-03-06T07:26:24.666494+0000","last_clean":"2024-03-06T07:26:24.666494+0000","last_became_active":"2024-03-06T07:26:16.620245+0000","last_became_peered":"2024-03-06T07:26:16.620245+0000","last_unstale":"2024-03-06T07:26:24.666494+0000","last_undegraded":"2024-03-06T07:26:24.666494+0000","last_fullsized":"2024-03-06T07:26:24.666494+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:39.028254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661458+0000","last_change":"2024-03-06T07:26:24.661458+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661458+0000","last_undegraded":"2024-03-06T07:26:24.661458+0000","last_fullsized":"2024-03-06T07:26:24.661458+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658283+0000","last_change":"2024-03-06T07:26:17.641468+0000","last_active":"2024-03-06T07:26:24.658283+0000","last_peered":"2024-03-06T07:26:24.658283+0000","last_clean":"2024-03-06T07:26:24.658283+0000","last_became_active":"2024-03-06T07:26:17.641155+0000","last_became_peered":"2024-03-06T07:26:17.641155+0000","last_unstale":"2024-03-06T07:26:24.658283+0000","last_undegraded":"2024-03-06T07:26:24.658283+0000","last_fullsized":"2024-03-06T07:26:24.658283+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:27.464355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"148'8","reported_seq":24,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664388+0000","last_change":"2024-03-06T07:26:16.620548+0000","last_active":"2024-03-06T07:26:24.664388+0000","last_peered":"2024-03-06T07:26:24.664388+0000","last_clean":"2024-03-06T07:26:24.664388+0000","last_became_active":"2024-03-06T07:26:16.620289+0000","last_became_peered":"2024-03-06T07:26:16.620289+0000","last_unstale":"2024-03-06T07:26:24.664388+0000","last_undegraded":"2024-03-06T07:26:24.664388+0000","last_fullsized":"2024-03-06T07:26:24.664388+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:08.155799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.660981+0000","last_change":"2024-03-06T07:26:24.660981+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.660981+0000","last_undegraded":"2024-03-06T07:26:24.660981+0000","last_fullsized":"2024-03-06T07:26:24.660981+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664277+0000","last_change":"2024-03-06T07:26:17.629187+0000","last_active":"2024-03-06T07:26:24.664277+0000","last_peered":"2024-03-06T07:26:24.664277+0000","last_clean":"2024-03-06T07:26:24.664277+0000","last_became_active":"2024-03-06T07:26:17.628903+0000","last_became_peered":"2024-03-06T07:26:17.628903+0000","last_unstale":"2024-03-06T07:26:24.664277+0000","last_undegraded":"2024-03-06T07:26:24.664277+0000","last_fullsized":"2024-03-06T07:26:24.664277+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:25.173456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"148'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662984+0000","last_change":"2024-03-06T07:26:16.617108+0000","last_active":"2024-03-06T07:26:24.662984+0000","last_peered":"2024-03-06T07:26:24.662984+0000","last_clean":"2024-03-06T07:26:24.662984+0000","last_became_active":"2024-03-06T07:26:16.616776+0000","last_became_peered":"2024-03-06T07:26:16.616776+0000","last_unstale":"2024-03-06T07:26:24.662984+0000","last_undegraded":"2024-03-06T07:26:24.662984+0000","last_fullsized":"2024-03-06T07:26:24.662984+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:01.223054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.664369+0000","last_change":"2024-03-06T07:26:24.664369+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.664369+0000","last_undegraded":"2024-03-06T07:26:24.664369+0000","last_fullsized":"2024-03-06T07:26:24.664369+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665818+0000","last_change":"2024-03-06T07:26:17.634792+0000","last_active":"2024-03-06T07:26:24.665818+0000","last_peered":"2024-03-06T07:26:24.665818+0000","last_clean":"2024-03-06T07:26:24.665818+0000","last_became_active":"2024-03-06T07:26:17.634367+0000","last_became_peered":"2024-03-06T07:26:17.634367+0000","last_unstale":"2024-03-06T07:26:24.665818+0000","last_undegraded":"2024-03-06T07:26:24.665818+0000","last_fullsized":"2024-03-06T07:26:24.665818+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:11.386543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665495+0000","last_change":"2024-03-06T07:26:16.619012+0000","last_active":"2024-03-06T07:26:24.665495+0000","last_peered":"2024-03-06T07:26:24.665495+0000","last_clean":"2024-03-06T07:26:24.665495+0000","last_became_active":"2024-03-06T07:26:16.617718+0000","last_became_peered":"2024-03-06T07:26:16.617718+0000","last_unstale":"2024-03-06T07:26:24.665495+0000","last_undegraded":"2024-03-06T07:26:24.665495+0000","last_fullsized":"2024-03-06T07:26:24.665495+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:08.220437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.666049+0000","last_change":"2024-03-06T07:26:24.666049+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.666049+0000","last_undegraded":"2024-03-06T07:26:24.666049+0000","last_fullsized":"2024-03-06T07:26:24.666049+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666499+0000","last_change":"2024-03-06T07:26:17.633394+0000","last_active":"2024-03-06T07:26:24.666499+0000","last_peered":"2024-03-06T07:26:24.666499+0000","last_clean":"2024-03-06T07:26:24.666499+0000","last_became_active":"2024-03-06T07:26:17.632438+0000","last_became_peered":"2024-03-06T07:26:17.632438+0000","last_unstale":"2024-03-06T07:26:24.666499+0000","last_undegraded":"2024-03-06T07:26:24.666499+0000","last_fullsized":"2024-03-06T07:26:24.666499+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:12.024002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"148'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668243+0000","last_change":"2024-03-06T07:26:16.628646+0000","last_active":"2024-03-06T07:26:24.668243+0000","last_peered":"2024-03-06T07:26:24.668243+0000","last_clean":"2024-03-06T07:26:24.668243+0000","last_became_active":"2024-03-06T07:26:16.627947+0000","last_became_peered":"2024-03-06T07:26:16.627947+0000","last_unstale":"2024-03-06T07:26:24.668243+0000","last_undegraded":"2024-03-06T07:26:24.668243+0000","last_fullsized":"2024-03-06T07:26:24.668243+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:25.875046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665349+0000","last_change":"2024-03-06T07:26:24.665349+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665349+0000","last_undegraded":"2024-03-06T07:26:24.665349+0000","last_fullsized":"2024-03-06T07:26:24.665349+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666918+0000","last_change":"2024-03-06T07:26:17.634254+0000","last_active":"2024-03-06T07:26:24.666918+0000","last_peered":"2024-03-06T07:26:24.666918+0000","last_clean":"2024-03-06T07:26:24.666918+0000","last_became_active":"2024-03-06T07:26:17.633961+0000","last_became_peered":"2024-03-06T07:26:17.633961+0000","last_unstale":"2024-03-06T07:26:24.666918+0000","last_undegraded":"2024-03-06T07:26:24.666918+0000","last_fullsized":"2024-03-06T07:26:24.666918+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:59.873611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665244+0000","last_change":"2024-03-06T07:26:16.619532+0000","last_active":"2024-03-06T07:26:24.665244+0000","last_peered":"2024-03-06T07:26:24.665244+0000","last_clean":"2024-03-06T07:26:24.665244+0000","last_became_active":"2024-03-06T07:26:16.619261+0000","last_became_peered":"2024-03-06T07:26:16.619261+0000","last_unstale":"2024-03-06T07:26:24.665244+0000","last_undegraded":"2024-03-06T07:26:24.665244+0000","last_fullsized":"2024-03-06T07:26:24.665244+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:34.031068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661217+0000","last_change":"2024-03-06T07:26:24.661217+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661217+0000","last_undegraded":"2024-03-06T07:26:24.661217+0000","last_fullsized":"2024-03-06T07:26:24.661217+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662447+0000","last_change":"2024-03-06T07:26:17.637879+0000","last_active":"2024-03-06T07:26:24.662447+0000","last_peered":"2024-03-06T07:26:24.662447+0000","last_clean":"2024-03-06T07:26:24.662447+0000","last_became_active":"2024-03-06T07:26:17.637094+0000","last_became_peered":"2024-03-06T07:26:17.637094+0000","last_unstale":"2024-03-06T07:26:24.662447+0000","last_undegraded":"2024-03-06T07:26:24.662447+0000","last_fullsized":"2024-03-06T07:26:24.662447+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:34.084422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668442+0000","last_change":"2024-03-06T07:26:16.622856+0000","last_active":"2024-03-06T07:26:24.668442+0000","last_peered":"2024-03-06T07:26:24.668442+0000","last_clean":"2024-03-06T07:26:24.668442+0000","last_became_active":"2024-03-06T07:26:16.622421+0000","last_became_peered":"2024-03-06T07:26:16.622421+0000","last_unstale":"2024-03-06T07:26:24.668442+0000","last_undegraded":"2024-03-06T07:26:24.668442+0000","last_fullsized":"2024-03-06T07:26:24.668442+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:40.651795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661343+0000","last_change":"2024-03-06T07:26:24.661343+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661343+0000","last_undegraded":"2024-03-06T07:26:24.661343+0000","last_fullsized":"2024-03-06T07:26:24.661343+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668215+0000","last_change":"2024-03-06T07:26:17.629188+0000","last_active":"2024-03-06T07:26:24.668215+0000","last_peered":"2024-03-06T07:26:24.668215+0000","last_clean":"2024-03-06T07:26:24.668215+0000","last_became_active":"2024-03-06T07:26:17.628881+0000","last_became_peered":"2024-03-06T07:26:17.628881+0000","last_unstale":"2024-03-06T07:26:24.668215+0000","last_undegraded":"2024-03-06T07:26:24.668215+0000","last_fullsized":"2024-03-06T07:26:24.668215+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:44.957682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666547+0000","last_change":"2024-03-06T07:26:16.625755+0000","last_active":"2024-03-06T07:26:24.666547+0000","last_peered":"2024-03-06T07:26:24.666547+0000","last_clean":"2024-03-06T07:26:24.666547+0000","last_became_active":"2024-03-06T07:26:16.625489+0000","last_became_peered":"2024-03-06T07:26:16.625489+0000","last_unstale":"2024-03-06T07:26:24.666547+0000","last_undegraded":"2024-03-06T07:26:24.666547+0000","last_fullsized":"2024-03-06T07:26:24.666547+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:03.053080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.666734+0000","last_change":"2024-03-06T07:26:24.666734+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.666734+0000","last_undegraded":"2024-03-06T07:26:24.666734+0000","last_fullsized":"2024-03-06T07:26:24.666734+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668192+0000","last_change":"2024-03-06T07:26:17.629894+0000","last_active":"2024-03-06T07:26:24.668192+0000","last_peered":"2024-03-06T07:26:24.668192+0000","last_clean":"2024-03-06T07:26:24.668192+0000","last_became_active":"2024-03-06T07:26:17.629624+0000","last_became_peered":"2024-03-06T07:26:17.629624+0000","last_unstale":"2024-03-06T07:26:24.668192+0000","last_undegraded":"2024-03-06T07:26:24.668192+0000","last_fullsized":"2024-03-06T07:26:24.668192+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:48.520981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658129+0000","last_change":"2024-03-06T07:26:16.618167+0000","last_active":"2024-03-06T07:26:24.658129+0000","last_peered":"2024-03-06T07:26:24.658129+0000","last_clean":"2024-03-06T07:26:24.658129+0000","last_became_active":"2024-03-06T07:26:16.617893+0000","last_became_peered":"2024-03-06T07:26:16.617893+0000","last_unstale":"2024-03-06T07:26:24.658129+0000","last_undegraded":"2024-03-06T07:26:24.658129+0000","last_fullsized":"2024-03-06T07:26:24.658129+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:33.738360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.660975+0000","last_change":"2024-03-06T07:26:24.660975+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.660975+0000","last_undegraded":"2024-03-06T07:26:24.660975+0000","last_fullsized":"2024-03-06T07:26:24.660975+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664802+0000","last_change":"2024-03-06T07:26:17.632248+0000","last_active":"2024-03-06T07:26:24.664802+0000","last_peered":"2024-03-06T07:26:24.664802+0000","last_clean":"2024-03-06T07:26:24.664802+0000","last_became_active":"2024-03-06T07:26:17.631912+0000","last_became_peered":"2024-03-06T07:26:17.631912+0000","last_unstale":"2024-03-06T07:26:24.664802+0000","last_undegraded":"2024-03-06T07:26:24.664802+0000","last_fullsized":"2024-03-06T07:26:24.664802+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:29.113308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664561+0000","last_change":"2024-03-06T07:26:16.616949+0000","last_active":"2024-03-06T07:26:24.664561+0000","last_peered":"2024-03-06T07:26:24.664561+0000","last_clean":"2024-03-06T07:26:24.664561+0000","last_became_active":"2024-03-06T07:26:16.616670+0000","last_became_peered":"2024-03-06T07:26:16.616670+0000","last_unstale":"2024-03-06T07:26:24.664561+0000","last_undegraded":"2024-03-06T07:26:24.664561+0000","last_fullsized":"2024-03-06T07:26:24.664561+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:59.587645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661489+0000","last_change":"2024-03-06T07:26:24.661489+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661489+0000","last_undegraded":"2024-03-06T07:26:24.661489+0000","last_fullsized":"2024-03-06T07:26:24.661489+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666760+0000","last_change":"2024-03-06T07:26:17.628994+0000","last_active":"2024-03-06T07:26:24.666760+0000","last_peered":"2024-03-06T07:26:24.666760+0000","last_clean":"2024-03-06T07:26:24.666760+0000","last_became_active":"2024-03-06T07:26:17.628722+0000","last_became_peered":"2024-03-06T07:26:17.628722+0000","last_unstale":"2024-03-06T07:26:24.666760+0000","last_undegraded":"2024-03-06T07:26:24.666760+0000","last_fullsized":"2024-03-06T07:26:24.666760+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:43.090011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668358+0000","last_change":"2024-03-06T07:26:16.620011+0000","last_active":"2024-03-06T07:26:24.668358+0000","last_peered":"2024-03-06T07:26:24.668358+0000","last_clean":"2024-03-06T07:26:24.668358+0000","last_became_active":"2024-03-06T07:26:16.619736+0000","last_became_peered":"2024-03-06T07:26:16.619736+0000","last_unstale":"2024-03-06T07:26:24.668358+0000","last_undegraded":"2024-03-06T07:26:24.668358+0000","last_fullsized":"2024-03-06T07:26:24.668358+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:24.169321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.663886+0000","last_change":"2024-03-06T07:26:24.663886+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.663886+0000","last_undegraded":"2024-03-06T07:26:24.663886+0000","last_fullsized":"2024-03-06T07:26:24.663886+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664813+0000","last_change":"2024-03-06T07:26:17.628485+0000","last_active":"2024-03-06T07:26:24.664813+0000","last_peered":"2024-03-06T07:26:24.664813+0000","last_clean":"2024-03-06T07:26:24.664813+0000","last_became_active":"2024-03-06T07:26:17.628204+0000","last_became_peered":"2024-03-06T07:26:17.628204+0000","last_unstale":"2024-03-06T07:26:24.664813+0000","last_undegraded":"2024-03-06T07:26:24.664813+0000","last_fullsized":"2024-03-06T07:26:24.664813+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:56.134884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"149'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662755+0000","last_change":"2024-03-06T07:26:16.627066+0000","last_active":"2024-03-06T07:26:24.662755+0000","last_peered":"2024-03-06T07:26:24.662755+0000","last_clean":"2024-03-06T07:26:24.662755+0000","last_became_active":"2024-03-06T07:26:16.626418+0000","last_became_peered":"2024-03-06T07:26:16.626418+0000","last_unstale":"2024-03-06T07:26:24.662755+0000","last_undegraded":"2024-03-06T07:26:24.662755+0000","last_fullsized":"2024-03-06T07:26:24.662755+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:37.036435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661600+0000","last_change":"2024-03-06T07:26:24.661600+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661600+0000","last_undegraded":"2024-03-06T07:26:24.661600+0000","last_fullsized":"2024-03-06T07:26:24.661600+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658129+0000","last_change":"2024-03-06T07:26:17.626185+0000","last_active":"2024-03-06T07:26:24.658129+0000","last_peered":"2024-03-06T07:26:24.658129+0000","last_clean":"2024-03-06T07:26:24.658129+0000","last_became_active":"2024-03-06T07:26:17.625895+0000","last_became_peered":"2024-03-06T07:26:17.625895+0000","last_unstale":"2024-03-06T07:26:24.658129+0000","last_undegraded":"2024-03-06T07:26:24.658129+0000","last_fullsized":"2024-03-06T07:26:24.658129+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:16.007003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662590+0000","last_change":"2024-03-06T07:26:16.625471+0000","last_active":"2024-03-06T07:26:24.662590+0000","last_peered":"2024-03-06T07:26:24.662590+0000","last_clean":"2024-03-06T07:26:24.662590+0000","last_became_active":"2024-03-06T07:26:16.625198+0000","last_became_peered":"2024-03-06T07:26:16.625198+0000","last_unstale":"2024-03-06T07:26:24.662590+0000","last_undegraded":"2024-03-06T07:26:24.662590+0000","last_fullsized":"2024-03-06T07:26:24.662590+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:37.984120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.664299+0000","last_change":"2024-03-06T07:26:24.664299+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.664299+0000","last_undegraded":"2024-03-06T07:26:24.664299+0000","last_fullsized":"2024-03-06T07:26:24.664299+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664790+0000","last_change":"2024-03-06T07:26:17.630631+0000","last_active":"2024-03-06T07:26:24.664790+0000","last_peered":"2024-03-06T07:26:24.664790+0000","last_clean":"2024-03-06T07:26:24.664790+0000","last_became_active":"2024-03-06T07:26:17.630365+0000","last_became_peered":"2024-03-06T07:26:17.630365+0000","last_unstale":"2024-03-06T07:26:24.664790+0000","last_undegraded":"2024-03-06T07:26:24.664790+0000","last_fullsized":"2024-03-06T07:26:24.664790+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:27.398343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658871+0000","last_change":"2024-03-06T07:26:16.611334+0000","last_active":"2024-03-06T07:26:24.658871+0000","last_peered":"2024-03-06T07:26:24.658871+0000","last_clean":"2024-03-06T07:26:24.658871+0000","last_became_active":"2024-03-06T07:26:16.610646+0000","last_became_peered":"2024-03-06T07:26:16.610646+0000","last_unstale":"2024-03-06T07:26:24.658871+0000","last_undegraded":"2024-03-06T07:26:24.658871+0000","last_fullsized":"2024-03-06T07:26:24.658871+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:32.639393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.664690+0000","last_change":"2024-03-06T07:26:24.664690+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.664690+0000","last_undegraded":"2024-03-06T07:26:24.664690+0000","last_fullsized":"2024-03-06T07:26:24.664690+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663654+0000","last_change":"2024-03-06T07:26:17.632379+0000","last_active":"2024-03-06T07:26:24.663654+0000","last_peered":"2024-03-06T07:26:24.663654+0000","last_clean":"2024-03-06T07:26:24.663654+0000","last_became_active":"2024-03-06T07:26:17.631899+0000","last_became_peered":"2024-03-06T07:26:17.631899+0000","last_unstale":"2024-03-06T07:26:24.663654+0000","last_undegraded":"2024-03-06T07:26:24.663654+0000","last_fullsized":"2024-03-06T07:26:24.663654+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:00.591780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663294+0000","last_change":"2024-03-06T07:26:16.623898+0000","last_active":"2024-03-06T07:26:24.663294+0000","last_peered":"2024-03-06T07:26:24.663294+0000","last_clean":"2024-03-06T07:26:24.663294+0000","last_became_active":"2024-03-06T07:26:16.623602+0000","last_became_peered":"2024-03-06T07:26:16.623602+0000","last_unstale":"2024-03-06T07:26:24.663294+0000","last_undegraded":"2024-03-06T07:26:24.663294+0000","last_fullsized":"2024-03-06T07:26:24.663294+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:56.210974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665305+0000","last_change":"2024-03-06T07:26:24.665305+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665305+0000","last_undegraded":"2024-03-06T07:26:24.665305+0000","last_fullsized":"2024-03-06T07:26:24.665305+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664900+0000","last_change":"2024-03-06T07:26:17.632856+0000","last_active":"2024-03-06T07:26:24.664900+0000","last_peered":"2024-03-06T07:26:24.664900+0000","last_clean":"2024-03-06T07:26:24.664900+0000","last_became_active":"2024-03-06T07:26:17.632027+0000","last_became_peered":"2024-03-06T07:26:17.632027+0000","last_unstale":"2024-03-06T07:26:24.664900+0000","last_undegraded":"2024-03-06T07:26:24.664900+0000","last_fullsized":"2024-03-06T07:26:24.664900+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:38.988254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662329+0000","last_change":"2024-03-06T07:26:16.627698+0000","last_active":"2024-03-06T07:26:24.662329+0000","last_peered":"2024-03-06T07:26:24.662329+0000","last_clean":"2024-03-06T07:26:24.662329+0000","last_became_active":"2024-03-06T07:26:16.627318+0000","last_became_peered":"2024-03-06T07:26:16.627318+0000","last_unstale":"2024-03-06T07:26:24.662329+0000","last_undegraded":"2024-03-06T07:26:24.662329+0000","last_fullsized":"2024-03-06T07:26:24.662329+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:08.249739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665228+0000","last_change":"2024-03-06T07:26:24.665228+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665228+0000","last_undegraded":"2024-03-06T07:26:24.665228+0000","last_fullsized":"2024-03-06T07:26:24.665228+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662577+0000","last_change":"2024-03-06T07:26:17.637676+0000","last_active":"2024-03-06T07:26:24.662577+0000","last_peered":"2024-03-06T07:26:24.662577+0000","last_clean":"2024-03-06T07:26:24.662577+0000","last_became_active":"2024-03-06T07:26:17.637375+0000","last_became_peered":"2024-03-06T07:26:17.637375+0000","last_unstale":"2024-03-06T07:26:24.662577+0000","last_undegraded":"2024-03-06T07:26:24.662577+0000","last_fullsized":"2024-03-06T07:26:24.662577+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:18.556057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666678+0000","last_change":"2024-03-06T07:26:16.620202+0000","last_active":"2024-03-06T07:26:24.666678+0000","last_peered":"2024-03-06T07:26:24.666678+0000","last_clean":"2024-03-06T07:26:24.666678+0000","last_became_active":"2024-03-06T07:26:16.619897+0000","last_became_peered":"2024-03-06T07:26:16.619897+0000","last_unstale":"2024-03-06T07:26:24.666678+0000","last_undegraded":"2024-03-06T07:26:24.666678+0000","last_fullsized":"2024-03-06T07:26:24.666678+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:48.709582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.666356+0000","last_change":"2024-03-06T07:26:24.666356+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.666356+0000","last_undegraded":"2024-03-06T07:26:24.666356+0000","last_fullsized":"2024-03-06T07:26:24.666356+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658331+0000","last_change":"2024-03-06T07:26:17.625716+0000","last_active":"2024-03-06T07:26:24.658331+0000","last_peered":"2024-03-06T07:26:24.658331+0000","last_clean":"2024-03-06T07:26:24.658331+0000","last_became_active":"2024-03-06T07:26:17.625429+0000","last_became_peered":"2024-03-06T07:26:17.625429+0000","last_unstale":"2024-03-06T07:26:24.658331+0000","last_undegraded":"2024-03-06T07:26:24.658331+0000","last_fullsized":"2024-03-06T07:26:24.658331+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:38.596798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667604+0000","last_change":"2024-03-06T07:26:16.620839+0000","last_active":"2024-03-06T07:26:24.667604+0000","last_peered":"2024-03-06T07:26:24.667604+0000","last_clean":"2024-03-06T07:26:24.667604+0000","last_became_active":"2024-03-06T07:26:16.620561+0000","last_became_peered":"2024-03-06T07:26:16.620561+0000","last_unstale":"2024-03-06T07:26:24.667604+0000","last_undegraded":"2024-03-06T07:26:24.667604+0000","last_fullsized":"2024-03-06T07:26:24.667604+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:26.394208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.663905+0000","last_change":"2024-03-06T07:26:24.663905+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.663905+0000","last_undegraded":"2024-03-06T07:26:24.663905+0000","last_fullsized":"2024-03-06T07:26:24.663905+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664986+0000","last_change":"2024-03-06T07:26:17.632976+0000","last_active":"2024-03-06T07:26:24.664986+0000","last_peered":"2024-03-06T07:26:24.664986+0000","last_clean":"2024-03-06T07:26:24.664986+0000","last_became_active":"2024-03-06T07:26:17.632721+0000","last_became_peered":"2024-03-06T07:26:17.632721+0000","last_unstale":"2024-03-06T07:26:24.664986+0000","last_undegraded":"2024-03-06T07:26:24.664986+0000","last_fullsized":"2024-03-06T07:26:24.664986+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:56.134884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662665+0000","last_change":"2024-03-06T07:26:16.626430+0000","last_active":"2024-03-06T07:26:24.662665+0000","last_peered":"2024-03-06T07:26:24.662665+0000","last_clean":"2024-03-06T07:26:24.662665+0000","last_became_active":"2024-03-06T07:26:16.625971+0000","last_became_peered":"2024-03-06T07:26:16.625971+0000","last_unstale":"2024-03-06T07:26:24.662665+0000","last_undegraded":"2024-03-06T07:26:24.662665+0000","last_fullsized":"2024-03-06T07:26:24.662665+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:28.109173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":419,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664587+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:26:24.664587+0000","last_peered":"2024-03-06T07:26:24.664587+0000","last_clean":"2024-03-06T07:26:24.664587+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:26:24.664587+0000","last_undegraded":"2024-03-06T07:26:24.664587+0000","last_fullsized":"2024-03-06T07:26:24.664587+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":1941504,"data_stored":1496336,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574924,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33016,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1875968,"data_stored":905968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574924,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1884160,"data_stored":906092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574925,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1892352,"data_stored":906732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574924,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33036,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338804,"statfs":{"total":96636764160,"available":96602935296,"internally_reserved":0,"allocated":1896448,"data_stored":908184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574924,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33016,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1875968,"data_stored":905968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1888256,"data_stored":907580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574925,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33088,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338752,"statfs":{"total":96636764160,"available":96602882048,"internally_reserved":0,"allocated":1949696,"data_stored":1496440,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:29.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:29.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.852+0000 7fa96dd0c640 1 Processor -- start 2024-03-06T07:26:29.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.852+0000 7fa96dd0c640 1 -- start start 2024-03-06T07:26:29.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.853+0000 7fa96dd0c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa9680c6b10 0x7fa968155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.853+0000 7fa96dd0c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa9680bf9d0 0x7fa9680c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.853+0000 7fa96dd0c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9680c2fc0 con 0x7fa968152650 2024-03-06T07:26:29.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.853+0000 7fa96dd0c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa968155800 con 0x7fa9680bf9d0 2024-03-06T07:26:29.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.853+0000 7fa96dd0c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa968158da0 con 0x7fa9680c6b10 2024-03-06T07:26:29.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.853+0000 7fa967fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa9680bf9d0 0x7fa9680c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.853+0000 7fa9677fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa968152650 0x7fa968158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60342/0 (socket says 172.21.15.1:60342) 2024-03-06T07:26:29.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.853+0000 7fa9677fe640 1 -- 172.21.15.1:0/532162215 learned_addr learned my addr 172.21.15.1:0/532162215 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:29.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.853+0000 7fa966ffd640 1 --2- 172.21.15.1:0/532162215 >> v2:172.21.15.98:3300/0 conn(0x7fa9680c6b10 0x7fa968155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.853+0000 7fa967fff640 1 -- 172.21.15.1:0/532162215 >> v2:172.21.15.98:3300/0 conn(0x7fa9680c6b10 msgr2=0x7fa968155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:29.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.853+0000 7fa967fff640 1 --2- 172.21.15.1:0/532162215 >> v2:172.21.15.98:3300/0 conn(0x7fa9680c6b10 0x7fa968155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.853+0000 7fa967fff640 1 -- 172.21.15.1:0/532162215 >> v1:172.21.15.1:6789/0 conn(0x7fa968152650 legacy=0x7fa968158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.853+0000 7fa967fff640 1 -- 172.21.15.1:0/532162215 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9680c6690 con 0x7fa9680bf9d0 2024-03-06T07:26:29.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.853+0000 7fa9667fc640 1 -- 172.21.15.1:0/532162215 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468659467 0 0) 0x7fa9680c2fc0 con 0x7fa968152650 2024-03-06T07:26:29.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.854+0000 7fa967fff640 1 --2- 172.21.15.1:0/532162215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9680bf9d0 0x7fa9680c1460 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fa9680cb8e0 tx=0x7fa96002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c0b02dd1db7eab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.854+0000 7fa9667fc640 1 -- 172.21.15.1:0/532162215 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa960039070 con 0x7fa9680bf9d0 2024-03-06T07:26:29.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.854+0000 7fa9667fc640 1 -- 172.21.15.1:0/532162215 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96002cbf0 con 0x7fa9680bf9d0 2024-03-06T07:26:29.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.854+0000 7fa9667fc640 1 -- 172.21.15.1:0/532162215 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa960034440 con 0x7fa9680bf9d0 2024-03-06T07:26:29.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.855+0000 7fa96dd0c640 1 -- 172.21.15.1:0/532162215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9680bf9d0 msgr2=0x7fa9680c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.855+0000 7fa96dd0c640 1 --2- 172.21.15.1:0/532162215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9680bf9d0 0x7fa9680c1460 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fa9680cb8e0 tx=0x7fa96002be40 comp rx=0 tx=0).stop 2024-03-06T07:26:29.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.855+0000 7fa96dd0c640 1 -- 172.21.15.1:0/532162215 shutdown_connections 2024-03-06T07:26:29.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.855+0000 7fa96dd0c640 1 --2- 172.21.15.1:0/532162215 >> v2:172.21.15.98:3300/0 conn(0x7fa9680c6b10 0x7fa968155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.855+0000 7fa96dd0c640 1 --2- 172.21.15.1:0/532162215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9680bf9d0 0x7fa9680c1460 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.855+0000 7fa96dd0c640 1 -- 172.21.15.1:0/532162215 >> 172.21.15.1:0/532162215 conn(0x7fa968066ee0 msgr2=0x7fa9680672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.855+0000 7fa96dd0c640 1 -- 172.21.15.1:0/532162215 shutdown_connections 2024-03-06T07:26:29.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.855+0000 7fa96dd0c640 1 -- 172.21.15.1:0/532162215 wait complete. 2024-03-06T07:26:29.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.856+0000 7fa96dd0c640 1 Processor -- start 2024-03-06T07:26:29.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.857+0000 7fa96dd0c640 1 -- start start 2024-03-06T07:26:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.858+0000 7fa96dd0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9680bf9d0 0x7fa9681ee900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.858+0000 7fa96dd0c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa968152650 0x7fa968154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.858+0000 7fa96dd0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9680c16d0 0x7fa968156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.858+0000 7fa96dd0c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9681551b0 con 0x7fa968152650 2024-03-06T07:26:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.858+0000 7fa96dd0c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9680c6fa0 con 0x7fa9680bf9d0 2024-03-06T07:26:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.858+0000 7fa96dd0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa968155490 con 0x7fa9680c16d0 2024-03-06T07:26:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.858+0000 7fa966ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa968152650 0x7fa968154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.858+0000 7fa966ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa968152650 0x7fa968154530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58634/0 (socket says 172.21.15.1:58634) 2024-03-06T07:26:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.858+0000 7fa966ffd640 1 -- 172.21.15.1:0/2940000305 learned_addr learned my addr 172.21.15.1:0/2940000305 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:29.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.858+0000 7fa9677fe640 1 --2- 172.21.15.1:0/2940000305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9680bf9d0 0x7fa9681ee900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.858+0000 7fa967fff640 1 --2- 172.21.15.1:0/2940000305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9680c16d0 0x7fa968156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.858+0000 7fa966ffd640 1 -- 172.21.15.1:0/2940000305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9680c16d0 msgr2=0x7fa968156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.858+0000 7fa966ffd640 1 --2- 172.21.15.1:0/2940000305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9680c16d0 0x7fa968156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.858+0000 7fa966ffd640 1 -- 172.21.15.1:0/2940000305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9680bf9d0 msgr2=0x7fa9681ee900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.858+0000 7fa966ffd640 1 --2- 172.21.15.1:0/2940000305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9680bf9d0 0x7fa9681ee900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.858+0000 7fa966ffd640 1 -- 172.21.15.1:0/2940000305 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa968152cf0 con 0x7fa968152650 2024-03-06T07:26:29.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.859+0000 7fa966ffd640 1 --2- 172.21.15.1:0/2940000305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa968152650 0x7fa968154530 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00c7f0 tx=0x7fa95c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.859+0000 7fa94ffff640 1 -- 172.21.15.1:0/2940000305 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c015070 con 0x7fa968152650 2024-03-06T07:26:29.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.859+0000 7fa94ffff640 1 -- 172.21.15.1:0/2940000305 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa95c004240 con 0x7fa968152650 2024-03-06T07:26:29.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.859+0000 7fa96dd0c640 1 -- 172.21.15.1:0/2940000305 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa968154bc0 con 0x7fa968152650 2024-03-06T07:26:29.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.859+0000 7fa94ffff640 1 -- 172.21.15.1:0/2940000305 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c003610 con 0x7fa968152650 2024-03-06T07:26:29.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.859+0000 7fa96dd0c640 1 -- 172.21.15.1:0/2940000305 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9681ede20 con 0x7fa968152650 2024-03-06T07:26:29.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.860+0000 7fa96dd0c640 1 -- 172.21.15.1:0/2940000305 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa968067fd0 con 0x7fa968152650 2024-03-06T07:26:29.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.861+0000 7fa94ffff640 1 -- 172.21.15.1:0/2940000305 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa95c0039c0 con 0x7fa968152650 2024-03-06T07:26:29.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.861+0000 7fa94ffff640 1 --2- 172.21.15.1:0/2940000305 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa95406b4d0 0x7fa95406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.861+0000 7fa94ffff640 1 -- 172.21.15.1:0/2940000305 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fa95c08aa50 con 0x7fa968152650 2024-03-06T07:26:29.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.861+0000 7fa9677fe640 1 --2- 172.21.15.1:0/2940000305 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa95406b4d0 0x7fa95406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.862+0000 7fa9677fe640 1 --2- 172.21.15.1:0/2940000305 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa95406b4d0 0x7fa95406d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fa958000f60 tx=0x7fa958007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:29.869+0000 7fa94ffff640 1 -- 172.21.15.1:0/2940000305 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa95c058db0 con 0x7fa968152650 2024-03-06T07:26:30.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.045+0000 7fa96dd0c640 1 -- 172.21.15.1:0/2940000305 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa96814fa80 con 0x7fa95406b4d0 2024-03-06T07:26:30.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.057+0000 7fa94ffff640 1 -- 172.21.15.1:0/2940000305 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+479322 (secure 0 0 0) 0x7fa96814fa80 con 0x7fa95406b4d0 2024-03-06T07:26:30.064 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:30.067 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:26:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.061+0000 7fa96dd0c640 1 -- 172.21.15.1:0/2940000305 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa95406b4d0 msgr2=0x7fa95406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.061+0000 7fa96dd0c640 1 --2- 172.21.15.1:0/2940000305 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa95406b4d0 0x7fa95406d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fa958000f60 tx=0x7fa958007d50 comp rx=0 tx=0).stop 2024-03-06T07:26:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.061+0000 7fa96dd0c640 1 -- 172.21.15.1:0/2940000305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa968152650 msgr2=0x7fa968154530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.061+0000 7fa96dd0c640 1 --2- 172.21.15.1:0/2940000305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa968152650 0x7fa968154530 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00c7f0 tx=0x7fa95c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.061+0000 7fa96dd0c640 1 -- 172.21.15.1:0/2940000305 shutdown_connections 2024-03-06T07:26:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.061+0000 7fa96dd0c640 1 --2- 172.21.15.1:0/2940000305 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa95406b4d0 0x7fa95406d990 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.061+0000 7fa96dd0c640 1 --2- 172.21.15.1:0/2940000305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9680c16d0 0x7fa968156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.061+0000 7fa96dd0c640 1 --2- 172.21.15.1:0/2940000305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa968152650 0x7fa968154530 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.061+0000 7fa96dd0c640 1 --2- 172.21.15.1:0/2940000305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9680bf9d0 0x7fa9681ee900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.061+0000 7fa96dd0c640 1 -- 172.21.15.1:0/2940000305 >> 172.21.15.1:0/2940000305 conn(0x7fa968066ee0 msgr2=0x7fa9680672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.061+0000 7fa96dd0c640 1 -- 172.21.15.1:0/2940000305 shutdown_connections 2024-03-06T07:26:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.061+0000 7fa96dd0c640 1 -- 172.21.15.1:0/2940000305 wait complete. 2024-03-06T07:26:30.073 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-03-06T07:26:28.774882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264320,"kb_used_data":14848,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710400,"statfs":{"total":773094113280,"available":772823449600,"internally_reserved":0,"allocated":15204352,"data_stored":8433300,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.115308"},"pg_stats":[{"pgid":"26.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3f","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663770+0000","last_change":"2024-03-06T07:26:17.632610+0000","last_active":"2024-03-06T07:26:24.663770+0000","last_peered":"2024-03-06T07:26:24.663770+0000","last_clean":"2024-03-06T07:26:24.663770+0000","last_became_active":"2024-03-06T07:26:17.630763+0000","last_became_peered":"2024-03-06T07:26:17.630763+0000","last_unstale":"2024-03-06T07:26:24.663770+0000","last_undegraded":"2024-03-06T07:26:24.663770+0000","last_fullsized":"2024-03-06T07:26:24.663770+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:35.035202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664635+0000","last_change":"2024-03-06T07:26:17.632425+0000","last_active":"2024-03-06T07:26:24.664635+0000","last_peered":"2024-03-06T07:26:24.664635+0000","last_clean":"2024-03-06T07:26:24.664635+0000","last_became_active":"2024-03-06T07:26:17.632153+0000","last_became_peered":"2024-03-06T07:26:17.632153+0000","last_unstale":"2024-03-06T07:26:24.664635+0000","last_undegraded":"2024-03-06T07:26:24.664635+0000","last_fullsized":"2024-03-06T07:26:24.664635+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:01.829378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663942+0000","last_change":"2024-03-06T07:26:17.630353+0000","last_active":"2024-03-06T07:26:24.663942+0000","last_peered":"2024-03-06T07:26:24.663942+0000","last_clean":"2024-03-06T07:26:24.663942+0000","last_became_active":"2024-03-06T07:26:17.630080+0000","last_became_peered":"2024-03-06T07:26:17.630080+0000","last_unstale":"2024-03-06T07:26:24.663942+0000","last_undegraded":"2024-03-06T07:26:24.663942+0000","last_fullsized":"2024-03-06T07:26:24.663942+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:49.713717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663552+0000","last_change":"2024-03-06T07:26:17.630434+0000","last_active":"2024-03-06T07:26:24.663552+0000","last_peered":"2024-03-06T07:26:24.663552+0000","last_clean":"2024-03-06T07:26:24.663552+0000","last_became_active":"2024-03-06T07:26:17.630139+0000","last_became_peered":"2024-03-06T07:26:17.630139+0000","last_unstale":"2024-03-06T07:26:24.663552+0000","last_undegraded":"2024-03-06T07:26:24.663552+0000","last_fullsized":"2024-03-06T07:26:24.663552+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:09.159934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668267+0000","last_change":"2024-03-06T07:26:17.632665+0000","last_active":"2024-03-06T07:26:24.668267+0000","last_peered":"2024-03-06T07:26:24.668267+0000","last_clean":"2024-03-06T07:26:24.668267+0000","last_became_active":"2024-03-06T07:26:17.632377+0000","last_became_peered":"2024-03-06T07:26:17.632377+0000","last_unstale":"2024-03-06T07:26:24.668267+0000","last_undegraded":"2024-03-06T07:26:24.668267+0000","last_fullsized":"2024-03-06T07:26:24.668267+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:23.225363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665614+0000","last_change":"2024-03-06T07:26:17.633911+0000","last_active":"2024-03-06T07:26:24.665614+0000","last_peered":"2024-03-06T07:26:24.665614+0000","last_clean":"2024-03-06T07:26:24.665614+0000","last_became_active":"2024-03-06T07:26:17.633638+0000","last_became_peered":"2024-03-06T07:26:17.633638+0000","last_unstale":"2024-03-06T07:26:24.665614+0000","last_undegraded":"2024-03-06T07:26:24.665614+0000","last_fullsized":"2024-03-06T07:26:24.665614+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:01.350963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664718+0000","last_change":"2024-03-06T07:26:17.629894+0000","last_active":"2024-03-06T07:26:24.664718+0000","last_peered":"2024-03-06T07:26:24.664718+0000","last_clean":"2024-03-06T07:26:24.664718+0000","last_became_active":"2024-03-06T07:26:17.629625+0000","last_became_peered":"2024-03-06T07:26:17.629625+0000","last_unstale":"2024-03-06T07:26:24.664718+0000","last_undegraded":"2024-03-06T07:26:24.664718+0000","last_fullsized":"2024-03-06T07:26:24.664718+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:46.687284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664295+0000","last_change":"2024-03-06T07:26:17.632711+0000","last_active":"2024-03-06T07:26:24.664295+0000","last_peered":"2024-03-06T07:26:24.664295+0000","last_clean":"2024-03-06T07:26:24.664295+0000","last_became_active":"2024-03-06T07:26:17.632173+0000","last_became_peered":"2024-03-06T07:26:17.632173+0000","last_unstale":"2024-03-06T07:26:24.664295+0000","last_undegraded":"2024-03-06T07:26:24.664295+0000","last_fullsized":"2024-03-06T07:26:24.664295+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:25.595909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665506+0000","last_change":"2024-03-06T07:26:17.638531+0000","last_active":"2024-03-06T07:26:24.665506+0000","last_peered":"2024-03-06T07:26:24.665506+0000","last_clean":"2024-03-06T07:26:24.665506+0000","last_became_active":"2024-03-06T07:26:17.638254+0000","last_became_peered":"2024-03-06T07:26:17.638254+0000","last_unstale":"2024-03-06T07:26:24.665506+0000","last_undegraded":"2024-03-06T07:26:24.665506+0000","last_fullsized":"2024-03-06T07:26:24.665506+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:01.350963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665485+0000","last_change":"2024-03-06T07:26:17.632455+0000","last_active":"2024-03-06T07:26:24.665485+0000","last_peered":"2024-03-06T07:26:24.665485+0000","last_clean":"2024-03-06T07:26:24.665485+0000","last_became_active":"2024-03-06T07:26:17.632195+0000","last_became_peered":"2024-03-06T07:26:17.632195+0000","last_unstale":"2024-03-06T07:26:24.665485+0000","last_undegraded":"2024-03-06T07:26:24.665485+0000","last_fullsized":"2024-03-06T07:26:24.665485+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:14.404340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667172+0000","last_change":"2024-03-06T07:26:17.633984+0000","last_active":"2024-03-06T07:26:24.667172+0000","last_peered":"2024-03-06T07:26:24.667172+0000","last_clean":"2024-03-06T07:26:24.667172+0000","last_became_active":"2024-03-06T07:26:17.633666+0000","last_became_peered":"2024-03-06T07:26:17.633666+0000","last_unstale":"2024-03-06T07:26:24.667172+0000","last_undegraded":"2024-03-06T07:26:24.667172+0000","last_fullsized":"2024-03-06T07:26:24.667172+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:43.884264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664561+0000","last_change":"2024-03-06T07:26:17.634798+0000","last_active":"2024-03-06T07:26:24.664561+0000","last_peered":"2024-03-06T07:26:24.664561+0000","last_clean":"2024-03-06T07:26:24.664561+0000","last_became_active":"2024-03-06T07:26:17.634537+0000","last_became_peered":"2024-03-06T07:26:17.634537+0000","last_unstale":"2024-03-06T07:26:24.664561+0000","last_undegraded":"2024-03-06T07:26:24.664561+0000","last_fullsized":"2024-03-06T07:26:24.664561+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:25.595909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662407+0000","last_change":"2024-03-06T07:26:17.640814+0000","last_active":"2024-03-06T07:26:24.662407+0000","last_peered":"2024-03-06T07:26:24.662407+0000","last_clean":"2024-03-06T07:26:24.662407+0000","last_became_active":"2024-03-06T07:26:17.640444+0000","last_became_peered":"2024-03-06T07:26:17.640444+0000","last_unstale":"2024-03-06T07:26:24.662407+0000","last_undegraded":"2024-03-06T07:26:24.662407+0000","last_fullsized":"2024-03-06T07:26:24.662407+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:35.920913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667754+0000","last_change":"2024-03-06T07:26:17.631892+0000","last_active":"2024-03-06T07:26:24.667754+0000","last_peered":"2024-03-06T07:26:24.667754+0000","last_clean":"2024-03-06T07:26:24.667754+0000","last_became_active":"2024-03-06T07:26:17.631498+0000","last_became_peered":"2024-03-06T07:26:17.631498+0000","last_unstale":"2024-03-06T07:26:24.667754+0000","last_undegraded":"2024-03-06T07:26:24.667754+0000","last_fullsized":"2024-03-06T07:26:24.667754+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:09.253874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666379+0000","last_change":"2024-03-06T07:26:17.634220+0000","last_active":"2024-03-06T07:26:24.666379+0000","last_peered":"2024-03-06T07:26:24.666379+0000","last_clean":"2024-03-06T07:26:24.666379+0000","last_became_active":"2024-03-06T07:26:17.633922+0000","last_became_peered":"2024-03-06T07:26:17.633922+0000","last_unstale":"2024-03-06T07:26:24.666379+0000","last_undegraded":"2024-03-06T07:26:24.666379+0000","last_fullsized":"2024-03-06T07:26:24.666379+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:51.900557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662529+0000","last_change":"2024-03-06T07:26:17.635856+0000","last_active":"2024-03-06T07:26:24.662529+0000","last_peered":"2024-03-06T07:26:24.662529+0000","last_clean":"2024-03-06T07:26:24.662529+0000","last_became_active":"2024-03-06T07:26:17.631353+0000","last_became_peered":"2024-03-06T07:26:17.631353+0000","last_unstale":"2024-03-06T07:26:24.662529+0000","last_undegraded":"2024-03-06T07:26:24.662529+0000","last_fullsized":"2024-03-06T07:26:24.662529+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:15.652152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665594+0000","last_change":"2024-03-06T07:26:17.628960+0000","last_active":"2024-03-06T07:26:24.665594+0000","last_peered":"2024-03-06T07:26:24.665594+0000","last_clean":"2024-03-06T07:26:24.665594+0000","last_became_active":"2024-03-06T07:26:17.628681+0000","last_became_peered":"2024-03-06T07:26:17.628681+0000","last_unstale":"2024-03-06T07:26:24.665594+0000","last_undegraded":"2024-03-06T07:26:24.665594+0000","last_fullsized":"2024-03-06T07:26:24.665594+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:40.950597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666241+0000","last_change":"2024-03-06T07:26:17.629582+0000","last_active":"2024-03-06T07:26:24.666241+0000","last_peered":"2024-03-06T07:26:24.666241+0000","last_clean":"2024-03-06T07:26:24.666241+0000","last_became_active":"2024-03-06T07:26:17.629246+0000","last_became_peered":"2024-03-06T07:26:17.629246+0000","last_unstale":"2024-03-06T07:26:24.666241+0000","last_undegraded":"2024-03-06T07:26:24.666241+0000","last_fullsized":"2024-03-06T07:26:24.666241+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:56.556155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667465+0000","last_change":"2024-03-06T07:26:17.632386+0000","last_active":"2024-03-06T07:26:24.667465+0000","last_peered":"2024-03-06T07:26:24.667465+0000","last_clean":"2024-03-06T07:26:24.667465+0000","last_became_active":"2024-03-06T07:26:17.631867+0000","last_became_peered":"2024-03-06T07:26:17.631867+0000","last_unstale":"2024-03-06T07:26:24.667465+0000","last_undegraded":"2024-03-06T07:26:24.667465+0000","last_fullsized":"2024-03-06T07:26:24.667465+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:46.687284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658784+0000","last_change":"2024-03-06T07:26:17.622563+0000","last_active":"2024-03-06T07:26:24.658784+0000","last_peered":"2024-03-06T07:26:24.658784+0000","last_clean":"2024-03-06T07:26:24.658784+0000","last_became_active":"2024-03-06T07:26:17.622293+0000","last_became_peered":"2024-03-06T07:26:17.622293+0000","last_unstale":"2024-03-06T07:26:24.658784+0000","last_undegraded":"2024-03-06T07:26:24.658784+0000","last_fullsized":"2024-03-06T07:26:24.658784+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:34.462291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662501+0000","last_change":"2024-03-06T07:26:17.637385+0000","last_active":"2024-03-06T07:26:24.662501+0000","last_peered":"2024-03-06T07:26:24.662501+0000","last_clean":"2024-03-06T07:26:24.662501+0000","last_became_active":"2024-03-06T07:26:17.637063+0000","last_became_peered":"2024-03-06T07:26:17.637063+0000","last_unstale":"2024-03-06T07:26:24.662501+0000","last_undegraded":"2024-03-06T07:26:24.662501+0000","last_fullsized":"2024-03-06T07:26:24.662501+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:49.333296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665712+0000","last_change":"2024-03-06T07:26:17.632278+0000","last_active":"2024-03-06T07:26:24.665712+0000","last_peered":"2024-03-06T07:26:24.665712+0000","last_clean":"2024-03-06T07:26:24.665712+0000","last_became_active":"2024-03-06T07:26:17.631791+0000","last_became_peered":"2024-03-06T07:26:17.631791+0000","last_unstale":"2024-03-06T07:26:24.665712+0000","last_undegraded":"2024-03-06T07:26:24.665712+0000","last_fullsized":"2024-03-06T07:26:24.665712+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:40.950597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667556+0000","last_change":"2024-03-06T07:26:17.627902+0000","last_active":"2024-03-06T07:26:24.667556+0000","last_peered":"2024-03-06T07:26:24.667556+0000","last_clean":"2024-03-06T07:26:24.667556+0000","last_became_active":"2024-03-06T07:26:17.627643+0000","last_became_peered":"2024-03-06T07:26:17.627643+0000","last_unstale":"2024-03-06T07:26:24.667556+0000","last_undegraded":"2024-03-06T07:26:24.667556+0000","last_fullsized":"2024-03-06T07:26:24.667556+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:30.600647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662946+0000","last_change":"2024-03-06T07:26:17.631421+0000","last_active":"2024-03-06T07:26:24.662946+0000","last_peered":"2024-03-06T07:26:24.662946+0000","last_clean":"2024-03-06T07:26:24.662946+0000","last_became_active":"2024-03-06T07:26:17.631022+0000","last_became_peered":"2024-03-06T07:26:17.631022+0000","last_unstale":"2024-03-06T07:26:24.662946+0000","last_undegraded":"2024-03-06T07:26:24.662946+0000","last_fullsized":"2024-03-06T07:26:24.662946+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:40.032388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662940+0000","last_change":"2024-03-06T07:26:17.637044+0000","last_active":"2024-03-06T07:26:24.662940+0000","last_peered":"2024-03-06T07:26:24.662940+0000","last_clean":"2024-03-06T07:26:24.662940+0000","last_became_active":"2024-03-06T07:26:17.636771+0000","last_became_peered":"2024-03-06T07:26:17.636771+0000","last_unstale":"2024-03-06T07:26:24.662940+0000","last_undegraded":"2024-03-06T07:26:24.662940+0000","last_fullsized":"2024-03-06T07:26:24.662940+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:29.354563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658522+0000","last_change":"2024-03-06T07:26:17.636369+0000","last_active":"2024-03-06T07:26:24.658522+0000","last_peered":"2024-03-06T07:26:24.658522+0000","last_clean":"2024-03-06T07:26:24.658522+0000","last_became_active":"2024-03-06T07:26:17.636090+0000","last_became_peered":"2024-03-06T07:26:17.636090+0000","last_unstale":"2024-03-06T07:26:24.658522+0000","last_undegraded":"2024-03-06T07:26:24.658522+0000","last_fullsized":"2024-03-06T07:26:24.658522+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:45.110614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664860+0000","last_change":"2024-03-06T07:26:17.633006+0000","last_active":"2024-03-06T07:26:24.664860+0000","last_peered":"2024-03-06T07:26:24.664860+0000","last_clean":"2024-03-06T07:26:24.664860+0000","last_became_active":"2024-03-06T07:26:17.632523+0000","last_became_peered":"2024-03-06T07:26:17.632523+0000","last_unstale":"2024-03-06T07:26:24.664860+0000","last_undegraded":"2024-03-06T07:26:24.664860+0000","last_fullsized":"2024-03-06T07:26:24.664860+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:41.655929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662348+0000","last_change":"2024-03-06T07:26:17.638448+0000","last_active":"2024-03-06T07:26:24.662348+0000","last_peered":"2024-03-06T07:26:24.662348+0000","last_clean":"2024-03-06T07:26:24.662348+0000","last_became_active":"2024-03-06T07:26:17.638188+0000","last_became_peered":"2024-03-06T07:26:17.638188+0000","last_unstale":"2024-03-06T07:26:24.662348+0000","last_undegraded":"2024-03-06T07:26:24.662348+0000","last_fullsized":"2024-03-06T07:26:24.662348+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:15.455961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666402+0000","last_change":"2024-03-06T07:26:17.634053+0000","last_active":"2024-03-06T07:26:24.666402+0000","last_peered":"2024-03-06T07:26:24.666402+0000","last_clean":"2024-03-06T07:26:24.666402+0000","last_became_active":"2024-03-06T07:26:17.633451+0000","last_became_peered":"2024-03-06T07:26:17.633451+0000","last_unstale":"2024-03-06T07:26:24.666402+0000","last_undegraded":"2024-03-06T07:26:24.666402+0000","last_fullsized":"2024-03-06T07:26:24.666402+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:25.173456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665705+0000","last_change":"2024-03-06T07:26:17.627891+0000","last_active":"2024-03-06T07:26:24.665705+0000","last_peered":"2024-03-06T07:26:24.665705+0000","last_clean":"2024-03-06T07:26:24.665705+0000","last_became_active":"2024-03-06T07:26:17.627624+0000","last_became_peered":"2024-03-06T07:26:17.627624+0000","last_unstale":"2024-03-06T07:26:24.665705+0000","last_undegraded":"2024-03-06T07:26:24.665705+0000","last_fullsized":"2024-03-06T07:26:24.665705+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:56.476412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666557+0000","last_change":"2024-03-06T07:26:17.635467+0000","last_active":"2024-03-06T07:26:24.666557+0000","last_peered":"2024-03-06T07:26:24.666557+0000","last_clean":"2024-03-06T07:26:24.666557+0000","last_became_active":"2024-03-06T07:26:17.635125+0000","last_became_peered":"2024-03-06T07:26:17.635125+0000","last_unstale":"2024-03-06T07:26:24.666557+0000","last_undegraded":"2024-03-06T07:26:24.666557+0000","last_fullsized":"2024-03-06T07:26:24.666557+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:39.829929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665165+0000","last_change":"2024-03-06T07:26:17.638944+0000","last_active":"2024-03-06T07:26:24.665165+0000","last_peered":"2024-03-06T07:26:24.665165+0000","last_clean":"2024-03-06T07:26:24.665165+0000","last_became_active":"2024-03-06T07:26:17.638649+0000","last_became_peered":"2024-03-06T07:26:17.638649+0000","last_unstale":"2024-03-06T07:26:24.665165+0000","last_undegraded":"2024-03-06T07:26:24.665165+0000","last_fullsized":"2024-03-06T07:26:24.665165+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:17.394090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665307+0000","last_change":"2024-03-06T07:26:24.665307+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665307+0000","last_undegraded":"2024-03-06T07:26:24.665307+0000","last_fullsized":"2024-03-06T07:26:24.665307+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664442+0000","last_change":"2024-03-06T07:26:17.635968+0000","last_active":"2024-03-06T07:26:24.664442+0000","last_peered":"2024-03-06T07:26:24.664442+0000","last_clean":"2024-03-06T07:26:24.664442+0000","last_became_active":"2024-03-06T07:26:17.635095+0000","last_became_peered":"2024-03-06T07:26:17.635095+0000","last_unstale":"2024-03-06T07:26:24.664442+0000","last_undegraded":"2024-03-06T07:26:24.664442+0000","last_fullsized":"2024-03-06T07:26:24.664442+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:26.879181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"148'4","reported_seq":20,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663882+0000","last_change":"2024-03-06T07:26:16.626240+0000","last_active":"2024-03-06T07:26:24.663882+0000","last_peered":"2024-03-06T07:26:24.663882+0000","last_clean":"2024-03-06T07:26:24.663882+0000","last_became_active":"2024-03-06T07:26:16.626011+0000","last_became_peered":"2024-03-06T07:26:16.626011+0000","last_unstale":"2024-03-06T07:26:24.663882+0000","last_undegraded":"2024-03-06T07:26:24.663882+0000","last_fullsized":"2024-03-06T07:26:24.663882+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:30.259380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665984+0000","last_change":"2024-03-06T07:26:24.665984+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665984+0000","last_undegraded":"2024-03-06T07:26:24.665984+0000","last_fullsized":"2024-03-06T07:26:24.665984+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662038+0000","last_change":"2024-03-06T07:26:17.627512+0000","last_active":"2024-03-06T07:26:24.662038+0000","last_peered":"2024-03-06T07:26:24.662038+0000","last_clean":"2024-03-06T07:26:24.662038+0000","last_became_active":"2024-03-06T07:26:17.627235+0000","last_became_peered":"2024-03-06T07:26:17.627235+0000","last_unstale":"2024-03-06T07:26:24.662038+0000","last_undegraded":"2024-03-06T07:26:24.662038+0000","last_fullsized":"2024-03-06T07:26:24.662038+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:19.245848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"148'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668199+0000","last_change":"2024-03-06T07:26:16.625703+0000","last_active":"2024-03-06T07:26:24.668199+0000","last_peered":"2024-03-06T07:26:24.668199+0000","last_clean":"2024-03-06T07:26:24.668199+0000","last_became_active":"2024-03-06T07:26:16.625426+0000","last_became_peered":"2024-03-06T07:26:16.625426+0000","last_unstale":"2024-03-06T07:26:24.668199+0000","last_undegraded":"2024-03-06T07:26:24.668199+0000","last_fullsized":"2024-03-06T07:26:24.668199+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:39.946462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.662178+0000","last_change":"2024-03-06T07:26:24.662178+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.662178+0000","last_undegraded":"2024-03-06T07:26:24.662178+0000","last_fullsized":"2024-03-06T07:26:24.662178+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667846+0000","last_change":"2024-03-06T07:26:17.632107+0000","last_active":"2024-03-06T07:26:24.667846+0000","last_peered":"2024-03-06T07:26:24.667846+0000","last_clean":"2024-03-06T07:26:24.667846+0000","last_became_active":"2024-03-06T07:26:17.631481+0000","last_became_peered":"2024-03-06T07:26:17.631481+0000","last_unstale":"2024-03-06T07:26:24.667846+0000","last_undegraded":"2024-03-06T07:26:24.667846+0000","last_fullsized":"2024-03-06T07:26:24.667846+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:20.662554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667577+0000","last_change":"2024-03-06T07:26:16.620199+0000","last_active":"2024-03-06T07:26:24.667577+0000","last_peered":"2024-03-06T07:26:24.667577+0000","last_clean":"2024-03-06T07:26:24.667577+0000","last_became_active":"2024-03-06T07:26:16.619877+0000","last_became_peered":"2024-03-06T07:26:16.619877+0000","last_unstale":"2024-03-06T07:26:24.667577+0000","last_undegraded":"2024-03-06T07:26:24.667577+0000","last_fullsized":"2024-03-06T07:26:24.667577+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:14.648017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.663658+0000","last_change":"2024-03-06T07:26:24.663658+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.663658+0000","last_undegraded":"2024-03-06T07:26:24.663658+0000","last_fullsized":"2024-03-06T07:26:24.663658+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666492+0000","last_change":"2024-03-06T07:26:17.634295+0000","last_active":"2024-03-06T07:26:24.666492+0000","last_peered":"2024-03-06T07:26:24.666492+0000","last_clean":"2024-03-06T07:26:24.666492+0000","last_became_active":"2024-03-06T07:26:17.633744+0000","last_became_peered":"2024-03-06T07:26:17.633744+0000","last_unstale":"2024-03-06T07:26:24.666492+0000","last_undegraded":"2024-03-06T07:26:24.666492+0000","last_fullsized":"2024-03-06T07:26:24.666492+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:14.404340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658789+0000","last_change":"2024-03-06T07:26:16.616715+0000","last_active":"2024-03-06T07:26:24.658789+0000","last_peered":"2024-03-06T07:26:24.658789+0000","last_clean":"2024-03-06T07:26:24.658789+0000","last_became_active":"2024-03-06T07:26:16.616466+0000","last_became_peered":"2024-03-06T07:26:16.616466+0000","last_unstale":"2024-03-06T07:26:24.658789+0000","last_undegraded":"2024-03-06T07:26:24.658789+0000","last_fullsized":"2024-03-06T07:26:24.658789+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:54.279225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665856+0000","last_change":"2024-03-06T07:26:24.665856+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665856+0000","last_undegraded":"2024-03-06T07:26:24.665856+0000","last_fullsized":"2024-03-06T07:26:24.665856+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665092+0000","last_change":"2024-03-06T07:26:17.631966+0000","last_active":"2024-03-06T07:26:24.665092+0000","last_peered":"2024-03-06T07:26:24.665092+0000","last_clean":"2024-03-06T07:26:24.665092+0000","last_became_active":"2024-03-06T07:26:17.631673+0000","last_became_peered":"2024-03-06T07:26:17.631673+0000","last_unstale":"2024-03-06T07:26:24.665092+0000","last_undegraded":"2024-03-06T07:26:24.665092+0000","last_fullsized":"2024-03-06T07:26:24.665092+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:26.879181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"148'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664861+0000","last_change":"2024-03-06T07:26:16.619602+0000","last_active":"2024-03-06T07:26:24.664861+0000","last_peered":"2024-03-06T07:26:24.664861+0000","last_clean":"2024-03-06T07:26:24.664861+0000","last_became_active":"2024-03-06T07:26:16.619282+0000","last_became_peered":"2024-03-06T07:26:16.619282+0000","last_unstale":"2024-03-06T07:26:24.664861+0000","last_undegraded":"2024-03-06T07:26:24.664861+0000","last_fullsized":"2024-03-06T07:26:24.664861+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:59.587645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.662223+0000","last_change":"2024-03-06T07:26:24.662223+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.662223+0000","last_undegraded":"2024-03-06T07:26:24.662223+0000","last_fullsized":"2024-03-06T07:26:24.662223+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658427+0000","last_change":"2024-03-06T07:26:17.628669+0000","last_active":"2024-03-06T07:26:24.658427+0000","last_peered":"2024-03-06T07:26:24.658427+0000","last_clean":"2024-03-06T07:26:24.658427+0000","last_became_active":"2024-03-06T07:26:17.628371+0000","last_became_peered":"2024-03-06T07:26:17.628371+0000","last_unstale":"2024-03-06T07:26:24.658427+0000","last_undegraded":"2024-03-06T07:26:24.658427+0000","last_fullsized":"2024-03-06T07:26:24.658427+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:16.499694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"148'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668006+0000","last_change":"2024-03-06T07:26:16.615912+0000","last_active":"2024-03-06T07:26:24.668006+0000","last_peered":"2024-03-06T07:26:24.668006+0000","last_clean":"2024-03-06T07:26:24.668006+0000","last_became_active":"2024-03-06T07:26:16.615625+0000","last_became_peered":"2024-03-06T07:26:16.615625+0000","last_unstale":"2024-03-06T07:26:24.668006+0000","last_undegraded":"2024-03-06T07:26:24.668006+0000","last_fullsized":"2024-03-06T07:26:24.668006+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:24.591774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661284+0000","last_change":"2024-03-06T07:26:24.661284+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661284+0000","last_undegraded":"2024-03-06T07:26:24.661284+0000","last_fullsized":"2024-03-06T07:26:24.661284+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658604+0000","last_change":"2024-03-06T07:26:17.626866+0000","last_active":"2024-03-06T07:26:24.658604+0000","last_peered":"2024-03-06T07:26:24.658604+0000","last_clean":"2024-03-06T07:26:24.658604+0000","last_became_active":"2024-03-06T07:26:17.626586+0000","last_became_peered":"2024-03-06T07:26:17.626586+0000","last_unstale":"2024-03-06T07:26:24.658604+0000","last_undegraded":"2024-03-06T07:26:24.658604+0000","last_fullsized":"2024-03-06T07:26:24.658604+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:34.182149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658031+0000","last_change":"2024-03-06T07:26:16.622797+0000","last_active":"2024-03-06T07:26:24.658031+0000","last_peered":"2024-03-06T07:26:24.658031+0000","last_clean":"2024-03-06T07:26:24.658031+0000","last_became_active":"2024-03-06T07:26:16.622534+0000","last_became_peered":"2024-03-06T07:26:16.622534+0000","last_unstale":"2024-03-06T07:26:24.658031+0000","last_undegraded":"2024-03-06T07:26:24.658031+0000","last_fullsized":"2024-03-06T07:26:24.658031+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:39.697277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.663648+0000","last_change":"2024-03-06T07:26:24.663648+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.663648+0000","last_undegraded":"2024-03-06T07:26:24.663648+0000","last_fullsized":"2024-03-06T07:26:24.663648+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663807+0000","last_change":"2024-03-06T07:26:17.632829+0000","last_active":"2024-03-06T07:26:24.663807+0000","last_peered":"2024-03-06T07:26:24.663807+0000","last_clean":"2024-03-06T07:26:24.663807+0000","last_became_active":"2024-03-06T07:26:17.631051+0000","last_became_peered":"2024-03-06T07:26:17.631051+0000","last_unstale":"2024-03-06T07:26:24.663807+0000","last_undegraded":"2024-03-06T07:26:24.663807+0000","last_fullsized":"2024-03-06T07:26:24.663807+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:09.224572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664647+0000","last_change":"2024-03-06T07:26:16.618575+0000","last_active":"2024-03-06T07:26:24.664647+0000","last_peered":"2024-03-06T07:26:24.664647+0000","last_clean":"2024-03-06T07:26:24.664647+0000","last_became_active":"2024-03-06T07:26:16.618143+0000","last_became_peered":"2024-03-06T07:26:16.618143+0000","last_unstale":"2024-03-06T07:26:24.664647+0000","last_undegraded":"2024-03-06T07:26:24.664647+0000","last_fullsized":"2024-03-06T07:26:24.664647+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:48.709582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661586+0000","last_change":"2024-03-06T07:26:24.661586+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661586+0000","last_undegraded":"2024-03-06T07:26:24.661586+0000","last_fullsized":"2024-03-06T07:26:24.661586+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664401+0000","last_change":"2024-03-06T07:26:17.636599+0000","last_active":"2024-03-06T07:26:24.664401+0000","last_peered":"2024-03-06T07:26:24.664401+0000","last_clean":"2024-03-06T07:26:24.664401+0000","last_became_active":"2024-03-06T07:26:17.635451+0000","last_became_peered":"2024-03-06T07:26:17.635451+0000","last_unstale":"2024-03-06T07:26:24.664401+0000","last_undegraded":"2024-03-06T07:26:24.664401+0000","last_fullsized":"2024-03-06T07:26:24.664401+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:27.398343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662893+0000","last_change":"2024-03-06T07:26:16.627460+0000","last_active":"2024-03-06T07:26:24.662893+0000","last_peered":"2024-03-06T07:26:24.662893+0000","last_clean":"2024-03-06T07:26:24.662893+0000","last_became_active":"2024-03-06T07:26:16.626345+0000","last_became_peered":"2024-03-06T07:26:16.626345+0000","last_unstale":"2024-03-06T07:26:24.662893+0000","last_undegraded":"2024-03-06T07:26:24.662893+0000","last_fullsized":"2024-03-06T07:26:24.662893+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:24.257795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661596+0000","last_change":"2024-03-06T07:26:24.661596+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661596+0000","last_undegraded":"2024-03-06T07:26:24.661596+0000","last_fullsized":"2024-03-06T07:26:24.661596+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662215+0000","last_change":"2024-03-06T07:26:17.634986+0000","last_active":"2024-03-06T07:26:24.662215+0000","last_peered":"2024-03-06T07:26:24.662215+0000","last_clean":"2024-03-06T07:26:24.662215+0000","last_became_active":"2024-03-06T07:26:17.634680+0000","last_became_peered":"2024-03-06T07:26:17.634680+0000","last_unstale":"2024-03-06T07:26:24.662215+0000","last_undegraded":"2024-03-06T07:26:24.662215+0000","last_fullsized":"2024-03-06T07:26:24.662215+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:48.372843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"148'3","reported_seq":19,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664051+0000","last_change":"2024-03-06T07:26:16.623947+0000","last_active":"2024-03-06T07:26:24.664051+0000","last_peered":"2024-03-06T07:26:24.664051+0000","last_clean":"2024-03-06T07:26:24.664051+0000","last_became_active":"2024-03-06T07:26:16.623662+0000","last_became_peered":"2024-03-06T07:26:16.623662+0000","last_unstale":"2024-03-06T07:26:24.664051+0000","last_undegraded":"2024-03-06T07:26:24.664051+0000","last_fullsized":"2024-03-06T07:26:24.664051+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:00.411414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.663853+0000","last_change":"2024-03-06T07:26:24.663853+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.663853+0000","last_undegraded":"2024-03-06T07:26:24.663853+0000","last_fullsized":"2024-03-06T07:26:24.663853+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658162+0000","last_change":"2024-03-06T07:26:17.626440+0000","last_active":"2024-03-06T07:26:24.658162+0000","last_peered":"2024-03-06T07:26:24.658162+0000","last_clean":"2024-03-06T07:26:24.658162+0000","last_became_active":"2024-03-06T07:26:17.626149+0000","last_became_peered":"2024-03-06T07:26:17.626149+0000","last_unstale":"2024-03-06T07:26:24.658162+0000","last_undegraded":"2024-03-06T07:26:24.658162+0000","last_fullsized":"2024-03-06T07:26:24.658162+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:46.191755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"148'6","reported_seq":22,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662564+0000","last_change":"2024-03-06T07:26:16.627536+0000","last_active":"2024-03-06T07:26:24.662564+0000","last_peered":"2024-03-06T07:26:24.662564+0000","last_clean":"2024-03-06T07:26:24.662564+0000","last_became_active":"2024-03-06T07:26:16.627229+0000","last_became_peered":"2024-03-06T07:26:16.627229+0000","last_unstale":"2024-03-06T07:26:24.662564+0000","last_undegraded":"2024-03-06T07:26:24.662564+0000","last_fullsized":"2024-03-06T07:26:24.662564+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:00.825243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665684+0000","last_change":"2024-03-06T07:26:24.665684+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665684+0000","last_undegraded":"2024-03-06T07:26:24.665684+0000","last_fullsized":"2024-03-06T07:26:24.665684+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665948+0000","last_change":"2024-03-06T07:26:17.638945+0000","last_active":"2024-03-06T07:26:24.665948+0000","last_peered":"2024-03-06T07:26:24.665948+0000","last_clean":"2024-03-06T07:26:24.665948+0000","last_became_active":"2024-03-06T07:26:17.638664+0000","last_became_peered":"2024-03-06T07:26:17.638664+0000","last_unstale":"2024-03-06T07:26:24.665948+0000","last_undegraded":"2024-03-06T07:26:24.665948+0000","last_fullsized":"2024-03-06T07:26:24.665948+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:15.414267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668359+0000","last_change":"2024-03-06T07:26:16.625884+0000","last_active":"2024-03-06T07:26:24.668359+0000","last_peered":"2024-03-06T07:26:24.668359+0000","last_clean":"2024-03-06T07:26:24.668359+0000","last_became_active":"2024-03-06T07:26:16.625605+0000","last_became_peered":"2024-03-06T07:26:16.625605+0000","last_unstale":"2024-03-06T07:26:24.668359+0000","last_undegraded":"2024-03-06T07:26:24.668359+0000","last_fullsized":"2024-03-06T07:26:24.668359+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:13.400206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661600+0000","last_change":"2024-03-06T07:26:24.661600+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661600+0000","last_undegraded":"2024-03-06T07:26:24.661600+0000","last_fullsized":"2024-03-06T07:26:24.661600+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664941+0000","last_change":"2024-03-06T07:26:17.631376+0000","last_active":"2024-03-06T07:26:24.664941+0000","last_peered":"2024-03-06T07:26:24.664941+0000","last_clean":"2024-03-06T07:26:24.664941+0000","last_became_active":"2024-03-06T07:26:17.631068+0000","last_became_peered":"2024-03-06T07:26:17.631068+0000","last_unstale":"2024-03-06T07:26:24.664941+0000","last_undegraded":"2024-03-06T07:26:24.664941+0000","last_fullsized":"2024-03-06T07:26:24.664941+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:46.783070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662415+0000","last_change":"2024-03-06T07:26:16.625352+0000","last_active":"2024-03-06T07:26:24.662415+0000","last_peered":"2024-03-06T07:26:24.662415+0000","last_clean":"2024-03-06T07:26:24.662415+0000","last_became_active":"2024-03-06T07:26:16.624893+0000","last_became_peered":"2024-03-06T07:26:16.624893+0000","last_unstale":"2024-03-06T07:26:24.662415+0000","last_undegraded":"2024-03-06T07:26:24.662415+0000","last_fullsized":"2024-03-06T07:26:24.662415+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:29.596513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.667139+0000","last_change":"2024-03-06T07:26:24.667139+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.667139+0000","last_undegraded":"2024-03-06T07:26:24.667139+0000","last_fullsized":"2024-03-06T07:26:24.667139+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667199+0000","last_change":"2024-03-06T07:26:17.639031+0000","last_active":"2024-03-06T07:26:24.667199+0000","last_peered":"2024-03-06T07:26:24.667199+0000","last_clean":"2024-03-06T07:26:24.667199+0000","last_became_active":"2024-03-06T07:26:17.638716+0000","last_became_peered":"2024-03-06T07:26:17.638716+0000","last_unstale":"2024-03-06T07:26:24.667199+0000","last_undegraded":"2024-03-06T07:26:24.667199+0000","last_fullsized":"2024-03-06T07:26:24.667199+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:46.783070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666494+0000","last_change":"2024-03-06T07:26:16.620528+0000","last_active":"2024-03-06T07:26:24.666494+0000","last_peered":"2024-03-06T07:26:24.666494+0000","last_clean":"2024-03-06T07:26:24.666494+0000","last_became_active":"2024-03-06T07:26:16.620245+0000","last_became_peered":"2024-03-06T07:26:16.620245+0000","last_unstale":"2024-03-06T07:26:24.666494+0000","last_undegraded":"2024-03-06T07:26:24.666494+0000","last_fullsized":"2024-03-06T07:26:24.666494+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:39.028254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661458+0000","last_change":"2024-03-06T07:26:24.661458+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661458+0000","last_undegraded":"2024-03-06T07:26:24.661458+0000","last_fullsized":"2024-03-06T07:26:24.661458+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658283+0000","last_change":"2024-03-06T07:26:17.641468+0000","last_active":"2024-03-06T07:26:24.658283+0000","last_peered":"2024-03-06T07:26:24.658283+0000","last_clean":"2024-03-06T07:26:24.658283+0000","last_became_active":"2024-03-06T07:26:17.641155+0000","last_became_peered":"2024-03-06T07:26:17.641155+0000","last_unstale":"2024-03-06T07:26:24.658283+0000","last_undegraded":"2024-03-06T07:26:24.658283+0000","last_fullsized":"2024-03-06T07:26:24.658283+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:27.464355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"148'8","reported_seq":24,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664388+0000","last_change":"2024-03-06T07:26:16.620548+0000","last_active":"2024-03-06T07:26:24.664388+0000","last_peered":"2024-03-06T07:26:24.664388+0000","last_clean":"2024-03-06T07:26:24.664388+0000","last_became_active":"2024-03-06T07:26:16.620289+0000","last_became_peered":"2024-03-06T07:26:16.620289+0000","last_unstale":"2024-03-06T07:26:24.664388+0000","last_undegraded":"2024-03-06T07:26:24.664388+0000","last_fullsized":"2024-03-06T07:26:24.664388+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:08.155799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.660981+0000","last_change":"2024-03-06T07:26:24.660981+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.660981+0000","last_undegraded":"2024-03-06T07:26:24.660981+0000","last_fullsized":"2024-03-06T07:26:24.660981+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664277+0000","last_change":"2024-03-06T07:26:17.629187+0000","last_active":"2024-03-06T07:26:24.664277+0000","last_peered":"2024-03-06T07:26:24.664277+0000","last_clean":"2024-03-06T07:26:24.664277+0000","last_became_active":"2024-03-06T07:26:17.628903+0000","last_became_peered":"2024-03-06T07:26:17.628903+0000","last_unstale":"2024-03-06T07:26:24.664277+0000","last_undegraded":"2024-03-06T07:26:24.664277+0000","last_fullsized":"2024-03-06T07:26:24.664277+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:25.173456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"148'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662984+0000","last_change":"2024-03-06T07:26:16.617108+0000","last_active":"2024-03-06T07:26:24.662984+0000","last_peered":"2024-03-06T07:26:24.662984+0000","last_clean":"2024-03-06T07:26:24.662984+0000","last_became_active":"2024-03-06T07:26:16.616776+0000","last_became_peered":"2024-03-06T07:26:16.616776+0000","last_unstale":"2024-03-06T07:26:24.662984+0000","last_undegraded":"2024-03-06T07:26:24.662984+0000","last_fullsized":"2024-03-06T07:26:24.662984+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:01.223054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.664369+0000","last_change":"2024-03-06T07:26:24.664369+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.664369+0000","last_undegraded":"2024-03-06T07:26:24.664369+0000","last_fullsized":"2024-03-06T07:26:24.664369+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665818+0000","last_change":"2024-03-06T07:26:17.634792+0000","last_active":"2024-03-06T07:26:24.665818+0000","last_peered":"2024-03-06T07:26:24.665818+0000","last_clean":"2024-03-06T07:26:24.665818+0000","last_became_active":"2024-03-06T07:26:17.634367+0000","last_became_peered":"2024-03-06T07:26:17.634367+0000","last_unstale":"2024-03-06T07:26:24.665818+0000","last_undegraded":"2024-03-06T07:26:24.665818+0000","last_fullsized":"2024-03-06T07:26:24.665818+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:11.386543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665495+0000","last_change":"2024-03-06T07:26:16.619012+0000","last_active":"2024-03-06T07:26:24.665495+0000","last_peered":"2024-03-06T07:26:24.665495+0000","last_clean":"2024-03-06T07:26:24.665495+0000","last_became_active":"2024-03-06T07:26:16.617718+0000","last_became_peered":"2024-03-06T07:26:16.617718+0000","last_unstale":"2024-03-06T07:26:24.665495+0000","last_undegraded":"2024-03-06T07:26:24.665495+0000","last_fullsized":"2024-03-06T07:26:24.665495+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:08.220437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.666049+0000","last_change":"2024-03-06T07:26:24.666049+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.666049+0000","last_undegraded":"2024-03-06T07:26:24.666049+0000","last_fullsized":"2024-03-06T07:26:24.666049+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666499+0000","last_change":"2024-03-06T07:26:17.633394+0000","last_active":"2024-03-06T07:26:24.666499+0000","last_peered":"2024-03-06T07:26:24.666499+0000","last_clean":"2024-03-06T07:26:24.666499+0000","last_became_active":"2024-03-06T07:26:17.632438+0000","last_became_peered":"2024-03-06T07:26:17.632438+0000","last_unstale":"2024-03-06T07:26:24.666499+0000","last_undegraded":"2024-03-06T07:26:24.666499+0000","last_fullsized":"2024-03-06T07:26:24.666499+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:12.024002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"148'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668243+0000","last_change":"2024-03-06T07:26:16.628646+0000","last_active":"2024-03-06T07:26:24.668243+0000","last_peered":"2024-03-06T07:26:24.668243+0000","last_clean":"2024-03-06T07:26:24.668243+0000","last_became_active":"2024-03-06T07:26:16.627947+0000","last_became_peered":"2024-03-06T07:26:16.627947+0000","last_unstale":"2024-03-06T07:26:24.668243+0000","last_undegraded":"2024-03-06T07:26:24.668243+0000","last_fullsized":"2024-03-06T07:26:24.668243+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:25.875046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665349+0000","last_change":"2024-03-06T07:26:24.665349+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665349+0000","last_undegraded":"2024-03-06T07:26:24.665349+0000","last_fullsized":"2024-03-06T07:26:24.665349+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666918+0000","last_change":"2024-03-06T07:26:17.634254+0000","last_active":"2024-03-06T07:26:24.666918+0000","last_peered":"2024-03-06T07:26:24.666918+0000","last_clean":"2024-03-06T07:26:24.666918+0000","last_became_active":"2024-03-06T07:26:17.633961+0000","last_became_peered":"2024-03-06T07:26:17.633961+0000","last_unstale":"2024-03-06T07:26:24.666918+0000","last_undegraded":"2024-03-06T07:26:24.666918+0000","last_fullsized":"2024-03-06T07:26:24.666918+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:59.873611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665244+0000","last_change":"2024-03-06T07:26:16.619532+0000","last_active":"2024-03-06T07:26:24.665244+0000","last_peered":"2024-03-06T07:26:24.665244+0000","last_clean":"2024-03-06T07:26:24.665244+0000","last_became_active":"2024-03-06T07:26:16.619261+0000","last_became_peered":"2024-03-06T07:26:16.619261+0000","last_unstale":"2024-03-06T07:26:24.665244+0000","last_undegraded":"2024-03-06T07:26:24.665244+0000","last_fullsized":"2024-03-06T07:26:24.665244+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:34.031068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661217+0000","last_change":"2024-03-06T07:26:24.661217+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661217+0000","last_undegraded":"2024-03-06T07:26:24.661217+0000","last_fullsized":"2024-03-06T07:26:24.661217+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662447+0000","last_change":"2024-03-06T07:26:17.637879+0000","last_active":"2024-03-06T07:26:24.662447+0000","last_peered":"2024-03-06T07:26:24.662447+0000","last_clean":"2024-03-06T07:26:24.662447+0000","last_became_active":"2024-03-06T07:26:17.637094+0000","last_became_peered":"2024-03-06T07:26:17.637094+0000","last_unstale":"2024-03-06T07:26:24.662447+0000","last_undegraded":"2024-03-06T07:26:24.662447+0000","last_fullsized":"2024-03-06T07:26:24.662447+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:34.084422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668442+0000","last_change":"2024-03-06T07:26:16.622856+0000","last_active":"2024-03-06T07:26:24.668442+0000","last_peered":"2024-03-06T07:26:24.668442+0000","last_clean":"2024-03-06T07:26:24.668442+0000","last_became_active":"2024-03-06T07:26:16.622421+0000","last_became_peered":"2024-03-06T07:26:16.622421+0000","last_unstale":"2024-03-06T07:26:24.668442+0000","last_undegraded":"2024-03-06T07:26:24.668442+0000","last_fullsized":"2024-03-06T07:26:24.668442+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:40.651795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661343+0000","last_change":"2024-03-06T07:26:24.661343+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661343+0000","last_undegraded":"2024-03-06T07:26:24.661343+0000","last_fullsized":"2024-03-06T07:26:24.661343+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668215+0000","last_change":"2024-03-06T07:26:17.629188+0000","last_active":"2024-03-06T07:26:24.668215+0000","last_peered":"2024-03-06T07:26:24.668215+0000","last_clean":"2024-03-06T07:26:24.668215+0000","last_became_active":"2024-03-06T07:26:17.628881+0000","last_became_peered":"2024-03-06T07:26:17.628881+0000","last_unstale":"2024-03-06T07:26:24.668215+0000","last_undegraded":"2024-03-06T07:26:24.668215+0000","last_fullsized":"2024-03-06T07:26:24.668215+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:44.957682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666547+0000","last_change":"2024-03-06T07:26:16.625755+0000","last_active":"2024-03-06T07:26:24.666547+0000","last_peered":"2024-03-06T07:26:24.666547+0000","last_clean":"2024-03-06T07:26:24.666547+0000","last_became_active":"2024-03-06T07:26:16.625489+0000","last_became_peered":"2024-03-06T07:26:16.625489+0000","last_unstale":"2024-03-06T07:26:24.666547+0000","last_undegraded":"2024-03-06T07:26:24.666547+0000","last_fullsized":"2024-03-06T07:26:24.666547+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:03.053080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.666734+0000","last_change":"2024-03-06T07:26:24.666734+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.666734+0000","last_undegraded":"2024-03-06T07:26:24.666734+0000","last_fullsized":"2024-03-06T07:26:24.666734+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668192+0000","last_change":"2024-03-06T07:26:17.629894+0000","last_active":"2024-03-06T07:26:24.668192+0000","last_peered":"2024-03-06T07:26:24.668192+0000","last_clean":"2024-03-06T07:26:24.668192+0000","last_became_active":"2024-03-06T07:26:17.629624+0000","last_became_peered":"2024-03-06T07:26:17.629624+0000","last_unstale":"2024-03-06T07:26:24.668192+0000","last_undegraded":"2024-03-06T07:26:24.668192+0000","last_fullsized":"2024-03-06T07:26:24.668192+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:48.520981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658129+0000","last_change":"2024-03-06T07:26:16.618167+0000","last_active":"2024-03-06T07:26:24.658129+0000","last_peered":"2024-03-06T07:26:24.658129+0000","last_clean":"2024-03-06T07:26:24.658129+0000","last_became_active":"2024-03-06T07:26:16.617893+0000","last_became_peered":"2024-03-06T07:26:16.617893+0000","last_unstale":"2024-03-06T07:26:24.658129+0000","last_undegraded":"2024-03-06T07:26:24.658129+0000","last_fullsized":"2024-03-06T07:26:24.658129+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:33.738360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.660975+0000","last_change":"2024-03-06T07:26:24.660975+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.660975+0000","last_undegraded":"2024-03-06T07:26:24.660975+0000","last_fullsized":"2024-03-06T07:26:24.660975+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664802+0000","last_change":"2024-03-06T07:26:17.632248+0000","last_active":"2024-03-06T07:26:24.664802+0000","last_peered":"2024-03-06T07:26:24.664802+0000","last_clean":"2024-03-06T07:26:24.664802+0000","last_became_active":"2024-03-06T07:26:17.631912+0000","last_became_peered":"2024-03-06T07:26:17.631912+0000","last_unstale":"2024-03-06T07:26:24.664802+0000","last_undegraded":"2024-03-06T07:26:24.664802+0000","last_fullsized":"2024-03-06T07:26:24.664802+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:29.113308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664561+0000","last_change":"2024-03-06T07:26:16.616949+0000","last_active":"2024-03-06T07:26:24.664561+0000","last_peered":"2024-03-06T07:26:24.664561+0000","last_clean":"2024-03-06T07:26:24.664561+0000","last_became_active":"2024-03-06T07:26:16.616670+0000","last_became_peered":"2024-03-06T07:26:16.616670+0000","last_unstale":"2024-03-06T07:26:24.664561+0000","last_undegraded":"2024-03-06T07:26:24.664561+0000","last_fullsized":"2024-03-06T07:26:24.664561+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:59.587645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661489+0000","last_change":"2024-03-06T07:26:24.661489+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661489+0000","last_undegraded":"2024-03-06T07:26:24.661489+0000","last_fullsized":"2024-03-06T07:26:24.661489+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666760+0000","last_change":"2024-03-06T07:26:17.628994+0000","last_active":"2024-03-06T07:26:24.666760+0000","last_peered":"2024-03-06T07:26:24.666760+0000","last_clean":"2024-03-06T07:26:24.666760+0000","last_became_active":"2024-03-06T07:26:17.628722+0000","last_became_peered":"2024-03-06T07:26:17.628722+0000","last_unstale":"2024-03-06T07:26:24.666760+0000","last_undegraded":"2024-03-06T07:26:24.666760+0000","last_fullsized":"2024-03-06T07:26:24.666760+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:43.090011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668358+0000","last_change":"2024-03-06T07:26:16.620011+0000","last_active":"2024-03-06T07:26:24.668358+0000","last_peered":"2024-03-06T07:26:24.668358+0000","last_clean":"2024-03-06T07:26:24.668358+0000","last_became_active":"2024-03-06T07:26:16.619736+0000","last_became_peered":"2024-03-06T07:26:16.619736+0000","last_unstale":"2024-03-06T07:26:24.668358+0000","last_undegraded":"2024-03-06T07:26:24.668358+0000","last_fullsized":"2024-03-06T07:26:24.668358+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:24.169321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.663886+0000","last_change":"2024-03-06T07:26:24.663886+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.663886+0000","last_undegraded":"2024-03-06T07:26:24.663886+0000","last_fullsized":"2024-03-06T07:26:24.663886+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664813+0000","last_change":"2024-03-06T07:26:17.628485+0000","last_active":"2024-03-06T07:26:24.664813+0000","last_peered":"2024-03-06T07:26:24.664813+0000","last_clean":"2024-03-06T07:26:24.664813+0000","last_became_active":"2024-03-06T07:26:17.628204+0000","last_became_peered":"2024-03-06T07:26:17.628204+0000","last_unstale":"2024-03-06T07:26:24.664813+0000","last_undegraded":"2024-03-06T07:26:24.664813+0000","last_fullsized":"2024-03-06T07:26:24.664813+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:56.134884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"149'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662755+0000","last_change":"2024-03-06T07:26:16.627066+0000","last_active":"2024-03-06T07:26:24.662755+0000","last_peered":"2024-03-06T07:26:24.662755+0000","last_clean":"2024-03-06T07:26:24.662755+0000","last_became_active":"2024-03-06T07:26:16.626418+0000","last_became_peered":"2024-03-06T07:26:16.626418+0000","last_unstale":"2024-03-06T07:26:24.662755+0000","last_undegraded":"2024-03-06T07:26:24.662755+0000","last_fullsized":"2024-03-06T07:26:24.662755+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:37.036435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661600+0000","last_change":"2024-03-06T07:26:24.661600+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661600+0000","last_undegraded":"2024-03-06T07:26:24.661600+0000","last_fullsized":"2024-03-06T07:26:24.661600+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658129+0000","last_change":"2024-03-06T07:26:17.626185+0000","last_active":"2024-03-06T07:26:24.658129+0000","last_peered":"2024-03-06T07:26:24.658129+0000","last_clean":"2024-03-06T07:26:24.658129+0000","last_became_active":"2024-03-06T07:26:17.625895+0000","last_became_peered":"2024-03-06T07:26:17.625895+0000","last_unstale":"2024-03-06T07:26:24.658129+0000","last_undegraded":"2024-03-06T07:26:24.658129+0000","last_fullsized":"2024-03-06T07:26:24.658129+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:16.007003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662590+0000","last_change":"2024-03-06T07:26:16.625471+0000","last_active":"2024-03-06T07:26:24.662590+0000","last_peered":"2024-03-06T07:26:24.662590+0000","last_clean":"2024-03-06T07:26:24.662590+0000","last_became_active":"2024-03-06T07:26:16.625198+0000","last_became_peered":"2024-03-06T07:26:16.625198+0000","last_unstale":"2024-03-06T07:26:24.662590+0000","last_undegraded":"2024-03-06T07:26:24.662590+0000","last_fullsized":"2024-03-06T07:26:24.662590+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:37.984120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.664299+0000","last_change":"2024-03-06T07:26:24.664299+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.664299+0000","last_undegraded":"2024-03-06T07:26:24.664299+0000","last_fullsized":"2024-03-06T07:26:24.664299+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664790+0000","last_change":"2024-03-06T07:26:17.630631+0000","last_active":"2024-03-06T07:26:24.664790+0000","last_peered":"2024-03-06T07:26:24.664790+0000","last_clean":"2024-03-06T07:26:24.664790+0000","last_became_active":"2024-03-06T07:26:17.630365+0000","last_became_peered":"2024-03-06T07:26:17.630365+0000","last_unstale":"2024-03-06T07:26:24.664790+0000","last_undegraded":"2024-03-06T07:26:24.664790+0000","last_fullsized":"2024-03-06T07:26:24.664790+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:27.398343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658871+0000","last_change":"2024-03-06T07:26:16.611334+0000","last_active":"2024-03-06T07:26:24.658871+0000","last_peered":"2024-03-06T07:26:24.658871+0000","last_clean":"2024-03-06T07:26:24.658871+0000","last_became_active":"2024-03-06T07:26:16.610646+0000","last_became_peered":"2024-03-06T07:26:16.610646+0000","last_unstale":"2024-03-06T07:26:24.658871+0000","last_undegraded":"2024-03-06T07:26:24.658871+0000","last_fullsized":"2024-03-06T07:26:24.658871+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:32.639393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.664690+0000","last_change":"2024-03-06T07:26:24.664690+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.664690+0000","last_undegraded":"2024-03-06T07:26:24.664690+0000","last_fullsized":"2024-03-06T07:26:24.664690+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663654+0000","last_change":"2024-03-06T07:26:17.632379+0000","last_active":"2024-03-06T07:26:24.663654+0000","last_peered":"2024-03-06T07:26:24.663654+0000","last_clean":"2024-03-06T07:26:24.663654+0000","last_became_active":"2024-03-06T07:26:17.631899+0000","last_became_peered":"2024-03-06T07:26:17.631899+0000","last_unstale":"2024-03-06T07:26:24.663654+0000","last_undegraded":"2024-03-06T07:26:24.663654+0000","last_fullsized":"2024-03-06T07:26:24.663654+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:00.591780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663294+0000","last_change":"2024-03-06T07:26:16.623898+0000","last_active":"2024-03-06T07:26:24.663294+0000","last_peered":"2024-03-06T07:26:24.663294+0000","last_clean":"2024-03-06T07:26:24.663294+0000","last_became_active":"2024-03-06T07:26:16.623602+0000","last_became_peered":"2024-03-06T07:26:16.623602+0000","last_unstale":"2024-03-06T07:26:24.663294+0000","last_undegraded":"2024-03-06T07:26:24.663294+0000","last_fullsized":"2024-03-06T07:26:24.663294+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:56.210974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665305+0000","last_change":"2024-03-06T07:26:24.665305+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665305+0000","last_undegraded":"2024-03-06T07:26:24.665305+0000","last_fullsized":"2024-03-06T07:26:24.665305+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664900+0000","last_change":"2024-03-06T07:26:17.632856+0000","last_active":"2024-03-06T07:26:24.664900+0000","last_peered":"2024-03-06T07:26:24.664900+0000","last_clean":"2024-03-06T07:26:24.664900+0000","last_became_active":"2024-03-06T07:26:17.632027+0000","last_became_peered":"2024-03-06T07:26:17.632027+0000","last_unstale":"2024-03-06T07:26:24.664900+0000","last_undegraded":"2024-03-06T07:26:24.664900+0000","last_fullsized":"2024-03-06T07:26:24.664900+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:38.988254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662329+0000","last_change":"2024-03-06T07:26:16.627698+0000","last_active":"2024-03-06T07:26:24.662329+0000","last_peered":"2024-03-06T07:26:24.662329+0000","last_clean":"2024-03-06T07:26:24.662329+0000","last_became_active":"2024-03-06T07:26:16.627318+0000","last_became_peered":"2024-03-06T07:26:16.627318+0000","last_unstale":"2024-03-06T07:26:24.662329+0000","last_undegraded":"2024-03-06T07:26:24.662329+0000","last_fullsized":"2024-03-06T07:26:24.662329+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:08.249739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665228+0000","last_change":"2024-03-06T07:26:24.665228+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665228+0000","last_undegraded":"2024-03-06T07:26:24.665228+0000","last_fullsized":"2024-03-06T07:26:24.665228+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662577+0000","last_change":"2024-03-06T07:26:17.637676+0000","last_active":"2024-03-06T07:26:24.662577+0000","last_peered":"2024-03-06T07:26:24.662577+0000","last_clean":"2024-03-06T07:26:24.662577+0000","last_became_active":"2024-03-06T07:26:17.637375+0000","last_became_peered":"2024-03-06T07:26:17.637375+0000","last_unstale":"2024-03-06T07:26:24.662577+0000","last_undegraded":"2024-03-06T07:26:24.662577+0000","last_fullsized":"2024-03-06T07:26:24.662577+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:18.556057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666678+0000","last_change":"2024-03-06T07:26:16.620202+0000","last_active":"2024-03-06T07:26:24.666678+0000","last_peered":"2024-03-06T07:26:24.666678+0000","last_clean":"2024-03-06T07:26:24.666678+0000","last_became_active":"2024-03-06T07:26:16.619897+0000","last_became_peered":"2024-03-06T07:26:16.619897+0000","last_unstale":"2024-03-06T07:26:24.666678+0000","last_undegraded":"2024-03-06T07:26:24.666678+0000","last_fullsized":"2024-03-06T07:26:24.666678+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:48.709582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.666356+0000","last_change":"2024-03-06T07:26:24.666356+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.666356+0000","last_undegraded":"2024-03-06T07:26:24.666356+0000","last_fullsized":"2024-03-06T07:26:24.666356+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658331+0000","last_change":"2024-03-06T07:26:17.625716+0000","last_active":"2024-03-06T07:26:24.658331+0000","last_peered":"2024-03-06T07:26:24.658331+0000","last_clean":"2024-03-06T07:26:24.658331+0000","last_became_active":"2024-03-06T07:26:17.625429+0000","last_became_peered":"2024-03-06T07:26:17.625429+0000","last_unstale":"2024-03-06T07:26:24.658331+0000","last_undegraded":"2024-03-06T07:26:24.658331+0000","last_fullsized":"2024-03-06T07:26:24.658331+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:38.596798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667604+0000","last_change":"2024-03-06T07:26:16.620839+0000","last_active":"2024-03-06T07:26:24.667604+0000","last_peered":"2024-03-06T07:26:24.667604+0000","last_clean":"2024-03-06T07:26:24.667604+0000","last_became_active":"2024-03-06T07:26:16.620561+0000","last_became_peered":"2024-03-06T07:26:16.620561+0000","last_unstale":"2024-03-06T07:26:24.667604+0000","last_undegraded":"2024-03-06T07:26:24.667604+0000","last_fullsized":"2024-03-06T07:26:24.667604+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:26.394208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.663905+0000","last_change":"2024-03-06T07:26:24.663905+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.663905+0000","last_undegraded":"2024-03-06T07:26:24.663905+0000","last_fullsized":"2024-03-06T07:26:24.663905+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664986+0000","last_change":"2024-03-06T07:26:17.632976+0000","last_active":"2024-03-06T07:26:24.664986+0000","last_peered":"2024-03-06T07:26:24.664986+0000","last_clean":"2024-03-06T07:26:24.664986+0000","last_became_active":"2024-03-06T07:26:17.632721+0000","last_became_peered":"2024-03-06T07:26:17.632721+0000","last_unstale":"2024-03-06T07:26:24.664986+0000","last_undegraded":"2024-03-06T07:26:24.664986+0000","last_fullsized":"2024-03-06T07:26:24.664986+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:56.134884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662665+0000","last_change":"2024-03-06T07:26:16.626430+0000","last_active":"2024-03-06T07:26:24.662665+0000","last_peered":"2024-03-06T07:26:24.662665+0000","last_clean":"2024-03-06T07:26:24.662665+0000","last_became_active":"2024-03-06T07:26:16.625971+0000","last_became_peered":"2024-03-06T07:26:16.625971+0000","last_unstale":"2024-03-06T07:26:24.662665+0000","last_undegraded":"2024-03-06T07:26:24.662665+0000","last_fullsized":"2024-03-06T07:26:24.662665+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:28.109173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":419,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664587+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:26:24.664587+0000","last_peered":"2024-03-06T07:26:24.664587+0000","last_clean":"2024-03-06T07:26:24.664587+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:26:24.664587+0000","last_undegraded":"2024-03-06T07:26:24.664587+0000","last_fullsized":"2024-03-06T07:26:24.664587+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":1941504,"data_stored":1496336,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574924,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33016,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1875968,"data_stored":905968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574924,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1884160,"data_stored":906092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574925,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1892352,"data_stored":906732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574924,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33036,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338804,"statfs":{"total":96636764160,"available":96602935296,"internally_reserved":0,"allocated":1896448,"data_stored":908184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574924,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33016,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1875968,"data_stored":905968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1888256,"data_stored":907580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574925,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33088,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338752,"statfs":{"total":96636764160,"available":96602882048,"internally_reserved":0,"allocated":1949696,"data_stored":1496440,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:30.080 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:30.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.236+0000 7fd9de1c7640 1 Processor -- start 2024-03-06T07:26:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.237+0000 7fd9de1c7640 1 -- start start 2024-03-06T07:26:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.237+0000 7fd9de1c7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d8152820 0x7fd9d80c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.237+0000 7fd9de1c7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d80c7180 con 0x7fd9d80af6f0 2024-03-06T07:26:30.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.237+0000 7fd9de1c7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d8155460 con 0x7fd9d80c0670 2024-03-06T07:26:30.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.237+0000 7fd9de1c7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d80c6bc0 con 0x7fd9d8152820 2024-03-06T07:26:30.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.237+0000 7fd9d7fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd9d80af6f0 0x7fd9d815fb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60356/0 (socket says 172.21.15.1:60356) 2024-03-06T07:26:30.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.237+0000 7fd9d7fff640 1 -- 172.21.15.1:0/1305032276 learned_addr learned my addr 172.21.15.1:0/1305032276 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:30.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.238+0000 7fd9d77fe640 1 --2- 172.21.15.1:0/1305032276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d8152820 0x7fd9d80c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.238+0000 7fd9d67fc640 1 -- 172.21.15.1:0/1305032276 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789580678 0 0) 0x7fd9d80c7180 con 0x7fd9d80af6f0 2024-03-06T07:26:30.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.238+0000 7fd9d67fc640 1 -- 172.21.15.1:0/1305032276 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c0003520 con 0x7fd9d80af6f0 2024-03-06T07:26:30.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.238+0000 7fd9d67fc640 1 -- 172.21.15.1:0/1305032276 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269023296 0 0) 0x7fd9d8155460 con 0x7fd9d80c0670 2024-03-06T07:26:30.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.238+0000 7fd9d67fc640 1 -- 172.21.15.1:0/1305032276 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d80c7180 con 0x7fd9d80c0670 2024-03-06T07:26:30.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.238+0000 7fd9d67fc640 1 -- 172.21.15.1:0/1305032276 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3556337553 0 0) 0x7fd9c0003520 con 0x7fd9d80af6f0 2024-03-06T07:26:30.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.238+0000 7fd9d77fe640 1 -- 172.21.15.1:0/1305032276 >> v1:172.21.15.98:6789/0 conn(0x7fd9d80c0670 legacy=0x7fd9d80c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.238+0000 7fd9d77fe640 1 -- 172.21.15.1:0/1305032276 >> v1:172.21.15.1:6789/0 conn(0x7fd9d80af6f0 legacy=0x7fd9d815fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.238+0000 7fd9d77fe640 1 -- 172.21.15.1:0/1305032276 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d80c6ea0 con 0x7fd9d8152820 2024-03-06T07:26:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.239+0000 7fd9d77fe640 1 --2- 172.21.15.1:0/1305032276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d8152820 0x7fd9d80c0130 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80097d0 tx=0x7fd9c802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=837c8fb4bf8d3a17 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.239+0000 7fd9d67fc640 1 -- 172.21.15.1:0/1305032276 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8037070 con 0x7fd9d8152820 2024-03-06T07:26:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.239+0000 7fd9d67fc640 1 -- 172.21.15.1:0/1305032276 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c8030510 con 0x7fd9d8152820 2024-03-06T07:26:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.239+0000 7fd9d67fc640 1 -- 172.21.15.1:0/1305032276 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0001970 con 0x7fd9d8152820 2024-03-06T07:26:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.240+0000 7fd9de1c7640 1 -- 172.21.15.1:0/1305032276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d8152820 msgr2=0x7fd9d80c0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.240+0000 7fd9de1c7640 1 --2- 172.21.15.1:0/1305032276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d8152820 0x7fd9d80c0130 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80097d0 tx=0x7fd9c802b800 comp rx=0 tx=0).stop 2024-03-06T07:26:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.240+0000 7fd9de1c7640 1 -- 172.21.15.1:0/1305032276 shutdown_connections 2024-03-06T07:26:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.240+0000 7fd9de1c7640 1 --2- 172.21.15.1:0/1305032276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d8152820 0x7fd9d80c0130 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.240+0000 7fd9de1c7640 1 -- 172.21.15.1:0/1305032276 >> 172.21.15.1:0/1305032276 conn(0x7fd9d8072720 msgr2=0x7fd9d8072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.240+0000 7fd9de1c7640 1 -- 172.21.15.1:0/1305032276 shutdown_connections 2024-03-06T07:26:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.240+0000 7fd9de1c7640 1 -- 172.21.15.1:0/1305032276 wait complete. 2024-03-06T07:26:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.241+0000 7fd9de1c7640 1 Processor -- start 2024-03-06T07:26:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.241+0000 7fd9de1c7640 1 -- start start 2024-03-06T07:26:30.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.242+0000 7fd9de1c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d80af6f0 0x7fd9d80c3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.242+0000 7fd9de1c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d80c0670 0x7fd9d80aa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.242+0000 7fd9de1c7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d8152820 0x7fd9d814d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.242+0000 7fd9de1c7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d80c7c80 con 0x7fd9d80c0670 2024-03-06T07:26:30.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.242+0000 7fd9de1c7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d80c7b00 con 0x7fd9d80af6f0 2024-03-06T07:26:30.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.242+0000 7fd9de1c7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d8152c00 con 0x7fd9d8152820 2024-03-06T07:26:30.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.242+0000 7fd9d6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d80c0670 0x7fd9d80aa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.242+0000 7fd9d6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d80c0670 0x7fd9d80aa1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58642/0 (socket says 172.21.15.1:58642) 2024-03-06T07:26:30.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.242+0000 7fd9d6ffd640 1 -- 172.21.15.1:0/4108807756 learned_addr learned my addr 172.21.15.1:0/4108807756 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:30.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.243+0000 7fd9d7fff640 1 --2- 172.21.15.1:0/4108807756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d8152820 0x7fd9d814d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.243+0000 7fd9d77fe640 1 --2- 172.21.15.1:0/4108807756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d80af6f0 0x7fd9d80c3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.243+0000 7fd9d6ffd640 1 -- 172.21.15.1:0/4108807756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d8152820 msgr2=0x7fd9d814d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.243+0000 7fd9d6ffd640 1 --2- 172.21.15.1:0/4108807756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d8152820 0x7fd9d814d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.243+0000 7fd9d6ffd640 1 -- 172.21.15.1:0/4108807756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d80af6f0 msgr2=0x7fd9d80c3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.243+0000 7fd9d6ffd640 1 --2- 172.21.15.1:0/4108807756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d80af6f0 0x7fd9d80c3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.243+0000 7fd9d6ffd640 1 -- 172.21.15.1:0/4108807756 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d8158b50 con 0x7fd9d80c0670 2024-03-06T07:26:30.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.243+0000 7fd9d7fff640 1 --2- 172.21.15.1:0/4108807756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d8152820 0x7fd9d814d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:30.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.243+0000 7fd9d77fe640 1 --2- 172.21.15.1:0/4108807756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d80af6f0 0x7fd9d80c3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:30.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.243+0000 7fd9d6ffd640 1 --2- 172.21.15.1:0/4108807756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d80c0670 0x7fd9d80aa1c0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc00c500 tx=0x7fd9cc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.243+0000 7fd9bffff640 1 -- 172.21.15.1:0/4108807756 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc0052d0 con 0x7fd9d80c0670 2024-03-06T07:26:30.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.243+0000 7fd9bffff640 1 -- 172.21.15.1:0/4108807756 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9cc005470 con 0x7fd9d80c0670 2024-03-06T07:26:30.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.243+0000 7fd9de1c7640 1 -- 172.21.15.1:0/4108807756 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d8158e30 con 0x7fd9d80c0670 2024-03-06T07:26:30.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.243+0000 7fd9bffff640 1 -- 172.21.15.1:0/4108807756 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc012870 con 0x7fd9d80c0670 2024-03-06T07:26:30.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.243+0000 7fd9de1c7640 1 -- 172.21.15.1:0/4108807756 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d81593f0 con 0x7fd9d80c0670 2024-03-06T07:26:30.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.244+0000 7fd9de1c7640 1 -- 172.21.15.1:0/4108807756 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d80686e0 con 0x7fd9d80c0670 2024-03-06T07:26:30.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.245+0000 7fd9bffff640 1 -- 172.21.15.1:0/4108807756 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9cc012ad0 con 0x7fd9d80c0670 2024-03-06T07:26:30.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.245+0000 7fd9bffff640 1 --2- 172.21.15.1:0/4108807756 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd9c406b4d0 0x7fd9c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.245+0000 7fd9bffff640 1 -- 172.21.15.1:0/4108807756 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fd9cc08a3d0 con 0x7fd9d80c0670 2024-03-06T07:26:30.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.246+0000 7fd9d77fe640 1 --2- 172.21.15.1:0/4108807756 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd9c406b4d0 0x7fd9c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.246+0000 7fd9d77fe640 1 --2- 172.21.15.1:0/4108807756 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd9c406b4d0 0x7fd9c406d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8034000 tx=0x7fd9c8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.250+0000 7fd9bffff640 1 -- 172.21.15.1:0/4108807756 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9cc058d80 con 0x7fd9d80c0670 2024-03-06T07:26:30.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.425+0000 7fd9de1c7640 1 -- 172.21.15.1:0/4108807756 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd9d8155ee0 con 0x7fd9c406b4d0 2024-03-06T07:26:30.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.437+0000 7fd9bffff640 1 -- 172.21.15.1:0/4108807756 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+479322 (secure 0 0 0) 0x7fd9c8030690 con 0x7fd9c406b4d0 2024-03-06T07:26:30.443 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:30.446 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:26:30.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.441+0000 7fd9de1c7640 1 -- 172.21.15.1:0/4108807756 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd9c406b4d0 msgr2=0x7fd9c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.441+0000 7fd9de1c7640 1 --2- 172.21.15.1:0/4108807756 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd9c406b4d0 0x7fd9c406d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8034000 tx=0x7fd9c8008030 comp rx=0 tx=0).stop 2024-03-06T07:26:30.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.441+0000 7fd9de1c7640 1 -- 172.21.15.1:0/4108807756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d80c0670 msgr2=0x7fd9d80aa1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.441+0000 7fd9de1c7640 1 --2- 172.21.15.1:0/4108807756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d80c0670 0x7fd9d80aa1c0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc00c500 tx=0x7fd9cc00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.441+0000 7fd9de1c7640 1 -- 172.21.15.1:0/4108807756 shutdown_connections 2024-03-06T07:26:30.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.441+0000 7fd9de1c7640 1 --2- 172.21.15.1:0/4108807756 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd9c406b4d0 0x7fd9c406d990 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.441+0000 7fd9de1c7640 1 --2- 172.21.15.1:0/4108807756 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d8152820 0x7fd9d814d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.441+0000 7fd9de1c7640 1 --2- 172.21.15.1:0/4108807756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d80c0670 0x7fd9d80aa1c0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.441+0000 7fd9de1c7640 1 --2- 172.21.15.1:0/4108807756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d80af6f0 0x7fd9d80c3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.441+0000 7fd9de1c7640 1 -- 172.21.15.1:0/4108807756 >> 172.21.15.1:0/4108807756 conn(0x7fd9d8072720 msgr2=0x7fd9d8151ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.442+0000 7fd9de1c7640 1 -- 172.21.15.1:0/4108807756 shutdown_connections 2024-03-06T07:26:30.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.442+0000 7fd9de1c7640 1 -- 172.21.15.1:0/4108807756 wait complete. 2024-03-06T07:26:30.454 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-03-06T07:26:28.774882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264320,"kb_used_data":14848,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710400,"statfs":{"total":773094113280,"available":772823449600,"internally_reserved":0,"allocated":15204352,"data_stored":8433300,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.115308"},"pg_stats":[{"pgid":"26.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3f","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663770+0000","last_change":"2024-03-06T07:26:17.632610+0000","last_active":"2024-03-06T07:26:24.663770+0000","last_peered":"2024-03-06T07:26:24.663770+0000","last_clean":"2024-03-06T07:26:24.663770+0000","last_became_active":"2024-03-06T07:26:17.630763+0000","last_became_peered":"2024-03-06T07:26:17.630763+0000","last_unstale":"2024-03-06T07:26:24.663770+0000","last_undegraded":"2024-03-06T07:26:24.663770+0000","last_fullsized":"2024-03-06T07:26:24.663770+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:35.035202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664635+0000","last_change":"2024-03-06T07:26:17.632425+0000","last_active":"2024-03-06T07:26:24.664635+0000","last_peered":"2024-03-06T07:26:24.664635+0000","last_clean":"2024-03-06T07:26:24.664635+0000","last_became_active":"2024-03-06T07:26:17.632153+0000","last_became_peered":"2024-03-06T07:26:17.632153+0000","last_unstale":"2024-03-06T07:26:24.664635+0000","last_undegraded":"2024-03-06T07:26:24.664635+0000","last_fullsized":"2024-03-06T07:26:24.664635+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:01.829378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663942+0000","last_change":"2024-03-06T07:26:17.630353+0000","last_active":"2024-03-06T07:26:24.663942+0000","last_peered":"2024-03-06T07:26:24.663942+0000","last_clean":"2024-03-06T07:26:24.663942+0000","last_became_active":"2024-03-06T07:26:17.630080+0000","last_became_peered":"2024-03-06T07:26:17.630080+0000","last_unstale":"2024-03-06T07:26:24.663942+0000","last_undegraded":"2024-03-06T07:26:24.663942+0000","last_fullsized":"2024-03-06T07:26:24.663942+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:49.713717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663552+0000","last_change":"2024-03-06T07:26:17.630434+0000","last_active":"2024-03-06T07:26:24.663552+0000","last_peered":"2024-03-06T07:26:24.663552+0000","last_clean":"2024-03-06T07:26:24.663552+0000","last_became_active":"2024-03-06T07:26:17.630139+0000","last_became_peered":"2024-03-06T07:26:17.630139+0000","last_unstale":"2024-03-06T07:26:24.663552+0000","last_undegraded":"2024-03-06T07:26:24.663552+0000","last_fullsized":"2024-03-06T07:26:24.663552+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:09.159934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668267+0000","last_change":"2024-03-06T07:26:17.632665+0000","last_active":"2024-03-06T07:26:24.668267+0000","last_peered":"2024-03-06T07:26:24.668267+0000","last_clean":"2024-03-06T07:26:24.668267+0000","last_became_active":"2024-03-06T07:26:17.632377+0000","last_became_peered":"2024-03-06T07:26:17.632377+0000","last_unstale":"2024-03-06T07:26:24.668267+0000","last_undegraded":"2024-03-06T07:26:24.668267+0000","last_fullsized":"2024-03-06T07:26:24.668267+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:23.225363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665614+0000","last_change":"2024-03-06T07:26:17.633911+0000","last_active":"2024-03-06T07:26:24.665614+0000","last_peered":"2024-03-06T07:26:24.665614+0000","last_clean":"2024-03-06T07:26:24.665614+0000","last_became_active":"2024-03-06T07:26:17.633638+0000","last_became_peered":"2024-03-06T07:26:17.633638+0000","last_unstale":"2024-03-06T07:26:24.665614+0000","last_undegraded":"2024-03-06T07:26:24.665614+0000","last_fullsized":"2024-03-06T07:26:24.665614+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:01.350963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664718+0000","last_change":"2024-03-06T07:26:17.629894+0000","last_active":"2024-03-06T07:26:24.664718+0000","last_peered":"2024-03-06T07:26:24.664718+0000","last_clean":"2024-03-06T07:26:24.664718+0000","last_became_active":"2024-03-06T07:26:17.629625+0000","last_became_peered":"2024-03-06T07:26:17.629625+0000","last_unstale":"2024-03-06T07:26:24.664718+0000","last_undegraded":"2024-03-06T07:26:24.664718+0000","last_fullsized":"2024-03-06T07:26:24.664718+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:46.687284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664295+0000","last_change":"2024-03-06T07:26:17.632711+0000","last_active":"2024-03-06T07:26:24.664295+0000","last_peered":"2024-03-06T07:26:24.664295+0000","last_clean":"2024-03-06T07:26:24.664295+0000","last_became_active":"2024-03-06T07:26:17.632173+0000","last_became_peered":"2024-03-06T07:26:17.632173+0000","last_unstale":"2024-03-06T07:26:24.664295+0000","last_undegraded":"2024-03-06T07:26:24.664295+0000","last_fullsized":"2024-03-06T07:26:24.664295+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:25.595909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665506+0000","last_change":"2024-03-06T07:26:17.638531+0000","last_active":"2024-03-06T07:26:24.665506+0000","last_peered":"2024-03-06T07:26:24.665506+0000","last_clean":"2024-03-06T07:26:24.665506+0000","last_became_active":"2024-03-06T07:26:17.638254+0000","last_became_peered":"2024-03-06T07:26:17.638254+0000","last_unstale":"2024-03-06T07:26:24.665506+0000","last_undegraded":"2024-03-06T07:26:24.665506+0000","last_fullsized":"2024-03-06T07:26:24.665506+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:01.350963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665485+0000","last_change":"2024-03-06T07:26:17.632455+0000","last_active":"2024-03-06T07:26:24.665485+0000","last_peered":"2024-03-06T07:26:24.665485+0000","last_clean":"2024-03-06T07:26:24.665485+0000","last_became_active":"2024-03-06T07:26:17.632195+0000","last_became_peered":"2024-03-06T07:26:17.632195+0000","last_unstale":"2024-03-06T07:26:24.665485+0000","last_undegraded":"2024-03-06T07:26:24.665485+0000","last_fullsized":"2024-03-06T07:26:24.665485+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:14.404340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667172+0000","last_change":"2024-03-06T07:26:17.633984+0000","last_active":"2024-03-06T07:26:24.667172+0000","last_peered":"2024-03-06T07:26:24.667172+0000","last_clean":"2024-03-06T07:26:24.667172+0000","last_became_active":"2024-03-06T07:26:17.633666+0000","last_became_peered":"2024-03-06T07:26:17.633666+0000","last_unstale":"2024-03-06T07:26:24.667172+0000","last_undegraded":"2024-03-06T07:26:24.667172+0000","last_fullsized":"2024-03-06T07:26:24.667172+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:43.884264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664561+0000","last_change":"2024-03-06T07:26:17.634798+0000","last_active":"2024-03-06T07:26:24.664561+0000","last_peered":"2024-03-06T07:26:24.664561+0000","last_clean":"2024-03-06T07:26:24.664561+0000","last_became_active":"2024-03-06T07:26:17.634537+0000","last_became_peered":"2024-03-06T07:26:17.634537+0000","last_unstale":"2024-03-06T07:26:24.664561+0000","last_undegraded":"2024-03-06T07:26:24.664561+0000","last_fullsized":"2024-03-06T07:26:24.664561+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:25.595909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662407+0000","last_change":"2024-03-06T07:26:17.640814+0000","last_active":"2024-03-06T07:26:24.662407+0000","last_peered":"2024-03-06T07:26:24.662407+0000","last_clean":"2024-03-06T07:26:24.662407+0000","last_became_active":"2024-03-06T07:26:17.640444+0000","last_became_peered":"2024-03-06T07:26:17.640444+0000","last_unstale":"2024-03-06T07:26:24.662407+0000","last_undegraded":"2024-03-06T07:26:24.662407+0000","last_fullsized":"2024-03-06T07:26:24.662407+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:35.920913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667754+0000","last_change":"2024-03-06T07:26:17.631892+0000","last_active":"2024-03-06T07:26:24.667754+0000","last_peered":"2024-03-06T07:26:24.667754+0000","last_clean":"2024-03-06T07:26:24.667754+0000","last_became_active":"2024-03-06T07:26:17.631498+0000","last_became_peered":"2024-03-06T07:26:17.631498+0000","last_unstale":"2024-03-06T07:26:24.667754+0000","last_undegraded":"2024-03-06T07:26:24.667754+0000","last_fullsized":"2024-03-06T07:26:24.667754+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:09.253874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666379+0000","last_change":"2024-03-06T07:26:17.634220+0000","last_active":"2024-03-06T07:26:24.666379+0000","last_peered":"2024-03-06T07:26:24.666379+0000","last_clean":"2024-03-06T07:26:24.666379+0000","last_became_active":"2024-03-06T07:26:17.633922+0000","last_became_peered":"2024-03-06T07:26:17.633922+0000","last_unstale":"2024-03-06T07:26:24.666379+0000","last_undegraded":"2024-03-06T07:26:24.666379+0000","last_fullsized":"2024-03-06T07:26:24.666379+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:51.900557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662529+0000","last_change":"2024-03-06T07:26:17.635856+0000","last_active":"2024-03-06T07:26:24.662529+0000","last_peered":"2024-03-06T07:26:24.662529+0000","last_clean":"2024-03-06T07:26:24.662529+0000","last_became_active":"2024-03-06T07:26:17.631353+0000","last_became_peered":"2024-03-06T07:26:17.631353+0000","last_unstale":"2024-03-06T07:26:24.662529+0000","last_undegraded":"2024-03-06T07:26:24.662529+0000","last_fullsized":"2024-03-06T07:26:24.662529+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:15.652152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665594+0000","last_change":"2024-03-06T07:26:17.628960+0000","last_active":"2024-03-06T07:26:24.665594+0000","last_peered":"2024-03-06T07:26:24.665594+0000","last_clean":"2024-03-06T07:26:24.665594+0000","last_became_active":"2024-03-06T07:26:17.628681+0000","last_became_peered":"2024-03-06T07:26:17.628681+0000","last_unstale":"2024-03-06T07:26:24.665594+0000","last_undegraded":"2024-03-06T07:26:24.665594+0000","last_fullsized":"2024-03-06T07:26:24.665594+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:40.950597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666241+0000","last_change":"2024-03-06T07:26:17.629582+0000","last_active":"2024-03-06T07:26:24.666241+0000","last_peered":"2024-03-06T07:26:24.666241+0000","last_clean":"2024-03-06T07:26:24.666241+0000","last_became_active":"2024-03-06T07:26:17.629246+0000","last_became_peered":"2024-03-06T07:26:17.629246+0000","last_unstale":"2024-03-06T07:26:24.666241+0000","last_undegraded":"2024-03-06T07:26:24.666241+0000","last_fullsized":"2024-03-06T07:26:24.666241+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:56.556155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667465+0000","last_change":"2024-03-06T07:26:17.632386+0000","last_active":"2024-03-06T07:26:24.667465+0000","last_peered":"2024-03-06T07:26:24.667465+0000","last_clean":"2024-03-06T07:26:24.667465+0000","last_became_active":"2024-03-06T07:26:17.631867+0000","last_became_peered":"2024-03-06T07:26:17.631867+0000","last_unstale":"2024-03-06T07:26:24.667465+0000","last_undegraded":"2024-03-06T07:26:24.667465+0000","last_fullsized":"2024-03-06T07:26:24.667465+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:46.687284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658784+0000","last_change":"2024-03-06T07:26:17.622563+0000","last_active":"2024-03-06T07:26:24.658784+0000","last_peered":"2024-03-06T07:26:24.658784+0000","last_clean":"2024-03-06T07:26:24.658784+0000","last_became_active":"2024-03-06T07:26:17.622293+0000","last_became_peered":"2024-03-06T07:26:17.622293+0000","last_unstale":"2024-03-06T07:26:24.658784+0000","last_undegraded":"2024-03-06T07:26:24.658784+0000","last_fullsized":"2024-03-06T07:26:24.658784+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:34.462291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662501+0000","last_change":"2024-03-06T07:26:17.637385+0000","last_active":"2024-03-06T07:26:24.662501+0000","last_peered":"2024-03-06T07:26:24.662501+0000","last_clean":"2024-03-06T07:26:24.662501+0000","last_became_active":"2024-03-06T07:26:17.637063+0000","last_became_peered":"2024-03-06T07:26:17.637063+0000","last_unstale":"2024-03-06T07:26:24.662501+0000","last_undegraded":"2024-03-06T07:26:24.662501+0000","last_fullsized":"2024-03-06T07:26:24.662501+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:49.333296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665712+0000","last_change":"2024-03-06T07:26:17.632278+0000","last_active":"2024-03-06T07:26:24.665712+0000","last_peered":"2024-03-06T07:26:24.665712+0000","last_clean":"2024-03-06T07:26:24.665712+0000","last_became_active":"2024-03-06T07:26:17.631791+0000","last_became_peered":"2024-03-06T07:26:17.631791+0000","last_unstale":"2024-03-06T07:26:24.665712+0000","last_undegraded":"2024-03-06T07:26:24.665712+0000","last_fullsized":"2024-03-06T07:26:24.665712+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:40.950597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667556+0000","last_change":"2024-03-06T07:26:17.627902+0000","last_active":"2024-03-06T07:26:24.667556+0000","last_peered":"2024-03-06T07:26:24.667556+0000","last_clean":"2024-03-06T07:26:24.667556+0000","last_became_active":"2024-03-06T07:26:17.627643+0000","last_became_peered":"2024-03-06T07:26:17.627643+0000","last_unstale":"2024-03-06T07:26:24.667556+0000","last_undegraded":"2024-03-06T07:26:24.667556+0000","last_fullsized":"2024-03-06T07:26:24.667556+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:30.600647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662946+0000","last_change":"2024-03-06T07:26:17.631421+0000","last_active":"2024-03-06T07:26:24.662946+0000","last_peered":"2024-03-06T07:26:24.662946+0000","last_clean":"2024-03-06T07:26:24.662946+0000","last_became_active":"2024-03-06T07:26:17.631022+0000","last_became_peered":"2024-03-06T07:26:17.631022+0000","last_unstale":"2024-03-06T07:26:24.662946+0000","last_undegraded":"2024-03-06T07:26:24.662946+0000","last_fullsized":"2024-03-06T07:26:24.662946+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:40.032388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662940+0000","last_change":"2024-03-06T07:26:17.637044+0000","last_active":"2024-03-06T07:26:24.662940+0000","last_peered":"2024-03-06T07:26:24.662940+0000","last_clean":"2024-03-06T07:26:24.662940+0000","last_became_active":"2024-03-06T07:26:17.636771+0000","last_became_peered":"2024-03-06T07:26:17.636771+0000","last_unstale":"2024-03-06T07:26:24.662940+0000","last_undegraded":"2024-03-06T07:26:24.662940+0000","last_fullsized":"2024-03-06T07:26:24.662940+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:29.354563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658522+0000","last_change":"2024-03-06T07:26:17.636369+0000","last_active":"2024-03-06T07:26:24.658522+0000","last_peered":"2024-03-06T07:26:24.658522+0000","last_clean":"2024-03-06T07:26:24.658522+0000","last_became_active":"2024-03-06T07:26:17.636090+0000","last_became_peered":"2024-03-06T07:26:17.636090+0000","last_unstale":"2024-03-06T07:26:24.658522+0000","last_undegraded":"2024-03-06T07:26:24.658522+0000","last_fullsized":"2024-03-06T07:26:24.658522+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:45.110614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664860+0000","last_change":"2024-03-06T07:26:17.633006+0000","last_active":"2024-03-06T07:26:24.664860+0000","last_peered":"2024-03-06T07:26:24.664860+0000","last_clean":"2024-03-06T07:26:24.664860+0000","last_became_active":"2024-03-06T07:26:17.632523+0000","last_became_peered":"2024-03-06T07:26:17.632523+0000","last_unstale":"2024-03-06T07:26:24.664860+0000","last_undegraded":"2024-03-06T07:26:24.664860+0000","last_fullsized":"2024-03-06T07:26:24.664860+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:41.655929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662348+0000","last_change":"2024-03-06T07:26:17.638448+0000","last_active":"2024-03-06T07:26:24.662348+0000","last_peered":"2024-03-06T07:26:24.662348+0000","last_clean":"2024-03-06T07:26:24.662348+0000","last_became_active":"2024-03-06T07:26:17.638188+0000","last_became_peered":"2024-03-06T07:26:17.638188+0000","last_unstale":"2024-03-06T07:26:24.662348+0000","last_undegraded":"2024-03-06T07:26:24.662348+0000","last_fullsized":"2024-03-06T07:26:24.662348+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:15.455961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666402+0000","last_change":"2024-03-06T07:26:17.634053+0000","last_active":"2024-03-06T07:26:24.666402+0000","last_peered":"2024-03-06T07:26:24.666402+0000","last_clean":"2024-03-06T07:26:24.666402+0000","last_became_active":"2024-03-06T07:26:17.633451+0000","last_became_peered":"2024-03-06T07:26:17.633451+0000","last_unstale":"2024-03-06T07:26:24.666402+0000","last_undegraded":"2024-03-06T07:26:24.666402+0000","last_fullsized":"2024-03-06T07:26:24.666402+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:25.173456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665705+0000","last_change":"2024-03-06T07:26:17.627891+0000","last_active":"2024-03-06T07:26:24.665705+0000","last_peered":"2024-03-06T07:26:24.665705+0000","last_clean":"2024-03-06T07:26:24.665705+0000","last_became_active":"2024-03-06T07:26:17.627624+0000","last_became_peered":"2024-03-06T07:26:17.627624+0000","last_unstale":"2024-03-06T07:26:24.665705+0000","last_undegraded":"2024-03-06T07:26:24.665705+0000","last_fullsized":"2024-03-06T07:26:24.665705+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:56.476412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666557+0000","last_change":"2024-03-06T07:26:17.635467+0000","last_active":"2024-03-06T07:26:24.666557+0000","last_peered":"2024-03-06T07:26:24.666557+0000","last_clean":"2024-03-06T07:26:24.666557+0000","last_became_active":"2024-03-06T07:26:17.635125+0000","last_became_peered":"2024-03-06T07:26:17.635125+0000","last_unstale":"2024-03-06T07:26:24.666557+0000","last_undegraded":"2024-03-06T07:26:24.666557+0000","last_fullsized":"2024-03-06T07:26:24.666557+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:39.829929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665165+0000","last_change":"2024-03-06T07:26:17.638944+0000","last_active":"2024-03-06T07:26:24.665165+0000","last_peered":"2024-03-06T07:26:24.665165+0000","last_clean":"2024-03-06T07:26:24.665165+0000","last_became_active":"2024-03-06T07:26:17.638649+0000","last_became_peered":"2024-03-06T07:26:17.638649+0000","last_unstale":"2024-03-06T07:26:24.665165+0000","last_undegraded":"2024-03-06T07:26:24.665165+0000","last_fullsized":"2024-03-06T07:26:24.665165+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:17.394090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665307+0000","last_change":"2024-03-06T07:26:24.665307+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665307+0000","last_undegraded":"2024-03-06T07:26:24.665307+0000","last_fullsized":"2024-03-06T07:26:24.665307+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664442+0000","last_change":"2024-03-06T07:26:17.635968+0000","last_active":"2024-03-06T07:26:24.664442+0000","last_peered":"2024-03-06T07:26:24.664442+0000","last_clean":"2024-03-06T07:26:24.664442+0000","last_became_active":"2024-03-06T07:26:17.635095+0000","last_became_peered":"2024-03-06T07:26:17.635095+0000","last_unstale":"2024-03-06T07:26:24.664442+0000","last_undegraded":"2024-03-06T07:26:24.664442+0000","last_fullsized":"2024-03-06T07:26:24.664442+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:26.879181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"148'4","reported_seq":20,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663882+0000","last_change":"2024-03-06T07:26:16.626240+0000","last_active":"2024-03-06T07:26:24.663882+0000","last_peered":"2024-03-06T07:26:24.663882+0000","last_clean":"2024-03-06T07:26:24.663882+0000","last_became_active":"2024-03-06T07:26:16.626011+0000","last_became_peered":"2024-03-06T07:26:16.626011+0000","last_unstale":"2024-03-06T07:26:24.663882+0000","last_undegraded":"2024-03-06T07:26:24.663882+0000","last_fullsized":"2024-03-06T07:26:24.663882+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:30.259380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665984+0000","last_change":"2024-03-06T07:26:24.665984+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665984+0000","last_undegraded":"2024-03-06T07:26:24.665984+0000","last_fullsized":"2024-03-06T07:26:24.665984+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662038+0000","last_change":"2024-03-06T07:26:17.627512+0000","last_active":"2024-03-06T07:26:24.662038+0000","last_peered":"2024-03-06T07:26:24.662038+0000","last_clean":"2024-03-06T07:26:24.662038+0000","last_became_active":"2024-03-06T07:26:17.627235+0000","last_became_peered":"2024-03-06T07:26:17.627235+0000","last_unstale":"2024-03-06T07:26:24.662038+0000","last_undegraded":"2024-03-06T07:26:24.662038+0000","last_fullsized":"2024-03-06T07:26:24.662038+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:19.245848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"148'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668199+0000","last_change":"2024-03-06T07:26:16.625703+0000","last_active":"2024-03-06T07:26:24.668199+0000","last_peered":"2024-03-06T07:26:24.668199+0000","last_clean":"2024-03-06T07:26:24.668199+0000","last_became_active":"2024-03-06T07:26:16.625426+0000","last_became_peered":"2024-03-06T07:26:16.625426+0000","last_unstale":"2024-03-06T07:26:24.668199+0000","last_undegraded":"2024-03-06T07:26:24.668199+0000","last_fullsized":"2024-03-06T07:26:24.668199+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:39.946462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.662178+0000","last_change":"2024-03-06T07:26:24.662178+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.662178+0000","last_undegraded":"2024-03-06T07:26:24.662178+0000","last_fullsized":"2024-03-06T07:26:24.662178+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667846+0000","last_change":"2024-03-06T07:26:17.632107+0000","last_active":"2024-03-06T07:26:24.667846+0000","last_peered":"2024-03-06T07:26:24.667846+0000","last_clean":"2024-03-06T07:26:24.667846+0000","last_became_active":"2024-03-06T07:26:17.631481+0000","last_became_peered":"2024-03-06T07:26:17.631481+0000","last_unstale":"2024-03-06T07:26:24.667846+0000","last_undegraded":"2024-03-06T07:26:24.667846+0000","last_fullsized":"2024-03-06T07:26:24.667846+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:20.662554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667577+0000","last_change":"2024-03-06T07:26:16.620199+0000","last_active":"2024-03-06T07:26:24.667577+0000","last_peered":"2024-03-06T07:26:24.667577+0000","last_clean":"2024-03-06T07:26:24.667577+0000","last_became_active":"2024-03-06T07:26:16.619877+0000","last_became_peered":"2024-03-06T07:26:16.619877+0000","last_unstale":"2024-03-06T07:26:24.667577+0000","last_undegraded":"2024-03-06T07:26:24.667577+0000","last_fullsized":"2024-03-06T07:26:24.667577+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:14.648017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.663658+0000","last_change":"2024-03-06T07:26:24.663658+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.663658+0000","last_undegraded":"2024-03-06T07:26:24.663658+0000","last_fullsized":"2024-03-06T07:26:24.663658+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666492+0000","last_change":"2024-03-06T07:26:17.634295+0000","last_active":"2024-03-06T07:26:24.666492+0000","last_peered":"2024-03-06T07:26:24.666492+0000","last_clean":"2024-03-06T07:26:24.666492+0000","last_became_active":"2024-03-06T07:26:17.633744+0000","last_became_peered":"2024-03-06T07:26:17.633744+0000","last_unstale":"2024-03-06T07:26:24.666492+0000","last_undegraded":"2024-03-06T07:26:24.666492+0000","last_fullsized":"2024-03-06T07:26:24.666492+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:14.404340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658789+0000","last_change":"2024-03-06T07:26:16.616715+0000","last_active":"2024-03-06T07:26:24.658789+0000","last_peered":"2024-03-06T07:26:24.658789+0000","last_clean":"2024-03-06T07:26:24.658789+0000","last_became_active":"2024-03-06T07:26:16.616466+0000","last_became_peered":"2024-03-06T07:26:16.616466+0000","last_unstale":"2024-03-06T07:26:24.658789+0000","last_undegraded":"2024-03-06T07:26:24.658789+0000","last_fullsized":"2024-03-06T07:26:24.658789+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:54.279225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665856+0000","last_change":"2024-03-06T07:26:24.665856+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665856+0000","last_undegraded":"2024-03-06T07:26:24.665856+0000","last_fullsized":"2024-03-06T07:26:24.665856+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665092+0000","last_change":"2024-03-06T07:26:17.631966+0000","last_active":"2024-03-06T07:26:24.665092+0000","last_peered":"2024-03-06T07:26:24.665092+0000","last_clean":"2024-03-06T07:26:24.665092+0000","last_became_active":"2024-03-06T07:26:17.631673+0000","last_became_peered":"2024-03-06T07:26:17.631673+0000","last_unstale":"2024-03-06T07:26:24.665092+0000","last_undegraded":"2024-03-06T07:26:24.665092+0000","last_fullsized":"2024-03-06T07:26:24.665092+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:26.879181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"148'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664861+0000","last_change":"2024-03-06T07:26:16.619602+0000","last_active":"2024-03-06T07:26:24.664861+0000","last_peered":"2024-03-06T07:26:24.664861+0000","last_clean":"2024-03-06T07:26:24.664861+0000","last_became_active":"2024-03-06T07:26:16.619282+0000","last_became_peered":"2024-03-06T07:26:16.619282+0000","last_unstale":"2024-03-06T07:26:24.664861+0000","last_undegraded":"2024-03-06T07:26:24.664861+0000","last_fullsized":"2024-03-06T07:26:24.664861+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:59.587645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.662223+0000","last_change":"2024-03-06T07:26:24.662223+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.662223+0000","last_undegraded":"2024-03-06T07:26:24.662223+0000","last_fullsized":"2024-03-06T07:26:24.662223+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658427+0000","last_change":"2024-03-06T07:26:17.628669+0000","last_active":"2024-03-06T07:26:24.658427+0000","last_peered":"2024-03-06T07:26:24.658427+0000","last_clean":"2024-03-06T07:26:24.658427+0000","last_became_active":"2024-03-06T07:26:17.628371+0000","last_became_peered":"2024-03-06T07:26:17.628371+0000","last_unstale":"2024-03-06T07:26:24.658427+0000","last_undegraded":"2024-03-06T07:26:24.658427+0000","last_fullsized":"2024-03-06T07:26:24.658427+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:16.499694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"148'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668006+0000","last_change":"2024-03-06T07:26:16.615912+0000","last_active":"2024-03-06T07:26:24.668006+0000","last_peered":"2024-03-06T07:26:24.668006+0000","last_clean":"2024-03-06T07:26:24.668006+0000","last_became_active":"2024-03-06T07:26:16.615625+0000","last_became_peered":"2024-03-06T07:26:16.615625+0000","last_unstale":"2024-03-06T07:26:24.668006+0000","last_undegraded":"2024-03-06T07:26:24.668006+0000","last_fullsized":"2024-03-06T07:26:24.668006+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:24.591774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661284+0000","last_change":"2024-03-06T07:26:24.661284+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661284+0000","last_undegraded":"2024-03-06T07:26:24.661284+0000","last_fullsized":"2024-03-06T07:26:24.661284+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658604+0000","last_change":"2024-03-06T07:26:17.626866+0000","last_active":"2024-03-06T07:26:24.658604+0000","last_peered":"2024-03-06T07:26:24.658604+0000","last_clean":"2024-03-06T07:26:24.658604+0000","last_became_active":"2024-03-06T07:26:17.626586+0000","last_became_peered":"2024-03-06T07:26:17.626586+0000","last_unstale":"2024-03-06T07:26:24.658604+0000","last_undegraded":"2024-03-06T07:26:24.658604+0000","last_fullsized":"2024-03-06T07:26:24.658604+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:34.182149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658031+0000","last_change":"2024-03-06T07:26:16.622797+0000","last_active":"2024-03-06T07:26:24.658031+0000","last_peered":"2024-03-06T07:26:24.658031+0000","last_clean":"2024-03-06T07:26:24.658031+0000","last_became_active":"2024-03-06T07:26:16.622534+0000","last_became_peered":"2024-03-06T07:26:16.622534+0000","last_unstale":"2024-03-06T07:26:24.658031+0000","last_undegraded":"2024-03-06T07:26:24.658031+0000","last_fullsized":"2024-03-06T07:26:24.658031+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:39.697277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.663648+0000","last_change":"2024-03-06T07:26:24.663648+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.663648+0000","last_undegraded":"2024-03-06T07:26:24.663648+0000","last_fullsized":"2024-03-06T07:26:24.663648+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663807+0000","last_change":"2024-03-06T07:26:17.632829+0000","last_active":"2024-03-06T07:26:24.663807+0000","last_peered":"2024-03-06T07:26:24.663807+0000","last_clean":"2024-03-06T07:26:24.663807+0000","last_became_active":"2024-03-06T07:26:17.631051+0000","last_became_peered":"2024-03-06T07:26:17.631051+0000","last_unstale":"2024-03-06T07:26:24.663807+0000","last_undegraded":"2024-03-06T07:26:24.663807+0000","last_fullsized":"2024-03-06T07:26:24.663807+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:09.224572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664647+0000","last_change":"2024-03-06T07:26:16.618575+0000","last_active":"2024-03-06T07:26:24.664647+0000","last_peered":"2024-03-06T07:26:24.664647+0000","last_clean":"2024-03-06T07:26:24.664647+0000","last_became_active":"2024-03-06T07:26:16.618143+0000","last_became_peered":"2024-03-06T07:26:16.618143+0000","last_unstale":"2024-03-06T07:26:24.664647+0000","last_undegraded":"2024-03-06T07:26:24.664647+0000","last_fullsized":"2024-03-06T07:26:24.664647+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:48.709582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661586+0000","last_change":"2024-03-06T07:26:24.661586+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661586+0000","last_undegraded":"2024-03-06T07:26:24.661586+0000","last_fullsized":"2024-03-06T07:26:24.661586+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664401+0000","last_change":"2024-03-06T07:26:17.636599+0000","last_active":"2024-03-06T07:26:24.664401+0000","last_peered":"2024-03-06T07:26:24.664401+0000","last_clean":"2024-03-06T07:26:24.664401+0000","last_became_active":"2024-03-06T07:26:17.635451+0000","last_became_peered":"2024-03-06T07:26:17.635451+0000","last_unstale":"2024-03-06T07:26:24.664401+0000","last_undegraded":"2024-03-06T07:26:24.664401+0000","last_fullsized":"2024-03-06T07:26:24.664401+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:27.398343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662893+0000","last_change":"2024-03-06T07:26:16.627460+0000","last_active":"2024-03-06T07:26:24.662893+0000","last_peered":"2024-03-06T07:26:24.662893+0000","last_clean":"2024-03-06T07:26:24.662893+0000","last_became_active":"2024-03-06T07:26:16.626345+0000","last_became_peered":"2024-03-06T07:26:16.626345+0000","last_unstale":"2024-03-06T07:26:24.662893+0000","last_undegraded":"2024-03-06T07:26:24.662893+0000","last_fullsized":"2024-03-06T07:26:24.662893+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:24.257795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661596+0000","last_change":"2024-03-06T07:26:24.661596+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661596+0000","last_undegraded":"2024-03-06T07:26:24.661596+0000","last_fullsized":"2024-03-06T07:26:24.661596+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662215+0000","last_change":"2024-03-06T07:26:17.634986+0000","last_active":"2024-03-06T07:26:24.662215+0000","last_peered":"2024-03-06T07:26:24.662215+0000","last_clean":"2024-03-06T07:26:24.662215+0000","last_became_active":"2024-03-06T07:26:17.634680+0000","last_became_peered":"2024-03-06T07:26:17.634680+0000","last_unstale":"2024-03-06T07:26:24.662215+0000","last_undegraded":"2024-03-06T07:26:24.662215+0000","last_fullsized":"2024-03-06T07:26:24.662215+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:48.372843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"148'3","reported_seq":19,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664051+0000","last_change":"2024-03-06T07:26:16.623947+0000","last_active":"2024-03-06T07:26:24.664051+0000","last_peered":"2024-03-06T07:26:24.664051+0000","last_clean":"2024-03-06T07:26:24.664051+0000","last_became_active":"2024-03-06T07:26:16.623662+0000","last_became_peered":"2024-03-06T07:26:16.623662+0000","last_unstale":"2024-03-06T07:26:24.664051+0000","last_undegraded":"2024-03-06T07:26:24.664051+0000","last_fullsized":"2024-03-06T07:26:24.664051+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:00.411414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.663853+0000","last_change":"2024-03-06T07:26:24.663853+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.663853+0000","last_undegraded":"2024-03-06T07:26:24.663853+0000","last_fullsized":"2024-03-06T07:26:24.663853+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658162+0000","last_change":"2024-03-06T07:26:17.626440+0000","last_active":"2024-03-06T07:26:24.658162+0000","last_peered":"2024-03-06T07:26:24.658162+0000","last_clean":"2024-03-06T07:26:24.658162+0000","last_became_active":"2024-03-06T07:26:17.626149+0000","last_became_peered":"2024-03-06T07:26:17.626149+0000","last_unstale":"2024-03-06T07:26:24.658162+0000","last_undegraded":"2024-03-06T07:26:24.658162+0000","last_fullsized":"2024-03-06T07:26:24.658162+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:46.191755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"148'6","reported_seq":22,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662564+0000","last_change":"2024-03-06T07:26:16.627536+0000","last_active":"2024-03-06T07:26:24.662564+0000","last_peered":"2024-03-06T07:26:24.662564+0000","last_clean":"2024-03-06T07:26:24.662564+0000","last_became_active":"2024-03-06T07:26:16.627229+0000","last_became_peered":"2024-03-06T07:26:16.627229+0000","last_unstale":"2024-03-06T07:26:24.662564+0000","last_undegraded":"2024-03-06T07:26:24.662564+0000","last_fullsized":"2024-03-06T07:26:24.662564+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:00.825243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665684+0000","last_change":"2024-03-06T07:26:24.665684+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665684+0000","last_undegraded":"2024-03-06T07:26:24.665684+0000","last_fullsized":"2024-03-06T07:26:24.665684+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665948+0000","last_change":"2024-03-06T07:26:17.638945+0000","last_active":"2024-03-06T07:26:24.665948+0000","last_peered":"2024-03-06T07:26:24.665948+0000","last_clean":"2024-03-06T07:26:24.665948+0000","last_became_active":"2024-03-06T07:26:17.638664+0000","last_became_peered":"2024-03-06T07:26:17.638664+0000","last_unstale":"2024-03-06T07:26:24.665948+0000","last_undegraded":"2024-03-06T07:26:24.665948+0000","last_fullsized":"2024-03-06T07:26:24.665948+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:15.414267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668359+0000","last_change":"2024-03-06T07:26:16.625884+0000","last_active":"2024-03-06T07:26:24.668359+0000","last_peered":"2024-03-06T07:26:24.668359+0000","last_clean":"2024-03-06T07:26:24.668359+0000","last_became_active":"2024-03-06T07:26:16.625605+0000","last_became_peered":"2024-03-06T07:26:16.625605+0000","last_unstale":"2024-03-06T07:26:24.668359+0000","last_undegraded":"2024-03-06T07:26:24.668359+0000","last_fullsized":"2024-03-06T07:26:24.668359+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:13.400206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661600+0000","last_change":"2024-03-06T07:26:24.661600+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661600+0000","last_undegraded":"2024-03-06T07:26:24.661600+0000","last_fullsized":"2024-03-06T07:26:24.661600+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664941+0000","last_change":"2024-03-06T07:26:17.631376+0000","last_active":"2024-03-06T07:26:24.664941+0000","last_peered":"2024-03-06T07:26:24.664941+0000","last_clean":"2024-03-06T07:26:24.664941+0000","last_became_active":"2024-03-06T07:26:17.631068+0000","last_became_peered":"2024-03-06T07:26:17.631068+0000","last_unstale":"2024-03-06T07:26:24.664941+0000","last_undegraded":"2024-03-06T07:26:24.664941+0000","last_fullsized":"2024-03-06T07:26:24.664941+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:46.783070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662415+0000","last_change":"2024-03-06T07:26:16.625352+0000","last_active":"2024-03-06T07:26:24.662415+0000","last_peered":"2024-03-06T07:26:24.662415+0000","last_clean":"2024-03-06T07:26:24.662415+0000","last_became_active":"2024-03-06T07:26:16.624893+0000","last_became_peered":"2024-03-06T07:26:16.624893+0000","last_unstale":"2024-03-06T07:26:24.662415+0000","last_undegraded":"2024-03-06T07:26:24.662415+0000","last_fullsized":"2024-03-06T07:26:24.662415+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:29.596513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.667139+0000","last_change":"2024-03-06T07:26:24.667139+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.667139+0000","last_undegraded":"2024-03-06T07:26:24.667139+0000","last_fullsized":"2024-03-06T07:26:24.667139+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667199+0000","last_change":"2024-03-06T07:26:17.639031+0000","last_active":"2024-03-06T07:26:24.667199+0000","last_peered":"2024-03-06T07:26:24.667199+0000","last_clean":"2024-03-06T07:26:24.667199+0000","last_became_active":"2024-03-06T07:26:17.638716+0000","last_became_peered":"2024-03-06T07:26:17.638716+0000","last_unstale":"2024-03-06T07:26:24.667199+0000","last_undegraded":"2024-03-06T07:26:24.667199+0000","last_fullsized":"2024-03-06T07:26:24.667199+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:46.783070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666494+0000","last_change":"2024-03-06T07:26:16.620528+0000","last_active":"2024-03-06T07:26:24.666494+0000","last_peered":"2024-03-06T07:26:24.666494+0000","last_clean":"2024-03-06T07:26:24.666494+0000","last_became_active":"2024-03-06T07:26:16.620245+0000","last_became_peered":"2024-03-06T07:26:16.620245+0000","last_unstale":"2024-03-06T07:26:24.666494+0000","last_undegraded":"2024-03-06T07:26:24.666494+0000","last_fullsized":"2024-03-06T07:26:24.666494+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:39.028254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661458+0000","last_change":"2024-03-06T07:26:24.661458+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661458+0000","last_undegraded":"2024-03-06T07:26:24.661458+0000","last_fullsized":"2024-03-06T07:26:24.661458+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658283+0000","last_change":"2024-03-06T07:26:17.641468+0000","last_active":"2024-03-06T07:26:24.658283+0000","last_peered":"2024-03-06T07:26:24.658283+0000","last_clean":"2024-03-06T07:26:24.658283+0000","last_became_active":"2024-03-06T07:26:17.641155+0000","last_became_peered":"2024-03-06T07:26:17.641155+0000","last_unstale":"2024-03-06T07:26:24.658283+0000","last_undegraded":"2024-03-06T07:26:24.658283+0000","last_fullsized":"2024-03-06T07:26:24.658283+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:27.464355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"148'8","reported_seq":24,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664388+0000","last_change":"2024-03-06T07:26:16.620548+0000","last_active":"2024-03-06T07:26:24.664388+0000","last_peered":"2024-03-06T07:26:24.664388+0000","last_clean":"2024-03-06T07:26:24.664388+0000","last_became_active":"2024-03-06T07:26:16.620289+0000","last_became_peered":"2024-03-06T07:26:16.620289+0000","last_unstale":"2024-03-06T07:26:24.664388+0000","last_undegraded":"2024-03-06T07:26:24.664388+0000","last_fullsized":"2024-03-06T07:26:24.664388+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:08.155799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.660981+0000","last_change":"2024-03-06T07:26:24.660981+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.660981+0000","last_undegraded":"2024-03-06T07:26:24.660981+0000","last_fullsized":"2024-03-06T07:26:24.660981+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664277+0000","last_change":"2024-03-06T07:26:17.629187+0000","last_active":"2024-03-06T07:26:24.664277+0000","last_peered":"2024-03-06T07:26:24.664277+0000","last_clean":"2024-03-06T07:26:24.664277+0000","last_became_active":"2024-03-06T07:26:17.628903+0000","last_became_peered":"2024-03-06T07:26:17.628903+0000","last_unstale":"2024-03-06T07:26:24.664277+0000","last_undegraded":"2024-03-06T07:26:24.664277+0000","last_fullsized":"2024-03-06T07:26:24.664277+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:25.173456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"148'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662984+0000","last_change":"2024-03-06T07:26:16.617108+0000","last_active":"2024-03-06T07:26:24.662984+0000","last_peered":"2024-03-06T07:26:24.662984+0000","last_clean":"2024-03-06T07:26:24.662984+0000","last_became_active":"2024-03-06T07:26:16.616776+0000","last_became_peered":"2024-03-06T07:26:16.616776+0000","last_unstale":"2024-03-06T07:26:24.662984+0000","last_undegraded":"2024-03-06T07:26:24.662984+0000","last_fullsized":"2024-03-06T07:26:24.662984+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:01.223054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.664369+0000","last_change":"2024-03-06T07:26:24.664369+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.664369+0000","last_undegraded":"2024-03-06T07:26:24.664369+0000","last_fullsized":"2024-03-06T07:26:24.664369+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665818+0000","last_change":"2024-03-06T07:26:17.634792+0000","last_active":"2024-03-06T07:26:24.665818+0000","last_peered":"2024-03-06T07:26:24.665818+0000","last_clean":"2024-03-06T07:26:24.665818+0000","last_became_active":"2024-03-06T07:26:17.634367+0000","last_became_peered":"2024-03-06T07:26:17.634367+0000","last_unstale":"2024-03-06T07:26:24.665818+0000","last_undegraded":"2024-03-06T07:26:24.665818+0000","last_fullsized":"2024-03-06T07:26:24.665818+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:11.386543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665495+0000","last_change":"2024-03-06T07:26:16.619012+0000","last_active":"2024-03-06T07:26:24.665495+0000","last_peered":"2024-03-06T07:26:24.665495+0000","last_clean":"2024-03-06T07:26:24.665495+0000","last_became_active":"2024-03-06T07:26:16.617718+0000","last_became_peered":"2024-03-06T07:26:16.617718+0000","last_unstale":"2024-03-06T07:26:24.665495+0000","last_undegraded":"2024-03-06T07:26:24.665495+0000","last_fullsized":"2024-03-06T07:26:24.665495+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:08.220437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.666049+0000","last_change":"2024-03-06T07:26:24.666049+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.666049+0000","last_undegraded":"2024-03-06T07:26:24.666049+0000","last_fullsized":"2024-03-06T07:26:24.666049+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666499+0000","last_change":"2024-03-06T07:26:17.633394+0000","last_active":"2024-03-06T07:26:24.666499+0000","last_peered":"2024-03-06T07:26:24.666499+0000","last_clean":"2024-03-06T07:26:24.666499+0000","last_became_active":"2024-03-06T07:26:17.632438+0000","last_became_peered":"2024-03-06T07:26:17.632438+0000","last_unstale":"2024-03-06T07:26:24.666499+0000","last_undegraded":"2024-03-06T07:26:24.666499+0000","last_fullsized":"2024-03-06T07:26:24.666499+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:12.024002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"148'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668243+0000","last_change":"2024-03-06T07:26:16.628646+0000","last_active":"2024-03-06T07:26:24.668243+0000","last_peered":"2024-03-06T07:26:24.668243+0000","last_clean":"2024-03-06T07:26:24.668243+0000","last_became_active":"2024-03-06T07:26:16.627947+0000","last_became_peered":"2024-03-06T07:26:16.627947+0000","last_unstale":"2024-03-06T07:26:24.668243+0000","last_undegraded":"2024-03-06T07:26:24.668243+0000","last_fullsized":"2024-03-06T07:26:24.668243+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:25.875046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665349+0000","last_change":"2024-03-06T07:26:24.665349+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665349+0000","last_undegraded":"2024-03-06T07:26:24.665349+0000","last_fullsized":"2024-03-06T07:26:24.665349+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666918+0000","last_change":"2024-03-06T07:26:17.634254+0000","last_active":"2024-03-06T07:26:24.666918+0000","last_peered":"2024-03-06T07:26:24.666918+0000","last_clean":"2024-03-06T07:26:24.666918+0000","last_became_active":"2024-03-06T07:26:17.633961+0000","last_became_peered":"2024-03-06T07:26:17.633961+0000","last_unstale":"2024-03-06T07:26:24.666918+0000","last_undegraded":"2024-03-06T07:26:24.666918+0000","last_fullsized":"2024-03-06T07:26:24.666918+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:59.873611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.665244+0000","last_change":"2024-03-06T07:26:16.619532+0000","last_active":"2024-03-06T07:26:24.665244+0000","last_peered":"2024-03-06T07:26:24.665244+0000","last_clean":"2024-03-06T07:26:24.665244+0000","last_became_active":"2024-03-06T07:26:16.619261+0000","last_became_peered":"2024-03-06T07:26:16.619261+0000","last_unstale":"2024-03-06T07:26:24.665244+0000","last_undegraded":"2024-03-06T07:26:24.665244+0000","last_fullsized":"2024-03-06T07:26:24.665244+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:34.031068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661217+0000","last_change":"2024-03-06T07:26:24.661217+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661217+0000","last_undegraded":"2024-03-06T07:26:24.661217+0000","last_fullsized":"2024-03-06T07:26:24.661217+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662447+0000","last_change":"2024-03-06T07:26:17.637879+0000","last_active":"2024-03-06T07:26:24.662447+0000","last_peered":"2024-03-06T07:26:24.662447+0000","last_clean":"2024-03-06T07:26:24.662447+0000","last_became_active":"2024-03-06T07:26:17.637094+0000","last_became_peered":"2024-03-06T07:26:17.637094+0000","last_unstale":"2024-03-06T07:26:24.662447+0000","last_undegraded":"2024-03-06T07:26:24.662447+0000","last_fullsized":"2024-03-06T07:26:24.662447+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:34.084422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668442+0000","last_change":"2024-03-06T07:26:16.622856+0000","last_active":"2024-03-06T07:26:24.668442+0000","last_peered":"2024-03-06T07:26:24.668442+0000","last_clean":"2024-03-06T07:26:24.668442+0000","last_became_active":"2024-03-06T07:26:16.622421+0000","last_became_peered":"2024-03-06T07:26:16.622421+0000","last_unstale":"2024-03-06T07:26:24.668442+0000","last_undegraded":"2024-03-06T07:26:24.668442+0000","last_fullsized":"2024-03-06T07:26:24.668442+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:40.651795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661343+0000","last_change":"2024-03-06T07:26:24.661343+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661343+0000","last_undegraded":"2024-03-06T07:26:24.661343+0000","last_fullsized":"2024-03-06T07:26:24.661343+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668215+0000","last_change":"2024-03-06T07:26:17.629188+0000","last_active":"2024-03-06T07:26:24.668215+0000","last_peered":"2024-03-06T07:26:24.668215+0000","last_clean":"2024-03-06T07:26:24.668215+0000","last_became_active":"2024-03-06T07:26:17.628881+0000","last_became_peered":"2024-03-06T07:26:17.628881+0000","last_unstale":"2024-03-06T07:26:24.668215+0000","last_undegraded":"2024-03-06T07:26:24.668215+0000","last_fullsized":"2024-03-06T07:26:24.668215+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:44.957682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666547+0000","last_change":"2024-03-06T07:26:16.625755+0000","last_active":"2024-03-06T07:26:24.666547+0000","last_peered":"2024-03-06T07:26:24.666547+0000","last_clean":"2024-03-06T07:26:24.666547+0000","last_became_active":"2024-03-06T07:26:16.625489+0000","last_became_peered":"2024-03-06T07:26:16.625489+0000","last_unstale":"2024-03-06T07:26:24.666547+0000","last_undegraded":"2024-03-06T07:26:24.666547+0000","last_fullsized":"2024-03-06T07:26:24.666547+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:03.053080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.666734+0000","last_change":"2024-03-06T07:26:24.666734+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.666734+0000","last_undegraded":"2024-03-06T07:26:24.666734+0000","last_fullsized":"2024-03-06T07:26:24.666734+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668192+0000","last_change":"2024-03-06T07:26:17.629894+0000","last_active":"2024-03-06T07:26:24.668192+0000","last_peered":"2024-03-06T07:26:24.668192+0000","last_clean":"2024-03-06T07:26:24.668192+0000","last_became_active":"2024-03-06T07:26:17.629624+0000","last_became_peered":"2024-03-06T07:26:17.629624+0000","last_unstale":"2024-03-06T07:26:24.668192+0000","last_undegraded":"2024-03-06T07:26:24.668192+0000","last_fullsized":"2024-03-06T07:26:24.668192+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:48.520981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658129+0000","last_change":"2024-03-06T07:26:16.618167+0000","last_active":"2024-03-06T07:26:24.658129+0000","last_peered":"2024-03-06T07:26:24.658129+0000","last_clean":"2024-03-06T07:26:24.658129+0000","last_became_active":"2024-03-06T07:26:16.617893+0000","last_became_peered":"2024-03-06T07:26:16.617893+0000","last_unstale":"2024-03-06T07:26:24.658129+0000","last_undegraded":"2024-03-06T07:26:24.658129+0000","last_fullsized":"2024-03-06T07:26:24.658129+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:33.738360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.660975+0000","last_change":"2024-03-06T07:26:24.660975+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.660975+0000","last_undegraded":"2024-03-06T07:26:24.660975+0000","last_fullsized":"2024-03-06T07:26:24.660975+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664802+0000","last_change":"2024-03-06T07:26:17.632248+0000","last_active":"2024-03-06T07:26:24.664802+0000","last_peered":"2024-03-06T07:26:24.664802+0000","last_clean":"2024-03-06T07:26:24.664802+0000","last_became_active":"2024-03-06T07:26:17.631912+0000","last_became_peered":"2024-03-06T07:26:17.631912+0000","last_unstale":"2024-03-06T07:26:24.664802+0000","last_undegraded":"2024-03-06T07:26:24.664802+0000","last_fullsized":"2024-03-06T07:26:24.664802+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:29.113308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664561+0000","last_change":"2024-03-06T07:26:16.616949+0000","last_active":"2024-03-06T07:26:24.664561+0000","last_peered":"2024-03-06T07:26:24.664561+0000","last_clean":"2024-03-06T07:26:24.664561+0000","last_became_active":"2024-03-06T07:26:16.616670+0000","last_became_peered":"2024-03-06T07:26:16.616670+0000","last_unstale":"2024-03-06T07:26:24.664561+0000","last_undegraded":"2024-03-06T07:26:24.664561+0000","last_fullsized":"2024-03-06T07:26:24.664561+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:59.587645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661489+0000","last_change":"2024-03-06T07:26:24.661489+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661489+0000","last_undegraded":"2024-03-06T07:26:24.661489+0000","last_fullsized":"2024-03-06T07:26:24.661489+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666760+0000","last_change":"2024-03-06T07:26:17.628994+0000","last_active":"2024-03-06T07:26:24.666760+0000","last_peered":"2024-03-06T07:26:24.666760+0000","last_clean":"2024-03-06T07:26:24.666760+0000","last_became_active":"2024-03-06T07:26:17.628722+0000","last_became_peered":"2024-03-06T07:26:17.628722+0000","last_unstale":"2024-03-06T07:26:24.666760+0000","last_undegraded":"2024-03-06T07:26:24.666760+0000","last_fullsized":"2024-03-06T07:26:24.666760+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:43.090011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"148'2","reported_seq":18,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.668358+0000","last_change":"2024-03-06T07:26:16.620011+0000","last_active":"2024-03-06T07:26:24.668358+0000","last_peered":"2024-03-06T07:26:24.668358+0000","last_clean":"2024-03-06T07:26:24.668358+0000","last_became_active":"2024-03-06T07:26:16.619736+0000","last_became_peered":"2024-03-06T07:26:16.619736+0000","last_unstale":"2024-03-06T07:26:24.668358+0000","last_undegraded":"2024-03-06T07:26:24.668358+0000","last_fullsized":"2024-03-06T07:26:24.668358+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:24.169321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.663886+0000","last_change":"2024-03-06T07:26:24.663886+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.663886+0000","last_undegraded":"2024-03-06T07:26:24.663886+0000","last_fullsized":"2024-03-06T07:26:24.663886+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664813+0000","last_change":"2024-03-06T07:26:17.628485+0000","last_active":"2024-03-06T07:26:24.664813+0000","last_peered":"2024-03-06T07:26:24.664813+0000","last_clean":"2024-03-06T07:26:24.664813+0000","last_became_active":"2024-03-06T07:26:17.628204+0000","last_became_peered":"2024-03-06T07:26:17.628204+0000","last_unstale":"2024-03-06T07:26:24.664813+0000","last_undegraded":"2024-03-06T07:26:24.664813+0000","last_fullsized":"2024-03-06T07:26:24.664813+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:56.134884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"149'1","reported_seq":17,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662755+0000","last_change":"2024-03-06T07:26:16.627066+0000","last_active":"2024-03-06T07:26:24.662755+0000","last_peered":"2024-03-06T07:26:24.662755+0000","last_clean":"2024-03-06T07:26:24.662755+0000","last_became_active":"2024-03-06T07:26:16.626418+0000","last_became_peered":"2024-03-06T07:26:16.626418+0000","last_unstale":"2024-03-06T07:26:24.662755+0000","last_undegraded":"2024-03-06T07:26:24.662755+0000","last_fullsized":"2024-03-06T07:26:24.662755+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:37.036435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.661600+0000","last_change":"2024-03-06T07:26:24.661600+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.661600+0000","last_undegraded":"2024-03-06T07:26:24.661600+0000","last_fullsized":"2024-03-06T07:26:24.661600+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658129+0000","last_change":"2024-03-06T07:26:17.626185+0000","last_active":"2024-03-06T07:26:24.658129+0000","last_peered":"2024-03-06T07:26:24.658129+0000","last_clean":"2024-03-06T07:26:24.658129+0000","last_became_active":"2024-03-06T07:26:17.625895+0000","last_became_peered":"2024-03-06T07:26:17.625895+0000","last_unstale":"2024-03-06T07:26:24.658129+0000","last_undegraded":"2024-03-06T07:26:24.658129+0000","last_fullsized":"2024-03-06T07:26:24.658129+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:16.007003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662590+0000","last_change":"2024-03-06T07:26:16.625471+0000","last_active":"2024-03-06T07:26:24.662590+0000","last_peered":"2024-03-06T07:26:24.662590+0000","last_clean":"2024-03-06T07:26:24.662590+0000","last_became_active":"2024-03-06T07:26:16.625198+0000","last_became_peered":"2024-03-06T07:26:16.625198+0000","last_unstale":"2024-03-06T07:26:24.662590+0000","last_undegraded":"2024-03-06T07:26:24.662590+0000","last_fullsized":"2024-03-06T07:26:24.662590+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:37.984120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.664299+0000","last_change":"2024-03-06T07:26:24.664299+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.664299+0000","last_undegraded":"2024-03-06T07:26:24.664299+0000","last_fullsized":"2024-03-06T07:26:24.664299+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664790+0000","last_change":"2024-03-06T07:26:17.630631+0000","last_active":"2024-03-06T07:26:24.664790+0000","last_peered":"2024-03-06T07:26:24.664790+0000","last_clean":"2024-03-06T07:26:24.664790+0000","last_became_active":"2024-03-06T07:26:17.630365+0000","last_became_peered":"2024-03-06T07:26:17.630365+0000","last_unstale":"2024-03-06T07:26:24.664790+0000","last_undegraded":"2024-03-06T07:26:24.664790+0000","last_fullsized":"2024-03-06T07:26:24.664790+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:27.398343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658871+0000","last_change":"2024-03-06T07:26:16.611334+0000","last_active":"2024-03-06T07:26:24.658871+0000","last_peered":"2024-03-06T07:26:24.658871+0000","last_clean":"2024-03-06T07:26:24.658871+0000","last_became_active":"2024-03-06T07:26:16.610646+0000","last_became_peered":"2024-03-06T07:26:16.610646+0000","last_unstale":"2024-03-06T07:26:24.658871+0000","last_undegraded":"2024-03-06T07:26:24.658871+0000","last_fullsized":"2024-03-06T07:26:24.658871+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:32.639393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.664690+0000","last_change":"2024-03-06T07:26:24.664690+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.664690+0000","last_undegraded":"2024-03-06T07:26:24.664690+0000","last_fullsized":"2024-03-06T07:26:24.664690+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663654+0000","last_change":"2024-03-06T07:26:17.632379+0000","last_active":"2024-03-06T07:26:24.663654+0000","last_peered":"2024-03-06T07:26:24.663654+0000","last_clean":"2024-03-06T07:26:24.663654+0000","last_became_active":"2024-03-06T07:26:17.631899+0000","last_became_peered":"2024-03-06T07:26:17.631899+0000","last_unstale":"2024-03-06T07:26:24.663654+0000","last_undegraded":"2024-03-06T07:26:24.663654+0000","last_fullsized":"2024-03-06T07:26:24.663654+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:00.591780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.663294+0000","last_change":"2024-03-06T07:26:16.623898+0000","last_active":"2024-03-06T07:26:24.663294+0000","last_peered":"2024-03-06T07:26:24.663294+0000","last_clean":"2024-03-06T07:26:24.663294+0000","last_became_active":"2024-03-06T07:26:16.623602+0000","last_became_peered":"2024-03-06T07:26:16.623602+0000","last_unstale":"2024-03-06T07:26:24.663294+0000","last_undegraded":"2024-03-06T07:26:24.663294+0000","last_fullsized":"2024-03-06T07:26:24.663294+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:56.210974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665305+0000","last_change":"2024-03-06T07:26:24.665305+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665305+0000","last_undegraded":"2024-03-06T07:26:24.665305+0000","last_fullsized":"2024-03-06T07:26:24.665305+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664900+0000","last_change":"2024-03-06T07:26:17.632856+0000","last_active":"2024-03-06T07:26:24.664900+0000","last_peered":"2024-03-06T07:26:24.664900+0000","last_clean":"2024-03-06T07:26:24.664900+0000","last_became_active":"2024-03-06T07:26:17.632027+0000","last_became_peered":"2024-03-06T07:26:17.632027+0000","last_unstale":"2024-03-06T07:26:24.664900+0000","last_undegraded":"2024-03-06T07:26:24.664900+0000","last_fullsized":"2024-03-06T07:26:24.664900+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:38.988254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662329+0000","last_change":"2024-03-06T07:26:16.627698+0000","last_active":"2024-03-06T07:26:24.662329+0000","last_peered":"2024-03-06T07:26:24.662329+0000","last_clean":"2024-03-06T07:26:24.662329+0000","last_became_active":"2024-03-06T07:26:16.627318+0000","last_became_peered":"2024-03-06T07:26:16.627318+0000","last_unstale":"2024-03-06T07:26:24.662329+0000","last_undegraded":"2024-03-06T07:26:24.662329+0000","last_fullsized":"2024-03-06T07:26:24.662329+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:08.249739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.665228+0000","last_change":"2024-03-06T07:26:24.665228+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.665228+0000","last_undegraded":"2024-03-06T07:26:24.665228+0000","last_fullsized":"2024-03-06T07:26:24.665228+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662577+0000","last_change":"2024-03-06T07:26:17.637676+0000","last_active":"2024-03-06T07:26:24.662577+0000","last_peered":"2024-03-06T07:26:24.662577+0000","last_clean":"2024-03-06T07:26:24.662577+0000","last_became_active":"2024-03-06T07:26:17.637375+0000","last_became_peered":"2024-03-06T07:26:17.637375+0000","last_unstale":"2024-03-06T07:26:24.662577+0000","last_undegraded":"2024-03-06T07:26:24.662577+0000","last_fullsized":"2024-03-06T07:26:24.662577+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:18.556057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.666678+0000","last_change":"2024-03-06T07:26:16.620202+0000","last_active":"2024-03-06T07:26:24.666678+0000","last_peered":"2024-03-06T07:26:24.666678+0000","last_clean":"2024-03-06T07:26:24.666678+0000","last_became_active":"2024-03-06T07:26:16.619897+0000","last_became_peered":"2024-03-06T07:26:16.619897+0000","last_unstale":"2024-03-06T07:26:24.666678+0000","last_undegraded":"2024-03-06T07:26:24.666678+0000","last_fullsized":"2024-03-06T07:26:24.666678+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:48.709582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.666356+0000","last_change":"2024-03-06T07:26:24.666356+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.666356+0000","last_undegraded":"2024-03-06T07:26:24.666356+0000","last_fullsized":"2024-03-06T07:26:24.666356+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.658331+0000","last_change":"2024-03-06T07:26:17.625716+0000","last_active":"2024-03-06T07:26:24.658331+0000","last_peered":"2024-03-06T07:26:24.658331+0000","last_clean":"2024-03-06T07:26:24.658331+0000","last_became_active":"2024-03-06T07:26:17.625429+0000","last_became_peered":"2024-03-06T07:26:17.625429+0000","last_unstale":"2024-03-06T07:26:24.658331+0000","last_undegraded":"2024-03-06T07:26:24.658331+0000","last_fullsized":"2024-03-06T07:26:24.658331+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:38.596798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.667604+0000","last_change":"2024-03-06T07:26:16.620839+0000","last_active":"2024-03-06T07:26:24.667604+0000","last_peered":"2024-03-06T07:26:24.667604+0000","last_clean":"2024-03-06T07:26:24.667604+0000","last_became_active":"2024-03-06T07:26:16.620561+0000","last_became_peered":"2024-03-06T07:26:16.620561+0000","last_unstale":"2024-03-06T07:26:24.667604+0000","last_undegraded":"2024-03-06T07:26:24.667604+0000","last_fullsized":"2024-03-06T07:26:24.667604+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:26.394208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:25.656216+0000","last_change":"2024-03-06T07:26:25.656216+0000","last_active":"2024-03-06T07:26:25.656216+0000","last_peered":"2024-03-06T07:26:25.656216+0000","last_clean":"2024-03-06T07:26:25.656216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:25.656216+0000","last_undegraded":"2024-03-06T07:26:25.656216+0000","last_fullsized":"2024-03-06T07:26:25.656216+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"creating+peering","last_fresh":"2024-03-06T07:26:24.663905+0000","last_change":"2024-03-06T07:26:24.663905+0000","last_active":"2024-03-06T07:26:24.654657+0000","last_peered":"2024-03-06T07:26:24.654657+0000","last_clean":"2024-03-06T07:26:24.654657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.663905+0000","last_undegraded":"2024-03-06T07:26:24.663905+0000","last_fullsized":"2024-03-06T07:26:24.663905+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664986+0000","last_change":"2024-03-06T07:26:17.632976+0000","last_active":"2024-03-06T07:26:24.664986+0000","last_peered":"2024-03-06T07:26:24.664986+0000","last_clean":"2024-03-06T07:26:24.664986+0000","last_became_active":"2024-03-06T07:26:17.632721+0000","last_became_peered":"2024-03-06T07:26:17.632721+0000","last_unstale":"2024-03-06T07:26:24.664986+0000","last_undegraded":"2024-03-06T07:26:24.664986+0000","last_fullsized":"2024-03-06T07:26:24.664986+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:56.134884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":16,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.662665+0000","last_change":"2024-03-06T07:26:16.626430+0000","last_active":"2024-03-06T07:26:24.662665+0000","last_peered":"2024-03-06T07:26:24.662665+0000","last_clean":"2024-03-06T07:26:24.662665+0000","last_became_active":"2024-03-06T07:26:16.625971+0000","last_became_peered":"2024-03-06T07:26:16.625971+0000","last_unstale":"2024-03-06T07:26:24.662665+0000","last_undegraded":"2024-03-06T07:26:24.662665+0000","last_fullsized":"2024-03-06T07:26:24.662665+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:28.109173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":419,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-06T07:26:24.664587+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:26:24.664587+0000","last_peered":"2024-03-06T07:26:24.664587+0000","last_clean":"2024-03-06T07:26:24.664587+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:26:24.664587+0000","last_undegraded":"2024-03-06T07:26:24.664587+0000","last_fullsized":"2024-03-06T07:26:24.664587+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":1941504,"data_stored":1496336,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574924,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33016,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1875968,"data_stored":905968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574924,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1884160,"data_stored":906092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574925,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1892352,"data_stored":906732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574924,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33036,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338804,"statfs":{"total":96636764160,"available":96602935296,"internally_reserved":0,"allocated":1896448,"data_stored":908184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574924,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33016,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1875968,"data_stored":905968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1888256,"data_stored":907580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574925,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33088,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338752,"statfs":{"total":96636764160,"available":96602882048,"internally_reserved":0,"allocated":1949696,"data_stored":1496440,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:30.461 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.3d is not active+clean 2024-03-06T07:26:30.461 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.461 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.3c is not active+clean 2024-03-06T07:26:30.461 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.462 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.3f is not active+clean 2024-03-06T07:26:30.462 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.462 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.3e is not active+clean 2024-03-06T07:26:30.462 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.462 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.39 is not active+clean 2024-03-06T07:26:30.462 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.462 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.38 is not active+clean 2024-03-06T07:26:30.462 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.462 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.3b is not active+clean 2024-03-06T07:26:30.462 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.462 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.3a is not active+clean 2024-03-06T07:26:30.462 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.463 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.35 is not active+clean 2024-03-06T07:26:30.463 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.463 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.34 is not active+clean 2024-03-06T07:26:30.463 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.463 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.37 is not active+clean 2024-03-06T07:26:30.463 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.463 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.36 is not active+clean 2024-03-06T07:26:30.463 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.463 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.31 is not active+clean 2024-03-06T07:26:30.463 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.463 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.30 is not active+clean 2024-03-06T07:26:30.463 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.463 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.33 is not active+clean 2024-03-06T07:26:30.464 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.464 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.32 is not active+clean 2024-03-06T07:26:30.464 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.464 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.2d is not active+clean 2024-03-06T07:26:30.464 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.464 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.2c is not active+clean 2024-03-06T07:26:30.464 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.464 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.2f is not active+clean 2024-03-06T07:26:30.464 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.464 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.2e is not active+clean 2024-03-06T07:26:30.464 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.465 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.29 is not active+clean 2024-03-06T07:26:30.465 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.465 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.28 is not active+clean 2024-03-06T07:26:30.465 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.465 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.2b is not active+clean 2024-03-06T07:26:30.465 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.465 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.2a is not active+clean 2024-03-06T07:26:30.465 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.465 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.25 is not active+clean 2024-03-06T07:26:30.465 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.465 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.24 is not active+clean 2024-03-06T07:26:30.465 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.465 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.27 is not active+clean 2024-03-06T07:26:30.465 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.466 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.26 is not active+clean 2024-03-06T07:26:30.466 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.466 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.21 is not active+clean 2024-03-06T07:26:30.466 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.466 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.20 is not active+clean 2024-03-06T07:26:30.466 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.466 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.23 is not active+clean 2024-03-06T07:26:30.466 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.466 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.22 is not active+clean 2024-03-06T07:26:30.466 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.466 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.12 is not active+clean 2024-03-06T07:26:30.466 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.467 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.11 is not active+clean 2024-03-06T07:26:30.467 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.665307+0000', 'last_change': '2024-03-06T07:26:24.665307+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.665307+0000', 'last_undegraded': '2024-03-06T07:26:24.665307+0000', 'last_fullsized': '2024-03-06T07:26:24.665307+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:26:30.467 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.13 is not active+clean 2024-03-06T07:26:30.467 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.467 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.10 is not active+clean 2024-03-06T07:26:30.467 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.665984+0000', 'last_change': '2024-03-06T07:26:24.665984+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.665984+0000', 'last_undegraded': '2024-03-06T07:26:24.665984+0000', 'last_fullsized': '2024-03-06T07:26:24.665984+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:26:30.467 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.10 is not active+clean 2024-03-06T07:26:30.467 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.467 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.13 is not active+clean 2024-03-06T07:26:30.467 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.662178+0000', 'last_change': '2024-03-06T07:26:24.662178+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.662178+0000', 'last_undegraded': '2024-03-06T07:26:24.662178+0000', 'last_fullsized': '2024-03-06T07:26:24.662178+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:26:30.467 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.11 is not active+clean 2024-03-06T07:26:30.467 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.468 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.12 is not active+clean 2024-03-06T07:26:30.468 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.663658+0000', 'last_change': '2024-03-06T07:26:24.663658+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.663658+0000', 'last_undegraded': '2024-03-06T07:26:24.663658+0000', 'last_fullsized': '2024-03-06T07:26:24.663658+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:26:30.468 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.16 is not active+clean 2024-03-06T07:26:30.468 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.468 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.15 is not active+clean 2024-03-06T07:26:30.468 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.665856+0000', 'last_change': '2024-03-06T07:26:24.665856+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.665856+0000', 'last_undegraded': '2024-03-06T07:26:24.665856+0000', 'last_fullsized': '2024-03-06T07:26:24.665856+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:26:30.468 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.17 is not active+clean 2024-03-06T07:26:30.468 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.468 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.14 is not active+clean 2024-03-06T07:26:30.468 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.662223+0000', 'last_change': '2024-03-06T07:26:24.662223+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.662223+0000', 'last_undegraded': '2024-03-06T07:26:24.662223+0000', 'last_fullsized': '2024-03-06T07:26:24.662223+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:26:30.468 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.14 is not active+clean 2024-03-06T07:26:30.468 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.17 is not active+clean 2024-03-06T07:26:30.469 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.661284+0000', 'last_change': '2024-03-06T07:26:24.661284+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.661284+0000', 'last_undegraded': '2024-03-06T07:26:24.661284+0000', 'last_fullsized': '2024-03-06T07:26:24.661284+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:26:30.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.15 is not active+clean 2024-03-06T07:26:30.469 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.16 is not active+clean 2024-03-06T07:26:30.469 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.663648+0000', 'last_change': '2024-03-06T07:26:24.663648+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.663648+0000', 'last_undegraded': '2024-03-06T07:26:24.663648+0000', 'last_fullsized': '2024-03-06T07:26:24.663648+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:26:30.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.1a is not active+clean 2024-03-06T07:26:30.469 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.19 is not active+clean 2024-03-06T07:26:30.469 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.661586+0000', 'last_change': '2024-03-06T07:26:24.661586+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.661586+0000', 'last_undegraded': '2024-03-06T07:26:24.661586+0000', 'last_fullsized': '2024-03-06T07:26:24.661586+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:26:30.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.18 is not active+clean 2024-03-06T07:26:30.469 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.1b is not active+clean 2024-03-06T07:26:30.470 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.661596+0000', 'last_change': '2024-03-06T07:26:24.661596+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.661596+0000', 'last_undegraded': '2024-03-06T07:26:24.661596+0000', 'last_fullsized': '2024-03-06T07:26:24.661596+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:26:30.470 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.19 is not active+clean 2024-03-06T07:26:30.470 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.470 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.1a is not active+clean 2024-03-06T07:26:30.470 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.663853+0000', 'last_change': '2024-03-06T07:26:24.663853+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.663853+0000', 'last_undegraded': '2024-03-06T07:26:24.663853+0000', 'last_fullsized': '2024-03-06T07:26:24.663853+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:26:30.470 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.1e is not active+clean 2024-03-06T07:26:30.470 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.470 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.1d is not active+clean 2024-03-06T07:26:30.470 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.665684+0000', 'last_change': '2024-03-06T07:26:24.665684+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.665684+0000', 'last_undegraded': '2024-03-06T07:26:24.665684+0000', 'last_fullsized': '2024-03-06T07:26:24.665684+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:26:30.470 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.1f is not active+clean 2024-03-06T07:26:30.470 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.471 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.1c is not active+clean 2024-03-06T07:26:30.471 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.661600+0000', 'last_change': '2024-03-06T07:26:24.661600+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.661600+0000', 'last_undegraded': '2024-03-06T07:26:24.661600+0000', 'last_fullsized': '2024-03-06T07:26:24.661600+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:26:30.471 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.1c is not active+clean 2024-03-06T07:26:30.471 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.471 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.1f is not active+clean 2024-03-06T07:26:30.471 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.667139+0000', 'last_change': '2024-03-06T07:26:24.667139+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.667139+0000', 'last_undegraded': '2024-03-06T07:26:24.667139+0000', 'last_fullsized': '2024-03-06T07:26:24.667139+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:26:30.471 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.1d is not active+clean 2024-03-06T07:26:30.471 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.471 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.1e is not active+clean 2024-03-06T07:26:30.471 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.661458+0000', 'last_change': '2024-03-06T07:26:24.661458+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.661458+0000', 'last_undegraded': '2024-03-06T07:26:24.661458+0000', 'last_fullsized': '2024-03-06T07:26:24.661458+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:26:30.471 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.2 is not active+clean 2024-03-06T07:26:30.471 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.1 is not active+clean 2024-03-06T07:26:30.472 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.660981+0000', 'last_change': '2024-03-06T07:26:24.660981+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.660981+0000', 'last_undegraded': '2024-03-06T07:26:24.660981+0000', 'last_fullsized': '2024-03-06T07:26:24.660981+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:26:30.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.3 is not active+clean 2024-03-06T07:26:30.472 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.0 is not active+clean 2024-03-06T07:26:30.472 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.664369+0000', 'last_change': '2024-03-06T07:26:24.664369+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.664369+0000', 'last_undegraded': '2024-03-06T07:26:24.664369+0000', 'last_fullsized': '2024-03-06T07:26:24.664369+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:26:30.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.0 is not active+clean 2024-03-06T07:26:30.472 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.3 is not active+clean 2024-03-06T07:26:30.472 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.666049+0000', 'last_change': '2024-03-06T07:26:24.666049+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.666049+0000', 'last_undegraded': '2024-03-06T07:26:24.666049+0000', 'last_fullsized': '2024-03-06T07:26:24.666049+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:26:30.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.1 is not active+clean 2024-03-06T07:26:30.472 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.473 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.2 is not active+clean 2024-03-06T07:26:30.473 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.665349+0000', 'last_change': '2024-03-06T07:26:24.665349+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.665349+0000', 'last_undegraded': '2024-03-06T07:26:24.665349+0000', 'last_fullsized': '2024-03-06T07:26:24.665349+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:26:30.473 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.6 is not active+clean 2024-03-06T07:26:30.473 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.473 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.5 is not active+clean 2024-03-06T07:26:30.473 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.661217+0000', 'last_change': '2024-03-06T07:26:24.661217+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.661217+0000', 'last_undegraded': '2024-03-06T07:26:24.661217+0000', 'last_fullsized': '2024-03-06T07:26:24.661217+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:26:30.473 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.7 is not active+clean 2024-03-06T07:26:30.473 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.473 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.4 is not active+clean 2024-03-06T07:26:30.473 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.661343+0000', 'last_change': '2024-03-06T07:26:24.661343+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.661343+0000', 'last_undegraded': '2024-03-06T07:26:24.661343+0000', 'last_fullsized': '2024-03-06T07:26:24.661343+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:26:30.473 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.4 is not active+clean 2024-03-06T07:26:30.473 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.473 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.7 is not active+clean 2024-03-06T07:26:30.474 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.666734+0000', 'last_change': '2024-03-06T07:26:24.666734+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.666734+0000', 'last_undegraded': '2024-03-06T07:26:24.666734+0000', 'last_fullsized': '2024-03-06T07:26:24.666734+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:26:30.474 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.5 is not active+clean 2024-03-06T07:26:30.474 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.474 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.6 is not active+clean 2024-03-06T07:26:30.474 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.660975+0000', 'last_change': '2024-03-06T07:26:24.660975+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.660975+0000', 'last_undegraded': '2024-03-06T07:26:24.660975+0000', 'last_fullsized': '2024-03-06T07:26:24.660975+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:26:30.474 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.a is not active+clean 2024-03-06T07:26:30.474 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.474 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.9 is not active+clean 2024-03-06T07:26:30.474 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.661489+0000', 'last_change': '2024-03-06T07:26:24.661489+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.661489+0000', 'last_undegraded': '2024-03-06T07:26:24.661489+0000', 'last_fullsized': '2024-03-06T07:26:24.661489+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:26:30.474 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.b is not active+clean 2024-03-06T07:26:30.474 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.474 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.8 is not active+clean 2024-03-06T07:26:30.474 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.663886+0000', 'last_change': '2024-03-06T07:26:24.663886+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.663886+0000', 'last_undegraded': '2024-03-06T07:26:24.663886+0000', 'last_fullsized': '2024-03-06T07:26:24.663886+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:26:30.475 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.8 is not active+clean 2024-03-06T07:26:30.475 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.475 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.b is not active+clean 2024-03-06T07:26:30.475 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.661600+0000', 'last_change': '2024-03-06T07:26:24.661600+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.661600+0000', 'last_undegraded': '2024-03-06T07:26:24.661600+0000', 'last_fullsized': '2024-03-06T07:26:24.661600+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:26:30.475 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.9 is not active+clean 2024-03-06T07:26:30.475 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.475 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.a is not active+clean 2024-03-06T07:26:30.475 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.664299+0000', 'last_change': '2024-03-06T07:26:24.664299+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.664299+0000', 'last_undegraded': '2024-03-06T07:26:24.664299+0000', 'last_fullsized': '2024-03-06T07:26:24.664299+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:26:30.475 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.e is not active+clean 2024-03-06T07:26:30.475 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.476 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.d is not active+clean 2024-03-06T07:26:30.476 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.664690+0000', 'last_change': '2024-03-06T07:26:24.664690+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.664690+0000', 'last_undegraded': '2024-03-06T07:26:24.664690+0000', 'last_fullsized': '2024-03-06T07:26:24.664690+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:26:30.476 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.f is not active+clean 2024-03-06T07:26:30.476 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.476 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.c is not active+clean 2024-03-06T07:26:30.476 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.665305+0000', 'last_change': '2024-03-06T07:26:24.665305+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.665305+0000', 'last_undegraded': '2024-03-06T07:26:24.665305+0000', 'last_fullsized': '2024-03-06T07:26:24.665305+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:26:30.476 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.c is not active+clean 2024-03-06T07:26:30.476 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.476 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.f is not active+clean 2024-03-06T07:26:30.476 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.665228+0000', 'last_change': '2024-03-06T07:26:24.665228+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.665228+0000', 'last_undegraded': '2024-03-06T07:26:24.665228+0000', 'last_fullsized': '2024-03-06T07:26:24.665228+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:26:30.476 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.d is not active+clean 2024-03-06T07:26:30.477 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.477 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.e is not active+clean 2024-03-06T07:26:30.477 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.666356+0000', 'last_change': '2024-03-06T07:26:24.666356+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.666356+0000', 'last_undegraded': '2024-03-06T07:26:24.666356+0000', 'last_fullsized': '2024-03-06T07:26:24.666356+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:26:30.477 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.1b is not active+clean 2024-03-06T07:26:30.477 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:25.656216+0000', 'last_change': '2024-03-06T07:26:25.656216+0000', 'last_active': '2024-03-06T07:26:25.656216+0000', 'last_peered': '2024-03-06T07:26:25.656216+0000', 'last_clean': '2024-03-06T07:26:25.656216+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:25.656216+0000', 'last_undegraded': '2024-03-06T07:26:25.656216+0000', 'last_fullsized': '2024-03-06T07:26:25.656216+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:25.656216+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:30.477 INFO:tasks.cephfs.filesystem.ceph_manager:PG 25.18 is not active+clean 2024-03-06T07:26:30.477 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '25.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 150, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:24.663905+0000', 'last_change': '2024-03-06T07:26:24.663905+0000', 'last_active': '2024-03-06T07:26:24.654657+0000', 'last_peered': '2024-03-06T07:26:24.654657+0000', 'last_clean': '2024-03-06T07:26:24.654657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:24.663905+0000', 'last_undegraded': '2024-03-06T07:26:24.663905+0000', 'last_fullsized': '2024-03-06T07:26:24.663905+0000', 'mapping_epoch': 150, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 150, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:24.654657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:26:30.478 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:26:30.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.593+0000 7feeb088d640 1 Processor -- start 2024-03-06T07:26:30.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.593+0000 7feeb088d640 1 -- start start 2024-03-06T07:26:30.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.594+0000 7feeb088d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac152650 0x7feeac155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.594+0000 7feeb088d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feeac0c6b10 0x7feeac158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.594+0000 7feeb088d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac0c2fc0 con 0x7feeac0bf9d0 2024-03-06T07:26:30.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.594+0000 7feeb088d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeac155800 con 0x7feeac0c6b10 2024-03-06T07:26:30.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.594+0000 7feeb088d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeac0c6690 con 0x7feeac152650 2024-03-06T07:26:30.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.594+0000 7feea9d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feeac0c6b10 0x7feeac158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.594+0000 7feea9d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feeac0c6b10 0x7feeac158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58656/0 (socket says 172.21.15.1:58656) 2024-03-06T07:26:30.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.594+0000 7feea9d9b640 1 -- 172.21.15.1:0/2450480605 learned_addr learned my addr 172.21.15.1:0/2450480605 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.594+0000 7feeaad9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7feeac0bf9d0 0x7feeac0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60362/0 (socket says 172.21.15.1:60362) 2024-03-06T07:26:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.594+0000 7feeaa59c640 1 --2- 172.21.15.1:0/2450480605 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac152650 0x7feeac155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.594+0000 7feea9d9b640 1 -- 172.21.15.1:0/2450480605 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac152650 msgr2=0x7feeac155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.594+0000 7feea9d9b640 1 --2- 172.21.15.1:0/2450480605 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac152650 0x7feeac155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.594+0000 7feea9d9b640 1 -- 172.21.15.1:0/2450480605 >> v1:172.21.15.1:6789/0 conn(0x7feeac0bf9d0 legacy=0x7feeac0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.594+0000 7feea9d9b640 1 -- 172.21.15.1:0/2450480605 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac0c7280 con 0x7feeac0c6b10 2024-03-06T07:26:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.594+0000 7feea959a640 1 -- 172.21.15.1:0/2450480605 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971315117 0 0) 0x7feeac0c2fc0 con 0x7feeac0bf9d0 2024-03-06T07:26:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.595+0000 7feea9d9b640 1 --2- 172.21.15.1:0/2450480605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0c6b10 0x7feeac158690 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7feea0009870 tx=0x7feea002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda8e8f08209dae0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.595+0000 7feea959a640 1 -- 172.21.15.1:0/2450480605 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea0035020 con 0x7feeac0c6b10 2024-03-06T07:26:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.595+0000 7feea959a640 1 -- 172.21.15.1:0/2450480605 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea0031640 con 0x7feeac0c6b10 2024-03-06T07:26:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.595+0000 7feea959a640 1 -- 172.21.15.1:0/2450480605 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea002d040 con 0x7feeac0c6b10 2024-03-06T07:26:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.596+0000 7feeb088d640 1 -- 172.21.15.1:0/2450480605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0c6b10 msgr2=0x7feeac158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.596+0000 7feeb088d640 1 --2- 172.21.15.1:0/2450480605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0c6b10 0x7feeac158690 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7feea0009870 tx=0x7feea002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.596+0000 7feeb088d640 1 -- 172.21.15.1:0/2450480605 shutdown_connections 2024-03-06T07:26:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.596+0000 7feeb088d640 1 --2- 172.21.15.1:0/2450480605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0c6b10 0x7feeac158690 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.596+0000 7feeb088d640 1 --2- 172.21.15.1:0/2450480605 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac152650 0x7feeac155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.596+0000 7feeb088d640 1 -- 172.21.15.1:0/2450480605 >> 172.21.15.1:0/2450480605 conn(0x7feeac066ee0 msgr2=0x7feeac0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.596+0000 7feeb088d640 1 -- 172.21.15.1:0/2450480605 shutdown_connections 2024-03-06T07:26:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.596+0000 7feeb088d640 1 -- 172.21.15.1:0/2450480605 wait complete. 2024-03-06T07:26:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.598+0000 7feeb088d640 1 Processor -- start 2024-03-06T07:26:30.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.598+0000 7feeb088d640 1 -- start start 2024-03-06T07:26:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.598+0000 7feeb088d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0bf9d0 0x7feeac1e0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.598+0000 7feeb088d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feeac152650 0x7feeac1e0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feeb088d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac142250 0x7feeac1ee710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feeb088d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac155490 con 0x7feeac0bf9d0 2024-03-06T07:26:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feeb088d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac1551b0 con 0x7feeac152650 2024-03-06T07:26:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feeb088d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeac152a30 con 0x7feeac142250 2024-03-06T07:26:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feeaa59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0bf9d0 0x7feeac1e0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feeaa59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0bf9d0 0x7feeac1e0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58658/0 (socket says 172.21.15.1:58658) 2024-03-06T07:26:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feeaa59c640 1 -- 172.21.15.1:0/1732780913 learned_addr learned my addr 172.21.15.1:0/1732780913 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feeaad9d640 1 --2- 172.21.15.1:0/1732780913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac142250 0x7feeac1ee710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feea9d9b640 1 --2- 172.21.15.1:0/1732780913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feeac152650 0x7feeac1e0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feeaa59c640 1 -- 172.21.15.1:0/1732780913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac142250 msgr2=0x7feeac1ee710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feeaa59c640 1 --2- 172.21.15.1:0/1732780913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac142250 0x7feeac1ee710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feeaa59c640 1 -- 172.21.15.1:0/1732780913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feeac152650 msgr2=0x7feeac1e0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feeaa59c640 1 --2- 172.21.15.1:0/1732780913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feeac152650 0x7feeac1e0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feeaa59c640 1 -- 172.21.15.1:0/1732780913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac1ef340 con 0x7feeac0bf9d0 2024-03-06T07:26:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feeaad9d640 1 --2- 172.21.15.1:0/1732780913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac142250 0x7feeac1ee710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feeaa59c640 1 --2- 172.21.15.1:0/1732780913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0bf9d0 0x7feeac1e0c10 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fee9c00d830 tx=0x7fee9c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7fee9affd640 1 -- 172.21.15.1:0/1732780913 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9c00b6c0 con 0x7feeac0bf9d0 2024-03-06T07:26:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7fee9affd640 1 -- 172.21.15.1:0/1732780913 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9c019070 con 0x7feeac0bf9d0 2024-03-06T07:26:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feeb088d640 1 -- 172.21.15.1:0/1732780913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeac14c1d0 con 0x7feeac0bf9d0 2024-03-06T07:26:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7fee9affd640 1 -- 172.21.15.1:0/1732780913 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9c00ba80 con 0x7feeac0bf9d0 2024-03-06T07:26:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.599+0000 7feeb088d640 1 -- 172.21.15.1:0/1732780913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeac14dea0 con 0x7feeac0bf9d0 2024-03-06T07:26:30.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.601+0000 7feeb088d640 1 -- 172.21.15.1:0/1732780913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeac067fd0 con 0x7feeac0bf9d0 2024-03-06T07:26:30.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.601+0000 7fee9affd640 1 -- 172.21.15.1:0/1732780913 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee9c00bc20 con 0x7feeac0bf9d0 2024-03-06T07:26:30.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.601+0000 7fee9affd640 1 --2- 172.21.15.1:0/1732780913 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee8c06b500 0x7fee8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.602+0000 7fee9affd640 1 -- 172.21.15.1:0/1732780913 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fee9c08aad0 con 0x7feeac0bf9d0 2024-03-06T07:26:30.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.602+0000 7feea9d9b640 1 --2- 172.21.15.1:0/1732780913 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee8c06b500 0x7fee8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.602+0000 7feea9d9b640 1 --2- 172.21.15.1:0/1732780913 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee8c06b500 0x7fee8c06d9c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7feea0036000 tx=0x7feea003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.606+0000 7fee9affd640 1 -- 172.21.15.1:0/1732780913 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee9c059470 con 0x7feeac0bf9d0 2024-03-06T07:26:30.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.859+0000 7feeb088d640 1 -- 172.21.15.1:0/1732780913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7feeac0b06f0 con 0x7feeac0bf9d0 2024-03-06T07:26:30.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.859+0000 7fee9affd640 1 -- 172.21.15.1:0/1732780913 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1639 (secure 0 0 0) 0x7fee9c058d70 con 0x7feeac0bf9d0 2024-03-06T07:26:30.861 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:30.861 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":386,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":153,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709614,"num_in_osds":8,"osd_in_since":1709709606,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97},{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32}],"num_pgs":193,"num_pools":5,"num_objects":24,"data_bytes":592778,"bytes_used":270663680,"bytes_avail":772823449600,"bytes_total":773094113280,"unknown_pgs_ratio":0.33160620927810669,"inactive_pgs_ratio":0.16580310463905334},"fsmap":{"epoch":140,"id":11,"up":1,"in":1,"max":1,"id":12,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":12,"rank":0,"name":"c","status":"up:active","gid":8820},{"filesystem_id":11,"rank":0,"name":"a","status":"up:active","gid":8823}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-03-06T07:26:04.769458+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:26:30.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.861+0000 7feeb088d640 1 -- 172.21.15.1:0/1732780913 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee8c06b500 msgr2=0x7fee8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.861+0000 7feeb088d640 1 --2- 172.21.15.1:0/1732780913 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee8c06b500 0x7fee8c06d9c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7feea0036000 tx=0x7feea003c040 comp rx=0 tx=0).stop 2024-03-06T07:26:30.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.861+0000 7feeb088d640 1 -- 172.21.15.1:0/1732780913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0bf9d0 msgr2=0x7feeac1e0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.861+0000 7feeb088d640 1 --2- 172.21.15.1:0/1732780913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0bf9d0 0x7feeac1e0c10 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fee9c00d830 tx=0x7fee9c00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.861+0000 7feeb088d640 1 -- 172.21.15.1:0/1732780913 shutdown_connections 2024-03-06T07:26:30.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.861+0000 7feeb088d640 1 --2- 172.21.15.1:0/1732780913 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee8c06b500 0x7fee8c06d9c0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.861+0000 7feeb088d640 1 --2- 172.21.15.1:0/1732780913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feeac142250 0x7feeac1ee710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.861+0000 7feeb088d640 1 --2- 172.21.15.1:0/1732780913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feeac152650 0x7feeac1e0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.861+0000 7feeb088d640 1 --2- 172.21.15.1:0/1732780913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feeac0bf9d0 0x7feeac1e0c10 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.861+0000 7feeb088d640 1 -- 172.21.15.1:0/1732780913 >> 172.21.15.1:0/1732780913 conn(0x7feeac066ee0 msgr2=0x7feeac0c5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.861+0000 7feeb088d640 1 -- 172.21.15.1:0/1732780913 shutdown_connections 2024-03-06T07:26:30.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.861+0000 7feeb088d640 1 -- 172.21.15.1:0/1732780913 wait complete. 2024-03-06T07:26:30.874 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:26:30.874 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:30.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.973+0000 7f7c8773e640 1 Processor -- start 2024-03-06T07:26:30.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.974+0000 7f7c8773e640 1 -- start start 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.974+0000 7f7c8773e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c80152650 0x7f7c80155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.974+0000 7f7c8773e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c800c2fc0 con 0x7f7c800bf9d0 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.974+0000 7f7c8773e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c80158da0 con 0x7f7c800c6b10 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.974+0000 7f7c8773e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c80155800 con 0x7f7c80152650 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.974+0000 7f7c854da640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c80152650 0x7f7c80155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.975+0000 7f7c854da640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c80152650 0x7f7c80155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58664/0 (socket says 172.21.15.1:58664) 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.975+0000 7f7c85cdb640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7c800bf9d0 0x7f7c800c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60370/0 (socket says 172.21.15.1:60370) 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.975+0000 7f7c854da640 1 -- 172.21.15.1:0/4194772096 learned_addr learned my addr 172.21.15.1:0/4194772096 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.975+0000 7f7c854da640 1 -- 172.21.15.1:0/4194772096 >> v1:172.21.15.98:6789/0 conn(0x7f7c800c6b10 legacy=0x7f7c80158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.975+0000 7f7c854da640 1 -- 172.21.15.1:0/4194772096 >> v1:172.21.15.1:6789/0 conn(0x7f7c800bf9d0 legacy=0x7f7c800c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.975+0000 7f7c854da640 1 -- 172.21.15.1:0/4194772096 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c800c6690 con 0x7f7c80152650 2024-03-06T07:26:30.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.976+0000 7f7c854da640 1 --2- 172.21.15.1:0/4194772096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c80152650 0x7f7c80155c90 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c009870 tx=0x7f7c7c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d2f9d742d5ea52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.976+0000 7f7c77fff640 1 -- 172.21.15.1:0/4194772096 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c035020 con 0x7f7c80152650 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.976+0000 7f7c77fff640 1 -- 172.21.15.1:0/4194772096 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7c031870 con 0x7f7c80152650 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.976+0000 7f7c77fff640 1 -- 172.21.15.1:0/4194772096 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c031c40 con 0x7f7c80152650 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.976+0000 7f7c8773e640 1 -- 172.21.15.1:0/4194772096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c80152650 msgr2=0x7f7c80155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.976+0000 7f7c8773e640 1 --2- 172.21.15.1:0/4194772096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c80152650 0x7f7c80155c90 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c009870 tx=0x7f7c7c02b910 comp rx=0 tx=0).stop 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.976+0000 7f7c8773e640 1 -- 172.21.15.1:0/4194772096 shutdown_connections 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.976+0000 7f7c8773e640 1 --2- 172.21.15.1:0/4194772096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c80152650 0x7f7c80155c90 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.976+0000 7f7c8773e640 1 -- 172.21.15.1:0/4194772096 >> 172.21.15.1:0/4194772096 conn(0x7f7c80066ee0 msgr2=0x7f7c800672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.977+0000 7f7c8773e640 1 -- 172.21.15.1:0/4194772096 shutdown_connections 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.977+0000 7f7c8773e640 1 -- 172.21.15.1:0/4194772096 wait complete. 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.978+0000 7f7c8773e640 1 Processor -- start 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.978+0000 7f7c8773e640 1 -- start start 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.978+0000 7f7c8773e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c800bf9d0 0x7f7c801eebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.978+0000 7f7c8773e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c80152650 0x7f7c8014e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.978+0000 7f7c8773e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c801588a0 0x7f7c801501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.978+0000 7f7c8773e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c801551b0 con 0x7f7c801588a0 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.978+0000 7f7c8773e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c800c6fa0 con 0x7f7c800bf9d0 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.978+0000 7f7c8773e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c80155490 con 0x7f7c80152650 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.978+0000 7f7c85cdb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c801588a0 0x7f7c801501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.978+0000 7f7c85cdb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c801588a0 0x7f7c801501a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58680/0 (socket says 172.21.15.1:58680) 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.978+0000 7f7c85cdb640 1 -- 172.21.15.1:0/1859718449 learned_addr learned my addr 172.21.15.1:0/1859718449 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.979+0000 7f7c854da640 1 --2- 172.21.15.1:0/1859718449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c800bf9d0 0x7f7c801eebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.979+0000 7f7c84cd9640 1 --2- 172.21.15.1:0/1859718449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c80152650 0x7f7c8014e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.979+0000 7f7c85cdb640 1 -- 172.21.15.1:0/1859718449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c80152650 msgr2=0x7f7c8014e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.979+0000 7f7c85cdb640 1 --2- 172.21.15.1:0/1859718449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c80152650 0x7f7c8014e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.979+0000 7f7c85cdb640 1 -- 172.21.15.1:0/1859718449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c800bf9d0 msgr2=0x7f7c801eebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.979+0000 7f7c85cdb640 1 --2- 172.21.15.1:0/1859718449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c800bf9d0 0x7f7c801eebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.979+0000 7f7c85cdb640 1 -- 172.21.15.1:0/1859718449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c801ef110 con 0x7f7c801588a0 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.979+0000 7f7c854da640 1 --2- 172.21.15.1:0/1859718449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c800bf9d0 0x7f7c801eebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.979+0000 7f7c85cdb640 1 --2- 172.21.15.1:0/1859718449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c801588a0 0x7f7c801501a0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f7c7800c730 tx=0x7f7c7800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.979+0000 7f7c75ffb640 1 -- 172.21.15.1:0/1859718449 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c78017070 con 0x7f7c801588a0 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.979+0000 7f7c75ffb640 1 -- 172.21.15.1:0/1859718449 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7800ce40 con 0x7f7c801588a0 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.979+0000 7f7c8773e640 1 -- 172.21.15.1:0/1859718449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c800c9e60 con 0x7f7c801588a0 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.979+0000 7f7c75ffb640 1 -- 172.21.15.1:0/1859718449 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7800a840 con 0x7f7c801588a0 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.979+0000 7f7c8773e640 1 -- 172.21.15.1:0/1859718449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c801ee3d0 con 0x7f7c801588a0 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.981+0000 7f7c75ffb640 1 -- 172.21.15.1:0/1859718449 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c7801a070 con 0x7f7c801588a0 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.981+0000 7f7c8773e640 1 -- 172.21.15.1:0/1859718449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c800ca380 con 0x7f7c801588a0 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.981+0000 7f7c75ffb640 1 --2- 172.21.15.1:0/1859718449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c6406b500 0x7f7c6406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.981+0000 7f7c75ffb640 1 -- 172.21.15.1:0/1859718449 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f7c78089a60 con 0x7f7c801588a0 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.981+0000 7f7c854da640 1 --2- 172.21.15.1:0/1859718449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c6406b500 0x7f7c6406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.982+0000 7f7c854da640 1 --2- 172.21.15.1:0/1859718449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c6406b500 0x7f7c6406d9c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c036000 tx=0x7f7c7c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:30.990+0000 7f7c75ffb640 1 -- 172.21.15.1:0/1859718449 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c800ca380 con 0x7f7c801588a0 2024-03-06T07:26:31.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:31.331+0000 7f7c8773e640 1 -- 172.21.15.1:0/1859718449 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7c801ee6e0 con 0x7f7c6406b500 2024-03-06T07:26:31.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:31.343+0000 7f7c75ffb640 1 -- 172.21.15.1:0/1859718449 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+491533 (secure 0 0 0) 0x7f7c801ee6e0 con 0x7f7c6406b500 2024-03-06T07:26:31.350 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:31.354 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:26:31.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:31.347+0000 7f7c8773e640 1 -- 172.21.15.1:0/1859718449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c6406b500 msgr2=0x7f7c6406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:31.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:31.347+0000 7f7c8773e640 1 --2- 172.21.15.1:0/1859718449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c6406b500 0x7f7c6406d9c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c036000 tx=0x7f7c7c03c040 comp rx=0 tx=0).stop 2024-03-06T07:26:31.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:31.347+0000 7f7c8773e640 1 -- 172.21.15.1:0/1859718449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c801588a0 msgr2=0x7f7c801501a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:31.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:31.347+0000 7f7c8773e640 1 --2- 172.21.15.1:0/1859718449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c801588a0 0x7f7c801501a0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f7c7800c730 tx=0x7f7c7800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:31.347+0000 7f7c8773e640 1 -- 172.21.15.1:0/1859718449 shutdown_connections 2024-03-06T07:26:31.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:31.347+0000 7f7c8773e640 1 --2- 172.21.15.1:0/1859718449 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7c6406b500 0x7f7c6406d9c0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:31.347+0000 7f7c8773e640 1 --2- 172.21.15.1:0/1859718449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c801588a0 0x7f7c801501a0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:31.347+0000 7f7c8773e640 1 --2- 172.21.15.1:0/1859718449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c80152650 0x7f7c8014e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:31.347+0000 7f7c8773e640 1 --2- 172.21.15.1:0/1859718449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c800bf9d0 0x7f7c801eebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:31.347+0000 7f7c8773e640 1 -- 172.21.15.1:0/1859718449 >> 172.21.15.1:0/1859718449 conn(0x7f7c80066ee0 msgr2=0x7f7c800c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:31.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:31.347+0000 7f7c8773e640 1 -- 172.21.15.1:0/1859718449 shutdown_connections 2024-03-06T07:26:31.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:31.347+0000 7f7c8773e640 1 -- 172.21.15.1:0/1859718449 wait complete. 2024-03-06T07:26:31.363 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-03-06T07:26:30.775427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":373,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264776,"kb_used_data":15296,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754709944,"statfs":{"total":773094113280,"available":772822982656,"internally_reserved":0,"allocated":15663104,"data_stored":8679392,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001727"},"pg_stats":[{"pgid":"26.3d","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854122+0000","last_change":"2024-03-06T07:26:26.694214+0000","last_active":"2024-03-06T07:26:27.854122+0000","last_peered":"2024-03-06T07:26:27.854122+0000","last_clean":"2024-03-06T07:26:27.854122+0000","last_became_active":"2024-03-06T07:26:26.693903+0000","last_became_peered":"2024-03-06T07:26:26.693903+0000","last_unstale":"2024-03-06T07:26:27.854122+0000","last_undegraded":"2024-03-06T07:26:27.854122+0000","last_fullsized":"2024-03-06T07:26:27.854122+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:49.088768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.3f","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683756+0000","last_change":"2024-03-06T07:26:17.632610+0000","last_active":"2024-03-06T07:26:27.683756+0000","last_peered":"2024-03-06T07:26:27.683756+0000","last_clean":"2024-03-06T07:26:27.683756+0000","last_became_active":"2024-03-06T07:26:17.630763+0000","last_became_peered":"2024-03-06T07:26:17.630763+0000","last_unstale":"2024-03-06T07:26:27.683756+0000","last_undegraded":"2024-03-06T07:26:27.683756+0000","last_fullsized":"2024-03-06T07:26:27.683756+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:35.035202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680368+0000","last_change":"2024-03-06T07:26:26.708189+0000","last_active":"2024-03-06T07:26:27.680368+0000","last_peered":"2024-03-06T07:26:27.680368+0000","last_clean":"2024-03-06T07:26:27.680368+0000","last_became_active":"2024-03-06T07:26:26.707895+0000","last_became_peered":"2024-03-06T07:26:26.707895+0000","last_unstale":"2024-03-06T07:26:27.680368+0000","last_undegraded":"2024-03-06T07:26:27.680368+0000","last_fullsized":"2024-03-06T07:26:27.680368+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:37.770304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684557+0000","last_change":"2024-03-06T07:26:17.632425+0000","last_active":"2024-03-06T07:26:27.684557+0000","last_peered":"2024-03-06T07:26:27.684557+0000","last_clean":"2024-03-06T07:26:27.684557+0000","last_became_active":"2024-03-06T07:26:17.632153+0000","last_became_peered":"2024-03-06T07:26:17.632153+0000","last_unstale":"2024-03-06T07:26:27.684557+0000","last_undegraded":"2024-03-06T07:26:27.684557+0000","last_fullsized":"2024-03-06T07:26:27.684557+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:01.829378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3f","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702806+0000","last_change":"2024-03-06T07:26:26.696360+0000","last_active":"2024-03-06T07:26:27.702806+0000","last_peered":"2024-03-06T07:26:27.702806+0000","last_clean":"2024-03-06T07:26:27.702806+0000","last_became_active":"2024-03-06T07:26:26.696053+0000","last_became_peered":"2024-03-06T07:26:26.696053+0000","last_unstale":"2024-03-06T07:26:27.702806+0000","last_undegraded":"2024-03-06T07:26:27.702806+0000","last_fullsized":"2024-03-06T07:26:27.702806+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:53.420127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683419+0000","last_change":"2024-03-06T07:26:17.630353+0000","last_active":"2024-03-06T07:26:27.683419+0000","last_peered":"2024-03-06T07:26:27.683419+0000","last_clean":"2024-03-06T07:26:27.683419+0000","last_became_active":"2024-03-06T07:26:17.630080+0000","last_became_peered":"2024-03-06T07:26:17.630080+0000","last_unstale":"2024-03-06T07:26:27.683419+0000","last_undegraded":"2024-03-06T07:26:27.683419+0000","last_fullsized":"2024-03-06T07:26:27.683419+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:49.713717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702711+0000","last_change":"2024-03-06T07:26:26.705829+0000","last_active":"2024-03-06T07:26:27.702711+0000","last_peered":"2024-03-06T07:26:27.702711+0000","last_clean":"2024-03-06T07:26:27.702711+0000","last_became_active":"2024-03-06T07:26:26.705541+0000","last_became_peered":"2024-03-06T07:26:26.705541+0000","last_unstale":"2024-03-06T07:26:27.702711+0000","last_undegraded":"2024-03-06T07:26:27.702711+0000","last_fullsized":"2024-03-06T07:26:27.702711+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:38.410943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683191+0000","last_change":"2024-03-06T07:26:17.630434+0000","last_active":"2024-03-06T07:26:27.683191+0000","last_peered":"2024-03-06T07:26:27.683191+0000","last_clean":"2024-03-06T07:26:27.683191+0000","last_became_active":"2024-03-06T07:26:17.630139+0000","last_became_peered":"2024-03-06T07:26:17.630139+0000","last_unstale":"2024-03-06T07:26:27.683191+0000","last_undegraded":"2024-03-06T07:26:27.683191+0000","last_fullsized":"2024-03-06T07:26:27.683191+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:09.159934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700303+0000","last_change":"2024-03-06T07:26:26.697421+0000","last_active":"2024-03-06T07:26:27.700303+0000","last_peered":"2024-03-06T07:26:27.700303+0000","last_clean":"2024-03-06T07:26:27.700303+0000","last_became_active":"2024-03-06T07:26:26.696894+0000","last_became_peered":"2024-03-06T07:26:26.696894+0000","last_unstale":"2024-03-06T07:26:27.700303+0000","last_undegraded":"2024-03-06T07:26:27.700303+0000","last_fullsized":"2024-03-06T07:26:27.700303+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:06.271489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680152+0000","last_change":"2024-03-06T07:26:17.632665+0000","last_active":"2024-03-06T07:26:27.680152+0000","last_peered":"2024-03-06T07:26:27.680152+0000","last_clean":"2024-03-06T07:26:27.680152+0000","last_became_active":"2024-03-06T07:26:17.632377+0000","last_became_peered":"2024-03-06T07:26:17.632377+0000","last_unstale":"2024-03-06T07:26:27.680152+0000","last_undegraded":"2024-03-06T07:26:27.680152+0000","last_fullsized":"2024-03-06T07:26:27.680152+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:23.225363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683788+0000","last_change":"2024-03-06T07:26:26.701290+0000","last_active":"2024-03-06T07:26:27.683788+0000","last_peered":"2024-03-06T07:26:27.683788+0000","last_clean":"2024-03-06T07:26:27.683788+0000","last_became_active":"2024-03-06T07:26:26.701010+0000","last_became_peered":"2024-03-06T07:26:26.701010+0000","last_unstale":"2024-03-06T07:26:27.683788+0000","last_undegraded":"2024-03-06T07:26:27.683788+0000","last_fullsized":"2024-03-06T07:26:27.683788+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:57.429223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684070+0000","last_change":"2024-03-06T07:26:17.633911+0000","last_active":"2024-03-06T07:26:27.684070+0000","last_peered":"2024-03-06T07:26:27.684070+0000","last_clean":"2024-03-06T07:26:27.684070+0000","last_became_active":"2024-03-06T07:26:17.633638+0000","last_became_peered":"2024-03-06T07:26:17.633638+0000","last_unstale":"2024-03-06T07:26:27.684070+0000","last_undegraded":"2024-03-06T07:26:27.684070+0000","last_fullsized":"2024-03-06T07:26:27.684070+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:01.350963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853311+0000","last_change":"2024-03-06T07:26:26.707191+0000","last_active":"2024-03-06T07:26:27.853311+0000","last_peered":"2024-03-06T07:26:27.853311+0000","last_clean":"2024-03-06T07:26:27.853311+0000","last_became_active":"2024-03-06T07:26:26.706927+0000","last_became_peered":"2024-03-06T07:26:26.706927+0000","last_unstale":"2024-03-06T07:26:27.853311+0000","last_undegraded":"2024-03-06T07:26:27.853311+0000","last_fullsized":"2024-03-06T07:26:27.853311+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:56.420351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702248+0000","last_change":"2024-03-06T07:26:17.629894+0000","last_active":"2024-03-06T07:26:27.702248+0000","last_peered":"2024-03-06T07:26:27.702248+0000","last_clean":"2024-03-06T07:26:27.702248+0000","last_became_active":"2024-03-06T07:26:17.629625+0000","last_became_peered":"2024-03-06T07:26:17.629625+0000","last_unstale":"2024-03-06T07:26:27.702248+0000","last_undegraded":"2024-03-06T07:26:27.702248+0000","last_fullsized":"2024-03-06T07:26:27.702248+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:46.687284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853129+0000","last_change":"2024-03-06T07:26:26.701868+0000","last_active":"2024-03-06T07:26:27.853129+0000","last_peered":"2024-03-06T07:26:27.853129+0000","last_clean":"2024-03-06T07:26:27.853129+0000","last_became_active":"2024-03-06T07:26:26.701540+0000","last_became_peered":"2024-03-06T07:26:26.701540+0000","last_unstale":"2024-03-06T07:26:27.853129+0000","last_undegraded":"2024-03-06T07:26:27.853129+0000","last_fullsized":"2024-03-06T07:26:27.853129+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:06.797123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702035+0000","last_change":"2024-03-06T07:26:17.632711+0000","last_active":"2024-03-06T07:26:27.702035+0000","last_peered":"2024-03-06T07:26:27.702035+0000","last_clean":"2024-03-06T07:26:27.702035+0000","last_became_active":"2024-03-06T07:26:17.632173+0000","last_became_peered":"2024-03-06T07:26:17.632173+0000","last_unstale":"2024-03-06T07:26:27.702035+0000","last_undegraded":"2024-03-06T07:26:27.702035+0000","last_fullsized":"2024-03-06T07:26:27.702035+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:25.595909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854605+0000","last_change":"2024-03-06T07:26:26.702364+0000","last_active":"2024-03-06T07:26:27.854605+0000","last_peered":"2024-03-06T07:26:27.854605+0000","last_clean":"2024-03-06T07:26:27.854605+0000","last_became_active":"2024-03-06T07:26:26.702102+0000","last_became_peered":"2024-03-06T07:26:26.702102+0000","last_unstale":"2024-03-06T07:26:27.854605+0000","last_undegraded":"2024-03-06T07:26:27.854605+0000","last_fullsized":"2024-03-06T07:26:27.854605+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:57.577361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703050+0000","last_change":"2024-03-06T07:26:17.638531+0000","last_active":"2024-03-06T07:26:27.703050+0000","last_peered":"2024-03-06T07:26:27.703050+0000","last_clean":"2024-03-06T07:26:27.703050+0000","last_became_active":"2024-03-06T07:26:17.638254+0000","last_became_peered":"2024-03-06T07:26:17.638254+0000","last_unstale":"2024-03-06T07:26:27.703050+0000","last_undegraded":"2024-03-06T07:26:27.703050+0000","last_fullsized":"2024-03-06T07:26:27.703050+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:01.350963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680372+0000","last_change":"2024-03-06T07:26:26.694408+0000","last_active":"2024-03-06T07:26:27.680372+0000","last_peered":"2024-03-06T07:26:27.680372+0000","last_clean":"2024-03-06T07:26:27.680372+0000","last_became_active":"2024-03-06T07:26:26.694144+0000","last_became_peered":"2024-03-06T07:26:26.694144+0000","last_unstale":"2024-03-06T07:26:27.680372+0000","last_undegraded":"2024-03-06T07:26:27.680372+0000","last_fullsized":"2024-03-06T07:26:27.680372+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:14.666630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703206+0000","last_change":"2024-03-06T07:26:17.632455+0000","last_active":"2024-03-06T07:26:27.703206+0000","last_peered":"2024-03-06T07:26:27.703206+0000","last_clean":"2024-03-06T07:26:27.703206+0000","last_became_active":"2024-03-06T07:26:17.632195+0000","last_became_peered":"2024-03-06T07:26:17.632195+0000","last_unstale":"2024-03-06T07:26:27.703206+0000","last_undegraded":"2024-03-06T07:26:27.703206+0000","last_fullsized":"2024-03-06T07:26:27.703206+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:14.404340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702916+0000","last_change":"2024-03-06T07:26:26.702603+0000","last_active":"2024-03-06T07:26:27.702916+0000","last_peered":"2024-03-06T07:26:27.702916+0000","last_clean":"2024-03-06T07:26:27.702916+0000","last_became_active":"2024-03-06T07:26:26.702329+0000","last_became_peered":"2024-03-06T07:26:26.702329+0000","last_unstale":"2024-03-06T07:26:27.702916+0000","last_undegraded":"2024-03-06T07:26:27.702916+0000","last_fullsized":"2024-03-06T07:26:27.702916+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:47.407016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854744+0000","last_change":"2024-03-06T07:26:17.633984+0000","last_active":"2024-03-06T07:26:27.854744+0000","last_peered":"2024-03-06T07:26:27.854744+0000","last_clean":"2024-03-06T07:26:27.854744+0000","last_became_active":"2024-03-06T07:26:17.633666+0000","last_became_peered":"2024-03-06T07:26:17.633666+0000","last_unstale":"2024-03-06T07:26:27.854744+0000","last_undegraded":"2024-03-06T07:26:27.854744+0000","last_fullsized":"2024-03-06T07:26:27.854744+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:43.884264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854240+0000","last_change":"2024-03-06T07:26:26.706601+0000","last_active":"2024-03-06T07:26:27.854240+0000","last_peered":"2024-03-06T07:26:27.854240+0000","last_clean":"2024-03-06T07:26:27.854240+0000","last_became_active":"2024-03-06T07:26:26.706312+0000","last_became_peered":"2024-03-06T07:26:26.706312+0000","last_unstale":"2024-03-06T07:26:27.854240+0000","last_undegraded":"2024-03-06T07:26:27.854240+0000","last_fullsized":"2024-03-06T07:26:27.854240+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:10.927406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684725+0000","last_change":"2024-03-06T07:26:17.634798+0000","last_active":"2024-03-06T07:26:27.684725+0000","last_peered":"2024-03-06T07:26:27.684725+0000","last_clean":"2024-03-06T07:26:27.684725+0000","last_became_active":"2024-03-06T07:26:17.634537+0000","last_became_peered":"2024-03-06T07:26:17.634537+0000","last_unstale":"2024-03-06T07:26:27.684725+0000","last_undegraded":"2024-03-06T07:26:27.684725+0000","last_fullsized":"2024-03-06T07:26:27.684725+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:25.595909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702423+0000","last_change":"2024-03-06T07:26:26.708042+0000","last_active":"2024-03-06T07:26:27.702423+0000","last_peered":"2024-03-06T07:26:27.702423+0000","last_clean":"2024-03-06T07:26:27.702423+0000","last_became_active":"2024-03-06T07:26:26.707778+0000","last_became_peered":"2024-03-06T07:26:26.707778+0000","last_unstale":"2024-03-06T07:26:27.702423+0000","last_undegraded":"2024-03-06T07:26:27.702423+0000","last_fullsized":"2024-03-06T07:26:27.702423+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:43.140802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853687+0000","last_change":"2024-03-06T07:26:17.640814+0000","last_active":"2024-03-06T07:26:27.853687+0000","last_peered":"2024-03-06T07:26:27.853687+0000","last_clean":"2024-03-06T07:26:27.853687+0000","last_became_active":"2024-03-06T07:26:17.640444+0000","last_became_peered":"2024-03-06T07:26:17.640444+0000","last_unstale":"2024-03-06T07:26:27.853687+0000","last_undegraded":"2024-03-06T07:26:27.853687+0000","last_fullsized":"2024-03-06T07:26:27.853687+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:35.920913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853140+0000","last_change":"2024-03-06T07:26:26.704032+0000","last_active":"2024-03-06T07:26:27.853140+0000","last_peered":"2024-03-06T07:26:27.853140+0000","last_clean":"2024-03-06T07:26:27.853140+0000","last_became_active":"2024-03-06T07:26:26.703752+0000","last_became_peered":"2024-03-06T07:26:26.703752+0000","last_unstale":"2024-03-06T07:26:27.853140+0000","last_undegraded":"2024-03-06T07:26:27.853140+0000","last_fullsized":"2024-03-06T07:26:27.853140+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:30.863087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680198+0000","last_change":"2024-03-06T07:26:17.631892+0000","last_active":"2024-03-06T07:26:27.680198+0000","last_peered":"2024-03-06T07:26:27.680198+0000","last_clean":"2024-03-06T07:26:27.680198+0000","last_became_active":"2024-03-06T07:26:17.631498+0000","last_became_peered":"2024-03-06T07:26:17.631498+0000","last_unstale":"2024-03-06T07:26:27.680198+0000","last_undegraded":"2024-03-06T07:26:27.680198+0000","last_fullsized":"2024-03-06T07:26:27.680198+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:09.253874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682875+0000","last_change":"2024-03-06T07:26:26.707066+0000","last_active":"2024-03-06T07:26:27.682875+0000","last_peered":"2024-03-06T07:26:27.682875+0000","last_clean":"2024-03-06T07:26:27.682875+0000","last_became_active":"2024-03-06T07:26:26.705596+0000","last_became_peered":"2024-03-06T07:26:26.705596+0000","last_unstale":"2024-03-06T07:26:27.682875+0000","last_undegraded":"2024-03-06T07:26:27.682875+0000","last_fullsized":"2024-03-06T07:26:27.682875+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:47.096950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853710+0000","last_change":"2024-03-06T07:26:17.634220+0000","last_active":"2024-03-06T07:26:27.853710+0000","last_peered":"2024-03-06T07:26:27.853710+0000","last_clean":"2024-03-06T07:26:27.853710+0000","last_became_active":"2024-03-06T07:26:17.633922+0000","last_became_peered":"2024-03-06T07:26:17.633922+0000","last_unstale":"2024-03-06T07:26:27.853710+0000","last_undegraded":"2024-03-06T07:26:27.853710+0000","last_fullsized":"2024-03-06T07:26:27.853710+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:51.900557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700487+0000","last_change":"2024-03-06T07:26:26.704753+0000","last_active":"2024-03-06T07:26:27.700487+0000","last_peered":"2024-03-06T07:26:27.700487+0000","last_clean":"2024-03-06T07:26:27.700487+0000","last_became_active":"2024-03-06T07:26:26.704488+0000","last_became_peered":"2024-03-06T07:26:26.704488+0000","last_unstale":"2024-03-06T07:26:27.700487+0000","last_undegraded":"2024-03-06T07:26:27.700487+0000","last_fullsized":"2024-03-06T07:26:27.700487+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:51.184434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682404+0000","last_change":"2024-03-06T07:26:17.635856+0000","last_active":"2024-03-06T07:26:27.682404+0000","last_peered":"2024-03-06T07:26:27.682404+0000","last_clean":"2024-03-06T07:26:27.682404+0000","last_became_active":"2024-03-06T07:26:17.631353+0000","last_became_peered":"2024-03-06T07:26:17.631353+0000","last_unstale":"2024-03-06T07:26:27.682404+0000","last_undegraded":"2024-03-06T07:26:27.682404+0000","last_fullsized":"2024-03-06T07:26:27.682404+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:15.652152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683086+0000","last_change":"2024-03-06T07:26:26.704283+0000","last_active":"2024-03-06T07:26:27.683086+0000","last_peered":"2024-03-06T07:26:27.683086+0000","last_clean":"2024-03-06T07:26:27.683086+0000","last_became_active":"2024-03-06T07:26:26.703817+0000","last_became_peered":"2024-03-06T07:26:26.703817+0000","last_unstale":"2024-03-06T07:26:27.683086+0000","last_undegraded":"2024-03-06T07:26:27.683086+0000","last_fullsized":"2024-03-06T07:26:27.683086+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:34.318310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703135+0000","last_change":"2024-03-06T07:26:17.628960+0000","last_active":"2024-03-06T07:26:27.703135+0000","last_peered":"2024-03-06T07:26:27.703135+0000","last_clean":"2024-03-06T07:26:27.703135+0000","last_became_active":"2024-03-06T07:26:17.628681+0000","last_became_peered":"2024-03-06T07:26:17.628681+0000","last_unstale":"2024-03-06T07:26:27.703135+0000","last_undegraded":"2024-03-06T07:26:27.703135+0000","last_fullsized":"2024-03-06T07:26:27.703135+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:40.950597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853602+0000","last_change":"2024-03-06T07:26:26.695129+0000","last_active":"2024-03-06T07:26:27.853602+0000","last_peered":"2024-03-06T07:26:27.853602+0000","last_clean":"2024-03-06T07:26:27.853602+0000","last_became_active":"2024-03-06T07:26:26.694943+0000","last_became_peered":"2024-03-06T07:26:26.694943+0000","last_unstale":"2024-03-06T07:26:27.853602+0000","last_undegraded":"2024-03-06T07:26:27.853602+0000","last_fullsized":"2024-03-06T07:26:27.853602+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:52.613907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854354+0000","last_change":"2024-03-06T07:26:17.629582+0000","last_active":"2024-03-06T07:26:27.854354+0000","last_peered":"2024-03-06T07:26:27.854354+0000","last_clean":"2024-03-06T07:26:27.854354+0000","last_became_active":"2024-03-06T07:26:17.629246+0000","last_became_peered":"2024-03-06T07:26:17.629246+0000","last_unstale":"2024-03-06T07:26:27.854354+0000","last_undegraded":"2024-03-06T07:26:27.854354+0000","last_fullsized":"2024-03-06T07:26:27.854354+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:56.556155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854336+0000","last_change":"2024-03-06T07:26:26.695709+0000","last_active":"2024-03-06T07:26:27.854336+0000","last_peered":"2024-03-06T07:26:27.854336+0000","last_clean":"2024-03-06T07:26:27.854336+0000","last_became_active":"2024-03-06T07:26:26.695471+0000","last_became_peered":"2024-03-06T07:26:26.695471+0000","last_unstale":"2024-03-06T07:26:27.854336+0000","last_undegraded":"2024-03-06T07:26:27.854336+0000","last_fullsized":"2024-03-06T07:26:27.854336+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:38.741123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854792+0000","last_change":"2024-03-06T07:26:17.632386+0000","last_active":"2024-03-06T07:26:27.854792+0000","last_peered":"2024-03-06T07:26:27.854792+0000","last_clean":"2024-03-06T07:26:27.854792+0000","last_became_active":"2024-03-06T07:26:17.631867+0000","last_became_peered":"2024-03-06T07:26:17.631867+0000","last_unstale":"2024-03-06T07:26:27.854792+0000","last_undegraded":"2024-03-06T07:26:27.854792+0000","last_fullsized":"2024-03-06T07:26:27.854792+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:46.687284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683476+0000","last_change":"2024-03-06T07:26:26.705909+0000","last_active":"2024-03-06T07:26:27.683476+0000","last_peered":"2024-03-06T07:26:27.683476+0000","last_clean":"2024-03-06T07:26:27.683476+0000","last_became_active":"2024-03-06T07:26:26.705632+0000","last_became_peered":"2024-03-06T07:26:26.705632+0000","last_unstale":"2024-03-06T07:26:27.683476+0000","last_undegraded":"2024-03-06T07:26:27.683476+0000","last_fullsized":"2024-03-06T07:26:27.683476+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:39.657027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700242+0000","last_change":"2024-03-06T07:26:17.622563+0000","last_active":"2024-03-06T07:26:27.700242+0000","last_peered":"2024-03-06T07:26:27.700242+0000","last_clean":"2024-03-06T07:26:27.700242+0000","last_became_active":"2024-03-06T07:26:17.622293+0000","last_became_peered":"2024-03-06T07:26:17.622293+0000","last_unstale":"2024-03-06T07:26:27.700242+0000","last_undegraded":"2024-03-06T07:26:27.700242+0000","last_fullsized":"2024-03-06T07:26:27.700242+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:34.462291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684627+0000","last_change":"2024-03-06T07:26:26.699335+0000","last_active":"2024-03-06T07:26:27.684627+0000","last_peered":"2024-03-06T07:26:27.684627+0000","last_clean":"2024-03-06T07:26:27.684627+0000","last_became_active":"2024-03-06T07:26:26.699041+0000","last_became_peered":"2024-03-06T07:26:26.699041+0000","last_unstale":"2024-03-06T07:26:27.684627+0000","last_undegraded":"2024-03-06T07:26:27.684627+0000","last_fullsized":"2024-03-06T07:26:27.684627+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:24.512341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853783+0000","last_change":"2024-03-06T07:26:17.637385+0000","last_active":"2024-03-06T07:26:27.853783+0000","last_peered":"2024-03-06T07:26:27.853783+0000","last_clean":"2024-03-06T07:26:27.853783+0000","last_became_active":"2024-03-06T07:26:17.637063+0000","last_became_peered":"2024-03-06T07:26:17.637063+0000","last_unstale":"2024-03-06T07:26:27.853783+0000","last_undegraded":"2024-03-06T07:26:27.853783+0000","last_fullsized":"2024-03-06T07:26:27.853783+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:49.333296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702261+0000","last_change":"2024-03-06T07:26:26.695440+0000","last_active":"2024-03-06T07:26:27.702261+0000","last_peered":"2024-03-06T07:26:27.702261+0000","last_clean":"2024-03-06T07:26:27.702261+0000","last_became_active":"2024-03-06T07:26:26.695168+0000","last_became_peered":"2024-03-06T07:26:26.695168+0000","last_unstale":"2024-03-06T07:26:27.702261+0000","last_undegraded":"2024-03-06T07:26:27.702261+0000","last_fullsized":"2024-03-06T07:26:27.702261+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:24.512341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684155+0000","last_change":"2024-03-06T07:26:17.632278+0000","last_active":"2024-03-06T07:26:27.684155+0000","last_peered":"2024-03-06T07:26:27.684155+0000","last_clean":"2024-03-06T07:26:27.684155+0000","last_became_active":"2024-03-06T07:26:17.631791+0000","last_became_peered":"2024-03-06T07:26:17.631791+0000","last_unstale":"2024-03-06T07:26:27.684155+0000","last_undegraded":"2024-03-06T07:26:27.684155+0000","last_fullsized":"2024-03-06T07:26:27.684155+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:40.950597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703000+0000","last_change":"2024-03-06T07:26:26.703449+0000","last_active":"2024-03-06T07:26:27.703000+0000","last_peered":"2024-03-06T07:26:27.703000+0000","last_clean":"2024-03-06T07:26:27.703000+0000","last_became_active":"2024-03-06T07:26:26.702643+0000","last_became_peered":"2024-03-06T07:26:26.702643+0000","last_unstale":"2024-03-06T07:26:27.703000+0000","last_undegraded":"2024-03-06T07:26:27.703000+0000","last_fullsized":"2024-03-06T07:26:27.703000+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:12.463109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.679900+0000","last_change":"2024-03-06T07:26:17.627902+0000","last_active":"2024-03-06T07:26:27.679900+0000","last_peered":"2024-03-06T07:26:27.679900+0000","last_clean":"2024-03-06T07:26:27.679900+0000","last_became_active":"2024-03-06T07:26:17.627643+0000","last_became_peered":"2024-03-06T07:26:17.627643+0000","last_unstale":"2024-03-06T07:26:27.679900+0000","last_undegraded":"2024-03-06T07:26:27.679900+0000","last_fullsized":"2024-03-06T07:26:27.679900+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:30.600647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702957+0000","last_change":"2024-03-06T07:26:26.703994+0000","last_active":"2024-03-06T07:26:27.702957+0000","last_peered":"2024-03-06T07:26:27.702957+0000","last_clean":"2024-03-06T07:26:27.702957+0000","last_became_active":"2024-03-06T07:26:26.703695+0000","last_became_peered":"2024-03-06T07:26:26.703695+0000","last_unstale":"2024-03-06T07:26:27.702957+0000","last_undegraded":"2024-03-06T07:26:27.702957+0000","last_fullsized":"2024-03-06T07:26:27.702957+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:57.436722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682452+0000","last_change":"2024-03-06T07:26:17.631421+0000","last_active":"2024-03-06T07:26:27.682452+0000","last_peered":"2024-03-06T07:26:27.682452+0000","last_clean":"2024-03-06T07:26:27.682452+0000","last_became_active":"2024-03-06T07:26:17.631022+0000","last_became_peered":"2024-03-06T07:26:17.631022+0000","last_unstale":"2024-03-06T07:26:27.682452+0000","last_undegraded":"2024-03-06T07:26:27.682452+0000","last_fullsized":"2024-03-06T07:26:27.682452+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:40.032388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684400+0000","last_change":"2024-03-06T07:26:26.700754+0000","last_active":"2024-03-06T07:26:27.684400+0000","last_peered":"2024-03-06T07:26:27.684400+0000","last_clean":"2024-03-06T07:26:27.684400+0000","last_became_active":"2024-03-06T07:26:26.700432+0000","last_became_peered":"2024-03-06T07:26:26.700432+0000","last_unstale":"2024-03-06T07:26:27.684400+0000","last_undegraded":"2024-03-06T07:26:27.684400+0000","last_fullsized":"2024-03-06T07:26:27.684400+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:27.612436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854481+0000","last_change":"2024-03-06T07:26:17.637044+0000","last_active":"2024-03-06T07:26:27.854481+0000","last_peered":"2024-03-06T07:26:27.854481+0000","last_clean":"2024-03-06T07:26:27.854481+0000","last_became_active":"2024-03-06T07:26:17.636771+0000","last_became_peered":"2024-03-06T07:26:17.636771+0000","last_unstale":"2024-03-06T07:26:27.854481+0000","last_undegraded":"2024-03-06T07:26:27.854481+0000","last_fullsized":"2024-03-06T07:26:27.854481+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:29.354563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853327+0000","last_change":"2024-03-06T07:26:26.706900+0000","last_active":"2024-03-06T07:26:27.853327+0000","last_peered":"2024-03-06T07:26:27.853327+0000","last_clean":"2024-03-06T07:26:27.853327+0000","last_became_active":"2024-03-06T07:26:26.706660+0000","last_became_peered":"2024-03-06T07:26:26.706660+0000","last_unstale":"2024-03-06T07:26:27.853327+0000","last_undegraded":"2024-03-06T07:26:27.853327+0000","last_fullsized":"2024-03-06T07:26:27.853327+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:34.561014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700173+0000","last_change":"2024-03-06T07:26:17.636369+0000","last_active":"2024-03-06T07:26:27.700173+0000","last_peered":"2024-03-06T07:26:27.700173+0000","last_clean":"2024-03-06T07:26:27.700173+0000","last_became_active":"2024-03-06T07:26:17.636090+0000","last_became_peered":"2024-03-06T07:26:17.636090+0000","last_unstale":"2024-03-06T07:26:27.700173+0000","last_undegraded":"2024-03-06T07:26:27.700173+0000","last_fullsized":"2024-03-06T07:26:27.700173+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:45.110614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700456+0000","last_change":"2024-03-06T07:26:26.690832+0000","last_active":"2024-03-06T07:26:27.700456+0000","last_peered":"2024-03-06T07:26:27.700456+0000","last_clean":"2024-03-06T07:26:27.700456+0000","last_became_active":"2024-03-06T07:26:26.690572+0000","last_became_peered":"2024-03-06T07:26:26.690572+0000","last_unstale":"2024-03-06T07:26:27.700456+0000","last_undegraded":"2024-03-06T07:26:27.700456+0000","last_fullsized":"2024-03-06T07:26:27.700456+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:10.471929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684323+0000","last_change":"2024-03-06T07:26:17.633006+0000","last_active":"2024-03-06T07:26:27.684323+0000","last_peered":"2024-03-06T07:26:27.684323+0000","last_clean":"2024-03-06T07:26:27.684323+0000","last_became_active":"2024-03-06T07:26:17.632523+0000","last_became_peered":"2024-03-06T07:26:17.632523+0000","last_unstale":"2024-03-06T07:26:27.684323+0000","last_undegraded":"2024-03-06T07:26:27.684323+0000","last_fullsized":"2024-03-06T07:26:27.684323+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:41.655929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853881+0000","last_change":"2024-03-06T07:26:26.701984+0000","last_active":"2024-03-06T07:26:27.853881+0000","last_peered":"2024-03-06T07:26:27.853881+0000","last_clean":"2024-03-06T07:26:27.853881+0000","last_became_active":"2024-03-06T07:26:26.701386+0000","last_became_peered":"2024-03-06T07:26:26.701386+0000","last_unstale":"2024-03-06T07:26:27.853881+0000","last_undegraded":"2024-03-06T07:26:27.853881+0000","last_fullsized":"2024-03-06T07:26:27.853881+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:39.657027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853431+0000","last_change":"2024-03-06T07:26:17.638448+0000","last_active":"2024-03-06T07:26:27.853431+0000","last_peered":"2024-03-06T07:26:27.853431+0000","last_clean":"2024-03-06T07:26:27.853431+0000","last_became_active":"2024-03-06T07:26:17.638188+0000","last_became_peered":"2024-03-06T07:26:17.638188+0000","last_unstale":"2024-03-06T07:26:27.853431+0000","last_undegraded":"2024-03-06T07:26:27.853431+0000","last_fullsized":"2024-03-06T07:26:27.853431+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:15.455961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700096+0000","last_change":"2024-03-06T07:26:26.688526+0000","last_active":"2024-03-06T07:26:27.700096+0000","last_peered":"2024-03-06T07:26:27.700096+0000","last_clean":"2024-03-06T07:26:27.700096+0000","last_became_active":"2024-03-06T07:26:26.688270+0000","last_became_peered":"2024-03-06T07:26:26.688270+0000","last_unstale":"2024-03-06T07:26:27.700096+0000","last_undegraded":"2024-03-06T07:26:27.700096+0000","last_fullsized":"2024-03-06T07:26:27.700096+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:20.442923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684388+0000","last_change":"2024-03-06T07:26:17.634053+0000","last_active":"2024-03-06T07:26:27.684388+0000","last_peered":"2024-03-06T07:26:27.684388+0000","last_clean":"2024-03-06T07:26:27.684388+0000","last_became_active":"2024-03-06T07:26:17.633451+0000","last_became_peered":"2024-03-06T07:26:17.633451+0000","last_unstale":"2024-03-06T07:26:27.684388+0000","last_undegraded":"2024-03-06T07:26:27.684388+0000","last_fullsized":"2024-03-06T07:26:27.684388+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:25.173456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853860+0000","last_change":"2024-03-06T07:26:26.704201+0000","last_active":"2024-03-06T07:26:27.853860+0000","last_peered":"2024-03-06T07:26:27.853860+0000","last_clean":"2024-03-06T07:26:27.853860+0000","last_became_active":"2024-03-06T07:26:26.703937+0000","last_became_peered":"2024-03-06T07:26:26.703937+0000","last_unstale":"2024-03-06T07:26:27.853860+0000","last_undegraded":"2024-03-06T07:26:27.853860+0000","last_fullsized":"2024-03-06T07:26:27.853860+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:00.129449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853564+0000","last_change":"2024-03-06T07:26:17.627891+0000","last_active":"2024-03-06T07:26:27.853564+0000","last_peered":"2024-03-06T07:26:27.853564+0000","last_clean":"2024-03-06T07:26:27.853564+0000","last_became_active":"2024-03-06T07:26:17.627624+0000","last_became_peered":"2024-03-06T07:26:17.627624+0000","last_unstale":"2024-03-06T07:26:27.853564+0000","last_undegraded":"2024-03-06T07:26:27.853564+0000","last_fullsized":"2024-03-06T07:26:27.853564+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:56.476412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854339+0000","last_change":"2024-03-06T07:26:26.695392+0000","last_active":"2024-03-06T07:26:27.854339+0000","last_peered":"2024-03-06T07:26:27.854339+0000","last_clean":"2024-03-06T07:26:27.854339+0000","last_became_active":"2024-03-06T07:26:26.695115+0000","last_became_peered":"2024-03-06T07:26:26.695115+0000","last_unstale":"2024-03-06T07:26:27.854339+0000","last_undegraded":"2024-03-06T07:26:27.854339+0000","last_fullsized":"2024-03-06T07:26:27.854339+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:32.281743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853828+0000","last_change":"2024-03-06T07:26:17.635467+0000","last_active":"2024-03-06T07:26:27.853828+0000","last_peered":"2024-03-06T07:26:27.853828+0000","last_clean":"2024-03-06T07:26:27.853828+0000","last_became_active":"2024-03-06T07:26:17.635125+0000","last_became_peered":"2024-03-06T07:26:17.635125+0000","last_unstale":"2024-03-06T07:26:27.853828+0000","last_undegraded":"2024-03-06T07:26:27.853828+0000","last_fullsized":"2024-03-06T07:26:27.853828+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:39.829929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854316+0000","last_change":"2024-03-06T07:26:26.702761+0000","last_active":"2024-03-06T07:26:27.854316+0000","last_peered":"2024-03-06T07:26:27.854316+0000","last_clean":"2024-03-06T07:26:27.854316+0000","last_became_active":"2024-03-06T07:26:26.700555+0000","last_became_peered":"2024-03-06T07:26:26.700555+0000","last_unstale":"2024-03-06T07:26:27.854316+0000","last_undegraded":"2024-03-06T07:26:27.854316+0000","last_fullsized":"2024-03-06T07:26:27.854316+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:29.718934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853712+0000","last_change":"2024-03-06T07:26:17.638944+0000","last_active":"2024-03-06T07:26:27.853712+0000","last_peered":"2024-03-06T07:26:27.853712+0000","last_clean":"2024-03-06T07:26:27.853712+0000","last_became_active":"2024-03-06T07:26:17.638649+0000","last_became_peered":"2024-03-06T07:26:17.638649+0000","last_unstale":"2024-03-06T07:26:27.853712+0000","last_undegraded":"2024-03-06T07:26:27.853712+0000","last_fullsized":"2024-03-06T07:26:27.853712+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:17.394090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682834+0000","last_change":"2024-03-06T07:26:26.703690+0000","last_active":"2024-03-06T07:26:27.682834+0000","last_peered":"2024-03-06T07:26:27.682834+0000","last_clean":"2024-03-06T07:26:27.682834+0000","last_became_active":"2024-03-06T07:26:26.703143+0000","last_became_peered":"2024-03-06T07:26:26.703143+0000","last_unstale":"2024-03-06T07:26:27.682834+0000","last_undegraded":"2024-03-06T07:26:27.682834+0000","last_fullsized":"2024-03-06T07:26:27.682834+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:38.169688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.679751+0000","last_change":"2024-03-06T07:26:25.675542+0000","last_active":"2024-03-06T07:26:27.679751+0000","last_peered":"2024-03-06T07:26:27.679751+0000","last_clean":"2024-03-06T07:26:27.679751+0000","last_became_active":"2024-03-06T07:26:25.675297+0000","last_became_peered":"2024-03-06T07:26:25.675297+0000","last_unstale":"2024-03-06T07:26:27.679751+0000","last_undegraded":"2024-03-06T07:26:27.679751+0000","last_fullsized":"2024-03-06T07:26:27.679751+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:37.409384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683746+0000","last_change":"2024-03-06T07:26:17.635968+0000","last_active":"2024-03-06T07:26:27.683746+0000","last_peered":"2024-03-06T07:26:27.683746+0000","last_clean":"2024-03-06T07:26:27.683746+0000","last_became_active":"2024-03-06T07:26:17.635095+0000","last_became_peered":"2024-03-06T07:26:17.635095+0000","last_unstale":"2024-03-06T07:26:27.683746+0000","last_undegraded":"2024-03-06T07:26:27.683746+0000","last_fullsized":"2024-03-06T07:26:27.683746+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:26.879181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"148'4","reported_seq":26,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683934+0000","last_change":"2024-03-06T07:26:16.626240+0000","last_active":"2024-03-06T07:26:27.683934+0000","last_peered":"2024-03-06T07:26:27.683934+0000","last_clean":"2024-03-06T07:26:27.683934+0000","last_became_active":"2024-03-06T07:26:16.626011+0000","last_became_peered":"2024-03-06T07:26:16.626011+0000","last_unstale":"2024-03-06T07:26:27.683934+0000","last_undegraded":"2024-03-06T07:26:27.683934+0000","last_fullsized":"2024-03-06T07:26:27.683934+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:30.259380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683188+0000","last_change":"2024-03-06T07:26:26.703425+0000","last_active":"2024-03-06T07:26:27.683188+0000","last_peered":"2024-03-06T07:26:27.683188+0000","last_clean":"2024-03-06T07:26:27.683188+0000","last_became_active":"2024-03-06T07:26:26.703116+0000","last_became_peered":"2024-03-06T07:26:26.703116+0000","last_unstale":"2024-03-06T07:26:27.683188+0000","last_undegraded":"2024-03-06T07:26:27.683188+0000","last_fullsized":"2024-03-06T07:26:27.683188+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:11.283568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"152'8","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854098+0000","last_change":"2024-03-06T07:26:25.674717+0000","last_active":"2024-03-06T07:26:27.854098+0000","last_peered":"2024-03-06T07:26:27.854098+0000","last_clean":"2024-03-06T07:26:27.854098+0000","last_became_active":"2024-03-06T07:26:25.674451+0000","last_became_peered":"2024-03-06T07:26:25.674451+0000","last_unstale":"2024-03-06T07:26:27.854098+0000","last_undegraded":"2024-03-06T07:26:27.854098+0000","last_fullsized":"2024-03-06T07:26:27.854098+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:33.650729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853174+0000","last_change":"2024-03-06T07:26:17.627512+0000","last_active":"2024-03-06T07:26:27.853174+0000","last_peered":"2024-03-06T07:26:27.853174+0000","last_clean":"2024-03-06T07:26:27.853174+0000","last_became_active":"2024-03-06T07:26:17.627235+0000","last_became_peered":"2024-03-06T07:26:17.627235+0000","last_unstale":"2024-03-06T07:26:27.853174+0000","last_undegraded":"2024-03-06T07:26:27.853174+0000","last_fullsized":"2024-03-06T07:26:27.853174+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:19.245848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"148'1","reported_seq":23,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680614+0000","last_change":"2024-03-06T07:26:16.625703+0000","last_active":"2024-03-06T07:26:27.680614+0000","last_peered":"2024-03-06T07:26:27.680614+0000","last_clean":"2024-03-06T07:26:27.680614+0000","last_became_active":"2024-03-06T07:26:16.625426+0000","last_became_peered":"2024-03-06T07:26:16.625426+0000","last_unstale":"2024-03-06T07:26:27.680614+0000","last_undegraded":"2024-03-06T07:26:27.680614+0000","last_fullsized":"2024-03-06T07:26:27.680614+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:39.946462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.699883+0000","last_change":"2024-03-06T07:26:26.695824+0000","last_active":"2024-03-06T07:26:27.699883+0000","last_peered":"2024-03-06T07:26:27.699883+0000","last_clean":"2024-03-06T07:26:27.699883+0000","last_became_active":"2024-03-06T07:26:26.695570+0000","last_became_peered":"2024-03-06T07:26:26.695570+0000","last_unstale":"2024-03-06T07:26:27.699883+0000","last_undegraded":"2024-03-06T07:26:27.699883+0000","last_fullsized":"2024-03-06T07:26:27.699883+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:05.612535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"152'2","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682794+0000","last_change":"2024-03-06T07:26:25.680772+0000","last_active":"2024-03-06T07:26:27.682794+0000","last_peered":"2024-03-06T07:26:27.682794+0000","last_clean":"2024-03-06T07:26:27.682794+0000","last_became_active":"2024-03-06T07:26:25.680514+0000","last_became_peered":"2024-03-06T07:26:25.680514+0000","last_unstale":"2024-03-06T07:26:27.682794+0000","last_undegraded":"2024-03-06T07:26:27.682794+0000","last_fullsized":"2024-03-06T07:26:27.682794+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:54.742105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680221+0000","last_change":"2024-03-06T07:26:17.632107+0000","last_active":"2024-03-06T07:26:27.680221+0000","last_peered":"2024-03-06T07:26:27.680221+0000","last_clean":"2024-03-06T07:26:27.680221+0000","last_became_active":"2024-03-06T07:26:17.631481+0000","last_became_peered":"2024-03-06T07:26:17.631481+0000","last_unstale":"2024-03-06T07:26:27.680221+0000","last_undegraded":"2024-03-06T07:26:27.680221+0000","last_fullsized":"2024-03-06T07:26:27.680221+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:20.662554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"148'2","reported_seq":24,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854829+0000","last_change":"2024-03-06T07:26:16.620199+0000","last_active":"2024-03-06T07:26:27.854829+0000","last_peered":"2024-03-06T07:26:27.854829+0000","last_clean":"2024-03-06T07:26:27.854829+0000","last_became_active":"2024-03-06T07:26:16.619877+0000","last_became_peered":"2024-03-06T07:26:16.619877+0000","last_unstale":"2024-03-06T07:26:27.854829+0000","last_undegraded":"2024-03-06T07:26:27.854829+0000","last_fullsized":"2024-03-06T07:26:27.854829+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:14.648017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853115+0000","last_change":"2024-03-06T07:26:26.697268+0000","last_active":"2024-03-06T07:26:27.853115+0000","last_peered":"2024-03-06T07:26:27.853115+0000","last_clean":"2024-03-06T07:26:27.853115+0000","last_became_active":"2024-03-06T07:26:26.697005+0000","last_became_peered":"2024-03-06T07:26:26.697005+0000","last_unstale":"2024-03-06T07:26:27.853115+0000","last_undegraded":"2024-03-06T07:26:27.853115+0000","last_fullsized":"2024-03-06T07:26:27.853115+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:33.213505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703297+0000","last_change":"2024-03-06T07:26:25.678879+0000","last_active":"2024-03-06T07:26:27.703297+0000","last_peered":"2024-03-06T07:26:27.703297+0000","last_clean":"2024-03-06T07:26:27.703297+0000","last_became_active":"2024-03-06T07:26:25.678532+0000","last_became_peered":"2024-03-06T07:26:25.678532+0000","last_unstale":"2024-03-06T07:26:27.703297+0000","last_undegraded":"2024-03-06T07:26:27.703297+0000","last_fullsized":"2024-03-06T07:26:27.703297+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:33.316750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684628+0000","last_change":"2024-03-06T07:26:17.634295+0000","last_active":"2024-03-06T07:26:27.684628+0000","last_peered":"2024-03-06T07:26:27.684628+0000","last_clean":"2024-03-06T07:26:27.684628+0000","last_became_active":"2024-03-06T07:26:17.633744+0000","last_became_peered":"2024-03-06T07:26:17.633744+0000","last_unstale":"2024-03-06T07:26:27.684628+0000","last_undegraded":"2024-03-06T07:26:27.684628+0000","last_fullsized":"2024-03-06T07:26:27.684628+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:14.404340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"148'2","reported_seq":24,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700248+0000","last_change":"2024-03-06T07:26:16.616715+0000","last_active":"2024-03-06T07:26:27.700248+0000","last_peered":"2024-03-06T07:26:27.700248+0000","last_clean":"2024-03-06T07:26:27.700248+0000","last_became_active":"2024-03-06T07:26:16.616466+0000","last_became_peered":"2024-03-06T07:26:16.616466+0000","last_unstale":"2024-03-06T07:26:27.700248+0000","last_undegraded":"2024-03-06T07:26:27.700248+0000","last_fullsized":"2024-03-06T07:26:27.700248+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:54.279225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683734+0000","last_change":"2024-03-06T07:26:26.705862+0000","last_active":"2024-03-06T07:26:27.683734+0000","last_peered":"2024-03-06T07:26:27.683734+0000","last_clean":"2024-03-06T07:26:27.683734+0000","last_became_active":"2024-03-06T07:26:26.705261+0000","last_became_peered":"2024-03-06T07:26:26.705261+0000","last_unstale":"2024-03-06T07:26:27.683734+0000","last_undegraded":"2024-03-06T07:26:27.683734+0000","last_fullsized":"2024-03-06T07:26:27.683734+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:18.310254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"152'3","reported_seq":17,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680332+0000","last_change":"2024-03-06T07:26:25.678093+0000","last_active":"2024-03-06T07:26:27.680332+0000","last_peered":"2024-03-06T07:26:27.680332+0000","last_clean":"2024-03-06T07:26:27.680332+0000","last_became_active":"2024-03-06T07:26:25.677685+0000","last_became_peered":"2024-03-06T07:26:25.677685+0000","last_unstale":"2024-03-06T07:26:27.680332+0000","last_undegraded":"2024-03-06T07:26:27.680332+0000","last_fullsized":"2024-03-06T07:26:27.680332+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:47.861522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702555+0000","last_change":"2024-03-06T07:26:17.631966+0000","last_active":"2024-03-06T07:26:27.702555+0000","last_peered":"2024-03-06T07:26:27.702555+0000","last_clean":"2024-03-06T07:26:27.702555+0000","last_became_active":"2024-03-06T07:26:17.631673+0000","last_became_peered":"2024-03-06T07:26:17.631673+0000","last_unstale":"2024-03-06T07:26:27.702555+0000","last_undegraded":"2024-03-06T07:26:27.702555+0000","last_fullsized":"2024-03-06T07:26:27.702555+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:26.879181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"148'1","reported_seq":23,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702812+0000","last_change":"2024-03-06T07:26:16.619602+0000","last_active":"2024-03-06T07:26:27.702812+0000","last_peered":"2024-03-06T07:26:27.702812+0000","last_clean":"2024-03-06T07:26:27.702812+0000","last_became_active":"2024-03-06T07:26:16.619282+0000","last_became_peered":"2024-03-06T07:26:16.619282+0000","last_unstale":"2024-03-06T07:26:27.702812+0000","last_undegraded":"2024-03-06T07:26:27.702812+0000","last_fullsized":"2024-03-06T07:26:27.702812+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:59.587645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702976+0000","last_change":"2024-03-06T07:26:26.703177+0000","last_active":"2024-03-06T07:26:27.702976+0000","last_peered":"2024-03-06T07:26:27.702976+0000","last_clean":"2024-03-06T07:26:27.702976+0000","last_became_active":"2024-03-06T07:26:26.702905+0000","last_became_peered":"2024-03-06T07:26:26.702905+0000","last_unstale":"2024-03-06T07:26:27.702976+0000","last_undegraded":"2024-03-06T07:26:27.702976+0000","last_fullsized":"2024-03-06T07:26:27.702976+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:51.224651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"152'6","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682827+0000","last_change":"2024-03-06T07:26:25.678836+0000","last_active":"2024-03-06T07:26:27.682827+0000","last_peered":"2024-03-06T07:26:27.682827+0000","last_clean":"2024-03-06T07:26:27.682827+0000","last_became_active":"2024-03-06T07:26:25.678543+0000","last_became_peered":"2024-03-06T07:26:25.678543+0000","last_unstale":"2024-03-06T07:26:27.682827+0000","last_undegraded":"2024-03-06T07:26:27.682827+0000","last_fullsized":"2024-03-06T07:26:27.682827+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:47.884750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700506+0000","last_change":"2024-03-06T07:26:17.628669+0000","last_active":"2024-03-06T07:26:27.700506+0000","last_peered":"2024-03-06T07:26:27.700506+0000","last_clean":"2024-03-06T07:26:27.700506+0000","last_became_active":"2024-03-06T07:26:17.628371+0000","last_became_peered":"2024-03-06T07:26:17.628371+0000","last_unstale":"2024-03-06T07:26:27.700506+0000","last_undegraded":"2024-03-06T07:26:27.700506+0000","last_fullsized":"2024-03-06T07:26:27.700506+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:16.499694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"148'1","reported_seq":23,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680358+0000","last_change":"2024-03-06T07:26:16.615912+0000","last_active":"2024-03-06T07:26:27.680358+0000","last_peered":"2024-03-06T07:26:27.680358+0000","last_clean":"2024-03-06T07:26:27.680358+0000","last_became_active":"2024-03-06T07:26:16.615625+0000","last_became_peered":"2024-03-06T07:26:16.615625+0000","last_unstale":"2024-03-06T07:26:27.680358+0000","last_undegraded":"2024-03-06T07:26:27.680358+0000","last_fullsized":"2024-03-06T07:26:27.680358+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:24.591774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684700+0000","last_change":"2024-03-06T07:26:26.708533+0000","last_active":"2024-03-06T07:26:27.684700+0000","last_peered":"2024-03-06T07:26:27.684700+0000","last_clean":"2024-03-06T07:26:27.684700+0000","last_became_active":"2024-03-06T07:26:26.708206+0000","last_became_peered":"2024-03-06T07:26:26.708206+0000","last_unstale":"2024-03-06T07:26:27.684700+0000","last_undegraded":"2024-03-06T07:26:27.684700+0000","last_fullsized":"2024-03-06T07:26:27.684700+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:38.410943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854315+0000","last_change":"2024-03-06T07:26:25.677850+0000","last_active":"2024-03-06T07:26:27.854315+0000","last_peered":"2024-03-06T07:26:27.854315+0000","last_clean":"2024-03-06T07:26:27.854315+0000","last_became_active":"2024-03-06T07:26:25.677154+0000","last_became_peered":"2024-03-06T07:26:25.677154+0000","last_unstale":"2024-03-06T07:26:27.854315+0000","last_undegraded":"2024-03-06T07:26:27.854315+0000","last_fullsized":"2024-03-06T07:26:27.854315+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:13.665071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700223+0000","last_change":"2024-03-06T07:26:17.626866+0000","last_active":"2024-03-06T07:26:27.700223+0000","last_peered":"2024-03-06T07:26:27.700223+0000","last_clean":"2024-03-06T07:26:27.700223+0000","last_became_active":"2024-03-06T07:26:17.626586+0000","last_became_peered":"2024-03-06T07:26:17.626586+0000","last_unstale":"2024-03-06T07:26:27.700223+0000","last_undegraded":"2024-03-06T07:26:27.700223+0000","last_fullsized":"2024-03-06T07:26:27.700223+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:34.182149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700639+0000","last_change":"2024-03-06T07:26:16.622797+0000","last_active":"2024-03-06T07:26:27.700639+0000","last_peered":"2024-03-06T07:26:27.700639+0000","last_clean":"2024-03-06T07:26:27.700639+0000","last_became_active":"2024-03-06T07:26:16.622534+0000","last_became_peered":"2024-03-06T07:26:16.622534+0000","last_unstale":"2024-03-06T07:26:27.700639+0000","last_undegraded":"2024-03-06T07:26:27.700639+0000","last_fullsized":"2024-03-06T07:26:27.700639+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:39.697277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854075+0000","last_change":"2024-03-06T07:26:26.688274+0000","last_active":"2024-03-06T07:26:27.854075+0000","last_peered":"2024-03-06T07:26:27.854075+0000","last_clean":"2024-03-06T07:26:27.854075+0000","last_became_active":"2024-03-06T07:26:26.687023+0000","last_became_peered":"2024-03-06T07:26:26.687023+0000","last_unstale":"2024-03-06T07:26:27.854075+0000","last_undegraded":"2024-03-06T07:26:27.854075+0000","last_fullsized":"2024-03-06T07:26:27.854075+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:39.420779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702829+0000","last_change":"2024-03-06T07:26:25.679443+0000","last_active":"2024-03-06T07:26:27.702829+0000","last_peered":"2024-03-06T07:26:27.702829+0000","last_clean":"2024-03-06T07:26:27.702829+0000","last_became_active":"2024-03-06T07:26:25.679202+0000","last_became_peered":"2024-03-06T07:26:25.679202+0000","last_unstale":"2024-03-06T07:26:27.702829+0000","last_undegraded":"2024-03-06T07:26:27.702829+0000","last_fullsized":"2024-03-06T07:26:27.702829+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:10.282009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683851+0000","last_change":"2024-03-06T07:26:17.632829+0000","last_active":"2024-03-06T07:26:27.683851+0000","last_peered":"2024-03-06T07:26:27.683851+0000","last_clean":"2024-03-06T07:26:27.683851+0000","last_became_active":"2024-03-06T07:26:17.631051+0000","last_became_peered":"2024-03-06T07:26:17.631051+0000","last_unstale":"2024-03-06T07:26:27.683851+0000","last_undegraded":"2024-03-06T07:26:27.683851+0000","last_fullsized":"2024-03-06T07:26:27.683851+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:09.224572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702266+0000","last_change":"2024-03-06T07:26:16.618575+0000","last_active":"2024-03-06T07:26:27.702266+0000","last_peered":"2024-03-06T07:26:27.702266+0000","last_clean":"2024-03-06T07:26:27.702266+0000","last_became_active":"2024-03-06T07:26:16.618143+0000","last_became_peered":"2024-03-06T07:26:16.618143+0000","last_unstale":"2024-03-06T07:26:27.702266+0000","last_undegraded":"2024-03-06T07:26:27.702266+0000","last_fullsized":"2024-03-06T07:26:27.702266+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:48.709582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703107+0000","last_change":"2024-03-06T07:26:26.707394+0000","last_active":"2024-03-06T07:26:27.703107+0000","last_peered":"2024-03-06T07:26:27.703107+0000","last_clean":"2024-03-06T07:26:27.703107+0000","last_became_active":"2024-03-06T07:26:26.706799+0000","last_became_peered":"2024-03-06T07:26:26.706799+0000","last_unstale":"2024-03-06T07:26:27.703107+0000","last_undegraded":"2024-03-06T07:26:27.703107+0000","last_fullsized":"2024-03-06T07:26:27.703107+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:19.825140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682565+0000","last_change":"2024-03-06T07:26:25.675283+0000","last_active":"2024-03-06T07:26:27.682565+0000","last_peered":"2024-03-06T07:26:27.682565+0000","last_clean":"2024-03-06T07:26:27.682565+0000","last_became_active":"2024-03-06T07:26:25.675001+0000","last_became_peered":"2024-03-06T07:26:25.675001+0000","last_unstale":"2024-03-06T07:26:27.682565+0000","last_undegraded":"2024-03-06T07:26:27.682565+0000","last_fullsized":"2024-03-06T07:26:27.682565+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:51.939085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683636+0000","last_change":"2024-03-06T07:26:17.636599+0000","last_active":"2024-03-06T07:26:27.683636+0000","last_peered":"2024-03-06T07:26:27.683636+0000","last_clean":"2024-03-06T07:26:27.683636+0000","last_became_active":"2024-03-06T07:26:17.635451+0000","last_became_peered":"2024-03-06T07:26:17.635451+0000","last_unstale":"2024-03-06T07:26:27.683636+0000","last_undegraded":"2024-03-06T07:26:27.683636+0000","last_fullsized":"2024-03-06T07:26:27.683636+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:27.398343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854475+0000","last_change":"2024-03-06T07:26:16.627460+0000","last_active":"2024-03-06T07:26:27.854475+0000","last_peered":"2024-03-06T07:26:27.854475+0000","last_clean":"2024-03-06T07:26:27.854475+0000","last_became_active":"2024-03-06T07:26:16.626345+0000","last_became_peered":"2024-03-06T07:26:16.626345+0000","last_unstale":"2024-03-06T07:26:27.854475+0000","last_undegraded":"2024-03-06T07:26:27.854475+0000","last_fullsized":"2024-03-06T07:26:27.854475+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:24.257795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682387+0000","last_change":"2024-03-06T07:26:26.704789+0000","last_active":"2024-03-06T07:26:27.682387+0000","last_peered":"2024-03-06T07:26:27.682387+0000","last_clean":"2024-03-06T07:26:27.682387+0000","last_became_active":"2024-03-06T07:26:26.703629+0000","last_became_peered":"2024-03-06T07:26:26.703629+0000","last_unstale":"2024-03-06T07:26:27.682387+0000","last_undegraded":"2024-03-06T07:26:27.682387+0000","last_fullsized":"2024-03-06T07:26:27.682387+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:48.044634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"152'1","reported_seq":15,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853372+0000","last_change":"2024-03-06T07:26:25.672473+0000","last_active":"2024-03-06T07:26:27.853372+0000","last_peered":"2024-03-06T07:26:27.853372+0000","last_clean":"2024-03-06T07:26:27.853372+0000","last_became_active":"2024-03-06T07:26:25.672210+0000","last_became_peered":"2024-03-06T07:26:25.672210+0000","last_unstale":"2024-03-06T07:26:27.853372+0000","last_undegraded":"2024-03-06T07:26:27.853372+0000","last_fullsized":"2024-03-06T07:26:27.853372+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:09.602353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853363+0000","last_change":"2024-03-06T07:26:17.634986+0000","last_active":"2024-03-06T07:26:27.853363+0000","last_peered":"2024-03-06T07:26:27.853363+0000","last_clean":"2024-03-06T07:26:27.853363+0000","last_became_active":"2024-03-06T07:26:17.634680+0000","last_became_peered":"2024-03-06T07:26:17.634680+0000","last_unstale":"2024-03-06T07:26:27.853363+0000","last_undegraded":"2024-03-06T07:26:27.853363+0000","last_fullsized":"2024-03-06T07:26:27.853363+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:48.372843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"148'3","reported_seq":25,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683510+0000","last_change":"2024-03-06T07:26:16.623947+0000","last_active":"2024-03-06T07:26:27.683510+0000","last_peered":"2024-03-06T07:26:27.683510+0000","last_clean":"2024-03-06T07:26:27.683510+0000","last_became_active":"2024-03-06T07:26:16.623662+0000","last_became_peered":"2024-03-06T07:26:16.623662+0000","last_unstale":"2024-03-06T07:26:27.683510+0000","last_undegraded":"2024-03-06T07:26:27.683510+0000","last_fullsized":"2024-03-06T07:26:27.683510+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:00.411414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702187+0000","last_change":"2024-03-06T07:26:26.687868+0000","last_active":"2024-03-06T07:26:27.702187+0000","last_peered":"2024-03-06T07:26:27.702187+0000","last_clean":"2024-03-06T07:26:27.702187+0000","last_became_active":"2024-03-06T07:26:26.687595+0000","last_became_peered":"2024-03-06T07:26:26.687595+0000","last_unstale":"2024-03-06T07:26:27.702187+0000","last_undegraded":"2024-03-06T07:26:27.702187+0000","last_fullsized":"2024-03-06T07:26:27.702187+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:27.612436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"152'1","reported_seq":15,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680108+0000","last_change":"2024-03-06T07:26:25.677952+0000","last_active":"2024-03-06T07:26:27.680108+0000","last_peered":"2024-03-06T07:26:27.680108+0000","last_clean":"2024-03-06T07:26:27.680108+0000","last_became_active":"2024-03-06T07:26:25.675695+0000","last_became_peered":"2024-03-06T07:26:25.675695+0000","last_unstale":"2024-03-06T07:26:27.680108+0000","last_undegraded":"2024-03-06T07:26:27.680108+0000","last_fullsized":"2024-03-06T07:26:27.680108+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:18.823580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.699991+0000","last_change":"2024-03-06T07:26:17.626440+0000","last_active":"2024-03-06T07:26:27.699991+0000","last_peered":"2024-03-06T07:26:27.699991+0000","last_clean":"2024-03-06T07:26:27.699991+0000","last_became_active":"2024-03-06T07:26:17.626149+0000","last_became_peered":"2024-03-06T07:26:17.626149+0000","last_unstale":"2024-03-06T07:26:27.699991+0000","last_undegraded":"2024-03-06T07:26:27.699991+0000","last_fullsized":"2024-03-06T07:26:27.699991+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:46.191755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"148'6","reported_seq":28,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853822+0000","last_change":"2024-03-06T07:26:16.627536+0000","last_active":"2024-03-06T07:26:27.853822+0000","last_peered":"2024-03-06T07:26:27.853822+0000","last_clean":"2024-03-06T07:26:27.853822+0000","last_became_active":"2024-03-06T07:26:16.627229+0000","last_became_peered":"2024-03-06T07:26:16.627229+0000","last_unstale":"2024-03-06T07:26:27.853822+0000","last_undegraded":"2024-03-06T07:26:27.853822+0000","last_fullsized":"2024-03-06T07:26:27.853822+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:00.825243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854293+0000","last_change":"2024-03-06T07:26:26.700560+0000","last_active":"2024-03-06T07:26:27.854293+0000","last_peered":"2024-03-06T07:26:27.854293+0000","last_clean":"2024-03-06T07:26:27.854293+0000","last_became_active":"2024-03-06T07:26:26.700215+0000","last_became_peered":"2024-03-06T07:26:26.700215+0000","last_unstale":"2024-03-06T07:26:27.854293+0000","last_undegraded":"2024-03-06T07:26:27.854293+0000","last_fullsized":"2024-03-06T07:26:27.854293+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:18.310254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"152'2","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854601+0000","last_change":"2024-03-06T07:26:25.678187+0000","last_active":"2024-03-06T07:26:27.854601+0000","last_peered":"2024-03-06T07:26:27.854601+0000","last_clean":"2024-03-06T07:26:27.854601+0000","last_became_active":"2024-03-06T07:26:25.677933+0000","last_became_peered":"2024-03-06T07:26:25.677933+0000","last_unstale":"2024-03-06T07:26:27.854601+0000","last_undegraded":"2024-03-06T07:26:27.854601+0000","last_fullsized":"2024-03-06T07:26:27.854601+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:22.459161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853863+0000","last_change":"2024-03-06T07:26:17.638945+0000","last_active":"2024-03-06T07:26:27.853863+0000","last_peered":"2024-03-06T07:26:27.853863+0000","last_clean":"2024-03-06T07:26:27.853863+0000","last_became_active":"2024-03-06T07:26:17.638664+0000","last_became_peered":"2024-03-06T07:26:17.638664+0000","last_unstale":"2024-03-06T07:26:27.853863+0000","last_undegraded":"2024-03-06T07:26:27.853863+0000","last_fullsized":"2024-03-06T07:26:27.853863+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:15.414267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"148'2","reported_seq":24,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680199+0000","last_change":"2024-03-06T07:26:16.625884+0000","last_active":"2024-03-06T07:26:27.680199+0000","last_peered":"2024-03-06T07:26:27.680199+0000","last_clean":"2024-03-06T07:26:27.680199+0000","last_became_active":"2024-03-06T07:26:16.625605+0000","last_became_peered":"2024-03-06T07:26:16.625605+0000","last_unstale":"2024-03-06T07:26:27.680199+0000","last_undegraded":"2024-03-06T07:26:27.680199+0000","last_fullsized":"2024-03-06T07:26:27.680199+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:13.400206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854578+0000","last_change":"2024-03-06T07:26:26.701244+0000","last_active":"2024-03-06T07:26:27.854578+0000","last_peered":"2024-03-06T07:26:27.854578+0000","last_clean":"2024-03-06T07:26:27.854578+0000","last_became_active":"2024-03-06T07:26:26.700943+0000","last_became_peered":"2024-03-06T07:26:26.700943+0000","last_unstale":"2024-03-06T07:26:27.854578+0000","last_undegraded":"2024-03-06T07:26:27.854578+0000","last_fullsized":"2024-03-06T07:26:27.854578+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:58.770097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"152'2","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682985+0000","last_change":"2024-03-06T07:26:25.677490+0000","last_active":"2024-03-06T07:26:27.682985+0000","last_peered":"2024-03-06T07:26:27.682985+0000","last_clean":"2024-03-06T07:26:27.682985+0000","last_became_active":"2024-03-06T07:26:25.677161+0000","last_became_peered":"2024-03-06T07:26:25.677161+0000","last_unstale":"2024-03-06T07:26:27.682985+0000","last_undegraded":"2024-03-06T07:26:27.682985+0000","last_fullsized":"2024-03-06T07:26:27.682985+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:04.189705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702839+0000","last_change":"2024-03-06T07:26:17.631376+0000","last_active":"2024-03-06T07:26:27.702839+0000","last_peered":"2024-03-06T07:26:27.702839+0000","last_clean":"2024-03-06T07:26:27.702839+0000","last_became_active":"2024-03-06T07:26:17.631068+0000","last_became_peered":"2024-03-06T07:26:17.631068+0000","last_unstale":"2024-03-06T07:26:27.702839+0000","last_undegraded":"2024-03-06T07:26:27.702839+0000","last_fullsized":"2024-03-06T07:26:27.702839+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:46.783070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853507+0000","last_change":"2024-03-06T07:26:16.625352+0000","last_active":"2024-03-06T07:26:27.853507+0000","last_peered":"2024-03-06T07:26:27.853507+0000","last_clean":"2024-03-06T07:26:27.853507+0000","last_became_active":"2024-03-06T07:26:16.624893+0000","last_became_peered":"2024-03-06T07:26:16.624893+0000","last_unstale":"2024-03-06T07:26:27.853507+0000","last_undegraded":"2024-03-06T07:26:27.853507+0000","last_fullsized":"2024-03-06T07:26:27.853507+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:29.596513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684298+0000","last_change":"2024-03-06T07:26:26.696441+0000","last_active":"2024-03-06T07:26:27.684298+0000","last_peered":"2024-03-06T07:26:27.684298+0000","last_clean":"2024-03-06T07:26:27.684298+0000","last_became_active":"2024-03-06T07:26:26.695929+0000","last_became_peered":"2024-03-06T07:26:26.695929+0000","last_unstale":"2024-03-06T07:26:27.684298+0000","last_undegraded":"2024-03-06T07:26:27.684298+0000","last_fullsized":"2024-03-06T07:26:27.684298+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:58.389676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"152'4","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680696+0000","last_change":"2024-03-06T07:26:25.672657+0000","last_active":"2024-03-06T07:26:27.680696+0000","last_peered":"2024-03-06T07:26:27.680696+0000","last_clean":"2024-03-06T07:26:27.680696+0000","last_became_active":"2024-03-06T07:26:25.672330+0000","last_became_peered":"2024-03-06T07:26:25.672330+0000","last_unstale":"2024-03-06T07:26:27.680696+0000","last_undegraded":"2024-03-06T07:26:27.680696+0000","last_fullsized":"2024-03-06T07:26:27.680696+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:50.223092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854791+0000","last_change":"2024-03-06T07:26:17.639031+0000","last_active":"2024-03-06T07:26:27.854791+0000","last_peered":"2024-03-06T07:26:27.854791+0000","last_clean":"2024-03-06T07:26:27.854791+0000","last_became_active":"2024-03-06T07:26:17.638716+0000","last_became_peered":"2024-03-06T07:26:17.638716+0000","last_unstale":"2024-03-06T07:26:27.854791+0000","last_undegraded":"2024-03-06T07:26:27.854791+0000","last_fullsized":"2024-03-06T07:26:27.854791+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:46.783070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853941+0000","last_change":"2024-03-06T07:26:16.620528+0000","last_active":"2024-03-06T07:26:27.853941+0000","last_peered":"2024-03-06T07:26:27.853941+0000","last_clean":"2024-03-06T07:26:27.853941+0000","last_became_active":"2024-03-06T07:26:16.620245+0000","last_became_peered":"2024-03-06T07:26:16.620245+0000","last_unstale":"2024-03-06T07:26:27.853941+0000","last_undegraded":"2024-03-06T07:26:27.853941+0000","last_fullsized":"2024-03-06T07:26:27.853941+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:39.028254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700169+0000","last_change":"2024-03-06T07:26:26.695916+0000","last_active":"2024-03-06T07:26:27.700169+0000","last_peered":"2024-03-06T07:26:27.700169+0000","last_clean":"2024-03-06T07:26:27.700169+0000","last_became_active":"2024-03-06T07:26:26.695663+0000","last_became_peered":"2024-03-06T07:26:26.695663+0000","last_unstale":"2024-03-06T07:26:27.700169+0000","last_undegraded":"2024-03-06T07:26:27.700169+0000","last_fullsized":"2024-03-06T07:26:27.700169+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:28.555022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"152'1","reported_seq":15,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854342+0000","last_change":"2024-03-06T07:26:25.673722+0000","last_active":"2024-03-06T07:26:27.854342+0000","last_peered":"2024-03-06T07:26:27.854342+0000","last_clean":"2024-03-06T07:26:27.854342+0000","last_became_active":"2024-03-06T07:26:25.672078+0000","last_became_peered":"2024-03-06T07:26:25.672078+0000","last_unstale":"2024-03-06T07:26:27.854342+0000","last_undegraded":"2024-03-06T07:26:27.854342+0000","last_fullsized":"2024-03-06T07:26:27.854342+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:36.768745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700563+0000","last_change":"2024-03-06T07:26:17.641468+0000","last_active":"2024-03-06T07:26:27.700563+0000","last_peered":"2024-03-06T07:26:27.700563+0000","last_clean":"2024-03-06T07:26:27.700563+0000","last_became_active":"2024-03-06T07:26:17.641155+0000","last_became_peered":"2024-03-06T07:26:17.641155+0000","last_unstale":"2024-03-06T07:26:27.700563+0000","last_undegraded":"2024-03-06T07:26:27.700563+0000","last_fullsized":"2024-03-06T07:26:27.700563+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:27.464355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"148'8","reported_seq":30,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702314+0000","last_change":"2024-03-06T07:26:16.620548+0000","last_active":"2024-03-06T07:26:27.702314+0000","last_peered":"2024-03-06T07:26:27.702314+0000","last_clean":"2024-03-06T07:26:27.702314+0000","last_became_active":"2024-03-06T07:26:16.620289+0000","last_became_peered":"2024-03-06T07:26:16.620289+0000","last_unstale":"2024-03-06T07:26:27.702314+0000","last_undegraded":"2024-03-06T07:26:27.702314+0000","last_fullsized":"2024-03-06T07:26:27.702314+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:08.155799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703250+0000","last_change":"2024-03-06T07:26:26.691751+0000","last_active":"2024-03-06T07:26:27.703250+0000","last_peered":"2024-03-06T07:26:27.703250+0000","last_clean":"2024-03-06T07:26:27.703250+0000","last_became_active":"2024-03-06T07:26:26.691505+0000","last_became_peered":"2024-03-06T07:26:26.691505+0000","last_unstale":"2024-03-06T07:26:27.703250+0000","last_undegraded":"2024-03-06T07:26:27.703250+0000","last_fullsized":"2024-03-06T07:26:27.703250+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:24.976580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.699874+0000","last_change":"2024-03-06T07:26:25.676004+0000","last_active":"2024-03-06T07:26:27.699874+0000","last_peered":"2024-03-06T07:26:27.699874+0000","last_clean":"2024-03-06T07:26:27.699874+0000","last_became_active":"2024-03-06T07:26:25.675756+0000","last_became_peered":"2024-03-06T07:26:25.675756+0000","last_unstale":"2024-03-06T07:26:27.699874+0000","last_undegraded":"2024-03-06T07:26:27.699874+0000","last_fullsized":"2024-03-06T07:26:27.699874+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:09.976162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702169+0000","last_change":"2024-03-06T07:26:17.629187+0000","last_active":"2024-03-06T07:26:27.702169+0000","last_peered":"2024-03-06T07:26:27.702169+0000","last_clean":"2024-03-06T07:26:27.702169+0000","last_became_active":"2024-03-06T07:26:17.628903+0000","last_became_peered":"2024-03-06T07:26:17.628903+0000","last_unstale":"2024-03-06T07:26:27.702169+0000","last_undegraded":"2024-03-06T07:26:27.702169+0000","last_fullsized":"2024-03-06T07:26:27.702169+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:25.173456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"148'1","reported_seq":23,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854561+0000","last_change":"2024-03-06T07:26:16.617108+0000","last_active":"2024-03-06T07:26:27.854561+0000","last_peered":"2024-03-06T07:26:27.854561+0000","last_clean":"2024-03-06T07:26:27.854561+0000","last_became_active":"2024-03-06T07:26:16.616776+0000","last_became_peered":"2024-03-06T07:26:16.616776+0000","last_unstale":"2024-03-06T07:26:27.854561+0000","last_undegraded":"2024-03-06T07:26:27.854561+0000","last_fullsized":"2024-03-06T07:26:27.854561+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:01.223054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684029+0000","last_change":"2024-03-06T07:26:26.699183+0000","last_active":"2024-03-06T07:26:27.684029+0000","last_peered":"2024-03-06T07:26:27.684029+0000","last_clean":"2024-03-06T07:26:27.684029+0000","last_became_active":"2024-03-06T07:26:26.698593+0000","last_became_peered":"2024-03-06T07:26:26.698593+0000","last_unstale":"2024-03-06T07:26:27.684029+0000","last_undegraded":"2024-03-06T07:26:27.684029+0000","last_fullsized":"2024-03-06T07:26:27.684029+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:44.977293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853384+0000","last_change":"2024-03-06T07:26:25.679689+0000","last_active":"2024-03-06T07:26:27.853384+0000","last_peered":"2024-03-06T07:26:27.853384+0000","last_clean":"2024-03-06T07:26:27.853384+0000","last_became_active":"2024-03-06T07:26:25.679402+0000","last_became_peered":"2024-03-06T07:26:25.679402+0000","last_unstale":"2024-03-06T07:26:27.853384+0000","last_undegraded":"2024-03-06T07:26:27.853384+0000","last_fullsized":"2024-03-06T07:26:27.853384+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:50.182875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853697+0000","last_change":"2024-03-06T07:26:17.634792+0000","last_active":"2024-03-06T07:26:27.853697+0000","last_peered":"2024-03-06T07:26:27.853697+0000","last_clean":"2024-03-06T07:26:27.853697+0000","last_became_active":"2024-03-06T07:26:17.634367+0000","last_became_peered":"2024-03-06T07:26:17.634367+0000","last_unstale":"2024-03-06T07:26:27.853697+0000","last_undegraded":"2024-03-06T07:26:27.853697+0000","last_fullsized":"2024-03-06T07:26:27.853697+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:11.386543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"148'2","reported_seq":24,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703255+0000","last_change":"2024-03-06T07:26:16.619012+0000","last_active":"2024-03-06T07:26:27.703255+0000","last_peered":"2024-03-06T07:26:27.703255+0000","last_clean":"2024-03-06T07:26:27.703255+0000","last_became_active":"2024-03-06T07:26:16.617718+0000","last_became_peered":"2024-03-06T07:26:16.617718+0000","last_unstale":"2024-03-06T07:26:27.703255+0000","last_undegraded":"2024-03-06T07:26:27.703255+0000","last_fullsized":"2024-03-06T07:26:27.703255+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:08.220437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853468+0000","last_change":"2024-03-06T07:26:26.689460+0000","last_active":"2024-03-06T07:26:27.853468+0000","last_peered":"2024-03-06T07:26:27.853468+0000","last_clean":"2024-03-06T07:26:27.853468+0000","last_became_active":"2024-03-06T07:26:26.689176+0000","last_became_peered":"2024-03-06T07:26:26.689176+0000","last_unstale":"2024-03-06T07:26:27.853468+0000","last_undegraded":"2024-03-06T07:26:27.853468+0000","last_fullsized":"2024-03-06T07:26:27.853468+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:13.113595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684084+0000","last_change":"2024-03-06T07:26:25.677892+0000","last_active":"2024-03-06T07:26:27.684084+0000","last_peered":"2024-03-06T07:26:27.684084+0000","last_clean":"2024-03-06T07:26:27.684084+0000","last_became_active":"2024-03-06T07:26:25.677625+0000","last_became_peered":"2024-03-06T07:26:25.677625+0000","last_unstale":"2024-03-06T07:26:27.684084+0000","last_undegraded":"2024-03-06T07:26:27.684084+0000","last_fullsized":"2024-03-06T07:26:27.684084+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:53.012502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854399+0000","last_change":"2024-03-06T07:26:17.633394+0000","last_active":"2024-03-06T07:26:27.854399+0000","last_peered":"2024-03-06T07:26:27.854399+0000","last_clean":"2024-03-06T07:26:27.854399+0000","last_became_active":"2024-03-06T07:26:17.632438+0000","last_became_peered":"2024-03-06T07:26:17.632438+0000","last_unstale":"2024-03-06T07:26:27.854399+0000","last_undegraded":"2024-03-06T07:26:27.854399+0000","last_fullsized":"2024-03-06T07:26:27.854399+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:12.024002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"148'1","reported_seq":23,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680591+0000","last_change":"2024-03-06T07:26:16.628646+0000","last_active":"2024-03-06T07:26:27.680591+0000","last_peered":"2024-03-06T07:26:27.680591+0000","last_clean":"2024-03-06T07:26:27.680591+0000","last_became_active":"2024-03-06T07:26:16.627947+0000","last_became_peered":"2024-03-06T07:26:16.627947+0000","last_unstale":"2024-03-06T07:26:27.680591+0000","last_undegraded":"2024-03-06T07:26:27.680591+0000","last_fullsized":"2024-03-06T07:26:27.680591+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:25.875046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682566+0000","last_change":"2024-03-06T07:26:26.703286+0000","last_active":"2024-03-06T07:26:27.682566+0000","last_peered":"2024-03-06T07:26:27.682566+0000","last_clean":"2024-03-06T07:26:27.682566+0000","last_became_active":"2024-03-06T07:26:26.702961+0000","last_became_peered":"2024-03-06T07:26:26.702961+0000","last_unstale":"2024-03-06T07:26:27.682566+0000","last_undegraded":"2024-03-06T07:26:27.682566+0000","last_fullsized":"2024-03-06T07:26:27.682566+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:10.885758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684052+0000","last_change":"2024-03-06T07:26:25.677636+0000","last_active":"2024-03-06T07:26:27.684052+0000","last_peered":"2024-03-06T07:26:27.684052+0000","last_clean":"2024-03-06T07:26:27.684052+0000","last_became_active":"2024-03-06T07:26:25.677362+0000","last_became_peered":"2024-03-06T07:26:25.677362+0000","last_unstale":"2024-03-06T07:26:27.684052+0000","last_undegraded":"2024-03-06T07:26:27.684052+0000","last_fullsized":"2024-03-06T07:26:27.684052+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:31.280183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854482+0000","last_change":"2024-03-06T07:26:17.634254+0000","last_active":"2024-03-06T07:26:27.854482+0000","last_peered":"2024-03-06T07:26:27.854482+0000","last_clean":"2024-03-06T07:26:27.854482+0000","last_became_active":"2024-03-06T07:26:17.633961+0000","last_became_peered":"2024-03-06T07:26:17.633961+0000","last_unstale":"2024-03-06T07:26:27.854482+0000","last_undegraded":"2024-03-06T07:26:27.854482+0000","last_fullsized":"2024-03-06T07:26:27.854482+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:59.873611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702650+0000","last_change":"2024-03-06T07:26:16.619532+0000","last_active":"2024-03-06T07:26:27.702650+0000","last_peered":"2024-03-06T07:26:27.702650+0000","last_clean":"2024-03-06T07:26:27.702650+0000","last_became_active":"2024-03-06T07:26:16.619261+0000","last_became_peered":"2024-03-06T07:26:16.619261+0000","last_unstale":"2024-03-06T07:26:27.702650+0000","last_undegraded":"2024-03-06T07:26:27.702650+0000","last_fullsized":"2024-03-06T07:26:27.702650+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:34.031068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684675+0000","last_change":"2024-03-06T07:26:26.695922+0000","last_active":"2024-03-06T07:26:27.684675+0000","last_peered":"2024-03-06T07:26:27.684675+0000","last_clean":"2024-03-06T07:26:27.684675+0000","last_became_active":"2024-03-06T07:26:26.695643+0000","last_became_peered":"2024-03-06T07:26:26.695643+0000","last_unstale":"2024-03-06T07:26:27.684675+0000","last_undegraded":"2024-03-06T07:26:27.684675+0000","last_fullsized":"2024-03-06T07:26:27.684675+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:43.140802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854148+0000","last_change":"2024-03-06T07:26:25.672463+0000","last_active":"2024-03-06T07:26:27.854148+0000","last_peered":"2024-03-06T07:26:27.854148+0000","last_clean":"2024-03-06T07:26:27.854148+0000","last_became_active":"2024-03-06T07:26:25.672192+0000","last_became_peered":"2024-03-06T07:26:25.672192+0000","last_unstale":"2024-03-06T07:26:27.854148+0000","last_undegraded":"2024-03-06T07:26:27.854148+0000","last_fullsized":"2024-03-06T07:26:27.854148+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:54.265085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853647+0000","last_change":"2024-03-06T07:26:17.637879+0000","last_active":"2024-03-06T07:26:27.853647+0000","last_peered":"2024-03-06T07:26:27.853647+0000","last_clean":"2024-03-06T07:26:27.853647+0000","last_became_active":"2024-03-06T07:26:17.637094+0000","last_became_peered":"2024-03-06T07:26:17.637094+0000","last_unstale":"2024-03-06T07:26:27.853647+0000","last_undegraded":"2024-03-06T07:26:27.853647+0000","last_fullsized":"2024-03-06T07:26:27.853647+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:34.084422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"148'2","reported_seq":24,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680265+0000","last_change":"2024-03-06T07:26:16.622856+0000","last_active":"2024-03-06T07:26:27.680265+0000","last_peered":"2024-03-06T07:26:27.680265+0000","last_clean":"2024-03-06T07:26:27.680265+0000","last_became_active":"2024-03-06T07:26:16.622421+0000","last_became_peered":"2024-03-06T07:26:16.622421+0000","last_unstale":"2024-03-06T07:26:27.680265+0000","last_undegraded":"2024-03-06T07:26:27.680265+0000","last_fullsized":"2024-03-06T07:26:27.680265+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:40.651795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700557+0000","last_change":"2024-03-06T07:26:26.701119+0000","last_active":"2024-03-06T07:26:27.700557+0000","last_peered":"2024-03-06T07:26:27.700557+0000","last_clean":"2024-03-06T07:26:27.700557+0000","last_became_active":"2024-03-06T07:26:26.700823+0000","last_became_peered":"2024-03-06T07:26:26.700823+0000","last_unstale":"2024-03-06T07:26:27.700557+0000","last_undegraded":"2024-03-06T07:26:27.700557+0000","last_fullsized":"2024-03-06T07:26:27.700557+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:23.853802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853732+0000","last_change":"2024-03-06T07:26:25.683133+0000","last_active":"2024-03-06T07:26:27.853732+0000","last_peered":"2024-03-06T07:26:27.853732+0000","last_clean":"2024-03-06T07:26:27.853732+0000","last_became_active":"2024-03-06T07:26:25.682836+0000","last_became_peered":"2024-03-06T07:26:25.682836+0000","last_unstale":"2024-03-06T07:26:27.853732+0000","last_undegraded":"2024-03-06T07:26:27.853732+0000","last_fullsized":"2024-03-06T07:26:27.853732+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:44.114707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680586+0000","last_change":"2024-03-06T07:26:17.629188+0000","last_active":"2024-03-06T07:26:27.680586+0000","last_peered":"2024-03-06T07:26:27.680586+0000","last_clean":"2024-03-06T07:26:27.680586+0000","last_became_active":"2024-03-06T07:26:17.628881+0000","last_became_peered":"2024-03-06T07:26:17.628881+0000","last_unstale":"2024-03-06T07:26:27.680586+0000","last_undegraded":"2024-03-06T07:26:27.680586+0000","last_fullsized":"2024-03-06T07:26:27.680586+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:44.957682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854605+0000","last_change":"2024-03-06T07:26:16.625755+0000","last_active":"2024-03-06T07:26:27.854605+0000","last_peered":"2024-03-06T07:26:27.854605+0000","last_clean":"2024-03-06T07:26:27.854605+0000","last_became_active":"2024-03-06T07:26:16.625489+0000","last_became_peered":"2024-03-06T07:26:16.625489+0000","last_unstale":"2024-03-06T07:26:27.854605+0000","last_undegraded":"2024-03-06T07:26:27.854605+0000","last_fullsized":"2024-03-06T07:26:27.854605+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:03.053080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700195+0000","last_change":"2024-03-06T07:26:26.708290+0000","last_active":"2024-03-06T07:26:27.700195+0000","last_peered":"2024-03-06T07:26:27.700195+0000","last_clean":"2024-03-06T07:26:27.700195+0000","last_became_active":"2024-03-06T07:26:26.708035+0000","last_became_peered":"2024-03-06T07:26:26.708035+0000","last_unstale":"2024-03-06T07:26:27.700195+0000","last_undegraded":"2024-03-06T07:26:27.700195+0000","last_fullsized":"2024-03-06T07:26:27.700195+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:40.319895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"152'2","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680716+0000","last_change":"2024-03-06T07:26:25.676230+0000","last_active":"2024-03-06T07:26:27.680716+0000","last_peered":"2024-03-06T07:26:27.680716+0000","last_clean":"2024-03-06T07:26:27.680716+0000","last_became_active":"2024-03-06T07:26:25.675724+0000","last_became_peered":"2024-03-06T07:26:25.675724+0000","last_unstale":"2024-03-06T07:26:27.680716+0000","last_undegraded":"2024-03-06T07:26:27.680716+0000","last_fullsized":"2024-03-06T07:26:27.680716+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:56.435162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680607+0000","last_change":"2024-03-06T07:26:17.629894+0000","last_active":"2024-03-06T07:26:27.680607+0000","last_peered":"2024-03-06T07:26:27.680607+0000","last_clean":"2024-03-06T07:26:27.680607+0000","last_became_active":"2024-03-06T07:26:17.629624+0000","last_became_peered":"2024-03-06T07:26:17.629624+0000","last_unstale":"2024-03-06T07:26:27.680607+0000","last_undegraded":"2024-03-06T07:26:27.680607+0000","last_fullsized":"2024-03-06T07:26:27.680607+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:48.520981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700694+0000","last_change":"2024-03-06T07:26:16.618167+0000","last_active":"2024-03-06T07:26:27.700694+0000","last_peered":"2024-03-06T07:26:27.700694+0000","last_clean":"2024-03-06T07:26:27.700694+0000","last_became_active":"2024-03-06T07:26:16.617893+0000","last_became_peered":"2024-03-06T07:26:16.617893+0000","last_unstale":"2024-03-06T07:26:27.700694+0000","last_undegraded":"2024-03-06T07:26:27.700694+0000","last_fullsized":"2024-03-06T07:26:27.700694+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:33.738360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702793+0000","last_change":"2024-03-06T07:26:26.687658+0000","last_active":"2024-03-06T07:26:27.702793+0000","last_peered":"2024-03-06T07:26:27.702793+0000","last_clean":"2024-03-06T07:26:27.702793+0000","last_became_active":"2024-03-06T07:26:26.687392+0000","last_became_peered":"2024-03-06T07:26:26.687392+0000","last_unstale":"2024-03-06T07:26:27.702793+0000","last_undegraded":"2024-03-06T07:26:27.702793+0000","last_fullsized":"2024-03-06T07:26:27.702793+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:48.863082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"153'1","reported_seq":15,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700114+0000","last_change":"2024-03-06T07:26:25.681294+0000","last_active":"2024-03-06T07:26:27.700114+0000","last_peered":"2024-03-06T07:26:27.700114+0000","last_clean":"2024-03-06T07:26:27.700114+0000","last_became_active":"2024-03-06T07:26:25.681036+0000","last_became_peered":"2024-03-06T07:26:25.681036+0000","last_unstale":"2024-03-06T07:26:27.700114+0000","last_undegraded":"2024-03-06T07:26:27.700114+0000","last_fullsized":"2024-03-06T07:26:27.700114+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:24.719315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684768+0000","last_change":"2024-03-06T07:26:17.632248+0000","last_active":"2024-03-06T07:26:27.684768+0000","last_peered":"2024-03-06T07:26:27.684768+0000","last_clean":"2024-03-06T07:26:27.684768+0000","last_became_active":"2024-03-06T07:26:17.631912+0000","last_became_peered":"2024-03-06T07:26:17.631912+0000","last_unstale":"2024-03-06T07:26:27.684768+0000","last_undegraded":"2024-03-06T07:26:27.684768+0000","last_fullsized":"2024-03-06T07:26:27.684768+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:29.113308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683862+0000","last_change":"2024-03-06T07:26:16.616949+0000","last_active":"2024-03-06T07:26:27.683862+0000","last_peered":"2024-03-06T07:26:27.683862+0000","last_clean":"2024-03-06T07:26:27.683862+0000","last_became_active":"2024-03-06T07:26:16.616670+0000","last_became_peered":"2024-03-06T07:26:16.616670+0000","last_unstale":"2024-03-06T07:26:27.683862+0000","last_undegraded":"2024-03-06T07:26:27.683862+0000","last_fullsized":"2024-03-06T07:26:27.683862+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:59.587645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680156+0000","last_change":"2024-03-06T07:26:26.701033+0000","last_active":"2024-03-06T07:26:27.680156+0000","last_peered":"2024-03-06T07:26:27.680156+0000","last_clean":"2024-03-06T07:26:27.680156+0000","last_became_active":"2024-03-06T07:26:26.700715+0000","last_became_peered":"2024-03-06T07:26:26.700715+0000","last_unstale":"2024-03-06T07:26:27.680156+0000","last_undegraded":"2024-03-06T07:26:27.680156+0000","last_fullsized":"2024-03-06T07:26:27.680156+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:55.266644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.699864+0000","last_change":"2024-03-06T07:26:25.674406+0000","last_active":"2024-03-06T07:26:27.699864+0000","last_peered":"2024-03-06T07:26:27.699864+0000","last_clean":"2024-03-06T07:26:27.699864+0000","last_became_active":"2024-03-06T07:26:25.673939+0000","last_became_peered":"2024-03-06T07:26:25.673939+0000","last_unstale":"2024-03-06T07:26:27.699864+0000","last_undegraded":"2024-03-06T07:26:27.699864+0000","last_fullsized":"2024-03-06T07:26:27.699864+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:45.214049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854395+0000","last_change":"2024-03-06T07:26:17.628994+0000","last_active":"2024-03-06T07:26:27.854395+0000","last_peered":"2024-03-06T07:26:27.854395+0000","last_clean":"2024-03-06T07:26:27.854395+0000","last_became_active":"2024-03-06T07:26:17.628722+0000","last_became_peered":"2024-03-06T07:26:17.628722+0000","last_unstale":"2024-03-06T07:26:27.854395+0000","last_undegraded":"2024-03-06T07:26:27.854395+0000","last_fullsized":"2024-03-06T07:26:27.854395+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:43.090011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"148'2","reported_seq":24,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680809+0000","last_change":"2024-03-06T07:26:16.620011+0000","last_active":"2024-03-06T07:26:27.680809+0000","last_peered":"2024-03-06T07:26:27.680809+0000","last_clean":"2024-03-06T07:26:27.680809+0000","last_became_active":"2024-03-06T07:26:16.619736+0000","last_became_peered":"2024-03-06T07:26:16.619736+0000","last_unstale":"2024-03-06T07:26:27.680809+0000","last_undegraded":"2024-03-06T07:26:27.680809+0000","last_fullsized":"2024-03-06T07:26:27.680809+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:24.169321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853636+0000","last_change":"2024-03-06T07:26:26.694087+0000","last_active":"2024-03-06T07:26:27.853636+0000","last_peered":"2024-03-06T07:26:27.853636+0000","last_clean":"2024-03-06T07:26:27.853636+0000","last_became_active":"2024-03-06T07:26:26.693812+0000","last_became_peered":"2024-03-06T07:26:26.693812+0000","last_unstale":"2024-03-06T07:26:27.853636+0000","last_undegraded":"2024-03-06T07:26:27.853636+0000","last_fullsized":"2024-03-06T07:26:27.853636+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:28.200856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853595+0000","last_change":"2024-03-06T07:26:25.670698+0000","last_active":"2024-03-06T07:26:27.853595+0000","last_peered":"2024-03-06T07:26:27.853595+0000","last_clean":"2024-03-06T07:26:27.853595+0000","last_became_active":"2024-03-06T07:26:25.670446+0000","last_became_peered":"2024-03-06T07:26:25.670446+0000","last_unstale":"2024-03-06T07:26:27.853595+0000","last_undegraded":"2024-03-06T07:26:27.853595+0000","last_fullsized":"2024-03-06T07:26:27.853595+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:22.852243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702343+0000","last_change":"2024-03-06T07:26:17.628485+0000","last_active":"2024-03-06T07:26:27.702343+0000","last_peered":"2024-03-06T07:26:27.702343+0000","last_clean":"2024-03-06T07:26:27.702343+0000","last_became_active":"2024-03-06T07:26:17.628204+0000","last_became_peered":"2024-03-06T07:26:17.628204+0000","last_unstale":"2024-03-06T07:26:27.702343+0000","last_undegraded":"2024-03-06T07:26:27.702343+0000","last_fullsized":"2024-03-06T07:26:27.702343+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:56.134884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"149'1","reported_seq":23,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854453+0000","last_change":"2024-03-06T07:26:16.627066+0000","last_active":"2024-03-06T07:26:27.854453+0000","last_peered":"2024-03-06T07:26:27.854453+0000","last_clean":"2024-03-06T07:26:27.854453+0000","last_became_active":"2024-03-06T07:26:16.626418+0000","last_became_peered":"2024-03-06T07:26:16.626418+0000","last_unstale":"2024-03-06T07:26:27.854453+0000","last_undegraded":"2024-03-06T07:26:27.854453+0000","last_fullsized":"2024-03-06T07:26:27.854453+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:37.036435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702097+0000","last_change":"2024-03-06T07:26:26.692608+0000","last_active":"2024-03-06T07:26:27.702097+0000","last_peered":"2024-03-06T07:26:27.702097+0000","last_clean":"2024-03-06T07:26:27.702097+0000","last_became_active":"2024-03-06T07:26:26.692333+0000","last_became_peered":"2024-03-06T07:26:26.692333+0000","last_unstale":"2024-03-06T07:26:27.702097+0000","last_undegraded":"2024-03-06T07:26:27.702097+0000","last_fullsized":"2024-03-06T07:26:27.702097+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:58.389676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"152'2","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682915+0000","last_change":"2024-03-06T07:26:25.683377+0000","last_active":"2024-03-06T07:26:27.682915+0000","last_peered":"2024-03-06T07:26:27.682915+0000","last_clean":"2024-03-06T07:26:27.682915+0000","last_became_active":"2024-03-06T07:26:25.683159+0000","last_became_peered":"2024-03-06T07:26:25.683159+0000","last_unstale":"2024-03-06T07:26:27.682915+0000","last_undegraded":"2024-03-06T07:26:27.682915+0000","last_fullsized":"2024-03-06T07:26:27.682915+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:54.837891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700119+0000","last_change":"2024-03-06T07:26:17.626185+0000","last_active":"2024-03-06T07:26:27.700119+0000","last_peered":"2024-03-06T07:26:27.700119+0000","last_clean":"2024-03-06T07:26:27.700119+0000","last_became_active":"2024-03-06T07:26:17.625895+0000","last_became_peered":"2024-03-06T07:26:17.625895+0000","last_unstale":"2024-03-06T07:26:27.700119+0000","last_undegraded":"2024-03-06T07:26:27.700119+0000","last_fullsized":"2024-03-06T07:26:27.700119+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:16.007003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854236+0000","last_change":"2024-03-06T07:26:16.625471+0000","last_active":"2024-03-06T07:26:27.854236+0000","last_peered":"2024-03-06T07:26:27.854236+0000","last_clean":"2024-03-06T07:26:27.854236+0000","last_became_active":"2024-03-06T07:26:16.625198+0000","last_became_peered":"2024-03-06T07:26:16.625198+0000","last_unstale":"2024-03-06T07:26:27.854236+0000","last_undegraded":"2024-03-06T07:26:27.854236+0000","last_fullsized":"2024-03-06T07:26:27.854236+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:37.984120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.679891+0000","last_change":"2024-03-06T07:26:26.706286+0000","last_active":"2024-03-06T07:26:27.679891+0000","last_peered":"2024-03-06T07:26:27.679891+0000","last_clean":"2024-03-06T07:26:27.679891+0000","last_became_active":"2024-03-06T07:26:26.706024+0000","last_became_peered":"2024-03-06T07:26:26.706024+0000","last_unstale":"2024-03-06T07:26:27.679891+0000","last_undegraded":"2024-03-06T07:26:27.679891+0000","last_fullsized":"2024-03-06T07:26:27.679891+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:10.603912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.679921+0000","last_change":"2024-03-06T07:26:25.677833+0000","last_active":"2024-03-06T07:26:27.679921+0000","last_peered":"2024-03-06T07:26:27.679921+0000","last_clean":"2024-03-06T07:26:27.679921+0000","last_became_active":"2024-03-06T07:26:25.675930+0000","last_became_peered":"2024-03-06T07:26:25.675930+0000","last_unstale":"2024-03-06T07:26:27.679921+0000","last_undegraded":"2024-03-06T07:26:27.679921+0000","last_fullsized":"2024-03-06T07:26:27.679921+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:11.461549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702647+0000","last_change":"2024-03-06T07:26:17.630631+0000","last_active":"2024-03-06T07:26:27.702647+0000","last_peered":"2024-03-06T07:26:27.702647+0000","last_clean":"2024-03-06T07:26:27.702647+0000","last_became_active":"2024-03-06T07:26:17.630365+0000","last_became_peered":"2024-03-06T07:26:17.630365+0000","last_unstale":"2024-03-06T07:26:27.702647+0000","last_undegraded":"2024-03-06T07:26:27.702647+0000","last_fullsized":"2024-03-06T07:26:27.702647+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:27.398343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700353+0000","last_change":"2024-03-06T07:26:16.611334+0000","last_active":"2024-03-06T07:26:27.700353+0000","last_peered":"2024-03-06T07:26:27.700353+0000","last_clean":"2024-03-06T07:26:27.700353+0000","last_became_active":"2024-03-06T07:26:16.610646+0000","last_became_peered":"2024-03-06T07:26:16.610646+0000","last_unstale":"2024-03-06T07:26:27.700353+0000","last_undegraded":"2024-03-06T07:26:27.700353+0000","last_fullsized":"2024-03-06T07:26:27.700353+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:32.639393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854569+0000","last_change":"2024-03-06T07:26:26.701760+0000","last_active":"2024-03-06T07:26:27.854569+0000","last_peered":"2024-03-06T07:26:27.854569+0000","last_clean":"2024-03-06T07:26:27.854569+0000","last_became_active":"2024-03-06T07:26:26.701124+0000","last_became_peered":"2024-03-06T07:26:26.701124+0000","last_unstale":"2024-03-06T07:26:27.854569+0000","last_undegraded":"2024-03-06T07:26:27.854569+0000","last_fullsized":"2024-03-06T07:26:27.854569+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:54.014062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"152'1","reported_seq":15,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854021+0000","last_change":"2024-03-06T07:26:25.674222+0000","last_active":"2024-03-06T07:26:27.854021+0000","last_peered":"2024-03-06T07:26:27.854021+0000","last_clean":"2024-03-06T07:26:27.854021+0000","last_became_active":"2024-03-06T07:26:25.673935+0000","last_became_peered":"2024-03-06T07:26:25.673935+0000","last_unstale":"2024-03-06T07:26:27.854021+0000","last_undegraded":"2024-03-06T07:26:27.854021+0000","last_fullsized":"2024-03-06T07:26:27.854021+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:12.179836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683192+0000","last_change":"2024-03-06T07:26:17.632379+0000","last_active":"2024-03-06T07:26:27.683192+0000","last_peered":"2024-03-06T07:26:27.683192+0000","last_clean":"2024-03-06T07:26:27.683192+0000","last_became_active":"2024-03-06T07:26:17.631899+0000","last_became_peered":"2024-03-06T07:26:17.631899+0000","last_unstale":"2024-03-06T07:26:27.683192+0000","last_undegraded":"2024-03-06T07:26:27.683192+0000","last_fullsized":"2024-03-06T07:26:27.683192+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:00.591780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683250+0000","last_change":"2024-03-06T07:26:16.623898+0000","last_active":"2024-03-06T07:26:27.683250+0000","last_peered":"2024-03-06T07:26:27.683250+0000","last_clean":"2024-03-06T07:26:27.683250+0000","last_became_active":"2024-03-06T07:26:16.623602+0000","last_became_peered":"2024-03-06T07:26:16.623602+0000","last_unstale":"2024-03-06T07:26:27.683250+0000","last_undegraded":"2024-03-06T07:26:27.683250+0000","last_fullsized":"2024-03-06T07:26:27.683250+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:56.210974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703136+0000","last_change":"2024-03-06T07:26:26.692493+0000","last_active":"2024-03-06T07:26:27.703136+0000","last_peered":"2024-03-06T07:26:27.703136+0000","last_clean":"2024-03-06T07:26:27.703136+0000","last_became_active":"2024-03-06T07:26:26.692242+0000","last_became_peered":"2024-03-06T07:26:26.692242+0000","last_unstale":"2024-03-06T07:26:27.703136+0000","last_undegraded":"2024-03-06T07:26:27.703136+0000","last_fullsized":"2024-03-06T07:26:27.703136+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:36.454193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854525+0000","last_change":"2024-03-06T07:26:25.669943+0000","last_active":"2024-03-06T07:26:27.854525+0000","last_peered":"2024-03-06T07:26:27.854525+0000","last_clean":"2024-03-06T07:26:27.854525+0000","last_became_active":"2024-03-06T07:26:25.669439+0000","last_became_peered":"2024-03-06T07:26:25.669439+0000","last_unstale":"2024-03-06T07:26:27.854525+0000","last_undegraded":"2024-03-06T07:26:27.854525+0000","last_fullsized":"2024-03-06T07:26:27.854525+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:49.710750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684463+0000","last_change":"2024-03-06T07:26:17.632856+0000","last_active":"2024-03-06T07:26:27.684463+0000","last_peered":"2024-03-06T07:26:27.684463+0000","last_clean":"2024-03-06T07:26:27.684463+0000","last_became_active":"2024-03-06T07:26:17.632027+0000","last_became_peered":"2024-03-06T07:26:17.632027+0000","last_unstale":"2024-03-06T07:26:27.684463+0000","last_undegraded":"2024-03-06T07:26:27.684463+0000","last_fullsized":"2024-03-06T07:26:27.684463+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:38.988254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853594+0000","last_change":"2024-03-06T07:26:16.627698+0000","last_active":"2024-03-06T07:26:27.853594+0000","last_peered":"2024-03-06T07:26:27.853594+0000","last_clean":"2024-03-06T07:26:27.853594+0000","last_became_active":"2024-03-06T07:26:16.627318+0000","last_became_peered":"2024-03-06T07:26:16.627318+0000","last_unstale":"2024-03-06T07:26:27.853594+0000","last_undegraded":"2024-03-06T07:26:27.853594+0000","last_fullsized":"2024-03-06T07:26:27.853594+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:08.249739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700175+0000","last_change":"2024-03-06T07:26:26.689324+0000","last_active":"2024-03-06T07:26:27.700175+0000","last_peered":"2024-03-06T07:26:27.700175+0000","last_clean":"2024-03-06T07:26:27.700175+0000","last_became_active":"2024-03-06T07:26:26.689060+0000","last_became_peered":"2024-03-06T07:26:26.689060+0000","last_unstale":"2024-03-06T07:26:27.700175+0000","last_undegraded":"2024-03-06T07:26:27.700175+0000","last_fullsized":"2024-03-06T07:26:27.700175+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:41.961151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"152'1","reported_seq":15,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853892+0000","last_change":"2024-03-06T07:26:25.675206+0000","last_active":"2024-03-06T07:26:27.853892+0000","last_peered":"2024-03-06T07:26:27.853892+0000","last_clean":"2024-03-06T07:26:27.853892+0000","last_became_active":"2024-03-06T07:26:25.674907+0000","last_became_peered":"2024-03-06T07:26:25.674907+0000","last_unstale":"2024-03-06T07:26:27.853892+0000","last_undegraded":"2024-03-06T07:26:27.853892+0000","last_fullsized":"2024-03-06T07:26:27.853892+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:33.228276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854360+0000","last_change":"2024-03-06T07:26:17.637676+0000","last_active":"2024-03-06T07:26:27.854360+0000","last_peered":"2024-03-06T07:26:27.854360+0000","last_clean":"2024-03-06T07:26:27.854360+0000","last_became_active":"2024-03-06T07:26:17.637375+0000","last_became_peered":"2024-03-06T07:26:17.637375+0000","last_unstale":"2024-03-06T07:26:27.854360+0000","last_undegraded":"2024-03-06T07:26:27.854360+0000","last_fullsized":"2024-03-06T07:26:27.854360+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:18.556057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854616+0000","last_change":"2024-03-06T07:26:16.620202+0000","last_active":"2024-03-06T07:26:27.854616+0000","last_peered":"2024-03-06T07:26:27.854616+0000","last_clean":"2024-03-06T07:26:27.854616+0000","last_became_active":"2024-03-06T07:26:16.619897+0000","last_became_peered":"2024-03-06T07:26:16.619897+0000","last_unstale":"2024-03-06T07:26:27.854616+0000","last_undegraded":"2024-03-06T07:26:27.854616+0000","last_fullsized":"2024-03-06T07:26:27.854616+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:48.709582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854090+0000","last_change":"2024-03-06T07:26:26.704246+0000","last_active":"2024-03-06T07:26:27.854090+0000","last_peered":"2024-03-06T07:26:27.854090+0000","last_clean":"2024-03-06T07:26:27.854090+0000","last_became_active":"2024-03-06T07:26:26.703980+0000","last_became_peered":"2024-03-06T07:26:26.703980+0000","last_unstale":"2024-03-06T07:26:27.854090+0000","last_undegraded":"2024-03-06T07:26:27.854090+0000","last_fullsized":"2024-03-06T07:26:27.854090+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:22.369760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"152'2","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680501+0000","last_change":"2024-03-06T07:26:25.676226+0000","last_active":"2024-03-06T07:26:27.680501+0000","last_peered":"2024-03-06T07:26:27.680501+0000","last_clean":"2024-03-06T07:26:27.680501+0000","last_became_active":"2024-03-06T07:26:25.675930+0000","last_became_peered":"2024-03-06T07:26:25.675930+0000","last_unstale":"2024-03-06T07:26:27.680501+0000","last_undegraded":"2024-03-06T07:26:27.680501+0000","last_fullsized":"2024-03-06T07:26:27.680501+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:46.405456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700605+0000","last_change":"2024-03-06T07:26:17.625716+0000","last_active":"2024-03-06T07:26:27.700605+0000","last_peered":"2024-03-06T07:26:27.700605+0000","last_clean":"2024-03-06T07:26:27.700605+0000","last_became_active":"2024-03-06T07:26:17.625429+0000","last_became_peered":"2024-03-06T07:26:17.625429+0000","last_unstale":"2024-03-06T07:26:27.700605+0000","last_undegraded":"2024-03-06T07:26:27.700605+0000","last_fullsized":"2024-03-06T07:26:27.700605+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:38.596798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.679809+0000","last_change":"2024-03-06T07:26:16.620839+0000","last_active":"2024-03-06T07:26:27.679809+0000","last_peered":"2024-03-06T07:26:27.679809+0000","last_clean":"2024-03-06T07:26:27.679809+0000","last_became_active":"2024-03-06T07:26:16.620561+0000","last_became_peered":"2024-03-06T07:26:16.620561+0000","last_unstale":"2024-03-06T07:26:27.679809+0000","last_undegraded":"2024-03-06T07:26:27.679809+0000","last_fullsized":"2024-03-06T07:26:27.679809+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:26.394208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853778+0000","last_change":"2024-03-06T07:26:26.693843+0000","last_active":"2024-03-06T07:26:27.853778+0000","last_peered":"2024-03-06T07:26:27.853778+0000","last_clean":"2024-03-06T07:26:27.853778+0000","last_became_active":"2024-03-06T07:26:26.693548+0000","last_became_peered":"2024-03-06T07:26:26.693548+0000","last_unstale":"2024-03-06T07:26:27.853778+0000","last_undegraded":"2024-03-06T07:26:27.853778+0000","last_fullsized":"2024-03-06T07:26:27.853778+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:24.708532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.679801+0000","last_change":"2024-03-06T07:26:25.675948+0000","last_active":"2024-03-06T07:26:27.679801+0000","last_peered":"2024-03-06T07:26:27.679801+0000","last_clean":"2024-03-06T07:26:27.679801+0000","last_became_active":"2024-03-06T07:26:25.675645+0000","last_became_peered":"2024-03-06T07:26:25.675645+0000","last_unstale":"2024-03-06T07:26:27.679801+0000","last_undegraded":"2024-03-06T07:26:27.679801+0000","last_fullsized":"2024-03-06T07:26:27.679801+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:52.418568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684466+0000","last_change":"2024-03-06T07:26:17.632976+0000","last_active":"2024-03-06T07:26:27.684466+0000","last_peered":"2024-03-06T07:26:27.684466+0000","last_clean":"2024-03-06T07:26:27.684466+0000","last_became_active":"2024-03-06T07:26:17.632721+0000","last_became_peered":"2024-03-06T07:26:17.632721+0000","last_unstale":"2024-03-06T07:26:27.684466+0000","last_undegraded":"2024-03-06T07:26:27.684466+0000","last_fullsized":"2024-03-06T07:26:27.684466+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:56.134884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854445+0000","last_change":"2024-03-06T07:26:16.626430+0000","last_active":"2024-03-06T07:26:27.854445+0000","last_peered":"2024-03-06T07:26:27.854445+0000","last_clean":"2024-03-06T07:26:27.854445+0000","last_became_active":"2024-03-06T07:26:16.625971+0000","last_became_peered":"2024-03-06T07:26:16.625971+0000","last_unstale":"2024-03-06T07:26:27.854445+0000","last_undegraded":"2024-03-06T07:26:27.854445+0000","last_fullsized":"2024-03-06T07:26:27.854445+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:28.109173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":425,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683924+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:26:27.683924+0000","last_peered":"2024-03-06T07:26:27.683924+0000","last_clean":"2024-03-06T07:26:27.683924+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:26:27.683924+0000","last_undegraded":"2024-03-06T07:26:27.683924+0000","last_fullsized":"2024-03-06T07:26:27.683924+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574926,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33128,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338712,"statfs":{"total":96636764160,"available":96602841088,"internally_reserved":0,"allocated":1990656,"data_stored":1526495,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574925,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":1929216,"data_stored":936197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574925,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33088,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338752,"statfs":{"total":96636764160,"available":96602882048,"internally_reserved":0,"allocated":1949696,"data_stored":937885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574926,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1957888,"data_stored":938537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574925,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33092,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338748,"statfs":{"total":96636764160,"available":96602877952,"internally_reserved":0,"allocated":1953792,"data_stored":938503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574925,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":1929216,"data_stored":936709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":1937408,"data_stored":937739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33156,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338684,"statfs":{"total":96636764160,"available":96602812416,"internally_reserved":0,"allocated":2015232,"data_stored":1527327,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:32.004 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:32.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:34.372 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:34.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.473+0000 7f35bfcfa640 1 Processor -- start 2024-03-06T07:26:34.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.473+0000 7f35bfcfa640 1 -- start start 2024-03-06T07:26:34.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.474+0000 7f35bfcfa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35b80c6b10 0x7f35b8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.474+0000 7f35bfcfa640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b80c2fc0 con 0x7f35b80bf9d0 2024-03-06T07:26:34.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.474+0000 7f35bfcfa640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b8158da0 con 0x7f35b8152650 2024-03-06T07:26:34.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.474+0000 7f35bfcfa640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b8155800 con 0x7f35b80c6b10 2024-03-06T07:26:34.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.474+0000 7f35bd295640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35b80c6b10 0x7f35b8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.474+0000 7f35be297640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f35b80bf9d0 0x7f35b80c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37240/0 (socket says 172.21.15.1:37240) 2024-03-06T07:26:34.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.474+0000 7f35be297640 1 -- 172.21.15.1:0/3767680328 learned_addr learned my addr 172.21.15.1:0/3767680328 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:34.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.475+0000 7f35bca94640 1 -- 172.21.15.1:0/3767680328 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925874365 0 0) 0x7f35b80c2fc0 con 0x7f35b80bf9d0 2024-03-06T07:26:34.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.475+0000 7f35bca94640 1 -- 172.21.15.1:0/3767680328 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a4003520 con 0x7f35b80bf9d0 2024-03-06T07:26:34.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.475+0000 7f35bd295640 1 -- 172.21.15.1:0/3767680328 >> v1:172.21.15.98:6789/0 conn(0x7f35b8152650 legacy=0x7f35b8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:34.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.475+0000 7f35bd295640 1 -- 172.21.15.1:0/3767680328 >> v1:172.21.15.1:6789/0 conn(0x7f35b80bf9d0 legacy=0x7f35b80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.475+0000 7f35bd295640 1 -- 172.21.15.1:0/3767680328 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b80c6690 con 0x7f35b80c6b10 2024-03-06T07:26:34.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.475+0000 7f35bd295640 1 --2- 172.21.15.1:0/3767680328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b80c6b10 0x7f35b8155c90 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f35a80097a0 tx=0x7f35a802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c69412132f5f96 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:34.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.476+0000 7f35bca94640 1 -- 172.21.15.1:0/3767680328 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a80315a0 con 0x7f35b80c6b10 2024-03-06T07:26:34.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.476+0000 7f35bca94640 1 -- 172.21.15.1:0/3767680328 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a8037070 con 0x7f35b80c6b10 2024-03-06T07:26:34.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.476+0000 7f35bca94640 1 -- 172.21.15.1:0/3767680328 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a8031960 con 0x7f35b80c6b10 2024-03-06T07:26:34.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.476+0000 7f35bfcfa640 1 -- 172.21.15.1:0/3767680328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b80c6b10 msgr2=0x7f35b8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.476+0000 7f35bfcfa640 1 --2- 172.21.15.1:0/3767680328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b80c6b10 0x7f35b8155c90 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f35a80097a0 tx=0x7f35a802b840 comp rx=0 tx=0).stop 2024-03-06T07:26:34.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.476+0000 7f35bfcfa640 1 -- 172.21.15.1:0/3767680328 shutdown_connections 2024-03-06T07:26:34.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.476+0000 7f35bfcfa640 1 --2- 172.21.15.1:0/3767680328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b80c6b10 0x7f35b8155c90 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.476+0000 7f35bfcfa640 1 -- 172.21.15.1:0/3767680328 >> 172.21.15.1:0/3767680328 conn(0x7f35b8066ee0 msgr2=0x7f35b80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:34.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.476+0000 7f35bfcfa640 1 -- 172.21.15.1:0/3767680328 shutdown_connections 2024-03-06T07:26:34.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.476+0000 7f35bfcfa640 1 -- 172.21.15.1:0/3767680328 wait complete. 2024-03-06T07:26:34.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.477+0000 7f35bfcfa640 1 Processor -- start 2024-03-06T07:26:34.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.477+0000 7f35bfcfa640 1 -- start start 2024-03-06T07:26:34.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.478+0000 7f35bfcfa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b80bf9d0 0x7f35b814e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.478+0000 7f35bfcfa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b8152650 0x7f35b81eb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.478+0000 7f35bfcfa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b814b3c0 0x7f35b81ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.478+0000 7f35bfcfa640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b81551b0 con 0x7f35b814b3c0 2024-03-06T07:26:34.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.478+0000 7f35bfcfa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b80c6fa0 con 0x7f35b8152650 2024-03-06T07:26:34.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.478+0000 7f35bfcfa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b8155490 con 0x7f35b80bf9d0 2024-03-06T07:26:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.478+0000 7f35be297640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b814b3c0 0x7f35b81ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.478+0000 7f35be297640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b814b3c0 0x7f35b81ee5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38594/0 (socket says 172.21.15.1:38594) 2024-03-06T07:26:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.478+0000 7f35be297640 1 -- 172.21.15.1:0/1931688393 learned_addr learned my addr 172.21.15.1:0/1931688393 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.478+0000 7f35be297640 1 -- 172.21.15.1:0/1931688393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b80bf9d0 msgr2=0x7f35b814e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.478+0000 7f35be297640 1 --2- 172.21.15.1:0/1931688393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b80bf9d0 0x7f35b814e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.479+0000 7f35be297640 1 -- 172.21.15.1:0/1931688393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b8152650 msgr2=0x7f35b81eb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.479+0000 7f35be297640 1 --2- 172.21.15.1:0/1931688393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b8152650 0x7f35b81eb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.479+0000 7f35be297640 1 -- 172.21.15.1:0/1931688393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b80c9bc0 con 0x7f35b814b3c0 2024-03-06T07:26:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.479+0000 7f35be297640 1 --2- 172.21.15.1:0/1931688393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b814b3c0 0x7f35b81ee5f0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f35b000caa0 tx=0x7f35b000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.479+0000 7f35ae7fc640 1 -- 172.21.15.1:0/1931688393 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b0017070 con 0x7f35b814b3c0 2024-03-06T07:26:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.479+0000 7f35ae7fc640 1 -- 172.21.15.1:0/1931688393 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b000ac00 con 0x7f35b814b3c0 2024-03-06T07:26:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.479+0000 7f35ae7fc640 1 -- 172.21.15.1:0/1931688393 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b0012400 con 0x7f35b814b3c0 2024-03-06T07:26:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.479+0000 7f35bfcfa640 1 -- 172.21.15.1:0/1931688393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b80c9ea0 con 0x7f35b814b3c0 2024-03-06T07:26:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.479+0000 7f35bfcfa640 1 -- 172.21.15.1:0/1931688393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b8152f10 con 0x7f35b814b3c0 2024-03-06T07:26:34.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.480+0000 7f35bfcfa640 1 -- 172.21.15.1:0/1931688393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b8067fd0 con 0x7f35b814b3c0 2024-03-06T07:26:34.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.480+0000 7f35ae7fc640 1 -- 172.21.15.1:0/1931688393 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f35b001a070 con 0x7f35b814b3c0 2024-03-06T07:26:34.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.481+0000 7f35ae7fc640 1 --2- 172.21.15.1:0/1931688393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b4d0 0x7f359c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.481+0000 7f35ae7fc640 1 -- 172.21.15.1:0/1931688393 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f35b008a3e0 con 0x7f35b814b3c0 2024-03-06T07:26:34.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.481+0000 7f35bda96640 1 --2- 172.21.15.1:0/1931688393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b4d0 0x7f359c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.482+0000 7f35bda96640 1 --2- 172.21.15.1:0/1931688393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b4d0 0x7f359c06d990 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f35b4003cc0 tx=0x7f35b40079b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:34.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.485+0000 7f35ae7fc640 1 -- 172.21.15.1:0/1931688393 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f35b0058e10 con 0x7f35b814b3c0 2024-03-06T07:26:34.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.662+0000 7f35bfcfa640 1 -- 172.21.15.1:0/1931688393 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f35b80c9970 con 0x7f359c06b4d0 2024-03-06T07:26:34.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.674+0000 7f35ae7fc640 1 -- 172.21.15.1:0/1931688393 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+491533 (secure 0 0 0) 0x7f35b80c9970 con 0x7f359c06b4d0 2024-03-06T07:26:34.682 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:34.685 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:26:34.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.679+0000 7f35bfcfa640 1 -- 172.21.15.1:0/1931688393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b4d0 msgr2=0x7f359c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.679+0000 7f35bfcfa640 1 --2- 172.21.15.1:0/1931688393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b4d0 0x7f359c06d990 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f35b4003cc0 tx=0x7f35b40079b0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.679+0000 7f35bfcfa640 1 -- 172.21.15.1:0/1931688393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b814b3c0 msgr2=0x7f35b81ee5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.679+0000 7f35bfcfa640 1 --2- 172.21.15.1:0/1931688393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b814b3c0 0x7f35b81ee5f0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f35b000caa0 tx=0x7f35b000cf60 comp rx=0 tx=0).stop 2024-03-06T07:26:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.679+0000 7f35bfcfa640 1 -- 172.21.15.1:0/1931688393 shutdown_connections 2024-03-06T07:26:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.679+0000 7f35bfcfa640 1 --2- 172.21.15.1:0/1931688393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b4d0 0x7f359c06d990 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.679+0000 7f35bfcfa640 1 --2- 172.21.15.1:0/1931688393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b814b3c0 0x7f35b81ee5f0 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.679+0000 7f35bfcfa640 1 --2- 172.21.15.1:0/1931688393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b8152650 0x7f35b81eb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.679+0000 7f35bfcfa640 1 --2- 172.21.15.1:0/1931688393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b80bf9d0 0x7f35b814e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.679+0000 7f35bfcfa640 1 -- 172.21.15.1:0/1931688393 >> 172.21.15.1:0/1931688393 conn(0x7f35b8066ee0 msgr2=0x7f35b80c3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.679+0000 7f35bfcfa640 1 -- 172.21.15.1:0/1931688393 shutdown_connections 2024-03-06T07:26:34.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.679+0000 7f35bfcfa640 1 -- 172.21.15.1:0/1931688393 wait complete. 2024-03-06T07:26:34.693 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-03-06T07:26:32.776180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":373,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264776,"kb_used_data":15296,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754709944,"statfs":{"total":773094113280,"available":772822982656,"internally_reserved":0,"allocated":15663104,"data_stored":8679392,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.116955"},"pg_stats":[{"pgid":"26.3d","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854122+0000","last_change":"2024-03-06T07:26:26.694214+0000","last_active":"2024-03-06T07:26:27.854122+0000","last_peered":"2024-03-06T07:26:27.854122+0000","last_clean":"2024-03-06T07:26:27.854122+0000","last_became_active":"2024-03-06T07:26:26.693903+0000","last_became_peered":"2024-03-06T07:26:26.693903+0000","last_unstale":"2024-03-06T07:26:27.854122+0000","last_undegraded":"2024-03-06T07:26:27.854122+0000","last_fullsized":"2024-03-06T07:26:27.854122+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:49.088768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.3f","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683756+0000","last_change":"2024-03-06T07:26:17.632610+0000","last_active":"2024-03-06T07:26:27.683756+0000","last_peered":"2024-03-06T07:26:27.683756+0000","last_clean":"2024-03-06T07:26:27.683756+0000","last_became_active":"2024-03-06T07:26:17.630763+0000","last_became_peered":"2024-03-06T07:26:17.630763+0000","last_unstale":"2024-03-06T07:26:27.683756+0000","last_undegraded":"2024-03-06T07:26:27.683756+0000","last_fullsized":"2024-03-06T07:26:27.683756+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:35.035202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680368+0000","last_change":"2024-03-06T07:26:26.708189+0000","last_active":"2024-03-06T07:26:27.680368+0000","last_peered":"2024-03-06T07:26:27.680368+0000","last_clean":"2024-03-06T07:26:27.680368+0000","last_became_active":"2024-03-06T07:26:26.707895+0000","last_became_peered":"2024-03-06T07:26:26.707895+0000","last_unstale":"2024-03-06T07:26:27.680368+0000","last_undegraded":"2024-03-06T07:26:27.680368+0000","last_fullsized":"2024-03-06T07:26:27.680368+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:37.770304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684557+0000","last_change":"2024-03-06T07:26:17.632425+0000","last_active":"2024-03-06T07:26:27.684557+0000","last_peered":"2024-03-06T07:26:27.684557+0000","last_clean":"2024-03-06T07:26:27.684557+0000","last_became_active":"2024-03-06T07:26:17.632153+0000","last_became_peered":"2024-03-06T07:26:17.632153+0000","last_unstale":"2024-03-06T07:26:27.684557+0000","last_undegraded":"2024-03-06T07:26:27.684557+0000","last_fullsized":"2024-03-06T07:26:27.684557+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:01.829378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3f","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702806+0000","last_change":"2024-03-06T07:26:26.696360+0000","last_active":"2024-03-06T07:26:27.702806+0000","last_peered":"2024-03-06T07:26:27.702806+0000","last_clean":"2024-03-06T07:26:27.702806+0000","last_became_active":"2024-03-06T07:26:26.696053+0000","last_became_peered":"2024-03-06T07:26:26.696053+0000","last_unstale":"2024-03-06T07:26:27.702806+0000","last_undegraded":"2024-03-06T07:26:27.702806+0000","last_fullsized":"2024-03-06T07:26:27.702806+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:53.420127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683419+0000","last_change":"2024-03-06T07:26:17.630353+0000","last_active":"2024-03-06T07:26:27.683419+0000","last_peered":"2024-03-06T07:26:27.683419+0000","last_clean":"2024-03-06T07:26:27.683419+0000","last_became_active":"2024-03-06T07:26:17.630080+0000","last_became_peered":"2024-03-06T07:26:17.630080+0000","last_unstale":"2024-03-06T07:26:27.683419+0000","last_undegraded":"2024-03-06T07:26:27.683419+0000","last_fullsized":"2024-03-06T07:26:27.683419+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:49.713717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702711+0000","last_change":"2024-03-06T07:26:26.705829+0000","last_active":"2024-03-06T07:26:27.702711+0000","last_peered":"2024-03-06T07:26:27.702711+0000","last_clean":"2024-03-06T07:26:27.702711+0000","last_became_active":"2024-03-06T07:26:26.705541+0000","last_became_peered":"2024-03-06T07:26:26.705541+0000","last_unstale":"2024-03-06T07:26:27.702711+0000","last_undegraded":"2024-03-06T07:26:27.702711+0000","last_fullsized":"2024-03-06T07:26:27.702711+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:38.410943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683191+0000","last_change":"2024-03-06T07:26:17.630434+0000","last_active":"2024-03-06T07:26:27.683191+0000","last_peered":"2024-03-06T07:26:27.683191+0000","last_clean":"2024-03-06T07:26:27.683191+0000","last_became_active":"2024-03-06T07:26:17.630139+0000","last_became_peered":"2024-03-06T07:26:17.630139+0000","last_unstale":"2024-03-06T07:26:27.683191+0000","last_undegraded":"2024-03-06T07:26:27.683191+0000","last_fullsized":"2024-03-06T07:26:27.683191+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:09.159934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700303+0000","last_change":"2024-03-06T07:26:26.697421+0000","last_active":"2024-03-06T07:26:27.700303+0000","last_peered":"2024-03-06T07:26:27.700303+0000","last_clean":"2024-03-06T07:26:27.700303+0000","last_became_active":"2024-03-06T07:26:26.696894+0000","last_became_peered":"2024-03-06T07:26:26.696894+0000","last_unstale":"2024-03-06T07:26:27.700303+0000","last_undegraded":"2024-03-06T07:26:27.700303+0000","last_fullsized":"2024-03-06T07:26:27.700303+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:06.271489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680152+0000","last_change":"2024-03-06T07:26:17.632665+0000","last_active":"2024-03-06T07:26:27.680152+0000","last_peered":"2024-03-06T07:26:27.680152+0000","last_clean":"2024-03-06T07:26:27.680152+0000","last_became_active":"2024-03-06T07:26:17.632377+0000","last_became_peered":"2024-03-06T07:26:17.632377+0000","last_unstale":"2024-03-06T07:26:27.680152+0000","last_undegraded":"2024-03-06T07:26:27.680152+0000","last_fullsized":"2024-03-06T07:26:27.680152+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:23.225363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683788+0000","last_change":"2024-03-06T07:26:26.701290+0000","last_active":"2024-03-06T07:26:27.683788+0000","last_peered":"2024-03-06T07:26:27.683788+0000","last_clean":"2024-03-06T07:26:27.683788+0000","last_became_active":"2024-03-06T07:26:26.701010+0000","last_became_peered":"2024-03-06T07:26:26.701010+0000","last_unstale":"2024-03-06T07:26:27.683788+0000","last_undegraded":"2024-03-06T07:26:27.683788+0000","last_fullsized":"2024-03-06T07:26:27.683788+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:57.429223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684070+0000","last_change":"2024-03-06T07:26:17.633911+0000","last_active":"2024-03-06T07:26:27.684070+0000","last_peered":"2024-03-06T07:26:27.684070+0000","last_clean":"2024-03-06T07:26:27.684070+0000","last_became_active":"2024-03-06T07:26:17.633638+0000","last_became_peered":"2024-03-06T07:26:17.633638+0000","last_unstale":"2024-03-06T07:26:27.684070+0000","last_undegraded":"2024-03-06T07:26:27.684070+0000","last_fullsized":"2024-03-06T07:26:27.684070+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:01.350963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853311+0000","last_change":"2024-03-06T07:26:26.707191+0000","last_active":"2024-03-06T07:26:27.853311+0000","last_peered":"2024-03-06T07:26:27.853311+0000","last_clean":"2024-03-06T07:26:27.853311+0000","last_became_active":"2024-03-06T07:26:26.706927+0000","last_became_peered":"2024-03-06T07:26:26.706927+0000","last_unstale":"2024-03-06T07:26:27.853311+0000","last_undegraded":"2024-03-06T07:26:27.853311+0000","last_fullsized":"2024-03-06T07:26:27.853311+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:56.420351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702248+0000","last_change":"2024-03-06T07:26:17.629894+0000","last_active":"2024-03-06T07:26:27.702248+0000","last_peered":"2024-03-06T07:26:27.702248+0000","last_clean":"2024-03-06T07:26:27.702248+0000","last_became_active":"2024-03-06T07:26:17.629625+0000","last_became_peered":"2024-03-06T07:26:17.629625+0000","last_unstale":"2024-03-06T07:26:27.702248+0000","last_undegraded":"2024-03-06T07:26:27.702248+0000","last_fullsized":"2024-03-06T07:26:27.702248+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:46.687284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853129+0000","last_change":"2024-03-06T07:26:26.701868+0000","last_active":"2024-03-06T07:26:27.853129+0000","last_peered":"2024-03-06T07:26:27.853129+0000","last_clean":"2024-03-06T07:26:27.853129+0000","last_became_active":"2024-03-06T07:26:26.701540+0000","last_became_peered":"2024-03-06T07:26:26.701540+0000","last_unstale":"2024-03-06T07:26:27.853129+0000","last_undegraded":"2024-03-06T07:26:27.853129+0000","last_fullsized":"2024-03-06T07:26:27.853129+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:06.797123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702035+0000","last_change":"2024-03-06T07:26:17.632711+0000","last_active":"2024-03-06T07:26:27.702035+0000","last_peered":"2024-03-06T07:26:27.702035+0000","last_clean":"2024-03-06T07:26:27.702035+0000","last_became_active":"2024-03-06T07:26:17.632173+0000","last_became_peered":"2024-03-06T07:26:17.632173+0000","last_unstale":"2024-03-06T07:26:27.702035+0000","last_undegraded":"2024-03-06T07:26:27.702035+0000","last_fullsized":"2024-03-06T07:26:27.702035+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:25.595909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854605+0000","last_change":"2024-03-06T07:26:26.702364+0000","last_active":"2024-03-06T07:26:27.854605+0000","last_peered":"2024-03-06T07:26:27.854605+0000","last_clean":"2024-03-06T07:26:27.854605+0000","last_became_active":"2024-03-06T07:26:26.702102+0000","last_became_peered":"2024-03-06T07:26:26.702102+0000","last_unstale":"2024-03-06T07:26:27.854605+0000","last_undegraded":"2024-03-06T07:26:27.854605+0000","last_fullsized":"2024-03-06T07:26:27.854605+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:57.577361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703050+0000","last_change":"2024-03-06T07:26:17.638531+0000","last_active":"2024-03-06T07:26:27.703050+0000","last_peered":"2024-03-06T07:26:27.703050+0000","last_clean":"2024-03-06T07:26:27.703050+0000","last_became_active":"2024-03-06T07:26:17.638254+0000","last_became_peered":"2024-03-06T07:26:17.638254+0000","last_unstale":"2024-03-06T07:26:27.703050+0000","last_undegraded":"2024-03-06T07:26:27.703050+0000","last_fullsized":"2024-03-06T07:26:27.703050+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:01.350963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680372+0000","last_change":"2024-03-06T07:26:26.694408+0000","last_active":"2024-03-06T07:26:27.680372+0000","last_peered":"2024-03-06T07:26:27.680372+0000","last_clean":"2024-03-06T07:26:27.680372+0000","last_became_active":"2024-03-06T07:26:26.694144+0000","last_became_peered":"2024-03-06T07:26:26.694144+0000","last_unstale":"2024-03-06T07:26:27.680372+0000","last_undegraded":"2024-03-06T07:26:27.680372+0000","last_fullsized":"2024-03-06T07:26:27.680372+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:14.666630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703206+0000","last_change":"2024-03-06T07:26:17.632455+0000","last_active":"2024-03-06T07:26:27.703206+0000","last_peered":"2024-03-06T07:26:27.703206+0000","last_clean":"2024-03-06T07:26:27.703206+0000","last_became_active":"2024-03-06T07:26:17.632195+0000","last_became_peered":"2024-03-06T07:26:17.632195+0000","last_unstale":"2024-03-06T07:26:27.703206+0000","last_undegraded":"2024-03-06T07:26:27.703206+0000","last_fullsized":"2024-03-06T07:26:27.703206+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:14.404340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702916+0000","last_change":"2024-03-06T07:26:26.702603+0000","last_active":"2024-03-06T07:26:27.702916+0000","last_peered":"2024-03-06T07:26:27.702916+0000","last_clean":"2024-03-06T07:26:27.702916+0000","last_became_active":"2024-03-06T07:26:26.702329+0000","last_became_peered":"2024-03-06T07:26:26.702329+0000","last_unstale":"2024-03-06T07:26:27.702916+0000","last_undegraded":"2024-03-06T07:26:27.702916+0000","last_fullsized":"2024-03-06T07:26:27.702916+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:47.407016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854744+0000","last_change":"2024-03-06T07:26:17.633984+0000","last_active":"2024-03-06T07:26:27.854744+0000","last_peered":"2024-03-06T07:26:27.854744+0000","last_clean":"2024-03-06T07:26:27.854744+0000","last_became_active":"2024-03-06T07:26:17.633666+0000","last_became_peered":"2024-03-06T07:26:17.633666+0000","last_unstale":"2024-03-06T07:26:27.854744+0000","last_undegraded":"2024-03-06T07:26:27.854744+0000","last_fullsized":"2024-03-06T07:26:27.854744+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:43.884264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854240+0000","last_change":"2024-03-06T07:26:26.706601+0000","last_active":"2024-03-06T07:26:27.854240+0000","last_peered":"2024-03-06T07:26:27.854240+0000","last_clean":"2024-03-06T07:26:27.854240+0000","last_became_active":"2024-03-06T07:26:26.706312+0000","last_became_peered":"2024-03-06T07:26:26.706312+0000","last_unstale":"2024-03-06T07:26:27.854240+0000","last_undegraded":"2024-03-06T07:26:27.854240+0000","last_fullsized":"2024-03-06T07:26:27.854240+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:10.927406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684725+0000","last_change":"2024-03-06T07:26:17.634798+0000","last_active":"2024-03-06T07:26:27.684725+0000","last_peered":"2024-03-06T07:26:27.684725+0000","last_clean":"2024-03-06T07:26:27.684725+0000","last_became_active":"2024-03-06T07:26:17.634537+0000","last_became_peered":"2024-03-06T07:26:17.634537+0000","last_unstale":"2024-03-06T07:26:27.684725+0000","last_undegraded":"2024-03-06T07:26:27.684725+0000","last_fullsized":"2024-03-06T07:26:27.684725+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:25.595909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702423+0000","last_change":"2024-03-06T07:26:26.708042+0000","last_active":"2024-03-06T07:26:27.702423+0000","last_peered":"2024-03-06T07:26:27.702423+0000","last_clean":"2024-03-06T07:26:27.702423+0000","last_became_active":"2024-03-06T07:26:26.707778+0000","last_became_peered":"2024-03-06T07:26:26.707778+0000","last_unstale":"2024-03-06T07:26:27.702423+0000","last_undegraded":"2024-03-06T07:26:27.702423+0000","last_fullsized":"2024-03-06T07:26:27.702423+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:43.140802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853687+0000","last_change":"2024-03-06T07:26:17.640814+0000","last_active":"2024-03-06T07:26:27.853687+0000","last_peered":"2024-03-06T07:26:27.853687+0000","last_clean":"2024-03-06T07:26:27.853687+0000","last_became_active":"2024-03-06T07:26:17.640444+0000","last_became_peered":"2024-03-06T07:26:17.640444+0000","last_unstale":"2024-03-06T07:26:27.853687+0000","last_undegraded":"2024-03-06T07:26:27.853687+0000","last_fullsized":"2024-03-06T07:26:27.853687+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:35.920913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853140+0000","last_change":"2024-03-06T07:26:26.704032+0000","last_active":"2024-03-06T07:26:27.853140+0000","last_peered":"2024-03-06T07:26:27.853140+0000","last_clean":"2024-03-06T07:26:27.853140+0000","last_became_active":"2024-03-06T07:26:26.703752+0000","last_became_peered":"2024-03-06T07:26:26.703752+0000","last_unstale":"2024-03-06T07:26:27.853140+0000","last_undegraded":"2024-03-06T07:26:27.853140+0000","last_fullsized":"2024-03-06T07:26:27.853140+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:30.863087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680198+0000","last_change":"2024-03-06T07:26:17.631892+0000","last_active":"2024-03-06T07:26:27.680198+0000","last_peered":"2024-03-06T07:26:27.680198+0000","last_clean":"2024-03-06T07:26:27.680198+0000","last_became_active":"2024-03-06T07:26:17.631498+0000","last_became_peered":"2024-03-06T07:26:17.631498+0000","last_unstale":"2024-03-06T07:26:27.680198+0000","last_undegraded":"2024-03-06T07:26:27.680198+0000","last_fullsized":"2024-03-06T07:26:27.680198+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:09.253874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682875+0000","last_change":"2024-03-06T07:26:26.707066+0000","last_active":"2024-03-06T07:26:27.682875+0000","last_peered":"2024-03-06T07:26:27.682875+0000","last_clean":"2024-03-06T07:26:27.682875+0000","last_became_active":"2024-03-06T07:26:26.705596+0000","last_became_peered":"2024-03-06T07:26:26.705596+0000","last_unstale":"2024-03-06T07:26:27.682875+0000","last_undegraded":"2024-03-06T07:26:27.682875+0000","last_fullsized":"2024-03-06T07:26:27.682875+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:47.096950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853710+0000","last_change":"2024-03-06T07:26:17.634220+0000","last_active":"2024-03-06T07:26:27.853710+0000","last_peered":"2024-03-06T07:26:27.853710+0000","last_clean":"2024-03-06T07:26:27.853710+0000","last_became_active":"2024-03-06T07:26:17.633922+0000","last_became_peered":"2024-03-06T07:26:17.633922+0000","last_unstale":"2024-03-06T07:26:27.853710+0000","last_undegraded":"2024-03-06T07:26:27.853710+0000","last_fullsized":"2024-03-06T07:26:27.853710+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:51.900557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700487+0000","last_change":"2024-03-06T07:26:26.704753+0000","last_active":"2024-03-06T07:26:27.700487+0000","last_peered":"2024-03-06T07:26:27.700487+0000","last_clean":"2024-03-06T07:26:27.700487+0000","last_became_active":"2024-03-06T07:26:26.704488+0000","last_became_peered":"2024-03-06T07:26:26.704488+0000","last_unstale":"2024-03-06T07:26:27.700487+0000","last_undegraded":"2024-03-06T07:26:27.700487+0000","last_fullsized":"2024-03-06T07:26:27.700487+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:51.184434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682404+0000","last_change":"2024-03-06T07:26:17.635856+0000","last_active":"2024-03-06T07:26:27.682404+0000","last_peered":"2024-03-06T07:26:27.682404+0000","last_clean":"2024-03-06T07:26:27.682404+0000","last_became_active":"2024-03-06T07:26:17.631353+0000","last_became_peered":"2024-03-06T07:26:17.631353+0000","last_unstale":"2024-03-06T07:26:27.682404+0000","last_undegraded":"2024-03-06T07:26:27.682404+0000","last_fullsized":"2024-03-06T07:26:27.682404+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:15.652152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683086+0000","last_change":"2024-03-06T07:26:26.704283+0000","last_active":"2024-03-06T07:26:27.683086+0000","last_peered":"2024-03-06T07:26:27.683086+0000","last_clean":"2024-03-06T07:26:27.683086+0000","last_became_active":"2024-03-06T07:26:26.703817+0000","last_became_peered":"2024-03-06T07:26:26.703817+0000","last_unstale":"2024-03-06T07:26:27.683086+0000","last_undegraded":"2024-03-06T07:26:27.683086+0000","last_fullsized":"2024-03-06T07:26:27.683086+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:34.318310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703135+0000","last_change":"2024-03-06T07:26:17.628960+0000","last_active":"2024-03-06T07:26:27.703135+0000","last_peered":"2024-03-06T07:26:27.703135+0000","last_clean":"2024-03-06T07:26:27.703135+0000","last_became_active":"2024-03-06T07:26:17.628681+0000","last_became_peered":"2024-03-06T07:26:17.628681+0000","last_unstale":"2024-03-06T07:26:27.703135+0000","last_undegraded":"2024-03-06T07:26:27.703135+0000","last_fullsized":"2024-03-06T07:26:27.703135+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:40.950597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853602+0000","last_change":"2024-03-06T07:26:26.695129+0000","last_active":"2024-03-06T07:26:27.853602+0000","last_peered":"2024-03-06T07:26:27.853602+0000","last_clean":"2024-03-06T07:26:27.853602+0000","last_became_active":"2024-03-06T07:26:26.694943+0000","last_became_peered":"2024-03-06T07:26:26.694943+0000","last_unstale":"2024-03-06T07:26:27.853602+0000","last_undegraded":"2024-03-06T07:26:27.853602+0000","last_fullsized":"2024-03-06T07:26:27.853602+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:52.613907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854354+0000","last_change":"2024-03-06T07:26:17.629582+0000","last_active":"2024-03-06T07:26:27.854354+0000","last_peered":"2024-03-06T07:26:27.854354+0000","last_clean":"2024-03-06T07:26:27.854354+0000","last_became_active":"2024-03-06T07:26:17.629246+0000","last_became_peered":"2024-03-06T07:26:17.629246+0000","last_unstale":"2024-03-06T07:26:27.854354+0000","last_undegraded":"2024-03-06T07:26:27.854354+0000","last_fullsized":"2024-03-06T07:26:27.854354+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:56.556155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854336+0000","last_change":"2024-03-06T07:26:26.695709+0000","last_active":"2024-03-06T07:26:27.854336+0000","last_peered":"2024-03-06T07:26:27.854336+0000","last_clean":"2024-03-06T07:26:27.854336+0000","last_became_active":"2024-03-06T07:26:26.695471+0000","last_became_peered":"2024-03-06T07:26:26.695471+0000","last_unstale":"2024-03-06T07:26:27.854336+0000","last_undegraded":"2024-03-06T07:26:27.854336+0000","last_fullsized":"2024-03-06T07:26:27.854336+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:38.741123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854792+0000","last_change":"2024-03-06T07:26:17.632386+0000","last_active":"2024-03-06T07:26:27.854792+0000","last_peered":"2024-03-06T07:26:27.854792+0000","last_clean":"2024-03-06T07:26:27.854792+0000","last_became_active":"2024-03-06T07:26:17.631867+0000","last_became_peered":"2024-03-06T07:26:17.631867+0000","last_unstale":"2024-03-06T07:26:27.854792+0000","last_undegraded":"2024-03-06T07:26:27.854792+0000","last_fullsized":"2024-03-06T07:26:27.854792+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:46.687284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683476+0000","last_change":"2024-03-06T07:26:26.705909+0000","last_active":"2024-03-06T07:26:27.683476+0000","last_peered":"2024-03-06T07:26:27.683476+0000","last_clean":"2024-03-06T07:26:27.683476+0000","last_became_active":"2024-03-06T07:26:26.705632+0000","last_became_peered":"2024-03-06T07:26:26.705632+0000","last_unstale":"2024-03-06T07:26:27.683476+0000","last_undegraded":"2024-03-06T07:26:27.683476+0000","last_fullsized":"2024-03-06T07:26:27.683476+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:39.657027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700242+0000","last_change":"2024-03-06T07:26:17.622563+0000","last_active":"2024-03-06T07:26:27.700242+0000","last_peered":"2024-03-06T07:26:27.700242+0000","last_clean":"2024-03-06T07:26:27.700242+0000","last_became_active":"2024-03-06T07:26:17.622293+0000","last_became_peered":"2024-03-06T07:26:17.622293+0000","last_unstale":"2024-03-06T07:26:27.700242+0000","last_undegraded":"2024-03-06T07:26:27.700242+0000","last_fullsized":"2024-03-06T07:26:27.700242+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:34.462291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684627+0000","last_change":"2024-03-06T07:26:26.699335+0000","last_active":"2024-03-06T07:26:27.684627+0000","last_peered":"2024-03-06T07:26:27.684627+0000","last_clean":"2024-03-06T07:26:27.684627+0000","last_became_active":"2024-03-06T07:26:26.699041+0000","last_became_peered":"2024-03-06T07:26:26.699041+0000","last_unstale":"2024-03-06T07:26:27.684627+0000","last_undegraded":"2024-03-06T07:26:27.684627+0000","last_fullsized":"2024-03-06T07:26:27.684627+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:24.512341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853783+0000","last_change":"2024-03-06T07:26:17.637385+0000","last_active":"2024-03-06T07:26:27.853783+0000","last_peered":"2024-03-06T07:26:27.853783+0000","last_clean":"2024-03-06T07:26:27.853783+0000","last_became_active":"2024-03-06T07:26:17.637063+0000","last_became_peered":"2024-03-06T07:26:17.637063+0000","last_unstale":"2024-03-06T07:26:27.853783+0000","last_undegraded":"2024-03-06T07:26:27.853783+0000","last_fullsized":"2024-03-06T07:26:27.853783+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:49.333296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702261+0000","last_change":"2024-03-06T07:26:26.695440+0000","last_active":"2024-03-06T07:26:27.702261+0000","last_peered":"2024-03-06T07:26:27.702261+0000","last_clean":"2024-03-06T07:26:27.702261+0000","last_became_active":"2024-03-06T07:26:26.695168+0000","last_became_peered":"2024-03-06T07:26:26.695168+0000","last_unstale":"2024-03-06T07:26:27.702261+0000","last_undegraded":"2024-03-06T07:26:27.702261+0000","last_fullsized":"2024-03-06T07:26:27.702261+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:24.512341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684155+0000","last_change":"2024-03-06T07:26:17.632278+0000","last_active":"2024-03-06T07:26:27.684155+0000","last_peered":"2024-03-06T07:26:27.684155+0000","last_clean":"2024-03-06T07:26:27.684155+0000","last_became_active":"2024-03-06T07:26:17.631791+0000","last_became_peered":"2024-03-06T07:26:17.631791+0000","last_unstale":"2024-03-06T07:26:27.684155+0000","last_undegraded":"2024-03-06T07:26:27.684155+0000","last_fullsized":"2024-03-06T07:26:27.684155+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:40.950597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703000+0000","last_change":"2024-03-06T07:26:26.703449+0000","last_active":"2024-03-06T07:26:27.703000+0000","last_peered":"2024-03-06T07:26:27.703000+0000","last_clean":"2024-03-06T07:26:27.703000+0000","last_became_active":"2024-03-06T07:26:26.702643+0000","last_became_peered":"2024-03-06T07:26:26.702643+0000","last_unstale":"2024-03-06T07:26:27.703000+0000","last_undegraded":"2024-03-06T07:26:27.703000+0000","last_fullsized":"2024-03-06T07:26:27.703000+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:12.463109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.679900+0000","last_change":"2024-03-06T07:26:17.627902+0000","last_active":"2024-03-06T07:26:27.679900+0000","last_peered":"2024-03-06T07:26:27.679900+0000","last_clean":"2024-03-06T07:26:27.679900+0000","last_became_active":"2024-03-06T07:26:17.627643+0000","last_became_peered":"2024-03-06T07:26:17.627643+0000","last_unstale":"2024-03-06T07:26:27.679900+0000","last_undegraded":"2024-03-06T07:26:27.679900+0000","last_fullsized":"2024-03-06T07:26:27.679900+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:30.600647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702957+0000","last_change":"2024-03-06T07:26:26.703994+0000","last_active":"2024-03-06T07:26:27.702957+0000","last_peered":"2024-03-06T07:26:27.702957+0000","last_clean":"2024-03-06T07:26:27.702957+0000","last_became_active":"2024-03-06T07:26:26.703695+0000","last_became_peered":"2024-03-06T07:26:26.703695+0000","last_unstale":"2024-03-06T07:26:27.702957+0000","last_undegraded":"2024-03-06T07:26:27.702957+0000","last_fullsized":"2024-03-06T07:26:27.702957+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:57.436722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682452+0000","last_change":"2024-03-06T07:26:17.631421+0000","last_active":"2024-03-06T07:26:27.682452+0000","last_peered":"2024-03-06T07:26:27.682452+0000","last_clean":"2024-03-06T07:26:27.682452+0000","last_became_active":"2024-03-06T07:26:17.631022+0000","last_became_peered":"2024-03-06T07:26:17.631022+0000","last_unstale":"2024-03-06T07:26:27.682452+0000","last_undegraded":"2024-03-06T07:26:27.682452+0000","last_fullsized":"2024-03-06T07:26:27.682452+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:40.032388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684400+0000","last_change":"2024-03-06T07:26:26.700754+0000","last_active":"2024-03-06T07:26:27.684400+0000","last_peered":"2024-03-06T07:26:27.684400+0000","last_clean":"2024-03-06T07:26:27.684400+0000","last_became_active":"2024-03-06T07:26:26.700432+0000","last_became_peered":"2024-03-06T07:26:26.700432+0000","last_unstale":"2024-03-06T07:26:27.684400+0000","last_undegraded":"2024-03-06T07:26:27.684400+0000","last_fullsized":"2024-03-06T07:26:27.684400+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:27.612436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854481+0000","last_change":"2024-03-06T07:26:17.637044+0000","last_active":"2024-03-06T07:26:27.854481+0000","last_peered":"2024-03-06T07:26:27.854481+0000","last_clean":"2024-03-06T07:26:27.854481+0000","last_became_active":"2024-03-06T07:26:17.636771+0000","last_became_peered":"2024-03-06T07:26:17.636771+0000","last_unstale":"2024-03-06T07:26:27.854481+0000","last_undegraded":"2024-03-06T07:26:27.854481+0000","last_fullsized":"2024-03-06T07:26:27.854481+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:29.354563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853327+0000","last_change":"2024-03-06T07:26:26.706900+0000","last_active":"2024-03-06T07:26:27.853327+0000","last_peered":"2024-03-06T07:26:27.853327+0000","last_clean":"2024-03-06T07:26:27.853327+0000","last_became_active":"2024-03-06T07:26:26.706660+0000","last_became_peered":"2024-03-06T07:26:26.706660+0000","last_unstale":"2024-03-06T07:26:27.853327+0000","last_undegraded":"2024-03-06T07:26:27.853327+0000","last_fullsized":"2024-03-06T07:26:27.853327+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:34.561014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700173+0000","last_change":"2024-03-06T07:26:17.636369+0000","last_active":"2024-03-06T07:26:27.700173+0000","last_peered":"2024-03-06T07:26:27.700173+0000","last_clean":"2024-03-06T07:26:27.700173+0000","last_became_active":"2024-03-06T07:26:17.636090+0000","last_became_peered":"2024-03-06T07:26:17.636090+0000","last_unstale":"2024-03-06T07:26:27.700173+0000","last_undegraded":"2024-03-06T07:26:27.700173+0000","last_fullsized":"2024-03-06T07:26:27.700173+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:45.110614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700456+0000","last_change":"2024-03-06T07:26:26.690832+0000","last_active":"2024-03-06T07:26:27.700456+0000","last_peered":"2024-03-06T07:26:27.700456+0000","last_clean":"2024-03-06T07:26:27.700456+0000","last_became_active":"2024-03-06T07:26:26.690572+0000","last_became_peered":"2024-03-06T07:26:26.690572+0000","last_unstale":"2024-03-06T07:26:27.700456+0000","last_undegraded":"2024-03-06T07:26:27.700456+0000","last_fullsized":"2024-03-06T07:26:27.700456+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:10.471929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684323+0000","last_change":"2024-03-06T07:26:17.633006+0000","last_active":"2024-03-06T07:26:27.684323+0000","last_peered":"2024-03-06T07:26:27.684323+0000","last_clean":"2024-03-06T07:26:27.684323+0000","last_became_active":"2024-03-06T07:26:17.632523+0000","last_became_peered":"2024-03-06T07:26:17.632523+0000","last_unstale":"2024-03-06T07:26:27.684323+0000","last_undegraded":"2024-03-06T07:26:27.684323+0000","last_fullsized":"2024-03-06T07:26:27.684323+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:41.655929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853881+0000","last_change":"2024-03-06T07:26:26.701984+0000","last_active":"2024-03-06T07:26:27.853881+0000","last_peered":"2024-03-06T07:26:27.853881+0000","last_clean":"2024-03-06T07:26:27.853881+0000","last_became_active":"2024-03-06T07:26:26.701386+0000","last_became_peered":"2024-03-06T07:26:26.701386+0000","last_unstale":"2024-03-06T07:26:27.853881+0000","last_undegraded":"2024-03-06T07:26:27.853881+0000","last_fullsized":"2024-03-06T07:26:27.853881+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:39.657027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853431+0000","last_change":"2024-03-06T07:26:17.638448+0000","last_active":"2024-03-06T07:26:27.853431+0000","last_peered":"2024-03-06T07:26:27.853431+0000","last_clean":"2024-03-06T07:26:27.853431+0000","last_became_active":"2024-03-06T07:26:17.638188+0000","last_became_peered":"2024-03-06T07:26:17.638188+0000","last_unstale":"2024-03-06T07:26:27.853431+0000","last_undegraded":"2024-03-06T07:26:27.853431+0000","last_fullsized":"2024-03-06T07:26:27.853431+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:15.455961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700096+0000","last_change":"2024-03-06T07:26:26.688526+0000","last_active":"2024-03-06T07:26:27.700096+0000","last_peered":"2024-03-06T07:26:27.700096+0000","last_clean":"2024-03-06T07:26:27.700096+0000","last_became_active":"2024-03-06T07:26:26.688270+0000","last_became_peered":"2024-03-06T07:26:26.688270+0000","last_unstale":"2024-03-06T07:26:27.700096+0000","last_undegraded":"2024-03-06T07:26:27.700096+0000","last_fullsized":"2024-03-06T07:26:27.700096+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:20.442923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684388+0000","last_change":"2024-03-06T07:26:17.634053+0000","last_active":"2024-03-06T07:26:27.684388+0000","last_peered":"2024-03-06T07:26:27.684388+0000","last_clean":"2024-03-06T07:26:27.684388+0000","last_became_active":"2024-03-06T07:26:17.633451+0000","last_became_peered":"2024-03-06T07:26:17.633451+0000","last_unstale":"2024-03-06T07:26:27.684388+0000","last_undegraded":"2024-03-06T07:26:27.684388+0000","last_fullsized":"2024-03-06T07:26:27.684388+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:25.173456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853860+0000","last_change":"2024-03-06T07:26:26.704201+0000","last_active":"2024-03-06T07:26:27.853860+0000","last_peered":"2024-03-06T07:26:27.853860+0000","last_clean":"2024-03-06T07:26:27.853860+0000","last_became_active":"2024-03-06T07:26:26.703937+0000","last_became_peered":"2024-03-06T07:26:26.703937+0000","last_unstale":"2024-03-06T07:26:27.853860+0000","last_undegraded":"2024-03-06T07:26:27.853860+0000","last_fullsized":"2024-03-06T07:26:27.853860+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:00.129449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853564+0000","last_change":"2024-03-06T07:26:17.627891+0000","last_active":"2024-03-06T07:26:27.853564+0000","last_peered":"2024-03-06T07:26:27.853564+0000","last_clean":"2024-03-06T07:26:27.853564+0000","last_became_active":"2024-03-06T07:26:17.627624+0000","last_became_peered":"2024-03-06T07:26:17.627624+0000","last_unstale":"2024-03-06T07:26:27.853564+0000","last_undegraded":"2024-03-06T07:26:27.853564+0000","last_fullsized":"2024-03-06T07:26:27.853564+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:56.476412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854339+0000","last_change":"2024-03-06T07:26:26.695392+0000","last_active":"2024-03-06T07:26:27.854339+0000","last_peered":"2024-03-06T07:26:27.854339+0000","last_clean":"2024-03-06T07:26:27.854339+0000","last_became_active":"2024-03-06T07:26:26.695115+0000","last_became_peered":"2024-03-06T07:26:26.695115+0000","last_unstale":"2024-03-06T07:26:27.854339+0000","last_undegraded":"2024-03-06T07:26:27.854339+0000","last_fullsized":"2024-03-06T07:26:27.854339+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:32.281743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853828+0000","last_change":"2024-03-06T07:26:17.635467+0000","last_active":"2024-03-06T07:26:27.853828+0000","last_peered":"2024-03-06T07:26:27.853828+0000","last_clean":"2024-03-06T07:26:27.853828+0000","last_became_active":"2024-03-06T07:26:17.635125+0000","last_became_peered":"2024-03-06T07:26:17.635125+0000","last_unstale":"2024-03-06T07:26:27.853828+0000","last_undegraded":"2024-03-06T07:26:27.853828+0000","last_fullsized":"2024-03-06T07:26:27.853828+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:39.829929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854316+0000","last_change":"2024-03-06T07:26:26.702761+0000","last_active":"2024-03-06T07:26:27.854316+0000","last_peered":"2024-03-06T07:26:27.854316+0000","last_clean":"2024-03-06T07:26:27.854316+0000","last_became_active":"2024-03-06T07:26:26.700555+0000","last_became_peered":"2024-03-06T07:26:26.700555+0000","last_unstale":"2024-03-06T07:26:27.854316+0000","last_undegraded":"2024-03-06T07:26:27.854316+0000","last_fullsized":"2024-03-06T07:26:27.854316+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:29.718934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853712+0000","last_change":"2024-03-06T07:26:17.638944+0000","last_active":"2024-03-06T07:26:27.853712+0000","last_peered":"2024-03-06T07:26:27.853712+0000","last_clean":"2024-03-06T07:26:27.853712+0000","last_became_active":"2024-03-06T07:26:17.638649+0000","last_became_peered":"2024-03-06T07:26:17.638649+0000","last_unstale":"2024-03-06T07:26:27.853712+0000","last_undegraded":"2024-03-06T07:26:27.853712+0000","last_fullsized":"2024-03-06T07:26:27.853712+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:17.394090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682834+0000","last_change":"2024-03-06T07:26:26.703690+0000","last_active":"2024-03-06T07:26:27.682834+0000","last_peered":"2024-03-06T07:26:27.682834+0000","last_clean":"2024-03-06T07:26:27.682834+0000","last_became_active":"2024-03-06T07:26:26.703143+0000","last_became_peered":"2024-03-06T07:26:26.703143+0000","last_unstale":"2024-03-06T07:26:27.682834+0000","last_undegraded":"2024-03-06T07:26:27.682834+0000","last_fullsized":"2024-03-06T07:26:27.682834+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:38.169688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.679751+0000","last_change":"2024-03-06T07:26:25.675542+0000","last_active":"2024-03-06T07:26:27.679751+0000","last_peered":"2024-03-06T07:26:27.679751+0000","last_clean":"2024-03-06T07:26:27.679751+0000","last_became_active":"2024-03-06T07:26:25.675297+0000","last_became_peered":"2024-03-06T07:26:25.675297+0000","last_unstale":"2024-03-06T07:26:27.679751+0000","last_undegraded":"2024-03-06T07:26:27.679751+0000","last_fullsized":"2024-03-06T07:26:27.679751+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:37.409384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683746+0000","last_change":"2024-03-06T07:26:17.635968+0000","last_active":"2024-03-06T07:26:27.683746+0000","last_peered":"2024-03-06T07:26:27.683746+0000","last_clean":"2024-03-06T07:26:27.683746+0000","last_became_active":"2024-03-06T07:26:17.635095+0000","last_became_peered":"2024-03-06T07:26:17.635095+0000","last_unstale":"2024-03-06T07:26:27.683746+0000","last_undegraded":"2024-03-06T07:26:27.683746+0000","last_fullsized":"2024-03-06T07:26:27.683746+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:26.879181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"148'4","reported_seq":26,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683934+0000","last_change":"2024-03-06T07:26:16.626240+0000","last_active":"2024-03-06T07:26:27.683934+0000","last_peered":"2024-03-06T07:26:27.683934+0000","last_clean":"2024-03-06T07:26:27.683934+0000","last_became_active":"2024-03-06T07:26:16.626011+0000","last_became_peered":"2024-03-06T07:26:16.626011+0000","last_unstale":"2024-03-06T07:26:27.683934+0000","last_undegraded":"2024-03-06T07:26:27.683934+0000","last_fullsized":"2024-03-06T07:26:27.683934+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:30.259380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683188+0000","last_change":"2024-03-06T07:26:26.703425+0000","last_active":"2024-03-06T07:26:27.683188+0000","last_peered":"2024-03-06T07:26:27.683188+0000","last_clean":"2024-03-06T07:26:27.683188+0000","last_became_active":"2024-03-06T07:26:26.703116+0000","last_became_peered":"2024-03-06T07:26:26.703116+0000","last_unstale":"2024-03-06T07:26:27.683188+0000","last_undegraded":"2024-03-06T07:26:27.683188+0000","last_fullsized":"2024-03-06T07:26:27.683188+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:11.283568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"152'8","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854098+0000","last_change":"2024-03-06T07:26:25.674717+0000","last_active":"2024-03-06T07:26:27.854098+0000","last_peered":"2024-03-06T07:26:27.854098+0000","last_clean":"2024-03-06T07:26:27.854098+0000","last_became_active":"2024-03-06T07:26:25.674451+0000","last_became_peered":"2024-03-06T07:26:25.674451+0000","last_unstale":"2024-03-06T07:26:27.854098+0000","last_undegraded":"2024-03-06T07:26:27.854098+0000","last_fullsized":"2024-03-06T07:26:27.854098+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:33.650729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853174+0000","last_change":"2024-03-06T07:26:17.627512+0000","last_active":"2024-03-06T07:26:27.853174+0000","last_peered":"2024-03-06T07:26:27.853174+0000","last_clean":"2024-03-06T07:26:27.853174+0000","last_became_active":"2024-03-06T07:26:17.627235+0000","last_became_peered":"2024-03-06T07:26:17.627235+0000","last_unstale":"2024-03-06T07:26:27.853174+0000","last_undegraded":"2024-03-06T07:26:27.853174+0000","last_fullsized":"2024-03-06T07:26:27.853174+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:19.245848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"148'1","reported_seq":23,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680614+0000","last_change":"2024-03-06T07:26:16.625703+0000","last_active":"2024-03-06T07:26:27.680614+0000","last_peered":"2024-03-06T07:26:27.680614+0000","last_clean":"2024-03-06T07:26:27.680614+0000","last_became_active":"2024-03-06T07:26:16.625426+0000","last_became_peered":"2024-03-06T07:26:16.625426+0000","last_unstale":"2024-03-06T07:26:27.680614+0000","last_undegraded":"2024-03-06T07:26:27.680614+0000","last_fullsized":"2024-03-06T07:26:27.680614+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:39.946462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.699883+0000","last_change":"2024-03-06T07:26:26.695824+0000","last_active":"2024-03-06T07:26:27.699883+0000","last_peered":"2024-03-06T07:26:27.699883+0000","last_clean":"2024-03-06T07:26:27.699883+0000","last_became_active":"2024-03-06T07:26:26.695570+0000","last_became_peered":"2024-03-06T07:26:26.695570+0000","last_unstale":"2024-03-06T07:26:27.699883+0000","last_undegraded":"2024-03-06T07:26:27.699883+0000","last_fullsized":"2024-03-06T07:26:27.699883+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:05.612535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"152'2","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682794+0000","last_change":"2024-03-06T07:26:25.680772+0000","last_active":"2024-03-06T07:26:27.682794+0000","last_peered":"2024-03-06T07:26:27.682794+0000","last_clean":"2024-03-06T07:26:27.682794+0000","last_became_active":"2024-03-06T07:26:25.680514+0000","last_became_peered":"2024-03-06T07:26:25.680514+0000","last_unstale":"2024-03-06T07:26:27.682794+0000","last_undegraded":"2024-03-06T07:26:27.682794+0000","last_fullsized":"2024-03-06T07:26:27.682794+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:54.742105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680221+0000","last_change":"2024-03-06T07:26:17.632107+0000","last_active":"2024-03-06T07:26:27.680221+0000","last_peered":"2024-03-06T07:26:27.680221+0000","last_clean":"2024-03-06T07:26:27.680221+0000","last_became_active":"2024-03-06T07:26:17.631481+0000","last_became_peered":"2024-03-06T07:26:17.631481+0000","last_unstale":"2024-03-06T07:26:27.680221+0000","last_undegraded":"2024-03-06T07:26:27.680221+0000","last_fullsized":"2024-03-06T07:26:27.680221+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:20.662554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"148'2","reported_seq":24,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854829+0000","last_change":"2024-03-06T07:26:16.620199+0000","last_active":"2024-03-06T07:26:27.854829+0000","last_peered":"2024-03-06T07:26:27.854829+0000","last_clean":"2024-03-06T07:26:27.854829+0000","last_became_active":"2024-03-06T07:26:16.619877+0000","last_became_peered":"2024-03-06T07:26:16.619877+0000","last_unstale":"2024-03-06T07:26:27.854829+0000","last_undegraded":"2024-03-06T07:26:27.854829+0000","last_fullsized":"2024-03-06T07:26:27.854829+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:14.648017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853115+0000","last_change":"2024-03-06T07:26:26.697268+0000","last_active":"2024-03-06T07:26:27.853115+0000","last_peered":"2024-03-06T07:26:27.853115+0000","last_clean":"2024-03-06T07:26:27.853115+0000","last_became_active":"2024-03-06T07:26:26.697005+0000","last_became_peered":"2024-03-06T07:26:26.697005+0000","last_unstale":"2024-03-06T07:26:27.853115+0000","last_undegraded":"2024-03-06T07:26:27.853115+0000","last_fullsized":"2024-03-06T07:26:27.853115+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:33.213505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703297+0000","last_change":"2024-03-06T07:26:25.678879+0000","last_active":"2024-03-06T07:26:27.703297+0000","last_peered":"2024-03-06T07:26:27.703297+0000","last_clean":"2024-03-06T07:26:27.703297+0000","last_became_active":"2024-03-06T07:26:25.678532+0000","last_became_peered":"2024-03-06T07:26:25.678532+0000","last_unstale":"2024-03-06T07:26:27.703297+0000","last_undegraded":"2024-03-06T07:26:27.703297+0000","last_fullsized":"2024-03-06T07:26:27.703297+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:33.316750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684628+0000","last_change":"2024-03-06T07:26:17.634295+0000","last_active":"2024-03-06T07:26:27.684628+0000","last_peered":"2024-03-06T07:26:27.684628+0000","last_clean":"2024-03-06T07:26:27.684628+0000","last_became_active":"2024-03-06T07:26:17.633744+0000","last_became_peered":"2024-03-06T07:26:17.633744+0000","last_unstale":"2024-03-06T07:26:27.684628+0000","last_undegraded":"2024-03-06T07:26:27.684628+0000","last_fullsized":"2024-03-06T07:26:27.684628+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:14.404340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"148'2","reported_seq":24,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700248+0000","last_change":"2024-03-06T07:26:16.616715+0000","last_active":"2024-03-06T07:26:27.700248+0000","last_peered":"2024-03-06T07:26:27.700248+0000","last_clean":"2024-03-06T07:26:27.700248+0000","last_became_active":"2024-03-06T07:26:16.616466+0000","last_became_peered":"2024-03-06T07:26:16.616466+0000","last_unstale":"2024-03-06T07:26:27.700248+0000","last_undegraded":"2024-03-06T07:26:27.700248+0000","last_fullsized":"2024-03-06T07:26:27.700248+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:54.279225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683734+0000","last_change":"2024-03-06T07:26:26.705862+0000","last_active":"2024-03-06T07:26:27.683734+0000","last_peered":"2024-03-06T07:26:27.683734+0000","last_clean":"2024-03-06T07:26:27.683734+0000","last_became_active":"2024-03-06T07:26:26.705261+0000","last_became_peered":"2024-03-06T07:26:26.705261+0000","last_unstale":"2024-03-06T07:26:27.683734+0000","last_undegraded":"2024-03-06T07:26:27.683734+0000","last_fullsized":"2024-03-06T07:26:27.683734+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:18.310254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"152'3","reported_seq":17,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680332+0000","last_change":"2024-03-06T07:26:25.678093+0000","last_active":"2024-03-06T07:26:27.680332+0000","last_peered":"2024-03-06T07:26:27.680332+0000","last_clean":"2024-03-06T07:26:27.680332+0000","last_became_active":"2024-03-06T07:26:25.677685+0000","last_became_peered":"2024-03-06T07:26:25.677685+0000","last_unstale":"2024-03-06T07:26:27.680332+0000","last_undegraded":"2024-03-06T07:26:27.680332+0000","last_fullsized":"2024-03-06T07:26:27.680332+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:47.861522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702555+0000","last_change":"2024-03-06T07:26:17.631966+0000","last_active":"2024-03-06T07:26:27.702555+0000","last_peered":"2024-03-06T07:26:27.702555+0000","last_clean":"2024-03-06T07:26:27.702555+0000","last_became_active":"2024-03-06T07:26:17.631673+0000","last_became_peered":"2024-03-06T07:26:17.631673+0000","last_unstale":"2024-03-06T07:26:27.702555+0000","last_undegraded":"2024-03-06T07:26:27.702555+0000","last_fullsized":"2024-03-06T07:26:27.702555+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:26.879181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"148'1","reported_seq":23,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702812+0000","last_change":"2024-03-06T07:26:16.619602+0000","last_active":"2024-03-06T07:26:27.702812+0000","last_peered":"2024-03-06T07:26:27.702812+0000","last_clean":"2024-03-06T07:26:27.702812+0000","last_became_active":"2024-03-06T07:26:16.619282+0000","last_became_peered":"2024-03-06T07:26:16.619282+0000","last_unstale":"2024-03-06T07:26:27.702812+0000","last_undegraded":"2024-03-06T07:26:27.702812+0000","last_fullsized":"2024-03-06T07:26:27.702812+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:59.587645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702976+0000","last_change":"2024-03-06T07:26:26.703177+0000","last_active":"2024-03-06T07:26:27.702976+0000","last_peered":"2024-03-06T07:26:27.702976+0000","last_clean":"2024-03-06T07:26:27.702976+0000","last_became_active":"2024-03-06T07:26:26.702905+0000","last_became_peered":"2024-03-06T07:26:26.702905+0000","last_unstale":"2024-03-06T07:26:27.702976+0000","last_undegraded":"2024-03-06T07:26:27.702976+0000","last_fullsized":"2024-03-06T07:26:27.702976+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:51.224651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"152'6","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682827+0000","last_change":"2024-03-06T07:26:25.678836+0000","last_active":"2024-03-06T07:26:27.682827+0000","last_peered":"2024-03-06T07:26:27.682827+0000","last_clean":"2024-03-06T07:26:27.682827+0000","last_became_active":"2024-03-06T07:26:25.678543+0000","last_became_peered":"2024-03-06T07:26:25.678543+0000","last_unstale":"2024-03-06T07:26:27.682827+0000","last_undegraded":"2024-03-06T07:26:27.682827+0000","last_fullsized":"2024-03-06T07:26:27.682827+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:47.884750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700506+0000","last_change":"2024-03-06T07:26:17.628669+0000","last_active":"2024-03-06T07:26:27.700506+0000","last_peered":"2024-03-06T07:26:27.700506+0000","last_clean":"2024-03-06T07:26:27.700506+0000","last_became_active":"2024-03-06T07:26:17.628371+0000","last_became_peered":"2024-03-06T07:26:17.628371+0000","last_unstale":"2024-03-06T07:26:27.700506+0000","last_undegraded":"2024-03-06T07:26:27.700506+0000","last_fullsized":"2024-03-06T07:26:27.700506+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:16.499694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"148'1","reported_seq":23,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680358+0000","last_change":"2024-03-06T07:26:16.615912+0000","last_active":"2024-03-06T07:26:27.680358+0000","last_peered":"2024-03-06T07:26:27.680358+0000","last_clean":"2024-03-06T07:26:27.680358+0000","last_became_active":"2024-03-06T07:26:16.615625+0000","last_became_peered":"2024-03-06T07:26:16.615625+0000","last_unstale":"2024-03-06T07:26:27.680358+0000","last_undegraded":"2024-03-06T07:26:27.680358+0000","last_fullsized":"2024-03-06T07:26:27.680358+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:24.591774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684700+0000","last_change":"2024-03-06T07:26:26.708533+0000","last_active":"2024-03-06T07:26:27.684700+0000","last_peered":"2024-03-06T07:26:27.684700+0000","last_clean":"2024-03-06T07:26:27.684700+0000","last_became_active":"2024-03-06T07:26:26.708206+0000","last_became_peered":"2024-03-06T07:26:26.708206+0000","last_unstale":"2024-03-06T07:26:27.684700+0000","last_undegraded":"2024-03-06T07:26:27.684700+0000","last_fullsized":"2024-03-06T07:26:27.684700+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:38.410943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854315+0000","last_change":"2024-03-06T07:26:25.677850+0000","last_active":"2024-03-06T07:26:27.854315+0000","last_peered":"2024-03-06T07:26:27.854315+0000","last_clean":"2024-03-06T07:26:27.854315+0000","last_became_active":"2024-03-06T07:26:25.677154+0000","last_became_peered":"2024-03-06T07:26:25.677154+0000","last_unstale":"2024-03-06T07:26:27.854315+0000","last_undegraded":"2024-03-06T07:26:27.854315+0000","last_fullsized":"2024-03-06T07:26:27.854315+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:13.665071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700223+0000","last_change":"2024-03-06T07:26:17.626866+0000","last_active":"2024-03-06T07:26:27.700223+0000","last_peered":"2024-03-06T07:26:27.700223+0000","last_clean":"2024-03-06T07:26:27.700223+0000","last_became_active":"2024-03-06T07:26:17.626586+0000","last_became_peered":"2024-03-06T07:26:17.626586+0000","last_unstale":"2024-03-06T07:26:27.700223+0000","last_undegraded":"2024-03-06T07:26:27.700223+0000","last_fullsized":"2024-03-06T07:26:27.700223+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:34.182149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700639+0000","last_change":"2024-03-06T07:26:16.622797+0000","last_active":"2024-03-06T07:26:27.700639+0000","last_peered":"2024-03-06T07:26:27.700639+0000","last_clean":"2024-03-06T07:26:27.700639+0000","last_became_active":"2024-03-06T07:26:16.622534+0000","last_became_peered":"2024-03-06T07:26:16.622534+0000","last_unstale":"2024-03-06T07:26:27.700639+0000","last_undegraded":"2024-03-06T07:26:27.700639+0000","last_fullsized":"2024-03-06T07:26:27.700639+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:39.697277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854075+0000","last_change":"2024-03-06T07:26:26.688274+0000","last_active":"2024-03-06T07:26:27.854075+0000","last_peered":"2024-03-06T07:26:27.854075+0000","last_clean":"2024-03-06T07:26:27.854075+0000","last_became_active":"2024-03-06T07:26:26.687023+0000","last_became_peered":"2024-03-06T07:26:26.687023+0000","last_unstale":"2024-03-06T07:26:27.854075+0000","last_undegraded":"2024-03-06T07:26:27.854075+0000","last_fullsized":"2024-03-06T07:26:27.854075+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:39.420779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702829+0000","last_change":"2024-03-06T07:26:25.679443+0000","last_active":"2024-03-06T07:26:27.702829+0000","last_peered":"2024-03-06T07:26:27.702829+0000","last_clean":"2024-03-06T07:26:27.702829+0000","last_became_active":"2024-03-06T07:26:25.679202+0000","last_became_peered":"2024-03-06T07:26:25.679202+0000","last_unstale":"2024-03-06T07:26:27.702829+0000","last_undegraded":"2024-03-06T07:26:27.702829+0000","last_fullsized":"2024-03-06T07:26:27.702829+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:10.282009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683851+0000","last_change":"2024-03-06T07:26:17.632829+0000","last_active":"2024-03-06T07:26:27.683851+0000","last_peered":"2024-03-06T07:26:27.683851+0000","last_clean":"2024-03-06T07:26:27.683851+0000","last_became_active":"2024-03-06T07:26:17.631051+0000","last_became_peered":"2024-03-06T07:26:17.631051+0000","last_unstale":"2024-03-06T07:26:27.683851+0000","last_undegraded":"2024-03-06T07:26:27.683851+0000","last_fullsized":"2024-03-06T07:26:27.683851+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:09.224572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702266+0000","last_change":"2024-03-06T07:26:16.618575+0000","last_active":"2024-03-06T07:26:27.702266+0000","last_peered":"2024-03-06T07:26:27.702266+0000","last_clean":"2024-03-06T07:26:27.702266+0000","last_became_active":"2024-03-06T07:26:16.618143+0000","last_became_peered":"2024-03-06T07:26:16.618143+0000","last_unstale":"2024-03-06T07:26:27.702266+0000","last_undegraded":"2024-03-06T07:26:27.702266+0000","last_fullsized":"2024-03-06T07:26:27.702266+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:48.709582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703107+0000","last_change":"2024-03-06T07:26:26.707394+0000","last_active":"2024-03-06T07:26:27.703107+0000","last_peered":"2024-03-06T07:26:27.703107+0000","last_clean":"2024-03-06T07:26:27.703107+0000","last_became_active":"2024-03-06T07:26:26.706799+0000","last_became_peered":"2024-03-06T07:26:26.706799+0000","last_unstale":"2024-03-06T07:26:27.703107+0000","last_undegraded":"2024-03-06T07:26:27.703107+0000","last_fullsized":"2024-03-06T07:26:27.703107+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:19.825140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682565+0000","last_change":"2024-03-06T07:26:25.675283+0000","last_active":"2024-03-06T07:26:27.682565+0000","last_peered":"2024-03-06T07:26:27.682565+0000","last_clean":"2024-03-06T07:26:27.682565+0000","last_became_active":"2024-03-06T07:26:25.675001+0000","last_became_peered":"2024-03-06T07:26:25.675001+0000","last_unstale":"2024-03-06T07:26:27.682565+0000","last_undegraded":"2024-03-06T07:26:27.682565+0000","last_fullsized":"2024-03-06T07:26:27.682565+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:51.939085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683636+0000","last_change":"2024-03-06T07:26:17.636599+0000","last_active":"2024-03-06T07:26:27.683636+0000","last_peered":"2024-03-06T07:26:27.683636+0000","last_clean":"2024-03-06T07:26:27.683636+0000","last_became_active":"2024-03-06T07:26:17.635451+0000","last_became_peered":"2024-03-06T07:26:17.635451+0000","last_unstale":"2024-03-06T07:26:27.683636+0000","last_undegraded":"2024-03-06T07:26:27.683636+0000","last_fullsized":"2024-03-06T07:26:27.683636+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:27.398343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854475+0000","last_change":"2024-03-06T07:26:16.627460+0000","last_active":"2024-03-06T07:26:27.854475+0000","last_peered":"2024-03-06T07:26:27.854475+0000","last_clean":"2024-03-06T07:26:27.854475+0000","last_became_active":"2024-03-06T07:26:16.626345+0000","last_became_peered":"2024-03-06T07:26:16.626345+0000","last_unstale":"2024-03-06T07:26:27.854475+0000","last_undegraded":"2024-03-06T07:26:27.854475+0000","last_fullsized":"2024-03-06T07:26:27.854475+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:24.257795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682387+0000","last_change":"2024-03-06T07:26:26.704789+0000","last_active":"2024-03-06T07:26:27.682387+0000","last_peered":"2024-03-06T07:26:27.682387+0000","last_clean":"2024-03-06T07:26:27.682387+0000","last_became_active":"2024-03-06T07:26:26.703629+0000","last_became_peered":"2024-03-06T07:26:26.703629+0000","last_unstale":"2024-03-06T07:26:27.682387+0000","last_undegraded":"2024-03-06T07:26:27.682387+0000","last_fullsized":"2024-03-06T07:26:27.682387+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:48.044634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"152'1","reported_seq":15,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853372+0000","last_change":"2024-03-06T07:26:25.672473+0000","last_active":"2024-03-06T07:26:27.853372+0000","last_peered":"2024-03-06T07:26:27.853372+0000","last_clean":"2024-03-06T07:26:27.853372+0000","last_became_active":"2024-03-06T07:26:25.672210+0000","last_became_peered":"2024-03-06T07:26:25.672210+0000","last_unstale":"2024-03-06T07:26:27.853372+0000","last_undegraded":"2024-03-06T07:26:27.853372+0000","last_fullsized":"2024-03-06T07:26:27.853372+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:09.602353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853363+0000","last_change":"2024-03-06T07:26:17.634986+0000","last_active":"2024-03-06T07:26:27.853363+0000","last_peered":"2024-03-06T07:26:27.853363+0000","last_clean":"2024-03-06T07:26:27.853363+0000","last_became_active":"2024-03-06T07:26:17.634680+0000","last_became_peered":"2024-03-06T07:26:17.634680+0000","last_unstale":"2024-03-06T07:26:27.853363+0000","last_undegraded":"2024-03-06T07:26:27.853363+0000","last_fullsized":"2024-03-06T07:26:27.853363+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:48.372843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"148'3","reported_seq":25,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683510+0000","last_change":"2024-03-06T07:26:16.623947+0000","last_active":"2024-03-06T07:26:27.683510+0000","last_peered":"2024-03-06T07:26:27.683510+0000","last_clean":"2024-03-06T07:26:27.683510+0000","last_became_active":"2024-03-06T07:26:16.623662+0000","last_became_peered":"2024-03-06T07:26:16.623662+0000","last_unstale":"2024-03-06T07:26:27.683510+0000","last_undegraded":"2024-03-06T07:26:27.683510+0000","last_fullsized":"2024-03-06T07:26:27.683510+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:00.411414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702187+0000","last_change":"2024-03-06T07:26:26.687868+0000","last_active":"2024-03-06T07:26:27.702187+0000","last_peered":"2024-03-06T07:26:27.702187+0000","last_clean":"2024-03-06T07:26:27.702187+0000","last_became_active":"2024-03-06T07:26:26.687595+0000","last_became_peered":"2024-03-06T07:26:26.687595+0000","last_unstale":"2024-03-06T07:26:27.702187+0000","last_undegraded":"2024-03-06T07:26:27.702187+0000","last_fullsized":"2024-03-06T07:26:27.702187+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:27.612436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"152'1","reported_seq":15,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680108+0000","last_change":"2024-03-06T07:26:25.677952+0000","last_active":"2024-03-06T07:26:27.680108+0000","last_peered":"2024-03-06T07:26:27.680108+0000","last_clean":"2024-03-06T07:26:27.680108+0000","last_became_active":"2024-03-06T07:26:25.675695+0000","last_became_peered":"2024-03-06T07:26:25.675695+0000","last_unstale":"2024-03-06T07:26:27.680108+0000","last_undegraded":"2024-03-06T07:26:27.680108+0000","last_fullsized":"2024-03-06T07:26:27.680108+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:18.823580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.699991+0000","last_change":"2024-03-06T07:26:17.626440+0000","last_active":"2024-03-06T07:26:27.699991+0000","last_peered":"2024-03-06T07:26:27.699991+0000","last_clean":"2024-03-06T07:26:27.699991+0000","last_became_active":"2024-03-06T07:26:17.626149+0000","last_became_peered":"2024-03-06T07:26:17.626149+0000","last_unstale":"2024-03-06T07:26:27.699991+0000","last_undegraded":"2024-03-06T07:26:27.699991+0000","last_fullsized":"2024-03-06T07:26:27.699991+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:46.191755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"148'6","reported_seq":28,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853822+0000","last_change":"2024-03-06T07:26:16.627536+0000","last_active":"2024-03-06T07:26:27.853822+0000","last_peered":"2024-03-06T07:26:27.853822+0000","last_clean":"2024-03-06T07:26:27.853822+0000","last_became_active":"2024-03-06T07:26:16.627229+0000","last_became_peered":"2024-03-06T07:26:16.627229+0000","last_unstale":"2024-03-06T07:26:27.853822+0000","last_undegraded":"2024-03-06T07:26:27.853822+0000","last_fullsized":"2024-03-06T07:26:27.853822+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:00.825243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854293+0000","last_change":"2024-03-06T07:26:26.700560+0000","last_active":"2024-03-06T07:26:27.854293+0000","last_peered":"2024-03-06T07:26:27.854293+0000","last_clean":"2024-03-06T07:26:27.854293+0000","last_became_active":"2024-03-06T07:26:26.700215+0000","last_became_peered":"2024-03-06T07:26:26.700215+0000","last_unstale":"2024-03-06T07:26:27.854293+0000","last_undegraded":"2024-03-06T07:26:27.854293+0000","last_fullsized":"2024-03-06T07:26:27.854293+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:18.310254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"152'2","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854601+0000","last_change":"2024-03-06T07:26:25.678187+0000","last_active":"2024-03-06T07:26:27.854601+0000","last_peered":"2024-03-06T07:26:27.854601+0000","last_clean":"2024-03-06T07:26:27.854601+0000","last_became_active":"2024-03-06T07:26:25.677933+0000","last_became_peered":"2024-03-06T07:26:25.677933+0000","last_unstale":"2024-03-06T07:26:27.854601+0000","last_undegraded":"2024-03-06T07:26:27.854601+0000","last_fullsized":"2024-03-06T07:26:27.854601+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:22.459161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853863+0000","last_change":"2024-03-06T07:26:17.638945+0000","last_active":"2024-03-06T07:26:27.853863+0000","last_peered":"2024-03-06T07:26:27.853863+0000","last_clean":"2024-03-06T07:26:27.853863+0000","last_became_active":"2024-03-06T07:26:17.638664+0000","last_became_peered":"2024-03-06T07:26:17.638664+0000","last_unstale":"2024-03-06T07:26:27.853863+0000","last_undegraded":"2024-03-06T07:26:27.853863+0000","last_fullsized":"2024-03-06T07:26:27.853863+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:15.414267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"148'2","reported_seq":24,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680199+0000","last_change":"2024-03-06T07:26:16.625884+0000","last_active":"2024-03-06T07:26:27.680199+0000","last_peered":"2024-03-06T07:26:27.680199+0000","last_clean":"2024-03-06T07:26:27.680199+0000","last_became_active":"2024-03-06T07:26:16.625605+0000","last_became_peered":"2024-03-06T07:26:16.625605+0000","last_unstale":"2024-03-06T07:26:27.680199+0000","last_undegraded":"2024-03-06T07:26:27.680199+0000","last_fullsized":"2024-03-06T07:26:27.680199+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:13.400206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854578+0000","last_change":"2024-03-06T07:26:26.701244+0000","last_active":"2024-03-06T07:26:27.854578+0000","last_peered":"2024-03-06T07:26:27.854578+0000","last_clean":"2024-03-06T07:26:27.854578+0000","last_became_active":"2024-03-06T07:26:26.700943+0000","last_became_peered":"2024-03-06T07:26:26.700943+0000","last_unstale":"2024-03-06T07:26:27.854578+0000","last_undegraded":"2024-03-06T07:26:27.854578+0000","last_fullsized":"2024-03-06T07:26:27.854578+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:58.770097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"152'2","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682985+0000","last_change":"2024-03-06T07:26:25.677490+0000","last_active":"2024-03-06T07:26:27.682985+0000","last_peered":"2024-03-06T07:26:27.682985+0000","last_clean":"2024-03-06T07:26:27.682985+0000","last_became_active":"2024-03-06T07:26:25.677161+0000","last_became_peered":"2024-03-06T07:26:25.677161+0000","last_unstale":"2024-03-06T07:26:27.682985+0000","last_undegraded":"2024-03-06T07:26:27.682985+0000","last_fullsized":"2024-03-06T07:26:27.682985+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:04.189705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702839+0000","last_change":"2024-03-06T07:26:17.631376+0000","last_active":"2024-03-06T07:26:27.702839+0000","last_peered":"2024-03-06T07:26:27.702839+0000","last_clean":"2024-03-06T07:26:27.702839+0000","last_became_active":"2024-03-06T07:26:17.631068+0000","last_became_peered":"2024-03-06T07:26:17.631068+0000","last_unstale":"2024-03-06T07:26:27.702839+0000","last_undegraded":"2024-03-06T07:26:27.702839+0000","last_fullsized":"2024-03-06T07:26:27.702839+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:46.783070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853507+0000","last_change":"2024-03-06T07:26:16.625352+0000","last_active":"2024-03-06T07:26:27.853507+0000","last_peered":"2024-03-06T07:26:27.853507+0000","last_clean":"2024-03-06T07:26:27.853507+0000","last_became_active":"2024-03-06T07:26:16.624893+0000","last_became_peered":"2024-03-06T07:26:16.624893+0000","last_unstale":"2024-03-06T07:26:27.853507+0000","last_undegraded":"2024-03-06T07:26:27.853507+0000","last_fullsized":"2024-03-06T07:26:27.853507+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:29.596513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684298+0000","last_change":"2024-03-06T07:26:26.696441+0000","last_active":"2024-03-06T07:26:27.684298+0000","last_peered":"2024-03-06T07:26:27.684298+0000","last_clean":"2024-03-06T07:26:27.684298+0000","last_became_active":"2024-03-06T07:26:26.695929+0000","last_became_peered":"2024-03-06T07:26:26.695929+0000","last_unstale":"2024-03-06T07:26:27.684298+0000","last_undegraded":"2024-03-06T07:26:27.684298+0000","last_fullsized":"2024-03-06T07:26:27.684298+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:58.389676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"152'4","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680696+0000","last_change":"2024-03-06T07:26:25.672657+0000","last_active":"2024-03-06T07:26:27.680696+0000","last_peered":"2024-03-06T07:26:27.680696+0000","last_clean":"2024-03-06T07:26:27.680696+0000","last_became_active":"2024-03-06T07:26:25.672330+0000","last_became_peered":"2024-03-06T07:26:25.672330+0000","last_unstale":"2024-03-06T07:26:27.680696+0000","last_undegraded":"2024-03-06T07:26:27.680696+0000","last_fullsized":"2024-03-06T07:26:27.680696+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:50.223092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854791+0000","last_change":"2024-03-06T07:26:17.639031+0000","last_active":"2024-03-06T07:26:27.854791+0000","last_peered":"2024-03-06T07:26:27.854791+0000","last_clean":"2024-03-06T07:26:27.854791+0000","last_became_active":"2024-03-06T07:26:17.638716+0000","last_became_peered":"2024-03-06T07:26:17.638716+0000","last_unstale":"2024-03-06T07:26:27.854791+0000","last_undegraded":"2024-03-06T07:26:27.854791+0000","last_fullsized":"2024-03-06T07:26:27.854791+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:46.783070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853941+0000","last_change":"2024-03-06T07:26:16.620528+0000","last_active":"2024-03-06T07:26:27.853941+0000","last_peered":"2024-03-06T07:26:27.853941+0000","last_clean":"2024-03-06T07:26:27.853941+0000","last_became_active":"2024-03-06T07:26:16.620245+0000","last_became_peered":"2024-03-06T07:26:16.620245+0000","last_unstale":"2024-03-06T07:26:27.853941+0000","last_undegraded":"2024-03-06T07:26:27.853941+0000","last_fullsized":"2024-03-06T07:26:27.853941+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:39.028254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700169+0000","last_change":"2024-03-06T07:26:26.695916+0000","last_active":"2024-03-06T07:26:27.700169+0000","last_peered":"2024-03-06T07:26:27.700169+0000","last_clean":"2024-03-06T07:26:27.700169+0000","last_became_active":"2024-03-06T07:26:26.695663+0000","last_became_peered":"2024-03-06T07:26:26.695663+0000","last_unstale":"2024-03-06T07:26:27.700169+0000","last_undegraded":"2024-03-06T07:26:27.700169+0000","last_fullsized":"2024-03-06T07:26:27.700169+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:28.555022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"152'1","reported_seq":15,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854342+0000","last_change":"2024-03-06T07:26:25.673722+0000","last_active":"2024-03-06T07:26:27.854342+0000","last_peered":"2024-03-06T07:26:27.854342+0000","last_clean":"2024-03-06T07:26:27.854342+0000","last_became_active":"2024-03-06T07:26:25.672078+0000","last_became_peered":"2024-03-06T07:26:25.672078+0000","last_unstale":"2024-03-06T07:26:27.854342+0000","last_undegraded":"2024-03-06T07:26:27.854342+0000","last_fullsized":"2024-03-06T07:26:27.854342+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:36.768745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700563+0000","last_change":"2024-03-06T07:26:17.641468+0000","last_active":"2024-03-06T07:26:27.700563+0000","last_peered":"2024-03-06T07:26:27.700563+0000","last_clean":"2024-03-06T07:26:27.700563+0000","last_became_active":"2024-03-06T07:26:17.641155+0000","last_became_peered":"2024-03-06T07:26:17.641155+0000","last_unstale":"2024-03-06T07:26:27.700563+0000","last_undegraded":"2024-03-06T07:26:27.700563+0000","last_fullsized":"2024-03-06T07:26:27.700563+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:27.464355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"148'8","reported_seq":30,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702314+0000","last_change":"2024-03-06T07:26:16.620548+0000","last_active":"2024-03-06T07:26:27.702314+0000","last_peered":"2024-03-06T07:26:27.702314+0000","last_clean":"2024-03-06T07:26:27.702314+0000","last_became_active":"2024-03-06T07:26:16.620289+0000","last_became_peered":"2024-03-06T07:26:16.620289+0000","last_unstale":"2024-03-06T07:26:27.702314+0000","last_undegraded":"2024-03-06T07:26:27.702314+0000","last_fullsized":"2024-03-06T07:26:27.702314+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:08.155799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703250+0000","last_change":"2024-03-06T07:26:26.691751+0000","last_active":"2024-03-06T07:26:27.703250+0000","last_peered":"2024-03-06T07:26:27.703250+0000","last_clean":"2024-03-06T07:26:27.703250+0000","last_became_active":"2024-03-06T07:26:26.691505+0000","last_became_peered":"2024-03-06T07:26:26.691505+0000","last_unstale":"2024-03-06T07:26:27.703250+0000","last_undegraded":"2024-03-06T07:26:27.703250+0000","last_fullsized":"2024-03-06T07:26:27.703250+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:24.976580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.699874+0000","last_change":"2024-03-06T07:26:25.676004+0000","last_active":"2024-03-06T07:26:27.699874+0000","last_peered":"2024-03-06T07:26:27.699874+0000","last_clean":"2024-03-06T07:26:27.699874+0000","last_became_active":"2024-03-06T07:26:25.675756+0000","last_became_peered":"2024-03-06T07:26:25.675756+0000","last_unstale":"2024-03-06T07:26:27.699874+0000","last_undegraded":"2024-03-06T07:26:27.699874+0000","last_fullsized":"2024-03-06T07:26:27.699874+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:09.976162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702169+0000","last_change":"2024-03-06T07:26:17.629187+0000","last_active":"2024-03-06T07:26:27.702169+0000","last_peered":"2024-03-06T07:26:27.702169+0000","last_clean":"2024-03-06T07:26:27.702169+0000","last_became_active":"2024-03-06T07:26:17.628903+0000","last_became_peered":"2024-03-06T07:26:17.628903+0000","last_unstale":"2024-03-06T07:26:27.702169+0000","last_undegraded":"2024-03-06T07:26:27.702169+0000","last_fullsized":"2024-03-06T07:26:27.702169+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:25.173456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"148'1","reported_seq":23,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854561+0000","last_change":"2024-03-06T07:26:16.617108+0000","last_active":"2024-03-06T07:26:27.854561+0000","last_peered":"2024-03-06T07:26:27.854561+0000","last_clean":"2024-03-06T07:26:27.854561+0000","last_became_active":"2024-03-06T07:26:16.616776+0000","last_became_peered":"2024-03-06T07:26:16.616776+0000","last_unstale":"2024-03-06T07:26:27.854561+0000","last_undegraded":"2024-03-06T07:26:27.854561+0000","last_fullsized":"2024-03-06T07:26:27.854561+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:01.223054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684029+0000","last_change":"2024-03-06T07:26:26.699183+0000","last_active":"2024-03-06T07:26:27.684029+0000","last_peered":"2024-03-06T07:26:27.684029+0000","last_clean":"2024-03-06T07:26:27.684029+0000","last_became_active":"2024-03-06T07:26:26.698593+0000","last_became_peered":"2024-03-06T07:26:26.698593+0000","last_unstale":"2024-03-06T07:26:27.684029+0000","last_undegraded":"2024-03-06T07:26:27.684029+0000","last_fullsized":"2024-03-06T07:26:27.684029+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:44.977293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853384+0000","last_change":"2024-03-06T07:26:25.679689+0000","last_active":"2024-03-06T07:26:27.853384+0000","last_peered":"2024-03-06T07:26:27.853384+0000","last_clean":"2024-03-06T07:26:27.853384+0000","last_became_active":"2024-03-06T07:26:25.679402+0000","last_became_peered":"2024-03-06T07:26:25.679402+0000","last_unstale":"2024-03-06T07:26:27.853384+0000","last_undegraded":"2024-03-06T07:26:27.853384+0000","last_fullsized":"2024-03-06T07:26:27.853384+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:50.182875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853697+0000","last_change":"2024-03-06T07:26:17.634792+0000","last_active":"2024-03-06T07:26:27.853697+0000","last_peered":"2024-03-06T07:26:27.853697+0000","last_clean":"2024-03-06T07:26:27.853697+0000","last_became_active":"2024-03-06T07:26:17.634367+0000","last_became_peered":"2024-03-06T07:26:17.634367+0000","last_unstale":"2024-03-06T07:26:27.853697+0000","last_undegraded":"2024-03-06T07:26:27.853697+0000","last_fullsized":"2024-03-06T07:26:27.853697+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:11.386543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"148'2","reported_seq":24,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703255+0000","last_change":"2024-03-06T07:26:16.619012+0000","last_active":"2024-03-06T07:26:27.703255+0000","last_peered":"2024-03-06T07:26:27.703255+0000","last_clean":"2024-03-06T07:26:27.703255+0000","last_became_active":"2024-03-06T07:26:16.617718+0000","last_became_peered":"2024-03-06T07:26:16.617718+0000","last_unstale":"2024-03-06T07:26:27.703255+0000","last_undegraded":"2024-03-06T07:26:27.703255+0000","last_fullsized":"2024-03-06T07:26:27.703255+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:08.220437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853468+0000","last_change":"2024-03-06T07:26:26.689460+0000","last_active":"2024-03-06T07:26:27.853468+0000","last_peered":"2024-03-06T07:26:27.853468+0000","last_clean":"2024-03-06T07:26:27.853468+0000","last_became_active":"2024-03-06T07:26:26.689176+0000","last_became_peered":"2024-03-06T07:26:26.689176+0000","last_unstale":"2024-03-06T07:26:27.853468+0000","last_undegraded":"2024-03-06T07:26:27.853468+0000","last_fullsized":"2024-03-06T07:26:27.853468+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:13.113595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684084+0000","last_change":"2024-03-06T07:26:25.677892+0000","last_active":"2024-03-06T07:26:27.684084+0000","last_peered":"2024-03-06T07:26:27.684084+0000","last_clean":"2024-03-06T07:26:27.684084+0000","last_became_active":"2024-03-06T07:26:25.677625+0000","last_became_peered":"2024-03-06T07:26:25.677625+0000","last_unstale":"2024-03-06T07:26:27.684084+0000","last_undegraded":"2024-03-06T07:26:27.684084+0000","last_fullsized":"2024-03-06T07:26:27.684084+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:53.012502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854399+0000","last_change":"2024-03-06T07:26:17.633394+0000","last_active":"2024-03-06T07:26:27.854399+0000","last_peered":"2024-03-06T07:26:27.854399+0000","last_clean":"2024-03-06T07:26:27.854399+0000","last_became_active":"2024-03-06T07:26:17.632438+0000","last_became_peered":"2024-03-06T07:26:17.632438+0000","last_unstale":"2024-03-06T07:26:27.854399+0000","last_undegraded":"2024-03-06T07:26:27.854399+0000","last_fullsized":"2024-03-06T07:26:27.854399+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:12.024002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"148'1","reported_seq":23,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680591+0000","last_change":"2024-03-06T07:26:16.628646+0000","last_active":"2024-03-06T07:26:27.680591+0000","last_peered":"2024-03-06T07:26:27.680591+0000","last_clean":"2024-03-06T07:26:27.680591+0000","last_became_active":"2024-03-06T07:26:16.627947+0000","last_became_peered":"2024-03-06T07:26:16.627947+0000","last_unstale":"2024-03-06T07:26:27.680591+0000","last_undegraded":"2024-03-06T07:26:27.680591+0000","last_fullsized":"2024-03-06T07:26:27.680591+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:25.875046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682566+0000","last_change":"2024-03-06T07:26:26.703286+0000","last_active":"2024-03-06T07:26:27.682566+0000","last_peered":"2024-03-06T07:26:27.682566+0000","last_clean":"2024-03-06T07:26:27.682566+0000","last_became_active":"2024-03-06T07:26:26.702961+0000","last_became_peered":"2024-03-06T07:26:26.702961+0000","last_unstale":"2024-03-06T07:26:27.682566+0000","last_undegraded":"2024-03-06T07:26:27.682566+0000","last_fullsized":"2024-03-06T07:26:27.682566+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:10.885758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684052+0000","last_change":"2024-03-06T07:26:25.677636+0000","last_active":"2024-03-06T07:26:27.684052+0000","last_peered":"2024-03-06T07:26:27.684052+0000","last_clean":"2024-03-06T07:26:27.684052+0000","last_became_active":"2024-03-06T07:26:25.677362+0000","last_became_peered":"2024-03-06T07:26:25.677362+0000","last_unstale":"2024-03-06T07:26:27.684052+0000","last_undegraded":"2024-03-06T07:26:27.684052+0000","last_fullsized":"2024-03-06T07:26:27.684052+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:31.280183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854482+0000","last_change":"2024-03-06T07:26:17.634254+0000","last_active":"2024-03-06T07:26:27.854482+0000","last_peered":"2024-03-06T07:26:27.854482+0000","last_clean":"2024-03-06T07:26:27.854482+0000","last_became_active":"2024-03-06T07:26:17.633961+0000","last_became_peered":"2024-03-06T07:26:17.633961+0000","last_unstale":"2024-03-06T07:26:27.854482+0000","last_undegraded":"2024-03-06T07:26:27.854482+0000","last_fullsized":"2024-03-06T07:26:27.854482+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:59.873611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702650+0000","last_change":"2024-03-06T07:26:16.619532+0000","last_active":"2024-03-06T07:26:27.702650+0000","last_peered":"2024-03-06T07:26:27.702650+0000","last_clean":"2024-03-06T07:26:27.702650+0000","last_became_active":"2024-03-06T07:26:16.619261+0000","last_became_peered":"2024-03-06T07:26:16.619261+0000","last_unstale":"2024-03-06T07:26:27.702650+0000","last_undegraded":"2024-03-06T07:26:27.702650+0000","last_fullsized":"2024-03-06T07:26:27.702650+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:34.031068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684675+0000","last_change":"2024-03-06T07:26:26.695922+0000","last_active":"2024-03-06T07:26:27.684675+0000","last_peered":"2024-03-06T07:26:27.684675+0000","last_clean":"2024-03-06T07:26:27.684675+0000","last_became_active":"2024-03-06T07:26:26.695643+0000","last_became_peered":"2024-03-06T07:26:26.695643+0000","last_unstale":"2024-03-06T07:26:27.684675+0000","last_undegraded":"2024-03-06T07:26:27.684675+0000","last_fullsized":"2024-03-06T07:26:27.684675+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:43.140802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854148+0000","last_change":"2024-03-06T07:26:25.672463+0000","last_active":"2024-03-06T07:26:27.854148+0000","last_peered":"2024-03-06T07:26:27.854148+0000","last_clean":"2024-03-06T07:26:27.854148+0000","last_became_active":"2024-03-06T07:26:25.672192+0000","last_became_peered":"2024-03-06T07:26:25.672192+0000","last_unstale":"2024-03-06T07:26:27.854148+0000","last_undegraded":"2024-03-06T07:26:27.854148+0000","last_fullsized":"2024-03-06T07:26:27.854148+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:54.265085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853647+0000","last_change":"2024-03-06T07:26:17.637879+0000","last_active":"2024-03-06T07:26:27.853647+0000","last_peered":"2024-03-06T07:26:27.853647+0000","last_clean":"2024-03-06T07:26:27.853647+0000","last_became_active":"2024-03-06T07:26:17.637094+0000","last_became_peered":"2024-03-06T07:26:17.637094+0000","last_unstale":"2024-03-06T07:26:27.853647+0000","last_undegraded":"2024-03-06T07:26:27.853647+0000","last_fullsized":"2024-03-06T07:26:27.853647+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:34.084422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"148'2","reported_seq":24,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680265+0000","last_change":"2024-03-06T07:26:16.622856+0000","last_active":"2024-03-06T07:26:27.680265+0000","last_peered":"2024-03-06T07:26:27.680265+0000","last_clean":"2024-03-06T07:26:27.680265+0000","last_became_active":"2024-03-06T07:26:16.622421+0000","last_became_peered":"2024-03-06T07:26:16.622421+0000","last_unstale":"2024-03-06T07:26:27.680265+0000","last_undegraded":"2024-03-06T07:26:27.680265+0000","last_fullsized":"2024-03-06T07:26:27.680265+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:40.651795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700557+0000","last_change":"2024-03-06T07:26:26.701119+0000","last_active":"2024-03-06T07:26:27.700557+0000","last_peered":"2024-03-06T07:26:27.700557+0000","last_clean":"2024-03-06T07:26:27.700557+0000","last_became_active":"2024-03-06T07:26:26.700823+0000","last_became_peered":"2024-03-06T07:26:26.700823+0000","last_unstale":"2024-03-06T07:26:27.700557+0000","last_undegraded":"2024-03-06T07:26:27.700557+0000","last_fullsized":"2024-03-06T07:26:27.700557+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:23.853802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853732+0000","last_change":"2024-03-06T07:26:25.683133+0000","last_active":"2024-03-06T07:26:27.853732+0000","last_peered":"2024-03-06T07:26:27.853732+0000","last_clean":"2024-03-06T07:26:27.853732+0000","last_became_active":"2024-03-06T07:26:25.682836+0000","last_became_peered":"2024-03-06T07:26:25.682836+0000","last_unstale":"2024-03-06T07:26:27.853732+0000","last_undegraded":"2024-03-06T07:26:27.853732+0000","last_fullsized":"2024-03-06T07:26:27.853732+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:44.114707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680586+0000","last_change":"2024-03-06T07:26:17.629188+0000","last_active":"2024-03-06T07:26:27.680586+0000","last_peered":"2024-03-06T07:26:27.680586+0000","last_clean":"2024-03-06T07:26:27.680586+0000","last_became_active":"2024-03-06T07:26:17.628881+0000","last_became_peered":"2024-03-06T07:26:17.628881+0000","last_unstale":"2024-03-06T07:26:27.680586+0000","last_undegraded":"2024-03-06T07:26:27.680586+0000","last_fullsized":"2024-03-06T07:26:27.680586+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:44.957682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854605+0000","last_change":"2024-03-06T07:26:16.625755+0000","last_active":"2024-03-06T07:26:27.854605+0000","last_peered":"2024-03-06T07:26:27.854605+0000","last_clean":"2024-03-06T07:26:27.854605+0000","last_became_active":"2024-03-06T07:26:16.625489+0000","last_became_peered":"2024-03-06T07:26:16.625489+0000","last_unstale":"2024-03-06T07:26:27.854605+0000","last_undegraded":"2024-03-06T07:26:27.854605+0000","last_fullsized":"2024-03-06T07:26:27.854605+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:03.053080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700195+0000","last_change":"2024-03-06T07:26:26.708290+0000","last_active":"2024-03-06T07:26:27.700195+0000","last_peered":"2024-03-06T07:26:27.700195+0000","last_clean":"2024-03-06T07:26:27.700195+0000","last_became_active":"2024-03-06T07:26:26.708035+0000","last_became_peered":"2024-03-06T07:26:26.708035+0000","last_unstale":"2024-03-06T07:26:27.700195+0000","last_undegraded":"2024-03-06T07:26:27.700195+0000","last_fullsized":"2024-03-06T07:26:27.700195+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:40.319895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"152'2","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680716+0000","last_change":"2024-03-06T07:26:25.676230+0000","last_active":"2024-03-06T07:26:27.680716+0000","last_peered":"2024-03-06T07:26:27.680716+0000","last_clean":"2024-03-06T07:26:27.680716+0000","last_became_active":"2024-03-06T07:26:25.675724+0000","last_became_peered":"2024-03-06T07:26:25.675724+0000","last_unstale":"2024-03-06T07:26:27.680716+0000","last_undegraded":"2024-03-06T07:26:27.680716+0000","last_fullsized":"2024-03-06T07:26:27.680716+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:56.435162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680607+0000","last_change":"2024-03-06T07:26:17.629894+0000","last_active":"2024-03-06T07:26:27.680607+0000","last_peered":"2024-03-06T07:26:27.680607+0000","last_clean":"2024-03-06T07:26:27.680607+0000","last_became_active":"2024-03-06T07:26:17.629624+0000","last_became_peered":"2024-03-06T07:26:17.629624+0000","last_unstale":"2024-03-06T07:26:27.680607+0000","last_undegraded":"2024-03-06T07:26:27.680607+0000","last_fullsized":"2024-03-06T07:26:27.680607+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:48.520981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700694+0000","last_change":"2024-03-06T07:26:16.618167+0000","last_active":"2024-03-06T07:26:27.700694+0000","last_peered":"2024-03-06T07:26:27.700694+0000","last_clean":"2024-03-06T07:26:27.700694+0000","last_became_active":"2024-03-06T07:26:16.617893+0000","last_became_peered":"2024-03-06T07:26:16.617893+0000","last_unstale":"2024-03-06T07:26:27.700694+0000","last_undegraded":"2024-03-06T07:26:27.700694+0000","last_fullsized":"2024-03-06T07:26:27.700694+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:33.738360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702793+0000","last_change":"2024-03-06T07:26:26.687658+0000","last_active":"2024-03-06T07:26:27.702793+0000","last_peered":"2024-03-06T07:26:27.702793+0000","last_clean":"2024-03-06T07:26:27.702793+0000","last_became_active":"2024-03-06T07:26:26.687392+0000","last_became_peered":"2024-03-06T07:26:26.687392+0000","last_unstale":"2024-03-06T07:26:27.702793+0000","last_undegraded":"2024-03-06T07:26:27.702793+0000","last_fullsized":"2024-03-06T07:26:27.702793+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:48.863082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"153'1","reported_seq":15,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700114+0000","last_change":"2024-03-06T07:26:25.681294+0000","last_active":"2024-03-06T07:26:27.700114+0000","last_peered":"2024-03-06T07:26:27.700114+0000","last_clean":"2024-03-06T07:26:27.700114+0000","last_became_active":"2024-03-06T07:26:25.681036+0000","last_became_peered":"2024-03-06T07:26:25.681036+0000","last_unstale":"2024-03-06T07:26:27.700114+0000","last_undegraded":"2024-03-06T07:26:27.700114+0000","last_fullsized":"2024-03-06T07:26:27.700114+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:24.719315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684768+0000","last_change":"2024-03-06T07:26:17.632248+0000","last_active":"2024-03-06T07:26:27.684768+0000","last_peered":"2024-03-06T07:26:27.684768+0000","last_clean":"2024-03-06T07:26:27.684768+0000","last_became_active":"2024-03-06T07:26:17.631912+0000","last_became_peered":"2024-03-06T07:26:17.631912+0000","last_unstale":"2024-03-06T07:26:27.684768+0000","last_undegraded":"2024-03-06T07:26:27.684768+0000","last_fullsized":"2024-03-06T07:26:27.684768+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:29.113308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683862+0000","last_change":"2024-03-06T07:26:16.616949+0000","last_active":"2024-03-06T07:26:27.683862+0000","last_peered":"2024-03-06T07:26:27.683862+0000","last_clean":"2024-03-06T07:26:27.683862+0000","last_became_active":"2024-03-06T07:26:16.616670+0000","last_became_peered":"2024-03-06T07:26:16.616670+0000","last_unstale":"2024-03-06T07:26:27.683862+0000","last_undegraded":"2024-03-06T07:26:27.683862+0000","last_fullsized":"2024-03-06T07:26:27.683862+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:59.587645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680156+0000","last_change":"2024-03-06T07:26:26.701033+0000","last_active":"2024-03-06T07:26:27.680156+0000","last_peered":"2024-03-06T07:26:27.680156+0000","last_clean":"2024-03-06T07:26:27.680156+0000","last_became_active":"2024-03-06T07:26:26.700715+0000","last_became_peered":"2024-03-06T07:26:26.700715+0000","last_unstale":"2024-03-06T07:26:27.680156+0000","last_undegraded":"2024-03-06T07:26:27.680156+0000","last_fullsized":"2024-03-06T07:26:27.680156+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:55.266644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.699864+0000","last_change":"2024-03-06T07:26:25.674406+0000","last_active":"2024-03-06T07:26:27.699864+0000","last_peered":"2024-03-06T07:26:27.699864+0000","last_clean":"2024-03-06T07:26:27.699864+0000","last_became_active":"2024-03-06T07:26:25.673939+0000","last_became_peered":"2024-03-06T07:26:25.673939+0000","last_unstale":"2024-03-06T07:26:27.699864+0000","last_undegraded":"2024-03-06T07:26:27.699864+0000","last_fullsized":"2024-03-06T07:26:27.699864+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:45.214049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854395+0000","last_change":"2024-03-06T07:26:17.628994+0000","last_active":"2024-03-06T07:26:27.854395+0000","last_peered":"2024-03-06T07:26:27.854395+0000","last_clean":"2024-03-06T07:26:27.854395+0000","last_became_active":"2024-03-06T07:26:17.628722+0000","last_became_peered":"2024-03-06T07:26:17.628722+0000","last_unstale":"2024-03-06T07:26:27.854395+0000","last_undegraded":"2024-03-06T07:26:27.854395+0000","last_fullsized":"2024-03-06T07:26:27.854395+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:43.090011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"148'2","reported_seq":24,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680809+0000","last_change":"2024-03-06T07:26:16.620011+0000","last_active":"2024-03-06T07:26:27.680809+0000","last_peered":"2024-03-06T07:26:27.680809+0000","last_clean":"2024-03-06T07:26:27.680809+0000","last_became_active":"2024-03-06T07:26:16.619736+0000","last_became_peered":"2024-03-06T07:26:16.619736+0000","last_unstale":"2024-03-06T07:26:27.680809+0000","last_undegraded":"2024-03-06T07:26:27.680809+0000","last_fullsized":"2024-03-06T07:26:27.680809+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:24.169321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853636+0000","last_change":"2024-03-06T07:26:26.694087+0000","last_active":"2024-03-06T07:26:27.853636+0000","last_peered":"2024-03-06T07:26:27.853636+0000","last_clean":"2024-03-06T07:26:27.853636+0000","last_became_active":"2024-03-06T07:26:26.693812+0000","last_became_peered":"2024-03-06T07:26:26.693812+0000","last_unstale":"2024-03-06T07:26:27.853636+0000","last_undegraded":"2024-03-06T07:26:27.853636+0000","last_fullsized":"2024-03-06T07:26:27.853636+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:28.200856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853595+0000","last_change":"2024-03-06T07:26:25.670698+0000","last_active":"2024-03-06T07:26:27.853595+0000","last_peered":"2024-03-06T07:26:27.853595+0000","last_clean":"2024-03-06T07:26:27.853595+0000","last_became_active":"2024-03-06T07:26:25.670446+0000","last_became_peered":"2024-03-06T07:26:25.670446+0000","last_unstale":"2024-03-06T07:26:27.853595+0000","last_undegraded":"2024-03-06T07:26:27.853595+0000","last_fullsized":"2024-03-06T07:26:27.853595+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:22.852243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702343+0000","last_change":"2024-03-06T07:26:17.628485+0000","last_active":"2024-03-06T07:26:27.702343+0000","last_peered":"2024-03-06T07:26:27.702343+0000","last_clean":"2024-03-06T07:26:27.702343+0000","last_became_active":"2024-03-06T07:26:17.628204+0000","last_became_peered":"2024-03-06T07:26:17.628204+0000","last_unstale":"2024-03-06T07:26:27.702343+0000","last_undegraded":"2024-03-06T07:26:27.702343+0000","last_fullsized":"2024-03-06T07:26:27.702343+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:56.134884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"149'1","reported_seq":23,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854453+0000","last_change":"2024-03-06T07:26:16.627066+0000","last_active":"2024-03-06T07:26:27.854453+0000","last_peered":"2024-03-06T07:26:27.854453+0000","last_clean":"2024-03-06T07:26:27.854453+0000","last_became_active":"2024-03-06T07:26:16.626418+0000","last_became_peered":"2024-03-06T07:26:16.626418+0000","last_unstale":"2024-03-06T07:26:27.854453+0000","last_undegraded":"2024-03-06T07:26:27.854453+0000","last_fullsized":"2024-03-06T07:26:27.854453+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:37.036435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702097+0000","last_change":"2024-03-06T07:26:26.692608+0000","last_active":"2024-03-06T07:26:27.702097+0000","last_peered":"2024-03-06T07:26:27.702097+0000","last_clean":"2024-03-06T07:26:27.702097+0000","last_became_active":"2024-03-06T07:26:26.692333+0000","last_became_peered":"2024-03-06T07:26:26.692333+0000","last_unstale":"2024-03-06T07:26:27.702097+0000","last_undegraded":"2024-03-06T07:26:27.702097+0000","last_fullsized":"2024-03-06T07:26:27.702097+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:58.389676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"152'2","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.682915+0000","last_change":"2024-03-06T07:26:25.683377+0000","last_active":"2024-03-06T07:26:27.682915+0000","last_peered":"2024-03-06T07:26:27.682915+0000","last_clean":"2024-03-06T07:26:27.682915+0000","last_became_active":"2024-03-06T07:26:25.683159+0000","last_became_peered":"2024-03-06T07:26:25.683159+0000","last_unstale":"2024-03-06T07:26:27.682915+0000","last_undegraded":"2024-03-06T07:26:27.682915+0000","last_fullsized":"2024-03-06T07:26:27.682915+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:54.837891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700119+0000","last_change":"2024-03-06T07:26:17.626185+0000","last_active":"2024-03-06T07:26:27.700119+0000","last_peered":"2024-03-06T07:26:27.700119+0000","last_clean":"2024-03-06T07:26:27.700119+0000","last_became_active":"2024-03-06T07:26:17.625895+0000","last_became_peered":"2024-03-06T07:26:17.625895+0000","last_unstale":"2024-03-06T07:26:27.700119+0000","last_undegraded":"2024-03-06T07:26:27.700119+0000","last_fullsized":"2024-03-06T07:26:27.700119+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:16.007003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854236+0000","last_change":"2024-03-06T07:26:16.625471+0000","last_active":"2024-03-06T07:26:27.854236+0000","last_peered":"2024-03-06T07:26:27.854236+0000","last_clean":"2024-03-06T07:26:27.854236+0000","last_became_active":"2024-03-06T07:26:16.625198+0000","last_became_peered":"2024-03-06T07:26:16.625198+0000","last_unstale":"2024-03-06T07:26:27.854236+0000","last_undegraded":"2024-03-06T07:26:27.854236+0000","last_fullsized":"2024-03-06T07:26:27.854236+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:37.984120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.679891+0000","last_change":"2024-03-06T07:26:26.706286+0000","last_active":"2024-03-06T07:26:27.679891+0000","last_peered":"2024-03-06T07:26:27.679891+0000","last_clean":"2024-03-06T07:26:27.679891+0000","last_became_active":"2024-03-06T07:26:26.706024+0000","last_became_peered":"2024-03-06T07:26:26.706024+0000","last_unstale":"2024-03-06T07:26:27.679891+0000","last_undegraded":"2024-03-06T07:26:27.679891+0000","last_fullsized":"2024-03-06T07:26:27.679891+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:10.603912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.679921+0000","last_change":"2024-03-06T07:26:25.677833+0000","last_active":"2024-03-06T07:26:27.679921+0000","last_peered":"2024-03-06T07:26:27.679921+0000","last_clean":"2024-03-06T07:26:27.679921+0000","last_became_active":"2024-03-06T07:26:25.675930+0000","last_became_peered":"2024-03-06T07:26:25.675930+0000","last_unstale":"2024-03-06T07:26:27.679921+0000","last_undegraded":"2024-03-06T07:26:27.679921+0000","last_fullsized":"2024-03-06T07:26:27.679921+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:11.461549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.702647+0000","last_change":"2024-03-06T07:26:17.630631+0000","last_active":"2024-03-06T07:26:27.702647+0000","last_peered":"2024-03-06T07:26:27.702647+0000","last_clean":"2024-03-06T07:26:27.702647+0000","last_became_active":"2024-03-06T07:26:17.630365+0000","last_became_peered":"2024-03-06T07:26:17.630365+0000","last_unstale":"2024-03-06T07:26:27.702647+0000","last_undegraded":"2024-03-06T07:26:27.702647+0000","last_fullsized":"2024-03-06T07:26:27.702647+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:27.398343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700353+0000","last_change":"2024-03-06T07:26:16.611334+0000","last_active":"2024-03-06T07:26:27.700353+0000","last_peered":"2024-03-06T07:26:27.700353+0000","last_clean":"2024-03-06T07:26:27.700353+0000","last_became_active":"2024-03-06T07:26:16.610646+0000","last_became_peered":"2024-03-06T07:26:16.610646+0000","last_unstale":"2024-03-06T07:26:27.700353+0000","last_undegraded":"2024-03-06T07:26:27.700353+0000","last_fullsized":"2024-03-06T07:26:27.700353+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:32.639393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854569+0000","last_change":"2024-03-06T07:26:26.701760+0000","last_active":"2024-03-06T07:26:27.854569+0000","last_peered":"2024-03-06T07:26:27.854569+0000","last_clean":"2024-03-06T07:26:27.854569+0000","last_became_active":"2024-03-06T07:26:26.701124+0000","last_became_peered":"2024-03-06T07:26:26.701124+0000","last_unstale":"2024-03-06T07:26:27.854569+0000","last_undegraded":"2024-03-06T07:26:27.854569+0000","last_fullsized":"2024-03-06T07:26:27.854569+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:54.014062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"152'1","reported_seq":15,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854021+0000","last_change":"2024-03-06T07:26:25.674222+0000","last_active":"2024-03-06T07:26:27.854021+0000","last_peered":"2024-03-06T07:26:27.854021+0000","last_clean":"2024-03-06T07:26:27.854021+0000","last_became_active":"2024-03-06T07:26:25.673935+0000","last_became_peered":"2024-03-06T07:26:25.673935+0000","last_unstale":"2024-03-06T07:26:27.854021+0000","last_undegraded":"2024-03-06T07:26:27.854021+0000","last_fullsized":"2024-03-06T07:26:27.854021+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:12.179836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683192+0000","last_change":"2024-03-06T07:26:17.632379+0000","last_active":"2024-03-06T07:26:27.683192+0000","last_peered":"2024-03-06T07:26:27.683192+0000","last_clean":"2024-03-06T07:26:27.683192+0000","last_became_active":"2024-03-06T07:26:17.631899+0000","last_became_peered":"2024-03-06T07:26:17.631899+0000","last_unstale":"2024-03-06T07:26:27.683192+0000","last_undegraded":"2024-03-06T07:26:27.683192+0000","last_fullsized":"2024-03-06T07:26:27.683192+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:00.591780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683250+0000","last_change":"2024-03-06T07:26:16.623898+0000","last_active":"2024-03-06T07:26:27.683250+0000","last_peered":"2024-03-06T07:26:27.683250+0000","last_clean":"2024-03-06T07:26:27.683250+0000","last_became_active":"2024-03-06T07:26:16.623602+0000","last_became_peered":"2024-03-06T07:26:16.623602+0000","last_unstale":"2024-03-06T07:26:27.683250+0000","last_undegraded":"2024-03-06T07:26:27.683250+0000","last_fullsized":"2024-03-06T07:26:27.683250+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:56.210974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.703136+0000","last_change":"2024-03-06T07:26:26.692493+0000","last_active":"2024-03-06T07:26:27.703136+0000","last_peered":"2024-03-06T07:26:27.703136+0000","last_clean":"2024-03-06T07:26:27.703136+0000","last_became_active":"2024-03-06T07:26:26.692242+0000","last_became_peered":"2024-03-06T07:26:26.692242+0000","last_unstale":"2024-03-06T07:26:27.703136+0000","last_undegraded":"2024-03-06T07:26:27.703136+0000","last_fullsized":"2024-03-06T07:26:27.703136+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:36.454193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854525+0000","last_change":"2024-03-06T07:26:25.669943+0000","last_active":"2024-03-06T07:26:27.854525+0000","last_peered":"2024-03-06T07:26:27.854525+0000","last_clean":"2024-03-06T07:26:27.854525+0000","last_became_active":"2024-03-06T07:26:25.669439+0000","last_became_peered":"2024-03-06T07:26:25.669439+0000","last_unstale":"2024-03-06T07:26:27.854525+0000","last_undegraded":"2024-03-06T07:26:27.854525+0000","last_fullsized":"2024-03-06T07:26:27.854525+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:49.710750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684463+0000","last_change":"2024-03-06T07:26:17.632856+0000","last_active":"2024-03-06T07:26:27.684463+0000","last_peered":"2024-03-06T07:26:27.684463+0000","last_clean":"2024-03-06T07:26:27.684463+0000","last_became_active":"2024-03-06T07:26:17.632027+0000","last_became_peered":"2024-03-06T07:26:17.632027+0000","last_unstale":"2024-03-06T07:26:27.684463+0000","last_undegraded":"2024-03-06T07:26:27.684463+0000","last_fullsized":"2024-03-06T07:26:27.684463+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:38.988254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853594+0000","last_change":"2024-03-06T07:26:16.627698+0000","last_active":"2024-03-06T07:26:27.853594+0000","last_peered":"2024-03-06T07:26:27.853594+0000","last_clean":"2024-03-06T07:26:27.853594+0000","last_became_active":"2024-03-06T07:26:16.627318+0000","last_became_peered":"2024-03-06T07:26:16.627318+0000","last_unstale":"2024-03-06T07:26:27.853594+0000","last_undegraded":"2024-03-06T07:26:27.853594+0000","last_fullsized":"2024-03-06T07:26:27.853594+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:08.249739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700175+0000","last_change":"2024-03-06T07:26:26.689324+0000","last_active":"2024-03-06T07:26:27.700175+0000","last_peered":"2024-03-06T07:26:27.700175+0000","last_clean":"2024-03-06T07:26:27.700175+0000","last_became_active":"2024-03-06T07:26:26.689060+0000","last_became_peered":"2024-03-06T07:26:26.689060+0000","last_unstale":"2024-03-06T07:26:27.700175+0000","last_undegraded":"2024-03-06T07:26:27.700175+0000","last_fullsized":"2024-03-06T07:26:27.700175+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:41.961151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"152'1","reported_seq":15,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853892+0000","last_change":"2024-03-06T07:26:25.675206+0000","last_active":"2024-03-06T07:26:27.853892+0000","last_peered":"2024-03-06T07:26:27.853892+0000","last_clean":"2024-03-06T07:26:27.853892+0000","last_became_active":"2024-03-06T07:26:25.674907+0000","last_became_peered":"2024-03-06T07:26:25.674907+0000","last_unstale":"2024-03-06T07:26:27.853892+0000","last_undegraded":"2024-03-06T07:26:27.853892+0000","last_fullsized":"2024-03-06T07:26:27.853892+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:33.228276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854360+0000","last_change":"2024-03-06T07:26:17.637676+0000","last_active":"2024-03-06T07:26:27.854360+0000","last_peered":"2024-03-06T07:26:27.854360+0000","last_clean":"2024-03-06T07:26:27.854360+0000","last_became_active":"2024-03-06T07:26:17.637375+0000","last_became_peered":"2024-03-06T07:26:17.637375+0000","last_unstale":"2024-03-06T07:26:27.854360+0000","last_undegraded":"2024-03-06T07:26:27.854360+0000","last_fullsized":"2024-03-06T07:26:27.854360+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:18.556057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854616+0000","last_change":"2024-03-06T07:26:16.620202+0000","last_active":"2024-03-06T07:26:27.854616+0000","last_peered":"2024-03-06T07:26:27.854616+0000","last_clean":"2024-03-06T07:26:27.854616+0000","last_became_active":"2024-03-06T07:26:16.619897+0000","last_became_peered":"2024-03-06T07:26:16.619897+0000","last_unstale":"2024-03-06T07:26:27.854616+0000","last_undegraded":"2024-03-06T07:26:27.854616+0000","last_fullsized":"2024-03-06T07:26:27.854616+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:48.709582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854090+0000","last_change":"2024-03-06T07:26:26.704246+0000","last_active":"2024-03-06T07:26:27.854090+0000","last_peered":"2024-03-06T07:26:27.854090+0000","last_clean":"2024-03-06T07:26:27.854090+0000","last_became_active":"2024-03-06T07:26:26.703980+0000","last_became_peered":"2024-03-06T07:26:26.703980+0000","last_unstale":"2024-03-06T07:26:27.854090+0000","last_undegraded":"2024-03-06T07:26:27.854090+0000","last_fullsized":"2024-03-06T07:26:27.854090+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:22.369760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"152'2","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.680501+0000","last_change":"2024-03-06T07:26:25.676226+0000","last_active":"2024-03-06T07:26:27.680501+0000","last_peered":"2024-03-06T07:26:27.680501+0000","last_clean":"2024-03-06T07:26:27.680501+0000","last_became_active":"2024-03-06T07:26:25.675930+0000","last_became_peered":"2024-03-06T07:26:25.675930+0000","last_unstale":"2024-03-06T07:26:27.680501+0000","last_undegraded":"2024-03-06T07:26:27.680501+0000","last_fullsized":"2024-03-06T07:26:27.680501+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:46.405456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.700605+0000","last_change":"2024-03-06T07:26:17.625716+0000","last_active":"2024-03-06T07:26:27.700605+0000","last_peered":"2024-03-06T07:26:27.700605+0000","last_clean":"2024-03-06T07:26:27.700605+0000","last_became_active":"2024-03-06T07:26:17.625429+0000","last_became_peered":"2024-03-06T07:26:17.625429+0000","last_unstale":"2024-03-06T07:26:27.700605+0000","last_undegraded":"2024-03-06T07:26:27.700605+0000","last_fullsized":"2024-03-06T07:26:27.700605+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:38.596798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.679809+0000","last_change":"2024-03-06T07:26:16.620839+0000","last_active":"2024-03-06T07:26:27.679809+0000","last_peered":"2024-03-06T07:26:27.679809+0000","last_clean":"2024-03-06T07:26:27.679809+0000","last_became_active":"2024-03-06T07:26:16.620561+0000","last_became_peered":"2024-03-06T07:26:16.620561+0000","last_unstale":"2024-03-06T07:26:27.679809+0000","last_undegraded":"2024-03-06T07:26:27.679809+0000","last_fullsized":"2024-03-06T07:26:27.679809+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:26.394208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.853778+0000","last_change":"2024-03-06T07:26:26.693843+0000","last_active":"2024-03-06T07:26:27.853778+0000","last_peered":"2024-03-06T07:26:27.853778+0000","last_clean":"2024-03-06T07:26:27.853778+0000","last_became_active":"2024-03-06T07:26:26.693548+0000","last_became_peered":"2024-03-06T07:26:26.693548+0000","last_unstale":"2024-03-06T07:26:27.853778+0000","last_undegraded":"2024-03-06T07:26:27.853778+0000","last_fullsized":"2024-03-06T07:26:27.853778+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:25.656216+0000","last_clean_scrub_stamp":"2024-03-06T07:26:25.656216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:24.708532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.679801+0000","last_change":"2024-03-06T07:26:25.675948+0000","last_active":"2024-03-06T07:26:27.679801+0000","last_peered":"2024-03-06T07:26:27.679801+0000","last_clean":"2024-03-06T07:26:27.679801+0000","last_became_active":"2024-03-06T07:26:25.675645+0000","last_became_peered":"2024-03-06T07:26:25.675645+0000","last_unstale":"2024-03-06T07:26:27.679801+0000","last_undegraded":"2024-03-06T07:26:27.679801+0000","last_fullsized":"2024-03-06T07:26:27.679801+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.654657+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.654657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:52.418568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.684466+0000","last_change":"2024-03-06T07:26:17.632976+0000","last_active":"2024-03-06T07:26:27.684466+0000","last_peered":"2024-03-06T07:26:27.684466+0000","last_clean":"2024-03-06T07:26:27.684466+0000","last_became_active":"2024-03-06T07:26:17.632721+0000","last_became_peered":"2024-03-06T07:26:17.632721+0000","last_unstale":"2024-03-06T07:26:27.684466+0000","last_undegraded":"2024-03-06T07:26:27.684466+0000","last_fullsized":"2024-03-06T07:26:27.684466+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.599836+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.599836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:56.134884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.854445+0000","last_change":"2024-03-06T07:26:16.626430+0000","last_active":"2024-03-06T07:26:27.854445+0000","last_peered":"2024-03-06T07:26:27.854445+0000","last_clean":"2024-03-06T07:26:27.854445+0000","last_became_active":"2024-03-06T07:26:16.625971+0000","last_became_peered":"2024-03-06T07:26:16.625971+0000","last_unstale":"2024-03-06T07:26:27.854445+0000","last_undegraded":"2024-03-06T07:26:27.854445+0000","last_fullsized":"2024-03-06T07:26:27.854445+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.595702+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.595702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:28.109173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":425,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.683924+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:26:27.683924+0000","last_peered":"2024-03-06T07:26:27.683924+0000","last_clean":"2024-03-06T07:26:27.683924+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:26:27.683924+0000","last_undegraded":"2024-03-06T07:26:27.683924+0000","last_fullsized":"2024-03-06T07:26:27.683924+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574926,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33128,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338712,"statfs":{"total":96636764160,"available":96602841088,"internally_reserved":0,"allocated":1990656,"data_stored":1526495,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574925,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":1929216,"data_stored":936197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574925,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33088,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338752,"statfs":{"total":96636764160,"available":96602882048,"internally_reserved":0,"allocated":1949696,"data_stored":937885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574926,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1957888,"data_stored":938537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574925,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33092,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338748,"statfs":{"total":96636764160,"available":96602877952,"internally_reserved":0,"allocated":1953792,"data_stored":938503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574925,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":1929216,"data_stored":936709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":1937408,"data_stored":937739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33156,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338684,"statfs":{"total":96636764160,"available":96602812416,"internally_reserved":0,"allocated":2015232,"data_stored":1527327,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:34.701 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:26:34.701 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:34.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.843+0000 7ffb4bf55640 1 Processor -- start 2024-03-06T07:26:34.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.844+0000 7ffb4bf55640 1 -- start start 2024-03-06T07:26:34.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.844+0000 7ffb4bf55640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffb44150520 0x7ffb441543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.844+0000 7ffb4bf55640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb44150900 con 0x7ffb44070c80 2024-03-06T07:26:34.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.844+0000 7ffb4bf55640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb44071790 con 0x7ffb440c32f0 2024-03-06T07:26:34.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.844+0000 7ffb4bf55640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb44153ed0 con 0x7ffb44150520 2024-03-06T07:26:34.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.845+0000 7ffb49cf1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffb44150520 0x7ffb441543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.845+0000 7ffb49cf1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffb44150520 0x7ffb441543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38598/0 (socket says 172.21.15.1:38598) 2024-03-06T07:26:34.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.845+0000 7ffb4a4f2640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffb44070c80 0x7ffb4415d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37256/0 (socket says 172.21.15.1:37256) 2024-03-06T07:26:34.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.845+0000 7ffb49cf1640 1 -- 172.21.15.1:0/4212880239 learned_addr learned my addr 172.21.15.1:0/4212880239 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:34.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.845+0000 7ffb49cf1640 1 -- 172.21.15.1:0/4212880239 >> v1:172.21.15.98:6789/0 conn(0x7ffb440c32f0 legacy=0x7ffb441537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:34.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.845+0000 7ffb49cf1640 1 -- 172.21.15.1:0/4212880239 >> v1:172.21.15.1:6789/0 conn(0x7ffb44070c80 legacy=0x7ffb4415d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.845+0000 7ffb49cf1640 1 -- 172.21.15.1:0/4212880239 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb44157050 con 0x7ffb44150520 2024-03-06T07:26:34.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.845+0000 7ffb48cef640 1 -- 172.21.15.1:0/4212880239 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100707984 0 0) 0x7ffb44150900 con 0x7ffb44070c80 2024-03-06T07:26:34.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.846+0000 7ffb49cf1640 1 --2- 172.21.15.1:0/4212880239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb44150520 0x7ffb441543a0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7ffb40009870 tx=0x7ffb4002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3fb046559d688f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:34.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.846+0000 7ffb48cef640 1 -- 172.21.15.1:0/4212880239 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb40034070 con 0x7ffb44150520 2024-03-06T07:26:34.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.846+0000 7ffb48cef640 1 -- 172.21.15.1:0/4212880239 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb40030860 con 0x7ffb44150520 2024-03-06T07:26:34.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.846+0000 7ffb48cef640 1 -- 172.21.15.1:0/4212880239 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb40030c30 con 0x7ffb44150520 2024-03-06T07:26:34.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.846+0000 7ffb4bf55640 1 -- 172.21.15.1:0/4212880239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb44150520 msgr2=0x7ffb441543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.846+0000 7ffb4bf55640 1 --2- 172.21.15.1:0/4212880239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb44150520 0x7ffb441543a0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7ffb40009870 tx=0x7ffb4002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.846+0000 7ffb4bf55640 1 -- 172.21.15.1:0/4212880239 shutdown_connections 2024-03-06T07:26:34.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.846+0000 7ffb4bf55640 1 --2- 172.21.15.1:0/4212880239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb44150520 0x7ffb441543a0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.846+0000 7ffb4bf55640 1 -- 172.21.15.1:0/4212880239 >> 172.21.15.1:0/4212880239 conn(0x7ffb440c88f0 msgr2=0x7ffb440c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:34.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.846+0000 7ffb4bf55640 1 -- 172.21.15.1:0/4212880239 shutdown_connections 2024-03-06T07:26:34.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.847+0000 7ffb4bf55640 1 -- 172.21.15.1:0/4212880239 wait complete. 2024-03-06T07:26:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.848+0000 7ffb4bf55640 1 Processor -- start 2024-03-06T07:26:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.848+0000 7ffb4bf55640 1 -- start start 2024-03-06T07:26:34.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.848+0000 7ffb4bf55640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb44070c80 0x7ffb44147b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.848+0000 7ffb4bf55640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb440c32f0 0x7ffb441ebbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.848+0000 7ffb4bf55640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb44150520 0x7ffb441ddee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.848+0000 7ffb4bf55640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb44157330 con 0x7ffb44070c80 2024-03-06T07:26:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.848+0000 7ffb4bf55640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb441532a0 con 0x7ffb44150520 2024-03-06T07:26:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.848+0000 7ffb4bf55640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb440c7ec0 con 0x7ffb440c32f0 2024-03-06T07:26:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.848+0000 7ffb49cf1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb44070c80 0x7ffb44147b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.848+0000 7ffb49cf1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb44070c80 0x7ffb44147b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38604/0 (socket says 172.21.15.1:38604) 2024-03-06T07:26:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.848+0000 7ffb49cf1640 1 -- 172.21.15.1:0/149134868 learned_addr learned my addr 172.21.15.1:0/149134868 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.849+0000 7ffb494f0640 1 --2- 172.21.15.1:0/149134868 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb440c32f0 0x7ffb441ebbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.849+0000 7ffb4a4f2640 1 --2- 172.21.15.1:0/149134868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb44150520 0x7ffb441ddee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.849+0000 7ffb49cf1640 1 -- 172.21.15.1:0/149134868 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb440c32f0 msgr2=0x7ffb441ebbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.849+0000 7ffb49cf1640 1 --2- 172.21.15.1:0/149134868 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb440c32f0 0x7ffb441ebbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.849+0000 7ffb49cf1640 1 -- 172.21.15.1:0/149134868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb44150520 msgr2=0x7ffb441ddee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.849+0000 7ffb49cf1640 1 --2- 172.21.15.1:0/149134868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb44150520 0x7ffb441ddee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.849+0000 7ffb49cf1640 1 -- 172.21.15.1:0/149134868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb441ec5e0 con 0x7ffb44070c80 2024-03-06T07:26:34.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.849+0000 7ffb49cf1640 1 --2- 172.21.15.1:0/149134868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb44070c80 0x7ffb44147b00 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7ffb40009870 tx=0x7ffb400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:34.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.849+0000 7ffb367fc640 1 -- 172.21.15.1:0/149134868 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb40034040 con 0x7ffb44070c80 2024-03-06T07:26:34.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.849+0000 7ffb367fc640 1 -- 172.21.15.1:0/149134868 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb40039d90 con 0x7ffb44070c80 2024-03-06T07:26:34.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.849+0000 7ffb4bf55640 1 -- 172.21.15.1:0/149134868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb44146050 con 0x7ffb44070c80 2024-03-06T07:26:34.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.849+0000 7ffb367fc640 1 -- 172.21.15.1:0/149134868 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb40043610 con 0x7ffb44070c80 2024-03-06T07:26:34.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.849+0000 7ffb4bf55640 1 -- 172.21.15.1:0/149134868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb441ec130 con 0x7ffb44070c80 2024-03-06T07:26:34.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.850+0000 7ffb4bf55640 1 -- 172.21.15.1:0/149134868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb441ec360 con 0x7ffb44070c80 2024-03-06T07:26:34.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.851+0000 7ffb367fc640 1 -- 172.21.15.1:0/149134868 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb4004a060 con 0x7ffb44070c80 2024-03-06T07:26:34.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.851+0000 7ffb367fc640 1 --2- 172.21.15.1:0/149134868 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb2806b4d0 0x7ffb2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.851+0000 7ffb367fc640 1 -- 172.21.15.1:0/149134868 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7ffb400ac890 con 0x7ffb44070c80 2024-03-06T07:26:34.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.851+0000 7ffb494f0640 1 --2- 172.21.15.1:0/149134868 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb2806b4d0 0x7ffb2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.852+0000 7ffb494f0640 1 --2- 172.21.15.1:0/149134868 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb2806b4d0 0x7ffb2806d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ffb441dfe90 tx=0x7ffb38009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:34.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:34.855+0000 7ffb367fc640 1 -- 172.21.15.1:0/149134868 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb441ec360 con 0x7ffb44070c80 2024-03-06T07:26:35.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.075+0000 7ffb4bf55640 1 -- 172.21.15.1:0/149134868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb440686e0 con 0x7ffb44070c80 2024-03-06T07:26:35.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.076+0000 7ffb367fc640 1 -- 172.21.15.1:0/149134868 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+6260 (secure 0 0 0) 0x7ffb4003e070 con 0x7ffb44070c80 2024-03-06T07:26:35.078 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:35.078 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":140,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6989,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2360179300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2360179300},{"type":"v1","addr":"172.21.15.98:6837","nonce":2360179300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":7774,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3360241462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3360241462},{"type":"v1","addr":"172.21.15.98:6836","nonce":3360241462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:16.977128+0000","modified":"2024-03-06T07:26:18.975595+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8823},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8823":{"gid":8823,"name":"a","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/67091521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":67091521},{"type":"v1","addr":"172.21.15.1:6835","nonce":67091521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8823,"qdb_cluster":[8823]},"id":11},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:26.024256+0000","modified":"2024-03-06T07:26:28.060419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8820},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8820":{"gid":8820,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":5,"addr":"172.21.15.1:6833/1089548586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1089548586},{"type":"v1","addr":"172.21.15.1:6833","nonce":1089548586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8820,"qdb_cluster":[8820]},"id":12}]} 2024-03-06T07:26:35.078 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 140 2024-03-06T07:26:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.078+0000 7ffb4bf55640 1 -- 172.21.15.1:0/149134868 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb2806b4d0 msgr2=0x7ffb2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.078+0000 7ffb4bf55640 1 --2- 172.21.15.1:0/149134868 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb2806b4d0 0x7ffb2806d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ffb441dfe90 tx=0x7ffb38009040 comp rx=0 tx=0).stop 2024-03-06T07:26:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.078+0000 7ffb4bf55640 1 -- 172.21.15.1:0/149134868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb44070c80 msgr2=0x7ffb44147b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.078+0000 7ffb4bf55640 1 --2- 172.21.15.1:0/149134868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb44070c80 0x7ffb44147b00 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7ffb40009870 tx=0x7ffb400046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.078+0000 7ffb4bf55640 1 -- 172.21.15.1:0/149134868 shutdown_connections 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.078+0000 7ffb4bf55640 1 --2- 172.21.15.1:0/149134868 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb2806b4d0 0x7ffb2806d990 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.078+0000 7ffb4bf55640 1 --2- 172.21.15.1:0/149134868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb44150520 0x7ffb441ddee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.078+0000 7ffb4bf55640 1 --2- 172.21.15.1:0/149134868 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb440c32f0 0x7ffb441ebbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.078+0000 7ffb4bf55640 1 --2- 172.21.15.1:0/149134868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb44070c80 0x7ffb44147b00 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.078+0000 7ffb4bf55640 1 -- 172.21.15.1:0/149134868 >> 172.21.15.1:0/149134868 conn(0x7ffb440c88f0 msgr2=0x7ffb440c6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.078+0000 7ffb4bf55640 1 -- 172.21.15.1:0/149134868 shutdown_connections 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.078+0000 7ffb4bf55640 1 -- 172.21.15.1:0/149134868 wait complete. 2024-03-06T07:26:35.091 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 140, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:26:16.977128+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 137, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8823": { "addr": "172.21.15.1:6835/67091521", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6834", "nonce": 67091521, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 67091521, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8823, "incarnation": 135, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-03-06T07:26:18.975595+0000", "qdb_cluster": [ 8823 ], "qdb_leader": 8823, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8823 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:26:26.024256+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 140, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_8820": { "addr": "172.21.15.1:6833/1089548586", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 1089548586, "type": "v2" }, { "addr": "172.21.15.1:6833", "nonce": 1089548586, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8820, "incarnation": 138, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-03-06T07:26:28.060419+0000", "qdb_cluster": [ 8820 ], "qdb_leader": 8820, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8820 } } } ], "standbys": [ { "addr": "172.21.15.98:6837/2360179300", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 2360179300, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2360179300, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6989, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6836/3360241462", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 3360241462, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 3360241462, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7774, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:26:35.091 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:26:35.092 INFO:tasks.cephfs.test_failover:current = [{'gid': 8823, 'name': 'a', 'rank': 0, 'incarnation': 135, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/67091521', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 67091521}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 67091521}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6989, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/2360179300', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 2360179300}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2360179300}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 134}, {'gid': 8820, 'name': 'c', 'rank': 0, 'incarnation': 138, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.1:6833/1089548586', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1089548586}, {'type': 'v1', 'addr': '172.21.15.1:6833', 'nonce': 1089548586}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7774, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/3360241462', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 3360241462}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 3360241462}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 134}] 2024-03-06T07:26:35.092 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:35.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.192+0000 7f7102b20640 1 Processor -- start 2024-03-06T07:26:35.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.192+0000 7f7102b20640 1 -- start start 2024-03-06T07:26:35.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.193+0000 7f7102b20640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70fc0c6b10 0x7f70fc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.193+0000 7f7102b20640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70fc0c2fc0 con 0x7f70fc152650 2024-03-06T07:26:35.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.193+0000 7f7102b20640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70fc158da0 con 0x7f70fc0bf9d0 2024-03-06T07:26:35.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.193+0000 7f7102b20640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc155800 con 0x7f70fc0c6b10 2024-03-06T07:26:35.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.193+0000 7f71008bc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f70fc152650 0x7f70fc158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37260/0 (socket says 172.21.15.1:37260) 2024-03-06T07:26:35.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.194+0000 7f71008bc640 1 -- 172.21.15.1:0/3835484337 learned_addr learned my addr 172.21.15.1:0/3835484337 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:35.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.194+0000 7f70fbfff640 1 --2- 172.21.15.1:0/3835484337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70fc0c6b10 0x7f70fc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.194+0000 7f70fb7fe640 1 -- 172.21.15.1:0/3835484337 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148244306 0 0) 0x7f70fc0c2fc0 con 0x7f70fc152650 2024-03-06T07:26:35.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.194+0000 7f70fb7fe640 1 -- 172.21.15.1:0/3835484337 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70ec003520 con 0x7f70fc152650 2024-03-06T07:26:35.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.194+0000 7f70fb7fe640 1 -- 172.21.15.1:0/3835484337 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206712688 0 0) 0x7f70fc158da0 con 0x7f70fc0bf9d0 2024-03-06T07:26:35.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.194+0000 7f70fbfff640 1 -- 172.21.15.1:0/3835484337 >> v1:172.21.15.98:6789/0 conn(0x7f70fc0bf9d0 legacy=0x7f70fc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.194+0000 7f70fbfff640 1 -- 172.21.15.1:0/3835484337 >> v1:172.21.15.1:6789/0 conn(0x7f70fc152650 legacy=0x7f70fc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.194+0000 7f70fbfff640 1 -- 172.21.15.1:0/3835484337 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc0c6690 con 0x7f70fc0c6b10 2024-03-06T07:26:35.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.195+0000 7f70fbfff640 1 --2- 172.21.15.1:0/3835484337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70fc0c6b10 0x7f70fc155c90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f70e80097d0 tx=0x7f70e802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c468565eaa5eacc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.195+0000 7f70fb7fe640 1 -- 172.21.15.1:0/3835484337 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8035020 con 0x7f70fc0c6b10 2024-03-06T07:26:35.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.195+0000 7f70fb7fe640 1 -- 172.21.15.1:0/3835484337 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e8031800 con 0x7f70fc0c6b10 2024-03-06T07:26:35.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.195+0000 7f70fb7fe640 1 -- 172.21.15.1:0/3835484337 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8031bd0 con 0x7f70fc0c6b10 2024-03-06T07:26:35.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.195+0000 7f7102b20640 1 -- 172.21.15.1:0/3835484337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70fc0c6b10 msgr2=0x7f70fc155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.195+0000 7f7102b20640 1 --2- 172.21.15.1:0/3835484337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70fc0c6b10 0x7f70fc155c90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f70e80097d0 tx=0x7f70e802b8a0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.196+0000 7f7102b20640 1 -- 172.21.15.1:0/3835484337 shutdown_connections 2024-03-06T07:26:35.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.196+0000 7f7102b20640 1 --2- 172.21.15.1:0/3835484337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70fc0c6b10 0x7f70fc155c90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.196+0000 7f7102b20640 1 -- 172.21.15.1:0/3835484337 >> 172.21.15.1:0/3835484337 conn(0x7f70fc066ee0 msgr2=0x7f70fc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:35.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.196+0000 7f7102b20640 1 -- 172.21.15.1:0/3835484337 shutdown_connections 2024-03-06T07:26:35.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.196+0000 7f7102b20640 1 -- 172.21.15.1:0/3835484337 wait complete. 2024-03-06T07:26:35.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.197+0000 7f7102b20640 1 Processor -- start 2024-03-06T07:26:35.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.197+0000 7f7102b20640 1 -- start start 2024-03-06T07:26:35.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.197+0000 7f7102b20640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70fc0bf9d0 0x7f70fc14cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.197+0000 7f7102b20640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70fc152650 0x7f70fc14fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.197+0000 7f7102b20640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70fc14dab0 0x7f70fc0c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.197+0000 7f7102b20640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc1551b0 con 0x7f70fc152650 2024-03-06T07:26:35.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.197+0000 7f7102b20640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc0c6fa0 con 0x7f70fc14dab0 2024-03-06T07:26:35.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.197+0000 7f7102b20640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc155490 con 0x7f70fc0bf9d0 2024-03-06T07:26:35.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f70fbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70fc152650 0x7f70fc14fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f70fbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70fc152650 0x7f70fc14fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38616/0 (socket says 172.21.15.1:38616) 2024-03-06T07:26:35.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f70fbfff640 1 -- 172.21.15.1:0/2599458345 learned_addr learned my addr 172.21.15.1:0/2599458345 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:35.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f71008bc640 1 --2- 172.21.15.1:0/2599458345 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70fc0bf9d0 0x7f70fc14cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f71010bd640 1 --2- 172.21.15.1:0/2599458345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70fc14dab0 0x7f70fc0c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f70fbfff640 1 -- 172.21.15.1:0/2599458345 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70fc0bf9d0 msgr2=0x7f70fc14cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f70fbfff640 1 --2- 172.21.15.1:0/2599458345 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70fc0bf9d0 0x7f70fc14cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f70fbfff640 1 -- 172.21.15.1:0/2599458345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70fc14dab0 msgr2=0x7f70fc0c9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f70fbfff640 1 --2- 172.21.15.1:0/2599458345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70fc14dab0 0x7f70fc0c9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f70fbfff640 1 -- 172.21.15.1:0/2599458345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc1ef270 con 0x7f70fc152650 2024-03-06T07:26:35.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f71008bc640 1 --2- 172.21.15.1:0/2599458345 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70fc0bf9d0 0x7f70fc14cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:35.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f71010bd640 1 --2- 172.21.15.1:0/2599458345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70fc14dab0 0x7f70fc0c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:35.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f70fbfff640 1 --2- 172.21.15.1:0/2599458345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70fc152650 0x7f70fc14fcd0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f70e80097d0 tx=0x7f70e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f70f97fa640 1 -- 172.21.15.1:0/2599458345 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8041070 con 0x7f70fc152650 2024-03-06T07:26:35.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f70f97fa640 1 -- 172.21.15.1:0/2599458345 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e803ac80 con 0x7f70fc152650 2024-03-06T07:26:35.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f7102b20640 1 -- 172.21.15.1:0/2599458345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70fc0ca600 con 0x7f70fc152650 2024-03-06T07:26:35.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f70f97fa640 1 -- 172.21.15.1:0/2599458345 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8035040 con 0x7f70fc152650 2024-03-06T07:26:35.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.198+0000 7f7102b20640 1 -- 172.21.15.1:0/2599458345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70fc150210 con 0x7f70fc152650 2024-03-06T07:26:35.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.199+0000 7f7102b20640 1 -- 172.21.15.1:0/2599458345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70fc0c2f50 con 0x7f70fc152650 2024-03-06T07:26:35.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.200+0000 7f70f97fa640 1 -- 172.21.15.1:0/2599458345 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70e8038020 con 0x7f70fc152650 2024-03-06T07:26:35.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.201+0000 7f70f97fa640 1 --2- 172.21.15.1:0/2599458345 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f70e006b4d0 0x7f70e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.201+0000 7f70f97fa640 1 -- 172.21.15.1:0/2599458345 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f70e80ac340 con 0x7f70fc152650 2024-03-06T07:26:35.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.201+0000 7f71008bc640 1 --2- 172.21.15.1:0/2599458345 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f70e006b4d0 0x7f70e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.202+0000 7f71008bc640 1 --2- 172.21.15.1:0/2599458345 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f70e006b4d0 0x7f70e006d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f70f0007fd0 tx=0x7f70f0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.205+0000 7f70f97fa640 1 -- 172.21.15.1:0/2599458345 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f70fc0c2f50 con 0x7f70fc152650 2024-03-06T07:26:35.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.423+0000 7f7102b20640 1 -- 172.21.15.1:0/2599458345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70fc067fd0 con 0x7f70fc152650 2024-03-06T07:26:35.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.423+0000 7f70f97fa640 1 -- 172.21.15.1:0/2599458345 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+6260 (secure 0 0 0) 0x7f70e807acd0 con 0x7f70fc152650 2024-03-06T07:26:35.425 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:35.425 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":140,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6989,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2360179300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2360179300},{"type":"v1","addr":"172.21.15.98:6837","nonce":2360179300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":7774,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3360241462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3360241462},{"type":"v1","addr":"172.21.15.98:6836","nonce":3360241462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:16.977128+0000","modified":"2024-03-06T07:26:18.975595+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8823},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8823":{"gid":8823,"name":"a","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/67091521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":67091521},{"type":"v1","addr":"172.21.15.1:6835","nonce":67091521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8823,"qdb_cluster":[8823]},"id":11},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:26.024256+0000","modified":"2024-03-06T07:26:28.060419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8820},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8820":{"gid":8820,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":5,"addr":"172.21.15.1:6833/1089548586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1089548586},{"type":"v1","addr":"172.21.15.1:6833","nonce":1089548586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8820,"qdb_cluster":[8820]},"id":12}]} 2024-03-06T07:26:35.426 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 140 2024-03-06T07:26:35.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.425+0000 7f7102b20640 1 -- 172.21.15.1:0/2599458345 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f70e006b4d0 msgr2=0x7f70e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.425+0000 7f7102b20640 1 --2- 172.21.15.1:0/2599458345 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f70e006b4d0 0x7f70e006d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f70f0007fd0 tx=0x7f70f0009040 comp rx=0 tx=0).stop 2024-03-06T07:26:35.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.425+0000 7f7102b20640 1 -- 172.21.15.1:0/2599458345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70fc152650 msgr2=0x7f70fc14fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.425+0000 7f7102b20640 1 --2- 172.21.15.1:0/2599458345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70fc152650 0x7f70fc14fcd0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f70e80097d0 tx=0x7f70e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.425+0000 7f7102b20640 1 -- 172.21.15.1:0/2599458345 shutdown_connections 2024-03-06T07:26:35.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.425+0000 7f7102b20640 1 --2- 172.21.15.1:0/2599458345 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f70e006b4d0 0x7f70e006d990 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.425+0000 7f7102b20640 1 --2- 172.21.15.1:0/2599458345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70fc14dab0 0x7f70fc0c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.425+0000 7f7102b20640 1 --2- 172.21.15.1:0/2599458345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70fc152650 0x7f70fc14fcd0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.425+0000 7f7102b20640 1 --2- 172.21.15.1:0/2599458345 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70fc0bf9d0 0x7f70fc14cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.425+0000 7f7102b20640 1 -- 172.21.15.1:0/2599458345 >> 172.21.15.1:0/2599458345 conn(0x7f70fc066ee0 msgr2=0x7f70fc0c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:35.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.426+0000 7f7102b20640 1 -- 172.21.15.1:0/2599458345 shutdown_connections 2024-03-06T07:26:35.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.426+0000 7f7102b20640 1 -- 172.21.15.1:0/2599458345 wait complete. 2024-03-06T07:26:35.439 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 140, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:26:16.977128+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 137, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8823": { "addr": "172.21.15.1:6835/67091521", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6834", "nonce": 67091521, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 67091521, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8823, "incarnation": 135, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-03-06T07:26:18.975595+0000", "qdb_cluster": [ 8823 ], "qdb_leader": 8823, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8823 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:26:26.024256+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 140, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_8820": { "addr": "172.21.15.1:6833/1089548586", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 1089548586, "type": "v2" }, { "addr": "172.21.15.1:6833", "nonce": 1089548586, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8820, "incarnation": 138, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-03-06T07:26:28.060419+0000", "qdb_cluster": [ 8820 ], "qdb_leader": 8820, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8820 } } } ], "standbys": [ { "addr": "172.21.15.98:6837/2360179300", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 2360179300, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2360179300, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6989, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6836/3360241462", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 3360241462, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 3360241462, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7774, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:26:35.439 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:26:35.439 INFO:tasks.cephfs.test_failover:current = [{'gid': 8823, 'name': 'a', 'rank': 0, 'incarnation': 135, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/67091521', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 67091521}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 67091521}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6989, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6837/2360179300', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 2360179300}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2360179300}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 134}, {'gid': 8820, 'name': 'c', 'rank': 0, 'incarnation': 138, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.1:6833/1089548586', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1089548586}, {'type': 'v1', 'addr': '172.21.15.1:6833', 'nonce': 1089548586}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7774, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.98:6836/3360241462', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 3360241462}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 3360241462}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 134}] 2024-03-06T07:26:35.440 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs2 2024-03-06T07:26:35.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.542+0000 7f357a99a640 1 Processor -- start 2024-03-06T07:26:35.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.542+0000 7f357a99a640 1 -- start start 2024-03-06T07:26:35.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.543+0000 7f357a99a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35740c4560 0x7f3574155c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.543+0000 7f357a99a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f35740c39c0 0x7f35741561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.543+0000 7f357a99a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35740c7420 con 0x7f35740c4940 2024-03-06T07:26:35.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.543+0000 7f357a99a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35741566f0 con 0x7f35740c39c0 2024-03-06T07:26:35.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.543+0000 7f357a99a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35740ab760 con 0x7f35740c4560 2024-03-06T07:26:35.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.543+0000 7f3578f37640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f35740c4940 0x7f35740ac220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37272/0 (socket says 172.21.15.1:37272) 2024-03-06T07:26:35.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.543+0000 7f3578f37640 1 -- 172.21.15.1:0/3707582009 learned_addr learned my addr 172.21.15.1:0/3707582009 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:35.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.544+0000 7f3573fff640 1 --2- 172.21.15.1:0/3707582009 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35740c4560 0x7f3574155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.544+0000 7f35737fe640 1 --2- 172.21.15.1:0/3707582009 >> v2:172.21.15.98:3300/0 conn(0x7f35740c39c0 0x7f35741561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.544+0000 7f3572ffd640 1 -- 172.21.15.1:0/3707582009 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403265583 0 0) 0x7f35740c7420 con 0x7f35740c4940 2024-03-06T07:26:35.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.544+0000 7f3572ffd640 1 -- 172.21.15.1:0/3707582009 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3560003520 con 0x7f35740c4940 2024-03-06T07:26:35.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.544+0000 7f3573fff640 1 -- 172.21.15.1:0/3707582009 >> v2:172.21.15.98:3300/0 conn(0x7f35740c39c0 msgr2=0x7f35741561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:35.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.544+0000 7f3573fff640 1 --2- 172.21.15.1:0/3707582009 >> v2:172.21.15.98:3300/0 conn(0x7f35740c39c0 0x7f35741561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.544+0000 7f3573fff640 1 -- 172.21.15.1:0/3707582009 >> v1:172.21.15.1:6789/0 conn(0x7f35740c4940 legacy=0x7f35740ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.544+0000 7f3572ffd640 1 -- 172.21.15.1:0/3707582009 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2152146194 0 0) 0x7f3560003520 con 0x7f35740c4940 2024-03-06T07:26:35.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.544+0000 7f3573fff640 1 -- 172.21.15.1:0/3707582009 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35740c76c0 con 0x7f35740c4560 2024-03-06T07:26:35.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.545+0000 7f3573fff640 1 --2- 172.21.15.1:0/3707582009 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35740c4560 0x7f3574155c70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f35640097a0 tx=0x7f356402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd7ba27e89e78a36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.545+0000 7f3572ffd640 1 -- 172.21.15.1:0/3707582009 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3564037070 con 0x7f35740c4560 2024-03-06T07:26:35.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.545+0000 7f3572ffd640 1 -- 172.21.15.1:0/3707582009 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3564030510 con 0x7f35740c4560 2024-03-06T07:26:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.545+0000 7f3572ffd640 1 -- 172.21.15.1:0/3707582009 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c0019f0 con 0x7f35740c4560 2024-03-06T07:26:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.546+0000 7f357a99a640 1 -- 172.21.15.1:0/3707582009 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35740c4560 msgr2=0x7f3574155c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.546+0000 7f357a99a640 1 --2- 172.21.15.1:0/3707582009 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35740c4560 0x7f3574155c70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f35640097a0 tx=0x7f356402b790 comp rx=0 tx=0).stop 2024-03-06T07:26:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.546+0000 7f357a99a640 1 -- 172.21.15.1:0/3707582009 shutdown_connections 2024-03-06T07:26:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.546+0000 7f357a99a640 1 --2- 172.21.15.1:0/3707582009 >> v2:172.21.15.98:3300/0 conn(0x7f35740c39c0 0x7f35741561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.546+0000 7f357a99a640 1 --2- 172.21.15.1:0/3707582009 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35740c4560 0x7f3574155c70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.546+0000 7f357a99a640 1 -- 172.21.15.1:0/3707582009 >> 172.21.15.1:0/3707582009 conn(0x7f35740c7a50 msgr2=0x7f35740c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.546+0000 7f357a99a640 1 -- 172.21.15.1:0/3707582009 shutdown_connections 2024-03-06T07:26:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.546+0000 7f357a99a640 1 -- 172.21.15.1:0/3707582009 wait complete. 2024-03-06T07:26:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.547+0000 7f357a99a640 1 Processor -- start 2024-03-06T07:26:35.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.547+0000 7f357a99a640 1 -- start start 2024-03-06T07:26:35.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.548+0000 7f357a99a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35740c39c0 0x7f35740c1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.548+0000 7f357a99a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35740c4560 0x7f35740a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.548+0000 7f357a99a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35740c4940 0x7f35740a8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.548+0000 7f357a99a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357406e6d0 con 0x7f35740c4560 2024-03-06T07:26:35.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.548+0000 7f357a99a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35740abd00 con 0x7f35740c4940 2024-03-06T07:26:35.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.548+0000 7f357a99a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f357406e850 con 0x7f35740c39c0 2024-03-06T07:26:35.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.548+0000 7f35737fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35740c4560 0x7f35740a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.548+0000 7f35737fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35740c4560 0x7f35740a6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38626/0 (socket says 172.21.15.1:38626) 2024-03-06T07:26:35.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.548+0000 7f35737fe640 1 -- 172.21.15.1:0/875507000 learned_addr learned my addr 172.21.15.1:0/875507000 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:35.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.548+0000 7f3573fff640 1 --2- 172.21.15.1:0/875507000 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35740c39c0 0x7f35740c1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.548+0000 7f3578f37640 1 --2- 172.21.15.1:0/875507000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35740c4940 0x7f35740a8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.548+0000 7f35737fe640 1 -- 172.21.15.1:0/875507000 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35740c39c0 msgr2=0x7f35740c1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.548+0000 7f35737fe640 1 --2- 172.21.15.1:0/875507000 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35740c39c0 0x7f35740c1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.548+0000 7f35737fe640 1 -- 172.21.15.1:0/875507000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35740c4940 msgr2=0x7f35740a8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.548+0000 7f35737fe640 1 --2- 172.21.15.1:0/875507000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35740c4940 0x7f35740a8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.548+0000 7f35737fe640 1 -- 172.21.15.1:0/875507000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35741eaa00 con 0x7f35740c4560 2024-03-06T07:26:35.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.549+0000 7f35737fe640 1 --2- 172.21.15.1:0/875507000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35740c4560 0x7f35740a6410 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f355c00e980 tx=0x7f355c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.549+0000 7f3570ff9640 1 -- 172.21.15.1:0/875507000 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c017070 con 0x7f35740c4560 2024-03-06T07:26:35.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.549+0000 7f3570ff9640 1 -- 172.21.15.1:0/875507000 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c00ca00 con 0x7f35740c4560 2024-03-06T07:26:35.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.549+0000 7f3570ff9640 1 -- 172.21.15.1:0/875507000 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c00cdd0 con 0x7f35740c4560 2024-03-06T07:26:35.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.549+0000 7f357a99a640 1 -- 172.21.15.1:0/875507000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3574150e70 con 0x7f35740c4560 2024-03-06T07:26:35.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.549+0000 7f357a99a640 1 -- 172.21.15.1:0/875507000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35741e9f20 con 0x7f35740c4560 2024-03-06T07:26:35.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.551+0000 7f357a99a640 1 -- 172.21.15.1:0/875507000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35740686e0 con 0x7f35740c4560 2024-03-06T07:26:35.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.551+0000 7f3570ff9640 1 -- 172.21.15.1:0/875507000 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f355c01a070 con 0x7f35740c4560 2024-03-06T07:26:35.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.551+0000 7f3570ff9640 1 --2- 172.21.15.1:0/875507000 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f355406b4d0 0x7f355406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.552+0000 7f3570ff9640 1 -- 172.21.15.1:0/875507000 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f355c08a6a0 con 0x7f35740c4560 2024-03-06T07:26:35.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.552+0000 7f3573fff640 1 --2- 172.21.15.1:0/875507000 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f355406b4d0 0x7f355406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.552+0000 7f3573fff640 1 --2- 172.21.15.1:0/875507000 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f355406b4d0 0x7f355406d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f3564034000 tx=0x7f356402f000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.556+0000 7f3570ff9640 1 -- 172.21.15.1:0/875507000 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f355c0590d0 con 0x7f35740c4560 2024-03-06T07:26:35.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.875+0000 7f357a99a640 1 -- 172.21.15.1:0/875507000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f35740a9540 con 0x7f35740c4560 2024-03-06T07:26:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.878+0000 7f3570ff9640 1 -- 172.21.15.1:0/875507000 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v52)=0 v52) v1 ==== 117+0+0 (secure 0 0 0) 0x7f355c0589d0 con 0x7f35740c4560 2024-03-06T07:26:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.881+0000 7f357a99a640 1 -- 172.21.15.1:0/875507000 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f355406b4d0 msgr2=0x7f355406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.881+0000 7f357a99a640 1 --2- 172.21.15.1:0/875507000 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f355406b4d0 0x7f355406d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f3564034000 tx=0x7f356402f000 comp rx=0 tx=0).stop 2024-03-06T07:26:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.881+0000 7f357a99a640 1 -- 172.21.15.1:0/875507000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35740c4560 msgr2=0x7f35740a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.881+0000 7f357a99a640 1 --2- 172.21.15.1:0/875507000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35740c4560 0x7f35740a6410 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f355c00e980 tx=0x7f355c00ee40 comp rx=0 tx=0).stop 2024-03-06T07:26:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.881+0000 7f357a99a640 1 -- 172.21.15.1:0/875507000 shutdown_connections 2024-03-06T07:26:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.881+0000 7f357a99a640 1 --2- 172.21.15.1:0/875507000 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f355406b4d0 0x7f355406d990 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.881+0000 7f357a99a640 1 --2- 172.21.15.1:0/875507000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35740c4940 0x7f35740a8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.881+0000 7f357a99a640 1 --2- 172.21.15.1:0/875507000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35740c4560 0x7f35740a6410 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.881+0000 7f357a99a640 1 --2- 172.21.15.1:0/875507000 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35740c39c0 0x7f35740c1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.881+0000 7f357a99a640 1 -- 172.21.15.1:0/875507000 >> 172.21.15.1:0/875507000 conn(0x7f35740c7a50 msgr2=0x7f3574155100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.881+0000 7f357a99a640 1 -- 172.21.15.1:0/875507000 shutdown_connections 2024-03-06T07:26:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:35.881+0000 7f357a99a640 1 -- 172.21.15.1:0/875507000 wait complete. 2024-03-06T07:26:35.897 INFO:tasks.cephfs.test_failover:assigning affinity to cephfs2 for active mds (mds.c) 2024-03-06T07:26:35.898 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs2 2024-03-06T07:26:36.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.000+0000 7fd4ae1bc640 1 Processor -- start 2024-03-06T07:26:36.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.000+0000 7fd4ae1bc640 1 -- start start 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.001+0000 7fd4ae1bc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd4a80c3970 0x7fd4a80c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.001+0000 7fd4ae1bc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd4a8152840 0x7fd4a81553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.001+0000 7fd4ae1bc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4a80c5e40 con 0x7fd4a80c60b0 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.001+0000 7fd4ae1bc640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4a80c67f0 con 0x7fd4a8152840 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.001+0000 7fd4ae1bc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4a80c6a40 con 0x7fd4a80c3970 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.001+0000 7fd4a7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd4a8152840 0x7fd4a81553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.001+0000 7fd4a7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd4a8152840 0x7fd4a81553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38640/0 (socket says 172.21.15.1:38640) 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.001+0000 7fd4a7fff640 1 -- 172.21.15.1:0/3751127074 learned_addr learned my addr 172.21.15.1:0/3751127074 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.001+0000 7fd4a77fe640 1 --2- 172.21.15.1:0/3751127074 >> v2:172.21.15.98:3300/0 conn(0x7fd4a80c3970 0x7fd4a80c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.001+0000 7fd4a7fff640 1 -- 172.21.15.1:0/3751127074 >> v2:172.21.15.98:3300/0 conn(0x7fd4a80c3970 msgr2=0x7fd4a80c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.001+0000 7fd4a7fff640 1 --2- 172.21.15.1:0/3751127074 >> v2:172.21.15.98:3300/0 conn(0x7fd4a80c3970 0x7fd4a80c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.001+0000 7fd4a7fff640 1 -- 172.21.15.1:0/3751127074 >> v1:172.21.15.98:6789/0 conn(0x7fd4a80c60b0 legacy=0x7fd4a8152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.001+0000 7fd4a7fff640 1 -- 172.21.15.1:0/3751127074 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a80c6d20 con 0x7fd4a8152840 2024-03-06T07:26:36.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.002+0000 7fd4a7fff640 1 --2- 172.21.15.1:0/3751127074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a8152840 0x7fd4a81553d0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fd4a80c34c0 tx=0x7fd4a002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f41fd83e5ce04a1d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.002+0000 7fd4a67fc640 1 -- 172.21.15.1:0/3751127074 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a002cb20 con 0x7fd4a8152840 2024-03-06T07:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.002+0000 7fd4a67fc640 1 -- 172.21.15.1:0/3751127074 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a002ccc0 con 0x7fd4a8152840 2024-03-06T07:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.002+0000 7fd4a67fc640 1 -- 172.21.15.1:0/3751127074 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a0034610 con 0x7fd4a8152840 2024-03-06T07:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.003+0000 7fd4ae1bc640 1 -- 172.21.15.1:0/3751127074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a8152840 msgr2=0x7fd4a81553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.003+0000 7fd4ae1bc640 1 --2- 172.21.15.1:0/3751127074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a8152840 0x7fd4a81553d0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fd4a80c34c0 tx=0x7fd4a002bf80 comp rx=0 tx=0).stop 2024-03-06T07:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.003+0000 7fd4ae1bc640 1 -- 172.21.15.1:0/3751127074 shutdown_connections 2024-03-06T07:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.003+0000 7fd4ae1bc640 1 --2- 172.21.15.1:0/3751127074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a8152840 0x7fd4a81553d0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.003+0000 7fd4ae1bc640 1 --2- 172.21.15.1:0/3751127074 >> v2:172.21.15.98:3300/0 conn(0x7fd4a80c3970 0x7fd4a80c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.003+0000 7fd4ae1bc640 1 -- 172.21.15.1:0/3751127074 >> 172.21.15.1:0/3751127074 conn(0x7fd4a80c88f0 msgr2=0x7fd4a80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.003+0000 7fd4ae1bc640 1 -- 172.21.15.1:0/3751127074 shutdown_connections 2024-03-06T07:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.003+0000 7fd4ae1bc640 1 -- 172.21.15.1:0/3751127074 wait complete. 2024-03-06T07:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.004+0000 7fd4ae1bc640 1 Processor -- start 2024-03-06T07:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.004+0000 7fd4ae1bc640 1 -- start start 2024-03-06T07:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4ae1bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4a80c3970 0x7fd4a814d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4ae1bc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a80c60b0 0x7fd4a80a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4ae1bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4a8152840 0x7fd4a81eb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4ae1bc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a80c75c0 con 0x7fd4a80c60b0 2024-03-06T07:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4ae1bc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a80c72e0 con 0x7fd4a8152840 2024-03-06T07:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4ae1bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a80c3d50 con 0x7fd4a80c3970 2024-03-06T07:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4a6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a80c60b0 0x7fd4a80a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4a6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a80c60b0 0x7fd4a80a97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38654/0 (socket says 172.21.15.1:38654) 2024-03-06T07:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4a6ffd640 1 -- 172.21.15.1:0/70058878 learned_addr learned my addr 172.21.15.1:0/70058878 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4a77fe640 1 --2- 172.21.15.1:0/70058878 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4a80c3970 0x7fd4a814d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4a6ffd640 1 -- 172.21.15.1:0/70058878 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4a80c3970 msgr2=0x7fd4a814d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4a6ffd640 1 --2- 172.21.15.1:0/70058878 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4a80c3970 0x7fd4a814d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4a7fff640 1 --2- 172.21.15.1:0/70058878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4a8152840 0x7fd4a81eb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4a6ffd640 1 -- 172.21.15.1:0/70058878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4a8152840 msgr2=0x7fd4a81eb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4a6ffd640 1 --2- 172.21.15.1:0/70058878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4a8152840 0x7fd4a81eb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4a6ffd640 1 -- 172.21.15.1:0/70058878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a8070d80 con 0x7fd4a80c60b0 2024-03-06T07:26:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4a77fe640 1 --2- 172.21.15.1:0/70058878 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4a80c3970 0x7fd4a814d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4a6ffd640 1 --2- 172.21.15.1:0/70058878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a80c60b0 0x7fd4a80a97d0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fd49800c3d0 tx=0x7fd49800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd49ffff640 1 -- 172.21.15.1:0/70058878 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4980052d0 con 0x7fd4a80c60b0 2024-03-06T07:26:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd49ffff640 1 -- 172.21.15.1:0/70058878 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd498005490 con 0x7fd4a80c60b0 2024-03-06T07:26:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd49ffff640 1 -- 172.21.15.1:0/70058878 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd498007260 con 0x7fd4a80c60b0 2024-03-06T07:26:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.005+0000 7fd4ae1bc640 1 -- 172.21.15.1:0/70058878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a8070ff0 con 0x7fd4a80c60b0 2024-03-06T07:26:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.006+0000 7fd4ae1bc640 1 -- 172.21.15.1:0/70058878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a81eab40 con 0x7fd4a80c60b0 2024-03-06T07:26:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.006+0000 7fd4ae1bc640 1 -- 172.21.15.1:0/70058878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a80686e0 con 0x7fd4a80c60b0 2024-03-06T07:26:36.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.008+0000 7fd49ffff640 1 -- 172.21.15.1:0/70058878 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd498007400 con 0x7fd4a80c60b0 2024-03-06T07:26:36.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.008+0000 7fd49ffff640 1 --2- 172.21.15.1:0/70058878 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd48806b4d0 0x7fd48806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.008+0000 7fd49ffff640 1 -- 172.21.15.1:0/70058878 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fd498089b40 con 0x7fd4a80c60b0 2024-03-06T07:26:36.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.008+0000 7fd4a77fe640 1 --2- 172.21.15.1:0/70058878 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd48806b4d0 0x7fd48806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.009+0000 7fd4a77fe640 1 --2- 172.21.15.1:0/70058878 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd48806b4d0 0x7fd48806d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8156540 tx=0x7fd490008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.014+0000 7fd49ffff640 1 -- 172.21.15.1:0/70058878 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd498058570 con 0x7fd4a80c60b0 2024-03-06T07:26:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.282+0000 7fd4ae1bc640 1 -- 172.21.15.1:0/70058878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fd4a80c56e0 con 0x7fd4a80c60b0 2024-03-06T07:26:36.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.286+0000 7fd49ffff640 1 -- 172.21.15.1:0/70058878 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v53)=0 v53) v1 ==== 117+0+0 (secure 0 0 0) 0x7fd498057e70 con 0x7fd4a80c60b0 2024-03-06T07:26:36.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.288+0000 7fd4ae1bc640 1 -- 172.21.15.1:0/70058878 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd48806b4d0 msgr2=0x7fd48806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.288+0000 7fd4ae1bc640 1 --2- 172.21.15.1:0/70058878 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd48806b4d0 0x7fd48806d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8156540 tx=0x7fd490008040 comp rx=0 tx=0).stop 2024-03-06T07:26:36.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.288+0000 7fd4ae1bc640 1 -- 172.21.15.1:0/70058878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a80c60b0 msgr2=0x7fd4a80a97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.288+0000 7fd4ae1bc640 1 --2- 172.21.15.1:0/70058878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a80c60b0 0x7fd4a80a97d0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fd49800c3d0 tx=0x7fd49800c890 comp rx=0 tx=0).stop 2024-03-06T07:26:36.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.288+0000 7fd4ae1bc640 1 -- 172.21.15.1:0/70058878 shutdown_connections 2024-03-06T07:26:36.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.288+0000 7fd4ae1bc640 1 --2- 172.21.15.1:0/70058878 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd48806b4d0 0x7fd48806d990 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.288+0000 7fd4ae1bc640 1 --2- 172.21.15.1:0/70058878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4a8152840 0x7fd4a81eb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.288+0000 7fd4ae1bc640 1 --2- 172.21.15.1:0/70058878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a80c60b0 0x7fd4a80a97d0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.288+0000 7fd4ae1bc640 1 --2- 172.21.15.1:0/70058878 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4a80c3970 0x7fd4a814d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.288+0000 7fd4ae1bc640 1 -- 172.21.15.1:0/70058878 >> 172.21.15.1:0/70058878 conn(0x7fd4a80c88f0 msgr2=0x7fd4a80aa680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:36.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.288+0000 7fd4ae1bc640 1 -- 172.21.15.1:0/70058878 shutdown_connections 2024-03-06T07:26:36.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.288+0000 7fd4ae1bc640 1 -- 172.21.15.1:0/70058878 wait complete. 2024-03-06T07:26:36.304 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 11:0 2024-03-06T07:26:36.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.413+0000 7f3845ced640 1 Processor -- start 2024-03-06T07:26:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.413+0000 7f3845ced640 1 -- start start 2024-03-06T07:26:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f3845ced640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38400c3260 0x7f38400c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f3845ced640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38400c2b90 con 0x7f38400c8cd0 2024-03-06T07:26:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f3845ced640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3840156020 con 0x7f3840155b70 2024-03-06T07:26:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f3845ced640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38400c37b0 con 0x7f38400c3260 2024-03-06T07:26:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f38400c8cd0 0x7f38400c7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37278/0 (socket says 172.21.15.1:37278) 2024-03-06T07:26:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383f7fe640 1 -- 172.21.15.1:0/4198058247 learned_addr learned my addr 172.21.15.1:0/4198058247 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383effd640 1 --2- 172.21.15.1:0/4198058247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38400c3260 0x7f38400c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383e7fc640 1 -- 172.21.15.1:0/4198058247 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233412010 0 0) 0x7f38400c2b90 con 0x7f38400c8cd0 2024-03-06T07:26:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383e7fc640 1 -- 172.21.15.1:0/4198058247 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f382c003520 con 0x7f38400c8cd0 2024-03-06T07:26:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383e7fc640 1 -- 172.21.15.1:0/4198058247 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 681686170 0 0) 0x7f382c003520 con 0x7f38400c8cd0 2024-03-06T07:26:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383e7fc640 1 -- 172.21.15.1:0/4198058247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38400c3260 msgr2=0x7f38400c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383e7fc640 1 --2- 172.21.15.1:0/4198058247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38400c3260 0x7f38400c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383e7fc640 1 -- 172.21.15.1:0/4198058247 >> v1:172.21.15.98:6789/0 conn(0x7f3840155b70 legacy=0x7f3840159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383e7fc640 1 -- 172.21.15.1:0/4198058247 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38400c3a90 con 0x7f38400c8cd0 2024-03-06T07:26:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383e7fc640 1 -- 172.21.15.1:0/4198058247 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f3830001670 con 0x7f38400c8cd0 2024-03-06T07:26:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383e7fc640 1 -- 172.21.15.1:0/4198058247 >> v1:172.21.15.1:6789/0 conn(0x7f38400c8cd0 legacy=0x7f38400c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383e7fc640 1 --2- 172.21.15.1:0/4198058247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f382c0039e0 0x7f382c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383e7fc640 1 --2- 172.21.15.1:0/4198058247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f382c004390 0x7f382c004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383e7fc640 1 --2- 172.21.15.1:0/4198058247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38400c3260 0x7f382c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383e7fc640 1 -- 172.21.15.1:0/4198058247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38400c37b0 con 0x7f382c004390 2024-03-06T07:26:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383e7fc640 1 -- 172.21.15.1:0/4198058247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c00ba70 con 0x7f382c0039e0 2024-03-06T07:26:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.415+0000 7f383e7fc640 1 -- 172.21.15.1:0/4198058247 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382c00bc70 con 0x7f38400c3260 2024-03-06T07:26:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.416+0000 7f383effd640 1 --2- 172.21.15.1:0/4198058247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f382c004390 0x7f382c004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.416+0000 7f383f7fe640 1 --2- 172.21.15.1:0/4198058247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f382c0039e0 0x7f382c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.416+0000 7f383ffff640 1 --2- 172.21.15.1:0/4198058247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38400c3260 0x7f382c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.416+0000 7f383effd640 1 -- 172.21.15.1:0/4198058247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38400c3260 msgr2=0x7f382c005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.416+0000 7f383effd640 1 --2- 172.21.15.1:0/4198058247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38400c3260 0x7f382c005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.416+0000 7f383effd640 1 -- 172.21.15.1:0/4198058247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f382c0039e0 msgr2=0x7f382c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.416+0000 7f383effd640 1 --2- 172.21.15.1:0/4198058247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f382c0039e0 0x7f382c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.416+0000 7f383effd640 1 -- 172.21.15.1:0/4198058247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382c003520 con 0x7f382c004390 2024-03-06T07:26:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.417+0000 7f383effd640 1 --2- 172.21.15.1:0/4198058247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f382c004390 0x7f382c004860 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f3828009a90 tx=0x7f382c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=696b80817bd71e50 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.417+0000 7f383e7fc640 1 -- 172.21.15.1:0/4198058247 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828039070 con 0x7f382c004390 2024-03-06T07:26:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.417+0000 7f383e7fc640 1 -- 172.21.15.1:0/4198058247 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382802c680 con 0x7f382c004390 2024-03-06T07:26:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.417+0000 7f383e7fc640 1 -- 172.21.15.1:0/4198058247 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828009690 con 0x7f382c004390 2024-03-06T07:26:36.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.417+0000 7f3845ced640 1 -- 172.21.15.1:0/4198058247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f382c004390 msgr2=0x7f382c004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.417+0000 7f3845ced640 1 --2- 172.21.15.1:0/4198058247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f382c004390 0x7f382c004860 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f3828009a90 tx=0x7f382c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:36.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.417+0000 7f383ffff640 1 -- 172.21.15.1:0/4198058247 reap_dead start 2024-03-06T07:26:36.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.417+0000 7f3845ced640 1 -- 172.21.15.1:0/4198058247 shutdown_connections 2024-03-06T07:26:36.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.417+0000 7f3845ced640 1 -- 172.21.15.1:0/4198058247 >> 172.21.15.1:0/4198058247 conn(0x7f384006b3e0 msgr2=0x7f384006b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:36.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.417+0000 7f3845ced640 1 -- 172.21.15.1:0/4198058247 shutdown_connections 2024-03-06T07:26:36.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.418+0000 7f3845ced640 1 -- 172.21.15.1:0/4198058247 wait complete. 2024-03-06T07:26:36.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.419+0000 7f3845ced640 1 Processor -- start 2024-03-06T07:26:36.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.419+0000 7f3845ced640 1 -- start start 2024-03-06T07:26:36.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.419+0000 7f3845ced640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f382c004390 0x7f3840154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.419+0000 7f3845ced640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3840071470 0x7f38401df9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.419+0000 7f3845ced640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38401dfef0 0x7f384006f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.419+0000 7f3845ced640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38400c8560 con 0x7f3840071470 2024-03-06T07:26:36.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.419+0000 7f3845ced640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3840158af0 con 0x7f382c004390 2024-03-06T07:26:36.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.419+0000 7f3845ced640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38400c86e0 con 0x7f38401dfef0 2024-03-06T07:26:36.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.420+0000 7f383effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3840071470 0x7f38401df9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.420+0000 7f383effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3840071470 0x7f38401df9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38672/0 (socket says 172.21.15.1:38672) 2024-03-06T07:26:36.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.420+0000 7f383effd640 1 -- 172.21.15.1:0/2112848077 learned_addr learned my addr 172.21.15.1:0/2112848077 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:36.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.420+0000 7f383f7fe640 1 --2- 172.21.15.1:0/2112848077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f382c004390 0x7f3840154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.420+0000 7f383ffff640 1 --2- 172.21.15.1:0/2112848077 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38401dfef0 0x7f384006f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.420+0000 7f383effd640 1 -- 172.21.15.1:0/2112848077 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38401dfef0 msgr2=0x7f384006f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.420+0000 7f383effd640 1 --2- 172.21.15.1:0/2112848077 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38401dfef0 0x7f384006f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.420+0000 7f383effd640 1 -- 172.21.15.1:0/2112848077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f382c004390 msgr2=0x7f3840154320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.420+0000 7f383effd640 1 --2- 172.21.15.1:0/2112848077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f382c004390 0x7f3840154320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.420+0000 7f383effd640 1 -- 172.21.15.1:0/2112848077 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38401eed80 con 0x7f3840071470 2024-03-06T07:26:36.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.420+0000 7f383effd640 1 --2- 172.21.15.1:0/2112848077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3840071470 0x7f38401df9b0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f382802c2f0 tx=0x7f3828004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.420+0000 7f3837fff640 1 -- 172.21.15.1:0/2112848077 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828042020 con 0x7f3840071470 2024-03-06T07:26:36.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.420+0000 7f3837fff640 1 -- 172.21.15.1:0/2112848077 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382803a040 con 0x7f3840071470 2024-03-06T07:26:36.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.420+0000 7f3845ced640 1 -- 172.21.15.1:0/2112848077 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38401edfc0 con 0x7f3840071470 2024-03-06T07:26:36.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.420+0000 7f3837fff640 1 -- 172.21.15.1:0/2112848077 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828039070 con 0x7f3840071470 2024-03-06T07:26:36.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.420+0000 7f3845ced640 1 -- 172.21.15.1:0/2112848077 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38401ee580 con 0x7f3840071470 2024-03-06T07:26:36.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.422+0000 7f3845ced640 1 -- 172.21.15.1:0/2112848077 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3840067fd0 con 0x7f3840071470 2024-03-06T07:26:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.422+0000 7f3837fff640 1 -- 172.21.15.1:0/2112848077 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3828045020 con 0x7f3840071470 2024-03-06T07:26:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.422+0000 7f3837fff640 1 --2- 172.21.15.1:0/2112848077 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f382006b4d0 0x7f382006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.422+0000 7f3837fff640 1 -- 172.21.15.1:0/2112848077 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f38280abd50 con 0x7f3840071470 2024-03-06T07:26:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.422+0000 7f383f7fe640 1 --2- 172.21.15.1:0/2112848077 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f382006b4d0 0x7f382006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.423+0000 7f383f7fe640 1 --2- 172.21.15.1:0/2112848077 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f382006b4d0 0x7f382006d990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f384014f2d0 tx=0x7f3830001090 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.427+0000 7f3837fff640 1 -- 172.21.15.1:0/2112848077 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f38280af030 con 0x7f3840071470 2024-03-06T07:26:36.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.681+0000 7f3845ced640 1 -- 172.21.15.1:0/2112848077 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "11:0"} v 0) v1 -- 0x7f384014d720 con 0x7f3840071470 2024-03-06T07:26:36.683 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:26:36.682+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:36.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.683+0000 7f3837fff640 1 -- 172.21.15.1:0/2112848077 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "11:0"}]=0 failed mds gid 8823 v141) v1 ==== 98+0+0 (secure 0 0 0) 0x7f382804c030 con 0x7f3840071470 2024-03-06T07:26:36.685 INFO:teuthology.orchestra.run.smithi001.stderr:failed mds gid 8823 2024-03-06T07:26:36.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.686+0000 7f3845ced640 1 -- 172.21.15.1:0/2112848077 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f382006b4d0 msgr2=0x7f382006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.686+0000 7f3845ced640 1 --2- 172.21.15.1:0/2112848077 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f382006b4d0 0x7f382006d990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f384014f2d0 tx=0x7f3830001090 comp rx=0 tx=0).stop 2024-03-06T07:26:36.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.686+0000 7f3845ced640 1 -- 172.21.15.1:0/2112848077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3840071470 msgr2=0x7f38401df9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.686+0000 7f3845ced640 1 --2- 172.21.15.1:0/2112848077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3840071470 0x7f38401df9b0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f382802c2f0 tx=0x7f3828004730 comp rx=0 tx=0).stop 2024-03-06T07:26:36.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.687+0000 7f3845ced640 1 -- 172.21.15.1:0/2112848077 shutdown_connections 2024-03-06T07:26:36.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.687+0000 7f3845ced640 1 --2- 172.21.15.1:0/2112848077 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f382006b4d0 0x7f382006d990 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.687+0000 7f3845ced640 1 --2- 172.21.15.1:0/2112848077 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f38401dfef0 0x7f384006f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.687+0000 7f3845ced640 1 --2- 172.21.15.1:0/2112848077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3840071470 0x7f38401df9b0 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.687+0000 7f3845ced640 1 --2- 172.21.15.1:0/2112848077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f382c004390 0x7f3840154320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.687+0000 7f3845ced640 1 -- 172.21.15.1:0/2112848077 >> 172.21.15.1:0/2112848077 conn(0x7f384006b3e0 msgr2=0x7f38400c71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:36.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.687+0000 7f3845ced640 1 -- 172.21.15.1:0/2112848077 shutdown_connections 2024-03-06T07:26:36.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.687+0000 7f3845ced640 1 -- 172.21.15.1:0/2112848077 wait complete. 2024-03-06T07:26:36.690 INFO:tasks.ceph.mds.a.smithi001.stderr: -445> 2024-03-06T07:26:22.901+0000 7fade67ea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:36.700 INFO:tasks.ceph.mds.c.smithi001.stderr: -276> 2024-03-06T07:26:27.902+0000 7f3676d68640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:36.703 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:36.722 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:26:36.729 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:26:36.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.809+0000 7f557edf7640 1 Processor -- start 2024-03-06T07:26:36.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.809+0000 7f557edf7640 1 -- start start 2024-03-06T07:26:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.809+0000 7f557edf7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f55780c3050 0x7f55780c05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.809+0000 7f557edf7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f557806c650 0x7f55780c0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.809+0000 7f557edf7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55780a8c40 con 0x7f55781550a0 2024-03-06T07:26:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.809+0000 7f557edf7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55780a8f20 con 0x7f55780c3050 2024-03-06T07:26:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.809+0000 7f557edf7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55780a9200 con 0x7f557806c650 2024-03-06T07:26:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.810+0000 7f557cb93640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f55780c3050 0x7f55780c05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.810+0000 7f5577fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f557806c650 0x7f55780c0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.810+0000 7f5577fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f557806c650 0x7f55780c0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:49028/0 (socket says 172.21.15.1:49028) 2024-03-06T07:26:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.810+0000 7f557cb93640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f55780c3050 0x7f55780c05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:42778/0 (socket says 172.21.15.1:42778) 2024-03-06T07:26:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.810+0000 7f5577fff640 1 -- 172.21.15.1:0/4165853628 learned_addr learned my addr 172.21.15.1:0/4165853628 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.810+0000 7f557d394640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f55781550a0 0x7f5578151ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:51444/0 (socket says 172.21.15.1:51444) 2024-03-06T07:26:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.810+0000 7f557cb93640 1 -- 172.21.15.1:0/4165853628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f557806c650 msgr2=0x7f55780c0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.810+0000 7f557cb93640 1 --2- 172.21.15.1:0/4165853628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f557806c650 0x7f55780c0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.810+0000 7f557cb93640 1 -- 172.21.15.1:0/4165853628 >> v1:172.21.15.98:6789/0 conn(0x7f55781550a0 legacy=0x7f5578151ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.810+0000 7f557cb93640 1 -- 172.21.15.1:0/4165853628 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55780a94e0 con 0x7f55780c3050 2024-03-06T07:26:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.810+0000 7f55777fe640 1 -- 172.21.15.1:0/4165853628 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871842776 0 0) 0x7f55780a8c40 con 0x7f55781550a0 2024-03-06T07:26:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.810+0000 7f5577fff640 1 --2- 172.21.15.1:0/4165853628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f557806c650 0x7f55780c0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.811+0000 7f557cb93640 1 --2- 172.21.15.1:0/4165853628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55780c3050 0x7f55780c05f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f55680097a0 tx=0x7f556802b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8c72afd149552fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.811+0000 7f55777fe640 1 -- 172.21.15.1:0/4165853628 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568038070 con 0x7f55780c3050 2024-03-06T07:26:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.811+0000 7f55777fe640 1 -- 172.21.15.1:0/4165853628 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5568031700 con 0x7f55780c3050 2024-03-06T07:26:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.811+0000 7f55777fe640 1 -- 172.21.15.1:0/4165853628 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568031ad0 con 0x7f55780c3050 2024-03-06T07:26:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.812+0000 7f557edf7640 1 -- 172.21.15.1:0/4165853628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55780c3050 msgr2=0x7f55780c05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.812+0000 7f557edf7640 1 --2- 172.21.15.1:0/4165853628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55780c3050 0x7f55780c05f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f55680097a0 tx=0x7f556802b7a0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.812+0000 7f557edf7640 1 -- 172.21.15.1:0/4165853628 shutdown_connections 2024-03-06T07:26:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.812+0000 7f557edf7640 1 --2- 172.21.15.1:0/4165853628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f557806c650 0x7f55780c0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.812+0000 7f557edf7640 1 --2- 172.21.15.1:0/4165853628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55780c3050 0x7f55780c05f0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.812+0000 7f557edf7640 1 -- 172.21.15.1:0/4165853628 >> 172.21.15.1:0/4165853628 conn(0x7f55780c3cc0 msgr2=0x7f55780c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:36.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.812+0000 7f557edf7640 1 -- 172.21.15.1:0/4165853628 shutdown_connections 2024-03-06T07:26:36.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.812+0000 7f557edf7640 1 -- 172.21.15.1:0/4165853628 wait complete. 2024-03-06T07:26:36.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.813+0000 7f557edf7640 1 Processor -- start 2024-03-06T07:26:36.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.813+0000 7f557edf7640 1 -- start start 2024-03-06T07:26:36.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.813+0000 7f557edf7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f557806c650 0x7f5578154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.813+0000 7f557edf7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55780c3050 0x7f55780c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.813+0000 7f557edf7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55781550a0 0x7f5578151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f557edf7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55780a9ce0 con 0x7f55780c3050 2024-03-06T07:26:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f557edf7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55780c3980 con 0x7f55781550a0 2024-03-06T07:26:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f557edf7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f557806f830 con 0x7f557806c650 2024-03-06T07:26:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f5577fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55780c3050 0x7f55780c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f5577fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55780c3050 0x7f55780c67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38700/0 (socket says 172.21.15.1:38700) 2024-03-06T07:26:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f5577fff640 1 -- 172.21.15.1:0/3152953236 learned_addr learned my addr 172.21.15.1:0/3152953236 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f557cb93640 1 --2- 172.21.15.1:0/3152953236 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f557806c650 0x7f5578154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f557d394640 1 --2- 172.21.15.1:0/3152953236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55781550a0 0x7f5578151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f5577fff640 1 -- 172.21.15.1:0/3152953236 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f557806c650 msgr2=0x7f5578154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f5577fff640 1 --2- 172.21.15.1:0/3152953236 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f557806c650 0x7f5578154650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f5577fff640 1 -- 172.21.15.1:0/3152953236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55781550a0 msgr2=0x7f5578151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f5577fff640 1 --2- 172.21.15.1:0/3152953236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55781550a0 0x7f5578151180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f5577fff640 1 -- 172.21.15.1:0/3152953236 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5578154b90 con 0x7f55780c3050 2024-03-06T07:26:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f5577fff640 1 --2- 172.21.15.1:0/3152953236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55780c3050 0x7f55780c67c0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f556000e9b0 tx=0x7f556000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f55757fa640 1 -- 172.21.15.1:0/3152953236 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5560017070 con 0x7f55780c3050 2024-03-06T07:26:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f55757fa640 1 -- 172.21.15.1:0/3152953236 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556000c9a0 con 0x7f55780c3050 2024-03-06T07:26:36.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f557edf7640 1 -- 172.21.15.1:0/3152953236 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5578154e60 con 0x7f55780c3050 2024-03-06T07:26:36.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f55757fa640 1 -- 172.21.15.1:0/3152953236 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556000cd50 con 0x7f55780c3050 2024-03-06T07:26:36.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.814+0000 7f557edf7640 1 -- 172.21.15.1:0/3152953236 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55781ea9a0 con 0x7f55780c3050 2024-03-06T07:26:36.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.815+0000 7f557edf7640 1 -- 172.21.15.1:0/3152953236 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5578067fd0 con 0x7f55780c3050 2024-03-06T07:26:36.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.816+0000 7f55757fa640 1 -- 172.21.15.1:0/3152953236 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f556001a070 con 0x7f55780c3050 2024-03-06T07:26:36.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.816+0000 7f55757fa640 1 --2- 172.21.15.1:0/3152953236 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f555806b4d0 0x7f555806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.816+0000 7f55757fa640 1 -- 172.21.15.1:0/3152953236 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f556008a410 con 0x7f55780c3050 2024-03-06T07:26:36.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.816+0000 7f557cb93640 1 --2- 172.21.15.1:0/3152953236 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f555806b4d0 0x7f555806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.817+0000 7f557cb93640 1 --2- 172.21.15.1:0/3152953236 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f555806b4d0 0x7f555806d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f5568035000 tx=0x7f5568008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:36.821+0000 7f55757fa640 1 -- 172.21.15.1:0/3152953236 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5560058740 con 0x7f55780c3050 2024-03-06T07:26:37.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:37.042+0000 7f557edf7640 1 -- 172.21.15.1:0/3152953236 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f557806b8a0 con 0x7f55780c3050 2024-03-06T07:26:37.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:37.042+0000 7f55757fa640 1 -- 172.21.15.1:0/3152953236 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+4679 (secure 0 0 0) 0x7f556005c7e0 con 0x7f55780c3050 2024-03-06T07:26:37.045 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:37.045 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":142,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:16.977128+0000","modified":"2024-03-06T07:26:36.687327+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7774":{"gid":7774,"name":"d","rank":0,"incarnation":142,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6836/3360241462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3360241462},{"type":"v1","addr":"172.21.15.98:6836","nonce":3360241462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":11},{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:26.024256+0000","modified":"2024-03-06T07:26:36.687518+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":155,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6989":{"gid":6989,"name":"b","rank":0,"incarnation":142,"state":"up:replay","state_seq":7,"addr":"172.21.15.98:6837/2360179300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2360179300},{"type":"v1","addr":"172.21.15.98:6837","nonce":2360179300}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":12}]} 2024-03-06T07:26:37.045 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 142 2024-03-06T07:26:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:37.044+0000 7f557edf7640 1 -- 172.21.15.1:0/3152953236 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f555806b4d0 msgr2=0x7f555806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:37.044+0000 7f557edf7640 1 --2- 172.21.15.1:0/3152953236 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f555806b4d0 0x7f555806d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f5568035000 tx=0x7f5568008030 comp rx=0 tx=0).stop 2024-03-06T07:26:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:37.045+0000 7f557edf7640 1 -- 172.21.15.1:0/3152953236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55780c3050 msgr2=0x7f55780c67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:37.045+0000 7f557edf7640 1 --2- 172.21.15.1:0/3152953236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55780c3050 0x7f55780c67c0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f556000e9b0 tx=0x7f556000ee70 comp rx=0 tx=0).stop 2024-03-06T07:26:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:37.045+0000 7f557edf7640 1 -- 172.21.15.1:0/3152953236 shutdown_connections 2024-03-06T07:26:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:37.045+0000 7f557edf7640 1 --2- 172.21.15.1:0/3152953236 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f555806b4d0 0x7f555806d990 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:37.045+0000 7f557edf7640 1 --2- 172.21.15.1:0/3152953236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55781550a0 0x7f5578151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:37.045+0000 7f557edf7640 1 --2- 172.21.15.1:0/3152953236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55780c3050 0x7f55780c67c0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:37.045+0000 7f557edf7640 1 --2- 172.21.15.1:0/3152953236 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f557806c650 0x7f5578154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:37.045+0000 7f557edf7640 1 -- 172.21.15.1:0/3152953236 >> 172.21.15.1:0/3152953236 conn(0x7f55780c3cc0 msgr2=0x7f557806bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:37.045+0000 7f557edf7640 1 -- 172.21.15.1:0/3152953236 shutdown_connections 2024-03-06T07:26:37.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:37.045+0000 7f557edf7640 1 -- 172.21.15.1:0/3152953236 wait complete. 2024-03-06T07:26:37.460 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 142, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:16.977128+0000', 'modified': '2024-03-06T07:26:36.687327+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 154, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7774}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7774': {'gid': 7774, 'name': 'd', 'rank': 0, 'incarnation': 142, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.98:6836/3360241462', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 3360241462}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 3360241462}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:37.460 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7774:up:replay 2024-03-06T07:26:38.461 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.573+0000 7f3c81833640 1 Processor -- start 2024-03-06T07:26:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.573+0000 7f3c81833640 1 -- start start 2024-03-06T07:26:38.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.574+0000 7f3c81833640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3c7c0afc40 0x7f3c7c160b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:38.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.574+0000 7f3c81833640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7c0c7180 con 0x7f3c7c152820 2024-03-06T07:26:38.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.574+0000 7f3c81833640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7c155460 con 0x7f3c7c0af6f0 2024-03-06T07:26:38.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.574+0000 7f3c81833640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c7c0c6bc0 con 0x7f3c7c0afc40 2024-03-06T07:26:38.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.574+0000 7f3c7affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3c7c152820 0x7f3c7c0c0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37304/0 (socket says 172.21.15.1:37304) 2024-03-06T07:26:38.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.574+0000 7f3c7affd640 1 -- 172.21.15.1:0/983379706 learned_addr learned my addr 172.21.15.1:0/983379706 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:38.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.574+0000 7f3c7b7fe640 1 --2- 172.21.15.1:0/983379706 >> v2:172.21.15.98:3300/0 conn(0x7f3c7c0afc40 0x7f3c7c160b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:38.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.574+0000 7f3c79ffb640 1 -- 172.21.15.1:0/983379706 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813681219 0 0) 0x7f3c7c0c7180 con 0x7f3c7c152820 2024-03-06T07:26:38.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.574+0000 7f3c79ffb640 1 -- 172.21.15.1:0/983379706 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c68003520 con 0x7f3c7c152820 2024-03-06T07:26:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c79ffb640 1 -- 172.21.15.1:0/983379706 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1317903264 0 0) 0x7f3c68003520 con 0x7f3c7c152820 2024-03-06T07:26:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c79ffb640 1 -- 172.21.15.1:0/983379706 >> v2:172.21.15.98:3300/0 conn(0x7f3c7c0afc40 msgr2=0x7f3c7c160b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c79ffb640 1 --2- 172.21.15.1:0/983379706 >> v2:172.21.15.98:3300/0 conn(0x7f3c7c0afc40 0x7f3c7c160b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c79ffb640 1 -- 172.21.15.1:0/983379706 >> v1:172.21.15.98:6789/0 conn(0x7f3c7c0af6f0 legacy=0x7f3c7c0c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c79ffb640 1 -- 172.21.15.1:0/983379706 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7c0c6ea0 con 0x7f3c7c152820 2024-03-06T07:26:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c79ffb640 1 -- 172.21.15.1:0/983379706 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f3c74001670 con 0x7f3c7c152820 2024-03-06T07:26:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c79ffb640 1 -- 172.21.15.1:0/983379706 >> v1:172.21.15.1:6789/0 conn(0x7f3c7c152820 legacy=0x7f3c7c0c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c79ffb640 1 --2- 172.21.15.1:0/983379706 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c680039b0 0x7f3c68003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c79ffb640 1 --2- 172.21.15.1:0/983379706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c68004800 0x7f3c68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c79ffb640 1 --2- 172.21.15.1:0/983379706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c680051c0 0x7f3c6800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c79ffb640 1 -- 172.21.15.1:0/983379706 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c0c6bc0 con 0x7f3c68004800 2024-03-06T07:26:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c79ffb640 1 -- 172.21.15.1:0/983379706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6800c060 con 0x7f3c680051c0 2024-03-06T07:26:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c79ffb640 1 -- 172.21.15.1:0/983379706 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6800c260 con 0x7f3c680039b0 2024-03-06T07:26:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c7a7fc640 1 --2- 172.21.15.1:0/983379706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c68004800 0x7f3c68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c7affd640 1 --2- 172.21.15.1:0/983379706 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c680039b0 0x7f3c68003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c7b7fe640 1 --2- 172.21.15.1:0/983379706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c680051c0 0x7f3c6800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c7a7fc640 1 -- 172.21.15.1:0/983379706 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c680039b0 msgr2=0x7f3c68003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c7a7fc640 1 --2- 172.21.15.1:0/983379706 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c680039b0 0x7f3c68003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c7a7fc640 1 -- 172.21.15.1:0/983379706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c680051c0 msgr2=0x7f3c6800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c7a7fc640 1 --2- 172.21.15.1:0/983379706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c680051c0 0x7f3c6800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c7a7fc640 1 -- 172.21.15.1:0/983379706 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c68003520 con 0x7f3c68004800 2024-03-06T07:26:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.575+0000 7f3c7b7fe640 1 -- 172.21.15.1:0/983379706 reap_dead start 2024-03-06T07:26:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.576+0000 7f3c7a7fc640 1 --2- 172.21.15.1:0/983379706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c68004800 0x7f3c68004c80 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f3c64001750 tx=0x7f3c68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=798b79060140d2a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.576+0000 7f3c79ffb640 1 -- 172.21.15.1:0/983379706 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c64001370 con 0x7f3c68004800 2024-03-06T07:26:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.576+0000 7f3c79ffb640 1 -- 172.21.15.1:0/983379706 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c640052d0 con 0x7f3c68004800 2024-03-06T07:26:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.576+0000 7f3c79ffb640 1 -- 172.21.15.1:0/983379706 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c64003830 con 0x7f3c68004800 2024-03-06T07:26:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.577+0000 7f3c81833640 1 -- 172.21.15.1:0/983379706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c68004800 msgr2=0x7f3c68004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.577+0000 7f3c81833640 1 --2- 172.21.15.1:0/983379706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c68004800 0x7f3c68004c80 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f3c64001750 tx=0x7f3c68005830 comp rx=0 tx=0).stop 2024-03-06T07:26:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.577+0000 7f3c81833640 1 -- 172.21.15.1:0/983379706 shutdown_connections 2024-03-06T07:26:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.577+0000 7f3c81833640 1 --2- 172.21.15.1:0/983379706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c68004800 0x7f3c68004c80 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.577+0000 7f3c81833640 1 -- 172.21.15.1:0/983379706 >> 172.21.15.1:0/983379706 conn(0x7f3c7c072720 msgr2=0x7f3c7c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.577+0000 7f3c81833640 1 -- 172.21.15.1:0/983379706 shutdown_connections 2024-03-06T07:26:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.577+0000 7f3c81833640 1 -- 172.21.15.1:0/983379706 wait complete. 2024-03-06T07:26:38.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.578+0000 7f3c81833640 1 Processor -- start 2024-03-06T07:26:38.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.578+0000 7f3c81833640 1 -- start start 2024-03-06T07:26:38.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.579+0000 7f3c81833640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c68004800 0x7f3c7c1ee280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:38.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.579+0000 7f3c81833640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c7c0c4940 0x7f3c7c14c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:38.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.579+0000 7f3c81833640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c7c14c970 0x7f3c7c157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:38.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.579+0000 7f3c81833640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c152d80 con 0x7f3c7c14c970 2024-03-06T07:26:38.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.579+0000 7f3c81833640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c152c00 con 0x7f3c68004800 2024-03-06T07:26:38.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.579+0000 7f3c81833640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c155f00 con 0x7f3c7c0c4940 2024-03-06T07:26:38.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.579+0000 7f3c7b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c7c14c970 0x7f3c7c157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:38.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.579+0000 7f3c7b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c7c14c970 0x7f3c7c157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38710/0 (socket says 172.21.15.1:38710) 2024-03-06T07:26:38.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.579+0000 7f3c7b7fe640 1 -- 172.21.15.1:0/77596817 learned_addr learned my addr 172.21.15.1:0/77596817 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:38.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.579+0000 7f3c7a7fc640 1 --2- 172.21.15.1:0/77596817 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c7c0c4940 0x7f3c7c14c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:38.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.579+0000 7f3c7affd640 1 --2- 172.21.15.1:0/77596817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c68004800 0x7f3c7c1ee280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:38.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.579+0000 7f3c7b7fe640 1 -- 172.21.15.1:0/77596817 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c7c0c4940 msgr2=0x7f3c7c14c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:38.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.579+0000 7f3c7b7fe640 1 --2- 172.21.15.1:0/77596817 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c7c0c4940 0x7f3c7c14c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.579+0000 7f3c7b7fe640 1 -- 172.21.15.1:0/77596817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c68004800 msgr2=0x7f3c7c1ee280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:38.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.579+0000 7f3c7b7fe640 1 --2- 172.21.15.1:0/77596817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c68004800 0x7f3c7c1ee280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.579+0000 7f3c7b7fe640 1 -- 172.21.15.1:0/77596817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7c1eeee0 con 0x7f3c7c14c970 2024-03-06T07:26:38.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.580+0000 7f3c7b7fe640 1 --2- 172.21.15.1:0/77596817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c7c14c970 0x7f3c7c157ce0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c011b40 tx=0x7f3c6c00f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:38.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.580+0000 7f3c73fff640 1 -- 172.21.15.1:0/77596817 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c00fcb0 con 0x7f3c7c14c970 2024-03-06T07:26:38.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.580+0000 7f3c73fff640 1 -- 172.21.15.1:0/77596817 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6c00fe50 con 0x7f3c7c14c970 2024-03-06T07:26:38.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.580+0000 7f3c73fff640 1 -- 172.21.15.1:0/77596817 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c015670 con 0x7f3c7c14c970 2024-03-06T07:26:38.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.580+0000 7f3c81833640 1 -- 172.21.15.1:0/77596817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c7c1eec00 con 0x7f3c7c14c970 2024-03-06T07:26:38.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.580+0000 7f3c81833640 1 -- 172.21.15.1:0/77596817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7c1dede0 con 0x7f3c7c14c970 2024-03-06T07:26:38.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.581+0000 7f3c81833640 1 -- 172.21.15.1:0/77596817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c7c0686e0 con 0x7f3c7c14c970 2024-03-06T07:26:38.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.581+0000 7f3c73fff640 1 -- 172.21.15.1:0/77596817 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c6c015810 con 0x7f3c7c14c970 2024-03-06T07:26:38.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.582+0000 7f3c73fff640 1 --2- 172.21.15.1:0/77596817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c5c06b500 0x7f3c5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:38.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.582+0000 7f3c73fff640 1 -- 172.21.15.1:0/77596817 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f3c6c08b550 con 0x7f3c7c14c970 2024-03-06T07:26:38.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.582+0000 7f3c7affd640 1 --2- 172.21.15.1:0/77596817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c5c06b500 0x7f3c5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:38.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.583+0000 7f3c7affd640 1 --2- 172.21.15.1:0/77596817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c5c06b500 0x7f3c5c06d9c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3c74007c10 tx=0x7f3c74009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:38.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.586+0000 7f3c73fff640 1 -- 172.21.15.1:0/77596817 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c6c019030 con 0x7f3c7c14c970 2024-03-06T07:26:38.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.810+0000 7f3c81833640 1 -- 172.21.15.1:0/77596817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c7c0c5b40 con 0x7f3c7c14c970 2024-03-06T07:26:38.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.811+0000 7f3c73fff640 1 -- 172.21.15.1:0/77596817 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+6256 (secure 0 0 0) 0x7f3c6c010040 con 0x7f3c7c14c970 2024-03-06T07:26:38.813 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:38.813 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":144,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9162,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/4198928111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4198928111},{"type":"v1","addr":"172.21.15.1:6833","nonce":4198928111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143},{"gid":9168,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2664051788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2664051788},{"type":"v1","addr":"172.21.15.1:6835","nonce":2664051788}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:16.977128+0000","modified":"2024-03-06T07:26:37.703504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7774":{"gid":7774,"name":"d","rank":0,"incarnation":142,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.98:6836/3360241462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3360241462},{"type":"v1","addr":"172.21.15.98:6836","nonce":3360241462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":11},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:26.024256+0000","modified":"2024-03-06T07:26:37.703595+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":155,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6989":{"gid":6989,"name":"b","rank":0,"incarnation":142,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.98:6837/2360179300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2360179300},{"type":"v1","addr":"172.21.15.98:6837","nonce":2360179300}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":12}]} 2024-03-06T07:26:38.813 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 144 2024-03-06T07:26:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.813+0000 7f3c81833640 1 -- 172.21.15.1:0/77596817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c5c06b500 msgr2=0x7f3c5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.813+0000 7f3c81833640 1 --2- 172.21.15.1:0/77596817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c5c06b500 0x7f3c5c06d9c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3c74007c10 tx=0x7f3c74009040 comp rx=0 tx=0).stop 2024-03-06T07:26:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.813+0000 7f3c81833640 1 -- 172.21.15.1:0/77596817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c7c14c970 msgr2=0x7f3c7c157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.813+0000 7f3c81833640 1 --2- 172.21.15.1:0/77596817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c7c14c970 0x7f3c7c157ce0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c011b40 tx=0x7f3c6c00f1b0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.813+0000 7f3c81833640 1 -- 172.21.15.1:0/77596817 shutdown_connections 2024-03-06T07:26:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.813+0000 7f3c81833640 1 --2- 172.21.15.1:0/77596817 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c5c06b500 0x7f3c5c06d9c0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.813+0000 7f3c81833640 1 --2- 172.21.15.1:0/77596817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c7c14c970 0x7f3c7c157ce0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.813+0000 7f3c81833640 1 --2- 172.21.15.1:0/77596817 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c7c0c4940 0x7f3c7c14c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.813+0000 7f3c81833640 1 --2- 172.21.15.1:0/77596817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c68004800 0x7f3c7c1ee280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.813+0000 7f3c81833640 1 -- 172.21.15.1:0/77596817 >> 172.21.15.1:0/77596817 conn(0x7f3c7c072720 msgr2=0x7f3c7c1512c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.813+0000 7f3c81833640 1 -- 172.21.15.1:0/77596817 shutdown_connections 2024-03-06T07:26:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:38.813+0000 7f3c81833640 1 -- 172.21.15.1:0/77596817 wait complete. 2024-03-06T07:26:39.228 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 144, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:16.977128+0000', 'modified': '2024-03-06T07:26:37.703504+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 154, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7774}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7774': {'gid': 7774, 'name': 'd', 'rank': 0, 'incarnation': 142, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.98:6836/3360241462', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 3360241462}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 3360241462}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:39.229 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7774:up:rejoin 2024-03-06T07:26:40.230 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:40.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.341+0000 7f13e09d7640 1 Processor -- start 2024-03-06T07:26:40.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.341+0000 7f13e09d7640 1 -- start start 2024-03-06T07:26:40.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.342+0000 7f13e09d7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f13dc152690 0x7f13dc155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:40.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.342+0000 7f13e09d7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f13dc156cd0 0x7f13dc157120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:40.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.342+0000 7f13e09d7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc1561e0 con 0x7f13dc0c6730 2024-03-06T07:26:40.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.342+0000 7f13e09d7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13dc0c3300 con 0x7f13dc156cd0 2024-03-06T07:26:40.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.342+0000 7f13e09d7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13dc0c35a0 con 0x7f13dc152690 2024-03-06T07:26:40.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.342+0000 7f13dad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f13dc156cd0 0x7f13dc157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:40.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.342+0000 7f13dad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f13dc156cd0 0x7f13dc157120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38716/0 (socket says 172.21.15.1:38716) 2024-03-06T07:26:40.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.342+0000 7f13dad9d640 1 -- 172.21.15.1:0/331535803 learned_addr learned my addr 172.21.15.1:0/331535803 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:40.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.342+0000 7f13da59c640 1 --2- 172.21.15.1:0/331535803 >> v2:172.21.15.98:3300/0 conn(0x7f13dc152690 0x7f13dc155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:40.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.342+0000 7f13dad9d640 1 -- 172.21.15.1:0/331535803 >> v2:172.21.15.98:3300/0 conn(0x7f13dc152690 msgr2=0x7f13dc155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:40.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.342+0000 7f13dad9d640 1 --2- 172.21.15.1:0/331535803 >> v2:172.21.15.98:3300/0 conn(0x7f13dc152690 0x7f13dc155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:40.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.342+0000 7f13dad9d640 1 -- 172.21.15.1:0/331535803 >> v1:172.21.15.98:6789/0 conn(0x7f13dc0c6730 legacy=0x7f13dc0c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:40.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.342+0000 7f13dad9d640 1 -- 172.21.15.1:0/331535803 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13dc0c3880 con 0x7f13dc156cd0 2024-03-06T07:26:40.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.343+0000 7f13dad9d640 1 --2- 172.21.15.1:0/331535803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13dc156cd0 0x7f13dc157120 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0ab5f0 tx=0x7f13d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17eb3ca5ca08d42c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:40.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.343+0000 7f13d959a640 1 -- 172.21.15.1:0/331535803 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d4035020 con 0x7f13dc156cd0 2024-03-06T07:26:40.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.343+0000 7f13d959a640 1 -- 172.21.15.1:0/331535803 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d4031870 con 0x7f13dc156cd0 2024-03-06T07:26:40.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.343+0000 7f13d959a640 1 -- 172.21.15.1:0/331535803 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d4031c40 con 0x7f13dc156cd0 2024-03-06T07:26:40.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.344+0000 7f13e09d7640 1 -- 172.21.15.1:0/331535803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13dc156cd0 msgr2=0x7f13dc157120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:40.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.344+0000 7f13e09d7640 1 --2- 172.21.15.1:0/331535803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13dc156cd0 0x7f13dc157120 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0ab5f0 tx=0x7f13d402b910 comp rx=0 tx=0).stop 2024-03-06T07:26:40.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.344+0000 7f13e09d7640 1 -- 172.21.15.1:0/331535803 shutdown_connections 2024-03-06T07:26:40.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.344+0000 7f13e09d7640 1 --2- 172.21.15.1:0/331535803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13dc156cd0 0x7f13dc157120 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:40.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.344+0000 7f13e09d7640 1 --2- 172.21.15.1:0/331535803 >> v2:172.21.15.98:3300/0 conn(0x7f13dc152690 0x7f13dc155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:40.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.344+0000 7f13e09d7640 1 -- 172.21.15.1:0/331535803 >> 172.21.15.1:0/331535803 conn(0x7f13dc06b410 msgr2=0x7f13dc06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:40.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.344+0000 7f13e09d7640 1 -- 172.21.15.1:0/331535803 shutdown_connections 2024-03-06T07:26:40.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.344+0000 7f13e09d7640 1 -- 172.21.15.1:0/331535803 wait complete. 2024-03-06T07:26:40.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.345+0000 7f13e09d7640 1 Processor -- start 2024-03-06T07:26:40.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.345+0000 7f13e09d7640 1 -- start start 2024-03-06T07:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.346+0000 7f13e09d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13dc0c6730 0x7f13dc1ee330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.346+0000 7f13e09d7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13dc152690 0x7f13dc155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.346+0000 7f13e09d7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13dc156cd0 0x7f13dc155580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.346+0000 7f13e09d7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc0c5ee0 con 0x7f13dc152690 2024-03-06T07:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.346+0000 7f13e09d7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc1567b0 con 0x7f13dc0c6730 2024-03-06T07:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.346+0000 7f13e09d7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc0c6060 con 0x7f13dc156cd0 2024-03-06T07:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.346+0000 7f13d9d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13dc152690 0x7f13dc155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.346+0000 7f13d9d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13dc152690 0x7f13dc155040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38724/0 (socket says 172.21.15.1:38724) 2024-03-06T07:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.346+0000 7f13d9d9b640 1 -- 172.21.15.1:0/718908659 learned_addr learned my addr 172.21.15.1:0/718908659 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.346+0000 7f13da59c640 1 --2- 172.21.15.1:0/718908659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13dc0c6730 0x7f13dc1ee330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.346+0000 7f13dad9d640 1 --2- 172.21.15.1:0/718908659 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13dc156cd0 0x7f13dc155580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.347+0000 7f13d9d9b640 1 -- 172.21.15.1:0/718908659 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13dc156cd0 msgr2=0x7f13dc155580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.347+0000 7f13d9d9b640 1 --2- 172.21.15.1:0/718908659 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13dc156cd0 0x7f13dc155580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.347+0000 7f13d9d9b640 1 -- 172.21.15.1:0/718908659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13dc0c6730 msgr2=0x7f13dc1ee330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.347+0000 7f13d9d9b640 1 --2- 172.21.15.1:0/718908659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13dc0c6730 0x7f13dc1ee330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.347+0000 7f13d9d9b640 1 -- 172.21.15.1:0/718908659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13dc0aa720 con 0x7f13dc152690 2024-03-06T07:26:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.347+0000 7f13d9d9b640 1 --2- 172.21.15.1:0/718908659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13dc152690 0x7f13dc155040 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f13d000c3c0 tx=0x7f13d000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.347+0000 7f13caffd640 1 -- 172.21.15.1:0/718908659 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d00052d0 con 0x7f13dc152690 2024-03-06T07:26:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.347+0000 7f13caffd640 1 -- 172.21.15.1:0/718908659 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d0005490 con 0x7f13dc152690 2024-03-06T07:26:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.347+0000 7f13e09d7640 1 -- 172.21.15.1:0/718908659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13dc1eec70 con 0x7f13dc152690 2024-03-06T07:26:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.347+0000 7f13caffd640 1 -- 172.21.15.1:0/718908659 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d0007260 con 0x7f13dc152690 2024-03-06T07:26:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.347+0000 7f13e09d7640 1 -- 172.21.15.1:0/718908659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13dc1ee870 con 0x7f13dc152690 2024-03-06T07:26:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.348+0000 7f13e09d7640 1 -- 172.21.15.1:0/718908659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13dc067fd0 con 0x7f13dc152690 2024-03-06T07:26:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.349+0000 7f13caffd640 1 -- 172.21.15.1:0/718908659 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f13d0007400 con 0x7f13dc152690 2024-03-06T07:26:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.349+0000 7f13caffd640 1 --2- 172.21.15.1:0/718908659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f13bc06b500 0x7f13bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.349+0000 7f13caffd640 1 -- 172.21.15.1:0/718908659 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f13d0089aa0 con 0x7f13dc152690 2024-03-06T07:26:40.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.349+0000 7f13da59c640 1 --2- 172.21.15.1:0/718908659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f13bc06b500 0x7f13bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:40.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.350+0000 7f13da59c640 1 --2- 172.21.15.1:0/718908659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f13bc06b500 0x7f13bc06d9c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f13dc151480 tx=0x7f13cc00b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:40.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.353+0000 7f13caffd640 1 -- 172.21.15.1:0/718908659 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f13dc1561e0 con 0x7f13dc152690 2024-03-06T07:26:40.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.574+0000 7f13e09d7640 1 -- 172.21.15.1:0/718908659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13dc0c2c40 con 0x7f13dc152690 2024-03-06T07:26:40.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.574+0000 7f13caffd640 1 -- 172.21.15.1:0/718908659 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6272 (secure 0 0 0) 0x7f13d0057dd0 con 0x7f13dc152690 2024-03-06T07:26:40.576 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:40.577 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":145,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9162,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/4198928111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4198928111},{"type":"v1","addr":"172.21.15.1:6833","nonce":4198928111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143},{"gid":9168,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2664051788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2664051788},{"type":"v1","addr":"172.21.15.1:6835","nonce":2664051788}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:16.977128+0000","modified":"2024-03-06T07:26:39.702748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7774":{"gid":7774,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.98:6836/3360241462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3360241462},{"type":"v1","addr":"172.21.15.98:6836","nonce":3360241462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7774,"qdb_cluster":[7774]},"id":11},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:26.024256+0000","modified":"2024-03-06T07:26:39.702748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":155,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6989":{"gid":6989,"name":"b","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.98:6837/2360179300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2360179300},{"type":"v1","addr":"172.21.15.98:6837","nonce":2360179300}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6989,"qdb_cluster":[6989]},"id":12}]} 2024-03-06T07:26:40.577 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 145 2024-03-06T07:26:40.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.576+0000 7f13e09d7640 1 -- 172.21.15.1:0/718908659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f13bc06b500 msgr2=0x7f13bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:40.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.576+0000 7f13e09d7640 1 --2- 172.21.15.1:0/718908659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f13bc06b500 0x7f13bc06d9c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f13dc151480 tx=0x7f13cc00b040 comp rx=0 tx=0).stop 2024-03-06T07:26:40.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.576+0000 7f13e09d7640 1 -- 172.21.15.1:0/718908659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13dc152690 msgr2=0x7f13dc155040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:40.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.576+0000 7f13e09d7640 1 --2- 172.21.15.1:0/718908659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13dc152690 0x7f13dc155040 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f13d000c3c0 tx=0x7f13d000c880 comp rx=0 tx=0).stop 2024-03-06T07:26:40.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.576+0000 7f13e09d7640 1 -- 172.21.15.1:0/718908659 shutdown_connections 2024-03-06T07:26:40.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.576+0000 7f13e09d7640 1 --2- 172.21.15.1:0/718908659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f13bc06b500 0x7f13bc06d9c0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:40.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.576+0000 7f13e09d7640 1 --2- 172.21.15.1:0/718908659 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f13dc156cd0 0x7f13dc155580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:40.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.576+0000 7f13e09d7640 1 --2- 172.21.15.1:0/718908659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13dc152690 0x7f13dc155040 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:40.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.576+0000 7f13e09d7640 1 --2- 172.21.15.1:0/718908659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13dc0c6730 0x7f13dc1ee330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:40.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.576+0000 7f13e09d7640 1 -- 172.21.15.1:0/718908659 >> 172.21.15.1:0/718908659 conn(0x7f13dc06b410 msgr2=0x7f13dc0c57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:40.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.576+0000 7f13e09d7640 1 -- 172.21.15.1:0/718908659 shutdown_connections 2024-03-06T07:26:40.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:40.576+0000 7f13e09d7640 1 -- 172.21.15.1:0/718908659 wait complete. 2024-03-06T07:26:40.991 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 145, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:16.977128+0000', 'modified': '2024-03-06T07:26:39.702748+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 154, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7774}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7774': {'gid': 7774, 'name': 'd', 'rank': 0, 'incarnation': 142, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.98:6836/3360241462', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 3360241462}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 3360241462}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7774, 'qdb_cluster': [7774]} 2024-03-06T07:26:40.991 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:26:40.991 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:26:41.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.095+0000 7f4649f0c640 1 Processor -- start 2024-03-06T07:26:41.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.095+0000 7f4649f0c640 1 -- start start 2024-03-06T07:26:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.095+0000 7f4649f0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46440c6730 0x7f46440c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.095+0000 7f4649f0c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f46440c5d10 0x7f4644156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.095+0000 7f4649f0c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46440c3300 con 0x7f4644152690 2024-03-06T07:26:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.095+0000 7f4649f0c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46440c35a0 con 0x7f46440c5d10 2024-03-06T07:26:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.095+0000 7f4649f0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46440c3880 con 0x7f46440c6730 2024-03-06T07:26:41.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.096+0000 7f46437fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4644152690 0x7f4644155ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37306/0 (socket says 172.21.15.1:37306) 2024-03-06T07:26:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.096+0000 7f46437fe640 1 -- 172.21.15.1:0/3608208016 learned_addr learned my addr 172.21.15.1:0/3608208016 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.096+0000 7f4642ffd640 1 --2- 172.21.15.1:0/3608208016 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46440c6730 0x7f46440c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.096+0000 7f4643fff640 1 --2- 172.21.15.1:0/3608208016 >> v2:172.21.15.98:3300/0 conn(0x7f46440c5d10 0x7f4644156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.096+0000 7f46427fc640 1 -- 172.21.15.1:0/3608208016 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170539498 0 0) 0x7f46440c3300 con 0x7f4644152690 2024-03-06T07:26:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.096+0000 7f46427fc640 1 -- 172.21.15.1:0/3608208016 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f462c003520 con 0x7f4644152690 2024-03-06T07:26:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.096+0000 7f4642ffd640 1 -- 172.21.15.1:0/3608208016 >> v2:172.21.15.98:3300/0 conn(0x7f46440c5d10 msgr2=0x7f4644156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.096+0000 7f4642ffd640 1 --2- 172.21.15.1:0/3608208016 >> v2:172.21.15.98:3300/0 conn(0x7f46440c5d10 0x7f4644156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.096+0000 7f4642ffd640 1 -- 172.21.15.1:0/3608208016 >> v1:172.21.15.1:6789/0 conn(0x7f4644152690 legacy=0x7f4644155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.096+0000 7f4642ffd640 1 -- 172.21.15.1:0/3608208016 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46440c3b60 con 0x7f46440c6730 2024-03-06T07:26:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.096+0000 7f4643fff640 1 --2- 172.21.15.1:0/3608208016 >> v2:172.21.15.98:3300/0 conn(0x7f46440c5d10 0x7f4644156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.097+0000 7f4642ffd640 1 --2- 172.21.15.1:0/3608208016 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46440c6730 0x7f46440c57d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f46380097a0 tx=0x7f463802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8feb9295265940a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.097+0000 7f46427fc640 1 -- 172.21.15.1:0/3608208016 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46380315a0 con 0x7f46440c6730 2024-03-06T07:26:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.097+0000 7f46427fc640 1 -- 172.21.15.1:0/3608208016 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4638037070 con 0x7f46440c6730 2024-03-06T07:26:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.097+0000 7f46427fc640 1 -- 172.21.15.1:0/3608208016 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638031960 con 0x7f46440c6730 2024-03-06T07:26:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.097+0000 7f4649f0c640 1 -- 172.21.15.1:0/3608208016 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46440c6730 msgr2=0x7f46440c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.097+0000 7f4649f0c640 1 --2- 172.21.15.1:0/3608208016 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46440c6730 0x7f46440c57d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f46380097a0 tx=0x7f463802b840 comp rx=0 tx=0).stop 2024-03-06T07:26:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.097+0000 7f4649f0c640 1 -- 172.21.15.1:0/3608208016 shutdown_connections 2024-03-06T07:26:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.097+0000 7f4649f0c640 1 --2- 172.21.15.1:0/3608208016 >> v2:172.21.15.98:3300/0 conn(0x7f46440c5d10 0x7f4644156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.097+0000 7f4649f0c640 1 --2- 172.21.15.1:0/3608208016 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46440c6730 0x7f46440c57d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.097+0000 7f4649f0c640 1 -- 172.21.15.1:0/3608208016 >> 172.21.15.1:0/3608208016 conn(0x7f464406b410 msgr2=0x7f464406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.097+0000 7f4649f0c640 1 -- 172.21.15.1:0/3608208016 shutdown_connections 2024-03-06T07:26:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.097+0000 7f4649f0c640 1 -- 172.21.15.1:0/3608208016 wait complete. 2024-03-06T07:26:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.098+0000 7f4649f0c640 1 Processor -- start 2024-03-06T07:26:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.099+0000 7f4649f0c640 1 -- start start 2024-03-06T07:26:41.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.099+0000 7f4649f0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46440c5d10 0x7f4644070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.099+0000 7f4649f0c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46440c6730 0x7f46441508a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.099+0000 7f4649f0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4644152690 0x7f4644150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.099+0000 7f4649f0c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4644152bf0 con 0x7f46440c6730 2024-03-06T07:26:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.099+0000 7f4649f0c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4644152a70 con 0x7f46440c5d10 2024-03-06T07:26:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.099+0000 7f4649f0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4644157210 con 0x7f4644152690 2024-03-06T07:26:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.099+0000 7f4642ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46440c6730 0x7f46441508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.099+0000 7f4642ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46440c6730 0x7f46441508a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38740/0 (socket says 172.21.15.1:38740) 2024-03-06T07:26:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.099+0000 7f4642ffd640 1 -- 172.21.15.1:0/2664303903 learned_addr learned my addr 172.21.15.1:0/2664303903 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.099+0000 7f4643fff640 1 --2- 172.21.15.1:0/2664303903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4644152690 0x7f4644150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.099+0000 7f46437fe640 1 --2- 172.21.15.1:0/2664303903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46440c5d10 0x7f4644070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.100+0000 7f4642ffd640 1 -- 172.21.15.1:0/2664303903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4644152690 msgr2=0x7f4644150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.100+0000 7f4642ffd640 1 --2- 172.21.15.1:0/2664303903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4644152690 0x7f4644150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.100+0000 7f4642ffd640 1 -- 172.21.15.1:0/2664303903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46440c5d10 msgr2=0x7f4644070ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.100+0000 7f4642ffd640 1 --2- 172.21.15.1:0/2664303903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46440c5d10 0x7f4644070ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.100+0000 7f4642ffd640 1 -- 172.21.15.1:0/2664303903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4644099af0 con 0x7f46440c6730 2024-03-06T07:26:41.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.100+0000 7f4642ffd640 1 --2- 172.21.15.1:0/2664303903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46440c6730 0x7f46441508a0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f46380098c0 tx=0x7f46380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.100+0000 7f462bfff640 1 -- 172.21.15.1:0/2664303903 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638032430 con 0x7f46440c6730 2024-03-06T07:26:41.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.100+0000 7f462bfff640 1 -- 172.21.15.1:0/2664303903 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46380325d0 con 0x7f46440c6730 2024-03-06T07:26:41.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.100+0000 7f4649f0c640 1 -- 172.21.15.1:0/2664303903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46441ee9c0 con 0x7f46440c6730 2024-03-06T07:26:41.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.100+0000 7f462bfff640 1 -- 172.21.15.1:0/2664303903 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46380329c0 con 0x7f46440c6730 2024-03-06T07:26:41.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.100+0000 7f4649f0c640 1 -- 172.21.15.1:0/2664303903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46441eeca0 con 0x7f46440c6730 2024-03-06T07:26:41.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.102+0000 7f462bfff640 1 -- 172.21.15.1:0/2664303903 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4638032b60 con 0x7f46440c6730 2024-03-06T07:26:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.102+0000 7f462bfff640 1 --2- 172.21.15.1:0/2664303903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f463006b500 0x7f463006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.102+0000 7f462bfff640 1 -- 172.21.15.1:0/2664303903 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f463802f030 con 0x7f46440c6730 2024-03-06T07:26:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.102+0000 7f46437fe640 1 --2- 172.21.15.1:0/2664303903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f463006b500 0x7f463006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.103+0000 7f46437fe640 1 --2- 172.21.15.1:0/2664303903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f463006b500 0x7f463006d9c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f4634005880 tx=0x7f4634009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.103+0000 7f464cbd4740 1 Processor -- start 2024-03-06T07:26:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.103+0000 7f464cbd4740 1 -- start start 2024-03-06T07:26:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.104+0000 7f464cbd4740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1cf50 0x55ad31c1aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.104+0000 7f464cbd4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c1d330 0x55ad31c1af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.104+0000 7f464cbd4740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1b4b0 0x55ad31c17680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.104+0000 7f464cbd4740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad31b816b0 con 0x55ad31c1cf50 2024-03-06T07:26:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.104+0000 7f464cbd4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad31b2aa60 con 0x55ad31c1d330 2024-03-06T07:26:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.104+0000 7f464cbd4740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad31b7d750 con 0x55ad31c1b4b0 2024-03-06T07:26:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.104+0000 7f46437fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1cf50 0x55ad31c1aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.104+0000 7f46437fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1cf50 0x55ad31c1aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38748/0 (socket says 172.21.15.1:38748) 2024-03-06T07:26:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.104+0000 7f46437fe640 1 -- 172.21.15.1:0/4148382909 learned_addr learned my addr 172.21.15.1:0/4148382909 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.104+0000 7f4642ffd640 1 --2- 172.21.15.1:0/4148382909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c1d330 0x55ad31c1af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.104+0000 7f4643fff640 1 --2- 172.21.15.1:0/4148382909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1b4b0 0x55ad31c17680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.104+0000 7f46437fe640 1 -- 172.21.15.1:0/4148382909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1b4b0 msgr2=0x55ad31c17680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.104+0000 7f46437fe640 1 --2- 172.21.15.1:0/4148382909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1b4b0 0x55ad31c17680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.104+0000 7f46437fe640 1 -- 172.21.15.1:0/4148382909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c1d330 msgr2=0x55ad31c1af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.104+0000 7f46437fe640 1 --2- 172.21.15.1:0/4148382909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c1d330 0x55ad31c1af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.104+0000 7f46437fe640 1 -- 172.21.15.1:0/4148382909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad31b48f30 con 0x55ad31c1cf50 2024-03-06T07:26:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.105+0000 7f46437fe640 1 --2- 172.21.15.1:0/4148382909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1cf50 0x55ad31c1aa30 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f463400fab0 tx=0x7f4634000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1abadbab586c9f25 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.105+0000 7f46297fa640 1 -- 172.21.15.1:0/4148382909 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4634025070 con 0x55ad31c1cf50 2024-03-06T07:26:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.105+0000 7f46297fa640 1 -- 172.21.15.1:0/4148382909 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4634015cb0 con 0x55ad31c1cf50 2024-03-06T07:26:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.105+0000 7f46297fa640 1 -- 172.21.15.1:0/4148382909 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4634015e60 con 0x55ad31c1cf50 2024-03-06T07:26:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.105+0000 7f464cbd4740 1 -- 172.21.15.1:0/4148382909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1cf50 msgr2=0x55ad31c1aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.105+0000 7f464cbd4740 1 --2- 172.21.15.1:0/4148382909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1cf50 0x55ad31c1aa30 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f463400fab0 tx=0x7f4634000ed0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.106+0000 7f464cbd4740 1 -- 172.21.15.1:0/4148382909 shutdown_connections 2024-03-06T07:26:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.106+0000 7f464cbd4740 1 --2- 172.21.15.1:0/4148382909 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1b4b0 0x55ad31c17680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.106+0000 7f464cbd4740 1 --2- 172.21.15.1:0/4148382909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c1d330 0x55ad31c1af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.106+0000 7f464cbd4740 1 --2- 172.21.15.1:0/4148382909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1cf50 0x55ad31c1aa30 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.106+0000 7f464cbd4740 1 -- 172.21.15.1:0/4148382909 >> 172.21.15.1:0/4148382909 conn(0x55ad31c197d0 msgr2=0x55ad31c3e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.106+0000 7f464cbd4740 1 -- 172.21.15.1:0/4148382909 shutdown_connections 2024-03-06T07:26:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.106+0000 7f464cbd4740 1 -- 172.21.15.1:0/4148382909 wait complete. 2024-03-06T07:26:41.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.106+0000 7f464cbd4740 1 -- start start 2024-03-06T07:26:41.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.106+0000 7f464cbd4740 1 Processor -- start 2024-03-06T07:26:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f464cbd4740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1cf50 0x55ad31c6d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f464cbd4740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1d330 0x55ad31c3cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f464cbd4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c38d20 0x55ad31c391a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f464cbd4740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad31c3f230 con 0x55ad31c1d330 2024-03-06T07:26:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f464cbd4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad31b3dea0 con 0x55ad31c38d20 2024-03-06T07:26:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f464cbd4740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad31c3ec70 con 0x55ad31c1cf50 2024-03-06T07:26:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f464cbd4740 10 client.0 _ll_register_callbacks cb 0x55ad31b3d870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f4642ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1d330 0x55ad31c3cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f4642ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1d330 0x55ad31c3cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38760/0 (socket says 172.21.15.1:38760) 2024-03-06T07:26:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f4642ffd640 1 -- 172.21.15.1:0/634256235 learned_addr learned my addr 172.21.15.1:0/634256235 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f46437fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1cf50 0x55ad31c6d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f4643fff640 1 --2- 172.21.15.1:0/634256235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c38d20 0x55ad31c391a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f4642ffd640 1 -- 172.21.15.1:0/634256235 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1cf50 msgr2=0x55ad31c6d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f4642ffd640 1 --2- 172.21.15.1:0/634256235 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1cf50 0x55ad31c6d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f4642ffd640 1 -- 172.21.15.1:0/634256235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c38d20 msgr2=0x55ad31c391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f4642ffd640 1 --2- 172.21.15.1:0/634256235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c38d20 0x55ad31c391a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f4642ffd640 1 -- 172.21.15.1:0/634256235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad31b3e270 con 0x55ad31c1d330 2024-03-06T07:26:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f4642ffd640 1 --2- 172.21.15.1:0/634256235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1d330 0x55ad31c3cfb0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f46380aa360 tx=0x7f463807adf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dba68ea7b94c09e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.107+0000 7f46227fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:26:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.108+0000 7f46227fc640 1 -- 172.21.15.1:0/634256235 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638092070 con 0x55ad31c1d330 2024-03-06T07:26:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.108+0000 7f46227fc640 1 -- 172.21.15.1:0/634256235 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f4638076c20 con 0x55ad31c1d330 2024-03-06T07:26:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.108+0000 7f464cbd4740 1 -- 172.21.15.1:0/634256235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4634015cb0 con 0x55ad31c1d330 2024-03-06T07:26:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.108+0000 7f46227fc640 1 -- 172.21.15.1:0/634256235 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=145) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46380aa080 con 0x55ad31c1d330 2024-03-06T07:26:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.108+0000 7f464cbd4740 10 client.9195 fetch_fsmap learned FSMap version 145 2024-03-06T07:26:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.108+0000 7f464cbd4740 1 -- 172.21.15.1:0/634256235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=145,monmap=2+}) v3 -- 0x55ad318962d0 con 0x55ad31c1d330 2024-03-06T07:26:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.108+0000 7f46227fc640 1 -- 172.21.15.1:0/634256235 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4634015cb0 con 0x55ad31c1d330 2024-03-06T07:26:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.108+0000 7f46227fc640 1 -- 172.21.15.1:0/634256235 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 145) v1 ==== 3929+0+0 (secure 0 0 0) 0x7f4638031070 con 0x55ad31c1d330 2024-03-06T07:26:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.108+0000 7f464cbd4740 10 client.9195 fetch_fsmap finished waiting for FSMap version 145 2024-03-06T07:26:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.108+0000 7f464cbd4740 10 client.9195 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:26:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.108+0000 7f464cbd4740 20 client.9195 populate_metadata read hostname 'smithi001' 2024-03-06T07:26:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.108+0000 7f464cbd4740 1 --2- 172.21.15.1:0/634256235 >> [v2:172.21.15.98:6834/3360241462,v1:172.21.15.98:6836/3360241462] conn(0x55ad31c76970 0x55ad31c78d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.108+0000 7f464cbd4740 4 client.9195 mds_command: new command op to 7774 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.108+0000 7f464cbd4740 1 -- 172.21.15.1:0/634256235 --> [v2:172.21.15.98:6834/3360241462,v1:172.21.15.98:6836/3360241462] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad31b8ebf0 con 0x55ad31c76970 2024-03-06T07:26:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.108+0000 7f46437fe640 1 --2- 172.21.15.1:0/634256235 >> [v2:172.21.15.98:6834/3360241462,v1:172.21.15.98:6836/3360241462] conn(0x55ad31c76970 0x55ad31c78d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.109+0000 7f46437fe640 1 --2- 172.21.15.1:0/634256235 >> [v2:172.21.15.98:6834/3360241462,v1:172.21.15.98:6836/3360241462] conn(0x55ad31c76970 0x55ad31c78d60 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcee64d81a87c5aa server_cookie=4ff630b8181995ce in_seq=0 out_seq=0 2024-03-06T07:26:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.109+0000 7f46227fc640 10 client.9195 ms_handle_connect on v2:172.21.15.98:6834/3360241462 2024-03-06T07:26:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.110+0000 7f46227fc640 1 -- 172.21.15.1:0/634256235 <== mds.0 v2:172.21.15.98:6834/3360241462 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ad31b8ebf0 con 0x55ad31c76970 2024-03-06T07:26:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.110+0000 7f46227fc640 10 client.9195 handle_command_reply: tid=0 2024-03-06T07:26:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.110+0000 7f464cbd4740 1 client.9195 shutdown 2024-03-06T07:26:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.110+0000 7f464cbd4740 1 -- 172.21.15.1:0/634256235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1d330 msgr2=0x55ad31c3cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.110+0000 7f464cbd4740 1 --2- 172.21.15.1:0/634256235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1d330 0x55ad31c3cfb0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f46380aa360 tx=0x7f463807adf0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.110+0000 7f464cbd4740 1 -- 172.21.15.1:0/634256235 shutdown_connections 2024-03-06T07:26:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.110+0000 7f464cbd4740 1 --2- 172.21.15.1:0/634256235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c38d20 0x55ad31c391a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.110+0000 7f464cbd4740 1 --2- 172.21.15.1:0/634256235 >> [v2:172.21.15.98:6834/3360241462,v1:172.21.15.98:6836/3360241462] conn(0x55ad31c76970 0x55ad31c78d60 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.110+0000 7f464cbd4740 1 --2- 172.21.15.1:0/634256235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1d330 0x55ad31c3cfb0 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.110+0000 7f46227fc640 0 client.9195 ms_handle_reset on v2:172.21.15.98:6834/3360241462 2024-03-06T07:26:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.110+0000 7f464cbd4740 1 --2- 172.21.15.1:0/634256235 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1cf50 0x55ad31c6d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.110+0000 7f464cbd4740 1 -- 172.21.15.1:0/634256235 >> 172.21.15.1:0/634256235 conn(0x55ad31c1b4b0 msgr2=0x55ad31c59b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.110+0000 7f464cbd4740 1 -- 172.21.15.1:0/634256235 shutdown_connections 2024-03-06T07:26:41.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.110+0000 7f464cbd4740 1 -- 172.21.15.1:0/634256235 wait complete. 2024-03-06T07:26:41.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.110+0000 7f464cbd4740 20 client.9195 trim_cache size 0 max 16384 2024-03-06T07:26:41.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.127+0000 7f464cbd4740 1 Processor -- start 2024-03-06T07:26:41.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.127+0000 7f464cbd4740 1 -- start start 2024-03-06T07:26:41.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f464cbd4740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1cf50 0x55ad31c1dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f464cbd4740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1d330 0x55ad31c1e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f464cbd4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c38d20 0x55ad31c16510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f464cbd4740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46441fac40 con 0x55ad31c1cf50 2024-03-06T07:26:41.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f464cbd4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46441f88a0 con 0x55ad31c38d20 2024-03-06T07:26:41.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f464cbd4740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46441f8ea0 con 0x55ad31c1d330 2024-03-06T07:26:41.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f46437fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1cf50 0x55ad31c1dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f46437fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1cf50 0x55ad31c1dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38776/0 (socket says 172.21.15.1:38776) 2024-03-06T07:26:41.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f46437fe640 1 -- 172.21.15.1:0/1099826144 learned_addr learned my addr 172.21.15.1:0/1099826144 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:41.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f4642ffd640 1 --2- 172.21.15.1:0/1099826144 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1d330 0x55ad31c1e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f4643fff640 1 --2- 172.21.15.1:0/1099826144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c38d20 0x55ad31c16510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f46437fe640 1 -- 172.21.15.1:0/1099826144 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1d330 msgr2=0x55ad31c1e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f46437fe640 1 --2- 172.21.15.1:0/1099826144 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1d330 0x55ad31c1e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f46437fe640 1 -- 172.21.15.1:0/1099826144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c38d20 msgr2=0x55ad31c16510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f46437fe640 1 --2- 172.21.15.1:0/1099826144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c38d20 0x55ad31c16510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f46437fe640 1 -- 172.21.15.1:0/1099826144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad319606b0 con 0x55ad31c1cf50 2024-03-06T07:26:41.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f4643fff640 1 --2- 172.21.15.1:0/1099826144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c38d20 0x55ad31c16510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f46437fe640 1 --2- 172.21.15.1:0/1099826144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1cf50 0x55ad31c1dcb0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f4634023040 tx=0x7f4634014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8b98af757c2a34f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f4628ff9640 1 -- 172.21.15.1:0/1099826144 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4634025040 con 0x55ad31c1cf50 2024-03-06T07:26:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f4628ff9640 1 -- 172.21.15.1:0/1099826144 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4634026040 con 0x55ad31c1cf50 2024-03-06T07:26:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.128+0000 7f4628ff9640 1 -- 172.21.15.1:0/1099826144 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4634015e60 con 0x55ad31c1cf50 2024-03-06T07:26:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.129+0000 7f464cbd4740 1 -- 172.21.15.1:0/1099826144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1cf50 msgr2=0x55ad31c1dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.129+0000 7f464cbd4740 1 --2- 172.21.15.1:0/1099826144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1cf50 0x55ad31c1dcb0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f4634023040 tx=0x7f4634014000 comp rx=0 tx=0).stop 2024-03-06T07:26:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.129+0000 7f464cbd4740 1 -- 172.21.15.1:0/1099826144 shutdown_connections 2024-03-06T07:26:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.129+0000 7f464cbd4740 1 --2- 172.21.15.1:0/1099826144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c38d20 0x55ad31c16510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.129+0000 7f464cbd4740 1 --2- 172.21.15.1:0/1099826144 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1d330 0x55ad31c1e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.129+0000 7f464cbd4740 1 --2- 172.21.15.1:0/1099826144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c1cf50 0x55ad31c1dcb0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.129+0000 7f464cbd4740 1 -- 172.21.15.1:0/1099826144 >> 172.21.15.1:0/1099826144 conn(0x55ad31c1b4b0 msgr2=0x55ad31c1aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.129+0000 7f464cbd4740 1 -- 172.21.15.1:0/1099826144 shutdown_connections 2024-03-06T07:26:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.129+0000 7f464cbd4740 1 -- 172.21.15.1:0/1099826144 wait complete. 2024-03-06T07:26:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.129+0000 7f464cbd4740 1 -- start start 2024-03-06T07:26:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.130+0000 7f464cbd4740 1 Processor -- start 2024-03-06T07:26:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.130+0000 7f464cbd4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c1cf50 0x55ad31cc9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.130+0000 7f464cbd4740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1d330 0x55ad31c406b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.130+0000 7f464cbd4740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c38d20 0x55ad31cc68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.130+0000 7f464cbd4740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad31c6d7a0 con 0x55ad31c38d20 2024-03-06T07:26:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.130+0000 7f464cbd4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad31c3e4b0 con 0x55ad31c1cf50 2024-03-06T07:26:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.130+0000 7f464cbd4740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad31b3dea0 con 0x55ad31c1d330 2024-03-06T07:26:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.130+0000 7f464cbd4740 10 client.0 _ll_register_callbacks cb 0x7f4638009130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.130+0000 7f4643fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c38d20 0x55ad31cc68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.130+0000 7f4643fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c38d20 0x55ad31cc68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38782/0 (socket says 172.21.15.1:38782) 2024-03-06T07:26:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.130+0000 7f4643fff640 1 -- 172.21.15.1:0/3528719257 learned_addr learned my addr 172.21.15.1:0/3528719257 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.130+0000 7f46437fe640 1 --2- 172.21.15.1:0/3528719257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c1cf50 0x55ad31cc9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.130+0000 7f4642ffd640 1 --2- 172.21.15.1:0/3528719257 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1d330 0x55ad31c406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f4643fff640 1 -- 172.21.15.1:0/3528719257 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1d330 msgr2=0x55ad31c406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f4643fff640 1 --2- 172.21.15.1:0/3528719257 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1d330 0x55ad31c406b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f4643fff640 1 -- 172.21.15.1:0/3528719257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c1cf50 msgr2=0x55ad31cc9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f4643fff640 1 --2- 172.21.15.1:0/3528719257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c1cf50 0x55ad31cc9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f4643fff640 1 -- 172.21.15.1:0/3528719257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad31c41a60 con 0x55ad31c38d20 2024-03-06T07:26:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f4643fff640 1 --2- 172.21.15.1:0/3528719257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c38d20 0x55ad31cc68b0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f463c017f40 tx=0x7f463c015820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=910a8d24d3491c04 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f46227fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:26:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f46227fc640 1 -- 172.21.15.1:0/3528719257 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c017260 con 0x55ad31c38d20 2024-03-06T07:26:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f46227fc640 1 -- 172.21.15.1:0/3528719257 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f463c003990 con 0x55ad31c38d20 2024-03-06T07:26:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f464cbd4740 1 -- 172.21.15.1:0/3528719257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4634026040 con 0x55ad31c38d20 2024-03-06T07:26:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f46227fc640 1 -- 172.21.15.1:0/3528719257 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=145) v2 ==== 24+0+0 (secure 0 0 0) 0x7f463c003bd0 con 0x55ad31c38d20 2024-03-06T07:26:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f464cbd4740 10 client.9201 fetch_fsmap learned FSMap version 145 2024-03-06T07:26:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f464cbd4740 1 -- 172.21.15.1:0/3528719257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=145,monmap=2+}) v3 -- 0x55ad31c40bf0 con 0x55ad31c38d20 2024-03-06T07:26:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f46227fc640 1 -- 172.21.15.1:0/3528719257 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4634026040 con 0x55ad31c38d20 2024-03-06T07:26:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f46227fc640 1 -- 172.21.15.1:0/3528719257 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 145) v1 ==== 3929+0+0 (secure 0 0 0) 0x7f463c004030 con 0x55ad31c38d20 2024-03-06T07:26:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f464cbd4740 10 client.9201 fetch_fsmap finished waiting for FSMap version 145 2024-03-06T07:26:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f464cbd4740 10 client.9201 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:26:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f464cbd4740 20 client.9201 populate_metadata read hostname 'smithi001' 2024-03-06T07:26:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f464cbd4740 1 --2- 172.21.15.1:0/3528719257 >> [v2:172.21.15.98:6834/3360241462,v1:172.21.15.98:6836/3360241462] conn(0x55ad31ccca00 0x55ad31cccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f464cbd4740 4 client.9201 mds_command: new command op to 7774 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.131+0000 7f464cbd4740 1 -- 172.21.15.1:0/3528719257 --> [v2:172.21.15.98:6834/3360241462,v1:172.21.15.98:6836/3360241462] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ad31c43760 con 0x55ad31ccca00 2024-03-06T07:26:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f46437fe640 1 --2- 172.21.15.1:0/3528719257 >> [v2:172.21.15.98:6834/3360241462,v1:172.21.15.98:6836/3360241462] conn(0x55ad31ccca00 0x55ad31cccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f46437fe640 1 --2- 172.21.15.1:0/3528719257 >> [v2:172.21.15.98:6834/3360241462,v1:172.21.15.98:6836/3360241462] conn(0x55ad31ccca00 0x55ad31cccde0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35b3efe199c47611 server_cookie=be0d8d335f2efb86 in_seq=0 out_seq=0 2024-03-06T07:26:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f46227fc640 10 client.9201 ms_handle_connect on v2:172.21.15.98:6834/3360241462 2024-03-06T07:26:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f46227fc640 1 -- 172.21.15.1:0/3528719257 <== mds.0 v2:172.21.15.98:6834/3360241462 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55ad31c43760 con 0x55ad31ccca00 2024-03-06T07:26:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f46227fc640 10 client.9201 handle_command_reply: tid=0 2024-03-06T07:26:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f464cbd4740 1 client.9201 shutdown 2024-03-06T07:26:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f464cbd4740 1 -- 172.21.15.1:0/3528719257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c38d20 msgr2=0x55ad31cc68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f464cbd4740 1 --2- 172.21.15.1:0/3528719257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c38d20 0x55ad31cc68b0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f463c017f40 tx=0x7f463c015820 comp rx=0 tx=0).stop 2024-03-06T07:26:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f464cbd4740 1 -- 172.21.15.1:0/3528719257 shutdown_connections 2024-03-06T07:26:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f464cbd4740 1 --2- 172.21.15.1:0/3528719257 >> [v2:172.21.15.98:6834/3360241462,v1:172.21.15.98:6836/3360241462] conn(0x55ad31ccca00 0x55ad31cccde0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f464cbd4740 1 --2- 172.21.15.1:0/3528719257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad31c38d20 0x55ad31cc68b0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f464cbd4740 1 --2- 172.21.15.1:0/3528719257 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55ad31c1d330 0x55ad31c406b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f46227fc640 0 client.9201 ms_handle_reset on v2:172.21.15.98:6834/3360241462 2024-03-06T07:26:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f464cbd4740 1 --2- 172.21.15.1:0/3528719257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ad31c1cf50 0x55ad31cc9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f464cbd4740 1 -- 172.21.15.1:0/3528719257 >> 172.21.15.1:0/3528719257 conn(0x55ad31c1b4b0 msgr2=0x55ad31c5c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f464cbd4740 1 -- 172.21.15.1:0/3528719257 shutdown_connections 2024-03-06T07:26:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f464cbd4740 1 -- 172.21.15.1:0/3528719257 wait complete. 2024-03-06T07:26:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.132+0000 7f464cbd4740 20 client.9201 trim_cache size 0 max 16384 2024-03-06T07:26:41.141 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:26:41.141 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:26:41.141 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:26:41.141 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 7774, 2024-03-06T07:26:41.141 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:26:41.141 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:26:41.141 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:26:41.141 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 4.4364510490000004, 2024-03-06T07:26:41.141 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 145, 2024-03-06T07:26:41.141 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 155, 2024-03-06T07:26:41.141 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 155, 2024-03-06T07:26:41.141 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 28.596109841000001 2024-03-06T07:26:41.142 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:26:41.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.133+0000 7f4649f0c640 1 -- 172.21.15.1:0/2664303903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f463006b500 msgr2=0x7f463006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.133+0000 7f4649f0c640 1 --2- 172.21.15.1:0/2664303903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f463006b500 0x7f463006d9c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f4634005880 tx=0x7f4634009040 comp rx=0 tx=0).stop 2024-03-06T07:26:41.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.133+0000 7f4649f0c640 1 -- 172.21.15.1:0/2664303903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46440c6730 msgr2=0x7f46441508a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.133+0000 7f4649f0c640 1 --2- 172.21.15.1:0/2664303903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46440c6730 0x7f46441508a0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f46380098c0 tx=0x7f46380046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.133+0000 7f4649f0c640 1 -- 172.21.15.1:0/2664303903 shutdown_connections 2024-03-06T07:26:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.133+0000 7f4649f0c640 1 --2- 172.21.15.1:0/2664303903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f463006b500 0x7f463006d9c0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.133+0000 7f4649f0c640 1 --2- 172.21.15.1:0/2664303903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4644152690 0x7f4644150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.133+0000 7f4649f0c640 1 --2- 172.21.15.1:0/2664303903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46440c6730 0x7f46441508a0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.133+0000 7f4649f0c640 1 --2- 172.21.15.1:0/2664303903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46440c5d10 0x7f4644070ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.133+0000 7f4649f0c640 1 -- 172.21.15.1:0/2664303903 >> 172.21.15.1:0/2664303903 conn(0x7f464406b410 msgr2=0x7f46441587a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.133+0000 7f4649f0c640 1 -- 172.21.15.1:0/2664303903 shutdown_connections 2024-03-06T07:26:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.133+0000 7f4649f0c640 1 -- 172.21.15.1:0/2664303903 wait complete. 2024-03-06T07:26:41.145 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 145, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:26:16.977128+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 145, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7774": { "addr": "172.21.15.98:6836/3360241462", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6834", "nonce": 3360241462, "type": "v2" }, { "addr": "172.21.15.98:6836", "nonce": 3360241462, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7774, "incarnation": 142, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 154, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-03-06T07:26:39.702748+0000", "qdb_cluster": [ 7774 ], "qdb_leader": 7774, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7774 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:26:26.024256+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 145, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_6989": { "addr": "172.21.15.98:6837/2360179300", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 2360179300, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2360179300, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6989, "incarnation": 142, "join_fscid": 12, "name": "b", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 155, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-03-06T07:26:39.702748+0000", "qdb_cluster": [ 6989 ], "qdb_leader": 6989, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6989 } } } ], "standbys": [ { "addr": "172.21.15.1:6833/4198928111", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6832", "nonce": 4198928111, "type": "v2" }, { "addr": "172.21.15.1:6833", "nonce": 4198928111, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 143, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9162, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.1:6835/2664051788", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6834", "nonce": 2664051788, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 2664051788, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 143, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9168, "incarnation": 0, "join_fscid": 12, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:26:41.147 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 12, 'name': 'b'}, {'join_fscid': 12, 'name': 'c'}, {'join_fscid': -1, 'name': 'd', 'state': 'up:active'}] 2024-03-06T07:26:41.147 INFO:tasks.cephfs.test_failover:current = [{'gid': 9162, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6833/4198928111', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 4198928111}, {'type': 'v1', 'addr': '172.21.15.1:6833', 'nonce': 4198928111}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 143}, {'gid': 6989, 'name': 'b', 'rank': 0, 'incarnation': 142, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.98:6837/2360179300', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 2360179300}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2360179300}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 9168, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.1:6835/2664051788', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 2664051788}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 2664051788}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 143}, {'gid': 7774, 'name': 'd', 'rank': 0, 'incarnation': 142, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.98:6836/3360241462', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 3360241462}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 3360241462}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-03-06T07:26:41.147 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-06T07:26:41.147 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:41.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.246+0000 7f51a7481640 1 Processor -- start 2024-03-06T07:26:41.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.246+0000 7f51a7481640 1 -- start start 2024-03-06T07:26:41.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.247+0000 7f51a7481640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51a0152690 0x7f51a0155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.247+0000 7f51a7481640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f51a00c6730 0x7f51a00c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.247+0000 7f51a7481640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51a01561e0 con 0x7f51a00c5d10 2024-03-06T07:26:41.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.247+0000 7f51a7481640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51a00c3300 con 0x7f51a0152690 2024-03-06T07:26:41.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.247+0000 7f51a7481640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51a00c35a0 con 0x7f51a00c6730 2024-03-06T07:26:41.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.247+0000 7f51a521d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51a0152690 0x7f51a0155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.247+0000 7f51a5a1e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f51a00c5d10 0x7f51a0156cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37314/0 (socket says 172.21.15.1:37314) 2024-03-06T07:26:41.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.247+0000 7f51a5a1e640 1 -- 172.21.15.1:0/2083001252 learned_addr learned my addr 172.21.15.1:0/2083001252 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:41.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.247+0000 7f51a4a1c640 1 --2- 172.21.15.1:0/2083001252 >> v2:172.21.15.98:3300/0 conn(0x7f51a00c6730 0x7f51a00c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.247+0000 7f51a4a1c640 1 -- 172.21.15.1:0/2083001252 >> v2:172.21.15.1:3300/0 conn(0x7f51a0152690 msgr2=0x7f51a0155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.247+0000 7f51a4a1c640 1 --2- 172.21.15.1:0/2083001252 >> v2:172.21.15.1:3300/0 conn(0x7f51a0152690 0x7f51a0155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.247+0000 7f51a4a1c640 1 -- 172.21.15.1:0/2083001252 >> v1:172.21.15.1:6789/0 conn(0x7f51a00c5d10 legacy=0x7f51a0156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.247+0000 7f51a4a1c640 1 -- 172.21.15.1:0/2083001252 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a00c3880 con 0x7f51a00c6730 2024-03-06T07:26:41.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.247+0000 7f5197fff640 1 -- 172.21.15.1:0/2083001252 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860978390 0 0) 0x7f5198001490 con 0x7f51a00c5d10 2024-03-06T07:26:41.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.247+0000 7f51a521d640 1 --2- 172.21.15.1:0/2083001252 >> v2:172.21.15.1:3300/0 conn(0x7f51a0152690 0x7f51a0155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.248+0000 7f51a4a1c640 1 --2- 172.21.15.1:0/2083001252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a00c6730 0x7f51a00c57d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f51a00c98d0 tx=0x7f519002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82062e7e3f963b56 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.248+0000 7f5197fff640 1 -- 172.21.15.1:0/2083001252 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51900315a0 con 0x7f51a00c6730 2024-03-06T07:26:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.248+0000 7f5197fff640 1 -- 172.21.15.1:0/2083001252 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5190037070 con 0x7f51a00c6730 2024-03-06T07:26:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.248+0000 7f5197fff640 1 -- 172.21.15.1:0/2083001252 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5190031960 con 0x7f51a00c6730 2024-03-06T07:26:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.249+0000 7f51a7481640 1 -- 172.21.15.1:0/2083001252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a00c6730 msgr2=0x7f51a00c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.249+0000 7f51a7481640 1 --2- 172.21.15.1:0/2083001252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a00c6730 0x7f51a00c57d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f51a00c98d0 tx=0x7f519002b840 comp rx=0 tx=0).stop 2024-03-06T07:26:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.249+0000 7f51a7481640 1 -- 172.21.15.1:0/2083001252 shutdown_connections 2024-03-06T07:26:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.249+0000 7f51a7481640 1 --2- 172.21.15.1:0/2083001252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a00c6730 0x7f51a00c57d0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.249+0000 7f51a7481640 1 --2- 172.21.15.1:0/2083001252 >> v2:172.21.15.1:3300/0 conn(0x7f51a0152690 0x7f51a0155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.249+0000 7f51a7481640 1 -- 172.21.15.1:0/2083001252 >> 172.21.15.1:0/2083001252 conn(0x7f51a006b410 msgr2=0x7f51a006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.249+0000 7f51a7481640 1 -- 172.21.15.1:0/2083001252 shutdown_connections 2024-03-06T07:26:41.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.249+0000 7f51a7481640 1 -- 172.21.15.1:0/2083001252 wait complete. 2024-03-06T07:26:41.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.250+0000 7f51a7481640 1 Processor -- start 2024-03-06T07:26:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.250+0000 7f51a7481640 1 -- start start 2024-03-06T07:26:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.250+0000 7f51a7481640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51a00c5d10 0x7f51a00acec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.250+0000 7f51a7481640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a00c6730 0x7f51a01ee1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.250+0000 7f51a7481640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51a0152690 0x7f51a00aa940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.250+0000 7f51a7481640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a0152a70 con 0x7f51a00c5d10 2024-03-06T07:26:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.250+0000 7f51a7481640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a01567b0 con 0x7f51a00c6730 2024-03-06T07:26:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.250+0000 7f51a7481640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51a0152bf0 con 0x7f51a0152690 2024-03-06T07:26:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.251+0000 7f51a521d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51a00c5d10 0x7f51a00acec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.251+0000 7f51a521d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51a00c5d10 0x7f51a00acec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38798/0 (socket says 172.21.15.1:38798) 2024-03-06T07:26:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.251+0000 7f51a521d640 1 -- 172.21.15.1:0/3918640266 learned_addr learned my addr 172.21.15.1:0/3918640266 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.251+0000 7f51a4a1c640 1 --2- 172.21.15.1:0/3918640266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a00c6730 0x7f51a01ee1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.251+0000 7f51a5a1e640 1 --2- 172.21.15.1:0/3918640266 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51a0152690 0x7f51a00aa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.251+0000 7f51a521d640 1 -- 172.21.15.1:0/3918640266 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51a0152690 msgr2=0x7f51a00aa940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.251+0000 7f51a521d640 1 --2- 172.21.15.1:0/3918640266 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51a0152690 0x7f51a00aa940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.251+0000 7f51a521d640 1 -- 172.21.15.1:0/3918640266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a00c6730 msgr2=0x7f51a01ee1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.251+0000 7f51a521d640 1 --2- 172.21.15.1:0/3918640266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a00c6730 0x7f51a01ee1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.251+0000 7f51a521d640 1 -- 172.21.15.1:0/3918640266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a00ad400 con 0x7f51a00c5d10 2024-03-06T07:26:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.251+0000 7f51a521d640 1 --2- 172.21.15.1:0/3918640266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51a00c5d10 0x7f51a00acec0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f519c00e990 tx=0x7f519c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.251+0000 7f5195ffb640 1 -- 172.21.15.1:0/3918640266 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c017070 con 0x7f51a00c5d10 2024-03-06T07:26:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.251+0000 7f5195ffb640 1 -- 172.21.15.1:0/3918640266 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519c00c9e0 con 0x7f51a00c5d10 2024-03-06T07:26:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.251+0000 7f51a7481640 1 -- 172.21.15.1:0/3918640266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a00ac700 con 0x7f51a00c5d10 2024-03-06T07:26:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.251+0000 7f5195ffb640 1 -- 172.21.15.1:0/3918640266 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c00cdb0 con 0x7f51a00c5d10 2024-03-06T07:26:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.251+0000 7f51a7481640 1 -- 172.21.15.1:0/3918640266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a00a8c60 con 0x7f51a00c5d10 2024-03-06T07:26:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.252+0000 7f51a7481640 1 -- 172.21.15.1:0/3918640266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a0067fd0 con 0x7f51a00c5d10 2024-03-06T07:26:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.253+0000 7f5195ffb640 1 -- 172.21.15.1:0/3918640266 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f519c01a070 con 0x7f51a00c5d10 2024-03-06T07:26:41.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.253+0000 7f5195ffb640 1 --2- 172.21.15.1:0/3918640266 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f518406b4d0 0x7f518406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.253+0000 7f5195ffb640 1 -- 172.21.15.1:0/3918640266 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f519c08a5a0 con 0x7f51a00c5d10 2024-03-06T07:26:41.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.253+0000 7f51a4a1c640 1 --2- 172.21.15.1:0/3918640266 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f518406b4d0 0x7f518406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.254+0000 7f51a4a1c640 1 --2- 172.21.15.1:0/3918640266 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f518406b4d0 0x7f518406d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5190036000 tx=0x7f5190008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.258+0000 7f5195ffb640 1 -- 172.21.15.1:0/3918640266 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f519c058840 con 0x7f51a00c5d10 2024-03-06T07:26:41.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.476+0000 7f51a7481640 1 -- 172.21.15.1:0/3918640266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51a00c2c40 con 0x7f51a00c5d10 2024-03-06T07:26:41.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.476+0000 7f5195ffb640 1 -- 172.21.15.1:0/3918640266 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6272 (secure 0 0 0) 0x7f519c05c8e0 con 0x7f51a00c5d10 2024-03-06T07:26:41.479 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:41.479 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":145,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9162,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/4198928111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4198928111},{"type":"v1","addr":"172.21.15.1:6833","nonce":4198928111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143},{"gid":9168,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2664051788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2664051788},{"type":"v1","addr":"172.21.15.1:6835","nonce":2664051788}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:16.977128+0000","modified":"2024-03-06T07:26:39.702748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7774":{"gid":7774,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.98:6836/3360241462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3360241462},{"type":"v1","addr":"172.21.15.98:6836","nonce":3360241462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7774,"qdb_cluster":[7774]},"id":11},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:26.024256+0000","modified":"2024-03-06T07:26:39.702748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":155,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6989":{"gid":6989,"name":"b","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.98:6837/2360179300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2360179300},{"type":"v1","addr":"172.21.15.98:6837","nonce":2360179300}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6989,"qdb_cluster":[6989]},"id":12}]} 2024-03-06T07:26:41.479 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 145 2024-03-06T07:26:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.478+0000 7f51a7481640 1 -- 172.21.15.1:0/3918640266 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f518406b4d0 msgr2=0x7f518406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.478+0000 7f51a7481640 1 --2- 172.21.15.1:0/3918640266 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f518406b4d0 0x7f518406d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5190036000 tx=0x7f5190008030 comp rx=0 tx=0).stop 2024-03-06T07:26:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.478+0000 7f51a7481640 1 -- 172.21.15.1:0/3918640266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51a00c5d10 msgr2=0x7f51a00acec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.478+0000 7f51a7481640 1 --2- 172.21.15.1:0/3918640266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51a00c5d10 0x7f51a00acec0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f519c00e990 tx=0x7f519c00ee50 comp rx=0 tx=0).stop 2024-03-06T07:26:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.478+0000 7f51a7481640 1 -- 172.21.15.1:0/3918640266 shutdown_connections 2024-03-06T07:26:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.478+0000 7f51a7481640 1 --2- 172.21.15.1:0/3918640266 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f518406b4d0 0x7f518406d990 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.478+0000 7f51a7481640 1 --2- 172.21.15.1:0/3918640266 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51a0152690 0x7f51a00aa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.478+0000 7f51a7481640 1 --2- 172.21.15.1:0/3918640266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a00c6730 0x7f51a01ee1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.478+0000 7f51a7481640 1 --2- 172.21.15.1:0/3918640266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51a00c5d10 0x7f51a00acec0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.478+0000 7f51a7481640 1 -- 172.21.15.1:0/3918640266 >> 172.21.15.1:0/3918640266 conn(0x7f51a006b410 msgr2=0x7f51a00a9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.478+0000 7f51a7481640 1 -- 172.21.15.1:0/3918640266 shutdown_connections 2024-03-06T07:26:41.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.478+0000 7f51a7481640 1 -- 172.21.15.1:0/3918640266 wait complete. 2024-03-06T07:26:41.491 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:41.491 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:41.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:41.580 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:41.580 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:26:41.580 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:41.594 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:26:41.595 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:26:41.595 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:41.609 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:26:41.610 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:41.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.737+0000 7f3a4753e640 1 Processor -- start 2024-03-06T07:26:41.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.737+0000 7f3a4753e640 1 -- start start 2024-03-06T07:26:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.737+0000 7f3a4753e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a400c5850 0x7f3a400c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.737+0000 7f3a4753e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3a40155220 0x7f3a401556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.738+0000 7f3a4753e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a40158910 con 0x7f3a40152920 2024-03-06T07:26:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.738+0000 7f3a4753e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a40152d00 con 0x7f3a40155220 2024-03-06T07:26:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.738+0000 7f3a4753e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a401596d0 con 0x7f3a400c5850 2024-03-06T07:26:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.738+0000 7f3a45adb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3a40155220 0x7f3a401556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.738+0000 7f3a44ad9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a400c5850 0x7f3a400c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.738+0000 7f3a452da640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3a40152920 0x7f3a4006aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:51474/0 (socket says 172.21.15.1:51474) 2024-03-06T07:26:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.738+0000 7f3a452da640 1 -- 172.21.15.1:0/1494320698 learned_addr learned my addr 172.21.15.1:0/1494320698 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.738+0000 7f3a44ad9640 1 -- 172.21.15.1:0/1494320698 >> v2:172.21.15.98:3300/0 conn(0x7f3a40155220 msgr2=0x7f3a401556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.738+0000 7f3a44ad9640 1 --2- 172.21.15.1:0/1494320698 >> v2:172.21.15.98:3300/0 conn(0x7f3a40155220 0x7f3a401556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.738+0000 7f3a44ad9640 1 -- 172.21.15.1:0/1494320698 >> v1:172.21.15.98:6789/0 conn(0x7f3a40152920 legacy=0x7f3a4006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.738+0000 7f3a44ad9640 1 -- 172.21.15.1:0/1494320698 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a40158bf0 con 0x7f3a400c5850 2024-03-06T07:26:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.738+0000 7f3a45adb640 1 --2- 172.21.15.1:0/1494320698 >> v2:172.21.15.98:3300/0 conn(0x7f3a40155220 0x7f3a401556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:41.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.739+0000 7f3a44ad9640 1 --2- 172.21.15.1:0/1494320698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a400c5850 0x7f3a400c5cb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3a300097a0 tx=0x7f3a3002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d83d23ea4b10cf95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.739+0000 7f3a37fff640 1 -- 172.21.15.1:0/1494320698 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a300315a0 con 0x7f3a400c5850 2024-03-06T07:26:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.739+0000 7f3a37fff640 1 -- 172.21.15.1:0/1494320698 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a30037070 con 0x7f3a400c5850 2024-03-06T07:26:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.739+0000 7f3a37fff640 1 -- 172.21.15.1:0/1494320698 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a30031960 con 0x7f3a400c5850 2024-03-06T07:26:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.739+0000 7f3a4753e640 1 -- 172.21.15.1:0/1494320698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a400c5850 msgr2=0x7f3a400c5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.739+0000 7f3a4753e640 1 --2- 172.21.15.1:0/1494320698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a400c5850 0x7f3a400c5cb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3a300097a0 tx=0x7f3a3002b840 comp rx=0 tx=0).stop 2024-03-06T07:26:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.740+0000 7f3a4753e640 1 -- 172.21.15.1:0/1494320698 shutdown_connections 2024-03-06T07:26:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.740+0000 7f3a4753e640 1 --2- 172.21.15.1:0/1494320698 >> v2:172.21.15.98:3300/0 conn(0x7f3a40155220 0x7f3a401556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.740+0000 7f3a4753e640 1 --2- 172.21.15.1:0/1494320698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a400c5850 0x7f3a400c5cb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.740+0000 7f3a4753e640 1 -- 172.21.15.1:0/1494320698 >> 172.21.15.1:0/1494320698 conn(0x7f3a400a9700 msgr2=0x7f3a400a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.740+0000 7f3a4753e640 1 -- 172.21.15.1:0/1494320698 shutdown_connections 2024-03-06T07:26:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.740+0000 7f3a4753e640 1 -- 172.21.15.1:0/1494320698 wait complete. 2024-03-06T07:26:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.741+0000 7f3a4753e640 1 Processor -- start 2024-03-06T07:26:41.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.741+0000 7f3a4753e640 1 -- start start 2024-03-06T07:26:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.741+0000 7f3a4753e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a400c5850 0x7f3a400c3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.741+0000 7f3a4753e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a40152920 0x7f3a4006f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.741+0000 7f3a4753e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a40155220 0x7f3a400c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.741+0000 7f3a4753e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400c6380 con 0x7f3a400c5850 2024-03-06T07:26:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.741+0000 7f3a4753e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a40159240 con 0x7f3a40152920 2024-03-06T07:26:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.741+0000 7f3a4753e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4006b700 con 0x7f3a40155220 2024-03-06T07:26:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.741+0000 7f3a452da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a400c5850 0x7f3a400c3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.741+0000 7f3a452da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a400c5850 0x7f3a400c3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38800/0 (socket says 172.21.15.1:38800) 2024-03-06T07:26:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.741+0000 7f3a452da640 1 -- 172.21.15.1:0/1973416618 learned_addr learned my addr 172.21.15.1:0/1973416618 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.741+0000 7f3a44ad9640 1 --2- 172.21.15.1:0/1973416618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a40152920 0x7f3a4006f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.741+0000 7f3a45adb640 1 --2- 172.21.15.1:0/1973416618 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a40155220 0x7f3a400c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.742+0000 7f3a452da640 1 -- 172.21.15.1:0/1973416618 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a40155220 msgr2=0x7f3a400c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.742+0000 7f3a452da640 1 --2- 172.21.15.1:0/1973416618 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a40155220 0x7f3a400c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.742+0000 7f3a452da640 1 -- 172.21.15.1:0/1973416618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a40152920 msgr2=0x7f3a4006f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.742+0000 7f3a452da640 1 --2- 172.21.15.1:0/1973416618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a40152920 0x7f3a4006f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.742+0000 7f3a452da640 1 -- 172.21.15.1:0/1973416618 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a400c44b0 con 0x7f3a400c5850 2024-03-06T07:26:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.742+0000 7f3a452da640 1 --2- 172.21.15.1:0/1973416618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a400c5850 0x7f3a400c3040 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00c430 tx=0x7f3a3c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.742+0000 7f3a35ffb640 1 -- 172.21.15.1:0/1973416618 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c017070 con 0x7f3a400c5850 2024-03-06T07:26:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.742+0000 7f3a35ffb640 1 -- 172.21.15.1:0/1973416618 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3c005ce0 con 0x7f3a400c5850 2024-03-06T07:26:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.742+0000 7f3a4753e640 1 -- 172.21.15.1:0/1973416618 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4014c170 con 0x7f3a400c5850 2024-03-06T07:26:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.742+0000 7f3a35ffb640 1 -- 172.21.15.1:0/1973416618 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c0072f0 con 0x7f3a400c5850 2024-03-06T07:26:41.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.742+0000 7f3a4753e640 1 -- 172.21.15.1:0/1973416618 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4006fbb0 con 0x7f3a400c5850 2024-03-06T07:26:41.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.743+0000 7f3a4753e640 1 -- 172.21.15.1:0/1973416618 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a40067fd0 con 0x7f3a400c5850 2024-03-06T07:26:41.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.743+0000 7f3a35ffb640 1 -- 172.21.15.1:0/1973416618 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a3c01a070 con 0x7f3a400c5850 2024-03-06T07:26:41.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.744+0000 7f3a35ffb640 1 --2- 172.21.15.1:0/1973416618 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a2406b500 0x7f3a2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.744+0000 7f3a35ffb640 1 -- 172.21.15.1:0/1973416618 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f3a3c0899b0 con 0x7f3a400c5850 2024-03-06T07:26:41.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.744+0000 7f3a44ad9640 1 --2- 172.21.15.1:0/1973416618 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a2406b500 0x7f3a2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.745+0000 7f3a44ad9640 1 --2- 172.21.15.1:0/1973416618 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a2406b500 0x7f3a2406d9c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f3a30036000 tx=0x7f3a3003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.748+0000 7f3a35ffb640 1 -- 172.21.15.1:0/1973416618 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a3c057ce0 con 0x7f3a400c5850 2024-03-06T07:26:41.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.967+0000 7f3a4753e640 1 -- 172.21.15.1:0/1973416618 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a4006bdf0 con 0x7f3a400c5850 2024-03-06T07:26:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.968+0000 7f3a35ffb640 1 -- 172.21.15.1:0/1973416618 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6272 (secure 0 0 0) 0x7f3a3c05bd80 con 0x7f3a400c5850 2024-03-06T07:26:41.970 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:41.970 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":145,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9162,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/4198928111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4198928111},{"type":"v1","addr":"172.21.15.1:6833","nonce":4198928111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143},{"gid":9168,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2664051788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2664051788},{"type":"v1","addr":"172.21.15.1:6835","nonce":2664051788}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:16.977128+0000","modified":"2024-03-06T07:26:39.702748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7774":{"gid":7774,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.98:6836/3360241462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3360241462},{"type":"v1","addr":"172.21.15.98:6836","nonce":3360241462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7774,"qdb_cluster":[7774]},"id":11},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:26.024256+0000","modified":"2024-03-06T07:26:39.702748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":155,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6989":{"gid":6989,"name":"b","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.98:6837/2360179300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2360179300},{"type":"v1","addr":"172.21.15.98:6837","nonce":2360179300}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6989,"qdb_cluster":[6989]},"id":12}]} 2024-03-06T07:26:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 145 2024-03-06T07:26:41.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.970+0000 7f3a4753e640 1 -- 172.21.15.1:0/1973416618 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a2406b500 msgr2=0x7f3a2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.970+0000 7f3a4753e640 1 --2- 172.21.15.1:0/1973416618 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a2406b500 0x7f3a2406d9c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f3a30036000 tx=0x7f3a3003d040 comp rx=0 tx=0).stop 2024-03-06T07:26:41.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.970+0000 7f3a4753e640 1 -- 172.21.15.1:0/1973416618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a400c5850 msgr2=0x7f3a400c3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.970+0000 7f3a4753e640 1 --2- 172.21.15.1:0/1973416618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a400c5850 0x7f3a400c3040 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00c430 tx=0x7f3a3c00c830 comp rx=0 tx=0).stop 2024-03-06T07:26:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.970+0000 7f3a4753e640 1 -- 172.21.15.1:0/1973416618 shutdown_connections 2024-03-06T07:26:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.970+0000 7f3a4753e640 1 --2- 172.21.15.1:0/1973416618 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3a2406b500 0x7f3a2406d9c0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.970+0000 7f3a4753e640 1 --2- 172.21.15.1:0/1973416618 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a40155220 0x7f3a400c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.970+0000 7f3a4753e640 1 --2- 172.21.15.1:0/1973416618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a40152920 0x7f3a4006f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.970+0000 7f3a4753e640 1 --2- 172.21.15.1:0/1973416618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a400c5850 0x7f3a400c3040 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.970+0000 7f3a4753e640 1 -- 172.21.15.1:0/1973416618 >> 172.21.15.1:0/1973416618 conn(0x7f3a400a9700 msgr2=0x7f3a40153320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.970+0000 7f3a4753e640 1 -- 172.21.15.1:0/1973416618 shutdown_connections 2024-03-06T07:26:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:41.970+0000 7f3a4753e640 1 -- 172.21.15.1:0/1973416618 wait complete. 2024-03-06T07:26:41.983 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:42.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.082+0000 7f21db4eb640 1 Processor -- start 2024-03-06T07:26:42.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.083+0000 7f21db4eb640 1 -- start start 2024-03-06T07:26:42.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.083+0000 7f21db4eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d4152650 0x7f21d4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.083+0000 7f21db4eb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f21d40c6b10 0x7f21d4158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.083+0000 7f21db4eb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f21d40bf9d0 0x7f21d40c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.083+0000 7f21db4eb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d4155800 con 0x7f21d40bf9d0 2024-03-06T07:26:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.083+0000 7f21db4eb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d40c2fc0 con 0x7f21d40c6b10 2024-03-06T07:26:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.083+0000 7f21db4eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d40c6690 con 0x7f21d4152650 2024-03-06T07:26:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.083+0000 7f21d9a88640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f21d40bf9d0 0x7f21d40c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.084+0000 7f21d9a88640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f21d40bf9d0 0x7f21d40c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38804/0 (socket says 172.21.15.1:38804) 2024-03-06T07:26:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.084+0000 7f21d8a86640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f21d40c6b10 0x7f21d4158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.084+0000 7f21d9a88640 1 -- 172.21.15.1:0/899856809 learned_addr learned my addr 172.21.15.1:0/899856809 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.084+0000 7f21d9287640 1 --2- 172.21.15.1:0/899856809 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d4152650 0x7f21d4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.084+0000 7f21d9a88640 1 -- 172.21.15.1:0/899856809 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d4152650 msgr2=0x7f21d4155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.084+0000 7f21d9a88640 1 --2- 172.21.15.1:0/899856809 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d4152650 0x7f21d4155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.084+0000 7f21d9a88640 1 -- 172.21.15.1:0/899856809 >> v2:172.21.15.98:3300/0 conn(0x7f21d40c6b10 msgr2=0x7f21d4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.084+0000 7f21d9a88640 1 --2- 172.21.15.1:0/899856809 >> v2:172.21.15.98:3300/0 conn(0x7f21d40c6b10 0x7f21d4158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.084+0000 7f21d9a88640 1 -- 172.21.15.1:0/899856809 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d40c7280 con 0x7f21d40bf9d0 2024-03-06T07:26:42.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.085+0000 7f21d9a88640 1 --2- 172.21.15.1:0/899856809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21d40bf9d0 0x7f21d40c1460 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f21d408c500 tx=0x7f21cc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9c81892787dfbb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.085+0000 7f21cbfff640 1 -- 172.21.15.1:0/899856809 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc02c9e0 con 0x7f21d40bf9d0 2024-03-06T07:26:42.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.085+0000 7f21cbfff640 1 -- 172.21.15.1:0/899856809 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21cc02cb80 con 0x7f21d40bf9d0 2024-03-06T07:26:42.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.085+0000 7f21cbfff640 1 -- 172.21.15.1:0/899856809 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc0343f0 con 0x7f21d40bf9d0 2024-03-06T07:26:42.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.085+0000 7f21db4eb640 1 -- 172.21.15.1:0/899856809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21d40bf9d0 msgr2=0x7f21d40c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.085+0000 7f21db4eb640 1 --2- 172.21.15.1:0/899856809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21d40bf9d0 0x7f21d40c1460 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f21d408c500 tx=0x7f21cc02be40 comp rx=0 tx=0).stop 2024-03-06T07:26:42.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.085+0000 7f21db4eb640 1 -- 172.21.15.1:0/899856809 shutdown_connections 2024-03-06T07:26:42.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.085+0000 7f21db4eb640 1 --2- 172.21.15.1:0/899856809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21d40bf9d0 0x7f21d40c1460 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.085+0000 7f21db4eb640 1 --2- 172.21.15.1:0/899856809 >> v2:172.21.15.98:3300/0 conn(0x7f21d40c6b10 0x7f21d4158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.085+0000 7f21db4eb640 1 --2- 172.21.15.1:0/899856809 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d4152650 0x7f21d4155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.085+0000 7f21db4eb640 1 -- 172.21.15.1:0/899856809 >> 172.21.15.1:0/899856809 conn(0x7f21d4066ee0 msgr2=0x7f21d40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:42.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.085+0000 7f21db4eb640 1 -- 172.21.15.1:0/899856809 shutdown_connections 2024-03-06T07:26:42.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.085+0000 7f21db4eb640 1 -- 172.21.15.1:0/899856809 wait complete. 2024-03-06T07:26:42.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.086+0000 7f21db4eb640 1 Processor -- start 2024-03-06T07:26:42.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.087+0000 7f21db4eb640 1 -- start start 2024-03-06T07:26:42.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.087+0000 7f21db4eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d40bf9d0 0x7f21d40ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.087+0000 7f21db4eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d4152650 0x7f21d40c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.087+0000 7f21db4eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21d414e5f0 0x7f21d41e0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.087+0000 7f21db4eb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d4155490 con 0x7f21d414e5f0 2024-03-06T07:26:42.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.087+0000 7f21db4eb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d41551b0 con 0x7f21d40bf9d0 2024-03-06T07:26:42.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.087+0000 7f21db4eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d4152a30 con 0x7f21d4152650 2024-03-06T07:26:42.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.087+0000 7f21d9a88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21d414e5f0 0x7f21d41e0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.087+0000 7f21d9a88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21d414e5f0 0x7f21d41e0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38818/0 (socket says 172.21.15.1:38818) 2024-03-06T07:26:42.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.087+0000 7f21d9a88640 1 -- 172.21.15.1:0/1203036551 learned_addr learned my addr 172.21.15.1:0/1203036551 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:42.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.087+0000 7f21d9287640 1 --2- 172.21.15.1:0/1203036551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d40bf9d0 0x7f21d40ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.087+0000 7f21d8a86640 1 --2- 172.21.15.1:0/1203036551 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d4152650 0x7f21d40c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.088+0000 7f21d9a88640 1 -- 172.21.15.1:0/1203036551 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d4152650 msgr2=0x7f21d40c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.088+0000 7f21d9a88640 1 --2- 172.21.15.1:0/1203036551 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d4152650 0x7f21d40c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.088+0000 7f21d9a88640 1 -- 172.21.15.1:0/1203036551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d40bf9d0 msgr2=0x7f21d40ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.088+0000 7f21d9a88640 1 --2- 172.21.15.1:0/1203036551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d40bf9d0 0x7f21d40ca400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.088+0000 7f21d9a88640 1 -- 172.21.15.1:0/1203036551 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d414c400 con 0x7f21d414e5f0 2024-03-06T07:26:42.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.088+0000 7f21d9a88640 1 --2- 172.21.15.1:0/1203036551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21d414e5f0 0x7f21d41e0e50 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009f40 tx=0x7f21cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.088+0000 7f21c9ffb640 1 -- 172.21.15.1:0/1203036551 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc043020 con 0x7f21d414e5f0 2024-03-06T07:26:42.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.088+0000 7f21c9ffb640 1 -- 172.21.15.1:0/1203036551 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21cc03e450 con 0x7f21d414e5f0 2024-03-06T07:26:42.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.088+0000 7f21c9ffb640 1 -- 172.21.15.1:0/1203036551 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc009510 con 0x7f21d414e5f0 2024-03-06T07:26:42.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.088+0000 7f21db4eb640 1 -- 172.21.15.1:0/1203036551 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d41e02c0 con 0x7f21d414e5f0 2024-03-06T07:26:42.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.088+0000 7f21db4eb640 1 -- 172.21.15.1:0/1203036551 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d414c6e0 con 0x7f21d414e5f0 2024-03-06T07:26:42.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.089+0000 7f21db4eb640 1 -- 172.21.15.1:0/1203036551 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d414c8b0 con 0x7f21d414e5f0 2024-03-06T07:26:42.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.090+0000 7f21c9ffb640 1 -- 172.21.15.1:0/1203036551 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f21cc046020 con 0x7f21d414e5f0 2024-03-06T07:26:42.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.090+0000 7f21c9ffb640 1 --2- 172.21.15.1:0/1203036551 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f21b806b4d0 0x7f21b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.090+0000 7f21c9ffb640 1 -- 172.21.15.1:0/1203036551 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f21cc0acfa0 con 0x7f21d414e5f0 2024-03-06T07:26:42.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.090+0000 7f21d9287640 1 --2- 172.21.15.1:0/1203036551 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f21b806b4d0 0x7f21b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.091+0000 7f21d9287640 1 --2- 172.21.15.1:0/1203036551 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f21b806b4d0 0x7f21b806d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f21d40c0cc0 tx=0x7f21d0009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.094+0000 7f21c9ffb640 1 -- 172.21.15.1:0/1203036551 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21cc04d020 con 0x7f21d414e5f0 2024-03-06T07:26:42.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.278+0000 7f21db4eb640 1 -- 172.21.15.1:0/1203036551 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21d4067fd0 con 0x7f21d414e5f0 2024-03-06T07:26:42.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.280+0000 7f21c9ffb640 1 -- 172.21.15.1:0/1203036551 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+20733 (secure 0 0 0) 0x7f21cc07b2d0 con 0x7f21d414e5f0 2024-03-06T07:26:42.282 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:42.282 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":155,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:26:36.687343+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:15.245426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:15.942343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:26:24.304587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:26:25.085741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/1089548586":"2024-03-07T07:26:36.687217+0000","172.21.15.1:6832/1089548586":"2024-03-07T07:26:36.687217+0000","172.21.15.1:6835/67091521":"2024-03-07T07:26:36.682683+0000","172.21.15.1:6834/67091521":"2024-03-07T07:26:36.682683+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:42.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.282+0000 7f21db4eb640 1 -- 172.21.15.1:0/1203036551 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f21b806b4d0 msgr2=0x7f21b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.282+0000 7f21db4eb640 1 --2- 172.21.15.1:0/1203036551 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f21b806b4d0 0x7f21b806d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f21d40c0cc0 tx=0x7f21d0009190 comp rx=0 tx=0).stop 2024-03-06T07:26:42.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.282+0000 7f21db4eb640 1 -- 172.21.15.1:0/1203036551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21d414e5f0 msgr2=0x7f21d41e0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.282+0000 7f21db4eb640 1 --2- 172.21.15.1:0/1203036551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21d414e5f0 0x7f21d41e0e50 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009f40 tx=0x7f21cc004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:42.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.282+0000 7f21db4eb640 1 -- 172.21.15.1:0/1203036551 shutdown_connections 2024-03-06T07:26:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.282+0000 7f21db4eb640 1 --2- 172.21.15.1:0/1203036551 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f21b806b4d0 0x7f21b806d990 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.282+0000 7f21db4eb640 1 --2- 172.21.15.1:0/1203036551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21d414e5f0 0x7f21d41e0e50 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.282+0000 7f21db4eb640 1 --2- 172.21.15.1:0/1203036551 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21d4152650 0x7f21d40c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.282+0000 7f21db4eb640 1 --2- 172.21.15.1:0/1203036551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d40bf9d0 0x7f21d40ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.282+0000 7f21db4eb640 1 -- 172.21.15.1:0/1203036551 >> 172.21.15.1:0/1203036551 conn(0x7f21d4066ee0 msgr2=0x7f21d40672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.282+0000 7f21db4eb640 1 -- 172.21.15.1:0/1203036551 shutdown_connections 2024-03-06T07:26:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.282+0000 7f21db4eb640 1 -- 172.21.15.1:0/1203036551 wait complete. 2024-03-06T07:26:42.296 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:26:15.245426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:26:15.942343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '149', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-06T07:26:24.304587+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '152', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-06T07:26:25.085741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:42.297 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:42.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.400+0000 7f408be15640 1 Processor -- start 2024-03-06T07:26:42.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.400+0000 7f408be15640 1 -- start start 2024-03-06T07:26:42.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.401+0000 7f408be15640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f40840c1390 0x7f40840c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.401+0000 7f408be15640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40841552e0 0x7f4084155700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.401+0000 7f408be15640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40841588e0 con 0x7f4084152730 2024-03-06T07:26:42.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.401+0000 7f408be15640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40840c1d60 con 0x7f40841552e0 2024-03-06T07:26:42.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.401+0000 7f408be15640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4084158bc0 con 0x7f40840c1390 2024-03-06T07:26:42.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.401+0000 7f408a3b2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40841552e0 0x7f4084155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.401+0000 7f4089bb1640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4084152730 0x7f40840c6a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37330/0 (socket says 172.21.15.1:37330) 2024-03-06T07:26:42.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.401+0000 7f4089bb1640 1 -- 172.21.15.1:0/649496942 learned_addr learned my addr 172.21.15.1:0/649496942 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:42.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.401+0000 7f40893b0640 1 --2- 172.21.15.1:0/649496942 >> v2:172.21.15.98:3300/0 conn(0x7f40840c1390 0x7f40840c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.401+0000 7f4088baf640 1 -- 172.21.15.1:0/649496942 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496375861 0 0) 0x7f40841588e0 con 0x7f4084152730 2024-03-06T07:26:42.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.401+0000 7f4088baf640 1 -- 172.21.15.1:0/649496942 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f406c003520 con 0x7f4084152730 2024-03-06T07:26:42.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.401+0000 7f408a3b2640 1 -- 172.21.15.1:0/649496942 >> v2:172.21.15.98:3300/0 conn(0x7f40840c1390 msgr2=0x7f40840c1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:42.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.401+0000 7f408a3b2640 1 --2- 172.21.15.1:0/649496942 >> v2:172.21.15.98:3300/0 conn(0x7f40840c1390 0x7f40840c1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.401+0000 7f408a3b2640 1 -- 172.21.15.1:0/649496942 >> v1:172.21.15.1:6789/0 conn(0x7f4084152730 legacy=0x7f40840c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.401+0000 7f408a3b2640 1 -- 172.21.15.1:0/649496942 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4084158ea0 con 0x7f40841552e0 2024-03-06T07:26:42.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.401+0000 7f40893b0640 1 --2- 172.21.15.1:0/649496942 >> v2:172.21.15.98:3300/0 conn(0x7f40840c1390 0x7f40840c1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:42.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.402+0000 7f408a3b2640 1 --2- 172.21.15.1:0/649496942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40841552e0 0x7f4084155700 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f408408ca60 tx=0x7f407402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107af2b9adbf68ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.402+0000 7f4088baf640 1 -- 172.21.15.1:0/649496942 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4074039070 con 0x7f40841552e0 2024-03-06T07:26:42.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.402+0000 7f4088baf640 1 -- 172.21.15.1:0/649496942 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407402cb60 con 0x7f40841552e0 2024-03-06T07:26:42.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.402+0000 7f4088baf640 1 -- 172.21.15.1:0/649496942 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4074034440 con 0x7f40841552e0 2024-03-06T07:26:42.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.403+0000 7f408be15640 1 -- 172.21.15.1:0/649496942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40841552e0 msgr2=0x7f4084155700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.403+0000 7f408be15640 1 --2- 172.21.15.1:0/649496942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40841552e0 0x7f4084155700 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f408408ca60 tx=0x7f407402bdb0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.403+0000 7f408be15640 1 -- 172.21.15.1:0/649496942 shutdown_connections 2024-03-06T07:26:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.403+0000 7f408be15640 1 --2- 172.21.15.1:0/649496942 >> v2:172.21.15.98:3300/0 conn(0x7f40840c1390 0x7f40840c1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.403+0000 7f408be15640 1 --2- 172.21.15.1:0/649496942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40841552e0 0x7f4084155700 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.403+0000 7f408be15640 1 -- 172.21.15.1:0/649496942 >> 172.21.15.1:0/649496942 conn(0x7f4084066d00 msgr2=0x7f4084067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:42.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.403+0000 7f408be15640 1 -- 172.21.15.1:0/649496942 shutdown_connections 2024-03-06T07:26:42.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.403+0000 7f408be15640 1 -- 172.21.15.1:0/649496942 wait complete. 2024-03-06T07:26:42.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.404+0000 7f408be15640 1 Processor -- start 2024-03-06T07:26:42.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.404+0000 7f408be15640 1 -- start start 2024-03-06T07:26:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f408be15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40840c1390 0x7f4084157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f408be15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4084152730 0x7f408414d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f408be15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40841552e0 0x7f408414d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f408be15640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4084152b10 con 0x7f40840c1390 2024-03-06T07:26:42.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f408be15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4084159460 con 0x7f40841552e0 2024-03-06T07:26:42.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f408be15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40840c71a0 con 0x7f4084152730 2024-03-06T07:26:42.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f4089bb1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40840c1390 0x7f4084157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f4089bb1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40840c1390 0x7f4084157950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38838/0 (socket says 172.21.15.1:38838) 2024-03-06T07:26:42.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f4089bb1640 1 -- 172.21.15.1:0/1317241063 learned_addr learned my addr 172.21.15.1:0/1317241063 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:42.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f40893b0640 1 --2- 172.21.15.1:0/1317241063 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4084152730 0x7f408414d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f408a3b2640 1 --2- 172.21.15.1:0/1317241063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40841552e0 0x7f408414d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f4089bb1640 1 -- 172.21.15.1:0/1317241063 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4084152730 msgr2=0x7f408414d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f4089bb1640 1 --2- 172.21.15.1:0/1317241063 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4084152730 0x7f408414d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f4089bb1640 1 -- 172.21.15.1:0/1317241063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40841552e0 msgr2=0x7f408414d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f4089bb1640 1 --2- 172.21.15.1:0/1317241063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40841552e0 0x7f408414d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f4089bb1640 1 -- 172.21.15.1:0/1317241063 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4084153e50 con 0x7f40840c1390 2024-03-06T07:26:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f4089bb1640 1 --2- 172.21.15.1:0/1317241063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40840c1390 0x7f4084157950 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f407c00c390 tx=0x7f407c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f40827fc640 1 -- 172.21.15.1:0/1317241063 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c017070 con 0x7f40840c1390 2024-03-06T07:26:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f40827fc640 1 -- 172.21.15.1:0/1317241063 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407c005ce0 con 0x7f40840c1390 2024-03-06T07:26:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f40827fc640 1 -- 172.21.15.1:0/1317241063 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c0072f0 con 0x7f40840c1390 2024-03-06T07:26:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.405+0000 7f408be15640 1 -- 172.21.15.1:0/1317241063 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40841eb780 con 0x7f40840c1390 2024-03-06T07:26:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.406+0000 7f408be15640 1 -- 172.21.15.1:0/1317241063 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4084157e90 con 0x7f40840c1390 2024-03-06T07:26:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.406+0000 7f408be15640 1 -- 172.21.15.1:0/1317241063 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4084067fd0 con 0x7f40840c1390 2024-03-06T07:26:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.407+0000 7f40827fc640 1 -- 172.21.15.1:0/1317241063 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f407c01a070 con 0x7f40840c1390 2024-03-06T07:26:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.407+0000 7f40827fc640 1 --2- 172.21.15.1:0/1317241063 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f407006b500 0x7f407006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.407+0000 7f40827fc640 1 -- 172.21.15.1:0/1317241063 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f407c089a20 con 0x7f40840c1390 2024-03-06T07:26:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.407+0000 7f40893b0640 1 --2- 172.21.15.1:0/1317241063 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f407006b500 0x7f407006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.408+0000 7f40893b0640 1 --2- 172.21.15.1:0/1317241063 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f407006b500 0x7f407006d9c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4078009560 tx=0x7f4078009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.412+0000 7f40827fc640 1 -- 172.21.15.1:0/1317241063 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f407c057d50 con 0x7f40840c1390 2024-03-06T07:26:42.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.591+0000 7f408be15640 1 -- 172.21.15.1:0/1317241063 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f40840c6870 con 0x7f40840c1390 2024-03-06T07:26:42.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.591+0000 7f40827fc640 1 -- 172.21.15.1:0/1317241063 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v155) v1 ==== 93+0+10 (secure 0 0 0) 0x7f407c05bdf0 con 0x7f40840c1390 2024-03-06T07:26:42.594 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:26:42.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.593+0000 7f408be15640 1 -- 172.21.15.1:0/1317241063 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f407006b500 msgr2=0x7f407006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.593+0000 7f408be15640 1 --2- 172.21.15.1:0/1317241063 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f407006b500 0x7f407006d9c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4078009560 tx=0x7f4078009190 comp rx=0 tx=0).stop 2024-03-06T07:26:42.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.593+0000 7f408be15640 1 -- 172.21.15.1:0/1317241063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40840c1390 msgr2=0x7f4084157950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.593+0000 7f408be15640 1 --2- 172.21.15.1:0/1317241063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40840c1390 0x7f4084157950 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f407c00c390 tx=0x7f407c00c850 comp rx=0 tx=0).stop 2024-03-06T07:26:42.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.593+0000 7f408be15640 1 -- 172.21.15.1:0/1317241063 shutdown_connections 2024-03-06T07:26:42.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.593+0000 7f408be15640 1 --2- 172.21.15.1:0/1317241063 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f407006b500 0x7f407006d9c0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.593+0000 7f408be15640 1 --2- 172.21.15.1:0/1317241063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40841552e0 0x7f408414d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.593+0000 7f408be15640 1 --2- 172.21.15.1:0/1317241063 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4084152730 0x7f408414d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.593+0000 7f408be15640 1 --2- 172.21.15.1:0/1317241063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40840c1390 0x7f4084157950 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.593+0000 7f408be15640 1 -- 172.21.15.1:0/1317241063 >> 172.21.15.1:0/1317241063 conn(0x7f4084066d00 msgr2=0x7f40840b0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:42.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.593+0000 7f408be15640 1 -- 172.21.15.1:0/1317241063 shutdown_connections 2024-03-06T07:26:42.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.593+0000 7f408be15640 1 -- 172.21.15.1:0/1317241063 wait complete. 2024-03-06T07:26:42.607 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:42.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.710+0000 7fd2611c7640 1 Processor -- start 2024-03-06T07:26:42.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.710+0000 7fd2611c7640 1 -- start start 2024-03-06T07:26:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.711+0000 7fd2611c7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd25c0c6ce0 0x7fd25c13e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.711+0000 7fd2611c7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd25c0c1a70 0x7fd25c158980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.711+0000 7fd2611c7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd25c1551b0 con 0x7fd25c13e6b0 2024-03-06T07:26:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.711+0000 7fd2611c7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd25c155470 con 0x7fd25c0c1a70 2024-03-06T07:26:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.711+0000 7fd2611c7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c155750 con 0x7fd25c0c6ce0 2024-03-06T07:26:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.711+0000 7fd25b59e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd25c0c1a70 0x7fd25c158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.711+0000 7fd25b59e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd25c0c1a70 0x7fd25c158980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38852/0 (socket says 172.21.15.1:38852) 2024-03-06T07:26:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.711+0000 7fd25b59e640 1 -- 172.21.15.1:0/3985673120 learned_addr learned my addr 172.21.15.1:0/3985673120 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.711+0000 7fd25ad9d640 1 --2- 172.21.15.1:0/3985673120 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd25c0c6ce0 0x7fd25c13e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.712+0000 7fd25b59e640 1 -- 172.21.15.1:0/3985673120 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd25c0c6ce0 msgr2=0x7fd25c13e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.712+0000 7fd25b59e640 1 --2- 172.21.15.1:0/3985673120 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd25c0c6ce0 0x7fd25c13e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.712+0000 7fd25b59e640 1 -- 172.21.15.1:0/3985673120 >> v1:172.21.15.98:6789/0 conn(0x7fd25c13e6b0 legacy=0x7fd25c0c1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.712+0000 7fd25b59e640 1 -- 172.21.15.1:0/3985673120 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c155a30 con 0x7fd25c0c1a70 2024-03-06T07:26:42.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.712+0000 7fd25b59e640 1 --2- 172.21.15.1:0/3985673120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd25c0c1a70 0x7fd25c158980 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fd254009870 tx=0x7fd25402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e494889414ebb8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.713+0000 7fd259d9b640 1 -- 172.21.15.1:0/3985673120 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd254035020 con 0x7fd25c0c1a70 2024-03-06T07:26:42.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.713+0000 7fd259d9b640 1 -- 172.21.15.1:0/3985673120 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd254031640 con 0x7fd25c0c1a70 2024-03-06T07:26:42.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.713+0000 7fd259d9b640 1 -- 172.21.15.1:0/3985673120 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25402d040 con 0x7fd25c0c1a70 2024-03-06T07:26:42.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.713+0000 7fd2611c7640 1 -- 172.21.15.1:0/3985673120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd25c0c1a70 msgr2=0x7fd25c158980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.713+0000 7fd2611c7640 1 --2- 172.21.15.1:0/3985673120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd25c0c1a70 0x7fd25c158980 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fd254009870 tx=0x7fd25402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.713+0000 7fd2611c7640 1 -- 172.21.15.1:0/3985673120 shutdown_connections 2024-03-06T07:26:42.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.713+0000 7fd2611c7640 1 --2- 172.21.15.1:0/3985673120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd25c0c1a70 0x7fd25c158980 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.713+0000 7fd2611c7640 1 --2- 172.21.15.1:0/3985673120 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd25c0c6ce0 0x7fd25c13e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.713+0000 7fd2611c7640 1 -- 172.21.15.1:0/3985673120 >> 172.21.15.1:0/3985673120 conn(0x7fd25c06da30 msgr2=0x7fd25c066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:42.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.713+0000 7fd2611c7640 1 -- 172.21.15.1:0/3985673120 shutdown_connections 2024-03-06T07:26:42.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.713+0000 7fd2611c7640 1 -- 172.21.15.1:0/3985673120 wait complete. 2024-03-06T07:26:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.714+0000 7fd2611c7640 1 Processor -- start 2024-03-06T07:26:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.715+0000 7fd2611c7640 1 -- start start 2024-03-06T07:26:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.715+0000 7fd2611c7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd25c0c1a70 0x7fd25c1374b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.715+0000 7fd2611c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd25c0c6ce0 0x7fd25c13cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.715+0000 7fd2611c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd25c13e6b0 0x7fd25c152000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.715+0000 7fd2611c7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c0c70c0 con 0x7fd25c0c6ce0 2024-03-06T07:26:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.715+0000 7fd2611c7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c155ff0 con 0x7fd25c13e6b0 2024-03-06T07:26:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.715+0000 7fd2611c7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c0c7240 con 0x7fd25c0c1a70 2024-03-06T07:26:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.715+0000 7fd25a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd25c0c6ce0 0x7fd25c13cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.715+0000 7fd25a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd25c0c6ce0 0x7fd25c13cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38854/0 (socket says 172.21.15.1:38854) 2024-03-06T07:26:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.715+0000 7fd25a59c640 1 -- 172.21.15.1:0/2054799881 learned_addr learned my addr 172.21.15.1:0/2054799881 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.715+0000 7fd25ad9d640 1 --2- 172.21.15.1:0/2054799881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd25c0c1a70 0x7fd25c1374b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.715+0000 7fd25b59e640 1 --2- 172.21.15.1:0/2054799881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd25c13e6b0 0x7fd25c152000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.716+0000 7fd25a59c640 1 -- 172.21.15.1:0/2054799881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd25c0c1a70 msgr2=0x7fd25c1374b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.716+0000 7fd25a59c640 1 --2- 172.21.15.1:0/2054799881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd25c0c1a70 0x7fd25c1374b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.716+0000 7fd25a59c640 1 -- 172.21.15.1:0/2054799881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd25c13e6b0 msgr2=0x7fd25c152000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.716+0000 7fd25a59c640 1 --2- 172.21.15.1:0/2054799881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd25c13e6b0 0x7fd25c152000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.716+0000 7fd25a59c640 1 -- 172.21.15.1:0/2054799881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c137210 con 0x7fd25c0c6ce0 2024-03-06T07:26:42.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.716+0000 7fd25a59c640 1 --2- 172.21.15.1:0/2054799881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd25c0c6ce0 0x7fd25c13cab0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fd25000c3f0 tx=0x7fd25000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.717+0000 7fd2437fe640 1 -- 172.21.15.1:0/2054799881 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2500052d0 con 0x7fd25c0c6ce0 2024-03-06T07:26:42.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.717+0000 7fd2437fe640 1 -- 172.21.15.1:0/2054799881 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd250005490 con 0x7fd25c0c6ce0 2024-03-06T07:26:42.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.717+0000 7fd2611c7640 1 -- 172.21.15.1:0/2054799881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd25c0cada0 con 0x7fd25c0c6ce0 2024-03-06T07:26:42.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.717+0000 7fd2437fe640 1 -- 172.21.15.1:0/2054799881 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd250007260 con 0x7fd25c0c6ce0 2024-03-06T07:26:42.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.717+0000 7fd2611c7640 1 -- 172.21.15.1:0/2054799881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd25c13cff0 con 0x7fd25c0c6ce0 2024-03-06T07:26:42.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.718+0000 7fd2611c7640 1 -- 172.21.15.1:0/2054799881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd25c0caf70 con 0x7fd25c0c6ce0 2024-03-06T07:26:42.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.718+0000 7fd2437fe640 1 -- 172.21.15.1:0/2054799881 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd250007400 con 0x7fd25c0c6ce0 2024-03-06T07:26:42.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.719+0000 7fd2437fe640 1 --2- 172.21.15.1:0/2054799881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd24806b4d0 0x7fd24806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.719+0000 7fd2437fe640 1 -- 172.21.15.1:0/2054799881 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fd250089b30 con 0x7fd25c0c6ce0 2024-03-06T07:26:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.719+0000 7fd25ad9d640 1 --2- 172.21.15.1:0/2054799881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd24806b4d0 0x7fd24806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.720+0000 7fd25ad9d640 1 --2- 172.21.15.1:0/2054799881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd24806b4d0 0x7fd24806d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fd25c154c70 tx=0x7fd24c00b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.723+0000 7fd2437fe640 1 -- 172.21.15.1:0/2054799881 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd250057e60 con 0x7fd25c0c6ce0 2024-03-06T07:26:42.904 INFO:tasks.ceph.mds.d.smithi098.stderr:2024-03-06T07:26:42.903+0000 7f2313d48640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:42.906 INFO:tasks.ceph.mds.b.smithi098.stderr:2024-03-06T07:26:42.904+0000 7ff26172f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:42.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.904+0000 7fd2611c7640 1 -- 172.21.15.1:0/2054799881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd25c0c85a0 con 0x7fd25c0c6ce0 2024-03-06T07:26:42.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.905+0000 7fd2437fe640 1 -- 172.21.15.1:0/2054799881 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v155) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd25c0c85a0 con 0x7fd25c0c6ce0 2024-03-06T07:26:42.907 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:26:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.907+0000 7fd2611c7640 1 -- 172.21.15.1:0/2054799881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd24806b4d0 msgr2=0x7fd24806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.907+0000 7fd2611c7640 1 --2- 172.21.15.1:0/2054799881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd24806b4d0 0x7fd24806d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fd25c154c70 tx=0x7fd24c00b040 comp rx=0 tx=0).stop 2024-03-06T07:26:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.907+0000 7fd2611c7640 1 -- 172.21.15.1:0/2054799881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd25c0c6ce0 msgr2=0x7fd25c13cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.907+0000 7fd2611c7640 1 --2- 172.21.15.1:0/2054799881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd25c0c6ce0 0x7fd25c13cab0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fd25000c3f0 tx=0x7fd25000c8b0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.907+0000 7fd2611c7640 1 -- 172.21.15.1:0/2054799881 shutdown_connections 2024-03-06T07:26:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.907+0000 7fd2611c7640 1 --2- 172.21.15.1:0/2054799881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd24806b4d0 0x7fd24806d990 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.907+0000 7fd2611c7640 1 --2- 172.21.15.1:0/2054799881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd25c13e6b0 0x7fd25c152000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.907+0000 7fd2611c7640 1 --2- 172.21.15.1:0/2054799881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd25c0c6ce0 0x7fd25c13cab0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.907+0000 7fd2611c7640 1 --2- 172.21.15.1:0/2054799881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd25c0c1a70 0x7fd25c1374b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.907+0000 7fd2611c7640 1 -- 172.21.15.1:0/2054799881 >> 172.21.15.1:0/2054799881 conn(0x7fd25c06da30 msgr2=0x7fd25c0bff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.907+0000 7fd2611c7640 1 -- 172.21.15.1:0/2054799881 shutdown_connections 2024-03-06T07:26:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:42.907+0000 7fd2611c7640 1 -- 172.21.15.1:0/2054799881 wait complete. 2024-03-06T07:26:42.919 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.021+0000 7efc74647640 1 Processor -- start 2024-03-06T07:26:43.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.021+0000 7efc74647640 1 -- start start 2024-03-06T07:26:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.022+0000 7efc74647640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efc6c0c6b10 0x7efc6c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.022+0000 7efc74647640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efc6c0bf9d0 0x7efc6c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.022+0000 7efc74647640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc6c0c2fc0 con 0x7efc6c152650 2024-03-06T07:26:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.022+0000 7efc74647640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc6c155800 con 0x7efc6c0c6b10 2024-03-06T07:26:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.022+0000 7efc74647640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc6c158da0 con 0x7efc6c0bf9d0 2024-03-06T07:26:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.022+0000 7efc71be2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efc6c0c6b10 0x7efc6c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.022+0000 7efc723e3640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efc6c152650 0x7efc6c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37344/0 (socket says 172.21.15.1:37344) 2024-03-06T07:26:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.022+0000 7efc723e3640 1 -- 172.21.15.1:0/3555663208 learned_addr learned my addr 172.21.15.1:0/3555663208 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:43.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.022+0000 7efc72be4640 1 --2- 172.21.15.1:0/3555663208 >> v2:172.21.15.98:3300/0 conn(0x7efc6c0bf9d0 0x7efc6c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.023+0000 7efc71be2640 1 -- 172.21.15.1:0/3555663208 >> v2:172.21.15.98:3300/0 conn(0x7efc6c0bf9d0 msgr2=0x7efc6c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.023+0000 7efc71be2640 1 --2- 172.21.15.1:0/3555663208 >> v2:172.21.15.98:3300/0 conn(0x7efc6c0bf9d0 0x7efc6c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.023+0000 7efc71be2640 1 -- 172.21.15.1:0/3555663208 >> v1:172.21.15.1:6789/0 conn(0x7efc6c152650 legacy=0x7efc6c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.023+0000 7efc71be2640 1 -- 172.21.15.1:0/3555663208 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6c0c6690 con 0x7efc6c0c6b10 2024-03-06T07:26:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.023+0000 7efc713e1640 1 -- 172.21.15.1:0/3555663208 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334880593 0 0) 0x7efc6c0c2fc0 con 0x7efc6c152650 2024-03-06T07:26:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.023+0000 7efc72be4640 1 --2- 172.21.15.1:0/3555663208 >> v2:172.21.15.98:3300/0 conn(0x7efc6c0bf9d0 0x7efc6c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:43.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.024+0000 7efc71be2640 1 --2- 172.21.15.1:0/3555663208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc6c0c6b10 0x7efc6c155c90 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7efc6c1597b0 tx=0x7efc5c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826766e79b8e86b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.024+0000 7efc713e1640 1 -- 172.21.15.1:0/3555663208 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5c0315a0 con 0x7efc6c0c6b10 2024-03-06T07:26:43.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.024+0000 7efc713e1640 1 -- 172.21.15.1:0/3555663208 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc5c037070 con 0x7efc6c0c6b10 2024-03-06T07:26:43.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.024+0000 7efc713e1640 1 -- 172.21.15.1:0/3555663208 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5c031960 con 0x7efc6c0c6b10 2024-03-06T07:26:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.024+0000 7efc74647640 1 -- 172.21.15.1:0/3555663208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc6c0c6b10 msgr2=0x7efc6c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.024+0000 7efc74647640 1 --2- 172.21.15.1:0/3555663208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc6c0c6b10 0x7efc6c155c90 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7efc6c1597b0 tx=0x7efc5c02b840 comp rx=0 tx=0).stop 2024-03-06T07:26:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.024+0000 7efc74647640 1 -- 172.21.15.1:0/3555663208 shutdown_connections 2024-03-06T07:26:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.024+0000 7efc74647640 1 --2- 172.21.15.1:0/3555663208 >> v2:172.21.15.98:3300/0 conn(0x7efc6c0bf9d0 0x7efc6c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.024+0000 7efc74647640 1 --2- 172.21.15.1:0/3555663208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc6c0c6b10 0x7efc6c155c90 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.024+0000 7efc74647640 1 -- 172.21.15.1:0/3555663208 >> 172.21.15.1:0/3555663208 conn(0x7efc6c066ee0 msgr2=0x7efc6c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.024+0000 7efc74647640 1 -- 172.21.15.1:0/3555663208 shutdown_connections 2024-03-06T07:26:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.024+0000 7efc74647640 1 -- 172.21.15.1:0/3555663208 wait complete. 2024-03-06T07:26:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.026+0000 7efc74647640 1 Processor -- start 2024-03-06T07:26:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.026+0000 7efc74647640 1 -- start start 2024-03-06T07:26:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.026+0000 7efc74647640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc6c0bf9d0 0x7efc6c0c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.026+0000 7efc74647640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc6c152650 0x7efc6c14b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.026+0000 7efc74647640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc6c156490 0x7efc6c14b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.026+0000 7efc74647640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c1551b0 con 0x7efc6c156490 2024-03-06T07:26:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.026+0000 7efc74647640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c0c6fa0 con 0x7efc6c152650 2024-03-06T07:26:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.026+0000 7efc74647640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c155490 con 0x7efc6c0bf9d0 2024-03-06T07:26:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.026+0000 7efc72be4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc6c156490 0x7efc6c14b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.026+0000 7efc72be4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc6c156490 0x7efc6c14b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38866/0 (socket says 172.21.15.1:38866) 2024-03-06T07:26:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.026+0000 7efc72be4640 1 -- 172.21.15.1:0/1361100099 learned_addr learned my addr 172.21.15.1:0/1361100099 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.026+0000 7efc71be2640 1 --2- 172.21.15.1:0/1361100099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc6c152650 0x7efc6c14b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.026+0000 7efc723e3640 1 --2- 172.21.15.1:0/1361100099 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc6c0bf9d0 0x7efc6c0c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.027+0000 7efc72be4640 1 -- 172.21.15.1:0/1361100099 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc6c0bf9d0 msgr2=0x7efc6c0c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.027+0000 7efc72be4640 1 --2- 172.21.15.1:0/1361100099 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc6c0bf9d0 0x7efc6c0c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.027+0000 7efc72be4640 1 -- 172.21.15.1:0/1361100099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc6c152650 msgr2=0x7efc6c14b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.027+0000 7efc72be4640 1 --2- 172.21.15.1:0/1361100099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc6c152650 0x7efc6c14b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.027+0000 7efc72be4640 1 -- 172.21.15.1:0/1361100099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6c14bc60 con 0x7efc6c156490 2024-03-06T07:26:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.027+0000 7efc723e3640 1 --2- 172.21.15.1:0/1361100099 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc6c0bf9d0 0x7efc6c0c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.027+0000 7efc72be4640 1 --2- 172.21.15.1:0/1361100099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc6c156490 0x7efc6c14b720 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7efc6400efc0 tx=0x7efc6400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.027+0000 7efc62ffd640 1 -- 172.21.15.1:0/1361100099 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc64018030 con 0x7efc6c156490 2024-03-06T07:26:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.027+0000 7efc62ffd640 1 -- 172.21.15.1:0/1361100099 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6400cdf0 con 0x7efc6c156490 2024-03-06T07:26:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.027+0000 7efc74647640 1 -- 172.21.15.1:0/1361100099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc6c154870 con 0x7efc6c156490 2024-03-06T07:26:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.027+0000 7efc62ffd640 1 -- 172.21.15.1:0/1361100099 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc64003bb0 con 0x7efc6c156490 2024-03-06T07:26:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.027+0000 7efc74647640 1 -- 172.21.15.1:0/1361100099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc6c1ef240 con 0x7efc6c156490 2024-03-06T07:26:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.028+0000 7efc74647640 1 -- 172.21.15.1:0/1361100099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc6c067fd0 con 0x7efc6c156490 2024-03-06T07:26:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.028+0000 7efc62ffd640 1 -- 172.21.15.1:0/1361100099 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc64005d10 con 0x7efc6c156490 2024-03-06T07:26:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.029+0000 7efc62ffd640 1 --2- 172.21.15.1:0/1361100099 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc5006b4d0 0x7efc5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.029+0000 7efc62ffd640 1 -- 172.21.15.1:0/1361100099 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7efc6408afd0 con 0x7efc6c156490 2024-03-06T07:26:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.029+0000 7efc723e3640 1 --2- 172.21.15.1:0/1361100099 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc5006b4d0 0x7efc5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.029+0000 7efc723e3640 1 --2- 172.21.15.1:0/1361100099 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc5006b4d0 0x7efc5006d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7efc68000f60 tx=0x7efc68007c10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.033+0000 7efc62ffd640 1 -- 172.21.15.1:0/1361100099 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc64022070 con 0x7efc6c156490 2024-03-06T07:26:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.213+0000 7efc74647640 1 -- 172.21.15.1:0/1361100099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efc6c0b01a0 con 0x7efc6c156490 2024-03-06T07:26:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.214+0000 7efc62ffd640 1 -- 172.21.15.1:0/1361100099 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v155) v1 ==== 100+0+11 (secure 0 0 0) 0x7efc64010040 con 0x7efc6c156490 2024-03-06T07:26:43.216 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:26:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.216+0000 7efc74647640 1 -- 172.21.15.1:0/1361100099 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc5006b4d0 msgr2=0x7efc5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.216+0000 7efc74647640 1 --2- 172.21.15.1:0/1361100099 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc5006b4d0 0x7efc5006d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7efc68000f60 tx=0x7efc68007c10 comp rx=0 tx=0).stop 2024-03-06T07:26:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.216+0000 7efc74647640 1 -- 172.21.15.1:0/1361100099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc6c156490 msgr2=0x7efc6c14b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.216+0000 7efc74647640 1 --2- 172.21.15.1:0/1361100099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc6c156490 0x7efc6c14b720 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7efc6400efc0 tx=0x7efc6400c490 comp rx=0 tx=0).stop 2024-03-06T07:26:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.216+0000 7efc74647640 1 -- 172.21.15.1:0/1361100099 shutdown_connections 2024-03-06T07:26:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.216+0000 7efc74647640 1 --2- 172.21.15.1:0/1361100099 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efc5006b4d0 0x7efc5006d990 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.216+0000 7efc74647640 1 --2- 172.21.15.1:0/1361100099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc6c156490 0x7efc6c14b720 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.216+0000 7efc74647640 1 --2- 172.21.15.1:0/1361100099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc6c152650 0x7efc6c14b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.216+0000 7efc74647640 1 --2- 172.21.15.1:0/1361100099 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc6c0bf9d0 0x7efc6c0c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.216+0000 7efc74647640 1 -- 172.21.15.1:0/1361100099 >> 172.21.15.1:0/1361100099 conn(0x7efc6c066ee0 msgr2=0x7efc6c0c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.216+0000 7efc74647640 1 -- 172.21.15.1:0/1361100099 shutdown_connections 2024-03-06T07:26:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.216+0000 7efc74647640 1 -- 172.21.15.1:0/1361100099 wait complete. 2024-03-06T07:26:43.229 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-06T07:26:43.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.331+0000 7f7669a02640 1 Processor -- start 2024-03-06T07:26:43.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.331+0000 7f7669a02640 1 -- start start 2024-03-06T07:26:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.331+0000 7f7669a02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7664152740 0x7f76640c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.332+0000 7f7669a02640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7664155330 con 0x7f76640c7790 2024-03-06T07:26:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.332+0000 7f7669a02640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7664155610 con 0x7f76640c7330 2024-03-06T07:26:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.332+0000 7f7669a02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76641558f0 con 0x7f7664152740 2024-03-06T07:26:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.332+0000 7f76637fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f76640c7790 0x7f766406d770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37350/0 (socket says 172.21.15.1:37350) 2024-03-06T07:26:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.332+0000 7f76637fe640 1 -- 172.21.15.1:0/439407939 learned_addr learned my addr 172.21.15.1:0/439407939 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.332+0000 7f7662ffd640 1 --2- 172.21.15.1:0/439407939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7664152740 0x7f76640c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.332+0000 7f76627fc640 1 -- 172.21.15.1:0/439407939 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735911486 0 0) 0x7f7664155610 con 0x7f76640c7330 2024-03-06T07:26:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.332+0000 7f76627fc640 1 -- 172.21.15.1:0/439407939 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7650003520 con 0x7f76640c7330 2024-03-06T07:26:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.332+0000 7f76627fc640 1 -- 172.21.15.1:0/439407939 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018021618 0 0) 0x7f7664155330 con 0x7f76640c7790 2024-03-06T07:26:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.332+0000 7f76627fc640 1 -- 172.21.15.1:0/439407939 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7664155610 con 0x7f76640c7790 2024-03-06T07:26:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.332+0000 7f7662ffd640 1 -- 172.21.15.1:0/439407939 >> v1:172.21.15.98:6789/0 conn(0x7f76640c7330 legacy=0x7f766406d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.332+0000 7f7662ffd640 1 -- 172.21.15.1:0/439407939 >> v1:172.21.15.1:6789/0 conn(0x7f76640c7790 legacy=0x7f766406d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.332+0000 7f7662ffd640 1 -- 172.21.15.1:0/439407939 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7664155bd0 con 0x7f7664152740 2024-03-06T07:26:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.333+0000 7f7662ffd640 1 --2- 172.21.15.1:0/439407939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7664152740 0x7f76640c6a50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f76540097d0 tx=0x7f765402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d36fbd834c60cb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.333+0000 7f76627fc640 1 -- 172.21.15.1:0/439407939 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654035020 con 0x7f7664152740 2024-03-06T07:26:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.333+0000 7f76627fc640 1 -- 172.21.15.1:0/439407939 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7654031800 con 0x7f7664152740 2024-03-06T07:26:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.333+0000 7f76627fc640 1 -- 172.21.15.1:0/439407939 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654031bb0 con 0x7f7664152740 2024-03-06T07:26:43.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.334+0000 7f7669a02640 1 -- 172.21.15.1:0/439407939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7664152740 msgr2=0x7f76640c6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.334+0000 7f7669a02640 1 --2- 172.21.15.1:0/439407939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7664152740 0x7f76640c6a50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f76540097d0 tx=0x7f765402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.334+0000 7f7669a02640 1 -- 172.21.15.1:0/439407939 shutdown_connections 2024-03-06T07:26:43.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.334+0000 7f7669a02640 1 --2- 172.21.15.1:0/439407939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7664152740 0x7f76640c6a50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.334+0000 7f7669a02640 1 -- 172.21.15.1:0/439407939 >> 172.21.15.1:0/439407939 conn(0x7f7664066d00 msgr2=0x7f7664067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.334+0000 7f7669a02640 1 -- 172.21.15.1:0/439407939 shutdown_connections 2024-03-06T07:26:43.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.334+0000 7f7669a02640 1 -- 172.21.15.1:0/439407939 wait complete. 2024-03-06T07:26:43.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.335+0000 7f7669a02640 1 Processor -- start 2024-03-06T07:26:43.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.335+0000 7f7669a02640 1 -- start start 2024-03-06T07:26:43.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.335+0000 7f7669a02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76640c7330 0x7f76640c4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.335+0000 7f7669a02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76640c7790 0x7f76640c9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.335+0000 7f7669a02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7664152740 0x7f766413e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.335+0000 7f7669a02640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76640c6f90 con 0x7f76640c7790 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.335+0000 7f7669a02640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7664152b20 con 0x7f7664152740 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.335+0000 7f7669a02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76640c7110 con 0x7f76640c7330 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.336+0000 7f765bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76640c7790 0x7f76640c9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.336+0000 7f7662ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76640c7330 0x7f76640c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.336+0000 7f7662ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76640c7330 0x7f76640c4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:49230/0 (socket says 172.21.15.1:49230) 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.336+0000 7f7662ffd640 1 -- 172.21.15.1:0/3746663176 learned_addr learned my addr 172.21.15.1:0/3746663176 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.336+0000 7f76637fe640 1 --2- 172.21.15.1:0/3746663176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7664152740 0x7f766413e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.336+0000 7f765bfff640 1 -- 172.21.15.1:0/3746663176 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76640c7330 msgr2=0x7f76640c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.336+0000 7f765bfff640 1 --2- 172.21.15.1:0/3746663176 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76640c7330 0x7f76640c4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.336+0000 7f765bfff640 1 -- 172.21.15.1:0/3746663176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7664152740 msgr2=0x7f766413e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.336+0000 7f765bfff640 1 --2- 172.21.15.1:0/3746663176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7664152740 0x7f766413e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.336+0000 7f765bfff640 1 -- 172.21.15.1:0/3746663176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76641eb810 con 0x7f76640c7790 2024-03-06T07:26:43.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.336+0000 7f7662ffd640 1 --2- 172.21.15.1:0/3746663176 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76640c7330 0x7f76640c4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:43.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.336+0000 7f765bfff640 1 --2- 172.21.15.1:0/3746663176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76640c7790 0x7f76640c9b10 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f764c00c3c0 tx=0x7f764c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.336+0000 7f7660ff9640 1 -- 172.21.15.1:0/3746663176 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764c017070 con 0x7f76640c7790 2024-03-06T07:26:43.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.336+0000 7f7660ff9640 1 -- 172.21.15.1:0/3746663176 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764c0052d0 con 0x7f76640c7790 2024-03-06T07:26:43.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.336+0000 7f7669a02640 1 -- 172.21.15.1:0/3746663176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7664156520 con 0x7f76640c7790 2024-03-06T07:26:43.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.336+0000 7f7660ff9640 1 -- 172.21.15.1:0/3746663176 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764c003830 con 0x7f76640c7790 2024-03-06T07:26:43.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.336+0000 7f7669a02640 1 -- 172.21.15.1:0/3746663176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76640ca050 con 0x7f76640c7790 2024-03-06T07:26:43.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.337+0000 7f7669a02640 1 -- 172.21.15.1:0/3746663176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7664067fd0 con 0x7f76640c7790 2024-03-06T07:26:43.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.338+0000 7f7660ff9640 1 -- 172.21.15.1:0/3746663176 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f764c01a070 con 0x7f76640c7790 2024-03-06T07:26:43.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.338+0000 7f7660ff9640 1 --2- 172.21.15.1:0/3746663176 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764406b4d0 0x7f764406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.338+0000 7f7660ff9640 1 -- 172.21.15.1:0/3746663176 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f764c0898f0 con 0x7f76640c7790 2024-03-06T07:26:43.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.338+0000 7f7662ffd640 1 --2- 172.21.15.1:0/3746663176 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764406b4d0 0x7f764406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.339+0000 7f7662ffd640 1 --2- 172.21.15.1:0/3746663176 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764406b4d0 0x7f764406d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f766414cfa0 tx=0x7f7654008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.342+0000 7f7660ff9640 1 -- 172.21.15.1:0/3746663176 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f764c057c20 con 0x7f76640c7790 2024-03-06T07:26:43.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.520+0000 7f7669a02640 1 -- 172.21.15.1:0/3746663176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f76640af5c0 con 0x7f76640c7790 2024-03-06T07:26:43.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.521+0000 7f7660ff9640 1 -- 172.21.15.1:0/3746663176 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v155) v1 ==== 105+0+11 (secure 0 0 0) 0x7f764c05bcc0 con 0x7f76640c7790 2024-03-06T07:26:43.523 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:26:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.523+0000 7f7669a02640 1 -- 172.21.15.1:0/3746663176 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764406b4d0 msgr2=0x7f764406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.523+0000 7f7669a02640 1 --2- 172.21.15.1:0/3746663176 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764406b4d0 0x7f764406d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f766414cfa0 tx=0x7f7654008030 comp rx=0 tx=0).stop 2024-03-06T07:26:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.523+0000 7f7669a02640 1 -- 172.21.15.1:0/3746663176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76640c7790 msgr2=0x7f76640c9b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.523+0000 7f7669a02640 1 --2- 172.21.15.1:0/3746663176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76640c7790 0x7f76640c9b10 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f764c00c3c0 tx=0x7f764c00c880 comp rx=0 tx=0).stop 2024-03-06T07:26:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.523+0000 7f7669a02640 1 -- 172.21.15.1:0/3746663176 shutdown_connections 2024-03-06T07:26:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.523+0000 7f7669a02640 1 --2- 172.21.15.1:0/3746663176 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f764406b4d0 0x7f764406d990 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.523+0000 7f7669a02640 1 --2- 172.21.15.1:0/3746663176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7664152740 0x7f766413e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.523+0000 7f7669a02640 1 --2- 172.21.15.1:0/3746663176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76640c7790 0x7f76640c9b10 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.523+0000 7f7669a02640 1 --2- 172.21.15.1:0/3746663176 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76640c7330 0x7f76640c4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.523+0000 7f7669a02640 1 -- 172.21.15.1:0/3746663176 >> 172.21.15.1:0/3746663176 conn(0x7f7664066d00 msgr2=0x7f7664152ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.523+0000 7f7669a02640 1 -- 172.21.15.1:0/3746663176 shutdown_connections 2024-03-06T07:26:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.523+0000 7f7669a02640 1 -- 172.21.15.1:0/3746663176 wait complete. 2024-03-06T07:26:43.536 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-06T07:26:43.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.638+0000 7f3b1330f640 1 Processor -- start 2024-03-06T07:26:43.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.638+0000 7f3b1330f640 1 -- start start 2024-03-06T07:26:43.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.639+0000 7f3b1330f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3b0c152690 0x7f3b0c155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.639+0000 7f3b1330f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3b0c156cd0 0x7f3b0c157120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.639+0000 7f3b1330f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b0c1561e0 con 0x7f3b0c0c6730 2024-03-06T07:26:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.639+0000 7f3b1330f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b0c0c3300 con 0x7f3b0c156cd0 2024-03-06T07:26:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.639+0000 7f3b1330f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b0c0c35a0 con 0x7f3b0c152690 2024-03-06T07:26:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.639+0000 7f3b118ac640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3b0c156cd0 0x7f3b0c157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.639+0000 7f3b118ac640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3b0c156cd0 0x7f3b0c157120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38876/0 (socket says 172.21.15.1:38876) 2024-03-06T07:26:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.639+0000 7f3b108aa640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3b0c0c6730 0x7f3b0c0c57d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37362/0 (socket says 172.21.15.1:37362) 2024-03-06T07:26:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.639+0000 7f3b118ac640 1 -- 172.21.15.1:0/1197382365 learned_addr learned my addr 172.21.15.1:0/1197382365 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.639+0000 7f3b110ab640 1 --2- 172.21.15.1:0/1197382365 >> v2:172.21.15.98:3300/0 conn(0x7f3b0c152690 0x7f3b0c155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.639+0000 7f3b118ac640 1 -- 172.21.15.1:0/1197382365 >> v2:172.21.15.98:3300/0 conn(0x7f3b0c152690 msgr2=0x7f3b0c155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.639+0000 7f3b118ac640 1 --2- 172.21.15.1:0/1197382365 >> v2:172.21.15.98:3300/0 conn(0x7f3b0c152690 0x7f3b0c155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.639+0000 7f3b118ac640 1 -- 172.21.15.1:0/1197382365 >> v1:172.21.15.1:6789/0 conn(0x7f3b0c0c6730 legacy=0x7f3b0c0c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.639+0000 7f3b118ac640 1 -- 172.21.15.1:0/1197382365 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0c0c3880 con 0x7f3b0c156cd0 2024-03-06T07:26:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.639+0000 7f3b0bfff640 1 -- 172.21.15.1:0/1197382365 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886244662 0 0) 0x7f3b00001110 con 0x7f3b0c0c6730 2024-03-06T07:26:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.640+0000 7f3b118ac640 1 --2- 172.21.15.1:0/1197382365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b0c156cd0 0x7f3b0c157120 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c0ab5f0 tx=0x7f3afc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b29a50bc97f4765 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.640+0000 7f3b0bfff640 1 -- 172.21.15.1:0/1197382365 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc035020 con 0x7f3b0c156cd0 2024-03-06T07:26:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.640+0000 7f3b0bfff640 1 -- 172.21.15.1:0/1197382365 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3afc031870 con 0x7f3b0c156cd0 2024-03-06T07:26:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.640+0000 7f3b0bfff640 1 -- 172.21.15.1:0/1197382365 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc031c40 con 0x7f3b0c156cd0 2024-03-06T07:26:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.641+0000 7f3b1330f640 1 -- 172.21.15.1:0/1197382365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b0c156cd0 msgr2=0x7f3b0c157120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.641+0000 7f3b1330f640 1 --2- 172.21.15.1:0/1197382365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b0c156cd0 0x7f3b0c157120 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c0ab5f0 tx=0x7f3afc02b910 comp rx=0 tx=0).stop 2024-03-06T07:26:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.641+0000 7f3b1330f640 1 -- 172.21.15.1:0/1197382365 shutdown_connections 2024-03-06T07:26:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.641+0000 7f3b1330f640 1 --2- 172.21.15.1:0/1197382365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b0c156cd0 0x7f3b0c157120 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.641+0000 7f3b1330f640 1 --2- 172.21.15.1:0/1197382365 >> v2:172.21.15.98:3300/0 conn(0x7f3b0c152690 0x7f3b0c155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.641+0000 7f3b1330f640 1 -- 172.21.15.1:0/1197382365 >> 172.21.15.1:0/1197382365 conn(0x7f3b0c06b410 msgr2=0x7f3b0c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.641+0000 7f3b1330f640 1 -- 172.21.15.1:0/1197382365 shutdown_connections 2024-03-06T07:26:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.641+0000 7f3b1330f640 1 -- 172.21.15.1:0/1197382365 wait complete. 2024-03-06T07:26:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.642+0000 7f3b1330f640 1 Processor -- start 2024-03-06T07:26:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.642+0000 7f3b1330f640 1 -- start start 2024-03-06T07:26:43.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b1330f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b0c0c6730 0x7f3b0c0c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b1330f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b0c152690 0x7f3b0c0a6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b1330f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3b0c156cd0 0x7f3b0c0a7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b1330f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0c0c5ee0 con 0x7f3b0c0c6730 2024-03-06T07:26:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b1330f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0c1567b0 con 0x7f3b0c152690 2024-03-06T07:26:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b1330f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0c0c6060 con 0x7f3b0c156cd0 2024-03-06T07:26:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b110ab640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b0c0c6730 0x7f3b0c0c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b110ab640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b0c0c6730 0x7f3b0c0c7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38878/0 (socket says 172.21.15.1:38878) 2024-03-06T07:26:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b110ab640 1 -- 172.21.15.1:0/397854059 learned_addr learned my addr 172.21.15.1:0/397854059 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b118ac640 1 --2- 172.21.15.1:0/397854059 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3b0c156cd0 0x7f3b0c0a7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b108aa640 1 --2- 172.21.15.1:0/397854059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b0c152690 0x7f3b0c0a6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b110ab640 1 -- 172.21.15.1:0/397854059 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3b0c156cd0 msgr2=0x7f3b0c0a7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b110ab640 1 --2- 172.21.15.1:0/397854059 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3b0c156cd0 0x7f3b0c0a7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b110ab640 1 -- 172.21.15.1:0/397854059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b0c152690 msgr2=0x7f3b0c0a6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b110ab640 1 --2- 172.21.15.1:0/397854059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b0c152690 0x7f3b0c0a6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b110ab640 1 -- 172.21.15.1:0/397854059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0c0aa580 con 0x7f3b0c0c6730 2024-03-06T07:26:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b110ab640 1 --2- 172.21.15.1:0/397854059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b0c0c6730 0x7f3b0c0c7eb0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400efc0 tx=0x7f3b0400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b09ffb640 1 -- 172.21.15.1:0/397854059 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b04018020 con 0x7f3b0c0c6730 2024-03-06T07:26:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b09ffb640 1 -- 172.21.15.1:0/397854059 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b04014480 con 0x7f3b0c0c6730 2024-03-06T07:26:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b1330f640 1 -- 172.21.15.1:0/397854059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b0c1eb5b0 con 0x7f3b0c0c6730 2024-03-06T07:26:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.643+0000 7f3b09ffb640 1 -- 172.21.15.1:0/397854059 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b04014850 con 0x7f3b0c0c6730 2024-03-06T07:26:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.644+0000 7f3b1330f640 1 -- 172.21.15.1:0/397854059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b0c154690 con 0x7f3b0c0c6730 2024-03-06T07:26:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.645+0000 7f3b1330f640 1 -- 172.21.15.1:0/397854059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b0c067fd0 con 0x7f3b0c0c6730 2024-03-06T07:26:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.645+0000 7f3b09ffb640 1 -- 172.21.15.1:0/397854059 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b04005d10 con 0x7f3b0c0c6730 2024-03-06T07:26:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.645+0000 7f3b09ffb640 1 --2- 172.21.15.1:0/397854059 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3af806b500 0x7f3af806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.645+0000 7f3b09ffb640 1 -- 172.21.15.1:0/397854059 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f3b0408b000 con 0x7f3b0c0c6730 2024-03-06T07:26:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.646+0000 7f3b108aa640 1 --2- 172.21.15.1:0/397854059 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3af806b500 0x7f3af806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.646+0000 7f3b108aa640 1 --2- 172.21.15.1:0/397854059 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3af806b500 0x7f3af806d9c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c06d490 tx=0x7f3b00007de0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.650+0000 7f3b09ffb640 1 -- 172.21.15.1:0/397854059 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b04022070 con 0x7f3b0c0c6730 2024-03-06T07:26:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.829+0000 7f3b1330f640 1 -- 172.21.15.1:0/397854059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7f3b0c0c2c40 con 0x7f3b0c0c6730 2024-03-06T07:26:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.829+0000 7f3b09ffb640 1 -- 172.21.15.1:0/397854059 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v155) v1 ==== 101+0+11 (secure 0 0 0) 0x7f3b04010040 con 0x7f3b0c0c6730 2024-03-06T07:26:43.832 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:26:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.832+0000 7f3b1330f640 1 -- 172.21.15.1:0/397854059 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3af806b500 msgr2=0x7f3af806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.832+0000 7f3b1330f640 1 --2- 172.21.15.1:0/397854059 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3af806b500 0x7f3af806d9c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c06d490 tx=0x7f3b00007de0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.832+0000 7f3b1330f640 1 -- 172.21.15.1:0/397854059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b0c0c6730 msgr2=0x7f3b0c0c7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.832+0000 7f3b1330f640 1 --2- 172.21.15.1:0/397854059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b0c0c6730 0x7f3b0c0c7eb0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400efc0 tx=0x7f3b0400c490 comp rx=0 tx=0).stop 2024-03-06T07:26:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.832+0000 7f3b1330f640 1 -- 172.21.15.1:0/397854059 shutdown_connections 2024-03-06T07:26:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.832+0000 7f3b1330f640 1 --2- 172.21.15.1:0/397854059 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3af806b500 0x7f3af806d9c0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.832+0000 7f3b1330f640 1 --2- 172.21.15.1:0/397854059 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3b0c156cd0 0x7f3b0c0a7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.832+0000 7f3b1330f640 1 --2- 172.21.15.1:0/397854059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b0c152690 0x7f3b0c0a6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.832+0000 7f3b1330f640 1 --2- 172.21.15.1:0/397854059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b0c0c6730 0x7f3b0c0c7eb0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.832+0000 7f3b1330f640 1 -- 172.21.15.1:0/397854059 >> 172.21.15.1:0/397854059 conn(0x7f3b0c06b410 msgr2=0x7f3b0c0c57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.832+0000 7f3b1330f640 1 -- 172.21.15.1:0/397854059 shutdown_connections 2024-03-06T07:26:43.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.832+0000 7f3b1330f640 1 -- 172.21.15.1:0/397854059 wait complete. 2024-03-06T07:26:43.845 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:43.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.944+0000 7f03058a4640 1 Processor -- start 2024-03-06T07:26:43.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.944+0000 7f03058a4640 1 -- start start 2024-03-06T07:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.945+0000 7f03058a4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0300149a90 0x7f03000c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.945+0000 7f03058a4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f03000c1260 0x7f03000c16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.945+0000 7f03058a4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03001530a0 con 0x7f03001565a0 2024-03-06T07:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.945+0000 7f03058a4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0300153380 con 0x7f0300149a90 2024-03-06T07:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.945+0000 7f03058a4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0300153660 con 0x7f03000c1260 2024-03-06T07:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.945+0000 7f02feffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0300149a90 0x7f03000c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.945+0000 7f02ff7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f03001565a0 0x7f0300156a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37366/0 (socket says 172.21.15.1:37366) 2024-03-06T07:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.945+0000 7f02ff7fe640 1 -- 172.21.15.1:0/942620566 learned_addr learned my addr 172.21.15.1:0/942620566 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.945+0000 7f02fe7fc640 1 --2- 172.21.15.1:0/942620566 >> v2:172.21.15.98:3300/0 conn(0x7f03000c1260 0x7f03000c16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.945+0000 7f02feffd640 1 -- 172.21.15.1:0/942620566 >> v2:172.21.15.98:3300/0 conn(0x7f03000c1260 msgr2=0x7f03000c16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.945+0000 7f02feffd640 1 --2- 172.21.15.1:0/942620566 >> v2:172.21.15.98:3300/0 conn(0x7f03000c1260 0x7f03000c16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.945+0000 7f02feffd640 1 -- 172.21.15.1:0/942620566 >> v1:172.21.15.1:6789/0 conn(0x7f03001565a0 legacy=0x7f0300156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.945+0000 7f02fdffb640 1 -- 172.21.15.1:0/942620566 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345302509 0 0) 0x7f03001530a0 con 0x7f03001565a0 2024-03-06T07:26:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.945+0000 7f02feffd640 1 -- 172.21.15.1:0/942620566 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0300153940 con 0x7f0300149a90 2024-03-06T07:26:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.946+0000 7f02feffd640 1 --2- 172.21.15.1:0/942620566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0300149a90 0x7f03000c4760 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f03000a6520 tx=0x7f02f802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b1be1e75aaa89f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.946+0000 7f02fdffb640 1 -- 172.21.15.1:0/942620566 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f80315a0 con 0x7f0300149a90 2024-03-06T07:26:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.946+0000 7f02fdffb640 1 -- 172.21.15.1:0/942620566 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f8037070 con 0x7f0300149a90 2024-03-06T07:26:43.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.946+0000 7f02fdffb640 1 -- 172.21.15.1:0/942620566 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f8031960 con 0x7f0300149a90 2024-03-06T07:26:43.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.947+0000 7f03058a4640 1 -- 172.21.15.1:0/942620566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0300149a90 msgr2=0x7f03000c4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.947+0000 7f03058a4640 1 --2- 172.21.15.1:0/942620566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0300149a90 0x7f03000c4760 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f03000a6520 tx=0x7f02f802b840 comp rx=0 tx=0).stop 2024-03-06T07:26:43.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.947+0000 7f03058a4640 1 -- 172.21.15.1:0/942620566 shutdown_connections 2024-03-06T07:26:43.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.947+0000 7f03058a4640 1 --2- 172.21.15.1:0/942620566 >> v2:172.21.15.98:3300/0 conn(0x7f03000c1260 0x7f03000c16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.947+0000 7f03058a4640 1 --2- 172.21.15.1:0/942620566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0300149a90 0x7f03000c4760 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.947+0000 7f03058a4640 1 -- 172.21.15.1:0/942620566 >> 172.21.15.1:0/942620566 conn(0x7f03000c6860 msgr2=0x7f03000c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.947+0000 7f03058a4640 1 -- 172.21.15.1:0/942620566 shutdown_connections 2024-03-06T07:26:43.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.947+0000 7f03058a4640 1 -- 172.21.15.1:0/942620566 wait complete. 2024-03-06T07:26:43.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.948+0000 7f03058a4640 1 Processor -- start 2024-03-06T07:26:43.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.948+0000 7f03058a4640 1 -- start start 2024-03-06T07:26:43.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f03058a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03000c1260 0x7f03000bff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f03058a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0300149a90 0x7f030014f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f03058a4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03001565a0 0x7f030014f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f03058a4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0300157120 con 0x7f03001565a0 2024-03-06T07:26:43.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f03058a4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03000c4ca0 con 0x7f0300149a90 2024-03-06T07:26:43.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f03058a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03001572a0 con 0x7f03000c1260 2024-03-06T07:26:43.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f02ff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03001565a0 0x7f030014f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f02ff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03001565a0 0x7f030014f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38888/0 (socket says 172.21.15.1:38888) 2024-03-06T07:26:43.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f02ff7fe640 1 -- 172.21.15.1:0/1507679494 learned_addr learned my addr 172.21.15.1:0/1507679494 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:43.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f02feffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03000c1260 0x7f03000bff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f02fe7fc640 1 --2- 172.21.15.1:0/1507679494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0300149a90 0x7f030014f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f02ff7fe640 1 -- 172.21.15.1:0/1507679494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03000c1260 msgr2=0x7f03000bff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f02ff7fe640 1 --2- 172.21.15.1:0/1507679494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03000c1260 0x7f03000bff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f02ff7fe640 1 -- 172.21.15.1:0/1507679494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0300149a90 msgr2=0x7f030014f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f02ff7fe640 1 --2- 172.21.15.1:0/1507679494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0300149a90 0x7f030014f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f02ff7fe640 1 -- 172.21.15.1:0/1507679494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03001eb570 con 0x7f03001565a0 2024-03-06T07:26:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f02ff7fe640 1 --2- 172.21.15.1:0/1507679494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03001565a0 0x7f030014f8f0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f02f000c7a0 tx=0x7f02f000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f02f7fff640 1 -- 172.21.15.1:0/1507679494 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f0018070 con 0x7f03001565a0 2024-03-06T07:26:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f02f7fff640 1 -- 172.21.15.1:0/1507679494 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f000a640 con 0x7f03001565a0 2024-03-06T07:26:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f02f7fff640 1 -- 172.21.15.1:0/1507679494 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f000aab0 con 0x7f03001565a0 2024-03-06T07:26:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.949+0000 7f03058a4640 1 -- 172.21.15.1:0/1507679494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03001eb850 con 0x7f03001565a0 2024-03-06T07:26:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.950+0000 7f03058a4640 1 -- 172.21.15.1:0/1507679494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03000c0440 con 0x7f03001565a0 2024-03-06T07:26:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.950+0000 7f03058a4640 1 -- 172.21.15.1:0/1507679494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0300067fd0 con 0x7f03001565a0 2024-03-06T07:26:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.951+0000 7f02f7fff640 1 -- 172.21.15.1:0/1507679494 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02f001b070 con 0x7f03001565a0 2024-03-06T07:26:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.951+0000 7f02f7fff640 1 --2- 172.21.15.1:0/1507679494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02e006b500 0x7f02e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.951+0000 7f02f7fff640 1 -- 172.21.15.1:0/1507679494 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f02f0089c20 con 0x7f03001565a0 2024-03-06T07:26:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.951+0000 7f02feffd640 1 --2- 172.21.15.1:0/1507679494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02e006b500 0x7f02e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.952+0000 7f02feffd640 1 --2- 172.21.15.1:0/1507679494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02e006b500 0x7f02e006d9c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f02f8036000 tx=0x7f02f803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:43.955+0000 7f02f7fff640 1 -- 172.21.15.1:0/1507679494 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02f0057f50 con 0x7f03001565a0 2024-03-06T07:26:44.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.173+0000 7f03058a4640 1 -- 172.21.15.1:0/1507679494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f030006f280 con 0x7f03001565a0 2024-03-06T07:26:44.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.174+0000 7f02f7fff640 1 -- 172.21.15.1:0/1507679494 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6272 (secure 0 0 0) 0x7f02f005bff0 con 0x7f03001565a0 2024-03-06T07:26:44.176 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:44.176 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":145,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9162,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/4198928111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4198928111},{"type":"v1","addr":"172.21.15.1:6833","nonce":4198928111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143},{"gid":9168,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2664051788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2664051788},{"type":"v1","addr":"172.21.15.1:6835","nonce":2664051788}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:16.977128+0000","modified":"2024-03-06T07:26:39.702748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7774":{"gid":7774,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.98:6836/3360241462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3360241462},{"type":"v1","addr":"172.21.15.98:6836","nonce":3360241462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7774,"qdb_cluster":[7774]},"id":11},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:26.024256+0000","modified":"2024-03-06T07:26:39.702748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":155,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6989":{"gid":6989,"name":"b","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.98:6837/2360179300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2360179300},{"type":"v1","addr":"172.21.15.98:6837","nonce":2360179300}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6989,"qdb_cluster":[6989]},"id":12}]} 2024-03-06T07:26:44.176 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 145 2024-03-06T07:26:44.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.176+0000 7f03058a4640 1 -- 172.21.15.1:0/1507679494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02e006b500 msgr2=0x7f02e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.176+0000 7f03058a4640 1 --2- 172.21.15.1:0/1507679494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02e006b500 0x7f02e006d9c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f02f8036000 tx=0x7f02f803d040 comp rx=0 tx=0).stop 2024-03-06T07:26:44.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.176+0000 7f03058a4640 1 -- 172.21.15.1:0/1507679494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03001565a0 msgr2=0x7f030014f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.176+0000 7f03058a4640 1 --2- 172.21.15.1:0/1507679494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03001565a0 0x7f030014f8f0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f02f000c7a0 tx=0x7f02f000cc60 comp rx=0 tx=0).stop 2024-03-06T07:26:44.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.176+0000 7f03058a4640 1 -- 172.21.15.1:0/1507679494 shutdown_connections 2024-03-06T07:26:44.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.176+0000 7f03058a4640 1 --2- 172.21.15.1:0/1507679494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f02e006b500 0x7f02e006d9c0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.176+0000 7f03058a4640 1 --2- 172.21.15.1:0/1507679494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03001565a0 0x7f030014f8f0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.176+0000 7f03058a4640 1 --2- 172.21.15.1:0/1507679494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0300149a90 0x7f030014f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.176+0000 7f03058a4640 1 --2- 172.21.15.1:0/1507679494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03000c1260 0x7f03000bff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.176+0000 7f03058a4640 1 -- 172.21.15.1:0/1507679494 >> 172.21.15.1:0/1507679494 conn(0x7f03000c6860 msgr2=0x7f0300152850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:44.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.176+0000 7f03058a4640 1 -- 172.21.15.1:0/1507679494 shutdown_connections 2024-03-06T07:26:44.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.176+0000 7f03058a4640 1 -- 172.21.15.1:0/1507679494 wait complete. 2024-03-06T07:26:44.189 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:44.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.288+0000 7f61cbf0c640 1 Processor -- start 2024-03-06T07:26:44.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.288+0000 7f61cbf0c640 1 -- start start 2024-03-06T07:26:44.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.289+0000 7f61cbf0c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f61c40a7ec0 0x7f61c41535b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.289+0000 7f61cbf0c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c41566e0 con 0x7f61c40c4dd0 2024-03-06T07:26:44.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.289+0000 7f61cbf0c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c41569c0 con 0x7f61c40c79c0 2024-03-06T07:26:44.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.289+0000 7f61cbf0c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c4156490 con 0x7f61c40a7ec0 2024-03-06T07:26:44.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.289+0000 7f61c9ca8640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f61c40c4dd0 0x7f61c40c5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37382/0 (socket says 172.21.15.1:37382) 2024-03-06T07:26:44.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.289+0000 7f61c9ca8640 1 -- 172.21.15.1:0/1646972023 learned_addr learned my addr 172.21.15.1:0/1646972023 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:44.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.289+0000 7f61ca4a9640 1 --2- 172.21.15.1:0/1646972023 >> v2:172.21.15.98:3300/0 conn(0x7f61c40a7ec0 0x7f61c41535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.289+0000 7f61c8ca6640 1 -- 172.21.15.1:0/1646972023 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105819507 0 0) 0x7f61c41566e0 con 0x7f61c40c4dd0 2024-03-06T07:26:44.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.289+0000 7f61c8ca6640 1 -- 172.21.15.1:0/1646972023 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61ac003520 con 0x7f61c40c4dd0 2024-03-06T07:26:44.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.289+0000 7f61ca4a9640 1 -- 172.21.15.1:0/1646972023 >> v1:172.21.15.98:6789/0 conn(0x7f61c40c79c0 legacy=0x7f61c4152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.289+0000 7f61ca4a9640 1 -- 172.21.15.1:0/1646972023 >> v1:172.21.15.1:6789/0 conn(0x7f61c40c4dd0 legacy=0x7f61c40c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.289+0000 7f61ca4a9640 1 -- 172.21.15.1:0/1646972023 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c4156ca0 con 0x7f61c40a7ec0 2024-03-06T07:26:44.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.290+0000 7f61ca4a9640 1 --2- 172.21.15.1:0/1646972023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61c40a7ec0 0x7f61c41535b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f61b4009d90 tx=0x7f61b402be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd2167ab43b88fdc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.290+0000 7f61c8ca6640 1 -- 172.21.15.1:0/1646972023 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b4039070 con 0x7f61c40a7ec0 2024-03-06T07:26:44.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.290+0000 7f61c8ca6640 1 -- 172.21.15.1:0/1646972023 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b402cbd0 con 0x7f61c40a7ec0 2024-03-06T07:26:44.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.290+0000 7f61c8ca6640 1 -- 172.21.15.1:0/1646972023 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b4034440 con 0x7f61c40a7ec0 2024-03-06T07:26:44.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.291+0000 7f61cbf0c640 1 -- 172.21.15.1:0/1646972023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61c40a7ec0 msgr2=0x7f61c41535b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.291+0000 7f61cbf0c640 1 --2- 172.21.15.1:0/1646972023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61c40a7ec0 0x7f61c41535b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f61b4009d90 tx=0x7f61b402be20 comp rx=0 tx=0).stop 2024-03-06T07:26:44.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.291+0000 7f61cbf0c640 1 -- 172.21.15.1:0/1646972023 shutdown_connections 2024-03-06T07:26:44.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.291+0000 7f61cbf0c640 1 --2- 172.21.15.1:0/1646972023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61c40a7ec0 0x7f61c41535b0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.291+0000 7f61cbf0c640 1 -- 172.21.15.1:0/1646972023 >> 172.21.15.1:0/1646972023 conn(0x7f61c406b410 msgr2=0x7f61c406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:44.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.291+0000 7f61cbf0c640 1 -- 172.21.15.1:0/1646972023 shutdown_connections 2024-03-06T07:26:44.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.291+0000 7f61cbf0c640 1 -- 172.21.15.1:0/1646972023 wait complete. 2024-03-06T07:26:44.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.292+0000 7f61cbf0c640 1 Processor -- start 2024-03-06T07:26:44.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.292+0000 7f61cbf0c640 1 -- start start 2024-03-06T07:26:44.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61cbf0c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61c40a7ec0 0x7f61c4151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61cbf0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61c40c4dd0 0x7f61c41dc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61cbf0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61c414d0d0 0x7f61c41dca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61cbf0c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c4153af0 con 0x7f61c40a7ec0 2024-03-06T07:26:44.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61cbf0c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c4157010 con 0x7f61c40c4dd0 2024-03-06T07:26:44.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61cbf0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c40715f0 con 0x7f61c414d0d0 2024-03-06T07:26:44.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61c9ca8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61c40a7ec0 0x7f61c4151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61c9ca8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61c40a7ec0 0x7f61c4151fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38904/0 (socket says 172.21.15.1:38904) 2024-03-06T07:26:44.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61c9ca8640 1 -- 172.21.15.1:0/1184313068 learned_addr learned my addr 172.21.15.1:0/1184313068 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:44.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61ca4a9640 1 --2- 172.21.15.1:0/1184313068 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61c414d0d0 0x7f61c41dca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61c94a7640 1 --2- 172.21.15.1:0/1184313068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61c40c4dd0 0x7f61c41dc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61c9ca8640 1 -- 172.21.15.1:0/1184313068 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61c414d0d0 msgr2=0x7f61c41dca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61c9ca8640 1 --2- 172.21.15.1:0/1184313068 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61c414d0d0 0x7f61c41dca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61c9ca8640 1 -- 172.21.15.1:0/1184313068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61c40c4dd0 msgr2=0x7f61c41dc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61c9ca8640 1 --2- 172.21.15.1:0/1184313068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61c40c4dd0 0x7f61c41dc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61c9ca8640 1 -- 172.21.15.1:0/1184313068 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c40c3fb0 con 0x7f61c40a7ec0 2024-03-06T07:26:44.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61c9ca8640 1 --2- 172.21.15.1:0/1184313068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61c40a7ec0 0x7f61c4151fd0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f61bc00c500 tx=0x7f61bc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61c27fc640 1 -- 172.21.15.1:0/1184313068 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc017070 con 0x7f61c40a7ec0 2024-03-06T07:26:44.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61c27fc640 1 -- 172.21.15.1:0/1184313068 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61bc005ce0 con 0x7f61c40a7ec0 2024-03-06T07:26:44.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61cbf0c640 1 -- 172.21.15.1:0/1184313068 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c414cd20 con 0x7f61c40a7ec0 2024-03-06T07:26:44.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.293+0000 7f61c27fc640 1 -- 172.21.15.1:0/1184313068 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc007330 con 0x7f61c40a7ec0 2024-03-06T07:26:44.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.294+0000 7f61cbf0c640 1 -- 172.21.15.1:0/1184313068 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c41ec1c0 con 0x7f61c40a7ec0 2024-03-06T07:26:44.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.294+0000 7f61cbf0c640 1 -- 172.21.15.1:0/1184313068 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c41ebcd0 con 0x7f61c40a7ec0 2024-03-06T07:26:44.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.295+0000 7f61c27fc640 1 -- 172.21.15.1:0/1184313068 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61bc01a070 con 0x7f61c40a7ec0 2024-03-06T07:26:44.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.295+0000 7f61c27fc640 1 --2- 172.21.15.1:0/1184313068 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f61b006b4d0 0x7f61b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.295+0000 7f61c27fc640 1 -- 172.21.15.1:0/1184313068 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f61bc089ba0 con 0x7f61c40a7ec0 2024-03-06T07:26:44.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.296+0000 7f61c94a7640 1 --2- 172.21.15.1:0/1184313068 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f61b006b4d0 0x7f61b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.296+0000 7f61c94a7640 1 --2- 172.21.15.1:0/1184313068 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f61b006b4d0 0x7f61b006d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f61c4153f80 tx=0x7f61b8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.300+0000 7f61c27fc640 1 -- 172.21.15.1:0/1184313068 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61bc057f10 con 0x7f61c40a7ec0 2024-03-06T07:26:44.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.485+0000 7f61cbf0c640 1 -- 172.21.15.1:0/1184313068 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61c4067fd0 con 0x7f61c40a7ec0 2024-03-06T07:26:44.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.487+0000 7f61c27fc640 1 -- 172.21.15.1:0/1184313068 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+20733 (secure 0 0 0) 0x7f61bc05bfb0 con 0x7f61c40a7ec0 2024-03-06T07:26:44.489 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:44.489 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":155,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:26:36.687343+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:15.245426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:15.942343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:26:24.304587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:26:25.085741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/1089548586":"2024-03-07T07:26:36.687217+0000","172.21.15.1:6832/1089548586":"2024-03-07T07:26:36.687217+0000","172.21.15.1:6835/67091521":"2024-03-07T07:26:36.682683+0000","172.21.15.1:6834/67091521":"2024-03-07T07:26:36.682683+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.488+0000 7f61cbf0c640 1 -- 172.21.15.1:0/1184313068 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f61b006b4d0 msgr2=0x7f61b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.488+0000 7f61cbf0c640 1 --2- 172.21.15.1:0/1184313068 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f61b006b4d0 0x7f61b006d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f61c4153f80 tx=0x7f61b8009040 comp rx=0 tx=0).stop 2024-03-06T07:26:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.488+0000 7f61cbf0c640 1 -- 172.21.15.1:0/1184313068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61c40a7ec0 msgr2=0x7f61c4151fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.489+0000 7f61cbf0c640 1 --2- 172.21.15.1:0/1184313068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61c40a7ec0 0x7f61c4151fd0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f61bc00c500 tx=0x7f61bc00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.489+0000 7f61cbf0c640 1 -- 172.21.15.1:0/1184313068 shutdown_connections 2024-03-06T07:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.489+0000 7f61cbf0c640 1 --2- 172.21.15.1:0/1184313068 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f61b006b4d0 0x7f61b006d990 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.489+0000 7f61cbf0c640 1 --2- 172.21.15.1:0/1184313068 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61c414d0d0 0x7f61c41dca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.489+0000 7f61cbf0c640 1 --2- 172.21.15.1:0/1184313068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61c40c4dd0 0x7f61c41dc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.489+0000 7f61cbf0c640 1 --2- 172.21.15.1:0/1184313068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61c40a7ec0 0x7f61c4151fd0 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.489+0000 7f61cbf0c640 1 -- 172.21.15.1:0/1184313068 >> 172.21.15.1:0/1184313068 conn(0x7f61c406b410 msgr2=0x7f61c41532f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.489+0000 7f61cbf0c640 1 -- 172.21.15.1:0/1184313068 shutdown_connections 2024-03-06T07:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.489+0000 7f61cbf0c640 1 -- 172.21.15.1:0/1184313068 wait complete. 2024-03-06T07:26:44.502 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:26:44.502 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.600+0000 7f6ee2956640 1 Processor -- start 2024-03-06T07:26:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.600+0000 7f6ee2956640 1 -- start start 2024-03-06T07:26:44.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.601+0000 7f6ee2956640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6edc0afc40 0x7f6edc160b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.601+0000 7f6ee2956640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6edc0c7180 con 0x7f6edc152820 2024-03-06T07:26:44.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.601+0000 7f6ee2956640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6edc155460 con 0x7f6edc0af6f0 2024-03-06T07:26:44.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.601+0000 7f6ee2956640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6edc0c6bc0 con 0x7f6edc0afc40 2024-03-06T07:26:44.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.601+0000 7f6edbfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6edc152820 0x7f6edc0c0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33556/0 (socket says 172.21.15.1:33556) 2024-03-06T07:26:44.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.601+0000 7f6edbfff640 1 -- 172.21.15.1:0/3413890772 learned_addr learned my addr 172.21.15.1:0/3413890772 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:44.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.601+0000 7f6ee0ef3640 1 --2- 172.21.15.1:0/3413890772 >> v2:172.21.15.98:3300/0 conn(0x7f6edc0afc40 0x7f6edc160b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.602+0000 7f6edaffd640 1 -- 172.21.15.1:0/3413890772 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106306522 0 0) 0x7f6edc0c7180 con 0x7f6edc152820 2024-03-06T07:26:44.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.602+0000 7f6edaffd640 1 -- 172.21.15.1:0/3413890772 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ec4003520 con 0x7f6edc152820 2024-03-06T07:26:44.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.602+0000 7f6ee0ef3640 1 -- 172.21.15.1:0/3413890772 >> v1:172.21.15.98:6789/0 conn(0x7f6edc0af6f0 legacy=0x7f6edc0c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.602+0000 7f6ee0ef3640 1 -- 172.21.15.1:0/3413890772 >> v1:172.21.15.1:6789/0 conn(0x7f6edc152820 legacy=0x7f6edc0c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.602+0000 7f6edaffd640 1 -- 172.21.15.1:0/3413890772 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747239487 0 0) 0x7f6edc155460 con 0x7f6edc0af6f0 2024-03-06T07:26:44.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.602+0000 7f6ee0ef3640 1 -- 172.21.15.1:0/3413890772 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc0c6ea0 con 0x7f6edc0afc40 2024-03-06T07:26:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.602+0000 7f6ee0ef3640 1 --2- 172.21.15.1:0/3413890772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6edc0afc40 0x7f6edc160b70 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4009d90 tx=0x7f6ed402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4033f2f72459e8da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.603+0000 7f6edaffd640 1 -- 172.21.15.1:0/3413890772 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4039070 con 0x7f6edc0afc40 2024-03-06T07:26:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.603+0000 7f6edaffd640 1 -- 172.21.15.1:0/3413890772 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed402cd00 con 0x7f6edc0afc40 2024-03-06T07:26:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.603+0000 7f6edaffd640 1 -- 172.21.15.1:0/3413890772 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4034640 con 0x7f6edc0afc40 2024-03-06T07:26:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.603+0000 7f6ee2956640 1 -- 172.21.15.1:0/3413890772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6edc0afc40 msgr2=0x7f6edc160b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.603+0000 7f6ee2956640 1 --2- 172.21.15.1:0/3413890772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6edc0afc40 0x7f6edc160b70 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4009d90 tx=0x7f6ed402bf50 comp rx=0 tx=0).stop 2024-03-06T07:26:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.603+0000 7f6ee2956640 1 -- 172.21.15.1:0/3413890772 shutdown_connections 2024-03-06T07:26:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.603+0000 7f6ee2956640 1 --2- 172.21.15.1:0/3413890772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6edc0afc40 0x7f6edc160b70 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.603+0000 7f6ee2956640 1 -- 172.21.15.1:0/3413890772 >> 172.21.15.1:0/3413890772 conn(0x7f6edc072720 msgr2=0x7f6edc072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.603+0000 7f6ee2956640 1 -- 172.21.15.1:0/3413890772 shutdown_connections 2024-03-06T07:26:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.603+0000 7f6ee2956640 1 -- 172.21.15.1:0/3413890772 wait complete. 2024-03-06T07:26:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.604+0000 7f6ee2956640 1 Processor -- start 2024-03-06T07:26:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.604+0000 7f6ee2956640 1 -- start start 2024-03-06T07:26:44.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.605+0000 7f6ee2956640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6edc0af6f0 0x7f6edc0c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.605+0000 7f6ee2956640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6edc0afc40 0x7f6edc1ed1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.605+0000 7f6ee2956640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6edc152820 0x7f6edc1ed6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.605+0000 7f6ee2956640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc0c7b00 con 0x7f6edc0afc40 2024-03-06T07:26:44.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.605+0000 7f6ee2956640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc0c0840 con 0x7f6edc152820 2024-03-06T07:26:44.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.605+0000 7f6ee2956640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc0c7c80 con 0x7f6edc0af6f0 2024-03-06T07:26:44.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.605+0000 7f6edb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6edc0afc40 0x7f6edc1ed1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.605+0000 7f6edb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6edc0afc40 0x7f6edc1ed1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39416/0 (socket says 172.21.15.1:39416) 2024-03-06T07:26:44.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.605+0000 7f6edb7fe640 1 -- 172.21.15.1:0/963994152 learned_addr learned my addr 172.21.15.1:0/963994152 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:44.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.605+0000 7f6edbfff640 1 --2- 172.21.15.1:0/963994152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6edc0af6f0 0x7f6edc0c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.605+0000 7f6ee0ef3640 1 --2- 172.21.15.1:0/963994152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6edc152820 0x7f6edc1ed6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.606+0000 7f6edb7fe640 1 -- 172.21.15.1:0/963994152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6edc0af6f0 msgr2=0x7f6edc0c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.606+0000 7f6edb7fe640 1 --2- 172.21.15.1:0/963994152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6edc0af6f0 0x7f6edc0c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.606+0000 7f6edb7fe640 1 -- 172.21.15.1:0/963994152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6edc152820 msgr2=0x7f6edc1ed6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.606+0000 7f6edb7fe640 1 --2- 172.21.15.1:0/963994152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6edc152820 0x7f6edc1ed6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.606+0000 7f6edb7fe640 1 -- 172.21.15.1:0/963994152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc0c4420 con 0x7f6edc0afc40 2024-03-06T07:26:44.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.606+0000 7f6edb7fe640 1 --2- 172.21.15.1:0/963994152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6edc0afc40 0x7f6edc1ed1a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000c4b0 tx=0x7f6ed000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.606+0000 7f6ed8ff9640 1 -- 172.21.15.1:0/963994152 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0017070 con 0x7f6edc0afc40 2024-03-06T07:26:44.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.606+0000 7f6ed8ff9640 1 -- 172.21.15.1:0/963994152 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed0005480 con 0x7f6edc0afc40 2024-03-06T07:26:44.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.606+0000 7f6ee2956640 1 -- 172.21.15.1:0/963994152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6edc0c3210 con 0x7f6edc0afc40 2024-03-06T07:26:44.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.606+0000 7f6ed8ff9640 1 -- 172.21.15.1:0/963994152 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0003830 con 0x7f6edc0afc40 2024-03-06T07:26:44.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.606+0000 7f6ee2956640 1 -- 172.21.15.1:0/963994152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6edc1ebba0 con 0x7f6edc0afc40 2024-03-06T07:26:44.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.607+0000 7f6ee2956640 1 -- 172.21.15.1:0/963994152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6edc0686e0 con 0x7f6edc0afc40 2024-03-06T07:26:44.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.607+0000 7f6ed8ff9640 1 -- 172.21.15.1:0/963994152 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ed001a070 con 0x7f6edc0afc40 2024-03-06T07:26:44.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.608+0000 7f6ed8ff9640 1 --2- 172.21.15.1:0/963994152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ec806b4d0 0x7f6ec806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.608+0000 7f6ed8ff9640 1 -- 172.21.15.1:0/963994152 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f6ed0089a90 con 0x7f6edc0afc40 2024-03-06T07:26:44.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.608+0000 7f6edbfff640 1 --2- 172.21.15.1:0/963994152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ec806b4d0 0x7f6ec806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.608+0000 7f6edbfff640 1 --2- 172.21.15.1:0/963994152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ec806b4d0 0x7f6ec806d990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc005880 tx=0x7f6ecc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.612+0000 7f6ed8ff9640 1 -- 172.21.15.1:0/963994152 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ed0057d30 con 0x7f6edc0afc40 2024-03-06T07:26:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.832+0000 7f6ee2956640 1 -- 172.21.15.1:0/963994152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6edc0745d0 con 0x7f6edc0afc40 2024-03-06T07:26:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.832+0000 7f6ed8ff9640 1 -- 172.21.15.1:0/963994152 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6272 (secure 0 0 0) 0x7f6ed005e080 con 0x7f6edc0afc40 2024-03-06T07:26:44.835 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:44.835 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":145,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9162,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/4198928111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4198928111},{"type":"v1","addr":"172.21.15.1:6833","nonce":4198928111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143},{"gid":9168,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2664051788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2664051788},{"type":"v1","addr":"172.21.15.1:6835","nonce":2664051788}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:16.977128+0000","modified":"2024-03-06T07:26:39.702748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7774":{"gid":7774,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.98:6836/3360241462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3360241462},{"type":"v1","addr":"172.21.15.98:6836","nonce":3360241462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7774,"qdb_cluster":[7774]},"id":11},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:26.024256+0000","modified":"2024-03-06T07:26:39.702748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":155,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6989":{"gid":6989,"name":"b","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.98:6837/2360179300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2360179300},{"type":"v1","addr":"172.21.15.98:6837","nonce":2360179300}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6989,"qdb_cluster":[6989]},"id":12}]} 2024-03-06T07:26:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 145 2024-03-06T07:26:44.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.835+0000 7f6ee2956640 1 -- 172.21.15.1:0/963994152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ec806b4d0 msgr2=0x7f6ec806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.835+0000 7f6ee2956640 1 --2- 172.21.15.1:0/963994152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ec806b4d0 0x7f6ec806d990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc005880 tx=0x7f6ecc009040 comp rx=0 tx=0).stop 2024-03-06T07:26:44.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.835+0000 7f6ee2956640 1 -- 172.21.15.1:0/963994152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6edc0afc40 msgr2=0x7f6edc1ed1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.835+0000 7f6ee2956640 1 --2- 172.21.15.1:0/963994152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6edc0afc40 0x7f6edc1ed1a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000c4b0 tx=0x7f6ed000c970 comp rx=0 tx=0).stop 2024-03-06T07:26:44.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.835+0000 7f6ee2956640 1 -- 172.21.15.1:0/963994152 shutdown_connections 2024-03-06T07:26:44.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.835+0000 7f6ee2956640 1 --2- 172.21.15.1:0/963994152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ec806b4d0 0x7f6ec806d990 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.835+0000 7f6ee2956640 1 --2- 172.21.15.1:0/963994152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6edc152820 0x7f6edc1ed6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.835+0000 7f6ee2956640 1 --2- 172.21.15.1:0/963994152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6edc0afc40 0x7f6edc1ed1a0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.835+0000 7f6ee2956640 1 --2- 172.21.15.1:0/963994152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6edc0af6f0 0x7f6edc0c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.835+0000 7f6ee2956640 1 -- 172.21.15.1:0/963994152 >> 172.21.15.1:0/963994152 conn(0x7f6edc072720 msgr2=0x7f6edc0c73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.835+0000 7f6ee2956640 1 -- 172.21.15.1:0/963994152 shutdown_connections 2024-03-06T07:26:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.835+0000 7f6ee2956640 1 -- 172.21.15.1:0/963994152 wait complete. 2024-03-06T07:26:44.847 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:44.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.946+0000 7f25c2f3a640 1 Processor -- start 2024-03-06T07:26:44.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.946+0000 7f25c2f3a640 1 -- start start 2024-03-06T07:26:44.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.947+0000 7f25c2f3a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f25bc152650 0x7f25bc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.947+0000 7f25c2f3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25bc0c6b10 0x7f25bc158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.947+0000 7f25c2f3a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f25bc0bf9d0 0x7f25bc0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.947+0000 7f25c2f3a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25bc155800 con 0x7f25bc152650 2024-03-06T07:26:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.947+0000 7f25c2f3a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25bc0c2fc0 con 0x7f25bc0bf9d0 2024-03-06T07:26:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.947+0000 7f25c2f3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc0c6690 con 0x7f25bc0c6b10 2024-03-06T07:26:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.947+0000 7f25c0cd6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f25bc152650 0x7f25bc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.947+0000 7f25c0cd6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f25bc152650 0x7f25bc155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39430/0 (socket says 172.21.15.1:39430) 2024-03-06T07:26:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.947+0000 7f25c0cd6640 1 -- 172.21.15.1:0/692324080 learned_addr learned my addr 172.21.15.1:0/692324080 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.947+0000 7f25bbfff640 1 --2- 172.21.15.1:0/692324080 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25bc0c6b10 0x7f25bc158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.947+0000 7f25c14d7640 1 --2- 172.21.15.1:0/692324080 >> v2:172.21.15.98:3300/0 conn(0x7f25bc0bf9d0 0x7f25bc0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.948+0000 7f25c0cd6640 1 -- 172.21.15.1:0/692324080 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25bc0c6b10 msgr2=0x7f25bc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.948+0000 7f25c0cd6640 1 --2- 172.21.15.1:0/692324080 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25bc0c6b10 0x7f25bc158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.948+0000 7f25c0cd6640 1 -- 172.21.15.1:0/692324080 >> v2:172.21.15.98:3300/0 conn(0x7f25bc0bf9d0 msgr2=0x7f25bc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.948+0000 7f25c0cd6640 1 --2- 172.21.15.1:0/692324080 >> v2:172.21.15.98:3300/0 conn(0x7f25bc0bf9d0 0x7f25bc0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.948+0000 7f25c0cd6640 1 -- 172.21.15.1:0/692324080 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25bc0c7280 con 0x7f25bc152650 2024-03-06T07:26:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.948+0000 7f25c0cd6640 1 --2- 172.21.15.1:0/692324080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25bc152650 0x7f25bc155c90 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f25bc1597b0 tx=0x7f25b402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6055c07bba82ebf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.949+0000 7f25bb7fe640 1 -- 172.21.15.1:0/692324080 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b402c9e0 con 0x7f25bc152650 2024-03-06T07:26:44.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.949+0000 7f25bb7fe640 1 -- 172.21.15.1:0/692324080 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b402cb80 con 0x7f25bc152650 2024-03-06T07:26:44.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.949+0000 7f25bb7fe640 1 -- 172.21.15.1:0/692324080 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b40343f0 con 0x7f25bc152650 2024-03-06T07:26:44.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.949+0000 7f25c2f3a640 1 -- 172.21.15.1:0/692324080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25bc152650 msgr2=0x7f25bc155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.949+0000 7f25c2f3a640 1 --2- 172.21.15.1:0/692324080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25bc152650 0x7f25bc155c90 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f25bc1597b0 tx=0x7f25b402be40 comp rx=0 tx=0).stop 2024-03-06T07:26:44.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.949+0000 7f25c2f3a640 1 -- 172.21.15.1:0/692324080 shutdown_connections 2024-03-06T07:26:44.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.949+0000 7f25c2f3a640 1 --2- 172.21.15.1:0/692324080 >> v2:172.21.15.98:3300/0 conn(0x7f25bc0bf9d0 0x7f25bc0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.949+0000 7f25c2f3a640 1 --2- 172.21.15.1:0/692324080 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25bc0c6b10 0x7f25bc158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.949+0000 7f25c2f3a640 1 --2- 172.21.15.1:0/692324080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25bc152650 0x7f25bc155c90 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.949+0000 7f25c2f3a640 1 -- 172.21.15.1:0/692324080 >> 172.21.15.1:0/692324080 conn(0x7f25bc066ee0 msgr2=0x7f25bc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:44.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.949+0000 7f25c2f3a640 1 -- 172.21.15.1:0/692324080 shutdown_connections 2024-03-06T07:26:44.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.949+0000 7f25c2f3a640 1 -- 172.21.15.1:0/692324080 wait complete. 2024-03-06T07:26:44.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.950+0000 7f25c2f3a640 1 Processor -- start 2024-03-06T07:26:44.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.950+0000 7f25c2f3a640 1 -- start start 2024-03-06T07:26:44.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.951+0000 7f25c2f3a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25bc0bf9d0 0x7f25bc1eb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.951+0000 7f25c2f3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25bc152650 0x7f25bc0c5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.951+0000 7f25c2f3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25bc14b950 0x7f25bc14ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.951+0000 7f25c2f3a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc155490 con 0x7f25bc0bf9d0 2024-03-06T07:26:44.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.951+0000 7f25c2f3a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc1551b0 con 0x7f25bc152650 2024-03-06T07:26:44.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.951+0000 7f25c2f3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc152a30 con 0x7f25bc14b950 2024-03-06T07:26:44.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.951+0000 7f25c0cd6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25bc0bf9d0 0x7f25bc1eb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.951+0000 7f25c0cd6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25bc0bf9d0 0x7f25bc1eb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39442/0 (socket says 172.21.15.1:39442) 2024-03-06T07:26:44.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.951+0000 7f25c0cd6640 1 -- 172.21.15.1:0/4242840863 learned_addr learned my addr 172.21.15.1:0/4242840863 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:44.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.951+0000 7f25bbfff640 1 --2- 172.21.15.1:0/4242840863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25bc152650 0x7f25bc0c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.951+0000 7f25c14d7640 1 --2- 172.21.15.1:0/4242840863 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25bc14b950 0x7f25bc14ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.951+0000 7f25c0cd6640 1 -- 172.21.15.1:0/4242840863 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25bc14b950 msgr2=0x7f25bc14ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.951+0000 7f25c0cd6640 1 --2- 172.21.15.1:0/4242840863 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25bc14b950 0x7f25bc14ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.951+0000 7f25c0cd6640 1 -- 172.21.15.1:0/4242840863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25bc152650 msgr2=0x7f25bc0c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.951+0000 7f25c0cd6640 1 --2- 172.21.15.1:0/4242840863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25bc152650 0x7f25bc0c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.951+0000 7f25c0cd6640 1 -- 172.21.15.1:0/4242840863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25bc1dede0 con 0x7f25bc0bf9d0 2024-03-06T07:26:44.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.952+0000 7f25c0cd6640 1 --2- 172.21.15.1:0/4242840863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25bc0bf9d0 0x7f25bc1eb300 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f25b4000f80 tx=0x7f25b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.952+0000 7f25b97fa640 1 -- 172.21.15.1:0/4242840863 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b4043020 con 0x7f25bc0bf9d0 2024-03-06T07:26:44.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.952+0000 7f25b97fa640 1 -- 172.21.15.1:0/4242840863 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b403e450 con 0x7f25bc0bf9d0 2024-03-06T07:26:44.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.952+0000 7f25c2f3a640 1 -- 172.21.15.1:0/4242840863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25bc1eba70 con 0x7f25bc0bf9d0 2024-03-06T07:26:44.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.952+0000 7f25b97fa640 1 -- 172.21.15.1:0/4242840863 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b4009510 con 0x7f25bc0bf9d0 2024-03-06T07:26:44.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.952+0000 7f25c2f3a640 1 -- 172.21.15.1:0/4242840863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25bc1ec030 con 0x7f25bc0bf9d0 2024-03-06T07:26:44.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.953+0000 7f25c2f3a640 1 -- 172.21.15.1:0/4242840863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25bc1ec270 con 0x7f25bc0bf9d0 2024-03-06T07:26:44.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.953+0000 7f25b97fa640 1 -- 172.21.15.1:0/4242840863 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25b4046020 con 0x7f25bc0bf9d0 2024-03-06T07:26:44.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.954+0000 7f25b97fa640 1 --2- 172.21.15.1:0/4242840863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f25a806b4d0 0x7f25a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.954+0000 7f25b97fa640 1 -- 172.21.15.1:0/4242840863 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f25b40accf0 con 0x7f25bc0bf9d0 2024-03-06T07:26:44.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.954+0000 7f25bbfff640 1 --2- 172.21.15.1:0/4242840863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f25a806b4d0 0x7f25a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.954+0000 7f25bbfff640 1 --2- 172.21.15.1:0/4242840863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f25a806b4d0 0x7f25a806d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f25bc151300 tx=0x7f25b0009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:44.958+0000 7f25b97fa640 1 -- 172.21.15.1:0/4242840863 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25b40b1070 con 0x7f25bc0bf9d0 2024-03-06T07:26:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.176+0000 7f25c2f3a640 1 -- 172.21.15.1:0/4242840863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25bc0b06f0 con 0x7f25bc0bf9d0 2024-03-06T07:26:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.177+0000 7f25b97fa640 1 -- 172.21.15.1:0/4242840863 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6272 (secure 0 0 0) 0x7f25b404d030 con 0x7f25bc0bf9d0 2024-03-06T07:26:45.179 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:45.179 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":145,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9162,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/4198928111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4198928111},{"type":"v1","addr":"172.21.15.1:6833","nonce":4198928111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143},{"gid":9168,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2664051788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2664051788},{"type":"v1","addr":"172.21.15.1:6835","nonce":2664051788}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:16.977128+0000","modified":"2024-03-06T07:26:39.702748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7774":{"gid":7774,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.98:6836/3360241462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3360241462},{"type":"v1","addr":"172.21.15.98:6836","nonce":3360241462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7774,"qdb_cluster":[7774]},"id":11},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:26.024256+0000","modified":"2024-03-06T07:26:39.702748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":155,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6989":{"gid":6989,"name":"b","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.98:6837/2360179300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2360179300},{"type":"v1","addr":"172.21.15.98:6837","nonce":2360179300}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6989,"qdb_cluster":[6989]},"id":12}]} 2024-03-06T07:26:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 145 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.179+0000 7f25c2f3a640 1 -- 172.21.15.1:0/4242840863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f25a806b4d0 msgr2=0x7f25a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.179+0000 7f25c2f3a640 1 --2- 172.21.15.1:0/4242840863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f25a806b4d0 0x7f25a806d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f25bc151300 tx=0x7f25b0009190 comp rx=0 tx=0).stop 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.179+0000 7f25c2f3a640 1 -- 172.21.15.1:0/4242840863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25bc0bf9d0 msgr2=0x7f25bc1eb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.179+0000 7f25c2f3a640 1 --2- 172.21.15.1:0/4242840863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25bc0bf9d0 0x7f25bc1eb300 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f25b4000f80 tx=0x7f25b4004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.179+0000 7f25c2f3a640 1 -- 172.21.15.1:0/4242840863 shutdown_connections 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.179+0000 7f25c2f3a640 1 --2- 172.21.15.1:0/4242840863 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f25a806b4d0 0x7f25a806d990 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.179+0000 7f25c2f3a640 1 --2- 172.21.15.1:0/4242840863 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25bc14b950 0x7f25bc14ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.179+0000 7f25c2f3a640 1 --2- 172.21.15.1:0/4242840863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25bc152650 0x7f25bc0c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.179+0000 7f25c2f3a640 1 --2- 172.21.15.1:0/4242840863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25bc0bf9d0 0x7f25bc1eb300 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.179+0000 7f25c2f3a640 1 -- 172.21.15.1:0/4242840863 >> 172.21.15.1:0/4242840863 conn(0x7f25bc066ee0 msgr2=0x7f25bc0c0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.179+0000 7f25c2f3a640 1 -- 172.21.15.1:0/4242840863 shutdown_connections 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.179+0000 7f25c2f3a640 1 -- 172.21.15.1:0/4242840863 wait complete. 2024-03-06T07:26:45.192 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:45.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.290+0000 7f0542076640 1 Processor -- start 2024-03-06T07:26:45.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.291+0000 7f0542076640 1 -- start start 2024-03-06T07:26:45.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.291+0000 7f0542076640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f053c152650 0x7f053c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.291+0000 7f0542076640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f053c0c6b10 0x7f053c158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.291+0000 7f0542076640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c0c2fc0 con 0x7f053c0bf9d0 2024-03-06T07:26:45.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.291+0000 7f0542076640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f053c155800 con 0x7f053c0c6b10 2024-03-06T07:26:45.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.291+0000 7f0542076640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f053c0c6690 con 0x7f053c152650 2024-03-06T07:26:45.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.291+0000 7f053affd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f053c0c6b10 0x7f053c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.291+0000 7f053affd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f053c0c6b10 0x7f053c158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39448/0 (socket says 172.21.15.1:39448) 2024-03-06T07:26:45.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.291+0000 7f053affd640 1 -- 172.21.15.1:0/2690848224 learned_addr learned my addr 172.21.15.1:0/2690848224 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:45.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.292+0000 7f053b7fe640 1 --2- 172.21.15.1:0/2690848224 >> v2:172.21.15.98:3300/0 conn(0x7f053c152650 0x7f053c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.292+0000 7f053affd640 1 -- 172.21.15.1:0/2690848224 >> v2:172.21.15.98:3300/0 conn(0x7f053c152650 msgr2=0x7f053c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:45.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.292+0000 7f053affd640 1 --2- 172.21.15.1:0/2690848224 >> v2:172.21.15.98:3300/0 conn(0x7f053c152650 0x7f053c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.292+0000 7f053affd640 1 -- 172.21.15.1:0/2690848224 >> v1:172.21.15.98:6789/0 conn(0x7f053c0bf9d0 legacy=0x7f053c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.292+0000 7f053affd640 1 -- 172.21.15.1:0/2690848224 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c0c7280 con 0x7f053c0c6b10 2024-03-06T07:26:45.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.292+0000 7f053a7fc640 1 -- 172.21.15.1:0/2690848224 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667283175 0 0) 0x7f053c0c2fc0 con 0x7f053c0bf9d0 2024-03-06T07:26:45.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.292+0000 7f053affd640 1 --2- 172.21.15.1:0/2690848224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c0c6b10 0x7f053c158690 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f053c0cac90 tx=0x7f053002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=423f856ca7a4616a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.293+0000 7f053a7fc640 1 -- 172.21.15.1:0/2690848224 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0530035020 con 0x7f053c0c6b10 2024-03-06T07:26:45.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.293+0000 7f053a7fc640 1 -- 172.21.15.1:0/2690848224 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0530031870 con 0x7f053c0c6b10 2024-03-06T07:26:45.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.293+0000 7f053a7fc640 1 -- 172.21.15.1:0/2690848224 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0530031c40 con 0x7f053c0c6b10 2024-03-06T07:26:45.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.293+0000 7f0542076640 1 -- 172.21.15.1:0/2690848224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c0c6b10 msgr2=0x7f053c158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.293+0000 7f0542076640 1 --2- 172.21.15.1:0/2690848224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c0c6b10 0x7f053c158690 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f053c0cac90 tx=0x7f053002b910 comp rx=0 tx=0).stop 2024-03-06T07:26:45.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.293+0000 7f0542076640 1 -- 172.21.15.1:0/2690848224 shutdown_connections 2024-03-06T07:26:45.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.293+0000 7f0542076640 1 --2- 172.21.15.1:0/2690848224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c0c6b10 0x7f053c158690 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.293+0000 7f0542076640 1 --2- 172.21.15.1:0/2690848224 >> v2:172.21.15.98:3300/0 conn(0x7f053c152650 0x7f053c155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.293+0000 7f0542076640 1 -- 172.21.15.1:0/2690848224 >> 172.21.15.1:0/2690848224 conn(0x7f053c066ee0 msgr2=0x7f053c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.293+0000 7f0542076640 1 -- 172.21.15.1:0/2690848224 shutdown_connections 2024-03-06T07:26:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.293+0000 7f0542076640 1 -- 172.21.15.1:0/2690848224 wait complete. 2024-03-06T07:26:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.294+0000 7f0542076640 1 Processor -- start 2024-03-06T07:26:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.295+0000 7f0542076640 1 -- start start 2024-03-06T07:26:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.295+0000 7f0542076640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f053c0bf9d0 0x7f053c153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.295+0000 7f0542076640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f053c152650 0x7f053c1451f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.295+0000 7f0542076640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c1e0ed0 0x7f053c145730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.295+0000 7f0542076640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c155490 con 0x7f053c1e0ed0 2024-03-06T07:26:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.295+0000 7f0542076640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c1551b0 con 0x7f053c0bf9d0 2024-03-06T07:26:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.295+0000 7f0542076640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053c152a30 con 0x7f053c152650 2024-03-06T07:26:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.295+0000 7f053bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c1e0ed0 0x7f053c145730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.295+0000 7f053bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c1e0ed0 0x7f053c145730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39460/0 (socket says 172.21.15.1:39460) 2024-03-06T07:26:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.295+0000 7f053bfff640 1 -- 172.21.15.1:0/3688087820 learned_addr learned my addr 172.21.15.1:0/3688087820 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.295+0000 7f053b7fe640 1 --2- 172.21.15.1:0/3688087820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f053c0bf9d0 0x7f053c153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.295+0000 7f053affd640 1 --2- 172.21.15.1:0/3688087820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f053c152650 0x7f053c1451f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.296+0000 7f053bfff640 1 -- 172.21.15.1:0/3688087820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f053c152650 msgr2=0x7f053c1451f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.296+0000 7f053bfff640 1 --2- 172.21.15.1:0/3688087820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f053c152650 0x7f053c1451f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.296+0000 7f053bfff640 1 -- 172.21.15.1:0/3688087820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f053c0bf9d0 msgr2=0x7f053c153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.296+0000 7f053bfff640 1 --2- 172.21.15.1:0/3688087820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f053c0bf9d0 0x7f053c153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.296+0000 7f053bfff640 1 -- 172.21.15.1:0/3688087820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c145c70 con 0x7f053c1e0ed0 2024-03-06T07:26:45.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.296+0000 7f053bfff640 1 --2- 172.21.15.1:0/3688087820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c1e0ed0 0x7f053c145730 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f053400c7b0 tx=0x7f053400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.296+0000 7f0523fff640 1 -- 172.21.15.1:0/3688087820 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0534003ba0 con 0x7f053c1e0ed0 2024-03-06T07:26:45.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.296+0000 7f0523fff640 1 -- 172.21.15.1:0/3688087820 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0534003d40 con 0x7f053c1e0ed0 2024-03-06T07:26:45.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.296+0000 7f0542076640 1 -- 172.21.15.1:0/3688087820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053c14fc80 con 0x7f053c1e0ed0 2024-03-06T07:26:45.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.296+0000 7f0523fff640 1 -- 172.21.15.1:0/3688087820 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0534005270 con 0x7f053c1e0ed0 2024-03-06T07:26:45.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.296+0000 7f0542076640 1 -- 172.21.15.1:0/3688087820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053c1eb6b0 con 0x7f053c1e0ed0 2024-03-06T07:26:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.297+0000 7f0542076640 1 -- 172.21.15.1:0/3688087820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053c067fd0 con 0x7f053c1e0ed0 2024-03-06T07:26:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.298+0000 7f0523fff640 1 -- 172.21.15.1:0/3688087820 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0534005410 con 0x7f053c1e0ed0 2024-03-06T07:26:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.298+0000 7f0523fff640 1 --2- 172.21.15.1:0/3688087820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f052806b500 0x7f052806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.298+0000 7f0523fff640 1 -- 172.21.15.1:0/3688087820 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f053408a3e0 con 0x7f053c1e0ed0 2024-03-06T07:26:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.298+0000 7f053b7fe640 1 --2- 172.21.15.1:0/3688087820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f052806b500 0x7f052806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.298+0000 7f053b7fe640 1 --2- 172.21.15.1:0/3688087820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f052806b500 0x7f052806d9c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f053c1eb5e0 tx=0x7f052c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.302+0000 7f0523fff640 1 -- 172.21.15.1:0/3688087820 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0534058710 con 0x7f053c1e0ed0 2024-03-06T07:26:45.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.488+0000 7f0542076640 1 -- 172.21.15.1:0/3688087820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f053c06b7b0 con 0x7f053c1e0ed0 2024-03-06T07:26:45.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.490+0000 7f0523fff640 1 -- 172.21.15.1:0/3688087820 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+20733 (secure 0 0 0) 0x7f053405c7b0 con 0x7f053c1e0ed0 2024-03-06T07:26:45.492 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:45.492 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":155,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:26:36.687343+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:15.245426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:15.942343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:26:24.304587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:26:25.085741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/1089548586":"2024-03-07T07:26:36.687217+0000","172.21.15.1:6832/1089548586":"2024-03-07T07:26:36.687217+0000","172.21.15.1:6835/67091521":"2024-03-07T07:26:36.682683+0000","172.21.15.1:6834/67091521":"2024-03-07T07:26:36.682683+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.492+0000 7f0542076640 1 -- 172.21.15.1:0/3688087820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f052806b500 msgr2=0x7f052806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.492+0000 7f0542076640 1 --2- 172.21.15.1:0/3688087820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f052806b500 0x7f052806d9c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f053c1eb5e0 tx=0x7f052c009190 comp rx=0 tx=0).stop 2024-03-06T07:26:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.492+0000 7f0542076640 1 -- 172.21.15.1:0/3688087820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c1e0ed0 msgr2=0x7f053c145730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.492+0000 7f0542076640 1 --2- 172.21.15.1:0/3688087820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c1e0ed0 0x7f053c145730 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f053400c7b0 tx=0x7f053400cbb0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.492+0000 7f0542076640 1 -- 172.21.15.1:0/3688087820 shutdown_connections 2024-03-06T07:26:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.492+0000 7f0542076640 1 --2- 172.21.15.1:0/3688087820 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f052806b500 0x7f052806d9c0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.492+0000 7f0542076640 1 --2- 172.21.15.1:0/3688087820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c1e0ed0 0x7f053c145730 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.492+0000 7f0542076640 1 --2- 172.21.15.1:0/3688087820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f053c152650 0x7f053c1451f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.492+0000 7f0542076640 1 --2- 172.21.15.1:0/3688087820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f053c0bf9d0 0x7f053c153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.492+0000 7f0542076640 1 -- 172.21.15.1:0/3688087820 >> 172.21.15.1:0/3688087820 conn(0x7f053c066ee0 msgr2=0x7f053c0c52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:45.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.492+0000 7f0542076640 1 -- 172.21.15.1:0/3688087820 shutdown_connections 2024-03-06T07:26:45.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.492+0000 7f0542076640 1 -- 172.21.15.1:0/3688087820 wait complete. 2024-03-06T07:26:45.505 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:26:45.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.605+0000 7f947b0eb640 1 Processor -- start 2024-03-06T07:26:45.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.605+0000 7f947b0eb640 1 -- start start 2024-03-06T07:26:45.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.606+0000 7f947b0eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94740c3970 0x7f94740c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.606+0000 7f947b0eb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94740c60b0 0x7f9474152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.606+0000 7f947b0eb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9474152670 0x7f94741553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.606+0000 7f947b0eb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9474152ad0 con 0x7f94740c60b0 2024-03-06T07:26:45.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.606+0000 7f947b0eb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94740c67f0 con 0x7f9474152670 2024-03-06T07:26:45.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.606+0000 7f947b0eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94740c5e40 con 0x7f94740c3970 2024-03-06T07:26:45.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.606+0000 7f9473fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94740c60b0 0x7f9474152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.606+0000 7f9478e87640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94740c3970 0x7f94740c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.606+0000 7f9479688640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9474152670 0x7f94741553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.606+0000 7f9478e87640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94740c3970 0x7f94740c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:57532/0 (socket says 172.21.15.1:57532) 2024-03-06T07:26:45.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.606+0000 7f9478e87640 1 -- 172.21.15.1:0/198043626 learned_addr learned my addr 172.21.15.1:0/198043626 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:45.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.606+0000 7f9473fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94740c60b0 0x7f9474152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39470/0 (socket says 172.21.15.1:39470) 2024-03-06T07:26:45.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.607+0000 7f9478e87640 1 -- 172.21.15.1:0/198043626 >> v2:172.21.15.98:3300/0 conn(0x7f9474152670 msgr2=0x7f94741553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:45.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.607+0000 7f9478e87640 1 --2- 172.21.15.1:0/198043626 >> v2:172.21.15.98:3300/0 conn(0x7f9474152670 0x7f94741553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.607+0000 7f9478e87640 1 -- 172.21.15.1:0/198043626 >> v2:172.21.15.1:3300/0 conn(0x7f94740c60b0 msgr2=0x7f9474152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:45.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.607+0000 7f9478e87640 1 --2- 172.21.15.1:0/198043626 >> v2:172.21.15.1:3300/0 conn(0x7f94740c60b0 0x7f9474152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.607+0000 7f9478e87640 1 -- 172.21.15.1:0/198043626 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94740c6a40 con 0x7f94740c3970 2024-03-06T07:26:45.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.607+0000 7f9473fff640 1 --2- 172.21.15.1:0/198043626 >> v2:172.21.15.1:3300/0 conn(0x7f94740c60b0 0x7f9474152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:45.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.607+0000 7f9478e87640 1 --2- 172.21.15.1:0/198043626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94740c3970 0x7f94740c8170 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f94740c4f70 tx=0x7f946c02bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ec4e138042a1995 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.607+0000 7f94737fe640 1 -- 172.21.15.1:0/198043626 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c02ca80 con 0x7f94740c3970 2024-03-06T07:26:45.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.608+0000 7f94737fe640 1 -- 172.21.15.1:0/198043626 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946c02cc20 con 0x7f94740c3970 2024-03-06T07:26:45.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.608+0000 7f94737fe640 1 -- 172.21.15.1:0/198043626 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c0343f0 con 0x7f94740c3970 2024-03-06T07:26:45.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.608+0000 7f947b0eb640 1 -- 172.21.15.1:0/198043626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94740c3970 msgr2=0x7f94740c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.608+0000 7f947b0eb640 1 --2- 172.21.15.1:0/198043626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94740c3970 0x7f94740c8170 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f94740c4f70 tx=0x7f946c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.608+0000 7f947b0eb640 1 -- 172.21.15.1:0/198043626 shutdown_connections 2024-03-06T07:26:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.608+0000 7f947b0eb640 1 --2- 172.21.15.1:0/198043626 >> v2:172.21.15.98:3300/0 conn(0x7f9474152670 0x7f94741553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.608+0000 7f947b0eb640 1 --2- 172.21.15.1:0/198043626 >> v2:172.21.15.1:3300/0 conn(0x7f94740c60b0 0x7f9474152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.608+0000 7f947b0eb640 1 --2- 172.21.15.1:0/198043626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94740c3970 0x7f94740c8170 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.608+0000 7f947b0eb640 1 -- 172.21.15.1:0/198043626 >> 172.21.15.1:0/198043626 conn(0x7f94740c88f0 msgr2=0x7f94740c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.608+0000 7f947b0eb640 1 -- 172.21.15.1:0/198043626 shutdown_connections 2024-03-06T07:26:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.608+0000 7f947b0eb640 1 -- 172.21.15.1:0/198043626 wait complete. 2024-03-06T07:26:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.609+0000 7f947b0eb640 1 Processor -- start 2024-03-06T07:26:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.609+0000 7f947b0eb640 1 -- start start 2024-03-06T07:26:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.610+0000 7f947b0eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94740c3970 0x7f94740711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.610+0000 7f947b0eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94740c60b0 0x7f94740c2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.610+0000 7f947b0eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9474152670 0x7f94741ea3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.610+0000 7f947b0eb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94740c72e0 con 0x7f94740c60b0 2024-03-06T07:26:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.610+0000 7f947b0eb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94740c7000 con 0x7f9474152670 2024-03-06T07:26:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.610+0000 7f947b0eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94740c75c0 con 0x7f94740c3970 2024-03-06T07:26:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.610+0000 7f9473fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94740c60b0 0x7f94740c2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.610+0000 7f9473fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94740c60b0 0x7f94740c2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39472/0 (socket says 172.21.15.1:39472) 2024-03-06T07:26:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.610+0000 7f9473fff640 1 -- 172.21.15.1:0/2740939552 learned_addr learned my addr 172.21.15.1:0/2740939552 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.610+0000 7f9478e87640 1 --2- 172.21.15.1:0/2740939552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94740c3970 0x7f94740711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.610+0000 7f9479688640 1 --2- 172.21.15.1:0/2740939552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9474152670 0x7f94741ea3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.610+0000 7f9473fff640 1 -- 172.21.15.1:0/2740939552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94740c3970 msgr2=0x7f94740711e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.610+0000 7f9473fff640 1 --2- 172.21.15.1:0/2740939552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94740c3970 0x7f94740711e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.610+0000 7f9473fff640 1 -- 172.21.15.1:0/2740939552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9474152670 msgr2=0x7f94741ea3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.610+0000 7f9473fff640 1 --2- 172.21.15.1:0/2740939552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9474152670 0x7f94741ea3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.610+0000 7f9473fff640 1 -- 172.21.15.1:0/2740939552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94740ade50 con 0x7f94740c60b0 2024-03-06T07:26:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.611+0000 7f9473fff640 1 --2- 172.21.15.1:0/2740939552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94740c60b0 0x7f94740c2530 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f946400e9b0 tx=0x7f946400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.611+0000 7f94717fa640 1 -- 172.21.15.1:0/2740939552 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464017070 con 0x7f94740c60b0 2024-03-06T07:26:45.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.611+0000 7f94717fa640 1 -- 172.21.15.1:0/2740939552 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946400c9a0 con 0x7f94740c60b0 2024-03-06T07:26:45.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.611+0000 7f947b0eb640 1 -- 172.21.15.1:0/2740939552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9474156590 con 0x7f94740c60b0 2024-03-06T07:26:45.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.611+0000 7f94717fa640 1 -- 172.21.15.1:0/2740939552 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946400cd70 con 0x7f94740c60b0 2024-03-06T07:26:45.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.611+0000 7f947b0eb640 1 -- 172.21.15.1:0/2740939552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94740ad670 con 0x7f94740c60b0 2024-03-06T07:26:45.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.612+0000 7f947b0eb640 1 -- 172.21.15.1:0/2740939552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94740686e0 con 0x7f94740c60b0 2024-03-06T07:26:45.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.612+0000 7f94717fa640 1 -- 172.21.15.1:0/2740939552 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f946401a070 con 0x7f94740c60b0 2024-03-06T07:26:45.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.613+0000 7f94717fa640 1 --2- 172.21.15.1:0/2740939552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f945406b4d0 0x7f945406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.613+0000 7f94717fa640 1 -- 172.21.15.1:0/2740939552 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f946408a290 con 0x7f94740c60b0 2024-03-06T07:26:45.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.613+0000 7f9478e87640 1 --2- 172.21.15.1:0/2740939552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f945406b4d0 0x7f945406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.613+0000 7f9478e87640 1 --2- 172.21.15.1:0/2740939552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f945406b4d0 0x7f945406d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f946c035000 tx=0x7f946c030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.617+0000 7f94717fa640 1 -- 172.21.15.1:0/2740939552 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9464058490 con 0x7f94740c60b0 2024-03-06T07:26:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.836+0000 7f947b0eb640 1 -- 172.21.15.1:0/2740939552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f947406eb70 con 0x7f94740c60b0 2024-03-06T07:26:45.838 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:26:45.836+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:45.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.838+0000 7f94717fa640 1 -- 172.21.15.1:0/2740939552 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v146) v1 ==== 161+0+0 (secure 0 0 0) 0x7f946405c530 con 0x7f94740c60b0 2024-03-06T07:26:45.839 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:26:45.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.840+0000 7f947b0eb640 1 -- 172.21.15.1:0/2740939552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f945406b4d0 msgr2=0x7f945406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.840+0000 7f947b0eb640 1 --2- 172.21.15.1:0/2740939552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f945406b4d0 0x7f945406d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f946c035000 tx=0x7f946c030000 comp rx=0 tx=0).stop 2024-03-06T07:26:45.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.840+0000 7f947b0eb640 1 -- 172.21.15.1:0/2740939552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94740c60b0 msgr2=0x7f94740c2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.840+0000 7f947b0eb640 1 --2- 172.21.15.1:0/2740939552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94740c60b0 0x7f94740c2530 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f946400e9b0 tx=0x7f946400ee70 comp rx=0 tx=0).stop 2024-03-06T07:26:45.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.840+0000 7f947b0eb640 1 -- 172.21.15.1:0/2740939552 shutdown_connections 2024-03-06T07:26:45.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.840+0000 7f947b0eb640 1 --2- 172.21.15.1:0/2740939552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f945406b4d0 0x7f945406d990 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.840+0000 7f947b0eb640 1 --2- 172.21.15.1:0/2740939552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9474152670 0x7f94741ea3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.840+0000 7f947b0eb640 1 --2- 172.21.15.1:0/2740939552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94740c60b0 0x7f94740c2530 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.840+0000 7f947b0eb640 1 --2- 172.21.15.1:0/2740939552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94740c3970 0x7f94740711e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.840+0000 7f947b0eb640 1 -- 172.21.15.1:0/2740939552 >> 172.21.15.1:0/2740939552 conn(0x7f94740c88f0 msgr2=0x7f9474151680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:45.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.840+0000 7f947b0eb640 1 -- 172.21.15.1:0/2740939552 shutdown_connections 2024-03-06T07:26:45.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.840+0000 7f947b0eb640 1 -- 172.21.15.1:0/2740939552 wait complete. 2024-03-06T07:26:45.843 INFO:tasks.ceph.mds.d.smithi098.stderr: -94> 2024-03-06T07:26:42.903+0000 7f2313d48640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:45.856 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:26:45.867 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:26:45.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.958+0000 7fd05cdac640 1 Processor -- start 2024-03-06T07:26:45.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.958+0000 7fd05cdac640 1 -- start start 2024-03-06T07:26:45.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.958+0000 7fd05cdac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0580c05f0 0x7fd0580c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.958+0000 7fd05cdac640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd05806f640 0x7fd0581551b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.958+0000 7fd05cdac640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd0580c14c0 0x7fd0581556f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.958+0000 7fd05cdac640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd05806fe90 con 0x7fd0580c14c0 2024-03-06T07:26:45.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.959+0000 7fd05cdac640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0580c2fc0 con 0x7fd05806f640 2024-03-06T07:26:45.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.959+0000 7fd05cdac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0580c31f0 con 0x7fd0580c05f0 2024-03-06T07:26:45.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.959+0000 7fd056d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd0580c14c0 0x7fd0581556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.959+0000 7fd05659c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0580c05f0 0x7fd0580c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.959+0000 7fd056d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd0580c14c0 0x7fd0581556f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39488/0 (socket says 172.21.15.1:39488) 2024-03-06T07:26:45.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.959+0000 7fd055d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd05806f640 0x7fd0581551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.959+0000 7fd056d9d640 1 -- 172.21.15.1:0/2504978328 learned_addr learned my addr 172.21.15.1:0/2504978328 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:45.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.959+0000 7fd056d9d640 1 -- 172.21.15.1:0/2504978328 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0580c05f0 msgr2=0x7fd0580c09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:45.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.959+0000 7fd056d9d640 1 --2- 172.21.15.1:0/2504978328 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0580c05f0 0x7fd0580c09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.959+0000 7fd056d9d640 1 -- 172.21.15.1:0/2504978328 >> v2:172.21.15.98:3300/0 conn(0x7fd05806f640 msgr2=0x7fd0581551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:45.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.959+0000 7fd056d9d640 1 --2- 172.21.15.1:0/2504978328 >> v2:172.21.15.98:3300/0 conn(0x7fd05806f640 0x7fd0581551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.959+0000 7fd056d9d640 1 -- 172.21.15.1:0/2504978328 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05806fbb0 con 0x7fd0580c14c0 2024-03-06T07:26:45.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.959+0000 7fd05659c640 1 --2- 172.21.15.1:0/2504978328 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0580c05f0 0x7fd0580c09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:45.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.959+0000 7fd055d9b640 1 --2- 172.21.15.1:0/2504978328 >> v2:172.21.15.98:3300/0 conn(0x7fd05806f640 0x7fd0581551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:45.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.960+0000 7fd056d9d640 1 --2- 172.21.15.1:0/2504978328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0580c14c0 0x7fd0581556f0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fd0580c4c90 tx=0x7fd05002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826cb63154aa0e7d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.960+0000 7fd05559a640 1 -- 172.21.15.1:0/2504978328 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05002ca70 con 0x7fd0580c14c0 2024-03-06T07:26:45.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.960+0000 7fd05559a640 1 -- 172.21.15.1:0/2504978328 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05002cc10 con 0x7fd0580c14c0 2024-03-06T07:26:45.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.960+0000 7fd05559a640 1 -- 172.21.15.1:0/2504978328 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0500343f0 con 0x7fd0580c14c0 2024-03-06T07:26:45.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.960+0000 7fd05cdac640 1 -- 172.21.15.1:0/2504978328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0580c14c0 msgr2=0x7fd0581556f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.960+0000 7fd05cdac640 1 --2- 172.21.15.1:0/2504978328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0580c14c0 0x7fd0581556f0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fd0580c4c90 tx=0x7fd05002bed0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.961+0000 7fd05cdac640 1 -- 172.21.15.1:0/2504978328 shutdown_connections 2024-03-06T07:26:45.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.961+0000 7fd05cdac640 1 --2- 172.21.15.1:0/2504978328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0580c14c0 0x7fd0581556f0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.961+0000 7fd05cdac640 1 --2- 172.21.15.1:0/2504978328 >> v2:172.21.15.98:3300/0 conn(0x7fd05806f640 0x7fd0581551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.961+0000 7fd05cdac640 1 --2- 172.21.15.1:0/2504978328 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0580c05f0 0x7fd0580c09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.961+0000 7fd05cdac640 1 -- 172.21.15.1:0/2504978328 >> 172.21.15.1:0/2504978328 conn(0x7fd0580c3cc0 msgr2=0x7fd0580c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:45.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.961+0000 7fd05cdac640 1 -- 172.21.15.1:0/2504978328 shutdown_connections 2024-03-06T07:26:45.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.961+0000 7fd05cdac640 1 -- 172.21.15.1:0/2504978328 wait complete. 2024-03-06T07:26:45.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.962+0000 7fd05cdac640 1 Processor -- start 2024-03-06T07:26:45.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.962+0000 7fd05cdac640 1 -- start start 2024-03-06T07:26:45.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.962+0000 7fd05cdac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd05806f640 0x7fd058157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd05cdac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0580c05f0 0x7fd058157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd05cdac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0580c14c0 0x7fd0581467d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd05cdac640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0580c1040 con 0x7fd0580c05f0 2024-03-06T07:26:45.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd05cdac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0580c3770 con 0x7fd0580c14c0 2024-03-06T07:26:45.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd05cdac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05814f460 con 0x7fd05806f640 2024-03-06T07:26:45.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd055d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0580c05f0 0x7fd058157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd055d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0580c05f0 0x7fd058157cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39496/0 (socket says 172.21.15.1:39496) 2024-03-06T07:26:45.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd055d9b640 1 -- 172.21.15.1:0/2857824797 learned_addr learned my addr 172.21.15.1:0/2857824797 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:45.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd05659c640 1 --2- 172.21.15.1:0/2857824797 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd05806f640 0x7fd058157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd056d9d640 1 --2- 172.21.15.1:0/2857824797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0580c14c0 0x7fd0581467d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd055d9b640 1 -- 172.21.15.1:0/2857824797 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd05806f640 msgr2=0x7fd058157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd055d9b640 1 --2- 172.21.15.1:0/2857824797 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd05806f640 0x7fd058157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd055d9b640 1 -- 172.21.15.1:0/2857824797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0580c14c0 msgr2=0x7fd0581467d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd055d9b640 1 --2- 172.21.15.1:0/2857824797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0580c14c0 0x7fd0581467d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd055d9b640 1 -- 172.21.15.1:0/2857824797 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd058158210 con 0x7fd0580c05f0 2024-03-06T07:26:45.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd05659c640 1 --2- 172.21.15.1:0/2857824797 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd05806f640 0x7fd058157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:45.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd056d9d640 1 --2- 172.21.15.1:0/2857824797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0580c14c0 0x7fd0581467d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:45.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd055d9b640 1 --2- 172.21.15.1:0/2857824797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0580c05f0 0x7fd058157cd0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fd04c00e8e0 tx=0x7fd04c00eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd046ffd640 1 -- 172.21.15.1:0/2857824797 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04c017070 con 0x7fd0580c05f0 2024-03-06T07:26:45.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd046ffd640 1 -- 172.21.15.1:0/2857824797 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04c00c940 con 0x7fd0580c05f0 2024-03-06T07:26:45.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd05cdac640 1 -- 172.21.15.1:0/2857824797 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0581eb480 con 0x7fd0580c05f0 2024-03-06T07:26:45.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd046ffd640 1 -- 172.21.15.1:0/2857824797 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04c00cd10 con 0x7fd0580c05f0 2024-03-06T07:26:45.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.963+0000 7fd05cdac640 1 -- 172.21.15.1:0/2857824797 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05814dee0 con 0x7fd0580c05f0 2024-03-06T07:26:45.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.964+0000 7fd05cdac640 1 -- 172.21.15.1:0/2857824797 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05814e530 con 0x7fd0580c05f0 2024-03-06T07:26:45.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.965+0000 7fd046ffd640 1 -- 172.21.15.1:0/2857824797 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd04c01a070 con 0x7fd0580c05f0 2024-03-06T07:26:45.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.965+0000 7fd046ffd640 1 --2- 172.21.15.1:0/2857824797 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd03806b4d0 0x7fd03806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.965+0000 7fd046ffd640 1 -- 172.21.15.1:0/2857824797 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 7215+0+0 (secure 0 0 0) 0x7fd04c08a5a0 con 0x7fd0580c05f0 2024-03-06T07:26:45.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.966+0000 7fd05659c640 1 --2- 172.21.15.1:0/2857824797 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd03806b4d0 0x7fd03806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.966+0000 7fd05659c640 1 --2- 172.21.15.1:0/2857824797 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd03806b4d0 0x7fd03806d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fd048009560 tx=0x7fd048009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:45.970+0000 7fd046ffd640 1 -- 172.21.15.1:0/2857824797 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd04c058880 con 0x7fd0580c05f0 2024-03-06T07:26:46.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.189+0000 7fd05cdac640 1 -- 172.21.15.1:0/2857824797 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd058067fd0 con 0x7fd0580c05f0 2024-03-06T07:26:46.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.839+0000 7fd046ffd640 1 -- 172.21.15.1:0/2857824797 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v147) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd04c05c920 con 0x7fd0580c05f0 2024-03-06T07:26:46.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.841+0000 7fd05cdac640 1 -- 172.21.15.1:0/2857824797 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd03806b4d0 msgr2=0x7fd03806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.841+0000 7fd05cdac640 1 --2- 172.21.15.1:0/2857824797 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd03806b4d0 0x7fd03806d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fd048009560 tx=0x7fd048009190 comp rx=0 tx=0).stop 2024-03-06T07:26:46.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.841+0000 7fd05cdac640 1 -- 172.21.15.1:0/2857824797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0580c05f0 msgr2=0x7fd058157cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.841+0000 7fd05cdac640 1 --2- 172.21.15.1:0/2857824797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0580c05f0 0x7fd058157cd0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fd04c00e8e0 tx=0x7fd04c00eda0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.841+0000 7fd05cdac640 1 -- 172.21.15.1:0/2857824797 shutdown_connections 2024-03-06T07:26:46.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.841+0000 7fd05cdac640 1 --2- 172.21.15.1:0/2857824797 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd03806b4d0 0x7fd03806d990 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.841+0000 7fd05cdac640 1 --2- 172.21.15.1:0/2857824797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0580c14c0 0x7fd0581467d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.841+0000 7fd05cdac640 1 --2- 172.21.15.1:0/2857824797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0580c05f0 0x7fd058157cd0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.841+0000 7fd05cdac640 1 --2- 172.21.15.1:0/2857824797 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd05806f640 0x7fd058157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.841+0000 7fd05cdac640 1 -- 172.21.15.1:0/2857824797 >> 172.21.15.1:0/2857824797 conn(0x7fd0580c3cc0 msgr2=0x7fd05806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:46.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.842+0000 7fd05cdac640 1 -- 172.21.15.1:0/2857824797 shutdown_connections 2024-03-06T07:26:46.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.842+0000 7fd05cdac640 1 -- 172.21.15.1:0/2857824797 wait complete. 2024-03-06T07:26:46.856 INFO:tasks.ceph.mds.b.smithi098.stderr: -151> 2024-03-06T07:26:42.904+0000 7ff26172f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:46.862 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:26:46.880 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:26:46.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.971+0000 7fb4f2d02640 1 Processor -- start 2024-03-06T07:26:46.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.971+0000 7fb4f2d02640 1 -- start start 2024-03-06T07:26:46.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.971+0000 7fb4f2d02640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb4ec0c16e0 0x7fb4ec152470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.971+0000 7fb4f2d02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4ec0c70c0 0x7fb4ec155550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.971+0000 7fb4f2d02640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb4ec1529b0 0x7fb4ec155a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.971+0000 7fb4f2d02640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4ec158890 con 0x7fb4ec0c16e0 2024-03-06T07:26:46.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.971+0000 7fb4f2d02640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4ec158b70 con 0x7fb4ec1529b0 2024-03-06T07:26:46.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.971+0000 7fb4f2d02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec158e50 con 0x7fb4ec0c70c0 2024-03-06T07:26:46.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.971+0000 7fb4f0a9e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb4ec0c16e0 0x7fb4ec152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.972+0000 7fb4f0a9e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb4ec0c16e0 0x7fb4ec152470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39512/0 (socket says 172.21.15.1:39512) 2024-03-06T07:26:46.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.972+0000 7fb4f0a9e640 1 -- 172.21.15.1:0/639075610 learned_addr learned my addr 172.21.15.1:0/639075610 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:46.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.972+0000 7fb4ebfff640 1 --2- 172.21.15.1:0/639075610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4ec0c70c0 0x7fb4ec155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.972+0000 7fb4f129f640 1 --2- 172.21.15.1:0/639075610 >> v2:172.21.15.98:3300/0 conn(0x7fb4ec1529b0 0x7fb4ec155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.972+0000 7fb4f0a9e640 1 -- 172.21.15.1:0/639075610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4ec0c70c0 msgr2=0x7fb4ec155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.972+0000 7fb4f0a9e640 1 --2- 172.21.15.1:0/639075610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4ec0c70c0 0x7fb4ec155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.972+0000 7fb4f0a9e640 1 -- 172.21.15.1:0/639075610 >> v2:172.21.15.98:3300/0 conn(0x7fb4ec1529b0 msgr2=0x7fb4ec155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.972+0000 7fb4f0a9e640 1 --2- 172.21.15.1:0/639075610 >> v2:172.21.15.98:3300/0 conn(0x7fb4ec1529b0 0x7fb4ec155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.972+0000 7fb4f0a9e640 1 -- 172.21.15.1:0/639075610 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec159130 con 0x7fb4ec0c16e0 2024-03-06T07:26:46.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.973+0000 7fb4f0a9e640 1 --2- 172.21.15.1:0/639075610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4ec0c16e0 0x7fb4ec152470 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec0cbb50 tx=0x7fb4e402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=245023dc01849a24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.973+0000 7fb4eb7fe640 1 -- 172.21.15.1:0/639075610 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e402c9e0 con 0x7fb4ec0c16e0 2024-03-06T07:26:46.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.973+0000 7fb4eb7fe640 1 -- 172.21.15.1:0/639075610 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e402cb80 con 0x7fb4ec0c16e0 2024-03-06T07:26:46.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.973+0000 7fb4eb7fe640 1 -- 172.21.15.1:0/639075610 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e40343f0 con 0x7fb4ec0c16e0 2024-03-06T07:26:46.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.973+0000 7fb4f2d02640 1 -- 172.21.15.1:0/639075610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4ec0c16e0 msgr2=0x7fb4ec152470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.973+0000 7fb4f2d02640 1 --2- 172.21.15.1:0/639075610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4ec0c16e0 0x7fb4ec152470 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec0cbb50 tx=0x7fb4e402be40 comp rx=0 tx=0).stop 2024-03-06T07:26:46.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.973+0000 7fb4f2d02640 1 -- 172.21.15.1:0/639075610 shutdown_connections 2024-03-06T07:26:46.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.973+0000 7fb4f2d02640 1 --2- 172.21.15.1:0/639075610 >> v2:172.21.15.98:3300/0 conn(0x7fb4ec1529b0 0x7fb4ec155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.973+0000 7fb4f2d02640 1 --2- 172.21.15.1:0/639075610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4ec0c70c0 0x7fb4ec155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.973+0000 7fb4f2d02640 1 --2- 172.21.15.1:0/639075610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4ec0c16e0 0x7fb4ec152470 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.973+0000 7fb4f2d02640 1 -- 172.21.15.1:0/639075610 >> 172.21.15.1:0/639075610 conn(0x7fb4ec066d00 msgr2=0x7fb4ec067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:46.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.973+0000 7fb4f2d02640 1 -- 172.21.15.1:0/639075610 shutdown_connections 2024-03-06T07:26:46.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.973+0000 7fb4f2d02640 1 -- 172.21.15.1:0/639075610 wait complete. 2024-03-06T07:26:46.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.974+0000 7fb4f2d02640 1 Processor -- start 2024-03-06T07:26:46.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.974+0000 7fb4f2d02640 1 -- start start 2024-03-06T07:26:46.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.975+0000 7fb4f2d02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4ec0c16e0 0x7fb4ec0c3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.975+0000 7fb4f2d02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4ec0c70c0 0x7fb4ec1eb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.975+0000 7fb4f2d02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4ec1529b0 0x7fb4ec1ebc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.975+0000 7fb4f2d02640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec0c68c0 con 0x7fb4ec0c16e0 2024-03-06T07:26:46.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.975+0000 7fb4f2d02640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec1596f0 con 0x7fb4ec0c70c0 2024-03-06T07:26:46.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.975+0000 7fb4f2d02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec0cb470 con 0x7fb4ec1529b0 2024-03-06T07:26:46.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.975+0000 7fb4f0a9e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4ec0c16e0 0x7fb4ec0c3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.975+0000 7fb4f0a9e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4ec0c16e0 0x7fb4ec0c3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39520/0 (socket says 172.21.15.1:39520) 2024-03-06T07:26:46.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.975+0000 7fb4f0a9e640 1 -- 172.21.15.1:0/3628875336 learned_addr learned my addr 172.21.15.1:0/3628875336 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:46.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.975+0000 7fb4f129f640 1 --2- 172.21.15.1:0/3628875336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4ec1529b0 0x7fb4ec1ebc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.975+0000 7fb4ebfff640 1 --2- 172.21.15.1:0/3628875336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4ec0c70c0 0x7fb4ec1eb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.976+0000 7fb4f0a9e640 1 -- 172.21.15.1:0/3628875336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4ec1529b0 msgr2=0x7fb4ec1ebc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.976+0000 7fb4f0a9e640 1 --2- 172.21.15.1:0/3628875336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4ec1529b0 0x7fb4ec1ebc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.976+0000 7fb4f0a9e640 1 -- 172.21.15.1:0/3628875336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4ec0c70c0 msgr2=0x7fb4ec1eb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.976+0000 7fb4f0a9e640 1 --2- 172.21.15.1:0/3628875336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4ec0c70c0 0x7fb4ec1eb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.976+0000 7fb4f0a9e640 1 -- 172.21.15.1:0/3628875336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec1ef500 con 0x7fb4ec0c16e0 2024-03-06T07:26:46.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.976+0000 7fb4f0a9e640 1 --2- 172.21.15.1:0/3628875336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4ec0c16e0 0x7fb4ec0c3650 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4036040 tx=0x7fb4e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.976+0000 7fb4e97fa640 1 -- 172.21.15.1:0/3628875336 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4043020 con 0x7fb4ec0c16e0 2024-03-06T07:26:46.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.976+0000 7fb4e97fa640 1 -- 172.21.15.1:0/3628875336 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e403e450 con 0x7fb4ec0c16e0 2024-03-06T07:26:46.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.976+0000 7fb4f2d02640 1 -- 172.21.15.1:0/3628875336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4ec157290 con 0x7fb4ec0c16e0 2024-03-06T07:26:46.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.976+0000 7fb4e97fa640 1 -- 172.21.15.1:0/3628875336 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4009510 con 0x7fb4ec0c16e0 2024-03-06T07:26:46.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.976+0000 7fb4f2d02640 1 -- 172.21.15.1:0/3628875336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4ec157850 con 0x7fb4ec0c16e0 2024-03-06T07:26:46.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.977+0000 7fb4f2d02640 1 -- 172.21.15.1:0/3628875336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4ec0680b0 con 0x7fb4ec0c16e0 2024-03-06T07:26:46.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.978+0000 7fb4e97fa640 1 -- 172.21.15.1:0/3628875336 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb4e4046020 con 0x7fb4ec0c16e0 2024-03-06T07:26:46.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.978+0000 7fb4e97fa640 1 --2- 172.21.15.1:0/3628875336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb4d806b500 0x7fb4d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.978+0000 7fb4e97fa640 1 -- 172.21.15.1:0/3628875336 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 7301+0+0 (secure 0 0 0) 0x7fb4e40accf0 con 0x7fb4ec0c16e0 2024-03-06T07:26:46.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.978+0000 7fb4ebfff640 1 --2- 172.21.15.1:0/3628875336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb4d806b500 0x7fb4d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.979+0000 7fb4ebfff640 1 --2- 172.21.15.1:0/3628875336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb4d806b500 0x7fb4d806d9c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec155200 tx=0x7fb4e0009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:46.982+0000 7fb4e97fa640 1 -- 172.21.15.1:0/3628875336 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb4e40b1070 con 0x7fb4ec0c16e0 2024-03-06T07:26:47.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.163+0000 7fb4f2d02640 1 -- 172.21.15.1:0/3628875336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb4ec0b0880 con 0x7fb4ec0c16e0 2024-03-06T07:26:47.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.845+0000 7fb4e97fa640 1 -- 172.21.15.1:0/3628875336 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v158) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb4e404d030 con 0x7fb4ec0c16e0 2024-03-06T07:26:47.848 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:26:47.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.850+0000 7fb4f2d02640 1 -- 172.21.15.1:0/3628875336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb4d806b500 msgr2=0x7fb4d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.850+0000 7fb4f2d02640 1 --2- 172.21.15.1:0/3628875336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb4d806b500 0x7fb4d806d9c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec155200 tx=0x7fb4e0009190 comp rx=0 tx=0).stop 2024-03-06T07:26:47.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.850+0000 7fb4f2d02640 1 -- 172.21.15.1:0/3628875336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4ec0c16e0 msgr2=0x7fb4ec0c3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.850+0000 7fb4f2d02640 1 --2- 172.21.15.1:0/3628875336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4ec0c16e0 0x7fb4ec0c3650 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4036040 tx=0x7fb4e4004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:47.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.850+0000 7fb4f2d02640 1 -- 172.21.15.1:0/3628875336 shutdown_connections 2024-03-06T07:26:47.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.850+0000 7fb4f2d02640 1 --2- 172.21.15.1:0/3628875336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb4d806b500 0x7fb4d806d9c0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.850+0000 7fb4f2d02640 1 --2- 172.21.15.1:0/3628875336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb4ec1529b0 0x7fb4ec1ebc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.850+0000 7fb4f2d02640 1 --2- 172.21.15.1:0/3628875336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4ec0c70c0 0x7fb4ec1eb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.850+0000 7fb4f2d02640 1 --2- 172.21.15.1:0/3628875336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4ec0c16e0 0x7fb4ec0c3650 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.850+0000 7fb4f2d02640 1 -- 172.21.15.1:0/3628875336 >> 172.21.15.1:0/3628875336 conn(0x7fb4ec066d00 msgr2=0x7fb4ec074450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:47.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.851+0000 7fb4f2d02640 1 -- 172.21.15.1:0/3628875336 shutdown_connections 2024-03-06T07:26:47.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.851+0000 7fb4f2d02640 1 -- 172.21.15.1:0/3628875336 wait complete. 2024-03-06T07:26:47.867 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:26:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.976+0000 7fdff649d640 1 Processor -- start 2024-03-06T07:26:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.976+0000 7fdff649d640 1 -- start start 2024-03-06T07:26:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.977+0000 7fdff649d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdff00c6800 0x7fdff0156a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.977+0000 7fdff649d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdff0156f90 0x7fdff01573c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.977+0000 7fdff649d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff0158560 con 0x7fdff0158180 2024-03-06T07:26:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.977+0000 7fdff649d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff0156210 con 0x7fdff0156f90 2024-03-06T07:26:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.977+0000 7fdff649d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff00c3ea0 con 0x7fdff00c6800 2024-03-06T07:26:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.977+0000 7fdff4a3a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdff0156f90 0x7fdff01573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.977+0000 7fdfeffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdff0158180 0x7fdff00c58e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33558/0 (socket says 172.21.15.1:33558) 2024-03-06T07:26:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.977+0000 7fdfeffff640 1 -- 172.21.15.1:0/1615819142 learned_addr learned my addr 172.21.15.1:0/1615819142 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.977+0000 7fdff4a3a640 1 -- 172.21.15.1:0/1615819142 >> v2:172.21.15.98:3300/0 conn(0x7fdff00c6800 msgr2=0x7fdff0156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.977+0000 7fdff4a3a640 1 --2- 172.21.15.1:0/1615819142 >> v2:172.21.15.98:3300/0 conn(0x7fdff00c6800 0x7fdff0156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.977+0000 7fdff4a3a640 1 -- 172.21.15.1:0/1615819142 >> v1:172.21.15.1:6789/0 conn(0x7fdff0158180 legacy=0x7fdff00c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.977+0000 7fdff4a3a640 1 -- 172.21.15.1:0/1615819142 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff0155600 con 0x7fdff0156f90 2024-03-06T07:26:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.977+0000 7fdfeeffd640 1 -- 172.21.15.1:0/1615819142 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550184781 0 0) 0x7fdff0158560 con 0x7fdff0158180 2024-03-06T07:26:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.978+0000 7fdff4a3a640 1 --2- 172.21.15.1:0/1615819142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdff0156f90 0x7fdff01573c0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fdff0159880 tx=0x7fdfe802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ad518be0d592e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.978+0000 7fdfeeffd640 1 -- 172.21.15.1:0/1615819142 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe8039070 con 0x7fdff0156f90 2024-03-06T07:26:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.978+0000 7fdfeeffd640 1 -- 172.21.15.1:0/1615819142 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe802cb60 con 0x7fdff0156f90 2024-03-06T07:26:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.978+0000 7fdfeeffd640 1 -- 172.21.15.1:0/1615819142 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe8034440 con 0x7fdff0156f90 2024-03-06T07:26:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.979+0000 7fdff649d640 1 -- 172.21.15.1:0/1615819142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdff0156f90 msgr2=0x7fdff01573c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.979+0000 7fdff649d640 1 --2- 172.21.15.1:0/1615819142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdff0156f90 0x7fdff01573c0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fdff0159880 tx=0x7fdfe802bdb0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.979+0000 7fdff649d640 1 -- 172.21.15.1:0/1615819142 shutdown_connections 2024-03-06T07:26:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.979+0000 7fdff649d640 1 --2- 172.21.15.1:0/1615819142 >> v2:172.21.15.98:3300/0 conn(0x7fdff00c6800 0x7fdff0156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.979+0000 7fdff649d640 1 --2- 172.21.15.1:0/1615819142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdff0156f90 0x7fdff01573c0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.979+0000 7fdff649d640 1 -- 172.21.15.1:0/1615819142 >> 172.21.15.1:0/1615819142 conn(0x7fdff006b410 msgr2=0x7fdff006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.979+0000 7fdff649d640 1 -- 172.21.15.1:0/1615819142 shutdown_connections 2024-03-06T07:26:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.979+0000 7fdff649d640 1 -- 172.21.15.1:0/1615819142 wait complete. 2024-03-06T07:26:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.980+0000 7fdff649d640 1 Processor -- start 2024-03-06T07:26:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.980+0000 7fdff649d640 1 -- start start 2024-03-06T07:26:47.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.981+0000 7fdff649d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdff00c6800 0x7fdff01527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.981+0000 7fdff649d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdff0156f90 0x7fdff01e7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.981+0000 7fdff649d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff0158180 0x7fdff0070080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.981+0000 7fdff649d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff0155ea0 con 0x7fdff00c6800 2024-03-06T07:26:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.981+0000 7fdff649d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff0155bc0 con 0x7fdff0158180 2024-03-06T07:26:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.981+0000 7fdff649d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff00c5ff0 con 0x7fdff0156f90 2024-03-06T07:26:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.981+0000 7fdfeffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdff00c6800 0x7fdff01527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.981+0000 7fdfeffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdff00c6800 0x7fdff01527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39524/0 (socket says 172.21.15.1:39524) 2024-03-06T07:26:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.981+0000 7fdfeffff640 1 -- 172.21.15.1:0/1908257196 learned_addr learned my addr 172.21.15.1:0/1908257196 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.981+0000 7fdfeffff640 1 -- 172.21.15.1:0/1908257196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdff0156f90 msgr2=0x7fdff01e7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.981+0000 7fdfeffff640 1 --2- 172.21.15.1:0/1908257196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdff0156f90 0x7fdff01e7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.981+0000 7fdfeffff640 1 -- 172.21.15.1:0/1908257196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff0158180 msgr2=0x7fdff0070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.981+0000 7fdfeffff640 1 --2- 172.21.15.1:0/1908257196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff0158180 0x7fdff0070080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.981+0000 7fdfeffff640 1 -- 172.21.15.1:0/1908257196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff0152e10 con 0x7fdff00c6800 2024-03-06T07:26:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.981+0000 7fdfeffff640 1 --2- 172.21.15.1:0/1908257196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdff00c6800 0x7fdff01527a0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fdfd800c430 tx=0x7fdfd800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.982+0000 7fdfecff9640 1 -- 172.21.15.1:0/1908257196 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd80074e0 con 0x7fdff00c6800 2024-03-06T07:26:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.982+0000 7fdfecff9640 1 -- 172.21.15.1:0/1908257196 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd80047e0 con 0x7fdff00c6800 2024-03-06T07:26:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.982+0000 7fdff649d640 1 -- 172.21.15.1:0/1908257196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff00ac560 con 0x7fdff00c6800 2024-03-06T07:26:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.982+0000 7fdfecff9640 1 -- 172.21.15.1:0/1908257196 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8004bd0 con 0x7fdff00c6800 2024-03-06T07:26:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.982+0000 7fdff649d640 1 -- 172.21.15.1:0/1908257196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff00acb20 con 0x7fdff00c6800 2024-03-06T07:26:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.983+0000 7fdff649d640 1 -- 172.21.15.1:0/1908257196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff00680b0 con 0x7fdff00c6800 2024-03-06T07:26:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.983+0000 7fdfecff9640 1 -- 172.21.15.1:0/1908257196 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdfd8011070 con 0x7fdff00c6800 2024-03-06T07:26:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.983+0000 7fdfecff9640 1 --2- 172.21.15.1:0/1908257196 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfd006b4d0 0x7fdfd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.984+0000 7fdfecff9640 1 -- 172.21.15.1:0/1908257196 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6863+0+0 (secure 0 0 0) 0x7fdfd808a220 con 0x7fdff00c6800 2024-03-06T07:26:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.984+0000 7fdfef7fe640 1 --2- 172.21.15.1:0/1908257196 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfd006b4d0 0x7fdfd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.984+0000 7fdfef7fe640 1 --2- 172.21.15.1:0/1908257196 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfd006b4d0 0x7fdfd006d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fdff00ad340 tx=0x7fdfe0006c90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:47.988+0000 7fdfecff9640 1 -- 172.21.15.1:0/1908257196 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdfd8058cb0 con 0x7fdff00c6800 2024-03-06T07:26:48.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.175+0000 7fdff649d640 1 -- 172.21.15.1:0/1908257196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdff00c2df0 con 0x7fdff00c6800 2024-03-06T07:26:48.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.861+0000 7fdfecff9640 1 -- 172.21.15.1:0/1908257196 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v159) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdfd80585b0 con 0x7fdff00c6800 2024-03-06T07:26:48.863 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-06T07:26:48.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.865+0000 7fdff649d640 1 -- 172.21.15.1:0/1908257196 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfd006b4d0 msgr2=0x7fdfd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.865+0000 7fdff649d640 1 --2- 172.21.15.1:0/1908257196 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfd006b4d0 0x7fdfd006d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fdff00ad340 tx=0x7fdfe0006c90 comp rx=0 tx=0).stop 2024-03-06T07:26:48.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.865+0000 7fdff649d640 1 -- 172.21.15.1:0/1908257196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdff00c6800 msgr2=0x7fdff01527a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.865+0000 7fdff649d640 1 --2- 172.21.15.1:0/1908257196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdff00c6800 0x7fdff01527a0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fdfd800c430 tx=0x7fdfd800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.865+0000 7fdff649d640 1 -- 172.21.15.1:0/1908257196 shutdown_connections 2024-03-06T07:26:48.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.865+0000 7fdff649d640 1 --2- 172.21.15.1:0/1908257196 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdfd006b4d0 0x7fdfd006d990 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.865+0000 7fdff649d640 1 --2- 172.21.15.1:0/1908257196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff0158180 0x7fdff0070080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.865+0000 7fdff649d640 1 --2- 172.21.15.1:0/1908257196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdff0156f90 0x7fdff01e7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.865+0000 7fdff649d640 1 --2- 172.21.15.1:0/1908257196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdff00c6800 0x7fdff01527a0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.865+0000 7fdff649d640 1 -- 172.21.15.1:0/1908257196 >> 172.21.15.1:0/1908257196 conn(0x7fdff006b410 msgr2=0x7fdff00c58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.865+0000 7fdff649d640 1 -- 172.21.15.1:0/1908257196 shutdown_connections 2024-03-06T07:26:48.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.865+0000 7fdff649d640 1 -- 172.21.15.1:0/1908257196 wait complete. 2024-03-06T07:26:48.884 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:48.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.988+0000 7f5084bc7640 1 Processor -- start 2024-03-06T07:26:48.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.988+0000 7f5084bc7640 1 -- start start 2024-03-06T07:26:48.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.989+0000 7f5084bc7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f50800c3eb0 0x7f50800c7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.989+0000 7f5084bc7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f508006d060 0x7f50801558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.989+0000 7f5084bc7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50800c71c0 con 0x7f50800c7940 2024-03-06T07:26:48.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.989+0000 7f5084bc7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50800c59b0 con 0x7f50800c3eb0 2024-03-06T07:26:48.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.989+0000 7f5084bc7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50800c4c00 con 0x7f508006d060 2024-03-06T07:26:48.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.989+0000 7f507e59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f50800c3eb0 0x7f50800c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.989+0000 7f5077fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f508006d060 0x7f50801558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.989+0000 7f5077fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f508006d060 0x7f50801558b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:57576/0 (socket says 172.21.15.1:57576) 2024-03-06T07:26:48.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.989+0000 7f507e59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f50800c3eb0 0x7f50800c7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:41750/0 (socket says 172.21.15.1:41750) 2024-03-06T07:26:48.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.989+0000 7f5077fff640 1 -- 172.21.15.1:0/1621797709 learned_addr learned my addr 172.21.15.1:0/1621797709 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:48.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.989+0000 7f507ed9d640 1 --1- 172.21.15.1:0/1621797709 >> v1:172.21.15.98:6789/0 conn(0x7f50800c7940 0x7f5080158760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:53548/0 (socket says 172.21.15.1:53548) 2024-03-06T07:26:48.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.989+0000 7f507e59c640 1 -- 172.21.15.1:0/1621797709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f508006d060 msgr2=0x7f50801558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:48.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.989+0000 7f507e59c640 1 --2- 172.21.15.1:0/1621797709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f508006d060 0x7f50801558b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.989+0000 7f507e59c640 1 -- 172.21.15.1:0/1621797709 >> v1:172.21.15.98:6789/0 conn(0x7f50800c7940 legacy=0x7f5080158760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.989+0000 7f507e59c640 1 -- 172.21.15.1:0/1621797709 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50800c4e50 con 0x7f50800c3eb0 2024-03-06T07:26:48.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.989+0000 7f5077fff640 1 --2- 172.21.15.1:0/1621797709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f508006d060 0x7f50801558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.990+0000 7f507e59c640 1 --2- 172.21.15.1:0/1621797709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50800c3eb0 0x7f50800c7400 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f50700097a0 tx=0x7f507002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33cb27096db3ddac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.990+0000 7f507dd9b640 1 -- 172.21.15.1:0/1621797709 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070038070 con 0x7f50800c3eb0 2024-03-06T07:26:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.990+0000 7f507dd9b640 1 -- 172.21.15.1:0/1621797709 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5070031700 con 0x7f50800c3eb0 2024-03-06T07:26:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.990+0000 7f507dd9b640 1 -- 172.21.15.1:0/1621797709 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070031ad0 con 0x7f50800c3eb0 2024-03-06T07:26:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.991+0000 7f5084bc7640 1 -- 172.21.15.1:0/1621797709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50800c3eb0 msgr2=0x7f50800c7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.991+0000 7f5084bc7640 1 --2- 172.21.15.1:0/1621797709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50800c3eb0 0x7f50800c7400 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f50700097a0 tx=0x7f507002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.991+0000 7f5084bc7640 1 -- 172.21.15.1:0/1621797709 shutdown_connections 2024-03-06T07:26:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.991+0000 7f5084bc7640 1 --2- 172.21.15.1:0/1621797709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f508006d060 0x7f50801558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.991+0000 7f5084bc7640 1 --2- 172.21.15.1:0/1621797709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50800c3eb0 0x7f50800c7400 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.991+0000 7f5084bc7640 1 -- 172.21.15.1:0/1621797709 >> 172.21.15.1:0/1621797709 conn(0x7f5080066d00 msgr2=0x7f5080067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.991+0000 7f5084bc7640 1 -- 172.21.15.1:0/1621797709 shutdown_connections 2024-03-06T07:26:48.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.991+0000 7f5084bc7640 1 -- 172.21.15.1:0/1621797709 wait complete. 2024-03-06T07:26:48.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.992+0000 7f5084bc7640 1 Processor -- start 2024-03-06T07:26:48.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.992+0000 7f5084bc7640 1 -- start start 2024-03-06T07:26:48.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f5084bc7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f508006d060 0x7f50800cac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f5084bc7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50800c3eb0 0x7f50801ee480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f5084bc7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50800c7940 0x7f50801ee9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f5084bc7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50800c4290 con 0x7f50800c7940 2024-03-06T07:26:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f5084bc7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5080155420 con 0x7f508006d060 2024-03-06T07:26:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f5084bc7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f508014a7c0 con 0x7f50800c3eb0 2024-03-06T07:26:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f507ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50800c7940 0x7f50801ee9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f507ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50800c7940 0x7f50801ee9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39538/0 (socket says 172.21.15.1:39538) 2024-03-06T07:26:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f507ed9d640 1 -- 172.21.15.1:0/3529741227 learned_addr learned my addr 172.21.15.1:0/3529741227 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f5077fff640 1 --2- 172.21.15.1:0/3529741227 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50800c3eb0 0x7f50801ee480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f507e59c640 1 --2- 172.21.15.1:0/3529741227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f508006d060 0x7f50800cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f507ed9d640 1 -- 172.21.15.1:0/3529741227 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50800c3eb0 msgr2=0x7f50801ee480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f507ed9d640 1 --2- 172.21.15.1:0/3529741227 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50800c3eb0 0x7f50801ee480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f507ed9d640 1 -- 172.21.15.1:0/3529741227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f508006d060 msgr2=0x7f50800cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f507ed9d640 1 --2- 172.21.15.1:0/3529741227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f508006d060 0x7f50800cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f507ed9d640 1 -- 172.21.15.1:0/3529741227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50801474e0 con 0x7f50800c7940 2024-03-06T07:26:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f5077fff640 1 --2- 172.21.15.1:0/3529741227 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50800c3eb0 0x7f50801ee480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f507e59c640 1 --2- 172.21.15.1:0/3529741227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f508006d060 0x7f50800cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.993+0000 7f507ed9d640 1 --2- 172.21.15.1:0/3529741227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50800c7940 0x7f50801ee9c0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f507800c780 tx=0x7f507800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.994+0000 7f5076ffd640 1 -- 172.21.15.1:0/3529741227 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5078017070 con 0x7f50800c7940 2024-03-06T07:26:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.994+0000 7f5076ffd640 1 -- 172.21.15.1:0/3529741227 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507800a640 con 0x7f50800c7940 2024-03-06T07:26:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.994+0000 7f5084bc7640 1 -- 172.21.15.1:0/3529741227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5080148a50 con 0x7f50800c7940 2024-03-06T07:26:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.994+0000 7f5076ffd640 1 -- 172.21.15.1:0/3529741227 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507800aa30 con 0x7f50800c7940 2024-03-06T07:26:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.994+0000 7f5084bc7640 1 -- 172.21.15.1:0/3529741227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5080149010 con 0x7f50800c7940 2024-03-06T07:26:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.995+0000 7f5084bc7640 1 -- 172.21.15.1:0/3529741227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5080067fd0 con 0x7f50800c7940 2024-03-06T07:26:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.995+0000 7f5076ffd640 1 -- 172.21.15.1:0/3529741227 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f507801a070 con 0x7f50800c7940 2024-03-06T07:26:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.995+0000 7f5076ffd640 1 --2- 172.21.15.1:0/3529741227 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f506006b500 0x7f506006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.995+0000 7f5076ffd640 1 -- 172.21.15.1:0/3529741227 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f5078089920 con 0x7f50800c7940 2024-03-06T07:26:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.996+0000 7f507e59c640 1 --2- 172.21.15.1:0/3529741227 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f506006b500 0x7f506006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:48.996+0000 7f507e59c640 1 --2- 172.21.15.1:0/3529741227 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f506006b500 0x7f506006d9c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f5070035000 tx=0x7f5070008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.000+0000 7f5076ffd640 1 -- 172.21.15.1:0/3529741227 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5078057ef0 con 0x7f50800c7940 2024-03-06T07:26:49.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.181+0000 7f5084bc7640 1 -- 172.21.15.1:0/3529741227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50800c2180 con 0x7f50800c7940 2024-03-06T07:26:49.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.182+0000 7f5076ffd640 1 -- 172.21.15.1:0/3529741227 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v159) v1 ==== 74+0+17283 (secure 0 0 0) 0x7f507805bf90 con 0x7f50800c7940 2024-03-06T07:26:49.184 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:49.184 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":159,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:26:48.853480+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:26:24.304587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:26:25.085741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6837/2360179300":"2024-03-07T07:26:46.842519+0000","172.21.15.98:6835/2360179300":"2024-03-07T07:26:46.842519+0000","172.21.15.1:6833/1089548586":"2024-03-07T07:26:36.687217+0000","172.21.15.1:6832/1089548586":"2024-03-07T07:26:36.687217+0000","172.21.15.98:6836/3360241462":"2024-03-07T07:26:45.837702+0000","172.21.15.98:6834/3360241462":"2024-03-07T07:26:45.837702+0000","172.21.15.1:6835/67091521":"2024-03-07T07:26:36.682683+0000","172.21.15.1:6834/67091521":"2024-03-07T07:26:36.682683+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:49.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.184+0000 7f5084bc7640 1 -- 172.21.15.1:0/3529741227 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f506006b500 msgr2=0x7f506006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.184+0000 7f5084bc7640 1 --2- 172.21.15.1:0/3529741227 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f506006b500 0x7f506006d9c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f5070035000 tx=0x7f5070008030 comp rx=0 tx=0).stop 2024-03-06T07:26:49.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.184+0000 7f5084bc7640 1 -- 172.21.15.1:0/3529741227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50800c7940 msgr2=0x7f50801ee9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.184+0000 7f5084bc7640 1 --2- 172.21.15.1:0/3529741227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50800c7940 0x7f50801ee9c0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f507800c780 tx=0x7f507800cc40 comp rx=0 tx=0).stop 2024-03-06T07:26:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.184+0000 7f5084bc7640 1 -- 172.21.15.1:0/3529741227 shutdown_connections 2024-03-06T07:26:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.184+0000 7f5084bc7640 1 --2- 172.21.15.1:0/3529741227 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f506006b500 0x7f506006d9c0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.184+0000 7f5084bc7640 1 --2- 172.21.15.1:0/3529741227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50800c7940 0x7f50801ee9c0 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.184+0000 7f5084bc7640 1 --2- 172.21.15.1:0/3529741227 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50800c3eb0 0x7f50801ee480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.184+0000 7f5084bc7640 1 --2- 172.21.15.1:0/3529741227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f508006d060 0x7f50800cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.184+0000 7f5084bc7640 1 -- 172.21.15.1:0/3529741227 >> 172.21.15.1:0/3529741227 conn(0x7f5080066d00 msgr2=0x7f50801539c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.184+0000 7f5084bc7640 1 -- 172.21.15.1:0/3529741227 shutdown_connections 2024-03-06T07:26:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.184+0000 7f5084bc7640 1 -- 172.21.15.1:0/3529741227 wait complete. 2024-03-06T07:26:49.197 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-06T07:26:24.304587+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '152', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-06T07:26:25.085741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:49.197 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:49.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.300+0000 7fb92f1e9640 1 Processor -- start 2024-03-06T07:26:49.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.300+0000 7fb92f1e9640 1 -- start start 2024-03-06T07:26:49.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.300+0000 7fb92f1e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9280c7560 0x7fb928155860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.300+0000 7fb92f1e9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb928155da0 0x7fb9280c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.300+0000 7fb92f1e9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb928155080 con 0x7fb928152890 2024-03-06T07:26:49.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.300+0000 7fb92f1e9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9280c35b0 con 0x7fb928155da0 2024-03-06T07:26:49.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.300+0000 7fb92f1e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280c3890 con 0x7fb9280c7560 2024-03-06T07:26:49.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.300+0000 7fb92d786640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb928155da0 0x7fb9280c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.301+0000 7fb92d786640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb928155da0 0x7fb9280c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39550/0 (socket says 172.21.15.1:39550) 2024-03-06T07:26:49.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.301+0000 7fb92d786640 1 -- 172.21.15.1:0/1810647596 learned_addr learned my addr 172.21.15.1:0/1810647596 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:49.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.301+0000 7fb927fff640 1 --2- 172.21.15.1:0/1810647596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9280c7560 0x7fb928155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.301+0000 7fb92d786640 1 -- 172.21.15.1:0/1810647596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9280c7560 msgr2=0x7fb928155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.301+0000 7fb92d786640 1 --2- 172.21.15.1:0/1810647596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9280c7560 0x7fb928155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.301+0000 7fb92d786640 1 -- 172.21.15.1:0/1810647596 >> v1:172.21.15.98:6789/0 conn(0x7fb928152890 legacy=0x7fb928159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.301+0000 7fb92d786640 1 -- 172.21.15.1:0/1810647596 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9280c3b70 con 0x7fb928155da0 2024-03-06T07:26:49.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.302+0000 7fb92d786640 1 --2- 172.21.15.1:0/1810647596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb928155da0 0x7fb9280c8cf0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fb920009e80 tx=0x7fb92002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20c1c809a1615425 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.302+0000 7fb9277fe640 1 -- 172.21.15.1:0/1810647596 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92002c9c0 con 0x7fb928155da0 2024-03-06T07:26:49.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.302+0000 7fb9277fe640 1 -- 172.21.15.1:0/1810647596 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92002cb60 con 0x7fb928155da0 2024-03-06T07:26:49.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.302+0000 7fb9277fe640 1 -- 172.21.15.1:0/1810647596 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9200343f0 con 0x7fb928155da0 2024-03-06T07:26:49.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.302+0000 7fb92f1e9640 1 -- 172.21.15.1:0/1810647596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb928155da0 msgr2=0x7fb9280c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.302+0000 7fb92f1e9640 1 --2- 172.21.15.1:0/1810647596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb928155da0 0x7fb9280c8cf0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fb920009e80 tx=0x7fb92002be20 comp rx=0 tx=0).stop 2024-03-06T07:26:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.302+0000 7fb92f1e9640 1 -- 172.21.15.1:0/1810647596 shutdown_connections 2024-03-06T07:26:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.302+0000 7fb92f1e9640 1 --2- 172.21.15.1:0/1810647596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb928155da0 0x7fb9280c8cf0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.302+0000 7fb92f1e9640 1 --2- 172.21.15.1:0/1810647596 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9280c7560 0x7fb928155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.302+0000 7fb92f1e9640 1 -- 172.21.15.1:0/1810647596 >> 172.21.15.1:0/1810647596 conn(0x7fb92806b3e0 msgr2=0x7fb92806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.302+0000 7fb92f1e9640 1 -- 172.21.15.1:0/1810647596 shutdown_connections 2024-03-06T07:26:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.302+0000 7fb92f1e9640 1 -- 172.21.15.1:0/1810647596 wait complete. 2024-03-06T07:26:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.303+0000 7fb92f1e9640 1 Processor -- start 2024-03-06T07:26:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.303+0000 7fb92f1e9640 1 -- start start 2024-03-06T07:26:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.304+0000 7fb92f1e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9280c7560 0x7fb928156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.304+0000 7fb92f1e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb928152890 0x7fb9280a8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.304+0000 7fb92f1e9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9281504d0 0x7fb928157250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.304+0000 7fb92f1e9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280a81f0 con 0x7fb9281504d0 2024-03-06T07:26:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.304+0000 7fb92f1e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280c3260 con 0x7fb928152890 2024-03-06T07:26:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.304+0000 7fb92f1e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280a8370 con 0x7fb9280c7560 2024-03-06T07:26:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.304+0000 7fb92d786640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9281504d0 0x7fb928157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.304+0000 7fb92d786640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9281504d0 0x7fb928157250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39552/0 (socket says 172.21.15.1:39552) 2024-03-06T07:26:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.304+0000 7fb92d786640 1 -- 172.21.15.1:0/4011510414 learned_addr learned my addr 172.21.15.1:0/4011510414 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.305+0000 7fb927fff640 1 --2- 172.21.15.1:0/4011510414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb928152890 0x7fb9280a8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.305+0000 7fb92cf85640 1 --2- 172.21.15.1:0/4011510414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9280c7560 0x7fb928156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.305+0000 7fb92d786640 1 -- 172.21.15.1:0/4011510414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9280c7560 msgr2=0x7fb928156570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.305+0000 7fb92d786640 1 --2- 172.21.15.1:0/4011510414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9280c7560 0x7fb928156570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.305+0000 7fb92d786640 1 -- 172.21.15.1:0/4011510414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb928152890 msgr2=0x7fb9280a8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.305+0000 7fb92d786640 1 --2- 172.21.15.1:0/4011510414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb928152890 0x7fb9280a8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.305+0000 7fb92d786640 1 -- 172.21.15.1:0/4011510414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9280a8a90 con 0x7fb9281504d0 2024-03-06T07:26:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.305+0000 7fb92cf85640 1 --2- 172.21.15.1:0/4011510414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9280c7560 0x7fb928156570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.305+0000 7fb92d786640 1 --2- 172.21.15.1:0/4011510414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9281504d0 0x7fb928157250 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fb92002c430 tx=0x7fb920004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.305+0000 7fb9257fa640 1 -- 172.21.15.1:0/4011510414 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb920042020 con 0x7fb9281504d0 2024-03-06T07:26:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.305+0000 7fb9257fa640 1 -- 172.21.15.1:0/4011510414 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92003e450 con 0x7fb9281504d0 2024-03-06T07:26:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.305+0000 7fb92f1e9640 1 -- 172.21.15.1:0/4011510414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9281eb330 con 0x7fb9281504d0 2024-03-06T07:26:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.305+0000 7fb9257fa640 1 -- 172.21.15.1:0/4011510414 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92003e820 con 0x7fb9281504d0 2024-03-06T07:26:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.305+0000 7fb92f1e9640 1 -- 172.21.15.1:0/4011510414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92814fcd0 con 0x7fb9281504d0 2024-03-06T07:26:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.306+0000 7fb92f1e9640 1 -- 172.21.15.1:0/4011510414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb928150000 con 0x7fb9281504d0 2024-03-06T07:26:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.306+0000 7fb9257fa640 1 -- 172.21.15.1:0/4011510414 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb920045020 con 0x7fb9281504d0 2024-03-06T07:26:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.307+0000 7fb9257fa640 1 --2- 172.21.15.1:0/4011510414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb90c06b500 0x7fb90c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.307+0000 7fb9257fa640 1 -- 172.21.15.1:0/4011510414 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6449+0+0 (secure 0 0 0) 0x7fb9200ac5e0 con 0x7fb9281504d0 2024-03-06T07:26:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.307+0000 7fb92cf85640 1 --2- 172.21.15.1:0/4011510414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb90c06b500 0x7fb90c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.308+0000 7fb92cf85640 1 --2- 172.21.15.1:0/4011510414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb90c06b500 0x7fb90c06d9c0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb91c001680 tx=0x7fb91c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.311+0000 7fb9257fa640 1 -- 172.21.15.1:0/4011510414 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb92004c070 con 0x7fb9281504d0 2024-03-06T07:26:49.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.494+0000 7fb92f1e9640 1 -- 172.21.15.1:0/4011510414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb928067fd0 con 0x7fb9281504d0 2024-03-06T07:26:49.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.494+0000 7fb9257fa640 1 -- 172.21.15.1:0/4011510414 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v159) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb92007b200 con 0x7fb9281504d0 2024-03-06T07:26:49.496 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:26:49.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.496+0000 7fb92f1e9640 1 -- 172.21.15.1:0/4011510414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb90c06b500 msgr2=0x7fb90c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.496+0000 7fb92f1e9640 1 --2- 172.21.15.1:0/4011510414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb90c06b500 0x7fb90c06d9c0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb91c001680 tx=0x7fb91c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:49.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.496+0000 7fb92f1e9640 1 -- 172.21.15.1:0/4011510414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9281504d0 msgr2=0x7fb928157250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.496+0000 7fb92f1e9640 1 --2- 172.21.15.1:0/4011510414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9281504d0 0x7fb928157250 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fb92002c430 tx=0x7fb920004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:49.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.497+0000 7fb92f1e9640 1 -- 172.21.15.1:0/4011510414 shutdown_connections 2024-03-06T07:26:49.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.497+0000 7fb92f1e9640 1 --2- 172.21.15.1:0/4011510414 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb90c06b500 0x7fb90c06d9c0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.497+0000 7fb92f1e9640 1 --2- 172.21.15.1:0/4011510414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9281504d0 0x7fb928157250 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.497+0000 7fb92f1e9640 1 --2- 172.21.15.1:0/4011510414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb928152890 0x7fb9280a8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.497+0000 7fb92f1e9640 1 --2- 172.21.15.1:0/4011510414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9280c7560 0x7fb928156570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.497+0000 7fb92f1e9640 1 -- 172.21.15.1:0/4011510414 >> 172.21.15.1:0/4011510414 conn(0x7fb92806b3e0 msgr2=0x7fb9280c6d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.497+0000 7fb92f1e9640 1 -- 172.21.15.1:0/4011510414 shutdown_connections 2024-03-06T07:26:49.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.497+0000 7fb92f1e9640 1 -- 172.21.15.1:0/4011510414 wait complete. 2024-03-06T07:26:49.509 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-06T07:26:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.611+0000 7fd58396c640 1 Processor -- start 2024-03-06T07:26:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.611+0000 7fd58396c640 1 -- start start 2024-03-06T07:26:49.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd58396c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd57c0c8cd0 0x7fd57c0c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd58396c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd57c155b70 0x7fd57c159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd58396c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57c0c2b90 con 0x7fd57c0c3260 2024-03-06T07:26:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd58396c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd57c156020 con 0x7fd57c155b70 2024-03-06T07:26:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd58396c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c0c37b0 con 0x7fd57c0c8cd0 2024-03-06T07:26:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd580f07640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd57c0c3260 0x7fd57c0c7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33566/0 (socket says 172.21.15.1:33566) 2024-03-06T07:26:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd580f07640 1 -- 172.21.15.1:0/1464205913 learned_addr learned my addr 172.21.15.1:0/1464205913 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd581708640 1 --2- 172.21.15.1:0/1464205913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd57c0c8cd0 0x7fd57c0c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd581f09640 1 --2- 172.21.15.1:0/1464205913 >> v2:172.21.15.98:3300/0 conn(0x7fd57c155b70 0x7fd57c159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd573fff640 1 -- 172.21.15.1:0/1464205913 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425610652 0 0) 0x7fd57c0c2b90 con 0x7fd57c0c3260 2024-03-06T07:26:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd573fff640 1 -- 172.21.15.1:0/1464205913 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd568003520 con 0x7fd57c0c3260 2024-03-06T07:26:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd573fff640 1 -- 172.21.15.1:0/1464205913 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3679947012 0 0) 0x7fd568003520 con 0x7fd57c0c3260 2024-03-06T07:26:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd573fff640 1 -- 172.21.15.1:0/1464205913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd57c0c8cd0 msgr2=0x7fd57c0c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd573fff640 1 --2- 172.21.15.1:0/1464205913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd57c0c8cd0 0x7fd57c0c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd573fff640 1 -- 172.21.15.1:0/1464205913 >> v2:172.21.15.98:3300/0 conn(0x7fd57c155b70 msgr2=0x7fd57c159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd573fff640 1 --2- 172.21.15.1:0/1464205913 >> v2:172.21.15.98:3300/0 conn(0x7fd57c155b70 0x7fd57c159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd573fff640 1 -- 172.21.15.1:0/1464205913 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c0c3a90 con 0x7fd57c0c3260 2024-03-06T07:26:49.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd573fff640 1 -- 172.21.15.1:0/1464205913 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fd56c001670 con 0x7fd57c0c3260 2024-03-06T07:26:49.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd573fff640 1 -- 172.21.15.1:0/1464205913 >> v1:172.21.15.1:6789/0 conn(0x7fd57c0c3260 legacy=0x7fd57c0c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd573fff640 1 --2- 172.21.15.1:0/1464205913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd57c0c8cd0 0x7fd5680039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd573fff640 1 --2- 172.21.15.1:0/1464205913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd568003f30 0x7fd568004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd573fff640 1 --2- 172.21.15.1:0/1464205913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd568004d40 0x7fd5680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd573fff640 1 -- 172.21.15.1:0/1464205913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c156020 con 0x7fd568004d40 2024-03-06T07:26:49.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd573fff640 1 -- 172.21.15.1:0/1464205913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c0c37b0 con 0x7fd568003f30 2024-03-06T07:26:49.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.612+0000 7fd573fff640 1 -- 172.21.15.1:0/1464205913 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56800ba10 con 0x7fd57c0c8cd0 2024-03-06T07:26:49.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.613+0000 7fd581f09640 1 --2- 172.21.15.1:0/1464205913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd568004d40 0x7fd5680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.613+0000 7fd581708640 1 --2- 172.21.15.1:0/1464205913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd57c0c8cd0 0x7fd5680039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.613+0000 7fd580f07640 1 --2- 172.21.15.1:0/1464205913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd568003f30 0x7fd568004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.613+0000 7fd581f09640 1 -- 172.21.15.1:0/1464205913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd57c0c8cd0 msgr2=0x7fd5680039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.613+0000 7fd581f09640 1 --2- 172.21.15.1:0/1464205913 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd57c0c8cd0 0x7fd5680039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.613+0000 7fd581f09640 1 -- 172.21.15.1:0/1464205913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd568003f30 msgr2=0x7fd568004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.613+0000 7fd581f09640 1 --2- 172.21.15.1:0/1464205913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd568003f30 0x7fd568004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.613+0000 7fd581f09640 1 -- 172.21.15.1:0/1464205913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd568003520 con 0x7fd568004d40 2024-03-06T07:26:49.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.614+0000 7fd581f09640 1 --2- 172.21.15.1:0/1464205913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd568004d40 0x7fd5680051c0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fd57400e170 tx=0x7fd568005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee74904293a8548 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.614+0000 7fd573fff640 1 -- 172.21.15.1:0/1464205913 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd574039070 con 0x7fd568004d40 2024-03-06T07:26:49.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.614+0000 7fd573fff640 1 -- 172.21.15.1:0/1464205913 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57402cc20 con 0x7fd568004d40 2024-03-06T07:26:49.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.614+0000 7fd573fff640 1 -- 172.21.15.1:0/1464205913 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd574009c10 con 0x7fd568004d40 2024-03-06T07:26:49.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.614+0000 7fd58396c640 1 -- 172.21.15.1:0/1464205913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd568004d40 msgr2=0x7fd5680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.614+0000 7fd58396c640 1 --2- 172.21.15.1:0/1464205913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd568004d40 0x7fd5680051c0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fd57400e170 tx=0x7fd568005830 comp rx=0 tx=0).stop 2024-03-06T07:26:49.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.614+0000 7fd581f09640 1 -- 172.21.15.1:0/1464205913 reap_dead start 2024-03-06T07:26:49.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.614+0000 7fd58396c640 1 -- 172.21.15.1:0/1464205913 shutdown_connections 2024-03-06T07:26:49.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.614+0000 7fd58396c640 1 -- 172.21.15.1:0/1464205913 >> 172.21.15.1:0/1464205913 conn(0x7fd57c06b3e0 msgr2=0x7fd57c06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.614+0000 7fd58396c640 1 -- 172.21.15.1:0/1464205913 shutdown_connections 2024-03-06T07:26:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.614+0000 7fd58396c640 1 -- 172.21.15.1:0/1464205913 wait complete. 2024-03-06T07:26:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.615+0000 7fd58396c640 1 Processor -- start 2024-03-06T07:26:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.615+0000 7fd58396c640 1 -- start start 2024-03-06T07:26:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.616+0000 7fd58396c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd568004d40 0x7fd57c14ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.616+0000 7fd58396c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd57c14ffc0 0x7fd57c153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.616+0000 7fd58396c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd57c154490 0x7fd57c1edf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.616+0000 7fd58396c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c159550 con 0x7fd57c154490 2024-03-06T07:26:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.616+0000 7fd58396c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c158af0 con 0x7fd568004d40 2024-03-06T07:26:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.616+0000 7fd58396c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c0c8560 con 0x7fd57c14ffc0 2024-03-06T07:26:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.616+0000 7fd581f09640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd57c154490 0x7fd57c1edf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.616+0000 7fd581f09640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd57c154490 0x7fd57c1edf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39564/0 (socket says 172.21.15.1:39564) 2024-03-06T07:26:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.616+0000 7fd581f09640 1 -- 172.21.15.1:0/158405446 learned_addr learned my addr 172.21.15.1:0/158405446 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.616+0000 7fd580f07640 1 --2- 172.21.15.1:0/158405446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd57c14ffc0 0x7fd57c153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.616+0000 7fd581708640 1 --2- 172.21.15.1:0/158405446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd568004d40 0x7fd57c14ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.616+0000 7fd581f09640 1 -- 172.21.15.1:0/158405446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd57c14ffc0 msgr2=0x7fd57c153f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.616+0000 7fd581f09640 1 --2- 172.21.15.1:0/158405446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd57c14ffc0 0x7fd57c153f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.616+0000 7fd581f09640 1 -- 172.21.15.1:0/158405446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd568004d40 msgr2=0x7fd57c14ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.616+0000 7fd581f09640 1 --2- 172.21.15.1:0/158405446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd568004d40 0x7fd57c14ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.617+0000 7fd581f09640 1 -- 172.21.15.1:0/158405446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c1eecb0 con 0x7fd57c154490 2024-03-06T07:26:49.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.617+0000 7fd581708640 1 --2- 172.21.15.1:0/158405446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd568004d40 0x7fd57c14ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:49.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.617+0000 7fd581f09640 1 --2- 172.21.15.1:0/158405446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd57c154490 0x7fd57c1edf90 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fd574009f90 tx=0x7fd574009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.617+0000 7fd571ffb640 1 -- 172.21.15.1:0/158405446 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd574039030 con 0x7fd57c154490 2024-03-06T07:26:49.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.617+0000 7fd571ffb640 1 -- 172.21.15.1:0/158405446 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57403a040 con 0x7fd57c154490 2024-03-06T07:26:49.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.617+0000 7fd571ffb640 1 -- 172.21.15.1:0/158405446 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57403d5b0 con 0x7fd57c154490 2024-03-06T07:26:49.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.617+0000 7fd58396c640 1 -- 172.21.15.1:0/158405446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57c1df4e0 con 0x7fd57c154490 2024-03-06T07:26:49.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.617+0000 7fd58396c640 1 -- 172.21.15.1:0/158405446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57c1ee4d0 con 0x7fd57c154490 2024-03-06T07:26:49.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.618+0000 7fd58396c640 1 -- 172.21.15.1:0/158405446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd57c1df300 con 0x7fd57c154490 2024-03-06T07:26:49.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.619+0000 7fd571ffb640 1 -- 172.21.15.1:0/158405446 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd574045020 con 0x7fd57c154490 2024-03-06T07:26:49.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.619+0000 7fd571ffb640 1 --2- 172.21.15.1:0/158405446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd56006b500 0x7fd56006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.619+0000 7fd571ffb640 1 -- 172.21.15.1:0/158405446 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6449+0+0 (secure 0 0 0) 0x7fd5740ac5e0 con 0x7fd57c154490 2024-03-06T07:26:49.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.619+0000 7fd581708640 1 --2- 172.21.15.1:0/158405446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd56006b500 0x7fd56006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.620+0000 7fd581708640 1 --2- 172.21.15.1:0/158405446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd56006b500 0x7fd56006d9c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0a8010 tx=0x7fd578008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.623+0000 7fd571ffb640 1 -- 172.21.15.1:0/158405446 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd57c1df300 con 0x7fd57c154490 2024-03-06T07:26:49.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.810+0000 7fd58396c640 1 -- 172.21.15.1:0/158405446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd57c067fd0 con 0x7fd57c154490 2024-03-06T07:26:49.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.811+0000 7fd571ffb640 1 -- 172.21.15.1:0/158405446 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v159) v1 ==== 105+0+11 (secure 0 0 0) 0x7fd57407b200 con 0x7fd57c154490 2024-03-06T07:26:49.813 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:26:49.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.813+0000 7fd58396c640 1 -- 172.21.15.1:0/158405446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd56006b500 msgr2=0x7fd56006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.813+0000 7fd58396c640 1 --2- 172.21.15.1:0/158405446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd56006b500 0x7fd56006d9c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0a8010 tx=0x7fd578008040 comp rx=0 tx=0).stop 2024-03-06T07:26:49.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.813+0000 7fd58396c640 1 -- 172.21.15.1:0/158405446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd57c154490 msgr2=0x7fd57c1edf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.813+0000 7fd58396c640 1 --2- 172.21.15.1:0/158405446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd57c154490 0x7fd57c1edf90 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fd574009f90 tx=0x7fd574009750 comp rx=0 tx=0).stop 2024-03-06T07:26:49.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.813+0000 7fd58396c640 1 -- 172.21.15.1:0/158405446 shutdown_connections 2024-03-06T07:26:49.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.813+0000 7fd58396c640 1 --2- 172.21.15.1:0/158405446 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd56006b500 0x7fd56006d9c0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.813+0000 7fd58396c640 1 --2- 172.21.15.1:0/158405446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd57c154490 0x7fd57c1edf90 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.813+0000 7fd58396c640 1 --2- 172.21.15.1:0/158405446 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd57c14ffc0 0x7fd57c153f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.813+0000 7fd58396c640 1 --2- 172.21.15.1:0/158405446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd568004d40 0x7fd57c14ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.813+0000 7fd58396c640 1 -- 172.21.15.1:0/158405446 >> 172.21.15.1:0/158405446 conn(0x7fd57c06b3e0 msgr2=0x7fd57c0c7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.813+0000 7fd58396c640 1 -- 172.21.15.1:0/158405446 shutdown_connections 2024-03-06T07:26:49.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.813+0000 7fd58396c640 1 -- 172.21.15.1:0/158405446 wait complete. 2024-03-06T07:26:49.826 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-06T07:26:49.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.946+0000 7fa66ebd0640 1 Processor -- start 2024-03-06T07:26:49.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.946+0000 7fa66ebd0640 1 -- start start 2024-03-06T07:26:49.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.947+0000 7fa66ebd0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa6680c5850 0x7fa6680c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.947+0000 7fa66ebd0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa668155220 0x7fa6681556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.947+0000 7fa66ebd0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa668158910 con 0x7fa668152920 2024-03-06T07:26:49.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.947+0000 7fa66ebd0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa668152d00 con 0x7fa6680c5850 2024-03-06T07:26:49.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.947+0000 7fa66ebd0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6681596d0 con 0x7fa668155220 2024-03-06T07:26:49.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.947+0000 7fa667fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa6680c5850 0x7fa6680c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.947+0000 7fa66c96c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa668152920 0x7fa66806aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33568/0 (socket says 172.21.15.1:33568) 2024-03-06T07:26:49.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.947+0000 7fa667fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa6680c5850 0x7fa6680c5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39576/0 (socket says 172.21.15.1:39576) 2024-03-06T07:26:49.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.947+0000 7fa66c96c640 1 -- 172.21.15.1:0/337772069 learned_addr learned my addr 172.21.15.1:0/337772069 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:49.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.947+0000 7fa667fff640 1 -- 172.21.15.1:0/337772069 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa668155220 msgr2=0x7fa6681556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.947+0000 7fa66d16d640 1 --2- 172.21.15.1:0/337772069 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa668155220 0x7fa6681556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.948+0000 7fa667fff640 1 --2- 172.21.15.1:0/337772069 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa668155220 0x7fa6681556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.948+0000 7fa667fff640 1 -- 172.21.15.1:0/337772069 >> v1:172.21.15.1:6789/0 conn(0x7fa668152920 legacy=0x7fa66806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.948+0000 7fa6677fe640 1 -- 172.21.15.1:0/337772069 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809309802 0 0) 0x7fa668158910 con 0x7fa668152920 2024-03-06T07:26:49.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.948+0000 7fa667fff640 1 -- 172.21.15.1:0/337772069 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa668158bf0 con 0x7fa6680c5850 2024-03-06T07:26:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.948+0000 7fa66d16d640 1 --2- 172.21.15.1:0/337772069 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa668155220 0x7fa6681556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.948+0000 7fa667fff640 1 --2- 172.21.15.1:0/337772069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6680c5850 0x7fa6680c5cb0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fa6580097a0 tx=0x7fa65802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=854936767e0ab604 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.948+0000 7fa6677fe640 1 -- 172.21.15.1:0/337772069 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa658031780 con 0x7fa6680c5850 2024-03-06T07:26:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.948+0000 7fa6677fe640 1 -- 172.21.15.1:0/337772069 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa658037070 con 0x7fa6680c5850 2024-03-06T07:26:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.948+0000 7fa6677fe640 1 -- 172.21.15.1:0/337772069 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa658031b40 con 0x7fa6680c5850 2024-03-06T07:26:49.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.949+0000 7fa66ebd0640 1 -- 172.21.15.1:0/337772069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6680c5850 msgr2=0x7fa6680c5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.949+0000 7fa66ebd0640 1 --2- 172.21.15.1:0/337772069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6680c5850 0x7fa6680c5cb0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fa6580097a0 tx=0x7fa65802b820 comp rx=0 tx=0).stop 2024-03-06T07:26:49.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.949+0000 7fa66ebd0640 1 -- 172.21.15.1:0/337772069 shutdown_connections 2024-03-06T07:26:49.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.949+0000 7fa66ebd0640 1 --2- 172.21.15.1:0/337772069 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa668155220 0x7fa6681556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.949+0000 7fa66ebd0640 1 --2- 172.21.15.1:0/337772069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6680c5850 0x7fa6680c5cb0 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.949+0000 7fa66ebd0640 1 -- 172.21.15.1:0/337772069 >> 172.21.15.1:0/337772069 conn(0x7fa6680a9700 msgr2=0x7fa6680a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.949+0000 7fa66ebd0640 1 -- 172.21.15.1:0/337772069 shutdown_connections 2024-03-06T07:26:49.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.949+0000 7fa66ebd0640 1 -- 172.21.15.1:0/337772069 wait complete. 2024-03-06T07:26:49.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.950+0000 7fa66ebd0640 1 Processor -- start 2024-03-06T07:26:49.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.950+0000 7fa66ebd0640 1 -- start start 2024-03-06T07:26:49.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.951+0000 7fa66ebd0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6680c5850 0x7fa668151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.951+0000 7fa66ebd0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa668152920 0x7fa6680a8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.951+0000 7fa66ebd0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa668155220 0x7fa6681eb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.951+0000 7fa66ebd0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6680c6380 con 0x7fa668155220 2024-03-06T07:26:49.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.951+0000 7fa66ebd0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa668159240 con 0x7fa668152920 2024-03-06T07:26:49.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.951+0000 7fa66ebd0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66806b700 con 0x7fa6680c5850 2024-03-06T07:26:49.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.951+0000 7fa66d16d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa668155220 0x7fa6681eb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.951+0000 7fa66d16d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa668155220 0x7fa6681eb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39586/0 (socket says 172.21.15.1:39586) 2024-03-06T07:26:49.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.951+0000 7fa66d16d640 1 -- 172.21.15.1:0/434562241 learned_addr learned my addr 172.21.15.1:0/434562241 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:49.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.951+0000 7fa66c96c640 1 --2- 172.21.15.1:0/434562241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6680c5850 0x7fa668151b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.951+0000 7fa667fff640 1 --2- 172.21.15.1:0/434562241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa668152920 0x7fa6680a8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.951+0000 7fa66d16d640 1 -- 172.21.15.1:0/434562241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6680c5850 msgr2=0x7fa668151b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.951+0000 7fa66d16d640 1 --2- 172.21.15.1:0/434562241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6680c5850 0x7fa668151b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.951+0000 7fa66d16d640 1 -- 172.21.15.1:0/434562241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa668152920 msgr2=0x7fa6680a8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.951+0000 7fa66d16d640 1 --2- 172.21.15.1:0/434562241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa668152920 0x7fa6680a8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.951+0000 7fa66d16d640 1 -- 172.21.15.1:0/434562241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6681502c0 con 0x7fa668155220 2024-03-06T07:26:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.952+0000 7fa66d16d640 1 --2- 172.21.15.1:0/434562241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa668155220 0x7fa6681eb9e0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fa66000cd80 tx=0x7fa66000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.952+0000 7fa6657fa640 1 -- 172.21.15.1:0/434562241 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66000f680 con 0x7fa668155220 2024-03-06T07:26:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.952+0000 7fa6657fa640 1 -- 172.21.15.1:0/434562241 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa66000f840 con 0x7fa668155220 2024-03-06T07:26:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.952+0000 7fa66ebd0640 1 -- 172.21.15.1:0/434562241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6681505a0 con 0x7fa668155220 2024-03-06T07:26:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.952+0000 7fa6657fa640 1 -- 172.21.15.1:0/434562241 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66000fc30 con 0x7fa668155220 2024-03-06T07:26:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.952+0000 7fa66ebd0640 1 -- 172.21.15.1:0/434562241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa668150b60 con 0x7fa668155220 2024-03-06T07:26:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.953+0000 7fa66ebd0640 1 -- 172.21.15.1:0/434562241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa668067fd0 con 0x7fa668155220 2024-03-06T07:26:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.953+0000 7fa6657fa640 1 -- 172.21.15.1:0/434562241 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6600213f0 con 0x7fa668155220 2024-03-06T07:26:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.954+0000 7fa6657fa640 1 --2- 172.21.15.1:0/434562241 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa64806b4d0 0x7fa64806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.954+0000 7fa6657fa640 1 -- 172.21.15.1:0/434562241 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6449+0+0 (secure 0 0 0) 0x7fa66008b180 con 0x7fa668155220 2024-03-06T07:26:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.954+0000 7fa66c96c640 1 --2- 172.21.15.1:0/434562241 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa64806b4d0 0x7fa64806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.955+0000 7fa66c96c640 1 --2- 172.21.15.1:0/434562241 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa64806b4d0 0x7fa64806d990 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fa650000f60 tx=0x7fa650007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:49.958+0000 7fa6657fa640 1 -- 172.21.15.1:0/434562241 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa660014030 con 0x7fa668155220 2024-03-06T07:26:50.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.137+0000 7fa66ebd0640 1 -- 172.21.15.1:0/434562241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7fa66806bdf0 con 0x7fa668155220 2024-03-06T07:26:50.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.138+0000 7fa6657fa640 1 -- 172.21.15.1:0/434562241 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v159) v1 ==== 101+0+11 (secure 0 0 0) 0x7fa6600596c0 con 0x7fa668155220 2024-03-06T07:26:50.140 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:26:50.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.140+0000 7fa66ebd0640 1 -- 172.21.15.1:0/434562241 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa64806b4d0 msgr2=0x7fa64806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.140+0000 7fa66ebd0640 1 --2- 172.21.15.1:0/434562241 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa64806b4d0 0x7fa64806d990 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fa650000f60 tx=0x7fa650007d50 comp rx=0 tx=0).stop 2024-03-06T07:26:50.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.140+0000 7fa66ebd0640 1 -- 172.21.15.1:0/434562241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa668155220 msgr2=0x7fa6681eb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.140+0000 7fa66ebd0640 1 --2- 172.21.15.1:0/434562241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa668155220 0x7fa6681eb9e0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fa66000cd80 tx=0x7fa66000a650 comp rx=0 tx=0).stop 2024-03-06T07:26:50.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.140+0000 7fa66ebd0640 1 -- 172.21.15.1:0/434562241 shutdown_connections 2024-03-06T07:26:50.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.140+0000 7fa66ebd0640 1 --2- 172.21.15.1:0/434562241 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa64806b4d0 0x7fa64806d990 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.140+0000 7fa66ebd0640 1 --2- 172.21.15.1:0/434562241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa668155220 0x7fa6681eb9e0 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.140+0000 7fa66ebd0640 1 --2- 172.21.15.1:0/434562241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa668152920 0x7fa6680a8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.140+0000 7fa66ebd0640 1 --2- 172.21.15.1:0/434562241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa6680c5850 0x7fa668151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.140+0000 7fa66ebd0640 1 -- 172.21.15.1:0/434562241 >> 172.21.15.1:0/434562241 conn(0x7fa6680a9700 msgr2=0x7fa66814c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.140+0000 7fa66ebd0640 1 -- 172.21.15.1:0/434562241 shutdown_connections 2024-03-06T07:26:50.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.140+0000 7fa66ebd0640 1 -- 172.21.15.1:0/434562241 wait complete. 2024-03-06T07:26:50.153 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:50.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.252+0000 7fed1ee95640 1 Processor -- start 2024-03-06T07:26:50.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.252+0000 7fed1ee95640 1 -- start start 2024-03-06T07:26:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.253+0000 7fed1ee95640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed180c60b0 0x7fed180c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.253+0000 7fed1ee95640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed180c5e40 con 0x7fed18152840 2024-03-06T07:26:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.253+0000 7fed1ee95640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed180c6a40 con 0x7fed180c3970 2024-03-06T07:26:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.253+0000 7fed1ee95640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed180c67f0 con 0x7fed180c60b0 2024-03-06T07:26:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.253+0000 7fed17fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fed18152840 0x7fed181553d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33574/0 (socket says 172.21.15.1:33574) 2024-03-06T07:26:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.253+0000 7fed17fff640 1 -- 172.21.15.1:0/3427821746 learned_addr learned my addr 172.21.15.1:0/3427821746 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.253+0000 7fed1d432640 1 --2- 172.21.15.1:0/3427821746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed180c60b0 0x7fed180c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.253+0000 7fed177fe640 1 -- 172.21.15.1:0/3427821746 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252223049 0 0) 0x7fed180c5e40 con 0x7fed18152840 2024-03-06T07:26:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.253+0000 7fed177fe640 1 -- 172.21.15.1:0/3427821746 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed08003520 con 0x7fed18152840 2024-03-06T07:26:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed177fe640 1 -- 172.21.15.1:0/3427821746 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1653381558 0 0) 0x7fed08003520 con 0x7fed18152840 2024-03-06T07:26:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed177fe640 1 -- 172.21.15.1:0/3427821746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed180c60b0 msgr2=0x7fed180c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed177fe640 1 --2- 172.21.15.1:0/3427821746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed180c60b0 0x7fed180c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed177fe640 1 -- 172.21.15.1:0/3427821746 >> v1:172.21.15.98:6789/0 conn(0x7fed180c3970 legacy=0x7fed18152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed177fe640 1 -- 172.21.15.1:0/3427821746 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed180c6d20 con 0x7fed18152840 2024-03-06T07:26:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed177fe640 1 -- 172.21.15.1:0/3427821746 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fed04001670 con 0x7fed18152840 2024-03-06T07:26:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed177fe640 1 -- 172.21.15.1:0/3427821746 >> v1:172.21.15.1:6789/0 conn(0x7fed18152840 legacy=0x7fed181553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed177fe640 1 --2- 172.21.15.1:0/3427821746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed180c60b0 0x7fed080039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed177fe640 1 --2- 172.21.15.1:0/3427821746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08003f60 0x7fed08004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed177fe640 1 --2- 172.21.15.1:0/3427821746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed08004d40 0x7fed080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed177fe640 1 -- 172.21.15.1:0/3427821746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed180c67f0 con 0x7fed08003f60 2024-03-06T07:26:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed177fe640 1 -- 172.21.15.1:0/3427821746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0800baa0 con 0x7fed08004d40 2024-03-06T07:26:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed177fe640 1 -- 172.21.15.1:0/3427821746 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0800bca0 con 0x7fed180c60b0 2024-03-06T07:26:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed17fff640 1 --2- 172.21.15.1:0/3427821746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08003f60 0x7fed08004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed1cc31640 1 --2- 172.21.15.1:0/3427821746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed180c60b0 0x7fed080039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed1d432640 1 --2- 172.21.15.1:0/3427821746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed08004d40 0x7fed080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed17fff640 1 -- 172.21.15.1:0/3427821746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed180c60b0 msgr2=0x7fed080039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed17fff640 1 --2- 172.21.15.1:0/3427821746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed180c60b0 0x7fed080039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed17fff640 1 -- 172.21.15.1:0/3427821746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed08004d40 msgr2=0x7fed080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed17fff640 1 --2- 172.21.15.1:0/3427821746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed08004d40 0x7fed080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.254+0000 7fed17fff640 1 -- 172.21.15.1:0/3427821746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed08003520 con 0x7fed08003f60 2024-03-06T07:26:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.255+0000 7fed17fff640 1 --2- 172.21.15.1:0/3427821746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08003f60 0x7fed08004800 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fed0400c210 tx=0x7fed08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d2dd774449848c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.255+0000 7fed177fe640 1 -- 172.21.15.1:0/3427821746 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed04007cf0 con 0x7fed08003f60 2024-03-06T07:26:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.255+0000 7fed177fe640 1 -- 172.21.15.1:0/3427821746 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed04030070 con 0x7fed08003f60 2024-03-06T07:26:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.255+0000 7fed177fe640 1 -- 172.21.15.1:0/3427821746 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed04005d30 con 0x7fed08003f60 2024-03-06T07:26:50.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.256+0000 7fed1ee95640 1 -- 172.21.15.1:0/3427821746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08003f60 msgr2=0x7fed08004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.256+0000 7fed1ee95640 1 --2- 172.21.15.1:0/3427821746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08003f60 0x7fed08004800 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fed0400c210 tx=0x7fed08005830 comp rx=0 tx=0).stop 2024-03-06T07:26:50.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.256+0000 7fed1d432640 1 -- 172.21.15.1:0/3427821746 reap_dead start 2024-03-06T07:26:50.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.256+0000 7fed1ee95640 1 -- 172.21.15.1:0/3427821746 shutdown_connections 2024-03-06T07:26:50.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.256+0000 7fed1ee95640 1 -- 172.21.15.1:0/3427821746 >> 172.21.15.1:0/3427821746 conn(0x7fed180c88f0 msgr2=0x7fed180c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.256+0000 7fed1ee95640 1 -- 172.21.15.1:0/3427821746 shutdown_connections 2024-03-06T07:26:50.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.256+0000 7fed1ee95640 1 -- 172.21.15.1:0/3427821746 wait complete. 2024-03-06T07:26:50.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.257+0000 7fed1ee95640 1 Processor -- start 2024-03-06T07:26:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.257+0000 7fed1ee95640 1 -- start start 2024-03-06T07:26:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.258+0000 7fed1ee95640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08003f60 0x7fed18157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.258+0000 7fed1ee95640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed180aaaf0 0x7fed180aaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.258+0000 7fed1ee95640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed1814a700 0x7fed1814ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.258+0000 7fed1ee95640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed180c75c0 con 0x7fed08003f60 2024-03-06T07:26:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.258+0000 7fed1ee95640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed180c72e0 con 0x7fed180aaaf0 2024-03-06T07:26:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.258+0000 7fed1ee95640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed180c3d50 con 0x7fed1814a700 2024-03-06T07:26:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.258+0000 7fed1cc31640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08003f60 0x7fed18157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.258+0000 7fed1cc31640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08003f60 0x7fed18157770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39612/0 (socket says 172.21.15.1:39612) 2024-03-06T07:26:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.258+0000 7fed1cc31640 1 -- 172.21.15.1:0/1029485448 learned_addr learned my addr 172.21.15.1:0/1029485448 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.258+0000 7fed1d432640 1 --2- 172.21.15.1:0/1029485448 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed1814a700 0x7fed1814ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.258+0000 7fed17fff640 1 --2- 172.21.15.1:0/1029485448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed180aaaf0 0x7fed180aaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.258+0000 7fed1cc31640 1 -- 172.21.15.1:0/1029485448 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed1814a700 msgr2=0x7fed1814ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.258+0000 7fed1cc31640 1 --2- 172.21.15.1:0/1029485448 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed1814a700 0x7fed1814ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.258+0000 7fed1cc31640 1 -- 172.21.15.1:0/1029485448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed180aaaf0 msgr2=0x7fed180aaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.259+0000 7fed1cc31640 1 --2- 172.21.15.1:0/1029485448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed180aaaf0 0x7fed180aaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.259+0000 7fed1cc31640 1 -- 172.21.15.1:0/1029485448 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1814c620 con 0x7fed08003f60 2024-03-06T07:26:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.259+0000 7fed1d432640 1 --2- 172.21.15.1:0/1029485448 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed1814a700 0x7fed1814ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.259+0000 7fed1cc31640 1 --2- 172.21.15.1:0/1029485448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08003f60 0x7fed18157770 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00f820 tx=0x7fed0c00fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.259+0000 7fed157fa640 1 -- 172.21.15.1:0/1029485448 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c0050d0 con 0x7fed08003f60 2024-03-06T07:26:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.259+0000 7fed157fa640 1 -- 172.21.15.1:0/1029485448 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0c018070 con 0x7fed08003f60 2024-03-06T07:26:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.259+0000 7fed1ee95640 1 -- 172.21.15.1:0/1029485448 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1814c900 con 0x7fed08003f60 2024-03-06T07:26:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.259+0000 7fed157fa640 1 -- 172.21.15.1:0/1029485448 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c005490 con 0x7fed08003f60 2024-03-06T07:26:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.259+0000 7fed1ee95640 1 -- 172.21.15.1:0/1029485448 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1814cec0 con 0x7fed08003f60 2024-03-06T07:26:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.260+0000 7fed1ee95640 1 -- 172.21.15.1:0/1029485448 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed180686e0 con 0x7fed08003f60 2024-03-06T07:26:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.261+0000 7fed157fa640 1 -- 172.21.15.1:0/1029485448 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed0c00d6d0 con 0x7fed08003f60 2024-03-06T07:26:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.261+0000 7fed157fa640 1 --2- 172.21.15.1:0/1029485448 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fecfc06b500 0x7fecfc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.261+0000 7fed157fa640 1 -- 172.21.15.1:0/1029485448 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6449+0+0 (secure 0 0 0) 0x7fed0c08a2c0 con 0x7fed08003f60 2024-03-06T07:26:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.261+0000 7fed17fff640 1 --2- 172.21.15.1:0/1029485448 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fecfc06b500 0x7fecfc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.262+0000 7fed17fff640 1 --2- 172.21.15.1:0/1029485448 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fecfc06b500 0x7fecfc06d9c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fed04034000 tx=0x7fed04007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.265+0000 7fed157fa640 1 -- 172.21.15.1:0/1029485448 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed0c058ee0 con 0x7fed08003f60 2024-03-06T07:26:50.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.483+0000 7fed1ee95640 1 -- 172.21.15.1:0/1029485448 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed180ad920 con 0x7fed08003f60 2024-03-06T07:26:50.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.483+0000 7fed157fa640 1 -- 172.21.15.1:0/1029485448 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+4984 (secure 0 0 0) 0x7fed0c08a0e0 con 0x7fed08003f60 2024-03-06T07:26:50.485 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:50.486 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":151,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1387389703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1387389703},{"type":"v1","addr":"172.21.15.98:6836","nonce":1387389703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":8137,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2268963727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2268963727},{"type":"v1","addr":"172.21.15.98:6837","nonce":2268963727}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":9162,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/4198928111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4198928111},{"type":"v1","addr":"172.21.15.1:6833","nonce":4198928111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:26.024256+0000","modified":"2024-03-06T07:26:49.862056+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9168},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9168":{"gid":9168,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6835/2664051788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2664051788},{"type":"v1","addr":"172.21.15.1:6835","nonce":2664051788}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9168,"qdb_cluster":[9168]},"id":12}]} 2024-03-06T07:26:50.486 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 151 2024-03-06T07:26:50.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.485+0000 7fed1ee95640 1 -- 172.21.15.1:0/1029485448 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fecfc06b500 msgr2=0x7fecfc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.485+0000 7fed1ee95640 1 --2- 172.21.15.1:0/1029485448 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fecfc06b500 0x7fecfc06d9c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fed04034000 tx=0x7fed04007000 comp rx=0 tx=0).stop 2024-03-06T07:26:50.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.485+0000 7fed1ee95640 1 -- 172.21.15.1:0/1029485448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08003f60 msgr2=0x7fed18157770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.485+0000 7fed1ee95640 1 --2- 172.21.15.1:0/1029485448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08003f60 0x7fed18157770 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00f820 tx=0x7fed0c00fce0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.485+0000 7fed1ee95640 1 -- 172.21.15.1:0/1029485448 shutdown_connections 2024-03-06T07:26:50.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.485+0000 7fed1ee95640 1 --2- 172.21.15.1:0/1029485448 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fecfc06b500 0x7fecfc06d9c0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.485+0000 7fed1ee95640 1 --2- 172.21.15.1:0/1029485448 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed1814a700 0x7fed1814ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.485+0000 7fed1ee95640 1 --2- 172.21.15.1:0/1029485448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed180aaaf0 0x7fed180aaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.485+0000 7fed1ee95640 1 --2- 172.21.15.1:0/1029485448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08003f60 0x7fed18157770 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.485+0000 7fed1ee95640 1 -- 172.21.15.1:0/1029485448 >> 172.21.15.1:0/1029485448 conn(0x7fed180c88f0 msgr2=0x7fed18151f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.486+0000 7fed1ee95640 1 -- 172.21.15.1:0/1029485448 shutdown_connections 2024-03-06T07:26:50.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.486+0000 7fed1ee95640 1 -- 172.21.15.1:0/1029485448 wait complete. 2024-03-06T07:26:50.498 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:50.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.598+0000 7f614e943640 1 Processor -- start 2024-03-06T07:26:50.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.598+0000 7f614e943640 1 -- start start 2024-03-06T07:26:50.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.599+0000 7f614e943640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61480c6b10 0x7f6148155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.599+0000 7f614e943640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61480c2fc0 con 0x7f61480bf9d0 2024-03-06T07:26:50.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.599+0000 7f614e943640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6148158da0 con 0x7f6148152650 2024-03-06T07:26:50.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.599+0000 7f614e943640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6148155800 con 0x7f61480c6b10 2024-03-06T07:26:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.599+0000 7f614cee0640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f61480bf9d0 0x7f61480c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33586/0 (socket says 172.21.15.1:33586) 2024-03-06T07:26:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.599+0000 7f614cee0640 1 -- 172.21.15.1:0/324477889 learned_addr learned my addr 172.21.15.1:0/324477889 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.599+0000 7f61477fe640 1 --2- 172.21.15.1:0/324477889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61480c6b10 0x7f6148155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.599+0000 7f6146ffd640 1 -- 172.21.15.1:0/324477889 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221451854 0 0) 0x7f61480c2fc0 con 0x7f61480bf9d0 2024-03-06T07:26:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.599+0000 7f6146ffd640 1 -- 172.21.15.1:0/324477889 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6130003520 con 0x7f61480bf9d0 2024-03-06T07:26:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.599+0000 7f6146ffd640 1 -- 172.21.15.1:0/324477889 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212984742 0 0) 0x7f6148158da0 con 0x7f6148152650 2024-03-06T07:26:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.599+0000 7f6146ffd640 1 -- 172.21.15.1:0/324477889 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61480c2fc0 con 0x7f6148152650 2024-03-06T07:26:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.599+0000 7f6146ffd640 1 -- 172.21.15.1:0/324477889 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1175322347 0 0) 0x7f6130003520 con 0x7f61480bf9d0 2024-03-06T07:26:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.599+0000 7f61477fe640 1 -- 172.21.15.1:0/324477889 >> v1:172.21.15.98:6789/0 conn(0x7f6148152650 legacy=0x7f6148158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.599+0000 7f61477fe640 1 -- 172.21.15.1:0/324477889 >> v1:172.21.15.1:6789/0 conn(0x7f61480bf9d0 legacy=0x7f61480c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.599+0000 7f61477fe640 1 -- 172.21.15.1:0/324477889 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61480c6690 con 0x7f61480c6b10 2024-03-06T07:26:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.600+0000 7f61477fe640 1 --2- 172.21.15.1:0/324477889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61480c6b10 0x7f6148155c90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f613c0097d0 tx=0x7f613c02b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b00809e5ff66503d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.600+0000 7f6146ffd640 1 -- 172.21.15.1:0/324477889 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c037070 con 0x7f61480c6b10 2024-03-06T07:26:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.600+0000 7f6146ffd640 1 -- 172.21.15.1:0/324477889 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613c030510 con 0x7f61480c6b10 2024-03-06T07:26:50.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.600+0000 7f6146ffd640 1 -- 172.21.15.1:0/324477889 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61400019f0 con 0x7f61480c6b10 2024-03-06T07:26:50.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.601+0000 7f614e943640 1 -- 172.21.15.1:0/324477889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61480c6b10 msgr2=0x7f6148155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.601+0000 7f614e943640 1 --2- 172.21.15.1:0/324477889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61480c6b10 0x7f6148155c90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f613c0097d0 tx=0x7f613c02b770 comp rx=0 tx=0).stop 2024-03-06T07:26:50.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.601+0000 7f614e943640 1 -- 172.21.15.1:0/324477889 shutdown_connections 2024-03-06T07:26:50.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.601+0000 7f614e943640 1 --2- 172.21.15.1:0/324477889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61480c6b10 0x7f6148155c90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.601+0000 7f614e943640 1 -- 172.21.15.1:0/324477889 >> 172.21.15.1:0/324477889 conn(0x7f6148066ee0 msgr2=0x7f61480672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.601+0000 7f614e943640 1 -- 172.21.15.1:0/324477889 shutdown_connections 2024-03-06T07:26:50.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.601+0000 7f614e943640 1 -- 172.21.15.1:0/324477889 wait complete. 2024-03-06T07:26:50.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.602+0000 7f614e943640 1 Processor -- start 2024-03-06T07:26:50.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.602+0000 7f614e943640 1 -- start start 2024-03-06T07:26:50.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f614e943640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61480bf9d0 0x7f61481eb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f614e943640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6148152650 0x7f6148157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f614e943640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6148152cd0 0x7f6148150340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f614e943640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61481551b0 con 0x7f6148152650 2024-03-06T07:26:50.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f614e943640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61480c6fa0 con 0x7f6148152cd0 2024-03-06T07:26:50.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f614e943640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6148155490 con 0x7f61480bf9d0 2024-03-06T07:26:50.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f61477fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6148152650 0x7f6148157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f61477fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6148152650 0x7f6148157750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39622/0 (socket says 172.21.15.1:39622) 2024-03-06T07:26:50.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f61477fe640 1 -- 172.21.15.1:0/3331500239 learned_addr learned my addr 172.21.15.1:0/3331500239 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:50.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f614cee0640 1 --2- 172.21.15.1:0/3331500239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6148152cd0 0x7f6148150340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f6147fff640 1 --2- 172.21.15.1:0/3331500239 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61480bf9d0 0x7f61481eb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f61477fe640 1 -- 172.21.15.1:0/3331500239 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61480bf9d0 msgr2=0x7f61481eb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f61477fe640 1 --2- 172.21.15.1:0/3331500239 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61480bf9d0 0x7f61481eb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f61477fe640 1 -- 172.21.15.1:0/3331500239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6148152cd0 msgr2=0x7f6148150340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f61477fe640 1 --2- 172.21.15.1:0/3331500239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6148152cd0 0x7f6148150340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f61477fe640 1 -- 172.21.15.1:0/3331500239 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614814fba0 con 0x7f6148152650 2024-03-06T07:26:50.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f614cee0640 1 --2- 172.21.15.1:0/3331500239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6148152cd0 0x7f6148150340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:50.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f6147fff640 1 --2- 172.21.15.1:0/3331500239 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61480bf9d0 0x7f61481eb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:50.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.603+0000 7f61477fe640 1 --2- 172.21.15.1:0/3331500239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6148152650 0x7f6148157750 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f613c0097d0 tx=0x7f613c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.604+0000 7f6144ff9640 1 -- 172.21.15.1:0/3331500239 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c037040 con 0x7f6148152650 2024-03-06T07:26:50.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.604+0000 7f6144ff9640 1 -- 172.21.15.1:0/3331500239 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613c004250 con 0x7f6148152650 2024-03-06T07:26:50.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.604+0000 7f614e943640 1 -- 172.21.15.1:0/3331500239 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6148150eb0 con 0x7f6148152650 2024-03-06T07:26:50.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.604+0000 7f6144ff9640 1 -- 172.21.15.1:0/3331500239 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c006e10 con 0x7f6148152650 2024-03-06T07:26:50.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.604+0000 7f614e943640 1 -- 172.21.15.1:0/3331500239 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6148151470 con 0x7f6148152650 2024-03-06T07:26:50.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.605+0000 7f614e943640 1 -- 172.21.15.1:0/3331500239 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6148067fd0 con 0x7f6148152650 2024-03-06T07:26:50.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.606+0000 7f6144ff9640 1 -- 172.21.15.1:0/3331500239 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f613c03a070 con 0x7f6148152650 2024-03-06T07:26:50.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.606+0000 7f6144ff9640 1 --2- 172.21.15.1:0/3331500239 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f613406b4d0 0x7f613406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.606+0000 7f6144ff9640 1 -- 172.21.15.1:0/3331500239 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f613c0abd10 con 0x7f6148152650 2024-03-06T07:26:50.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.606+0000 7f6147fff640 1 --2- 172.21.15.1:0/3331500239 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f613406b4d0 0x7f613406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.607+0000 7f6147fff640 1 --2- 172.21.15.1:0/3331500239 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f613406b4d0 0x7f613406d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f61380011e0 tx=0x7f6138009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.611+0000 7f6144ff9640 1 -- 172.21.15.1:0/3331500239 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f613c041020 con 0x7f6148152650 2024-03-06T07:26:50.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.820+0000 7f614e943640 1 -- 172.21.15.1:0/3331500239 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61480b06f0 con 0x7f6148152650 2024-03-06T07:26:50.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.821+0000 7f6144ff9640 1 -- 172.21.15.1:0/3331500239 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v159) v1 ==== 74+0+17283 (secure 0 0 0) 0x7f613c07a930 con 0x7f6148152650 2024-03-06T07:26:50.823 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:50.824 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":159,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:26:48.853480+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:26:24.304587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:26:25.085741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6837/2360179300":"2024-03-07T07:26:46.842519+0000","172.21.15.98:6835/2360179300":"2024-03-07T07:26:46.842519+0000","172.21.15.1:6833/1089548586":"2024-03-07T07:26:36.687217+0000","172.21.15.1:6832/1089548586":"2024-03-07T07:26:36.687217+0000","172.21.15.98:6836/3360241462":"2024-03-07T07:26:45.837702+0000","172.21.15.98:6834/3360241462":"2024-03-07T07:26:45.837702+0000","172.21.15.1:6835/67091521":"2024-03-07T07:26:36.682683+0000","172.21.15.1:6834/67091521":"2024-03-07T07:26:36.682683+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:50.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.823+0000 7f614e943640 1 -- 172.21.15.1:0/3331500239 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f613406b4d0 msgr2=0x7f613406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.823+0000 7f614e943640 1 --2- 172.21.15.1:0/3331500239 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f613406b4d0 0x7f613406d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f61380011e0 tx=0x7f6138009040 comp rx=0 tx=0).stop 2024-03-06T07:26:50.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.823+0000 7f614e943640 1 -- 172.21.15.1:0/3331500239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6148152650 msgr2=0x7f6148157750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.823+0000 7f614e943640 1 --2- 172.21.15.1:0/3331500239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6148152650 0x7f6148157750 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f613c0097d0 tx=0x7f613c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.823+0000 7f614e943640 1 -- 172.21.15.1:0/3331500239 shutdown_connections 2024-03-06T07:26:50.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.823+0000 7f614e943640 1 --2- 172.21.15.1:0/3331500239 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f613406b4d0 0x7f613406d990 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.823+0000 7f614e943640 1 --2- 172.21.15.1:0/3331500239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6148152cd0 0x7f6148150340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.823+0000 7f614e943640 1 --2- 172.21.15.1:0/3331500239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6148152650 0x7f6148157750 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.823+0000 7f614e943640 1 --2- 172.21.15.1:0/3331500239 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61480bf9d0 0x7f61481eb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.823+0000 7f614e943640 1 -- 172.21.15.1:0/3331500239 >> 172.21.15.1:0/3331500239 conn(0x7f6148066ee0 msgr2=0x7f61480c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.823+0000 7f614e943640 1 -- 172.21.15.1:0/3331500239 shutdown_connections 2024-03-06T07:26:50.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.823+0000 7f614e943640 1 -- 172.21.15.1:0/3331500239 wait complete. 2024-03-06T07:26:50.840 INFO:tasks.cephfs.filesystem:Destroying file system cephfs2 and related pools 2024-03-06T07:26:50.840 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:50.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.944+0000 7f4f064eb640 1 Processor -- start 2024-03-06T07:26:50.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.944+0000 7f4f064eb640 1 -- start start 2024-03-06T07:26:50.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.945+0000 7f4f064eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f000c5850 0x7f4f000c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.945+0000 7f4f064eb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4f00155220 0x7f4f001556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.945+0000 7f4f064eb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f00158910 con 0x7f4f00152920 2024-03-06T07:26:50.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.945+0000 7f4f064eb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f00152d00 con 0x7f4f00155220 2024-03-06T07:26:50.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.945+0000 7f4f064eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f001596d0 con 0x7f4f000c5850 2024-03-06T07:26:50.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.945+0000 7f4eff7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f000c5850 0x7f4f000c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.945+0000 7f4f04a88640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4f00155220 0x7f4f001556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.945+0000 7f4eff7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f000c5850 0x7f4f000c5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:57690/0 (socket says 172.21.15.1:57690) 2024-03-06T07:26:50.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.945+0000 7f4f04a88640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4f00155220 0x7f4f001556b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:41806/0 (socket says 172.21.15.1:41806) 2024-03-06T07:26:50.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.945+0000 7f4eff7fe640 1 -- 172.21.15.1:0/2416768525 learned_addr learned my addr 172.21.15.1:0/2416768525 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:50.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.945+0000 7f4efffff640 1 --1- 172.21.15.1:0/2416768525 >> v1:172.21.15.98:6789/0 conn(0x7f4f00152920 0x7f4f0006aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:53592/0 (socket says 172.21.15.1:53592) 2024-03-06T07:26:50.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.946+0000 7f4eff7fe640 1 -- 172.21.15.1:0/2416768525 >> v2:172.21.15.98:3300/0 conn(0x7f4f00155220 msgr2=0x7f4f001556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.946+0000 7f4eff7fe640 1 --2- 172.21.15.1:0/2416768525 >> v2:172.21.15.98:3300/0 conn(0x7f4f00155220 0x7f4f001556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.946+0000 7f4eff7fe640 1 -- 172.21.15.1:0/2416768525 >> v1:172.21.15.98:6789/0 conn(0x7f4f00152920 legacy=0x7f4f0006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.946+0000 7f4eff7fe640 1 -- 172.21.15.1:0/2416768525 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f00158bf0 con 0x7f4f000c5850 2024-03-06T07:26:50.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.946+0000 7f4f04a88640 1 --2- 172.21.15.1:0/2416768525 >> v2:172.21.15.98:3300/0 conn(0x7f4f00155220 0x7f4f001556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:50.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.946+0000 7f4eff7fe640 1 --2- 172.21.15.1:0/2416768525 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f000c5850 0x7f4f000c5cb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4009870 tx=0x7f4ef402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74db276dec886fe8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.947+0000 7f4efeffd640 1 -- 172.21.15.1:0/2416768525 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4035020 con 0x7f4f000c5850 2024-03-06T07:26:50.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.947+0000 7f4efeffd640 1 -- 172.21.15.1:0/2416768525 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef4031640 con 0x7f4f000c5850 2024-03-06T07:26:50.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.947+0000 7f4efeffd640 1 -- 172.21.15.1:0/2416768525 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef402d040 con 0x7f4f000c5850 2024-03-06T07:26:50.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.947+0000 7f4f064eb640 1 -- 172.21.15.1:0/2416768525 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f000c5850 msgr2=0x7f4f000c5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.947+0000 7f4f064eb640 1 --2- 172.21.15.1:0/2416768525 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f000c5850 0x7f4f000c5cb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4009870 tx=0x7f4ef402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.947+0000 7f4f064eb640 1 -- 172.21.15.1:0/2416768525 shutdown_connections 2024-03-06T07:26:50.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.947+0000 7f4f064eb640 1 --2- 172.21.15.1:0/2416768525 >> v2:172.21.15.98:3300/0 conn(0x7f4f00155220 0x7f4f001556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.947+0000 7f4f064eb640 1 --2- 172.21.15.1:0/2416768525 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f000c5850 0x7f4f000c5cb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.947+0000 7f4f064eb640 1 -- 172.21.15.1:0/2416768525 >> 172.21.15.1:0/2416768525 conn(0x7f4f000a9700 msgr2=0x7f4f000a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.947+0000 7f4f064eb640 1 -- 172.21.15.1:0/2416768525 shutdown_connections 2024-03-06T07:26:50.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.947+0000 7f4f064eb640 1 -- 172.21.15.1:0/2416768525 wait complete. 2024-03-06T07:26:50.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.949+0000 7f4f064eb640 1 Processor -- start 2024-03-06T07:26:50.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.949+0000 7f4f064eb640 1 -- start start 2024-03-06T07:26:50.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.950+0000 7f4f064eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f000c5850 0x7f4f000c3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.950+0000 7f4f064eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f00152920 0x7f4f0006f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.950+0000 7f4f064eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f00155220 0x7f4f000c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.950+0000 7f4f064eb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f000c6380 con 0x7f4f00152920 2024-03-06T07:26:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.950+0000 7f4f064eb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f00159240 con 0x7f4f00155220 2024-03-06T07:26:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.950+0000 7f4f064eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0006b700 con 0x7f4f000c5850 2024-03-06T07:26:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.950+0000 7f4eff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f00152920 0x7f4f0006f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.950+0000 7f4eff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f00152920 0x7f4f0006f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39636/0 (socket says 172.21.15.1:39636) 2024-03-06T07:26:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.950+0000 7f4eff7fe640 1 -- 172.21.15.1:0/912606907 learned_addr learned my addr 172.21.15.1:0/912606907 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.950+0000 7f4f04a88640 1 --2- 172.21.15.1:0/912606907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f00155220 0x7f4f000c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.951+0000 7f4efffff640 1 --2- 172.21.15.1:0/912606907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f000c5850 0x7f4f000c3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.951+0000 7f4eff7fe640 1 -- 172.21.15.1:0/912606907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f000c5850 msgr2=0x7f4f000c3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.951+0000 7f4eff7fe640 1 --2- 172.21.15.1:0/912606907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f000c5850 0x7f4f000c3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.951+0000 7f4eff7fe640 1 -- 172.21.15.1:0/912606907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f00155220 msgr2=0x7f4f000c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.951+0000 7f4eff7fe640 1 --2- 172.21.15.1:0/912606907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f00155220 0x7f4f000c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.951+0000 7f4eff7fe640 1 -- 172.21.15.1:0/912606907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f000c44b0 con 0x7f4f00152920 2024-03-06T07:26:50.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.951+0000 7f4eff7fe640 1 --2- 172.21.15.1:0/912606907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f00152920 0x7f4f0006f670 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f4ef402b930 tx=0x7f4ef40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.951+0000 7f4efcff9640 1 -- 172.21.15.1:0/912606907 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4040070 con 0x7f4f00152920 2024-03-06T07:26:50.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.951+0000 7f4efcff9640 1 -- 172.21.15.1:0/912606907 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef403e070 con 0x7f4f00152920 2024-03-06T07:26:50.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.951+0000 7f4f064eb640 1 -- 172.21.15.1:0/912606907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0014c170 con 0x7f4f00152920 2024-03-06T07:26:50.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.951+0000 7f4efcff9640 1 -- 172.21.15.1:0/912606907 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4035040 con 0x7f4f00152920 2024-03-06T07:26:50.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.951+0000 7f4f064eb640 1 -- 172.21.15.1:0/912606907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f0006fbb0 con 0x7f4f00152920 2024-03-06T07:26:50.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.952+0000 7f4f064eb640 1 -- 172.21.15.1:0/912606907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f00067fd0 con 0x7f4f00152920 2024-03-06T07:26:50.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.953+0000 7f4efcff9640 1 -- 172.21.15.1:0/912606907 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ef4038020 con 0x7f4f00152920 2024-03-06T07:26:50.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.953+0000 7f4efcff9640 1 --2- 172.21.15.1:0/912606907 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4eec06b500 0x7f4eec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.953+0000 7f4efcff9640 1 -- 172.21.15.1:0/912606907 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f4ef40abc70 con 0x7f4f00152920 2024-03-06T07:26:50.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.953+0000 7f4efffff640 1 --2- 172.21.15.1:0/912606907 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4eec06b500 0x7f4eec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.954+0000 7f4efffff640 1 --2- 172.21.15.1:0/912606907 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4eec06b500 0x7f4eec06d9c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f4f0014f730 tx=0x7f4ef0007e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:50.957+0000 7f4efcff9640 1 -- 172.21.15.1:0/912606907 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ef407a890 con 0x7f4f00152920 2024-03-06T07:26:51.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.179+0000 7f4f064eb640 1 -- 172.21.15.1:0/912606907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f0006bdf0 con 0x7f4f00152920 2024-03-06T07:26:51.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.179+0000 7f4efcff9640 1 -- 172.21.15.1:0/912606907 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+4984 (secure 0 0 0) 0x7f4ef407a190 con 0x7f4f00152920 2024-03-06T07:26:51.182 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:51.182 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":151,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1387389703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1387389703},{"type":"v1","addr":"172.21.15.98:6836","nonce":1387389703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":8137,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2268963727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2268963727},{"type":"v1","addr":"172.21.15.98:6837","nonce":2268963727}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":9162,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/4198928111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4198928111},{"type":"v1","addr":"172.21.15.1:6833","nonce":4198928111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:26.024256+0000","modified":"2024-03-06T07:26:49.862056+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9168},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9168":{"gid":9168,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6835/2664051788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2664051788},{"type":"v1","addr":"172.21.15.1:6835","nonce":2664051788}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9168,"qdb_cluster":[9168]},"id":12}]} 2024-03-06T07:26:51.182 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 151 2024-03-06T07:26:51.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.181+0000 7f4f064eb640 1 -- 172.21.15.1:0/912606907 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4eec06b500 msgr2=0x7f4eec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.181+0000 7f4f064eb640 1 --2- 172.21.15.1:0/912606907 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4eec06b500 0x7f4eec06d9c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f4f0014f730 tx=0x7f4ef0007e80 comp rx=0 tx=0).stop 2024-03-06T07:26:51.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.181+0000 7f4f064eb640 1 -- 172.21.15.1:0/912606907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f00152920 msgr2=0x7f4f0006f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.181+0000 7f4f064eb640 1 --2- 172.21.15.1:0/912606907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f00152920 0x7f4f0006f670 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f4ef402b930 tx=0x7f4ef40046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.181+0000 7f4f064eb640 1 -- 172.21.15.1:0/912606907 shutdown_connections 2024-03-06T07:26:51.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.181+0000 7f4f064eb640 1 --2- 172.21.15.1:0/912606907 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4eec06b500 0x7f4eec06d9c0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.181+0000 7f4f064eb640 1 --2- 172.21.15.1:0/912606907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f00155220 0x7f4f000c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.181+0000 7f4f064eb640 1 --2- 172.21.15.1:0/912606907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f00152920 0x7f4f0006f670 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.181+0000 7f4f064eb640 1 --2- 172.21.15.1:0/912606907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4f000c5850 0x7f4f000c3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.181+0000 7f4f064eb640 1 -- 172.21.15.1:0/912606907 >> 172.21.15.1:0/912606907 conn(0x7f4f000a9700 msgr2=0x7f4f00153320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.181+0000 7f4f064eb640 1 -- 172.21.15.1:0/912606907 shutdown_connections 2024-03-06T07:26:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.181+0000 7f4f064eb640 1 -- 172.21.15.1:0/912606907 wait complete. 2024-03-06T07:26:51.194 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:51.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.294+0000 7f97a0a2a640 1 Processor -- start 2024-03-06T07:26:51.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.294+0000 7f97a0a2a640 1 -- start start 2024-03-06T07:26:51.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.295+0000 7f97a0a2a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f979c152650 0x7f979c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.295+0000 7f97a0a2a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f979c0c6b10 0x7f979c158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.295+0000 7f97a0a2a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f979c0bf9d0 0x7f979c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.295+0000 7f97a0a2a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f979c155800 con 0x7f979c152650 2024-03-06T07:26:51.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.295+0000 7f97a0a2a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f979c0c2fc0 con 0x7f979c0bf9d0 2024-03-06T07:26:51.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.295+0000 7f97a0a2a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f979c0c6690 con 0x7f979c0c6b10 2024-03-06T07:26:51.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.295+0000 7f979a59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f979c152650 0x7f979c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.295+0000 7f979a59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f979c152650 0x7f979c155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39652/0 (socket says 172.21.15.1:39652) 2024-03-06T07:26:51.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.295+0000 7f979a59c640 1 -- 172.21.15.1:0/3696957709 learned_addr learned my addr 172.21.15.1:0/3696957709 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:51.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.295+0000 7f9793fff640 1 --2- 172.21.15.1:0/3696957709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f979c0c6b10 0x7f979c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.295+0000 7f979ad9d640 1 --2- 172.21.15.1:0/3696957709 >> v2:172.21.15.98:3300/0 conn(0x7f979c0bf9d0 0x7f979c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.295+0000 7f979a59c640 1 -- 172.21.15.1:0/3696957709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f979c0c6b10 msgr2=0x7f979c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:51.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.295+0000 7f979a59c640 1 --2- 172.21.15.1:0/3696957709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f979c0c6b10 0x7f979c158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.295+0000 7f979a59c640 1 -- 172.21.15.1:0/3696957709 >> v2:172.21.15.98:3300/0 conn(0x7f979c0bf9d0 msgr2=0x7f979c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:51.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.295+0000 7f979a59c640 1 --2- 172.21.15.1:0/3696957709 >> v2:172.21.15.98:3300/0 conn(0x7f979c0bf9d0 0x7f979c0c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.295+0000 7f979a59c640 1 -- 172.21.15.1:0/3696957709 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c0c7280 con 0x7f979c152650 2024-03-06T07:26:51.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.296+0000 7f979a59c640 1 --2- 172.21.15.1:0/3696957709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c152650 0x7f979c155c90 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f979c1597b0 tx=0x7f978c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ffa0340cdefe84d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.296+0000 7f9799d9b640 1 -- 172.21.15.1:0/3696957709 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978c02e070 con 0x7f979c152650 2024-03-06T07:26:51.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.296+0000 7f9799d9b640 1 -- 172.21.15.1:0/3696957709 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978c037070 con 0x7f979c152650 2024-03-06T07:26:51.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.296+0000 7f9799d9b640 1 -- 172.21.15.1:0/3696957709 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978c031a00 con 0x7f979c152650 2024-03-06T07:26:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.297+0000 7f97a0a2a640 1 -- 172.21.15.1:0/3696957709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c152650 msgr2=0x7f979c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.297+0000 7f97a0a2a640 1 --2- 172.21.15.1:0/3696957709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c152650 0x7f979c155c90 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f979c1597b0 tx=0x7f978c02b880 comp rx=0 tx=0).stop 2024-03-06T07:26:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.297+0000 7f97a0a2a640 1 -- 172.21.15.1:0/3696957709 shutdown_connections 2024-03-06T07:26:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.297+0000 7f97a0a2a640 1 --2- 172.21.15.1:0/3696957709 >> v2:172.21.15.98:3300/0 conn(0x7f979c0bf9d0 0x7f979c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.297+0000 7f97a0a2a640 1 --2- 172.21.15.1:0/3696957709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f979c0c6b10 0x7f979c158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.297+0000 7f97a0a2a640 1 --2- 172.21.15.1:0/3696957709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c152650 0x7f979c155c90 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.297+0000 7f97a0a2a640 1 -- 172.21.15.1:0/3696957709 >> 172.21.15.1:0/3696957709 conn(0x7f979c066ee0 msgr2=0x7f979c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.297+0000 7f97a0a2a640 1 -- 172.21.15.1:0/3696957709 shutdown_connections 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.297+0000 7f97a0a2a640 1 -- 172.21.15.1:0/3696957709 wait complete. 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.298+0000 7f97a0a2a640 1 Processor -- start 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.298+0000 7f97a0a2a640 1 -- start start 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.299+0000 7f97a0a2a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f979c0bf9d0 0x7f979c14ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.299+0000 7f97a0a2a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c152650 0x7f979c142050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.299+0000 7f97a0a2a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f979c0c87d0 0x7f979c142590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.299+0000 7f97a0a2a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979c155490 con 0x7f979c152650 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.299+0000 7f97a0a2a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979c1551b0 con 0x7f979c0bf9d0 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.299+0000 7f97a0a2a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f979c152a30 con 0x7f979c0c87d0 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.299+0000 7f9793fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c152650 0x7f979c142050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.299+0000 7f9793fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c152650 0x7f979c142050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39664/0 (socket says 172.21.15.1:39664) 2024-03-06T07:26:51.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.299+0000 7f9793fff640 1 -- 172.21.15.1:0/1108696542 learned_addr learned my addr 172.21.15.1:0/1108696542 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:51.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.299+0000 7f979ad9d640 1 --2- 172.21.15.1:0/1108696542 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f979c0c87d0 0x7f979c142590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.299+0000 7f979a59c640 1 --2- 172.21.15.1:0/1108696542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f979c0bf9d0 0x7f979c14ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.299+0000 7f9793fff640 1 -- 172.21.15.1:0/1108696542 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f979c0c87d0 msgr2=0x7f979c142590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.299+0000 7f9793fff640 1 --2- 172.21.15.1:0/1108696542 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f979c0c87d0 0x7f979c142590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.299+0000 7f9793fff640 1 -- 172.21.15.1:0/1108696542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f979c0bf9d0 msgr2=0x7f979c14ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.299+0000 7f9793fff640 1 --2- 172.21.15.1:0/1108696542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f979c0bf9d0 0x7f979c14ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.300+0000 7f9793fff640 1 -- 172.21.15.1:0/1108696542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c1ebe10 con 0x7f979c152650 2024-03-06T07:26:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.300+0000 7f979a59c640 1 --2- 172.21.15.1:0/1108696542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f979c0bf9d0 0x7f979c14ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.300+0000 7f9793fff640 1 --2- 172.21.15.1:0/1108696542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c152650 0x7f979c142050 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f978400c7f0 tx=0x7f978400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.300+0000 7f9792ffd640 1 -- 172.21.15.1:0/1108696542 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784015070 con 0x7f979c152650 2024-03-06T07:26:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.300+0000 7f9792ffd640 1 -- 172.21.15.1:0/1108696542 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9784004240 con 0x7f979c152650 2024-03-06T07:26:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.300+0000 7f9792ffd640 1 -- 172.21.15.1:0/1108696542 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784007620 con 0x7f979c152650 2024-03-06T07:26:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.300+0000 7f97a0a2a640 1 -- 172.21.15.1:0/1108696542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f979c0ca250 con 0x7f979c152650 2024-03-06T07:26:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.300+0000 7f97a0a2a640 1 -- 172.21.15.1:0/1108696542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f979c14d870 con 0x7f979c152650 2024-03-06T07:26:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.301+0000 7f97a0a2a640 1 -- 172.21.15.1:0/1108696542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f979c0ca420 con 0x7f979c152650 2024-03-06T07:26:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.301+0000 7f9792ffd640 1 -- 172.21.15.1:0/1108696542 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97840079d0 con 0x7f979c152650 2024-03-06T07:26:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.302+0000 7f9792ffd640 1 --2- 172.21.15.1:0/1108696542 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f977c06b4d0 0x7f977c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.302+0000 7f9792ffd640 1 -- 172.21.15.1:0/1108696542 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f978408aa10 con 0x7f979c152650 2024-03-06T07:26:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.302+0000 7f979a59c640 1 --2- 172.21.15.1:0/1108696542 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f977c06b4d0 0x7f977c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.302+0000 7f979a59c640 1 --2- 172.21.15.1:0/1108696542 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f977c06b4d0 0x7f977c06d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f978c036000 tx=0x7f978c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.306+0000 7f9792ffd640 1 -- 172.21.15.1:0/1108696542 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9784059500 con 0x7f979c152650 2024-03-06T07:26:51.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.526+0000 7f97a0a2a640 1 -- 172.21.15.1:0/1108696542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f979c06b540 con 0x7f979c152650 2024-03-06T07:26:51.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.526+0000 7f9792ffd640 1 -- 172.21.15.1:0/1108696542 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+4984 (secure 0 0 0) 0x7f979c06b540 con 0x7f979c152650 2024-03-06T07:26:51.529 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:51.529 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":151,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1387389703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1387389703},{"type":"v1","addr":"172.21.15.98:6836","nonce":1387389703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":8137,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2268963727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2268963727},{"type":"v1","addr":"172.21.15.98:6837","nonce":2268963727}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":9162,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/4198928111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4198928111},{"type":"v1","addr":"172.21.15.1:6833","nonce":4198928111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:26.024256+0000","modified":"2024-03-06T07:26:49.862056+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9168},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9168":{"gid":9168,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6835/2664051788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2664051788},{"type":"v1","addr":"172.21.15.1:6835","nonce":2664051788}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9168,"qdb_cluster":[9168]},"id":12}]} 2024-03-06T07:26:51.529 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 151 2024-03-06T07:26:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.529+0000 7f97a0a2a640 1 -- 172.21.15.1:0/1108696542 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f977c06b4d0 msgr2=0x7f977c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.529+0000 7f97a0a2a640 1 --2- 172.21.15.1:0/1108696542 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f977c06b4d0 0x7f977c06d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f978c036000 tx=0x7f978c008030 comp rx=0 tx=0).stop 2024-03-06T07:26:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.529+0000 7f97a0a2a640 1 -- 172.21.15.1:0/1108696542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c152650 msgr2=0x7f979c142050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.529+0000 7f97a0a2a640 1 --2- 172.21.15.1:0/1108696542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c152650 0x7f979c142050 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f978400c7f0 tx=0x7f978400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.529+0000 7f97a0a2a640 1 -- 172.21.15.1:0/1108696542 shutdown_connections 2024-03-06T07:26:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.529+0000 7f97a0a2a640 1 --2- 172.21.15.1:0/1108696542 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f977c06b4d0 0x7f977c06d990 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.529+0000 7f97a0a2a640 1 --2- 172.21.15.1:0/1108696542 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f979c0c87d0 0x7f979c142590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.529+0000 7f97a0a2a640 1 --2- 172.21.15.1:0/1108696542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c152650 0x7f979c142050 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.529+0000 7f97a0a2a640 1 --2- 172.21.15.1:0/1108696542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f979c0bf9d0 0x7f979c14ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.529+0000 7f97a0a2a640 1 -- 172.21.15.1:0/1108696542 >> 172.21.15.1:0/1108696542 conn(0x7f979c066ee0 msgr2=0x7f979c0c52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.529+0000 7f97a0a2a640 1 -- 172.21.15.1:0/1108696542 shutdown_connections 2024-03-06T07:26:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.529+0000 7f97a0a2a640 1 -- 172.21.15.1:0/1108696542 wait complete. 2024-03-06T07:26:51.543 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.647+0000 7f3cd6e66640 1 Processor -- start 2024-03-06T07:26:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.647+0000 7f3cd6e66640 1 -- start start 2024-03-06T07:26:51.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.647+0000 7f3cd6e66640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cd00c6b10 0x7f3cd0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.647+0000 7f3cd6e66640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3cd00bf9d0 0x7f3cd00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.647+0000 7f3cd6e66640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd00c2fc0 con 0x7f3cd0152650 2024-03-06T07:26:51.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.647+0000 7f3cd6e66640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd0155800 con 0x7f3cd00bf9d0 2024-03-06T07:26:51.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.647+0000 7f3cd6e66640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd0158da0 con 0x7f3cd00c6b10 2024-03-06T07:26:51.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.648+0000 7f3cd4c02640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3cd0152650 0x7f3cd0158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33588/0 (socket says 172.21.15.1:33588) 2024-03-06T07:26:51.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.648+0000 7f3cd4c02640 1 -- 172.21.15.1:0/2525136424 learned_addr learned my addr 172.21.15.1:0/2525136424 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:51.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.648+0000 7f3ccffff640 1 --2- 172.21.15.1:0/2525136424 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cd00c6b10 0x7f3cd0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.648+0000 7f3cd5403640 1 --2- 172.21.15.1:0/2525136424 >> v2:172.21.15.98:3300/0 conn(0x7f3cd00bf9d0 0x7f3cd00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.648+0000 7f3ccf7fe640 1 -- 172.21.15.1:0/2525136424 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079208001 0 0) 0x7f3cd00c2fc0 con 0x7f3cd0152650 2024-03-06T07:26:51.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.648+0000 7f3ccf7fe640 1 -- 172.21.15.1:0/2525136424 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cb8003520 con 0x7f3cd0152650 2024-03-06T07:26:51.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.648+0000 7f3ccffff640 1 -- 172.21.15.1:0/2525136424 >> v2:172.21.15.98:3300/0 conn(0x7f3cd00bf9d0 msgr2=0x7f3cd00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:51.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.648+0000 7f3ccffff640 1 --2- 172.21.15.1:0/2525136424 >> v2:172.21.15.98:3300/0 conn(0x7f3cd00bf9d0 0x7f3cd00c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.648+0000 7f3ccffff640 1 -- 172.21.15.1:0/2525136424 >> v1:172.21.15.1:6789/0 conn(0x7f3cd0152650 legacy=0x7f3cd0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.648+0000 7f3ccffff640 1 -- 172.21.15.1:0/2525136424 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd00c6690 con 0x7f3cd00c6b10 2024-03-06T07:26:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.649+0000 7f3ccffff640 1 --2- 172.21.15.1:0/2525136424 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cd00c6b10 0x7f3cd0155c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3cc40097a0 tx=0x7f3cc402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2e8a038e13e923f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.649+0000 7f3ccf7fe640 1 -- 172.21.15.1:0/2525136424 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc40315a0 con 0x7f3cd00c6b10 2024-03-06T07:26:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.649+0000 7f3ccf7fe640 1 -- 172.21.15.1:0/2525136424 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc4037070 con 0x7f3cd00c6b10 2024-03-06T07:26:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.649+0000 7f3ccf7fe640 1 -- 172.21.15.1:0/2525136424 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4031960 con 0x7f3cd00c6b10 2024-03-06T07:26:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.650+0000 7f3cd6e66640 1 -- 172.21.15.1:0/2525136424 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cd00c6b10 msgr2=0x7f3cd0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.650+0000 7f3cd6e66640 1 --2- 172.21.15.1:0/2525136424 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cd00c6b10 0x7f3cd0155c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3cc40097a0 tx=0x7f3cc402b840 comp rx=0 tx=0).stop 2024-03-06T07:26:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.650+0000 7f3cd6e66640 1 -- 172.21.15.1:0/2525136424 shutdown_connections 2024-03-06T07:26:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.650+0000 7f3cd6e66640 1 --2- 172.21.15.1:0/2525136424 >> v2:172.21.15.98:3300/0 conn(0x7f3cd00bf9d0 0x7f3cd00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.650+0000 7f3cd6e66640 1 --2- 172.21.15.1:0/2525136424 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cd00c6b10 0x7f3cd0155c90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.650+0000 7f3cd6e66640 1 -- 172.21.15.1:0/2525136424 >> 172.21.15.1:0/2525136424 conn(0x7f3cd0066ee0 msgr2=0x7f3cd00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.650+0000 7f3cd6e66640 1 -- 172.21.15.1:0/2525136424 shutdown_connections 2024-03-06T07:26:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.650+0000 7f3cd6e66640 1 -- 172.21.15.1:0/2525136424 wait complete. 2024-03-06T07:26:51.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.652+0000 7f3cd6e66640 1 Processor -- start 2024-03-06T07:26:51.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.652+0000 7f3cd6e66640 1 -- start start 2024-03-06T07:26:51.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.653+0000 7f3cd6e66640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cd00bf9d0 0x7f3cd0157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.653+0000 7f3cd6e66640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cd0152650 0x7f3cd01de700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.653+0000 7f3cd6e66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cd0157950 0x7f3cd01dec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.653+0000 7f3cd6e66640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd01551b0 con 0x7f3cd0152650 2024-03-06T07:26:51.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.653+0000 7f3cd6e66640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd00c6fa0 con 0x7f3cd0157950 2024-03-06T07:26:51.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.653+0000 7f3cd6e66640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd0155490 con 0x7f3cd00bf9d0 2024-03-06T07:26:51.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.653+0000 7f3ccffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cd0152650 0x7f3cd01de700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.653+0000 7f3ccffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cd0152650 0x7f3cd01de700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39668/0 (socket says 172.21.15.1:39668) 2024-03-06T07:26:51.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.653+0000 7f3ccffff640 1 -- 172.21.15.1:0/3574399615 learned_addr learned my addr 172.21.15.1:0/3574399615 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:51.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.653+0000 7f3cd4c02640 1 --2- 172.21.15.1:0/3574399615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cd00bf9d0 0x7f3cd0157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.653+0000 7f3cd5403640 1 --2- 172.21.15.1:0/3574399615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cd0157950 0x7f3cd01dec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.653+0000 7f3ccffff640 1 -- 172.21.15.1:0/3574399615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cd00bf9d0 msgr2=0x7f3cd0157410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.653+0000 7f3ccffff640 1 --2- 172.21.15.1:0/3574399615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cd00bf9d0 0x7f3cd0157410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.654+0000 7f3ccffff640 1 -- 172.21.15.1:0/3574399615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cd0157950 msgr2=0x7f3cd01dec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.654+0000 7f3ccffff640 1 --2- 172.21.15.1:0/3574399615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cd0157950 0x7f3cd01dec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.654+0000 7f3ccffff640 1 -- 172.21.15.1:0/3574399615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd01df180 con 0x7f3cd0152650 2024-03-06T07:26:51.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.654+0000 7f3ccffff640 1 --2- 172.21.15.1:0/3574399615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cd0152650 0x7f3cd01de700 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4034040 tx=0x7f3cc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.654+0000 7f3ccd7fa640 1 -- 172.21.15.1:0/3574399615 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4032630 con 0x7f3cd0152650 2024-03-06T07:26:51.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.654+0000 7f3ccd7fa640 1 -- 172.21.15.1:0/3574399615 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc40327f0 con 0x7f3cd0152650 2024-03-06T07:26:51.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.654+0000 7f3cd6e66640 1 -- 172.21.15.1:0/3574399615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd01eb840 con 0x7f3cd0152650 2024-03-06T07:26:51.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.654+0000 7f3ccd7fa640 1 -- 172.21.15.1:0/3574399615 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4032be0 con 0x7f3cd0152650 2024-03-06T07:26:51.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.654+0000 7f3cd6e66640 1 -- 172.21.15.1:0/3574399615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd01ebae0 con 0x7f3cd0152650 2024-03-06T07:26:51.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.655+0000 7f3cd6e66640 1 -- 172.21.15.1:0/3574399615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd01ebcb0 con 0x7f3cd0152650 2024-03-06T07:26:51.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.655+0000 7f3ccd7fa640 1 -- 172.21.15.1:0/3574399615 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3cc404e430 con 0x7f3cd0152650 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.656+0000 7f3ccd7fa640 1 --2- 172.21.15.1:0/3574399615 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cbc06b4d0 0x7f3cbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.656+0000 7f3ccd7fa640 1 -- 172.21.15.1:0/3574399615 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f3cc402f030 con 0x7f3cd0152650 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.656+0000 7f3cd4c02640 1 --2- 172.21.15.1:0/3574399615 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cbc06b4d0 0x7f3cbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.657+0000 7f3cd4c02640 1 --2- 172.21.15.1:0/3574399615 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cbc06b4d0 0x7f3cbc06d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8005c00 tx=0x7f3cc8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.660+0000 7f3ccd7fa640 1 -- 172.21.15.1:0/3574399615 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3cc4040070 con 0x7f3cd0152650 2024-03-06T07:26:51.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.844+0000 7f3cd6e66640 1 -- 172.21.15.1:0/3574399615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3cd00b0a40 con 0x7f3cd0152650 2024-03-06T07:26:51.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.846+0000 7f3ccd7fa640 1 -- 172.21.15.1:0/3574399615 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v159) v1 ==== 74+0+17283 (secure 0 0 0) 0x7f3cc407a6a0 con 0x7f3cd0152650 2024-03-06T07:26:51.848 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:51.848 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":159,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:26:48.853480+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:26:24.304587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:26:25.085741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6837/2360179300":"2024-03-07T07:26:46.842519+0000","172.21.15.98:6835/2360179300":"2024-03-07T07:26:46.842519+0000","172.21.15.1:6833/1089548586":"2024-03-07T07:26:36.687217+0000","172.21.15.1:6832/1089548586":"2024-03-07T07:26:36.687217+0000","172.21.15.98:6836/3360241462":"2024-03-07T07:26:45.837702+0000","172.21.15.98:6834/3360241462":"2024-03-07T07:26:45.837702+0000","172.21.15.1:6835/67091521":"2024-03-07T07:26:36.682683+0000","172.21.15.1:6834/67091521":"2024-03-07T07:26:36.682683+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:51.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.848+0000 7f3cd6e66640 1 -- 172.21.15.1:0/3574399615 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cbc06b4d0 msgr2=0x7f3cbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.848+0000 7f3cd6e66640 1 --2- 172.21.15.1:0/3574399615 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cbc06b4d0 0x7f3cbc06d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8005c00 tx=0x7f3cc8009040 comp rx=0 tx=0).stop 2024-03-06T07:26:51.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.848+0000 7f3cd6e66640 1 -- 172.21.15.1:0/3574399615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cd0152650 msgr2=0x7f3cd01de700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.848+0000 7f3cd6e66640 1 --2- 172.21.15.1:0/3574399615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cd0152650 0x7f3cd01de700 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4034040 tx=0x7f3cc40046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.848+0000 7f3cd6e66640 1 -- 172.21.15.1:0/3574399615 shutdown_connections 2024-03-06T07:26:51.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.848+0000 7f3cd6e66640 1 --2- 172.21.15.1:0/3574399615 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cbc06b4d0 0x7f3cbc06d990 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.848+0000 7f3cd6e66640 1 --2- 172.21.15.1:0/3574399615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cd0157950 0x7f3cd01dec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.848+0000 7f3cd6e66640 1 --2- 172.21.15.1:0/3574399615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cd0152650 0x7f3cd01de700 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.848+0000 7f3cd6e66640 1 --2- 172.21.15.1:0/3574399615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cd00bf9d0 0x7f3cd0157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.848+0000 7f3cd6e66640 1 -- 172.21.15.1:0/3574399615 >> 172.21.15.1:0/3574399615 conn(0x7f3cd0066ee0 msgr2=0x7f3cd00c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.848+0000 7f3cd6e66640 1 -- 172.21.15.1:0/3574399615 shutdown_connections 2024-03-06T07:26:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.848+0000 7f3cd6e66640 1 -- 172.21.15.1:0/3574399615 wait complete. 2024-03-06T07:26:51.864 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs2 2024-03-06T07:26:51.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.964+0000 7f1c4ee8d640 1 Processor -- start 2024-03-06T07:26:51.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.965+0000 7f1c4ee8d640 1 -- start start 2024-03-06T07:26:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.965+0000 7f1c4ee8d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1c480c6b10 0x7f1c48155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.965+0000 7f1c4ee8d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1c480bf9d0 0x7f1c480c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.965+0000 7f1c4ee8d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c480c2fc0 con 0x7f1c48152650 2024-03-06T07:26:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.965+0000 7f1c4ee8d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c48155800 con 0x7f1c480bf9d0 2024-03-06T07:26:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.965+0000 7f1c4ee8d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c48158da0 con 0x7f1c480c6b10 2024-03-06T07:26:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.965+0000 7f1c4d42a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1c480bf9d0 0x7f1c480c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.965+0000 7f1c4d42a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1c480bf9d0 0x7f1c480c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39676/0 (socket says 172.21.15.1:39676) 2024-03-06T07:26:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.965+0000 7f1c4d42a640 1 -- 172.21.15.1:0/1394407911 learned_addr learned my addr 172.21.15.1:0/1394407911 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.965+0000 7f1c4cc29640 1 --1- 172.21.15.1:0/1394407911 >> v1:172.21.15.1:6789/0 conn(0x7f1c48152650 0x7f1c48158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33600/0 (socket says 172.21.15.1:33600) 2024-03-06T07:26:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.966+0000 7f1c47fff640 1 --2- 172.21.15.1:0/1394407911 >> v2:172.21.15.98:3300/0 conn(0x7f1c480c6b10 0x7f1c48155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.966+0000 7f1c4d42a640 1 -- 172.21.15.1:0/1394407911 >> v2:172.21.15.98:3300/0 conn(0x7f1c480c6b10 msgr2=0x7f1c48155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.966+0000 7f1c4d42a640 1 --2- 172.21.15.1:0/1394407911 >> v2:172.21.15.98:3300/0 conn(0x7f1c480c6b10 0x7f1c48155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.966+0000 7f1c4d42a640 1 -- 172.21.15.1:0/1394407911 >> v1:172.21.15.1:6789/0 conn(0x7f1c48152650 legacy=0x7f1c48158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.966+0000 7f1c4d42a640 1 -- 172.21.15.1:0/1394407911 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c480c6690 con 0x7f1c480bf9d0 2024-03-06T07:26:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.966+0000 7f1c477fe640 1 -- 172.21.15.1:0/1394407911 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750722229 0 0) 0x7f1c480c2fc0 con 0x7f1c48152650 2024-03-06T07:26:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.966+0000 7f1c47fff640 1 --2- 172.21.15.1:0/1394407911 >> v2:172.21.15.98:3300/0 conn(0x7f1c480c6b10 0x7f1c48155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.967+0000 7f1c4d42a640 1 --2- 172.21.15.1:0/1394407911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c480bf9d0 0x7f1c480c1460 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f1c480cb8e0 tx=0x7f1c4002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85c8825a46e9de1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.967+0000 7f1c477fe640 1 -- 172.21.15.1:0/1394407911 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40035020 con 0x7f1c480bf9d0 2024-03-06T07:26:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.967+0000 7f1c477fe640 1 -- 172.21.15.1:0/1394407911 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c40031870 con 0x7f1c480bf9d0 2024-03-06T07:26:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.967+0000 7f1c477fe640 1 -- 172.21.15.1:0/1394407911 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40031c40 con 0x7f1c480bf9d0 2024-03-06T07:26:51.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.967+0000 7f1c4ee8d640 1 -- 172.21.15.1:0/1394407911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c480bf9d0 msgr2=0x7f1c480c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.967+0000 7f1c4ee8d640 1 --2- 172.21.15.1:0/1394407911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c480bf9d0 0x7f1c480c1460 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f1c480cb8e0 tx=0x7f1c4002b910 comp rx=0 tx=0).stop 2024-03-06T07:26:51.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.967+0000 7f1c4ee8d640 1 -- 172.21.15.1:0/1394407911 shutdown_connections 2024-03-06T07:26:51.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.967+0000 7f1c4ee8d640 1 --2- 172.21.15.1:0/1394407911 >> v2:172.21.15.98:3300/0 conn(0x7f1c480c6b10 0x7f1c48155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.967+0000 7f1c4ee8d640 1 --2- 172.21.15.1:0/1394407911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c480bf9d0 0x7f1c480c1460 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.967+0000 7f1c4ee8d640 1 -- 172.21.15.1:0/1394407911 >> 172.21.15.1:0/1394407911 conn(0x7f1c48066ee0 msgr2=0x7f1c480672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.967+0000 7f1c4ee8d640 1 -- 172.21.15.1:0/1394407911 shutdown_connections 2024-03-06T07:26:51.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.967+0000 7f1c4ee8d640 1 -- 172.21.15.1:0/1394407911 wait complete. 2024-03-06T07:26:51.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.969+0000 7f1c4ee8d640 1 Processor -- start 2024-03-06T07:26:51.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.969+0000 7f1c4ee8d640 1 -- start start 2024-03-06T07:26:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.970+0000 7f1c4ee8d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c480bf9d0 0x7f1c48156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.970+0000 7f1c4ee8d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c48152650 0x7f1c481eb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.970+0000 7f1c4ee8d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c480c9a90 0x7f1c481533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.970+0000 7f1c4ee8d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c481551b0 con 0x7f1c480c9a90 2024-03-06T07:26:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.970+0000 7f1c4ee8d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c480c6fa0 con 0x7f1c480bf9d0 2024-03-06T07:26:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.970+0000 7f1c4ee8d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c48155490 con 0x7f1c48152650 2024-03-06T07:26:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.970+0000 7f1c4d42a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c480c9a90 0x7f1c481533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.970+0000 7f1c4d42a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c480c9a90 0x7f1c481533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39688/0 (socket says 172.21.15.1:39688) 2024-03-06T07:26:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.970+0000 7f1c4d42a640 1 -- 172.21.15.1:0/2947508291 learned_addr learned my addr 172.21.15.1:0/2947508291 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.971+0000 7f1c4d42a640 1 -- 172.21.15.1:0/2947508291 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c48152650 msgr2=0x7f1c481eb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.971+0000 7f1c47fff640 1 --2- 172.21.15.1:0/2947508291 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c48152650 0x7f1c481eb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.971+0000 7f1c4cc29640 1 --2- 172.21.15.1:0/2947508291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c480bf9d0 0x7f1c48156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.971+0000 7f1c4d42a640 1 --2- 172.21.15.1:0/2947508291 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c48152650 0x7f1c481eb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.971+0000 7f1c4d42a640 1 -- 172.21.15.1:0/2947508291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c480bf9d0 msgr2=0x7f1c48156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.971+0000 7f1c4d42a640 1 --2- 172.21.15.1:0/2947508291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c480bf9d0 0x7f1c48156850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.971+0000 7f1c4d42a640 1 -- 172.21.15.1:0/2947508291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c481deca0 con 0x7f1c480c9a90 2024-03-06T07:26:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.971+0000 7f1c4d42a640 1 --2- 172.21.15.1:0/2947508291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c480c9a90 0x7f1c481533d0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f1c40034040 tx=0x7f1c400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.971+0000 7f1c457fa640 1 -- 172.21.15.1:0/2947508291 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40040070 con 0x7f1c480c9a90 2024-03-06T07:26:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.971+0000 7f1c457fa640 1 -- 172.21.15.1:0/2947508291 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c40030c80 con 0x7f1c480c9a90 2024-03-06T07:26:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.971+0000 7f1c4ee8d640 1 -- 172.21.15.1:0/2947508291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c481540a0 con 0x7f1c480c9a90 2024-03-06T07:26:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.971+0000 7f1c457fa640 1 -- 172.21.15.1:0/2947508291 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40035040 con 0x7f1c480c9a90 2024-03-06T07:26:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.971+0000 7f1c4ee8d640 1 -- 172.21.15.1:0/2947508291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c48154660 con 0x7f1c480c9a90 2024-03-06T07:26:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.972+0000 7f1c4ee8d640 1 -- 172.21.15.1:0/2947508291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c481548a0 con 0x7f1c480c9a90 2024-03-06T07:26:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.973+0000 7f1c457fa640 1 -- 172.21.15.1:0/2947508291 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c40038020 con 0x7f1c480c9a90 2024-03-06T07:26:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.973+0000 7f1c457fa640 1 --2- 172.21.15.1:0/2947508291 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c2806b4d0 0x7f1c2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.973+0000 7f1c457fa640 1 -- 172.21.15.1:0/2947508291 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f1c400ac310 con 0x7f1c480c9a90 2024-03-06T07:26:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.974+0000 7f1c4cc29640 1 --2- 172.21.15.1:0/2947508291 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c2806b4d0 0x7f1c2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.974+0000 7f1c4cc29640 1 --2- 172.21.15.1:0/2947508291 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c2806b4d0 0x7f1c2806d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f1c30007fd0 tx=0x7f1c30009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:51.978+0000 7f1c457fa640 1 -- 172.21.15.1:0/2947508291 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c4007af30 con 0x7f1c480c9a90 2024-03-06T07:26:52.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.196+0000 7f1c4ee8d640 1 -- 172.21.15.1:0/2947508291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs2"} v 0) v1 -- 0x7f1c48067fd0 con 0x7f1c480c9a90 2024-03-06T07:26:52.198 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:26:52.197+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:52.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.198+0000 7f1c457fa640 1 -- 172.21.15.1:0/2947508291 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs2"}]=0 cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v152) v1 ==== 163+0+0 (secure 0 0 0) 0x7f1c4004ce80 con 0x7f1c480c9a90 2024-03-06T07:26:52.200 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:26:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.201+0000 7f1c4ee8d640 1 -- 172.21.15.1:0/2947508291 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c2806b4d0 msgr2=0x7f1c2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.201+0000 7f1c4ee8d640 1 --2- 172.21.15.1:0/2947508291 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c2806b4d0 0x7f1c2806d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f1c30007fd0 tx=0x7f1c30009040 comp rx=0 tx=0).stop 2024-03-06T07:26:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.201+0000 7f1c4ee8d640 1 -- 172.21.15.1:0/2947508291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c480c9a90 msgr2=0x7f1c481533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.201+0000 7f1c4ee8d640 1 --2- 172.21.15.1:0/2947508291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c480c9a90 0x7f1c481533d0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f1c40034040 tx=0x7f1c400046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.201+0000 7f1c4ee8d640 1 -- 172.21.15.1:0/2947508291 shutdown_connections 2024-03-06T07:26:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.201+0000 7f1c4ee8d640 1 --2- 172.21.15.1:0/2947508291 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1c2806b4d0 0x7f1c2806d990 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.201+0000 7f1c4ee8d640 1 --2- 172.21.15.1:0/2947508291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c480c9a90 0x7f1c481533d0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.201+0000 7f1c4ee8d640 1 --2- 172.21.15.1:0/2947508291 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1c48152650 0x7f1c481eb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.201+0000 7f1c4ee8d640 1 --2- 172.21.15.1:0/2947508291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c480bf9d0 0x7f1c48156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.201+0000 7f1c4ee8d640 1 -- 172.21.15.1:0/2947508291 >> 172.21.15.1:0/2947508291 conn(0x7f1c48066ee0 msgr2=0x7f1c480c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.201+0000 7f1c4ee8d640 1 -- 172.21.15.1:0/2947508291 shutdown_connections 2024-03-06T07:26:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.201+0000 7f1c4ee8d640 1 -- 172.21.15.1:0/2947508291 wait complete. 2024-03-06T07:26:52.216 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs2 --yes-i-really-mean-it 2024-03-06T07:26:52.228 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:26:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.317+0000 7f5e37b8a640 1 Processor -- start 2024-03-06T07:26:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.317+0000 7f5e37b8a640 1 -- start start 2024-03-06T07:26:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.318+0000 7f5e37b8a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5e30072bd0 0x7f5e300c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.318+0000 7f5e37b8a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e30152b70 0x7f5e300c3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.318+0000 7f5e37b8a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e30151dd0 con 0x7f5e300c2b30 2024-03-06T07:26:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.318+0000 7f5e37b8a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e301526e0 con 0x7f5e30072bd0 2024-03-06T07:26:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.318+0000 7f5e37b8a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e30156190 con 0x7f5e30152b70 2024-03-06T07:26:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.318+0000 7f5e35125640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5e30072bd0 0x7f5e300c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.318+0000 7f5e35125640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5e30072bd0 0x7f5e300c32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39708/0 (socket says 172.21.15.1:39708) 2024-03-06T07:26:52.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.318+0000 7f5e35125640 1 -- 172.21.15.1:0/1760353574 learned_addr learned my addr 172.21.15.1:0/1760353574 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.318+0000 7f5e36127640 1 --2- 172.21.15.1:0/1760353574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e30152b70 0x7f5e300c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.318+0000 7f5e35125640 1 -- 172.21.15.1:0/1760353574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e30152b70 msgr2=0x7f5e300c3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.318+0000 7f5e35125640 1 --2- 172.21.15.1:0/1760353574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e30152b70 0x7f5e300c3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.318+0000 7f5e35125640 1 -- 172.21.15.1:0/1760353574 >> v1:172.21.15.98:6789/0 conn(0x7f5e300c2b30 legacy=0x7f5e300724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.318+0000 7f5e35125640 1 -- 172.21.15.1:0/1760353574 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e30152090 con 0x7f5e30072bd0 2024-03-06T07:26:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.318+0000 7f5e36127640 1 --2- 172.21.15.1:0/1760353574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e30152b70 0x7f5e300c3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.319+0000 7f5e35125640 1 --2- 172.21.15.1:0/1760353574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e30072bd0 0x7f5e300c32f0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f5e30152860 tx=0x7f5e2002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a96ad1c3a23a67fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.319+0000 7f5e34924640 1 -- 172.21.15.1:0/1760353574 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e20035020 con 0x7f5e30072bd0 2024-03-06T07:26:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.319+0000 7f5e34924640 1 -- 172.21.15.1:0/1760353574 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e20031640 con 0x7f5e30072bd0 2024-03-06T07:26:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.319+0000 7f5e34924640 1 -- 172.21.15.1:0/1760353574 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2002d040 con 0x7f5e30072bd0 2024-03-06T07:26:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.319+0000 7f5e37b8a640 1 -- 172.21.15.1:0/1760353574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e30072bd0 msgr2=0x7f5e300c32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.319+0000 7f5e37b8a640 1 --2- 172.21.15.1:0/1760353574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e30072bd0 0x7f5e300c32f0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f5e30152860 tx=0x7f5e2002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.320+0000 7f5e37b8a640 1 -- 172.21.15.1:0/1760353574 shutdown_connections 2024-03-06T07:26:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.320+0000 7f5e37b8a640 1 --2- 172.21.15.1:0/1760353574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e30152b70 0x7f5e300c3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.320+0000 7f5e37b8a640 1 --2- 172.21.15.1:0/1760353574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e30072bd0 0x7f5e300c32f0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.320+0000 7f5e37b8a640 1 -- 172.21.15.1:0/1760353574 >> 172.21.15.1:0/1760353574 conn(0x7f5e300c6860 msgr2=0x7f5e300c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.320+0000 7f5e37b8a640 1 -- 172.21.15.1:0/1760353574 shutdown_connections 2024-03-06T07:26:52.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.320+0000 7f5e37b8a640 1 -- 172.21.15.1:0/1760353574 wait complete. 2024-03-06T07:26:52.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.321+0000 7f5e37b8a640 1 Processor -- start 2024-03-06T07:26:52.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.321+0000 7f5e37b8a640 1 -- start start 2024-03-06T07:26:52.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.321+0000 7f5e37b8a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e30072bd0 0x7f5e300abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.321+0000 7f5e37b8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e300c2b30 0x7f5e3014e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.321+0000 7f5e37b8a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e30152b70 0x7f5e301dd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.321+0000 7f5e37b8a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e300c3ef0 con 0x7f5e30152b70 2024-03-06T07:26:52.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.321+0000 7f5e37b8a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e300c3d70 con 0x7f5e300c2b30 2024-03-06T07:26:52.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.321+0000 7f5e37b8a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e30155d80 con 0x7f5e30072bd0 2024-03-06T07:26:52.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.321+0000 7f5e36127640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e30152b70 0x7f5e301dd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.321+0000 7f5e36127640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e30152b70 0x7f5e301dd5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39712/0 (socket says 172.21.15.1:39712) 2024-03-06T07:26:52.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.321+0000 7f5e36127640 1 -- 172.21.15.1:0/1891876737 learned_addr learned my addr 172.21.15.1:0/1891876737 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:52.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.322+0000 7f5e35926640 1 --2- 172.21.15.1:0/1891876737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e30072bd0 0x7f5e300abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.322+0000 7f5e35125640 1 --2- 172.21.15.1:0/1891876737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e300c2b30 0x7f5e3014e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.322+0000 7f5e36127640 1 -- 172.21.15.1:0/1891876737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e30072bd0 msgr2=0x7f5e300abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.322+0000 7f5e36127640 1 --2- 172.21.15.1:0/1891876737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e30072bd0 0x7f5e300abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.322+0000 7f5e36127640 1 -- 172.21.15.1:0/1891876737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e300c2b30 msgr2=0x7f5e3014e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.322+0000 7f5e36127640 1 --2- 172.21.15.1:0/1891876737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e300c2b30 0x7f5e3014e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.322+0000 7f5e36127640 1 -- 172.21.15.1:0/1891876737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e301538e0 con 0x7f5e30152b70 2024-03-06T07:26:52.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.322+0000 7f5e36127640 1 --2- 172.21.15.1:0/1891876737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e30152b70 0x7f5e301dd5f0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800cd40 tx=0x7f5e2800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.322+0000 7f5e267fc640 1 -- 172.21.15.1:0/1891876737 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28003b90 con 0x7f5e30152b70 2024-03-06T07:26:52.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.322+0000 7f5e267fc640 1 -- 172.21.15.1:0/1891876737 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e28003d50 con 0x7f5e30152b70 2024-03-06T07:26:52.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.322+0000 7f5e37b8a640 1 -- 172.21.15.1:0/1891876737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e30153600 con 0x7f5e30152b70 2024-03-06T07:26:52.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.322+0000 7f5e267fc640 1 -- 172.21.15.1:0/1891876737 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28004260 con 0x7f5e30152b70 2024-03-06T07:26:52.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.322+0000 7f5e37b8a640 1 -- 172.21.15.1:0/1891876737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e301deeb0 con 0x7f5e30152b70 2024-03-06T07:26:52.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.323+0000 7f5e37b8a640 1 -- 172.21.15.1:0/1891876737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e301ef830 con 0x7f5e30152b70 2024-03-06T07:26:52.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.324+0000 7f5e267fc640 1 -- 172.21.15.1:0/1891876737 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e28004400 con 0x7f5e30152b70 2024-03-06T07:26:52.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.324+0000 7f5e267fc640 1 --2- 172.21.15.1:0/1891876737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e1406b4d0 0x7f5e1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.324+0000 7f5e267fc640 1 -- 172.21.15.1:0/1891876737 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6535+0+0 (secure 0 0 0) 0x7f5e2808ae40 con 0x7f5e30152b70 2024-03-06T07:26:52.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.324+0000 7f5e35926640 1 --2- 172.21.15.1:0/1891876737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e1406b4d0 0x7f5e1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.325+0000 7f5e35926640 1 --2- 172.21.15.1:0/1891876737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e1406b4d0 0x7f5e1406d990 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f5e300bfcb0 tx=0x7f5e2c007e30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.328+0000 7f5e267fc640 1 -- 172.21.15.1:0/1891876737 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e28014030 con 0x7f5e30152b70 2024-03-06T07:26:52.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:52.549+0000 7f5e37b8a640 1 -- 172.21.15.1:0/1891876737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5e30067fd0 con 0x7f5e30152b70 2024-03-06T07:26:53.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.202+0000 7f5e267fc640 1 -- 172.21.15.1:0/1891876737 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true}]=0 v153) v1 ==== 105+0+0 (secure 0 0 0) 0x7f5e280593c0 con 0x7f5e30152b70 2024-03-06T07:26:53.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.206+0000 7f5e37b8a640 1 -- 172.21.15.1:0/1891876737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e1406b4d0 msgr2=0x7f5e1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.206+0000 7f5e37b8a640 1 --2- 172.21.15.1:0/1891876737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e1406b4d0 0x7f5e1406d990 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f5e300bfcb0 tx=0x7f5e2c007e30 comp rx=0 tx=0).stop 2024-03-06T07:26:53.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.206+0000 7f5e37b8a640 1 -- 172.21.15.1:0/1891876737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e30152b70 msgr2=0x7f5e301dd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.206+0000 7f5e37b8a640 1 --2- 172.21.15.1:0/1891876737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e30152b70 0x7f5e301dd5f0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800cd40 tx=0x7f5e2800a650 comp rx=0 tx=0).stop 2024-03-06T07:26:53.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.206+0000 7f5e37b8a640 1 -- 172.21.15.1:0/1891876737 shutdown_connections 2024-03-06T07:26:53.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.206+0000 7f5e37b8a640 1 --2- 172.21.15.1:0/1891876737 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e1406b4d0 0x7f5e1406d990 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.206+0000 7f5e37b8a640 1 --2- 172.21.15.1:0/1891876737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e30152b70 0x7f5e301dd5f0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.206+0000 7f5e37b8a640 1 --2- 172.21.15.1:0/1891876737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e300c2b30 0x7f5e3014e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.206+0000 7f5e37b8a640 1 --2- 172.21.15.1:0/1891876737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e30072bd0 0x7f5e300abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.206+0000 7f5e37b8a640 1 -- 172.21.15.1:0/1891876737 >> 172.21.15.1:0/1891876737 conn(0x7f5e300c6860 msgr2=0x7f5e301508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.206+0000 7f5e37b8a640 1 -- 172.21.15.1:0/1891876737 shutdown_connections 2024-03-06T07:26:53.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.206+0000 7f5e37b8a640 1 -- 172.21.15.1:0/1891876737 wait complete. 2024-03-06T07:26:53.221 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_metadata cephfs2_metadata --yes-i-really-really-mean-it 2024-03-06T07:26:53.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.374+0000 7f2dc810f640 1 Processor -- start 2024-03-06T07:26:53.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.374+0000 7f2dc810f640 1 -- start start 2024-03-06T07:26:53.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.375+0000 7f2dc810f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2dc00c16e0 0x7f2dc0152470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.375+0000 7f2dc810f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2dc00c70c0 0x7f2dc0155550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.375+0000 7f2dc810f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2dc01529b0 0x7f2dc0155a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.375+0000 7f2dc810f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc0158890 con 0x7f2dc00c70c0 2024-03-06T07:26:53.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.375+0000 7f2dc810f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc0158b70 con 0x7f2dc01529b0 2024-03-06T07:26:53.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.375+0000 7f2dc810f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc0158e50 con 0x7f2dc00c16e0 2024-03-06T07:26:53.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.375+0000 7f2dc56aa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2dc00c70c0 0x7f2dc0155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.375+0000 7f2dc56aa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2dc00c70c0 0x7f2dc0155550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39714/0 (socket says 172.21.15.1:39714) 2024-03-06T07:26:53.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.375+0000 7f2dc56aa640 1 -- 172.21.15.1:0/1823359485 learned_addr learned my addr 172.21.15.1:0/1823359485 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:53.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.375+0000 7f2dc66ac640 1 --2- 172.21.15.1:0/1823359485 >> v2:172.21.15.98:3300/0 conn(0x7f2dc01529b0 0x7f2dc0155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.375+0000 7f2dc5eab640 1 --2- 172.21.15.1:0/1823359485 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2dc00c16e0 0x7f2dc0152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.375+0000 7f2dc56aa640 1 -- 172.21.15.1:0/1823359485 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2dc00c16e0 msgr2=0x7f2dc0152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.375+0000 7f2dc56aa640 1 --2- 172.21.15.1:0/1823359485 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2dc00c16e0 0x7f2dc0152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.375+0000 7f2dc56aa640 1 -- 172.21.15.1:0/1823359485 >> v2:172.21.15.98:3300/0 conn(0x7f2dc01529b0 msgr2=0x7f2dc0155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.375+0000 7f2dc56aa640 1 --2- 172.21.15.1:0/1823359485 >> v2:172.21.15.98:3300/0 conn(0x7f2dc01529b0 0x7f2dc0155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.376+0000 7f2dc56aa640 1 -- 172.21.15.1:0/1823359485 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc0159130 con 0x7f2dc00c70c0 2024-03-06T07:26:53.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.376+0000 7f2dc56aa640 1 --2- 172.21.15.1:0/1823359485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dc00c70c0 0x7f2dc0155550 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f2dc00cbb50 tx=0x7f2db002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e353632f620f69 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.376+0000 7f2dc4ea9640 1 -- 172.21.15.1:0/1823359485 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db002e070 con 0x7f2dc00c70c0 2024-03-06T07:26:53.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.376+0000 7f2dc4ea9640 1 -- 172.21.15.1:0/1823359485 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db0037070 con 0x7f2dc00c70c0 2024-03-06T07:26:53.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.376+0000 7f2dc4ea9640 1 -- 172.21.15.1:0/1823359485 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db0031a00 con 0x7f2dc00c70c0 2024-03-06T07:26:53.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.377+0000 7f2dc810f640 1 -- 172.21.15.1:0/1823359485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dc00c70c0 msgr2=0x7f2dc0155550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.377+0000 7f2dc810f640 1 --2- 172.21.15.1:0/1823359485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dc00c70c0 0x7f2dc0155550 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f2dc00cbb50 tx=0x7f2db002b880 comp rx=0 tx=0).stop 2024-03-06T07:26:53.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.377+0000 7f2dc810f640 1 -- 172.21.15.1:0/1823359485 shutdown_connections 2024-03-06T07:26:53.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.377+0000 7f2dc810f640 1 --2- 172.21.15.1:0/1823359485 >> v2:172.21.15.98:3300/0 conn(0x7f2dc01529b0 0x7f2dc0155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.377+0000 7f2dc810f640 1 --2- 172.21.15.1:0/1823359485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dc00c70c0 0x7f2dc0155550 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.377+0000 7f2dc810f640 1 --2- 172.21.15.1:0/1823359485 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2dc00c16e0 0x7f2dc0152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.377+0000 7f2dc810f640 1 -- 172.21.15.1:0/1823359485 >> 172.21.15.1:0/1823359485 conn(0x7f2dc0066d00 msgr2=0x7f2dc0067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.377+0000 7f2dc810f640 1 -- 172.21.15.1:0/1823359485 shutdown_connections 2024-03-06T07:26:53.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.377+0000 7f2dc810f640 1 -- 172.21.15.1:0/1823359485 wait complete. 2024-03-06T07:26:53.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.378+0000 7f2dc810f640 1 Processor -- start 2024-03-06T07:26:53.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.378+0000 7f2dc810f640 1 -- start start 2024-03-06T07:26:53.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.379+0000 7f2dc810f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2dc00c16e0 0x7f2dc014b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.379+0000 7f2dc810f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dc00c70c0 0x7f2dc01eaf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.379+0000 7f2dc810f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc01529b0 0x7f2dc01eb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.379+0000 7f2dc810f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc00c68c0 con 0x7f2dc00c70c0 2024-03-06T07:26:53.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.379+0000 7f2dc810f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc01596f0 con 0x7f2dc01529b0 2024-03-06T07:26:53.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.379+0000 7f2dc810f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc00cb470 con 0x7f2dc00c16e0 2024-03-06T07:26:53.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.379+0000 7f2dc56aa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dc00c70c0 0x7f2dc01eaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.379+0000 7f2dc56aa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dc00c70c0 0x7f2dc01eaf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39718/0 (socket says 172.21.15.1:39718) 2024-03-06T07:26:53.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.379+0000 7f2dc56aa640 1 -- 172.21.15.1:0/1207573754 learned_addr learned my addr 172.21.15.1:0/1207573754 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:53.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.379+0000 7f2dc56aa640 1 -- 172.21.15.1:0/1207573754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2dc00c16e0 msgr2=0x7f2dc014b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.379+0000 7f2dc66ac640 1 --2- 172.21.15.1:0/1207573754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc01529b0 0x7f2dc01eb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.379+0000 7f2dc5eab640 1 --2- 172.21.15.1:0/1207573754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2dc00c16e0 0x7f2dc014b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.379+0000 7f2dc56aa640 1 --2- 172.21.15.1:0/1207573754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2dc00c16e0 0x7f2dc014b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.379+0000 7f2dc56aa640 1 -- 172.21.15.1:0/1207573754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc01529b0 msgr2=0x7f2dc01eb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.379+0000 7f2dc56aa640 1 --2- 172.21.15.1:0/1207573754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc01529b0 0x7f2dc01eb460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.379+0000 7f2dc56aa640 1 -- 172.21.15.1:0/1207573754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc0158090 con 0x7f2dc00c70c0 2024-03-06T07:26:53.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.379+0000 7f2dc56aa640 1 --2- 172.21.15.1:0/1207573754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dc00c70c0 0x7f2dc01eaf20 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f2db002b8c0 tx=0x7f2db00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.380+0000 7f2db67fc640 1 -- 172.21.15.1:0/1207573754 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db0032880 con 0x7f2dc00c70c0 2024-03-06T07:26:53.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.380+0000 7f2db67fc640 1 -- 172.21.15.1:0/1207573754 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db0032a40 con 0x7f2dc00c70c0 2024-03-06T07:26:53.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.380+0000 7f2dc810f640 1 -- 172.21.15.1:0/1207573754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc0154a70 con 0x7f2dc00c70c0 2024-03-06T07:26:53.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.380+0000 7f2db67fc640 1 -- 172.21.15.1:0/1207573754 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db0032e30 con 0x7f2dc00c70c0 2024-03-06T07:26:53.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.380+0000 7f2dc810f640 1 -- 172.21.15.1:0/1207573754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc014b9b0 con 0x7f2dc00c70c0 2024-03-06T07:26:53.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.381+0000 7f2dc810f640 1 -- 172.21.15.1:0/1207573754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc00680b0 con 0x7f2dc00c70c0 2024-03-06T07:26:53.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.381+0000 7f2db67fc640 1 -- 172.21.15.1:0/1207573754 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2db00453f0 con 0x7f2dc00c70c0 2024-03-06T07:26:53.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.382+0000 7f2db67fc640 1 --2- 172.21.15.1:0/1207573754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2da406b4d0 0x7f2da406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.382+0000 7f2db67fc640 1 -- 172.21.15.1:0/1207573754 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6535+0+0 (secure 0 0 0) 0x7f2db00ac2f0 con 0x7f2dc00c70c0 2024-03-06T07:26:53.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.382+0000 7f2dc5eab640 1 --2- 172.21.15.1:0/1207573754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2da406b4d0 0x7f2da406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.382+0000 7f2dc5eab640 1 --2- 172.21.15.1:0/1207573754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2da406b4d0 0x7f2da406d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f2db8006fd0 tx=0x7f2db8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.386+0000 7f2db67fc640 1 -- 172.21.15.1:0/1207573754 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2db0040070 con 0x7f2dc00c70c0 2024-03-06T07:26:53.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:53.566+0000 7f2dc810f640 1 -- 172.21.15.1:0/1207573754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2dc00b0880 con 0x7f2dc00c70c0 2024-03-06T07:26:54.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.204+0000 7f2db67fc640 1 -- 172.21.15.1:0/1207573754 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_metadata' removed v161) v1 ==== 184+0+0 (secure 0 0 0) 0x7f2db007aec0 con 0x7f2dc00c70c0 2024-03-06T07:26:54.206 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs2_metadata' removed 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.207+0000 7f2dc810f640 1 -- 172.21.15.1:0/1207573754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2da406b4d0 msgr2=0x7f2da406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.207+0000 7f2dc810f640 1 --2- 172.21.15.1:0/1207573754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2da406b4d0 0x7f2da406d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f2db8006fd0 tx=0x7f2db8008040 comp rx=0 tx=0).stop 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.207+0000 7f2dc810f640 1 -- 172.21.15.1:0/1207573754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dc00c70c0 msgr2=0x7f2dc01eaf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.207+0000 7f2dc810f640 1 --2- 172.21.15.1:0/1207573754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dc00c70c0 0x7f2dc01eaf20 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f2db002b8c0 tx=0x7f2db00046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.208+0000 7f2dc810f640 1 -- 172.21.15.1:0/1207573754 shutdown_connections 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.208+0000 7f2dc810f640 1 --2- 172.21.15.1:0/1207573754 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2da406b4d0 0x7f2da406d990 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.208+0000 7f2dc810f640 1 --2- 172.21.15.1:0/1207573754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc01529b0 0x7f2dc01eb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.208+0000 7f2dc810f640 1 --2- 172.21.15.1:0/1207573754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dc00c70c0 0x7f2dc01eaf20 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.208+0000 7f2dc810f640 1 --2- 172.21.15.1:0/1207573754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2dc00c16e0 0x7f2dc014b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.208+0000 7f2dc810f640 1 -- 172.21.15.1:0/1207573754 >> 172.21.15.1:0/1207573754 conn(0x7f2dc0066d00 msgr2=0x7f2dc00670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.208+0000 7f2dc810f640 1 -- 172.21.15.1:0/1207573754 shutdown_connections 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.208+0000 7f2dc810f640 1 -- 172.21.15.1:0/1207573754 wait complete. 2024-03-06T07:26:54.225 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_data cephfs2_data --yes-i-really-really-mean-it 2024-03-06T07:26:54.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.334+0000 7f42d7f5c640 1 Processor -- start 2024-03-06T07:26:54.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.334+0000 7f42d7f5c640 1 -- start start 2024-03-06T07:26:54.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d7f5c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f42d00c5950 0x7f42d00c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d7f5c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d00728c0 con 0x7f42d00a60b0 2024-03-06T07:26:54.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d7f5c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d0072ba0 con 0x7f42d006eae0 2024-03-06T07:26:54.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d7f5c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42d0072690 con 0x7f42d00c5950 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d54f7640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f42d00a60b0 0x7f42d0152910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33614/0 (socket says 172.21.15.1:33614) 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d54f7640 1 -- 172.21.15.1:0/934322707 learned_addr learned my addr 172.21.15.1:0/934322707 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d5cf8640 1 --2- 172.21.15.1:0/934322707 >> v2:172.21.15.98:3300/0 conn(0x7f42d00c5950 0x7f42d00c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d4cf6640 1 -- 172.21.15.1:0/934322707 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228025496 0 0) 0x7f42d00728c0 con 0x7f42d00a60b0 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d4cf6640 1 -- 172.21.15.1:0/934322707 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42bc003520 con 0x7f42d00a60b0 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d4cf6640 1 -- 172.21.15.1:0/934322707 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170602467 0 0) 0x7f42d0072ba0 con 0x7f42d006eae0 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d4cf6640 1 -- 172.21.15.1:0/934322707 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42d00728c0 con 0x7f42d006eae0 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d4cf6640 1 -- 172.21.15.1:0/934322707 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3243170704 0 0) 0x7f42bc003520 con 0x7f42d00a60b0 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d4cf6640 1 -- 172.21.15.1:0/934322707 >> v2:172.21.15.98:3300/0 conn(0x7f42d00c5950 msgr2=0x7f42d00c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d4cf6640 1 --2- 172.21.15.1:0/934322707 >> v2:172.21.15.98:3300/0 conn(0x7f42d00c5950 0x7f42d00c5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d4cf6640 1 -- 172.21.15.1:0/934322707 >> v1:172.21.15.98:6789/0 conn(0x7f42d006eae0 legacy=0x7f42d00c1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d4cf6640 1 -- 172.21.15.1:0/934322707 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d0072e80 con 0x7f42d00a60b0 2024-03-06T07:26:54.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d4cf6640 1 -- 172.21.15.1:0/934322707 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f42c0001670 con 0x7f42d00a60b0 2024-03-06T07:26:54.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d4cf6640 1 -- 172.21.15.1:0/934322707 >> v1:172.21.15.1:6789/0 conn(0x7f42d00a60b0 legacy=0x7f42d0152910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d4cf6640 1 --2- 172.21.15.1:0/934322707 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42bc003b90 0x7f42bc003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d4cf6640 1 --2- 172.21.15.1:0/934322707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42bc004800 0x7f42bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d4cf6640 1 --2- 172.21.15.1:0/934322707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42bc0051c0 0x7f42bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d4cf6640 1 -- 172.21.15.1:0/934322707 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc00c090 con 0x7f42bc0051c0 2024-03-06T07:26:54.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d4cf6640 1 -- 172.21.15.1:0/934322707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc00c210 con 0x7f42bc004800 2024-03-06T07:26:54.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d4cf6640 1 -- 172.21.15.1:0/934322707 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc00c410 con 0x7f42bc003b90 2024-03-06T07:26:54.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.335+0000 7f42d5cf8640 1 --2- 172.21.15.1:0/934322707 >> v2:172.21.15.98:3300/0 conn(0x7f42d00c5950 0x7f42d00c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:54.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.336+0000 7f42d64f9640 1 --2- 172.21.15.1:0/934322707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42bc0051c0 0x7f42bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.336+0000 7f42d54f7640 1 --2- 172.21.15.1:0/934322707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42bc004800 0x7f42bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.336+0000 7f42d5cf8640 1 --2- 172.21.15.1:0/934322707 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42bc003b90 0x7f42bc003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.336+0000 7f42d64f9640 1 -- 172.21.15.1:0/934322707 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42bc003b90 msgr2=0x7f42bc003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.336+0000 7f42d64f9640 1 --2- 172.21.15.1:0/934322707 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42bc003b90 0x7f42bc003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.336+0000 7f42d64f9640 1 -- 172.21.15.1:0/934322707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42bc004800 msgr2=0x7f42bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.336+0000 7f42d64f9640 1 --2- 172.21.15.1:0/934322707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42bc004800 0x7f42bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.336+0000 7f42d64f9640 1 -- 172.21.15.1:0/934322707 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc003520 con 0x7f42bc0051c0 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.337+0000 7f42d64f9640 1 -- 172.21.15.1:0/934322707 reap_dead start 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.337+0000 7f42d64f9640 1 --2- 172.21.15.1:0/934322707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42bc0051c0 0x7f42bc00b9c0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f42c8001fd0 tx=0x7f42bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d64d98dc27aa718d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.337+0000 7f42d4cf6640 1 -- 172.21.15.1:0/934322707 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c8001bf0 con 0x7f42bc0051c0 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.337+0000 7f42d4cf6640 1 -- 172.21.15.1:0/934322707 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c8003b70 con 0x7f42bc0051c0 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.337+0000 7f42d4cf6640 1 -- 172.21.15.1:0/934322707 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c80050d0 con 0x7f42bc0051c0 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.337+0000 7f42d7f5c640 1 -- 172.21.15.1:0/934322707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42bc0051c0 msgr2=0x7f42bc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.337+0000 7f42d7f5c640 1 --2- 172.21.15.1:0/934322707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42bc0051c0 0x7f42bc00b9c0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f42c8001fd0 tx=0x7f42bc005830 comp rx=0 tx=0).stop 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.337+0000 7f42d7f5c640 1 -- 172.21.15.1:0/934322707 shutdown_connections 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.337+0000 7f42d7f5c640 1 --2- 172.21.15.1:0/934322707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42bc0051c0 0x7f42bc00b9c0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.337+0000 7f42d7f5c640 1 -- 172.21.15.1:0/934322707 >> 172.21.15.1:0/934322707 conn(0x7f42d00a8e10 msgr2=0x7f42d00c3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.337+0000 7f42d7f5c640 1 -- 172.21.15.1:0/934322707 shutdown_connections 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.337+0000 7f42d7f5c640 1 -- 172.21.15.1:0/934322707 wait complete. 2024-03-06T07:26:54.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.338+0000 7f42d7f5c640 1 Processor -- start 2024-03-06T07:26:54.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.339+0000 7f42d7f5c640 1 -- start start 2024-03-06T07:26:54.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.339+0000 7f42d7f5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42bc0051c0 0x7f42d01ea440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.339+0000 7f42d7f5c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42d01547c0 0x7f42d00bffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.339+0000 7f42d7f5c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42d00a99f0 0x7f42d00a9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.339+0000 7f42d7f5c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d00c5eb0 con 0x7f42d00a99f0 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.339+0000 7f42d7f5c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d00c5d30 con 0x7f42bc0051c0 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.339+0000 7f42d7f5c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d00c6030 con 0x7f42d01547c0 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.339+0000 7f42d64f9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42d00a99f0 0x7f42d00a9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.339+0000 7f42d64f9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42d00a99f0 0x7f42d00a9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39738/0 (socket says 172.21.15.1:39738) 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.339+0000 7f42d64f9640 1 -- 172.21.15.1:0/1075497193 learned_addr learned my addr 172.21.15.1:0/1075497193 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.339+0000 7f42d5cf8640 1 --2- 172.21.15.1:0/1075497193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42bc0051c0 0x7f42d01ea440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.339+0000 7f42d54f7640 1 --2- 172.21.15.1:0/1075497193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42d01547c0 0x7f42d00bffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.340+0000 7f42d64f9640 1 -- 172.21.15.1:0/1075497193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42d01547c0 msgr2=0x7f42d00bffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.340+0000 7f42d64f9640 1 --2- 172.21.15.1:0/1075497193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42d01547c0 0x7f42d00bffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.340+0000 7f42d64f9640 1 -- 172.21.15.1:0/1075497193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42bc0051c0 msgr2=0x7f42d01ea440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.340+0000 7f42d64f9640 1 --2- 172.21.15.1:0/1075497193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42bc0051c0 0x7f42d01ea440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.340+0000 7f42d64f9640 1 -- 172.21.15.1:0/1075497193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d0153720 con 0x7f42d00a99f0 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.340+0000 7f42d64f9640 1 --2- 172.21.15.1:0/1075497193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42d00a99f0 0x7f42d00a9ea0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f42c802c1e0 tx=0x7f42c8001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.340+0000 7f42c67fc640 1 -- 172.21.15.1:0/1075497193 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c8001bf0 con 0x7f42d00a99f0 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.340+0000 7f42c67fc640 1 -- 172.21.15.1:0/1075497193 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c8003970 con 0x7f42d00a99f0 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.340+0000 7f42c67fc640 1 -- 172.21.15.1:0/1075497193 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c8005810 con 0x7f42d00a99f0 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.340+0000 7f42d7f5c640 1 -- 172.21.15.1:0/1075497193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d0153a00 con 0x7f42d00a99f0 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.340+0000 7f42d7f5c640 1 -- 172.21.15.1:0/1075497193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d0153fc0 con 0x7f42d00a99f0 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.341+0000 7f42d7f5c640 1 -- 172.21.15.1:0/1075497193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42d00680b0 con 0x7f42d00a99f0 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.341+0000 7f42c67fc640 1 -- 172.21.15.1:0/1075497193 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f42c802c570 con 0x7f42d00a99f0 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.342+0000 7f42c67fc640 1 --2- 172.21.15.1:0/1075497193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f42b406b4d0 0x7f42b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.342+0000 7f42c67fc640 1 -- 172.21.15.1:0/1075497193 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6095+0+0 (secure 0 0 0) 0x7f42c80ab130 con 0x7f42d00a99f0 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.342+0000 7f42d5cf8640 1 --2- 172.21.15.1:0/1075497193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f42b406b4d0 0x7f42b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.343+0000 7f42d5cf8640 1 --2- 172.21.15.1:0/1075497193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f42b406b4d0 0x7f42b406d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0093d0 tx=0x7f42cc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.346+0000 7f42c67fc640 1 -- 172.21.15.1:0/1075497193 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42c8046020 con 0x7f42d00a99f0 2024-03-06T07:26:54.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:54.528+0000 7f42d7f5c640 1 -- 172.21.15.1:0/1075497193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f42d00c2ab0 con 0x7f42d00a99f0 2024-03-06T07:26:55.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.212+0000 7f42c67fc640 1 -- 172.21.15.1:0/1075497193 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_data' removed v162) v1 ==== 172+0+0 (secure 0 0 0) 0x7f42c8003b40 con 0x7f42d00a99f0 2024-03-06T07:26:55.214 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs2_data' removed 2024-03-06T07:26:55.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.215+0000 7f42d7f5c640 1 -- 172.21.15.1:0/1075497193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f42b406b4d0 msgr2=0x7f42b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.215+0000 7f42d7f5c640 1 --2- 172.21.15.1:0/1075497193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f42b406b4d0 0x7f42b406d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0093d0 tx=0x7f42cc008040 comp rx=0 tx=0).stop 2024-03-06T07:26:55.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.215+0000 7f42d7f5c640 1 -- 172.21.15.1:0/1075497193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42d00a99f0 msgr2=0x7f42d00a9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.215+0000 7f42d7f5c640 1 --2- 172.21.15.1:0/1075497193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42d00a99f0 0x7f42d00a9ea0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f42c802c1e0 tx=0x7f42c8001650 comp rx=0 tx=0).stop 2024-03-06T07:26:55.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.216+0000 7f42d7f5c640 1 -- 172.21.15.1:0/1075497193 shutdown_connections 2024-03-06T07:26:55.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.216+0000 7f42d7f5c640 1 --2- 172.21.15.1:0/1075497193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f42b406b4d0 0x7f42b406d990 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.216+0000 7f42d7f5c640 1 --2- 172.21.15.1:0/1075497193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42d00a99f0 0x7f42d00a9ea0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.216+0000 7f42d7f5c640 1 --2- 172.21.15.1:0/1075497193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42d01547c0 0x7f42d00bffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.216+0000 7f42d7f5c640 1 --2- 172.21.15.1:0/1075497193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42bc0051c0 0x7f42d01ea440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.216+0000 7f42d7f5c640 1 -- 172.21.15.1:0/1075497193 >> 172.21.15.1:0/1075497193 conn(0x7f42d00a8e10 msgr2=0x7f42d0151720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.216+0000 7f42d7f5c640 1 -- 172.21.15.1:0/1075497193 shutdown_connections 2024-03-06T07:26:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.216+0000 7f42d7f5c640 1 -- 172.21.15.1:0/1075497193 wait complete. 2024-03-06T07:26:55.233 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:26:55.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.345+0000 7fa10ecfa640 1 Processor -- start 2024-03-06T07:26:55.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.346+0000 7fa10ecfa640 1 -- start start 2024-03-06T07:26:55.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.346+0000 7fa10ecfa640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa1000d5840 0x7fa1000dc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.346+0000 7fa10ecfa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa10004bb60 0x7fa1000483c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.346+0000 7fa10ecfa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa100048900 0x7fa10004a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.346+0000 7fa10ecfa640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1000d5c20 con 0x7fa10004bb60 2024-03-06T07:26:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.346+0000 7fa10ecfa640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa10004c480 con 0x7fa1000d5840 2024-03-06T07:26:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.346+0000 7fa10ecfa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1000dc6a0 con 0x7fa100048900 2024-03-06T07:26:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.346+0000 7fa10d4f7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa10004bb60 0x7fa1000483c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.346+0000 7fa10d4f7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa10004bb60 0x7fa1000483c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45054/0 (socket says 172.21.15.1:45054) 2024-03-06T07:26:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.346+0000 7fa10d4f7640 1 -- 172.21.15.1:0/1170742610 learned_addr learned my addr 172.21.15.1:0/1170742610 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.347+0000 7fa10e4f9640 1 --2- 172.21.15.1:0/1170742610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa100048900 0x7fa10004a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.347+0000 7fa10dcf8640 1 --2- 172.21.15.1:0/1170742610 >> v2:172.21.15.98:3300/0 conn(0x7fa1000d5840 0x7fa1000dc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.347+0000 7fa10d4f7640 1 -- 172.21.15.1:0/1170742610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa100048900 msgr2=0x7fa10004a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.347+0000 7fa10d4f7640 1 --2- 172.21.15.1:0/1170742610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa100048900 0x7fa10004a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.347+0000 7fa10d4f7640 1 -- 172.21.15.1:0/1170742610 >> v2:172.21.15.98:3300/0 conn(0x7fa1000d5840 msgr2=0x7fa1000dc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.347+0000 7fa10d4f7640 1 --2- 172.21.15.1:0/1170742610 >> v2:172.21.15.98:3300/0 conn(0x7fa1000d5840 0x7fa1000dc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.347+0000 7fa10d4f7640 1 -- 172.21.15.1:0/1170742610 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10004c060 con 0x7fa10004bb60 2024-03-06T07:26:55.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.347+0000 7fa10e4f9640 1 --2- 172.21.15.1:0/1170742610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa100048900 0x7fa10004a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:55.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.348+0000 7fa10d4f7640 1 --2- 172.21.15.1:0/1170742610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa10004bb60 0x7fa1000483c0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fa10000e9f0 tx=0x7fa0f802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54484ffaddd5ec9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.348+0000 7fa10ccf6640 1 -- 172.21.15.1:0/1170742610 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f80317c0 con 0x7fa10004bb60 2024-03-06T07:26:55.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.348+0000 7fa10ccf6640 1 -- 172.21.15.1:0/1170742610 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f8037070 con 0x7fa10004bb60 2024-03-06T07:26:55.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.348+0000 7fa10ccf6640 1 -- 172.21.15.1:0/1170742610 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8031b80 con 0x7fa10004bb60 2024-03-06T07:26:55.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.348+0000 7fa10ecfa640 1 -- 172.21.15.1:0/1170742610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa10004bb60 msgr2=0x7fa1000483c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.348+0000 7fa10ecfa640 1 --2- 172.21.15.1:0/1170742610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa10004bb60 0x7fa1000483c0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fa10000e9f0 tx=0x7fa0f802b860 comp rx=0 tx=0).stop 2024-03-06T07:26:55.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.348+0000 7fa10ecfa640 1 -- 172.21.15.1:0/1170742610 shutdown_connections 2024-03-06T07:26:55.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.348+0000 7fa10ecfa640 1 --2- 172.21.15.1:0/1170742610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa100048900 0x7fa10004a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.348+0000 7fa10ecfa640 1 --2- 172.21.15.1:0/1170742610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa10004bb60 0x7fa1000483c0 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.348+0000 7fa10ecfa640 1 --2- 172.21.15.1:0/1170742610 >> v2:172.21.15.98:3300/0 conn(0x7fa1000d5840 0x7fa1000dc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.348+0000 7fa10ecfa640 1 -- 172.21.15.1:0/1170742610 >> 172.21.15.1:0/1170742610 conn(0x7fa10002f590 msgr2=0x7fa10002f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.349+0000 7fa10ecfa640 1 -- 172.21.15.1:0/1170742610 shutdown_connections 2024-03-06T07:26:55.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.349+0000 7fa10ecfa640 1 -- 172.21.15.1:0/1170742610 wait complete. 2024-03-06T07:26:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.350+0000 7fa10ecfa640 1 Processor -- start 2024-03-06T07:26:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.350+0000 7fa10ecfa640 1 -- start start 2024-03-06T07:26:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.350+0000 7fa10ecfa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa100048900 0x7fa100008520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.350+0000 7fa10ecfa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa10004bb60 0x7fa1000440b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.350+0000 7fa10ecfa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1000d5840 0x7fa100009330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.350+0000 7fa10ecfa640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1000d8550 con 0x7fa100048900 2024-03-06T07:26:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.350+0000 7fa10ecfa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1000d8ba0 con 0x7fa10004bb60 2024-03-06T07:26:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.350+0000 7fa10ecfa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1000d8830 con 0x7fa1000d5840 2024-03-06T07:26:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.350+0000 7fa10dcf8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa100048900 0x7fa100008520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.350+0000 7fa10dcf8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa100048900 0x7fa100008520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45066/0 (socket says 172.21.15.1:45066) 2024-03-06T07:26:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.350+0000 7fa10dcf8640 1 -- 172.21.15.1:0/1079909833 learned_addr learned my addr 172.21.15.1:0/1079909833 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.350+0000 7fa10e4f9640 1 --2- 172.21.15.1:0/1079909833 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1000d5840 0x7fa100009330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.351+0000 7fa10d4f7640 1 --2- 172.21.15.1:0/1079909833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa10004bb60 0x7fa1000440b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.351+0000 7fa10dcf8640 1 -- 172.21.15.1:0/1079909833 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1000d5840 msgr2=0x7fa100009330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.351+0000 7fa10dcf8640 1 --2- 172.21.15.1:0/1079909833 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1000d5840 0x7fa100009330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.351+0000 7fa10dcf8640 1 -- 172.21.15.1:0/1079909833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa10004bb60 msgr2=0x7fa1000440b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.351+0000 7fa10dcf8640 1 --2- 172.21.15.1:0/1079909833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa10004bb60 0x7fa1000440b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.351+0000 7fa10dcf8640 1 -- 172.21.15.1:0/1079909833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1000d0fa0 con 0x7fa100048900 2024-03-06T07:26:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.351+0000 7fa10e4f9640 1 --2- 172.21.15.1:0/1079909833 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1000d5840 0x7fa100009330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.351+0000 7fa10dcf8640 1 --2- 172.21.15.1:0/1079909833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa100048900 0x7fa100008520 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fa10400d800 tx=0x7fa10400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.351+0000 7fa0fe7fc640 1 -- 172.21.15.1:0/1079909833 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10400b6c0 con 0x7fa100048900 2024-03-06T07:26:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.351+0000 7fa0fe7fc640 1 -- 172.21.15.1:0/1079909833 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa104019070 con 0x7fa100048900 2024-03-06T07:26:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.351+0000 7fa10ecfa640 1 -- 172.21.15.1:0/1079909833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1000d1280 con 0x7fa100048900 2024-03-06T07:26:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.351+0000 7fa0fe7fc640 1 -- 172.21.15.1:0/1079909833 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10400ba80 con 0x7fa100048900 2024-03-06T07:26:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.351+0000 7fa10ecfa640 1 -- 172.21.15.1:0/1079909833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1000cc5a0 con 0x7fa100048900 2024-03-06T07:26:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.352+0000 7fa10ecfa640 1 -- 172.21.15.1:0/1079909833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa100005710 con 0x7fa100048900 2024-03-06T07:26:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.352+0000 7fa0fe7fc640 1 -- 172.21.15.1:0/1079909833 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa10400bc20 con 0x7fa100048900 2024-03-06T07:26:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.353+0000 7fa0fe7fc640 1 --2- 172.21.15.1:0/1079909833 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0ec06b4d0 0x7fa0ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.353+0000 7fa0fe7fc640 1 -- 172.21.15.1:0/1079909833 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fa10408a6c0 con 0x7fa100048900 2024-03-06T07:26:55.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.353+0000 7fa10d4f7640 1 --2- 172.21.15.1:0/1079909833 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0ec06b4d0 0x7fa0ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.354+0000 7fa10d4f7640 1 --2- 172.21.15.1:0/1079909833 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0ec06b4d0 0x7fa0ec06d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8036000 tx=0x7fa0f803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.357+0000 7fa0fe7fc640 1 -- 172.21.15.1:0/1079909833 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa104059560 con 0x7fa100048900 2024-03-06T07:26:55.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.536+0000 7fa10ecfa640 1 -- 172.21.15.1:0/1079909833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa10004c9b0 con 0x7fa100048900 2024-03-06T07:26:55.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.539+0000 7fa0fe7fc640 1 -- 172.21.15.1:0/1079909833 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v54) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa104058e60 con 0x7fa100048900 2024-03-06T07:26:55.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.541+0000 7fa10ecfa640 1 -- 172.21.15.1:0/1079909833 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0ec06b4d0 msgr2=0x7fa0ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.541+0000 7fa10ecfa640 1 --2- 172.21.15.1:0/1079909833 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0ec06b4d0 0x7fa0ec06d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8036000 tx=0x7fa0f803d040 comp rx=0 tx=0).stop 2024-03-06T07:26:55.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.541+0000 7fa10ecfa640 1 -- 172.21.15.1:0/1079909833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa100048900 msgr2=0x7fa100008520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.541+0000 7fa10ecfa640 1 --2- 172.21.15.1:0/1079909833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa100048900 0x7fa100008520 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fa10400d800 tx=0x7fa10400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.541+0000 7fa10ecfa640 1 -- 172.21.15.1:0/1079909833 shutdown_connections 2024-03-06T07:26:55.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.541+0000 7fa10ecfa640 1 --2- 172.21.15.1:0/1079909833 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0ec06b4d0 0x7fa0ec06d990 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.541+0000 7fa10ecfa640 1 --2- 172.21.15.1:0/1079909833 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1000d5840 0x7fa100009330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.541+0000 7fa10ecfa640 1 --2- 172.21.15.1:0/1079909833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa10004bb60 0x7fa1000440b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.541+0000 7fa10ecfa640 1 --2- 172.21.15.1:0/1079909833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa100048900 0x7fa100008520 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.541+0000 7fa10ecfa640 1 -- 172.21.15.1:0/1079909833 >> 172.21.15.1:0/1079909833 conn(0x7fa10002f590 msgr2=0x7fa10004cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.541+0000 7fa10ecfa640 1 -- 172.21.15.1:0/1079909833 shutdown_connections 2024-03-06T07:26:55.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.541+0000 7fa10ecfa640 1 -- 172.21.15.1:0/1079909833 wait complete. 2024-03-06T07:26:55.554 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-03-06T07:26:55.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.655+0000 7f2ca620e640 1 Processor -- start 2024-03-06T07:26:55.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.655+0000 7f2ca620e640 1 -- start start 2024-03-06T07:26:55.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.656+0000 7f2ca620e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2ca00c2b30 0x7f2ca00724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.656+0000 7f2ca620e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ca00a5fa0 0x7f2ca0072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.656+0000 7f2ca620e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca0151dd0 con 0x7f2ca0152b70 2024-03-06T07:26:55.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.656+0000 7f2ca620e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca01526e0 con 0x7f2ca00c2b30 2024-03-06T07:26:55.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.656+0000 7f2ca620e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca0156190 con 0x7f2ca00a5fa0 2024-03-06T07:26:55.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.656+0000 7f2c9f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2ca00c2b30 0x7f2ca00724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.656+0000 7f2c9f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2ca00c2b30 0x7f2ca00724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45082/0 (socket says 172.21.15.1:45082) 2024-03-06T07:26:55.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.656+0000 7f2c9f7fe640 1 -- 172.21.15.1:0/1614671419 learned_addr learned my addr 172.21.15.1:0/1614671419 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:55.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.656+0000 7f2c9ffff640 1 --2- 172.21.15.1:0/1614671419 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ca00a5fa0 0x7f2ca0072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.656+0000 7f2c9f7fe640 1 -- 172.21.15.1:0/1614671419 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ca00a5fa0 msgr2=0x7f2ca0072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.656+0000 7f2c9f7fe640 1 --2- 172.21.15.1:0/1614671419 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ca00a5fa0 0x7f2ca0072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.656+0000 7f2c9f7fe640 1 -- 172.21.15.1:0/1614671419 >> v1:172.21.15.98:6789/0 conn(0x7f2ca0152b70 legacy=0x7f2ca00c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.656+0000 7f2c9f7fe640 1 -- 172.21.15.1:0/1614671419 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca0152090 con 0x7f2ca00c2b30 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.656+0000 7f2c9ffff640 1 --2- 172.21.15.1:0/1614671419 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ca00a5fa0 0x7f2ca0072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.657+0000 7f2c9f7fe640 1 --2- 172.21.15.1:0/1614671419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca00c2b30 0x7f2ca00724c0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f2c90009870 tx=0x7f2c9002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d2f2f0a14b86550 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.657+0000 7f2c9e7fc640 1 -- 172.21.15.1:0/1614671419 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c90035020 con 0x7f2ca00c2b30 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.657+0000 7f2c9e7fc640 1 -- 172.21.15.1:0/1614671419 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c90031640 con 0x7f2ca00c2b30 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.657+0000 7f2c9e7fc640 1 -- 172.21.15.1:0/1614671419 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9002d040 con 0x7f2ca00c2b30 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.658+0000 7f2ca620e640 1 -- 172.21.15.1:0/1614671419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca00c2b30 msgr2=0x7f2ca00724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.658+0000 7f2ca620e640 1 --2- 172.21.15.1:0/1614671419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca00c2b30 0x7f2ca00724c0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f2c90009870 tx=0x7f2c9002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.658+0000 7f2ca620e640 1 -- 172.21.15.1:0/1614671419 shutdown_connections 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.658+0000 7f2ca620e640 1 --2- 172.21.15.1:0/1614671419 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ca00a5fa0 0x7f2ca0072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.658+0000 7f2ca620e640 1 --2- 172.21.15.1:0/1614671419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca00c2b30 0x7f2ca00724c0 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.658+0000 7f2ca620e640 1 -- 172.21.15.1:0/1614671419 >> 172.21.15.1:0/1614671419 conn(0x7f2ca00c6860 msgr2=0x7f2ca00c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.658+0000 7f2ca620e640 1 -- 172.21.15.1:0/1614671419 shutdown_connections 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.658+0000 7f2ca620e640 1 -- 172.21.15.1:0/1614671419 wait complete. 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.659+0000 7f2ca620e640 1 Processor -- start 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.659+0000 7f2ca620e640 1 -- start start 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2ca620e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ca00a5fa0 0x7f2ca00bd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2ca620e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca00c2b30 0x7f2ca0148cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2ca620e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ca0152b70 0x7f2ca0149210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2ca620e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca0155f00 con 0x7f2ca00c2b30 2024-03-06T07:26:55.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2ca620e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca0155d80 con 0x7f2ca00a5fa0 2024-03-06T07:26:55.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2ca620e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca00c3a00 con 0x7f2ca0152b70 2024-03-06T07:26:55.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2c9effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca00c2b30 0x7f2ca0148cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2c9effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca00c2b30 0x7f2ca0148cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45088/0 (socket says 172.21.15.1:45088) 2024-03-06T07:26:55.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2c9effd640 1 -- 172.21.15.1:0/3180311524 learned_addr learned my addr 172.21.15.1:0/3180311524 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:55.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2c9f7fe640 1 --2- 172.21.15.1:0/3180311524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ca00a5fa0 0x7f2ca00bd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2c9ffff640 1 --2- 172.21.15.1:0/3180311524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ca0152b70 0x7f2ca0149210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2c9effd640 1 -- 172.21.15.1:0/3180311524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ca0152b70 msgr2=0x7f2ca0149210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2c9effd640 1 --2- 172.21.15.1:0/3180311524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ca0152b70 0x7f2ca0149210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2c9effd640 1 -- 172.21.15.1:0/3180311524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ca00a5fa0 msgr2=0x7f2ca00bd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2c9effd640 1 --2- 172.21.15.1:0/3180311524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ca00a5fa0 0x7f2ca00bd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2c9effd640 1 -- 172.21.15.1:0/3180311524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca0149750 con 0x7f2ca00c2b30 2024-03-06T07:26:55.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2c9effd640 1 --2- 172.21.15.1:0/3180311524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca00c2b30 0x7f2ca0148cd0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f2c9400b420 tx=0x7f2c9400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2c87fff640 1 -- 172.21.15.1:0/3180311524 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c94011830 con 0x7f2ca00c2b30 2024-03-06T07:26:55.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2c87fff640 1 -- 172.21.15.1:0/3180311524 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c94016070 con 0x7f2ca00c2b30 2024-03-06T07:26:55.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2c87fff640 1 -- 172.21.15.1:0/3180311524 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c94011c10 con 0x7f2ca00c2b30 2024-03-06T07:26:55.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2ca620e640 1 -- 172.21.15.1:0/3180311524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca00bbe40 con 0x7f2ca00c2b30 2024-03-06T07:26:55.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.660+0000 7f2ca620e640 1 -- 172.21.15.1:0/3180311524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca00bc400 con 0x7f2ca00c2b30 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.661+0000 7f2ca620e640 1 -- 172.21.15.1:0/3180311524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca0067fd0 con 0x7f2ca00c2b30 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.662+0000 7f2c87fff640 1 -- 172.21.15.1:0/3180311524 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c9401b460 con 0x7f2ca00c2b30 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.662+0000 7f2c87fff640 1 --2- 172.21.15.1:0/3180311524 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c8c06b4d0 0x7f2c8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.662+0000 7f2c87fff640 1 -- 172.21.15.1:0/3180311524 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f2c94089b60 con 0x7f2ca00c2b30 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.662+0000 7f2c9f7fe640 1 --2- 172.21.15.1:0/3180311524 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c8c06b4d0 0x7f2c8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.663+0000 7f2c9f7fe640 1 --2- 172.21.15.1:0/3180311524 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c8c06b4d0 0x7f2c8c06d990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f2c90036000 tx=0x7f2c9003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.667+0000 7f2c87fff640 1 -- 172.21.15.1:0/3180311524 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c94020020 con 0x7f2ca00c2b30 2024-03-06T07:26:55.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.847+0000 7f2ca620e640 1 -- 172.21.15.1:0/3180311524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7f2ca0071790 con 0x7f2ca00c2b30 2024-03-06T07:26:55.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.850+0000 7f2c87fff640 1 -- 172.21.15.1:0/3180311524 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v55) v1 ==== 96+0+0 (secure 0 0 0) 0x7f2c940583b0 con 0x7f2ca00c2b30 2024-03-06T07:26:55.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.852+0000 7f2ca620e640 1 -- 172.21.15.1:0/3180311524 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c8c06b4d0 msgr2=0x7f2c8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.852+0000 7f2ca620e640 1 --2- 172.21.15.1:0/3180311524 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c8c06b4d0 0x7f2c8c06d990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f2c90036000 tx=0x7f2c9003c040 comp rx=0 tx=0).stop 2024-03-06T07:26:55.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.852+0000 7f2ca620e640 1 -- 172.21.15.1:0/3180311524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca00c2b30 msgr2=0x7f2ca0148cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.852+0000 7f2ca620e640 1 --2- 172.21.15.1:0/3180311524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca00c2b30 0x7f2ca0148cd0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f2c9400b420 tx=0x7f2c9400b8e0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.852+0000 7f2ca620e640 1 -- 172.21.15.1:0/3180311524 shutdown_connections 2024-03-06T07:26:55.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.852+0000 7f2ca620e640 1 --2- 172.21.15.1:0/3180311524 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c8c06b4d0 0x7f2c8c06d990 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.852+0000 7f2ca620e640 1 --2- 172.21.15.1:0/3180311524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ca0152b70 0x7f2ca0149210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.852+0000 7f2ca620e640 1 --2- 172.21.15.1:0/3180311524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca00c2b30 0x7f2ca0148cd0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.852+0000 7f2ca620e640 1 --2- 172.21.15.1:0/3180311524 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ca00a5fa0 0x7f2ca00bd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.852+0000 7f2ca620e640 1 -- 172.21.15.1:0/3180311524 >> 172.21.15.1:0/3180311524 conn(0x7f2ca00c6860 msgr2=0x7f2ca0150190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.852+0000 7f2ca620e640 1 -- 172.21.15.1:0/3180311524 shutdown_connections 2024-03-06T07:26:55.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.852+0000 7f2ca620e640 1 -- 172.21.15.1:0/3180311524 wait complete. 2024-03-06T07:26:55.865 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-03-06T07:26:55.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.966+0000 7fb33ced6640 1 Processor -- start 2024-03-06T07:26:55.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.966+0000 7fb33ced6640 1 -- start start 2024-03-06T07:26:55.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.966+0000 7fb33ced6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb3380c2b30 0x7fb3380724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.966+0000 7fb33ced6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb338152b70 0x7fb338072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.966+0000 7fb33ced6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb338151dd0 con 0x7fb3380a5fa0 2024-03-06T07:26:55.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.966+0000 7fb33ced6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3381526e0 con 0x7fb338152b70 2024-03-06T07:26:55.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.966+0000 7fb33ced6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb338156190 con 0x7fb3380c2b30 2024-03-06T07:26:55.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.967+0000 7fb335d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb338152b70 0x7fb338072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.967+0000 7fb336d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb3380a5fa0 0x7fb3380c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36750/0 (socket says 172.21.15.1:36750) 2024-03-06T07:26:55.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.967+0000 7fb336d9d640 1 -- 172.21.15.1:0/1835213169 learned_addr learned my addr 172.21.15.1:0/1835213169 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:55.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.967+0000 7fb33659c640 1 --2- 172.21.15.1:0/1835213169 >> v2:172.21.15.98:3300/0 conn(0x7fb3380c2b30 0x7fb3380724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.967+0000 7fb335d9b640 1 -- 172.21.15.1:0/1835213169 >> v2:172.21.15.98:3300/0 conn(0x7fb3380c2b30 msgr2=0x7fb3380724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.967+0000 7fb335d9b640 1 --2- 172.21.15.1:0/1835213169 >> v2:172.21.15.98:3300/0 conn(0x7fb3380c2b30 0x7fb3380724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.967+0000 7fb335d9b640 1 -- 172.21.15.1:0/1835213169 >> v1:172.21.15.1:6789/0 conn(0x7fb3380a5fa0 legacy=0x7fb3380c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.967+0000 7fb33559a640 1 -- 172.21.15.1:0/1835213169 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888639348 0 0) 0x7fb338151dd0 con 0x7fb3380a5fa0 2024-03-06T07:26:55.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.967+0000 7fb335d9b640 1 -- 172.21.15.1:0/1835213169 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb338152090 con 0x7fb338152b70 2024-03-06T07:26:55.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.968+0000 7fb335d9b640 1 --2- 172.21.15.1:0/1835213169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb338152b70 0x7fb338072a00 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fb3380c6570 tx=0x7fb32c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef4ce9b387fba0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.968+0000 7fb33559a640 1 -- 172.21.15.1:0/1835213169 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32c035020 con 0x7fb338152b70 2024-03-06T07:26:55.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.968+0000 7fb33559a640 1 -- 172.21.15.1:0/1835213169 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32c031830 con 0x7fb338152b70 2024-03-06T07:26:55.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.968+0000 7fb33559a640 1 -- 172.21.15.1:0/1835213169 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32c031c00 con 0x7fb338152b70 2024-03-06T07:26:55.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.968+0000 7fb33ced6640 1 -- 172.21.15.1:0/1835213169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb338152b70 msgr2=0x7fb338072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.968+0000 7fb33ced6640 1 --2- 172.21.15.1:0/1835213169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb338152b70 0x7fb338072a00 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fb3380c6570 tx=0x7fb32c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.968+0000 7fb33ced6640 1 -- 172.21.15.1:0/1835213169 shutdown_connections 2024-03-06T07:26:55.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.968+0000 7fb33ced6640 1 --2- 172.21.15.1:0/1835213169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb338152b70 0x7fb338072a00 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.969+0000 7fb33ced6640 1 --2- 172.21.15.1:0/1835213169 >> v2:172.21.15.98:3300/0 conn(0x7fb3380c2b30 0x7fb3380724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.969+0000 7fb33ced6640 1 -- 172.21.15.1:0/1835213169 >> 172.21.15.1:0/1835213169 conn(0x7fb3380c6860 msgr2=0x7fb3380c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.969+0000 7fb33ced6640 1 -- 172.21.15.1:0/1835213169 shutdown_connections 2024-03-06T07:26:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.969+0000 7fb33ced6640 1 -- 172.21.15.1:0/1835213169 wait complete. 2024-03-06T07:26:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.970+0000 7fb33ced6640 1 Processor -- start 2024-03-06T07:26:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.970+0000 7fb33ced6640 1 -- start start 2024-03-06T07:26:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.970+0000 7fb33ced6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3380a5fa0 0x7fb33814cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.970+0000 7fb33ced6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3380c2b30 0x7fb3380ab1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.970+0000 7fb33ced6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb338152b70 0x7fb3380ab730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.970+0000 7fb33ced6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb338155f00 con 0x7fb3380a5fa0 2024-03-06T07:26:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.970+0000 7fb33ced6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb338155d80 con 0x7fb3380c2b30 2024-03-06T07:26:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.970+0000 7fb33ced6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3380c3a00 con 0x7fb338152b70 2024-03-06T07:26:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.971+0000 7fb33659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3380a5fa0 0x7fb33814cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.971+0000 7fb33659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3380a5fa0 0x7fb33814cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45096/0 (socket says 172.21.15.1:45096) 2024-03-06T07:26:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.971+0000 7fb33659c640 1 -- 172.21.15.1:0/2797655248 learned_addr learned my addr 172.21.15.1:0/2797655248 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.971+0000 7fb335d9b640 1 --2- 172.21.15.1:0/2797655248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3380c2b30 0x7fb3380ab1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.971+0000 7fb336d9d640 1 --2- 172.21.15.1:0/2797655248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb338152b70 0x7fb3380ab730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.971+0000 7fb33659c640 1 -- 172.21.15.1:0/2797655248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb338152b70 msgr2=0x7fb3380ab730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.971+0000 7fb33659c640 1 --2- 172.21.15.1:0/2797655248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb338152b70 0x7fb3380ab730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.971+0000 7fb33659c640 1 -- 172.21.15.1:0/2797655248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3380c2b30 msgr2=0x7fb3380ab1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.971+0000 7fb33659c640 1 --2- 172.21.15.1:0/2797655248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3380c2b30 0x7fb3380ab1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.971+0000 7fb33659c640 1 -- 172.21.15.1:0/2797655248 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3380aaa30 con 0x7fb3380a5fa0 2024-03-06T07:26:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.971+0000 7fb33659c640 1 --2- 172.21.15.1:0/2797655248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3380a5fa0 0x7fb33814cfe0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fb32800d800 tx=0x7fb32800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.971+0000 7fb326ffd640 1 -- 172.21.15.1:0/2797655248 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32800b6c0 con 0x7fb3380a5fa0 2024-03-06T07:26:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.971+0000 7fb326ffd640 1 -- 172.21.15.1:0/2797655248 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb328019070 con 0x7fb3380a5fa0 2024-03-06T07:26:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.971+0000 7fb33ced6640 1 -- 172.21.15.1:0/2797655248 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3381eef30 con 0x7fb3380a5fa0 2024-03-06T07:26:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.971+0000 7fb326ffd640 1 -- 172.21.15.1:0/2797655248 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32800ba80 con 0x7fb3380a5fa0 2024-03-06T07:26:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.971+0000 7fb33ced6640 1 -- 172.21.15.1:0/2797655248 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3381ef210 con 0x7fb3380a5fa0 2024-03-06T07:26:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.972+0000 7fb33ced6640 1 -- 172.21.15.1:0/2797655248 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb338067fd0 con 0x7fb3380a5fa0 2024-03-06T07:26:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.973+0000 7fb326ffd640 1 -- 172.21.15.1:0/2797655248 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb32800bc20 con 0x7fb3380a5fa0 2024-03-06T07:26:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.973+0000 7fb326ffd640 1 --2- 172.21.15.1:0/2797655248 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb31806b4d0 0x7fb31806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.973+0000 7fb326ffd640 1 -- 172.21.15.1:0/2797655248 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fb32808a520 con 0x7fb3380a5fa0 2024-03-06T07:26:55.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.973+0000 7fb335d9b640 1 --2- 172.21.15.1:0/2797655248 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb31806b4d0 0x7fb31806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.974+0000 7fb335d9b640 1 --2- 172.21.15.1:0/2797655248 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb31806b4d0 0x7fb31806d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fb32c036000 tx=0x7fb32c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:55.978+0000 7fb326ffd640 1 -- 172.21.15.1:0/2797655248 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb328058d70 con 0x7fb3380a5fa0 2024-03-06T07:26:56.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.159+0000 7fb33ced6640 1 -- 172.21.15.1:0/2797655248 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7fb338071790 con 0x7fb3380a5fa0 2024-03-06T07:26:56.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.162+0000 7fb326ffd640 1 -- 172.21.15.1:0/2797655248 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v56) v1 ==== 96+0+0 (secure 0 0 0) 0x7fb32805f0c0 con 0x7fb3380a5fa0 2024-03-06T07:26:56.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.164+0000 7fb33ced6640 1 -- 172.21.15.1:0/2797655248 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb31806b4d0 msgr2=0x7fb31806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.164+0000 7fb33ced6640 1 --2- 172.21.15.1:0/2797655248 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb31806b4d0 0x7fb31806d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fb32c036000 tx=0x7fb32c008030 comp rx=0 tx=0).stop 2024-03-06T07:26:56.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.164+0000 7fb33ced6640 1 -- 172.21.15.1:0/2797655248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3380a5fa0 msgr2=0x7fb33814cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.164+0000 7fb33ced6640 1 --2- 172.21.15.1:0/2797655248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3380a5fa0 0x7fb33814cfe0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fb32800d800 tx=0x7fb32800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.164+0000 7fb33ced6640 1 -- 172.21.15.1:0/2797655248 shutdown_connections 2024-03-06T07:26:56.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.164+0000 7fb33ced6640 1 --2- 172.21.15.1:0/2797655248 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb31806b4d0 0x7fb31806d990 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.164+0000 7fb33ced6640 1 --2- 172.21.15.1:0/2797655248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb338152b70 0x7fb3380ab730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.164+0000 7fb33ced6640 1 --2- 172.21.15.1:0/2797655248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3380c2b30 0x7fb3380ab1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.164+0000 7fb33ced6640 1 --2- 172.21.15.1:0/2797655248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3380a5fa0 0x7fb33814cfe0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.164+0000 7fb33ced6640 1 -- 172.21.15.1:0/2797655248 >> 172.21.15.1:0/2797655248 conn(0x7fb3380c6860 msgr2=0x7fb338153d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.164+0000 7fb33ced6640 1 -- 172.21.15.1:0/2797655248 shutdown_connections 2024-03-06T07:26:56.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.164+0000 7fb33ced6640 1 -- 172.21.15.1:0/2797655248 wait complete. 2024-03-06T07:26:56.177 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla' 2024-03-06T07:26:56.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.276+0000 7f90b7ec2640 1 Processor -- start 2024-03-06T07:26:56.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.276+0000 7f90b7ec2640 1 -- start start 2024-03-06T07:26:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.277+0000 7f90b7ec2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f90b0152650 0x7f90b0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.277+0000 7f90b7ec2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90b00c6b10 0x7f90b0158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.277+0000 7f90b7ec2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f90b00bf9d0 0x7f90b00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.277+0000 7f90b7ec2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b0155800 con 0x7f90b0152650 2024-03-06T07:26:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.277+0000 7f90b7ec2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b00c2fc0 con 0x7f90b00bf9d0 2024-03-06T07:26:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.277+0000 7f90b7ec2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b00c6690 con 0x7f90b00c6b10 2024-03-06T07:26:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.277+0000 7f90b5c5e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f90b0152650 0x7f90b0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.277+0000 7f90b5c5e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f90b0152650 0x7f90b0155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45098/0 (socket says 172.21.15.1:45098) 2024-03-06T07:26:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.277+0000 7f90b5c5e640 1 -- 172.21.15.1:0/2344859532 learned_addr learned my addr 172.21.15.1:0/2344859532 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.278+0000 7f90b5c5e640 1 -- 172.21.15.1:0/2344859532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90b00c6b10 msgr2=0x7f90b0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.278+0000 7f90b5c5e640 1 --2- 172.21.15.1:0/2344859532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90b00c6b10 0x7f90b0158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.278+0000 7f90b5c5e640 1 -- 172.21.15.1:0/2344859532 >> v2:172.21.15.98:3300/0 conn(0x7f90b00bf9d0 msgr2=0x7f90b00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.278+0000 7f90b645f640 1 --2- 172.21.15.1:0/2344859532 >> v2:172.21.15.98:3300/0 conn(0x7f90b00bf9d0 0x7f90b00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.278+0000 7f90b5c5e640 1 --2- 172.21.15.1:0/2344859532 >> v2:172.21.15.98:3300/0 conn(0x7f90b00bf9d0 0x7f90b00c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.278+0000 7f90b5c5e640 1 -- 172.21.15.1:0/2344859532 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b00c7280 con 0x7f90b0152650 2024-03-06T07:26:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.278+0000 7f90b5c5e640 1 --2- 172.21.15.1:0/2344859532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b0152650 0x7f90b0155c90 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f90b01597b0 tx=0x7f90ac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0427c5378e0178 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.279+0000 7f90b4c5c640 1 -- 172.21.15.1:0/2344859532 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac02e070 con 0x7f90b0152650 2024-03-06T07:26:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.279+0000 7f90b4c5c640 1 -- 172.21.15.1:0/2344859532 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac037070 con 0x7f90b0152650 2024-03-06T07:26:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.279+0000 7f90b4c5c640 1 -- 172.21.15.1:0/2344859532 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac031a00 con 0x7f90b0152650 2024-03-06T07:26:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.279+0000 7f90b7ec2640 1 -- 172.21.15.1:0/2344859532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b0152650 msgr2=0x7f90b0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.279+0000 7f90b7ec2640 1 --2- 172.21.15.1:0/2344859532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b0152650 0x7f90b0155c90 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f90b01597b0 tx=0x7f90ac02b880 comp rx=0 tx=0).stop 2024-03-06T07:26:56.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.279+0000 7f90b7ec2640 1 -- 172.21.15.1:0/2344859532 shutdown_connections 2024-03-06T07:26:56.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.279+0000 7f90b7ec2640 1 --2- 172.21.15.1:0/2344859532 >> v2:172.21.15.98:3300/0 conn(0x7f90b00bf9d0 0x7f90b00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.279+0000 7f90b7ec2640 1 --2- 172.21.15.1:0/2344859532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90b00c6b10 0x7f90b0158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.279+0000 7f90b7ec2640 1 --2- 172.21.15.1:0/2344859532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b0152650 0x7f90b0155c90 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.279+0000 7f90b7ec2640 1 -- 172.21.15.1:0/2344859532 >> 172.21.15.1:0/2344859532 conn(0x7f90b0066ee0 msgr2=0x7f90b00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.279+0000 7f90b7ec2640 1 -- 172.21.15.1:0/2344859532 shutdown_connections 2024-03-06T07:26:56.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.279+0000 7f90b7ec2640 1 -- 172.21.15.1:0/2344859532 wait complete. 2024-03-06T07:26:56.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.280+0000 7f90b7ec2640 1 Processor -- start 2024-03-06T07:26:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.280+0000 7f90b7ec2640 1 -- start start 2024-03-06T07:26:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b7ec2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b00bf9d0 0x7f90b01ef000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b7ec2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b0152650 0x7f90b00c5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b7ec2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90b01eb390 0x7f90b01eb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b7ec2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b0155490 con 0x7f90b00bf9d0 2024-03-06T07:26:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b7ec2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b01551b0 con 0x7f90b0152650 2024-03-06T07:26:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b7ec2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b0152a30 con 0x7f90b01eb390 2024-03-06T07:26:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b5c5e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b00bf9d0 0x7f90b01ef000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b5c5e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b00bf9d0 0x7f90b01ef000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45110/0 (socket says 172.21.15.1:45110) 2024-03-06T07:26:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b5c5e640 1 -- 172.21.15.1:0/1385820365 learned_addr learned my addr 172.21.15.1:0/1385820365 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b545d640 1 --2- 172.21.15.1:0/1385820365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b0152650 0x7f90b00c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b645f640 1 --2- 172.21.15.1:0/1385820365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90b01eb390 0x7f90b01eb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b5c5e640 1 -- 172.21.15.1:0/1385820365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90b01eb390 msgr2=0x7f90b01eb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b5c5e640 1 --2- 172.21.15.1:0/1385820365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90b01eb390 0x7f90b01eb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b5c5e640 1 -- 172.21.15.1:0/1385820365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b0152650 msgr2=0x7f90b00c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b5c5e640 1 --2- 172.21.15.1:0/1385820365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b0152650 0x7f90b00c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b5c5e640 1 -- 172.21.15.1:0/1385820365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b00ca510 con 0x7f90b00bf9d0 2024-03-06T07:26:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b545d640 1 --2- 172.21.15.1:0/1385820365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b0152650 0x7f90b00c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.281+0000 7f90b5c5e640 1 --2- 172.21.15.1:0/1385820365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b00bf9d0 0x7f90b01ef000 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f90ac02b8c0 tx=0x7f90ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.282+0000 7f90a67fc640 1 -- 172.21.15.1:0/1385820365 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac032880 con 0x7f90b00bf9d0 2024-03-06T07:26:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.282+0000 7f90a67fc640 1 -- 172.21.15.1:0/1385820365 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac032a20 con 0x7f90b00bf9d0 2024-03-06T07:26:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.282+0000 7f90b7ec2640 1 -- 172.21.15.1:0/1385820365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b0149f90 con 0x7f90b00bf9d0 2024-03-06T07:26:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.282+0000 7f90a67fc640 1 -- 172.21.15.1:0/1385820365 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac032e10 con 0x7f90b00bf9d0 2024-03-06T07:26:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.282+0000 7f90b7ec2640 1 -- 172.21.15.1:0/1385820365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b00c5a60 con 0x7f90b00bf9d0 2024-03-06T07:26:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.283+0000 7f90b7ec2640 1 -- 172.21.15.1:0/1385820365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b0067fd0 con 0x7f90b00bf9d0 2024-03-06T07:26:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.283+0000 7f90a67fc640 1 -- 172.21.15.1:0/1385820365 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90ac0453f0 con 0x7f90b00bf9d0 2024-03-06T07:26:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.283+0000 7f90a67fc640 1 --2- 172.21.15.1:0/1385820365 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f909406b4d0 0x7f909406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.284+0000 7f90a67fc640 1 -- 172.21.15.1:0/1385820365 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f90ac0ac0c0 con 0x7f90b00bf9d0 2024-03-06T07:26:56.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.284+0000 7f90b545d640 1 --2- 172.21.15.1:0/1385820365 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f909406b4d0 0x7f909406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.284+0000 7f90b545d640 1 --2- 172.21.15.1:0/1385820365 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f909406b4d0 0x7f909406d990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f90b014f920 tx=0x7f90a0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.288+0000 7f90a67fc640 1 -- 172.21.15.1:0/1385820365 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90ac040070 con 0x7f90b00bf9d0 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:56.539+0000 7f90b7ec2640 1 -- 172.21.15.1:0/1385820365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]} v 0) v1 -- 0x7f90b01eb0f0 con 0x7f90b00bf9d0 2024-03-06T07:26:57.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.218+0000 7f90a67fc640 1 -- 172.21.15.1:0/1385820365 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]}]=0 v383) v1 ==== 146+0+0 (secure 0 0 0) 0x7f90ac07b110 con 0x7f90b00bf9d0 2024-03-06T07:26:57.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.221+0000 7f90b7ec2640 1 -- 172.21.15.1:0/1385820365 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f909406b4d0 msgr2=0x7f909406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.221+0000 7f90b7ec2640 1 --2- 172.21.15.1:0/1385820365 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f909406b4d0 0x7f909406d990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f90b014f920 tx=0x7f90a0008040 comp rx=0 tx=0).stop 2024-03-06T07:26:57.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.221+0000 7f90b7ec2640 1 -- 172.21.15.1:0/1385820365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b00bf9d0 msgr2=0x7f90b01ef000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.221+0000 7f90b7ec2640 1 --2- 172.21.15.1:0/1385820365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b00bf9d0 0x7f90b01ef000 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f90ac02b8c0 tx=0x7f90ac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.221+0000 7f90b7ec2640 1 -- 172.21.15.1:0/1385820365 shutdown_connections 2024-03-06T07:26:57.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.221+0000 7f90b7ec2640 1 --2- 172.21.15.1:0/1385820365 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f909406b4d0 0x7f909406d990 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.221+0000 7f90b7ec2640 1 --2- 172.21.15.1:0/1385820365 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90b01eb390 0x7f90b01eb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.221+0000 7f90b7ec2640 1 --2- 172.21.15.1:0/1385820365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b0152650 0x7f90b00c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.221+0000 7f90b7ec2640 1 --2- 172.21.15.1:0/1385820365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b00bf9d0 0x7f90b01ef000 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.221+0000 7f90b7ec2640 1 -- 172.21.15.1:0/1385820365 >> 172.21.15.1:0/1385820365 conn(0x7f90b0066ee0 msgr2=0x7f90b00c4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.221+0000 7f90b7ec2640 1 -- 172.21.15.1:0/1385820365 shutdown_connections 2024-03-06T07:26:57.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.221+0000 7f90b7ec2640 1 -- 172.21.15.1:0/1385820365 wait complete. 2024-03-06T07:26:57.236 INFO:tasks.cephfs_test_runner:That a vanilla standby is preferred over others with mds_join_fs set to another fs. ... ok 2024-03-06T07:26:57.236 INFO:tasks.cephfs_test_runner:Starting test: test_down (tasks.cephfs.test_failover.TestClusterResize) That down setting toggles and sets max_mds appropriately. 2024-03-06T07:26:57.237 INFO:tasks.cephfs_test_runner:test_down (tasks.cephfs.test_failover.TestClusterResize) 2024-03-06T07:26:57.237 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:57.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.342+0000 7f6b61f02640 1 Processor -- start 2024-03-06T07:26:57.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.342+0000 7f6b61f02640 1 -- start start 2024-03-06T07:26:57.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.343+0000 7f6b61f02640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6b5c152650 0x7f6b5c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.343+0000 7f6b61f02640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6b5c0bf9d0 0x7f6b5c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.343+0000 7f6b61f02640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5c0c2fc0 con 0x7f6b5c0c6b10 2024-03-06T07:26:57.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.343+0000 7f6b61f02640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b5c155800 con 0x7f6b5c0bf9d0 2024-03-06T07:26:57.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.343+0000 7f6b61f02640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b5c158da0 con 0x7f6b5c152650 2024-03-06T07:26:57.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.343+0000 7f6b5bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6b5c0bf9d0 0x7f6b5c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.343+0000 7f6b5bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6b5c0bf9d0 0x7f6b5c0c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45120/0 (socket says 172.21.15.1:45120) 2024-03-06T07:26:57.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.343+0000 7f6b5bfff640 1 -- 172.21.15.1:0/2545146118 learned_addr learned my addr 172.21.15.1:0/2545146118 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:57.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.343+0000 7f6b5affd640 1 --1- 172.21.15.1:0/2545146118 >> v1:172.21.15.1:6789/0 conn(0x7f6b5c0c6b10 0x7f6b5c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36766/0 (socket says 172.21.15.1:36766) 2024-03-06T07:26:57.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.343+0000 7f6b5b7fe640 1 --2- 172.21.15.1:0/2545146118 >> v2:172.21.15.98:3300/0 conn(0x7f6b5c152650 0x7f6b5c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.343+0000 7f6b5bfff640 1 -- 172.21.15.1:0/2545146118 >> v2:172.21.15.98:3300/0 conn(0x7f6b5c152650 msgr2=0x7f6b5c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.343+0000 7f6b5bfff640 1 --2- 172.21.15.1:0/2545146118 >> v2:172.21.15.98:3300/0 conn(0x7f6b5c152650 0x7f6b5c155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.343+0000 7f6b5bfff640 1 -- 172.21.15.1:0/2545146118 >> v1:172.21.15.1:6789/0 conn(0x7f6b5c0c6b10 legacy=0x7f6b5c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.343+0000 7f6b5bfff640 1 -- 172.21.15.1:0/2545146118 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c0c6690 con 0x7f6b5c0bf9d0 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.344+0000 7f6b5bfff640 1 --2- 172.21.15.1:0/2545146118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b5c0bf9d0 0x7f6b5c0c1460 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c0cac90 tx=0x7f6b5402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3870e53895eee906 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.344+0000 7f6b5a7fc640 1 -- 172.21.15.1:0/2545146118 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5402ca80 con 0x7f6b5c0bf9d0 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.344+0000 7f6b5a7fc640 1 -- 172.21.15.1:0/2545146118 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5402cc20 con 0x7f6b5c0bf9d0 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.344+0000 7f6b5a7fc640 1 -- 172.21.15.1:0/2545146118 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b540343f0 con 0x7f6b5c0bf9d0 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.345+0000 7f6b61f02640 1 -- 172.21.15.1:0/2545146118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b5c0bf9d0 msgr2=0x7f6b5c0c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.345+0000 7f6b61f02640 1 --2- 172.21.15.1:0/2545146118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b5c0bf9d0 0x7f6b5c0c1460 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c0cac90 tx=0x7f6b5402bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.345+0000 7f6b61f02640 1 -- 172.21.15.1:0/2545146118 shutdown_connections 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.345+0000 7f6b61f02640 1 --2- 172.21.15.1:0/2545146118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b5c0bf9d0 0x7f6b5c0c1460 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.345+0000 7f6b61f02640 1 --2- 172.21.15.1:0/2545146118 >> v2:172.21.15.98:3300/0 conn(0x7f6b5c152650 0x7f6b5c155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.345+0000 7f6b61f02640 1 -- 172.21.15.1:0/2545146118 >> 172.21.15.1:0/2545146118 conn(0x7f6b5c066ee0 msgr2=0x7f6b5c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.345+0000 7f6b61f02640 1 -- 172.21.15.1:0/2545146118 shutdown_connections 2024-03-06T07:26:57.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.345+0000 7f6b61f02640 1 -- 172.21.15.1:0/2545146118 wait complete. 2024-03-06T07:26:57.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.346+0000 7f6b61f02640 1 Processor -- start 2024-03-06T07:26:57.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.346+0000 7f6b61f02640 1 -- start start 2024-03-06T07:26:57.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b61f02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b5c0bf9d0 0x7f6b5c0c5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b61f02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c152650 0x7f6b5c14b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b61f02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b5c0ca380 0x7f6b5c14b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b61f02640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c1551b0 con 0x7f6b5c0ca380 2024-03-06T07:26:57.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b61f02640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c0c6fa0 con 0x7f6b5c152650 2024-03-06T07:26:57.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b61f02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c155490 con 0x7f6b5c0bf9d0 2024-03-06T07:26:57.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b5bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b5c0ca380 0x7f6b5c14b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b5bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b5c0ca380 0x7f6b5c14b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45126/0 (socket says 172.21.15.1:45126) 2024-03-06T07:26:57.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b5bfff640 1 -- 172.21.15.1:0/3799960475 learned_addr learned my addr 172.21.15.1:0/3799960475 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:57.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b5b7fe640 1 --2- 172.21.15.1:0/3799960475 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b5c0bf9d0 0x7f6b5c0c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b5affd640 1 --2- 172.21.15.1:0/3799960475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c152650 0x7f6b5c14b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b5bfff640 1 -- 172.21.15.1:0/3799960475 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b5c0bf9d0 msgr2=0x7f6b5c0c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b5bfff640 1 --2- 172.21.15.1:0/3799960475 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b5c0bf9d0 0x7f6b5c0c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b5bfff640 1 -- 172.21.15.1:0/3799960475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c152650 msgr2=0x7f6b5c14b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b5bfff640 1 --2- 172.21.15.1:0/3799960475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c152650 0x7f6b5c14b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b5bfff640 1 -- 172.21.15.1:0/3799960475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c14bc60 con 0x7f6b5c0ca380 2024-03-06T07:26:57.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b5affd640 1 --2- 172.21.15.1:0/3799960475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c152650 0x7f6b5c14b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:57.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b5bfff640 1 --2- 172.21.15.1:0/3799960475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b5c0ca380 0x7f6b5c14b720 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009f40 tx=0x7f6b54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.347+0000 7f6b43fff640 1 -- 172.21.15.1:0/3799960475 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b54042020 con 0x7f6b5c0ca380 2024-03-06T07:26:57.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.348+0000 7f6b43fff640 1 -- 172.21.15.1:0/3799960475 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5403e450 con 0x7f6b5c0ca380 2024-03-06T07:26:57.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.348+0000 7f6b61f02640 1 -- 172.21.15.1:0/3799960475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b5c151990 con 0x7f6b5c0ca380 2024-03-06T07:26:57.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.348+0000 7f6b43fff640 1 -- 172.21.15.1:0/3799960475 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b54009510 con 0x7f6b5c0ca380 2024-03-06T07:26:57.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.348+0000 7f6b61f02640 1 -- 172.21.15.1:0/3799960475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b5c14fcb0 con 0x7f6b5c0ca380 2024-03-06T07:26:57.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.349+0000 7f6b61f02640 1 -- 172.21.15.1:0/3799960475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b5c151eb0 con 0x7f6b5c0ca380 2024-03-06T07:26:57.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.349+0000 7f6b43fff640 1 -- 172.21.15.1:0/3799960475 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b54045020 con 0x7f6b5c0ca380 2024-03-06T07:26:57.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.349+0000 7f6b43fff640 1 --2- 172.21.15.1:0/3799960475 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b4806b500 0x7f6b4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.350+0000 7f6b43fff640 1 -- 172.21.15.1:0/3799960475 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f6b540ac960 con 0x7f6b5c0ca380 2024-03-06T07:26:57.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.350+0000 7f6b5b7fe640 1 --2- 172.21.15.1:0/3799960475 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b4806b500 0x7f6b4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.350+0000 7f6b5b7fe640 1 --2- 172.21.15.1:0/3799960475 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b4806b500 0x7f6b4806d9c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c00a970 tx=0x7f6b4c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.354+0000 7f6b43fff640 1 -- 172.21.15.1:0/3799960475 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b5c151eb0 con 0x7f6b5c0ca380 2024-03-06T07:26:57.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.536+0000 7f6b61f02640 1 -- 172.21.15.1:0/3799960475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b5c14ffc0 con 0x7f6b5c0ca380 2024-03-06T07:26:57.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.537+0000 7f6b43fff640 1 -- 172.21.15.1:0/3799960475 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+13693 (secure 0 0 0) 0x7f6b5404c030 con 0x7f6b5c0ca380 2024-03-06T07:26:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:26:57.539 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":162,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:26:55.205672+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6835/2664051788":"2024-03-07T07:26:52.197824+0000","172.21.15.98:6837/2360179300":"2024-03-07T07:26:46.842519+0000","172.21.15.98:6835/2360179300":"2024-03-07T07:26:46.842519+0000","172.21.15.1:6833/1089548586":"2024-03-07T07:26:36.687217+0000","172.21.15.1:6834/2664051788":"2024-03-07T07:26:52.197824+0000","172.21.15.1:6832/1089548586":"2024-03-07T07:26:36.687217+0000","172.21.15.98:6836/3360241462":"2024-03-07T07:26:45.837702+0000","172.21.15.98:6834/3360241462":"2024-03-07T07:26:45.837702+0000","172.21.15.1:6835/67091521":"2024-03-07T07:26:36.682683+0000","172.21.15.1:6834/67091521":"2024-03-07T07:26:36.682683+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:57.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.539+0000 7f6b61f02640 1 -- 172.21.15.1:0/3799960475 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b4806b500 msgr2=0x7f6b4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.539+0000 7f6b61f02640 1 --2- 172.21.15.1:0/3799960475 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b4806b500 0x7f6b4806d9c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c00a970 tx=0x7f6b4c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:57.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.539+0000 7f6b61f02640 1 -- 172.21.15.1:0/3799960475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b5c0ca380 msgr2=0x7f6b5c14b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.539+0000 7f6b61f02640 1 --2- 172.21.15.1:0/3799960475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b5c0ca380 0x7f6b5c14b720 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009f40 tx=0x7f6b54004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:57.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.539+0000 7f6b61f02640 1 -- 172.21.15.1:0/3799960475 shutdown_connections 2024-03-06T07:26:57.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.539+0000 7f6b61f02640 1 --2- 172.21.15.1:0/3799960475 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b4806b500 0x7f6b4806d9c0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.539+0000 7f6b61f02640 1 --2- 172.21.15.1:0/3799960475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b5c0ca380 0x7f6b5c14b720 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.539+0000 7f6b61f02640 1 --2- 172.21.15.1:0/3799960475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b5c152650 0x7f6b5c14b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.539+0000 7f6b61f02640 1 --2- 172.21.15.1:0/3799960475 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b5c0bf9d0 0x7f6b5c0c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.539+0000 7f6b61f02640 1 -- 172.21.15.1:0/3799960475 >> 172.21.15.1:0/3799960475 conn(0x7f6b5c066ee0 msgr2=0x7f6b5c0c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.539+0000 7f6b61f02640 1 -- 172.21.15.1:0/3799960475 shutdown_connections 2024-03-06T07:26:57.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.539+0000 7f6b61f02640 1 -- 172.21.15.1:0/3799960475 wait complete. 2024-03-06T07:26:57.552 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:57.552 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:57.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.655+0000 7f82f7516640 1 Processor -- start 2024-03-06T07:26:57.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.655+0000 7f82f7516640 1 -- start start 2024-03-06T07:26:57.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.655+0000 7f82f7516640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82f00c79c0 0x7f82f0152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.655+0000 7f82f7516640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f82f00a7ec0 0x7f82f01533e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.655+0000 7f82f7516640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f01566e0 con 0x7f82f00c4dd0 2024-03-06T07:26:57.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.655+0000 7f82f7516640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f0156490 con 0x7f82f00a7ec0 2024-03-06T07:26:57.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82f7516640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f0157010 con 0x7f82f00c79c0 2024-03-06T07:26:57.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82f52b2640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f82f00c4dd0 0x7f82f00c5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36782/0 (socket says 172.21.15.1:36782) 2024-03-06T07:26:57.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82f52b2640 1 -- 172.21.15.1:0/3075973213 learned_addr learned my addr 172.21.15.1:0/3075973213 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:57.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82f4ab1640 1 --2- 172.21.15.1:0/3075973213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82f00c79c0 0x7f82f0152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82f5ab3640 1 --2- 172.21.15.1:0/3075973213 >> v2:172.21.15.98:3300/0 conn(0x7f82f00a7ec0 0x7f82f01533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82e7fff640 1 -- 172.21.15.1:0/3075973213 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755722442 0 0) 0x7f82f01566e0 con 0x7f82f00c4dd0 2024-03-06T07:26:57.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82e7fff640 1 -- 172.21.15.1:0/3075973213 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82dc003520 con 0x7f82f00c4dd0 2024-03-06T07:26:57.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82e7fff640 1 -- 172.21.15.1:0/3075973213 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2945228219 0 0) 0x7f82dc003520 con 0x7f82f00c4dd0 2024-03-06T07:26:57.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82e7fff640 1 -- 172.21.15.1:0/3075973213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82f00c79c0 msgr2=0x7f82f0152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82e7fff640 1 --2- 172.21.15.1:0/3075973213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82f00c79c0 0x7f82f0152ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82e7fff640 1 -- 172.21.15.1:0/3075973213 >> v2:172.21.15.98:3300/0 conn(0x7f82f00a7ec0 msgr2=0x7f82f01533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82e7fff640 1 --2- 172.21.15.1:0/3075973213 >> v2:172.21.15.98:3300/0 conn(0x7f82f00a7ec0 0x7f82f01533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82e7fff640 1 -- 172.21.15.1:0/3075973213 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f01569c0 con 0x7f82f00c4dd0 2024-03-06T07:26:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82e7fff640 1 -- 172.21.15.1:0/3075973213 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f82ec001670 con 0x7f82f00c4dd0 2024-03-06T07:26:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82e7fff640 1 -- 172.21.15.1:0/3075973213 >> v1:172.21.15.1:6789/0 conn(0x7f82f00c4dd0 legacy=0x7f82f00c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82e7fff640 1 --2- 172.21.15.1:0/3075973213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82dc0039b0 0x7f82dc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82e7fff640 1 --2- 172.21.15.1:0/3075973213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82dc004310 0x7f82dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82e7fff640 1 --2- 172.21.15.1:0/3075973213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82dc004ef0 0x7f82dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82e7fff640 1 -- 172.21.15.1:0/3075973213 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f0156490 con 0x7f82dc004310 2024-03-06T07:26:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82e7fff640 1 -- 172.21.15.1:0/3075973213 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f0157010 con 0x7f82dc004ef0 2024-03-06T07:26:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82e7fff640 1 -- 172.21.15.1:0/3075973213 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc00bf00 con 0x7f82dc0039b0 2024-03-06T07:26:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82f4ab1640 1 --2- 172.21.15.1:0/3075973213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82dc004310 0x7f82dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82f52b2640 1 --2- 172.21.15.1:0/3075973213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82dc0039b0 0x7f82dc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.656+0000 7f82f5ab3640 1 --2- 172.21.15.1:0/3075973213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82dc004ef0 0x7f82dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.657+0000 7f82f4ab1640 1 -- 172.21.15.1:0/3075973213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82dc0039b0 msgr2=0x7f82dc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.657+0000 7f82f4ab1640 1 --2- 172.21.15.1:0/3075973213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82dc0039b0 0x7f82dc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.657+0000 7f82f4ab1640 1 -- 172.21.15.1:0/3075973213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82dc004ef0 msgr2=0x7f82dc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.657+0000 7f82f4ab1640 1 --2- 172.21.15.1:0/3075973213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82dc004ef0 0x7f82dc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.657+0000 7f82f4ab1640 1 -- 172.21.15.1:0/3075973213 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82dc003520 con 0x7f82dc004310 2024-03-06T07:26:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.657+0000 7f82f4ab1640 1 --2- 172.21.15.1:0/3075973213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82dc004310 0x7f82dc004800 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009ac0 tx=0x7f82dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e11dc6c6cf56c1ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.658+0000 7f82e7fff640 1 -- 172.21.15.1:0/3075973213 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0039070 con 0x7f82dc004310 2024-03-06T07:26:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.658+0000 7f82e7fff640 1 -- 172.21.15.1:0/3075973213 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e002c6b0 con 0x7f82dc004310 2024-03-06T07:26:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.658+0000 7f82e7fff640 1 -- 172.21.15.1:0/3075973213 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0009690 con 0x7f82dc004310 2024-03-06T07:26:57.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.658+0000 7f82f7516640 1 -- 172.21.15.1:0/3075973213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82dc004310 msgr2=0x7f82dc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.658+0000 7f82f7516640 1 --2- 172.21.15.1:0/3075973213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82dc004310 0x7f82dc004800 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009ac0 tx=0x7f82dc005830 comp rx=0 tx=0).stop 2024-03-06T07:26:57.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.658+0000 7f82f5ab3640 1 -- 172.21.15.1:0/3075973213 reap_dead start 2024-03-06T07:26:57.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.658+0000 7f82f7516640 1 -- 172.21.15.1:0/3075973213 shutdown_connections 2024-03-06T07:26:57.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.658+0000 7f82f7516640 1 -- 172.21.15.1:0/3075973213 >> 172.21.15.1:0/3075973213 conn(0x7f82f006b410 msgr2=0x7f82f006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.658+0000 7f82f7516640 1 -- 172.21.15.1:0/3075973213 shutdown_connections 2024-03-06T07:26:57.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.658+0000 7f82f7516640 1 -- 172.21.15.1:0/3075973213 wait complete. 2024-03-06T07:26:57.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.659+0000 7f82f7516640 1 Processor -- start 2024-03-06T07:26:57.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.659+0000 7f82f7516640 1 -- start start 2024-03-06T07:26:57.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.660+0000 7f82f7516640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82dc004310 0x7f82f00c3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.660+0000 7f82f7516640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82f00c27c0 0x7f82f0152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.660+0000 7f82f7516640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82f00ad1e0 0x7f82f0152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.660+0000 7f82f7516640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f00c60d0 con 0x7f82f00ad1e0 2024-03-06T07:26:57.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.660+0000 7f82f7516640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f00c5f50 con 0x7f82f00c27c0 2024-03-06T07:26:57.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.660+0000 7f82f7516640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f00715f0 con 0x7f82dc004310 2024-03-06T07:26:57.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.660+0000 7f82f5ab3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82f00ad1e0 0x7f82f0152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.660+0000 7f82f5ab3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82f00ad1e0 0x7f82f0152c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45144/0 (socket says 172.21.15.1:45144) 2024-03-06T07:26:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.660+0000 7f82f5ab3640 1 -- 172.21.15.1:0/249330043 learned_addr learned my addr 172.21.15.1:0/249330043 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.660+0000 7f82f5ab3640 1 -- 172.21.15.1:0/249330043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82dc004310 msgr2=0x7f82f00c3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.660+0000 7f82f5ab3640 1 --2- 172.21.15.1:0/249330043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82dc004310 0x7f82f00c3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.660+0000 7f82f5ab3640 1 -- 172.21.15.1:0/249330043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82f00c27c0 msgr2=0x7f82f0152730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.660+0000 7f82f4ab1640 1 --2- 172.21.15.1:0/249330043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82f00c27c0 0x7f82f0152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.660+0000 7f82f5ab3640 1 --2- 172.21.15.1:0/249330043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82f00c27c0 0x7f82f0152730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.660+0000 7f82f5ab3640 1 -- 172.21.15.1:0/249330043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f006dc50 con 0x7f82f00ad1e0 2024-03-06T07:26:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.661+0000 7f82f5ab3640 1 --2- 172.21.15.1:0/249330043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82f00ad1e0 0x7f82f0152c70 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f82f00a83c0 tx=0x7f82e800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.661+0000 7f82e5ffb640 1 -- 172.21.15.1:0/249330043 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e800fde0 con 0x7f82f00ad1e0 2024-03-06T07:26:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.661+0000 7f82e5ffb640 1 -- 172.21.15.1:0/249330043 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e801d070 con 0x7f82f00ad1e0 2024-03-06T07:26:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.661+0000 7f82f7516640 1 -- 172.21.15.1:0/249330043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f006df30 con 0x7f82f00ad1e0 2024-03-06T07:26:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.661+0000 7f82e5ffb640 1 -- 172.21.15.1:0/249330043 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8015670 con 0x7f82f00ad1e0 2024-03-06T07:26:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.661+0000 7f82f7516640 1 -- 172.21.15.1:0/249330043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f00c4160 con 0x7f82f00ad1e0 2024-03-06T07:26:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.662+0000 7f82f7516640 1 -- 172.21.15.1:0/249330043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f00c1c20 con 0x7f82f00ad1e0 2024-03-06T07:26:57.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.662+0000 7f82e5ffb640 1 -- 172.21.15.1:0/249330043 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82e8015810 con 0x7f82f00ad1e0 2024-03-06T07:26:57.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.663+0000 7f82e5ffb640 1 --2- 172.21.15.1:0/249330043 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f82d406b500 0x7f82d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.663+0000 7f82e5ffb640 1 -- 172.21.15.1:0/249330043 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f82e808abe0 con 0x7f82f00ad1e0 2024-03-06T07:26:57.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.663+0000 7f82f52b2640 1 --2- 172.21.15.1:0/249330043 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f82d406b500 0x7f82d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.664+0000 7f82f52b2640 1 --2- 172.21.15.1:0/249330043 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f82d406b500 0x7f82d406d9c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0014f0 tx=0x7f82ec0010a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.667+0000 7f82e5ffb640 1 -- 172.21.15.1:0/249330043 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82e8010040 con 0x7f82f00ad1e0 2024-03-06T07:26:57.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.847+0000 7f82f7516640 1 -- 172.21.15.1:0/249330043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f82f0067fd0 con 0x7f82f00ad1e0 2024-03-06T07:26:57.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.848+0000 7f82e5ffb640 1 -- 172.21.15.1:0/249330043 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v162) v1 ==== 93+0+10 (secure 0 0 0) 0x7f82e80593a0 con 0x7f82f00ad1e0 2024-03-06T07:26:57.850 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:26:57.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.850+0000 7f82f7516640 1 -- 172.21.15.1:0/249330043 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f82d406b500 msgr2=0x7f82d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.850+0000 7f82f7516640 1 --2- 172.21.15.1:0/249330043 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f82d406b500 0x7f82d406d9c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0014f0 tx=0x7f82ec0010a0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.850+0000 7f82f7516640 1 -- 172.21.15.1:0/249330043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82f00ad1e0 msgr2=0x7f82f0152c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.850+0000 7f82f7516640 1 --2- 172.21.15.1:0/249330043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82f00ad1e0 0x7f82f0152c70 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f82f00a83c0 tx=0x7f82e800f580 comp rx=0 tx=0).stop 2024-03-06T07:26:57.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.850+0000 7f82f7516640 1 -- 172.21.15.1:0/249330043 shutdown_connections 2024-03-06T07:26:57.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.850+0000 7f82f7516640 1 --2- 172.21.15.1:0/249330043 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f82d406b500 0x7f82d406d9c0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.850+0000 7f82f7516640 1 --2- 172.21.15.1:0/249330043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82f00ad1e0 0x7f82f0152c70 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.850+0000 7f82f7516640 1 --2- 172.21.15.1:0/249330043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82f00c27c0 0x7f82f0152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.850+0000 7f82f7516640 1 --2- 172.21.15.1:0/249330043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82dc004310 0x7f82f00c3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.850+0000 7f82f7516640 1 -- 172.21.15.1:0/249330043 >> 172.21.15.1:0/249330043 conn(0x7f82f006b410 msgr2=0x7f82f00c32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.850+0000 7f82f7516640 1 -- 172.21.15.1:0/249330043 shutdown_connections 2024-03-06T07:26:57.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.850+0000 7f82f7516640 1 -- 172.21.15.1:0/249330043 wait complete. 2024-03-06T07:26:57.863 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down' 2024-03-06T07:26:57.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.962+0000 7fe9c619a640 1 Processor -- start 2024-03-06T07:26:57.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.962+0000 7fe9c619a640 1 -- start start 2024-03-06T07:26:57.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.963+0000 7fe9c619a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9c01506b0 0x7fe9c00c5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.963+0000 7fe9c619a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe9c00a8090 0x7fe9c01538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.963+0000 7fe9c619a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c00c5770 con 0x7fe9c00c6720 2024-03-06T07:26:57.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.963+0000 7fe9c619a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c00c6dc0 con 0x7fe9c00a8090 2024-03-06T07:26:57.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.963+0000 7fe9c619a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c00c2ef0 con 0x7fe9c01506b0 2024-03-06T07:26:57.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.963+0000 7fe9beffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe9c00a8090 0x7fe9c01538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.963+0000 7fe9beffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe9c00a8090 0x7fe9c01538b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45146/0 (socket says 172.21.15.1:45146) 2024-03-06T07:26:57.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.963+0000 7fe9beffd640 1 -- 172.21.15.1:0/1831515534 learned_addr learned my addr 172.21.15.1:0/1831515534 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:57.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.963+0000 7fe9bf7fe640 1 --2- 172.21.15.1:0/1831515534 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9c01506b0 0x7fe9c00c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.963+0000 7fe9beffd640 1 -- 172.21.15.1:0/1831515534 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9c01506b0 msgr2=0x7fe9c00c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.963+0000 7fe9beffd640 1 --2- 172.21.15.1:0/1831515534 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9c01506b0 0x7fe9c00c5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.963+0000 7fe9beffd640 1 -- 172.21.15.1:0/1831515534 >> v1:172.21.15.98:6789/0 conn(0x7fe9c00c6720 legacy=0x7fe9c00c31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.963+0000 7fe9beffd640 1 -- 172.21.15.1:0/1831515534 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c01565c0 con 0x7fe9c00a8090 2024-03-06T07:26:57.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.964+0000 7fe9beffd640 1 --2- 172.21.15.1:0/1831515534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c00a8090 0x7fe9c01538b0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4009870 tx=0x7fe9b402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898a9949ae84ce5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.964+0000 7fe9be7fc640 1 -- 172.21.15.1:0/1831515534 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b40317c0 con 0x7fe9c00a8090 2024-03-06T07:26:57.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.964+0000 7fe9be7fc640 1 -- 172.21.15.1:0/1831515534 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b4037070 con 0x7fe9c00a8090 2024-03-06T07:26:57.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.964+0000 7fe9be7fc640 1 -- 172.21.15.1:0/1831515534 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4031b80 con 0x7fe9c00a8090 2024-03-06T07:26:57.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.965+0000 7fe9c619a640 1 -- 172.21.15.1:0/1831515534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c00a8090 msgr2=0x7fe9c01538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.965+0000 7fe9c619a640 1 --2- 172.21.15.1:0/1831515534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c00a8090 0x7fe9c01538b0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4009870 tx=0x7fe9b402b860 comp rx=0 tx=0).stop 2024-03-06T07:26:57.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.965+0000 7fe9c619a640 1 -- 172.21.15.1:0/1831515534 shutdown_connections 2024-03-06T07:26:57.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.965+0000 7fe9c619a640 1 --2- 172.21.15.1:0/1831515534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c00a8090 0x7fe9c01538b0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.965+0000 7fe9c619a640 1 --2- 172.21.15.1:0/1831515534 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9c01506b0 0x7fe9c00c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.965+0000 7fe9c619a640 1 -- 172.21.15.1:0/1831515534 >> 172.21.15.1:0/1831515534 conn(0x7fe9c00aa550 msgr2=0x7fe9c00aa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.965+0000 7fe9c619a640 1 -- 172.21.15.1:0/1831515534 shutdown_connections 2024-03-06T07:26:57.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.965+0000 7fe9c619a640 1 -- 172.21.15.1:0/1831515534 wait complete. 2024-03-06T07:26:57.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.966+0000 7fe9c619a640 1 Processor -- start 2024-03-06T07:26:57.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.966+0000 7fe9c619a640 1 -- start start 2024-03-06T07:26:57.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.966+0000 7fe9c619a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c00a8090 0x7fe9c006dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.966+0000 7fe9c619a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c00c6720 0x7fe9c006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.966+0000 7fe9c619a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9c01506b0 0x7fe9c00c1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.966+0000 7fe9c619a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c0153df0 con 0x7fe9c00c6720 2024-03-06T07:26:57.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.966+0000 7fe9c619a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c0153390 con 0x7fe9c00a8090 2024-03-06T07:26:57.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.966+0000 7fe9c619a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c0150a90 con 0x7fe9c01506b0 2024-03-06T07:26:57.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.967+0000 7fe9beffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c00c6720 0x7fe9c006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.967+0000 7fe9beffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c00c6720 0x7fe9c006d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45158/0 (socket says 172.21.15.1:45158) 2024-03-06T07:26:57.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.967+0000 7fe9beffd640 1 -- 172.21.15.1:0/4045947416 learned_addr learned my addr 172.21.15.1:0/4045947416 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:57.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.967+0000 7fe9bf7fe640 1 --2- 172.21.15.1:0/4045947416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c00a8090 0x7fe9c006dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.967+0000 7fe9bffff640 1 --2- 172.21.15.1:0/4045947416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9c01506b0 0x7fe9c00c1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.967+0000 7fe9beffd640 1 -- 172.21.15.1:0/4045947416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9c01506b0 msgr2=0x7fe9c00c1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.967+0000 7fe9beffd640 1 --2- 172.21.15.1:0/4045947416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9c01506b0 0x7fe9c00c1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.967+0000 7fe9beffd640 1 -- 172.21.15.1:0/4045947416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c00a8090 msgr2=0x7fe9c006dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.967+0000 7fe9beffd640 1 --2- 172.21.15.1:0/4045947416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c00a8090 0x7fe9c006dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.967+0000 7fe9beffd640 1 -- 172.21.15.1:0/4045947416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c014b9d0 con 0x7fe9c00c6720 2024-03-06T07:26:57.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.967+0000 7fe9beffd640 1 --2- 172.21.15.1:0/4045947416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c00c6720 0x7fe9c006d3d0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4000c00 tx=0x7fe9b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.967+0000 7fe9a7fff640 1 -- 172.21.15.1:0/4045947416 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4032650 con 0x7fe9c00c6720 2024-03-06T07:26:57.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.967+0000 7fe9a7fff640 1 -- 172.21.15.1:0/4045947416 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b40327f0 con 0x7fe9c00c6720 2024-03-06T07:26:57.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.967+0000 7fe9c619a640 1 -- 172.21.15.1:0/4045947416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c014bcb0 con 0x7fe9c00c6720 2024-03-06T07:26:57.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.967+0000 7fe9a7fff640 1 -- 172.21.15.1:0/4045947416 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4032be0 con 0x7fe9c00c6720 2024-03-06T07:26:57.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.967+0000 7fe9c619a640 1 -- 172.21.15.1:0/4045947416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c014bf90 con 0x7fe9c00c6720 2024-03-06T07:26:57.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.968+0000 7fe9c619a640 1 -- 172.21.15.1:0/4045947416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c0067fd0 con 0x7fe9c00c6720 2024-03-06T07:26:57.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.969+0000 7fe9a7fff640 1 -- 172.21.15.1:0/4045947416 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9b404c060 con 0x7fe9c00c6720 2024-03-06T07:26:57.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.969+0000 7fe9a7fff640 1 --2- 172.21.15.1:0/4045947416 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9ac06b4d0 0x7fe9ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.969+0000 7fe9a7fff640 1 -- 172.21.15.1:0/4045947416 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fe9b40abcb0 con 0x7fe9c00c6720 2024-03-06T07:26:57.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.969+0000 7fe9bf7fe640 1 --2- 172.21.15.1:0/4045947416 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9ac06b4d0 0x7fe9ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.970+0000 7fe9bf7fe640 1 --2- 172.21.15.1:0/4045947416 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9ac06b4d0 0x7fe9ac06d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0006fd0 tx=0x7fe9b0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:57.973+0000 7fe9a7fff640 1 -- 172.21.15.1:0/4045947416 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe9b4040070 con 0x7fe9c00c6720 2024-03-06T07:26:58.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:58.229+0000 7fe9c619a640 1 -- 172.21.15.1:0/4045947416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down"]} v 0) v1 -- 0x7fe9c00c7e80 con 0x7fe9c00c6720 2024-03-06T07:26:59.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.225+0000 7fe9a7fff640 1 -- 172.21.15.1:0/4045947416 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down"]}]=0 v385) v1 ==== 136+0+0 (secure 0 0 0) 0x7fe9b407a460 con 0x7fe9c00c6720 2024-03-06T07:26:59.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.228+0000 7fe9c619a640 1 -- 172.21.15.1:0/4045947416 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9ac06b4d0 msgr2=0x7fe9ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.228+0000 7fe9c619a640 1 --2- 172.21.15.1:0/4045947416 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9ac06b4d0 0x7fe9ac06d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0006fd0 tx=0x7fe9b0008040 comp rx=0 tx=0).stop 2024-03-06T07:26:59.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.228+0000 7fe9c619a640 1 -- 172.21.15.1:0/4045947416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c00c6720 msgr2=0x7fe9c006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.228+0000 7fe9c619a640 1 --2- 172.21.15.1:0/4045947416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c00c6720 0x7fe9c006d3d0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4000c00 tx=0x7fe9b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.228+0000 7fe9c619a640 1 -- 172.21.15.1:0/4045947416 shutdown_connections 2024-03-06T07:26:59.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.228+0000 7fe9c619a640 1 --2- 172.21.15.1:0/4045947416 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe9ac06b4d0 0x7fe9ac06d990 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.228+0000 7fe9c619a640 1 --2- 172.21.15.1:0/4045947416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9c01506b0 0x7fe9c00c1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.228+0000 7fe9c619a640 1 --2- 172.21.15.1:0/4045947416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9c00c6720 0x7fe9c006d3d0 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.228+0000 7fe9c619a640 1 --2- 172.21.15.1:0/4045947416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9c00a8090 0x7fe9c006dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.228+0000 7fe9c619a640 1 -- 172.21.15.1:0/4045947416 >> 172.21.15.1:0/4045947416 conn(0x7fe9c00aa550 msgr2=0x7fe9c00c30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.228+0000 7fe9c619a640 1 -- 172.21.15.1:0/4045947416 shutdown_connections 2024-03-06T07:26:59.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.228+0000 7fe9c619a640 1 -- 172.21.15.1:0/4045947416 wait complete. 2024-03-06T07:26:59.243 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:26:59.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.352+0000 7f1546ed4640 1 Processor -- start 2024-03-06T07:26:59.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.352+0000 7f1546ed4640 1 -- start start 2024-03-06T07:26:59.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.353+0000 7f1546ed4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f15400c2b30 0x7f15400724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.353+0000 7f1546ed4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f15400a5fa0 0x7f1540072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.353+0000 7f1546ed4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1540151dd0 con 0x7f1540152b70 2024-03-06T07:26:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.353+0000 7f1546ed4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15401526e0 con 0x7f15400c2b30 2024-03-06T07:26:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.353+0000 7f1546ed4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1540156190 con 0x7f15400a5fa0 2024-03-06T07:26:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.353+0000 7f1544c70640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f15400c2b30 0x7f15400724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.353+0000 7f1544c70640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f15400c2b30 0x7f15400724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45166/0 (socket says 172.21.15.1:45166) 2024-03-06T07:26:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.353+0000 7f153ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1540152b70 0x7f15400c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36790/0 (socket says 172.21.15.1:36790) 2024-03-06T07:26:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.353+0000 7f1544c70640 1 -- 172.21.15.1:0/658303936 learned_addr learned my addr 172.21.15.1:0/658303936 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.353+0000 7f1545471640 1 --2- 172.21.15.1:0/658303936 >> v2:172.21.15.98:3300/0 conn(0x7f15400a5fa0 0x7f1540072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.353+0000 7f1544c70640 1 -- 172.21.15.1:0/658303936 >> v2:172.21.15.98:3300/0 conn(0x7f15400a5fa0 msgr2=0x7f1540072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.353+0000 7f1544c70640 1 --2- 172.21.15.1:0/658303936 >> v2:172.21.15.98:3300/0 conn(0x7f15400a5fa0 0x7f1540072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.353+0000 7f1544c70640 1 -- 172.21.15.1:0/658303936 >> v1:172.21.15.1:6789/0 conn(0x7f1540152b70 legacy=0x7f15400c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.353+0000 7f1544c70640 1 -- 172.21.15.1:0/658303936 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1540152090 con 0x7f15400c2b30 2024-03-06T07:26:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.354+0000 7f1544c70640 1 --2- 172.21.15.1:0/658303936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15400c2b30 0x7f15400724c0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f154006f710 tx=0x7f153402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6197bfc95d02f773 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.354+0000 7f153f7fe640 1 -- 172.21.15.1:0/658303936 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1534034070 con 0x7f15400c2b30 2024-03-06T07:26:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.354+0000 7f153f7fe640 1 -- 172.21.15.1:0/658303936 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1534030860 con 0x7f15400c2b30 2024-03-06T07:26:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.354+0000 7f153f7fe640 1 -- 172.21.15.1:0/658303936 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1534030c30 con 0x7f15400c2b30 2024-03-06T07:26:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.355+0000 7f1546ed4640 1 -- 172.21.15.1:0/658303936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15400c2b30 msgr2=0x7f15400724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.355+0000 7f1546ed4640 1 --2- 172.21.15.1:0/658303936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15400c2b30 0x7f15400724c0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f154006f710 tx=0x7f153402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.355+0000 7f1546ed4640 1 -- 172.21.15.1:0/658303936 shutdown_connections 2024-03-06T07:26:59.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.355+0000 7f1546ed4640 1 --2- 172.21.15.1:0/658303936 >> v2:172.21.15.98:3300/0 conn(0x7f15400a5fa0 0x7f1540072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.355+0000 7f1546ed4640 1 --2- 172.21.15.1:0/658303936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15400c2b30 0x7f15400724c0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.355+0000 7f1546ed4640 1 -- 172.21.15.1:0/658303936 >> 172.21.15.1:0/658303936 conn(0x7f15400c6860 msgr2=0x7f15400c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.355+0000 7f1546ed4640 1 -- 172.21.15.1:0/658303936 shutdown_connections 2024-03-06T07:26:59.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.355+0000 7f1546ed4640 1 -- 172.21.15.1:0/658303936 wait complete. 2024-03-06T07:26:59.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.356+0000 7f1546ed4640 1 Processor -- start 2024-03-06T07:26:59.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.356+0000 7f1546ed4640 1 -- start start 2024-03-06T07:26:59.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f1546ed4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15400a5fa0 0x7f15401e74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f1546ed4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15400c2b30 0x7f154014fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f1546ed4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1540152b70 0x7f15400bbff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f1546ed4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1540155f00 con 0x7f1540152b70 2024-03-06T07:26:59.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f1546ed4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1540155d80 con 0x7f15400c2b30 2024-03-06T07:26:59.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f1546ed4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15400c3a00 con 0x7f15400a5fa0 2024-03-06T07:26:59.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f1545471640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1540152b70 0x7f15400bbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f1545471640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1540152b70 0x7f15400bbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45174/0 (socket says 172.21.15.1:45174) 2024-03-06T07:26:59.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f1545471640 1 -- 172.21.15.1:0/2378306610 learned_addr learned my addr 172.21.15.1:0/2378306610 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:59.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f1544c70640 1 --2- 172.21.15.1:0/2378306610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15400a5fa0 0x7f15401e74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f153ffff640 1 --2- 172.21.15.1:0/2378306610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15400c2b30 0x7f154014fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f1545471640 1 -- 172.21.15.1:0/2378306610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15400a5fa0 msgr2=0x7f15401e74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f1545471640 1 --2- 172.21.15.1:0/2378306610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15400a5fa0 0x7f15401e74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f1545471640 1 -- 172.21.15.1:0/2378306610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15400c2b30 msgr2=0x7f154014fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f1545471640 1 --2- 172.21.15.1:0/2378306610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15400c2b30 0x7f154014fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f1545471640 1 -- 172.21.15.1:0/2378306610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15401e7a20 con 0x7f1540152b70 2024-03-06T07:26:59.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f1545471640 1 --2- 172.21.15.1:0/2378306610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1540152b70 0x7f15400bbff0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f1538007e70 tx=0x7f153800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f153d7fa640 1 -- 172.21.15.1:0/2378306610 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1538014620 con 0x7f1540152b70 2024-03-06T07:26:59.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.357+0000 7f153d7fa640 1 -- 172.21.15.1:0/2378306610 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1538019070 con 0x7f1540152b70 2024-03-06T07:26:59.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.358+0000 7f1546ed4640 1 -- 172.21.15.1:0/2378306610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15400bd360 con 0x7f1540152b70 2024-03-06T07:26:59.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.358+0000 7f153d7fa640 1 -- 172.21.15.1:0/2378306610 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1538014a20 con 0x7f1540152b70 2024-03-06T07:26:59.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.358+0000 7f1546ed4640 1 -- 172.21.15.1:0/2378306610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15400bd920 con 0x7f1540152b70 2024-03-06T07:26:59.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.358+0000 7f1546ed4640 1 -- 172.21.15.1:0/2378306610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15400c7630 con 0x7f1540152b70 2024-03-06T07:26:59.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.359+0000 7f153d7fa640 1 -- 172.21.15.1:0/2378306610 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1538014bc0 con 0x7f1540152b70 2024-03-06T07:26:59.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.359+0000 7f153d7fa640 1 --2- 172.21.15.1:0/2378306610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f152406b550 0x7f152406da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.359+0000 7f153d7fa640 1 -- 172.21.15.1:0/2378306610 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f153808a7b0 con 0x7f1540152b70 2024-03-06T07:26:59.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.360+0000 7f1544c70640 1 --2- 172.21.15.1:0/2378306610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f152406b550 0x7f152406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.360+0000 7f1544c70640 1 --2- 172.21.15.1:0/2378306610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f152406b550 0x7f152406da10 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f1534036000 tx=0x7f1534008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.364+0000 7f153d7fa640 1 -- 172.21.15.1:0/2378306610 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f15400c7630 con 0x7f1540152b70 2024-03-06T07:26:59.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.547+0000 7f1546ed4640 1 -- 172.21.15.1:0/2378306610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1540067fd0 con 0x7f1540152b70 2024-03-06T07:26:59.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.550+0000 7f153d7fa640 1 -- 172.21.15.1:0/2378306610 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v57)=0 v57) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1538059170 con 0x7f1540152b70 2024-03-06T07:26:59.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.552+0000 7f1546ed4640 1 -- 172.21.15.1:0/2378306610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f152406b550 msgr2=0x7f152406da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.552+0000 7f1546ed4640 1 --2- 172.21.15.1:0/2378306610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f152406b550 0x7f152406da10 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f1534036000 tx=0x7f1534008030 comp rx=0 tx=0).stop 2024-03-06T07:26:59.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.552+0000 7f1546ed4640 1 -- 172.21.15.1:0/2378306610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1540152b70 msgr2=0x7f15400bbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.552+0000 7f1546ed4640 1 --2- 172.21.15.1:0/2378306610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1540152b70 0x7f15400bbff0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f1538007e70 tx=0x7f153800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.552+0000 7f1546ed4640 1 -- 172.21.15.1:0/2378306610 shutdown_connections 2024-03-06T07:26:59.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.552+0000 7f1546ed4640 1 --2- 172.21.15.1:0/2378306610 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f152406b550 0x7f152406da10 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.552+0000 7f1546ed4640 1 --2- 172.21.15.1:0/2378306610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1540152b70 0x7f15400bbff0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.552+0000 7f1546ed4640 1 --2- 172.21.15.1:0/2378306610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15400c2b30 0x7f154014fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.552+0000 7f1546ed4640 1 --2- 172.21.15.1:0/2378306610 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15400a5fa0 0x7f15401e74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.552+0000 7f1546ed4640 1 -- 172.21.15.1:0/2378306610 >> 172.21.15.1:0/2378306610 conn(0x7f15400c6860 msgr2=0x7f15400c5e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.552+0000 7f1546ed4640 1 -- 172.21.15.1:0/2378306610 shutdown_connections 2024-03-06T07:26:59.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.552+0000 7f1546ed4640 1 -- 172.21.15.1:0/2378306610 wait complete. 2024-03-06T07:26:59.565 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:59.565 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:59.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:59.674 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:59.674 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:26:59.674 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:59.689 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:26:59.690 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:26:59.690 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:59.706 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:26:59.706 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:26:59.706 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:59.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.829+0000 7f2916a2a640 1 Processor -- start 2024-03-06T07:26:59.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.829+0000 7f2916a2a640 1 -- start start 2024-03-06T07:26:59.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f2916a2a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f29100c1260 0x7f29100c16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f2916a2a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29101565a0 0x7f2910156a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f2916a2a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29101530a0 con 0x7f2910149a90 2024-03-06T07:26:59.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f2916a2a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2910153380 con 0x7f29100c1260 2024-03-06T07:26:59.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f2916a2a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2910153660 con 0x7f29101565a0 2024-03-06T07:26:59.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2910149a90 0x7f29100c4760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36798/0 (socket says 172.21.15.1:36798) 2024-03-06T07:26:59.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290ffff640 1 -- 172.21.15.1:0/2773871020 learned_addr learned my addr 172.21.15.1:0/2773871020 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:26:59.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f2914fc7640 1 --2- 172.21.15.1:0/2773871020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29101565a0 0x7f2910156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290f7fe640 1 --2- 172.21.15.1:0/2773871020 >> v2:172.21.15.98:3300/0 conn(0x7f29100c1260 0x7f29100c16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290effd640 1 -- 172.21.15.1:0/2773871020 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225962985 0 0) 0x7f29101530a0 con 0x7f2910149a90 2024-03-06T07:26:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290effd640 1 -- 172.21.15.1:0/2773871020 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28f8003520 con 0x7f2910149a90 2024-03-06T07:26:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290effd640 1 -- 172.21.15.1:0/2773871020 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1171833612 0 0) 0x7f28f8003520 con 0x7f2910149a90 2024-03-06T07:26:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290effd640 1 -- 172.21.15.1:0/2773871020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29101565a0 msgr2=0x7f2910156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290effd640 1 --2- 172.21.15.1:0/2773871020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29101565a0 0x7f2910156a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290effd640 1 -- 172.21.15.1:0/2773871020 >> v2:172.21.15.98:3300/0 conn(0x7f29100c1260 msgr2=0x7f29100c16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290effd640 1 --2- 172.21.15.1:0/2773871020 >> v2:172.21.15.98:3300/0 conn(0x7f29100c1260 0x7f29100c16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290effd640 1 -- 172.21.15.1:0/2773871020 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2910153940 con 0x7f2910149a90 2024-03-06T07:26:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290effd640 1 -- 172.21.15.1:0/2773871020 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f2904001670 con 0x7f2910149a90 2024-03-06T07:26:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290effd640 1 -- 172.21.15.1:0/2773871020 >> v1:172.21.15.1:6789/0 conn(0x7f2910149a90 legacy=0x7f29100c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290effd640 1 --2- 172.21.15.1:0/2773871020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29101565a0 0x7f28f80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290effd640 1 --2- 172.21.15.1:0/2773871020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28f8003f30 0x7f28f8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290effd640 1 --2- 172.21.15.1:0/2773871020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28f8004d40 0x7f28f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290effd640 1 -- 172.21.15.1:0/2773871020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2910153380 con 0x7f28f8004d40 2024-03-06T07:26:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.830+0000 7f290effd640 1 -- 172.21.15.1:0/2773871020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2910153660 con 0x7f28f8003f30 2024-03-06T07:26:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.831+0000 7f290effd640 1 -- 172.21.15.1:0/2773871020 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f800ba10 con 0x7f29101565a0 2024-03-06T07:26:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.831+0000 7f2914fc7640 1 --2- 172.21.15.1:0/2773871020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28f8004d40 0x7f28f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.831+0000 7f290ffff640 1 --2- 172.21.15.1:0/2773871020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29101565a0 0x7f28f80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.831+0000 7f290f7fe640 1 --2- 172.21.15.1:0/2773871020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28f8003f30 0x7f28f8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.831+0000 7f2914fc7640 1 -- 172.21.15.1:0/2773871020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29101565a0 msgr2=0x7f28f80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.831+0000 7f2914fc7640 1 --2- 172.21.15.1:0/2773871020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29101565a0 0x7f28f80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.831+0000 7f2914fc7640 1 -- 172.21.15.1:0/2773871020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28f8003f30 msgr2=0x7f28f8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.831+0000 7f2914fc7640 1 --2- 172.21.15.1:0/2773871020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28f8003f30 0x7f28f8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.831+0000 7f2914fc7640 1 -- 172.21.15.1:0/2773871020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f8003520 con 0x7f28f8004d40 2024-03-06T07:26:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.832+0000 7f2914fc7640 1 --2- 172.21.15.1:0/2773871020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28f8004d40 0x7f28f80051c0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f290800e170 tx=0x7f28f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed87203ece9354fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.832+0000 7f290effd640 1 -- 172.21.15.1:0/2773871020 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2908039070 con 0x7f28f8004d40 2024-03-06T07:26:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.832+0000 7f290effd640 1 -- 172.21.15.1:0/2773871020 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290802cbf0 con 0x7f28f8004d40 2024-03-06T07:26:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.832+0000 7f290effd640 1 -- 172.21.15.1:0/2773871020 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2908009c10 con 0x7f28f8004d40 2024-03-06T07:26:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.832+0000 7f2916a2a640 1 -- 172.21.15.1:0/2773871020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28f8004d40 msgr2=0x7f28f80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.832+0000 7f2916a2a640 1 --2- 172.21.15.1:0/2773871020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28f8004d40 0x7f28f80051c0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f290800e170 tx=0x7f28f8005830 comp rx=0 tx=0).stop 2024-03-06T07:26:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.832+0000 7f2914fc7640 1 -- 172.21.15.1:0/2773871020 reap_dead start 2024-03-06T07:26:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.832+0000 7f2916a2a640 1 -- 172.21.15.1:0/2773871020 shutdown_connections 2024-03-06T07:26:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.832+0000 7f2916a2a640 1 -- 172.21.15.1:0/2773871020 >> 172.21.15.1:0/2773871020 conn(0x7f29100c6860 msgr2=0x7f29100c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.832+0000 7f2916a2a640 1 -- 172.21.15.1:0/2773871020 shutdown_connections 2024-03-06T07:26:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.832+0000 7f2916a2a640 1 -- 172.21.15.1:0/2773871020 wait complete. 2024-03-06T07:26:59.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.833+0000 7f2916a2a640 1 Processor -- start 2024-03-06T07:26:59.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.834+0000 7f2916a2a640 1 -- start start 2024-03-06T07:26:59.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.834+0000 7f2916a2a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28f8004d40 0x7f29100abab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.834+0000 7f2916a2a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29100bc9b0 0x7f291014f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.834+0000 7f2916a2a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f291014f7e0 0x7f29101ea460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.834+0000 7f2916a2a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291006f620 con 0x7f29100bc9b0 2024-03-06T07:26:59.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.834+0000 7f2916a2a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291006f4a0 con 0x7f28f8004d40 2024-03-06T07:26:59.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.834+0000 7f2916a2a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2910156a10 con 0x7f291014f7e0 2024-03-06T07:26:59.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.834+0000 7f290f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29100bc9b0 0x7f291014f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.834+0000 7f290f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29100bc9b0 0x7f291014f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45186/0 (socket says 172.21.15.1:45186) 2024-03-06T07:26:59.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.834+0000 7f290f7fe640 1 -- 172.21.15.1:0/3232667707 learned_addr learned my addr 172.21.15.1:0/3232667707 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:26:59.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.834+0000 7f2914fc7640 1 --2- 172.21.15.1:0/3232667707 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f291014f7e0 0x7f29101ea460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.834+0000 7f290ffff640 1 --2- 172.21.15.1:0/3232667707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28f8004d40 0x7f29100abab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.835+0000 7f290f7fe640 1 -- 172.21.15.1:0/3232667707 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f291014f7e0 msgr2=0x7f29101ea460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.835+0000 7f290f7fe640 1 --2- 172.21.15.1:0/3232667707 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f291014f7e0 0x7f29101ea460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.835+0000 7f290f7fe640 1 -- 172.21.15.1:0/3232667707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28f8004d40 msgr2=0x7f29100abab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.835+0000 7f290f7fe640 1 --2- 172.21.15.1:0/3232667707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28f8004d40 0x7f29100abab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.835+0000 7f290f7fe640 1 -- 172.21.15.1:0/3232667707 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29101ec890 con 0x7f29100bc9b0 2024-03-06T07:26:59.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.835+0000 7f290ffff640 1 --2- 172.21.15.1:0/3232667707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28f8004d40 0x7f29100abab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:59.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.835+0000 7f290f7fe640 1 --2- 172.21.15.1:0/3232667707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29100bc9b0 0x7f291014f2a0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f2900011960 tx=0x7f2900011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.835+0000 7f290cff9640 1 -- 172.21.15.1:0/3232667707 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290000faf0 con 0x7f29100bc9b0 2024-03-06T07:26:59.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.835+0000 7f290cff9640 1 -- 172.21.15.1:0/3232667707 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290000fc90 con 0x7f29100bc9b0 2024-03-06T07:26:59.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.835+0000 7f2916a2a640 1 -- 172.21.15.1:0/3232667707 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29101ec5b0 con 0x7f29100bc9b0 2024-03-06T07:26:59.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.835+0000 7f290cff9640 1 -- 172.21.15.1:0/3232667707 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2900014440 con 0x7f29100bc9b0 2024-03-06T07:26:59.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.835+0000 7f2916a2a640 1 -- 172.21.15.1:0/3232667707 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29101e9c60 con 0x7f29100bc9b0 2024-03-06T07:26:59.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.836+0000 7f2916a2a640 1 -- 172.21.15.1:0/3232667707 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291014fbc0 con 0x7f29100bc9b0 2024-03-06T07:26:59.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.836+0000 7f290cff9640 1 -- 172.21.15.1:0/3232667707 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2900012040 con 0x7f29100bc9b0 2024-03-06T07:26:59.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.837+0000 7f290cff9640 1 --2- 172.21.15.1:0/3232667707 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f28fc06b4d0 0x7f28fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.837+0000 7f290cff9640 1 -- 172.21.15.1:0/3232667707 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f290008a740 con 0x7f29100bc9b0 2024-03-06T07:26:59.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.837+0000 7f290ffff640 1 --2- 172.21.15.1:0/3232667707 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f28fc06b4d0 0x7f28fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.837+0000 7f290ffff640 1 --2- 172.21.15.1:0/3232667707 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f28fc06b4d0 0x7f28fc06d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f2904007fd0 tx=0x7f2904009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:26:59.841+0000 7f290cff9640 1 -- 172.21.15.1:0/3232667707 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f291014fbc0 con 0x7f29100bc9b0 2024-03-06T07:27:00.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.059+0000 7f2916a2a640 1 -- 172.21.15.1:0/3232667707 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2910067fd0 con 0x7f29100bc9b0 2024-03-06T07:27:00.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.060+0000 7f290cff9640 1 -- 172.21.15.1:0/3232667707 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+3698 (secure 0 0 0) 0x7f2900019030 con 0x7f29100bc9b0 2024-03-06T07:27:00.062 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:00.062 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":153,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/1387389703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":1387389703},{"type":"v1","addr":"172.21.15.98:6836","nonce":1387389703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":8137,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2268963727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2268963727},{"type":"v1","addr":"172.21.15.98:6837","nonce":2268963727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":9162,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/4198928111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4198928111},{"type":"v1","addr":"172.21.15.1:6833","nonce":4198928111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143},{"gid":9396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/4274948232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":4274948232},{"type":"v1","addr":"172.21.15.1:6835","nonce":4274948232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[]} 2024-03-06T07:27:00.063 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 153 2024-03-06T07:27:00.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.062+0000 7f2916a2a640 1 -- 172.21.15.1:0/3232667707 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f28fc06b4d0 msgr2=0x7f28fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.062+0000 7f2916a2a640 1 --2- 172.21.15.1:0/3232667707 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f28fc06b4d0 0x7f28fc06d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f2904007fd0 tx=0x7f2904009040 comp rx=0 tx=0).stop 2024-03-06T07:27:00.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.062+0000 7f2916a2a640 1 -- 172.21.15.1:0/3232667707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29100bc9b0 msgr2=0x7f291014f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.062+0000 7f2916a2a640 1 --2- 172.21.15.1:0/3232667707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29100bc9b0 0x7f291014f2a0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f2900011960 tx=0x7f2900011e20 comp rx=0 tx=0).stop 2024-03-06T07:27:00.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.062+0000 7f2916a2a640 1 -- 172.21.15.1:0/3232667707 shutdown_connections 2024-03-06T07:27:00.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.062+0000 7f2916a2a640 1 --2- 172.21.15.1:0/3232667707 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f28fc06b4d0 0x7f28fc06d990 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.062+0000 7f2916a2a640 1 --2- 172.21.15.1:0/3232667707 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f291014f7e0 0x7f29101ea460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.062+0000 7f2916a2a640 1 --2- 172.21.15.1:0/3232667707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29100bc9b0 0x7f291014f2a0 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.062+0000 7f2916a2a640 1 --2- 172.21.15.1:0/3232667707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28f8004d40 0x7f29100abab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.062+0000 7f2916a2a640 1 -- 172.21.15.1:0/3232667707 >> 172.21.15.1:0/3232667707 conn(0x7f29100c6860 msgr2=0x7f29100a8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.062+0000 7f2916a2a640 1 -- 172.21.15.1:0/3232667707 shutdown_connections 2024-03-06T07:27:00.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.062+0000 7f2916a2a640 1 -- 172.21.15.1:0/3232667707 wait complete. 2024-03-06T07:27:00.075 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:00.075 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:27:00.075 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:27:00.075 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:00.076 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:00.076 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:27:00.076 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:27:00.076 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:00.076 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:00.076 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:27:00.076 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:27:00.077 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:00.077 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:00.077 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:27:00.077 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:27:00.077 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:00.085 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:27:00.085 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:00.087 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:27:00.087 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:00.088 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:27:00.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:00.089 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:27:00.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:00.128 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:00.128 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:00.129 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:00.129 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:00.131 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:00.131 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:00.132 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:00.132 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:00.132 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:27:00.205 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:27:00.206 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:27:00.214 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:27:00.214 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:27:00.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.239+0000 7fa895b79640 1 Processor -- start 2024-03-06T07:27:00.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.239+0000 7fa895b79640 1 -- start start 2024-03-06T07:27:00.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.240+0000 7fa895b79640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa890152820 0x7fa8900c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.240+0000 7fa895b79640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa8900af6f0 0x7fa89015fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.240+0000 7fa895b79640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8900c7180 con 0x7fa8900c0670 2024-03-06T07:27:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.240+0000 7fa895b79640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa890155460 con 0x7fa8900af6f0 2024-03-06T07:27:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.240+0000 7fa895b79640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8900c6bc0 con 0x7fa890152820 2024-03-06T07:27:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.240+0000 7fa88f7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa890152820 0x7fa8900c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.240+0000 7fa88effd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa8900c0670 0x7fa8900c73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:52634/0 (socket says 172.21.15.1:52634) 2024-03-06T07:27:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.240+0000 7fa88effd640 1 -- 172.21.15.1:0/293431844 learned_addr learned my addr 172.21.15.1:0/293431844 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.240+0000 7fa88ffff640 1 --2- 172.21.15.1:0/293431844 >> v2:172.21.15.98:3300/0 conn(0x7fa8900af6f0 0x7fa89015fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.240+0000 7fa88e7fc640 1 -- 172.21.15.1:0/293431844 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397213128 0 0) 0x7fa8900c7180 con 0x7fa8900c0670 2024-03-06T07:27:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.240+0000 7fa88f7fe640 1 -- 172.21.15.1:0/293431844 >> v2:172.21.15.98:3300/0 conn(0x7fa8900af6f0 msgr2=0x7fa89015fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.240+0000 7fa88f7fe640 1 --2- 172.21.15.1:0/293431844 >> v2:172.21.15.98:3300/0 conn(0x7fa8900af6f0 0x7fa89015fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.240+0000 7fa88f7fe640 1 -- 172.21.15.1:0/293431844 >> v1:172.21.15.98:6789/0 conn(0x7fa8900c0670 legacy=0x7fa8900c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.241+0000 7fa88f7fe640 1 -- 172.21.15.1:0/293431844 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8900c6ea0 con 0x7fa890152820 2024-03-06T07:27:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.241+0000 7fa88ffff640 1 --2- 172.21.15.1:0/293431844 >> v2:172.21.15.98:3300/0 conn(0x7fa8900af6f0 0x7fa89015fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:00.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.241+0000 7fa88f7fe640 1 --2- 172.21.15.1:0/293431844 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa890152820 0x7fa8900c0130 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa8800097a0 tx=0x7fa88002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71a2fa733f473cb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.242+0000 7fa88e7fc640 1 -- 172.21.15.1:0/293431844 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa880035020 con 0x7fa890152820 2024-03-06T07:27:00.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.242+0000 7fa88e7fc640 1 -- 172.21.15.1:0/293431844 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa880031830 con 0x7fa890152820 2024-03-06T07:27:00.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.242+0000 7fa88e7fc640 1 -- 172.21.15.1:0/293431844 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa880031c00 con 0x7fa890152820 2024-03-06T07:27:00.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.242+0000 7fa895b79640 1 -- 172.21.15.1:0/293431844 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa890152820 msgr2=0x7fa8900c0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.242+0000 7fa895b79640 1 --2- 172.21.15.1:0/293431844 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa890152820 0x7fa8900c0130 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa8800097a0 tx=0x7fa88002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.242+0000 7fa895b79640 1 -- 172.21.15.1:0/293431844 shutdown_connections 2024-03-06T07:27:00.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.242+0000 7fa895b79640 1 --2- 172.21.15.1:0/293431844 >> v2:172.21.15.98:3300/0 conn(0x7fa8900af6f0 0x7fa89015fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.242+0000 7fa895b79640 1 --2- 172.21.15.1:0/293431844 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa890152820 0x7fa8900c0130 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.242+0000 7fa895b79640 1 -- 172.21.15.1:0/293431844 >> 172.21.15.1:0/293431844 conn(0x7fa890072720 msgr2=0x7fa890072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.242+0000 7fa895b79640 1 -- 172.21.15.1:0/293431844 shutdown_connections 2024-03-06T07:27:00.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.242+0000 7fa895b79640 1 -- 172.21.15.1:0/293431844 wait complete. 2024-03-06T07:27:00.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.243+0000 7fa895b79640 1 Processor -- start 2024-03-06T07:27:00.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.244+0000 7fa895b79640 1 -- start start 2024-03-06T07:27:00.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.244+0000 7fa895b79640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8900af6f0 0x7fa8900c3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.244+0000 7fa895b79640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8900c0670 0x7fa89014a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.244+0000 7fa895b79640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa890152820 0x7fa89014ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.244+0000 7fa895b79640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8900c7c80 con 0x7fa8900c0670 2024-03-06T07:27:00.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.244+0000 7fa895b79640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8900c7b00 con 0x7fa8900af6f0 2024-03-06T07:27:00.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.244+0000 7fa895b79640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa890152c00 con 0x7fa890152820 2024-03-06T07:27:00.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.244+0000 7fa88effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8900c0670 0x7fa89014a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.245+0000 7fa88effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8900c0670 0x7fa89014a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45222/0 (socket says 172.21.15.1:45222) 2024-03-06T07:27:00.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.245+0000 7fa88effd640 1 -- 172.21.15.1:0/3112623408 learned_addr learned my addr 172.21.15.1:0/3112623408 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:00.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.245+0000 7fa88ffff640 1 --2- 172.21.15.1:0/3112623408 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa890152820 0x7fa89014ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.245+0000 7fa88f7fe640 1 --2- 172.21.15.1:0/3112623408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8900af6f0 0x7fa8900c3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.245+0000 7fa88effd640 1 -- 172.21.15.1:0/3112623408 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa890152820 msgr2=0x7fa89014ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.245+0000 7fa88effd640 1 --2- 172.21.15.1:0/3112623408 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa890152820 0x7fa89014ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.245+0000 7fa88effd640 1 -- 172.21.15.1:0/3112623408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8900af6f0 msgr2=0x7fa8900c3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.245+0000 7fa88effd640 1 --2- 172.21.15.1:0/3112623408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8900af6f0 0x7fa8900c3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.245+0000 7fa88effd640 1 -- 172.21.15.1:0/3112623408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8901eac10 con 0x7fa8900c0670 2024-03-06T07:27:00.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.245+0000 7fa88effd640 1 --2- 172.21.15.1:0/3112623408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8900c0670 0x7fa89014a8c0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fa88400c460 tx=0x7fa88400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.245+0000 7fa877fff640 1 -- 172.21.15.1:0/3112623408 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884018070 con 0x7fa8900c0670 2024-03-06T07:27:00.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.245+0000 7fa877fff640 1 -- 172.21.15.1:0/3112623408 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa884007520 con 0x7fa8900c0670 2024-03-06T07:27:00.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.245+0000 7fa895b79640 1 -- 172.21.15.1:0/3112623408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8901eaed0 con 0x7fa8900c0670 2024-03-06T07:27:00.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.245+0000 7fa877fff640 1 -- 172.21.15.1:0/3112623408 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884004a20 con 0x7fa8900c0670 2024-03-06T07:27:00.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.245+0000 7fa895b79640 1 -- 172.21.15.1:0/3112623408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8901eb490 con 0x7fa8900c0670 2024-03-06T07:27:00.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.246+0000 7fa895b79640 1 -- 172.21.15.1:0/3112623408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8900c1e60 con 0x7fa8900c0670 2024-03-06T07:27:00.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.247+0000 7fa877fff640 1 -- 172.21.15.1:0/3112623408 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa88401b070 con 0x7fa8900c0670 2024-03-06T07:27:00.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.247+0000 7fa877fff640 1 --2- 172.21.15.1:0/3112623408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa87c06b4d0 0x7fa87c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.247+0000 7fa877fff640 1 -- 172.21.15.1:0/3112623408 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fa884089590 con 0x7fa8900c0670 2024-03-06T07:27:00.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.247+0000 7fa88f7fe640 1 --2- 172.21.15.1:0/3112623408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa87c06b4d0 0x7fa87c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.248+0000 7fa88f7fe640 1 --2- 172.21.15.1:0/3112623408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa87c06b4d0 0x7fa87c06d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa880036000 tx=0x7fa88003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.252+0000 7fa877fff640 1 -- 172.21.15.1:0/3112623408 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8900c1e60 con 0x7fa8900c0670 2024-03-06T07:27:00.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:00.443+0000 7fa895b79640 1 -- 172.21.15.1:0/3112623408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa8900686e0 con 0x7fa8900c0670 2024-03-06T07:27:01.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.239+0000 7fa877fff640 1 -- 172.21.15.1:0/3112623408 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v167) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa884058430 con 0x7fa8900c0670 2024-03-06T07:27:01.241 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-06T07:27:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.243+0000 7fa895b79640 1 -- 172.21.15.1:0/3112623408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa87c06b4d0 msgr2=0x7fa87c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.243+0000 7fa895b79640 1 --2- 172.21.15.1:0/3112623408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa87c06b4d0 0x7fa87c06d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa880036000 tx=0x7fa88003c040 comp rx=0 tx=0).stop 2024-03-06T07:27:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.243+0000 7fa895b79640 1 -- 172.21.15.1:0/3112623408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8900c0670 msgr2=0x7fa89014a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.243+0000 7fa895b79640 1 --2- 172.21.15.1:0/3112623408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8900c0670 0x7fa89014a8c0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fa88400c460 tx=0x7fa88400c920 comp rx=0 tx=0).stop 2024-03-06T07:27:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.243+0000 7fa895b79640 1 -- 172.21.15.1:0/3112623408 shutdown_connections 2024-03-06T07:27:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.243+0000 7fa895b79640 1 --2- 172.21.15.1:0/3112623408 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa87c06b4d0 0x7fa87c06d990 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.243+0000 7fa895b79640 1 --2- 172.21.15.1:0/3112623408 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa890152820 0x7fa89014ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.243+0000 7fa895b79640 1 --2- 172.21.15.1:0/3112623408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8900c0670 0x7fa89014a8c0 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.243+0000 7fa895b79640 1 --2- 172.21.15.1:0/3112623408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8900af6f0 0x7fa8900c3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.243+0000 7fa895b79640 1 -- 172.21.15.1:0/3112623408 >> 172.21.15.1:0/3112623408 conn(0x7fa890072720 msgr2=0x7fa89006cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.243+0000 7fa895b79640 1 -- 172.21.15.1:0/3112623408 shutdown_connections 2024-03-06T07:27:01.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.243+0000 7fa895b79640 1 -- 172.21.15.1:0/3112623408 wait complete. 2024-03-06T07:27:01.258 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:27:01.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.364+0000 7fbacb69d640 1 Processor -- start 2024-03-06T07:27:01.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.364+0000 7fbacb69d640 1 -- start start 2024-03-06T07:27:01.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.365+0000 7fbacb69d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbac406ea50 0x7fbac40c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.365+0000 7fbacb69d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac406f060 con 0x7fbac406e670 2024-03-06T07:27:01.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.365+0000 7fbacb69d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac40c26e0 con 0x7fbac40a8520 2024-03-06T07:27:01.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.365+0000 7fbacb69d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac40c2ca0 con 0x7fbac406ea50 2024-03-06T07:27:01.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.365+0000 7fbac9c3a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbac406ea50 0x7fbac40c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.365+0000 7fbac9c3a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbac406ea50 0x7fbac40c1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45224/0 (socket says 172.21.15.1:45224) 2024-03-06T07:27:01.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.365+0000 7fbac9c3a640 1 -- 172.21.15.1:0/1446406770 learned_addr learned my addr 172.21.15.1:0/1446406770 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:01.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.365+0000 7fbac9c3a640 1 -- 172.21.15.1:0/1446406770 >> v1:172.21.15.98:6789/0 conn(0x7fbac40a8520 legacy=0x7fbac4153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.365+0000 7fbac9c3a640 1 -- 172.21.15.1:0/1446406770 >> v1:172.21.15.1:6789/0 conn(0x7fbac406e670 legacy=0x7fbac4150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.365+0000 7fbac9c3a640 1 -- 172.21.15.1:0/1446406770 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac40c29c0 con 0x7fbac406ea50 2024-03-06T07:27:01.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.365+0000 7fbabbfff640 1 -- 172.21.15.1:0/1446406770 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919486275 0 0) 0x7fbac406f060 con 0x7fbac406e670 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.366+0000 7fbac9c3a640 1 --2- 172.21.15.1:0/1446406770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbac406ea50 0x7fbac40c1bc0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009e50 tx=0x7fbabc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c248a91d587c0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.366+0000 7fbabbfff640 1 -- 172.21.15.1:0/1446406770 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc02cb20 con 0x7fbac406ea50 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.366+0000 7fbabbfff640 1 -- 172.21.15.1:0/1446406770 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbabc02ccc0 con 0x7fbac406ea50 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.366+0000 7fbabbfff640 1 -- 172.21.15.1:0/1446406770 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc034610 con 0x7fbac406ea50 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.367+0000 7fbacb69d640 1 -- 172.21.15.1:0/1446406770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbac406ea50 msgr2=0x7fbac40c1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.367+0000 7fbacb69d640 1 --2- 172.21.15.1:0/1446406770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbac406ea50 0x7fbac40c1bc0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009e50 tx=0x7fbabc02bf80 comp rx=0 tx=0).stop 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.367+0000 7fbacb69d640 1 -- 172.21.15.1:0/1446406770 shutdown_connections 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.367+0000 7fbacb69d640 1 --2- 172.21.15.1:0/1446406770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbac406ea50 0x7fbac40c1bc0 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.367+0000 7fbacb69d640 1 -- 172.21.15.1:0/1446406770 >> 172.21.15.1:0/1446406770 conn(0x7fbac40c3cc0 msgr2=0x7fbac40c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.367+0000 7fbacb69d640 1 -- 172.21.15.1:0/1446406770 shutdown_connections 2024-03-06T07:27:01.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.367+0000 7fbacb69d640 1 -- 172.21.15.1:0/1446406770 wait complete. 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.368+0000 7fbacb69d640 1 Processor -- start 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.368+0000 7fbacb69d640 1 -- start start 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbacb69d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbac406e670 0x7fbac414f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbacb69d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbac406ea50 0x7fbac4152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbacb69d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbac40a8520 0x7fbac41ea5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbacb69d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac4153b20 con 0x7fbac40a8520 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbacb69d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac41539a0 con 0x7fbac406e670 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbacb69d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac4153ca0 con 0x7fbac406ea50 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbac9c3a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbac40a8520 0x7fbac41ea5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbac9c3a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbac40a8520 0x7fbac41ea5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45226/0 (socket says 172.21.15.1:45226) 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbac9c3a640 1 -- 172.21.15.1:0/2060975874 learned_addr learned my addr 172.21.15.1:0/2060975874 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbac9439640 1 --2- 172.21.15.1:0/2060975874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbac406e670 0x7fbac414f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbac8c38640 1 --2- 172.21.15.1:0/2060975874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbac406ea50 0x7fbac4152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbac9c3a640 1 -- 172.21.15.1:0/2060975874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbac406ea50 msgr2=0x7fbac4152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbac9c3a640 1 --2- 172.21.15.1:0/2060975874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbac406ea50 0x7fbac4152530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbac9c3a640 1 -- 172.21.15.1:0/2060975874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbac406e670 msgr2=0x7fbac414f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbac9c3a640 1 --2- 172.21.15.1:0/2060975874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbac406e670 0x7fbac414f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbac9c3a640 1 -- 172.21.15.1:0/2060975874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac40ac420 con 0x7fbac40a8520 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbac8c38640 1 --2- 172.21.15.1:0/2060975874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbac406ea50 0x7fbac4152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.369+0000 7fbac9c3a640 1 --2- 172.21.15.1:0/2060975874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbac40a8520 0x7fbac41ea5f0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009e50 tx=0x7fbabc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.370+0000 7fbab9ffb640 1 -- 172.21.15.1:0/2060975874 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc043030 con 0x7fbac40a8520 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.370+0000 7fbab9ffb640 1 -- 172.21.15.1:0/2060975874 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbabc003990 con 0x7fbac40a8520 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.370+0000 7fbacb69d640 1 -- 172.21.15.1:0/2060975874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac40ac140 con 0x7fbac40a8520 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.370+0000 7fbab9ffb640 1 -- 172.21.15.1:0/2060975874 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc003b60 con 0x7fbac40a8520 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.370+0000 7fbacb69d640 1 -- 172.21.15.1:0/2060975874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac40aa7d0 con 0x7fbac40a8520 2024-03-06T07:27:01.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.371+0000 7fbacb69d640 1 -- 172.21.15.1:0/2060975874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac4067fd0 con 0x7fbac40a8520 2024-03-06T07:27:01.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.371+0000 7fbab9ffb640 1 -- 172.21.15.1:0/2060975874 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbabc02cb30 con 0x7fbac40a8520 2024-03-06T07:27:01.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.372+0000 7fbab9ffb640 1 --2- 172.21.15.1:0/2060975874 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbaa806b4d0 0x7fbaa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.372+0000 7fbab9ffb640 1 -- 172.21.15.1:0/2060975874 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbabc0ac3b0 con 0x7fbac40a8520 2024-03-06T07:27:01.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.372+0000 7fbac9439640 1 --2- 172.21.15.1:0/2060975874 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbaa806b4d0 0x7fbaa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.372+0000 7fbac9439640 1 --2- 172.21.15.1:0/2060975874 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbaa806b4d0 0x7fbaa806d990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fbac0000f60 tx=0x7fbac0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.376+0000 7fbab9ffb640 1 -- 172.21.15.1:0/2060975874 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbabc04d020 con 0x7fbac40a8520 2024-03-06T07:27:01.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.626+0000 7fbacb69d640 1 -- 172.21.15.1:0/2060975874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbac4150320 con 0x7fbac40a8520 2024-03-06T07:27:01.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.626+0000 7fbab9ffb640 1 -- 172.21.15.1:0/2060975874 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+4724 (secure 0 0 0) 0x7fbabc0aae20 con 0x7fbac40a8520 2024-03-06T07:27:01.628 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:01.628 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:27:01.629 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-06T07:27:01.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:01.629 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-06T07:27:01.629 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlnxyWARAAKIPXU0IafNQ1RzBPX+3DYw==", 2024-03-06T07:27:01.629 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:01.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:27:01.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:27:01.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:27:01.629 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:01.629 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:01.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:01.629 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-06T07:27:01.630 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl2fXxDRAAuLqH+HpKcN1JTts9ZtDTXA==", 2024-03-06T07:27:01.630 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:01.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:27:01.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:27:01.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:27:01.630 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:01.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:01.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:01.630 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-06T07:27:01.630 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl+IzqCBAAmgLkarFGt9Ihs/8kJYicBQ==", 2024-03-06T07:27:01.630 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:01.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:27:01.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:27:01.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:27:01.631 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:01.631 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:01.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:01.631 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-06T07:27:01.631 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlipdRFRAAA34vrsTEoX3L1CKb1YAV9g==", 2024-03-06T07:27:01.631 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:01.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:27:01.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:27:01.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:27:01.632 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:01.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:01.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:01.632 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-06T07:27:01.632 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAGGehlQK39LxAAOpPIZvl3FqQwCgyo3yoOGQ==", 2024-03-06T07:27:01.632 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:01.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:01.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:27:01.632 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:01.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:01.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:01.633 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-06T07:27:01.633 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAJGehlTxwDLhAAMwN9E0+3Jh5XNXnLo+fw3g==", 2024-03-06T07:27:01.633 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:01.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:01.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:27:01.633 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:01.633 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:01.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:01.633 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-06T07:27:01.633 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAMGehl4qrlLBAAm0VWQRNvSQIpDx5xf4Fyig==", 2024-03-06T07:27:01.633 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:01.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:01.634 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:27:01.634 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:01.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:01.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:01.634 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-06T07:27:01.634 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAPGehllO5HKxAAozm6ycG+fRxatoZTogeJTA==", 2024-03-06T07:27:01.634 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:01.634 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:01.634 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:27:01.634 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:01.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:01.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:01.635 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-06T07:27:01.635 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAVGehlLtLBMhAAUrV6q6ZCQluIcoo/S/NdbA==", 2024-03-06T07:27:01.635 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:01.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:01.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:27:01.635 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:01.635 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:01.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:01.635 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-06T07:27:01.635 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAYGehlHp59MRAAxxgpSPTEZkYKjl3fr0xeeQ==", 2024-03-06T07:27:01.635 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:01.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:01.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:27:01.636 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:01.636 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:01.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:01.636 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-06T07:27:01.636 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAbGehlTnXcMBAAFOBqcnh5ubAz+o6AGjccWQ==", 2024-03-06T07:27:01.636 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:01.636 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:01.636 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:27:01.636 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:01.636 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:01.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:01.636 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-06T07:27:01.636 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAeGehl1qH1LhAAaSl7Aj7qlN6d4xt5SKAmcg==", 2024-03-06T07:27:01.637 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:01.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:01.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:27:01.637 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:01.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:01.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:01.637 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-06T07:27:01.637 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng==", 2024-03-06T07:27:01.637 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:01.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:27:01.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-06T07:27:01.637 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:01.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:01.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:01.638 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-06T07:27:01.638 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlMyv7HBAAi7vt7aZWEaZuMpFrB5ofZw==", 2024-03-06T07:27:01.638 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:01.638 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:27:01.638 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-06T07:27:01.638 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-06T07:27:01.638 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-06T07:27:01.638 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:01.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:01.638 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-06T07:27:01.638 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlBS/vIBAAN86N2K4ze5PJB8kR869Lfg==", 2024-03-06T07:27:01.638 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:01.638 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:27:01.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:27:01.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:27:01.639 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:01.639 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:01.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:01.639 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-06T07:27:01.639 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlJ/jPORAAhLk5MTGafsGKhZSYhvhBZg==", 2024-03-06T07:27:01.639 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:01.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:27:01.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:01.639 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:01.639 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:01.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:01.640 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-06T07:27:01.640 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlAl8rNhAAvdU0Qvo/FJYnxc4/H8MIqQ==", 2024-03-06T07:27:01.640 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:01.640 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:27:01.640 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:01.640 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:01.640 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.640 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:01.640 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-06T07:27:01.640 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.628+0000 7fbacb69d640 1 -- 172.21.15.1:0/2060975874 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbaa806b4d0 msgr2=0x7fbaa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.628+0000 7fbacb69d640 1 --2- 172.21.15.1:0/2060975874 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbaa806b4d0 0x7fbaa806d990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fbac0000f60 tx=0x7fbac0009040 comp rx=0 tx=0).stop 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.628+0000 7fbacb69d640 1 -- 172.21.15.1:0/2060975874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbac40a8520 msgr2=0x7fbac41ea5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.628+0000 7fbacb69d640 1 --2- 172.21.15.1:0/2060975874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbac40a8520 0x7fbac41ea5f0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009e50 tx=0x7fbabc004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.628+0000 7fbacb69d640 1 -- 172.21.15.1:0/2060975874 shutdown_connections 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.628+0000 7fbacb69d640 1 --2- 172.21.15.1:0/2060975874 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbaa806b4d0 0x7fbaa806d990 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.628+0000 7fbacb69d640 1 --2- 172.21.15.1:0/2060975874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbac40a8520 0x7fbac41ea5f0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.628+0000 7fbacb69d640 1 --2- 172.21.15.1:0/2060975874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbac406ea50 0x7fbac4152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.628+0000 7fbacb69d640 1 --2- 172.21.15.1:0/2060975874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbac406e670 0x7fbac414f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.628+0000 7fbacb69d640 1 -- 172.21.15.1:0/2060975874 >> 172.21.15.1:0/2060975874 conn(0x7fbac40c3cc0 msgr2=0x7fbac40a9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.628+0000 7fbacb69d640 1 -- 172.21.15.1:0/2060975874 shutdown_connections 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.628+0000 7fbacb69d640 1 -- 172.21.15.1:0/2060975874 wait complete. 2024-03-06T07:27:01.643 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:01.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.742+0000 7f3c25f3e640 1 Processor -- start 2024-03-06T07:27:01.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.742+0000 7f3c25f3e640 1 -- start start 2024-03-06T07:27:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.743+0000 7f3c25f3e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3c20150fe0 0x7f3c201513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.743+0000 7f3c25f3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c20072f20 0x7f3c2006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.743+0000 7f3c25f3e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c200a6140 con 0x7f3c2006f2f0 2024-03-06T07:27:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.743+0000 7f3c25f3e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c201526b0 con 0x7f3c20150fe0 2024-03-06T07:27:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.743+0000 7f3c25f3e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c20151aa0 con 0x7f3c20072f20 2024-03-06T07:27:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.743+0000 7f3c1f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3c20150fe0 0x7f3c201513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.743+0000 7f3c1f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3c20150fe0 0x7f3c201513c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45240/0 (socket says 172.21.15.1:45240) 2024-03-06T07:27:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.743+0000 7f3c1f7fe640 1 -- 172.21.15.1:0/496677745 learned_addr learned my addr 172.21.15.1:0/496677745 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.743+0000 7f3c1effd640 1 --2- 172.21.15.1:0/496677745 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c20072f20 0x7f3c2006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.743+0000 7f3c1f7fe640 1 -- 172.21.15.1:0/496677745 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c20072f20 msgr2=0x7f3c2006edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.743+0000 7f3c1f7fe640 1 --2- 172.21.15.1:0/496677745 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c20072f20 0x7f3c2006edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.743+0000 7f3c1f7fe640 1 -- 172.21.15.1:0/496677745 >> v1:172.21.15.1:6789/0 conn(0x7f3c2006f2f0 legacy=0x7f3c20071f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.743+0000 7f3c1f7fe640 1 -- 172.21.15.1:0/496677745 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c20151d80 con 0x7f3c20150fe0 2024-03-06T07:27:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.743+0000 7f3c1e7fc640 1 -- 172.21.15.1:0/496677745 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184025830 0 0) 0x7f3c200a6140 con 0x7f3c2006f2f0 2024-03-06T07:27:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.743+0000 7f3c1effd640 1 --2- 172.21.15.1:0/496677745 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c20072f20 0x7f3c2006edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.744+0000 7f3c1f7fe640 1 --2- 172.21.15.1:0/496677745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c20150fe0 0x7f3c201513c0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f3c10009870 tx=0x7f3c1002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e296487269502a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.744+0000 7f3c1e7fc640 1 -- 172.21.15.1:0/496677745 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c10035020 con 0x7f3c20150fe0 2024-03-06T07:27:01.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.744+0000 7f3c1e7fc640 1 -- 172.21.15.1:0/496677745 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c10031640 con 0x7f3c20150fe0 2024-03-06T07:27:01.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.744+0000 7f3c1e7fc640 1 -- 172.21.15.1:0/496677745 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1002d040 con 0x7f3c20150fe0 2024-03-06T07:27:01.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.745+0000 7f3c25f3e640 1 -- 172.21.15.1:0/496677745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c20150fe0 msgr2=0x7f3c201513c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.745+0000 7f3c25f3e640 1 --2- 172.21.15.1:0/496677745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c20150fe0 0x7f3c201513c0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f3c10009870 tx=0x7f3c1002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.745+0000 7f3c25f3e640 1 -- 172.21.15.1:0/496677745 shutdown_connections 2024-03-06T07:27:01.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.745+0000 7f3c25f3e640 1 --2- 172.21.15.1:0/496677745 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c20072f20 0x7f3c2006edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.745+0000 7f3c25f3e640 1 --2- 172.21.15.1:0/496677745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c20150fe0 0x7f3c201513c0 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.745+0000 7f3c25f3e640 1 -- 172.21.15.1:0/496677745 >> 172.21.15.1:0/496677745 conn(0x7f3c200c6860 msgr2=0x7f3c200c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.745+0000 7f3c25f3e640 1 -- 172.21.15.1:0/496677745 shutdown_connections 2024-03-06T07:27:01.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.745+0000 7f3c25f3e640 1 -- 172.21.15.1:0/496677745 wait complete. 2024-03-06T07:27:01.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.746+0000 7f3c25f3e640 1 Processor -- start 2024-03-06T07:27:01.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.746+0000 7f3c25f3e640 1 -- start start 2024-03-06T07:27:01.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.747+0000 7f3c25f3e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c2006f2f0 0x7f3c2014adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.747+0000 7f3c25f3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c20072f20 0x7f3c2014e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.747+0000 7f3c25f3e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c20150fe0 0x7f3c201e6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.747+0000 7f3c25f3e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c200726a0 con 0x7f3c20150fe0 2024-03-06T07:27:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.747+0000 7f3c25f3e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c20152340 con 0x7f3c2006f2f0 2024-03-06T07:27:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.747+0000 7f3c25f3e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c20072820 con 0x7f3c20072f20 2024-03-06T07:27:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.747+0000 7f3c1ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c20150fe0 0x7f3c201e6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.747+0000 7f3c1ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c20150fe0 0x7f3c201e6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45250/0 (socket says 172.21.15.1:45250) 2024-03-06T07:27:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.747+0000 7f3c1ffff640 1 -- 172.21.15.1:0/1631660106 learned_addr learned my addr 172.21.15.1:0/1631660106 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.747+0000 7f3c1f7fe640 1 --2- 172.21.15.1:0/1631660106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c2006f2f0 0x7f3c2014adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.747+0000 7f3c1effd640 1 --2- 172.21.15.1:0/1631660106 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c20072f20 0x7f3c2014e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.747+0000 7f3c1ffff640 1 -- 172.21.15.1:0/1631660106 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c20072f20 msgr2=0x7f3c2014e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.747+0000 7f3c1ffff640 1 --2- 172.21.15.1:0/1631660106 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c20072f20 0x7f3c2014e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.747+0000 7f3c1ffff640 1 -- 172.21.15.1:0/1631660106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c2006f2f0 msgr2=0x7f3c2014adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.747+0000 7f3c1ffff640 1 --2- 172.21.15.1:0/1631660106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c2006f2f0 0x7f3c2014adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.747+0000 7f3c1ffff640 1 -- 172.21.15.1:0/1631660106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c201e67c0 con 0x7f3c20150fe0 2024-03-06T07:27:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.748+0000 7f3c1ffff640 1 --2- 172.21.15.1:0/1631660106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c20150fe0 0x7f3c201e6ca0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f3c1800c760 tx=0x7f3c1800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.748+0000 7f3c07fff640 1 -- 172.21.15.1:0/1631660106 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c18003da0 con 0x7f3c20150fe0 2024-03-06T07:27:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.748+0000 7f3c07fff640 1 -- 172.21.15.1:0/1631660106 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c18005070 con 0x7f3c20150fe0 2024-03-06T07:27:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.748+0000 7f3c25f3e640 1 -- 172.21.15.1:0/1631660106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c201ea980 con 0x7f3c20150fe0 2024-03-06T07:27:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.748+0000 7f3c07fff640 1 -- 172.21.15.1:0/1631660106 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c18005460 con 0x7f3c20150fe0 2024-03-06T07:27:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.748+0000 7f3c25f3e640 1 -- 172.21.15.1:0/1631660106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c200bc6e0 con 0x7f3c20150fe0 2024-03-06T07:27:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.749+0000 7f3c25f3e640 1 -- 172.21.15.1:0/1631660106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c201e6990 con 0x7f3c20150fe0 2024-03-06T07:27:01.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.749+0000 7f3c07fff640 1 -- 172.21.15.1:0/1631660106 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c18005600 con 0x7f3c20150fe0 2024-03-06T07:27:01.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.750+0000 7f3c07fff640 1 --2- 172.21.15.1:0/1631660106 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c0c06b4d0 0x7f3c0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.750+0000 7f3c07fff640 1 -- 172.21.15.1:0/1631660106 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f3c18089da0 con 0x7f3c20150fe0 2024-03-06T07:27:01.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.750+0000 7f3c1f7fe640 1 --2- 172.21.15.1:0/1631660106 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c0c06b4d0 0x7f3c0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.751+0000 7f3c1f7fe640 1 --2- 172.21.15.1:0/1631660106 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c0c06b4d0 0x7f3c0c06d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f3c10036000 tx=0x7f3c1003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.754+0000 7f3c07fff640 1 -- 172.21.15.1:0/1631660106 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c18058790 con 0x7f3c20150fe0 2024-03-06T07:27:01.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.936+0000 7f3c25f3e640 1 -- 172.21.15.1:0/1631660106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c20067fd0 con 0x7f3c20150fe0 2024-03-06T07:27:01.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.937+0000 7f3c07fff640 1 -- 172.21.15.1:0/1631660106 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) v1 ==== 74+0+13054 (secure 0 0 0) 0x7f3c1805c830 con 0x7f3c20150fe0 2024-03-06T07:27:01.939 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:01.939 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":167,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:27:01.238959+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:01.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.938+0000 7f3c25f3e640 1 -- 172.21.15.1:0/1631660106 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c0c06b4d0 msgr2=0x7f3c0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.938+0000 7f3c25f3e640 1 --2- 172.21.15.1:0/1631660106 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c0c06b4d0 0x7f3c0c06d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f3c10036000 tx=0x7f3c1003c040 comp rx=0 tx=0).stop 2024-03-06T07:27:01.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.938+0000 7f3c25f3e640 1 -- 172.21.15.1:0/1631660106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c20150fe0 msgr2=0x7f3c201e6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.938+0000 7f3c25f3e640 1 --2- 172.21.15.1:0/1631660106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c20150fe0 0x7f3c201e6ca0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f3c1800c760 tx=0x7f3c1800cc20 comp rx=0 tx=0).stop 2024-03-06T07:27:01.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.938+0000 7f3c25f3e640 1 -- 172.21.15.1:0/1631660106 shutdown_connections 2024-03-06T07:27:01.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.938+0000 7f3c25f3e640 1 --2- 172.21.15.1:0/1631660106 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c0c06b4d0 0x7f3c0c06d990 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.939+0000 7f3c25f3e640 1 --2- 172.21.15.1:0/1631660106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c20150fe0 0x7f3c201e6ca0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.939+0000 7f3c25f3e640 1 --2- 172.21.15.1:0/1631660106 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c20072f20 0x7f3c2014e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.939+0000 7f3c25f3e640 1 --2- 172.21.15.1:0/1631660106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c2006f2f0 0x7f3c2014adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.939+0000 7f3c25f3e640 1 -- 172.21.15.1:0/1631660106 >> 172.21.15.1:0/1631660106 conn(0x7f3c200c6860 msgr2=0x7f3c2014ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.939+0000 7f3c25f3e640 1 -- 172.21.15.1:0/1631660106 shutdown_connections 2024-03-06T07:27:01.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:01.939+0000 7f3c25f3e640 1 -- 172.21.15.1:0/1631660106 wait complete. 2024-03-06T07:27:01.952 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:01.953 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:02.057 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:02.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.055+0000 7f5c9fd0c640 1 Processor -- start 2024-03-06T07:27:02.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.055+0000 7f5c9fd0c640 1 -- start start 2024-03-06T07:27:02.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.056+0000 7f5c9fd0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980c2b30 0x7f5c980724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.056+0000 7f5c9fd0c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c98152b70 0x7f5c98072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.056+0000 7f5c9fd0c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5c980a5fa0 0x7f5c980c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.056+0000 7f5c9fd0c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c981526e0 con 0x7f5c98152b70 2024-03-06T07:27:02.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.056+0000 7f5c9fd0c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c98151dd0 con 0x7f5c980a5fa0 2024-03-06T07:27:02.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.056+0000 7f5c9fd0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c98156190 con 0x7f5c980c2b30 2024-03-06T07:27:02.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.056+0000 7f5c9d2a7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c98152b70 0x7f5c98072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.056+0000 7f5c9d2a7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c98152b70 0x7f5c98072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45264/0 (socket says 172.21.15.1:45264) 2024-03-06T07:27:02.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.056+0000 7f5c9d2a7640 1 -- 172.21.15.1:0/405461371 learned_addr learned my addr 172.21.15.1:0/405461371 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:02.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.056+0000 7f5c9daa8640 1 --2- 172.21.15.1:0/405461371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980c2b30 0x7f5c980724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.056+0000 7f5c9e2a9640 1 --2- 172.21.15.1:0/405461371 >> v2:172.21.15.98:3300/0 conn(0x7f5c980a5fa0 0x7f5c980c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.056+0000 7f5c9d2a7640 1 -- 172.21.15.1:0/405461371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980c2b30 msgr2=0x7f5c980724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.056+0000 7f5c9d2a7640 1 --2- 172.21.15.1:0/405461371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980c2b30 0x7f5c980724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.056+0000 7f5c9d2a7640 1 -- 172.21.15.1:0/405461371 >> v2:172.21.15.98:3300/0 conn(0x7f5c980a5fa0 msgr2=0x7f5c980c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.056+0000 7f5c9d2a7640 1 --2- 172.21.15.1:0/405461371 >> v2:172.21.15.98:3300/0 conn(0x7f5c980a5fa0 0x7f5c980c32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.056+0000 7f5c9d2a7640 1 -- 172.21.15.1:0/405461371 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c98152090 con 0x7f5c98152b70 2024-03-06T07:27:02.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.057+0000 7f5c9d2a7640 1 --2- 172.21.15.1:0/405461371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98152b70 0x7f5c98072a00 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f5c980c0e20 tx=0x7f5c8c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc0ce309f3eb9391 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.057+0000 7f5c9caa6640 1 -- 172.21.15.1:0/405461371 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c035020 con 0x7f5c98152b70 2024-03-06T07:27:02.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.057+0000 7f5c9caa6640 1 -- 172.21.15.1:0/405461371 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8c031870 con 0x7f5c98152b70 2024-03-06T07:27:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.057+0000 7f5c9caa6640 1 -- 172.21.15.1:0/405461371 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c031c40 con 0x7f5c98152b70 2024-03-06T07:27:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.058+0000 7f5c9fd0c640 1 -- 172.21.15.1:0/405461371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98152b70 msgr2=0x7f5c98072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.058+0000 7f5c9fd0c640 1 --2- 172.21.15.1:0/405461371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98152b70 0x7f5c98072a00 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f5c980c0e20 tx=0x7f5c8c02b910 comp rx=0 tx=0).stop 2024-03-06T07:27:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.058+0000 7f5c9fd0c640 1 -- 172.21.15.1:0/405461371 shutdown_connections 2024-03-06T07:27:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.058+0000 7f5c9fd0c640 1 --2- 172.21.15.1:0/405461371 >> v2:172.21.15.98:3300/0 conn(0x7f5c980a5fa0 0x7f5c980c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.058+0000 7f5c9fd0c640 1 --2- 172.21.15.1:0/405461371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98152b70 0x7f5c98072a00 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.058+0000 7f5c9fd0c640 1 --2- 172.21.15.1:0/405461371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980c2b30 0x7f5c980724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.058+0000 7f5c9fd0c640 1 -- 172.21.15.1:0/405461371 >> 172.21.15.1:0/405461371 conn(0x7f5c980c6860 msgr2=0x7f5c980c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.058+0000 7f5c9fd0c640 1 -- 172.21.15.1:0/405461371 shutdown_connections 2024-03-06T07:27:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.058+0000 7f5c9fd0c640 1 -- 172.21.15.1:0/405461371 wait complete. 2024-03-06T07:27:02.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.059+0000 7f5c9fd0c640 1 Processor -- start 2024-03-06T07:27:02.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.059+0000 7f5c9fd0c640 1 -- start start 2024-03-06T07:27:02.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.060+0000 7f5c9fd0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980a5fa0 0x7f5c980bd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.060+0000 7f5c9fd0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c980c2b30 0x7f5c980bc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.060+0000 7f5c9fd0c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98152b70 0x7f5c98153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.060+0000 7f5c9fd0c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c98155f00 con 0x7f5c98152b70 2024-03-06T07:27:02.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.060+0000 7f5c9fd0c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c98155d80 con 0x7f5c980c2b30 2024-03-06T07:27:02.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.060+0000 7f5c9fd0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c980c3830 con 0x7f5c980a5fa0 2024-03-06T07:27:02.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:02.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.060+0000 7f5c9e2a9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98152b70 0x7f5c98153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.060+0000 7f5c9e2a9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98152b70 0x7f5c98153430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45278/0 (socket says 172.21.15.1:45278) 2024-03-06T07:27:02.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.060+0000 7f5c9e2a9640 1 -- 172.21.15.1:0/2874327345 learned_addr learned my addr 172.21.15.1:0/2874327345 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:02.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.060+0000 7f5c9daa8640 1 --2- 172.21.15.1:0/2874327345 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980a5fa0 0x7f5c980bd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.060+0000 7f5c9d2a7640 1 --2- 172.21.15.1:0/2874327345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c980c2b30 0x7f5c980bc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.060+0000 7f5c9e2a9640 1 -- 172.21.15.1:0/2874327345 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980a5fa0 msgr2=0x7f5c980bd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.060+0000 7f5c9e2a9640 1 --2- 172.21.15.1:0/2874327345 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980a5fa0 0x7f5c980bd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.060+0000 7f5c9e2a9640 1 -- 172.21.15.1:0/2874327345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c980c2b30 msgr2=0x7f5c980bc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.060+0000 7f5c9e2a9640 1 --2- 172.21.15.1:0/2874327345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c980c2b30 0x7f5c980bc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.060+0000 7f5c9e2a9640 1 -- 172.21.15.1:0/2874327345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c981de070 con 0x7f5c98152b70 2024-03-06T07:27:02.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.060+0000 7f5c9e2a9640 1 --2- 172.21.15.1:0/2874327345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98152b70 0x7f5c98153430 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f5c88007e60 tx=0x7f5c8800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.061+0000 7f5c967fc640 1 -- 172.21.15.1:0/2874327345 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c88014620 con 0x7f5c98152b70 2024-03-06T07:27:02.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.061+0000 7f5c967fc640 1 -- 172.21.15.1:0/2874327345 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c88019070 con 0x7f5c98152b70 2024-03-06T07:27:02.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.061+0000 7f5c9fd0c640 1 -- 172.21.15.1:0/2874327345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c981de350 con 0x7f5c98152b70 2024-03-06T07:27:02.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.061+0000 7f5c967fc640 1 -- 172.21.15.1:0/2874327345 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c88014a20 con 0x7f5c98152b70 2024-03-06T07:27:02.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.061+0000 7f5c9fd0c640 1 -- 172.21.15.1:0/2874327345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c98153970 con 0x7f5c98152b70 2024-03-06T07:27:02.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.062+0000 7f5c9fd0c640 1 -- 172.21.15.1:0/2874327345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c981e8220 con 0x7f5c98152b70 2024-03-06T07:27:02.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.062+0000 7f5c967fc640 1 -- 172.21.15.1:0/2874327345 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c88014bc0 con 0x7f5c98152b70 2024-03-06T07:27:02.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.063+0000 7f5c967fc640 1 --2- 172.21.15.1:0/2874327345 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c8406b4d0 0x7f5c8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.063+0000 7f5c967fc640 1 -- 172.21.15.1:0/2874327345 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5c8808a730 con 0x7f5c98152b70 2024-03-06T07:27:02.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.063+0000 7f5c9daa8640 1 --2- 172.21.15.1:0/2874327345 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c8406b4d0 0x7f5c8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.064+0000 7f5c9daa8640 1 --2- 172.21.15.1:0/2874327345 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c8406b4d0 0x7f5c8406d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5c90006fd0 tx=0x7f5c90008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.067+0000 7f5c967fc640 1 -- 172.21.15.1:0/2874327345 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c88059130 con 0x7f5c98152b70 2024-03-06T07:27:02.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.248+0000 7f5c9fd0c640 1 -- 172.21.15.1:0/2874327345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5c981e7d00 con 0x7f5c98152b70 2024-03-06T07:27:02.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.249+0000 7f5c967fc640 1 -- 172.21.15.1:0/2874327345 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v167) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5c8805d1d0 con 0x7f5c98152b70 2024-03-06T07:27:02.251 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:27:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.251+0000 7f5c9fd0c640 1 -- 172.21.15.1:0/2874327345 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c8406b4d0 msgr2=0x7f5c8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.251+0000 7f5c9fd0c640 1 --2- 172.21.15.1:0/2874327345 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c8406b4d0 0x7f5c8406d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5c90006fd0 tx=0x7f5c90008040 comp rx=0 tx=0).stop 2024-03-06T07:27:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.251+0000 7f5c9fd0c640 1 -- 172.21.15.1:0/2874327345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98152b70 msgr2=0x7f5c98153430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.251+0000 7f5c9fd0c640 1 --2- 172.21.15.1:0/2874327345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98152b70 0x7f5c98153430 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f5c88007e60 tx=0x7f5c8800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.251+0000 7f5c9fd0c640 1 -- 172.21.15.1:0/2874327345 shutdown_connections 2024-03-06T07:27:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.251+0000 7f5c9fd0c640 1 --2- 172.21.15.1:0/2874327345 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c8406b4d0 0x7f5c8406d990 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.251+0000 7f5c9fd0c640 1 --2- 172.21.15.1:0/2874327345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98152b70 0x7f5c98153430 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.251+0000 7f5c9fd0c640 1 --2- 172.21.15.1:0/2874327345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c980c2b30 0x7f5c980bc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.251+0000 7f5c9fd0c640 1 --2- 172.21.15.1:0/2874327345 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c980a5fa0 0x7f5c980bd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.251+0000 7f5c9fd0c640 1 -- 172.21.15.1:0/2874327345 >> 172.21.15.1:0/2874327345 conn(0x7f5c980c6860 msgr2=0x7f5c98072410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.251+0000 7f5c9fd0c640 1 -- 172.21.15.1:0/2874327345 shutdown_connections 2024-03-06T07:27:02.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.251+0000 7f5c9fd0c640 1 -- 172.21.15.1:0/2874327345 wait complete. 2024-03-06T07:27:02.264 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:27:02.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.363+0000 7f0842aeb640 1 Processor -- start 2024-03-06T07:27:02.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.363+0000 7f0842aeb640 1 -- start start 2024-03-06T07:27:02.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.364+0000 7f0842aeb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f083c0c2b30 0x7f083c0724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.364+0000 7f0842aeb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f083c152b70 0x7f083c072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.364+0000 7f0842aeb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083c151dd0 con 0x7f083c0a5fa0 2024-03-06T07:27:02.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.364+0000 7f0842aeb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f083c1526e0 con 0x7f083c0c2b30 2024-03-06T07:27:02.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.364+0000 7f0842aeb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f083c156190 con 0x7f083c152b70 2024-03-06T07:27:02.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.364+0000 7f0840887640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f083c0c2b30 0x7f083c0724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.364+0000 7f0841088640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f083c0a5fa0 0x7f083c0c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36818/0 (socket says 172.21.15.1:36818) 2024-03-06T07:27:02.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.364+0000 7f0841088640 1 -- 172.21.15.1:0/738761144 learned_addr learned my addr 172.21.15.1:0/738761144 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:02.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.364+0000 7f0840887640 1 -- 172.21.15.1:0/738761144 >> v2:172.21.15.98:3300/0 conn(0x7f083c152b70 msgr2=0x7f083c072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.364+0000 7f0840887640 1 --2- 172.21.15.1:0/738761144 >> v2:172.21.15.98:3300/0 conn(0x7f083c152b70 0x7f083c072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.364+0000 7f0840887640 1 -- 172.21.15.1:0/738761144 >> v1:172.21.15.1:6789/0 conn(0x7f083c0a5fa0 legacy=0x7f083c0c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.364+0000 7f0840887640 1 -- 172.21.15.1:0/738761144 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c152090 con 0x7f083c0c2b30 2024-03-06T07:27:02.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.364+0000 7f083b7fe640 1 -- 172.21.15.1:0/738761144 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087980625 0 0) 0x7f083c151dd0 con 0x7f083c0a5fa0 2024-03-06T07:27:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.365+0000 7f0840887640 1 --2- 172.21.15.1:0/738761144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f083c0c2b30 0x7f083c0724c0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f083c0c1090 tx=0x7f083402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af0183525cbc3d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.365+0000 7f083b7fe640 1 -- 172.21.15.1:0/738761144 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0834039070 con 0x7f083c0c2b30 2024-03-06T07:27:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.365+0000 7f083b7fe640 1 -- 172.21.15.1:0/738761144 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083402cc80 con 0x7f083c0c2b30 2024-03-06T07:27:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.365+0000 7f083b7fe640 1 -- 172.21.15.1:0/738761144 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0834034660 con 0x7f083c0c2b30 2024-03-06T07:27:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.366+0000 7f0842aeb640 1 -- 172.21.15.1:0/738761144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f083c0c2b30 msgr2=0x7f083c0724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.366+0000 7f0842aeb640 1 --2- 172.21.15.1:0/738761144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f083c0c2b30 0x7f083c0724c0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f083c0c1090 tx=0x7f083402bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.366+0000 7f0842aeb640 1 -- 172.21.15.1:0/738761144 shutdown_connections 2024-03-06T07:27:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.366+0000 7f0842aeb640 1 --2- 172.21.15.1:0/738761144 >> v2:172.21.15.98:3300/0 conn(0x7f083c152b70 0x7f083c072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.366+0000 7f0842aeb640 1 --2- 172.21.15.1:0/738761144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f083c0c2b30 0x7f083c0724c0 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.366+0000 7f0842aeb640 1 -- 172.21.15.1:0/738761144 >> 172.21.15.1:0/738761144 conn(0x7f083c0c6860 msgr2=0x7f083c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.366+0000 7f0842aeb640 1 -- 172.21.15.1:0/738761144 shutdown_connections 2024-03-06T07:27:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.366+0000 7f0842aeb640 1 -- 172.21.15.1:0/738761144 wait complete. 2024-03-06T07:27:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.367+0000 7f0842aeb640 1 Processor -- start 2024-03-06T07:27:02.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.367+0000 7f0842aeb640 1 -- start start 2024-03-06T07:27:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f0842aeb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f083c0a5fa0 0x7f083c1ee640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f0842aeb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f083c0c2b30 0x7f083c14e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f0842aeb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c152b70 0x7f083c14c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f0842aeb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c155f00 con 0x7f083c0c2b30 2024-03-06T07:27:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f0842aeb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c155d80 con 0x7f083c152b70 2024-03-06T07:27:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f0842aeb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083c0c3a00 con 0x7f083c0a5fa0 2024-03-06T07:27:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f083bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f083c0c2b30 0x7f083c14e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f083bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f083c0c2b30 0x7f083c14e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45296/0 (socket says 172.21.15.1:45296) 2024-03-06T07:27:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f083bfff640 1 -- 172.21.15.1:0/3706730067 learned_addr learned my addr 172.21.15.1:0/3706730067 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f0840887640 1 --2- 172.21.15.1:0/3706730067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f083c0a5fa0 0x7f083c1ee640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f0841088640 1 --2- 172.21.15.1:0/3706730067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c152b70 0x7f083c14c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f083bfff640 1 -- 172.21.15.1:0/3706730067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f083c0a5fa0 msgr2=0x7f083c1ee640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f083bfff640 1 --2- 172.21.15.1:0/3706730067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f083c0a5fa0 0x7f083c1ee640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f083bfff640 1 -- 172.21.15.1:0/3706730067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c152b70 msgr2=0x7f083c14c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f083bfff640 1 --2- 172.21.15.1:0/3706730067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c152b70 0x7f083c14c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f083bfff640 1 -- 172.21.15.1:0/3706730067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c14d2b0 con 0x7f083c0c2b30 2024-03-06T07:27:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f0840887640 1 --2- 172.21.15.1:0/3706730067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f083c0a5fa0 0x7f083c1ee640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f0841088640 1 --2- 172.21.15.1:0/3706730067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c152b70 0x7f083c14c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.368+0000 7f083bfff640 1 --2- 172.21.15.1:0/3706730067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f083c0c2b30 0x7f083c14e270 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f083000b410 tx=0x7f083000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.369+0000 7f08397fa640 1 -- 172.21.15.1:0/3706730067 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0830015020 con 0x7f083c0c2b30 2024-03-06T07:27:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.369+0000 7f08397fa640 1 -- 172.21.15.1:0/3706730067 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0830011850 con 0x7f083c0c2b30 2024-03-06T07:27:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.369+0000 7f0842aeb640 1 -- 172.21.15.1:0/3706730067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083c1ebcf0 con 0x7f083c0c2b30 2024-03-06T07:27:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.369+0000 7f08397fa640 1 -- 172.21.15.1:0/3706730067 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0830011c40 con 0x7f083c0c2b30 2024-03-06T07:27:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.369+0000 7f0842aeb640 1 -- 172.21.15.1:0/3706730067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083c1ebfd0 con 0x7f083c0c2b30 2024-03-06T07:27:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.370+0000 7f0842aeb640 1 -- 172.21.15.1:0/3706730067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f083c067fd0 con 0x7f083c0c2b30 2024-03-06T07:27:02.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.370+0000 7f08397fa640 1 -- 172.21.15.1:0/3706730067 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0830018020 con 0x7f083c0c2b30 2024-03-06T07:27:02.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.370+0000 7f08397fa640 1 --2- 172.21.15.1:0/3706730067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f082806b4d0 0x7f082806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.371+0000 7f08397fa640 1 -- 172.21.15.1:0/3706730067 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f0830089ef0 con 0x7f083c0c2b30 2024-03-06T07:27:02.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.371+0000 7f0840887640 1 --2- 172.21.15.1:0/3706730067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f082806b4d0 0x7f082806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.371+0000 7f0840887640 1 --2- 172.21.15.1:0/3706730067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f082806b4d0 0x7f082806d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0834036000 tx=0x7f0834030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.375+0000 7f08397fa640 1 -- 172.21.15.1:0/3706730067 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f083001f020 con 0x7f083c0c2b30 2024-03-06T07:27:02.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.558+0000 7f0842aeb640 1 -- 172.21.15.1:0/3706730067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f083c071790 con 0x7f083c0c2b30 2024-03-06T07:27:02.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.558+0000 7f08397fa640 1 -- 172.21.15.1:0/3706730067 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v167) v1 ==== 84+0+65 (secure 0 0 0) 0x7f08300172c0 con 0x7f083c0c2b30 2024-03-06T07:27:02.560 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:02.560 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-06T07:27:02.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.560+0000 7f0842aeb640 1 -- 172.21.15.1:0/3706730067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f082806b4d0 msgr2=0x7f082806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.560+0000 7f0842aeb640 1 --2- 172.21.15.1:0/3706730067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f082806b4d0 0x7f082806d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0834036000 tx=0x7f0834030000 comp rx=0 tx=0).stop 2024-03-06T07:27:02.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.560+0000 7f0842aeb640 1 -- 172.21.15.1:0/3706730067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f083c0c2b30 msgr2=0x7f083c14e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.560+0000 7f0842aeb640 1 --2- 172.21.15.1:0/3706730067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f083c0c2b30 0x7f083c14e270 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f083000b410 tx=0x7f083000b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.560+0000 7f0842aeb640 1 -- 172.21.15.1:0/3706730067 shutdown_connections 2024-03-06T07:27:02.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.560+0000 7f0842aeb640 1 --2- 172.21.15.1:0/3706730067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f082806b4d0 0x7f082806d990 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.560+0000 7f0842aeb640 1 --2- 172.21.15.1:0/3706730067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c152b70 0x7f083c14c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.560+0000 7f0842aeb640 1 --2- 172.21.15.1:0/3706730067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f083c0c2b30 0x7f083c14e270 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.560+0000 7f0842aeb640 1 --2- 172.21.15.1:0/3706730067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f083c0a5fa0 0x7f083c1ee640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.560+0000 7f0842aeb640 1 -- 172.21.15.1:0/3706730067 >> 172.21.15.1:0/3706730067 conn(0x7f083c0c6860 msgr2=0x7f083c153d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.560+0000 7f0842aeb640 1 -- 172.21.15.1:0/3706730067 shutdown_connections 2024-03-06T07:27:02.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.560+0000 7f0842aeb640 1 -- 172.21.15.1:0/3706730067 wait complete. 2024-03-06T07:27:02.573 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:27:02.573 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:27:02.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.676+0000 7f2b05381640 1 Processor -- start 2024-03-06T07:27:02.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.676+0000 7f2b05381640 1 -- start start 2024-03-06T07:27:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2b05381640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b000c5950 0x7f2b000c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2b05381640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2b000a60b0 0x7f2b00152910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2b05381640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2b0006eae0 0x7f2b0006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2b05381640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b00072690 con 0x7f2b0006eae0 2024-03-06T07:27:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2b05381640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b0006f490 con 0x7f2b000a60b0 2024-03-06T07:27:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2b05381640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b00152e50 con 0x7f2b000c5950 2024-03-06T07:27:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2aff7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2b0006eae0 0x7f2b0006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2aff7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2b0006eae0 0x7f2b0006ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45310/0 (socket says 172.21.15.1:45310) 2024-03-06T07:27:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2aff7fe640 1 -- 172.21.15.1:0/1578165950 learned_addr learned my addr 172.21.15.1:0/1578165950 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2af7fff640 1 --2- 172.21.15.1:0/1578165950 >> v2:172.21.15.98:3300/0 conn(0x7f2b000a60b0 0x7f2b00152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2afeffd640 1 --2- 172.21.15.1:0/1578165950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b000c5950 0x7f2b000c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2aff7fe640 1 -- 172.21.15.1:0/1578165950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b000c5950 msgr2=0x7f2b000c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2aff7fe640 1 --2- 172.21.15.1:0/1578165950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b000c5950 0x7f2b000c5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2aff7fe640 1 -- 172.21.15.1:0/1578165950 >> v2:172.21.15.98:3300/0 conn(0x7f2b000a60b0 msgr2=0x7f2b00152910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2aff7fe640 1 --2- 172.21.15.1:0/1578165950 >> v2:172.21.15.98:3300/0 conn(0x7f2b000a60b0 0x7f2b00152910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2aff7fe640 1 -- 172.21.15.1:0/1578165950 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b000728c0 con 0x7f2b0006eae0 2024-03-06T07:27:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2afeffd640 1 --2- 172.21.15.1:0/1578165950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b000c5950 0x7f2b000c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.677+0000 7f2af7fff640 1 --2- 172.21.15.1:0/1578165950 >> v2:172.21.15.98:3300/0 conn(0x7f2b000a60b0 0x7f2b00152910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.678+0000 7f2aff7fe640 1 --2- 172.21.15.1:0/1578165950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b0006eae0 0x7f2b0006ef50 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f2b0006f9b0 tx=0x7f2af802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=430461209cf9003c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.678+0000 7f2afe7fc640 1 -- 172.21.15.1:0/1578165950 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af802ea80 con 0x7f2b0006eae0 2024-03-06T07:27:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.678+0000 7f2afe7fc640 1 -- 172.21.15.1:0/1578165950 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af802ec20 con 0x7f2b0006eae0 2024-03-06T07:27:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.678+0000 7f2afe7fc640 1 -- 172.21.15.1:0/1578165950 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af8009600 con 0x7f2b0006eae0 2024-03-06T07:27:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.679+0000 7f2b05381640 1 -- 172.21.15.1:0/1578165950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b0006eae0 msgr2=0x7f2b0006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.679+0000 7f2b05381640 1 --2- 172.21.15.1:0/1578165950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b0006eae0 0x7f2b0006ef50 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f2b0006f9b0 tx=0x7f2af802e0e0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.679+0000 7f2b05381640 1 -- 172.21.15.1:0/1578165950 shutdown_connections 2024-03-06T07:27:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.679+0000 7f2b05381640 1 --2- 172.21.15.1:0/1578165950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b0006eae0 0x7f2b0006ef50 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.679+0000 7f2b05381640 1 --2- 172.21.15.1:0/1578165950 >> v2:172.21.15.98:3300/0 conn(0x7f2b000a60b0 0x7f2b00152910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.679+0000 7f2b05381640 1 --2- 172.21.15.1:0/1578165950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b000c5950 0x7f2b000c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.679+0000 7f2b05381640 1 -- 172.21.15.1:0/1578165950 >> 172.21.15.1:0/1578165950 conn(0x7f2b000a8e10 msgr2=0x7f2b000c3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.679+0000 7f2b05381640 1 -- 172.21.15.1:0/1578165950 shutdown_connections 2024-03-06T07:27:02.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.679+0000 7f2b05381640 1 -- 172.21.15.1:0/1578165950 wait complete. 2024-03-06T07:27:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.680+0000 7f2b05381640 1 Processor -- start 2024-03-06T07:27:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.680+0000 7f2b05381640 1 -- start start 2024-03-06T07:27:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.680+0000 7f2b05381640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b0006eae0 0x7f2b001edc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.680+0000 7f2b05381640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b000a60b0 0x7f2b000bffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.680+0000 7f2b05381640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b000c5950 0x7f2b000ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.680+0000 7f2b05381640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b00073160 con 0x7f2b000c5950 2024-03-06T07:27:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.680+0000 7f2b05381640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b00072e80 con 0x7f2b0006eae0 2024-03-06T07:27:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.680+0000 7f2b05381640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b000c1720 con 0x7f2b000a60b0 2024-03-06T07:27:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.680+0000 7f2aff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b000c5950 0x7f2b000ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.681+0000 7f2aff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b000c5950 0x7f2b000ac2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45324/0 (socket says 172.21.15.1:45324) 2024-03-06T07:27:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.681+0000 7f2aff7fe640 1 -- 172.21.15.1:0/3898374652 learned_addr learned my addr 172.21.15.1:0/3898374652 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.681+0000 7f2af7fff640 1 --2- 172.21.15.1:0/3898374652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b000a60b0 0x7f2b000bffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.681+0000 7f2afeffd640 1 --2- 172.21.15.1:0/3898374652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b0006eae0 0x7f2b001edc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.681+0000 7f2aff7fe640 1 -- 172.21.15.1:0/3898374652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b000a60b0 msgr2=0x7f2b000bffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.681+0000 7f2aff7fe640 1 --2- 172.21.15.1:0/3898374652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b000a60b0 0x7f2b000bffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.681+0000 7f2aff7fe640 1 -- 172.21.15.1:0/3898374652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b0006eae0 msgr2=0x7f2b001edc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.681+0000 7f2aff7fe640 1 --2- 172.21.15.1:0/3898374652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b0006eae0 0x7f2b001edc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.681+0000 7f2aff7fe640 1 -- 172.21.15.1:0/3898374652 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b00151640 con 0x7f2b000c5950 2024-03-06T07:27:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.681+0000 7f2aff7fe640 1 --2- 172.21.15.1:0/3898374652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b000c5950 0x7f2b000ac2e0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f2af800d240 tx=0x7f2af8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.681+0000 7f2af77fe640 1 -- 172.21.15.1:0/3898374652 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af802ea80 con 0x7f2b000c5950 2024-03-06T07:27:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.681+0000 7f2af77fe640 1 -- 172.21.15.1:0/3898374652 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af8045070 con 0x7f2b000c5950 2024-03-06T07:27:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.681+0000 7f2b05381640 1 -- 172.21.15.1:0/3898374652 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b001eb2e0 con 0x7f2b000c5950 2024-03-06T07:27:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.681+0000 7f2af77fe640 1 -- 172.21.15.1:0/3898374652 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af800b720 con 0x7f2b000c5950 2024-03-06T07:27:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.681+0000 7f2b05381640 1 -- 172.21.15.1:0/3898374652 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b001eb5c0 con 0x7f2b000c5950 2024-03-06T07:27:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.682+0000 7f2b05381640 1 -- 172.21.15.1:0/3898374652 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b000680b0 con 0x7f2b000c5950 2024-03-06T07:27:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.683+0000 7f2af77fe640 1 -- 172.21.15.1:0/3898374652 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2af8036a60 con 0x7f2b000c5950 2024-03-06T07:27:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.683+0000 7f2af77fe640 1 --2- 172.21.15.1:0/3898374652 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2ae006b4d0 0x7f2ae006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.683+0000 7f2af77fe640 1 -- 172.21.15.1:0/3898374652 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f2af80ad0b0 con 0x7f2b000c5950 2024-03-06T07:27:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.683+0000 7f2afeffd640 1 --2- 172.21.15.1:0/3898374652 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2ae006b4d0 0x7f2ae006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.684+0000 7f2afeffd640 1 --2- 172.21.15.1:0/3898374652 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2ae006b4d0 0x7f2ae006d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f2af0006fd0 tx=0x7f2af0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.687+0000 7f2af77fe640 1 -- 172.21.15.1:0/3898374652 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2af807c220 con 0x7f2b000c5950 2024-03-06T07:27:02.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:02.868+0000 7f2b05381640 1 -- 172.21.15.1:0/3898374652 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2b0014e780 con 0x7f2b000c5950 2024-03-06T07:27:03.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.245+0000 7f2af77fe640 1 -- 172.21.15.1:0/3898374652 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v168) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2af807bb20 con 0x7f2b000c5950 2024-03-06T07:27:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-06T07:27:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.247+0000 7f2b05381640 1 -- 172.21.15.1:0/3898374652 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2ae006b4d0 msgr2=0x7f2ae006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.247+0000 7f2b05381640 1 --2- 172.21.15.1:0/3898374652 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2ae006b4d0 0x7f2ae006d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f2af0006fd0 tx=0x7f2af0008040 comp rx=0 tx=0).stop 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.247+0000 7f2b05381640 1 -- 172.21.15.1:0/3898374652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b000c5950 msgr2=0x7f2b000ac2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.247+0000 7f2b05381640 1 --2- 172.21.15.1:0/3898374652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b000c5950 0x7f2b000ac2e0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f2af800d240 tx=0x7f2af8004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.247+0000 7f2b05381640 1 -- 172.21.15.1:0/3898374652 shutdown_connections 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.247+0000 7f2b05381640 1 --2- 172.21.15.1:0/3898374652 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2ae006b4d0 0x7f2ae006d990 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.247+0000 7f2b05381640 1 --2- 172.21.15.1:0/3898374652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b000c5950 0x7f2b000ac2e0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.247+0000 7f2b05381640 1 --2- 172.21.15.1:0/3898374652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b000a60b0 0x7f2b000bffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.247+0000 7f2b05381640 1 --2- 172.21.15.1:0/3898374652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b0006eae0 0x7f2b001edc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.247+0000 7f2b05381640 1 -- 172.21.15.1:0/3898374652 >> 172.21.15.1:0/3898374652 conn(0x7f2b000a8e10 msgr2=0x7f2b000c3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.247+0000 7f2b05381640 1 -- 172.21.15.1:0/3898374652 shutdown_connections 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.247+0000 7f2b05381640 1 -- 172.21.15.1:0/3898374652 wait complete. 2024-03-06T07:27:03.479 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:27:03.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.589+0000 7f918e3d0640 1 Processor -- start 2024-03-06T07:27:03.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.589+0000 7f918e3d0640 1 -- start start 2024-03-06T07:27:03.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.589+0000 7f918e3d0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f91880c6730 0x7f91880c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.589+0000 7f918e3d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91880c5d10 0x7f9188156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.589+0000 7f918e3d0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91880c3300 con 0x7f9188152690 2024-03-06T07:27:03.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.589+0000 7f918e3d0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91880c35a0 con 0x7f91880c6730 2024-03-06T07:27:03.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.589+0000 7f918e3d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91880c3880 con 0x7f91880c5d10 2024-03-06T07:27:03.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.590+0000 7f918c96d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91880c5d10 0x7f9188156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.590+0000 7f91877fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f91880c6730 0x7f91880c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.590+0000 7f9187fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9188152690 0x7f9188155ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:52654/0 (socket says 172.21.15.1:52654) 2024-03-06T07:27:03.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.590+0000 7f9187fff640 1 -- 172.21.15.1:0/3538748714 learned_addr learned my addr 172.21.15.1:0/3538748714 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:03.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.590+0000 7f91877fe640 1 -- 172.21.15.1:0/3538748714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91880c5d10 msgr2=0x7f9188156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.590+0000 7f91877fe640 1 --2- 172.21.15.1:0/3538748714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91880c5d10 0x7f9188156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.590+0000 7f91877fe640 1 -- 172.21.15.1:0/3538748714 >> v1:172.21.15.98:6789/0 conn(0x7f9188152690 legacy=0x7f9188155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.590+0000 7f91877fe640 1 -- 172.21.15.1:0/3538748714 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91880c3b60 con 0x7f91880c6730 2024-03-06T07:27:03.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.591+0000 7f91877fe640 1 --2- 172.21.15.1:0/3538748714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880c6730 0x7f91880c57d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f917c0097a0 tx=0x7f917c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5099d121827bf175 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.591+0000 7f9186ffd640 1 -- 172.21.15.1:0/3538748714 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c0315a0 con 0x7f91880c6730 2024-03-06T07:27:03.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.591+0000 7f9186ffd640 1 -- 172.21.15.1:0/3538748714 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917c037070 con 0x7f91880c6730 2024-03-06T07:27:03.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.591+0000 7f9186ffd640 1 -- 172.21.15.1:0/3538748714 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c031960 con 0x7f91880c6730 2024-03-06T07:27:03.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.592+0000 7f918e3d0640 1 -- 172.21.15.1:0/3538748714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880c6730 msgr2=0x7f91880c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.592+0000 7f918e3d0640 1 --2- 172.21.15.1:0/3538748714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880c6730 0x7f91880c57d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f917c0097a0 tx=0x7f917c02b840 comp rx=0 tx=0).stop 2024-03-06T07:27:03.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.592+0000 7f918e3d0640 1 -- 172.21.15.1:0/3538748714 shutdown_connections 2024-03-06T07:27:03.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.592+0000 7f918e3d0640 1 --2- 172.21.15.1:0/3538748714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f91880c5d10 0x7f9188156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.592+0000 7f918e3d0640 1 --2- 172.21.15.1:0/3538748714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880c6730 0x7f91880c57d0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.592+0000 7f918e3d0640 1 -- 172.21.15.1:0/3538748714 >> 172.21.15.1:0/3538748714 conn(0x7f918806b410 msgr2=0x7f918806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.592+0000 7f918e3d0640 1 -- 172.21.15.1:0/3538748714 shutdown_connections 2024-03-06T07:27:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.592+0000 7f918e3d0640 1 -- 172.21.15.1:0/3538748714 wait complete. 2024-03-06T07:27:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.593+0000 7f918e3d0640 1 Processor -- start 2024-03-06T07:27:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.593+0000 7f918e3d0640 1 -- start start 2024-03-06T07:27:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f918e3d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880c5d10 0x7f918814b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f918e3d0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91880c6730 0x7f91880aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f918e3d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9188152690 0x7f918814d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f918e3d0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9188152bf0 con 0x7f91880c6730 2024-03-06T07:27:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f918e3d0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9188152a70 con 0x7f91880c5d10 2024-03-06T07:27:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f918e3d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9188157210 con 0x7f9188152690 2024-03-06T07:27:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f91877fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91880c6730 0x7f91880aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f91877fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91880c6730 0x7f91880aa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45338/0 (socket says 172.21.15.1:45338) 2024-03-06T07:27:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f91877fe640 1 -- 172.21.15.1:0/1975668293 learned_addr learned my addr 172.21.15.1:0/1975668293 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f918c96d640 1 --2- 172.21.15.1:0/1975668293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9188152690 0x7f918814d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f9187fff640 1 --2- 172.21.15.1:0/1975668293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880c5d10 0x7f918814b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f91877fe640 1 -- 172.21.15.1:0/1975668293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9188152690 msgr2=0x7f918814d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f91877fe640 1 --2- 172.21.15.1:0/1975668293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9188152690 0x7f918814d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f91877fe640 1 -- 172.21.15.1:0/1975668293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880c5d10 msgr2=0x7f918814b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f91877fe640 1 --2- 172.21.15.1:0/1975668293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880c5d10 0x7f918814b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f91877fe640 1 -- 172.21.15.1:0/1975668293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91881eeab0 con 0x7f91880c6730 2024-03-06T07:27:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f91877fe640 1 --2- 172.21.15.1:0/1975668293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91880c6730 0x7f91880aa5c0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f917c034040 tx=0x7f917c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.594+0000 7f9184ff9640 1 -- 172.21.15.1:0/1975668293 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c032630 con 0x7f91880c6730 2024-03-06T07:27:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.595+0000 7f9184ff9640 1 -- 172.21.15.1:0/1975668293 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917c0327f0 con 0x7f91880c6730 2024-03-06T07:27:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.595+0000 7f918e3d0640 1 -- 172.21.15.1:0/1975668293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91880aaf30 con 0x7f91880c6730 2024-03-06T07:27:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.595+0000 7f9184ff9640 1 -- 172.21.15.1:0/1975668293 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c032be0 con 0x7f91880c6730 2024-03-06T07:27:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.595+0000 7f918e3d0640 1 -- 172.21.15.1:0/1975668293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918814d9f0 con 0x7f91880c6730 2024-03-06T07:27:03.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.596+0000 7f918e3d0640 1 -- 172.21.15.1:0/1975668293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9188067fd0 con 0x7f91880c6730 2024-03-06T07:27:03.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.596+0000 7f9184ff9640 1 -- 172.21.15.1:0/1975668293 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f917c04e430 con 0x7f91880c6730 2024-03-06T07:27:03.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.596+0000 7f9184ff9640 1 --2- 172.21.15.1:0/1975668293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f917406b4d0 0x7f917406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.596+0000 7f9184ff9640 1 -- 172.21.15.1:0/1975668293 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f917c02f030 con 0x7f91880c6730 2024-03-06T07:27:03.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.597+0000 7f9187fff640 1 --2- 172.21.15.1:0/1975668293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f917406b4d0 0x7f917406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.597+0000 7f9187fff640 1 --2- 172.21.15.1:0/1975668293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f917406b4d0 0x7f917406d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f918814dd20 tx=0x7f9178009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.601+0000 7f9184ff9640 1 -- 172.21.15.1:0/1975668293 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f917c040070 con 0x7f91880c6730 2024-03-06T07:27:03.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:03.784+0000 7f918e3d0640 1 -- 172.21.15.1:0/1975668293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9188156590 con 0x7f91880c6730 2024-03-06T07:27:04.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.247+0000 7f9184ff9640 1 -- 172.21.15.1:0/1975668293 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v169) v1 ==== 170+0+0 (secure 0 0 0) 0x7f917c07aec0 con 0x7f91880c6730 2024-03-06T07:27:04.249 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.251+0000 7f918e3d0640 1 -- 172.21.15.1:0/1975668293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f917406b4d0 msgr2=0x7f917406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.251+0000 7f918e3d0640 1 --2- 172.21.15.1:0/1975668293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f917406b4d0 0x7f917406d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f918814dd20 tx=0x7f9178009040 comp rx=0 tx=0).stop 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.251+0000 7f918e3d0640 1 -- 172.21.15.1:0/1975668293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91880c6730 msgr2=0x7f91880aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.251+0000 7f918e3d0640 1 --2- 172.21.15.1:0/1975668293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91880c6730 0x7f91880aa5c0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f917c034040 tx=0x7f917c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.251+0000 7f918e3d0640 1 -- 172.21.15.1:0/1975668293 shutdown_connections 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.251+0000 7f918e3d0640 1 --2- 172.21.15.1:0/1975668293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f917406b4d0 0x7f917406d990 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.251+0000 7f918e3d0640 1 --2- 172.21.15.1:0/1975668293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9188152690 0x7f918814d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.251+0000 7f918e3d0640 1 --2- 172.21.15.1:0/1975668293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91880c6730 0x7f91880aa5c0 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.251+0000 7f918e3d0640 1 --2- 172.21.15.1:0/1975668293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880c5d10 0x7f918814b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.251+0000 7f918e3d0640 1 -- 172.21.15.1:0/1975668293 >> 172.21.15.1:0/1975668293 conn(0x7f918806b410 msgr2=0x7f9188070ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.251+0000 7f918e3d0640 1 -- 172.21.15.1:0/1975668293 shutdown_connections 2024-03-06T07:27:04.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.251+0000 7f918e3d0640 1 -- 172.21.15.1:0/1975668293 wait complete. 2024-03-06T07:27:04.279 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:27:04.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.383+0000 7f78611bf640 1 Processor -- start 2024-03-06T07:27:04.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.384+0000 7f78611bf640 1 -- start start 2024-03-06T07:27:04.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.384+0000 7f78611bf640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f785c0c1390 0x7f785c0c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.384+0000 7f78611bf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f785c1552e0 0x7f785c155750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.384+0000 7f78611bf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f785c1588e0 con 0x7f785c152730 2024-03-06T07:27:04.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.384+0000 7f78611bf640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f785c0c1d60 con 0x7f785c0c1390 2024-03-06T07:27:04.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.384+0000 7f78611bf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785c158bc0 con 0x7f785c1552e0 2024-03-06T07:27:04.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.384+0000 7f785a59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f785c0c1390 0x7f785c0c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.384+0000 7f785ad9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f785c152730 0x7f785c0c6a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44346/0 (socket says 172.21.15.1:44346) 2024-03-06T07:27:04.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.384+0000 7f785ad9d640 1 -- 172.21.15.1:0/1906395291 learned_addr learned my addr 172.21.15.1:0/1906395291 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:04.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.385+0000 7f785b59e640 1 --2- 172.21.15.1:0/1906395291 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f785c1552e0 0x7f785c155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.385+0000 7f785a59c640 1 -- 172.21.15.1:0/1906395291 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f785c1552e0 msgr2=0x7f785c155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:04.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.385+0000 7f785a59c640 1 --2- 172.21.15.1:0/1906395291 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f785c1552e0 0x7f785c155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.385+0000 7f785a59c640 1 -- 172.21.15.1:0/1906395291 >> v1:172.21.15.1:6789/0 conn(0x7f785c152730 legacy=0x7f785c0c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.385+0000 7f785a59c640 1 -- 172.21.15.1:0/1906395291 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c158ea0 con 0x7f785c0c1390 2024-03-06T07:27:04.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.385+0000 7f7859d9b640 1 -- 172.21.15.1:0/1906395291 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939173423 0 0) 0x7f785c1588e0 con 0x7f785c152730 2024-03-06T07:27:04.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.385+0000 7f785a59c640 1 --2- 172.21.15.1:0/1906395291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f785c0c1390 0x7f785c0c1820 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f78500097a0 tx=0x7f785002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd514776f113f07e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.386+0000 7f7859d9b640 1 -- 172.21.15.1:0/1906395291 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7850038070 con 0x7f785c0c1390 2024-03-06T07:27:04.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.386+0000 7f7859d9b640 1 -- 172.21.15.1:0/1906395291 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7850031700 con 0x7f785c0c1390 2024-03-06T07:27:04.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.386+0000 7f7859d9b640 1 -- 172.21.15.1:0/1906395291 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7850031ad0 con 0x7f785c0c1390 2024-03-06T07:27:04.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.386+0000 7f78611bf640 1 -- 172.21.15.1:0/1906395291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f785c0c1390 msgr2=0x7f785c0c1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.386+0000 7f78611bf640 1 --2- 172.21.15.1:0/1906395291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f785c0c1390 0x7f785c0c1820 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f78500097a0 tx=0x7f785002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.386+0000 7f78611bf640 1 -- 172.21.15.1:0/1906395291 shutdown_connections 2024-03-06T07:27:04.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.386+0000 7f78611bf640 1 --2- 172.21.15.1:0/1906395291 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f785c1552e0 0x7f785c155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.386+0000 7f78611bf640 1 --2- 172.21.15.1:0/1906395291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f785c0c1390 0x7f785c0c1820 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.386+0000 7f78611bf640 1 -- 172.21.15.1:0/1906395291 >> 172.21.15.1:0/1906395291 conn(0x7f785c066d00 msgr2=0x7f785c067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.386+0000 7f78611bf640 1 -- 172.21.15.1:0/1906395291 shutdown_connections 2024-03-06T07:27:04.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.386+0000 7f78611bf640 1 -- 172.21.15.1:0/1906395291 wait complete. 2024-03-06T07:27:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.388+0000 7f78611bf640 1 Processor -- start 2024-03-06T07:27:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.388+0000 7f78611bf640 1 -- start start 2024-03-06T07:27:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.388+0000 7f78611bf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f785c0c1390 0x7f785c0c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.388+0000 7f78611bf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f785c152730 0x7f785c14cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.388+0000 7f78611bf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f785c1552e0 0x7f785c14f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.388+0000 7f78611bf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c152b10 con 0x7f785c152730 2024-03-06T07:27:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.388+0000 7f78611bf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c159460 con 0x7f785c0c1390 2024-03-06T07:27:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.388+0000 7f78611bf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785c0c71a0 con 0x7f785c1552e0 2024-03-06T07:27:04.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.388+0000 7f785a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f785c152730 0x7f785c14cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.388+0000 7f785a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f785c152730 0x7f785c14cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59356/0 (socket says 172.21.15.1:59356) 2024-03-06T07:27:04.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.388+0000 7f785a59c640 1 -- 172.21.15.1:0/1140519783 learned_addr learned my addr 172.21.15.1:0/1140519783 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:04.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.389+0000 7f785b59e640 1 --2- 172.21.15.1:0/1140519783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f785c1552e0 0x7f785c14f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.389+0000 7f785ad9d640 1 --2- 172.21.15.1:0/1140519783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f785c0c1390 0x7f785c0c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.389+0000 7f785a59c640 1 -- 172.21.15.1:0/1140519783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f785c1552e0 msgr2=0x7f785c14f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.389+0000 7f785a59c640 1 --2- 172.21.15.1:0/1140519783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f785c1552e0 0x7f785c14f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.389+0000 7f785a59c640 1 -- 172.21.15.1:0/1140519783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f785c0c1390 msgr2=0x7f785c0c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.389+0000 7f785a59c640 1 --2- 172.21.15.1:0/1140519783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f785c0c1390 0x7f785c0c8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.389+0000 7f785a59c640 1 -- 172.21.15.1:0/1140519783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c1475a0 con 0x7f785c152730 2024-03-06T07:27:04.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.389+0000 7f785b59e640 1 --2- 172.21.15.1:0/1140519783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f785c1552e0 0x7f785c14f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:04.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.389+0000 7f785ad9d640 1 --2- 172.21.15.1:0/1140519783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f785c0c1390 0x7f785c0c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:04.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.389+0000 7f785a59c640 1 --2- 172.21.15.1:0/1140519783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f785c152730 0x7f785c14cac0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f78500097a0 tx=0x7f78500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.389+0000 7f78437fe640 1 -- 172.21.15.1:0/1140519783 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7850038040 con 0x7f785c152730 2024-03-06T07:27:04.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.389+0000 7f78437fe640 1 -- 172.21.15.1:0/1140519783 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7850004030 con 0x7f785c152730 2024-03-06T07:27:04.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.389+0000 7f78611bf640 1 -- 172.21.15.1:0/1140519783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f785c14b080 con 0x7f785c152730 2024-03-06T07:27:04.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.389+0000 7f78437fe640 1 -- 172.21.15.1:0/1140519783 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7850004420 con 0x7f785c152730 2024-03-06T07:27:04.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.389+0000 7f78611bf640 1 -- 172.21.15.1:0/1140519783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f785c14d6d0 con 0x7f785c152730 2024-03-06T07:27:04.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.390+0000 7f78611bf640 1 -- 172.21.15.1:0/1140519783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f785c067fd0 con 0x7f785c152730 2024-03-06T07:27:04.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.391+0000 7f78437fe640 1 -- 172.21.15.1:0/1140519783 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7850033780 con 0x7f785c152730 2024-03-06T07:27:04.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.391+0000 7f78437fe640 1 --2- 172.21.15.1:0/1140519783 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f784806b500 0x7f784806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.391+0000 7f78437fe640 1 -- 172.21.15.1:0/1140519783 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f78500ac550 con 0x7f785c152730 2024-03-06T07:27:04.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.391+0000 7f785ad9d640 1 --2- 172.21.15.1:0/1140519783 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f784806b500 0x7f784806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.392+0000 7f785ad9d640 1 --2- 172.21.15.1:0/1140519783 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f784806b500 0x7f784806d9c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f78540012e0 tx=0x7f7854009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.396+0000 7f78437fe640 1 -- 172.21.15.1:0/1140519783 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7850030070 con 0x7f785c152730 2024-03-06T07:27:04.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:04.617+0000 7f78611bf640 1 -- 172.21.15.1:0/1140519783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f785c14b250 con 0x7f785c152730 2024-03-06T07:27:05.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.260+0000 7f78437fe640 1 -- 172.21.15.1:0/1140519783 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 27 and data pool 28 v155) v1 ==== 174+0+0 (secure 0 0 0) 0x7f785007b290 con 0x7f785c152730 2024-03-06T07:27:05.262 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 27 and data pool 28 2024-03-06T07:27:05.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.267+0000 7f78611bf640 1 -- 172.21.15.1:0/1140519783 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f784806b500 msgr2=0x7f784806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.267+0000 7f78611bf640 1 --2- 172.21.15.1:0/1140519783 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f784806b500 0x7f784806d9c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f78540012e0 tx=0x7f7854009040 comp rx=0 tx=0).stop 2024-03-06T07:27:05.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.267+0000 7f78611bf640 1 -- 172.21.15.1:0/1140519783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f785c152730 msgr2=0x7f785c14cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.267+0000 7f78611bf640 1 --2- 172.21.15.1:0/1140519783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f785c152730 0x7f785c14cac0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f78500097a0 tx=0x7f78500046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.271+0000 7f78611bf640 1 -- 172.21.15.1:0/1140519783 shutdown_connections 2024-03-06T07:27:05.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.271+0000 7f78611bf640 1 --2- 172.21.15.1:0/1140519783 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f784806b500 0x7f784806d9c0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.271+0000 7f78611bf640 1 --2- 172.21.15.1:0/1140519783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f785c1552e0 0x7f785c14f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.271+0000 7f78611bf640 1 --2- 172.21.15.1:0/1140519783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f785c152730 0x7f785c14cac0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.271+0000 7f78611bf640 1 --2- 172.21.15.1:0/1140519783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f785c0c1390 0x7f785c0c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.271+0000 7f78611bf640 1 -- 172.21.15.1:0/1140519783 >> 172.21.15.1:0/1140519783 conn(0x7f785c066d00 msgr2=0x7f785c0c0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.277+0000 7f78611bf640 1 -- 172.21.15.1:0/1140519783 shutdown_connections 2024-03-06T07:27:05.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.279+0000 7f78611bf640 1 -- 172.21.15.1:0/1140519783 wait complete. 2024-03-06T07:27:05.302 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.405+0000 7f109f75c640 1 Processor -- start 2024-03-06T07:27:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.405+0000 7f109f75c640 1 -- start start 2024-03-06T07:27:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.406+0000 7f109f75c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1098152650 0x7f1098155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.406+0000 7f109f75c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10980bf9d0 0x7f10980c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.406+0000 7f109f75c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10980c2fc0 con 0x7f10980c6b10 2024-03-06T07:27:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.406+0000 7f109f75c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1098155800 con 0x7f1098152650 2024-03-06T07:27:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.406+0000 7f109f75c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1098158da0 con 0x7f10980bf9d0 2024-03-06T07:27:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.406+0000 7f109d4f8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1098152650 0x7f1098155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.406+0000 7f109d4f8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1098152650 0x7f1098155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59366/0 (socket says 172.21.15.1:59366) 2024-03-06T07:27:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.406+0000 7f109d4f8640 1 -- 172.21.15.1:0/1329291412 learned_addr learned my addr 172.21.15.1:0/1329291412 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.407+0000 7f109dcf9640 1 --2- 172.21.15.1:0/1329291412 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10980bf9d0 0x7f10980c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.407+0000 7f109d4f8640 1 -- 172.21.15.1:0/1329291412 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10980bf9d0 msgr2=0x7f10980c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.407+0000 7f109d4f8640 1 --2- 172.21.15.1:0/1329291412 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10980bf9d0 0x7f10980c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.407+0000 7f109d4f8640 1 -- 172.21.15.1:0/1329291412 >> v1:172.21.15.98:6789/0 conn(0x7f10980c6b10 legacy=0x7f1098158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.407+0000 7f109d4f8640 1 -- 172.21.15.1:0/1329291412 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10980c6690 con 0x7f1098152650 2024-03-06T07:27:05.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.407+0000 7f109dcf9640 1 --2- 172.21.15.1:0/1329291412 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10980bf9d0 0x7f10980c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.407+0000 7f109d4f8640 1 --2- 172.21.15.1:0/1329291412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1098152650 0x7f1098155c90 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f10980cac90 tx=0x7f109002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e956d1fae0d97f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.407+0000 7f1097fff640 1 -- 172.21.15.1:0/1329291412 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10900315a0 con 0x7f1098152650 2024-03-06T07:27:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.408+0000 7f1097fff640 1 -- 172.21.15.1:0/1329291412 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1090037070 con 0x7f1098152650 2024-03-06T07:27:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.408+0000 7f1097fff640 1 -- 172.21.15.1:0/1329291412 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1090031960 con 0x7f1098152650 2024-03-06T07:27:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.408+0000 7f109f75c640 1 -- 172.21.15.1:0/1329291412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1098152650 msgr2=0x7f1098155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.408+0000 7f109f75c640 1 --2- 172.21.15.1:0/1329291412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1098152650 0x7f1098155c90 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f10980cac90 tx=0x7f109002b840 comp rx=0 tx=0).stop 2024-03-06T07:27:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.408+0000 7f109f75c640 1 -- 172.21.15.1:0/1329291412 shutdown_connections 2024-03-06T07:27:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.408+0000 7f109f75c640 1 --2- 172.21.15.1:0/1329291412 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10980bf9d0 0x7f10980c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.408+0000 7f109f75c640 1 --2- 172.21.15.1:0/1329291412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1098152650 0x7f1098155c90 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.408+0000 7f109f75c640 1 -- 172.21.15.1:0/1329291412 >> 172.21.15.1:0/1329291412 conn(0x7f1098066ee0 msgr2=0x7f10980672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.408+0000 7f109f75c640 1 -- 172.21.15.1:0/1329291412 shutdown_connections 2024-03-06T07:27:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.408+0000 7f109f75c640 1 -- 172.21.15.1:0/1329291412 wait complete. 2024-03-06T07:27:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.409+0000 7f109f75c640 1 Processor -- start 2024-03-06T07:27:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.409+0000 7f109f75c640 1 -- start start 2024-03-06T07:27:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.410+0000 7f109f75c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10980bf9d0 0x7f10981ebe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.410+0000 7f109f75c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1098152650 0x7f109814fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.410+0000 7f109f75c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1098156490 0x7f10981582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.410+0000 7f109f75c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10981551b0 con 0x7f1098152650 2024-03-06T07:27:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.410+0000 7f109f75c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10980c6fa0 con 0x7f1098156490 2024-03-06T07:27:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.410+0000 7f109f75c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1098155490 con 0x7f10980bf9d0 2024-03-06T07:27:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.410+0000 7f109ccf7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1098152650 0x7f109814fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.410+0000 7f109ccf7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1098152650 0x7f109814fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59374/0 (socket says 172.21.15.1:59374) 2024-03-06T07:27:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.410+0000 7f109ccf7640 1 -- 172.21.15.1:0/2885556847 learned_addr learned my addr 172.21.15.1:0/2885556847 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.410+0000 7f109d4f8640 1 --2- 172.21.15.1:0/2885556847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10980bf9d0 0x7f10981ebe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.410+0000 7f109dcf9640 1 --2- 172.21.15.1:0/2885556847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1098156490 0x7f10981582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.410+0000 7f109ccf7640 1 -- 172.21.15.1:0/2885556847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10980bf9d0 msgr2=0x7f10981ebe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.410+0000 7f109ccf7640 1 --2- 172.21.15.1:0/2885556847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10980bf9d0 0x7f10981ebe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.410+0000 7f109ccf7640 1 -- 172.21.15.1:0/2885556847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1098156490 msgr2=0x7f10981582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.410+0000 7f109ccf7640 1 --2- 172.21.15.1:0/2885556847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1098156490 0x7f10981582b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.410+0000 7f109ccf7640 1 -- 172.21.15.1:0/2885556847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1098151a40 con 0x7f1098152650 2024-03-06T07:27:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.410+0000 7f109d4f8640 1 --2- 172.21.15.1:0/2885556847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10980bf9d0 0x7f10981ebe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.411+0000 7f109ccf7640 1 --2- 172.21.15.1:0/2885556847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1098152650 0x7f109814fc30 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f108c00c460 tx=0x7f108c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.411+0000 7f1095ffb640 1 -- 172.21.15.1:0/2885556847 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108c016020 con 0x7f1098152650 2024-03-06T07:27:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.411+0000 7f1095ffb640 1 -- 172.21.15.1:0/2885556847 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108c012870 con 0x7f1098152650 2024-03-06T07:27:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.411+0000 7f109f75c640 1 -- 172.21.15.1:0/2885556847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1098151d20 con 0x7f1098152650 2024-03-06T07:27:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.411+0000 7f1095ffb640 1 -- 172.21.15.1:0/2885556847 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108c012c60 con 0x7f1098152650 2024-03-06T07:27:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.411+0000 7f109f75c640 1 -- 172.21.15.1:0/2885556847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10981eb910 con 0x7f1098152650 2024-03-06T07:27:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.412+0000 7f109f75c640 1 -- 172.21.15.1:0/2885556847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1098067fd0 con 0x7f1098152650 2024-03-06T07:27:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.412+0000 7f1095ffb640 1 -- 172.21.15.1:0/2885556847 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f108c019020 con 0x7f1098152650 2024-03-06T07:27:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.413+0000 7f1095ffb640 1 --2- 172.21.15.1:0/2885556847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f108406b4d0 0x7f108406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.413+0000 7f1095ffb640 1 -- 172.21.15.1:0/2885556847 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f108c08a0c0 con 0x7f1098152650 2024-03-06T07:27:05.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.413+0000 7f109d4f8640 1 --2- 172.21.15.1:0/2885556847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f108406b4d0 0x7f108406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.413+0000 7f109d4f8640 1 --2- 172.21.15.1:0/2885556847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f108406b4d0 0x7f108406d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f1090036000 tx=0x7f109003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.417+0000 7f1095ffb640 1 -- 172.21.15.1:0/2885556847 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f108c020030 con 0x7f1098152650 2024-03-06T07:27:05.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.599+0000 7f109f75c640 1 -- 172.21.15.1:0/2885556847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10980b01a0 con 0x7f1098152650 2024-03-06T07:27:05.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.600+0000 7f1095ffb640 1 -- 172.21.15.1:0/2885556847 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) v1 ==== 74+0+16845 (secure 0 0 0) 0x7f108c058da0 con 0x7f1098152650 2024-03-06T07:27:05.603 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:05.603 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":170,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:27:04.618780+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:02.888292+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:03.804423+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:05.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.603+0000 7f109f75c640 1 -- 172.21.15.1:0/2885556847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f108406b4d0 msgr2=0x7f108406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.603+0000 7f109f75c640 1 --2- 172.21.15.1:0/2885556847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f108406b4d0 0x7f108406d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f1090036000 tx=0x7f109003d040 comp rx=0 tx=0).stop 2024-03-06T07:27:05.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.603+0000 7f109f75c640 1 -- 172.21.15.1:0/2885556847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1098152650 msgr2=0x7f109814fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.603+0000 7f109f75c640 1 --2- 172.21.15.1:0/2885556847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1098152650 0x7f109814fc30 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f108c00c460 tx=0x7f108c00c920 comp rx=0 tx=0).stop 2024-03-06T07:27:05.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.603+0000 7f109f75c640 1 -- 172.21.15.1:0/2885556847 shutdown_connections 2024-03-06T07:27:05.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.603+0000 7f109f75c640 1 --2- 172.21.15.1:0/2885556847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f108406b4d0 0x7f108406d990 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.603+0000 7f109f75c640 1 --2- 172.21.15.1:0/2885556847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1098156490 0x7f10981582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.603+0000 7f109f75c640 1 --2- 172.21.15.1:0/2885556847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1098152650 0x7f109814fc30 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.603+0000 7f109f75c640 1 --2- 172.21.15.1:0/2885556847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10980bf9d0 0x7f10981ebe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.603+0000 7f109f75c640 1 -- 172.21.15.1:0/2885556847 >> 172.21.15.1:0/2885556847 conn(0x7f1098066ee0 msgr2=0x7f10980c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.603+0000 7f109f75c640 1 -- 172.21.15.1:0/2885556847 shutdown_connections 2024-03-06T07:27:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.603+0000 7f109f75c640 1 -- 172.21.15.1:0/2885556847 wait complete. 2024-03-06T07:27:05.616 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:05.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.715+0000 7fb14355c640 1 Processor -- start 2024-03-06T07:27:05.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.715+0000 7fb14355c640 1 -- start start 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.716+0000 7fb14355c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb13c0c6b10 0x7fb13c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.716+0000 7fb14355c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb13c0bf9d0 0x7fb13c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.716+0000 7fb14355c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13c0c2fc0 con 0x7fb13c152650 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.716+0000 7fb14355c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13c155800 con 0x7fb13c0bf9d0 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.716+0000 7fb14355c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c158da0 con 0x7fb13c0c6b10 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.716+0000 7fb141af9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb13c0bf9d0 0x7fb13c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.716+0000 7fb1412f8640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb13c152650 0x7fb13c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44352/0 (socket says 172.21.15.1:44352) 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.716+0000 7fb1412f8640 1 -- 172.21.15.1:0/4027798715 learned_addr learned my addr 172.21.15.1:0/4027798715 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.717+0000 7fb13bfff640 1 -- 172.21.15.1:0/4027798715 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835187481 0 0) 0x7fb13c0c2fc0 con 0x7fb13c152650 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.717+0000 7fb140af7640 1 --2- 172.21.15.1:0/4027798715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb13c0c6b10 0x7fb13c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.717+0000 7fb13bfff640 1 -- 172.21.15.1:0/4027798715 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb124003520 con 0x7fb13c152650 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.717+0000 7fb141af9640 1 -- 172.21.15.1:0/4027798715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb13c0c6b10 msgr2=0x7fb13c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.717+0000 7fb141af9640 1 --2- 172.21.15.1:0/4027798715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb13c0c6b10 0x7fb13c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.717+0000 7fb141af9640 1 -- 172.21.15.1:0/4027798715 >> v1:172.21.15.1:6789/0 conn(0x7fb13c152650 legacy=0x7fb13c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.717+0000 7fb141af9640 1 -- 172.21.15.1:0/4027798715 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c0c6690 con 0x7fb13c0bf9d0 2024-03-06T07:27:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.717+0000 7fb141af9640 1 --2- 172.21.15.1:0/4027798715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb13c0bf9d0 0x7fb13c0c1460 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0097a0 tx=0x7fb12c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711b61c334f2c7a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.718+0000 7fb13bfff640 1 -- 172.21.15.1:0/4027798715 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c0315a0 con 0x7fb13c0bf9d0 2024-03-06T07:27:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.718+0000 7fb13bfff640 1 -- 172.21.15.1:0/4027798715 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12c037070 con 0x7fb13c0bf9d0 2024-03-06T07:27:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.718+0000 7fb13bfff640 1 -- 172.21.15.1:0/4027798715 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c031960 con 0x7fb13c0bf9d0 2024-03-06T07:27:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.718+0000 7fb14355c640 1 -- 172.21.15.1:0/4027798715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb13c0bf9d0 msgr2=0x7fb13c0c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.718+0000 7fb14355c640 1 --2- 172.21.15.1:0/4027798715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb13c0bf9d0 0x7fb13c0c1460 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0097a0 tx=0x7fb12c02b840 comp rx=0 tx=0).stop 2024-03-06T07:27:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.718+0000 7fb14355c640 1 -- 172.21.15.1:0/4027798715 shutdown_connections 2024-03-06T07:27:05.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.718+0000 7fb14355c640 1 --2- 172.21.15.1:0/4027798715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb13c0c6b10 0x7fb13c155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.718+0000 7fb14355c640 1 --2- 172.21.15.1:0/4027798715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb13c0bf9d0 0x7fb13c0c1460 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.718+0000 7fb14355c640 1 -- 172.21.15.1:0/4027798715 >> 172.21.15.1:0/4027798715 conn(0x7fb13c066ee0 msgr2=0x7fb13c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.718+0000 7fb14355c640 1 -- 172.21.15.1:0/4027798715 shutdown_connections 2024-03-06T07:27:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.718+0000 7fb14355c640 1 -- 172.21.15.1:0/4027798715 wait complete. 2024-03-06T07:27:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.719+0000 7fb14355c640 1 Processor -- start 2024-03-06T07:27:05.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.719+0000 7fb14355c640 1 -- start start 2024-03-06T07:27:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb14355c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13c0bf9d0 0x7fb13c1eb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb14355c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb13c152650 0x7fb13c1541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb14355c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb13c1587c0 0x7fb13c14d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb14355c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c1551b0 con 0x7fb13c152650 2024-03-06T07:27:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb14355c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c0c6fa0 con 0x7fb13c0bf9d0 2024-03-06T07:27:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb14355c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c155490 con 0x7fb13c1587c0 2024-03-06T07:27:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb140af7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb13c152650 0x7fb13c1541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb140af7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb13c152650 0x7fb13c1541b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59388/0 (socket says 172.21.15.1:59388) 2024-03-06T07:27:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb140af7640 1 -- 172.21.15.1:0/3114358061 learned_addr learned my addr 172.21.15.1:0/3114358061 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb141af9640 1 --2- 172.21.15.1:0/3114358061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb13c1587c0 0x7fb13c14d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb1412f8640 1 --2- 172.21.15.1:0/3114358061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13c0bf9d0 0x7fb13c1eb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb140af7640 1 -- 172.21.15.1:0/3114358061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb13c1587c0 msgr2=0x7fb13c14d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb140af7640 1 --2- 172.21.15.1:0/3114358061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb13c1587c0 0x7fb13c14d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb140af7640 1 -- 172.21.15.1:0/3114358061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13c0bf9d0 msgr2=0x7fb13c1eb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb140af7640 1 --2- 172.21.15.1:0/3114358061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13c0bf9d0 0x7fb13c1eb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb140af7640 1 -- 172.21.15.1:0/3114358061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c0aa140 con 0x7fb13c152650 2024-03-06T07:27:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb141af9640 1 --2- 172.21.15.1:0/3114358061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb13c1587c0 0x7fb13c14d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.720+0000 7fb1412f8640 1 --2- 172.21.15.1:0/3114358061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13c0bf9d0 0x7fb13c1eb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.721+0000 7fb140af7640 1 --2- 172.21.15.1:0/3114358061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb13c152650 0x7fb13c1541b0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fb13000d800 tx=0x7fb13000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.721+0000 7fb139ffb640 1 -- 172.21.15.1:0/3114358061 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13000b6c0 con 0x7fb13c152650 2024-03-06T07:27:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.721+0000 7fb139ffb640 1 -- 172.21.15.1:0/3114358061 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb130019070 con 0x7fb13c152650 2024-03-06T07:27:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.721+0000 7fb14355c640 1 -- 172.21.15.1:0/3114358061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13c157ed0 con 0x7fb13c152650 2024-03-06T07:27:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.721+0000 7fb139ffb640 1 -- 172.21.15.1:0/3114358061 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13000ba80 con 0x7fb13c152650 2024-03-06T07:27:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.721+0000 7fb14355c640 1 -- 172.21.15.1:0/3114358061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13c157bf0 con 0x7fb13c152650 2024-03-06T07:27:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.722+0000 7fb14355c640 1 -- 172.21.15.1:0/3114358061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13c06b540 con 0x7fb13c152650 2024-03-06T07:27:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.722+0000 7fb139ffb640 1 -- 172.21.15.1:0/3114358061 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb13000bc20 con 0x7fb13c152650 2024-03-06T07:27:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.723+0000 7fb139ffb640 1 --2- 172.21.15.1:0/3114358061 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb12806b4d0 0x7fb12806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.723+0000 7fb139ffb640 1 -- 172.21.15.1:0/3114358061 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb13008a7e0 con 0x7fb13c152650 2024-03-06T07:27:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.723+0000 7fb1412f8640 1 --2- 172.21.15.1:0/3114358061 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb12806b4d0 0x7fb12806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.724+0000 7fb1412f8640 1 --2- 172.21.15.1:0/3114358061 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb12806b4d0 0x7fb12806d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb134005c00 tx=0x7fb134009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.727+0000 7fb139ffb640 1 -- 172.21.15.1:0/3114358061 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb13c06b540 con 0x7fb13c152650 2024-03-06T07:27:05.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.911+0000 7fb14355c640 1 -- 172.21.15.1:0/3114358061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb13c067fd0 con 0x7fb13c152650 2024-03-06T07:27:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.912+0000 7fb139ffb640 1 -- 172.21.15.1:0/3114358061 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) v1 ==== 74+0+16845 (secure 0 0 0) 0x7fb130059560 con 0x7fb13c152650 2024-03-06T07:27:05.914 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:05.914 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":170,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:27:04.618780+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:02.888292+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:03.804423+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:05.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.914+0000 7fb14355c640 1 -- 172.21.15.1:0/3114358061 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb12806b4d0 msgr2=0x7fb12806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.914+0000 7fb14355c640 1 --2- 172.21.15.1:0/3114358061 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb12806b4d0 0x7fb12806d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb134005c00 tx=0x7fb134009040 comp rx=0 tx=0).stop 2024-03-06T07:27:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.914+0000 7fb14355c640 1 -- 172.21.15.1:0/3114358061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb13c152650 msgr2=0x7fb13c1541b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.914+0000 7fb14355c640 1 --2- 172.21.15.1:0/3114358061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb13c152650 0x7fb13c1541b0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fb13000d800 tx=0x7fb13000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.914+0000 7fb14355c640 1 -- 172.21.15.1:0/3114358061 shutdown_connections 2024-03-06T07:27:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.914+0000 7fb14355c640 1 --2- 172.21.15.1:0/3114358061 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb12806b4d0 0x7fb12806d990 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.914+0000 7fb14355c640 1 --2- 172.21.15.1:0/3114358061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb13c1587c0 0x7fb13c14d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.914+0000 7fb14355c640 1 --2- 172.21.15.1:0/3114358061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb13c152650 0x7fb13c1541b0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.914+0000 7fb14355c640 1 --2- 172.21.15.1:0/3114358061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13c0bf9d0 0x7fb13c1eb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.914+0000 7fb14355c640 1 -- 172.21.15.1:0/3114358061 >> 172.21.15.1:0/3114358061 conn(0x7fb13c066ee0 msgr2=0x7fb13c0c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.914+0000 7fb14355c640 1 -- 172.21.15.1:0/3114358061 shutdown_connections 2024-03-06T07:27:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:05.914+0000 7fb14355c640 1 -- 172.21.15.1:0/3114358061 wait complete. 2024-03-06T07:27:05.927 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:27:06.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.029+0000 7fe597276640 1 Processor -- start 2024-03-06T07:27:06.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.030+0000 7fe597276640 1 -- start start 2024-03-06T07:27:06.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.030+0000 7fe597276640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe5900c6b10 0x7fe590155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.030+0000 7fe597276640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5900c2fc0 con 0x7fe590152650 2024-03-06T07:27:06.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.030+0000 7fe597276640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe590158da0 con 0x7fe5900bf9d0 2024-03-06T07:27:06.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.030+0000 7fe597276640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe590155800 con 0x7fe5900c6b10 2024-03-06T07:27:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.030+0000 7fe595012640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe590152650 0x7fe590158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44356/0 (socket says 172.21.15.1:44356) 2024-03-06T07:27:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.030+0000 7fe595012640 1 -- 172.21.15.1:0/551763369 learned_addr learned my addr 172.21.15.1:0/551763369 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe594811640 1 --2- 172.21.15.1:0/551763369 >> v2:172.21.15.98:3300/0 conn(0x7fe5900c6b10 0x7fe590155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe58ffff640 1 -- 172.21.15.1:0/551763369 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309468911 0 0) 0x7fe5900c2fc0 con 0x7fe590152650 2024-03-06T07:27:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe58ffff640 1 -- 172.21.15.1:0/551763369 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe578003520 con 0x7fe590152650 2024-03-06T07:27:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe58ffff640 1 -- 172.21.15.1:0/551763369 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2701646299 0 0) 0x7fe578003520 con 0x7fe590152650 2024-03-06T07:27:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe58ffff640 1 -- 172.21.15.1:0/551763369 >> v2:172.21.15.98:3300/0 conn(0x7fe5900c6b10 msgr2=0x7fe590155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe58ffff640 1 --2- 172.21.15.1:0/551763369 >> v2:172.21.15.98:3300/0 conn(0x7fe5900c6b10 0x7fe590155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe58ffff640 1 -- 172.21.15.1:0/551763369 >> v1:172.21.15.98:6789/0 conn(0x7fe5900bf9d0 legacy=0x7fe5900c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe58ffff640 1 -- 172.21.15.1:0/551763369 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5900c6690 con 0x7fe590152650 2024-03-06T07:27:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe58ffff640 1 -- 172.21.15.1:0/551763369 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fe588001670 con 0x7fe590152650 2024-03-06T07:27:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe58ffff640 1 -- 172.21.15.1:0/551763369 >> v1:172.21.15.1:6789/0 conn(0x7fe590152650 legacy=0x7fe590158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe58ffff640 1 --2- 172.21.15.1:0/551763369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe578003a00 0x7fe578003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe58ffff640 1 --2- 172.21.15.1:0/551763369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe578004800 0x7fe578004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe58ffff640 1 --2- 172.21.15.1:0/551763369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe578005210 0x7fe57800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe58ffff640 1 -- 172.21.15.1:0/551763369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe590155800 con 0x7fe578004800 2024-03-06T07:27:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe58ffff640 1 -- 172.21.15.1:0/551763369 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57800c160 con 0x7fe578003a00 2024-03-06T07:27:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe58ffff640 1 -- 172.21.15.1:0/551763369 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57800c360 con 0x7fe578005210 2024-03-06T07:27:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe594811640 1 --2- 172.21.15.1:0/551763369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe578004800 0x7fe578004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe595012640 1 --2- 172.21.15.1:0/551763369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe578003a00 0x7fe578003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe594811640 1 -- 172.21.15.1:0/551763369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe578005210 msgr2=0x7fe57800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe595813640 1 --2- 172.21.15.1:0/551763369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe578005210 0x7fe57800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe594811640 1 --2- 172.21.15.1:0/551763369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe578005210 0x7fe57800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe594811640 1 -- 172.21.15.1:0/551763369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe578003a00 msgr2=0x7fe578003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe594811640 1 --2- 172.21.15.1:0/551763369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe578003a00 0x7fe578003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe594811640 1 -- 172.21.15.1:0/551763369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe578003520 con 0x7fe578004800 2024-03-06T07:27:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe595012640 1 --2- 172.21.15.1:0/551763369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe578003a00 0x7fe578003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe595813640 1 --2- 172.21.15.1:0/551763369 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe578005210 0x7fe57800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.031+0000 7fe595813640 1 -- 172.21.15.1:0/551763369 reap_dead start 2024-03-06T07:27:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.032+0000 7fe594811640 1 --2- 172.21.15.1:0/551763369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe578004800 0x7fe578004c80 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fe584009a90 tx=0x7fe578005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a057d3634aa263 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.032+0000 7fe58ffff640 1 -- 172.21.15.1:0/551763369 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe584039070 con 0x7fe578004800 2024-03-06T07:27:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.032+0000 7fe58ffff640 1 -- 172.21.15.1:0/551763369 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58402c680 con 0x7fe578004800 2024-03-06T07:27:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.032+0000 7fe58ffff640 1 -- 172.21.15.1:0/551763369 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe584009690 con 0x7fe578004800 2024-03-06T07:27:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.033+0000 7fe597276640 1 -- 172.21.15.1:0/551763369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe578004800 msgr2=0x7fe578004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.033+0000 7fe597276640 1 --2- 172.21.15.1:0/551763369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe578004800 0x7fe578004c80 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fe584009a90 tx=0x7fe578005830 comp rx=0 tx=0).stop 2024-03-06T07:27:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.033+0000 7fe597276640 1 -- 172.21.15.1:0/551763369 shutdown_connections 2024-03-06T07:27:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.033+0000 7fe597276640 1 --2- 172.21.15.1:0/551763369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe578004800 0x7fe578004c80 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.033+0000 7fe597276640 1 -- 172.21.15.1:0/551763369 >> 172.21.15.1:0/551763369 conn(0x7fe590066ee0 msgr2=0x7fe5900672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.033+0000 7fe597276640 1 -- 172.21.15.1:0/551763369 shutdown_connections 2024-03-06T07:27:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.033+0000 7fe597276640 1 -- 172.21.15.1:0/551763369 wait complete. 2024-03-06T07:27:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.034+0000 7fe597276640 1 Processor -- start 2024-03-06T07:27:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.034+0000 7fe597276640 1 -- start start 2024-03-06T07:27:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe597276640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe578004800 0x7fe59014e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe597276640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe5901eb710 0x7fe59014b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe597276640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5901588c0 0x7fe59014b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe597276640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5901551b0 con 0x7fe578004800 2024-03-06T07:27:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe597276640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5900c6fa0 con 0x7fe5901588c0 2024-03-06T07:27:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe597276640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe590155490 con 0x7fe5901eb710 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe595012640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe578004800 0x7fe59014e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe595012640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe578004800 0x7fe59014e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59410/0 (socket says 172.21.15.1:59410) 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe595012640 1 -- 172.21.15.1:0/924782476 learned_addr learned my addr 172.21.15.1:0/924782476 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe595813640 1 --2- 172.21.15.1:0/924782476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5901588c0 0x7fe59014b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe594811640 1 --2- 172.21.15.1:0/924782476 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe5901eb710 0x7fe59014b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe595012640 1 -- 172.21.15.1:0/924782476 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe5901eb710 msgr2=0x7fe59014b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe595012640 1 --2- 172.21.15.1:0/924782476 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe5901eb710 0x7fe59014b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe595012640 1 -- 172.21.15.1:0/924782476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5901588c0 msgr2=0x7fe59014b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe595012640 1 --2- 172.21.15.1:0/924782476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5901588c0 0x7fe59014b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe595012640 1 -- 172.21.15.1:0/924782476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5901513f0 con 0x7fe578004800 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe594811640 1 --2- 172.21.15.1:0/924782476 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe5901eb710 0x7fe59014b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.035+0000 7fe595012640 1 --2- 172.21.15.1:0/924782476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe578004800 0x7fe59014e710 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fe588000fb0 tx=0x7fe58800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.036+0000 7fe58dffb640 1 -- 172.21.15.1:0/924782476 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58800ee60 con 0x7fe578004800 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.036+0000 7fe58dffb640 1 -- 172.21.15.1:0/924782476 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe588018070 con 0x7fe578004800 2024-03-06T07:27:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.036+0000 7fe597276640 1 -- 172.21.15.1:0/924782476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5901516d0 con 0x7fe578004800 2024-03-06T07:27:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.036+0000 7fe58dffb640 1 -- 172.21.15.1:0/924782476 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58800cad0 con 0x7fe578004800 2024-03-06T07:27:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.036+0000 7fe597276640 1 -- 172.21.15.1:0/924782476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe590151c90 con 0x7fe578004800 2024-03-06T07:27:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.037+0000 7fe597276640 1 -- 172.21.15.1:0/924782476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe590067fd0 con 0x7fe578004800 2024-03-06T07:27:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.037+0000 7fe58dffb640 1 -- 172.21.15.1:0/924782476 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe58800cc70 con 0x7fe578004800 2024-03-06T07:27:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.037+0000 7fe58dffb640 1 --2- 172.21.15.1:0/924782476 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe57c06b4d0 0x7fe57c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.037+0000 7fe58dffb640 1 -- 172.21.15.1:0/924782476 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe58808a2b0 con 0x7fe578004800 2024-03-06T07:27:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.038+0000 7fe594811640 1 --2- 172.21.15.1:0/924782476 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe57c06b4d0 0x7fe57c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.038+0000 7fe594811640 1 --2- 172.21.15.1:0/924782476 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe57c06b4d0 0x7fe57c06d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fe584036000 tx=0x7fe584030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.042+0000 7fe58dffb640 1 -- 172.21.15.1:0/924782476 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe588058fa0 con 0x7fe578004800 2024-03-06T07:27:06.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.260+0000 7fe597276640 1 -- 172.21.15.1:0/924782476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe590151e60 con 0x7fe578004800 2024-03-06T07:27:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.270+0000 7fe58dffb640 1 -- 172.21.15.1:0/924782476 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v156) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe5880588a0 con 0x7fe578004800 2024-03-06T07:27:06.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.272+0000 7fe597276640 1 -- 172.21.15.1:0/924782476 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe57c06b4d0 msgr2=0x7fe57c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.272+0000 7fe597276640 1 --2- 172.21.15.1:0/924782476 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe57c06b4d0 0x7fe57c06d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fe584036000 tx=0x7fe584030000 comp rx=0 tx=0).stop 2024-03-06T07:27:06.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.273+0000 7fe597276640 1 -- 172.21.15.1:0/924782476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe578004800 msgr2=0x7fe59014e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.273+0000 7fe597276640 1 --2- 172.21.15.1:0/924782476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe578004800 0x7fe59014e710 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fe588000fb0 tx=0x7fe58800ec10 comp rx=0 tx=0).stop 2024-03-06T07:27:06.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.273+0000 7fe597276640 1 -- 172.21.15.1:0/924782476 shutdown_connections 2024-03-06T07:27:06.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.273+0000 7fe597276640 1 --2- 172.21.15.1:0/924782476 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe57c06b4d0 0x7fe57c06d990 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.273+0000 7fe597276640 1 --2- 172.21.15.1:0/924782476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5901588c0 0x7fe59014b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.273+0000 7fe597276640 1 --2- 172.21.15.1:0/924782476 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe5901eb710 0x7fe59014b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.273+0000 7fe597276640 1 --2- 172.21.15.1:0/924782476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe578004800 0x7fe59014e710 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.273+0000 7fe597276640 1 -- 172.21.15.1:0/924782476 >> 172.21.15.1:0/924782476 conn(0x7fe590066ee0 msgr2=0x7fe5900c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.273+0000 7fe597276640 1 -- 172.21.15.1:0/924782476 shutdown_connections 2024-03-06T07:27:06.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.273+0000 7fe597276640 1 -- 172.21.15.1:0/924782476 wait complete. 2024-03-06T07:27:06.290 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:27:06.290 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:27:06.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.392+0000 7f1ebe839640 1 Processor -- start 2024-03-06T07:27:06.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.392+0000 7f1ebe839640 1 -- start start 2024-03-06T07:27:06.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.392+0000 7f1ebe839640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1eb8152650 0x7f1eb8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.392+0000 7f1ebe839640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1eb80c6b10 0x7f1eb8158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.392+0000 7f1ebe839640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb80c2fc0 con 0x7f1eb80bf9d0 2024-03-06T07:27:06.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.392+0000 7f1ebe839640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb8155800 con 0x7f1eb80c6b10 2024-03-06T07:27:06.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.392+0000 7f1ebe839640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb80c6690 con 0x7f1eb8152650 2024-03-06T07:27:06.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.392+0000 7f1eb77fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1eb80c6b10 0x7f1eb8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.392+0000 7f1eb77fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1eb80c6b10 0x7f1eb8158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59424/0 (socket says 172.21.15.1:59424) 2024-03-06T07:27:06.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.392+0000 7f1ebcdd6640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1eb80bf9d0 0x7f1eb80c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44362/0 (socket says 172.21.15.1:44362) 2024-03-06T07:27:06.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.392+0000 7f1eb77fe640 1 -- 172.21.15.1:0/300674685 learned_addr learned my addr 172.21.15.1:0/300674685 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:06.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.393+0000 7f1eb7fff640 1 --2- 172.21.15.1:0/300674685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1eb8152650 0x7f1eb8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.393+0000 7f1eb77fe640 1 -- 172.21.15.1:0/300674685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1eb8152650 msgr2=0x7f1eb8155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.393+0000 7f1eb77fe640 1 --2- 172.21.15.1:0/300674685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1eb8152650 0x7f1eb8155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.393+0000 7f1eb77fe640 1 -- 172.21.15.1:0/300674685 >> v1:172.21.15.1:6789/0 conn(0x7f1eb80bf9d0 legacy=0x7f1eb80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.393+0000 7f1eb6ffd640 1 -- 172.21.15.1:0/300674685 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566438349 0 0) 0x7f1eb80c2fc0 con 0x7f1eb80bf9d0 2024-03-06T07:27:06.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.393+0000 7f1eb77fe640 1 -- 172.21.15.1:0/300674685 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb80c7280 con 0x7f1eb80c6b10 2024-03-06T07:27:06.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.393+0000 7f1eb7fff640 1 --2- 172.21.15.1:0/300674685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1eb8152650 0x7f1eb8155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:06.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.394+0000 7f1eb77fe640 1 --2- 172.21.15.1:0/300674685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1eb80c6b10 0x7f1eb8158690 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f1eac009870 tx=0x7f1eac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ee396d083f803c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.394+0000 7f1eb6ffd640 1 -- 172.21.15.1:0/300674685 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac035020 con 0x7f1eb80c6b10 2024-03-06T07:27:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.394+0000 7f1eb6ffd640 1 -- 172.21.15.1:0/300674685 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac031640 con 0x7f1eb80c6b10 2024-03-06T07:27:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.394+0000 7f1eb6ffd640 1 -- 172.21.15.1:0/300674685 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac02d040 con 0x7f1eb80c6b10 2024-03-06T07:27:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.394+0000 7f1ebe839640 1 -- 172.21.15.1:0/300674685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1eb80c6b10 msgr2=0x7f1eb8158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.394+0000 7f1ebe839640 1 --2- 172.21.15.1:0/300674685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1eb80c6b10 0x7f1eb8158690 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f1eac009870 tx=0x7f1eac02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.394+0000 7f1ebe839640 1 -- 172.21.15.1:0/300674685 shutdown_connections 2024-03-06T07:27:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.394+0000 7f1ebe839640 1 --2- 172.21.15.1:0/300674685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1eb80c6b10 0x7f1eb8158690 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.394+0000 7f1ebe839640 1 --2- 172.21.15.1:0/300674685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1eb8152650 0x7f1eb8155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.394+0000 7f1ebe839640 1 -- 172.21.15.1:0/300674685 >> 172.21.15.1:0/300674685 conn(0x7f1eb8066ee0 msgr2=0x7f1eb80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.394+0000 7f1ebe839640 1 -- 172.21.15.1:0/300674685 shutdown_connections 2024-03-06T07:27:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.394+0000 7f1ebe839640 1 -- 172.21.15.1:0/300674685 wait complete. 2024-03-06T07:27:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.395+0000 7f1ebe839640 1 Processor -- start 2024-03-06T07:27:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.396+0000 7f1ebe839640 1 -- start start 2024-03-06T07:27:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.396+0000 7f1ebe839640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1eb80bf9d0 0x7f1eb8152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.396+0000 7f1ebe839640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1eb8152650 0x7f1eb81eb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.396+0000 7f1ebe839640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eb8154630 0x7f1eb814d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.396+0000 7f1ebe839640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb8155490 con 0x7f1eb80bf9d0 2024-03-06T07:27:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.396+0000 7f1ebe839640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb81551b0 con 0x7f1eb8154630 2024-03-06T07:27:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.396+0000 7f1ebe839640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb8152a30 con 0x7f1eb8152650 2024-03-06T07:27:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.396+0000 7f1eb7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1eb80bf9d0 0x7f1eb8152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.396+0000 7f1eb7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1eb80bf9d0 0x7f1eb8152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59438/0 (socket says 172.21.15.1:59438) 2024-03-06T07:27:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.396+0000 7f1eb7fff640 1 -- 172.21.15.1:0/3702486188 learned_addr learned my addr 172.21.15.1:0/3702486188 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.397+0000 7f1eb77fe640 1 --2- 172.21.15.1:0/3702486188 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1eb8152650 0x7f1eb81eb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.397+0000 7f1eb7fff640 1 -- 172.21.15.1:0/3702486188 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1eb8152650 msgr2=0x7f1eb81eb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.397+0000 7f1eb7fff640 1 --2- 172.21.15.1:0/3702486188 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1eb8152650 0x7f1eb81eb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.397+0000 7f1eb7fff640 1 -- 172.21.15.1:0/3702486188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eb8154630 msgr2=0x7f1eb814d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.397+0000 7f1eb7fff640 1 --2- 172.21.15.1:0/3702486188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eb8154630 0x7f1eb814d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.397+0000 7f1eb7fff640 1 -- 172.21.15.1:0/3702486188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb8153450 con 0x7f1eb80bf9d0 2024-03-06T07:27:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.397+0000 7f1eb77fe640 1 --2- 172.21.15.1:0/3702486188 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1eb8152650 0x7f1eb81eb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.397+0000 7f1eb7fff640 1 --2- 172.21.15.1:0/3702486188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1eb80bf9d0 0x7f1eb8152f10 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f1ea800c7f0 tx=0x7f1ea800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.397+0000 7f1eb4ff9640 1 -- 172.21.15.1:0/3702486188 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8015070 con 0x7f1eb80bf9d0 2024-03-06T07:27:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.397+0000 7f1eb4ff9640 1 -- 172.21.15.1:0/3702486188 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea8007a30 con 0x7f1eb80bf9d0 2024-03-06T07:27:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.397+0000 7f1ebe839640 1 -- 172.21.15.1:0/3702486188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb81ebdd0 con 0x7f1eb80bf9d0 2024-03-06T07:27:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.397+0000 7f1eb4ff9640 1 -- 172.21.15.1:0/3702486188 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8007e00 con 0x7f1eb80bf9d0 2024-03-06T07:27:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.397+0000 7f1ebe839640 1 -- 172.21.15.1:0/3702486188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb8154da0 con 0x7f1eb80bf9d0 2024-03-06T07:27:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.398+0000 7f1ebe839640 1 -- 172.21.15.1:0/3702486188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb8067fd0 con 0x7f1eb80bf9d0 2024-03-06T07:27:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.398+0000 7f1eb4ff9640 1 -- 172.21.15.1:0/3702486188 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ea8018070 con 0x7f1eb80bf9d0 2024-03-06T07:27:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.399+0000 7f1eb4ff9640 1 --2- 172.21.15.1:0/3702486188 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1ea406b500 0x7f1ea406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.399+0000 7f1eb4ff9640 1 -- 172.21.15.1:0/3702486188 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1ea808a250 con 0x7f1eb80bf9d0 2024-03-06T07:27:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.399+0000 7f1eb77fe640 1 --2- 172.21.15.1:0/3702486188 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1ea406b500 0x7f1ea406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.400+0000 7f1eb77fe640 1 --2- 172.21.15.1:0/3702486188 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1ea406b500 0x7f1ea406d9c0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f1eac036000 tx=0x7f1eac008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.407+0000 7f1eb4ff9640 1 -- 172.21.15.1:0/3702486188 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ea8058fd0 con 0x7f1eb80bf9d0 2024-03-06T07:27:06.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:06.626+0000 7f1ebe839640 1 -- 172.21.15.1:0/3702486188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1eb80b06f0 con 0x7f1eb80bf9d0 2024-03-06T07:27:07.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.281+0000 7f1eb4ff9640 1 -- 172.21.15.1:0/3702486188 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v157) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1ea80588d0 con 0x7f1eb80bf9d0 2024-03-06T07:27:07.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.283+0000 7f1ebe839640 1 -- 172.21.15.1:0/3702486188 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1ea406b500 msgr2=0x7f1ea406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.283+0000 7f1ebe839640 1 --2- 172.21.15.1:0/3702486188 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1ea406b500 0x7f1ea406d9c0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f1eac036000 tx=0x7f1eac008030 comp rx=0 tx=0).stop 2024-03-06T07:27:07.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.283+0000 7f1ebe839640 1 -- 172.21.15.1:0/3702486188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1eb80bf9d0 msgr2=0x7f1eb8152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.283+0000 7f1ebe839640 1 --2- 172.21.15.1:0/3702486188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1eb80bf9d0 0x7f1eb8152f10 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f1ea800c7f0 tx=0x7f1ea800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.283+0000 7f1ebe839640 1 -- 172.21.15.1:0/3702486188 shutdown_connections 2024-03-06T07:27:07.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.283+0000 7f1ebe839640 1 --2- 172.21.15.1:0/3702486188 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1ea406b500 0x7f1ea406d9c0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.283+0000 7f1ebe839640 1 --2- 172.21.15.1:0/3702486188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eb8154630 0x7f1eb814d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.283+0000 7f1ebe839640 1 --2- 172.21.15.1:0/3702486188 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1eb8152650 0x7f1eb81eb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.283+0000 7f1ebe839640 1 --2- 172.21.15.1:0/3702486188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1eb80bf9d0 0x7f1eb8152f10 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.283+0000 7f1ebe839640 1 -- 172.21.15.1:0/3702486188 >> 172.21.15.1:0/3702486188 conn(0x7f1eb8066ee0 msgr2=0x7f1eb80672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.283+0000 7f1ebe839640 1 -- 172.21.15.1:0/3702486188 shutdown_connections 2024-03-06T07:27:07.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.283+0000 7f1ebe839640 1 -- 172.21.15.1:0/3702486188 wait complete. 2024-03-06T07:27:07.299 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:27:07.300 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:07.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.426+0000 7f095a313640 1 Processor -- start 2024-03-06T07:27:07.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.426+0000 7f095a313640 1 -- start start 2024-03-06T07:27:07.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.427+0000 7f095a313640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0954152730 0x7f09540c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.427+0000 7f095a313640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09541588e0 con 0x7f09540c6fd0 2024-03-06T07:27:07.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.427+0000 7f095a313640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0954158bc0 con 0x7f09540c1aa0 2024-03-06T07:27:07.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.427+0000 7f095a313640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0954158ea0 con 0x7f0954152730 2024-03-06T07:27:07.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.427+0000 7f09537fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f09540c6fd0 0x7f09540c1390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44374/0 (socket says 172.21.15.1:44374) 2024-03-06T07:27:07.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.427+0000 7f09537fe640 1 -- 172.21.15.1:0/2785912628 learned_addr learned my addr 172.21.15.1:0/2785912628 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.427+0000 7f0953fff640 1 --2- 172.21.15.1:0/2785912628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0954152730 0x7f09540c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.427+0000 7f0952ffd640 1 -- 172.21.15.1:0/2785912628 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083293355 0 0) 0x7f09541588e0 con 0x7f09540c6fd0 2024-03-06T07:27:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.427+0000 7f0952ffd640 1 -- 172.21.15.1:0/2785912628 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f093c003520 con 0x7f09540c6fd0 2024-03-06T07:27:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.427+0000 7f0952ffd640 1 -- 172.21.15.1:0/2785912628 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 481066019 0 0) 0x7f093c003520 con 0x7f09540c6fd0 2024-03-06T07:27:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0952ffd640 1 -- 172.21.15.1:0/2785912628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0954152730 msgr2=0x7f09540c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0952ffd640 1 --2- 172.21.15.1:0/2785912628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0954152730 0x7f09540c6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0952ffd640 1 -- 172.21.15.1:0/2785912628 >> v1:172.21.15.98:6789/0 conn(0x7f09540c1aa0 legacy=0x7f09541552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0952ffd640 1 -- 172.21.15.1:0/2785912628 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0954159180 con 0x7f09540c6fd0 2024-03-06T07:27:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0952ffd640 1 -- 172.21.15.1:0/2785912628 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f0944001690 con 0x7f09540c6fd0 2024-03-06T07:27:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0952ffd640 1 -- 172.21.15.1:0/2785912628 >> v1:172.21.15.1:6789/0 conn(0x7f09540c6fd0 legacy=0x7f09540c1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0952ffd640 1 --2- 172.21.15.1:0/2785912628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f093c0039e0 0x7f093c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0952ffd640 1 --2- 172.21.15.1:0/2785912628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093c004800 0x7f093c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0952ffd640 1 --2- 172.21.15.1:0/2785912628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0954152730 0x7f093c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0952ffd640 1 -- 172.21.15.1:0/2785912628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0954158ea0 con 0x7f093c0039e0 2024-03-06T07:27:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0952ffd640 1 -- 172.21.15.1:0/2785912628 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093c00baf0 con 0x7f093c004800 2024-03-06T07:27:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0952ffd640 1 -- 172.21.15.1:0/2785912628 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093c00bcf0 con 0x7f0954152730 2024-03-06T07:27:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0953fff640 1 --2- 172.21.15.1:0/2785912628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f093c0039e0 0x7f093c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f09537fe640 1 --2- 172.21.15.1:0/2785912628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093c004800 0x7f093c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f09588b0640 1 --2- 172.21.15.1:0/2785912628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0954152730 0x7f093c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0953fff640 1 -- 172.21.15.1:0/2785912628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0954152730 msgr2=0x7f093c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0953fff640 1 --2- 172.21.15.1:0/2785912628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0954152730 0x7f093c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0953fff640 1 -- 172.21.15.1:0/2785912628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093c004800 msgr2=0x7f093c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0953fff640 1 --2- 172.21.15.1:0/2785912628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f093c004800 0x7f093c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f0953fff640 1 -- 172.21.15.1:0/2785912628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093c003520 con 0x7f093c0039e0 2024-03-06T07:27:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.428+0000 7f09588b0640 1 --2- 172.21.15.1:0/2785912628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0954152730 0x7f093c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.429+0000 7f0953fff640 1 --2- 172.21.15.1:0/2785912628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f093c0039e0 0x7f093c003dc0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f0948009ac0 tx=0x7f093c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f19e7979f3bcb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.429+0000 7f0952ffd640 1 -- 172.21.15.1:0/2785912628 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948039070 con 0x7f093c0039e0 2024-03-06T07:27:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.429+0000 7f0952ffd640 1 -- 172.21.15.1:0/2785912628 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094802c6b0 con 0x7f093c0039e0 2024-03-06T07:27:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.429+0000 7f0952ffd640 1 -- 172.21.15.1:0/2785912628 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948009690 con 0x7f093c0039e0 2024-03-06T07:27:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.430+0000 7f095a313640 1 -- 172.21.15.1:0/2785912628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f093c0039e0 msgr2=0x7f093c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.430+0000 7f095a313640 1 --2- 172.21.15.1:0/2785912628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f093c0039e0 0x7f093c003dc0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f0948009ac0 tx=0x7f093c005830 comp rx=0 tx=0).stop 2024-03-06T07:27:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.430+0000 7f09588b0640 1 -- 172.21.15.1:0/2785912628 reap_dead start 2024-03-06T07:27:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.430+0000 7f095a313640 1 -- 172.21.15.1:0/2785912628 shutdown_connections 2024-03-06T07:27:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.430+0000 7f095a313640 1 -- 172.21.15.1:0/2785912628 >> 172.21.15.1:0/2785912628 conn(0x7f0954066d00 msgr2=0x7f0954067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.430+0000 7f095a313640 1 -- 172.21.15.1:0/2785912628 shutdown_connections 2024-03-06T07:27:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.430+0000 7f095a313640 1 -- 172.21.15.1:0/2785912628 wait complete. 2024-03-06T07:27:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.431+0000 7f095a313640 1 Processor -- start 2024-03-06T07:27:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.431+0000 7f095a313640 1 -- start start 2024-03-06T07:27:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f095a313640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f093c0039e0 0x7f09540afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f095a313640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09540aabd0 0x7f09540c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f095a313640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09541eac10 0x7f09541eb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f095a313640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09540c6a90 con 0x7f093c0039e0 2024-03-06T07:27:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f095a313640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0954152b10 con 0x7f09541eac10 2024-03-06T07:27:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f095a313640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09540c6c10 con 0x7f09540aabd0 2024-03-06T07:27:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f0953fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f093c0039e0 0x7f09540afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f0953fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f093c0039e0 0x7f09540afcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59446/0 (socket says 172.21.15.1:59446) 2024-03-06T07:27:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f0953fff640 1 -- 172.21.15.1:0/2810362970 learned_addr learned my addr 172.21.15.1:0/2810362970 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f09537fe640 1 --2- 172.21.15.1:0/2810362970 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09540aabd0 0x7f09540c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f09588b0640 1 --2- 172.21.15.1:0/2810362970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09541eac10 0x7f09541eb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f0953fff640 1 -- 172.21.15.1:0/2810362970 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09540aabd0 msgr2=0x7f09540c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f0953fff640 1 --2- 172.21.15.1:0/2810362970 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09540aabd0 0x7f09540c9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f0953fff640 1 -- 172.21.15.1:0/2810362970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09541eac10 msgr2=0x7f09541eb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f0953fff640 1 --2- 172.21.15.1:0/2810362970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09541eac10 0x7f09541eb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f0953fff640 1 -- 172.21.15.1:0/2810362970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09541ee220 con 0x7f093c0039e0 2024-03-06T07:27:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f09537fe640 1 --2- 172.21.15.1:0/2810362970 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09540aabd0 0x7f09540c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.432+0000 7f09588b0640 1 --2- 172.21.15.1:0/2810362970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09541eac10 0x7f09541eb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.433+0000 7f0953fff640 1 --2- 172.21.15.1:0/2810362970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f093c0039e0 0x7f09540afcb0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f0948000c00 tx=0x7f0948004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.433+0000 7f0950ff9640 1 -- 172.21.15.1:0/2810362970 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948042020 con 0x7f093c0039e0 2024-03-06T07:27:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.433+0000 7f0950ff9640 1 -- 172.21.15.1:0/2810362970 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094803a040 con 0x7f093c0039e0 2024-03-06T07:27:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.433+0000 7f095a313640 1 -- 172.21.15.1:0/2810362970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09541ee490 con 0x7f093c0039e0 2024-03-06T07:27:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.433+0000 7f0950ff9640 1 -- 172.21.15.1:0/2810362970 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948039070 con 0x7f093c0039e0 2024-03-06T07:27:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.433+0000 7f095a313640 1 -- 172.21.15.1:0/2810362970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09541eea50 con 0x7f093c0039e0 2024-03-06T07:27:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.434+0000 7f095a313640 1 -- 172.21.15.1:0/2810362970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09540aafb0 con 0x7f093c0039e0 2024-03-06T07:27:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.434+0000 7f0950ff9640 1 -- 172.21.15.1:0/2810362970 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0948045020 con 0x7f093c0039e0 2024-03-06T07:27:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.435+0000 7f0950ff9640 1 --2- 172.21.15.1:0/2810362970 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f094006b500 0x7f094006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.435+0000 7f0950ff9640 1 -- 172.21.15.1:0/2810362970 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f09480ab6b0 con 0x7f093c0039e0 2024-03-06T07:27:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.435+0000 7f09537fe640 1 --2- 172.21.15.1:0/2810362970 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f094006b500 0x7f094006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.436+0000 7f09537fe640 1 --2- 172.21.15.1:0/2810362970 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f094006b500 0x7f094006d9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f09540c3b50 tx=0x7f0944009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.439+0000 7f0950ff9640 1 -- 172.21.15.1:0/2810362970 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f094804c070 con 0x7f093c0039e0 2024-03-06T07:27:07.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.796+0000 7f095a313640 1 -- 172.21.15.1:0/2810362970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0954067fd0 con 0x7f093c0039e0 2024-03-06T07:27:07.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.797+0000 7f0950ff9640 1 -- 172.21.15.1:0/2810362970 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+4969 (secure 0 0 0) 0x7f094807a430 con 0x7f093c0039e0 2024-03-06T07:27:07.799 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:07.799 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":157,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:06.627754+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9489":{"gid":9489,"name":"c","rank":0,"incarnation":155,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1751109564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1751109564},{"type":"v1","addr":"172.21.15.1:6835","nonce":1751109564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9489,"qdb_cluster":[9489]},"id":13}]} 2024-03-06T07:27:07.799 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 157 2024-03-06T07:27:07.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.799+0000 7f095a313640 1 -- 172.21.15.1:0/2810362970 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f094006b500 msgr2=0x7f094006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.799+0000 7f095a313640 1 --2- 172.21.15.1:0/2810362970 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f094006b500 0x7f094006d9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f09540c3b50 tx=0x7f0944009040 comp rx=0 tx=0).stop 2024-03-06T07:27:07.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.799+0000 7f095a313640 1 -- 172.21.15.1:0/2810362970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f093c0039e0 msgr2=0x7f09540afcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.799+0000 7f095a313640 1 --2- 172.21.15.1:0/2810362970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f093c0039e0 0x7f09540afcb0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f0948000c00 tx=0x7f0948004730 comp rx=0 tx=0).stop 2024-03-06T07:27:07.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.799+0000 7f095a313640 1 -- 172.21.15.1:0/2810362970 shutdown_connections 2024-03-06T07:27:07.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.799+0000 7f095a313640 1 --2- 172.21.15.1:0/2810362970 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f094006b500 0x7f094006d9c0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.799+0000 7f095a313640 1 --2- 172.21.15.1:0/2810362970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09541eac10 0x7f09541eb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.799+0000 7f095a313640 1 --2- 172.21.15.1:0/2810362970 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09540aabd0 0x7f09540c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.799+0000 7f095a313640 1 --2- 172.21.15.1:0/2810362970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f093c0039e0 0x7f09540afcb0 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.799+0000 7f095a313640 1 -- 172.21.15.1:0/2810362970 >> 172.21.15.1:0/2810362970 conn(0x7f0954066d00 msgr2=0x7f09541519e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.799+0000 7f095a313640 1 -- 172.21.15.1:0/2810362970 shutdown_connections 2024-03-06T07:27:07.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.799+0000 7f095a313640 1 -- 172.21.15.1:0/2810362970 wait complete. 2024-03-06T07:27:07.815 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:07.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.942+0000 7f9cac6af640 1 Processor -- start 2024-03-06T07:27:07.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.942+0000 7f9cac6af640 1 -- start start 2024-03-06T07:27:07.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.943+0000 7f9cac6af640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9ca40c3970 0x7f9ca40c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.943+0000 7f9cac6af640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca40c5e40 con 0x7f9ca4152840 2024-03-06T07:27:07.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.943+0000 7f9cac6af640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca40c6a40 con 0x7f9ca40c60b0 2024-03-06T07:27:07.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.943+0000 7f9cac6af640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca40c67f0 con 0x7f9ca40c3970 2024-03-06T07:27:07.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.944+0000 7f9caa44b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9ca40c3970 0x7f9ca40c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.944+0000 7f9caac4c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9ca4152840 0x7f9ca41553d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44388/0 (socket says 172.21.15.1:44388) 2024-03-06T07:27:07.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.944+0000 7f9caa44b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9ca40c3970 0x7f9ca40c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59456/0 (socket says 172.21.15.1:59456) 2024-03-06T07:27:07.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.944+0000 7f9caac4c640 1 -- 172.21.15.1:0/307045394 learned_addr learned my addr 172.21.15.1:0/307045394 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:07.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.944+0000 7f9caa44b640 1 -- 172.21.15.1:0/307045394 >> v1:172.21.15.98:6789/0 conn(0x7f9ca40c60b0 legacy=0x7f9ca4152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.944+0000 7f9caa44b640 1 -- 172.21.15.1:0/307045394 >> v1:172.21.15.1:6789/0 conn(0x7f9ca4152840 legacy=0x7f9ca41553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.944+0000 7f9caa44b640 1 -- 172.21.15.1:0/307045394 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca40c6d20 con 0x7f9ca40c3970 2024-03-06T07:27:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.945+0000 7f9caa44b640 1 --2- 172.21.15.1:0/307045394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ca40c3970 0x7f9ca40c8170 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00097a0 tx=0x7f9ca002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fb64a2792a45a28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.945+0000 7f9ca9449640 1 -- 172.21.15.1:0/307045394 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0035020 con 0x7f9ca40c3970 2024-03-06T07:27:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.945+0000 7f9ca9449640 1 -- 172.21.15.1:0/307045394 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca0031830 con 0x7f9ca40c3970 2024-03-06T07:27:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.945+0000 7f9ca9449640 1 -- 172.21.15.1:0/307045394 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0031c00 con 0x7f9ca40c3970 2024-03-06T07:27:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.946+0000 7f9cac6af640 1 -- 172.21.15.1:0/307045394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ca40c3970 msgr2=0x7f9ca40c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.946+0000 7f9cac6af640 1 --2- 172.21.15.1:0/307045394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ca40c3970 0x7f9ca40c8170 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00097a0 tx=0x7f9ca002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.946+0000 7f9cac6af640 1 -- 172.21.15.1:0/307045394 shutdown_connections 2024-03-06T07:27:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.946+0000 7f9cac6af640 1 --2- 172.21.15.1:0/307045394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ca40c3970 0x7f9ca40c8170 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.946+0000 7f9cac6af640 1 -- 172.21.15.1:0/307045394 >> 172.21.15.1:0/307045394 conn(0x7f9ca40c88f0 msgr2=0x7f9ca40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.946+0000 7f9cac6af640 1 -- 172.21.15.1:0/307045394 shutdown_connections 2024-03-06T07:27:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.946+0000 7f9cac6af640 1 -- 172.21.15.1:0/307045394 wait complete. 2024-03-06T07:27:07.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.947+0000 7f9cac6af640 1 Processor -- start 2024-03-06T07:27:07.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.947+0000 7f9cac6af640 1 -- start start 2024-03-06T07:27:07.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.947+0000 7f9cac6af640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ca40c3970 0x7f9ca40a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.947+0000 7f9cac6af640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ca40c60b0 0x7f9ca41edfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.947+0000 7f9cac6af640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ca4152840 0x7f9ca41ee500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.947+0000 7f9cac6af640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40c75c0 con 0x7f9ca4152840 2024-03-06T07:27:07.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.947+0000 7f9cac6af640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40c72e0 con 0x7f9ca40c60b0 2024-03-06T07:27:07.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.947+0000 7f9cac6af640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40c3d50 con 0x7f9ca40c3970 2024-03-06T07:27:07.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9caac4c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ca4152840 0x7f9ca41ee500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9caac4c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ca4152840 0x7f9ca41ee500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59460/0 (socket says 172.21.15.1:59460) 2024-03-06T07:27:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9caac4c640 1 -- 172.21.15.1:0/180054293 learned_addr learned my addr 172.21.15.1:0/180054293 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9caa44b640 1 --2- 172.21.15.1:0/180054293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ca40c3970 0x7f9ca40a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9ca9c4a640 1 --2- 172.21.15.1:0/180054293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ca40c60b0 0x7f9ca41edfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9caac4c640 1 -- 172.21.15.1:0/180054293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ca40c3970 msgr2=0x7f9ca40a97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9caac4c640 1 --2- 172.21.15.1:0/180054293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ca40c3970 0x7f9ca40a97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9caac4c640 1 -- 172.21.15.1:0/180054293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ca40c60b0 msgr2=0x7f9ca41edfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9caac4c640 1 --2- 172.21.15.1:0/180054293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ca40c60b0 0x7f9ca41edfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9caac4c640 1 -- 172.21.15.1:0/180054293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca406f730 con 0x7f9ca4152840 2024-03-06T07:27:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9caa44b640 1 --2- 172.21.15.1:0/180054293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ca40c3970 0x7f9ca40a97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9ca9c4a640 1 --2- 172.21.15.1:0/180054293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ca40c60b0 0x7f9ca41edfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9caac4c640 1 --2- 172.21.15.1:0/180054293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ca4152840 0x7f9ca41ee500 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c00cad0 tx=0x7f9c9c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9c9affd640 1 -- 172.21.15.1:0/180054293 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c017070 con 0x7f9ca4152840 2024-03-06T07:27:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9c9affd640 1 -- 172.21.15.1:0/180054293 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9c00ac30 con 0x7f9ca4152840 2024-03-06T07:27:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9cac6af640 1 -- 172.21.15.1:0/180054293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca406fa10 con 0x7f9ca4152840 2024-03-06T07:27:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9c9affd640 1 -- 172.21.15.1:0/180054293 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c012400 con 0x7f9ca4152840 2024-03-06T07:27:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.948+0000 7f9cac6af640 1 -- 172.21.15.1:0/180054293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca406ffd0 con 0x7f9ca4152840 2024-03-06T07:27:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.949+0000 7f9cac6af640 1 -- 172.21.15.1:0/180054293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca41edd50 con 0x7f9ca4152840 2024-03-06T07:27:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.950+0000 7f9c9affd640 1 -- 172.21.15.1:0/180054293 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c9c01a070 con 0x7f9ca4152840 2024-03-06T07:27:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.950+0000 7f9c9affd640 1 --2- 172.21.15.1:0/180054293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c8806b4d0 0x7f9c8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.950+0000 7f9c9affd640 1 -- 172.21.15.1:0/180054293 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9c9c08a6b0 con 0x7f9ca4152840 2024-03-06T07:27:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.951+0000 7f9caa44b640 1 --2- 172.21.15.1:0/180054293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c8806b4d0 0x7f9c8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.951+0000 7f9caa44b640 1 --2- 172.21.15.1:0/180054293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c8806b4d0 0x7f9c8806d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0036000 tx=0x7f9ca003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:07.954+0000 7f9c9affd640 1 -- 172.21.15.1:0/180054293 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ca41edd50 con 0x7f9ca4152840 2024-03-06T07:27:08.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.134+0000 7f9cac6af640 1 -- 172.21.15.1:0/180054293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ca40686e0 con 0x7f9ca4152840 2024-03-06T07:27:08.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.135+0000 7f9c9affd640 1 -- 172.21.15.1:0/180054293 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+16832 (secure 0 0 0) 0x7f9c9c058ec0 con 0x7f9ca4152840 2024-03-06T07:27:08.138 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:08.138 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":171,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:27:06.270883+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:02.888292+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:03.804423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:08.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.137+0000 7f9cac6af640 1 -- 172.21.15.1:0/180054293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c8806b4d0 msgr2=0x7f9c8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.137+0000 7f9cac6af640 1 --2- 172.21.15.1:0/180054293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c8806b4d0 0x7f9c8806d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0036000 tx=0x7f9ca003c040 comp rx=0 tx=0).stop 2024-03-06T07:27:08.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.137+0000 7f9cac6af640 1 -- 172.21.15.1:0/180054293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ca4152840 msgr2=0x7f9ca41ee500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.137+0000 7f9cac6af640 1 --2- 172.21.15.1:0/180054293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ca4152840 0x7f9ca41ee500 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c00cad0 tx=0x7f9c9c00cf90 comp rx=0 tx=0).stop 2024-03-06T07:27:08.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.137+0000 7f9cac6af640 1 -- 172.21.15.1:0/180054293 shutdown_connections 2024-03-06T07:27:08.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.137+0000 7f9cac6af640 1 --2- 172.21.15.1:0/180054293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c8806b4d0 0x7f9c8806d990 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.137+0000 7f9cac6af640 1 --2- 172.21.15.1:0/180054293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ca4152840 0x7f9ca41ee500 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.137+0000 7f9cac6af640 1 --2- 172.21.15.1:0/180054293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ca40c60b0 0x7f9ca41edfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.137+0000 7f9cac6af640 1 --2- 172.21.15.1:0/180054293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ca40c3970 0x7f9ca40a97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.137+0000 7f9cac6af640 1 -- 172.21.15.1:0/180054293 >> 172.21.15.1:0/180054293 conn(0x7f9ca40c88f0 msgr2=0x7f9ca4152f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.137+0000 7f9cac6af640 1 -- 172.21.15.1:0/180054293 shutdown_connections 2024-03-06T07:27:08.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.137+0000 7f9cac6af640 1 -- 172.21.15.1:0/180054293 wait complete. 2024-03-06T07:27:08.150 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:27:08.151 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:08.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.289+0000 7f68461ac640 1 Processor -- start 2024-03-06T07:27:08.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.289+0000 7f68461ac640 1 -- start start 2024-03-06T07:27:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.290+0000 7f68461ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f68400c32f0 0x7f68401543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.290+0000 7f68461ac640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6840070c80 0x7f68400710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.290+0000 7f68461ac640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6840150900 con 0x7f6840150520 2024-03-06T07:27:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.290+0000 7f68461ac640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6840153ed0 con 0x7f6840070c80 2024-03-06T07:27:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.290+0000 7f68461ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6840071790 con 0x7f68400c32f0 2024-03-06T07:27:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.290+0000 7f683f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6840150520 0x7f68401537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44398/0 (socket says 172.21.15.1:44398) 2024-03-06T07:27:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.290+0000 7f683f7fe640 1 -- 172.21.15.1:0/3652165215 learned_addr learned my addr 172.21.15.1:0/3652165215 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.290+0000 7f683ffff640 1 --2- 172.21.15.1:0/3652165215 >> v2:172.21.15.98:3300/0 conn(0x7f6840070c80 0x7f68400710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.290+0000 7f683effd640 1 --2- 172.21.15.1:0/3652165215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f68400c32f0 0x7f68401543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.290+0000 7f683e7fc640 1 -- 172.21.15.1:0/3652165215 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214322554 0 0) 0x7f6840150900 con 0x7f6840150520 2024-03-06T07:27:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.290+0000 7f683e7fc640 1 -- 172.21.15.1:0/3652165215 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6828003520 con 0x7f6840150520 2024-03-06T07:27:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.290+0000 7f683ffff640 1 -- 172.21.15.1:0/3652165215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f68400c32f0 msgr2=0x7f68401543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.290+0000 7f683ffff640 1 --2- 172.21.15.1:0/3652165215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f68400c32f0 0x7f68401543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.290+0000 7f683ffff640 1 -- 172.21.15.1:0/3652165215 >> v1:172.21.15.1:6789/0 conn(0x7f6840150520 legacy=0x7f68401537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.290+0000 7f683e7fc640 1 -- 172.21.15.1:0/3652165215 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 716530450 0 0) 0x7f6828003520 con 0x7f6840150520 2024-03-06T07:27:08.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.290+0000 7f683ffff640 1 -- 172.21.15.1:0/3652165215 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6840157050 con 0x7f6840070c80 2024-03-06T07:27:08.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.290+0000 7f683effd640 1 --2- 172.21.15.1:0/3652165215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f68400c32f0 0x7f68401543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:08.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.291+0000 7f683ffff640 1 --2- 172.21.15.1:0/3652165215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840070c80 0x7f68400710f0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6838009d90 tx=0x7f683802bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=732968997a80873e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.291+0000 7f683e7fc640 1 -- 172.21.15.1:0/3652165215 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6838039070 con 0x7f6840070c80 2024-03-06T07:27:08.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.291+0000 7f683e7fc640 1 -- 172.21.15.1:0/3652165215 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683802c8a0 con 0x7f6840070c80 2024-03-06T07:27:08.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.291+0000 7f683e7fc640 1 -- 172.21.15.1:0/3652165215 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830001670 con 0x7f6840070c80 2024-03-06T07:27:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.292+0000 7f68461ac640 1 -- 172.21.15.1:0/3652165215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840070c80 msgr2=0x7f68400710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.292+0000 7f68461ac640 1 --2- 172.21.15.1:0/3652165215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840070c80 0x7f68400710f0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6838009d90 tx=0x7f683802bd70 comp rx=0 tx=0).stop 2024-03-06T07:27:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.292+0000 7f68461ac640 1 -- 172.21.15.1:0/3652165215 shutdown_connections 2024-03-06T07:27:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.292+0000 7f68461ac640 1 --2- 172.21.15.1:0/3652165215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840070c80 0x7f68400710f0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.292+0000 7f68461ac640 1 --2- 172.21.15.1:0/3652165215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f68400c32f0 0x7f68401543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.292+0000 7f68461ac640 1 -- 172.21.15.1:0/3652165215 >> 172.21.15.1:0/3652165215 conn(0x7f68400c88f0 msgr2=0x7f68400c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.292+0000 7f68461ac640 1 -- 172.21.15.1:0/3652165215 shutdown_connections 2024-03-06T07:27:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.292+0000 7f68461ac640 1 -- 172.21.15.1:0/3652165215 wait complete. 2024-03-06T07:27:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.293+0000 7f68461ac640 1 Processor -- start 2024-03-06T07:27:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.293+0000 7f68461ac640 1 -- start start 2024-03-06T07:27:08.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.294+0000 7f68461ac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840070c80 0x7f6840154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.294+0000 7f68461ac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68400c32f0 0x7f68400c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.294+0000 7f68461ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6840150520 0x7f6840156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.294+0000 7f68461ac640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6840157330 con 0x7f68400c32f0 2024-03-06T07:27:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.294+0000 7f68461ac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68401532a0 con 0x7f6840070c80 2024-03-06T07:27:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.294+0000 7f68461ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68400c7ec0 con 0x7f6840150520 2024-03-06T07:27:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.294+0000 7f683effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68400c32f0 0x7f68400c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.294+0000 7f683effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68400c32f0 0x7f68400c1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59472/0 (socket says 172.21.15.1:59472) 2024-03-06T07:27:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.294+0000 7f683effd640 1 -- 172.21.15.1:0/1503948391 learned_addr learned my addr 172.21.15.1:0/1503948391 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.294+0000 7f683ffff640 1 --2- 172.21.15.1:0/1503948391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6840150520 0x7f6840156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.294+0000 7f683f7fe640 1 --2- 172.21.15.1:0/1503948391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840070c80 0x7f6840154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.294+0000 7f683effd640 1 -- 172.21.15.1:0/1503948391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6840150520 msgr2=0x7f6840156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.294+0000 7f683effd640 1 --2- 172.21.15.1:0/1503948391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6840150520 0x7f6840156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.294+0000 7f683effd640 1 -- 172.21.15.1:0/1503948391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840070c80 msgr2=0x7f6840154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.294+0000 7f683effd640 1 --2- 172.21.15.1:0/1503948391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840070c80 0x7f6840154d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.294+0000 7f683effd640 1 -- 172.21.15.1:0/1503948391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684006ddf0 con 0x7f68400c32f0 2024-03-06T07:27:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.294+0000 7f683ffff640 1 --2- 172.21.15.1:0/1503948391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6840150520 0x7f6840156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:08.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.295+0000 7f683effd640 1 --2- 172.21.15.1:0/1503948391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68400c32f0 0x7f68400c1f90 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f683400ebe0 tx=0x7f683400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.295+0000 7f6827fff640 1 -- 172.21.15.1:0/1503948391 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6834017070 con 0x7f68400c32f0 2024-03-06T07:27:08.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.295+0000 7f6827fff640 1 -- 172.21.15.1:0/1503948391 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683400cc30 con 0x7f68400c32f0 2024-03-06T07:27:08.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.295+0000 7f6827fff640 1 -- 172.21.15.1:0/1503948391 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6834013470 con 0x7f68400c32f0 2024-03-06T07:27:08.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.295+0000 7f68461ac640 1 -- 172.21.15.1:0/1503948391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68401eae10 con 0x7f68400c32f0 2024-03-06T07:27:08.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.295+0000 7f68461ac640 1 -- 172.21.15.1:0/1503948391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68400c24d0 con 0x7f68400c32f0 2024-03-06T07:27:08.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.296+0000 7f68461ac640 1 -- 172.21.15.1:0/1503948391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6840154990 con 0x7f68400c32f0 2024-03-06T07:27:08.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.296+0000 7f6827fff640 1 -- 172.21.15.1:0/1503948391 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f683401a070 con 0x7f68400c32f0 2024-03-06T07:27:08.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.297+0000 7f6827fff640 1 --2- 172.21.15.1:0/1503948391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f682c06b500 0x7f682c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.297+0000 7f6827fff640 1 -- 172.21.15.1:0/1503948391 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f683408a2d0 con 0x7f68400c32f0 2024-03-06T07:27:08.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.297+0000 7f683f7fe640 1 --2- 172.21.15.1:0/1503948391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f682c06b500 0x7f682c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.297+0000 7f683f7fe640 1 --2- 172.21.15.1:0/1503948391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f682c06b500 0x7f682c06d9c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f6830007940 tx=0x7f6830009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.305+0000 7f6827fff640 1 -- 172.21.15.1:0/1503948391 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6834058fc0 con 0x7f68400c32f0 2024-03-06T07:27:08.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.482+0000 7f68461ac640 1 -- 172.21.15.1:0/1503948391 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f68400686e0 con 0x7f682c06b500 2024-03-06T07:27:08.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.488+0000 7f6827fff640 1 -- 172.21.15.1:0/1503948391 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247159 (secure 0 0 0) 0x7f68400686e0 con 0x7f682c06b500 2024-03-06T07:27:08.493 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:08.494 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.491+0000 7f68461ac640 1 -- 172.21.15.1:0/1503948391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f682c06b500 msgr2=0x7f682c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.491+0000 7f68461ac640 1 --2- 172.21.15.1:0/1503948391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f682c06b500 0x7f682c06d9c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f6830007940 tx=0x7f6830009040 comp rx=0 tx=0).stop 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.491+0000 7f68461ac640 1 -- 172.21.15.1:0/1503948391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68400c32f0 msgr2=0x7f68400c1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.491+0000 7f68461ac640 1 --2- 172.21.15.1:0/1503948391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68400c32f0 0x7f68400c1f90 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f683400ebe0 tx=0x7f683400c520 comp rx=0 tx=0).stop 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.491+0000 7f68461ac640 1 -- 172.21.15.1:0/1503948391 shutdown_connections 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.491+0000 7f68461ac640 1 --2- 172.21.15.1:0/1503948391 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f682c06b500 0x7f682c06d9c0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.491+0000 7f68461ac640 1 --2- 172.21.15.1:0/1503948391 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6840150520 0x7f6840156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.491+0000 7f68461ac640 1 --2- 172.21.15.1:0/1503948391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68400c32f0 0x7f68400c1f90 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.491+0000 7f68461ac640 1 --2- 172.21.15.1:0/1503948391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840070c80 0x7f6840154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.491+0000 7f68461ac640 1 -- 172.21.15.1:0/1503948391 >> 172.21.15.1:0/1503948391 conn(0x7f68400c88f0 msgr2=0x7f6840071060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.491+0000 7f68461ac640 1 -- 172.21.15.1:0/1503948391 shutdown_connections 2024-03-06T07:27:08.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.491+0000 7f68461ac640 1 -- 172.21.15.1:0/1503948391 wait complete. 2024-03-06T07:27:08.505 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-03-06T07:27:06.784467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":266240,"kb_used_data":16784,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754708480,"statfs":{"total":773094113280,"available":772821483520,"internally_reserved":0,"allocated":17186816,"data_stored":9494844,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001086"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254547+0000","last_change":"2024-03-06T07:27:04.254547+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254547+0000","last_undegraded":"2024-03-06T07:27:04.254547+0000","last_fullsized":"2024-03-06T07:27:04.254547+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.255495+0000","last_change":"2024-03-06T07:27:04.255495+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.255495+0000","last_undegraded":"2024-03-06T07:27:04.255495+0000","last_fullsized":"2024-03-06T07:27:04.255495+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259837+0000","last_change":"2024-03-06T07:27:04.259837+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259837+0000","last_undegraded":"2024-03-06T07:27:04.259837+0000","last_fullsized":"2024-03-06T07:27:04.259837+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259747+0000","last_change":"2024-03-06T07:27:04.259747+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259747+0000","last_undegraded":"2024-03-06T07:27:04.259747+0000","last_fullsized":"2024-03-06T07:27:04.259747+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258205+0000","last_change":"2024-03-06T07:27:04.258205+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258205+0000","last_undegraded":"2024-03-06T07:27:04.258205+0000","last_fullsized":"2024-03-06T07:27:04.258205+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259513+0000","last_change":"2024-03-06T07:27:04.259513+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259513+0000","last_undegraded":"2024-03-06T07:27:04.259513+0000","last_fullsized":"2024-03-06T07:27:04.259513+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257656+0000","last_change":"2024-03-06T07:27:04.257656+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257656+0000","last_undegraded":"2024-03-06T07:27:04.257656+0000","last_fullsized":"2024-03-06T07:27:04.257656+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257414+0000","last_change":"2024-03-06T07:27:04.257414+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257414+0000","last_undegraded":"2024-03-06T07:27:04.257414+0000","last_fullsized":"2024-03-06T07:27:04.257414+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258740+0000","last_change":"2024-03-06T07:27:04.258740+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258740+0000","last_undegraded":"2024-03-06T07:27:04.258740+0000","last_fullsized":"2024-03-06T07:27:04.258740+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258528+0000","last_change":"2024-03-06T07:27:04.258528+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258528+0000","last_undegraded":"2024-03-06T07:27:04.258528+0000","last_fullsized":"2024-03-06T07:27:04.258528+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259375+0000","last_change":"2024-03-06T07:27:04.259375+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259375+0000","last_undegraded":"2024-03-06T07:27:04.259375+0000","last_fullsized":"2024-03-06T07:27:04.259375+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257246+0000","last_change":"2024-03-06T07:27:04.257246+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257246+0000","last_undegraded":"2024-03-06T07:27:04.257246+0000","last_fullsized":"2024-03-06T07:27:04.257246+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259445+0000","last_change":"2024-03-06T07:27:04.259445+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259445+0000","last_undegraded":"2024-03-06T07:27:04.259445+0000","last_fullsized":"2024-03-06T07:27:04.259445+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257927+0000","last_change":"2024-03-06T07:27:04.257927+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257927+0000","last_undegraded":"2024-03-06T07:27:04.257927+0000","last_fullsized":"2024-03-06T07:27:04.257927+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.263169+0000","last_change":"2024-03-06T07:27:04.263169+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.263169+0000","last_undegraded":"2024-03-06T07:27:04.263169+0000","last_fullsized":"2024-03-06T07:27:04.263169+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257264+0000","last_change":"2024-03-06T07:27:04.257264+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257264+0000","last_undegraded":"2024-03-06T07:27:04.257264+0000","last_fullsized":"2024-03-06T07:27:04.257264+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258290+0000","last_change":"2024-03-06T07:27:04.258290+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258290+0000","last_undegraded":"2024-03-06T07:27:04.258290+0000","last_fullsized":"2024-03-06T07:27:04.258290+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258702+0000","last_change":"2024-03-06T07:27:04.258702+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258702+0000","last_undegraded":"2024-03-06T07:27:04.258702+0000","last_fullsized":"2024-03-06T07:27:04.258702+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258065+0000","last_change":"2024-03-06T07:27:04.258065+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258065+0000","last_undegraded":"2024-03-06T07:27:04.258065+0000","last_fullsized":"2024-03-06T07:27:04.258065+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258945+0000","last_change":"2024-03-06T07:27:04.258945+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258945+0000","last_undegraded":"2024-03-06T07:27:04.258945+0000","last_fullsized":"2024-03-06T07:27:04.258945+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257755+0000","last_change":"2024-03-06T07:27:04.257755+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257755+0000","last_undegraded":"2024-03-06T07:27:04.257755+0000","last_fullsized":"2024-03-06T07:27:04.257755+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257343+0000","last_change":"2024-03-06T07:27:04.257343+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257343+0000","last_undegraded":"2024-03-06T07:27:04.257343+0000","last_fullsized":"2024-03-06T07:27:04.257343+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.253999+0000","last_change":"2024-03-06T07:27:04.253999+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.253999+0000","last_undegraded":"2024-03-06T07:27:04.253999+0000","last_fullsized":"2024-03-06T07:27:04.253999+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256789+0000","last_change":"2024-03-06T07:27:04.256789+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256789+0000","last_undegraded":"2024-03-06T07:27:04.256789+0000","last_fullsized":"2024-03-06T07:27:04.256789+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257184+0000","last_change":"2024-03-06T07:27:04.257184+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257184+0000","last_undegraded":"2024-03-06T07:27:04.257184+0000","last_fullsized":"2024-03-06T07:27:04.257184+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257217+0000","last_change":"2024-03-06T07:27:04.257217+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257217+0000","last_undegraded":"2024-03-06T07:27:04.257217+0000","last_fullsized":"2024-03-06T07:27:04.257217+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259006+0000","last_change":"2024-03-06T07:27:04.259006+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259006+0000","last_undegraded":"2024-03-06T07:27:04.259006+0000","last_fullsized":"2024-03-06T07:27:04.259006+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.255903+0000","last_change":"2024-03-06T07:27:04.255903+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.255903+0000","last_undegraded":"2024-03-06T07:27:04.255903+0000","last_fullsized":"2024-03-06T07:27:04.255903+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254435+0000","last_change":"2024-03-06T07:27:04.254435+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254435+0000","last_undegraded":"2024-03-06T07:27:04.254435+0000","last_fullsized":"2024-03-06T07:27:04.254435+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.264763+0000","last_change":"2024-03-06T07:27:04.264763+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.264763+0000","last_undegraded":"2024-03-06T07:27:04.264763+0000","last_fullsized":"2024-03-06T07:27:04.264763+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261771+0000","last_change":"2024-03-06T07:27:04.261771+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261771+0000","last_undegraded":"2024-03-06T07:27:04.261771+0000","last_fullsized":"2024-03-06T07:27:04.261771+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.260482+0000","last_change":"2024-03-06T07:27:04.260482+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.260482+0000","last_undegraded":"2024-03-06T07:27:04.260482+0000","last_fullsized":"2024-03-06T07:27:04.260482+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254981+0000","last_change":"2024-03-06T07:27:04.254981+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254981+0000","last_undegraded":"2024-03-06T07:27:04.254981+0000","last_fullsized":"2024-03-06T07:27:04.254981+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.278552+0000","last_change":"2024-03-06T07:27:04.271712+0000","last_active":"2024-03-06T07:27:04.278552+0000","last_peered":"2024-03-06T07:27:04.278552+0000","last_clean":"2024-03-06T07:27:04.278552+0000","last_became_active":"2024-03-06T07:27:04.271429+0000","last_became_peered":"2024-03-06T07:27:04.271429+0000","last_unstale":"2024-03-06T07:27:04.278552+0000","last_undegraded":"2024-03-06T07:27:04.278552+0000","last_fullsized":"2024-03-06T07:27:04.278552+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:20.726754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257054+0000","last_change":"2024-03-06T07:27:04.257054+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257054+0000","last_undegraded":"2024-03-06T07:27:04.257054+0000","last_fullsized":"2024-03-06T07:27:04.257054+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.268329+0000","last_change":"2024-03-06T07:27:04.266276+0000","last_active":"2024-03-06T07:27:04.268329+0000","last_peered":"2024-03-06T07:27:04.268329+0000","last_clean":"2024-03-06T07:27:04.268329+0000","last_became_active":"2024-03-06T07:27:04.265418+0000","last_became_peered":"2024-03-06T07:27:04.265418+0000","last_unstale":"2024-03-06T07:27:04.268329+0000","last_undegraded":"2024-03-06T07:27:04.268329+0000","last_fullsized":"2024-03-06T07:27:04.268329+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:24.992967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256890+0000","last_change":"2024-03-06T07:27:04.256890+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256890+0000","last_undegraded":"2024-03-06T07:27:04.256890+0000","last_fullsized":"2024-03-06T07:27:04.256890+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267032+0000","last_change":"2024-03-06T07:27:04.262189+0000","last_active":"2024-03-06T07:27:04.267032+0000","last_peered":"2024-03-06T07:27:04.267032+0000","last_clean":"2024-03-06T07:27:04.267032+0000","last_became_active":"2024-03-06T07:27:04.261909+0000","last_became_peered":"2024-03-06T07:27:04.261909+0000","last_unstale":"2024-03-06T07:27:04.267032+0000","last_undegraded":"2024-03-06T07:27:04.267032+0000","last_fullsized":"2024-03-06T07:27:04.267032+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:15:43.228599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256537+0000","last_change":"2024-03-06T07:27:04.256537+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256537+0000","last_undegraded":"2024-03-06T07:27:04.256537+0000","last_fullsized":"2024-03-06T07:27:04.256537+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271246+0000","last_change":"2024-03-06T07:27:04.270472+0000","last_active":"2024-03-06T07:27:04.271246+0000","last_peered":"2024-03-06T07:27:04.271246+0000","last_clean":"2024-03-06T07:27:04.271246+0000","last_became_active":"2024-03-06T07:27:04.270185+0000","last_became_peered":"2024-03-06T07:27:04.270185+0000","last_unstale":"2024-03-06T07:27:04.271246+0000","last_undegraded":"2024-03-06T07:27:04.271246+0000","last_fullsized":"2024-03-06T07:27:04.271246+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:37.715400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.263186+0000","last_change":"2024-03-06T07:27:04.263186+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.263186+0000","last_undegraded":"2024-03-06T07:27:04.263186+0000","last_fullsized":"2024-03-06T07:27:04.263186+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.275596+0000","last_change":"2024-03-06T07:27:04.272491+0000","last_active":"2024-03-06T07:27:04.275596+0000","last_peered":"2024-03-06T07:27:04.275596+0000","last_clean":"2024-03-06T07:27:04.275596+0000","last_became_active":"2024-03-06T07:27:04.272190+0000","last_became_peered":"2024-03-06T07:27:04.272190+0000","last_unstale":"2024-03-06T07:27:04.275596+0000","last_undegraded":"2024-03-06T07:27:04.275596+0000","last_fullsized":"2024-03-06T07:27:04.275596+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:17.476526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.263035+0000","last_change":"2024-03-06T07:27:04.263035+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.263035+0000","last_undegraded":"2024-03-06T07:27:04.263035+0000","last_fullsized":"2024-03-06T07:27:04.263035+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.278476+0000","last_change":"2024-03-06T07:27:04.273573+0000","last_active":"2024-03-06T07:27:04.278476+0000","last_peered":"2024-03-06T07:27:04.278476+0000","last_clean":"2024-03-06T07:27:04.278476+0000","last_became_active":"2024-03-06T07:27:04.273253+0000","last_became_peered":"2024-03-06T07:27:04.273253+0000","last_unstale":"2024-03-06T07:27:04.278476+0000","last_undegraded":"2024-03-06T07:27:04.278476+0000","last_fullsized":"2024-03-06T07:27:04.278476+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:56.531270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254166+0000","last_change":"2024-03-06T07:27:04.254166+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254166+0000","last_undegraded":"2024-03-06T07:27:04.254166+0000","last_fullsized":"2024-03-06T07:27:04.254166+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.272537+0000","last_change":"2024-03-06T07:27:04.271894+0000","last_active":"2024-03-06T07:27:04.272537+0000","last_peered":"2024-03-06T07:27:04.272537+0000","last_clean":"2024-03-06T07:27:04.272537+0000","last_became_active":"2024-03-06T07:27:04.271305+0000","last_became_peered":"2024-03-06T07:27:04.271305+0000","last_unstale":"2024-03-06T07:27:04.272537+0000","last_undegraded":"2024-03-06T07:27:04.272537+0000","last_fullsized":"2024-03-06T07:27:04.272537+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:26.898488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.266335+0000","last_change":"2024-03-06T07:27:04.266335+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.266335+0000","last_undegraded":"2024-03-06T07:27:04.266335+0000","last_fullsized":"2024-03-06T07:27:04.266335+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271368+0000","last_change":"2024-03-06T07:27:04.269100+0000","last_active":"2024-03-06T07:27:04.271368+0000","last_peered":"2024-03-06T07:27:04.271368+0000","last_clean":"2024-03-06T07:27:04.271368+0000","last_became_active":"2024-03-06T07:27:04.268811+0000","last_became_peered":"2024-03-06T07:27:04.268811+0000","last_unstale":"2024-03-06T07:27:04.271368+0000","last_undegraded":"2024-03-06T07:27:04.271368+0000","last_fullsized":"2024-03-06T07:27:04.271368+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:34.615045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258161+0000","last_change":"2024-03-06T07:27:04.258161+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258161+0000","last_undegraded":"2024-03-06T07:27:04.258161+0000","last_fullsized":"2024-03-06T07:27:04.258161+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.275534+0000","last_change":"2024-03-06T07:27:04.271341+0000","last_active":"2024-03-06T07:27:04.275534+0000","last_peered":"2024-03-06T07:27:04.275534+0000","last_clean":"2024-03-06T07:27:04.275534+0000","last_became_active":"2024-03-06T07:27:04.271023+0000","last_became_peered":"2024-03-06T07:27:04.271023+0000","last_unstale":"2024-03-06T07:27:04.275534+0000","last_undegraded":"2024-03-06T07:27:04.275534+0000","last_fullsized":"2024-03-06T07:27:04.275534+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:24.173726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257145+0000","last_change":"2024-03-06T07:27:04.257145+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257145+0000","last_undegraded":"2024-03-06T07:27:04.257145+0000","last_fullsized":"2024-03-06T07:27:04.257145+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.266679+0000","last_change":"2024-03-06T07:27:04.264830+0000","last_active":"2024-03-06T07:27:04.266679+0000","last_peered":"2024-03-06T07:27:04.266679+0000","last_clean":"2024-03-06T07:27:04.266679+0000","last_became_active":"2024-03-06T07:27:04.264562+0000","last_became_peered":"2024-03-06T07:27:04.264562+0000","last_unstale":"2024-03-06T07:27:04.266679+0000","last_undegraded":"2024-03-06T07:27:04.266679+0000","last_fullsized":"2024-03-06T07:27:04.266679+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:12.146965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256102+0000","last_change":"2024-03-06T07:27:04.256102+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256102+0000","last_undegraded":"2024-03-06T07:27:04.256102+0000","last_fullsized":"2024-03-06T07:27:04.256102+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.278496+0000","last_change":"2024-03-06T07:27:04.273664+0000","last_active":"2024-03-06T07:27:04.278496+0000","last_peered":"2024-03-06T07:27:04.278496+0000","last_clean":"2024-03-06T07:27:04.278496+0000","last_became_active":"2024-03-06T07:27:04.273372+0000","last_became_peered":"2024-03-06T07:27:04.273372+0000","last_unstale":"2024-03-06T07:27:04.278496+0000","last_undegraded":"2024-03-06T07:27:04.278496+0000","last_fullsized":"2024-03-06T07:27:04.278496+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:51.519017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.255456+0000","last_change":"2024-03-06T07:27:04.255456+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.255456+0000","last_undegraded":"2024-03-06T07:27:04.255456+0000","last_fullsized":"2024-03-06T07:27:04.255456+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267158+0000","last_change":"2024-03-06T07:27:04.263075+0000","last_active":"2024-03-06T07:27:04.267158+0000","last_peered":"2024-03-06T07:27:04.267158+0000","last_clean":"2024-03-06T07:27:04.267158+0000","last_became_active":"2024-03-06T07:27:04.261822+0000","last_became_peered":"2024-03-06T07:27:04.261822+0000","last_unstale":"2024-03-06T07:27:04.267158+0000","last_undegraded":"2024-03-06T07:27:04.267158+0000","last_fullsized":"2024-03-06T07:27:04.267158+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:50.416453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258591+0000","last_change":"2024-03-06T07:27:04.258591+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258591+0000","last_undegraded":"2024-03-06T07:27:04.258591+0000","last_fullsized":"2024-03-06T07:27:04.258591+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.270193+0000","last_change":"2024-03-06T07:27:04.268027+0000","last_active":"2024-03-06T07:27:04.270193+0000","last_peered":"2024-03-06T07:27:04.270193+0000","last_clean":"2024-03-06T07:27:04.270193+0000","last_became_active":"2024-03-06T07:27:04.267726+0000","last_became_peered":"2024-03-06T07:27:04.267726+0000","last_unstale":"2024-03-06T07:27:04.270193+0000","last_undegraded":"2024-03-06T07:27:04.270193+0000","last_fullsized":"2024-03-06T07:27:04.270193+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:11.943331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254886+0000","last_change":"2024-03-06T07:27:04.254886+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254886+0000","last_undegraded":"2024-03-06T07:27:04.254886+0000","last_fullsized":"2024-03-06T07:27:04.254886+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.266762+0000","last_change":"2024-03-06T07:27:04.265676+0000","last_active":"2024-03-06T07:27:04.266762+0000","last_peered":"2024-03-06T07:27:04.266762+0000","last_clean":"2024-03-06T07:27:04.266762+0000","last_became_active":"2024-03-06T07:27:04.265374+0000","last_became_peered":"2024-03-06T07:27:04.265374+0000","last_unstale":"2024-03-06T07:27:04.266762+0000","last_undegraded":"2024-03-06T07:27:04.266762+0000","last_fullsized":"2024-03-06T07:27:04.266762+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:30.199858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261058+0000","last_change":"2024-03-06T07:27:04.261058+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261058+0000","last_undegraded":"2024-03-06T07:27:04.261058+0000","last_fullsized":"2024-03-06T07:27:04.261058+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271352+0000","last_change":"2024-03-06T07:27:04.269119+0000","last_active":"2024-03-06T07:27:04.271352+0000","last_peered":"2024-03-06T07:27:04.271352+0000","last_clean":"2024-03-06T07:27:04.271352+0000","last_became_active":"2024-03-06T07:27:04.268844+0000","last_became_peered":"2024-03-06T07:27:04.268844+0000","last_unstale":"2024-03-06T07:27:04.271352+0000","last_undegraded":"2024-03-06T07:27:04.271352+0000","last_fullsized":"2024-03-06T07:27:04.271352+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:47.701531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256965+0000","last_change":"2024-03-06T07:27:04.256965+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256965+0000","last_undegraded":"2024-03-06T07:27:04.256965+0000","last_fullsized":"2024-03-06T07:27:04.256965+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.268206+0000","last_change":"2024-03-06T07:27:04.262327+0000","last_active":"2024-03-06T07:27:04.268206+0000","last_peered":"2024-03-06T07:27:04.268206+0000","last_clean":"2024-03-06T07:27:04.268206+0000","last_became_active":"2024-03-06T07:27:04.262046+0000","last_became_peered":"2024-03-06T07:27:04.262046+0000","last_unstale":"2024-03-06T07:27:04.268206+0000","last_undegraded":"2024-03-06T07:27:04.268206+0000","last_fullsized":"2024-03-06T07:27:04.268206+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:31.006078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256647+0000","last_change":"2024-03-06T07:27:04.256647+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256647+0000","last_undegraded":"2024-03-06T07:27:04.256647+0000","last_fullsized":"2024-03-06T07:27:04.256647+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.268350+0000","last_change":"2024-03-06T07:27:04.266893+0000","last_active":"2024-03-06T07:27:04.268350+0000","last_peered":"2024-03-06T07:27:04.268350+0000","last_clean":"2024-03-06T07:27:04.268350+0000","last_became_active":"2024-03-06T07:27:04.265563+0000","last_became_peered":"2024-03-06T07:27:04.265563+0000","last_unstale":"2024-03-06T07:27:04.268350+0000","last_undegraded":"2024-03-06T07:27:04.268350+0000","last_fullsized":"2024-03-06T07:27:04.268350+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:35.022673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254208+0000","last_change":"2024-03-06T07:27:04.254208+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254208+0000","last_undegraded":"2024-03-06T07:27:04.254208+0000","last_fullsized":"2024-03-06T07:27:04.254208+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267134+0000","last_change":"2024-03-06T07:27:04.265277+0000","last_active":"2024-03-06T07:27:04.267134+0000","last_peered":"2024-03-06T07:27:04.267134+0000","last_clean":"2024-03-06T07:27:04.267134+0000","last_became_active":"2024-03-06T07:27:04.265018+0000","last_became_peered":"2024-03-06T07:27:04.265018+0000","last_unstale":"2024-03-06T07:27:04.267134+0000","last_undegraded":"2024-03-06T07:27:04.267134+0000","last_fullsized":"2024-03-06T07:27:04.267134+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:51.929743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257320+0000","last_change":"2024-03-06T07:27:04.257320+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257320+0000","last_undegraded":"2024-03-06T07:27:04.257320+0000","last_fullsized":"2024-03-06T07:27:04.257320+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267517+0000","last_change":"2024-03-06T07:27:04.265842+0000","last_active":"2024-03-06T07:27:04.267517+0000","last_peered":"2024-03-06T07:27:04.267517+0000","last_clean":"2024-03-06T07:27:04.267517+0000","last_became_active":"2024-03-06T07:27:04.265568+0000","last_became_peered":"2024-03-06T07:27:04.265568+0000","last_unstale":"2024-03-06T07:27:04.267517+0000","last_undegraded":"2024-03-06T07:27:04.267517+0000","last_fullsized":"2024-03-06T07:27:04.267517+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:31.100594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254138+0000","last_change":"2024-03-06T07:27:04.254138+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254138+0000","last_undegraded":"2024-03-06T07:27:04.254138+0000","last_fullsized":"2024-03-06T07:27:04.254138+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.264364+0000","last_change":"2024-03-06T07:27:04.262729+0000","last_active":"2024-03-06T07:27:04.264364+0000","last_peered":"2024-03-06T07:27:04.264364+0000","last_clean":"2024-03-06T07:27:04.264364+0000","last_became_active":"2024-03-06T07:27:04.262484+0000","last_became_peered":"2024-03-06T07:27:04.262484+0000","last_unstale":"2024-03-06T07:27:04.264364+0000","last_undegraded":"2024-03-06T07:27:04.264364+0000","last_fullsized":"2024-03-06T07:27:04.264364+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:38.945976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261717+0000","last_change":"2024-03-06T07:27:04.261717+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261717+0000","last_undegraded":"2024-03-06T07:27:04.261717+0000","last_fullsized":"2024-03-06T07:27:04.261717+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267085+0000","last_change":"2024-03-06T07:27:04.263995+0000","last_active":"2024-03-06T07:27:04.267085+0000","last_peered":"2024-03-06T07:27:04.267085+0000","last_clean":"2024-03-06T07:27:04.267085+0000","last_became_active":"2024-03-06T07:27:04.263629+0000","last_became_peered":"2024-03-06T07:27:04.263629+0000","last_unstale":"2024-03-06T07:27:04.267085+0000","last_undegraded":"2024-03-06T07:27:04.267085+0000","last_fullsized":"2024-03-06T07:27:04.267085+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:44.864909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254160+0000","last_change":"2024-03-06T07:27:04.254160+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254160+0000","last_undegraded":"2024-03-06T07:27:04.254160+0000","last_fullsized":"2024-03-06T07:27:04.254160+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.278500+0000","last_change":"2024-03-06T07:27:04.271282+0000","last_active":"2024-03-06T07:27:04.278500+0000","last_peered":"2024-03-06T07:27:04.278500+0000","last_clean":"2024-03-06T07:27:04.278500+0000","last_became_active":"2024-03-06T07:27:04.270987+0000","last_became_peered":"2024-03-06T07:27:04.270987+0000","last_unstale":"2024-03-06T07:27:04.278500+0000","last_undegraded":"2024-03-06T07:27:04.278500+0000","last_fullsized":"2024-03-06T07:27:04.278500+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:02.098293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258625+0000","last_change":"2024-03-06T07:27:04.258625+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258625+0000","last_undegraded":"2024-03-06T07:27:04.258625+0000","last_fullsized":"2024-03-06T07:27:04.258625+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.275695+0000","last_change":"2024-03-06T07:27:04.272151+0000","last_active":"2024-03-06T07:27:04.275695+0000","last_peered":"2024-03-06T07:27:04.275695+0000","last_clean":"2024-03-06T07:27:04.275695+0000","last_became_active":"2024-03-06T07:27:04.271894+0000","last_became_peered":"2024-03-06T07:27:04.271894+0000","last_unstale":"2024-03-06T07:27:04.275695+0000","last_undegraded":"2024-03-06T07:27:04.275695+0000","last_fullsized":"2024-03-06T07:27:04.275695+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:50.049060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256530+0000","last_change":"2024-03-06T07:27:04.256530+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256530+0000","last_undegraded":"2024-03-06T07:27:04.256530+0000","last_fullsized":"2024-03-06T07:27:04.256530+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.270258+0000","last_change":"2024-03-06T07:27:04.268865+0000","last_active":"2024-03-06T07:27:04.270258+0000","last_peered":"2024-03-06T07:27:04.270258+0000","last_clean":"2024-03-06T07:27:04.270258+0000","last_became_active":"2024-03-06T07:27:04.268566+0000","last_became_peered":"2024-03-06T07:27:04.268566+0000","last_unstale":"2024-03-06T07:27:04.270258+0000","last_undegraded":"2024-03-06T07:27:04.270258+0000","last_fullsized":"2024-03-06T07:27:04.270258+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:05.786807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257744+0000","last_change":"2024-03-06T07:27:04.257744+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257744+0000","last_undegraded":"2024-03-06T07:27:04.257744+0000","last_fullsized":"2024-03-06T07:27:04.257744+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.270471+0000","last_change":"2024-03-06T07:27:04.264784+0000","last_active":"2024-03-06T07:27:04.270471+0000","last_peered":"2024-03-06T07:27:04.270471+0000","last_clean":"2024-03-06T07:27:04.270471+0000","last_became_active":"2024-03-06T07:27:04.264499+0000","last_became_peered":"2024-03-06T07:27:04.264499+0000","last_unstale":"2024-03-06T07:27:04.270471+0000","last_undegraded":"2024-03-06T07:27:04.270471+0000","last_fullsized":"2024-03-06T07:27:04.270471+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:38.542889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254229+0000","last_change":"2024-03-06T07:27:04.254229+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254229+0000","last_undegraded":"2024-03-06T07:27:04.254229+0000","last_fullsized":"2024-03-06T07:27:04.254229+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.275452+0000","last_change":"2024-03-06T07:27:04.269541+0000","last_active":"2024-03-06T07:27:04.275452+0000","last_peered":"2024-03-06T07:27:04.275452+0000","last_clean":"2024-03-06T07:27:04.275452+0000","last_became_active":"2024-03-06T07:27:04.269228+0000","last_became_peered":"2024-03-06T07:27:04.269228+0000","last_unstale":"2024-03-06T07:27:04.275452+0000","last_undegraded":"2024-03-06T07:27:04.275452+0000","last_fullsized":"2024-03-06T07:27:04.275452+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:05.198388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254889+0000","last_change":"2024-03-06T07:27:04.254889+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254889+0000","last_undegraded":"2024-03-06T07:27:04.254889+0000","last_fullsized":"2024-03-06T07:27:04.254889+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.264329+0000","last_change":"2024-03-06T07:27:04.260635+0000","last_active":"2024-03-06T07:27:04.264329+0000","last_peered":"2024-03-06T07:27:04.264329+0000","last_clean":"2024-03-06T07:27:04.264329+0000","last_became_active":"2024-03-06T07:27:04.260370+0000","last_became_peered":"2024-03-06T07:27:04.260370+0000","last_unstale":"2024-03-06T07:27:04.264329+0000","last_undegraded":"2024-03-06T07:27:04.264329+0000","last_fullsized":"2024-03-06T07:27:04.264329+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:26.449033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257795+0000","last_change":"2024-03-06T07:27:04.257795+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257795+0000","last_undegraded":"2024-03-06T07:27:04.257795+0000","last_fullsized":"2024-03-06T07:27:04.257795+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.272959+0000","last_change":"2024-03-06T07:27:04.269073+0000","last_active":"2024-03-06T07:27:04.272959+0000","last_peered":"2024-03-06T07:27:04.272959+0000","last_clean":"2024-03-06T07:27:04.272959+0000","last_became_active":"2024-03-06T07:27:04.268831+0000","last_became_peered":"2024-03-06T07:27:04.268831+0000","last_unstale":"2024-03-06T07:27:04.272959+0000","last_undegraded":"2024-03-06T07:27:04.272959+0000","last_fullsized":"2024-03-06T07:27:04.272959+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:30.526595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261310+0000","last_change":"2024-03-06T07:27:04.261310+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261310+0000","last_undegraded":"2024-03-06T07:27:04.261310+0000","last_fullsized":"2024-03-06T07:27:04.261310+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271907+0000","last_change":"2024-03-06T07:27:04.271202+0000","last_active":"2024-03-06T07:27:04.271907+0000","last_peered":"2024-03-06T07:27:04.271907+0000","last_clean":"2024-03-06T07:27:04.271907+0000","last_became_active":"2024-03-06T07:27:04.270881+0000","last_became_peered":"2024-03-06T07:27:04.270881+0000","last_unstale":"2024-03-06T07:27:04.271907+0000","last_undegraded":"2024-03-06T07:27:04.271907+0000","last_fullsized":"2024-03-06T07:27:04.271907+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:17.006730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261395+0000","last_change":"2024-03-06T07:27:04.261395+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261395+0000","last_undegraded":"2024-03-06T07:27:04.261395+0000","last_fullsized":"2024-03-06T07:27:04.261395+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.273065+0000","last_change":"2024-03-06T07:27:04.269025+0000","last_active":"2024-03-06T07:27:04.273065+0000","last_peered":"2024-03-06T07:27:04.273065+0000","last_clean":"2024-03-06T07:27:04.273065+0000","last_became_active":"2024-03-06T07:27:04.268768+0000","last_became_peered":"2024-03-06T07:27:04.268768+0000","last_unstale":"2024-03-06T07:27:04.273065+0000","last_undegraded":"2024-03-06T07:27:04.273065+0000","last_fullsized":"2024-03-06T07:27:04.273065+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:28.810603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.260194+0000","last_change":"2024-03-06T07:27:04.260194+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.260194+0000","last_undegraded":"2024-03-06T07:27:04.260194+0000","last_fullsized":"2024-03-06T07:27:04.260194+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271722+0000","last_change":"2024-03-06T07:27:04.269800+0000","last_active":"2024-03-06T07:27:04.271722+0000","last_peered":"2024-03-06T07:27:04.271722+0000","last_clean":"2024-03-06T07:27:04.271722+0000","last_became_active":"2024-03-06T07:27:04.269489+0000","last_became_peered":"2024-03-06T07:27:04.269489+0000","last_unstale":"2024-03-06T07:27:04.271722+0000","last_undegraded":"2024-03-06T07:27:04.271722+0000","last_fullsized":"2024-03-06T07:27:04.271722+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:58.666333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254142+0000","last_change":"2024-03-06T07:27:04.254142+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254142+0000","last_undegraded":"2024-03-06T07:27:04.254142+0000","last_fullsized":"2024-03-06T07:27:04.254142+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.259397+0000","last_change":"2024-03-06T07:27:04.258208+0000","last_active":"2024-03-06T07:27:04.259397+0000","last_peered":"2024-03-06T07:27:04.259397+0000","last_clean":"2024-03-06T07:27:04.259397+0000","last_became_active":"2024-03-06T07:27:04.257866+0000","last_became_peered":"2024-03-06T07:27:04.257866+0000","last_unstale":"2024-03-06T07:27:04.259397+0000","last_undegraded":"2024-03-06T07:27:04.259397+0000","last_fullsized":"2024-03-06T07:27:04.259397+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:34.006302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":456,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.262911+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:27:04.262911+0000","last_peered":"2024-03-06T07:27:04.262911+0000","last_clean":"2024-03-06T07:27:04.262911+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:27:04.262911+0000","last_undegraded":"2024-03-06T07:27:04.262911+0000","last_fullsized":"2024-03-06T07:27:04.262911+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33320,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338520,"statfs":{"total":96636764160,"available":96602644480,"internally_reserved":0,"allocated":2191360,"data_stored":1626298,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574932,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1042597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1042597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1035930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574932,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1042597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1042597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1035930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574933,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33320,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338520,"statfs":{"total":96636764160,"available":96602644480,"internally_reserved":0,"allocated":2191360,"data_stored":1626298,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:08.509 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:08.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.643+0000 7f6dbe484640 1 Processor -- start 2024-03-06T07:27:08.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.643+0000 7f6dbe484640 1 -- start start 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.644+0000 7f6dbe484640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6db80c6b10 0x7f6db8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.644+0000 7f6dbe484640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6db80bf9d0 0x7f6db80c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.644+0000 7f6dbe484640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db80c2fc0 con 0x7f6db8152650 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.644+0000 7f6dbe484640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db8155800 con 0x7f6db80c6b10 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.644+0000 7f6dbe484640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db8158da0 con 0x7f6db80bf9d0 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.644+0000 7f6db77fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6db80c6b10 0x7f6db8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.644+0000 7f6db77fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6db80c6b10 0x7f6db8155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59474/0 (socket says 172.21.15.1:59474) 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.644+0000 7f6db77fe640 1 -- 172.21.15.1:0/884724373 learned_addr learned my addr 172.21.15.1:0/884724373 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.644+0000 7f6dbca21640 1 --2- 172.21.15.1:0/884724373 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6db80bf9d0 0x7f6db80c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.645+0000 7f6db77fe640 1 -- 172.21.15.1:0/884724373 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6db80bf9d0 msgr2=0x7f6db80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.645+0000 7f6db77fe640 1 --2- 172.21.15.1:0/884724373 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6db80bf9d0 0x7f6db80c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.645+0000 7f6db77fe640 1 -- 172.21.15.1:0/884724373 >> v1:172.21.15.98:6789/0 conn(0x7f6db8152650 legacy=0x7f6db8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.645+0000 7f6db77fe640 1 -- 172.21.15.1:0/884724373 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db80c6690 con 0x7f6db80c6b10 2024-03-06T07:27:08.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.645+0000 7f6db6ffd640 1 -- 172.21.15.1:0/884724373 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792368319 0 0) 0x7f6db80c2fc0 con 0x7f6db8152650 2024-03-06T07:27:08.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.645+0000 7f6db77fe640 1 --2- 172.21.15.1:0/884724373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db80c6b10 0x7f6db8155c90 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f6dac009870 tx=0x7f6dac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde06438f629978e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.646+0000 7f6db6ffd640 1 -- 172.21.15.1:0/884724373 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dac035020 con 0x7f6db80c6b10 2024-03-06T07:27:08.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.646+0000 7f6db6ffd640 1 -- 172.21.15.1:0/884724373 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dac031640 con 0x7f6db80c6b10 2024-03-06T07:27:08.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.646+0000 7f6db6ffd640 1 -- 172.21.15.1:0/884724373 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dac02d040 con 0x7f6db80c6b10 2024-03-06T07:27:08.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.646+0000 7f6dbe484640 1 -- 172.21.15.1:0/884724373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db80c6b10 msgr2=0x7f6db8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.646+0000 7f6dbe484640 1 --2- 172.21.15.1:0/884724373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db80c6b10 0x7f6db8155c90 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f6dac009870 tx=0x7f6dac02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.646+0000 7f6dbe484640 1 -- 172.21.15.1:0/884724373 shutdown_connections 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.646+0000 7f6dbe484640 1 --2- 172.21.15.1:0/884724373 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6db80bf9d0 0x7f6db80c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.646+0000 7f6dbe484640 1 --2- 172.21.15.1:0/884724373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db80c6b10 0x7f6db8155c90 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.646+0000 7f6dbe484640 1 -- 172.21.15.1:0/884724373 >> 172.21.15.1:0/884724373 conn(0x7f6db8066ee0 msgr2=0x7f6db80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.646+0000 7f6dbe484640 1 -- 172.21.15.1:0/884724373 shutdown_connections 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.646+0000 7f6dbe484640 1 -- 172.21.15.1:0/884724373 wait complete. 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.647+0000 7f6dbe484640 1 Processor -- start 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.647+0000 7f6dbe484640 1 -- start start 2024-03-06T07:27:08.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.648+0000 7f6dbe484640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6db80bf9d0 0x7f6db8153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.648+0000 7f6dbe484640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6db8152650 0x7f6db81dfca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.648+0000 7f6dbe484640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db814b2c0 0x7f6db81e01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.648+0000 7f6dbe484640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db81551b0 con 0x7f6db814b2c0 2024-03-06T07:27:08.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.648+0000 7f6dbe484640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db80c6fa0 con 0x7f6db8152650 2024-03-06T07:27:08.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.648+0000 7f6dbe484640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db8155490 con 0x7f6db80bf9d0 2024-03-06T07:27:08.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.648+0000 7f6dbca21640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db814b2c0 0x7f6db81e01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.648+0000 7f6dbca21640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db814b2c0 0x7f6db81e01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59476/0 (socket says 172.21.15.1:59476) 2024-03-06T07:27:08.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.648+0000 7f6dbca21640 1 -- 172.21.15.1:0/244847192 learned_addr learned my addr 172.21.15.1:0/244847192 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:08.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.648+0000 7f6dbca21640 1 -- 172.21.15.1:0/244847192 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6db80bf9d0 msgr2=0x7f6db8153110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.648+0000 7f6dbca21640 1 --2- 172.21.15.1:0/244847192 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6db80bf9d0 0x7f6db8153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.648+0000 7f6dbca21640 1 -- 172.21.15.1:0/244847192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6db8152650 msgr2=0x7f6db81dfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.648+0000 7f6dbca21640 1 --2- 172.21.15.1:0/244847192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6db8152650 0x7f6db81dfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.648+0000 7f6dbca21640 1 -- 172.21.15.1:0/244847192 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db814b070 con 0x7f6db814b2c0 2024-03-06T07:27:08.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.649+0000 7f6dbca21640 1 --2- 172.21.15.1:0/244847192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db814b2c0 0x7f6db81e01e0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f6db000efc0 tx=0x7f6db000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.649+0000 7f6db4ff9640 1 -- 172.21.15.1:0/244847192 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db0018020 con 0x7f6db814b2c0 2024-03-06T07:27:08.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.649+0000 7f6db4ff9640 1 -- 172.21.15.1:0/244847192 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db0014480 con 0x7f6db814b2c0 2024-03-06T07:27:08.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.649+0000 7f6dbe484640 1 -- 172.21.15.1:0/244847192 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db81e0720 con 0x7f6db814b2c0 2024-03-06T07:27:08.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.649+0000 7f6db4ff9640 1 -- 172.21.15.1:0/244847192 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db0014870 con 0x7f6db814b2c0 2024-03-06T07:27:08.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.649+0000 7f6dbe484640 1 -- 172.21.15.1:0/244847192 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db814ad90 con 0x7f6db814b2c0 2024-03-06T07:27:08.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.650+0000 7f6dbe484640 1 -- 172.21.15.1:0/244847192 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db8067fd0 con 0x7f6db814b2c0 2024-03-06T07:27:08.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.650+0000 7f6db4ff9640 1 -- 172.21.15.1:0/244847192 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6db0005d10 con 0x7f6db814b2c0 2024-03-06T07:27:08.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.651+0000 7f6db4ff9640 1 --2- 172.21.15.1:0/244847192 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6da406b4d0 0x7f6da406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.651+0000 7f6db4ff9640 1 -- 172.21.15.1:0/244847192 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6db008ada0 con 0x7f6db814b2c0 2024-03-06T07:27:08.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.651+0000 7f6db7fff640 1 --2- 172.21.15.1:0/244847192 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6da406b4d0 0x7f6da406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.652+0000 7f6db7fff640 1 --2- 172.21.15.1:0/244847192 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6da406b4d0 0x7f6da406d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f6db81e1720 tx=0x7f6da8007930 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.655+0000 7f6db4ff9640 1 -- 172.21.15.1:0/244847192 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6db0022070 con 0x7f6db814b2c0 2024-03-06T07:27:08.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.836+0000 7f6dbe484640 1 -- 172.21.15.1:0/244847192 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6db8067440 con 0x7f6da406b4d0 2024-03-06T07:27:08.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.842+0000 7f6db4ff9640 1 -- 172.21.15.1:0/244847192 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247159 (secure 0 0 0) 0x7f6db8067440 con 0x7f6da406b4d0 2024-03-06T07:27:08.846 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:08.848 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:27:08.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.845+0000 7f6dbe484640 1 -- 172.21.15.1:0/244847192 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6da406b4d0 msgr2=0x7f6da406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.845+0000 7f6dbe484640 1 --2- 172.21.15.1:0/244847192 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6da406b4d0 0x7f6da406d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f6db81e1720 tx=0x7f6da8007930 comp rx=0 tx=0).stop 2024-03-06T07:27:08.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.845+0000 7f6dbe484640 1 -- 172.21.15.1:0/244847192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db814b2c0 msgr2=0x7f6db81e01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.845+0000 7f6dbe484640 1 --2- 172.21.15.1:0/244847192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db814b2c0 0x7f6db81e01e0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f6db000efc0 tx=0x7f6db000c490 comp rx=0 tx=0).stop 2024-03-06T07:27:08.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.845+0000 7f6dbe484640 1 -- 172.21.15.1:0/244847192 shutdown_connections 2024-03-06T07:27:08.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.845+0000 7f6dbe484640 1 --2- 172.21.15.1:0/244847192 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6da406b4d0 0x7f6da406d990 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.845+0000 7f6dbe484640 1 --2- 172.21.15.1:0/244847192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db814b2c0 0x7f6db81e01e0 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.845+0000 7f6dbe484640 1 --2- 172.21.15.1:0/244847192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6db8152650 0x7f6db81dfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.845+0000 7f6dbe484640 1 --2- 172.21.15.1:0/244847192 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6db80bf9d0 0x7f6db8153110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.845+0000 7f6dbe484640 1 -- 172.21.15.1:0/244847192 >> 172.21.15.1:0/244847192 conn(0x7f6db8066ee0 msgr2=0x7f6db80c0420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.845+0000 7f6dbe484640 1 -- 172.21.15.1:0/244847192 shutdown_connections 2024-03-06T07:27:08.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.845+0000 7f6dbe484640 1 -- 172.21.15.1:0/244847192 wait complete. 2024-03-06T07:27:08.858 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-03-06T07:27:08.784835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":266240,"kb_used_data":16784,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754708480,"statfs":{"total":773094113280,"available":772821483520,"internally_reserved":0,"allocated":17186816,"data_stored":9494844,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.533351"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254547+0000","last_change":"2024-03-06T07:27:04.254547+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254547+0000","last_undegraded":"2024-03-06T07:27:04.254547+0000","last_fullsized":"2024-03-06T07:27:04.254547+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.255495+0000","last_change":"2024-03-06T07:27:04.255495+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.255495+0000","last_undegraded":"2024-03-06T07:27:04.255495+0000","last_fullsized":"2024-03-06T07:27:04.255495+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259837+0000","last_change":"2024-03-06T07:27:04.259837+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259837+0000","last_undegraded":"2024-03-06T07:27:04.259837+0000","last_fullsized":"2024-03-06T07:27:04.259837+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259747+0000","last_change":"2024-03-06T07:27:04.259747+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259747+0000","last_undegraded":"2024-03-06T07:27:04.259747+0000","last_fullsized":"2024-03-06T07:27:04.259747+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258205+0000","last_change":"2024-03-06T07:27:04.258205+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258205+0000","last_undegraded":"2024-03-06T07:27:04.258205+0000","last_fullsized":"2024-03-06T07:27:04.258205+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259513+0000","last_change":"2024-03-06T07:27:04.259513+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259513+0000","last_undegraded":"2024-03-06T07:27:04.259513+0000","last_fullsized":"2024-03-06T07:27:04.259513+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257656+0000","last_change":"2024-03-06T07:27:04.257656+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257656+0000","last_undegraded":"2024-03-06T07:27:04.257656+0000","last_fullsized":"2024-03-06T07:27:04.257656+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257414+0000","last_change":"2024-03-06T07:27:04.257414+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257414+0000","last_undegraded":"2024-03-06T07:27:04.257414+0000","last_fullsized":"2024-03-06T07:27:04.257414+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258740+0000","last_change":"2024-03-06T07:27:04.258740+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258740+0000","last_undegraded":"2024-03-06T07:27:04.258740+0000","last_fullsized":"2024-03-06T07:27:04.258740+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258528+0000","last_change":"2024-03-06T07:27:04.258528+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258528+0000","last_undegraded":"2024-03-06T07:27:04.258528+0000","last_fullsized":"2024-03-06T07:27:04.258528+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259375+0000","last_change":"2024-03-06T07:27:04.259375+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259375+0000","last_undegraded":"2024-03-06T07:27:04.259375+0000","last_fullsized":"2024-03-06T07:27:04.259375+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257246+0000","last_change":"2024-03-06T07:27:04.257246+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257246+0000","last_undegraded":"2024-03-06T07:27:04.257246+0000","last_fullsized":"2024-03-06T07:27:04.257246+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259445+0000","last_change":"2024-03-06T07:27:04.259445+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259445+0000","last_undegraded":"2024-03-06T07:27:04.259445+0000","last_fullsized":"2024-03-06T07:27:04.259445+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257927+0000","last_change":"2024-03-06T07:27:04.257927+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257927+0000","last_undegraded":"2024-03-06T07:27:04.257927+0000","last_fullsized":"2024-03-06T07:27:04.257927+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.263169+0000","last_change":"2024-03-06T07:27:04.263169+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.263169+0000","last_undegraded":"2024-03-06T07:27:04.263169+0000","last_fullsized":"2024-03-06T07:27:04.263169+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257264+0000","last_change":"2024-03-06T07:27:04.257264+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257264+0000","last_undegraded":"2024-03-06T07:27:04.257264+0000","last_fullsized":"2024-03-06T07:27:04.257264+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258290+0000","last_change":"2024-03-06T07:27:04.258290+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258290+0000","last_undegraded":"2024-03-06T07:27:04.258290+0000","last_fullsized":"2024-03-06T07:27:04.258290+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258702+0000","last_change":"2024-03-06T07:27:04.258702+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258702+0000","last_undegraded":"2024-03-06T07:27:04.258702+0000","last_fullsized":"2024-03-06T07:27:04.258702+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258065+0000","last_change":"2024-03-06T07:27:04.258065+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258065+0000","last_undegraded":"2024-03-06T07:27:04.258065+0000","last_fullsized":"2024-03-06T07:27:04.258065+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258945+0000","last_change":"2024-03-06T07:27:04.258945+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258945+0000","last_undegraded":"2024-03-06T07:27:04.258945+0000","last_fullsized":"2024-03-06T07:27:04.258945+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257755+0000","last_change":"2024-03-06T07:27:04.257755+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257755+0000","last_undegraded":"2024-03-06T07:27:04.257755+0000","last_fullsized":"2024-03-06T07:27:04.257755+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257343+0000","last_change":"2024-03-06T07:27:04.257343+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257343+0000","last_undegraded":"2024-03-06T07:27:04.257343+0000","last_fullsized":"2024-03-06T07:27:04.257343+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.253999+0000","last_change":"2024-03-06T07:27:04.253999+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.253999+0000","last_undegraded":"2024-03-06T07:27:04.253999+0000","last_fullsized":"2024-03-06T07:27:04.253999+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256789+0000","last_change":"2024-03-06T07:27:04.256789+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256789+0000","last_undegraded":"2024-03-06T07:27:04.256789+0000","last_fullsized":"2024-03-06T07:27:04.256789+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257184+0000","last_change":"2024-03-06T07:27:04.257184+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257184+0000","last_undegraded":"2024-03-06T07:27:04.257184+0000","last_fullsized":"2024-03-06T07:27:04.257184+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257217+0000","last_change":"2024-03-06T07:27:04.257217+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257217+0000","last_undegraded":"2024-03-06T07:27:04.257217+0000","last_fullsized":"2024-03-06T07:27:04.257217+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259006+0000","last_change":"2024-03-06T07:27:04.259006+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259006+0000","last_undegraded":"2024-03-06T07:27:04.259006+0000","last_fullsized":"2024-03-06T07:27:04.259006+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.255903+0000","last_change":"2024-03-06T07:27:04.255903+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.255903+0000","last_undegraded":"2024-03-06T07:27:04.255903+0000","last_fullsized":"2024-03-06T07:27:04.255903+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254435+0000","last_change":"2024-03-06T07:27:04.254435+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254435+0000","last_undegraded":"2024-03-06T07:27:04.254435+0000","last_fullsized":"2024-03-06T07:27:04.254435+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.264763+0000","last_change":"2024-03-06T07:27:04.264763+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.264763+0000","last_undegraded":"2024-03-06T07:27:04.264763+0000","last_fullsized":"2024-03-06T07:27:04.264763+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261771+0000","last_change":"2024-03-06T07:27:04.261771+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261771+0000","last_undegraded":"2024-03-06T07:27:04.261771+0000","last_fullsized":"2024-03-06T07:27:04.261771+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.260482+0000","last_change":"2024-03-06T07:27:04.260482+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.260482+0000","last_undegraded":"2024-03-06T07:27:04.260482+0000","last_fullsized":"2024-03-06T07:27:04.260482+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254981+0000","last_change":"2024-03-06T07:27:04.254981+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254981+0000","last_undegraded":"2024-03-06T07:27:04.254981+0000","last_fullsized":"2024-03-06T07:27:04.254981+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.278552+0000","last_change":"2024-03-06T07:27:04.271712+0000","last_active":"2024-03-06T07:27:04.278552+0000","last_peered":"2024-03-06T07:27:04.278552+0000","last_clean":"2024-03-06T07:27:04.278552+0000","last_became_active":"2024-03-06T07:27:04.271429+0000","last_became_peered":"2024-03-06T07:27:04.271429+0000","last_unstale":"2024-03-06T07:27:04.278552+0000","last_undegraded":"2024-03-06T07:27:04.278552+0000","last_fullsized":"2024-03-06T07:27:04.278552+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:20.726754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257054+0000","last_change":"2024-03-06T07:27:04.257054+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257054+0000","last_undegraded":"2024-03-06T07:27:04.257054+0000","last_fullsized":"2024-03-06T07:27:04.257054+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.268329+0000","last_change":"2024-03-06T07:27:04.266276+0000","last_active":"2024-03-06T07:27:04.268329+0000","last_peered":"2024-03-06T07:27:04.268329+0000","last_clean":"2024-03-06T07:27:04.268329+0000","last_became_active":"2024-03-06T07:27:04.265418+0000","last_became_peered":"2024-03-06T07:27:04.265418+0000","last_unstale":"2024-03-06T07:27:04.268329+0000","last_undegraded":"2024-03-06T07:27:04.268329+0000","last_fullsized":"2024-03-06T07:27:04.268329+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:24.992967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256890+0000","last_change":"2024-03-06T07:27:04.256890+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256890+0000","last_undegraded":"2024-03-06T07:27:04.256890+0000","last_fullsized":"2024-03-06T07:27:04.256890+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267032+0000","last_change":"2024-03-06T07:27:04.262189+0000","last_active":"2024-03-06T07:27:04.267032+0000","last_peered":"2024-03-06T07:27:04.267032+0000","last_clean":"2024-03-06T07:27:04.267032+0000","last_became_active":"2024-03-06T07:27:04.261909+0000","last_became_peered":"2024-03-06T07:27:04.261909+0000","last_unstale":"2024-03-06T07:27:04.267032+0000","last_undegraded":"2024-03-06T07:27:04.267032+0000","last_fullsized":"2024-03-06T07:27:04.267032+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:15:43.228599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256537+0000","last_change":"2024-03-06T07:27:04.256537+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256537+0000","last_undegraded":"2024-03-06T07:27:04.256537+0000","last_fullsized":"2024-03-06T07:27:04.256537+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271246+0000","last_change":"2024-03-06T07:27:04.270472+0000","last_active":"2024-03-06T07:27:04.271246+0000","last_peered":"2024-03-06T07:27:04.271246+0000","last_clean":"2024-03-06T07:27:04.271246+0000","last_became_active":"2024-03-06T07:27:04.270185+0000","last_became_peered":"2024-03-06T07:27:04.270185+0000","last_unstale":"2024-03-06T07:27:04.271246+0000","last_undegraded":"2024-03-06T07:27:04.271246+0000","last_fullsized":"2024-03-06T07:27:04.271246+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:37.715400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.263186+0000","last_change":"2024-03-06T07:27:04.263186+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.263186+0000","last_undegraded":"2024-03-06T07:27:04.263186+0000","last_fullsized":"2024-03-06T07:27:04.263186+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.275596+0000","last_change":"2024-03-06T07:27:04.272491+0000","last_active":"2024-03-06T07:27:04.275596+0000","last_peered":"2024-03-06T07:27:04.275596+0000","last_clean":"2024-03-06T07:27:04.275596+0000","last_became_active":"2024-03-06T07:27:04.272190+0000","last_became_peered":"2024-03-06T07:27:04.272190+0000","last_unstale":"2024-03-06T07:27:04.275596+0000","last_undegraded":"2024-03-06T07:27:04.275596+0000","last_fullsized":"2024-03-06T07:27:04.275596+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:17.476526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.263035+0000","last_change":"2024-03-06T07:27:04.263035+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.263035+0000","last_undegraded":"2024-03-06T07:27:04.263035+0000","last_fullsized":"2024-03-06T07:27:04.263035+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.278476+0000","last_change":"2024-03-06T07:27:04.273573+0000","last_active":"2024-03-06T07:27:04.278476+0000","last_peered":"2024-03-06T07:27:04.278476+0000","last_clean":"2024-03-06T07:27:04.278476+0000","last_became_active":"2024-03-06T07:27:04.273253+0000","last_became_peered":"2024-03-06T07:27:04.273253+0000","last_unstale":"2024-03-06T07:27:04.278476+0000","last_undegraded":"2024-03-06T07:27:04.278476+0000","last_fullsized":"2024-03-06T07:27:04.278476+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:56.531270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254166+0000","last_change":"2024-03-06T07:27:04.254166+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254166+0000","last_undegraded":"2024-03-06T07:27:04.254166+0000","last_fullsized":"2024-03-06T07:27:04.254166+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.272537+0000","last_change":"2024-03-06T07:27:04.271894+0000","last_active":"2024-03-06T07:27:04.272537+0000","last_peered":"2024-03-06T07:27:04.272537+0000","last_clean":"2024-03-06T07:27:04.272537+0000","last_became_active":"2024-03-06T07:27:04.271305+0000","last_became_peered":"2024-03-06T07:27:04.271305+0000","last_unstale":"2024-03-06T07:27:04.272537+0000","last_undegraded":"2024-03-06T07:27:04.272537+0000","last_fullsized":"2024-03-06T07:27:04.272537+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:26.898488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.266335+0000","last_change":"2024-03-06T07:27:04.266335+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.266335+0000","last_undegraded":"2024-03-06T07:27:04.266335+0000","last_fullsized":"2024-03-06T07:27:04.266335+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271368+0000","last_change":"2024-03-06T07:27:04.269100+0000","last_active":"2024-03-06T07:27:04.271368+0000","last_peered":"2024-03-06T07:27:04.271368+0000","last_clean":"2024-03-06T07:27:04.271368+0000","last_became_active":"2024-03-06T07:27:04.268811+0000","last_became_peered":"2024-03-06T07:27:04.268811+0000","last_unstale":"2024-03-06T07:27:04.271368+0000","last_undegraded":"2024-03-06T07:27:04.271368+0000","last_fullsized":"2024-03-06T07:27:04.271368+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:34.615045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258161+0000","last_change":"2024-03-06T07:27:04.258161+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258161+0000","last_undegraded":"2024-03-06T07:27:04.258161+0000","last_fullsized":"2024-03-06T07:27:04.258161+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.275534+0000","last_change":"2024-03-06T07:27:04.271341+0000","last_active":"2024-03-06T07:27:04.275534+0000","last_peered":"2024-03-06T07:27:04.275534+0000","last_clean":"2024-03-06T07:27:04.275534+0000","last_became_active":"2024-03-06T07:27:04.271023+0000","last_became_peered":"2024-03-06T07:27:04.271023+0000","last_unstale":"2024-03-06T07:27:04.275534+0000","last_undegraded":"2024-03-06T07:27:04.275534+0000","last_fullsized":"2024-03-06T07:27:04.275534+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:24.173726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257145+0000","last_change":"2024-03-06T07:27:04.257145+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257145+0000","last_undegraded":"2024-03-06T07:27:04.257145+0000","last_fullsized":"2024-03-06T07:27:04.257145+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.266679+0000","last_change":"2024-03-06T07:27:04.264830+0000","last_active":"2024-03-06T07:27:04.266679+0000","last_peered":"2024-03-06T07:27:04.266679+0000","last_clean":"2024-03-06T07:27:04.266679+0000","last_became_active":"2024-03-06T07:27:04.264562+0000","last_became_peered":"2024-03-06T07:27:04.264562+0000","last_unstale":"2024-03-06T07:27:04.266679+0000","last_undegraded":"2024-03-06T07:27:04.266679+0000","last_fullsized":"2024-03-06T07:27:04.266679+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:12.146965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256102+0000","last_change":"2024-03-06T07:27:04.256102+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256102+0000","last_undegraded":"2024-03-06T07:27:04.256102+0000","last_fullsized":"2024-03-06T07:27:04.256102+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.278496+0000","last_change":"2024-03-06T07:27:04.273664+0000","last_active":"2024-03-06T07:27:04.278496+0000","last_peered":"2024-03-06T07:27:04.278496+0000","last_clean":"2024-03-06T07:27:04.278496+0000","last_became_active":"2024-03-06T07:27:04.273372+0000","last_became_peered":"2024-03-06T07:27:04.273372+0000","last_unstale":"2024-03-06T07:27:04.278496+0000","last_undegraded":"2024-03-06T07:27:04.278496+0000","last_fullsized":"2024-03-06T07:27:04.278496+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:51.519017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.255456+0000","last_change":"2024-03-06T07:27:04.255456+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.255456+0000","last_undegraded":"2024-03-06T07:27:04.255456+0000","last_fullsized":"2024-03-06T07:27:04.255456+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267158+0000","last_change":"2024-03-06T07:27:04.263075+0000","last_active":"2024-03-06T07:27:04.267158+0000","last_peered":"2024-03-06T07:27:04.267158+0000","last_clean":"2024-03-06T07:27:04.267158+0000","last_became_active":"2024-03-06T07:27:04.261822+0000","last_became_peered":"2024-03-06T07:27:04.261822+0000","last_unstale":"2024-03-06T07:27:04.267158+0000","last_undegraded":"2024-03-06T07:27:04.267158+0000","last_fullsized":"2024-03-06T07:27:04.267158+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:50.416453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258591+0000","last_change":"2024-03-06T07:27:04.258591+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258591+0000","last_undegraded":"2024-03-06T07:27:04.258591+0000","last_fullsized":"2024-03-06T07:27:04.258591+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.270193+0000","last_change":"2024-03-06T07:27:04.268027+0000","last_active":"2024-03-06T07:27:04.270193+0000","last_peered":"2024-03-06T07:27:04.270193+0000","last_clean":"2024-03-06T07:27:04.270193+0000","last_became_active":"2024-03-06T07:27:04.267726+0000","last_became_peered":"2024-03-06T07:27:04.267726+0000","last_unstale":"2024-03-06T07:27:04.270193+0000","last_undegraded":"2024-03-06T07:27:04.270193+0000","last_fullsized":"2024-03-06T07:27:04.270193+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:11.943331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254886+0000","last_change":"2024-03-06T07:27:04.254886+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254886+0000","last_undegraded":"2024-03-06T07:27:04.254886+0000","last_fullsized":"2024-03-06T07:27:04.254886+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.266762+0000","last_change":"2024-03-06T07:27:04.265676+0000","last_active":"2024-03-06T07:27:04.266762+0000","last_peered":"2024-03-06T07:27:04.266762+0000","last_clean":"2024-03-06T07:27:04.266762+0000","last_became_active":"2024-03-06T07:27:04.265374+0000","last_became_peered":"2024-03-06T07:27:04.265374+0000","last_unstale":"2024-03-06T07:27:04.266762+0000","last_undegraded":"2024-03-06T07:27:04.266762+0000","last_fullsized":"2024-03-06T07:27:04.266762+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:30.199858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261058+0000","last_change":"2024-03-06T07:27:04.261058+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261058+0000","last_undegraded":"2024-03-06T07:27:04.261058+0000","last_fullsized":"2024-03-06T07:27:04.261058+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271352+0000","last_change":"2024-03-06T07:27:04.269119+0000","last_active":"2024-03-06T07:27:04.271352+0000","last_peered":"2024-03-06T07:27:04.271352+0000","last_clean":"2024-03-06T07:27:04.271352+0000","last_became_active":"2024-03-06T07:27:04.268844+0000","last_became_peered":"2024-03-06T07:27:04.268844+0000","last_unstale":"2024-03-06T07:27:04.271352+0000","last_undegraded":"2024-03-06T07:27:04.271352+0000","last_fullsized":"2024-03-06T07:27:04.271352+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:47.701531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256965+0000","last_change":"2024-03-06T07:27:04.256965+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256965+0000","last_undegraded":"2024-03-06T07:27:04.256965+0000","last_fullsized":"2024-03-06T07:27:04.256965+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.268206+0000","last_change":"2024-03-06T07:27:04.262327+0000","last_active":"2024-03-06T07:27:04.268206+0000","last_peered":"2024-03-06T07:27:04.268206+0000","last_clean":"2024-03-06T07:27:04.268206+0000","last_became_active":"2024-03-06T07:27:04.262046+0000","last_became_peered":"2024-03-06T07:27:04.262046+0000","last_unstale":"2024-03-06T07:27:04.268206+0000","last_undegraded":"2024-03-06T07:27:04.268206+0000","last_fullsized":"2024-03-06T07:27:04.268206+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:31.006078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256647+0000","last_change":"2024-03-06T07:27:04.256647+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256647+0000","last_undegraded":"2024-03-06T07:27:04.256647+0000","last_fullsized":"2024-03-06T07:27:04.256647+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.268350+0000","last_change":"2024-03-06T07:27:04.266893+0000","last_active":"2024-03-06T07:27:04.268350+0000","last_peered":"2024-03-06T07:27:04.268350+0000","last_clean":"2024-03-06T07:27:04.268350+0000","last_became_active":"2024-03-06T07:27:04.265563+0000","last_became_peered":"2024-03-06T07:27:04.265563+0000","last_unstale":"2024-03-06T07:27:04.268350+0000","last_undegraded":"2024-03-06T07:27:04.268350+0000","last_fullsized":"2024-03-06T07:27:04.268350+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:35.022673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254208+0000","last_change":"2024-03-06T07:27:04.254208+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254208+0000","last_undegraded":"2024-03-06T07:27:04.254208+0000","last_fullsized":"2024-03-06T07:27:04.254208+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267134+0000","last_change":"2024-03-06T07:27:04.265277+0000","last_active":"2024-03-06T07:27:04.267134+0000","last_peered":"2024-03-06T07:27:04.267134+0000","last_clean":"2024-03-06T07:27:04.267134+0000","last_became_active":"2024-03-06T07:27:04.265018+0000","last_became_peered":"2024-03-06T07:27:04.265018+0000","last_unstale":"2024-03-06T07:27:04.267134+0000","last_undegraded":"2024-03-06T07:27:04.267134+0000","last_fullsized":"2024-03-06T07:27:04.267134+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:51.929743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257320+0000","last_change":"2024-03-06T07:27:04.257320+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257320+0000","last_undegraded":"2024-03-06T07:27:04.257320+0000","last_fullsized":"2024-03-06T07:27:04.257320+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267517+0000","last_change":"2024-03-06T07:27:04.265842+0000","last_active":"2024-03-06T07:27:04.267517+0000","last_peered":"2024-03-06T07:27:04.267517+0000","last_clean":"2024-03-06T07:27:04.267517+0000","last_became_active":"2024-03-06T07:27:04.265568+0000","last_became_peered":"2024-03-06T07:27:04.265568+0000","last_unstale":"2024-03-06T07:27:04.267517+0000","last_undegraded":"2024-03-06T07:27:04.267517+0000","last_fullsized":"2024-03-06T07:27:04.267517+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:31.100594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254138+0000","last_change":"2024-03-06T07:27:04.254138+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254138+0000","last_undegraded":"2024-03-06T07:27:04.254138+0000","last_fullsized":"2024-03-06T07:27:04.254138+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.264364+0000","last_change":"2024-03-06T07:27:04.262729+0000","last_active":"2024-03-06T07:27:04.264364+0000","last_peered":"2024-03-06T07:27:04.264364+0000","last_clean":"2024-03-06T07:27:04.264364+0000","last_became_active":"2024-03-06T07:27:04.262484+0000","last_became_peered":"2024-03-06T07:27:04.262484+0000","last_unstale":"2024-03-06T07:27:04.264364+0000","last_undegraded":"2024-03-06T07:27:04.264364+0000","last_fullsized":"2024-03-06T07:27:04.264364+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:38.945976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261717+0000","last_change":"2024-03-06T07:27:04.261717+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261717+0000","last_undegraded":"2024-03-06T07:27:04.261717+0000","last_fullsized":"2024-03-06T07:27:04.261717+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267085+0000","last_change":"2024-03-06T07:27:04.263995+0000","last_active":"2024-03-06T07:27:04.267085+0000","last_peered":"2024-03-06T07:27:04.267085+0000","last_clean":"2024-03-06T07:27:04.267085+0000","last_became_active":"2024-03-06T07:27:04.263629+0000","last_became_peered":"2024-03-06T07:27:04.263629+0000","last_unstale":"2024-03-06T07:27:04.267085+0000","last_undegraded":"2024-03-06T07:27:04.267085+0000","last_fullsized":"2024-03-06T07:27:04.267085+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:44.864909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254160+0000","last_change":"2024-03-06T07:27:04.254160+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254160+0000","last_undegraded":"2024-03-06T07:27:04.254160+0000","last_fullsized":"2024-03-06T07:27:04.254160+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.278500+0000","last_change":"2024-03-06T07:27:04.271282+0000","last_active":"2024-03-06T07:27:04.278500+0000","last_peered":"2024-03-06T07:27:04.278500+0000","last_clean":"2024-03-06T07:27:04.278500+0000","last_became_active":"2024-03-06T07:27:04.270987+0000","last_became_peered":"2024-03-06T07:27:04.270987+0000","last_unstale":"2024-03-06T07:27:04.278500+0000","last_undegraded":"2024-03-06T07:27:04.278500+0000","last_fullsized":"2024-03-06T07:27:04.278500+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:02.098293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258625+0000","last_change":"2024-03-06T07:27:04.258625+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258625+0000","last_undegraded":"2024-03-06T07:27:04.258625+0000","last_fullsized":"2024-03-06T07:27:04.258625+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.275695+0000","last_change":"2024-03-06T07:27:04.272151+0000","last_active":"2024-03-06T07:27:04.275695+0000","last_peered":"2024-03-06T07:27:04.275695+0000","last_clean":"2024-03-06T07:27:04.275695+0000","last_became_active":"2024-03-06T07:27:04.271894+0000","last_became_peered":"2024-03-06T07:27:04.271894+0000","last_unstale":"2024-03-06T07:27:04.275695+0000","last_undegraded":"2024-03-06T07:27:04.275695+0000","last_fullsized":"2024-03-06T07:27:04.275695+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:50.049060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256530+0000","last_change":"2024-03-06T07:27:04.256530+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256530+0000","last_undegraded":"2024-03-06T07:27:04.256530+0000","last_fullsized":"2024-03-06T07:27:04.256530+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.270258+0000","last_change":"2024-03-06T07:27:04.268865+0000","last_active":"2024-03-06T07:27:04.270258+0000","last_peered":"2024-03-06T07:27:04.270258+0000","last_clean":"2024-03-06T07:27:04.270258+0000","last_became_active":"2024-03-06T07:27:04.268566+0000","last_became_peered":"2024-03-06T07:27:04.268566+0000","last_unstale":"2024-03-06T07:27:04.270258+0000","last_undegraded":"2024-03-06T07:27:04.270258+0000","last_fullsized":"2024-03-06T07:27:04.270258+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:05.786807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257744+0000","last_change":"2024-03-06T07:27:04.257744+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257744+0000","last_undegraded":"2024-03-06T07:27:04.257744+0000","last_fullsized":"2024-03-06T07:27:04.257744+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.270471+0000","last_change":"2024-03-06T07:27:04.264784+0000","last_active":"2024-03-06T07:27:04.270471+0000","last_peered":"2024-03-06T07:27:04.270471+0000","last_clean":"2024-03-06T07:27:04.270471+0000","last_became_active":"2024-03-06T07:27:04.264499+0000","last_became_peered":"2024-03-06T07:27:04.264499+0000","last_unstale":"2024-03-06T07:27:04.270471+0000","last_undegraded":"2024-03-06T07:27:04.270471+0000","last_fullsized":"2024-03-06T07:27:04.270471+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:38.542889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254229+0000","last_change":"2024-03-06T07:27:04.254229+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254229+0000","last_undegraded":"2024-03-06T07:27:04.254229+0000","last_fullsized":"2024-03-06T07:27:04.254229+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.275452+0000","last_change":"2024-03-06T07:27:04.269541+0000","last_active":"2024-03-06T07:27:04.275452+0000","last_peered":"2024-03-06T07:27:04.275452+0000","last_clean":"2024-03-06T07:27:04.275452+0000","last_became_active":"2024-03-06T07:27:04.269228+0000","last_became_peered":"2024-03-06T07:27:04.269228+0000","last_unstale":"2024-03-06T07:27:04.275452+0000","last_undegraded":"2024-03-06T07:27:04.275452+0000","last_fullsized":"2024-03-06T07:27:04.275452+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:05.198388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254889+0000","last_change":"2024-03-06T07:27:04.254889+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254889+0000","last_undegraded":"2024-03-06T07:27:04.254889+0000","last_fullsized":"2024-03-06T07:27:04.254889+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.264329+0000","last_change":"2024-03-06T07:27:04.260635+0000","last_active":"2024-03-06T07:27:04.264329+0000","last_peered":"2024-03-06T07:27:04.264329+0000","last_clean":"2024-03-06T07:27:04.264329+0000","last_became_active":"2024-03-06T07:27:04.260370+0000","last_became_peered":"2024-03-06T07:27:04.260370+0000","last_unstale":"2024-03-06T07:27:04.264329+0000","last_undegraded":"2024-03-06T07:27:04.264329+0000","last_fullsized":"2024-03-06T07:27:04.264329+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:26.449033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257795+0000","last_change":"2024-03-06T07:27:04.257795+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257795+0000","last_undegraded":"2024-03-06T07:27:04.257795+0000","last_fullsized":"2024-03-06T07:27:04.257795+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.272959+0000","last_change":"2024-03-06T07:27:04.269073+0000","last_active":"2024-03-06T07:27:04.272959+0000","last_peered":"2024-03-06T07:27:04.272959+0000","last_clean":"2024-03-06T07:27:04.272959+0000","last_became_active":"2024-03-06T07:27:04.268831+0000","last_became_peered":"2024-03-06T07:27:04.268831+0000","last_unstale":"2024-03-06T07:27:04.272959+0000","last_undegraded":"2024-03-06T07:27:04.272959+0000","last_fullsized":"2024-03-06T07:27:04.272959+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:30.526595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261310+0000","last_change":"2024-03-06T07:27:04.261310+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261310+0000","last_undegraded":"2024-03-06T07:27:04.261310+0000","last_fullsized":"2024-03-06T07:27:04.261310+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271907+0000","last_change":"2024-03-06T07:27:04.271202+0000","last_active":"2024-03-06T07:27:04.271907+0000","last_peered":"2024-03-06T07:27:04.271907+0000","last_clean":"2024-03-06T07:27:04.271907+0000","last_became_active":"2024-03-06T07:27:04.270881+0000","last_became_peered":"2024-03-06T07:27:04.270881+0000","last_unstale":"2024-03-06T07:27:04.271907+0000","last_undegraded":"2024-03-06T07:27:04.271907+0000","last_fullsized":"2024-03-06T07:27:04.271907+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:17.006730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261395+0000","last_change":"2024-03-06T07:27:04.261395+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261395+0000","last_undegraded":"2024-03-06T07:27:04.261395+0000","last_fullsized":"2024-03-06T07:27:04.261395+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.273065+0000","last_change":"2024-03-06T07:27:04.269025+0000","last_active":"2024-03-06T07:27:04.273065+0000","last_peered":"2024-03-06T07:27:04.273065+0000","last_clean":"2024-03-06T07:27:04.273065+0000","last_became_active":"2024-03-06T07:27:04.268768+0000","last_became_peered":"2024-03-06T07:27:04.268768+0000","last_unstale":"2024-03-06T07:27:04.273065+0000","last_undegraded":"2024-03-06T07:27:04.273065+0000","last_fullsized":"2024-03-06T07:27:04.273065+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:28.810603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.260194+0000","last_change":"2024-03-06T07:27:04.260194+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.260194+0000","last_undegraded":"2024-03-06T07:27:04.260194+0000","last_fullsized":"2024-03-06T07:27:04.260194+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271722+0000","last_change":"2024-03-06T07:27:04.269800+0000","last_active":"2024-03-06T07:27:04.271722+0000","last_peered":"2024-03-06T07:27:04.271722+0000","last_clean":"2024-03-06T07:27:04.271722+0000","last_became_active":"2024-03-06T07:27:04.269489+0000","last_became_peered":"2024-03-06T07:27:04.269489+0000","last_unstale":"2024-03-06T07:27:04.271722+0000","last_undegraded":"2024-03-06T07:27:04.271722+0000","last_fullsized":"2024-03-06T07:27:04.271722+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:58.666333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254142+0000","last_change":"2024-03-06T07:27:04.254142+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254142+0000","last_undegraded":"2024-03-06T07:27:04.254142+0000","last_fullsized":"2024-03-06T07:27:04.254142+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.259397+0000","last_change":"2024-03-06T07:27:04.258208+0000","last_active":"2024-03-06T07:27:04.259397+0000","last_peered":"2024-03-06T07:27:04.259397+0000","last_clean":"2024-03-06T07:27:04.259397+0000","last_became_active":"2024-03-06T07:27:04.257866+0000","last_became_peered":"2024-03-06T07:27:04.257866+0000","last_unstale":"2024-03-06T07:27:04.259397+0000","last_undegraded":"2024-03-06T07:27:04.259397+0000","last_fullsized":"2024-03-06T07:27:04.259397+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:34.006302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":456,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.262911+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:27:04.262911+0000","last_peered":"2024-03-06T07:27:04.262911+0000","last_clean":"2024-03-06T07:27:04.262911+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:27:04.262911+0000","last_undegraded":"2024-03-06T07:27:04.262911+0000","last_fullsized":"2024-03-06T07:27:04.262911+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33320,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338520,"statfs":{"total":96636764160,"available":96602644480,"internally_reserved":0,"allocated":2191360,"data_stored":1626298,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574932,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1042597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1042597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1035930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574932,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1042597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1042597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1035930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574933,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33320,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338520,"statfs":{"total":96636764160,"available":96602644480,"internally_reserved":0,"allocated":2191360,"data_stored":1626298,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:08.862 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:09.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.999+0000 7fc5f6bf7640 1 Processor -- start 2024-03-06T07:27:09.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:08.999+0000 7fc5f6bf7640 1 -- start start 2024-03-06T07:27:09.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.000+0000 7fc5f6bf7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5f0152650 0x7fc5f0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.000+0000 7fc5f6bf7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5f00bf9d0 0x7fc5f00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.000+0000 7fc5f6bf7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f00c2fc0 con 0x7fc5f00c6b10 2024-03-06T07:27:09.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.000+0000 7fc5f6bf7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5f0155800 con 0x7fc5f0152650 2024-03-06T07:27:09.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.000+0000 7fc5f6bf7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f0158da0 con 0x7fc5f00bf9d0 2024-03-06T07:27:09.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.000+0000 7fc5f4993640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5f0152650 0x7fc5f0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.000+0000 7fc5f4993640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5f0152650 0x7fc5f0155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59480/0 (socket says 172.21.15.1:59480) 2024-03-06T07:27:09.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.000+0000 7fc5e7fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc5f00c6b10 0x7fc5f0158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44400/0 (socket says 172.21.15.1:44400) 2024-03-06T07:27:09.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.000+0000 7fc5f4993640 1 -- 172.21.15.1:0/172903100 learned_addr learned my addr 172.21.15.1:0/172903100 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:09.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.000+0000 7fc5f5194640 1 --2- 172.21.15.1:0/172903100 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5f00bf9d0 0x7fc5f00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.000+0000 7fc5f4993640 1 -- 172.21.15.1:0/172903100 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5f00bf9d0 msgr2=0x7fc5f00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.000+0000 7fc5f4993640 1 --2- 172.21.15.1:0/172903100 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5f00bf9d0 0x7fc5f00c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.000+0000 7fc5f4993640 1 -- 172.21.15.1:0/172903100 >> v1:172.21.15.1:6789/0 conn(0x7fc5f00c6b10 legacy=0x7fc5f0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.000+0000 7fc5f4993640 1 -- 172.21.15.1:0/172903100 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f00c6690 con 0x7fc5f0152650 2024-03-06T07:27:09.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.000+0000 7fc5f5194640 1 --2- 172.21.15.1:0/172903100 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5f00bf9d0 0x7fc5f00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:09.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.001+0000 7fc5f4993640 1 --2- 172.21.15.1:0/172903100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f0152650 0x7fc5f0155c90 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec009870 tx=0x7fc5ec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d24d698a69212c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.001+0000 7fc5e77fe640 1 -- 172.21.15.1:0/172903100 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec035020 con 0x7fc5f0152650 2024-03-06T07:27:09.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.001+0000 7fc5e77fe640 1 -- 172.21.15.1:0/172903100 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ec031640 con 0x7fc5f0152650 2024-03-06T07:27:09.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.001+0000 7fc5e77fe640 1 -- 172.21.15.1:0/172903100 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec02d040 con 0x7fc5f0152650 2024-03-06T07:27:09.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.002+0000 7fc5f6bf7640 1 -- 172.21.15.1:0/172903100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f0152650 msgr2=0x7fc5f0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.002+0000 7fc5f6bf7640 1 --2- 172.21.15.1:0/172903100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f0152650 0x7fc5f0155c90 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec009870 tx=0x7fc5ec02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.002+0000 7fc5f6bf7640 1 -- 172.21.15.1:0/172903100 shutdown_connections 2024-03-06T07:27:09.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.002+0000 7fc5f6bf7640 1 --2- 172.21.15.1:0/172903100 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5f00bf9d0 0x7fc5f00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.002+0000 7fc5f6bf7640 1 --2- 172.21.15.1:0/172903100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f0152650 0x7fc5f0155c90 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.002+0000 7fc5f6bf7640 1 -- 172.21.15.1:0/172903100 >> 172.21.15.1:0/172903100 conn(0x7fc5f0066ee0 msgr2=0x7fc5f00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.002+0000 7fc5f6bf7640 1 -- 172.21.15.1:0/172903100 shutdown_connections 2024-03-06T07:27:09.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.002+0000 7fc5f6bf7640 1 -- 172.21.15.1:0/172903100 wait complete. 2024-03-06T07:27:09.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.003+0000 7fc5f6bf7640 1 Processor -- start 2024-03-06T07:27:09.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.003+0000 7fc5f6bf7640 1 -- start start 2024-03-06T07:27:09.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5f6bf7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5f00bf9d0 0x7fc5f00c5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5f6bf7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f0152650 0x7fc5f014a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5f6bf7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5f014e080 0x7fc5f014a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5f6bf7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f01551b0 con 0x7fc5f0152650 2024-03-06T07:27:09.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5f6bf7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f00c6fa0 con 0x7fc5f014e080 2024-03-06T07:27:09.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5f6bf7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f0155490 con 0x7fc5f00bf9d0 2024-03-06T07:27:09.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5e7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f0152650 0x7fc5f014a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5e7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f0152650 0x7fc5f014a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59488/0 (socket says 172.21.15.1:59488) 2024-03-06T07:27:09.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5e7fff640 1 -- 172.21.15.1:0/60058315 learned_addr learned my addr 172.21.15.1:0/60058315 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:09.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5f4993640 1 --2- 172.21.15.1:0/60058315 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5f00bf9d0 0x7fc5f00c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5f5194640 1 --2- 172.21.15.1:0/60058315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5f014e080 0x7fc5f014a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5e7fff640 1 -- 172.21.15.1:0/60058315 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5f00bf9d0 msgr2=0x7fc5f00c5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5e7fff640 1 --2- 172.21.15.1:0/60058315 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5f00bf9d0 0x7fc5f00c5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5e7fff640 1 -- 172.21.15.1:0/60058315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5f014e080 msgr2=0x7fc5f014a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5e7fff640 1 --2- 172.21.15.1:0/60058315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5f014e080 0x7fc5f014a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5e7fff640 1 -- 172.21.15.1:0/60058315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f0151c30 con 0x7fc5f0152650 2024-03-06T07:27:09.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5f4993640 1 --2- 172.21.15.1:0/60058315 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5f00bf9d0 0x7fc5f00c5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:09.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.004+0000 7fc5e7fff640 1 --2- 172.21.15.1:0/60058315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f0152650 0x7fc5f014a040 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000c430 tx=0x7fc5e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.005+0000 7fc5e57fa640 1 -- 172.21.15.1:0/60058315 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e00074e0 con 0x7fc5f0152650 2024-03-06T07:27:09.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.005+0000 7fc5e57fa640 1 -- 172.21.15.1:0/60058315 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e0007680 con 0x7fc5f0152650 2024-03-06T07:27:09.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.005+0000 7fc5f6bf7640 1 -- 172.21.15.1:0/60058315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f014d370 con 0x7fc5f0152650 2024-03-06T07:27:09.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.005+0000 7fc5e57fa640 1 -- 172.21.15.1:0/60058315 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e0004a00 con 0x7fc5f0152650 2024-03-06T07:27:09.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.005+0000 7fc5f6bf7640 1 -- 172.21.15.1:0/60058315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f014d930 con 0x7fc5f0152650 2024-03-06T07:27:09.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.006+0000 7fc5f6bf7640 1 -- 172.21.15.1:0/60058315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f0067fd0 con 0x7fc5f0152650 2024-03-06T07:27:09.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.006+0000 7fc5e57fa640 1 -- 172.21.15.1:0/60058315 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5e001a020 con 0x7fc5f0152650 2024-03-06T07:27:09.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.006+0000 7fc5e57fa640 1 --2- 172.21.15.1:0/60058315 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5dc06b500 0x7fc5dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.007+0000 7fc5e57fa640 1 -- 172.21.15.1:0/60058315 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc5e0089b50 con 0x7fc5f0152650 2024-03-06T07:27:09.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.007+0000 7fc5f4993640 1 --2- 172.21.15.1:0/60058315 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5dc06b500 0x7fc5dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.007+0000 7fc5f4993640 1 --2- 172.21.15.1:0/60058315 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5dc06b500 0x7fc5dc06d9c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec036000 tx=0x7fc5ec03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.015+0000 7fc5e57fa640 1 -- 172.21.15.1:0/60058315 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5e0058840 con 0x7fc5f0152650 2024-03-06T07:27:09.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.192+0000 7fc5f6bf7640 1 -- 172.21.15.1:0/60058315 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc5f01519e0 con 0x7fc5dc06b500 2024-03-06T07:27:09.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.198+0000 7fc5e57fa640 1 -- 172.21.15.1:0/60058315 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247159 (secure 0 0 0) 0x7fc5f01519e0 con 0x7fc5dc06b500 2024-03-06T07:27:09.202 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:09.204 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:27:09.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.201+0000 7fc5f6bf7640 1 -- 172.21.15.1:0/60058315 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5dc06b500 msgr2=0x7fc5dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.201+0000 7fc5f6bf7640 1 --2- 172.21.15.1:0/60058315 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5dc06b500 0x7fc5dc06d9c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec036000 tx=0x7fc5ec03c040 comp rx=0 tx=0).stop 2024-03-06T07:27:09.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.201+0000 7fc5f6bf7640 1 -- 172.21.15.1:0/60058315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f0152650 msgr2=0x7fc5f014a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.201+0000 7fc5f6bf7640 1 --2- 172.21.15.1:0/60058315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f0152650 0x7fc5f014a040 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000c430 tx=0x7fc5e000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.201+0000 7fc5f6bf7640 1 -- 172.21.15.1:0/60058315 shutdown_connections 2024-03-06T07:27:09.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.201+0000 7fc5f6bf7640 1 --2- 172.21.15.1:0/60058315 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc5dc06b500 0x7fc5dc06d9c0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.201+0000 7fc5f6bf7640 1 --2- 172.21.15.1:0/60058315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5f014e080 0x7fc5f014a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.201+0000 7fc5f6bf7640 1 --2- 172.21.15.1:0/60058315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f0152650 0x7fc5f014a040 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.201+0000 7fc5f6bf7640 1 --2- 172.21.15.1:0/60058315 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5f00bf9d0 0x7fc5f00c5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.201+0000 7fc5f6bf7640 1 -- 172.21.15.1:0/60058315 >> 172.21.15.1:0/60058315 conn(0x7fc5f0066ee0 msgr2=0x7fc5f00c3cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.201+0000 7fc5f6bf7640 1 -- 172.21.15.1:0/60058315 shutdown_connections 2024-03-06T07:27:09.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.201+0000 7fc5f6bf7640 1 -- 172.21.15.1:0/60058315 wait complete. 2024-03-06T07:27:09.214 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-03-06T07:27:08.784835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":266240,"kb_used_data":16784,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754708480,"statfs":{"total":773094113280,"available":772821483520,"internally_reserved":0,"allocated":17186816,"data_stored":9494844,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.533351"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254547+0000","last_change":"2024-03-06T07:27:04.254547+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254547+0000","last_undegraded":"2024-03-06T07:27:04.254547+0000","last_fullsized":"2024-03-06T07:27:04.254547+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.255495+0000","last_change":"2024-03-06T07:27:04.255495+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.255495+0000","last_undegraded":"2024-03-06T07:27:04.255495+0000","last_fullsized":"2024-03-06T07:27:04.255495+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259837+0000","last_change":"2024-03-06T07:27:04.259837+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259837+0000","last_undegraded":"2024-03-06T07:27:04.259837+0000","last_fullsized":"2024-03-06T07:27:04.259837+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259747+0000","last_change":"2024-03-06T07:27:04.259747+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259747+0000","last_undegraded":"2024-03-06T07:27:04.259747+0000","last_fullsized":"2024-03-06T07:27:04.259747+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258205+0000","last_change":"2024-03-06T07:27:04.258205+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258205+0000","last_undegraded":"2024-03-06T07:27:04.258205+0000","last_fullsized":"2024-03-06T07:27:04.258205+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259513+0000","last_change":"2024-03-06T07:27:04.259513+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259513+0000","last_undegraded":"2024-03-06T07:27:04.259513+0000","last_fullsized":"2024-03-06T07:27:04.259513+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257656+0000","last_change":"2024-03-06T07:27:04.257656+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257656+0000","last_undegraded":"2024-03-06T07:27:04.257656+0000","last_fullsized":"2024-03-06T07:27:04.257656+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257414+0000","last_change":"2024-03-06T07:27:04.257414+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257414+0000","last_undegraded":"2024-03-06T07:27:04.257414+0000","last_fullsized":"2024-03-06T07:27:04.257414+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258740+0000","last_change":"2024-03-06T07:27:04.258740+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258740+0000","last_undegraded":"2024-03-06T07:27:04.258740+0000","last_fullsized":"2024-03-06T07:27:04.258740+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258528+0000","last_change":"2024-03-06T07:27:04.258528+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258528+0000","last_undegraded":"2024-03-06T07:27:04.258528+0000","last_fullsized":"2024-03-06T07:27:04.258528+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259375+0000","last_change":"2024-03-06T07:27:04.259375+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259375+0000","last_undegraded":"2024-03-06T07:27:04.259375+0000","last_fullsized":"2024-03-06T07:27:04.259375+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257246+0000","last_change":"2024-03-06T07:27:04.257246+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257246+0000","last_undegraded":"2024-03-06T07:27:04.257246+0000","last_fullsized":"2024-03-06T07:27:04.257246+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259445+0000","last_change":"2024-03-06T07:27:04.259445+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259445+0000","last_undegraded":"2024-03-06T07:27:04.259445+0000","last_fullsized":"2024-03-06T07:27:04.259445+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257927+0000","last_change":"2024-03-06T07:27:04.257927+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257927+0000","last_undegraded":"2024-03-06T07:27:04.257927+0000","last_fullsized":"2024-03-06T07:27:04.257927+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.263169+0000","last_change":"2024-03-06T07:27:04.263169+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.263169+0000","last_undegraded":"2024-03-06T07:27:04.263169+0000","last_fullsized":"2024-03-06T07:27:04.263169+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257264+0000","last_change":"2024-03-06T07:27:04.257264+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257264+0000","last_undegraded":"2024-03-06T07:27:04.257264+0000","last_fullsized":"2024-03-06T07:27:04.257264+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258290+0000","last_change":"2024-03-06T07:27:04.258290+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258290+0000","last_undegraded":"2024-03-06T07:27:04.258290+0000","last_fullsized":"2024-03-06T07:27:04.258290+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258702+0000","last_change":"2024-03-06T07:27:04.258702+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258702+0000","last_undegraded":"2024-03-06T07:27:04.258702+0000","last_fullsized":"2024-03-06T07:27:04.258702+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258065+0000","last_change":"2024-03-06T07:27:04.258065+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258065+0000","last_undegraded":"2024-03-06T07:27:04.258065+0000","last_fullsized":"2024-03-06T07:27:04.258065+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258945+0000","last_change":"2024-03-06T07:27:04.258945+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258945+0000","last_undegraded":"2024-03-06T07:27:04.258945+0000","last_fullsized":"2024-03-06T07:27:04.258945+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257755+0000","last_change":"2024-03-06T07:27:04.257755+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257755+0000","last_undegraded":"2024-03-06T07:27:04.257755+0000","last_fullsized":"2024-03-06T07:27:04.257755+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257343+0000","last_change":"2024-03-06T07:27:04.257343+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257343+0000","last_undegraded":"2024-03-06T07:27:04.257343+0000","last_fullsized":"2024-03-06T07:27:04.257343+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.253999+0000","last_change":"2024-03-06T07:27:04.253999+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.253999+0000","last_undegraded":"2024-03-06T07:27:04.253999+0000","last_fullsized":"2024-03-06T07:27:04.253999+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256789+0000","last_change":"2024-03-06T07:27:04.256789+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256789+0000","last_undegraded":"2024-03-06T07:27:04.256789+0000","last_fullsized":"2024-03-06T07:27:04.256789+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257184+0000","last_change":"2024-03-06T07:27:04.257184+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257184+0000","last_undegraded":"2024-03-06T07:27:04.257184+0000","last_fullsized":"2024-03-06T07:27:04.257184+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257217+0000","last_change":"2024-03-06T07:27:04.257217+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257217+0000","last_undegraded":"2024-03-06T07:27:04.257217+0000","last_fullsized":"2024-03-06T07:27:04.257217+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259006+0000","last_change":"2024-03-06T07:27:04.259006+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259006+0000","last_undegraded":"2024-03-06T07:27:04.259006+0000","last_fullsized":"2024-03-06T07:27:04.259006+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.255903+0000","last_change":"2024-03-06T07:27:04.255903+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.255903+0000","last_undegraded":"2024-03-06T07:27:04.255903+0000","last_fullsized":"2024-03-06T07:27:04.255903+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254435+0000","last_change":"2024-03-06T07:27:04.254435+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254435+0000","last_undegraded":"2024-03-06T07:27:04.254435+0000","last_fullsized":"2024-03-06T07:27:04.254435+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.264763+0000","last_change":"2024-03-06T07:27:04.264763+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.264763+0000","last_undegraded":"2024-03-06T07:27:04.264763+0000","last_fullsized":"2024-03-06T07:27:04.264763+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261771+0000","last_change":"2024-03-06T07:27:04.261771+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261771+0000","last_undegraded":"2024-03-06T07:27:04.261771+0000","last_fullsized":"2024-03-06T07:27:04.261771+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.260482+0000","last_change":"2024-03-06T07:27:04.260482+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.260482+0000","last_undegraded":"2024-03-06T07:27:04.260482+0000","last_fullsized":"2024-03-06T07:27:04.260482+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254981+0000","last_change":"2024-03-06T07:27:04.254981+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254981+0000","last_undegraded":"2024-03-06T07:27:04.254981+0000","last_fullsized":"2024-03-06T07:27:04.254981+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.278552+0000","last_change":"2024-03-06T07:27:04.271712+0000","last_active":"2024-03-06T07:27:04.278552+0000","last_peered":"2024-03-06T07:27:04.278552+0000","last_clean":"2024-03-06T07:27:04.278552+0000","last_became_active":"2024-03-06T07:27:04.271429+0000","last_became_peered":"2024-03-06T07:27:04.271429+0000","last_unstale":"2024-03-06T07:27:04.278552+0000","last_undegraded":"2024-03-06T07:27:04.278552+0000","last_fullsized":"2024-03-06T07:27:04.278552+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:20.726754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257054+0000","last_change":"2024-03-06T07:27:04.257054+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257054+0000","last_undegraded":"2024-03-06T07:27:04.257054+0000","last_fullsized":"2024-03-06T07:27:04.257054+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.268329+0000","last_change":"2024-03-06T07:27:04.266276+0000","last_active":"2024-03-06T07:27:04.268329+0000","last_peered":"2024-03-06T07:27:04.268329+0000","last_clean":"2024-03-06T07:27:04.268329+0000","last_became_active":"2024-03-06T07:27:04.265418+0000","last_became_peered":"2024-03-06T07:27:04.265418+0000","last_unstale":"2024-03-06T07:27:04.268329+0000","last_undegraded":"2024-03-06T07:27:04.268329+0000","last_fullsized":"2024-03-06T07:27:04.268329+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:24.992967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256890+0000","last_change":"2024-03-06T07:27:04.256890+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256890+0000","last_undegraded":"2024-03-06T07:27:04.256890+0000","last_fullsized":"2024-03-06T07:27:04.256890+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267032+0000","last_change":"2024-03-06T07:27:04.262189+0000","last_active":"2024-03-06T07:27:04.267032+0000","last_peered":"2024-03-06T07:27:04.267032+0000","last_clean":"2024-03-06T07:27:04.267032+0000","last_became_active":"2024-03-06T07:27:04.261909+0000","last_became_peered":"2024-03-06T07:27:04.261909+0000","last_unstale":"2024-03-06T07:27:04.267032+0000","last_undegraded":"2024-03-06T07:27:04.267032+0000","last_fullsized":"2024-03-06T07:27:04.267032+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:15:43.228599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256537+0000","last_change":"2024-03-06T07:27:04.256537+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256537+0000","last_undegraded":"2024-03-06T07:27:04.256537+0000","last_fullsized":"2024-03-06T07:27:04.256537+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271246+0000","last_change":"2024-03-06T07:27:04.270472+0000","last_active":"2024-03-06T07:27:04.271246+0000","last_peered":"2024-03-06T07:27:04.271246+0000","last_clean":"2024-03-06T07:27:04.271246+0000","last_became_active":"2024-03-06T07:27:04.270185+0000","last_became_peered":"2024-03-06T07:27:04.270185+0000","last_unstale":"2024-03-06T07:27:04.271246+0000","last_undegraded":"2024-03-06T07:27:04.271246+0000","last_fullsized":"2024-03-06T07:27:04.271246+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:37.715400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.263186+0000","last_change":"2024-03-06T07:27:04.263186+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.263186+0000","last_undegraded":"2024-03-06T07:27:04.263186+0000","last_fullsized":"2024-03-06T07:27:04.263186+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.275596+0000","last_change":"2024-03-06T07:27:04.272491+0000","last_active":"2024-03-06T07:27:04.275596+0000","last_peered":"2024-03-06T07:27:04.275596+0000","last_clean":"2024-03-06T07:27:04.275596+0000","last_became_active":"2024-03-06T07:27:04.272190+0000","last_became_peered":"2024-03-06T07:27:04.272190+0000","last_unstale":"2024-03-06T07:27:04.275596+0000","last_undegraded":"2024-03-06T07:27:04.275596+0000","last_fullsized":"2024-03-06T07:27:04.275596+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:17.476526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.263035+0000","last_change":"2024-03-06T07:27:04.263035+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.263035+0000","last_undegraded":"2024-03-06T07:27:04.263035+0000","last_fullsized":"2024-03-06T07:27:04.263035+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.278476+0000","last_change":"2024-03-06T07:27:04.273573+0000","last_active":"2024-03-06T07:27:04.278476+0000","last_peered":"2024-03-06T07:27:04.278476+0000","last_clean":"2024-03-06T07:27:04.278476+0000","last_became_active":"2024-03-06T07:27:04.273253+0000","last_became_peered":"2024-03-06T07:27:04.273253+0000","last_unstale":"2024-03-06T07:27:04.278476+0000","last_undegraded":"2024-03-06T07:27:04.278476+0000","last_fullsized":"2024-03-06T07:27:04.278476+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:56.531270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254166+0000","last_change":"2024-03-06T07:27:04.254166+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254166+0000","last_undegraded":"2024-03-06T07:27:04.254166+0000","last_fullsized":"2024-03-06T07:27:04.254166+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.272537+0000","last_change":"2024-03-06T07:27:04.271894+0000","last_active":"2024-03-06T07:27:04.272537+0000","last_peered":"2024-03-06T07:27:04.272537+0000","last_clean":"2024-03-06T07:27:04.272537+0000","last_became_active":"2024-03-06T07:27:04.271305+0000","last_became_peered":"2024-03-06T07:27:04.271305+0000","last_unstale":"2024-03-06T07:27:04.272537+0000","last_undegraded":"2024-03-06T07:27:04.272537+0000","last_fullsized":"2024-03-06T07:27:04.272537+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:26.898488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.266335+0000","last_change":"2024-03-06T07:27:04.266335+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.266335+0000","last_undegraded":"2024-03-06T07:27:04.266335+0000","last_fullsized":"2024-03-06T07:27:04.266335+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271368+0000","last_change":"2024-03-06T07:27:04.269100+0000","last_active":"2024-03-06T07:27:04.271368+0000","last_peered":"2024-03-06T07:27:04.271368+0000","last_clean":"2024-03-06T07:27:04.271368+0000","last_became_active":"2024-03-06T07:27:04.268811+0000","last_became_peered":"2024-03-06T07:27:04.268811+0000","last_unstale":"2024-03-06T07:27:04.271368+0000","last_undegraded":"2024-03-06T07:27:04.271368+0000","last_fullsized":"2024-03-06T07:27:04.271368+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:34.615045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258161+0000","last_change":"2024-03-06T07:27:04.258161+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258161+0000","last_undegraded":"2024-03-06T07:27:04.258161+0000","last_fullsized":"2024-03-06T07:27:04.258161+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.275534+0000","last_change":"2024-03-06T07:27:04.271341+0000","last_active":"2024-03-06T07:27:04.275534+0000","last_peered":"2024-03-06T07:27:04.275534+0000","last_clean":"2024-03-06T07:27:04.275534+0000","last_became_active":"2024-03-06T07:27:04.271023+0000","last_became_peered":"2024-03-06T07:27:04.271023+0000","last_unstale":"2024-03-06T07:27:04.275534+0000","last_undegraded":"2024-03-06T07:27:04.275534+0000","last_fullsized":"2024-03-06T07:27:04.275534+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:24.173726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257145+0000","last_change":"2024-03-06T07:27:04.257145+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257145+0000","last_undegraded":"2024-03-06T07:27:04.257145+0000","last_fullsized":"2024-03-06T07:27:04.257145+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.266679+0000","last_change":"2024-03-06T07:27:04.264830+0000","last_active":"2024-03-06T07:27:04.266679+0000","last_peered":"2024-03-06T07:27:04.266679+0000","last_clean":"2024-03-06T07:27:04.266679+0000","last_became_active":"2024-03-06T07:27:04.264562+0000","last_became_peered":"2024-03-06T07:27:04.264562+0000","last_unstale":"2024-03-06T07:27:04.266679+0000","last_undegraded":"2024-03-06T07:27:04.266679+0000","last_fullsized":"2024-03-06T07:27:04.266679+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:12.146965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256102+0000","last_change":"2024-03-06T07:27:04.256102+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256102+0000","last_undegraded":"2024-03-06T07:27:04.256102+0000","last_fullsized":"2024-03-06T07:27:04.256102+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.278496+0000","last_change":"2024-03-06T07:27:04.273664+0000","last_active":"2024-03-06T07:27:04.278496+0000","last_peered":"2024-03-06T07:27:04.278496+0000","last_clean":"2024-03-06T07:27:04.278496+0000","last_became_active":"2024-03-06T07:27:04.273372+0000","last_became_peered":"2024-03-06T07:27:04.273372+0000","last_unstale":"2024-03-06T07:27:04.278496+0000","last_undegraded":"2024-03-06T07:27:04.278496+0000","last_fullsized":"2024-03-06T07:27:04.278496+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:51.519017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.255456+0000","last_change":"2024-03-06T07:27:04.255456+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.255456+0000","last_undegraded":"2024-03-06T07:27:04.255456+0000","last_fullsized":"2024-03-06T07:27:04.255456+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267158+0000","last_change":"2024-03-06T07:27:04.263075+0000","last_active":"2024-03-06T07:27:04.267158+0000","last_peered":"2024-03-06T07:27:04.267158+0000","last_clean":"2024-03-06T07:27:04.267158+0000","last_became_active":"2024-03-06T07:27:04.261822+0000","last_became_peered":"2024-03-06T07:27:04.261822+0000","last_unstale":"2024-03-06T07:27:04.267158+0000","last_undegraded":"2024-03-06T07:27:04.267158+0000","last_fullsized":"2024-03-06T07:27:04.267158+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:50.416453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258591+0000","last_change":"2024-03-06T07:27:04.258591+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258591+0000","last_undegraded":"2024-03-06T07:27:04.258591+0000","last_fullsized":"2024-03-06T07:27:04.258591+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.270193+0000","last_change":"2024-03-06T07:27:04.268027+0000","last_active":"2024-03-06T07:27:04.270193+0000","last_peered":"2024-03-06T07:27:04.270193+0000","last_clean":"2024-03-06T07:27:04.270193+0000","last_became_active":"2024-03-06T07:27:04.267726+0000","last_became_peered":"2024-03-06T07:27:04.267726+0000","last_unstale":"2024-03-06T07:27:04.270193+0000","last_undegraded":"2024-03-06T07:27:04.270193+0000","last_fullsized":"2024-03-06T07:27:04.270193+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:11.943331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254886+0000","last_change":"2024-03-06T07:27:04.254886+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254886+0000","last_undegraded":"2024-03-06T07:27:04.254886+0000","last_fullsized":"2024-03-06T07:27:04.254886+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.266762+0000","last_change":"2024-03-06T07:27:04.265676+0000","last_active":"2024-03-06T07:27:04.266762+0000","last_peered":"2024-03-06T07:27:04.266762+0000","last_clean":"2024-03-06T07:27:04.266762+0000","last_became_active":"2024-03-06T07:27:04.265374+0000","last_became_peered":"2024-03-06T07:27:04.265374+0000","last_unstale":"2024-03-06T07:27:04.266762+0000","last_undegraded":"2024-03-06T07:27:04.266762+0000","last_fullsized":"2024-03-06T07:27:04.266762+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:30.199858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261058+0000","last_change":"2024-03-06T07:27:04.261058+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261058+0000","last_undegraded":"2024-03-06T07:27:04.261058+0000","last_fullsized":"2024-03-06T07:27:04.261058+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271352+0000","last_change":"2024-03-06T07:27:04.269119+0000","last_active":"2024-03-06T07:27:04.271352+0000","last_peered":"2024-03-06T07:27:04.271352+0000","last_clean":"2024-03-06T07:27:04.271352+0000","last_became_active":"2024-03-06T07:27:04.268844+0000","last_became_peered":"2024-03-06T07:27:04.268844+0000","last_unstale":"2024-03-06T07:27:04.271352+0000","last_undegraded":"2024-03-06T07:27:04.271352+0000","last_fullsized":"2024-03-06T07:27:04.271352+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:47.701531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256965+0000","last_change":"2024-03-06T07:27:04.256965+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256965+0000","last_undegraded":"2024-03-06T07:27:04.256965+0000","last_fullsized":"2024-03-06T07:27:04.256965+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.268206+0000","last_change":"2024-03-06T07:27:04.262327+0000","last_active":"2024-03-06T07:27:04.268206+0000","last_peered":"2024-03-06T07:27:04.268206+0000","last_clean":"2024-03-06T07:27:04.268206+0000","last_became_active":"2024-03-06T07:27:04.262046+0000","last_became_peered":"2024-03-06T07:27:04.262046+0000","last_unstale":"2024-03-06T07:27:04.268206+0000","last_undegraded":"2024-03-06T07:27:04.268206+0000","last_fullsized":"2024-03-06T07:27:04.268206+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:31.006078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256647+0000","last_change":"2024-03-06T07:27:04.256647+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256647+0000","last_undegraded":"2024-03-06T07:27:04.256647+0000","last_fullsized":"2024-03-06T07:27:04.256647+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.268350+0000","last_change":"2024-03-06T07:27:04.266893+0000","last_active":"2024-03-06T07:27:04.268350+0000","last_peered":"2024-03-06T07:27:04.268350+0000","last_clean":"2024-03-06T07:27:04.268350+0000","last_became_active":"2024-03-06T07:27:04.265563+0000","last_became_peered":"2024-03-06T07:27:04.265563+0000","last_unstale":"2024-03-06T07:27:04.268350+0000","last_undegraded":"2024-03-06T07:27:04.268350+0000","last_fullsized":"2024-03-06T07:27:04.268350+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:35.022673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254208+0000","last_change":"2024-03-06T07:27:04.254208+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254208+0000","last_undegraded":"2024-03-06T07:27:04.254208+0000","last_fullsized":"2024-03-06T07:27:04.254208+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267134+0000","last_change":"2024-03-06T07:27:04.265277+0000","last_active":"2024-03-06T07:27:04.267134+0000","last_peered":"2024-03-06T07:27:04.267134+0000","last_clean":"2024-03-06T07:27:04.267134+0000","last_became_active":"2024-03-06T07:27:04.265018+0000","last_became_peered":"2024-03-06T07:27:04.265018+0000","last_unstale":"2024-03-06T07:27:04.267134+0000","last_undegraded":"2024-03-06T07:27:04.267134+0000","last_fullsized":"2024-03-06T07:27:04.267134+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:51.929743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257320+0000","last_change":"2024-03-06T07:27:04.257320+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257320+0000","last_undegraded":"2024-03-06T07:27:04.257320+0000","last_fullsized":"2024-03-06T07:27:04.257320+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267517+0000","last_change":"2024-03-06T07:27:04.265842+0000","last_active":"2024-03-06T07:27:04.267517+0000","last_peered":"2024-03-06T07:27:04.267517+0000","last_clean":"2024-03-06T07:27:04.267517+0000","last_became_active":"2024-03-06T07:27:04.265568+0000","last_became_peered":"2024-03-06T07:27:04.265568+0000","last_unstale":"2024-03-06T07:27:04.267517+0000","last_undegraded":"2024-03-06T07:27:04.267517+0000","last_fullsized":"2024-03-06T07:27:04.267517+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:31.100594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254138+0000","last_change":"2024-03-06T07:27:04.254138+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254138+0000","last_undegraded":"2024-03-06T07:27:04.254138+0000","last_fullsized":"2024-03-06T07:27:04.254138+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.264364+0000","last_change":"2024-03-06T07:27:04.262729+0000","last_active":"2024-03-06T07:27:04.264364+0000","last_peered":"2024-03-06T07:27:04.264364+0000","last_clean":"2024-03-06T07:27:04.264364+0000","last_became_active":"2024-03-06T07:27:04.262484+0000","last_became_peered":"2024-03-06T07:27:04.262484+0000","last_unstale":"2024-03-06T07:27:04.264364+0000","last_undegraded":"2024-03-06T07:27:04.264364+0000","last_fullsized":"2024-03-06T07:27:04.264364+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:38.945976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261717+0000","last_change":"2024-03-06T07:27:04.261717+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261717+0000","last_undegraded":"2024-03-06T07:27:04.261717+0000","last_fullsized":"2024-03-06T07:27:04.261717+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267085+0000","last_change":"2024-03-06T07:27:04.263995+0000","last_active":"2024-03-06T07:27:04.267085+0000","last_peered":"2024-03-06T07:27:04.267085+0000","last_clean":"2024-03-06T07:27:04.267085+0000","last_became_active":"2024-03-06T07:27:04.263629+0000","last_became_peered":"2024-03-06T07:27:04.263629+0000","last_unstale":"2024-03-06T07:27:04.267085+0000","last_undegraded":"2024-03-06T07:27:04.267085+0000","last_fullsized":"2024-03-06T07:27:04.267085+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:44.864909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254160+0000","last_change":"2024-03-06T07:27:04.254160+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254160+0000","last_undegraded":"2024-03-06T07:27:04.254160+0000","last_fullsized":"2024-03-06T07:27:04.254160+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.278500+0000","last_change":"2024-03-06T07:27:04.271282+0000","last_active":"2024-03-06T07:27:04.278500+0000","last_peered":"2024-03-06T07:27:04.278500+0000","last_clean":"2024-03-06T07:27:04.278500+0000","last_became_active":"2024-03-06T07:27:04.270987+0000","last_became_peered":"2024-03-06T07:27:04.270987+0000","last_unstale":"2024-03-06T07:27:04.278500+0000","last_undegraded":"2024-03-06T07:27:04.278500+0000","last_fullsized":"2024-03-06T07:27:04.278500+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:02.098293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258625+0000","last_change":"2024-03-06T07:27:04.258625+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258625+0000","last_undegraded":"2024-03-06T07:27:04.258625+0000","last_fullsized":"2024-03-06T07:27:04.258625+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.275695+0000","last_change":"2024-03-06T07:27:04.272151+0000","last_active":"2024-03-06T07:27:04.275695+0000","last_peered":"2024-03-06T07:27:04.275695+0000","last_clean":"2024-03-06T07:27:04.275695+0000","last_became_active":"2024-03-06T07:27:04.271894+0000","last_became_peered":"2024-03-06T07:27:04.271894+0000","last_unstale":"2024-03-06T07:27:04.275695+0000","last_undegraded":"2024-03-06T07:27:04.275695+0000","last_fullsized":"2024-03-06T07:27:04.275695+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:50.049060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256530+0000","last_change":"2024-03-06T07:27:04.256530+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256530+0000","last_undegraded":"2024-03-06T07:27:04.256530+0000","last_fullsized":"2024-03-06T07:27:04.256530+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.270258+0000","last_change":"2024-03-06T07:27:04.268865+0000","last_active":"2024-03-06T07:27:04.270258+0000","last_peered":"2024-03-06T07:27:04.270258+0000","last_clean":"2024-03-06T07:27:04.270258+0000","last_became_active":"2024-03-06T07:27:04.268566+0000","last_became_peered":"2024-03-06T07:27:04.268566+0000","last_unstale":"2024-03-06T07:27:04.270258+0000","last_undegraded":"2024-03-06T07:27:04.270258+0000","last_fullsized":"2024-03-06T07:27:04.270258+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:05.786807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257744+0000","last_change":"2024-03-06T07:27:04.257744+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257744+0000","last_undegraded":"2024-03-06T07:27:04.257744+0000","last_fullsized":"2024-03-06T07:27:04.257744+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.270471+0000","last_change":"2024-03-06T07:27:04.264784+0000","last_active":"2024-03-06T07:27:04.270471+0000","last_peered":"2024-03-06T07:27:04.270471+0000","last_clean":"2024-03-06T07:27:04.270471+0000","last_became_active":"2024-03-06T07:27:04.264499+0000","last_became_peered":"2024-03-06T07:27:04.264499+0000","last_unstale":"2024-03-06T07:27:04.270471+0000","last_undegraded":"2024-03-06T07:27:04.270471+0000","last_fullsized":"2024-03-06T07:27:04.270471+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:38.542889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254229+0000","last_change":"2024-03-06T07:27:04.254229+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254229+0000","last_undegraded":"2024-03-06T07:27:04.254229+0000","last_fullsized":"2024-03-06T07:27:04.254229+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.275452+0000","last_change":"2024-03-06T07:27:04.269541+0000","last_active":"2024-03-06T07:27:04.275452+0000","last_peered":"2024-03-06T07:27:04.275452+0000","last_clean":"2024-03-06T07:27:04.275452+0000","last_became_active":"2024-03-06T07:27:04.269228+0000","last_became_peered":"2024-03-06T07:27:04.269228+0000","last_unstale":"2024-03-06T07:27:04.275452+0000","last_undegraded":"2024-03-06T07:27:04.275452+0000","last_fullsized":"2024-03-06T07:27:04.275452+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:05.198388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254889+0000","last_change":"2024-03-06T07:27:04.254889+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254889+0000","last_undegraded":"2024-03-06T07:27:04.254889+0000","last_fullsized":"2024-03-06T07:27:04.254889+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.264329+0000","last_change":"2024-03-06T07:27:04.260635+0000","last_active":"2024-03-06T07:27:04.264329+0000","last_peered":"2024-03-06T07:27:04.264329+0000","last_clean":"2024-03-06T07:27:04.264329+0000","last_became_active":"2024-03-06T07:27:04.260370+0000","last_became_peered":"2024-03-06T07:27:04.260370+0000","last_unstale":"2024-03-06T07:27:04.264329+0000","last_undegraded":"2024-03-06T07:27:04.264329+0000","last_fullsized":"2024-03-06T07:27:04.264329+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:26.449033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257795+0000","last_change":"2024-03-06T07:27:04.257795+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257795+0000","last_undegraded":"2024-03-06T07:27:04.257795+0000","last_fullsized":"2024-03-06T07:27:04.257795+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.272959+0000","last_change":"2024-03-06T07:27:04.269073+0000","last_active":"2024-03-06T07:27:04.272959+0000","last_peered":"2024-03-06T07:27:04.272959+0000","last_clean":"2024-03-06T07:27:04.272959+0000","last_became_active":"2024-03-06T07:27:04.268831+0000","last_became_peered":"2024-03-06T07:27:04.268831+0000","last_unstale":"2024-03-06T07:27:04.272959+0000","last_undegraded":"2024-03-06T07:27:04.272959+0000","last_fullsized":"2024-03-06T07:27:04.272959+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:30.526595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261310+0000","last_change":"2024-03-06T07:27:04.261310+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261310+0000","last_undegraded":"2024-03-06T07:27:04.261310+0000","last_fullsized":"2024-03-06T07:27:04.261310+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271907+0000","last_change":"2024-03-06T07:27:04.271202+0000","last_active":"2024-03-06T07:27:04.271907+0000","last_peered":"2024-03-06T07:27:04.271907+0000","last_clean":"2024-03-06T07:27:04.271907+0000","last_became_active":"2024-03-06T07:27:04.270881+0000","last_became_peered":"2024-03-06T07:27:04.270881+0000","last_unstale":"2024-03-06T07:27:04.271907+0000","last_undegraded":"2024-03-06T07:27:04.271907+0000","last_fullsized":"2024-03-06T07:27:04.271907+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:17.006730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261395+0000","last_change":"2024-03-06T07:27:04.261395+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261395+0000","last_undegraded":"2024-03-06T07:27:04.261395+0000","last_fullsized":"2024-03-06T07:27:04.261395+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.273065+0000","last_change":"2024-03-06T07:27:04.269025+0000","last_active":"2024-03-06T07:27:04.273065+0000","last_peered":"2024-03-06T07:27:04.273065+0000","last_clean":"2024-03-06T07:27:04.273065+0000","last_became_active":"2024-03-06T07:27:04.268768+0000","last_became_peered":"2024-03-06T07:27:04.268768+0000","last_unstale":"2024-03-06T07:27:04.273065+0000","last_undegraded":"2024-03-06T07:27:04.273065+0000","last_fullsized":"2024-03-06T07:27:04.273065+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:28.810603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.260194+0000","last_change":"2024-03-06T07:27:04.260194+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.260194+0000","last_undegraded":"2024-03-06T07:27:04.260194+0000","last_fullsized":"2024-03-06T07:27:04.260194+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271722+0000","last_change":"2024-03-06T07:27:04.269800+0000","last_active":"2024-03-06T07:27:04.271722+0000","last_peered":"2024-03-06T07:27:04.271722+0000","last_clean":"2024-03-06T07:27:04.271722+0000","last_became_active":"2024-03-06T07:27:04.269489+0000","last_became_peered":"2024-03-06T07:27:04.269489+0000","last_unstale":"2024-03-06T07:27:04.271722+0000","last_undegraded":"2024-03-06T07:27:04.271722+0000","last_fullsized":"2024-03-06T07:27:04.271722+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:58.666333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254142+0000","last_change":"2024-03-06T07:27:04.254142+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254142+0000","last_undegraded":"2024-03-06T07:27:04.254142+0000","last_fullsized":"2024-03-06T07:27:04.254142+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.259397+0000","last_change":"2024-03-06T07:27:04.258208+0000","last_active":"2024-03-06T07:27:04.259397+0000","last_peered":"2024-03-06T07:27:04.259397+0000","last_clean":"2024-03-06T07:27:04.259397+0000","last_became_active":"2024-03-06T07:27:04.257866+0000","last_became_peered":"2024-03-06T07:27:04.257866+0000","last_unstale":"2024-03-06T07:27:04.259397+0000","last_undegraded":"2024-03-06T07:27:04.259397+0000","last_fullsized":"2024-03-06T07:27:04.259397+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:34.006302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":456,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.262911+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:27:04.262911+0000","last_peered":"2024-03-06T07:27:04.262911+0000","last_clean":"2024-03-06T07:27:04.262911+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:27:04.262911+0000","last_undegraded":"2024-03-06T07:27:04.262911+0000","last_fullsized":"2024-03-06T07:27:04.262911+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33320,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338520,"statfs":{"total":96636764160,"available":96602644480,"internally_reserved":0,"allocated":2191360,"data_stored":1626298,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574932,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1042597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1042597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1035930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574932,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1042597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1042597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1035930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574933,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33320,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338520,"statfs":{"total":96636764160,"available":96602644480,"internally_reserved":0,"allocated":2191360,"data_stored":1626298,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:09.217 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.3f is not active+clean 2024-03-06T07:27:09.217 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.254547+0000', 'last_change': '2024-03-06T07:27:04.254547+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.254547+0000', 'last_undegraded': '2024-03-06T07:27:04.254547+0000', 'last_fullsized': '2024-03-06T07:27:04.254547+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:09.217 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.3e is not active+clean 2024-03-06T07:27:09.217 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.255495+0000', 'last_change': '2024-03-06T07:27:04.255495+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.255495+0000', 'last_undegraded': '2024-03-06T07:27:04.255495+0000', 'last_fullsized': '2024-03-06T07:27:04.255495+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:09.218 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.3d is not active+clean 2024-03-06T07:27:09.218 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.259837+0000', 'last_change': '2024-03-06T07:27:04.259837+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.259837+0000', 'last_undegraded': '2024-03-06T07:27:04.259837+0000', 'last_fullsized': '2024-03-06T07:27:04.259837+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:09.218 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.3c is not active+clean 2024-03-06T07:27:09.218 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.259747+0000', 'last_change': '2024-03-06T07:27:04.259747+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.259747+0000', 'last_undegraded': '2024-03-06T07:27:04.259747+0000', 'last_fullsized': '2024-03-06T07:27:04.259747+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:09.218 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.3b is not active+clean 2024-03-06T07:27:09.218 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.258205+0000', 'last_change': '2024-03-06T07:27:04.258205+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.258205+0000', 'last_undegraded': '2024-03-06T07:27:04.258205+0000', 'last_fullsized': '2024-03-06T07:27:04.258205+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:09.218 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.3a is not active+clean 2024-03-06T07:27:09.218 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.259513+0000', 'last_change': '2024-03-06T07:27:04.259513+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.259513+0000', 'last_undegraded': '2024-03-06T07:27:04.259513+0000', 'last_fullsized': '2024-03-06T07:27:04.259513+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:09.218 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.39 is not active+clean 2024-03-06T07:27:09.218 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.257656+0000', 'last_change': '2024-03-06T07:27:04.257656+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.257656+0000', 'last_undegraded': '2024-03-06T07:27:04.257656+0000', 'last_fullsized': '2024-03-06T07:27:04.257656+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:09.218 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.38 is not active+clean 2024-03-06T07:27:09.218 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.257414+0000', 'last_change': '2024-03-06T07:27:04.257414+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.257414+0000', 'last_undegraded': '2024-03-06T07:27:04.257414+0000', 'last_fullsized': '2024-03-06T07:27:04.257414+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:09.219 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.37 is not active+clean 2024-03-06T07:27:09.219 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.258740+0000', 'last_change': '2024-03-06T07:27:04.258740+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.258740+0000', 'last_undegraded': '2024-03-06T07:27:04.258740+0000', 'last_fullsized': '2024-03-06T07:27:04.258740+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:09.219 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.36 is not active+clean 2024-03-06T07:27:09.219 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.258528+0000', 'last_change': '2024-03-06T07:27:04.258528+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.258528+0000', 'last_undegraded': '2024-03-06T07:27:04.258528+0000', 'last_fullsized': '2024-03-06T07:27:04.258528+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:09.219 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.35 is not active+clean 2024-03-06T07:27:09.219 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.259375+0000', 'last_change': '2024-03-06T07:27:04.259375+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.259375+0000', 'last_undegraded': '2024-03-06T07:27:04.259375+0000', 'last_fullsized': '2024-03-06T07:27:04.259375+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:09.219 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.34 is not active+clean 2024-03-06T07:27:09.219 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.257246+0000', 'last_change': '2024-03-06T07:27:04.257246+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.257246+0000', 'last_undegraded': '2024-03-06T07:27:04.257246+0000', 'last_fullsized': '2024-03-06T07:27:04.257246+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:09.219 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.33 is not active+clean 2024-03-06T07:27:09.219 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.259445+0000', 'last_change': '2024-03-06T07:27:04.259445+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.259445+0000', 'last_undegraded': '2024-03-06T07:27:04.259445+0000', 'last_fullsized': '2024-03-06T07:27:04.259445+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:09.219 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.32 is not active+clean 2024-03-06T07:27:09.219 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.257927+0000', 'last_change': '2024-03-06T07:27:04.257927+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.257927+0000', 'last_undegraded': '2024-03-06T07:27:04.257927+0000', 'last_fullsized': '2024-03-06T07:27:04.257927+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:09.219 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.31 is not active+clean 2024-03-06T07:27:09.220 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.263169+0000', 'last_change': '2024-03-06T07:27:04.263169+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.263169+0000', 'last_undegraded': '2024-03-06T07:27:04.263169+0000', 'last_fullsized': '2024-03-06T07:27:04.263169+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:09.220 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.30 is not active+clean 2024-03-06T07:27:09.220 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.257264+0000', 'last_change': '2024-03-06T07:27:04.257264+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.257264+0000', 'last_undegraded': '2024-03-06T07:27:04.257264+0000', 'last_fullsized': '2024-03-06T07:27:04.257264+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:09.220 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.2f is not active+clean 2024-03-06T07:27:09.220 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.258290+0000', 'last_change': '2024-03-06T07:27:04.258290+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.258290+0000', 'last_undegraded': '2024-03-06T07:27:04.258290+0000', 'last_fullsized': '2024-03-06T07:27:04.258290+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:09.220 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.2e is not active+clean 2024-03-06T07:27:09.220 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.258702+0000', 'last_change': '2024-03-06T07:27:04.258702+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.258702+0000', 'last_undegraded': '2024-03-06T07:27:04.258702+0000', 'last_fullsized': '2024-03-06T07:27:04.258702+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:09.220 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.2d is not active+clean 2024-03-06T07:27:09.220 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.258065+0000', 'last_change': '2024-03-06T07:27:04.258065+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.258065+0000', 'last_undegraded': '2024-03-06T07:27:04.258065+0000', 'last_fullsized': '2024-03-06T07:27:04.258065+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:09.220 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.2c is not active+clean 2024-03-06T07:27:09.220 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.258945+0000', 'last_change': '2024-03-06T07:27:04.258945+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.258945+0000', 'last_undegraded': '2024-03-06T07:27:04.258945+0000', 'last_fullsized': '2024-03-06T07:27:04.258945+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:09.220 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.2b is not active+clean 2024-03-06T07:27:09.220 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.257755+0000', 'last_change': '2024-03-06T07:27:04.257755+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.257755+0000', 'last_undegraded': '2024-03-06T07:27:04.257755+0000', 'last_fullsized': '2024-03-06T07:27:04.257755+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:09.221 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.2a is not active+clean 2024-03-06T07:27:09.221 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.257343+0000', 'last_change': '2024-03-06T07:27:04.257343+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.257343+0000', 'last_undegraded': '2024-03-06T07:27:04.257343+0000', 'last_fullsized': '2024-03-06T07:27:04.257343+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:09.221 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.29 is not active+clean 2024-03-06T07:27:09.221 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.253999+0000', 'last_change': '2024-03-06T07:27:04.253999+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.253999+0000', 'last_undegraded': '2024-03-06T07:27:04.253999+0000', 'last_fullsized': '2024-03-06T07:27:04.253999+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:09.221 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.28 is not active+clean 2024-03-06T07:27:09.221 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.256789+0000', 'last_change': '2024-03-06T07:27:04.256789+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.256789+0000', 'last_undegraded': '2024-03-06T07:27:04.256789+0000', 'last_fullsized': '2024-03-06T07:27:04.256789+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:09.221 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.27 is not active+clean 2024-03-06T07:27:09.221 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.257184+0000', 'last_change': '2024-03-06T07:27:04.257184+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.257184+0000', 'last_undegraded': '2024-03-06T07:27:04.257184+0000', 'last_fullsized': '2024-03-06T07:27:04.257184+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:09.221 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.26 is not active+clean 2024-03-06T07:27:09.221 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.257217+0000', 'last_change': '2024-03-06T07:27:04.257217+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.257217+0000', 'last_undegraded': '2024-03-06T07:27:04.257217+0000', 'last_fullsized': '2024-03-06T07:27:04.257217+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:09.221 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.25 is not active+clean 2024-03-06T07:27:09.221 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.259006+0000', 'last_change': '2024-03-06T07:27:04.259006+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.259006+0000', 'last_undegraded': '2024-03-06T07:27:04.259006+0000', 'last_fullsized': '2024-03-06T07:27:04.259006+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:09.222 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.24 is not active+clean 2024-03-06T07:27:09.222 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.255903+0000', 'last_change': '2024-03-06T07:27:04.255903+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.255903+0000', 'last_undegraded': '2024-03-06T07:27:04.255903+0000', 'last_fullsized': '2024-03-06T07:27:04.255903+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:09.222 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.23 is not active+clean 2024-03-06T07:27:09.222 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.254435+0000', 'last_change': '2024-03-06T07:27:04.254435+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.254435+0000', 'last_undegraded': '2024-03-06T07:27:04.254435+0000', 'last_fullsized': '2024-03-06T07:27:04.254435+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:09.222 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.22 is not active+clean 2024-03-06T07:27:09.222 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.264763+0000', 'last_change': '2024-03-06T07:27:04.264763+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.264763+0000', 'last_undegraded': '2024-03-06T07:27:04.264763+0000', 'last_fullsized': '2024-03-06T07:27:04.264763+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:09.222 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.21 is not active+clean 2024-03-06T07:27:09.222 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.261771+0000', 'last_change': '2024-03-06T07:27:04.261771+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.261771+0000', 'last_undegraded': '2024-03-06T07:27:04.261771+0000', 'last_fullsized': '2024-03-06T07:27:04.261771+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:09.222 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.20 is not active+clean 2024-03-06T07:27:09.222 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.260482+0000', 'last_change': '2024-03-06T07:27:04.260482+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.260482+0000', 'last_undegraded': '2024-03-06T07:27:04.260482+0000', 'last_fullsized': '2024-03-06T07:27:04.260482+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:09.222 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.18 is not active+clean 2024-03-06T07:27:09.223 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.254981+0000', 'last_change': '2024-03-06T07:27:04.254981+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.254981+0000', 'last_undegraded': '2024-03-06T07:27:04.254981+0000', 'last_fullsized': '2024-03-06T07:27:04.254981+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:09.223 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.19 is not active+clean 2024-03-06T07:27:09.223 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.257054+0000', 'last_change': '2024-03-06T07:27:04.257054+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.257054+0000', 'last_undegraded': '2024-03-06T07:27:04.257054+0000', 'last_fullsized': '2024-03-06T07:27:04.257054+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:09.223 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.1a is not active+clean 2024-03-06T07:27:09.223 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.256890+0000', 'last_change': '2024-03-06T07:27:04.256890+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.256890+0000', 'last_undegraded': '2024-03-06T07:27:04.256890+0000', 'last_fullsized': '2024-03-06T07:27:04.256890+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:09.223 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.1b is not active+clean 2024-03-06T07:27:09.223 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.256537+0000', 'last_change': '2024-03-06T07:27:04.256537+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.256537+0000', 'last_undegraded': '2024-03-06T07:27:04.256537+0000', 'last_fullsized': '2024-03-06T07:27:04.256537+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:09.223 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.1c is not active+clean 2024-03-06T07:27:09.223 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.263186+0000', 'last_change': '2024-03-06T07:27:04.263186+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.263186+0000', 'last_undegraded': '2024-03-06T07:27:04.263186+0000', 'last_fullsized': '2024-03-06T07:27:04.263186+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:09.223 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.1e is not active+clean 2024-03-06T07:27:09.223 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.263035+0000', 'last_change': '2024-03-06T07:27:04.263035+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.263035+0000', 'last_undegraded': '2024-03-06T07:27:04.263035+0000', 'last_fullsized': '2024-03-06T07:27:04.263035+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:09.223 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.1f is not active+clean 2024-03-06T07:27:09.223 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.254166+0000', 'last_change': '2024-03-06T07:27:04.254166+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.254166+0000', 'last_undegraded': '2024-03-06T07:27:04.254166+0000', 'last_fullsized': '2024-03-06T07:27:04.254166+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:09.224 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.10 is not active+clean 2024-03-06T07:27:09.224 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.266335+0000', 'last_change': '2024-03-06T07:27:04.266335+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.266335+0000', 'last_undegraded': '2024-03-06T07:27:04.266335+0000', 'last_fullsized': '2024-03-06T07:27:04.266335+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:09.224 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.11 is not active+clean 2024-03-06T07:27:09.224 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.258161+0000', 'last_change': '2024-03-06T07:27:04.258161+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.258161+0000', 'last_undegraded': '2024-03-06T07:27:04.258161+0000', 'last_fullsized': '2024-03-06T07:27:04.258161+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:09.224 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.12 is not active+clean 2024-03-06T07:27:09.224 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.257145+0000', 'last_change': '2024-03-06T07:27:04.257145+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.257145+0000', 'last_undegraded': '2024-03-06T07:27:04.257145+0000', 'last_fullsized': '2024-03-06T07:27:04.257145+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:09.224 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.13 is not active+clean 2024-03-06T07:27:09.224 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.256102+0000', 'last_change': '2024-03-06T07:27:04.256102+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.256102+0000', 'last_undegraded': '2024-03-06T07:27:04.256102+0000', 'last_fullsized': '2024-03-06T07:27:04.256102+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:09.224 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.14 is not active+clean 2024-03-06T07:27:09.224 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.255456+0000', 'last_change': '2024-03-06T07:27:04.255456+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.255456+0000', 'last_undegraded': '2024-03-06T07:27:04.255456+0000', 'last_fullsized': '2024-03-06T07:27:04.255456+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:09.224 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.15 is not active+clean 2024-03-06T07:27:09.224 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.258591+0000', 'last_change': '2024-03-06T07:27:04.258591+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.258591+0000', 'last_undegraded': '2024-03-06T07:27:04.258591+0000', 'last_fullsized': '2024-03-06T07:27:04.258591+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:09.225 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.16 is not active+clean 2024-03-06T07:27:09.225 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.254886+0000', 'last_change': '2024-03-06T07:27:04.254886+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.254886+0000', 'last_undegraded': '2024-03-06T07:27:04.254886+0000', 'last_fullsized': '2024-03-06T07:27:04.254886+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:09.225 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.17 is not active+clean 2024-03-06T07:27:09.225 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.261058+0000', 'last_change': '2024-03-06T07:27:04.261058+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.261058+0000', 'last_undegraded': '2024-03-06T07:27:04.261058+0000', 'last_fullsized': '2024-03-06T07:27:04.261058+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:09.225 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.8 is not active+clean 2024-03-06T07:27:09.225 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.256965+0000', 'last_change': '2024-03-06T07:27:04.256965+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.256965+0000', 'last_undegraded': '2024-03-06T07:27:04.256965+0000', 'last_fullsized': '2024-03-06T07:27:04.256965+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:09.225 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.9 is not active+clean 2024-03-06T07:27:09.225 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.256647+0000', 'last_change': '2024-03-06T07:27:04.256647+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.256647+0000', 'last_undegraded': '2024-03-06T07:27:04.256647+0000', 'last_fullsized': '2024-03-06T07:27:04.256647+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:09.225 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.a is not active+clean 2024-03-06T07:27:09.225 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.254208+0000', 'last_change': '2024-03-06T07:27:04.254208+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.254208+0000', 'last_undegraded': '2024-03-06T07:27:04.254208+0000', 'last_fullsized': '2024-03-06T07:27:04.254208+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:09.225 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.b is not active+clean 2024-03-06T07:27:09.225 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.257320+0000', 'last_change': '2024-03-06T07:27:04.257320+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.257320+0000', 'last_undegraded': '2024-03-06T07:27:04.257320+0000', 'last_fullsized': '2024-03-06T07:27:04.257320+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:09.226 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.c is not active+clean 2024-03-06T07:27:09.226 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.254138+0000', 'last_change': '2024-03-06T07:27:04.254138+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.254138+0000', 'last_undegraded': '2024-03-06T07:27:04.254138+0000', 'last_fullsized': '2024-03-06T07:27:04.254138+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:09.226 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.d is not active+clean 2024-03-06T07:27:09.226 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.261717+0000', 'last_change': '2024-03-06T07:27:04.261717+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.261717+0000', 'last_undegraded': '2024-03-06T07:27:04.261717+0000', 'last_fullsized': '2024-03-06T07:27:04.261717+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:09.226 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.e is not active+clean 2024-03-06T07:27:09.226 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.254160+0000', 'last_change': '2024-03-06T07:27:04.254160+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.254160+0000', 'last_undegraded': '2024-03-06T07:27:04.254160+0000', 'last_fullsized': '2024-03-06T07:27:04.254160+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:09.226 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.f is not active+clean 2024-03-06T07:27:09.226 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.258625+0000', 'last_change': '2024-03-06T07:27:04.258625+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.258625+0000', 'last_undegraded': '2024-03-06T07:27:04.258625+0000', 'last_fullsized': '2024-03-06T07:27:04.258625+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:09.226 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.0 is not active+clean 2024-03-06T07:27:09.226 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.256530+0000', 'last_change': '2024-03-06T07:27:04.256530+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.256530+0000', 'last_undegraded': '2024-03-06T07:27:04.256530+0000', 'last_fullsized': '2024-03-06T07:27:04.256530+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:09.226 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.1 is not active+clean 2024-03-06T07:27:09.226 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.257744+0000', 'last_change': '2024-03-06T07:27:04.257744+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.257744+0000', 'last_undegraded': '2024-03-06T07:27:04.257744+0000', 'last_fullsized': '2024-03-06T07:27:04.257744+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:09.227 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.2 is not active+clean 2024-03-06T07:27:09.227 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.254229+0000', 'last_change': '2024-03-06T07:27:04.254229+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.254229+0000', 'last_undegraded': '2024-03-06T07:27:04.254229+0000', 'last_fullsized': '2024-03-06T07:27:04.254229+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:09.227 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.3 is not active+clean 2024-03-06T07:27:09.227 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.254889+0000', 'last_change': '2024-03-06T07:27:04.254889+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.254889+0000', 'last_undegraded': '2024-03-06T07:27:04.254889+0000', 'last_fullsized': '2024-03-06T07:27:04.254889+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:09.227 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.4 is not active+clean 2024-03-06T07:27:09.227 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.257795+0000', 'last_change': '2024-03-06T07:27:04.257795+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.257795+0000', 'last_undegraded': '2024-03-06T07:27:04.257795+0000', 'last_fullsized': '2024-03-06T07:27:04.257795+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:09.227 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.5 is not active+clean 2024-03-06T07:27:09.227 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.261310+0000', 'last_change': '2024-03-06T07:27:04.261310+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.261310+0000', 'last_undegraded': '2024-03-06T07:27:04.261310+0000', 'last_fullsized': '2024-03-06T07:27:04.261310+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:09.227 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.6 is not active+clean 2024-03-06T07:27:09.227 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.261395+0000', 'last_change': '2024-03-06T07:27:04.261395+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.261395+0000', 'last_undegraded': '2024-03-06T07:27:04.261395+0000', 'last_fullsized': '2024-03-06T07:27:04.261395+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:09.227 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.7 is not active+clean 2024-03-06T07:27:09.227 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.260194+0000', 'last_change': '2024-03-06T07:27:04.260194+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.260194+0000', 'last_undegraded': '2024-03-06T07:27:04.260194+0000', 'last_fullsized': '2024-03-06T07:27:04.260194+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:09.228 INFO:tasks.cephfs.filesystem.ceph_manager:PG 28.1d is not active+clean 2024-03-06T07:27:09.228 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '28.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 169, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:04.254142+0000', 'last_change': '2024-03-06T07:27:04.254142+0000', 'last_active': '2024-03-06T07:27:04.246424+0000', 'last_peered': '2024-03-06T07:27:04.246424+0000', 'last_clean': '2024-03-06T07:27:04.246424+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:04.254142+0000', 'last_undegraded': '2024-03-06T07:27:04.254142+0000', 'last_fullsized': '2024-03-06T07:27:04.254142+0000', 'mapping_epoch': 169, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 169, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:04.246424+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:09.228 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:27:09.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.352+0000 7f810f7d7640 1 Processor -- start 2024-03-06T07:27:09.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.352+0000 7f810f7d7640 1 -- start start 2024-03-06T07:27:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.353+0000 7f810f7d7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81080c32f0 0x7f81081543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.353+0000 7f810f7d7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8108150900 con 0x7f8108150520 2024-03-06T07:27:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.353+0000 7f810f7d7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8108071790 con 0x7f8108070c80 2024-03-06T07:27:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.353+0000 7f810f7d7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8108153ed0 con 0x7f81080c32f0 2024-03-06T07:27:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.354+0000 7f810cd72640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81080c32f0 0x7f81081543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.354+0000 7f810d573640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8108150520 0x7f81081537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44416/0 (socket says 172.21.15.1:44416) 2024-03-06T07:27:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.354+0000 7f810d573640 1 -- 172.21.15.1:0/1977593685 learned_addr learned my addr 172.21.15.1:0/1977593685 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.354+0000 7f810cd72640 1 -- 172.21.15.1:0/1977593685 >> v1:172.21.15.98:6789/0 conn(0x7f8108070c80 legacy=0x7f810815d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.354+0000 7f810cd72640 1 -- 172.21.15.1:0/1977593685 >> v1:172.21.15.1:6789/0 conn(0x7f8108150520 legacy=0x7f81081537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.354+0000 7f80fffff640 1 -- 172.21.15.1:0/1977593685 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689568137 0 0) 0x7f8108150900 con 0x7f8108150520 2024-03-06T07:27:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.354+0000 7f810cd72640 1 -- 172.21.15.1:0/1977593685 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8108157050 con 0x7f81080c32f0 2024-03-06T07:27:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.355+0000 7f810cd72640 1 --2- 172.21.15.1:0/1977593685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81080c32f0 0x7f81081543a0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f80f80097a0 tx=0x7f80f802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d1dc7422793003e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.355+0000 7f80fffff640 1 -- 172.21.15.1:0/1977593685 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8034070 con 0x7f81080c32f0 2024-03-06T07:27:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.355+0000 7f80fffff640 1 -- 172.21.15.1:0/1977593685 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f8030610 con 0x7f81080c32f0 2024-03-06T07:27:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.355+0000 7f80fffff640 1 -- 172.21.15.1:0/1977593685 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f80309e0 con 0x7f81080c32f0 2024-03-06T07:27:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.356+0000 7f810f7d7640 1 -- 172.21.15.1:0/1977593685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81080c32f0 msgr2=0x7f81081543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.356+0000 7f810f7d7640 1 --2- 172.21.15.1:0/1977593685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81080c32f0 0x7f81081543a0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f80f80097a0 tx=0x7f80f802b960 comp rx=0 tx=0).stop 2024-03-06T07:27:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.356+0000 7f810f7d7640 1 -- 172.21.15.1:0/1977593685 shutdown_connections 2024-03-06T07:27:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.356+0000 7f810f7d7640 1 --2- 172.21.15.1:0/1977593685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81080c32f0 0x7f81081543a0 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.356+0000 7f810f7d7640 1 -- 172.21.15.1:0/1977593685 >> 172.21.15.1:0/1977593685 conn(0x7f81080c88f0 msgr2=0x7f81080c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.356+0000 7f810f7d7640 1 -- 172.21.15.1:0/1977593685 shutdown_connections 2024-03-06T07:27:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.356+0000 7f810f7d7640 1 -- 172.21.15.1:0/1977593685 wait complete. 2024-03-06T07:27:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.357+0000 7f810f7d7640 1 Processor -- start 2024-03-06T07:27:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.357+0000 7f810f7d7640 1 -- start start 2024-03-06T07:27:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810f7d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8108070c80 0x7f81080c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810f7d7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81080c32f0 0x7f8108146170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810f7d7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8108150520 0x7f81081466b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810f7d7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8108157330 con 0x7f8108150520 2024-03-06T07:27:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810f7d7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81081532a0 con 0x7f8108070c80 2024-03-06T07:27:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810f7d7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81080c7ec0 con 0x7f81080c32f0 2024-03-06T07:27:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810dd74640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8108150520 0x7f81081466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810dd74640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8108150520 0x7f81081466b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59496/0 (socket says 172.21.15.1:59496) 2024-03-06T07:27:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810dd74640 1 -- 172.21.15.1:0/2610625272 learned_addr learned my addr 172.21.15.1:0/2610625272 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810d573640 1 --2- 172.21.15.1:0/2610625272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8108070c80 0x7f81080c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810cd72640 1 --2- 172.21.15.1:0/2610625272 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81080c32f0 0x7f8108146170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810dd74640 1 -- 172.21.15.1:0/2610625272 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81080c32f0 msgr2=0x7f8108146170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810dd74640 1 --2- 172.21.15.1:0/2610625272 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81080c32f0 0x7f8108146170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810dd74640 1 -- 172.21.15.1:0/2610625272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8108070c80 msgr2=0x7f81080c1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810dd74640 1 --2- 172.21.15.1:0/2610625272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8108070c80 0x7f81080c1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810dd74640 1 -- 172.21.15.1:0/2610625272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81081eee90 con 0x7f8108150520 2024-03-06T07:27:09.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810d573640 1 --2- 172.21.15.1:0/2610625272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8108070c80 0x7f81080c1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:09.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f810dd74640 1 --2- 172.21.15.1:0/2610625272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8108150520 0x7f81081466b0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f810000c7d0 tx=0x7f810000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f80fdffb640 1 -- 172.21.15.1:0/2610625272 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8100003b70 con 0x7f8108150520 2024-03-06T07:27:09.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.358+0000 7f80fdffb640 1 -- 172.21.15.1:0/2610625272 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8100003d30 con 0x7f8108150520 2024-03-06T07:27:09.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.359+0000 7f810f7d7640 1 -- 172.21.15.1:0/2610625272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810814b960 con 0x7f8108150520 2024-03-06T07:27:09.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.359+0000 7f80fdffb640 1 -- 172.21.15.1:0/2610625272 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810000aa90 con 0x7f8108150520 2024-03-06T07:27:09.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.359+0000 7f810f7d7640 1 -- 172.21.15.1:0/2610625272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81081ee280 con 0x7f8108150520 2024-03-06T07:27:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.360+0000 7f810f7d7640 1 -- 172.21.15.1:0/2610625272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81080686e0 con 0x7f8108150520 2024-03-06T07:27:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.360+0000 7f80fdffb640 1 -- 172.21.15.1:0/2610625272 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f810000ac30 con 0x7f8108150520 2024-03-06T07:27:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.361+0000 7f80fdffb640 1 --2- 172.21.15.1:0/2610625272 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f80ec06b500 0x7f80ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.361+0000 7f80fdffb640 1 -- 172.21.15.1:0/2610625272 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f810008a230 con 0x7f8108150520 2024-03-06T07:27:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.361+0000 7f810d573640 1 --2- 172.21.15.1:0/2610625272 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f80ec06b500 0x7f80ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.362+0000 7f810d573640 1 --2- 172.21.15.1:0/2610625272 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f80ec06b500 0x7f80ec06d9c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f8104000f60 tx=0x7f8104009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.365+0000 7f80fdffb640 1 -- 172.21.15.1:0/2610625272 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8100058fb0 con 0x7f8108150520 2024-03-06T07:27:09.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.615+0000 7f810f7d7640 1 -- 172.21.15.1:0/2610625272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f810806c580 con 0x7f8108150520 2024-03-06T07:27:09.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.616+0000 7f80fdffb640 1 -- 172.21.15.1:0/2610625272 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1458 (secure 0 0 0) 0x7f81000588b0 con 0x7f8108150520 2024-03-06T07:27:09.618 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:09.618 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":425,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":171,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709614,"num_in_osds":8,"osd_in_since":1709709606,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":272629760,"bytes_avail":772821483520,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":157,"id":13,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":13,"rank":0,"name":"c","status":"up:active","gid":9489}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-03-06T07:26:52.781305+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:27:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.618+0000 7f810f7d7640 1 -- 172.21.15.1:0/2610625272 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f80ec06b500 msgr2=0x7f80ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.618+0000 7f810f7d7640 1 --2- 172.21.15.1:0/2610625272 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f80ec06b500 0x7f80ec06d9c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f8104000f60 tx=0x7f8104009040 comp rx=0 tx=0).stop 2024-03-06T07:27:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.618+0000 7f810f7d7640 1 -- 172.21.15.1:0/2610625272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8108150520 msgr2=0x7f81081466b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.618+0000 7f810f7d7640 1 --2- 172.21.15.1:0/2610625272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8108150520 0x7f81081466b0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f810000c7d0 tx=0x7f810000cc90 comp rx=0 tx=0).stop 2024-03-06T07:27:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.618+0000 7f810f7d7640 1 -- 172.21.15.1:0/2610625272 shutdown_connections 2024-03-06T07:27:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.618+0000 7f810f7d7640 1 --2- 172.21.15.1:0/2610625272 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f80ec06b500 0x7f80ec06d9c0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.618+0000 7f810f7d7640 1 --2- 172.21.15.1:0/2610625272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8108150520 0x7f81081466b0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.618+0000 7f810f7d7640 1 --2- 172.21.15.1:0/2610625272 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81080c32f0 0x7f8108146170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.618+0000 7f810f7d7640 1 --2- 172.21.15.1:0/2610625272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8108070c80 0x7f81080c1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.618+0000 7f810f7d7640 1 -- 172.21.15.1:0/2610625272 >> 172.21.15.1:0/2610625272 conn(0x7f81080c88f0 msgr2=0x7f81080c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.618+0000 7f810f7d7640 1 -- 172.21.15.1:0/2610625272 shutdown_connections 2024-03-06T07:27:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.618+0000 7f810f7d7640 1 -- 172.21.15.1:0/2610625272 wait complete. 2024-03-06T07:27:09.630 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:27:09.631 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:09.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.733+0000 7f64a735e640 1 Processor -- start 2024-03-06T07:27:09.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.733+0000 7f64a735e640 1 -- start start 2024-03-06T07:27:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.734+0000 7f64a735e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64a00c6b10 0x7f64a0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.734+0000 7f64a735e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f64a00bf9d0 0x7f64a00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.734+0000 7f64a735e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a00c2fc0 con 0x7f64a0152650 2024-03-06T07:27:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.734+0000 7f64a735e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a0155800 con 0x7f64a00bf9d0 2024-03-06T07:27:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.734+0000 7f64a735e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a0158da0 con 0x7f64a00c6b10 2024-03-06T07:27:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.734+0000 7f64a58fb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f64a00bf9d0 0x7f64a00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.734+0000 7f64a58fb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f64a00bf9d0 0x7f64a00c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59506/0 (socket says 172.21.15.1:59506) 2024-03-06T07:27:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.734+0000 7f64a58fb640 1 -- 172.21.15.1:0/2788112283 learned_addr learned my addr 172.21.15.1:0/2788112283 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.735+0000 7f64a48f9640 1 --2- 172.21.15.1:0/2788112283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64a00c6b10 0x7f64a0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.735+0000 7f64a58fb640 1 -- 172.21.15.1:0/2788112283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64a00c6b10 msgr2=0x7f64a0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.735+0000 7f64a58fb640 1 --2- 172.21.15.1:0/2788112283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64a00c6b10 0x7f64a0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.735+0000 7f64a58fb640 1 -- 172.21.15.1:0/2788112283 >> v1:172.21.15.98:6789/0 conn(0x7f64a0152650 legacy=0x7f64a0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.735+0000 7f64a58fb640 1 -- 172.21.15.1:0/2788112283 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a00c6690 con 0x7f64a00bf9d0 2024-03-06T07:27:09.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.736+0000 7f64a58fb640 1 --2- 172.21.15.1:0/2788112283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a00bf9d0 0x7f64a00c1460 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f6498009e50 tx=0x7f649802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a7ca3594fc9a76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.736+0000 7f6497fff640 1 -- 172.21.15.1:0/2788112283 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649802ca70 con 0x7f64a00bf9d0 2024-03-06T07:27:09.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.736+0000 7f6497fff640 1 -- 172.21.15.1:0/2788112283 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649802cc10 con 0x7f64a00bf9d0 2024-03-06T07:27:09.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.736+0000 7f6497fff640 1 -- 172.21.15.1:0/2788112283 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64980343f0 con 0x7f64a00bf9d0 2024-03-06T07:27:09.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.736+0000 7f64a735e640 1 -- 172.21.15.1:0/2788112283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a00bf9d0 msgr2=0x7f64a00c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.736+0000 7f64a735e640 1 --2- 172.21.15.1:0/2788112283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a00bf9d0 0x7f64a00c1460 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f6498009e50 tx=0x7f649802bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.736+0000 7f64a735e640 1 -- 172.21.15.1:0/2788112283 shutdown_connections 2024-03-06T07:27:09.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.736+0000 7f64a735e640 1 --2- 172.21.15.1:0/2788112283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a00bf9d0 0x7f64a00c1460 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.736+0000 7f64a735e640 1 --2- 172.21.15.1:0/2788112283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64a00c6b10 0x7f64a0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.736+0000 7f64a735e640 1 -- 172.21.15.1:0/2788112283 >> 172.21.15.1:0/2788112283 conn(0x7f64a0066ee0 msgr2=0x7f64a00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.736+0000 7f64a735e640 1 -- 172.21.15.1:0/2788112283 shutdown_connections 2024-03-06T07:27:09.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.737+0000 7f64a735e640 1 -- 172.21.15.1:0/2788112283 wait complete. 2024-03-06T07:27:09.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.738+0000 7f64a735e640 1 Processor -- start 2024-03-06T07:27:09.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.738+0000 7f64a735e640 1 -- start start 2024-03-06T07:27:09.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.738+0000 7f64a735e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a00bf9d0 0x7f64a0154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.738+0000 7f64a735e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a0152650 0x7f64a014e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.738+0000 7f64a735e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64a014cfb0 0x7f64a00c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.738+0000 7f64a735e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a01551b0 con 0x7f64a00bf9d0 2024-03-06T07:27:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.738+0000 7f64a735e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a00c6fa0 con 0x7f64a0152650 2024-03-06T07:27:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.738+0000 7f64a735e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a0155490 con 0x7f64a014cfb0 2024-03-06T07:27:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.738+0000 7f64a50fa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a00bf9d0 0x7f64a0154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.738+0000 7f64a50fa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a00bf9d0 0x7f64a0154660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59514/0 (socket says 172.21.15.1:59514) 2024-03-06T07:27:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.739+0000 7f64a50fa640 1 -- 172.21.15.1:0/860999257 learned_addr learned my addr 172.21.15.1:0/860999257 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.739+0000 7f64a58fb640 1 --2- 172.21.15.1:0/860999257 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64a014cfb0 0x7f64a00c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.739+0000 7f64a48f9640 1 --2- 172.21.15.1:0/860999257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a0152650 0x7f64a014e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.739+0000 7f64a50fa640 1 -- 172.21.15.1:0/860999257 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64a014cfb0 msgr2=0x7f64a00c8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.739+0000 7f64a50fa640 1 --2- 172.21.15.1:0/860999257 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64a014cfb0 0x7f64a00c8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.739+0000 7f64a50fa640 1 -- 172.21.15.1:0/860999257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a0152650 msgr2=0x7f64a014e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.739+0000 7f64a50fa640 1 --2- 172.21.15.1:0/860999257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a0152650 0x7f64a014e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.739+0000 7f64a50fa640 1 -- 172.21.15.1:0/860999257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a0154ba0 con 0x7f64a00bf9d0 2024-03-06T07:27:09.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.739+0000 7f64a50fa640 1 --2- 172.21.15.1:0/860999257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a00bf9d0 0x7f64a0154660 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f649c00c400 tx=0x7f649c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.739+0000 7f6495ffb640 1 -- 172.21.15.1:0/860999257 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c0052d0 con 0x7f64a00bf9d0 2024-03-06T07:27:09.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.739+0000 7f6495ffb640 1 -- 172.21.15.1:0/860999257 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649c005490 con 0x7f64a00bf9d0 2024-03-06T07:27:09.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.739+0000 7f64a735e640 1 -- 172.21.15.1:0/860999257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a0154e60 con 0x7f64a00bf9d0 2024-03-06T07:27:09.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.739+0000 7f6495ffb640 1 -- 172.21.15.1:0/860999257 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c007260 con 0x7f64a00bf9d0 2024-03-06T07:27:09.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.739+0000 7f64a735e640 1 -- 172.21.15.1:0/860999257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a014c7b0 con 0x7f64a00bf9d0 2024-03-06T07:27:09.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.740+0000 7f64a735e640 1 -- 172.21.15.1:0/860999257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a0155d40 con 0x7f64a00bf9d0 2024-03-06T07:27:09.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.741+0000 7f6495ffb640 1 -- 172.21.15.1:0/860999257 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f649c007400 con 0x7f64a00bf9d0 2024-03-06T07:27:09.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.741+0000 7f6495ffb640 1 --2- 172.21.15.1:0/860999257 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f648406b500 0x7f648406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.741+0000 7f6495ffb640 1 -- 172.21.15.1:0/860999257 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f649c089730 con 0x7f64a00bf9d0 2024-03-06T07:27:09.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.741+0000 7f64a48f9640 1 --2- 172.21.15.1:0/860999257 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f648406b500 0x7f648406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.742+0000 7f64a48f9640 1 --2- 172.21.15.1:0/860999257 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f648406b500 0x7f648406d9c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f64a00c6400 tx=0x7f6490009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.746+0000 7f6495ffb640 1 -- 172.21.15.1:0/860999257 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64a0155d40 con 0x7f64a00bf9d0 2024-03-06T07:27:09.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.923+0000 7f64a735e640 1 -- 172.21.15.1:0/860999257 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f64a0067fd0 con 0x7f648406b500 2024-03-06T07:27:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.929+0000 7f6495ffb640 1 -- 172.21.15.1:0/860999257 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247159 (secure 0 0 0) 0x7f64a0067fd0 con 0x7f648406b500 2024-03-06T07:27:09.934 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:09.935 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:27:09.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.933+0000 7f64a735e640 1 -- 172.21.15.1:0/860999257 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f648406b500 msgr2=0x7f648406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.933+0000 7f64a735e640 1 --2- 172.21.15.1:0/860999257 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f648406b500 0x7f648406d9c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f64a00c6400 tx=0x7f6490009190 comp rx=0 tx=0).stop 2024-03-06T07:27:09.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.933+0000 7f64a735e640 1 -- 172.21.15.1:0/860999257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a00bf9d0 msgr2=0x7f64a0154660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.933+0000 7f64a735e640 1 --2- 172.21.15.1:0/860999257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a00bf9d0 0x7f64a0154660 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f649c00c400 tx=0x7f649c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.933+0000 7f64a735e640 1 -- 172.21.15.1:0/860999257 shutdown_connections 2024-03-06T07:27:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.933+0000 7f64a735e640 1 --2- 172.21.15.1:0/860999257 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f648406b500 0x7f648406d9c0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.933+0000 7f64a735e640 1 --2- 172.21.15.1:0/860999257 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f64a014cfb0 0x7f64a00c8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.933+0000 7f64a735e640 1 --2- 172.21.15.1:0/860999257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a0152650 0x7f64a014e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.933+0000 7f64a735e640 1 --2- 172.21.15.1:0/860999257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64a00bf9d0 0x7f64a0154660 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.933+0000 7f64a735e640 1 -- 172.21.15.1:0/860999257 >> 172.21.15.1:0/860999257 conn(0x7f64a0066ee0 msgr2=0x7f64a00cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.933+0000 7f64a735e640 1 -- 172.21.15.1:0/860999257 shutdown_connections 2024-03-06T07:27:09.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:09.933+0000 7f64a735e640 1 -- 172.21.15.1:0/860999257 wait complete. 2024-03-06T07:27:09.946 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-03-06T07:27:08.784835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":266240,"kb_used_data":16784,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754708480,"statfs":{"total":773094113280,"available":772821483520,"internally_reserved":0,"allocated":17186816,"data_stored":9494844,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.533351"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254547+0000","last_change":"2024-03-06T07:27:04.254547+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254547+0000","last_undegraded":"2024-03-06T07:27:04.254547+0000","last_fullsized":"2024-03-06T07:27:04.254547+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.255495+0000","last_change":"2024-03-06T07:27:04.255495+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.255495+0000","last_undegraded":"2024-03-06T07:27:04.255495+0000","last_fullsized":"2024-03-06T07:27:04.255495+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259837+0000","last_change":"2024-03-06T07:27:04.259837+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259837+0000","last_undegraded":"2024-03-06T07:27:04.259837+0000","last_fullsized":"2024-03-06T07:27:04.259837+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259747+0000","last_change":"2024-03-06T07:27:04.259747+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259747+0000","last_undegraded":"2024-03-06T07:27:04.259747+0000","last_fullsized":"2024-03-06T07:27:04.259747+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258205+0000","last_change":"2024-03-06T07:27:04.258205+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258205+0000","last_undegraded":"2024-03-06T07:27:04.258205+0000","last_fullsized":"2024-03-06T07:27:04.258205+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259513+0000","last_change":"2024-03-06T07:27:04.259513+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259513+0000","last_undegraded":"2024-03-06T07:27:04.259513+0000","last_fullsized":"2024-03-06T07:27:04.259513+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257656+0000","last_change":"2024-03-06T07:27:04.257656+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257656+0000","last_undegraded":"2024-03-06T07:27:04.257656+0000","last_fullsized":"2024-03-06T07:27:04.257656+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257414+0000","last_change":"2024-03-06T07:27:04.257414+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257414+0000","last_undegraded":"2024-03-06T07:27:04.257414+0000","last_fullsized":"2024-03-06T07:27:04.257414+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258740+0000","last_change":"2024-03-06T07:27:04.258740+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258740+0000","last_undegraded":"2024-03-06T07:27:04.258740+0000","last_fullsized":"2024-03-06T07:27:04.258740+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258528+0000","last_change":"2024-03-06T07:27:04.258528+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258528+0000","last_undegraded":"2024-03-06T07:27:04.258528+0000","last_fullsized":"2024-03-06T07:27:04.258528+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259375+0000","last_change":"2024-03-06T07:27:04.259375+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259375+0000","last_undegraded":"2024-03-06T07:27:04.259375+0000","last_fullsized":"2024-03-06T07:27:04.259375+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257246+0000","last_change":"2024-03-06T07:27:04.257246+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257246+0000","last_undegraded":"2024-03-06T07:27:04.257246+0000","last_fullsized":"2024-03-06T07:27:04.257246+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259445+0000","last_change":"2024-03-06T07:27:04.259445+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259445+0000","last_undegraded":"2024-03-06T07:27:04.259445+0000","last_fullsized":"2024-03-06T07:27:04.259445+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257927+0000","last_change":"2024-03-06T07:27:04.257927+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257927+0000","last_undegraded":"2024-03-06T07:27:04.257927+0000","last_fullsized":"2024-03-06T07:27:04.257927+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.263169+0000","last_change":"2024-03-06T07:27:04.263169+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.263169+0000","last_undegraded":"2024-03-06T07:27:04.263169+0000","last_fullsized":"2024-03-06T07:27:04.263169+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257264+0000","last_change":"2024-03-06T07:27:04.257264+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257264+0000","last_undegraded":"2024-03-06T07:27:04.257264+0000","last_fullsized":"2024-03-06T07:27:04.257264+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258290+0000","last_change":"2024-03-06T07:27:04.258290+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258290+0000","last_undegraded":"2024-03-06T07:27:04.258290+0000","last_fullsized":"2024-03-06T07:27:04.258290+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258702+0000","last_change":"2024-03-06T07:27:04.258702+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258702+0000","last_undegraded":"2024-03-06T07:27:04.258702+0000","last_fullsized":"2024-03-06T07:27:04.258702+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258065+0000","last_change":"2024-03-06T07:27:04.258065+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258065+0000","last_undegraded":"2024-03-06T07:27:04.258065+0000","last_fullsized":"2024-03-06T07:27:04.258065+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258945+0000","last_change":"2024-03-06T07:27:04.258945+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258945+0000","last_undegraded":"2024-03-06T07:27:04.258945+0000","last_fullsized":"2024-03-06T07:27:04.258945+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257755+0000","last_change":"2024-03-06T07:27:04.257755+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257755+0000","last_undegraded":"2024-03-06T07:27:04.257755+0000","last_fullsized":"2024-03-06T07:27:04.257755+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257343+0000","last_change":"2024-03-06T07:27:04.257343+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257343+0000","last_undegraded":"2024-03-06T07:27:04.257343+0000","last_fullsized":"2024-03-06T07:27:04.257343+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.253999+0000","last_change":"2024-03-06T07:27:04.253999+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.253999+0000","last_undegraded":"2024-03-06T07:27:04.253999+0000","last_fullsized":"2024-03-06T07:27:04.253999+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256789+0000","last_change":"2024-03-06T07:27:04.256789+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256789+0000","last_undegraded":"2024-03-06T07:27:04.256789+0000","last_fullsized":"2024-03-06T07:27:04.256789+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257184+0000","last_change":"2024-03-06T07:27:04.257184+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257184+0000","last_undegraded":"2024-03-06T07:27:04.257184+0000","last_fullsized":"2024-03-06T07:27:04.257184+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257217+0000","last_change":"2024-03-06T07:27:04.257217+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257217+0000","last_undegraded":"2024-03-06T07:27:04.257217+0000","last_fullsized":"2024-03-06T07:27:04.257217+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.259006+0000","last_change":"2024-03-06T07:27:04.259006+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.259006+0000","last_undegraded":"2024-03-06T07:27:04.259006+0000","last_fullsized":"2024-03-06T07:27:04.259006+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.255903+0000","last_change":"2024-03-06T07:27:04.255903+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.255903+0000","last_undegraded":"2024-03-06T07:27:04.255903+0000","last_fullsized":"2024-03-06T07:27:04.255903+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254435+0000","last_change":"2024-03-06T07:27:04.254435+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254435+0000","last_undegraded":"2024-03-06T07:27:04.254435+0000","last_fullsized":"2024-03-06T07:27:04.254435+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.264763+0000","last_change":"2024-03-06T07:27:04.264763+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.264763+0000","last_undegraded":"2024-03-06T07:27:04.264763+0000","last_fullsized":"2024-03-06T07:27:04.264763+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261771+0000","last_change":"2024-03-06T07:27:04.261771+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261771+0000","last_undegraded":"2024-03-06T07:27:04.261771+0000","last_fullsized":"2024-03-06T07:27:04.261771+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.260482+0000","last_change":"2024-03-06T07:27:04.260482+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.260482+0000","last_undegraded":"2024-03-06T07:27:04.260482+0000","last_fullsized":"2024-03-06T07:27:04.260482+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254981+0000","last_change":"2024-03-06T07:27:04.254981+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254981+0000","last_undegraded":"2024-03-06T07:27:04.254981+0000","last_fullsized":"2024-03-06T07:27:04.254981+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.278552+0000","last_change":"2024-03-06T07:27:04.271712+0000","last_active":"2024-03-06T07:27:04.278552+0000","last_peered":"2024-03-06T07:27:04.278552+0000","last_clean":"2024-03-06T07:27:04.278552+0000","last_became_active":"2024-03-06T07:27:04.271429+0000","last_became_peered":"2024-03-06T07:27:04.271429+0000","last_unstale":"2024-03-06T07:27:04.278552+0000","last_undegraded":"2024-03-06T07:27:04.278552+0000","last_fullsized":"2024-03-06T07:27:04.278552+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:20.726754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257054+0000","last_change":"2024-03-06T07:27:04.257054+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257054+0000","last_undegraded":"2024-03-06T07:27:04.257054+0000","last_fullsized":"2024-03-06T07:27:04.257054+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.268329+0000","last_change":"2024-03-06T07:27:04.266276+0000","last_active":"2024-03-06T07:27:04.268329+0000","last_peered":"2024-03-06T07:27:04.268329+0000","last_clean":"2024-03-06T07:27:04.268329+0000","last_became_active":"2024-03-06T07:27:04.265418+0000","last_became_peered":"2024-03-06T07:27:04.265418+0000","last_unstale":"2024-03-06T07:27:04.268329+0000","last_undegraded":"2024-03-06T07:27:04.268329+0000","last_fullsized":"2024-03-06T07:27:04.268329+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:24.992967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256890+0000","last_change":"2024-03-06T07:27:04.256890+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256890+0000","last_undegraded":"2024-03-06T07:27:04.256890+0000","last_fullsized":"2024-03-06T07:27:04.256890+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267032+0000","last_change":"2024-03-06T07:27:04.262189+0000","last_active":"2024-03-06T07:27:04.267032+0000","last_peered":"2024-03-06T07:27:04.267032+0000","last_clean":"2024-03-06T07:27:04.267032+0000","last_became_active":"2024-03-06T07:27:04.261909+0000","last_became_peered":"2024-03-06T07:27:04.261909+0000","last_unstale":"2024-03-06T07:27:04.267032+0000","last_undegraded":"2024-03-06T07:27:04.267032+0000","last_fullsized":"2024-03-06T07:27:04.267032+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:15:43.228599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256537+0000","last_change":"2024-03-06T07:27:04.256537+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256537+0000","last_undegraded":"2024-03-06T07:27:04.256537+0000","last_fullsized":"2024-03-06T07:27:04.256537+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271246+0000","last_change":"2024-03-06T07:27:04.270472+0000","last_active":"2024-03-06T07:27:04.271246+0000","last_peered":"2024-03-06T07:27:04.271246+0000","last_clean":"2024-03-06T07:27:04.271246+0000","last_became_active":"2024-03-06T07:27:04.270185+0000","last_became_peered":"2024-03-06T07:27:04.270185+0000","last_unstale":"2024-03-06T07:27:04.271246+0000","last_undegraded":"2024-03-06T07:27:04.271246+0000","last_fullsized":"2024-03-06T07:27:04.271246+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:37.715400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.263186+0000","last_change":"2024-03-06T07:27:04.263186+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.263186+0000","last_undegraded":"2024-03-06T07:27:04.263186+0000","last_fullsized":"2024-03-06T07:27:04.263186+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.275596+0000","last_change":"2024-03-06T07:27:04.272491+0000","last_active":"2024-03-06T07:27:04.275596+0000","last_peered":"2024-03-06T07:27:04.275596+0000","last_clean":"2024-03-06T07:27:04.275596+0000","last_became_active":"2024-03-06T07:27:04.272190+0000","last_became_peered":"2024-03-06T07:27:04.272190+0000","last_unstale":"2024-03-06T07:27:04.275596+0000","last_undegraded":"2024-03-06T07:27:04.275596+0000","last_fullsized":"2024-03-06T07:27:04.275596+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:17.476526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.263035+0000","last_change":"2024-03-06T07:27:04.263035+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.263035+0000","last_undegraded":"2024-03-06T07:27:04.263035+0000","last_fullsized":"2024-03-06T07:27:04.263035+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.278476+0000","last_change":"2024-03-06T07:27:04.273573+0000","last_active":"2024-03-06T07:27:04.278476+0000","last_peered":"2024-03-06T07:27:04.278476+0000","last_clean":"2024-03-06T07:27:04.278476+0000","last_became_active":"2024-03-06T07:27:04.273253+0000","last_became_peered":"2024-03-06T07:27:04.273253+0000","last_unstale":"2024-03-06T07:27:04.278476+0000","last_undegraded":"2024-03-06T07:27:04.278476+0000","last_fullsized":"2024-03-06T07:27:04.278476+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:56.531270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254166+0000","last_change":"2024-03-06T07:27:04.254166+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254166+0000","last_undegraded":"2024-03-06T07:27:04.254166+0000","last_fullsized":"2024-03-06T07:27:04.254166+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.272537+0000","last_change":"2024-03-06T07:27:04.271894+0000","last_active":"2024-03-06T07:27:04.272537+0000","last_peered":"2024-03-06T07:27:04.272537+0000","last_clean":"2024-03-06T07:27:04.272537+0000","last_became_active":"2024-03-06T07:27:04.271305+0000","last_became_peered":"2024-03-06T07:27:04.271305+0000","last_unstale":"2024-03-06T07:27:04.272537+0000","last_undegraded":"2024-03-06T07:27:04.272537+0000","last_fullsized":"2024-03-06T07:27:04.272537+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:26.898488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.266335+0000","last_change":"2024-03-06T07:27:04.266335+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.266335+0000","last_undegraded":"2024-03-06T07:27:04.266335+0000","last_fullsized":"2024-03-06T07:27:04.266335+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271368+0000","last_change":"2024-03-06T07:27:04.269100+0000","last_active":"2024-03-06T07:27:04.271368+0000","last_peered":"2024-03-06T07:27:04.271368+0000","last_clean":"2024-03-06T07:27:04.271368+0000","last_became_active":"2024-03-06T07:27:04.268811+0000","last_became_peered":"2024-03-06T07:27:04.268811+0000","last_unstale":"2024-03-06T07:27:04.271368+0000","last_undegraded":"2024-03-06T07:27:04.271368+0000","last_fullsized":"2024-03-06T07:27:04.271368+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:34.615045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258161+0000","last_change":"2024-03-06T07:27:04.258161+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258161+0000","last_undegraded":"2024-03-06T07:27:04.258161+0000","last_fullsized":"2024-03-06T07:27:04.258161+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.275534+0000","last_change":"2024-03-06T07:27:04.271341+0000","last_active":"2024-03-06T07:27:04.275534+0000","last_peered":"2024-03-06T07:27:04.275534+0000","last_clean":"2024-03-06T07:27:04.275534+0000","last_became_active":"2024-03-06T07:27:04.271023+0000","last_became_peered":"2024-03-06T07:27:04.271023+0000","last_unstale":"2024-03-06T07:27:04.275534+0000","last_undegraded":"2024-03-06T07:27:04.275534+0000","last_fullsized":"2024-03-06T07:27:04.275534+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:24.173726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257145+0000","last_change":"2024-03-06T07:27:04.257145+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257145+0000","last_undegraded":"2024-03-06T07:27:04.257145+0000","last_fullsized":"2024-03-06T07:27:04.257145+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.266679+0000","last_change":"2024-03-06T07:27:04.264830+0000","last_active":"2024-03-06T07:27:04.266679+0000","last_peered":"2024-03-06T07:27:04.266679+0000","last_clean":"2024-03-06T07:27:04.266679+0000","last_became_active":"2024-03-06T07:27:04.264562+0000","last_became_peered":"2024-03-06T07:27:04.264562+0000","last_unstale":"2024-03-06T07:27:04.266679+0000","last_undegraded":"2024-03-06T07:27:04.266679+0000","last_fullsized":"2024-03-06T07:27:04.266679+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:12.146965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256102+0000","last_change":"2024-03-06T07:27:04.256102+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256102+0000","last_undegraded":"2024-03-06T07:27:04.256102+0000","last_fullsized":"2024-03-06T07:27:04.256102+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.278496+0000","last_change":"2024-03-06T07:27:04.273664+0000","last_active":"2024-03-06T07:27:04.278496+0000","last_peered":"2024-03-06T07:27:04.278496+0000","last_clean":"2024-03-06T07:27:04.278496+0000","last_became_active":"2024-03-06T07:27:04.273372+0000","last_became_peered":"2024-03-06T07:27:04.273372+0000","last_unstale":"2024-03-06T07:27:04.278496+0000","last_undegraded":"2024-03-06T07:27:04.278496+0000","last_fullsized":"2024-03-06T07:27:04.278496+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:51.519017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.255456+0000","last_change":"2024-03-06T07:27:04.255456+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.255456+0000","last_undegraded":"2024-03-06T07:27:04.255456+0000","last_fullsized":"2024-03-06T07:27:04.255456+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267158+0000","last_change":"2024-03-06T07:27:04.263075+0000","last_active":"2024-03-06T07:27:04.267158+0000","last_peered":"2024-03-06T07:27:04.267158+0000","last_clean":"2024-03-06T07:27:04.267158+0000","last_became_active":"2024-03-06T07:27:04.261822+0000","last_became_peered":"2024-03-06T07:27:04.261822+0000","last_unstale":"2024-03-06T07:27:04.267158+0000","last_undegraded":"2024-03-06T07:27:04.267158+0000","last_fullsized":"2024-03-06T07:27:04.267158+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:50.416453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258591+0000","last_change":"2024-03-06T07:27:04.258591+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258591+0000","last_undegraded":"2024-03-06T07:27:04.258591+0000","last_fullsized":"2024-03-06T07:27:04.258591+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.270193+0000","last_change":"2024-03-06T07:27:04.268027+0000","last_active":"2024-03-06T07:27:04.270193+0000","last_peered":"2024-03-06T07:27:04.270193+0000","last_clean":"2024-03-06T07:27:04.270193+0000","last_became_active":"2024-03-06T07:27:04.267726+0000","last_became_peered":"2024-03-06T07:27:04.267726+0000","last_unstale":"2024-03-06T07:27:04.270193+0000","last_undegraded":"2024-03-06T07:27:04.270193+0000","last_fullsized":"2024-03-06T07:27:04.270193+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:11.943331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254886+0000","last_change":"2024-03-06T07:27:04.254886+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254886+0000","last_undegraded":"2024-03-06T07:27:04.254886+0000","last_fullsized":"2024-03-06T07:27:04.254886+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.266762+0000","last_change":"2024-03-06T07:27:04.265676+0000","last_active":"2024-03-06T07:27:04.266762+0000","last_peered":"2024-03-06T07:27:04.266762+0000","last_clean":"2024-03-06T07:27:04.266762+0000","last_became_active":"2024-03-06T07:27:04.265374+0000","last_became_peered":"2024-03-06T07:27:04.265374+0000","last_unstale":"2024-03-06T07:27:04.266762+0000","last_undegraded":"2024-03-06T07:27:04.266762+0000","last_fullsized":"2024-03-06T07:27:04.266762+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:30.199858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261058+0000","last_change":"2024-03-06T07:27:04.261058+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261058+0000","last_undegraded":"2024-03-06T07:27:04.261058+0000","last_fullsized":"2024-03-06T07:27:04.261058+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271352+0000","last_change":"2024-03-06T07:27:04.269119+0000","last_active":"2024-03-06T07:27:04.271352+0000","last_peered":"2024-03-06T07:27:04.271352+0000","last_clean":"2024-03-06T07:27:04.271352+0000","last_became_active":"2024-03-06T07:27:04.268844+0000","last_became_peered":"2024-03-06T07:27:04.268844+0000","last_unstale":"2024-03-06T07:27:04.271352+0000","last_undegraded":"2024-03-06T07:27:04.271352+0000","last_fullsized":"2024-03-06T07:27:04.271352+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:47.701531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256965+0000","last_change":"2024-03-06T07:27:04.256965+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256965+0000","last_undegraded":"2024-03-06T07:27:04.256965+0000","last_fullsized":"2024-03-06T07:27:04.256965+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.268206+0000","last_change":"2024-03-06T07:27:04.262327+0000","last_active":"2024-03-06T07:27:04.268206+0000","last_peered":"2024-03-06T07:27:04.268206+0000","last_clean":"2024-03-06T07:27:04.268206+0000","last_became_active":"2024-03-06T07:27:04.262046+0000","last_became_peered":"2024-03-06T07:27:04.262046+0000","last_unstale":"2024-03-06T07:27:04.268206+0000","last_undegraded":"2024-03-06T07:27:04.268206+0000","last_fullsized":"2024-03-06T07:27:04.268206+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:31.006078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256647+0000","last_change":"2024-03-06T07:27:04.256647+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256647+0000","last_undegraded":"2024-03-06T07:27:04.256647+0000","last_fullsized":"2024-03-06T07:27:04.256647+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.268350+0000","last_change":"2024-03-06T07:27:04.266893+0000","last_active":"2024-03-06T07:27:04.268350+0000","last_peered":"2024-03-06T07:27:04.268350+0000","last_clean":"2024-03-06T07:27:04.268350+0000","last_became_active":"2024-03-06T07:27:04.265563+0000","last_became_peered":"2024-03-06T07:27:04.265563+0000","last_unstale":"2024-03-06T07:27:04.268350+0000","last_undegraded":"2024-03-06T07:27:04.268350+0000","last_fullsized":"2024-03-06T07:27:04.268350+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:35.022673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254208+0000","last_change":"2024-03-06T07:27:04.254208+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254208+0000","last_undegraded":"2024-03-06T07:27:04.254208+0000","last_fullsized":"2024-03-06T07:27:04.254208+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267134+0000","last_change":"2024-03-06T07:27:04.265277+0000","last_active":"2024-03-06T07:27:04.267134+0000","last_peered":"2024-03-06T07:27:04.267134+0000","last_clean":"2024-03-06T07:27:04.267134+0000","last_became_active":"2024-03-06T07:27:04.265018+0000","last_became_peered":"2024-03-06T07:27:04.265018+0000","last_unstale":"2024-03-06T07:27:04.267134+0000","last_undegraded":"2024-03-06T07:27:04.267134+0000","last_fullsized":"2024-03-06T07:27:04.267134+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:51.929743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257320+0000","last_change":"2024-03-06T07:27:04.257320+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257320+0000","last_undegraded":"2024-03-06T07:27:04.257320+0000","last_fullsized":"2024-03-06T07:27:04.257320+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267517+0000","last_change":"2024-03-06T07:27:04.265842+0000","last_active":"2024-03-06T07:27:04.267517+0000","last_peered":"2024-03-06T07:27:04.267517+0000","last_clean":"2024-03-06T07:27:04.267517+0000","last_became_active":"2024-03-06T07:27:04.265568+0000","last_became_peered":"2024-03-06T07:27:04.265568+0000","last_unstale":"2024-03-06T07:27:04.267517+0000","last_undegraded":"2024-03-06T07:27:04.267517+0000","last_fullsized":"2024-03-06T07:27:04.267517+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:31.100594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254138+0000","last_change":"2024-03-06T07:27:04.254138+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254138+0000","last_undegraded":"2024-03-06T07:27:04.254138+0000","last_fullsized":"2024-03-06T07:27:04.254138+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.264364+0000","last_change":"2024-03-06T07:27:04.262729+0000","last_active":"2024-03-06T07:27:04.264364+0000","last_peered":"2024-03-06T07:27:04.264364+0000","last_clean":"2024-03-06T07:27:04.264364+0000","last_became_active":"2024-03-06T07:27:04.262484+0000","last_became_peered":"2024-03-06T07:27:04.262484+0000","last_unstale":"2024-03-06T07:27:04.264364+0000","last_undegraded":"2024-03-06T07:27:04.264364+0000","last_fullsized":"2024-03-06T07:27:04.264364+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:38.945976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261717+0000","last_change":"2024-03-06T07:27:04.261717+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261717+0000","last_undegraded":"2024-03-06T07:27:04.261717+0000","last_fullsized":"2024-03-06T07:27:04.261717+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.267085+0000","last_change":"2024-03-06T07:27:04.263995+0000","last_active":"2024-03-06T07:27:04.267085+0000","last_peered":"2024-03-06T07:27:04.267085+0000","last_clean":"2024-03-06T07:27:04.267085+0000","last_became_active":"2024-03-06T07:27:04.263629+0000","last_became_peered":"2024-03-06T07:27:04.263629+0000","last_unstale":"2024-03-06T07:27:04.267085+0000","last_undegraded":"2024-03-06T07:27:04.267085+0000","last_fullsized":"2024-03-06T07:27:04.267085+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:44.864909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254160+0000","last_change":"2024-03-06T07:27:04.254160+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254160+0000","last_undegraded":"2024-03-06T07:27:04.254160+0000","last_fullsized":"2024-03-06T07:27:04.254160+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.278500+0000","last_change":"2024-03-06T07:27:04.271282+0000","last_active":"2024-03-06T07:27:04.278500+0000","last_peered":"2024-03-06T07:27:04.278500+0000","last_clean":"2024-03-06T07:27:04.278500+0000","last_became_active":"2024-03-06T07:27:04.270987+0000","last_became_peered":"2024-03-06T07:27:04.270987+0000","last_unstale":"2024-03-06T07:27:04.278500+0000","last_undegraded":"2024-03-06T07:27:04.278500+0000","last_fullsized":"2024-03-06T07:27:04.278500+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:02.098293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.258625+0000","last_change":"2024-03-06T07:27:04.258625+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.258625+0000","last_undegraded":"2024-03-06T07:27:04.258625+0000","last_fullsized":"2024-03-06T07:27:04.258625+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.275695+0000","last_change":"2024-03-06T07:27:04.272151+0000","last_active":"2024-03-06T07:27:04.275695+0000","last_peered":"2024-03-06T07:27:04.275695+0000","last_clean":"2024-03-06T07:27:04.275695+0000","last_became_active":"2024-03-06T07:27:04.271894+0000","last_became_peered":"2024-03-06T07:27:04.271894+0000","last_unstale":"2024-03-06T07:27:04.275695+0000","last_undegraded":"2024-03-06T07:27:04.275695+0000","last_fullsized":"2024-03-06T07:27:04.275695+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:50.049060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.256530+0000","last_change":"2024-03-06T07:27:04.256530+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.256530+0000","last_undegraded":"2024-03-06T07:27:04.256530+0000","last_fullsized":"2024-03-06T07:27:04.256530+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.270258+0000","last_change":"2024-03-06T07:27:04.268865+0000","last_active":"2024-03-06T07:27:04.270258+0000","last_peered":"2024-03-06T07:27:04.270258+0000","last_clean":"2024-03-06T07:27:04.270258+0000","last_became_active":"2024-03-06T07:27:04.268566+0000","last_became_peered":"2024-03-06T07:27:04.268566+0000","last_unstale":"2024-03-06T07:27:04.270258+0000","last_undegraded":"2024-03-06T07:27:04.270258+0000","last_fullsized":"2024-03-06T07:27:04.270258+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:05.786807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257744+0000","last_change":"2024-03-06T07:27:04.257744+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257744+0000","last_undegraded":"2024-03-06T07:27:04.257744+0000","last_fullsized":"2024-03-06T07:27:04.257744+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.270471+0000","last_change":"2024-03-06T07:27:04.264784+0000","last_active":"2024-03-06T07:27:04.270471+0000","last_peered":"2024-03-06T07:27:04.270471+0000","last_clean":"2024-03-06T07:27:04.270471+0000","last_became_active":"2024-03-06T07:27:04.264499+0000","last_became_peered":"2024-03-06T07:27:04.264499+0000","last_unstale":"2024-03-06T07:27:04.270471+0000","last_undegraded":"2024-03-06T07:27:04.270471+0000","last_fullsized":"2024-03-06T07:27:04.270471+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:38.542889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254229+0000","last_change":"2024-03-06T07:27:04.254229+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254229+0000","last_undegraded":"2024-03-06T07:27:04.254229+0000","last_fullsized":"2024-03-06T07:27:04.254229+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.275452+0000","last_change":"2024-03-06T07:27:04.269541+0000","last_active":"2024-03-06T07:27:04.275452+0000","last_peered":"2024-03-06T07:27:04.275452+0000","last_clean":"2024-03-06T07:27:04.275452+0000","last_became_active":"2024-03-06T07:27:04.269228+0000","last_became_peered":"2024-03-06T07:27:04.269228+0000","last_unstale":"2024-03-06T07:27:04.275452+0000","last_undegraded":"2024-03-06T07:27:04.275452+0000","last_fullsized":"2024-03-06T07:27:04.275452+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:05.198388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254889+0000","last_change":"2024-03-06T07:27:04.254889+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254889+0000","last_undegraded":"2024-03-06T07:27:04.254889+0000","last_fullsized":"2024-03-06T07:27:04.254889+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.264329+0000","last_change":"2024-03-06T07:27:04.260635+0000","last_active":"2024-03-06T07:27:04.264329+0000","last_peered":"2024-03-06T07:27:04.264329+0000","last_clean":"2024-03-06T07:27:04.264329+0000","last_became_active":"2024-03-06T07:27:04.260370+0000","last_became_peered":"2024-03-06T07:27:04.260370+0000","last_unstale":"2024-03-06T07:27:04.264329+0000","last_undegraded":"2024-03-06T07:27:04.264329+0000","last_fullsized":"2024-03-06T07:27:04.264329+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:26.449033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.257795+0000","last_change":"2024-03-06T07:27:04.257795+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.257795+0000","last_undegraded":"2024-03-06T07:27:04.257795+0000","last_fullsized":"2024-03-06T07:27:04.257795+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.272959+0000","last_change":"2024-03-06T07:27:04.269073+0000","last_active":"2024-03-06T07:27:04.272959+0000","last_peered":"2024-03-06T07:27:04.272959+0000","last_clean":"2024-03-06T07:27:04.272959+0000","last_became_active":"2024-03-06T07:27:04.268831+0000","last_became_peered":"2024-03-06T07:27:04.268831+0000","last_unstale":"2024-03-06T07:27:04.272959+0000","last_undegraded":"2024-03-06T07:27:04.272959+0000","last_fullsized":"2024-03-06T07:27:04.272959+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:30.526595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261310+0000","last_change":"2024-03-06T07:27:04.261310+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261310+0000","last_undegraded":"2024-03-06T07:27:04.261310+0000","last_fullsized":"2024-03-06T07:27:04.261310+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271907+0000","last_change":"2024-03-06T07:27:04.271202+0000","last_active":"2024-03-06T07:27:04.271907+0000","last_peered":"2024-03-06T07:27:04.271907+0000","last_clean":"2024-03-06T07:27:04.271907+0000","last_became_active":"2024-03-06T07:27:04.270881+0000","last_became_peered":"2024-03-06T07:27:04.270881+0000","last_unstale":"2024-03-06T07:27:04.271907+0000","last_undegraded":"2024-03-06T07:27:04.271907+0000","last_fullsized":"2024-03-06T07:27:04.271907+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:17.006730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.261395+0000","last_change":"2024-03-06T07:27:04.261395+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.261395+0000","last_undegraded":"2024-03-06T07:27:04.261395+0000","last_fullsized":"2024-03-06T07:27:04.261395+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.273065+0000","last_change":"2024-03-06T07:27:04.269025+0000","last_active":"2024-03-06T07:27:04.273065+0000","last_peered":"2024-03-06T07:27:04.273065+0000","last_clean":"2024-03-06T07:27:04.273065+0000","last_became_active":"2024-03-06T07:27:04.268768+0000","last_became_peered":"2024-03-06T07:27:04.268768+0000","last_unstale":"2024-03-06T07:27:04.273065+0000","last_undegraded":"2024-03-06T07:27:04.273065+0000","last_fullsized":"2024-03-06T07:27:04.273065+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:28.810603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.260194+0000","last_change":"2024-03-06T07:27:04.260194+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.260194+0000","last_undegraded":"2024-03-06T07:27:04.260194+0000","last_fullsized":"2024-03-06T07:27:04.260194+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.271722+0000","last_change":"2024-03-06T07:27:04.269800+0000","last_active":"2024-03-06T07:27:04.271722+0000","last_peered":"2024-03-06T07:27:04.271722+0000","last_clean":"2024-03-06T07:27:04.271722+0000","last_became_active":"2024-03-06T07:27:04.269489+0000","last_became_peered":"2024-03-06T07:27:04.269489+0000","last_unstale":"2024-03-06T07:27:04.271722+0000","last_undegraded":"2024-03-06T07:27:04.271722+0000","last_fullsized":"2024-03-06T07:27:04.271722+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:58.666333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":1,"reported_epoch":169,"state":"creating+peering","last_fresh":"2024-03-06T07:27:04.254142+0000","last_change":"2024-03-06T07:27:04.254142+0000","last_active":"2024-03-06T07:27:04.246424+0000","last_peered":"2024-03-06T07:27:04.246424+0000","last_clean":"2024-03-06T07:27:04.246424+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:04.254142+0000","last_undegraded":"2024-03-06T07:27:04.254142+0000","last_fullsized":"2024-03-06T07:27:04.254142+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.259397+0000","last_change":"2024-03-06T07:27:04.258208+0000","last_active":"2024-03-06T07:27:04.259397+0000","last_peered":"2024-03-06T07:27:04.259397+0000","last_clean":"2024-03-06T07:27:04.259397+0000","last_became_active":"2024-03-06T07:27:04.257866+0000","last_became_peered":"2024-03-06T07:27:04.257866+0000","last_unstale":"2024-03-06T07:27:04.259397+0000","last_undegraded":"2024-03-06T07:27:04.259397+0000","last_fullsized":"2024-03-06T07:27:04.259397+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:34.006302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":456,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.262911+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:27:04.262911+0000","last_peered":"2024-03-06T07:27:04.262911+0000","last_clean":"2024-03-06T07:27:04.262911+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:27:04.262911+0000","last_undegraded":"2024-03-06T07:27:04.262911+0000","last_fullsized":"2024-03-06T07:27:04.262911+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33320,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338520,"statfs":{"total":96636764160,"available":96602644480,"internally_reserved":0,"allocated":2191360,"data_stored":1626298,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574932,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1042597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1042597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1035930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574932,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1042597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1042597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1035930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574933,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33320,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338520,"statfs":{"total":96636764160,"available":96602644480,"internally_reserved":0,"allocated":2191360,"data_stored":1626298,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:10.558 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-06T07:27:10.556+0000 7fbb7c614640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:12.950 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:13.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.050+0000 7f5a9129a640 1 Processor -- start 2024-03-06T07:27:13.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.050+0000 7f5a9129a640 1 -- start start 2024-03-06T07:27:13.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.051+0000 7f5a9129a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5a8c06c370 0x7f5a8c0c3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.051+0000 7f5a9129a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a8c06dd60 0x7f5a8c155350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.051+0000 7f5a9129a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5a8c155890 0x7f5a8c14f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.051+0000 7f5a9129a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a8c06e1c0 con 0x7f5a8c155890 2024-03-06T07:27:13.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.051+0000 7f5a9129a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a8c06f810 con 0x7f5a8c06c370 2024-03-06T07:27:13.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.051+0000 7f5a9129a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c06fdb0 con 0x7f5a8c06dd60 2024-03-06T07:27:13.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.052+0000 7f5a8b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5a8c155890 0x7f5a8c14f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.052+0000 7f5a8b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5a8c155890 0x7f5a8c14f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59524/0 (socket says 172.21.15.1:59524) 2024-03-06T07:27:13.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.052+0000 7f5a8b7fe640 1 -- 172.21.15.1:0/2363913642 learned_addr learned my addr 172.21.15.1:0/2363913642 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:13.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.052+0000 7f5a8a7fc640 1 --2- 172.21.15.1:0/2363913642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a8c06dd60 0x7f5a8c155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.052+0000 7f5a8affd640 1 --2- 172.21.15.1:0/2363913642 >> v2:172.21.15.98:3300/0 conn(0x7f5a8c06c370 0x7f5a8c0c3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.052+0000 7f5a8b7fe640 1 -- 172.21.15.1:0/2363913642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a8c06dd60 msgr2=0x7f5a8c155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.052+0000 7f5a8b7fe640 1 --2- 172.21.15.1:0/2363913642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a8c06dd60 0x7f5a8c155350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.052+0000 7f5a8b7fe640 1 -- 172.21.15.1:0/2363913642 >> v2:172.21.15.98:3300/0 conn(0x7f5a8c06c370 msgr2=0x7f5a8c0c3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.052+0000 7f5a8b7fe640 1 --2- 172.21.15.1:0/2363913642 >> v2:172.21.15.98:3300/0 conn(0x7f5a8c06c370 0x7f5a8c0c3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.052+0000 7f5a8b7fe640 1 -- 172.21.15.1:0/2363913642 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8c0c2010 con 0x7f5a8c155890 2024-03-06T07:27:13.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.053+0000 7f5a8b7fe640 1 --2- 172.21.15.1:0/2363913642 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c155890 0x7f5a8c14f030 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c0c8180 tx=0x7f5a8402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fddea38650258d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.053+0000 7f5a89ffb640 1 -- 172.21.15.1:0/2363913642 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8402ca70 con 0x7f5a8c155890 2024-03-06T07:27:13.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.053+0000 7f5a89ffb640 1 -- 172.21.15.1:0/2363913642 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8402cc10 con 0x7f5a8c155890 2024-03-06T07:27:13.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.053+0000 7f5a89ffb640 1 -- 172.21.15.1:0/2363913642 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a840343f0 con 0x7f5a8c155890 2024-03-06T07:27:13.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.053+0000 7f5a9129a640 1 -- 172.21.15.1:0/2363913642 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c155890 msgr2=0x7f5a8c14f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.053+0000 7f5a9129a640 1 --2- 172.21.15.1:0/2363913642 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c155890 0x7f5a8c14f030 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c0c8180 tx=0x7f5a8402bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.053+0000 7f5a9129a640 1 -- 172.21.15.1:0/2363913642 shutdown_connections 2024-03-06T07:27:13.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.053+0000 7f5a9129a640 1 --2- 172.21.15.1:0/2363913642 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c155890 0x7f5a8c14f030 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.053+0000 7f5a9129a640 1 --2- 172.21.15.1:0/2363913642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a8c06dd60 0x7f5a8c155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.053+0000 7f5a9129a640 1 --2- 172.21.15.1:0/2363913642 >> v2:172.21.15.98:3300/0 conn(0x7f5a8c06c370 0x7f5a8c0c3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.053+0000 7f5a9129a640 1 -- 172.21.15.1:0/2363913642 >> 172.21.15.1:0/2363913642 conn(0x7f5a8c0c3cc0 msgr2=0x7f5a8c0c40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.053+0000 7f5a9129a640 1 -- 172.21.15.1:0/2363913642 shutdown_connections 2024-03-06T07:27:13.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.053+0000 7f5a9129a640 1 -- 172.21.15.1:0/2363913642 wait complete. 2024-03-06T07:27:13.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.055+0000 7f5a9129a640 1 Processor -- start 2024-03-06T07:27:13.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.055+0000 7f5a9129a640 1 -- start start 2024-03-06T07:27:13.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.055+0000 7f5a9129a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c06c370 0x7f5a8c150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.055+0000 7f5a9129a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a8c06dd60 0x7f5a8c14e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.055+0000 7f5a9129a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a8c155890 0x7f5a8c1eaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.055+0000 7f5a9129a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c0a9880 con 0x7f5a8c06c370 2024-03-06T07:27:13.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.055+0000 7f5a9129a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c0a95a0 con 0x7f5a8c06dd60 2024-03-06T07:27:13.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.055+0000 7f5a9129a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c0a9da0 con 0x7f5a8c155890 2024-03-06T07:27:13.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.055+0000 7f5a8affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c06c370 0x7f5a8c150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.056+0000 7f5a8affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c06c370 0x7f5a8c150df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59526/0 (socket says 172.21.15.1:59526) 2024-03-06T07:27:13.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.056+0000 7f5a8affd640 1 -- 172.21.15.1:0/573682434 learned_addr learned my addr 172.21.15.1:0/573682434 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:13.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.056+0000 7f5a8a7fc640 1 --2- 172.21.15.1:0/573682434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a8c06dd60 0x7f5a8c14e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.056+0000 7f5a8b7fe640 1 --2- 172.21.15.1:0/573682434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a8c155890 0x7f5a8c1eaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.056+0000 7f5a8affd640 1 -- 172.21.15.1:0/573682434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a8c155890 msgr2=0x7f5a8c1eaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.056+0000 7f5a8affd640 1 --2- 172.21.15.1:0/573682434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a8c155890 0x7f5a8c1eaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.056+0000 7f5a8affd640 1 -- 172.21.15.1:0/573682434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a8c06dd60 msgr2=0x7f5a8c14e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.056+0000 7f5a8affd640 1 --2- 172.21.15.1:0/573682434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a8c06dd60 0x7f5a8c14e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.056+0000 7f5a8affd640 1 -- 172.21.15.1:0/573682434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8c0c6470 con 0x7f5a8c06c370 2024-03-06T07:27:13.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.056+0000 7f5a8affd640 1 --2- 172.21.15.1:0/573682434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c06c370 0x7f5a8c150df0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00e9b0 tx=0x7f5a7c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.056+0000 7f5a737fe640 1 -- 172.21.15.1:0/573682434 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c017070 con 0x7f5a8c06c370 2024-03-06T07:27:13.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.056+0000 7f5a737fe640 1 -- 172.21.15.1:0/573682434 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7c00c9a0 con 0x7f5a8c06c370 2024-03-06T07:27:13.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.056+0000 7f5a9129a640 1 -- 172.21.15.1:0/573682434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8c0c6750 con 0x7f5a8c06c370 2024-03-06T07:27:13.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.056+0000 7f5a737fe640 1 -- 172.21.15.1:0/573682434 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c00cd90 con 0x7f5a8c06c370 2024-03-06T07:27:13.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.056+0000 7f5a9129a640 1 -- 172.21.15.1:0/573682434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8c0c6d10 con 0x7f5a8c06c370 2024-03-06T07:27:13.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.057+0000 7f5a9129a640 1 -- 172.21.15.1:0/573682434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a8c0ad560 con 0x7f5a8c06c370 2024-03-06T07:27:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.058+0000 7f5a737fe640 1 -- 172.21.15.1:0/573682434 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a7c01a070 con 0x7f5a8c06c370 2024-03-06T07:27:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.058+0000 7f5a737fe640 1 --2- 172.21.15.1:0/573682434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a7806b4d0 0x7f5a7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.058+0000 7f5a737fe640 1 -- 172.21.15.1:0/573682434 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5a7c08a5e0 con 0x7f5a8c06c370 2024-03-06T07:27:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.058+0000 7f5a8a7fc640 1 --2- 172.21.15.1:0/573682434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a7806b4d0 0x7f5a7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.059+0000 7f5a8a7fc640 1 --2- 172.21.15.1:0/573682434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a7806b4d0 0x7f5a7806d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c14ca40 tx=0x7f5a80009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.062+0000 7f5a737fe640 1 -- 172.21.15.1:0/573682434 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a8c0ad560 con 0x7f5a8c06c370 2024-03-06T07:27:13.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.239+0000 7f5a9129a640 1 -- 172.21.15.1:0/573682434 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5a8c0686e0 con 0x7f5a7806b4d0 2024-03-06T07:27:13.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.245+0000 7f5a737fe640 1 -- 172.21.15.1:0/573682434 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252570 (secure 0 0 0) 0x7f5a8c0686e0 con 0x7f5a7806b4d0 2024-03-06T07:27:13.249 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:13.251 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:27:13.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.248+0000 7f5a9129a640 1 -- 172.21.15.1:0/573682434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a7806b4d0 msgr2=0x7f5a7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.248+0000 7f5a9129a640 1 --2- 172.21.15.1:0/573682434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a7806b4d0 0x7f5a7806d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c14ca40 tx=0x7f5a80009190 comp rx=0 tx=0).stop 2024-03-06T07:27:13.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.248+0000 7f5a9129a640 1 -- 172.21.15.1:0/573682434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c06c370 msgr2=0x7f5a8c150df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.248+0000 7f5a9129a640 1 --2- 172.21.15.1:0/573682434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c06c370 0x7f5a8c150df0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00e9b0 tx=0x7f5a7c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:27:13.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.248+0000 7f5a9129a640 1 -- 172.21.15.1:0/573682434 shutdown_connections 2024-03-06T07:27:13.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.248+0000 7f5a9129a640 1 --2- 172.21.15.1:0/573682434 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a7806b4d0 0x7f5a7806d990 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.248+0000 7f5a9129a640 1 --2- 172.21.15.1:0/573682434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a8c155890 0x7f5a8c1eaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.248+0000 7f5a9129a640 1 --2- 172.21.15.1:0/573682434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a8c06dd60 0x7f5a8c14e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.248+0000 7f5a9129a640 1 --2- 172.21.15.1:0/573682434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c06c370 0x7f5a8c150df0 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.248+0000 7f5a9129a640 1 -- 172.21.15.1:0/573682434 >> 172.21.15.1:0/573682434 conn(0x7f5a8c0c3cc0 msgr2=0x7f5a8c071d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.248+0000 7f5a9129a640 1 -- 172.21.15.1:0/573682434 shutdown_connections 2024-03-06T07:27:13.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.248+0000 7f5a9129a640 1 -- 172.21.15.1:0/573682434 wait complete. 2024-03-06T07:27:13.261 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-03-06T07:27:12.786027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":266556,"kb_used_data":17088,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754708164,"statfs":{"total":773094113280,"available":772821159936,"internally_reserved":0,"allocated":17498112,"data_stored":9637204,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002244"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211082+0000","last_change":"2024-03-06T07:27:05.285459+0000","last_active":"2024-03-06T07:27:07.211082+0000","last_peered":"2024-03-06T07:27:07.211082+0000","last_clean":"2024-03-06T07:27:07.211082+0000","last_became_active":"2024-03-06T07:27:05.284664+0000","last_became_peered":"2024-03-06T07:27:05.284664+0000","last_unstale":"2024-03-06T07:27:07.211082+0000","last_undegraded":"2024-03-06T07:27:07.211082+0000","last_fullsized":"2024-03-06T07:27:07.211082+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:53.626511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.279284+0000","last_change":"2024-03-06T07:27:05.273546+0000","last_active":"2024-03-06T07:27:06.279284+0000","last_peered":"2024-03-06T07:27:06.279284+0000","last_clean":"2024-03-06T07:27:06.279284+0000","last_became_active":"2024-03-06T07:27:05.273219+0000","last_became_peered":"2024-03-06T07:27:05.273219+0000","last_unstale":"2024-03-06T07:27:06.279284+0000","last_undegraded":"2024-03-06T07:27:06.279284+0000","last_fullsized":"2024-03-06T07:27:06.279284+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:42.836787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.279841+0000","last_change":"2024-03-06T07:27:05.292546+0000","last_active":"2024-03-06T07:27:06.279841+0000","last_peered":"2024-03-06T07:27:06.279841+0000","last_clean":"2024-03-06T07:27:06.279841+0000","last_became_active":"2024-03-06T07:27:05.291976+0000","last_became_peered":"2024-03-06T07:27:05.291976+0000","last_unstale":"2024-03-06T07:27:06.279841+0000","last_undegraded":"2024-03-06T07:27:06.279841+0000","last_fullsized":"2024-03-06T07:27:06.279841+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:08.977664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210524+0000","last_change":"2024-03-06T07:27:05.277990+0000","last_active":"2024-03-06T07:27:07.210524+0000","last_peered":"2024-03-06T07:27:07.210524+0000","last_clean":"2024-03-06T07:27:07.210524+0000","last_became_active":"2024-03-06T07:27:05.277695+0000","last_became_peered":"2024-03-06T07:27:05.277695+0000","last_unstale":"2024-03-06T07:27:07.210524+0000","last_undegraded":"2024-03-06T07:27:07.210524+0000","last_fullsized":"2024-03-06T07:27:07.210524+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:17.331331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.276988+0000","last_change":"2024-03-06T07:27:05.275782+0000","last_active":"2024-03-06T07:27:06.276988+0000","last_peered":"2024-03-06T07:27:06.276988+0000","last_clean":"2024-03-06T07:27:06.276988+0000","last_became_active":"2024-03-06T07:27:05.275466+0000","last_became_peered":"2024-03-06T07:27:05.275466+0000","last_unstale":"2024-03-06T07:27:06.276988+0000","last_undegraded":"2024-03-06T07:27:06.276988+0000","last_fullsized":"2024-03-06T07:27:06.276988+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:29.302517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210296+0000","last_change":"2024-03-06T07:27:05.292784+0000","last_active":"2024-03-06T07:27:07.210296+0000","last_peered":"2024-03-06T07:27:07.210296+0000","last_clean":"2024-03-06T07:27:07.210296+0000","last_became_active":"2024-03-06T07:27:05.292471+0000","last_became_peered":"2024-03-06T07:27:05.292471+0000","last_unstale":"2024-03-06T07:27:07.210296+0000","last_undegraded":"2024-03-06T07:27:07.210296+0000","last_fullsized":"2024-03-06T07:27:07.210296+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:00.959968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.276779+0000","last_change":"2024-03-06T07:27:05.275273+0000","last_active":"2024-03-06T07:27:06.276779+0000","last_peered":"2024-03-06T07:27:06.276779+0000","last_clean":"2024-03-06T07:27:06.276779+0000","last_became_active":"2024-03-06T07:27:05.274527+0000","last_became_peered":"2024-03-06T07:27:05.274527+0000","last_unstale":"2024-03-06T07:27:06.276779+0000","last_undegraded":"2024-03-06T07:27:06.276779+0000","last_fullsized":"2024-03-06T07:27:06.276779+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:02.050928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210509+0000","last_change":"2024-03-06T07:27:05.293419+0000","last_active":"2024-03-06T07:27:07.210509+0000","last_peered":"2024-03-06T07:27:07.210509+0000","last_clean":"2024-03-06T07:27:07.210509+0000","last_became_active":"2024-03-06T07:27:05.293137+0000","last_became_peered":"2024-03-06T07:27:05.293137+0000","last_unstale":"2024-03-06T07:27:07.210509+0000","last_undegraded":"2024-03-06T07:27:07.210509+0000","last_fullsized":"2024-03-06T07:27:07.210509+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:25.687158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.209356+0000","last_change":"2024-03-06T07:27:05.289503+0000","last_active":"2024-03-06T07:27:07.209356+0000","last_peered":"2024-03-06T07:27:07.209356+0000","last_clean":"2024-03-06T07:27:07.209356+0000","last_became_active":"2024-03-06T07:27:05.289219+0000","last_became_peered":"2024-03-06T07:27:05.289219+0000","last_unstale":"2024-03-06T07:27:07.209356+0000","last_undegraded":"2024-03-06T07:27:07.209356+0000","last_fullsized":"2024-03-06T07:27:07.209356+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:52.933999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210865+0000","last_change":"2024-03-06T07:27:05.279901+0000","last_active":"2024-03-06T07:27:07.210865+0000","last_peered":"2024-03-06T07:27:07.210865+0000","last_clean":"2024-03-06T07:27:07.210865+0000","last_became_active":"2024-03-06T07:27:05.279356+0000","last_became_peered":"2024-03-06T07:27:05.279356+0000","last_unstale":"2024-03-06T07:27:07.210865+0000","last_undegraded":"2024-03-06T07:27:07.210865+0000","last_fullsized":"2024-03-06T07:27:07.210865+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:18.247235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.279961+0000","last_change":"2024-03-06T07:27:05.292569+0000","last_active":"2024-03-06T07:27:06.279961+0000","last_peered":"2024-03-06T07:27:06.279961+0000","last_clean":"2024-03-06T07:27:06.279961+0000","last_became_active":"2024-03-06T07:27:05.292187+0000","last_became_peered":"2024-03-06T07:27:05.292187+0000","last_unstale":"2024-03-06T07:27:06.279961+0000","last_undegraded":"2024-03-06T07:27:06.279961+0000","last_fullsized":"2024-03-06T07:27:06.279961+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:46.180033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.277119+0000","last_change":"2024-03-06T07:27:05.284208+0000","last_active":"2024-03-06T07:27:06.277119+0000","last_peered":"2024-03-06T07:27:06.277119+0000","last_clean":"2024-03-06T07:27:06.277119+0000","last_became_active":"2024-03-06T07:27:05.283881+0000","last_became_peered":"2024-03-06T07:27:05.283881+0000","last_unstale":"2024-03-06T07:27:06.277119+0000","last_undegraded":"2024-03-06T07:27:06.277119+0000","last_fullsized":"2024-03-06T07:27:06.277119+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:14:16.759896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210623+0000","last_change":"2024-03-06T07:27:05.280940+0000","last_active":"2024-03-06T07:27:07.210623+0000","last_peered":"2024-03-06T07:27:07.210623+0000","last_clean":"2024-03-06T07:27:07.210623+0000","last_became_active":"2024-03-06T07:27:05.280133+0000","last_became_peered":"2024-03-06T07:27:05.280133+0000","last_unstale":"2024-03-06T07:27:07.210623+0000","last_undegraded":"2024-03-06T07:27:07.210623+0000","last_fullsized":"2024-03-06T07:27:07.210623+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:16.569605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210530+0000","last_change":"2024-03-06T07:27:05.288617+0000","last_active":"2024-03-06T07:27:07.210530+0000","last_peered":"2024-03-06T07:27:07.210530+0000","last_clean":"2024-03-06T07:27:07.210530+0000","last_became_active":"2024-03-06T07:27:05.288351+0000","last_became_peered":"2024-03-06T07:27:05.288351+0000","last_unstale":"2024-03-06T07:27:07.210530+0000","last_undegraded":"2024-03-06T07:27:07.210530+0000","last_fullsized":"2024-03-06T07:27:07.210530+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:10.871951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.932111+0000","last_change":"2024-03-06T07:27:05.273806+0000","last_active":"2024-03-06T07:27:06.932111+0000","last_peered":"2024-03-06T07:27:06.932111+0000","last_clean":"2024-03-06T07:27:06.932111+0000","last_became_active":"2024-03-06T07:27:05.273536+0000","last_became_peered":"2024-03-06T07:27:05.273536+0000","last_unstale":"2024-03-06T07:27:06.932111+0000","last_undegraded":"2024-03-06T07:27:06.932111+0000","last_fullsized":"2024-03-06T07:27:06.932111+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:23.706475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.279467+0000","last_change":"2024-03-06T07:27:05.291182+0000","last_active":"2024-03-06T07:27:06.279467+0000","last_peered":"2024-03-06T07:27:06.279467+0000","last_clean":"2024-03-06T07:27:06.279467+0000","last_became_active":"2024-03-06T07:27:05.290864+0000","last_became_peered":"2024-03-06T07:27:05.290864+0000","last_unstale":"2024-03-06T07:27:06.279467+0000","last_undegraded":"2024-03-06T07:27:06.279467+0000","last_fullsized":"2024-03-06T07:27:06.279467+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:49.517614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.209462+0000","last_change":"2024-03-06T07:27:05.281372+0000","last_active":"2024-03-06T07:27:07.209462+0000","last_peered":"2024-03-06T07:27:07.209462+0000","last_clean":"2024-03-06T07:27:07.209462+0000","last_became_active":"2024-03-06T07:27:05.281098+0000","last_became_peered":"2024-03-06T07:27:05.281098+0000","last_unstale":"2024-03-06T07:27:07.209462+0000","last_undegraded":"2024-03-06T07:27:07.209462+0000","last_fullsized":"2024-03-06T07:27:07.209462+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:00.491268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.209388+0000","last_change":"2024-03-06T07:27:05.295592+0000","last_active":"2024-03-06T07:27:07.209388+0000","last_peered":"2024-03-06T07:27:07.209388+0000","last_clean":"2024-03-06T07:27:07.209388+0000","last_became_active":"2024-03-06T07:27:05.295259+0000","last_became_peered":"2024-03-06T07:27:05.295259+0000","last_unstale":"2024-03-06T07:27:07.209388+0000","last_undegraded":"2024-03-06T07:27:07.209388+0000","last_fullsized":"2024-03-06T07:27:07.209388+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:15.077030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211132+0000","last_change":"2024-03-06T07:27:05.281128+0000","last_active":"2024-03-06T07:27:07.211132+0000","last_peered":"2024-03-06T07:27:07.211132+0000","last_clean":"2024-03-06T07:27:07.211132+0000","last_became_active":"2024-03-06T07:27:05.280767+0000","last_became_peered":"2024-03-06T07:27:05.280767+0000","last_unstale":"2024-03-06T07:27:07.211132+0000","last_undegraded":"2024-03-06T07:27:07.211132+0000","last_fullsized":"2024-03-06T07:27:07.211132+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:32.604270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211166+0000","last_change":"2024-03-06T07:27:05.283713+0000","last_active":"2024-03-06T07:27:07.211166+0000","last_peered":"2024-03-06T07:27:07.211166+0000","last_clean":"2024-03-06T07:27:07.211166+0000","last_became_active":"2024-03-06T07:27:05.283434+0000","last_became_peered":"2024-03-06T07:27:05.283434+0000","last_unstale":"2024-03-06T07:27:07.211166+0000","last_undegraded":"2024-03-06T07:27:07.211166+0000","last_fullsized":"2024-03-06T07:27:07.211166+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:42.244828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210882+0000","last_change":"2024-03-06T07:27:05.287502+0000","last_active":"2024-03-06T07:27:07.210882+0000","last_peered":"2024-03-06T07:27:07.210882+0000","last_clean":"2024-03-06T07:27:07.210882+0000","last_became_active":"2024-03-06T07:27:05.287194+0000","last_became_peered":"2024-03-06T07:27:05.287194+0000","last_unstale":"2024-03-06T07:27:07.210882+0000","last_undegraded":"2024-03-06T07:27:07.210882+0000","last_fullsized":"2024-03-06T07:27:07.210882+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:56.900462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210806+0000","last_change":"2024-03-06T07:27:05.287966+0000","last_active":"2024-03-06T07:27:07.210806+0000","last_peered":"2024-03-06T07:27:07.210806+0000","last_clean":"2024-03-06T07:27:07.210806+0000","last_became_active":"2024-03-06T07:27:05.287699+0000","last_became_peered":"2024-03-06T07:27:05.287699+0000","last_unstale":"2024-03-06T07:27:07.210806+0000","last_undegraded":"2024-03-06T07:27:07.210806+0000","last_fullsized":"2024-03-06T07:27:07.210806+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:49.873776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.278928+0000","last_change":"2024-03-06T07:27:05.286872+0000","last_active":"2024-03-06T07:27:06.278928+0000","last_peered":"2024-03-06T07:27:06.278928+0000","last_clean":"2024-03-06T07:27:06.278928+0000","last_became_active":"2024-03-06T07:27:05.286573+0000","last_became_peered":"2024-03-06T07:27:05.286573+0000","last_unstale":"2024-03-06T07:27:06.278928+0000","last_undegraded":"2024-03-06T07:27:06.278928+0000","last_fullsized":"2024-03-06T07:27:06.278928+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:30.513586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.276716+0000","last_change":"2024-03-06T07:27:05.278963+0000","last_active":"2024-03-06T07:27:06.276716+0000","last_peered":"2024-03-06T07:27:06.276716+0000","last_clean":"2024-03-06T07:27:06.276716+0000","last_became_active":"2024-03-06T07:27:05.278681+0000","last_became_peered":"2024-03-06T07:27:05.278681+0000","last_unstale":"2024-03-06T07:27:06.276716+0000","last_undegraded":"2024-03-06T07:27:06.276716+0000","last_fullsized":"2024-03-06T07:27:06.276716+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:12.820044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.209497+0000","last_change":"2024-03-06T07:27:05.295854+0000","last_active":"2024-03-06T07:27:07.209497+0000","last_peered":"2024-03-06T07:27:07.209497+0000","last_clean":"2024-03-06T07:27:07.209497+0000","last_became_active":"2024-03-06T07:27:05.295550+0000","last_became_peered":"2024-03-06T07:27:05.295550+0000","last_unstale":"2024-03-06T07:27:07.209497+0000","last_undegraded":"2024-03-06T07:27:07.209497+0000","last_fullsized":"2024-03-06T07:27:07.209497+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:56.217917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211921+0000","last_change":"2024-03-06T07:27:05.286426+0000","last_active":"2024-03-06T07:27:07.211921+0000","last_peered":"2024-03-06T07:27:07.211921+0000","last_clean":"2024-03-06T07:27:07.211921+0000","last_became_active":"2024-03-06T07:27:05.285720+0000","last_became_peered":"2024-03-06T07:27:05.285720+0000","last_unstale":"2024-03-06T07:27:07.211921+0000","last_undegraded":"2024-03-06T07:27:07.211921+0000","last_fullsized":"2024-03-06T07:27:07.211921+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:27.902745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.280046+0000","last_change":"2024-03-06T07:27:05.292301+0000","last_active":"2024-03-06T07:27:06.280046+0000","last_peered":"2024-03-06T07:27:06.280046+0000","last_clean":"2024-03-06T07:27:06.280046+0000","last_became_active":"2024-03-06T07:27:05.291910+0000","last_became_peered":"2024-03-06T07:27:05.291910+0000","last_unstale":"2024-03-06T07:27:06.280046+0000","last_undegraded":"2024-03-06T07:27:06.280046+0000","last_fullsized":"2024-03-06T07:27:06.280046+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:21.091752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.277107+0000","last_change":"2024-03-06T07:27:05.274128+0000","last_active":"2024-03-06T07:27:06.277107+0000","last_peered":"2024-03-06T07:27:06.277107+0000","last_clean":"2024-03-06T07:27:06.277107+0000","last_became_active":"2024-03-06T07:27:05.273849+0000","last_became_peered":"2024-03-06T07:27:05.273849+0000","last_unstale":"2024-03-06T07:27:06.277107+0000","last_undegraded":"2024-03-06T07:27:06.277107+0000","last_fullsized":"2024-03-06T07:27:06.277107+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:49.475966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.209197+0000","last_change":"2024-03-06T07:27:05.272548+0000","last_active":"2024-03-06T07:27:07.209197+0000","last_peered":"2024-03-06T07:27:07.209197+0000","last_clean":"2024-03-06T07:27:07.209197+0000","last_became_active":"2024-03-06T07:27:05.272217+0000","last_became_peered":"2024-03-06T07:27:05.272217+0000","last_unstale":"2024-03-06T07:27:07.209197+0000","last_undegraded":"2024-03-06T07:27:07.209197+0000","last_fullsized":"2024-03-06T07:27:07.209197+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:51.420709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.932044+0000","last_change":"2024-03-06T07:27:05.271495+0000","last_active":"2024-03-06T07:27:06.932044+0000","last_peered":"2024-03-06T07:27:06.932044+0000","last_clean":"2024-03-06T07:27:06.932044+0000","last_became_active":"2024-03-06T07:27:05.271203+0000","last_became_peered":"2024-03-06T07:27:05.271203+0000","last_unstale":"2024-03-06T07:27:06.932044+0000","last_undegraded":"2024-03-06T07:27:06.932044+0000","last_fullsized":"2024-03-06T07:27:06.932044+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:19.856061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210188+0000","last_change":"2024-03-06T07:27:05.277784+0000","last_active":"2024-03-06T07:27:07.210188+0000","last_peered":"2024-03-06T07:27:07.210188+0000","last_clean":"2024-03-06T07:27:07.210188+0000","last_became_active":"2024-03-06T07:27:05.277520+0000","last_became_peered":"2024-03-06T07:27:05.277520+0000","last_unstale":"2024-03-06T07:27:07.210188+0000","last_undegraded":"2024-03-06T07:27:07.210188+0000","last_fullsized":"2024-03-06T07:27:07.210188+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:18.010987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210154+0000","last_change":"2024-03-06T07:27:05.280699+0000","last_active":"2024-03-06T07:27:07.210154+0000","last_peered":"2024-03-06T07:27:07.210154+0000","last_clean":"2024-03-06T07:27:07.210154+0000","last_became_active":"2024-03-06T07:27:05.280425+0000","last_became_peered":"2024-03-06T07:27:05.280425+0000","last_unstale":"2024-03-06T07:27:07.210154+0000","last_undegraded":"2024-03-06T07:27:07.210154+0000","last_fullsized":"2024-03-06T07:27:07.210154+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:38.719657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.278843+0000","last_change":"2024-03-06T07:27:05.286829+0000","last_active":"2024-03-06T07:27:06.278843+0000","last_peered":"2024-03-06T07:27:06.278843+0000","last_clean":"2024-03-06T07:27:06.278843+0000","last_became_active":"2024-03-06T07:27:05.286524+0000","last_became_peered":"2024-03-06T07:27:05.286524+0000","last_unstale":"2024-03-06T07:27:06.278843+0000","last_undegraded":"2024-03-06T07:27:06.278843+0000","last_fullsized":"2024-03-06T07:27:06.278843+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:24.250434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"170'4","reported_seq":18,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.932303+0000","last_change":"2024-03-06T07:27:04.271712+0000","last_active":"2024-03-06T07:27:06.932303+0000","last_peered":"2024-03-06T07:27:06.932303+0000","last_clean":"2024-03-06T07:27:06.932303+0000","last_became_active":"2024-03-06T07:27:04.271429+0000","last_became_peered":"2024-03-06T07:27:04.271429+0000","last_unstale":"2024-03-06T07:27:06.932303+0000","last_undegraded":"2024-03-06T07:27:06.932303+0000","last_fullsized":"2024-03-06T07:27:06.932303+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:15.996894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211491+0000","last_change":"2024-03-06T07:27:05.285443+0000","last_active":"2024-03-06T07:27:07.211491+0000","last_peered":"2024-03-06T07:27:07.211491+0000","last_clean":"2024-03-06T07:27:07.211491+0000","last_became_active":"2024-03-06T07:27:05.285180+0000","last_became_peered":"2024-03-06T07:27:05.285180+0000","last_unstale":"2024-03-06T07:27:07.211491+0000","last_undegraded":"2024-03-06T07:27:07.211491+0000","last_fullsized":"2024-03-06T07:27:07.211491+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:55.169285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"170'1","reported_seq":15,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211854+0000","last_change":"2024-03-06T07:27:04.266276+0000","last_active":"2024-03-06T07:27:07.211854+0000","last_peered":"2024-03-06T07:27:07.211854+0000","last_clean":"2024-03-06T07:27:07.211854+0000","last_became_active":"2024-03-06T07:27:04.265418+0000","last_became_peered":"2024-03-06T07:27:04.265418+0000","last_unstale":"2024-03-06T07:27:07.211854+0000","last_undegraded":"2024-03-06T07:27:07.211854+0000","last_fullsized":"2024-03-06T07:27:07.211854+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:22.702218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211145+0000","last_change":"2024-03-06T07:27:05.278602+0000","last_active":"2024-03-06T07:27:07.211145+0000","last_peered":"2024-03-06T07:27:07.211145+0000","last_clean":"2024-03-06T07:27:07.211145+0000","last_became_active":"2024-03-06T07:27:05.278303+0000","last_became_peered":"2024-03-06T07:27:05.278303+0000","last_unstale":"2024-03-06T07:27:07.211145+0000","last_undegraded":"2024-03-06T07:27:07.211145+0000","last_fullsized":"2024-03-06T07:27:07.211145+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:36.167569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"170'2","reported_seq":16,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.279212+0000","last_change":"2024-03-06T07:27:04.262189+0000","last_active":"2024-03-06T07:27:06.279212+0000","last_peered":"2024-03-06T07:27:06.279212+0000","last_clean":"2024-03-06T07:27:06.279212+0000","last_became_active":"2024-03-06T07:27:04.261909+0000","last_became_peered":"2024-03-06T07:27:04.261909+0000","last_unstale":"2024-03-06T07:27:06.279212+0000","last_undegraded":"2024-03-06T07:27:06.279212+0000","last_fullsized":"2024-03-06T07:27:06.279212+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:14.072773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.277065+0000","last_change":"2024-03-06T07:27:05.269100+0000","last_active":"2024-03-06T07:27:06.277065+0000","last_peered":"2024-03-06T07:27:06.277065+0000","last_clean":"2024-03-06T07:27:06.277065+0000","last_became_active":"2024-03-06T07:27:05.268810+0000","last_became_peered":"2024-03-06T07:27:05.268810+0000","last_unstale":"2024-03-06T07:27:06.277065+0000","last_undegraded":"2024-03-06T07:27:06.277065+0000","last_fullsized":"2024-03-06T07:27:06.277065+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:26.634842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"170'2","reported_seq":16,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.209310+0000","last_change":"2024-03-06T07:27:04.270472+0000","last_active":"2024-03-06T07:27:07.209310+0000","last_peered":"2024-03-06T07:27:07.209310+0000","last_clean":"2024-03-06T07:27:07.209310+0000","last_became_active":"2024-03-06T07:27:04.270185+0000","last_became_peered":"2024-03-06T07:27:04.270185+0000","last_unstale":"2024-03-06T07:27:07.209310+0000","last_undegraded":"2024-03-06T07:27:07.209310+0000","last_fullsized":"2024-03-06T07:27:07.209310+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:16.327074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.932294+0000","last_change":"2024-03-06T07:27:05.276044+0000","last_active":"2024-03-06T07:27:06.932294+0000","last_peered":"2024-03-06T07:27:06.932294+0000","last_clean":"2024-03-06T07:27:06.932294+0000","last_became_active":"2024-03-06T07:27:05.275779+0000","last_became_peered":"2024-03-06T07:27:05.275779+0000","last_unstale":"2024-03-06T07:27:06.932294+0000","last_undegraded":"2024-03-06T07:27:06.932294+0000","last_fullsized":"2024-03-06T07:27:06.932294+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:49.194120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"170'1","reported_seq":15,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210729+0000","last_change":"2024-03-06T07:27:04.272491+0000","last_active":"2024-03-06T07:27:07.210729+0000","last_peered":"2024-03-06T07:27:07.210729+0000","last_clean":"2024-03-06T07:27:07.210729+0000","last_became_active":"2024-03-06T07:27:04.272190+0000","last_became_peered":"2024-03-06T07:27:04.272190+0000","last_unstale":"2024-03-06T07:27:07.210729+0000","last_undegraded":"2024-03-06T07:27:07.210729+0000","last_fullsized":"2024-03-06T07:27:07.210729+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:52.622254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.932322+0000","last_change":"2024-03-06T07:27:05.276200+0000","last_active":"2024-03-06T07:27:06.932322+0000","last_peered":"2024-03-06T07:27:06.932322+0000","last_clean":"2024-03-06T07:27:06.932322+0000","last_became_active":"2024-03-06T07:27:05.275507+0000","last_became_peered":"2024-03-06T07:27:05.275507+0000","last_unstale":"2024-03-06T07:27:06.932322+0000","last_undegraded":"2024-03-06T07:27:06.932322+0000","last_fullsized":"2024-03-06T07:27:06.932322+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:33.856852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210308+0000","last_change":"2024-03-06T07:27:04.273573+0000","last_active":"2024-03-06T07:27:07.210308+0000","last_peered":"2024-03-06T07:27:07.210308+0000","last_clean":"2024-03-06T07:27:07.210308+0000","last_became_active":"2024-03-06T07:27:04.273253+0000","last_became_peered":"2024-03-06T07:27:04.273253+0000","last_unstale":"2024-03-06T07:27:07.210308+0000","last_undegraded":"2024-03-06T07:27:07.210308+0000","last_fullsized":"2024-03-06T07:27:07.210308+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:21.965084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.209592+0000","last_change":"2024-03-06T07:27:05.289447+0000","last_active":"2024-03-06T07:27:07.209592+0000","last_peered":"2024-03-06T07:27:07.209592+0000","last_clean":"2024-03-06T07:27:07.209592+0000","last_became_active":"2024-03-06T07:27:05.289107+0000","last_became_peered":"2024-03-06T07:27:05.289107+0000","last_unstale":"2024-03-06T07:27:07.209592+0000","last_undegraded":"2024-03-06T07:27:07.209592+0000","last_fullsized":"2024-03-06T07:27:07.209592+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:31.255403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.279495+0000","last_change":"2024-03-06T07:27:04.271894+0000","last_active":"2024-03-06T07:27:06.279495+0000","last_peered":"2024-03-06T07:27:06.279495+0000","last_clean":"2024-03-06T07:27:06.279495+0000","last_became_active":"2024-03-06T07:27:04.271305+0000","last_became_peered":"2024-03-06T07:27:04.271305+0000","last_unstale":"2024-03-06T07:27:06.279495+0000","last_undegraded":"2024-03-06T07:27:06.279495+0000","last_fullsized":"2024-03-06T07:27:06.279495+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:55.694331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.931928+0000","last_change":"2024-03-06T07:27:05.281211+0000","last_active":"2024-03-06T07:27:06.931928+0000","last_peered":"2024-03-06T07:27:06.931928+0000","last_clean":"2024-03-06T07:27:06.931928+0000","last_became_active":"2024-03-06T07:27:05.279990+0000","last_became_peered":"2024-03-06T07:27:05.279990+0000","last_unstale":"2024-03-06T07:27:06.931928+0000","last_undegraded":"2024-03-06T07:27:06.931928+0000","last_fullsized":"2024-03-06T07:27:06.931928+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:03.566788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.277492+0000","last_change":"2024-03-06T07:27:04.269100+0000","last_active":"2024-03-06T07:27:06.277492+0000","last_peered":"2024-03-06T07:27:06.277492+0000","last_clean":"2024-03-06T07:27:06.277492+0000","last_became_active":"2024-03-06T07:27:04.268811+0000","last_became_peered":"2024-03-06T07:27:04.268811+0000","last_unstale":"2024-03-06T07:27:06.277492+0000","last_undegraded":"2024-03-06T07:27:06.277492+0000","last_fullsized":"2024-03-06T07:27:06.277492+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:26.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.279781+0000","last_change":"2024-03-06T07:27:05.291339+0000","last_active":"2024-03-06T07:27:06.279781+0000","last_peered":"2024-03-06T07:27:06.279781+0000","last_clean":"2024-03-06T07:27:06.279781+0000","last_became_active":"2024-03-06T07:27:05.291058+0000","last_became_peered":"2024-03-06T07:27:05.291058+0000","last_unstale":"2024-03-06T07:27:06.279781+0000","last_undegraded":"2024-03-06T07:27:06.279781+0000","last_fullsized":"2024-03-06T07:27:06.279781+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:16.569605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211566+0000","last_change":"2024-03-06T07:27:04.271341+0000","last_active":"2024-03-06T07:27:07.211566+0000","last_peered":"2024-03-06T07:27:07.211566+0000","last_clean":"2024-03-06T07:27:07.211566+0000","last_became_active":"2024-03-06T07:27:04.271023+0000","last_became_peered":"2024-03-06T07:27:04.271023+0000","last_unstale":"2024-03-06T07:27:07.211566+0000","last_undegraded":"2024-03-06T07:27:07.211566+0000","last_fullsized":"2024-03-06T07:27:07.211566+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:54.165028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.209209+0000","last_change":"2024-03-06T07:27:05.289670+0000","last_active":"2024-03-06T07:27:07.209209+0000","last_peered":"2024-03-06T07:27:07.209209+0000","last_clean":"2024-03-06T07:27:07.209209+0000","last_became_active":"2024-03-06T07:27:05.289394+0000","last_became_peered":"2024-03-06T07:27:05.289394+0000","last_unstale":"2024-03-06T07:27:07.209209+0000","last_undegraded":"2024-03-06T07:27:07.209209+0000","last_fullsized":"2024-03-06T07:27:07.209209+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:45.869166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"170'3","reported_seq":17,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.209389+0000","last_change":"2024-03-06T07:27:04.264830+0000","last_active":"2024-03-06T07:27:07.209389+0000","last_peered":"2024-03-06T07:27:07.209389+0000","last_clean":"2024-03-06T07:27:07.209389+0000","last_became_active":"2024-03-06T07:27:04.264562+0000","last_became_peered":"2024-03-06T07:27:04.264562+0000","last_unstale":"2024-03-06T07:27:07.209389+0000","last_undegraded":"2024-03-06T07:27:07.209389+0000","last_fullsized":"2024-03-06T07:27:07.209389+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:59.955711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.277145+0000","last_change":"2024-03-06T07:27:05.284450+0000","last_active":"2024-03-06T07:27:06.277145+0000","last_peered":"2024-03-06T07:27:06.277145+0000","last_clean":"2024-03-06T07:27:06.277145+0000","last_became_active":"2024-03-06T07:27:05.284164+0000","last_became_peered":"2024-03-06T07:27:05.284164+0000","last_unstale":"2024-03-06T07:27:06.277145+0000","last_undegraded":"2024-03-06T07:27:06.277145+0000","last_fullsized":"2024-03-06T07:27:06.277145+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:25.687158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"170'6","reported_seq":20,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210634+0000","last_change":"2024-03-06T07:27:04.273664+0000","last_active":"2024-03-06T07:27:07.210634+0000","last_peered":"2024-03-06T07:27:07.210634+0000","last_clean":"2024-03-06T07:27:07.210634+0000","last_became_active":"2024-03-06T07:27:04.273372+0000","last_became_peered":"2024-03-06T07:27:04.273372+0000","last_unstale":"2024-03-06T07:27:07.210634+0000","last_undegraded":"2024-03-06T07:27:07.210634+0000","last_fullsized":"2024-03-06T07:27:07.210634+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:53.738091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.279186+0000","last_change":"2024-03-06T07:27:05.275290+0000","last_active":"2024-03-06T07:27:06.279186+0000","last_peered":"2024-03-06T07:27:06.279186+0000","last_clean":"2024-03-06T07:27:06.279186+0000","last_became_active":"2024-03-06T07:27:05.274921+0000","last_became_peered":"2024-03-06T07:27:05.274921+0000","last_unstale":"2024-03-06T07:27:06.279186+0000","last_undegraded":"2024-03-06T07:27:06.279186+0000","last_fullsized":"2024-03-06T07:27:06.279186+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:37.335341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"170'2","reported_seq":16,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.278979+0000","last_change":"2024-03-06T07:27:04.263075+0000","last_active":"2024-03-06T07:27:06.278979+0000","last_peered":"2024-03-06T07:27:06.278979+0000","last_clean":"2024-03-06T07:27:06.278979+0000","last_became_active":"2024-03-06T07:27:04.261822+0000","last_became_peered":"2024-03-06T07:27:04.261822+0000","last_unstale":"2024-03-06T07:27:06.278979+0000","last_undegraded":"2024-03-06T07:27:06.278979+0000","last_fullsized":"2024-03-06T07:27:06.278979+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:55.213660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.280078+0000","last_change":"2024-03-06T07:27:05.293140+0000","last_active":"2024-03-06T07:27:06.280078+0000","last_peered":"2024-03-06T07:27:06.280078+0000","last_clean":"2024-03-06T07:27:06.280078+0000","last_became_active":"2024-03-06T07:27:05.292863+0000","last_became_peered":"2024-03-06T07:27:05.292863+0000","last_unstale":"2024-03-06T07:27:06.280078+0000","last_undegraded":"2024-03-06T07:27:06.280078+0000","last_fullsized":"2024-03-06T07:27:06.280078+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:51.289461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.279970+0000","last_change":"2024-03-06T07:27:04.268027+0000","last_active":"2024-03-06T07:27:06.279970+0000","last_peered":"2024-03-06T07:27:06.279970+0000","last_clean":"2024-03-06T07:27:06.279970+0000","last_became_active":"2024-03-06T07:27:04.267726+0000","last_became_peered":"2024-03-06T07:27:04.267726+0000","last_unstale":"2024-03-06T07:27:06.279970+0000","last_undegraded":"2024-03-06T07:27:06.279970+0000","last_fullsized":"2024-03-06T07:27:06.279970+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:14:10.799456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.209421+0000","last_change":"2024-03-06T07:27:05.276254+0000","last_active":"2024-03-06T07:27:07.209421+0000","last_peered":"2024-03-06T07:27:07.209421+0000","last_clean":"2024-03-06T07:27:07.209421+0000","last_became_active":"2024-03-06T07:27:05.275987+0000","last_became_peered":"2024-03-06T07:27:05.275987+0000","last_unstale":"2024-03-06T07:27:07.209421+0000","last_undegraded":"2024-03-06T07:27:07.209421+0000","last_fullsized":"2024-03-06T07:27:07.209421+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:38.321310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.209082+0000","last_change":"2024-03-06T07:27:04.265676+0000","last_active":"2024-03-06T07:27:07.209082+0000","last_peered":"2024-03-06T07:27:07.209082+0000","last_clean":"2024-03-06T07:27:07.209082+0000","last_became_active":"2024-03-06T07:27:04.265374+0000","last_became_peered":"2024-03-06T07:27:04.265374+0000","last_unstale":"2024-03-06T07:27:07.209082+0000","last_undegraded":"2024-03-06T07:27:07.209082+0000","last_fullsized":"2024-03-06T07:27:07.209082+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:48.513357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210312+0000","last_change":"2024-03-06T07:27:05.287985+0000","last_active":"2024-03-06T07:27:07.210312+0000","last_peered":"2024-03-06T07:27:07.210312+0000","last_clean":"2024-03-06T07:27:07.210312+0000","last_became_active":"2024-03-06T07:27:05.287382+0000","last_became_peered":"2024-03-06T07:27:05.287382+0000","last_unstale":"2024-03-06T07:27:07.210312+0000","last_undegraded":"2024-03-06T07:27:07.210312+0000","last_fullsized":"2024-03-06T07:27:07.210312+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:49.517614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"170'8","reported_seq":22,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.276676+0000","last_change":"2024-03-06T07:27:04.269119+0000","last_active":"2024-03-06T07:27:06.276676+0000","last_peered":"2024-03-06T07:27:06.276676+0000","last_clean":"2024-03-06T07:27:06.276676+0000","last_became_active":"2024-03-06T07:27:04.268844+0000","last_became_peered":"2024-03-06T07:27:04.268844+0000","last_unstale":"2024-03-06T07:27:06.276676+0000","last_undegraded":"2024-03-06T07:27:06.276676+0000","last_fullsized":"2024-03-06T07:27:06.276676+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:29.732342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210878+0000","last_change":"2024-03-06T07:27:05.279095+0000","last_active":"2024-03-06T07:27:07.210878+0000","last_peered":"2024-03-06T07:27:07.210878+0000","last_clean":"2024-03-06T07:27:07.210878+0000","last_became_active":"2024-03-06T07:27:05.278801+0000","last_became_peered":"2024-03-06T07:27:05.278801+0000","last_unstale":"2024-03-06T07:27:07.210878+0000","last_undegraded":"2024-03-06T07:27:07.210878+0000","last_fullsized":"2024-03-06T07:27:07.210878+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:08.309142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"170'1","reported_seq":15,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211180+0000","last_change":"2024-03-06T07:27:04.262327+0000","last_active":"2024-03-06T07:27:07.211180+0000","last_peered":"2024-03-06T07:27:07.211180+0000","last_clean":"2024-03-06T07:27:07.211180+0000","last_became_active":"2024-03-06T07:27:04.262046+0000","last_became_peered":"2024-03-06T07:27:04.262046+0000","last_unstale":"2024-03-06T07:27:07.211180+0000","last_undegraded":"2024-03-06T07:27:07.211180+0000","last_fullsized":"2024-03-06T07:27:07.211180+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:48.189863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211657+0000","last_change":"2024-03-06T07:27:05.284611+0000","last_active":"2024-03-06T07:27:07.211657+0000","last_peered":"2024-03-06T07:27:07.211657+0000","last_clean":"2024-03-06T07:27:07.211657+0000","last_became_active":"2024-03-06T07:27:05.284329+0000","last_became_peered":"2024-03-06T07:27:05.284329+0000","last_unstale":"2024-03-06T07:27:07.211657+0000","last_undegraded":"2024-03-06T07:27:07.211657+0000","last_fullsized":"2024-03-06T07:27:07.211657+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:12.947588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"170'2","reported_seq":16,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211941+0000","last_change":"2024-03-06T07:27:04.266893+0000","last_active":"2024-03-06T07:27:07.211941+0000","last_peered":"2024-03-06T07:27:07.211941+0000","last_clean":"2024-03-06T07:27:07.211941+0000","last_became_active":"2024-03-06T07:27:04.265563+0000","last_became_peered":"2024-03-06T07:27:04.265563+0000","last_unstale":"2024-03-06T07:27:07.211941+0000","last_undegraded":"2024-03-06T07:27:07.211941+0000","last_fullsized":"2024-03-06T07:27:07.211941+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:32.852595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211637+0000","last_change":"2024-03-06T07:27:05.286933+0000","last_active":"2024-03-06T07:27:07.211637+0000","last_peered":"2024-03-06T07:27:07.211637+0000","last_clean":"2024-03-06T07:27:07.211637+0000","last_became_active":"2024-03-06T07:27:05.286564+0000","last_became_peered":"2024-03-06T07:27:05.286564+0000","last_unstale":"2024-03-06T07:27:07.211637+0000","last_undegraded":"2024-03-06T07:27:07.211637+0000","last_fullsized":"2024-03-06T07:27:07.211637+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:54.742348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"170'1","reported_seq":15,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.279265+0000","last_change":"2024-03-06T07:27:04.265277+0000","last_active":"2024-03-06T07:27:06.279265+0000","last_peered":"2024-03-06T07:27:06.279265+0000","last_clean":"2024-03-06T07:27:06.279265+0000","last_became_active":"2024-03-06T07:27:04.265018+0000","last_became_peered":"2024-03-06T07:27:04.265018+0000","last_unstale":"2024-03-06T07:27:06.279265+0000","last_undegraded":"2024-03-06T07:27:06.279265+0000","last_fullsized":"2024-03-06T07:27:06.279265+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:59.487011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211257+0000","last_change":"2024-03-06T07:27:05.286988+0000","last_active":"2024-03-06T07:27:07.211257+0000","last_peered":"2024-03-06T07:27:07.211257+0000","last_clean":"2024-03-06T07:27:07.211257+0000","last_became_active":"2024-03-06T07:27:05.284571+0000","last_became_peered":"2024-03-06T07:27:05.284571+0000","last_unstale":"2024-03-06T07:27:07.211257+0000","last_undegraded":"2024-03-06T07:27:07.211257+0000","last_fullsized":"2024-03-06T07:27:07.211257+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:17.063634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.277205+0000","last_change":"2024-03-06T07:27:04.265842+0000","last_active":"2024-03-06T07:27:06.277205+0000","last_peered":"2024-03-06T07:27:06.277205+0000","last_clean":"2024-03-06T07:27:06.277205+0000","last_became_active":"2024-03-06T07:27:04.265568+0000","last_became_peered":"2024-03-06T07:27:04.265568+0000","last_unstale":"2024-03-06T07:27:06.277205+0000","last_undegraded":"2024-03-06T07:27:06.277205+0000","last_fullsized":"2024-03-06T07:27:06.277205+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:38.542889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.279173+0000","last_change":"2024-03-06T07:27:05.274459+0000","last_active":"2024-03-06T07:27:06.279173+0000","last_peered":"2024-03-06T07:27:06.279173+0000","last_clean":"2024-03-06T07:27:06.279173+0000","last_became_active":"2024-03-06T07:27:05.274157+0000","last_became_peered":"2024-03-06T07:27:05.274157+0000","last_unstale":"2024-03-06T07:27:06.279173+0000","last_undegraded":"2024-03-06T07:27:06.279173+0000","last_fullsized":"2024-03-06T07:27:06.279173+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:24.867156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"170'2","reported_seq":16,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.277161+0000","last_change":"2024-03-06T07:27:04.262729+0000","last_active":"2024-03-06T07:27:06.277161+0000","last_peered":"2024-03-06T07:27:06.277161+0000","last_clean":"2024-03-06T07:27:06.277161+0000","last_became_active":"2024-03-06T07:27:04.262484+0000","last_became_peered":"2024-03-06T07:27:04.262484+0000","last_unstale":"2024-03-06T07:27:06.277161+0000","last_undegraded":"2024-03-06T07:27:06.277161+0000","last_fullsized":"2024-03-06T07:27:06.277161+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:30.526595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210970+0000","last_change":"2024-03-06T07:27:05.278488+0000","last_active":"2024-03-06T07:27:07.210970+0000","last_peered":"2024-03-06T07:27:07.210970+0000","last_clean":"2024-03-06T07:27:07.210970+0000","last_became_active":"2024-03-06T07:27:05.278169+0000","last_became_peered":"2024-03-06T07:27:05.278169+0000","last_unstale":"2024-03-06T07:27:07.210970+0000","last_undegraded":"2024-03-06T07:27:07.210970+0000","last_fullsized":"2024-03-06T07:27:07.210970+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:08.977664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.278981+0000","last_change":"2024-03-06T07:27:04.263995+0000","last_active":"2024-03-06T07:27:06.278981+0000","last_peered":"2024-03-06T07:27:06.278981+0000","last_clean":"2024-03-06T07:27:06.278981+0000","last_became_active":"2024-03-06T07:27:04.263629+0000","last_became_peered":"2024-03-06T07:27:04.263629+0000","last_unstale":"2024-03-06T07:27:06.278981+0000","last_undegraded":"2024-03-06T07:27:06.278981+0000","last_fullsized":"2024-03-06T07:27:06.278981+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:37.317053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.209511+0000","last_change":"2024-03-06T07:27:05.278120+0000","last_active":"2024-03-06T07:27:07.209511+0000","last_peered":"2024-03-06T07:27:07.209511+0000","last_clean":"2024-03-06T07:27:07.209511+0000","last_became_active":"2024-03-06T07:27:05.277795+0000","last_became_peered":"2024-03-06T07:27:05.277795+0000","last_unstale":"2024-03-06T07:27:07.209511+0000","last_undegraded":"2024-03-06T07:27:07.209511+0000","last_fullsized":"2024-03-06T07:27:07.209511+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:43.528201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.932117+0000","last_change":"2024-03-06T07:27:04.271282+0000","last_active":"2024-03-06T07:27:06.932117+0000","last_peered":"2024-03-06T07:27:06.932117+0000","last_clean":"2024-03-06T07:27:06.932117+0000","last_became_active":"2024-03-06T07:27:04.270987+0000","last_became_peered":"2024-03-06T07:27:04.270987+0000","last_unstale":"2024-03-06T07:27:06.932117+0000","last_undegraded":"2024-03-06T07:27:06.932117+0000","last_fullsized":"2024-03-06T07:27:06.932117+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:26.449033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211423+0000","last_change":"2024-03-06T07:27:05.287890+0000","last_active":"2024-03-06T07:27:07.211423+0000","last_peered":"2024-03-06T07:27:07.211423+0000","last_clean":"2024-03-06T07:27:07.211423+0000","last_became_active":"2024-03-06T07:27:05.287605+0000","last_became_peered":"2024-03-06T07:27:05.287605+0000","last_unstale":"2024-03-06T07:27:07.211423+0000","last_undegraded":"2024-03-06T07:27:07.211423+0000","last_fullsized":"2024-03-06T07:27:07.211423+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:08.735003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210180+0000","last_change":"2024-03-06T07:27:04.272151+0000","last_active":"2024-03-06T07:27:07.210180+0000","last_peered":"2024-03-06T07:27:07.210180+0000","last_clean":"2024-03-06T07:27:07.210180+0000","last_became_active":"2024-03-06T07:27:04.271894+0000","last_became_peered":"2024-03-06T07:27:04.271894+0000","last_unstale":"2024-03-06T07:27:07.210180+0000","last_undegraded":"2024-03-06T07:27:07.210180+0000","last_fullsized":"2024-03-06T07:27:07.210180+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:14.040144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210678+0000","last_change":"2024-03-06T07:27:05.276409+0000","last_active":"2024-03-06T07:27:07.210678+0000","last_peered":"2024-03-06T07:27:07.210678+0000","last_clean":"2024-03-06T07:27:07.210678+0000","last_became_active":"2024-03-06T07:27:05.276148+0000","last_became_peered":"2024-03-06T07:27:05.276148+0000","last_unstale":"2024-03-06T07:27:07.210678+0000","last_undegraded":"2024-03-06T07:27:07.210678+0000","last_fullsized":"2024-03-06T07:27:07.210678+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:12.908518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"170'2","reported_seq":16,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.209551+0000","last_change":"2024-03-06T07:27:04.268865+0000","last_active":"2024-03-06T07:27:07.209551+0000","last_peered":"2024-03-06T07:27:07.209551+0000","last_clean":"2024-03-06T07:27:07.209551+0000","last_became_active":"2024-03-06T07:27:04.268566+0000","last_became_peered":"2024-03-06T07:27:04.268566+0000","last_unstale":"2024-03-06T07:27:07.209551+0000","last_undegraded":"2024-03-06T07:27:07.209551+0000","last_fullsized":"2024-03-06T07:27:07.209551+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:01.274790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.279857+0000","last_change":"2024-03-06T07:27:05.288883+0000","last_active":"2024-03-06T07:27:06.279857+0000","last_peered":"2024-03-06T07:27:06.279857+0000","last_clean":"2024-03-06T07:27:06.279857+0000","last_became_active":"2024-03-06T07:27:05.288594+0000","last_became_peered":"2024-03-06T07:27:05.288594+0000","last_unstale":"2024-03-06T07:27:06.279857+0000","last_undegraded":"2024-03-06T07:27:06.279857+0000","last_fullsized":"2024-03-06T07:27:06.279857+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:02.279047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"171'1","reported_seq":15,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211021+0000","last_change":"2024-03-06T07:27:04.264784+0000","last_active":"2024-03-06T07:27:07.211021+0000","last_peered":"2024-03-06T07:27:07.211021+0000","last_clean":"2024-03-06T07:27:07.211021+0000","last_became_active":"2024-03-06T07:27:04.264499+0000","last_became_peered":"2024-03-06T07:27:04.264499+0000","last_unstale":"2024-03-06T07:27:07.211021+0000","last_undegraded":"2024-03-06T07:27:07.211021+0000","last_fullsized":"2024-03-06T07:27:07.211021+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:26.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.209159+0000","last_change":"2024-03-06T07:27:05.279272+0000","last_active":"2024-03-06T07:27:07.209159+0000","last_peered":"2024-03-06T07:27:07.209159+0000","last_clean":"2024-03-06T07:27:07.209159+0000","last_became_active":"2024-03-06T07:27:05.278963+0000","last_became_peered":"2024-03-06T07:27:05.278963+0000","last_unstale":"2024-03-06T07:27:07.209159+0000","last_undegraded":"2024-03-06T07:27:07.209159+0000","last_fullsized":"2024-03-06T07:27:07.209159+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:15:44.232856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.932247+0000","last_change":"2024-03-06T07:27:04.269541+0000","last_active":"2024-03-06T07:27:06.932247+0000","last_peered":"2024-03-06T07:27:06.932247+0000","last_clean":"2024-03-06T07:27:06.932247+0000","last_became_active":"2024-03-06T07:27:04.269228+0000","last_became_peered":"2024-03-06T07:27:04.269228+0000","last_unstale":"2024-03-06T07:27:06.932247+0000","last_undegraded":"2024-03-06T07:27:06.932247+0000","last_fullsized":"2024-03-06T07:27:06.932247+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:31.006078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210572+0000","last_change":"2024-03-06T07:27:05.284537+0000","last_active":"2024-03-06T07:27:07.210572+0000","last_peered":"2024-03-06T07:27:07.210572+0000","last_clean":"2024-03-06T07:27:07.210572+0000","last_became_active":"2024-03-06T07:27:05.283331+0000","last_became_peered":"2024-03-06T07:27:05.283331+0000","last_unstale":"2024-03-06T07:27:07.210572+0000","last_undegraded":"2024-03-06T07:27:07.210572+0000","last_fullsized":"2024-03-06T07:27:07.210572+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:52.523273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210554+0000","last_change":"2024-03-06T07:27:04.260635+0000","last_active":"2024-03-06T07:27:07.210554+0000","last_peered":"2024-03-06T07:27:07.210554+0000","last_clean":"2024-03-06T07:27:07.210554+0000","last_became_active":"2024-03-06T07:27:04.260370+0000","last_became_peered":"2024-03-06T07:27:04.260370+0000","last_unstale":"2024-03-06T07:27:07.210554+0000","last_undegraded":"2024-03-06T07:27:07.210554+0000","last_fullsized":"2024-03-06T07:27:07.210554+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:02.562532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211062+0000","last_change":"2024-03-06T07:27:05.294326+0000","last_active":"2024-03-06T07:27:07.211062+0000","last_peered":"2024-03-06T07:27:07.211062+0000","last_clean":"2024-03-06T07:27:07.211062+0000","last_became_active":"2024-03-06T07:27:05.294034+0000","last_became_peered":"2024-03-06T07:27:05.294034+0000","last_unstale":"2024-03-06T07:27:07.211062+0000","last_undegraded":"2024-03-06T07:27:07.211062+0000","last_fullsized":"2024-03-06T07:27:07.211062+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:31.591715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211056+0000","last_change":"2024-03-06T07:27:04.269073+0000","last_active":"2024-03-06T07:27:07.211056+0000","last_peered":"2024-03-06T07:27:07.211056+0000","last_clean":"2024-03-06T07:27:07.211056+0000","last_became_active":"2024-03-06T07:27:04.268831+0000","last_became_peered":"2024-03-06T07:27:04.268831+0000","last_unstale":"2024-03-06T07:27:07.211056+0000","last_undegraded":"2024-03-06T07:27:07.211056+0000","last_fullsized":"2024-03-06T07:27:07.211056+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:33.329615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211118+0000","last_change":"2024-03-06T07:27:05.280613+0000","last_active":"2024-03-06T07:27:07.211118+0000","last_peered":"2024-03-06T07:27:07.211118+0000","last_clean":"2024-03-06T07:27:07.211118+0000","last_became_active":"2024-03-06T07:27:05.279503+0000","last_became_peered":"2024-03-06T07:27:05.279503+0000","last_unstale":"2024-03-06T07:27:07.211118+0000","last_undegraded":"2024-03-06T07:27:07.211118+0000","last_fullsized":"2024-03-06T07:27:07.211118+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:02.279047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.209383+0000","last_change":"2024-03-06T07:27:04.271202+0000","last_active":"2024-03-06T07:27:07.209383+0000","last_peered":"2024-03-06T07:27:07.209383+0000","last_clean":"2024-03-06T07:27:07.209383+0000","last_became_active":"2024-03-06T07:27:04.270881+0000","last_became_peered":"2024-03-06T07:27:04.270881+0000","last_unstale":"2024-03-06T07:27:07.209383+0000","last_undegraded":"2024-03-06T07:27:07.209383+0000","last_fullsized":"2024-03-06T07:27:07.209383+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:07.973408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211471+0000","last_change":"2024-03-06T07:27:05.279835+0000","last_active":"2024-03-06T07:27:07.211471+0000","last_peered":"2024-03-06T07:27:07.211471+0000","last_clean":"2024-03-06T07:27:07.211471+0000","last_became_active":"2024-03-06T07:27:05.279575+0000","last_became_peered":"2024-03-06T07:27:05.279575+0000","last_unstale":"2024-03-06T07:27:07.211471+0000","last_undegraded":"2024-03-06T07:27:07.211471+0000","last_fullsized":"2024-03-06T07:27:07.211471+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:46.180033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.211800+0000","last_change":"2024-03-06T07:27:04.269025+0000","last_active":"2024-03-06T07:27:07.211800+0000","last_peered":"2024-03-06T07:27:07.211800+0000","last_clean":"2024-03-06T07:27:07.211800+0000","last_became_active":"2024-03-06T07:27:04.268768+0000","last_became_peered":"2024-03-06T07:27:04.268768+0000","last_unstale":"2024-03-06T07:27:07.211800+0000","last_undegraded":"2024-03-06T07:27:07.211800+0000","last_fullsized":"2024-03-06T07:27:07.211800+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:18.851804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.279686+0000","last_change":"2024-03-06T07:27:05.295457+0000","last_active":"2024-03-06T07:27:06.279686+0000","last_peered":"2024-03-06T07:27:06.279686+0000","last_clean":"2024-03-06T07:27:06.279686+0000","last_became_active":"2024-03-06T07:27:05.295174+0000","last_became_peered":"2024-03-06T07:27:05.295174+0000","last_unstale":"2024-03-06T07:27:06.279686+0000","last_undegraded":"2024-03-06T07:27:06.279686+0000","last_fullsized":"2024-03-06T07:27:06.279686+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:17.331331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.276795+0000","last_change":"2024-03-06T07:27:04.269800+0000","last_active":"2024-03-06T07:27:06.276795+0000","last_peered":"2024-03-06T07:27:06.276795+0000","last_clean":"2024-03-06T07:27:06.276795+0000","last_became_active":"2024-03-06T07:27:04.269489+0000","last_became_peered":"2024-03-06T07:27:04.269489+0000","last_unstale":"2024-03-06T07:27:06.276795+0000","last_undegraded":"2024-03-06T07:27:06.276795+0000","last_fullsized":"2024-03-06T07:27:06.276795+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:46.515942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210744+0000","last_change":"2024-03-06T07:27:05.283349+0000","last_active":"2024-03-06T07:27:07.210744+0000","last_peered":"2024-03-06T07:27:07.210744+0000","last_clean":"2024-03-06T07:27:07.210744+0000","last_became_active":"2024-03-06T07:27:05.283076+0000","last_became_peered":"2024-03-06T07:27:05.283076+0000","last_unstale":"2024-03-06T07:27:07.210744+0000","last_undegraded":"2024-03-06T07:27:07.210744+0000","last_fullsized":"2024-03-06T07:27:07.210744+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.246424+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.246424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:22.969341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"170'1","reported_seq":15,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.209258+0000","last_change":"2024-03-06T07:27:04.258208+0000","last_active":"2024-03-06T07:27:07.209258+0000","last_peered":"2024-03-06T07:27:07.209258+0000","last_clean":"2024-03-06T07:27:07.209258+0000","last_became_active":"2024-03-06T07:27:04.257866+0000","last_became_peered":"2024-03-06T07:27:04.257866+0000","last_unstale":"2024-03-06T07:27:07.209258+0000","last_undegraded":"2024-03-06T07:27:07.209258+0000","last_fullsized":"2024-03-06T07:27:07.209258+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.242167+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.242167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:15.565348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":460,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-06T07:27:07.210146+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:27:07.210146+0000","last_peered":"2024-03-06T07:27:07.210146+0000","last_clean":"2024-03-06T07:27:07.210146+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:27:07.210146+0000","last_undegraded":"2024-03-06T07:27:07.210146+0000","last_fullsized":"2024-03-06T07:27:07.210146+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33376,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338464,"statfs":{"total":96636764160,"available":96602587136,"internally_reserved":0,"allocated":2244608,"data_stored":1648506,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574933,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33312,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338528,"statfs":{"total":96636764160,"available":96602652672,"internally_reserved":0,"allocated":2179072,"data_stored":1058650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574933,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2170880,"data_stored":1056512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574934,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2166784,"data_stored":1057038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574933,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33308,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338532,"statfs":{"total":96636764160,"available":96602656768,"internally_reserved":0,"allocated":2174976,"data_stored":1056582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574933,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33296,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338544,"statfs":{"total":96636764160,"available":96602669056,"internally_reserved":0,"allocated":2166784,"data_stored":1056546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33296,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338544,"statfs":{"total":96636764160,"available":96602669056,"internally_reserved":0,"allocated":2162688,"data_stored":1056456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33364,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338476,"statfs":{"total":96636764160,"available":96602599424,"internally_reserved":0,"allocated":2232320,"data_stored":1646914,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:13.265 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:27:13.265 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:27:13.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.408+0000 7f23912af640 1 Processor -- start 2024-03-06T07:27:13.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.408+0000 7f23912af640 1 -- start start 2024-03-06T07:27:13.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.409+0000 7f23912af640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f238c152a30 0x7f238c0c7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.409+0000 7f23912af640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f238c0afcb0 0x7f238c06d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.409+0000 7f23912af640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f238c0c7940 0x7f238c06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.409+0000 7f23912af640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f238c155260 con 0x7f238c152a30 2024-03-06T07:27:13.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.409+0000 7f23912af640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f238c0c6fb0 con 0x7f238c0afcb0 2024-03-06T07:27:13.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.409+0000 7f23912af640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238c155520 con 0x7f238c0c7940 2024-03-06T07:27:13.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.409+0000 7f238affd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f238c152a30 0x7f238c0c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.409+0000 7f238affd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f238c152a30 0x7f238c0c7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59538/0 (socket says 172.21.15.1:59538) 2024-03-06T07:27:13.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.409+0000 7f238affd640 1 -- 172.21.15.1:0/1227321578 learned_addr learned my addr 172.21.15.1:0/1227321578 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:13.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.409+0000 7f238b7fe640 1 --2- 172.21.15.1:0/1227321578 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f238c0c7940 0x7f238c06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.409+0000 7f2383fff640 1 --2- 172.21.15.1:0/1227321578 >> v2:172.21.15.98:3300/0 conn(0x7f238c0afcb0 0x7f238c06d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.409+0000 7f238affd640 1 -- 172.21.15.1:0/1227321578 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f238c0c7940 msgr2=0x7f238c06d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.409+0000 7f238affd640 1 --2- 172.21.15.1:0/1227321578 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f238c0c7940 0x7f238c06d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.409+0000 7f238affd640 1 -- 172.21.15.1:0/1227321578 >> v2:172.21.15.98:3300/0 conn(0x7f238c0afcb0 msgr2=0x7f238c06d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.409+0000 7f238affd640 1 --2- 172.21.15.1:0/1227321578 >> v2:172.21.15.98:3300/0 conn(0x7f238c0afcb0 0x7f238c06d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.409+0000 7f238affd640 1 -- 172.21.15.1:0/1227321578 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c155800 con 0x7f238c152a30 2024-03-06T07:27:13.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.410+0000 7f238affd640 1 --2- 172.21.15.1:0/1227321578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f238c152a30 0x7f238c0c7400 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f238c0cba90 tx=0x7f237c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195f7d37b2ae35de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.410+0000 7f238a7fc640 1 -- 172.21.15.1:0/1227321578 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c02e070 con 0x7f238c152a30 2024-03-06T07:27:13.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.410+0000 7f238a7fc640 1 -- 172.21.15.1:0/1227321578 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237c037070 con 0x7f238c152a30 2024-03-06T07:27:13.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.410+0000 7f238a7fc640 1 -- 172.21.15.1:0/1227321578 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c031a00 con 0x7f238c152a30 2024-03-06T07:27:13.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.411+0000 7f23912af640 1 -- 172.21.15.1:0/1227321578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f238c152a30 msgr2=0x7f238c0c7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.411+0000 7f23912af640 1 --2- 172.21.15.1:0/1227321578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f238c152a30 0x7f238c0c7400 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f238c0cba90 tx=0x7f237c02b880 comp rx=0 tx=0).stop 2024-03-06T07:27:13.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.411+0000 7f23912af640 1 -- 172.21.15.1:0/1227321578 shutdown_connections 2024-03-06T07:27:13.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.411+0000 7f23912af640 1 --2- 172.21.15.1:0/1227321578 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f238c0c7940 0x7f238c06d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.411+0000 7f23912af640 1 --2- 172.21.15.1:0/1227321578 >> v2:172.21.15.98:3300/0 conn(0x7f238c0afcb0 0x7f238c06d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.411+0000 7f23912af640 1 --2- 172.21.15.1:0/1227321578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f238c152a30 0x7f238c0c7400 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.411+0000 7f23912af640 1 -- 172.21.15.1:0/1227321578 >> 172.21.15.1:0/1227321578 conn(0x7f238c08b8e0 msgr2=0x7f238c066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.411+0000 7f23912af640 1 -- 172.21.15.1:0/1227321578 shutdown_connections 2024-03-06T07:27:13.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.411+0000 7f23912af640 1 -- 172.21.15.1:0/1227321578 wait complete. 2024-03-06T07:27:13.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.412+0000 7f23912af640 1 Processor -- start 2024-03-06T07:27:13.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.412+0000 7f23912af640 1 -- start start 2024-03-06T07:27:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.412+0000 7f23912af640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f238c0afcb0 0x7f238c1576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.412+0000 7f23912af640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f238c0c7940 0x7f238c0c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.412+0000 7f23912af640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f238c152a30 0x7f238c1eb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.412+0000 7f23912af640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c0c6cc0 con 0x7f238c0c7940 2024-03-06T07:27:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.412+0000 7f23912af640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c0c6b40 con 0x7f238c0afcb0 2024-03-06T07:27:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.412+0000 7f23912af640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238c0af6d0 con 0x7f238c152a30 2024-03-06T07:27:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f2383fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f238c0c7940 0x7f238c0c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f2383fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f238c0c7940 0x7f238c0c9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59548/0 (socket says 172.21.15.1:59548) 2024-03-06T07:27:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f2383fff640 1 -- 172.21.15.1:0/3872196686 learned_addr learned my addr 172.21.15.1:0/3872196686 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f238b7fe640 1 --2- 172.21.15.1:0/3872196686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f238c152a30 0x7f238c1eb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f238affd640 1 --2- 172.21.15.1:0/3872196686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f238c0afcb0 0x7f238c1576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f2383fff640 1 -- 172.21.15.1:0/3872196686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f238c152a30 msgr2=0x7f238c1eb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f2383fff640 1 --2- 172.21.15.1:0/3872196686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f238c152a30 0x7f238c1eb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f2383fff640 1 -- 172.21.15.1:0/3872196686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f238c0afcb0 msgr2=0x7f238c1576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f2383fff640 1 --2- 172.21.15.1:0/3872196686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f238c0afcb0 0x7f238c1576d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f2383fff640 1 -- 172.21.15.1:0/3872196686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c1eaaf0 con 0x7f238c0c7940 2024-03-06T07:27:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f238affd640 1 --2- 172.21.15.1:0/3872196686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f238c0afcb0 0x7f238c1576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:13.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f238b7fe640 1 --2- 172.21.15.1:0/3872196686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f238c152a30 0x7f238c1eb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:13.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f2383fff640 1 --2- 172.21.15.1:0/3872196686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f238c0c7940 0x7f238c0c9be0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f237400d7d0 tx=0x7f237400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f23837fe640 1 -- 172.21.15.1:0/3872196686 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23740096a0 con 0x7f238c0c7940 2024-03-06T07:27:13.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f23837fe640 1 -- 172.21.15.1:0/3872196686 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2374019070 con 0x7f238c0c7940 2024-03-06T07:27:13.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f23837fe640 1 -- 172.21.15.1:0/3872196686 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2374009a60 con 0x7f238c0c7940 2024-03-06T07:27:13.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f23912af640 1 -- 172.21.15.1:0/3872196686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f238c156820 con 0x7f238c0c7940 2024-03-06T07:27:13.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.413+0000 7f23912af640 1 -- 172.21.15.1:0/3872196686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f238c1ead90 con 0x7f238c0c7940 2024-03-06T07:27:13.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.414+0000 7f23912af640 1 -- 172.21.15.1:0/3872196686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f238c0af4b0 con 0x7f238c0c7940 2024-03-06T07:27:13.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.415+0000 7f23837fe640 1 -- 172.21.15.1:0/3872196686 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2374009c00 con 0x7f238c0c7940 2024-03-06T07:27:13.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.415+0000 7f23837fe640 1 --2- 172.21.15.1:0/3872196686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f236c06b4d0 0x7f236c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.415+0000 7f23837fe640 1 -- 172.21.15.1:0/3872196686 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f237408a7b0 con 0x7f238c0c7940 2024-03-06T07:27:13.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.415+0000 7f238affd640 1 --2- 172.21.15.1:0/3872196686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f236c06b4d0 0x7f236c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.416+0000 7f238affd640 1 --2- 172.21.15.1:0/3872196686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f236c06b4d0 0x7f236c06d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f237c036000 tx=0x7f237c03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.420+0000 7f23837fe640 1 -- 172.21.15.1:0/3872196686 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f238c0af4b0 con 0x7f238c0c7940 2024-03-06T07:27:13.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.669+0000 7f23912af640 1 -- 172.21.15.1:0/3872196686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f238c067fd0 con 0x7f238c0c7940 2024-03-06T07:27:13.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.670+0000 7f23837fe640 1 -- 172.21.15.1:0/3872196686 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+147 (secure 0 0 0) 0x7f23740594a0 con 0x7f238c0c7940 2024-03-06T07:27:13.673 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-06T07:27:13.673 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng== 2024-03-06T07:27:13.673 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-06T07:27:13.673 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-06T07:27:13.673 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:13.673 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-06T07:27:13.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.673+0000 7f23912af640 1 -- 172.21.15.1:0/3872196686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f236c06b4d0 msgr2=0x7f236c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.673+0000 7f23912af640 1 --2- 172.21.15.1:0/3872196686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f236c06b4d0 0x7f236c06d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f237c036000 tx=0x7f237c03d040 comp rx=0 tx=0).stop 2024-03-06T07:27:13.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.673+0000 7f23912af640 1 -- 172.21.15.1:0/3872196686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f238c0c7940 msgr2=0x7f238c0c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.673+0000 7f23912af640 1 --2- 172.21.15.1:0/3872196686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f238c0c7940 0x7f238c0c9be0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f237400d7d0 tx=0x7f237400dc90 comp rx=0 tx=0).stop 2024-03-06T07:27:13.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.673+0000 7f23912af640 1 -- 172.21.15.1:0/3872196686 shutdown_connections 2024-03-06T07:27:13.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.673+0000 7f23912af640 1 --2- 172.21.15.1:0/3872196686 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f236c06b4d0 0x7f236c06d990 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.673+0000 7f23912af640 1 --2- 172.21.15.1:0/3872196686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f238c152a30 0x7f238c1eb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.673+0000 7f23912af640 1 --2- 172.21.15.1:0/3872196686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f238c0c7940 0x7f238c0c9be0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.673+0000 7f23912af640 1 --2- 172.21.15.1:0/3872196686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f238c0afcb0 0x7f238c1576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.673+0000 7f23912af640 1 -- 172.21.15.1:0/3872196686 >> 172.21.15.1:0/3872196686 conn(0x7f238c08b8e0 msgr2=0x7f238c066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.673+0000 7f23912af640 1 -- 172.21.15.1:0/3872196686 shutdown_connections 2024-03-06T07:27:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.673+0000 7f23912af640 1 -- 172.21.15.1:0/3872196686 wait complete. 2024-03-06T07:27:13.686 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:13.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.791+0000 7ff12d5e9640 1 Processor -- start 2024-03-06T07:27:13.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.791+0000 7ff12d5e9640 1 -- start start 2024-03-06T07:27:13.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.792+0000 7ff12d5e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff12814cdc0 0x7ff128152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.792+0000 7ff12d5e9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff128156500 0x7ff1280c2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.792+0000 7ff12d5e9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1281569e0 con 0x7ff1280c2890 2024-03-06T07:27:13.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.792+0000 7ff12d5e9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1280c2dd0 con 0x7ff128156500 2024-03-06T07:27:13.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.792+0000 7ff12d5e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff128156cc0 con 0x7ff12814cdc0 2024-03-06T07:27:13.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.792+0000 7ff126ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff12814cdc0 0x7ff128152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.792+0000 7ff1267fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff128156500 0x7ff1280c2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.792+0000 7ff126ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff12814cdc0 0x7ff128152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:56234/0 (socket says 172.21.15.1:56234) 2024-03-06T07:27:13.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.792+0000 7ff126ffd640 1 -- 172.21.15.1:0/275913692 learned_addr learned my addr 172.21.15.1:0/275913692 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:13.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.792+0000 7ff1267fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff128156500 0x7ff1280c2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:54360/0 (socket says 172.21.15.1:54360) 2024-03-06T07:27:13.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.792+0000 7ff1267fc640 1 -- 172.21.15.1:0/275913692 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff12814cdc0 msgr2=0x7ff128152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.792+0000 7ff1267fc640 1 --2- 172.21.15.1:0/275913692 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff12814cdc0 0x7ff128152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.792+0000 7ff1267fc640 1 -- 172.21.15.1:0/275913692 >> v1:172.21.15.98:6789/0 conn(0x7ff1280c2890 legacy=0x7ff12806b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.792+0000 7ff1267fc640 1 -- 172.21.15.1:0/275913692 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff128156fa0 con 0x7ff128156500 2024-03-06T07:27:13.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.792+0000 7ff126ffd640 1 --2- 172.21.15.1:0/275913692 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff12814cdc0 0x7ff128152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:13.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.793+0000 7ff1267fc640 1 --2- 172.21.15.1:0/275913692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff128156500 0x7ff1280c2350 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff1180097a0 tx=0x7ff11802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=748b3ef0f1774474 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.793+0000 7ff125ffb640 1 -- 172.21.15.1:0/275913692 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff118035020 con 0x7ff128156500 2024-03-06T07:27:13.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.794+0000 7ff125ffb640 1 -- 172.21.15.1:0/275913692 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff118031830 con 0x7ff128156500 2024-03-06T07:27:13.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.794+0000 7ff125ffb640 1 -- 172.21.15.1:0/275913692 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff118031c00 con 0x7ff128156500 2024-03-06T07:27:13.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.794+0000 7ff12d5e9640 1 -- 172.21.15.1:0/275913692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff128156500 msgr2=0x7ff1280c2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.794+0000 7ff12d5e9640 1 --2- 172.21.15.1:0/275913692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff128156500 0x7ff1280c2350 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff1180097a0 tx=0x7ff11802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.794+0000 7ff12d5e9640 1 -- 172.21.15.1:0/275913692 shutdown_connections 2024-03-06T07:27:13.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.794+0000 7ff12d5e9640 1 --2- 172.21.15.1:0/275913692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff128156500 0x7ff1280c2350 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.794+0000 7ff12d5e9640 1 --2- 172.21.15.1:0/275913692 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff12814cdc0 0x7ff128152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.794+0000 7ff12d5e9640 1 -- 172.21.15.1:0/275913692 >> 172.21.15.1:0/275913692 conn(0x7ff1280a9700 msgr2=0x7ff1280a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.794+0000 7ff12d5e9640 1 -- 172.21.15.1:0/275913692 shutdown_connections 2024-03-06T07:27:13.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.794+0000 7ff12d5e9640 1 -- 172.21.15.1:0/275913692 wait complete. 2024-03-06T07:27:13.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.795+0000 7ff12d5e9640 1 Processor -- start 2024-03-06T07:27:13.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.795+0000 7ff12d5e9640 1 -- start start 2024-03-06T07:27:13.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.796+0000 7ff12d5e9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1280c2890 0x7ff1281dfc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.796+0000 7ff12d5e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff12814cdc0 0x7ff12814f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.796+0000 7ff12d5e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff128156500 0x7ff1281ec3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.796+0000 7ff12d5e9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1280c65e0 con 0x7ff1280c2890 2024-03-06T07:27:13.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.796+0000 7ff12d5e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1280c5a70 con 0x7ff128156500 2024-03-06T07:27:13.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.796+0000 7ff12d5e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff12814d1a0 con 0x7ff12814cdc0 2024-03-06T07:27:13.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.796+0000 7ff126ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1280c2890 0x7ff1281dfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.796+0000 7ff126ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1280c2890 0x7ff1281dfc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59560/0 (socket says 172.21.15.1:59560) 2024-03-06T07:27:13.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.796+0000 7ff126ffd640 1 -- 172.21.15.1:0/1305775777 learned_addr learned my addr 172.21.15.1:0/1305775777 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:13.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.796+0000 7ff1277fe640 1 --2- 172.21.15.1:0/1305775777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff128156500 0x7ff1281ec3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.796+0000 7ff1267fc640 1 --2- 172.21.15.1:0/1305775777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff12814cdc0 0x7ff12814f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.796+0000 7ff126ffd640 1 -- 172.21.15.1:0/1305775777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff12814cdc0 msgr2=0x7ff12814f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.796+0000 7ff126ffd640 1 --2- 172.21.15.1:0/1305775777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff12814cdc0 0x7ff12814f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.796+0000 7ff126ffd640 1 -- 172.21.15.1:0/1305775777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff128156500 msgr2=0x7ff1281ec3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.796+0000 7ff126ffd640 1 --2- 172.21.15.1:0/1305775777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff128156500 0x7ff1281ec3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.796+0000 7ff126ffd640 1 -- 172.21.15.1:0/1305775777 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12814c230 con 0x7ff1280c2890 2024-03-06T07:27:13.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.797+0000 7ff126ffd640 1 --2- 172.21.15.1:0/1305775777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1280c2890 0x7ff1281dfc60 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7ff11000e9d0 tx=0x7ff11000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.797+0000 7ff11f7fe640 1 -- 172.21.15.1:0/1305775777 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff110017070 con 0x7ff1280c2890 2024-03-06T07:27:13.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.797+0000 7ff12d5e9640 1 -- 172.21.15.1:0/1305775777 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1280ada70 con 0x7ff1280c2890 2024-03-06T07:27:13.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.797+0000 7ff11f7fe640 1 -- 172.21.15.1:0/1305775777 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff11000ca60 con 0x7ff1280c2890 2024-03-06T07:27:13.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.797+0000 7ff11f7fe640 1 -- 172.21.15.1:0/1305775777 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11000ce10 con 0x7ff1280c2890 2024-03-06T07:27:13.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.797+0000 7ff12d5e9640 1 -- 172.21.15.1:0/1305775777 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1281e01a0 con 0x7ff1280c2890 2024-03-06T07:27:13.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.798+0000 7ff12d5e9640 1 -- 172.21.15.1:0/1305775777 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff128067fd0 con 0x7ff1280c2890 2024-03-06T07:27:13.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.799+0000 7ff11f7fe640 1 -- 172.21.15.1:0/1305775777 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff11001a070 con 0x7ff1280c2890 2024-03-06T07:27:13.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.799+0000 7ff11f7fe640 1 --2- 172.21.15.1:0/1305775777 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff10806b4d0 0x7ff10806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.799+0000 7ff11f7fe640 1 -- 172.21.15.1:0/1305775777 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff11008a2c0 con 0x7ff1280c2890 2024-03-06T07:27:13.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.799+0000 7ff1267fc640 1 --2- 172.21.15.1:0/1305775777 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff10806b4d0 0x7ff10806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.800+0000 7ff1267fc640 1 --2- 172.21.15.1:0/1305775777 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff10806b4d0 0x7ff10806d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff118036000 tx=0x7ff11803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:13.803+0000 7ff11f7fe640 1 -- 172.21.15.1:0/1305775777 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff110058fc0 con 0x7ff1280c2890 2024-03-06T07:27:14.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.022+0000 7ff12d5e9640 1 -- 172.21.15.1:0/1305775777 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1280c5850 con 0x7ff1280c2890 2024-03-06T07:27:14.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.023+0000 7ff11f7fe640 1 -- 172.21.15.1:0/1305775777 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+4969 (secure 0 0 0) 0x7ff1100588c0 con 0x7ff1280c2890 2024-03-06T07:27:14.025 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:14.025 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":157,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:06.627754+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9489":{"gid":9489,"name":"c","rank":0,"incarnation":155,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1751109564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1751109564},{"type":"v1","addr":"172.21.15.1:6835","nonce":1751109564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9489,"qdb_cluster":[9489]},"id":13}]} 2024-03-06T07:27:14.025 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 157 2024-03-06T07:27:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.025+0000 7ff12d5e9640 1 -- 172.21.15.1:0/1305775777 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff10806b4d0 msgr2=0x7ff10806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.025+0000 7ff12d5e9640 1 --2- 172.21.15.1:0/1305775777 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff10806b4d0 0x7ff10806d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff118036000 tx=0x7ff11803c040 comp rx=0 tx=0).stop 2024-03-06T07:27:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.025+0000 7ff12d5e9640 1 -- 172.21.15.1:0/1305775777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1280c2890 msgr2=0x7ff1281dfc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.025+0000 7ff12d5e9640 1 --2- 172.21.15.1:0/1305775777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1280c2890 0x7ff1281dfc60 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7ff11000e9d0 tx=0x7ff11000ee90 comp rx=0 tx=0).stop 2024-03-06T07:27:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.025+0000 7ff12d5e9640 1 -- 172.21.15.1:0/1305775777 shutdown_connections 2024-03-06T07:27:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.025+0000 7ff12d5e9640 1 --2- 172.21.15.1:0/1305775777 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff10806b4d0 0x7ff10806d990 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.025+0000 7ff12d5e9640 1 --2- 172.21.15.1:0/1305775777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff128156500 0x7ff1281ec3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.025+0000 7ff12d5e9640 1 --2- 172.21.15.1:0/1305775777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff12814cdc0 0x7ff12814f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.025+0000 7ff12d5e9640 1 --2- 172.21.15.1:0/1305775777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1280c2890 0x7ff1281dfc60 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.025+0000 7ff12d5e9640 1 -- 172.21.15.1:0/1305775777 >> 172.21.15.1:0/1305775777 conn(0x7ff1280a9700 msgr2=0x7ff128152410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.025+0000 7ff12d5e9640 1 -- 172.21.15.1:0/1305775777 shutdown_connections 2024-03-06T07:27:14.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.025+0000 7ff12d5e9640 1 -- 172.21.15.1:0/1305775777 wait complete. 2024-03-06T07:27:14.440 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 157, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:04.619091+0000', 'modified': '2024-03-06T07:27:06.627754+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9489': {'gid': 9489, 'name': 'c', 'rank': 0, 'incarnation': 155, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1751109564', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1751109564}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1751109564}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9489, 'qdb_cluster': [9489]} 2024-03-06T07:27:14.441 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:14.441 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:27:14.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.557+0000 7fb8bc69d640 1 Processor -- start 2024-03-06T07:27:14.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.558+0000 7fb8bc69d640 1 -- start start 2024-03-06T07:27:14.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.559+0000 7fb8bc69d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b40c1820 0x7fb8b41559f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.559+0000 7fb8bc69d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b41588e0 con 0x7fb8b4152730 2024-03-06T07:27:14.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.559+0000 7fb8bc69d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b4158bc0 con 0x7fb8b40c1390 2024-03-06T07:27:14.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.559+0000 7fb8bc69d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b4158ea0 con 0x7fb8b40c1820 2024-03-06T07:27:14.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.559+0000 7fb8ba439640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb8b4152730 0x7fb8b40c6a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42930/0 (socket says 172.21.15.1:42930) 2024-03-06T07:27:14.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.559+0000 7fb8ba439640 1 -- 172.21.15.1:0/3360657935 learned_addr learned my addr 172.21.15.1:0/3360657935 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:14.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.559+0000 7fb8bac3a640 1 --2- 172.21.15.1:0/3360657935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b40c1820 0x7fb8b41559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.559+0000 7fb8b9437640 1 -- 172.21.15.1:0/3360657935 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514796804 0 0) 0x7fb8b41588e0 con 0x7fb8b4152730 2024-03-06T07:27:14.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.559+0000 7fb8b9437640 1 -- 172.21.15.1:0/3360657935 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a0003520 con 0x7fb8b4152730 2024-03-06T07:27:14.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8b9437640 1 -- 172.21.15.1:0/3360657935 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1297597686 0 0) 0x7fb8a0003520 con 0x7fb8b4152730 2024-03-06T07:27:14.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8b9437640 1 -- 172.21.15.1:0/3360657935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b40c1820 msgr2=0x7fb8b41559f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8b9437640 1 --2- 172.21.15.1:0/3360657935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b40c1820 0x7fb8b41559f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8b9437640 1 -- 172.21.15.1:0/3360657935 >> v1:172.21.15.98:6789/0 conn(0x7fb8b40c1390 legacy=0x7fb8b41552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8b9437640 1 -- 172.21.15.1:0/3360657935 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b4159180 con 0x7fb8b4152730 2024-03-06T07:27:14.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8b9437640 1 -- 172.21.15.1:0/3360657935 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fb8a4001690 con 0x7fb8b4152730 2024-03-06T07:27:14.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8b9437640 1 -- 172.21.15.1:0/3360657935 >> v1:172.21.15.1:6789/0 conn(0x7fb8b4152730 legacy=0x7fb8b40c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8b9437640 1 --2- 172.21.15.1:0/3360657935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b40c1820 0x7fb8a00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8b9437640 1 --2- 172.21.15.1:0/3360657935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8a0003f60 0x7fb8a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8b9437640 1 --2- 172.21.15.1:0/3360657935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a0004d40 0x7fb8a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8b9437640 1 -- 172.21.15.1:0/3360657935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b4158ea0 con 0x7fb8a0004d40 2024-03-06T07:27:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8b9437640 1 -- 172.21.15.1:0/3360657935 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a000bc00 con 0x7fb8a0003f60 2024-03-06T07:27:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8b9437640 1 -- 172.21.15.1:0/3360657935 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a000be00 con 0x7fb8b40c1820 2024-03-06T07:27:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8bac3a640 1 --2- 172.21.15.1:0/3360657935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a0004d40 0x7fb8a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8ba439640 1 --2- 172.21.15.1:0/3360657935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b40c1820 0x7fb8a00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8b9c38640 1 --2- 172.21.15.1:0/3360657935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8a0003f60 0x7fb8a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8bac3a640 1 -- 172.21.15.1:0/3360657935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b40c1820 msgr2=0x7fb8a00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8bac3a640 1 --2- 172.21.15.1:0/3360657935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b40c1820 0x7fb8a00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8bac3a640 1 -- 172.21.15.1:0/3360657935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8a0003f60 msgr2=0x7fb8a0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8bac3a640 1 --2- 172.21.15.1:0/3360657935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8a0003f60 0x7fb8a0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.560+0000 7fb8bac3a640 1 -- 172.21.15.1:0/3360657935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a0003520 con 0x7fb8a0004d40 2024-03-06T07:27:14.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.561+0000 7fb8bac3a640 1 --2- 172.21.15.1:0/3360657935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a0004d40 0x7fb8a00051c0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009fc0 tx=0x7fb8a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209af610b8b98c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.561+0000 7fb8b9437640 1 -- 172.21.15.1:0/3360657935 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac039070 con 0x7fb8a0004d40 2024-03-06T07:27:14.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.561+0000 7fb8b9437640 1 -- 172.21.15.1:0/3360657935 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ac02cc20 con 0x7fb8a0004d40 2024-03-06T07:27:14.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.561+0000 7fb8b9437640 1 -- 172.21.15.1:0/3360657935 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac009c10 con 0x7fb8a0004d40 2024-03-06T07:27:14.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.562+0000 7fb8bc69d640 1 -- 172.21.15.1:0/3360657935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a0004d40 msgr2=0x7fb8a00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.562+0000 7fb8bc69d640 1 --2- 172.21.15.1:0/3360657935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a0004d40 0x7fb8a00051c0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009fc0 tx=0x7fb8a0005830 comp rx=0 tx=0).stop 2024-03-06T07:27:14.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.562+0000 7fb8bac3a640 1 -- 172.21.15.1:0/3360657935 reap_dead start 2024-03-06T07:27:14.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.562+0000 7fb8bc69d640 1 -- 172.21.15.1:0/3360657935 shutdown_connections 2024-03-06T07:27:14.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.562+0000 7fb8bc69d640 1 -- 172.21.15.1:0/3360657935 >> 172.21.15.1:0/3360657935 conn(0x7fb8b4066d00 msgr2=0x7fb8b4067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.562+0000 7fb8bc69d640 1 -- 172.21.15.1:0/3360657935 shutdown_connections 2024-03-06T07:27:14.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.562+0000 7fb8bc69d640 1 -- 172.21.15.1:0/3360657935 wait complete. 2024-03-06T07:27:14.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.563+0000 7fb8bc69d640 1 Processor -- start 2024-03-06T07:27:14.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.563+0000 7fb8bc69d640 1 -- start start 2024-03-06T07:27:14.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.563+0000 7fb8bc69d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8a0004d40 0x7fb8b4155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.563+0000 7fb8bc69d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8b40aa920 0x7fb8b4156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.563+0000 7fb8bc69d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b4156f60 0x7fb8b40c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.563+0000 7fb8bc69d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b4152b10 con 0x7fb8b40aa920 2024-03-06T07:27:14.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.563+0000 7fb8bc69d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b40c1c90 con 0x7fb8a0004d40 2024-03-06T07:27:14.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.563+0000 7fb8bc69d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b40c71a0 con 0x7fb8b4156f60 2024-03-06T07:27:14.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.564+0000 7fb8b9c38640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8b40aa920 0x7fb8b4156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.564+0000 7fb8b9c38640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8b40aa920 0x7fb8b4156a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48220/0 (socket says 172.21.15.1:48220) 2024-03-06T07:27:14.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.564+0000 7fb8b9c38640 1 -- 172.21.15.1:0/921034704 learned_addr learned my addr 172.21.15.1:0/921034704 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:14.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.564+0000 7fb8ba439640 1 --2- 172.21.15.1:0/921034704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8a0004d40 0x7fb8b4155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.564+0000 7fb8bac3a640 1 --2- 172.21.15.1:0/921034704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b4156f60 0x7fb8b40c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.564+0000 7fb8b9c38640 1 -- 172.21.15.1:0/921034704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b4156f60 msgr2=0x7fb8b40c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.564+0000 7fb8b9c38640 1 --2- 172.21.15.1:0/921034704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b4156f60 0x7fb8b40c8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.564+0000 7fb8b9c38640 1 -- 172.21.15.1:0/921034704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8a0004d40 msgr2=0x7fb8b4155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.564+0000 7fb8b9c38640 1 --2- 172.21.15.1:0/921034704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8a0004d40 0x7fb8b4155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.564+0000 7fb8b9c38640 1 -- 172.21.15.1:0/921034704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b4157f00 con 0x7fb8b40aa920 2024-03-06T07:27:14.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.564+0000 7fb8b9c38640 1 --2- 172.21.15.1:0/921034704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8b40aa920 0x7fb8b4156a20 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000f820 tx=0x7fb8b000fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.564+0000 7fb8aaffd640 1 -- 172.21.15.1:0/921034704 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b00050d0 con 0x7fb8b40aa920 2024-03-06T07:27:14.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.564+0000 7fb8aaffd640 1 -- 172.21.15.1:0/921034704 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b0018070 con 0x7fb8b40aa920 2024-03-06T07:27:14.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.564+0000 7fb8bc69d640 1 -- 172.21.15.1:0/921034704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b41581e0 con 0x7fb8b40aa920 2024-03-06T07:27:14.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.564+0000 7fb8aaffd640 1 -- 172.21.15.1:0/921034704 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0005490 con 0x7fb8b40aa920 2024-03-06T07:27:14.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.564+0000 7fb8bc69d640 1 -- 172.21.15.1:0/921034704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b40c9970 con 0x7fb8b40aa920 2024-03-06T07:27:14.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.566+0000 7fb8aaffd640 1 -- 172.21.15.1:0/921034704 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb8b000d730 con 0x7fb8b40aa920 2024-03-06T07:27:14.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.566+0000 7fb8aaffd640 1 --2- 172.21.15.1:0/921034704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb89806b500 0x7fb89806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.566+0000 7fb8aaffd640 1 -- 172.21.15.1:0/921034704 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb8b0089fd0 con 0x7fb8b40aa920 2024-03-06T07:27:14.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.566+0000 7fb8ba439640 1 --2- 172.21.15.1:0/921034704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb89806b500 0x7fb89806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.567+0000 7fb8ba439640 1 --2- 172.21.15.1:0/921034704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb89806b500 0x7fb89806d9c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4001800 tx=0x7fb8a4001630 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.568+0000 7fb8bf363740 1 Processor -- start 2024-03-06T07:27:14.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.568+0000 7fb8bf363740 1 -- start start 2024-03-06T07:27:14.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.569+0000 7fb8bf363740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacaf50 0x55a52eac8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.569+0000 7fb8bf363740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eacb330 0x55a52eac8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.569+0000 7fb8bf363740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eac94b0 0x55a52eac5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.569+0000 7fb8bf363740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a52ea2f6b0 con 0x55a52eacaf50 2024-03-06T07:27:14.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.569+0000 7fb8bf363740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a52e9d8a60 con 0x55a52eacb330 2024-03-06T07:27:14.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.569+0000 7fb8bf363740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a52ea2b750 con 0x55a52eac94b0 2024-03-06T07:27:14.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.569+0000 7fb8ba439640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacaf50 0x55a52eac8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.569+0000 7fb8ba439640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacaf50 0x55a52eac8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48230/0 (socket says 172.21.15.1:48230) 2024-03-06T07:27:14.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.569+0000 7fb8ba439640 1 -- 172.21.15.1:0/3570937934 learned_addr learned my addr 172.21.15.1:0/3570937934 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:14.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.570+0000 7fb8b9c38640 1 --2- 172.21.15.1:0/3570937934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eacb330 0x55a52eac8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.570+0000 7fb8bac3a640 1 --2- 172.21.15.1:0/3570937934 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eac94b0 0x55a52eac5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.570+0000 7fb8ba439640 1 -- 172.21.15.1:0/3570937934 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eac94b0 msgr2=0x55a52eac5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.570+0000 7fb8ba439640 1 --2- 172.21.15.1:0/3570937934 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eac94b0 0x55a52eac5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.570+0000 7fb8ba439640 1 -- 172.21.15.1:0/3570937934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eacb330 msgr2=0x55a52eac8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.570+0000 7fb8ba439640 1 --2- 172.21.15.1:0/3570937934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eacb330 0x55a52eac8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.570+0000 7fb8ba439640 1 -- 172.21.15.1:0/3570937934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a52e9f6f30 con 0x55a52eacaf50 2024-03-06T07:27:14.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.570+0000 7fb8b9c38640 1 --2- 172.21.15.1:0/3570937934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eacb330 0x55a52eac8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:14.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.570+0000 7fb8bac3a640 1 --2- 172.21.15.1:0/3570937934 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eac94b0 0x55a52eac5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:14.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.570+0000 7fb8ba439640 1 --2- 172.21.15.1:0/3570937934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacaf50 0x55a52eac8a30 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4007610 tx=0x7fb8a400c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a194a897c894be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.570+0000 7fb897fff640 1 -- 172.21.15.1:0/3570937934 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4028070 con 0x55a52eacaf50 2024-03-06T07:27:14.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.570+0000 7fb897fff640 1 -- 172.21.15.1:0/3570937934 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a4015bd0 con 0x55a52eacaf50 2024-03-06T07:27:14.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.570+0000 7fb897fff640 1 -- 172.21.15.1:0/3570937934 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a40213c0 con 0x55a52eacaf50 2024-03-06T07:27:14.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.571+0000 7fb8bf363740 1 -- 172.21.15.1:0/3570937934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacaf50 msgr2=0x55a52eac8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.571+0000 7fb8bf363740 1 --2- 172.21.15.1:0/3570937934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacaf50 0x55a52eac8a30 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4007610 tx=0x7fb8a400c000 comp rx=0 tx=0).stop 2024-03-06T07:27:14.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.571+0000 7fb8bf363740 1 -- 172.21.15.1:0/3570937934 shutdown_connections 2024-03-06T07:27:14.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.571+0000 7fb8bf363740 1 --2- 172.21.15.1:0/3570937934 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eac94b0 0x55a52eac5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.571+0000 7fb8bf363740 1 --2- 172.21.15.1:0/3570937934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eacb330 0x55a52eac8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.571+0000 7fb8bf363740 1 --2- 172.21.15.1:0/3570937934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacaf50 0x55a52eac8a30 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.571+0000 7fb8bf363740 1 -- 172.21.15.1:0/3570937934 >> 172.21.15.1:0/3570937934 conn(0x55a52eac77d0 msgr2=0x55a52eaec3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.571+0000 7fb8bf363740 1 -- 172.21.15.1:0/3570937934 shutdown_connections 2024-03-06T07:27:14.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.571+0000 7fb8bf363740 1 -- 172.21.15.1:0/3570937934 wait complete. 2024-03-06T07:27:14.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.571+0000 7fb8bf363740 1 -- start start 2024-03-06T07:27:14.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.572+0000 7fb8bf363740 1 Processor -- start 2024-03-06T07:27:14.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.572+0000 7fb8bf363740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eacaf50 0x55a52eaefff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.572+0000 7fb8bf363740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eacb330 0x55a52eaebc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.572+0000 7fb8bf363740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eaf1440 0x55a52eaf0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.572+0000 7fb8bf363740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a52eaed230 con 0x55a52eaf1440 2024-03-06T07:27:14.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.572+0000 7fb8bf363740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a52e9ebea0 con 0x55a52eacaf50 2024-03-06T07:27:14.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.572+0000 7fb8bf363740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a52eaecc70 con 0x55a52eacb330 2024-03-06T07:27:14.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.572+0000 7fb8bf363740 10 client.0 _ll_register_callbacks cb 0x55a52e9eb870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:14.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.572+0000 7fb8bac3a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eaf1440 0x55a52eaf0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.573+0000 7fb8bac3a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eaf1440 0x55a52eaf0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48244/0 (socket says 172.21.15.1:48244) 2024-03-06T07:27:14.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.573+0000 7fb8bac3a640 1 -- 172.21.15.1:0/1007593652 learned_addr learned my addr 172.21.15.1:0/1007593652 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:14.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.573+0000 7fb8b9c38640 1 --2- 172.21.15.1:0/1007593652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eacb330 0x55a52eaebc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.573+0000 7fb8ba439640 1 --2- 172.21.15.1:0/1007593652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eacaf50 0x55a52eaefff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.573+0000 7fb8bac3a640 1 -- 172.21.15.1:0/1007593652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eacb330 msgr2=0x55a52eaebc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.573+0000 7fb8bac3a640 1 --2- 172.21.15.1:0/1007593652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eacb330 0x55a52eaebc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.573+0000 7fb8bac3a640 1 -- 172.21.15.1:0/1007593652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eacaf50 msgr2=0x55a52eaefff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.573+0000 7fb8bac3a640 1 --2- 172.21.15.1:0/1007593652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eacaf50 0x55a52eaefff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.573+0000 7fb8bac3a640 1 -- 172.21.15.1:0/1007593652 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a52e9ec270 con 0x55a52eaf1440 2024-03-06T07:27:14.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.573+0000 7fb8bac3a640 1 --2- 172.21.15.1:0/1007593652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eaf1440 0x55a52eaf0a30 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00e230 tx=0x7fb8ac004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baaadfd365f0fc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.573+0000 7fb8957fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:27:14.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.573+0000 7fb8957fa640 1 -- 172.21.15.1:0/1007593652 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac044070 con 0x55a52eaf1440 2024-03-06T07:27:14.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.573+0000 7fb8957fa640 1 -- 172.21.15.1:0/1007593652 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb8ac003b90 con 0x55a52eaf1440 2024-03-06T07:27:14.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.573+0000 7fb8bf363740 1 -- 172.21.15.1:0/1007593652 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb8a4015bd0 con 0x55a52eaf1440 2024-03-06T07:27:14.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.573+0000 7fb8957fa640 1 -- 172.21.15.1:0/1007593652 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=157) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8ac04e050 con 0x55a52eaf1440 2024-03-06T07:27:14.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.573+0000 7fb8bf363740 10 client.9663 fetch_fsmap learned FSMap version 157 2024-03-06T07:27:14.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.573+0000 7fb8bf363740 1 -- 172.21.15.1:0/1007593652 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=157,monmap=2+}) v3 -- 0x55a52e7442d0 con 0x55a52eaf1440 2024-03-06T07:27:14.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.574+0000 7fb8957fa640 1 -- 172.21.15.1:0/1007593652 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a4015bd0 con 0x55a52eaf1440 2024-03-06T07:27:14.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.574+0000 7fb8957fa640 1 -- 172.21.15.1:0/1007593652 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 157) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb8ac051020 con 0x55a52eaf1440 2024-03-06T07:27:14.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.574+0000 7fb8bf363740 10 client.9663 fetch_fsmap finished waiting for FSMap version 157 2024-03-06T07:27:14.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.574+0000 7fb8bf363740 10 client.9663 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:14.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.574+0000 7fb8bf363740 20 client.9663 populate_metadata read hostname 'smithi001' 2024-03-06T07:27:14.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.574+0000 7fb8bf363740 1 --2- 172.21.15.1:0/1007593652 >> [v2:172.21.15.1:6833/1751109564,v1:172.21.15.1:6835/1751109564] conn(0x55a52eae8130 0x55a52eae8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.574+0000 7fb8bf363740 4 client.9663 mds_command: new command op to 9489 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:14.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.574+0000 7fb8bf363740 1 -- 172.21.15.1:0/1007593652 --> [v2:172.21.15.1:6833/1751109564,v1:172.21.15.1:6835/1751109564] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a52ea3cbf0 con 0x55a52eae8130 2024-03-06T07:27:14.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.574+0000 7fb8ba439640 1 --2- 172.21.15.1:0/1007593652 >> [v2:172.21.15.1:6833/1751109564,v1:172.21.15.1:6835/1751109564] conn(0x55a52eae8130 0x55a52eae8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.574+0000 7fb8ba439640 1 --2- 172.21.15.1:0/1007593652 >> [v2:172.21.15.1:6833/1751109564,v1:172.21.15.1:6835/1751109564] conn(0x55a52eae8130 0x55a52eae8510 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dce8f19a0efb5940 server_cookie=2d1420c4d496f97c in_seq=0 out_seq=0 2024-03-06T07:27:14.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.574+0000 7fb8957fa640 10 client.9663 ms_handle_connect on v2:172.21.15.1:6833/1751109564 2024-03-06T07:27:14.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.575+0000 7fb8957fa640 1 -- 172.21.15.1:0/1007593652 <== mds.0 v2:172.21.15.1:6833/1751109564 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a52ea3cbf0 con 0x55a52eae8130 2024-03-06T07:27:14.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.575+0000 7fb8957fa640 10 client.9663 handle_command_reply: tid=0 2024-03-06T07:27:14.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.575+0000 7fb8bf363740 1 client.9663 shutdown 2024-03-06T07:27:14.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.575+0000 7fb8bf363740 1 -- 172.21.15.1:0/1007593652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eaf1440 msgr2=0x55a52eaf0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.575+0000 7fb8bf363740 1 --2- 172.21.15.1:0/1007593652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eaf1440 0x55a52eaf0a30 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00e230 tx=0x7fb8ac004ab0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.575+0000 7fb8bf363740 1 -- 172.21.15.1:0/1007593652 shutdown_connections 2024-03-06T07:27:14.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.575+0000 7fb8bf363740 1 --2- 172.21.15.1:0/1007593652 >> [v2:172.21.15.1:6833/1751109564,v1:172.21.15.1:6835/1751109564] conn(0x55a52eae8130 0x55a52eae8510 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.575+0000 7fb8bf363740 1 --2- 172.21.15.1:0/1007593652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eaf1440 0x55a52eaf0a30 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.575+0000 7fb8bf363740 1 --2- 172.21.15.1:0/1007593652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eacb330 0x55a52eaebc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.575+0000 7fb8957fa640 0 client.9663 ms_handle_reset on v2:172.21.15.1:6833/1751109564 2024-03-06T07:27:14.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.575+0000 7fb8bf363740 1 --2- 172.21.15.1:0/1007593652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eacaf50 0x55a52eaefff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.575+0000 7fb8bf363740 1 -- 172.21.15.1:0/1007593652 >> 172.21.15.1:0/1007593652 conn(0x55a52eac94b0 msgr2=0x55a52eb07b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.575+0000 7fb8bf363740 1 -- 172.21.15.1:0/1007593652 shutdown_connections 2024-03-06T07:27:14.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.575+0000 7fb8bf363740 1 -- 172.21.15.1:0/1007593652 wait complete. 2024-03-06T07:27:14.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.575+0000 7fb8bf363740 20 client.9663 trim_cache size 0 max 16384 2024-03-06T07:27:14.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.592+0000 7fb8bf363740 1 Processor -- start 2024-03-06T07:27:14.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.592+0000 7fb8bf363740 1 -- start start 2024-03-06T07:27:14.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8bf363740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacaf50 0x55a52eac8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8bf363740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eacb330 0x55a52eac9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8bf363740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eaf1440 0x55a52eaea520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8bf363740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b41f8420 con 0x55a52eacaf50 2024-03-06T07:27:14.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8bf363740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b41f7580 con 0x55a52eaf1440 2024-03-06T07:27:14.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8bf363740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b41f69b0 con 0x55a52eacb330 2024-03-06T07:27:14.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8ba439640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacaf50 0x55a52eac8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8ba439640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacaf50 0x55a52eac8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48258/0 (socket says 172.21.15.1:48258) 2024-03-06T07:27:14.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8ba439640 1 -- 172.21.15.1:0/1036495171 learned_addr learned my addr 172.21.15.1:0/1036495171 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:14.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8b9c38640 1 --2- 172.21.15.1:0/1036495171 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eacb330 0x55a52eac9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8bac3a640 1 --2- 172.21.15.1:0/1036495171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eaf1440 0x55a52eaea520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8ba439640 1 -- 172.21.15.1:0/1036495171 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eacb330 msgr2=0x55a52eac9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8ba439640 1 --2- 172.21.15.1:0/1036495171 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eacb330 0x55a52eac9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8ba439640 1 -- 172.21.15.1:0/1036495171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eaf1440 msgr2=0x55a52eaea520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8ba439640 1 --2- 172.21.15.1:0/1036495171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eaf1440 0x55a52eaea520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8ba439640 1 -- 172.21.15.1:0/1036495171 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a52eaf0da0 con 0x55a52eacaf50 2024-03-06T07:27:14.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8bac3a640 1 --2- 172.21.15.1:0/1036495171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eaf1440 0x55a52eaea520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:14.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8b9c38640 1 --2- 172.21.15.1:0/1036495171 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eacb330 0x55a52eac9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:14.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8ba439640 1 --2- 172.21.15.1:0/1036495171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacaf50 0x55a52eac8650 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4026040 tx=0x7fb8a4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fda60f6b4bf3132 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8977fe640 1 -- 172.21.15.1:0/1036495171 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4028070 con 0x55a52eacaf50 2024-03-06T07:27:14.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8977fe640 1 -- 172.21.15.1:0/1036495171 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a4029040 con 0x55a52eacaf50 2024-03-06T07:27:14.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.593+0000 7fb8977fe640 1 -- 172.21.15.1:0/1036495171 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4004d30 con 0x55a52eacaf50 2024-03-06T07:27:14.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.594+0000 7fb8bf363740 1 -- 172.21.15.1:0/1036495171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacaf50 msgr2=0x55a52eac8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.594+0000 7fb8bf363740 1 --2- 172.21.15.1:0/1036495171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacaf50 0x55a52eac8650 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4026040 tx=0x7fb8a4017000 comp rx=0 tx=0).stop 2024-03-06T07:27:14.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.594+0000 7fb8bf363740 1 -- 172.21.15.1:0/1036495171 shutdown_connections 2024-03-06T07:27:14.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.594+0000 7fb8bf363740 1 --2- 172.21.15.1:0/1036495171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eaf1440 0x55a52eaea520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.594+0000 7fb8bf363740 1 --2- 172.21.15.1:0/1036495171 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eacb330 0x55a52eac9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.594+0000 7fb8bf363740 1 --2- 172.21.15.1:0/1036495171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacaf50 0x55a52eac8650 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.594+0000 7fb8bf363740 1 -- 172.21.15.1:0/1036495171 >> 172.21.15.1:0/1036495171 conn(0x55a52eac94b0 msgr2=0x55a52eaefe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.594+0000 7fb8bf363740 1 -- 172.21.15.1:0/1036495171 shutdown_connections 2024-03-06T07:27:14.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.594+0000 7fb8bf363740 1 -- 172.21.15.1:0/1036495171 wait complete. 2024-03-06T07:27:14.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.595+0000 7fb8bf363740 1 -- start start 2024-03-06T07:27:14.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.595+0000 7fb8bf363740 1 Processor -- start 2024-03-06T07:27:14.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.595+0000 7fb8bf363740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eacaf50 0x55a52eac6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.595+0000 7fb8bf363740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacb330 0x55a52eb2ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.595+0000 7fb8bf363740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eaf1440 0x55a52eb2b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.595+0000 7fb8bf363740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a52eaec190 con 0x55a52eacb330 2024-03-06T07:27:14.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.595+0000 7fb8bf363740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b41f6fb0 con 0x55a52eacaf50 2024-03-06T07:27:14.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.595+0000 7fb8bf363740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a52eaef3c0 con 0x55a52eaf1440 2024-03-06T07:27:14.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.595+0000 7fb8bf363740 10 client.0 _ll_register_callbacks cb 0x7fb8a0004620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:14.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8b9c38640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacb330 0x55a52eb2ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8b9c38640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacb330 0x55a52eb2ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48264/0 (socket says 172.21.15.1:48264) 2024-03-06T07:27:14.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8b9c38640 1 -- 172.21.15.1:0/931522442 learned_addr learned my addr 172.21.15.1:0/931522442 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:14.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8bac3a640 1 --2- 172.21.15.1:0/931522442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eaf1440 0x55a52eb2b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8ba439640 1 --2- 172.21.15.1:0/931522442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eacaf50 0x55a52eac6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8b9c38640 1 -- 172.21.15.1:0/931522442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eaf1440 msgr2=0x55a52eb2b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8b9c38640 1 --2- 172.21.15.1:0/931522442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eaf1440 0x55a52eb2b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8b9c38640 1 -- 172.21.15.1:0/931522442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eacaf50 msgr2=0x55a52eac6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8b9c38640 1 --2- 172.21.15.1:0/931522442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eacaf50 0x55a52eac6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8b9c38640 1 -- 172.21.15.1:0/931522442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a52e80e6b0 con 0x55a52eacb330 2024-03-06T07:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8bac3a640 1 --2- 172.21.15.1:0/931522442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eaf1440 0x55a52eb2b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8b9c38640 1 --2- 172.21.15.1:0/931522442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacb330 0x55a52eb2ad60 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0060830 tx=0x7fb8b0058830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c33a8e13664e6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8957fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8957fa640 1 -- 172.21.15.1:0/931522442 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b00700c0 con 0x55a52eacb330 2024-03-06T07:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8957fa640 1 -- 172.21.15.1:0/931522442 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb8b0057b50 con 0x55a52eacb330 2024-03-06T07:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8bf363740 1 -- 172.21.15.1:0/931522442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb8a4029040 con 0x55a52eacb330 2024-03-06T07:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8957fa640 1 -- 172.21.15.1:0/931522442 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=157) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8b0077170 con 0x55a52eacb330 2024-03-06T07:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8bf363740 10 client.9669 fetch_fsmap learned FSMap version 157 2024-03-06T07:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.596+0000 7fb8bf363740 1 -- 172.21.15.1:0/931522442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=157,monmap=2+}) v3 -- 0x55a52eaca5a0 con 0x55a52eacb330 2024-03-06T07:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.597+0000 7fb8957fa640 1 -- 172.21.15.1:0/931522442 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a4029040 con 0x55a52eacb330 2024-03-06T07:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.597+0000 7fb8957fa640 1 -- 172.21.15.1:0/931522442 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 157) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb8b0054450 con 0x55a52eacb330 2024-03-06T07:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.597+0000 7fb8bf363740 10 client.9669 fetch_fsmap finished waiting for FSMap version 157 2024-03-06T07:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.597+0000 7fb8bf363740 10 client.9669 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.597+0000 7fb8bf363740 20 client.9669 populate_metadata read hostname 'smithi001' 2024-03-06T07:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.597+0000 7fb8bf363740 1 --2- 172.21.15.1:0/931522442 >> [v2:172.21.15.1:6833/1751109564,v1:172.21.15.1:6835/1751109564] conn(0x55a52eb74c00 0x55a52eb17a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.597+0000 7fb8bf363740 4 client.9669 mds_command: new command op to 9489 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.597+0000 7fb8bf363740 1 -- 172.21.15.1:0/931522442 --> [v2:172.21.15.1:6833/1751109564,v1:172.21.15.1:6835/1751109564] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a52ea36f40 con 0x55a52eb74c00 2024-03-06T07:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.597+0000 7fb8ba439640 1 --2- 172.21.15.1:0/931522442 >> [v2:172.21.15.1:6833/1751109564,v1:172.21.15.1:6835/1751109564] conn(0x55a52eb74c00 0x55a52eb17a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.597+0000 7fb8ba439640 1 --2- 172.21.15.1:0/931522442 >> [v2:172.21.15.1:6833/1751109564,v1:172.21.15.1:6835/1751109564] conn(0x55a52eb74c00 0x55a52eb17a20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cfa22523fe03dda server_cookie=bae9810b848b3e7 in_seq=0 out_seq=0 2024-03-06T07:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.597+0000 7fb8957fa640 10 client.9669 ms_handle_connect on v2:172.21.15.1:6833/1751109564 2024-03-06T07:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.597+0000 7fb8957fa640 1 -- 172.21.15.1:0/931522442 <== mds.0 v2:172.21.15.1:6833/1751109564 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55a52ea36f40 con 0x55a52eb74c00 2024-03-06T07:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.597+0000 7fb8957fa640 10 client.9669 handle_command_reply: tid=0 2024-03-06T07:27:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.597+0000 7fb8bf363740 1 client.9669 shutdown 2024-03-06T07:27:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.597+0000 7fb8bf363740 1 -- 172.21.15.1:0/931522442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacb330 msgr2=0x55a52eb2ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.597+0000 7fb8bf363740 1 --2- 172.21.15.1:0/931522442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacb330 0x55a52eb2ad60 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0060830 tx=0x7fb8b0058830 comp rx=0 tx=0).stop 2024-03-06T07:27:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bf363740 1 -- 172.21.15.1:0/931522442 shutdown_connections 2024-03-06T07:27:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bf363740 1 --2- 172.21.15.1:0/931522442 >> [v2:172.21.15.1:6833/1751109564,v1:172.21.15.1:6835/1751109564] conn(0x55a52eb74c00 0x55a52eb17a20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bf363740 1 --2- 172.21.15.1:0/931522442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55a52eaf1440 0x55a52eb2b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bf363740 1 --2- 172.21.15.1:0/931522442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a52eacb330 0x55a52eb2ad60 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8957fa640 0 client.9669 ms_handle_reset on v2:172.21.15.1:6833/1751109564 2024-03-06T07:27:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bf363740 1 --2- 172.21.15.1:0/931522442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a52eacaf50 0x55a52eac6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bf363740 1 -- 172.21.15.1:0/931522442 >> 172.21.15.1:0/931522442 conn(0x55a52eac94b0 msgr2=0x55a52eac67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bf363740 1 -- 172.21.15.1:0/931522442 shutdown_connections 2024-03-06T07:27:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bf363740 1 -- 172.21.15.1:0/931522442 wait complete. 2024-03-06T07:27:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bf363740 20 client.9669 trim_cache size 0 max 16384 2024-03-06T07:27:14.607 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:27:14.607 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:27:14.607 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:27:14.607 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 9489, 2024-03-06T07:27:14.607 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:27:14.607 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:27:14.607 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:27:14.607 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 9.3371524539999999, 2024-03-06T07:27:14.607 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 157, 2024-03-06T07:27:14.607 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 171, 2024-03-06T07:27:14.607 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 171, 2024-03-06T07:27:14.607 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 14.392095619999999 2024-03-06T07:27:14.607 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:27:14.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bc69d640 1 -- 172.21.15.1:0/921034704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb89806b500 msgr2=0x7fb89806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bc69d640 1 --2- 172.21.15.1:0/921034704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb89806b500 0x7fb89806d9c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4001800 tx=0x7fb8a4001630 comp rx=0 tx=0).stop 2024-03-06T07:27:14.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bc69d640 1 -- 172.21.15.1:0/921034704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8b40aa920 msgr2=0x7fb8b4156a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bc69d640 1 --2- 172.21.15.1:0/921034704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8b40aa920 0x7fb8b4156a20 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000f820 tx=0x7fb8b000fce0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bc69d640 1 -- 172.21.15.1:0/921034704 shutdown_connections 2024-03-06T07:27:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bc69d640 1 --2- 172.21.15.1:0/921034704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb89806b500 0x7fb89806d9c0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bc69d640 1 --2- 172.21.15.1:0/921034704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8b4156f60 0x7fb8b40c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bc69d640 1 --2- 172.21.15.1:0/921034704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8b40aa920 0x7fb8b4156a20 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bc69d640 1 --2- 172.21.15.1:0/921034704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8a0004d40 0x7fb8b4155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bc69d640 1 -- 172.21.15.1:0/921034704 >> 172.21.15.1:0/921034704 conn(0x7fb8b4066d00 msgr2=0x7fb8b4067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bc69d640 1 -- 172.21.15.1:0/921034704 shutdown_connections 2024-03-06T07:27:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.598+0000 7fb8bc69d640 1 -- 172.21.15.1:0/921034704 wait complete. 2024-03-06T07:27:14.611 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2024-03-06T07:27:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.713+0000 7f44ac8a4640 1 Processor -- start 2024-03-06T07:27:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.713+0000 7f44ac8a4640 1 -- start start 2024-03-06T07:27:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.714+0000 7f44ac8a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a8152650 0x7f44a8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.714+0000 7f44ac8a4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f44a80c6b10 0x7f44a8158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.714+0000 7f44ac8a4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a80c2fc0 con 0x7f44a80bf9d0 2024-03-06T07:27:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.714+0000 7f44ac8a4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44a8155800 con 0x7f44a80c6b10 2024-03-06T07:27:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.714+0000 7f44ac8a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80c6690 con 0x7f44a8152650 2024-03-06T07:27:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.714+0000 7f44a6d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f44a80bf9d0 0x7f44a80c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42946/0 (socket says 172.21.15.1:42946) 2024-03-06T07:27:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.714+0000 7f44a6d9d640 1 -- 172.21.15.1:0/1074801911 learned_addr learned my addr 172.21.15.1:0/1074801911 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a659c640 1 --2- 172.21.15.1:0/1074801911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a8152650 0x7f44a8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a5d9b640 1 --2- 172.21.15.1:0/1074801911 >> v2:172.21.15.98:3300/0 conn(0x7f44a80c6b10 0x7f44a8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a559a640 1 -- 172.21.15.1:0/1074801911 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870721793 0 0) 0x7f44a80c2fc0 con 0x7f44a80bf9d0 2024-03-06T07:27:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a559a640 1 -- 172.21.15.1:0/1074801911 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4490003520 con 0x7f44a80bf9d0 2024-03-06T07:27:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a559a640 1 -- 172.21.15.1:0/1074801911 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4255216781 0 0) 0x7f4490003520 con 0x7f44a80bf9d0 2024-03-06T07:27:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a559a640 1 -- 172.21.15.1:0/1074801911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a8152650 msgr2=0x7f44a8155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a559a640 1 --2- 172.21.15.1:0/1074801911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a8152650 0x7f44a8155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a559a640 1 -- 172.21.15.1:0/1074801911 >> v2:172.21.15.98:3300/0 conn(0x7f44a80c6b10 msgr2=0x7f44a8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a559a640 1 --2- 172.21.15.1:0/1074801911 >> v2:172.21.15.98:3300/0 conn(0x7f44a80c6b10 0x7f44a8158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a559a640 1 -- 172.21.15.1:0/1074801911 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a80c7280 con 0x7f44a80bf9d0 2024-03-06T07:27:14.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a559a640 1 -- 172.21.15.1:0/1074801911 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f44a00019f0 con 0x7f44a80bf9d0 2024-03-06T07:27:14.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a559a640 1 -- 172.21.15.1:0/1074801911 >> v1:172.21.15.1:6789/0 conn(0x7f44a80bf9d0 legacy=0x7f44a80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a559a640 1 --2- 172.21.15.1:0/1074801911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44900039b0 0x7f4490003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a559a640 1 --2- 172.21.15.1:0/1074801911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4490004800 0x7f4490004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a559a640 1 --2- 172.21.15.1:0/1074801911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a8152650 0x7f44900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a559a640 1 -- 172.21.15.1:0/1074801911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a8155800 con 0x7f4490004800 2024-03-06T07:27:14.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a559a640 1 -- 172.21.15.1:0/1074801911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80c6690 con 0x7f44900039b0 2024-03-06T07:27:14.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a559a640 1 -- 172.21.15.1:0/1074801911 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449000b9c0 con 0x7f44a8152650 2024-03-06T07:27:14.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a5d9b640 1 --2- 172.21.15.1:0/1074801911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4490004800 0x7f4490004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a659c640 1 --2- 172.21.15.1:0/1074801911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44900039b0 0x7f4490003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a6d9d640 1 --2- 172.21.15.1:0/1074801911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a8152650 0x7f44900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a5d9b640 1 -- 172.21.15.1:0/1074801911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a8152650 msgr2=0x7f44900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a5d9b640 1 --2- 172.21.15.1:0/1074801911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a8152650 0x7f44900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a5d9b640 1 -- 172.21.15.1:0/1074801911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44900039b0 msgr2=0x7f4490003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a5d9b640 1 --2- 172.21.15.1:0/1074801911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44900039b0 0x7f4490003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.715+0000 7f44a5d9b640 1 -- 172.21.15.1:0/1074801911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4490003520 con 0x7f4490004800 2024-03-06T07:27:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.716+0000 7f44a5d9b640 1 --2- 172.21.15.1:0/1074801911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4490004800 0x7f4490004c80 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f449c009af0 tx=0x7f4490005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d9a1033a4d5141d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.716+0000 7f44a559a640 1 -- 172.21.15.1:0/1074801911 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c039070 con 0x7f4490004800 2024-03-06T07:27:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.716+0000 7f44a559a640 1 -- 172.21.15.1:0/1074801911 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449c02c710 con 0x7f4490004800 2024-03-06T07:27:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.716+0000 7f44a559a640 1 -- 172.21.15.1:0/1074801911 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c009690 con 0x7f4490004800 2024-03-06T07:27:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.717+0000 7f44ac8a4640 1 -- 172.21.15.1:0/1074801911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4490004800 msgr2=0x7f4490004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.717+0000 7f44ac8a4640 1 --2- 172.21.15.1:0/1074801911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4490004800 0x7f4490004c80 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f449c009af0 tx=0x7f4490005830 comp rx=0 tx=0).stop 2024-03-06T07:27:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.717+0000 7f44a6d9d640 1 -- 172.21.15.1:0/1074801911 reap_dead start 2024-03-06T07:27:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.717+0000 7f44ac8a4640 1 -- 172.21.15.1:0/1074801911 shutdown_connections 2024-03-06T07:27:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.717+0000 7f44ac8a4640 1 -- 172.21.15.1:0/1074801911 >> 172.21.15.1:0/1074801911 conn(0x7f44a8066ee0 msgr2=0x7f44a80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.717+0000 7f44ac8a4640 1 -- 172.21.15.1:0/1074801911 shutdown_connections 2024-03-06T07:27:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.717+0000 7f44ac8a4640 1 -- 172.21.15.1:0/1074801911 wait complete. 2024-03-06T07:27:14.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.719+0000 7f44ac8a4640 1 Processor -- start 2024-03-06T07:27:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.719+0000 7f44ac8a4640 1 -- start start 2024-03-06T07:27:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44ac8a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4490004800 0x7f44a8144dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44ac8a4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a814faf0 0x7f44a814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44ac8a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44a814afd0 0x7f44a8157d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44ac8a4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a8155490 con 0x7f44a814faf0 2024-03-06T07:27:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44ac8a4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a81551b0 con 0x7f44a814afd0 2024-03-06T07:27:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44ac8a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a8152a30 con 0x7f4490004800 2024-03-06T07:27:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44a5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a814faf0 0x7f44a814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44a5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a814faf0 0x7f44a814aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48276/0 (socket says 172.21.15.1:48276) 2024-03-06T07:27:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44a5d9b640 1 -- 172.21.15.1:0/4225722069 learned_addr learned my addr 172.21.15.1:0/4225722069 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44a659c640 1 --2- 172.21.15.1:0/4225722069 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4490004800 0x7f44a8144dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44a6d9d640 1 --2- 172.21.15.1:0/4225722069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44a814afd0 0x7f44a8157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44a5d9b640 1 -- 172.21.15.1:0/4225722069 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4490004800 msgr2=0x7f44a8144dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44a5d9b640 1 --2- 172.21.15.1:0/4225722069 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4490004800 0x7f44a8144dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44a5d9b640 1 -- 172.21.15.1:0/4225722069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44a814afd0 msgr2=0x7f44a8157d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44a5d9b640 1 --2- 172.21.15.1:0/4225722069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44a814afd0 0x7f44a8157d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44a5d9b640 1 -- 172.21.15.1:0/4225722069 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a80ca190 con 0x7f44a814faf0 2024-03-06T07:27:14.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44a6d9d640 1 --2- 172.21.15.1:0/4225722069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44a814afd0 0x7f44a8157d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:14.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f44a5d9b640 1 --2- 172.21.15.1:0/4225722069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a814faf0 0x7f44a814aa90 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f449c009af0 tx=0x7f449c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.720+0000 7f4496ffd640 1 -- 172.21.15.1:0/4225722069 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c042020 con 0x7f44a814faf0 2024-03-06T07:27:14.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.721+0000 7f4496ffd640 1 -- 172.21.15.1:0/4225722069 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449c03a040 con 0x7f44a814faf0 2024-03-06T07:27:14.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.721+0000 7f44ac8a4640 1 -- 172.21.15.1:0/4225722069 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a8156530 con 0x7f44a814faf0 2024-03-06T07:27:14.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.721+0000 7f4496ffd640 1 -- 172.21.15.1:0/4225722069 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c039070 con 0x7f44a814faf0 2024-03-06T07:27:14.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.721+0000 7f44ac8a4640 1 -- 172.21.15.1:0/4225722069 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a81eefe0 con 0x7f44a814faf0 2024-03-06T07:27:14.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.722+0000 7f44ac8a4640 1 -- 172.21.15.1:0/4225722069 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a8067fd0 con 0x7f44a814faf0 2024-03-06T07:27:14.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.722+0000 7f4496ffd640 1 -- 172.21.15.1:0/4225722069 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f449c045020 con 0x7f44a814faf0 2024-03-06T07:27:14.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.723+0000 7f4496ffd640 1 --2- 172.21.15.1:0/4225722069 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 0x7f448806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.723+0000 7f4496ffd640 1 -- 172.21.15.1:0/4225722069 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f449c0abe30 con 0x7f44a814faf0 2024-03-06T07:27:14.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.723+0000 7f44a659c640 1 --2- 172.21.15.1:0/4225722069 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 0x7f448806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.723+0000 7f44a659c640 1 --2- 172.21.15.1:0/4225722069 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 0x7f448806d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f44a8145af0 tx=0x7f4498008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.726+0000 7f4496ffd640 1 -- 172.21.15.1:0/4225722069 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f449c04c070 con 0x7f44a814faf0 2024-03-06T07:27:14.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:14.949+0000 7f44ac8a4640 1 -- 172.21.15.1:0/4225722069 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"} v 0) v1 -- 0x7f44a806b500 con 0x7f44a814faf0 2024-03-06T07:27:15.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.306+0000 7f4496ffd640 1 -- 172.21.15.1:0/4225722069 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"}]=0 cephfs marked down. v158) v1 ==== 125+0+0 (secure 0 0 0) 0x7f449c07ad20 con 0x7f44a814faf0 2024-03-06T07:27:15.308 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked down. 2024-03-06T07:27:15.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.309+0000 7f44ac8a4640 1 -- 172.21.15.1:0/4225722069 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 msgr2=0x7f448806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.309+0000 7f44ac8a4640 1 --2- 172.21.15.1:0/4225722069 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 0x7f448806d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f44a8145af0 tx=0x7f4498008040 comp rx=0 tx=0).stop 2024-03-06T07:27:15.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.309+0000 7f44ac8a4640 1 -- 172.21.15.1:0/4225722069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a814faf0 msgr2=0x7f44a814aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.309+0000 7f44ac8a4640 1 --2- 172.21.15.1:0/4225722069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a814faf0 0x7f44a814aa90 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f449c009af0 tx=0x7f449c004730 comp rx=0 tx=0).stop 2024-03-06T07:27:15.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.309+0000 7f44ac8a4640 1 -- 172.21.15.1:0/4225722069 shutdown_connections 2024-03-06T07:27:15.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.309+0000 7f44ac8a4640 1 --2- 172.21.15.1:0/4225722069 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 0x7f448806d990 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.309+0000 7f44ac8a4640 1 --2- 172.21.15.1:0/4225722069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44a814afd0 0x7f44a8157d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.309+0000 7f44ac8a4640 1 --2- 172.21.15.1:0/4225722069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a814faf0 0x7f44a814aa90 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.309+0000 7f44ac8a4640 1 --2- 172.21.15.1:0/4225722069 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4490004800 0x7f44a8144dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.309+0000 7f44ac8a4640 1 -- 172.21.15.1:0/4225722069 >> 172.21.15.1:0/4225722069 conn(0x7f44a8066ee0 msgr2=0x7f44a80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.309+0000 7f44ac8a4640 1 -- 172.21.15.1:0/4225722069 shutdown_connections 2024-03-06T07:27:15.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.309+0000 7f44ac8a4640 1 -- 172.21.15.1:0/4225722069 wait complete. 2024-03-06T07:27:15.322 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:15.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.422+0000 7f21370df640 1 Processor -- start 2024-03-06T07:27:15.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.422+0000 7f21370df640 1 -- start start 2024-03-06T07:27:15.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.422+0000 7f21370df640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2130156500 0x7f2130152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.422+0000 7f21370df640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21301569e0 con 0x7f213014cdc0 2024-03-06T07:27:15.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.422+0000 7f21370df640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2130156cc0 con 0x7f213006b510 2024-03-06T07:27:15.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.422+0000 7f21370df640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21300c2dd0 con 0x7f2130156500 2024-03-06T07:27:15.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.423+0000 7f212ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2130156500 0x7f2130152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.423+0000 7f212ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2130156500 0x7f2130152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48288/0 (socket says 172.21.15.1:48288) 2024-03-06T07:27:15.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.423+0000 7f2134e7b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f213014cdc0 0x7f21300c2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42960/0 (socket says 172.21.15.1:42960) 2024-03-06T07:27:15.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.423+0000 7f212ffff640 1 -- 172.21.15.1:0/1059777603 learned_addr learned my addr 172.21.15.1:0/1059777603 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:15.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.423+0000 7f212ffff640 1 -- 172.21.15.1:0/1059777603 >> v1:172.21.15.98:6789/0 conn(0x7f213006b510 legacy=0x7f213006b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.423+0000 7f212ffff640 1 -- 172.21.15.1:0/1059777603 >> v1:172.21.15.1:6789/0 conn(0x7f213014cdc0 legacy=0x7f21300c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.423+0000 7f212ffff640 1 -- 172.21.15.1:0/1059777603 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2130156fa0 con 0x7f2130156500 2024-03-06T07:27:15.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.424+0000 7f212ffff640 1 --2- 172.21.15.1:0/1059777603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2130156500 0x7f2130152f10 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f2118009870 tx=0x7f211802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a020eb6da1ffc798 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.424+0000 7f212f7fe640 1 -- 172.21.15.1:0/1059777603 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2118035020 con 0x7f2130156500 2024-03-06T07:27:15.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.424+0000 7f212f7fe640 1 -- 172.21.15.1:0/1059777603 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2118031870 con 0x7f2130156500 2024-03-06T07:27:15.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.424+0000 7f212f7fe640 1 -- 172.21.15.1:0/1059777603 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2118031c40 con 0x7f2130156500 2024-03-06T07:27:15.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.424+0000 7f21370df640 1 -- 172.21.15.1:0/1059777603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2130156500 msgr2=0x7f2130152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.424+0000 7f21370df640 1 --2- 172.21.15.1:0/1059777603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2130156500 0x7f2130152f10 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f2118009870 tx=0x7f211802b910 comp rx=0 tx=0).stop 2024-03-06T07:27:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.424+0000 7f21370df640 1 -- 172.21.15.1:0/1059777603 shutdown_connections 2024-03-06T07:27:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.424+0000 7f21370df640 1 --2- 172.21.15.1:0/1059777603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2130156500 0x7f2130152f10 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.424+0000 7f21370df640 1 -- 172.21.15.1:0/1059777603 >> 172.21.15.1:0/1059777603 conn(0x7f21300a9700 msgr2=0x7f21300a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.424+0000 7f21370df640 1 -- 172.21.15.1:0/1059777603 shutdown_connections 2024-03-06T07:27:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.424+0000 7f21370df640 1 -- 172.21.15.1:0/1059777603 wait complete. 2024-03-06T07:27:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.425+0000 7f21370df640 1 Processor -- start 2024-03-06T07:27:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.425+0000 7f21370df640 1 -- start start 2024-03-06T07:27:15.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.426+0000 7f21370df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f213006b510 0x7f21301ec660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.426+0000 7f21370df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f213014cdc0 0x7f21300ab700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.426+0000 7f21370df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2130156500 0x7f213014ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.426+0000 7f21370df640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21300c65e0 con 0x7f2130156500 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.426+0000 7f21370df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21300c5a70 con 0x7f213014cdc0 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.426+0000 7f21370df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21300c2a60 con 0x7f213006b510 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.426+0000 7f213567c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2130156500 0x7f213014ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.426+0000 7f213567c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2130156500 0x7f213014ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48300/0 (socket says 172.21.15.1:48300) 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.426+0000 7f213567c640 1 -- 172.21.15.1:0/1843048795 learned_addr learned my addr 172.21.15.1:0/1843048795 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.427+0000 7f2134e7b640 1 --2- 172.21.15.1:0/1843048795 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f213006b510 0x7f21301ec660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.427+0000 7f212ffff640 1 --2- 172.21.15.1:0/1843048795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f213014cdc0 0x7f21300ab700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.427+0000 7f213567c640 1 -- 172.21.15.1:0/1843048795 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f213006b510 msgr2=0x7f21301ec660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.427+0000 7f213567c640 1 --2- 172.21.15.1:0/1843048795 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f213006b510 0x7f21301ec660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.427+0000 7f213567c640 1 -- 172.21.15.1:0/1843048795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f213014cdc0 msgr2=0x7f21300ab700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.427+0000 7f213567c640 1 --2- 172.21.15.1:0/1843048795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f213014cdc0 0x7f21300ab700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.427+0000 7f213567c640 1 -- 172.21.15.1:0/1843048795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21301ecf30 con 0x7f2130156500 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.427+0000 7f212ffff640 1 --2- 172.21.15.1:0/1843048795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f213014cdc0 0x7f21300ab700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:15.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.427+0000 7f213567c640 1 --2- 172.21.15.1:0/1843048795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2130156500 0x7f213014ba40 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f212000c410 tx=0x7f212000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.427+0000 7f212d7fa640 1 -- 172.21.15.1:0/1843048795 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2120011790 con 0x7f2130156500 2024-03-06T07:27:15.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.427+0000 7f212d7fa640 1 -- 172.21.15.1:0/1843048795 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2120017070 con 0x7f2130156500 2024-03-06T07:27:15.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.427+0000 7f21370df640 1 -- 172.21.15.1:0/1843048795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21300ad140 con 0x7f2130156500 2024-03-06T07:27:15.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.427+0000 7f212d7fa640 1 -- 172.21.15.1:0/1843048795 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2120011b70 con 0x7f2130156500 2024-03-06T07:27:15.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.427+0000 7f21370df640 1 -- 172.21.15.1:0/1843048795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f213014feb0 con 0x7f2130156500 2024-03-06T07:27:15.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.428+0000 7f21370df640 1 -- 172.21.15.1:0/1843048795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2130067fd0 con 0x7f2130156500 2024-03-06T07:27:15.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.429+0000 7f212d7fa640 1 -- 172.21.15.1:0/1843048795 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2120011d10 con 0x7f2130156500 2024-03-06T07:27:15.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.429+0000 7f212d7fa640 1 --2- 172.21.15.1:0/1843048795 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f211006b500 0x7f211006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.429+0000 7f212d7fa640 1 -- 172.21.15.1:0/1843048795 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2120089ef0 con 0x7f2130156500 2024-03-06T07:27:15.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.429+0000 7f2134e7b640 1 --2- 172.21.15.1:0/1843048795 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f211006b500 0x7f211006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.430+0000 7f2134e7b640 1 --2- 172.21.15.1:0/1843048795 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f211006b500 0x7f211006d9c0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f21280012e0 tx=0x7f2128005b70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.433+0000 7f212d7fa640 1 -- 172.21.15.1:0/1843048795 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2120021020 con 0x7f2130156500 2024-03-06T07:27:15.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.655+0000 7f21370df640 1 -- 172.21.15.1:0/1843048795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21300c5850 con 0x7f2130156500 2024-03-06T07:27:15.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.656+0000 7f212d7fa640 1 -- 172.21.15.1:0/1843048795 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f2120058c70 con 0x7f2130156500 2024-03-06T07:27:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:15.658 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":159,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:15.309451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9489":{"gid":9489,"name":"c","rank":0,"incarnation":155,"state":"up:stopping","state_seq":3,"addr":"172.21.15.1:6835/1751109564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1751109564},{"type":"v1","addr":"172.21.15.1:6835","nonce":1751109564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:27:15.658 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 159 2024-03-06T07:27:15.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.658+0000 7f21370df640 1 -- 172.21.15.1:0/1843048795 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f211006b500 msgr2=0x7f211006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.658+0000 7f21370df640 1 --2- 172.21.15.1:0/1843048795 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f211006b500 0x7f211006d9c0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f21280012e0 tx=0x7f2128005b70 comp rx=0 tx=0).stop 2024-03-06T07:27:15.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.658+0000 7f21370df640 1 -- 172.21.15.1:0/1843048795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2130156500 msgr2=0x7f213014ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.658+0000 7f21370df640 1 --2- 172.21.15.1:0/1843048795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2130156500 0x7f213014ba40 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f212000c410 tx=0x7f212000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.658+0000 7f21370df640 1 -- 172.21.15.1:0/1843048795 shutdown_connections 2024-03-06T07:27:15.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.658+0000 7f21370df640 1 --2- 172.21.15.1:0/1843048795 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f211006b500 0x7f211006d9c0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.658+0000 7f21370df640 1 --2- 172.21.15.1:0/1843048795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2130156500 0x7f213014ba40 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.658+0000 7f21370df640 1 --2- 172.21.15.1:0/1843048795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f213014cdc0 0x7f21300ab700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.658+0000 7f21370df640 1 --2- 172.21.15.1:0/1843048795 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f213006b510 0x7f21301ec660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.658+0000 7f21370df640 1 -- 172.21.15.1:0/1843048795 >> 172.21.15.1:0/1843048795 conn(0x7f21300a9700 msgr2=0x7f21300c4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.658+0000 7f21370df640 1 -- 172.21.15.1:0/1843048795 shutdown_connections 2024-03-06T07:27:15.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:15.658+0000 7f21370df640 1 -- 172.21.15.1:0/1843048795 wait complete. 2024-03-06T07:27:16.074 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 159, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:04.619091+0000', 'modified': '2024-03-06T07:27:15.309451+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9489': {'gid': 9489, 'name': 'c', 'rank': 0, 'incarnation': 155, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.1:6835/1751109564', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1751109564}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1751109564}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:16.074 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9489:up:stopping 2024-03-06T07:27:17.075 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:17.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.187+0000 7f5556565640 1 Processor -- start 2024-03-06T07:27:17.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.187+0000 7f5556565640 1 -- start start 2024-03-06T07:27:17.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.188+0000 7f5556565640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55500c5850 0x7f55500c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.188+0000 7f5556565640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5550155dc0 con 0x7f5550155220 2024-03-06T07:27:17.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.188+0000 7f5556565640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5550158910 con 0x7f5550152920 2024-03-06T07:27:17.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.188+0000 7f5556565640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5550152d00 con 0x7f55500c5850 2024-03-06T07:27:17.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.188+0000 7f5554b02640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5550155220 0x7f55501556b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42966/0 (socket says 172.21.15.1:42966) 2024-03-06T07:27:17.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.188+0000 7f5554b02640 1 -- 172.21.15.1:0/1656906770 learned_addr learned my addr 172.21.15.1:0/1656906770 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:17.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.188+0000 7f554f7fe640 1 --2- 172.21.15.1:0/1656906770 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55500c5850 0x7f55500c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.188+0000 7f554effd640 1 -- 172.21.15.1:0/1656906770 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569008510 0 0) 0x7f5550155dc0 con 0x7f5550155220 2024-03-06T07:27:17.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.188+0000 7f554effd640 1 -- 172.21.15.1:0/1656906770 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5538003520 con 0x7f5550155220 2024-03-06T07:27:17.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.189+0000 7f554effd640 1 -- 172.21.15.1:0/1656906770 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076390768 0 0) 0x7f5550158910 con 0x7f5550152920 2024-03-06T07:27:17.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.189+0000 7f554effd640 1 -- 172.21.15.1:0/1656906770 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5550155dc0 con 0x7f5550152920 2024-03-06T07:27:17.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.189+0000 7f554f7fe640 1 -- 172.21.15.1:0/1656906770 >> v1:172.21.15.98:6789/0 conn(0x7f5550152920 legacy=0x7f555006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.189+0000 7f554f7fe640 1 -- 172.21.15.1:0/1656906770 >> v1:172.21.15.1:6789/0 conn(0x7f5550155220 legacy=0x7f55501556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.189+0000 7f554effd640 1 -- 172.21.15.1:0/1656906770 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2761504034 0 0) 0x7f5538003520 con 0x7f5550155220 2024-03-06T07:27:17.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.189+0000 7f554f7fe640 1 -- 172.21.15.1:0/1656906770 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5550158bf0 con 0x7f55500c5850 2024-03-06T07:27:17.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.189+0000 7f554f7fe640 1 --2- 172.21.15.1:0/1656906770 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55500c5850 0x7f55500c5cb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f55440097d0 tx=0x7f554402b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d55614493746e511 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.190+0000 7f554effd640 1 -- 172.21.15.1:0/1656906770 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5544037070 con 0x7f55500c5850 2024-03-06T07:27:17.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.190+0000 7f554effd640 1 -- 172.21.15.1:0/1656906770 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5544030510 con 0x7f55500c5850 2024-03-06T07:27:17.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.190+0000 7f554effd640 1 -- 172.21.15.1:0/1656906770 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55480019f0 con 0x7f55500c5850 2024-03-06T07:27:17.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.190+0000 7f5556565640 1 -- 172.21.15.1:0/1656906770 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55500c5850 msgr2=0x7f55500c5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.190+0000 7f5556565640 1 --2- 172.21.15.1:0/1656906770 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55500c5850 0x7f55500c5cb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f55440097d0 tx=0x7f554402b770 comp rx=0 tx=0).stop 2024-03-06T07:27:17.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.190+0000 7f5556565640 1 -- 172.21.15.1:0/1656906770 shutdown_connections 2024-03-06T07:27:17.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.190+0000 7f5556565640 1 --2- 172.21.15.1:0/1656906770 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55500c5850 0x7f55500c5cb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.190+0000 7f5556565640 1 -- 172.21.15.1:0/1656906770 >> 172.21.15.1:0/1656906770 conn(0x7f55500a9700 msgr2=0x7f55500a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.190+0000 7f5556565640 1 -- 172.21.15.1:0/1656906770 shutdown_connections 2024-03-06T07:27:17.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.190+0000 7f5556565640 1 -- 172.21.15.1:0/1656906770 wait complete. 2024-03-06T07:27:17.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.191+0000 7f5556565640 1 Processor -- start 2024-03-06T07:27:17.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.192+0000 7f5556565640 1 -- start start 2024-03-06T07:27:17.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.192+0000 7f5556565640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55500c5850 0x7f55500a8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.192+0000 7f5556565640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5550152920 0x7f5550149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.192+0000 7f5556565640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5550155220 0x7f55501eab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.192+0000 7f5556565640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5550159240 con 0x7f55500c5850 2024-03-06T07:27:17.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.192+0000 7f5556565640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55501596d0 con 0x7f5550155220 2024-03-06T07:27:17.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.192+0000 7f5556565640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55500c6380 con 0x7f5550152920 2024-03-06T07:27:17.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.192+0000 7f554ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55500c5850 0x7f55500a8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.193+0000 7f5554b02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5550155220 0x7f55501eab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.193+0000 7f5554b02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5550155220 0x7f55501eab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:56752/0 (socket says 172.21.15.1:56752) 2024-03-06T07:27:17.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.193+0000 7f554ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55500c5850 0x7f55500a8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48316/0 (socket says 172.21.15.1:48316) 2024-03-06T07:27:17.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.193+0000 7f5554b02640 1 -- 172.21.15.1:0/1129932569 learned_addr learned my addr 172.21.15.1:0/1129932569 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:17.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.193+0000 7f554f7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5550152920 0x7f5550149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.193+0000 7f554ffff640 1 -- 172.21.15.1:0/1129932569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5550152920 msgr2=0x7f5550149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.193+0000 7f554ffff640 1 --2- 172.21.15.1:0/1129932569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5550152920 0x7f5550149f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.193+0000 7f554ffff640 1 -- 172.21.15.1:0/1129932569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5550155220 msgr2=0x7f55501eab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.193+0000 7f554ffff640 1 --2- 172.21.15.1:0/1129932569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5550155220 0x7f55501eab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.193+0000 7f554ffff640 1 -- 172.21.15.1:0/1129932569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555006e3b0 con 0x7f55500c5850 2024-03-06T07:27:17.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.193+0000 7f5554b02640 1 --2- 172.21.15.1:0/1129932569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5550155220 0x7f55501eab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:17.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.193+0000 7f554ffff640 1 --2- 172.21.15.1:0/1129932569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55500c5850 0x7f55500a8bd0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f554000c400 tx=0x7f554000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.193+0000 7f554cff9640 1 -- 172.21.15.1:0/1129932569 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5540017070 con 0x7f55500c5850 2024-03-06T07:27:17.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.193+0000 7f554cff9640 1 -- 172.21.15.1:0/1129932569 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55400052d0 con 0x7f55500c5850 2024-03-06T07:27:17.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.193+0000 7f554cff9640 1 -- 172.21.15.1:0/1129932569 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5540003830 con 0x7f55500c5850 2024-03-06T07:27:17.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.193+0000 7f5556565640 1 -- 172.21.15.1:0/1129932569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55501567a0 con 0x7f55500c5850 2024-03-06T07:27:17.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.193+0000 7f5556565640 1 -- 172.21.15.1:0/1129932569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5550147f10 con 0x7f55500c5850 2024-03-06T07:27:17.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.194+0000 7f5556565640 1 -- 172.21.15.1:0/1129932569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5550067fd0 con 0x7f55500c5850 2024-03-06T07:27:17.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.195+0000 7f554cff9640 1 -- 172.21.15.1:0/1129932569 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f554001a070 con 0x7f55500c5850 2024-03-06T07:27:17.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.195+0000 7f554cff9640 1 --2- 172.21.15.1:0/1129932569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f553c06b4d0 0x7f553c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.195+0000 7f554cff9640 1 -- 172.21.15.1:0/1129932569 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5540089670 con 0x7f55500c5850 2024-03-06T07:27:17.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.195+0000 7f554f7fe640 1 --2- 172.21.15.1:0/1129932569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f553c06b4d0 0x7f553c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.196+0000 7f554f7fe640 1 --2- 172.21.15.1:0/1129932569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f553c06b4d0 0x7f553c06d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f5544034000 tx=0x7f5544008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.200+0000 7f554cff9640 1 -- 172.21.15.1:0/1129932569 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5540057da0 con 0x7f55500c5850 2024-03-06T07:27:17.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.418+0000 7f5556565640 1 -- 172.21.15.1:0/1129932569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f555006bda0 con 0x7f55500c5850 2024-03-06T07:27:17.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.418+0000 7f554cff9640 1 -- 172.21.15.1:0/1129932569 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f554005be40 con 0x7f55500c5850 2024-03-06T07:27:17.421 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:17.421 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":159,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:15.309451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9489":{"gid":9489,"name":"c","rank":0,"incarnation":155,"state":"up:stopping","state_seq":3,"addr":"172.21.15.1:6835/1751109564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1751109564},{"type":"v1","addr":"172.21.15.1:6835","nonce":1751109564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:27:17.421 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 159 2024-03-06T07:27:17.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.420+0000 7f5556565640 1 -- 172.21.15.1:0/1129932569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f553c06b4d0 msgr2=0x7f553c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.420+0000 7f5556565640 1 --2- 172.21.15.1:0/1129932569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f553c06b4d0 0x7f553c06d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f5544034000 tx=0x7f5544008030 comp rx=0 tx=0).stop 2024-03-06T07:27:17.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.420+0000 7f5556565640 1 -- 172.21.15.1:0/1129932569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55500c5850 msgr2=0x7f55500a8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.420+0000 7f5556565640 1 --2- 172.21.15.1:0/1129932569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55500c5850 0x7f55500a8bd0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f554000c400 tx=0x7f554000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.420+0000 7f5556565640 1 -- 172.21.15.1:0/1129932569 shutdown_connections 2024-03-06T07:27:17.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.420+0000 7f5556565640 1 --2- 172.21.15.1:0/1129932569 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f553c06b4d0 0x7f553c06d990 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.420+0000 7f5556565640 1 --2- 172.21.15.1:0/1129932569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5550155220 0x7f55501eab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.420+0000 7f5556565640 1 --2- 172.21.15.1:0/1129932569 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5550152920 0x7f5550149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.420+0000 7f5556565640 1 --2- 172.21.15.1:0/1129932569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55500c5850 0x7f55500a8bd0 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.420+0000 7f5556565640 1 -- 172.21.15.1:0/1129932569 >> 172.21.15.1:0/1129932569 conn(0x7f55500a9700 msgr2=0x7f5550153f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.421+0000 7f5556565640 1 -- 172.21.15.1:0/1129932569 shutdown_connections 2024-03-06T07:27:17.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:17.421+0000 7f5556565640 1 -- 172.21.15.1:0/1129932569 wait complete. 2024-03-06T07:27:17.835 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 159, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:04.619091+0000', 'modified': '2024-03-06T07:27:15.309451+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9489': {'gid': 9489, 'name': 'c', 'rank': 0, 'incarnation': 155, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.1:6835/1751109564', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1751109564}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1751109564}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:17.836 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9489:up:stopping 2024-03-06T07:27:18.836 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:18.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.948+0000 7f8bfb7bc640 1 Processor -- start 2024-03-06T07:27:18.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.949+0000 7f8bfb7bc640 1 -- start start 2024-03-06T07:27:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.949+0000 7f8bfb7bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bf4152650 0x7f8bf4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.949+0000 7f8bfb7bc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8bf40c6b10 0x7f8bf4158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.949+0000 7f8bfb7bc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf40c2fc0 con 0x7f8bf40bf9d0 2024-03-06T07:27:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.949+0000 7f8bfb7bc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf4155800 con 0x7f8bf40c6b10 2024-03-06T07:27:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.949+0000 7f8bfb7bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf40c6690 con 0x7f8bf4152650 2024-03-06T07:27:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.950+0000 7f8bf9558640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bf4152650 0x7f8bf4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.950+0000 7f8bf8d57640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8bf40c6b10 0x7f8bf4158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.950+0000 7f8bf9d59640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8bf40bf9d0 0x7f8bf40c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:38976/0 (socket says 172.21.15.1:38976) 2024-03-06T07:27:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.950+0000 7f8bf9558640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bf4152650 0x7f8bf4155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:46386/0 (socket says 172.21.15.1:46386) 2024-03-06T07:27:18.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.950+0000 7f8bf9d59640 1 -- 172.21.15.1:0/2760179403 learned_addr learned my addr 172.21.15.1:0/2760179403 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.950+0000 7f8bf9558640 1 -- 172.21.15.1:0/2760179403 >> v2:172.21.15.98:3300/0 conn(0x7f8bf40c6b10 msgr2=0x7f8bf4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.950+0000 7f8bf9558640 1 --2- 172.21.15.1:0/2760179403 >> v2:172.21.15.98:3300/0 conn(0x7f8bf40c6b10 0x7f8bf4158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.950+0000 7f8bf9558640 1 -- 172.21.15.1:0/2760179403 >> v1:172.21.15.98:6789/0 conn(0x7f8bf40bf9d0 legacy=0x7f8bf40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.950+0000 7f8bf9558640 1 -- 172.21.15.1:0/2760179403 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf40c7280 con 0x7f8bf4152650 2024-03-06T07:27:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.951+0000 7f8bf9558640 1 --2- 172.21.15.1:0/2760179403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bf4152650 0x7f8bf4155c90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8bec0097a0 tx=0x7f8bec02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c90ad1cca6a4d86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.951+0000 7f8bf3fff640 1 -- 172.21.15.1:0/2760179403 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec0315a0 con 0x7f8bf4152650 2024-03-06T07:27:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.951+0000 7f8bf3fff640 1 -- 172.21.15.1:0/2760179403 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bec037070 con 0x7f8bf4152650 2024-03-06T07:27:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.951+0000 7f8bf3fff640 1 -- 172.21.15.1:0/2760179403 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec031960 con 0x7f8bf4152650 2024-03-06T07:27:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.952+0000 7f8bfb7bc640 1 -- 172.21.15.1:0/2760179403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bf4152650 msgr2=0x7f8bf4155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.952+0000 7f8bfb7bc640 1 --2- 172.21.15.1:0/2760179403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bf4152650 0x7f8bf4155c90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8bec0097a0 tx=0x7f8bec02b840 comp rx=0 tx=0).stop 2024-03-06T07:27:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.952+0000 7f8bfb7bc640 1 -- 172.21.15.1:0/2760179403 shutdown_connections 2024-03-06T07:27:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.952+0000 7f8bfb7bc640 1 --2- 172.21.15.1:0/2760179403 >> v2:172.21.15.98:3300/0 conn(0x7f8bf40c6b10 0x7f8bf4158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.952+0000 7f8bfb7bc640 1 --2- 172.21.15.1:0/2760179403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bf4152650 0x7f8bf4155c90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.952+0000 7f8bfb7bc640 1 -- 172.21.15.1:0/2760179403 >> 172.21.15.1:0/2760179403 conn(0x7f8bf4066ee0 msgr2=0x7f8bf40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.952+0000 7f8bfb7bc640 1 -- 172.21.15.1:0/2760179403 shutdown_connections 2024-03-06T07:27:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.952+0000 7f8bfb7bc640 1 -- 172.21.15.1:0/2760179403 wait complete. 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.953+0000 7f8bfb7bc640 1 Processor -- start 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.953+0000 7f8bfb7bc640 1 -- start start 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.953+0000 7f8bfb7bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bf40bf9d0 0x7f8bf41e0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bfb7bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bf4152650 0x7f8bf4141c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bfb7bc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bf41427c0 0x7f8bf41421b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bfb7bc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf4155490 con 0x7f8bf41427c0 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bfb7bc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf41551b0 con 0x7f8bf4152650 2024-03-06T07:27:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bfb7bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf4152a30 con 0x7f8bf40bf9d0 2024-03-06T07:27:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bf9d59640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bf41427c0 0x7f8bf41421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bf9d59640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bf41427c0 0x7f8bf41421b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48328/0 (socket says 172.21.15.1:48328) 2024-03-06T07:27:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bf9d59640 1 -- 172.21.15.1:0/3931338639 learned_addr learned my addr 172.21.15.1:0/3931338639 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bf9558640 1 --2- 172.21.15.1:0/3931338639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bf40bf9d0 0x7f8bf41e0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bf8d57640 1 --2- 172.21.15.1:0/3931338639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bf4152650 0x7f8bf4141c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bf9d59640 1 -- 172.21.15.1:0/3931338639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bf40bf9d0 msgr2=0x7f8bf41e0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bf9d59640 1 --2- 172.21.15.1:0/3931338639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bf40bf9d0 0x7f8bf41e0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bf9d59640 1 -- 172.21.15.1:0/3931338639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bf4152650 msgr2=0x7f8bf4141c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bf9d59640 1 --2- 172.21.15.1:0/3931338639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bf4152650 0x7f8bf4141c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bf9d59640 1 -- 172.21.15.1:0/3931338639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf4142d80 con 0x7f8bf41427c0 2024-03-06T07:27:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bf9d59640 1 --2- 172.21.15.1:0/3931338639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bf41427c0 0x7f8bf41421b0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f8be400ca20 tx=0x7f8be400cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bf1ffb640 1 -- 172.21.15.1:0/3931338639 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be4017070 con 0x7f8bf41427c0 2024-03-06T07:27:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bf1ffb640 1 -- 172.21.15.1:0/3931338639 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be400abd0 con 0x7f8bf41427c0 2024-03-06T07:27:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bfb7bc640 1 -- 172.21.15.1:0/3931338639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf41ef310 con 0x7f8bf41427c0 2024-03-06T07:27:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.954+0000 7f8bf1ffb640 1 -- 172.21.15.1:0/3931338639 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be4012400 con 0x7f8bf41427c0 2024-03-06T07:27:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.955+0000 7f8bfb7bc640 1 -- 172.21.15.1:0/3931338639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf41543b0 con 0x7f8bf41427c0 2024-03-06T07:27:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.956+0000 7f8bfb7bc640 1 -- 172.21.15.1:0/3931338639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf4067fd0 con 0x7f8bf41427c0 2024-03-06T07:27:18.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.956+0000 7f8bf1ffb640 1 -- 172.21.15.1:0/3931338639 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8be401a070 con 0x7f8bf41427c0 2024-03-06T07:27:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.956+0000 7f8bf1ffb640 1 --2- 172.21.15.1:0/3931338639 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8be006b500 0x7f8be006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.956+0000 7f8bf1ffb640 1 -- 172.21.15.1:0/3931338639 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8be4089f20 con 0x7f8bf41427c0 2024-03-06T07:27:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.957+0000 7f8bf9558640 1 --2- 172.21.15.1:0/3931338639 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8be006b500 0x7f8be006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.957+0000 7f8bf9558640 1 --2- 172.21.15.1:0/3931338639 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8be006b500 0x7f8be006d9c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f8bec036000 tx=0x7f8bec03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:18.961+0000 7f8bf1ffb640 1 -- 172.21.15.1:0/3931338639 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8be4058650 con 0x7f8bf41427c0 2024-03-06T07:27:19.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:19.178+0000 7f8bfb7bc640 1 -- 172.21.15.1:0/3931338639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bf40b06f0 con 0x7f8bf41427c0 2024-03-06T07:27:19.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:19.179+0000 7f8bf1ffb640 1 -- 172.21.15.1:0/3931338639 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f8be405c6f0 con 0x7f8bf41427c0 2024-03-06T07:27:19.181 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:19.181 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":159,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:15.309451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9489":{"gid":9489,"name":"c","rank":0,"incarnation":155,"state":"up:stopping","state_seq":3,"addr":"172.21.15.1:6835/1751109564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1751109564},{"type":"v1","addr":"172.21.15.1:6835","nonce":1751109564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:27:19.181 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 159 2024-03-06T07:27:19.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:19.181+0000 7f8bfb7bc640 1 -- 172.21.15.1:0/3931338639 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8be006b500 msgr2=0x7f8be006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:19.181+0000 7f8bfb7bc640 1 --2- 172.21.15.1:0/3931338639 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8be006b500 0x7f8be006d9c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f8bec036000 tx=0x7f8bec03d040 comp rx=0 tx=0).stop 2024-03-06T07:27:19.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:19.181+0000 7f8bfb7bc640 1 -- 172.21.15.1:0/3931338639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bf41427c0 msgr2=0x7f8bf41421b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:19.181+0000 7f8bfb7bc640 1 --2- 172.21.15.1:0/3931338639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bf41427c0 0x7f8bf41421b0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f8be400ca20 tx=0x7f8be400cee0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:19.181+0000 7f8bfb7bc640 1 -- 172.21.15.1:0/3931338639 shutdown_connections 2024-03-06T07:27:19.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:19.181+0000 7f8bfb7bc640 1 --2- 172.21.15.1:0/3931338639 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8be006b500 0x7f8be006d9c0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:19.181+0000 7f8bfb7bc640 1 --2- 172.21.15.1:0/3931338639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bf41427c0 0x7f8bf41421b0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:19.181+0000 7f8bfb7bc640 1 --2- 172.21.15.1:0/3931338639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bf4152650 0x7f8bf4141c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:19.181+0000 7f8bfb7bc640 1 --2- 172.21.15.1:0/3931338639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8bf40bf9d0 0x7f8bf41e0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:19.181+0000 7f8bfb7bc640 1 -- 172.21.15.1:0/3931338639 >> 172.21.15.1:0/3931338639 conn(0x7f8bf4066ee0 msgr2=0x7f8bf40cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:19.181+0000 7f8bfb7bc640 1 -- 172.21.15.1:0/3931338639 shutdown_connections 2024-03-06T07:27:19.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:19.181+0000 7f8bfb7bc640 1 -- 172.21.15.1:0/3931338639 wait complete. 2024-03-06T07:27:19.596 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 159, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:04.619091+0000', 'modified': '2024-03-06T07:27:15.309451+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9489': {'gid': 9489, 'name': 'c', 'rank': 0, 'incarnation': 155, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.1:6835/1751109564', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1751109564}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1751109564}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:19.596 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9489:up:stopping 2024-03-06T07:27:20.598 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:20.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.709+0000 7f6be4281640 1 Processor -- start 2024-03-06T07:27:20.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.709+0000 7f6be4281640 1 -- start start 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.710+0000 7f6be4281640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6bdc0c2b30 0x7f6bdc0724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.710+0000 7f6be4281640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bdc0a5fa0 0x7f6bdc072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.710+0000 7f6be4281640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bdc151dd0 con 0x7f6bdc152b70 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.710+0000 7f6be4281640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bdc1526e0 con 0x7f6bdc0c2b30 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.710+0000 7f6be4281640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc156190 con 0x7f6bdc0a5fa0 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.710+0000 7f6be201d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6bdc0c2b30 0x7f6bdc0724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.710+0000 7f6be201d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6bdc0c2b30 0x7f6bdc0724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48330/0 (socket says 172.21.15.1:48330) 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.710+0000 7f6be201d640 1 -- 172.21.15.1:0/761378968 learned_addr learned my addr 172.21.15.1:0/761378968 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.710+0000 7f6be281e640 1 --2- 172.21.15.1:0/761378968 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bdc0a5fa0 0x7f6bdc072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.711+0000 7f6be201d640 1 -- 172.21.15.1:0/761378968 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bdc0a5fa0 msgr2=0x7f6bdc072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.711+0000 7f6be201d640 1 --2- 172.21.15.1:0/761378968 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bdc0a5fa0 0x7f6bdc072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.711+0000 7f6be201d640 1 -- 172.21.15.1:0/761378968 >> v1:172.21.15.98:6789/0 conn(0x7f6bdc152b70 legacy=0x7f6bdc0c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.711+0000 7f6be201d640 1 -- 172.21.15.1:0/761378968 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc152090 con 0x7f6bdc0c2b30 2024-03-06T07:27:20.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.711+0000 7f6be201d640 1 --2- 172.21.15.1:0/761378968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bdc0c2b30 0x7f6bdc0724c0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f6bd400bb10 tx=0x7f6bd402e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f0ed39e10c0895 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.711+0000 7f6be101b640 1 -- 172.21.15.1:0/761378968 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd403b070 con 0x7f6bdc0c2b30 2024-03-06T07:27:20.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.712+0000 7f6be101b640 1 -- 172.21.15.1:0/761378968 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd4036440 con 0x7f6bdc0c2b30 2024-03-06T07:27:20.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.712+0000 7f6be101b640 1 -- 172.21.15.1:0/761378968 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4036810 con 0x7f6bdc0c2b30 2024-03-06T07:27:20.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.712+0000 7f6be4281640 1 -- 172.21.15.1:0/761378968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bdc0c2b30 msgr2=0x7f6bdc0724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.712+0000 7f6be4281640 1 --2- 172.21.15.1:0/761378968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bdc0c2b30 0x7f6bdc0724c0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f6bd400bb10 tx=0x7f6bd402e150 comp rx=0 tx=0).stop 2024-03-06T07:27:20.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.712+0000 7f6be4281640 1 -- 172.21.15.1:0/761378968 shutdown_connections 2024-03-06T07:27:20.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.712+0000 7f6be4281640 1 --2- 172.21.15.1:0/761378968 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bdc0a5fa0 0x7f6bdc072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.712+0000 7f6be4281640 1 --2- 172.21.15.1:0/761378968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bdc0c2b30 0x7f6bdc0724c0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.712+0000 7f6be4281640 1 -- 172.21.15.1:0/761378968 >> 172.21.15.1:0/761378968 conn(0x7f6bdc0c6860 msgr2=0x7f6bdc0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.712+0000 7f6be4281640 1 -- 172.21.15.1:0/761378968 shutdown_connections 2024-03-06T07:27:20.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.712+0000 7f6be4281640 1 -- 172.21.15.1:0/761378968 wait complete. 2024-03-06T07:27:20.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.713+0000 7f6be4281640 1 Processor -- start 2024-03-06T07:27:20.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.713+0000 7f6be4281640 1 -- start start 2024-03-06T07:27:20.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.714+0000 7f6be4281640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bdc0a5fa0 0x7f6bdc1e7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.714+0000 7f6be4281640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bdc0c2b30 0x7f6bdc0ac6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.714+0000 7f6be4281640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bdc152b70 0x7f6bdc1ea850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.714+0000 7f6be4281640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc155f00 con 0x7f6bdc152b70 2024-03-06T07:27:20.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.714+0000 7f6be4281640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc155d80 con 0x7f6bdc0c2b30 2024-03-06T07:27:20.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.714+0000 7f6be4281640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc0c3a00 con 0x7f6bdc0a5fa0 2024-03-06T07:27:20.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.714+0000 7f6be281e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bdc152b70 0x7f6bdc1ea850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.714+0000 7f6be281e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bdc152b70 0x7f6bdc1ea850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48336/0 (socket says 172.21.15.1:48336) 2024-03-06T07:27:20.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.714+0000 7f6be281e640 1 -- 172.21.15.1:0/1824689762 learned_addr learned my addr 172.21.15.1:0/1824689762 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:20.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.714+0000 7f6be201d640 1 --2- 172.21.15.1:0/1824689762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bdc0a5fa0 0x7f6bdc1e7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.714+0000 7f6be181c640 1 --2- 172.21.15.1:0/1824689762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bdc0c2b30 0x7f6bdc0ac6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.714+0000 7f6be281e640 1 -- 172.21.15.1:0/1824689762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bdc0a5fa0 msgr2=0x7f6bdc1e7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.714+0000 7f6be281e640 1 --2- 172.21.15.1:0/1824689762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bdc0a5fa0 0x7f6bdc1e7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.714+0000 7f6be281e640 1 -- 172.21.15.1:0/1824689762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bdc0c2b30 msgr2=0x7f6bdc0ac6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.714+0000 7f6be281e640 1 --2- 172.21.15.1:0/1824689762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bdc0c2b30 0x7f6bdc0ac6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.714+0000 7f6be281e640 1 -- 172.21.15.1:0/1824689762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc1e7ab0 con 0x7f6bdc152b70 2024-03-06T07:27:20.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.714+0000 7f6be181c640 1 --2- 172.21.15.1:0/1824689762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bdc0c2b30 0x7f6bdc0ac6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:20.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.715+0000 7f6be281e640 1 --2- 172.21.15.1:0/1824689762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bdc152b70 0x7f6bdc1ea850 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800efc0 tx=0x7f6bd800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.715+0000 7f6bd2ffd640 1 -- 172.21.15.1:0/1824689762 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8018020 con 0x7f6bdc152b70 2024-03-06T07:27:20.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.715+0000 7f6bd2ffd640 1 -- 172.21.15.1:0/1824689762 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd8014480 con 0x7f6bdc152b70 2024-03-06T07:27:20.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.715+0000 7f6be4281640 1 -- 172.21.15.1:0/1824689762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bdc0acc30 con 0x7f6bdc152b70 2024-03-06T07:27:20.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.715+0000 7f6bd2ffd640 1 -- 172.21.15.1:0/1824689762 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8014870 con 0x7f6bdc152b70 2024-03-06T07:27:20.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.715+0000 7f6be4281640 1 -- 172.21.15.1:0/1824689762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bdc0bd640 con 0x7f6bdc152b70 2024-03-06T07:27:20.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.716+0000 7f6be4281640 1 -- 172.21.15.1:0/1824689762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bdc067fd0 con 0x7f6bdc152b70 2024-03-06T07:27:20.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.716+0000 7f6bd2ffd640 1 -- 172.21.15.1:0/1824689762 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6bd8005d10 con 0x7f6bdc152b70 2024-03-06T07:27:20.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.717+0000 7f6bd2ffd640 1 --2- 172.21.15.1:0/1824689762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6bc006b4d0 0x7f6bc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.717+0000 7f6bd2ffd640 1 -- 172.21.15.1:0/1824689762 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6bd808ad80 con 0x7f6bdc152b70 2024-03-06T07:27:20.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.717+0000 7f6be201d640 1 --2- 172.21.15.1:0/1824689762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6bc006b4d0 0x7f6bc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.718+0000 7f6be201d640 1 --2- 172.21.15.1:0/1824689762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6bc006b4d0 0x7f6bc006d990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4038000 tx=0x7f6bd4032000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.721+0000 7f6bd2ffd640 1 -- 172.21.15.1:0/1824689762 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6bd8022070 con 0x7f6bdc152b70 2024-03-06T07:27:20.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.940+0000 7f6be4281640 1 -- 172.21.15.1:0/1824689762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bdc155580 con 0x7f6bdc152b70 2024-03-06T07:27:20.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.941+0000 7f6bd2ffd640 1 -- 172.21.15.1:0/1824689762 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f6bdc0acc30 con 0x7f6bdc152b70 2024-03-06T07:27:20.943 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:20.943 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":159,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:15.309451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9489":{"gid":9489,"name":"c","rank":0,"incarnation":155,"state":"up:stopping","state_seq":3,"addr":"172.21.15.1:6835/1751109564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1751109564},{"type":"v1","addr":"172.21.15.1:6835","nonce":1751109564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:27:20.943 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 159 2024-03-06T07:27:20.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.943+0000 7f6be4281640 1 -- 172.21.15.1:0/1824689762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6bc006b4d0 msgr2=0x7f6bc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.943+0000 7f6be4281640 1 --2- 172.21.15.1:0/1824689762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6bc006b4d0 0x7f6bc006d990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4038000 tx=0x7f6bd4032000 comp rx=0 tx=0).stop 2024-03-06T07:27:20.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.943+0000 7f6be4281640 1 -- 172.21.15.1:0/1824689762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bdc152b70 msgr2=0x7f6bdc1ea850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.943+0000 7f6be4281640 1 --2- 172.21.15.1:0/1824689762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bdc152b70 0x7f6bdc1ea850 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800efc0 tx=0x7f6bd800c490 comp rx=0 tx=0).stop 2024-03-06T07:27:20.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.943+0000 7f6be4281640 1 -- 172.21.15.1:0/1824689762 shutdown_connections 2024-03-06T07:27:20.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.943+0000 7f6be4281640 1 --2- 172.21.15.1:0/1824689762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6bc006b4d0 0x7f6bc006d990 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.943+0000 7f6be4281640 1 --2- 172.21.15.1:0/1824689762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bdc152b70 0x7f6bdc1ea850 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.943+0000 7f6be4281640 1 --2- 172.21.15.1:0/1824689762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bdc0c2b30 0x7f6bdc0ac6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.943+0000 7f6be4281640 1 --2- 172.21.15.1:0/1824689762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6bdc0a5fa0 0x7f6bdc1e7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.943+0000 7f6be4281640 1 -- 172.21.15.1:0/1824689762 >> 172.21.15.1:0/1824689762 conn(0x7f6bdc0c6860 msgr2=0x7f6bdc151690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.943+0000 7f6be4281640 1 -- 172.21.15.1:0/1824689762 shutdown_connections 2024-03-06T07:27:20.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:20.943+0000 7f6be4281640 1 -- 172.21.15.1:0/1824689762 wait complete. 2024-03-06T07:27:21.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 159, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:04.619091+0000', 'modified': '2024-03-06T07:27:15.309451+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9489': {'gid': 9489, 'name': 'c', 'rank': 0, 'incarnation': 155, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.1:6835/1751109564', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1751109564}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1751109564}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:21.358 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9489:up:stopping 2024-03-06T07:27:22.359 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:22.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.470+0000 7f78b0df6640 1 Processor -- start 2024-03-06T07:27:22.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.471+0000 7f78b0df6640 1 -- start start 2024-03-06T07:27:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.471+0000 7f78b0df6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78ac0c6730 0x7f78ac0c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.471+0000 7f78b0df6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f78ac0c5d10 0x7f78ac156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.471+0000 7f78b0df6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78ac0c3300 con 0x7f78ac152690 2024-03-06T07:27:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.471+0000 7f78b0df6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78ac0c35a0 con 0x7f78ac0c5d10 2024-03-06T07:27:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.471+0000 7f78b0df6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac0c3880 con 0x7f78ac0c6730 2024-03-06T07:27:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.471+0000 7f78aad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f78ac0c5d10 0x7f78ac156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.471+0000 7f78aad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f78ac0c5d10 0x7f78ac156cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48344/0 (socket says 172.21.15.1:48344) 2024-03-06T07:27:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.471+0000 7f78aad9d640 1 -- 172.21.15.1:0/3809646473 learned_addr learned my addr 172.21.15.1:0/3809646473 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.472+0000 7f78a3fff640 1 --2- 172.21.15.1:0/3809646473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78ac0c6730 0x7f78ac0c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.472+0000 7f78aad9d640 1 -- 172.21.15.1:0/3809646473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78ac0c6730 msgr2=0x7f78ac0c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.472+0000 7f78aad9d640 1 --2- 172.21.15.1:0/3809646473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78ac0c6730 0x7f78ac0c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.472+0000 7f78aad9d640 1 -- 172.21.15.1:0/3809646473 >> v1:172.21.15.98:6789/0 conn(0x7f78ac152690 legacy=0x7f78ac155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.472+0000 7f78aad9d640 1 -- 172.21.15.1:0/3809646473 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac0c3b60 con 0x7f78ac0c5d10 2024-03-06T07:27:22.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.473+0000 7f78aad9d640 1 --2- 172.21.15.1:0/3809646473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78ac0c5d10 0x7f78ac156cd0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009e50 tx=0x7f78a402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=260e5ae6d8aae476 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.473+0000 7f78a9d9b640 1 -- 172.21.15.1:0/3809646473 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a402ca70 con 0x7f78ac0c5d10 2024-03-06T07:27:22.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.473+0000 7f78a9d9b640 1 -- 172.21.15.1:0/3809646473 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a402cc10 con 0x7f78ac0c5d10 2024-03-06T07:27:22.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.473+0000 7f78a9d9b640 1 -- 172.21.15.1:0/3809646473 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a40343f0 con 0x7f78ac0c5d10 2024-03-06T07:27:22.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.473+0000 7f78b0df6640 1 -- 172.21.15.1:0/3809646473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78ac0c5d10 msgr2=0x7f78ac156cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.473+0000 7f78b0df6640 1 --2- 172.21.15.1:0/3809646473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78ac0c5d10 0x7f78ac156cd0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009e50 tx=0x7f78a402bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.473+0000 7f78b0df6640 1 -- 172.21.15.1:0/3809646473 shutdown_connections 2024-03-06T07:27:22.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.473+0000 7f78b0df6640 1 --2- 172.21.15.1:0/3809646473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78ac0c5d10 0x7f78ac156cd0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.473+0000 7f78b0df6640 1 --2- 172.21.15.1:0/3809646473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78ac0c6730 0x7f78ac0c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.473+0000 7f78b0df6640 1 -- 172.21.15.1:0/3809646473 >> 172.21.15.1:0/3809646473 conn(0x7f78ac06b410 msgr2=0x7f78ac06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.473+0000 7f78b0df6640 1 -- 172.21.15.1:0/3809646473 shutdown_connections 2024-03-06T07:27:22.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.473+0000 7f78b0df6640 1 -- 172.21.15.1:0/3809646473 wait complete. 2024-03-06T07:27:22.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.475+0000 7f78b0df6640 1 Processor -- start 2024-03-06T07:27:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.475+0000 7f78b0df6640 1 -- start start 2024-03-06T07:27:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.475+0000 7f78b0df6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78ac0c5d10 0x7f78ac14bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.475+0000 7f78b0df6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78ac0c6730 0x7f78ac14ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.475+0000 7f78b0df6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac152690 0x7f78ac14b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.475+0000 7f78b0df6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac152bf0 con 0x7f78ac0c6730 2024-03-06T07:27:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.475+0000 7f78b0df6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac152a70 con 0x7f78ac152690 2024-03-06T07:27:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.475+0000 7f78b0df6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac157210 con 0x7f78ac0c5d10 2024-03-06T07:27:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.476+0000 7f78a3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78ac0c6730 0x7f78ac14ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.476+0000 7f78a3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78ac0c6730 0x7f78ac14ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48360/0 (socket says 172.21.15.1:48360) 2024-03-06T07:27:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.476+0000 7f78a3fff640 1 -- 172.21.15.1:0/791730866 learned_addr learned my addr 172.21.15.1:0/791730866 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.476+0000 7f78aa59c640 1 --2- 172.21.15.1:0/791730866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78ac0c5d10 0x7f78ac14bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.476+0000 7f78aad9d640 1 --2- 172.21.15.1:0/791730866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac152690 0x7f78ac14b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.476+0000 7f78a3fff640 1 -- 172.21.15.1:0/791730866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78ac0c5d10 msgr2=0x7f78ac14bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.476+0000 7f78a3fff640 1 --2- 172.21.15.1:0/791730866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78ac0c5d10 0x7f78ac14bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.476+0000 7f78a3fff640 1 -- 172.21.15.1:0/791730866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac152690 msgr2=0x7f78ac14b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.476+0000 7f78a3fff640 1 --2- 172.21.15.1:0/791730866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac152690 0x7f78ac14b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.476+0000 7f78a3fff640 1 -- 172.21.15.1:0/791730866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac0c7b30 con 0x7f78ac0c6730 2024-03-06T07:27:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.476+0000 7f78a3fff640 1 --2- 172.21.15.1:0/791730866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78ac0c6730 0x7f78ac14ad00 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f789400d800 tx=0x7f789400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.476+0000 7f78a2ffd640 1 -- 172.21.15.1:0/791730866 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789400b6c0 con 0x7f78ac0c6730 2024-03-06T07:27:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.476+0000 7f78a2ffd640 1 -- 172.21.15.1:0/791730866 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7894019070 con 0x7f78ac0c6730 2024-03-06T07:27:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.476+0000 7f78b0df6640 1 -- 172.21.15.1:0/791730866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78ac14c830 con 0x7f78ac0c6730 2024-03-06T07:27:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.476+0000 7f78a2ffd640 1 -- 172.21.15.1:0/791730866 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789400ba80 con 0x7f78ac0c6730 2024-03-06T07:27:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.476+0000 7f78b0df6640 1 -- 172.21.15.1:0/791730866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78ac14b780 con 0x7f78ac0c6730 2024-03-06T07:27:22.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.477+0000 7f78b0df6640 1 -- 172.21.15.1:0/791730866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78ac067fd0 con 0x7f78ac0c6730 2024-03-06T07:27:22.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.478+0000 7f78a2ffd640 1 -- 172.21.15.1:0/791730866 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f789400bc20 con 0x7f78ac0c6730 2024-03-06T07:27:22.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.478+0000 7f78a2ffd640 1 --2- 172.21.15.1:0/791730866 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f788c06b4d0 0x7f788c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.478+0000 7f78a2ffd640 1 -- 172.21.15.1:0/791730866 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f789408a700 con 0x7f78ac0c6730 2024-03-06T07:27:22.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.479+0000 7f78aa59c640 1 --2- 172.21.15.1:0/791730866 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f788c06b4d0 0x7f788c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.479+0000 7f78aa59c640 1 --2- 172.21.15.1:0/791730866 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f788c06b4d0 0x7f788c06d990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0710e0 tx=0x7f789c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.483+0000 7f78a2ffd640 1 -- 172.21.15.1:0/791730866 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7894058d90 con 0x7f78ac0c6730 2024-03-06T07:27:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.700+0000 7f78b0df6640 1 -- 172.21.15.1:0/791730866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78ac0c2c40 con 0x7f78ac0c6730 2024-03-06T07:27:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.701+0000 7f78a2ffd640 1 -- 172.21.15.1:0/791730866 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f7894088e20 con 0x7f78ac0c6730 2024-03-06T07:27:22.703 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:22.703 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":159,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:15.309451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9489":{"gid":9489,"name":"c","rank":0,"incarnation":155,"state":"up:stopping","state_seq":3,"addr":"172.21.15.1:6835/1751109564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1751109564},{"type":"v1","addr":"172.21.15.1:6835","nonce":1751109564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:27:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 159 2024-03-06T07:27:22.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.703+0000 7f78b0df6640 1 -- 172.21.15.1:0/791730866 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f788c06b4d0 msgr2=0x7f788c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.703+0000 7f78b0df6640 1 --2- 172.21.15.1:0/791730866 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f788c06b4d0 0x7f788c06d990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0710e0 tx=0x7f789c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:22.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.703+0000 7f78b0df6640 1 -- 172.21.15.1:0/791730866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78ac0c6730 msgr2=0x7f78ac14ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.703+0000 7f78b0df6640 1 --2- 172.21.15.1:0/791730866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78ac0c6730 0x7f78ac14ad00 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f789400d800 tx=0x7f789400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.703+0000 7f78b0df6640 1 -- 172.21.15.1:0/791730866 shutdown_connections 2024-03-06T07:27:22.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.703+0000 7f78b0df6640 1 --2- 172.21.15.1:0/791730866 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f788c06b4d0 0x7f788c06d990 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.703+0000 7f78b0df6640 1 --2- 172.21.15.1:0/791730866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78ac152690 0x7f78ac14b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.703+0000 7f78b0df6640 1 --2- 172.21.15.1:0/791730866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78ac0c6730 0x7f78ac14ad00 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.703+0000 7f78b0df6640 1 --2- 172.21.15.1:0/791730866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78ac0c5d10 0x7f78ac14bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.703+0000 7f78b0df6640 1 -- 172.21.15.1:0/791730866 >> 172.21.15.1:0/791730866 conn(0x7f78ac06b410 msgr2=0x7f78ac0a9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.703+0000 7f78b0df6640 1 -- 172.21.15.1:0/791730866 shutdown_connections 2024-03-06T07:27:22.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:22.703+0000 7f78b0df6640 1 -- 172.21.15.1:0/791730866 wait complete. 2024-03-06T07:27:23.118 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 159, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:04.619091+0000', 'modified': '2024-03-06T07:27:15.309451+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9489': {'gid': 9489, 'name': 'c', 'rank': 0, 'incarnation': 155, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.1:6835/1751109564', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1751109564}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1751109564}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:23.118 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9489:up:stopping 2024-03-06T07:27:24.120 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:24.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.232+0000 7fe292c8f640 1 Processor -- start 2024-03-06T07:27:24.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.232+0000 7fe292c8f640 1 -- start start 2024-03-06T07:27:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.233+0000 7fe292c8f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe28c0733d0 0x7fe28c0c2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.233+0000 7fe292c8f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28c06fb80 con 0x7fe28c14f7d0 2024-03-06T07:27:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.233+0000 7fe292c8f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28c06fe60 con 0x7fe28c072830 2024-03-06T07:27:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.233+0000 7fe292c8f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe28c070140 con 0x7fe28c0733d0 2024-03-06T07:27:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.233+0000 7fe28bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe28c14f7d0 0x7fe28c0c2630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42982/0 (socket says 172.21.15.1:42982) 2024-03-06T07:27:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.233+0000 7fe28bfff640 1 -- 172.21.15.1:0/3458649100 learned_addr learned my addr 172.21.15.1:0/3458649100 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.233+0000 7fe29122c640 1 --2- 172.21.15.1:0/3458649100 >> v2:172.21.15.98:3300/0 conn(0x7fe28c0733d0 0x7fe28c0c2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.233+0000 7fe28b7fe640 1 -- 172.21.15.1:0/3458649100 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323792771 0 0) 0x7fe28c06fb80 con 0x7fe28c14f7d0 2024-03-06T07:27:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.233+0000 7fe28b7fe640 1 -- 172.21.15.1:0/3458649100 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe274003520 con 0x7fe28c14f7d0 2024-03-06T07:27:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe28b7fe640 1 -- 172.21.15.1:0/3458649100 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633727430 0 0) 0x7fe28c06fe60 con 0x7fe28c072830 2024-03-06T07:27:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe28b7fe640 1 -- 172.21.15.1:0/3458649100 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe28c06fb80 con 0x7fe28c072830 2024-03-06T07:27:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe28b7fe640 1 -- 172.21.15.1:0/3458649100 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4195916660 0 0) 0x7fe274003520 con 0x7fe28c14f7d0 2024-03-06T07:27:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe28b7fe640 1 -- 172.21.15.1:0/3458649100 >> v2:172.21.15.98:3300/0 conn(0x7fe28c0733d0 msgr2=0x7fe28c0c2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe28b7fe640 1 --2- 172.21.15.1:0/3458649100 >> v2:172.21.15.98:3300/0 conn(0x7fe28c0733d0 0x7fe28c0c2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe28b7fe640 1 -- 172.21.15.1:0/3458649100 >> v1:172.21.15.98:6789/0 conn(0x7fe28c072830 legacy=0x7fe28c14f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe28b7fe640 1 -- 172.21.15.1:0/3458649100 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c070420 con 0x7fe28c14f7d0 2024-03-06T07:27:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe28b7fe640 1 -- 172.21.15.1:0/3458649100 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fe280001670 con 0x7fe28c14f7d0 2024-03-06T07:27:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe28b7fe640 1 -- 172.21.15.1:0/3458649100 >> v1:172.21.15.1:6789/0 conn(0x7fe28c14f7d0 legacy=0x7fe28c0c2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe28b7fe640 1 --2- 172.21.15.1:0/3458649100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe274003b90 0x7fe274003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe28b7fe640 1 --2- 172.21.15.1:0/3458649100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe274004800 0x7fe274004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe28b7fe640 1 --2- 172.21.15.1:0/3458649100 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe2740051c0 0x7fe27400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe28b7fe640 1 -- 172.21.15.1:0/3458649100 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c070140 con 0x7fe274003b90 2024-03-06T07:27:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe28b7fe640 1 -- 172.21.15.1:0/3458649100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27400e1f0 con 0x7fe274004800 2024-03-06T07:27:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe28b7fe640 1 -- 172.21.15.1:0/3458649100 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe27400e3f0 con 0x7fe2740051c0 2024-03-06T07:27:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe290a2b640 1 --2- 172.21.15.1:0/3458649100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe274003b90 0x7fe274003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe28bfff640 1 --2- 172.21.15.1:0/3458649100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe274004800 0x7fe274004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe29122c640 1 --2- 172.21.15.1:0/3458649100 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe2740051c0 0x7fe27400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe290a2b640 1 -- 172.21.15.1:0/3458649100 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe2740051c0 msgr2=0x7fe27400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe290a2b640 1 --2- 172.21.15.1:0/3458649100 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe2740051c0 0x7fe27400dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe290a2b640 1 -- 172.21.15.1:0/3458649100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe274004800 msgr2=0x7fe274004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe290a2b640 1 --2- 172.21.15.1:0/3458649100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe274004800 0x7fe274004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe290a2b640 1 -- 172.21.15.1:0/3458649100 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe274003520 con 0x7fe274003b90 2024-03-06T07:27:24.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe29122c640 1 --2- 172.21.15.1:0/3458649100 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe2740051c0 0x7fe27400dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:24.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.234+0000 7fe29122c640 1 -- 172.21.15.1:0/3458649100 reap_dead start 2024-03-06T07:27:24.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.235+0000 7fe290a2b640 1 --2- 172.21.15.1:0/3458649100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe274003b90 0x7fe274003f90 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fe284001fd0 tx=0x7fe274005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=942d7ad39b021f2c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.235+0000 7fe28b7fe640 1 -- 172.21.15.1:0/3458649100 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe284001bf0 con 0x7fe274003b90 2024-03-06T07:27:24.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.235+0000 7fe28b7fe640 1 -- 172.21.15.1:0/3458649100 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe284003b70 con 0x7fe274003b90 2024-03-06T07:27:24.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.235+0000 7fe28b7fe640 1 -- 172.21.15.1:0/3458649100 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2840050d0 con 0x7fe274003b90 2024-03-06T07:27:24.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.236+0000 7fe292c8f640 1 -- 172.21.15.1:0/3458649100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe274003b90 msgr2=0x7fe274003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.236+0000 7fe292c8f640 1 --2- 172.21.15.1:0/3458649100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe274003b90 0x7fe274003f90 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fe284001fd0 tx=0x7fe274005830 comp rx=0 tx=0).stop 2024-03-06T07:27:24.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.236+0000 7fe292c8f640 1 -- 172.21.15.1:0/3458649100 shutdown_connections 2024-03-06T07:27:24.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.236+0000 7fe292c8f640 1 --2- 172.21.15.1:0/3458649100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe274003b90 0x7fe274003f90 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.236+0000 7fe292c8f640 1 -- 172.21.15.1:0/3458649100 >> 172.21.15.1:0/3458649100 conn(0x7fe28c0a9700 msgr2=0x7fe28c0a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.236+0000 7fe292c8f640 1 -- 172.21.15.1:0/3458649100 shutdown_connections 2024-03-06T07:27:24.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.236+0000 7fe292c8f640 1 -- 172.21.15.1:0/3458649100 wait complete. 2024-03-06T07:27:24.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.237+0000 7fe292c8f640 1 Processor -- start 2024-03-06T07:27:24.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.237+0000 7fe292c8f640 1 -- start start 2024-03-06T07:27:24.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe292c8f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe274003b90 0x7fe28c154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe292c8f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe28c14e930 0x7fe28c1ea210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe292c8f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe28c155d50 0x7fe28c1486c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe292c8f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c072c10 con 0x7fe274003b90 2024-03-06T07:27:24.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe292c8f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c06f940 con 0x7fe28c14e930 2024-03-06T07:27:24.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe292c8f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c072d90 con 0x7fe28c155d50 2024-03-06T07:27:24.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe290a2b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe274003b90 0x7fe28c154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe290a2b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe274003b90 0x7fe28c154a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48382/0 (socket says 172.21.15.1:48382) 2024-03-06T07:27:24.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe290a2b640 1 -- 172.21.15.1:0/1069012709 learned_addr learned my addr 172.21.15.1:0/1069012709 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:24.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe29122c640 1 --2- 172.21.15.1:0/1069012709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe28c155d50 0x7fe28c1486c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe28bfff640 1 --2- 172.21.15.1:0/1069012709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe28c14e930 0x7fe28c1ea210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe290a2b640 1 -- 172.21.15.1:0/1069012709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe28c155d50 msgr2=0x7fe28c1486c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe290a2b640 1 --2- 172.21.15.1:0/1069012709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe28c155d50 0x7fe28c1486c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe290a2b640 1 -- 172.21.15.1:0/1069012709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe28c14e930 msgr2=0x7fe28c1ea210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe290a2b640 1 --2- 172.21.15.1:0/1069012709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe28c14e930 0x7fe28c1ea210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe290a2b640 1 -- 172.21.15.1:0/1069012709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c154f60 con 0x7fe274003b90 2024-03-06T07:27:24.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe28bfff640 1 --2- 172.21.15.1:0/1069012709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe28c14e930 0x7fe28c1ea210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:24.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe290a2b640 1 --2- 172.21.15.1:0/1069012709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe274003b90 0x7fe28c154a20 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7fe284001fd0 tx=0x7fe284001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.238+0000 7fe2897fa640 1 -- 172.21.15.1:0/1069012709 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe284039040 con 0x7fe274003b90 2024-03-06T07:27:24.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.239+0000 7fe2897fa640 1 -- 172.21.15.1:0/1069012709 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe284005500 con 0x7fe274003b90 2024-03-06T07:27:24.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.239+0000 7fe292c8f640 1 -- 172.21.15.1:0/1069012709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28c1db510 con 0x7fe274003b90 2024-03-06T07:27:24.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.239+0000 7fe2897fa640 1 -- 172.21.15.1:0/1069012709 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe284001bf0 con 0x7fe274003b90 2024-03-06T07:27:24.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.239+0000 7fe292c8f640 1 -- 172.21.15.1:0/1069012709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28c154220 con 0x7fe274003b90 2024-03-06T07:27:24.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.240+0000 7fe292c8f640 1 -- 172.21.15.1:0/1069012709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28c068870 con 0x7fe274003b90 2024-03-06T07:27:24.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.240+0000 7fe2897fa640 1 -- 172.21.15.1:0/1069012709 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe28402c530 con 0x7fe274003b90 2024-03-06T07:27:24.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.241+0000 7fe2897fa640 1 --2- 172.21.15.1:0/1069012709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe27806b500 0x7fe27806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.241+0000 7fe2897fa640 1 -- 172.21.15.1:0/1069012709 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe2840ac0f0 con 0x7fe274003b90 2024-03-06T07:27:24.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.241+0000 7fe28bfff640 1 --2- 172.21.15.1:0/1069012709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe27806b500 0x7fe27806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.241+0000 7fe28bfff640 1 --2- 172.21.15.1:0/1069012709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe27806b500 0x7fe27806d9c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fe28c1de370 tx=0x7fe280009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.245+0000 7fe2897fa640 1 -- 172.21.15.1:0/1069012709 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe28c068870 con 0x7fe274003b90 2024-03-06T07:27:24.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.464+0000 7fe292c8f640 1 -- 172.21.15.1:0/1069012709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe28c067fd0 con 0x7fe274003b90 2024-03-06T07:27:24.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.464+0000 7fe2897fa640 1 -- 172.21.15.1:0/1069012709 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+4964 (secure 0 0 0) 0x7fe28407aef0 con 0x7fe274003b90 2024-03-06T07:27:24.466 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:24.466 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":159,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:15.309451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9489":{"gid":9489,"name":"c","rank":0,"incarnation":155,"state":"up:stopping","state_seq":3,"addr":"172.21.15.1:6835/1751109564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1751109564},{"type":"v1","addr":"172.21.15.1:6835","nonce":1751109564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:27:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 159 2024-03-06T07:27:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.466+0000 7fe292c8f640 1 -- 172.21.15.1:0/1069012709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe27806b500 msgr2=0x7fe27806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.466+0000 7fe292c8f640 1 --2- 172.21.15.1:0/1069012709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe27806b500 0x7fe27806d9c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fe28c1de370 tx=0x7fe280009040 comp rx=0 tx=0).stop 2024-03-06T07:27:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.466+0000 7fe292c8f640 1 -- 172.21.15.1:0/1069012709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe274003b90 msgr2=0x7fe28c154a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.466+0000 7fe292c8f640 1 --2- 172.21.15.1:0/1069012709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe274003b90 0x7fe28c154a20 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7fe284001fd0 tx=0x7fe284001650 comp rx=0 tx=0).stop 2024-03-06T07:27:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.467+0000 7fe292c8f640 1 -- 172.21.15.1:0/1069012709 shutdown_connections 2024-03-06T07:27:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.467+0000 7fe292c8f640 1 --2- 172.21.15.1:0/1069012709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe27806b500 0x7fe27806d9c0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.467+0000 7fe292c8f640 1 --2- 172.21.15.1:0/1069012709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe28c155d50 0x7fe28c1486c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.467+0000 7fe292c8f640 1 --2- 172.21.15.1:0/1069012709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe28c14e930 0x7fe28c1ea210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.467+0000 7fe292c8f640 1 --2- 172.21.15.1:0/1069012709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe274003b90 0x7fe28c154a20 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.467+0000 7fe292c8f640 1 -- 172.21.15.1:0/1069012709 >> 172.21.15.1:0/1069012709 conn(0x7fe28c0a9700 msgr2=0x7fe28c0a7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.467+0000 7fe292c8f640 1 -- 172.21.15.1:0/1069012709 shutdown_connections 2024-03-06T07:27:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:24.467+0000 7fe292c8f640 1 -- 172.21.15.1:0/1069012709 wait complete. 2024-03-06T07:27:24.882 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 159, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:04.619091+0000', 'modified': '2024-03-06T07:27:15.309451+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9489': {'gid': 9489, 'name': 'c', 'rank': 0, 'incarnation': 155, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.1:6835/1751109564', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1751109564}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1751109564}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:24.882 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9489:up:stopping 2024-03-06T07:27:25.883 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:25.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.995+0000 7f43f7864640 1 Processor -- start 2024-03-06T07:27:25.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.995+0000 7f43f7864640 1 -- start start 2024-03-06T07:27:25.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.996+0000 7f43f7864640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f43f0152920 0x7f43f006aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.996+0000 7f43f7864640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43f006b530 0x7f43f00c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.996+0000 7f43f7864640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f0158910 con 0x7f43f00c5d90 2024-03-06T07:27:25.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.996+0000 7f43f7864640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f0152d00 con 0x7f43f0152920 2024-03-06T07:27:25.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.996+0000 7f43f7864640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f01596d0 con 0x7f43f006b530 2024-03-06T07:27:25.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.996+0000 7f43f5600640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f43f0152920 0x7f43f006aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.996+0000 7f43f5600640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f43f0152920 0x7f43f006aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59306/0 (socket says 172.21.15.1:59306) 2024-03-06T07:27:25.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.996+0000 7f43f5600640 1 -- 172.21.15.1:0/2665907155 learned_addr learned my addr 172.21.15.1:0/2665907155 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:25.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.996+0000 7f43f4dff640 1 --2- 172.21.15.1:0/2665907155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43f006b530 0x7f43f00c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.997+0000 7f43f5600640 1 -- 172.21.15.1:0/2665907155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43f006b530 msgr2=0x7f43f00c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.997+0000 7f43f5600640 1 --2- 172.21.15.1:0/2665907155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43f006b530 0x7f43f00c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.997+0000 7f43f5600640 1 -- 172.21.15.1:0/2665907155 >> v1:172.21.15.1:6789/0 conn(0x7f43f00c5d90 legacy=0x7f43f0155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.997+0000 7f43f5600640 1 -- 172.21.15.1:0/2665907155 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f0158bf0 con 0x7f43f0152920 2024-03-06T07:27:26.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.997+0000 7f43f5600640 1 --2- 172.21.15.1:0/2665907155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43f0152920 0x7f43f006aff0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f43ec0098a0 tx=0x7f43ec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ffb97a83d55d523 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.998+0000 7f43e7fff640 1 -- 172.21.15.1:0/2665907155 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ec035020 con 0x7f43f0152920 2024-03-06T07:27:26.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.998+0000 7f43e7fff640 1 -- 172.21.15.1:0/2665907155 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43ec031640 con 0x7f43f0152920 2024-03-06T07:27:26.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.998+0000 7f43e7fff640 1 -- 172.21.15.1:0/2665907155 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ec02d040 con 0x7f43f0152920 2024-03-06T07:27:26.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.998+0000 7f43f7864640 1 -- 172.21.15.1:0/2665907155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43f0152920 msgr2=0x7f43f006aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.998+0000 7f43f7864640 1 --2- 172.21.15.1:0/2665907155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43f0152920 0x7f43f006aff0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f43ec0098a0 tx=0x7f43ec02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.998+0000 7f43f7864640 1 -- 172.21.15.1:0/2665907155 shutdown_connections 2024-03-06T07:27:26.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.998+0000 7f43f7864640 1 --2- 172.21.15.1:0/2665907155 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43f006b530 0x7f43f00c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.998+0000 7f43f7864640 1 --2- 172.21.15.1:0/2665907155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43f0152920 0x7f43f006aff0 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.998+0000 7f43f7864640 1 -- 172.21.15.1:0/2665907155 >> 172.21.15.1:0/2665907155 conn(0x7f43f00a9700 msgr2=0x7f43f00a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.998+0000 7f43f7864640 1 -- 172.21.15.1:0/2665907155 shutdown_connections 2024-03-06T07:27:26.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.998+0000 7f43f7864640 1 -- 172.21.15.1:0/2665907155 wait complete. 2024-03-06T07:27:26.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:25.999+0000 7f43f7864640 1 Processor -- start 2024-03-06T07:27:26.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.000+0000 7f43f7864640 1 -- start start 2024-03-06T07:27:26.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.000+0000 7f43f7864640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43f006b530 0x7f43f014a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.000+0000 7f43f7864640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f00c5d90 0x7f43f0147b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.000+0000 7f43f7864640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43f0152920 0x7f43f01480b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.000+0000 7f43f7864640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f00c6380 con 0x7f43f0152920 2024-03-06T07:27:26.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.000+0000 7f43f7864640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f0159240 con 0x7f43f00c5d90 2024-03-06T07:27:26.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.000+0000 7f43f7864640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f0155930 con 0x7f43f006b530 2024-03-06T07:27:26.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.000+0000 7f43f5e01640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43f0152920 0x7f43f01480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.000+0000 7f43f5e01640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43f0152920 0x7f43f01480b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59318/0 (socket says 172.21.15.1:59318) 2024-03-06T07:27:26.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.000+0000 7f43f5e01640 1 -- 172.21.15.1:0/991578154 learned_addr learned my addr 172.21.15.1:0/991578154 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:26.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.001+0000 7f43f5600640 1 --2- 172.21.15.1:0/991578154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43f006b530 0x7f43f014a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.001+0000 7f43f5e01640 1 -- 172.21.15.1:0/991578154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43f006b530 msgr2=0x7f43f014a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.001+0000 7f43f5e01640 1 --2- 172.21.15.1:0/991578154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43f006b530 0x7f43f014a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.001+0000 7f43f5e01640 1 -- 172.21.15.1:0/991578154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f00c5d90 msgr2=0x7f43f0147b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.001+0000 7f43f5e01640 1 --2- 172.21.15.1:0/991578154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f00c5d90 0x7f43f0147b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.001+0000 7f43f5e01640 1 -- 172.21.15.1:0/991578154 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f0154370 con 0x7f43f0152920 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.001+0000 7f43f5600640 1 --2- 172.21.15.1:0/991578154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43f006b530 0x7f43f014a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.001+0000 7f43f5e01640 1 --2- 172.21.15.1:0/991578154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43f0152920 0x7f43f01480b0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f43e800c7d0 tx=0x7f43e800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.001+0000 7f43e5ffb640 1 -- 172.21.15.1:0/991578154 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e8003b70 con 0x7f43f0152920 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.001+0000 7f43e5ffb640 1 -- 172.21.15.1:0/991578154 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e8003d30 con 0x7f43f0152920 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.001+0000 7f43e5ffb640 1 -- 172.21.15.1:0/991578154 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e800aa90 con 0x7f43f0152920 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.001+0000 7f43f7864640 1 -- 172.21.15.1:0/991578154 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f0149800 con 0x7f43f0152920 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.001+0000 7f43f7864640 1 -- 172.21.15.1:0/991578154 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f014a620 con 0x7f43f0152920 2024-03-06T07:27:26.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.002+0000 7f43f7864640 1 -- 172.21.15.1:0/991578154 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f0067fd0 con 0x7f43f0152920 2024-03-06T07:27:26.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.003+0000 7f43e5ffb640 1 -- 172.21.15.1:0/991578154 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43e800ac30 con 0x7f43f0152920 2024-03-06T07:27:26.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.003+0000 7f43e5ffb640 1 --2- 172.21.15.1:0/991578154 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43d406b4d0 0x7f43d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.003+0000 7f43e5ffb640 1 -- 172.21.15.1:0/991578154 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f43e808a240 con 0x7f43f0152920 2024-03-06T07:27:26.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.003+0000 7f43f5600640 1 --2- 172.21.15.1:0/991578154 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43d406b4d0 0x7f43d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.004+0000 7f43f5600640 1 --2- 172.21.15.1:0/991578154 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43d406b4d0 0x7f43d406d990 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f43ec036000 tx=0x7f43ec008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.008+0000 7f43e5ffb640 1 -- 172.21.15.1:0/991578154 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43e8058fc0 con 0x7f43f0152920 2024-03-06T07:27:26.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.226+0000 7f43f7864640 1 -- 172.21.15.1:0/991578154 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43f006bdf0 con 0x7f43f0152920 2024-03-06T07:27:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.227+0000 7f43e5ffb640 1 -- 172.21.15.1:0/991578154 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f43e80588c0 con 0x7f43f0152920 2024-03-06T07:27:26.229 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:26.229 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":159,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:15.309451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9489":{"gid":9489,"name":"c","rank":0,"incarnation":155,"state":"up:stopping","state_seq":3,"addr":"172.21.15.1:6835/1751109564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1751109564},{"type":"v1","addr":"172.21.15.1:6835","nonce":1751109564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:27:26.230 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 159 2024-03-06T07:27:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.229+0000 7f43f7864640 1 -- 172.21.15.1:0/991578154 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43d406b4d0 msgr2=0x7f43d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.229+0000 7f43f7864640 1 --2- 172.21.15.1:0/991578154 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43d406b4d0 0x7f43d406d990 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f43ec036000 tx=0x7f43ec008030 comp rx=0 tx=0).stop 2024-03-06T07:27:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.229+0000 7f43f7864640 1 -- 172.21.15.1:0/991578154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43f0152920 msgr2=0x7f43f01480b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.229+0000 7f43f7864640 1 --2- 172.21.15.1:0/991578154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43f0152920 0x7f43f01480b0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f43e800c7d0 tx=0x7f43e800cc90 comp rx=0 tx=0).stop 2024-03-06T07:27:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.229+0000 7f43f7864640 1 -- 172.21.15.1:0/991578154 shutdown_connections 2024-03-06T07:27:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.229+0000 7f43f7864640 1 --2- 172.21.15.1:0/991578154 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43d406b4d0 0x7f43d406d990 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.229+0000 7f43f7864640 1 --2- 172.21.15.1:0/991578154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43f0152920 0x7f43f01480b0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.229+0000 7f43f7864640 1 --2- 172.21.15.1:0/991578154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f00c5d90 0x7f43f0147b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.229+0000 7f43f7864640 1 --2- 172.21.15.1:0/991578154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43f006b530 0x7f43f014a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.229+0000 7f43f7864640 1 -- 172.21.15.1:0/991578154 >> 172.21.15.1:0/991578154 conn(0x7f43f00a9700 msgr2=0x7f43f01503f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.229+0000 7f43f7864640 1 -- 172.21.15.1:0/991578154 shutdown_connections 2024-03-06T07:27:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:26.229+0000 7f43f7864640 1 -- 172.21.15.1:0/991578154 wait complete. 2024-03-06T07:27:26.645 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 159, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:04.619091+0000', 'modified': '2024-03-06T07:27:15.309451+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9489': {'gid': 9489, 'name': 'c', 'rank': 0, 'incarnation': 155, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.1:6835/1751109564', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1751109564}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1751109564}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:26.645 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9489:up:stopping 2024-03-06T07:27:27.646 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:27.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.758+0000 7f331f9e2640 1 Processor -- start 2024-03-06T07:27:27.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.759+0000 7f331f9e2640 1 -- start start 2024-03-06T07:27:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.759+0000 7f331f9e2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3318152650 0x7f3318155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.759+0000 7f331f9e2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f33180bf9d0 0x7f33180c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.759+0000 7f331f9e2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33180c2fc0 con 0x7f33180c6b10 2024-03-06T07:27:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.759+0000 7f331f9e2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3318155800 con 0x7f3318152650 2024-03-06T07:27:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.759+0000 7f331f9e2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3318158da0 con 0x7f33180bf9d0 2024-03-06T07:27:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.759+0000 7f331d77e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3318152650 0x7f3318155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.759+0000 7f331d77e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3318152650 0x7f3318155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59326/0 (socket says 172.21.15.1:59326) 2024-03-06T07:27:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.759+0000 7f331d77e640 1 -- 172.21.15.1:0/698926410 learned_addr learned my addr 172.21.15.1:0/698926410 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.760+0000 7f331df7f640 1 --2- 172.21.15.1:0/698926410 >> v2:172.21.15.98:3300/0 conn(0x7f33180bf9d0 0x7f33180c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.760+0000 7f331d77e640 1 -- 172.21.15.1:0/698926410 >> v2:172.21.15.98:3300/0 conn(0x7f33180bf9d0 msgr2=0x7f33180c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.760+0000 7f331d77e640 1 --2- 172.21.15.1:0/698926410 >> v2:172.21.15.98:3300/0 conn(0x7f33180bf9d0 0x7f33180c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.760+0000 7f331d77e640 1 -- 172.21.15.1:0/698926410 >> v1:172.21.15.98:6789/0 conn(0x7f33180c6b10 legacy=0x7f3318158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.760+0000 7f331d77e640 1 -- 172.21.15.1:0/698926410 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33180c6690 con 0x7f3318152650 2024-03-06T07:27:27.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.760+0000 7f331d77e640 1 --2- 172.21.15.1:0/698926410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3318152650 0x7f3318155c90 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f33181597b0 tx=0x7f331002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4b8babe0512c7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.761+0000 7f3317fff640 1 -- 172.21.15.1:0/698926410 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3310035020 con 0x7f3318152650 2024-03-06T07:27:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.761+0000 7f3317fff640 1 -- 172.21.15.1:0/698926410 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3310031870 con 0x7f3318152650 2024-03-06T07:27:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.761+0000 7f3317fff640 1 -- 172.21.15.1:0/698926410 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3310031c40 con 0x7f3318152650 2024-03-06T07:27:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.761+0000 7f331f9e2640 1 -- 172.21.15.1:0/698926410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3318152650 msgr2=0x7f3318155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.761+0000 7f331f9e2640 1 --2- 172.21.15.1:0/698926410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3318152650 0x7f3318155c90 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f33181597b0 tx=0x7f331002b910 comp rx=0 tx=0).stop 2024-03-06T07:27:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.761+0000 7f331f9e2640 1 -- 172.21.15.1:0/698926410 shutdown_connections 2024-03-06T07:27:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.761+0000 7f331f9e2640 1 --2- 172.21.15.1:0/698926410 >> v2:172.21.15.98:3300/0 conn(0x7f33180bf9d0 0x7f33180c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.761+0000 7f331f9e2640 1 --2- 172.21.15.1:0/698926410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3318152650 0x7f3318155c90 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.761+0000 7f331f9e2640 1 -- 172.21.15.1:0/698926410 >> 172.21.15.1:0/698926410 conn(0x7f3318066ee0 msgr2=0x7f33180672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.761+0000 7f331f9e2640 1 -- 172.21.15.1:0/698926410 shutdown_connections 2024-03-06T07:27:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.761+0000 7f331f9e2640 1 -- 172.21.15.1:0/698926410 wait complete. 2024-03-06T07:27:27.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.762+0000 7f331f9e2640 1 Processor -- start 2024-03-06T07:27:27.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.762+0000 7f331f9e2640 1 -- start start 2024-03-06T07:27:27.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.763+0000 7f331f9e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33180bf9d0 0x7f3318158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.763+0000 7f331f9e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3318152650 0x7f331814ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.763+0000 7f331f9e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f331814b360 0x7f33181ee1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.763+0000 7f331f9e2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33181551b0 con 0x7f331814b360 2024-03-06T07:27:27.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.763+0000 7f331f9e2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33180c6fa0 con 0x7f33180bf9d0 2024-03-06T07:27:27.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.763+0000 7f331f9e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3318155490 con 0x7f3318152650 2024-03-06T07:27:27.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.763+0000 7f331df7f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f331814b360 0x7f33181ee1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.763+0000 7f331df7f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f331814b360 0x7f33181ee1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59332/0 (socket says 172.21.15.1:59332) 2024-03-06T07:27:27.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.763+0000 7f331df7f640 1 -- 172.21.15.1:0/1471103532 learned_addr learned my addr 172.21.15.1:0/1471103532 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:27.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.763+0000 7f331d77e640 1 --2- 172.21.15.1:0/1471103532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33180bf9d0 0x7f3318158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.763+0000 7f331cf7d640 1 --2- 172.21.15.1:0/1471103532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3318152650 0x7f331814ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.764+0000 7f331df7f640 1 -- 172.21.15.1:0/1471103532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3318152650 msgr2=0x7f331814ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.764+0000 7f331df7f640 1 --2- 172.21.15.1:0/1471103532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3318152650 0x7f331814ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.764+0000 7f331df7f640 1 -- 172.21.15.1:0/1471103532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33180bf9d0 msgr2=0x7f3318158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.764+0000 7f331df7f640 1 --2- 172.21.15.1:0/1471103532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33180bf9d0 0x7f3318158790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.764+0000 7f331df7f640 1 -- 172.21.15.1:0/1471103532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3318156e90 con 0x7f331814b360 2024-03-06T07:27:27.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.764+0000 7f331d77e640 1 --2- 172.21.15.1:0/1471103532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33180bf9d0 0x7f3318158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:27.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.764+0000 7f331df7f640 1 --2- 172.21.15.1:0/1471103532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f331814b360 0x7f33181ee1b0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f3308007e60 tx=0x7f330800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.764+0000 7f3315ffb640 1 -- 172.21.15.1:0/1471103532 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3308014420 con 0x7f331814b360 2024-03-06T07:27:27.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.764+0000 7f3315ffb640 1 -- 172.21.15.1:0/1471103532 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3308019070 con 0x7f331814b360 2024-03-06T07:27:27.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.764+0000 7f331f9e2640 1 -- 172.21.15.1:0/1471103532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3318157170 con 0x7f331814b360 2024-03-06T07:27:27.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.764+0000 7f3315ffb640 1 -- 172.21.15.1:0/1471103532 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3308014820 con 0x7f331814b360 2024-03-06T07:27:27.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.764+0000 7f331f9e2640 1 -- 172.21.15.1:0/1471103532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3318151d80 con 0x7f331814b360 2024-03-06T07:27:27.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.765+0000 7f331f9e2640 1 -- 172.21.15.1:0/1471103532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3318067fd0 con 0x7f331814b360 2024-03-06T07:27:27.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.766+0000 7f3315ffb640 1 -- 172.21.15.1:0/1471103532 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33080149c0 con 0x7f331814b360 2024-03-06T07:27:27.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.766+0000 7f3315ffb640 1 --2- 172.21.15.1:0/1471103532 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f330406b500 0x7f330406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.766+0000 7f3315ffb640 1 -- 172.21.15.1:0/1471103532 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f330808ac30 con 0x7f331814b360 2024-03-06T07:27:27.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.766+0000 7f331d77e640 1 --2- 172.21.15.1:0/1471103532 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f330406b500 0x7f330406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.767+0000 7f331d77e640 1 --2- 172.21.15.1:0/1471103532 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f330406b500 0x7f330406d9c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3310036000 tx=0x7f331003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.770+0000 7f3315ffb640 1 -- 172.21.15.1:0/1471103532 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3308059890 con 0x7f331814b360 2024-03-06T07:27:27.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.993+0000 7f331f9e2640 1 -- 172.21.15.1:0/1471103532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33180b06f0 con 0x7f331814b360 2024-03-06T07:27:27.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.993+0000 7f3315ffb640 1 -- 172.21.15.1:0/1471103532 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f3308059190 con 0x7f331814b360 2024-03-06T07:27:27.996 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:27.996 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":159,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:15.309451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9489":{"gid":9489,"name":"c","rank":0,"incarnation":155,"state":"up:stopping","state_seq":3,"addr":"172.21.15.1:6835/1751109564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1751109564},{"type":"v1","addr":"172.21.15.1:6835","nonce":1751109564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:27:27.996 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 159 2024-03-06T07:27:27.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.996+0000 7f331f9e2640 1 -- 172.21.15.1:0/1471103532 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f330406b500 msgr2=0x7f330406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.996+0000 7f331f9e2640 1 --2- 172.21.15.1:0/1471103532 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f330406b500 0x7f330406d9c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3310036000 tx=0x7f331003c040 comp rx=0 tx=0).stop 2024-03-06T07:27:27.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.996+0000 7f331f9e2640 1 -- 172.21.15.1:0/1471103532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f331814b360 msgr2=0x7f33181ee1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.996+0000 7f331f9e2640 1 --2- 172.21.15.1:0/1471103532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f331814b360 0x7f33181ee1b0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f3308007e60 tx=0x7f330800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.996+0000 7f331f9e2640 1 -- 172.21.15.1:0/1471103532 shutdown_connections 2024-03-06T07:27:27.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.996+0000 7f331f9e2640 1 --2- 172.21.15.1:0/1471103532 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f330406b500 0x7f330406d9c0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.996+0000 7f331f9e2640 1 --2- 172.21.15.1:0/1471103532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f331814b360 0x7f33181ee1b0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.996+0000 7f331f9e2640 1 --2- 172.21.15.1:0/1471103532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3318152650 0x7f331814ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.996+0000 7f331f9e2640 1 --2- 172.21.15.1:0/1471103532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33180bf9d0 0x7f3318158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.996+0000 7f331f9e2640 1 -- 172.21.15.1:0/1471103532 >> 172.21.15.1:0/1471103532 conn(0x7f3318066ee0 msgr2=0x7f33180c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.996+0000 7f331f9e2640 1 -- 172.21.15.1:0/1471103532 shutdown_connections 2024-03-06T07:27:27.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:27.996+0000 7f331f9e2640 1 -- 172.21.15.1:0/1471103532 wait complete. 2024-03-06T07:27:28.411 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 159, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:04.619091+0000', 'modified': '2024-03-06T07:27:15.309451+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9489': {'gid': 9489, 'name': 'c', 'rank': 0, 'incarnation': 155, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.1:6835/1751109564', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1751109564}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1751109564}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:28.411 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9489:up:stopping 2024-03-06T07:27:29.411 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:29.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.523+0000 7f4c527c9640 1 Processor -- start 2024-03-06T07:27:29.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.523+0000 7f4c527c9640 1 -- start start 2024-03-06T07:27:29.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.524+0000 7f4c527c9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c4c152690 0x7f4c4c155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.524+0000 7f4c527c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c4c156cd0 0x7f4c4c157160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.524+0000 7f4c527c9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c4c1561e0 con 0x7f4c4c0c6730 2024-03-06T07:27:29.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.524+0000 7f4c527c9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4c0c3300 con 0x7f4c4c152690 2024-03-06T07:27:29.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.524+0000 7f4c527c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c0c35a0 con 0x7f4c4c156cd0 2024-03-06T07:27:29.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.524+0000 7f4c4b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4c4c0c6730 0x7f4c4c0c57d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34782/0 (socket says 172.21.15.1:34782) 2024-03-06T07:27:29.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.524+0000 7f4c4b7fe640 1 -- 172.21.15.1:0/499167045 learned_addr learned my addr 172.21.15.1:0/499167045 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:29.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.524+0000 7f4c50d66640 1 --2- 172.21.15.1:0/499167045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c4c156cd0 0x7f4c4c157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.524+0000 7f4c4bfff640 1 --2- 172.21.15.1:0/499167045 >> v2:172.21.15.98:3300/0 conn(0x7f4c4c152690 0x7f4c4c155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.524+0000 7f4c4affd640 1 -- 172.21.15.1:0/499167045 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922779146 0 0) 0x7f4c3c001110 con 0x7f4c4c0c6730 2024-03-06T07:27:29.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.524+0000 7f4c4affd640 1 -- 172.21.15.1:0/499167045 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c34003520 con 0x7f4c4c0c6730 2024-03-06T07:27:29.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4affd640 1 -- 172.21.15.1:0/499167045 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1485839304 0 0) 0x7f4c34003520 con 0x7f4c4c0c6730 2024-03-06T07:27:29.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4affd640 1 -- 172.21.15.1:0/499167045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c4c156cd0 msgr2=0x7f4c4c157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4affd640 1 --2- 172.21.15.1:0/499167045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c4c156cd0 0x7f4c4c157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4affd640 1 -- 172.21.15.1:0/499167045 >> v2:172.21.15.98:3300/0 conn(0x7f4c4c152690 msgr2=0x7f4c4c155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4affd640 1 --2- 172.21.15.1:0/499167045 >> v2:172.21.15.98:3300/0 conn(0x7f4c4c152690 0x7f4c4c155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4affd640 1 -- 172.21.15.1:0/499167045 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c0c3880 con 0x7f4c4c0c6730 2024-03-06T07:27:29.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4affd640 1 -- 172.21.15.1:0/499167045 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f4c3c001840 con 0x7f4c4c0c6730 2024-03-06T07:27:29.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4affd640 1 -- 172.21.15.1:0/499167045 >> v1:172.21.15.1:6789/0 conn(0x7f4c4c0c6730 legacy=0x7f4c4c0c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4affd640 1 --2- 172.21.15.1:0/499167045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c340039b0 0x7f4c34003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4affd640 1 --2- 172.21.15.1:0/499167045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c4c156cd0 0x7f4c34004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4affd640 1 --2- 172.21.15.1:0/499167045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c34004d40 0x7f4c340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4affd640 1 -- 172.21.15.1:0/499167045 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c0c3300 con 0x7f4c340039b0 2024-03-06T07:27:29.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4affd640 1 -- 172.21.15.1:0/499167045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c0c35a0 con 0x7f4c34004d40 2024-03-06T07:27:29.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4affd640 1 -- 172.21.15.1:0/499167045 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3400b9c0 con 0x7f4c4c156cd0 2024-03-06T07:27:29.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4bfff640 1 --2- 172.21.15.1:0/499167045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c340039b0 0x7f4c34003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4b7fe640 1 --2- 172.21.15.1:0/499167045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c4c156cd0 0x7f4c34004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c50d66640 1 --2- 172.21.15.1:0/499167045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c34004d40 0x7f4c340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4bfff640 1 -- 172.21.15.1:0/499167045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c4c156cd0 msgr2=0x7f4c34004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4bfff640 1 --2- 172.21.15.1:0/499167045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c4c156cd0 0x7f4c34004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4bfff640 1 -- 172.21.15.1:0/499167045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c34004d40 msgr2=0x7f4c340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4bfff640 1 --2- 172.21.15.1:0/499167045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c34004d40 0x7f4c340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.525+0000 7f4c4bfff640 1 -- 172.21.15.1:0/499167045 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c34003520 con 0x7f4c340039b0 2024-03-06T07:27:29.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.526+0000 7f4c4bfff640 1 --2- 172.21.15.1:0/499167045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c340039b0 0x7f4c34003dd0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f4c40009ac0 tx=0x7f4c34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64713db0fb4f1770 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.526+0000 7f4c4affd640 1 -- 172.21.15.1:0/499167045 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c40039070 con 0x7f4c340039b0 2024-03-06T07:27:29.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.526+0000 7f4c4affd640 1 -- 172.21.15.1:0/499167045 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4002c710 con 0x7f4c340039b0 2024-03-06T07:27:29.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.526+0000 7f4c4affd640 1 -- 172.21.15.1:0/499167045 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c40009690 con 0x7f4c340039b0 2024-03-06T07:27:29.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.527+0000 7f4c527c9640 1 -- 172.21.15.1:0/499167045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c340039b0 msgr2=0x7f4c34003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.527+0000 7f4c527c9640 1 --2- 172.21.15.1:0/499167045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c340039b0 0x7f4c34003dd0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f4c40009ac0 tx=0x7f4c34005830 comp rx=0 tx=0).stop 2024-03-06T07:27:29.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.527+0000 7f4c50d66640 1 -- 172.21.15.1:0/499167045 reap_dead start 2024-03-06T07:27:29.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.527+0000 7f4c527c9640 1 -- 172.21.15.1:0/499167045 shutdown_connections 2024-03-06T07:27:29.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.527+0000 7f4c527c9640 1 -- 172.21.15.1:0/499167045 >> 172.21.15.1:0/499167045 conn(0x7f4c4c06b410 msgr2=0x7f4c4c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.527+0000 7f4c527c9640 1 -- 172.21.15.1:0/499167045 shutdown_connections 2024-03-06T07:27:29.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.527+0000 7f4c527c9640 1 -- 172.21.15.1:0/499167045 wait complete. 2024-03-06T07:27:29.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.528+0000 7f4c527c9640 1 Processor -- start 2024-03-06T07:27:29.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.528+0000 7f4c527c9640 1 -- start start 2024-03-06T07:27:29.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.529+0000 7f4c527c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c340039b0 0x7f4c4c070d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.529+0000 7f4c527c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c4c0ad000 0x7f4c4c14ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.529+0000 7f4c527c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c4c06dc40 0x7f4c4c14f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.529+0000 7f4c527c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c0c5ee0 con 0x7f4c4c06dc40 2024-03-06T07:27:29.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.529+0000 7f4c527c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c1567b0 con 0x7f4c4c0ad000 2024-03-06T07:27:29.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.529+0000 7f4c527c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c0c6060 con 0x7f4c340039b0 2024-03-06T07:27:29.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.529+0000 7f4c50d66640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c4c06dc40 0x7f4c4c14f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.529+0000 7f4c50d66640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c4c06dc40 0x7f4c4c14f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59342/0 (socket says 172.21.15.1:59342) 2024-03-06T07:27:29.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.529+0000 7f4c50d66640 1 -- 172.21.15.1:0/2385343945 learned_addr learned my addr 172.21.15.1:0/2385343945 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:29.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.529+0000 7f4c4b7fe640 1 --2- 172.21.15.1:0/2385343945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c4c0ad000 0x7f4c4c14ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.529+0000 7f4c4bfff640 1 --2- 172.21.15.1:0/2385343945 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c340039b0 0x7f4c4c070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.529+0000 7f4c50d66640 1 -- 172.21.15.1:0/2385343945 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c340039b0 msgr2=0x7f4c4c070d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.529+0000 7f4c50d66640 1 --2- 172.21.15.1:0/2385343945 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c340039b0 0x7f4c4c070d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.529+0000 7f4c50d66640 1 -- 172.21.15.1:0/2385343945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c4c0ad000 msgr2=0x7f4c4c14ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.529+0000 7f4c50d66640 1 --2- 172.21.15.1:0/2385343945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c4c0ad000 0x7f4c4c14ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.529+0000 7f4c50d66640 1 -- 172.21.15.1:0/2385343945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c1ee010 con 0x7f4c4c06dc40 2024-03-06T07:27:29.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.529+0000 7f4c50d66640 1 --2- 172.21.15.1:0/2385343945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c4c06dc40 0x7f4c4c14f2a0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0c4f30 tx=0x7f4c4400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.530+0000 7f4c48ff9640 1 -- 172.21.15.1:0/2385343945 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4400fde0 con 0x7f4c4c06dc40 2024-03-06T07:27:29.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.530+0000 7f4c48ff9640 1 -- 172.21.15.1:0/2385343945 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4401d070 con 0x7f4c4c06dc40 2024-03-06T07:27:29.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.530+0000 7f4c527c9640 1 -- 172.21.15.1:0/2385343945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4c1ee2b0 con 0x7f4c4c06dc40 2024-03-06T07:27:29.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.530+0000 7f4c48ff9640 1 -- 172.21.15.1:0/2385343945 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44015670 con 0x7f4c4c06dc40 2024-03-06T07:27:29.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.530+0000 7f4c527c9640 1 -- 172.21.15.1:0/2385343945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4c1ee870 con 0x7f4c4c06dc40 2024-03-06T07:27:29.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.531+0000 7f4c527c9640 1 -- 172.21.15.1:0/2385343945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c4c067fd0 con 0x7f4c4c06dc40 2024-03-06T07:27:29.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.531+0000 7f4c48ff9640 1 -- 172.21.15.1:0/2385343945 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c44015810 con 0x7f4c4c06dc40 2024-03-06T07:27:29.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.531+0000 7f4c48ff9640 1 --2- 172.21.15.1:0/2385343945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c3806b500 0x7f4c3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.532+0000 7f4c48ff9640 1 -- 172.21.15.1:0/2385343945 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4c4408ad10 con 0x7f4c4c06dc40 2024-03-06T07:27:29.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.532+0000 7f4c4bfff640 1 --2- 172.21.15.1:0/2385343945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c3806b500 0x7f4c3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.532+0000 7f4c4bfff640 1 --2- 172.21.15.1:0/2385343945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c3806b500 0x7f4c3806d9c0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f4c40036000 tx=0x7f4c40030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.536+0000 7f4c48ff9640 1 -- 172.21.15.1:0/2385343945 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c44010040 con 0x7f4c4c06dc40 2024-03-06T07:27:29.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.754+0000 7f4c527c9640 1 -- 172.21.15.1:0/2385343945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c4c0c8d40 con 0x7f4c4c06dc40 2024-03-06T07:27:29.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.755+0000 7f4c48ff9640 1 -- 172.21.15.1:0/2385343945 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f4c44059a00 con 0x7f4c4c06dc40 2024-03-06T07:27:29.757 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:29.757 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":159,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:15.309451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9489":{"gid":9489,"name":"c","rank":0,"incarnation":155,"state":"up:stopping","state_seq":3,"addr":"172.21.15.1:6835/1751109564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1751109564},{"type":"v1","addr":"172.21.15.1:6835","nonce":1751109564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:27:29.758 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 159 2024-03-06T07:27:29.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.757+0000 7f4c527c9640 1 -- 172.21.15.1:0/2385343945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c3806b500 msgr2=0x7f4c3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.757+0000 7f4c527c9640 1 --2- 172.21.15.1:0/2385343945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c3806b500 0x7f4c3806d9c0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f4c40036000 tx=0x7f4c40030000 comp rx=0 tx=0).stop 2024-03-06T07:27:29.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.757+0000 7f4c527c9640 1 -- 172.21.15.1:0/2385343945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c4c06dc40 msgr2=0x7f4c4c14f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.757+0000 7f4c527c9640 1 --2- 172.21.15.1:0/2385343945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c4c06dc40 0x7f4c4c14f2a0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0c4f30 tx=0x7f4c4400f580 comp rx=0 tx=0).stop 2024-03-06T07:27:29.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.757+0000 7f4c527c9640 1 -- 172.21.15.1:0/2385343945 shutdown_connections 2024-03-06T07:27:29.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.757+0000 7f4c527c9640 1 --2- 172.21.15.1:0/2385343945 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c3806b500 0x7f4c3806d9c0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.757+0000 7f4c527c9640 1 --2- 172.21.15.1:0/2385343945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c4c06dc40 0x7f4c4c14f2a0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.757+0000 7f4c527c9640 1 --2- 172.21.15.1:0/2385343945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c4c0ad000 0x7f4c4c14ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.757+0000 7f4c527c9640 1 --2- 172.21.15.1:0/2385343945 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c340039b0 0x7f4c4c070d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.757+0000 7f4c527c9640 1 -- 172.21.15.1:0/2385343945 >> 172.21.15.1:0/2385343945 conn(0x7f4c4c06b410 msgr2=0x7f4c4c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.757+0000 7f4c527c9640 1 -- 172.21.15.1:0/2385343945 shutdown_connections 2024-03-06T07:27:29.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:29.757+0000 7f4c527c9640 1 -- 172.21.15.1:0/2385343945 wait complete. 2024-03-06T07:27:30.173 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 159, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:04.619091+0000', 'modified': '2024-03-06T07:27:15.309451+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9489': {'gid': 9489, 'name': 'c', 'rank': 0, 'incarnation': 155, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.1:6835/1751109564', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1751109564}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1751109564}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:30.173 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9489:up:stopping 2024-03-06T07:27:30.348 INFO:tasks.ceph.mds.c.smithi001.stderr: -782> 2024-03-06T07:27:10.556+0000 7fbb7c614640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:30.374 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:27:31.174 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:31.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.286+0000 7f51c9502640 1 Processor -- start 2024-03-06T07:27:31.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.286+0000 7f51c9502640 1 -- start start 2024-03-06T07:27:31.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.287+0000 7f51c9502640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51c4156cd0 0x7f51c4157160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.287+0000 7f51c9502640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c40c3300 con 0x7f51c40c6730 2024-03-06T07:27:31.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.287+0000 7f51c9502640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c40c35a0 con 0x7f51c4152690 2024-03-06T07:27:31.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.287+0000 7f51c9502640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c40c3880 con 0x7f51c4156cd0 2024-03-06T07:27:31.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.287+0000 7f51c27fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f51c40c6730 0x7f51c40c57d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34802/0 (socket says 172.21.15.1:34802) 2024-03-06T07:27:31.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.287+0000 7f51c27fc640 1 -- 172.21.15.1:0/669669025 learned_addr learned my addr 172.21.15.1:0/669669025 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:31.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.287+0000 7f51c37fe640 1 --2- 172.21.15.1:0/669669025 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51c4156cd0 0x7f51c4157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.287+0000 7f51c1ffb640 1 -- 172.21.15.1:0/669669025 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454296329 0 0) 0x7f51c40c3300 con 0x7f51c40c6730 2024-03-06T07:27:31.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.287+0000 7f51c1ffb640 1 -- 172.21.15.1:0/669669025 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51b0003520 con 0x7f51c40c6730 2024-03-06T07:27:31.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.287+0000 7f51c1ffb640 1 -- 172.21.15.1:0/669669025 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1217864610 0 0) 0x7f51c40c35a0 con 0x7f51c4152690 2024-03-06T07:27:31.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.287+0000 7f51c1ffb640 1 -- 172.21.15.1:0/669669025 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51c40c3300 con 0x7f51c4152690 2024-03-06T07:27:31.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.287+0000 7f51c37fe640 1 -- 172.21.15.1:0/669669025 >> v1:172.21.15.98:6789/0 conn(0x7f51c4152690 legacy=0x7f51c4155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.287+0000 7f51c37fe640 1 -- 172.21.15.1:0/669669025 >> v1:172.21.15.1:6789/0 conn(0x7f51c40c6730 legacy=0x7f51c40c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.287+0000 7f51c37fe640 1 -- 172.21.15.1:0/669669025 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c40c3b60 con 0x7f51c4156cd0 2024-03-06T07:27:31.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.288+0000 7f51c37fe640 1 --2- 172.21.15.1:0/669669025 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51c4156cd0 0x7f51c4157160 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f51bc0097d0 tx=0x7f51bc02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91c66cd52d0ed5a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.288+0000 7f51c1ffb640 1 -- 172.21.15.1:0/669669025 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc035020 con 0x7f51c4156cd0 2024-03-06T07:27:31.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.289+0000 7f51c1ffb640 1 -- 172.21.15.1:0/669669025 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51bc031800 con 0x7f51c4156cd0 2024-03-06T07:27:31.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.289+0000 7f51c1ffb640 1 -- 172.21.15.1:0/669669025 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc031bd0 con 0x7f51c4156cd0 2024-03-06T07:27:31.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.289+0000 7f51c9502640 1 -- 172.21.15.1:0/669669025 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51c4156cd0 msgr2=0x7f51c4157160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.289+0000 7f51c9502640 1 --2- 172.21.15.1:0/669669025 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51c4156cd0 0x7f51c4157160 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f51bc0097d0 tx=0x7f51bc02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.289+0000 7f51c9502640 1 -- 172.21.15.1:0/669669025 shutdown_connections 2024-03-06T07:27:31.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.289+0000 7f51c9502640 1 --2- 172.21.15.1:0/669669025 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51c4156cd0 0x7f51c4157160 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.289+0000 7f51c9502640 1 -- 172.21.15.1:0/669669025 >> 172.21.15.1:0/669669025 conn(0x7f51c406b410 msgr2=0x7f51c406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.289+0000 7f51c9502640 1 -- 172.21.15.1:0/669669025 shutdown_connections 2024-03-06T07:27:31.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.289+0000 7f51c9502640 1 -- 172.21.15.1:0/669669025 wait complete. 2024-03-06T07:27:31.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.290+0000 7f51c9502640 1 Processor -- start 2024-03-06T07:27:31.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.290+0000 7f51c9502640 1 -- start start 2024-03-06T07:27:31.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.291+0000 7f51c9502640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c40c6730 0x7f51c406d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.291+0000 7f51c9502640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51c4152690 0x7f51c414e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.291+0000 7f51c9502640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51c4156cd0 0x7f51c414ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.291+0000 7f51c9502640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c40c6060 con 0x7f51c40c6730 2024-03-06T07:27:31.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.291+0000 7f51c9502640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c40c5ee0 con 0x7f51c4156cd0 2024-03-06T07:27:31.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.291+0000 7f51c9502640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c4152a70 con 0x7f51c4152690 2024-03-06T07:27:31.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.291+0000 7f51c2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c40c6730 0x7f51c406d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.291+0000 7f51c2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c40c6730 0x7f51c406d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59358/0 (socket says 172.21.15.1:59358) 2024-03-06T07:27:31.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.291+0000 7f51c2ffd640 1 -- 172.21.15.1:0/219211939 learned_addr learned my addr 172.21.15.1:0/219211939 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:31.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.291+0000 7f51c27fc640 1 --2- 172.21.15.1:0/219211939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51c4152690 0x7f51c414e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.291+0000 7f51c2ffd640 1 -- 172.21.15.1:0/219211939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51c4152690 msgr2=0x7f51c414e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.291+0000 7f51c37fe640 1 --2- 172.21.15.1:0/219211939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51c4156cd0 0x7f51c414ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.291+0000 7f51c2ffd640 1 --2- 172.21.15.1:0/219211939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51c4152690 0x7f51c414e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.291+0000 7f51c2ffd640 1 -- 172.21.15.1:0/219211939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51c4156cd0 msgr2=0x7f51c414ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.292+0000 7f51c2ffd640 1 --2- 172.21.15.1:0/219211939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51c4156cd0 0x7f51c414ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.292+0000 7f51c2ffd640 1 -- 172.21.15.1:0/219211939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c41ea830 con 0x7f51c40c6730 2024-03-06T07:27:31.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.292+0000 7f51c27fc640 1 --2- 172.21.15.1:0/219211939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51c4152690 0x7f51c414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:31.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.292+0000 7f51c2ffd640 1 --2- 172.21.15.1:0/219211939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c40c6730 0x7f51c406d4f0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f51b400c870 tx=0x7f51b400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.292+0000 7f51bb7fe640 1 -- 172.21.15.1:0/219211939 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b4003b70 con 0x7f51c40c6730 2024-03-06T07:27:31.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.292+0000 7f51bb7fe640 1 -- 172.21.15.1:0/219211939 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b4003d30 con 0x7f51c40c6730 2024-03-06T07:27:31.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.292+0000 7f51c9502640 1 -- 172.21.15.1:0/219211939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c414dc50 con 0x7f51c40c6730 2024-03-06T07:27:31.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.292+0000 7f51bb7fe640 1 -- 172.21.15.1:0/219211939 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b400aa30 con 0x7f51c40c6730 2024-03-06T07:27:31.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.292+0000 7f51c9502640 1 -- 172.21.15.1:0/219211939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c414e210 con 0x7f51c40c6730 2024-03-06T07:27:31.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.293+0000 7f51c9502640 1 -- 172.21.15.1:0/219211939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c414e410 con 0x7f51c40c6730 2024-03-06T07:27:31.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.293+0000 7f51bb7fe640 1 -- 172.21.15.1:0/219211939 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f51b400abd0 con 0x7f51c40c6730 2024-03-06T07:27:31.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.294+0000 7f51bb7fe640 1 --2- 172.21.15.1:0/219211939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51a406b4d0 0x7f51a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.294+0000 7f51bb7fe640 1 -- 172.21.15.1:0/219211939 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f51b408a2d0 con 0x7f51c40c6730 2024-03-06T07:27:31.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.294+0000 7f51c27fc640 1 --2- 172.21.15.1:0/219211939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51a406b4d0 0x7f51a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.295+0000 7f51c27fc640 1 --2- 172.21.15.1:0/219211939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51a406b4d0 0x7f51a406d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f51c4068340 tx=0x7f51ac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.298+0000 7f51bb7fe640 1 -- 172.21.15.1:0/219211939 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51c414e410 con 0x7f51c40c6730 2024-03-06T07:27:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.520+0000 7f51c9502640 1 -- 172.21.15.1:0/219211939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51c4067fd0 con 0x7f51c40c6730 2024-03-06T07:27:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.521+0000 7f51bb7fe640 1 -- 172.21.15.1:0/219211939 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+4952 (secure 0 0 0) 0x7f51b4059050 con 0x7f51c40c6730 2024-03-06T07:27:31.523 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:31.523 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":161,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9738,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1021164277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1021164277},{"type":"v1","addr":"172.21.15.1:6835","nonce":1021164277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":161}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:30.209688+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:27:31.523 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 161 2024-03-06T07:27:31.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.523+0000 7f51c9502640 1 -- 172.21.15.1:0/219211939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51a406b4d0 msgr2=0x7f51a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.523+0000 7f51c9502640 1 --2- 172.21.15.1:0/219211939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51a406b4d0 0x7f51a406d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f51c4068340 tx=0x7f51ac009040 comp rx=0 tx=0).stop 2024-03-06T07:27:31.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.523+0000 7f51c9502640 1 -- 172.21.15.1:0/219211939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c40c6730 msgr2=0x7f51c406d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.523+0000 7f51c9502640 1 --2- 172.21.15.1:0/219211939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c40c6730 0x7f51c406d4f0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f51b400c870 tx=0x7f51b400cd30 comp rx=0 tx=0).stop 2024-03-06T07:27:31.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.523+0000 7f51c9502640 1 -- 172.21.15.1:0/219211939 shutdown_connections 2024-03-06T07:27:31.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.523+0000 7f51c9502640 1 --2- 172.21.15.1:0/219211939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f51a406b4d0 0x7f51a406d990 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.523+0000 7f51c9502640 1 --2- 172.21.15.1:0/219211939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51c4156cd0 0x7f51c414ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.523+0000 7f51c9502640 1 --2- 172.21.15.1:0/219211939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51c4152690 0x7f51c414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.523+0000 7f51c9502640 1 --2- 172.21.15.1:0/219211939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c40c6730 0x7f51c406d4f0 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.523+0000 7f51c9502640 1 -- 172.21.15.1:0/219211939 >> 172.21.15.1:0/219211939 conn(0x7f51c406b410 msgr2=0x7f51c4152f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.523+0000 7f51c9502640 1 -- 172.21.15.1:0/219211939 shutdown_connections 2024-03-06T07:27:31.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:31.523+0000 7f51c9502640 1 -- 172.21.15.1:0/219211939 wait complete. 2024-03-06T07:27:31.938 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 160, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:04.619091+0000', 'modified': '2024-03-06T07:27:30.209688+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [], 'up': {}, 'failed': [], 'damaged': [], 'stopped': [0], 'info': {}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:31.938 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/0 2024-03-06T07:27:31.938 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:32.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.040+0000 7f28102d4640 1 Processor -- start 2024-03-06T07:27:32.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.040+0000 7f28102d4640 1 -- start start 2024-03-06T07:27:32.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.041+0000 7f28102d4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f280806e670 0x7f280806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.041+0000 7f28102d4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f28080c1bc0 0x7f2808153290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.041+0000 7f28102d4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f280806f060 con 0x7f28080a8520 2024-03-06T07:27:32.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.041+0000 7f28102d4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28080c2ca0 con 0x7f28080c1bc0 2024-03-06T07:27:32.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.041+0000 7f28102d4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28080c26e0 con 0x7f280806e670 2024-03-06T07:27:32.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.041+0000 7f280e871640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f28080c1bc0 0x7f2808153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.041+0000 7f280e871640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f28080c1bc0 0x7f2808153290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59368/0 (socket says 172.21.15.1:59368) 2024-03-06T07:27:32.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.041+0000 7f280e871640 1 -- 172.21.15.1:0/944450523 learned_addr learned my addr 172.21.15.1:0/944450523 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:32.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.041+0000 7f280e070640 1 --2- 172.21.15.1:0/944450523 >> v2:172.21.15.98:3300/0 conn(0x7f280806e670 0x7f280806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.041+0000 7f280e871640 1 -- 172.21.15.1:0/944450523 >> v2:172.21.15.98:3300/0 conn(0x7f280806e670 msgr2=0x7f280806ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.041+0000 7f280e871640 1 --2- 172.21.15.1:0/944450523 >> v2:172.21.15.98:3300/0 conn(0x7f280806e670 0x7f280806ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.041+0000 7f280e871640 1 -- 172.21.15.1:0/944450523 >> v1:172.21.15.98:6789/0 conn(0x7f28080a8520 legacy=0x7f2808150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.041+0000 7f280e871640 1 -- 172.21.15.1:0/944450523 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28080c29c0 con 0x7f28080c1bc0 2024-03-06T07:27:32.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.042+0000 7f280e871640 1 --2- 172.21.15.1:0/944450523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28080c1bc0 0x7f2808153290 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f28080732d0 tx=0x7f280002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2fd41bab5da23c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.042+0000 7f280d06e640 1 -- 172.21.15.1:0/944450523 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280002c9f0 con 0x7f28080c1bc0 2024-03-06T07:27:32.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.042+0000 7f280d06e640 1 -- 172.21.15.1:0/944450523 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280002cb90 con 0x7f28080c1bc0 2024-03-06T07:27:32.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.042+0000 7f280d06e640 1 -- 172.21.15.1:0/944450523 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28000343f0 con 0x7f28080c1bc0 2024-03-06T07:27:32.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.043+0000 7f28102d4640 1 -- 172.21.15.1:0/944450523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28080c1bc0 msgr2=0x7f2808153290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.043+0000 7f28102d4640 1 --2- 172.21.15.1:0/944450523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28080c1bc0 0x7f2808153290 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f28080732d0 tx=0x7f280002be50 comp rx=0 tx=0).stop 2024-03-06T07:27:32.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.043+0000 7f28102d4640 1 -- 172.21.15.1:0/944450523 shutdown_connections 2024-03-06T07:27:32.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.043+0000 7f28102d4640 1 --2- 172.21.15.1:0/944450523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28080c1bc0 0x7f2808153290 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.043+0000 7f28102d4640 1 --2- 172.21.15.1:0/944450523 >> v2:172.21.15.98:3300/0 conn(0x7f280806e670 0x7f280806ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.043+0000 7f28102d4640 1 -- 172.21.15.1:0/944450523 >> 172.21.15.1:0/944450523 conn(0x7f28080c3cc0 msgr2=0x7f28080c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.043+0000 7f28102d4640 1 -- 172.21.15.1:0/944450523 shutdown_connections 2024-03-06T07:27:32.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.043+0000 7f28102d4640 1 -- 172.21.15.1:0/944450523 wait complete. 2024-03-06T07:27:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.044+0000 7f28102d4640 1 Processor -- start 2024-03-06T07:27:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.044+0000 7f28102d4640 1 -- start start 2024-03-06T07:27:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f28102d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f280806e670 0x7f2808145b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f28102d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28080a8520 0x7f28080c0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f28102d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f28080c1bc0 0x7f2808144d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f28102d4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2808153950 con 0x7f280806e670 2024-03-06T07:27:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f28102d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28081537d0 con 0x7f28080a8520 2024-03-06T07:27:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f28102d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2808153ad0 con 0x7f28080c1bc0 2024-03-06T07:27:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f280e070640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f280806e670 0x7f2808145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f280e070640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f280806e670 0x7f2808145b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59372/0 (socket says 172.21.15.1:59372) 2024-03-06T07:27:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f280e070640 1 -- 172.21.15.1:0/4189001933 learned_addr learned my addr 172.21.15.1:0/4189001933 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f280e871640 1 --2- 172.21.15.1:0/4189001933 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f28080c1bc0 0x7f2808144d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f280d86f640 1 --2- 172.21.15.1:0/4189001933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28080a8520 0x7f28080c0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f280e070640 1 -- 172.21.15.1:0/4189001933 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f28080c1bc0 msgr2=0x7f2808144d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f280e070640 1 --2- 172.21.15.1:0/4189001933 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f28080c1bc0 0x7f2808144d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f280e070640 1 -- 172.21.15.1:0/4189001933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28080a8520 msgr2=0x7f28080c0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f280e070640 1 --2- 172.21.15.1:0/4189001933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28080a8520 0x7f28080c0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f280e070640 1 -- 172.21.15.1:0/4189001933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280814fd00 con 0x7f280806e670 2024-03-06T07:27:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f280d86f640 1 --2- 172.21.15.1:0/4189001933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28080a8520 0x7f28080c0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f280e070640 1 --2- 172.21.15.1:0/4189001933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f280806e670 0x7f2808145b00 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f280400d830 tx=0x7f280400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f27feffd640 1 -- 172.21.15.1:0/4189001933 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280400b6c0 con 0x7f280806e670 2024-03-06T07:27:32.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f27feffd640 1 -- 172.21.15.1:0/4189001933 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2804019070 con 0x7f280806e670 2024-03-06T07:27:32.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f28102d4640 1 -- 172.21.15.1:0/4189001933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2808145820 con 0x7f280806e670 2024-03-06T07:27:32.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f27feffd640 1 -- 172.21.15.1:0/4189001933 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280400ba80 con 0x7f280806e670 2024-03-06T07:27:32.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.045+0000 7f28102d4640 1 -- 172.21.15.1:0/4189001933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2808152ba0 con 0x7f280806e670 2024-03-06T07:27:32.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.046+0000 7f28102d4640 1 -- 172.21.15.1:0/4189001933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2808067fd0 con 0x7f280806e670 2024-03-06T07:27:32.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.047+0000 7f27feffd640 1 -- 172.21.15.1:0/4189001933 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f280401e460 con 0x7f280806e670 2024-03-06T07:27:32.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.047+0000 7f27feffd640 1 --2- 172.21.15.1:0/4189001933 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f27ec06b4d0 0x7f27ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.047+0000 7f27feffd640 1 -- 172.21.15.1:0/4189001933 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f280408aa00 con 0x7f280806e670 2024-03-06T07:27:32.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.047+0000 7f280d86f640 1 --2- 172.21.15.1:0/4189001933 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f27ec06b4d0 0x7f27ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.048+0000 7f280d86f640 1 --2- 172.21.15.1:0/4189001933 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f27ec06b4d0 0x7f27ec06d990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f280814c170 tx=0x7f27f8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.051+0000 7f27feffd640 1 -- 172.21.15.1:0/4189001933 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2804059780 con 0x7f280806e670 2024-03-06T07:27:32.214 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:32.216 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:32.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.272+0000 7f28102d4640 1 -- 172.21.15.1:0/4189001933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2808067570 con 0x7f280806e670 2024-03-06T07:27:32.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.272+0000 7f27feffd640 1 -- 172.21.15.1:0/4189001933 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+4952 (secure 0 0 0) 0x7f2804059080 con 0x7f280806e670 2024-03-06T07:27:32.274 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:32.274 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":161,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9738,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1021164277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1021164277},{"type":"v1","addr":"172.21.15.1:6835","nonce":1021164277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":161}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:30.209688+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:27:32.274 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 161 2024-03-06T07:27:32.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.274+0000 7f28102d4640 1 -- 172.21.15.1:0/4189001933 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f27ec06b4d0 msgr2=0x7f27ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.274+0000 7f28102d4640 1 --2- 172.21.15.1:0/4189001933 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f27ec06b4d0 0x7f27ec06d990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f280814c170 tx=0x7f27f8009040 comp rx=0 tx=0).stop 2024-03-06T07:27:32.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.274+0000 7f28102d4640 1 -- 172.21.15.1:0/4189001933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f280806e670 msgr2=0x7f2808145b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.274+0000 7f28102d4640 1 --2- 172.21.15.1:0/4189001933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f280806e670 0x7f2808145b00 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f280400d830 tx=0x7f280400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.274+0000 7f28102d4640 1 -- 172.21.15.1:0/4189001933 shutdown_connections 2024-03-06T07:27:32.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.274+0000 7f28102d4640 1 --2- 172.21.15.1:0/4189001933 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f27ec06b4d0 0x7f27ec06d990 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.274+0000 7f28102d4640 1 --2- 172.21.15.1:0/4189001933 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f28080c1bc0 0x7f2808144d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.274+0000 7f28102d4640 1 --2- 172.21.15.1:0/4189001933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28080a8520 0x7f28080c0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.274+0000 7f28102d4640 1 --2- 172.21.15.1:0/4189001933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f280806e670 0x7f2808145b00 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.274+0000 7f28102d4640 1 -- 172.21.15.1:0/4189001933 >> 172.21.15.1:0/4189001933 conn(0x7f28080c3cc0 msgr2=0x7f280806ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.274+0000 7f28102d4640 1 -- 172.21.15.1:0/4189001933 shutdown_connections 2024-03-06T07:27:32.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.274+0000 7f28102d4640 1 -- 172.21.15.1:0/4189001933 wait complete. 2024-03-06T07:27:32.287 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down false 2024-03-06T07:27:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.429+0000 7f9ed3695640 1 Processor -- start 2024-03-06T07:27:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.429+0000 7f9ed3695640 1 -- start start 2024-03-06T07:27:32.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.430+0000 7f9ed3695640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ecc0c6b10 0x7f9ecc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.430+0000 7f9ed3695640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ecc0c2fc0 con 0x7f9ecc152650 2024-03-06T07:27:32.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.430+0000 7f9ed3695640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ecc158da0 con 0x7f9ecc0bf9d0 2024-03-06T07:27:32.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.430+0000 7f9ed3695640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc155800 con 0x7f9ecc0c6b10 2024-03-06T07:27:32.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.430+0000 7f9ed1431640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9ecc152650 0x7f9ecc158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34812/0 (socket says 172.21.15.1:34812) 2024-03-06T07:27:32.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.430+0000 7f9ed1431640 1 -- 172.21.15.1:0/1321311248 learned_addr learned my addr 172.21.15.1:0/1321311248 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:32.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.430+0000 7f9ecbfff640 1 -- 172.21.15.1:0/1321311248 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565353369 0 0) 0x7f9ecc0c2fc0 con 0x7f9ecc152650 2024-03-06T07:27:32.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.430+0000 7f9ecbfff640 1 -- 172.21.15.1:0/1321311248 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9eb4003520 con 0x7f9ecc152650 2024-03-06T07:27:32.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.430+0000 7f9ed0c30640 1 --2- 172.21.15.1:0/1321311248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ecc0c6b10 0x7f9ecc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ecbfff640 1 -- 172.21.15.1:0/1321311248 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 147476176 0 0) 0x7f9eb4003520 con 0x7f9ecc152650 2024-03-06T07:27:32.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ecbfff640 1 -- 172.21.15.1:0/1321311248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ecc0c6b10 msgr2=0x7f9ecc155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ecbfff640 1 --2- 172.21.15.1:0/1321311248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ecc0c6b10 0x7f9ecc155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ecbfff640 1 -- 172.21.15.1:0/1321311248 >> v1:172.21.15.98:6789/0 conn(0x7f9ecc0bf9d0 legacy=0x7f9ecc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ecbfff640 1 -- 172.21.15.1:0/1321311248 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ecc0c6690 con 0x7f9ecc152650 2024-03-06T07:27:32.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ecbfff640 1 -- 172.21.15.1:0/1321311248 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f9ebc0019f0 con 0x7f9ecc152650 2024-03-06T07:27:32.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ecbfff640 1 -- 172.21.15.1:0/1321311248 >> v1:172.21.15.1:6789/0 conn(0x7f9ecc152650 legacy=0x7f9ecc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ecbfff640 1 --2- 172.21.15.1:0/1321311248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb40039e0 0x7f9eb4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ecbfff640 1 --2- 172.21.15.1:0/1321311248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb4004800 0x7f9eb4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ecbfff640 1 --2- 172.21.15.1:0/1321311248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb4005180 0x7f9eb400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ecbfff640 1 -- 172.21.15.1:0/1321311248 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc155800 con 0x7f9eb40039e0 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ecbfff640 1 -- 172.21.15.1:0/1321311248 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb400bf00 con 0x7f9eb4005180 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ecbfff640 1 -- 172.21.15.1:0/1321311248 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb400c0d0 con 0x7f9eb4004800 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ed1431640 1 --2- 172.21.15.1:0/1321311248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb40039e0 0x7f9eb4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ed1c32640 1 --2- 172.21.15.1:0/1321311248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb4005180 0x7f9eb400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ed0c30640 1 --2- 172.21.15.1:0/1321311248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb4004800 0x7f9eb4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ed1431640 1 -- 172.21.15.1:0/1321311248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb4004800 msgr2=0x7f9eb4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ed1431640 1 --2- 172.21.15.1:0/1321311248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb4004800 0x7f9eb4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ed1431640 1 -- 172.21.15.1:0/1321311248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb4005180 msgr2=0x7f9eb400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ed1431640 1 --2- 172.21.15.1:0/1321311248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb4005180 0x7f9eb400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.431+0000 7f9ed1431640 1 -- 172.21.15.1:0/1321311248 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb4003520 con 0x7f9eb40039e0 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.432+0000 7f9ed1431640 1 --2- 172.21.15.1:0/1321311248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb40039e0 0x7f9eb4003e00 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc00a840 tx=0x7f9eb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4626ab5e90e0335f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.432+0000 7f9ecbfff640 1 -- 172.21.15.1:0/1321311248 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc00a460 con 0x7f9eb40039e0 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.432+0000 7f9ecbfff640 1 -- 172.21.15.1:0/1321311248 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ebc034650 con 0x7f9eb40039e0 2024-03-06T07:27:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.432+0000 7f9ecbfff640 1 -- 172.21.15.1:0/1321311248 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc034a40 con 0x7f9eb40039e0 2024-03-06T07:27:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.432+0000 7f9ed3695640 1 -- 172.21.15.1:0/1321311248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb40039e0 msgr2=0x7f9eb4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.432+0000 7f9ed3695640 1 --2- 172.21.15.1:0/1321311248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb40039e0 0x7f9eb4003e00 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc00a840 tx=0x7f9eb4005830 comp rx=0 tx=0).stop 2024-03-06T07:27:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.432+0000 7f9ed1c32640 1 -- 172.21.15.1:0/1321311248 reap_dead start 2024-03-06T07:27:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.432+0000 7f9ed3695640 1 -- 172.21.15.1:0/1321311248 shutdown_connections 2024-03-06T07:27:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.432+0000 7f9ed3695640 1 -- 172.21.15.1:0/1321311248 >> 172.21.15.1:0/1321311248 conn(0x7f9ecc066ee0 msgr2=0x7f9ecc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.433+0000 7f9ed3695640 1 -- 172.21.15.1:0/1321311248 shutdown_connections 2024-03-06T07:27:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.433+0000 7f9ed3695640 1 -- 172.21.15.1:0/1321311248 wait complete. 2024-03-06T07:27:32.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.434+0000 7f9ed3695640 1 Processor -- start 2024-03-06T07:27:32.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.434+0000 7f9ed3695640 1 -- start start 2024-03-06T07:27:32.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.434+0000 7f9ed3695640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb40039e0 0x7f9ecc14d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.434+0000 7f9ed3695640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ecc14b6a0 0x7f9ecc14fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.434+0000 7f9ed3695640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ecc151bd0 0x7f9ecc1503b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.434+0000 7f9ed3695640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc1551b0 con 0x7f9ecc151bd0 2024-03-06T07:27:32.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.434+0000 7f9ed3695640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc0c6fa0 con 0x7f9ecc14b6a0 2024-03-06T07:27:32.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.434+0000 7f9ed3695640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc155490 con 0x7f9eb40039e0 2024-03-06T07:27:32.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.434+0000 7f9ed1c32640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ecc151bd0 0x7f9ecc1503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.434+0000 7f9ed1c32640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ecc151bd0 0x7f9ecc1503b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59400/0 (socket says 172.21.15.1:59400) 2024-03-06T07:27:32.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.434+0000 7f9ed1c32640 1 -- 172.21.15.1:0/3766151189 learned_addr learned my addr 172.21.15.1:0/3766151189 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:32.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.435+0000 7f9ed1431640 1 --2- 172.21.15.1:0/3766151189 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb40039e0 0x7f9ecc14d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.435+0000 7f9ed0c30640 1 --2- 172.21.15.1:0/3766151189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ecc14b6a0 0x7f9ecc14fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.435+0000 7f9ed1c32640 1 -- 172.21.15.1:0/3766151189 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb40039e0 msgr2=0x7f9ecc14d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.435+0000 7f9ed1c32640 1 --2- 172.21.15.1:0/3766151189 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb40039e0 0x7f9ecc14d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.435+0000 7f9ed1c32640 1 -- 172.21.15.1:0/3766151189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ecc14b6a0 msgr2=0x7f9ecc14fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.435+0000 7f9ed1c32640 1 --2- 172.21.15.1:0/3766151189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ecc14b6a0 0x7f9ecc14fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.435+0000 7f9ed1c32640 1 -- 172.21.15.1:0/3766151189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ecc156630 con 0x7f9ecc151bd0 2024-03-06T07:27:32.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.435+0000 7f9ed1431640 1 --2- 172.21.15.1:0/3766151189 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb40039e0 0x7f9ecc14d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:32.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.435+0000 7f9ed0c30640 1 --2- 172.21.15.1:0/3766151189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ecc14b6a0 0x7f9ecc14fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:32.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.435+0000 7f9ed1c32640 1 --2- 172.21.15.1:0/3766151189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ecc151bd0 0x7f9ecc1503b0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc08bd40 tx=0x7f9ec400e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.435+0000 7f9ec9ffb640 1 -- 172.21.15.1:0/3766151189 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4019070 con 0x7f9ecc151bd0 2024-03-06T07:27:32.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.435+0000 7f9ec9ffb640 1 -- 172.21.15.1:0/3766151189 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec40052e0 con 0x7f9ecc151bd0 2024-03-06T07:27:32.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.435+0000 7f9ec9ffb640 1 -- 172.21.15.1:0/3766151189 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4003830 con 0x7f9ecc151bd0 2024-03-06T07:27:32.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.435+0000 7f9ed3695640 1 -- 172.21.15.1:0/3766151189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ecc156910 con 0x7f9ecc151bd0 2024-03-06T07:27:32.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.435+0000 7f9ed3695640 1 -- 172.21.15.1:0/3766151189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ecc156ed0 con 0x7f9ecc151bd0 2024-03-06T07:27:32.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.436+0000 7f9ed3695640 1 -- 172.21.15.1:0/3766151189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ecc1570d0 con 0x7f9ecc151bd0 2024-03-06T07:27:32.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.437+0000 7f9ec9ffb640 1 -- 172.21.15.1:0/3766151189 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ec40039d0 con 0x7f9ecc151bd0 2024-03-06T07:27:32.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.437+0000 7f9ec9ffb640 1 --2- 172.21.15.1:0/3766151189 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9eb806b500 0x7f9eb806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.437+0000 7f9ec9ffb640 1 -- 172.21.15.1:0/3766151189 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9ec4089630 con 0x7f9ecc151bd0 2024-03-06T07:27:32.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.437+0000 7f9ed1431640 1 --2- 172.21.15.1:0/3766151189 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9eb806b500 0x7f9eb806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.438+0000 7f9ed1431640 1 --2- 172.21.15.1:0/3766151189 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9eb806b500 0x7f9eb806d9c0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc034000 tx=0x7f9ebc009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.442+0000 7f9ec9ffb640 1 -- 172.21.15.1:0/3766151189 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ec404ee10 con 0x7f9ecc151bd0 2024-03-06T07:27:32.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:32.664+0000 7f9ed3695640 1 -- 172.21.15.1:0/3766151189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "false"} v 0) v1 -- 0x7f9ecc067fd0 con 0x7f9ecc151bd0 2024-03-06T07:27:33.356 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:27:33.355+0000 7fb853791640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.359+0000 7f9ec9ffb640 1 -- 172.21.15.1:0/3766151189 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "false"}]=0 cephfs marked up, max_mds = 1 v162) v1 ==== 135+0+0 (secure 0 0 0) 0x7f9ecc1570d0 con 0x7f9ecc151bd0 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked up, max_mds = 1 2024-03-06T07:27:33.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.362+0000 7f9ed3695640 1 -- 172.21.15.1:0/3766151189 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9eb806b500 msgr2=0x7f9eb806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.362+0000 7f9ed3695640 1 --2- 172.21.15.1:0/3766151189 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9eb806b500 0x7f9eb806d9c0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc034000 tx=0x7f9ebc009030 comp rx=0 tx=0).stop 2024-03-06T07:27:33.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.362+0000 7f9ed3695640 1 -- 172.21.15.1:0/3766151189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ecc151bd0 msgr2=0x7f9ecc1503b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.362+0000 7f9ed3695640 1 --2- 172.21.15.1:0/3766151189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ecc151bd0 0x7f9ecc1503b0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc08bd40 tx=0x7f9ec400e8a0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.362+0000 7f9ed3695640 1 -- 172.21.15.1:0/3766151189 shutdown_connections 2024-03-06T07:27:33.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.362+0000 7f9ed3695640 1 --2- 172.21.15.1:0/3766151189 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9eb806b500 0x7f9eb806d9c0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.362+0000 7f9ed3695640 1 --2- 172.21.15.1:0/3766151189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ecc151bd0 0x7f9ecc1503b0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.362+0000 7f9ed3695640 1 --2- 172.21.15.1:0/3766151189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ecc14b6a0 0x7f9ecc14fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.362+0000 7f9ed3695640 1 --2- 172.21.15.1:0/3766151189 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb40039e0 0x7f9ecc14d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.362+0000 7f9ed3695640 1 -- 172.21.15.1:0/3766151189 >> 172.21.15.1:0/3766151189 conn(0x7f9ecc066ee0 msgr2=0x7f9ecc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.362+0000 7f9ed3695640 1 -- 172.21.15.1:0/3766151189 shutdown_connections 2024-03-06T07:27:33.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.362+0000 7f9ed3695640 1 -- 172.21.15.1:0/3766151189 wait complete. 2024-03-06T07:27:33.380 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:33.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.485+0000 7f0c6cd4b640 1 Processor -- start 2024-03-06T07:27:33.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.485+0000 7f0c6cd4b640 1 -- start start 2024-03-06T07:27:33.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.486+0000 7f0c6cd4b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c680c5850 0x7f0c680c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.486+0000 7f0c6cd4b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0c68155220 0x7f0c681556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.486+0000 7f0c6cd4b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c68158910 con 0x7f0c68152920 2024-03-06T07:27:33.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.486+0000 7f0c6cd4b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c68152d00 con 0x7f0c68155220 2024-03-06T07:27:33.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.486+0000 7f0c6cd4b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c681596d0 con 0x7f0c680c5850 2024-03-06T07:27:33.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.486+0000 7f0c66d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0c68155220 0x7f0c681556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.486+0000 7f0c66d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0c68155220 0x7f0c681556b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59402/0 (socket says 172.21.15.1:59402) 2024-03-06T07:27:33.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.486+0000 7f0c66d9d640 1 -- 172.21.15.1:0/1092697816 learned_addr learned my addr 172.21.15.1:0/1092697816 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:33.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.486+0000 7f0c5ffff640 1 --2- 172.21.15.1:0/1092697816 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c680c5850 0x7f0c680c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.486+0000 7f0c66d9d640 1 -- 172.21.15.1:0/1092697816 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c680c5850 msgr2=0x7f0c680c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.486+0000 7f0c66d9d640 1 --2- 172.21.15.1:0/1092697816 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c680c5850 0x7f0c680c5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.486+0000 7f0c66d9d640 1 -- 172.21.15.1:0/1092697816 >> v1:172.21.15.98:6789/0 conn(0x7f0c68152920 legacy=0x7f0c6806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.486+0000 7f0c66d9d640 1 -- 172.21.15.1:0/1092697816 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c68158bf0 con 0x7f0c68155220 2024-03-06T07:27:33.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.486+0000 7f0c5ffff640 1 --2- 172.21.15.1:0/1092697816 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c680c5850 0x7f0c680c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:33.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.487+0000 7f0c66d9d640 1 --2- 172.21.15.1:0/1092697816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c68155220 0x7f0c681556b0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009e50 tx=0x7f0c6002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60bd63d7e0d7ce44 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.487+0000 7f0c65d9b640 1 -- 172.21.15.1:0/1092697816 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6002c9e0 con 0x7f0c68155220 2024-03-06T07:27:33.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.487+0000 7f0c65d9b640 1 -- 172.21.15.1:0/1092697816 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6002cb80 con 0x7f0c68155220 2024-03-06T07:27:33.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.487+0000 7f0c65d9b640 1 -- 172.21.15.1:0/1092697816 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c600343f0 con 0x7f0c68155220 2024-03-06T07:27:33.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.488+0000 7f0c6cd4b640 1 -- 172.21.15.1:0/1092697816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c68155220 msgr2=0x7f0c681556b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.488+0000 7f0c6cd4b640 1 --2- 172.21.15.1:0/1092697816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c68155220 0x7f0c681556b0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009e50 tx=0x7f0c6002be40 comp rx=0 tx=0).stop 2024-03-06T07:27:33.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.488+0000 7f0c6cd4b640 1 -- 172.21.15.1:0/1092697816 shutdown_connections 2024-03-06T07:27:33.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.488+0000 7f0c6cd4b640 1 --2- 172.21.15.1:0/1092697816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c68155220 0x7f0c681556b0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.488+0000 7f0c6cd4b640 1 --2- 172.21.15.1:0/1092697816 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c680c5850 0x7f0c680c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.488+0000 7f0c6cd4b640 1 -- 172.21.15.1:0/1092697816 >> 172.21.15.1:0/1092697816 conn(0x7f0c680a9700 msgr2=0x7f0c680a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.488+0000 7f0c6cd4b640 1 -- 172.21.15.1:0/1092697816 shutdown_connections 2024-03-06T07:27:33.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.488+0000 7f0c6cd4b640 1 -- 172.21.15.1:0/1092697816 wait complete. 2024-03-06T07:27:33.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.489+0000 7f0c6cd4b640 1 Processor -- start 2024-03-06T07:27:33.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.489+0000 7f0c6cd4b640 1 -- start start 2024-03-06T07:27:33.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c6cd4b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c680c5850 0x7f0c680ad3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c6cd4b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c68152920 0x7f0c680c4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c6cd4b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c68155220 0x7f0c6814ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c6cd4b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c680c6380 con 0x7f0c68152920 2024-03-06T07:27:33.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c6cd4b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c68159240 con 0x7f0c68155220 2024-03-06T07:27:33.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c6cd4b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6806b700 con 0x7f0c680c5850 2024-03-06T07:27:33.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c5ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c68152920 0x7f0c680c4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c5ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c68152920 0x7f0c680c4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59418/0 (socket says 172.21.15.1:59418) 2024-03-06T07:27:33.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c5ffff640 1 -- 172.21.15.1:0/3216114344 learned_addr learned my addr 172.21.15.1:0/3216114344 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:33.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c6659c640 1 --2- 172.21.15.1:0/3216114344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c680c5850 0x7f0c680ad3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c66d9d640 1 --2- 172.21.15.1:0/3216114344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c68155220 0x7f0c6814ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c5ffff640 1 -- 172.21.15.1:0/3216114344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c680c5850 msgr2=0x7f0c680ad3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c5ffff640 1 --2- 172.21.15.1:0/3216114344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c680c5850 0x7f0c680ad3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c5ffff640 1 -- 172.21.15.1:0/3216114344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c68155220 msgr2=0x7f0c6814ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c5ffff640 1 --2- 172.21.15.1:0/3216114344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c68155220 0x7f0c6814ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c5ffff640 1 -- 172.21.15.1:0/3216114344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6814b820 con 0x7f0c68152920 2024-03-06T07:27:33.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c66d9d640 1 --2- 172.21.15.1:0/3216114344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c68155220 0x7f0c6814ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:33.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c5ffff640 1 --2- 172.21.15.1:0/3216114344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c68152920 0x7f0c680c4e10 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000c7f0 tx=0x7f0c5000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.490+0000 7f0c5effd640 1 -- 172.21.15.1:0/3216114344 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50015070 con 0x7f0c68152920 2024-03-06T07:27:33.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.491+0000 7f0c5effd640 1 -- 172.21.15.1:0/3216114344 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c500079d0 con 0x7f0c68152920 2024-03-06T07:27:33.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.491+0000 7f0c6cd4b640 1 -- 172.21.15.1:0/3216114344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c680c43a0 con 0x7f0c68152920 2024-03-06T07:27:33.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.491+0000 7f0c5effd640 1 -- 172.21.15.1:0/3216114344 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50007da0 con 0x7f0c68152920 2024-03-06T07:27:33.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.491+0000 7f0c6cd4b640 1 -- 172.21.15.1:0/3216114344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c68155ff0 con 0x7f0c68152920 2024-03-06T07:27:33.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.491+0000 7f0c6cd4b640 1 -- 172.21.15.1:0/3216114344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c68067fd0 con 0x7f0c68152920 2024-03-06T07:27:33.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.492+0000 7f0c5effd640 1 -- 172.21.15.1:0/3216114344 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c50018070 con 0x7f0c68152920 2024-03-06T07:27:33.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.492+0000 7f0c5effd640 1 --2- 172.21.15.1:0/3216114344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c4806b4d0 0x7f0c4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.493+0000 7f0c5effd640 1 -- 172.21.15.1:0/3216114344 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0c5008a160 con 0x7f0c68152920 2024-03-06T07:27:33.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.493+0000 7f0c6659c640 1 --2- 172.21.15.1:0/3216114344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c4806b4d0 0x7f0c4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.493+0000 7f0c6659c640 1 --2- 172.21.15.1:0/3216114344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c4806b4d0 0x7f0c4806d990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f0c6814ca90 tx=0x7f0c58007e30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.497+0000 7f0c5effd640 1 -- 172.21.15.1:0/3216114344 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c50058890 con 0x7f0c68152920 2024-03-06T07:27:33.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.716+0000 7f0c6cd4b640 1 -- 172.21.15.1:0/3216114344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c6806bdf0 con 0x7f0c68152920 2024-03-06T07:27:33.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.716+0000 7f0c5effd640 1 -- 172.21.15.1:0/3216114344 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f0c5005c930 con 0x7f0c68152920 2024-03-06T07:27:33.718 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:33.718 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":163,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:33.362862+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9738":{"gid":9738,"name":"c","rank":0,"incarnation":163,"state":"up:starting","state_seq":1,"addr":"172.21.15.1:6835/1021164277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1021164277},{"type":"v1","addr":"172.21.15.1:6835","nonce":1021164277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:27:33.719 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 163 2024-03-06T07:27:33.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.718+0000 7f0c6cd4b640 1 -- 172.21.15.1:0/3216114344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c4806b4d0 msgr2=0x7f0c4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.718+0000 7f0c6cd4b640 1 --2- 172.21.15.1:0/3216114344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c4806b4d0 0x7f0c4806d990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f0c6814ca90 tx=0x7f0c58007e30 comp rx=0 tx=0).stop 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.718+0000 7f0c6cd4b640 1 -- 172.21.15.1:0/3216114344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c68152920 msgr2=0x7f0c680c4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.718+0000 7f0c6cd4b640 1 --2- 172.21.15.1:0/3216114344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c68152920 0x7f0c680c4e10 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000c7f0 tx=0x7f0c5000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.718+0000 7f0c6cd4b640 1 -- 172.21.15.1:0/3216114344 shutdown_connections 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.718+0000 7f0c6cd4b640 1 --2- 172.21.15.1:0/3216114344 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c4806b4d0 0x7f0c4806d990 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.718+0000 7f0c6cd4b640 1 --2- 172.21.15.1:0/3216114344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c68155220 0x7f0c6814ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.718+0000 7f0c6cd4b640 1 --2- 172.21.15.1:0/3216114344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c68152920 0x7f0c680c4e10 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.718+0000 7f0c6cd4b640 1 --2- 172.21.15.1:0/3216114344 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c680c5850 0x7f0c680ad3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.718+0000 7f0c6cd4b640 1 -- 172.21.15.1:0/3216114344 >> 172.21.15.1:0/3216114344 conn(0x7f0c680a9700 msgr2=0x7f0c680c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.718+0000 7f0c6cd4b640 1 -- 172.21.15.1:0/3216114344 shutdown_connections 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.718+0000 7f0c6cd4b640 1 -- 172.21.15.1:0/3216114344 wait complete. 2024-03-06T07:27:33.731 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:33.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.830+0000 7fb0dced4640 1 Processor -- start 2024-03-06T07:27:33.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.830+0000 7fb0dced4640 1 -- start start 2024-03-06T07:27:33.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.831+0000 7fb0dced4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d8072f20 0x7fb0d81513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.831+0000 7fb0dced4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb0d8071f90 0x7fb0d8072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.831+0000 7fb0dced4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d80a6140 con 0x7fb0d8150fe0 2024-03-06T07:27:33.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.831+0000 7fb0dced4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d81526b0 con 0x7fb0d8071f90 2024-03-06T07:27:33.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.831+0000 7fb0dced4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d8151aa0 con 0x7fb0d8072f20 2024-03-06T07:27:33.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.831+0000 7fb0d5d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d8072f20 0x7fb0d81513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.831+0000 7fb0d659c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb0d8150fe0 0x7fb0d806edb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:33822/0 (socket says 172.21.15.1:33822) 2024-03-06T07:27:33.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.831+0000 7fb0d659c640 1 -- 172.21.15.1:0/1145104479 learned_addr learned my addr 172.21.15.1:0/1145104479 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:33.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.831+0000 7fb0d6d9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb0d8071f90 0x7fb0d8072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.831+0000 7fb0d5d9b640 1 -- 172.21.15.1:0/1145104479 >> v2:172.21.15.98:3300/0 conn(0x7fb0d8071f90 msgr2=0x7fb0d8072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.831+0000 7fb0d5d9b640 1 --2- 172.21.15.1:0/1145104479 >> v2:172.21.15.98:3300/0 conn(0x7fb0d8071f90 0x7fb0d8072420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.831+0000 7fb0d5d9b640 1 -- 172.21.15.1:0/1145104479 >> v1:172.21.15.98:6789/0 conn(0x7fb0d8150fe0 legacy=0x7fb0d806edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.831+0000 7fb0d5d9b640 1 -- 172.21.15.1:0/1145104479 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d8151d80 con 0x7fb0d8072f20 2024-03-06T07:27:33.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.832+0000 7fb0d5d9b640 1 --2- 172.21.15.1:0/1145104479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d8072f20 0x7fb0d81513c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0097a0 tx=0x7fb0cc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c189c5c2f20908bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.832+0000 7fb0d559a640 1 -- 172.21.15.1:0/1145104479 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc0315a0 con 0x7fb0d8072f20 2024-03-06T07:27:33.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.832+0000 7fb0d559a640 1 -- 172.21.15.1:0/1145104479 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0cc037070 con 0x7fb0d8072f20 2024-03-06T07:27:33.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.832+0000 7fb0d559a640 1 -- 172.21.15.1:0/1145104479 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc031960 con 0x7fb0d8072f20 2024-03-06T07:27:33.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.833+0000 7fb0dced4640 1 -- 172.21.15.1:0/1145104479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d8072f20 msgr2=0x7fb0d81513c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.833+0000 7fb0dced4640 1 --2- 172.21.15.1:0/1145104479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d8072f20 0x7fb0d81513c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0097a0 tx=0x7fb0cc02b840 comp rx=0 tx=0).stop 2024-03-06T07:27:33.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.833+0000 7fb0dced4640 1 -- 172.21.15.1:0/1145104479 shutdown_connections 2024-03-06T07:27:33.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.833+0000 7fb0dced4640 1 --2- 172.21.15.1:0/1145104479 >> v2:172.21.15.98:3300/0 conn(0x7fb0d8071f90 0x7fb0d8072420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.833+0000 7fb0dced4640 1 --2- 172.21.15.1:0/1145104479 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d8072f20 0x7fb0d81513c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.833+0000 7fb0dced4640 1 -- 172.21.15.1:0/1145104479 >> 172.21.15.1:0/1145104479 conn(0x7fb0d80c6860 msgr2=0x7fb0d80c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.833+0000 7fb0dced4640 1 -- 172.21.15.1:0/1145104479 shutdown_connections 2024-03-06T07:27:33.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.833+0000 7fb0dced4640 1 -- 172.21.15.1:0/1145104479 wait complete. 2024-03-06T07:27:33.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.834+0000 7fb0dced4640 1 Processor -- start 2024-03-06T07:27:33.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.834+0000 7fb0dced4640 1 -- start start 2024-03-06T07:27:33.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.835+0000 7fb0dced4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d8071f90 0x7fb0d80c4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.835+0000 7fb0dced4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d8072f20 0x7fb0d8149fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.835+0000 7fb0dced4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d8150fe0 0x7fb0d814a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.835+0000 7fb0dced4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d806f4c0 con 0x7fb0d8072f20 2024-03-06T07:27:33.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.835+0000 7fb0dced4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d8152340 con 0x7fb0d8071f90 2024-03-06T07:27:33.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.835+0000 7fb0dced4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d806f640 con 0x7fb0d8150fe0 2024-03-06T07:27:33.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.835+0000 7fb0d5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d8072f20 0x7fb0d8149fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.835+0000 7fb0d5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d8072f20 0x7fb0d8149fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59434/0 (socket says 172.21.15.1:59434) 2024-03-06T07:27:33.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.835+0000 7fb0d5d9b640 1 -- 172.21.15.1:0/3296290118 learned_addr learned my addr 172.21.15.1:0/3296290118 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:33.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.835+0000 7fb0d5d9b640 1 -- 172.21.15.1:0/3296290118 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d8150fe0 msgr2=0x7fb0d814a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.835+0000 7fb0d5d9b640 1 --2- 172.21.15.1:0/3296290118 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d8150fe0 0x7fb0d814a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.835+0000 7fb0d5d9b640 1 -- 172.21.15.1:0/3296290118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d8071f90 msgr2=0x7fb0d80c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.835+0000 7fb0d5d9b640 1 --2- 172.21.15.1:0/3296290118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d8071f90 0x7fb0d80c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.835+0000 7fb0d5d9b640 1 -- 172.21.15.1:0/3296290118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d80c5550 con 0x7fb0d8072f20 2024-03-06T07:27:33.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.836+0000 7fb0d5d9b640 1 --2- 172.21.15.1:0/3296290118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d8072f20 0x7fb0d8149fc0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0098c0 tx=0x7fb0cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.836+0000 7fb0c6ffd640 1 -- 172.21.15.1:0/3296290118 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc032430 con 0x7fb0d8072f20 2024-03-06T07:27:33.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.836+0000 7fb0dced4640 1 -- 172.21.15.1:0/3296290118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d81eca60 con 0x7fb0d8072f20 2024-03-06T07:27:33.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.836+0000 7fb0c6ffd640 1 -- 172.21.15.1:0/3296290118 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0cc0325f0 con 0x7fb0d8072f20 2024-03-06T07:27:33.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.836+0000 7fb0c6ffd640 1 -- 172.21.15.1:0/3296290118 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc0329e0 con 0x7fb0d8072f20 2024-03-06T07:27:33.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.836+0000 7fb0dced4640 1 -- 172.21.15.1:0/3296290118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d81425e0 con 0x7fb0d8072f20 2024-03-06T07:27:33.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.837+0000 7fb0dced4640 1 -- 172.21.15.1:0/3296290118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d8067fd0 con 0x7fb0d8072f20 2024-03-06T07:27:33.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.837+0000 7fb0c6ffd640 1 -- 172.21.15.1:0/3296290118 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0cc032b80 con 0x7fb0d8072f20 2024-03-06T07:27:33.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.838+0000 7fb0c6ffd640 1 --2- 172.21.15.1:0/3296290118 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0b806b4d0 0x7fb0b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.838+0000 7fb0c6ffd640 1 -- 172.21.15.1:0/3296290118 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb0cc02f030 con 0x7fb0d8072f20 2024-03-06T07:27:33.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.838+0000 7fb0d659c640 1 --2- 172.21.15.1:0/3296290118 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0b806b4d0 0x7fb0b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.838+0000 7fb0d659c640 1 --2- 172.21.15.1:0/3296290118 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0b806b4d0 0x7fb0b806d990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8000f60 tx=0x7fb0c8007a80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:33.842+0000 7fb0c6ffd640 1 -- 172.21.15.1:0/3296290118 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0cc040070 con 0x7fb0d8072f20 2024-03-06T07:27:34.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:34.058+0000 7fb0dced4640 1 -- 172.21.15.1:0/3296290118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0d8151460 con 0x7fb0d8072f20 2024-03-06T07:27:34.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:34.059+0000 7fb0c6ffd640 1 -- 172.21.15.1:0/3296290118 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+4964 (secure 0 0 0) 0x7fb0cc07ae70 con 0x7fb0d8072f20 2024-03-06T07:27:34.061 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:34.061 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":163,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:33.362862+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9738":{"gid":9738,"name":"c","rank":0,"incarnation":163,"state":"up:starting","state_seq":1,"addr":"172.21.15.1:6835/1021164277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1021164277},{"type":"v1","addr":"172.21.15.1:6835","nonce":1021164277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:27:34.061 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 163 2024-03-06T07:27:34.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:34.061+0000 7fb0dced4640 1 -- 172.21.15.1:0/3296290118 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0b806b4d0 msgr2=0x7fb0b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:34.061+0000 7fb0dced4640 1 --2- 172.21.15.1:0/3296290118 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0b806b4d0 0x7fb0b806d990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8000f60 tx=0x7fb0c8007a80 comp rx=0 tx=0).stop 2024-03-06T07:27:34.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:34.061+0000 7fb0dced4640 1 -- 172.21.15.1:0/3296290118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d8072f20 msgr2=0x7fb0d8149fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:34.061+0000 7fb0dced4640 1 --2- 172.21.15.1:0/3296290118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d8072f20 0x7fb0d8149fc0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0098c0 tx=0x7fb0cc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:34.061+0000 7fb0dced4640 1 -- 172.21.15.1:0/3296290118 shutdown_connections 2024-03-06T07:27:34.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:34.061+0000 7fb0dced4640 1 --2- 172.21.15.1:0/3296290118 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb0b806b4d0 0x7fb0b806d990 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:34.061+0000 7fb0dced4640 1 --2- 172.21.15.1:0/3296290118 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb0d8150fe0 0x7fb0d814a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:34.061+0000 7fb0dced4640 1 --2- 172.21.15.1:0/3296290118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0d8072f20 0x7fb0d8149fc0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:34.061+0000 7fb0dced4640 1 --2- 172.21.15.1:0/3296290118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d8071f90 0x7fb0d80c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:34.061+0000 7fb0dced4640 1 -- 172.21.15.1:0/3296290118 >> 172.21.15.1:0/3296290118 conn(0x7fb0d80c6860 msgr2=0x7fb0d80c6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:34.061+0000 7fb0dced4640 1 -- 172.21.15.1:0/3296290118 shutdown_connections 2024-03-06T07:27:34.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:34.061+0000 7fb0dced4640 1 -- 172.21.15.1:0/3296290118 wait complete. 2024-03-06T07:27:34.476 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 163, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:04.619091+0000', 'modified': '2024-03-06T07:27:33.362862+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9738}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9738': {'gid': 9738, 'name': 'c', 'rank': 0, 'incarnation': 163, 'state': 'up:starting', 'state_seq': 1, 'addr': '172.21.15.1:6835/1021164277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1021164277}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1021164277}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:34.476 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9738:up:starting 2024-03-06T07:27:35.477 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:35.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.580+0000 7f1dc3a8f640 1 Processor -- start 2024-03-06T07:27:35.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.580+0000 7f1dc3a8f640 1 -- start start 2024-03-06T07:27:35.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.581+0000 7f1dc3a8f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1dbc0bf9d0 0x7f1dbc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.581+0000 7f1dc3a8f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dbc0c2fc0 con 0x7f1dbc152650 2024-03-06T07:27:35.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc3a8f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dbc158da0 con 0x7f1dbc0c6b10 2024-03-06T07:27:35.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc3a8f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc155800 con 0x7f1dbc0bf9d0 2024-03-06T07:27:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc182b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1dbc152650 0x7f1dbc158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33960/0 (socket says 172.21.15.1:33960) 2024-03-06T07:27:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc182b640 1 -- 172.21.15.1:0/594329872 learned_addr learned my addr 172.21.15.1:0/594329872 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc0829640 1 -- 172.21.15.1:0/594329872 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828536149 0 0) 0x7f1dbc0c2fc0 con 0x7f1dbc152650 2024-03-06T07:27:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc202c640 1 --2- 172.21.15.1:0/594329872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1dbc0bf9d0 0x7f1dbc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc0829640 1 -- 172.21.15.1:0/594329872 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1da8003520 con 0x7f1dbc152650 2024-03-06T07:27:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc0829640 1 -- 172.21.15.1:0/594329872 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3127035489 0 0) 0x7f1da8003520 con 0x7f1dbc152650 2024-03-06T07:27:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc0829640 1 -- 172.21.15.1:0/594329872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1dbc0bf9d0 msgr2=0x7f1dbc155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc0829640 1 --2- 172.21.15.1:0/594329872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1dbc0bf9d0 0x7f1dbc155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc0829640 1 -- 172.21.15.1:0/594329872 >> v1:172.21.15.98:6789/0 conn(0x7f1dbc0c6b10 legacy=0x7f1dbc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc0829640 1 -- 172.21.15.1:0/594329872 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dbc0c6690 con 0x7f1dbc152650 2024-03-06T07:27:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc0829640 1 -- 172.21.15.1:0/594329872 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f1db8001690 con 0x7f1dbc152650 2024-03-06T07:27:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc0829640 1 -- 172.21.15.1:0/594329872 >> v1:172.21.15.1:6789/0 conn(0x7f1dbc152650 legacy=0x7f1dbc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc0829640 1 --2- 172.21.15.1:0/594329872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1dbc0bf9d0 0x7f1da80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc0829640 1 --2- 172.21.15.1:0/594329872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1da8003f60 0x7f1da8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc0829640 1 --2- 172.21.15.1:0/594329872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1da8004d40 0x7f1da80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc0829640 1 -- 172.21.15.1:0/594329872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc155800 con 0x7f1da8004d40 2024-03-06T07:27:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc0829640 1 -- 172.21.15.1:0/594329872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da800bb50 con 0x7f1da8003f60 2024-03-06T07:27:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc0829640 1 -- 172.21.15.1:0/594329872 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da800bd50 con 0x7f1dbc0bf9d0 2024-03-06T07:27:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.582+0000 7f1dc202c640 1 --2- 172.21.15.1:0/594329872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1da8004d40 0x7f1da80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.583+0000 7f1dc182b640 1 --2- 172.21.15.1:0/594329872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1dbc0bf9d0 0x7f1da80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.583+0000 7f1dc102a640 1 --2- 172.21.15.1:0/594329872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1da8003f60 0x7f1da8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.583+0000 7f1dc202c640 1 -- 172.21.15.1:0/594329872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1dbc0bf9d0 msgr2=0x7f1da80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.583+0000 7f1dc202c640 1 --2- 172.21.15.1:0/594329872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1dbc0bf9d0 0x7f1da80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.583+0000 7f1dc202c640 1 -- 172.21.15.1:0/594329872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1da8003f60 msgr2=0x7f1da8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.583+0000 7f1dc202c640 1 --2- 172.21.15.1:0/594329872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1da8003f60 0x7f1da8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.583+0000 7f1dc202c640 1 -- 172.21.15.1:0/594329872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da8003520 con 0x7f1da8004d40 2024-03-06T07:27:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.583+0000 7f1dc102a640 1 --2- 172.21.15.1:0/594329872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1da8003f60 0x7f1da8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.583+0000 7f1dc182b640 1 --2- 172.21.15.1:0/594329872 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1dbc0bf9d0 0x7f1da80039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.583+0000 7f1dc202c640 1 --2- 172.21.15.1:0/594329872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1da8004d40 0x7f1da80051c0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009fc0 tx=0x7f1da8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb2401d6257fce4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.584+0000 7f1dc0829640 1 -- 172.21.15.1:0/594329872 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db4039070 con 0x7f1da8004d40 2024-03-06T07:27:35.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.584+0000 7f1dc0829640 1 -- 172.21.15.1:0/594329872 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db402cc20 con 0x7f1da8004d40 2024-03-06T07:27:35.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.584+0000 7f1dc0829640 1 -- 172.21.15.1:0/594329872 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db4009c10 con 0x7f1da8004d40 2024-03-06T07:27:35.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.584+0000 7f1dc3a8f640 1 -- 172.21.15.1:0/594329872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1da8004d40 msgr2=0x7f1da80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.584+0000 7f1dc3a8f640 1 --2- 172.21.15.1:0/594329872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1da8004d40 0x7f1da80051c0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009fc0 tx=0x7f1da8005830 comp rx=0 tx=0).stop 2024-03-06T07:27:35.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.584+0000 7f1dc202c640 1 -- 172.21.15.1:0/594329872 reap_dead start 2024-03-06T07:27:35.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.584+0000 7f1dc3a8f640 1 -- 172.21.15.1:0/594329872 shutdown_connections 2024-03-06T07:27:35.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.584+0000 7f1dc3a8f640 1 -- 172.21.15.1:0/594329872 >> 172.21.15.1:0/594329872 conn(0x7f1dbc066ee0 msgr2=0x7f1dbc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:35.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.584+0000 7f1dc3a8f640 1 -- 172.21.15.1:0/594329872 shutdown_connections 2024-03-06T07:27:35.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.584+0000 7f1dc3a8f640 1 -- 172.21.15.1:0/594329872 wait complete. 2024-03-06T07:27:35.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.586+0000 7f1dc3a8f640 1 Processor -- start 2024-03-06T07:27:35.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.586+0000 7f1dc3a8f640 1 -- start start 2024-03-06T07:27:35.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc3a8f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1da8004d40 0x7f1dbc0c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc3a8f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dbc1ee050 0x7f1dbc1ee4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc3a8f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dbc14e380 0x7f1dbc14e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc3a8f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc1551b0 con 0x7f1dbc1ee050 2024-03-06T07:27:35.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc3a8f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc0c6fa0 con 0x7f1dbc14e380 2024-03-06T07:27:35.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc3a8f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc155490 con 0x7f1da8004d40 2024-03-06T07:27:35.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc102a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dbc1ee050 0x7f1dbc1ee4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc102a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dbc1ee050 0x7f1dbc1ee4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58756/0 (socket says 172.21.15.1:58756) 2024-03-06T07:27:35.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc102a640 1 -- 172.21.15.1:0/334743087 learned_addr learned my addr 172.21.15.1:0/334743087 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:35.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc182b640 1 --2- 172.21.15.1:0/334743087 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1da8004d40 0x7f1dbc0c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc202c640 1 --2- 172.21.15.1:0/334743087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dbc14e380 0x7f1dbc14e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc102a640 1 -- 172.21.15.1:0/334743087 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1da8004d40 msgr2=0x7f1dbc0c7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc102a640 1 --2- 172.21.15.1:0/334743087 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1da8004d40 0x7f1dbc0c7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc102a640 1 -- 172.21.15.1:0/334743087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dbc14e380 msgr2=0x7f1dbc14e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc102a640 1 --2- 172.21.15.1:0/334743087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dbc14e380 0x7f1dbc14e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc102a640 1 -- 172.21.15.1:0/334743087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dbc1586c0 con 0x7f1dbc1ee050 2024-03-06T07:27:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc202c640 1 --2- 172.21.15.1:0/334743087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dbc14e380 0x7f1dbc14e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dc102a640 1 --2- 172.21.15.1:0/334743087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dbc1ee050 0x7f1dbc1ee4d0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f1db000e500 tx=0x7f1db000e900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dae7fc640 1 -- 172.21.15.1:0/334743087 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db00052e0 con 0x7f1dbc1ee050 2024-03-06T07:27:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.587+0000 7f1dae7fc640 1 -- 172.21.15.1:0/334743087 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db0018070 con 0x7f1dbc1ee050 2024-03-06T07:27:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.588+0000 7f1dc3a8f640 1 -- 172.21.15.1:0/334743087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dbc1589a0 con 0x7f1dbc1ee050 2024-03-06T07:27:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.588+0000 7f1dae7fc640 1 -- 172.21.15.1:0/334743087 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db0003810 con 0x7f1dbc1ee050 2024-03-06T07:27:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.588+0000 7f1dc3a8f640 1 -- 172.21.15.1:0/334743087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dbc151490 con 0x7f1dbc1ee050 2024-03-06T07:27:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.589+0000 7f1dc3a8f640 1 -- 172.21.15.1:0/334743087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dbc150fa0 con 0x7f1dbc1ee050 2024-03-06T07:27:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.589+0000 7f1dae7fc640 1 -- 172.21.15.1:0/334743087 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1db00039b0 con 0x7f1dbc1ee050 2024-03-06T07:27:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.589+0000 7f1dae7fc640 1 --2- 172.21.15.1:0/334743087 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1da006b4d0 0x7f1da006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.589+0000 7f1dae7fc640 1 -- 172.21.15.1:0/334743087 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1db0089eb0 con 0x7f1dbc1ee050 2024-03-06T07:27:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.590+0000 7f1dc182b640 1 --2- 172.21.15.1:0/334743087 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1da006b4d0 0x7f1da006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.590+0000 7f1dc182b640 1 --2- 172.21.15.1:0/334743087 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1da006b4d0 0x7f1da006d990 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f1db8001800 tx=0x7f1db8007040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.598+0000 7f1dae7fc640 1 -- 172.21.15.1:0/334743087 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1db0058c30 con 0x7f1dbc1ee050 2024-03-06T07:27:35.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.817+0000 7f1dc3a8f640 1 -- 172.21.15.1:0/334743087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dbc067fd0 con 0x7f1dbc1ee050 2024-03-06T07:27:35.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.818+0000 7f1dae7fc640 1 -- 172.21.15.1:0/334743087 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+4969 (secure 0 0 0) 0x7f1db0058530 con 0x7f1dbc1ee050 2024-03-06T07:27:35.820 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:35.820 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":164,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:34.364010+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9738":{"gid":9738,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":2,"addr":"172.21.15.1:6835/1021164277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1021164277},{"type":"v1","addr":"172.21.15.1:6835","nonce":1021164277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9738,"qdb_cluster":[9738]},"id":13}]} 2024-03-06T07:27:35.820 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 164 2024-03-06T07:27:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.820+0000 7f1dc3a8f640 1 -- 172.21.15.1:0/334743087 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1da006b4d0 msgr2=0x7f1da006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.820+0000 7f1dc3a8f640 1 --2- 172.21.15.1:0/334743087 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1da006b4d0 0x7f1da006d990 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f1db8001800 tx=0x7f1db8007040 comp rx=0 tx=0).stop 2024-03-06T07:27:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.820+0000 7f1dc3a8f640 1 -- 172.21.15.1:0/334743087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dbc1ee050 msgr2=0x7f1dbc1ee4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.820+0000 7f1dc3a8f640 1 --2- 172.21.15.1:0/334743087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dbc1ee050 0x7f1dbc1ee4d0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f1db000e500 tx=0x7f1db000e900 comp rx=0 tx=0).stop 2024-03-06T07:27:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.820+0000 7f1dc3a8f640 1 -- 172.21.15.1:0/334743087 shutdown_connections 2024-03-06T07:27:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.820+0000 7f1dc3a8f640 1 --2- 172.21.15.1:0/334743087 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1da006b4d0 0x7f1da006d990 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.820+0000 7f1dc3a8f640 1 --2- 172.21.15.1:0/334743087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dbc14e380 0x7f1dbc14e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.820+0000 7f1dc3a8f640 1 --2- 172.21.15.1:0/334743087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dbc1ee050 0x7f1dbc1ee4d0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.820+0000 7f1dc3a8f640 1 --2- 172.21.15.1:0/334743087 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1da8004d40 0x7f1dbc0c7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.820+0000 7f1dc3a8f640 1 -- 172.21.15.1:0/334743087 >> 172.21.15.1:0/334743087 conn(0x7f1dbc066ee0 msgr2=0x7f1dbc0cac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.820+0000 7f1dc3a8f640 1 -- 172.21.15.1:0/334743087 shutdown_connections 2024-03-06T07:27:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:35.820+0000 7f1dc3a8f640 1 -- 172.21.15.1:0/334743087 wait complete. 2024-03-06T07:27:36.235 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 164, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:04.619091+0000', 'modified': '2024-03-06T07:27:34.364010+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9738}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9738': {'gid': 9738, 'name': 'c', 'rank': 0, 'incarnation': 163, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.1:6835/1021164277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1021164277}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1021164277}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9738, 'qdb_cluster': [9738]} 2024-03-06T07:27:36.236 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:36.236 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:27:36.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.337+0000 7fe838b1e640 1 Processor -- start 2024-03-06T07:27:36.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.337+0000 7fe838b1e640 1 -- start start 2024-03-06T07:27:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.338+0000 7fe838b1e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe834072bd0 0x7fe8340c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.338+0000 7fe838b1e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe834151dd0 con 0x7fe8340c2b30 2024-03-06T07:27:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.338+0000 7fe838b1e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe834156190 con 0x7fe834152b70 2024-03-06T07:27:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.338+0000 7fe838b1e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8340c3f40 con 0x7fe834072bd0 2024-03-06T07:27:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.338+0000 7fe829d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe834072bd0 0x7fe8340c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.338+0000 7fe829d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe834072bd0 0x7fe8340c32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58766/0 (socket says 172.21.15.1:58766) 2024-03-06T07:27:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.338+0000 7fe829d9b640 1 -- 172.21.15.1:0/1617051428 learned_addr learned my addr 172.21.15.1:0/1617051428 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.338+0000 7fe83259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe8340c2b30 0x7fe8340724c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33966/0 (socket says 172.21.15.1:33966) 2024-03-06T07:27:36.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.338+0000 7fe829d9b640 1 -- 172.21.15.1:0/1617051428 >> v1:172.21.15.98:6789/0 conn(0x7fe834152b70 legacy=0x7fe8340c3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.338+0000 7fe829d9b640 1 -- 172.21.15.1:0/1617051428 >> v1:172.21.15.1:6789/0 conn(0x7fe8340c2b30 legacy=0x7fe8340724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.338+0000 7fe829d9b640 1 -- 172.21.15.1:0/1617051428 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe834152090 con 0x7fe834072bd0 2024-03-06T07:27:36.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.338+0000 7fe831d9b640 1 -- 172.21.15.1:0/1617051428 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332408320 0 0) 0x7fe834151dd0 con 0x7fe8340c2b30 2024-03-06T07:27:36.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.339+0000 7fe829d9b640 1 --2- 172.21.15.1:0/1617051428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe834072bd0 0x7fe8340c32f0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009870 tx=0x7fe81c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=901e8917c8fd4850 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.339+0000 7fe831d9b640 1 -- 172.21.15.1:0/1617051428 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c034070 con 0x7fe834072bd0 2024-03-06T07:27:36.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.339+0000 7fe831d9b640 1 -- 172.21.15.1:0/1617051428 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81c030860 con 0x7fe834072bd0 2024-03-06T07:27:36.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.339+0000 7fe831d9b640 1 -- 172.21.15.1:0/1617051428 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c030c30 con 0x7fe834072bd0 2024-03-06T07:27:36.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.340+0000 7fe838b1e640 1 -- 172.21.15.1:0/1617051428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe834072bd0 msgr2=0x7fe8340c32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.340+0000 7fe838b1e640 1 --2- 172.21.15.1:0/1617051428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe834072bd0 0x7fe8340c32f0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009870 tx=0x7fe81c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.340+0000 7fe838b1e640 1 -- 172.21.15.1:0/1617051428 shutdown_connections 2024-03-06T07:27:36.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.340+0000 7fe838b1e640 1 --2- 172.21.15.1:0/1617051428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe834072bd0 0x7fe8340c32f0 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.340+0000 7fe838b1e640 1 -- 172.21.15.1:0/1617051428 >> 172.21.15.1:0/1617051428 conn(0x7fe8340c6860 msgr2=0x7fe8340c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:36.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.340+0000 7fe838b1e640 1 -- 172.21.15.1:0/1617051428 shutdown_connections 2024-03-06T07:27:36.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.340+0000 7fe838b1e640 1 -- 172.21.15.1:0/1617051428 wait complete. 2024-03-06T07:27:36.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.341+0000 7fe838b1e640 1 Processor -- start 2024-03-06T07:27:36.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.341+0000 7fe838b1e640 1 -- start start 2024-03-06T07:27:36.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.342+0000 7fe838b1e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe834072bd0 0x7fe8341dd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.342+0000 7fe838b1e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8340c2b30 0x7fe8341439e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.342+0000 7fe838b1e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe834152b70 0x7fe834143f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.342+0000 7fe838b1e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe834155d80 con 0x7fe834152b70 2024-03-06T07:27:36.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.342+0000 7fe838b1e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8341526e0 con 0x7fe8340c2b30 2024-03-06T07:27:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.342+0000 7fe838b1e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe834155f00 con 0x7fe834072bd0 2024-03-06T07:27:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.342+0000 7fe832d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe834152b70 0x7fe834143f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.342+0000 7fe832d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe834152b70 0x7fe834143f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58780/0 (socket says 172.21.15.1:58780) 2024-03-06T07:27:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.342+0000 7fe832d9d640 1 -- 172.21.15.1:0/4259380899 learned_addr learned my addr 172.21.15.1:0/4259380899 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.343+0000 7fe83259c640 1 --2- 172.21.15.1:0/4259380899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe834072bd0 0x7fe8341dd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.343+0000 7fe829d9b640 1 --2- 172.21.15.1:0/4259380899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8340c2b30 0x7fe8341439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.343+0000 7fe832d9d640 1 -- 172.21.15.1:0/4259380899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe834072bd0 msgr2=0x7fe8341dd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.343+0000 7fe832d9d640 1 --2- 172.21.15.1:0/4259380899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe834072bd0 0x7fe8341dd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.343+0000 7fe832d9d640 1 -- 172.21.15.1:0/4259380899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8340c2b30 msgr2=0x7fe8341439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.343+0000 7fe832d9d640 1 --2- 172.21.15.1:0/4259380899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8340c2b30 0x7fe8341439e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.343+0000 7fe832d9d640 1 -- 172.21.15.1:0/4259380899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8341e03d0 con 0x7fe834152b70 2024-03-06T07:27:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.343+0000 7fe83259c640 1 --2- 172.21.15.1:0/4259380899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe834072bd0 0x7fe8341dd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.343+0000 7fe832d9d640 1 --2- 172.21.15.1:0/4259380899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe834152b70 0x7fe834143f20 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fe82c00c700 tx=0x7fe82c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.343+0000 7fe82b7fe640 1 -- 172.21.15.1:0/4259380899 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c016020 con 0x7fe834152b70 2024-03-06T07:27:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.343+0000 7fe82b7fe640 1 -- 172.21.15.1:0/4259380899 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82c00aa50 con 0x7fe834152b70 2024-03-06T07:27:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.343+0000 7fe82b7fe640 1 -- 172.21.15.1:0/4259380899 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c00ae40 con 0x7fe834152b70 2024-03-06T07:27:36.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.343+0000 7fe838b1e640 1 -- 172.21.15.1:0/4259380899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe834150af0 con 0x7fe834152b70 2024-03-06T07:27:36.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.343+0000 7fe838b1e640 1 -- 172.21.15.1:0/4259380899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8341ef4d0 con 0x7fe834152b70 2024-03-06T07:27:36.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.345+0000 7fe82b7fe640 1 -- 172.21.15.1:0/4259380899 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe82c019020 con 0x7fe834152b70 2024-03-06T07:27:36.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.345+0000 7fe82b7fe640 1 --2- 172.21.15.1:0/4259380899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe81406b4d0 0x7fe81406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.345+0000 7fe82b7fe640 1 -- 172.21.15.1:0/4259380899 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe82c08a550 con 0x7fe834152b70 2024-03-06T07:27:36.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.346+0000 7fe83259c640 1 --2- 172.21.15.1:0/4259380899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe81406b4d0 0x7fe81406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.346+0000 7fe83b7e7740 1 Processor -- start 2024-03-06T07:27:36.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.346+0000 7fe83b7e7740 1 -- start start 2024-03-06T07:27:36.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.346+0000 7fe83259c640 1 --2- 172.21.15.1:0/4259380899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe81406b4d0 0x7fe81406d990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fe824000f60 tx=0x7fe824009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.346+0000 7fe83b7e7740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a7ff50 0x55e534a7da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.346+0000 7fe83b7e7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a80330 0x55e534a7df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.346+0000 7fe83b7e7740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a7e4b0 0x55e534a7a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.346+0000 7fe83b7e7740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5349e46b0 con 0x55e534a7ff50 2024-03-06T07:27:36.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.346+0000 7fe83b7e7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e53498da60 con 0x55e534a80330 2024-03-06T07:27:36.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.346+0000 7fe83b7e7740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e5349e0750 con 0x55e534a7e4b0 2024-03-06T07:27:36.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.347+0000 7fe83259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a7ff50 0x55e534a7da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.347+0000 7fe83259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a7ff50 0x55e534a7da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58792/0 (socket says 172.21.15.1:58792) 2024-03-06T07:27:36.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.347+0000 7fe83259c640 1 -- 172.21.15.1:0/2942220693 learned_addr learned my addr 172.21.15.1:0/2942220693 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:36.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.347+0000 7fe832d9d640 1 --2- 172.21.15.1:0/2942220693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a7e4b0 0x55e534a7a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.347+0000 7fe829d9b640 1 --2- 172.21.15.1:0/2942220693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a80330 0x55e534a7df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.347+0000 7fe83259c640 1 -- 172.21.15.1:0/2942220693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a7e4b0 msgr2=0x55e534a7a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.347+0000 7fe83259c640 1 --2- 172.21.15.1:0/2942220693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a7e4b0 0x55e534a7a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.347+0000 7fe83259c640 1 -- 172.21.15.1:0/2942220693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a80330 msgr2=0x55e534a7df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.347+0000 7fe83259c640 1 --2- 172.21.15.1:0/2942220693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a80330 0x55e534a7df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.347+0000 7fe83259c640 1 -- 172.21.15.1:0/2942220693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5349abf30 con 0x55e534a7ff50 2024-03-06T07:27:36.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.347+0000 7fe829d9b640 1 --2- 172.21.15.1:0/2942220693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a80330 0x55e534a7df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:36.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.347+0000 7fe83259c640 1 --2- 172.21.15.1:0/2942220693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a7ff50 0x55e534a7da30 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fe824007360 tx=0x7fe824010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2071135d3dd0389 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.347+0000 7fe813fff640 1 -- 172.21.15.1:0/2942220693 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82400dbe0 con 0x55e534a7ff50 2024-03-06T07:27:36.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.347+0000 7fe813fff640 1 -- 172.21.15.1:0/2942220693 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe824001400 con 0x55e534a7ff50 2024-03-06T07:27:36.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.347+0000 7fe813fff640 1 -- 172.21.15.1:0/2942220693 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe824020450 con 0x55e534a7ff50 2024-03-06T07:27:36.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.348+0000 7fe83b7e7740 1 -- 172.21.15.1:0/2942220693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a7ff50 msgr2=0x55e534a7da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.348+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/2942220693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a7ff50 0x55e534a7da30 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fe824007360 tx=0x7fe824010000 comp rx=0 tx=0).stop 2024-03-06T07:27:36.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.348+0000 7fe83b7e7740 1 -- 172.21.15.1:0/2942220693 shutdown_connections 2024-03-06T07:27:36.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.348+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/2942220693 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a7e4b0 0x55e534a7a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.348+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/2942220693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a80330 0x55e534a7df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.348+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/2942220693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a7ff50 0x55e534a7da30 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.348+0000 7fe83b7e7740 1 -- 172.21.15.1:0/2942220693 >> 172.21.15.1:0/2942220693 conn(0x55e534a7c7d0 msgr2=0x55e534aa13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:36.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.348+0000 7fe83b7e7740 1 -- 172.21.15.1:0/2942220693 shutdown_connections 2024-03-06T07:27:36.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.348+0000 7fe83b7e7740 1 -- 172.21.15.1:0/2942220693 wait complete. 2024-03-06T07:27:36.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.348+0000 7fe83b7e7740 1 -- start start 2024-03-06T07:27:36.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.349+0000 7fe83b7e7740 1 Processor -- start 2024-03-06T07:27:36.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.349+0000 7fe83b7e7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a7ff50 0x55e534aa0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.349+0000 7fe83b7e7740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a80330 0x55e534acff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.349+0000 7fe83b7e7740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a9cec0 0x55e534a9d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.349+0000 7fe83b7e7740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e534aa2230 con 0x55e534a9cec0 2024-03-06T07:27:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.349+0000 7fe83b7e7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5349a0ea0 con 0x55e534a7ff50 2024-03-06T07:27:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.349+0000 7fe83b7e7740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e534aa1c70 con 0x55e534a80330 2024-03-06T07:27:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.349+0000 7fe83b7e7740 10 client.0 _ll_register_callbacks cb 0x55e5349a0870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe832d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a9cec0 0x55e534a9d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe832d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a9cec0 0x55e534a9d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58800/0 (socket says 172.21.15.1:58800) 2024-03-06T07:27:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe832d9d640 1 -- 172.21.15.1:0/3084523059 learned_addr learned my addr 172.21.15.1:0/3084523059 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe829d9b640 1 --2- 172.21.15.1:0/3084523059 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a80330 0x55e534acff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe832d9d640 1 -- 172.21.15.1:0/3084523059 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a80330 msgr2=0x55e534acff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe832d9d640 1 --2- 172.21.15.1:0/3084523059 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a80330 0x55e534acff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe832d9d640 1 -- 172.21.15.1:0/3084523059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a7ff50 msgr2=0x55e534aa0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe83259c640 1 --2- 172.21.15.1:0/3084523059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a7ff50 0x55e534aa0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe832d9d640 1 --2- 172.21.15.1:0/3084523059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a7ff50 0x55e534aa0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe832d9d640 1 -- 172.21.15.1:0/3084523059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5349a1270 con 0x55e534a9cec0 2024-03-06T07:27:36.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe83259c640 1 --2- 172.21.15.1:0/3084523059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a7ff50 0x55e534aa0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:36.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe832d9d640 1 --2- 172.21.15.1:0/3084523059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a9cec0 0x55e534a9d340 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fe82c00a430 tx=0x7fe82c05ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc27131362114c53 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe8117fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:27:36.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe8117fa640 1 -- 172.21.15.1:0/3084523059 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c070070 con 0x55e534a9cec0 2024-03-06T07:27:36.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe8117fa640 1 -- 172.21.15.1:0/3084523059 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe82c06cd90 con 0x55e534a9cec0 2024-03-06T07:27:36.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe83b7e7740 1 -- 172.21.15.1:0/3084523059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe824001400 con 0x55e534a9cec0 2024-03-06T07:27:36.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe8117fa640 1 -- 172.21.15.1:0/3084523059 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=164) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe82c057040 con 0x55e534a9cec0 2024-03-06T07:27:36.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe83b7e7740 10 client.9786 fetch_fsmap learned FSMap version 164 2024-03-06T07:27:36.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe83b7e7740 1 -- 172.21.15.1:0/3084523059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=164,monmap=2+}) v3 -- 0x55e5346f92d0 con 0x55e534a9cec0 2024-03-06T07:27:36.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.350+0000 7fe8117fa640 1 -- 172.21.15.1:0/3084523059 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe824001400 con 0x55e534a9cec0 2024-03-06T07:27:36.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.351+0000 7fe8117fa640 1 -- 172.21.15.1:0/3084523059 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 164) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe82c076050 con 0x55e534a9cec0 2024-03-06T07:27:36.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.351+0000 7fe83b7e7740 10 client.9786 fetch_fsmap finished waiting for FSMap version 164 2024-03-06T07:27:36.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.351+0000 7fe83b7e7740 10 client.9786 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:36.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.351+0000 7fe83b7e7740 20 client.9786 populate_metadata read hostname 'smithi001' 2024-03-06T07:27:36.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.351+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/3084523059 >> [v2:172.21.15.1:6833/1021164277,v1:172.21.15.1:6835/1021164277] conn(0x55e534ad9a90 0x55e534adbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.351+0000 7fe83b7e7740 4 client.9786 mds_command: new command op to 9738 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:36.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.351+0000 7fe83b7e7740 1 -- 172.21.15.1:0/3084523059 --> [v2:172.21.15.1:6833/1021164277,v1:172.21.15.1:6835/1021164277] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e5349f1bf0 con 0x55e534ad9a90 2024-03-06T07:27:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.351+0000 7fe83259c640 1 --2- 172.21.15.1:0/3084523059 >> [v2:172.21.15.1:6833/1021164277,v1:172.21.15.1:6835/1021164277] conn(0x55e534ad9a90 0x55e534adbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.351+0000 7fe8117fa640 10 client.9786 ms_handle_connect on v2:172.21.15.1:6833/1021164277 2024-03-06T07:27:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.351+0000 7fe83259c640 1 --2- 172.21.15.1:0/3084523059 >> [v2:172.21.15.1:6833/1021164277,v1:172.21.15.1:6835/1021164277] conn(0x55e534ad9a90 0x55e534adbe80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1770eda810420442 server_cookie=a6b37b3e07f5121b in_seq=0 out_seq=0 2024-03-06T07:27:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.352+0000 7fe8117fa640 1 -- 172.21.15.1:0/3084523059 <== mds.0 v2:172.21.15.1:6833/1021164277 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e5349f1bf0 con 0x55e534ad9a90 2024-03-06T07:27:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.352+0000 7fe8117fa640 10 client.9786 handle_command_reply: tid=0 2024-03-06T07:27:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.352+0000 7fe83b7e7740 1 client.9786 shutdown 2024-03-06T07:27:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.352+0000 7fe83b7e7740 1 -- 172.21.15.1:0/3084523059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a9cec0 msgr2=0x55e534a9d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.352+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/3084523059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a9cec0 0x55e534a9d340 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fe82c00a430 tx=0x7fe82c05ce60 comp rx=0 tx=0).stop 2024-03-06T07:27:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.352+0000 7fe83b7e7740 1 -- 172.21.15.1:0/3084523059 shutdown_connections 2024-03-06T07:27:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.352+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/3084523059 >> [v2:172.21.15.1:6833/1021164277,v1:172.21.15.1:6835/1021164277] conn(0x55e534ad9a90 0x55e534adbe80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.352+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/3084523059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a9cec0 0x55e534a9d340 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.352+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/3084523059 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a80330 0x55e534acff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.352+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/3084523059 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a7ff50 0x55e534aa0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.352+0000 7fe8117fa640 0 client.9786 ms_handle_reset on v2:172.21.15.1:6833/1021164277 2024-03-06T07:27:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.352+0000 7fe83b7e7740 1 -- 172.21.15.1:0/3084523059 >> 172.21.15.1:0/3084523059 conn(0x55e534a7e4b0 msgr2=0x55e534abcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.352+0000 7fe83b7e7740 1 -- 172.21.15.1:0/3084523059 shutdown_connections 2024-03-06T07:27:36.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.352+0000 7fe83b7e7740 1 -- 172.21.15.1:0/3084523059 wait complete. 2024-03-06T07:27:36.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.352+0000 7fe83b7e7740 20 client.9786 trim_cache size 0 max 16384 2024-03-06T07:27:36.360 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-06T07:27:36.357+0000 7f476c47d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:36.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.369+0000 7fe83b7e7740 1 Processor -- start 2024-03-06T07:27:36.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.369+0000 7fe83b7e7740 1 -- start start 2024-03-06T07:27:36.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe83b7e7740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a7ff50 0x55e534aa2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe83b7e7740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a80330 0x55e534a9fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe83b7e7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a9cec0 0x55e534aa0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe83b7e7740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8341f7d70 con 0x55e534a7ff50 2024-03-06T07:27:36.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe83b7e7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8341f6f10 con 0x55e534a9cec0 2024-03-06T07:27:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe83b7e7740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8341ebe50 con 0x55e534a80330 2024-03-06T07:27:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe83259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a7ff50 0x55e534aa2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe83259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a7ff50 0x55e534aa2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58812/0 (socket says 172.21.15.1:58812) 2024-03-06T07:27:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe83259c640 1 -- 172.21.15.1:0/2748934882 learned_addr learned my addr 172.21.15.1:0/2748934882 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe829d9b640 1 --2- 172.21.15.1:0/2748934882 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a80330 0x55e534a9fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe832d9d640 1 --2- 172.21.15.1:0/2748934882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a9cec0 0x55e534aa0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe83259c640 1 -- 172.21.15.1:0/2748934882 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a80330 msgr2=0x55e534a9fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe83259c640 1 --2- 172.21.15.1:0/2748934882 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a80330 0x55e534a9fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe83259c640 1 -- 172.21.15.1:0/2748934882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a9cec0 msgr2=0x55e534aa0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe83259c640 1 --2- 172.21.15.1:0/2748934882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a9cec0 0x55e534aa0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe83259c640 1 -- 172.21.15.1:0/2748934882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5347c36b0 con 0x55e534a7ff50 2024-03-06T07:27:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe83259c640 1 --2- 172.21.15.1:0/2748934882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a7ff50 0x55e534aa2990 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fe824013000 tx=0x7fe824007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb01df5084121e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe8137fe640 1 -- 172.21.15.1:0/2748934882 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82402f270 con 0x55e534a7ff50 2024-03-06T07:27:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe8137fe640 1 -- 172.21.15.1:0/2748934882 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe824034070 con 0x55e534a7ff50 2024-03-06T07:27:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.370+0000 7fe8137fe640 1 -- 172.21.15.1:0/2748934882 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82400de00 con 0x55e534a7ff50 2024-03-06T07:27:36.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.371+0000 7fe83b7e7740 1 -- 172.21.15.1:0/2748934882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a7ff50 msgr2=0x55e534aa2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.371+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/2748934882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a7ff50 0x55e534aa2990 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fe824013000 tx=0x7fe824007000 comp rx=0 tx=0).stop 2024-03-06T07:27:36.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.371+0000 7fe83b7e7740 1 -- 172.21.15.1:0/2748934882 shutdown_connections 2024-03-06T07:27:36.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.371+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/2748934882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a9cec0 0x55e534aa0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.371+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/2748934882 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a80330 0x55e534a9fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.371+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/2748934882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a7ff50 0x55e534aa2990 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.371+0000 7fe83b7e7740 1 -- 172.21.15.1:0/2748934882 >> 172.21.15.1:0/2748934882 conn(0x55e534a7e4b0 msgr2=0x55e534aa4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:36.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.371+0000 7fe83b7e7740 1 -- 172.21.15.1:0/2748934882 shutdown_connections 2024-03-06T07:27:36.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.371+0000 7fe83b7e7740 1 -- 172.21.15.1:0/2748934882 wait complete. 2024-03-06T07:27:36.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.372+0000 7fe83b7e7740 1 -- start start 2024-03-06T07:27:36.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.372+0000 7fe83b7e7740 1 Processor -- start 2024-03-06T07:27:36.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.373+0000 7fe83b7e7740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a7ff50 0x55e534aa6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.373+0000 7fe83b7e7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a80330 0x55e534a7ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.373+0000 7fe83b7e7740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a9cec0 0x55e534a7d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.373+0000 7fe83b7e7740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e534ab9cc0 con 0x55e534a9cec0 2024-03-06T07:27:36.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.373+0000 7fe83b7e7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8340c0980 con 0x55e534a80330 2024-03-06T07:27:36.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.373+0000 7fe83b7e7740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e534acd1e0 con 0x55e534a7ff50 2024-03-06T07:27:36.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.373+0000 7fe83b7e7740 10 client.0 _ll_register_callbacks cb 0x7fe82c058c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:36.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.373+0000 7fe832d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a9cec0 0x55e534a7d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.373+0000 7fe832d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a9cec0 0x55e534a7d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58816/0 (socket says 172.21.15.1:58816) 2024-03-06T07:27:36.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.373+0000 7fe832d9d640 1 -- 172.21.15.1:0/4069732996 learned_addr learned my addr 172.21.15.1:0/4069732996 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:36.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.373+0000 7fe83259c640 1 --2- 172.21.15.1:0/4069732996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a7ff50 0x55e534aa6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.373+0000 7fe829d9b640 1 --2- 172.21.15.1:0/4069732996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a80330 0x55e534a7ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.374+0000 7fe832d9d640 1 -- 172.21.15.1:0/4069732996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a7ff50 msgr2=0x55e534aa6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.374+0000 7fe832d9d640 1 --2- 172.21.15.1:0/4069732996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a7ff50 0x55e534aa6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.374+0000 7fe832d9d640 1 -- 172.21.15.1:0/4069732996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a80330 msgr2=0x55e534a7ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.374+0000 7fe832d9d640 1 --2- 172.21.15.1:0/4069732996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a80330 0x55e534a7ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.374+0000 7fe832d9d640 1 -- 172.21.15.1:0/4069732996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e534ac02d0 con 0x55e534a9cec0 2024-03-06T07:27:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.374+0000 7fe832d9d640 1 --2- 172.21.15.1:0/4069732996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a9cec0 0x55e534a7d860 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fe82c003ff0 tx=0x7fe82c088850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32888cf57b9b1699 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.374+0000 7fe8117fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:27:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.374+0000 7fe8117fa640 1 -- 172.21.15.1:0/4069732996 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c070040 con 0x55e534a9cec0 2024-03-06T07:27:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.374+0000 7fe8117fa640 1 -- 172.21.15.1:0/4069732996 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe82c082020 con 0x55e534a9cec0 2024-03-06T07:27:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.374+0000 7fe83b7e7740 1 -- 172.21.15.1:0/4069732996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe824034070 con 0x55e534a9cec0 2024-03-06T07:27:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.374+0000 7fe8117fa640 1 -- 172.21.15.1:0/4069732996 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=164) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe82c08a360 con 0x55e534a9cec0 2024-03-06T07:27:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.374+0000 7fe83b7e7740 10 client.9792 fetch_fsmap learned FSMap version 164 2024-03-06T07:27:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.374+0000 7fe83b7e7740 1 -- 172.21.15.1:0/4069732996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=164,monmap=2+}) v3 -- 0x55e534b2bb10 con 0x55e534a9cec0 2024-03-06T07:27:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.374+0000 7fe8117fa640 1 -- 172.21.15.1:0/4069732996 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe824034070 con 0x55e534a9cec0 2024-03-06T07:27:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.375+0000 7fe8117fa640 1 -- 172.21.15.1:0/4069732996 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 164) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe82c076070 con 0x55e534a9cec0 2024-03-06T07:27:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.375+0000 7fe83b7e7740 10 client.9792 fetch_fsmap finished waiting for FSMap version 164 2024-03-06T07:27:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.375+0000 7fe83b7e7740 10 client.9792 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.375+0000 7fe83b7e7740 20 client.9792 populate_metadata read hostname 'smithi001' 2024-03-06T07:27:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.375+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/4069732996 >> [v2:172.21.15.1:6833/1021164277,v1:172.21.15.1:6835/1021164277] conn(0x55e534b294c0 0x55e534b2f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.375+0000 7fe83b7e7740 4 client.9792 mds_command: new command op to 9738 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.375+0000 7fe83b7e7740 1 -- 172.21.15.1:0/4069732996 --> [v2:172.21.15.1:6833/1021164277,v1:172.21.15.1:6835/1021164277] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e5349ebf40 con 0x55e534b294c0 2024-03-06T07:27:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.375+0000 7fe83259c640 1 --2- 172.21.15.1:0/4069732996 >> [v2:172.21.15.1:6833/1021164277,v1:172.21.15.1:6835/1021164277] conn(0x55e534b294c0 0x55e534b2f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.375+0000 7fe8117fa640 10 client.9792 ms_handle_connect on v2:172.21.15.1:6833/1021164277 2024-03-06T07:27:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.375+0000 7fe83259c640 1 --2- 172.21.15.1:0/4069732996 >> [v2:172.21.15.1:6833/1021164277,v1:172.21.15.1:6835/1021164277] conn(0x55e534b294c0 0x55e534b2f080 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ace3c2e986969287 server_cookie=7f17a516d6524842 in_seq=0 out_seq=0 2024-03-06T07:27:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.375+0000 7fe8117fa640 1 -- 172.21.15.1:0/4069732996 <== mds.0 v2:172.21.15.1:6833/1021164277 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55e5349ebf40 con 0x55e534b294c0 2024-03-06T07:27:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.375+0000 7fe8117fa640 10 client.9792 handle_command_reply: tid=0 2024-03-06T07:27:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.375+0000 7fe83b7e7740 1 client.9792 shutdown 2024-03-06T07:27:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.375+0000 7fe83b7e7740 1 -- 172.21.15.1:0/4069732996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a9cec0 msgr2=0x55e534a7d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.375+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/4069732996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a9cec0 0x55e534a7d860 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fe82c003ff0 tx=0x7fe82c088850 comp rx=0 tx=0).stop 2024-03-06T07:27:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe83b7e7740 1 -- 172.21.15.1:0/4069732996 shutdown_connections 2024-03-06T07:27:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/4069732996 >> [v2:172.21.15.1:6833/1021164277,v1:172.21.15.1:6835/1021164277] conn(0x55e534b294c0 0x55e534b2f080 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/4069732996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e534a9cec0 0x55e534a7d860 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/4069732996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e534a80330 0x55e534a7ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe83b7e7740 1 --2- 172.21.15.1:0/4069732996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e534a7ff50 0x55e534aa6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe8117fa640 0 client.9792 ms_handle_reset on v2:172.21.15.1:6833/1021164277 2024-03-06T07:27:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe83b7e7740 1 -- 172.21.15.1:0/4069732996 >> 172.21.15.1:0/4069732996 conn(0x55e534a7e4b0 msgr2=0x55e534aa2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe83b7e7740 1 -- 172.21.15.1:0/4069732996 shutdown_connections 2024-03-06T07:27:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe83b7e7740 1 -- 172.21.15.1:0/4069732996 wait complete. 2024-03-06T07:27:36.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe83b7e7740 20 client.9792 trim_cache size 0 max 16384 2024-03-06T07:27:36.382 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:27:36.383 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:27:36.383 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:27:36.383 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 9738, 2024-03-06T07:27:36.383 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:27:36.383 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:27:36.383 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:27:36.383 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 3.0121814040000001, 2024-03-06T07:27:36.383 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 164, 2024-03-06T07:27:36.383 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 171, 2024-03-06T07:27:36.383 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 171, 2024-03-06T07:27:36.383 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 6.0011314530000002 2024-03-06T07:27:36.383 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:27:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe838b1e640 1 -- 172.21.15.1:0/4259380899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe81406b4d0 msgr2=0x7fe81406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe838b1e640 1 --2- 172.21.15.1:0/4259380899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe81406b4d0 0x7fe81406d990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fe824000f60 tx=0x7fe824009040 comp rx=0 tx=0).stop 2024-03-06T07:27:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe838b1e640 1 -- 172.21.15.1:0/4259380899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe834152b70 msgr2=0x7fe834143f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe838b1e640 1 --2- 172.21.15.1:0/4259380899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe834152b70 0x7fe834143f20 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fe82c00c700 tx=0x7fe82c00cbc0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe838b1e640 1 -- 172.21.15.1:0/4259380899 shutdown_connections 2024-03-06T07:27:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe838b1e640 1 --2- 172.21.15.1:0/4259380899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe81406b4d0 0x7fe81406d990 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe838b1e640 1 --2- 172.21.15.1:0/4259380899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe834152b70 0x7fe834143f20 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe838b1e640 1 --2- 172.21.15.1:0/4259380899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8340c2b30 0x7fe8341439e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe838b1e640 1 --2- 172.21.15.1:0/4259380899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe834072bd0 0x7fe8341dd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe838b1e640 1 -- 172.21.15.1:0/4259380899 >> 172.21.15.1:0/4259380899 conn(0x7fe8340c6860 msgr2=0x7fe8340c3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:36.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe838b1e640 1 -- 172.21.15.1:0/4259380899 shutdown_connections 2024-03-06T07:27:36.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.376+0000 7fe838b1e640 1 -- 172.21.15.1:0/4259380899 wait complete. 2024-03-06T07:27:36.389 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:36.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.488+0000 7f154d81c640 1 Processor -- start 2024-03-06T07:27:36.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.488+0000 7f154d81c640 1 -- start start 2024-03-06T07:27:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.489+0000 7f154d81c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1548152690 0x7f1548155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.489+0000 7f154d81c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15480c6730 0x7f15480c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.489+0000 7f154d81c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15481561e0 con 0x7f15480c5d10 2024-03-06T07:27:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.489+0000 7f154d81c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15480c3300 con 0x7f1548152690 2024-03-06T07:27:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.489+0000 7f154d81c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15480c35a0 con 0x7f15480c6730 2024-03-06T07:27:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.489+0000 7f15467fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15480c6730 0x7f15480c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.489+0000 7f1546ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1548152690 0x7f1548155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.489+0000 7f15467fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15480c6730 0x7f15480c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:44496/0 (socket says 172.21.15.1:44496) 2024-03-06T07:27:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.489+0000 7f15467fc640 1 -- 172.21.15.1:0/4278885558 learned_addr learned my addr 172.21.15.1:0/4278885558 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.489+0000 7f15477fe640 1 --1- 172.21.15.1:0/4278885558 >> v1:172.21.15.98:6789/0 conn(0x7f15480c5d10 0x7f1548156cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:37650/0 (socket says 172.21.15.1:37650) 2024-03-06T07:27:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.489+0000 7f1546ffd640 1 -- 172.21.15.1:0/4278885558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15480c6730 msgr2=0x7f15480c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.489+0000 7f1546ffd640 1 --2- 172.21.15.1:0/4278885558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15480c6730 0x7f15480c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.489+0000 7f1546ffd640 1 -- 172.21.15.1:0/4278885558 >> v1:172.21.15.98:6789/0 conn(0x7f15480c5d10 legacy=0x7f1548156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.490+0000 7f1546ffd640 1 -- 172.21.15.1:0/4278885558 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15480c3880 con 0x7f1548152690 2024-03-06T07:27:36.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.489+0000 7f1545ffb640 1 -- 172.21.15.1:0/4278885558 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999372374 0 0) 0x7f1538001490 con 0x7f15480c5d10 2024-03-06T07:27:36.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.490+0000 7f15467fc640 1 --2- 172.21.15.1:0/4278885558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15480c6730 0x7f15480c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:36.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.490+0000 7f1546ffd640 1 --2- 172.21.15.1:0/4278885558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1548152690 0x7f1548155ca0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f15400097a0 tx=0x7f154002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3431108631a45337 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.490+0000 7f1545ffb640 1 -- 172.21.15.1:0/4278885558 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1540031780 con 0x7f1548152690 2024-03-06T07:27:36.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.491+0000 7f1545ffb640 1 -- 172.21.15.1:0/4278885558 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1540037070 con 0x7f1548152690 2024-03-06T07:27:36.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.491+0000 7f1545ffb640 1 -- 172.21.15.1:0/4278885558 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1540031b40 con 0x7f1548152690 2024-03-06T07:27:36.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.491+0000 7f154d81c640 1 -- 172.21.15.1:0/4278885558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1548152690 msgr2=0x7f1548155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.491+0000 7f154d81c640 1 --2- 172.21.15.1:0/4278885558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1548152690 0x7f1548155ca0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f15400097a0 tx=0x7f154002b820 comp rx=0 tx=0).stop 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.491+0000 7f154d81c640 1 -- 172.21.15.1:0/4278885558 shutdown_connections 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.491+0000 7f154d81c640 1 --2- 172.21.15.1:0/4278885558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15480c6730 0x7f15480c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.491+0000 7f154d81c640 1 --2- 172.21.15.1:0/4278885558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1548152690 0x7f1548155ca0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.491+0000 7f154d81c640 1 -- 172.21.15.1:0/4278885558 >> 172.21.15.1:0/4278885558 conn(0x7f154806b410 msgr2=0x7f154806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.491+0000 7f154d81c640 1 -- 172.21.15.1:0/4278885558 shutdown_connections 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.491+0000 7f154d81c640 1 -- 172.21.15.1:0/4278885558 wait complete. 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.492+0000 7f154d81c640 1 Processor -- start 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.492+0000 7f154d81c640 1 -- start start 2024-03-06T07:27:36.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.493+0000 7f154d81c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15480c5d10 0x7f154814fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.493+0000 7f154d81c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15480c6730 0x7f15481ee480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.493+0000 7f154d81c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1548152690 0x7f15481ee9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.493+0000 7f154d81c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1548152a70 con 0x7f15480c6730 2024-03-06T07:27:36.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.493+0000 7f154d81c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15481567b0 con 0x7f1548152690 2024-03-06T07:27:36.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.493+0000 7f154d81c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1548152bf0 con 0x7f15480c5d10 2024-03-06T07:27:36.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.493+0000 7f15467fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15480c6730 0x7f15481ee480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.493+0000 7f15467fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15480c6730 0x7f15481ee480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58824/0 (socket says 172.21.15.1:58824) 2024-03-06T07:27:36.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.493+0000 7f15467fc640 1 -- 172.21.15.1:0/3004388220 learned_addr learned my addr 172.21.15.1:0/3004388220 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:36.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.493+0000 7f1546ffd640 1 --2- 172.21.15.1:0/3004388220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15480c5d10 0x7f154814fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.493+0000 7f15477fe640 1 --2- 172.21.15.1:0/3004388220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1548152690 0x7f15481ee9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.493+0000 7f15467fc640 1 -- 172.21.15.1:0/3004388220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15480c5d10 msgr2=0x7f154814fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.493+0000 7f15467fc640 1 --2- 172.21.15.1:0/3004388220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15480c5d10 0x7f154814fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.493+0000 7f15467fc640 1 -- 172.21.15.1:0/3004388220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1548152690 msgr2=0x7f15481ee9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.493+0000 7f15467fc640 1 --2- 172.21.15.1:0/3004388220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1548152690 0x7f15481ee9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.493+0000 7f15467fc640 1 -- 172.21.15.1:0/3004388220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15480add70 con 0x7f15480c6730 2024-03-06T07:27:36.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.493+0000 7f15477fe640 1 --2- 172.21.15.1:0/3004388220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1548152690 0x7f15481ee9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:36.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.494+0000 7f15467fc640 1 --2- 172.21.15.1:0/3004388220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15480c6730 0x7f15481ee480 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f153c00e9e0 tx=0x7f153c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.494+0000 7f152ffff640 1 -- 172.21.15.1:0/3004388220 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c017070 con 0x7f15480c6730 2024-03-06T07:27:36.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.494+0000 7f154d81c640 1 -- 172.21.15.1:0/3004388220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15480c76a0 con 0x7f15480c6730 2024-03-06T07:27:36.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.494+0000 7f152ffff640 1 -- 172.21.15.1:0/3004388220 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153c00c9d0 con 0x7f15480c6730 2024-03-06T07:27:36.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.494+0000 7f152ffff640 1 -- 172.21.15.1:0/3004388220 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c00cd80 con 0x7f15480c6730 2024-03-06T07:27:36.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.494+0000 7f154d81c640 1 -- 172.21.15.1:0/3004388220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154806cae0 con 0x7f15480c6730 2024-03-06T07:27:36.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.495+0000 7f154d81c640 1 -- 172.21.15.1:0/3004388220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15480c2eb0 con 0x7f15480c6730 2024-03-06T07:27:36.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.496+0000 7f152ffff640 1 -- 172.21.15.1:0/3004388220 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f153c01a070 con 0x7f15480c6730 2024-03-06T07:27:36.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.496+0000 7f152ffff640 1 --2- 172.21.15.1:0/3004388220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f153406b500 0x7f153406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.496+0000 7f152ffff640 1 -- 172.21.15.1:0/3004388220 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f153c08a5a0 con 0x7f15480c6730 2024-03-06T07:27:36.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.496+0000 7f1546ffd640 1 --2- 172.21.15.1:0/3004388220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f153406b500 0x7f153406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.497+0000 7f1546ffd640 1 --2- 172.21.15.1:0/3004388220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f153406b500 0x7f153406d9c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f1540036000 tx=0x7f154003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.501+0000 7f152ffff640 1 -- 172.21.15.1:0/3004388220 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f15480c2eb0 con 0x7f15480c6730 2024-03-06T07:27:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.723+0000 7f154d81c640 1 -- 172.21.15.1:0/3004388220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15480c2c40 con 0x7f15480c6730 2024-03-06T07:27:36.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.724+0000 7f152ffff640 1 -- 172.21.15.1:0/3004388220 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+4969 (secure 0 0 0) 0x7f15480add70 con 0x7f15480c6730 2024-03-06T07:27:36.726 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:36.726 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":164,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:34.364010+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9738":{"gid":9738,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":2,"addr":"172.21.15.1:6835/1021164277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1021164277},{"type":"v1","addr":"172.21.15.1:6835","nonce":1021164277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9738,"qdb_cluster":[9738]},"id":13}]} 2024-03-06T07:27:36.726 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 164 2024-03-06T07:27:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.726+0000 7f154d81c640 1 -- 172.21.15.1:0/3004388220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f153406b500 msgr2=0x7f153406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.726+0000 7f154d81c640 1 --2- 172.21.15.1:0/3004388220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f153406b500 0x7f153406d9c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f1540036000 tx=0x7f154003d040 comp rx=0 tx=0).stop 2024-03-06T07:27:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.726+0000 7f154d81c640 1 -- 172.21.15.1:0/3004388220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15480c6730 msgr2=0x7f15481ee480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.726+0000 7f154d81c640 1 --2- 172.21.15.1:0/3004388220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15480c6730 0x7f15481ee480 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f153c00e9e0 tx=0x7f153c00eea0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.726+0000 7f154d81c640 1 -- 172.21.15.1:0/3004388220 shutdown_connections 2024-03-06T07:27:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.726+0000 7f154d81c640 1 --2- 172.21.15.1:0/3004388220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f153406b500 0x7f153406d9c0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.726+0000 7f154d81c640 1 --2- 172.21.15.1:0/3004388220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1548152690 0x7f15481ee9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.726+0000 7f154d81c640 1 --2- 172.21.15.1:0/3004388220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15480c6730 0x7f15481ee480 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.726+0000 7f154d81c640 1 --2- 172.21.15.1:0/3004388220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15480c5d10 0x7f154814fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.726+0000 7f154d81c640 1 -- 172.21.15.1:0/3004388220 >> 172.21.15.1:0/3004388220 conn(0x7f154806b410 msgr2=0x7f15481508f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.726+0000 7f154d81c640 1 -- 172.21.15.1:0/3004388220 shutdown_connections 2024-03-06T07:27:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.726+0000 7f154d81c640 1 -- 172.21.15.1:0/3004388220 wait complete. 2024-03-06T07:27:36.739 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:27:36.739 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:36.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:36.826 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:27:36.827 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:27:36.827 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:36.841 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:27:36.841 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:27:36.841 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:36.884 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:27:36.884 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.983+0000 7f8739e39640 1 Processor -- start 2024-03-06T07:27:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.983+0000 7f8739e39640 1 -- start start 2024-03-06T07:27:36.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.984+0000 7f8739e39640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8734152820 0x7f87340c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.984+0000 7f8739e39640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f87340af6f0 0x7f873415fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.984+0000 7f8739e39640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87340c7180 con 0x7f87340c0670 2024-03-06T07:27:36.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.984+0000 7f8739e39640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8734155460 con 0x7f8734152820 2024-03-06T07:27:36.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.984+0000 7f8739e39640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87340c6bc0 con 0x7f87340af6f0 2024-03-06T07:27:36.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.985+0000 7f87337fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8734152820 0x7f87340c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.985+0000 7f87337fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8734152820 0x7f87340c0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58840/0 (socket says 172.21.15.1:58840) 2024-03-06T07:27:36.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.985+0000 7f87337fe640 1 -- 172.21.15.1:0/3399764572 learned_addr learned my addr 172.21.15.1:0/3399764572 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:36.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.985+0000 7f8733fff640 1 --2- 172.21.15.1:0/3399764572 >> v2:172.21.15.98:3300/0 conn(0x7f87340af6f0 0x7f873415fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.985+0000 7f87337fe640 1 -- 172.21.15.1:0/3399764572 >> v2:172.21.15.98:3300/0 conn(0x7f87340af6f0 msgr2=0x7f873415fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.985+0000 7f87337fe640 1 --2- 172.21.15.1:0/3399764572 >> v2:172.21.15.98:3300/0 conn(0x7f87340af6f0 0x7f873415fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.985+0000 7f87337fe640 1 -- 172.21.15.1:0/3399764572 >> v1:172.21.15.98:6789/0 conn(0x7f87340c0670 legacy=0x7f87340c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.985+0000 7f87337fe640 1 -- 172.21.15.1:0/3399764572 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87340c6ea0 con 0x7f8734152820 2024-03-06T07:27:36.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.986+0000 7f87337fe640 1 --2- 172.21.15.1:0/3399764572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8734152820 0x7f87340c0130 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f873406c030 tx=0x7f872402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9543ffacf17f921 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.986+0000 7f87327fc640 1 -- 172.21.15.1:0/3399764572 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8724035020 con 0x7f8734152820 2024-03-06T07:27:36.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.986+0000 7f87327fc640 1 -- 172.21.15.1:0/3399764572 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8724031870 con 0x7f8734152820 2024-03-06T07:27:36.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.986+0000 7f87327fc640 1 -- 172.21.15.1:0/3399764572 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8724031c40 con 0x7f8734152820 2024-03-06T07:27:36.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.986+0000 7f8739e39640 1 -- 172.21.15.1:0/3399764572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8734152820 msgr2=0x7f87340c0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.986+0000 7f8739e39640 1 --2- 172.21.15.1:0/3399764572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8734152820 0x7f87340c0130 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f873406c030 tx=0x7f872402b910 comp rx=0 tx=0).stop 2024-03-06T07:27:36.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.986+0000 7f8739e39640 1 -- 172.21.15.1:0/3399764572 shutdown_connections 2024-03-06T07:27:36.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.986+0000 7f8739e39640 1 --2- 172.21.15.1:0/3399764572 >> v2:172.21.15.98:3300/0 conn(0x7f87340af6f0 0x7f873415fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.986+0000 7f8739e39640 1 --2- 172.21.15.1:0/3399764572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8734152820 0x7f87340c0130 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.986+0000 7f8739e39640 1 -- 172.21.15.1:0/3399764572 >> 172.21.15.1:0/3399764572 conn(0x7f8734072720 msgr2=0x7f8734072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:36.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.986+0000 7f8739e39640 1 -- 172.21.15.1:0/3399764572 shutdown_connections 2024-03-06T07:27:36.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.986+0000 7f8739e39640 1 -- 172.21.15.1:0/3399764572 wait complete. 2024-03-06T07:27:36.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.988+0000 7f8739e39640 1 Processor -- start 2024-03-06T07:27:36.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.988+0000 7f8739e39640 1 -- start start 2024-03-06T07:27:36.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.989+0000 7f8739e39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87340af6f0 0x7f87340aac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.989+0000 7f8739e39640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87340c0670 0x7f87341584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.989+0000 7f8739e39640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8734152820 0x7f873414f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.989+0000 7f8739e39640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87340c7c80 con 0x7f8734152820 2024-03-06T07:27:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.989+0000 7f8739e39640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87340c7b00 con 0x7f87340af6f0 2024-03-06T07:27:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.989+0000 7f8739e39640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8734152c00 con 0x7f87340c0670 2024-03-06T07:27:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.989+0000 7f8733fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8734152820 0x7f873414f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.990+0000 7f8733fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8734152820 0x7f873414f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58844/0 (socket says 172.21.15.1:58844) 2024-03-06T07:27:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.990+0000 7f8733fff640 1 -- 172.21.15.1:0/648773282 learned_addr learned my addr 172.21.15.1:0/648773282 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.990+0000 7f8732ffd640 1 --2- 172.21.15.1:0/648773282 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87340c0670 0x7f87341584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.990+0000 7f87337fe640 1 --2- 172.21.15.1:0/648773282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87340af6f0 0x7f87340aac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.990+0000 7f8733fff640 1 -- 172.21.15.1:0/648773282 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87340c0670 msgr2=0x7f87341584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.990+0000 7f8733fff640 1 --2- 172.21.15.1:0/648773282 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87340c0670 0x7f87341584e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.990+0000 7f8733fff640 1 -- 172.21.15.1:0/648773282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87340af6f0 msgr2=0x7f87340aac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.990+0000 7f8733fff640 1 --2- 172.21.15.1:0/648773282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87340af6f0 0x7f87340aac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.990+0000 7f8733fff640 1 -- 172.21.15.1:0/648773282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8734154810 con 0x7f8734152820 2024-03-06T07:27:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.990+0000 7f8732ffd640 1 --2- 172.21.15.1:0/648773282 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87340c0670 0x7f87341584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.990+0000 7f87337fe640 1 --2- 172.21.15.1:0/648773282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87340af6f0 0x7f87340aac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.990+0000 7f8733fff640 1 --2- 172.21.15.1:0/648773282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8734152820 0x7f873414f3c0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f872c00cda0 tx=0x7f872c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.990+0000 7f871bfff640 1 -- 172.21.15.1:0/648773282 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c003b90 con 0x7f8734152820 2024-03-06T07:27:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.990+0000 7f871bfff640 1 -- 172.21.15.1:0/648773282 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872c003d30 con 0x7f8734152820 2024-03-06T07:27:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.990+0000 7f8739e39640 1 -- 172.21.15.1:0/648773282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8734159680 con 0x7f8734152820 2024-03-06T07:27:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.990+0000 7f871bfff640 1 -- 172.21.15.1:0/648773282 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c004240 con 0x7f8734152820 2024-03-06T07:27:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.990+0000 7f8739e39640 1 -- 172.21.15.1:0/648773282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8734157a00 con 0x7f8734152820 2024-03-06T07:27:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.991+0000 7f8739e39640 1 -- 172.21.15.1:0/648773282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87340686e0 con 0x7f8734152820 2024-03-06T07:27:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.992+0000 7f871bfff640 1 -- 172.21.15.1:0/648773282 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f872c0043e0 con 0x7f8734152820 2024-03-06T07:27:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.992+0000 7f871bfff640 1 --2- 172.21.15.1:0/648773282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f872006b4d0 0x7f872006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.992+0000 7f871bfff640 1 -- 172.21.15.1:0/648773282 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f872c08af10 con 0x7f8734152820 2024-03-06T07:27:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.992+0000 7f87337fe640 1 --2- 172.21.15.1:0/648773282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f872006b4d0 0x7f872006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:36.993+0000 7f87337fe640 1 --2- 172.21.15.1:0/648773282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f872006b4d0 0x7f872006d990 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f8724036000 tx=0x7f872403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.001+0000 7f871bfff640 1 -- 172.21.15.1:0/648773282 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f872c01d340 con 0x7f8734152820 2024-03-06T07:27:37.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.219+0000 7f8739e39640 1 -- 172.21.15.1:0/648773282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87340c5b40 con 0x7f8734152820 2024-03-06T07:27:37.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.219+0000 7f871bfff640 1 -- 172.21.15.1:0/648773282 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+4969 (secure 0 0 0) 0x7f872c059c00 con 0x7f8734152820 2024-03-06T07:27:37.222 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:37.222 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":164,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:34.364010+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9738":{"gid":9738,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":2,"addr":"172.21.15.1:6835/1021164277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1021164277},{"type":"v1","addr":"172.21.15.1:6835","nonce":1021164277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9738,"qdb_cluster":[9738]},"id":13}]} 2024-03-06T07:27:37.222 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 164 2024-03-06T07:27:37.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.221+0000 7f8739e39640 1 -- 172.21.15.1:0/648773282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f872006b4d0 msgr2=0x7f872006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.221+0000 7f8739e39640 1 --2- 172.21.15.1:0/648773282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f872006b4d0 0x7f872006d990 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f8724036000 tx=0x7f872403c040 comp rx=0 tx=0).stop 2024-03-06T07:27:37.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.221+0000 7f8739e39640 1 -- 172.21.15.1:0/648773282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8734152820 msgr2=0x7f873414f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.221+0000 7f8739e39640 1 --2- 172.21.15.1:0/648773282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8734152820 0x7f873414f3c0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f872c00cda0 tx=0x7f872c00a660 comp rx=0 tx=0).stop 2024-03-06T07:27:37.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.221+0000 7f8739e39640 1 -- 172.21.15.1:0/648773282 shutdown_connections 2024-03-06T07:27:37.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.221+0000 7f8739e39640 1 --2- 172.21.15.1:0/648773282 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f872006b4d0 0x7f872006d990 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.221+0000 7f8739e39640 1 --2- 172.21.15.1:0/648773282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8734152820 0x7f873414f3c0 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.221+0000 7f8739e39640 1 --2- 172.21.15.1:0/648773282 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87340c0670 0x7f87341584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.221+0000 7f8739e39640 1 --2- 172.21.15.1:0/648773282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87340af6f0 0x7f87340aac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.221+0000 7f8739e39640 1 -- 172.21.15.1:0/648773282 >> 172.21.15.1:0/648773282 conn(0x7f8734072720 msgr2=0x7f87340c73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.221+0000 7f8739e39640 1 -- 172.21.15.1:0/648773282 shutdown_connections 2024-03-06T07:27:37.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.222+0000 7f8739e39640 1 -- 172.21.15.1:0/648773282 wait complete. 2024-03-06T07:27:37.234 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:37.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.353+0000 7f4d5bf93640 1 Processor -- start 2024-03-06T07:27:37.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.353+0000 7f4d5bf93640 1 -- start start 2024-03-06T07:27:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.353+0000 7f4d5bf93640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d54152730 0x7f4d540c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.353+0000 7f4d5bf93640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4d540c6fd0 0x7f4d540c1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.353+0000 7f4d5bf93640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d541588e0 con 0x7f4d540c18d0 2024-03-06T07:27:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.353+0000 7f4d5bf93640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d540c1d20 con 0x7f4d540c6fd0 2024-03-06T07:27:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.353+0000 7f4d5bf93640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d54158bc0 con 0x7f4d54152730 2024-03-06T07:27:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.354+0000 7f4d59d2f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d54152730 0x7f4d540c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.354+0000 7f4d59d2f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d54152730 0x7f4d540c6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:44504/0 (socket says 172.21.15.1:44504) 2024-03-06T07:27:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.354+0000 7f4d59d2f640 1 -- 172.21.15.1:0/230248943 learned_addr learned my addr 172.21.15.1:0/230248943 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.354+0000 7f4d5952e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4d540c6fd0 0x7f4d540c1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.354+0000 7f4d59d2f640 1 -- 172.21.15.1:0/230248943 >> v2:172.21.15.98:3300/0 conn(0x7f4d540c6fd0 msgr2=0x7f4d540c1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.354+0000 7f4d59d2f640 1 --2- 172.21.15.1:0/230248943 >> v2:172.21.15.98:3300/0 conn(0x7f4d540c6fd0 0x7f4d540c1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.354+0000 7f4d59d2f640 1 -- 172.21.15.1:0/230248943 >> v1:172.21.15.98:6789/0 conn(0x7f4d540c18d0 legacy=0x7f4d541552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.354+0000 7f4d59d2f640 1 -- 172.21.15.1:0/230248943 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d54158ea0 con 0x7f4d54152730 2024-03-06T07:27:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.355+0000 7f4d59d2f640 1 --2- 172.21.15.1:0/230248943 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d54152730 0x7f4d540c6a90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009870 tx=0x7f4d5002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6c710d0f3293c36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.355+0000 7f4d58d2d640 1 -- 172.21.15.1:0/230248943 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d500317c0 con 0x7f4d54152730 2024-03-06T07:27:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.355+0000 7f4d58d2d640 1 -- 172.21.15.1:0/230248943 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d50037070 con 0x7f4d54152730 2024-03-06T07:27:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.355+0000 7f4d58d2d640 1 -- 172.21.15.1:0/230248943 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50031b80 con 0x7f4d54152730 2024-03-06T07:27:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.356+0000 7f4d5bf93640 1 -- 172.21.15.1:0/230248943 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d54152730 msgr2=0x7f4d540c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.356+0000 7f4d5bf93640 1 --2- 172.21.15.1:0/230248943 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d54152730 0x7f4d540c6a90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009870 tx=0x7f4d5002b860 comp rx=0 tx=0).stop 2024-03-06T07:27:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.356+0000 7f4d5bf93640 1 -- 172.21.15.1:0/230248943 shutdown_connections 2024-03-06T07:27:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.356+0000 7f4d5bf93640 1 --2- 172.21.15.1:0/230248943 >> v2:172.21.15.98:3300/0 conn(0x7f4d540c6fd0 0x7f4d540c1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.356+0000 7f4d5bf93640 1 --2- 172.21.15.1:0/230248943 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d54152730 0x7f4d540c6a90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.356+0000 7f4d5bf93640 1 -- 172.21.15.1:0/230248943 >> 172.21.15.1:0/230248943 conn(0x7f4d54066d00 msgr2=0x7f4d54067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.356+0000 7f4d5bf93640 1 -- 172.21.15.1:0/230248943 shutdown_connections 2024-03-06T07:27:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.356+0000 7f4d5bf93640 1 -- 172.21.15.1:0/230248943 wait complete. 2024-03-06T07:27:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.358+0000 7f4d5bf93640 1 Processor -- start 2024-03-06T07:27:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.358+0000 7f4d5bf93640 1 -- start start 2024-03-06T07:27:37.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.359+0000 7f4d5bf93640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d540c18d0 0x7f4d541e1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.359+0000 7f4d5bf93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d540c6fd0 0x7f4d54150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.359+0000 7f4d5bf93640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d54152730 0x7f4d540c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.359+0000 7f4d5bf93640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d54152b10 con 0x7f4d540c6fd0 2024-03-06T07:27:37.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.359+0000 7f4d5bf93640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d54159460 con 0x7f4d54152730 2024-03-06T07:27:37.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.359+0000 7f4d5bf93640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d541559f0 con 0x7f4d540c18d0 2024-03-06T07:27:37.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.359+0000 7f4d5952e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d540c6fd0 0x7f4d54150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.359+0000 7f4d5952e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d540c6fd0 0x7f4d54150a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58856/0 (socket says 172.21.15.1:58856) 2024-03-06T07:27:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.359+0000 7f4d5952e640 1 -- 172.21.15.1:0/10684846 learned_addr learned my addr 172.21.15.1:0/10684846 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.359+0000 7f4d59d2f640 1 --2- 172.21.15.1:0/10684846 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d540c18d0 0x7f4d541e1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.359+0000 7f4d5a530640 1 --2- 172.21.15.1:0/10684846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d54152730 0x7f4d540c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.359+0000 7f4d5952e640 1 -- 172.21.15.1:0/10684846 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d540c18d0 msgr2=0x7f4d541e1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.359+0000 7f4d5952e640 1 --2- 172.21.15.1:0/10684846 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d540c18d0 0x7f4d541e1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.359+0000 7f4d5952e640 1 -- 172.21.15.1:0/10684846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d54152730 msgr2=0x7f4d540c8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.359+0000 7f4d5952e640 1 --2- 172.21.15.1:0/10684846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d54152730 0x7f4d540c8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.359+0000 7f4d5952e640 1 -- 172.21.15.1:0/10684846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d541e1d40 con 0x7f4d540c6fd0 2024-03-06T07:27:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.359+0000 7f4d5952e640 1 --2- 172.21.15.1:0/10684846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d540c6fd0 0x7f4d54150a10 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400e9b0 tx=0x7f4d4400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.360+0000 7f4d4a7fc640 1 -- 172.21.15.1:0/10684846 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d44017070 con 0x7f4d540c6fd0 2024-03-06T07:27:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.360+0000 7f4d4a7fc640 1 -- 172.21.15.1:0/10684846 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4400c9a0 con 0x7f4d540c6fd0 2024-03-06T07:27:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.360+0000 7f4d4a7fc640 1 -- 172.21.15.1:0/10684846 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4400cd90 con 0x7f4d540c6fd0 2024-03-06T07:27:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.360+0000 7f4d5bf93640 1 -- 172.21.15.1:0/10684846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d5414e930 con 0x7f4d540c6fd0 2024-03-06T07:27:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.360+0000 7f4d5bf93640 1 -- 172.21.15.1:0/10684846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d54150f50 con 0x7f4d540c6fd0 2024-03-06T07:27:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.361+0000 7f4d5bf93640 1 -- 172.21.15.1:0/10684846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d54067fd0 con 0x7f4d540c6fd0 2024-03-06T07:27:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.361+0000 7f4d4a7fc640 1 -- 172.21.15.1:0/10684846 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d4401a070 con 0x7f4d540c6fd0 2024-03-06T07:27:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.362+0000 7f4d4a7fc640 1 --2- 172.21.15.1:0/10684846 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4d3806b4d0 0x7f4d3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.362+0000 7f4d4a7fc640 1 -- 172.21.15.1:0/10684846 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4d4408a670 con 0x7f4d540c6fd0 2024-03-06T07:27:37.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.362+0000 7f4d59d2f640 1 --2- 172.21.15.1:0/10684846 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4d3806b4d0 0x7f4d3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.362+0000 7f4d59d2f640 1 --2- 172.21.15.1:0/10684846 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4d3806b4d0 0x7f4d3806d990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4d50036000 tx=0x7f4d5003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.366+0000 7f4d4a7fc640 1 -- 172.21.15.1:0/10684846 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d44058d10 con 0x7f4d540c6fd0 2024-03-06T07:27:37.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.550+0000 7f4d5bf93640 1 -- 172.21.15.1:0/10684846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d540c6870 con 0x7f4d540c6fd0 2024-03-06T07:27:37.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.551+0000 7f4d4a7fc640 1 -- 172.21.15.1:0/10684846 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+16832 (secure 0 0 0) 0x7f4d4405cdb0 con 0x7f4d540c6fd0 2024-03-06T07:27:37.553 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:37.553 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":171,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:27:06.270883+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:02.888292+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:03.804423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:37.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.553+0000 7f4d5bf93640 1 -- 172.21.15.1:0/10684846 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4d3806b4d0 msgr2=0x7f4d3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.553+0000 7f4d5bf93640 1 --2- 172.21.15.1:0/10684846 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4d3806b4d0 0x7f4d3806d990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4d50036000 tx=0x7f4d5003d040 comp rx=0 tx=0).stop 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.553+0000 7f4d5bf93640 1 -- 172.21.15.1:0/10684846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d540c6fd0 msgr2=0x7f4d54150a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.553+0000 7f4d5bf93640 1 --2- 172.21.15.1:0/10684846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d540c6fd0 0x7f4d54150a10 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400e9b0 tx=0x7f4d4400ee70 comp rx=0 tx=0).stop 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.553+0000 7f4d5bf93640 1 -- 172.21.15.1:0/10684846 shutdown_connections 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.553+0000 7f4d5bf93640 1 --2- 172.21.15.1:0/10684846 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4d3806b4d0 0x7f4d3806d990 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.553+0000 7f4d5bf93640 1 --2- 172.21.15.1:0/10684846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d54152730 0x7f4d540c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.553+0000 7f4d5bf93640 1 --2- 172.21.15.1:0/10684846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d540c6fd0 0x7f4d54150a10 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.553+0000 7f4d5bf93640 1 --2- 172.21.15.1:0/10684846 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d540c18d0 0x7f4d541e1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.553+0000 7f4d5bf93640 1 -- 172.21.15.1:0/10684846 >> 172.21.15.1:0/10684846 conn(0x7f4d54066d00 msgr2=0x7f4d540670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.553+0000 7f4d5bf93640 1 -- 172.21.15.1:0/10684846 shutdown_connections 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.553+0000 7f4d5bf93640 1 -- 172.21.15.1:0/10684846 wait complete. 2024-03-06T07:27:37.570 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:02.888292+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '170', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:03.804423+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:37.570 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:37.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.673+0000 7f2c51ac2640 1 Processor -- start 2024-03-06T07:27:37.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.673+0000 7f2c51ac2640 1 -- start start 2024-03-06T07:27:37.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.674+0000 7f2c51ac2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2c4c152650 0x7f2c4c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.674+0000 7f2c51ac2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2c4c0c6b10 0x7f2c4c158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.674+0000 7f2c51ac2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4c0c2fc0 con 0x7f2c4c0bf9d0 2024-03-06T07:27:37.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.674+0000 7f2c51ac2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c4c155800 con 0x7f2c4c152650 2024-03-06T07:27:37.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.674+0000 7f2c51ac2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c4c0c6690 con 0x7f2c4c0c6b10 2024-03-06T07:27:37.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.674+0000 7f2c4b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2c4c152650 0x7f2c4c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.674+0000 7f2c4bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2c4c0bf9d0 0x7f2c4c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33974/0 (socket says 172.21.15.1:33974) 2024-03-06T07:27:37.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.674+0000 7f2c4bfff640 1 -- 172.21.15.1:0/4214528152 learned_addr learned my addr 172.21.15.1:0/4214528152 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:37.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.674+0000 7f2c4affd640 1 --2- 172.21.15.1:0/4214528152 >> v2:172.21.15.98:3300/0 conn(0x7f2c4c0c6b10 0x7f2c4c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.674+0000 7f2c4b7fe640 1 -- 172.21.15.1:0/4214528152 >> v2:172.21.15.98:3300/0 conn(0x7f2c4c0c6b10 msgr2=0x7f2c4c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.674+0000 7f2c4b7fe640 1 --2- 172.21.15.1:0/4214528152 >> v2:172.21.15.98:3300/0 conn(0x7f2c4c0c6b10 0x7f2c4c158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.674+0000 7f2c4b7fe640 1 -- 172.21.15.1:0/4214528152 >> v1:172.21.15.1:6789/0 conn(0x7f2c4c0bf9d0 legacy=0x7f2c4c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.674+0000 7f2c4a7fc640 1 -- 172.21.15.1:0/4214528152 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877368114 0 0) 0x7f2c4c0c2fc0 con 0x7f2c4c0bf9d0 2024-03-06T07:27:37.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.675+0000 7f2c4b7fe640 1 -- 172.21.15.1:0/4214528152 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4c0c7280 con 0x7f2c4c152650 2024-03-06T07:27:37.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.675+0000 7f2c4b7fe640 1 --2- 172.21.15.1:0/4214528152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4c152650 0x7f2c4c155c90 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c0cb8e0 tx=0x7f2c3c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369a9156e10e4375 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.675+0000 7f2c4a7fc640 1 -- 172.21.15.1:0/4214528152 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c0315a0 con 0x7f2c4c152650 2024-03-06T07:27:37.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.675+0000 7f2c4a7fc640 1 -- 172.21.15.1:0/4214528152 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3c037070 con 0x7f2c4c152650 2024-03-06T07:27:37.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.675+0000 7f2c4a7fc640 1 -- 172.21.15.1:0/4214528152 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c031960 con 0x7f2c4c152650 2024-03-06T07:27:37.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.676+0000 7f2c51ac2640 1 -- 172.21.15.1:0/4214528152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4c152650 msgr2=0x7f2c4c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.676+0000 7f2c51ac2640 1 --2- 172.21.15.1:0/4214528152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4c152650 0x7f2c4c155c90 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c0cb8e0 tx=0x7f2c3c02b840 comp rx=0 tx=0).stop 2024-03-06T07:27:37.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.676+0000 7f2c51ac2640 1 -- 172.21.15.1:0/4214528152 shutdown_connections 2024-03-06T07:27:37.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.676+0000 7f2c51ac2640 1 --2- 172.21.15.1:0/4214528152 >> v2:172.21.15.98:3300/0 conn(0x7f2c4c0c6b10 0x7f2c4c158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.676+0000 7f2c51ac2640 1 --2- 172.21.15.1:0/4214528152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4c152650 0x7f2c4c155c90 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.676+0000 7f2c51ac2640 1 -- 172.21.15.1:0/4214528152 >> 172.21.15.1:0/4214528152 conn(0x7f2c4c066ee0 msgr2=0x7f2c4c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.676+0000 7f2c51ac2640 1 -- 172.21.15.1:0/4214528152 shutdown_connections 2024-03-06T07:27:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.676+0000 7f2c51ac2640 1 -- 172.21.15.1:0/4214528152 wait complete. 2024-03-06T07:27:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.677+0000 7f2c51ac2640 1 Processor -- start 2024-03-06T07:27:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.677+0000 7f2c51ac2640 1 -- start start 2024-03-06T07:27:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.678+0000 7f2c51ac2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c4c0bf9d0 0x7f2c4c153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.678+0000 7f2c51ac2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c4c152650 0x7f2c4c1deaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.678+0000 7f2c51ac2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4c154410 0x7f2c4c1defe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.678+0000 7f2c51ac2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c155490 con 0x7f2c4c154410 2024-03-06T07:27:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.678+0000 7f2c51ac2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c1551b0 con 0x7f2c4c0bf9d0 2024-03-06T07:27:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.678+0000 7f2c51ac2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c152a30 con 0x7f2c4c152650 2024-03-06T07:27:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.678+0000 7f2c4bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4c154410 0x7f2c4c1defe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.678+0000 7f2c4bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4c154410 0x7f2c4c1defe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58870/0 (socket says 172.21.15.1:58870) 2024-03-06T07:27:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.678+0000 7f2c4bfff640 1 -- 172.21.15.1:0/1890514536 learned_addr learned my addr 172.21.15.1:0/1890514536 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:37.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.678+0000 7f2c4affd640 1 --2- 172.21.15.1:0/1890514536 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c4c152650 0x7f2c4c1deaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.678+0000 7f2c4b7fe640 1 --2- 172.21.15.1:0/1890514536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c4c0bf9d0 0x7f2c4c153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.678+0000 7f2c4bfff640 1 -- 172.21.15.1:0/1890514536 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c4c152650 msgr2=0x7f2c4c1deaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.678+0000 7f2c4bfff640 1 --2- 172.21.15.1:0/1890514536 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c4c152650 0x7f2c4c1deaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.678+0000 7f2c4bfff640 1 -- 172.21.15.1:0/1890514536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c4c0bf9d0 msgr2=0x7f2c4c153b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.678+0000 7f2c4bfff640 1 --2- 172.21.15.1:0/1890514536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c4c0bf9d0 0x7f2c4c153b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.678+0000 7f2c4bfff640 1 -- 172.21.15.1:0/1890514536 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4c154c60 con 0x7f2c4c154410 2024-03-06T07:27:37.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.679+0000 7f2c4bfff640 1 --2- 172.21.15.1:0/1890514536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4c154410 0x7f2c4c1defe0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f2c4400c700 tx=0x7f2c4400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.679+0000 7f2c43fff640 1 -- 172.21.15.1:0/1890514536 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c44018070 con 0x7f2c4c154410 2024-03-06T07:27:37.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.679+0000 7f2c43fff640 1 -- 172.21.15.1:0/1890514536 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4400ce10 con 0x7f2c4c154410 2024-03-06T07:27:37.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.679+0000 7f2c51ac2640 1 -- 172.21.15.1:0/1890514536 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c4c1eb8b0 con 0x7f2c4c154410 2024-03-06T07:27:37.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.679+0000 7f2c43fff640 1 -- 172.21.15.1:0/1890514536 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4400a880 con 0x7f2c4c154410 2024-03-06T07:27:37.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.679+0000 7f2c51ac2640 1 -- 172.21.15.1:0/1890514536 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4c154960 con 0x7f2c4c154410 2024-03-06T07:27:37.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.680+0000 7f2c51ac2640 1 -- 172.21.15.1:0/1890514536 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c4c067fd0 con 0x7f2c4c154410 2024-03-06T07:27:37.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.681+0000 7f2c43fff640 1 -- 172.21.15.1:0/1890514536 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c4401b070 con 0x7f2c4c154410 2024-03-06T07:27:37.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.681+0000 7f2c43fff640 1 --2- 172.21.15.1:0/1890514536 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c2c06b4d0 0x7f2c2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.681+0000 7f2c43fff640 1 -- 172.21.15.1:0/1890514536 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2c44089890 con 0x7f2c4c154410 2024-03-06T07:27:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.681+0000 7f2c4b7fe640 1 --2- 172.21.15.1:0/1890514536 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c2c06b4d0 0x7f2c2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.682+0000 7f2c4b7fe640 1 --2- 172.21.15.1:0/1890514536 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c2c06b4d0 0x7f2c2c06d990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c036000 tx=0x7f2c3c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.685+0000 7f2c43fff640 1 -- 172.21.15.1:0/1890514536 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c44058610 con 0x7f2c4c154410 2024-03-06T07:27:37.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.935+0000 7f2c51ac2640 1 -- 172.21.15.1:0/1890514536 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2c4c0b06f0 con 0x7f2c4c154410 2024-03-06T07:27:37.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.936+0000 7f2c43fff640 1 -- 172.21.15.1:0/1890514536 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v171) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2c44057f10 con 0x7f2c4c154410 2024-03-06T07:27:37.938 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:27:37.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.938+0000 7f2c51ac2640 1 -- 172.21.15.1:0/1890514536 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c2c06b4d0 msgr2=0x7f2c2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.938+0000 7f2c51ac2640 1 --2- 172.21.15.1:0/1890514536 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c2c06b4d0 0x7f2c2c06d990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c036000 tx=0x7f2c3c008030 comp rx=0 tx=0).stop 2024-03-06T07:27:37.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.938+0000 7f2c51ac2640 1 -- 172.21.15.1:0/1890514536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4c154410 msgr2=0x7f2c4c1defe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.938+0000 7f2c51ac2640 1 --2- 172.21.15.1:0/1890514536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4c154410 0x7f2c4c1defe0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f2c4400c700 tx=0x7f2c4400cbc0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.938+0000 7f2c51ac2640 1 -- 172.21.15.1:0/1890514536 shutdown_connections 2024-03-06T07:27:37.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.938+0000 7f2c51ac2640 1 --2- 172.21.15.1:0/1890514536 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2c2c06b4d0 0x7f2c2c06d990 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.938+0000 7f2c51ac2640 1 --2- 172.21.15.1:0/1890514536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4c154410 0x7f2c4c1defe0 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.938+0000 7f2c51ac2640 1 --2- 172.21.15.1:0/1890514536 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2c4c152650 0x7f2c4c1deaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.938+0000 7f2c51ac2640 1 --2- 172.21.15.1:0/1890514536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c4c0bf9d0 0x7f2c4c153b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.938+0000 7f2c51ac2640 1 -- 172.21.15.1:0/1890514536 >> 172.21.15.1:0/1890514536 conn(0x7f2c4c066ee0 msgr2=0x7f2c4c14e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.938+0000 7f2c51ac2640 1 -- 172.21.15.1:0/1890514536 shutdown_connections 2024-03-06T07:27:37.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:37.938+0000 7f2c51ac2640 1 -- 172.21.15.1:0/1890514536 wait complete. 2024-03-06T07:27:37.950 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:38.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.052+0000 7f0c77d25640 1 Processor -- start 2024-03-06T07:27:38.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.053+0000 7f0c77d25640 1 -- start start 2024-03-06T07:27:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.053+0000 7f0c77d25640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c70152840 0x7f0c700c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.053+0000 7f0c77d25640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0c700c60b0 0x7f0c701553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.053+0000 7f0c77d25640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c700c5e40 con 0x7f0c700c3970 2024-03-06T07:27:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.053+0000 7f0c77d25640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c700c67f0 con 0x7f0c700c60b0 2024-03-06T07:27:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.053+0000 7f0c77d25640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700c6a40 con 0x7f0c70152840 2024-03-06T07:27:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.053+0000 7f0c762c2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0c700c60b0 0x7f0c701553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.054+0000 7f0c762c2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0c700c60b0 0x7f0c701553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58880/0 (socket says 172.21.15.1:58880) 2024-03-06T07:27:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.054+0000 7f0c75ac1640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0c700c3970 0x7f0c70152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33976/0 (socket says 172.21.15.1:33976) 2024-03-06T07:27:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.054+0000 7f0c762c2640 1 -- 172.21.15.1:0/4116468102 learned_addr learned my addr 172.21.15.1:0/4116468102 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.054+0000 7f0c752c0640 1 --2- 172.21.15.1:0/4116468102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c70152840 0x7f0c700c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.054+0000 7f0c762c2640 1 -- 172.21.15.1:0/4116468102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c70152840 msgr2=0x7f0c700c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.054+0000 7f0c762c2640 1 --2- 172.21.15.1:0/4116468102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c70152840 0x7f0c700c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.054+0000 7f0c762c2640 1 -- 172.21.15.1:0/4116468102 >> v1:172.21.15.1:6789/0 conn(0x7f0c700c3970 legacy=0x7f0c70152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.054+0000 7f0c762c2640 1 -- 172.21.15.1:0/4116468102 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c700c6d20 con 0x7f0c700c60b0 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.055+0000 7f0c762c2640 1 --2- 172.21.15.1:0/4116468102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c700c60b0 0x7f0c701553d0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f0c68009e50 tx=0x7f0c6802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc28e7c9f29f5f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.055+0000 7f0c74abf640 1 -- 172.21.15.1:0/4116468102 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6802cb00 con 0x7f0c700c60b0 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.055+0000 7f0c74abf640 1 -- 172.21.15.1:0/4116468102 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6802cca0 con 0x7f0c700c60b0 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.055+0000 7f0c74abf640 1 -- 172.21.15.1:0/4116468102 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68034610 con 0x7f0c700c60b0 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.055+0000 7f0c77d25640 1 -- 172.21.15.1:0/4116468102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c700c60b0 msgr2=0x7f0c701553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.055+0000 7f0c77d25640 1 --2- 172.21.15.1:0/4116468102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c700c60b0 0x7f0c701553d0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f0c68009e50 tx=0x7f0c6802bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.055+0000 7f0c77d25640 1 -- 172.21.15.1:0/4116468102 shutdown_connections 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.055+0000 7f0c77d25640 1 --2- 172.21.15.1:0/4116468102 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c70152840 0x7f0c700c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.055+0000 7f0c77d25640 1 --2- 172.21.15.1:0/4116468102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c700c60b0 0x7f0c701553d0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.055+0000 7f0c77d25640 1 -- 172.21.15.1:0/4116468102 >> 172.21.15.1:0/4116468102 conn(0x7f0c700c88f0 msgr2=0x7f0c700c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.055+0000 7f0c77d25640 1 -- 172.21.15.1:0/4116468102 shutdown_connections 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.055+0000 7f0c77d25640 1 -- 172.21.15.1:0/4116468102 wait complete. 2024-03-06T07:27:38.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.056+0000 7f0c77d25640 1 Processor -- start 2024-03-06T07:27:38.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.057+0000 7f0c77d25640 1 -- start start 2024-03-06T07:27:38.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.057+0000 7f0c77d25640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c700c3970 0x7f0c70157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.057+0000 7f0c77d25640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c700c60b0 0x7f0c701ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.057+0000 7f0c77d25640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c70152840 0x7f0c701ee570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.057+0000 7f0c77d25640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700c75c0 con 0x7f0c700c60b0 2024-03-06T07:27:38.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.057+0000 7f0c77d25640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700c72e0 con 0x7f0c70152840 2024-03-06T07:27:38.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.057+0000 7f0c77d25640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700c3d50 con 0x7f0c700c3970 2024-03-06T07:27:38.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.057+0000 7f0c752c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c700c60b0 0x7f0c701ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.057+0000 7f0c752c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c700c60b0 0x7f0c701ee030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58896/0 (socket says 172.21.15.1:58896) 2024-03-06T07:27:38.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.057+0000 7f0c752c0640 1 -- 172.21.15.1:0/441583539 learned_addr learned my addr 172.21.15.1:0/441583539 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:38.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.057+0000 7f0c75ac1640 1 --2- 172.21.15.1:0/441583539 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c700c3970 0x7f0c70157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.057+0000 7f0c762c2640 1 --2- 172.21.15.1:0/441583539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c70152840 0x7f0c701ee570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.058+0000 7f0c752c0640 1 -- 172.21.15.1:0/441583539 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c700c3970 msgr2=0x7f0c70157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.058+0000 7f0c752c0640 1 --2- 172.21.15.1:0/441583539 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c700c3970 0x7f0c70157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.058+0000 7f0c752c0640 1 -- 172.21.15.1:0/441583539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c70152840 msgr2=0x7f0c701ee570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.058+0000 7f0c752c0640 1 --2- 172.21.15.1:0/441583539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c70152840 0x7f0c701ee570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.058+0000 7f0c752c0640 1 -- 172.21.15.1:0/441583539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c701eeab0 con 0x7f0c700c60b0 2024-03-06T07:27:38.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.058+0000 7f0c75ac1640 1 --2- 172.21.15.1:0/441583539 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c700c3970 0x7f0c70157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:38.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.058+0000 7f0c752c0640 1 --2- 172.21.15.1:0/441583539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c700c60b0 0x7f0c701ee030 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000d5f0 tx=0x7f0c6000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.058+0000 7f0c667fc640 1 -- 172.21.15.1:0/441583539 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c600125f0 con 0x7f0c700c60b0 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.058+0000 7f0c667fc640 1 -- 172.21.15.1:0/441583539 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c60019070 con 0x7f0c700c60b0 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.058+0000 7f0c77d25640 1 -- 172.21.15.1:0/441583539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c7014f9c0 con 0x7f0c700c60b0 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.058+0000 7f0c667fc640 1 -- 172.21.15.1:0/441583539 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c600129d0 con 0x7f0c700c60b0 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.058+0000 7f0c77d25640 1 -- 172.21.15.1:0/441583539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c700abd10 con 0x7f0c700c60b0 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.059+0000 7f0c77d25640 1 -- 172.21.15.1:0/441583539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c700686e0 con 0x7f0c700c60b0 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.060+0000 7f0c667fc640 1 -- 172.21.15.1:0/441583539 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c60012b70 con 0x7f0c700c60b0 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.060+0000 7f0c667fc640 1 --2- 172.21.15.1:0/441583539 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c5406b4d0 0x7f0c5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.060+0000 7f0c667fc640 1 -- 172.21.15.1:0/441583539 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0c6008a670 con 0x7f0c700c60b0 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.061+0000 7f0c75ac1640 1 --2- 172.21.15.1:0/441583539 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c5406b4d0 0x7f0c5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.061+0000 7f0c75ac1640 1 --2- 172.21.15.1:0/441583539 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c5406b4d0 0x7f0c5406d990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c007fd0 tx=0x7f0c6c007d60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.065+0000 7f0c667fc640 1 -- 172.21.15.1:0/441583539 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c600593f0 con 0x7f0c700c60b0 2024-03-06T07:27:38.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.245+0000 7f0c77d25640 1 -- 172.21.15.1:0/441583539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0c7006eb70 con 0x7f0c700c60b0 2024-03-06T07:27:38.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.245+0000 7f0c667fc640 1 -- 172.21.15.1:0/441583539 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v171) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0c6005f0f0 con 0x7f0c700c60b0 2024-03-06T07:27:38.248 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:27:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.247+0000 7f0c77d25640 1 -- 172.21.15.1:0/441583539 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c5406b4d0 msgr2=0x7f0c5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.247+0000 7f0c77d25640 1 --2- 172.21.15.1:0/441583539 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c5406b4d0 0x7f0c5406d990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c007fd0 tx=0x7f0c6c007d60 comp rx=0 tx=0).stop 2024-03-06T07:27:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.247+0000 7f0c77d25640 1 -- 172.21.15.1:0/441583539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c700c60b0 msgr2=0x7f0c701ee030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.247+0000 7f0c77d25640 1 --2- 172.21.15.1:0/441583539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c700c60b0 0x7f0c701ee030 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000d5f0 tx=0x7f0c6000dab0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.247+0000 7f0c77d25640 1 -- 172.21.15.1:0/441583539 shutdown_connections 2024-03-06T07:27:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.247+0000 7f0c77d25640 1 --2- 172.21.15.1:0/441583539 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c5406b4d0 0x7f0c5406d990 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.247+0000 7f0c77d25640 1 --2- 172.21.15.1:0/441583539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c70152840 0x7f0c701ee570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.247+0000 7f0c77d25640 1 --2- 172.21.15.1:0/441583539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c700c60b0 0x7f0c701ee030 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.247+0000 7f0c77d25640 1 --2- 172.21.15.1:0/441583539 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c700c3970 0x7f0c70157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.247+0000 7f0c77d25640 1 -- 172.21.15.1:0/441583539 >> 172.21.15.1:0/441583539 conn(0x7f0c700c88f0 msgr2=0x7f0c7006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.248+0000 7f0c77d25640 1 -- 172.21.15.1:0/441583539 shutdown_connections 2024-03-06T07:27:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.248+0000 7f0c77d25640 1 -- 172.21.15.1:0/441583539 wait complete. 2024-03-06T07:27:38.260 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:38.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.396+0000 7f1a2c1f7640 1 Processor -- start 2024-03-06T07:27:38.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.397+0000 7f1a2c1f7640 1 -- start start 2024-03-06T07:27:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.397+0000 7f1a2c1f7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1a24152890 0x7f1a24159030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.397+0000 7f1a2c1f7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1a240c8cf0 0x7f1a240c9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.397+0000 7f1a2c1f7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a24155080 con 0x7f1a240c7560 2024-03-06T07:27:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.397+0000 7f1a2c1f7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a24155f70 con 0x7f1a24152890 2024-03-06T07:27:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.397+0000 7f1a2c1f7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a240c35b0 con 0x7f1a240c8cf0 2024-03-06T07:27:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.397+0000 7f1a29f93640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1a24152890 0x7f1a24159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.397+0000 7f1a29792640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1a240c7560 0x7f1a24155860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33982/0 (socket says 172.21.15.1:33982) 2024-03-06T07:27:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.397+0000 7f1a29f93640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1a24152890 0x7f1a24159030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58902/0 (socket says 172.21.15.1:58902) 2024-03-06T07:27:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.397+0000 7f1a29792640 1 -- 172.21.15.1:0/4098363852 learned_addr learned my addr 172.21.15.1:0/4098363852 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.398+0000 7f1a29f93640 1 -- 172.21.15.1:0/4098363852 >> v2:172.21.15.98:3300/0 conn(0x7f1a240c8cf0 msgr2=0x7f1a240c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.398+0000 7f1a2a794640 1 --2- 172.21.15.1:0/4098363852 >> v2:172.21.15.98:3300/0 conn(0x7f1a240c8cf0 0x7f1a240c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.398+0000 7f1a29f93640 1 --2- 172.21.15.1:0/4098363852 >> v2:172.21.15.98:3300/0 conn(0x7f1a240c8cf0 0x7f1a240c9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.398+0000 7f1a29f93640 1 -- 172.21.15.1:0/4098363852 >> v1:172.21.15.1:6789/0 conn(0x7f1a240c7560 legacy=0x7f1a24155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.398+0000 7f1a29f93640 1 -- 172.21.15.1:0/4098363852 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a240c3890 con 0x7f1a24152890 2024-03-06T07:27:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.398+0000 7f1a28f91640 1 -- 172.21.15.1:0/4098363852 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723323529 0 0) 0x7f1a24155080 con 0x7f1a240c7560 2024-03-06T07:27:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.398+0000 7f1a2a794640 1 --2- 172.21.15.1:0/4098363852 >> v2:172.21.15.98:3300/0 conn(0x7f1a240c8cf0 0x7f1a240c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.399+0000 7f1a29f93640 1 --2- 172.21.15.1:0/4098363852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a24152890 0x7f1a24159030 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f1a14009d90 tx=0x7f1a1402bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeaf7c9b4b8057d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.399+0000 7f1a28f91640 1 -- 172.21.15.1:0/4098363852 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a14039070 con 0x7f1a24152890 2024-03-06T07:27:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.399+0000 7f1a28f91640 1 -- 172.21.15.1:0/4098363852 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1402cb40 con 0x7f1a24152890 2024-03-06T07:27:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.399+0000 7f1a28f91640 1 -- 172.21.15.1:0/4098363852 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a14034440 con 0x7f1a24152890 2024-03-06T07:27:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.399+0000 7f1a2c1f7640 1 -- 172.21.15.1:0/4098363852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a24152890 msgr2=0x7f1a24159030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.399+0000 7f1a2c1f7640 1 --2- 172.21.15.1:0/4098363852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a24152890 0x7f1a24159030 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f1a14009d90 tx=0x7f1a1402bd90 comp rx=0 tx=0).stop 2024-03-06T07:27:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.399+0000 7f1a2c1f7640 1 -- 172.21.15.1:0/4098363852 shutdown_connections 2024-03-06T07:27:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.399+0000 7f1a2c1f7640 1 --2- 172.21.15.1:0/4098363852 >> v2:172.21.15.98:3300/0 conn(0x7f1a240c8cf0 0x7f1a240c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.399+0000 7f1a2c1f7640 1 --2- 172.21.15.1:0/4098363852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a24152890 0x7f1a24159030 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.399+0000 7f1a2c1f7640 1 -- 172.21.15.1:0/4098363852 >> 172.21.15.1:0/4098363852 conn(0x7f1a2406b3e0 msgr2=0x7f1a2406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.399+0000 7f1a2c1f7640 1 -- 172.21.15.1:0/4098363852 shutdown_connections 2024-03-06T07:27:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.399+0000 7f1a2c1f7640 1 -- 172.21.15.1:0/4098363852 wait complete. 2024-03-06T07:27:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.400+0000 7f1a2c1f7640 1 Processor -- start 2024-03-06T07:27:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.400+0000 7f1a2c1f7640 1 -- start start 2024-03-06T07:27:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a2c1f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a240c7560 0x7f1a24151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a2c1f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a240c8cf0 0x7f1a241573c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a2c1f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a24152890 0x7f1a24153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a2c1f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a24159570 con 0x7f1a240c8cf0 2024-03-06T07:27:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a2c1f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a24155340 con 0x7f1a24152890 2024-03-06T07:27:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a2c1f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a240c3260 con 0x7f1a240c7560 2024-03-06T07:27:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a29792640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a240c8cf0 0x7f1a241573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a29792640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a240c8cf0 0x7f1a241573c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58906/0 (socket says 172.21.15.1:58906) 2024-03-06T07:27:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a29792640 1 -- 172.21.15.1:0/1804994762 learned_addr learned my addr 172.21.15.1:0/1804994762 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a29f93640 1 --2- 172.21.15.1:0/1804994762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a240c7560 0x7f1a24151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a2a794640 1 --2- 172.21.15.1:0/1804994762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a24152890 0x7f1a24153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a29792640 1 -- 172.21.15.1:0/1804994762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a240c7560 msgr2=0x7f1a24151890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a29792640 1 --2- 172.21.15.1:0/1804994762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a240c7560 0x7f1a24151890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a29792640 1 -- 172.21.15.1:0/1804994762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a24152890 msgr2=0x7f1a24153900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a29792640 1 --2- 172.21.15.1:0/1804994762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a24152890 0x7f1a24153900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a29792640 1 -- 172.21.15.1:0/1804994762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a24156500 con 0x7f1a240c8cf0 2024-03-06T07:27:38.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a29f93640 1 --2- 172.21.15.1:0/1804994762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a240c7560 0x7f1a24151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:38.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.401+0000 7f1a2a794640 1 --2- 172.21.15.1:0/1804994762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a24152890 0x7f1a24153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:38.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.402+0000 7f1a29792640 1 --2- 172.21.15.1:0/1804994762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a240c8cf0 0x7f1a241573c0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f1a1800c430 tx=0x7f1a1800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.402+0000 7f1a227fc640 1 -- 172.21.15.1:0/1804994762 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a180074e0 con 0x7f1a240c8cf0 2024-03-06T07:27:38.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.402+0000 7f1a227fc640 1 -- 172.21.15.1:0/1804994762 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a180047e0 con 0x7f1a240c8cf0 2024-03-06T07:27:38.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.402+0000 7f1a2c1f7640 1 -- 172.21.15.1:0/1804994762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a241567e0 con 0x7f1a240c8cf0 2024-03-06T07:27:38.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.402+0000 7f1a227fc640 1 -- 172.21.15.1:0/1804994762 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a18004bd0 con 0x7f1a240c8cf0 2024-03-06T07:27:38.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.402+0000 7f1a2c1f7640 1 -- 172.21.15.1:0/1804994762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a24158850 con 0x7f1a240c8cf0 2024-03-06T07:27:38.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.403+0000 7f1a2c1f7640 1 -- 172.21.15.1:0/1804994762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a24067fd0 con 0x7f1a240c8cf0 2024-03-06T07:27:38.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.403+0000 7f1a227fc640 1 -- 172.21.15.1:0/1804994762 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a18011070 con 0x7f1a240c8cf0 2024-03-06T07:27:38.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.404+0000 7f1a227fc640 1 --2- 172.21.15.1:0/1804994762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a1006b4d0 0x7f1a1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.404+0000 7f1a227fc640 1 -- 172.21.15.1:0/1804994762 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1a18089fc0 con 0x7f1a240c8cf0 2024-03-06T07:27:38.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.404+0000 7f1a29f93640 1 --2- 172.21.15.1:0/1804994762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a1006b4d0 0x7f1a1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.404+0000 7f1a29f93640 1 --2- 172.21.15.1:0/1804994762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a1006b4d0 0x7f1a1006d990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f1a14036000 tx=0x7f1a14030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.408+0000 7f1a227fc640 1 -- 172.21.15.1:0/1804994762 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a18058d40 con 0x7f1a240c8cf0 2024-03-06T07:27:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.631+0000 7f1a2c1f7640 1 -- 172.21.15.1:0/1804994762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1a240c2d80 con 0x7f1a240c8cf0 2024-03-06T07:27:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.632+0000 7f1a227fc640 1 -- 172.21.15.1:0/1804994762 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v171) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1a18058640 con 0x7f1a240c8cf0 2024-03-06T07:27:38.634 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:27:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.634+0000 7f1a2c1f7640 1 -- 172.21.15.1:0/1804994762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a1006b4d0 msgr2=0x7f1a1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.634+0000 7f1a2c1f7640 1 --2- 172.21.15.1:0/1804994762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a1006b4d0 0x7f1a1006d990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f1a14036000 tx=0x7f1a14030000 comp rx=0 tx=0).stop 2024-03-06T07:27:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.634+0000 7f1a2c1f7640 1 -- 172.21.15.1:0/1804994762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a240c8cf0 msgr2=0x7f1a241573c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.634+0000 7f1a2c1f7640 1 --2- 172.21.15.1:0/1804994762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a240c8cf0 0x7f1a241573c0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f1a1800c430 tx=0x7f1a1800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.634+0000 7f1a2c1f7640 1 -- 172.21.15.1:0/1804994762 shutdown_connections 2024-03-06T07:27:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.634+0000 7f1a2c1f7640 1 --2- 172.21.15.1:0/1804994762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a1006b4d0 0x7f1a1006d990 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.634+0000 7f1a2c1f7640 1 --2- 172.21.15.1:0/1804994762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a24152890 0x7f1a24153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.634+0000 7f1a2c1f7640 1 --2- 172.21.15.1:0/1804994762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a240c8cf0 0x7f1a241573c0 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.634+0000 7f1a2c1f7640 1 --2- 172.21.15.1:0/1804994762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a240c7560 0x7f1a24151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.634+0000 7f1a2c1f7640 1 -- 172.21.15.1:0/1804994762 >> 172.21.15.1:0/1804994762 conn(0x7f1a2406b3e0 msgr2=0x7f1a240a9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.634+0000 7f1a2c1f7640 1 -- 172.21.15.1:0/1804994762 shutdown_connections 2024-03-06T07:27:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.634+0000 7f1a2c1f7640 1 -- 172.21.15.1:0/1804994762 wait complete. 2024-03-06T07:27:38.647 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.745+0000 7ff382b3a640 1 Processor -- start 2024-03-06T07:27:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.745+0000 7ff382b3a640 1 -- start start 2024-03-06T07:27:38.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.746+0000 7ff382b3a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff37c0c5cb0 0x7ff37c155930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.746+0000 7ff382b3a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff37c158910 con 0x7ff37c152920 2024-03-06T07:27:38.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.746+0000 7ff382b3a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff37c158bf0 con 0x7ff37c0c5850 2024-03-06T07:27:38.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.746+0000 7ff382b3a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff37c152d00 con 0x7ff37c0c5cb0 2024-03-06T07:27:38.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.746+0000 7ff3808d6640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff37c152920 0x7ff37c06aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33992/0 (socket says 172.21.15.1:33992) 2024-03-06T07:27:38.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.746+0000 7ff3808d6640 1 -- 172.21.15.1:0/2928556576 learned_addr learned my addr 172.21.15.1:0/2928556576 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:38.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.746+0000 7ff3810d7640 1 --2- 172.21.15.1:0/2928556576 >> v2:172.21.15.98:3300/0 conn(0x7ff37c0c5cb0 0x7ff37c155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.747+0000 7ff37b7fe640 1 -- 172.21.15.1:0/2928556576 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423557366 0 0) 0x7ff37c158910 con 0x7ff37c152920 2024-03-06T07:27:38.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.747+0000 7ff37b7fe640 1 -- 172.21.15.1:0/2928556576 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff368003520 con 0x7ff37c152920 2024-03-06T07:27:38.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.747+0000 7ff37b7fe640 1 -- 172.21.15.1:0/2928556576 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487424785 0 0) 0x7ff37c158bf0 con 0x7ff37c0c5850 2024-03-06T07:27:38.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.747+0000 7ff37b7fe640 1 -- 172.21.15.1:0/2928556576 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff37c158910 con 0x7ff37c0c5850 2024-03-06T07:27:38.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.747+0000 7ff37b7fe640 1 -- 172.21.15.1:0/2928556576 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1199530325 0 0) 0x7ff368003520 con 0x7ff37c152920 2024-03-06T07:27:38.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.747+0000 7ff3810d7640 1 -- 172.21.15.1:0/2928556576 >> v1:172.21.15.98:6789/0 conn(0x7ff37c0c5850 legacy=0x7ff37c155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.747+0000 7ff3810d7640 1 -- 172.21.15.1:0/2928556576 >> v1:172.21.15.1:6789/0 conn(0x7ff37c152920 legacy=0x7ff37c06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.747+0000 7ff3810d7640 1 -- 172.21.15.1:0/2928556576 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37c158ed0 con 0x7ff37c0c5cb0 2024-03-06T07:27:38.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.748+0000 7ff3810d7640 1 --2- 172.21.15.1:0/2928556576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff37c0c5cb0 0x7ff37c155930 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff374009d90 tx=0x7ff37402bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55f9015b45ad95bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.748+0000 7ff37b7fe640 1 -- 172.21.15.1:0/2928556576 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374039070 con 0x7ff37c0c5cb0 2024-03-06T07:27:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.748+0000 7ff37b7fe640 1 -- 172.21.15.1:0/2928556576 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37402c8b0 con 0x7ff37c0c5cb0 2024-03-06T07:27:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.748+0000 7ff37b7fe640 1 -- 172.21.15.1:0/2928556576 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff364001670 con 0x7ff37c0c5cb0 2024-03-06T07:27:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.748+0000 7ff382b3a640 1 -- 172.21.15.1:0/2928556576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff37c0c5cb0 msgr2=0x7ff37c155930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.748+0000 7ff382b3a640 1 --2- 172.21.15.1:0/2928556576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff37c0c5cb0 0x7ff37c155930 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff374009d90 tx=0x7ff37402bd80 comp rx=0 tx=0).stop 2024-03-06T07:27:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.748+0000 7ff382b3a640 1 -- 172.21.15.1:0/2928556576 shutdown_connections 2024-03-06T07:27:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.748+0000 7ff382b3a640 1 --2- 172.21.15.1:0/2928556576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff37c0c5cb0 0x7ff37c155930 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.748+0000 7ff382b3a640 1 -- 172.21.15.1:0/2928556576 >> 172.21.15.1:0/2928556576 conn(0x7ff37c0a9700 msgr2=0x7ff37c0a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.749+0000 7ff382b3a640 1 -- 172.21.15.1:0/2928556576 shutdown_connections 2024-03-06T07:27:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.749+0000 7ff382b3a640 1 -- 172.21.15.1:0/2928556576 wait complete. 2024-03-06T07:27:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.750+0000 7ff382b3a640 1 Processor -- start 2024-03-06T07:27:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.750+0000 7ff382b3a640 1 -- start start 2024-03-06T07:27:38.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.750+0000 7ff382b3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff37c0c5850 0x7ff37c06e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.750+0000 7ff382b3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff37c0c5cb0 0x7ff37c157850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.750+0000 7ff382b3a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff37c152920 0x7ff37c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.750+0000 7ff382b3a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37c159240 con 0x7ff37c152920 2024-03-06T07:27:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.750+0000 7ff382b3a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37c1596d0 con 0x7ff37c0c5850 2024-03-06T07:27:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.750+0000 7ff382b3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff37c06b700 con 0x7ff37c0c5cb0 2024-03-06T07:27:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.750+0000 7ff3810d7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff37c152920 0x7ff37c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.751+0000 7ff3810d7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff37c152920 0x7ff37c06ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58914/0 (socket says 172.21.15.1:58914) 2024-03-06T07:27:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.751+0000 7ff3810d7640 1 -- 172.21.15.1:0/1328884563 learned_addr learned my addr 172.21.15.1:0/1328884563 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.751+0000 7ff3808d6640 1 --2- 172.21.15.1:0/1328884563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff37c0c5850 0x7ff37c06e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.751+0000 7ff37bfff640 1 --2- 172.21.15.1:0/1328884563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff37c0c5cb0 0x7ff37c157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.751+0000 7ff3810d7640 1 -- 172.21.15.1:0/1328884563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff37c0c5cb0 msgr2=0x7ff37c157850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.751+0000 7ff3810d7640 1 --2- 172.21.15.1:0/1328884563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff37c0c5cb0 0x7ff37c157850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.751+0000 7ff3810d7640 1 -- 172.21.15.1:0/1328884563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff37c0c5850 msgr2=0x7ff37c06e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.751+0000 7ff3810d7640 1 --2- 172.21.15.1:0/1328884563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff37c0c5850 0x7ff37c06e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.751+0000 7ff3810d7640 1 -- 172.21.15.1:0/1328884563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37c14f700 con 0x7ff37c152920 2024-03-06T07:27:38.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.751+0000 7ff3810d7640 1 --2- 172.21.15.1:0/1328884563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff37c152920 0x7ff37c06ec80 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7ff374009eb0 tx=0x7ff374004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.751+0000 7ff3797fa640 1 -- 172.21.15.1:0/1328884563 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374039050 con 0x7ff37c152920 2024-03-06T07:27:38.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.751+0000 7ff3797fa640 1 -- 172.21.15.1:0/1328884563 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37403a040 con 0x7ff37c152920 2024-03-06T07:27:38.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.751+0000 7ff382b3a640 1 -- 172.21.15.1:0/1328884563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37c1ef090 con 0x7ff37c152920 2024-03-06T07:27:38.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.751+0000 7ff3797fa640 1 -- 172.21.15.1:0/1328884563 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374009450 con 0x7ff37c152920 2024-03-06T07:27:38.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.751+0000 7ff382b3a640 1 -- 172.21.15.1:0/1328884563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37c14fca0 con 0x7ff37c152920 2024-03-06T07:27:38.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.752+0000 7ff382b3a640 1 -- 172.21.15.1:0/1328884563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff37c067fd0 con 0x7ff37c152920 2024-03-06T07:27:38.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.753+0000 7ff3797fa640 1 -- 172.21.15.1:0/1328884563 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff374047020 con 0x7ff37c152920 2024-03-06T07:27:38.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.753+0000 7ff3797fa640 1 --2- 172.21.15.1:0/1328884563 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff35c06b4d0 0x7ff35c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.753+0000 7ff3797fa640 1 -- 172.21.15.1:0/1328884563 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff3740ac250 con 0x7ff37c152920 2024-03-06T07:27:38.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.753+0000 7ff3808d6640 1 --2- 172.21.15.1:0/1328884563 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff35c06b4d0 0x7ff35c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.754+0000 7ff3808d6640 1 --2- 172.21.15.1:0/1328884563 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff35c06b4d0 0x7ff35c06d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7ff3640011e0 tx=0x7ff364009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.761+0000 7ff3797fa640 1 -- 172.21.15.1:0/1328884563 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff37404e020 con 0x7ff37c152920 2024-03-06T07:27:38.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.979+0000 7ff382b3a640 1 -- 172.21.15.1:0/1328884563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff37c06bdf0 con 0x7ff37c152920 2024-03-06T07:27:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.980+0000 7ff3797fa640 1 -- 172.21.15.1:0/1328884563 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+4969 (secure 0 0 0) 0x7ff37403d360 con 0x7ff37c152920 2024-03-06T07:27:38.982 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:38.982 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":164,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:34.364010+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9738":{"gid":9738,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":2,"addr":"172.21.15.1:6835/1021164277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1021164277},{"type":"v1","addr":"172.21.15.1:6835","nonce":1021164277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9738,"qdb_cluster":[9738]},"id":13}]} 2024-03-06T07:27:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 164 2024-03-06T07:27:38.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.981+0000 7ff382b3a640 1 -- 172.21.15.1:0/1328884563 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff35c06b4d0 msgr2=0x7ff35c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.981+0000 7ff382b3a640 1 --2- 172.21.15.1:0/1328884563 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff35c06b4d0 0x7ff35c06d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7ff3640011e0 tx=0x7ff364009040 comp rx=0 tx=0).stop 2024-03-06T07:27:38.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.981+0000 7ff382b3a640 1 -- 172.21.15.1:0/1328884563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff37c152920 msgr2=0x7ff37c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.981+0000 7ff382b3a640 1 --2- 172.21.15.1:0/1328884563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff37c152920 0x7ff37c06ec80 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7ff374009eb0 tx=0x7ff374004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:38.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.982+0000 7ff382b3a640 1 -- 172.21.15.1:0/1328884563 shutdown_connections 2024-03-06T07:27:38.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.982+0000 7ff382b3a640 1 --2- 172.21.15.1:0/1328884563 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff35c06b4d0 0x7ff35c06d990 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.982+0000 7ff382b3a640 1 --2- 172.21.15.1:0/1328884563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff37c152920 0x7ff37c06ec80 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.982+0000 7ff382b3a640 1 --2- 172.21.15.1:0/1328884563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff37c0c5cb0 0x7ff37c157850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.982+0000 7ff382b3a640 1 --2- 172.21.15.1:0/1328884563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff37c0c5850 0x7ff37c06e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.982+0000 7ff382b3a640 1 -- 172.21.15.1:0/1328884563 >> 172.21.15.1:0/1328884563 conn(0x7ff37c0a9700 msgr2=0x7ff37c0c0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.982+0000 7ff382b3a640 1 -- 172.21.15.1:0/1328884563 shutdown_connections 2024-03-06T07:27:38.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:38.982+0000 7ff382b3a640 1 -- 172.21.15.1:0/1328884563 wait complete. 2024-03-06T07:27:38.998 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:39.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.097+0000 7fad931e9640 1 Processor -- start 2024-03-06T07:27:39.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.097+0000 7fad931e9640 1 -- start start 2024-03-06T07:27:39.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.098+0000 7fad931e9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fad8c152650 0x7fad8c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.098+0000 7fad931e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad8c0c6b10 0x7fad8c158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.098+0000 7fad931e9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8c0c2fc0 con 0x7fad8c0bf9d0 2024-03-06T07:27:39.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.098+0000 7fad931e9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad8c155800 con 0x7fad8c152650 2024-03-06T07:27:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.098+0000 7fad931e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c0c6690 con 0x7fad8c0c6b10 2024-03-06T07:27:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.098+0000 7fad91786640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fad8c0bf9d0 0x7fad8c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33996/0 (socket says 172.21.15.1:33996) 2024-03-06T07:27:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.098+0000 7fad91786640 1 -- 172.21.15.1:0/3137522125 learned_addr learned my addr 172.21.15.1:0/3137522125 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.098+0000 7fad8bfff640 1 --2- 172.21.15.1:0/3137522125 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad8c0c6b10 0x7fad8c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.098+0000 7fad90f85640 1 --2- 172.21.15.1:0/3137522125 >> v2:172.21.15.98:3300/0 conn(0x7fad8c152650 0x7fad8c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.098+0000 7fad8b7fe640 1 -- 172.21.15.1:0/3137522125 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122287019 0 0) 0x7fad8c0c2fc0 con 0x7fad8c0bf9d0 2024-03-06T07:27:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.098+0000 7fad8b7fe640 1 -- 172.21.15.1:0/3137522125 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad7c003520 con 0x7fad8c0bf9d0 2024-03-06T07:27:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.099+0000 7fad90f85640 1 -- 172.21.15.1:0/3137522125 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad8c0c6b10 msgr2=0x7fad8c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.099+0000 7fad90f85640 1 --2- 172.21.15.1:0/3137522125 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad8c0c6b10 0x7fad8c158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.099+0000 7fad90f85640 1 -- 172.21.15.1:0/3137522125 >> v1:172.21.15.1:6789/0 conn(0x7fad8c0bf9d0 legacy=0x7fad8c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.099+0000 7fad8b7fe640 1 -- 172.21.15.1:0/3137522125 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3253972311 0 0) 0x7fad7c003520 con 0x7fad8c0bf9d0 2024-03-06T07:27:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.099+0000 7fad90f85640 1 -- 172.21.15.1:0/3137522125 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8c0c7280 con 0x7fad8c152650 2024-03-06T07:27:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.099+0000 7fad90f85640 1 --2- 172.21.15.1:0/3137522125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad8c152650 0x7fad8c155c90 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fad800097a0 tx=0x7fad8002b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96033ab783fd44f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.100+0000 7fad8b7fe640 1 -- 172.21.15.1:0/3137522125 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad80037070 con 0x7fad8c152650 2024-03-06T07:27:39.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.100+0000 7fad8b7fe640 1 -- 172.21.15.1:0/3137522125 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad80030510 con 0x7fad8c152650 2024-03-06T07:27:39.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.100+0000 7fad8b7fe640 1 -- 172.21.15.1:0/3137522125 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad840019f0 con 0x7fad8c152650 2024-03-06T07:27:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.100+0000 7fad931e9640 1 -- 172.21.15.1:0/3137522125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad8c152650 msgr2=0x7fad8c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.100+0000 7fad931e9640 1 --2- 172.21.15.1:0/3137522125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad8c152650 0x7fad8c155c90 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fad800097a0 tx=0x7fad8002b710 comp rx=0 tx=0).stop 2024-03-06T07:27:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.100+0000 7fad931e9640 1 -- 172.21.15.1:0/3137522125 shutdown_connections 2024-03-06T07:27:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.100+0000 7fad931e9640 1 --2- 172.21.15.1:0/3137522125 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad8c0c6b10 0x7fad8c158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.100+0000 7fad931e9640 1 --2- 172.21.15.1:0/3137522125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad8c152650 0x7fad8c155c90 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.100+0000 7fad931e9640 1 -- 172.21.15.1:0/3137522125 >> 172.21.15.1:0/3137522125 conn(0x7fad8c066ee0 msgr2=0x7fad8c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.100+0000 7fad931e9640 1 -- 172.21.15.1:0/3137522125 shutdown_connections 2024-03-06T07:27:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.100+0000 7fad931e9640 1 -- 172.21.15.1:0/3137522125 wait complete. 2024-03-06T07:27:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.101+0000 7fad931e9640 1 Processor -- start 2024-03-06T07:27:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.101+0000 7fad931e9640 1 -- start start 2024-03-06T07:27:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.102+0000 7fad931e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad8c0bf9d0 0x7fad8c1eb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.102+0000 7fad931e9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad8c152650 0x7fad8c141fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.102+0000 7fad931e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad8c142500 0x7fad8c14c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.102+0000 7fad931e9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c155490 con 0x7fad8c152650 2024-03-06T07:27:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.102+0000 7fad931e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c1551b0 con 0x7fad8c142500 2024-03-06T07:27:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.102+0000 7fad931e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c152a30 con 0x7fad8c0bf9d0 2024-03-06T07:27:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.102+0000 7fad8bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad8c152650 0x7fad8c141fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.102+0000 7fad8bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad8c152650 0x7fad8c141fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58916/0 (socket says 172.21.15.1:58916) 2024-03-06T07:27:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.102+0000 7fad8bfff640 1 -- 172.21.15.1:0/3481153424 learned_addr learned my addr 172.21.15.1:0/3481153424 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.102+0000 7fad91786640 1 --2- 172.21.15.1:0/3481153424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad8c142500 0x7fad8c14c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.102+0000 7fad90f85640 1 --2- 172.21.15.1:0/3481153424 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad8c0bf9d0 0x7fad8c1eb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.103+0000 7fad8bfff640 1 -- 172.21.15.1:0/3481153424 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad8c0bf9d0 msgr2=0x7fad8c1eb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.103+0000 7fad8bfff640 1 --2- 172.21.15.1:0/3481153424 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad8c0bf9d0 0x7fad8c1eb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.103+0000 7fad8bfff640 1 -- 172.21.15.1:0/3481153424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad8c142500 msgr2=0x7fad8c14c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.103+0000 7fad8bfff640 1 --2- 172.21.15.1:0/3481153424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad8c142500 0x7fad8c14c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.103+0000 7fad8bfff640 1 -- 172.21.15.1:0/3481153424 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8c0c9b60 con 0x7fad8c152650 2024-03-06T07:27:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.103+0000 7fad8bfff640 1 --2- 172.21.15.1:0/3481153424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad8c152650 0x7fad8c141fc0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fad7800e9e0 tx=0x7fad7800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.103+0000 7fad897fa640 1 -- 172.21.15.1:0/3481153424 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad78017070 con 0x7fad8c152650 2024-03-06T07:27:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.103+0000 7fad897fa640 1 -- 172.21.15.1:0/3481153424 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7800c9d0 con 0x7fad8c152650 2024-03-06T07:27:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.103+0000 7fad931e9640 1 -- 172.21.15.1:0/3481153424 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad8c14c1d0 con 0x7fad8c152650 2024-03-06T07:27:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.103+0000 7fad897fa640 1 -- 172.21.15.1:0/3481153424 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7800cdc0 con 0x7fad8c152650 2024-03-06T07:27:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.103+0000 7fad931e9640 1 -- 172.21.15.1:0/3481153424 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad8c0c9e00 con 0x7fad8c152650 2024-03-06T07:27:39.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.104+0000 7fad931e9640 1 -- 172.21.15.1:0/3481153424 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad8c067fd0 con 0x7fad8c152650 2024-03-06T07:27:39.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.105+0000 7fad897fa640 1 -- 172.21.15.1:0/3481153424 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad7801a070 con 0x7fad8c152650 2024-03-06T07:27:39.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.105+0000 7fad897fa640 1 --2- 172.21.15.1:0/3481153424 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad7006b4d0 0x7fad7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.105+0000 7fad897fa640 1 -- 172.21.15.1:0/3481153424 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fad7808a350 con 0x7fad8c152650 2024-03-06T07:27:39.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.105+0000 7fad90f85640 1 --2- 172.21.15.1:0/3481153424 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad7006b4d0 0x7fad7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.106+0000 7fad90f85640 1 --2- 172.21.15.1:0/3481153424 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad7006b4d0 0x7fad7006d990 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fad80034000 tx=0x7fad80008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.110+0000 7fad897fa640 1 -- 172.21.15.1:0/3481153424 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad780590d0 con 0x7fad8c152650 2024-03-06T07:27:39.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.291+0000 7fad931e9640 1 -- 172.21.15.1:0/3481153424 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad8c0b06f0 con 0x7fad8c152650 2024-03-06T07:27:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.292+0000 7fad897fa640 1 -- 172.21.15.1:0/3481153424 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+16832 (secure 0 0 0) 0x7fad780589d0 con 0x7fad8c152650 2024-03-06T07:27:39.294 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:39.294 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":171,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:27:06.270883+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:02.888292+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:03.804423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:39.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.294+0000 7fad931e9640 1 -- 172.21.15.1:0/3481153424 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad7006b4d0 msgr2=0x7fad7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.294+0000 7fad931e9640 1 --2- 172.21.15.1:0/3481153424 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad7006b4d0 0x7fad7006d990 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fad80034000 tx=0x7fad80008030 comp rx=0 tx=0).stop 2024-03-06T07:27:39.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.294+0000 7fad931e9640 1 -- 172.21.15.1:0/3481153424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad8c152650 msgr2=0x7fad8c141fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.294+0000 7fad931e9640 1 --2- 172.21.15.1:0/3481153424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad8c152650 0x7fad8c141fc0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fad7800e9e0 tx=0x7fad7800eea0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.294+0000 7fad931e9640 1 -- 172.21.15.1:0/3481153424 shutdown_connections 2024-03-06T07:27:39.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.294+0000 7fad931e9640 1 --2- 172.21.15.1:0/3481153424 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad7006b4d0 0x7fad7006d990 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.294+0000 7fad931e9640 1 --2- 172.21.15.1:0/3481153424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad8c142500 0x7fad8c14c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.294+0000 7fad931e9640 1 --2- 172.21.15.1:0/3481153424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad8c152650 0x7fad8c141fc0 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.294+0000 7fad931e9640 1 --2- 172.21.15.1:0/3481153424 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad8c0bf9d0 0x7fad8c1eb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.294+0000 7fad931e9640 1 -- 172.21.15.1:0/3481153424 >> 172.21.15.1:0/3481153424 conn(0x7fad8c066ee0 msgr2=0x7fad8c0672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.294+0000 7fad931e9640 1 -- 172.21.15.1:0/3481153424 shutdown_connections 2024-03-06T07:27:39.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.294+0000 7fad931e9640 1 -- 172.21.15.1:0/3481153424 wait complete. 2024-03-06T07:27:39.307 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:27:39.307 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:39.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.410+0000 7f86ce9f7640 1 Processor -- start 2024-03-06T07:27:39.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.410+0000 7f86ce9f7640 1 -- start start 2024-03-06T07:27:39.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.411+0000 7f86ce9f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86c8071fb0 0x7f86c80c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.411+0000 7f86ce9f7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f86c80730c0 0x7f86c80c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.411+0000 7f86ce9f7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c80a6610 con 0x7f86c8159250 2024-03-06T07:27:39.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.411+0000 7f86ce9f7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c80c1e60 con 0x7f86c80730c0 2024-03-06T07:27:39.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.411+0000 7f86ce9f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c80c2090 con 0x7f86c8071fb0 2024-03-06T07:27:39.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.411+0000 7f86ccf94640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f86c80730c0 0x7f86c80c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.411+0000 7f86ccf94640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f86c80730c0 0x7f86c80c5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58924/0 (socket says 172.21.15.1:58924) 2024-03-06T07:27:39.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.411+0000 7f86ccf94640 1 -- 172.21.15.1:0/2909956095 learned_addr learned my addr 172.21.15.1:0/2909956095 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:39.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.411+0000 7f86c7fff640 1 --1- 172.21.15.1:0/2909956095 >> v1:172.21.15.1:6789/0 conn(0x7f86c8159250 0x7f86c80c3890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34010/0 (socket says 172.21.15.1:34010) 2024-03-06T07:27:39.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.411+0000 7f86c77fe640 1 --2- 172.21.15.1:0/2909956095 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86c8071fb0 0x7f86c80c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.411+0000 7f86ccf94640 1 -- 172.21.15.1:0/2909956095 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86c8071fb0 msgr2=0x7f86c80c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:39.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.411+0000 7f86ccf94640 1 --2- 172.21.15.1:0/2909956095 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86c8071fb0 0x7f86c80c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.411+0000 7f86ccf94640 1 -- 172.21.15.1:0/2909956095 >> v1:172.21.15.1:6789/0 conn(0x7f86c8159250 legacy=0x7f86c80c3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.411+0000 7f86ccf94640 1 -- 172.21.15.1:0/2909956095 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c80c2370 con 0x7f86c80730c0 2024-03-06T07:27:39.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.411+0000 7f86c6ffd640 1 -- 172.21.15.1:0/2909956095 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612042917 0 0) 0x7f86c80a6610 con 0x7f86c8159250 2024-03-06T07:27:39.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.411+0000 7f86c77fe640 1 --2- 172.21.15.1:0/2909956095 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86c8071fb0 0x7f86c80c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:39.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.412+0000 7f86ccf94640 1 --2- 172.21.15.1:0/2909956095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86c80730c0 0x7f86c80c5090 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009e50 tx=0x7f86c002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9fce3722b80e79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.412+0000 7f86c6ffd640 1 -- 172.21.15.1:0/2909956095 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c002cb00 con 0x7f86c80730c0 2024-03-06T07:27:39.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.412+0000 7f86c6ffd640 1 -- 172.21.15.1:0/2909956095 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c002cca0 con 0x7f86c80730c0 2024-03-06T07:27:39.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.412+0000 7f86c6ffd640 1 -- 172.21.15.1:0/2909956095 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0034610 con 0x7f86c80730c0 2024-03-06T07:27:39.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.413+0000 7f86ce9f7640 1 -- 172.21.15.1:0/2909956095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86c80730c0 msgr2=0x7f86c80c5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.413+0000 7f86ce9f7640 1 --2- 172.21.15.1:0/2909956095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86c80730c0 0x7f86c80c5090 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009e50 tx=0x7f86c002bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:39.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.413+0000 7f86ce9f7640 1 -- 172.21.15.1:0/2909956095 shutdown_connections 2024-03-06T07:27:39.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.413+0000 7f86ce9f7640 1 --2- 172.21.15.1:0/2909956095 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86c8071fb0 0x7f86c80c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.413+0000 7f86ce9f7640 1 --2- 172.21.15.1:0/2909956095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86c80730c0 0x7f86c80c5090 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.413+0000 7f86ce9f7640 1 -- 172.21.15.1:0/2909956095 >> 172.21.15.1:0/2909956095 conn(0x7f86c80c6f40 msgr2=0x7f86c80c7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.413+0000 7f86ce9f7640 1 -- 172.21.15.1:0/2909956095 shutdown_connections 2024-03-06T07:27:39.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.413+0000 7f86ce9f7640 1 -- 172.21.15.1:0/2909956095 wait complete. 2024-03-06T07:27:39.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.414+0000 7f86ce9f7640 1 Processor -- start 2024-03-06T07:27:39.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.414+0000 7f86ce9f7640 1 -- start start 2024-03-06T07:27:39.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.415+0000 7f86ce9f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86c8071fb0 0x7f86c8158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.415+0000 7f86ce9f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86c80730c0 0x7f86c81ebc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.415+0000 7f86ce9f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86c8159250 0x7f86c81ec160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.415+0000 7f86ce9f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c80c2c10 con 0x7f86c8159250 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.415+0000 7f86ce9f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c80c2930 con 0x7f86c80730c0 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.415+0000 7f86ce9f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c80c3fa0 con 0x7f86c8071fb0 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.415+0000 7f86ccf94640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86c8159250 0x7f86c81ec160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.415+0000 7f86ccf94640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86c8159250 0x7f86c81ec160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58932/0 (socket says 172.21.15.1:58932) 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.415+0000 7f86ccf94640 1 -- 172.21.15.1:0/2629948226 learned_addr learned my addr 172.21.15.1:0/2629948226 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.415+0000 7f86c77fe640 1 --2- 172.21.15.1:0/2629948226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86c80730c0 0x7f86c81ebc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.415+0000 7f86c7fff640 1 --2- 172.21.15.1:0/2629948226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86c8071fb0 0x7f86c8158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.415+0000 7f86ccf94640 1 -- 172.21.15.1:0/2629948226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86c8071fb0 msgr2=0x7f86c8158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.415+0000 7f86ccf94640 1 --2- 172.21.15.1:0/2629948226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86c8071fb0 0x7f86c8158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.415+0000 7f86ccf94640 1 -- 172.21.15.1:0/2629948226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86c80730c0 msgr2=0x7f86c81ebc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.415+0000 7f86ccf94640 1 --2- 172.21.15.1:0/2629948226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86c80730c0 0x7f86c81ebc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.415+0000 7f86ccf94640 1 -- 172.21.15.1:0/2629948226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c81eec30 con 0x7f86c8159250 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.415+0000 7f86ccf94640 1 --2- 172.21.15.1:0/2629948226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86c8159250 0x7f86c81ec160 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009e50 tx=0x7f86c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.416+0000 7f86c4ff9640 1 -- 172.21.15.1:0/2629948226 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0043020 con 0x7f86c8159250 2024-03-06T07:27:39.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.416+0000 7f86c4ff9640 1 -- 172.21.15.1:0/2629948226 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c002ce50 con 0x7f86c8159250 2024-03-06T07:27:39.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.416+0000 7f86ce9f7640 1 -- 172.21.15.1:0/2629948226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c80aaa50 con 0x7f86c8159250 2024-03-06T07:27:39.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.416+0000 7f86c4ff9640 1 -- 172.21.15.1:0/2629948226 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0009510 con 0x7f86c8159250 2024-03-06T07:27:39.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.416+0000 7f86ce9f7640 1 -- 172.21.15.1:0/2629948226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c80ab010 con 0x7f86c8159250 2024-03-06T07:27:39.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.417+0000 7f86ce9f7640 1 -- 172.21.15.1:0/2629948226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c8067fd0 con 0x7f86c8159250 2024-03-06T07:27:39.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.417+0000 7f86c4ff9640 1 -- 172.21.15.1:0/2629948226 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86c0046020 con 0x7f86c8159250 2024-03-06T07:27:39.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.417+0000 7f86c4ff9640 1 --2- 172.21.15.1:0/2629948226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86b406b4d0 0x7f86b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.418+0000 7f86c4ff9640 1 -- 172.21.15.1:0/2629948226 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f86c00aca90 con 0x7f86c8159250 2024-03-06T07:27:39.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.418+0000 7f86c7fff640 1 --2- 172.21.15.1:0/2629948226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86b406b4d0 0x7f86b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.418+0000 7f86c7fff640 1 --2- 172.21.15.1:0/2629948226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86b406b4d0 0x7f86b406d990 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f86b8000f60 tx=0x7f86b8007b50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.422+0000 7f86c4ff9640 1 -- 172.21.15.1:0/2629948226 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86c004d030 con 0x7f86c8159250 2024-03-06T07:27:39.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.640+0000 7f86ce9f7640 1 -- 172.21.15.1:0/2629948226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86c80a82f0 con 0x7f86c8159250 2024-03-06T07:27:39.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.640+0000 7f86c4ff9640 1 -- 172.21.15.1:0/2629948226 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+4969 (secure 0 0 0) 0x7f86c007b810 con 0x7f86c8159250 2024-03-06T07:27:39.643 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:39.643 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":164,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:34.364010+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9738":{"gid":9738,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":2,"addr":"172.21.15.1:6835/1021164277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1021164277},{"type":"v1","addr":"172.21.15.1:6835","nonce":1021164277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9738,"qdb_cluster":[9738]},"id":13}]} 2024-03-06T07:27:39.643 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 164 2024-03-06T07:27:39.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.643+0000 7f86ce9f7640 1 -- 172.21.15.1:0/2629948226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86b406b4d0 msgr2=0x7f86b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.643+0000 7f86ce9f7640 1 --2- 172.21.15.1:0/2629948226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86b406b4d0 0x7f86b406d990 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f86b8000f60 tx=0x7f86b8007b50 comp rx=0 tx=0).stop 2024-03-06T07:27:39.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.643+0000 7f86ce9f7640 1 -- 172.21.15.1:0/2629948226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86c8159250 msgr2=0x7f86c81ec160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.643+0000 7f86ce9f7640 1 --2- 172.21.15.1:0/2629948226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86c8159250 0x7f86c81ec160 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009e50 tx=0x7f86c0004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:39.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.643+0000 7f86ce9f7640 1 -- 172.21.15.1:0/2629948226 shutdown_connections 2024-03-06T07:27:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.643+0000 7f86ce9f7640 1 --2- 172.21.15.1:0/2629948226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86b406b4d0 0x7f86b406d990 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.643+0000 7f86ce9f7640 1 --2- 172.21.15.1:0/2629948226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86c8159250 0x7f86c81ec160 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.643+0000 7f86ce9f7640 1 --2- 172.21.15.1:0/2629948226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86c80730c0 0x7f86c81ebc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.643+0000 7f86ce9f7640 1 --2- 172.21.15.1:0/2629948226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86c8071fb0 0x7f86c8158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.643+0000 7f86ce9f7640 1 -- 172.21.15.1:0/2629948226 >> 172.21.15.1:0/2629948226 conn(0x7f86c80c6f40 msgr2=0x7f86c80c3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.643+0000 7f86ce9f7640 1 -- 172.21.15.1:0/2629948226 shutdown_connections 2024-03-06T07:27:39.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.643+0000 7f86ce9f7640 1 -- 172.21.15.1:0/2629948226 wait complete. 2024-03-06T07:27:39.659 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:39.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.776+0000 7f762081c640 1 Processor -- start 2024-03-06T07:27:39.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.776+0000 7f762081c640 1 -- start start 2024-03-06T07:27:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.777+0000 7f762081c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f761c0c32f0 0x7f761c1543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.777+0000 7f762081c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c150900 con 0x7f761c150520 2024-03-06T07:27:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.777+0000 7f762081c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c071790 con 0x7f761c070c80 2024-03-06T07:27:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.777+0000 7f762081c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f761c153ed0 con 0x7f761c0c32f0 2024-03-06T07:27:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.777+0000 7f761a59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f761c150520 0x7f761c1537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34014/0 (socket says 172.21.15.1:34014) 2024-03-06T07:27:39.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.777+0000 7f761a59c640 1 -- 172.21.15.1:0/1089906884 learned_addr learned my addr 172.21.15.1:0/1089906884 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:39.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.777+0000 7f7619d9b640 1 --2- 172.21.15.1:0/1089906884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f761c0c32f0 0x7f761c1543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.778+0000 7f761959a640 1 -- 172.21.15.1:0/1089906884 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531245423 0 0) 0x7f761c150900 con 0x7f761c150520 2024-03-06T07:27:39.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.778+0000 7f761959a640 1 -- 172.21.15.1:0/1089906884 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7608003520 con 0x7f761c150520 2024-03-06T07:27:39.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.778+0000 7f761959a640 1 -- 172.21.15.1:0/1089906884 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864567102 0 0) 0x7f761c071790 con 0x7f761c070c80 2024-03-06T07:27:39.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.778+0000 7f761959a640 1 -- 172.21.15.1:0/1089906884 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f761c150900 con 0x7f761c070c80 2024-03-06T07:27:39.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.778+0000 7f761959a640 1 -- 172.21.15.1:0/1089906884 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3931122294 0 0) 0x7f7608003520 con 0x7f761c150520 2024-03-06T07:27:39.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.778+0000 7f7619d9b640 1 -- 172.21.15.1:0/1089906884 >> v1:172.21.15.98:6789/0 conn(0x7f761c070c80 legacy=0x7f761c15d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.778+0000 7f7619d9b640 1 -- 172.21.15.1:0/1089906884 >> v1:172.21.15.1:6789/0 conn(0x7f761c150520 legacy=0x7f761c1537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.778+0000 7f7619d9b640 1 -- 172.21.15.1:0/1089906884 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c157050 con 0x7f761c0c32f0 2024-03-06T07:27:39.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.779+0000 7f7619d9b640 1 --2- 172.21.15.1:0/1089906884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f761c0c32f0 0x7f761c1543a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f76040097d0 tx=0x7f760402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2176ec2a073a67c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.779+0000 7f761959a640 1 -- 172.21.15.1:0/1089906884 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7604037070 con 0x7f761c0c32f0 2024-03-06T07:27:39.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.779+0000 7f761959a640 1 -- 172.21.15.1:0/1089906884 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7604030510 con 0x7f761c0c32f0 2024-03-06T07:27:39.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.779+0000 7f761959a640 1 -- 172.21.15.1:0/1089906884 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7614001670 con 0x7f761c0c32f0 2024-03-06T07:27:39.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.779+0000 7f762081c640 1 -- 172.21.15.1:0/1089906884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f761c0c32f0 msgr2=0x7f761c1543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.779+0000 7f762081c640 1 --2- 172.21.15.1:0/1089906884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f761c0c32f0 0x7f761c1543a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f76040097d0 tx=0x7f760402b800 comp rx=0 tx=0).stop 2024-03-06T07:27:39.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.779+0000 7f762081c640 1 -- 172.21.15.1:0/1089906884 shutdown_connections 2024-03-06T07:27:39.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.779+0000 7f762081c640 1 --2- 172.21.15.1:0/1089906884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f761c0c32f0 0x7f761c1543a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.779+0000 7f762081c640 1 -- 172.21.15.1:0/1089906884 >> 172.21.15.1:0/1089906884 conn(0x7f761c0c88f0 msgr2=0x7f761c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.780+0000 7f762081c640 1 -- 172.21.15.1:0/1089906884 shutdown_connections 2024-03-06T07:27:39.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.780+0000 7f762081c640 1 -- 172.21.15.1:0/1089906884 wait complete. 2024-03-06T07:27:39.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.781+0000 7f762081c640 1 Processor -- start 2024-03-06T07:27:39.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.781+0000 7f762081c640 1 -- start start 2024-03-06T07:27:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.782+0000 7f762081c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f761c070c80 0x7f761c0c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.782+0000 7f762081c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f761c0c32f0 0x7f761c156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.782+0000 7f762081c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f761c150520 0x7f761c146070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.782+0000 7f762081c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761c157330 con 0x7f761c070c80 2024-03-06T07:27:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.782+0000 7f762081c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761c1532a0 con 0x7f761c0c32f0 2024-03-06T07:27:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.782+0000 7f762081c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f761c0c7ec0 con 0x7f761c150520 2024-03-06T07:27:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.782+0000 7f761a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f761c070c80 0x7f761c0c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.782+0000 7f761a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f761c070c80 0x7f761c0c49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58944/0 (socket says 172.21.15.1:58944) 2024-03-06T07:27:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.782+0000 7f761a59c640 1 -- 172.21.15.1:0/4156983233 learned_addr learned my addr 172.21.15.1:0/4156983233 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.782+0000 7f7619d9b640 1 --2- 172.21.15.1:0/4156983233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f761c0c32f0 0x7f761c156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.782+0000 7f761ad9d640 1 --2- 172.21.15.1:0/4156983233 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f761c150520 0x7f761c146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.782+0000 7f761a59c640 1 -- 172.21.15.1:0/4156983233 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f761c150520 msgr2=0x7f761c146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.782+0000 7f761a59c640 1 --2- 172.21.15.1:0/4156983233 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f761c150520 0x7f761c146070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.782+0000 7f761a59c640 1 -- 172.21.15.1:0/4156983233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f761c0c32f0 msgr2=0x7f761c156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.782+0000 7f761a59c640 1 --2- 172.21.15.1:0/4156983233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f761c0c32f0 0x7f761c156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.782+0000 7f761a59c640 1 -- 172.21.15.1:0/4156983233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c156ae0 con 0x7f761c070c80 2024-03-06T07:27:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.782+0000 7f761a59c640 1 --2- 172.21.15.1:0/4156983233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f761c070c80 0x7f761c0c49b0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f761400c550 tx=0x7f761400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.783+0000 7f7612ffd640 1 -- 172.21.15.1:0/4156983233 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7614017070 con 0x7f761c070c80 2024-03-06T07:27:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.783+0000 7f7612ffd640 1 -- 172.21.15.1:0/4156983233 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7614005d00 con 0x7f761c070c80 2024-03-06T07:27:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.783+0000 7f7612ffd640 1 -- 172.21.15.1:0/4156983233 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76140072b0 con 0x7f761c070c80 2024-03-06T07:27:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.783+0000 7f762081c640 1 -- 172.21.15.1:0/4156983233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761c1ec220 con 0x7f761c070c80 2024-03-06T07:27:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.783+0000 7f762081c640 1 -- 172.21.15.1:0/4156983233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f761c1ec7e0 con 0x7f761c070c80 2024-03-06T07:27:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.784+0000 7f762081c640 1 -- 172.21.15.1:0/4156983233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f761c0686e0 con 0x7f761c070c80 2024-03-06T07:27:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.785+0000 7f7612ffd640 1 -- 172.21.15.1:0/4156983233 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f761401a070 con 0x7f761c070c80 2024-03-06T07:27:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.785+0000 7f7612ffd640 1 --2- 172.21.15.1:0/4156983233 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75fc06b500 0x7f75fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.785+0000 7f7612ffd640 1 -- 172.21.15.1:0/4156983233 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f76140897b0 con 0x7f761c070c80 2024-03-06T07:27:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.785+0000 7f7619d9b640 1 --2- 172.21.15.1:0/4156983233 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75fc06b500 0x7f75fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.786+0000 7f7619d9b640 1 --2- 172.21.15.1:0/4156983233 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75fc06b500 0x7f75fc06d9c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f7604034000 tx=0x7f7604008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:39.793+0000 7f7612ffd640 1 -- 172.21.15.1:0/4156983233 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7614057e50 con 0x7f761c070c80 2024-03-06T07:27:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.079+0000 7f762081c640 1 -- 172.21.15.1:0/4156983233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f761c0c7b10 con 0x7f761c070c80 2024-03-06T07:27:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.080+0000 7f7612ffd640 1 -- 172.21.15.1:0/4156983233 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+4969 (secure 0 0 0) 0x7f761405bef0 con 0x7f761c070c80 2024-03-06T07:27:40.082 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:40.082 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":164,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.619091+0000","modified":"2024-03-06T07:27:34.364010+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9738":{"gid":9738,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":2,"addr":"172.21.15.1:6835/1021164277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1021164277},{"type":"v1","addr":"172.21.15.1:6835","nonce":1021164277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9738,"qdb_cluster":[9738]},"id":13}]} 2024-03-06T07:27:40.082 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 164 2024-03-06T07:27:40.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.081+0000 7f762081c640 1 -- 172.21.15.1:0/4156983233 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75fc06b500 msgr2=0x7f75fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.081+0000 7f762081c640 1 --2- 172.21.15.1:0/4156983233 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75fc06b500 0x7f75fc06d9c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f7604034000 tx=0x7f7604008030 comp rx=0 tx=0).stop 2024-03-06T07:27:40.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.081+0000 7f762081c640 1 -- 172.21.15.1:0/4156983233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f761c070c80 msgr2=0x7f761c0c49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.082+0000 7f762081c640 1 --2- 172.21.15.1:0/4156983233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f761c070c80 0x7f761c0c49b0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f761400c550 tx=0x7f761400ca10 comp rx=0 tx=0).stop 2024-03-06T07:27:40.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.082+0000 7f762081c640 1 -- 172.21.15.1:0/4156983233 shutdown_connections 2024-03-06T07:27:40.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.082+0000 7f762081c640 1 --2- 172.21.15.1:0/4156983233 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75fc06b500 0x7f75fc06d9c0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.082+0000 7f762081c640 1 --2- 172.21.15.1:0/4156983233 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f761c150520 0x7f761c146070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.082+0000 7f762081c640 1 --2- 172.21.15.1:0/4156983233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f761c0c32f0 0x7f761c156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.082+0000 7f762081c640 1 --2- 172.21.15.1:0/4156983233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f761c070c80 0x7f761c0c49b0 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.082+0000 7f762081c640 1 -- 172.21.15.1:0/4156983233 >> 172.21.15.1:0/4156983233 conn(0x7f761c0c88f0 msgr2=0x7f761c14c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:40.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.082+0000 7f762081c640 1 -- 172.21.15.1:0/4156983233 shutdown_connections 2024-03-06T07:27:40.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.082+0000 7f762081c640 1 -- 172.21.15.1:0/4156983233 wait complete. 2024-03-06T07:27:40.094 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:40.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.197+0000 7f11bfff7640 1 Processor -- start 2024-03-06T07:27:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.197+0000 7f11bfff7640 1 -- start start 2024-03-06T07:27:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.198+0000 7f11bfff7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11b8150520 0x7f11b81543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.198+0000 7f11bfff7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11b80c32f0 0x7f11b81537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.198+0000 7f11bfff7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b8150900 con 0x7f11b8070c80 2024-03-06T07:27:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.198+0000 7f11bfff7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b8071790 con 0x7f11b8150520 2024-03-06T07:27:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.198+0000 7f11bfff7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b8157050 con 0x7f11b80c32f0 2024-03-06T07:27:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.198+0000 7f11bdd93640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11b8150520 0x7f11b81543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.198+0000 7f11be594640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f11b8070c80 0x7f11b80c67f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34022/0 (socket says 172.21.15.1:34022) 2024-03-06T07:27:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.198+0000 7f11bdd93640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11b8150520 0x7f11b81543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58956/0 (socket says 172.21.15.1:58956) 2024-03-06T07:27:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.198+0000 7f11be594640 1 -- 172.21.15.1:0/474396237 learned_addr learned my addr 172.21.15.1:0/474396237 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.198+0000 7f11bd592640 1 --2- 172.21.15.1:0/474396237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11b80c32f0 0x7f11b81537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.198+0000 7f11bdd93640 1 -- 172.21.15.1:0/474396237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11b80c32f0 msgr2=0x7f11b81537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:40.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.198+0000 7f11bdd93640 1 --2- 172.21.15.1:0/474396237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11b80c32f0 0x7f11b81537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.198+0000 7f11bdd93640 1 -- 172.21.15.1:0/474396237 >> v1:172.21.15.1:6789/0 conn(0x7f11b8070c80 legacy=0x7f11b80c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.199+0000 7f11bcd91640 1 -- 172.21.15.1:0/474396237 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884223184 0 0) 0x7f11b8150900 con 0x7f11b8070c80 2024-03-06T07:27:40.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.199+0000 7f11bdd93640 1 -- 172.21.15.1:0/474396237 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b8152fc0 con 0x7f11b8150520 2024-03-06T07:27:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.199+0000 7f11bdd93640 1 --2- 172.21.15.1:0/474396237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11b8150520 0x7f11b81543a0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f11b0009d90 tx=0x7f11b002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6856ceb898dcf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.199+0000 7f11bcd91640 1 -- 172.21.15.1:0/474396237 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b0039070 con 0x7f11b8150520 2024-03-06T07:27:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.199+0000 7f11bcd91640 1 -- 172.21.15.1:0/474396237 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b002cc50 con 0x7f11b8150520 2024-03-06T07:27:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.199+0000 7f11bcd91640 1 -- 172.21.15.1:0/474396237 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b0034440 con 0x7f11b8150520 2024-03-06T07:27:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.200+0000 7f11bfff7640 1 -- 172.21.15.1:0/474396237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11b8150520 msgr2=0x7f11b81543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.200+0000 7f11bfff7640 1 --2- 172.21.15.1:0/474396237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11b8150520 0x7f11b81543a0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f11b0009d90 tx=0x7f11b002bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.200+0000 7f11bfff7640 1 -- 172.21.15.1:0/474396237 shutdown_connections 2024-03-06T07:27:40.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.200+0000 7f11bfff7640 1 --2- 172.21.15.1:0/474396237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11b80c32f0 0x7f11b81537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.200+0000 7f11bfff7640 1 --2- 172.21.15.1:0/474396237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11b8150520 0x7f11b81543a0 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.200+0000 7f11bfff7640 1 -- 172.21.15.1:0/474396237 >> 172.21.15.1:0/474396237 conn(0x7f11b80c88f0 msgr2=0x7f11b80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:40.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.200+0000 7f11bfff7640 1 -- 172.21.15.1:0/474396237 shutdown_connections 2024-03-06T07:27:40.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.200+0000 7f11bfff7640 1 -- 172.21.15.1:0/474396237 wait complete. 2024-03-06T07:27:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.202+0000 7f11bfff7640 1 Processor -- start 2024-03-06T07:27:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.202+0000 7f11bfff7640 1 -- start start 2024-03-06T07:27:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11bfff7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11b8070c80 0x7f11b80c2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11bfff7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11b80c32f0 0x7f11b8143790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11bfff7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b8150520 0x7f11b8145360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11bfff7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b80c7ec0 con 0x7f11b80c32f0 2024-03-06T07:27:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11bfff7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b8157330 con 0x7f11b8150520 2024-03-06T07:27:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11bfff7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b8153d00 con 0x7f11b8070c80 2024-03-06T07:27:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11bd592640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11b80c32f0 0x7f11b8143790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11bd592640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11b80c32f0 0x7f11b8143790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58966/0 (socket says 172.21.15.1:58966) 2024-03-06T07:27:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11bd592640 1 -- 172.21.15.1:0/2089210417 learned_addr learned my addr 172.21.15.1:0/2089210417 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11bdd93640 1 --2- 172.21.15.1:0/2089210417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11b8070c80 0x7f11b80c2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11bd592640 1 -- 172.21.15.1:0/2089210417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11b8070c80 msgr2=0x7f11b80c2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11bd592640 1 --2- 172.21.15.1:0/2089210417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11b8070c80 0x7f11b80c2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11bd592640 1 -- 172.21.15.1:0/2089210417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b8150520 msgr2=0x7f11b8145360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11be594640 1 --2- 172.21.15.1:0/2089210417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b8150520 0x7f11b8145360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11bd592640 1 --2- 172.21.15.1:0/2089210417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b8150520 0x7f11b8145360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11bd592640 1 -- 172.21.15.1:0/2089210417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b8146010 con 0x7f11b80c32f0 2024-03-06T07:27:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11bdd93640 1 --2- 172.21.15.1:0/2089210417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11b8070c80 0x7f11b80c2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:40.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.203+0000 7f11be594640 1 --2- 172.21.15.1:0/2089210417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b8150520 0x7f11b8145360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.204+0000 7f11bd592640 1 --2- 172.21.15.1:0/2089210417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11b80c32f0 0x7f11b8143790 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f11a800d830 tx=0x7f11a800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.204+0000 7f11ae7fc640 1 -- 172.21.15.1:0/2089210417 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a800b6c0 con 0x7f11b80c32f0 2024-03-06T07:27:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.204+0000 7f11ae7fc640 1 -- 172.21.15.1:0/2089210417 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a8019070 con 0x7f11b80c32f0 2024-03-06T07:27:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.204+0000 7f11ae7fc640 1 -- 172.21.15.1:0/2089210417 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a800ba80 con 0x7f11b80c32f0 2024-03-06T07:27:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.204+0000 7f11bfff7640 1 -- 172.21.15.1:0/2089210417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b8155500 con 0x7f11b80c32f0 2024-03-06T07:27:40.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.204+0000 7f11bfff7640 1 -- 172.21.15.1:0/2089210417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b8155df0 con 0x7f11b80c32f0 2024-03-06T07:27:40.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.205+0000 7f11bfff7640 1 -- 172.21.15.1:0/2089210417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b806fc70 con 0x7f11b80c32f0 2024-03-06T07:27:40.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.205+0000 7f11ae7fc640 1 -- 172.21.15.1:0/2089210417 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11a801e460 con 0x7f11b80c32f0 2024-03-06T07:27:40.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.206+0000 7f11ae7fc640 1 --2- 172.21.15.1:0/2089210417 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f119c06b500 0x7f119c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.206+0000 7f11ae7fc640 1 -- 172.21.15.1:0/2089210417 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f11a808a9a0 con 0x7f11b80c32f0 2024-03-06T07:27:40.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.206+0000 7f11bdd93640 1 --2- 172.21.15.1:0/2089210417 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f119c06b500 0x7f119c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.206+0000 7f11bdd93640 1 --2- 172.21.15.1:0/2089210417 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f119c06b500 0x7f119c06d9c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f11b0036000 tx=0x7f11b0030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.210+0000 7f11ae7fc640 1 -- 172.21.15.1:0/2089210417 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11b806fc70 con 0x7f11b80c32f0 2024-03-06T07:27:40.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.393+0000 7f11bfff7640 1 -- 172.21.15.1:0/2089210417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11b806f4e0 con 0x7f11b80c32f0 2024-03-06T07:27:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.394+0000 7f11ae7fc640 1 -- 172.21.15.1:0/2089210417 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+16832 (secure 0 0 0) 0x7f11b806f4e0 con 0x7f11b80c32f0 2024-03-06T07:27:40.396 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:40.397 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":171,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:27:06.270883+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:02.888292+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:03.804423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.396+0000 7f11bfff7640 1 -- 172.21.15.1:0/2089210417 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f119c06b500 msgr2=0x7f119c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.396+0000 7f11bfff7640 1 --2- 172.21.15.1:0/2089210417 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f119c06b500 0x7f119c06d9c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f11b0036000 tx=0x7f11b0030000 comp rx=0 tx=0).stop 2024-03-06T07:27:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.396+0000 7f11bfff7640 1 -- 172.21.15.1:0/2089210417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11b80c32f0 msgr2=0x7f11b8143790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.396+0000 7f11bfff7640 1 --2- 172.21.15.1:0/2089210417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11b80c32f0 0x7f11b8143790 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f11a800d830 tx=0x7f11a800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.396+0000 7f11bfff7640 1 -- 172.21.15.1:0/2089210417 shutdown_connections 2024-03-06T07:27:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.396+0000 7f11bfff7640 1 --2- 172.21.15.1:0/2089210417 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f119c06b500 0x7f119c06d9c0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.396+0000 7f11bfff7640 1 --2- 172.21.15.1:0/2089210417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b8150520 0x7f11b8145360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.396+0000 7f11bfff7640 1 --2- 172.21.15.1:0/2089210417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11b80c32f0 0x7f11b8143790 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.396+0000 7f11bfff7640 1 --2- 172.21.15.1:0/2089210417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11b8070c80 0x7f11b80c2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.396+0000 7f11bfff7640 1 -- 172.21.15.1:0/2089210417 >> 172.21.15.1:0/2089210417 conn(0x7f11b80c88f0 msgr2=0x7f11b814ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:40.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.397+0000 7f11bfff7640 1 -- 172.21.15.1:0/2089210417 shutdown_connections 2024-03-06T07:27:40.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.397+0000 7f11bfff7640 1 -- 172.21.15.1:0/2089210417 wait complete. 2024-03-06T07:27:40.409 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:27:40.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.510+0000 7fb24f556640 1 Processor -- start 2024-03-06T07:27:40.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.510+0000 7fb24f556640 1 -- start start 2024-03-06T07:27:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.511+0000 7fb24f556640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb248152650 0x7fb248155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.511+0000 7fb24f556640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb2480bf9d0 0x7fb2480c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.511+0000 7fb24f556640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2480c2fc0 con 0x7fb2480c6b10 2024-03-06T07:27:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.511+0000 7fb24f556640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb248155800 con 0x7fb248152650 2024-03-06T07:27:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.511+0000 7fb24f556640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb248158da0 con 0x7fb2480bf9d0 2024-03-06T07:27:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.511+0000 7fb24d2f2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb248152650 0x7fb248155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.511+0000 7fb24d2f2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb248152650 0x7fb248155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58982/0 (socket says 172.21.15.1:58982) 2024-03-06T07:27:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.511+0000 7fb24d2f2640 1 -- 172.21.15.1:0/2080604995 learned_addr learned my addr 172.21.15.1:0/2080604995 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.512+0000 7fb24d2f2640 1 -- 172.21.15.1:0/2080604995 >> v2:172.21.15.98:3300/0 conn(0x7fb2480bf9d0 msgr2=0x7fb2480c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.512+0000 7fb24d2f2640 1 --2- 172.21.15.1:0/2080604995 >> v2:172.21.15.98:3300/0 conn(0x7fb2480bf9d0 0x7fb2480c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.512+0000 7fb24d2f2640 1 -- 172.21.15.1:0/2080604995 >> v1:172.21.15.98:6789/0 conn(0x7fb2480c6b10 legacy=0x7fb248158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.512+0000 7fb24d2f2640 1 -- 172.21.15.1:0/2080604995 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2480c6690 con 0x7fb248152650 2024-03-06T07:27:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.513+0000 7fb24d2f2640 1 --2- 172.21.15.1:0/2080604995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb248152650 0x7fb248155c90 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7fb2481597b0 tx=0x7fb24402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f116bd2c554b7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.513+0000 7fb23ffff640 1 -- 172.21.15.1:0/2080604995 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24402e070 con 0x7fb248152650 2024-03-06T07:27:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.513+0000 7fb23ffff640 1 -- 172.21.15.1:0/2080604995 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb244037070 con 0x7fb248152650 2024-03-06T07:27:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.513+0000 7fb23ffff640 1 -- 172.21.15.1:0/2080604995 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244031a00 con 0x7fb248152650 2024-03-06T07:27:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.513+0000 7fb24f556640 1 -- 172.21.15.1:0/2080604995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb248152650 msgr2=0x7fb248155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.513+0000 7fb24f556640 1 --2- 172.21.15.1:0/2080604995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb248152650 0x7fb248155c90 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7fb2481597b0 tx=0x7fb24402b880 comp rx=0 tx=0).stop 2024-03-06T07:27:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.513+0000 7fb24f556640 1 -- 172.21.15.1:0/2080604995 shutdown_connections 2024-03-06T07:27:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.513+0000 7fb24f556640 1 --2- 172.21.15.1:0/2080604995 >> v2:172.21.15.98:3300/0 conn(0x7fb2480bf9d0 0x7fb2480c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.513+0000 7fb24f556640 1 --2- 172.21.15.1:0/2080604995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb248152650 0x7fb248155c90 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.513+0000 7fb24f556640 1 -- 172.21.15.1:0/2080604995 >> 172.21.15.1:0/2080604995 conn(0x7fb248066ee0 msgr2=0x7fb2480672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.513+0000 7fb24f556640 1 -- 172.21.15.1:0/2080604995 shutdown_connections 2024-03-06T07:27:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.514+0000 7fb24f556640 1 -- 172.21.15.1:0/2080604995 wait complete. 2024-03-06T07:27:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.515+0000 7fb24f556640 1 Processor -- start 2024-03-06T07:27:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.515+0000 7fb24f556640 1 -- start start 2024-03-06T07:27:40.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.515+0000 7fb24f556640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2480bf9d0 0x7fb24814c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.515+0000 7fb24f556640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb248152650 0x7fb24814e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.515+0000 7fb24f556640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2481eb7b0 0x7fb24814b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.515+0000 7fb24f556640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2481551b0 con 0x7fb2480bf9d0 2024-03-06T07:27:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.515+0000 7fb24f556640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2480c6fa0 con 0x7fb248152650 2024-03-06T07:27:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.515+0000 7fb24f556640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb248155490 con 0x7fb2481eb7b0 2024-03-06T07:27:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.515+0000 7fb24d2f2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2480bf9d0 0x7fb24814c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.515+0000 7fb24d2f2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2480bf9d0 0x7fb24814c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58998/0 (socket says 172.21.15.1:58998) 2024-03-06T07:27:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.515+0000 7fb24d2f2640 1 -- 172.21.15.1:0/2734800389 learned_addr learned my addr 172.21.15.1:0/2734800389 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.516+0000 7fb24caf1640 1 --2- 172.21.15.1:0/2734800389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb248152650 0x7fb24814e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.516+0000 7fb24daf3640 1 --2- 172.21.15.1:0/2734800389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2481eb7b0 0x7fb24814b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.516+0000 7fb24d2f2640 1 -- 172.21.15.1:0/2734800389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2481eb7b0 msgr2=0x7fb24814b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.516+0000 7fb24d2f2640 1 --2- 172.21.15.1:0/2734800389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2481eb7b0 0x7fb24814b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.516+0000 7fb24d2f2640 1 -- 172.21.15.1:0/2734800389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb248152650 msgr2=0x7fb24814e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.516+0000 7fb24d2f2640 1 --2- 172.21.15.1:0/2734800389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb248152650 0x7fb24814e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.516+0000 7fb24d2f2640 1 -- 172.21.15.1:0/2734800389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24814f3e0 con 0x7fb2480bf9d0 2024-03-06T07:27:40.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.516+0000 7fb24caf1640 1 --2- 172.21.15.1:0/2734800389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb248152650 0x7fb24814e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:40.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.516+0000 7fb24d2f2640 1 --2- 172.21.15.1:0/2734800389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2480bf9d0 0x7fb24814c380 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fb244000c00 tx=0x7fb2440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.516+0000 7fb23dffb640 1 -- 172.21.15.1:0/2734800389 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244032880 con 0x7fb2480bf9d0 2024-03-06T07:27:40.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.516+0000 7fb23dffb640 1 -- 172.21.15.1:0/2734800389 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb244032a20 con 0x7fb2480bf9d0 2024-03-06T07:27:40.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.516+0000 7fb24f556640 1 -- 172.21.15.1:0/2734800389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb248152bd0 con 0x7fb2480bf9d0 2024-03-06T07:27:40.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.516+0000 7fb23dffb640 1 -- 172.21.15.1:0/2734800389 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244032e10 con 0x7fb2480bf9d0 2024-03-06T07:27:40.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.516+0000 7fb24f556640 1 -- 172.21.15.1:0/2734800389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24814ec50 con 0x7fb2480bf9d0 2024-03-06T07:27:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.518+0000 7fb24f556640 1 -- 172.21.15.1:0/2734800389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb248067fd0 con 0x7fb2480bf9d0 2024-03-06T07:27:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.518+0000 7fb23dffb640 1 -- 172.21.15.1:0/2734800389 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb24404c060 con 0x7fb2480bf9d0 2024-03-06T07:27:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.518+0000 7fb23dffb640 1 --2- 172.21.15.1:0/2734800389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb22c06b4d0 0x7fb22c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.518+0000 7fb23dffb640 1 -- 172.21.15.1:0/2734800389 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb2440abf30 con 0x7fb2480bf9d0 2024-03-06T07:27:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.519+0000 7fb24caf1640 1 --2- 172.21.15.1:0/2734800389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb22c06b4d0 0x7fb22c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.519+0000 7fb24caf1640 1 --2- 172.21.15.1:0/2734800389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb22c06b4d0 0x7fb22c06d990 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb248151740 tx=0x7fb238009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.522+0000 7fb23dffb640 1 -- 172.21.15.1:0/2734800389 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb244040070 con 0x7fb2480bf9d0 2024-03-06T07:27:40.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.757+0000 7fb24f556640 1 -- 172.21.15.1:0/2734800389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb2480686e0 con 0x7fb2480bf9d0 2024-03-06T07:27:40.760 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:27:40.758+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:27:40.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.759+0000 7fb23dffb640 1 -- 172.21.15.1:0/2734800389 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v165) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb24407acb0 con 0x7fb2480bf9d0 2024-03-06T07:27:40.761 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:27:40.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.762+0000 7fb24f556640 1 -- 172.21.15.1:0/2734800389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb22c06b4d0 msgr2=0x7fb22c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.762+0000 7fb24f556640 1 --2- 172.21.15.1:0/2734800389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb22c06b4d0 0x7fb22c06d990 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb248151740 tx=0x7fb238009040 comp rx=0 tx=0).stop 2024-03-06T07:27:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.762+0000 7fb24f556640 1 -- 172.21.15.1:0/2734800389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2480bf9d0 msgr2=0x7fb24814c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.762+0000 7fb24f556640 1 --2- 172.21.15.1:0/2734800389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2480bf9d0 0x7fb24814c380 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fb244000c00 tx=0x7fb2440046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.762+0000 7fb24f556640 1 -- 172.21.15.1:0/2734800389 shutdown_connections 2024-03-06T07:27:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.762+0000 7fb24f556640 1 --2- 172.21.15.1:0/2734800389 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb22c06b4d0 0x7fb22c06d990 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.762+0000 7fb24f556640 1 --2- 172.21.15.1:0/2734800389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2481eb7b0 0x7fb24814b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.762+0000 7fb24f556640 1 --2- 172.21.15.1:0/2734800389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb248152650 0x7fb24814e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.762+0000 7fb24f556640 1 --2- 172.21.15.1:0/2734800389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2480bf9d0 0x7fb24814c380 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.762+0000 7fb24f556640 1 -- 172.21.15.1:0/2734800389 >> 172.21.15.1:0/2734800389 conn(0x7fb248066ee0 msgr2=0x7fb2480c5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.762+0000 7fb24f556640 1 -- 172.21.15.1:0/2734800389 shutdown_connections 2024-03-06T07:27:40.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.762+0000 7fb24f556640 1 -- 172.21.15.1:0/2734800389 wait complete. 2024-03-06T07:27:40.765 INFO:tasks.ceph.mds.c.smithi001.stderr: -178> 2024-03-06T07:27:36.357+0000 7f476c47d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:40.776 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:27:40.788 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:27:40.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.878+0000 7f2979b93640 1 Processor -- start 2024-03-06T07:27:40.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.878+0000 7f2979b93640 1 -- start start 2024-03-06T07:27:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.879+0000 7f2979b93640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2974150520 0x7f29741543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.879+0000 7f2979b93640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29740c32f0 0x7f29741537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.879+0000 7f2979b93640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2974070c80 0x7f29740710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.879+0000 7f2979b93640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2974150900 con 0x7f2974150520 2024-03-06T07:27:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.879+0000 7f2979b93640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2974071790 con 0x7f2974070c80 2024-03-06T07:27:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.879+0000 7f2979b93640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2974157050 con 0x7f29740c32f0 2024-03-06T07:27:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.879+0000 7f29737fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2974150520 0x7f29741543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.879+0000 7f29737fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2974150520 0x7f29741543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59020/0 (socket says 172.21.15.1:59020) 2024-03-06T07:27:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.879+0000 7f29737fe640 1 -- 172.21.15.1:0/2544056746 learned_addr learned my addr 172.21.15.1:0/2544056746 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.879+0000 7f296bfff640 1 --2- 172.21.15.1:0/2544056746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29740c32f0 0x7f29741537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.880+0000 7f2973fff640 1 --2- 172.21.15.1:0/2544056746 >> v2:172.21.15.98:3300/0 conn(0x7f2974070c80 0x7f29740710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.880+0000 7f29737fe640 1 -- 172.21.15.1:0/2544056746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29740c32f0 msgr2=0x7f29741537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.880+0000 7f29737fe640 1 --2- 172.21.15.1:0/2544056746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29740c32f0 0x7f29741537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.880+0000 7f29737fe640 1 -- 172.21.15.1:0/2544056746 >> v2:172.21.15.98:3300/0 conn(0x7f2974070c80 msgr2=0x7f29740710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:40.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.880+0000 7f29737fe640 1 --2- 172.21.15.1:0/2544056746 >> v2:172.21.15.98:3300/0 conn(0x7f2974070c80 0x7f29740710f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.880+0000 7f29737fe640 1 -- 172.21.15.1:0/2544056746 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2974152fc0 con 0x7f2974150520 2024-03-06T07:27:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.880+0000 7f29737fe640 1 --2- 172.21.15.1:0/2544056746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2974150520 0x7f29741543a0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f29740c9990 tx=0x7f296402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2835d2df2a77be01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.881+0000 7f2972ffd640 1 -- 172.21.15.1:0/2544056746 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2964035020 con 0x7f2974150520 2024-03-06T07:27:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.881+0000 7f2972ffd640 1 -- 172.21.15.1:0/2544056746 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2964031870 con 0x7f2974150520 2024-03-06T07:27:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.881+0000 7f2972ffd640 1 -- 172.21.15.1:0/2544056746 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2964031c40 con 0x7f2974150520 2024-03-06T07:27:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.881+0000 7f2979b93640 1 -- 172.21.15.1:0/2544056746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2974150520 msgr2=0x7f29741543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.881+0000 7f2979b93640 1 --2- 172.21.15.1:0/2544056746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2974150520 0x7f29741543a0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f29740c9990 tx=0x7f296402b910 comp rx=0 tx=0).stop 2024-03-06T07:27:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.881+0000 7f2979b93640 1 -- 172.21.15.1:0/2544056746 shutdown_connections 2024-03-06T07:27:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.881+0000 7f2979b93640 1 --2- 172.21.15.1:0/2544056746 >> v2:172.21.15.98:3300/0 conn(0x7f2974070c80 0x7f29740710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.881+0000 7f2979b93640 1 --2- 172.21.15.1:0/2544056746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29740c32f0 0x7f29741537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.881+0000 7f2979b93640 1 --2- 172.21.15.1:0/2544056746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2974150520 0x7f29741543a0 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.881+0000 7f2979b93640 1 -- 172.21.15.1:0/2544056746 >> 172.21.15.1:0/2544056746 conn(0x7f29740c88f0 msgr2=0x7f29740c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.881+0000 7f2979b93640 1 -- 172.21.15.1:0/2544056746 shutdown_connections 2024-03-06T07:27:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.881+0000 7f2979b93640 1 -- 172.21.15.1:0/2544056746 wait complete. 2024-03-06T07:27:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.882+0000 7f2979b93640 1 Processor -- start 2024-03-06T07:27:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.882+0000 7f2979b93640 1 -- start start 2024-03-06T07:27:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.883+0000 7f2979b93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2974070c80 0x7f2974145960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.883+0000 7f2979b93640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29740c32f0 0x7f297414e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.883+0000 7f2979b93640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2974150520 0x7f297414e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.883+0000 7f2979b93640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29740c7ec0 con 0x7f2974070c80 2024-03-06T07:27:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.883+0000 7f2979b93640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2974157330 con 0x7f2974150520 2024-03-06T07:27:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.883+0000 7f2979b93640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2974153d00 con 0x7f29740c32f0 2024-03-06T07:27:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.883+0000 7f29737fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2974070c80 0x7f2974145960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.883+0000 7f29737fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2974070c80 0x7f2974145960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59030/0 (socket says 172.21.15.1:59030) 2024-03-06T07:27:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.883+0000 7f29737fe640 1 -- 172.21.15.1:0/266448586 learned_addr learned my addr 172.21.15.1:0/266448586 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.883+0000 7f296bfff640 1 --2- 172.21.15.1:0/266448586 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29740c32f0 0x7f297414e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.883+0000 7f29737fe640 1 -- 172.21.15.1:0/266448586 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29740c32f0 msgr2=0x7f297414e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.883+0000 7f29737fe640 1 --2- 172.21.15.1:0/266448586 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29740c32f0 0x7f297414e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.883+0000 7f2973fff640 1 --2- 172.21.15.1:0/266448586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2974150520 0x7f297414e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.883+0000 7f29737fe640 1 -- 172.21.15.1:0/266448586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2974150520 msgr2=0x7f297414e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.883+0000 7f29737fe640 1 --2- 172.21.15.1:0/266448586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2974150520 0x7f297414e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.883+0000 7f29737fe640 1 -- 172.21.15.1:0/266448586 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2974156990 con 0x7f2974070c80 2024-03-06T07:27:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.883+0000 7f2973fff640 1 --2- 172.21.15.1:0/266448586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2974150520 0x7f297414e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.884+0000 7f29737fe640 1 --2- 172.21.15.1:0/266448586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2974070c80 0x7f2974145960 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f296402b950 tx=0x7f29640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.884+0000 7f2970ff9640 1 -- 172.21.15.1:0/266448586 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2964035040 con 0x7f2974070c80 2024-03-06T07:27:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.884+0000 7f2970ff9640 1 -- 172.21.15.1:0/266448586 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2964030d00 con 0x7f2974070c80 2024-03-06T07:27:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.884+0000 7f2979b93640 1 -- 172.21.15.1:0/266448586 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f297414b520 con 0x7f2974070c80 2024-03-06T07:27:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.884+0000 7f2970ff9640 1 -- 172.21.15.1:0/266448586 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29640456a0 con 0x7f2974070c80 2024-03-06T07:27:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.884+0000 7f2979b93640 1 -- 172.21.15.1:0/266448586 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29740a8990 con 0x7f2974070c80 2024-03-06T07:27:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.885+0000 7f2979b93640 1 -- 172.21.15.1:0/266448586 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f297406e900 con 0x7f2974070c80 2024-03-06T07:27:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.885+0000 7f2970ff9640 1 -- 172.21.15.1:0/266448586 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2964038020 con 0x7f2974070c80 2024-03-06T07:27:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.886+0000 7f2970ff9640 1 --2- 172.21.15.1:0/266448586 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f295406b4d0 0x7f295406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.886+0000 7f2970ff9640 1 -- 172.21.15.1:0/266448586 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f29640ac620 con 0x7f2974070c80 2024-03-06T07:27:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.886+0000 7f296bfff640 1 --2- 172.21.15.1:0/266448586 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f295406b4d0 0x7f295406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.887+0000 7f296bfff640 1 --2- 172.21.15.1:0/266448586 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f295406b4d0 0x7f295406d990 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2974146790 tx=0x7f295c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:40.890+0000 7f2970ff9640 1 -- 172.21.15.1:0/266448586 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f297406e900 con 0x7f2974070c80 2024-03-06T07:27:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.114+0000 7f2979b93640 1 -- 172.21.15.1:0/266448586 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f297406f4e0 con 0x7f2974070c80 2024-03-06T07:27:41.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.766+0000 7f2970ff9640 1 -- 172.21.15.1:0/266448586 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v166) v1 ==== 104+0+0 (secure 0 0 0) 0x7f297406f4e0 con 0x7f2974070c80 2024-03-06T07:27:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.769+0000 7f2979b93640 1 -- 172.21.15.1:0/266448586 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f295406b4d0 msgr2=0x7f295406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.769+0000 7f2979b93640 1 --2- 172.21.15.1:0/266448586 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f295406b4d0 0x7f295406d990 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2974146790 tx=0x7f295c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.769+0000 7f2979b93640 1 -- 172.21.15.1:0/266448586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2974070c80 msgr2=0x7f2974145960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.769+0000 7f2979b93640 1 --2- 172.21.15.1:0/266448586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2974070c80 0x7f2974145960 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f296402b950 tx=0x7f29640046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.769+0000 7f2979b93640 1 -- 172.21.15.1:0/266448586 shutdown_connections 2024-03-06T07:27:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.769+0000 7f2979b93640 1 --2- 172.21.15.1:0/266448586 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f295406b4d0 0x7f295406d990 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.770+0000 7f2979b93640 1 --2- 172.21.15.1:0/266448586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2974150520 0x7f297414e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.770+0000 7f2979b93640 1 --2- 172.21.15.1:0/266448586 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29740c32f0 0x7f297414e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.770+0000 7f2979b93640 1 --2- 172.21.15.1:0/266448586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2974070c80 0x7f2974145960 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.770+0000 7f2979b93640 1 -- 172.21.15.1:0/266448586 >> 172.21.15.1:0/266448586 conn(0x7f29740c88f0 msgr2=0x7f29740c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.770+0000 7f2979b93640 1 -- 172.21.15.1:0/266448586 shutdown_connections 2024-03-06T07:27:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.770+0000 7f2979b93640 1 -- 172.21.15.1:0/266448586 wait complete. 2024-03-06T07:27:41.786 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:27:41.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.935+0000 7fdc80288640 1 Processor -- start 2024-03-06T07:27:41.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.935+0000 7fdc80288640 1 -- start start 2024-03-06T07:27:41.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.936+0000 7fdc80288640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdc780c79c0 0x7fdc78154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.936+0000 7fdc80288640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc781571c0 con 0x7fdc780c3400 2024-03-06T07:27:41.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.936+0000 7fdc80288640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc78153130 con 0x7fdc78150750 2024-03-06T07:27:41.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.936+0000 7fdc80288640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc78154040 con 0x7fdc780c79c0 2024-03-06T07:27:41.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.936+0000 7fdc7d823640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdc780c3400 0x7fdc78070d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34038/0 (socket says 172.21.15.1:34038) 2024-03-06T07:27:41.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.936+0000 7fdc7d823640 1 -- 172.21.15.1:0/2954477712 learned_addr learned my addr 172.21.15.1:0/2954477712 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:41.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.936+0000 7fdc7e825640 1 --2- 172.21.15.1:0/2954477712 >> v2:172.21.15.98:3300/0 conn(0x7fdc780c79c0 0x7fdc78154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.937+0000 7fdc7d022640 1 -- 172.21.15.1:0/2954477712 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2406109142 0 0) 0x7fdc781571c0 con 0x7fdc780c3400 2024-03-06T07:27:41.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.937+0000 7fdc7d022640 1 -- 172.21.15.1:0/2954477712 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc60003520 con 0x7fdc780c3400 2024-03-06T07:27:41.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.937+0000 7fdc7d022640 1 -- 172.21.15.1:0/2954477712 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010295757 0 0) 0x7fdc78153130 con 0x7fdc78150750 2024-03-06T07:27:41.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.937+0000 7fdc7d022640 1 -- 172.21.15.1:0/2954477712 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc781571c0 con 0x7fdc78150750 2024-03-06T07:27:41.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.937+0000 7fdc7e825640 1 -- 172.21.15.1:0/2954477712 >> v1:172.21.15.98:6789/0 conn(0x7fdc78150750 legacy=0x7fdc78153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.937+0000 7fdc7e825640 1 -- 172.21.15.1:0/2954477712 >> v1:172.21.15.1:6789/0 conn(0x7fdc780c3400 legacy=0x7fdc78070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.937+0000 7fdc7e825640 1 -- 172.21.15.1:0/2954477712 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc78153410 con 0x7fdc780c79c0 2024-03-06T07:27:41.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.938+0000 7fdc7e825640 1 --2- 172.21.15.1:0/2954477712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780c79c0 0x7fdc78154510 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdc68009d90 tx=0x7fdc6802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4ad0523e3a800fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:41.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.938+0000 7fdc7d022640 1 -- 172.21.15.1:0/2954477712 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68039070 con 0x7fdc780c79c0 2024-03-06T07:27:41.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.938+0000 7fdc7d022640 1 -- 172.21.15.1:0/2954477712 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc6802cd00 con 0x7fdc780c79c0 2024-03-06T07:27:41.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.938+0000 7fdc7d022640 1 -- 172.21.15.1:0/2954477712 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68034640 con 0x7fdc780c79c0 2024-03-06T07:27:41.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.938+0000 7fdc80288640 1 -- 172.21.15.1:0/2954477712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780c79c0 msgr2=0x7fdc78154510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.938+0000 7fdc80288640 1 --2- 172.21.15.1:0/2954477712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780c79c0 0x7fdc78154510 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdc68009d90 tx=0x7fdc6802bf50 comp rx=0 tx=0).stop 2024-03-06T07:27:41.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.939+0000 7fdc80288640 1 -- 172.21.15.1:0/2954477712 shutdown_connections 2024-03-06T07:27:41.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.939+0000 7fdc80288640 1 --2- 172.21.15.1:0/2954477712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780c79c0 0x7fdc78154510 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.939+0000 7fdc80288640 1 -- 172.21.15.1:0/2954477712 >> 172.21.15.1:0/2954477712 conn(0x7fdc7806b620 msgr2=0x7fdc780c8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:41.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.939+0000 7fdc80288640 1 -- 172.21.15.1:0/2954477712 shutdown_connections 2024-03-06T07:27:41.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.939+0000 7fdc80288640 1 -- 172.21.15.1:0/2954477712 wait complete. 2024-03-06T07:27:41.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.940+0000 7fdc80288640 1 Processor -- start 2024-03-06T07:27:41.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.940+0000 7fdc80288640 1 -- start start 2024-03-06T07:27:41.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.940+0000 7fdc80288640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780c3400 0x7fdc780ad680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.940+0000 7fdc80288640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc780c79c0 0x7fdc780abab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.940+0000 7fdc80288640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc78150750 0x7fdc781e7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.940+0000 7fdc80288640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc780c7e40 con 0x7fdc78150750 2024-03-06T07:27:41.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.940+0000 7fdc80288640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc781574a0 con 0x7fdc780c3400 2024-03-06T07:27:41.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.940+0000 7fdc80288640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc780c7fc0 con 0x7fdc780c79c0 2024-03-06T07:27:41.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.940+0000 7fdc7e825640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc78150750 0x7fdc781e7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.940+0000 7fdc7e825640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc78150750 0x7fdc781e7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59038/0 (socket says 172.21.15.1:59038) 2024-03-06T07:27:41.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.940+0000 7fdc7e825640 1 -- 172.21.15.1:0/3063472881 learned_addr learned my addr 172.21.15.1:0/3063472881 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:41.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.941+0000 7fdc7e024640 1 --2- 172.21.15.1:0/3063472881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780c3400 0x7fdc780ad680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.941+0000 7fdc7d823640 1 --2- 172.21.15.1:0/3063472881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc780c79c0 0x7fdc780abab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.941+0000 7fdc7e825640 1 -- 172.21.15.1:0/3063472881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc780c79c0 msgr2=0x7fdc780abab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.941+0000 7fdc7e825640 1 --2- 172.21.15.1:0/3063472881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc780c79c0 0x7fdc780abab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.941+0000 7fdc7e825640 1 -- 172.21.15.1:0/3063472881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780c3400 msgr2=0x7fdc780ad680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.941+0000 7fdc7e825640 1 --2- 172.21.15.1:0/3063472881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780c3400 0x7fdc780ad680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.941+0000 7fdc7e825640 1 -- 172.21.15.1:0/3063472881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc780c2b80 con 0x7fdc78150750 2024-03-06T07:27:41.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.941+0000 7fdc7e825640 1 --2- 172.21.15.1:0/3063472881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc78150750 0x7fdc781e7710 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fdc68009eb0 tx=0x7fdc68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:41.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.941+0000 7fdc76ffd640 1 -- 172.21.15.1:0/3063472881 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68039050 con 0x7fdc78150750 2024-03-06T07:27:41.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.941+0000 7fdc76ffd640 1 -- 172.21.15.1:0/3063472881 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc68045070 con 0x7fdc78150750 2024-03-06T07:27:41.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.941+0000 7fdc80288640 1 -- 172.21.15.1:0/3063472881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc781eae20 con 0x7fdc78150750 2024-03-06T07:27:41.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.941+0000 7fdc76ffd640 1 -- 172.21.15.1:0/3063472881 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68009450 con 0x7fdc78150750 2024-03-06T07:27:41.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.941+0000 7fdc80288640 1 -- 172.21.15.1:0/3063472881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc781eb100 con 0x7fdc78150750 2024-03-06T07:27:41.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.942+0000 7fdc80288640 1 -- 172.21.15.1:0/3063472881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc780686e0 con 0x7fdc78150750 2024-03-06T07:27:41.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.943+0000 7fdc76ffd640 1 -- 172.21.15.1:0/3063472881 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc6803e830 con 0x7fdc78150750 2024-03-06T07:27:41.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.943+0000 7fdc76ffd640 1 --2- 172.21.15.1:0/3063472881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdc6406b4d0 0x7fdc6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.943+0000 7fdc76ffd640 1 -- 172.21.15.1:0/3063472881 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fdc680ad080 con 0x7fdc78150750 2024-03-06T07:27:41.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.943+0000 7fdc7e024640 1 --2- 172.21.15.1:0/3063472881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdc6406b4d0 0x7fdc6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.944+0000 7fdc7e024640 1 --2- 172.21.15.1:0/3063472881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdc6406b4d0 0x7fdc6406d990 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fdc700015b0 tx=0x7fdc70009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:41.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:41.947+0000 7fdc76ffd640 1 -- 172.21.15.1:0/3063472881 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc68032e20 con 0x7fdc78150750 2024-03-06T07:27:42.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.130+0000 7fdc80288640 1 -- 172.21.15.1:0/3063472881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdc7806ec90 con 0x7fdc78150750 2024-03-06T07:27:42.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.767+0000 7fdc76ffd640 1 -- 172.21.15.1:0/3063472881 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v173) v1 ==== 181+0+0 (secure 0 0 0) 0x7fdc6807be40 con 0x7fdc78150750 2024-03-06T07:27:42.770 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:27:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.771+0000 7fdc80288640 1 -- 172.21.15.1:0/3063472881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdc6406b4d0 msgr2=0x7fdc6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.771+0000 7fdc80288640 1 --2- 172.21.15.1:0/3063472881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdc6406b4d0 0x7fdc6406d990 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fdc700015b0 tx=0x7fdc70009040 comp rx=0 tx=0).stop 2024-03-06T07:27:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.771+0000 7fdc80288640 1 -- 172.21.15.1:0/3063472881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc78150750 msgr2=0x7fdc781e7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.771+0000 7fdc80288640 1 --2- 172.21.15.1:0/3063472881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc78150750 0x7fdc781e7710 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fdc68009eb0 tx=0x7fdc68004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:42.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.771+0000 7fdc80288640 1 -- 172.21.15.1:0/3063472881 shutdown_connections 2024-03-06T07:27:42.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.771+0000 7fdc80288640 1 --2- 172.21.15.1:0/3063472881 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdc6406b4d0 0x7fdc6406d990 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.771+0000 7fdc80288640 1 --2- 172.21.15.1:0/3063472881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc78150750 0x7fdc781e7710 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.771+0000 7fdc80288640 1 --2- 172.21.15.1:0/3063472881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc780c79c0 0x7fdc780abab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.771+0000 7fdc80288640 1 --2- 172.21.15.1:0/3063472881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc780c3400 0x7fdc780ad680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.771+0000 7fdc80288640 1 -- 172.21.15.1:0/3063472881 >> 172.21.15.1:0/3063472881 conn(0x7fdc7806b620 msgr2=0x7fdc78154320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:42.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.772+0000 7fdc80288640 1 -- 172.21.15.1:0/3063472881 shutdown_connections 2024-03-06T07:27:42.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.772+0000 7fdc80288640 1 -- 172.21.15.1:0/3063472881 wait complete. 2024-03-06T07:27:42.790 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:27:42.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.899+0000 7f98f83ae640 1 Processor -- start 2024-03-06T07:27:42.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.899+0000 7f98f83ae640 1 -- start start 2024-03-06T07:27:42.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.899+0000 7f98f83ae640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98f0152780 0x7f98f00c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.899+0000 7f98f83ae640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f98f00c1010 0x7f98f00c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.899+0000 7f98f83ae640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f00c69b0 con 0x7f98f00c7500 2024-03-06T07:27:42.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.899+0000 7f98f83ae640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98f00c6c90 con 0x7f98f00c1010 2024-03-06T07:27:42.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.899+0000 7f98f83ae640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f0155370 con 0x7f98f0152780 2024-03-06T07:27:42.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f694b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f98f00c7500 0x7f98f00cab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34040/0 (socket says 172.21.15.1:34040) 2024-03-06T07:27:42.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f694b640 1 -- 172.21.15.1:0/3968991639 learned_addr learned my addr 172.21.15.1:0/3968991639 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:42.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5949640 1 --2- 172.21.15.1:0/3968991639 >> v2:172.21.15.98:3300/0 conn(0x7f98f00c1010 0x7f98f00c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f614a640 1 --2- 172.21.15.1:0/3968991639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98f0152780 0x7f98f00c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5148640 1 -- 172.21.15.1:0/3968991639 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266906408 0 0) 0x7f98f00c69b0 con 0x7f98f00c7500 2024-03-06T07:27:42.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5148640 1 -- 172.21.15.1:0/3968991639 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d8003520 con 0x7f98f00c7500 2024-03-06T07:27:42.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5148640 1 -- 172.21.15.1:0/3968991639 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1874472236 0 0) 0x7f98d8003520 con 0x7f98f00c7500 2024-03-06T07:27:42.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5148640 1 -- 172.21.15.1:0/3968991639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98f0152780 msgr2=0x7f98f00c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:42.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5148640 1 --2- 172.21.15.1:0/3968991639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98f0152780 0x7f98f00c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5148640 1 -- 172.21.15.1:0/3968991639 >> v2:172.21.15.98:3300/0 conn(0x7f98f00c1010 msgr2=0x7f98f00c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:42.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5148640 1 --2- 172.21.15.1:0/3968991639 >> v2:172.21.15.98:3300/0 conn(0x7f98f00c1010 0x7f98f00c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5148640 1 -- 172.21.15.1:0/3968991639 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f0155650 con 0x7f98f00c7500 2024-03-06T07:27:42.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5148640 1 -- 172.21.15.1:0/3968991639 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f98e00019f0 con 0x7f98f00c7500 2024-03-06T07:27:42.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5148640 1 -- 172.21.15.1:0/3968991639 >> v1:172.21.15.1:6789/0 conn(0x7f98f00c7500 legacy=0x7f98f00cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5148640 1 --2- 172.21.15.1:0/3968991639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98d80039b0 0x7f98d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5148640 1 --2- 172.21.15.1:0/3968991639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98d8004800 0x7f98d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5148640 1 --2- 172.21.15.1:0/3968991639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98f0152780 0x7f98d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5148640 1 -- 172.21.15.1:0/3968991639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f00c6c90 con 0x7f98d8004800 2024-03-06T07:27:42.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5148640 1 -- 172.21.15.1:0/3968991639 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f0155370 con 0x7f98d80039b0 2024-03-06T07:27:42.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5148640 1 -- 172.21.15.1:0/3968991639 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d800b9c0 con 0x7f98f0152780 2024-03-06T07:27:42.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f5949640 1 --2- 172.21.15.1:0/3968991639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98d8004800 0x7f98d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f614a640 1 --2- 172.21.15.1:0/3968991639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98d80039b0 0x7f98d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.900+0000 7f98f694b640 1 --2- 172.21.15.1:0/3968991639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98f0152780 0x7f98d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.901+0000 7f98f5949640 1 -- 172.21.15.1:0/3968991639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98f0152780 msgr2=0x7f98d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:42.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.901+0000 7f98f5949640 1 --2- 172.21.15.1:0/3968991639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98f0152780 0x7f98d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.901+0000 7f98f5949640 1 -- 172.21.15.1:0/3968991639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98d80039b0 msgr2=0x7f98d8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:42.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.901+0000 7f98f5949640 1 --2- 172.21.15.1:0/3968991639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98d80039b0 0x7f98d8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.901+0000 7f98f5949640 1 -- 172.21.15.1:0/3968991639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d8003520 con 0x7f98d8004800 2024-03-06T07:27:42.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.901+0000 7f98f614a640 1 --2- 172.21.15.1:0/3968991639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98d80039b0 0x7f98d8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:42.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.901+0000 7f98f5949640 1 --2- 172.21.15.1:0/3968991639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98d8004800 0x7f98d8004c80 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f98e4009ac0 tx=0x7f98d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31c4d0121998062 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.902+0000 7f98f5148640 1 -- 172.21.15.1:0/3968991639 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e4039070 con 0x7f98d8004800 2024-03-06T07:27:42.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.902+0000 7f98f5148640 1 -- 172.21.15.1:0/3968991639 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e402c6b0 con 0x7f98d8004800 2024-03-06T07:27:42.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.902+0000 7f98f5148640 1 -- 172.21.15.1:0/3968991639 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e4009690 con 0x7f98d8004800 2024-03-06T07:27:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.902+0000 7f98f83ae640 1 -- 172.21.15.1:0/3968991639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98d8004800 msgr2=0x7f98d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.902+0000 7f98f83ae640 1 --2- 172.21.15.1:0/3968991639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98d8004800 0x7f98d8004c80 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f98e4009ac0 tx=0x7f98d8005830 comp rx=0 tx=0).stop 2024-03-06T07:27:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.902+0000 7f98f694b640 1 -- 172.21.15.1:0/3968991639 reap_dead start 2024-03-06T07:27:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.902+0000 7f98f83ae640 1 -- 172.21.15.1:0/3968991639 shutdown_connections 2024-03-06T07:27:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.902+0000 7f98f83ae640 1 -- 172.21.15.1:0/3968991639 >> 172.21.15.1:0/3968991639 conn(0x7f98f008b720 msgr2=0x7f98f006d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.902+0000 7f98f83ae640 1 -- 172.21.15.1:0/3968991639 shutdown_connections 2024-03-06T07:27:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.902+0000 7f98f83ae640 1 -- 172.21.15.1:0/3968991639 wait complete. 2024-03-06T07:27:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.903+0000 7f98f83ae640 1 Processor -- start 2024-03-06T07:27:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.903+0000 7f98f83ae640 1 -- start start 2024-03-06T07:27:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.904+0000 7f98f83ae640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98d8004800 0x7f98f0066f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.904+0000 7f98f83ae640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98f01ea500 0x7f98f0146f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.904+0000 7f98f83ae640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98f01e2260 0x7f98f01474c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.904+0000 7f98f83ae640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f00cb210 con 0x7f98f01e2260 2024-03-06T07:27:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.904+0000 7f98f83ae640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f0152b60 con 0x7f98f01ea500 2024-03-06T07:27:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.904+0000 7f98f83ae640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f00cb390 con 0x7f98d8004800 2024-03-06T07:27:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.904+0000 7f98f694b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98f01e2260 0x7f98f01474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.904+0000 7f98f694b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98f01e2260 0x7f98f01474c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59062/0 (socket says 172.21.15.1:59062) 2024-03-06T07:27:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.904+0000 7f98f694b640 1 -- 172.21.15.1:0/329782027 learned_addr learned my addr 172.21.15.1:0/329782027 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.904+0000 7f98f614a640 1 --2- 172.21.15.1:0/329782027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98d8004800 0x7f98f0066f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.904+0000 7f98f5949640 1 --2- 172.21.15.1:0/329782027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98f01ea500 0x7f98f0146f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.904+0000 7f98f694b640 1 -- 172.21.15.1:0/329782027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98d8004800 msgr2=0x7f98f0066f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.904+0000 7f98f694b640 1 --2- 172.21.15.1:0/329782027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98d8004800 0x7f98f0066f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.904+0000 7f98f694b640 1 -- 172.21.15.1:0/329782027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98f01ea500 msgr2=0x7f98f0146f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.904+0000 7f98f694b640 1 --2- 172.21.15.1:0/329782027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98f01ea500 0x7f98f0146f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.904+0000 7f98f694b640 1 -- 172.21.15.1:0/329782027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f014b0c0 con 0x7f98f01e2260 2024-03-06T07:27:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.905+0000 7f98f614a640 1 --2- 172.21.15.1:0/329782027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98d8004800 0x7f98f0066f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.905+0000 7f98f5949640 1 --2- 172.21.15.1:0/329782027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98f01ea500 0x7f98f0146f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.905+0000 7f98f694b640 1 --2- 172.21.15.1:0/329782027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98f01e2260 0x7f98f01474c0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f98f00c00d0 tx=0x7f98e0007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.905+0000 7f98eeffd640 1 -- 172.21.15.1:0/329782027 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e0017020 con 0x7f98f01e2260 2024-03-06T07:27:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.905+0000 7f98eeffd640 1 -- 172.21.15.1:0/329782027 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e000abc0 con 0x7f98f01e2260 2024-03-06T07:27:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.905+0000 7f98f83ae640 1 -- 172.21.15.1:0/329782027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f0158430 con 0x7f98f01e2260 2024-03-06T07:27:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.905+0000 7f98eeffd640 1 -- 172.21.15.1:0/329782027 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8009690 con 0x7f98f01e2260 2024-03-06T07:27:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.905+0000 7f98f83ae640 1 -- 172.21.15.1:0/329782027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f01eace0 con 0x7f98f01e2260 2024-03-06T07:27:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.906+0000 7f98f83ae640 1 -- 172.21.15.1:0/329782027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f00680b0 con 0x7f98f01e2260 2024-03-06T07:27:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.906+0000 7f98eeffd640 1 -- 172.21.15.1:0/329782027 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98e001a020 con 0x7f98f01e2260 2024-03-06T07:27:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.907+0000 7f98eeffd640 1 --2- 172.21.15.1:0/329782027 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f98dc06b4d0 0x7f98dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.907+0000 7f98eeffd640 1 -- 172.21.15.1:0/329782027 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f98e008a850 con 0x7f98f01e2260 2024-03-06T07:27:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.907+0000 7f98f614a640 1 --2- 172.21.15.1:0/329782027 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f98dc06b4d0 0x7f98dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.907+0000 7f98f614a640 1 --2- 172.21.15.1:0/329782027 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f98dc06b4d0 0x7f98dc06d990 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f98e8009b00 tx=0x7f98e8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:42.911+0000 7f98eeffd640 1 -- 172.21.15.1:0/329782027 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98e0021020 con 0x7f98f01e2260 2024-03-06T07:27:43.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.094+0000 7f98f83ae640 1 -- 172.21.15.1:0/329782027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f98f00c5de0 con 0x7f98f01e2260 2024-03-06T07:27:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.773+0000 7f98eeffd640 1 -- 172.21.15.1:0/329782027 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v174) v1 ==== 169+0+0 (secure 0 0 0) 0x7f98e0059730 con 0x7f98f01e2260 2024-03-06T07:27:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-06T07:27:43.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.776+0000 7f98f83ae640 1 -- 172.21.15.1:0/329782027 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f98dc06b4d0 msgr2=0x7f98dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.776+0000 7f98f83ae640 1 --2- 172.21.15.1:0/329782027 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f98dc06b4d0 0x7f98dc06d990 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f98e8009b00 tx=0x7f98e8008040 comp rx=0 tx=0).stop 2024-03-06T07:27:43.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.776+0000 7f98f83ae640 1 -- 172.21.15.1:0/329782027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98f01e2260 msgr2=0x7f98f01474c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.776+0000 7f98f83ae640 1 --2- 172.21.15.1:0/329782027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98f01e2260 0x7f98f01474c0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f98f00c00d0 tx=0x7f98e0007b20 comp rx=0 tx=0).stop 2024-03-06T07:27:43.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.776+0000 7f98f83ae640 1 -- 172.21.15.1:0/329782027 shutdown_connections 2024-03-06T07:27:43.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.776+0000 7f98f83ae640 1 --2- 172.21.15.1:0/329782027 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f98dc06b4d0 0x7f98dc06d990 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.776+0000 7f98f83ae640 1 --2- 172.21.15.1:0/329782027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98f01e2260 0x7f98f01474c0 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.776+0000 7f98f83ae640 1 --2- 172.21.15.1:0/329782027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98f01ea500 0x7f98f0146f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.776+0000 7f98f83ae640 1 --2- 172.21.15.1:0/329782027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98d8004800 0x7f98f0066f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.777+0000 7f98f83ae640 1 -- 172.21.15.1:0/329782027 >> 172.21.15.1:0/329782027 conn(0x7f98f008b720 msgr2=0x7f98f0151800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:43.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.777+0000 7f98f83ae640 1 -- 172.21.15.1:0/329782027 shutdown_connections 2024-03-06T07:27:43.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.777+0000 7f98f83ae640 1 -- 172.21.15.1:0/329782027 wait complete. 2024-03-06T07:27:43.793 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:27:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.900+0000 7f9dd2b79640 1 Processor -- start 2024-03-06T07:27:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.900+0000 7f9dd2b79640 1 -- start start 2024-03-06T07:27:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.901+0000 7f9dd2b79640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9dcc06e670 0x7f9dcc06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.901+0000 7f9dd2b79640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9dcc0a8520 0x7f9dcc0c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.901+0000 7f9dd2b79640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dcc150290 0x7f9dcc150740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.901+0000 7f9dd2b79640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dcc0c2ca0 con 0x7f9dcc06e670 2024-03-06T07:27:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.901+0000 7f9dd2b79640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dcc06f060 con 0x7f9dcc0a8520 2024-03-06T07:27:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.901+0000 7f9dd2b79640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc0c26e0 con 0x7f9dcc150290 2024-03-06T07:27:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.901+0000 7f9dd0915640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9dcc06e670 0x7f9dcc06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.901+0000 7f9dd0915640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9dcc06e670 0x7f9dcc06ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59072/0 (socket says 172.21.15.1:59072) 2024-03-06T07:27:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.901+0000 7f9dd0915640 1 -- 172.21.15.1:0/2432042154 learned_addr learned my addr 172.21.15.1:0/2432042154 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.901+0000 7f9dd1116640 1 --2- 172.21.15.1:0/2432042154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dcc150290 0x7f9dcc150740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.901+0000 7f9dcbfff640 1 --2- 172.21.15.1:0/2432042154 >> v2:172.21.15.98:3300/0 conn(0x7f9dcc0a8520 0x7f9dcc0c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.901+0000 7f9dd0915640 1 -- 172.21.15.1:0/2432042154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dcc150290 msgr2=0x7f9dcc150740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.901+0000 7f9dd0915640 1 --2- 172.21.15.1:0/2432042154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dcc150290 0x7f9dcc150740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.901+0000 7f9dd0915640 1 -- 172.21.15.1:0/2432042154 >> v2:172.21.15.98:3300/0 conn(0x7f9dcc0a8520 msgr2=0x7f9dcc0c1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.901+0000 7f9dd0915640 1 --2- 172.21.15.1:0/2432042154 >> v2:172.21.15.98:3300/0 conn(0x7f9dcc0a8520 0x7f9dcc0c1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.901+0000 7f9dd0915640 1 -- 172.21.15.1:0/2432042154 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dcc0c29c0 con 0x7f9dcc06e670 2024-03-06T07:27:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.902+0000 7f9dd0915640 1 --2- 172.21.15.1:0/2432042154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dcc06e670 0x7f9dcc06ea50 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc072e60 tx=0x7f9dc002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae7abd253e4bf99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.902+0000 7f9dcb7fe640 1 -- 172.21.15.1:0/2432042154 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc002e070 con 0x7f9dcc06e670 2024-03-06T07:27:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.902+0000 7f9dcb7fe640 1 -- 172.21.15.1:0/2432042154 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc0037070 con 0x7f9dcc06e670 2024-03-06T07:27:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.902+0000 7f9dcb7fe640 1 -- 172.21.15.1:0/2432042154 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0031a00 con 0x7f9dcc06e670 2024-03-06T07:27:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.903+0000 7f9dd2b79640 1 -- 172.21.15.1:0/2432042154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dcc06e670 msgr2=0x7f9dcc06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.903+0000 7f9dd2b79640 1 --2- 172.21.15.1:0/2432042154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dcc06e670 0x7f9dcc06ea50 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc072e60 tx=0x7f9dc002b880 comp rx=0 tx=0).stop 2024-03-06T07:27:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.903+0000 7f9dd2b79640 1 -- 172.21.15.1:0/2432042154 shutdown_connections 2024-03-06T07:27:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.903+0000 7f9dd2b79640 1 --2- 172.21.15.1:0/2432042154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dcc150290 0x7f9dcc150740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.903+0000 7f9dd2b79640 1 --2- 172.21.15.1:0/2432042154 >> v2:172.21.15.98:3300/0 conn(0x7f9dcc0a8520 0x7f9dcc0c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.903+0000 7f9dd2b79640 1 --2- 172.21.15.1:0/2432042154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dcc06e670 0x7f9dcc06ea50 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.903+0000 7f9dd2b79640 1 -- 172.21.15.1:0/2432042154 >> 172.21.15.1:0/2432042154 conn(0x7f9dcc0c3cc0 msgr2=0x7f9dcc0c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.903+0000 7f9dd2b79640 1 -- 172.21.15.1:0/2432042154 shutdown_connections 2024-03-06T07:27:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.903+0000 7f9dd2b79640 1 -- 172.21.15.1:0/2432042154 wait complete. 2024-03-06T07:27:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.904+0000 7f9dd2b79640 1 Processor -- start 2024-03-06T07:27:43.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.904+0000 7f9dd2b79640 1 -- start start 2024-03-06T07:27:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd2b79640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9dcc06e670 0x7f9dcc1ebc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd2b79640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dcc0a8520 0x7f9dcc1519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd2b79640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dcc150290 0x7f9dcc0c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd2b79640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc153660 con 0x7f9dcc150290 2024-03-06T07:27:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd2b79640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc1534e0 con 0x7f9dcc06e670 2024-03-06T07:27:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd2b79640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc1537e0 con 0x7f9dcc0a8520 2024-03-06T07:27:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd1116640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dcc150290 0x7f9dcc0c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd1116640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dcc150290 0x7f9dcc0c5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59080/0 (socket says 172.21.15.1:59080) 2024-03-06T07:27:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd1116640 1 -- 172.21.15.1:0/2572213178 learned_addr learned my addr 172.21.15.1:0/2572213178 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dcbfff640 1 --2- 172.21.15.1:0/2572213178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dcc0a8520 0x7f9dcc1519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd0915640 1 --2- 172.21.15.1:0/2572213178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9dcc06e670 0x7f9dcc1ebc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd1116640 1 -- 172.21.15.1:0/2572213178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dcc0a8520 msgr2=0x7f9dcc1519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd1116640 1 --2- 172.21.15.1:0/2572213178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dcc0a8520 0x7f9dcc1519b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd1116640 1 -- 172.21.15.1:0/2572213178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9dcc06e670 msgr2=0x7f9dcc1ebc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd1116640 1 --2- 172.21.15.1:0/2572213178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9dcc06e670 0x7f9dcc1ebc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd1116640 1 -- 172.21.15.1:0/2572213178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dcc0c4db0 con 0x7f9dcc150290 2024-03-06T07:27:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd0915640 1 --2- 172.21.15.1:0/2572213178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9dcc06e670 0x7f9dcc1ebc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd1116640 1 --2- 172.21.15.1:0/2572213178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dcc150290 0x7f9dcc0c5ec0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f9dc400cc30 tx=0x7f9dc400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dc97fa640 1 -- 172.21.15.1:0/2572213178 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc400ce40 con 0x7f9dcc150290 2024-03-06T07:27:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dc97fa640 1 -- 172.21.15.1:0/2572213178 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc4003990 con 0x7f9dcc150290 2024-03-06T07:27:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd2b79640 1 -- 172.21.15.1:0/2572213178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dcc0c5090 con 0x7f9dcc150290 2024-03-06T07:27:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dc97fa640 1 -- 172.21.15.1:0/2572213178 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4003d40 con 0x7f9dcc150290 2024-03-06T07:27:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.905+0000 7f9dd2b79640 1 -- 172.21.15.1:0/2572213178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dcc0c56c0 con 0x7f9dcc150290 2024-03-06T07:27:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.906+0000 7f9dd2b79640 1 -- 172.21.15.1:0/2572213178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dcc0c5bb0 con 0x7f9dcc150290 2024-03-06T07:27:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.907+0000 7f9dc97fa640 1 -- 172.21.15.1:0/2572213178 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9dc4003ee0 con 0x7f9dcc150290 2024-03-06T07:27:43.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.907+0000 7f9dc97fa640 1 --2- 172.21.15.1:0/2572213178 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9db006b4d0 0x7f9db006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.907+0000 7f9dc97fa640 1 -- 172.21.15.1:0/2572213178 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f9dc408a400 con 0x7f9dcc150290 2024-03-06T07:27:43.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.908+0000 7f9dd0915640 1 --2- 172.21.15.1:0/2572213178 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9db006b4d0 0x7f9db006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.908+0000 7f9dd0915640 1 --2- 172.21.15.1:0/2572213178 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9db006b4d0 0x7f9db006d990 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0036000 tx=0x7f9dc003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:43.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:43.912+0000 7f9dc97fa640 1 -- 172.21.15.1:0/2572213178 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9dc4014030 con 0x7f9dcc150290 2024-03-06T07:27:44.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.091+0000 7f9dd2b79640 1 -- 172.21.15.1:0/2572213178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9dcc067fd0 con 0x7f9dcc150290 2024-03-06T07:27:44.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.094+0000 7f9dc97fa640 1 -- 172.21.15.1:0/2572213178 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v58) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9dc4058e30 con 0x7f9dcc150290 2024-03-06T07:27:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.096+0000 7f9dd2b79640 1 -- 172.21.15.1:0/2572213178 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9db006b4d0 msgr2=0x7f9db006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.096+0000 7f9dd2b79640 1 --2- 172.21.15.1:0/2572213178 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9db006b4d0 0x7f9db006d990 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0036000 tx=0x7f9dc003d040 comp rx=0 tx=0).stop 2024-03-06T07:27:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.096+0000 7f9dd2b79640 1 -- 172.21.15.1:0/2572213178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dcc150290 msgr2=0x7f9dcc0c5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.096+0000 7f9dd2b79640 1 --2- 172.21.15.1:0/2572213178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dcc150290 0x7f9dcc0c5ec0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f9dc400cc30 tx=0x7f9dc400a5c0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.097+0000 7f9dd2b79640 1 -- 172.21.15.1:0/2572213178 shutdown_connections 2024-03-06T07:27:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.097+0000 7f9dd2b79640 1 --2- 172.21.15.1:0/2572213178 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9db006b4d0 0x7f9db006d990 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.097+0000 7f9dd2b79640 1 --2- 172.21.15.1:0/2572213178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dcc150290 0x7f9dcc0c5ec0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.097+0000 7f9dd2b79640 1 --2- 172.21.15.1:0/2572213178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9dcc0a8520 0x7f9dcc1519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.097+0000 7f9dd2b79640 1 --2- 172.21.15.1:0/2572213178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9dcc06e670 0x7f9dcc1ebc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.097+0000 7f9dd2b79640 1 -- 172.21.15.1:0/2572213178 >> 172.21.15.1:0/2572213178 conn(0x7f9dcc0c3cc0 msgr2=0x7f9dcc0c40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:44.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.097+0000 7f9dd2b79640 1 -- 172.21.15.1:0/2572213178 shutdown_connections 2024-03-06T07:27:44.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.097+0000 7f9dd2b79640 1 -- 172.21.15.1:0/2572213178 wait complete. 2024-03-06T07:27:44.109 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down' 2024-03-06T07:27:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.208+0000 7f87033d7640 1 Processor -- start 2024-03-06T07:27:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.208+0000 7f87033d7640 1 -- start start 2024-03-06T07:27:44.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.209+0000 7f87033d7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f86fc159140 0x7f86fc0c4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.209+0000 7f87033d7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86fc072d00 con 0x7f86fc0c69b0 2024-03-06T07:27:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.209+0000 7f87033d7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86fc073180 con 0x7f86fc0c7510 2024-03-06T07:27:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.209+0000 7f87033d7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86fc0c0e80 con 0x7f86fc159140 2024-03-06T07:27:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.209+0000 7f8701173640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f86fc159140 0x7f86fc0c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.209+0000 7f8700972640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f86fc0c69b0 0x7f86fc0c2b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34050/0 (socket says 172.21.15.1:34050) 2024-03-06T07:27:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.209+0000 7f8700972640 1 -- 172.21.15.1:0/2016327283 learned_addr learned my addr 172.21.15.1:0/2016327283 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.209+0000 7f8701173640 1 -- 172.21.15.1:0/2016327283 >> v1:172.21.15.98:6789/0 conn(0x7f86fc0c7510 legacy=0x7f86fc15fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.209+0000 7f8701173640 1 -- 172.21.15.1:0/2016327283 >> v1:172.21.15.1:6789/0 conn(0x7f86fc0c69b0 legacy=0x7f86fc0c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.209+0000 7f8701173640 1 -- 172.21.15.1:0/2016327283 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86fc073460 con 0x7f86fc159140 2024-03-06T07:27:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.209+0000 7f86f3fff640 1 -- 172.21.15.1:0/2016327283 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092858594 0 0) 0x7f86fc072d00 con 0x7f86fc0c69b0 2024-03-06T07:27:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.210+0000 7f8701173640 1 --2- 172.21.15.1:0/2016327283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86fc159140 0x7f86fc0c4490 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f86f80097a0 tx=0x7f86f802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde08e7b6bd5c5de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.210+0000 7f86f3fff640 1 -- 172.21.15.1:0/2016327283 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f80315a0 con 0x7f86fc159140 2024-03-06T07:27:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.210+0000 7f86f3fff640 1 -- 172.21.15.1:0/2016327283 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f8037070 con 0x7f86fc159140 2024-03-06T07:27:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.210+0000 7f86f3fff640 1 -- 172.21.15.1:0/2016327283 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f8031960 con 0x7f86fc159140 2024-03-06T07:27:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.211+0000 7f87033d7640 1 -- 172.21.15.1:0/2016327283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86fc159140 msgr2=0x7f86fc0c4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.211+0000 7f87033d7640 1 --2- 172.21.15.1:0/2016327283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86fc159140 0x7f86fc0c4490 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f86f80097a0 tx=0x7f86f802b840 comp rx=0 tx=0).stop 2024-03-06T07:27:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.211+0000 7f87033d7640 1 -- 172.21.15.1:0/2016327283 shutdown_connections 2024-03-06T07:27:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.211+0000 7f87033d7640 1 --2- 172.21.15.1:0/2016327283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86fc159140 0x7f86fc0c4490 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.211+0000 7f87033d7640 1 -- 172.21.15.1:0/2016327283 >> 172.21.15.1:0/2016327283 conn(0x7f86fc0c7a50 msgr2=0x7f86fc0c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.211+0000 7f87033d7640 1 -- 172.21.15.1:0/2016327283 shutdown_connections 2024-03-06T07:27:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.211+0000 7f87033d7640 1 -- 172.21.15.1:0/2016327283 wait complete. 2024-03-06T07:27:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.212+0000 7f87033d7640 1 Processor -- start 2024-03-06T07:27:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.212+0000 7f87033d7640 1 -- start start 2024-03-06T07:27:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f87033d7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86fc0c69b0 0x7f86fc1ee7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f87033d7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86fc0c7510 0x7f86fc09a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f87033d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86fc159140 0x7f86fc0ab6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f87033d7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86fc0c6520 con 0x7f86fc0c69b0 2024-03-06T07:27:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f87033d7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86fc0c13b0 con 0x7f86fc159140 2024-03-06T07:27:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f87033d7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86fc159520 con 0x7f86fc0c7510 2024-03-06T07:27:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f8701173640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86fc0c69b0 0x7f86fc1ee7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f8701173640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86fc0c69b0 0x7f86fc1ee7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59098/0 (socket says 172.21.15.1:59098) 2024-03-06T07:27:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f8701173640 1 -- 172.21.15.1:0/1216003222 learned_addr learned my addr 172.21.15.1:0/1216003222 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f8701173640 1 -- 172.21.15.1:0/1216003222 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86fc0c7510 msgr2=0x7f86fc09a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f8701173640 1 --2- 172.21.15.1:0/1216003222 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86fc0c7510 0x7f86fc09a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f8701173640 1 -- 172.21.15.1:0/1216003222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86fc159140 msgr2=0x7f86fc0ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f8701173640 1 --2- 172.21.15.1:0/1216003222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86fc159140 0x7f86fc0ab6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f8701173640 1 -- 172.21.15.1:0/1216003222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86fc1ea990 con 0x7f86fc0c69b0 2024-03-06T07:27:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f8701173640 1 --2- 172.21.15.1:0/1216003222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86fc0c69b0 0x7f86fc1ee7b0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f86f80097a0 tx=0x7f86f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f86f1ffb640 1 -- 172.21.15.1:0/1216003222 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f8032840 con 0x7f86fc0c69b0 2024-03-06T07:27:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f86f1ffb640 1 -- 172.21.15.1:0/1216003222 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f80329e0 con 0x7f86fc0c69b0 2024-03-06T07:27:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f87033d7640 1 -- 172.21.15.1:0/1216003222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86fc0aa210 con 0x7f86fc0c69b0 2024-03-06T07:27:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.213+0000 7f86f1ffb640 1 -- 172.21.15.1:0/1216003222 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f8032dd0 con 0x7f86fc0c69b0 2024-03-06T07:27:44.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.214+0000 7f87033d7640 1 -- 172.21.15.1:0/1216003222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86fc0abc20 con 0x7f86fc0c69b0 2024-03-06T07:27:44.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.215+0000 7f87033d7640 1 -- 172.21.15.1:0/1216003222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86fc0686e0 con 0x7f86fc0c69b0 2024-03-06T07:27:44.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.215+0000 7f86f1ffb640 1 -- 172.21.15.1:0/1216003222 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86f803b3f0 con 0x7f86fc0c69b0 2024-03-06T07:27:44.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.215+0000 7f86f1ffb640 1 --2- 172.21.15.1:0/1216003222 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86e006b4d0 0x7f86e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.215+0000 7f86f1ffb640 1 -- 172.21.15.1:0/1216003222 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f86f802f030 con 0x7f86fc0c69b0 2024-03-06T07:27:44.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.216+0000 7f8700972640 1 --2- 172.21.15.1:0/1216003222 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86e006b4d0 0x7f86e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.216+0000 7f8700972640 1 --2- 172.21.15.1:0/1216003222 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86e006b4d0 0x7f86e006d990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f86ec007fd0 tx=0x7f86ec007920 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.220+0000 7f86f1ffb640 1 -- 172.21.15.1:0/1216003222 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86f8040070 con 0x7f86fc0c69b0 2024-03-06T07:27:44.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.469+0000 7f87033d7640 1 -- 172.21.15.1:0/1216003222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down"]} v 0) v1 -- 0x7f86fc0c5310 con 0x7f86fc0c69b0 2024-03-06T07:27:44.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.774+0000 7f86f1ffb640 1 -- 172.21.15.1:0/1216003222 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down"]}]=0 v426) v1 ==== 133+0+0 (secure 0 0 0) 0x7f86f807b040 con 0x7f86fc0c69b0 2024-03-06T07:27:44.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.777+0000 7f87033d7640 1 -- 172.21.15.1:0/1216003222 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86e006b4d0 msgr2=0x7f86e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.777+0000 7f87033d7640 1 --2- 172.21.15.1:0/1216003222 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86e006b4d0 0x7f86e006d990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f86ec007fd0 tx=0x7f86ec007920 comp rx=0 tx=0).stop 2024-03-06T07:27:44.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.777+0000 7f87033d7640 1 -- 172.21.15.1:0/1216003222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86fc0c69b0 msgr2=0x7f86fc1ee7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.777+0000 7f87033d7640 1 --2- 172.21.15.1:0/1216003222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86fc0c69b0 0x7f86fc1ee7b0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f86f80097a0 tx=0x7f86f80046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.777+0000 7f87033d7640 1 -- 172.21.15.1:0/1216003222 shutdown_connections 2024-03-06T07:27:44.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.777+0000 7f87033d7640 1 --2- 172.21.15.1:0/1216003222 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86e006b4d0 0x7f86e006d990 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.777+0000 7f87033d7640 1 --2- 172.21.15.1:0/1216003222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86fc159140 0x7f86fc0ab6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.777+0000 7f87033d7640 1 --2- 172.21.15.1:0/1216003222 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86fc0c7510 0x7f86fc09a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.777+0000 7f87033d7640 1 --2- 172.21.15.1:0/1216003222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86fc0c69b0 0x7f86fc1ee7b0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.777+0000 7f87033d7640 1 -- 172.21.15.1:0/1216003222 >> 172.21.15.1:0/1216003222 conn(0x7f86fc0c7a50 msgr2=0x7f86fc1583f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:44.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.777+0000 7f87033d7640 1 -- 172.21.15.1:0/1216003222 shutdown_connections 2024-03-06T07:27:44.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.777+0000 7f87033d7640 1 -- 172.21.15.1:0/1216003222 wait complete. 2024-03-06T07:27:44.791 INFO:tasks.cephfs_test_runner:That down setting toggles and sets max_mds appropriately. ... ok 2024-03-06T07:27:44.791 INFO:tasks.cephfs_test_runner:Starting test: test_down_grow (tasks.cephfs.test_failover.TestClusterResize) That setting max_mds undoes down. 2024-03-06T07:27:44.791 INFO:tasks.cephfs_test_runner:test_down_grow (tasks.cephfs.test_failover.TestClusterResize) 2024-03-06T07:27:44.791 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.892+0000 7f5f05202640 1 Processor -- start 2024-03-06T07:27:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.892+0000 7f5f05202640 1 -- start start 2024-03-06T07:27:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.892+0000 7f5f05202640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f000c6360 0x7f5f000c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.892+0000 7f5f05202640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f000c6990 con 0x7f5f00155a60 2024-03-06T07:27:44.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.892+0000 7f5f05202640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f000c6c30 con 0x7f5f0014b8f0 2024-03-06T07:27:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.892+0000 7f5f05202640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f00158940 con 0x7f5f000c6360 2024-03-06T07:27:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5eff59e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5f00155a60 0x7f5f000c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57802/0 (socket says 172.21.15.1:57802) 2024-03-06T07:27:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5eff59e640 1 -- 172.21.15.1:0/213446938 learned_addr learned my addr 172.21.15.1:0/213446938 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efdd9b640 1 -- 172.21.15.1:0/213446938 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082857395 0 0) 0x7f5f000c6990 con 0x7f5f00155a60 2024-03-06T07:27:44.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efe59c640 1 --2- 172.21.15.1:0/213446938 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f000c6360 0x7f5f000c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efdd9b640 1 -- 172.21.15.1:0/213446938 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee8003520 con 0x7f5f00155a60 2024-03-06T07:27:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efdd9b640 1 -- 172.21.15.1:0/213446938 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2591237219 0 0) 0x7f5ee8003520 con 0x7f5f00155a60 2024-03-06T07:27:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efdd9b640 1 -- 172.21.15.1:0/213446938 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f000c6360 msgr2=0x7f5f000c2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efdd9b640 1 --2- 172.21.15.1:0/213446938 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f000c6360 0x7f5f000c2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efdd9b640 1 -- 172.21.15.1:0/213446938 >> v1:172.21.15.98:6789/0 conn(0x7f5f0014b8f0 legacy=0x7f5f00158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efdd9b640 1 -- 172.21.15.1:0/213446938 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f000c6f10 con 0x7f5f00155a60 2024-03-06T07:27:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efdd9b640 1 -- 172.21.15.1:0/213446938 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f5ef80019f0 con 0x7f5f00155a60 2024-03-06T07:27:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efdd9b640 1 -- 172.21.15.1:0/213446938 >> v1:172.21.15.1:6789/0 conn(0x7f5f00155a60 legacy=0x7f5f000c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efdd9b640 1 --2- 172.21.15.1:0/213446938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ee80039e0 0x7f5ee8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efdd9b640 1 --2- 172.21.15.1:0/213446938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ee8004800 0x7f5ee8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efdd9b640 1 --2- 172.21.15.1:0/213446938 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f000c6360 0x7f5ee8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efdd9b640 1 -- 172.21.15.1:0/213446938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f00158940 con 0x7f5ee80039e0 2024-03-06T07:27:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efdd9b640 1 -- 172.21.15.1:0/213446938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee800bb20 con 0x7f5ee8004800 2024-03-06T07:27:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efdd9b640 1 -- 172.21.15.1:0/213446938 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee800bd20 con 0x7f5f000c6360 2024-03-06T07:27:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efed9d640 1 --2- 172.21.15.1:0/213446938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ee80039e0 0x7f5ee8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5efe59c640 1 --2- 172.21.15.1:0/213446938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ee8004800 0x7f5ee8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.893+0000 7f5eff59e640 1 --2- 172.21.15.1:0/213446938 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f000c6360 0x7f5ee8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.894+0000 7f5efed9d640 1 -- 172.21.15.1:0/213446938 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f000c6360 msgr2=0x7f5ee8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.894+0000 7f5efed9d640 1 --2- 172.21.15.1:0/213446938 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f000c6360 0x7f5ee8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.894+0000 7f5efed9d640 1 -- 172.21.15.1:0/213446938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ee8004800 msgr2=0x7f5ee8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.894+0000 7f5efed9d640 1 --2- 172.21.15.1:0/213446938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ee8004800 0x7f5ee8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.894+0000 7f5efed9d640 1 -- 172.21.15.1:0/213446938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee8003520 con 0x7f5ee80039e0 2024-03-06T07:27:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.894+0000 7f5efed9d640 1 --2- 172.21.15.1:0/213446938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ee80039e0 0x7f5ee8003e00 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0001750 tx=0x7f5ee8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef4802d7047ba6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.895+0000 7f5efdd9b640 1 -- 172.21.15.1:0/213446938 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0001370 con 0x7f5ee80039e0 2024-03-06T07:27:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.895+0000 7f5efdd9b640 1 -- 172.21.15.1:0/213446938 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef00052d0 con 0x7f5ee80039e0 2024-03-06T07:27:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.895+0000 7f5efdd9b640 1 -- 172.21.15.1:0/213446938 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0003830 con 0x7f5ee80039e0 2024-03-06T07:27:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.895+0000 7f5f05202640 1 -- 172.21.15.1:0/213446938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ee80039e0 msgr2=0x7f5ee8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.895+0000 7f5f05202640 1 --2- 172.21.15.1:0/213446938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ee80039e0 0x7f5ee8003e00 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0001750 tx=0x7f5ee8005830 comp rx=0 tx=0).stop 2024-03-06T07:27:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.895+0000 7f5eff59e640 1 -- 172.21.15.1:0/213446938 reap_dead start 2024-03-06T07:27:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.895+0000 7f5f05202640 1 -- 172.21.15.1:0/213446938 shutdown_connections 2024-03-06T07:27:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.895+0000 7f5f05202640 1 -- 172.21.15.1:0/213446938 >> 172.21.15.1:0/213446938 conn(0x7f5f0006d950 msgr2=0x7f5f00066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.895+0000 7f5f05202640 1 -- 172.21.15.1:0/213446938 shutdown_connections 2024-03-06T07:27:44.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.895+0000 7f5f05202640 1 -- 172.21.15.1:0/213446938 wait complete. 2024-03-06T07:27:44.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.896+0000 7f5f05202640 1 Processor -- start 2024-03-06T07:27:44.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.897+0000 7f5f05202640 1 -- start start 2024-03-06T07:27:44.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.897+0000 7f5f05202640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f000c0d10 0x7f5f00152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.897+0000 7f5f05202640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f001ee830 0x7f5f00153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.897+0000 7f5f05202640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f001535a0 0x7f5f0014a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.897+0000 7f5f05202640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f00155ed0 con 0x7f5f001ee830 2024-03-06T07:27:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.897+0000 7f5f05202640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f001594e0 con 0x7f5f001535a0 2024-03-06T07:27:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.897+0000 7f5f05202640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f000c7a60 con 0x7f5f000c0d10 2024-03-06T07:27:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.897+0000 7f5efe59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f001ee830 0x7f5f00153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.897+0000 7f5efe59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f001ee830 0x7f5f00153060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58590/0 (socket says 172.21.15.1:58590) 2024-03-06T07:27:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.897+0000 7f5efe59c640 1 -- 172.21.15.1:0/3968168910 learned_addr learned my addr 172.21.15.1:0/3968168910 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.897+0000 7f5efed9d640 1 --2- 172.21.15.1:0/3968168910 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f000c0d10 0x7f5f00152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.897+0000 7f5eff59e640 1 --2- 172.21.15.1:0/3968168910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f001535a0 0x7f5f0014a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.898+0000 7f5efe59c640 1 -- 172.21.15.1:0/3968168910 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f000c0d10 msgr2=0x7f5f00152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.898+0000 7f5efe59c640 1 --2- 172.21.15.1:0/3968168910 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f000c0d10 0x7f5f00152140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.898+0000 7f5efe59c640 1 -- 172.21.15.1:0/3968168910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f001535a0 msgr2=0x7f5f0014a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.898+0000 7f5efe59c640 1 --2- 172.21.15.1:0/3968168910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f001535a0 0x7f5f0014a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.898+0000 7f5efe59c640 1 -- 172.21.15.1:0/3968168910 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f001eaba0 con 0x7f5f001ee830 2024-03-06T07:27:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.898+0000 7f5efed9d640 1 --2- 172.21.15.1:0/3968168910 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f000c0d10 0x7f5f00152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.898+0000 7f5eff59e640 1 --2- 172.21.15.1:0/3968168910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f001535a0 0x7f5f0014a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.898+0000 7f5efe59c640 1 --2- 172.21.15.1:0/3968168910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f001ee830 0x7f5f00153060 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f5ef40118e0 tx=0x7f5ef4011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.898+0000 7f5ee77fe640 1 -- 172.21.15.1:0/3968168910 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef400fa90 con 0x7f5f001ee830 2024-03-06T07:27:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.898+0000 7f5ee77fe640 1 -- 172.21.15.1:0/3968168910 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef400fc30 con 0x7f5f001ee830 2024-03-06T07:27:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.898+0000 7f5f05202640 1 -- 172.21.15.1:0/3968168910 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f001eae60 con 0x7f5f001ee830 2024-03-06T07:27:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.898+0000 7f5ee77fe640 1 -- 172.21.15.1:0/3968168910 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef4014440 con 0x7f5f001ee830 2024-03-06T07:27:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.898+0000 7f5f05202640 1 -- 172.21.15.1:0/3968168910 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f000ca610 con 0x7f5f001ee830 2024-03-06T07:27:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.899+0000 7f5f05202640 1 -- 172.21.15.1:0/3968168910 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f000c19f0 con 0x7f5f001ee830 2024-03-06T07:27:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.899+0000 7f5ee77fe640 1 -- 172.21.15.1:0/3968168910 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ef4012040 con 0x7f5f001ee830 2024-03-06T07:27:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.900+0000 7f5ee77fe640 1 --2- 172.21.15.1:0/3968168910 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eec06b4d0 0x7f5eec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.900+0000 7f5ee77fe640 1 -- 172.21.15.1:0/3968168910 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f5ef408a5d0 con 0x7f5f001ee830 2024-03-06T07:27:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.900+0000 7f5efed9d640 1 --2- 172.21.15.1:0/3968168910 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eec06b4d0 0x7f5eec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.901+0000 7f5efed9d640 1 --2- 172.21.15.1:0/3968168910 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eec06b4d0 0x7f5eec06d990 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0033000 tx=0x7f5ef0009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:44.904+0000 7f5ee77fe640 1 -- 172.21.15.1:0/3968168910 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f000c19f0 con 0x7f5f001ee830 2024-03-06T07:27:45.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.085+0000 7f5f05202640 1 -- 172.21.15.1:0/3968168910 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f00067fd0 con 0x7f5f001ee830 2024-03-06T07:27:45.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.086+0000 7f5ee77fe640 1 -- 172.21.15.1:0/3968168910 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+13181 (secure 0 0 0) 0x7f5ef4019030 con 0x7f5f001ee830 2024-03-06T07:27:45.088 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:45.088 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":174,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:27:43.769153+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6835/1021164277":"2024-03-07T07:27:40.759158+0000","172.21.15.1:6833/1021164277":"2024-03-07T07:27:40.759158+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:45.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.088+0000 7f5f05202640 1 -- 172.21.15.1:0/3968168910 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eec06b4d0 msgr2=0x7f5eec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.088+0000 7f5f05202640 1 --2- 172.21.15.1:0/3968168910 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eec06b4d0 0x7f5eec06d990 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0033000 tx=0x7f5ef0009030 comp rx=0 tx=0).stop 2024-03-06T07:27:45.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.088+0000 7f5f05202640 1 -- 172.21.15.1:0/3968168910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f001ee830 msgr2=0x7f5f00153060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.088+0000 7f5f05202640 1 --2- 172.21.15.1:0/3968168910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f001ee830 0x7f5f00153060 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f5ef40118e0 tx=0x7f5ef4011da0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.088+0000 7f5f05202640 1 -- 172.21.15.1:0/3968168910 shutdown_connections 2024-03-06T07:27:45.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.088+0000 7f5f05202640 1 --2- 172.21.15.1:0/3968168910 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5eec06b4d0 0x7f5eec06d990 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.088+0000 7f5f05202640 1 --2- 172.21.15.1:0/3968168910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f001535a0 0x7f5f0014a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.088+0000 7f5f05202640 1 --2- 172.21.15.1:0/3968168910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f001ee830 0x7f5f00153060 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.088+0000 7f5f05202640 1 --2- 172.21.15.1:0/3968168910 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5f000c0d10 0x7f5f00152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.088+0000 7f5f05202640 1 -- 172.21.15.1:0/3968168910 >> 172.21.15.1:0/3968168910 conn(0x7f5f0006d950 msgr2=0x7f5f000c2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.088+0000 7f5f05202640 1 -- 172.21.15.1:0/3968168910 shutdown_connections 2024-03-06T07:27:45.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.088+0000 7f5f05202640 1 -- 172.21.15.1:0/3968168910 wait complete. 2024-03-06T07:27:45.101 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:45.101 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:45.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.203+0000 7f2f4185c640 1 Processor -- start 2024-03-06T07:27:45.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.203+0000 7f2f4185c640 1 -- start start 2024-03-06T07:27:45.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.204+0000 7f2f4185c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2f3c152820 0x7f2f3c0c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.204+0000 7f2f4185c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f3c0af6f0 0x7f2f3c15fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.204+0000 7f2f4185c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f3c0c7180 con 0x7f2f3c0c0670 2024-03-06T07:27:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.204+0000 7f2f4185c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f3c155460 con 0x7f2f3c152820 2024-03-06T07:27:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.204+0000 7f2f4185c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c0c6bc0 con 0x7f2f3c0af6f0 2024-03-06T07:27:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.204+0000 7f2f3a7fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2f3c0c0670 0x7f2f3c0c73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57808/0 (socket says 172.21.15.1:57808) 2024-03-06T07:27:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.204+0000 7f2f3a7fc640 1 -- 172.21.15.1:0/669312104 learned_addr learned my addr 172.21.15.1:0/669312104 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.204+0000 7f2f3affd640 1 --2- 172.21.15.1:0/669312104 >> v2:172.21.15.98:3300/0 conn(0x7f2f3c152820 0x7f2f3c0c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.204+0000 7f2f3b7fe640 1 --2- 172.21.15.1:0/669312104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f3c0af6f0 0x7f2f3c15fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.204+0000 7f2f39ffb640 1 -- 172.21.15.1:0/669312104 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004485705 0 0) 0x7f2f3c0c7180 con 0x7f2f3c0c0670 2024-03-06T07:27:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.205+0000 7f2f39ffb640 1 -- 172.21.15.1:0/669312104 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f28003520 con 0x7f2f3c0c0670 2024-03-06T07:27:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.205+0000 7f2f3affd640 1 -- 172.21.15.1:0/669312104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f3c0af6f0 msgr2=0x7f2f3c15fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.205+0000 7f2f3affd640 1 --2- 172.21.15.1:0/669312104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f3c0af6f0 0x7f2f3c15fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.205+0000 7f2f3affd640 1 -- 172.21.15.1:0/669312104 >> v1:172.21.15.1:6789/0 conn(0x7f2f3c0c0670 legacy=0x7f2f3c0c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.205+0000 7f2f3affd640 1 -- 172.21.15.1:0/669312104 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3c0c6ea0 con 0x7f2f3c152820 2024-03-06T07:27:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.205+0000 7f2f3affd640 1 --2- 172.21.15.1:0/669312104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f3c152820 0x7f2f3c0c0130 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f2f340097a0 tx=0x7f2f3402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cb0cae022ec9a03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.206+0000 7f2f39ffb640 1 -- 172.21.15.1:0/669312104 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f34035020 con 0x7f2f3c152820 2024-03-06T07:27:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.206+0000 7f2f39ffb640 1 -- 172.21.15.1:0/669312104 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f34031830 con 0x7f2f3c152820 2024-03-06T07:27:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.206+0000 7f2f39ffb640 1 -- 172.21.15.1:0/669312104 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f34031be0 con 0x7f2f3c152820 2024-03-06T07:27:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.206+0000 7f2f4185c640 1 -- 172.21.15.1:0/669312104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f3c152820 msgr2=0x7f2f3c0c0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.206+0000 7f2f4185c640 1 --2- 172.21.15.1:0/669312104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f3c152820 0x7f2f3c0c0130 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f2f340097a0 tx=0x7f2f3402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.206+0000 7f2f4185c640 1 -- 172.21.15.1:0/669312104 shutdown_connections 2024-03-06T07:27:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.206+0000 7f2f4185c640 1 --2- 172.21.15.1:0/669312104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f3c0af6f0 0x7f2f3c15fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.206+0000 7f2f4185c640 1 --2- 172.21.15.1:0/669312104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f3c152820 0x7f2f3c0c0130 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.206+0000 7f2f4185c640 1 -- 172.21.15.1:0/669312104 >> 172.21.15.1:0/669312104 conn(0x7f2f3c072720 msgr2=0x7f2f3c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.206+0000 7f2f4185c640 1 -- 172.21.15.1:0/669312104 shutdown_connections 2024-03-06T07:27:45.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.206+0000 7f2f4185c640 1 -- 172.21.15.1:0/669312104 wait complete. 2024-03-06T07:27:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.207+0000 7f2f4185c640 1 Processor -- start 2024-03-06T07:27:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.208+0000 7f2f4185c640 1 -- start start 2024-03-06T07:27:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.208+0000 7f2f4185c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f3c0af6f0 0x7f2f3c0c3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.208+0000 7f2f4185c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f3c0c0670 0x7f2f3c14b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.208+0000 7f2f4185c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f3c152820 0x7f2f3c14b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.208+0000 7f2f4185c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c0c7c80 con 0x7f2f3c0af6f0 2024-03-06T07:27:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.208+0000 7f2f4185c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c0c7b00 con 0x7f2f3c0c0670 2024-03-06T07:27:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.208+0000 7f2f4185c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c152c00 con 0x7f2f3c152820 2024-03-06T07:27:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.208+0000 7f2f3affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f3c0af6f0 0x7f2f3c0c3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.208+0000 7f2f3affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f3c0af6f0 0x7f2f3c0c3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58602/0 (socket says 172.21.15.1:58602) 2024-03-06T07:27:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.208+0000 7f2f3affd640 1 -- 172.21.15.1:0/3907778916 learned_addr learned my addr 172.21.15.1:0/3907778916 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.208+0000 7f2f3b7fe640 1 --2- 172.21.15.1:0/3907778916 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f3c152820 0x7f2f3c14b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.208+0000 7f2f3a7fc640 1 --2- 172.21.15.1:0/3907778916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f3c0c0670 0x7f2f3c14b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.209+0000 7f2f3affd640 1 -- 172.21.15.1:0/3907778916 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f3c152820 msgr2=0x7f2f3c14b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.209+0000 7f2f3affd640 1 --2- 172.21.15.1:0/3907778916 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f3c152820 0x7f2f3c14b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.209+0000 7f2f3affd640 1 -- 172.21.15.1:0/3907778916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f3c0c0670 msgr2=0x7f2f3c14b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.209+0000 7f2f3affd640 1 --2- 172.21.15.1:0/3907778916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f3c0c0670 0x7f2f3c14b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.209+0000 7f2f3affd640 1 -- 172.21.15.1:0/3907778916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3c1eec80 con 0x7f2f3c0af6f0 2024-03-06T07:27:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.209+0000 7f2f3affd640 1 --2- 172.21.15.1:0/3907778916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f3c0af6f0 0x7f2f3c0c3920 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f2f3402b910 tx=0x7f2f340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.209+0000 7f2f33fff640 1 -- 172.21.15.1:0/3907778916 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f34040070 con 0x7f2f3c0af6f0 2024-03-06T07:27:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.209+0000 7f2f33fff640 1 -- 172.21.15.1:0/3907778916 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3403a910 con 0x7f2f3c0af6f0 2024-03-06T07:27:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.209+0000 7f2f33fff640 1 -- 172.21.15.1:0/3907778916 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f34035040 con 0x7f2f3c0af6f0 2024-03-06T07:27:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.209+0000 7f2f4185c640 1 -- 172.21.15.1:0/3907778916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f3c06d850 con 0x7f2f3c0af6f0 2024-03-06T07:27:45.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.209+0000 7f2f4185c640 1 -- 172.21.15.1:0/3907778916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f3c1ee9a0 con 0x7f2f3c0af6f0 2024-03-06T07:27:45.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.210+0000 7f2f4185c640 1 -- 172.21.15.1:0/3907778916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f3c1ee790 con 0x7f2f3c0af6f0 2024-03-06T07:27:45.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.210+0000 7f2f33fff640 1 -- 172.21.15.1:0/3907778916 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f3404b060 con 0x7f2f3c0af6f0 2024-03-06T07:27:45.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.211+0000 7f2f33fff640 1 --2- 172.21.15.1:0/3907778916 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f1c06b500 0x7f2f1c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.211+0000 7f2f33fff640 1 -- 172.21.15.1:0/3907778916 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f2f340ab7b0 con 0x7f2f3c0af6f0 2024-03-06T07:27:45.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.211+0000 7f2f3a7fc640 1 --2- 172.21.15.1:0/3907778916 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f1c06b500 0x7f2f1c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.211+0000 7f2f3a7fc640 1 --2- 172.21.15.1:0/3907778916 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f1c06b500 0x7f2f1c06d9c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c0c7460 tx=0x7f2f24009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.215+0000 7f2f33fff640 1 -- 172.21.15.1:0/3907778916 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f3407a7b0 con 0x7f2f3c0af6f0 2024-03-06T07:27:45.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.395+0000 7f2f4185c640 1 -- 172.21.15.1:0/3907778916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2f3c0c5b40 con 0x7f2f3c0af6f0 2024-03-06T07:27:45.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.396+0000 7f2f33fff640 1 -- 172.21.15.1:0/3907778916 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v174) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2f3407a0b0 con 0x7f2f3c0af6f0 2024-03-06T07:27:45.398 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:27:45.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.398+0000 7f2f4185c640 1 -- 172.21.15.1:0/3907778916 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f1c06b500 msgr2=0x7f2f1c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.398+0000 7f2f4185c640 1 --2- 172.21.15.1:0/3907778916 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f1c06b500 0x7f2f1c06d9c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c0c7460 tx=0x7f2f24009040 comp rx=0 tx=0).stop 2024-03-06T07:27:45.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.398+0000 7f2f4185c640 1 -- 172.21.15.1:0/3907778916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f3c0af6f0 msgr2=0x7f2f3c0c3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.398+0000 7f2f4185c640 1 --2- 172.21.15.1:0/3907778916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f3c0af6f0 0x7f2f3c0c3920 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f2f3402b910 tx=0x7f2f340046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.398+0000 7f2f4185c640 1 -- 172.21.15.1:0/3907778916 shutdown_connections 2024-03-06T07:27:45.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.398+0000 7f2f4185c640 1 --2- 172.21.15.1:0/3907778916 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f1c06b500 0x7f2f1c06d9c0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.398+0000 7f2f4185c640 1 --2- 172.21.15.1:0/3907778916 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2f3c152820 0x7f2f3c14b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.398+0000 7f2f4185c640 1 --2- 172.21.15.1:0/3907778916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f3c0c0670 0x7f2f3c14b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.398+0000 7f2f4185c640 1 --2- 172.21.15.1:0/3907778916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f3c0af6f0 0x7f2f3c0c3920 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.398+0000 7f2f4185c640 1 -- 172.21.15.1:0/3907778916 >> 172.21.15.1:0/3907778916 conn(0x7f2f3c072720 msgr2=0x7f2f3c06cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.398+0000 7f2f4185c640 1 -- 172.21.15.1:0/3907778916 shutdown_connections 2024-03-06T07:27:45.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.398+0000 7f2f4185c640 1 -- 172.21.15.1:0/3907778916 wait complete. 2024-03-06T07:27:45.411 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_grow' 2024-03-06T07:27:45.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.510+0000 7f4801ec3640 1 Processor -- start 2024-03-06T07:27:45.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.510+0000 7f4801ec3640 1 -- start start 2024-03-06T07:27:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.511+0000 7f4801ec3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f47fc152920 0x7f47fc06aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.511+0000 7f4801ec3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47fc155dc0 con 0x7f47fc06b530 2024-03-06T07:27:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.511+0000 7f4801ec3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47fc158910 con 0x7f47fc155220 2024-03-06T07:27:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.511+0000 7f4801ec3640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47fc152d00 con 0x7f47fc152920 2024-03-06T07:27:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.511+0000 7f47fb7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f47fc152920 0x7f47fc06aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.511+0000 7f47faffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f47fc06b530 0x7f47fc0c5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57818/0 (socket says 172.21.15.1:57818) 2024-03-06T07:27:45.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.511+0000 7f47faffd640 1 -- 172.21.15.1:0/1082661461 learned_addr learned my addr 172.21.15.1:0/1082661461 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:45.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.511+0000 7f47fb7fe640 1 -- 172.21.15.1:0/1082661461 >> v1:172.21.15.98:6789/0 conn(0x7f47fc155220 legacy=0x7f47fc1556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.511+0000 7f47fb7fe640 1 -- 172.21.15.1:0/1082661461 >> v1:172.21.15.1:6789/0 conn(0x7f47fc06b530 legacy=0x7f47fc0c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.511+0000 7f47fa7fc640 1 -- 172.21.15.1:0/1082661461 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532598195 0 0) 0x7f47fc155dc0 con 0x7f47fc06b530 2024-03-06T07:27:45.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.511+0000 7f47fb7fe640 1 -- 172.21.15.1:0/1082661461 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47fc158bf0 con 0x7f47fc152920 2024-03-06T07:27:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.512+0000 7f47fb7fe640 1 --2- 172.21.15.1:0/1082661461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47fc152920 0x7f47fc06aff0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f47ec0097a0 tx=0x7f47ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702400b8d6c89bc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.512+0000 7f47fa7fc640 1 -- 172.21.15.1:0/1082661461 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec035020 con 0x7f47fc152920 2024-03-06T07:27:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.512+0000 7f47fa7fc640 1 -- 172.21.15.1:0/1082661461 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec031830 con 0x7f47fc152920 2024-03-06T07:27:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.512+0000 7f47fa7fc640 1 -- 172.21.15.1:0/1082661461 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec031c00 con 0x7f47fc152920 2024-03-06T07:27:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.513+0000 7f4801ec3640 1 -- 172.21.15.1:0/1082661461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47fc152920 msgr2=0x7f47fc06aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.513+0000 7f4801ec3640 1 --2- 172.21.15.1:0/1082661461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47fc152920 0x7f47fc06aff0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f47ec0097a0 tx=0x7f47ec02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.513+0000 7f4801ec3640 1 -- 172.21.15.1:0/1082661461 shutdown_connections 2024-03-06T07:27:45.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.513+0000 7f4801ec3640 1 --2- 172.21.15.1:0/1082661461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47fc152920 0x7f47fc06aff0 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.513+0000 7f4801ec3640 1 -- 172.21.15.1:0/1082661461 >> 172.21.15.1:0/1082661461 conn(0x7f47fc0a9700 msgr2=0x7f47fc0a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.513+0000 7f4801ec3640 1 -- 172.21.15.1:0/1082661461 shutdown_connections 2024-03-06T07:27:45.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.513+0000 7f4801ec3640 1 -- 172.21.15.1:0/1082661461 wait complete. 2024-03-06T07:27:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.514+0000 7f4801ec3640 1 Processor -- start 2024-03-06T07:27:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.514+0000 7f4801ec3640 1 -- start start 2024-03-06T07:27:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f4801ec3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47fc06b530 0x7f47fc06f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f4801ec3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47fc152920 0x7f47fc149380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f4801ec3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47fc155220 0x7f47fc1486c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f4801ec3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc159240 con 0x7f47fc152920 2024-03-06T07:27:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f4801ec3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc1596d0 con 0x7f47fc155220 2024-03-06T07:27:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f4801ec3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc0c6380 con 0x7f47fc06b530 2024-03-06T07:27:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f47faffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47fc152920 0x7f47fc149380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f47faffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47fc152920 0x7f47fc149380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58626/0 (socket says 172.21.15.1:58626) 2024-03-06T07:27:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f47faffd640 1 -- 172.21.15.1:0/524375166 learned_addr learned my addr 172.21.15.1:0/524375166 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f47fb7fe640 1 --2- 172.21.15.1:0/524375166 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47fc06b530 0x7f47fc06f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f47fbfff640 1 --2- 172.21.15.1:0/524375166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47fc155220 0x7f47fc1486c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f47faffd640 1 -- 172.21.15.1:0/524375166 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47fc06b530 msgr2=0x7f47fc06f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f47faffd640 1 --2- 172.21.15.1:0/524375166 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47fc06b530 0x7f47fc06f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f47faffd640 1 -- 172.21.15.1:0/524375166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47fc155220 msgr2=0x7f47fc1486c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f47faffd640 1 --2- 172.21.15.1:0/524375166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47fc155220 0x7f47fc1486c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f47faffd640 1 -- 172.21.15.1:0/524375166 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47fc06fc80 con 0x7f47fc152920 2024-03-06T07:27:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f47faffd640 1 --2- 172.21.15.1:0/524375166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47fc152920 0x7f47fc149380 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f47f000c460 tx=0x7f47f000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f47e3fff640 1 -- 172.21.15.1:0/524375166 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f0018070 con 0x7f47fc152920 2024-03-06T07:27:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f4801ec3640 1 -- 172.21.15.1:0/524375166 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47fc070380 con 0x7f47fc152920 2024-03-06T07:27:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f47e3fff640 1 -- 172.21.15.1:0/524375166 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f0007520 con 0x7f47fc152920 2024-03-06T07:27:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.515+0000 7f47e3fff640 1 -- 172.21.15.1:0/524375166 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f0004a20 con 0x7f47fc152920 2024-03-06T07:27:45.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.516+0000 7f4801ec3640 1 -- 172.21.15.1:0/524375166 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47fc1ebb80 con 0x7f47fc152920 2024-03-06T07:27:45.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.517+0000 7f4801ec3640 1 -- 172.21.15.1:0/524375166 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47fc067fd0 con 0x7f47fc152920 2024-03-06T07:27:45.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.517+0000 7f47e3fff640 1 -- 172.21.15.1:0/524375166 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f47f0004bc0 con 0x7f47fc152920 2024-03-06T07:27:45.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.517+0000 7f47e3fff640 1 --2- 172.21.15.1:0/524375166 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47e806b4d0 0x7f47e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.517+0000 7f47e3fff640 1 -- 172.21.15.1:0/524375166 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f47f0089270 con 0x7f47fc152920 2024-03-06T07:27:45.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.517+0000 7f47fb7fe640 1 --2- 172.21.15.1:0/524375166 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47e806b4d0 0x7f47e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.518+0000 7f47fb7fe640 1 --2- 172.21.15.1:0/524375166 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47e806b4d0 0x7f47e806d990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f47ec036000 tx=0x7f47ec008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.521+0000 7f47e3fff640 1 -- 172.21.15.1:0/524375166 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f47f0057c20 con 0x7f47fc152920 2024-03-06T07:27:45.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:45.779+0000 7f4801ec3640 1 -- 172.21.15.1:0/524375166 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]} v 0) v1 -- 0x7f47fc06bdf0 con 0x7f47fc152920 2024-03-06T07:27:46.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.780+0000 7f47e3fff640 1 -- 172.21.15.1:0/524375166 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]}]=0 v428) v1 ==== 141+0+0 (secure 0 0 0) 0x7f47f005bcc0 con 0x7f47fc152920 2024-03-06T07:27:46.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.783+0000 7f4801ec3640 1 -- 172.21.15.1:0/524375166 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47e806b4d0 msgr2=0x7f47e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.783+0000 7f4801ec3640 1 --2- 172.21.15.1:0/524375166 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47e806b4d0 0x7f47e806d990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f47ec036000 tx=0x7f47ec008030 comp rx=0 tx=0).stop 2024-03-06T07:27:46.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.783+0000 7f4801ec3640 1 -- 172.21.15.1:0/524375166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47fc152920 msgr2=0x7f47fc149380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.783+0000 7f4801ec3640 1 --2- 172.21.15.1:0/524375166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47fc152920 0x7f47fc149380 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f47f000c460 tx=0x7f47f000c920 comp rx=0 tx=0).stop 2024-03-06T07:27:46.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.783+0000 7f4801ec3640 1 -- 172.21.15.1:0/524375166 shutdown_connections 2024-03-06T07:27:46.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.783+0000 7f4801ec3640 1 --2- 172.21.15.1:0/524375166 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f47e806b4d0 0x7f47e806d990 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.783+0000 7f4801ec3640 1 --2- 172.21.15.1:0/524375166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47fc155220 0x7f47fc1486c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.783+0000 7f4801ec3640 1 --2- 172.21.15.1:0/524375166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47fc152920 0x7f47fc149380 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.783+0000 7f4801ec3640 1 --2- 172.21.15.1:0/524375166 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47fc06b530 0x7f47fc06f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.783+0000 7f4801ec3640 1 -- 172.21.15.1:0/524375166 >> 172.21.15.1:0/524375166 conn(0x7f47fc0a9700 msgr2=0x7f47fc0c65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:46.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.783+0000 7f4801ec3640 1 -- 172.21.15.1:0/524375166 shutdown_connections 2024-03-06T07:27:46.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.783+0000 7f4801ec3640 1 -- 172.21.15.1:0/524375166 wait complete. 2024-03-06T07:27:46.798 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:27:46.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.907+0000 7f050173a640 1 Processor -- start 2024-03-06T07:27:46.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.907+0000 7f050173a640 1 -- start start 2024-03-06T07:27:46.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.908+0000 7f050173a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f04fc155dc0 0x7f04fc160a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.908+0000 7f050173a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04fc155200 con 0x7f04fc0c90f0 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.908+0000 7f050173a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04fc0c9550 con 0x7f04fc0c8d10 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.908+0000 7f050173a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04fc0c33e0 con 0x7f04fc155dc0 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.908+0000 7f04f3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f04fc0c90f0 0x7f04fc1556b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57822/0 (socket says 172.21.15.1:57822) 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.908+0000 7f04f3fff640 1 -- 172.21.15.1:0/1905792690 learned_addr learned my addr 172.21.15.1:0/1905792690 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.908+0000 7f04fb7fe640 1 --2- 172.21.15.1:0/1905792690 >> v2:172.21.15.98:3300/0 conn(0x7f04fc155dc0 0x7f04fc160a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.908+0000 7f04fa7fc640 1 -- 172.21.15.1:0/1905792690 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662903634 0 0) 0x7f04fc155200 con 0x7f04fc0c90f0 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.908+0000 7f04fa7fc640 1 -- 172.21.15.1:0/1905792690 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e8003520 con 0x7f04fc0c90f0 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.908+0000 7f04fa7fc640 1 -- 172.21.15.1:0/1905792690 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782746540 0 0) 0x7f04fc0c9550 con 0x7f04fc0c8d10 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.908+0000 7f04fa7fc640 1 -- 172.21.15.1:0/1905792690 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04fc155200 con 0x7f04fc0c8d10 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.908+0000 7f04fa7fc640 1 -- 172.21.15.1:0/1905792690 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3494706547 0 0) 0x7f04e8003520 con 0x7f04fc0c90f0 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.908+0000 7f04fb7fe640 1 -- 172.21.15.1:0/1905792690 >> v1:172.21.15.98:6789/0 conn(0x7f04fc0c8d10 legacy=0x7f04fc1590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.908+0000 7f04fb7fe640 1 -- 172.21.15.1:0/1905792690 >> v1:172.21.15.1:6789/0 conn(0x7f04fc0c90f0 legacy=0x7f04fc1556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.908+0000 7f04fb7fe640 1 -- 172.21.15.1:0/1905792690 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc0c36c0 con 0x7f04fc155dc0 2024-03-06T07:27:46.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.909+0000 7f04fb7fe640 1 --2- 172.21.15.1:0/1905792690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04fc155dc0 0x7f04fc160a80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f04f4009d90 tx=0x7f04f402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9fd9f6c6be85d04 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.909+0000 7f04fa7fc640 1 -- 172.21.15.1:0/1905792690 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f4039070 con 0x7f04fc155dc0 2024-03-06T07:27:46.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.909+0000 7f04fa7fc640 1 -- 172.21.15.1:0/1905792690 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f402c940 con 0x7f04fc155dc0 2024-03-06T07:27:46.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.909+0000 7f04fa7fc640 1 -- 172.21.15.1:0/1905792690 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e4001670 con 0x7f04fc155dc0 2024-03-06T07:27:46.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.910+0000 7f050173a640 1 -- 172.21.15.1:0/1905792690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04fc155dc0 msgr2=0x7f04fc160a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.910+0000 7f050173a640 1 --2- 172.21.15.1:0/1905792690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04fc155dc0 0x7f04fc160a80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f04f4009d90 tx=0x7f04f402be10 comp rx=0 tx=0).stop 2024-03-06T07:27:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.910+0000 7f050173a640 1 -- 172.21.15.1:0/1905792690 shutdown_connections 2024-03-06T07:27:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.910+0000 7f050173a640 1 --2- 172.21.15.1:0/1905792690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04fc155dc0 0x7f04fc160a80 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.910+0000 7f050173a640 1 -- 172.21.15.1:0/1905792690 >> 172.21.15.1:0/1905792690 conn(0x7f04fc06b3e0 msgr2=0x7f04fc06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.910+0000 7f050173a640 1 -- 172.21.15.1:0/1905792690 shutdown_connections 2024-03-06T07:27:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.910+0000 7f050173a640 1 -- 172.21.15.1:0/1905792690 wait complete. 2024-03-06T07:27:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.911+0000 7f050173a640 1 Processor -- start 2024-03-06T07:27:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.911+0000 7f050173a640 1 -- start start 2024-03-06T07:27:46.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f050173a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04fc0c8d10 0x7f04fc0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f050173a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04fc0c90f0 0x7f04fc0ac210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f050173a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04fc155dc0 0x7f04fc1ee510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f050173a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc1529e0 con 0x7f04fc0c90f0 2024-03-06T07:27:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f050173a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc0c7560 con 0x7f04fc155dc0 2024-03-06T07:27:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f050173a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc152b60 con 0x7f04fc0c8d10 2024-03-06T07:27:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f04f3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04fc0c90f0 0x7f04fc0ac210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f04f3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04fc0c90f0 0x7f04fc0ac210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58636/0 (socket says 172.21.15.1:58636) 2024-03-06T07:27:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f04f3fff640 1 -- 172.21.15.1:0/156477033 learned_addr learned my addr 172.21.15.1:0/156477033 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f04f3fff640 1 -- 172.21.15.1:0/156477033 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04fc0c8d10 msgr2=0x7f04fc0ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f04f3fff640 1 --2- 172.21.15.1:0/156477033 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04fc0c8d10 0x7f04fc0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f04f3fff640 1 -- 172.21.15.1:0/156477033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04fc155dc0 msgr2=0x7f04fc1ee510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f04fb7fe640 1 --2- 172.21.15.1:0/156477033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04fc155dc0 0x7f04fc1ee510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f04f3fff640 1 --2- 172.21.15.1:0/156477033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04fc155dc0 0x7f04fc1ee510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f04f3fff640 1 -- 172.21.15.1:0/156477033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc150a10 con 0x7f04fc0c90f0 2024-03-06T07:27:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f04fb7fe640 1 --2- 172.21.15.1:0/156477033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04fc155dc0 0x7f04fc1ee510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f04f3fff640 1 --2- 172.21.15.1:0/156477033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04fc0c90f0 0x7f04fc0ac210 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f04e400c780 tx=0x7f04e400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.912+0000 7f04f37fe640 1 -- 172.21.15.1:0/156477033 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e4017070 con 0x7f04fc0c90f0 2024-03-06T07:27:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.913+0000 7f04f37fe640 1 -- 172.21.15.1:0/156477033 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e4005ad0 con 0x7f04fc0c90f0 2024-03-06T07:27:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.913+0000 7f04f37fe640 1 -- 172.21.15.1:0/156477033 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e4007070 con 0x7f04fc0c90f0 2024-03-06T07:27:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.913+0000 7f050173a640 1 -- 172.21.15.1:0/156477033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04fc0ad700 con 0x7f04fc0c90f0 2024-03-06T07:27:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.913+0000 7f050173a640 1 -- 172.21.15.1:0/156477033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04fc0ac750 con 0x7f04fc0c90f0 2024-03-06T07:27:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.913+0000 7f050173a640 1 -- 172.21.15.1:0/156477033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04fc0aa220 con 0x7f04fc0c90f0 2024-03-06T07:27:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.914+0000 7f04f37fe640 1 -- 172.21.15.1:0/156477033 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04e401a070 con 0x7f04fc0c90f0 2024-03-06T07:27:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.914+0000 7f04f37fe640 1 --2- 172.21.15.1:0/156477033 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f04dc06b4d0 0x7f04dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.915+0000 7f04f37fe640 1 -- 172.21.15.1:0/156477033 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f04e4089670 con 0x7f04fc0c90f0 2024-03-06T07:27:46.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.915+0000 7f04faffd640 1 --2- 172.21.15.1:0/156477033 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f04dc06b4d0 0x7f04dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.915+0000 7f04faffd640 1 --2- 172.21.15.1:0/156477033 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f04dc06b4d0 0x7f04dc06d990 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0010e0 tx=0x7f04ec007950 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:46.919+0000 7f04f37fe640 1 -- 172.21.15.1:0/156477033 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04fc0aa220 con 0x7f04fc0c90f0 2024-03-06T07:27:47.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.103+0000 7f050173a640 1 -- 172.21.15.1:0/156477033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f04fc067fd0 con 0x7f04fc0c90f0 2024-03-06T07:27:47.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.106+0000 7f04f37fe640 1 -- 172.21.15.1:0/156477033 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v59)=0 v59) v1 ==== 122+0+0 (secure 0 0 0) 0x7f04e4058660 con 0x7f04fc0c90f0 2024-03-06T07:27:47.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.108+0000 7f050173a640 1 -- 172.21.15.1:0/156477033 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f04dc06b4d0 msgr2=0x7f04dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.108+0000 7f050173a640 1 --2- 172.21.15.1:0/156477033 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f04dc06b4d0 0x7f04dc06d990 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0010e0 tx=0x7f04ec007950 comp rx=0 tx=0).stop 2024-03-06T07:27:47.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.108+0000 7f050173a640 1 -- 172.21.15.1:0/156477033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04fc0c90f0 msgr2=0x7f04fc0ac210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.108+0000 7f050173a640 1 --2- 172.21.15.1:0/156477033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04fc0c90f0 0x7f04fc0ac210 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f04e400c780 tx=0x7f04e400cc40 comp rx=0 tx=0).stop 2024-03-06T07:27:47.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.108+0000 7f050173a640 1 -- 172.21.15.1:0/156477033 shutdown_connections 2024-03-06T07:27:47.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.108+0000 7f050173a640 1 --2- 172.21.15.1:0/156477033 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f04dc06b4d0 0x7f04dc06d990 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.108+0000 7f050173a640 1 --2- 172.21.15.1:0/156477033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04fc155dc0 0x7f04fc1ee510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.108+0000 7f050173a640 1 --2- 172.21.15.1:0/156477033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04fc0c90f0 0x7f04fc0ac210 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.108+0000 7f050173a640 1 --2- 172.21.15.1:0/156477033 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f04fc0c8d10 0x7f04fc0ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.108+0000 7f050173a640 1 -- 172.21.15.1:0/156477033 >> 172.21.15.1:0/156477033 conn(0x7f04fc06b3e0 msgr2=0x7f04fc06b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:47.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.108+0000 7f050173a640 1 -- 172.21.15.1:0/156477033 shutdown_connections 2024-03-06T07:27:47.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.108+0000 7f050173a640 1 -- 172.21.15.1:0/156477033 wait complete. 2024-03-06T07:27:47.121 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:27:47.121 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:47.164 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:47.207 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:27:47.207 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:27:47.207 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:47.222 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:27:47.222 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:27:47.222 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:47.264 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:27:47.264 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:27:47.264 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:47.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.363+0000 7fe1992b1640 1 Processor -- start 2024-03-06T07:27:47.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.363+0000 7fe1992b1640 1 -- start start 2024-03-06T07:27:47.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.364+0000 7fe1992b1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1940c32f0 0x7fe1941543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.364+0000 7fe1992b1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe194070c80 0x7fe1940710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.364+0000 7fe1992b1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe194150900 con 0x7fe194150520 2024-03-06T07:27:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.364+0000 7fe1992b1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe194153ed0 con 0x7fe1940c32f0 2024-03-06T07:27:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.364+0000 7fe1992b1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe194071790 con 0x7fe194070c80 2024-03-06T07:27:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.364+0000 7fe1937fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe194070c80 0x7fe1940710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.364+0000 7fe1927fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1940c32f0 0x7fe1941543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.364+0000 7fe192ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe194150520 0x7fe1941537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:35426/0 (socket says 172.21.15.1:35426) 2024-03-06T07:27:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.364+0000 7fe1937fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe194070c80 0x7fe1940710c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:50270/0 (socket says 172.21.15.1:50270) 2024-03-06T07:27:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.364+0000 7fe192ffd640 1 -- 172.21.15.1:0/4291502119 learned_addr learned my addr 172.21.15.1:0/4291502119 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.365+0000 7fe191ffb640 1 -- 172.21.15.1:0/4291502119 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956920369 0 0) 0x7fe194150900 con 0x7fe194150520 2024-03-06T07:27:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.365+0000 7fe191ffb640 1 -- 172.21.15.1:0/4291502119 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe180003520 con 0x7fe194150520 2024-03-06T07:27:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.365+0000 7fe1937fe640 1 -- 172.21.15.1:0/4291502119 >> v2:172.21.15.98:3300/0 conn(0x7fe1940c32f0 msgr2=0x7fe1941543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.365+0000 7fe1937fe640 1 --2- 172.21.15.1:0/4291502119 >> v2:172.21.15.98:3300/0 conn(0x7fe1940c32f0 0x7fe1941543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.365+0000 7fe1937fe640 1 -- 172.21.15.1:0/4291502119 >> v1:172.21.15.98:6789/0 conn(0x7fe194150520 legacy=0x7fe1941537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.365+0000 7fe1937fe640 1 -- 172.21.15.1:0/4291502119 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe194157050 con 0x7fe194070c80 2024-03-06T07:27:47.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.365+0000 7fe1927fc640 1 --2- 172.21.15.1:0/4291502119 >> v2:172.21.15.98:3300/0 conn(0x7fe1940c32f0 0x7fe1941543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:47.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.365+0000 7fe1937fe640 1 --2- 172.21.15.1:0/4291502119 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe194070c80 0x7fe1940710c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe184009d90 tx=0x7fe18402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37aa945be13ab2fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.366+0000 7fe191ffb640 1 -- 172.21.15.1:0/4291502119 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184039070 con 0x7fe194070c80 2024-03-06T07:27:47.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.366+0000 7fe191ffb640 1 -- 172.21.15.1:0/4291502119 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18402cc50 con 0x7fe194070c80 2024-03-06T07:27:47.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.366+0000 7fe191ffb640 1 -- 172.21.15.1:0/4291502119 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184034440 con 0x7fe194070c80 2024-03-06T07:27:47.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.366+0000 7fe1992b1640 1 -- 172.21.15.1:0/4291502119 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe194070c80 msgr2=0x7fe1940710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.366+0000 7fe1992b1640 1 --2- 172.21.15.1:0/4291502119 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe194070c80 0x7fe1940710c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe184009d90 tx=0x7fe18402bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.366+0000 7fe1992b1640 1 -- 172.21.15.1:0/4291502119 shutdown_connections 2024-03-06T07:27:47.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.366+0000 7fe1992b1640 1 --2- 172.21.15.1:0/4291502119 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe194070c80 0x7fe1940710c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.366+0000 7fe1992b1640 1 --2- 172.21.15.1:0/4291502119 >> v2:172.21.15.98:3300/0 conn(0x7fe1940c32f0 0x7fe1941543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.366+0000 7fe1992b1640 1 -- 172.21.15.1:0/4291502119 >> 172.21.15.1:0/4291502119 conn(0x7fe1940c88f0 msgr2=0x7fe1940c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:47.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.366+0000 7fe1992b1640 1 -- 172.21.15.1:0/4291502119 shutdown_connections 2024-03-06T07:27:47.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.366+0000 7fe1992b1640 1 -- 172.21.15.1:0/4291502119 wait complete. 2024-03-06T07:27:47.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.367+0000 7fe1992b1640 1 Processor -- start 2024-03-06T07:27:47.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.367+0000 7fe1992b1640 1 -- start start 2024-03-06T07:27:47.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.368+0000 7fe1992b1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe194070c80 0x7fe194143210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.368+0000 7fe1992b1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1940c32f0 0x7fe1941ebd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.368+0000 7fe1992b1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe194150520 0x7fe1940c40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.368+0000 7fe1992b1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe194157330 con 0x7fe194150520 2024-03-06T07:27:47.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.368+0000 7fe1992b1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1941532a0 con 0x7fe1940c32f0 2024-03-06T07:27:47.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.368+0000 7fe1992b1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1940c7ec0 con 0x7fe194070c80 2024-03-06T07:27:47.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.368+0000 7fe1937fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe194150520 0x7fe1940c40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.368+0000 7fe1937fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe194150520 0x7fe1940c40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58640/0 (socket says 172.21.15.1:58640) 2024-03-06T07:27:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.368+0000 7fe1937fe640 1 -- 172.21.15.1:0/2181852590 learned_addr learned my addr 172.21.15.1:0/2181852590 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.368+0000 7fe1927fc640 1 --2- 172.21.15.1:0/2181852590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1940c32f0 0x7fe1941ebd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.368+0000 7fe192ffd640 1 --2- 172.21.15.1:0/2181852590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe194070c80 0x7fe194143210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.369+0000 7fe1937fe640 1 -- 172.21.15.1:0/2181852590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe194070c80 msgr2=0x7fe194143210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.369+0000 7fe1937fe640 1 --2- 172.21.15.1:0/2181852590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe194070c80 0x7fe194143210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.369+0000 7fe1937fe640 1 -- 172.21.15.1:0/2181852590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1940c32f0 msgr2=0x7fe1941ebd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.369+0000 7fe1937fe640 1 --2- 172.21.15.1:0/2181852590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1940c32f0 0x7fe1941ebd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.369+0000 7fe1937fe640 1 -- 172.21.15.1:0/2181852590 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19414e750 con 0x7fe194150520 2024-03-06T07:27:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.369+0000 7fe1937fe640 1 --2- 172.21.15.1:0/2181852590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe194150520 0x7fe1940c40a0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fe184009eb0 tx=0x7fe184004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.369+0000 7fe18b7fe640 1 -- 172.21.15.1:0/2181852590 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184039050 con 0x7fe194150520 2024-03-06T07:27:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.369+0000 7fe18b7fe640 1 -- 172.21.15.1:0/2181852590 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe184045070 con 0x7fe194150520 2024-03-06T07:27:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.369+0000 7fe1992b1640 1 -- 172.21.15.1:0/2181852590 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1940c5770 con 0x7fe194150520 2024-03-06T07:27:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.369+0000 7fe18b7fe640 1 -- 172.21.15.1:0/2181852590 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184009450 con 0x7fe194150520 2024-03-06T07:27:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.369+0000 7fe1992b1640 1 -- 172.21.15.1:0/2181852590 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19414af90 con 0x7fe194150520 2024-03-06T07:27:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.370+0000 7fe1992b1640 1 -- 172.21.15.1:0/2181852590 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1940686e0 con 0x7fe194150520 2024-03-06T07:27:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.370+0000 7fe18b7fe640 1 -- 172.21.15.1:0/2181852590 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe18403e610 con 0x7fe194150520 2024-03-06T07:27:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.371+0000 7fe18b7fe640 1 --2- 172.21.15.1:0/2181852590 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe17406b500 0x7fe17406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.371+0000 7fe18b7fe640 1 -- 172.21.15.1:0/2181852590 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fe1840ac900 con 0x7fe194150520 2024-03-06T07:27:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.371+0000 7fe192ffd640 1 --2- 172.21.15.1:0/2181852590 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe17406b500 0x7fe17406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.372+0000 7fe192ffd640 1 --2- 172.21.15.1:0/2181852590 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe17406b500 0x7fe17406d9c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fe18c005880 tx=0x7fe18c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.375+0000 7fe18b7fe640 1 -- 172.21.15.1:0/2181852590 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe18407b480 con 0x7fe194150520 2024-03-06T07:27:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.596+0000 7fe1992b1640 1 -- 172.21.15.1:0/2181852590 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1940c7b10 con 0x7fe194150520 2024-03-06T07:27:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.596+0000 7fe18b7fe640 1 -- 172.21.15.1:0/2181852590 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+3680 (secure 0 0 0) 0x7fe18407f520 con 0x7fe194150520 2024-03-06T07:27:47.598 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:47.598 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":166,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/555882678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":555882678},{"type":"v1","addr":"172.21.15.98:6836","nonce":555882678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":8284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/69628419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":69628419},{"type":"v1","addr":"172.21.15.98:6837","nonce":69628419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/840119087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":840119087},{"type":"v1","addr":"172.21.15.1:6834","nonce":840119087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":9882,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/48122843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":48122843},{"type":"v1","addr":"172.21.15.1:6835","nonce":48122843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":166}],"filesystems":[]} 2024-03-06T07:27:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 166 2024-03-06T07:27:47.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.598+0000 7fe1992b1640 1 -- 172.21.15.1:0/2181852590 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe17406b500 msgr2=0x7fe17406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.598+0000 7fe1992b1640 1 --2- 172.21.15.1:0/2181852590 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe17406b500 0x7fe17406d9c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fe18c005880 tx=0x7fe18c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.598+0000 7fe1992b1640 1 -- 172.21.15.1:0/2181852590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe194150520 msgr2=0x7fe1940c40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.598+0000 7fe1992b1640 1 --2- 172.21.15.1:0/2181852590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe194150520 0x7fe1940c40a0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fe184009eb0 tx=0x7fe184004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.598+0000 7fe1992b1640 1 -- 172.21.15.1:0/2181852590 shutdown_connections 2024-03-06T07:27:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.598+0000 7fe1992b1640 1 --2- 172.21.15.1:0/2181852590 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe17406b500 0x7fe17406d9c0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.598+0000 7fe1992b1640 1 --2- 172.21.15.1:0/2181852590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe194150520 0x7fe1940c40a0 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.598+0000 7fe1992b1640 1 --2- 172.21.15.1:0/2181852590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1940c32f0 0x7fe1941ebd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.598+0000 7fe1992b1640 1 --2- 172.21.15.1:0/2181852590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe194070c80 0x7fe194143210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.598+0000 7fe1992b1640 1 -- 172.21.15.1:0/2181852590 >> 172.21.15.1:0/2181852590 conn(0x7fe1940c88f0 msgr2=0x7fe194071060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.598+0000 7fe1992b1640 1 -- 172.21.15.1:0/2181852590 shutdown_connections 2024-03-06T07:27:47.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.598+0000 7fe1992b1640 1 -- 172.21.15.1:0/2181852590 wait complete. 2024-03-06T07:27:47.612 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:47.612 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:27:47.612 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:27:47.612 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:47.612 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:47.612 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:27:47.612 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:27:47.613 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:47.613 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:47.613 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:27:47.613 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:27:47.613 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:47.613 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:47.613 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:27:47.613 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:27:47.614 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:47.621 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:27:47.621 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:47.622 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:27:47.622 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:47.627 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:27:47.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:47.628 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:27:47.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:47.664 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:47.664 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:47.665 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:47.665 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:47.670 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:47.670 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:47.671 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:47.671 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:47.671 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:27:47.740 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:27:47.740 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:27:47.753 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:27:47.753 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:27:47.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.776+0000 7febc44df640 1 Processor -- start 2024-03-06T07:27:47.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.776+0000 7febc44df640 1 -- start start 2024-03-06T07:27:47.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.777+0000 7febc44df640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7febbc152650 0x7febbc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.777+0000 7febc44df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febbc0bf9d0 0x7febbc0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.777+0000 7febc44df640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc0c2fc0 con 0x7febbc0c6b10 2024-03-06T07:27:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.777+0000 7febc44df640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febbc155800 con 0x7febbc152650 2024-03-06T07:27:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.777+0000 7febc44df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febbc158da0 con 0x7febbc0bf9d0 2024-03-06T07:27:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.777+0000 7febc1a7a640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7febbc0c6b10 0x7febbc158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57838/0 (socket says 172.21.15.1:57838) 2024-03-06T07:27:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.777+0000 7febc1a7a640 1 -- 172.21.15.1:0/2863039495 learned_addr learned my addr 172.21.15.1:0/2863039495 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.777+0000 7febc227b640 1 --2- 172.21.15.1:0/2863039495 >> v2:172.21.15.1:3300/0 conn(0x7febbc152650 0x7febbc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.777+0000 7febc2a7c640 1 --2- 172.21.15.1:0/2863039495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febbc0bf9d0 0x7febbc0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.777+0000 7febc227b640 1 -- 172.21.15.1:0/2863039495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febbc0bf9d0 msgr2=0x7febbc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.777+0000 7febc227b640 1 --2- 172.21.15.1:0/2863039495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febbc0bf9d0 0x7febbc0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.777+0000 7febc227b640 1 -- 172.21.15.1:0/2863039495 >> v1:172.21.15.1:6789/0 conn(0x7febbc0c6b10 legacy=0x7febbc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.778+0000 7febc227b640 1 -- 172.21.15.1:0/2863039495 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febbc0c6690 con 0x7febbc152650 2024-03-06T07:27:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.777+0000 7febc1279640 1 -- 172.21.15.1:0/2863039495 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838563722 0 0) 0x7febbc0c2fc0 con 0x7febbc0c6b10 2024-03-06T07:27:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.778+0000 7febc2a7c640 1 --2- 172.21.15.1:0/2863039495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febbc0bf9d0 0x7febbc0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.778+0000 7febc227b640 1 --2- 172.21.15.1:0/2863039495 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febbc152650 0x7febbc155c90 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7febb80097a0 tx=0x7febb802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e927d050c6f2e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.778+0000 7febc1279640 1 -- 172.21.15.1:0/2863039495 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb8031780 con 0x7febbc152650 2024-03-06T07:27:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.778+0000 7febc1279640 1 -- 172.21.15.1:0/2863039495 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb8037070 con 0x7febbc152650 2024-03-06T07:27:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.778+0000 7febc1279640 1 -- 172.21.15.1:0/2863039495 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb8031b40 con 0x7febbc152650 2024-03-06T07:27:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.779+0000 7febc44df640 1 -- 172.21.15.1:0/2863039495 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febbc152650 msgr2=0x7febbc155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.779+0000 7febc44df640 1 --2- 172.21.15.1:0/2863039495 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febbc152650 0x7febbc155c90 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7febb80097a0 tx=0x7febb802b820 comp rx=0 tx=0).stop 2024-03-06T07:27:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.779+0000 7febc44df640 1 -- 172.21.15.1:0/2863039495 shutdown_connections 2024-03-06T07:27:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.779+0000 7febc44df640 1 --2- 172.21.15.1:0/2863039495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febbc0bf9d0 0x7febbc0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.779+0000 7febc44df640 1 --2- 172.21.15.1:0/2863039495 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febbc152650 0x7febbc155c90 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.779+0000 7febc44df640 1 -- 172.21.15.1:0/2863039495 >> 172.21.15.1:0/2863039495 conn(0x7febbc066ee0 msgr2=0x7febbc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.779+0000 7febc44df640 1 -- 172.21.15.1:0/2863039495 shutdown_connections 2024-03-06T07:27:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.779+0000 7febc44df640 1 -- 172.21.15.1:0/2863039495 wait complete. 2024-03-06T07:27:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.780+0000 7febc44df640 1 Processor -- start 2024-03-06T07:27:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.780+0000 7febc44df640 1 -- start start 2024-03-06T07:27:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.781+0000 7febc44df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febbc0bf9d0 0x7febbc1ef020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.781+0000 7febc44df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febbc152650 0x7febbc14e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.781+0000 7febc44df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febbc152e20 0x7febbc14b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.781+0000 7febc44df640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febbc1551b0 con 0x7febbc0bf9d0 2024-03-06T07:27:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.781+0000 7febc44df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febbc0c6fa0 con 0x7febbc152e20 2024-03-06T07:27:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.781+0000 7febc227b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febbc0bf9d0 0x7febbc1ef020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.781+0000 7febc44df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febbc155490 con 0x7febbc152650 2024-03-06T07:27:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.781+0000 7febc227b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febbc0bf9d0 0x7febbc1ef020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58682/0 (socket says 172.21.15.1:58682) 2024-03-06T07:27:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.781+0000 7febc1a7a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febbc152650 0x7febbc14e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.781+0000 7febc227b640 1 -- 172.21.15.1:0/2119341028 learned_addr learned my addr 172.21.15.1:0/2119341028 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.781+0000 7febc2a7c640 1 --2- 172.21.15.1:0/2119341028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febbc152e20 0x7febbc14b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.781+0000 7febc227b640 1 -- 172.21.15.1:0/2119341028 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febbc152650 msgr2=0x7febbc14e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.781+0000 7febc227b640 1 --2- 172.21.15.1:0/2119341028 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febbc152650 0x7febbc14e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.781+0000 7febc227b640 1 -- 172.21.15.1:0/2119341028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febbc152e20 msgr2=0x7febbc14b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.781+0000 7febc227b640 1 --2- 172.21.15.1:0/2119341028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febbc152e20 0x7febbc14b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.782+0000 7febc227b640 1 -- 172.21.15.1:0/2119341028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febbc0ca300 con 0x7febbc0bf9d0 2024-03-06T07:27:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.782+0000 7febc1a7a640 1 --2- 172.21.15.1:0/2119341028 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febbc152650 0x7febbc14e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.782+0000 7febc227b640 1 --2- 172.21.15.1:0/2119341028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febbc0bf9d0 0x7febbc1ef020 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7febb80098c0 tx=0x7febb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.783+0000 7febb2ffd640 1 -- 172.21.15.1:0/2119341028 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb8032820 con 0x7febbc0bf9d0 2024-03-06T07:27:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.783+0000 7febb2ffd640 1 -- 172.21.15.1:0/2119341028 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb80329e0 con 0x7febbc0bf9d0 2024-03-06T07:27:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.783+0000 7febc44df640 1 -- 172.21.15.1:0/2119341028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febbc0ca5e0 con 0x7febbc0bf9d0 2024-03-06T07:27:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.783+0000 7febc44df640 1 -- 172.21.15.1:0/2119341028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febbc14eeb0 con 0x7febbc0bf9d0 2024-03-06T07:27:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.783+0000 7febb2ffd640 1 -- 172.21.15.1:0/2119341028 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb8032dd0 con 0x7febbc0bf9d0 2024-03-06T07:27:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.785+0000 7febc44df640 1 -- 172.21.15.1:0/2119341028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febbc0c59c0 con 0x7febbc0bf9d0 2024-03-06T07:27:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.785+0000 7febb2ffd640 1 -- 172.21.15.1:0/2119341028 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7febb804e440 con 0x7febbc0bf9d0 2024-03-06T07:27:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.785+0000 7febb2ffd640 1 --2- 172.21.15.1:0/2119341028 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feba006b4d0 0x7feba006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.786+0000 7febb2ffd640 1 -- 172.21.15.1:0/2119341028 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5335+0+0 (secure 0 0 0) 0x7febb80abc90 con 0x7febbc0bf9d0 2024-03-06T07:27:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.786+0000 7febc1a7a640 1 --2- 172.21.15.1:0/2119341028 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feba006b4d0 0x7feba006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.786+0000 7febc1a7a640 1 --2- 172.21.15.1:0/2119341028 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feba006b4d0 0x7feba006d990 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7febbc1e8f30 tx=0x7febac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.790+0000 7febb2ffd640 1 -- 172.21.15.1:0/2119341028 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7febbc0c59c0 con 0x7febbc0bf9d0 2024-03-06T07:27:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:47.985+0000 7febc44df640 1 -- 172.21.15.1:0/2119341028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7febbc0c85a0 con 0x7febbc0bf9d0 2024-03-06T07:27:48.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.802+0000 7febb2ffd640 1 -- 172.21.15.1:0/2119341028 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v179) v1 ==== 97+0+0 (secure 0 0 0) 0x7febbc0c85a0 con 0x7febbc0bf9d0 2024-03-06T07:27:48.804 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-06T07:27:48.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.804+0000 7febc44df640 1 -- 172.21.15.1:0/2119341028 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feba006b4d0 msgr2=0x7feba006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.804+0000 7febc44df640 1 --2- 172.21.15.1:0/2119341028 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feba006b4d0 0x7feba006d990 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7febbc1e8f30 tx=0x7febac009040 comp rx=0 tx=0).stop 2024-03-06T07:27:48.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.804+0000 7febc44df640 1 -- 172.21.15.1:0/2119341028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febbc0bf9d0 msgr2=0x7febbc1ef020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.804+0000 7febc44df640 1 --2- 172.21.15.1:0/2119341028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febbc0bf9d0 0x7febbc1ef020 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7febb80098c0 tx=0x7febb80046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.805+0000 7febc44df640 1 -- 172.21.15.1:0/2119341028 shutdown_connections 2024-03-06T07:27:48.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.805+0000 7febc44df640 1 --2- 172.21.15.1:0/2119341028 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feba006b4d0 0x7feba006d990 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.805+0000 7febc44df640 1 --2- 172.21.15.1:0/2119341028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febbc152e20 0x7febbc14b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.805+0000 7febc44df640 1 --2- 172.21.15.1:0/2119341028 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febbc152650 0x7febbc14e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.805+0000 7febc44df640 1 --2- 172.21.15.1:0/2119341028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febbc0bf9d0 0x7febbc1ef020 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.805+0000 7febc44df640 1 -- 172.21.15.1:0/2119341028 >> 172.21.15.1:0/2119341028 conn(0x7febbc066ee0 msgr2=0x7febbc0c52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:48.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.805+0000 7febc44df640 1 -- 172.21.15.1:0/2119341028 shutdown_connections 2024-03-06T07:27:48.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.805+0000 7febc44df640 1 -- 172.21.15.1:0/2119341028 wait complete. 2024-03-06T07:27:48.820 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:27:48.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.929+0000 7f0838f5e640 1 Processor -- start 2024-03-06T07:27:48.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.929+0000 7f0838f5e640 1 -- start start 2024-03-06T07:27:48.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.930+0000 7f0838f5e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f08340c16f0 0x7f0834156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.930+0000 7f0838f5e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08341530a0 con 0x7f0834149a90 2024-03-06T07:27:48.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.930+0000 7f0838f5e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0834153380 con 0x7f08340c1260 2024-03-06T07:27:48.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.930+0000 7f0838f5e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0834153660 con 0x7f08340c16f0 2024-03-06T07:27:48.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.930+0000 7f083259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0834149a90 0x7f08340c4760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57840/0 (socket says 172.21.15.1:57840) 2024-03-06T07:27:48.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.930+0000 7f083259c640 1 -- 172.21.15.1:0/1875717190 learned_addr learned my addr 172.21.15.1:0/1875717190 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:48.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.930+0000 7f0832d9d640 1 --2- 172.21.15.1:0/1875717190 >> v2:172.21.15.1:3300/0 conn(0x7f08340c16f0 0x7f0834156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.930+0000 7f0832d9d640 1 -- 172.21.15.1:0/1875717190 >> v1:172.21.15.98:6789/0 conn(0x7f08340c1260 legacy=0x7f08341565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.930+0000 7f0832d9d640 1 -- 172.21.15.1:0/1875717190 >> v1:172.21.15.1:6789/0 conn(0x7f0834149a90 legacy=0x7f08340c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.930+0000 7f083159a640 1 -- 172.21.15.1:0/1875717190 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891620231 0 0) 0x7f08341530a0 con 0x7f0834149a90 2024-03-06T07:27:48.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.931+0000 7f0832d9d640 1 -- 172.21.15.1:0/1875717190 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0834153940 con 0x7f08340c16f0 2024-03-06T07:27:48.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.931+0000 7f0832d9d640 1 --2- 172.21.15.1:0/1875717190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08340c16f0 0x7f0834156cb0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f0824009d90 tx=0x7f082402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a76b998475f3e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.931+0000 7f083159a640 1 -- 172.21.15.1:0/1875717190 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0824039070 con 0x7f08340c16f0 2024-03-06T07:27:48.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.931+0000 7f083159a640 1 -- 172.21.15.1:0/1875717190 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082402cc60 con 0x7f08340c16f0 2024-03-06T07:27:48.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.931+0000 7f083159a640 1 -- 172.21.15.1:0/1875717190 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0824034660 con 0x7f08340c16f0 2024-03-06T07:27:48.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.932+0000 7f0838f5e640 1 -- 172.21.15.1:0/1875717190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08340c16f0 msgr2=0x7f0834156cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.932+0000 7f0838f5e640 1 --2- 172.21.15.1:0/1875717190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08340c16f0 0x7f0834156cb0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f0824009d90 tx=0x7f082402beb0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.932+0000 7f0838f5e640 1 -- 172.21.15.1:0/1875717190 shutdown_connections 2024-03-06T07:27:48.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.932+0000 7f0838f5e640 1 --2- 172.21.15.1:0/1875717190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08340c16f0 0x7f0834156cb0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.932+0000 7f0838f5e640 1 -- 172.21.15.1:0/1875717190 >> 172.21.15.1:0/1875717190 conn(0x7f08340c6860 msgr2=0x7f08340c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:48.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.932+0000 7f0838f5e640 1 -- 172.21.15.1:0/1875717190 shutdown_connections 2024-03-06T07:27:48.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.932+0000 7f0838f5e640 1 -- 172.21.15.1:0/1875717190 wait complete. 2024-03-06T07:27:48.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.933+0000 7f0838f5e640 1 Processor -- start 2024-03-06T07:27:48.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.933+0000 7f0838f5e640 1 -- start start 2024-03-06T07:27:48.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.934+0000 7f0838f5e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08340c1260 0x7f08341dd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.934+0000 7f0838f5e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08340c16f0 0x7f083414fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.934+0000 7f0838f5e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0834149a90 0x7f0834150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.934+0000 7f0838f5e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083406f4a0 con 0x7f08340c1260 2024-03-06T07:27:48.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.934+0000 7f0838f5e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08341571f0 con 0x7f0834149a90 2024-03-06T07:27:48.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.934+0000 7f0838f5e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083406f620 con 0x7f08340c16f0 2024-03-06T07:27:48.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.934+0000 7f083259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08340c1260 0x7f08341dd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.934+0000 7f083259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08340c1260 0x7f08341dd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58708/0 (socket says 172.21.15.1:58708) 2024-03-06T07:27:48.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.934+0000 7f083259c640 1 -- 172.21.15.1:0/350212553 learned_addr learned my addr 172.21.15.1:0/350212553 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:48.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.934+0000 7f0831d9b640 1 --2- 172.21.15.1:0/350212553 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08340c16f0 0x7f083414fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.934+0000 7f083259c640 1 -- 172.21.15.1:0/350212553 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08340c16f0 msgr2=0x7f083414fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.934+0000 7f0832d9d640 1 --2- 172.21.15.1:0/350212553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0834149a90 0x7f0834150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.934+0000 7f083259c640 1 --2- 172.21.15.1:0/350212553 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08340c16f0 0x7f083414fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.934+0000 7f083259c640 1 -- 172.21.15.1:0/350212553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0834149a90 msgr2=0x7f0834150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.935+0000 7f083259c640 1 --2- 172.21.15.1:0/350212553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0834149a90 0x7f0834150010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.935+0000 7f083259c640 1 -- 172.21.15.1:0/350212553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08341ec730 con 0x7f08340c1260 2024-03-06T07:27:48.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.935+0000 7f0832d9d640 1 --2- 172.21.15.1:0/350212553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0834149a90 0x7f0834150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:48.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.935+0000 7f083259c640 1 --2- 172.21.15.1:0/350212553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08340c1260 0x7f08341dd6e0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f082c00c3f0 tx=0x7f082c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.935+0000 7f082affd640 1 -- 172.21.15.1:0/350212553 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c0074e0 con 0x7f08340c1260 2024-03-06T07:27:48.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.935+0000 7f082affd640 1 -- 172.21.15.1:0/350212553 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c0047e0 con 0x7f08340c1260 2024-03-06T07:27:48.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.935+0000 7f082affd640 1 -- 172.21.15.1:0/350212553 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c004c10 con 0x7f08340c1260 2024-03-06T07:27:48.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.935+0000 7f0838f5e640 1 -- 172.21.15.1:0/350212553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08341dc640 con 0x7f08340c1260 2024-03-06T07:27:48.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.935+0000 7f0838f5e640 1 -- 172.21.15.1:0/350212553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08341ddc20 con 0x7f08340c1260 2024-03-06T07:27:48.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.936+0000 7f0838f5e640 1 -- 172.21.15.1:0/350212553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0834067fd0 con 0x7f08340c1260 2024-03-06T07:27:48.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.936+0000 7f082affd640 1 -- 172.21.15.1:0/350212553 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f082c011070 con 0x7f08340c1260 2024-03-06T07:27:48.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.937+0000 7f082affd640 1 --2- 172.21.15.1:0/350212553 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f081406b4d0 0x7f081406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.937+0000 7f082affd640 1 -- 172.21.15.1:0/350212553 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f082c089c30 con 0x7f08340c1260 2024-03-06T07:27:48.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.937+0000 7f0831d9b640 1 --2- 172.21.15.1:0/350212553 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f081406b4d0 0x7f081406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.938+0000 7f0831d9b640 1 --2- 172.21.15.1:0/350212553 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f081406b4d0 0x7f081406d990 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f083414f590 tx=0x7f081c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:48.941+0000 7f082affd640 1 -- 172.21.15.1:0/350212553 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f082c058c70 con 0x7f08340c1260 2024-03-06T07:27:49.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.196+0000 7f0838f5e640 1 -- 172.21.15.1:0/350212553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f083406f280 con 0x7f08340c1260 2024-03-06T07:27:49.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.197+0000 7f082affd640 1 -- 172.21.15.1:0/350212553 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f082c058570 con 0x7f08340c1260 2024-03-06T07:27:49.199 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:49.199 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:27:49.199 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-06T07:27:49.199 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:49.199 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-06T07:27:49.200 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlnxyWARAAKIPXU0IafNQ1RzBPX+3DYw==", 2024-03-06T07:27:49.200 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:49.200 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:27:49.200 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:27:49.200 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:27:49.200 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:49.200 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.200 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:49.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:49.200 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-06T07:27:49.200 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl2fXxDRAAuLqH+HpKcN1JTts9ZtDTXA==", 2024-03-06T07:27:49.200 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:49.201 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:27:49.201 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:27:49.201 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:27:49.201 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:49.201 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:49.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:49.201 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-06T07:27:49.201 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl+IzqCBAAmgLkarFGt9Ihs/8kJYicBQ==", 2024-03-06T07:27:49.201 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:49.201 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:27:49.201 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:27:49.201 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:27:49.201 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:49.202 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:49.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:49.202 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-06T07:27:49.202 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlipdRFRAAA34vrsTEoX3L1CKb1YAV9g==", 2024-03-06T07:27:49.202 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:49.202 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:27:49.202 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:27:49.202 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:27:49.202 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:49.202 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:49.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:49.203 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-06T07:27:49.203 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAGGehlQK39LxAAOpPIZvl3FqQwCgyo3yoOGQ==", 2024-03-06T07:27:49.203 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:49.203 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:49.203 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:27:49.203 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:49.203 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.203 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:49.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:49.203 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-06T07:27:49.203 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAJGehlTxwDLhAAMwN9E0+3Jh5XNXnLo+fw3g==", 2024-03-06T07:27:49.204 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:49.204 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:49.204 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:27:49.204 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:49.204 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.204 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:49.204 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:49.204 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-06T07:27:49.204 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAMGehl4qrlLBAAm0VWQRNvSQIpDx5xf4Fyig==", 2024-03-06T07:27:49.204 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:49.204 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:49.204 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:27:49.204 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:49.204 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:49.205 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:49.205 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-06T07:27:49.205 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAPGehllO5HKxAAozm6ycG+fRxatoZTogeJTA==", 2024-03-06T07:27:49.205 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:49.205 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:49.205 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:27:49.205 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:49.205 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:49.205 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:49.205 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-06T07:27:49.205 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAVGehlLtLBMhAAUrV6q6ZCQluIcoo/S/NdbA==", 2024-03-06T07:27:49.206 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:49.206 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:49.206 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:27:49.206 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:49.206 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.206 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:49.206 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:49.206 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-06T07:27:49.206 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAYGehlHp59MRAAxxgpSPTEZkYKjl3fr0xeeQ==", 2024-03-06T07:27:49.206 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:49.206 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:49.206 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:27:49.206 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:49.206 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:49.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:49.207 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-06T07:27:49.207 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAbGehlTnXcMBAAFOBqcnh5ubAz+o6AGjccWQ==", 2024-03-06T07:27:49.207 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:49.207 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:49.207 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:27:49.207 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:49.207 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:49.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:49.207 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-06T07:27:49.207 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAeGehl1qH1LhAAaSl7Aj7qlN6d4xt5SKAmcg==", 2024-03-06T07:27:49.207 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:49.207 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:49.208 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:27:49.208 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:49.208 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.208 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:49.208 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:49.208 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-06T07:27:49.208 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng==", 2024-03-06T07:27:49.208 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:49.208 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:27:49.208 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-06T07:27:49.208 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:49.208 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.208 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:49.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:49.209 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-06T07:27:49.209 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlMyv7HBAAi7vt7aZWEaZuMpFrB5ofZw==", 2024-03-06T07:27:49.209 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:49.209 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:27:49.209 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-06T07:27:49.209 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-06T07:27:49.209 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-06T07:27:49.209 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.209 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:49.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:49.209 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-06T07:27:49.209 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlBS/vIBAAN86N2K4ze5PJB8kR869Lfg==", 2024-03-06T07:27:49.209 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:49.209 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:27:49.210 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:27:49.210 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:27:49.210 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:49.210 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:49.210 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:49.210 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-06T07:27:49.210 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlJ/jPORAAhLk5MTGafsGKhZSYhvhBZg==", 2024-03-06T07:27:49.210 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:49.210 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:27:49.210 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:49.210 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:49.210 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:27:49.210 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:49.211 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-06T07:27:49.211 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlAl8rNhAAvdU0Qvo/FJYnxc4/H8MIqQ==", 2024-03-06T07:27:49.211 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:27:49.211 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:27:49.211 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:49.211 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:27:49.211 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.211 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:49.211 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-06T07:27:49.211 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:27:49.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.199+0000 7f0838f5e640 1 -- 172.21.15.1:0/350212553 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f081406b4d0 msgr2=0x7f081406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.199+0000 7f0838f5e640 1 --2- 172.21.15.1:0/350212553 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f081406b4d0 0x7f081406d990 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f083414f590 tx=0x7f081c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:49.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.199+0000 7f0838f5e640 1 -- 172.21.15.1:0/350212553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08340c1260 msgr2=0x7f08341dd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.199+0000 7f0838f5e640 1 --2- 172.21.15.1:0/350212553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08340c1260 0x7f08341dd6e0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f082c00c3f0 tx=0x7f082c00c8b0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.199+0000 7f0838f5e640 1 -- 172.21.15.1:0/350212553 shutdown_connections 2024-03-06T07:27:49.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.199+0000 7f0838f5e640 1 --2- 172.21.15.1:0/350212553 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f081406b4d0 0x7f081406d990 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.199+0000 7f0838f5e640 1 --2- 172.21.15.1:0/350212553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0834149a90 0x7f0834150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.199+0000 7f0838f5e640 1 --2- 172.21.15.1:0/350212553 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08340c16f0 0x7f083414fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.199+0000 7f0838f5e640 1 --2- 172.21.15.1:0/350212553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08340c1260 0x7f08341dd6e0 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.199+0000 7f0838f5e640 1 -- 172.21.15.1:0/350212553 >> 172.21.15.1:0/350212553 conn(0x7f08340c6860 msgr2=0x7f08341559a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.199+0000 7f0838f5e640 1 -- 172.21.15.1:0/350212553 shutdown_connections 2024-03-06T07:27:49.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.199+0000 7f0838f5e640 1 -- 172.21.15.1:0/350212553 wait complete. 2024-03-06T07:27:49.214 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:49.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.313+0000 7f19f1777640 1 Processor -- start 2024-03-06T07:27:49.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.313+0000 7f19f1777640 1 -- start start 2024-03-06T07:27:49.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.313+0000 7f19f1777640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19ec14cdc0 0x7f19ec152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.313+0000 7f19f1777640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f19ec156500 0x7f19ec0c2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.313+0000 7f19f1777640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec1569e0 con 0x7f19ec0c2890 2024-03-06T07:27:49.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.313+0000 7f19f1777640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ec0c2dd0 con 0x7f19ec156500 2024-03-06T07:27:49.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.313+0000 7f19f1777640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec156cc0 con 0x7f19ec14cdc0 2024-03-06T07:27:49.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.314+0000 7f19ea7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f19ec156500 0x7f19ec0c2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.314+0000 7f19ea7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f19ec156500 0x7f19ec0c2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58710/0 (socket says 172.21.15.1:58710) 2024-03-06T07:27:49.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.314+0000 7f19ea7fc640 1 -- 172.21.15.1:0/3630593165 learned_addr learned my addr 172.21.15.1:0/3630593165 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:49.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.314+0000 7f19eaffd640 1 --2- 172.21.15.1:0/3630593165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19ec14cdc0 0x7f19ec152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.314+0000 7f19ea7fc640 1 -- 172.21.15.1:0/3630593165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19ec14cdc0 msgr2=0x7f19ec152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:49.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.314+0000 7f19ea7fc640 1 --2- 172.21.15.1:0/3630593165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19ec14cdc0 0x7f19ec152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.314+0000 7f19ea7fc640 1 -- 172.21.15.1:0/3630593165 >> v1:172.21.15.98:6789/0 conn(0x7f19ec0c2890 legacy=0x7f19ec06b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.314+0000 7f19ea7fc640 1 -- 172.21.15.1:0/3630593165 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec156fa0 con 0x7f19ec156500 2024-03-06T07:27:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.314+0000 7f19eaffd640 1 --2- 172.21.15.1:0/3630593165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19ec14cdc0 0x7f19ec152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.315+0000 7f19ea7fc640 1 --2- 172.21.15.1:0/3630593165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19ec156500 0x7f19ec0c2350 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009870 tx=0x7f19e002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d7bcc3fdf69178 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.315+0000 7f19e9ffb640 1 -- 172.21.15.1:0/3630593165 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0034070 con 0x7f19ec156500 2024-03-06T07:27:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.315+0000 7f19e9ffb640 1 -- 172.21.15.1:0/3630593165 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e0030840 con 0x7f19ec156500 2024-03-06T07:27:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.315+0000 7f19e9ffb640 1 -- 172.21.15.1:0/3630593165 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0030c10 con 0x7f19ec156500 2024-03-06T07:27:49.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.315+0000 7f19f1777640 1 -- 172.21.15.1:0/3630593165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19ec156500 msgr2=0x7f19ec0c2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.315+0000 7f19f1777640 1 --2- 172.21.15.1:0/3630593165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19ec156500 0x7f19ec0c2350 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009870 tx=0x7f19e002b980 comp rx=0 tx=0).stop 2024-03-06T07:27:49.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.315+0000 7f19f1777640 1 -- 172.21.15.1:0/3630593165 shutdown_connections 2024-03-06T07:27:49.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.315+0000 7f19f1777640 1 --2- 172.21.15.1:0/3630593165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19ec156500 0x7f19ec0c2350 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.315+0000 7f19f1777640 1 --2- 172.21.15.1:0/3630593165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19ec14cdc0 0x7f19ec152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.315+0000 7f19f1777640 1 -- 172.21.15.1:0/3630593165 >> 172.21.15.1:0/3630593165 conn(0x7f19ec0a9700 msgr2=0x7f19ec0a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.315+0000 7f19f1777640 1 -- 172.21.15.1:0/3630593165 shutdown_connections 2024-03-06T07:27:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.315+0000 7f19f1777640 1 -- 172.21.15.1:0/3630593165 wait complete. 2024-03-06T07:27:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.317+0000 7f19f1777640 1 Processor -- start 2024-03-06T07:27:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.317+0000 7f19f1777640 1 -- start start 2024-03-06T07:27:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.317+0000 7f19f1777640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19ec0c2890 0x7f19ec0ab940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.317+0000 7f19f1777640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19ec14cdc0 0x7f19ec1dfc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.317+0000 7f19f1777640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19ec156500 0x7f19ec1ef0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.317+0000 7f19f1777640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0c65e0 con 0x7f19ec0c2890 2024-03-06T07:27:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.317+0000 7f19f1777640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0c5a70 con 0x7f19ec156500 2024-03-06T07:27:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.317+0000 7f19f1777640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec14d1a0 con 0x7f19ec14cdc0 2024-03-06T07:27:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.317+0000 7f19eaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19ec0c2890 0x7f19ec0ab940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.317+0000 7f19eaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19ec0c2890 0x7f19ec0ab940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58714/0 (socket says 172.21.15.1:58714) 2024-03-06T07:27:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.317+0000 7f19eaffd640 1 -- 172.21.15.1:0/2382043152 learned_addr learned my addr 172.21.15.1:0/2382043152 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.318+0000 7f19ea7fc640 1 --2- 172.21.15.1:0/2382043152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19ec14cdc0 0x7f19ec1dfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.318+0000 7f19eb7fe640 1 --2- 172.21.15.1:0/2382043152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19ec156500 0x7f19ec1ef0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.318+0000 7f19eaffd640 1 -- 172.21.15.1:0/2382043152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19ec14cdc0 msgr2=0x7f19ec1dfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.318+0000 7f19eaffd640 1 --2- 172.21.15.1:0/2382043152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19ec14cdc0 0x7f19ec1dfc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.318+0000 7f19eaffd640 1 -- 172.21.15.1:0/2382043152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19ec156500 msgr2=0x7f19ec1ef0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.318+0000 7f19eaffd640 1 --2- 172.21.15.1:0/2382043152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19ec156500 0x7f19ec1ef0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.318+0000 7f19eaffd640 1 -- 172.21.15.1:0/2382043152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec06f590 con 0x7f19ec0c2890 2024-03-06T07:27:49.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.318+0000 7f19ea7fc640 1 --2- 172.21.15.1:0/2382043152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19ec14cdc0 0x7f19ec1dfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:49.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.318+0000 7f19eb7fe640 1 --2- 172.21.15.1:0/2382043152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19ec156500 0x7f19ec1ef0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:49.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.318+0000 7f19eaffd640 1 --2- 172.21.15.1:0/2382043152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19ec0c2890 0x7f19ec0ab940 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f19dc00e810 tx=0x7f19dc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.318+0000 7f19d37fe640 1 -- 172.21.15.1:0/2382043152 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc00f040 con 0x7f19ec0c2890 2024-03-06T07:27:49.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.318+0000 7f19d37fe640 1 -- 172.21.15.1:0/2382043152 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19dc004030 con 0x7f19ec0c2890 2024-03-06T07:27:49.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.318+0000 7f19f1777640 1 -- 172.21.15.1:0/2382043152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ec06f870 con 0x7f19ec0c2890 2024-03-06T07:27:49.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.318+0000 7f19d37fe640 1 -- 172.21.15.1:0/2382043152 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc004400 con 0x7f19ec0c2890 2024-03-06T07:27:49.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.318+0000 7f19f1777640 1 -- 172.21.15.1:0/2382043152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ec1eb680 con 0x7f19ec0c2890 2024-03-06T07:27:49.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.319+0000 7f19f1777640 1 -- 172.21.15.1:0/2382043152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19ec067fd0 con 0x7f19ec0c2890 2024-03-06T07:27:49.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.319+0000 7f19d37fe640 1 -- 172.21.15.1:0/2382043152 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19dc00c6d0 con 0x7f19ec0c2890 2024-03-06T07:27:49.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.320+0000 7f19d37fe640 1 --2- 172.21.15.1:0/2382043152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19d806b4d0 0x7f19d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.320+0000 7f19d37fe640 1 -- 172.21.15.1:0/2382043152 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f19dc089d90 con 0x7f19ec0c2890 2024-03-06T07:27:49.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.320+0000 7f19ea7fc640 1 --2- 172.21.15.1:0/2382043152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19d806b4d0 0x7f19d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.320+0000 7f19ea7fc640 1 --2- 172.21.15.1:0/2382043152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19d806b4d0 0x7f19d806d990 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f19e0036000 tx=0x7f19e003b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.324+0000 7f19d37fe640 1 -- 172.21.15.1:0/2382043152 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19dc017020 con 0x7f19ec0c2890 2024-03-06T07:27:49.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.506+0000 7f19f1777640 1 -- 172.21.15.1:0/2382043152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19ec153e00 con 0x7f19ec0c2890 2024-03-06T07:27:49.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.507+0000 7f19d37fe640 1 -- 172.21.15.1:0/2382043152 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+13054 (secure 0 0 0) 0x7f19dc058e60 con 0x7f19ec0c2890 2024-03-06T07:27:49.509 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:49.509 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":179,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:27:48.801997+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:49.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.508+0000 7f19f1777640 1 -- 172.21.15.1:0/2382043152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19d806b4d0 msgr2=0x7f19d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.508+0000 7f19f1777640 1 --2- 172.21.15.1:0/2382043152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19d806b4d0 0x7f19d806d990 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f19e0036000 tx=0x7f19e003b040 comp rx=0 tx=0).stop 2024-03-06T07:27:49.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.508+0000 7f19f1777640 1 -- 172.21.15.1:0/2382043152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19ec0c2890 msgr2=0x7f19ec0ab940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.508+0000 7f19f1777640 1 --2- 172.21.15.1:0/2382043152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19ec0c2890 0x7f19ec0ab940 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f19dc00e810 tx=0x7f19dc00ecd0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.508+0000 7f19f1777640 1 -- 172.21.15.1:0/2382043152 shutdown_connections 2024-03-06T07:27:49.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.508+0000 7f19f1777640 1 --2- 172.21.15.1:0/2382043152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19d806b4d0 0x7f19d806d990 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.508+0000 7f19f1777640 1 --2- 172.21.15.1:0/2382043152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19ec156500 0x7f19ec1ef0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.508+0000 7f19f1777640 1 --2- 172.21.15.1:0/2382043152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19ec14cdc0 0x7f19ec1dfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.508+0000 7f19f1777640 1 --2- 172.21.15.1:0/2382043152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19ec0c2890 0x7f19ec0ab940 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.508+0000 7f19f1777640 1 -- 172.21.15.1:0/2382043152 >> 172.21.15.1:0/2382043152 conn(0x7f19ec0a9700 msgr2=0x7f19ec152410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.509+0000 7f19f1777640 1 -- 172.21.15.1:0/2382043152 shutdown_connections 2024-03-06T07:27:49.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.509+0000 7f19f1777640 1 -- 172.21.15.1:0/2382043152 wait complete. 2024-03-06T07:27:49.522 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:49.522 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:49.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.625+0000 7fad5afa0640 1 Processor -- start 2024-03-06T07:27:49.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.625+0000 7fad5afa0640 1 -- start start 2024-03-06T07:27:49.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.626+0000 7fad5afa0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad54152840 0x7fad540ca890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.626+0000 7fad5afa0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fad540c6ee0 0x7fad540c5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.626+0000 7fad5afa0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad540cadd0 0x7fad540c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.626+0000 7fad5afa0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad540c1b20 con 0x7fad540cadd0 2024-03-06T07:27:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.626+0000 7fad5afa0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad54155e90 con 0x7fad540c6ee0 2024-03-06T07:27:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.626+0000 7fad5afa0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad540c14d0 con 0x7fad54152840 2024-03-06T07:27:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.626+0000 7fad5a79f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad540cadd0 0x7fad540c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.626+0000 7fad5a79f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad540cadd0 0x7fad540c59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58724/0 (socket says 172.21.15.1:58724) 2024-03-06T07:27:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.626+0000 7fad5a79f640 1 -- 172.21.15.1:0/1139502697 learned_addr learned my addr 172.21.15.1:0/1139502697 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.626+0000 7fad59f9e640 1 --2- 172.21.15.1:0/1139502697 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad54152840 0x7fad540ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.626+0000 7fad5979d640 1 --2- 172.21.15.1:0/1139502697 >> v2:172.21.15.98:3300/0 conn(0x7fad540c6ee0 0x7fad540c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.626+0000 7fad5a79f640 1 -- 172.21.15.1:0/1139502697 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad54152840 msgr2=0x7fad540ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.626+0000 7fad5a79f640 1 --2- 172.21.15.1:0/1139502697 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad54152840 0x7fad540ca890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.626+0000 7fad5a79f640 1 -- 172.21.15.1:0/1139502697 >> v2:172.21.15.98:3300/0 conn(0x7fad540c6ee0 msgr2=0x7fad540c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.626+0000 7fad5a79f640 1 --2- 172.21.15.1:0/1139502697 >> v2:172.21.15.98:3300/0 conn(0x7fad540c6ee0 0x7fad540c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.626+0000 7fad5a79f640 1 -- 172.21.15.1:0/1139502697 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad540c17b0 con 0x7fad540cadd0 2024-03-06T07:27:49.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.627+0000 7fad5a79f640 1 --2- 172.21.15.1:0/1139502697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad540cadd0 0x7fad540c59d0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fad540c7340 tx=0x7fad4c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e19f9031df7ace server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.627+0000 7fad58f9c640 1 -- 172.21.15.1:0/1139502697 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c02c9e0 con 0x7fad540cadd0 2024-03-06T07:27:49.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.627+0000 7fad58f9c640 1 -- 172.21.15.1:0/1139502697 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4c02cb80 con 0x7fad540cadd0 2024-03-06T07:27:49.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.627+0000 7fad58f9c640 1 -- 172.21.15.1:0/1139502697 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c0343f0 con 0x7fad540cadd0 2024-03-06T07:27:49.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.628+0000 7fad5afa0640 1 -- 172.21.15.1:0/1139502697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad540cadd0 msgr2=0x7fad540c59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.628+0000 7fad5afa0640 1 --2- 172.21.15.1:0/1139502697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad540cadd0 0x7fad540c59d0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fad540c7340 tx=0x7fad4c02be40 comp rx=0 tx=0).stop 2024-03-06T07:27:49.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.628+0000 7fad5afa0640 1 -- 172.21.15.1:0/1139502697 shutdown_connections 2024-03-06T07:27:49.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.628+0000 7fad5afa0640 1 --2- 172.21.15.1:0/1139502697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad540cadd0 0x7fad540c59d0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.628+0000 7fad5afa0640 1 --2- 172.21.15.1:0/1139502697 >> v2:172.21.15.98:3300/0 conn(0x7fad540c6ee0 0x7fad540c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.628+0000 7fad5afa0640 1 --2- 172.21.15.1:0/1139502697 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad54152840 0x7fad540ca890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.628+0000 7fad5afa0640 1 -- 172.21.15.1:0/1139502697 >> 172.21.15.1:0/1139502697 conn(0x7fad54067520 msgr2=0x7fad5406d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.628+0000 7fad5afa0640 1 -- 172.21.15.1:0/1139502697 shutdown_connections 2024-03-06T07:27:49.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.628+0000 7fad5afa0640 1 -- 172.21.15.1:0/1139502697 wait complete. 2024-03-06T07:27:49.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.629+0000 7fad5afa0640 1 Processor -- start 2024-03-06T07:27:49.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.629+0000 7fad5afa0640 1 -- start start 2024-03-06T07:27:49.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5afa0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad540c6ee0 0x7fad540aa680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5afa0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad540cadd0 0x7fad541566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5afa0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad54152840 0x7fad540c9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5afa0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad540c6090 con 0x7fad54152840 2024-03-06T07:27:49.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5afa0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad540c5f10 con 0x7fad540c6ee0 2024-03-06T07:27:49.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5afa0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad540bf900 con 0x7fad540cadd0 2024-03-06T07:27:49.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5a79f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad54152840 0x7fad540c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5a79f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad54152840 0x7fad540c9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58740/0 (socket says 172.21.15.1:58740) 2024-03-06T07:27:49.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5a79f640 1 -- 172.21.15.1:0/2667128224 learned_addr learned my addr 172.21.15.1:0/2667128224 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:49.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad59f9e640 1 --2- 172.21.15.1:0/2667128224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad540c6ee0 0x7fad540aa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5979d640 1 --2- 172.21.15.1:0/2667128224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad540cadd0 0x7fad541566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5a79f640 1 -- 172.21.15.1:0/2667128224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad540cadd0 msgr2=0x7fad541566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5a79f640 1 --2- 172.21.15.1:0/2667128224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad540cadd0 0x7fad541566c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5a79f640 1 -- 172.21.15.1:0/2667128224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad540c6ee0 msgr2=0x7fad540aa680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5a79f640 1 --2- 172.21.15.1:0/2667128224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad540c6ee0 0x7fad540aa680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5a79f640 1 -- 172.21.15.1:0/2667128224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad540c8950 con 0x7fad54152840 2024-03-06T07:27:49.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5979d640 1 --2- 172.21.15.1:0/2667128224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad540cadd0 0x7fad541566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5a79f640 1 --2- 172.21.15.1:0/2667128224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad54152840 0x7fad540c9710 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fad4c02c450 tx=0x7fad4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad4a7fc640 1 -- 172.21.15.1:0/2667128224 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c043020 con 0x7fad54152840 2024-03-06T07:27:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad4a7fc640 1 -- 172.21.15.1:0/2667128224 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4c03e450 con 0x7fad54152840 2024-03-06T07:27:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad5afa0640 1 -- 172.21.15.1:0/2667128224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad540c8c30 con 0x7fad54152840 2024-03-06T07:27:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.630+0000 7fad4a7fc640 1 -- 172.21.15.1:0/2667128224 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c009510 con 0x7fad54152840 2024-03-06T07:27:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.631+0000 7fad5afa0640 1 -- 172.21.15.1:0/2667128224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad540c91f0 con 0x7fad54152840 2024-03-06T07:27:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.631+0000 7fad5afa0640 1 -- 172.21.15.1:0/2667128224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad540c86e0 con 0x7fad54152840 2024-03-06T07:27:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.632+0000 7fad4a7fc640 1 -- 172.21.15.1:0/2667128224 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad4c046020 con 0x7fad54152840 2024-03-06T07:27:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.632+0000 7fad4a7fc640 1 --2- 172.21.15.1:0/2667128224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad3806b4d0 0x7fad3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.632+0000 7fad4a7fc640 1 -- 172.21.15.1:0/2667128224 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fad4c0ac930 con 0x7fad54152840 2024-03-06T07:27:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.632+0000 7fad59f9e640 1 --2- 172.21.15.1:0/2667128224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad3806b4d0 0x7fad3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.633+0000 7fad59f9e640 1 --2- 172.21.15.1:0/2667128224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad3806b4d0 0x7fad3806d990 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fad50009aa0 tx=0x7fad50009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.636+0000 7fad4a7fc640 1 -- 172.21.15.1:0/2667128224 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad4c04d020 con 0x7fad54152840 2024-03-06T07:27:49.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.816+0000 7fad5afa0640 1 -- 172.21.15.1:0/2667128224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fad540c93f0 con 0x7fad54152840 2024-03-06T07:27:49.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.817+0000 7fad4a7fc640 1 -- 172.21.15.1:0/2667128224 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v179) v1 ==== 93+0+10 (secure 0 0 0) 0x7fad540c93f0 con 0x7fad54152840 2024-03-06T07:27:49.819 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:27:49.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.819+0000 7fad5afa0640 1 -- 172.21.15.1:0/2667128224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad3806b4d0 msgr2=0x7fad3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.819+0000 7fad5afa0640 1 --2- 172.21.15.1:0/2667128224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad3806b4d0 0x7fad3806d990 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fad50009aa0 tx=0x7fad50009190 comp rx=0 tx=0).stop 2024-03-06T07:27:49.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.819+0000 7fad5afa0640 1 -- 172.21.15.1:0/2667128224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad54152840 msgr2=0x7fad540c9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.819+0000 7fad5afa0640 1 --2- 172.21.15.1:0/2667128224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad54152840 0x7fad540c9710 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fad4c02c450 tx=0x7fad4c004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:49.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.819+0000 7fad5afa0640 1 -- 172.21.15.1:0/2667128224 shutdown_connections 2024-03-06T07:27:49.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.819+0000 7fad5afa0640 1 --2- 172.21.15.1:0/2667128224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fad3806b4d0 0x7fad3806d990 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.819+0000 7fad5afa0640 1 --2- 172.21.15.1:0/2667128224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad54152840 0x7fad540c9710 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.819+0000 7fad5afa0640 1 --2- 172.21.15.1:0/2667128224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad540cadd0 0x7fad541566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.819+0000 7fad5afa0640 1 --2- 172.21.15.1:0/2667128224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad540c6ee0 0x7fad540aa680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.819+0000 7fad5afa0640 1 -- 172.21.15.1:0/2667128224 >> 172.21.15.1:0/2667128224 conn(0x7fad54067520 msgr2=0x7fad540c3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.819+0000 7fad5afa0640 1 -- 172.21.15.1:0/2667128224 shutdown_connections 2024-03-06T07:27:49.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.819+0000 7fad5afa0640 1 -- 172.21.15.1:0/2667128224 wait complete. 2024-03-06T07:27:49.831 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:27:49.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.931+0000 7f794c5bc640 1 Processor -- start 2024-03-06T07:27:49.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.931+0000 7f794c5bc640 1 -- start start 2024-03-06T07:27:49.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.932+0000 7f794c5bc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f79440c6730 0x7f79440c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.932+0000 7f794c5bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79440c5d10 0x7f7944156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.932+0000 7f794c5bc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79440c3300 con 0x7f7944152690 2024-03-06T07:27:49.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.932+0000 7f794c5bc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79440c35a0 con 0x7f79440c6730 2024-03-06T07:27:49.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.932+0000 7f794c5bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79440c3880 con 0x7f79440c5d10 2024-03-06T07:27:49.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.932+0000 7f794ab59640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79440c5d10 0x7f7944156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.932+0000 7f7949b57640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f79440c6730 0x7f79440c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.932+0000 7f794ab59640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79440c5d10 0x7f7944156cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:50376/0 (socket says 172.21.15.1:50376) 2024-03-06T07:27:49.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.932+0000 7f794ab59640 1 -- 172.21.15.1:0/2809912901 learned_addr learned my addr 172.21.15.1:0/2809912901 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:49.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.932+0000 7f794ab59640 1 -- 172.21.15.1:0/2809912901 >> v2:172.21.15.98:3300/0 conn(0x7f79440c6730 msgr2=0x7f79440c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:49.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.932+0000 7f794ab59640 1 --2- 172.21.15.1:0/2809912901 >> v2:172.21.15.98:3300/0 conn(0x7f79440c6730 0x7f79440c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.932+0000 7f794ab59640 1 -- 172.21.15.1:0/2809912901 >> v1:172.21.15.98:6789/0 conn(0x7f7944152690 legacy=0x7f7944155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.932+0000 7f794ab59640 1 -- 172.21.15.1:0/2809912901 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79440c3b60 con 0x7f79440c5d10 2024-03-06T07:27:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.933+0000 7f794ab59640 1 --2- 172.21.15.1:0/2809912901 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79440c5d10 0x7f7944156cd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f793c009e50 tx=0x7f793c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bf035a43ee36fd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.933+0000 7f7949356640 1 -- 172.21.15.1:0/2809912901 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c02ca70 con 0x7f79440c5d10 2024-03-06T07:27:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.933+0000 7f7949356640 1 -- 172.21.15.1:0/2809912901 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793c02cc10 con 0x7f79440c5d10 2024-03-06T07:27:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.933+0000 7f7949356640 1 -- 172.21.15.1:0/2809912901 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c0343f0 con 0x7f79440c5d10 2024-03-06T07:27:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.934+0000 7f794c5bc640 1 -- 172.21.15.1:0/2809912901 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79440c5d10 msgr2=0x7f7944156cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.934+0000 7f794c5bc640 1 --2- 172.21.15.1:0/2809912901 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79440c5d10 0x7f7944156cd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f793c009e50 tx=0x7f793c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.934+0000 7f794c5bc640 1 -- 172.21.15.1:0/2809912901 shutdown_connections 2024-03-06T07:27:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.934+0000 7f794c5bc640 1 --2- 172.21.15.1:0/2809912901 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79440c5d10 0x7f7944156cd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.934+0000 7f794c5bc640 1 --2- 172.21.15.1:0/2809912901 >> v2:172.21.15.98:3300/0 conn(0x7f79440c6730 0x7f79440c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.934+0000 7f794c5bc640 1 -- 172.21.15.1:0/2809912901 >> 172.21.15.1:0/2809912901 conn(0x7f794406b410 msgr2=0x7f794406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.934+0000 7f794c5bc640 1 -- 172.21.15.1:0/2809912901 shutdown_connections 2024-03-06T07:27:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.934+0000 7f794c5bc640 1 -- 172.21.15.1:0/2809912901 wait complete. 2024-03-06T07:27:49.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.935+0000 7f794c5bc640 1 Processor -- start 2024-03-06T07:27:49.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.935+0000 7f794c5bc640 1 -- start start 2024-03-06T07:27:49.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f794c5bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79440c5d10 0x7f79440710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f794c5bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79440c6730 0x7f79440a8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f794c5bc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7944152690 0x7f7944150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f794c5bc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7944152bf0 con 0x7f7944152690 2024-03-06T07:27:49.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f794c5bc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7944152a70 con 0x7f79440c5d10 2024-03-06T07:27:49.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f794c5bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7944157210 con 0x7f79440c6730 2024-03-06T07:27:49.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f794ab59640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7944152690 0x7f7944150bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f794ab59640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7944152690 0x7f7944150bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58754/0 (socket says 172.21.15.1:58754) 2024-03-06T07:27:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f794ab59640 1 -- 172.21.15.1:0/2982586058 learned_addr learned my addr 172.21.15.1:0/2982586058 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f7949b57640 1 --2- 172.21.15.1:0/2982586058 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79440c6730 0x7f79440a8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f794a358640 1 --2- 172.21.15.1:0/2982586058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79440c5d10 0x7f79440710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f794ab59640 1 -- 172.21.15.1:0/2982586058 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79440c6730 msgr2=0x7f79440a8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f794ab59640 1 --2- 172.21.15.1:0/2982586058 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79440c6730 0x7f79440a8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f794ab59640 1 -- 172.21.15.1:0/2982586058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79440c5d10 msgr2=0x7f79440710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f794ab59640 1 --2- 172.21.15.1:0/2982586058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79440c5d10 0x7f79440710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f794ab59640 1 -- 172.21.15.1:0/2982586058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79441ee9d0 con 0x7f7944152690 2024-03-06T07:27:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f7949b57640 1 --2- 172.21.15.1:0/2982586058 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79440c6730 0x7f79440a8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.936+0000 7f794ab59640 1 --2- 172.21.15.1:0/2982586058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7944152690 0x7f7944150bc0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f793c036040 tx=0x7f793c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.937+0000 7f793affd640 1 -- 172.21.15.1:0/2982586058 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c043020 con 0x7f7944152690 2024-03-06T07:27:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.937+0000 7f793affd640 1 -- 172.21.15.1:0/2982586058 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793c03e450 con 0x7f7944152690 2024-03-06T07:27:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.937+0000 7f794c5bc640 1 -- 172.21.15.1:0/2982586058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79441eecb0 con 0x7f7944152690 2024-03-06T07:27:49.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.937+0000 7f793affd640 1 -- 172.21.15.1:0/2982586058 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c009510 con 0x7f7944152690 2024-03-06T07:27:49.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.937+0000 7f794c5bc640 1 -- 172.21.15.1:0/2982586058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7944099f80 con 0x7f7944152690 2024-03-06T07:27:49.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.938+0000 7f794c5bc640 1 -- 172.21.15.1:0/2982586058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7944067fd0 con 0x7f7944152690 2024-03-06T07:27:49.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.938+0000 7f793affd640 1 -- 172.21.15.1:0/2982586058 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f793c046020 con 0x7f7944152690 2024-03-06T07:27:49.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.938+0000 7f793affd640 1 --2- 172.21.15.1:0/2982586058 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f792806b4d0 0x7f792806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.939+0000 7f793affd640 1 -- 172.21.15.1:0/2982586058 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f793c0ac780 con 0x7f7944152690 2024-03-06T07:27:49.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.939+0000 7f794a358640 1 --2- 172.21.15.1:0/2982586058 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f792806b4d0 0x7f792806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.939+0000 7f794a358640 1 --2- 172.21.15.1:0/2982586058 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f792806b4d0 0x7f792806d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f7940001640 tx=0x7f7940009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:49.943+0000 7f793affd640 1 -- 172.21.15.1:0/2982586058 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f793c04d030 con 0x7f7944152690 2024-03-06T07:27:50.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.125+0000 7f794c5bc640 1 -- 172.21.15.1:0/2982586058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f79440c2c40 con 0x7f7944152690 2024-03-06T07:27:50.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.125+0000 7f793affd640 1 -- 172.21.15.1:0/2982586058 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v179) v1 ==== 84+0+65 (secure 0 0 0) 0x7f793c07b7c0 con 0x7f7944152690 2024-03-06T07:27:50.128 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:50.128 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-06T07:27:50.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:27:50.128 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-06T07:27:50.128 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-06T07:27:50.128 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:27:50.128 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-06T07:27:50.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.127+0000 7f794c5bc640 1 -- 172.21.15.1:0/2982586058 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f792806b4d0 msgr2=0x7f792806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.127+0000 7f794c5bc640 1 --2- 172.21.15.1:0/2982586058 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f792806b4d0 0x7f792806d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f7940001640 tx=0x7f7940009040 comp rx=0 tx=0).stop 2024-03-06T07:27:50.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.127+0000 7f794c5bc640 1 -- 172.21.15.1:0/2982586058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7944152690 msgr2=0x7f7944150bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.127+0000 7f794c5bc640 1 --2- 172.21.15.1:0/2982586058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7944152690 0x7f7944150bc0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f793c036040 tx=0x7f793c004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:50.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.127+0000 7f794c5bc640 1 -- 172.21.15.1:0/2982586058 shutdown_connections 2024-03-06T07:27:50.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.127+0000 7f794c5bc640 1 --2- 172.21.15.1:0/2982586058 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f792806b4d0 0x7f792806d990 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.127+0000 7f794c5bc640 1 --2- 172.21.15.1:0/2982586058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7944152690 0x7f7944150bc0 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.127+0000 7f794c5bc640 1 --2- 172.21.15.1:0/2982586058 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79440c6730 0x7f79440a8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.127+0000 7f794c5bc640 1 --2- 172.21.15.1:0/2982586058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79440c5d10 0x7f79440710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.127+0000 7f794c5bc640 1 -- 172.21.15.1:0/2982586058 >> 172.21.15.1:0/2982586058 conn(0x7f794406b410 msgr2=0x7f7944151e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:50.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.127+0000 7f794c5bc640 1 -- 172.21.15.1:0/2982586058 shutdown_connections 2024-03-06T07:27:50.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.127+0000 7f794c5bc640 1 -- 172.21.15.1:0/2982586058 wait complete. 2024-03-06T07:27:50.140 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:27:50.141 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:27:50.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.244+0000 7f2fb4bac640 1 Processor -- start 2024-03-06T07:27:50.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.244+0000 7f2fb4bac640 1 -- start start 2024-03-06T07:27:50.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.245+0000 7f2fb4bac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fb00c5dd0 0x7f2fb0153b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.245+0000 7f2fb4bac640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb006dab0 con 0x7f2fb00c4940 2024-03-06T07:27:50.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.245+0000 7f2fb4bac640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb00c2db0 con 0x7f2fb00c1e90 2024-03-06T07:27:50.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.245+0000 7f2fb4bac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb01540c0 con 0x7f2fb00c5dd0 2024-03-06T07:27:50.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.245+0000 7f2faed9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2fb00c4940 0x7f2fb00c4dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57856/0 (socket says 172.21.15.1:57856) 2024-03-06T07:27:50.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.245+0000 7f2faed9d640 1 -- 172.21.15.1:0/679937231 learned_addr learned my addr 172.21.15.1:0/679937231 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:50.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.245+0000 7f2fad59a640 1 -- 172.21.15.1:0/679937231 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228426033 0 0) 0x7f2fb006dab0 con 0x7f2fb00c4940 2024-03-06T07:27:50.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.245+0000 7f2fad59a640 1 -- 172.21.15.1:0/679937231 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f98003520 con 0x7f2fb00c4940 2024-03-06T07:27:50.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.245+0000 7f2fae59c640 1 --2- 172.21.15.1:0/679937231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fb00c5dd0 0x7f2fb0153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.245+0000 7f2fad59a640 1 -- 172.21.15.1:0/679937231 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 229666497 0 0) 0x7f2f98003520 con 0x7f2fb00c4940 2024-03-06T07:27:50.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fad59a640 1 -- 172.21.15.1:0/679937231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fb00c5dd0 msgr2=0x7f2fb0153b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:50.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fad59a640 1 --2- 172.21.15.1:0/679937231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fb00c5dd0 0x7f2fb0153b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fad59a640 1 -- 172.21.15.1:0/679937231 >> v1:172.21.15.98:6789/0 conn(0x7f2fb00c1e90 legacy=0x7f2fb00c1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fad59a640 1 -- 172.21.15.1:0/679937231 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb00c3090 con 0x7f2fb00c4940 2024-03-06T07:27:50.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fad59a640 1 -- 172.21.15.1:0/679937231 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f2fa00019f0 con 0x7f2fb00c4940 2024-03-06T07:27:50.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fad59a640 1 -- 172.21.15.1:0/679937231 >> v1:172.21.15.1:6789/0 conn(0x7f2fb00c4940 legacy=0x7f2fb00c4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fad59a640 1 --2- 172.21.15.1:0/679937231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f980039e0 0x7f2f98003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fad59a640 1 --2- 172.21.15.1:0/679937231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fb00c5dd0 0x7f2f98004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fad59a640 1 --2- 172.21.15.1:0/679937231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f98004d40 0x7f2f980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fad59a640 1 -- 172.21.15.1:0/679937231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb01540c0 con 0x7f2f980039e0 2024-03-06T07:27:50.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fad59a640 1 -- 172.21.15.1:0/679937231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9800baf0 con 0x7f2f98004d40 2024-03-06T07:27:50.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fad59a640 1 -- 172.21.15.1:0/679937231 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9800bcf0 con 0x7f2fb00c5dd0 2024-03-06T07:27:50.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fae59c640 1 --2- 172.21.15.1:0/679937231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f980039e0 0x7f2f98003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fadd9b640 1 --2- 172.21.15.1:0/679937231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fb00c5dd0 0x7f2f98004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2faed9d640 1 --2- 172.21.15.1:0/679937231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f98004d40 0x7f2f980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fae59c640 1 -- 172.21.15.1:0/679937231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fb00c5dd0 msgr2=0x7f2f98004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:50.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fae59c640 1 --2- 172.21.15.1:0/679937231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fb00c5dd0 0x7f2f98004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fae59c640 1 -- 172.21.15.1:0/679937231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f98004d40 msgr2=0x7f2f980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:50.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fae59c640 1 --2- 172.21.15.1:0/679937231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f98004d40 0x7f2f980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2fae59c640 1 -- 172.21.15.1:0/679937231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f98003520 con 0x7f2f980039e0 2024-03-06T07:27:50.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.246+0000 7f2faed9d640 1 --2- 172.21.15.1:0/679937231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f98004d40 0x7f2f980051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:50.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.247+0000 7f2fae59c640 1 --2- 172.21.15.1:0/679937231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f980039e0 0x7f2f98003e00 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009ac0 tx=0x7f2f98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c7b0273adddc62 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:50.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.247+0000 7f2fad59a640 1 -- 172.21.15.1:0/679937231 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8039070 con 0x7f2f980039e0 2024-03-06T07:27:50.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.247+0000 7f2fad59a640 1 -- 172.21.15.1:0/679937231 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa802c600 con 0x7f2f980039e0 2024-03-06T07:27:50.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.247+0000 7f2fad59a640 1 -- 172.21.15.1:0/679937231 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8009690 con 0x7f2f980039e0 2024-03-06T07:27:50.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.247+0000 7f2fb4bac640 1 -- 172.21.15.1:0/679937231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f980039e0 msgr2=0x7f2f98003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.247+0000 7f2fb4bac640 1 --2- 172.21.15.1:0/679937231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f980039e0 0x7f2f98003e00 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009ac0 tx=0x7f2f98005830 comp rx=0 tx=0).stop 2024-03-06T07:27:50.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.247+0000 7f2faed9d640 1 -- 172.21.15.1:0/679937231 reap_dead start 2024-03-06T07:27:50.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.248+0000 7f2fb4bac640 1 -- 172.21.15.1:0/679937231 shutdown_connections 2024-03-06T07:27:50.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.248+0000 7f2fb4bac640 1 -- 172.21.15.1:0/679937231 >> 172.21.15.1:0/679937231 conn(0x7f2fb00a8e10 msgr2=0x7f2fb00c3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:50.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.248+0000 7f2fb4bac640 1 -- 172.21.15.1:0/679937231 shutdown_connections 2024-03-06T07:27:50.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.248+0000 7f2fb4bac640 1 -- 172.21.15.1:0/679937231 wait complete. 2024-03-06T07:27:50.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.249+0000 7f2fb4bac640 1 Processor -- start 2024-03-06T07:27:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.249+0000 7f2fb4bac640 1 -- start start 2024-03-06T07:27:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fb4bac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f980039e0 0x7f2fb00bfd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fb4bac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fb00c02b0 0x7f2fb00aacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fb4bac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fb014a280 0x7f2fb00ab210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fb4bac640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb00c61b0 con 0x7f2fb00c02b0 2024-03-06T07:27:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fb4bac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb006e050 con 0x7f2f980039e0 2024-03-06T07:27:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fb4bac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb00c1a50 con 0x7f2fb014a280 2024-03-06T07:27:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fadd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fb00c02b0 0x7f2fb00aacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fadd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fb00c02b0 0x7f2fb00aacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58776/0 (socket says 172.21.15.1:58776) 2024-03-06T07:27:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fadd9b640 1 -- 172.21.15.1:0/4227368392 learned_addr learned my addr 172.21.15.1:0/4227368392 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fae59c640 1 --2- 172.21.15.1:0/4227368392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f980039e0 0x7f2fb00bfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2faed9d640 1 --2- 172.21.15.1:0/4227368392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fb014a280 0x7f2fb00ab210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fadd9b640 1 -- 172.21.15.1:0/4227368392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fb014a280 msgr2=0x7f2fb00ab210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fadd9b640 1 --2- 172.21.15.1:0/4227368392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fb014a280 0x7f2fb00ab210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fadd9b640 1 -- 172.21.15.1:0/4227368392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f980039e0 msgr2=0x7f2fb00bfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fadd9b640 1 --2- 172.21.15.1:0/4227368392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f980039e0 0x7f2fb00bfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fadd9b640 1 -- 172.21.15.1:0/4227368392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb00726d0 con 0x7f2fb00c02b0 2024-03-06T07:27:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fadd9b640 1 --2- 172.21.15.1:0/4227368392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fb00c02b0 0x7f2fb00aacd0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400f820 tx=0x7f2fa400fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2f9effd640 1 -- 172.21.15.1:0/4227368392 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa40050d0 con 0x7f2fb00c02b0 2024-03-06T07:27:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2f9effd640 1 -- 172.21.15.1:0/4227368392 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa4018070 con 0x7f2fb00c02b0 2024-03-06T07:27:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fb4bac640 1 -- 172.21.15.1:0/4227368392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb01eed10 con 0x7f2fb00c02b0 2024-03-06T07:27:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2f9effd640 1 -- 172.21.15.1:0/4227368392 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4005490 con 0x7f2fb00c02b0 2024-03-06T07:27:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.250+0000 7f2fb4bac640 1 -- 172.21.15.1:0/4227368392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb0072c50 con 0x7f2fb00c02b0 2024-03-06T07:27:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.251+0000 7f2fb4bac640 1 -- 172.21.15.1:0/4227368392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb00680b0 con 0x7f2fb00c02b0 2024-03-06T07:27:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.252+0000 7f2f9effd640 1 -- 172.21.15.1:0/4227368392 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fa400d6d0 con 0x7f2fb00c02b0 2024-03-06T07:27:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.252+0000 7f2f9effd640 1 --2- 172.21.15.1:0/4227368392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f9006b500 0x7f2f9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.252+0000 7f2f9effd640 1 -- 172.21.15.1:0/4227368392 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f2fa4089d30 con 0x7f2fb00c02b0 2024-03-06T07:27:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.252+0000 7f2fae59c640 1 --2- 172.21.15.1:0/4227368392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f9006b500 0x7f2f9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.253+0000 7f2fae59c640 1 --2- 172.21.15.1:0/4227368392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f9006b500 0x7f2f9006d9c0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8036000 tx=0x7f2fa8030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.256+0000 7f2f9effd640 1 -- 172.21.15.1:0/4227368392 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fa4058e00 con 0x7f2fb00c02b0 2024-03-06T07:27:50.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.437+0000 7f2fb4bac640 1 -- 172.21.15.1:0/4227368392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2fb00c1770 con 0x7f2fb00c02b0 2024-03-06T07:27:50.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.807+0000 7f2f9effd640 1 -- 172.21.15.1:0/4227368392 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v180) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2fa4058700 con 0x7f2fb00c02b0 2024-03-06T07:27:50.810 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-06T07:27:50.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.810+0000 7f2fb4bac640 1 -- 172.21.15.1:0/4227368392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f9006b500 msgr2=0x7f2f9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.810+0000 7f2fb4bac640 1 --2- 172.21.15.1:0/4227368392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f9006b500 0x7f2f9006d9c0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8036000 tx=0x7f2fa8030000 comp rx=0 tx=0).stop 2024-03-06T07:27:50.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.810+0000 7f2fb4bac640 1 -- 172.21.15.1:0/4227368392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fb00c02b0 msgr2=0x7f2fb00aacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.810+0000 7f2fb4bac640 1 --2- 172.21.15.1:0/4227368392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fb00c02b0 0x7f2fb00aacd0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400f820 tx=0x7f2fa400fce0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.810+0000 7f2fb4bac640 1 -- 172.21.15.1:0/4227368392 shutdown_connections 2024-03-06T07:27:50.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.810+0000 7f2fb4bac640 1 --2- 172.21.15.1:0/4227368392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2f9006b500 0x7f2f9006d9c0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.810+0000 7f2fb4bac640 1 --2- 172.21.15.1:0/4227368392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fb014a280 0x7f2fb00ab210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.810+0000 7f2fb4bac640 1 --2- 172.21.15.1:0/4227368392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fb00c02b0 0x7f2fb00aacd0 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.810+0000 7f2fb4bac640 1 --2- 172.21.15.1:0/4227368392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f980039e0 0x7f2fb00bfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.810+0000 7f2fb4bac640 1 -- 172.21.15.1:0/4227368392 >> 172.21.15.1:0/4227368392 conn(0x7f2fb00a8e10 msgr2=0x7f2fb0153050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:50.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.810+0000 7f2fb4bac640 1 -- 172.21.15.1:0/4227368392 shutdown_connections 2024-03-06T07:27:50.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.810+0000 7f2fb4bac640 1 -- 172.21.15.1:0/4227368392 wait complete. 2024-03-06T07:27:50.827 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:27:50.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.934+0000 7ff1215ae640 1 Processor -- start 2024-03-06T07:27:50.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.934+0000 7ff1215ae640 1 -- start start 2024-03-06T07:27:50.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.935+0000 7ff1215ae640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff11c0af6f0 0x7ff11c0c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.935+0000 7ff1215ae640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11c0c7180 con 0x7ff11c0c7930 2024-03-06T07:27:50.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.935+0000 7ff1215ae640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11c155460 con 0x7ff11c152820 2024-03-06T07:27:50.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.935+0000 7ff1215ae640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff11c0c6bc0 con 0x7ff11c0af6f0 2024-03-06T07:27:50.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.935+0000 7ff11b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff11c0c7930 0x7ff11c15fb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57870/0 (socket says 172.21.15.1:57870) 2024-03-06T07:27:50.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.935+0000 7ff11b7fe640 1 -- 172.21.15.1:0/2448619421 learned_addr learned my addr 172.21.15.1:0/2448619421 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:50.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.935+0000 7ff11a7fc640 1 --2- 172.21.15.1:0/2448619421 >> v2:172.21.15.98:3300/0 conn(0x7ff11c0af6f0 0x7ff11c0c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff119ffb640 1 -- 172.21.15.1:0/2448619421 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171530437 0 0) 0x7ff11c0c7180 con 0x7ff11c0c7930 2024-03-06T07:27:50.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff119ffb640 1 -- 172.21.15.1:0/2448619421 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff108003520 con 0x7ff11c0c7930 2024-03-06T07:27:50.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff119ffb640 1 -- 172.21.15.1:0/2448619421 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 403476385 0 0) 0x7ff108003520 con 0x7ff11c0c7930 2024-03-06T07:27:50.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff119ffb640 1 -- 172.21.15.1:0/2448619421 >> v2:172.21.15.98:3300/0 conn(0x7ff11c0af6f0 msgr2=0x7ff11c0c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:50.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff119ffb640 1 --2- 172.21.15.1:0/2448619421 >> v2:172.21.15.98:3300/0 conn(0x7ff11c0af6f0 0x7ff11c0c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff119ffb640 1 -- 172.21.15.1:0/2448619421 >> v1:172.21.15.98:6789/0 conn(0x7ff11c152820 legacy=0x7ff11c0c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff119ffb640 1 -- 172.21.15.1:0/2448619421 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11c0c6ea0 con 0x7ff11c0c7930 2024-03-06T07:27:50.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff119ffb640 1 -- 172.21.15.1:0/2448619421 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7ff10c0019f0 con 0x7ff11c0c7930 2024-03-06T07:27:50.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff11a7fc640 1 --2- 172.21.15.1:0/2448619421 >> v2:172.21.15.98:3300/0 conn(0x7ff11c0af6f0 0x7ff11c0c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:50.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff119ffb640 1 -- 172.21.15.1:0/2448619421 >> v1:172.21.15.1:6789/0 conn(0x7ff11c0c7930 legacy=0x7ff11c15fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff119ffb640 1 --2- 172.21.15.1:0/2448619421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1080039b0 0x7ff108003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff119ffb640 1 --2- 172.21.15.1:0/2448619421 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff108004800 0x7ff108004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff119ffb640 1 --2- 172.21.15.1:0/2448619421 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1080051c0 0x7ff10800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff119ffb640 1 -- 172.21.15.1:0/2448619421 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11c0c6bc0 con 0x7ff108004800 2024-03-06T07:27:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff119ffb640 1 -- 172.21.15.1:0/2448619421 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10800c060 con 0x7ff1080039b0 2024-03-06T07:27:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff119ffb640 1 -- 172.21.15.1:0/2448619421 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff10800c260 con 0x7ff1080051c0 2024-03-06T07:27:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff11a7fc640 1 --2- 172.21.15.1:0/2448619421 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff108004800 0x7ff108004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff11affd640 1 --2- 172.21.15.1:0/2448619421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1080039b0 0x7ff108003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff11b7fe640 1 --2- 172.21.15.1:0/2448619421 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1080051c0 0x7ff10800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff11a7fc640 1 -- 172.21.15.1:0/2448619421 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1080051c0 msgr2=0x7ff10800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff11a7fc640 1 --2- 172.21.15.1:0/2448619421 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1080051c0 0x7ff10800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff11a7fc640 1 -- 172.21.15.1:0/2448619421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1080039b0 msgr2=0x7ff108003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff11a7fc640 1 --2- 172.21.15.1:0/2448619421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1080039b0 0x7ff108003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff11a7fc640 1 -- 172.21.15.1:0/2448619421 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff108003520 con 0x7ff108004800 2024-03-06T07:27:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.936+0000 7ff11b7fe640 1 -- 172.21.15.1:0/2448619421 reap_dead start 2024-03-06T07:27:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.937+0000 7ff11a7fc640 1 --2- 172.21.15.1:0/2448619421 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff108004800 0x7ff108004c80 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7ff104009c90 tx=0x7ff108005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7eb96eab8572e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.937+0000 7ff119ffb640 1 -- 172.21.15.1:0/2448619421 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff104039070 con 0x7ff108004800 2024-03-06T07:27:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.937+0000 7ff119ffb640 1 -- 172.21.15.1:0/2448619421 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10402c8f0 con 0x7ff108004800 2024-03-06T07:27:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.937+0000 7ff119ffb640 1 -- 172.21.15.1:0/2448619421 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff104009860 con 0x7ff108004800 2024-03-06T07:27:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.938+0000 7ff1215ae640 1 -- 172.21.15.1:0/2448619421 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff108004800 msgr2=0x7ff108004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.938+0000 7ff1215ae640 1 --2- 172.21.15.1:0/2448619421 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff108004800 0x7ff108004c80 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7ff104009c90 tx=0x7ff108005830 comp rx=0 tx=0).stop 2024-03-06T07:27:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.938+0000 7ff1215ae640 1 -- 172.21.15.1:0/2448619421 shutdown_connections 2024-03-06T07:27:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.938+0000 7ff1215ae640 1 --2- 172.21.15.1:0/2448619421 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff108004800 0x7ff108004c80 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.938+0000 7ff1215ae640 1 -- 172.21.15.1:0/2448619421 >> 172.21.15.1:0/2448619421 conn(0x7ff11c072720 msgr2=0x7ff11c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.938+0000 7ff1215ae640 1 -- 172.21.15.1:0/2448619421 shutdown_connections 2024-03-06T07:27:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.938+0000 7ff1215ae640 1 -- 172.21.15.1:0/2448619421 wait complete. 2024-03-06T07:27:50.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.939+0000 7ff1215ae640 1 Processor -- start 2024-03-06T07:27:50.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.939+0000 7ff1215ae640 1 -- start start 2024-03-06T07:27:50.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff1215ae640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff108004800 0x7ff11c1543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff1215ae640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff11c1575f0 0x7ff11c14c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff1215ae640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff11c157a50 0x7ff11c14cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff1215ae640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11c152d80 con 0x7ff11c157a50 2024-03-06T07:27:50.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff1215ae640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11c152c00 con 0x7ff11c1575f0 2024-03-06T07:27:50.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff1215ae640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11c0afc40 con 0x7ff108004800 2024-03-06T07:27:50.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff11b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff11c157a50 0x7ff11c14cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff11b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff11c157a50 0x7ff11c14cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58804/0 (socket says 172.21.15.1:58804) 2024-03-06T07:27:50.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff11b7fe640 1 -- 172.21.15.1:0/2632948458 learned_addr learned my addr 172.21.15.1:0/2632948458 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:50.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff11affd640 1 --2- 172.21.15.1:0/2632948458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff108004800 0x7ff11c1543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff11a7fc640 1 --2- 172.21.15.1:0/2632948458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff11c1575f0 0x7ff11c14c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff11b7fe640 1 -- 172.21.15.1:0/2632948458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff108004800 msgr2=0x7ff11c1543f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff11b7fe640 1 --2- 172.21.15.1:0/2632948458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff108004800 0x7ff11c1543f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff11b7fe640 1 -- 172.21.15.1:0/2632948458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff11c1575f0 msgr2=0x7ff11c14c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff11b7fe640 1 --2- 172.21.15.1:0/2632948458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff11c1575f0 0x7ff11c14c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff11b7fe640 1 -- 172.21.15.1:0/2632948458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11c156520 con 0x7ff11c157a50 2024-03-06T07:27:50.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff11b7fe640 1 --2- 172.21.15.1:0/2632948458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff11c157a50 0x7ff11c14cb90 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0c5b10 tx=0x7ff10c007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:50.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff1137fe640 1 -- 172.21.15.1:0/2632948458 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c00aa60 con 0x7ff11c157a50 2024-03-06T07:27:50.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff1137fe640 1 -- 172.21.15.1:0/2632948458 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10c018070 con 0x7ff11c157a50 2024-03-06T07:27:50.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff1215ae640 1 -- 172.21.15.1:0/2632948458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff11c156800 con 0x7ff11c157a50 2024-03-06T07:27:50.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff1137fe640 1 -- 172.21.15.1:0/2632948458 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c00ae40 con 0x7ff11c157a50 2024-03-06T07:27:50.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.940+0000 7ff1215ae640 1 -- 172.21.15.1:0/2632948458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff11c153ed0 con 0x7ff11c157a50 2024-03-06T07:27:50.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.941+0000 7ff1215ae640 1 -- 172.21.15.1:0/2632948458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff11c1540d0 con 0x7ff11c157a50 2024-03-06T07:27:50.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.942+0000 7ff1137fe640 1 -- 172.21.15.1:0/2632948458 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff10c01d460 con 0x7ff11c157a50 2024-03-06T07:27:50.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.942+0000 7ff1137fe640 1 --2- 172.21.15.1:0/2632948458 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff0fc06b500 0x7ff0fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.943+0000 7ff1137fe640 1 -- 172.21.15.1:0/2632948458 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5619+0+0 (secure 0 0 0) 0x7ff10c08b080 con 0x7ff11c157a50 2024-03-06T07:27:50.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.943+0000 7ff11affd640 1 --2- 172.21.15.1:0/2632948458 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff0fc06b500 0x7ff0fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.943+0000 7ff11affd640 1 --2- 172.21.15.1:0/2632948458 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff0fc06b500 0x7ff0fc06d9c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7ff114000c00 tx=0x7ff114009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:50.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:50.946+0000 7ff1137fe640 1 -- 172.21.15.1:0/2632948458 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff11c1540d0 con 0x7ff11c157a50 2024-03-06T07:27:51.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.127+0000 7ff1215ae640 1 -- 172.21.15.1:0/2632948458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff11c0686e0 con 0x7ff11c157a50 2024-03-06T07:27:51.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.810+0000 7ff1137fe640 1 -- 172.21.15.1:0/2632948458 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v181) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff10c059fb0 con 0x7ff11c157a50 2024-03-06T07:27:51.812 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-06T07:27:51.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.813+0000 7ff1215ae640 1 -- 172.21.15.1:0/2632948458 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff0fc06b500 msgr2=0x7ff0fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.813+0000 7ff1215ae640 1 --2- 172.21.15.1:0/2632948458 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff0fc06b500 0x7ff0fc06d9c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7ff114000c00 tx=0x7ff114009040 comp rx=0 tx=0).stop 2024-03-06T07:27:51.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.813+0000 7ff1215ae640 1 -- 172.21.15.1:0/2632948458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff11c157a50 msgr2=0x7ff11c14cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.813+0000 7ff1215ae640 1 --2- 172.21.15.1:0/2632948458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff11c157a50 0x7ff11c14cb90 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0c5b10 tx=0x7ff10c007a90 comp rx=0 tx=0).stop 2024-03-06T07:27:51.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.813+0000 7ff1215ae640 1 -- 172.21.15.1:0/2632948458 shutdown_connections 2024-03-06T07:27:51.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.813+0000 7ff1215ae640 1 --2- 172.21.15.1:0/2632948458 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff0fc06b500 0x7ff0fc06d9c0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.813+0000 7ff1215ae640 1 --2- 172.21.15.1:0/2632948458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff11c157a50 0x7ff11c14cb90 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.813+0000 7ff1215ae640 1 --2- 172.21.15.1:0/2632948458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff11c1575f0 0x7ff11c14c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.813+0000 7ff1215ae640 1 --2- 172.21.15.1:0/2632948458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff108004800 0x7ff11c1543f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.813+0000 7ff1215ae640 1 -- 172.21.15.1:0/2632948458 >> 172.21.15.1:0/2632948458 conn(0x7ff11c072720 msgr2=0x7ff11c150070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:51.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.813+0000 7ff1215ae640 1 -- 172.21.15.1:0/2632948458 shutdown_connections 2024-03-06T07:27:51.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.813+0000 7ff1215ae640 1 -- 172.21.15.1:0/2632948458 wait complete. 2024-03-06T07:27:51.840 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:27:51.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.949+0000 7feab61b6640 1 Processor -- start 2024-03-06T07:27:51.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.949+0000 7feab61b6640 1 -- start start 2024-03-06T07:27:51.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.949+0000 7feab61b6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feab00c05f0 0x7feab00c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.949+0000 7feab61b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feab00c14c0 0x7feab01558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.949+0000 7feab61b6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab00c31f0 con 0x7feab006f640 2024-03-06T07:27:51.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.949+0000 7feab61b6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab006fe90 con 0x7feab00c05f0 2024-03-06T07:27:51.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.949+0000 7feab61b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab00c2fc0 con 0x7feab00c14c0 2024-03-06T07:27:51.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.950+0000 7feaaf7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feab00c05f0 0x7feab00c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.950+0000 7feaaf7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feab00c05f0 0x7feab00c09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58806/0 (socket says 172.21.15.1:58806) 2024-03-06T07:27:51.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.950+0000 7feaaeffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7feab006f640 0x7feab01551b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57876/0 (socket says 172.21.15.1:57876) 2024-03-06T07:27:51.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.950+0000 7feaaf7fe640 1 -- 172.21.15.1:0/484384169 learned_addr learned my addr 172.21.15.1:0/484384169 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:51.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.950+0000 7feaaffff640 1 --2- 172.21.15.1:0/484384169 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feab00c14c0 0x7feab01558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.950+0000 7feaaf7fe640 1 -- 172.21.15.1:0/484384169 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feab00c14c0 msgr2=0x7feab01558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:51.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.950+0000 7feaaf7fe640 1 --2- 172.21.15.1:0/484384169 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feab00c14c0 0x7feab01558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.950+0000 7feaaf7fe640 1 -- 172.21.15.1:0/484384169 >> v1:172.21.15.1:6789/0 conn(0x7feab006f640 legacy=0x7feab01551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.950+0000 7feaaf7fe640 1 -- 172.21.15.1:0/484384169 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab0155e00 con 0x7feab00c05f0 2024-03-06T07:27:51.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.950+0000 7feaae7fc640 1 -- 172.21.15.1:0/484384169 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194390090 0 0) 0x7feab00c31f0 con 0x7feab006f640 2024-03-06T07:27:51.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.951+0000 7feaaf7fe640 1 --2- 172.21.15.1:0/484384169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feab00c05f0 0x7feab00c09d0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7feaa0009870 tx=0x7feaa002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc9aae8bac0ae71f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.951+0000 7feaae7fc640 1 -- 172.21.15.1:0/484384169 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa0034070 con 0x7feab00c05f0 2024-03-06T07:27:51.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.951+0000 7feaae7fc640 1 -- 172.21.15.1:0/484384169 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa0030840 con 0x7feab00c05f0 2024-03-06T07:27:51.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.951+0000 7feaae7fc640 1 -- 172.21.15.1:0/484384169 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa0030c10 con 0x7feab00c05f0 2024-03-06T07:27:51.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.951+0000 7feab61b6640 1 -- 172.21.15.1:0/484384169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feab00c05f0 msgr2=0x7feab00c09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.951+0000 7feab61b6640 1 --2- 172.21.15.1:0/484384169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feab00c05f0 0x7feab00c09d0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7feaa0009870 tx=0x7feaa002b980 comp rx=0 tx=0).stop 2024-03-06T07:27:51.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.951+0000 7feab61b6640 1 -- 172.21.15.1:0/484384169 shutdown_connections 2024-03-06T07:27:51.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.951+0000 7feab61b6640 1 --2- 172.21.15.1:0/484384169 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feab00c14c0 0x7feab01558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.951+0000 7feab61b6640 1 --2- 172.21.15.1:0/484384169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feab00c05f0 0x7feab00c09d0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.951+0000 7feab61b6640 1 -- 172.21.15.1:0/484384169 >> 172.21.15.1:0/484384169 conn(0x7feab00c3cc0 msgr2=0x7feab00c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:51.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.951+0000 7feab61b6640 1 -- 172.21.15.1:0/484384169 shutdown_connections 2024-03-06T07:27:51.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.951+0000 7feab61b6640 1 -- 172.21.15.1:0/484384169 wait complete. 2024-03-06T07:27:51.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.952+0000 7feab61b6640 1 Processor -- start 2024-03-06T07:27:51.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.953+0000 7feab61b6640 1 -- start start 2024-03-06T07:27:51.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.953+0000 7feab61b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab006f640 0x7feab00aa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.953+0000 7feab61b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feab00c05f0 0x7feab00a8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.953+0000 7feab61b6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feab00c14c0 0x7feab01eb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.953+0000 7feab61b6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab00c3770 con 0x7feab00c14c0 2024-03-06T07:27:51.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.953+0000 7feab61b6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab00c3490 con 0x7feab006f640 2024-03-06T07:27:51.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.953+0000 7feab61b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab00c1040 con 0x7feab00c05f0 2024-03-06T07:27:51.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.953+0000 7feaaffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feab00c14c0 0x7feab01eb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.953+0000 7feaaffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feab00c14c0 0x7feab01eb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58814/0 (socket says 172.21.15.1:58814) 2024-03-06T07:27:51.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.953+0000 7feaaffff640 1 -- 172.21.15.1:0/1370835918 learned_addr learned my addr 172.21.15.1:0/1370835918 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:51.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.954+0000 7feaaeffd640 1 --2- 172.21.15.1:0/1370835918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feab00c05f0 0x7feab00a8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.954+0000 7feaaf7fe640 1 --2- 172.21.15.1:0/1370835918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab006f640 0x7feab00aa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.954+0000 7feaaffff640 1 -- 172.21.15.1:0/1370835918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feab00c05f0 msgr2=0x7feab00a8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.954+0000 7feaaffff640 1 --2- 172.21.15.1:0/1370835918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feab00c05f0 0x7feab00a8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.954+0000 7feaaffff640 1 -- 172.21.15.1:0/1370835918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab006f640 msgr2=0x7feab00aa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.954+0000 7feaaffff640 1 --2- 172.21.15.1:0/1370835918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab006f640 0x7feab00aa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.954+0000 7feaaffff640 1 -- 172.21.15.1:0/1370835918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab00abd10 con 0x7feab00c14c0 2024-03-06T07:27:51.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.954+0000 7feaaffff640 1 --2- 172.21.15.1:0/1370835918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feab00c14c0 0x7feab01eb9c0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7feaa8007e90 tx=0x7feaa800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.954+0000 7fea97fff640 1 -- 172.21.15.1:0/1370835918 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8014420 con 0x7feab00c14c0 2024-03-06T07:27:51.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.954+0000 7fea97fff640 1 -- 172.21.15.1:0/1370835918 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa8019070 con 0x7feab00c14c0 2024-03-06T07:27:51.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.954+0000 7feab61b6640 1 -- 172.21.15.1:0/1370835918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab01eb1c0 con 0x7feab00c14c0 2024-03-06T07:27:51.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.954+0000 7fea97fff640 1 -- 172.21.15.1:0/1370835918 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8014800 con 0x7feab00c14c0 2024-03-06T07:27:51.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.954+0000 7feab61b6640 1 -- 172.21.15.1:0/1370835918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab00a8c60 con 0x7feab00c14c0 2024-03-06T07:27:51.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.955+0000 7feab61b6640 1 -- 172.21.15.1:0/1370835918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab0067fd0 con 0x7feab00c14c0 2024-03-06T07:27:51.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.956+0000 7fea97fff640 1 -- 172.21.15.1:0/1370835918 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feaa80149a0 con 0x7feab00c14c0 2024-03-06T07:27:51.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.956+0000 7fea97fff640 1 --2- 172.21.15.1:0/1370835918 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fea9c06b4d0 0x7fea9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.956+0000 7fea97fff640 1 -- 172.21.15.1:0/1370835918 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6001+0+0 (secure 0 0 0) 0x7feaa808aaa0 con 0x7feab00c14c0 2024-03-06T07:27:51.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.956+0000 7feaaf7fe640 1 --2- 172.21.15.1:0/1370835918 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fea9c06b4d0 0x7fea9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.957+0000 7feaaf7fe640 1 --2- 172.21.15.1:0/1370835918 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fea9c06b4d0 0x7fea9c06d990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7feaa0036000 tx=0x7feaa003b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:51.961+0000 7fea97fff640 1 -- 172.21.15.1:0/1370835918 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feaa8059880 con 0x7feab00c14c0 2024-03-06T07:27:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.178+0000 7feab61b6640 1 -- 172.21.15.1:0/1370835918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7feab01529f0 con 0x7feab00c14c0 2024-03-06T07:27:52.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.831+0000 7fea97fff640 1 -- 172.21.15.1:0/1370835918 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 29 and data pool 30 v168) v1 ==== 174+0+0 (secure 0 0 0) 0x7feaa8059180 con 0x7feab00c14c0 2024-03-06T07:27:52.833 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 29 and data pool 30 2024-03-06T07:27:52.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.839+0000 7feab61b6640 1 -- 172.21.15.1:0/1370835918 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fea9c06b4d0 msgr2=0x7fea9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.840+0000 7feab61b6640 1 --2- 172.21.15.1:0/1370835918 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fea9c06b4d0 0x7fea9c06d990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7feaa0036000 tx=0x7feaa003b040 comp rx=0 tx=0).stop 2024-03-06T07:27:52.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.840+0000 7feab61b6640 1 -- 172.21.15.1:0/1370835918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feab00c14c0 msgr2=0x7feab01eb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.840+0000 7feab61b6640 1 --2- 172.21.15.1:0/1370835918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feab00c14c0 0x7feab01eb9c0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7feaa8007e90 tx=0x7feaa800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.842+0000 7feab61b6640 1 -- 172.21.15.1:0/1370835918 shutdown_connections 2024-03-06T07:27:52.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.842+0000 7feab61b6640 1 --2- 172.21.15.1:0/1370835918 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fea9c06b4d0 0x7fea9c06d990 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.842+0000 7feab61b6640 1 --2- 172.21.15.1:0/1370835918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feab00c14c0 0x7feab01eb9c0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.842+0000 7feab61b6640 1 --2- 172.21.15.1:0/1370835918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feab00c05f0 0x7feab00a8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.842+0000 7feab61b6640 1 --2- 172.21.15.1:0/1370835918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab006f640 0x7feab00aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.843+0000 7feab61b6640 1 -- 172.21.15.1:0/1370835918 >> 172.21.15.1:0/1370835918 conn(0x7feab00c3cc0 msgr2=0x7feab01551e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:52.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.844+0000 7feab61b6640 1 -- 172.21.15.1:0/1370835918 shutdown_connections 2024-03-06T07:27:52.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.847+0000 7feab61b6640 1 -- 172.21.15.1:0/1370835918 wait complete. 2024-03-06T07:27:52.867 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:52.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.970+0000 7f665a5ac640 1 Processor -- start 2024-03-06T07:27:52.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.970+0000 7f665a5ac640 1 -- start start 2024-03-06T07:27:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.971+0000 7f665a5ac640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6654152690 0x7f6654155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.971+0000 7f665a5ac640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f66540c6730 0x7f66540c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.971+0000 7f665a5ac640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66541561e0 con 0x7f66540c5d10 2024-03-06T07:27:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.971+0000 7f665a5ac640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66540c3300 con 0x7f6654152690 2024-03-06T07:27:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.971+0000 7f665a5ac640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66540c35a0 con 0x7f66540c6730 2024-03-06T07:27:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.971+0000 7f6653fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6654152690 0x7f6654155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.971+0000 7f6653fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6654152690 0x7f6654155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58818/0 (socket says 172.21.15.1:58818) 2024-03-06T07:27:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.971+0000 7f6653fff640 1 -- 172.21.15.1:0/4252059802 learned_addr learned my addr 172.21.15.1:0/4252059802 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.971+0000 7f66537fe640 1 --2- 172.21.15.1:0/4252059802 >> v2:172.21.15.98:3300/0 conn(0x7f66540c6730 0x7f66540c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.971+0000 7f6653fff640 1 -- 172.21.15.1:0/4252059802 >> v2:172.21.15.98:3300/0 conn(0x7f66540c6730 msgr2=0x7f66540c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.971+0000 7f6653fff640 1 --2- 172.21.15.1:0/4252059802 >> v2:172.21.15.98:3300/0 conn(0x7f66540c6730 0x7f66540c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.971+0000 7f6653fff640 1 -- 172.21.15.1:0/4252059802 >> v1:172.21.15.98:6789/0 conn(0x7f66540c5d10 legacy=0x7f6654156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.971+0000 7f6653fff640 1 -- 172.21.15.1:0/4252059802 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66540c3880 con 0x7f6654152690 2024-03-06T07:27:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.972+0000 7f6653fff640 1 --2- 172.21.15.1:0/4252059802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6654152690 0x7f6654155ca0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f66540c98d0 tx=0x7f663c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da605b4f6e9783f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.972+0000 7f6652ffd640 1 -- 172.21.15.1:0/4252059802 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c02e070 con 0x7f6654152690 2024-03-06T07:27:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.972+0000 7f6652ffd640 1 -- 172.21.15.1:0/4252059802 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663c037070 con 0x7f6654152690 2024-03-06T07:27:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.972+0000 7f6652ffd640 1 -- 172.21.15.1:0/4252059802 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c031a00 con 0x7f6654152690 2024-03-06T07:27:52.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.973+0000 7f665a5ac640 1 -- 172.21.15.1:0/4252059802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6654152690 msgr2=0x7f6654155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.973+0000 7f665a5ac640 1 --2- 172.21.15.1:0/4252059802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6654152690 0x7f6654155ca0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f66540c98d0 tx=0x7f663c02b880 comp rx=0 tx=0).stop 2024-03-06T07:27:52.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.973+0000 7f665a5ac640 1 -- 172.21.15.1:0/4252059802 shutdown_connections 2024-03-06T07:27:52.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.973+0000 7f665a5ac640 1 --2- 172.21.15.1:0/4252059802 >> v2:172.21.15.98:3300/0 conn(0x7f66540c6730 0x7f66540c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.973+0000 7f665a5ac640 1 --2- 172.21.15.1:0/4252059802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6654152690 0x7f6654155ca0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.973+0000 7f665a5ac640 1 -- 172.21.15.1:0/4252059802 >> 172.21.15.1:0/4252059802 conn(0x7f665406b410 msgr2=0x7f665406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:52.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.973+0000 7f665a5ac640 1 -- 172.21.15.1:0/4252059802 shutdown_connections 2024-03-06T07:27:52.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.973+0000 7f665a5ac640 1 -- 172.21.15.1:0/4252059802 wait complete. 2024-03-06T07:27:52.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.974+0000 7f665a5ac640 1 Processor -- start 2024-03-06T07:27:52.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.974+0000 7f665a5ac640 1 -- start start 2024-03-06T07:27:52.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f665a5ac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66540c5d10 0x7f665414d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f665a5ac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66540c6730 0x7f66541eb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f665a5ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6654152690 0x7f66541eb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f665a5ac640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6654152a70 con 0x7f66540c5d10 2024-03-06T07:27:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f665a5ac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66541567b0 con 0x7f66540c6730 2024-03-06T07:27:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f665a5ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6654152bf0 con 0x7f6654152690 2024-03-06T07:27:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f6653fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66540c5d10 0x7f665414d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f6653fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66540c5d10 0x7f665414d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58826/0 (socket says 172.21.15.1:58826) 2024-03-06T07:27:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f6653fff640 1 -- 172.21.15.1:0/748783221 learned_addr learned my addr 172.21.15.1:0/748783221 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f66537fe640 1 --2- 172.21.15.1:0/748783221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66540c6730 0x7f66541eb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f6658b49640 1 --2- 172.21.15.1:0/748783221 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6654152690 0x7f66541eb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f6653fff640 1 -- 172.21.15.1:0/748783221 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6654152690 msgr2=0x7f66541eb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f6653fff640 1 --2- 172.21.15.1:0/748783221 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6654152690 0x7f66541eb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f6653fff640 1 -- 172.21.15.1:0/748783221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66540c6730 msgr2=0x7f66541eb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f6653fff640 1 --2- 172.21.15.1:0/748783221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66540c6730 0x7f66541eb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f6653fff640 1 -- 172.21.15.1:0/748783221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6654153aa0 con 0x7f66540c5d10 2024-03-06T07:27:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f6658b49640 1 --2- 172.21.15.1:0/748783221 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6654152690 0x7f66541eb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.975+0000 7f6653fff640 1 --2- 172.21.15.1:0/748783221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66540c5d10 0x7f665414d200 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f663c02b8c0 tx=0x7f663c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.976+0000 7f6650ff9640 1 -- 172.21.15.1:0/748783221 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c032670 con 0x7f66540c5d10 2024-03-06T07:27:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.976+0000 7f6650ff9640 1 -- 172.21.15.1:0/748783221 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663c032830 con 0x7f66540c5d10 2024-03-06T07:27:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.976+0000 7f665a5ac640 1 -- 172.21.15.1:0/748783221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66541eacc0 con 0x7f66540c5d10 2024-03-06T07:27:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.976+0000 7f6650ff9640 1 -- 172.21.15.1:0/748783221 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c032c20 con 0x7f66540c5d10 2024-03-06T07:27:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.976+0000 7f665a5ac640 1 -- 172.21.15.1:0/748783221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6654153d10 con 0x7f66540c5d10 2024-03-06T07:27:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.977+0000 7f665a5ac640 1 -- 172.21.15.1:0/748783221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6654067fd0 con 0x7f66540c5d10 2024-03-06T07:27:52.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.977+0000 7f6650ff9640 1 -- 172.21.15.1:0/748783221 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f663c04c060 con 0x7f66540c5d10 2024-03-06T07:27:52.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.978+0000 7f6650ff9640 1 --2- 172.21.15.1:0/748783221 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f663406b4d0 0x7f663406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.978+0000 7f6650ff9640 1 -- 172.21.15.1:0/748783221 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f663c0abd60 con 0x7f66540c5d10 2024-03-06T07:27:52.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.978+0000 7f66537fe640 1 --2- 172.21.15.1:0/748783221 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f663406b4d0 0x7f663406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.978+0000 7f66537fe640 1 --2- 172.21.15.1:0/748783221 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f663406b4d0 0x7f663406d990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f66540a9770 tx=0x7f6644008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:52.982+0000 7f6650ff9640 1 -- 172.21.15.1:0/748783221 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f663c040070 con 0x7f66540c5d10 2024-03-06T07:27:53.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.166+0000 7f665a5ac640 1 -- 172.21.15.1:0/748783221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66540a9dc0 con 0x7f66540c5d10 2024-03-06T07:27:53.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.167+0000 7f6650ff9640 1 -- 172.21.15.1:0/748783221 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v182) v1 ==== 74+0+16777 (secure 0 0 0) 0x7f66541561e0 con 0x7f66540c5d10 2024-03-06T07:27:53.169 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:53.169 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":182,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:27:52.179530+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:50.456844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:51.147612+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.169+0000 7f665a5ac640 1 -- 172.21.15.1:0/748783221 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f663406b4d0 msgr2=0x7f663406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.169+0000 7f665a5ac640 1 --2- 172.21.15.1:0/748783221 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f663406b4d0 0x7f663406d990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f66540a9770 tx=0x7f6644008040 comp rx=0 tx=0).stop 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.169+0000 7f665a5ac640 1 -- 172.21.15.1:0/748783221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66540c5d10 msgr2=0x7f665414d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.169+0000 7f665a5ac640 1 --2- 172.21.15.1:0/748783221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66540c5d10 0x7f665414d200 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f663c02b8c0 tx=0x7f663c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.169+0000 7f665a5ac640 1 -- 172.21.15.1:0/748783221 shutdown_connections 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.169+0000 7f665a5ac640 1 --2- 172.21.15.1:0/748783221 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f663406b4d0 0x7f663406d990 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.169+0000 7f665a5ac640 1 --2- 172.21.15.1:0/748783221 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6654152690 0x7f66541eb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.169+0000 7f665a5ac640 1 --2- 172.21.15.1:0/748783221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66540c6730 0x7f66541eb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.169+0000 7f665a5ac640 1 --2- 172.21.15.1:0/748783221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66540c5d10 0x7f665414d200 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.169+0000 7f665a5ac640 1 -- 172.21.15.1:0/748783221 >> 172.21.15.1:0/748783221 conn(0x7f665406b410 msgr2=0x7f6654151340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:53.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.170+0000 7f665a5ac640 1 -- 172.21.15.1:0/748783221 shutdown_connections 2024-03-06T07:27:53.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.170+0000 7f665a5ac640 1 -- 172.21.15.1:0/748783221 wait complete. 2024-03-06T07:27:53.183 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.283+0000 7f94ed165640 1 Processor -- start 2024-03-06T07:27:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.283+0000 7f94ed165640 1 -- start start 2024-03-06T07:27:53.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.284+0000 7f94ed165640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94e80c3560 0x7f94e8152560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.284+0000 7f94ed165640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94e80a5fd0 0x7f94e8152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.284+0000 7f94ed165640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e81588c0 con 0x7f94e80c59e0 2024-03-06T07:27:53.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.284+0000 7f94ed165640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e8158ba0 con 0x7f94e80a5fd0 2024-03-06T07:27:53.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.284+0000 7f94ed165640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e8158e80 con 0x7f94e80c3560 2024-03-06T07:27:53.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.284+0000 7f94e759e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94e80a5fd0 0x7f94e8152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.284+0000 7f94dffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f94e80c59e0 0x7f94e81551b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57892/0 (socket says 172.21.15.1:57892) 2024-03-06T07:27:53.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.284+0000 7f94dffff640 1 -- 172.21.15.1:0/2867696521 learned_addr learned my addr 172.21.15.1:0/2867696521 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:53.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.284+0000 7f94e6d9d640 1 --2- 172.21.15.1:0/2867696521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94e80c3560 0x7f94e8152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.284+0000 7f94e759e640 1 -- 172.21.15.1:0/2867696521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94e80c3560 msgr2=0x7f94e8152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:53.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.284+0000 7f94e759e640 1 --2- 172.21.15.1:0/2867696521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94e80c3560 0x7f94e8152560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.284+0000 7f94e759e640 1 -- 172.21.15.1:0/2867696521 >> v1:172.21.15.1:6789/0 conn(0x7f94e80c59e0 legacy=0x7f94e81551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.284+0000 7f94e659c640 1 -- 172.21.15.1:0/2867696521 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223367449 0 0) 0x7f94e81588c0 con 0x7f94e80c59e0 2024-03-06T07:27:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.284+0000 7f94e759e640 1 -- 172.21.15.1:0/2867696521 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e8159160 con 0x7f94e80a5fd0 2024-03-06T07:27:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.285+0000 7f94e759e640 1 --2- 172.21.15.1:0/2867696521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94e80a5fd0 0x7f94e8152aa0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f94e0009d90 tx=0x7f94e002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f611612205be8a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.285+0000 7f94e659c640 1 -- 172.21.15.1:0/2867696521 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e0039070 con 0x7f94e80a5fd0 2024-03-06T07:27:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.285+0000 7f94e659c640 1 -- 172.21.15.1:0/2867696521 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e002cc50 con 0x7f94e80a5fd0 2024-03-06T07:27:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.285+0000 7f94e659c640 1 -- 172.21.15.1:0/2867696521 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e0034440 con 0x7f94e80a5fd0 2024-03-06T07:27:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.285+0000 7f94ed165640 1 -- 172.21.15.1:0/2867696521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94e80a5fd0 msgr2=0x7f94e8152aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.285+0000 7f94ed165640 1 --2- 172.21.15.1:0/2867696521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94e80a5fd0 0x7f94e8152aa0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f94e0009d90 tx=0x7f94e002bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.286+0000 7f94ed165640 1 -- 172.21.15.1:0/2867696521 shutdown_connections 2024-03-06T07:27:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.286+0000 7f94ed165640 1 --2- 172.21.15.1:0/2867696521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94e80a5fd0 0x7f94e8152aa0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.286+0000 7f94ed165640 1 --2- 172.21.15.1:0/2867696521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94e80c3560 0x7f94e8152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.286+0000 7f94ed165640 1 -- 172.21.15.1:0/2867696521 >> 172.21.15.1:0/2867696521 conn(0x7f94e80c6860 msgr2=0x7f94e80c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.286+0000 7f94ed165640 1 -- 172.21.15.1:0/2867696521 shutdown_connections 2024-03-06T07:27:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.286+0000 7f94ed165640 1 -- 172.21.15.1:0/2867696521 wait complete. 2024-03-06T07:27:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.287+0000 7f94ed165640 1 Processor -- start 2024-03-06T07:27:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.287+0000 7f94ed165640 1 -- start start 2024-03-06T07:27:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.287+0000 7f94ed165640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94e80a5fd0 0x7f94e81ea860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.287+0000 7f94ed165640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94e80c3560 0x7f94e8149be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.287+0000 7f94ed165640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e80c59e0 0x7f94e8153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94ed165640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e80c3ac0 con 0x7f94e80c3560 2024-03-06T07:27:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94ed165640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e80c3940 con 0x7f94e80c59e0 2024-03-06T07:27:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94ed165640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e81558c0 con 0x7f94e80a5fd0 2024-03-06T07:27:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94e80c3560 0x7f94e8149be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94e80c3560 0x7f94e8149be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58846/0 (socket says 172.21.15.1:58846) 2024-03-06T07:27:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94dffff640 1 -- 172.21.15.1:0/2658907226 learned_addr learned my addr 172.21.15.1:0/2658907226 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94e6d9d640 1 --2- 172.21.15.1:0/2658907226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94e80a5fd0 0x7f94e81ea860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94dffff640 1 -- 172.21.15.1:0/2658907226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94e80a5fd0 msgr2=0x7f94e81ea860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94dffff640 1 --2- 172.21.15.1:0/2658907226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94e80a5fd0 0x7f94e81ea860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94dffff640 1 -- 172.21.15.1:0/2658907226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e80c59e0 msgr2=0x7f94e8153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94dffff640 1 --2- 172.21.15.1:0/2658907226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e80c59e0 0x7f94e8153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94dffff640 1 -- 172.21.15.1:0/2658907226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e81eaeb0 con 0x7f94e80c3560 2024-03-06T07:27:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94dffff640 1 --2- 172.21.15.1:0/2658907226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94e80c3560 0x7f94e8149be0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f94d000c410 tx=0x7f94d000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94df7fe640 1 -- 172.21.15.1:0/2658907226 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d00074e0 con 0x7f94e80c3560 2024-03-06T07:27:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94df7fe640 1 -- 172.21.15.1:0/2658907226 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d00047e0 con 0x7f94e80c3560 2024-03-06T07:27:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94df7fe640 1 -- 172.21.15.1:0/2658907226 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d0004bd0 con 0x7f94e80c3560 2024-03-06T07:27:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94ed165640 1 -- 172.21.15.1:0/2658907226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e81eb190 con 0x7f94e80c3560 2024-03-06T07:27:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.288+0000 7f94ed165640 1 -- 172.21.15.1:0/2658907226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e81eb750 con 0x7f94e80c3560 2024-03-06T07:27:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.289+0000 7f94ed165640 1 -- 172.21.15.1:0/2658907226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e8067fd0 con 0x7f94e80c3560 2024-03-06T07:27:53.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.290+0000 7f94df7fe640 1 -- 172.21.15.1:0/2658907226 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94d0011070 con 0x7f94e80c3560 2024-03-06T07:27:53.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.290+0000 7f94df7fe640 1 --2- 172.21.15.1:0/2658907226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94c806b4d0 0x7f94c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.290+0000 7f94df7fe640 1 -- 172.21.15.1:0/2658907226 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f94d0089f60 con 0x7f94e80c3560 2024-03-06T07:27:53.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.291+0000 7f94e6d9d640 1 --2- 172.21.15.1:0/2658907226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94c806b4d0 0x7f94c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.291+0000 7f94e6d9d640 1 --2- 172.21.15.1:0/2658907226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94c806b4d0 0x7f94c806d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f94d8006fd0 tx=0x7f94d8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.295+0000 7f94df7fe640 1 -- 172.21.15.1:0/2658907226 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94d0058ce0 con 0x7f94e80c3560 2024-03-06T07:27:53.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.476+0000 7f94ed165640 1 -- 172.21.15.1:0/2658907226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94e80c1e90 con 0x7f94e80c3560 2024-03-06T07:27:53.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.477+0000 7f94df7fe640 1 -- 172.21.15.1:0/2658907226 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v182) v1 ==== 74+0+16777 (secure 0 0 0) 0x7f94d00585e0 con 0x7f94e80c3560 2024-03-06T07:27:53.479 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:53.479 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":182,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:27:52.179530+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:50.456844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:51.147612+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.479+0000 7f94ed165640 1 -- 172.21.15.1:0/2658907226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94c806b4d0 msgr2=0x7f94c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.479+0000 7f94ed165640 1 --2- 172.21.15.1:0/2658907226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94c806b4d0 0x7f94c806d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f94d8006fd0 tx=0x7f94d8008040 comp rx=0 tx=0).stop 2024-03-06T07:27:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.479+0000 7f94ed165640 1 -- 172.21.15.1:0/2658907226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94e80c3560 msgr2=0x7f94e8149be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.479+0000 7f94ed165640 1 --2- 172.21.15.1:0/2658907226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94e80c3560 0x7f94e8149be0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f94d000c410 tx=0x7f94d000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.479+0000 7f94ed165640 1 -- 172.21.15.1:0/2658907226 shutdown_connections 2024-03-06T07:27:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.479+0000 7f94ed165640 1 --2- 172.21.15.1:0/2658907226 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94c806b4d0 0x7f94c806d990 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.479+0000 7f94ed165640 1 --2- 172.21.15.1:0/2658907226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e80c59e0 0x7f94e8153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.479+0000 7f94ed165640 1 --2- 172.21.15.1:0/2658907226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94e80c3560 0x7f94e8149be0 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.479+0000 7f94ed165640 1 --2- 172.21.15.1:0/2658907226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94e80a5fd0 0x7f94e81ea860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.479+0000 7f94ed165640 1 -- 172.21.15.1:0/2658907226 >> 172.21.15.1:0/2658907226 conn(0x7f94e80c6860 msgr2=0x7f94e806eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:53.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.479+0000 7f94ed165640 1 -- 172.21.15.1:0/2658907226 shutdown_connections 2024-03-06T07:27:53.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.479+0000 7f94ed165640 1 -- 172.21.15.1:0/2658907226 wait complete. 2024-03-06T07:27:53.492 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:27:53.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.595+0000 7f1bd7ab5640 1 Processor -- start 2024-03-06T07:27:53.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.595+0000 7f1bd7ab5640 1 -- start start 2024-03-06T07:27:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.595+0000 7f1bd7ab5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1bd00c2b30 0x7f1bd00724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.595+0000 7f1bd7ab5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1bd0152b70 0x7f1bd0072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.595+0000 7f1bd7ab5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd0151dd0 con 0x7f1bd00a5fa0 2024-03-06T07:27:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.595+0000 7f1bd7ab5640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bd01526e0 con 0x7f1bd0152b70 2024-03-06T07:27:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.595+0000 7f1bd7ab5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bd0156190 con 0x7f1bd00c2b30 2024-03-06T07:27:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.596+0000 7f1bd5050640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1bd0152b70 0x7f1bd0072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.596+0000 7f1bd5050640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1bd0152b70 0x7f1bd0072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58852/0 (socket says 172.21.15.1:58852) 2024-03-06T07:27:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.596+0000 7f1bd5050640 1 -- 172.21.15.1:0/966253637 learned_addr learned my addr 172.21.15.1:0/966253637 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.596+0000 7f1bd5851640 1 --2- 172.21.15.1:0/966253637 >> v2:172.21.15.98:3300/0 conn(0x7f1bd00c2b30 0x7f1bd00724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.596+0000 7f1bd5050640 1 -- 172.21.15.1:0/966253637 >> v2:172.21.15.98:3300/0 conn(0x7f1bd00c2b30 msgr2=0x7f1bd00724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.596+0000 7f1bd5050640 1 --2- 172.21.15.1:0/966253637 >> v2:172.21.15.98:3300/0 conn(0x7f1bd00c2b30 0x7f1bd00724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.596+0000 7f1bd5050640 1 -- 172.21.15.1:0/966253637 >> v1:172.21.15.98:6789/0 conn(0x7f1bd00a5fa0 legacy=0x7f1bd00c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.596+0000 7f1bd5050640 1 -- 172.21.15.1:0/966253637 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd0152090 con 0x7f1bd0152b70 2024-03-06T07:27:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.597+0000 7f1bd5050640 1 --2- 172.21.15.1:0/966253637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bd0152b70 0x7f1bd0072a00 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f1bd00c6570 tx=0x7f1bc002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68cb36b647be64c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.597+0000 7f1bd484f640 1 -- 172.21.15.1:0/966253637 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0035020 con 0x7f1bd0152b70 2024-03-06T07:27:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.597+0000 7f1bd484f640 1 -- 172.21.15.1:0/966253637 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc0031870 con 0x7f1bd0152b70 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.597+0000 7f1bd484f640 1 -- 172.21.15.1:0/966253637 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0031c40 con 0x7f1bd0152b70 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.597+0000 7f1bd7ab5640 1 -- 172.21.15.1:0/966253637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bd0152b70 msgr2=0x7f1bd0072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.597+0000 7f1bd7ab5640 1 --2- 172.21.15.1:0/966253637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bd0152b70 0x7f1bd0072a00 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f1bd00c6570 tx=0x7f1bc002b910 comp rx=0 tx=0).stop 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.597+0000 7f1bd7ab5640 1 -- 172.21.15.1:0/966253637 shutdown_connections 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.597+0000 7f1bd7ab5640 1 --2- 172.21.15.1:0/966253637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bd0152b70 0x7f1bd0072a00 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.597+0000 7f1bd7ab5640 1 --2- 172.21.15.1:0/966253637 >> v2:172.21.15.98:3300/0 conn(0x7f1bd00c2b30 0x7f1bd00724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.597+0000 7f1bd7ab5640 1 -- 172.21.15.1:0/966253637 >> 172.21.15.1:0/966253637 conn(0x7f1bd00c6860 msgr2=0x7f1bd00c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.598+0000 7f1bd7ab5640 1 -- 172.21.15.1:0/966253637 shutdown_connections 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.598+0000 7f1bd7ab5640 1 -- 172.21.15.1:0/966253637 wait complete. 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.599+0000 7f1bd7ab5640 1 Processor -- start 2024-03-06T07:27:53.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.599+0000 7f1bd7ab5640 1 -- start start 2024-03-06T07:27:53.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.599+0000 7f1bd7ab5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bd00a5fa0 0x7f1bd01539d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.599+0000 7f1bd7ab5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bd00c2b30 0x7f1bd00bf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.599+0000 7f1bd7ab5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd0152b70 0x7f1bd00bff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.599+0000 7f1bd7ab5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd0155f00 con 0x7f1bd00c2b30 2024-03-06T07:27:53.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.599+0000 7f1bd7ab5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd0155d80 con 0x7f1bd0152b70 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.599+0000 7f1bd7ab5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd00c3a00 con 0x7f1bd00a5fa0 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.599+0000 7f1bd5050640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bd00c2b30 0x7f1bd00bf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.599+0000 7f1bd5050640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bd00c2b30 0x7f1bd00bf9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58868/0 (socket says 172.21.15.1:58868) 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.599+0000 7f1bd5050640 1 -- 172.21.15.1:0/507340212 learned_addr learned my addr 172.21.15.1:0/507340212 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.600+0000 7f1bd5851640 1 --2- 172.21.15.1:0/507340212 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bd00a5fa0 0x7f1bd01539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.600+0000 7f1bd6052640 1 --2- 172.21.15.1:0/507340212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd0152b70 0x7f1bd00bff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.600+0000 7f1bd5050640 1 -- 172.21.15.1:0/507340212 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bd00a5fa0 msgr2=0x7f1bd01539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.600+0000 7f1bd5050640 1 --2- 172.21.15.1:0/507340212 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bd00a5fa0 0x7f1bd01539d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.600+0000 7f1bd5050640 1 -- 172.21.15.1:0/507340212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd0152b70 msgr2=0x7f1bd00bff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.600+0000 7f1bd5050640 1 --2- 172.21.15.1:0/507340212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd0152b70 0x7f1bd00bff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.600+0000 7f1bd5050640 1 -- 172.21.15.1:0/507340212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd0147770 con 0x7f1bd00c2b30 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.600+0000 7f1bd6052640 1 --2- 172.21.15.1:0/507340212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd0152b70 0x7f1bd00bff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.600+0000 7f1bd5050640 1 --2- 172.21.15.1:0/507340212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bd00c2b30 0x7f1bd00bf9f0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0000c00 tx=0x7f1bc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.600+0000 7f1bc67fc640 1 -- 172.21.15.1:0/507340212 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0040070 con 0x7f1bd00c2b30 2024-03-06T07:27:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.600+0000 7f1bc67fc640 1 -- 172.21.15.1:0/507340212 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc003a420 con 0x7f1bd00c2b30 2024-03-06T07:27:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.600+0000 7f1bd7ab5640 1 -- 172.21.15.1:0/507340212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd014e6b0 con 0x7f1bd00c2b30 2024-03-06T07:27:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.600+0000 7f1bc67fc640 1 -- 172.21.15.1:0/507340212 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0035040 con 0x7f1bd00c2b30 2024-03-06T07:27:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.600+0000 7f1bd7ab5640 1 -- 172.21.15.1:0/507340212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd014ec70 con 0x7f1bd00c2b30 2024-03-06T07:27:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.601+0000 7f1bd7ab5640 1 -- 172.21.15.1:0/507340212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd0153f70 con 0x7f1bd00c2b30 2024-03-06T07:27:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.602+0000 7f1bc67fc640 1 -- 172.21.15.1:0/507340212 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1bc004b060 con 0x7f1bd00c2b30 2024-03-06T07:27:53.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.602+0000 7f1bc67fc640 1 --2- 172.21.15.1:0/507340212 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1bb406b500 0x7f1bb406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.602+0000 7f1bc67fc640 1 -- 172.21.15.1:0/507340212 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1bc00ac220 con 0x7f1bd00c2b30 2024-03-06T07:27:53.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.602+0000 7f1bd5851640 1 --2- 172.21.15.1:0/507340212 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1bb406b500 0x7f1bb406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.603+0000 7f1bd5851640 1 --2- 172.21.15.1:0/507340212 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1bb406b500 0x7f1bb406d9c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc006fd0 tx=0x7f1bcc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.606+0000 7f1bc67fc640 1 -- 172.21.15.1:0/507340212 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1bc007afa0 con 0x7f1bd00c2b30 2024-03-06T07:27:53.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.828+0000 7f1bd7ab5640 1 -- 172.21.15.1:0/507340212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1bd0067fd0 con 0x7f1bd00c2b30 2024-03-06T07:27:53.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.833+0000 7f1bc67fc640 1 -- 172.21.15.1:0/507340212 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v169) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1bc007a8a0 con 0x7f1bd00c2b30 2024-03-06T07:27:53.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.835+0000 7f1bd7ab5640 1 -- 172.21.15.1:0/507340212 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1bb406b500 msgr2=0x7f1bb406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.835+0000 7f1bd7ab5640 1 --2- 172.21.15.1:0/507340212 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1bb406b500 0x7f1bb406d9c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc006fd0 tx=0x7f1bcc008040 comp rx=0 tx=0).stop 2024-03-06T07:27:53.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.835+0000 7f1bd7ab5640 1 -- 172.21.15.1:0/507340212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bd00c2b30 msgr2=0x7f1bd00bf9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.835+0000 7f1bd7ab5640 1 --2- 172.21.15.1:0/507340212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bd00c2b30 0x7f1bd00bf9f0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0000c00 tx=0x7f1bc00046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.835+0000 7f1bd7ab5640 1 -- 172.21.15.1:0/507340212 shutdown_connections 2024-03-06T07:27:53.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.835+0000 7f1bd7ab5640 1 --2- 172.21.15.1:0/507340212 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1bb406b500 0x7f1bb406d9c0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.835+0000 7f1bd7ab5640 1 --2- 172.21.15.1:0/507340212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd0152b70 0x7f1bd00bff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.835+0000 7f1bd7ab5640 1 --2- 172.21.15.1:0/507340212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bd00c2b30 0x7f1bd00bf9f0 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.835+0000 7f1bd7ab5640 1 --2- 172.21.15.1:0/507340212 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bd00a5fa0 0x7f1bd01539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.835+0000 7f1bd7ab5640 1 -- 172.21.15.1:0/507340212 >> 172.21.15.1:0/507340212 conn(0x7f1bd00c6860 msgr2=0x7f1bd0150470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:53.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.835+0000 7f1bd7ab5640 1 -- 172.21.15.1:0/507340212 shutdown_connections 2024-03-06T07:27:53.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.836+0000 7f1bd7ab5640 1 -- 172.21.15.1:0/507340212 wait complete. 2024-03-06T07:27:53.849 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:27:53.849 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:27:53.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.954+0000 7f85712d4640 1 Processor -- start 2024-03-06T07:27:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.954+0000 7f85712d4640 1 -- start start 2024-03-06T07:27:53.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.955+0000 7f85712d4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f856c152730 0x7f856c0c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.955+0000 7f85712d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f856c0c1aa0 0x7f856c1552e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.955+0000 7f85712d4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856c1588e0 con 0x7f856c0c6fd0 2024-03-06T07:27:53.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.955+0000 7f85712d4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856c158bc0 con 0x7f856c152730 2024-03-06T07:27:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.955+0000 7f85712d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c158ea0 con 0x7f856c0c1aa0 2024-03-06T07:27:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.955+0000 7f856a7fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f856c0c6fd0 0x7f856c0c1390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57896/0 (socket says 172.21.15.1:57896) 2024-03-06T07:27:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.955+0000 7f856a7fc640 1 -- 172.21.15.1:0/417321249 learned_addr learned my addr 172.21.15.1:0/417321249 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.955+0000 7f856b7fe640 1 --2- 172.21.15.1:0/417321249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f856c0c1aa0 0x7f856c1552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.955+0000 7f856affd640 1 --2- 172.21.15.1:0/417321249 >> v2:172.21.15.98:3300/0 conn(0x7f856c152730 0x7f856c0c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.955+0000 7f8569ffb640 1 -- 172.21.15.1:0/417321249 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692888123 0 0) 0x7f856c1588e0 con 0x7f856c0c6fd0 2024-03-06T07:27:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.955+0000 7f8569ffb640 1 -- 172.21.15.1:0/417321249 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8554003520 con 0x7f856c0c6fd0 2024-03-06T07:27:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.956+0000 7f856b7fe640 1 -- 172.21.15.1:0/417321249 >> v2:172.21.15.98:3300/0 conn(0x7f856c152730 msgr2=0x7f856c0c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.956+0000 7f856b7fe640 1 --2- 172.21.15.1:0/417321249 >> v2:172.21.15.98:3300/0 conn(0x7f856c152730 0x7f856c0c6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.956+0000 7f856b7fe640 1 -- 172.21.15.1:0/417321249 >> v1:172.21.15.1:6789/0 conn(0x7f856c0c6fd0 legacy=0x7f856c0c1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.956+0000 7f8569ffb640 1 -- 172.21.15.1:0/417321249 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4170452433 0 0) 0x7f8554003520 con 0x7f856c0c6fd0 2024-03-06T07:27:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.956+0000 7f856b7fe640 1 -- 172.21.15.1:0/417321249 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c159180 con 0x7f856c0c1aa0 2024-03-06T07:27:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.956+0000 7f856affd640 1 --2- 172.21.15.1:0/417321249 >> v2:172.21.15.98:3300/0 conn(0x7f856c152730 0x7f856c0c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.956+0000 7f856b7fe640 1 --2- 172.21.15.1:0/417321249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f856c0c1aa0 0x7f856c1552e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f855c009d90 tx=0x7f855c02bc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c48ef50df8dfbdb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.957+0000 7f8569ffb640 1 -- 172.21.15.1:0/417321249 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855c039070 con 0x7f856c0c1aa0 2024-03-06T07:27:54.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.957+0000 7f8569ffb640 1 -- 172.21.15.1:0/417321249 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f855c02c780 con 0x7f856c0c1aa0 2024-03-06T07:27:54.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.957+0000 7f8569ffb640 1 -- 172.21.15.1:0/417321249 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8560001670 con 0x7f856c0c1aa0 2024-03-06T07:27:54.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.957+0000 7f85712d4640 1 -- 172.21.15.1:0/417321249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f856c0c1aa0 msgr2=0x7f856c1552e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.957+0000 7f85712d4640 1 --2- 172.21.15.1:0/417321249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f856c0c1aa0 0x7f856c1552e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f855c009d90 tx=0x7f855c02bc50 comp rx=0 tx=0).stop 2024-03-06T07:27:54.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.957+0000 7f85712d4640 1 -- 172.21.15.1:0/417321249 shutdown_connections 2024-03-06T07:27:54.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.957+0000 7f85712d4640 1 --2- 172.21.15.1:0/417321249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f856c0c1aa0 0x7f856c1552e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.957+0000 7f85712d4640 1 --2- 172.21.15.1:0/417321249 >> v2:172.21.15.98:3300/0 conn(0x7f856c152730 0x7f856c0c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.957+0000 7f85712d4640 1 -- 172.21.15.1:0/417321249 >> 172.21.15.1:0/417321249 conn(0x7f856c066d00 msgr2=0x7f856c067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.957+0000 7f85712d4640 1 -- 172.21.15.1:0/417321249 shutdown_connections 2024-03-06T07:27:54.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.957+0000 7f85712d4640 1 -- 172.21.15.1:0/417321249 wait complete. 2024-03-06T07:27:54.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.958+0000 7f85712d4640 1 Processor -- start 2024-03-06T07:27:54.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.958+0000 7f85712d4640 1 -- start start 2024-03-06T07:27:54.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.959+0000 7f85712d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f856c0c1aa0 0x7f856c1ee280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.959+0000 7f85712d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f856c0c6fd0 0x7f856c1ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.959+0000 7f85712d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c152730 0x7f856c0afe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.959+0000 7f85712d4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c155820 con 0x7f856c152730 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.959+0000 7f85712d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c152b10 con 0x7f856c0c6fd0 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.959+0000 7f85712d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c1559a0 con 0x7f856c0c1aa0 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.959+0000 7f856b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c152730 0x7f856c0afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.959+0000 7f856b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c152730 0x7f856c0afe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58882/0 (socket says 172.21.15.1:58882) 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.959+0000 7f856b7fe640 1 -- 172.21.15.1:0/1490552534 learned_addr learned my addr 172.21.15.1:0/1490552534 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.959+0000 7f856affd640 1 --2- 172.21.15.1:0/1490552534 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f856c0c1aa0 0x7f856c1ee280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.959+0000 7f856a7fc640 1 --2- 172.21.15.1:0/1490552534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f856c0c6fd0 0x7f856c1ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.959+0000 7f856b7fe640 1 -- 172.21.15.1:0/1490552534 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f856c0c1aa0 msgr2=0x7f856c1ee280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.959+0000 7f856b7fe640 1 --2- 172.21.15.1:0/1490552534 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f856c0c1aa0 0x7f856c1ee280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.959+0000 7f856b7fe640 1 -- 172.21.15.1:0/1490552534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f856c0c6fd0 msgr2=0x7f856c1ee7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.959+0000 7f856b7fe640 1 --2- 172.21.15.1:0/1490552534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f856c0c6fd0 0x7f856c1ee7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.959+0000 7f856b7fe640 1 -- 172.21.15.1:0/1490552534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c1df8d0 con 0x7f856c152730 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.959+0000 7f856affd640 1 --2- 172.21.15.1:0/1490552534 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f856c0c1aa0 0x7f856c1ee280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.960+0000 7f856a7fc640 1 --2- 172.21.15.1:0/1490552534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f856c0c6fd0 0x7f856c1ee7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.960+0000 7f856b7fe640 1 --2- 172.21.15.1:0/1490552534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c152730 0x7f856c0afe70 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f855c037040 tx=0x7f855c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.960+0000 7f85537fe640 1 -- 172.21.15.1:0/1490552534 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855c039050 con 0x7f856c152730 2024-03-06T07:27:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.960+0000 7f85537fe640 1 -- 172.21.15.1:0/1490552534 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f855c045070 con 0x7f856c152730 2024-03-06T07:27:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.960+0000 7f85537fe640 1 -- 172.21.15.1:0/1490552534 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855c034e40 con 0x7f856c152730 2024-03-06T07:27:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.960+0000 7f85712d4640 1 -- 172.21.15.1:0/1490552534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856c1df310 con 0x7f856c152730 2024-03-06T07:27:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.960+0000 7f85712d4640 1 -- 172.21.15.1:0/1490552534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856c0c9970 con 0x7f856c152730 2024-03-06T07:27:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.961+0000 7f85712d4640 1 -- 172.21.15.1:0/1490552534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856c067fd0 con 0x7f856c152730 2024-03-06T07:27:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.961+0000 7f85537fe640 1 -- 172.21.15.1:0/1490552534 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f855c04a460 con 0x7f856c152730 2024-03-06T07:27:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.962+0000 7f85537fe640 1 --2- 172.21.15.1:0/1490552534 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f855806b4d0 0x7f855806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.962+0000 7f85537fe640 1 -- 172.21.15.1:0/1490552534 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f855c0acae0 con 0x7f856c152730 2024-03-06T07:27:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.962+0000 7f856affd640 1 --2- 172.21.15.1:0/1490552534 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f855806b4d0 0x7f855806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.963+0000 7f856affd640 1 --2- 172.21.15.1:0/1490552534 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f855806b4d0 0x7f855806d990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f8564009a20 tx=0x7f8564008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:53.966+0000 7f85537fe640 1 -- 172.21.15.1:0/1490552534 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f855c07b7c0 con 0x7f856c152730 2024-03-06T07:27:54.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.185+0000 7f85712d4640 1 -- 172.21.15.1:0/1490552534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f856c0c9b40 con 0x7f856c152730 2024-03-06T07:27:54.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.841+0000 7f85537fe640 1 -- 172.21.15.1:0/1490552534 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v170) v1 ==== 122+0+0 (secure 0 0 0) 0x7f855c07b0c0 con 0x7f856c152730 2024-03-06T07:27:54.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.844+0000 7f85712d4640 1 -- 172.21.15.1:0/1490552534 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f855806b4d0 msgr2=0x7f855806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.844+0000 7f85712d4640 1 --2- 172.21.15.1:0/1490552534 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f855806b4d0 0x7f855806d990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f8564009a20 tx=0x7f8564008040 comp rx=0 tx=0).stop 2024-03-06T07:27:54.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.844+0000 7f85712d4640 1 -- 172.21.15.1:0/1490552534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c152730 msgr2=0x7f856c0afe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.844+0000 7f85712d4640 1 --2- 172.21.15.1:0/1490552534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c152730 0x7f856c0afe70 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f855c037040 tx=0x7f855c004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:54.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.844+0000 7f85712d4640 1 -- 172.21.15.1:0/1490552534 shutdown_connections 2024-03-06T07:27:54.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.844+0000 7f85712d4640 1 --2- 172.21.15.1:0/1490552534 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f855806b4d0 0x7f855806d990 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.844+0000 7f85712d4640 1 --2- 172.21.15.1:0/1490552534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c152730 0x7f856c0afe70 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.844+0000 7f85712d4640 1 --2- 172.21.15.1:0/1490552534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f856c0c6fd0 0x7f856c1ee7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.844+0000 7f85712d4640 1 --2- 172.21.15.1:0/1490552534 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f856c0c1aa0 0x7f856c1ee280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.844+0000 7f85712d4640 1 -- 172.21.15.1:0/1490552534 >> 172.21.15.1:0/1490552534 conn(0x7f856c066d00 msgr2=0x7f856c1534c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.844+0000 7f85712d4640 1 -- 172.21.15.1:0/1490552534 shutdown_connections 2024-03-06T07:27:54.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.844+0000 7f85712d4640 1 -- 172.21.15.1:0/1490552534 wait complete. 2024-03-06T07:27:54.860 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:27:54.861 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:54.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.960+0000 7f7222d3e640 1 Processor -- start 2024-03-06T07:27:54.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.960+0000 7f7222d3e640 1 -- start start 2024-03-06T07:27:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.961+0000 7f7222d3e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f721c152650 0x7f721c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.961+0000 7f7222d3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f721c0bf9d0 0x7f721c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.961+0000 7f7222d3e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f721c0c2fc0 con 0x7f721c0c6b10 2024-03-06T07:27:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.961+0000 7f7222d3e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f721c155800 con 0x7f721c152650 2024-03-06T07:27:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.961+0000 7f7222d3e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721c158da0 con 0x7f721c0bf9d0 2024-03-06T07:27:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.961+0000 7f7220ada640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f721c152650 0x7f721c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.961+0000 7f7220ada640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f721c152650 0x7f721c155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60394/0 (socket says 172.21.15.1:60394) 2024-03-06T07:27:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.961+0000 7f7220ada640 1 -- 172.21.15.1:0/354868881 learned_addr learned my addr 172.21.15.1:0/354868881 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.962+0000 7f72212db640 1 --2- 172.21.15.1:0/354868881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f721c0bf9d0 0x7f721c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.962+0000 7f7220ada640 1 -- 172.21.15.1:0/354868881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f721c0bf9d0 msgr2=0x7f721c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.962+0000 7f7220ada640 1 --2- 172.21.15.1:0/354868881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f721c0bf9d0 0x7f721c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.962+0000 7f7220ada640 1 -- 172.21.15.1:0/354868881 >> v1:172.21.15.98:6789/0 conn(0x7f721c0c6b10 legacy=0x7f721c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.962+0000 7f7220ada640 1 -- 172.21.15.1:0/354868881 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c0c6690 con 0x7f721c152650 2024-03-06T07:27:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.962+0000 7f7220ada640 1 --2- 172.21.15.1:0/354868881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c152650 0x7f721c155c90 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f721c0cac90 tx=0x7f721002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28992ceb1e43e9df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.963+0000 7f721b7fe640 1 -- 172.21.15.1:0/354868881 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72100315a0 con 0x7f721c152650 2024-03-06T07:27:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.963+0000 7f721b7fe640 1 -- 172.21.15.1:0/354868881 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7210037070 con 0x7f721c152650 2024-03-06T07:27:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.963+0000 7f721b7fe640 1 -- 172.21.15.1:0/354868881 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7210031960 con 0x7f721c152650 2024-03-06T07:27:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.963+0000 7f7222d3e640 1 -- 172.21.15.1:0/354868881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c152650 msgr2=0x7f721c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.963+0000 7f7222d3e640 1 --2- 172.21.15.1:0/354868881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c152650 0x7f721c155c90 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f721c0cac90 tx=0x7f721002b840 comp rx=0 tx=0).stop 2024-03-06T07:27:54.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.963+0000 7f7222d3e640 1 -- 172.21.15.1:0/354868881 shutdown_connections 2024-03-06T07:27:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.963+0000 7f7222d3e640 1 --2- 172.21.15.1:0/354868881 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f721c0bf9d0 0x7f721c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.963+0000 7f7222d3e640 1 --2- 172.21.15.1:0/354868881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c152650 0x7f721c155c90 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.963+0000 7f7222d3e640 1 -- 172.21.15.1:0/354868881 >> 172.21.15.1:0/354868881 conn(0x7f721c066ee0 msgr2=0x7f721c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.963+0000 7f7222d3e640 1 -- 172.21.15.1:0/354868881 shutdown_connections 2024-03-06T07:27:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.963+0000 7f7222d3e640 1 -- 172.21.15.1:0/354868881 wait complete. 2024-03-06T07:27:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.964+0000 7f7222d3e640 1 Processor -- start 2024-03-06T07:27:54.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.964+0000 7f7222d3e640 1 -- start start 2024-03-06T07:27:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.965+0000 7f7222d3e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f721c0bf9d0 0x7f721c1ef000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.965+0000 7f7222d3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f721c152650 0x7f721c0c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.965+0000 7f7222d3e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c1514f0 0x7f721c13d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.965+0000 7f7222d3e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c1551b0 con 0x7f721c1514f0 2024-03-06T07:27:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.965+0000 7f7222d3e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c0c6fa0 con 0x7f721c0bf9d0 2024-03-06T07:27:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.965+0000 7f7222d3e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721c155490 con 0x7f721c152650 2024-03-06T07:27:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.965+0000 7f72212db640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c1514f0 0x7f721c13d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.965+0000 7f72212db640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c1514f0 0x7f721c13d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60402/0 (socket says 172.21.15.1:60402) 2024-03-06T07:27:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.965+0000 7f72212db640 1 -- 172.21.15.1:0/3407028761 learned_addr learned my addr 172.21.15.1:0/3407028761 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.965+0000 7f721bfff640 1 --2- 172.21.15.1:0/3407028761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f721c152650 0x7f721c0c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.965+0000 7f72212db640 1 -- 172.21.15.1:0/3407028761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f721c152650 msgr2=0x7f721c0c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.965+0000 7f72212db640 1 --2- 172.21.15.1:0/3407028761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f721c152650 0x7f721c0c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.965+0000 7f72212db640 1 -- 172.21.15.1:0/3407028761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f721c0bf9d0 msgr2=0x7f721c1ef000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.965+0000 7f7220ada640 1 --2- 172.21.15.1:0/3407028761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f721c0bf9d0 0x7f721c1ef000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.965+0000 7f72212db640 1 --2- 172.21.15.1:0/3407028761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f721c0bf9d0 0x7f721c1ef000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.965+0000 7f72212db640 1 -- 172.21.15.1:0/3407028761 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c151a40 con 0x7f721c1514f0 2024-03-06T07:27:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.966+0000 7f72212db640 1 --2- 172.21.15.1:0/3407028761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c1514f0 0x7f721c13d120 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f7214007ec0 tx=0x7f721400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.966+0000 7f72197fa640 1 -- 172.21.15.1:0/3407028761 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7214014620 con 0x7f721c1514f0 2024-03-06T07:27:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.966+0000 7f72197fa640 1 -- 172.21.15.1:0/3407028761 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7214019070 con 0x7f721c1514f0 2024-03-06T07:27:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.966+0000 7f7222d3e640 1 -- 172.21.15.1:0/3407028761 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721c156a10 con 0x7f721c1514f0 2024-03-06T07:27:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.966+0000 7f72197fa640 1 -- 172.21.15.1:0/3407028761 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7214014a20 con 0x7f721c1514f0 2024-03-06T07:27:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.966+0000 7f7222d3e640 1 -- 172.21.15.1:0/3407028761 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721c0c9fd0 con 0x7f721c1514f0 2024-03-06T07:27:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.967+0000 7f7222d3e640 1 -- 172.21.15.1:0/3407028761 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721c067fd0 con 0x7f721c1514f0 2024-03-06T07:27:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.967+0000 7f72197fa640 1 -- 172.21.15.1:0/3407028761 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7214014bc0 con 0x7f721c1514f0 2024-03-06T07:27:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.968+0000 7f72197fa640 1 --2- 172.21.15.1:0/3407028761 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f720006b500 0x7f720006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.968+0000 7f72197fa640 1 -- 172.21.15.1:0/3407028761 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f721408ab20 con 0x7f721c1514f0 2024-03-06T07:27:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.968+0000 7f7220ada640 1 --2- 172.21.15.1:0/3407028761 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f720006b500 0x7f720006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.969+0000 7f7220ada640 1 --2- 172.21.15.1:0/3407028761 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f720006b500 0x7f720006d9c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f7210036000 tx=0x7f7210008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:54.976+0000 7f72197fa640 1 -- 172.21.15.1:0/3407028761 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f72140598a0 con 0x7f721c1514f0 2024-03-06T07:27:55.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.194+0000 7f7222d3e640 1 -- 172.21.15.1:0/3407028761 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f721c0b01a0 con 0x7f721c1514f0 2024-03-06T07:27:55.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.194+0000 7f72197fa640 1 -- 172.21.15.1:0/3407028761 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+4981 (secure 0 0 0) 0x7f72140591a0 con 0x7f721c1514f0 2024-03-06T07:27:55.197 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:55.197 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":170,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":9951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:27:54.186815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9954":{"gid":9954,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1300509547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1300509547},{"type":"v1","addr":"172.21.15.1:6835","nonce":1300509547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9954,"qdb_cluster":[9954]},"id":14}]} 2024-03-06T07:27:55.197 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 170 2024-03-06T07:27:55.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.196+0000 7f7222d3e640 1 -- 172.21.15.1:0/3407028761 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f720006b500 msgr2=0x7f720006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.196+0000 7f7222d3e640 1 --2- 172.21.15.1:0/3407028761 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f720006b500 0x7f720006d9c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f7210036000 tx=0x7f7210008030 comp rx=0 tx=0).stop 2024-03-06T07:27:55.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.196+0000 7f7222d3e640 1 -- 172.21.15.1:0/3407028761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c1514f0 msgr2=0x7f721c13d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.196+0000 7f7222d3e640 1 --2- 172.21.15.1:0/3407028761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c1514f0 0x7f721c13d120 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f7214007ec0 tx=0x7f721400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.196+0000 7f7222d3e640 1 -- 172.21.15.1:0/3407028761 shutdown_connections 2024-03-06T07:27:55.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.196+0000 7f7222d3e640 1 --2- 172.21.15.1:0/3407028761 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f720006b500 0x7f720006d9c0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.196+0000 7f7222d3e640 1 --2- 172.21.15.1:0/3407028761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f721c1514f0 0x7f721c13d120 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.196+0000 7f7222d3e640 1 --2- 172.21.15.1:0/3407028761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f721c152650 0x7f721c0c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.196+0000 7f7222d3e640 1 --2- 172.21.15.1:0/3407028761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f721c0bf9d0 0x7f721c1ef000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.196+0000 7f7222d3e640 1 -- 172.21.15.1:0/3407028761 >> 172.21.15.1:0/3407028761 conn(0x7f721c066ee0 msgr2=0x7f721c0c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.196+0000 7f7222d3e640 1 -- 172.21.15.1:0/3407028761 shutdown_connections 2024-03-06T07:27:55.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.196+0000 7f7222d3e640 1 -- 172.21.15.1:0/3407028761 wait complete. 2024-03-06T07:27:55.209 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:55.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.309+0000 7f1627d8a640 1 Processor -- start 2024-03-06T07:27:55.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.310+0000 7f1627d8a640 1 -- start start 2024-03-06T07:27:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.310+0000 7f1627d8a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f16200c6b10 0x7f1620155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.310+0000 7f1627d8a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16200bf9d0 0x7f16200c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.310+0000 7f1627d8a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16200c2fc0 con 0x7f1620152650 2024-03-06T07:27:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.310+0000 7f1627d8a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1620155800 con 0x7f16200c6b10 2024-03-06T07:27:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.310+0000 7f1627d8a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1620158da0 con 0x7f16200bf9d0 2024-03-06T07:27:55.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.311+0000 7f1626327640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16200bf9d0 0x7f16200c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.311+0000 7f1626327640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16200bf9d0 0x7f16200c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:33476/0 (socket says 172.21.15.1:33476) 2024-03-06T07:27:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.311+0000 7f1626327640 1 -- 172.21.15.1:0/826976952 learned_addr learned my addr 172.21.15.1:0/826976952 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.311+0000 7f1625325640 1 --2- 172.21.15.1:0/826976952 >> v2:172.21.15.98:3300/0 conn(0x7f16200c6b10 0x7f1620155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.311+0000 7f1626327640 1 -- 172.21.15.1:0/826976952 >> v2:172.21.15.98:3300/0 conn(0x7f16200c6b10 msgr2=0x7f1620155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.311+0000 7f1626327640 1 --2- 172.21.15.1:0/826976952 >> v2:172.21.15.98:3300/0 conn(0x7f16200c6b10 0x7f1620155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.311+0000 7f1626327640 1 -- 172.21.15.1:0/826976952 >> v1:172.21.15.98:6789/0 conn(0x7f1620152650 legacy=0x7f1620158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.311+0000 7f1626327640 1 -- 172.21.15.1:0/826976952 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16200c6690 con 0x7f16200bf9d0 2024-03-06T07:27:55.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.312+0000 7f1626327640 1 --2- 172.21.15.1:0/826976952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16200bf9d0 0x7f16200c1460 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1618009e50 tx=0x7f161802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a7ddf33919c4966 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.312+0000 7f1624b24640 1 -- 172.21.15.1:0/826976952 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161802ca70 con 0x7f16200bf9d0 2024-03-06T07:27:55.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.312+0000 7f1624b24640 1 -- 172.21.15.1:0/826976952 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f161802cc10 con 0x7f16200bf9d0 2024-03-06T07:27:55.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.312+0000 7f1624b24640 1 -- 172.21.15.1:0/826976952 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16180343f0 con 0x7f16200bf9d0 2024-03-06T07:27:55.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.313+0000 7f1627d8a640 1 -- 172.21.15.1:0/826976952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16200bf9d0 msgr2=0x7f16200c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.313+0000 7f1627d8a640 1 --2- 172.21.15.1:0/826976952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16200bf9d0 0x7f16200c1460 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1618009e50 tx=0x7f161802bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.313+0000 7f1627d8a640 1 -- 172.21.15.1:0/826976952 shutdown_connections 2024-03-06T07:27:55.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.313+0000 7f1627d8a640 1 --2- 172.21.15.1:0/826976952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16200bf9d0 0x7f16200c1460 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.313+0000 7f1627d8a640 1 --2- 172.21.15.1:0/826976952 >> v2:172.21.15.98:3300/0 conn(0x7f16200c6b10 0x7f1620155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.313+0000 7f1627d8a640 1 -- 172.21.15.1:0/826976952 >> 172.21.15.1:0/826976952 conn(0x7f1620066ee0 msgr2=0x7f16200672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.313+0000 7f1627d8a640 1 -- 172.21.15.1:0/826976952 shutdown_connections 2024-03-06T07:27:55.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.313+0000 7f1627d8a640 1 -- 172.21.15.1:0/826976952 wait complete. 2024-03-06T07:27:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.314+0000 7f1627d8a640 1 Processor -- start 2024-03-06T07:27:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.314+0000 7f1627d8a640 1 -- start start 2024-03-06T07:27:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f1627d8a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16200bf9d0 0x7f162014b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f1627d8a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1620152650 0x7f1620150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f1627d8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1620151be0 0x7f1620153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f1627d8a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16201551b0 con 0x7f16200bf9d0 2024-03-06T07:27:55.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f1627d8a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16200c6fa0 con 0x7f1620151be0 2024-03-06T07:27:55.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f1627d8a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1620155490 con 0x7f1620152650 2024-03-06T07:27:55.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f1625b26640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16200bf9d0 0x7f162014b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f1625b26640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16200bf9d0 0x7f162014b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60410/0 (socket says 172.21.15.1:60410) 2024-03-06T07:27:55.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f1625b26640 1 -- 172.21.15.1:0/1519931606 learned_addr learned my addr 172.21.15.1:0/1519931606 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:55.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f1625325640 1 --2- 172.21.15.1:0/1519931606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1620152650 0x7f1620150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f1625b26640 1 -- 172.21.15.1:0/1519931606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1620152650 msgr2=0x7f1620150800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f1625b26640 1 --2- 172.21.15.1:0/1519931606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1620152650 0x7f1620150800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f1625b26640 1 -- 172.21.15.1:0/1519931606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1620151be0 msgr2=0x7f1620153160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f1625b26640 1 --2- 172.21.15.1:0/1519931606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1620151be0 0x7f1620153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f1625b26640 1 -- 172.21.15.1:0/1519931606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1620158760 con 0x7f16200bf9d0 2024-03-06T07:27:55.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f1625325640 1 --2- 172.21.15.1:0/1519931606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1620152650 0x7f1620150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:55.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f1625b26640 1 --2- 172.21.15.1:0/1519931606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16200bf9d0 0x7f162014b1e0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f161c00c430 tx=0x7f161c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.315+0000 7f16167fc640 1 -- 172.21.15.1:0/1519931606 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c017070 con 0x7f16200bf9d0 2024-03-06T07:27:55.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.316+0000 7f16167fc640 1 -- 172.21.15.1:0/1519931606 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f161c0052d0 con 0x7f16200bf9d0 2024-03-06T07:27:55.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.316+0000 7f1627d8a640 1 -- 172.21.15.1:0/1519931606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1620158a40 con 0x7f16200bf9d0 2024-03-06T07:27:55.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.316+0000 7f16167fc640 1 -- 172.21.15.1:0/1519931606 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c003850 con 0x7f16200bf9d0 2024-03-06T07:27:55.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.316+0000 7f1627d8a640 1 -- 172.21.15.1:0/1519931606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16201516c0 con 0x7f16200bf9d0 2024-03-06T07:27:55.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.317+0000 7f1627d8a640 1 -- 172.21.15.1:0/1519931606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1620151210 con 0x7f16200bf9d0 2024-03-06T07:27:55.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.318+0000 7f16167fc640 1 -- 172.21.15.1:0/1519931606 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f161c01a070 con 0x7f16200bf9d0 2024-03-06T07:27:55.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.318+0000 7f16167fc640 1 --2- 172.21.15.1:0/1519931606 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f160406b4d0 0x7f160406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.318+0000 7f16167fc640 1 -- 172.21.15.1:0/1519931606 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f161c089730 con 0x7f16200bf9d0 2024-03-06T07:27:55.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.318+0000 7f1625325640 1 --2- 172.21.15.1:0/1519931606 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f160406b4d0 0x7f160406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.319+0000 7f1625325640 1 --2- 172.21.15.1:0/1519931606 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f160406b4d0 0x7f160406d990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1620155ce0 tx=0x7f1610008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.323+0000 7f16167fc640 1 -- 172.21.15.1:0/1519931606 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f161c058420 con 0x7f16200bf9d0 2024-03-06T07:27:55.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.503+0000 7f1627d8a640 1 -- 172.21.15.1:0/1519931606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1620067fd0 con 0x7f16200bf9d0 2024-03-06T07:27:55.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.504+0000 7f16167fc640 1 -- 172.21.15.1:0/1519931606 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v183) v1 ==== 74+0+16764 (secure 0 0 0) 0x7f161c057d20 con 0x7f16200bf9d0 2024-03-06T07:27:55.506 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:55.506 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":183,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:27:53.821050+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:50.456844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:51.147612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.506+0000 7f1627d8a640 1 -- 172.21.15.1:0/1519931606 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f160406b4d0 msgr2=0x7f160406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.506+0000 7f1627d8a640 1 --2- 172.21.15.1:0/1519931606 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f160406b4d0 0x7f160406d990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1620155ce0 tx=0x7f1610008040 comp rx=0 tx=0).stop 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.506+0000 7f1627d8a640 1 -- 172.21.15.1:0/1519931606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16200bf9d0 msgr2=0x7f162014b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.506+0000 7f1627d8a640 1 --2- 172.21.15.1:0/1519931606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16200bf9d0 0x7f162014b1e0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f161c00c430 tx=0x7f161c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.506+0000 7f1627d8a640 1 -- 172.21.15.1:0/1519931606 shutdown_connections 2024-03-06T07:27:55.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.506+0000 7f1627d8a640 1 --2- 172.21.15.1:0/1519931606 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f160406b4d0 0x7f160406d990 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.506+0000 7f1627d8a640 1 --2- 172.21.15.1:0/1519931606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1620151be0 0x7f1620153160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.506+0000 7f1627d8a640 1 --2- 172.21.15.1:0/1519931606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1620152650 0x7f1620150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.506+0000 7f1627d8a640 1 --2- 172.21.15.1:0/1519931606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16200bf9d0 0x7f162014b1e0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.506+0000 7f1627d8a640 1 -- 172.21.15.1:0/1519931606 >> 172.21.15.1:0/1519931606 conn(0x7f1620066ee0 msgr2=0x7f16200c04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.506+0000 7f1627d8a640 1 -- 172.21.15.1:0/1519931606 shutdown_connections 2024-03-06T07:27:55.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.506+0000 7f1627d8a640 1 -- 172.21.15.1:0/1519931606 wait complete. 2024-03-06T07:27:55.519 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:27:55.519 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:55.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.622+0000 7feea82df640 1 Processor -- start 2024-03-06T07:27:55.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.622+0000 7feea82df640 1 -- start start 2024-03-06T07:27:55.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.623+0000 7feea82df640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feea00c6b10 0x7feea0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.623+0000 7feea82df640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feea00bf9d0 0x7feea00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.623+0000 7feea82df640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea00c2fc0 con 0x7feea0152650 2024-03-06T07:27:55.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.623+0000 7feea82df640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea0155800 con 0x7feea00c6b10 2024-03-06T07:27:55.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.623+0000 7feea82df640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea0158da0 con 0x7feea00bf9d0 2024-03-06T07:27:55.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.624+0000 7feea607b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7feea0152650 0x7feea0158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50602/0 (socket says 172.21.15.1:50602) 2024-03-06T07:27:55.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.624+0000 7feea607b640 1 -- 172.21.15.1:0/670295446 learned_addr learned my addr 172.21.15.1:0/670295446 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:55.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.624+0000 7feea587a640 1 --2- 172.21.15.1:0/670295446 >> v2:172.21.15.1:3300/0 conn(0x7feea00c6b10 0x7feea0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.624+0000 7feea587a640 1 -- 172.21.15.1:0/670295446 >> v2:172.21.15.98:3300/0 conn(0x7feea00bf9d0 msgr2=0x7feea00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:55.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.624+0000 7feea687c640 1 --2- 172.21.15.1:0/670295446 >> v2:172.21.15.98:3300/0 conn(0x7feea00bf9d0 0x7feea00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.624+0000 7feea5079640 1 -- 172.21.15.1:0/670295446 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881757810 0 0) 0x7feea00c2fc0 con 0x7feea0152650 2024-03-06T07:27:55.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.624+0000 7feea587a640 1 --2- 172.21.15.1:0/670295446 >> v2:172.21.15.98:3300/0 conn(0x7feea00bf9d0 0x7feea00c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.624+0000 7feea587a640 1 -- 172.21.15.1:0/670295446 >> v1:172.21.15.1:6789/0 conn(0x7feea0152650 legacy=0x7feea0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.624+0000 7feea587a640 1 -- 172.21.15.1:0/670295446 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea00c6690 con 0x7feea00c6b10 2024-03-06T07:27:55.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.624+0000 7feea687c640 1 --2- 172.21.15.1:0/670295446 >> v2:172.21.15.98:3300/0 conn(0x7feea00bf9d0 0x7feea00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:55.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.625+0000 7feea587a640 1 --2- 172.21.15.1:0/670295446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feea00c6b10 0x7feea0155c90 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7feea01597b0 tx=0x7fee9002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5f2db7a5e9758e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.625+0000 7feea5079640 1 -- 172.21.15.1:0/670295446 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee900315a0 con 0x7feea00c6b10 2024-03-06T07:27:55.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.625+0000 7feea5079640 1 -- 172.21.15.1:0/670295446 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee90037070 con 0x7feea00c6b10 2024-03-06T07:27:55.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.625+0000 7feea5079640 1 -- 172.21.15.1:0/670295446 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee90031960 con 0x7feea00c6b10 2024-03-06T07:27:55.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.625+0000 7feea82df640 1 -- 172.21.15.1:0/670295446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feea00c6b10 msgr2=0x7feea0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.625+0000 7feea82df640 1 --2- 172.21.15.1:0/670295446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feea00c6b10 0x7feea0155c90 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7feea01597b0 tx=0x7fee9002b840 comp rx=0 tx=0).stop 2024-03-06T07:27:55.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.625+0000 7feea82df640 1 -- 172.21.15.1:0/670295446 shutdown_connections 2024-03-06T07:27:55.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.625+0000 7feea82df640 1 --2- 172.21.15.1:0/670295446 >> v2:172.21.15.98:3300/0 conn(0x7feea00bf9d0 0x7feea00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.625+0000 7feea82df640 1 --2- 172.21.15.1:0/670295446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feea00c6b10 0x7feea0155c90 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.625+0000 7feea82df640 1 -- 172.21.15.1:0/670295446 >> 172.21.15.1:0/670295446 conn(0x7feea0066ee0 msgr2=0x7feea00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.625+0000 7feea82df640 1 -- 172.21.15.1:0/670295446 shutdown_connections 2024-03-06T07:27:55.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.625+0000 7feea82df640 1 -- 172.21.15.1:0/670295446 wait complete. 2024-03-06T07:27:55.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.626+0000 7feea82df640 1 Processor -- start 2024-03-06T07:27:55.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.627+0000 7feea82df640 1 -- start start 2024-03-06T07:27:55.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.627+0000 7feea82df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feea00bf9d0 0x7feea01df860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.627+0000 7feea82df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feea0152650 0x7feea014ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.627+0000 7feea82df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feea01583b0 0x7feea014d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.627+0000 7feea82df640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea01551b0 con 0x7feea0152650 2024-03-06T07:27:55.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.627+0000 7feea82df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea00c6fa0 con 0x7feea00bf9d0 2024-03-06T07:27:55.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.627+0000 7feea82df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea0155490 con 0x7feea01583b0 2024-03-06T07:27:55.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.627+0000 7feea587a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feea0152650 0x7feea014ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.627+0000 7feea587a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feea0152650 0x7feea014ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60422/0 (socket says 172.21.15.1:60422) 2024-03-06T07:27:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.627+0000 7feea587a640 1 -- 172.21.15.1:0/3618699307 learned_addr learned my addr 172.21.15.1:0/3618699307 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.628+0000 7feea607b640 1 --2- 172.21.15.1:0/3618699307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feea00bf9d0 0x7feea01df860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.628+0000 7feea587a640 1 -- 172.21.15.1:0/3618699307 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feea01583b0 msgr2=0x7feea014d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.628+0000 7feea587a640 1 --2- 172.21.15.1:0/3618699307 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feea01583b0 0x7feea014d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.628+0000 7feea587a640 1 -- 172.21.15.1:0/3618699307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feea00bf9d0 msgr2=0x7feea01df860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.628+0000 7feea587a640 1 --2- 172.21.15.1:0/3618699307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feea00bf9d0 0x7feea01df860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.628+0000 7feea587a640 1 -- 172.21.15.1:0/3618699307 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea00c6090 con 0x7feea0152650 2024-03-06T07:27:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.628+0000 7feea587a640 1 --2- 172.21.15.1:0/3618699307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feea0152650 0x7feea014ce20 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fee90000c00 tx=0x7fee900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.628+0000 7fee96ffd640 1 -- 172.21.15.1:0/3618699307 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee90032840 con 0x7feea0152650 2024-03-06T07:27:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.628+0000 7fee96ffd640 1 -- 172.21.15.1:0/3618699307 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee90032a00 con 0x7feea0152650 2024-03-06T07:27:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.628+0000 7feea82df640 1 -- 172.21.15.1:0/3618699307 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea01de950 con 0x7feea0152650 2024-03-06T07:27:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.628+0000 7fee96ffd640 1 -- 172.21.15.1:0/3618699307 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee90032df0 con 0x7feea0152650 2024-03-06T07:27:55.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.628+0000 7feea82df640 1 -- 172.21.15.1:0/3618699307 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea01def10 con 0x7feea0152650 2024-03-06T07:27:55.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.629+0000 7feea82df640 1 -- 172.21.15.1:0/3618699307 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea00b0d50 con 0x7feea0152650 2024-03-06T07:27:55.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.630+0000 7fee96ffd640 1 -- 172.21.15.1:0/3618699307 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee9004d3f0 con 0x7feea0152650 2024-03-06T07:27:55.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.630+0000 7fee96ffd640 1 --2- 172.21.15.1:0/3618699307 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee8406b4d0 0x7fee8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.630+0000 7fee96ffd640 1 -- 172.21.15.1:0/3618699307 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fee9002f030 con 0x7feea0152650 2024-03-06T07:27:55.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.630+0000 7feea607b640 1 --2- 172.21.15.1:0/3618699307 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee8406b4d0 0x7fee8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.631+0000 7feea607b640 1 --2- 172.21.15.1:0/3618699307 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee8406b4d0 0x7fee8406d990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fee9c007fd0 tx=0x7fee9c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.634+0000 7fee96ffd640 1 -- 172.21.15.1:0/3618699307 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feea00b0d50 con 0x7feea0152650 2024-03-06T07:27:55.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.887+0000 7feea82df640 1 -- 172.21.15.1:0/3618699307 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feea00c0310 con 0x7fee8406b4d0 2024-03-06T07:27:55.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.893+0000 7fee96ffd640 1 -- 172.21.15.1:0/3618699307 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252563 (secure 0 0 0) 0x7feea00c0310 con 0x7fee8406b4d0 2024-03-06T07:27:55.899 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:55.900 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:27:55.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.897+0000 7feea82df640 1 -- 172.21.15.1:0/3618699307 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee8406b4d0 msgr2=0x7fee8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.897+0000 7feea82df640 1 --2- 172.21.15.1:0/3618699307 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee8406b4d0 0x7fee8406d990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fee9c007fd0 tx=0x7fee9c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:55.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.897+0000 7feea82df640 1 -- 172.21.15.1:0/3618699307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feea0152650 msgr2=0x7feea014ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.897+0000 7feea82df640 1 --2- 172.21.15.1:0/3618699307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feea0152650 0x7feea014ce20 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fee90000c00 tx=0x7fee900046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.897+0000 7feea82df640 1 -- 172.21.15.1:0/3618699307 shutdown_connections 2024-03-06T07:27:55.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.897+0000 7feea82df640 1 --2- 172.21.15.1:0/3618699307 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee8406b4d0 0x7fee8406d990 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.897+0000 7feea82df640 1 --2- 172.21.15.1:0/3618699307 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feea01583b0 0x7feea014d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.897+0000 7feea82df640 1 --2- 172.21.15.1:0/3618699307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feea0152650 0x7feea014ce20 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.897+0000 7feea82df640 1 --2- 172.21.15.1:0/3618699307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feea00bf9d0 0x7feea01df860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.897+0000 7feea82df640 1 -- 172.21.15.1:0/3618699307 >> 172.21.15.1:0/3618699307 conn(0x7feea0066ee0 msgr2=0x7feea00c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.897+0000 7feea82df640 1 -- 172.21.15.1:0/3618699307 shutdown_connections 2024-03-06T07:27:55.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:55.897+0000 7feea82df640 1 -- 172.21.15.1:0/3618699307 wait complete. 2024-03-06T07:27:55.910 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":406,"stamp":"2024-03-06T07:27:54.796831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":125,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":276052,"kb_used_data":18268,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754698668,"statfs":{"total":773094113280,"available":772811436032,"internally_reserved":0,"allocated":18706432,"data_stored":10202880,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001092"},"pg_stats":[{"pgid":"30.3f","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228517+0000","last_change":"2024-03-06T07:27:52.842860+0000","last_active":"2024-03-06T07:27:54.228517+0000","last_peered":"2024-03-06T07:27:54.228517+0000","last_clean":"2024-03-06T07:27:54.228517+0000","last_became_active":"2024-03-06T07:27:52.842531+0000","last_became_peered":"2024-03-06T07:27:52.842531+0000","last_unstale":"2024-03-06T07:27:54.228517+0000","last_undegraded":"2024-03-06T07:27:54.228517+0000","last_fullsized":"2024-03-06T07:27:54.228517+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:14.199266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227456+0000","last_change":"2024-03-06T07:27:52.843586+0000","last_active":"2024-03-06T07:27:54.227456+0000","last_peered":"2024-03-06T07:27:54.227456+0000","last_clean":"2024-03-06T07:27:54.227456+0000","last_became_active":"2024-03-06T07:27:52.843302+0000","last_became_peered":"2024-03-06T07:27:52.843302+0000","last_unstale":"2024-03-06T07:27:54.227456+0000","last_undegraded":"2024-03-06T07:27:54.227456+0000","last_fullsized":"2024-03-06T07:27:54.227456+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:04.175670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827599+0000","last_change":"2024-03-06T07:27:52.841423+0000","last_active":"2024-03-06T07:27:53.827599+0000","last_peered":"2024-03-06T07:27:53.827599+0000","last_clean":"2024-03-06T07:27:53.827599+0000","last_became_active":"2024-03-06T07:27:52.841148+0000","last_became_peered":"2024-03-06T07:27:52.841148+0000","last_unstale":"2024-03-06T07:27:53.827599+0000","last_undegraded":"2024-03-06T07:27:53.827599+0000","last_fullsized":"2024-03-06T07:27:53.827599+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:44.654522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.828425+0000","last_change":"2024-03-06T07:27:52.836948+0000","last_active":"2024-03-06T07:27:53.828425+0000","last_peered":"2024-03-06T07:27:53.828425+0000","last_clean":"2024-03-06T07:27:53.828425+0000","last_became_active":"2024-03-06T07:27:52.836693+0000","last_became_peered":"2024-03-06T07:27:52.836693+0000","last_unstale":"2024-03-06T07:27:53.828425+0000","last_undegraded":"2024-03-06T07:27:53.828425+0000","last_fullsized":"2024-03-06T07:27:53.828425+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:33.742320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228131+0000","last_change":"2024-03-06T07:27:52.848857+0000","last_active":"2024-03-06T07:27:54.228131+0000","last_peered":"2024-03-06T07:27:54.228131+0000","last_clean":"2024-03-06T07:27:54.228131+0000","last_became_active":"2024-03-06T07:27:52.848253+0000","last_became_peered":"2024-03-06T07:27:52.848253+0000","last_unstale":"2024-03-06T07:27:54.228131+0000","last_undegraded":"2024-03-06T07:27:54.228131+0000","last_fullsized":"2024-03-06T07:27:54.228131+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:17.947842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116875+0000","last_change":"2024-03-06T07:27:52.839769+0000","last_active":"2024-03-06T07:27:54.116875+0000","last_peered":"2024-03-06T07:27:54.116875+0000","last_clean":"2024-03-06T07:27:54.116875+0000","last_became_active":"2024-03-06T07:27:52.839488+0000","last_became_peered":"2024-03-06T07:27:52.839488+0000","last_unstale":"2024-03-06T07:27:54.116875+0000","last_undegraded":"2024-03-06T07:27:54.116875+0000","last_fullsized":"2024-03-06T07:27:54.116875+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:43.780203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827357+0000","last_change":"2024-03-06T07:27:52.845031+0000","last_active":"2024-03-06T07:27:53.827357+0000","last_peered":"2024-03-06T07:27:53.827357+0000","last_clean":"2024-03-06T07:27:53.827357+0000","last_became_active":"2024-03-06T07:27:52.844686+0000","last_became_peered":"2024-03-06T07:27:52.844686+0000","last_unstale":"2024-03-06T07:27:53.827357+0000","last_undegraded":"2024-03-06T07:27:53.827357+0000","last_fullsized":"2024-03-06T07:27:53.827357+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:18.206755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116506+0000","last_change":"2024-03-06T07:27:52.838158+0000","last_active":"2024-03-06T07:27:54.116506+0000","last_peered":"2024-03-06T07:27:54.116506+0000","last_clean":"2024-03-06T07:27:54.116506+0000","last_became_active":"2024-03-06T07:27:52.837890+0000","last_became_peered":"2024-03-06T07:27:52.837890+0000","last_unstale":"2024-03-06T07:27:54.116506+0000","last_undegraded":"2024-03-06T07:27:54.116506+0000","last_fullsized":"2024-03-06T07:27:54.116506+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:33.431453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824522+0000","last_change":"2024-03-06T07:27:52.848310+0000","last_active":"2024-03-06T07:27:53.824522+0000","last_peered":"2024-03-06T07:27:53.824522+0000","last_clean":"2024-03-06T07:27:53.824522+0000","last_became_active":"2024-03-06T07:27:52.848053+0000","last_became_peered":"2024-03-06T07:27:52.848053+0000","last_unstale":"2024-03-06T07:27:53.824522+0000","last_undegraded":"2024-03-06T07:27:53.824522+0000","last_fullsized":"2024-03-06T07:27:53.824522+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:03.922799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827674+0000","last_change":"2024-03-06T07:27:52.842430+0000","last_active":"2024-03-06T07:27:53.827674+0000","last_peered":"2024-03-06T07:27:53.827674+0000","last_clean":"2024-03-06T07:27:53.827674+0000","last_became_active":"2024-03-06T07:27:52.842176+0000","last_became_peered":"2024-03-06T07:27:52.842176+0000","last_unstale":"2024-03-06T07:27:53.827674+0000","last_undegraded":"2024-03-06T07:27:53.827674+0000","last_fullsized":"2024-03-06T07:27:53.827674+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:02.576405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.828524+0000","last_change":"2024-03-06T07:27:52.832071+0000","last_active":"2024-03-06T07:27:53.828524+0000","last_peered":"2024-03-06T07:27:53.828524+0000","last_clean":"2024-03-06T07:27:53.828524+0000","last_became_active":"2024-03-06T07:27:52.831756+0000","last_became_peered":"2024-03-06T07:27:52.831756+0000","last_unstale":"2024-03-06T07:27:53.828524+0000","last_undegraded":"2024-03-06T07:27:53.828524+0000","last_fullsized":"2024-03-06T07:27:53.828524+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:08.654039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229066+0000","last_change":"2024-03-06T07:27:52.849912+0000","last_active":"2024-03-06T07:27:54.229066+0000","last_peered":"2024-03-06T07:27:54.229066+0000","last_clean":"2024-03-06T07:27:54.229066+0000","last_became_active":"2024-03-06T07:27:52.849284+0000","last_became_peered":"2024-03-06T07:27:52.849284+0000","last_unstale":"2024-03-06T07:27:54.229066+0000","last_undegraded":"2024-03-06T07:27:54.229066+0000","last_fullsized":"2024-03-06T07:27:54.229066+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:38.851748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116740+0000","last_change":"2024-03-06T07:27:52.835245+0000","last_active":"2024-03-06T07:27:54.116740+0000","last_peered":"2024-03-06T07:27:54.116740+0000","last_clean":"2024-03-06T07:27:54.116740+0000","last_became_active":"2024-03-06T07:27:52.834973+0000","last_became_peered":"2024-03-06T07:27:52.834973+0000","last_unstale":"2024-03-06T07:27:54.116740+0000","last_undegraded":"2024-03-06T07:27:54.116740+0000","last_fullsized":"2024-03-06T07:27:54.116740+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:25.883597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228795+0000","last_change":"2024-03-06T07:27:52.850092+0000","last_active":"2024-03-06T07:27:54.228795+0000","last_peered":"2024-03-06T07:27:54.228795+0000","last_clean":"2024-03-06T07:27:54.228795+0000","last_became_active":"2024-03-06T07:27:52.849770+0000","last_became_peered":"2024-03-06T07:27:52.849770+0000","last_unstale":"2024-03-06T07:27:54.228795+0000","last_undegraded":"2024-03-06T07:27:54.228795+0000","last_fullsized":"2024-03-06T07:27:54.228795+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:49.841334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842594+0000","last_change":"2024-03-06T07:27:52.847806+0000","last_active":"2024-03-06T07:27:53.842594+0000","last_peered":"2024-03-06T07:27:53.842594+0000","last_clean":"2024-03-06T07:27:53.842594+0000","last_became_active":"2024-03-06T07:27:52.847516+0000","last_became_peered":"2024-03-06T07:27:52.847516+0000","last_unstale":"2024-03-06T07:27:53.842594+0000","last_undegraded":"2024-03-06T07:27:53.842594+0000","last_fullsized":"2024-03-06T07:27:53.842594+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:17.947842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116592+0000","last_change":"2024-03-06T07:27:52.840358+0000","last_active":"2024-03-06T07:27:54.116592+0000","last_peered":"2024-03-06T07:27:54.116592+0000","last_clean":"2024-03-06T07:27:54.116592+0000","last_became_active":"2024-03-06T07:27:52.840039+0000","last_became_peered":"2024-03-06T07:27:52.840039+0000","last_unstale":"2024-03-06T07:27:54.116592+0000","last_undegraded":"2024-03-06T07:27:54.116592+0000","last_fullsized":"2024-03-06T07:27:54.116592+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:40.496286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229137+0000","last_change":"2024-03-06T07:27:52.848809+0000","last_active":"2024-03-06T07:27:54.229137+0000","last_peered":"2024-03-06T07:27:54.229137+0000","last_clean":"2024-03-06T07:27:54.229137+0000","last_became_active":"2024-03-06T07:27:52.847002+0000","last_became_peered":"2024-03-06T07:27:52.847002+0000","last_unstale":"2024-03-06T07:27:54.229137+0000","last_undegraded":"2024-03-06T07:27:54.229137+0000","last_fullsized":"2024-03-06T07:27:54.229137+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:09.978274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827669+0000","last_change":"2024-03-06T07:27:52.841828+0000","last_active":"2024-03-06T07:27:53.827669+0000","last_peered":"2024-03-06T07:27:53.827669+0000","last_clean":"2024-03-06T07:27:53.827669+0000","last_became_active":"2024-03-06T07:27:52.840848+0000","last_became_peered":"2024-03-06T07:27:52.840848+0000","last_unstale":"2024-03-06T07:27:53.827669+0000","last_undegraded":"2024-03-06T07:27:53.827669+0000","last_fullsized":"2024-03-06T07:27:53.827669+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:56.120793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228291+0000","last_change":"2024-03-06T07:27:52.843757+0000","last_active":"2024-03-06T07:27:54.228291+0000","last_peered":"2024-03-06T07:27:54.228291+0000","last_clean":"2024-03-06T07:27:54.228291+0000","last_became_active":"2024-03-06T07:27:52.843251+0000","last_became_peered":"2024-03-06T07:27:52.843251+0000","last_unstale":"2024-03-06T07:27:54.228291+0000","last_undegraded":"2024-03-06T07:27:54.228291+0000","last_fullsized":"2024-03-06T07:27:54.228291+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:02.906107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116915+0000","last_change":"2024-03-06T07:27:52.846815+0000","last_active":"2024-03-06T07:27:54.116915+0000","last_peered":"2024-03-06T07:27:54.116915+0000","last_clean":"2024-03-06T07:27:54.116915+0000","last_became_active":"2024-03-06T07:27:52.846395+0000","last_became_peered":"2024-03-06T07:27:52.846395+0000","last_unstale":"2024-03-06T07:27:54.116915+0000","last_undegraded":"2024-03-06T07:27:54.116915+0000","last_fullsized":"2024-03-06T07:27:54.116915+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:31.090488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827344+0000","last_change":"2024-03-06T07:27:52.840352+0000","last_active":"2024-03-06T07:27:53.827344+0000","last_peered":"2024-03-06T07:27:53.827344+0000","last_clean":"2024-03-06T07:27:53.827344+0000","last_became_active":"2024-03-06T07:27:52.840069+0000","last_became_peered":"2024-03-06T07:27:52.840069+0000","last_unstale":"2024-03-06T07:27:53.827344+0000","last_undegraded":"2024-03-06T07:27:53.827344+0000","last_fullsized":"2024-03-06T07:27:53.827344+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:58.583352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842762+0000","last_change":"2024-03-06T07:27:52.842867+0000","last_active":"2024-03-06T07:27:53.842762+0000","last_peered":"2024-03-06T07:27:53.842762+0000","last_clean":"2024-03-06T07:27:53.842762+0000","last_became_active":"2024-03-06T07:27:52.842617+0000","last_became_peered":"2024-03-06T07:27:52.842617+0000","last_unstale":"2024-03-06T07:27:53.842762+0000","last_undegraded":"2024-03-06T07:27:53.842762+0000","last_fullsized":"2024-03-06T07:27:53.842762+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:14.199266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824426+0000","last_change":"2024-03-06T07:27:52.849788+0000","last_active":"2024-03-06T07:27:53.824426+0000","last_peered":"2024-03-06T07:27:53.824426+0000","last_clean":"2024-03-06T07:27:53.824426+0000","last_became_active":"2024-03-06T07:27:52.848829+0000","last_became_peered":"2024-03-06T07:27:52.848829+0000","last_unstale":"2024-03-06T07:27:53.824426+0000","last_undegraded":"2024-03-06T07:27:53.824426+0000","last_fullsized":"2024-03-06T07:27:53.824426+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:07.418348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227869+0000","last_change":"2024-03-06T07:27:52.844099+0000","last_active":"2024-03-06T07:27:54.227869+0000","last_peered":"2024-03-06T07:27:54.227869+0000","last_clean":"2024-03-06T07:27:54.227869+0000","last_became_active":"2024-03-06T07:27:52.843467+0000","last_became_peered":"2024-03-06T07:27:52.843467+0000","last_unstale":"2024-03-06T07:27:54.227869+0000","last_undegraded":"2024-03-06T07:27:54.227869+0000","last_fullsized":"2024-03-06T07:27:54.227869+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:30.502225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228427+0000","last_change":"2024-03-06T07:27:52.843062+0000","last_active":"2024-03-06T07:27:54.228427+0000","last_peered":"2024-03-06T07:27:54.228427+0000","last_clean":"2024-03-06T07:27:54.228427+0000","last_became_active":"2024-03-06T07:27:52.842694+0000","last_became_peered":"2024-03-06T07:27:52.842694+0000","last_unstale":"2024-03-06T07:27:54.228427+0000","last_undegraded":"2024-03-06T07:27:54.228427+0000","last_fullsized":"2024-03-06T07:27:54.228427+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:57:54.032945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827779+0000","last_change":"2024-03-06T07:27:52.854005+0000","last_active":"2024-03-06T07:27:53.827779+0000","last_peered":"2024-03-06T07:27:53.827779+0000","last_clean":"2024-03-06T07:27:53.827779+0000","last_became_active":"2024-03-06T07:27:52.853490+0000","last_became_peered":"2024-03-06T07:27:52.853490+0000","last_unstale":"2024-03-06T07:27:53.827779+0000","last_undegraded":"2024-03-06T07:27:53.827779+0000","last_fullsized":"2024-03-06T07:27:53.827779+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:08.487751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.828536+0000","last_change":"2024-03-06T07:27:52.848560+0000","last_active":"2024-03-06T07:27:53.828536+0000","last_peered":"2024-03-06T07:27:53.828536+0000","last_clean":"2024-03-06T07:27:53.828536+0000","last_became_active":"2024-03-06T07:27:52.848293+0000","last_became_peered":"2024-03-06T07:27:52.848293+0000","last_unstale":"2024-03-06T07:27:53.828536+0000","last_undegraded":"2024-03-06T07:27:53.828536+0000","last_fullsized":"2024-03-06T07:27:53.828536+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:38.851748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229139+0000","last_change":"2024-03-06T07:27:52.849159+0000","last_active":"2024-03-06T07:27:54.229139+0000","last_peered":"2024-03-06T07:27:54.229139+0000","last_clean":"2024-03-06T07:27:54.229139+0000","last_became_active":"2024-03-06T07:27:52.847083+0000","last_became_peered":"2024-03-06T07:27:52.847083+0000","last_unstale":"2024-03-06T07:27:54.229139+0000","last_undegraded":"2024-03-06T07:27:54.229139+0000","last_fullsized":"2024-03-06T07:27:54.229139+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:49.125933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227349+0000","last_change":"2024-03-06T07:27:52.832481+0000","last_active":"2024-03-06T07:27:54.227349+0000","last_peered":"2024-03-06T07:27:54.227349+0000","last_clean":"2024-03-06T07:27:54.227349+0000","last_became_active":"2024-03-06T07:27:52.832194+0000","last_became_peered":"2024-03-06T07:27:52.832194+0000","last_unstale":"2024-03-06T07:27:54.227349+0000","last_undegraded":"2024-03-06T07:27:54.227349+0000","last_fullsized":"2024-03-06T07:27:54.227349+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:46.848368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116738+0000","last_change":"2024-03-06T07:27:52.851822+0000","last_active":"2024-03-06T07:27:54.116738+0000","last_peered":"2024-03-06T07:27:54.116738+0000","last_clean":"2024-03-06T07:27:54.116738+0000","last_became_active":"2024-03-06T07:27:52.851543+0000","last_became_peered":"2024-03-06T07:27:52.851543+0000","last_unstale":"2024-03-06T07:27:54.116738+0000","last_undegraded":"2024-03-06T07:27:54.116738+0000","last_fullsized":"2024-03-06T07:27:54.116738+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:48.053555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824343+0000","last_change":"2024-03-06T07:27:52.827809+0000","last_active":"2024-03-06T07:27:53.824343+0000","last_peered":"2024-03-06T07:27:53.824343+0000","last_clean":"2024-03-06T07:27:53.824343+0000","last_became_active":"2024-03-06T07:27:52.827502+0000","last_became_peered":"2024-03-06T07:27:52.827502+0000","last_unstale":"2024-03-06T07:27:53.824343+0000","last_undegraded":"2024-03-06T07:27:53.824343+0000","last_fullsized":"2024-03-06T07:27:53.824343+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:51.129075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116711+0000","last_change":"2024-03-06T07:27:52.847729+0000","last_active":"2024-03-06T07:27:54.116711+0000","last_peered":"2024-03-06T07:27:54.116711+0000","last_clean":"2024-03-06T07:27:54.116711+0000","last_became_active":"2024-03-06T07:27:52.847452+0000","last_became_peered":"2024-03-06T07:27:52.847452+0000","last_unstale":"2024-03-06T07:27:54.116711+0000","last_undegraded":"2024-03-06T07:27:54.116711+0000","last_fullsized":"2024-03-06T07:27:54.116711+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:02.639317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827559+0000","last_change":"2024-03-06T07:27:52.843112+0000","last_active":"2024-03-06T07:27:53.827559+0000","last_peered":"2024-03-06T07:27:53.827559+0000","last_clean":"2024-03-06T07:27:53.827559+0000","last_became_active":"2024-03-06T07:27:52.842787+0000","last_became_peered":"2024-03-06T07:27:52.842787+0000","last_unstale":"2024-03-06T07:27:53.827559+0000","last_undegraded":"2024-03-06T07:27:53.827559+0000","last_fullsized":"2024-03-06T07:27:53.827559+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:35.725347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"182'4","reported_seq":18,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824576+0000","last_change":"2024-03-06T07:27:51.829598+0000","last_active":"2024-03-06T07:27:53.824576+0000","last_peered":"2024-03-06T07:27:53.824576+0000","last_clean":"2024-03-06T07:27:53.824576+0000","last_became_active":"2024-03-06T07:27:51.829308+0000","last_became_peered":"2024-03-06T07:27:51.829308+0000","last_unstale":"2024-03-06T07:27:53.824576+0000","last_undegraded":"2024-03-06T07:27:53.824576+0000","last_fullsized":"2024-03-06T07:27:53.824576+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:18.568415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827622+0000","last_change":"2024-03-06T07:27:52.849160+0000","last_active":"2024-03-06T07:27:53.827622+0000","last_peered":"2024-03-06T07:27:53.827622+0000","last_clean":"2024-03-06T07:27:53.827622+0000","last_became_active":"2024-03-06T07:27:52.848836+0000","last_became_peered":"2024-03-06T07:27:52.848836+0000","last_unstale":"2024-03-06T07:27:53.827622+0000","last_undegraded":"2024-03-06T07:27:53.827622+0000","last_fullsized":"2024-03-06T07:27:53.827622+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:27.108826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"182'1","reported_seq":15,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229080+0000","last_change":"2024-03-06T07:27:51.830745+0000","last_active":"2024-03-06T07:27:54.229080+0000","last_peered":"2024-03-06T07:27:54.229080+0000","last_clean":"2024-03-06T07:27:54.229080+0000","last_became_active":"2024-03-06T07:27:51.830490+0000","last_became_peered":"2024-03-06T07:27:51.830490+0000","last_unstale":"2024-03-06T07:27:54.229080+0000","last_undegraded":"2024-03-06T07:27:54.229080+0000","last_fullsized":"2024-03-06T07:27:54.229080+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:40.184591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228894+0000","last_change":"2024-03-06T07:27:52.845892+0000","last_active":"2024-03-06T07:27:54.228894+0000","last_peered":"2024-03-06T07:27:54.228894+0000","last_clean":"2024-03-06T07:27:54.228894+0000","last_became_active":"2024-03-06T07:27:52.845611+0000","last_became_peered":"2024-03-06T07:27:52.845611+0000","last_unstale":"2024-03-06T07:27:54.228894+0000","last_undegraded":"2024-03-06T07:27:54.228894+0000","last_fullsized":"2024-03-06T07:27:54.228894+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:33.742320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1d","version":"182'2","reported_seq":16,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116786+0000","last_change":"2024-03-06T07:27:51.827920+0000","last_active":"2024-03-06T07:27:54.116786+0000","last_peered":"2024-03-06T07:27:54.116786+0000","last_clean":"2024-03-06T07:27:54.116786+0000","last_became_active":"2024-03-06T07:27:51.827650+0000","last_became_peered":"2024-03-06T07:27:51.827650+0000","last_unstale":"2024-03-06T07:27:54.116786+0000","last_undegraded":"2024-03-06T07:27:54.116786+0000","last_fullsized":"2024-03-06T07:27:54.116786+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:07.649833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827224+0000","last_change":"2024-03-06T07:27:52.841717+0000","last_active":"2024-03-06T07:27:53.827224+0000","last_peered":"2024-03-06T07:27:53.827224+0000","last_clean":"2024-03-06T07:27:53.827224+0000","last_became_active":"2024-03-06T07:27:52.841440+0000","last_became_peered":"2024-03-06T07:27:52.841440+0000","last_unstale":"2024-03-06T07:27:53.827224+0000","last_undegraded":"2024-03-06T07:27:53.827224+0000","last_fullsized":"2024-03-06T07:27:53.827224+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:48.085929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1b","version":"182'1","reported_seq":15,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842773+0000","last_change":"2024-03-06T07:27:51.830565+0000","last_active":"2024-03-06T07:27:53.842773+0000","last_peered":"2024-03-06T07:27:53.842773+0000","last_clean":"2024-03-06T07:27:53.842773+0000","last_became_active":"2024-03-06T07:27:51.830283+0000","last_became_peered":"2024-03-06T07:27:51.830283+0000","last_unstale":"2024-03-06T07:27:53.842773+0000","last_undegraded":"2024-03-06T07:27:53.842773+0000","last_fullsized":"2024-03-06T07:27:53.842773+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:23.893421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227905+0000","last_change":"2024-03-06T07:27:52.851487+0000","last_active":"2024-03-06T07:27:54.227905+0000","last_peered":"2024-03-06T07:27:54.227905+0000","last_clean":"2024-03-06T07:27:54.227905+0000","last_became_active":"2024-03-06T07:27:52.851074+0000","last_became_peered":"2024-03-06T07:27:52.851074+0000","last_unstale":"2024-03-06T07:27:54.227905+0000","last_undegraded":"2024-03-06T07:27:54.227905+0000","last_fullsized":"2024-03-06T07:27:54.227905+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:04.893618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"182'1","reported_seq":15,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227974+0000","last_change":"2024-03-06T07:27:51.832998+0000","last_active":"2024-03-06T07:27:54.227974+0000","last_peered":"2024-03-06T07:27:54.227974+0000","last_clean":"2024-03-06T07:27:54.227974+0000","last_became_active":"2024-03-06T07:27:51.832605+0000","last_became_peered":"2024-03-06T07:27:51.832605+0000","last_unstale":"2024-03-06T07:27:54.227974+0000","last_undegraded":"2024-03-06T07:27:54.227974+0000","last_fullsized":"2024-03-06T07:27:54.227974+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:57.043951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842605+0000","last_change":"2024-03-06T07:27:52.842376+0000","last_active":"2024-03-06T07:27:53.842605+0000","last_peered":"2024-03-06T07:27:53.842605+0000","last_clean":"2024-03-06T07:27:53.842605+0000","last_became_active":"2024-03-06T07:27:52.841989+0000","last_became_peered":"2024-03-06T07:27:52.841989+0000","last_unstale":"2024-03-06T07:27:53.842605+0000","last_undegraded":"2024-03-06T07:27:53.842605+0000","last_fullsized":"2024-03-06T07:27:53.842605+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:02.906107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.828387+0000","last_change":"2024-03-06T07:27:51.835979+0000","last_active":"2024-03-06T07:27:53.828387+0000","last_peered":"2024-03-06T07:27:53.828387+0000","last_clean":"2024-03-06T07:27:53.828387+0000","last_became_active":"2024-03-06T07:27:51.835707+0000","last_became_peered":"2024-03-06T07:27:51.835707+0000","last_unstale":"2024-03-06T07:27:53.828387+0000","last_undegraded":"2024-03-06T07:27:53.828387+0000","last_fullsized":"2024-03-06T07:27:53.828387+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:25.277737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824512+0000","last_change":"2024-03-06T07:27:52.836761+0000","last_active":"2024-03-06T07:27:53.824512+0000","last_peered":"2024-03-06T07:27:53.824512+0000","last_clean":"2024-03-06T07:27:53.824512+0000","last_became_active":"2024-03-06T07:27:52.836464+0000","last_became_peered":"2024-03-06T07:27:52.836464+0000","last_unstale":"2024-03-06T07:27:53.824512+0000","last_undegraded":"2024-03-06T07:27:53.824512+0000","last_fullsized":"2024-03-06T07:27:53.824512+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:21.419139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827332+0000","last_change":"2024-03-06T07:27:51.832162+0000","last_active":"2024-03-06T07:27:53.827332+0000","last_peered":"2024-03-06T07:27:53.827332+0000","last_clean":"2024-03-06T07:27:53.827332+0000","last_became_active":"2024-03-06T07:27:51.831873+0000","last_became_peered":"2024-03-06T07:27:51.831873+0000","last_unstale":"2024-03-06T07:27:53.827332+0000","last_undegraded":"2024-03-06T07:27:53.827332+0000","last_fullsized":"2024-03-06T07:27:53.827332+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:45.844161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842775+0000","last_change":"2024-03-06T07:27:52.847447+0000","last_active":"2024-03-06T07:27:53.842775+0000","last_peered":"2024-03-06T07:27:53.842775+0000","last_clean":"2024-03-06T07:27:53.842775+0000","last_became_active":"2024-03-06T07:27:52.847129+0000","last_became_peered":"2024-03-06T07:27:52.847129+0000","last_unstale":"2024-03-06T07:27:53.842775+0000","last_undegraded":"2024-03-06T07:27:53.842775+0000","last_fullsized":"2024-03-06T07:27:53.842775+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:57:54.032945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228630+0000","last_change":"2024-03-06T07:27:51.832113+0000","last_active":"2024-03-06T07:27:54.228630+0000","last_peered":"2024-03-06T07:27:54.228630+0000","last_clean":"2024-03-06T07:27:54.228630+0000","last_became_active":"2024-03-06T07:27:51.831777+0000","last_became_peered":"2024-03-06T07:27:51.831777+0000","last_unstale":"2024-03-06T07:27:54.228630+0000","last_undegraded":"2024-03-06T07:27:54.228630+0000","last_fullsized":"2024-03-06T07:27:54.228630+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:26.154047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842954+0000","last_change":"2024-03-06T07:27:52.847490+0000","last_active":"2024-03-06T07:27:53.842954+0000","last_peered":"2024-03-06T07:27:53.842954+0000","last_clean":"2024-03-06T07:27:53.842954+0000","last_became_active":"2024-03-06T07:27:52.846888+0000","last_became_peered":"2024-03-06T07:27:52.846888+0000","last_unstale":"2024-03-06T07:27:53.842954+0000","last_undegraded":"2024-03-06T07:27:53.842954+0000","last_fullsized":"2024-03-06T07:27:53.842954+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:01.915969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.828380+0000","last_change":"2024-03-06T07:27:51.833447+0000","last_active":"2024-03-06T07:27:53.828380+0000","last_peered":"2024-03-06T07:27:53.828380+0000","last_clean":"2024-03-06T07:27:53.828380+0000","last_became_active":"2024-03-06T07:27:51.833173+0000","last_became_peered":"2024-03-06T07:27:51.833173+0000","last_unstale":"2024-03-06T07:27:53.828380+0000","last_undegraded":"2024-03-06T07:27:53.828380+0000","last_fullsized":"2024-03-06T07:27:53.828380+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:04.569067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827836+0000","last_change":"2024-03-06T07:27:52.838853+0000","last_active":"2024-03-06T07:27:53.827836+0000","last_peered":"2024-03-06T07:27:53.827836+0000","last_clean":"2024-03-06T07:27:53.827836+0000","last_became_active":"2024-03-06T07:27:52.838528+0000","last_became_peered":"2024-03-06T07:27:52.838528+0000","last_unstale":"2024-03-06T07:27:53.827836+0000","last_undegraded":"2024-03-06T07:27:53.827836+0000","last_fullsized":"2024-03-06T07:27:53.827836+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:59.780024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"182'3","reported_seq":17,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228515+0000","last_change":"2024-03-06T07:27:51.832456+0000","last_active":"2024-03-06T07:27:54.228515+0000","last_peered":"2024-03-06T07:27:54.228515+0000","last_clean":"2024-03-06T07:27:54.228515+0000","last_became_active":"2024-03-06T07:27:51.832176+0000","last_became_peered":"2024-03-06T07:27:51.832176+0000","last_unstale":"2024-03-06T07:27:54.228515+0000","last_undegraded":"2024-03-06T07:27:54.228515+0000","last_fullsized":"2024-03-06T07:27:54.228515+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:10.808514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.828607+0000","last_change":"2024-03-06T07:27:52.831146+0000","last_active":"2024-03-06T07:27:53.828607+0000","last_peered":"2024-03-06T07:27:53.828607+0000","last_clean":"2024-03-06T07:27:53.828607+0000","last_became_active":"2024-03-06T07:27:52.830862+0000","last_became_peered":"2024-03-06T07:27:52.830862+0000","last_unstale":"2024-03-06T07:27:53.828607+0000","last_undegraded":"2024-03-06T07:27:53.828607+0000","last_fullsized":"2024-03-06T07:27:53.828607+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:09.978274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.14","version":"182'6","reported_seq":20,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116926+0000","last_change":"2024-03-06T07:27:51.830005+0000","last_active":"2024-03-06T07:27:54.116926+0000","last_peered":"2024-03-06T07:27:54.116926+0000","last_clean":"2024-03-06T07:27:54.116926+0000","last_became_active":"2024-03-06T07:27:51.829747+0000","last_became_peered":"2024-03-06T07:27:51.829747+0000","last_unstale":"2024-03-06T07:27:54.116926+0000","last_undegraded":"2024-03-06T07:27:54.116926+0000","last_fullsized":"2024-03-06T07:27:54.116926+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:37.847541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227137+0000","last_change":"2024-03-06T07:27:52.833504+0000","last_active":"2024-03-06T07:27:54.227137+0000","last_peered":"2024-03-06T07:27:54.227137+0000","last_clean":"2024-03-06T07:27:54.227137+0000","last_became_active":"2024-03-06T07:27:52.833228+0000","last_became_peered":"2024-03-06T07:27:52.833228+0000","last_unstale":"2024-03-06T07:27:54.227137+0000","last_undegraded":"2024-03-06T07:27:54.227137+0000","last_fullsized":"2024-03-06T07:27:54.227137+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:35.655688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"182'2","reported_seq":16,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228970+0000","last_change":"2024-03-06T07:27:51.830207+0000","last_active":"2024-03-06T07:27:54.228970+0000","last_peered":"2024-03-06T07:27:54.228970+0000","last_clean":"2024-03-06T07:27:54.228970+0000","last_became_active":"2024-03-06T07:27:51.829897+0000","last_became_peered":"2024-03-06T07:27:51.829897+0000","last_unstale":"2024-03-06T07:27:54.228970+0000","last_undegraded":"2024-03-06T07:27:54.228970+0000","last_fullsized":"2024-03-06T07:27:54.228970+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:14.460825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228493+0000","last_change":"2024-03-06T07:27:52.845627+0000","last_active":"2024-03-06T07:27:54.228493+0000","last_peered":"2024-03-06T07:27:54.228493+0000","last_clean":"2024-03-06T07:27:54.228493+0000","last_became_active":"2024-03-06T07:27:52.845234+0000","last_became_peered":"2024-03-06T07:27:52.845234+0000","last_unstale":"2024-03-06T07:27:54.228493+0000","last_undegraded":"2024-03-06T07:27:54.228493+0000","last_fullsized":"2024-03-06T07:27:54.228493+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:56.539951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227189+0000","last_change":"2024-03-06T07:27:51.822548+0000","last_active":"2024-03-06T07:27:54.227189+0000","last_peered":"2024-03-06T07:27:54.227189+0000","last_clean":"2024-03-06T07:27:54.227189+0000","last_became_active":"2024-03-06T07:27:51.822262+0000","last_became_peered":"2024-03-06T07:27:51.822262+0000","last_unstale":"2024-03-06T07:27:54.227189+0000","last_undegraded":"2024-03-06T07:27:54.227189+0000","last_fullsized":"2024-03-06T07:27:54.227189+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:16.943635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827425+0000","last_change":"2024-03-06T07:27:52.848419+0000","last_active":"2024-03-06T07:27:53.827425+0000","last_peered":"2024-03-06T07:27:53.827425+0000","last_clean":"2024-03-06T07:27:53.827425+0000","last_became_active":"2024-03-06T07:27:52.846559+0000","last_became_peered":"2024-03-06T07:27:52.846559+0000","last_unstale":"2024-03-06T07:27:53.827425+0000","last_undegraded":"2024-03-06T07:27:53.827425+0000","last_fullsized":"2024-03-06T07:27:53.827425+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:00.807566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116474+0000","last_change":"2024-03-06T07:27:51.833477+0000","last_active":"2024-03-06T07:27:54.116474+0000","last_peered":"2024-03-06T07:27:54.116474+0000","last_clean":"2024-03-06T07:27:54.116474+0000","last_became_active":"2024-03-06T07:27:51.833189+0000","last_became_peered":"2024-03-06T07:27:51.833189+0000","last_unstale":"2024-03-06T07:27:54.116474+0000","last_undegraded":"2024-03-06T07:27:54.116474+0000","last_fullsized":"2024-03-06T07:27:54.116474+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:32.738113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842738+0000","last_change":"2024-03-06T07:27:52.842524+0000","last_active":"2024-03-06T07:27:53.842738+0000","last_peered":"2024-03-06T07:27:53.842738+0000","last_clean":"2024-03-06T07:27:53.842738+0000","last_became_active":"2024-03-06T07:27:52.842171+0000","last_became_peered":"2024-03-06T07:27:52.842171+0000","last_unstale":"2024-03-06T07:27:53.842738+0000","last_undegraded":"2024-03-06T07:27:53.842738+0000","last_fullsized":"2024-03-06T07:27:53.842738+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:21.929558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"182'8","reported_seq":22,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228065+0000","last_change":"2024-03-06T07:27:51.836500+0000","last_active":"2024-03-06T07:27:54.228065+0000","last_peered":"2024-03-06T07:27:54.228065+0000","last_clean":"2024-03-06T07:27:54.228065+0000","last_became_active":"2024-03-06T07:27:51.836156+0000","last_became_peered":"2024-03-06T07:27:51.836156+0000","last_unstale":"2024-03-06T07:27:54.228065+0000","last_undegraded":"2024-03-06T07:27:54.228065+0000","last_fullsized":"2024-03-06T07:27:54.228065+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:41.727365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227314+0000","last_change":"2024-03-06T07:27:52.851096+0000","last_active":"2024-03-06T07:27:54.227314+0000","last_peered":"2024-03-06T07:27:54.227314+0000","last_clean":"2024-03-06T07:27:54.227314+0000","last_became_active":"2024-03-06T07:27:52.850775+0000","last_became_peered":"2024-03-06T07:27:52.850775+0000","last_unstale":"2024-03-06T07:27:54.227314+0000","last_undegraded":"2024-03-06T07:27:54.227314+0000","last_fullsized":"2024-03-06T07:27:54.227314+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:12.204157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"182'1","reported_seq":15,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827751+0000","last_change":"2024-03-06T07:27:51.831464+0000","last_active":"2024-03-06T07:27:53.827751+0000","last_peered":"2024-03-06T07:27:53.827751+0000","last_clean":"2024-03-06T07:27:53.827751+0000","last_became_active":"2024-03-06T07:27:51.831197+0000","last_became_peered":"2024-03-06T07:27:51.831197+0000","last_unstale":"2024-03-06T07:27:53.827751+0000","last_undegraded":"2024-03-06T07:27:53.827751+0000","last_fullsized":"2024-03-06T07:27:53.827751+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:03.171463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228399+0000","last_change":"2024-03-06T07:27:52.837210+0000","last_active":"2024-03-06T07:27:54.228399+0000","last_peered":"2024-03-06T07:27:54.228399+0000","last_clean":"2024-03-06T07:27:54.228399+0000","last_became_active":"2024-03-06T07:27:52.836950+0000","last_became_peered":"2024-03-06T07:27:52.836950+0000","last_unstale":"2024-03-06T07:27:54.228399+0000","last_undegraded":"2024-03-06T07:27:54.228399+0000","last_fullsized":"2024-03-06T07:27:54.228399+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:21.929558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.e","version":"182'2","reported_seq":16,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824397+0000","last_change":"2024-03-06T07:27:51.828775+0000","last_active":"2024-03-06T07:27:53.824397+0000","last_peered":"2024-03-06T07:27:53.824397+0000","last_clean":"2024-03-06T07:27:53.824397+0000","last_became_active":"2024-03-06T07:27:51.828483+0000","last_became_peered":"2024-03-06T07:27:51.828483+0000","last_unstale":"2024-03-06T07:27:53.824397+0000","last_undegraded":"2024-03-06T07:27:53.824397+0000","last_fullsized":"2024-03-06T07:27:53.824397+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:14.011370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827853+0000","last_change":"2024-03-06T07:27:52.845785+0000","last_active":"2024-03-06T07:27:53.827853+0000","last_peered":"2024-03-06T07:27:53.827853+0000","last_clean":"2024-03-06T07:27:53.827853+0000","last_became_active":"2024-03-06T07:27:52.845523+0000","last_became_peered":"2024-03-06T07:27:52.845523+0000","last_unstale":"2024-03-06T07:27:53.827853+0000","last_undegraded":"2024-03-06T07:27:53.827853+0000","last_fullsized":"2024-03-06T07:27:53.827853+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:44.197118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"182'1","reported_seq":15,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228897+0000","last_change":"2024-03-06T07:27:51.828077+0000","last_active":"2024-03-06T07:27:54.228897+0000","last_peered":"2024-03-06T07:27:54.228897+0000","last_clean":"2024-03-06T07:27:54.228897+0000","last_became_active":"2024-03-06T07:27:51.827762+0000","last_became_peered":"2024-03-06T07:27:51.827762+0000","last_unstale":"2024-03-06T07:27:54.228897+0000","last_undegraded":"2024-03-06T07:27:54.228897+0000","last_fullsized":"2024-03-06T07:27:54.228897+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:44.093607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229327+0000","last_change":"2024-03-06T07:27:52.850839+0000","last_active":"2024-03-06T07:27:54.229327+0000","last_peered":"2024-03-06T07:27:54.229327+0000","last_clean":"2024-03-06T07:27:54.229327+0000","last_became_active":"2024-03-06T07:27:52.850576+0000","last_became_peered":"2024-03-06T07:27:52.850576+0000","last_unstale":"2024-03-06T07:27:54.229327+0000","last_undegraded":"2024-03-06T07:27:54.229327+0000","last_fullsized":"2024-03-06T07:27:54.229327+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:59.783306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229199+0000","last_change":"2024-03-06T07:27:51.828930+0000","last_active":"2024-03-06T07:27:54.229199+0000","last_peered":"2024-03-06T07:27:54.229199+0000","last_clean":"2024-03-06T07:27:54.229199+0000","last_became_active":"2024-03-06T07:27:51.828653+0000","last_became_peered":"2024-03-06T07:27:51.828653+0000","last_unstale":"2024-03-06T07:27:54.229199+0000","last_undegraded":"2024-03-06T07:27:54.229199+0000","last_fullsized":"2024-03-06T07:27:54.229199+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:39.081353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227932+0000","last_change":"2024-03-06T07:27:52.851231+0000","last_active":"2024-03-06T07:27:54.227932+0000","last_peered":"2024-03-06T07:27:54.227932+0000","last_clean":"2024-03-06T07:27:54.227932+0000","last_became_active":"2024-03-06T07:27:52.850951+0000","last_became_peered":"2024-03-06T07:27:52.850951+0000","last_unstale":"2024-03-06T07:27:54.227932+0000","last_undegraded":"2024-03-06T07:27:54.227932+0000","last_fullsized":"2024-03-06T07:27:54.227932+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:04.131892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"182'2","reported_seq":16,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229064+0000","last_change":"2024-03-06T07:27:51.830105+0000","last_active":"2024-03-06T07:27:54.229064+0000","last_peered":"2024-03-06T07:27:54.229064+0000","last_clean":"2024-03-06T07:27:54.229064+0000","last_became_active":"2024-03-06T07:27:51.829820+0000","last_became_peered":"2024-03-06T07:27:51.829820+0000","last_unstale":"2024-03-06T07:27:54.229064+0000","last_undegraded":"2024-03-06T07:27:54.229064+0000","last_fullsized":"2024-03-06T07:27:54.229064+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:43.256668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227999+0000","last_change":"2024-03-06T07:27:52.843200+0000","last_active":"2024-03-06T07:27:54.227999+0000","last_peered":"2024-03-06T07:27:54.227999+0000","last_clean":"2024-03-06T07:27:54.227999+0000","last_became_active":"2024-03-06T07:27:52.842838+0000","last_became_peered":"2024-03-06T07:27:52.842838+0000","last_unstale":"2024-03-06T07:27:54.227999+0000","last_undegraded":"2024-03-06T07:27:54.227999+0000","last_fullsized":"2024-03-06T07:27:54.227999+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:31.323213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842712+0000","last_change":"2024-03-06T07:27:51.831216+0000","last_active":"2024-03-06T07:27:53.842712+0000","last_peered":"2024-03-06T07:27:53.842712+0000","last_clean":"2024-03-06T07:27:53.842712+0000","last_became_active":"2024-03-06T07:27:51.830947+0000","last_became_peered":"2024-03-06T07:27:51.830947+0000","last_unstale":"2024-03-06T07:27:53.842712+0000","last_undegraded":"2024-03-06T07:27:53.842712+0000","last_fullsized":"2024-03-06T07:27:53.842712+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:29.394867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227231+0000","last_change":"2024-03-06T07:27:52.851117+0000","last_active":"2024-03-06T07:27:54.227231+0000","last_peered":"2024-03-06T07:27:54.227231+0000","last_clean":"2024-03-06T07:27:54.227231+0000","last_became_active":"2024-03-06T07:27:52.850801+0000","last_became_peered":"2024-03-06T07:27:52.850801+0000","last_unstale":"2024-03-06T07:27:54.227231+0000","last_undegraded":"2024-03-06T07:27:54.227231+0000","last_fullsized":"2024-03-06T07:27:54.227231+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:50.603524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842581+0000","last_change":"2024-03-06T07:27:51.832578+0000","last_active":"2024-03-06T07:27:53.842581+0000","last_peered":"2024-03-06T07:27:53.842581+0000","last_clean":"2024-03-06T07:27:53.842581+0000","last_became_active":"2024-03-06T07:27:51.832234+0000","last_became_peered":"2024-03-06T07:27:51.832234+0000","last_unstale":"2024-03-06T07:27:53.842581+0000","last_undegraded":"2024-03-06T07:27:53.842581+0000","last_fullsized":"2024-03-06T07:27:53.842581+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:10.808514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827527+0000","last_change":"2024-03-06T07:27:52.852438+0000","last_active":"2024-03-06T07:27:53.827527+0000","last_peered":"2024-03-06T07:27:53.827527+0000","last_clean":"2024-03-06T07:27:53.827527+0000","last_became_active":"2024-03-06T07:27:52.851964+0000","last_became_peered":"2024-03-06T07:27:52.851964+0000","last_unstale":"2024-03-06T07:27:53.827527+0000","last_undegraded":"2024-03-06T07:27:53.827527+0000","last_fullsized":"2024-03-06T07:27:53.827527+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:28.314013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824174+0000","last_change":"2024-03-06T07:27:51.829009+0000","last_active":"2024-03-06T07:27:53.824174+0000","last_peered":"2024-03-06T07:27:53.824174+0000","last_clean":"2024-03-06T07:27:53.824174+0000","last_became_active":"2024-03-06T07:27:51.828505+0000","last_became_peered":"2024-03-06T07:27:51.828505+0000","last_unstale":"2024-03-06T07:27:53.824174+0000","last_undegraded":"2024-03-06T07:27:53.824174+0000","last_fullsized":"2024-03-06T07:27:53.824174+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:03.559231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227324+0000","last_change":"2024-03-06T07:27:52.827928+0000","last_active":"2024-03-06T07:27:54.227324+0000","last_peered":"2024-03-06T07:27:54.227324+0000","last_clean":"2024-03-06T07:27:54.227324+0000","last_became_active":"2024-03-06T07:27:52.827635+0000","last_became_peered":"2024-03-06T07:27:52.827635+0000","last_unstale":"2024-03-06T07:27:54.227324+0000","last_undegraded":"2024-03-06T07:27:54.227324+0000","last_fullsized":"2024-03-06T07:27:54.227324+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:09.730573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.7","version":"182'2","reported_seq":16,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229256+0000","last_change":"2024-03-06T07:27:51.825754+0000","last_active":"2024-03-06T07:27:54.229256+0000","last_peered":"2024-03-06T07:27:54.229256+0000","last_clean":"2024-03-06T07:27:54.229256+0000","last_became_active":"2024-03-06T07:27:51.825459+0000","last_became_peered":"2024-03-06T07:27:51.825459+0000","last_unstale":"2024-03-06T07:27:54.229256+0000","last_undegraded":"2024-03-06T07:27:54.229256+0000","last_fullsized":"2024-03-06T07:27:54.229256+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:41.300428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228483+0000","last_change":"2024-03-06T07:27:52.843802+0000","last_active":"2024-03-06T07:27:54.228483+0000","last_peered":"2024-03-06T07:27:54.228483+0000","last_clean":"2024-03-06T07:27:54.228483+0000","last_became_active":"2024-03-06T07:27:52.843472+0000","last_became_peered":"2024-03-06T07:27:52.843472+0000","last_unstale":"2024-03-06T07:27:54.228483+0000","last_undegraded":"2024-03-06T07:27:54.228483+0000","last_fullsized":"2024-03-06T07:27:54.228483+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:01.915969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"183'1","reported_seq":15,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.844256+0000","last_change":"2024-03-06T07:27:51.834543+0000","last_active":"2024-03-06T07:27:53.844256+0000","last_peered":"2024-03-06T07:27:53.844256+0000","last_clean":"2024-03-06T07:27:53.844256+0000","last_became_active":"2024-03-06T07:27:51.834294+0000","last_became_peered":"2024-03-06T07:27:51.834294+0000","last_unstale":"2024-03-06T07:27:53.844256+0000","last_undegraded":"2024-03-06T07:27:53.844256+0000","last_fullsized":"2024-03-06T07:27:53.844256+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:11.425236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116961+0000","last_change":"2024-03-06T07:27:52.847982+0000","last_active":"2024-03-06T07:27:54.116961+0000","last_peered":"2024-03-06T07:27:54.116961+0000","last_clean":"2024-03-06T07:27:54.116961+0000","last_became_active":"2024-03-06T07:27:52.847477+0000","last_became_peered":"2024-03-06T07:27:52.847477+0000","last_unstale":"2024-03-06T07:27:54.116961+0000","last_undegraded":"2024-03-06T07:27:54.116961+0000","last_fullsized":"2024-03-06T07:27:54.116961+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:18.817690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842959+0000","last_change":"2024-03-06T07:27:51.825939+0000","last_active":"2024-03-06T07:27:53.842959+0000","last_peered":"2024-03-06T07:27:53.842959+0000","last_clean":"2024-03-06T07:27:53.842959+0000","last_became_active":"2024-03-06T07:27:51.825569+0000","last_became_peered":"2024-03-06T07:27:51.825569+0000","last_unstale":"2024-03-06T07:27:53.842959+0000","last_undegraded":"2024-03-06T07:27:53.842959+0000","last_fullsized":"2024-03-06T07:27:53.842959+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:17.071666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824502+0000","last_change":"2024-03-06T07:27:52.828053+0000","last_active":"2024-03-06T07:27:53.824502+0000","last_peered":"2024-03-06T07:27:53.824502+0000","last_clean":"2024-03-06T07:27:53.824502+0000","last_became_active":"2024-03-06T07:27:52.827582+0000","last_became_peered":"2024-03-06T07:27:52.827582+0000","last_unstale":"2024-03-06T07:27:53.824502+0000","last_undegraded":"2024-03-06T07:27:53.824502+0000","last_fullsized":"2024-03-06T07:27:53.824502+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:36.756407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229210+0000","last_change":"2024-03-06T07:27:51.829977+0000","last_active":"2024-03-06T07:27:54.229210+0000","last_peered":"2024-03-06T07:27:54.229210+0000","last_clean":"2024-03-06T07:27:54.229210+0000","last_became_active":"2024-03-06T07:27:51.829718+0000","last_became_peered":"2024-03-06T07:27:51.829718+0000","last_unstale":"2024-03-06T07:27:54.229210+0000","last_undegraded":"2024-03-06T07:27:54.229210+0000","last_fullsized":"2024-03-06T07:27:54.229210+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:59.505668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827316+0000","last_change":"2024-03-06T07:27:52.853302+0000","last_active":"2024-03-06T07:27:53.827316+0000","last_peered":"2024-03-06T07:27:53.827316+0000","last_clean":"2024-03-06T07:27:53.827316+0000","last_became_active":"2024-03-06T07:27:52.852745+0000","last_became_peered":"2024-03-06T07:27:52.852745+0000","last_unstale":"2024-03-06T07:27:53.827316+0000","last_undegraded":"2024-03-06T07:27:53.827316+0000","last_fullsized":"2024-03-06T07:27:53.827316+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:03.769930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229136+0000","last_change":"2024-03-06T07:27:51.828299+0000","last_active":"2024-03-06T07:27:54.229136+0000","last_peered":"2024-03-06T07:27:54.229136+0000","last_clean":"2024-03-06T07:27:54.229136+0000","last_became_active":"2024-03-06T07:27:51.827999+0000","last_became_peered":"2024-03-06T07:27:51.827999+0000","last_unstale":"2024-03-06T07:27:54.229136+0000","last_undegraded":"2024-03-06T07:27:54.229136+0000","last_fullsized":"2024-03-06T07:27:54.229136+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:14:58.361793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827379+0000","last_change":"2024-03-06T07:27:52.840395+0000","last_active":"2024-03-06T07:27:53.827379+0000","last_peered":"2024-03-06T07:27:53.827379+0000","last_clean":"2024-03-06T07:27:53.827379+0000","last_became_active":"2024-03-06T07:27:52.840117+0000","last_became_peered":"2024-03-06T07:27:52.840117+0000","last_unstale":"2024-03-06T07:27:53.827379+0000","last_undegraded":"2024-03-06T07:27:53.827379+0000","last_fullsized":"2024-03-06T07:27:53.827379+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:26.160506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.828367+0000","last_change":"2024-03-06T07:27:51.830896+0000","last_active":"2024-03-06T07:27:53.828367+0000","last_peered":"2024-03-06T07:27:53.828367+0000","last_clean":"2024-03-06T07:27:53.828367+0000","last_became_active":"2024-03-06T07:27:51.830630+0000","last_became_peered":"2024-03-06T07:27:51.830630+0000","last_unstale":"2024-03-06T07:27:53.828367+0000","last_undegraded":"2024-03-06T07:27:53.828367+0000","last_fullsized":"2024-03-06T07:27:53.828367+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:53.349144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824496+0000","last_change":"2024-03-06T07:27:52.831089+0000","last_active":"2024-03-06T07:27:53.824496+0000","last_peered":"2024-03-06T07:27:53.824496+0000","last_clean":"2024-03-06T07:27:53.824496+0000","last_became_active":"2024-03-06T07:27:52.830222+0000","last_became_peered":"2024-03-06T07:27:52.830222+0000","last_unstale":"2024-03-06T07:27:53.824496+0000","last_undegraded":"2024-03-06T07:27:53.824496+0000","last_fullsized":"2024-03-06T07:27:53.824496+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:11.268762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228227+0000","last_change":"2024-03-06T07:27:51.834741+0000","last_active":"2024-03-06T07:27:54.228227+0000","last_peered":"2024-03-06T07:27:54.228227+0000","last_clean":"2024-03-06T07:27:54.228227+0000","last_became_active":"2024-03-06T07:27:51.833107+0000","last_became_peered":"2024-03-06T07:27:51.833107+0000","last_unstale":"2024-03-06T07:27:54.228227+0000","last_undegraded":"2024-03-06T07:27:54.228227+0000","last_fullsized":"2024-03-06T07:27:54.228227+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:51.680814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228937+0000","last_change":"2024-03-06T07:27:52.850358+0000","last_active":"2024-03-06T07:27:54.228937+0000","last_peered":"2024-03-06T07:27:54.228937+0000","last_clean":"2024-03-06T07:27:54.228937+0000","last_became_active":"2024-03-06T07:27:52.850040+0000","last_became_peered":"2024-03-06T07:27:52.850040+0000","last_unstale":"2024-03-06T07:27:54.228937+0000","last_undegraded":"2024-03-06T07:27:54.228937+0000","last_fullsized":"2024-03-06T07:27:54.228937+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:08.654039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228178+0000","last_change":"2024-03-06T07:27:51.833317+0000","last_active":"2024-03-06T07:27:54.228178+0000","last_peered":"2024-03-06T07:27:54.228178+0000","last_clean":"2024-03-06T07:27:54.228178+0000","last_became_active":"2024-03-06T07:27:51.828026+0000","last_became_peered":"2024-03-06T07:27:51.828026+0000","last_unstale":"2024-03-06T07:27:54.228178+0000","last_undegraded":"2024-03-06T07:27:54.228178+0000","last_fullsized":"2024-03-06T07:27:54.228178+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:09.527421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827701+0000","last_change":"2024-03-06T07:27:52.851417+0000","last_active":"2024-03-06T07:27:53.827701+0000","last_peered":"2024-03-06T07:27:53.827701+0000","last_clean":"2024-03-06T07:27:53.827701+0000","last_became_active":"2024-03-06T07:27:52.851110+0000","last_became_peered":"2024-03-06T07:27:52.851110+0000","last_unstale":"2024-03-06T07:27:53.827701+0000","last_undegraded":"2024-03-06T07:27:53.827701+0000","last_fullsized":"2024-03-06T07:27:53.827701+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:22.148483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1c","version":"182'2","reported_seq":16,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228051+0000","last_change":"2024-03-06T07:27:51.829812+0000","last_active":"2024-03-06T07:27:54.228051+0000","last_peered":"2024-03-06T07:27:54.228051+0000","last_clean":"2024-03-06T07:27:54.228051+0000","last_became_active":"2024-03-06T07:27:51.829502+0000","last_became_peered":"2024-03-06T07:27:51.829502+0000","last_unstale":"2024-03-06T07:27:54.228051+0000","last_undegraded":"2024-03-06T07:27:54.228051+0000","last_fullsized":"2024-03-06T07:27:54.228051+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:20.358143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":483,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227820+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:27:54.227820+0000","last_peered":"2024-03-06T07:27:54.227820+0000","last_clean":"2024-03-06T07:27:54.227820+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:27:54.227820+0000","last_undegraded":"2024-03-06T07:27:54.227820+0000","last_fullsized":"2024-03-06T07:27:54.227820+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574943,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33512,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338328,"statfs":{"total":96636764160,"available":96602447872,"internally_reserved":0,"allocated":2379776,"data_stored":1714023,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33464,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":2330624,"data_stored":1124721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574942,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33460,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338380,"statfs":{"total":96636764160,"available":96602501120,"internally_reserved":0,"allocated":2334720,"data_stored":1131357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574943,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33448,"kb_used_data":2268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338392,"statfs":{"total":96636764160,"available":96602513408,"internally_reserved":0,"allocated":2322432,"data_stored":1129765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574942,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33448,"kb_used_data":2268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338392,"statfs":{"total":96636764160,"available":96602513408,"internally_reserved":0,"allocated":2322432,"data_stored":1129697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574942,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37612,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334228,"statfs":{"total":96636764160,"available":96598249472,"internally_reserved":0,"allocated":2326528,"data_stored":1130291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33444,"kb_used_data":2264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338396,"statfs":{"total":96636764160,"available":96602517504,"internally_reserved":0,"allocated":2318336,"data_stored":1129675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574943,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37664,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334176,"statfs":{"total":96636764160,"available":96598196224,"internally_reserved":0,"allocated":2371584,"data_stored":1713351,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:55.913 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:56.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.049+0000 7f32e4f4b640 1 Processor -- start 2024-03-06T07:27:56.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.049+0000 7f32e4f4b640 1 -- start start 2024-03-06T07:27:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.049+0000 7f32e4f4b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32e00c3a90 0x7f32e0159250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.049+0000 7f32e4f4b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f32e00c5440 0x7f32e00c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.049+0000 7f32e4f4b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32e0155e50 0x7f32e01524f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.049+0000 7f32e4f4b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e00c5c50 con 0x7f32e0155e50 2024-03-06T07:27:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.049+0000 7f32e4f4b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e01559c0 con 0x7f32e00c5440 2024-03-06T07:27:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.049+0000 7f32e4f4b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00c5970 con 0x7f32e00c3a90 2024-03-06T07:27:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.050+0000 7f32ded9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32e0155e50 0x7f32e01524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.050+0000 7f32ded9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32e0155e50 0x7f32e01524f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60434/0 (socket says 172.21.15.1:60434) 2024-03-06T07:27:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.050+0000 7f32ded9d640 1 -- 172.21.15.1:0/3987628700 learned_addr learned my addr 172.21.15.1:0/3987628700 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.050+0000 7f32d5d9b640 1 --2- 172.21.15.1:0/3987628700 >> v2:172.21.15.98:3300/0 conn(0x7f32e00c5440 0x7f32e00c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.050+0000 7f32de59c640 1 --2- 172.21.15.1:0/3987628700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32e00c3a90 0x7f32e0159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.050+0000 7f32ded9d640 1 -- 172.21.15.1:0/3987628700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32e00c3a90 msgr2=0x7f32e0159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.050+0000 7f32ded9d640 1 --2- 172.21.15.1:0/3987628700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32e00c3a90 0x7f32e0159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.050+0000 7f32ded9d640 1 -- 172.21.15.1:0/3987628700 >> v2:172.21.15.98:3300/0 conn(0x7f32e00c5440 msgr2=0x7f32e00c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.050+0000 7f32ded9d640 1 --2- 172.21.15.1:0/3987628700 >> v2:172.21.15.98:3300/0 conn(0x7f32e00c5440 0x7f32e00c67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.050+0000 7f32ded9d640 1 -- 172.21.15.1:0/3987628700 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e0155370 con 0x7f32e0155e50 2024-03-06T07:27:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.050+0000 7f32d5d9b640 1 --2- 172.21.15.1:0/3987628700 >> v2:172.21.15.98:3300/0 conn(0x7f32e00c5440 0x7f32e00c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:56.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.051+0000 7f32ded9d640 1 --2- 172.21.15.1:0/3987628700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e0155e50 0x7f32e01524f0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f32e0073210 tx=0x7f32d802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227c19b37906ef0f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.051+0000 7f32ddd9b640 1 -- 172.21.15.1:0/3987628700 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d802c9e0 con 0x7f32e0155e50 2024-03-06T07:27:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.051+0000 7f32ddd9b640 1 -- 172.21.15.1:0/3987628700 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d802cb80 con 0x7f32e0155e50 2024-03-06T07:27:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.051+0000 7f32ddd9b640 1 -- 172.21.15.1:0/3987628700 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d80343f0 con 0x7f32e0155e50 2024-03-06T07:27:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.052+0000 7f32e4f4b640 1 -- 172.21.15.1:0/3987628700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e0155e50 msgr2=0x7f32e01524f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.052+0000 7f32e4f4b640 1 --2- 172.21.15.1:0/3987628700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e0155e50 0x7f32e01524f0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f32e0073210 tx=0x7f32d802be40 comp rx=0 tx=0).stop 2024-03-06T07:27:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.052+0000 7f32e4f4b640 1 -- 172.21.15.1:0/3987628700 shutdown_connections 2024-03-06T07:27:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.052+0000 7f32e4f4b640 1 --2- 172.21.15.1:0/3987628700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e0155e50 0x7f32e01524f0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.052+0000 7f32e4f4b640 1 --2- 172.21.15.1:0/3987628700 >> v2:172.21.15.98:3300/0 conn(0x7f32e00c5440 0x7f32e00c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.052+0000 7f32e4f4b640 1 --2- 172.21.15.1:0/3987628700 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32e00c3a90 0x7f32e0159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.052+0000 7f32e4f4b640 1 -- 172.21.15.1:0/3987628700 >> 172.21.15.1:0/3987628700 conn(0x7f32e00c88f0 msgr2=0x7f32e00c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.052+0000 7f32e4f4b640 1 -- 172.21.15.1:0/3987628700 shutdown_connections 2024-03-06T07:27:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.052+0000 7f32e4f4b640 1 -- 172.21.15.1:0/3987628700 wait complete. 2024-03-06T07:27:56.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.053+0000 7f32e4f4b640 1 Processor -- start 2024-03-06T07:27:56.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.053+0000 7f32e4f4b640 1 -- start start 2024-03-06T07:27:56.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32e4f4b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32e00c3a90 0x7f32e0150ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32e4f4b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e00c5440 0x7f32e01edf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32e4f4b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e0155e50 0x7f32e014dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32e4f4b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00c3e70 con 0x7f32e00c5440 2024-03-06T07:27:56.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32e4f4b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00c84a0 con 0x7f32e0155e50 2024-03-06T07:27:56.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32e4f4b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00c3ff0 con 0x7f32e00c3a90 2024-03-06T07:27:56.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32d5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e00c5440 0x7f32e01edf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32d5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e00c5440 0x7f32e01edf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60442/0 (socket says 172.21.15.1:60442) 2024-03-06T07:27:56.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32d5d9b640 1 -- 172.21.15.1:0/1267869264 learned_addr learned my addr 172.21.15.1:0/1267869264 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:56.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32de59c640 1 --2- 172.21.15.1:0/1267869264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32e00c3a90 0x7f32e0150ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32ded9d640 1 --2- 172.21.15.1:0/1267869264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e0155e50 0x7f32e014dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32d5d9b640 1 -- 172.21.15.1:0/1267869264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32e00c3a90 msgr2=0x7f32e0150ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32d5d9b640 1 --2- 172.21.15.1:0/1267869264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32e00c3a90 0x7f32e0150ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32d5d9b640 1 -- 172.21.15.1:0/1267869264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e0155e50 msgr2=0x7f32e014dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32d5d9b640 1 --2- 172.21.15.1:0/1267869264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e0155e50 0x7f32e014dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32d5d9b640 1 -- 172.21.15.1:0/1267869264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e01eeb50 con 0x7f32e00c5440 2024-03-06T07:27:56.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32de59c640 1 --2- 172.21.15.1:0/1267869264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32e00c3a90 0x7f32e0150ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:56.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.054+0000 7f32ded9d640 1 --2- 172.21.15.1:0/1267869264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e0155e50 0x7f32e014dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:56.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.055+0000 7f32d5d9b640 1 --2- 172.21.15.1:0/1267869264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e00c5440 0x7f32e01edf50 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f32c800e760 tx=0x7f32c800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.055+0000 7f32d77fe640 1 -- 172.21.15.1:0/1267869264 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c800f040 con 0x7f32e00c5440 2024-03-06T07:27:56.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.055+0000 7f32d77fe640 1 -- 172.21.15.1:0/1267869264 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c8004230 con 0x7f32e00c5440 2024-03-06T07:27:56.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.055+0000 7f32e4f4b640 1 -- 172.21.15.1:0/1267869264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e0071600 con 0x7f32e00c5440 2024-03-06T07:27:56.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.055+0000 7f32d77fe640 1 -- 172.21.15.1:0/1267869264 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c800ee70 con 0x7f32e00c5440 2024-03-06T07:27:56.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.055+0000 7f32e4f4b640 1 -- 172.21.15.1:0/1267869264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e0070560 con 0x7f32e00c5440 2024-03-06T07:27:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.056+0000 7f32e4f4b640 1 -- 172.21.15.1:0/1267869264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e00686e0 con 0x7f32e00c5440 2024-03-06T07:27:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.056+0000 7f32d77fe640 1 -- 172.21.15.1:0/1267869264 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f32c800c6d0 con 0x7f32e00c5440 2024-03-06T07:27:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.057+0000 7f32d77fe640 1 --2- 172.21.15.1:0/1267869264 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32c006b4d0 0x7f32c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.057+0000 7f32d77fe640 1 -- 172.21.15.1:0/1267869264 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f32c808a160 con 0x7f32e00c5440 2024-03-06T07:27:56.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.057+0000 7f32de59c640 1 --2- 172.21.15.1:0/1267869264 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32c006b4d0 0x7f32c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.057+0000 7f32de59c640 1 --2- 172.21.15.1:0/1267869264 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32c006b4d0 0x7f32c006d990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f32d0009b10 tx=0x7f32d0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.066+0000 7f32d77fe640 1 -- 172.21.15.1:0/1267869264 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32c8017020 con 0x7f32e00c5440 2024-03-06T07:27:56.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.256+0000 7f32e4f4b640 1 -- 172.21.15.1:0/1267869264 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f32e0158d10 con 0x7f32c006b4d0 2024-03-06T07:27:56.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.262+0000 7f32d77fe640 1 -- 172.21.15.1:0/1267869264 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252563 (secure 0 0 0) 0x7f32e0158d10 con 0x7f32c006b4d0 2024-03-06T07:27:56.267 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:56.268 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:27:56.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.265+0000 7f32e4f4b640 1 -- 172.21.15.1:0/1267869264 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32c006b4d0 msgr2=0x7f32c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.265+0000 7f32e4f4b640 1 --2- 172.21.15.1:0/1267869264 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32c006b4d0 0x7f32c006d990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f32d0009b10 tx=0x7f32d0008040 comp rx=0 tx=0).stop 2024-03-06T07:27:56.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.265+0000 7f32e4f4b640 1 -- 172.21.15.1:0/1267869264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e00c5440 msgr2=0x7f32e01edf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.265+0000 7f32e4f4b640 1 --2- 172.21.15.1:0/1267869264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e00c5440 0x7f32e01edf50 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f32c800e760 tx=0x7f32c800ec20 comp rx=0 tx=0).stop 2024-03-06T07:27:56.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.265+0000 7f32e4f4b640 1 -- 172.21.15.1:0/1267869264 shutdown_connections 2024-03-06T07:27:56.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.265+0000 7f32e4f4b640 1 --2- 172.21.15.1:0/1267869264 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f32c006b4d0 0x7f32c006d990 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.265+0000 7f32e4f4b640 1 --2- 172.21.15.1:0/1267869264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e0155e50 0x7f32e014dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.265+0000 7f32e4f4b640 1 --2- 172.21.15.1:0/1267869264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e00c5440 0x7f32e01edf50 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.265+0000 7f32e4f4b640 1 --2- 172.21.15.1:0/1267869264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f32e00c3a90 0x7f32e0150ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.265+0000 7f32e4f4b640 1 -- 172.21.15.1:0/1267869264 >> 172.21.15.1:0/1267869264 conn(0x7f32e00c88f0 msgr2=0x7f32e01522c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.265+0000 7f32e4f4b640 1 -- 172.21.15.1:0/1267869264 shutdown_connections 2024-03-06T07:27:56.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.265+0000 7f32e4f4b640 1 -- 172.21.15.1:0/1267869264 wait complete. 2024-03-06T07:27:56.278 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":406,"stamp":"2024-03-06T07:27:54.796831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":125,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":276052,"kb_used_data":18268,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754698668,"statfs":{"total":773094113280,"available":772811436032,"internally_reserved":0,"allocated":18706432,"data_stored":10202880,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001092"},"pg_stats":[{"pgid":"30.3f","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228517+0000","last_change":"2024-03-06T07:27:52.842860+0000","last_active":"2024-03-06T07:27:54.228517+0000","last_peered":"2024-03-06T07:27:54.228517+0000","last_clean":"2024-03-06T07:27:54.228517+0000","last_became_active":"2024-03-06T07:27:52.842531+0000","last_became_peered":"2024-03-06T07:27:52.842531+0000","last_unstale":"2024-03-06T07:27:54.228517+0000","last_undegraded":"2024-03-06T07:27:54.228517+0000","last_fullsized":"2024-03-06T07:27:54.228517+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:14.199266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227456+0000","last_change":"2024-03-06T07:27:52.843586+0000","last_active":"2024-03-06T07:27:54.227456+0000","last_peered":"2024-03-06T07:27:54.227456+0000","last_clean":"2024-03-06T07:27:54.227456+0000","last_became_active":"2024-03-06T07:27:52.843302+0000","last_became_peered":"2024-03-06T07:27:52.843302+0000","last_unstale":"2024-03-06T07:27:54.227456+0000","last_undegraded":"2024-03-06T07:27:54.227456+0000","last_fullsized":"2024-03-06T07:27:54.227456+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:04.175670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827599+0000","last_change":"2024-03-06T07:27:52.841423+0000","last_active":"2024-03-06T07:27:53.827599+0000","last_peered":"2024-03-06T07:27:53.827599+0000","last_clean":"2024-03-06T07:27:53.827599+0000","last_became_active":"2024-03-06T07:27:52.841148+0000","last_became_peered":"2024-03-06T07:27:52.841148+0000","last_unstale":"2024-03-06T07:27:53.827599+0000","last_undegraded":"2024-03-06T07:27:53.827599+0000","last_fullsized":"2024-03-06T07:27:53.827599+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:44.654522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.828425+0000","last_change":"2024-03-06T07:27:52.836948+0000","last_active":"2024-03-06T07:27:53.828425+0000","last_peered":"2024-03-06T07:27:53.828425+0000","last_clean":"2024-03-06T07:27:53.828425+0000","last_became_active":"2024-03-06T07:27:52.836693+0000","last_became_peered":"2024-03-06T07:27:52.836693+0000","last_unstale":"2024-03-06T07:27:53.828425+0000","last_undegraded":"2024-03-06T07:27:53.828425+0000","last_fullsized":"2024-03-06T07:27:53.828425+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:33.742320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228131+0000","last_change":"2024-03-06T07:27:52.848857+0000","last_active":"2024-03-06T07:27:54.228131+0000","last_peered":"2024-03-06T07:27:54.228131+0000","last_clean":"2024-03-06T07:27:54.228131+0000","last_became_active":"2024-03-06T07:27:52.848253+0000","last_became_peered":"2024-03-06T07:27:52.848253+0000","last_unstale":"2024-03-06T07:27:54.228131+0000","last_undegraded":"2024-03-06T07:27:54.228131+0000","last_fullsized":"2024-03-06T07:27:54.228131+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:17.947842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116875+0000","last_change":"2024-03-06T07:27:52.839769+0000","last_active":"2024-03-06T07:27:54.116875+0000","last_peered":"2024-03-06T07:27:54.116875+0000","last_clean":"2024-03-06T07:27:54.116875+0000","last_became_active":"2024-03-06T07:27:52.839488+0000","last_became_peered":"2024-03-06T07:27:52.839488+0000","last_unstale":"2024-03-06T07:27:54.116875+0000","last_undegraded":"2024-03-06T07:27:54.116875+0000","last_fullsized":"2024-03-06T07:27:54.116875+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:43.780203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827357+0000","last_change":"2024-03-06T07:27:52.845031+0000","last_active":"2024-03-06T07:27:53.827357+0000","last_peered":"2024-03-06T07:27:53.827357+0000","last_clean":"2024-03-06T07:27:53.827357+0000","last_became_active":"2024-03-06T07:27:52.844686+0000","last_became_peered":"2024-03-06T07:27:52.844686+0000","last_unstale":"2024-03-06T07:27:53.827357+0000","last_undegraded":"2024-03-06T07:27:53.827357+0000","last_fullsized":"2024-03-06T07:27:53.827357+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:18.206755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116506+0000","last_change":"2024-03-06T07:27:52.838158+0000","last_active":"2024-03-06T07:27:54.116506+0000","last_peered":"2024-03-06T07:27:54.116506+0000","last_clean":"2024-03-06T07:27:54.116506+0000","last_became_active":"2024-03-06T07:27:52.837890+0000","last_became_peered":"2024-03-06T07:27:52.837890+0000","last_unstale":"2024-03-06T07:27:54.116506+0000","last_undegraded":"2024-03-06T07:27:54.116506+0000","last_fullsized":"2024-03-06T07:27:54.116506+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:33.431453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824522+0000","last_change":"2024-03-06T07:27:52.848310+0000","last_active":"2024-03-06T07:27:53.824522+0000","last_peered":"2024-03-06T07:27:53.824522+0000","last_clean":"2024-03-06T07:27:53.824522+0000","last_became_active":"2024-03-06T07:27:52.848053+0000","last_became_peered":"2024-03-06T07:27:52.848053+0000","last_unstale":"2024-03-06T07:27:53.824522+0000","last_undegraded":"2024-03-06T07:27:53.824522+0000","last_fullsized":"2024-03-06T07:27:53.824522+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:03.922799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827674+0000","last_change":"2024-03-06T07:27:52.842430+0000","last_active":"2024-03-06T07:27:53.827674+0000","last_peered":"2024-03-06T07:27:53.827674+0000","last_clean":"2024-03-06T07:27:53.827674+0000","last_became_active":"2024-03-06T07:27:52.842176+0000","last_became_peered":"2024-03-06T07:27:52.842176+0000","last_unstale":"2024-03-06T07:27:53.827674+0000","last_undegraded":"2024-03-06T07:27:53.827674+0000","last_fullsized":"2024-03-06T07:27:53.827674+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:02.576405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.828524+0000","last_change":"2024-03-06T07:27:52.832071+0000","last_active":"2024-03-06T07:27:53.828524+0000","last_peered":"2024-03-06T07:27:53.828524+0000","last_clean":"2024-03-06T07:27:53.828524+0000","last_became_active":"2024-03-06T07:27:52.831756+0000","last_became_peered":"2024-03-06T07:27:52.831756+0000","last_unstale":"2024-03-06T07:27:53.828524+0000","last_undegraded":"2024-03-06T07:27:53.828524+0000","last_fullsized":"2024-03-06T07:27:53.828524+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:08.654039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229066+0000","last_change":"2024-03-06T07:27:52.849912+0000","last_active":"2024-03-06T07:27:54.229066+0000","last_peered":"2024-03-06T07:27:54.229066+0000","last_clean":"2024-03-06T07:27:54.229066+0000","last_became_active":"2024-03-06T07:27:52.849284+0000","last_became_peered":"2024-03-06T07:27:52.849284+0000","last_unstale":"2024-03-06T07:27:54.229066+0000","last_undegraded":"2024-03-06T07:27:54.229066+0000","last_fullsized":"2024-03-06T07:27:54.229066+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:38.851748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116740+0000","last_change":"2024-03-06T07:27:52.835245+0000","last_active":"2024-03-06T07:27:54.116740+0000","last_peered":"2024-03-06T07:27:54.116740+0000","last_clean":"2024-03-06T07:27:54.116740+0000","last_became_active":"2024-03-06T07:27:52.834973+0000","last_became_peered":"2024-03-06T07:27:52.834973+0000","last_unstale":"2024-03-06T07:27:54.116740+0000","last_undegraded":"2024-03-06T07:27:54.116740+0000","last_fullsized":"2024-03-06T07:27:54.116740+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:25.883597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228795+0000","last_change":"2024-03-06T07:27:52.850092+0000","last_active":"2024-03-06T07:27:54.228795+0000","last_peered":"2024-03-06T07:27:54.228795+0000","last_clean":"2024-03-06T07:27:54.228795+0000","last_became_active":"2024-03-06T07:27:52.849770+0000","last_became_peered":"2024-03-06T07:27:52.849770+0000","last_unstale":"2024-03-06T07:27:54.228795+0000","last_undegraded":"2024-03-06T07:27:54.228795+0000","last_fullsized":"2024-03-06T07:27:54.228795+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:49.841334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842594+0000","last_change":"2024-03-06T07:27:52.847806+0000","last_active":"2024-03-06T07:27:53.842594+0000","last_peered":"2024-03-06T07:27:53.842594+0000","last_clean":"2024-03-06T07:27:53.842594+0000","last_became_active":"2024-03-06T07:27:52.847516+0000","last_became_peered":"2024-03-06T07:27:52.847516+0000","last_unstale":"2024-03-06T07:27:53.842594+0000","last_undegraded":"2024-03-06T07:27:53.842594+0000","last_fullsized":"2024-03-06T07:27:53.842594+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:17.947842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116592+0000","last_change":"2024-03-06T07:27:52.840358+0000","last_active":"2024-03-06T07:27:54.116592+0000","last_peered":"2024-03-06T07:27:54.116592+0000","last_clean":"2024-03-06T07:27:54.116592+0000","last_became_active":"2024-03-06T07:27:52.840039+0000","last_became_peered":"2024-03-06T07:27:52.840039+0000","last_unstale":"2024-03-06T07:27:54.116592+0000","last_undegraded":"2024-03-06T07:27:54.116592+0000","last_fullsized":"2024-03-06T07:27:54.116592+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:40.496286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229137+0000","last_change":"2024-03-06T07:27:52.848809+0000","last_active":"2024-03-06T07:27:54.229137+0000","last_peered":"2024-03-06T07:27:54.229137+0000","last_clean":"2024-03-06T07:27:54.229137+0000","last_became_active":"2024-03-06T07:27:52.847002+0000","last_became_peered":"2024-03-06T07:27:52.847002+0000","last_unstale":"2024-03-06T07:27:54.229137+0000","last_undegraded":"2024-03-06T07:27:54.229137+0000","last_fullsized":"2024-03-06T07:27:54.229137+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:09.978274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827669+0000","last_change":"2024-03-06T07:27:52.841828+0000","last_active":"2024-03-06T07:27:53.827669+0000","last_peered":"2024-03-06T07:27:53.827669+0000","last_clean":"2024-03-06T07:27:53.827669+0000","last_became_active":"2024-03-06T07:27:52.840848+0000","last_became_peered":"2024-03-06T07:27:52.840848+0000","last_unstale":"2024-03-06T07:27:53.827669+0000","last_undegraded":"2024-03-06T07:27:53.827669+0000","last_fullsized":"2024-03-06T07:27:53.827669+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:56.120793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228291+0000","last_change":"2024-03-06T07:27:52.843757+0000","last_active":"2024-03-06T07:27:54.228291+0000","last_peered":"2024-03-06T07:27:54.228291+0000","last_clean":"2024-03-06T07:27:54.228291+0000","last_became_active":"2024-03-06T07:27:52.843251+0000","last_became_peered":"2024-03-06T07:27:52.843251+0000","last_unstale":"2024-03-06T07:27:54.228291+0000","last_undegraded":"2024-03-06T07:27:54.228291+0000","last_fullsized":"2024-03-06T07:27:54.228291+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:02.906107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116915+0000","last_change":"2024-03-06T07:27:52.846815+0000","last_active":"2024-03-06T07:27:54.116915+0000","last_peered":"2024-03-06T07:27:54.116915+0000","last_clean":"2024-03-06T07:27:54.116915+0000","last_became_active":"2024-03-06T07:27:52.846395+0000","last_became_peered":"2024-03-06T07:27:52.846395+0000","last_unstale":"2024-03-06T07:27:54.116915+0000","last_undegraded":"2024-03-06T07:27:54.116915+0000","last_fullsized":"2024-03-06T07:27:54.116915+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:31.090488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827344+0000","last_change":"2024-03-06T07:27:52.840352+0000","last_active":"2024-03-06T07:27:53.827344+0000","last_peered":"2024-03-06T07:27:53.827344+0000","last_clean":"2024-03-06T07:27:53.827344+0000","last_became_active":"2024-03-06T07:27:52.840069+0000","last_became_peered":"2024-03-06T07:27:52.840069+0000","last_unstale":"2024-03-06T07:27:53.827344+0000","last_undegraded":"2024-03-06T07:27:53.827344+0000","last_fullsized":"2024-03-06T07:27:53.827344+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:58.583352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842762+0000","last_change":"2024-03-06T07:27:52.842867+0000","last_active":"2024-03-06T07:27:53.842762+0000","last_peered":"2024-03-06T07:27:53.842762+0000","last_clean":"2024-03-06T07:27:53.842762+0000","last_became_active":"2024-03-06T07:27:52.842617+0000","last_became_peered":"2024-03-06T07:27:52.842617+0000","last_unstale":"2024-03-06T07:27:53.842762+0000","last_undegraded":"2024-03-06T07:27:53.842762+0000","last_fullsized":"2024-03-06T07:27:53.842762+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:14.199266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824426+0000","last_change":"2024-03-06T07:27:52.849788+0000","last_active":"2024-03-06T07:27:53.824426+0000","last_peered":"2024-03-06T07:27:53.824426+0000","last_clean":"2024-03-06T07:27:53.824426+0000","last_became_active":"2024-03-06T07:27:52.848829+0000","last_became_peered":"2024-03-06T07:27:52.848829+0000","last_unstale":"2024-03-06T07:27:53.824426+0000","last_undegraded":"2024-03-06T07:27:53.824426+0000","last_fullsized":"2024-03-06T07:27:53.824426+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:07.418348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227869+0000","last_change":"2024-03-06T07:27:52.844099+0000","last_active":"2024-03-06T07:27:54.227869+0000","last_peered":"2024-03-06T07:27:54.227869+0000","last_clean":"2024-03-06T07:27:54.227869+0000","last_became_active":"2024-03-06T07:27:52.843467+0000","last_became_peered":"2024-03-06T07:27:52.843467+0000","last_unstale":"2024-03-06T07:27:54.227869+0000","last_undegraded":"2024-03-06T07:27:54.227869+0000","last_fullsized":"2024-03-06T07:27:54.227869+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:30.502225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228427+0000","last_change":"2024-03-06T07:27:52.843062+0000","last_active":"2024-03-06T07:27:54.228427+0000","last_peered":"2024-03-06T07:27:54.228427+0000","last_clean":"2024-03-06T07:27:54.228427+0000","last_became_active":"2024-03-06T07:27:52.842694+0000","last_became_peered":"2024-03-06T07:27:52.842694+0000","last_unstale":"2024-03-06T07:27:54.228427+0000","last_undegraded":"2024-03-06T07:27:54.228427+0000","last_fullsized":"2024-03-06T07:27:54.228427+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:57:54.032945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827779+0000","last_change":"2024-03-06T07:27:52.854005+0000","last_active":"2024-03-06T07:27:53.827779+0000","last_peered":"2024-03-06T07:27:53.827779+0000","last_clean":"2024-03-06T07:27:53.827779+0000","last_became_active":"2024-03-06T07:27:52.853490+0000","last_became_peered":"2024-03-06T07:27:52.853490+0000","last_unstale":"2024-03-06T07:27:53.827779+0000","last_undegraded":"2024-03-06T07:27:53.827779+0000","last_fullsized":"2024-03-06T07:27:53.827779+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:08.487751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.828536+0000","last_change":"2024-03-06T07:27:52.848560+0000","last_active":"2024-03-06T07:27:53.828536+0000","last_peered":"2024-03-06T07:27:53.828536+0000","last_clean":"2024-03-06T07:27:53.828536+0000","last_became_active":"2024-03-06T07:27:52.848293+0000","last_became_peered":"2024-03-06T07:27:52.848293+0000","last_unstale":"2024-03-06T07:27:53.828536+0000","last_undegraded":"2024-03-06T07:27:53.828536+0000","last_fullsized":"2024-03-06T07:27:53.828536+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:38.851748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229139+0000","last_change":"2024-03-06T07:27:52.849159+0000","last_active":"2024-03-06T07:27:54.229139+0000","last_peered":"2024-03-06T07:27:54.229139+0000","last_clean":"2024-03-06T07:27:54.229139+0000","last_became_active":"2024-03-06T07:27:52.847083+0000","last_became_peered":"2024-03-06T07:27:52.847083+0000","last_unstale":"2024-03-06T07:27:54.229139+0000","last_undegraded":"2024-03-06T07:27:54.229139+0000","last_fullsized":"2024-03-06T07:27:54.229139+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:49.125933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227349+0000","last_change":"2024-03-06T07:27:52.832481+0000","last_active":"2024-03-06T07:27:54.227349+0000","last_peered":"2024-03-06T07:27:54.227349+0000","last_clean":"2024-03-06T07:27:54.227349+0000","last_became_active":"2024-03-06T07:27:52.832194+0000","last_became_peered":"2024-03-06T07:27:52.832194+0000","last_unstale":"2024-03-06T07:27:54.227349+0000","last_undegraded":"2024-03-06T07:27:54.227349+0000","last_fullsized":"2024-03-06T07:27:54.227349+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:46.848368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116738+0000","last_change":"2024-03-06T07:27:52.851822+0000","last_active":"2024-03-06T07:27:54.116738+0000","last_peered":"2024-03-06T07:27:54.116738+0000","last_clean":"2024-03-06T07:27:54.116738+0000","last_became_active":"2024-03-06T07:27:52.851543+0000","last_became_peered":"2024-03-06T07:27:52.851543+0000","last_unstale":"2024-03-06T07:27:54.116738+0000","last_undegraded":"2024-03-06T07:27:54.116738+0000","last_fullsized":"2024-03-06T07:27:54.116738+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:48.053555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824343+0000","last_change":"2024-03-06T07:27:52.827809+0000","last_active":"2024-03-06T07:27:53.824343+0000","last_peered":"2024-03-06T07:27:53.824343+0000","last_clean":"2024-03-06T07:27:53.824343+0000","last_became_active":"2024-03-06T07:27:52.827502+0000","last_became_peered":"2024-03-06T07:27:52.827502+0000","last_unstale":"2024-03-06T07:27:53.824343+0000","last_undegraded":"2024-03-06T07:27:53.824343+0000","last_fullsized":"2024-03-06T07:27:53.824343+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:51.129075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116711+0000","last_change":"2024-03-06T07:27:52.847729+0000","last_active":"2024-03-06T07:27:54.116711+0000","last_peered":"2024-03-06T07:27:54.116711+0000","last_clean":"2024-03-06T07:27:54.116711+0000","last_became_active":"2024-03-06T07:27:52.847452+0000","last_became_peered":"2024-03-06T07:27:52.847452+0000","last_unstale":"2024-03-06T07:27:54.116711+0000","last_undegraded":"2024-03-06T07:27:54.116711+0000","last_fullsized":"2024-03-06T07:27:54.116711+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:02.639317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827559+0000","last_change":"2024-03-06T07:27:52.843112+0000","last_active":"2024-03-06T07:27:53.827559+0000","last_peered":"2024-03-06T07:27:53.827559+0000","last_clean":"2024-03-06T07:27:53.827559+0000","last_became_active":"2024-03-06T07:27:52.842787+0000","last_became_peered":"2024-03-06T07:27:52.842787+0000","last_unstale":"2024-03-06T07:27:53.827559+0000","last_undegraded":"2024-03-06T07:27:53.827559+0000","last_fullsized":"2024-03-06T07:27:53.827559+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:35.725347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"182'4","reported_seq":18,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824576+0000","last_change":"2024-03-06T07:27:51.829598+0000","last_active":"2024-03-06T07:27:53.824576+0000","last_peered":"2024-03-06T07:27:53.824576+0000","last_clean":"2024-03-06T07:27:53.824576+0000","last_became_active":"2024-03-06T07:27:51.829308+0000","last_became_peered":"2024-03-06T07:27:51.829308+0000","last_unstale":"2024-03-06T07:27:53.824576+0000","last_undegraded":"2024-03-06T07:27:53.824576+0000","last_fullsized":"2024-03-06T07:27:53.824576+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:18.568415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827622+0000","last_change":"2024-03-06T07:27:52.849160+0000","last_active":"2024-03-06T07:27:53.827622+0000","last_peered":"2024-03-06T07:27:53.827622+0000","last_clean":"2024-03-06T07:27:53.827622+0000","last_became_active":"2024-03-06T07:27:52.848836+0000","last_became_peered":"2024-03-06T07:27:52.848836+0000","last_unstale":"2024-03-06T07:27:53.827622+0000","last_undegraded":"2024-03-06T07:27:53.827622+0000","last_fullsized":"2024-03-06T07:27:53.827622+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:27.108826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"182'1","reported_seq":15,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229080+0000","last_change":"2024-03-06T07:27:51.830745+0000","last_active":"2024-03-06T07:27:54.229080+0000","last_peered":"2024-03-06T07:27:54.229080+0000","last_clean":"2024-03-06T07:27:54.229080+0000","last_became_active":"2024-03-06T07:27:51.830490+0000","last_became_peered":"2024-03-06T07:27:51.830490+0000","last_unstale":"2024-03-06T07:27:54.229080+0000","last_undegraded":"2024-03-06T07:27:54.229080+0000","last_fullsized":"2024-03-06T07:27:54.229080+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:40.184591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228894+0000","last_change":"2024-03-06T07:27:52.845892+0000","last_active":"2024-03-06T07:27:54.228894+0000","last_peered":"2024-03-06T07:27:54.228894+0000","last_clean":"2024-03-06T07:27:54.228894+0000","last_became_active":"2024-03-06T07:27:52.845611+0000","last_became_peered":"2024-03-06T07:27:52.845611+0000","last_unstale":"2024-03-06T07:27:54.228894+0000","last_undegraded":"2024-03-06T07:27:54.228894+0000","last_fullsized":"2024-03-06T07:27:54.228894+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:33.742320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1d","version":"182'2","reported_seq":16,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116786+0000","last_change":"2024-03-06T07:27:51.827920+0000","last_active":"2024-03-06T07:27:54.116786+0000","last_peered":"2024-03-06T07:27:54.116786+0000","last_clean":"2024-03-06T07:27:54.116786+0000","last_became_active":"2024-03-06T07:27:51.827650+0000","last_became_peered":"2024-03-06T07:27:51.827650+0000","last_unstale":"2024-03-06T07:27:54.116786+0000","last_undegraded":"2024-03-06T07:27:54.116786+0000","last_fullsized":"2024-03-06T07:27:54.116786+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:07.649833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827224+0000","last_change":"2024-03-06T07:27:52.841717+0000","last_active":"2024-03-06T07:27:53.827224+0000","last_peered":"2024-03-06T07:27:53.827224+0000","last_clean":"2024-03-06T07:27:53.827224+0000","last_became_active":"2024-03-06T07:27:52.841440+0000","last_became_peered":"2024-03-06T07:27:52.841440+0000","last_unstale":"2024-03-06T07:27:53.827224+0000","last_undegraded":"2024-03-06T07:27:53.827224+0000","last_fullsized":"2024-03-06T07:27:53.827224+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:48.085929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1b","version":"182'1","reported_seq":15,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842773+0000","last_change":"2024-03-06T07:27:51.830565+0000","last_active":"2024-03-06T07:27:53.842773+0000","last_peered":"2024-03-06T07:27:53.842773+0000","last_clean":"2024-03-06T07:27:53.842773+0000","last_became_active":"2024-03-06T07:27:51.830283+0000","last_became_peered":"2024-03-06T07:27:51.830283+0000","last_unstale":"2024-03-06T07:27:53.842773+0000","last_undegraded":"2024-03-06T07:27:53.842773+0000","last_fullsized":"2024-03-06T07:27:53.842773+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:23.893421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227905+0000","last_change":"2024-03-06T07:27:52.851487+0000","last_active":"2024-03-06T07:27:54.227905+0000","last_peered":"2024-03-06T07:27:54.227905+0000","last_clean":"2024-03-06T07:27:54.227905+0000","last_became_active":"2024-03-06T07:27:52.851074+0000","last_became_peered":"2024-03-06T07:27:52.851074+0000","last_unstale":"2024-03-06T07:27:54.227905+0000","last_undegraded":"2024-03-06T07:27:54.227905+0000","last_fullsized":"2024-03-06T07:27:54.227905+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:04.893618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"182'1","reported_seq":15,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227974+0000","last_change":"2024-03-06T07:27:51.832998+0000","last_active":"2024-03-06T07:27:54.227974+0000","last_peered":"2024-03-06T07:27:54.227974+0000","last_clean":"2024-03-06T07:27:54.227974+0000","last_became_active":"2024-03-06T07:27:51.832605+0000","last_became_peered":"2024-03-06T07:27:51.832605+0000","last_unstale":"2024-03-06T07:27:54.227974+0000","last_undegraded":"2024-03-06T07:27:54.227974+0000","last_fullsized":"2024-03-06T07:27:54.227974+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:57.043951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842605+0000","last_change":"2024-03-06T07:27:52.842376+0000","last_active":"2024-03-06T07:27:53.842605+0000","last_peered":"2024-03-06T07:27:53.842605+0000","last_clean":"2024-03-06T07:27:53.842605+0000","last_became_active":"2024-03-06T07:27:52.841989+0000","last_became_peered":"2024-03-06T07:27:52.841989+0000","last_unstale":"2024-03-06T07:27:53.842605+0000","last_undegraded":"2024-03-06T07:27:53.842605+0000","last_fullsized":"2024-03-06T07:27:53.842605+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:02.906107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.828387+0000","last_change":"2024-03-06T07:27:51.835979+0000","last_active":"2024-03-06T07:27:53.828387+0000","last_peered":"2024-03-06T07:27:53.828387+0000","last_clean":"2024-03-06T07:27:53.828387+0000","last_became_active":"2024-03-06T07:27:51.835707+0000","last_became_peered":"2024-03-06T07:27:51.835707+0000","last_unstale":"2024-03-06T07:27:53.828387+0000","last_undegraded":"2024-03-06T07:27:53.828387+0000","last_fullsized":"2024-03-06T07:27:53.828387+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:25.277737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824512+0000","last_change":"2024-03-06T07:27:52.836761+0000","last_active":"2024-03-06T07:27:53.824512+0000","last_peered":"2024-03-06T07:27:53.824512+0000","last_clean":"2024-03-06T07:27:53.824512+0000","last_became_active":"2024-03-06T07:27:52.836464+0000","last_became_peered":"2024-03-06T07:27:52.836464+0000","last_unstale":"2024-03-06T07:27:53.824512+0000","last_undegraded":"2024-03-06T07:27:53.824512+0000","last_fullsized":"2024-03-06T07:27:53.824512+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:21.419139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827332+0000","last_change":"2024-03-06T07:27:51.832162+0000","last_active":"2024-03-06T07:27:53.827332+0000","last_peered":"2024-03-06T07:27:53.827332+0000","last_clean":"2024-03-06T07:27:53.827332+0000","last_became_active":"2024-03-06T07:27:51.831873+0000","last_became_peered":"2024-03-06T07:27:51.831873+0000","last_unstale":"2024-03-06T07:27:53.827332+0000","last_undegraded":"2024-03-06T07:27:53.827332+0000","last_fullsized":"2024-03-06T07:27:53.827332+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:45.844161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842775+0000","last_change":"2024-03-06T07:27:52.847447+0000","last_active":"2024-03-06T07:27:53.842775+0000","last_peered":"2024-03-06T07:27:53.842775+0000","last_clean":"2024-03-06T07:27:53.842775+0000","last_became_active":"2024-03-06T07:27:52.847129+0000","last_became_peered":"2024-03-06T07:27:52.847129+0000","last_unstale":"2024-03-06T07:27:53.842775+0000","last_undegraded":"2024-03-06T07:27:53.842775+0000","last_fullsized":"2024-03-06T07:27:53.842775+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:57:54.032945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228630+0000","last_change":"2024-03-06T07:27:51.832113+0000","last_active":"2024-03-06T07:27:54.228630+0000","last_peered":"2024-03-06T07:27:54.228630+0000","last_clean":"2024-03-06T07:27:54.228630+0000","last_became_active":"2024-03-06T07:27:51.831777+0000","last_became_peered":"2024-03-06T07:27:51.831777+0000","last_unstale":"2024-03-06T07:27:54.228630+0000","last_undegraded":"2024-03-06T07:27:54.228630+0000","last_fullsized":"2024-03-06T07:27:54.228630+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:26.154047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842954+0000","last_change":"2024-03-06T07:27:52.847490+0000","last_active":"2024-03-06T07:27:53.842954+0000","last_peered":"2024-03-06T07:27:53.842954+0000","last_clean":"2024-03-06T07:27:53.842954+0000","last_became_active":"2024-03-06T07:27:52.846888+0000","last_became_peered":"2024-03-06T07:27:52.846888+0000","last_unstale":"2024-03-06T07:27:53.842954+0000","last_undegraded":"2024-03-06T07:27:53.842954+0000","last_fullsized":"2024-03-06T07:27:53.842954+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:01.915969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.828380+0000","last_change":"2024-03-06T07:27:51.833447+0000","last_active":"2024-03-06T07:27:53.828380+0000","last_peered":"2024-03-06T07:27:53.828380+0000","last_clean":"2024-03-06T07:27:53.828380+0000","last_became_active":"2024-03-06T07:27:51.833173+0000","last_became_peered":"2024-03-06T07:27:51.833173+0000","last_unstale":"2024-03-06T07:27:53.828380+0000","last_undegraded":"2024-03-06T07:27:53.828380+0000","last_fullsized":"2024-03-06T07:27:53.828380+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:04.569067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827836+0000","last_change":"2024-03-06T07:27:52.838853+0000","last_active":"2024-03-06T07:27:53.827836+0000","last_peered":"2024-03-06T07:27:53.827836+0000","last_clean":"2024-03-06T07:27:53.827836+0000","last_became_active":"2024-03-06T07:27:52.838528+0000","last_became_peered":"2024-03-06T07:27:52.838528+0000","last_unstale":"2024-03-06T07:27:53.827836+0000","last_undegraded":"2024-03-06T07:27:53.827836+0000","last_fullsized":"2024-03-06T07:27:53.827836+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:59.780024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"182'3","reported_seq":17,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228515+0000","last_change":"2024-03-06T07:27:51.832456+0000","last_active":"2024-03-06T07:27:54.228515+0000","last_peered":"2024-03-06T07:27:54.228515+0000","last_clean":"2024-03-06T07:27:54.228515+0000","last_became_active":"2024-03-06T07:27:51.832176+0000","last_became_peered":"2024-03-06T07:27:51.832176+0000","last_unstale":"2024-03-06T07:27:54.228515+0000","last_undegraded":"2024-03-06T07:27:54.228515+0000","last_fullsized":"2024-03-06T07:27:54.228515+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:10.808514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.828607+0000","last_change":"2024-03-06T07:27:52.831146+0000","last_active":"2024-03-06T07:27:53.828607+0000","last_peered":"2024-03-06T07:27:53.828607+0000","last_clean":"2024-03-06T07:27:53.828607+0000","last_became_active":"2024-03-06T07:27:52.830862+0000","last_became_peered":"2024-03-06T07:27:52.830862+0000","last_unstale":"2024-03-06T07:27:53.828607+0000","last_undegraded":"2024-03-06T07:27:53.828607+0000","last_fullsized":"2024-03-06T07:27:53.828607+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:09.978274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.14","version":"182'6","reported_seq":20,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116926+0000","last_change":"2024-03-06T07:27:51.830005+0000","last_active":"2024-03-06T07:27:54.116926+0000","last_peered":"2024-03-06T07:27:54.116926+0000","last_clean":"2024-03-06T07:27:54.116926+0000","last_became_active":"2024-03-06T07:27:51.829747+0000","last_became_peered":"2024-03-06T07:27:51.829747+0000","last_unstale":"2024-03-06T07:27:54.116926+0000","last_undegraded":"2024-03-06T07:27:54.116926+0000","last_fullsized":"2024-03-06T07:27:54.116926+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:37.847541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227137+0000","last_change":"2024-03-06T07:27:52.833504+0000","last_active":"2024-03-06T07:27:54.227137+0000","last_peered":"2024-03-06T07:27:54.227137+0000","last_clean":"2024-03-06T07:27:54.227137+0000","last_became_active":"2024-03-06T07:27:52.833228+0000","last_became_peered":"2024-03-06T07:27:52.833228+0000","last_unstale":"2024-03-06T07:27:54.227137+0000","last_undegraded":"2024-03-06T07:27:54.227137+0000","last_fullsized":"2024-03-06T07:27:54.227137+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:35.655688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"182'2","reported_seq":16,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228970+0000","last_change":"2024-03-06T07:27:51.830207+0000","last_active":"2024-03-06T07:27:54.228970+0000","last_peered":"2024-03-06T07:27:54.228970+0000","last_clean":"2024-03-06T07:27:54.228970+0000","last_became_active":"2024-03-06T07:27:51.829897+0000","last_became_peered":"2024-03-06T07:27:51.829897+0000","last_unstale":"2024-03-06T07:27:54.228970+0000","last_undegraded":"2024-03-06T07:27:54.228970+0000","last_fullsized":"2024-03-06T07:27:54.228970+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:14.460825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228493+0000","last_change":"2024-03-06T07:27:52.845627+0000","last_active":"2024-03-06T07:27:54.228493+0000","last_peered":"2024-03-06T07:27:54.228493+0000","last_clean":"2024-03-06T07:27:54.228493+0000","last_became_active":"2024-03-06T07:27:52.845234+0000","last_became_peered":"2024-03-06T07:27:52.845234+0000","last_unstale":"2024-03-06T07:27:54.228493+0000","last_undegraded":"2024-03-06T07:27:54.228493+0000","last_fullsized":"2024-03-06T07:27:54.228493+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:56.539951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227189+0000","last_change":"2024-03-06T07:27:51.822548+0000","last_active":"2024-03-06T07:27:54.227189+0000","last_peered":"2024-03-06T07:27:54.227189+0000","last_clean":"2024-03-06T07:27:54.227189+0000","last_became_active":"2024-03-06T07:27:51.822262+0000","last_became_peered":"2024-03-06T07:27:51.822262+0000","last_unstale":"2024-03-06T07:27:54.227189+0000","last_undegraded":"2024-03-06T07:27:54.227189+0000","last_fullsized":"2024-03-06T07:27:54.227189+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:16.943635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827425+0000","last_change":"2024-03-06T07:27:52.848419+0000","last_active":"2024-03-06T07:27:53.827425+0000","last_peered":"2024-03-06T07:27:53.827425+0000","last_clean":"2024-03-06T07:27:53.827425+0000","last_became_active":"2024-03-06T07:27:52.846559+0000","last_became_peered":"2024-03-06T07:27:52.846559+0000","last_unstale":"2024-03-06T07:27:53.827425+0000","last_undegraded":"2024-03-06T07:27:53.827425+0000","last_fullsized":"2024-03-06T07:27:53.827425+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:00.807566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116474+0000","last_change":"2024-03-06T07:27:51.833477+0000","last_active":"2024-03-06T07:27:54.116474+0000","last_peered":"2024-03-06T07:27:54.116474+0000","last_clean":"2024-03-06T07:27:54.116474+0000","last_became_active":"2024-03-06T07:27:51.833189+0000","last_became_peered":"2024-03-06T07:27:51.833189+0000","last_unstale":"2024-03-06T07:27:54.116474+0000","last_undegraded":"2024-03-06T07:27:54.116474+0000","last_fullsized":"2024-03-06T07:27:54.116474+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:32.738113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842738+0000","last_change":"2024-03-06T07:27:52.842524+0000","last_active":"2024-03-06T07:27:53.842738+0000","last_peered":"2024-03-06T07:27:53.842738+0000","last_clean":"2024-03-06T07:27:53.842738+0000","last_became_active":"2024-03-06T07:27:52.842171+0000","last_became_peered":"2024-03-06T07:27:52.842171+0000","last_unstale":"2024-03-06T07:27:53.842738+0000","last_undegraded":"2024-03-06T07:27:53.842738+0000","last_fullsized":"2024-03-06T07:27:53.842738+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:21.929558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"182'8","reported_seq":22,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228065+0000","last_change":"2024-03-06T07:27:51.836500+0000","last_active":"2024-03-06T07:27:54.228065+0000","last_peered":"2024-03-06T07:27:54.228065+0000","last_clean":"2024-03-06T07:27:54.228065+0000","last_became_active":"2024-03-06T07:27:51.836156+0000","last_became_peered":"2024-03-06T07:27:51.836156+0000","last_unstale":"2024-03-06T07:27:54.228065+0000","last_undegraded":"2024-03-06T07:27:54.228065+0000","last_fullsized":"2024-03-06T07:27:54.228065+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:41.727365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227314+0000","last_change":"2024-03-06T07:27:52.851096+0000","last_active":"2024-03-06T07:27:54.227314+0000","last_peered":"2024-03-06T07:27:54.227314+0000","last_clean":"2024-03-06T07:27:54.227314+0000","last_became_active":"2024-03-06T07:27:52.850775+0000","last_became_peered":"2024-03-06T07:27:52.850775+0000","last_unstale":"2024-03-06T07:27:54.227314+0000","last_undegraded":"2024-03-06T07:27:54.227314+0000","last_fullsized":"2024-03-06T07:27:54.227314+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:12.204157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"182'1","reported_seq":15,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827751+0000","last_change":"2024-03-06T07:27:51.831464+0000","last_active":"2024-03-06T07:27:53.827751+0000","last_peered":"2024-03-06T07:27:53.827751+0000","last_clean":"2024-03-06T07:27:53.827751+0000","last_became_active":"2024-03-06T07:27:51.831197+0000","last_became_peered":"2024-03-06T07:27:51.831197+0000","last_unstale":"2024-03-06T07:27:53.827751+0000","last_undegraded":"2024-03-06T07:27:53.827751+0000","last_fullsized":"2024-03-06T07:27:53.827751+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:03.171463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228399+0000","last_change":"2024-03-06T07:27:52.837210+0000","last_active":"2024-03-06T07:27:54.228399+0000","last_peered":"2024-03-06T07:27:54.228399+0000","last_clean":"2024-03-06T07:27:54.228399+0000","last_became_active":"2024-03-06T07:27:52.836950+0000","last_became_peered":"2024-03-06T07:27:52.836950+0000","last_unstale":"2024-03-06T07:27:54.228399+0000","last_undegraded":"2024-03-06T07:27:54.228399+0000","last_fullsized":"2024-03-06T07:27:54.228399+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:21.929558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.e","version":"182'2","reported_seq":16,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824397+0000","last_change":"2024-03-06T07:27:51.828775+0000","last_active":"2024-03-06T07:27:53.824397+0000","last_peered":"2024-03-06T07:27:53.824397+0000","last_clean":"2024-03-06T07:27:53.824397+0000","last_became_active":"2024-03-06T07:27:51.828483+0000","last_became_peered":"2024-03-06T07:27:51.828483+0000","last_unstale":"2024-03-06T07:27:53.824397+0000","last_undegraded":"2024-03-06T07:27:53.824397+0000","last_fullsized":"2024-03-06T07:27:53.824397+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:14.011370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827853+0000","last_change":"2024-03-06T07:27:52.845785+0000","last_active":"2024-03-06T07:27:53.827853+0000","last_peered":"2024-03-06T07:27:53.827853+0000","last_clean":"2024-03-06T07:27:53.827853+0000","last_became_active":"2024-03-06T07:27:52.845523+0000","last_became_peered":"2024-03-06T07:27:52.845523+0000","last_unstale":"2024-03-06T07:27:53.827853+0000","last_undegraded":"2024-03-06T07:27:53.827853+0000","last_fullsized":"2024-03-06T07:27:53.827853+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:44.197118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"182'1","reported_seq":15,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228897+0000","last_change":"2024-03-06T07:27:51.828077+0000","last_active":"2024-03-06T07:27:54.228897+0000","last_peered":"2024-03-06T07:27:54.228897+0000","last_clean":"2024-03-06T07:27:54.228897+0000","last_became_active":"2024-03-06T07:27:51.827762+0000","last_became_peered":"2024-03-06T07:27:51.827762+0000","last_unstale":"2024-03-06T07:27:54.228897+0000","last_undegraded":"2024-03-06T07:27:54.228897+0000","last_fullsized":"2024-03-06T07:27:54.228897+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:44.093607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229327+0000","last_change":"2024-03-06T07:27:52.850839+0000","last_active":"2024-03-06T07:27:54.229327+0000","last_peered":"2024-03-06T07:27:54.229327+0000","last_clean":"2024-03-06T07:27:54.229327+0000","last_became_active":"2024-03-06T07:27:52.850576+0000","last_became_peered":"2024-03-06T07:27:52.850576+0000","last_unstale":"2024-03-06T07:27:54.229327+0000","last_undegraded":"2024-03-06T07:27:54.229327+0000","last_fullsized":"2024-03-06T07:27:54.229327+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:59.783306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229199+0000","last_change":"2024-03-06T07:27:51.828930+0000","last_active":"2024-03-06T07:27:54.229199+0000","last_peered":"2024-03-06T07:27:54.229199+0000","last_clean":"2024-03-06T07:27:54.229199+0000","last_became_active":"2024-03-06T07:27:51.828653+0000","last_became_peered":"2024-03-06T07:27:51.828653+0000","last_unstale":"2024-03-06T07:27:54.229199+0000","last_undegraded":"2024-03-06T07:27:54.229199+0000","last_fullsized":"2024-03-06T07:27:54.229199+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:39.081353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227932+0000","last_change":"2024-03-06T07:27:52.851231+0000","last_active":"2024-03-06T07:27:54.227932+0000","last_peered":"2024-03-06T07:27:54.227932+0000","last_clean":"2024-03-06T07:27:54.227932+0000","last_became_active":"2024-03-06T07:27:52.850951+0000","last_became_peered":"2024-03-06T07:27:52.850951+0000","last_unstale":"2024-03-06T07:27:54.227932+0000","last_undegraded":"2024-03-06T07:27:54.227932+0000","last_fullsized":"2024-03-06T07:27:54.227932+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:04.131892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"182'2","reported_seq":16,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229064+0000","last_change":"2024-03-06T07:27:51.830105+0000","last_active":"2024-03-06T07:27:54.229064+0000","last_peered":"2024-03-06T07:27:54.229064+0000","last_clean":"2024-03-06T07:27:54.229064+0000","last_became_active":"2024-03-06T07:27:51.829820+0000","last_became_peered":"2024-03-06T07:27:51.829820+0000","last_unstale":"2024-03-06T07:27:54.229064+0000","last_undegraded":"2024-03-06T07:27:54.229064+0000","last_fullsized":"2024-03-06T07:27:54.229064+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:43.256668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227999+0000","last_change":"2024-03-06T07:27:52.843200+0000","last_active":"2024-03-06T07:27:54.227999+0000","last_peered":"2024-03-06T07:27:54.227999+0000","last_clean":"2024-03-06T07:27:54.227999+0000","last_became_active":"2024-03-06T07:27:52.842838+0000","last_became_peered":"2024-03-06T07:27:52.842838+0000","last_unstale":"2024-03-06T07:27:54.227999+0000","last_undegraded":"2024-03-06T07:27:54.227999+0000","last_fullsized":"2024-03-06T07:27:54.227999+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:31.323213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842712+0000","last_change":"2024-03-06T07:27:51.831216+0000","last_active":"2024-03-06T07:27:53.842712+0000","last_peered":"2024-03-06T07:27:53.842712+0000","last_clean":"2024-03-06T07:27:53.842712+0000","last_became_active":"2024-03-06T07:27:51.830947+0000","last_became_peered":"2024-03-06T07:27:51.830947+0000","last_unstale":"2024-03-06T07:27:53.842712+0000","last_undegraded":"2024-03-06T07:27:53.842712+0000","last_fullsized":"2024-03-06T07:27:53.842712+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:29.394867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227231+0000","last_change":"2024-03-06T07:27:52.851117+0000","last_active":"2024-03-06T07:27:54.227231+0000","last_peered":"2024-03-06T07:27:54.227231+0000","last_clean":"2024-03-06T07:27:54.227231+0000","last_became_active":"2024-03-06T07:27:52.850801+0000","last_became_peered":"2024-03-06T07:27:52.850801+0000","last_unstale":"2024-03-06T07:27:54.227231+0000","last_undegraded":"2024-03-06T07:27:54.227231+0000","last_fullsized":"2024-03-06T07:27:54.227231+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:50.603524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842581+0000","last_change":"2024-03-06T07:27:51.832578+0000","last_active":"2024-03-06T07:27:53.842581+0000","last_peered":"2024-03-06T07:27:53.842581+0000","last_clean":"2024-03-06T07:27:53.842581+0000","last_became_active":"2024-03-06T07:27:51.832234+0000","last_became_peered":"2024-03-06T07:27:51.832234+0000","last_unstale":"2024-03-06T07:27:53.842581+0000","last_undegraded":"2024-03-06T07:27:53.842581+0000","last_fullsized":"2024-03-06T07:27:53.842581+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:10.808514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827527+0000","last_change":"2024-03-06T07:27:52.852438+0000","last_active":"2024-03-06T07:27:53.827527+0000","last_peered":"2024-03-06T07:27:53.827527+0000","last_clean":"2024-03-06T07:27:53.827527+0000","last_became_active":"2024-03-06T07:27:52.851964+0000","last_became_peered":"2024-03-06T07:27:52.851964+0000","last_unstale":"2024-03-06T07:27:53.827527+0000","last_undegraded":"2024-03-06T07:27:53.827527+0000","last_fullsized":"2024-03-06T07:27:53.827527+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:28.314013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824174+0000","last_change":"2024-03-06T07:27:51.829009+0000","last_active":"2024-03-06T07:27:53.824174+0000","last_peered":"2024-03-06T07:27:53.824174+0000","last_clean":"2024-03-06T07:27:53.824174+0000","last_became_active":"2024-03-06T07:27:51.828505+0000","last_became_peered":"2024-03-06T07:27:51.828505+0000","last_unstale":"2024-03-06T07:27:53.824174+0000","last_undegraded":"2024-03-06T07:27:53.824174+0000","last_fullsized":"2024-03-06T07:27:53.824174+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:03.559231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227324+0000","last_change":"2024-03-06T07:27:52.827928+0000","last_active":"2024-03-06T07:27:54.227324+0000","last_peered":"2024-03-06T07:27:54.227324+0000","last_clean":"2024-03-06T07:27:54.227324+0000","last_became_active":"2024-03-06T07:27:52.827635+0000","last_became_peered":"2024-03-06T07:27:52.827635+0000","last_unstale":"2024-03-06T07:27:54.227324+0000","last_undegraded":"2024-03-06T07:27:54.227324+0000","last_fullsized":"2024-03-06T07:27:54.227324+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:09.730573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.7","version":"182'2","reported_seq":16,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229256+0000","last_change":"2024-03-06T07:27:51.825754+0000","last_active":"2024-03-06T07:27:54.229256+0000","last_peered":"2024-03-06T07:27:54.229256+0000","last_clean":"2024-03-06T07:27:54.229256+0000","last_became_active":"2024-03-06T07:27:51.825459+0000","last_became_peered":"2024-03-06T07:27:51.825459+0000","last_unstale":"2024-03-06T07:27:54.229256+0000","last_undegraded":"2024-03-06T07:27:54.229256+0000","last_fullsized":"2024-03-06T07:27:54.229256+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:41.300428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228483+0000","last_change":"2024-03-06T07:27:52.843802+0000","last_active":"2024-03-06T07:27:54.228483+0000","last_peered":"2024-03-06T07:27:54.228483+0000","last_clean":"2024-03-06T07:27:54.228483+0000","last_became_active":"2024-03-06T07:27:52.843472+0000","last_became_peered":"2024-03-06T07:27:52.843472+0000","last_unstale":"2024-03-06T07:27:54.228483+0000","last_undegraded":"2024-03-06T07:27:54.228483+0000","last_fullsized":"2024-03-06T07:27:54.228483+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:01.915969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"183'1","reported_seq":15,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.844256+0000","last_change":"2024-03-06T07:27:51.834543+0000","last_active":"2024-03-06T07:27:53.844256+0000","last_peered":"2024-03-06T07:27:53.844256+0000","last_clean":"2024-03-06T07:27:53.844256+0000","last_became_active":"2024-03-06T07:27:51.834294+0000","last_became_peered":"2024-03-06T07:27:51.834294+0000","last_unstale":"2024-03-06T07:27:53.844256+0000","last_undegraded":"2024-03-06T07:27:53.844256+0000","last_fullsized":"2024-03-06T07:27:53.844256+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:11.425236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.116961+0000","last_change":"2024-03-06T07:27:52.847982+0000","last_active":"2024-03-06T07:27:54.116961+0000","last_peered":"2024-03-06T07:27:54.116961+0000","last_clean":"2024-03-06T07:27:54.116961+0000","last_became_active":"2024-03-06T07:27:52.847477+0000","last_became_peered":"2024-03-06T07:27:52.847477+0000","last_unstale":"2024-03-06T07:27:54.116961+0000","last_undegraded":"2024-03-06T07:27:54.116961+0000","last_fullsized":"2024-03-06T07:27:54.116961+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:18.817690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.842959+0000","last_change":"2024-03-06T07:27:51.825939+0000","last_active":"2024-03-06T07:27:53.842959+0000","last_peered":"2024-03-06T07:27:53.842959+0000","last_clean":"2024-03-06T07:27:53.842959+0000","last_became_active":"2024-03-06T07:27:51.825569+0000","last_became_peered":"2024-03-06T07:27:51.825569+0000","last_unstale":"2024-03-06T07:27:53.842959+0000","last_undegraded":"2024-03-06T07:27:53.842959+0000","last_fullsized":"2024-03-06T07:27:53.842959+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:17.071666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824502+0000","last_change":"2024-03-06T07:27:52.828053+0000","last_active":"2024-03-06T07:27:53.824502+0000","last_peered":"2024-03-06T07:27:53.824502+0000","last_clean":"2024-03-06T07:27:53.824502+0000","last_became_active":"2024-03-06T07:27:52.827582+0000","last_became_peered":"2024-03-06T07:27:52.827582+0000","last_unstale":"2024-03-06T07:27:53.824502+0000","last_undegraded":"2024-03-06T07:27:53.824502+0000","last_fullsized":"2024-03-06T07:27:53.824502+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:36.756407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229210+0000","last_change":"2024-03-06T07:27:51.829977+0000","last_active":"2024-03-06T07:27:54.229210+0000","last_peered":"2024-03-06T07:27:54.229210+0000","last_clean":"2024-03-06T07:27:54.229210+0000","last_became_active":"2024-03-06T07:27:51.829718+0000","last_became_peered":"2024-03-06T07:27:51.829718+0000","last_unstale":"2024-03-06T07:27:54.229210+0000","last_undegraded":"2024-03-06T07:27:54.229210+0000","last_fullsized":"2024-03-06T07:27:54.229210+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:59.505668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827316+0000","last_change":"2024-03-06T07:27:52.853302+0000","last_active":"2024-03-06T07:27:53.827316+0000","last_peered":"2024-03-06T07:27:53.827316+0000","last_clean":"2024-03-06T07:27:53.827316+0000","last_became_active":"2024-03-06T07:27:52.852745+0000","last_became_peered":"2024-03-06T07:27:52.852745+0000","last_unstale":"2024-03-06T07:27:53.827316+0000","last_undegraded":"2024-03-06T07:27:53.827316+0000","last_fullsized":"2024-03-06T07:27:53.827316+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:03.769930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.229136+0000","last_change":"2024-03-06T07:27:51.828299+0000","last_active":"2024-03-06T07:27:54.229136+0000","last_peered":"2024-03-06T07:27:54.229136+0000","last_clean":"2024-03-06T07:27:54.229136+0000","last_became_active":"2024-03-06T07:27:51.827999+0000","last_became_peered":"2024-03-06T07:27:51.827999+0000","last_unstale":"2024-03-06T07:27:54.229136+0000","last_undegraded":"2024-03-06T07:27:54.229136+0000","last_fullsized":"2024-03-06T07:27:54.229136+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:14:58.361793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827379+0000","last_change":"2024-03-06T07:27:52.840395+0000","last_active":"2024-03-06T07:27:53.827379+0000","last_peered":"2024-03-06T07:27:53.827379+0000","last_clean":"2024-03-06T07:27:53.827379+0000","last_became_active":"2024-03-06T07:27:52.840117+0000","last_became_peered":"2024-03-06T07:27:52.840117+0000","last_unstale":"2024-03-06T07:27:53.827379+0000","last_undegraded":"2024-03-06T07:27:53.827379+0000","last_fullsized":"2024-03-06T07:27:53.827379+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:26.160506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.828367+0000","last_change":"2024-03-06T07:27:51.830896+0000","last_active":"2024-03-06T07:27:53.828367+0000","last_peered":"2024-03-06T07:27:53.828367+0000","last_clean":"2024-03-06T07:27:53.828367+0000","last_became_active":"2024-03-06T07:27:51.830630+0000","last_became_peered":"2024-03-06T07:27:51.830630+0000","last_unstale":"2024-03-06T07:27:53.828367+0000","last_undegraded":"2024-03-06T07:27:53.828367+0000","last_fullsized":"2024-03-06T07:27:53.828367+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:53.349144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.824496+0000","last_change":"2024-03-06T07:27:52.831089+0000","last_active":"2024-03-06T07:27:53.824496+0000","last_peered":"2024-03-06T07:27:53.824496+0000","last_clean":"2024-03-06T07:27:53.824496+0000","last_became_active":"2024-03-06T07:27:52.830222+0000","last_became_peered":"2024-03-06T07:27:52.830222+0000","last_unstale":"2024-03-06T07:27:53.824496+0000","last_undegraded":"2024-03-06T07:27:53.824496+0000","last_fullsized":"2024-03-06T07:27:53.824496+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:11.268762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228227+0000","last_change":"2024-03-06T07:27:51.834741+0000","last_active":"2024-03-06T07:27:54.228227+0000","last_peered":"2024-03-06T07:27:54.228227+0000","last_clean":"2024-03-06T07:27:54.228227+0000","last_became_active":"2024-03-06T07:27:51.833107+0000","last_became_peered":"2024-03-06T07:27:51.833107+0000","last_unstale":"2024-03-06T07:27:54.228227+0000","last_undegraded":"2024-03-06T07:27:54.228227+0000","last_fullsized":"2024-03-06T07:27:54.228227+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:51.680814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228937+0000","last_change":"2024-03-06T07:27:52.850358+0000","last_active":"2024-03-06T07:27:54.228937+0000","last_peered":"2024-03-06T07:27:54.228937+0000","last_clean":"2024-03-06T07:27:54.228937+0000","last_became_active":"2024-03-06T07:27:52.850040+0000","last_became_peered":"2024-03-06T07:27:52.850040+0000","last_unstale":"2024-03-06T07:27:54.228937+0000","last_undegraded":"2024-03-06T07:27:54.228937+0000","last_fullsized":"2024-03-06T07:27:54.228937+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:08.654039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228178+0000","last_change":"2024-03-06T07:27:51.833317+0000","last_active":"2024-03-06T07:27:54.228178+0000","last_peered":"2024-03-06T07:27:54.228178+0000","last_clean":"2024-03-06T07:27:54.228178+0000","last_became_active":"2024-03-06T07:27:51.828026+0000","last_became_peered":"2024-03-06T07:27:51.828026+0000","last_unstale":"2024-03-06T07:27:54.228178+0000","last_undegraded":"2024-03-06T07:27:54.228178+0000","last_fullsized":"2024-03-06T07:27:54.228178+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:09.527421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:53.827701+0000","last_change":"2024-03-06T07:27:52.851417+0000","last_active":"2024-03-06T07:27:53.827701+0000","last_peered":"2024-03-06T07:27:53.827701+0000","last_clean":"2024-03-06T07:27:53.827701+0000","last_became_active":"2024-03-06T07:27:52.851110+0000","last_became_peered":"2024-03-06T07:27:52.851110+0000","last_unstale":"2024-03-06T07:27:53.827701+0000","last_undegraded":"2024-03-06T07:27:53.827701+0000","last_fullsized":"2024-03-06T07:27:53.827701+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:51.808711+0000","last_clean_scrub_stamp":"2024-03-06T07:27:51.808711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:22.148483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1c","version":"182'2","reported_seq":16,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.228051+0000","last_change":"2024-03-06T07:27:51.829812+0000","last_active":"2024-03-06T07:27:54.228051+0000","last_peered":"2024-03-06T07:27:54.228051+0000","last_clean":"2024-03-06T07:27:54.228051+0000","last_became_active":"2024-03-06T07:27:51.829502+0000","last_became_peered":"2024-03-06T07:27:51.829502+0000","last_unstale":"2024-03-06T07:27:54.228051+0000","last_undegraded":"2024-03-06T07:27:54.228051+0000","last_fullsized":"2024-03-06T07:27:54.228051+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:50.804504+0000","last_clean_scrub_stamp":"2024-03-06T07:27:50.804504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:20.358143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":483,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-06T07:27:54.227820+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:27:54.227820+0000","last_peered":"2024-03-06T07:27:54.227820+0000","last_clean":"2024-03-06T07:27:54.227820+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:27:54.227820+0000","last_undegraded":"2024-03-06T07:27:54.227820+0000","last_fullsized":"2024-03-06T07:27:54.227820+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574943,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33512,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338328,"statfs":{"total":96636764160,"available":96602447872,"internally_reserved":0,"allocated":2379776,"data_stored":1714023,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33464,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":2330624,"data_stored":1124721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574942,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33460,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338380,"statfs":{"total":96636764160,"available":96602501120,"internally_reserved":0,"allocated":2334720,"data_stored":1131357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574943,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33448,"kb_used_data":2268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338392,"statfs":{"total":96636764160,"available":96602513408,"internally_reserved":0,"allocated":2322432,"data_stored":1129765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574942,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33448,"kb_used_data":2268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338392,"statfs":{"total":96636764160,"available":96602513408,"internally_reserved":0,"allocated":2322432,"data_stored":1129697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574942,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37612,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334228,"statfs":{"total":96636764160,"available":96598249472,"internally_reserved":0,"allocated":2326528,"data_stored":1130291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33444,"kb_used_data":2264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338396,"statfs":{"total":96636764160,"available":96602517504,"internally_reserved":0,"allocated":2318336,"data_stored":1129675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574943,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37664,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334176,"statfs":{"total":96636764160,"available":96598196224,"internally_reserved":0,"allocated":2371584,"data_stored":1713351,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:56.282 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:27:56.282 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:27:56.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.424+0000 7f43e44d4640 1 Processor -- start 2024-03-06T07:27:56.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.424+0000 7f43e44d4640 1 -- start start 2024-03-06T07:27:56.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e44d4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43dc152820 0x7f43dc0c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e44d4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43dc0c7180 con 0x7f43dc0c0670 2024-03-06T07:27:56.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e44d4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43dc155460 con 0x7f43dc0af6f0 2024-03-06T07:27:56.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e44d4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43dc0c6bc0 con 0x7f43dc152820 2024-03-06T07:27:56.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e1a6f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f43dc0c0670 0x7f43dc0c73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50604/0 (socket says 172.21.15.1:50604) 2024-03-06T07:27:56.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e1a6f640 1 -- 172.21.15.1:0/3081153592 learned_addr learned my addr 172.21.15.1:0/3081153592 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:56.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e2270640 1 --2- 172.21.15.1:0/3081153592 >> v2:172.21.15.98:3300/0 conn(0x7f43dc152820 0x7f43dc0c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e126e640 1 -- 172.21.15.1:0/3081153592 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618895643 0 0) 0x7f43dc0c7180 con 0x7f43dc0c0670 2024-03-06T07:27:56.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e126e640 1 -- 172.21.15.1:0/3081153592 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43c8003520 con 0x7f43dc0c0670 2024-03-06T07:27:56.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e126e640 1 -- 172.21.15.1:0/3081153592 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4142723690 0 0) 0x7f43c8003520 con 0x7f43dc0c0670 2024-03-06T07:27:56.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e126e640 1 -- 172.21.15.1:0/3081153592 >> v2:172.21.15.98:3300/0 conn(0x7f43dc152820 msgr2=0x7f43dc0c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e126e640 1 --2- 172.21.15.1:0/3081153592 >> v2:172.21.15.98:3300/0 conn(0x7f43dc152820 0x7f43dc0c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e126e640 1 -- 172.21.15.1:0/3081153592 >> v1:172.21.15.98:6789/0 conn(0x7f43dc0af6f0 legacy=0x7f43dc15fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e126e640 1 -- 172.21.15.1:0/3081153592 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43dc0c6ea0 con 0x7f43dc0c0670 2024-03-06T07:27:56.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e126e640 1 -- 172.21.15.1:0/3081153592 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f43d8001670 con 0x7f43dc0c0670 2024-03-06T07:27:56.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e126e640 1 -- 172.21.15.1:0/3081153592 >> v1:172.21.15.1:6789/0 conn(0x7f43dc0c0670 legacy=0x7f43dc0c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e126e640 1 --2- 172.21.15.1:0/3081153592 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43c80039b0 0x7f43c8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e126e640 1 --2- 172.21.15.1:0/3081153592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43c8004800 0x7f43c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e126e640 1 --2- 172.21.15.1:0/3081153592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43c80051c0 0x7f43c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e126e640 1 -- 172.21.15.1:0/3081153592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc0c6bc0 con 0x7f43c80051c0 2024-03-06T07:27:56.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e126e640 1 -- 172.21.15.1:0/3081153592 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c800c060 con 0x7f43c8004800 2024-03-06T07:27:56.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e126e640 1 -- 172.21.15.1:0/3081153592 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c800c260 con 0x7f43c80039b0 2024-03-06T07:27:56.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e2a71640 1 --2- 172.21.15.1:0/3081153592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43c80051c0 0x7f43c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e1a6f640 1 --2- 172.21.15.1:0/3081153592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43c8004800 0x7f43c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e2270640 1 --2- 172.21.15.1:0/3081153592 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43c80039b0 0x7f43c8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e2a71640 1 -- 172.21.15.1:0/3081153592 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43c80039b0 msgr2=0x7f43c8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e2a71640 1 --2- 172.21.15.1:0/3081153592 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43c80039b0 0x7f43c8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e2a71640 1 -- 172.21.15.1:0/3081153592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43c8004800 msgr2=0x7f43c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e2a71640 1 --2- 172.21.15.1:0/3081153592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43c8004800 0x7f43c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.426+0000 7f43e2a71640 1 -- 172.21.15.1:0/3081153592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c8003520 con 0x7f43c80051c0 2024-03-06T07:27:56.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.427+0000 7f43e2a71640 1 -- 172.21.15.1:0/3081153592 reap_dead start 2024-03-06T07:27:56.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.427+0000 7f43e2a71640 1 --2- 172.21.15.1:0/3081153592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43c80051c0 0x7f43c800b9c0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f43d4001e50 tx=0x7f43c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da9906517e7af8c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.427+0000 7f43e126e640 1 -- 172.21.15.1:0/3081153592 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d4001a70 con 0x7f43c80051c0 2024-03-06T07:27:56.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.427+0000 7f43e126e640 1 -- 172.21.15.1:0/3081153592 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d4003bb0 con 0x7f43c80051c0 2024-03-06T07:27:56.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.427+0000 7f43e126e640 1 -- 172.21.15.1:0/3081153592 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d40050d0 con 0x7f43c80051c0 2024-03-06T07:27:56.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.427+0000 7f43e44d4640 1 -- 172.21.15.1:0/3081153592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43c80051c0 msgr2=0x7f43c800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.427+0000 7f43e44d4640 1 --2- 172.21.15.1:0/3081153592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43c80051c0 0x7f43c800b9c0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f43d4001e50 tx=0x7f43c8005830 comp rx=0 tx=0).stop 2024-03-06T07:27:56.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.428+0000 7f43e44d4640 1 -- 172.21.15.1:0/3081153592 shutdown_connections 2024-03-06T07:27:56.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.428+0000 7f43e44d4640 1 --2- 172.21.15.1:0/3081153592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43c80051c0 0x7f43c800b9c0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.428+0000 7f43e44d4640 1 -- 172.21.15.1:0/3081153592 >> 172.21.15.1:0/3081153592 conn(0x7f43dc072720 msgr2=0x7f43dc072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.428+0000 7f43e44d4640 1 -- 172.21.15.1:0/3081153592 shutdown_connections 2024-03-06T07:27:56.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.428+0000 7f43e44d4640 1 -- 172.21.15.1:0/3081153592 wait complete. 2024-03-06T07:27:56.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.429+0000 7f43e44d4640 1 Processor -- start 2024-03-06T07:27:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.429+0000 7f43e44d4640 1 -- start start 2024-03-06T07:27:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.429+0000 7f43e44d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43c80051c0 0x7f43dc0aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.429+0000 7f43e44d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43dc1ee7b0 0x7f43dc154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.429+0000 7f43e44d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43dc0c2660 0x7f43dc157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.429+0000 7f43e44d4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc152d80 con 0x7f43c80051c0 2024-03-06T07:27:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.429+0000 7f43e44d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc152c00 con 0x7f43dc1ee7b0 2024-03-06T07:27:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.429+0000 7f43e44d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc0afc40 con 0x7f43dc0c2660 2024-03-06T07:27:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.430+0000 7f43e2270640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43c80051c0 0x7f43dc0aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.430+0000 7f43e2270640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43c80051c0 0x7f43dc0aed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60464/0 (socket says 172.21.15.1:60464) 2024-03-06T07:27:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.430+0000 7f43e2270640 1 -- 172.21.15.1:0/2800072594 learned_addr learned my addr 172.21.15.1:0/2800072594 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.430+0000 7f43e2a71640 1 --2- 172.21.15.1:0/2800072594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43dc0c2660 0x7f43dc157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.430+0000 7f43e1a6f640 1 --2- 172.21.15.1:0/2800072594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43dc1ee7b0 0x7f43dc154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.430+0000 7f43e2270640 1 -- 172.21.15.1:0/2800072594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43dc0c2660 msgr2=0x7f43dc157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.430+0000 7f43e2270640 1 --2- 172.21.15.1:0/2800072594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43dc0c2660 0x7f43dc157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.430+0000 7f43e2270640 1 -- 172.21.15.1:0/2800072594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43dc1ee7b0 msgr2=0x7f43dc154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.430+0000 7f43e2270640 1 --2- 172.21.15.1:0/2800072594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43dc1ee7b0 0x7f43dc154520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.430+0000 7f43e2270640 1 -- 172.21.15.1:0/2800072594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43dc0af260 con 0x7f43c80051c0 2024-03-06T07:27:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.430+0000 7f43e2270640 1 --2- 172.21.15.1:0/2800072594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43c80051c0 0x7f43dc0aed20 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f43d00118e0 tx=0x7f43d0011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.430+0000 7f43ceffd640 1 -- 172.21.15.1:0/2800072594 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d000faf0 con 0x7f43c80051c0 2024-03-06T07:27:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.430+0000 7f43ceffd640 1 -- 172.21.15.1:0/2800072594 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d000fc90 con 0x7f43c80051c0 2024-03-06T07:27:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.430+0000 7f43e44d4640 1 -- 172.21.15.1:0/2800072594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43dc159490 con 0x7f43c80051c0 2024-03-06T07:27:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.430+0000 7f43ceffd640 1 -- 172.21.15.1:0/2800072594 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d0014440 con 0x7f43c80051c0 2024-03-06T07:27:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.430+0000 7f43e44d4640 1 -- 172.21.15.1:0/2800072594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43dc1edfb0 con 0x7f43c80051c0 2024-03-06T07:27:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.431+0000 7f43e44d4640 1 -- 172.21.15.1:0/2800072594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43dc0686e0 con 0x7f43c80051c0 2024-03-06T07:27:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.432+0000 7f43ceffd640 1 -- 172.21.15.1:0/2800072594 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43d0012040 con 0x7f43c80051c0 2024-03-06T07:27:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.432+0000 7f43ceffd640 1 --2- 172.21.15.1:0/2800072594 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43c006b500 0x7f43c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.432+0000 7f43ceffd640 1 -- 172.21.15.1:0/2800072594 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f43d008a9c0 con 0x7f43c80051c0 2024-03-06T07:27:56.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.433+0000 7f43e1a6f640 1 --2- 172.21.15.1:0/2800072594 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43c006b500 0x7f43c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.433+0000 7f43e1a6f640 1 --2- 172.21.15.1:0/2800072594 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43c006b500 0x7f43c006d9c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f43dc06dab0 tx=0x7f43d8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.437+0000 7f43ceffd640 1 -- 172.21.15.1:0/2800072594 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43d008ce20 con 0x7f43c80051c0 2024-03-06T07:27:56.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.689+0000 7f43e44d4640 1 -- 172.21.15.1:0/2800072594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f43dc0c5b40 con 0x7f43c80051c0 2024-03-06T07:27:56.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.691+0000 7f43ceffd640 1 -- 172.21.15.1:0/2800072594 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+147 (secure 0 0 0) 0x7f43d0019030 con 0x7f43c80051c0 2024-03-06T07:27:56.693 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-06T07:27:56.693 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng== 2024-03-06T07:27:56.693 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-06T07:27:56.693 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-06T07:27:56.693 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:56.694 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-06T07:27:56.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.693+0000 7f43e44d4640 1 -- 172.21.15.1:0/2800072594 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43c006b500 msgr2=0x7f43c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.693+0000 7f43e44d4640 1 --2- 172.21.15.1:0/2800072594 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43c006b500 0x7f43c006d9c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f43dc06dab0 tx=0x7f43d8009040 comp rx=0 tx=0).stop 2024-03-06T07:27:56.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.693+0000 7f43e44d4640 1 -- 172.21.15.1:0/2800072594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43c80051c0 msgr2=0x7f43dc0aed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.693+0000 7f43e44d4640 1 --2- 172.21.15.1:0/2800072594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43c80051c0 0x7f43dc0aed20 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f43d00118e0 tx=0x7f43d0011da0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.693+0000 7f43e44d4640 1 -- 172.21.15.1:0/2800072594 shutdown_connections 2024-03-06T07:27:56.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.693+0000 7f43e44d4640 1 --2- 172.21.15.1:0/2800072594 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43c006b500 0x7f43c006d9c0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.693+0000 7f43e44d4640 1 --2- 172.21.15.1:0/2800072594 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43dc0c2660 0x7f43dc157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.693+0000 7f43e44d4640 1 --2- 172.21.15.1:0/2800072594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43dc1ee7b0 0x7f43dc154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.693+0000 7f43e44d4640 1 --2- 172.21.15.1:0/2800072594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43c80051c0 0x7f43dc0aed20 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.693+0000 7f43e44d4640 1 -- 172.21.15.1:0/2800072594 >> 172.21.15.1:0/2800072594 conn(0x7f43dc072720 msgr2=0x7f43dc151190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.693+0000 7f43e44d4640 1 -- 172.21.15.1:0/2800072594 shutdown_connections 2024-03-06T07:27:56.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.693+0000 7f43e44d4640 1 -- 172.21.15.1:0/2800072594 wait complete. 2024-03-06T07:27:56.706 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:56.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.829+0000 7ff9e4caa640 1 Processor -- start 2024-03-06T07:27:56.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.829+0000 7ff9e4caa640 1 -- start start 2024-03-06T07:27:56.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.829+0000 7ff9e4caa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9e0150fe0 0x7ff9e01513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.829+0000 7ff9e4caa640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff9e0072f20 0x7ff9e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.829+0000 7ff9e4caa640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e00a6140 con 0x7ff9e006f2f0 2024-03-06T07:27:56.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.829+0000 7ff9e4caa640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e01526b0 con 0x7ff9e0072f20 2024-03-06T07:27:56.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.830+0000 7ff9e4caa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e0151aa0 con 0x7ff9e0150fe0 2024-03-06T07:27:56.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.830+0000 7ff9ddd9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff9e0072f20 0x7ff9e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.830+0000 7ff9de59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9e0150fe0 0x7ff9e01513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.830+0000 7ff9ddd9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff9e0072f20 0x7ff9e006edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:56404/0 (socket says 172.21.15.1:56404) 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.830+0000 7ff9de59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9e0150fe0 0x7ff9e01513c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:33554/0 (socket says 172.21.15.1:33554) 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.830+0000 7ff9ddd9b640 1 -- 172.21.15.1:0/1007891564 learned_addr learned my addr 172.21.15.1:0/1007891564 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.830+0000 7ff9de59c640 1 -- 172.21.15.1:0/1007891564 >> v2:172.21.15.98:3300/0 conn(0x7ff9e0072f20 msgr2=0x7ff9e006edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.830+0000 7ff9de59c640 1 --2- 172.21.15.1:0/1007891564 >> v2:172.21.15.98:3300/0 conn(0x7ff9e0072f20 0x7ff9e006edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.830+0000 7ff9de59c640 1 -- 172.21.15.1:0/1007891564 >> v1:172.21.15.98:6789/0 conn(0x7ff9e006f2f0 legacy=0x7ff9e0071f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.830+0000 7ff9de59c640 1 -- 172.21.15.1:0/1007891564 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e0151d80 con 0x7ff9e0150fe0 2024-03-06T07:27:56.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.830+0000 7ff9ddd9b640 1 --2- 172.21.15.1:0/1007891564 >> v2:172.21.15.98:3300/0 conn(0x7ff9e0072f20 0x7ff9e006edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:56.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.831+0000 7ff9de59c640 1 --2- 172.21.15.1:0/1007891564 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9e0150fe0 0x7ff9e01513c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009d90 tx=0x7ff9d002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb397c9aa964751b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.831+0000 7ff9dd59a640 1 -- 172.21.15.1:0/1007891564 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0039070 con 0x7ff9e0150fe0 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.831+0000 7ff9dd59a640 1 -- 172.21.15.1:0/1007891564 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d002cbc0 con 0x7ff9e0150fe0 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.831+0000 7ff9dd59a640 1 -- 172.21.15.1:0/1007891564 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0034440 con 0x7ff9e0150fe0 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.832+0000 7ff9e4caa640 1 -- 172.21.15.1:0/1007891564 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9e0150fe0 msgr2=0x7ff9e01513c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.832+0000 7ff9e4caa640 1 --2- 172.21.15.1:0/1007891564 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9e0150fe0 0x7ff9e01513c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009d90 tx=0x7ff9d002be10 comp rx=0 tx=0).stop 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.832+0000 7ff9e4caa640 1 -- 172.21.15.1:0/1007891564 shutdown_connections 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.832+0000 7ff9e4caa640 1 --2- 172.21.15.1:0/1007891564 >> v2:172.21.15.98:3300/0 conn(0x7ff9e0072f20 0x7ff9e006edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.832+0000 7ff9e4caa640 1 --2- 172.21.15.1:0/1007891564 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9e0150fe0 0x7ff9e01513c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.832+0000 7ff9e4caa640 1 -- 172.21.15.1:0/1007891564 >> 172.21.15.1:0/1007891564 conn(0x7ff9e00c6860 msgr2=0x7ff9e00c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.832+0000 7ff9e4caa640 1 -- 172.21.15.1:0/1007891564 shutdown_connections 2024-03-06T07:27:56.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.832+0000 7ff9e4caa640 1 -- 172.21.15.1:0/1007891564 wait complete. 2024-03-06T07:27:56.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.833+0000 7ff9e4caa640 1 Processor -- start 2024-03-06T07:27:56.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.833+0000 7ff9e4caa640 1 -- start start 2024-03-06T07:27:56.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.834+0000 7ff9e4caa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9e006f2f0 0x7ff9e00bfb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.834+0000 7ff9e4caa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9e0072f20 0x7ff9e0154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.834+0000 7ff9e4caa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9e0150fe0 0x7ff9e0143b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.834+0000 7ff9e4caa640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e00726a0 con 0x7ff9e0150fe0 2024-03-06T07:27:56.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.834+0000 7ff9e4caa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e0152340 con 0x7ff9e006f2f0 2024-03-06T07:27:56.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.834+0000 7ff9e4caa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e0072820 con 0x7ff9e0072f20 2024-03-06T07:27:56.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.834+0000 7ff9ded9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9e0150fe0 0x7ff9e0143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.834+0000 7ff9ded9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9e0150fe0 0x7ff9e0143b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60472/0 (socket says 172.21.15.1:60472) 2024-03-06T07:27:56.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.834+0000 7ff9ded9d640 1 -- 172.21.15.1:0/2476967888 learned_addr learned my addr 172.21.15.1:0/2476967888 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:56.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.834+0000 7ff9de59c640 1 --2- 172.21.15.1:0/2476967888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9e006f2f0 0x7ff9e00bfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.834+0000 7ff9ddd9b640 1 --2- 172.21.15.1:0/2476967888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9e0072f20 0x7ff9e0154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.834+0000 7ff9ded9d640 1 -- 172.21.15.1:0/2476967888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9e0072f20 msgr2=0x7ff9e0154e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.834+0000 7ff9ded9d640 1 --2- 172.21.15.1:0/2476967888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9e0072f20 0x7ff9e0154e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.834+0000 7ff9ded9d640 1 -- 172.21.15.1:0/2476967888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9e006f2f0 msgr2=0x7ff9e00bfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.834+0000 7ff9ded9d640 1 --2- 172.21.15.1:0/2476967888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9e006f2f0 0x7ff9e00bfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.834+0000 7ff9ded9d640 1 -- 172.21.15.1:0/2476967888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e00beaa0 con 0x7ff9e0150fe0 2024-03-06T07:27:56.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.835+0000 7ff9ded9d640 1 --2- 172.21.15.1:0/2476967888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9e0150fe0 0x7ff9e0143b40 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800c400 tx=0x7ff9d800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.835+0000 7ff9d6ffd640 1 -- 172.21.15.1:0/2476967888 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8017070 con 0x7ff9e0150fe0 2024-03-06T07:27:56.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.835+0000 7ff9d6ffd640 1 -- 172.21.15.1:0/2476967888 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d80052d0 con 0x7ff9e0150fe0 2024-03-06T07:27:56.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.835+0000 7ff9d6ffd640 1 -- 172.21.15.1:0/2476967888 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8003810 con 0x7ff9e0150fe0 2024-03-06T07:27:56.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.835+0000 7ff9e4caa640 1 -- 172.21.15.1:0/2476967888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e00bed40 con 0x7ff9e0150fe0 2024-03-06T07:27:56.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.835+0000 7ff9e4caa640 1 -- 172.21.15.1:0/2476967888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e00bf300 con 0x7ff9e0150fe0 2024-03-06T07:27:56.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.836+0000 7ff9e4caa640 1 -- 172.21.15.1:0/2476967888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e0067fd0 con 0x7ff9e0150fe0 2024-03-06T07:27:56.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.836+0000 7ff9d6ffd640 1 -- 172.21.15.1:0/2476967888 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff9d801a070 con 0x7ff9e0150fe0 2024-03-06T07:27:56.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.837+0000 7ff9d6ffd640 1 --2- 172.21.15.1:0/2476967888 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff9c006b500 0x7ff9c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.837+0000 7ff9d6ffd640 1 -- 172.21.15.1:0/2476967888 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff9d80895b0 con 0x7ff9e0150fe0 2024-03-06T07:27:56.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.837+0000 7ff9de59c640 1 --2- 172.21.15.1:0/2476967888 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff9c006b500 0x7ff9c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.838+0000 7ff9de59c640 1 --2- 172.21.15.1:0/2476967888 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff9c006b500 0x7ff9c006d9c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0036000 tx=0x7ff9d0030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:56.842+0000 7ff9d6ffd640 1 -- 172.21.15.1:0/2476967888 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff9d8057ce0 con 0x7ff9e0150fe0 2024-03-06T07:27:57.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.067+0000 7ff9e4caa640 1 -- 172.21.15.1:0/2476967888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9e006eb90 con 0x7ff9e0150fe0 2024-03-06T07:27:57.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.068+0000 7ff9d6ffd640 1 -- 172.21.15.1:0/2476967888 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+4981 (secure 0 0 0) 0x7ff9d805bd80 con 0x7ff9e0150fe0 2024-03-06T07:27:57.070 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:57.070 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":170,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":9951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:27:54.186815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9954":{"gid":9954,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1300509547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1300509547},{"type":"v1","addr":"172.21.15.1:6835","nonce":1300509547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9954,"qdb_cluster":[9954]},"id":14}]} 2024-03-06T07:27:57.070 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 170 2024-03-06T07:27:57.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.071+0000 7ff9e4caa640 1 -- 172.21.15.1:0/2476967888 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff9c006b500 msgr2=0x7ff9c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.071+0000 7ff9e4caa640 1 --2- 172.21.15.1:0/2476967888 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff9c006b500 0x7ff9c006d9c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0036000 tx=0x7ff9d0030000 comp rx=0 tx=0).stop 2024-03-06T07:27:57.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.071+0000 7ff9e4caa640 1 -- 172.21.15.1:0/2476967888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9e0150fe0 msgr2=0x7ff9e0143b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.071+0000 7ff9e4caa640 1 --2- 172.21.15.1:0/2476967888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9e0150fe0 0x7ff9e0143b40 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800c400 tx=0x7ff9d800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.071+0000 7ff9e4caa640 1 -- 172.21.15.1:0/2476967888 shutdown_connections 2024-03-06T07:27:57.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.071+0000 7ff9e4caa640 1 --2- 172.21.15.1:0/2476967888 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff9c006b500 0x7ff9c006d9c0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.071+0000 7ff9e4caa640 1 --2- 172.21.15.1:0/2476967888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9e0150fe0 0x7ff9e0143b40 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.071+0000 7ff9e4caa640 1 --2- 172.21.15.1:0/2476967888 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff9e0072f20 0x7ff9e0154e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.071+0000 7ff9e4caa640 1 --2- 172.21.15.1:0/2476967888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9e006f2f0 0x7ff9e00bfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.071+0000 7ff9e4caa640 1 -- 172.21.15.1:0/2476967888 >> 172.21.15.1:0/2476967888 conn(0x7ff9e00c6860 msgr2=0x7ff9e014e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.071+0000 7ff9e4caa640 1 -- 172.21.15.1:0/2476967888 shutdown_connections 2024-03-06T07:27:57.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.071+0000 7ff9e4caa640 1 -- 172.21.15.1:0/2476967888 wait complete. 2024-03-06T07:27:57.485 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 170, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:52.179842+0000', 'modified': '2024-03-06T07:27:54.186815+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9954}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9954': {'gid': 9954, 'name': 'c', 'rank': 0, 'incarnation': 168, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1300509547', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 1300509547}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1300509547}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9954, 'qdb_cluster': [9954]} 2024-03-06T07:27:57.485 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:57.486 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:27:57.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.588+0000 7fb2c0393640 1 Processor -- start 2024-03-06T07:27:57.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.588+0000 7fb2c0393640 1 -- start start 2024-03-06T07:27:57.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.589+0000 7fb2c0393640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2b8150520 0x7fb2b81543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.589+0000 7fb2c0393640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2b8070c80 0x7fb2b80710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.589+0000 7fb2c0393640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b8150900 con 0x7fb2b80c32f0 2024-03-06T07:27:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.589+0000 7fb2c0393640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2b8153ed0 con 0x7fb2b8070c80 2024-03-06T07:27:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.589+0000 7fb2c0393640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b8071790 con 0x7fb2b8150520 2024-03-06T07:27:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.589+0000 7fb2be930640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2b8070c80 0x7fb2b80710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.589+0000 7fb2be930640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2b8070c80 0x7fb2b80710f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60474/0 (socket says 172.21.15.1:60474) 2024-03-06T07:27:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.589+0000 7fb2be930640 1 -- 172.21.15.1:0/4232738840 learned_addr learned my addr 172.21.15.1:0/4232738840 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.589+0000 7fb2be12f640 1 --2- 172.21.15.1:0/4232738840 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2b8150520 0x7fb2b81543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.589+0000 7fb2be930640 1 -- 172.21.15.1:0/4232738840 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2b8150520 msgr2=0x7fb2b81543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.589+0000 7fb2be930640 1 --2- 172.21.15.1:0/4232738840 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2b8150520 0x7fb2b81543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.589+0000 7fb2be930640 1 -- 172.21.15.1:0/4232738840 >> v1:172.21.15.98:6789/0 conn(0x7fb2b80c32f0 legacy=0x7fb2b81537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.589+0000 7fb2be930640 1 -- 172.21.15.1:0/4232738840 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b8157050 con 0x7fb2b8070c80 2024-03-06T07:27:57.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.590+0000 7fb2be930640 1 --2- 172.21.15.1:0/4232738840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2b8070c80 0x7fb2b80710f0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0009e50 tx=0x7fb2b002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24261897f1998c51 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.590+0000 7fb2bd12d640 1 -- 172.21.15.1:0/4232738840 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b002ca70 con 0x7fb2b8070c80 2024-03-06T07:27:57.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.590+0000 7fb2bd12d640 1 -- 172.21.15.1:0/4232738840 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b002cc10 con 0x7fb2b8070c80 2024-03-06T07:27:57.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.590+0000 7fb2bd12d640 1 -- 172.21.15.1:0/4232738840 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b00343f0 con 0x7fb2b8070c80 2024-03-06T07:27:57.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.591+0000 7fb2c0393640 1 -- 172.21.15.1:0/4232738840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2b8070c80 msgr2=0x7fb2b80710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.591+0000 7fb2c0393640 1 --2- 172.21.15.1:0/4232738840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2b8070c80 0x7fb2b80710f0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0009e50 tx=0x7fb2b002bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.591+0000 7fb2c0393640 1 -- 172.21.15.1:0/4232738840 shutdown_connections 2024-03-06T07:27:57.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.591+0000 7fb2c0393640 1 --2- 172.21.15.1:0/4232738840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2b8070c80 0x7fb2b80710f0 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.591+0000 7fb2c0393640 1 --2- 172.21.15.1:0/4232738840 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2b8150520 0x7fb2b81543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.591+0000 7fb2c0393640 1 -- 172.21.15.1:0/4232738840 >> 172.21.15.1:0/4232738840 conn(0x7fb2b80c88f0 msgr2=0x7fb2b80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.591+0000 7fb2c0393640 1 -- 172.21.15.1:0/4232738840 shutdown_connections 2024-03-06T07:27:57.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.591+0000 7fb2c0393640 1 -- 172.21.15.1:0/4232738840 wait complete. 2024-03-06T07:27:57.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.592+0000 7fb2c0393640 1 Processor -- start 2024-03-06T07:27:57.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.593+0000 7fb2c0393640 1 -- start start 2024-03-06T07:27:57.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2c0393640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2b8070c80 0x7fb2b80c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2c0393640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2b80c32f0 0x7fb2b8146170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2c0393640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2b8150520 0x7fb2b81466b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2c0393640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b8157330 con 0x7fb2b8150520 2024-03-06T07:27:57.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2c0393640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b81532a0 con 0x7fb2b80c32f0 2024-03-06T07:27:57.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2c0393640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b80c7ec0 con 0x7fb2b8070c80 2024-03-06T07:27:57.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2be930640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2b8150520 0x7fb2b81466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2be930640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2b8150520 0x7fb2b81466b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60480/0 (socket says 172.21.15.1:60480) 2024-03-06T07:27:57.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2be930640 1 -- 172.21.15.1:0/3551879237 learned_addr learned my addr 172.21.15.1:0/3551879237 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:57.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2be12f640 1 --2- 172.21.15.1:0/3551879237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2b8070c80 0x7fb2b80c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2bd92e640 1 --2- 172.21.15.1:0/3551879237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2b80c32f0 0x7fb2b8146170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2be930640 1 -- 172.21.15.1:0/3551879237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2b8070c80 msgr2=0x7fb2b80c1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2be930640 1 --2- 172.21.15.1:0/3551879237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2b8070c80 0x7fb2b80c1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2be930640 1 -- 172.21.15.1:0/3551879237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2b80c32f0 msgr2=0x7fb2b8146170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2be930640 1 --2- 172.21.15.1:0/3551879237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2b80c32f0 0x7fb2b8146170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2be930640 1 -- 172.21.15.1:0/3551879237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b806cfd0 con 0x7fb2b8150520 2024-03-06T07:27:57.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2be12f640 1 --2- 172.21.15.1:0/3551879237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2b8070c80 0x7fb2b80c1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:57.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2be930640 1 --2- 172.21.15.1:0/3551879237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2b8150520 0x7fb2b81466b0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0009e50 tx=0x7fb2b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2aeffd640 1 -- 172.21.15.1:0/3551879237 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b0043020 con 0x7fb2b8150520 2024-03-06T07:27:57.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.594+0000 7fb2aeffd640 1 -- 172.21.15.1:0/3551879237 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b003e450 con 0x7fb2b8150520 2024-03-06T07:27:57.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.595+0000 7fb2c0393640 1 -- 172.21.15.1:0/3551879237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b806d2b0 con 0x7fb2b8150520 2024-03-06T07:27:57.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.595+0000 7fb2aeffd640 1 -- 172.21.15.1:0/3551879237 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b0009510 con 0x7fb2b8150520 2024-03-06T07:27:57.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.595+0000 7fb2c0393640 1 -- 172.21.15.1:0/3551879237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b80c24d0 con 0x7fb2b8150520 2024-03-06T07:27:57.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.596+0000 7fb2aeffd640 1 -- 172.21.15.1:0/3551879237 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2b0046020 con 0x7fb2b8150520 2024-03-06T07:27:57.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.597+0000 7fb2aeffd640 1 --2- 172.21.15.1:0/3551879237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb29c06b4d0 0x7fb29c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.597+0000 7fb2aeffd640 1 -- 172.21.15.1:0/3551879237 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb2b00acc80 con 0x7fb2b8150520 2024-03-06T07:27:57.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.597+0000 7fb2be12f640 1 --2- 172.21.15.1:0/3551879237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb29c06b4d0 0x7fb29c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.598+0000 7fb2c2f80740 1 Processor -- start 2024-03-06T07:27:57.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.598+0000 7fb2be12f640 1 --2- 172.21.15.1:0/3551879237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb29c06b4d0 0x7fb29c06d990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4006fd0 tx=0x7fb2b4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.598+0000 7fb2c2f80740 1 -- start start 2024-03-06T07:27:57.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb2c2f80740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5dbef50 0x558de5dbca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb2c2f80740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbf330 0x558de5dbcf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb2c2f80740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbd4b0 0x558de5db9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb2c2f80740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x558de5d236b0 con 0x558de5dbd4b0 2024-03-06T07:27:57.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb2c2f80740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x558de5ccca60 con 0x558de5dbef50 2024-03-06T07:27:57.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb2c2f80740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x558de5d1f750 con 0x558de5dbf330 2024-03-06T07:27:57.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb2be930640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbd4b0 0x558de5db9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb2be930640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbd4b0 0x558de5db9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60490/0 (socket says 172.21.15.1:60490) 2024-03-06T07:27:57.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb2be930640 1 -- 172.21.15.1:0/3114366372 learned_addr learned my addr 172.21.15.1:0/3114366372 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:57.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb2be12f640 1 --2- 172.21.15.1:0/3114366372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5dbef50 0x558de5dbca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb2bd92e640 1 --2- 172.21.15.1:0/3114366372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbf330 0x558de5dbcf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb2be930640 1 -- 172.21.15.1:0/3114366372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbf330 msgr2=0x558de5dbcf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb2be930640 1 --2- 172.21.15.1:0/3114366372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbf330 0x558de5dbcf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb2be930640 1 -- 172.21.15.1:0/3114366372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5dbef50 msgr2=0x558de5dbca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb2be930640 1 --2- 172.21.15.1:0/3114366372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5dbef50 0x558de5dbca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb2be930640 1 -- 172.21.15.1:0/3114366372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558de5ceaf30 con 0x558de5dbd4b0 2024-03-06T07:27:57.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb2be930640 1 --2- 172.21.15.1:0/3114366372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbd4b0 0x558de5db9680 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fb2b002c4e0 tx=0x7fb2b003c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2f0af0a434e65d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb29bfff640 1 -- 172.21.15.1:0/3114366372 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b007b380 con 0x558de5dbd4b0 2024-03-06T07:27:57.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb29bfff640 1 -- 172.21.15.1:0/3114366372 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b007f4d0 con 0x558de5dbd4b0 2024-03-06T07:27:57.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.599+0000 7fb29bfff640 1 -- 172.21.15.1:0/3114366372 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b0083a90 con 0x558de5dbd4b0 2024-03-06T07:27:57.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.600+0000 7fb2c2f80740 1 -- 172.21.15.1:0/3114366372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbd4b0 msgr2=0x558de5db9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.600+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/3114366372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbd4b0 0x558de5db9680 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fb2b002c4e0 tx=0x7fb2b003c000 comp rx=0 tx=0).stop 2024-03-06T07:27:57.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.600+0000 7fb2c2f80740 1 -- 172.21.15.1:0/3114366372 shutdown_connections 2024-03-06T07:27:57.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.600+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/3114366372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbd4b0 0x558de5db9680 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.600+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/3114366372 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbf330 0x558de5dbcf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.600+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/3114366372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5dbef50 0x558de5dbca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.600+0000 7fb2c2f80740 1 -- 172.21.15.1:0/3114366372 >> 172.21.15.1:0/3114366372 conn(0x558de5dbb7d0 msgr2=0x558de5de03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.600+0000 7fb2c2f80740 1 -- 172.21.15.1:0/3114366372 shutdown_connections 2024-03-06T07:27:57.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.600+0000 7fb2c2f80740 1 -- 172.21.15.1:0/3114366372 wait complete. 2024-03-06T07:27:57.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.601+0000 7fb2c2f80740 1 -- start start 2024-03-06T07:27:57.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.601+0000 7fb2c2f80740 1 Processor -- start 2024-03-06T07:27:57.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2c2f80740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbef50 0x558de5ddfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2c2f80740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbf330 0x558de5e0c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2c2f80740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5de1b20 0x558de5de1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2c2f80740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x558de5d236b0 con 0x558de5dbf330 2024-03-06T07:27:57.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2c2f80740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x558de5cdfea0 con 0x558de5de1b20 2024-03-06T07:27:57.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2c2f80740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x558de5de1230 con 0x558de5dbef50 2024-03-06T07:27:57.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2c2f80740 10 client.0 _ll_register_callbacks cb 0x558de5cdf870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:57.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2bd92e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbf330 0x558de5e0c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2bd92e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbf330 0x558de5e0c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60506/0 (socket says 172.21.15.1:60506) 2024-03-06T07:27:57.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2bd92e640 1 -- 172.21.15.1:0/3514954353 learned_addr learned my addr 172.21.15.1:0/3514954353 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:57.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2be12f640 1 --2- 172.21.15.1:0/3514954353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbef50 0x558de5ddfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2be930640 1 --2- 172.21.15.1:0/3514954353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5de1b20 0x558de5de1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2bd92e640 1 -- 172.21.15.1:0/3514954353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbef50 msgr2=0x558de5ddfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2bd92e640 1 --2- 172.21.15.1:0/3514954353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbef50 0x558de5ddfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2bd92e640 1 -- 172.21.15.1:0/3514954353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5de1b20 msgr2=0x558de5de1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2bd92e640 1 --2- 172.21.15.1:0/3514954353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5de1b20 0x558de5de1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2bd92e640 1 -- 172.21.15.1:0/3514954353 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558de5ce0270 con 0x558de5dbf330 2024-03-06T07:27:57.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2bd92e640 1 --2- 172.21.15.1:0/3514954353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbf330 0x558de5e0c4b0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fb2b814fd70 tx=0x7fb2a8011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ec0a04335d9a2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.602+0000 7fb2997fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:27:57.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.603+0000 7fb2997fa640 1 -- 172.21.15.1:0/3514954353 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a8019070 con 0x558de5dbf330 2024-03-06T07:27:57.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.603+0000 7fb2997fa640 1 -- 172.21.15.1:0/3514954353 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb2a8015640 con 0x558de5dbf330 2024-03-06T07:27:57.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.603+0000 7fb2c2f80740 1 -- 172.21.15.1:0/3514954353 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb2b007f4d0 con 0x558de5dbf330 2024-03-06T07:27:57.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.603+0000 7fb2997fa640 1 -- 172.21.15.1:0/3514954353 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=170) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2a8024050 con 0x558de5dbf330 2024-03-06T07:27:57.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.603+0000 7fb2c2f80740 10 client.10092 fetch_fsmap learned FSMap version 170 2024-03-06T07:27:57.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.603+0000 7fb2c2f80740 1 -- 172.21.15.1:0/3514954353 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=170,monmap=2+}) v3 -- 0x558de5a382d0 con 0x558de5dbf330 2024-03-06T07:27:57.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.603+0000 7fb2997fa640 1 -- 172.21.15.1:0/3514954353 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b007f4d0 con 0x558de5dbf330 2024-03-06T07:27:57.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.603+0000 7fb2997fa640 1 -- 172.21.15.1:0/3514954353 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 170) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb2a8015840 con 0x558de5dbf330 2024-03-06T07:27:57.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.603+0000 7fb2c2f80740 10 client.10092 fetch_fsmap finished waiting for FSMap version 170 2024-03-06T07:27:57.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.603+0000 7fb2c2f80740 10 client.10092 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:57.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.603+0000 7fb2c2f80740 20 client.10092 populate_metadata read hostname 'smithi001' 2024-03-06T07:27:57.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.603+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/3514954353 >> [v2:172.21.15.1:6834/1300509547,v1:172.21.15.1:6835/1300509547] conn(0x558de5e0fa20 0x558de5e1a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.603+0000 7fb2c2f80740 4 client.10092 mds_command: new command op to 9954 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:57.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.603+0000 7fb2c2f80740 1 -- 172.21.15.1:0/3514954353 --> [v2:172.21.15.1:6834/1300509547,v1:172.21.15.1:6835/1300509547] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558de5d30bf0 con 0x558de5e0fa20 2024-03-06T07:27:57.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.603+0000 7fb2be12f640 1 --2- 172.21.15.1:0/3514954353 >> [v2:172.21.15.1:6834/1300509547,v1:172.21.15.1:6835/1300509547] conn(0x558de5e0fa20 0x558de5e1a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.604+0000 7fb2be12f640 1 --2- 172.21.15.1:0/3514954353 >> [v2:172.21.15.1:6834/1300509547,v1:172.21.15.1:6835/1300509547] conn(0x558de5e0fa20 0x558de5e1a5e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8d1da71cf754c51 server_cookie=a954c6a6d42b5a82 in_seq=0 out_seq=0 2024-03-06T07:27:57.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.604+0000 7fb2997fa640 10 client.10092 ms_handle_connect on v2:172.21.15.1:6834/1300509547 2024-03-06T07:27:57.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.604+0000 7fb2997fa640 1 -- 172.21.15.1:0/3514954353 <== mds.0 v2:172.21.15.1:6834/1300509547 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558de5d30bf0 con 0x558de5e0fa20 2024-03-06T07:27:57.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.604+0000 7fb2997fa640 10 client.10092 handle_command_reply: tid=0 2024-03-06T07:27:57.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.604+0000 7fb2c2f80740 1 client.10092 shutdown 2024-03-06T07:27:57.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.604+0000 7fb2c2f80740 1 -- 172.21.15.1:0/3514954353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbf330 msgr2=0x558de5e0c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.604+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/3514954353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbf330 0x558de5e0c4b0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fb2b814fd70 tx=0x7fb2a8011750 comp rx=0 tx=0).stop 2024-03-06T07:27:57.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.604+0000 7fb2c2f80740 1 -- 172.21.15.1:0/3514954353 shutdown_connections 2024-03-06T07:27:57.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.604+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/3514954353 >> [v2:172.21.15.1:6834/1300509547,v1:172.21.15.1:6835/1300509547] conn(0x558de5e0fa20 0x558de5e1a5e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.604+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/3514954353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5de1b20 0x558de5de1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.604+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/3514954353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbf330 0x558de5e0c4b0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.604+0000 7fb2997fa640 0 client.10092 ms_handle_reset on v2:172.21.15.1:6834/1300509547 2024-03-06T07:27:57.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.604+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/3514954353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbef50 0x558de5ddfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.604+0000 7fb2c2f80740 1 -- 172.21.15.1:0/3514954353 >> 172.21.15.1:0/3514954353 conn(0x558de5dbd4b0 msgr2=0x558de5dbe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.605+0000 7fb2c2f80740 1 -- 172.21.15.1:0/3514954353 shutdown_connections 2024-03-06T07:27:57.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.605+0000 7fb2c2f80740 1 -- 172.21.15.1:0/3514954353 wait complete. 2024-03-06T07:27:57.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.605+0000 7fb2c2f80740 20 client.10092 trim_cache size 0 max 16384 2024-03-06T07:27:57.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.621+0000 7fb2c2f80740 1 Processor -- start 2024-03-06T07:27:57.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.622+0000 7fb2c2f80740 1 -- start start 2024-03-06T07:27:57.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.622+0000 7fb2c2f80740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5dbef50 0x558de5df8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.622+0000 7fb2c2f80740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbf330 0x558de5db8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.622+0000 7fb2c2f80740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5de1b20 0x558de5dded50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.622+0000 7fb2c2f80740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b81f7dc0 con 0x558de5de1b20 2024-03-06T07:27:57.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.622+0000 7fb2c2f80740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b81f6f50 con 0x558de5dbef50 2024-03-06T07:27:57.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.622+0000 7fb2c2f80740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b81f63d0 con 0x558de5dbf330 2024-03-06T07:27:57.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.622+0000 7fb2be930640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5de1b20 0x558de5dded50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.622+0000 7fb2be930640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5de1b20 0x558de5dded50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60520/0 (socket says 172.21.15.1:60520) 2024-03-06T07:27:57.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.622+0000 7fb2be930640 1 -- 172.21.15.1:0/2860739626 learned_addr learned my addr 172.21.15.1:0/2860739626 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:57.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.622+0000 7fb2be930640 1 -- 172.21.15.1:0/2860739626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbf330 msgr2=0x558de5db8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.622+0000 7fb2be930640 1 --2- 172.21.15.1:0/2860739626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbf330 0x558de5db8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.622+0000 7fb2be930640 1 -- 172.21.15.1:0/2860739626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5dbef50 msgr2=0x558de5df8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.622+0000 7fb2be930640 1 --2- 172.21.15.1:0/2860739626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5dbef50 0x558de5df8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.623+0000 7fb2be930640 1 -- 172.21.15.1:0/2860739626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558de5b026b0 con 0x558de5de1b20 2024-03-06T07:27:57.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.623+0000 7fb2be930640 1 --2- 172.21.15.1:0/2860739626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5de1b20 0x558de5dded50 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fb2b003c040 tx=0x7fb2b00ac000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259250263b4f46d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.623+0000 7fb29b7fe640 1 -- 172.21.15.1:0/2860739626 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b0077620 con 0x558de5de1b20 2024-03-06T07:27:57.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.623+0000 7fb29b7fe640 1 -- 172.21.15.1:0/2860739626 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b007f4d0 con 0x558de5de1b20 2024-03-06T07:27:57.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.623+0000 7fb29b7fe640 1 -- 172.21.15.1:0/2860739626 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b0083a90 con 0x558de5de1b20 2024-03-06T07:27:57.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.623+0000 7fb2c2f80740 1 -- 172.21.15.1:0/2860739626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5de1b20 msgr2=0x558de5dded50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.623+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/2860739626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5de1b20 0x558de5dded50 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fb2b003c040 tx=0x7fb2b00ac000 comp rx=0 tx=0).stop 2024-03-06T07:27:57.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.623+0000 7fb2c2f80740 1 -- 172.21.15.1:0/2860739626 shutdown_connections 2024-03-06T07:27:57.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.623+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/2860739626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5de1b20 0x558de5dded50 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.623+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/2860739626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbf330 0x558de5db8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.623+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/2860739626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5dbef50 0x558de5df8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.623+0000 7fb2c2f80740 1 -- 172.21.15.1:0/2860739626 >> 172.21.15.1:0/2860739626 conn(0x558de5dbd4b0 msgr2=0x558de5e1f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.623+0000 7fb2c2f80740 1 -- 172.21.15.1:0/2860739626 shutdown_connections 2024-03-06T07:27:57.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.623+0000 7fb2c2f80740 1 -- 172.21.15.1:0/2860739626 wait complete. 2024-03-06T07:27:57.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.624+0000 7fb2c2f80740 1 -- start start 2024-03-06T07:27:57.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.624+0000 7fb2c2f80740 1 Processor -- start 2024-03-06T07:27:57.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2c2f80740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbef50 0x558de5e6f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2c2f80740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbf330 0x558de5dbfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2c2f80740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5de1b20 0x558de5e684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2c2f80740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x558de5e0c9f0 con 0x558de5dbf330 2024-03-06T07:27:57.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2c2f80740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b81f6950 con 0x558de5de1b20 2024-03-06T07:27:57.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2c2f80740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x558de5de3890 con 0x558de5dbef50 2024-03-06T07:27:57.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2c2f80740 10 client.0 _ll_register_callbacks cb 0x7fb2a80052f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:57.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2bd92e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbf330 0x558de5dbfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2bd92e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbf330 0x558de5dbfdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60526/0 (socket says 172.21.15.1:60526) 2024-03-06T07:27:57.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2bd92e640 1 -- 172.21.15.1:0/1090993906 learned_addr learned my addr 172.21.15.1:0/1090993906 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:57.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2be12f640 1 --2- 172.21.15.1:0/1090993906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbef50 0x558de5e6f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2bd92e640 1 -- 172.21.15.1:0/1090993906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbef50 msgr2=0x558de5e6f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2bd92e640 1 --2- 172.21.15.1:0/1090993906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbef50 0x558de5e6f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2bd92e640 1 -- 172.21.15.1:0/1090993906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5de1b20 msgr2=0x558de5e684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2bd92e640 1 --2- 172.21.15.1:0/1090993906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5de1b20 0x558de5e684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2bd92e640 1 -- 172.21.15.1:0/1090993906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558de5df0a90 con 0x558de5dbf330 2024-03-06T07:27:57.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2bd92e640 1 --2- 172.21.15.1:0/1090993906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbf330 0x558de5dbfdd0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8000c00 tx=0x7fb2a80036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18d6aad38777f6c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.625+0000 7fb2997fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:27:57.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.626+0000 7fb2997fa640 1 -- 172.21.15.1:0/1090993906 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a8019070 con 0x558de5dbf330 2024-03-06T07:27:57.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.626+0000 7fb2997fa640 1 -- 172.21.15.1:0/1090993906 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb2a8015640 con 0x558de5dbf330 2024-03-06T07:27:57.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.626+0000 7fb2c2f80740 1 -- 172.21.15.1:0/1090993906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb2b007f4d0 con 0x558de5dbf330 2024-03-06T07:27:57.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.626+0000 7fb2997fa640 1 -- 172.21.15.1:0/1090993906 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=170) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2a8034050 con 0x558de5dbf330 2024-03-06T07:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.626+0000 7fb2c2f80740 10 client.10098 fetch_fsmap learned FSMap version 170 2024-03-06T07:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.626+0000 7fb2c2f80740 1 -- 172.21.15.1:0/1090993906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=170,monmap=2+}) v3 -- 0x558de5df8a50 con 0x558de5dbf330 2024-03-06T07:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.626+0000 7fb2997fa640 1 -- 172.21.15.1:0/1090993906 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b007f4d0 con 0x558de5dbf330 2024-03-06T07:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.626+0000 7fb2997fa640 1 -- 172.21.15.1:0/1090993906 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 170) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb2a8037020 con 0x558de5dbf330 2024-03-06T07:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.626+0000 7fb2c2f80740 10 client.10098 fetch_fsmap finished waiting for FSMap version 170 2024-03-06T07:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.626+0000 7fb2c2f80740 10 client.10098 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.626+0000 7fb2c2f80740 20 client.10098 populate_metadata read hostname 'smithi001' 2024-03-06T07:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.626+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/1090993906 >> [v2:172.21.15.1:6834/1300509547,v1:172.21.15.1:6835/1300509547] conn(0x558de5e07a20 0x558de5e07e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.626+0000 7fb2c2f80740 4 client.10098 mds_command: new command op to 9954 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.626+0000 7fb2c2f80740 1 -- 172.21.15.1:0/1090993906 --> [v2:172.21.15.1:6834/1300509547,v1:172.21.15.1:6835/1300509547] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558de5d2af40 con 0x558de5e07a20 2024-03-06T07:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.626+0000 7fb2be12f640 1 --2- 172.21.15.1:0/1090993906 >> [v2:172.21.15.1:6834/1300509547,v1:172.21.15.1:6835/1300509547] conn(0x558de5e07a20 0x558de5e07e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.627+0000 7fb2be12f640 1 --2- 172.21.15.1:0/1090993906 >> [v2:172.21.15.1:6834/1300509547,v1:172.21.15.1:6835/1300509547] conn(0x558de5e07a20 0x558de5e07e00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2019692bc82a4d4c server_cookie=f22b055d0431ee6c in_seq=0 out_seq=0 2024-03-06T07:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.627+0000 7fb2997fa640 10 client.10098 ms_handle_connect on v2:172.21.15.1:6834/1300509547 2024-03-06T07:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.627+0000 7fb2997fa640 1 -- 172.21.15.1:0/1090993906 <== mds.0 v2:172.21.15.1:6834/1300509547 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x558de5d2af40 con 0x558de5e07a20 2024-03-06T07:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.627+0000 7fb2997fa640 10 client.10098 handle_command_reply: tid=0 2024-03-06T07:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.627+0000 7fb2c2f80740 1 client.10098 shutdown 2024-03-06T07:27:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.627+0000 7fb2c2f80740 1 -- 172.21.15.1:0/1090993906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbf330 msgr2=0x558de5dbfdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.627+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/1090993906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbf330 0x558de5dbfdd0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8000c00 tx=0x7fb2a80036f0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.627+0000 7fb2c2f80740 1 -- 172.21.15.1:0/1090993906 shutdown_connections 2024-03-06T07:27:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.627+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/1090993906 >> [v2:172.21.15.1:6834/1300509547,v1:172.21.15.1:6835/1300509547] conn(0x558de5e07a20 0x558de5e07e00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.627+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/1090993906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558de5de1b20 0x558de5e684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.627+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/1090993906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558de5dbf330 0x558de5dbfdd0 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.627+0000 7fb2c2f80740 1 --2- 172.21.15.1:0/1090993906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558de5dbef50 0x558de5e6f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.627+0000 7fb2997fa640 0 client.10098 ms_handle_reset on v2:172.21.15.1:6834/1300509547 2024-03-06T07:27:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.627+0000 7fb2c2f80740 1 -- 172.21.15.1:0/1090993906 >> 172.21.15.1:0/1090993906 conn(0x558de5dbd4b0 msgr2=0x558de5db8510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.627+0000 7fb2c2f80740 1 -- 172.21.15.1:0/1090993906 shutdown_connections 2024-03-06T07:27:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.627+0000 7fb2c2f80740 1 -- 172.21.15.1:0/1090993906 wait complete. 2024-03-06T07:27:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.627+0000 7fb2c2f80740 20 client.10098 trim_cache size 0 max 16384 2024-03-06T07:27:57.637 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:27:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:27:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:27:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 9954, 2024-03-06T07:27:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:27:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:27:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:27:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 4.7959106289999998, 2024-03-06T07:27:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 170, 2024-03-06T07:27:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 183, 2024-03-06T07:27:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 183, 2024-03-06T07:27:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 9.8850079429999997 2024-03-06T07:27:57.637 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:27:57.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.628+0000 7fb2c0393640 1 -- 172.21.15.1:0/3551879237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb29c06b4d0 msgr2=0x7fb29c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.628+0000 7fb2c0393640 1 --2- 172.21.15.1:0/3551879237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb29c06b4d0 0x7fb29c06d990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4006fd0 tx=0x7fb2b4008040 comp rx=0 tx=0).stop 2024-03-06T07:27:57.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.628+0000 7fb2c0393640 1 -- 172.21.15.1:0/3551879237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2b8150520 msgr2=0x7fb2b81466b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.628+0000 7fb2c0393640 1 --2- 172.21.15.1:0/3551879237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2b8150520 0x7fb2b81466b0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0009e50 tx=0x7fb2b0004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:57.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.628+0000 7fb2c0393640 1 -- 172.21.15.1:0/3551879237 shutdown_connections 2024-03-06T07:27:57.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.628+0000 7fb2c0393640 1 --2- 172.21.15.1:0/3551879237 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb29c06b4d0 0x7fb29c06d990 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.628+0000 7fb2c0393640 1 --2- 172.21.15.1:0/3551879237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2b8150520 0x7fb2b81466b0 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.628+0000 7fb2c0393640 1 --2- 172.21.15.1:0/3551879237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2b80c32f0 0x7fb2b8146170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.628+0000 7fb2c0393640 1 --2- 172.21.15.1:0/3551879237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2b8070c80 0x7fb2b80c1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.628+0000 7fb2c0393640 1 -- 172.21.15.1:0/3551879237 >> 172.21.15.1:0/3551879237 conn(0x7fb2b80c88f0 msgr2=0x7fb2b814af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.628+0000 7fb2c0393640 1 -- 172.21.15.1:0/3551879237 shutdown_connections 2024-03-06T07:27:57.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.628+0000 7fb2c0393640 1 -- 172.21.15.1:0/3551879237 wait complete. 2024-03-06T07:27:57.640 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2024-03-06T07:27:57.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.743+0000 7f2e7ce0e640 1 Processor -- start 2024-03-06T07:27:57.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.743+0000 7f2e7ce0e640 1 -- start start 2024-03-06T07:27:57.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.744+0000 7f2e7ce0e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e78150520 0x7f2e781543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.744+0000 7f2e7ce0e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2e78070c80 0x7f2e780710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.744+0000 7f2e7ce0e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e78150900 con 0x7f2e780c32f0 2024-03-06T07:27:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.744+0000 7f2e7ce0e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e78153ed0 con 0x7f2e78150520 2024-03-06T07:27:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.744+0000 7f2e7ce0e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e78071790 con 0x7f2e78070c80 2024-03-06T07:27:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.744+0000 7f2e7659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e78150520 0x7f2e781543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.744+0000 7f2e75d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2e780c32f0 0x7f2e781537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50616/0 (socket says 172.21.15.1:50616) 2024-03-06T07:27:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.744+0000 7f2e75d9b640 1 -- 172.21.15.1:0/2931884640 learned_addr learned my addr 172.21.15.1:0/2931884640 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:27:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.744+0000 7f2e76d9d640 1 --2- 172.21.15.1:0/2931884640 >> v2:172.21.15.98:3300/0 conn(0x7f2e78070c80 0x7f2e780710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.745+0000 7f2e7659c640 1 -- 172.21.15.1:0/2931884640 >> v2:172.21.15.98:3300/0 conn(0x7f2e78070c80 msgr2=0x7f2e780710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.745+0000 7f2e7659c640 1 --2- 172.21.15.1:0/2931884640 >> v2:172.21.15.98:3300/0 conn(0x7f2e78070c80 0x7f2e780710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.745+0000 7f2e7659c640 1 -- 172.21.15.1:0/2931884640 >> v1:172.21.15.1:6789/0 conn(0x7f2e780c32f0 legacy=0x7f2e781537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.745+0000 7f2e7659c640 1 -- 172.21.15.1:0/2931884640 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e78157050 con 0x7f2e78150520 2024-03-06T07:27:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.745+0000 7f2e7559a640 1 -- 172.21.15.1:0/2931884640 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490844798 0 0) 0x7f2e78150900 con 0x7f2e780c32f0 2024-03-06T07:27:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.745+0000 7f2e7659c640 1 --2- 172.21.15.1:0/2931884640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e78150520 0x7f2e781543a0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f2e780c3190 tx=0x7f2e6802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd6841145972fc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.746+0000 7f2e7559a640 1 -- 172.21.15.1:0/2931884640 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e68035020 con 0x7f2e78150520 2024-03-06T07:27:57.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.746+0000 7f2e7559a640 1 -- 172.21.15.1:0/2931884640 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e68031830 con 0x7f2e78150520 2024-03-06T07:27:57.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.746+0000 7f2e7559a640 1 -- 172.21.15.1:0/2931884640 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e68031c00 con 0x7f2e78150520 2024-03-06T07:27:57.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.746+0000 7f2e7ce0e640 1 -- 172.21.15.1:0/2931884640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e78150520 msgr2=0x7f2e781543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.746+0000 7f2e7ce0e640 1 --2- 172.21.15.1:0/2931884640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e78150520 0x7f2e781543a0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f2e780c3190 tx=0x7f2e6802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.746+0000 7f2e7ce0e640 1 -- 172.21.15.1:0/2931884640 shutdown_connections 2024-03-06T07:27:57.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.746+0000 7f2e7ce0e640 1 --2- 172.21.15.1:0/2931884640 >> v2:172.21.15.98:3300/0 conn(0x7f2e78070c80 0x7f2e780710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.746+0000 7f2e7ce0e640 1 --2- 172.21.15.1:0/2931884640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e78150520 0x7f2e781543a0 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.746+0000 7f2e7ce0e640 1 -- 172.21.15.1:0/2931884640 >> 172.21.15.1:0/2931884640 conn(0x7f2e780c88f0 msgr2=0x7f2e780c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.746+0000 7f2e7ce0e640 1 -- 172.21.15.1:0/2931884640 shutdown_connections 2024-03-06T07:27:57.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.746+0000 7f2e7ce0e640 1 -- 172.21.15.1:0/2931884640 wait complete. 2024-03-06T07:27:57.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.748+0000 7f2e7ce0e640 1 Processor -- start 2024-03-06T07:27:57.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.748+0000 7f2e7ce0e640 1 -- start start 2024-03-06T07:27:57.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.749+0000 7f2e7ce0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e78070c80 0x7f2e781ebaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.749+0000 7f2e7ce0e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e780c32f0 0x7f2e78155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.749+0000 7f2e7ce0e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e78150520 0x7f2e7814eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.749+0000 7f2e7ce0e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e78157330 con 0x7f2e780c32f0 2024-03-06T07:27:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.749+0000 7f2e7ce0e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e781532a0 con 0x7f2e78070c80 2024-03-06T07:27:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.749+0000 7f2e7ce0e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e780c7ec0 con 0x7f2e78150520 2024-03-06T07:27:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.749+0000 7f2e75d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e780c32f0 0x7f2e78155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.749+0000 7f2e75d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e780c32f0 0x7f2e78155310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60542/0 (socket says 172.21.15.1:60542) 2024-03-06T07:27:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.749+0000 7f2e75d9b640 1 -- 172.21.15.1:0/2866035177 learned_addr learned my addr 172.21.15.1:0/2866035177 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.750+0000 7f2e76d9d640 1 --2- 172.21.15.1:0/2866035177 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e78150520 0x7f2e7814eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.750+0000 7f2e7659c640 1 --2- 172.21.15.1:0/2866035177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e78070c80 0x7f2e781ebaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.750+0000 7f2e75d9b640 1 -- 172.21.15.1:0/2866035177 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e78150520 msgr2=0x7f2e7814eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.750+0000 7f2e75d9b640 1 --2- 172.21.15.1:0/2866035177 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e78150520 0x7f2e7814eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.750+0000 7f2e75d9b640 1 -- 172.21.15.1:0/2866035177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e78070c80 msgr2=0x7f2e781ebaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.750+0000 7f2e75d9b640 1 --2- 172.21.15.1:0/2866035177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e78070c80 0x7f2e781ebaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.750+0000 7f2e75d9b640 1 -- 172.21.15.1:0/2866035177 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e780a87d0 con 0x7f2e780c32f0 2024-03-06T07:27:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.750+0000 7f2e75d9b640 1 --2- 172.21.15.1:0/2866035177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e780c32f0 0x7f2e78155310 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c00c4d0 tx=0x7f2e6c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.750+0000 7f2e66ffd640 1 -- 172.21.15.1:0/2866035177 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c018070 con 0x7f2e780c32f0 2024-03-06T07:27:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.750+0000 7f2e66ffd640 1 -- 172.21.15.1:0/2866035177 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6c007500 con 0x7f2e780c32f0 2024-03-06T07:27:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.750+0000 7f2e66ffd640 1 -- 172.21.15.1:0/2866035177 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c004a20 con 0x7f2e780c32f0 2024-03-06T07:27:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.750+0000 7f2e7ce0e640 1 -- 172.21.15.1:0/2866035177 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e78156480 con 0x7f2e780c32f0 2024-03-06T07:27:57.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.750+0000 7f2e7ce0e640 1 -- 172.21.15.1:0/2866035177 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e78155850 con 0x7f2e780c32f0 2024-03-06T07:27:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.751+0000 7f2e7ce0e640 1 -- 172.21.15.1:0/2866035177 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e780686e0 con 0x7f2e780c32f0 2024-03-06T07:27:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.752+0000 7f2e66ffd640 1 -- 172.21.15.1:0/2866035177 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e6c01b070 con 0x7f2e780c32f0 2024-03-06T07:27:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.752+0000 7f2e66ffd640 1 --2- 172.21.15.1:0/2866035177 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e5806b4d0 0x7f2e5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.752+0000 7f2e66ffd640 1 -- 172.21.15.1:0/2866035177 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2e6c089600 con 0x7f2e780c32f0 2024-03-06T07:27:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.753+0000 7f2e7659c640 1 --2- 172.21.15.1:0/2866035177 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e5806b4d0 0x7f2e5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.753+0000 7f2e7659c640 1 --2- 172.21.15.1:0/2866035177 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e5806b4d0 0x7f2e5806d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f2e68036000 tx=0x7f2e6803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.757+0000 7f2e66ffd640 1 -- 172.21.15.1:0/2866035177 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e6c058380 con 0x7f2e780c32f0 2024-03-06T07:27:57.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:57.978+0000 7f2e7ce0e640 1 -- 172.21.15.1:0/2866035177 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"} v 0) v1 -- 0x7f2e780c5e50 con 0x7f2e780c32f0 2024-03-06T07:27:58.113 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-06T07:27:58.113+0000 7f74ade0b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:58.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.851+0000 7f2e66ffd640 1 -- 172.21.15.1:0/2866035177 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"}]=0 cephfs marked down. v171) v1 ==== 125+0+0 (secure 0 0 0) 0x7f2e6c057c80 con 0x7f2e780c32f0 2024-03-06T07:27:58.853 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked down. 2024-03-06T07:27:58.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.853+0000 7f2e7ce0e640 1 -- 172.21.15.1:0/2866035177 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e5806b4d0 msgr2=0x7f2e5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.853+0000 7f2e7ce0e640 1 --2- 172.21.15.1:0/2866035177 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e5806b4d0 0x7f2e5806d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f2e68036000 tx=0x7f2e6803c040 comp rx=0 tx=0).stop 2024-03-06T07:27:58.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.853+0000 7f2e7ce0e640 1 -- 172.21.15.1:0/2866035177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e780c32f0 msgr2=0x7f2e78155310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.853+0000 7f2e7ce0e640 1 --2- 172.21.15.1:0/2866035177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e780c32f0 0x7f2e78155310 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c00c4d0 tx=0x7f2e6c00c990 comp rx=0 tx=0).stop 2024-03-06T07:27:58.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.853+0000 7f2e7ce0e640 1 -- 172.21.15.1:0/2866035177 shutdown_connections 2024-03-06T07:27:58.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.853+0000 7f2e7ce0e640 1 --2- 172.21.15.1:0/2866035177 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2e5806b4d0 0x7f2e5806d990 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.853+0000 7f2e7ce0e640 1 --2- 172.21.15.1:0/2866035177 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2e78150520 0x7f2e7814eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.853+0000 7f2e7ce0e640 1 --2- 172.21.15.1:0/2866035177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e780c32f0 0x7f2e78155310 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.853+0000 7f2e7ce0e640 1 --2- 172.21.15.1:0/2866035177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e78070c80 0x7f2e781ebaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.853+0000 7f2e7ce0e640 1 -- 172.21.15.1:0/2866035177 >> 172.21.15.1:0/2866035177 conn(0x7f2e780c88f0 msgr2=0x7f2e78154050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.853+0000 7f2e7ce0e640 1 -- 172.21.15.1:0/2866035177 shutdown_connections 2024-03-06T07:27:58.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.853+0000 7f2e7ce0e640 1 -- 172.21.15.1:0/2866035177 wait complete. 2024-03-06T07:27:58.866 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:58.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.965+0000 7fd4d3aed640 1 Processor -- start 2024-03-06T07:27:58.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.966+0000 7fd4d3aed640 1 -- start start 2024-03-06T07:27:58.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.966+0000 7fd4d3aed640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd4cc152920 0x7fd4cc06aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.966+0000 7fd4d3aed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4cc155220 0x7fd4cc1556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.966+0000 7fd4d3aed640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4cc158910 con 0x7fd4cc06b530 2024-03-06T07:27:58.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.966+0000 7fd4d3aed640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4cc152d00 con 0x7fd4cc152920 2024-03-06T07:27:58.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.966+0000 7fd4d3aed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc1596d0 con 0x7fd4cc155220 2024-03-06T07:27:58.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.966+0000 7fd4d1889640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd4cc152920 0x7fd4cc06aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.966+0000 7fd4d1889640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd4cc152920 0x7fd4cc06aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60544/0 (socket says 172.21.15.1:60544) 2024-03-06T07:27:58.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.967+0000 7fd4d1889640 1 -- 172.21.15.1:0/1393520529 learned_addr learned my addr 172.21.15.1:0/1393520529 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:58.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.967+0000 7fd4d208a640 1 --2- 172.21.15.1:0/1393520529 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4cc155220 0x7fd4cc1556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.967+0000 7fd4d1889640 1 -- 172.21.15.1:0/1393520529 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4cc155220 msgr2=0x7fd4cc1556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:58.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.967+0000 7fd4d1889640 1 --2- 172.21.15.1:0/1393520529 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4cc155220 0x7fd4cc1556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.967+0000 7fd4d1889640 1 -- 172.21.15.1:0/1393520529 >> v1:172.21.15.98:6789/0 conn(0x7fd4cc06b530 legacy=0x7fd4cc0c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.967+0000 7fd4d1889640 1 -- 172.21.15.1:0/1393520529 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4cc158bf0 con 0x7fd4cc152920 2024-03-06T07:27:58.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.968+0000 7fd4d1889640 1 --2- 172.21.15.1:0/1393520529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4cc152920 0x7fd4cc06aff0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8009870 tx=0x7fd4c802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91984d7085a954e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.968+0000 7fd4d0887640 1 -- 172.21.15.1:0/1393520529 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8035020 con 0x7fd4cc152920 2024-03-06T07:27:58.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.968+0000 7fd4d0887640 1 -- 172.21.15.1:0/1393520529 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c8031640 con 0x7fd4cc152920 2024-03-06T07:27:58.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.968+0000 7fd4d0887640 1 -- 172.21.15.1:0/1393520529 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c802d040 con 0x7fd4cc152920 2024-03-06T07:27:58.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.968+0000 7fd4d3aed640 1 -- 172.21.15.1:0/1393520529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4cc152920 msgr2=0x7fd4cc06aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.968+0000 7fd4d3aed640 1 --2- 172.21.15.1:0/1393520529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4cc152920 0x7fd4cc06aff0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8009870 tx=0x7fd4c802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.968+0000 7fd4d3aed640 1 -- 172.21.15.1:0/1393520529 shutdown_connections 2024-03-06T07:27:58.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.968+0000 7fd4d3aed640 1 --2- 172.21.15.1:0/1393520529 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4cc155220 0x7fd4cc1556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.968+0000 7fd4d3aed640 1 --2- 172.21.15.1:0/1393520529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4cc152920 0x7fd4cc06aff0 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.968+0000 7fd4d3aed640 1 -- 172.21.15.1:0/1393520529 >> 172.21.15.1:0/1393520529 conn(0x7fd4cc0a9700 msgr2=0x7fd4cc0a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.968+0000 7fd4d3aed640 1 -- 172.21.15.1:0/1393520529 shutdown_connections 2024-03-06T07:27:58.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.968+0000 7fd4d3aed640 1 -- 172.21.15.1:0/1393520529 wait complete. 2024-03-06T07:27:58.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.969+0000 7fd4d3aed640 1 Processor -- start 2024-03-06T07:27:58.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.969+0000 7fd4d3aed640 1 -- start start 2024-03-06T07:27:58.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.970+0000 7fd4d3aed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4cc06b530 0x7fd4cc157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.970+0000 7fd4d3aed640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4cc152920 0x7fd4cc1edf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.970+0000 7fd4d3aed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4cc155220 0x7fd4cc14a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.970+0000 7fd4d3aed640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc0c6380 con 0x7fd4cc152920 2024-03-06T07:27:58.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.970+0000 7fd4d3aed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc159240 con 0x7fd4cc06b530 2024-03-06T07:27:58.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.970+0000 7fd4d3aed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc0c5f60 con 0x7fd4cc155220 2024-03-06T07:27:58.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.970+0000 7fd4d1088640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4cc152920 0x7fd4cc1edf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.970+0000 7fd4d1088640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4cc152920 0x7fd4cc1edf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60560/0 (socket says 172.21.15.1:60560) 2024-03-06T07:27:58.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.970+0000 7fd4d1088640 1 -- 172.21.15.1:0/1550405106 learned_addr learned my addr 172.21.15.1:0/1550405106 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:27:58.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.970+0000 7fd4d208a640 1 --2- 172.21.15.1:0/1550405106 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4cc155220 0x7fd4cc14a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.970+0000 7fd4d1889640 1 --2- 172.21.15.1:0/1550405106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4cc06b530 0x7fd4cc157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.970+0000 7fd4d1088640 1 -- 172.21.15.1:0/1550405106 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4cc155220 msgr2=0x7fd4cc14a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.970+0000 7fd4d1088640 1 --2- 172.21.15.1:0/1550405106 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4cc155220 0x7fd4cc14a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.970+0000 7fd4d1088640 1 -- 172.21.15.1:0/1550405106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4cc06b530 msgr2=0x7fd4cc157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.970+0000 7fd4d1088640 1 --2- 172.21.15.1:0/1550405106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4cc06b530 0x7fd4cc157e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.970+0000 7fd4d1088640 1 -- 172.21.15.1:0/1550405106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4cc158340 con 0x7fd4cc152920 2024-03-06T07:27:58.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.971+0000 7fd4d1088640 1 --2- 172.21.15.1:0/1550405106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4cc152920 0x7fd4cc1edf70 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc00c440 tx=0x7fd4bc001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.971+0000 7fd4c27fc640 1 -- 172.21.15.1:0/1550405106 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc0052d0 con 0x7fd4cc152920 2024-03-06T07:27:58.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.971+0000 7fd4c27fc640 1 -- 172.21.15.1:0/1550405106 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4bc005470 con 0x7fd4cc152920 2024-03-06T07:27:58.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.971+0000 7fd4d3aed640 1 -- 172.21.15.1:0/1550405106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4cc157640 con 0x7fd4cc152920 2024-03-06T07:27:58.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.971+0000 7fd4c27fc640 1 -- 172.21.15.1:0/1550405106 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc007260 con 0x7fd4cc152920 2024-03-06T07:27:58.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.971+0000 7fd4d3aed640 1 -- 172.21.15.1:0/1550405106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4cc153f50 con 0x7fd4cc152920 2024-03-06T07:27:58.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.972+0000 7fd4d3aed640 1 -- 172.21.15.1:0/1550405106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4cc067fd0 con 0x7fd4cc152920 2024-03-06T07:27:58.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.972+0000 7fd4c27fc640 1 -- 172.21.15.1:0/1550405106 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4bc007400 con 0x7fd4cc152920 2024-03-06T07:27:58.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.973+0000 7fd4c27fc640 1 --2- 172.21.15.1:0/1550405106 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd4b006b4d0 0x7fd4b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.973+0000 7fd4c27fc640 1 -- 172.21.15.1:0/1550405106 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd4bc089840 con 0x7fd4cc152920 2024-03-06T07:27:58.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.973+0000 7fd4d1889640 1 --2- 172.21.15.1:0/1550405106 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd4b006b4d0 0x7fd4b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.973+0000 7fd4d1889640 1 --2- 172.21.15.1:0/1550405106 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd4b006b4d0 0x7fd4b006d990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8036000 tx=0x7fd4c803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:58.977+0000 7fd4c27fc640 1 -- 172.21.15.1:0/1550405106 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd4bc0585c0 con 0x7fd4cc152920 2024-03-06T07:27:59.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:59.195+0000 7fd4d3aed640 1 -- 172.21.15.1:0/1550405106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4cc06bdf0 con 0x7fd4cc152920 2024-03-06T07:27:59.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:59.195+0000 7fd4c27fc640 1 -- 172.21.15.1:0/1550405106 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4976 (secure 0 0 0) 0x7fd4bc057ec0 con 0x7fd4cc152920 2024-03-06T07:27:59.197 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:27:59.197 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":172,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":9951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:27:58.854215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9954":{"gid":9954,"name":"c","rank":0,"incarnation":168,"state":"up:stopping","state_seq":3,"addr":"172.21.15.1:6835/1300509547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1300509547},{"type":"v1","addr":"172.21.15.1:6835","nonce":1300509547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:27:59.197 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 172 2024-03-06T07:27:59.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:59.197+0000 7fd4d3aed640 1 -- 172.21.15.1:0/1550405106 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd4b006b4d0 msgr2=0x7fd4b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:59.197+0000 7fd4d3aed640 1 --2- 172.21.15.1:0/1550405106 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd4b006b4d0 0x7fd4b006d990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8036000 tx=0x7fd4c803c040 comp rx=0 tx=0).stop 2024-03-06T07:27:59.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:59.197+0000 7fd4d3aed640 1 -- 172.21.15.1:0/1550405106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4cc152920 msgr2=0x7fd4cc1edf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:59.197+0000 7fd4d3aed640 1 --2- 172.21.15.1:0/1550405106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4cc152920 0x7fd4cc1edf70 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc00c440 tx=0x7fd4bc001e70 comp rx=0 tx=0).stop 2024-03-06T07:27:59.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:59.197+0000 7fd4d3aed640 1 -- 172.21.15.1:0/1550405106 shutdown_connections 2024-03-06T07:27:59.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:59.197+0000 7fd4d3aed640 1 --2- 172.21.15.1:0/1550405106 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd4b006b4d0 0x7fd4b006d990 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:59.197+0000 7fd4d3aed640 1 --2- 172.21.15.1:0/1550405106 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4cc155220 0x7fd4cc14a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:59.197+0000 7fd4d3aed640 1 --2- 172.21.15.1:0/1550405106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4cc152920 0x7fd4cc1edf70 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:59.197+0000 7fd4d3aed640 1 --2- 172.21.15.1:0/1550405106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4cc06b530 0x7fd4cc157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:59.197+0000 7fd4d3aed640 1 -- 172.21.15.1:0/1550405106 >> 172.21.15.1:0/1550405106 conn(0x7fd4cc0a9700 msgr2=0x7fd4cc0c6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:59.197+0000 7fd4d3aed640 1 -- 172.21.15.1:0/1550405106 shutdown_connections 2024-03-06T07:27:59.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:27:59.197+0000 7fd4d3aed640 1 -- 172.21.15.1:0/1550405106 wait complete. 2024-03-06T07:27:59.612 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 172, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:52.179842+0000', 'modified': '2024-03-06T07:27:58.854215+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9954}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9954': {'gid': 9954, 'name': 'c', 'rank': 0, 'incarnation': 168, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.1:6835/1300509547', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 1300509547}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1300509547}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:59.612 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9954:up:stopping 2024-03-06T07:28:00.613 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:00.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.724+0000 7f7da34df640 1 Processor -- start 2024-03-06T07:28:00.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.724+0000 7f7da34df640 1 -- start start 2024-03-06T07:28:00.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.725+0000 7f7da34df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d9c152690 0x7f7d9c155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.725+0000 7f7da34df640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d9c156cd0 0x7f7d9c157160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.725+0000 7f7da34df640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9c1561e0 con 0x7f7d9c0c6730 2024-03-06T07:28:00.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.725+0000 7f7da34df640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d9c0c3300 con 0x7f7d9c156cd0 2024-03-06T07:28:00.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.725+0000 7f7da34df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c0c35a0 con 0x7f7d9c152690 2024-03-06T07:28:00.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.725+0000 7f7da1a7c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d9c156cd0 0x7f7d9c157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.725+0000 7f7da1a7c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d9c156cd0 0x7f7d9c157160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60562/0 (socket says 172.21.15.1:60562) 2024-03-06T07:28:00.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.725+0000 7f7da1a7c640 1 -- 172.21.15.1:0/2287576650 learned_addr learned my addr 172.21.15.1:0/2287576650 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:00.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.725+0000 7f7da127b640 1 --2- 172.21.15.1:0/2287576650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d9c152690 0x7f7d9c155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.725+0000 7f7da1a7c640 1 -- 172.21.15.1:0/2287576650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d9c152690 msgr2=0x7f7d9c155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:00.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.725+0000 7f7da1a7c640 1 --2- 172.21.15.1:0/2287576650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d9c152690 0x7f7d9c155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.725+0000 7f7da1a7c640 1 -- 172.21.15.1:0/2287576650 >> v1:172.21.15.98:6789/0 conn(0x7f7d9c0c6730 legacy=0x7f7d9c0c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:00.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.725+0000 7f7da1a7c640 1 -- 172.21.15.1:0/2287576650 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9c0c3880 con 0x7f7d9c156cd0 2024-03-06T07:28:00.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.726+0000 7f7da1a7c640 1 --2- 172.21.15.1:0/2287576650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c156cd0 0x7f7d9c157160 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f7d94009870 tx=0x7f7d9402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94524677f9da2a5f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.726+0000 7f7d93fff640 1 -- 172.21.15.1:0/2287576650 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d940317c0 con 0x7f7d9c156cd0 2024-03-06T07:28:00.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.726+0000 7f7d93fff640 1 -- 172.21.15.1:0/2287576650 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d94037070 con 0x7f7d9c156cd0 2024-03-06T07:28:00.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.726+0000 7f7d93fff640 1 -- 172.21.15.1:0/2287576650 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d94031b80 con 0x7f7d9c156cd0 2024-03-06T07:28:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.726+0000 7f7da34df640 1 -- 172.21.15.1:0/2287576650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c156cd0 msgr2=0x7f7d9c157160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.726+0000 7f7da34df640 1 --2- 172.21.15.1:0/2287576650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c156cd0 0x7f7d9c157160 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f7d94009870 tx=0x7f7d9402b860 comp rx=0 tx=0).stop 2024-03-06T07:28:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.727+0000 7f7da34df640 1 -- 172.21.15.1:0/2287576650 shutdown_connections 2024-03-06T07:28:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.727+0000 7f7da34df640 1 --2- 172.21.15.1:0/2287576650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c156cd0 0x7f7d9c157160 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.727+0000 7f7da34df640 1 --2- 172.21.15.1:0/2287576650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d9c152690 0x7f7d9c155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.727+0000 7f7da34df640 1 -- 172.21.15.1:0/2287576650 >> 172.21.15.1:0/2287576650 conn(0x7f7d9c06b410 msgr2=0x7f7d9c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.727+0000 7f7da34df640 1 -- 172.21.15.1:0/2287576650 shutdown_connections 2024-03-06T07:28:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.727+0000 7f7da34df640 1 -- 172.21.15.1:0/2287576650 wait complete. 2024-03-06T07:28:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.728+0000 7f7da34df640 1 Processor -- start 2024-03-06T07:28:00.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.728+0000 7f7da34df640 1 -- start start 2024-03-06T07:28:00.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.728+0000 7f7da34df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d9c0c6730 0x7f7d9c1eea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.728+0000 7f7da34df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c152690 0x7f7d9c14c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.728+0000 7f7da34df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d9c156cd0 0x7f7d9c0709b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.728+0000 7f7da34df640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c0c5ee0 con 0x7f7d9c152690 2024-03-06T07:28:00.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.728+0000 7f7da34df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c1567b0 con 0x7f7d9c0c6730 2024-03-06T07:28:00.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.728+0000 7f7da34df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c0c6060 con 0x7f7d9c156cd0 2024-03-06T07:28:00.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.729+0000 7f7da0a7a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c152690 0x7f7d9c14c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.729+0000 7f7da0a7a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c152690 0x7f7d9c14c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60566/0 (socket says 172.21.15.1:60566) 2024-03-06T07:28:00.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.729+0000 7f7da0a7a640 1 -- 172.21.15.1:0/863865658 learned_addr learned my addr 172.21.15.1:0/863865658 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:00.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.729+0000 7f7da127b640 1 --2- 172.21.15.1:0/863865658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d9c0c6730 0x7f7d9c1eea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.729+0000 7f7da1a7c640 1 --2- 172.21.15.1:0/863865658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d9c156cd0 0x7f7d9c0709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.729+0000 7f7da0a7a640 1 -- 172.21.15.1:0/863865658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d9c156cd0 msgr2=0x7f7d9c0709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.729+0000 7f7da0a7a640 1 --2- 172.21.15.1:0/863865658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d9c156cd0 0x7f7d9c0709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.729+0000 7f7da0a7a640 1 -- 172.21.15.1:0/863865658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d9c0c6730 msgr2=0x7f7d9c1eea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.729+0000 7f7da0a7a640 1 --2- 172.21.15.1:0/863865658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d9c0c6730 0x7f7d9c1eea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.729+0000 7f7da0a7a640 1 -- 172.21.15.1:0/863865658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9c0c85c0 con 0x7f7d9c152690 2024-03-06T07:28:00.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.729+0000 7f7da0a7a640 1 --2- 172.21.15.1:0/863865658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c152690 0x7f7d9c14c780 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c00b3f0 tx=0x7f7d8c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.729+0000 7f7d91ffb640 1 -- 172.21.15.1:0/863865658 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c015020 con 0x7f7d9c152690 2024-03-06T07:28:00.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.729+0000 7f7d91ffb640 1 -- 172.21.15.1:0/863865658 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8c011830 con 0x7f7d9c152690 2024-03-06T07:28:00.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.729+0000 7f7d91ffb640 1 -- 172.21.15.1:0/863865658 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c011c20 con 0x7f7d9c152690 2024-03-06T07:28:00.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.729+0000 7f7da34df640 1 -- 172.21.15.1:0/863865658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d9c1eefa0 con 0x7f7d9c152690 2024-03-06T07:28:00.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.730+0000 7f7da34df640 1 -- 172.21.15.1:0/863865658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d9c0c8830 con 0x7f7d9c152690 2024-03-06T07:28:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.731+0000 7f7da34df640 1 -- 172.21.15.1:0/863865658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d9c067fd0 con 0x7f7d9c152690 2024-03-06T07:28:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.731+0000 7f7d91ffb640 1 -- 172.21.15.1:0/863865658 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d8c018020 con 0x7f7d9c152690 2024-03-06T07:28:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.731+0000 7f7d91ffb640 1 --2- 172.21.15.1:0/863865658 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d8006b4d0 0x7f7d8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.732+0000 7f7d91ffb640 1 -- 172.21.15.1:0/863865658 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7d8c089d70 con 0x7f7d9c152690 2024-03-06T07:28:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.732+0000 7f7da127b640 1 --2- 172.21.15.1:0/863865658 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d8006b4d0 0x7f7d8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.732+0000 7f7da127b640 1 --2- 172.21.15.1:0/863865658 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d8006b4d0 0x7f7d8006d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c0711b0 tx=0x7f7d98007710 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.736+0000 7f7d91ffb640 1 -- 172.21.15.1:0/863865658 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d8c01f030 con 0x7f7d9c152690 2024-03-06T07:28:00.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.956+0000 7f7da34df640 1 -- 172.21.15.1:0/863865658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d9c0a98c0 con 0x7f7d9c152690 2024-03-06T07:28:00.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.957+0000 7f7d91ffb640 1 -- 172.21.15.1:0/863865658 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4976 (secure 0 0 0) 0x7f7d8c0584a0 con 0x7f7d9c152690 2024-03-06T07:28:00.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:00.959 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":172,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":9951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:27:58.854215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9954":{"gid":9954,"name":"c","rank":0,"incarnation":168,"state":"up:stopping","state_seq":3,"addr":"172.21.15.1:6835/1300509547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1300509547},{"type":"v1","addr":"172.21.15.1:6835","nonce":1300509547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:28:00.960 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 172 2024-03-06T07:28:00.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.959+0000 7f7da34df640 1 -- 172.21.15.1:0/863865658 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d8006b4d0 msgr2=0x7f7d8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.959+0000 7f7da34df640 1 --2- 172.21.15.1:0/863865658 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d8006b4d0 0x7f7d8006d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c0711b0 tx=0x7f7d98007710 comp rx=0 tx=0).stop 2024-03-06T07:28:00.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.959+0000 7f7da34df640 1 -- 172.21.15.1:0/863865658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c152690 msgr2=0x7f7d9c14c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.959+0000 7f7da34df640 1 --2- 172.21.15.1:0/863865658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c152690 0x7f7d9c14c780 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c00b3f0 tx=0x7f7d8c00b8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.959+0000 7f7da34df640 1 -- 172.21.15.1:0/863865658 shutdown_connections 2024-03-06T07:28:00.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.959+0000 7f7da34df640 1 --2- 172.21.15.1:0/863865658 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d8006b4d0 0x7f7d8006d990 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.959+0000 7f7da34df640 1 --2- 172.21.15.1:0/863865658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d9c156cd0 0x7f7d9c0709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.959+0000 7f7da34df640 1 --2- 172.21.15.1:0/863865658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c152690 0x7f7d9c14c780 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.959+0000 7f7da34df640 1 --2- 172.21.15.1:0/863865658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d9c0c6730 0x7f7d9c1eea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.959+0000 7f7da34df640 1 -- 172.21.15.1:0/863865658 >> 172.21.15.1:0/863865658 conn(0x7f7d9c06b410 msgr2=0x7f7d9c0a9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.959+0000 7f7da34df640 1 -- 172.21.15.1:0/863865658 shutdown_connections 2024-03-06T07:28:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:00.959+0000 7f7da34df640 1 -- 172.21.15.1:0/863865658 wait complete. 2024-03-06T07:28:01.374 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 172, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:52.179842+0000', 'modified': '2024-03-06T07:27:58.854215+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9954}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9954': {'gid': 9954, 'name': 'c', 'rank': 0, 'incarnation': 168, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.1:6835/1300509547', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 1300509547}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1300509547}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:28:01.374 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9954:up:stopping 2024-03-06T07:28:02.370 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:02.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:02.375 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.521+0000 7f937569a640 1 Processor -- start 2024-03-06T07:28:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.521+0000 7f937569a640 1 -- start start 2024-03-06T07:28:02.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.522+0000 7f937569a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9370152820 0x7f93700c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.522+0000 7f937569a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f93700c0670 0x7f93700c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.522+0000 7f937569a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93700c7930 0x7f9370158830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.522+0000 7f937569a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93700c7180 con 0x7f93700c0670 2024-03-06T07:28:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.522+0000 7f937569a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9370155460 con 0x7f9370152820 2024-03-06T07:28:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.522+0000 7f937569a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93700c6bc0 con 0x7f93700c7930 2024-03-06T07:28:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.522+0000 7f936e7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f93700c0670 0x7f93700c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.522+0000 7f936e7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f93700c0670 0x7f93700c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60568/0 (socket says 172.21.15.1:60568) 2024-03-06T07:28:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.522+0000 7f936e7fc640 1 -- 172.21.15.1:0/1079332570 learned_addr learned my addr 172.21.15.1:0/1079332570 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.522+0000 7f936f7fe640 1 --2- 172.21.15.1:0/1079332570 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93700c7930 0x7f9370158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.522+0000 7f936effd640 1 --2- 172.21.15.1:0/1079332570 >> v2:172.21.15.98:3300/0 conn(0x7f9370152820 0x7f93700c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.522+0000 7f936e7fc640 1 -- 172.21.15.1:0/1079332570 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93700c7930 msgr2=0x7f9370158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.522+0000 7f936e7fc640 1 --2- 172.21.15.1:0/1079332570 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93700c7930 0x7f9370158830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.522+0000 7f936e7fc640 1 -- 172.21.15.1:0/1079332570 >> v2:172.21.15.98:3300/0 conn(0x7f9370152820 msgr2=0x7f93700c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.522+0000 7f936e7fc640 1 --2- 172.21.15.1:0/1079332570 >> v2:172.21.15.98:3300/0 conn(0x7f9370152820 0x7f93700c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.523+0000 7f936e7fc640 1 -- 172.21.15.1:0/1079332570 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93700c6ea0 con 0x7f93700c0670 2024-03-06T07:28:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.523+0000 7f936f7fe640 1 --2- 172.21.15.1:0/1079332570 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93700c7930 0x7f9370158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.523+0000 7f936effd640 1 --2- 172.21.15.1:0/1079332570 >> v2:172.21.15.98:3300/0 conn(0x7f9370152820 0x7f93700c0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:02.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.523+0000 7f936e7fc640 1 --2- 172.21.15.1:0/1079332570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93700c0670 0x7f93700c73f0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f937008ccc0 tx=0x7f936402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70bd087ed1ff2d99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.523+0000 7f936dffb640 1 -- 172.21.15.1:0/1079332570 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9364035020 con 0x7f93700c0670 2024-03-06T07:28:02.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.523+0000 7f936dffb640 1 -- 172.21.15.1:0/1079332570 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9364031870 con 0x7f93700c0670 2024-03-06T07:28:02.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.523+0000 7f936dffb640 1 -- 172.21.15.1:0/1079332570 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9364031c40 con 0x7f93700c0670 2024-03-06T07:28:02.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.524+0000 7f937569a640 1 -- 172.21.15.1:0/1079332570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93700c0670 msgr2=0x7f93700c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.524+0000 7f937569a640 1 --2- 172.21.15.1:0/1079332570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93700c0670 0x7f93700c73f0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f937008ccc0 tx=0x7f936402b910 comp rx=0 tx=0).stop 2024-03-06T07:28:02.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.524+0000 7f937569a640 1 -- 172.21.15.1:0/1079332570 shutdown_connections 2024-03-06T07:28:02.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.524+0000 7f937569a640 1 --2- 172.21.15.1:0/1079332570 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93700c7930 0x7f9370158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.524+0000 7f937569a640 1 --2- 172.21.15.1:0/1079332570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93700c0670 0x7f93700c73f0 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.524+0000 7f937569a640 1 --2- 172.21.15.1:0/1079332570 >> v2:172.21.15.98:3300/0 conn(0x7f9370152820 0x7f93700c0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.524+0000 7f937569a640 1 -- 172.21.15.1:0/1079332570 >> 172.21.15.1:0/1079332570 conn(0x7f9370072720 msgr2=0x7f9370072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.524+0000 7f937569a640 1 -- 172.21.15.1:0/1079332570 shutdown_connections 2024-03-06T07:28:02.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.524+0000 7f937569a640 1 -- 172.21.15.1:0/1079332570 wait complete. 2024-03-06T07:28:02.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.525+0000 7f937569a640 1 Processor -- start 2024-03-06T07:28:02.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.525+0000 7f937569a640 1 -- start start 2024-03-06T07:28:02.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.526+0000 7f937569a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93700c0670 0x7f93701ee0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.526+0000 7f937569a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93700c7930 0x7f937014b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.526+0000 7f937569a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9370152820 0x7f937014bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.526+0000 7f937569a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9370152d80 con 0x7f9370152820 2024-03-06T07:28:02.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.526+0000 7f937569a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9370152c00 con 0x7f93700c0670 2024-03-06T07:28:02.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.526+0000 7f937569a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93700af6f0 con 0x7f93700c7930 2024-03-06T07:28:02.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.526+0000 7f936f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9370152820 0x7f937014bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.526+0000 7f936f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9370152820 0x7f937014bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60582/0 (socket says 172.21.15.1:60582) 2024-03-06T07:28:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.526+0000 7f936f7fe640 1 -- 172.21.15.1:0/1047057335 learned_addr learned my addr 172.21.15.1:0/1047057335 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.526+0000 7f936e7fc640 1 --2- 172.21.15.1:0/1047057335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93700c7930 0x7f937014b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.526+0000 7f936effd640 1 --2- 172.21.15.1:0/1047057335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93700c0670 0x7f93701ee0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.526+0000 7f936f7fe640 1 -- 172.21.15.1:0/1047057335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93700c7930 msgr2=0x7f937014b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.526+0000 7f936f7fe640 1 --2- 172.21.15.1:0/1047057335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93700c7930 0x7f937014b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.527+0000 7f936f7fe640 1 -- 172.21.15.1:0/1047057335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93700c0670 msgr2=0x7f93701ee0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.527+0000 7f936f7fe640 1 --2- 172.21.15.1:0/1047057335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93700c0670 0x7f93701ee0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.527+0000 7f936f7fe640 1 -- 172.21.15.1:0/1047057335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93701df330 con 0x7f9370152820 2024-03-06T07:28:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.527+0000 7f936e7fc640 1 --2- 172.21.15.1:0/1047057335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93700c7930 0x7f937014b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.527+0000 7f936effd640 1 --2- 172.21.15.1:0/1047057335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93700c0670 0x7f93701ee0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.527+0000 7f936f7fe640 1 --2- 172.21.15.1:0/1047057335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9370152820 0x7f937014bb60 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f936800e610 tx=0x7f936800ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.527+0000 7f93577fe640 1 -- 172.21.15.1:0/1047057335 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936800f040 con 0x7f9370152820 2024-03-06T07:28:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.527+0000 7f93577fe640 1 -- 172.21.15.1:0/1047057335 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9368004230 con 0x7f9370152820 2024-03-06T07:28:02.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.527+0000 7f937569a640 1 -- 172.21.15.1:0/1047057335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9370156320 con 0x7f9370152820 2024-03-06T07:28:02.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.527+0000 7f93577fe640 1 -- 172.21.15.1:0/1047057335 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9368003610 con 0x7f9370152820 2024-03-06T07:28:02.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.527+0000 7f937569a640 1 -- 172.21.15.1:0/1047057335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93701eecc0 con 0x7f9370152820 2024-03-06T07:28:02.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.528+0000 7f937569a640 1 -- 172.21.15.1:0/1047057335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93700686e0 con 0x7f9370152820 2024-03-06T07:28:02.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.528+0000 7f93577fe640 1 -- 172.21.15.1:0/1047057335 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f93680037b0 con 0x7f9370152820 2024-03-06T07:28:02.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.529+0000 7f93577fe640 1 --2- 172.21.15.1:0/1047057335 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f935c06b4d0 0x7f935c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.529+0000 7f93577fe640 1 -- 172.21.15.1:0/1047057335 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9368089f70 con 0x7f9370152820 2024-03-06T07:28:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.529+0000 7f936effd640 1 --2- 172.21.15.1:0/1047057335 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f935c06b4d0 0x7f935c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.530+0000 7f936effd640 1 --2- 172.21.15.1:0/1047057335 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f935c06b4d0 0x7f935c06d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f9360009540 tx=0x7f936000b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.533+0000 7f93577fe640 1 -- 172.21.15.1:0/1047057335 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9368017020 con 0x7f9370152820 2024-03-06T07:28:02.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.750+0000 7f937569a640 1 -- 172.21.15.1:0/1047057335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93700c5b40 con 0x7f9370152820 2024-03-06T07:28:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.750+0000 7f93577fe640 1 -- 172.21.15.1:0/1047057335 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4976 (secure 0 0 0) 0x7f9368058cf0 con 0x7f9370152820 2024-03-06T07:28:02.752 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:02.752 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":172,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":9951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:27:58.854215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9954":{"gid":9954,"name":"c","rank":0,"incarnation":168,"state":"up:stopping","state_seq":3,"addr":"172.21.15.1:6835/1300509547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1300509547},{"type":"v1","addr":"172.21.15.1:6835","nonce":1300509547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:28:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 172 2024-03-06T07:28:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.752+0000 7f937569a640 1 -- 172.21.15.1:0/1047057335 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f935c06b4d0 msgr2=0x7f935c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.752+0000 7f937569a640 1 --2- 172.21.15.1:0/1047057335 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f935c06b4d0 0x7f935c06d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f9360009540 tx=0x7f936000b040 comp rx=0 tx=0).stop 2024-03-06T07:28:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.752+0000 7f937569a640 1 -- 172.21.15.1:0/1047057335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9370152820 msgr2=0x7f937014bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.752+0000 7f937569a640 1 --2- 172.21.15.1:0/1047057335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9370152820 0x7f937014bb60 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f936800e610 tx=0x7f936800ead0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.752+0000 7f937569a640 1 -- 172.21.15.1:0/1047057335 shutdown_connections 2024-03-06T07:28:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.752+0000 7f937569a640 1 --2- 172.21.15.1:0/1047057335 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f935c06b4d0 0x7f935c06d990 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.752+0000 7f937569a640 1 --2- 172.21.15.1:0/1047057335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9370152820 0x7f937014bb60 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.752+0000 7f937569a640 1 --2- 172.21.15.1:0/1047057335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93700c7930 0x7f937014b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.752+0000 7f937569a640 1 --2- 172.21.15.1:0/1047057335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93700c0670 0x7f93701ee0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.752+0000 7f937569a640 1 -- 172.21.15.1:0/1047057335 >> 172.21.15.1:0/1047057335 conn(0x7f9370072720 msgr2=0x7f93700745c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.752+0000 7f937569a640 1 -- 172.21.15.1:0/1047057335 shutdown_connections 2024-03-06T07:28:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:02.752+0000 7f937569a640 1 -- 172.21.15.1:0/1047057335 wait complete. 2024-03-06T07:28:03.168 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 172, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:52.179842+0000', 'modified': '2024-03-06T07:27:58.854215+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9954}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9954': {'gid': 9954, 'name': 'c', 'rank': 0, 'incarnation': 168, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.1:6835/1300509547', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 1300509547}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1300509547}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:28:03.168 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9954:up:stopping 2024-03-06T07:28:04.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:04.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.280+0000 7fc3a5a5c640 1 Processor -- start 2024-03-06T07:28:04.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.280+0000 7fc3a5a5c640 1 -- start start 2024-03-06T07:28:04.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.281+0000 7fc3a5a5c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc3a00af6f0 0x7fc3a00c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.281+0000 7fc3a5a5c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3a00c7930 0x7fc3a015fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.281+0000 7fc3a5a5c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a00c7180 con 0x7fc3a0152820 2024-03-06T07:28:04.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.281+0000 7fc3a5a5c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a0155460 con 0x7fc3a00af6f0 2024-03-06T07:28:04.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.281+0000 7fc3a5a5c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a00c6bc0 con 0x7fc3a00c7930 2024-03-06T07:28:04.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.281+0000 7fc39e7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc3a00af6f0 0x7fc3a00c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.281+0000 7fc39f7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3a00c7930 0x7fc3a015fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.282+0000 7fc39e7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc3a00af6f0 0x7fc3a00c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:56494/0 (socket says 172.21.15.1:56494) 2024-03-06T07:28:04.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.282+0000 7fc39e7fc640 1 -- 172.21.15.1:0/2333242456 learned_addr learned my addr 172.21.15.1:0/2333242456 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:04.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.282+0000 7fc39e7fc640 1 -- 172.21.15.1:0/2333242456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3a00c7930 msgr2=0x7fc3a015fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:04.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.282+0000 7fc39e7fc640 1 --2- 172.21.15.1:0/2333242456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3a00c7930 0x7fc3a015fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.282+0000 7fc39e7fc640 1 -- 172.21.15.1:0/2333242456 >> v1:172.21.15.98:6789/0 conn(0x7fc3a0152820 legacy=0x7fc3a00c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.282+0000 7fc39e7fc640 1 -- 172.21.15.1:0/2333242456 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a00c6ea0 con 0x7fc3a00af6f0 2024-03-06T07:28:04.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.282+0000 7fc39f7fe640 1 --2- 172.21.15.1:0/2333242456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3a00c7930 0x7fc3a015fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.283+0000 7fc39e7fc640 1 --2- 172.21.15.1:0/2333242456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3a00af6f0 0x7fc3a00c73f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fc394009870 tx=0x7fc39402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41628a58534b62f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.283+0000 7fc39dffb640 1 -- 172.21.15.1:0/2333242456 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394034070 con 0x7fc3a00af6f0 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.283+0000 7fc39dffb640 1 -- 172.21.15.1:0/2333242456 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc394030840 con 0x7fc3a00af6f0 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.283+0000 7fc39dffb640 1 -- 172.21.15.1:0/2333242456 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394030c10 con 0x7fc3a00af6f0 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.283+0000 7fc3a5a5c640 1 -- 172.21.15.1:0/2333242456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3a00af6f0 msgr2=0x7fc3a00c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.283+0000 7fc3a5a5c640 1 --2- 172.21.15.1:0/2333242456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3a00af6f0 0x7fc3a00c73f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fc394009870 tx=0x7fc39402b980 comp rx=0 tx=0).stop 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.283+0000 7fc3a5a5c640 1 -- 172.21.15.1:0/2333242456 shutdown_connections 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.284+0000 7fc3a5a5c640 1 --2- 172.21.15.1:0/2333242456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3a00c7930 0x7fc3a015fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.284+0000 7fc3a5a5c640 1 --2- 172.21.15.1:0/2333242456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3a00af6f0 0x7fc3a00c73f0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.284+0000 7fc3a5a5c640 1 -- 172.21.15.1:0/2333242456 >> 172.21.15.1:0/2333242456 conn(0x7fc3a0072720 msgr2=0x7fc3a0072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.284+0000 7fc3a5a5c640 1 -- 172.21.15.1:0/2333242456 shutdown_connections 2024-03-06T07:28:04.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.284+0000 7fc3a5a5c640 1 -- 172.21.15.1:0/2333242456 wait complete. 2024-03-06T07:28:04.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.285+0000 7fc3a5a5c640 1 Processor -- start 2024-03-06T07:28:04.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.285+0000 7fc3a5a5c640 1 -- start start 2024-03-06T07:28:04.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.285+0000 7fc3a5a5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3a00af6f0 0x7fc3a0159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.285+0000 7fc3a5a5c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3a00c7930 0x7fc3a00c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.285+0000 7fc3a5a5c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3a0152820 0x7fc3a0157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.285+0000 7fc3a5a5c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a0152c00 con 0x7fc3a00c7930 2024-03-06T07:28:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.285+0000 7fc3a5a5c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a00c0840 con 0x7fc3a00af6f0 2024-03-06T07:28:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.285+0000 7fc3a5a5c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a0152d80 con 0x7fc3a0152820 2024-03-06T07:28:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.286+0000 7fc39e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3a00c7930 0x7fc3a00c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.286+0000 7fc39effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3a00af6f0 0x7fc3a0159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.286+0000 7fc39e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3a00c7930 0x7fc3a00c9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60584/0 (socket says 172.21.15.1:60584) 2024-03-06T07:28:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.286+0000 7fc39e7fc640 1 -- 172.21.15.1:0/2987001168 learned_addr learned my addr 172.21.15.1:0/2987001168 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.286+0000 7fc39f7fe640 1 --2- 172.21.15.1:0/2987001168 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3a0152820 0x7fc3a0157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.286+0000 7fc39e7fc640 1 -- 172.21.15.1:0/2987001168 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3a0152820 msgr2=0x7fc3a0157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.286+0000 7fc39e7fc640 1 --2- 172.21.15.1:0/2987001168 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3a0152820 0x7fc3a0157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.286+0000 7fc39e7fc640 1 -- 172.21.15.1:0/2987001168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3a00af6f0 msgr2=0x7fc3a0159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.286+0000 7fc39e7fc640 1 --2- 172.21.15.1:0/2987001168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3a00af6f0 0x7fc3a0159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.286+0000 7fc39e7fc640 1 -- 172.21.15.1:0/2987001168 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a014c7e0 con 0x7fc3a00c7930 2024-03-06T07:28:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.286+0000 7fc39e7fc640 1 --2- 172.21.15.1:0/2987001168 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3a00c7930 0x7fc3a00c9960 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7fc394000c00 tx=0x7fc394004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.286+0000 7fc387fff640 1 -- 172.21.15.1:0/2987001168 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394048070 con 0x7fc3a00c7930 2024-03-06T07:28:04.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.286+0000 7fc387fff640 1 -- 172.21.15.1:0/2987001168 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc394030640 con 0x7fc3a00c7930 2024-03-06T07:28:04.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.286+0000 7fc3a5a5c640 1 -- 172.21.15.1:0/2987001168 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a00ca410 con 0x7fc3a00c7930 2024-03-06T07:28:04.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.286+0000 7fc387fff640 1 -- 172.21.15.1:0/2987001168 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394034040 con 0x7fc3a00c7930 2024-03-06T07:28:04.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.286+0000 7fc3a5a5c640 1 -- 172.21.15.1:0/2987001168 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a00ca6f0 con 0x7fc3a00c7930 2024-03-06T07:28:04.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.287+0000 7fc3a5a5c640 1 -- 172.21.15.1:0/2987001168 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a00686e0 con 0x7fc3a00c7930 2024-03-06T07:28:04.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.288+0000 7fc387fff640 1 -- 172.21.15.1:0/2987001168 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc39404b070 con 0x7fc3a00c7930 2024-03-06T07:28:04.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.288+0000 7fc387fff640 1 --2- 172.21.15.1:0/2987001168 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc38c06b500 0x7fc38c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.288+0000 7fc387fff640 1 -- 172.21.15.1:0/2987001168 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc3940ac2a0 con 0x7fc3a00c7930 2024-03-06T07:28:04.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.289+0000 7fc39effd640 1 --2- 172.21.15.1:0/2987001168 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc38c06b500 0x7fc38c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.289+0000 7fc39effd640 1 --2- 172.21.15.1:0/2987001168 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc38c06b500 0x7fc38c06d9c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc3a00afe50 tx=0x7fc398009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.293+0000 7fc387fff640 1 -- 172.21.15.1:0/2987001168 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc394030810 con 0x7fc3a00c7930 2024-03-06T07:28:04.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.514+0000 7fc3a5a5c640 1 -- 172.21.15.1:0/2987001168 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3a00c60a0 con 0x7fc3a00c7930 2024-03-06T07:28:04.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.514+0000 7fc387fff640 1 -- 172.21.15.1:0/2987001168 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4976 (secure 0 0 0) 0x7fc39407aaf0 con 0x7fc3a00c7930 2024-03-06T07:28:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:04.516 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":172,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":9951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:27:58.854215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9954":{"gid":9954,"name":"c","rank":0,"incarnation":168,"state":"up:stopping","state_seq":3,"addr":"172.21.15.1:6835/1300509547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1300509547},{"type":"v1","addr":"172.21.15.1:6835","nonce":1300509547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:28:04.517 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 172 2024-03-06T07:28:04.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.516+0000 7fc3a5a5c640 1 -- 172.21.15.1:0/2987001168 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc38c06b500 msgr2=0x7fc38c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.516+0000 7fc3a5a5c640 1 --2- 172.21.15.1:0/2987001168 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc38c06b500 0x7fc38c06d9c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc3a00afe50 tx=0x7fc398009040 comp rx=0 tx=0).stop 2024-03-06T07:28:04.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.516+0000 7fc3a5a5c640 1 -- 172.21.15.1:0/2987001168 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3a00c7930 msgr2=0x7fc3a00c9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.516+0000 7fc3a5a5c640 1 --2- 172.21.15.1:0/2987001168 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3a00c7930 0x7fc3a00c9960 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7fc394000c00 tx=0x7fc394004730 comp rx=0 tx=0).stop 2024-03-06T07:28:04.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.516+0000 7fc3a5a5c640 1 -- 172.21.15.1:0/2987001168 shutdown_connections 2024-03-06T07:28:04.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.516+0000 7fc3a5a5c640 1 --2- 172.21.15.1:0/2987001168 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc38c06b500 0x7fc38c06d9c0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.516+0000 7fc3a5a5c640 1 --2- 172.21.15.1:0/2987001168 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3a0152820 0x7fc3a0157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.516+0000 7fc3a5a5c640 1 --2- 172.21.15.1:0/2987001168 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3a00c7930 0x7fc3a00c9960 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.516+0000 7fc3a5a5c640 1 --2- 172.21.15.1:0/2987001168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3a00af6f0 0x7fc3a0159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.516+0000 7fc3a5a5c640 1 -- 172.21.15.1:0/2987001168 >> 172.21.15.1:0/2987001168 conn(0x7fc3a0072720 msgr2=0x7fc3a0151a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.516+0000 7fc3a5a5c640 1 -- 172.21.15.1:0/2987001168 shutdown_connections 2024-03-06T07:28:04.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:04.516+0000 7fc3a5a5c640 1 -- 172.21.15.1:0/2987001168 wait complete. 2024-03-06T07:28:04.932 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 172, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:52.179842+0000', 'modified': '2024-03-06T07:27:58.854215+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9954}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9954': {'gid': 9954, 'name': 'c', 'rank': 0, 'incarnation': 168, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.1:6835/1300509547', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 1300509547}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1300509547}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:28:04.932 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9954:up:stopping 2024-03-06T07:28:05.933 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.045+0000 7fc4d6e9a640 1 Processor -- start 2024-03-06T07:28:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.045+0000 7fc4d6e9a640 1 -- start start 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.046+0000 7fc4d6e9a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc4d01505b0 0x7fc4d00c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.046+0000 7fc4d6e9a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d006f5c0 con 0x7fc4d00c1260 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.046+0000 7fc4d6e9a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d0152f80 con 0x7fc4d00c16e0 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.046+0000 7fc4d6e9a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4d0150990 con 0x7fc4d01505b0 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.046+0000 7fc4d4c36640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc4d01505b0 0x7fc4d00c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.046+0000 7fc4cffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc4d00c1260 0x7fc4d01566e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47560/0 (socket says 172.21.15.1:47560) 2024-03-06T07:28:06.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.046+0000 7fc4cffff640 1 -- 172.21.15.1:0/3057027256 learned_addr learned my addr 172.21.15.1:0/3057027256 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:06.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.046+0000 7fc4d4c36640 1 -- 172.21.15.1:0/3057027256 >> v1:172.21.15.98:6789/0 conn(0x7fc4d00c16e0 legacy=0x7fc4d0156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.046+0000 7fc4d4c36640 1 -- 172.21.15.1:0/3057027256 >> v1:172.21.15.1:6789/0 conn(0x7fc4d00c1260 legacy=0x7fc4d01566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.046+0000 7fc4d4c36640 1 -- 172.21.15.1:0/3057027256 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d0153240 con 0x7fc4d01505b0 2024-03-06T07:28:06.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.046+0000 7fc4cf7fe640 1 -- 172.21.15.1:0/3057027256 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790261600 0 0) 0x7fc4d006f5c0 con 0x7fc4d00c1260 2024-03-06T07:28:06.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.047+0000 7fc4d4c36640 1 --2- 172.21.15.1:0/3057027256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d01505b0 0x7fc4d00c4760 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40097a0 tx=0x7fc4c402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbc95ce812c346a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.047+0000 7fc4cf7fe640 1 -- 172.21.15.1:0/3057027256 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4038070 con 0x7fc4d01505b0 2024-03-06T07:28:06.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.047+0000 7fc4cf7fe640 1 -- 172.21.15.1:0/3057027256 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c4031700 con 0x7fc4d01505b0 2024-03-06T07:28:06.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.047+0000 7fc4cf7fe640 1 -- 172.21.15.1:0/3057027256 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4031ad0 con 0x7fc4d01505b0 2024-03-06T07:28:06.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.048+0000 7fc4d6e9a640 1 -- 172.21.15.1:0/3057027256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d01505b0 msgr2=0x7fc4d00c4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.048+0000 7fc4d6e9a640 1 --2- 172.21.15.1:0/3057027256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d01505b0 0x7fc4d00c4760 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40097a0 tx=0x7fc4c402b7a0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.048+0000 7fc4d6e9a640 1 -- 172.21.15.1:0/3057027256 shutdown_connections 2024-03-06T07:28:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.048+0000 7fc4d6e9a640 1 --2- 172.21.15.1:0/3057027256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d01505b0 0x7fc4d00c4760 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.048+0000 7fc4d6e9a640 1 -- 172.21.15.1:0/3057027256 >> 172.21.15.1:0/3057027256 conn(0x7fc4d00c6860 msgr2=0x7fc4d00c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.048+0000 7fc4d6e9a640 1 -- 172.21.15.1:0/3057027256 shutdown_connections 2024-03-06T07:28:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.048+0000 7fc4d6e9a640 1 -- 172.21.15.1:0/3057027256 wait complete. 2024-03-06T07:28:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.049+0000 7fc4d6e9a640 1 Processor -- start 2024-03-06T07:28:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.049+0000 7fc4d6e9a640 1 -- start start 2024-03-06T07:28:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4d6e9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d00c1260 0x7fc4d0149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4d6e9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4d00c16e0 0x7fc4d00721e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4d6e9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4d01505b0 0x7fc4d0072720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4d6e9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d0153ae0 con 0x7fc4d00c1260 2024-03-06T07:28:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4d6e9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d0153800 con 0x7fc4d00c16e0 2024-03-06T07:28:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4d6e9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d00c4ca0 con 0x7fc4d01505b0 2024-03-06T07:28:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4d4c36640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d00c1260 0x7fc4d0149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4d4c36640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d00c1260 0x7fc4d0149fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40322/0 (socket says 172.21.15.1:40322) 2024-03-06T07:28:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4d4c36640 1 -- 172.21.15.1:0/2995306960 learned_addr learned my addr 172.21.15.1:0/2995306960 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4cffff640 1 --2- 172.21.15.1:0/2995306960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4d00c16e0 0x7fc4d00721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4d5437640 1 --2- 172.21.15.1:0/2995306960 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4d01505b0 0x7fc4d0072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4d4c36640 1 -- 172.21.15.1:0/2995306960 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4d01505b0 msgr2=0x7fc4d0072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4d4c36640 1 --2- 172.21.15.1:0/2995306960 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4d01505b0 0x7fc4d0072720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4d4c36640 1 -- 172.21.15.1:0/2995306960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4d00c16e0 msgr2=0x7fc4d00721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4d4c36640 1 --2- 172.21.15.1:0/2995306960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4d00c16e0 0x7fc4d00721e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4d4c36640 1 -- 172.21.15.1:0/2995306960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d00c0840 con 0x7fc4d00c1260 2024-03-06T07:28:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4cffff640 1 --2- 172.21.15.1:0/2995306960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4d00c16e0 0x7fc4d00721e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4d4c36640 1 --2- 172.21.15.1:0/2995306960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d00c1260 0x7fc4d0149fe0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40097a0 tx=0x7fc4c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4cd7fa640 1 -- 172.21.15.1:0/2995306960 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4043070 con 0x7fc4d00c1260 2024-03-06T07:28:06.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.050+0000 7fc4cd7fa640 1 -- 172.21.15.1:0/2995306960 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c4033a30 con 0x7fc4d00c1260 2024-03-06T07:28:06.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.051+0000 7fc4d6e9a640 1 -- 172.21.15.1:0/2995306960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d01ebb70 con 0x7fc4d00c1260 2024-03-06T07:28:06.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.051+0000 7fc4cd7fa640 1 -- 172.21.15.1:0/2995306960 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4038040 con 0x7fc4d00c1260 2024-03-06T07:28:06.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.051+0000 7fc4d6e9a640 1 -- 172.21.15.1:0/2995306960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d014bf20 con 0x7fc4d00c1260 2024-03-06T07:28:06.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.051+0000 7fc4d6e9a640 1 -- 172.21.15.1:0/2995306960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d0067fd0 con 0x7fc4d00c1260 2024-03-06T07:28:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.052+0000 7fc4cd7fa640 1 -- 172.21.15.1:0/2995306960 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc4c4046070 con 0x7fc4d00c1260 2024-03-06T07:28:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.052+0000 7fc4cd7fa640 1 --2- 172.21.15.1:0/2995306960 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc4b406b4d0 0x7fc4b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.053+0000 7fc4cd7fa640 1 -- 172.21.15.1:0/2995306960 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc4c40ac390 con 0x7fc4d00c1260 2024-03-06T07:28:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.053+0000 7fc4cffff640 1 --2- 172.21.15.1:0/2995306960 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc4b406b4d0 0x7fc4b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.053+0000 7fc4cffff640 1 --2- 172.21.15.1:0/2995306960 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc4b406b4d0 0x7fc4b406d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fc4d014f420 tx=0x7fc4bc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.057+0000 7fc4cd7fa640 1 -- 172.21.15.1:0/2995306960 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc4c4030070 con 0x7fc4d00c1260 2024-03-06T07:28:06.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.279+0000 7fc4d6e9a640 1 -- 172.21.15.1:0/2995306960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4d0152010 con 0x7fc4d00c1260 2024-03-06T07:28:06.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.279+0000 7fc4cd7fa640 1 -- 172.21.15.1:0/2995306960 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4976 (secure 0 0 0) 0x7fc4c407b230 con 0x7fc4d00c1260 2024-03-06T07:28:06.281 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:06.281 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":172,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":9951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:27:58.854215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9954":{"gid":9954,"name":"c","rank":0,"incarnation":168,"state":"up:stopping","state_seq":3,"addr":"172.21.15.1:6835/1300509547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1300509547},{"type":"v1","addr":"172.21.15.1:6835","nonce":1300509547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:28:06.281 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 172 2024-03-06T07:28:06.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.281+0000 7fc4d6e9a640 1 -- 172.21.15.1:0/2995306960 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc4b406b4d0 msgr2=0x7fc4b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.281+0000 7fc4d6e9a640 1 --2- 172.21.15.1:0/2995306960 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc4b406b4d0 0x7fc4b406d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fc4d014f420 tx=0x7fc4bc009040 comp rx=0 tx=0).stop 2024-03-06T07:28:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.281+0000 7fc4d6e9a640 1 -- 172.21.15.1:0/2995306960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d00c1260 msgr2=0x7fc4d0149fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.281+0000 7fc4d6e9a640 1 --2- 172.21.15.1:0/2995306960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d00c1260 0x7fc4d0149fe0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40097a0 tx=0x7fc4c40046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.281+0000 7fc4d6e9a640 1 -- 172.21.15.1:0/2995306960 shutdown_connections 2024-03-06T07:28:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.281+0000 7fc4d6e9a640 1 --2- 172.21.15.1:0/2995306960 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc4b406b4d0 0x7fc4b406d990 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.281+0000 7fc4d6e9a640 1 --2- 172.21.15.1:0/2995306960 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4d01505b0 0x7fc4d0072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.281+0000 7fc4d6e9a640 1 --2- 172.21.15.1:0/2995306960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4d00c16e0 0x7fc4d00721e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.281+0000 7fc4d6e9a640 1 --2- 172.21.15.1:0/2995306960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d00c1260 0x7fc4d0149fe0 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.281+0000 7fc4d6e9a640 1 -- 172.21.15.1:0/2995306960 >> 172.21.15.1:0/2995306960 conn(0x7fc4d00c6860 msgr2=0x7fc4d014f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.281+0000 7fc4d6e9a640 1 -- 172.21.15.1:0/2995306960 shutdown_connections 2024-03-06T07:28:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:06.281+0000 7fc4d6e9a640 1 -- 172.21.15.1:0/2995306960 wait complete. 2024-03-06T07:28:06.696 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 172, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:52.179842+0000', 'modified': '2024-03-06T07:27:58.854215+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9954}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9954': {'gid': 9954, 'name': 'c', 'rank': 0, 'incarnation': 168, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.1:6835/1300509547', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 1300509547}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1300509547}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:28:06.697 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9954:up:stopping 2024-03-06T07:28:07.698 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:07.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.809+0000 7f19f85a0640 1 Processor -- start 2024-03-06T07:28:07.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.810+0000 7f19f85a0640 1 -- start start 2024-03-06T07:28:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.810+0000 7f19f85a0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f19f00c1390 0x7f19f00c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.810+0000 7f19f85a0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19f01552e0 0x7f19f0155700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.811+0000 7f19f85a0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f01588e0 con 0x7f19f0152730 2024-03-06T07:28:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.811+0000 7f19f85a0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f00c1d60 con 0x7f19f00c1390 2024-03-06T07:28:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.811+0000 7f19f85a0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f0158bc0 con 0x7f19f01552e0 2024-03-06T07:28:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.811+0000 7f19f5b3b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f19f00c1390 0x7f19f00c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.811+0000 7f19f633c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f19f0152730 0x7f19f00c6a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47568/0 (socket says 172.21.15.1:47568) 2024-03-06T07:28:07.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.811+0000 7f19f633c640 1 -- 172.21.15.1:0/4161077663 learned_addr learned my addr 172.21.15.1:0/4161077663 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:07.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.811+0000 7f19f6b3d640 1 --2- 172.21.15.1:0/4161077663 >> v2:172.21.15.98:3300/0 conn(0x7f19f01552e0 0x7f19f0155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.811+0000 7f19f5b3b640 1 -- 172.21.15.1:0/4161077663 >> v2:172.21.15.98:3300/0 conn(0x7f19f01552e0 msgr2=0x7f19f0155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.811+0000 7f19f5b3b640 1 --2- 172.21.15.1:0/4161077663 >> v2:172.21.15.98:3300/0 conn(0x7f19f01552e0 0x7f19f0155700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.811+0000 7f19f5b3b640 1 -- 172.21.15.1:0/4161077663 >> v1:172.21.15.1:6789/0 conn(0x7f19f0152730 legacy=0x7f19f00c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.811+0000 7f19f533a640 1 -- 172.21.15.1:0/4161077663 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542360458 0 0) 0x7f19f01588e0 con 0x7f19f0152730 2024-03-06T07:28:07.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.811+0000 7f19f5b3b640 1 -- 172.21.15.1:0/4161077663 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f0158ea0 con 0x7f19f00c1390 2024-03-06T07:28:07.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.811+0000 7f19f6b3d640 1 --2- 172.21.15.1:0/4161077663 >> v2:172.21.15.98:3300/0 conn(0x7f19f01552e0 0x7f19f0155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:07.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.812+0000 7f19f5b3b640 1 --2- 172.21.15.1:0/4161077663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19f00c1390 0x7f19f00c1820 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f19f008c2e0 tx=0x7f19e002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906148e2846d589d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.812+0000 7f19f533a640 1 -- 172.21.15.1:0/4161077663 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0038070 con 0x7f19f00c1390 2024-03-06T07:28:07.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.812+0000 7f19f533a640 1 -- 172.21.15.1:0/4161077663 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e0031710 con 0x7f19f00c1390 2024-03-06T07:28:07.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.812+0000 7f19f533a640 1 -- 172.21.15.1:0/4161077663 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0031ae0 con 0x7f19f00c1390 2024-03-06T07:28:07.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.812+0000 7f19f85a0640 1 -- 172.21.15.1:0/4161077663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19f00c1390 msgr2=0x7f19f00c1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.813+0000 7f19f85a0640 1 --2- 172.21.15.1:0/4161077663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19f00c1390 0x7f19f00c1820 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f19f008c2e0 tx=0x7f19e002b7b0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.813+0000 7f19f85a0640 1 -- 172.21.15.1:0/4161077663 shutdown_connections 2024-03-06T07:28:07.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.813+0000 7f19f85a0640 1 --2- 172.21.15.1:0/4161077663 >> v2:172.21.15.98:3300/0 conn(0x7f19f01552e0 0x7f19f0155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.813+0000 7f19f85a0640 1 --2- 172.21.15.1:0/4161077663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19f00c1390 0x7f19f00c1820 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.813+0000 7f19f85a0640 1 -- 172.21.15.1:0/4161077663 >> 172.21.15.1:0/4161077663 conn(0x7f19f0066d00 msgr2=0x7f19f0067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.813+0000 7f19f85a0640 1 -- 172.21.15.1:0/4161077663 shutdown_connections 2024-03-06T07:28:07.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.813+0000 7f19f85a0640 1 -- 172.21.15.1:0/4161077663 wait complete. 2024-03-06T07:28:07.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.814+0000 7f19f85a0640 1 Processor -- start 2024-03-06T07:28:07.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.814+0000 7f19f85a0640 1 -- start start 2024-03-06T07:28:07.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.814+0000 7f19f85a0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19f00c1390 0x7f19f00ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.814+0000 7f19f85a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f0152730 0x7f19f00c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f85a0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f01552e0 0x7f19f014d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f85a0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f0152b10 con 0x7f19f00c1390 2024-03-06T07:28:07.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f85a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f0159460 con 0x7f19f0152730 2024-03-06T07:28:07.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f85a0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f00c71a0 con 0x7f19f01552e0 2024-03-06T07:28:07.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f633c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19f00c1390 0x7f19f00ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f633c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19f00c1390 0x7f19f00ca1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40328/0 (socket says 172.21.15.1:40328) 2024-03-06T07:28:07.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f633c640 1 -- 172.21.15.1:0/2239667224 learned_addr learned my addr 172.21.15.1:0/2239667224 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:07.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f5b3b640 1 --2- 172.21.15.1:0/2239667224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f0152730 0x7f19f00c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f6b3d640 1 --2- 172.21.15.1:0/2239667224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f01552e0 0x7f19f014d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f633c640 1 -- 172.21.15.1:0/2239667224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f01552e0 msgr2=0x7f19f014d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f633c640 1 --2- 172.21.15.1:0/2239667224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f01552e0 0x7f19f014d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f633c640 1 -- 172.21.15.1:0/2239667224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f0152730 msgr2=0x7f19f00c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f633c640 1 --2- 172.21.15.1:0/2239667224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f0152730 0x7f19f00c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f633c640 1 -- 172.21.15.1:0/2239667224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f00cacb0 con 0x7f19f00c1390 2024-03-06T07:28:07.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f6b3d640 1 --2- 172.21.15.1:0/2239667224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f01552e0 0x7f19f014d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:07.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f633c640 1 --2- 172.21.15.1:0/2239667224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19f00c1390 0x7f19f00ca1f0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00c430 tx=0x7f19ec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19e6ffd640 1 -- 172.21.15.1:0/2239667224 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec0074e0 con 0x7f19f00c1390 2024-03-06T07:28:07.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19e6ffd640 1 -- 172.21.15.1:0/2239667224 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec0047e0 con 0x7f19f00c1390 2024-03-06T07:28:07.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f85a0640 1 -- 172.21.15.1:0/2239667224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f01ee1a0 con 0x7f19f00c1390 2024-03-06T07:28:07.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19e6ffd640 1 -- 172.21.15.1:0/2239667224 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec004bd0 con 0x7f19f00c1390 2024-03-06T07:28:07.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.815+0000 7f19f85a0640 1 -- 172.21.15.1:0/2239667224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f01ee760 con 0x7f19f00c1390 2024-03-06T07:28:07.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.816+0000 7f19f85a0640 1 -- 172.21.15.1:0/2239667224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f0067fd0 con 0x7f19f00c1390 2024-03-06T07:28:07.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.817+0000 7f19e6ffd640 1 -- 172.21.15.1:0/2239667224 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19ec011070 con 0x7f19f00c1390 2024-03-06T07:28:07.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.817+0000 7f19e6ffd640 1 --2- 172.21.15.1:0/2239667224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19d406b4d0 0x7f19d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.817+0000 7f19e6ffd640 1 -- 172.21.15.1:0/2239667224 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f19ec089f50 con 0x7f19f00c1390 2024-03-06T07:28:07.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.818+0000 7f19f5b3b640 1 --2- 172.21.15.1:0/2239667224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19d406b4d0 0x7f19d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.818+0000 7f19f5b3b640 1 --2- 172.21.15.1:0/2239667224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19d406b4d0 0x7f19d406d990 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f19e0035000 tx=0x7f19e0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:07.822+0000 7f19e6ffd640 1 -- 172.21.15.1:0/2239667224 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19f00cacb0 con 0x7f19f00c1390 2024-03-06T07:28:08.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:08.043+0000 7f19f85a0640 1 -- 172.21.15.1:0/2239667224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19f0067600 con 0x7f19f00c1390 2024-03-06T07:28:08.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:08.044+0000 7f19e6ffd640 1 -- 172.21.15.1:0/2239667224 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4976 (secure 0 0 0) 0x7f19ec058c30 con 0x7f19f00c1390 2024-03-06T07:28:08.047 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:08.047 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":172,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":9951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:27:58.854215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9954":{"gid":9954,"name":"c","rank":0,"incarnation":168,"state":"up:stopping","state_seq":3,"addr":"172.21.15.1:6835/1300509547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1300509547},{"type":"v1","addr":"172.21.15.1:6835","nonce":1300509547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:28:08.047 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 172 2024-03-06T07:28:08.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:08.046+0000 7f19f85a0640 1 -- 172.21.15.1:0/2239667224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19d406b4d0 msgr2=0x7f19d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:08.046+0000 7f19f85a0640 1 --2- 172.21.15.1:0/2239667224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19d406b4d0 0x7f19d406d990 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f19e0035000 tx=0x7f19e0008030 comp rx=0 tx=0).stop 2024-03-06T07:28:08.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:08.046+0000 7f19f85a0640 1 -- 172.21.15.1:0/2239667224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19f00c1390 msgr2=0x7f19f00ca1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:08.046+0000 7f19f85a0640 1 --2- 172.21.15.1:0/2239667224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19f00c1390 0x7f19f00ca1f0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00c430 tx=0x7f19ec00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:08.046+0000 7f19f85a0640 1 -- 172.21.15.1:0/2239667224 shutdown_connections 2024-03-06T07:28:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:08.046+0000 7f19f85a0640 1 --2- 172.21.15.1:0/2239667224 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19d406b4d0 0x7f19d406d990 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:08.046+0000 7f19f85a0640 1 --2- 172.21.15.1:0/2239667224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f01552e0 0x7f19f014d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:08.046+0000 7f19f85a0640 1 --2- 172.21.15.1:0/2239667224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f0152730 0x7f19f00c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:08.046+0000 7f19f85a0640 1 --2- 172.21.15.1:0/2239667224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19f00c1390 0x7f19f00ca1f0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:08.046+0000 7f19f85a0640 1 -- 172.21.15.1:0/2239667224 >> 172.21.15.1:0/2239667224 conn(0x7f19f0066d00 msgr2=0x7f19f00c6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:08.046+0000 7f19f85a0640 1 -- 172.21.15.1:0/2239667224 shutdown_connections 2024-03-06T07:28:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:08.046+0000 7f19f85a0640 1 -- 172.21.15.1:0/2239667224 wait complete. 2024-03-06T07:28:08.462 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 172, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:52.179842+0000', 'modified': '2024-03-06T07:27:58.854215+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9954}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9954': {'gid': 9954, 'name': 'c', 'rank': 0, 'incarnation': 168, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.1:6835/1300509547', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 1300509547}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1300509547}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:28:08.462 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9954:up:stopping 2024-03-06T07:28:09.463 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:09.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.575+0000 7f589233e640 1 Processor -- start 2024-03-06T07:28:09.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.575+0000 7f589233e640 1 -- start start 2024-03-06T07:28:09.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.576+0000 7f589233e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f588c0c6b10 0x7f588c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.576+0000 7f589233e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f588c0bf9d0 0x7f588c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.576+0000 7f589233e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588c0c2fc0 con 0x7f588c152650 2024-03-06T07:28:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.576+0000 7f589233e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f588c155800 con 0x7f588c0c6b10 2024-03-06T07:28:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.576+0000 7f589233e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f588c158da0 con 0x7f588c0bf9d0 2024-03-06T07:28:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.576+0000 7f588b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f588c0c6b10 0x7f588c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.576+0000 7f588b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f588c0c6b10 0x7f588c155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40330/0 (socket says 172.21.15.1:40330) 2024-03-06T07:28:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.576+0000 7f588b7fe640 1 -- 172.21.15.1:0/2472548250 learned_addr learned my addr 172.21.15.1:0/2472548250 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.576+0000 7f58908db640 1 --2- 172.21.15.1:0/2472548250 >> v2:172.21.15.98:3300/0 conn(0x7f588c0bf9d0 0x7f588c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.576+0000 7f588b7fe640 1 -- 172.21.15.1:0/2472548250 >> v2:172.21.15.98:3300/0 conn(0x7f588c0bf9d0 msgr2=0x7f588c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.576+0000 7f588b7fe640 1 --2- 172.21.15.1:0/2472548250 >> v2:172.21.15.98:3300/0 conn(0x7f588c0bf9d0 0x7f588c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.576+0000 7f588b7fe640 1 -- 172.21.15.1:0/2472548250 >> v1:172.21.15.98:6789/0 conn(0x7f588c152650 legacy=0x7f588c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.576+0000 7f588b7fe640 1 -- 172.21.15.1:0/2472548250 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c0c6690 con 0x7f588c0c6b10 2024-03-06T07:28:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.576+0000 7f58908db640 1 --2- 172.21.15.1:0/2472548250 >> v2:172.21.15.98:3300/0 conn(0x7f588c0bf9d0 0x7f588c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:09.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.577+0000 7f588b7fe640 1 --2- 172.21.15.1:0/2472548250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f588c0c6b10 0x7f588c155c90 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f588c1597b0 tx=0x7f587c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=995fe615e8782d3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.577+0000 7f588affd640 1 -- 172.21.15.1:0/2472548250 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c02e070 con 0x7f588c0c6b10 2024-03-06T07:28:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.577+0000 7f588affd640 1 -- 172.21.15.1:0/2472548250 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f587c037070 con 0x7f588c0c6b10 2024-03-06T07:28:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.577+0000 7f588affd640 1 -- 172.21.15.1:0/2472548250 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c031a00 con 0x7f588c0c6b10 2024-03-06T07:28:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.578+0000 7f589233e640 1 -- 172.21.15.1:0/2472548250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f588c0c6b10 msgr2=0x7f588c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.578+0000 7f589233e640 1 --2- 172.21.15.1:0/2472548250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f588c0c6b10 0x7f588c155c90 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f588c1597b0 tx=0x7f587c02b880 comp rx=0 tx=0).stop 2024-03-06T07:28:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.578+0000 7f589233e640 1 -- 172.21.15.1:0/2472548250 shutdown_connections 2024-03-06T07:28:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.578+0000 7f589233e640 1 --2- 172.21.15.1:0/2472548250 >> v2:172.21.15.98:3300/0 conn(0x7f588c0bf9d0 0x7f588c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.578+0000 7f589233e640 1 --2- 172.21.15.1:0/2472548250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f588c0c6b10 0x7f588c155c90 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.578+0000 7f589233e640 1 -- 172.21.15.1:0/2472548250 >> 172.21.15.1:0/2472548250 conn(0x7f588c066ee0 msgr2=0x7f588c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.578+0000 7f589233e640 1 -- 172.21.15.1:0/2472548250 shutdown_connections 2024-03-06T07:28:09.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.578+0000 7f589233e640 1 -- 172.21.15.1:0/2472548250 wait complete. 2024-03-06T07:28:09.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.579+0000 7f589233e640 1 Processor -- start 2024-03-06T07:28:09.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.579+0000 7f589233e640 1 -- start start 2024-03-06T07:28:09.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.580+0000 7f589233e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f588c0bf9d0 0x7f588c14ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.580+0000 7f589233e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f588c152650 0x7f588c158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.580+0000 7f589233e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f588c14fce0 0x7f588c14b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.580+0000 7f589233e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c1551b0 con 0x7f588c0bf9d0 2024-03-06T07:28:09.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.580+0000 7f589233e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c0c6fa0 con 0x7f588c14fce0 2024-03-06T07:28:09.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.580+0000 7f589233e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588c155490 con 0x7f588c152650 2024-03-06T07:28:09.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.580+0000 7f588bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f588c0bf9d0 0x7f588c14ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.580+0000 7f588bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f588c0bf9d0 0x7f588c14ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40336/0 (socket says 172.21.15.1:40336) 2024-03-06T07:28:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.580+0000 7f588bfff640 1 -- 172.21.15.1:0/3847051337 learned_addr learned my addr 172.21.15.1:0/3847051337 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.580+0000 7f588b7fe640 1 --2- 172.21.15.1:0/3847051337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f588c152650 0x7f588c158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.580+0000 7f58908db640 1 --2- 172.21.15.1:0/3847051337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f588c14fce0 0x7f588c14b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.580+0000 7f588bfff640 1 -- 172.21.15.1:0/3847051337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f588c152650 msgr2=0x7f588c158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.580+0000 7f588bfff640 1 --2- 172.21.15.1:0/3847051337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f588c152650 0x7f588c158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.580+0000 7f588bfff640 1 -- 172.21.15.1:0/3847051337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f588c14fce0 msgr2=0x7f588c14b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.580+0000 7f588bfff640 1 --2- 172.21.15.1:0/3847051337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f588c14fce0 0x7f588c14b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.580+0000 7f588bfff640 1 -- 172.21.15.1:0/3847051337 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c14d150 con 0x7f588c0bf9d0 2024-03-06T07:28:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.581+0000 7f588bfff640 1 --2- 172.21.15.1:0/3847051337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f588c0bf9d0 0x7f588c14ddf0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f588000c460 tx=0x7f588000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.581+0000 7f5888ff9640 1 -- 172.21.15.1:0/3847051337 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5880016020 con 0x7f588c0bf9d0 2024-03-06T07:28:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.581+0000 7f5888ff9640 1 -- 172.21.15.1:0/3847051337 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5880012870 con 0x7f588c0bf9d0 2024-03-06T07:28:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.581+0000 7f589233e640 1 -- 172.21.15.1:0/3847051337 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588c14cb90 con 0x7f588c0bf9d0 2024-03-06T07:28:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.581+0000 7f5888ff9640 1 -- 172.21.15.1:0/3847051337 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5880012c60 con 0x7f588c0bf9d0 2024-03-06T07:28:09.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.581+0000 7f589233e640 1 -- 172.21.15.1:0/3847051337 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588c156530 con 0x7f588c0bf9d0 2024-03-06T07:28:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.582+0000 7f589233e640 1 -- 172.21.15.1:0/3847051337 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f588c0b0d50 con 0x7f588c0bf9d0 2024-03-06T07:28:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.582+0000 7f5888ff9640 1 -- 172.21.15.1:0/3847051337 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5880019020 con 0x7f588c0bf9d0 2024-03-06T07:28:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.583+0000 7f5888ff9640 1 --2- 172.21.15.1:0/3847051337 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f587806b4d0 0x7f587806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.583+0000 7f5888ff9640 1 -- 172.21.15.1:0/3847051337 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5880089d80 con 0x7f588c0bf9d0 2024-03-06T07:28:09.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.583+0000 7f588b7fe640 1 --2- 172.21.15.1:0/3847051337 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f587806b4d0 0x7f587806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.583+0000 7f588b7fe640 1 --2- 172.21.15.1:0/3847051337 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f587806b4d0 0x7f587806d990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f587c036000 tx=0x7f587c03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.587+0000 7f5888ff9640 1 -- 172.21.15.1:0/3847051337 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f588c0b0d50 con 0x7f588c0bf9d0 2024-03-06T07:28:09.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.807+0000 7f589233e640 1 -- 172.21.15.1:0/3847051337 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f588c067fd0 con 0x7f588c0bf9d0 2024-03-06T07:28:09.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.807+0000 7f5888ff9640 1 -- 172.21.15.1:0/3847051337 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4976 (secure 0 0 0) 0x7f5880020070 con 0x7f588c0bf9d0 2024-03-06T07:28:09.809 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:09.809 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":172,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":9951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:27:58.854215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9954":{"gid":9954,"name":"c","rank":0,"incarnation":168,"state":"up:stopping","state_seq":3,"addr":"172.21.15.1:6835/1300509547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1300509547},{"type":"v1","addr":"172.21.15.1:6835","nonce":1300509547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:28:09.810 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 172 2024-03-06T07:28:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.810+0000 7f589233e640 1 -- 172.21.15.1:0/3847051337 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f587806b4d0 msgr2=0x7f587806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.810+0000 7f589233e640 1 --2- 172.21.15.1:0/3847051337 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f587806b4d0 0x7f587806d990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f587c036000 tx=0x7f587c03d040 comp rx=0 tx=0).stop 2024-03-06T07:28:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.810+0000 7f589233e640 1 -- 172.21.15.1:0/3847051337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f588c0bf9d0 msgr2=0x7f588c14ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.810+0000 7f589233e640 1 --2- 172.21.15.1:0/3847051337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f588c0bf9d0 0x7f588c14ddf0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f588000c460 tx=0x7f588000c920 comp rx=0 tx=0).stop 2024-03-06T07:28:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.810+0000 7f589233e640 1 -- 172.21.15.1:0/3847051337 shutdown_connections 2024-03-06T07:28:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.810+0000 7f589233e640 1 --2- 172.21.15.1:0/3847051337 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f587806b4d0 0x7f587806d990 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.810+0000 7f589233e640 1 --2- 172.21.15.1:0/3847051337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f588c14fce0 0x7f588c14b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.810+0000 7f589233e640 1 --2- 172.21.15.1:0/3847051337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f588c152650 0x7f588c158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.810+0000 7f589233e640 1 --2- 172.21.15.1:0/3847051337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f588c0bf9d0 0x7f588c14ddf0 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.810+0000 7f589233e640 1 -- 172.21.15.1:0/3847051337 >> 172.21.15.1:0/3847051337 conn(0x7f588c066ee0 msgr2=0x7f588c0c03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.810+0000 7f589233e640 1 -- 172.21.15.1:0/3847051337 shutdown_connections 2024-03-06T07:28:09.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:09.810+0000 7f589233e640 1 -- 172.21.15.1:0/3847051337 wait complete. 2024-03-06T07:28:10.225 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 172, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:52.179842+0000', 'modified': '2024-03-06T07:27:58.854215+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9954}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9954': {'gid': 9954, 'name': 'c', 'rank': 0, 'incarnation': 168, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.1:6835/1300509547', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 1300509547}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1300509547}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:28:10.225 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9954:up:stopping 2024-03-06T07:28:11.227 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:11.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.338+0000 7f7775a88640 1 Processor -- start 2024-03-06T07:28:11.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.338+0000 7f7775a88640 1 -- start start 2024-03-06T07:28:11.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.339+0000 7f7775a88640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f777014cdc0 0x7f7770152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.339+0000 7f7775a88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f777006b510 0x7f777006b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.339+0000 7f7775a88640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77701569e0 con 0x7f7770156500 2024-03-06T07:28:11.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.339+0000 7f7775a88640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77700c2dd0 con 0x7f777014cdc0 2024-03-06T07:28:11.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.339+0000 7f7775a88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7770156cc0 con 0x7f777006b510 2024-03-06T07:28:11.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.339+0000 7f776f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f777014cdc0 0x7f7770152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.339+0000 7f776f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f777014cdc0 0x7f7770152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40348/0 (socket says 172.21.15.1:40348) 2024-03-06T07:28:11.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.339+0000 7f776f7fe640 1 -- 172.21.15.1:0/1128261089 learned_addr learned my addr 172.21.15.1:0/1128261089 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:11.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.339+0000 7f776ffff640 1 --2- 172.21.15.1:0/1128261089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f777006b510 0x7f777006b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.339+0000 7f776f7fe640 1 -- 172.21.15.1:0/1128261089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f777006b510 msgr2=0x7f777006b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:11.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.339+0000 7f776f7fe640 1 --2- 172.21.15.1:0/1128261089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f777006b510 0x7f777006b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.339+0000 7f776f7fe640 1 -- 172.21.15.1:0/1128261089 >> v1:172.21.15.98:6789/0 conn(0x7f7770156500 legacy=0x7f77700c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.339+0000 7f776f7fe640 1 -- 172.21.15.1:0/1128261089 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7770156fa0 con 0x7f777014cdc0 2024-03-06T07:28:11.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.339+0000 7f776ffff640 1 --2- 172.21.15.1:0/1128261089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f777006b510 0x7f777006b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:11.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.340+0000 7f776f7fe640 1 --2- 172.21.15.1:0/1128261089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f777014cdc0 0x7f7770152f10 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f7760009870 tx=0x7f776002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f747d60f3d5a56d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.340+0000 7f776e7fc640 1 -- 172.21.15.1:0/1128261089 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7760034070 con 0x7f777014cdc0 2024-03-06T07:28:11.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.340+0000 7f776e7fc640 1 -- 172.21.15.1:0/1128261089 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7760030840 con 0x7f777014cdc0 2024-03-06T07:28:11.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.340+0000 7f776e7fc640 1 -- 172.21.15.1:0/1128261089 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7760030c10 con 0x7f777014cdc0 2024-03-06T07:28:11.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.341+0000 7f7775a88640 1 -- 172.21.15.1:0/1128261089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f777014cdc0 msgr2=0x7f7770152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.341+0000 7f7775a88640 1 --2- 172.21.15.1:0/1128261089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f777014cdc0 0x7f7770152f10 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f7760009870 tx=0x7f776002b980 comp rx=0 tx=0).stop 2024-03-06T07:28:11.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.341+0000 7f7775a88640 1 -- 172.21.15.1:0/1128261089 shutdown_connections 2024-03-06T07:28:11.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.341+0000 7f7775a88640 1 --2- 172.21.15.1:0/1128261089 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f777006b510 0x7f777006b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.341+0000 7f7775a88640 1 --2- 172.21.15.1:0/1128261089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f777014cdc0 0x7f7770152f10 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.341+0000 7f7775a88640 1 -- 172.21.15.1:0/1128261089 >> 172.21.15.1:0/1128261089 conn(0x7f77700a9700 msgr2=0x7f77700a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.341+0000 7f7775a88640 1 -- 172.21.15.1:0/1128261089 shutdown_connections 2024-03-06T07:28:11.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.341+0000 7f7775a88640 1 -- 172.21.15.1:0/1128261089 wait complete. 2024-03-06T07:28:11.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.342+0000 7f7775a88640 1 Processor -- start 2024-03-06T07:28:11.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.342+0000 7f7775a88640 1 -- start start 2024-03-06T07:28:11.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.343+0000 7f7775a88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f777006b510 0x7f77700ac9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.343+0000 7f7775a88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f777014cdc0 0x7f77700acf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.343+0000 7f7775a88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7770156500 0x7f77701ec4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.343+0000 7f7775a88640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777006bea0 con 0x7f777006b510 2024-03-06T07:28:11.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.343+0000 7f7775a88640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77700c5a70 con 0x7f777014cdc0 2024-03-06T07:28:11.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.343+0000 7f7775a88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77700c65e0 con 0x7f7770156500 2024-03-06T07:28:11.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.343+0000 7f776f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f777006b510 0x7f77700ac9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.343+0000 7f776f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f777006b510 0x7f77700ac9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40350/0 (socket says 172.21.15.1:40350) 2024-03-06T07:28:11.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.343+0000 7f776f7fe640 1 -- 172.21.15.1:0/1875861682 learned_addr learned my addr 172.21.15.1:0/1875861682 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:11.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.343+0000 7f776ffff640 1 --2- 172.21.15.1:0/1875861682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7770156500 0x7f77701ec4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.343+0000 7f776effd640 1 --2- 172.21.15.1:0/1875861682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f777014cdc0 0x7f77700acf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.343+0000 7f776f7fe640 1 -- 172.21.15.1:0/1875861682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7770156500 msgr2=0x7f77701ec4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.343+0000 7f776f7fe640 1 --2- 172.21.15.1:0/1875861682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7770156500 0x7f77701ec4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.343+0000 7f776f7fe640 1 -- 172.21.15.1:0/1875861682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f777014cdc0 msgr2=0x7f77700acf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.343+0000 7f776f7fe640 1 --2- 172.21.15.1:0/1875861682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f777014cdc0 0x7f77700acf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.343+0000 7f776f7fe640 1 -- 172.21.15.1:0/1875861682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77701ed1c0 con 0x7f777006b510 2024-03-06T07:28:11.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.344+0000 7f776f7fe640 1 --2- 172.21.15.1:0/1875861682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f777006b510 0x7f77700ac9e0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f7760000c00 tx=0x7f7760004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.344+0000 7f7757fff640 1 -- 172.21.15.1:0/1875861682 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7760048070 con 0x7f777006b510 2024-03-06T07:28:11.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.344+0000 7f7757fff640 1 -- 172.21.15.1:0/1875861682 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7760030640 con 0x7f777006b510 2024-03-06T07:28:11.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.344+0000 7f7775a88640 1 -- 172.21.15.1:0/1875861682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77700aae00 con 0x7f777006b510 2024-03-06T07:28:11.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.344+0000 7f7757fff640 1 -- 172.21.15.1:0/1875861682 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7760034040 con 0x7f777006b510 2024-03-06T07:28:11.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.344+0000 7f7775a88640 1 -- 172.21.15.1:0/1875861682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7770150c30 con 0x7f777006b510 2024-03-06T07:28:11.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.345+0000 7f7775a88640 1 -- 172.21.15.1:0/1875861682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7770067fd0 con 0x7f777006b510 2024-03-06T07:28:11.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.345+0000 7f7757fff640 1 -- 172.21.15.1:0/1875861682 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f776004b070 con 0x7f777006b510 2024-03-06T07:28:11.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.346+0000 7f7757fff640 1 --2- 172.21.15.1:0/1875861682 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f775c06b500 0x7f775c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.346+0000 7f7757fff640 1 -- 172.21.15.1:0/1875861682 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f77600ac220 con 0x7f777006b510 2024-03-06T07:28:11.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.346+0000 7f776effd640 1 --2- 172.21.15.1:0/1875861682 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f775c06b500 0x7f775c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.346+0000 7f776effd640 1 --2- 172.21.15.1:0/1875861682 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f775c06b500 0x7f775c06d9c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f777014e530 tx=0x7f7764009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.350+0000 7f7757fff640 1 -- 172.21.15.1:0/1875861682 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f776007af30 con 0x7f777006b510 2024-03-06T07:28:11.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.570+0000 7f7775a88640 1 -- 172.21.15.1:0/1875861682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77700c1c90 con 0x7f777006b510 2024-03-06T07:28:11.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.570+0000 7f7757fff640 1 -- 172.21.15.1:0/1875861682 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4976 (secure 0 0 0) 0x7f7760030810 con 0x7f777006b510 2024-03-06T07:28:11.572 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:11.572 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":172,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":9951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:27:58.854215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9954":{"gid":9954,"name":"c","rank":0,"incarnation":168,"state":"up:stopping","state_seq":3,"addr":"172.21.15.1:6835/1300509547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1300509547},{"type":"v1","addr":"172.21.15.1:6835","nonce":1300509547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:28:11.573 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 172 2024-03-06T07:28:11.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.572+0000 7f7775a88640 1 -- 172.21.15.1:0/1875861682 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f775c06b500 msgr2=0x7f775c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.572+0000 7f7775a88640 1 --2- 172.21.15.1:0/1875861682 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f775c06b500 0x7f775c06d9c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f777014e530 tx=0x7f7764009040 comp rx=0 tx=0).stop 2024-03-06T07:28:11.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.572+0000 7f7775a88640 1 -- 172.21.15.1:0/1875861682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f777006b510 msgr2=0x7f77700ac9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.572+0000 7f7775a88640 1 --2- 172.21.15.1:0/1875861682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f777006b510 0x7f77700ac9e0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f7760000c00 tx=0x7f7760004730 comp rx=0 tx=0).stop 2024-03-06T07:28:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.572+0000 7f7775a88640 1 -- 172.21.15.1:0/1875861682 shutdown_connections 2024-03-06T07:28:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.572+0000 7f7775a88640 1 --2- 172.21.15.1:0/1875861682 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f775c06b500 0x7f775c06d9c0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.572+0000 7f7775a88640 1 --2- 172.21.15.1:0/1875861682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7770156500 0x7f77701ec4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.572+0000 7f7775a88640 1 --2- 172.21.15.1:0/1875861682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f777014cdc0 0x7f77700acf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.572+0000 7f7775a88640 1 --2- 172.21.15.1:0/1875861682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f777006b510 0x7f77700ac9e0 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.572+0000 7f7775a88640 1 -- 172.21.15.1:0/1875861682 >> 172.21.15.1:0/1875861682 conn(0x7f77700a9700 msgr2=0x7f7770154ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.572+0000 7f7775a88640 1 -- 172.21.15.1:0/1875861682 shutdown_connections 2024-03-06T07:28:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:11.572+0000 7f7775a88640 1 -- 172.21.15.1:0/1875861682 wait complete. 2024-03-06T07:28:11.987 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 172, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:52.179842+0000', 'modified': '2024-03-06T07:27:58.854215+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9954}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9954': {'gid': 9954, 'name': 'c', 'rank': 0, 'incarnation': 168, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.1:6835/1300509547', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 1300509547}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1300509547}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:28:11.987 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9954:up:stopping 2024-03-06T07:28:12.913 INFO:tasks.ceph.mds.c.smithi001.stderr: -622> 2024-03-06T07:27:58.113+0000 7f74ade0b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:12.941 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:28:12.988 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:13.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.096+0000 7fd35f53e640 1 Processor -- start 2024-03-06T07:28:13.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.096+0000 7fd35f53e640 1 -- start start 2024-03-06T07:28:13.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.097+0000 7fd35f53e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd3580c2b30 0x7fd3580724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.097+0000 7fd35f53e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd3580a5fa0 0x7fd358072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.097+0000 7fd35f53e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd358151dd0 con 0x7fd358152b70 2024-03-06T07:28:13.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.097+0000 7fd35f53e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3581526e0 con 0x7fd3580a5fa0 2024-03-06T07:28:13.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.097+0000 7fd35f53e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd358156190 con 0x7fd3580c2b30 2024-03-06T07:28:13.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.097+0000 7fd35dadb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd3580a5fa0 0x7fd358072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.097+0000 7fd35dadb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd3580a5fa0 0x7fd358072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40360/0 (socket says 172.21.15.1:40360) 2024-03-06T07:28:13.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.097+0000 7fd35dadb640 1 -- 172.21.15.1:0/955528107 learned_addr learned my addr 172.21.15.1:0/955528107 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:13.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.097+0000 7fd35d2da640 1 --2- 172.21.15.1:0/955528107 >> v2:172.21.15.98:3300/0 conn(0x7fd3580c2b30 0x7fd3580724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.097+0000 7fd35dadb640 1 -- 172.21.15.1:0/955528107 >> v2:172.21.15.98:3300/0 conn(0x7fd3580c2b30 msgr2=0x7fd3580724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.097+0000 7fd35dadb640 1 --2- 172.21.15.1:0/955528107 >> v2:172.21.15.98:3300/0 conn(0x7fd3580c2b30 0x7fd3580724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.097+0000 7fd35dadb640 1 -- 172.21.15.1:0/955528107 >> v1:172.21.15.98:6789/0 conn(0x7fd358152b70 legacy=0x7fd3580c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.097+0000 7fd35dadb640 1 -- 172.21.15.1:0/955528107 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd358152090 con 0x7fd3580a5fa0 2024-03-06T07:28:13.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.098+0000 7fd35dadb640 1 --2- 172.21.15.1:0/955528107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3580a5fa0 0x7fd358072a00 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7fd3580c6570 tx=0x7fd35002e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965f7b73d3c87152 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.098+0000 7fd34ffff640 1 -- 172.21.15.1:0/955528107 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35002ec90 con 0x7fd3580a5fa0 2024-03-06T07:28:13.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.098+0000 7fd34ffff640 1 -- 172.21.15.1:0/955528107 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35002ee30 con 0x7fd3580a5fa0 2024-03-06T07:28:13.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.098+0000 7fd34ffff640 1 -- 172.21.15.1:0/955528107 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3500365f0 con 0x7fd3580a5fa0 2024-03-06T07:28:13.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.099+0000 7fd35f53e640 1 -- 172.21.15.1:0/955528107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3580a5fa0 msgr2=0x7fd358072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.099+0000 7fd35f53e640 1 --2- 172.21.15.1:0/955528107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3580a5fa0 0x7fd358072a00 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7fd3580c6570 tx=0x7fd35002e0f0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.099+0000 7fd35f53e640 1 -- 172.21.15.1:0/955528107 shutdown_connections 2024-03-06T07:28:13.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.099+0000 7fd35f53e640 1 --2- 172.21.15.1:0/955528107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3580a5fa0 0x7fd358072a00 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.099+0000 7fd35f53e640 1 --2- 172.21.15.1:0/955528107 >> v2:172.21.15.98:3300/0 conn(0x7fd3580c2b30 0x7fd3580724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.099+0000 7fd35f53e640 1 -- 172.21.15.1:0/955528107 >> 172.21.15.1:0/955528107 conn(0x7fd3580c6860 msgr2=0x7fd3580c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.099+0000 7fd35f53e640 1 -- 172.21.15.1:0/955528107 shutdown_connections 2024-03-06T07:28:13.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.099+0000 7fd35f53e640 1 -- 172.21.15.1:0/955528107 wait complete. 2024-03-06T07:28:13.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.100+0000 7fd35f53e640 1 Processor -- start 2024-03-06T07:28:13.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.100+0000 7fd35f53e640 1 -- start start 2024-03-06T07:28:13.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35f53e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3580a5fa0 0x7fd3581dc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35f53e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3580c2b30 0x7fd3581e7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35f53e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd358152b70 0x7fd35806e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35f53e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd358155f00 con 0x7fd3580c2b30 2024-03-06T07:28:13.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35f53e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd358155d80 con 0x7fd3580a5fa0 2024-03-06T07:28:13.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35f53e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3580c3a00 con 0x7fd358152b70 2024-03-06T07:28:13.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35cad9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3580c2b30 0x7fd3581e7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35cad9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3580c2b30 0x7fd3581e7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40368/0 (socket says 172.21.15.1:40368) 2024-03-06T07:28:13.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35cad9640 1 -- 172.21.15.1:0/2959487110 learned_addr learned my addr 172.21.15.1:0/2959487110 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:13.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35d2da640 1 --2- 172.21.15.1:0/2959487110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3580a5fa0 0x7fd3581dc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35dadb640 1 --2- 172.21.15.1:0/2959487110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd358152b70 0x7fd35806e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35cad9640 1 -- 172.21.15.1:0/2959487110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd358152b70 msgr2=0x7fd35806e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35cad9640 1 --2- 172.21.15.1:0/2959487110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd358152b70 0x7fd35806e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35cad9640 1 -- 172.21.15.1:0/2959487110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3580a5fa0 msgr2=0x7fd3581dc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35cad9640 1 --2- 172.21.15.1:0/2959487110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3580a5fa0 0x7fd3581dc250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35cad9640 1 -- 172.21.15.1:0/2959487110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3580c09b0 con 0x7fd3580c2b30 2024-03-06T07:28:13.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35cad9640 1 --2- 172.21.15.1:0/2959487110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3580c2b30 0x7fd3581e7850 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fd34800b3f0 tx=0x7fd34800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd34dffb640 1 -- 172.21.15.1:0/2959487110 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348015020 con 0x7fd3580c2b30 2024-03-06T07:28:13.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd34dffb640 1 -- 172.21.15.1:0/2959487110 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd348011830 con 0x7fd3580c2b30 2024-03-06T07:28:13.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd35f53e640 1 -- 172.21.15.1:0/2959487110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3581db770 con 0x7fd3580c2b30 2024-03-06T07:28:13.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.101+0000 7fd34dffb640 1 -- 172.21.15.1:0/2959487110 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348011c20 con 0x7fd3580c2b30 2024-03-06T07:28:13.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.102+0000 7fd35f53e640 1 -- 172.21.15.1:0/2959487110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35806e710 con 0x7fd3580c2b30 2024-03-06T07:28:13.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.103+0000 7fd35f53e640 1 -- 172.21.15.1:0/2959487110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd358067fd0 con 0x7fd3580c2b30 2024-03-06T07:28:13.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.103+0000 7fd34dffb640 1 -- 172.21.15.1:0/2959487110 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd348018020 con 0x7fd3580c2b30 2024-03-06T07:28:13.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.103+0000 7fd34dffb640 1 --2- 172.21.15.1:0/2959487110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd33c06b4d0 0x7fd33c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.103+0000 7fd34dffb640 1 -- 172.21.15.1:0/2959487110 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7fd348089fa0 con 0x7fd3580c2b30 2024-03-06T07:28:13.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.103+0000 7fd35d2da640 1 --2- 172.21.15.1:0/2959487110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd33c06b4d0 0x7fd33c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.104+0000 7fd35d2da640 1 --2- 172.21.15.1:0/2959487110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd33c06b4d0 0x7fd33c06d990 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fd354006fd0 tx=0x7fd354008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.108+0000 7fd34dffb640 1 -- 172.21.15.1:0/2959487110 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd34801f030 con 0x7fd3580c2b30 2024-03-06T07:28:13.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.326+0000 7fd35f53e640 1 -- 172.21.15.1:0/2959487110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd358071790 con 0x7fd3580c2b30 2024-03-06T07:28:13.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.327+0000 7fd34dffb640 1 -- 172.21.15.1:0/2959487110 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+4175 (secure 0 0 0) 0x7fd348058cb0 con 0x7fd3580c2b30 2024-03-06T07:28:13.329 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:13.329 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":173,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":9951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:28:12.745294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:28:13.329 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 173 2024-03-06T07:28:13.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.328+0000 7fd35f53e640 1 -- 172.21.15.1:0/2959487110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd33c06b4d0 msgr2=0x7fd33c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.328+0000 7fd35f53e640 1 --2- 172.21.15.1:0/2959487110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd33c06b4d0 0x7fd33c06d990 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fd354006fd0 tx=0x7fd354008040 comp rx=0 tx=0).stop 2024-03-06T07:28:13.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.328+0000 7fd35f53e640 1 -- 172.21.15.1:0/2959487110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3580c2b30 msgr2=0x7fd3581e7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.328+0000 7fd35f53e640 1 --2- 172.21.15.1:0/2959487110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3580c2b30 0x7fd3581e7850 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fd34800b3f0 tx=0x7fd34800b8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.328+0000 7fd35f53e640 1 -- 172.21.15.1:0/2959487110 shutdown_connections 2024-03-06T07:28:13.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.328+0000 7fd35f53e640 1 --2- 172.21.15.1:0/2959487110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd33c06b4d0 0x7fd33c06d990 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.328+0000 7fd35f53e640 1 --2- 172.21.15.1:0/2959487110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd358152b70 0x7fd35806e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.328+0000 7fd35f53e640 1 --2- 172.21.15.1:0/2959487110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3580c2b30 0x7fd3581e7850 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.328+0000 7fd35f53e640 1 --2- 172.21.15.1:0/2959487110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3580a5fa0 0x7fd3581dc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.328+0000 7fd35f53e640 1 -- 172.21.15.1:0/2959487110 >> 172.21.15.1:0/2959487110 conn(0x7fd3580c6860 msgr2=0x7fd3580c2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.329+0000 7fd35f53e640 1 -- 172.21.15.1:0/2959487110 shutdown_connections 2024-03-06T07:28:13.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.329+0000 7fd35f53e640 1 -- 172.21.15.1:0/2959487110 wait complete. 2024-03-06T07:28:13.745 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 173, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:52.179842+0000', 'modified': '2024-03-06T07:28:12.745294+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [], 'up': {}, 'failed': [], 'damaged': [], 'stopped': [0], 'info': {}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:28:13.745 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/0 2024-03-06T07:28:13.746 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.848+0000 7f39be46f640 1 Processor -- start 2024-03-06T07:28:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.848+0000 7f39be46f640 1 -- start start 2024-03-06T07:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.848+0000 7f39be46f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f39b8072bd0 0x7f39b80c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.848+0000 7f39be46f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39b8152b70 0x7f39b80c3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.848+0000 7f39be46f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b8151dd0 con 0x7f39b80c2b30 2024-03-06T07:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.848+0000 7f39be46f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b81526e0 con 0x7f39b8072bd0 2024-03-06T07:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.848+0000 7f39be46f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b8156190 con 0x7f39b8152b70 2024-03-06T07:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.848+0000 7f39b77fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f39b8072bd0 0x7f39b80c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.849+0000 7f39b77fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f39b8072bd0 0x7f39b80c32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40384/0 (socket says 172.21.15.1:40384) 2024-03-06T07:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.849+0000 7f39b7fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f39b80c2b30 0x7f39b80724c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47584/0 (socket says 172.21.15.1:47584) 2024-03-06T07:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.849+0000 7f39b77fe640 1 -- 172.21.15.1:0/1390918683 learned_addr learned my addr 172.21.15.1:0/1390918683 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.849+0000 7f39bca0c640 1 --2- 172.21.15.1:0/1390918683 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39b8152b70 0x7f39b80c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.849+0000 7f39b77fe640 1 -- 172.21.15.1:0/1390918683 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39b8152b70 msgr2=0x7f39b80c3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.849+0000 7f39b77fe640 1 --2- 172.21.15.1:0/1390918683 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39b8152b70 0x7f39b80c3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.849+0000 7f39b77fe640 1 -- 172.21.15.1:0/1390918683 >> v1:172.21.15.1:6789/0 conn(0x7f39b80c2b30 legacy=0x7f39b80724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.849+0000 7f39b6ffd640 1 -- 172.21.15.1:0/1390918683 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155641574 0 0) 0x7f39b8151dd0 con 0x7f39b80c2b30 2024-03-06T07:28:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.849+0000 7f39b77fe640 1 -- 172.21.15.1:0/1390918683 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b8152090 con 0x7f39b8072bd0 2024-03-06T07:28:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.850+0000 7f39b77fe640 1 --2- 172.21.15.1:0/1390918683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b8072bd0 0x7f39b80c32f0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f39b8152860 tx=0x7f39a802db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9fb66f1e868f4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.850+0000 7f39b6ffd640 1 -- 172.21.15.1:0/1390918683 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a803b070 con 0x7f39b8072bd0 2024-03-06T07:28:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.850+0000 7f39b6ffd640 1 -- 172.21.15.1:0/1390918683 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a802e940 con 0x7f39b8072bd0 2024-03-06T07:28:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.850+0000 7f39b6ffd640 1 -- 172.21.15.1:0/1390918683 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a802ed10 con 0x7f39b8072bd0 2024-03-06T07:28:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.850+0000 7f39be46f640 1 -- 172.21.15.1:0/1390918683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b8072bd0 msgr2=0x7f39b80c32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.850+0000 7f39be46f640 1 --2- 172.21.15.1:0/1390918683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b8072bd0 0x7f39b80c32f0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f39b8152860 tx=0x7f39a802db90 comp rx=0 tx=0).stop 2024-03-06T07:28:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.850+0000 7f39be46f640 1 -- 172.21.15.1:0/1390918683 shutdown_connections 2024-03-06T07:28:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.850+0000 7f39be46f640 1 --2- 172.21.15.1:0/1390918683 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39b8152b70 0x7f39b80c3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.850+0000 7f39be46f640 1 --2- 172.21.15.1:0/1390918683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b8072bd0 0x7f39b80c32f0 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.850+0000 7f39be46f640 1 -- 172.21.15.1:0/1390918683 >> 172.21.15.1:0/1390918683 conn(0x7f39b80c6860 msgr2=0x7f39b80c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.850+0000 7f39be46f640 1 -- 172.21.15.1:0/1390918683 shutdown_connections 2024-03-06T07:28:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.850+0000 7f39be46f640 1 -- 172.21.15.1:0/1390918683 wait complete. 2024-03-06T07:28:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.851+0000 7f39be46f640 1 Processor -- start 2024-03-06T07:28:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.852+0000 7f39be46f640 1 -- start start 2024-03-06T07:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.852+0000 7f39be46f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39b8072bd0 0x7f39b81537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.852+0000 7f39be46f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b80c2b30 0x7f39b81e6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.852+0000 7f39be46f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39b8152b70 0x7f39b81e6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.852+0000 7f39be46f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80c3ef0 con 0x7f39b80c2b30 2024-03-06T07:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.852+0000 7f39be46f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80c3d70 con 0x7f39b8072bd0 2024-03-06T07:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.852+0000 7f39be46f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b8155d80 con 0x7f39b8152b70 2024-03-06T07:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.852+0000 7f39b77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b80c2b30 0x7f39b81e6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.852+0000 7f39b77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b80c2b30 0x7f39b81e6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40394/0 (socket says 172.21.15.1:40394) 2024-03-06T07:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.852+0000 7f39b77fe640 1 -- 172.21.15.1:0/3849229370 learned_addr learned my addr 172.21.15.1:0/3849229370 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.853+0000 7f39bca0c640 1 --2- 172.21.15.1:0/3849229370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39b8152b70 0x7f39b81e6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.853+0000 7f39b7fff640 1 --2- 172.21.15.1:0/3849229370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39b8072bd0 0x7f39b81537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.853+0000 7f39b77fe640 1 -- 172.21.15.1:0/3849229370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39b8152b70 msgr2=0x7f39b81e6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.853+0000 7f39b77fe640 1 --2- 172.21.15.1:0/3849229370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39b8152b70 0x7f39b81e6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.853+0000 7f39b77fe640 1 -- 172.21.15.1:0/3849229370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39b8072bd0 msgr2=0x7f39b81537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.853+0000 7f39b77fe640 1 --2- 172.21.15.1:0/3849229370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39b8072bd0 0x7f39b81537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.853+0000 7f39b77fe640 1 -- 172.21.15.1:0/3849229370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b8149bc0 con 0x7f39b80c2b30 2024-03-06T07:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.853+0000 7f39b7fff640 1 --2- 172.21.15.1:0/3849229370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39b8072bd0 0x7f39b81537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.853+0000 7f39b77fe640 1 --2- 172.21.15.1:0/3849229370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b80c2b30 0x7f39b81e6ab0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f39a8039040 tx=0x7f39a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.853+0000 7f39b4ff9640 1 -- 172.21.15.1:0/3849229370 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a803b040 con 0x7f39b80c2b30 2024-03-06T07:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.853+0000 7f39b4ff9640 1 -- 172.21.15.1:0/3849229370 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a8036b70 con 0x7f39b80c2b30 2024-03-06T07:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.853+0000 7f39be46f640 1 -- 172.21.15.1:0/3849229370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b81543c0 con 0x7f39b80c2b30 2024-03-06T07:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.853+0000 7f39b4ff9640 1 -- 172.21.15.1:0/3849229370 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a8004030 con 0x7f39b80c2b30 2024-03-06T07:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.853+0000 7f39be46f640 1 -- 172.21.15.1:0/3849229370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b806ead0 con 0x7f39b80c2b30 2024-03-06T07:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.854+0000 7f39be46f640 1 -- 172.21.15.1:0/3849229370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b81541b0 con 0x7f39b80c2b30 2024-03-06T07:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.854+0000 7f39b4ff9640 1 -- 172.21.15.1:0/3849229370 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39a80041d0 con 0x7f39b80c2b30 2024-03-06T07:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.855+0000 7f39b4ff9640 1 --2- 172.21.15.1:0/3849229370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f399806b4d0 0x7f399806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.855+0000 7f39b4ff9640 1 -- 172.21.15.1:0/3849229370 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f39a80ae8e0 con 0x7f39b80c2b30 2024-03-06T07:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.855+0000 7f39b7fff640 1 --2- 172.21.15.1:0/3849229370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f399806b4d0 0x7f399806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.856+0000 7f39b7fff640 1 --2- 172.21.15.1:0/3849229370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f399806b4d0 0x7f399806d990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f39a0000f60 tx=0x7f39a0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:13.859+0000 7f39b4ff9640 1 -- 172.21.15.1:0/3849229370 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39a803e350 con 0x7f39b80c2b30 2024-03-06T07:28:14.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.081+0000 7f39be46f640 1 -- 172.21.15.1:0/3849229370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39b8067fd0 con 0x7f39b80c2b30 2024-03-06T07:28:14.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.082+0000 7f39b4ff9640 1 -- 172.21.15.1:0/3849229370 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+4965 (secure 0 0 0) 0x7f39a807d5f0 con 0x7f39b80c2b30 2024-03-06T07:28:14.084 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:14.084 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":174,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":9951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":10161,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1525330395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1525330395},{"type":"v1","addr":"172.21.15.1:6835","nonce":1525330395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":174}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:28:12.745294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:28:14.084 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 174 2024-03-06T07:28:14.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.084+0000 7f39be46f640 1 -- 172.21.15.1:0/3849229370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f399806b4d0 msgr2=0x7f399806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.084+0000 7f39be46f640 1 --2- 172.21.15.1:0/3849229370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f399806b4d0 0x7f399806d990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f39a0000f60 tx=0x7f39a0009040 comp rx=0 tx=0).stop 2024-03-06T07:28:14.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.084+0000 7f39be46f640 1 -- 172.21.15.1:0/3849229370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b80c2b30 msgr2=0x7f39b81e6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.084+0000 7f39be46f640 1 --2- 172.21.15.1:0/3849229370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b80c2b30 0x7f39b81e6ab0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f39a8039040 tx=0x7f39a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.084+0000 7f39be46f640 1 -- 172.21.15.1:0/3849229370 shutdown_connections 2024-03-06T07:28:14.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.084+0000 7f39be46f640 1 --2- 172.21.15.1:0/3849229370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f399806b4d0 0x7f399806d990 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.084+0000 7f39be46f640 1 --2- 172.21.15.1:0/3849229370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f39b8152b70 0x7f39b81e6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.084+0000 7f39be46f640 1 --2- 172.21.15.1:0/3849229370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b80c2b30 0x7f39b81e6ab0 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.084+0000 7f39be46f640 1 --2- 172.21.15.1:0/3849229370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39b8072bd0 0x7f39b81537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.084+0000 7f39be46f640 1 -- 172.21.15.1:0/3849229370 >> 172.21.15.1:0/3849229370 conn(0x7f39b80c6860 msgr2=0x7f39b8150780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.084+0000 7f39be46f640 1 -- 172.21.15.1:0/3849229370 shutdown_connections 2024-03-06T07:28:14.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.084+0000 7f39be46f640 1 -- 172.21.15.1:0/3849229370 wait complete. 2024-03-06T07:28:14.097 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:28:14.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.199+0000 7f21c9e1c640 1 Processor -- start 2024-03-06T07:28:14.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.199+0000 7f21c9e1c640 1 -- start start 2024-03-06T07:28:14.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.200+0000 7f21c9e1c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c4152920 0x7f21c406aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.200+0000 7f21c9e1c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f21c4155220 0x7f21c41556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.200+0000 7f21c9e1c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c4158910 con 0x7f21c406b530 2024-03-06T07:28:14.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.200+0000 7f21c9e1c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c4152d00 con 0x7f21c4155220 2024-03-06T07:28:14.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.200+0000 7f21c9e1c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c41596d0 con 0x7f21c4152920 2024-03-06T07:28:14.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.200+0000 7f21c3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f21c4155220 0x7f21c41556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.200+0000 7f21c3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f21c4155220 0x7f21c41556b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40408/0 (socket says 172.21.15.1:40408) 2024-03-06T07:28:14.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.200+0000 7f21c3fff640 1 -- 172.21.15.1:0/2483793874 learned_addr learned my addr 172.21.15.1:0/2483793874 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:14.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.200+0000 7f21c37fe640 1 --2- 172.21.15.1:0/2483793874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c4152920 0x7f21c406aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.200+0000 7f21c3fff640 1 -- 172.21.15.1:0/2483793874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c4152920 msgr2=0x7f21c406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.200+0000 7f21c3fff640 1 --2- 172.21.15.1:0/2483793874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c4152920 0x7f21c406aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.200+0000 7f21c3fff640 1 -- 172.21.15.1:0/2483793874 >> v1:172.21.15.98:6789/0 conn(0x7f21c406b530 legacy=0x7f21c40c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.200+0000 7f21c3fff640 1 -- 172.21.15.1:0/2483793874 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c4158bf0 con 0x7f21c4155220 2024-03-06T07:28:14.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.201+0000 7f21c3fff640 1 --2- 172.21.15.1:0/2483793874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21c4155220 0x7f21c41556b0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009e50 tx=0x7f21bc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8deb4ba0edb99ca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.201+0000 7f21c27fc640 1 -- 172.21.15.1:0/2483793874 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc02ca70 con 0x7f21c4155220 2024-03-06T07:28:14.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.201+0000 7f21c27fc640 1 -- 172.21.15.1:0/2483793874 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc02cc10 con 0x7f21c4155220 2024-03-06T07:28:14.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.201+0000 7f21c27fc640 1 -- 172.21.15.1:0/2483793874 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0343f0 con 0x7f21c4155220 2024-03-06T07:28:14.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.201+0000 7f21c9e1c640 1 -- 172.21.15.1:0/2483793874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21c4155220 msgr2=0x7f21c41556b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.202+0000 7f21c9e1c640 1 --2- 172.21.15.1:0/2483793874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21c4155220 0x7f21c41556b0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009e50 tx=0x7f21bc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.202+0000 7f21c9e1c640 1 -- 172.21.15.1:0/2483793874 shutdown_connections 2024-03-06T07:28:14.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.202+0000 7f21c9e1c640 1 --2- 172.21.15.1:0/2483793874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21c4155220 0x7f21c41556b0 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.202+0000 7f21c9e1c640 1 --2- 172.21.15.1:0/2483793874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c4152920 0x7f21c406aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.202+0000 7f21c9e1c640 1 -- 172.21.15.1:0/2483793874 >> 172.21.15.1:0/2483793874 conn(0x7f21c40a9700 msgr2=0x7f21c40a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.202+0000 7f21c9e1c640 1 -- 172.21.15.1:0/2483793874 shutdown_connections 2024-03-06T07:28:14.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.202+0000 7f21c9e1c640 1 -- 172.21.15.1:0/2483793874 wait complete. 2024-03-06T07:28:14.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.203+0000 7f21c9e1c640 1 Processor -- start 2024-03-06T07:28:14.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.203+0000 7f21c9e1c640 1 -- start start 2024-03-06T07:28:14.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.203+0000 7f21c9e1c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c406b530 0x7f21c41de7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.203+0000 7f21c9e1c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21c4152920 0x7f21c406fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.203+0000 7f21c9e1c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21c4155220 0x7f21c41eafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.203+0000 7f21c9e1c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c40c6380 con 0x7f21c4152920 2024-03-06T07:28:14.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.203+0000 7f21c9e1c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c4159240 con 0x7f21c4155220 2024-03-06T07:28:14.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.203+0000 7f21c9e1c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c40c5f60 con 0x7f21c406b530 2024-03-06T07:28:14.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.204+0000 7f21c2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21c4152920 0x7f21c406fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.204+0000 7f21c2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21c4152920 0x7f21c406fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40410/0 (socket says 172.21.15.1:40410) 2024-03-06T07:28:14.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.204+0000 7f21c2ffd640 1 -- 172.21.15.1:0/1924548712 learned_addr learned my addr 172.21.15.1:0/1924548712 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:14.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.204+0000 7f21c37fe640 1 --2- 172.21.15.1:0/1924548712 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c406b530 0x7f21c41de7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.204+0000 7f21c3fff640 1 --2- 172.21.15.1:0/1924548712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21c4155220 0x7f21c41eafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.204+0000 7f21c2ffd640 1 -- 172.21.15.1:0/1924548712 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c406b530 msgr2=0x7f21c41de7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.204+0000 7f21c2ffd640 1 --2- 172.21.15.1:0/1924548712 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c406b530 0x7f21c41de7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.204+0000 7f21c2ffd640 1 -- 172.21.15.1:0/1924548712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21c4155220 msgr2=0x7f21c41eafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.204+0000 7f21c2ffd640 1 --2- 172.21.15.1:0/1924548712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21c4155220 0x7f21c41eafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.204+0000 7f21c2ffd640 1 -- 172.21.15.1:0/1924548712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c41e25d0 con 0x7f21c4152920 2024-03-06T07:28:14.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.204+0000 7f21c2ffd640 1 --2- 172.21.15.1:0/1924548712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21c4152920 0x7f21c406fad0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f21b800c470 tx=0x7f21b8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.204+0000 7f21abfff640 1 -- 172.21.15.1:0/1924548712 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b80052d0 con 0x7f21c4152920 2024-03-06T07:28:14.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.204+0000 7f21abfff640 1 -- 172.21.15.1:0/1924548712 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b8005470 con 0x7f21c4152920 2024-03-06T07:28:14.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.204+0000 7f21c9e1c640 1 -- 172.21.15.1:0/1924548712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c41eed30 con 0x7f21c4152920 2024-03-06T07:28:14.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.204+0000 7f21abfff640 1 -- 172.21.15.1:0/1924548712 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b8007260 con 0x7f21c4152920 2024-03-06T07:28:14.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.204+0000 7f21c9e1c640 1 -- 172.21.15.1:0/1924548712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c4151290 con 0x7f21c4152920 2024-03-06T07:28:14.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.205+0000 7f21c9e1c640 1 -- 172.21.15.1:0/1924548712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c4067fd0 con 0x7f21c4152920 2024-03-06T07:28:14.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.206+0000 7f21abfff640 1 -- 172.21.15.1:0/1924548712 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f21b8007400 con 0x7f21c4152920 2024-03-06T07:28:14.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.206+0000 7f21abfff640 1 --2- 172.21.15.1:0/1924548712 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f21b006b4d0 0x7f21b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.206+0000 7f21abfff640 1 -- 172.21.15.1:0/1924548712 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f21b80897d0 con 0x7f21c4152920 2024-03-06T07:28:14.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.207+0000 7f21c37fe640 1 --2- 172.21.15.1:0/1924548712 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f21b006b4d0 0x7f21b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.207+0000 7f21c37fe640 1 --2- 172.21.15.1:0/1924548712 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f21b006b4d0 0x7f21b006d990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f21b4006fd0 tx=0x7f21b4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.211+0000 7f21abfff640 1 -- 172.21.15.1:0/1924548712 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21b8057e90 con 0x7f21c4152920 2024-03-06T07:28:14.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.431+0000 7f21c9e1c640 1 -- 172.21.15.1:0/1924548712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f21c406bdf0 con 0x7f21c4152920 2024-03-06T07:28:14.923 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:28:14.921+0000 7fb853791640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:28:14.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.931+0000 7f21abfff640 1 -- 172.21.15.1:0/1924548712 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v175) v1 ==== 105+0+0 (secure 0 0 0) 0x7f21b805bf30 con 0x7f21c4152920 2024-03-06T07:28:14.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.934+0000 7f21c9e1c640 1 -- 172.21.15.1:0/1924548712 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f21b006b4d0 msgr2=0x7f21b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.934+0000 7f21c9e1c640 1 --2- 172.21.15.1:0/1924548712 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f21b006b4d0 0x7f21b006d990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f21b4006fd0 tx=0x7f21b4008040 comp rx=0 tx=0).stop 2024-03-06T07:28:14.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.934+0000 7f21c9e1c640 1 -- 172.21.15.1:0/1924548712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21c4152920 msgr2=0x7f21c406fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.934+0000 7f21c9e1c640 1 --2- 172.21.15.1:0/1924548712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21c4152920 0x7f21c406fad0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f21b800c470 tx=0x7f21b8001e70 comp rx=0 tx=0).stop 2024-03-06T07:28:14.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.934+0000 7f21c9e1c640 1 -- 172.21.15.1:0/1924548712 shutdown_connections 2024-03-06T07:28:14.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.934+0000 7f21c9e1c640 1 --2- 172.21.15.1:0/1924548712 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f21b006b4d0 0x7f21b006d990 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.934+0000 7f21c9e1c640 1 --2- 172.21.15.1:0/1924548712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21c4155220 0x7f21c41eafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.934+0000 7f21c9e1c640 1 --2- 172.21.15.1:0/1924548712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21c4152920 0x7f21c406fad0 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.934+0000 7f21c9e1c640 1 --2- 172.21.15.1:0/1924548712 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21c406b530 0x7f21c41de7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.934+0000 7f21c9e1c640 1 -- 172.21.15.1:0/1924548712 >> 172.21.15.1:0/1924548712 conn(0x7f21c40a9700 msgr2=0x7f21c41545b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.934+0000 7f21c9e1c640 1 -- 172.21.15.1:0/1924548712 shutdown_connections 2024-03-06T07:28:14.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:14.934+0000 7f21c9e1c640 1 -- 172.21.15.1:0/1924548712 wait complete. 2024-03-06T07:28:14.951 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:15.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.053+0000 7fe1c64c3640 1 Processor -- start 2024-03-06T07:28:15.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.053+0000 7fe1c64c3640 1 -- start start 2024-03-06T07:28:15.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.054+0000 7fe1c64c3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1c00c1260 0x7fe1c00c16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.054+0000 7fe1c64c3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1c01565a0 0x7fe1c0156a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.054+0000 7fe1c64c3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c01530a0 con 0x7fe1c0149a90 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.054+0000 7fe1c64c3640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c0153380 con 0x7fe1c01565a0 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.054+0000 7fe1c64c3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c0153660 con 0x7fe1c00c1260 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.054+0000 7fe1bf7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1c00c1260 0x7fe1c00c16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.054+0000 7fe1bf7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1c00c1260 0x7fe1c00c16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:41472/0 (socket says 172.21.15.1:41472) 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.054+0000 7fe1bf7fe640 1 -- 172.21.15.1:0/1084302371 learned_addr learned my addr 172.21.15.1:0/1084302371 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.054+0000 7fe1c4a60640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1c01565a0 0x7fe1c0156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.054+0000 7fe1bf7fe640 1 -- 172.21.15.1:0/1084302371 >> v2:172.21.15.98:3300/0 conn(0x7fe1c01565a0 msgr2=0x7fe1c0156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.054+0000 7fe1bf7fe640 1 --2- 172.21.15.1:0/1084302371 >> v2:172.21.15.98:3300/0 conn(0x7fe1c01565a0 0x7fe1c0156a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.054+0000 7fe1bf7fe640 1 -- 172.21.15.1:0/1084302371 >> v1:172.21.15.98:6789/0 conn(0x7fe1c0149a90 legacy=0x7fe1c00c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.054+0000 7fe1bf7fe640 1 -- 172.21.15.1:0/1084302371 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c0153940 con 0x7fe1c00c1260 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.054+0000 7fe1c4a60640 1 --2- 172.21.15.1:0/1084302371 >> v2:172.21.15.98:3300/0 conn(0x7fe1c01565a0 0x7fe1c0156a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.055+0000 7fe1bf7fe640 1 --2- 172.21.15.1:0/1084302371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1c00c1260 0x7fe1c00c16f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009870 tx=0x7fe1b002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e10eff133ae02d7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.055+0000 7fe1beffd640 1 -- 172.21.15.1:0/1084302371 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b0035020 con 0x7fe1c00c1260 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.055+0000 7fe1beffd640 1 -- 172.21.15.1:0/1084302371 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b0031640 con 0x7fe1c00c1260 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.055+0000 7fe1beffd640 1 -- 172.21.15.1:0/1084302371 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b002d040 con 0x7fe1c00c1260 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.056+0000 7fe1c64c3640 1 -- 172.21.15.1:0/1084302371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1c00c1260 msgr2=0x7fe1c00c16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.056+0000 7fe1c64c3640 1 --2- 172.21.15.1:0/1084302371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1c00c1260 0x7fe1c00c16f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009870 tx=0x7fe1b002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.056+0000 7fe1c64c3640 1 -- 172.21.15.1:0/1084302371 shutdown_connections 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.056+0000 7fe1c64c3640 1 --2- 172.21.15.1:0/1084302371 >> v2:172.21.15.98:3300/0 conn(0x7fe1c01565a0 0x7fe1c0156a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.056+0000 7fe1c64c3640 1 --2- 172.21.15.1:0/1084302371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1c00c1260 0x7fe1c00c16f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.056+0000 7fe1c64c3640 1 -- 172.21.15.1:0/1084302371 >> 172.21.15.1:0/1084302371 conn(0x7fe1c00c6860 msgr2=0x7fe1c00c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.056+0000 7fe1c64c3640 1 -- 172.21.15.1:0/1084302371 shutdown_connections 2024-03-06T07:28:15.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.056+0000 7fe1c64c3640 1 -- 172.21.15.1:0/1084302371 wait complete. 2024-03-06T07:28:15.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.057+0000 7fe1c64c3640 1 Processor -- start 2024-03-06T07:28:15.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.057+0000 7fe1c64c3640 1 -- start start 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.058+0000 7fe1c64c3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1c00c1260 0x7fe1c01e0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.058+0000 7fe1c64c3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c0149a90 0x7fe1c01ec1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.058+0000 7fe1c64c3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c01565a0 0x7fe1c014e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.058+0000 7fe1c64c3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c006f620 con 0x7fe1c0149a90 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.058+0000 7fe1c64c3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c006f4a0 con 0x7fe1c01565a0 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.058+0000 7fe1c64c3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c0156f70 con 0x7fe1c00c1260 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.058+0000 7fe1bf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c0149a90 0x7fe1c01ec1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.058+0000 7fe1bf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c0149a90 0x7fe1c01ec1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50800/0 (socket says 172.21.15.1:50800) 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.058+0000 7fe1bf7fe640 1 -- 172.21.15.1:0/4000035165 learned_addr learned my addr 172.21.15.1:0/4000035165 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.058+0000 7fe1c4a60640 1 --2- 172.21.15.1:0/4000035165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c01565a0 0x7fe1c014e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.058+0000 7fe1bffff640 1 --2- 172.21.15.1:0/4000035165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1c00c1260 0x7fe1c01e0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.058+0000 7fe1bf7fe640 1 -- 172.21.15.1:0/4000035165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1c00c1260 msgr2=0x7fe1c01e0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.058+0000 7fe1bf7fe640 1 --2- 172.21.15.1:0/4000035165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1c00c1260 0x7fe1c01e0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.058+0000 7fe1bf7fe640 1 -- 172.21.15.1:0/4000035165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c01565a0 msgr2=0x7fe1c014e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.058+0000 7fe1bf7fe640 1 --2- 172.21.15.1:0/4000035165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c01565a0 0x7fe1c014e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.058+0000 7fe1bf7fe640 1 -- 172.21.15.1:0/4000035165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c01ecc60 con 0x7fe1c0149a90 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.059+0000 7fe1bf7fe640 1 --2- 172.21.15.1:0/4000035165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c0149a90 0x7fe1c01ec1b0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0000c00 tx=0x7fe1b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.059+0000 7fe1bcff9640 1 -- 172.21.15.1:0/4000035165 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b0040070 con 0x7fe1c0149a90 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.059+0000 7fe1bcff9640 1 -- 172.21.15.1:0/4000035165 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b00399f0 con 0x7fe1c0149a90 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.059+0000 7fe1c64c3640 1 -- 172.21.15.1:0/4000035165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c01ecf00 con 0x7fe1c0149a90 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.059+0000 7fe1bcff9640 1 -- 172.21.15.1:0/4000035165 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b0035040 con 0x7fe1c0149a90 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.059+0000 7fe1c64c3640 1 -- 172.21.15.1:0/4000035165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c01507b0 con 0x7fe1c0149a90 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.060+0000 7fe1c64c3640 1 -- 172.21.15.1:0/4000035165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c0150ae0 con 0x7fe1c0149a90 2024-03-06T07:28:15.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.060+0000 7fe1bcff9640 1 -- 172.21.15.1:0/4000035165 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1b004b060 con 0x7fe1c0149a90 2024-03-06T07:28:15.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.061+0000 7fe1bcff9640 1 --2- 172.21.15.1:0/4000035165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe1ac06b4d0 0x7fe1ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.061+0000 7fe1bcff9640 1 -- 172.21.15.1:0/4000035165 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7fe1b00abc10 con 0x7fe1c0149a90 2024-03-06T07:28:15.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.061+0000 7fe1bffff640 1 --2- 172.21.15.1:0/4000035165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe1ac06b4d0 0x7fe1ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.062+0000 7fe1bffff640 1 --2- 172.21.15.1:0/4000035165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe1ac06b4d0 0x7fe1ac06d990 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4001640 tx=0x7fe1b4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.065+0000 7fe1bcff9640 1 -- 172.21.15.1:0/4000035165 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1b007a920 con 0x7fe1c0149a90 2024-03-06T07:28:15.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.283+0000 7fe1c64c3640 1 -- 172.21.15.1:0/4000035165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1c0067fd0 con 0x7fe1c0149a90 2024-03-06T07:28:15.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.283+0000 7fe1bcff9640 1 -- 172.21.15.1:0/4000035165 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+4979 (secure 0 0 0) 0x7fe1b007a220 con 0x7fe1c0149a90 2024-03-06T07:28:15.285 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:15.286 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":176,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":9951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:28:14.934186+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0],"up":{"mds_0":10161},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10161":{"gid":10161,"name":"c","rank":0,"incarnation":176,"state":"up:starting","state_seq":1,"addr":"172.21.15.1:6835/1525330395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1525330395},{"type":"v1","addr":"172.21.15.1:6835","nonce":1525330395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:28:15.286 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 176 2024-03-06T07:28:15.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.285+0000 7fe1c64c3640 1 -- 172.21.15.1:0/4000035165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe1ac06b4d0 msgr2=0x7fe1ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.285+0000 7fe1c64c3640 1 --2- 172.21.15.1:0/4000035165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe1ac06b4d0 0x7fe1ac06d990 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4001640 tx=0x7fe1b4009040 comp rx=0 tx=0).stop 2024-03-06T07:28:15.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.285+0000 7fe1c64c3640 1 -- 172.21.15.1:0/4000035165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c0149a90 msgr2=0x7fe1c01ec1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.285+0000 7fe1c64c3640 1 --2- 172.21.15.1:0/4000035165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c0149a90 0x7fe1c01ec1b0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0000c00 tx=0x7fe1b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.285+0000 7fe1c64c3640 1 -- 172.21.15.1:0/4000035165 shutdown_connections 2024-03-06T07:28:15.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.285+0000 7fe1c64c3640 1 --2- 172.21.15.1:0/4000035165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe1ac06b4d0 0x7fe1ac06d990 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.285+0000 7fe1c64c3640 1 --2- 172.21.15.1:0/4000035165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c01565a0 0x7fe1c014e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.285+0000 7fe1c64c3640 1 --2- 172.21.15.1:0/4000035165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c0149a90 0x7fe1c01ec1b0 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.285+0000 7fe1c64c3640 1 --2- 172.21.15.1:0/4000035165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe1c00c1260 0x7fe1c01e0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.285+0000 7fe1c64c3640 1 -- 172.21.15.1:0/4000035165 >> 172.21.15.1:0/4000035165 conn(0x7fe1c00c6860 msgr2=0x7fe1c00c6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.285+0000 7fe1c64c3640 1 -- 172.21.15.1:0/4000035165 shutdown_connections 2024-03-06T07:28:15.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:15.286+0000 7fe1c64c3640 1 -- 172.21.15.1:0/4000035165 wait complete. 2024-03-06T07:28:15.700 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 176, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:52.179842+0000', 'modified': '2024-03-06T07:28:14.934186+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 10161}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10161': {'gid': 10161, 'name': 'c', 'rank': 0, 'incarnation': 176, 'state': 'up:starting', 'state_seq': 1, 'addr': '172.21.15.1:6835/1525330395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 1525330395}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1525330395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:28:15.701 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10161:up:starting 2024-03-06T07:28:16.701 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:16.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.801+0000 7f726ffeb640 1 Processor -- start 2024-03-06T07:28:16.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.801+0000 7f726ffeb640 1 -- start start 2024-03-06T07:28:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.802+0000 7f726ffeb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f72681505b0 0x7f72680c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.802+0000 7f726ffeb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72680c1260 0x7f72680c16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.802+0000 7f726ffeb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726806f5c0 con 0x7f72681566e0 2024-03-06T07:28:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.802+0000 7f726ffeb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7268152f80 con 0x7f72681505b0 2024-03-06T07:28:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.802+0000 7f726ffeb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7268150990 con 0x7f72680c1260 2024-03-06T07:28:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.802+0000 7f726dd87640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f72681505b0 0x7f72680c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.802+0000 7f726d586640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72680c1260 0x7f72680c16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.802+0000 7f726dd87640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f72681505b0 0x7f72680c4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:43332/0 (socket says 172.21.15.1:43332) 2024-03-06T07:28:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.802+0000 7f726d586640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72680c1260 0x7f72680c16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:41486/0 (socket says 172.21.15.1:41486) 2024-03-06T07:28:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.802+0000 7f726dd87640 1 -- 172.21.15.1:0/1144379598 learned_addr learned my addr 172.21.15.1:0/1144379598 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.803+0000 7f726d586640 1 -- 172.21.15.1:0/1144379598 >> v2:172.21.15.98:3300/0 conn(0x7f72681505b0 msgr2=0x7f72680c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.803+0000 7f726d586640 1 --2- 172.21.15.1:0/1144379598 >> v2:172.21.15.98:3300/0 conn(0x7f72681505b0 0x7f72680c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.803+0000 7f726d586640 1 -- 172.21.15.1:0/1144379598 >> v1:172.21.15.98:6789/0 conn(0x7f72681566e0 legacy=0x7f7268156b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.803+0000 7f726d586640 1 -- 172.21.15.1:0/1144379598 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7268153240 con 0x7f72680c1260 2024-03-06T07:28:16.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.803+0000 7f726dd87640 1 --2- 172.21.15.1:0/1144379598 >> v2:172.21.15.98:3300/0 conn(0x7f72681505b0 0x7f72680c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.804+0000 7f726d586640 1 --2- 172.21.15.1:0/1144379598 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72680c1260 0x7f72680c16e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f725c0097a0 tx=0x7f725c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2af937ca73976f6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.804+0000 7f726cd85640 1 -- 172.21.15.1:0/1144379598 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c0315a0 con 0x7f72680c1260 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.804+0000 7f726cd85640 1 -- 172.21.15.1:0/1144379598 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725c037070 con 0x7f72680c1260 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.804+0000 7f726cd85640 1 -- 172.21.15.1:0/1144379598 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c031960 con 0x7f72680c1260 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.804+0000 7f726ffeb640 1 -- 172.21.15.1:0/1144379598 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72680c1260 msgr2=0x7f72680c16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.804+0000 7f726ffeb640 1 --2- 172.21.15.1:0/1144379598 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72680c1260 0x7f72680c16e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f725c0097a0 tx=0x7f725c02b840 comp rx=0 tx=0).stop 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.804+0000 7f726ffeb640 1 -- 172.21.15.1:0/1144379598 shutdown_connections 2024-03-06T07:28:16.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.804+0000 7f726ffeb640 1 --2- 172.21.15.1:0/1144379598 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72680c1260 0x7f72680c16e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.804+0000 7f726ffeb640 1 --2- 172.21.15.1:0/1144379598 >> v2:172.21.15.98:3300/0 conn(0x7f72681505b0 0x7f72680c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.804+0000 7f726ffeb640 1 -- 172.21.15.1:0/1144379598 >> 172.21.15.1:0/1144379598 conn(0x7f72680c6860 msgr2=0x7f72680c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.804+0000 7f726ffeb640 1 -- 172.21.15.1:0/1144379598 shutdown_connections 2024-03-06T07:28:16.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.804+0000 7f726ffeb640 1 -- 172.21.15.1:0/1144379598 wait complete. 2024-03-06T07:28:16.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.805+0000 7f726ffeb640 1 Processor -- start 2024-03-06T07:28:16.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.805+0000 7f726ffeb640 1 -- start start 2024-03-06T07:28:16.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.806+0000 7f726ffeb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72680c1260 0x7f72680c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.806+0000 7f726ffeb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72681505b0 0x7f72681e9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.806+0000 7f726ffeb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72681566e0 0x7f72681ea110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.806+0000 7f726ffeb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7268153ae0 con 0x7f72681505b0 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.806+0000 7f726ffeb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7268153800 con 0x7f72680c1260 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.806+0000 7f726ffeb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7268157280 con 0x7f72681566e0 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.806+0000 7f726d586640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72681505b0 0x7f72681e9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.806+0000 7f726d586640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72681505b0 0x7f72681e9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50810/0 (socket says 172.21.15.1:50810) 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.806+0000 7f726d586640 1 -- 172.21.15.1:0/2896959658 learned_addr learned my addr 172.21.15.1:0/2896959658 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.806+0000 7f726e588640 1 --2- 172.21.15.1:0/2896959658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72681566e0 0x7f72681ea110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.806+0000 7f726dd87640 1 --2- 172.21.15.1:0/2896959658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72680c1260 0x7f72680c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.807+0000 7f726d586640 1 -- 172.21.15.1:0/2896959658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72681566e0 msgr2=0x7f72681ea110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.807+0000 7f726d586640 1 --2- 172.21.15.1:0/2896959658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72681566e0 0x7f72681ea110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.807+0000 7f726d586640 1 -- 172.21.15.1:0/2896959658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72680c1260 msgr2=0x7f72680c5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.807+0000 7f726d586640 1 --2- 172.21.15.1:0/2896959658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72680c1260 0x7f72680c5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.807+0000 7f726d586640 1 -- 172.21.15.1:0/2896959658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72681496e0 con 0x7f72681505b0 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.807+0000 7f726dd87640 1 --2- 172.21.15.1:0/2896959658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72680c1260 0x7f72680c5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.807+0000 7f726e588640 1 --2- 172.21.15.1:0/2896959658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72681566e0 0x7f72681ea110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.807+0000 7f726d586640 1 --2- 172.21.15.1:0/2896959658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72681505b0 0x7f72681e9bd0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f725c034040 tx=0x7f725c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.807+0000 7f725a7fc640 1 -- 172.21.15.1:0/2896959658 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c032630 con 0x7f72681505b0 2024-03-06T07:28:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.807+0000 7f725a7fc640 1 -- 172.21.15.1:0/2896959658 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725c0327d0 con 0x7f72681505b0 2024-03-06T07:28:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.807+0000 7f726ffeb640 1 -- 172.21.15.1:0/2896959658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72681562e0 con 0x7f72681505b0 2024-03-06T07:28:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.807+0000 7f725a7fc640 1 -- 172.21.15.1:0/2896959658 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c032bc0 con 0x7f72681505b0 2024-03-06T07:28:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.807+0000 7f726ffeb640 1 -- 172.21.15.1:0/2896959658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72681554b0 con 0x7f72681505b0 2024-03-06T07:28:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.808+0000 7f726ffeb640 1 -- 172.21.15.1:0/2896959658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72681560d0 con 0x7f72681505b0 2024-03-06T07:28:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.808+0000 7f725a7fc640 1 -- 172.21.15.1:0/2896959658 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f725c04c020 con 0x7f72681505b0 2024-03-06T07:28:16.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.809+0000 7f725a7fc640 1 --2- 172.21.15.1:0/2896959658 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f724c06b500 0x7f724c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.809+0000 7f725a7fc640 1 -- 172.21.15.1:0/2896959658 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f725c02f020 con 0x7f72681505b0 2024-03-06T07:28:16.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.809+0000 7f726dd87640 1 --2- 172.21.15.1:0/2896959658 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f724c06b500 0x7f724c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.810+0000 7f726dd87640 1 --2- 172.21.15.1:0/2896959658 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f724c06b500 0x7f724c06d9c0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f7264009b80 tx=0x7f7264008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:16.817+0000 7f725a7fc640 1 -- 172.21.15.1:0/2896959658 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f725c040070 con 0x7f72681505b0 2024-03-06T07:28:17.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.048+0000 7f726ffeb640 1 -- 172.21.15.1:0/2896959658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7268067fd0 con 0x7f72681505b0 2024-03-06T07:28:17.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.049+0000 7f725a7fc640 1 -- 172.21.15.1:0/2896959658 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+5005 (secure 0 0 0) 0x7f725c0302f0 con 0x7f72681505b0 2024-03-06T07:28:17.051 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:17.051 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":179,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:28:16.947047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10161,"mds_1":9951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9951":{"gid":9951,"name":"a","rank":1,"incarnation":178,"state":"up:active","state_seq":9,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10161":{"gid":10161,"name":"c","rank":0,"incarnation":176,"state":"up:active","state_seq":2,"addr":"172.21.15.1:6835/1525330395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1525330395},{"type":"v1","addr":"172.21.15.1:6835","nonce":1525330395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10161,"qdb_cluster":[9951,10161]},"id":14}]} 2024-03-06T07:28:17.051 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 179 2024-03-06T07:28:17.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.051+0000 7f726ffeb640 1 -- 172.21.15.1:0/2896959658 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f724c06b500 msgr2=0x7f724c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.051+0000 7f726ffeb640 1 --2- 172.21.15.1:0/2896959658 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f724c06b500 0x7f724c06d9c0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f7264009b80 tx=0x7f7264008040 comp rx=0 tx=0).stop 2024-03-06T07:28:17.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.051+0000 7f726ffeb640 1 -- 172.21.15.1:0/2896959658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72681505b0 msgr2=0x7f72681e9bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.051+0000 7f726ffeb640 1 --2- 172.21.15.1:0/2896959658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72681505b0 0x7f72681e9bd0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f725c034040 tx=0x7f725c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.051+0000 7f726ffeb640 1 -- 172.21.15.1:0/2896959658 shutdown_connections 2024-03-06T07:28:17.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.051+0000 7f726ffeb640 1 --2- 172.21.15.1:0/2896959658 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f724c06b500 0x7f724c06d9c0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.051+0000 7f726ffeb640 1 --2- 172.21.15.1:0/2896959658 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72681566e0 0x7f72681ea110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.051+0000 7f726ffeb640 1 --2- 172.21.15.1:0/2896959658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72681505b0 0x7f72681e9bd0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.051+0000 7f726ffeb640 1 --2- 172.21.15.1:0/2896959658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72680c1260 0x7f72680c5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.051+0000 7f726ffeb640 1 -- 172.21.15.1:0/2896959658 >> 172.21.15.1:0/2896959658 conn(0x7f72680c6860 msgr2=0x7f726814dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.051+0000 7f726ffeb640 1 -- 172.21.15.1:0/2896959658 shutdown_connections 2024-03-06T07:28:17.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.051+0000 7f726ffeb640 1 -- 172.21.15.1:0/2896959658 wait complete. 2024-03-06T07:28:17.467 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 179, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:52.179842+0000', 'modified': '2024-03-06T07:28:16.947047+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 10161, 'mds_1': 9951}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9951': {'gid': 9951, 'name': 'a', 'rank': 1, 'incarnation': 178, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.1:6833/3895143314', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 3895143314}, {'type': 'v1', 'addr': '172.21.15.1:6833', 'nonce': 3895143314}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10161': {'gid': 10161, 'name': 'c', 'rank': 0, 'incarnation': 176, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.1:6835/1525330395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 1525330395}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1525330395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10161, 'qdb_cluster': [9951, 10161]} 2024-03-06T07:28:17.467 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-03-06T07:28:17.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:28:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.576+0000 7f53b99a0640 1 Processor -- start 2024-03-06T07:28:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.576+0000 7f53b99a0640 1 -- start start 2024-03-06T07:28:17.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.577+0000 7f53b99a0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f53b40c6ee0 0x7f53b40c5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.577+0000 7f53b99a0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b4155e90 con 0x7f53b4152840 2024-03-06T07:28:17.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.577+0000 7f53b99a0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b40c14d0 con 0x7f53b40bf900 2024-03-06T07:28:17.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.577+0000 7f53b99a0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b40c1b20 con 0x7f53b40c6ee0 2024-03-06T07:28:17.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.577+0000 7f53b3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f53b40c6ee0 0x7f53b40c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.577+0000 7f53b899e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f53b4152840 0x7f53b40ca890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39736/0 (socket says 172.21.15.1:39736) 2024-03-06T07:28:17.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.577+0000 7f53b3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f53b40c6ee0 0x7f53b40c5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50820/0 (socket says 172.21.15.1:50820) 2024-03-06T07:28:17.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.577+0000 7f53b899e640 1 -- 172.21.15.1:0/1196584389 learned_addr learned my addr 172.21.15.1:0/1196584389 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:17.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.578+0000 7f53b3fff640 1 -- 172.21.15.1:0/1196584389 >> v1:172.21.15.98:6789/0 conn(0x7f53b40bf900 legacy=0x7f53b40c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.578+0000 7f53b3fff640 1 -- 172.21.15.1:0/1196584389 >> v1:172.21.15.1:6789/0 conn(0x7f53b4152840 legacy=0x7f53b40ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.578+0000 7f53b3fff640 1 -- 172.21.15.1:0/1196584389 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b40c17b0 con 0x7f53b40c6ee0 2024-03-06T07:28:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.578+0000 7f53b3fff640 1 --2- 172.21.15.1:0/1196584389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53b40c6ee0 0x7f53b40c5490 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f53a00097a0 tx=0x7f53a002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f7b082ececcb93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.579+0000 7f53b37fe640 1 -- 172.21.15.1:0/1196584389 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a0038070 con 0x7f53b40c6ee0 2024-03-06T07:28:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.579+0000 7f53b37fe640 1 -- 172.21.15.1:0/1196584389 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a0031700 con 0x7f53b40c6ee0 2024-03-06T07:28:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.579+0000 7f53b37fe640 1 -- 172.21.15.1:0/1196584389 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a0031ad0 con 0x7f53b40c6ee0 2024-03-06T07:28:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.579+0000 7f53b99a0640 1 -- 172.21.15.1:0/1196584389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53b40c6ee0 msgr2=0x7f53b40c5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.579+0000 7f53b99a0640 1 --2- 172.21.15.1:0/1196584389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53b40c6ee0 0x7f53b40c5490 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f53a00097a0 tx=0x7f53a002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.579+0000 7f53b99a0640 1 -- 172.21.15.1:0/1196584389 shutdown_connections 2024-03-06T07:28:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.579+0000 7f53b99a0640 1 --2- 172.21.15.1:0/1196584389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53b40c6ee0 0x7f53b40c5490 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.579+0000 7f53b99a0640 1 -- 172.21.15.1:0/1196584389 >> 172.21.15.1:0/1196584389 conn(0x7f53b4067520 msgr2=0x7f53b406d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.579+0000 7f53b99a0640 1 -- 172.21.15.1:0/1196584389 shutdown_connections 2024-03-06T07:28:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.579+0000 7f53b99a0640 1 -- 172.21.15.1:0/1196584389 wait complete. 2024-03-06T07:28:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.580+0000 7f53b99a0640 1 Processor -- start 2024-03-06T07:28:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.580+0000 7f53b99a0640 1 -- start start 2024-03-06T07:28:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.582+0000 7f53b99a0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53b40bf900 0x7f53b414a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.582+0000 7f53b99a0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53b40c6ee0 0x7f53b40ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.582+0000 7f53b99a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b4152840 0x7f53b40ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.582+0000 7f53b99a0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b40cafa0 con 0x7f53b40c6ee0 2024-03-06T07:28:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.582+0000 7f53b99a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b4155b60 con 0x7f53b4152840 2024-03-06T07:28:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.582+0000 7f53b99a0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b40cb120 con 0x7f53b40bf900 2024-03-06T07:28:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.582+0000 7f53b3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53b40c6ee0 0x7f53b40ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.582+0000 7f53b3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53b40c6ee0 0x7f53b40ca3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50826/0 (socket says 172.21.15.1:50826) 2024-03-06T07:28:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.582+0000 7f53b3fff640 1 -- 172.21.15.1:0/2795391830 learned_addr learned my addr 172.21.15.1:0/2795391830 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:17.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.582+0000 7f53b899e640 1 --2- 172.21.15.1:0/2795391830 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53b40bf900 0x7f53b414a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.582+0000 7f53b919f640 1 --2- 172.21.15.1:0/2795391830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b4152840 0x7f53b40ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.582+0000 7f53b3fff640 1 -- 172.21.15.1:0/2795391830 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53b40bf900 msgr2=0x7f53b414a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.582+0000 7f53b3fff640 1 --2- 172.21.15.1:0/2795391830 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53b40bf900 0x7f53b414a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.582+0000 7f53b3fff640 1 -- 172.21.15.1:0/2795391830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b4152840 msgr2=0x7f53b40ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.582+0000 7f53b3fff640 1 --2- 172.21.15.1:0/2795391830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b4152840 0x7f53b40ca8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.582+0000 7f53b3fff640 1 -- 172.21.15.1:0/2795391830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b4147720 con 0x7f53b40c6ee0 2024-03-06T07:28:17.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.582+0000 7f53b3fff640 1 --2- 172.21.15.1:0/2795391830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53b40c6ee0 0x7f53b40ca3a0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f53a002b7e0 tx=0x7f53a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.583+0000 7f53b17fa640 1 -- 172.21.15.1:0/2795391830 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a0043070 con 0x7f53b40c6ee0 2024-03-06T07:28:17.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.583+0000 7f53b17fa640 1 -- 172.21.15.1:0/2795391830 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a0033a30 con 0x7f53b40c6ee0 2024-03-06T07:28:17.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.583+0000 7f53b99a0640 1 -- 172.21.15.1:0/2795391830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b4151c50 con 0x7f53b40c6ee0 2024-03-06T07:28:17.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.583+0000 7f53b17fa640 1 -- 172.21.15.1:0/2795391830 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a0038040 con 0x7f53b40c6ee0 2024-03-06T07:28:17.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.583+0000 7f53b99a0640 1 -- 172.21.15.1:0/2795391830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b4147990 con 0x7f53b40c6ee0 2024-03-06T07:28:17.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.585+0000 7f53b17fa640 1 -- 172.21.15.1:0/2795391830 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53a0046070 con 0x7f53b40c6ee0 2024-03-06T07:28:17.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.585+0000 7f53b17fa640 1 --2- 172.21.15.1:0/2795391830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f539806b4d0 0x7f539806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.585+0000 7f53b17fa640 1 -- 172.21.15.1:0/2795391830 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f53a00ac300 con 0x7f53b40c6ee0 2024-03-06T07:28:17.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.585+0000 7f53b899e640 1 --2- 172.21.15.1:0/2795391830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f539806b4d0 0x7f539806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.586+0000 7f53bd7ed740 1 Processor -- start 2024-03-06T07:28:17.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.586+0000 7f53b899e640 1 --2- 172.21.15.1:0/2795391830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f539806b4d0 0x7f539806d990 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f53a8007fd0 tx=0x7f53a8007e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.586+0000 7f53bd7ed740 1 -- start start 2024-03-06T07:28:17.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.587+0000 7f53bd7ed740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b100 0x565532c98be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.587+0000 7f53bd7ed740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532c9b4e0 0x565532c99120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.587+0000 7f53bd7ed740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c99660 0x565532c95830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.587+0000 7f53bd7ed740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x565532bff6b0 con 0x565532c99660 2024-03-06T07:28:17.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.587+0000 7f53bd7ed740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x565532ba8a60 con 0x565532c9b100 2024-03-06T07:28:17.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.587+0000 7f53bd7ed740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x565532bfb750 con 0x565532c9b4e0 2024-03-06T07:28:17.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.587+0000 7f53b919f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c99660 0x565532c95830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.587+0000 7f53b919f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c99660 0x565532c95830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50828/0 (socket says 172.21.15.1:50828) 2024-03-06T07:28:17.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.587+0000 7f53b919f640 1 -- 172.21.15.1:0/1405099486 learned_addr learned my addr 172.21.15.1:0/1405099486 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:17.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.587+0000 7f53b899e640 1 --2- 172.21.15.1:0/1405099486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b100 0x565532c98be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.587+0000 7f53b3fff640 1 --2- 172.21.15.1:0/1405099486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532c9b4e0 0x565532c99120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.587+0000 7f53b919f640 1 -- 172.21.15.1:0/1405099486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532c9b4e0 msgr2=0x565532c99120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.587+0000 7f53b919f640 1 --2- 172.21.15.1:0/1405099486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532c9b4e0 0x565532c99120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.587+0000 7f53b919f640 1 -- 172.21.15.1:0/1405099486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b100 msgr2=0x565532c98be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.587+0000 7f53b919f640 1 --2- 172.21.15.1:0/1405099486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b100 0x565532c98be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.588+0000 7f53b919f640 1 -- 172.21.15.1:0/1405099486 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565532bc6f30 con 0x565532c99660 2024-03-06T07:28:17.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.588+0000 7f53b919f640 1 --2- 172.21.15.1:0/1405099486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c99660 0x565532c95830 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f53ac001f50 tx=0x7f53ac0015d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13571ce0aaf58fe2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.588+0000 7f539e7fc640 1 -- 172.21.15.1:0/1405099486 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac00fe50 con 0x565532c99660 2024-03-06T07:28:17.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.588+0000 7f539e7fc640 1 -- 172.21.15.1:0/1405099486 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac018070 con 0x565532c99660 2024-03-06T07:28:17.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.588+0000 7f539e7fc640 1 -- 172.21.15.1:0/1405099486 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac003bb0 con 0x565532c99660 2024-03-06T07:28:17.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.588+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1405099486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c99660 msgr2=0x565532c95830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.588+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/1405099486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c99660 0x565532c95830 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f53ac001f50 tx=0x7f53ac0015d0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.588+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1405099486 shutdown_connections 2024-03-06T07:28:17.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.588+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/1405099486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c99660 0x565532c95830 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.588+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/1405099486 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532c9b4e0 0x565532c99120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.588+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/1405099486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b100 0x565532c98be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.588+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1405099486 >> 172.21.15.1:0/1405099486 conn(0x565532c97980 msgr2=0x565532cbc570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.588+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1405099486 shutdown_connections 2024-03-06T07:28:17.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.589+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1405099486 wait complete. 2024-03-06T07:28:17.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.589+0000 7f53bd7ed740 1 -- start start 2024-03-06T07:28:17.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.590+0000 7f53bd7ed740 1 Processor -- start 2024-03-06T07:28:17.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53bd7ed740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b100 0x565532ce8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53bd7ed740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532c9b4e0 0x565532cbbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53bd7ed740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532cbf8a0 0x565532cd3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53bd7ed740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x565532cbd3e0 con 0x565532cbf8a0 2024-03-06T07:28:17.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53bd7ed740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x565532bbbea0 con 0x565532c9b100 2024-03-06T07:28:17.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53bd7ed740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x565532cbce20 con 0x565532c9b4e0 2024-03-06T07:28:17.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53bd7ed740 10 client.0 _ll_register_callbacks cb 0x565532bbb870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:17.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53b919f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532cbf8a0 0x565532cd3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53b919f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532cbf8a0 0x565532cd3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50832/0 (socket says 172.21.15.1:50832) 2024-03-06T07:28:17.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53b919f640 1 -- 172.21.15.1:0/3223428247 learned_addr learned my addr 172.21.15.1:0/3223428247 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:17.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53b899e640 1 --2- 172.21.15.1:0/3223428247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b100 0x565532ce8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53b3fff640 1 --2- 172.21.15.1:0/3223428247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532c9b4e0 0x565532cbbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53b919f640 1 -- 172.21.15.1:0/3223428247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532c9b4e0 msgr2=0x565532cbbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53b919f640 1 --2- 172.21.15.1:0/3223428247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532c9b4e0 0x565532cbbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53b919f640 1 -- 172.21.15.1:0/3223428247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b100 msgr2=0x565532ce8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53b919f640 1 --2- 172.21.15.1:0/3223428247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b100 0x565532ce8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53b919f640 1 -- 172.21.15.1:0/3223428247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565532bbc270 con 0x565532cbf8a0 2024-03-06T07:28:17.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53b919f640 1 --2- 172.21.15.1:0/3223428247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532cbf8a0 0x565532cd3ab0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00fa30 tx=0x7f53ac001330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d828413695052b77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.591+0000 7f53937fe640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:28:17.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.592+0000 7f53937fe640 1 -- 172.21.15.1:0/3223428247 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac003d80 con 0x565532cbf8a0 2024-03-06T07:28:17.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.592+0000 7f53bd7ed740 1 -- 172.21.15.1:0/3223428247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f53ac018070 con 0x565532cbf8a0 2024-03-06T07:28:17.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.592+0000 7f53937fe640 1 -- 172.21.15.1:0/3223428247 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f53ac0050d0 con 0x565532cbf8a0 2024-03-06T07:28:17.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.592+0000 7f53937fe640 1 -- 172.21.15.1:0/3223428247 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=179) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53ac02d050 con 0x565532cbf8a0 2024-03-06T07:28:17.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.592+0000 7f53bd7ed740 10 client.10203 fetch_fsmap learned FSMap version 179 2024-03-06T07:28:17.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.592+0000 7f53bd7ed740 1 -- 172.21.15.1:0/3223428247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=179,monmap=2+}) v3 -- 0x5655329142d0 con 0x565532cbf8a0 2024-03-06T07:28:17.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.592+0000 7f53937fe640 1 -- 172.21.15.1:0/3223428247 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac018070 con 0x565532cbf8a0 2024-03-06T07:28:17.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.592+0000 7f53937fe640 1 -- 172.21.15.1:0/3223428247 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 179) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f53ac030020 con 0x565532cbf8a0 2024-03-06T07:28:17.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.592+0000 7f53bd7ed740 10 client.10203 fetch_fsmap finished waiting for FSMap version 179 2024-03-06T07:28:17.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.592+0000 7f53bd7ed740 10 client.10203 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:28:17.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.592+0000 7f53bd7ed740 20 client.10203 populate_metadata read hostname 'smithi001' 2024-03-06T07:28:17.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.592+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/3223428247 >> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] conn(0x565532ceb8b0 0x565532cf66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.592+0000 7f53bd7ed740 4 client.10203 mds_command: new command op to 9951 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:28:17.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.592+0000 7f53bd7ed740 1 -- 172.21.15.1:0/3223428247 --> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565532c0cbf0 con 0x565532ceb8b0 2024-03-06T07:28:17.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.592+0000 7f53b899e640 1 --2- 172.21.15.1:0/3223428247 >> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] conn(0x565532ceb8b0 0x565532cf66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.593+0000 7f53937fe640 10 client.10203 ms_handle_connect on v2:172.21.15.1:6832/3895143314 2024-03-06T07:28:17.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.593+0000 7f53b899e640 1 --2- 172.21.15.1:0/3223428247 >> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] conn(0x565532ceb8b0 0x565532cf66d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c43eabf1cc1da218 server_cookie=2d4f9100d0341f11 in_seq=0 out_seq=0 2024-03-06T07:28:17.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.594+0000 7f53937fe640 1 -- 172.21.15.1:0/3223428247 <== mds.1 v2:172.21.15.1:6832/3895143314 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x565532c0cbf0 con 0x565532ceb8b0 2024-03-06T07:28:17.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.594+0000 7f53937fe640 10 client.10203 handle_command_reply: tid=0 2024-03-06T07:28:17.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.594+0000 7f53bd7ed740 1 client.10203 shutdown 2024-03-06T07:28:17.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.594+0000 7f53bd7ed740 1 -- 172.21.15.1:0/3223428247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532cbf8a0 msgr2=0x565532cd3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.594+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/3223428247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532cbf8a0 0x565532cd3ab0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00fa30 tx=0x7f53ac001330 comp rx=0 tx=0).stop 2024-03-06T07:28:17.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.594+0000 7f53bd7ed740 1 -- 172.21.15.1:0/3223428247 shutdown_connections 2024-03-06T07:28:17.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.594+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/3223428247 >> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] conn(0x565532ceb8b0 0x565532cf66d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.594+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/3223428247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532cbf8a0 0x565532cd3ab0 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.594+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/3223428247 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532c9b4e0 0x565532cbbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.594+0000 7f53937fe640 0 client.10203 ms_handle_reset on v2:172.21.15.1:6832/3895143314 2024-03-06T07:28:17.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.594+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/3223428247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b100 0x565532ce8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.594+0000 7f53bd7ed740 1 -- 172.21.15.1:0/3223428247 >> 172.21.15.1:0/3223428247 conn(0x565532c99660 msgr2=0x565532c961e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.594+0000 7f53bd7ed740 1 -- 172.21.15.1:0/3223428247 shutdown_connections 2024-03-06T07:28:17.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.594+0000 7f53bd7ed740 1 -- 172.21.15.1:0/3223428247 wait complete. 2024-03-06T07:28:17.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.594+0000 7f53bd7ed740 20 client.10203 trim_cache size 0 max 16384 2024-03-06T07:28:17.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.611+0000 7f53bd7ed740 1 Processor -- start 2024-03-06T07:28:17.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.611+0000 7f53bd7ed740 1 -- start start 2024-03-06T07:28:17.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.611+0000 7f53bd7ed740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b100 0x565532c94c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.611+0000 7f53bd7ed740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c9b4e0 0x565532cdaf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.611+0000 7f53bd7ed740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532cbf8a0 0x565532cdb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.611+0000 7f53bd7ed740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b41f7df0 con 0x565532c9b4e0 2024-03-06T07:28:17.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.611+0000 7f53bd7ed740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b41f6390 con 0x565532c9b100 2024-03-06T07:28:17.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.611+0000 7f53bd7ed740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b41f6930 con 0x565532cbf8a0 2024-03-06T07:28:17.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.612+0000 7f53b3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c9b4e0 0x565532cdaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.612+0000 7f53b3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c9b4e0 0x565532cdaf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50842/0 (socket says 172.21.15.1:50842) 2024-03-06T07:28:17.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.612+0000 7f53b3fff640 1 -- 172.21.15.1:0/1890194027 learned_addr learned my addr 172.21.15.1:0/1890194027 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:17.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.612+0000 7f53b3fff640 1 -- 172.21.15.1:0/1890194027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532cbf8a0 msgr2=0x565532cdb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.612+0000 7f53b3fff640 1 --2- 172.21.15.1:0/1890194027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532cbf8a0 0x565532cdb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.612+0000 7f53b3fff640 1 -- 172.21.15.1:0/1890194027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b100 msgr2=0x565532c94c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.612+0000 7f53b899e640 1 --2- 172.21.15.1:0/1890194027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b100 0x565532c94c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.612+0000 7f53b3fff640 1 --2- 172.21.15.1:0/1890194027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b100 0x565532c94c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.612+0000 7f53b3fff640 1 -- 172.21.15.1:0/1890194027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5655329de6b0 con 0x565532c9b4e0 2024-03-06T07:28:17.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.612+0000 7f53b899e640 1 --2- 172.21.15.1:0/1890194027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b100 0x565532c94c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:17.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.612+0000 7f53b3fff640 1 --2- 172.21.15.1:0/1890194027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c9b4e0 0x565532cdaf80 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f53a007a7e0 tx=0x7f53a00ac040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a37b79a9710854c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.612+0000 7f539dffb640 1 -- 172.21.15.1:0/1890194027 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a0083140 con 0x565532c9b4e0 2024-03-06T07:28:17.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.612+0000 7f539dffb640 1 -- 172.21.15.1:0/1890194027 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a0078b80 con 0x565532c9b4e0 2024-03-06T07:28:17.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.612+0000 7f539dffb640 1 -- 172.21.15.1:0/1890194027 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a0078de0 con 0x565532c9b4e0 2024-03-06T07:28:17.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.613+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1890194027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c9b4e0 msgr2=0x565532cdaf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.613+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/1890194027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c9b4e0 0x565532cdaf80 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f53a007a7e0 tx=0x7f53a00ac040 comp rx=0 tx=0).stop 2024-03-06T07:28:17.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.613+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1890194027 shutdown_connections 2024-03-06T07:28:17.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.613+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/1890194027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532cbf8a0 0x565532cdb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.613+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/1890194027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c9b4e0 0x565532cdaf80 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.613+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/1890194027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b100 0x565532c94c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.613+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1890194027 >> 172.21.15.1:0/1890194027 conn(0x565532c99660 msgr2=0x565532c99a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.613+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1890194027 shutdown_connections 2024-03-06T07:28:17.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.613+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1890194027 wait complete. 2024-03-06T07:28:17.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.613+0000 7f53bd7ed740 1 -- start start 2024-03-06T07:28:17.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.614+0000 7f53bd7ed740 1 Processor -- start 2024-03-06T07:28:17.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53bd7ed740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c9b100 0x565532ce3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53bd7ed740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b4e0 0x565532c98520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53bd7ed740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532cbf8a0 0x565532c95430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53bd7ed740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x565532cbdcb0 con 0x565532c9b100 2024-03-06T07:28:17.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53bd7ed740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x565532cd44c0 con 0x565532c9b4e0 2024-03-06T07:28:17.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53bd7ed740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x565532cbd8d0 con 0x565532cbf8a0 2024-03-06T07:28:17.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53bd7ed740 10 client.0 _ll_register_callbacks cb 0x7f53ac001dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:17.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53b899e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c9b100 0x565532ce3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53b899e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c9b100 0x565532ce3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50846/0 (socket says 172.21.15.1:50846) 2024-03-06T07:28:17.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53b899e640 1 -- 172.21.15.1:0/1059192795 learned_addr learned my addr 172.21.15.1:0/1059192795 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:17.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53b3fff640 1 --2- 172.21.15.1:0/1059192795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b4e0 0x565532c98520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53b919f640 1 --2- 172.21.15.1:0/1059192795 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532cbf8a0 0x565532c95430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53b899e640 1 -- 172.21.15.1:0/1059192795 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532cbf8a0 msgr2=0x565532c95430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53b899e640 1 --2- 172.21.15.1:0/1059192795 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532cbf8a0 0x565532c95430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53b899e640 1 -- 172.21.15.1:0/1059192795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b4e0 msgr2=0x565532c98520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53b899e640 1 --2- 172.21.15.1:0/1059192795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b4e0 0x565532c98520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53b899e640 1 -- 172.21.15.1:0/1059192795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565532c99120 con 0x565532c9b100 2024-03-06T07:28:17.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53b919f640 1 --2- 172.21.15.1:0/1059192795 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532cbf8a0 0x565532c95430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:17.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53b899e640 1 --2- 172.21.15.1:0/1059192795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c9b100 0x565532ce3eb0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x565532cbdc00 tx=0x7f53a8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f40c0e92b73bc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.615+0000 7f53937fe640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:28:17.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.616+0000 7f53937fe640 1 -- 172.21.15.1:0/1059192795 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a80291c0 con 0x565532c9b100 2024-03-06T07:28:17.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.616+0000 7f53937fe640 1 -- 172.21.15.1:0/1059192795 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f53a80133f0 con 0x565532c9b100 2024-03-06T07:28:17.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.616+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1059192795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f53a0078b80 con 0x565532c9b100 2024-03-06T07:28:17.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.616+0000 7f53937fe640 1 -- 172.21.15.1:0/1059192795 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=179) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53a8038050 con 0x565532c9b100 2024-03-06T07:28:17.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.616+0000 7f53bd7ed740 10 client.10209 fetch_fsmap learned FSMap version 179 2024-03-06T07:28:17.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.616+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1059192795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=179,monmap=2+}) v3 -- 0x565532d47a40 con 0x565532c9b100 2024-03-06T07:28:17.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.616+0000 7f53937fe640 1 -- 172.21.15.1:0/1059192795 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a0078b80 con 0x565532c9b100 2024-03-06T07:28:17.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.616+0000 7f53937fe640 1 -- 172.21.15.1:0/1059192795 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 179) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f53a803b020 con 0x565532c9b100 2024-03-06T07:28:17.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.616+0000 7f53bd7ed740 10 client.10209 fetch_fsmap finished waiting for FSMap version 179 2024-03-06T07:28:17.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.616+0000 7f53bd7ed740 10 client.10209 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:28:17.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.616+0000 7f53bd7ed740 20 client.10209 populate_metadata read hostname 'smithi001' 2024-03-06T07:28:17.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.616+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/1059192795 >> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] conn(0x565532d40d00 0x565532c95970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.616+0000 7f53bd7ed740 4 client.10209 mds_command: new command op to 9951 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:28:17.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.616+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1059192795 --> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] -- command(tid 0: {"prefix": "status"}) v1 -- 0x565532c06f40 con 0x565532d40d00 2024-03-06T07:28:17.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.616+0000 7f53b3fff640 1 --2- 172.21.15.1:0/1059192795 >> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] conn(0x565532d40d00 0x565532c95970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.617+0000 7f53937fe640 10 client.10209 ms_handle_connect on v2:172.21.15.1:6832/3895143314 2024-03-06T07:28:17.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.617+0000 7f53b3fff640 1 --2- 172.21.15.1:0/1059192795 >> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] conn(0x565532d40d00 0x565532c95970 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9da288d5bd037d19 server_cookie=2889f463cff03c24 in_seq=0 out_seq=0 2024-03-06T07:28:17.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.617+0000 7f53937fe640 1 -- 172.21.15.1:0/1059192795 <== mds.1 v2:172.21.15.1:6832/3895143314 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x565532c06f40 con 0x565532d40d00 2024-03-06T07:28:17.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.617+0000 7f53937fe640 10 client.10209 handle_command_reply: tid=0 2024-03-06T07:28:17.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.617+0000 7f53bd7ed740 1 client.10209 shutdown 2024-03-06T07:28:17.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.617+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1059192795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c9b100 msgr2=0x565532ce3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.617+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/1059192795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c9b100 0x565532ce3eb0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x565532cbdc00 tx=0x7f53a8000bc0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.617+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1059192795 shutdown_connections 2024-03-06T07:28:17.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.617+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/1059192795 >> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] conn(0x565532d40d00 0x565532c95970 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.617+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/1059192795 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x565532cbf8a0 0x565532c95430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.617+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/1059192795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x565532c9b4e0 0x565532c98520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.617+0000 7f53bd7ed740 1 --2- 172.21.15.1:0/1059192795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x565532c9b100 0x565532ce3eb0 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.617+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1059192795 >> 172.21.15.1:0/1059192795 conn(0x565532c99660 msgr2=0x565532c99a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.617+0000 7f53937fe640 0 client.10209 ms_handle_reset on v2:172.21.15.1:6832/3895143314 2024-03-06T07:28:17.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.617+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1059192795 shutdown_connections 2024-03-06T07:28:17.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.617+0000 7f53bd7ed740 1 -- 172.21.15.1:0/1059192795 wait complete. 2024-03-06T07:28:17.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.617+0000 7f53bd7ed740 20 client.10209 trim_cache size 0 max 16384 2024-03-06T07:28:17.625 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:28:17.625 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:28:17.625 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 1, 2024-03-06T07:28:17.625 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 9951, 2024-03-06T07:28:17.625 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:28:17.625 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:28:17.625 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:28:17.625 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 1.6707856489999999, 2024-03-06T07:28:17.625 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 179, 2024-03-06T07:28:17.625 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 185, 2024-03-06T07:28:17.626 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 185, 2024-03-06T07:28:17.626 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 29.877515560999999 2024-03-06T07:28:17.626 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:28:17.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.618+0000 7f53b99a0640 1 -- 172.21.15.1:0/2795391830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f539806b4d0 msgr2=0x7f539806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.618+0000 7f53b99a0640 1 --2- 172.21.15.1:0/2795391830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f539806b4d0 0x7f539806d990 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f53a8007fd0 tx=0x7f53a8007e80 comp rx=0 tx=0).stop 2024-03-06T07:28:17.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.618+0000 7f53b99a0640 1 -- 172.21.15.1:0/2795391830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53b40c6ee0 msgr2=0x7f53b40ca3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.618+0000 7f53b99a0640 1 --2- 172.21.15.1:0/2795391830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53b40c6ee0 0x7f53b40ca3a0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f53a002b7e0 tx=0x7f53a00046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.618+0000 7f53b99a0640 1 -- 172.21.15.1:0/2795391830 shutdown_connections 2024-03-06T07:28:17.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.618+0000 7f53b99a0640 1 --2- 172.21.15.1:0/2795391830 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f539806b4d0 0x7f539806d990 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.618+0000 7f53b99a0640 1 --2- 172.21.15.1:0/2795391830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b4152840 0x7f53b40ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.618+0000 7f53b99a0640 1 --2- 172.21.15.1:0/2795391830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53b40c6ee0 0x7f53b40ca3a0 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.618+0000 7f53b99a0640 1 --2- 172.21.15.1:0/2795391830 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53b40bf900 0x7f53b414a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.618+0000 7f53b99a0640 1 -- 172.21.15.1:0/2795391830 >> 172.21.15.1:0/2795391830 conn(0x7f53b4067520 msgr2=0x7f53b406d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.618+0000 7f53b99a0640 1 -- 172.21.15.1:0/2795391830 shutdown_connections 2024-03-06T07:28:17.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.618+0000 7f53b99a0640 1 -- 172.21.15.1:0/2795391830 wait complete. 2024-03-06T07:28:17.634 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:28:17.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.738+0000 7f7c20088640 1 Processor -- start 2024-03-06T07:28:17.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.738+0000 7f7c20088640 1 -- start start 2024-03-06T07:28:17.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.739+0000 7f7c20088640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c180c3970 0x7f7c180c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.739+0000 7f7c20088640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7c180c60b0 0x7f7c18152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.739+0000 7f7c20088640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c180c5e40 con 0x7f7c18152670 2024-03-06T07:28:17.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.739+0000 7f7c20088640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c18152ad0 con 0x7f7c180c60b0 2024-03-06T07:28:17.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.739+0000 7f7c20088640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c180c67f0 con 0x7f7c180c3970 2024-03-06T07:28:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.739+0000 7f7c1d623640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7c180c60b0 0x7f7c18152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.739+0000 7f7c1de24640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c180c3970 0x7f7c180c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.739+0000 7f7c1d623640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7c180c60b0 0x7f7c18152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:43396/0 (socket says 172.21.15.1:43396) 2024-03-06T07:28:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.739+0000 7f7c1d623640 1 -- 172.21.15.1:0/741773984 learned_addr learned my addr 172.21.15.1:0/741773984 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.739+0000 7f7c1de24640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c180c3970 0x7f7c180c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:41548/0 (socket says 172.21.15.1:41548) 2024-03-06T07:28:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.739+0000 7f7c1d623640 1 -- 172.21.15.1:0/741773984 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c180c3970 msgr2=0x7f7c180c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.739+0000 7f7c1d623640 1 --2- 172.21.15.1:0/741773984 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c180c3970 0x7f7c180c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.739+0000 7f7c1d623640 1 -- 172.21.15.1:0/741773984 >> v1:172.21.15.98:6789/0 conn(0x7f7c18152670 legacy=0x7f7c181553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.739+0000 7f7c1d623640 1 -- 172.21.15.1:0/741773984 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c180c6a40 con 0x7f7c180c60b0 2024-03-06T07:28:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.739+0000 7f7c1de24640 1 --2- 172.21.15.1:0/741773984 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c180c3970 0x7f7c180c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:17.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.741+0000 7f7c1d623640 1 --2- 172.21.15.1:0/741773984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c180c60b0 0x7f7c18152130 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009870 tx=0x7f7c0802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a8f7b26f980e7f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.741+0000 7f7c1ce22640 1 -- 172.21.15.1:0/741773984 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c08034070 con 0x7f7c180c60b0 2024-03-06T07:28:17.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.741+0000 7f7c1ce22640 1 -- 172.21.15.1:0/741773984 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c08030840 con 0x7f7c180c60b0 2024-03-06T07:28:17.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.741+0000 7f7c1ce22640 1 -- 172.21.15.1:0/741773984 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c08030c10 con 0x7f7c180c60b0 2024-03-06T07:28:17.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.741+0000 7f7c20088640 1 -- 172.21.15.1:0/741773984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c180c60b0 msgr2=0x7f7c18152130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.741+0000 7f7c20088640 1 --2- 172.21.15.1:0/741773984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c180c60b0 0x7f7c18152130 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009870 tx=0x7f7c0802b980 comp rx=0 tx=0).stop 2024-03-06T07:28:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.741+0000 7f7c20088640 1 -- 172.21.15.1:0/741773984 shutdown_connections 2024-03-06T07:28:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.741+0000 7f7c20088640 1 --2- 172.21.15.1:0/741773984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c180c60b0 0x7f7c18152130 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.741+0000 7f7c20088640 1 --2- 172.21.15.1:0/741773984 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c180c3970 0x7f7c180c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.741+0000 7f7c20088640 1 -- 172.21.15.1:0/741773984 >> 172.21.15.1:0/741773984 conn(0x7f7c180c88f0 msgr2=0x7f7c180c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.741+0000 7f7c20088640 1 -- 172.21.15.1:0/741773984 shutdown_connections 2024-03-06T07:28:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.741+0000 7f7c20088640 1 -- 172.21.15.1:0/741773984 wait complete. 2024-03-06T07:28:17.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.743+0000 7f7c20088640 1 Processor -- start 2024-03-06T07:28:17.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.743+0000 7f7c20088640 1 -- start start 2024-03-06T07:28:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.744+0000 7f7c20088640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c180c3970 0x7f7c1814d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.744+0000 7f7c20088640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180c60b0 0x7f7c181574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.744+0000 7f7c20088640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c18152670 0x7f7c180ad0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.744+0000 7f7c20088640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c180c72e0 con 0x7f7c180c60b0 2024-03-06T07:28:17.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.744+0000 7f7c20088640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c180c7000 con 0x7f7c18152670 2024-03-06T07:28:17.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.744+0000 7f7c20088640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c180c75c0 con 0x7f7c180c3970 2024-03-06T07:28:17.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.744+0000 7f7c1d623640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180c60b0 0x7f7c181574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.744+0000 7f7c1d623640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180c60b0 0x7f7c181574b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50856/0 (socket says 172.21.15.1:50856) 2024-03-06T07:28:17.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.744+0000 7f7c1d623640 1 -- 172.21.15.1:0/2954433965 learned_addr learned my addr 172.21.15.1:0/2954433965 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:17.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.745+0000 7f7c1de24640 1 --2- 172.21.15.1:0/2954433965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c180c3970 0x7f7c1814d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.745+0000 7f7c1e625640 1 --2- 172.21.15.1:0/2954433965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c18152670 0x7f7c180ad0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.745+0000 7f7c1d623640 1 -- 172.21.15.1:0/2954433965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c180c3970 msgr2=0x7f7c1814d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.745+0000 7f7c1d623640 1 --2- 172.21.15.1:0/2954433965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c180c3970 0x7f7c1814d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.745+0000 7f7c1d623640 1 -- 172.21.15.1:0/2954433965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c18152670 msgr2=0x7f7c180ad0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.745+0000 7f7c1d623640 1 --2- 172.21.15.1:0/2954433965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c18152670 0x7f7c180ad0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.745+0000 7f7c1d623640 1 -- 172.21.15.1:0/2954433965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c181eb990 con 0x7f7c180c60b0 2024-03-06T07:28:17.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.745+0000 7f7c1d623640 1 --2- 172.21.15.1:0/2954433965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180c60b0 0x7f7c181574b0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f7c0802b9c0 tx=0x7f7c08004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.745+0000 7f7c0e7fc640 1 -- 172.21.15.1:0/2954433965 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c08048070 con 0x7f7c180c60b0 2024-03-06T07:28:17.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.745+0000 7f7c0e7fc640 1 -- 172.21.15.1:0/2954433965 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c08030640 con 0x7f7c180c60b0 2024-03-06T07:28:17.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.745+0000 7f7c20088640 1 -- 172.21.15.1:0/2954433965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c180acc10 con 0x7f7c180c60b0 2024-03-06T07:28:17.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.745+0000 7f7c0e7fc640 1 -- 172.21.15.1:0/2954433965 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c08034040 con 0x7f7c180c60b0 2024-03-06T07:28:17.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.745+0000 7f7c20088640 1 -- 172.21.15.1:0/2954433965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c181eb6b0 con 0x7f7c180c60b0 2024-03-06T07:28:17.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.747+0000 7f7c0e7fc640 1 -- 172.21.15.1:0/2954433965 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c0804b070 con 0x7f7c180c60b0 2024-03-06T07:28:17.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.748+0000 7f7c0e7fc640 1 --2- 172.21.15.1:0/2954433965 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bfc06b4d0 0x7f7bfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.748+0000 7f7c0e7fc640 1 -- 172.21.15.1:0/2954433965 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f7c080ac1a0 con 0x7f7c180c60b0 2024-03-06T07:28:17.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.748+0000 7f7c22c75740 1 Processor -- start 2024-03-06T07:28:17.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.748+0000 7f7c1de24640 1 --2- 172.21.15.1:0/2954433965 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bfc06b4d0 0x7f7bfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.748+0000 7f7c22c75740 1 -- start start 2024-03-06T07:28:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.749+0000 7f7c1de24640 1 --2- 172.21.15.1:0/2954433965 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bfc06b4d0 0x7f7bfc06d990 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f7c180a9ca0 tx=0x7f7c14009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.749+0000 7f7c22c75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9b1f50 0x55e43b9afa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.749+0000 7f7c22c75740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9b2330 0x55e43b9aff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.749+0000 7f7c22c75740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b04b0 0x55e43b9ac680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.749+0000 7f7c22c75740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e43b9166b0 con 0x55e43b9b04b0 2024-03-06T07:28:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.749+0000 7f7c22c75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e43b8bfa60 con 0x55e43b9b1f50 2024-03-06T07:28:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.749+0000 7f7c22c75740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e43b912750 con 0x55e43b9b2330 2024-03-06T07:28:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.749+0000 7f7c1e625640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b04b0 0x55e43b9ac680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.749+0000 7f7c1e625640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b04b0 0x55e43b9ac680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50872/0 (socket says 172.21.15.1:50872) 2024-03-06T07:28:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.749+0000 7f7c1e625640 1 -- 172.21.15.1:0/4294620444 learned_addr learned my addr 172.21.15.1:0/4294620444 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.749+0000 7f7c1de24640 1 --2- 172.21.15.1:0/4294620444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9b1f50 0x55e43b9afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.749+0000 7f7c1d623640 1 --2- 172.21.15.1:0/4294620444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9b2330 0x55e43b9aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.749+0000 7f7c1e625640 1 -- 172.21.15.1:0/4294620444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9b2330 msgr2=0x55e43b9aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.749+0000 7f7c1e625640 1 --2- 172.21.15.1:0/4294620444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9b2330 0x55e43b9aff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.749+0000 7f7c1e625640 1 -- 172.21.15.1:0/4294620444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9b1f50 msgr2=0x55e43b9afa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.749+0000 7f7c1e625640 1 --2- 172.21.15.1:0/4294620444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9b1f50 0x55e43b9afa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.749+0000 7f7c1e625640 1 -- 172.21.15.1:0/4294620444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e43b8ddf30 con 0x55e43b9b04b0 2024-03-06T07:28:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.750+0000 7f7c1de24640 1 --2- 172.21.15.1:0/4294620444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9b1f50 0x55e43b9afa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.750+0000 7f7c1e625640 1 --2- 172.21.15.1:0/4294620444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b04b0 0x55e43b9ac680 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f7c10001e90 tx=0x7f7c100015d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38876cc5a3d003e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.750+0000 7f7bfb7fe640 1 -- 172.21.15.1:0/4294620444 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c100020a0 con 0x55e43b9b04b0 2024-03-06T07:28:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.750+0000 7f7bfb7fe640 1 -- 172.21.15.1:0/4294620444 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c10018070 con 0x55e43b9b04b0 2024-03-06T07:28:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.750+0000 7f7bfb7fe640 1 -- 172.21.15.1:0/4294620444 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10003b70 con 0x55e43b9b04b0 2024-03-06T07:28:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.750+0000 7f7c22c75740 1 -- 172.21.15.1:0/4294620444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b04b0 msgr2=0x55e43b9ac680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.750+0000 7f7c22c75740 1 --2- 172.21.15.1:0/4294620444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b04b0 0x55e43b9ac680 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f7c10001e90 tx=0x7f7c100015d0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.751+0000 7f7c22c75740 1 -- 172.21.15.1:0/4294620444 shutdown_connections 2024-03-06T07:28:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.751+0000 7f7c22c75740 1 --2- 172.21.15.1:0/4294620444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b04b0 0x55e43b9ac680 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.751+0000 7f7c22c75740 1 --2- 172.21.15.1:0/4294620444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9b2330 0x55e43b9aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.751+0000 7f7c22c75740 1 --2- 172.21.15.1:0/4294620444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9b1f50 0x55e43b9afa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.751+0000 7f7c22c75740 1 -- 172.21.15.1:0/4294620444 >> 172.21.15.1:0/4294620444 conn(0x55e43b9ae7d0 msgr2=0x55e43b9d33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.751+0000 7f7c22c75740 1 -- 172.21.15.1:0/4294620444 shutdown_connections 2024-03-06T07:28:17.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.751+0000 7f7c22c75740 1 -- 172.21.15.1:0/4294620444 wait complete. 2024-03-06T07:28:17.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.751+0000 7f7c22c75740 1 -- start start 2024-03-06T07:28:17.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.751+0000 7f7c22c75740 1 Processor -- start 2024-03-06T07:28:17.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.752+0000 7f7c22c75740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b1f50 0x55e43b9d2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.752+0000 7f7c22c75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9b2330 0x55e43b9d6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.752+0000 7f7c22c75740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9ea9c0 0x55e43b9eae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.752+0000 7f7c22c75740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e43b9d4230 con 0x55e43b9b1f50 2024-03-06T07:28:17.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.752+0000 7f7c22c75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e43b8d2ea0 con 0x55e43b9b2330 2024-03-06T07:28:17.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.752+0000 7f7c22c75740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e43b9d3c70 con 0x55e43b9ea9c0 2024-03-06T07:28:17.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.752+0000 7f7c22c75740 10 client.0 _ll_register_callbacks cb 0x55e43b8d2870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:17.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.752+0000 7f7c1de24640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b1f50 0x55e43b9d2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.752+0000 7f7c1de24640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b1f50 0x55e43b9d2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50884/0 (socket says 172.21.15.1:50884) 2024-03-06T07:28:17.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.752+0000 7f7c1de24640 1 -- 172.21.15.1:0/2109555027 learned_addr learned my addr 172.21.15.1:0/2109555027 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:17.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.752+0000 7f7c1d623640 1 --2- 172.21.15.1:0/2109555027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9b2330 0x55e43b9d6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.752+0000 7f7c1e625640 1 --2- 172.21.15.1:0/2109555027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9ea9c0 0x55e43b9eae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7c1de24640 1 -- 172.21.15.1:0/2109555027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9ea9c0 msgr2=0x55e43b9eae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7c1de24640 1 --2- 172.21.15.1:0/2109555027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9ea9c0 0x55e43b9eae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7c1de24640 1 -- 172.21.15.1:0/2109555027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9b2330 msgr2=0x55e43b9d6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7c1de24640 1 --2- 172.21.15.1:0/2109555027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9b2330 0x55e43b9d6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7c1de24640 1 -- 172.21.15.1:0/2109555027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e43b8d3270 con 0x55e43b9b1f50 2024-03-06T07:28:17.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7c1d623640 1 --2- 172.21.15.1:0/2109555027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9b2330 0x55e43b9d6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:17.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7c1de24640 1 --2- 172.21.15.1:0/2109555027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b1f50 0x55e43b9d2a60 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f7c140045b0 tx=0x7f7c140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99466b57d74b7834 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7bf8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:28:17.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7bf8ff9640 1 -- 172.21.15.1:0/2109555027 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c14026070 con 0x55e43b9b1f50 2024-03-06T07:28:17.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7bf8ff9640 1 -- 172.21.15.1:0/2109555027 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f7c14014440 con 0x55e43b9b1f50 2024-03-06T07:28:17.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7c22c75740 1 -- 172.21.15.1:0/2109555027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7c10018070 con 0x55e43b9b1f50 2024-03-06T07:28:17.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7bf8ff9640 1 -- 172.21.15.1:0/2109555027 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=179) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c14014640 con 0x55e43b9b1f50 2024-03-06T07:28:17.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7c22c75740 10 client.10218 fetch_fsmap learned FSMap version 179 2024-03-06T07:28:17.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7c22c75740 1 -- 172.21.15.1:0/2109555027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=179,monmap=2+}) v3 -- 0x55e43b62b2d0 con 0x55e43b9b1f50 2024-03-06T07:28:17.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7bf8ff9640 1 -- 172.21.15.1:0/2109555027 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c10018070 con 0x55e43b9b1f50 2024-03-06T07:28:17.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7bf8ff9640 1 -- 172.21.15.1:0/2109555027 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 179) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f7c14034080 con 0x55e43b9b1f50 2024-03-06T07:28:17.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7c22c75740 10 client.10218 fetch_fsmap finished waiting for FSMap version 179 2024-03-06T07:28:17.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7c22c75740 10 client.10218 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:28:17.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7c22c75740 20 client.10218 populate_metadata read hostname 'smithi001' 2024-03-06T07:28:17.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.753+0000 7f7c22c75740 1 --2- 172.21.15.1:0/2109555027 >> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] conn(0x55e43b9fe970 0x55e43ba0dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.754+0000 7f7c22c75740 4 client.10218 mds_command: new command op to 10161 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:28:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.754+0000 7f7c22c75740 1 -- 172.21.15.1:0/2109555027 --> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e43b923bf0 con 0x55e43b9fe970 2024-03-06T07:28:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.754+0000 7f7c1d623640 1 --2- 172.21.15.1:0/2109555027 >> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] conn(0x55e43b9fe970 0x55e43ba0dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.754+0000 7f7bf8ff9640 10 client.10218 ms_handle_connect on v2:172.21.15.1:6834/1525330395 2024-03-06T07:28:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.754+0000 7f7c1d623640 1 --2- 172.21.15.1:0/2109555027 >> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] conn(0x55e43b9fe970 0x55e43ba0dad0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=565038b1803496f1 server_cookie=605936f580eb2e8a in_seq=0 out_seq=0 2024-03-06T07:28:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.754+0000 7f7bf8ff9640 1 -- 172.21.15.1:0/2109555027 <== mds.0 v2:172.21.15.1:6834/1525330395 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e43b923bf0 con 0x55e43b9fe970 2024-03-06T07:28:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.754+0000 7f7bf8ff9640 10 client.10218 handle_command_reply: tid=0 2024-03-06T07:28:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.755+0000 7f7c22c75740 1 client.10218 shutdown 2024-03-06T07:28:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.755+0000 7f7c22c75740 1 -- 172.21.15.1:0/2109555027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b1f50 msgr2=0x55e43b9d2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.755+0000 7f7c22c75740 1 --2- 172.21.15.1:0/2109555027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b1f50 0x55e43b9d2a60 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f7c140045b0 tx=0x7f7c140046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.755+0000 7f7c22c75740 1 -- 172.21.15.1:0/2109555027 shutdown_connections 2024-03-06T07:28:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.755+0000 7f7c22c75740 1 --2- 172.21.15.1:0/2109555027 >> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] conn(0x55e43b9fe970 0x55e43ba0dad0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.755+0000 7f7c22c75740 1 --2- 172.21.15.1:0/2109555027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9ea9c0 0x55e43b9eae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.755+0000 7f7c22c75740 1 --2- 172.21.15.1:0/2109555027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9b2330 0x55e43b9d6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.755+0000 7f7bf8ff9640 0 client.10218 ms_handle_reset on v2:172.21.15.1:6834/1525330395 2024-03-06T07:28:17.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.755+0000 7f7c22c75740 1 --2- 172.21.15.1:0/2109555027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b1f50 0x55e43b9d2a60 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.755+0000 7f7c22c75740 1 -- 172.21.15.1:0/2109555027 >> 172.21.15.1:0/2109555027 conn(0x55e43b9b04b0 msgr2=0x55e43b9b08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.755+0000 7f7c22c75740 1 -- 172.21.15.1:0/2109555027 shutdown_connections 2024-03-06T07:28:17.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.755+0000 7f7c22c75740 1 -- 172.21.15.1:0/2109555027 wait complete. 2024-03-06T07:28:17.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.755+0000 7f7c22c75740 20 client.10218 trim_cache size 0 max 16384 2024-03-06T07:28:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.772+0000 7f7c22c75740 1 Processor -- start 2024-03-06T07:28:17.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.772+0000 7f7c22c75740 1 -- start start 2024-03-06T07:28:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.772+0000 7f7c22c75740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b2330 0x55e43b9ab510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.772+0000 7f7c22c75740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9ea9c0 0x55e43b9d6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.772+0000 7f7c22c75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9fe970 0x55e43b9d67a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.772+0000 7f7c22c75740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c181f7930 con 0x55e43b9b2330 2024-03-06T07:28:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.772+0000 7f7c22c75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c181f7f90 con 0x55e43b9fe970 2024-03-06T07:28:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.772+0000 7f7c22c75740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c181f64d0 con 0x55e43b9ea9c0 2024-03-06T07:28:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.773+0000 7f7c1de24640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b2330 0x55e43b9ab510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.773+0000 7f7c1de24640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b2330 0x55e43b9ab510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50900/0 (socket says 172.21.15.1:50900) 2024-03-06T07:28:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.773+0000 7f7c1de24640 1 -- 172.21.15.1:0/1283488814 learned_addr learned my addr 172.21.15.1:0/1283488814 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.773+0000 7f7c1de24640 1 -- 172.21.15.1:0/1283488814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9ea9c0 msgr2=0x55e43b9d6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.773+0000 7f7c1de24640 1 --2- 172.21.15.1:0/1283488814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9ea9c0 0x55e43b9d6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.773+0000 7f7c1de24640 1 -- 172.21.15.1:0/1283488814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9fe970 msgr2=0x55e43b9d67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.773+0000 7f7c1e625640 1 --2- 172.21.15.1:0/1283488814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9fe970 0x55e43b9d67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.773+0000 7f7c1de24640 1 --2- 172.21.15.1:0/1283488814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9fe970 0x55e43b9d67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.773+0000 7f7c1de24640 1 -- 172.21.15.1:0/1283488814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e43b9eb1b0 con 0x55e43b9b2330 2024-03-06T07:28:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.773+0000 7f7c1de24640 1 --2- 172.21.15.1:0/1283488814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b2330 0x55e43b9ab510 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f7c14006da0 tx=0x7f7c1401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e24673dd75bf916 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.773+0000 7f7bfaffd640 1 -- 172.21.15.1:0/1283488814 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c14026040 con 0x55e43b9b2330 2024-03-06T07:28:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.773+0000 7f7bfaffd640 1 -- 172.21.15.1:0/1283488814 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1401b030 con 0x55e43b9b2330 2024-03-06T07:28:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.773+0000 7f7bfaffd640 1 -- 172.21.15.1:0/1283488814 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c14014c80 con 0x55e43b9b2330 2024-03-06T07:28:17.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.774+0000 7f7c22c75740 1 -- 172.21.15.1:0/1283488814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b2330 msgr2=0x55e43b9ab510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.774+0000 7f7c22c75740 1 --2- 172.21.15.1:0/1283488814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b2330 0x55e43b9ab510 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f7c14006da0 tx=0x7f7c1401d000 comp rx=0 tx=0).stop 2024-03-06T07:28:17.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.774+0000 7f7c22c75740 1 -- 172.21.15.1:0/1283488814 shutdown_connections 2024-03-06T07:28:17.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.774+0000 7f7c22c75740 1 --2- 172.21.15.1:0/1283488814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9fe970 0x55e43b9d67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.774+0000 7f7c22c75740 1 --2- 172.21.15.1:0/1283488814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9ea9c0 0x55e43b9d6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.774+0000 7f7c22c75740 1 --2- 172.21.15.1:0/1283488814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9b2330 0x55e43b9ab510 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.774+0000 7f7c22c75740 1 -- 172.21.15.1:0/1283488814 >> 172.21.15.1:0/1283488814 conn(0x55e43b9b04b0 msgr2=0x55e43b9d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.774+0000 7f7c22c75740 1 -- 172.21.15.1:0/1283488814 shutdown_connections 2024-03-06T07:28:17.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.774+0000 7f7c22c75740 1 -- 172.21.15.1:0/1283488814 wait complete. 2024-03-06T07:28:17.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.775+0000 7f7c22c75740 1 -- start start 2024-03-06T07:28:17.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.775+0000 7f7c22c75740 1 Processor -- start 2024-03-06T07:28:17.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.776+0000 7f7c22c75740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9b2330 0x55e43b9d2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.776+0000 7f7c22c75740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9ea9c0 0x55e43ba61810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.776+0000 7f7c22c75740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9fe970 0x55e43ba5b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.776+0000 7f7c22c75740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e43b9d7030 con 0x55e43b9ea9c0 2024-03-06T07:28:17.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.776+0000 7f7c22c75740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c181f6a80 con 0x55e43b9fe970 2024-03-06T07:28:17.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.776+0000 7f7c22c75740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e43ba02cf0 con 0x55e43b9b2330 2024-03-06T07:28:17.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.776+0000 7f7c22c75740 10 client.0 _ll_register_callbacks cb 0x7f7c14011c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:17.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.776+0000 7f7c1d623640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9ea9c0 0x55e43ba61810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.776+0000 7f7c1d623640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9ea9c0 0x55e43ba61810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50908/0 (socket says 172.21.15.1:50908) 2024-03-06T07:28:17.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.776+0000 7f7c1d623640 1 -- 172.21.15.1:0/3370207393 learned_addr learned my addr 172.21.15.1:0/3370207393 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:17.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.776+0000 7f7c1d623640 1 -- 172.21.15.1:0/3370207393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9b2330 msgr2=0x55e43b9d2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.776+0000 7f7c1d623640 1 --2- 172.21.15.1:0/3370207393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9b2330 0x55e43b9d2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.776+0000 7f7c1d623640 1 -- 172.21.15.1:0/3370207393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9fe970 msgr2=0x55e43ba5b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.776+0000 7f7c1d623640 1 --2- 172.21.15.1:0/3370207393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9fe970 0x55e43ba5b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.776+0000 7f7c1d623640 1 -- 172.21.15.1:0/3370207393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e43b6f56b0 con 0x55e43b9ea9c0 2024-03-06T07:28:17.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.777+0000 7f7c1d623640 1 --2- 172.21.15.1:0/3370207393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9ea9c0 0x55e43ba61810 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f7c08000c00 tx=0x7f7c08039b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881052e92f278976 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.777+0000 7f7bf8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:28:17.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.777+0000 7f7bf8ff9640 1 -- 172.21.15.1:0/3370207393 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c08092070 con 0x55e43b9ea9c0 2024-03-06T07:28:17.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.777+0000 7f7bf8ff9640 1 -- 172.21.15.1:0/3370207393 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f7c08078ad0 con 0x55e43b9ea9c0 2024-03-06T07:28:17.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.777+0000 7f7c22c75740 1 -- 172.21.15.1:0/3370207393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7c1401b030 con 0x55e43b9ea9c0 2024-03-06T07:28:17.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.777+0000 7f7bf8ff9640 1 -- 172.21.15.1:0/3370207393 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=179) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c0807e870 con 0x55e43b9ea9c0 2024-03-06T07:28:17.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.777+0000 7f7c22c75740 10 client.10224 fetch_fsmap learned FSMap version 179 2024-03-06T07:28:17.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.777+0000 7f7c22c75740 1 -- 172.21.15.1:0/3370207393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=179,monmap=2+}) v3 -- 0x55e43b9ac3b0 con 0x55e43b9ea9c0 2024-03-06T07:28:17.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.777+0000 7f7bf8ff9640 1 -- 172.21.15.1:0/3370207393 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1401b030 con 0x55e43b9ea9c0 2024-03-06T07:28:17.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.777+0000 7f7bf8ff9640 1 -- 172.21.15.1:0/3370207393 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 179) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f7c080aa080 con 0x55e43b9ea9c0 2024-03-06T07:28:17.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.777+0000 7f7c22c75740 10 client.10224 fetch_fsmap finished waiting for FSMap version 179 2024-03-06T07:28:17.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.777+0000 7f7c22c75740 10 client.10224 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:28:17.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.777+0000 7f7c22c75740 20 client.10224 populate_metadata read hostname 'smithi001' 2024-03-06T07:28:17.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.777+0000 7f7c22c75740 1 --2- 172.21.15.1:0/3370207393 >> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] conn(0x55e43ba5bc70 0x55e43ba5e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.777+0000 7f7c22c75740 4 client.10224 mds_command: new command op to 10161 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:28:17.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.777+0000 7f7c22c75740 1 -- 172.21.15.1:0/3370207393 --> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e43b9d3f40 con 0x55e43ba5bc70 2024-03-06T07:28:17.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.777+0000 7f7c1de24640 1 --2- 172.21.15.1:0/3370207393 >> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] conn(0x55e43ba5bc70 0x55e43ba5e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.778+0000 7f7bf8ff9640 10 client.10224 ms_handle_connect on v2:172.21.15.1:6834/1525330395 2024-03-06T07:28:17.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.778+0000 7f7c1de24640 1 --2- 172.21.15.1:0/3370207393 >> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] conn(0x55e43ba5bc70 0x55e43ba5e2a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49b3163ef227fd68 server_cookie=f0a363132af12576 in_seq=0 out_seq=0 2024-03-06T07:28:17.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.778+0000 7f7bf8ff9640 1 -- 172.21.15.1:0/3370207393 <== mds.0 v2:172.21.15.1:6834/1525330395 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55e43b9d3f40 con 0x55e43ba5bc70 2024-03-06T07:28:17.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.778+0000 7f7bf8ff9640 10 client.10224 handle_command_reply: tid=0 2024-03-06T07:28:17.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.778+0000 7f7c22c75740 1 client.10224 shutdown 2024-03-06T07:28:17.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.778+0000 7f7c22c75740 1 -- 172.21.15.1:0/3370207393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9ea9c0 msgr2=0x55e43ba61810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.778+0000 7f7c22c75740 1 --2- 172.21.15.1:0/3370207393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9ea9c0 0x55e43ba61810 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f7c08000c00 tx=0x7f7c08039b00 comp rx=0 tx=0).stop 2024-03-06T07:28:17.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.778+0000 7f7c22c75740 1 -- 172.21.15.1:0/3370207393 shutdown_connections 2024-03-06T07:28:17.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.778+0000 7f7c22c75740 1 --2- 172.21.15.1:0/3370207393 >> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] conn(0x55e43ba5bc70 0x55e43ba5e2a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.778+0000 7f7c22c75740 1 --2- 172.21.15.1:0/3370207393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e43b9fe970 0x55e43ba5b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.778+0000 7f7c22c75740 1 --2- 172.21.15.1:0/3370207393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e43b9ea9c0 0x55e43ba61810 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.778+0000 7f7c22c75740 1 --2- 172.21.15.1:0/3370207393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55e43b9b2330 0x55e43b9d2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.778+0000 7f7c22c75740 1 -- 172.21.15.1:0/3370207393 >> 172.21.15.1:0/3370207393 conn(0x55e43b9b04b0 msgr2=0x55e43b9afbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.778+0000 7f7bf8ff9640 0 client.10224 ms_handle_reset on v2:172.21.15.1:6834/1525330395 2024-03-06T07:28:17.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.778+0000 7f7c22c75740 1 -- 172.21.15.1:0/3370207393 shutdown_connections 2024-03-06T07:28:17.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.778+0000 7f7c22c75740 1 -- 172.21.15.1:0/3370207393 wait complete. 2024-03-06T07:28:17.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.778+0000 7f7c22c75740 20 client.10224 trim_cache size 0 max 16384 2024-03-06T07:28:17.786 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:28:17.786 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:28:17.786 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:28:17.786 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 10161, 2024-03-06T07:28:17.786 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:28:17.786 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:28:17.787 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:28:17.787 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 2.8387659190000001, 2024-03-06T07:28:17.787 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 179, 2024-03-06T07:28:17.787 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 185, 2024-03-06T07:28:17.787 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 185, 2024-03-06T07:28:17.787 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 4.8375281530000001 2024-03-06T07:28:17.787 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:28:17.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.779+0000 7f7c20088640 1 -- 172.21.15.1:0/2954433965 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bfc06b4d0 msgr2=0x7f7bfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.779+0000 7f7c20088640 1 --2- 172.21.15.1:0/2954433965 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bfc06b4d0 0x7f7bfc06d990 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f7c180a9ca0 tx=0x7f7c14009190 comp rx=0 tx=0).stop 2024-03-06T07:28:17.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.779+0000 7f7c20088640 1 -- 172.21.15.1:0/2954433965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180c60b0 msgr2=0x7f7c181574b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.779+0000 7f7c20088640 1 --2- 172.21.15.1:0/2954433965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180c60b0 0x7f7c181574b0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f7c0802b9c0 tx=0x7f7c08004730 comp rx=0 tx=0).stop 2024-03-06T07:28:17.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.779+0000 7f7c20088640 1 -- 172.21.15.1:0/2954433965 shutdown_connections 2024-03-06T07:28:17.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.779+0000 7f7c20088640 1 --2- 172.21.15.1:0/2954433965 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7bfc06b4d0 0x7f7bfc06d990 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.779+0000 7f7c20088640 1 --2- 172.21.15.1:0/2954433965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c18152670 0x7f7c180ad0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.779+0000 7f7c20088640 1 --2- 172.21.15.1:0/2954433965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180c60b0 0x7f7c181574b0 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.779+0000 7f7c20088640 1 --2- 172.21.15.1:0/2954433965 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7c180c3970 0x7f7c1814d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.779+0000 7f7c20088640 1 -- 172.21.15.1:0/2954433965 >> 172.21.15.1:0/2954433965 conn(0x7f7c180c88f0 msgr2=0x7f7c180c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.779+0000 7f7c20088640 1 -- 172.21.15.1:0/2954433965 shutdown_connections 2024-03-06T07:28:17.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.779+0000 7f7c20088640 1 -- 172.21.15.1:0/2954433965 wait complete. 2024-03-06T07:28:17.792 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:17.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.894+0000 7f67b2c08640 1 Processor -- start 2024-03-06T07:28:17.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.894+0000 7f67b2c08640 1 -- start start 2024-03-06T07:28:17.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.895+0000 7f67b2c08640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67ac0c6b10 0x7f67ac155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.895+0000 7f67b2c08640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f67ac0bf9d0 0x7f67ac0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.895+0000 7f67b2c08640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67ac0c2fc0 con 0x7f67ac152650 2024-03-06T07:28:17.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.895+0000 7f67b2c08640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67ac155800 con 0x7f67ac0bf9d0 2024-03-06T07:28:17.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.895+0000 7f67b2c08640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac158da0 con 0x7f67ac0c6b10 2024-03-06T07:28:17.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.895+0000 7f67b11a5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f67ac0bf9d0 0x7f67ac0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.895+0000 7f67b09a4640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f67ac152650 0x7f67ac158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39742/0 (socket says 172.21.15.1:39742) 2024-03-06T07:28:17.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.895+0000 7f67b09a4640 1 -- 172.21.15.1:0/1041000679 learned_addr learned my addr 172.21.15.1:0/1041000679 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:17.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.895+0000 7f67abfff640 1 --2- 172.21.15.1:0/1041000679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67ac0c6b10 0x7f67ac155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.895+0000 7f67b11a5640 1 -- 172.21.15.1:0/1041000679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67ac0c6b10 msgr2=0x7f67ac155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.895+0000 7f67b11a5640 1 --2- 172.21.15.1:0/1041000679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67ac0c6b10 0x7f67ac155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.895+0000 7f67b11a5640 1 -- 172.21.15.1:0/1041000679 >> v1:172.21.15.1:6789/0 conn(0x7f67ac152650 legacy=0x7f67ac158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.895+0000 7f67b11a5640 1 -- 172.21.15.1:0/1041000679 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ac0c6690 con 0x7f67ac0bf9d0 2024-03-06T07:28:17.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.896+0000 7f67b11a5640 1 --2- 172.21.15.1:0/1041000679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67ac0bf9d0 0x7f67ac0c1460 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f67a4009d90 tx=0x7f67a402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c2f00d9730f462 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.896+0000 7f67ab7fe640 1 -- 172.21.15.1:0/1041000679 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a4039070 con 0x7f67ac0bf9d0 2024-03-06T07:28:17.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.896+0000 7f67ab7fe640 1 -- 172.21.15.1:0/1041000679 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a402cbc0 con 0x7f67ac0bf9d0 2024-03-06T07:28:17.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.896+0000 7f67ab7fe640 1 -- 172.21.15.1:0/1041000679 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a4034440 con 0x7f67ac0bf9d0 2024-03-06T07:28:17.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.897+0000 7f67b2c08640 1 -- 172.21.15.1:0/1041000679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67ac0bf9d0 msgr2=0x7f67ac0c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.897+0000 7f67b2c08640 1 --2- 172.21.15.1:0/1041000679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67ac0bf9d0 0x7f67ac0c1460 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f67a4009d90 tx=0x7f67a402be10 comp rx=0 tx=0).stop 2024-03-06T07:28:17.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.897+0000 7f67b2c08640 1 -- 172.21.15.1:0/1041000679 shutdown_connections 2024-03-06T07:28:17.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.897+0000 7f67b2c08640 1 --2- 172.21.15.1:0/1041000679 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67ac0c6b10 0x7f67ac155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.897+0000 7f67b2c08640 1 --2- 172.21.15.1:0/1041000679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67ac0bf9d0 0x7f67ac0c1460 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.897+0000 7f67b2c08640 1 -- 172.21.15.1:0/1041000679 >> 172.21.15.1:0/1041000679 conn(0x7f67ac066ee0 msgr2=0x7f67ac0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.897+0000 7f67b2c08640 1 -- 172.21.15.1:0/1041000679 shutdown_connections 2024-03-06T07:28:17.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.897+0000 7f67b2c08640 1 -- 172.21.15.1:0/1041000679 wait complete. 2024-03-06T07:28:17.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.898+0000 7f67b2c08640 1 Processor -- start 2024-03-06T07:28:17.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.898+0000 7f67b2c08640 1 -- start start 2024-03-06T07:28:17.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.899+0000 7f67b2c08640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67ac0bf9d0 0x7f67ac1eb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.899+0000 7f67b2c08640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67ac152650 0x7f67ac14ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.899+0000 7f67b2c08640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67ac14d2d0 0x7f67ac14d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.899+0000 7f67b2c08640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac1551b0 con 0x7f67ac0bf9d0 2024-03-06T07:28:17.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.899+0000 7f67b2c08640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac0c6fa0 con 0x7f67ac152650 2024-03-06T07:28:17.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.899+0000 7f67b2c08640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac155490 con 0x7f67ac14d2d0 2024-03-06T07:28:17.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.899+0000 7f67b09a4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67ac0bf9d0 0x7f67ac1eb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.899+0000 7f67b09a4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67ac0bf9d0 0x7f67ac1eb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50930/0 (socket says 172.21.15.1:50930) 2024-03-06T07:28:17.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.899+0000 7f67b09a4640 1 -- 172.21.15.1:0/1514916665 learned_addr learned my addr 172.21.15.1:0/1514916665 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:17.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.899+0000 7f67abfff640 1 --2- 172.21.15.1:0/1514916665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67ac152650 0x7f67ac14ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.899+0000 7f67b11a5640 1 --2- 172.21.15.1:0/1514916665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67ac14d2d0 0x7f67ac14d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.899+0000 7f67b09a4640 1 -- 172.21.15.1:0/1514916665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67ac14d2d0 msgr2=0x7f67ac14d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.899+0000 7f67b09a4640 1 --2- 172.21.15.1:0/1514916665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67ac14d2d0 0x7f67ac14d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.899+0000 7f67b09a4640 1 -- 172.21.15.1:0/1514916665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67ac152650 msgr2=0x7f67ac14ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.899+0000 7f67b09a4640 1 --2- 172.21.15.1:0/1514916665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67ac152650 0x7f67ac14ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.899+0000 7f67b09a4640 1 -- 172.21.15.1:0/1514916665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ac154a80 con 0x7f67ac0bf9d0 2024-03-06T07:28:17.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.899+0000 7f67b09a4640 1 --2- 172.21.15.1:0/1514916665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67ac0bf9d0 0x7f67ac1eb6f0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f67a000c430 tx=0x7f67a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.900+0000 7f67a97fa640 1 -- 172.21.15.1:0/1514916665 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a0017070 con 0x7f67ac0bf9d0 2024-03-06T07:28:17.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.900+0000 7f67a97fa640 1 -- 172.21.15.1:0/1514916665 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a0005ce0 con 0x7f67ac0bf9d0 2024-03-06T07:28:17.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.900+0000 7f67b2c08640 1 -- 172.21.15.1:0/1514916665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67ac1541e0 con 0x7f67ac0bf9d0 2024-03-06T07:28:17.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.900+0000 7f67a97fa640 1 -- 172.21.15.1:0/1514916665 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a00072f0 con 0x7f67ac0bf9d0 2024-03-06T07:28:17.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.900+0000 7f67b2c08640 1 -- 172.21.15.1:0/1514916665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67ac1547a0 con 0x7f67ac0bf9d0 2024-03-06T07:28:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.901+0000 7f67b2c08640 1 -- 172.21.15.1:0/1514916665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67ac067fd0 con 0x7f67ac0bf9d0 2024-03-06T07:28:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.901+0000 7f67a97fa640 1 -- 172.21.15.1:0/1514916665 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67a001a070 con 0x7f67ac0bf9d0 2024-03-06T07:28:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.902+0000 7f67a97fa640 1 --2- 172.21.15.1:0/1514916665 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f679006b4d0 0x7f679006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.902+0000 7f67a97fa640 1 -- 172.21.15.1:0/1514916665 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f67a0089780 con 0x7f67ac0bf9d0 2024-03-06T07:28:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.902+0000 7f67abfff640 1 --2- 172.21.15.1:0/1514916665 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f679006b4d0 0x7f679006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.902+0000 7f67abfff640 1 --2- 172.21.15.1:0/1514916665 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f679006b4d0 0x7f679006d990 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f67ac1ebc30 tx=0x7f6798008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:17.906+0000 7f67a97fa640 1 -- 172.21.15.1:0/1514916665 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67a0057da0 con 0x7f67ac0bf9d0 2024-03-06T07:28:18.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.127+0000 7f67b2c08640 1 -- 172.21.15.1:0/1514916665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67ac152f30 con 0x7f67ac0bf9d0 2024-03-06T07:28:18.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.128+0000 7f67a97fa640 1 -- 172.21.15.1:0/1514916665 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+5005 (secure 0 0 0) 0x7f67a005be40 con 0x7f67ac0bf9d0 2024-03-06T07:28:18.130 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:18.130 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":179,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:28:16.947047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10161,"mds_1":9951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9951":{"gid":9951,"name":"a","rank":1,"incarnation":178,"state":"up:active","state_seq":9,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10161":{"gid":10161,"name":"c","rank":0,"incarnation":176,"state":"up:active","state_seq":2,"addr":"172.21.15.1:6835/1525330395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1525330395},{"type":"v1","addr":"172.21.15.1:6835","nonce":1525330395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10161,"qdb_cluster":[9951,10161]},"id":14}]} 2024-03-06T07:28:18.130 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 179 2024-03-06T07:28:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.129+0000 7f67b2c08640 1 -- 172.21.15.1:0/1514916665 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f679006b4d0 msgr2=0x7f679006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.129+0000 7f67b2c08640 1 --2- 172.21.15.1:0/1514916665 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f679006b4d0 0x7f679006d990 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f67ac1ebc30 tx=0x7f6798008040 comp rx=0 tx=0).stop 2024-03-06T07:28:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.129+0000 7f67b2c08640 1 -- 172.21.15.1:0/1514916665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67ac0bf9d0 msgr2=0x7f67ac1eb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.129+0000 7f67b2c08640 1 --2- 172.21.15.1:0/1514916665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67ac0bf9d0 0x7f67ac1eb6f0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f67a000c430 tx=0x7f67a000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.130+0000 7f67b2c08640 1 -- 172.21.15.1:0/1514916665 shutdown_connections 2024-03-06T07:28:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.130+0000 7f67b2c08640 1 --2- 172.21.15.1:0/1514916665 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f679006b4d0 0x7f679006d990 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.130+0000 7f67b2c08640 1 --2- 172.21.15.1:0/1514916665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67ac14d2d0 0x7f67ac14d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.130+0000 7f67b2c08640 1 --2- 172.21.15.1:0/1514916665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67ac152650 0x7f67ac14ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.130+0000 7f67b2c08640 1 --2- 172.21.15.1:0/1514916665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67ac0bf9d0 0x7f67ac1eb6f0 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.130+0000 7f67b2c08640 1 -- 172.21.15.1:0/1514916665 >> 172.21.15.1:0/1514916665 conn(0x7f67ac066ee0 msgr2=0x7f67ac0c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.130+0000 7f67b2c08640 1 -- 172.21.15.1:0/1514916665 shutdown_connections 2024-03-06T07:28:18.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.130+0000 7f67b2c08640 1 -- 172.21.15.1:0/1514916665 wait complete. 2024-03-06T07:28:18.545 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 179, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:52.179842+0000', 'modified': '2024-03-06T07:28:16.947047+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 10161, 'mds_1': 9951}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9951': {'gid': 9951, 'name': 'a', 'rank': 1, 'incarnation': 178, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.1:6833/3895143314', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 3895143314}, {'type': 'v1', 'addr': '172.21.15.1:6833', 'nonce': 3895143314}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10161': {'gid': 10161, 'name': 'c', 'rank': 0, 'incarnation': 176, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.1:6835/1525330395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 1525330395}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1525330395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10161, 'qdb_cluster': [9951, 10161]} 2024-03-06T07:28:18.545 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-03-06T07:28:18.545 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:28:18.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.646+0000 7f36de9c9640 1 Processor -- start 2024-03-06T07:28:18.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.646+0000 7f36de9c9640 1 -- start start 2024-03-06T07:28:18.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.646+0000 7f36de9c9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36d8150520 0x7f36d81543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.646+0000 7f36de9c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36d8070c80 0x7f36d80710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.647+0000 7f36de9c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d8150900 con 0x7f36d80c32f0 2024-03-06T07:28:18.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.647+0000 7f36de9c9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d8153ed0 con 0x7f36d8150520 2024-03-06T07:28:18.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.647+0000 7f36de9c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d8071790 con 0x7f36d8070c80 2024-03-06T07:28:18.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.647+0000 7f36dcf66640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36d8070c80 0x7f36d80710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.647+0000 7f36d7fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36d8150520 0x7f36d81543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.647+0000 7f36dcf66640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36d8070c80 0x7f36d80710c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:41634/0 (socket says 172.21.15.1:41634) 2024-03-06T07:28:18.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.647+0000 7f36dcf66640 1 -- 172.21.15.1:0/1104837150 learned_addr learned my addr 172.21.15.1:0/1104837150 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:18.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.647+0000 7f36d7fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36d8150520 0x7f36d81543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:43470/0 (socket says 172.21.15.1:43470) 2024-03-06T07:28:18.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.647+0000 7f36dcf66640 1 -- 172.21.15.1:0/1104837150 >> v2:172.21.15.98:3300/0 conn(0x7f36d8150520 msgr2=0x7f36d81543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.647+0000 7f36dcf66640 1 --2- 172.21.15.1:0/1104837150 >> v2:172.21.15.98:3300/0 conn(0x7f36d8150520 0x7f36d81543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.647+0000 7f36dcf66640 1 -- 172.21.15.1:0/1104837150 >> v1:172.21.15.98:6789/0 conn(0x7f36d80c32f0 legacy=0x7f36d81537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.647+0000 7f36dcf66640 1 -- 172.21.15.1:0/1104837150 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d8157050 con 0x7f36d8070c80 2024-03-06T07:28:18.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.647+0000 7f36d7fff640 1 --2- 172.21.15.1:0/1104837150 >> v2:172.21.15.98:3300/0 conn(0x7f36d8150520 0x7f36d81543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:18.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.648+0000 7f36dcf66640 1 --2- 172.21.15.1:0/1104837150 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36d8070c80 0x7f36d80710c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f36d0009e50 tx=0x7f36d002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e08744189ef5f107 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.648+0000 7f36d6ffd640 1 -- 172.21.15.1:0/1104837150 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d002cb00 con 0x7f36d8070c80 2024-03-06T07:28:18.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.648+0000 7f36d6ffd640 1 -- 172.21.15.1:0/1104837150 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d002cca0 con 0x7f36d8070c80 2024-03-06T07:28:18.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.648+0000 7f36d6ffd640 1 -- 172.21.15.1:0/1104837150 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d0034610 con 0x7f36d8070c80 2024-03-06T07:28:18.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.649+0000 7f36de9c9640 1 -- 172.21.15.1:0/1104837150 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36d8070c80 msgr2=0x7f36d80710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.649+0000 7f36de9c9640 1 --2- 172.21.15.1:0/1104837150 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36d8070c80 0x7f36d80710c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f36d0009e50 tx=0x7f36d002bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:18.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.649+0000 7f36de9c9640 1 -- 172.21.15.1:0/1104837150 shutdown_connections 2024-03-06T07:28:18.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.649+0000 7f36de9c9640 1 --2- 172.21.15.1:0/1104837150 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36d8070c80 0x7f36d80710c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.649+0000 7f36de9c9640 1 --2- 172.21.15.1:0/1104837150 >> v2:172.21.15.98:3300/0 conn(0x7f36d8150520 0x7f36d81543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.649+0000 7f36de9c9640 1 -- 172.21.15.1:0/1104837150 >> 172.21.15.1:0/1104837150 conn(0x7f36d80c88f0 msgr2=0x7f36d80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.649+0000 7f36de9c9640 1 -- 172.21.15.1:0/1104837150 shutdown_connections 2024-03-06T07:28:18.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.649+0000 7f36de9c9640 1 -- 172.21.15.1:0/1104837150 wait complete. 2024-03-06T07:28:18.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.650+0000 7f36de9c9640 1 Processor -- start 2024-03-06T07:28:18.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.650+0000 7f36de9c9640 1 -- start start 2024-03-06T07:28:18.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.651+0000 7f36de9c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d8070c80 0x7f36d814eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.651+0000 7f36de9c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36d80c32f0 0x7f36d81dcd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.651+0000 7f36de9c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36d8150520 0x7f36d8146170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.651+0000 7f36de9c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d8157330 con 0x7f36d8070c80 2024-03-06T07:28:18.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.651+0000 7f36de9c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d81532a0 con 0x7f36d80c32f0 2024-03-06T07:28:18.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.651+0000 7f36de9c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d80c7ec0 con 0x7f36d8150520 2024-03-06T07:28:18.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.651+0000 7f36d7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d8070c80 0x7f36d814eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.651+0000 7f36d7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d8070c80 0x7f36d814eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50934/0 (socket says 172.21.15.1:50934) 2024-03-06T07:28:18.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.651+0000 7f36d7fff640 1 -- 172.21.15.1:0/402828442 learned_addr learned my addr 172.21.15.1:0/402828442 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:18.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.651+0000 7f36d77fe640 1 --2- 172.21.15.1:0/402828442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36d80c32f0 0x7f36d81dcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.651+0000 7f36dcf66640 1 --2- 172.21.15.1:0/402828442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36d8150520 0x7f36d8146170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.651+0000 7f36d7fff640 1 -- 172.21.15.1:0/402828442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36d8150520 msgr2=0x7f36d8146170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.651+0000 7f36d7fff640 1 --2- 172.21.15.1:0/402828442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36d8150520 0x7f36d8146170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.651+0000 7f36d7fff640 1 -- 172.21.15.1:0/402828442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36d80c32f0 msgr2=0x7f36d81dcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.651+0000 7f36d7fff640 1 --2- 172.21.15.1:0/402828442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36d80c32f0 0x7f36d81dcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.651+0000 7f36d7fff640 1 -- 172.21.15.1:0/402828442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d81ebf10 con 0x7f36d8070c80 2024-03-06T07:28:18.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.651+0000 7f36d77fe640 1 --2- 172.21.15.1:0/402828442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36d80c32f0 0x7f36d81dcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:18.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.652+0000 7f36d7fff640 1 --2- 172.21.15.1:0/402828442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d8070c80 0x7f36d814eb00 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f36c800e990 tx=0x7f36c800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.652+0000 7f36d4ff9640 1 -- 172.21.15.1:0/402828442 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c8017070 con 0x7f36d8070c80 2024-03-06T07:28:18.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.652+0000 7f36d4ff9640 1 -- 172.21.15.1:0/402828442 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c800c9e0 con 0x7f36d8070c80 2024-03-06T07:28:18.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.652+0000 7f36de9c9640 1 -- 172.21.15.1:0/402828442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d814f450 con 0x7f36d8070c80 2024-03-06T07:28:18.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.652+0000 7f36d4ff9640 1 -- 172.21.15.1:0/402828442 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c800cdb0 con 0x7f36d8070c80 2024-03-06T07:28:18.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.652+0000 7f36de9c9640 1 -- 172.21.15.1:0/402828442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d81ec490 con 0x7f36d8070c80 2024-03-06T07:28:18.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.654+0000 7f36d4ff9640 1 -- 172.21.15.1:0/402828442 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36c801a070 con 0x7f36d8070c80 2024-03-06T07:28:18.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.654+0000 7f36d4ff9640 1 --2- 172.21.15.1:0/402828442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f36c406b4d0 0x7f36c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.655+0000 7f36d4ff9640 1 -- 172.21.15.1:0/402828442 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f36c808a2f0 con 0x7f36d8070c80 2024-03-06T07:28:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.655+0000 7f36d77fe640 1 --2- 172.21.15.1:0/402828442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f36c406b4d0 0x7f36c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.655+0000 7f36e15b2740 1 Processor -- start 2024-03-06T07:28:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.655+0000 7f36e15b2740 1 -- start start 2024-03-06T07:28:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.655+0000 7f36d77fe640 1 --2- 172.21.15.1:0/402828442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f36c406b4d0 0x7f36c406d990 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f36d814a250 tx=0x7f36cc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.656+0000 7f36e15b2740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ecf50 0x5624934eaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.656+0000 7f36e15b2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624934ed330 0x5624934eaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.656+0000 7f36e15b2740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934eb4b0 0x5624934e7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.656+0000 7f36e15b2740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624934516b0 con 0x5624934eb4b0 2024-03-06T07:28:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.656+0000 7f36e15b2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624933faa60 con 0x5624934ed330 2024-03-06T07:28:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.656+0000 7f36e15b2740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x56249344d750 con 0x5624934ecf50 2024-03-06T07:28:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.656+0000 7f36dcf66640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934eb4b0 0x5624934e7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.656+0000 7f36d7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ecf50 0x5624934eaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.656+0000 7f36dcf66640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934eb4b0 0x5624934e7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50936/0 (socket says 172.21.15.1:50936) 2024-03-06T07:28:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.656+0000 7f36dcf66640 1 -- 172.21.15.1:0/3714046673 learned_addr learned my addr 172.21.15.1:0/3714046673 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.656+0000 7f36d77fe640 1 --2- 172.21.15.1:0/3714046673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624934ed330 0x5624934eaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.656+0000 7f36dcf66640 1 -- 172.21.15.1:0/3714046673 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ecf50 msgr2=0x5624934eaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.656+0000 7f36dcf66640 1 --2- 172.21.15.1:0/3714046673 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ecf50 0x5624934eaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.656+0000 7f36dcf66640 1 -- 172.21.15.1:0/3714046673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624934ed330 msgr2=0x5624934eaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.656+0000 7f36dcf66640 1 --2- 172.21.15.1:0/3714046673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624934ed330 0x5624934eaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.656+0000 7f36dcf66640 1 -- 172.21.15.1:0/3714046673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562493418f30 con 0x5624934eb4b0 2024-03-06T07:28:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.656+0000 7f36d7fff640 1 --2- 172.21.15.1:0/3714046673 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ecf50 0x5624934eaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.657+0000 7f36dcf66640 1 --2- 172.21.15.1:0/3714046673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934eb4b0 0x5624934e7680 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f36d0036040 tx=0x7f36d002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62dd490d505a7269 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.657+0000 7f36bdffb640 1 -- 172.21.15.1:0/3714046673 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d002cb00 con 0x5624934eb4b0 2024-03-06T07:28:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.657+0000 7f36bdffb640 1 -- 172.21.15.1:0/3714046673 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d002ce50 con 0x5624934eb4b0 2024-03-06T07:28:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.657+0000 7f36bdffb640 1 -- 172.21.15.1:0/3714046673 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d003e670 con 0x5624934eb4b0 2024-03-06T07:28:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.657+0000 7f36e15b2740 1 -- 172.21.15.1:0/3714046673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934eb4b0 msgr2=0x5624934e7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.657+0000 7f36e15b2740 1 --2- 172.21.15.1:0/3714046673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934eb4b0 0x5624934e7680 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f36d0036040 tx=0x7f36d002e000 comp rx=0 tx=0).stop 2024-03-06T07:28:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.657+0000 7f36e15b2740 1 -- 172.21.15.1:0/3714046673 shutdown_connections 2024-03-06T07:28:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.657+0000 7f36e15b2740 1 --2- 172.21.15.1:0/3714046673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934eb4b0 0x5624934e7680 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.657+0000 7f36e15b2740 1 --2- 172.21.15.1:0/3714046673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624934ed330 0x5624934eaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.657+0000 7f36e15b2740 1 --2- 172.21.15.1:0/3714046673 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ecf50 0x5624934eaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.657+0000 7f36e15b2740 1 -- 172.21.15.1:0/3714046673 >> 172.21.15.1:0/3714046673 conn(0x5624934e97d0 msgr2=0x56249350e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.657+0000 7f36e15b2740 1 -- 172.21.15.1:0/3714046673 shutdown_connections 2024-03-06T07:28:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.657+0000 7f36e15b2740 1 -- 172.21.15.1:0/3714046673 wait complete. 2024-03-06T07:28:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.658+0000 7f36e15b2740 1 -- start start 2024-03-06T07:28:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.658+0000 7f36e15b2740 1 Processor -- start 2024-03-06T07:28:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36e15b2740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ecf50 0x56249350d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36e15b2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624934ed330 0x56249353a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36e15b2740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56249353d130 0x56249353d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36e15b2740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249350f230 con 0x56249353d130 2024-03-06T07:28:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36e15b2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249340dea0 con 0x5624934ed330 2024-03-06T07:28:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36e15b2740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x56249350ec70 con 0x5624934ecf50 2024-03-06T07:28:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36e15b2740 10 client.0 _ll_register_callbacks cb 0x56249340d870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36dcf66640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56249353d130 0x56249353d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36dcf66640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56249353d130 0x56249353d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50950/0 (socket says 172.21.15.1:50950) 2024-03-06T07:28:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36dcf66640 1 -- 172.21.15.1:0/2056467110 learned_addr learned my addr 172.21.15.1:0/2056467110 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36d7fff640 1 --2- 172.21.15.1:0/2056467110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ecf50 0x56249350d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36d77fe640 1 --2- 172.21.15.1:0/2056467110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624934ed330 0x56249353a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36dcf66640 1 -- 172.21.15.1:0/2056467110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ecf50 msgr2=0x56249350d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36dcf66640 1 --2- 172.21.15.1:0/2056467110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ecf50 0x56249350d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36dcf66640 1 -- 172.21.15.1:0/2056467110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624934ed330 msgr2=0x56249353a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36dcf66640 1 --2- 172.21.15.1:0/2056467110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624934ed330 0x56249353a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36dcf66640 1 -- 172.21.15.1:0/2056467110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56249340e270 con 0x56249353d130 2024-03-06T07:28:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36d77fe640 1 --2- 172.21.15.1:0/2056467110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624934ed330 0x56249353a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36dcf66640 1 --2- 172.21.15.1:0/2056467110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56249353d130 0x56249353d5b0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f36d003e660 tx=0x7f36d0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb1afa6b43f4e739 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:28:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36b6ffd640 1 -- 172.21.15.1:0/2056467110 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d004c020 con 0x56249353d130 2024-03-06T07:28:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36b6ffd640 1 -- 172.21.15.1:0/2056467110 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f36d0003ba0 con 0x56249353d130 2024-03-06T07:28:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.660+0000 7f36e15b2740 1 -- 172.21.15.1:0/2056467110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36d002ce50 con 0x56249353d130 2024-03-06T07:28:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.661+0000 7f36b6ffd640 1 -- 172.21.15.1:0/2056467110 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=179) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36d0056050 con 0x56249353d130 2024-03-06T07:28:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.661+0000 7f36e15b2740 10 client.10242 fetch_fsmap learned FSMap version 179 2024-03-06T07:28:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.661+0000 7f36e15b2740 1 -- 172.21.15.1:0/2056467110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=179,monmap=2+}) v3 -- 0x5624931662d0 con 0x56249353d130 2024-03-06T07:28:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.661+0000 7f36b6ffd640 1 -- 172.21.15.1:0/2056467110 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d002ce50 con 0x56249353d130 2024-03-06T07:28:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.661+0000 7f36b6ffd640 1 -- 172.21.15.1:0/2056467110 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 179) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f36d0059020 con 0x56249353d130 2024-03-06T07:28:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.661+0000 7f36e15b2740 10 client.10242 fetch_fsmap finished waiting for FSMap version 179 2024-03-06T07:28:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.661+0000 7f36e15b2740 10 client.10242 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:28:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.661+0000 7f36e15b2740 20 client.10242 populate_metadata read hostname 'smithi001' 2024-03-06T07:28:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.661+0000 7f36e15b2740 1 --2- 172.21.15.1:0/2056467110 >> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] conn(0x56249350fd60 0x562493548460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.661+0000 7f36e15b2740 4 client.10242 mds_command: new command op to 9951 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:28:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.661+0000 7f36e15b2740 1 -- 172.21.15.1:0/2056467110 --> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56249345ebf0 con 0x56249350fd60 2024-03-06T07:28:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.661+0000 7f36d7fff640 1 --2- 172.21.15.1:0/2056467110 >> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] conn(0x56249350fd60 0x562493548460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.661+0000 7f36b6ffd640 10 client.10242 ms_handle_connect on v2:172.21.15.1:6832/3895143314 2024-03-06T07:28:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.661+0000 7f36d7fff640 1 --2- 172.21.15.1:0/2056467110 >> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] conn(0x56249350fd60 0x562493548460 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=59e725b0451b9956 server_cookie=f9cd274c307067d in_seq=0 out_seq=0 2024-03-06T07:28:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.662+0000 7f36b6ffd640 1 -- 172.21.15.1:0/2056467110 <== mds.1 v2:172.21.15.1:6832/3895143314 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56249345ebf0 con 0x56249350fd60 2024-03-06T07:28:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.662+0000 7f36b6ffd640 10 client.10242 handle_command_reply: tid=0 2024-03-06T07:28:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.662+0000 7f36e15b2740 1 client.10242 shutdown 2024-03-06T07:28:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.663+0000 7f36e15b2740 1 -- 172.21.15.1:0/2056467110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56249353d130 msgr2=0x56249353d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.663+0000 7f36e15b2740 1 --2- 172.21.15.1:0/2056467110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56249353d130 0x56249353d5b0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f36d003e660 tx=0x7f36d0004ab0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.663+0000 7f36e15b2740 1 -- 172.21.15.1:0/2056467110 shutdown_connections 2024-03-06T07:28:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.663+0000 7f36e15b2740 1 --2- 172.21.15.1:0/2056467110 >> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] conn(0x56249350fd60 0x562493548460 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.663+0000 7f36e15b2740 1 --2- 172.21.15.1:0/2056467110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56249353d130 0x56249353d5b0 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.663+0000 7f36e15b2740 1 --2- 172.21.15.1:0/2056467110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624934ed330 0x56249353a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.663+0000 7f36e15b2740 1 --2- 172.21.15.1:0/2056467110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ecf50 0x56249350d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.663+0000 7f36e15b2740 1 -- 172.21.15.1:0/2056467110 >> 172.21.15.1:0/2056467110 conn(0x5624934eb4b0 msgr2=0x5624934eb890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.663+0000 7f36b6ffd640 0 client.10242 ms_handle_reset on v2:172.21.15.1:6832/3895143314 2024-03-06T07:28:18.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.663+0000 7f36e15b2740 1 -- 172.21.15.1:0/2056467110 shutdown_connections 2024-03-06T07:28:18.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.663+0000 7f36e15b2740 1 -- 172.21.15.1:0/2056467110 wait complete. 2024-03-06T07:28:18.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.663+0000 7f36e15b2740 20 client.10242 trim_cache size 0 max 16384 2024-03-06T07:28:18.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.680+0000 7f36e15b2740 1 Processor -- start 2024-03-06T07:28:18.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.680+0000 7f36e15b2740 1 -- start start 2024-03-06T07:28:18.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.680+0000 7f36e15b2740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934ecf50 0x56249350d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.680+0000 7f36e15b2740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ed330 0x562493539e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.680+0000 7f36e15b2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249353d130 0x56249353a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.680+0000 7f36e15b2740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d81f56b0 con 0x5624934ecf50 2024-03-06T07:28:18.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.680+0000 7f36e15b2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d81f5d10 con 0x56249353d130 2024-03-06T07:28:18.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.680+0000 7f36e15b2740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d80c5580 con 0x5624934ed330 2024-03-06T07:28:18.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.680+0000 7f36d7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934ecf50 0x56249350d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.680+0000 7f36d7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934ecf50 0x56249350d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50956/0 (socket says 172.21.15.1:50956) 2024-03-06T07:28:18.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.680+0000 7f36d7fff640 1 -- 172.21.15.1:0/4226726444 learned_addr learned my addr 172.21.15.1:0/4226726444 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.681+0000 7f36d77fe640 1 --2- 172.21.15.1:0/4226726444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ed330 0x562493539e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.681+0000 7f36dcf66640 1 --2- 172.21.15.1:0/4226726444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249353d130 0x56249353a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.681+0000 7f36d7fff640 1 -- 172.21.15.1:0/4226726444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ed330 msgr2=0x562493539e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.681+0000 7f36d7fff640 1 --2- 172.21.15.1:0/4226726444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ed330 0x562493539e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.681+0000 7f36d7fff640 1 -- 172.21.15.1:0/4226726444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249353d130 msgr2=0x56249353a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.681+0000 7f36d7fff640 1 --2- 172.21.15.1:0/4226726444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249353d130 0x56249353a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.681+0000 7f36d7fff640 1 -- 172.21.15.1:0/4226726444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624932306b0 con 0x5624934ecf50 2024-03-06T07:28:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.681+0000 7f36dcf66640 1 --2- 172.21.15.1:0/4226726444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249353d130 0x56249353a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.681+0000 7f36d77fe640 1 --2- 172.21.15.1:0/4226726444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ed330 0x562493539e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.681+0000 7f36d7fff640 1 --2- 172.21.15.1:0/4226726444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934ecf50 0x56249350d5c0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f36c8054c70 tx=0x7f36c808a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f597d665b172a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.681+0000 7f36bd7fa640 1 -- 172.21.15.1:0/4226726444 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c8086070 con 0x5624934ecf50 2024-03-06T07:28:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.681+0000 7f36bd7fa640 1 -- 172.21.15.1:0/4226726444 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c8060f70 con 0x5624934ecf50 2024-03-06T07:28:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.681+0000 7f36bd7fa640 1 -- 172.21.15.1:0/4226726444 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c8021040 con 0x5624934ecf50 2024-03-06T07:28:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.682+0000 7f36e15b2740 1 -- 172.21.15.1:0/4226726444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934ecf50 msgr2=0x56249350d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.682+0000 7f36e15b2740 1 --2- 172.21.15.1:0/4226726444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934ecf50 0x56249350d5c0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f36c8054c70 tx=0x7f36c808a040 comp rx=0 tx=0).stop 2024-03-06T07:28:18.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.682+0000 7f36e15b2740 1 -- 172.21.15.1:0/4226726444 shutdown_connections 2024-03-06T07:28:18.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.682+0000 7f36e15b2740 1 --2- 172.21.15.1:0/4226726444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249353d130 0x56249353a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.682+0000 7f36e15b2740 1 --2- 172.21.15.1:0/4226726444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ed330 0x562493539e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.682+0000 7f36e15b2740 1 --2- 172.21.15.1:0/4226726444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934ecf50 0x56249350d5c0 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.682+0000 7f36e15b2740 1 -- 172.21.15.1:0/4226726444 >> 172.21.15.1:0/4226726444 conn(0x5624934eb4b0 msgr2=0x5624934e8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.682+0000 7f36e15b2740 1 -- 172.21.15.1:0/4226726444 shutdown_connections 2024-03-06T07:28:18.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.682+0000 7f36e15b2740 1 -- 172.21.15.1:0/4226726444 wait complete. 2024-03-06T07:28:18.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.683+0000 7f36e15b2740 1 -- start start 2024-03-06T07:28:18.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.683+0000 7f36e15b2740 1 Processor -- start 2024-03-06T07:28:18.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.684+0000 7f36e15b2740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ecf50 0x562493599070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.684+0000 7f36e15b2740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934ed330 0x562493510190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.684+0000 7f36e15b2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249353d130 0x5624935966b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.684+0000 7f36e15b2740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249350f230 con 0x5624934ed330 2024-03-06T07:28:18.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.684+0000 7f36e15b2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d81f4870 con 0x56249353d130 2024-03-06T07:28:18.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.684+0000 7f36e15b2740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x562493511030 con 0x5624934ecf50 2024-03-06T07:28:18.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.684+0000 7f36e15b2740 10 client.0 _ll_register_callbacks cb 0x7f36d0009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:18.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.684+0000 7f36d77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934ed330 0x562493510190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.684+0000 7f36d77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934ed330 0x562493510190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50972/0 (socket says 172.21.15.1:50972) 2024-03-06T07:28:18.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.684+0000 7f36d77fe640 1 -- 172.21.15.1:0/986128606 learned_addr learned my addr 172.21.15.1:0/986128606 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:18.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.684+0000 7f36d7fff640 1 --2- 172.21.15.1:0/986128606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ecf50 0x562493599070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.684+0000 7f36dcf66640 1 --2- 172.21.15.1:0/986128606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249353d130 0x5624935966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.684+0000 7f36d77fe640 1 -- 172.21.15.1:0/986128606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ecf50 msgr2=0x562493599070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.684+0000 7f36d77fe640 1 --2- 172.21.15.1:0/986128606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ecf50 0x562493599070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.684+0000 7f36d77fe640 1 -- 172.21.15.1:0/986128606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249353d130 msgr2=0x5624935966b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.684+0000 7f36d77fe640 1 --2- 172.21.15.1:0/986128606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249353d130 0x5624935966b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.684+0000 7f36d77fe640 1 -- 172.21.15.1:0/986128606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56249359cff0 con 0x5624934ed330 2024-03-06T07:28:18.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.685+0000 7f36d77fe640 1 --2- 172.21.15.1:0/986128606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934ed330 0x562493510190 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f36cc007fd0 tx=0x7f36cc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a1d3b47e1fe5c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.685+0000 7f36b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:28:18.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.685+0000 7f36b6ffd640 1 -- 172.21.15.1:0/986128606 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc028060 con 0x5624934ed330 2024-03-06T07:28:18.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.685+0000 7f36b6ffd640 1 -- 172.21.15.1:0/986128606 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f36cc010040 con 0x5624934ed330 2024-03-06T07:28:18.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.685+0000 7f36e15b2740 1 -- 172.21.15.1:0/986128606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36c8060f70 con 0x5624934ed330 2024-03-06T07:28:18.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.685+0000 7f36b6ffd640 1 -- 172.21.15.1:0/986128606 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=179) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36cc02e050 con 0x5624934ed330 2024-03-06T07:28:18.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.685+0000 7f36e15b2740 10 client.10248 fetch_fsmap learned FSMap version 179 2024-03-06T07:28:18.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.685+0000 7f36e15b2740 1 -- 172.21.15.1:0/986128606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=179,monmap=2+}) v3 -- 0x5624934e6d90 con 0x5624934ed330 2024-03-06T07:28:18.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.685+0000 7f36b6ffd640 1 -- 172.21.15.1:0/986128606 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c8060f70 con 0x5624934ed330 2024-03-06T07:28:18.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.685+0000 7f36b6ffd640 1 -- 172.21.15.1:0/986128606 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 179) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f36cc031020 con 0x5624934ed330 2024-03-06T07:28:18.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.685+0000 7f36e15b2740 10 client.10248 fetch_fsmap finished waiting for FSMap version 179 2024-03-06T07:28:18.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.685+0000 7f36e15b2740 10 client.10248 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:28:18.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.685+0000 7f36e15b2740 20 client.10248 populate_metadata read hostname 'smithi001' 2024-03-06T07:28:18.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.685+0000 7f36e15b2740 1 --2- 172.21.15.1:0/986128606 >> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] conn(0x5624934eab00 0x562493512a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.685+0000 7f36e15b2740 4 client.10248 mds_command: new command op to 9951 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:28:18.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.685+0000 7f36e15b2740 1 -- 172.21.15.1:0/986128606 --> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562493458f40 con 0x5624934eab00 2024-03-06T07:28:18.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.685+0000 7f36d7fff640 1 --2- 172.21.15.1:0/986128606 >> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] conn(0x5624934eab00 0x562493512a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.686+0000 7f36b6ffd640 10 client.10248 ms_handle_connect on v2:172.21.15.1:6832/3895143314 2024-03-06T07:28:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.686+0000 7f36d7fff640 1 --2- 172.21.15.1:0/986128606 >> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] conn(0x5624934eab00 0x562493512a30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a9f1b94d51b4b2c4 server_cookie=271fdcba9d29e8d7 in_seq=0 out_seq=0 2024-03-06T07:28:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.686+0000 7f36b6ffd640 1 -- 172.21.15.1:0/986128606 <== mds.1 v2:172.21.15.1:6832/3895143314 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x562493458f40 con 0x5624934eab00 2024-03-06T07:28:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.686+0000 7f36b6ffd640 10 client.10248 handle_command_reply: tid=0 2024-03-06T07:28:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.686+0000 7f36e15b2740 1 client.10248 shutdown 2024-03-06T07:28:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.686+0000 7f36e15b2740 1 -- 172.21.15.1:0/986128606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934ed330 msgr2=0x562493510190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.686+0000 7f36e15b2740 1 --2- 172.21.15.1:0/986128606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934ed330 0x562493510190 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f36cc007fd0 tx=0x7f36cc000bc0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.686+0000 7f36e15b2740 1 -- 172.21.15.1:0/986128606 shutdown_connections 2024-03-06T07:28:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.686+0000 7f36e15b2740 1 --2- 172.21.15.1:0/986128606 >> [v2:172.21.15.1:6832/3895143314,v1:172.21.15.1:6833/3895143314] conn(0x5624934eab00 0x562493512a30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.686+0000 7f36e15b2740 1 --2- 172.21.15.1:0/986128606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249353d130 0x5624935966b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.686+0000 7f36e15b2740 1 --2- 172.21.15.1:0/986128606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624934ed330 0x562493510190 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.686+0000 7f36e15b2740 1 --2- 172.21.15.1:0/986128606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5624934ecf50 0x562493599070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.686+0000 7f36e15b2740 1 -- 172.21.15.1:0/986128606 >> 172.21.15.1:0/986128606 conn(0x5624934eb4b0 msgr2=0x5624934e6510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.686+0000 7f36b6ffd640 0 client.10248 ms_handle_reset on v2:172.21.15.1:6832/3895143314 2024-03-06T07:28:18.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.686+0000 7f36e15b2740 1 -- 172.21.15.1:0/986128606 shutdown_connections 2024-03-06T07:28:18.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.686+0000 7f36e15b2740 1 -- 172.21.15.1:0/986128606 wait complete. 2024-03-06T07:28:18.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.686+0000 7f36e15b2740 20 client.10248 trim_cache size 0 max 16384 2024-03-06T07:28:18.694 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:28:18.694 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:28:18.694 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 1, 2024-03-06T07:28:18.694 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 9951, 2024-03-06T07:28:18.694 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:28:18.694 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:28:18.694 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:28:18.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 2.7399106299999998, 2024-03-06T07:28:18.695 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 179, 2024-03-06T07:28:18.695 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 185, 2024-03-06T07:28:18.695 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 185, 2024-03-06T07:28:18.695 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 30.946639144999999 2024-03-06T07:28:18.695 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:28:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.687+0000 7f36de9c9640 1 -- 172.21.15.1:0/402828442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f36c406b4d0 msgr2=0x7f36c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.687+0000 7f36de9c9640 1 --2- 172.21.15.1:0/402828442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f36c406b4d0 0x7f36c406d990 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f36d814a250 tx=0x7f36cc009040 comp rx=0 tx=0).stop 2024-03-06T07:28:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.687+0000 7f36de9c9640 1 -- 172.21.15.1:0/402828442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d8070c80 msgr2=0x7f36d814eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.687+0000 7f36de9c9640 1 --2- 172.21.15.1:0/402828442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d8070c80 0x7f36d814eb00 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f36c800e990 tx=0x7f36c800ee50 comp rx=0 tx=0).stop 2024-03-06T07:28:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.687+0000 7f36de9c9640 1 -- 172.21.15.1:0/402828442 shutdown_connections 2024-03-06T07:28:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.687+0000 7f36de9c9640 1 --2- 172.21.15.1:0/402828442 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f36c406b4d0 0x7f36c406d990 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.687+0000 7f36de9c9640 1 --2- 172.21.15.1:0/402828442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f36d8150520 0x7f36d8146170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.687+0000 7f36de9c9640 1 --2- 172.21.15.1:0/402828442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36d80c32f0 0x7f36d81dcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.687+0000 7f36de9c9640 1 --2- 172.21.15.1:0/402828442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36d8070c80 0x7f36d814eb00 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.687+0000 7f36de9c9640 1 -- 172.21.15.1:0/402828442 >> 172.21.15.1:0/402828442 conn(0x7f36d80c88f0 msgr2=0x7f36d80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.687+0000 7f36de9c9640 1 -- 172.21.15.1:0/402828442 shutdown_connections 2024-03-06T07:28:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.687+0000 7f36de9c9640 1 -- 172.21.15.1:0/402828442 wait complete. 2024-03-06T07:28:18.700 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:28:18.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.807+0000 7f8b502df640 1 Processor -- start 2024-03-06T07:28:18.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.807+0000 7f8b502df640 1 -- start start 2024-03-06T07:28:18.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.808+0000 7f8b502df640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8b480c6b10 0x7f8b48155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.808+0000 7f8b502df640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8b480bf9d0 0x7f8b480c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.808+0000 7f8b502df640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b480c2fc0 con 0x7f8b48152650 2024-03-06T07:28:18.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.808+0000 7f8b502df640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b48155800 con 0x7f8b480c6b10 2024-03-06T07:28:18.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.808+0000 7f8b502df640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b48158da0 con 0x7f8b480bf9d0 2024-03-06T07:28:18.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.808+0000 7f8b4d87a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8b480c6b10 0x7f8b48155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.808+0000 7f8b4e07b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8b48152650 0x7f8b48158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39750/0 (socket says 172.21.15.1:39750) 2024-03-06T07:28:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.808+0000 7f8b4e07b640 1 -- 172.21.15.1:0/1280166780 learned_addr learned my addr 172.21.15.1:0/1280166780 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.808+0000 7f8b4e87c640 1 --2- 172.21.15.1:0/1280166780 >> v2:172.21.15.98:3300/0 conn(0x7f8b480bf9d0 0x7f8b480c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.808+0000 7f8b4d87a640 1 -- 172.21.15.1:0/1280166780 >> v2:172.21.15.98:3300/0 conn(0x7f8b480bf9d0 msgr2=0x7f8b480c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.808+0000 7f8b4d87a640 1 --2- 172.21.15.1:0/1280166780 >> v2:172.21.15.98:3300/0 conn(0x7f8b480bf9d0 0x7f8b480c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.808+0000 7f8b4d87a640 1 -- 172.21.15.1:0/1280166780 >> v1:172.21.15.1:6789/0 conn(0x7f8b48152650 legacy=0x7f8b48158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.808+0000 7f8b4d87a640 1 -- 172.21.15.1:0/1280166780 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b480c6690 con 0x7f8b480c6b10 2024-03-06T07:28:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.808+0000 7f8b4d079640 1 -- 172.21.15.1:0/1280166780 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1200094312 0 0) 0x7f8b480c2fc0 con 0x7f8b48152650 2024-03-06T07:28:18.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.808+0000 7f8b4e87c640 1 --2- 172.21.15.1:0/1280166780 >> v2:172.21.15.98:3300/0 conn(0x7f8b480bf9d0 0x7f8b480c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:18.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.809+0000 7f8b4d87a640 1 --2- 172.21.15.1:0/1280166780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b480c6b10 0x7f8b48155c90 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f8b481597b0 tx=0x7f8b3802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb22fef0809696c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.809+0000 7f8b4d079640 1 -- 172.21.15.1:0/1280166780 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b380315a0 con 0x7f8b480c6b10 2024-03-06T07:28:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.809+0000 7f8b4d079640 1 -- 172.21.15.1:0/1280166780 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b38037070 con 0x7f8b480c6b10 2024-03-06T07:28:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.809+0000 7f8b4d079640 1 -- 172.21.15.1:0/1280166780 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b38031960 con 0x7f8b480c6b10 2024-03-06T07:28:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.809+0000 7f8b502df640 1 -- 172.21.15.1:0/1280166780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b480c6b10 msgr2=0x7f8b48155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.809+0000 7f8b502df640 1 --2- 172.21.15.1:0/1280166780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b480c6b10 0x7f8b48155c90 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f8b481597b0 tx=0x7f8b3802b840 comp rx=0 tx=0).stop 2024-03-06T07:28:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.810+0000 7f8b502df640 1 -- 172.21.15.1:0/1280166780 shutdown_connections 2024-03-06T07:28:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.810+0000 7f8b502df640 1 --2- 172.21.15.1:0/1280166780 >> v2:172.21.15.98:3300/0 conn(0x7f8b480bf9d0 0x7f8b480c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.810+0000 7f8b502df640 1 --2- 172.21.15.1:0/1280166780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b480c6b10 0x7f8b48155c90 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.810+0000 7f8b502df640 1 -- 172.21.15.1:0/1280166780 >> 172.21.15.1:0/1280166780 conn(0x7f8b48066ee0 msgr2=0x7f8b480672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.810+0000 7f8b502df640 1 -- 172.21.15.1:0/1280166780 shutdown_connections 2024-03-06T07:28:18.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.810+0000 7f8b502df640 1 -- 172.21.15.1:0/1280166780 wait complete. 2024-03-06T07:28:18.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.811+0000 7f8b502df640 1 Processor -- start 2024-03-06T07:28:18.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.811+0000 7f8b502df640 1 -- start start 2024-03-06T07:28:18.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.812+0000 7f8b502df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b480bf9d0 0x7f8b481540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.812+0000 7f8b502df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b48152650 0x7f8b48156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.812+0000 7f8b502df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b48154c00 0x7f8b481569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.812+0000 7f8b502df640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b481551b0 con 0x7f8b48152650 2024-03-06T07:28:18.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.812+0000 7f8b502df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b480c6fa0 con 0x7f8b480bf9d0 2024-03-06T07:28:18.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.812+0000 7f8b502df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b48155490 con 0x7f8b48154c00 2024-03-06T07:28:18.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.813+0000 7f8b4d87a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b48152650 0x7f8b48156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.813+0000 7f8b4d87a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b48152650 0x7f8b48156490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50994/0 (socket says 172.21.15.1:50994) 2024-03-06T07:28:18.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.813+0000 7f8b4d87a640 1 -- 172.21.15.1:0/2047882671 learned_addr learned my addr 172.21.15.1:0/2047882671 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:18.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.813+0000 7f8b4e07b640 1 --2- 172.21.15.1:0/2047882671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b480bf9d0 0x7f8b481540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.813+0000 7f8b4e87c640 1 --2- 172.21.15.1:0/2047882671 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b48154c00 0x7f8b481569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.813+0000 7f8b4d87a640 1 -- 172.21.15.1:0/2047882671 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b48154c00 msgr2=0x7f8b481569d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.813+0000 7f8b4d87a640 1 --2- 172.21.15.1:0/2047882671 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b48154c00 0x7f8b481569d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.813+0000 7f8b4d87a640 1 -- 172.21.15.1:0/2047882671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b480bf9d0 msgr2=0x7f8b481540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.813+0000 7f8b4d87a640 1 --2- 172.21.15.1:0/2047882671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b480bf9d0 0x7f8b481540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.813+0000 7f8b4d87a640 1 -- 172.21.15.1:0/2047882671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4814b550 con 0x7f8b48152650 2024-03-06T07:28:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.813+0000 7f8b4d87a640 1 --2- 172.21.15.1:0/2047882671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b48152650 0x7f8b48156490 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f8b38034040 tx=0x7f8b380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.813+0000 7f8b3effd640 1 -- 172.21.15.1:0/2047882671 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b38004230 con 0x7f8b48152650 2024-03-06T07:28:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.813+0000 7f8b3effd640 1 -- 172.21.15.1:0/2047882671 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b380043d0 con 0x7f8b48152650 2024-03-06T07:28:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.813+0000 7f8b3effd640 1 -- 172.21.15.1:0/2047882671 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b38032860 con 0x7f8b48152650 2024-03-06T07:28:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.813+0000 7f8b502df640 1 -- 172.21.15.1:0/2047882671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b480c9cf0 con 0x7f8b48152650 2024-03-06T07:28:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.813+0000 7f8b502df640 1 -- 172.21.15.1:0/2047882671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b4814b7c0 con 0x7f8b48152650 2024-03-06T07:28:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.815+0000 7f8b3effd640 1 -- 172.21.15.1:0/2047882671 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b38032a00 con 0x7f8b48152650 2024-03-06T07:28:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.815+0000 7f8b3effd640 1 --2- 172.21.15.1:0/2047882671 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8b2c06b4d0 0x7f8b2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.815+0000 7f8b3effd640 1 -- 172.21.15.1:0/2047882671 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f8b3802f030 con 0x7f8b48152650 2024-03-06T07:28:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.816+0000 7f8b4e07b640 1 --2- 172.21.15.1:0/2047882671 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8b2c06b4d0 0x7f8b2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.816+0000 7f8b4e07b640 1 --2- 172.21.15.1:0/2047882671 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8b2c06b4d0 0x7f8b2c06d990 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f8b44000f60 tx=0x7f8b44007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.817+0000 7f8b52fa1740 1 Processor -- start 2024-03-06T07:28:18.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.817+0000 7f8b52fa1740 1 -- start start 2024-03-06T07:28:18.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.818+0000 7f8b52fa1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d4fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.818+0000 7f8b52fa1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d52330 0x5611a9d4ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.818+0000 7f8b52fa1740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d504b0 0x5611a9d4c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.818+0000 7f8b52fa1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a9cb66b0 con 0x5611a9d51f50 2024-03-06T07:28:18.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.818+0000 7f8b52fa1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a9c5fa60 con 0x5611a9d52330 2024-03-06T07:28:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.818+0000 7f8b52fa1740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5611a9cb2750 con 0x5611a9d504b0 2024-03-06T07:28:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.818+0000 7f8b4e07b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d4fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.818+0000 7f8b4e07b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d4fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50996/0 (socket says 172.21.15.1:50996) 2024-03-06T07:28:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.818+0000 7f8b4e07b640 1 -- 172.21.15.1:0/1311202265 learned_addr learned my addr 172.21.15.1:0/1311202265 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.819+0000 7f8b4d87a640 1 --2- 172.21.15.1:0/1311202265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d52330 0x5611a9d4ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.819+0000 7f8b4e87c640 1 --2- 172.21.15.1:0/1311202265 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d504b0 0x5611a9d4c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.819+0000 7f8b4e07b640 1 -- 172.21.15.1:0/1311202265 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d504b0 msgr2=0x5611a9d4c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.819+0000 7f8b4e07b640 1 --2- 172.21.15.1:0/1311202265 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d504b0 0x5611a9d4c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.819+0000 7f8b4e07b640 1 -- 172.21.15.1:0/1311202265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d52330 msgr2=0x5611a9d4ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.819+0000 7f8b4e07b640 1 --2- 172.21.15.1:0/1311202265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d52330 0x5611a9d4ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.819+0000 7f8b4e07b640 1 -- 172.21.15.1:0/1311202265 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611a9c7df30 con 0x5611a9d51f50 2024-03-06T07:28:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.819+0000 7f8b4e07b640 1 --2- 172.21.15.1:0/1311202265 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d4fa30 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f8b44004f90 tx=0x7f8b44013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3130b4c626cf12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.819+0000 7f8b2bfff640 1 -- 172.21.15.1:0/1311202265 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b44026070 con 0x5611a9d51f50 2024-03-06T07:28:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.819+0000 7f8b2bfff640 1 -- 172.21.15.1:0/1311202265 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b44013b80 con 0x5611a9d51f50 2024-03-06T07:28:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.819+0000 7f8b2bfff640 1 -- 172.21.15.1:0/1311202265 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b44012350 con 0x5611a9d51f50 2024-03-06T07:28:18.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.820+0000 7f8b52fa1740 1 -- 172.21.15.1:0/1311202265 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 msgr2=0x5611a9d4fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.820+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/1311202265 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d4fa30 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f8b44004f90 tx=0x7f8b44013000 comp rx=0 tx=0).stop 2024-03-06T07:28:18.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.820+0000 7f8b52fa1740 1 -- 172.21.15.1:0/1311202265 shutdown_connections 2024-03-06T07:28:18.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.820+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/1311202265 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d504b0 0x5611a9d4c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.820+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/1311202265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d52330 0x5611a9d4ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.820+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/1311202265 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d4fa30 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.820+0000 7f8b52fa1740 1 -- 172.21.15.1:0/1311202265 >> 172.21.15.1:0/1311202265 conn(0x5611a9d4e7d0 msgr2=0x5611a9d733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.820+0000 7f8b52fa1740 1 -- 172.21.15.1:0/1311202265 shutdown_connections 2024-03-06T07:28:18.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.820+0000 7f8b52fa1740 1 -- 172.21.15.1:0/1311202265 wait complete. 2024-03-06T07:28:18.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.821+0000 7f8b52fa1740 1 -- start start 2024-03-06T07:28:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.821+0000 7f8b52fa1740 1 Processor -- start 2024-03-06T07:28:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.822+0000 7f8b52fa1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d77a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.822+0000 7f8b52fa1740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d52330 0x5611a9d78a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.822+0000 7f8b52fa1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d78130 0x5611a9d747b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.822+0000 7f8b52fa1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a9d74230 con 0x5611a9d51f50 2024-03-06T07:28:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.822+0000 7f8b52fa1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a9c72ea0 con 0x5611a9d78130 2024-03-06T07:28:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.822+0000 7f8b52fa1740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5611a9d73c70 con 0x5611a9d52330 2024-03-06T07:28:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.822+0000 7f8b52fa1740 10 client.0 _ll_register_callbacks cb 0x5611a9c72870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.822+0000 7f8b4e07b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d77a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.822+0000 7f8b4e07b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d77a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51006/0 (socket says 172.21.15.1:51006) 2024-03-06T07:28:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.822+0000 7f8b4e07b640 1 -- 172.21.15.1:0/2426990952 learned_addr learned my addr 172.21.15.1:0/2426990952 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.822+0000 7f8b4d87a640 1 --2- 172.21.15.1:0/2426990952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d52330 0x5611a9d78a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.822+0000 7f8b4e87c640 1 --2- 172.21.15.1:0/2426990952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d78130 0x5611a9d747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b4e07b640 1 -- 172.21.15.1:0/2426990952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d52330 msgr2=0x5611a9d78a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b4e07b640 1 --2- 172.21.15.1:0/2426990952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d52330 0x5611a9d78a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b4e07b640 1 -- 172.21.15.1:0/2426990952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d78130 msgr2=0x5611a9d747b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b4e07b640 1 --2- 172.21.15.1:0/2426990952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d78130 0x5611a9d747b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b4e07b640 1 -- 172.21.15.1:0/2426990952 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5611a9c73270 con 0x5611a9d51f50 2024-03-06T07:28:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b4d87a640 1 --2- 172.21.15.1:0/2426990952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d52330 0x5611a9d78a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b4e87c640 1 --2- 172.21.15.1:0/2426990952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d78130 0x5611a9d747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b4e07b640 1 --2- 172.21.15.1:0/2426990952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d77a30 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x5611a9d915d0 tx=0x7f8b44000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d5d1fbba42acd4e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b297fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:28:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b297fa640 1 -- 172.21.15.1:0/2426990952 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b44030060 con 0x5611a9d51f50 2024-03-06T07:28:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b297fa640 1 -- 172.21.15.1:0/2426990952 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f8b44013d30 con 0x5611a9d51f50 2024-03-06T07:28:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b52fa1740 1 -- 172.21.15.1:0/2426990952 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8b44013b80 con 0x5611a9d51f50 2024-03-06T07:28:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b297fa640 1 -- 172.21.15.1:0/2426990952 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=179) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b44036050 con 0x5611a9d51f50 2024-03-06T07:28:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b52fa1740 10 client.10263 fetch_fsmap learned FSMap version 179 2024-03-06T07:28:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b52fa1740 1 -- 172.21.15.1:0/2426990952 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=179,monmap=2+}) v3 -- 0x5611a99cb2d0 con 0x5611a9d51f50 2024-03-06T07:28:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b297fa640 1 -- 172.21.15.1:0/2426990952 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b44013b80 con 0x5611a9d51f50 2024-03-06T07:28:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b297fa640 1 -- 172.21.15.1:0/2426990952 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 179) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f8b44039020 con 0x5611a9d51f50 2024-03-06T07:28:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b52fa1740 10 client.10263 fetch_fsmap finished waiting for FSMap version 179 2024-03-06T07:28:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b52fa1740 10 client.10263 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:28:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.823+0000 7f8b52fa1740 20 client.10263 populate_metadata read hostname 'smithi001' 2024-03-06T07:28:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.824+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/2426990952 >> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] conn(0x5611a9d8add0 0x5611a9da27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.824+0000 7f8b52fa1740 4 client.10263 mds_command: new command op to 10161 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:28:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.824+0000 7f8b52fa1740 1 -- 172.21.15.1:0/2426990952 --> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5611a9cc3bf0 con 0x5611a9d8add0 2024-03-06T07:28:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.824+0000 7f8b4d87a640 1 --2- 172.21.15.1:0/2426990952 >> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] conn(0x5611a9d8add0 0x5611a9da27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.824+0000 7f8b297fa640 10 client.10263 ms_handle_connect on v2:172.21.15.1:6834/1525330395 2024-03-06T07:28:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.824+0000 7f8b4d87a640 1 --2- 172.21.15.1:0/2426990952 >> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] conn(0x5611a9d8add0 0x5611a9da27c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=483a8197513b0b55 server_cookie=3d6acea1e89d6320 in_seq=0 out_seq=0 2024-03-06T07:28:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.825+0000 7f8b297fa640 1 -- 172.21.15.1:0/2426990952 <== mds.0 v2:172.21.15.1:6834/1525330395 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5611a9cc3bf0 con 0x5611a9d8add0 2024-03-06T07:28:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.825+0000 7f8b297fa640 10 client.10263 handle_command_reply: tid=0 2024-03-06T07:28:18.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.825+0000 7f8b52fa1740 1 client.10263 shutdown 2024-03-06T07:28:18.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.825+0000 7f8b52fa1740 1 -- 172.21.15.1:0/2426990952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 msgr2=0x5611a9d77a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.825+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/2426990952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d77a30 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x5611a9d915d0 tx=0x7f8b44000ed0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.825+0000 7f8b52fa1740 1 -- 172.21.15.1:0/2426990952 shutdown_connections 2024-03-06T07:28:18.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.825+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/2426990952 >> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] conn(0x5611a9d8add0 0x5611a9da27c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.825+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/2426990952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d78130 0x5611a9d747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.825+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/2426990952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d52330 0x5611a9d78a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.825+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/2426990952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d77a30 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.825+0000 7f8b52fa1740 1 -- 172.21.15.1:0/2426990952 >> 172.21.15.1:0/2426990952 conn(0x5611a9d504b0 msgr2=0x5611a9d4b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.825+0000 7f8b297fa640 0 client.10263 ms_handle_reset on v2:172.21.15.1:6834/1525330395 2024-03-06T07:28:18.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.825+0000 7f8b52fa1740 1 -- 172.21.15.1:0/2426990952 shutdown_connections 2024-03-06T07:28:18.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.825+0000 7f8b52fa1740 1 -- 172.21.15.1:0/2426990952 wait complete. 2024-03-06T07:28:18.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.825+0000 7f8b52fa1740 20 client.10263 trim_cache size 0 max 16384 2024-03-06T07:28:18.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.848+0000 7f8b52fa1740 1 Processor -- start 2024-03-06T07:28:18.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.849+0000 7f8b52fa1740 1 -- start start 2024-03-06T07:28:18.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.849+0000 7f8b52fa1740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d51f50 0x5611a9d917f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.849+0000 7f8b52fa1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d52330 0x5611a9db15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.849+0000 7f8b52fa1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d78130 0x5611a9db1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.849+0000 7f8b52fa1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b481f7cb0 con 0x5611a9d78130 2024-03-06T07:28:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.849+0000 7f8b52fa1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b481eec70 con 0x5611a9d52330 2024-03-06T07:28:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.849+0000 7f8b52fa1740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b481f67a0 con 0x5611a9d51f50 2024-03-06T07:28:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.849+0000 7f8b4e87c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d78130 0x5611a9db1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.849+0000 7f8b4e87c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d78130 0x5611a9db1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51012/0 (socket says 172.21.15.1:51012) 2024-03-06T07:28:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.849+0000 7f8b4e87c640 1 -- 172.21.15.1:0/2064060710 learned_addr learned my addr 172.21.15.1:0/2064060710 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.849+0000 7f8b4e07b640 1 --2- 172.21.15.1:0/2064060710 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d51f50 0x5611a9d917f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.849+0000 7f8b4d87a640 1 --2- 172.21.15.1:0/2064060710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d52330 0x5611a9db15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.850+0000 7f8b4e87c640 1 -- 172.21.15.1:0/2064060710 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d51f50 msgr2=0x5611a9d917f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.850+0000 7f8b4e87c640 1 --2- 172.21.15.1:0/2064060710 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d51f50 0x5611a9d917f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.850+0000 7f8b4e87c640 1 -- 172.21.15.1:0/2064060710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d52330 msgr2=0x5611a9db15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.850+0000 7f8b4e87c640 1 --2- 172.21.15.1:0/2064060710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d52330 0x5611a9db15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.850+0000 7f8b4e87c640 1 -- 172.21.15.1:0/2064060710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611a9a956b0 con 0x5611a9d78130 2024-03-06T07:28:18.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.850+0000 7f8b4e07b640 1 --2- 172.21.15.1:0/2064060710 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d51f50 0x5611a9d917f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:18.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.850+0000 7f8b4e87c640 1 --2- 172.21.15.1:0/2064060710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d78130 0x5611a9db1af0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000f5e0 tx=0x7f8b4000faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2131a1d1fe61d6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.850+0000 7f8b2b7fe640 1 -- 172.21.15.1:0/2064060710 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4001e070 con 0x5611a9d78130 2024-03-06T07:28:18.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.850+0000 7f8b2b7fe640 1 -- 172.21.15.1:0/2064060710 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b40003b90 con 0x5611a9d78130 2024-03-06T07:28:18.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.850+0000 7f8b2b7fe640 1 -- 172.21.15.1:0/2064060710 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4000e7f0 con 0x5611a9d78130 2024-03-06T07:28:18.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.851+0000 7f8b52fa1740 1 -- 172.21.15.1:0/2064060710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d78130 msgr2=0x5611a9db1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.851+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/2064060710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d78130 0x5611a9db1af0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000f5e0 tx=0x7f8b4000faa0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.851+0000 7f8b52fa1740 1 -- 172.21.15.1:0/2064060710 shutdown_connections 2024-03-06T07:28:18.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.851+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/2064060710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d78130 0x5611a9db1af0 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.851+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/2064060710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d52330 0x5611a9db15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.851+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/2064060710 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d51f50 0x5611a9d917f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.851+0000 7f8b52fa1740 1 -- 172.21.15.1:0/2064060710 >> 172.21.15.1:0/2064060710 conn(0x5611a9d504b0 msgr2=0x5611a9d8ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.851+0000 7f8b52fa1740 1 -- 172.21.15.1:0/2064060710 shutdown_connections 2024-03-06T07:28:18.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.851+0000 7f8b52fa1740 1 -- 172.21.15.1:0/2064060710 wait complete. 2024-03-06T07:28:18.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.851+0000 7f8b52fa1740 1 -- start start 2024-03-06T07:28:18.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.851+0000 7f8b52fa1740 1 Processor -- start 2024-03-06T07:28:18.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.852+0000 7f8b52fa1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d8d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.852+0000 7f8b52fa1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d52330 0x5611a9d4c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.852+0000 7f8b52fa1740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d78130 0x5611a9d4b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.852+0000 7f8b52fa1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a9d75300 con 0x5611a9d51f50 2024-03-06T07:28:18.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.852+0000 7f8b52fa1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a9d9f470 con 0x5611a9d52330 2024-03-06T07:28:18.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.852+0000 7f8b52fa1740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5611a9d8b1c0 con 0x5611a9d78130 2024-03-06T07:28:18.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.852+0000 7f8b52fa1740 10 client.0 _ll_register_callbacks cb 0x7f8b44022770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:18.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.852+0000 7f8b4e07b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d8d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.852+0000 7f8b4e07b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d8d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51028/0 (socket says 172.21.15.1:51028) 2024-03-06T07:28:18.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.852+0000 7f8b4e07b640 1 -- 172.21.15.1:0/1544897276 learned_addr learned my addr 172.21.15.1:0/1544897276 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:18.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.852+0000 7f8b4d87a640 1 --2- 172.21.15.1:0/1544897276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d52330 0x5611a9d4c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.852+0000 7f8b4e87c640 1 --2- 172.21.15.1:0/1544897276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d78130 0x5611a9d4b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b4e07b640 1 -- 172.21.15.1:0/1544897276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d78130 msgr2=0x5611a9d4b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b4e07b640 1 --2- 172.21.15.1:0/1544897276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d78130 0x5611a9d4b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b4e07b640 1 -- 172.21.15.1:0/1544897276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d52330 msgr2=0x5611a9d4c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b4e07b640 1 --2- 172.21.15.1:0/1544897276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d52330 0x5611a9d4c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b4e07b640 1 -- 172.21.15.1:0/1544897276 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5611a9d922b0 con 0x5611a9d51f50 2024-03-06T07:28:18.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b4e07b640 1 --2- 172.21.15.1:0/1544897276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d8d4e0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f8b44023a80 tx=0x7f8b4401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10bbace35fb8b714 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b297fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:28:18.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b297fa640 1 -- 172.21.15.1:0/1544897276 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b44026040 con 0x5611a9d51f50 2024-03-06T07:28:18.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b297fa640 1 -- 172.21.15.1:0/1544897276 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f8b44010040 con 0x5611a9d51f50 2024-03-06T07:28:18.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b52fa1740 1 -- 172.21.15.1:0/1544897276 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8b40003b90 con 0x5611a9d51f50 2024-03-06T07:28:18.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b297fa640 1 -- 172.21.15.1:0/1544897276 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=179) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b44045050 con 0x5611a9d51f50 2024-03-06T07:28:18.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b52fa1740 10 client.10269 fetch_fsmap learned FSMap version 179 2024-03-06T07:28:18.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b52fa1740 1 -- 172.21.15.1:0/1544897276 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=179,monmap=2+}) v3 -- 0x5611a9da4e60 con 0x5611a9d51f50 2024-03-06T07:28:18.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b297fa640 1 -- 172.21.15.1:0/1544897276 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b40003b90 con 0x5611a9d51f50 2024-03-06T07:28:18.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b297fa640 1 -- 172.21.15.1:0/1544897276 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 179) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f8b44048020 con 0x5611a9d51f50 2024-03-06T07:28:18.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b52fa1740 10 client.10269 fetch_fsmap finished waiting for FSMap version 179 2024-03-06T07:28:18.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b52fa1740 10 client.10269 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:28:18.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b52fa1740 20 client.10269 populate_metadata read hostname 'smithi001' 2024-03-06T07:28:18.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.853+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/1544897276 >> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] conn(0x5611a9d4c050 0x5611a9dfaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.854+0000 7f8b52fa1740 4 client.10269 mds_command: new command op to 10161 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:28:18.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.854+0000 7f8b52fa1740 1 -- 172.21.15.1:0/1544897276 --> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5611a9cbdf40 con 0x5611a9d4c050 2024-03-06T07:28:18.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.854+0000 7f8b4d87a640 1 --2- 172.21.15.1:0/1544897276 >> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] conn(0x5611a9d4c050 0x5611a9dfaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.854+0000 7f8b297fa640 10 client.10269 ms_handle_connect on v2:172.21.15.1:6834/1525330395 2024-03-06T07:28:18.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.854+0000 7f8b4d87a640 1 --2- 172.21.15.1:0/1544897276 >> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] conn(0x5611a9d4c050 0x5611a9dfaa30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5578341137655fd4 server_cookie=eecf6fe6ae3d1339 in_seq=0 out_seq=0 2024-03-06T07:28:18.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.854+0000 7f8b297fa640 1 -- 172.21.15.1:0/1544897276 <== mds.0 v2:172.21.15.1:6834/1525330395 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x5611a9cbdf40 con 0x5611a9d4c050 2024-03-06T07:28:18.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.854+0000 7f8b297fa640 10 client.10269 handle_command_reply: tid=0 2024-03-06T07:28:18.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.854+0000 7f8b52fa1740 1 client.10269 shutdown 2024-03-06T07:28:18.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.854+0000 7f8b52fa1740 1 -- 172.21.15.1:0/1544897276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 msgr2=0x5611a9d8d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.854+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/1544897276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d8d4e0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f8b44023a80 tx=0x7f8b4401e000 comp rx=0 tx=0).stop 2024-03-06T07:28:18.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.854+0000 7f8b52fa1740 1 -- 172.21.15.1:0/1544897276 shutdown_connections 2024-03-06T07:28:18.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.854+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/1544897276 >> [v2:172.21.15.1:6834/1525330395,v1:172.21.15.1:6835/1525330395] conn(0x5611a9d4c050 0x5611a9dfaa30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.854+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/1544897276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5611a9d78130 0x5611a9d4b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.854+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/1544897276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5611a9d52330 0x5611a9d4c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.854+0000 7f8b297fa640 0 client.10269 ms_handle_reset on v2:172.21.15.1:6834/1525330395 2024-03-06T07:28:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.854+0000 7f8b52fa1740 1 --2- 172.21.15.1:0/1544897276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5611a9d51f50 0x5611a9d8d4e0 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.855+0000 7f8b52fa1740 1 -- 172.21.15.1:0/1544897276 >> 172.21.15.1:0/1544897276 conn(0x5611a9d504b0 msgr2=0x5611a9d72fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.855+0000 7f8b52fa1740 1 -- 172.21.15.1:0/1544897276 shutdown_connections 2024-03-06T07:28:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.855+0000 7f8b52fa1740 1 -- 172.21.15.1:0/1544897276 wait complete. 2024-03-06T07:28:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.855+0000 7f8b52fa1740 20 client.10269 trim_cache size 0 max 16384 2024-03-06T07:28:18.873 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:28:18.873 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:28:18.873 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:28:18.873 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 10161, 2024-03-06T07:28:18.873 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:28:18.873 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:28:18.873 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:28:18.873 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 3.9150692550000001, 2024-03-06T07:28:18.874 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 179, 2024-03-06T07:28:18.874 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 185, 2024-03-06T07:28:18.874 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 185, 2024-03-06T07:28:18.874 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 5.9138348839999999 2024-03-06T07:28:18.874 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:28:18.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.855+0000 7f8b502df640 1 -- 172.21.15.1:0/2047882671 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8b2c06b4d0 msgr2=0x7f8b2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.855+0000 7f8b502df640 1 --2- 172.21.15.1:0/2047882671 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8b2c06b4d0 0x7f8b2c06d990 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f8b44000f60 tx=0x7f8b44007d50 comp rx=0 tx=0).stop 2024-03-06T07:28:18.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.855+0000 7f8b502df640 1 -- 172.21.15.1:0/2047882671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b48152650 msgr2=0x7f8b48156490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.855+0000 7f8b502df640 1 --2- 172.21.15.1:0/2047882671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b48152650 0x7f8b48156490 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f8b38034040 tx=0x7f8b380046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.855+0000 7f8b502df640 1 -- 172.21.15.1:0/2047882671 shutdown_connections 2024-03-06T07:28:18.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.855+0000 7f8b502df640 1 --2- 172.21.15.1:0/2047882671 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8b2c06b4d0 0x7f8b2c06d990 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.855+0000 7f8b502df640 1 --2- 172.21.15.1:0/2047882671 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8b48154c00 0x7f8b481569d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.855+0000 7f8b502df640 1 --2- 172.21.15.1:0/2047882671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b48152650 0x7f8b48156490 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.855+0000 7f8b502df640 1 --2- 172.21.15.1:0/2047882671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b480bf9d0 0x7f8b481540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.855+0000 7f8b502df640 1 -- 172.21.15.1:0/2047882671 >> 172.21.15.1:0/2047882671 conn(0x7f8b48066ee0 msgr2=0x7f8b480c3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.855+0000 7f8b502df640 1 -- 172.21.15.1:0/2047882671 shutdown_connections 2024-03-06T07:28:18.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:18.855+0000 7f8b502df640 1 -- 172.21.15.1:0/2047882671 wait complete. 2024-03-06T07:28:18.877 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:28:18.878 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:18.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:18.960 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:28:18.960 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:28:18.961 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:18.977 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:28:18.977 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:28:18.977 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:19.018 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:28:19.018 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:19.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.120+0000 7fcb95913640 1 Processor -- start 2024-03-06T07:28:19.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.120+0000 7fcb95913640 1 -- start start 2024-03-06T07:28:19.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.122+0000 7fcb95913640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcb90152820 0x7fcb900c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.122+0000 7fcb95913640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb900af6f0 0x7fcb9015fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.122+0000 7fcb95913640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb900c7180 con 0x7fcb900c0670 2024-03-06T07:28:19.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.122+0000 7fcb95913640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb90155460 con 0x7fcb90152820 2024-03-06T07:28:19.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.122+0000 7fcb95913640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb900c6bc0 con 0x7fcb900af6f0 2024-03-06T07:28:19.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.122+0000 7fcb8effd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcb90152820 0x7fcb900c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.122+0000 7fcb8f7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb900af6f0 0x7fcb9015fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.122+0000 7fcb8effd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcb90152820 0x7fcb900c0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:43554/0 (socket says 172.21.15.1:43554) 2024-03-06T07:28:19.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.122+0000 7fcb8effd640 1 -- 172.21.15.1:0/1907857178 learned_addr learned my addr 172.21.15.1:0/1907857178 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:19.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.122+0000 7fcb8f7fe640 1 -- 172.21.15.1:0/1907857178 >> v2:172.21.15.98:3300/0 conn(0x7fcb90152820 msgr2=0x7fcb900c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.122+0000 7fcb8f7fe640 1 --2- 172.21.15.1:0/1907857178 >> v2:172.21.15.98:3300/0 conn(0x7fcb90152820 0x7fcb900c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.122+0000 7fcb8f7fe640 1 -- 172.21.15.1:0/1907857178 >> v1:172.21.15.98:6789/0 conn(0x7fcb900c0670 legacy=0x7fcb900c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.122+0000 7fcb8f7fe640 1 -- 172.21.15.1:0/1907857178 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb900c6ea0 con 0x7fcb900af6f0 2024-03-06T07:28:19.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.123+0000 7fcb8f7fe640 1 --2- 172.21.15.1:0/1907857178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb900af6f0 0x7fcb9015fb60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcb880097a0 tx=0x7fcb8802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7feb83567e4b05f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.123+0000 7fcb8dffb640 1 -- 172.21.15.1:0/1907857178 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88035020 con 0x7fcb900af6f0 2024-03-06T07:28:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.123+0000 7fcb8dffb640 1 -- 172.21.15.1:0/1907857178 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb88031830 con 0x7fcb900af6f0 2024-03-06T07:28:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.123+0000 7fcb8dffb640 1 -- 172.21.15.1:0/1907857178 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88031c00 con 0x7fcb900af6f0 2024-03-06T07:28:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.124+0000 7fcb95913640 1 -- 172.21.15.1:0/1907857178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb900af6f0 msgr2=0x7fcb9015fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.124+0000 7fcb95913640 1 --2- 172.21.15.1:0/1907857178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb900af6f0 0x7fcb9015fb60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcb880097a0 tx=0x7fcb8802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.124+0000 7fcb95913640 1 -- 172.21.15.1:0/1907857178 shutdown_connections 2024-03-06T07:28:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.124+0000 7fcb95913640 1 --2- 172.21.15.1:0/1907857178 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb900af6f0 0x7fcb9015fb60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.124+0000 7fcb95913640 1 --2- 172.21.15.1:0/1907857178 >> v2:172.21.15.98:3300/0 conn(0x7fcb90152820 0x7fcb900c0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.124+0000 7fcb95913640 1 -- 172.21.15.1:0/1907857178 >> 172.21.15.1:0/1907857178 conn(0x7fcb90072720 msgr2=0x7fcb90072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.124+0000 7fcb95913640 1 -- 172.21.15.1:0/1907857178 shutdown_connections 2024-03-06T07:28:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.124+0000 7fcb95913640 1 -- 172.21.15.1:0/1907857178 wait complete. 2024-03-06T07:28:19.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.126+0000 7fcb95913640 1 Processor -- start 2024-03-06T07:28:19.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.126+0000 7fcb95913640 1 -- start start 2024-03-06T07:28:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.127+0000 7fcb95913640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb900af6f0 0x7fcb900c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.127+0000 7fcb95913640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb900c0670 0x7fcb900ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.127+0000 7fcb95913640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb90152820 0x7fcb900ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.127+0000 7fcb95913640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb900c7c80 con 0x7fcb90152820 2024-03-06T07:28:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.127+0000 7fcb95913640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb900c7b00 con 0x7fcb900c0670 2024-03-06T07:28:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.127+0000 7fcb95913640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb90152c00 con 0x7fcb900af6f0 2024-03-06T07:28:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.127+0000 7fcb8f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb90152820 0x7fcb900ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.127+0000 7fcb8f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb90152820 0x7fcb900ae640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51032/0 (socket says 172.21.15.1:51032) 2024-03-06T07:28:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.127+0000 7fcb8f7fe640 1 -- 172.21.15.1:0/3806124475 learned_addr learned my addr 172.21.15.1:0/3806124475 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.127+0000 7fcb8e7fc640 1 --2- 172.21.15.1:0/3806124475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb900c0670 0x7fcb900ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.127+0000 7fcb8effd640 1 --2- 172.21.15.1:0/3806124475 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb900af6f0 0x7fcb900c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.128+0000 7fcb8f7fe640 1 -- 172.21.15.1:0/3806124475 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb900af6f0 msgr2=0x7fcb900c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.128+0000 7fcb8f7fe640 1 --2- 172.21.15.1:0/3806124475 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb900af6f0 0x7fcb900c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.128+0000 7fcb8f7fe640 1 -- 172.21.15.1:0/3806124475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb900c0670 msgr2=0x7fcb900ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.128+0000 7fcb8f7fe640 1 --2- 172.21.15.1:0/3806124475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb900c0670 0x7fcb900ae100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.128+0000 7fcb8f7fe640 1 -- 172.21.15.1:0/3806124475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb90158540 con 0x7fcb90152820 2024-03-06T07:28:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.128+0000 7fcb8f7fe640 1 --2- 172.21.15.1:0/3806124475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb90152820 0x7fcb900ae640 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7fcb88034040 tx=0x7fcb880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.128+0000 7fcb87fff640 1 -- 172.21.15.1:0/3806124475 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88040070 con 0x7fcb90152820 2024-03-06T07:28:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.128+0000 7fcb87fff640 1 -- 172.21.15.1:0/3806124475 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8803a850 con 0x7fcb90152820 2024-03-06T07:28:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.128+0000 7fcb87fff640 1 -- 172.21.15.1:0/3806124475 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88035040 con 0x7fcb90152820 2024-03-06T07:28:19.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.128+0000 7fcb95913640 1 -- 172.21.15.1:0/3806124475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb901ee850 con 0x7fcb90152820 2024-03-06T07:28:19.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.128+0000 7fcb95913640 1 -- 172.21.15.1:0/3806124475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb901587d0 con 0x7fcb90152820 2024-03-06T07:28:19.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.129+0000 7fcb95913640 1 -- 172.21.15.1:0/3806124475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb900c1ec0 con 0x7fcb90152820 2024-03-06T07:28:19.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.130+0000 7fcb87fff640 1 -- 172.21.15.1:0/3806124475 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb88038020 con 0x7fcb90152820 2024-03-06T07:28:19.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.130+0000 7fcb87fff640 1 --2- 172.21.15.1:0/3806124475 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcb7006b4d0 0x7fcb7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.130+0000 7fcb87fff640 1 -- 172.21.15.1:0/3806124475 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7fcb880abbc0 con 0x7fcb90152820 2024-03-06T07:28:19.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.131+0000 7fcb8effd640 1 --2- 172.21.15.1:0/3806124475 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcb7006b4d0 0x7fcb7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.131+0000 7fcb8effd640 1 --2- 172.21.15.1:0/3806124475 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcb7006b4d0 0x7fcb7006d990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fcb90150af0 tx=0x7fcb8000b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.139+0000 7fcb87fff640 1 -- 172.21.15.1:0/3806124475 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb900c1ec0 con 0x7fcb90152820 2024-03-06T07:28:19.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.360+0000 7fcb95913640 1 -- 172.21.15.1:0/3806124475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb900686e0 con 0x7fcb90152820 2024-03-06T07:28:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.361+0000 7fcb87fff640 1 -- 172.21.15.1:0/3806124475 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+5005 (secure 0 0 0) 0x7fcb88080030 con 0x7fcb90152820 2024-03-06T07:28:19.363 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:19.363 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":179,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:28:16.947047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10161,"mds_1":9951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9951":{"gid":9951,"name":"a","rank":1,"incarnation":178,"state":"up:active","state_seq":9,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10161":{"gid":10161,"name":"c","rank":0,"incarnation":176,"state":"up:active","state_seq":2,"addr":"172.21.15.1:6835/1525330395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1525330395},{"type":"v1","addr":"172.21.15.1:6835","nonce":1525330395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10161,"qdb_cluster":[9951,10161]},"id":14}]} 2024-03-06T07:28:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 179 2024-03-06T07:28:19.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.363+0000 7fcb95913640 1 -- 172.21.15.1:0/3806124475 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcb7006b4d0 msgr2=0x7fcb7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.363+0000 7fcb95913640 1 --2- 172.21.15.1:0/3806124475 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcb7006b4d0 0x7fcb7006d990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fcb90150af0 tx=0x7fcb8000b040 comp rx=0 tx=0).stop 2024-03-06T07:28:19.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.363+0000 7fcb95913640 1 -- 172.21.15.1:0/3806124475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb90152820 msgr2=0x7fcb900ae640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.363+0000 7fcb95913640 1 --2- 172.21.15.1:0/3806124475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb90152820 0x7fcb900ae640 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7fcb88034040 tx=0x7fcb880046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.363+0000 7fcb95913640 1 -- 172.21.15.1:0/3806124475 shutdown_connections 2024-03-06T07:28:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.363+0000 7fcb95913640 1 --2- 172.21.15.1:0/3806124475 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcb7006b4d0 0x7fcb7006d990 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.363+0000 7fcb95913640 1 --2- 172.21.15.1:0/3806124475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb90152820 0x7fcb900ae640 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.363+0000 7fcb95913640 1 --2- 172.21.15.1:0/3806124475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb900c0670 0x7fcb900ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.363+0000 7fcb95913640 1 --2- 172.21.15.1:0/3806124475 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb900af6f0 0x7fcb900c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.363+0000 7fcb95913640 1 -- 172.21.15.1:0/3806124475 >> 172.21.15.1:0/3806124475 conn(0x7fcb90072720 msgr2=0x7fcb901500b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.363+0000 7fcb95913640 1 -- 172.21.15.1:0/3806124475 shutdown_connections 2024-03-06T07:28:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.363+0000 7fcb95913640 1 -- 172.21.15.1:0/3806124475 wait complete. 2024-03-06T07:28:19.376 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:19.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.475+0000 7fe885cd4640 1 Processor -- start 2024-03-06T07:28:19.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.475+0000 7fe885cd4640 1 -- start start 2024-03-06T07:28:19.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.476+0000 7fe885cd4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8800c6b10 0x7fe880155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.476+0000 7fe885cd4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe8800bf9d0 0x7fe8800c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.476+0000 7fe885cd4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8800c2fc0 con 0x7fe880152650 2024-03-06T07:28:19.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.476+0000 7fe885cd4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe880155800 con 0x7fe8800bf9d0 2024-03-06T07:28:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.476+0000 7fe885cd4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe880158da0 con 0x7fe8800c6b10 2024-03-06T07:28:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.476+0000 7fe87ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe8800bf9d0 0x7fe8800c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.476+0000 7fe87f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe880152650 0x7fe880158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39752/0 (socket says 172.21.15.1:39752) 2024-03-06T07:28:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.476+0000 7fe87ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe8800bf9d0 0x7fe8800c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51046/0 (socket says 172.21.15.1:51046) 2024-03-06T07:28:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.476+0000 7fe87f7fe640 1 -- 172.21.15.1:0/2691743749 learned_addr learned my addr 172.21.15.1:0/2691743749 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.476+0000 7fe87effd640 1 --2- 172.21.15.1:0/2691743749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8800c6b10 0x7fe880155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.476+0000 7fe87ffff640 1 -- 172.21.15.1:0/2691743749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8800c6b10 msgr2=0x7fe880155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.476+0000 7fe87ffff640 1 --2- 172.21.15.1:0/2691743749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8800c6b10 0x7fe880155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.476+0000 7fe87ffff640 1 -- 172.21.15.1:0/2691743749 >> v1:172.21.15.1:6789/0 conn(0x7fe880152650 legacy=0x7fe880158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.476+0000 7fe87ffff640 1 -- 172.21.15.1:0/2691743749 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8800c6690 con 0x7fe8800bf9d0 2024-03-06T07:28:19.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.477+0000 7fe87ffff640 1 --2- 172.21.15.1:0/2691743749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8800bf9d0 0x7fe8800c1460 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fe878009d90 tx=0x7fe87802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3ffdb6a955c398 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.477+0000 7fe87e7fc640 1 -- 172.21.15.1:0/2691743749 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe878039070 con 0x7fe8800bf9d0 2024-03-06T07:28:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.477+0000 7fe87e7fc640 1 -- 172.21.15.1:0/2691743749 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87802cbc0 con 0x7fe8800bf9d0 2024-03-06T07:28:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.477+0000 7fe87e7fc640 1 -- 172.21.15.1:0/2691743749 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe878034440 con 0x7fe8800bf9d0 2024-03-06T07:28:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.478+0000 7fe885cd4640 1 -- 172.21.15.1:0/2691743749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8800bf9d0 msgr2=0x7fe8800c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.478+0000 7fe885cd4640 1 --2- 172.21.15.1:0/2691743749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8800bf9d0 0x7fe8800c1460 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fe878009d90 tx=0x7fe87802be10 comp rx=0 tx=0).stop 2024-03-06T07:28:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.478+0000 7fe885cd4640 1 -- 172.21.15.1:0/2691743749 shutdown_connections 2024-03-06T07:28:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.478+0000 7fe885cd4640 1 --2- 172.21.15.1:0/2691743749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8800c6b10 0x7fe880155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.478+0000 7fe885cd4640 1 --2- 172.21.15.1:0/2691743749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8800bf9d0 0x7fe8800c1460 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.478+0000 7fe885cd4640 1 -- 172.21.15.1:0/2691743749 >> 172.21.15.1:0/2691743749 conn(0x7fe880066ee0 msgr2=0x7fe8800672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.478+0000 7fe885cd4640 1 -- 172.21.15.1:0/2691743749 shutdown_connections 2024-03-06T07:28:19.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.478+0000 7fe885cd4640 1 -- 172.21.15.1:0/2691743749 wait complete. 2024-03-06T07:28:19.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.479+0000 7fe885cd4640 1 Processor -- start 2024-03-06T07:28:19.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.479+0000 7fe885cd4640 1 -- start start 2024-03-06T07:28:19.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.480+0000 7fe885cd4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8800bf9d0 0x7fe8801eb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.480+0000 7fe885cd4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe880152650 0x7fe88014ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.480+0000 7fe885cd4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe88014d2d0 0x7fe88014d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.480+0000 7fe885cd4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8801551b0 con 0x7fe8800bf9d0 2024-03-06T07:28:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.480+0000 7fe885cd4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8800c6fa0 con 0x7fe880152650 2024-03-06T07:28:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.480+0000 7fe885cd4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe880155490 con 0x7fe88014d2d0 2024-03-06T07:28:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.480+0000 7fe87f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8800bf9d0 0x7fe8801eb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.480+0000 7fe87f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8800bf9d0 0x7fe8801eb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51050/0 (socket says 172.21.15.1:51050) 2024-03-06T07:28:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.480+0000 7fe87f7fe640 1 -- 172.21.15.1:0/3141329809 learned_addr learned my addr 172.21.15.1:0/3141329809 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.480+0000 7fe87f7fe640 1 -- 172.21.15.1:0/3141329809 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe88014d2d0 msgr2=0x7fe88014d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.480+0000 7fe87f7fe640 1 --2- 172.21.15.1:0/3141329809 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe88014d2d0 0x7fe88014d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.480+0000 7fe87f7fe640 1 -- 172.21.15.1:0/3141329809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe880152650 msgr2=0x7fe88014ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.480+0000 7fe87f7fe640 1 --2- 172.21.15.1:0/3141329809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe880152650 0x7fe88014ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.480+0000 7fe87f7fe640 1 -- 172.21.15.1:0/3141329809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe880154a80 con 0x7fe8800bf9d0 2024-03-06T07:28:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.480+0000 7fe87f7fe640 1 --2- 172.21.15.1:0/3141329809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8800bf9d0 0x7fe8801eb6f0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fe87000c430 tx=0x7fe87000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.480+0000 7fe877fff640 1 -- 172.21.15.1:0/3141329809 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe870017070 con 0x7fe8800bf9d0 2024-03-06T07:28:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.481+0000 7fe877fff640 1 -- 172.21.15.1:0/3141329809 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe870005ce0 con 0x7fe8800bf9d0 2024-03-06T07:28:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.481+0000 7fe885cd4640 1 -- 172.21.15.1:0/3141329809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8801541e0 con 0x7fe8800bf9d0 2024-03-06T07:28:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.481+0000 7fe877fff640 1 -- 172.21.15.1:0/3141329809 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8700072f0 con 0x7fe8800bf9d0 2024-03-06T07:28:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.481+0000 7fe885cd4640 1 -- 172.21.15.1:0/3141329809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8801547a0 con 0x7fe8800bf9d0 2024-03-06T07:28:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.482+0000 7fe885cd4640 1 -- 172.21.15.1:0/3141329809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe880067fd0 con 0x7fe8800bf9d0 2024-03-06T07:28:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.482+0000 7fe877fff640 1 -- 172.21.15.1:0/3141329809 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe87001a070 con 0x7fe8800bf9d0 2024-03-06T07:28:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.483+0000 7fe877fff640 1 --2- 172.21.15.1:0/3141329809 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe86006b4d0 0x7fe86006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.483+0000 7fe877fff640 1 -- 172.21.15.1:0/3141329809 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7fe870089790 con 0x7fe8800bf9d0 2024-03-06T07:28:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.483+0000 7fe87effd640 1 --2- 172.21.15.1:0/3141329809 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe86006b4d0 0x7fe86006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.483+0000 7fe87effd640 1 --2- 172.21.15.1:0/3141329809 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe86006b4d0 0x7fe86006d990 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fe868006a90 tx=0x7fe868006a20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.487+0000 7fe877fff640 1 -- 172.21.15.1:0/3141329809 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8700584a0 con 0x7fe8800bf9d0 2024-03-06T07:28:19.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.714+0000 7fe885cd4640 1 -- 172.21.15.1:0/3141329809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe880152f30 con 0x7fe8800bf9d0 2024-03-06T07:28:19.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.715+0000 7fe877fff640 1 -- 172.21.15.1:0/3141329809 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) v1 ==== 74+0+16947 (secure 0 0 0) 0x7fe870057da0 con 0x7fe8800bf9d0 2024-03-06T07:28:19.717 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:19.717 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":185,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:28:10.884250+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:50.456844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:51.147612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"29.0","mappings":[{"from":6,"to":0}]},{"pgid":"29.c","mappings":[{"from":6,"to":0}]},{"pgid":"29.f","mappings":[{"from":6,"to":0}]},{"pgid":"29.16","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.717+0000 7fe885cd4640 1 -- 172.21.15.1:0/3141329809 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe86006b4d0 msgr2=0x7fe86006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.717+0000 7fe885cd4640 1 --2- 172.21.15.1:0/3141329809 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe86006b4d0 0x7fe86006d990 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fe868006a90 tx=0x7fe868006a20 comp rx=0 tx=0).stop 2024-03-06T07:28:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.717+0000 7fe885cd4640 1 -- 172.21.15.1:0/3141329809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8800bf9d0 msgr2=0x7fe8801eb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.717+0000 7fe885cd4640 1 --2- 172.21.15.1:0/3141329809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8800bf9d0 0x7fe8801eb6f0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fe87000c430 tx=0x7fe87000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.717+0000 7fe885cd4640 1 -- 172.21.15.1:0/3141329809 shutdown_connections 2024-03-06T07:28:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.717+0000 7fe885cd4640 1 --2- 172.21.15.1:0/3141329809 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe86006b4d0 0x7fe86006d990 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.717+0000 7fe885cd4640 1 --2- 172.21.15.1:0/3141329809 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe88014d2d0 0x7fe88014d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.717+0000 7fe885cd4640 1 --2- 172.21.15.1:0/3141329809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe880152650 0x7fe88014ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.717+0000 7fe885cd4640 1 --2- 172.21.15.1:0/3141329809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8800bf9d0 0x7fe8801eb6f0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.717+0000 7fe885cd4640 1 -- 172.21.15.1:0/3141329809 >> 172.21.15.1:0/3141329809 conn(0x7fe880066ee0 msgr2=0x7fe8800c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.718+0000 7fe885cd4640 1 -- 172.21.15.1:0/3141329809 shutdown_connections 2024-03-06T07:28:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.718+0000 7fe885cd4640 1 -- 172.21.15.1:0/3141329809 wait complete. 2024-03-06T07:28:19.731 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:50.456844+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '182', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:51.147612+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '183', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:19.731 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:19.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.838+0000 7f7463a88640 1 Processor -- start 2024-03-06T07:28:19.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.838+0000 7f7463a88640 1 -- start start 2024-03-06T07:28:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.840+0000 7f7463a88640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f745c0c6b10 0x7f745c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.840+0000 7f7463a88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f745c0bf9d0 0x7f745c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.840+0000 7f7463a88640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f745c0c2fc0 con 0x7f745c152650 2024-03-06T07:28:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.840+0000 7f7463a88640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f745c155800 con 0x7f745c0c6b10 2024-03-06T07:28:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.840+0000 7f7463a88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745c158da0 con 0x7f745c0bf9d0 2024-03-06T07:28:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.840+0000 7f7461023640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f745c0c6b10 0x7f745c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.840+0000 7f7461824640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f745c152650 0x7f745c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39758/0 (socket says 172.21.15.1:39758) 2024-03-06T07:28:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.840+0000 7f7461824640 1 -- 172.21.15.1:0/2261031143 learned_addr learned my addr 172.21.15.1:0/2261031143 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.840+0000 7f7462025640 1 --2- 172.21.15.1:0/2261031143 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f745c0bf9d0 0x7f745c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.840+0000 7f7461023640 1 -- 172.21.15.1:0/2261031143 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f745c0bf9d0 msgr2=0x7f745c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.840+0000 7f7461023640 1 --2- 172.21.15.1:0/2261031143 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f745c0bf9d0 0x7f745c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.840+0000 7f7461023640 1 -- 172.21.15.1:0/2261031143 >> v1:172.21.15.1:6789/0 conn(0x7f745c152650 legacy=0x7f745c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.840+0000 7f7460822640 1 -- 172.21.15.1:0/2261031143 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514413234 0 0) 0x7f745c0c2fc0 con 0x7f745c152650 2024-03-06T07:28:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.840+0000 7f7461023640 1 -- 172.21.15.1:0/2261031143 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c0c6690 con 0x7f745c0c6b10 2024-03-06T07:28:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.841+0000 7f7461023640 1 --2- 172.21.15.1:0/2261031143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f745c0c6b10 0x7f745c155c90 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f744c0097a0 tx=0x7f744c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=877ff595a5aa510 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.841+0000 7f7460822640 1 -- 172.21.15.1:0/2261031143 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c0315a0 con 0x7f745c0c6b10 2024-03-06T07:28:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.841+0000 7f7460822640 1 -- 172.21.15.1:0/2261031143 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744c037070 con 0x7f745c0c6b10 2024-03-06T07:28:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.841+0000 7f7460822640 1 -- 172.21.15.1:0/2261031143 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c031960 con 0x7f745c0c6b10 2024-03-06T07:28:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.842+0000 7f7463a88640 1 -- 172.21.15.1:0/2261031143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f745c0c6b10 msgr2=0x7f745c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.842+0000 7f7463a88640 1 --2- 172.21.15.1:0/2261031143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f745c0c6b10 0x7f745c155c90 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f744c0097a0 tx=0x7f744c02b840 comp rx=0 tx=0).stop 2024-03-06T07:28:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.842+0000 7f7463a88640 1 -- 172.21.15.1:0/2261031143 shutdown_connections 2024-03-06T07:28:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.842+0000 7f7463a88640 1 --2- 172.21.15.1:0/2261031143 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f745c0bf9d0 0x7f745c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.842+0000 7f7463a88640 1 --2- 172.21.15.1:0/2261031143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f745c0c6b10 0x7f745c155c90 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.842+0000 7f7463a88640 1 -- 172.21.15.1:0/2261031143 >> 172.21.15.1:0/2261031143 conn(0x7f745c066ee0 msgr2=0x7f745c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.842+0000 7f7463a88640 1 -- 172.21.15.1:0/2261031143 shutdown_connections 2024-03-06T07:28:19.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.842+0000 7f7463a88640 1 -- 172.21.15.1:0/2261031143 wait complete. 2024-03-06T07:28:19.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.843+0000 7f7463a88640 1 Processor -- start 2024-03-06T07:28:19.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.843+0000 7f7463a88640 1 -- start start 2024-03-06T07:28:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7463a88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f745c0bf9d0 0x7f745c14d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7463a88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f745c152650 0x7f745c14ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7463a88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f745c14b700 0x7f745c14b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7463a88640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c1551b0 con 0x7f745c14b700 2024-03-06T07:28:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7463a88640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c0c6fa0 con 0x7f745c0bf9d0 2024-03-06T07:28:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7463a88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745c155490 con 0x7f745c152650 2024-03-06T07:28:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7462025640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f745c14b700 0x7f745c14b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7462025640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f745c14b700 0x7f745c14b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51078/0 (socket says 172.21.15.1:51078) 2024-03-06T07:28:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7462025640 1 -- 172.21.15.1:0/2169564070 learned_addr learned my addr 172.21.15.1:0/2169564070 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7461023640 1 --2- 172.21.15.1:0/2169564070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f745c152650 0x7f745c14ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7461824640 1 --2- 172.21.15.1:0/2169564070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f745c0bf9d0 0x7f745c14d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7462025640 1 -- 172.21.15.1:0/2169564070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f745c152650 msgr2=0x7f745c14ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7462025640 1 --2- 172.21.15.1:0/2169564070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f745c152650 0x7f745c14ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7462025640 1 -- 172.21.15.1:0/2169564070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f745c0bf9d0 msgr2=0x7f745c14d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7462025640 1 --2- 172.21.15.1:0/2169564070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f745c0bf9d0 0x7f745c14d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7462025640 1 -- 172.21.15.1:0/2169564070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c1eb710 con 0x7f745c14b700 2024-03-06T07:28:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7461824640 1 --2- 172.21.15.1:0/2169564070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f745c0bf9d0 0x7f745c14d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7462025640 1 --2- 172.21.15.1:0/2169564070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f745c14b700 0x7f745c14b140 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f7454007ec0 tx=0x7f745400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f74527fc640 1 -- 172.21.15.1:0/2169564070 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454014420 con 0x7f745c14b700 2024-03-06T07:28:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f74527fc640 1 -- 172.21.15.1:0/2169564070 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7454019070 con 0x7f745c14b700 2024-03-06T07:28:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.844+0000 7f7463a88640 1 -- 172.21.15.1:0/2169564070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f745c1eb990 con 0x7f745c14b700 2024-03-06T07:28:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.845+0000 7f74527fc640 1 -- 172.21.15.1:0/2169564070 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454014820 con 0x7f745c14b700 2024-03-06T07:28:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.845+0000 7f7463a88640 1 -- 172.21.15.1:0/2169564070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f745c151340 con 0x7f745c14b700 2024-03-06T07:28:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.846+0000 7f7463a88640 1 -- 172.21.15.1:0/2169564070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f745c152200 con 0x7f745c14b700 2024-03-06T07:28:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.846+0000 7f74527fc640 1 -- 172.21.15.1:0/2169564070 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74540149c0 con 0x7f745c14b700 2024-03-06T07:28:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.847+0000 7f74527fc640 1 --2- 172.21.15.1:0/2169564070 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f744006b500 0x7f744006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.847+0000 7f74527fc640 1 -- 172.21.15.1:0/2169564070 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f745408aa10 con 0x7f745c14b700 2024-03-06T07:28:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.847+0000 7f7461824640 1 --2- 172.21.15.1:0/2169564070 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f744006b500 0x7f744006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.847+0000 7f7461824640 1 --2- 172.21.15.1:0/2169564070 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f744006b500 0x7f744006d9c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f7458007fd0 tx=0x7f7458009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:19.851+0000 7f74527fc640 1 -- 172.21.15.1:0/2169564070 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f745c152200 con 0x7f745c14b700 2024-03-06T07:28:19.917 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-06T07:28:19.915+0000 7f6a441f2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:19.917 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-06T07:28:19.915+0000 7f6a441f2640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:28:20.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.073+0000 7f7463a88640 1 -- 172.21.15.1:0/2169564070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f745c067fd0 con 0x7f745c14b700 2024-03-06T07:28:20.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.074+0000 7f74527fc640 1 -- 172.21.15.1:0/2169564070 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v185) v1 ==== 93+0+10 (secure 0 0 0) 0x7f745c1eb710 con 0x7f745c14b700 2024-03-06T07:28:20.076 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:28:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.076+0000 7f7463a88640 1 -- 172.21.15.1:0/2169564070 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f744006b500 msgr2=0x7f744006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.076+0000 7f7463a88640 1 --2- 172.21.15.1:0/2169564070 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f744006b500 0x7f744006d9c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f7458007fd0 tx=0x7f7458009040 comp rx=0 tx=0).stop 2024-03-06T07:28:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.076+0000 7f7463a88640 1 -- 172.21.15.1:0/2169564070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f745c14b700 msgr2=0x7f745c14b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.076+0000 7f7463a88640 1 --2- 172.21.15.1:0/2169564070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f745c14b700 0x7f745c14b140 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f7454007ec0 tx=0x7f745400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.076+0000 7f7463a88640 1 -- 172.21.15.1:0/2169564070 shutdown_connections 2024-03-06T07:28:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.076+0000 7f7463a88640 1 --2- 172.21.15.1:0/2169564070 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f744006b500 0x7f744006d9c0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.076+0000 7f7463a88640 1 --2- 172.21.15.1:0/2169564070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f745c14b700 0x7f745c14b140 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.076+0000 7f7463a88640 1 --2- 172.21.15.1:0/2169564070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f745c152650 0x7f745c14ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.076+0000 7f7463a88640 1 --2- 172.21.15.1:0/2169564070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f745c0bf9d0 0x7f745c14d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.076+0000 7f7463a88640 1 -- 172.21.15.1:0/2169564070 >> 172.21.15.1:0/2169564070 conn(0x7f745c066ee0 msgr2=0x7f745c0cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.076+0000 7f7463a88640 1 -- 172.21.15.1:0/2169564070 shutdown_connections 2024-03-06T07:28:20.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.076+0000 7f7463a88640 1 -- 172.21.15.1:0/2169564070 wait complete. 2024-03-06T07:28:20.089 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:20.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.193+0000 7ff8a909a640 1 Processor -- start 2024-03-06T07:28:20.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.193+0000 7ff8a909a640 1 -- start start 2024-03-06T07:28:20.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.194+0000 7ff8a909a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff8a40c8590 0x7ff8a40c8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.194+0000 7ff8a909a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff8a40c3210 0x7ff8a41564d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.194+0000 7ff8a909a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a4156a10 0x7ff8a40c6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.194+0000 7ff8a909a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a4153a20 con 0x7ff8a40c8590 2024-03-06T07:28:20.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.194+0000 7ff8a909a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a40c9010 con 0x7ff8a40c3210 2024-03-06T07:28:20.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.194+0000 7ff8a909a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a41530f0 con 0x7ff8a4156a10 2024-03-06T07:28:20.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.194+0000 7ff8a2d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff8a40c8590 0x7ff8a40c8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.194+0000 7ff8a2d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff8a40c8590 0x7ff8a40c8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51088/0 (socket says 172.21.15.1:51088) 2024-03-06T07:28:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.194+0000 7ff8a2d9d640 1 -- 172.21.15.1:0/2223347994 learned_addr learned my addr 172.21.15.1:0/2223347994 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.194+0000 7ff8a259c640 1 --2- 172.21.15.1:0/2223347994 >> v2:172.21.15.98:3300/0 conn(0x7ff8a40c3210 0x7ff8a41564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.194+0000 7ff8a359e640 1 --2- 172.21.15.1:0/2223347994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a4156a10 0x7ff8a40c6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.194+0000 7ff8a2d9d640 1 -- 172.21.15.1:0/2223347994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a4156a10 msgr2=0x7ff8a40c6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.194+0000 7ff8a2d9d640 1 --2- 172.21.15.1:0/2223347994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a4156a10 0x7ff8a40c6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.194+0000 7ff8a2d9d640 1 -- 172.21.15.1:0/2223347994 >> v2:172.21.15.98:3300/0 conn(0x7ff8a40c3210 msgr2=0x7ff8a41564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.194+0000 7ff8a2d9d640 1 --2- 172.21.15.1:0/2223347994 >> v2:172.21.15.98:3300/0 conn(0x7ff8a40c3210 0x7ff8a41564d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.194+0000 7ff8a2d9d640 1 -- 172.21.15.1:0/2223347994 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a41533d0 con 0x7ff8a40c8590 2024-03-06T07:28:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.194+0000 7ff8a259c640 1 --2- 172.21.15.1:0/2223347994 >> v2:172.21.15.98:3300/0 conn(0x7ff8a40c3210 0x7ff8a41564d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.195+0000 7ff8a2d9d640 1 --2- 172.21.15.1:0/2223347994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a40c8590 0x7ff8a40c8970 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7ff8a40c92f0 tx=0x7ff89402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d386dc9b27f2e8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.195+0000 7ff8a1d9b640 1 -- 172.21.15.1:0/2223347994 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff894035020 con 0x7ff8a40c8590 2024-03-06T07:28:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.195+0000 7ff8a1d9b640 1 -- 172.21.15.1:0/2223347994 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89402d040 con 0x7ff8a40c8590 2024-03-06T07:28:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.195+0000 7ff8a1d9b640 1 -- 172.21.15.1:0/2223347994 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff894031a80 con 0x7ff8a40c8590 2024-03-06T07:28:20.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.196+0000 7ff8a909a640 1 -- 172.21.15.1:0/2223347994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a40c8590 msgr2=0x7ff8a40c8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.196+0000 7ff8a909a640 1 --2- 172.21.15.1:0/2223347994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a40c8590 0x7ff8a40c8970 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7ff8a40c92f0 tx=0x7ff89402b900 comp rx=0 tx=0).stop 2024-03-06T07:28:20.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.196+0000 7ff8a909a640 1 -- 172.21.15.1:0/2223347994 shutdown_connections 2024-03-06T07:28:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.196+0000 7ff8a909a640 1 --2- 172.21.15.1:0/2223347994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a4156a10 0x7ff8a40c6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.196+0000 7ff8a909a640 1 --2- 172.21.15.1:0/2223347994 >> v2:172.21.15.98:3300/0 conn(0x7ff8a40c3210 0x7ff8a41564d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.196+0000 7ff8a909a640 1 --2- 172.21.15.1:0/2223347994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a40c8590 0x7ff8a40c8970 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.196+0000 7ff8a909a640 1 -- 172.21.15.1:0/2223347994 >> 172.21.15.1:0/2223347994 conn(0x7ff8a40aa7a0 msgr2=0x7ff8a40aab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.196+0000 7ff8a909a640 1 -- 172.21.15.1:0/2223347994 shutdown_connections 2024-03-06T07:28:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.196+0000 7ff8a909a640 1 -- 172.21.15.1:0/2223347994 wait complete. 2024-03-06T07:28:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.197+0000 7ff8a909a640 1 Processor -- start 2024-03-06T07:28:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.197+0000 7ff8a909a640 1 -- start start 2024-03-06T07:28:20.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.198+0000 7ff8a909a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a40c3210 0x7ff8a41ebf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.198+0000 7ff8a909a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a40c8590 0x7ff8a4151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.198+0000 7ff8a909a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a4156a10 0x7ff8a4151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.198+0000 7ff8a909a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a40701e0 con 0x7ff8a40c8590 2024-03-06T07:28:20.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.198+0000 7ff8a909a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a4070060 con 0x7ff8a4156a10 2024-03-06T07:28:20.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.198+0000 7ff8a909a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a40c3650 con 0x7ff8a40c3210 2024-03-06T07:28:20.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.198+0000 7ff8a259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a40c8590 0x7ff8a4151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.198+0000 7ff8a259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a40c8590 0x7ff8a4151780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51090/0 (socket says 172.21.15.1:51090) 2024-03-06T07:28:20.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.198+0000 7ff8a259c640 1 -- 172.21.15.1:0/3966059041 learned_addr learned my addr 172.21.15.1:0/3966059041 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:20.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.198+0000 7ff8a2d9d640 1 --2- 172.21.15.1:0/3966059041 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a40c3210 0x7ff8a41ebf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.198+0000 7ff8a359e640 1 --2- 172.21.15.1:0/3966059041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a4156a10 0x7ff8a4151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.198+0000 7ff8a259c640 1 -- 172.21.15.1:0/3966059041 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a40c3210 msgr2=0x7ff8a41ebf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.198+0000 7ff8a259c640 1 --2- 172.21.15.1:0/3966059041 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a40c3210 0x7ff8a41ebf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.199+0000 7ff8a259c640 1 -- 172.21.15.1:0/3966059041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a4156a10 msgr2=0x7ff8a4151cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.199+0000 7ff8a259c640 1 --2- 172.21.15.1:0/3966059041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a4156a10 0x7ff8a4151cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.199+0000 7ff8a259c640 1 -- 172.21.15.1:0/3966059041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a41ec7c0 con 0x7ff8a40c8590 2024-03-06T07:28:20.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.199+0000 7ff8a359e640 1 --2- 172.21.15.1:0/3966059041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a4156a10 0x7ff8a4151cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:20.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.199+0000 7ff8a259c640 1 --2- 172.21.15.1:0/3966059041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a40c8590 0x7ff8a4151780 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7ff89800d800 tx=0x7ff89800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.199+0000 7ff88b7fe640 1 -- 172.21.15.1:0/3966059041 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898018070 con 0x7ff8a40c8590 2024-03-06T07:28:20.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.199+0000 7ff88b7fe640 1 -- 172.21.15.1:0/3966059041 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff898004030 con 0x7ff8a40c8590 2024-03-06T07:28:20.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.199+0000 7ff8a909a640 1 -- 172.21.15.1:0/3966059041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a41ecaa0 con 0x7ff8a40c8590 2024-03-06T07:28:20.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.199+0000 7ff88b7fe640 1 -- 172.21.15.1:0/3966059041 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898004400 con 0x7ff8a40c8590 2024-03-06T07:28:20.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.199+0000 7ff8a909a640 1 -- 172.21.15.1:0/3966059041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a40a8970 con 0x7ff8a40c8590 2024-03-06T07:28:20.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.200+0000 7ff8a909a640 1 -- 172.21.15.1:0/3966059041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a40686e0 con 0x7ff8a40c8590 2024-03-06T07:28:20.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.200+0000 7ff88b7fe640 1 -- 172.21.15.1:0/3966059041 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8980096a0 con 0x7ff8a40c8590 2024-03-06T07:28:20.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.201+0000 7ff88b7fe640 1 --2- 172.21.15.1:0/3966059041 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff89006b4d0 0x7ff89006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.201+0000 7ff88b7fe640 1 -- 172.21.15.1:0/3966059041 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7ff89808a7a0 con 0x7ff8a40c8590 2024-03-06T07:28:20.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.201+0000 7ff8a2d9d640 1 --2- 172.21.15.1:0/3966059041 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff89006b4d0 0x7ff89006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.202+0000 7ff8a2d9d640 1 --2- 172.21.15.1:0/3966059041 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff89006b4d0 0x7ff89006d990 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7ff894036000 tx=0x7ff89403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.205+0000 7ff88b7fe640 1 -- 172.21.15.1:0/3966059041 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff898059420 con 0x7ff8a40c8590 2024-03-06T07:28:20.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.384+0000 7ff8a909a640 1 -- 172.21.15.1:0/3966059041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff8a4150d30 con 0x7ff8a40c8590 2024-03-06T07:28:20.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.385+0000 7ff88b7fe640 1 -- 172.21.15.1:0/3966059041 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v185) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff898058d20 con 0x7ff8a40c8590 2024-03-06T07:28:20.387 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:28:20.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.387+0000 7ff8a909a640 1 -- 172.21.15.1:0/3966059041 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff89006b4d0 msgr2=0x7ff89006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.387+0000 7ff8a909a640 1 --2- 172.21.15.1:0/3966059041 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff89006b4d0 0x7ff89006d990 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7ff894036000 tx=0x7ff89403c040 comp rx=0 tx=0).stop 2024-03-06T07:28:20.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.387+0000 7ff8a909a640 1 -- 172.21.15.1:0/3966059041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a40c8590 msgr2=0x7ff8a4151780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.387+0000 7ff8a909a640 1 --2- 172.21.15.1:0/3966059041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a40c8590 0x7ff8a4151780 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7ff89800d800 tx=0x7ff89800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.387+0000 7ff8a909a640 1 -- 172.21.15.1:0/3966059041 shutdown_connections 2024-03-06T07:28:20.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.387+0000 7ff8a909a640 1 --2- 172.21.15.1:0/3966059041 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff89006b4d0 0x7ff89006d990 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.387+0000 7ff8a909a640 1 --2- 172.21.15.1:0/3966059041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a4156a10 0x7ff8a4151cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.387+0000 7ff8a909a640 1 --2- 172.21.15.1:0/3966059041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a40c8590 0x7ff8a4151780 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.387+0000 7ff8a909a640 1 --2- 172.21.15.1:0/3966059041 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8a40c3210 0x7ff8a41ebf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.387+0000 7ff8a909a640 1 -- 172.21.15.1:0/3966059041 >> 172.21.15.1:0/3966059041 conn(0x7ff8a40aa7a0 msgr2=0x7ff8a40c3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.387+0000 7ff8a909a640 1 -- 172.21.15.1:0/3966059041 shutdown_connections 2024-03-06T07:28:20.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.387+0000 7ff8a909a640 1 -- 172.21.15.1:0/3966059041 wait complete. 2024-03-06T07:28:20.400 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:20.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.504+0000 7fe680a88640 1 Processor -- start 2024-03-06T07:28:20.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.504+0000 7fe680a88640 1 -- start start 2024-03-06T07:28:20.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.505+0000 7fe680a88640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe67c0c6b10 0x7fe67c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.505+0000 7fe680a88640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe67c0bf9d0 0x7fe67c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.505+0000 7fe680a88640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67c0c2fc0 con 0x7fe67c152650 2024-03-06T07:28:20.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.505+0000 7fe680a88640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe67c155800 con 0x7fe67c0bf9d0 2024-03-06T07:28:20.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.505+0000 7fe680a88640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe67c158da0 con 0x7fe67c0c6b10 2024-03-06T07:28:20.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.505+0000 7fe67ad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe67c0bf9d0 0x7fe67c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.505+0000 7fe67a59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe67c152650 0x7fe67c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39766/0 (socket says 172.21.15.1:39766) 2024-03-06T07:28:20.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.505+0000 7fe67a59c640 1 -- 172.21.15.1:0/1814620866 learned_addr learned my addr 172.21.15.1:0/1814620866 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:20.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.506+0000 7fe679d9b640 1 --2- 172.21.15.1:0/1814620866 >> v2:172.21.15.98:3300/0 conn(0x7fe67c0c6b10 0x7fe67c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.506+0000 7fe67959a640 1 -- 172.21.15.1:0/1814620866 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975666138 0 0) 0x7fe67c0c2fc0 con 0x7fe67c152650 2024-03-06T07:28:20.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.506+0000 7fe67959a640 1 -- 172.21.15.1:0/1814620866 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe664003520 con 0x7fe67c152650 2024-03-06T07:28:20.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.506+0000 7fe67ad9d640 1 -- 172.21.15.1:0/1814620866 >> v2:172.21.15.98:3300/0 conn(0x7fe67c0c6b10 msgr2=0x7fe67c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.506+0000 7fe67ad9d640 1 --2- 172.21.15.1:0/1814620866 >> v2:172.21.15.98:3300/0 conn(0x7fe67c0c6b10 0x7fe67c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.506+0000 7fe67ad9d640 1 -- 172.21.15.1:0/1814620866 >> v1:172.21.15.1:6789/0 conn(0x7fe67c152650 legacy=0x7fe67c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.506+0000 7fe67ad9d640 1 -- 172.21.15.1:0/1814620866 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c0c6690 con 0x7fe67c0bf9d0 2024-03-06T07:28:20.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.507+0000 7fe67ad9d640 1 --2- 172.21.15.1:0/1814620866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe67c0bf9d0 0x7fe67c0c1460 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fe67c0cb8e0 tx=0x7fe67402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f31131858f536c8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.507+0000 7fe67959a640 1 -- 172.21.15.1:0/1814620866 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe674039070 con 0x7fe67c0bf9d0 2024-03-06T07:28:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.507+0000 7fe67959a640 1 -- 172.21.15.1:0/1814620866 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67402cbf0 con 0x7fe67c0bf9d0 2024-03-06T07:28:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.507+0000 7fe67959a640 1 -- 172.21.15.1:0/1814620866 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe674034440 con 0x7fe67c0bf9d0 2024-03-06T07:28:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.507+0000 7fe680a88640 1 -- 172.21.15.1:0/1814620866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe67c0bf9d0 msgr2=0x7fe67c0c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.507+0000 7fe680a88640 1 --2- 172.21.15.1:0/1814620866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe67c0bf9d0 0x7fe67c0c1460 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fe67c0cb8e0 tx=0x7fe67402be40 comp rx=0 tx=0).stop 2024-03-06T07:28:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.507+0000 7fe680a88640 1 -- 172.21.15.1:0/1814620866 shutdown_connections 2024-03-06T07:28:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.507+0000 7fe680a88640 1 --2- 172.21.15.1:0/1814620866 >> v2:172.21.15.98:3300/0 conn(0x7fe67c0c6b10 0x7fe67c155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.507+0000 7fe680a88640 1 --2- 172.21.15.1:0/1814620866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe67c0bf9d0 0x7fe67c0c1460 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.507+0000 7fe680a88640 1 -- 172.21.15.1:0/1814620866 >> 172.21.15.1:0/1814620866 conn(0x7fe67c066ee0 msgr2=0x7fe67c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.507+0000 7fe680a88640 1 -- 172.21.15.1:0/1814620866 shutdown_connections 2024-03-06T07:28:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.507+0000 7fe680a88640 1 -- 172.21.15.1:0/1814620866 wait complete. 2024-03-06T07:28:20.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.509+0000 7fe680a88640 1 Processor -- start 2024-03-06T07:28:20.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.509+0000 7fe680a88640 1 -- start start 2024-03-06T07:28:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.510+0000 7fe680a88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe67c0bf9d0 0x7fe67c157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.510+0000 7fe680a88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe67c152650 0x7fe67c1536b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.510+0000 7fe680a88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe67c154c70 0x7fe67c153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.510+0000 7fe680a88640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c1551b0 con 0x7fe67c152650 2024-03-06T07:28:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.510+0000 7fe680a88640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c0c6fa0 con 0x7fe67c154c70 2024-03-06T07:28:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.510+0000 7fe680a88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c155490 con 0x7fe67c0bf9d0 2024-03-06T07:28:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.511+0000 7fe679d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe67c152650 0x7fe67c1536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.511+0000 7fe679d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe67c152650 0x7fe67c1536b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51108/0 (socket says 172.21.15.1:51108) 2024-03-06T07:28:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.511+0000 7fe679d9b640 1 -- 172.21.15.1:0/1823093895 learned_addr learned my addr 172.21.15.1:0/1823093895 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.511+0000 7fe67ad9d640 1 --2- 172.21.15.1:0/1823093895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe67c154c70 0x7fe67c153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.511+0000 7fe67a59c640 1 --2- 172.21.15.1:0/1823093895 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe67c0bf9d0 0x7fe67c157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.511+0000 7fe679d9b640 1 -- 172.21.15.1:0/1823093895 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe67c0bf9d0 msgr2=0x7fe67c157750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.511+0000 7fe679d9b640 1 --2- 172.21.15.1:0/1823093895 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe67c0bf9d0 0x7fe67c157750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.511+0000 7fe679d9b640 1 -- 172.21.15.1:0/1823093895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe67c154c70 msgr2=0x7fe67c153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.511+0000 7fe679d9b640 1 --2- 172.21.15.1:0/1823093895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe67c154c70 0x7fe67c153bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.511+0000 7fe679d9b640 1 -- 172.21.15.1:0/1823093895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c1eaaa0 con 0x7fe67c152650 2024-03-06T07:28:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.511+0000 7fe679d9b640 1 --2- 172.21.15.1:0/1823093895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe67c152650 0x7fe67c1536b0 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fe67000e9b0 tx=0x7fe67000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.511+0000 7fe66affd640 1 -- 172.21.15.1:0/1823093895 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670017070 con 0x7fe67c152650 2024-03-06T07:28:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.511+0000 7fe66affd640 1 -- 172.21.15.1:0/1823093895 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67000c9a0 con 0x7fe67c152650 2024-03-06T07:28:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.511+0000 7fe680a88640 1 -- 172.21.15.1:0/1823093895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67c152bd0 con 0x7fe67c152650 2024-03-06T07:28:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.511+0000 7fe66affd640 1 -- 172.21.15.1:0/1823093895 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67000cd90 con 0x7fe67c152650 2024-03-06T07:28:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.511+0000 7fe680a88640 1 -- 172.21.15.1:0/1823093895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67c157c90 con 0x7fe67c152650 2024-03-06T07:28:20.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.512+0000 7fe680a88640 1 -- 172.21.15.1:0/1823093895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe67c067fd0 con 0x7fe67c152650 2024-03-06T07:28:20.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.513+0000 7fe66affd640 1 -- 172.21.15.1:0/1823093895 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe67001a070 con 0x7fe67c152650 2024-03-06T07:28:20.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.514+0000 7fe66affd640 1 --2- 172.21.15.1:0/1823093895 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe65c06b4d0 0x7fe65c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.514+0000 7fe66affd640 1 -- 172.21.15.1:0/1823093895 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7fe67008a340 con 0x7fe67c152650 2024-03-06T07:28:20.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.514+0000 7fe67a59c640 1 --2- 172.21.15.1:0/1823093895 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe65c06b4d0 0x7fe65c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.515+0000 7fe67a59c640 1 --2- 172.21.15.1:0/1823093895 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe65c06b4d0 0x7fe65c06d990 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fe66c005880 tx=0x7fe66c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.518+0000 7fe66affd640 1 -- 172.21.15.1:0/1823093895 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe670058970 con 0x7fe67c152650 2024-03-06T07:28:20.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.696+0000 7fe680a88640 1 -- 172.21.15.1:0/1823093895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe67c0b06f0 con 0x7fe67c152650 2024-03-06T07:28:20.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.697+0000 7fe66affd640 1 -- 172.21.15.1:0/1823093895 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v185) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe67005ca10 con 0x7fe67c152650 2024-03-06T07:28:20.699 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:28:20.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.699+0000 7fe680a88640 1 -- 172.21.15.1:0/1823093895 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe65c06b4d0 msgr2=0x7fe65c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.699+0000 7fe680a88640 1 --2- 172.21.15.1:0/1823093895 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe65c06b4d0 0x7fe65c06d990 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fe66c005880 tx=0x7fe66c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:20.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.699+0000 7fe680a88640 1 -- 172.21.15.1:0/1823093895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe67c152650 msgr2=0x7fe67c1536b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.699+0000 7fe680a88640 1 --2- 172.21.15.1:0/1823093895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe67c152650 0x7fe67c1536b0 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fe67000e9b0 tx=0x7fe67000ee70 comp rx=0 tx=0).stop 2024-03-06T07:28:20.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.699+0000 7fe680a88640 1 -- 172.21.15.1:0/1823093895 shutdown_connections 2024-03-06T07:28:20.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.699+0000 7fe680a88640 1 --2- 172.21.15.1:0/1823093895 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe65c06b4d0 0x7fe65c06d990 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.699+0000 7fe680a88640 1 --2- 172.21.15.1:0/1823093895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe67c154c70 0x7fe67c153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.699+0000 7fe680a88640 1 --2- 172.21.15.1:0/1823093895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe67c152650 0x7fe67c1536b0 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.699+0000 7fe680a88640 1 --2- 172.21.15.1:0/1823093895 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe67c0bf9d0 0x7fe67c157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.699+0000 7fe680a88640 1 -- 172.21.15.1:0/1823093895 >> 172.21.15.1:0/1823093895 conn(0x7fe67c066ee0 msgr2=0x7fe67c0c0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.699+0000 7fe680a88640 1 -- 172.21.15.1:0/1823093895 shutdown_connections 2024-03-06T07:28:20.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.699+0000 7fe680a88640 1 -- 172.21.15.1:0/1823093895 wait complete. 2024-03-06T07:28:20.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:20.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.812+0000 7f54f7883640 1 Processor -- start 2024-03-06T07:28:20.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.812+0000 7f54f7883640 1 -- start start 2024-03-06T07:28:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.813+0000 7f54f7883640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54f0152650 0x7f54f0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.813+0000 7f54f7883640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f54f00bf9d0 0x7f54f00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.813+0000 7f54f7883640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f00c2fc0 con 0x7f54f00c6b10 2024-03-06T07:28:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.813+0000 7f54f7883640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54f0155800 con 0x7f54f0152650 2024-03-06T07:28:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.813+0000 7f54f7883640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54f0158da0 con 0x7f54f00bf9d0 2024-03-06T07:28:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.813+0000 7f54f561f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54f0152650 0x7f54f0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.813+0000 7f54f561f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54f0152650 0x7f54f0155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51124/0 (socket says 172.21.15.1:51124) 2024-03-06T07:28:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.813+0000 7f54f561f640 1 -- 172.21.15.1:0/2983718255 learned_addr learned my addr 172.21.15.1:0/2983718255 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.813+0000 7f54f5e20640 1 --2- 172.21.15.1:0/2983718255 >> v2:172.21.15.98:3300/0 conn(0x7f54f00bf9d0 0x7f54f00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.813+0000 7f54f561f640 1 -- 172.21.15.1:0/2983718255 >> v2:172.21.15.98:3300/0 conn(0x7f54f00bf9d0 msgr2=0x7f54f00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.813+0000 7f54f561f640 1 --2- 172.21.15.1:0/2983718255 >> v2:172.21.15.98:3300/0 conn(0x7f54f00bf9d0 0x7f54f00c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.813+0000 7f54f561f640 1 -- 172.21.15.1:0/2983718255 >> v1:172.21.15.98:6789/0 conn(0x7f54f00c6b10 legacy=0x7f54f0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.813+0000 7f54f561f640 1 -- 172.21.15.1:0/2983718255 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f00c6690 con 0x7f54f0152650 2024-03-06T07:28:20.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.814+0000 7f54f561f640 1 --2- 172.21.15.1:0/2983718255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f0152650 0x7f54f0155c90 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f54f01597b0 tx=0x7f54ec02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a97c237f84281c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.814+0000 7f54e7fff640 1 -- 172.21.15.1:0/2983718255 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ec02e070 con 0x7f54f0152650 2024-03-06T07:28:20.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.814+0000 7f54e7fff640 1 -- 172.21.15.1:0/2983718255 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ec037070 con 0x7f54f0152650 2024-03-06T07:28:20.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.814+0000 7f54e7fff640 1 -- 172.21.15.1:0/2983718255 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ec031a00 con 0x7f54f0152650 2024-03-06T07:28:20.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.815+0000 7f54f7883640 1 -- 172.21.15.1:0/2983718255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f0152650 msgr2=0x7f54f0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.815+0000 7f54f7883640 1 --2- 172.21.15.1:0/2983718255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f0152650 0x7f54f0155c90 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f54f01597b0 tx=0x7f54ec02b880 comp rx=0 tx=0).stop 2024-03-06T07:28:20.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.815+0000 7f54f7883640 1 -- 172.21.15.1:0/2983718255 shutdown_connections 2024-03-06T07:28:20.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.815+0000 7f54f7883640 1 --2- 172.21.15.1:0/2983718255 >> v2:172.21.15.98:3300/0 conn(0x7f54f00bf9d0 0x7f54f00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.815+0000 7f54f7883640 1 --2- 172.21.15.1:0/2983718255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f0152650 0x7f54f0155c90 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.815+0000 7f54f7883640 1 -- 172.21.15.1:0/2983718255 >> 172.21.15.1:0/2983718255 conn(0x7f54f0066ee0 msgr2=0x7f54f00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.815+0000 7f54f7883640 1 -- 172.21.15.1:0/2983718255 shutdown_connections 2024-03-06T07:28:20.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.815+0000 7f54f7883640 1 -- 172.21.15.1:0/2983718255 wait complete. 2024-03-06T07:28:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.816+0000 7f54f7883640 1 Processor -- start 2024-03-06T07:28:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.816+0000 7f54f7883640 1 -- start start 2024-03-06T07:28:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.817+0000 7f54f7883640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54f00bf9d0 0x7f54f01eb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.817+0000 7f54f7883640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f0152650 0x7f54f00c5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.817+0000 7f54f7883640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54f0156490 0x7f54f01584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.817+0000 7f54f7883640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f01551b0 con 0x7f54f0152650 2024-03-06T07:28:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.817+0000 7f54f7883640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f00c6fa0 con 0x7f54f0156490 2024-03-06T07:28:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.817+0000 7f54f7883640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54f0155490 con 0x7f54f00bf9d0 2024-03-06T07:28:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.817+0000 7f54f4e1e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f0152650 0x7f54f00c5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.817+0000 7f54f4e1e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f0152650 0x7f54f00c5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51136/0 (socket says 172.21.15.1:51136) 2024-03-06T07:28:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.817+0000 7f54f4e1e640 1 -- 172.21.15.1:0/2403549722 learned_addr learned my addr 172.21.15.1:0/2403549722 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.817+0000 7f54f561f640 1 --2- 172.21.15.1:0/2403549722 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54f00bf9d0 0x7f54f01eb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.817+0000 7f54f5e20640 1 --2- 172.21.15.1:0/2403549722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54f0156490 0x7f54f01584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.817+0000 7f54f4e1e640 1 -- 172.21.15.1:0/2403549722 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54f00bf9d0 msgr2=0x7f54f01eb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.817+0000 7f54f4e1e640 1 --2- 172.21.15.1:0/2403549722 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54f00bf9d0 0x7f54f01eb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.817+0000 7f54f4e1e640 1 -- 172.21.15.1:0/2403549722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54f0156490 msgr2=0x7f54f01584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.817+0000 7f54f4e1e640 1 --2- 172.21.15.1:0/2403549722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54f0156490 0x7f54f01584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.817+0000 7f54f4e1e640 1 -- 172.21.15.1:0/2403549722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f0158000 con 0x7f54f0152650 2024-03-06T07:28:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.818+0000 7f54f4e1e640 1 --2- 172.21.15.1:0/2403549722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f0152650 0x7f54f00c5b30 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f54e000c410 tx=0x7f54e000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.818+0000 7f54e5ffb640 1 -- 172.21.15.1:0/2403549722 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e0017070 con 0x7f54f0152650 2024-03-06T07:28:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.818+0000 7f54e5ffb640 1 -- 172.21.15.1:0/2403549722 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e00035f0 con 0x7f54f0152650 2024-03-06T07:28:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.818+0000 7f54f7883640 1 -- 172.21.15.1:0/2403549722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f01e14d0 con 0x7f54f0152650 2024-03-06T07:28:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.818+0000 7f54e5ffb640 1 -- 172.21.15.1:0/2403549722 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e0003a20 con 0x7f54f0152650 2024-03-06T07:28:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.818+0000 7f54f7883640 1 -- 172.21.15.1:0/2403549722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f0151ad0 con 0x7f54f0152650 2024-03-06T07:28:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.819+0000 7f54f7883640 1 -- 172.21.15.1:0/2403549722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f01518f0 con 0x7f54f0152650 2024-03-06T07:28:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.819+0000 7f54e5ffb640 1 -- 172.21.15.1:0/2403549722 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54e001a070 con 0x7f54f0152650 2024-03-06T07:28:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.820+0000 7f54e5ffb640 1 --2- 172.21.15.1:0/2403549722 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f54d406b4d0 0x7f54d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.820+0000 7f54e5ffb640 1 -- 172.21.15.1:0/2403549722 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f54e0089760 con 0x7f54f0152650 2024-03-06T07:28:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.820+0000 7f54f561f640 1 --2- 172.21.15.1:0/2403549722 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f54d406b4d0 0x7f54d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.821+0000 7f54f561f640 1 --2- 172.21.15.1:0/2403549722 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f54d406b4d0 0x7f54d406d990 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f54ec036000 tx=0x7f54ec03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:20.824+0000 7f54e5ffb640 1 -- 172.21.15.1:0/2403549722 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54f01518f0 con 0x7f54f0152650 2024-03-06T07:28:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.172+0000 7f54f7883640 1 -- 172.21.15.1:0/2403549722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54f00c48b0 con 0x7f54f0152650 2024-03-06T07:28:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.172+0000 7f54e5ffb640 1 -- 172.21.15.1:0/2403549722 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+5005 (secure 0 0 0) 0x7f54f00c48b0 con 0x7f54f0152650 2024-03-06T07:28:21.174 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:21.175 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":179,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:28:16.947047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10161,"mds_1":9951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9951":{"gid":9951,"name":"a","rank":1,"incarnation":178,"state":"up:active","state_seq":9,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10161":{"gid":10161,"name":"c","rank":0,"incarnation":176,"state":"up:active","state_seq":2,"addr":"172.21.15.1:6835/1525330395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1525330395},{"type":"v1","addr":"172.21.15.1:6835","nonce":1525330395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10161,"qdb_cluster":[9951,10161]},"id":14}]} 2024-03-06T07:28:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 179 2024-03-06T07:28:21.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.174+0000 7f54f7883640 1 -- 172.21.15.1:0/2403549722 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f54d406b4d0 msgr2=0x7f54d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.174+0000 7f54f7883640 1 --2- 172.21.15.1:0/2403549722 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f54d406b4d0 0x7f54d406d990 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f54ec036000 tx=0x7f54ec03d040 comp rx=0 tx=0).stop 2024-03-06T07:28:21.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.174+0000 7f54f7883640 1 -- 172.21.15.1:0/2403549722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f0152650 msgr2=0x7f54f00c5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.174+0000 7f54f7883640 1 --2- 172.21.15.1:0/2403549722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f0152650 0x7f54f00c5b30 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f54e000c410 tx=0x7f54e000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.174+0000 7f54f7883640 1 -- 172.21.15.1:0/2403549722 shutdown_connections 2024-03-06T07:28:21.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.174+0000 7f54f7883640 1 --2- 172.21.15.1:0/2403549722 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f54d406b4d0 0x7f54d406d990 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.174+0000 7f54f7883640 1 --2- 172.21.15.1:0/2403549722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54f0156490 0x7f54f01584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.174+0000 7f54f7883640 1 --2- 172.21.15.1:0/2403549722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f0152650 0x7f54f00c5b30 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.174+0000 7f54f7883640 1 --2- 172.21.15.1:0/2403549722 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f54f00bf9d0 0x7f54f01eb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.174+0000 7f54f7883640 1 -- 172.21.15.1:0/2403549722 >> 172.21.15.1:0/2403549722 conn(0x7f54f0066ee0 msgr2=0x7f54f00c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.174+0000 7f54f7883640 1 -- 172.21.15.1:0/2403549722 shutdown_connections 2024-03-06T07:28:21.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.174+0000 7f54f7883640 1 -- 172.21.15.1:0/2403549722 wait complete. 2024-03-06T07:28:21.191 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:21.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.297+0000 7f27b604e640 1 Processor -- start 2024-03-06T07:28:21.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.297+0000 7f27b604e640 1 -- start start 2024-03-06T07:28:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.298+0000 7f27b604e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27b0152820 0x7f27b00c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.298+0000 7f27b604e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b00c7180 con 0x7f27b00c0670 2024-03-06T07:28:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.298+0000 7f27b604e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b0155460 con 0x7f27b00af6f0 2024-03-06T07:28:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.298+0000 7f27b604e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27b00c6bc0 con 0x7f27b0152820 2024-03-06T07:28:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.298+0000 7f27af7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27b0152820 0x7f27b00c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.298+0000 7f27af7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27b0152820 0x7f27b00c0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51146/0 (socket says 172.21.15.1:51146) 2024-03-06T07:28:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.298+0000 7f27af7fe640 1 -- 172.21.15.1:0/2251080655 learned_addr learned my addr 172.21.15.1:0/2251080655 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.298+0000 7f27aeffd640 1 --1- 172.21.15.1:0/2251080655 >> v1:172.21.15.1:6789/0 conn(0x7f27b00c0670 0x7f27b00c73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39778/0 (socket says 172.21.15.1:39778) 2024-03-06T07:28:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.299+0000 7f27af7fe640 1 -- 172.21.15.1:0/2251080655 >> v1:172.21.15.98:6789/0 conn(0x7f27b00af6f0 legacy=0x7f27b015fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.299+0000 7f27af7fe640 1 -- 172.21.15.1:0/2251080655 >> v1:172.21.15.1:6789/0 conn(0x7f27b00c0670 legacy=0x7f27b00c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.299+0000 7f27af7fe640 1 -- 172.21.15.1:0/2251080655 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b00c6ea0 con 0x7f27b0152820 2024-03-06T07:28:21.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.299+0000 7f27af7fe640 1 --2- 172.21.15.1:0/2251080655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b0152820 0x7f27b00c0130 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f27a0009870 tx=0x7f27a002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5f6e31ccf98f9c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.300+0000 7f27ae7fc640 1 -- 172.21.15.1:0/2251080655 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a0034070 con 0x7f27b0152820 2024-03-06T07:28:21.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.300+0000 7f27ae7fc640 1 -- 172.21.15.1:0/2251080655 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a0030860 con 0x7f27b0152820 2024-03-06T07:28:21.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.300+0000 7f27ae7fc640 1 -- 172.21.15.1:0/2251080655 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a0030c30 con 0x7f27b0152820 2024-03-06T07:28:21.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.300+0000 7f27b604e640 1 -- 172.21.15.1:0/2251080655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b0152820 msgr2=0x7f27b00c0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.300+0000 7f27b604e640 1 --2- 172.21.15.1:0/2251080655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b0152820 0x7f27b00c0130 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f27a0009870 tx=0x7f27a002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.300+0000 7f27b604e640 1 -- 172.21.15.1:0/2251080655 shutdown_connections 2024-03-06T07:28:21.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.300+0000 7f27b604e640 1 --2- 172.21.15.1:0/2251080655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b0152820 0x7f27b00c0130 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.300+0000 7f27b604e640 1 -- 172.21.15.1:0/2251080655 >> 172.21.15.1:0/2251080655 conn(0x7f27b0072720 msgr2=0x7f27b0072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.300+0000 7f27b604e640 1 -- 172.21.15.1:0/2251080655 shutdown_connections 2024-03-06T07:28:21.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.301+0000 7f27b604e640 1 -- 172.21.15.1:0/2251080655 wait complete. 2024-03-06T07:28:21.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.302+0000 7f27b604e640 1 Processor -- start 2024-03-06T07:28:21.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.302+0000 7f27b604e640 1 -- start start 2024-03-06T07:28:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.303+0000 7f27b604e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b00af6f0 0x7f27b014ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.303+0000 7f27b604e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27b00c0670 0x7f27b00c3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.303+0000 7f27b604e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27b0152820 0x7f27b0156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.303+0000 7f27b604e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b00c7c80 con 0x7f27b00af6f0 2024-03-06T07:28:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.303+0000 7f27b604e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b00c7b00 con 0x7f27b0152820 2024-03-06T07:28:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.303+0000 7f27b604e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27b0152c00 con 0x7f27b00c0670 2024-03-06T07:28:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.303+0000 7f27af7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b00af6f0 0x7f27b014ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.304+0000 7f27af7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b00af6f0 0x7f27b014ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51162/0 (socket says 172.21.15.1:51162) 2024-03-06T07:28:21.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.304+0000 7f27af7fe640 1 -- 172.21.15.1:0/639059086 learned_addr learned my addr 172.21.15.1:0/639059086 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:21.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.304+0000 7f27aeffd640 1 --2- 172.21.15.1:0/639059086 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27b00c0670 0x7f27b00c3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.304+0000 7f27affff640 1 --2- 172.21.15.1:0/639059086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27b0152820 0x7f27b0156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.304+0000 7f27af7fe640 1 -- 172.21.15.1:0/639059086 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27b00c0670 msgr2=0x7f27b00c3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.304+0000 7f27af7fe640 1 --2- 172.21.15.1:0/639059086 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27b00c0670 0x7f27b00c3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.304+0000 7f27af7fe640 1 -- 172.21.15.1:0/639059086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27b0152820 msgr2=0x7f27b0156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.304+0000 7f27af7fe640 1 --2- 172.21.15.1:0/639059086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27b0152820 0x7f27b0156300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.304+0000 7f27af7fe640 1 -- 172.21.15.1:0/639059086 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b01df150 con 0x7f27b00af6f0 2024-03-06T07:28:21.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.304+0000 7f27affff640 1 --2- 172.21.15.1:0/639059086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27b0152820 0x7f27b0156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:21.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.304+0000 7f27af7fe640 1 --2- 172.21.15.1:0/639059086 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b00af6f0 0x7f27b014ca60 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f27a0000c00 tx=0x7f27a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.304+0000 7f2797fff640 1 -- 172.21.15.1:0/639059086 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a0034040 con 0x7f27b00af6f0 2024-03-06T07:28:21.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.304+0000 7f2797fff640 1 -- 172.21.15.1:0/639059086 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a00433f0 con 0x7f27b00af6f0 2024-03-06T07:28:21.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.304+0000 7f2797fff640 1 -- 172.21.15.1:0/639059086 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a00437c0 con 0x7f27b00af6f0 2024-03-06T07:28:21.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.304+0000 7f27b604e640 1 -- 172.21.15.1:0/639059086 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b00c36e0 con 0x7f27b00af6f0 2024-03-06T07:28:21.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.305+0000 7f27b604e640 1 -- 172.21.15.1:0/639059086 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b00c3ca0 con 0x7f27b00af6f0 2024-03-06T07:28:21.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.306+0000 7f2797fff640 1 -- 172.21.15.1:0/639059086 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f27a004a060 con 0x7f27b00af6f0 2024-03-06T07:28:21.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.306+0000 7f27b604e640 1 -- 172.21.15.1:0/639059086 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b00686e0 con 0x7f27b00af6f0 2024-03-06T07:28:21.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.306+0000 7f2797fff640 1 --2- 172.21.15.1:0/639059086 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f279c06b500 0x7f279c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.306+0000 7f2797fff640 1 -- 172.21.15.1:0/639059086 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f27a00ac530 con 0x7f27b00af6f0 2024-03-06T07:28:21.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.307+0000 7f27aeffd640 1 --2- 172.21.15.1:0/639059086 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f279c06b500 0x7f279c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.307+0000 7f27aeffd640 1 --2- 172.21.15.1:0/639059086 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f279c06b500 0x7f279c06d9c0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f27b00af3f0 tx=0x7f27a4007e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.311+0000 7f2797fff640 1 -- 172.21.15.1:0/639059086 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f27a003e070 con 0x7f27b00af6f0 2024-03-06T07:28:21.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.491+0000 7f27b604e640 1 -- 172.21.15.1:0/639059086 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27b00c5b40 con 0x7f27b00af6f0 2024-03-06T07:28:21.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.493+0000 7f2797fff640 1 -- 172.21.15.1:0/639059086 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) v1 ==== 74+0+16947 (secure 0 0 0) 0x7f27a007b240 con 0x7f27b00af6f0 2024-03-06T07:28:21.495 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:21.495 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":185,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:28:10.884250+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:50.456844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:51.147612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"29.0","mappings":[{"from":6,"to":0}]},{"pgid":"29.c","mappings":[{"from":6,"to":0}]},{"pgid":"29.f","mappings":[{"from":6,"to":0}]},{"pgid":"29.16","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.495+0000 7f27b604e640 1 -- 172.21.15.1:0/639059086 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f279c06b500 msgr2=0x7f279c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.495+0000 7f27b604e640 1 --2- 172.21.15.1:0/639059086 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f279c06b500 0x7f279c06d9c0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f27b00af3f0 tx=0x7f27a4007e80 comp rx=0 tx=0).stop 2024-03-06T07:28:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.495+0000 7f27b604e640 1 -- 172.21.15.1:0/639059086 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b00af6f0 msgr2=0x7f27b014ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.495+0000 7f27b604e640 1 --2- 172.21.15.1:0/639059086 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b00af6f0 0x7f27b014ca60 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f27a0000c00 tx=0x7f27a00046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.495+0000 7f27b604e640 1 -- 172.21.15.1:0/639059086 shutdown_connections 2024-03-06T07:28:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.495+0000 7f27b604e640 1 --2- 172.21.15.1:0/639059086 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f279c06b500 0x7f279c06d9c0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.495+0000 7f27b604e640 1 --2- 172.21.15.1:0/639059086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27b0152820 0x7f27b0156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.495+0000 7f27b604e640 1 --2- 172.21.15.1:0/639059086 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27b00c0670 0x7f27b00c3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.495+0000 7f27b604e640 1 --2- 172.21.15.1:0/639059086 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b00af6f0 0x7f27b014ca60 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.495+0000 7f27b604e640 1 -- 172.21.15.1:0/639059086 >> 172.21.15.1:0/639059086 conn(0x7f27b0072720 msgr2=0x7f27b00745e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.495+0000 7f27b604e640 1 -- 172.21.15.1:0/639059086 shutdown_connections 2024-03-06T07:28:21.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.495+0000 7f27b604e640 1 -- 172.21.15.1:0/639059086 wait complete. 2024-03-06T07:28:21.511 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:28:21.512 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:21.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.615+0000 7fec201d7640 1 Processor -- start 2024-03-06T07:28:21.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.615+0000 7fec201d7640 1 -- start start 2024-03-06T07:28:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.616+0000 7fec201d7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fec180c6b10 0x7fec18155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.616+0000 7fec201d7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec180bf9d0 0x7fec180c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.616+0000 7fec201d7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec180c2fc0 con 0x7fec18152650 2024-03-06T07:28:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.616+0000 7fec201d7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec18155800 con 0x7fec180c6b10 2024-03-06T07:28:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.616+0000 7fec201d7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec18158da0 con 0x7fec180bf9d0 2024-03-06T07:28:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.617+0000 7fec1d772640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fec180c6b10 0x7fec18155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.617+0000 7fec1e774640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec180bf9d0 0x7fec180c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.617+0000 7fec1d772640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fec180c6b10 0x7fec18155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:43636/0 (socket says 172.21.15.1:43636) 2024-03-06T07:28:21.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.617+0000 7fec1d772640 1 -- 172.21.15.1:0/1562491507 learned_addr learned my addr 172.21.15.1:0/1562491507 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:21.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.617+0000 7fec1d772640 1 -- 172.21.15.1:0/1562491507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec180bf9d0 msgr2=0x7fec180c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.617+0000 7fec1d772640 1 --2- 172.21.15.1:0/1562491507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec180bf9d0 0x7fec180c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.617+0000 7fec1d772640 1 -- 172.21.15.1:0/1562491507 >> v1:172.21.15.98:6789/0 conn(0x7fec18152650 legacy=0x7fec18158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.617+0000 7fec1d772640 1 -- 172.21.15.1:0/1562491507 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec180c6690 con 0x7fec180c6b10 2024-03-06T07:28:21.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.617+0000 7fec1e774640 1 --2- 172.21.15.1:0/1562491507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec180bf9d0 0x7fec180c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:21.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.618+0000 7fec1d772640 1 --2- 172.21.15.1:0/1562491507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec180c6b10 0x7fec18155c90 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fec08009870 tx=0x7fec0802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4e9e84cfd7f5405 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.618+0000 7fec1cf71640 1 -- 172.21.15.1:0/1562491507 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec08035020 con 0x7fec180c6b10 2024-03-06T07:28:21.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.618+0000 7fec1cf71640 1 -- 172.21.15.1:0/1562491507 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec08031640 con 0x7fec180c6b10 2024-03-06T07:28:21.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.618+0000 7fec1cf71640 1 -- 172.21.15.1:0/1562491507 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0802d040 con 0x7fec180c6b10 2024-03-06T07:28:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.619+0000 7fec201d7640 1 -- 172.21.15.1:0/1562491507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec180c6b10 msgr2=0x7fec18155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.619+0000 7fec201d7640 1 --2- 172.21.15.1:0/1562491507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec180c6b10 0x7fec18155c90 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fec08009870 tx=0x7fec0802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.619+0000 7fec201d7640 1 -- 172.21.15.1:0/1562491507 shutdown_connections 2024-03-06T07:28:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.619+0000 7fec201d7640 1 --2- 172.21.15.1:0/1562491507 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec180bf9d0 0x7fec180c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.619+0000 7fec201d7640 1 --2- 172.21.15.1:0/1562491507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec180c6b10 0x7fec18155c90 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.619+0000 7fec201d7640 1 -- 172.21.15.1:0/1562491507 >> 172.21.15.1:0/1562491507 conn(0x7fec18066ee0 msgr2=0x7fec180672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.619+0000 7fec201d7640 1 -- 172.21.15.1:0/1562491507 shutdown_connections 2024-03-06T07:28:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.619+0000 7fec201d7640 1 -- 172.21.15.1:0/1562491507 wait complete. 2024-03-06T07:28:21.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.621+0000 7fec201d7640 1 Processor -- start 2024-03-06T07:28:21.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.621+0000 7fec201d7640 1 -- start start 2024-03-06T07:28:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.622+0000 7fec201d7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec180bf9d0 0x7fec1814b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.622+0000 7fec201d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec18152650 0x7fec180c5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.622+0000 7fec201d7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec1814fbf0 0x7fec181edfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.622+0000 7fec201d7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec181551b0 con 0x7fec1814fbf0 2024-03-06T07:28:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.622+0000 7fec201d7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec180c6fa0 con 0x7fec18152650 2024-03-06T07:28:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.622+0000 7fec201d7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec18155490 con 0x7fec180bf9d0 2024-03-06T07:28:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.622+0000 7fec1e774640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec1814fbf0 0x7fec181edfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.622+0000 7fec1e774640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec1814fbf0 0x7fec181edfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51174/0 (socket says 172.21.15.1:51174) 2024-03-06T07:28:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.622+0000 7fec1e774640 1 -- 172.21.15.1:0/3873904722 learned_addr learned my addr 172.21.15.1:0/3873904722 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.622+0000 7fec1df73640 1 --2- 172.21.15.1:0/3873904722 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec180bf9d0 0x7fec1814b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.622+0000 7fec1d772640 1 --2- 172.21.15.1:0/3873904722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec18152650 0x7fec180c5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.622+0000 7fec1e774640 1 -- 172.21.15.1:0/3873904722 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec180bf9d0 msgr2=0x7fec1814b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.622+0000 7fec1e774640 1 --2- 172.21.15.1:0/3873904722 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec180bf9d0 0x7fec1814b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.622+0000 7fec1e774640 1 -- 172.21.15.1:0/3873904722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec18152650 msgr2=0x7fec180c5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.622+0000 7fec1e774640 1 --2- 172.21.15.1:0/3873904722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec18152650 0x7fec180c5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.622+0000 7fec1e774640 1 -- 172.21.15.1:0/3873904722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec18158760 con 0x7fec1814fbf0 2024-03-06T07:28:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.622+0000 7fec1e774640 1 --2- 172.21.15.1:0/3873904722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec1814fbf0 0x7fec181edfb0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fec10009e60 tx=0x7fec1000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.623+0000 7fec0e7fc640 1 -- 172.21.15.1:0/3873904722 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec10018030 con 0x7fec1814fbf0 2024-03-06T07:28:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.623+0000 7fec0e7fc640 1 -- 172.21.15.1:0/3873904722 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1000cdf0 con 0x7fec1814fbf0 2024-03-06T07:28:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.623+0000 7fec0e7fc640 1 -- 172.21.15.1:0/3873904722 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec10003bb0 con 0x7fec1814fbf0 2024-03-06T07:28:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.623+0000 7fec201d7640 1 -- 172.21.15.1:0/3873904722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec18158a40 con 0x7fec1814fbf0 2024-03-06T07:28:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.623+0000 7fec201d7640 1 -- 172.21.15.1:0/3873904722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec18151340 con 0x7fec1814fbf0 2024-03-06T07:28:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.624+0000 7fec201d7640 1 -- 172.21.15.1:0/3873904722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec18067fd0 con 0x7fec1814fbf0 2024-03-06T07:28:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.624+0000 7fec0e7fc640 1 -- 172.21.15.1:0/3873904722 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec10005d10 con 0x7fec1814fbf0 2024-03-06T07:28:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.625+0000 7fec0e7fc640 1 --2- 172.21.15.1:0/3873904722 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febfc06b4d0 0x7febfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.625+0000 7fec0e7fc640 1 -- 172.21.15.1:0/3873904722 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7fec1008b240 con 0x7fec1814fbf0 2024-03-06T07:28:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.625+0000 7fec1df73640 1 --2- 172.21.15.1:0/3873904722 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febfc06b4d0 0x7febfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.626+0000 7fec1df73640 1 --2- 172.21.15.1:0/3873904722 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febfc06b4d0 0x7febfc06d990 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fec180b0eb0 tx=0x7fec14009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.628+0000 7fec0e7fc640 1 -- 172.21.15.1:0/3873904722 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec10022070 con 0x7fec1814fbf0 2024-03-06T07:28:21.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.848+0000 7fec201d7640 1 -- 172.21.15.1:0/3873904722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec180c6190 con 0x7fec1814fbf0 2024-03-06T07:28:21.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.848+0000 7fec0e7fc640 1 -- 172.21.15.1:0/3873904722 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+5005 (secure 0 0 0) 0x7fec10010040 con 0x7fec1814fbf0 2024-03-06T07:28:21.850 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:21.850 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":179,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:28:16.947047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10161,"mds_1":9951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9951":{"gid":9951,"name":"a","rank":1,"incarnation":178,"state":"up:active","state_seq":9,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10161":{"gid":10161,"name":"c","rank":0,"incarnation":176,"state":"up:active","state_seq":2,"addr":"172.21.15.1:6835/1525330395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1525330395},{"type":"v1","addr":"172.21.15.1:6835","nonce":1525330395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10161,"qdb_cluster":[9951,10161]},"id":14}]} 2024-03-06T07:28:21.851 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 179 2024-03-06T07:28:21.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.850+0000 7fec201d7640 1 -- 172.21.15.1:0/3873904722 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febfc06b4d0 msgr2=0x7febfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.850+0000 7fec201d7640 1 --2- 172.21.15.1:0/3873904722 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febfc06b4d0 0x7febfc06d990 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fec180b0eb0 tx=0x7fec14009040 comp rx=0 tx=0).stop 2024-03-06T07:28:21.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.850+0000 7fec201d7640 1 -- 172.21.15.1:0/3873904722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec1814fbf0 msgr2=0x7fec181edfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.850+0000 7fec201d7640 1 --2- 172.21.15.1:0/3873904722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec1814fbf0 0x7fec181edfb0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fec10009e60 tx=0x7fec1000c490 comp rx=0 tx=0).stop 2024-03-06T07:28:21.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.850+0000 7fec201d7640 1 -- 172.21.15.1:0/3873904722 shutdown_connections 2024-03-06T07:28:21.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.850+0000 7fec201d7640 1 --2- 172.21.15.1:0/3873904722 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febfc06b4d0 0x7febfc06d990 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.850+0000 7fec201d7640 1 --2- 172.21.15.1:0/3873904722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec1814fbf0 0x7fec181edfb0 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.850+0000 7fec201d7640 1 --2- 172.21.15.1:0/3873904722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec18152650 0x7fec180c5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.850+0000 7fec201d7640 1 --2- 172.21.15.1:0/3873904722 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec180bf9d0 0x7fec1814b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.850+0000 7fec201d7640 1 -- 172.21.15.1:0/3873904722 >> 172.21.15.1:0/3873904722 conn(0x7fec18066ee0 msgr2=0x7fec180c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.850+0000 7fec201d7640 1 -- 172.21.15.1:0/3873904722 shutdown_connections 2024-03-06T07:28:21.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.850+0000 7fec201d7640 1 -- 172.21.15.1:0/3873904722 wait complete. 2024-03-06T07:28:21.865 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:21.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.965+0000 7fee74b13640 1 Processor -- start 2024-03-06T07:28:21.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.965+0000 7fee74b13640 1 -- start start 2024-03-06T07:28:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.966+0000 7fee74b13640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee700c3970 0x7fee700c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.966+0000 7fee74b13640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fee70152840 0x7fee701553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.966+0000 7fee74b13640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee700c5e40 con 0x7fee700c60b0 2024-03-06T07:28:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.966+0000 7fee74b13640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee700c67f0 con 0x7fee70152840 2024-03-06T07:28:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.966+0000 7fee74b13640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee700c6a40 con 0x7fee700c3970 2024-03-06T07:28:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.966+0000 7fee67fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fee700c60b0 0x7fee70152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39790/0 (socket says 172.21.15.1:39790) 2024-03-06T07:28:21.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.966+0000 7fee67fff640 1 -- 172.21.15.1:0/3401689120 learned_addr learned my addr 172.21.15.1:0/3401689120 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.967+0000 7fee6e59c640 1 --2- 172.21.15.1:0/3401689120 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee700c3970 0x7fee700c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.967+0000 7fee6ed9d640 1 --2- 172.21.15.1:0/3401689120 >> v2:172.21.15.98:3300/0 conn(0x7fee70152840 0x7fee701553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.967+0000 7fee6dd9b640 1 -- 172.21.15.1:0/3401689120 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861091293 0 0) 0x7fee700c5e40 con 0x7fee700c60b0 2024-03-06T07:28:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.967+0000 7fee6dd9b640 1 -- 172.21.15.1:0/3401689120 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee5c003520 con 0x7fee700c60b0 2024-03-06T07:28:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.967+0000 7fee6e59c640 1 -- 172.21.15.1:0/3401689120 >> v2:172.21.15.98:3300/0 conn(0x7fee70152840 msgr2=0x7fee701553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.967+0000 7fee6e59c640 1 --2- 172.21.15.1:0/3401689120 >> v2:172.21.15.98:3300/0 conn(0x7fee70152840 0x7fee701553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.967+0000 7fee6e59c640 1 -- 172.21.15.1:0/3401689120 >> v1:172.21.15.1:6789/0 conn(0x7fee700c60b0 legacy=0x7fee70152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.967+0000 7fee6e59c640 1 -- 172.21.15.1:0/3401689120 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee700c6d20 con 0x7fee700c3970 2024-03-06T07:28:21.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.967+0000 7fee6ed9d640 1 --2- 172.21.15.1:0/3401689120 >> v2:172.21.15.98:3300/0 conn(0x7fee70152840 0x7fee701553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:21.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.968+0000 7fee6e59c640 1 --2- 172.21.15.1:0/3401689120 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee700c3970 0x7fee700c8170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fee600097a0 tx=0x7fee6002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4af04f382201e1a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.968+0000 7fee6dd9b640 1 -- 172.21.15.1:0/3401689120 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee60035020 con 0x7fee700c3970 2024-03-06T07:28:21.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.968+0000 7fee6dd9b640 1 -- 172.21.15.1:0/3401689120 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee60031830 con 0x7fee700c3970 2024-03-06T07:28:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.968+0000 7fee6dd9b640 1 -- 172.21.15.1:0/3401689120 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee60031c00 con 0x7fee700c3970 2024-03-06T07:28:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.969+0000 7fee74b13640 1 -- 172.21.15.1:0/3401689120 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee700c3970 msgr2=0x7fee700c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.969+0000 7fee74b13640 1 --2- 172.21.15.1:0/3401689120 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee700c3970 0x7fee700c8170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fee600097a0 tx=0x7fee6002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.969+0000 7fee74b13640 1 -- 172.21.15.1:0/3401689120 shutdown_connections 2024-03-06T07:28:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.969+0000 7fee74b13640 1 --2- 172.21.15.1:0/3401689120 >> v2:172.21.15.98:3300/0 conn(0x7fee70152840 0x7fee701553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.969+0000 7fee74b13640 1 --2- 172.21.15.1:0/3401689120 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee700c3970 0x7fee700c8170 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.969+0000 7fee74b13640 1 -- 172.21.15.1:0/3401689120 >> 172.21.15.1:0/3401689120 conn(0x7fee700c88f0 msgr2=0x7fee700c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.969+0000 7fee74b13640 1 -- 172.21.15.1:0/3401689120 shutdown_connections 2024-03-06T07:28:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.969+0000 7fee74b13640 1 -- 172.21.15.1:0/3401689120 wait complete. 2024-03-06T07:28:21.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.970+0000 7fee74b13640 1 Processor -- start 2024-03-06T07:28:21.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.970+0000 7fee74b13640 1 -- start start 2024-03-06T07:28:21.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee74b13640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee700c3970 0x7fee70153800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee74b13640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee700c60b0 0x7fee70153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee74b13640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee70152840 0x7fee7014c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee74b13640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee700c75c0 con 0x7fee700c3970 2024-03-06T07:28:21.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee74b13640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee700c72e0 con 0x7fee700c60b0 2024-03-06T07:28:21.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee74b13640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee700c3d50 con 0x7fee70152840 2024-03-06T07:28:21.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee6e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee700c3970 0x7fee70153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee6e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee700c3970 0x7fee70153800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51186/0 (socket says 172.21.15.1:51186) 2024-03-06T07:28:21.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee6e59c640 1 -- 172.21.15.1:0/3726517852 learned_addr learned my addr 172.21.15.1:0/3726517852 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:21.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee67fff640 1 --2- 172.21.15.1:0/3726517852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee700c60b0 0x7fee70153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee6ed9d640 1 --2- 172.21.15.1:0/3726517852 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee70152840 0x7fee7014c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee6e59c640 1 -- 172.21.15.1:0/3726517852 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee70152840 msgr2=0x7fee7014c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee6e59c640 1 --2- 172.21.15.1:0/3726517852 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee70152840 0x7fee7014c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee6e59c640 1 -- 172.21.15.1:0/3726517852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee700c60b0 msgr2=0x7fee70153d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee6e59c640 1 --2- 172.21.15.1:0/3726517852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee700c60b0 0x7fee70153d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee6e59c640 1 -- 172.21.15.1:0/3726517852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee70070ba0 con 0x7fee700c3970 2024-03-06T07:28:21.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee67fff640 1 --2- 172.21.15.1:0/3726517852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee700c60b0 0x7fee70153d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:21.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.971+0000 7fee6ed9d640 1 --2- 172.21.15.1:0/3726517852 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee70152840 0x7fee7014c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:21.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.972+0000 7fee6e59c640 1 --2- 172.21.15.1:0/3726517852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee700c3970 0x7fee70153800 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7fee6002bee0 tx=0x7fee600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.972+0000 7fee66ffd640 1 -- 172.21.15.1:0/3726517852 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee60035040 con 0x7fee700c3970 2024-03-06T07:28:21.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.972+0000 7fee66ffd640 1 -- 172.21.15.1:0/3726517852 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6003b920 con 0x7fee700c3970 2024-03-06T07:28:21.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.972+0000 7fee74b13640 1 -- 172.21.15.1:0/3726517852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee70070e80 con 0x7fee700c3970 2024-03-06T07:28:21.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.972+0000 7fee66ffd640 1 -- 172.21.15.1:0/3726517852 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6003bd10 con 0x7fee700c3970 2024-03-06T07:28:21.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.972+0000 7fee74b13640 1 -- 172.21.15.1:0/3726517852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee700a8a40 con 0x7fee700c3970 2024-03-06T07:28:21.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.973+0000 7fee74b13640 1 -- 172.21.15.1:0/3726517852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee700686e0 con 0x7fee700c3970 2024-03-06T07:28:21.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.973+0000 7fee66ffd640 1 -- 172.21.15.1:0/3726517852 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee60038020 con 0x7fee700c3970 2024-03-06T07:28:21.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.974+0000 7fee66ffd640 1 --2- 172.21.15.1:0/3726517852 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee5006b4d0 0x7fee5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.974+0000 7fee66ffd640 1 -- 172.21.15.1:0/3726517852 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7fee600ac440 con 0x7fee700c3970 2024-03-06T07:28:21.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.974+0000 7fee67fff640 1 --2- 172.21.15.1:0/3726517852 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee5006b4d0 0x7fee5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.974+0000 7fee67fff640 1 --2- 172.21.15.1:0/3726517852 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee5006b4d0 0x7fee5006d990 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fee7014ec50 tx=0x7fee58009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:21.978+0000 7fee66ffd640 1 -- 172.21.15.1:0/3726517852 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee6007b150 con 0x7fee700c3970 2024-03-06T07:28:22.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.198+0000 7fee74b13640 1 -- 172.21.15.1:0/3726517852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee70151930 con 0x7fee700c3970 2024-03-06T07:28:22.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.199+0000 7fee66ffd640 1 -- 172.21.15.1:0/3726517852 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+5005 (secure 0 0 0) 0x7fee6007aa50 con 0x7fee700c3970 2024-03-06T07:28:22.201 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:22.201 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":179,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:52.179842+0000","modified":"2024-03-06T07:28:16.947047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10161,"mds_1":9951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9951":{"gid":9951,"name":"a","rank":1,"incarnation":178,"state":"up:active","state_seq":9,"addr":"172.21.15.1:6833/3895143314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":3895143314},{"type":"v1","addr":"172.21.15.1:6833","nonce":3895143314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10161":{"gid":10161,"name":"c","rank":0,"incarnation":176,"state":"up:active","state_seq":2,"addr":"172.21.15.1:6835/1525330395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1525330395},{"type":"v1","addr":"172.21.15.1:6835","nonce":1525330395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10161,"qdb_cluster":[9951,10161]},"id":14}]} 2024-03-06T07:28:22.201 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 179 2024-03-06T07:28:22.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.200+0000 7fee74b13640 1 -- 172.21.15.1:0/3726517852 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee5006b4d0 msgr2=0x7fee5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.201+0000 7fee74b13640 1 --2- 172.21.15.1:0/3726517852 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee5006b4d0 0x7fee5006d990 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fee7014ec50 tx=0x7fee58009040 comp rx=0 tx=0).stop 2024-03-06T07:28:22.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.201+0000 7fee74b13640 1 -- 172.21.15.1:0/3726517852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee700c3970 msgr2=0x7fee70153800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.201+0000 7fee74b13640 1 --2- 172.21.15.1:0/3726517852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee700c3970 0x7fee70153800 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7fee6002bee0 tx=0x7fee600046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.201+0000 7fee74b13640 1 -- 172.21.15.1:0/3726517852 shutdown_connections 2024-03-06T07:28:22.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.201+0000 7fee74b13640 1 --2- 172.21.15.1:0/3726517852 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fee5006b4d0 0x7fee5006d990 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.201+0000 7fee74b13640 1 --2- 172.21.15.1:0/3726517852 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee70152840 0x7fee7014c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.201+0000 7fee74b13640 1 --2- 172.21.15.1:0/3726517852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee700c60b0 0x7fee70153d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.201+0000 7fee74b13640 1 --2- 172.21.15.1:0/3726517852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee700c3970 0x7fee70153800 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.201+0000 7fee74b13640 1 -- 172.21.15.1:0/3726517852 >> 172.21.15.1:0/3726517852 conn(0x7fee700c88f0 msgr2=0x7fee70074450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.201+0000 7fee74b13640 1 -- 172.21.15.1:0/3726517852 shutdown_connections 2024-03-06T07:28:22.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.201+0000 7fee74b13640 1 -- 172.21.15.1:0/3726517852 wait complete. 2024-03-06T07:28:22.213 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.316+0000 7f8676b9a640 1 Processor -- start 2024-03-06T07:28:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.316+0000 7f8676b9a640 1 -- start start 2024-03-06T07:28:22.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.317+0000 7f8676b9a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8670153070 0x7f8670156b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.317+0000 7f8676b9a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8670156680 con 0x7f8670150640 2024-03-06T07:28:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.317+0000 7f8676b9a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867006c2a0 con 0x7f86700726c0 2024-03-06T07:28:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.317+0000 7f8676b9a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f867006c580 con 0x7f8670153070 2024-03-06T07:28:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.317+0000 7f8674936640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8670150640 0x7f867006ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39796/0 (socket says 172.21.15.1:39796) 2024-03-06T07:28:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.317+0000 7f8674936640 1 -- 172.21.15.1:0/1364745099 learned_addr learned my addr 172.21.15.1:0/1364745099 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f866ffff640 1 --2- 172.21.15.1:0/1364745099 >> v2:172.21.15.98:3300/0 conn(0x7f8670153070 0x7f8670156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f866f7fe640 1 -- 172.21.15.1:0/1364745099 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623857335 0 0) 0x7f8670156680 con 0x7f8670150640 2024-03-06T07:28:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f866f7fe640 1 -- 172.21.15.1:0/1364745099 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8660003520 con 0x7f8670150640 2024-03-06T07:28:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f866f7fe640 1 -- 172.21.15.1:0/1364745099 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3312036553 0 0) 0x7f8660003520 con 0x7f8670150640 2024-03-06T07:28:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f866f7fe640 1 -- 172.21.15.1:0/1364745099 >> v2:172.21.15.98:3300/0 conn(0x7f8670153070 msgr2=0x7f8670156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f866f7fe640 1 --2- 172.21.15.1:0/1364745099 >> v2:172.21.15.98:3300/0 conn(0x7f8670153070 0x7f8670156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f866f7fe640 1 -- 172.21.15.1:0/1364745099 >> v1:172.21.15.98:6789/0 conn(0x7f86700726c0 legacy=0x7f867015d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f866f7fe640 1 -- 172.21.15.1:0/1364745099 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867006c860 con 0x7f8670150640 2024-03-06T07:28:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f866f7fe640 1 -- 172.21.15.1:0/1364745099 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f8664001690 con 0x7f8670150640 2024-03-06T07:28:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f866f7fe640 1 -- 172.21.15.1:0/1364745099 >> v1:172.21.15.1:6789/0 conn(0x7f8670150640 legacy=0x7f867006ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f866f7fe640 1 --2- 172.21.15.1:0/1364745099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86600039b0 0x7f8660003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f866f7fe640 1 --2- 172.21.15.1:0/1364745099 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8660004800 0x7f8660004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f866f7fe640 1 --2- 172.21.15.1:0/1364745099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86600052e0 0x7f866000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f866f7fe640 1 -- 172.21.15.1:0/1364745099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867006c580 con 0x7f86600039b0 2024-03-06T07:28:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f866f7fe640 1 -- 172.21.15.1:0/1364745099 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866000c1b0 con 0x7f86600052e0 2024-03-06T07:28:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f866f7fe640 1 -- 172.21.15.1:0/1364745099 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866000c3b0 con 0x7f8660004800 2024-03-06T07:28:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f8674936640 1 --2- 172.21.15.1:0/1364745099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86600039b0 0x7f8660003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f8675137640 1 --2- 172.21.15.1:0/1364745099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86600052e0 0x7f866000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f866ffff640 1 --2- 172.21.15.1:0/1364745099 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8660004800 0x7f8660004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f8674936640 1 -- 172.21.15.1:0/1364745099 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8660004800 msgr2=0x7f8660004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.318+0000 7f8674936640 1 --2- 172.21.15.1:0/1364745099 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8660004800 0x7f8660004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.319+0000 7f8674936640 1 -- 172.21.15.1:0/1364745099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86600052e0 msgr2=0x7f866000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.319+0000 7f8674936640 1 --2- 172.21.15.1:0/1364745099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86600052e0 0x7f866000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.319+0000 7f8674936640 1 -- 172.21.15.1:0/1364745099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8660003520 con 0x7f86600039b0 2024-03-06T07:28:22.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.319+0000 7f8675137640 1 -- 172.21.15.1:0/1364745099 reap_dead start 2024-03-06T07:28:22.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.320+0000 7f8674936640 1 --2- 172.21.15.1:0/1364745099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86600039b0 0x7f8660003dd0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f8670156850 tx=0x7f8660005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=698d796068032be9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.320+0000 7f866f7fe640 1 -- 172.21.15.1:0/1364745099 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8664007d50 con 0x7f86600039b0 2024-03-06T07:28:22.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.320+0000 7f866f7fe640 1 -- 172.21.15.1:0/1364745099 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8664030070 con 0x7f86600039b0 2024-03-06T07:28:22.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.320+0000 7f866f7fe640 1 -- 172.21.15.1:0/1364745099 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8664005d30 con 0x7f86600039b0 2024-03-06T07:28:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.320+0000 7f8676b9a640 1 -- 172.21.15.1:0/1364745099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86600039b0 msgr2=0x7f8660003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.320+0000 7f8676b9a640 1 --2- 172.21.15.1:0/1364745099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86600039b0 0x7f8660003dd0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f8670156850 tx=0x7f8660005830 comp rx=0 tx=0).stop 2024-03-06T07:28:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.321+0000 7f8676b9a640 1 -- 172.21.15.1:0/1364745099 shutdown_connections 2024-03-06T07:28:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.321+0000 7f8676b9a640 1 --2- 172.21.15.1:0/1364745099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86600039b0 0x7f8660003dd0 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.321+0000 7f8676b9a640 1 -- 172.21.15.1:0/1364745099 >> 172.21.15.1:0/1364745099 conn(0x7f86700c71c0 msgr2=0x7f86700c75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.321+0000 7f8676b9a640 1 -- 172.21.15.1:0/1364745099 shutdown_connections 2024-03-06T07:28:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.321+0000 7f8676b9a640 1 -- 172.21.15.1:0/1364745099 wait complete. 2024-03-06T07:28:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.322+0000 7f8676b9a640 1 Processor -- start 2024-03-06T07:28:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.322+0000 7f8676b9a640 1 -- start start 2024-03-06T07:28:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.323+0000 7f8676b9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86600039b0 0x7f86700bc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.323+0000 7f8676b9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86701dcd10 0x7f86701ebf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.323+0000 7f8676b9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86701dd190 0x7f86701ec460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.323+0000 7f8676b9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8670150a20 con 0x7f86701dd190 2024-03-06T07:28:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.323+0000 7f8676b9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86700a90d0 con 0x7f86701dcd10 2024-03-06T07:28:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.323+0000 7f8676b9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8670150ba0 con 0x7f86600039b0 2024-03-06T07:28:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.323+0000 7f8675137640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86701dd190 0x7f86701ec460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.323+0000 7f8675137640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86701dd190 0x7f86701ec460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51206/0 (socket says 172.21.15.1:51206) 2024-03-06T07:28:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.323+0000 7f8675137640 1 -- 172.21.15.1:0/656244755 learned_addr learned my addr 172.21.15.1:0/656244755 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.323+0000 7f866ffff640 1 --2- 172.21.15.1:0/656244755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86701dcd10 0x7f86701ebf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.323+0000 7f8674936640 1 --2- 172.21.15.1:0/656244755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86600039b0 0x7f86700bc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.323+0000 7f8675137640 1 -- 172.21.15.1:0/656244755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86600039b0 msgr2=0x7f86700bc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.323+0000 7f8675137640 1 --2- 172.21.15.1:0/656244755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86600039b0 0x7f86700bc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.323+0000 7f8675137640 1 -- 172.21.15.1:0/656244755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86701dcd10 msgr2=0x7f86701ebf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.323+0000 7f8675137640 1 --2- 172.21.15.1:0/656244755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86701dcd10 0x7f86701ebf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.323+0000 7f8675137640 1 -- 172.21.15.1:0/656244755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86701ddff0 con 0x7f86701dd190 2024-03-06T07:28:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.324+0000 7f8675137640 1 --2- 172.21.15.1:0/656244755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86701dd190 0x7f86701ec460 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f8670072ad0 tx=0x7f866800fb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.324+0000 7f866d7fa640 1 -- 172.21.15.1:0/656244755 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866801a070 con 0x7f86701dd190 2024-03-06T07:28:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.324+0000 7f866d7fa640 1 -- 172.21.15.1:0/656244755 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8668006b50 con 0x7f86701dd190 2024-03-06T07:28:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.324+0000 7f8676b9a640 1 -- 172.21.15.1:0/656244755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86701ec9a0 con 0x7f86701dd190 2024-03-06T07:28:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.324+0000 7f866d7fa640 1 -- 172.21.15.1:0/656244755 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8668003970 con 0x7f86701dd190 2024-03-06T07:28:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.324+0000 7f8676b9a640 1 -- 172.21.15.1:0/656244755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86701ddd10 con 0x7f86701dd190 2024-03-06T07:28:22.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.325+0000 7f8676b9a640 1 -- 172.21.15.1:0/656244755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86700686e0 con 0x7f86701dd190 2024-03-06T07:28:22.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.326+0000 7f866d7fa640 1 -- 172.21.15.1:0/656244755 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8668003b10 con 0x7f86701dd190 2024-03-06T07:28:22.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.326+0000 7f866d7fa640 1 --2- 172.21.15.1:0/656244755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f865406b500 0x7f865406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.326+0000 7f866d7fa640 1 -- 172.21.15.1:0/656244755 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f8668089790 con 0x7f86701dd190 2024-03-06T07:28:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.326+0000 7f8674936640 1 --2- 172.21.15.1:0/656244755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f865406b500 0x7f865406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.327+0000 7f8674936640 1 --2- 172.21.15.1:0/656244755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f865406b500 0x7f865406d9c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f8664034000 tx=0x7f8664009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.331+0000 7f866d7fa640 1 -- 172.21.15.1:0/656244755 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86680584a0 con 0x7f86701dd190 2024-03-06T07:28:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.513+0000 7f8676b9a640 1 -- 172.21.15.1:0/656244755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f867006fa60 con 0x7f86701dd190 2024-03-06T07:28:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.514+0000 7f866d7fa640 1 -- 172.21.15.1:0/656244755 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) v1 ==== 74+0+16947 (secure 0 0 0) 0x7f8668057da0 con 0x7f86701dd190 2024-03-06T07:28:22.517 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:22.517 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":185,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:28:10.884250+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:50.456844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:51.147612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"29.0","mappings":[{"from":6,"to":0}]},{"pgid":"29.c","mappings":[{"from":6,"to":0}]},{"pgid":"29.f","mappings":[{"from":6,"to":0}]},{"pgid":"29.16","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:22.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.517+0000 7f8676b9a640 1 -- 172.21.15.1:0/656244755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f865406b500 msgr2=0x7f865406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.517+0000 7f8676b9a640 1 --2- 172.21.15.1:0/656244755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f865406b500 0x7f865406d9c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f8664034000 tx=0x7f8664009030 comp rx=0 tx=0).stop 2024-03-06T07:28:22.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.517+0000 7f8676b9a640 1 -- 172.21.15.1:0/656244755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86701dd190 msgr2=0x7f86701ec460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.517+0000 7f8676b9a640 1 --2- 172.21.15.1:0/656244755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86701dd190 0x7f86701ec460 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f8670072ad0 tx=0x7f866800fb20 comp rx=0 tx=0).stop 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.517+0000 7f8676b9a640 1 -- 172.21.15.1:0/656244755 shutdown_connections 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.517+0000 7f8676b9a640 1 --2- 172.21.15.1:0/656244755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f865406b500 0x7f865406d9c0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.517+0000 7f8676b9a640 1 --2- 172.21.15.1:0/656244755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86701dd190 0x7f86701ec460 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.517+0000 7f8676b9a640 1 --2- 172.21.15.1:0/656244755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86701dcd10 0x7f86701ebf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.517+0000 7f8676b9a640 1 --2- 172.21.15.1:0/656244755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86600039b0 0x7f86700bc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.517+0000 7f8676b9a640 1 -- 172.21.15.1:0/656244755 >> 172.21.15.1:0/656244755 conn(0x7f86700c71c0 msgr2=0x7f867014fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.517+0000 7f8676b9a640 1 -- 172.21.15.1:0/656244755 shutdown_connections 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.517+0000 7f8676b9a640 1 -- 172.21.15.1:0/656244755 wait complete. 2024-03-06T07:28:22.531 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:28:22.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.652+0000 7f6e47e15640 1 Processor -- start 2024-03-06T07:28:22.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.652+0000 7f6e47e15640 1 -- start start 2024-03-06T07:28:22.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.653+0000 7f6e47e15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e400c2b30 0x7f6e400724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.653+0000 7f6e47e15640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e40151dd0 con 0x7f6e40152b70 2024-03-06T07:28:22.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.653+0000 7f6e47e15640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e40156190 con 0x7f6e400a5fa0 2024-03-06T07:28:22.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.653+0000 7f6e47e15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e401526e0 con 0x7f6e400c2b30 2024-03-06T07:28:22.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.653+0000 7f6e453b0640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6e40152b70 0x7f6e400c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39804/0 (socket says 172.21.15.1:39804) 2024-03-06T07:28:22.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.653+0000 7f6e453b0640 1 -- 172.21.15.1:0/2369478947 learned_addr learned my addr 172.21.15.1:0/2369478947 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:22.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.654+0000 7f6e45bb1640 1 --2- 172.21.15.1:0/2369478947 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e400c2b30 0x7f6e400724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.654+0000 7f6e44baf640 1 -- 172.21.15.1:0/2369478947 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084348171 0 0) 0x7f6e40151dd0 con 0x7f6e40152b70 2024-03-06T07:28:22.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.654+0000 7f6e44baf640 1 -- 172.21.15.1:0/2369478947 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e2c003520 con 0x7f6e40152b70 2024-03-06T07:28:22.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.654+0000 7f6e45bb1640 1 -- 172.21.15.1:0/2369478947 >> v1:172.21.15.98:6789/0 conn(0x7f6e400a5fa0 legacy=0x7f6e4015c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.654+0000 7f6e45bb1640 1 -- 172.21.15.1:0/2369478947 >> v1:172.21.15.1:6789/0 conn(0x7f6e40152b70 legacy=0x7f6e400c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.654+0000 7f6e44baf640 1 -- 172.21.15.1:0/2369478947 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877371716 0 0) 0x7f6e40156190 con 0x7f6e400a5fa0 2024-03-06T07:28:22.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.654+0000 7f6e45bb1640 1 -- 172.21.15.1:0/2369478947 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e40152090 con 0x7f6e400c2b30 2024-03-06T07:28:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.655+0000 7f6e45bb1640 1 --2- 172.21.15.1:0/2369478947 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e400c2b30 0x7f6e400724c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c0097d0 tx=0x7f6e3c02b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4400c90c0f448f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.655+0000 7f6e44baf640 1 -- 172.21.15.1:0/2369478947 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c037070 con 0x7f6e400c2b30 2024-03-06T07:28:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.655+0000 7f6e44baf640 1 -- 172.21.15.1:0/2369478947 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3c030510 con 0x7f6e400c2b30 2024-03-06T07:28:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.655+0000 7f6e44baf640 1 -- 172.21.15.1:0/2369478947 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e30001670 con 0x7f6e400c2b30 2024-03-06T07:28:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.656+0000 7f6e47e15640 1 -- 172.21.15.1:0/2369478947 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e400c2b30 msgr2=0x7f6e400724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.656+0000 7f6e47e15640 1 --2- 172.21.15.1:0/2369478947 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e400c2b30 0x7f6e400724c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c0097d0 tx=0x7f6e3c02b730 comp rx=0 tx=0).stop 2024-03-06T07:28:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.656+0000 7f6e47e15640 1 -- 172.21.15.1:0/2369478947 shutdown_connections 2024-03-06T07:28:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.656+0000 7f6e47e15640 1 --2- 172.21.15.1:0/2369478947 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e400c2b30 0x7f6e400724c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.656+0000 7f6e47e15640 1 -- 172.21.15.1:0/2369478947 >> 172.21.15.1:0/2369478947 conn(0x7f6e400c6860 msgr2=0x7f6e400c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.656+0000 7f6e47e15640 1 -- 172.21.15.1:0/2369478947 shutdown_connections 2024-03-06T07:28:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.656+0000 7f6e47e15640 1 -- 172.21.15.1:0/2369478947 wait complete. 2024-03-06T07:28:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.657+0000 7f6e47e15640 1 Processor -- start 2024-03-06T07:28:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.657+0000 7f6e47e15640 1 -- start start 2024-03-06T07:28:22.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.658+0000 7f6e47e15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e400a5fa0 0x7f6e4014e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.658+0000 7f6e47e15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e400c2b30 0x7f6e401dec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.658+0000 7f6e47e15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e40152b70 0x7f6e401439e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.658+0000 7f6e47e15640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e40155f00 con 0x7f6e400c2b30 2024-03-06T07:28:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.658+0000 7f6e47e15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e40155d80 con 0x7f6e40152b70 2024-03-06T07:28:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.658+0000 7f6e47e15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e40072a00 con 0x7f6e400a5fa0 2024-03-06T07:28:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.658+0000 7f6e453b0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e400c2b30 0x7f6e401dec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.658+0000 7f6e453b0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e400c2b30 0x7f6e401dec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51210/0 (socket says 172.21.15.1:51210) 2024-03-06T07:28:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.658+0000 7f6e453b0640 1 -- 172.21.15.1:0/3508883611 learned_addr learned my addr 172.21.15.1:0/3508883611 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.658+0000 7f6e463b2640 1 --2- 172.21.15.1:0/3508883611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e40152b70 0x7f6e401439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.658+0000 7f6e45bb1640 1 --2- 172.21.15.1:0/3508883611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e400a5fa0 0x7f6e4014e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.658+0000 7f6e453b0640 1 -- 172.21.15.1:0/3508883611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e400a5fa0 msgr2=0x7f6e4014e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.658+0000 7f6e453b0640 1 --2- 172.21.15.1:0/3508883611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e400a5fa0 0x7f6e4014e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.659+0000 7f6e453b0640 1 -- 172.21.15.1:0/3508883611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e40152b70 msgr2=0x7f6e401439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.659+0000 7f6e453b0640 1 --2- 172.21.15.1:0/3508883611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e40152b70 0x7f6e401439e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.659+0000 7f6e453b0640 1 -- 172.21.15.1:0/3508883611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4014e870 con 0x7f6e400c2b30 2024-03-06T07:28:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.659+0000 7f6e463b2640 1 --2- 172.21.15.1:0/3508883611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e40152b70 0x7f6e401439e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.659+0000 7f6e45bb1640 1 --2- 172.21.15.1:0/3508883611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e400a5fa0 0x7f6e4014e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.659+0000 7f6e453b0640 1 --2- 172.21.15.1:0/3508883611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e400c2b30 0x7f6e401dec40 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f6e3000c7f0 tx=0x7f6e3000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.659+0000 7f6e367fc640 1 -- 172.21.15.1:0/3508883611 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e30018070 con 0x7f6e400c2b30 2024-03-06T07:28:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.659+0000 7f6e367fc640 1 -- 172.21.15.1:0/3508883611 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e30007500 con 0x7f6e400c2b30 2024-03-06T07:28:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.659+0000 7f6e367fc640 1 -- 172.21.15.1:0/3508883611 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e30004a20 con 0x7f6e400c2b30 2024-03-06T07:28:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.659+0000 7f6e47e15640 1 -- 172.21.15.1:0/3508883611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e401ddba0 con 0x7f6e400c2b30 2024-03-06T07:28:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.659+0000 7f6e47e15640 1 -- 172.21.15.1:0/3508883611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e401de440 con 0x7f6e400c2b30 2024-03-06T07:28:22.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.660+0000 7f6e47e15640 1 -- 172.21.15.1:0/3508883611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e40067fd0 con 0x7f6e400c2b30 2024-03-06T07:28:22.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.661+0000 7f6e367fc640 1 -- 172.21.15.1:0/3508883611 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e3001b070 con 0x7f6e400c2b30 2024-03-06T07:28:22.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.661+0000 7f6e367fc640 1 --2- 172.21.15.1:0/3508883611 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e2406b4d0 0x7f6e2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.661+0000 7f6e367fc640 1 -- 172.21.15.1:0/3508883611 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f6e30089bd0 con 0x7f6e400c2b30 2024-03-06T07:28:22.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.661+0000 7f6e45bb1640 1 --2- 172.21.15.1:0/3508883611 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e2406b4d0 0x7f6e2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.662+0000 7f6e45bb1640 1 --2- 172.21.15.1:0/3508883611 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e2406b4d0 0x7f6e2406d990 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c034000 tx=0x7f6e3c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.669+0000 7f6e367fc640 1 -- 172.21.15.1:0/3508883611 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e300589c0 con 0x7f6e400c2b30 2024-03-06T07:28:22.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.888+0000 7f6e47e15640 1 -- 172.21.15.1:0/3508883611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6e400c76c0 con 0x7f6e400c2b30 2024-03-06T07:28:22.890 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-06T07:28:22.888+0000 7fb850f8c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:28:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.890+0000 7f6e367fc640 1 -- 172.21.15.1:0/3508883611 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v180) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6e300582c0 con 0x7f6e400c2b30 2024-03-06T07:28:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:28:22.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.893+0000 7f6e47e15640 1 -- 172.21.15.1:0/3508883611 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e2406b4d0 msgr2=0x7f6e2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.893+0000 7f6e47e15640 1 --2- 172.21.15.1:0/3508883611 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e2406b4d0 0x7f6e2406d990 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c034000 tx=0x7f6e3c008030 comp rx=0 tx=0).stop 2024-03-06T07:28:22.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.893+0000 7f6e47e15640 1 -- 172.21.15.1:0/3508883611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e400c2b30 msgr2=0x7f6e401dec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.893+0000 7f6e47e15640 1 --2- 172.21.15.1:0/3508883611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e400c2b30 0x7f6e401dec40 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f6e3000c7f0 tx=0x7f6e3000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.893+0000 7f6e47e15640 1 -- 172.21.15.1:0/3508883611 shutdown_connections 2024-03-06T07:28:22.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.893+0000 7f6e47e15640 1 --2- 172.21.15.1:0/3508883611 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6e2406b4d0 0x7f6e2406d990 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.893+0000 7f6e47e15640 1 --2- 172.21.15.1:0/3508883611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e40152b70 0x7f6e401439e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.893+0000 7f6e47e15640 1 --2- 172.21.15.1:0/3508883611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e400c2b30 0x7f6e401dec40 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.893+0000 7f6e47e15640 1 --2- 172.21.15.1:0/3508883611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e400a5fa0 0x7f6e4014e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.893+0000 7f6e47e15640 1 -- 172.21.15.1:0/3508883611 >> 172.21.15.1:0/3508883611 conn(0x7f6e400c6860 msgr2=0x7f6e400c4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.893+0000 7f6e47e15640 1 -- 172.21.15.1:0/3508883611 shutdown_connections 2024-03-06T07:28:22.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:22.893+0000 7f6e47e15640 1 -- 172.21.15.1:0/3508883611 wait complete. 2024-03-06T07:28:22.896 INFO:tasks.ceph.mds.c.smithi001.stderr: -104> 2024-03-06T07:28:19.915+0000 7f6a441f2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:22.896 INFO:tasks.ceph.mds.c.smithi001.stderr: -102> 2024-03-06T07:28:19.915+0000 7f6a441f2640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:28:22.908 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:28:22.921 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:28:22.921 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:28:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.011+0000 7f430c7d7640 1 Processor -- start 2024-03-06T07:28:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.011+0000 7f430c7d7640 1 -- start start 2024-03-06T07:28:23.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.012+0000 7f430c7d7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43040c2b30 0x7f43040724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.012+0000 7f430c7d7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43040a5fa0 0x7f4304072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.012+0000 7f430c7d7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4304151dd0 con 0x7f4304152b70 2024-03-06T07:28:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.012+0000 7f430c7d7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43041526e0 con 0x7f43040a5fa0 2024-03-06T07:28:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.012+0000 7f430c7d7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4304156190 con 0x7f43040c2b30 2024-03-06T07:28:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.012+0000 7f4309d72640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4304152b70 0x7f43040c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39836/0 (socket says 172.21.15.1:39836) 2024-03-06T07:28:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309d72640 1 -- 172.21.15.1:0/3491375928 learned_addr learned my addr 172.21.15.1:0/3491375928 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f430a573640 1 --2- 172.21.15.1:0/3491375928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43040c2b30 0x7f43040724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f430ad74640 1 --2- 172.21.15.1:0/3491375928 >> v2:172.21.15.98:3300/0 conn(0x7f43040a5fa0 0x7f4304072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309571640 1 -- 172.21.15.1:0/3491375928 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125945664 0 0) 0x7f4304151dd0 con 0x7f4304152b70 2024-03-06T07:28:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309571640 1 -- 172.21.15.1:0/3491375928 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42f0003520 con 0x7f4304152b70 2024-03-06T07:28:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309571640 1 -- 172.21.15.1:0/3491375928 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3554893040 0 0) 0x7f42f0003520 con 0x7f4304152b70 2024-03-06T07:28:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309571640 1 -- 172.21.15.1:0/3491375928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43040c2b30 msgr2=0x7f43040724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309571640 1 --2- 172.21.15.1:0/3491375928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43040c2b30 0x7f43040724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309571640 1 -- 172.21.15.1:0/3491375928 >> v2:172.21.15.98:3300/0 conn(0x7f43040a5fa0 msgr2=0x7f4304072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309571640 1 --2- 172.21.15.1:0/3491375928 >> v2:172.21.15.98:3300/0 conn(0x7f43040a5fa0 0x7f4304072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309571640 1 -- 172.21.15.1:0/3491375928 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4304152090 con 0x7f4304152b70 2024-03-06T07:28:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309571640 1 -- 172.21.15.1:0/3491375928 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f42f4001670 con 0x7f4304152b70 2024-03-06T07:28:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309571640 1 -- 172.21.15.1:0/3491375928 >> v1:172.21.15.1:6789/0 conn(0x7f4304152b70 legacy=0x7f43040c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309571640 1 --2- 172.21.15.1:0/3491375928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f00039b0 0x7f42f0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309571640 1 --2- 172.21.15.1:0/3491375928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42f0004800 0x7f42f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309571640 1 --2- 172.21.15.1:0/3491375928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43040c2b30 0x7f42f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309571640 1 -- 172.21.15.1:0/3491375928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43041526e0 con 0x7f42f00039b0 2024-03-06T07:28:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309571640 1 -- 172.21.15.1:0/3491375928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4304156190 con 0x7f42f0004800 2024-03-06T07:28:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309571640 1 -- 172.21.15.1:0/3491375928 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f000b9c0 con 0x7f43040c2b30 2024-03-06T07:28:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f430a573640 1 --2- 172.21.15.1:0/3491375928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f00039b0 0x7f42f0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f4309d72640 1 --2- 172.21.15.1:0/3491375928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42f0004800 0x7f42f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.013+0000 7f430ad74640 1 --2- 172.21.15.1:0/3491375928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43040c2b30 0x7f42f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.014+0000 7f430a573640 1 -- 172.21.15.1:0/3491375928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43040c2b30 msgr2=0x7f42f00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.014+0000 7f430a573640 1 --2- 172.21.15.1:0/3491375928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43040c2b30 0x7f42f00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.014+0000 7f430a573640 1 -- 172.21.15.1:0/3491375928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42f0004800 msgr2=0x7f42f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.014+0000 7f430a573640 1 --2- 172.21.15.1:0/3491375928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42f0004800 0x7f42f0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.014+0000 7f430a573640 1 -- 172.21.15.1:0/3491375928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f0003520 con 0x7f42f00039b0 2024-03-06T07:28:23.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.014+0000 7f430a573640 1 --2- 172.21.15.1:0/3491375928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f00039b0 0x7f42f0003dd0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f4300009650 tx=0x7f42f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=546773d5d62f5b09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.014+0000 7f4309571640 1 -- 172.21.15.1:0/3491375928 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430003b070 con 0x7f42f00039b0 2024-03-06T07:28:23.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.014+0000 7f4309571640 1 -- 172.21.15.1:0/3491375928 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430002e8c0 con 0x7f42f00039b0 2024-03-06T07:28:23.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.014+0000 7f4309571640 1 -- 172.21.15.1:0/3491375928 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4300009690 con 0x7f42f00039b0 2024-03-06T07:28:23.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.015+0000 7f430c7d7640 1 -- 172.21.15.1:0/3491375928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f00039b0 msgr2=0x7f42f0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.015+0000 7f430c7d7640 1 --2- 172.21.15.1:0/3491375928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f00039b0 0x7f42f0003dd0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f4300009650 tx=0x7f42f0005830 comp rx=0 tx=0).stop 2024-03-06T07:28:23.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.015+0000 7f430ad74640 1 -- 172.21.15.1:0/3491375928 reap_dead start 2024-03-06T07:28:23.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.015+0000 7f430c7d7640 1 -- 172.21.15.1:0/3491375928 shutdown_connections 2024-03-06T07:28:23.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.015+0000 7f430c7d7640 1 -- 172.21.15.1:0/3491375928 >> 172.21.15.1:0/3491375928 conn(0x7f43040c6860 msgr2=0x7f43040c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.015+0000 7f430c7d7640 1 -- 172.21.15.1:0/3491375928 shutdown_connections 2024-03-06T07:28:23.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.015+0000 7f430c7d7640 1 -- 172.21.15.1:0/3491375928 wait complete. 2024-03-06T07:28:23.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.016+0000 7f430c7d7640 1 Processor -- start 2024-03-06T07:28:23.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.016+0000 7f430c7d7640 1 -- start start 2024-03-06T07:28:23.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f430c7d7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42f00039b0 0x7f4304150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f430c7d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f430414a280 0x7f4304149700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f430c7d7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43041eebf0 0x7f4304149c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f430c7d7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4304155f00 con 0x7f43041eebf0 2024-03-06T07:28:23.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f430c7d7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4304155d80 con 0x7f430414a280 2024-03-06T07:28:23.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f430c7d7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43040724c0 con 0x7f42f00039b0 2024-03-06T07:28:23.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f430ad74640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43041eebf0 0x7f4304149c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f430ad74640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43041eebf0 0x7f4304149c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51272/0 (socket says 172.21.15.1:51272) 2024-03-06T07:28:23.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f430ad74640 1 -- 172.21.15.1:0/1388011664 learned_addr learned my addr 172.21.15.1:0/1388011664 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:23.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f4309d72640 1 --2- 172.21.15.1:0/1388011664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f430414a280 0x7f4304149700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f430a573640 1 --2- 172.21.15.1:0/1388011664 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42f00039b0 0x7f4304150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f430ad74640 1 -- 172.21.15.1:0/1388011664 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42f00039b0 msgr2=0x7f4304150310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f430ad74640 1 --2- 172.21.15.1:0/1388011664 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42f00039b0 0x7f4304150310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f430ad74640 1 -- 172.21.15.1:0/1388011664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f430414a280 msgr2=0x7f4304149700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f430ad74640 1 --2- 172.21.15.1:0/1388011664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f430414a280 0x7f4304149700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f430ad74640 1 -- 172.21.15.1:0/1388011664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430406ed80 con 0x7f43041eebf0 2024-03-06T07:28:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f430a573640 1 --2- 172.21.15.1:0/1388011664 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42f00039b0 0x7f4304150310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.017+0000 7f430ad74640 1 --2- 172.21.15.1:0/1388011664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43041eebf0 0x7f4304149c40 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f43040a5f30 tx=0x7f42fc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.018+0000 7f42faffd640 1 -- 172.21.15.1:0/1388011664 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc011e60 con 0x7f43041eebf0 2024-03-06T07:28:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.018+0000 7f42faffd640 1 -- 172.21.15.1:0/1388011664 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42fc015430 con 0x7f43041eebf0 2024-03-06T07:28:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.018+0000 7f430c7d7640 1 -- 172.21.15.1:0/1388011664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f430406f060 con 0x7f43041eebf0 2024-03-06T07:28:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.018+0000 7f42faffd640 1 -- 172.21.15.1:0/1388011664 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc015820 con 0x7f43041eebf0 2024-03-06T07:28:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.018+0000 7f430c7d7640 1 -- 172.21.15.1:0/1388011664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43041ddbe0 con 0x7f43041eebf0 2024-03-06T07:28:23.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.019+0000 7f430c7d7640 1 -- 172.21.15.1:0/1388011664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4304067fd0 con 0x7f43041eebf0 2024-03-06T07:28:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.019+0000 7f42faffd640 1 -- 172.21.15.1:0/1388011664 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f42fc0159c0 con 0x7f43041eebf0 2024-03-06T07:28:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.019+0000 7f42faffd640 1 --2- 172.21.15.1:0/1388011664 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f42e806b4d0 0x7f42e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.020+0000 7f42faffd640 1 -- 172.21.15.1:0/1388011664 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f42fc08b190 con 0x7f43041eebf0 2024-03-06T07:28:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.020+0000 7f430a573640 1 --2- 172.21.15.1:0/1388011664 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f42e806b4d0 0x7f42e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.020+0000 7f430a573640 1 --2- 172.21.15.1:0/1388011664 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f42e806b4d0 0x7f42e806d990 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f4300038000 tx=0x7f4300032000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.024+0000 7f42faffd640 1 -- 172.21.15.1:0/1388011664 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42fc019030 con 0x7f43041eebf0 2024-03-06T07:28:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.244+0000 7f430c7d7640 1 -- 172.21.15.1:0/1388011664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4304071790 con 0x7f43041eebf0 2024-03-06T07:28:23.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.894+0000 7f42faffd640 1 -- 172.21.15.1:0/1388011664 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v181) v1 ==== 104+0+0 (secure 0 0 0) 0x7f42fc010040 con 0x7f43041eebf0 2024-03-06T07:28:23.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.896+0000 7f430c7d7640 1 -- 172.21.15.1:0/1388011664 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f42e806b4d0 msgr2=0x7f42e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.896+0000 7f430c7d7640 1 --2- 172.21.15.1:0/1388011664 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f42e806b4d0 0x7f42e806d990 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f4300038000 tx=0x7f4300032000 comp rx=0 tx=0).stop 2024-03-06T07:28:23.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.896+0000 7f430c7d7640 1 -- 172.21.15.1:0/1388011664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43041eebf0 msgr2=0x7f4304149c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.896+0000 7f430c7d7640 1 --2- 172.21.15.1:0/1388011664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43041eebf0 0x7f4304149c40 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f43040a5f30 tx=0x7f42fc00f580 comp rx=0 tx=0).stop 2024-03-06T07:28:23.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.896+0000 7f430c7d7640 1 -- 172.21.15.1:0/1388011664 shutdown_connections 2024-03-06T07:28:23.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.896+0000 7f430c7d7640 1 --2- 172.21.15.1:0/1388011664 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f42e806b4d0 0x7f42e806d990 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.896+0000 7f430c7d7640 1 --2- 172.21.15.1:0/1388011664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43041eebf0 0x7f4304149c40 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.896+0000 7f430c7d7640 1 --2- 172.21.15.1:0/1388011664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f430414a280 0x7f4304149700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.896+0000 7f430c7d7640 1 --2- 172.21.15.1:0/1388011664 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42f00039b0 0x7f4304150310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.896+0000 7f430c7d7640 1 -- 172.21.15.1:0/1388011664 >> 172.21.15.1:0/1388011664 conn(0x7f43040c6860 msgr2=0x7f43040c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.896+0000 7f430c7d7640 1 -- 172.21.15.1:0/1388011664 shutdown_connections 2024-03-06T07:28:23.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:23.897+0000 7f430c7d7640 1 -- 172.21.15.1:0/1388011664 wait complete. 2024-03-06T07:28:23.912 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:28:24.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.060+0000 7f9ea126b640 1 Processor -- start 2024-03-06T07:28:24.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.060+0000 7f9ea126b640 1 -- start start 2024-03-06T07:28:24.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.061+0000 7f9ea126b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9e9c0c79c0 0x7f9e9c154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.061+0000 7f9ea126b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c1571c0 con 0x7f9e9c0c3400 2024-03-06T07:28:24.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.061+0000 7f9ea126b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c153130 con 0x7f9e9c150750 2024-03-06T07:28:24.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.061+0000 7f9ea126b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e9c154040 con 0x7f9e9c0c79c0 2024-03-06T07:28:24.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.061+0000 7f9e9a7fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9e9c0c3400 0x7f9e9c070d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39840/0 (socket says 172.21.15.1:39840) 2024-03-06T07:28:24.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.061+0000 7f9e9a7fc640 1 -- 172.21.15.1:0/486929216 learned_addr learned my addr 172.21.15.1:0/486929216 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:24.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.061+0000 7f9e9b7fe640 1 --2- 172.21.15.1:0/486929216 >> v2:172.21.15.98:3300/0 conn(0x7f9e9c0c79c0 0x7f9e9c154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.062+0000 7f9e99ffb640 1 -- 172.21.15.1:0/486929216 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673843707 0 0) 0x7f9e9c1571c0 con 0x7f9e9c0c3400 2024-03-06T07:28:24.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.062+0000 7f9e99ffb640 1 -- 172.21.15.1:0/486929216 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e88003520 con 0x7f9e9c0c3400 2024-03-06T07:28:24.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.062+0000 7f9e9b7fe640 1 -- 172.21.15.1:0/486929216 >> v1:172.21.15.98:6789/0 conn(0x7f9e9c150750 legacy=0x7f9e9c153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.062+0000 7f9e99ffb640 1 -- 172.21.15.1:0/486929216 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2880616070 0 0) 0x7f9e88003520 con 0x7f9e9c0c3400 2024-03-06T07:28:24.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.062+0000 7f9e9b7fe640 1 -- 172.21.15.1:0/486929216 >> v1:172.21.15.1:6789/0 conn(0x7f9e9c0c3400 legacy=0x7f9e9c070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.062+0000 7f9e9b7fe640 1 -- 172.21.15.1:0/486929216 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c153410 con 0x7f9e9c0c79c0 2024-03-06T07:28:24.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.063+0000 7f9e9b7fe640 1 --2- 172.21.15.1:0/486929216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9c0c79c0 0x7f9e9c154510 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009d90 tx=0x7f9e8c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0415b03247a3df9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.063+0000 7f9e99ffb640 1 -- 172.21.15.1:0/486929216 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c039070 con 0x7f9e9c0c79c0 2024-03-06T07:28:24.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.063+0000 7f9e99ffb640 1 -- 172.21.15.1:0/486929216 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8c02c940 con 0x7f9e9c0c79c0 2024-03-06T07:28:24.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.063+0000 7f9e99ffb640 1 -- 172.21.15.1:0/486929216 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e84001670 con 0x7f9e9c0c79c0 2024-03-06T07:28:24.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.063+0000 7f9ea126b640 1 -- 172.21.15.1:0/486929216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9c0c79c0 msgr2=0x7f9e9c154510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.063+0000 7f9ea126b640 1 --2- 172.21.15.1:0/486929216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9c0c79c0 0x7f9e9c154510 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009d90 tx=0x7f9e8c02be10 comp rx=0 tx=0).stop 2024-03-06T07:28:24.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.063+0000 7f9ea126b640 1 -- 172.21.15.1:0/486929216 shutdown_connections 2024-03-06T07:28:24.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.063+0000 7f9ea126b640 1 --2- 172.21.15.1:0/486929216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9c0c79c0 0x7f9e9c154510 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.063+0000 7f9ea126b640 1 -- 172.21.15.1:0/486929216 >> 172.21.15.1:0/486929216 conn(0x7f9e9c06b620 msgr2=0x7f9e9c0c8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.063+0000 7f9ea126b640 1 -- 172.21.15.1:0/486929216 shutdown_connections 2024-03-06T07:28:24.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.063+0000 7f9ea126b640 1 -- 172.21.15.1:0/486929216 wait complete. 2024-03-06T07:28:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.065+0000 7f9ea126b640 1 Processor -- start 2024-03-06T07:28:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.065+0000 7f9ea126b640 1 -- start start 2024-03-06T07:28:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.065+0000 7f9ea126b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e9c0c3400 0x7f9e9c14caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.065+0000 7f9ea126b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e9c0c79c0 0x7f9e9c1dfa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.065+0000 7f9ea126b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9c150750 0x7f9e9c0c3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.065+0000 7f9ea126b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c0c7e40 con 0x7f9e9c0c3400 2024-03-06T07:28:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.065+0000 7f9ea126b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c1574a0 con 0x7f9e9c150750 2024-03-06T07:28:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.065+0000 7f9ea126b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c0c7fc0 con 0x7f9e9c0c79c0 2024-03-06T07:28:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.065+0000 7f9e9affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e9c0c3400 0x7f9e9c14caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.065+0000 7f9e9affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e9c0c3400 0x7f9e9c14caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51274/0 (socket says 172.21.15.1:51274) 2024-03-06T07:28:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.065+0000 7f9e9affd640 1 -- 172.21.15.1:0/3405483243 learned_addr learned my addr 172.21.15.1:0/3405483243 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.066+0000 7f9e9a7fc640 1 --2- 172.21.15.1:0/3405483243 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e9c0c79c0 0x7f9e9c1dfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.066+0000 7f9e9b7fe640 1 --2- 172.21.15.1:0/3405483243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9c150750 0x7f9e9c0c3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.066+0000 7f9e9affd640 1 -- 172.21.15.1:0/3405483243 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e9c0c79c0 msgr2=0x7f9e9c1dfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.066+0000 7f9e9affd640 1 --2- 172.21.15.1:0/3405483243 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e9c0c79c0 0x7f9e9c1dfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.066+0000 7f9e9affd640 1 -- 172.21.15.1:0/3405483243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9c150750 msgr2=0x7f9e9c0c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.066+0000 7f9e9affd640 1 --2- 172.21.15.1:0/3405483243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9c150750 0x7f9e9c0c3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.066+0000 7f9e9affd640 1 -- 172.21.15.1:0/3405483243 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c14c850 con 0x7f9e9c0c3400 2024-03-06T07:28:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.066+0000 7f9e9a7fc640 1 --2- 172.21.15.1:0/3405483243 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e9c0c79c0 0x7f9e9c1dfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.066+0000 7f9e9affd640 1 --2- 172.21.15.1:0/3405483243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e9c0c3400 0x7f9e9c14caf0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400c3f0 tx=0x7f9e9400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.066+0000 7f9e937fe640 1 -- 172.21.15.1:0/3405483243 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e940052d0 con 0x7f9e9c0c3400 2024-03-06T07:28:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.066+0000 7f9e937fe640 1 -- 172.21.15.1:0/3405483243 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e94005490 con 0x7f9e9c0c3400 2024-03-06T07:28:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.066+0000 7f9ea126b640 1 -- 172.21.15.1:0/3405483243 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9c0c5120 con 0x7f9e9c0c3400 2024-03-06T07:28:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.066+0000 7f9e937fe640 1 -- 172.21.15.1:0/3405483243 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94007260 con 0x7f9e9c0c3400 2024-03-06T07:28:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.066+0000 7f9ea126b640 1 -- 172.21.15.1:0/3405483243 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e9c0c5c60 con 0x7f9e9c0c3400 2024-03-06T07:28:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.067+0000 7f9ea126b640 1 -- 172.21.15.1:0/3405483243 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e9c06ea20 con 0x7f9e9c0c3400 2024-03-06T07:28:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.068+0000 7f9e937fe640 1 -- 172.21.15.1:0/3405483243 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e94007400 con 0x7f9e9c0c3400 2024-03-06T07:28:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.068+0000 7f9e937fe640 1 --2- 172.21.15.1:0/3405483243 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e7c06b4d0 0x7f9e7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.068+0000 7f9e937fe640 1 -- 172.21.15.1:0/3405483243 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f9e940898e0 con 0x7f9e9c0c3400 2024-03-06T07:28:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.068+0000 7f9e9a7fc640 1 --2- 172.21.15.1:0/3405483243 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e7c06b4d0 0x7f9e7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.069+0000 7f9e9a7fc640 1 --2- 172.21.15.1:0/3405483243 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e7c06b4d0 0x7f9e7c06d990 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f9e84001eb0 tx=0x7f9e84009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.072+0000 7f9e937fe640 1 -- 172.21.15.1:0/3405483243 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e9c06ea20 con 0x7f9e9c0c3400 2024-03-06T07:28:24.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.257+0000 7f9ea126b640 1 -- 172.21.15.1:0/3405483243 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9e9c0686e0 con 0x7f9e9c0c3400 2024-03-06T07:28:24.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.987+0000 7f9e937fe640 1 -- 172.21.15.1:0/3405483243 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v187) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9e9405ce20 con 0x7f9e9c0c3400 2024-03-06T07:28:24.989 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:28:24.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.993+0000 7f9ea126b640 1 -- 172.21.15.1:0/3405483243 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e7c06b4d0 msgr2=0x7f9e7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.993+0000 7f9ea126b640 1 --2- 172.21.15.1:0/3405483243 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e7c06b4d0 0x7f9e7c06d990 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f9e84001eb0 tx=0x7f9e84009040 comp rx=0 tx=0).stop 2024-03-06T07:28:24.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.994+0000 7f9ea126b640 1 -- 172.21.15.1:0/3405483243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e9c0c3400 msgr2=0x7f9e9c14caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.994+0000 7f9ea126b640 1 --2- 172.21.15.1:0/3405483243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e9c0c3400 0x7f9e9c14caf0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400c3f0 tx=0x7f9e9400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.994+0000 7f9ea126b640 1 -- 172.21.15.1:0/3405483243 shutdown_connections 2024-03-06T07:28:24.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.994+0000 7f9ea126b640 1 --2- 172.21.15.1:0/3405483243 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e7c06b4d0 0x7f9e7c06d990 secure :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f9e84001eb0 tx=0x7f9e84009040 comp rx=0 tx=0).stop 2024-03-06T07:28:24.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.994+0000 7f9ea126b640 1 --2- 172.21.15.1:0/3405483243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e9c150750 0x7f9e9c0c3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.994+0000 7f9ea126b640 1 --2- 172.21.15.1:0/3405483243 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e9c0c79c0 0x7f9e9c1dfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.994+0000 7f9ea126b640 1 --2- 172.21.15.1:0/3405483243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e9c0c3400 0x7f9e9c14caf0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.994+0000 7f9ea126b640 1 -- 172.21.15.1:0/3405483243 >> 172.21.15.1:0/3405483243 conn(0x7f9e9c06b620 msgr2=0x7f9e9c1513f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.994+0000 7f9ea126b640 1 -- 172.21.15.1:0/3405483243 shutdown_connections 2024-03-06T07:28:24.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:24.995+0000 7f9ea126b640 1 -- 172.21.15.1:0/3405483243 wait complete. 2024-03-06T07:28:25.012 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:28:25.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.121+0000 7fdba6adf640 1 Processor -- start 2024-03-06T07:28:25.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.121+0000 7fdba6adf640 1 -- start start 2024-03-06T07:28:25.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.122+0000 7fdba6adf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba006c3f0 0x7fdba0158800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.122+0000 7fdba6adf640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdba0158d40 0x7fdba006f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.122+0000 7fdba6adf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba00a9790 con 0x7fdba0152c00 2024-03-06T07:28:25.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.122+0000 7fdba6adf640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba00c3990 con 0x7fdba0158d40 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.122+0000 7fdba6adf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba00a9560 con 0x7fdba006c3f0 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.122+0000 7fdba487b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdba0152c00 0x7fdba0072e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42426/0 (socket says 172.21.15.1:42426) 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.122+0000 7fdba487b640 1 -- 172.21.15.1:0/3990182184 learned_addr learned my addr 172.21.15.1:0/3990182184 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.122+0000 7fdba507c640 1 --2- 172.21.15.1:0/3990182184 >> v2:172.21.15.1:3300/0 conn(0x7fdba0158d40 0x7fdba006f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.122+0000 7fdb9ffff640 1 --2- 172.21.15.1:0/3990182184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba006c3f0 0x7fdba0158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.122+0000 7fdba507c640 1 -- 172.21.15.1:0/3990182184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba006c3f0 msgr2=0x7fdba0158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.122+0000 7fdba507c640 1 --2- 172.21.15.1:0/3990182184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba006c3f0 0x7fdba0158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.122+0000 7fdba507c640 1 -- 172.21.15.1:0/3990182184 >> v1:172.21.15.1:6789/0 conn(0x7fdba0152c00 legacy=0x7fdba0072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.122+0000 7fdba507c640 1 -- 172.21.15.1:0/3990182184 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba00a9a70 con 0x7fdba0158d40 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.122+0000 7fdb9f7fe640 1 -- 172.21.15.1:0/3990182184 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609516492 0 0) 0x7fdba00a9790 con 0x7fdba0152c00 2024-03-06T07:28:25.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.123+0000 7fdba507c640 1 --2- 172.21.15.1:0/3990182184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdba0158d40 0x7fdba006f100 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7fdb98009d90 tx=0x7fdb9802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67eeafb3aee9573d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.123+0000 7fdb9f7fe640 1 -- 172.21.15.1:0/3990182184 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb98039070 con 0x7fdba0158d40 2024-03-06T07:28:25.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.123+0000 7fdb9f7fe640 1 -- 172.21.15.1:0/3990182184 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9802cc50 con 0x7fdba0158d40 2024-03-06T07:28:25.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.123+0000 7fdb9f7fe640 1 -- 172.21.15.1:0/3990182184 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb98034440 con 0x7fdba0158d40 2024-03-06T07:28:25.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.124+0000 7fdba6adf640 1 -- 172.21.15.1:0/3990182184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdba0158d40 msgr2=0x7fdba006f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.124+0000 7fdba6adf640 1 --2- 172.21.15.1:0/3990182184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdba0158d40 0x7fdba006f100 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7fdb98009d90 tx=0x7fdb9802bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.124+0000 7fdba6adf640 1 -- 172.21.15.1:0/3990182184 shutdown_connections 2024-03-06T07:28:25.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.124+0000 7fdba6adf640 1 --2- 172.21.15.1:0/3990182184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba006c3f0 0x7fdba0158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.124+0000 7fdba6adf640 1 --2- 172.21.15.1:0/3990182184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdba0158d40 0x7fdba006f100 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.124+0000 7fdba6adf640 1 -- 172.21.15.1:0/3990182184 >> 172.21.15.1:0/3990182184 conn(0x7fdba00c76b0 msgr2=0x7fdba00c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.124+0000 7fdba6adf640 1 -- 172.21.15.1:0/3990182184 shutdown_connections 2024-03-06T07:28:25.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.124+0000 7fdba6adf640 1 -- 172.21.15.1:0/3990182184 wait complete. 2024-03-06T07:28:25.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.125+0000 7fdba6adf640 1 Processor -- start 2024-03-06T07:28:25.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.125+0000 7fdba6adf640 1 -- start start 2024-03-06T07:28:25.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba6adf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdba006c3f0 0x7fdba0157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba6adf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba0152c00 0x7fdba00ab2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba6adf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba0158d40 0x7fdba00ab810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba6adf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba006bf60 con 0x7fdba006c3f0 2024-03-06T07:28:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba6adf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba00aa030 con 0x7fdba0152c00 2024-03-06T07:28:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba6adf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba00724a0 con 0x7fdba0158d40 2024-03-06T07:28:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba487b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdba006c3f0 0x7fdba0157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba487b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdba006c3f0 0x7fdba0157bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34840/0 (socket says 172.21.15.1:34840) 2024-03-06T07:28:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba487b640 1 -- 172.21.15.1:0/2533444123 learned_addr learned my addr 172.21.15.1:0/2533444123 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba507c640 1 --2- 172.21.15.1:0/2533444123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba0158d40 0x7fdba00ab810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdb9ffff640 1 --2- 172.21.15.1:0/2533444123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba0152c00 0x7fdba00ab2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba487b640 1 -- 172.21.15.1:0/2533444123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba0158d40 msgr2=0x7fdba00ab810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba487b640 1 --2- 172.21.15.1:0/2533444123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba0158d40 0x7fdba00ab810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba487b640 1 -- 172.21.15.1:0/2533444123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba0152c00 msgr2=0x7fdba00ab2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba487b640 1 --2- 172.21.15.1:0/2533444123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba0152c00 0x7fdba00ab2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba487b640 1 -- 172.21.15.1:0/2533444123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba01eacd0 con 0x7fdba006c3f0 2024-03-06T07:28:25.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba507c640 1 --2- 172.21.15.1:0/2533444123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba0158d40 0x7fdba00ab810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:25.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba487b640 1 --2- 172.21.15.1:0/2533444123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdba006c3f0 0x7fdba0157bb0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fdb9400c4d0 tx=0x7fdb9400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdb9d7fa640 1 -- 172.21.15.1:0/2533444123 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb94018070 con 0x7fdba006c3f0 2024-03-06T07:28:25.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdb9d7fa640 1 -- 172.21.15.1:0/2533444123 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb94007500 con 0x7fdba006c3f0 2024-03-06T07:28:25.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdb9d7fa640 1 -- 172.21.15.1:0/2533444123 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb94004a20 con 0x7fdba006c3f0 2024-03-06T07:28:25.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba6adf640 1 -- 172.21.15.1:0/2533444123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba01eafb0 con 0x7fdba006c3f0 2024-03-06T07:28:25.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.126+0000 7fdba6adf640 1 -- 172.21.15.1:0/2533444123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba01eb570 con 0x7fdba006c3f0 2024-03-06T07:28:25.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.127+0000 7fdba6adf640 1 -- 172.21.15.1:0/2533444123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba00680b0 con 0x7fdba006c3f0 2024-03-06T07:28:25.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.128+0000 7fdb9d7fa640 1 -- 172.21.15.1:0/2533444123 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb9401b070 con 0x7fdba006c3f0 2024-03-06T07:28:25.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.128+0000 7fdb9d7fa640 1 --2- 172.21.15.1:0/2533444123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb8406b4d0 0x7fdb8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.128+0000 7fdb9d7fa640 1 -- 172.21.15.1:0/2533444123 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5835+0+0 (secure 0 0 0) 0x7fdb94089590 con 0x7fdba006c3f0 2024-03-06T07:28:25.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.129+0000 7fdb9ffff640 1 --2- 172.21.15.1:0/2533444123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb8406b4d0 0x7fdb8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.129+0000 7fdb9ffff640 1 --2- 172.21.15.1:0/2533444123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb8406b4d0 0x7fdb8406d990 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c006fd0 tx=0x7fdb8c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.133+0000 7fdb9d7fa640 1 -- 172.21.15.1:0/2533444123 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb94058390 con 0x7fdba006c3f0 2024-03-06T07:28:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.317+0000 7fdba6adf640 1 -- 172.21.15.1:0/2533444123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdba0073280 con 0x7fdba006c3f0 2024-03-06T07:28:25.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.989+0000 7fdb9d7fa640 1 -- 172.21.15.1:0/2533444123 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v188) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdb94057c90 con 0x7fdba006c3f0 2024-03-06T07:28:25.991 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-06T07:28:25.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.992+0000 7fdba6adf640 1 -- 172.21.15.1:0/2533444123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb8406b4d0 msgr2=0x7fdb8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.992+0000 7fdba6adf640 1 --2- 172.21.15.1:0/2533444123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb8406b4d0 0x7fdb8406d990 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c006fd0 tx=0x7fdb8c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:25.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.992+0000 7fdba6adf640 1 -- 172.21.15.1:0/2533444123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdba006c3f0 msgr2=0x7fdba0157bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.992+0000 7fdba6adf640 1 --2- 172.21.15.1:0/2533444123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdba006c3f0 0x7fdba0157bb0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fdb9400c4d0 tx=0x7fdb9400c990 comp rx=0 tx=0).stop 2024-03-06T07:28:25.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.993+0000 7fdba6adf640 1 -- 172.21.15.1:0/2533444123 shutdown_connections 2024-03-06T07:28:25.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.993+0000 7fdba6adf640 1 --2- 172.21.15.1:0/2533444123 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb8406b4d0 0x7fdb8406d990 secure :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c006fd0 tx=0x7fdb8c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:25.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.993+0000 7fdba6adf640 1 --2- 172.21.15.1:0/2533444123 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba0158d40 0x7fdba00ab810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.994+0000 7fdba6adf640 1 --2- 172.21.15.1:0/2533444123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba0152c00 0x7fdba00ab2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.994+0000 7fdba6adf640 1 --2- 172.21.15.1:0/2533444123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdba006c3f0 0x7fdba0157bb0 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.994+0000 7fdba6adf640 1 -- 172.21.15.1:0/2533444123 >> 172.21.15.1:0/2533444123 conn(0x7fdba00c76b0 msgr2=0x7fdba014b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.995+0000 7fdba6adf640 1 -- 172.21.15.1:0/2533444123 shutdown_connections 2024-03-06T07:28:25.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:25.995+0000 7fdba6adf640 1 -- 172.21.15.1:0/2533444123 wait complete. 2024-03-06T07:28:26.013 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:28:26.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.118+0000 7f3d00d9a640 1 Processor -- start 2024-03-06T07:28:26.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.118+0000 7f3d00d9a640 1 -- start start 2024-03-06T07:28:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.119+0000 7f3d00d9a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3cfc1543a0 0x7f3cfc15e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.119+0000 7f3d00d9a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cfc150900 con 0x7f3cfc150520 2024-03-06T07:28:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.119+0000 7f3d00d9a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cfc071790 con 0x7f3cfc0c32f0 2024-03-06T07:28:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.119+0000 7f3d00d9a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cfc153ed0 con 0x7f3cfc1543a0 2024-03-06T07:28:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.119+0000 7f3cfa59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3cfc150520 0x7f3cfc1537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42438/0 (socket says 172.21.15.1:42438) 2024-03-06T07:28:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.119+0000 7f3cfa59c640 1 -- 172.21.15.1:0/1066951136 learned_addr learned my addr 172.21.15.1:0/1066951136 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf9d9b640 1 -- 172.21.15.1:0/1066951136 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124222644 0 0) 0x7f3cfc150900 con 0x7f3cfc150520 2024-03-06T07:28:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf9d9b640 1 -- 172.21.15.1:0/1066951136 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ce8003520 con 0x7f3cfc150520 2024-03-06T07:28:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cfad9d640 1 --2- 172.21.15.1:0/1066951136 >> v2:172.21.15.98:3300/0 conn(0x7f3cfc1543a0 0x7f3cfc15e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf9d9b640 1 -- 172.21.15.1:0/1066951136 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3006577649 0 0) 0x7f3ce8003520 con 0x7f3cfc150520 2024-03-06T07:28:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf9d9b640 1 -- 172.21.15.1:0/1066951136 >> v2:172.21.15.98:3300/0 conn(0x7f3cfc1543a0 msgr2=0x7f3cfc15e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf9d9b640 1 --2- 172.21.15.1:0/1066951136 >> v2:172.21.15.98:3300/0 conn(0x7f3cfc1543a0 0x7f3cfc15e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf9d9b640 1 -- 172.21.15.1:0/1066951136 >> v1:172.21.15.98:6789/0 conn(0x7f3cfc0c32f0 legacy=0x7f3cfc070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf9d9b640 1 -- 172.21.15.1:0/1066951136 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc157050 con 0x7f3cfc150520 2024-03-06T07:28:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf9d9b640 1 -- 172.21.15.1:0/1066951136 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f3cec001670 con 0x7f3cfc150520 2024-03-06T07:28:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf9d9b640 1 -- 172.21.15.1:0/1066951136 >> v1:172.21.15.1:6789/0 conn(0x7f3cfc150520 legacy=0x7f3cfc1537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf9d9b640 1 --2- 172.21.15.1:0/1066951136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ce80039b0 0x7f3ce8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf9d9b640 1 --2- 172.21.15.1:0/1066951136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ce8004800 0x7f3ce8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf9d9b640 1 --2- 172.21.15.1:0/1066951136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce80051c0 0x7f3ce800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf9d9b640 1 -- 172.21.15.1:0/1066951136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc153ed0 con 0x7f3ce8004800 2024-03-06T07:28:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf9d9b640 1 -- 172.21.15.1:0/1066951136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce800c060 con 0x7f3ce80051c0 2024-03-06T07:28:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf9d9b640 1 -- 172.21.15.1:0/1066951136 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce800c260 con 0x7f3ce80039b0 2024-03-06T07:28:26.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf3fff640 1 --2- 172.21.15.1:0/1066951136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ce8004800 0x7f3ce8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cfa59c640 1 --2- 172.21.15.1:0/1066951136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ce80039b0 0x7f3ce8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cfad9d640 1 --2- 172.21.15.1:0/1066951136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce80051c0 0x7f3ce800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf3fff640 1 -- 172.21.15.1:0/1066951136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ce80039b0 msgr2=0x7f3ce8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf3fff640 1 --2- 172.21.15.1:0/1066951136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ce80039b0 0x7f3ce8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf3fff640 1 -- 172.21.15.1:0/1066951136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce80051c0 msgr2=0x7f3ce800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf3fff640 1 --2- 172.21.15.1:0/1066951136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce80051c0 0x7f3ce800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cf3fff640 1 -- 172.21.15.1:0/1066951136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce8003520 con 0x7f3ce8004800 2024-03-06T07:28:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cfa59c640 1 --2- 172.21.15.1:0/1066951136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ce80039b0 0x7f3ce8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cfad9d640 1 --2- 172.21.15.1:0/1066951136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce80051c0 0x7f3ce800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.120+0000 7f3cfad9d640 1 -- 172.21.15.1:0/1066951136 reap_dead start 2024-03-06T07:28:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.121+0000 7f3cf3fff640 1 --2- 172.21.15.1:0/1066951136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ce8004800 0x7f3ce8004c80 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4001750 tx=0x7f3ce8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7e491859ea6531 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.121+0000 7f3cf9d9b640 1 -- 172.21.15.1:0/1066951136 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4001370 con 0x7f3ce8004800 2024-03-06T07:28:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.121+0000 7f3cf9d9b640 1 -- 172.21.15.1:0/1066951136 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce40052d0 con 0x7f3ce8004800 2024-03-06T07:28:26.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.121+0000 7f3cf9d9b640 1 -- 172.21.15.1:0/1066951136 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4003830 con 0x7f3ce8004800 2024-03-06T07:28:26.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.122+0000 7f3d00d9a640 1 -- 172.21.15.1:0/1066951136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ce8004800 msgr2=0x7f3ce8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.122+0000 7f3d00d9a640 1 --2- 172.21.15.1:0/1066951136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ce8004800 0x7f3ce8004c80 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4001750 tx=0x7f3ce8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:26.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.122+0000 7f3d00d9a640 1 -- 172.21.15.1:0/1066951136 shutdown_connections 2024-03-06T07:28:26.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.122+0000 7f3d00d9a640 1 --2- 172.21.15.1:0/1066951136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ce8004800 0x7f3ce8004c80 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.122+0000 7f3d00d9a640 1 -- 172.21.15.1:0/1066951136 >> 172.21.15.1:0/1066951136 conn(0x7f3cfc0c88f0 msgr2=0x7f3cfc0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.122+0000 7f3d00d9a640 1 -- 172.21.15.1:0/1066951136 shutdown_connections 2024-03-06T07:28:26.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.122+0000 7f3d00d9a640 1 -- 172.21.15.1:0/1066951136 wait complete. 2024-03-06T07:28:26.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.123+0000 7f3d00d9a640 1 Processor -- start 2024-03-06T07:28:26.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.123+0000 7f3d00d9a640 1 -- start start 2024-03-06T07:28:26.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.124+0000 7f3d00d9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce8004800 0x7f3cfc154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.124+0000 7f3d00d9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cfc1eea60 0x7f3cfc0ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.124+0000 7f3d00d9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc145ee0 0x7f3cfc146340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.124+0000 7f3d00d9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc1532a0 con 0x7f3cfc145ee0 2024-03-06T07:28:26.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.124+0000 7f3d00d9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc1547e0 con 0x7f3ce8004800 2024-03-06T07:28:26.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.124+0000 7f3d00d9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc157330 con 0x7f3cfc1eea60 2024-03-06T07:28:26.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.124+0000 7f3cfad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc145ee0 0x7f3cfc146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.124+0000 7f3cfad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc145ee0 0x7f3cfc146340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34848/0 (socket says 172.21.15.1:34848) 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.124+0000 7f3cfad9d640 1 -- 172.21.15.1:0/2422428977 learned_addr learned my addr 172.21.15.1:0/2422428977 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.124+0000 7f3cf3fff640 1 --2- 172.21.15.1:0/2422428977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cfc1eea60 0x7f3cfc0ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.124+0000 7f3cfa59c640 1 --2- 172.21.15.1:0/2422428977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce8004800 0x7f3cfc154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.124+0000 7f3cfad9d640 1 -- 172.21.15.1:0/2422428977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cfc1eea60 msgr2=0x7f3cfc0ac9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.124+0000 7f3cfad9d640 1 --2- 172.21.15.1:0/2422428977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cfc1eea60 0x7f3cfc0ac9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.124+0000 7f3cfad9d640 1 -- 172.21.15.1:0/2422428977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce8004800 msgr2=0x7f3cfc154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.124+0000 7f3cfad9d640 1 --2- 172.21.15.1:0/2422428977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce8004800 0x7f3cfc154b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.124+0000 7f3cfad9d640 1 -- 172.21.15.1:0/2422428977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc1554a0 con 0x7f3cfc145ee0 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.124+0000 7f3cfad9d640 1 --2- 172.21.15.1:0/2422428977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc145ee0 0x7f3cfc146340 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4007cf0 tx=0x7f3cf4010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.125+0000 7f3cf2ffd640 1 -- 172.21.15.1:0/2422428977 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf401a030 con 0x7f3cfc145ee0 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.125+0000 7f3cf2ffd640 1 -- 172.21.15.1:0/2422428977 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf4010e30 con 0x7f3cfc145ee0 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.125+0000 7f3d00d9a640 1 -- 172.21.15.1:0/2422428977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cfc155780 con 0x7f3cfc145ee0 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.125+0000 7f3cf2ffd640 1 -- 172.21.15.1:0/2422428977 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf4003bd0 con 0x7f3cfc145ee0 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.125+0000 7f3d00d9a640 1 -- 172.21.15.1:0/2422428977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cfc06c5a0 con 0x7f3cfc145ee0 2024-03-06T07:28:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.126+0000 7f3d00d9a640 1 -- 172.21.15.1:0/2422428977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cfc0686e0 con 0x7f3cfc145ee0 2024-03-06T07:28:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.126+0000 7f3cf2ffd640 1 -- 172.21.15.1:0/2422428977 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3cf401d070 con 0x7f3cfc145ee0 2024-03-06T07:28:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.126+0000 7f3cf2ffd640 1 --2- 172.21.15.1:0/2422428977 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cdc06b4d0 0x7f3cdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.126+0000 7f3cf2ffd640 1 -- 172.21.15.1:0/2422428977 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f3cf408abb0 con 0x7f3cfc145ee0 2024-03-06T07:28:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.127+0000 7f3cfa59c640 1 --2- 172.21.15.1:0/2422428977 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cdc06b4d0 0x7f3cdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.127+0000 7f3cfa59c640 1 --2- 172.21.15.1:0/2422428977 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cdc06b4d0 0x7f3cdc06d990 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f3cec001cc0 tx=0x7f3cec009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.131+0000 7f3cf2ffd640 1 -- 172.21.15.1:0/2422428977 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3cf4059be0 con 0x7f3cfc145ee0 2024-03-06T07:28:26.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.313+0000 7f3d00d9a640 1 -- 172.21.15.1:0/2422428977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3cfc151630 con 0x7f3cfc145ee0 2024-03-06T07:28:26.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.317+0000 7f3cf2ffd640 1 -- 172.21.15.1:0/2422428977 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v60) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3cf40594e0 con 0x7f3cfc145ee0 2024-03-06T07:28:26.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.319+0000 7f3d00d9a640 1 -- 172.21.15.1:0/2422428977 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cdc06b4d0 msgr2=0x7f3cdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.319+0000 7f3d00d9a640 1 --2- 172.21.15.1:0/2422428977 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cdc06b4d0 0x7f3cdc06d990 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f3cec001cc0 tx=0x7f3cec009040 comp rx=0 tx=0).stop 2024-03-06T07:28:26.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.319+0000 7f3d00d9a640 1 -- 172.21.15.1:0/2422428977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc145ee0 msgr2=0x7f3cfc146340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.319+0000 7f3d00d9a640 1 --2- 172.21.15.1:0/2422428977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc145ee0 0x7f3cfc146340 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4007cf0 tx=0x7f3cf4010590 comp rx=0 tx=0).stop 2024-03-06T07:28:26.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.319+0000 7f3d00d9a640 1 -- 172.21.15.1:0/2422428977 shutdown_connections 2024-03-06T07:28:26.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.319+0000 7f3d00d9a640 1 --2- 172.21.15.1:0/2422428977 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3cdc06b4d0 0x7f3cdc06d990 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.319+0000 7f3d00d9a640 1 --2- 172.21.15.1:0/2422428977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc145ee0 0x7f3cfc146340 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.319+0000 7f3d00d9a640 1 --2- 172.21.15.1:0/2422428977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cfc1eea60 0x7f3cfc0ac9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.319+0000 7f3d00d9a640 1 --2- 172.21.15.1:0/2422428977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce8004800 0x7f3cfc154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.319+0000 7f3d00d9a640 1 -- 172.21.15.1:0/2422428977 >> 172.21.15.1:0/2422428977 conn(0x7f3cfc0c88f0 msgr2=0x7f3cfc0c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.319+0000 7f3d00d9a640 1 -- 172.21.15.1:0/2422428977 shutdown_connections 2024-03-06T07:28:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.319+0000 7f3d00d9a640 1 -- 172.21.15.1:0/2422428977 wait complete. 2024-03-06T07:28:26.332 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_grow' 2024-03-06T07:28:26.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.430+0000 7f872dedf640 1 Processor -- start 2024-03-06T07:28:26.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.430+0000 7f872dedf640 1 -- start start 2024-03-06T07:28:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.431+0000 7f872dedf640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87280afc40 0x7f8728160b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.431+0000 7f872dedf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87280c7180 con 0x7f87280af6f0 2024-03-06T07:28:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.431+0000 7f872dedf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8728155460 con 0x7f8728152820 2024-03-06T07:28:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.431+0000 7f872dedf640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87280c6bc0 con 0x7f87280afc40 2024-03-06T07:28:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.431+0000 7f8727fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87280afc40 0x7f8728160b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.431+0000 7f8726ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f87280af6f0 0x7f87280c73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42452/0 (socket says 172.21.15.1:42452) 2024-03-06T07:28:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.431+0000 7f8726ffd640 1 -- 172.21.15.1:0/9159892 learned_addr learned my addr 172.21.15.1:0/9159892 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.431+0000 7f8727fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87280afc40 0x7f8728160b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34862/0 (socket says 172.21.15.1:34862) 2024-03-06T07:28:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.432+0000 7f8727fff640 1 -- 172.21.15.1:0/9159892 >> v1:172.21.15.98:6789/0 conn(0x7f8728152820 legacy=0x7f87280c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.432+0000 7f8727fff640 1 -- 172.21.15.1:0/9159892 >> v1:172.21.15.1:6789/0 conn(0x7f87280af6f0 legacy=0x7f87280c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.432+0000 7f87267fc640 1 -- 172.21.15.1:0/9159892 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329293769 0 0) 0x7f87280c7180 con 0x7f87280af6f0 2024-03-06T07:28:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.432+0000 7f8727fff640 1 -- 172.21.15.1:0/9159892 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87280c6ea0 con 0x7f87280afc40 2024-03-06T07:28:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.432+0000 7f8727fff640 1 --2- 172.21.15.1:0/9159892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87280afc40 0x7f8728160b70 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f8720009d90 tx=0x7f872002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aec9bb85c14ca1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.433+0000 7f87267fc640 1 -- 172.21.15.1:0/9159892 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8720039070 con 0x7f87280afc40 2024-03-06T07:28:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.433+0000 7f87267fc640 1 -- 172.21.15.1:0/9159892 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872002cc60 con 0x7f87280afc40 2024-03-06T07:28:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.433+0000 7f87267fc640 1 -- 172.21.15.1:0/9159892 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8720034660 con 0x7f87280afc40 2024-03-06T07:28:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.433+0000 7f872dedf640 1 -- 172.21.15.1:0/9159892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87280afc40 msgr2=0x7f8728160b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.433+0000 7f872dedf640 1 --2- 172.21.15.1:0/9159892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87280afc40 0x7f8728160b70 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f8720009d90 tx=0x7f872002beb0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.433+0000 7f872dedf640 1 -- 172.21.15.1:0/9159892 shutdown_connections 2024-03-06T07:28:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.433+0000 7f872dedf640 1 --2- 172.21.15.1:0/9159892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87280afc40 0x7f8728160b70 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.433+0000 7f872dedf640 1 -- 172.21.15.1:0/9159892 >> 172.21.15.1:0/9159892 conn(0x7f8728072720 msgr2=0x7f8728072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.433+0000 7f872dedf640 1 -- 172.21.15.1:0/9159892 shutdown_connections 2024-03-06T07:28:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.433+0000 7f872dedf640 1 -- 172.21.15.1:0/9159892 wait complete. 2024-03-06T07:28:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.434+0000 7f872dedf640 1 Processor -- start 2024-03-06T07:28:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.434+0000 7f872dedf640 1 -- start start 2024-03-06T07:28:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.435+0000 7f872dedf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87280af6f0 0x7f87280c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.435+0000 7f872dedf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87280afc40 0x7f8728157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.435+0000 7f872dedf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8728152820 0x7f8728157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.435+0000 7f872dedf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87280c7b00 con 0x7f87280afc40 2024-03-06T07:28:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.435+0000 7f872dedf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87280c0840 con 0x7f8728152820 2024-03-06T07:28:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.435+0000 7f872dedf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87280c7c80 con 0x7f87280af6f0 2024-03-06T07:28:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.435+0000 7f8726ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87280afc40 0x7f8728157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.435+0000 7f8726ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87280afc40 0x7f8728157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34876/0 (socket says 172.21.15.1:34876) 2024-03-06T07:28:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.435+0000 7f8726ffd640 1 -- 172.21.15.1:0/3510536765 learned_addr learned my addr 172.21.15.1:0/3510536765 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.435+0000 7f8726ffd640 1 -- 172.21.15.1:0/3510536765 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87280af6f0 msgr2=0x7f87280c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.435+0000 7f8726ffd640 1 --2- 172.21.15.1:0/3510536765 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87280af6f0 0x7f87280c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.435+0000 7f8726ffd640 1 -- 172.21.15.1:0/3510536765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8728152820 msgr2=0x7f8728157a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.435+0000 7f8727fff640 1 --2- 172.21.15.1:0/3510536765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8728152820 0x7f8728157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.435+0000 7f8726ffd640 1 --2- 172.21.15.1:0/3510536765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8728152820 0x7f8728157a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.435+0000 7f8726ffd640 1 -- 172.21.15.1:0/3510536765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87280c3ea0 con 0x7f87280afc40 2024-03-06T07:28:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.436+0000 7f8726ffd640 1 --2- 172.21.15.1:0/3510536765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87280afc40 0x7f8728157520 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f871c00c4d0 tx=0x7f871c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.436+0000 7f870ffff640 1 -- 172.21.15.1:0/3510536765 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c018070 con 0x7f87280afc40 2024-03-06T07:28:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.436+0000 7f870ffff640 1 -- 172.21.15.1:0/3510536765 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871c007500 con 0x7f87280afc40 2024-03-06T07:28:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.436+0000 7f872dedf640 1 -- 172.21.15.1:0/3510536765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87281eb380 con 0x7f87280afc40 2024-03-06T07:28:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.436+0000 7f870ffff640 1 -- 172.21.15.1:0/3510536765 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c004a20 con 0x7f87280afc40 2024-03-06T07:28:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.436+0000 7f872dedf640 1 -- 172.21.15.1:0/3510536765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87280c9fe0 con 0x7f87280afc40 2024-03-06T07:28:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.437+0000 7f872dedf640 1 -- 172.21.15.1:0/3510536765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87280686e0 con 0x7f87280afc40 2024-03-06T07:28:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.437+0000 7f870ffff640 1 -- 172.21.15.1:0/3510536765 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f871c01b070 con 0x7f87280afc40 2024-03-06T07:28:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.438+0000 7f870ffff640 1 --2- 172.21.15.1:0/3510536765 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f871406b4d0 0x7f871406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.438+0000 7f870ffff640 1 -- 172.21.15.1:0/3510536765 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f871c089490 con 0x7f87280afc40 2024-03-06T07:28:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.438+0000 7f87277fe640 1 --2- 172.21.15.1:0/3510536765 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f871406b4d0 0x7f871406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.438+0000 7f87277fe640 1 --2- 172.21.15.1:0/3510536765 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f871406b4d0 0x7f871406d990 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8718006fd0 tx=0x7f8718006bc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.442+0000 7f870ffff640 1 -- 172.21.15.1:0/3510536765 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f871c058390 con 0x7f87280afc40 2024-03-06T07:28:26.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.697+0000 7f872dedf640 1 -- 172.21.15.1:0/3510536765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]} v 0) v1 -- 0x7f87280c5b40 con 0x7f87280afc40 2024-03-06T07:28:26.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.995+0000 7f870ffff640 1 -- 172.21.15.1:0/3510536765 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]}]=0 v467) v1 ==== 138+0+0 (secure 0 0 0) 0x7f871c057c90 con 0x7f87280afc40 2024-03-06T07:28:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.997+0000 7f872dedf640 1 -- 172.21.15.1:0/3510536765 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f871406b4d0 msgr2=0x7f871406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.997+0000 7f872dedf640 1 --2- 172.21.15.1:0/3510536765 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f871406b4d0 0x7f871406d990 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8718006fd0 tx=0x7f8718006bc0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.997+0000 7f872dedf640 1 -- 172.21.15.1:0/3510536765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87280afc40 msgr2=0x7f8728157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.997+0000 7f872dedf640 1 --2- 172.21.15.1:0/3510536765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87280afc40 0x7f8728157520 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f871c00c4d0 tx=0x7f871c00c990 comp rx=0 tx=0).stop 2024-03-06T07:28:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.997+0000 7f872dedf640 1 -- 172.21.15.1:0/3510536765 shutdown_connections 2024-03-06T07:28:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.997+0000 7f872dedf640 1 --2- 172.21.15.1:0/3510536765 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f871406b4d0 0x7f871406d990 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.997+0000 7f872dedf640 1 --2- 172.21.15.1:0/3510536765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8728152820 0x7f8728157a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.997+0000 7f872dedf640 1 --2- 172.21.15.1:0/3510536765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87280afc40 0x7f8728157520 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.997+0000 7f872dedf640 1 --2- 172.21.15.1:0/3510536765 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87280af6f0 0x7f87280c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.997+0000 7f872dedf640 1 -- 172.21.15.1:0/3510536765 >> 172.21.15.1:0/3510536765 conn(0x7f8728072720 msgr2=0x7f872806cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.997+0000 7f872dedf640 1 -- 172.21.15.1:0/3510536765 shutdown_connections 2024-03-06T07:28:27.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:26.997+0000 7f872dedf640 1 -- 172.21.15.1:0/3510536765 wait complete. 2024-03-06T07:28:27.010 INFO:tasks.cephfs_test_runner:That setting max_mds undoes down. ... ok 2024-03-06T07:28:27.011 INFO:tasks.cephfs_test_runner:Starting test: test_down_health (tasks.cephfs.test_failover.TestClusterResize) That marking a FS down does not generate a health warning 2024-03-06T07:28:27.011 INFO:tasks.cephfs_test_runner:test_down_health (tasks.cephfs.test_failover.TestClusterResize) 2024-03-06T07:28:27.011 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:27.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.111+0000 7f9a6f73a640 1 Processor -- start 2024-03-06T07:28:27.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.111+0000 7f9a6f73a640 1 -- start start 2024-03-06T07:28:27.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.111+0000 7f9a6f73a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9a68152890 0x7f9a68159030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.111+0000 7f9a6f73a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a680c7560 0x7f9a68155860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.111+0000 7f9a6f73a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a68155080 con 0x7f9a68155da0 2024-03-06T07:28:27.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.111+0000 7f9a6f73a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a680c35b0 con 0x7f9a68152890 2024-03-06T07:28:27.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.111+0000 7f9a6f73a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a680c3890 con 0x7f9a680c7560 2024-03-06T07:28:27.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.112+0000 7f9a6d4d6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9a68152890 0x7f9a68159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.112+0000 7f9a6d4d6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9a68152890 0x7f9a68159030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34892/0 (socket says 172.21.15.1:34892) 2024-03-06T07:28:27.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.112+0000 7f9a6d4d6640 1 -- 172.21.15.1:0/2440147728 learned_addr learned my addr 172.21.15.1:0/2440147728 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:27.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.112+0000 7f9a6ccd5640 1 --2- 172.21.15.1:0/2440147728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a680c7560 0x7f9a68155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.112+0000 7f9a6d4d6640 1 -- 172.21.15.1:0/2440147728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a680c7560 msgr2=0x7f9a68155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.112+0000 7f9a6d4d6640 1 --2- 172.21.15.1:0/2440147728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a680c7560 0x7f9a68155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.112+0000 7f9a6d4d6640 1 -- 172.21.15.1:0/2440147728 >> v1:172.21.15.98:6789/0 conn(0x7f9a68155da0 legacy=0x7f9a680c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.112+0000 7f9a6d4d6640 1 -- 172.21.15.1:0/2440147728 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a680c3b70 con 0x7f9a68152890 2024-03-06T07:28:27.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.113+0000 7f9a6d4d6640 1 --2- 172.21.15.1:0/2440147728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a68152890 0x7f9a68159030 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f9a64009870 tx=0x7f9a6402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d42cf46a5619f31 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.113+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2440147728 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a640317c0 con 0x7f9a68152890 2024-03-06T07:28:27.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.113+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2440147728 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a64037070 con 0x7f9a68152890 2024-03-06T07:28:27.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.113+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2440147728 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64031b80 con 0x7f9a68152890 2024-03-06T07:28:27.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.113+0000 7f9a6f73a640 1 -- 172.21.15.1:0/2440147728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a68152890 msgr2=0x7f9a68159030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.113+0000 7f9a6f73a640 1 --2- 172.21.15.1:0/2440147728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a68152890 0x7f9a68159030 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f9a64009870 tx=0x7f9a6402b860 comp rx=0 tx=0).stop 2024-03-06T07:28:27.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.113+0000 7f9a6f73a640 1 -- 172.21.15.1:0/2440147728 shutdown_connections 2024-03-06T07:28:27.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.113+0000 7f9a6f73a640 1 --2- 172.21.15.1:0/2440147728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a680c7560 0x7f9a68155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.113+0000 7f9a6f73a640 1 --2- 172.21.15.1:0/2440147728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a68152890 0x7f9a68159030 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.113+0000 7f9a6f73a640 1 -- 172.21.15.1:0/2440147728 >> 172.21.15.1:0/2440147728 conn(0x7f9a6806b3e0 msgr2=0x7f9a6806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.114+0000 7f9a6f73a640 1 -- 172.21.15.1:0/2440147728 shutdown_connections 2024-03-06T07:28:27.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.114+0000 7f9a6f73a640 1 -- 172.21.15.1:0/2440147728 wait complete. 2024-03-06T07:28:27.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.115+0000 7f9a6f73a640 1 Processor -- start 2024-03-06T07:28:27.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.115+0000 7f9a6f73a640 1 -- start start 2024-03-06T07:28:27.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.115+0000 7f9a6f73a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a680c7560 0x7f9a681519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.115+0000 7f9a6f73a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a68152890 0x7f9a680abbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.115+0000 7f9a6f73a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a68158790 0x7f9a680aa860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.115+0000 7f9a6f73a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a680c3260 con 0x7f9a68152890 2024-03-06T07:28:27.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.115+0000 7f9a6f73a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a68159570 con 0x7f9a680c7560 2024-03-06T07:28:27.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.115+0000 7f9a6f73a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a680c9400 con 0x7f9a68158790 2024-03-06T07:28:27.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.115+0000 7f9a6ccd5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a68152890 0x7f9a680abbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.115+0000 7f9a6ccd5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a68152890 0x7f9a680abbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34906/0 (socket says 172.21.15.1:34906) 2024-03-06T07:28:27.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.115+0000 7f9a6ccd5640 1 -- 172.21.15.1:0/3667274273 learned_addr learned my addr 172.21.15.1:0/3667274273 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:27.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.116+0000 7f9a6dcd7640 1 --2- 172.21.15.1:0/3667274273 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a68158790 0x7f9a680aa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.116+0000 7f9a6d4d6640 1 --2- 172.21.15.1:0/3667274273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a680c7560 0x7f9a681519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.116+0000 7f9a6ccd5640 1 -- 172.21.15.1:0/3667274273 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a68158790 msgr2=0x7f9a680aa860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.116+0000 7f9a6ccd5640 1 --2- 172.21.15.1:0/3667274273 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a68158790 0x7f9a680aa860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.116+0000 7f9a6ccd5640 1 -- 172.21.15.1:0/3667274273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a680c7560 msgr2=0x7f9a681519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.116+0000 7f9a6ccd5640 1 --2- 172.21.15.1:0/3667274273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a680c7560 0x7f9a681519c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.116+0000 7f9a6ccd5640 1 -- 172.21.15.1:0/3667274273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a681516e0 con 0x7f9a68152890 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.116+0000 7f9a6dcd7640 1 --2- 172.21.15.1:0/3667274273 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a68158790 0x7f9a680aa860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.116+0000 7f9a6ccd5640 1 --2- 172.21.15.1:0/3667274273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a68152890 0x7f9a680abbd0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800d600 tx=0x7f9a5800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.116+0000 7f9a5dffb640 1 -- 172.21.15.1:0/3667274273 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a580125f0 con 0x7f9a68152890 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.116+0000 7f9a5dffb640 1 -- 172.21.15.1:0/3667274273 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a58019070 con 0x7f9a68152890 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.116+0000 7f9a5dffb640 1 -- 172.21.15.1:0/3667274273 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a580129f0 con 0x7f9a68152890 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.116+0000 7f9a6f73a640 1 -- 172.21.15.1:0/3667274273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a680ad0d0 con 0x7f9a68152890 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.116+0000 7f9a6f73a640 1 -- 172.21.15.1:0/3667274273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a680a9430 con 0x7f9a68152890 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.117+0000 7f9a6f73a640 1 -- 172.21.15.1:0/3667274273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a68067fd0 con 0x7f9a68152890 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.118+0000 7f9a5dffb640 1 -- 172.21.15.1:0/3667274273 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a58012b90 con 0x7f9a68152890 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.118+0000 7f9a5dffb640 1 --2- 172.21.15.1:0/3667274273 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a4c06b4d0 0x7f9a4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.118+0000 7f9a5dffb640 1 -- 172.21.15.1:0/3667274273 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f9a5808a550 con 0x7f9a68152890 2024-03-06T07:28:27.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.118+0000 7f9a6d4d6640 1 --2- 172.21.15.1:0/3667274273 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a4c06b4d0 0x7f9a4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.119+0000 7f9a6d4d6640 1 --2- 172.21.15.1:0/3667274273 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a4c06b4d0 0x7f9a4c06d990 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f9a64036000 tx=0x7f9a6403d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.122+0000 7f9a5dffb640 1 -- 172.21.15.1:0/3667274273 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a580594f0 con 0x7f9a68152890 2024-03-06T07:28:27.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.306+0000 7f9a6f73a640 1 -- 172.21.15.1:0/3667274273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a680c2d80 con 0x7f9a68152890 2024-03-06T07:28:27.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.307+0000 7f9a5dffb640 1 -- 172.21.15.1:0/3667274273 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+13309 (secure 0 0 0) 0x7f9a58058df0 con 0x7f9a68152890 2024-03-06T07:28:27.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:27.309 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":188,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:28:25.988469+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6835/1525330395":"2024-03-07T07:28:22.889407+0000","172.21.15.1:6834/1525330395":"2024-03-07T07:28:22.889407+0000","172.21.15.1:6833/3895143314":"2024-03-07T07:28:22.889395+0000","172.21.15.1:6832/3895143314":"2024-03-07T07:28:22.889395+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:27.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.309+0000 7f9a6f73a640 1 -- 172.21.15.1:0/3667274273 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a4c06b4d0 msgr2=0x7f9a4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.309+0000 7f9a6f73a640 1 --2- 172.21.15.1:0/3667274273 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a4c06b4d0 0x7f9a4c06d990 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f9a64036000 tx=0x7f9a6403d040 comp rx=0 tx=0).stop 2024-03-06T07:28:27.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.309+0000 7f9a6f73a640 1 -- 172.21.15.1:0/3667274273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a68152890 msgr2=0x7f9a680abbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.309+0000 7f9a6f73a640 1 --2- 172.21.15.1:0/3667274273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a68152890 0x7f9a680abbd0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800d600 tx=0x7f9a5800dac0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.309+0000 7f9a6f73a640 1 -- 172.21.15.1:0/3667274273 shutdown_connections 2024-03-06T07:28:27.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.309+0000 7f9a6f73a640 1 --2- 172.21.15.1:0/3667274273 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a4c06b4d0 0x7f9a4c06d990 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.309+0000 7f9a6f73a640 1 --2- 172.21.15.1:0/3667274273 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a68158790 0x7f9a680aa860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.309+0000 7f9a6f73a640 1 --2- 172.21.15.1:0/3667274273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a68152890 0x7f9a680abbd0 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.309+0000 7f9a6f73a640 1 --2- 172.21.15.1:0/3667274273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a680c7560 0x7f9a681519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.309+0000 7f9a6f73a640 1 -- 172.21.15.1:0/3667274273 >> 172.21.15.1:0/3667274273 conn(0x7f9a6806b3e0 msgr2=0x7f9a681de6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.309+0000 7f9a6f73a640 1 -- 172.21.15.1:0/3667274273 shutdown_connections 2024-03-06T07:28:27.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.309+0000 7f9a6f73a640 1 -- 172.21.15.1:0/3667274273 wait complete. 2024-03-06T07:28:27.322 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:27.322 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:27.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.424+0000 7f775785c640 1 Processor -- start 2024-03-06T07:28:27.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.424+0000 7f775785c640 1 -- start start 2024-03-06T07:28:27.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.425+0000 7f775785c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7750152a30 0x7f77500c7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.425+0000 7f775785c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77500afcb0 0x7f775006d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.425+0000 7f775785c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7750155260 con 0x7f77500c7940 2024-03-06T07:28:27.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.425+0000 7f775785c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77500c6fb0 con 0x7f7750152a30 2024-03-06T07:28:27.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.425+0000 7f775785c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7750155520 con 0x7f77500afcb0 2024-03-06T07:28:27.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.425+0000 7f77555f8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7750152a30 0x7f77500c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.425+0000 7f7754df7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77500afcb0 0x7f775006d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.425+0000 7f7754df7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77500afcb0 0x7f775006d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:56684/0 (socket says 172.21.15.1:56684) 2024-03-06T07:28:27.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.425+0000 7f77555f8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7750152a30 0x7f77500c7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:46154/0 (socket says 172.21.15.1:46154) 2024-03-06T07:28:27.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.425+0000 7f7754df7640 1 -- 172.21.15.1:0/3614608153 learned_addr learned my addr 172.21.15.1:0/3614608153 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:27.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.426+0000 7f77555f8640 1 -- 172.21.15.1:0/3614608153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77500afcb0 msgr2=0x7f775006d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.426+0000 7f77555f8640 1 --2- 172.21.15.1:0/3614608153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77500afcb0 0x7f775006d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.426+0000 7f77555f8640 1 -- 172.21.15.1:0/3614608153 >> v1:172.21.15.98:6789/0 conn(0x7f77500c7940 legacy=0x7f77501587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.426+0000 7f77555f8640 1 -- 172.21.15.1:0/3614608153 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7750155800 con 0x7f7750152a30 2024-03-06T07:28:27.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.426+0000 7f7754df7640 1 --2- 172.21.15.1:0/3614608153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77500afcb0 0x7f775006d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:27.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.427+0000 7f77555f8640 1 --2- 172.21.15.1:0/3614608153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7750152a30 0x7f77500c7400 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f77480097a0 tx=0x7f774802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31750aacb04c1164 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.427+0000 7f774ffff640 1 -- 172.21.15.1:0/3614608153 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77480315a0 con 0x7f7750152a30 2024-03-06T07:28:27.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.427+0000 7f774ffff640 1 -- 172.21.15.1:0/3614608153 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7748037070 con 0x7f7750152a30 2024-03-06T07:28:27.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.427+0000 7f774ffff640 1 -- 172.21.15.1:0/3614608153 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7748031960 con 0x7f7750152a30 2024-03-06T07:28:27.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.427+0000 7f775785c640 1 -- 172.21.15.1:0/3614608153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7750152a30 msgr2=0x7f77500c7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.427+0000 7f775785c640 1 --2- 172.21.15.1:0/3614608153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7750152a30 0x7f77500c7400 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f77480097a0 tx=0x7f774802b840 comp rx=0 tx=0).stop 2024-03-06T07:28:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.427+0000 7f775785c640 1 -- 172.21.15.1:0/3614608153 shutdown_connections 2024-03-06T07:28:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.427+0000 7f775785c640 1 --2- 172.21.15.1:0/3614608153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77500afcb0 0x7f775006d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.427+0000 7f775785c640 1 --2- 172.21.15.1:0/3614608153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7750152a30 0x7f77500c7400 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.427+0000 7f775785c640 1 -- 172.21.15.1:0/3614608153 >> 172.21.15.1:0/3614608153 conn(0x7f775008b8e0 msgr2=0x7f7750066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.427+0000 7f775785c640 1 -- 172.21.15.1:0/3614608153 shutdown_connections 2024-03-06T07:28:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.427+0000 7f775785c640 1 -- 172.21.15.1:0/3614608153 wait complete. 2024-03-06T07:28:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.428+0000 7f775785c640 1 Processor -- start 2024-03-06T07:28:27.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.428+0000 7f775785c640 1 -- start start 2024-03-06T07:28:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.429+0000 7f775785c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77500afcb0 0x7f77500c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.429+0000 7f775785c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77500c7940 0x7f775014a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.429+0000 7f775785c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7750152a30 0x7f775014aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.429+0000 7f775785c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77500c6cc0 con 0x7f77500c7940 2024-03-06T07:28:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.429+0000 7f775785c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77500c6b40 con 0x7f7750152a30 2024-03-06T07:28:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.429+0000 7f775785c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77500af6d0 con 0x7f77500afcb0 2024-03-06T07:28:27.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.429+0000 7f7754df7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77500c7940 0x7f775014a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.429+0000 7f7754df7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77500c7940 0x7f775014a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34920/0 (socket says 172.21.15.1:34920) 2024-03-06T07:28:27.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.429+0000 7f7754df7640 1 -- 172.21.15.1:0/1074398403 learned_addr learned my addr 172.21.15.1:0/1074398403 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:27.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.429+0000 7f7755df9640 1 --2- 172.21.15.1:0/1074398403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7750152a30 0x7f775014aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.429+0000 7f77555f8640 1 --2- 172.21.15.1:0/1074398403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77500afcb0 0x7f77500c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.430+0000 7f7754df7640 1 -- 172.21.15.1:0/1074398403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77500afcb0 msgr2=0x7f77500c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.430+0000 7f7754df7640 1 --2- 172.21.15.1:0/1074398403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77500afcb0 0x7f77500c3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.430+0000 7f7754df7640 1 -- 172.21.15.1:0/1074398403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7750152a30 msgr2=0x7f775014aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.430+0000 7f7754df7640 1 --2- 172.21.15.1:0/1074398403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7750152a30 0x7f775014aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.430+0000 7f7754df7640 1 -- 172.21.15.1:0/1074398403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77501eed90 con 0x7f77500c7940 2024-03-06T07:28:27.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.430+0000 7f7754df7640 1 --2- 172.21.15.1:0/1074398403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77500c7940 0x7f775014a520 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f774400e9d0 tx=0x7f774400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.430+0000 7f774dffb640 1 -- 172.21.15.1:0/1074398403 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7744017070 con 0x7f77500c7940 2024-03-06T07:28:27.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.430+0000 7f774dffb640 1 -- 172.21.15.1:0/1074398403 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f774400ca60 con 0x7f77500c7940 2024-03-06T07:28:27.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.430+0000 7f775785c640 1 -- 172.21.15.1:0/1074398403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77500ca590 con 0x7f77500c7940 2024-03-06T07:28:27.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.430+0000 7f774dffb640 1 -- 172.21.15.1:0/1074398403 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774400ce10 con 0x7f77500c7940 2024-03-06T07:28:27.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.430+0000 7f775785c640 1 -- 172.21.15.1:0/1074398403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77500c4340 con 0x7f77500c7940 2024-03-06T07:28:27.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.431+0000 7f775785c640 1 -- 172.21.15.1:0/1074398403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7750152590 con 0x7f77500c7940 2024-03-06T07:28:27.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.432+0000 7f774dffb640 1 -- 172.21.15.1:0/1074398403 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f774401a070 con 0x7f77500c7940 2024-03-06T07:28:27.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.432+0000 7f774dffb640 1 --2- 172.21.15.1:0/1074398403 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f773c06b4d0 0x7f773c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.432+0000 7f774dffb640 1 -- 172.21.15.1:0/1074398403 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f774408a3c0 con 0x7f77500c7940 2024-03-06T07:28:27.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.432+0000 7f77555f8640 1 --2- 172.21.15.1:0/1074398403 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f773c06b4d0 0x7f773c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.433+0000 7f77555f8640 1 --2- 172.21.15.1:0/1074398403 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f773c06b4d0 0x7f773c06d990 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f7748036000 tx=0x7f774803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.436+0000 7f774dffb640 1 -- 172.21.15.1:0/1074398403 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7750152590 con 0x7f77500c7940 2024-03-06T07:28:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.618+0000 7f775785c640 1 -- 172.21.15.1:0/1074398403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7750067fd0 con 0x7f77500c7940 2024-03-06T07:28:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.618+0000 7f774dffb640 1 -- 172.21.15.1:0/1074398403 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v188) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7744058c90 con 0x7f77500c7940 2024-03-06T07:28:27.621 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:28:27.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.620+0000 7f775785c640 1 -- 172.21.15.1:0/1074398403 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f773c06b4d0 msgr2=0x7f773c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.620+0000 7f775785c640 1 --2- 172.21.15.1:0/1074398403 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f773c06b4d0 0x7f773c06d990 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f7748036000 tx=0x7f774803d040 comp rx=0 tx=0).stop 2024-03-06T07:28:27.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.620+0000 7f775785c640 1 -- 172.21.15.1:0/1074398403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77500c7940 msgr2=0x7f775014a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.620+0000 7f775785c640 1 --2- 172.21.15.1:0/1074398403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77500c7940 0x7f775014a520 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f774400e9d0 tx=0x7f774400ee90 comp rx=0 tx=0).stop 2024-03-06T07:28:27.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.620+0000 7f775785c640 1 -- 172.21.15.1:0/1074398403 shutdown_connections 2024-03-06T07:28:27.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.620+0000 7f775785c640 1 --2- 172.21.15.1:0/1074398403 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f773c06b4d0 0x7f773c06d990 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.620+0000 7f775785c640 1 --2- 172.21.15.1:0/1074398403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7750152a30 0x7f775014aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.620+0000 7f775785c640 1 --2- 172.21.15.1:0/1074398403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77500c7940 0x7f775014a520 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.620+0000 7f775785c640 1 --2- 172.21.15.1:0/1074398403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f77500afcb0 0x7f77500c3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.620+0000 7f775785c640 1 -- 172.21.15.1:0/1074398403 >> 172.21.15.1:0/1074398403 conn(0x7f775008b8e0 msgr2=0x7f775014fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.621+0000 7f775785c640 1 -- 172.21.15.1:0/1074398403 shutdown_connections 2024-03-06T07:28:27.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.621+0000 7f775785c640 1 -- 172.21.15.1:0/1074398403 wait complete. 2024-03-06T07:28:27.633 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_health' 2024-03-06T07:28:27.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.732+0000 7fd8a7fb5640 1 Processor -- start 2024-03-06T07:28:27.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.732+0000 7fd8a7fb5640 1 -- start start 2024-03-06T07:28:27.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.733+0000 7fd8a7fb5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8a00c3970 0x7fd8a00c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.733+0000 7fd8a7fb5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8a00c60b0 0x7fd8a0152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.733+0000 7fd8a7fb5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd8a0152670 0x7fd8a01553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.733+0000 7fd8a7fb5640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8a00c67f0 con 0x7fd8a00c3970 2024-03-06T07:28:27.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.733+0000 7fd8a7fb5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8a00c5e40 con 0x7fd8a0152670 2024-03-06T07:28:27.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.733+0000 7fd8a7fb5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a00c6a40 con 0x7fd8a00c60b0 2024-03-06T07:28:27.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.733+0000 7fd8a5d51640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8a00c3970 0x7fd8a00c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.733+0000 7fd8a5d51640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8a00c3970 0x7fd8a00c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34922/0 (socket says 172.21.15.1:34922) 2024-03-06T07:28:27.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.733+0000 7fd8a5d51640 1 -- 172.21.15.1:0/808793917 learned_addr learned my addr 172.21.15.1:0/808793917 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:27.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.733+0000 7fd8a5550640 1 --2- 172.21.15.1:0/808793917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8a00c60b0 0x7fd8a0152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.733+0000 7fd8a6552640 1 --2- 172.21.15.1:0/808793917 >> v2:172.21.15.98:3300/0 conn(0x7fd8a0152670 0x7fd8a01553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.734+0000 7fd8a5d51640 1 -- 172.21.15.1:0/808793917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8a00c60b0 msgr2=0x7fd8a0152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.734+0000 7fd8a5d51640 1 --2- 172.21.15.1:0/808793917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8a00c60b0 0x7fd8a0152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.734+0000 7fd8a5d51640 1 -- 172.21.15.1:0/808793917 >> v2:172.21.15.98:3300/0 conn(0x7fd8a0152670 msgr2=0x7fd8a01553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.734+0000 7fd8a5d51640 1 --2- 172.21.15.1:0/808793917 >> v2:172.21.15.98:3300/0 conn(0x7fd8a0152670 0x7fd8a01553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.734+0000 7fd8a5d51640 1 -- 172.21.15.1:0/808793917 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a00c6d20 con 0x7fd8a00c3970 2024-03-06T07:28:27.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.734+0000 7fd8a6552640 1 --2- 172.21.15.1:0/808793917 >> v2:172.21.15.98:3300/0 conn(0x7fd8a0152670 0x7fd8a01553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:27.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.734+0000 7fd8a5d51640 1 --2- 172.21.15.1:0/808793917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a00c3970 0x7fd8a00c8170 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00ae4a0 tx=0x7fd89802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67006726fb12c83 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.734+0000 7fd8a4d4f640 1 -- 172.21.15.1:0/808793917 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898035020 con 0x7fd8a00c3970 2024-03-06T07:28:27.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.734+0000 7fd8a4d4f640 1 -- 172.21.15.1:0/808793917 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd898031870 con 0x7fd8a00c3970 2024-03-06T07:28:27.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.734+0000 7fd8a4d4f640 1 -- 172.21.15.1:0/808793917 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898031c40 con 0x7fd8a00c3970 2024-03-06T07:28:27.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.735+0000 7fd8a7fb5640 1 -- 172.21.15.1:0/808793917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a00c3970 msgr2=0x7fd8a00c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.735+0000 7fd8a7fb5640 1 --2- 172.21.15.1:0/808793917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a00c3970 0x7fd8a00c8170 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00ae4a0 tx=0x7fd89802b910 comp rx=0 tx=0).stop 2024-03-06T07:28:27.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.735+0000 7fd8a7fb5640 1 -- 172.21.15.1:0/808793917 shutdown_connections 2024-03-06T07:28:27.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.735+0000 7fd8a7fb5640 1 --2- 172.21.15.1:0/808793917 >> v2:172.21.15.98:3300/0 conn(0x7fd8a0152670 0x7fd8a01553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.735+0000 7fd8a7fb5640 1 --2- 172.21.15.1:0/808793917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8a00c60b0 0x7fd8a0152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.735+0000 7fd8a7fb5640 1 --2- 172.21.15.1:0/808793917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a00c3970 0x7fd8a00c8170 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.735+0000 7fd8a7fb5640 1 -- 172.21.15.1:0/808793917 >> 172.21.15.1:0/808793917 conn(0x7fd8a00c88f0 msgr2=0x7fd8a00c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.735+0000 7fd8a7fb5640 1 -- 172.21.15.1:0/808793917 shutdown_connections 2024-03-06T07:28:27.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.735+0000 7fd8a7fb5640 1 -- 172.21.15.1:0/808793917 wait complete. 2024-03-06T07:28:27.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.736+0000 7fd8a7fb5640 1 Processor -- start 2024-03-06T07:28:27.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.736+0000 7fd8a7fb5640 1 -- start start 2024-03-06T07:28:27.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a7fb5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8a00c3970 0x7fd8a00acd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a7fb5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a00c60b0 0x7fd8a0155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a7fb5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a0152670 0x7fd8a01eb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a7fb5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a00c75c0 con 0x7fd8a0152670 2024-03-06T07:28:27.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a7fb5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a00c72e0 con 0x7fd8a00c60b0 2024-03-06T07:28:27.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a7fb5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a00c3d50 con 0x7fd8a00c3970 2024-03-06T07:28:27.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a6552640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a0152670 0x7fd8a01eb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a6552640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a0152670 0x7fd8a01eb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34934/0 (socket says 172.21.15.1:34934) 2024-03-06T07:28:27.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a6552640 1 -- 172.21.15.1:0/3673348165 learned_addr learned my addr 172.21.15.1:0/3673348165 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:27.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a5d51640 1 --2- 172.21.15.1:0/3673348165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8a00c3970 0x7fd8a00acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a5550640 1 --2- 172.21.15.1:0/3673348165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a00c60b0 0x7fd8a0155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a6552640 1 -- 172.21.15.1:0/3673348165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8a00c3970 msgr2=0x7fd8a00acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a6552640 1 --2- 172.21.15.1:0/3673348165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8a00c3970 0x7fd8a00acd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a6552640 1 -- 172.21.15.1:0/3673348165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a00c60b0 msgr2=0x7fd8a0155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a6552640 1 --2- 172.21.15.1:0/3673348165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a00c60b0 0x7fd8a0155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a6552640 1 -- 172.21.15.1:0/3673348165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a00abb40 con 0x7fd8a0152670 2024-03-06T07:28:27.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a5d51640 1 --2- 172.21.15.1:0/3673348165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8a00c3970 0x7fd8a00acd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:27.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a5550640 1 --2- 172.21.15.1:0/3673348165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a00c60b0 0x7fd8a0155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:27.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd8a6552640 1 --2- 172.21.15.1:0/3673348165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a0152670 0x7fd8a01eb020 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7fd890007df0 tx=0x7fd89000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.737+0000 7fd89e7fc640 1 -- 172.21.15.1:0/3673348165 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89001a070 con 0x7fd8a0152670 2024-03-06T07:28:27.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.738+0000 7fd89e7fc640 1 -- 172.21.15.1:0/3673348165 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd890003990 con 0x7fd8a0152670 2024-03-06T07:28:27.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.738+0000 7fd8a7fb5640 1 -- 172.21.15.1:0/3673348165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a01eb560 con 0x7fd8a0152670 2024-03-06T07:28:27.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.738+0000 7fd89e7fc640 1 -- 172.21.15.1:0/3673348165 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd890003d80 con 0x7fd8a0152670 2024-03-06T07:28:27.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.738+0000 7fd8a7fb5640 1 -- 172.21.15.1:0/3673348165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a00ad640 con 0x7fd8a0152670 2024-03-06T07:28:27.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.738+0000 7fd8a7fb5640 1 -- 172.21.15.1:0/3673348165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a00c50c0 con 0x7fd8a0152670 2024-03-06T07:28:27.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.739+0000 7fd89e7fc640 1 -- 172.21.15.1:0/3673348165 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd89001d070 con 0x7fd8a0152670 2024-03-06T07:28:27.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.739+0000 7fd89e7fc640 1 --2- 172.21.15.1:0/3673348165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd88c06b4d0 0x7fd88c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.739+0000 7fd89e7fc640 1 -- 172.21.15.1:0/3673348165 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fd89008a370 con 0x7fd8a0152670 2024-03-06T07:28:27.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.740+0000 7fd8a5d51640 1 --2- 172.21.15.1:0/3673348165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd88c06b4d0 0x7fd88c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.740+0000 7fd8a5d51640 1 --2- 172.21.15.1:0/3673348165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd88c06b4d0 0x7fd88c06d990 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fd898036000 tx=0x7fd898008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.744+0000 7fd89e7fc640 1 -- 172.21.15.1:0/3673348165 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8a00c50c0 con 0x7fd8a0152670 2024-03-06T07:28:27.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:27.995+0000 7fd8a7fb5640 1 -- 172.21.15.1:0/3673348165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]} v 0) v1 -- 0x7fd8a00686e0 con 0x7fd8a0152670 2024-03-06T07:28:29.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.002+0000 7fd89e7fc640 1 -- 172.21.15.1:0/3673348165 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]}]=0 v469) v1 ==== 143+0+0 (secure 0 0 0) 0x7fd890010070 con 0x7fd8a0152670 2024-03-06T07:28:29.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.005+0000 7fd8a7fb5640 1 -- 172.21.15.1:0/3673348165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd88c06b4d0 msgr2=0x7fd88c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.005+0000 7fd8a7fb5640 1 --2- 172.21.15.1:0/3673348165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd88c06b4d0 0x7fd88c06d990 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fd898036000 tx=0x7fd898008030 comp rx=0 tx=0).stop 2024-03-06T07:28:29.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.005+0000 7fd8a7fb5640 1 -- 172.21.15.1:0/3673348165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a0152670 msgr2=0x7fd8a01eb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.005+0000 7fd8a7fb5640 1 --2- 172.21.15.1:0/3673348165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a0152670 0x7fd8a01eb020 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7fd890007df0 tx=0x7fd89000d4a0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.005+0000 7fd8a7fb5640 1 -- 172.21.15.1:0/3673348165 shutdown_connections 2024-03-06T07:28:29.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.005+0000 7fd8a7fb5640 1 --2- 172.21.15.1:0/3673348165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd88c06b4d0 0x7fd88c06d990 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.005+0000 7fd8a7fb5640 1 --2- 172.21.15.1:0/3673348165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a0152670 0x7fd8a01eb020 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.005+0000 7fd8a7fb5640 1 --2- 172.21.15.1:0/3673348165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a00c60b0 0x7fd8a0155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.005+0000 7fd8a7fb5640 1 --2- 172.21.15.1:0/3673348165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8a00c3970 0x7fd8a00acd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.005+0000 7fd8a7fb5640 1 -- 172.21.15.1:0/3673348165 >> 172.21.15.1:0/3673348165 conn(0x7fd8a00c88f0 msgr2=0x7fd8a00c8d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.005+0000 7fd8a7fb5640 1 -- 172.21.15.1:0/3673348165 shutdown_connections 2024-03-06T07:28:29.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.005+0000 7fd8a7fb5640 1 -- 172.21.15.1:0/3673348165 wait complete. 2024-03-06T07:28:29.020 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:28:29.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.129+0000 7f299ed1e640 1 Processor -- start 2024-03-06T07:28:29.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.129+0000 7f299ed1e640 1 -- start start 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.130+0000 7f299ed1e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2998152890 0x7f2998159030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.130+0000 7f299ed1e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f29980c8cf0 0x7f29980c9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.130+0000 7f299ed1e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2998155080 con 0x7f29980c7560 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.130+0000 7f299ed1e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2998155f70 con 0x7f2998152890 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.130+0000 7f299ed1e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29980c35b0 con 0x7f29980c8cf0 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.130+0000 7f299caba640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2998152890 0x7f2998159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.130+0000 7f299caba640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2998152890 0x7f2998159030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34942/0 (socket says 172.21.15.1:34942) 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.130+0000 7f299caba640 1 -- 172.21.15.1:0/1436927003 learned_addr learned my addr 172.21.15.1:0/1436927003 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.130+0000 7f299d2bb640 1 --2- 172.21.15.1:0/1436927003 >> v2:172.21.15.98:3300/0 conn(0x7f29980c8cf0 0x7f29980c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.130+0000 7f299caba640 1 -- 172.21.15.1:0/1436927003 >> v2:172.21.15.98:3300/0 conn(0x7f29980c8cf0 msgr2=0x7f29980c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.130+0000 7f299caba640 1 --2- 172.21.15.1:0/1436927003 >> v2:172.21.15.98:3300/0 conn(0x7f29980c8cf0 0x7f29980c9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.130+0000 7f299caba640 1 -- 172.21.15.1:0/1436927003 >> v1:172.21.15.98:6789/0 conn(0x7f29980c7560 legacy=0x7f2998155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.130+0000 7f299caba640 1 -- 172.21.15.1:0/1436927003 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29980c3890 con 0x7f2998152890 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.131+0000 7f299caba640 1 --2- 172.21.15.1:0/1436927003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2998152890 0x7f2998159030 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f298c009870 tx=0x7f298c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8de13843c2e72b16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.131+0000 7f29977fe640 1 -- 172.21.15.1:0/1436927003 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c038070 con 0x7f2998152890 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.131+0000 7f29977fe640 1 -- 172.21.15.1:0/1436927003 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f298c02f050 con 0x7f2998152890 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.131+0000 7f29977fe640 1 -- 172.21.15.1:0/1436927003 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c031970 con 0x7f2998152890 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.131+0000 7f299ed1e640 1 -- 172.21.15.1:0/1436927003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2998152890 msgr2=0x7f2998159030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.131+0000 7f299ed1e640 1 --2- 172.21.15.1:0/1436927003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2998152890 0x7f2998159030 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f298c009870 tx=0x7f298c02b7f0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.132+0000 7f299ed1e640 1 -- 172.21.15.1:0/1436927003 shutdown_connections 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.132+0000 7f299ed1e640 1 --2- 172.21.15.1:0/1436927003 >> v2:172.21.15.98:3300/0 conn(0x7f29980c8cf0 0x7f29980c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.132+0000 7f299ed1e640 1 --2- 172.21.15.1:0/1436927003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2998152890 0x7f2998159030 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.132+0000 7f299ed1e640 1 -- 172.21.15.1:0/1436927003 >> 172.21.15.1:0/1436927003 conn(0x7f299806b3e0 msgr2=0x7f299806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.132+0000 7f299ed1e640 1 -- 172.21.15.1:0/1436927003 shutdown_connections 2024-03-06T07:28:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.132+0000 7f299ed1e640 1 -- 172.21.15.1:0/1436927003 wait complete. 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.133+0000 7f299ed1e640 1 Processor -- start 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.133+0000 7f299ed1e640 1 -- start start 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.133+0000 7f299ed1e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29980c7560 0x7f29981e0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.133+0000 7f299ed1e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29980c8cf0 0x7f29981e1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.133+0000 7f299ed1e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2998152890 0x7f299814c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f299ed1e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2998159570 con 0x7f29980c7560 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f299ed1e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2998155340 con 0x7f29980c8cf0 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f299ed1e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29980c3260 con 0x7f2998152890 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f299caba640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29980c7560 0x7f29981e0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f299caba640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29980c7560 0x7f29981e0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34944/0 (socket says 172.21.15.1:34944) 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f299caba640 1 -- 172.21.15.1:0/2475954333 learned_addr learned my addr 172.21.15.1:0/2475954333 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f2997fff640 1 --2- 172.21.15.1:0/2475954333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29980c8cf0 0x7f29981e1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f299d2bb640 1 --2- 172.21.15.1:0/2475954333 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2998152890 0x7f299814c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f299caba640 1 -- 172.21.15.1:0/2475954333 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2998152890 msgr2=0x7f299814c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f299caba640 1 --2- 172.21.15.1:0/2475954333 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2998152890 0x7f299814c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f299caba640 1 -- 172.21.15.1:0/2475954333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29980c8cf0 msgr2=0x7f29981e1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f299caba640 1 --2- 172.21.15.1:0/2475954333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29980c8cf0 0x7f29981e1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f299caba640 1 -- 172.21.15.1:0/2475954333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29981ef590 con 0x7f29980c7560 2024-03-06T07:28:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f299d2bb640 1 --2- 172.21.15.1:0/2475954333 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2998152890 0x7f299814c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f299caba640 1 --2- 172.21.15.1:0/2475954333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29980c7560 0x7f29981e0c00 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f298c000c00 tx=0x7f298c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f29957fa640 1 -- 172.21.15.1:0/2475954333 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c042070 con 0x7f29980c7560 2024-03-06T07:28:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f29957fa640 1 -- 172.21.15.1:0/2475954333 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f298c03d890 con 0x7f29980c7560 2024-03-06T07:28:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f299ed1e640 1 -- 172.21.15.1:0/2475954333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29981deec0 con 0x7f29980c7560 2024-03-06T07:28:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f29957fa640 1 -- 172.21.15.1:0/2475954333 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c038040 con 0x7f29980c7560 2024-03-06T07:28:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.134+0000 7f299ed1e640 1 -- 172.21.15.1:0/2475954333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29981ef870 con 0x7f29980c7560 2024-03-06T07:28:29.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.135+0000 7f299ed1e640 1 -- 172.21.15.1:0/2475954333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2998154190 con 0x7f29980c7560 2024-03-06T07:28:29.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.136+0000 7f29957fa640 1 -- 172.21.15.1:0/2475954333 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f298c045070 con 0x7f29980c7560 2024-03-06T07:28:29.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.136+0000 7f29957fa640 1 --2- 172.21.15.1:0/2475954333 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f297c06b4d0 0x7f297c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.136+0000 7f29957fa640 1 -- 172.21.15.1:0/2475954333 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f298c0abf00 con 0x7f29980c7560 2024-03-06T07:28:29.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.136+0000 7f2997fff640 1 --2- 172.21.15.1:0/2475954333 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f297c06b4d0 0x7f297c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.137+0000 7f2997fff640 1 --2- 172.21.15.1:0/2475954333 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f297c06b4d0 0x7f297c06d990 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f2984006fd0 tx=0x7f2984008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.141+0000 7f29957fa640 1 -- 172.21.15.1:0/2475954333 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2998154190 con 0x7f29980c7560 2024-03-06T07:28:29.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.322+0000 7f299ed1e640 1 -- 172.21.15.1:0/2475954333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2998067fd0 con 0x7f29980c7560 2024-03-06T07:28:29.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.325+0000 7f29957fa640 1 -- 172.21.15.1:0/2475954333 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v61)=0 v61) v1 ==== 122+0+0 (secure 0 0 0) 0x7f298c07b050 con 0x7f29980c7560 2024-03-06T07:28:29.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.326+0000 7f299ed1e640 1 -- 172.21.15.1:0/2475954333 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f297c06b4d0 msgr2=0x7f297c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.326+0000 7f299ed1e640 1 --2- 172.21.15.1:0/2475954333 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f297c06b4d0 0x7f297c06d990 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f2984006fd0 tx=0x7f2984008040 comp rx=0 tx=0).stop 2024-03-06T07:28:29.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.326+0000 7f299ed1e640 1 -- 172.21.15.1:0/2475954333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29980c7560 msgr2=0x7f29981e0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.326+0000 7f299ed1e640 1 --2- 172.21.15.1:0/2475954333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29980c7560 0x7f29981e0c00 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f298c000c00 tx=0x7f298c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.327+0000 7f299ed1e640 1 -- 172.21.15.1:0/2475954333 shutdown_connections 2024-03-06T07:28:29.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.327+0000 7f299ed1e640 1 --2- 172.21.15.1:0/2475954333 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f297c06b4d0 0x7f297c06d990 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.327+0000 7f299ed1e640 1 --2- 172.21.15.1:0/2475954333 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2998152890 0x7f299814c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.327+0000 7f299ed1e640 1 --2- 172.21.15.1:0/2475954333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29980c8cf0 0x7f29981e1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.327+0000 7f299ed1e640 1 --2- 172.21.15.1:0/2475954333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29980c7560 0x7f29981e0c00 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.327+0000 7f299ed1e640 1 -- 172.21.15.1:0/2475954333 >> 172.21.15.1:0/2475954333 conn(0x7f299806b3e0 msgr2=0x7f2998158c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.327+0000 7f299ed1e640 1 -- 172.21.15.1:0/2475954333 shutdown_connections 2024-03-06T07:28:29.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.327+0000 7f299ed1e640 1 -- 172.21.15.1:0/2475954333 wait complete. 2024-03-06T07:28:29.339 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:28:29.340 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:29.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:29.425 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:28:29.425 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:28:29.425 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:29.440 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:28:29.440 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:28:29.440 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:29.483 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:28:29.483 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:28:29.483 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:29.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.582+0000 7fdedd3b6640 1 Processor -- start 2024-03-06T07:28:29.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.582+0000 7fdedd3b6640 1 -- start start 2024-03-06T07:28:29.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.583+0000 7fdedd3b6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fded80c8cd0 0x7fded80c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.583+0000 7fdedd3b6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fded80c3260 0x7fded8155b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.583+0000 7fdedd3b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fded80c7560 0x7fded8159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.583+0000 7fdedd3b6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fded80c2b90 con 0x7fded80c8cd0 2024-03-06T07:28:29.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.583+0000 7fdedd3b6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fded80c37b0 con 0x7fded80c3260 2024-03-06T07:28:29.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.583+0000 7fdedd3b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded80c3a90 con 0x7fded80c7560 2024-03-06T07:28:29.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.583+0000 7fded6ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fded80c8cd0 0x7fded80c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.583+0000 7fded6ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fded80c8cd0 0x7fded80c90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34956/0 (socket says 172.21.15.1:34956) 2024-03-06T07:28:29.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.583+0000 7fded6ffd640 1 -- 172.21.15.1:0/1361576404 learned_addr learned my addr 172.21.15.1:0/1361576404 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:29.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.583+0000 7fdecffff640 1 --2- 172.21.15.1:0/1361576404 >> v2:172.21.15.98:3300/0 conn(0x7fded80c3260 0x7fded8155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.583+0000 7fded77fe640 1 --2- 172.21.15.1:0/1361576404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fded80c7560 0x7fded8159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.583+0000 7fded6ffd640 1 -- 172.21.15.1:0/1361576404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fded80c7560 msgr2=0x7fded8159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.583+0000 7fded6ffd640 1 --2- 172.21.15.1:0/1361576404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fded80c7560 0x7fded8159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.583+0000 7fded6ffd640 1 -- 172.21.15.1:0/1361576404 >> v2:172.21.15.98:3300/0 conn(0x7fded80c3260 msgr2=0x7fded8155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.583+0000 7fded6ffd640 1 --2- 172.21.15.1:0/1361576404 >> v2:172.21.15.98:3300/0 conn(0x7fded80c3260 0x7fded8155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.583+0000 7fded6ffd640 1 -- 172.21.15.1:0/1361576404 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded8158810 con 0x7fded80c8cd0 2024-03-06T07:28:29.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.584+0000 7fded6ffd640 1 --2- 172.21.15.1:0/1361576404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded80c8cd0 0x7fded80c90b0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fded80c4ce0 tx=0x7fdec802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62ead704c422035 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.584+0000 7fded67fc640 1 -- 172.21.15.1:0/1361576404 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec802e070 con 0x7fded80c8cd0 2024-03-06T07:28:29.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.584+0000 7fded67fc640 1 -- 172.21.15.1:0/1361576404 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec8037070 con 0x7fded80c8cd0 2024-03-06T07:28:29.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.584+0000 7fded67fc640 1 -- 172.21.15.1:0/1361576404 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec8031a00 con 0x7fded80c8cd0 2024-03-06T07:28:29.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.585+0000 7fdedd3b6640 1 -- 172.21.15.1:0/1361576404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded80c8cd0 msgr2=0x7fded80c90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.585+0000 7fdedd3b6640 1 --2- 172.21.15.1:0/1361576404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded80c8cd0 0x7fded80c90b0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fded80c4ce0 tx=0x7fdec802b880 comp rx=0 tx=0).stop 2024-03-06T07:28:29.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.585+0000 7fdedd3b6640 1 -- 172.21.15.1:0/1361576404 shutdown_connections 2024-03-06T07:28:29.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.585+0000 7fdedd3b6640 1 --2- 172.21.15.1:0/1361576404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fded80c7560 0x7fded8159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.585+0000 7fdedd3b6640 1 --2- 172.21.15.1:0/1361576404 >> v2:172.21.15.98:3300/0 conn(0x7fded80c3260 0x7fded8155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.585+0000 7fdedd3b6640 1 --2- 172.21.15.1:0/1361576404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded80c8cd0 0x7fded80c90b0 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.585+0000 7fdedd3b6640 1 -- 172.21.15.1:0/1361576404 >> 172.21.15.1:0/1361576404 conn(0x7fded806b3e0 msgr2=0x7fded806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.585+0000 7fdedd3b6640 1 -- 172.21.15.1:0/1361576404 shutdown_connections 2024-03-06T07:28:29.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.585+0000 7fdedd3b6640 1 -- 172.21.15.1:0/1361576404 wait complete. 2024-03-06T07:28:29.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.586+0000 7fdedd3b6640 1 Processor -- start 2024-03-06T07:28:29.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.586+0000 7fdedd3b6640 1 -- start start 2024-03-06T07:28:29.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.587+0000 7fdedd3b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded80c3260 0x7fded814ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.587+0000 7fdedd3b6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded80c7560 0x7fded80a9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.587+0000 7fdedd3b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fded80c8cd0 0x7fded80a8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.587+0000 7fdedd3b6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded80c7a10 con 0x7fded80c7560 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.587+0000 7fdedd3b6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded8159550 con 0x7fded80c3260 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.587+0000 7fdedd3b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded80c8560 con 0x7fded80c8cd0 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.587+0000 7fdecffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded80c7560 0x7fded80a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.587+0000 7fdecffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded80c7560 0x7fded80a9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34970/0 (socket says 172.21.15.1:34970) 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.587+0000 7fdecffff640 1 -- 172.21.15.1:0/1432824445 learned_addr learned my addr 172.21.15.1:0/1432824445 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.587+0000 7fded6ffd640 1 --2- 172.21.15.1:0/1432824445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded80c3260 0x7fded814ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.587+0000 7fded77fe640 1 --2- 172.21.15.1:0/1432824445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fded80c8cd0 0x7fded80a8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.587+0000 7fdecffff640 1 -- 172.21.15.1:0/1432824445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fded80c8cd0 msgr2=0x7fded80a8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.587+0000 7fdecffff640 1 --2- 172.21.15.1:0/1432824445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fded80c8cd0 0x7fded80a8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.587+0000 7fdecffff640 1 -- 172.21.15.1:0/1432824445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded80c3260 msgr2=0x7fded814ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.587+0000 7fdecffff640 1 --2- 172.21.15.1:0/1432824445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded80c3260 0x7fded814ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.587+0000 7fdecffff640 1 -- 172.21.15.1:0/1432824445 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded81eefe0 con 0x7fded80c7560 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.587+0000 7fdecffff640 1 --2- 172.21.15.1:0/1432824445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded80c7560 0x7fded80a9960 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fdec000e840 tx=0x7fdec000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.588+0000 7fdecf7fe640 1 -- 172.21.15.1:0/1432824445 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec0017070 con 0x7fded80c7560 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.588+0000 7fdecf7fe640 1 -- 172.21.15.1:0/1432824445 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec000c730 con 0x7fded80c7560 2024-03-06T07:28:29.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.588+0000 7fdedd3b6640 1 -- 172.21.15.1:0/1432824445 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded8153080 con 0x7fded80c7560 2024-03-06T07:28:29.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.588+0000 7fdecf7fe640 1 -- 172.21.15.1:0/1432824445 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec000cb00 con 0x7fded80c7560 2024-03-06T07:28:29.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.588+0000 7fdedd3b6640 1 -- 172.21.15.1:0/1432824445 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded80a8bd0 con 0x7fded80c7560 2024-03-06T07:28:29.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.589+0000 7fdedd3b6640 1 -- 172.21.15.1:0/1432824445 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded8067fd0 con 0x7fded80c7560 2024-03-06T07:28:29.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.589+0000 7fdecf7fe640 1 -- 172.21.15.1:0/1432824445 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdec001a070 con 0x7fded80c7560 2024-03-06T07:28:29.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.589+0000 7fdecf7fe640 1 --2- 172.21.15.1:0/1432824445 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdeb806b4d0 0x7fdeb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.590+0000 7fdecf7fe640 1 -- 172.21.15.1:0/1432824445 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fdec008a170 con 0x7fded80c7560 2024-03-06T07:28:29.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.590+0000 7fded6ffd640 1 --2- 172.21.15.1:0/1432824445 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdeb806b4d0 0x7fdeb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.590+0000 7fded6ffd640 1 --2- 172.21.15.1:0/1432824445 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdeb806b4d0 0x7fdeb806d990 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fdec8036000 tx=0x7fdec803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.594+0000 7fdecf7fe640 1 -- 172.21.15.1:0/1432824445 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdec0058b50 con 0x7fded80c7560 2024-03-06T07:28:29.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.814+0000 7fdedd3b6640 1 -- 172.21.15.1:0/1432824445 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fded80c2d80 con 0x7fded80c7560 2024-03-06T07:28:29.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.814+0000 7fdecf7fe640 1 -- 172.21.15.1:0/1432824445 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+3694 (secure 0 0 0) 0x7fdec005cbf0 con 0x7fded80c7560 2024-03-06T07:28:29.817 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:29.817 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":181,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3574900332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3574900332},{"type":"v1","addr":"172.21.15.98:6836","nonce":3574900332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":8626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3840365296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3840365296},{"type":"v1","addr":"172.21.15.98:6837","nonce":3840365296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167},{"gid":10350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/12515206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":12515206},{"type":"v1","addr":"172.21.15.1:6834","nonce":12515206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":181},{"gid":10353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/3342612898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":3342612898},{"type":"v1","addr":"172.21.15.1:6835","nonce":3342612898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":181}],"filesystems":[]} 2024-03-06T07:28:29.817 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 181 2024-03-06T07:28:29.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.816+0000 7fdedd3b6640 1 -- 172.21.15.1:0/1432824445 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdeb806b4d0 msgr2=0x7fdeb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.816+0000 7fdedd3b6640 1 --2- 172.21.15.1:0/1432824445 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdeb806b4d0 0x7fdeb806d990 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fdec8036000 tx=0x7fdec803d040 comp rx=0 tx=0).stop 2024-03-06T07:28:29.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.816+0000 7fdedd3b6640 1 -- 172.21.15.1:0/1432824445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded80c7560 msgr2=0x7fded80a9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.816+0000 7fdedd3b6640 1 --2- 172.21.15.1:0/1432824445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded80c7560 0x7fded80a9960 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fdec000e840 tx=0x7fdec000ed00 comp rx=0 tx=0).stop 2024-03-06T07:28:29.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.816+0000 7fdedd3b6640 1 -- 172.21.15.1:0/1432824445 shutdown_connections 2024-03-06T07:28:29.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.816+0000 7fdedd3b6640 1 --2- 172.21.15.1:0/1432824445 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdeb806b4d0 0x7fdeb806d990 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.817+0000 7fdedd3b6640 1 --2- 172.21.15.1:0/1432824445 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fded80c8cd0 0x7fded80a8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.817+0000 7fdedd3b6640 1 --2- 172.21.15.1:0/1432824445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded80c7560 0x7fded80a9960 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.817+0000 7fdedd3b6640 1 --2- 172.21.15.1:0/1432824445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fded80c3260 0x7fded814ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.817+0000 7fdedd3b6640 1 -- 172.21.15.1:0/1432824445 >> 172.21.15.1:0/1432824445 conn(0x7fded806b3e0 msgr2=0x7fded8157d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.817+0000 7fdedd3b6640 1 -- 172.21.15.1:0/1432824445 shutdown_connections 2024-03-06T07:28:29.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.817+0000 7fdedd3b6640 1 -- 172.21.15.1:0/1432824445 wait complete. 2024-03-06T07:28:29.830 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:28:29.830 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:28:29.830 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:28:29.830 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:29.830 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:28:29.830 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:28:29.831 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:28:29.831 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:29.831 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:28:29.831 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:28:29.831 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:28:29.831 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:29.831 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:28:29.831 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:28:29.832 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:28:29.832 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:29.841 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:28:29.841 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:28:29.842 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:28:29.842 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:28:29.843 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:28:29.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:28:29.844 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:28:29.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:28:29.886 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:28:29.886 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:29.887 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:28:29.887 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:28:29.887 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:28:29.888 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:29.888 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:29.888 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:29.888 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:28:29.962 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:28:29.962 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:28:29.969 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:28:29.969 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:28:29.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.990+0000 7f35b7ebb640 1 Processor -- start 2024-03-06T07:28:29.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.991+0000 7f35b7ebb640 1 -- start start 2024-03-06T07:28:29.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.991+0000 7f35b7ebb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f35b0152650 0x7f35b0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.991+0000 7f35b7ebb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35b00bf9d0 0x7f35b00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.991+0000 7f35b7ebb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b00c2fc0 con 0x7f35b00c6b10 2024-03-06T07:28:29.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.991+0000 7f35b7ebb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b0155800 con 0x7f35b00bf9d0 2024-03-06T07:28:29.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.991+0000 7f35b7ebb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b0158da0 con 0x7f35b0152650 2024-03-06T07:28:29.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.991+0000 7f35b6458640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35b00bf9d0 0x7f35b00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.992+0000 7f35b6458640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35b00bf9d0 0x7f35b00c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34998/0 (socket says 172.21.15.1:34998) 2024-03-06T07:28:29.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.992+0000 7f35b6458640 1 -- 172.21.15.1:0/2776977076 learned_addr learned my addr 172.21.15.1:0/2776977076 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:29.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.992+0000 7f35b6458640 1 -- 172.21.15.1:0/2776977076 >> v2:172.21.15.98:3300/0 conn(0x7f35b0152650 msgr2=0x7f35b0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.992+0000 7f35b6458640 1 --2- 172.21.15.1:0/2776977076 >> v2:172.21.15.98:3300/0 conn(0x7f35b0152650 0x7f35b0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.992+0000 7f35b6458640 1 -- 172.21.15.1:0/2776977076 >> v1:172.21.15.98:6789/0 conn(0x7f35b00c6b10 legacy=0x7f35b0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.992+0000 7f35b6458640 1 -- 172.21.15.1:0/2776977076 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b00c6690 con 0x7f35b00bf9d0 2024-03-06T07:28:29.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.992+0000 7f35b6458640 1 --2- 172.21.15.1:0/2776977076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b00bf9d0 0x7f35b00c1460 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f35b00cac90 tx=0x7f35a002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=100ac963a82c735e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.993+0000 7f35b4c55640 1 -- 172.21.15.1:0/2776977076 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a002c9f0 con 0x7f35b00bf9d0 2024-03-06T07:28:29.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.993+0000 7f35b4c55640 1 -- 172.21.15.1:0/2776977076 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a002cb90 con 0x7f35b00bf9d0 2024-03-06T07:28:29.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.993+0000 7f35b4c55640 1 -- 172.21.15.1:0/2776977076 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a00343f0 con 0x7f35b00bf9d0 2024-03-06T07:28:29.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.993+0000 7f35b7ebb640 1 -- 172.21.15.1:0/2776977076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b00bf9d0 msgr2=0x7f35b00c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.993+0000 7f35b7ebb640 1 --2- 172.21.15.1:0/2776977076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b00bf9d0 0x7f35b00c1460 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f35b00cac90 tx=0x7f35a002be50 comp rx=0 tx=0).stop 2024-03-06T07:28:29.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.993+0000 7f35b7ebb640 1 -- 172.21.15.1:0/2776977076 shutdown_connections 2024-03-06T07:28:29.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.993+0000 7f35b7ebb640 1 --2- 172.21.15.1:0/2776977076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b00bf9d0 0x7f35b00c1460 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.993+0000 7f35b7ebb640 1 --2- 172.21.15.1:0/2776977076 >> v2:172.21.15.98:3300/0 conn(0x7f35b0152650 0x7f35b0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.993+0000 7f35b7ebb640 1 -- 172.21.15.1:0/2776977076 >> 172.21.15.1:0/2776977076 conn(0x7f35b0066ee0 msgr2=0x7f35b00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.993+0000 7f35b7ebb640 1 -- 172.21.15.1:0/2776977076 shutdown_connections 2024-03-06T07:28:29.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.993+0000 7f35b7ebb640 1 -- 172.21.15.1:0/2776977076 wait complete. 2024-03-06T07:28:29.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.994+0000 7f35b7ebb640 1 Processor -- start 2024-03-06T07:28:29.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.994+0000 7f35b7ebb640 1 -- start start 2024-03-06T07:28:29.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.995+0000 7f35b7ebb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b00bf9d0 0x7f35b01e1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.995+0000 7f35b7ebb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b0152650 0x7f35b014a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.995+0000 7f35b7ebb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b014ba40 0x7f35b014aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.995+0000 7f35b7ebb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b01551b0 con 0x7f35b0152650 2024-03-06T07:28:29.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.995+0000 7f35b7ebb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b00c6fa0 con 0x7f35b014ba40 2024-03-06T07:28:29.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.995+0000 7f35b7ebb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b0155490 con 0x7f35b00bf9d0 2024-03-06T07:28:29.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.995+0000 7f35b5456640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b0152650 0x7f35b014a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.995+0000 7f35b5456640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b0152650 0x7f35b014a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35014/0 (socket says 172.21.15.1:35014) 2024-03-06T07:28:29.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.995+0000 7f35b5456640 1 -- 172.21.15.1:0/4235665194 learned_addr learned my addr 172.21.15.1:0/4235665194 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:29.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.995+0000 7f35b6458640 1 --2- 172.21.15.1:0/4235665194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b014ba40 0x7f35b014aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.995+0000 7f35b5c57640 1 --2- 172.21.15.1:0/4235665194 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b00bf9d0 0x7f35b01e1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.995+0000 7f35b5456640 1 -- 172.21.15.1:0/4235665194 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b00bf9d0 msgr2=0x7f35b01e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.995+0000 7f35b5456640 1 --2- 172.21.15.1:0/4235665194 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b00bf9d0 0x7f35b01e1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.995+0000 7f35b5456640 1 -- 172.21.15.1:0/4235665194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b014ba40 msgr2=0x7f35b014aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.995+0000 7f35b5456640 1 --2- 172.21.15.1:0/4235665194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b014ba40 0x7f35b014aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.995+0000 7f35b5456640 1 -- 172.21.15.1:0/4235665194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b0153010 con 0x7f35b0152650 2024-03-06T07:28:30.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.996+0000 7f35b6458640 1 --2- 172.21.15.1:0/4235665194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b014ba40 0x7f35b014aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:30.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.996+0000 7f35b5c57640 1 --2- 172.21.15.1:0/4235665194 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b00bf9d0 0x7f35b01e1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:30.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.996+0000 7f35b5456640 1 --2- 172.21.15.1:0/4235665194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b0152650 0x7f35b014a970 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f35a400c410 tx=0x7f35a400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.996+0000 7f35ae7fc640 1 -- 172.21.15.1:0/4235665194 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a4017070 con 0x7f35b0152650 2024-03-06T07:28:30.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.996+0000 7f35ae7fc640 1 -- 172.21.15.1:0/4235665194 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a40035f0 con 0x7f35b0152650 2024-03-06T07:28:30.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.996+0000 7f35b7ebb640 1 -- 172.21.15.1:0/4235665194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b014d5b0 con 0x7f35b0152650 2024-03-06T07:28:30.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.996+0000 7f35ae7fc640 1 -- 172.21.15.1:0/4235665194 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a40039e0 con 0x7f35b0152650 2024-03-06T07:28:30.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.996+0000 7f35b7ebb640 1 -- 172.21.15.1:0/4235665194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b014db70 con 0x7f35b0152650 2024-03-06T07:28:30.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.997+0000 7f35b7ebb640 1 -- 172.21.15.1:0/4235665194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b0067fd0 con 0x7f35b0152650 2024-03-06T07:28:30.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.997+0000 7f35ae7fc640 1 -- 172.21.15.1:0/4235665194 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f35a401a070 con 0x7f35b0152650 2024-03-06T07:28:30.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.998+0000 7f35ae7fc640 1 --2- 172.21.15.1:0/4235665194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b4d0 0x7f359c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.998+0000 7f35ae7fc640 1 -- 172.21.15.1:0/4235665194 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f35a4089560 con 0x7f35b0152650 2024-03-06T07:28:30.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.998+0000 7f35b5c57640 1 --2- 172.21.15.1:0/4235665194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b4d0 0x7f359c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:29.999+0000 7f35b5c57640 1 --2- 172.21.15.1:0/4235665194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b4d0 0x7f359c06d990 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f35a8006fd0 tx=0x7f35a8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:30.002+0000 7f35ae7fc640 1 -- 172.21.15.1:0/4235665194 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f35a40584f0 con 0x7f35b0152650 2024-03-06T07:28:30.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:30.200+0000 7f35b7ebb640 1 -- 172.21.15.1:0/4235665194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f35b00c8cb0 con 0x7f35b0152650 2024-03-06T07:28:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.013+0000 7f35ae7fc640 1 -- 172.21.15.1:0/4235665194 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v193) v1 ==== 97+0+0 (secure 0 0 0) 0x7f35a40010c0 con 0x7f35b0152650 2024-03-06T07:28:31.015 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-06T07:28:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.016+0000 7f35b7ebb640 1 -- 172.21.15.1:0/4235665194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b4d0 msgr2=0x7f359c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.016+0000 7f35b7ebb640 1 --2- 172.21.15.1:0/4235665194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b4d0 0x7f359c06d990 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f35a8006fd0 tx=0x7f35a8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.016+0000 7f35b7ebb640 1 -- 172.21.15.1:0/4235665194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b0152650 msgr2=0x7f35b014a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.016+0000 7f35b7ebb640 1 --2- 172.21.15.1:0/4235665194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b0152650 0x7f35b014a970 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f35a400c410 tx=0x7f35a400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.017+0000 7f35b7ebb640 1 -- 172.21.15.1:0/4235665194 shutdown_connections 2024-03-06T07:28:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.017+0000 7f35b7ebb640 1 --2- 172.21.15.1:0/4235665194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b4d0 0x7f359c06d990 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.017+0000 7f35b7ebb640 1 --2- 172.21.15.1:0/4235665194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b014ba40 0x7f35b014aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.017+0000 7f35b7ebb640 1 --2- 172.21.15.1:0/4235665194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b0152650 0x7f35b014a970 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.017+0000 7f35b7ebb640 1 --2- 172.21.15.1:0/4235665194 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b00bf9d0 0x7f35b01e1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.017+0000 7f35b7ebb640 1 -- 172.21.15.1:0/4235665194 >> 172.21.15.1:0/4235665194 conn(0x7f35b0066ee0 msgr2=0x7f35b00c3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.017+0000 7f35b7ebb640 1 -- 172.21.15.1:0/4235665194 shutdown_connections 2024-03-06T07:28:31.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.017+0000 7f35b7ebb640 1 -- 172.21.15.1:0/4235665194 wait complete. 2024-03-06T07:28:31.032 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:28:31.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.138+0000 7ff1c9ca4640 1 Processor -- start 2024-03-06T07:28:31.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.138+0000 7ff1c9ca4640 1 -- start start 2024-03-06T07:28:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.139+0000 7ff1c9ca4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff1c40c5cb0 0x7ff1c4155930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.139+0000 7ff1c9ca4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c4158910 con 0x7ff1c4152920 2024-03-06T07:28:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.139+0000 7ff1c9ca4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c4158bf0 con 0x7ff1c40c5850 2024-03-06T07:28:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.139+0000 7ff1c9ca4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c4152d00 con 0x7ff1c40c5cb0 2024-03-06T07:28:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.139+0000 7ff1c3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff1c40c5cb0 0x7ff1c4155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.139+0000 7ff1c37fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff1c4152920 0x7ff1c406aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42464/0 (socket says 172.21.15.1:42464) 2024-03-06T07:28:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.139+0000 7ff1c37fe640 1 -- 172.21.15.1:0/4200011774 learned_addr learned my addr 172.21.15.1:0/4200011774 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:31.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.139+0000 7ff1c3fff640 1 -- 172.21.15.1:0/4200011774 >> v1:172.21.15.98:6789/0 conn(0x7ff1c40c5850 legacy=0x7ff1c4155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.139+0000 7ff1c3fff640 1 -- 172.21.15.1:0/4200011774 >> v1:172.21.15.1:6789/0 conn(0x7ff1c4152920 legacy=0x7ff1c406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.139+0000 7ff1c3fff640 1 -- 172.21.15.1:0/4200011774 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c4158ed0 con 0x7ff1c40c5cb0 2024-03-06T07:28:31.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.140+0000 7ff1c3fff640 1 --2- 172.21.15.1:0/4200011774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1c40c5cb0 0x7ff1c4155930 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009d90 tx=0x7ff1bc02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c106525c0806fe5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.140+0000 7ff1c27fc640 1 -- 172.21.15.1:0/4200011774 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc039070 con 0x7ff1c40c5cb0 2024-03-06T07:28:31.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.140+0000 7ff1c27fc640 1 -- 172.21.15.1:0/4200011774 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1bc02cbd0 con 0x7ff1c40c5cb0 2024-03-06T07:28:31.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.140+0000 7ff1c27fc640 1 -- 172.21.15.1:0/4200011774 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc034440 con 0x7ff1c40c5cb0 2024-03-06T07:28:31.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.141+0000 7ff1c9ca4640 1 -- 172.21.15.1:0/4200011774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1c40c5cb0 msgr2=0x7ff1c4155930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.141+0000 7ff1c9ca4640 1 --2- 172.21.15.1:0/4200011774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1c40c5cb0 0x7ff1c4155930 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009d90 tx=0x7ff1bc02be20 comp rx=0 tx=0).stop 2024-03-06T07:28:31.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.141+0000 7ff1c9ca4640 1 -- 172.21.15.1:0/4200011774 shutdown_connections 2024-03-06T07:28:31.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.141+0000 7ff1c9ca4640 1 --2- 172.21.15.1:0/4200011774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1c40c5cb0 0x7ff1c4155930 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.141+0000 7ff1c9ca4640 1 -- 172.21.15.1:0/4200011774 >> 172.21.15.1:0/4200011774 conn(0x7ff1c40a9700 msgr2=0x7ff1c40a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.141+0000 7ff1c9ca4640 1 -- 172.21.15.1:0/4200011774 shutdown_connections 2024-03-06T07:28:31.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.141+0000 7ff1c9ca4640 1 -- 172.21.15.1:0/4200011774 wait complete. 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.142+0000 7ff1c9ca4640 1 Processor -- start 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.142+0000 7ff1c9ca4640 1 -- start start 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c9ca4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1c40c5850 0x7ff1c4147a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c9ca4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1c40c5cb0 0x7ff1c4156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c9ca4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1c4152920 0x7ff1c40a7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c9ca4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c4159240 con 0x7ff1c4152920 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c9ca4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c41596d0 con 0x7ff1c40c5cb0 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c9ca4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c406b700 con 0x7ff1c40c5850 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1c4152920 0x7ff1c40a7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1c4152920 0x7ff1c40a7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35028/0 (socket says 172.21.15.1:35028) 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c3fff640 1 -- 172.21.15.1:0/820603427 learned_addr learned my addr 172.21.15.1:0/820603427 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c37fe640 1 --2- 172.21.15.1:0/820603427 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1c40c5850 0x7ff1c4147a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c3fff640 1 -- 172.21.15.1:0/820603427 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1c40c5850 msgr2=0x7ff1c4147a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c3fff640 1 --2- 172.21.15.1:0/820603427 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1c40c5850 0x7ff1c4147a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c3fff640 1 -- 172.21.15.1:0/820603427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1c40c5cb0 msgr2=0x7ff1c4156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c2ffd640 1 --2- 172.21.15.1:0/820603427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1c40c5cb0 0x7ff1c4156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c3fff640 1 --2- 172.21.15.1:0/820603427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1c40c5cb0 0x7ff1c4156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c3fff640 1 -- 172.21.15.1:0/820603427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c41497a0 con 0x7ff1c4152920 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c37fe640 1 --2- 172.21.15.1:0/820603427 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1c40c5850 0x7ff1c4147a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c3fff640 1 --2- 172.21.15.1:0/820603427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1c4152920 0x7ff1c40a7840 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009d90 tx=0x7ff1bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1abfff640 1 -- 172.21.15.1:0/820603427 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc039050 con 0x7ff1c4152920 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1abfff640 1 -- 172.21.15.1:0/820603427 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1bc03a040 con 0x7ff1c4152920 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c9ca4640 1 -- 172.21.15.1:0/820603427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c4149a80 con 0x7ff1c4152920 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1abfff640 1 -- 172.21.15.1:0/820603427 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc03e610 con 0x7ff1c4152920 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.143+0000 7ff1c9ca4640 1 -- 172.21.15.1:0/820603427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c414a040 con 0x7ff1c4152920 2024-03-06T07:28:31.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.144+0000 7ff1c9ca4640 1 -- 172.21.15.1:0/820603427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c4067fd0 con 0x7ff1c4152920 2024-03-06T07:28:31.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.145+0000 7ff1abfff640 1 -- 172.21.15.1:0/820603427 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1bc047020 con 0x7ff1c4152920 2024-03-06T07:28:31.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.145+0000 7ff1abfff640 1 --2- 172.21.15.1:0/820603427 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1b006b4d0 0x7ff1b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.145+0000 7ff1abfff640 1 -- 172.21.15.1:0/820603427 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ff1bc0ac3d0 con 0x7ff1c4152920 2024-03-06T07:28:31.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.146+0000 7ff1c37fe640 1 --2- 172.21.15.1:0/820603427 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1b006b4d0 0x7ff1b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.146+0000 7ff1c37fe640 1 --2- 172.21.15.1:0/820603427 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1b006b4d0 0x7ff1b006d990 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4000f60 tx=0x7ff1b4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.150+0000 7ff1abfff640 1 -- 172.21.15.1:0/820603427 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1bc04e020 con 0x7ff1c4152920 2024-03-06T07:28:31.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.397+0000 7ff1c9ca4640 1 -- 172.21.15.1:0/820603427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff1c4157430 con 0x7ff1c4152920 2024-03-06T07:28:31.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.398+0000 7ff1abfff640 1 -- 172.21.15.1:0/820603427 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v23) v1 ==== 80+0+4724 (secure 0 0 0) 0x7ff1bc07b560 con 0x7ff1c4152920 2024-03-06T07:28:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:31.400 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:28:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-06T07:28:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-06T07:28:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlnxyWARAAKIPXU0IafNQ1RzBPX+3DYw==", 2024-03-06T07:28:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:28:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:28:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:28:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:28:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:28:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:28:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-06T07:28:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl2fXxDRAAuLqH+HpKcN1JTts9ZtDTXA==", 2024-03-06T07:28:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:28:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:28:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:28:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:28:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:28:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:28:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-06T07:28:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl+IzqCBAAmgLkarFGt9Ihs/8kJYicBQ==", 2024-03-06T07:28:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:28:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:28:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:28:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:28:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:28:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:28:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-06T07:28:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlipdRFRAAA34vrsTEoX3L1CKb1YAV9g==", 2024-03-06T07:28:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:28:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:28:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:28:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:28:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:28:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:28:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-06T07:28:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAGGehlQK39LxAAOpPIZvl3FqQwCgyo3yoOGQ==", 2024-03-06T07:28:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:28:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:28:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:28:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:28:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-06T07:28:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAJGehlTxwDLhAAMwN9E0+3Jh5XNXnLo+fw3g==", 2024-03-06T07:28:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:28:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:28:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:28:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:28:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-06T07:28:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAMGehl4qrlLBAAm0VWQRNvSQIpDx5xf4Fyig==", 2024-03-06T07:28:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:28:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:28:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:28:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:28:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-06T07:28:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAPGehllO5HKxAAozm6ycG+fRxatoZTogeJTA==", 2024-03-06T07:28:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:28:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:28:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:28:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:28:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-06T07:28:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAVGehlLtLBMhAAUrV6q6ZCQluIcoo/S/NdbA==", 2024-03-06T07:28:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:28:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:28:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:28:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:28:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-06T07:28:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAYGehlHp59MRAAxxgpSPTEZkYKjl3fr0xeeQ==", 2024-03-06T07:28:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:28:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:28:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:28:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:28:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-06T07:28:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAbGehlTnXcMBAAFOBqcnh5ubAz+o6AGjccWQ==", 2024-03-06T07:28:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:28:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:28:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:28:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:28:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-06T07:28:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAeGehl1qH1LhAAaSl7Aj7qlN6d4xt5SKAmcg==", 2024-03-06T07:28:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:28:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:28:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:28:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:28:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-06T07:28:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng==", 2024-03-06T07:28:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:28:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:28:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-06T07:28:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:28:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:28:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-06T07:28:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlMyv7HBAAi7vt7aZWEaZuMpFrB5ofZw==", 2024-03-06T07:28:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:28:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:28:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-06T07:28:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-06T07:28:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-06T07:28:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:28:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-06T07:28:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlBS/vIBAAN86N2K4ze5PJB8kR869Lfg==", 2024-03-06T07:28:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:28:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:28:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:28:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:28:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:28:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:28:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-06T07:28:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlJ/jPORAAhLk5MTGafsGKhZSYhvhBZg==", 2024-03-06T07:28:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:28:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:28:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:28:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:28:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-06T07:28:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlAl8rNhAAvdU0Qvo/FJYnxc4/H8MIqQ==", 2024-03-06T07:28:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:28:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:28:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:28:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.411 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:31.411 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-06T07:28:31.411 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:28:31.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.399+0000 7ff1c9ca4640 1 -- 172.21.15.1:0/820603427 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1b006b4d0 msgr2=0x7ff1b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.399+0000 7ff1c9ca4640 1 --2- 172.21.15.1:0/820603427 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1b006b4d0 0x7ff1b006d990 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4000f60 tx=0x7ff1b4009040 comp rx=0 tx=0).stop 2024-03-06T07:28:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.399+0000 7ff1c9ca4640 1 -- 172.21.15.1:0/820603427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1c4152920 msgr2=0x7ff1c40a7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.399+0000 7ff1c9ca4640 1 --2- 172.21.15.1:0/820603427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1c4152920 0x7ff1c40a7840 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009d90 tx=0x7ff1bc004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.400+0000 7ff1c9ca4640 1 -- 172.21.15.1:0/820603427 shutdown_connections 2024-03-06T07:28:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.400+0000 7ff1c9ca4640 1 --2- 172.21.15.1:0/820603427 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff1b006b4d0 0x7ff1b006d990 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.400+0000 7ff1c9ca4640 1 --2- 172.21.15.1:0/820603427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1c4152920 0x7ff1c40a7840 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.400+0000 7ff1c9ca4640 1 --2- 172.21.15.1:0/820603427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1c40c5cb0 0x7ff1c4156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.400+0000 7ff1c9ca4640 1 --2- 172.21.15.1:0/820603427 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1c40c5850 0x7ff1c4147a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.400+0000 7ff1c9ca4640 1 -- 172.21.15.1:0/820603427 >> 172.21.15.1:0/820603427 conn(0x7ff1c40a9700 msgr2=0x7ff1c40c3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.400+0000 7ff1c9ca4640 1 -- 172.21.15.1:0/820603427 shutdown_connections 2024-03-06T07:28:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.400+0000 7ff1c9ca4640 1 -- 172.21.15.1:0/820603427 wait complete. 2024-03-06T07:28:31.413 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:31.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.512+0000 7fbd3a55e640 1 Processor -- start 2024-03-06T07:28:31.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.512+0000 7fbd3a55e640 1 -- start start 2024-03-06T07:28:31.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.513+0000 7fbd3a55e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd34152650 0x7fbd34155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.513+0000 7fbd3a55e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbd340bf9d0 0x7fbd340c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.513+0000 7fbd3a55e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd340c2fc0 con 0x7fbd340c6b10 2024-03-06T07:28:31.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.513+0000 7fbd3a55e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd34155800 con 0x7fbd340bf9d0 2024-03-06T07:28:31.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.513+0000 7fbd3a55e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd34158da0 con 0x7fbd34152650 2024-03-06T07:28:31.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.514+0000 7fbd38afb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbd340bf9d0 0x7fbd340c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.514+0000 7fbd33fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd34152650 0x7fbd34155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.514+0000 7fbd337fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbd340c6b10 0x7fbd34158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:60638/0 (socket says 172.21.15.1:60638) 2024-03-06T07:28:31.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.514+0000 7fbd337fe640 1 -- 172.21.15.1:0/622148150 learned_addr learned my addr 172.21.15.1:0/622148150 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:31.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.514+0000 7fbd32ffd640 1 -- 172.21.15.1:0/622148150 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500528352 0 0) 0x7fbd340c2fc0 con 0x7fbd340c6b10 2024-03-06T07:28:31.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.514+0000 7fbd32ffd640 1 -- 172.21.15.1:0/622148150 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd1c003520 con 0x7fbd340c6b10 2024-03-06T07:28:31.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.514+0000 7fbd38afb640 1 -- 172.21.15.1:0/622148150 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd34152650 msgr2=0x7fbd34155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.514+0000 7fbd38afb640 1 --2- 172.21.15.1:0/622148150 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd34152650 0x7fbd34155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.514+0000 7fbd38afb640 1 -- 172.21.15.1:0/622148150 >> v1:172.21.15.98:6789/0 conn(0x7fbd340c6b10 legacy=0x7fbd34158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.514+0000 7fbd38afb640 1 -- 172.21.15.1:0/622148150 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd340c6690 con 0x7fbd340bf9d0 2024-03-06T07:28:31.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.514+0000 7fbd33fff640 1 --2- 172.21.15.1:0/622148150 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd34152650 0x7fbd34155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:31.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.515+0000 7fbd38afb640 1 --2- 172.21.15.1:0/622148150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd340bf9d0 0x7fbd340c1460 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c009d90 tx=0x7fbd2c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41bb82d8f429e61a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.515+0000 7fbd32ffd640 1 -- 172.21.15.1:0/622148150 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c039070 con 0x7fbd340bf9d0 2024-03-06T07:28:31.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.515+0000 7fbd32ffd640 1 -- 172.21.15.1:0/622148150 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2c02cbc0 con 0x7fbd340bf9d0 2024-03-06T07:28:31.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.515+0000 7fbd32ffd640 1 -- 172.21.15.1:0/622148150 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c034440 con 0x7fbd340bf9d0 2024-03-06T07:28:31.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.515+0000 7fbd3a55e640 1 -- 172.21.15.1:0/622148150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd340bf9d0 msgr2=0x7fbd340c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.515+0000 7fbd3a55e640 1 --2- 172.21.15.1:0/622148150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd340bf9d0 0x7fbd340c1460 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c009d90 tx=0x7fbd2c02be10 comp rx=0 tx=0).stop 2024-03-06T07:28:31.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.515+0000 7fbd3a55e640 1 -- 172.21.15.1:0/622148150 shutdown_connections 2024-03-06T07:28:31.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.515+0000 7fbd3a55e640 1 --2- 172.21.15.1:0/622148150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd340bf9d0 0x7fbd340c1460 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.515+0000 7fbd3a55e640 1 --2- 172.21.15.1:0/622148150 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd34152650 0x7fbd34155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.515+0000 7fbd3a55e640 1 -- 172.21.15.1:0/622148150 >> 172.21.15.1:0/622148150 conn(0x7fbd34066ee0 msgr2=0x7fbd340672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.515+0000 7fbd3a55e640 1 -- 172.21.15.1:0/622148150 shutdown_connections 2024-03-06T07:28:31.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.515+0000 7fbd3a55e640 1 -- 172.21.15.1:0/622148150 wait complete. 2024-03-06T07:28:31.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.516+0000 7fbd3a55e640 1 Processor -- start 2024-03-06T07:28:31.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.517+0000 7fbd3a55e640 1 -- start start 2024-03-06T07:28:31.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.517+0000 7fbd3a55e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd340bf9d0 0x7fbd340c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.517+0000 7fbd3a55e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd34152650 0x7fbd34154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.517+0000 7fbd3a55e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd341e1200 0x7fbd3414af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.517+0000 7fbd3a55e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd341551b0 con 0x7fbd34152650 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.517+0000 7fbd3a55e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd340c6fa0 con 0x7fbd341e1200 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.517+0000 7fbd3a55e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd34155490 con 0x7fbd340bf9d0 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.517+0000 7fbd337fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd34152650 0x7fbd34154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.517+0000 7fbd337fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd34152650 0x7fbd34154560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35040/0 (socket says 172.21.15.1:35040) 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.517+0000 7fbd337fe640 1 -- 172.21.15.1:0/3162718755 learned_addr learned my addr 172.21.15.1:0/3162718755 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.518+0000 7fbd33fff640 1 --2- 172.21.15.1:0/3162718755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd340bf9d0 0x7fbd340c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.518+0000 7fbd38afb640 1 --2- 172.21.15.1:0/3162718755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd341e1200 0x7fbd3414af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.518+0000 7fbd337fe640 1 -- 172.21.15.1:0/3162718755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd340bf9d0 msgr2=0x7fbd340c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.518+0000 7fbd337fe640 1 --2- 172.21.15.1:0/3162718755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd340bf9d0 0x7fbd340c8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.518+0000 7fbd337fe640 1 -- 172.21.15.1:0/3162718755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd341e1200 msgr2=0x7fbd3414af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.518+0000 7fbd337fe640 1 --2- 172.21.15.1:0/3162718755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd341e1200 0x7fbd3414af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.518+0000 7fbd337fe640 1 -- 172.21.15.1:0/3162718755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd340ca7e0 con 0x7fbd34152650 2024-03-06T07:28:31.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.518+0000 7fbd337fe640 1 --2- 172.21.15.1:0/3162718755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd34152650 0x7fbd34154560 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7fbd2800c390 tx=0x7fbd2800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.518+0000 7fbd30ff9640 1 -- 172.21.15.1:0/3162718755 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd28017070 con 0x7fbd34152650 2024-03-06T07:28:31.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.518+0000 7fbd30ff9640 1 -- 172.21.15.1:0/3162718755 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd28005ce0 con 0x7fbd34152650 2024-03-06T07:28:31.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.518+0000 7fbd3a55e640 1 -- 172.21.15.1:0/3162718755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd3414fda0 con 0x7fbd34152650 2024-03-06T07:28:31.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.518+0000 7fbd30ff9640 1 -- 172.21.15.1:0/3162718755 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd280072f0 con 0x7fbd34152650 2024-03-06T07:28:31.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.518+0000 7fbd3a55e640 1 -- 172.21.15.1:0/3162718755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd34150320 con 0x7fbd34152650 2024-03-06T07:28:31.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.519+0000 7fbd3a55e640 1 -- 172.21.15.1:0/3162718755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd340b0330 con 0x7fbd34152650 2024-03-06T07:28:31.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.520+0000 7fbd30ff9640 1 -- 172.21.15.1:0/3162718755 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd2801a070 con 0x7fbd34152650 2024-03-06T07:28:31.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.520+0000 7fbd30ff9640 1 --2- 172.21.15.1:0/3162718755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbd2006b4d0 0x7fbd2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.520+0000 7fbd30ff9640 1 -- 172.21.15.1:0/3162718755 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbd280892f0 con 0x7fbd34152650 2024-03-06T07:28:31.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.520+0000 7fbd33fff640 1 --2- 172.21.15.1:0/3162718755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbd2006b4d0 0x7fbd2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.521+0000 7fbd33fff640 1 --2- 172.21.15.1:0/3162718755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbd2006b4d0 0x7fbd2006d990 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fbd24009560 tx=0x7fbd24009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.524+0000 7fbd30ff9640 1 -- 172.21.15.1:0/3162718755 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd340b0330 con 0x7fbd34152650 2024-03-06T07:28:31.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.707+0000 7fbd3a55e640 1 -- 172.21.15.1:0/3162718755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd34152ec0 con 0x7fbd34152650 2024-03-06T07:28:31.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.708+0000 7fbd30ff9640 1 -- 172.21.15.1:0/3162718755 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+13054 (secure 0 0 0) 0x7fbd34152ec0 con 0x7fbd34152650 2024-03-06T07:28:31.710 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:31.710 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":193,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:28:31.013405+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:31.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.710+0000 7fbd3a55e640 1 -- 172.21.15.1:0/3162718755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbd2006b4d0 msgr2=0x7fbd2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.710+0000 7fbd3a55e640 1 --2- 172.21.15.1:0/3162718755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbd2006b4d0 0x7fbd2006d990 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fbd24009560 tx=0x7fbd24009190 comp rx=0 tx=0).stop 2024-03-06T07:28:31.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.710+0000 7fbd3a55e640 1 -- 172.21.15.1:0/3162718755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd34152650 msgr2=0x7fbd34154560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.710+0000 7fbd3a55e640 1 --2- 172.21.15.1:0/3162718755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd34152650 0x7fbd34154560 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7fbd2800c390 tx=0x7fbd2800c850 comp rx=0 tx=0).stop 2024-03-06T07:28:31.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.710+0000 7fbd3a55e640 1 -- 172.21.15.1:0/3162718755 shutdown_connections 2024-03-06T07:28:31.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.710+0000 7fbd3a55e640 1 --2- 172.21.15.1:0/3162718755 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbd2006b4d0 0x7fbd2006d990 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.710+0000 7fbd3a55e640 1 --2- 172.21.15.1:0/3162718755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd341e1200 0x7fbd3414af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.710+0000 7fbd3a55e640 1 --2- 172.21.15.1:0/3162718755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd34152650 0x7fbd34154560 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.710+0000 7fbd3a55e640 1 --2- 172.21.15.1:0/3162718755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd340bf9d0 0x7fbd340c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.710+0000 7fbd3a55e640 1 -- 172.21.15.1:0/3162718755 >> 172.21.15.1:0/3162718755 conn(0x7fbd34066ee0 msgr2=0x7fbd340cb030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.710+0000 7fbd3a55e640 1 -- 172.21.15.1:0/3162718755 shutdown_connections 2024-03-06T07:28:31.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.710+0000 7fbd3a55e640 1 -- 172.21.15.1:0/3162718755 wait complete. 2024-03-06T07:28:31.723 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:31.724 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:31.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.826+0000 7fc4e5e84640 1 Processor -- start 2024-03-06T07:28:31.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.826+0000 7fc4e5e84640 1 -- start start 2024-03-06T07:28:31.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.827+0000 7fc4e5e84640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc4e0152650 0x7fc4e0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.827+0000 7fc4e5e84640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4e00c6b10 0x7fc4e0158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.827+0000 7fc4e5e84640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4e00c2fc0 con 0x7fc4e00bf9d0 2024-03-06T07:28:31.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.827+0000 7fc4e5e84640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4e0155800 con 0x7fc4e0152650 2024-03-06T07:28:31.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.827+0000 7fc4e5e84640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e00c6690 con 0x7fc4e00c6b10 2024-03-06T07:28:31.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.827+0000 7fc4deffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4e00c6b10 0x7fc4e0158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.827+0000 7fc4df7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc4e0152650 0x7fc4e0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.827+0000 7fc4deffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4e00c6b10 0x7fc4e0158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:56776/0 (socket says 172.21.15.1:56776) 2024-03-06T07:28:31.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.827+0000 7fc4df7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc4e0152650 0x7fc4e0155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:46296/0 (socket says 172.21.15.1:46296) 2024-03-06T07:28:31.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.827+0000 7fc4dffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc4e00bf9d0 0x7fc4e00c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:60650/0 (socket says 172.21.15.1:60650) 2024-03-06T07:28:31.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.827+0000 7fc4deffd640 1 -- 172.21.15.1:0/1099804500 learned_addr learned my addr 172.21.15.1:0/1099804500 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:31.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.827+0000 7fc4df7fe640 1 -- 172.21.15.1:0/1099804500 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4e00c6b10 msgr2=0x7fc4e0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.827+0000 7fc4df7fe640 1 --2- 172.21.15.1:0/1099804500 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4e00c6b10 0x7fc4e0158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.827+0000 7fc4df7fe640 1 -- 172.21.15.1:0/1099804500 >> v1:172.21.15.98:6789/0 conn(0x7fc4e00bf9d0 legacy=0x7fc4e00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.827+0000 7fc4df7fe640 1 -- 172.21.15.1:0/1099804500 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e00c7280 con 0x7fc4e0152650 2024-03-06T07:28:31.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.827+0000 7fc4deffd640 1 --2- 172.21.15.1:0/1099804500 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4e00c6b10 0x7fc4e0158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:31.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.828+0000 7fc4df7fe640 1 --2- 172.21.15.1:0/1099804500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e0152650 0x7fc4e0155c90 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00097a0 tx=0x7fc4d002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd2c1dbec1fef7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.828+0000 7fc4de7fc640 1 -- 172.21.15.1:0/1099804500 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d00315a0 con 0x7fc4e0152650 2024-03-06T07:28:31.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.828+0000 7fc4de7fc640 1 -- 172.21.15.1:0/1099804500 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d0037070 con 0x7fc4e0152650 2024-03-06T07:28:31.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.828+0000 7fc4de7fc640 1 -- 172.21.15.1:0/1099804500 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0031960 con 0x7fc4e0152650 2024-03-06T07:28:31.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.829+0000 7fc4e5e84640 1 -- 172.21.15.1:0/1099804500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e0152650 msgr2=0x7fc4e0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.829+0000 7fc4e5e84640 1 --2- 172.21.15.1:0/1099804500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e0152650 0x7fc4e0155c90 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00097a0 tx=0x7fc4d002b840 comp rx=0 tx=0).stop 2024-03-06T07:28:31.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.829+0000 7fc4e5e84640 1 -- 172.21.15.1:0/1099804500 shutdown_connections 2024-03-06T07:28:31.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.829+0000 7fc4e5e84640 1 --2- 172.21.15.1:0/1099804500 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4e00c6b10 0x7fc4e0158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.829+0000 7fc4e5e84640 1 --2- 172.21.15.1:0/1099804500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e0152650 0x7fc4e0155c90 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.829+0000 7fc4e5e84640 1 -- 172.21.15.1:0/1099804500 >> 172.21.15.1:0/1099804500 conn(0x7fc4e0066ee0 msgr2=0x7fc4e00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.829+0000 7fc4e5e84640 1 -- 172.21.15.1:0/1099804500 shutdown_connections 2024-03-06T07:28:31.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.829+0000 7fc4e5e84640 1 -- 172.21.15.1:0/1099804500 wait complete. 2024-03-06T07:28:31.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.830+0000 7fc4e5e84640 1 Processor -- start 2024-03-06T07:28:31.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.830+0000 7fc4e5e84640 1 -- start start 2024-03-06T07:28:31.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.831+0000 7fc4e5e84640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4e00bf9d0 0x7fc4e0144a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.831+0000 7fc4e5e84640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4e0152650 0x7fc4e01431f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.831+0000 7fc4e5e84640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e014e660 0x7fc4e01421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.831+0000 7fc4e5e84640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e0155490 con 0x7fc4e0152650 2024-03-06T07:28:31.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.831+0000 7fc4e5e84640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e01551b0 con 0x7fc4e014e660 2024-03-06T07:28:31.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.831+0000 7fc4e5e84640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e0152a30 con 0x7fc4e00bf9d0 2024-03-06T07:28:31.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.831+0000 7fc4deffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4e0152650 0x7fc4e01431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.831+0000 7fc4deffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4e0152650 0x7fc4e01431f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35050/0 (socket says 172.21.15.1:35050) 2024-03-06T07:28:31.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.831+0000 7fc4deffd640 1 -- 172.21.15.1:0/4285460194 learned_addr learned my addr 172.21.15.1:0/4285460194 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:31.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.831+0000 7fc4df7fe640 1 --2- 172.21.15.1:0/4285460194 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4e00bf9d0 0x7fc4e0144a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.831+0000 7fc4dffff640 1 --2- 172.21.15.1:0/4285460194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e014e660 0x7fc4e01421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.831+0000 7fc4deffd640 1 -- 172.21.15.1:0/4285460194 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4e00bf9d0 msgr2=0x7fc4e0144a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.831+0000 7fc4deffd640 1 --2- 172.21.15.1:0/4285460194 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4e00bf9d0 0x7fc4e0144a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.831+0000 7fc4deffd640 1 -- 172.21.15.1:0/4285460194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e014e660 msgr2=0x7fc4e01421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.831+0000 7fc4deffd640 1 --2- 172.21.15.1:0/4285460194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e014e660 0x7fc4e01421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.831+0000 7fc4deffd640 1 -- 172.21.15.1:0/4285460194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e0145030 con 0x7fc4e0152650 2024-03-06T07:28:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.831+0000 7fc4deffd640 1 --2- 172.21.15.1:0/4285460194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4e0152650 0x7fc4e01431f0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fc4d400e9d0 tx=0x7fc4d400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.832+0000 7fc4c7fff640 1 -- 172.21.15.1:0/4285460194 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d4017070 con 0x7fc4e0152650 2024-03-06T07:28:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.832+0000 7fc4c7fff640 1 -- 172.21.15.1:0/4285460194 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d400ca00 con 0x7fc4e0152650 2024-03-06T07:28:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.832+0000 7fc4e5e84640 1 -- 172.21.15.1:0/4285460194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e01ee7c0 con 0x7fc4e0152650 2024-03-06T07:28:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.832+0000 7fc4c7fff640 1 -- 172.21.15.1:0/4285460194 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d400cdb0 con 0x7fc4e0152650 2024-03-06T07:28:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.832+0000 7fc4e5e84640 1 -- 172.21.15.1:0/4285460194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e01eed80 con 0x7fc4e0152650 2024-03-06T07:28:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.833+0000 7fc4e5e84640 1 -- 172.21.15.1:0/4285460194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4e0067fd0 con 0x7fc4e0152650 2024-03-06T07:28:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.833+0000 7fc4c7fff640 1 -- 172.21.15.1:0/4285460194 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc4d401a070 con 0x7fc4e0152650 2024-03-06T07:28:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.833+0000 7fc4c7fff640 1 --2- 172.21.15.1:0/4285460194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc4cc06b500 0x7fc4cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.834+0000 7fc4c7fff640 1 -- 172.21.15.1:0/4285460194 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc4d408a290 con 0x7fc4e0152650 2024-03-06T07:28:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.834+0000 7fc4df7fe640 1 --2- 172.21.15.1:0/4285460194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc4cc06b500 0x7fc4cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.834+0000 7fc4df7fe640 1 --2- 172.21.15.1:0/4285460194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc4cc06b500 0x7fc4cc06d9c0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0036000 tx=0x7fc4d003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:31.838+0000 7fc4c7fff640 1 -- 172.21.15.1:0/4285460194 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc4d4058bf0 con 0x7fc4e0152650 2024-03-06T07:28:32.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.018+0000 7fc4e5e84640 1 -- 172.21.15.1:0/4285460194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc4e00c4910 con 0x7fc4e0152650 2024-03-06T07:28:32.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.019+0000 7fc4c7fff640 1 -- 172.21.15.1:0/4285460194 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v193) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc4d405cc90 con 0x7fc4e0152650 2024-03-06T07:28:32.021 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:28:32.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.021+0000 7fc4e5e84640 1 -- 172.21.15.1:0/4285460194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc4cc06b500 msgr2=0x7fc4cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.021+0000 7fc4e5e84640 1 --2- 172.21.15.1:0/4285460194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc4cc06b500 0x7fc4cc06d9c0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0036000 tx=0x7fc4d003d040 comp rx=0 tx=0).stop 2024-03-06T07:28:32.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.021+0000 7fc4e5e84640 1 -- 172.21.15.1:0/4285460194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4e0152650 msgr2=0x7fc4e01431f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.021+0000 7fc4e5e84640 1 --2- 172.21.15.1:0/4285460194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4e0152650 0x7fc4e01431f0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fc4d400e9d0 tx=0x7fc4d400ee90 comp rx=0 tx=0).stop 2024-03-06T07:28:32.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.021+0000 7fc4e5e84640 1 -- 172.21.15.1:0/4285460194 shutdown_connections 2024-03-06T07:28:32.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.021+0000 7fc4e5e84640 1 --2- 172.21.15.1:0/4285460194 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc4cc06b500 0x7fc4cc06d9c0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.021+0000 7fc4e5e84640 1 --2- 172.21.15.1:0/4285460194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e014e660 0x7fc4e01421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.021+0000 7fc4e5e84640 1 --2- 172.21.15.1:0/4285460194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4e0152650 0x7fc4e01431f0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.021+0000 7fc4e5e84640 1 --2- 172.21.15.1:0/4285460194 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4e00bf9d0 0x7fc4e0144a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.021+0000 7fc4e5e84640 1 -- 172.21.15.1:0/4285460194 >> 172.21.15.1:0/4285460194 conn(0x7fc4e0066ee0 msgr2=0x7fc4e00c3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.021+0000 7fc4e5e84640 1 -- 172.21.15.1:0/4285460194 shutdown_connections 2024-03-06T07:28:32.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.021+0000 7fc4e5e84640 1 -- 172.21.15.1:0/4285460194 wait complete. 2024-03-06T07:28:32.034 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:28:32.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.133+0000 7fe227f9a640 1 Processor -- start 2024-03-06T07:28:32.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.133+0000 7fe227f9a640 1 -- start start 2024-03-06T07:28:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.134+0000 7fe227f9a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe220152650 0x7fe220155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.134+0000 7fe227f9a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe2200c6b10 0x7fe220158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.134+0000 7fe227f9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe2200bf9d0 0x7fe2200c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.134+0000 7fe227f9a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe220155800 con 0x7fe2200c6b10 2024-03-06T07:28:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.134+0000 7fe227f9a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2200c2fc0 con 0x7fe220152650 2024-03-06T07:28:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.134+0000 7fe227f9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2200c6690 con 0x7fe2200bf9d0 2024-03-06T07:28:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.134+0000 7fe225535640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe2200c6b10 0x7fe220158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.134+0000 7fe225535640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe2200c6b10 0x7fe220158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35054/0 (socket says 172.21.15.1:35054) 2024-03-06T07:28:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.134+0000 7fe225535640 1 -- 172.21.15.1:0/1961672032 learned_addr learned my addr 172.21.15.1:0/1961672032 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.134+0000 7fe226537640 1 --2- 172.21.15.1:0/1961672032 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe2200bf9d0 0x7fe2200c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.135+0000 7fe225535640 1 -- 172.21.15.1:0/1961672032 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe2200bf9d0 msgr2=0x7fe2200c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.135+0000 7fe225d36640 1 --2- 172.21.15.1:0/1961672032 >> v2:172.21.15.98:3300/0 conn(0x7fe220152650 0x7fe220155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.135+0000 7fe225535640 1 --2- 172.21.15.1:0/1961672032 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe2200bf9d0 0x7fe2200c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.135+0000 7fe225535640 1 -- 172.21.15.1:0/1961672032 >> v2:172.21.15.98:3300/0 conn(0x7fe220152650 msgr2=0x7fe220155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.135+0000 7fe225535640 1 --2- 172.21.15.1:0/1961672032 >> v2:172.21.15.98:3300/0 conn(0x7fe220152650 0x7fe220155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.135+0000 7fe225535640 1 -- 172.21.15.1:0/1961672032 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2200c7280 con 0x7fe2200c6b10 2024-03-06T07:28:32.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.135+0000 7fe225d36640 1 --2- 172.21.15.1:0/1961672032 >> v2:172.21.15.98:3300/0 conn(0x7fe220152650 0x7fe220155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.135+0000 7fe225535640 1 --2- 172.21.15.1:0/1961672032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2200c6b10 0x7fe220158690 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fe2200cac90 tx=0x7fe21402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d5948db2c20fd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.135+0000 7fe224d34640 1 -- 172.21.15.1:0/1961672032 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21402e070 con 0x7fe2200c6b10 2024-03-06T07:28:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.136+0000 7fe224d34640 1 -- 172.21.15.1:0/1961672032 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe214037070 con 0x7fe2200c6b10 2024-03-06T07:28:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.136+0000 7fe224d34640 1 -- 172.21.15.1:0/1961672032 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214031a00 con 0x7fe2200c6b10 2024-03-06T07:28:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.136+0000 7fe227f9a640 1 -- 172.21.15.1:0/1961672032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2200c6b10 msgr2=0x7fe220158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.136+0000 7fe227f9a640 1 --2- 172.21.15.1:0/1961672032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2200c6b10 0x7fe220158690 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fe2200cac90 tx=0x7fe21402b880 comp rx=0 tx=0).stop 2024-03-06T07:28:32.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.136+0000 7fe227f9a640 1 -- 172.21.15.1:0/1961672032 shutdown_connections 2024-03-06T07:28:32.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.136+0000 7fe227f9a640 1 --2- 172.21.15.1:0/1961672032 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe2200bf9d0 0x7fe2200c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.136+0000 7fe227f9a640 1 --2- 172.21.15.1:0/1961672032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2200c6b10 0x7fe220158690 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.136+0000 7fe227f9a640 1 --2- 172.21.15.1:0/1961672032 >> v2:172.21.15.98:3300/0 conn(0x7fe220152650 0x7fe220155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.136+0000 7fe227f9a640 1 -- 172.21.15.1:0/1961672032 >> 172.21.15.1:0/1961672032 conn(0x7fe220066ee0 msgr2=0x7fe2200672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.136+0000 7fe227f9a640 1 -- 172.21.15.1:0/1961672032 shutdown_connections 2024-03-06T07:28:32.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.136+0000 7fe227f9a640 1 -- 172.21.15.1:0/1961672032 wait complete. 2024-03-06T07:28:32.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.137+0000 7fe227f9a640 1 Processor -- start 2024-03-06T07:28:32.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.137+0000 7fe227f9a640 1 -- start start 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.138+0000 7fe227f9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2200bf9d0 0x7fe220142100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.138+0000 7fe227f9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe220152650 0x7fe2201ee630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.138+0000 7fe227f9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe2200c9870 0x7fe2201eeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.138+0000 7fe227f9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe220155490 con 0x7fe2200bf9d0 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.138+0000 7fe227f9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2201551b0 con 0x7fe220152650 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.138+0000 7fe227f9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe220152a30 con 0x7fe2200c9870 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.138+0000 7fe225d36640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2200bf9d0 0x7fe220142100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.138+0000 7fe225d36640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2200bf9d0 0x7fe220142100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35056/0 (socket says 172.21.15.1:35056) 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.138+0000 7fe225d36640 1 -- 172.21.15.1:0/2149100691 learned_addr learned my addr 172.21.15.1:0/2149100691 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.138+0000 7fe225d36640 1 -- 172.21.15.1:0/2149100691 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe2200c9870 msgr2=0x7fe2201eeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.138+0000 7fe225d36640 1 --2- 172.21.15.1:0/2149100691 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe2200c9870 0x7fe2201eeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.138+0000 7fe225d36640 1 -- 172.21.15.1:0/2149100691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe220152650 msgr2=0x7fe2201ee630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.138+0000 7fe225d36640 1 --2- 172.21.15.1:0/2149100691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe220152650 0x7fe2201ee630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.138+0000 7fe225d36640 1 -- 172.21.15.1:0/2149100691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2200c5900 con 0x7fe2200bf9d0 2024-03-06T07:28:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.138+0000 7fe225d36640 1 --2- 172.21.15.1:0/2149100691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2200bf9d0 0x7fe220142100 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fe21800c820 tx=0x7fe21800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.139+0000 7fe21e7fc640 1 -- 172.21.15.1:0/2149100691 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe218015070 con 0x7fe2200bf9d0 2024-03-06T07:28:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.139+0000 7fe21e7fc640 1 -- 172.21.15.1:0/2149100691 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe218007a30 con 0x7fe2200bf9d0 2024-03-06T07:28:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.139+0000 7fe21e7fc640 1 -- 172.21.15.1:0/2149100691 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe218007e00 con 0x7fe2200bf9d0 2024-03-06T07:28:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.139+0000 7fe227f9a640 1 -- 172.21.15.1:0/2149100691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2200ca130 con 0x7fe2200bf9d0 2024-03-06T07:28:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.139+0000 7fe227f9a640 1 -- 172.21.15.1:0/2149100691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe220142640 con 0x7fe2200bf9d0 2024-03-06T07:28:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.140+0000 7fe227f9a640 1 -- 172.21.15.1:0/2149100691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe220067fd0 con 0x7fe2200bf9d0 2024-03-06T07:28:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.140+0000 7fe21e7fc640 1 -- 172.21.15.1:0/2149100691 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe218018070 con 0x7fe2200bf9d0 2024-03-06T07:28:32.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.140+0000 7fe21e7fc640 1 --2- 172.21.15.1:0/2149100691 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe20c06b4d0 0x7fe20c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.141+0000 7fe21e7fc640 1 -- 172.21.15.1:0/2149100691 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fe218089fd0 con 0x7fe2200bf9d0 2024-03-06T07:28:32.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.141+0000 7fe225535640 1 --2- 172.21.15.1:0/2149100691 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe20c06b4d0 0x7fe20c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.141+0000 7fe225535640 1 --2- 172.21.15.1:0/2149100691 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe20c06b4d0 0x7fe20c06d990 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fe214036000 tx=0x7fe214030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.145+0000 7fe21e7fc640 1 -- 172.21.15.1:0/2149100691 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe218059010 con 0x7fe2200bf9d0 2024-03-06T07:28:32.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.328+0000 7fe227f9a640 1 -- 172.21.15.1:0/2149100691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe2200b06f0 con 0x7fe2200bf9d0 2024-03-06T07:28:32.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.328+0000 7fe21e7fc640 1 -- 172.21.15.1:0/2149100691 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v193) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe218058910 con 0x7fe2200bf9d0 2024-03-06T07:28:32.330 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:32.331 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-06T07:28:32.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:28:32.331 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-06T07:28:32.331 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-06T07:28:32.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:28:32.331 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-06T07:28:32.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.331+0000 7fe227f9a640 1 -- 172.21.15.1:0/2149100691 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe20c06b4d0 msgr2=0x7fe20c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.331+0000 7fe227f9a640 1 --2- 172.21.15.1:0/2149100691 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe20c06b4d0 0x7fe20c06d990 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fe214036000 tx=0x7fe214030000 comp rx=0 tx=0).stop 2024-03-06T07:28:32.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.331+0000 7fe227f9a640 1 -- 172.21.15.1:0/2149100691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2200bf9d0 msgr2=0x7fe220142100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.331+0000 7fe227f9a640 1 --2- 172.21.15.1:0/2149100691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2200bf9d0 0x7fe220142100 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fe21800c820 tx=0x7fe21800cc20 comp rx=0 tx=0).stop 2024-03-06T07:28:32.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.331+0000 7fe227f9a640 1 -- 172.21.15.1:0/2149100691 shutdown_connections 2024-03-06T07:28:32.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.331+0000 7fe227f9a640 1 --2- 172.21.15.1:0/2149100691 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe20c06b4d0 0x7fe20c06d990 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.331+0000 7fe227f9a640 1 --2- 172.21.15.1:0/2149100691 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe2200c9870 0x7fe2201eeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.331+0000 7fe227f9a640 1 --2- 172.21.15.1:0/2149100691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe220152650 0x7fe2201ee630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.331+0000 7fe227f9a640 1 --2- 172.21.15.1:0/2149100691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2200bf9d0 0x7fe220142100 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.331+0000 7fe227f9a640 1 -- 172.21.15.1:0/2149100691 >> 172.21.15.1:0/2149100691 conn(0x7fe220066ee0 msgr2=0x7fe2200cac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.331+0000 7fe227f9a640 1 -- 172.21.15.1:0/2149100691 shutdown_connections 2024-03-06T07:28:32.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.331+0000 7fe227f9a640 1 -- 172.21.15.1:0/2149100691 wait complete. 2024-03-06T07:28:32.343 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:28:32.344 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:28:32.434 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:32.437 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:32.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.446+0000 7f053ff93640 1 Processor -- start 2024-03-06T07:28:32.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.447+0000 7f053ff93640 1 -- start start 2024-03-06T07:28:32.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.447+0000 7f053ff93640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05380c0760 0x7f05380c0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.447+0000 7f053ff93640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f05380a8540 0x7f053815fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.447+0000 7f053ff93640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0538073420 con 0x7f0538072e70 2024-03-06T07:28:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.447+0000 7f053ff93640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0538159180 con 0x7f05380a8540 2024-03-06T07:28:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.447+0000 7f053ff93640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05380c19f0 con 0x7f05380c0760 2024-03-06T07:28:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053d52e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0538072e70 0x7f05381551c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42478/0 (socket says 172.21.15.1:42478) 2024-03-06T07:28:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053d52e640 1 -- 172.21.15.1:0/603067703 learned_addr learned my addr 172.21.15.1:0/603067703 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053dd2f640 1 --2- 172.21.15.1:0/603067703 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05380c0760 0x7f05380c0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053e530640 1 --2- 172.21.15.1:0/603067703 >> v2:172.21.15.98:3300/0 conn(0x7f05380a8540 0x7f053815fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053cd2d640 1 -- 172.21.15.1:0/603067703 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9243102 0 0) 0x7f0538073420 con 0x7f0538072e70 2024-03-06T07:28:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053cd2d640 1 -- 172.21.15.1:0/603067703 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0524003520 con 0x7f0538072e70 2024-03-06T07:28:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053cd2d640 1 -- 172.21.15.1:0/603067703 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2398972068 0 0) 0x7f0524003520 con 0x7f0538072e70 2024-03-06T07:28:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053cd2d640 1 -- 172.21.15.1:0/603067703 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05380c0760 msgr2=0x7f05380c0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053cd2d640 1 --2- 172.21.15.1:0/603067703 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05380c0760 0x7f05380c0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053cd2d640 1 -- 172.21.15.1:0/603067703 >> v2:172.21.15.98:3300/0 conn(0x7f05380a8540 msgr2=0x7f053815fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053cd2d640 1 --2- 172.21.15.1:0/603067703 >> v2:172.21.15.98:3300/0 conn(0x7f05380a8540 0x7f053815fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053cd2d640 1 -- 172.21.15.1:0/603067703 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053806c3b0 con 0x7f0538072e70 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053cd2d640 1 -- 172.21.15.1:0/603067703 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f0534001670 con 0x7f0538072e70 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053cd2d640 1 -- 172.21.15.1:0/603067703 >> v1:172.21.15.1:6789/0 conn(0x7f0538072e70 legacy=0x7f05381551c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053cd2d640 1 --2- 172.21.15.1:0/603067703 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05240039b0 0x7f0524003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053cd2d640 1 --2- 172.21.15.1:0/603067703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05240043a0 0x7f0524004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053cd2d640 1 --2- 172.21.15.1:0/603067703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0524004f90 0x7f052400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053cd2d640 1 -- 172.21.15.1:0/603067703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0538159180 con 0x7f05240043a0 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053cd2d640 1 -- 172.21.15.1:0/603067703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05380c19f0 con 0x7f0524004f90 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053cd2d640 1 -- 172.21.15.1:0/603067703 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f052400bf30 con 0x7f05240039b0 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053dd2f640 1 --2- 172.21.15.1:0/603067703 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05380c0760 0x7f05380c0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.448+0000 7f053d52e640 1 --2- 172.21.15.1:0/603067703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05240043a0 0x7f0524004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.449+0000 7f053dd2f640 1 --2- 172.21.15.1:0/603067703 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05240039b0 0x7f0524003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.449+0000 7f053e530640 1 --2- 172.21.15.1:0/603067703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0524004f90 0x7f052400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.449+0000 7f053d52e640 1 -- 172.21.15.1:0/603067703 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05240039b0 msgr2=0x7f0524003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.449+0000 7f053d52e640 1 --2- 172.21.15.1:0/603067703 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05240039b0 0x7f0524003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.449+0000 7f053d52e640 1 -- 172.21.15.1:0/603067703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0524004f90 msgr2=0x7f052400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.449+0000 7f053d52e640 1 --2- 172.21.15.1:0/603067703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0524004f90 0x7f052400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.449+0000 7f053d52e640 1 -- 172.21.15.1:0/603067703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0524003520 con 0x7f05240043a0 2024-03-06T07:28:32.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.449+0000 7f053e530640 1 --2- 172.21.15.1:0/603067703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0524004f90 0x7f052400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:32.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.449+0000 7f053d52e640 1 --2- 172.21.15.1:0/603067703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05240043a0 0x7f0524004880 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f053400c290 tx=0x7f0524005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=594cd6d0a2246d9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.450+0000 7f053cd2d640 1 -- 172.21.15.1:0/603067703 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0534007d50 con 0x7f05240043a0 2024-03-06T07:28:32.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.450+0000 7f053cd2d640 1 -- 172.21.15.1:0/603067703 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0534030070 con 0x7f05240043a0 2024-03-06T07:28:32.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.450+0000 7f053cd2d640 1 -- 172.21.15.1:0/603067703 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053402cdb0 con 0x7f05240043a0 2024-03-06T07:28:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.450+0000 7f053ff93640 1 -- 172.21.15.1:0/603067703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05240043a0 msgr2=0x7f0524004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.450+0000 7f053ff93640 1 --2- 172.21.15.1:0/603067703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05240043a0 0x7f0524004880 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f053400c290 tx=0x7f0524005830 comp rx=0 tx=0).stop 2024-03-06T07:28:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.450+0000 7f053e530640 1 -- 172.21.15.1:0/603067703 reap_dead start 2024-03-06T07:28:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.450+0000 7f053ff93640 1 -- 172.21.15.1:0/603067703 shutdown_connections 2024-03-06T07:28:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.450+0000 7f053ff93640 1 -- 172.21.15.1:0/603067703 >> 172.21.15.1:0/603067703 conn(0x7f053806ed60 msgr2=0x7f053806f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.450+0000 7f053ff93640 1 -- 172.21.15.1:0/603067703 shutdown_connections 2024-03-06T07:28:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.450+0000 7f053ff93640 1 -- 172.21.15.1:0/603067703 wait complete. 2024-03-06T07:28:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.451+0000 7f053ff93640 1 Processor -- start 2024-03-06T07:28:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.452+0000 7f053ff93640 1 -- start start 2024-03-06T07:28:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.452+0000 7f053ff93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05240043a0 0x7f05381482e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.452+0000 7f053ff93640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05380c6e00 0x7f0538157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.452+0000 7f053ff93640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05380c2150 0x7f05380c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.452+0000 7f053ff93640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053806fdd0 con 0x7f05240043a0 2024-03-06T07:28:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.452+0000 7f053ff93640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053806fc50 con 0x7f05380c6e00 2024-03-06T07:28:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.452+0000 7f053ff93640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0538152a20 con 0x7f05380c2150 2024-03-06T07:28:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.452+0000 7f053dd2f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05240043a0 0x7f05381482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.452+0000 7f053dd2f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05240043a0 0x7f05381482e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35078/0 (socket says 172.21.15.1:35078) 2024-03-06T07:28:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.452+0000 7f053dd2f640 1 -- 172.21.15.1:0/494534791 learned_addr learned my addr 172.21.15.1:0/494534791 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.452+0000 7f053e530640 1 --2- 172.21.15.1:0/494534791 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05380c2150 0x7f05380c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.452+0000 7f053d52e640 1 --2- 172.21.15.1:0/494534791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05380c6e00 0x7f0538157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.453+0000 7f053dd2f640 1 -- 172.21.15.1:0/494534791 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05380c2150 msgr2=0x7f05380c25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.453+0000 7f053dd2f640 1 --2- 172.21.15.1:0/494534791 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05380c2150 0x7f05380c25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.453+0000 7f053dd2f640 1 -- 172.21.15.1:0/494534791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05380c6e00 msgr2=0x7f0538157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.453+0000 7f053dd2f640 1 --2- 172.21.15.1:0/494534791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05380c6e00 0x7f0538157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.453+0000 7f053dd2f640 1 -- 172.21.15.1:0/494534791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0538157020 con 0x7f05240043a0 2024-03-06T07:28:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.453+0000 7f053e530640 1 --2- 172.21.15.1:0/494534791 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05380c2150 0x7f05380c25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.453+0000 7f053d52e640 1 --2- 172.21.15.1:0/494534791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05380c6e00 0x7f0538157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.453+0000 7f053dd2f640 1 --2- 172.21.15.1:0/494534791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05240043a0 0x7f05381482e0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f052c011a00 tx=0x7f052c011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.453+0000 7f052a7fc640 1 -- 172.21.15.1:0/494534791 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c00fc10 con 0x7f05240043a0 2024-03-06T07:28:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.453+0000 7f052a7fc640 1 -- 172.21.15.1:0/494534791 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c00fdb0 con 0x7f05240043a0 2024-03-06T07:28:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.453+0000 7f053ff93640 1 -- 172.21.15.1:0/494534791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05380c5df0 con 0x7f05240043a0 2024-03-06T07:28:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.453+0000 7f052a7fc640 1 -- 172.21.15.1:0/494534791 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c014640 con 0x7f05240043a0 2024-03-06T07:28:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.453+0000 7f053ff93640 1 -- 172.21.15.1:0/494534791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0538157c60 con 0x7f05240043a0 2024-03-06T07:28:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.454+0000 7f053ff93640 1 -- 172.21.15.1:0/494534791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05380686e0 con 0x7f05240043a0 2024-03-06T07:28:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.454+0000 7f052a7fc640 1 -- 172.21.15.1:0/494534791 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f052c012040 con 0x7f05240043a0 2024-03-06T07:28:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.455+0000 7f052a7fc640 1 --2- 172.21.15.1:0/494534791 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f051c06b4d0 0x7f051c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.455+0000 7f052a7fc640 1 -- 172.21.15.1:0/494534791 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f052c08a730 con 0x7f05240043a0 2024-03-06T07:28:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.455+0000 7f053d52e640 1 --2- 172.21.15.1:0/494534791 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f051c06b4d0 0x7f051c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.455+0000 7f053d52e640 1 --2- 172.21.15.1:0/494534791 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f051c06b4d0 0x7f051c06d990 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f0534034000 tx=0x7f0534007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.459+0000 7f052a7fc640 1 -- 172.21.15.1:0/494534791 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f052c019030 con 0x7f05240043a0 2024-03-06T07:28:32.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:32.640+0000 7f053ff93640 1 -- 172.21.15.1:0/494534791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f053806fa30 con 0x7f05240043a0 2024-03-06T07:28:33.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.020+0000 7f052a7fc640 1 -- 172.21.15.1:0/494534791 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v194) v1 ==== 138+0+0 (secure 0 0 0) 0x7f052c059800 con 0x7f05240043a0 2024-03-06T07:28:33.022 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-06T07:28:33.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.023+0000 7f053ff93640 1 -- 172.21.15.1:0/494534791 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f051c06b4d0 msgr2=0x7f051c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.023+0000 7f053ff93640 1 --2- 172.21.15.1:0/494534791 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f051c06b4d0 0x7f051c06d990 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f0534034000 tx=0x7f0534007000 comp rx=0 tx=0).stop 2024-03-06T07:28:33.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.023+0000 7f053ff93640 1 -- 172.21.15.1:0/494534791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05240043a0 msgr2=0x7f05381482e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.023+0000 7f053ff93640 1 --2- 172.21.15.1:0/494534791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05240043a0 0x7f05381482e0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f052c011a00 tx=0x7f052c011ec0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.023+0000 7f053ff93640 1 -- 172.21.15.1:0/494534791 shutdown_connections 2024-03-06T07:28:33.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.023+0000 7f053ff93640 1 --2- 172.21.15.1:0/494534791 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f051c06b4d0 0x7f051c06d990 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.023+0000 7f053ff93640 1 --2- 172.21.15.1:0/494534791 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05380c2150 0x7f05380c25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.023+0000 7f053ff93640 1 --2- 172.21.15.1:0/494534791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05380c6e00 0x7f0538157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.023+0000 7f053ff93640 1 --2- 172.21.15.1:0/494534791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05240043a0 0x7f05381482e0 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.023+0000 7f053ff93640 1 -- 172.21.15.1:0/494534791 >> 172.21.15.1:0/494534791 conn(0x7f053806ed60 msgr2=0x7f05381513d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.023+0000 7f053ff93640 1 -- 172.21.15.1:0/494534791 shutdown_connections 2024-03-06T07:28:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.023+0000 7f053ff93640 1 -- 172.21.15.1:0/494534791 wait complete. 2024-03-06T07:28:33.040 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:28:33.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.147+0000 7ff59cd8c640 1 Processor -- start 2024-03-06T07:28:33.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.147+0000 7ff59cd8c640 1 -- start start 2024-03-06T07:28:33.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.148+0000 7ff59cd8c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff5980c2b30 0x7ff5980724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.148+0000 7ff59cd8c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff598151dd0 con 0x7ff598152b70 2024-03-06T07:28:33.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.148+0000 7ff59cd8c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff598156190 con 0x7ff5980a5fa0 2024-03-06T07:28:33.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.148+0000 7ff59cd8c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5981526e0 con 0x7ff5980c2b30 2024-03-06T07:28:33.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.148+0000 7ff595d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff598152b70 0x7ff5980c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42486/0 (socket says 172.21.15.1:42486) 2024-03-06T07:28:33.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.148+0000 7ff595d9b640 1 -- 172.21.15.1:0/216979597 learned_addr learned my addr 172.21.15.1:0/216979597 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:33.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.148+0000 7ff59559a640 1 -- 172.21.15.1:0/216979597 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929018664 0 0) 0x7ff598151dd0 con 0x7ff598152b70 2024-03-06T07:28:33.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.148+0000 7ff59659c640 1 --2- 172.21.15.1:0/216979597 >> v2:172.21.15.98:3300/0 conn(0x7ff5980c2b30 0x7ff5980724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.148+0000 7ff59559a640 1 -- 172.21.15.1:0/216979597 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff584005730 con 0x7ff598152b70 2024-03-06T07:28:33.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff59559a640 1 -- 172.21.15.1:0/216979597 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2265800788 0 0) 0x7ff584005730 con 0x7ff598152b70 2024-03-06T07:28:33.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff59559a640 1 -- 172.21.15.1:0/216979597 >> v2:172.21.15.98:3300/0 conn(0x7ff5980c2b30 msgr2=0x7ff5980724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff59559a640 1 --2- 172.21.15.1:0/216979597 >> v2:172.21.15.98:3300/0 conn(0x7ff5980c2b30 0x7ff5980724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff59559a640 1 -- 172.21.15.1:0/216979597 >> v1:172.21.15.98:6789/0 conn(0x7ff5980a5fa0 legacy=0x7ff59815c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff59559a640 1 -- 172.21.15.1:0/216979597 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff598152090 con 0x7ff598152b70 2024-03-06T07:28:33.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff59559a640 1 -- 172.21.15.1:0/216979597 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7ff580001670 con 0x7ff598152b70 2024-03-06T07:28:33.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff59559a640 1 -- 172.21.15.1:0/216979597 >> v1:172.21.15.1:6789/0 conn(0x7ff598152b70 legacy=0x7ff5980c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff59559a640 1 --2- 172.21.15.1:0/216979597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff584005bc0 0x7ff584005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff59559a640 1 --2- 172.21.15.1:0/216979597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff584006a10 0x7ff584006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff59559a640 1 --2- 172.21.15.1:0/216979597 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5840073d0 0x7ff58400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff59559a640 1 -- 172.21.15.1:0/216979597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5981526e0 con 0x7ff584006a10 2024-03-06T07:28:33.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff59559a640 1 -- 172.21.15.1:0/216979597 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58400e270 con 0x7ff584005bc0 2024-03-06T07:28:33.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff59559a640 1 -- 172.21.15.1:0/216979597 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58400e470 con 0x7ff5840073d0 2024-03-06T07:28:33.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff595d9b640 1 --2- 172.21.15.1:0/216979597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff584006a10 0x7ff584006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff59659c640 1 --2- 172.21.15.1:0/216979597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff584005bc0 0x7ff584005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff596d9d640 1 --2- 172.21.15.1:0/216979597 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5840073d0 0x7ff58400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff595d9b640 1 -- 172.21.15.1:0/216979597 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5840073d0 msgr2=0x7ff58400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff595d9b640 1 --2- 172.21.15.1:0/216979597 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5840073d0 0x7ff58400dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff595d9b640 1 -- 172.21.15.1:0/216979597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff584005bc0 msgr2=0x7ff584005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff595d9b640 1 --2- 172.21.15.1:0/216979597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff584005bc0 0x7ff584005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff595d9b640 1 -- 172.21.15.1:0/216979597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff584005730 con 0x7ff584006a10 2024-03-06T07:28:33.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.149+0000 7ff596d9d640 1 -- 172.21.15.1:0/216979597 reap_dead start 2024-03-06T07:28:33.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.150+0000 7ff595d9b640 1 --2- 172.21.15.1:0/216979597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff584006a10 0x7ff584006e90 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7ff58000c240 tx=0x7ff584007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae30c86fd0ae003e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.150+0000 7ff59559a640 1 -- 172.21.15.1:0/216979597 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff580007d10 con 0x7ff584006a10 2024-03-06T07:28:33.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.150+0000 7ff59559a640 1 -- 172.21.15.1:0/216979597 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff580030070 con 0x7ff584006a10 2024-03-06T07:28:33.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.150+0000 7ff59559a640 1 -- 172.21.15.1:0/216979597 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58002cdc0 con 0x7ff584006a10 2024-03-06T07:28:33.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.150+0000 7ff59cd8c640 1 -- 172.21.15.1:0/216979597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff584006a10 msgr2=0x7ff584006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.150+0000 7ff59cd8c640 1 --2- 172.21.15.1:0/216979597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff584006a10 0x7ff584006e90 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7ff58000c240 tx=0x7ff584007a40 comp rx=0 tx=0).stop 2024-03-06T07:28:33.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.151+0000 7ff59cd8c640 1 -- 172.21.15.1:0/216979597 shutdown_connections 2024-03-06T07:28:33.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.151+0000 7ff59cd8c640 1 --2- 172.21.15.1:0/216979597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff584006a10 0x7ff584006e90 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.151+0000 7ff59cd8c640 1 -- 172.21.15.1:0/216979597 >> 172.21.15.1:0/216979597 conn(0x7ff5980c6860 msgr2=0x7ff5980c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.151+0000 7ff59cd8c640 1 -- 172.21.15.1:0/216979597 shutdown_connections 2024-03-06T07:28:33.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.151+0000 7ff59cd8c640 1 -- 172.21.15.1:0/216979597 wait complete. 2024-03-06T07:28:33.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.152+0000 7ff59cd8c640 1 Processor -- start 2024-03-06T07:28:33.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.152+0000 7ff59cd8c640 1 -- start start 2024-03-06T07:28:33.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.152+0000 7ff59cd8c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff584006a10 0x7ff5980c03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.152+0000 7ff59cd8c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5980c4680 0x7ff5980ab270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.152+0000 7ff59cd8c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5980ab7b0 0x7ff5981eb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff59cd8c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff598155f00 con 0x7ff584006a10 2024-03-06T07:28:33.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff59cd8c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff598155d80 con 0x7ff5980ab7b0 2024-03-06T07:28:33.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff59cd8c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5980724c0 con 0x7ff5980c4680 2024-03-06T07:28:33.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff59659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff584006a10 0x7ff5980c03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff59659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff584006a10 0x7ff5980c03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35090/0 (socket says 172.21.15.1:35090) 2024-03-06T07:28:33.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff59659c640 1 -- 172.21.15.1:0/1074784581 learned_addr learned my addr 172.21.15.1:0/1074784581 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:33.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff595d9b640 1 --2- 172.21.15.1:0/1074784581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5980c4680 0x7ff5980ab270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff596d9d640 1 --2- 172.21.15.1:0/1074784581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5980ab7b0 0x7ff5981eb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff59659c640 1 -- 172.21.15.1:0/1074784581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5980c4680 msgr2=0x7ff5980ab270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff59659c640 1 --2- 172.21.15.1:0/1074784581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5980c4680 0x7ff5980ab270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff59659c640 1 -- 172.21.15.1:0/1074784581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5980ab7b0 msgr2=0x7ff5981eb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff59659c640 1 --2- 172.21.15.1:0/1074784581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5980ab7b0 0x7ff5981eb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff59659c640 1 -- 172.21.15.1:0/1074784581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59814c9b0 con 0x7ff584006a10 2024-03-06T07:28:33.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff596d9d640 1 --2- 172.21.15.1:0/1074784581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5980ab7b0 0x7ff5981eb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:33.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff59659c640 1 --2- 172.21.15.1:0/1074784581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff584006a10 0x7ff5980c03e0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7ff590011aa0 tx=0x7ff590011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff58effd640 1 -- 172.21.15.1:0/1074784581 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59000fb80 con 0x7ff584006a10 2024-03-06T07:28:33.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff58effd640 1 -- 172.21.15.1:0/1074784581 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59000fd20 con 0x7ff584006a10 2024-03-06T07:28:33.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff59cd8c640 1 -- 172.21.15.1:0/1074784581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5980c0920 con 0x7ff584006a10 2024-03-06T07:28:33.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff58effd640 1 -- 172.21.15.1:0/1074784581 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff590014660 con 0x7ff584006a10 2024-03-06T07:28:33.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.153+0000 7ff59cd8c640 1 -- 172.21.15.1:0/1074784581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5981ef020 con 0x7ff584006a10 2024-03-06T07:28:33.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.154+0000 7ff59cd8c640 1 -- 172.21.15.1:0/1074784581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff598067fd0 con 0x7ff584006a10 2024-03-06T07:28:33.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.155+0000 7ff58effd640 1 -- 172.21.15.1:0/1074784581 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff590012040 con 0x7ff584006a10 2024-03-06T07:28:33.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.155+0000 7ff58effd640 1 --2- 172.21.15.1:0/1074784581 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff57806b500 0x7ff57806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.155+0000 7ff58effd640 1 -- 172.21.15.1:0/1074784581 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 5619+0+0 (secure 0 0 0) 0x7ff59008ada0 con 0x7ff584006a10 2024-03-06T07:28:33.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.156+0000 7ff595d9b640 1 --2- 172.21.15.1:0/1074784581 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff57806b500 0x7ff57806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.156+0000 7ff595d9b640 1 --2- 172.21.15.1:0/1074784581 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff57806b500 0x7ff57806d9c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7ff580034000 tx=0x7ff580039040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.159+0000 7ff58effd640 1 -- 172.21.15.1:0/1074784581 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff59002b020 con 0x7ff584006a10 2024-03-06T07:28:33.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:33.343+0000 7ff59cd8c640 1 -- 172.21.15.1:0/1074784581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff598071790 con 0x7ff584006a10 2024-03-06T07:28:34.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.025+0000 7ff58effd640 1 -- 172.21.15.1:0/1074784581 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v195) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff590019030 con 0x7ff584006a10 2024-03-06T07:28:34.027 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-06T07:28:34.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.028+0000 7ff59cd8c640 1 -- 172.21.15.1:0/1074784581 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff57806b500 msgr2=0x7ff57806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.028+0000 7ff59cd8c640 1 --2- 172.21.15.1:0/1074784581 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff57806b500 0x7ff57806d9c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7ff580034000 tx=0x7ff580039040 comp rx=0 tx=0).stop 2024-03-06T07:28:34.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.028+0000 7ff59cd8c640 1 -- 172.21.15.1:0/1074784581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff584006a10 msgr2=0x7ff5980c03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.028+0000 7ff59cd8c640 1 --2- 172.21.15.1:0/1074784581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff584006a10 0x7ff5980c03e0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7ff590011aa0 tx=0x7ff590011f60 comp rx=0 tx=0).stop 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.028+0000 7ff59cd8c640 1 -- 172.21.15.1:0/1074784581 shutdown_connections 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.028+0000 7ff59cd8c640 1 --2- 172.21.15.1:0/1074784581 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff57806b500 0x7ff57806d9c0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.028+0000 7ff59cd8c640 1 --2- 172.21.15.1:0/1074784581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5980ab7b0 0x7ff5981eb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.028+0000 7ff59cd8c640 1 --2- 172.21.15.1:0/1074784581 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5980c4680 0x7ff5980ab270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.028+0000 7ff59cd8c640 1 --2- 172.21.15.1:0/1074784581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff584006a10 0x7ff5980c03e0 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.028+0000 7ff59cd8c640 1 -- 172.21.15.1:0/1074784581 >> 172.21.15.1:0/1074784581 conn(0x7ff5980c6860 msgr2=0x7ff5980c6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.028+0000 7ff59cd8c640 1 -- 172.21.15.1:0/1074784581 shutdown_connections 2024-03-06T07:28:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.028+0000 7ff59cd8c640 1 -- 172.21.15.1:0/1074784581 wait complete. 2024-03-06T07:28:34.055 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:28:34.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.160+0000 7fc58702a640 1 Processor -- start 2024-03-06T07:28:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.160+0000 7fc58702a640 1 -- start start 2024-03-06T07:28:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.160+0000 7fc58702a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc5800bf9d0 0x7fc580155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.160+0000 7fc58702a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5800c2fc0 con 0x7fc5800c6b10 2024-03-06T07:28:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.160+0000 7fc58702a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc580158da0 con 0x7fc580152650 2024-03-06T07:28:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.160+0000 7fc58702a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc580155800 con 0x7fc5800bf9d0 2024-03-06T07:28:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.160+0000 7fc57ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc5800c6b10 0x7fc5800c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42498/0 (socket says 172.21.15.1:42498) 2024-03-06T07:28:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.160+0000 7fc57ffff640 1 -- 172.21.15.1:0/3713334571 learned_addr learned my addr 172.21.15.1:0/3713334571 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.161+0000 7fc5855c7640 1 --2- 172.21.15.1:0/3713334571 >> v2:172.21.15.98:3300/0 conn(0x7fc5800bf9d0 0x7fc580155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.161+0000 7fc57f7fe640 1 -- 172.21.15.1:0/3713334571 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544202400 0 0) 0x7fc5800c2fc0 con 0x7fc5800c6b10 2024-03-06T07:28:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.161+0000 7fc57f7fe640 1 -- 172.21.15.1:0/3713334571 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc568003520 con 0x7fc5800c6b10 2024-03-06T07:28:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.161+0000 7fc5855c7640 1 -- 172.21.15.1:0/3713334571 >> v1:172.21.15.98:6789/0 conn(0x7fc580152650 legacy=0x7fc580158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.161+0000 7fc5855c7640 1 -- 172.21.15.1:0/3713334571 >> v1:172.21.15.1:6789/0 conn(0x7fc5800c6b10 legacy=0x7fc5800c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.161+0000 7fc5855c7640 1 -- 172.21.15.1:0/3713334571 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5800c6690 con 0x7fc5800bf9d0 2024-03-06T07:28:34.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.162+0000 7fc5855c7640 1 --2- 172.21.15.1:0/3713334571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5800bf9d0 0x7fc580155c90 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc5700097a0 tx=0x7fc57002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3890035a0b8c906 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.162+0000 7fc57f7fe640 1 -- 172.21.15.1:0/3713334571 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570035020 con 0x7fc5800bf9d0 2024-03-06T07:28:34.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.162+0000 7fc57f7fe640 1 -- 172.21.15.1:0/3713334571 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc570031830 con 0x7fc5800bf9d0 2024-03-06T07:28:34.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.162+0000 7fc57f7fe640 1 -- 172.21.15.1:0/3713334571 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570031be0 con 0x7fc5800bf9d0 2024-03-06T07:28:34.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.162+0000 7fc58702a640 1 -- 172.21.15.1:0/3713334571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5800bf9d0 msgr2=0x7fc580155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.162+0000 7fc58702a640 1 --2- 172.21.15.1:0/3713334571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5800bf9d0 0x7fc580155c90 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc5700097a0 tx=0x7fc57002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.162+0000 7fc58702a640 1 -- 172.21.15.1:0/3713334571 shutdown_connections 2024-03-06T07:28:34.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.162+0000 7fc58702a640 1 --2- 172.21.15.1:0/3713334571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5800bf9d0 0x7fc580155c90 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.162+0000 7fc58702a640 1 -- 172.21.15.1:0/3713334571 >> 172.21.15.1:0/3713334571 conn(0x7fc580066ee0 msgr2=0x7fc5800672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.163+0000 7fc58702a640 1 -- 172.21.15.1:0/3713334571 shutdown_connections 2024-03-06T07:28:34.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.163+0000 7fc58702a640 1 -- 172.21.15.1:0/3713334571 wait complete. 2024-03-06T07:28:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.164+0000 7fc58702a640 1 Processor -- start 2024-03-06T07:28:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.164+0000 7fc58702a640 1 -- start start 2024-03-06T07:28:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.164+0000 7fc58702a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5800bf9d0 0x7fc580151880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.164+0000 7fc58702a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc580152650 0x7fc580150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.164+0000 7fc58702a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc580158870 0x7fc5801505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.164+0000 7fc58702a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5801551b0 con 0x7fc580152650 2024-03-06T07:28:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.164+0000 7fc58702a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5800c6fa0 con 0x7fc580158870 2024-03-06T07:28:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.164+0000 7fc58702a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc580155490 con 0x7fc5800bf9d0 2024-03-06T07:28:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.164+0000 7fc57ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc580152650 0x7fc580150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.165+0000 7fc57ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc580152650 0x7fc580150070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35098/0 (socket says 172.21.15.1:35098) 2024-03-06T07:28:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.165+0000 7fc57ffff640 1 -- 172.21.15.1:0/3123507989 learned_addr learned my addr 172.21.15.1:0/3123507989 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.165+0000 7fc584dc6640 1 --2- 172.21.15.1:0/3123507989 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5800bf9d0 0x7fc580151880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.165+0000 7fc5855c7640 1 --2- 172.21.15.1:0/3123507989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc580158870 0x7fc5801505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.165+0000 7fc57ffff640 1 -- 172.21.15.1:0/3123507989 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5800bf9d0 msgr2=0x7fc580151880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.165+0000 7fc57ffff640 1 --2- 172.21.15.1:0/3123507989 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5800bf9d0 0x7fc580151880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.165+0000 7fc57ffff640 1 -- 172.21.15.1:0/3123507989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc580158870 msgr2=0x7fc5801505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.165+0000 7fc57ffff640 1 --2- 172.21.15.1:0/3123507989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc580158870 0x7fc5801505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.165+0000 7fc57ffff640 1 -- 172.21.15.1:0/3123507989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc580156cb0 con 0x7fc580152650 2024-03-06T07:28:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.165+0000 7fc57ffff640 1 --2- 172.21.15.1:0/3123507989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc580152650 0x7fc580150070 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fc57400c500 tx=0x7fc57400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.165+0000 7fc57d7fa640 1 -- 172.21.15.1:0/3123507989 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc574017070 con 0x7fc580152650 2024-03-06T07:28:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.165+0000 7fc57d7fa640 1 -- 172.21.15.1:0/3123507989 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc574005ce0 con 0x7fc580152650 2024-03-06T07:28:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.165+0000 7fc58702a640 1 -- 172.21.15.1:0/3123507989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5801eb8b0 con 0x7fc580152650 2024-03-06T07:28:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.165+0000 7fc57d7fa640 1 -- 172.21.15.1:0/3123507989 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc574007330 con 0x7fc580152650 2024-03-06T07:28:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.165+0000 7fc58702a640 1 -- 172.21.15.1:0/3123507989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5801ebb90 con 0x7fc580152650 2024-03-06T07:28:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.166+0000 7fc58702a640 1 -- 172.21.15.1:0/3123507989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5800c2f50 con 0x7fc580152650 2024-03-06T07:28:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.167+0000 7fc57d7fa640 1 -- 172.21.15.1:0/3123507989 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc57401a070 con 0x7fc580152650 2024-03-06T07:28:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.167+0000 7fc57d7fa640 1 --2- 172.21.15.1:0/3123507989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc56c06b4d0 0x7fc56c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.167+0000 7fc57d7fa640 1 -- 172.21.15.1:0/3123507989 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fc574089700 con 0x7fc580152650 2024-03-06T07:28:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.167+0000 7fc584dc6640 1 --2- 172.21.15.1:0/3123507989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc56c06b4d0 0x7fc56c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.168+0000 7fc584dc6640 1 --2- 172.21.15.1:0/3123507989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc56c06b4d0 0x7fc56c06d990 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fc58014f040 tx=0x7fc578009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.172+0000 7fc57d7fa640 1 -- 172.21.15.1:0/3123507989 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5800c2f50 con 0x7fc580152650 2024-03-06T07:28:34.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:34.393+0000 7fc58702a640 1 -- 172.21.15.1:0/3123507989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc580067fd0 con 0x7fc580152650 2024-03-06T07:28:35.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.046+0000 7fc57d7fa640 1 -- 172.21.15.1:0/3123507989 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 31 and data pool 32 v183) v1 ==== 174+0+0 (secure 0 0 0) 0x7fc574057eb0 con 0x7fc580152650 2024-03-06T07:28:35.050 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 31 and data pool 32 2024-03-06T07:28:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.055+0000 7fc58702a640 1 -- 172.21.15.1:0/3123507989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc56c06b4d0 msgr2=0x7fc56c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.055+0000 7fc58702a640 1 --2- 172.21.15.1:0/3123507989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc56c06b4d0 0x7fc56c06d990 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fc58014f040 tx=0x7fc578009040 comp rx=0 tx=0).stop 2024-03-06T07:28:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.055+0000 7fc58702a640 1 -- 172.21.15.1:0/3123507989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc580152650 msgr2=0x7fc580150070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.055+0000 7fc58702a640 1 --2- 172.21.15.1:0/3123507989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc580152650 0x7fc580150070 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fc57400c500 tx=0x7fc57400c9c0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.055+0000 7fc58702a640 1 -- 172.21.15.1:0/3123507989 shutdown_connections 2024-03-06T07:28:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.055+0000 7fc58702a640 1 --2- 172.21.15.1:0/3123507989 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc56c06b4d0 0x7fc56c06d990 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.055+0000 7fc58702a640 1 --2- 172.21.15.1:0/3123507989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc580158870 0x7fc5801505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.055+0000 7fc58702a640 1 --2- 172.21.15.1:0/3123507989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc580152650 0x7fc580150070 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.055+0000 7fc58702a640 1 --2- 172.21.15.1:0/3123507989 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5800bf9d0 0x7fc580151880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.055+0000 7fc58702a640 1 -- 172.21.15.1:0/3123507989 >> 172.21.15.1:0/3123507989 conn(0x7fc580066ee0 msgr2=0x7fc5800c39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.055+0000 7fc58702a640 1 -- 172.21.15.1:0/3123507989 shutdown_connections 2024-03-06T07:28:35.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.055+0000 7fc58702a640 1 -- 172.21.15.1:0/3123507989 wait complete. 2024-03-06T07:28:35.081 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:35.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.183+0000 7f44af1d6640 1 Processor -- start 2024-03-06T07:28:35.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.183+0000 7f44af1d6640 1 -- start start 2024-03-06T07:28:35.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.184+0000 7f44af1d6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f44a80c90b0 0x7f44a8155b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.184+0000 7f44af1d6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a80c2b90 con 0x7f44a80c8cd0 2024-03-06T07:28:35.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.184+0000 7f44af1d6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a80c37b0 con 0x7f44a80c3260 2024-03-06T07:28:35.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.184+0000 7f44af1d6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44a80c9540 con 0x7f44a80c90b0 2024-03-06T07:28:35.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.184+0000 7f44acf72640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f44a80c8cd0 0x7f44a80c7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45178/0 (socket says 172.21.15.1:45178) 2024-03-06T07:28:35.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.184+0000 7f44acf72640 1 -- 172.21.15.1:0/849202992 learned_addr learned my addr 172.21.15.1:0/849202992 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:35.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.184+0000 7f44ad773640 1 --2- 172.21.15.1:0/849202992 >> v2:172.21.15.1:3300/0 conn(0x7f44a80c90b0 0x7f44a8155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.184+0000 7f44a77fe640 1 -- 172.21.15.1:0/849202992 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111729410 0 0) 0x7f44a80c2b90 con 0x7f44a80c8cd0 2024-03-06T07:28:35.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.184+0000 7f44a77fe640 1 -- 172.21.15.1:0/849202992 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4494003520 con 0x7f44a80c8cd0 2024-03-06T07:28:35.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.184+0000 7f44ad773640 1 -- 172.21.15.1:0/849202992 >> v1:172.21.15.98:6789/0 conn(0x7f44a80c3260 legacy=0x7f44a8159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.184+0000 7f44ad773640 1 -- 172.21.15.1:0/849202992 >> v1:172.21.15.1:6789/0 conn(0x7f44a80c8cd0 legacy=0x7f44a80c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.184+0000 7f44ad773640 1 -- 172.21.15.1:0/849202992 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a80c3a90 con 0x7f44a80c90b0 2024-03-06T07:28:35.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.185+0000 7f44ad773640 1 --2- 172.21.15.1:0/849202992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c90b0 0x7f44a8155b70 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f44900097a0 tx=0x7f449002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826ba2af475ee4e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.185+0000 7f44a77fe640 1 -- 172.21.15.1:0/849202992 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44900315a0 con 0x7f44a80c90b0 2024-03-06T07:28:35.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.185+0000 7f44a77fe640 1 -- 172.21.15.1:0/849202992 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4490037070 con 0x7f44a80c90b0 2024-03-06T07:28:35.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.185+0000 7f44a77fe640 1 -- 172.21.15.1:0/849202992 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4490031960 con 0x7f44a80c90b0 2024-03-06T07:28:35.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.186+0000 7f44af1d6640 1 -- 172.21.15.1:0/849202992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c90b0 msgr2=0x7f44a8155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.186+0000 7f44af1d6640 1 --2- 172.21.15.1:0/849202992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c90b0 0x7f44a8155b70 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f44900097a0 tx=0x7f449002b840 comp rx=0 tx=0).stop 2024-03-06T07:28:35.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.186+0000 7f44af1d6640 1 -- 172.21.15.1:0/849202992 shutdown_connections 2024-03-06T07:28:35.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.186+0000 7f44af1d6640 1 --2- 172.21.15.1:0/849202992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c90b0 0x7f44a8155b70 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.186+0000 7f44af1d6640 1 -- 172.21.15.1:0/849202992 >> 172.21.15.1:0/849202992 conn(0x7f44a806b3e0 msgr2=0x7f44a806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.186+0000 7f44af1d6640 1 -- 172.21.15.1:0/849202992 shutdown_connections 2024-03-06T07:28:35.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.186+0000 7f44af1d6640 1 -- 172.21.15.1:0/849202992 wait complete. 2024-03-06T07:28:35.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.187+0000 7f44af1d6640 1 Processor -- start 2024-03-06T07:28:35.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.187+0000 7f44af1d6640 1 -- start start 2024-03-06T07:28:35.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44af1d6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c3260 0x7f44a80aae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44af1d6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44a80c8cd0 0x7f44a80ab700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44af1d6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a80c90b0 0x7f44a81edfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44af1d6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80c8560 con 0x7f44a80c3260 2024-03-06T07:28:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44af1d6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a8158af0 con 0x7f44a80c8cd0 2024-03-06T07:28:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44af1d6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80c86e0 con 0x7f44a80c90b0 2024-03-06T07:28:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44acf72640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c3260 0x7f44a80aae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44acf72640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c3260 0x7f44a80aae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54414/0 (socket says 172.21.15.1:54414) 2024-03-06T07:28:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44acf72640 1 -- 172.21.15.1:0/1767578768 learned_addr learned my addr 172.21.15.1:0/1767578768 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44ad773640 1 --2- 172.21.15.1:0/1767578768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a80c90b0 0x7f44a81edfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44a7fff640 1 --2- 172.21.15.1:0/1767578768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44a80c8cd0 0x7f44a80ab700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44acf72640 1 -- 172.21.15.1:0/1767578768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a80c90b0 msgr2=0x7f44a81edfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44acf72640 1 --2- 172.21.15.1:0/1767578768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a80c90b0 0x7f44a81edfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44acf72640 1 -- 172.21.15.1:0/1767578768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44a80c8cd0 msgr2=0x7f44a80ab700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44acf72640 1 --2- 172.21.15.1:0/1767578768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44a80c8cd0 0x7f44a80ab700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44acf72640 1 -- 172.21.15.1:0/1767578768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a80ab380 con 0x7f44a80c3260 2024-03-06T07:28:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44ad773640 1 --2- 172.21.15.1:0/1767578768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a80c90b0 0x7f44a81edfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.188+0000 7f44acf72640 1 --2- 172.21.15.1:0/1767578768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c3260 0x7f44a80aae40 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f44a000cb70 tx=0x7f44a000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.189+0000 7f44a57fa640 1 -- 172.21.15.1:0/1767578768 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a0017070 con 0x7f44a80c3260 2024-03-06T07:28:35.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.189+0000 7f44a57fa640 1 -- 172.21.15.1:0/1767578768 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a000ad40 con 0x7f44a80c3260 2024-03-06T07:28:35.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.189+0000 7f44af1d6640 1 -- 172.21.15.1:0/1767578768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a81de750 con 0x7f44a80c3260 2024-03-06T07:28:35.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.189+0000 7f44a57fa640 1 -- 172.21.15.1:0/1767578768 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a0012620 con 0x7f44a80c3260 2024-03-06T07:28:35.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.189+0000 7f44af1d6640 1 -- 172.21.15.1:0/1767578768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a81eead0 con 0x7f44a80c3260 2024-03-06T07:28:35.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.190+0000 7f44af1d6640 1 -- 172.21.15.1:0/1767578768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a8067fd0 con 0x7f44a80c3260 2024-03-06T07:28:35.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.190+0000 7f44a57fa640 1 -- 172.21.15.1:0/1767578768 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44a001a070 con 0x7f44a80c3260 2024-03-06T07:28:35.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.190+0000 7f44a57fa640 1 --2- 172.21.15.1:0/1767578768 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 0x7f448806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.190+0000 7f44a57fa640 1 -- 172.21.15.1:0/1767578768 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f44a008a210 con 0x7f44a80c3260 2024-03-06T07:28:35.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.191+0000 7f44a7fff640 1 --2- 172.21.15.1:0/1767578768 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 0x7f448806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.191+0000 7f44a7fff640 1 --2- 172.21.15.1:0/1767578768 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 0x7f448806d990 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f44a80adbf0 tx=0x7f4498009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.195+0000 7f44a57fa640 1 -- 172.21.15.1:0/1767578768 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44a0058f00 con 0x7f44a80c3260 2024-03-06T07:28:35.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.375+0000 7f44af1d6640 1 -- 172.21.15.1:0/1767578768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44a80c6760 con 0x7f44a80c3260 2024-03-06T07:28:35.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.376+0000 7f44a57fa640 1 -- 172.21.15.1:0/1767578768 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+16845 (secure 0 0 0) 0x7f44a0058800 con 0x7f44a80c3260 2024-03-06T07:28:35.379 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:35.379 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":196,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:28:34.395152+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:32.660232+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:33.362968+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:35.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.378+0000 7f44af1d6640 1 -- 172.21.15.1:0/1767578768 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 msgr2=0x7f448806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.378+0000 7f44af1d6640 1 --2- 172.21.15.1:0/1767578768 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 0x7f448806d990 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f44a80adbf0 tx=0x7f4498009040 comp rx=0 tx=0).stop 2024-03-06T07:28:35.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.378+0000 7f44af1d6640 1 -- 172.21.15.1:0/1767578768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c3260 msgr2=0x7f44a80aae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.378+0000 7f44af1d6640 1 --2- 172.21.15.1:0/1767578768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c3260 0x7f44a80aae40 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f44a000cb70 tx=0x7f44a000a430 comp rx=0 tx=0).stop 2024-03-06T07:28:35.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.378+0000 7f44af1d6640 1 -- 172.21.15.1:0/1767578768 shutdown_connections 2024-03-06T07:28:35.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.378+0000 7f44af1d6640 1 --2- 172.21.15.1:0/1767578768 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f448806b4d0 0x7f448806d990 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.378+0000 7f44af1d6640 1 --2- 172.21.15.1:0/1767578768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f44a80c90b0 0x7f44a81edfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.378+0000 7f44af1d6640 1 --2- 172.21.15.1:0/1767578768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44a80c8cd0 0x7f44a80ab700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.378+0000 7f44af1d6640 1 --2- 172.21.15.1:0/1767578768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44a80c3260 0x7f44a80aae40 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.378+0000 7f44af1d6640 1 -- 172.21.15.1:0/1767578768 >> 172.21.15.1:0/1767578768 conn(0x7f44a806b3e0 msgr2=0x7f44a806b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.379+0000 7f44af1d6640 1 -- 172.21.15.1:0/1767578768 shutdown_connections 2024-03-06T07:28:35.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.379+0000 7f44af1d6640 1 -- 172.21.15.1:0/1767578768 wait complete. 2024-03-06T07:28:35.392 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:35.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.491+0000 7fa9794ed640 1 Processor -- start 2024-03-06T07:28:35.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.491+0000 7fa9794ed640 1 -- start start 2024-03-06T07:28:35.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.492+0000 7fa9794ed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9740c05f0 0x7fa9740c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.492+0000 7fa9794ed640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa9740c14c0 0x7fa9741558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.492+0000 7fa9794ed640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9740c31f0 con 0x7fa97406f640 2024-03-06T07:28:35.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.492+0000 7fa9794ed640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa97406fe90 con 0x7fa9740c14c0 2024-03-06T07:28:35.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.492+0000 7fa9794ed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9740c2fc0 con 0x7fa9740c05f0 2024-03-06T07:28:35.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.492+0000 7fa9727fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa97406f640 0x7fa9741551b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45192/0 (socket says 172.21.15.1:45192) 2024-03-06T07:28:35.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.492+0000 7fa9727fc640 1 -- 172.21.15.1:0/3863526508 learned_addr learned my addr 172.21.15.1:0/3863526508 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.492+0000 7fa972ffd640 1 --2- 172.21.15.1:0/3863526508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9740c05f0 0x7fa9740c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.492+0000 7fa9737fe640 1 --2- 172.21.15.1:0/3863526508 >> v2:172.21.15.98:3300/0 conn(0x7fa9740c14c0 0x7fa9741558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.492+0000 7fa971ffb640 1 -- 172.21.15.1:0/3863526508 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 426958771 0 0) 0x7fa9740c31f0 con 0x7fa97406f640 2024-03-06T07:28:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.492+0000 7fa971ffb640 1 -- 172.21.15.1:0/3863526508 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa960003520 con 0x7fa97406f640 2024-03-06T07:28:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.492+0000 7fa971ffb640 1 -- 172.21.15.1:0/3863526508 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 844563742 0 0) 0x7fa960003520 con 0x7fa97406f640 2024-03-06T07:28:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.492+0000 7fa971ffb640 1 -- 172.21.15.1:0/3863526508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9740c05f0 msgr2=0x7fa9740c09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.492+0000 7fa971ffb640 1 --2- 172.21.15.1:0/3863526508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9740c05f0 0x7fa9740c09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa971ffb640 1 -- 172.21.15.1:0/3863526508 >> v2:172.21.15.98:3300/0 conn(0x7fa9740c14c0 msgr2=0x7fa9741558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa971ffb640 1 --2- 172.21.15.1:0/3863526508 >> v2:172.21.15.98:3300/0 conn(0x7fa9740c14c0 0x7fa9741558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa971ffb640 1 -- 172.21.15.1:0/3863526508 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa974155e00 con 0x7fa97406f640 2024-03-06T07:28:35.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa971ffb640 1 -- 172.21.15.1:0/3863526508 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fa95c001670 con 0x7fa97406f640 2024-03-06T07:28:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa971ffb640 1 -- 172.21.15.1:0/3863526508 >> v1:172.21.15.1:6789/0 conn(0x7fa97406f640 legacy=0x7fa9741551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa971ffb640 1 --2- 172.21.15.1:0/3863526508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9740c05f0 0x7fa9600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa971ffb640 1 --2- 172.21.15.1:0/3863526508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa960003f30 0x7fa960004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa971ffb640 1 --2- 172.21.15.1:0/3863526508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa960004d40 0x7fa9600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa971ffb640 1 -- 172.21.15.1:0/3863526508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97406fe90 con 0x7fa960004d40 2024-03-06T07:28:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa971ffb640 1 -- 172.21.15.1:0/3863526508 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96000b9c0 con 0x7fa960003f30 2024-03-06T07:28:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa971ffb640 1 -- 172.21.15.1:0/3863526508 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa96000bb40 con 0x7fa9740c05f0 2024-03-06T07:28:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa9737fe640 1 --2- 172.21.15.1:0/3863526508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa960004d40 0x7fa9600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa972ffd640 1 --2- 172.21.15.1:0/3863526508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9740c05f0 0x7fa9600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa9727fc640 1 --2- 172.21.15.1:0/3863526508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa960003f30 0x7fa960004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa9737fe640 1 -- 172.21.15.1:0/3863526508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9740c05f0 msgr2=0x7fa9600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa9737fe640 1 --2- 172.21.15.1:0/3863526508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9740c05f0 0x7fa9600039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa9737fe640 1 -- 172.21.15.1:0/3863526508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa960003f30 msgr2=0x7fa960004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa9737fe640 1 --2- 172.21.15.1:0/3863526508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa960003f30 0x7fa960004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.493+0000 7fa9737fe640 1 -- 172.21.15.1:0/3863526508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa960003520 con 0x7fa960004d40 2024-03-06T07:28:35.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.494+0000 7fa9737fe640 1 --2- 172.21.15.1:0/3863526508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa960004d40 0x7fa9600051c0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7fa96c009ac0 tx=0x7fa960005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c70534ab31aa5b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.494+0000 7fa971ffb640 1 -- 172.21.15.1:0/3863526508 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96c039070 con 0x7fa960004d40 2024-03-06T07:28:35.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.494+0000 7fa971ffb640 1 -- 172.21.15.1:0/3863526508 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96c02c6d0 con 0x7fa960004d40 2024-03-06T07:28:35.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.494+0000 7fa971ffb640 1 -- 172.21.15.1:0/3863526508 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96c009690 con 0x7fa960004d40 2024-03-06T07:28:35.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.494+0000 7fa9794ed640 1 -- 172.21.15.1:0/3863526508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa960004d40 msgr2=0x7fa9600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.494+0000 7fa9794ed640 1 --2- 172.21.15.1:0/3863526508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa960004d40 0x7fa9600051c0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7fa96c009ac0 tx=0x7fa960005830 comp rx=0 tx=0).stop 2024-03-06T07:28:35.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.494+0000 7fa9737fe640 1 -- 172.21.15.1:0/3863526508 reap_dead start 2024-03-06T07:28:35.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.494+0000 7fa9794ed640 1 -- 172.21.15.1:0/3863526508 shutdown_connections 2024-03-06T07:28:35.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.494+0000 7fa9794ed640 1 -- 172.21.15.1:0/3863526508 >> 172.21.15.1:0/3863526508 conn(0x7fa9740c3cc0 msgr2=0x7fa9740c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.494+0000 7fa9794ed640 1 -- 172.21.15.1:0/3863526508 shutdown_connections 2024-03-06T07:28:35.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.495+0000 7fa9794ed640 1 -- 172.21.15.1:0/3863526508 wait complete. 2024-03-06T07:28:35.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.496+0000 7fa9794ed640 1 Processor -- start 2024-03-06T07:28:35.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.496+0000 7fa9794ed640 1 -- start start 2024-03-06T07:28:35.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.496+0000 7fa9794ed640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa974158800 0x7fa974152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.496+0000 7fa9794ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa97414d600 0x7fa97414ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.496+0000 7fa9794ed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9740aaa60 0x7fa97413f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.496+0000 7fa9794ed640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9740c3770 con 0x7fa974158800 2024-03-06T07:28:35.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.496+0000 7fa9794ed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9740c3490 con 0x7fa97414d600 2024-03-06T07:28:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.496+0000 7fa9794ed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9740c1040 con 0x7fa9740aaa60 2024-03-06T07:28:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.496+0000 7fa972ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa974158800 0x7fa974152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.496+0000 7fa972ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa974158800 0x7fa974152e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54430/0 (socket says 172.21.15.1:54430) 2024-03-06T07:28:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.496+0000 7fa972ffd640 1 -- 172.21.15.1:0/3381713531 learned_addr learned my addr 172.21.15.1:0/3381713531 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.497+0000 7fa9737fe640 1 --2- 172.21.15.1:0/3381713531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9740aaa60 0x7fa97413f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.497+0000 7fa9727fc640 1 --2- 172.21.15.1:0/3381713531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa97414d600 0x7fa97414ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.497+0000 7fa972ffd640 1 -- 172.21.15.1:0/3381713531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9740aaa60 msgr2=0x7fa97413f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.497+0000 7fa972ffd640 1 --2- 172.21.15.1:0/3381713531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9740aaa60 0x7fa97413f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.497+0000 7fa972ffd640 1 -- 172.21.15.1:0/3381713531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa97414d600 msgr2=0x7fa97414ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.497+0000 7fa972ffd640 1 --2- 172.21.15.1:0/3381713531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa97414d600 0x7fa97414ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.497+0000 7fa972ffd640 1 -- 172.21.15.1:0/3381713531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa974159130 con 0x7fa974158800 2024-03-06T07:28:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.497+0000 7fa9737fe640 1 --2- 172.21.15.1:0/3381713531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9740aaa60 0x7fa97413f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.497+0000 7fa9727fc640 1 --2- 172.21.15.1:0/3381713531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa97414d600 0x7fa97414ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:35.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.497+0000 7fa972ffd640 1 --2- 172.21.15.1:0/3381713531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa974158800 0x7fa974152e00 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7fa964011e60 tx=0x7fa96400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.497+0000 7fa96b7fe640 1 -- 172.21.15.1:0/3381713531 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa964015630 con 0x7fa974158800 2024-03-06T07:28:35.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.497+0000 7fa96b7fe640 1 -- 172.21.15.1:0/3381713531 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9640157d0 con 0x7fa974158800 2024-03-06T07:28:35.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.497+0000 7fa9794ed640 1 -- 172.21.15.1:0/3381713531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9740abd00 con 0x7fa974158800 2024-03-06T07:28:35.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.497+0000 7fa96b7fe640 1 -- 172.21.15.1:0/3381713531 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa964015ba0 con 0x7fa974158800 2024-03-06T07:28:35.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.497+0000 7fa9794ed640 1 -- 172.21.15.1:0/3381713531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9741546f0 con 0x7fa974158800 2024-03-06T07:28:35.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.498+0000 7fa9794ed640 1 -- 172.21.15.1:0/3381713531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa974067fd0 con 0x7fa974158800 2024-03-06T07:28:35.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.499+0000 7fa96b7fe640 1 -- 172.21.15.1:0/3381713531 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa96401d400 con 0x7fa974158800 2024-03-06T07:28:35.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.499+0000 7fa96b7fe640 1 --2- 172.21.15.1:0/3381713531 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa95406b4d0 0x7fa95406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.499+0000 7fa96b7fe640 1 -- 172.21.15.1:0/3381713531 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa96408b590 con 0x7fa974158800 2024-03-06T07:28:35.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.499+0000 7fa9727fc640 1 --2- 172.21.15.1:0/3381713531 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa95406b4d0 0x7fa95406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.500+0000 7fa9727fc640 1 --2- 172.21.15.1:0/3381713531 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa95406b4d0 0x7fa95406d990 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fa974158540 tx=0x7fa95c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.503+0000 7fa96b7fe640 1 -- 172.21.15.1:0/3381713531 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa974159130 con 0x7fa974158800 2024-03-06T07:28:35.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.684+0000 7fa9794ed640 1 -- 172.21.15.1:0/3381713531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9740ac750 con 0x7fa974158800 2024-03-06T07:28:35.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.685+0000 7fa96b7fe640 1 -- 172.21.15.1:0/3381713531 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+16845 (secure 0 0 0) 0x7fa964019030 con 0x7fa974158800 2024-03-06T07:28:35.687 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:35.687 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":196,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:28:34.395152+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:32.660232+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:33.362968+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:35.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.687+0000 7fa9794ed640 1 -- 172.21.15.1:0/3381713531 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa95406b4d0 msgr2=0x7fa95406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.687+0000 7fa9794ed640 1 --2- 172.21.15.1:0/3381713531 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa95406b4d0 0x7fa95406d990 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fa974158540 tx=0x7fa95c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:35.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.687+0000 7fa9794ed640 1 -- 172.21.15.1:0/3381713531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa974158800 msgr2=0x7fa974152e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.687+0000 7fa9794ed640 1 --2- 172.21.15.1:0/3381713531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa974158800 0x7fa974152e00 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7fa964011e60 tx=0x7fa96400f580 comp rx=0 tx=0).stop 2024-03-06T07:28:35.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.687+0000 7fa9794ed640 1 -- 172.21.15.1:0/3381713531 shutdown_connections 2024-03-06T07:28:35.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.687+0000 7fa9794ed640 1 --2- 172.21.15.1:0/3381713531 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa95406b4d0 0x7fa95406d990 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.687+0000 7fa9794ed640 1 --2- 172.21.15.1:0/3381713531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9740aaa60 0x7fa97413f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.687+0000 7fa9794ed640 1 --2- 172.21.15.1:0/3381713531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa97414d600 0x7fa97414ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.687+0000 7fa9794ed640 1 --2- 172.21.15.1:0/3381713531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa974158800 0x7fa974152e00 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.687+0000 7fa9794ed640 1 -- 172.21.15.1:0/3381713531 >> 172.21.15.1:0/3381713531 conn(0x7fa9740c3cc0 msgr2=0x7fa97414a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.687+0000 7fa9794ed640 1 -- 172.21.15.1:0/3381713531 shutdown_connections 2024-03-06T07:28:35.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.687+0000 7fa9794ed640 1 -- 172.21.15.1:0/3381713531 wait complete. 2024-03-06T07:28:35.700 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:28:35.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.802+0000 7f420b088640 1 Processor -- start 2024-03-06T07:28:35.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.802+0000 7f420b088640 1 -- start start 2024-03-06T07:28:35.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f420b088640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42040c3260 0x7f42040c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f420b088640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4204155b70 0x7f4204159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f420b088640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42040c2b90 con 0x7f42040c8cd0 2024-03-06T07:28:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f420b088640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4204156020 con 0x7f4204155b70 2024-03-06T07:28:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f420b088640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42040c37b0 con 0x7f42040c3260 2024-03-06T07:28:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f4208e24640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f42040c8cd0 0x7f42040c7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45196/0 (socket says 172.21.15.1:45196) 2024-03-06T07:28:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f4208e24640 1 -- 172.21.15.1:0/3495788146 learned_addr learned my addr 172.21.15.1:0/3495788146 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f42037fe640 1 -- 172.21.15.1:0/3495788146 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090617744 0 0) 0x7f42040c2b90 con 0x7f42040c8cd0 2024-03-06T07:28:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f42037fe640 1 -- 172.21.15.1:0/3495788146 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ec003520 con 0x7f42040c8cd0 2024-03-06T07:28:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f4209625640 1 --2- 172.21.15.1:0/3495788146 >> v2:172.21.15.98:3300/0 conn(0x7f4204155b70 0x7f4204159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f4203fff640 1 --2- 172.21.15.1:0/3495788146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42040c3260 0x7f42040c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f42037fe640 1 -- 172.21.15.1:0/3495788146 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3144520054 0 0) 0x7f41ec003520 con 0x7f42040c8cd0 2024-03-06T07:28:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f42037fe640 1 -- 172.21.15.1:0/3495788146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42040c3260 msgr2=0x7f42040c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f42037fe640 1 --2- 172.21.15.1:0/3495788146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42040c3260 0x7f42040c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f42037fe640 1 -- 172.21.15.1:0/3495788146 >> v2:172.21.15.98:3300/0 conn(0x7f4204155b70 msgr2=0x7f4204159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f42037fe640 1 --2- 172.21.15.1:0/3495788146 >> v2:172.21.15.98:3300/0 conn(0x7f4204155b70 0x7f4204159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f42037fe640 1 -- 172.21.15.1:0/3495788146 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42040c3a90 con 0x7f42040c8cd0 2024-03-06T07:28:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f42037fe640 1 -- 172.21.15.1:0/3495788146 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f41fc0019f0 con 0x7f42040c8cd0 2024-03-06T07:28:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f42037fe640 1 -- 172.21.15.1:0/3495788146 >> v1:172.21.15.1:6789/0 conn(0x7f42040c8cd0 legacy=0x7f42040c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.803+0000 7f42037fe640 1 --2- 172.21.15.1:0/3495788146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41ec0039b0 0x7f41ec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.804+0000 7f42037fe640 1 --2- 172.21.15.1:0/3495788146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ec004800 0x7f41ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.804+0000 7f42037fe640 1 --2- 172.21.15.1:0/3495788146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42040c3260 0x7f41ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.804+0000 7f42037fe640 1 -- 172.21.15.1:0/3495788146 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4204156020 con 0x7f41ec0039b0 2024-03-06T07:28:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.804+0000 7f42037fe640 1 -- 172.21.15.1:0/3495788146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42040c37b0 con 0x7f41ec004800 2024-03-06T07:28:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.804+0000 7f42037fe640 1 -- 172.21.15.1:0/3495788146 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec00ba70 con 0x7f42040c3260 2024-03-06T07:28:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.804+0000 7f4208e24640 1 --2- 172.21.15.1:0/3495788146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41ec0039b0 0x7f41ec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.804+0000 7f4203fff640 1 --2- 172.21.15.1:0/3495788146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ec004800 0x7f41ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.804+0000 7f4209625640 1 --2- 172.21.15.1:0/3495788146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42040c3260 0x7f41ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.804+0000 7f4208e24640 1 -- 172.21.15.1:0/3495788146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42040c3260 msgr2=0x7f41ec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.804+0000 7f4208e24640 1 --2- 172.21.15.1:0/3495788146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42040c3260 0x7f41ec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.804+0000 7f4208e24640 1 -- 172.21.15.1:0/3495788146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ec004800 msgr2=0x7f41ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.804+0000 7f4208e24640 1 --2- 172.21.15.1:0/3495788146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ec004800 0x7f41ec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.804+0000 7f4208e24640 1 -- 172.21.15.1:0/3495788146 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ec003520 con 0x7f41ec0039b0 2024-03-06T07:28:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.804+0000 7f4209625640 1 --2- 172.21.15.1:0/3495788146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42040c3260 0x7f41ec0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.805+0000 7f4208e24640 1 --2- 172.21.15.1:0/3495788146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41ec0039b0 0x7f41ec003dd0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f41fc00a8f0 tx=0x7f41ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5687f438993f26 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.805+0000 7f42037fe640 1 -- 172.21.15.1:0/3495788146 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc00a4c0 con 0x7f41ec0039b0 2024-03-06T07:28:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.805+0000 7f42037fe640 1 -- 172.21.15.1:0/3495788146 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41fc034650 con 0x7f41ec0039b0 2024-03-06T07:28:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.805+0000 7f42037fe640 1 -- 172.21.15.1:0/3495788146 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc034a40 con 0x7f41ec0039b0 2024-03-06T07:28:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.805+0000 7f420b088640 1 -- 172.21.15.1:0/3495788146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41ec0039b0 msgr2=0x7f41ec003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.805+0000 7f420b088640 1 --2- 172.21.15.1:0/3495788146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41ec0039b0 0x7f41ec003dd0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f41fc00a8f0 tx=0x7f41ec005830 comp rx=0 tx=0).stop 2024-03-06T07:28:35.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.805+0000 7f4209625640 1 -- 172.21.15.1:0/3495788146 reap_dead start 2024-03-06T07:28:35.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.805+0000 7f420b088640 1 -- 172.21.15.1:0/3495788146 shutdown_connections 2024-03-06T07:28:35.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.805+0000 7f420b088640 1 -- 172.21.15.1:0/3495788146 >> 172.21.15.1:0/3495788146 conn(0x7f420406b3e0 msgr2=0x7f420406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.805+0000 7f420b088640 1 -- 172.21.15.1:0/3495788146 shutdown_connections 2024-03-06T07:28:35.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.805+0000 7f420b088640 1 -- 172.21.15.1:0/3495788146 wait complete. 2024-03-06T07:28:35.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.806+0000 7f420b088640 1 Processor -- start 2024-03-06T07:28:35.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.807+0000 7f420b088640 1 -- start start 2024-03-06T07:28:35.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.807+0000 7f420b088640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ec0039b0 0x7f42040a8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.807+0000 7f420b088640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4204154bf0 0x7f420414bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.807+0000 7f420b088640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42040a9b20 0x7f420414c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.807+0000 7f420b088640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4204159550 con 0x7f42040a9b20 2024-03-06T07:28:35.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.807+0000 7f420b088640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4204158af0 con 0x7f41ec0039b0 2024-03-06T07:28:35.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.807+0000 7f420b088640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42040c8560 con 0x7f4204154bf0 2024-03-06T07:28:35.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.807+0000 7f4209625640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42040a9b20 0x7f420414c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.807+0000 7f4209625640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42040a9b20 0x7f420414c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54448/0 (socket says 172.21.15.1:54448) 2024-03-06T07:28:35.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.807+0000 7f4209625640 1 -- 172.21.15.1:0/2875620790 learned_addr learned my addr 172.21.15.1:0/2875620790 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:35.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.808+0000 7f4208e24640 1 --2- 172.21.15.1:0/2875620790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ec0039b0 0x7f42040a8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.808+0000 7f4203fff640 1 --2- 172.21.15.1:0/2875620790 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4204154bf0 0x7f420414bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.808+0000 7f4209625640 1 -- 172.21.15.1:0/2875620790 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4204154bf0 msgr2=0x7f420414bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.808+0000 7f4209625640 1 --2- 172.21.15.1:0/2875620790 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4204154bf0 0x7f420414bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.808+0000 7f4209625640 1 -- 172.21.15.1:0/2875620790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ec0039b0 msgr2=0x7f42040a8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.808+0000 7f4209625640 1 --2- 172.21.15.1:0/2875620790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ec0039b0 0x7f42040a8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.808+0000 7f4209625640 1 -- 172.21.15.1:0/2875620790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42041e0fb0 con 0x7f42040a9b20 2024-03-06T07:28:35.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.808+0000 7f4209625640 1 --2- 172.21.15.1:0/2875620790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42040a9b20 0x7f420414c190 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f4204071a00 tx=0x7f41f4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.808+0000 7f42017fa640 1 -- 172.21.15.1:0/2875620790 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4019020 con 0x7f42040a9b20 2024-03-06T07:28:35.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.808+0000 7f42017fa640 1 -- 172.21.15.1:0/2875620790 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f4004440 con 0x7f42040a9b20 2024-03-06T07:28:35.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.808+0000 7f420b088640 1 -- 172.21.15.1:0/2875620790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f420414c6d0 con 0x7f42040a9b20 2024-03-06T07:28:35.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.808+0000 7f42017fa640 1 -- 172.21.15.1:0/2875620790 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4003850 con 0x7f42040a9b20 2024-03-06T07:28:35.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.808+0000 7f420b088640 1 -- 172.21.15.1:0/2875620790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42041e1250 con 0x7f42040a9b20 2024-03-06T07:28:35.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.809+0000 7f420b088640 1 -- 172.21.15.1:0/2875620790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42041e1650 con 0x7f42040a9b20 2024-03-06T07:28:35.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.810+0000 7f42017fa640 1 -- 172.21.15.1:0/2875620790 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f41f401c020 con 0x7f42040a9b20 2024-03-06T07:28:35.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.810+0000 7f42017fa640 1 --2- 172.21.15.1:0/2875620790 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f41f006b4d0 0x7f41f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.810+0000 7f42017fa640 1 -- 172.21.15.1:0/2875620790 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f41f408a470 con 0x7f42040a9b20 2024-03-06T07:28:35.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.810+0000 7f4208e24640 1 --2- 172.21.15.1:0/2875620790 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f41f006b4d0 0x7f41f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.811+0000 7f4208e24640 1 --2- 172.21.15.1:0/2875620790 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f41f006b4d0 0x7f41f006d990 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f41fc034000 tx=0x7f41fc009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:35.814+0000 7f42017fa640 1 -- 172.21.15.1:0/2875620790 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42041e1650 con 0x7f42040a9b20 2024-03-06T07:28:36.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.034+0000 7f420b088640 1 -- 172.21.15.1:0/2875620790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4204067fd0 con 0x7f42040a9b20 2024-03-06T07:28:36.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.047+0000 7f42017fa640 1 -- 172.21.15.1:0/2875620790 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v184) v1 ==== 118+0+0 (secure 0 0 0) 0x7f41f4023030 con 0x7f42040a9b20 2024-03-06T07:28:36.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.049+0000 7f420b088640 1 -- 172.21.15.1:0/2875620790 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f41f006b4d0 msgr2=0x7f41f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.049+0000 7f420b088640 1 --2- 172.21.15.1:0/2875620790 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f41f006b4d0 0x7f41f006d990 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f41fc034000 tx=0x7f41fc009030 comp rx=0 tx=0).stop 2024-03-06T07:28:36.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.049+0000 7f420b088640 1 -- 172.21.15.1:0/2875620790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42040a9b20 msgr2=0x7f420414c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.049+0000 7f420b088640 1 --2- 172.21.15.1:0/2875620790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42040a9b20 0x7f420414c190 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f4204071a00 tx=0x7f41f4009f90 comp rx=0 tx=0).stop 2024-03-06T07:28:36.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.049+0000 7f420b088640 1 -- 172.21.15.1:0/2875620790 shutdown_connections 2024-03-06T07:28:36.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.049+0000 7f420b088640 1 --2- 172.21.15.1:0/2875620790 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f41f006b4d0 0x7f41f006d990 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.049+0000 7f420b088640 1 --2- 172.21.15.1:0/2875620790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42040a9b20 0x7f420414c190 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.049+0000 7f420b088640 1 --2- 172.21.15.1:0/2875620790 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4204154bf0 0x7f420414bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.049+0000 7f420b088640 1 --2- 172.21.15.1:0/2875620790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41ec0039b0 0x7f42040a8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.049+0000 7f420b088640 1 -- 172.21.15.1:0/2875620790 >> 172.21.15.1:0/2875620790 conn(0x7f420406b3e0 msgr2=0x7f420406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.049+0000 7f420b088640 1 -- 172.21.15.1:0/2875620790 shutdown_connections 2024-03-06T07:28:36.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.049+0000 7f420b088640 1 -- 172.21.15.1:0/2875620790 wait complete. 2024-03-06T07:28:36.063 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:28:36.063 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:28:36.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.166+0000 7fd8c296c640 1 Processor -- start 2024-03-06T07:28:36.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.166+0000 7fd8c296c640 1 -- start start 2024-03-06T07:28:36.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.167+0000 7fd8c296c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8bc152650 0x7fd8bc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.167+0000 7fd8c296c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd8bc0c6b10 0x7fd8bc158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.167+0000 7fd8c296c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc0c2fc0 con 0x7fd8bc0bf9d0 2024-03-06T07:28:36.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.167+0000 7fd8c296c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8bc155800 con 0x7fd8bc152650 2024-03-06T07:28:36.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.167+0000 7fd8c296c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8bc0c6690 con 0x7fd8bc0c6b10 2024-03-06T07:28:36.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.167+0000 7fd8bbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8bc152650 0x7fd8bc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.167+0000 7fd8c0f09640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd8bc0bf9d0 0x7fd8bc0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45208/0 (socket says 172.21.15.1:45208) 2024-03-06T07:28:36.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.167+0000 7fd8c0f09640 1 -- 172.21.15.1:0/1694996449 learned_addr learned my addr 172.21.15.1:0/1694996449 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:36.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.168+0000 7fd8bb7fe640 1 --2- 172.21.15.1:0/1694996449 >> v2:172.21.15.98:3300/0 conn(0x7fd8bc0c6b10 0x7fd8bc158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.168+0000 7fd8bbfff640 1 -- 172.21.15.1:0/1694996449 >> v2:172.21.15.98:3300/0 conn(0x7fd8bc0c6b10 msgr2=0x7fd8bc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.168+0000 7fd8bbfff640 1 --2- 172.21.15.1:0/1694996449 >> v2:172.21.15.98:3300/0 conn(0x7fd8bc0c6b10 0x7fd8bc158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.168+0000 7fd8bbfff640 1 -- 172.21.15.1:0/1694996449 >> v1:172.21.15.1:6789/0 conn(0x7fd8bc0bf9d0 legacy=0x7fd8bc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.168+0000 7fd8bbfff640 1 -- 172.21.15.1:0/1694996449 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc0c7280 con 0x7fd8bc152650 2024-03-06T07:28:36.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.168+0000 7fd8baffd640 1 -- 172.21.15.1:0/1694996449 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030220377 0 0) 0x7fd8bc0c2fc0 con 0x7fd8bc0bf9d0 2024-03-06T07:28:36.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.168+0000 7fd8bbfff640 1 --2- 172.21.15.1:0/1694996449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc152650 0x7fd8bc155c90 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc0cb8e0 tx=0x7fd8ac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=703021ae8f27c343 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.169+0000 7fd8baffd640 1 -- 172.21.15.1:0/1694996449 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac0315a0 con 0x7fd8bc152650 2024-03-06T07:28:36.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.169+0000 7fd8baffd640 1 -- 172.21.15.1:0/1694996449 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ac037070 con 0x7fd8bc152650 2024-03-06T07:28:36.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.169+0000 7fd8baffd640 1 -- 172.21.15.1:0/1694996449 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac031960 con 0x7fd8bc152650 2024-03-06T07:28:36.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.169+0000 7fd8c296c640 1 -- 172.21.15.1:0/1694996449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc152650 msgr2=0x7fd8bc155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.169+0000 7fd8c296c640 1 --2- 172.21.15.1:0/1694996449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc152650 0x7fd8bc155c90 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc0cb8e0 tx=0x7fd8ac02b840 comp rx=0 tx=0).stop 2024-03-06T07:28:36.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.169+0000 7fd8c296c640 1 -- 172.21.15.1:0/1694996449 shutdown_connections 2024-03-06T07:28:36.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.169+0000 7fd8c296c640 1 --2- 172.21.15.1:0/1694996449 >> v2:172.21.15.98:3300/0 conn(0x7fd8bc0c6b10 0x7fd8bc158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.169+0000 7fd8c296c640 1 --2- 172.21.15.1:0/1694996449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc152650 0x7fd8bc155c90 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.169+0000 7fd8c296c640 1 -- 172.21.15.1:0/1694996449 >> 172.21.15.1:0/1694996449 conn(0x7fd8bc066ee0 msgr2=0x7fd8bc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.169+0000 7fd8c296c640 1 -- 172.21.15.1:0/1694996449 shutdown_connections 2024-03-06T07:28:36.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.169+0000 7fd8c296c640 1 -- 172.21.15.1:0/1694996449 wait complete. 2024-03-06T07:28:36.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.170+0000 7fd8c296c640 1 Processor -- start 2024-03-06T07:28:36.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.171+0000 7fd8c296c640 1 -- start start 2024-03-06T07:28:36.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.171+0000 7fd8c296c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc0bf9d0 0x7fd8bc153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.171+0000 7fd8c296c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8bc152650 0x7fd8bc1deaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.171+0000 7fd8c296c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8bc154410 0x7fd8bc1defe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.171+0000 7fd8c296c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc155490 con 0x7fd8bc0bf9d0 2024-03-06T07:28:36.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.171+0000 7fd8c296c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc1551b0 con 0x7fd8bc154410 2024-03-06T07:28:36.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.171+0000 7fd8c296c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc152a30 con 0x7fd8bc152650 2024-03-06T07:28:36.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.171+0000 7fd8bbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc0bf9d0 0x7fd8bc153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.171+0000 7fd8bbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc0bf9d0 0x7fd8bc153b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54462/0 (socket says 172.21.15.1:54462) 2024-03-06T07:28:36.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.171+0000 7fd8bbfff640 1 -- 172.21.15.1:0/2358850716 learned_addr learned my addr 172.21.15.1:0/2358850716 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:36.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.172+0000 7fd8bb7fe640 1 --2- 172.21.15.1:0/2358850716 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8bc152650 0x7fd8bc1deaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.172+0000 7fd8bbfff640 1 -- 172.21.15.1:0/2358850716 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8bc152650 msgr2=0x7fd8bc1deaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.172+0000 7fd8bbfff640 1 --2- 172.21.15.1:0/2358850716 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8bc152650 0x7fd8bc1deaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.172+0000 7fd8c0f09640 1 --2- 172.21.15.1:0/2358850716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8bc154410 0x7fd8bc1defe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.172+0000 7fd8bbfff640 1 -- 172.21.15.1:0/2358850716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8bc154410 msgr2=0x7fd8bc1defe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.172+0000 7fd8bbfff640 1 --2- 172.21.15.1:0/2358850716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8bc154410 0x7fd8bc1defe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.172+0000 7fd8bbfff640 1 -- 172.21.15.1:0/2358850716 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc154c60 con 0x7fd8bc0bf9d0 2024-03-06T07:28:36.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.172+0000 7fd8bb7fe640 1 --2- 172.21.15.1:0/2358850716 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8bc152650 0x7fd8bc1deaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:36.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.172+0000 7fd8c0f09640 1 --2- 172.21.15.1:0/2358850716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8bc154410 0x7fd8bc1defe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:36.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.172+0000 7fd8bbfff640 1 --2- 172.21.15.1:0/2358850716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc0bf9d0 0x7fd8bc153b90 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac000c00 tx=0x7fd8ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.172+0000 7fd8b8ff9640 1 -- 172.21.15.1:0/2358850716 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac004230 con 0x7fd8bc0bf9d0 2024-03-06T07:28:36.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.172+0000 7fd8b8ff9640 1 -- 172.21.15.1:0/2358850716 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ac0043d0 con 0x7fd8bc0bf9d0 2024-03-06T07:28:36.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.172+0000 7fd8c296c640 1 -- 172.21.15.1:0/2358850716 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8bc1eb8b0 con 0x7fd8bc0bf9d0 2024-03-06T07:28:36.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.172+0000 7fd8b8ff9640 1 -- 172.21.15.1:0/2358850716 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac003850 con 0x7fd8bc0bf9d0 2024-03-06T07:28:36.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.172+0000 7fd8c296c640 1 -- 172.21.15.1:0/2358850716 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8bc154960 con 0x7fd8bc0bf9d0 2024-03-06T07:28:36.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.173+0000 7fd8c296c640 1 -- 172.21.15.1:0/2358850716 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8bc1de7b0 con 0x7fd8bc0bf9d0 2024-03-06T07:28:36.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.173+0000 7fd8b8ff9640 1 -- 172.21.15.1:0/2358850716 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8ac0039f0 con 0x7fd8bc0bf9d0 2024-03-06T07:28:36.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.174+0000 7fd8b8ff9640 1 --2- 172.21.15.1:0/2358850716 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8a806b4d0 0x7fd8a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.174+0000 7fd8b8ff9640 1 -- 172.21.15.1:0/2358850716 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd8ac02f030 con 0x7fd8bc0bf9d0 2024-03-06T07:28:36.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.174+0000 7fd8bb7fe640 1 --2- 172.21.15.1:0/2358850716 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8a806b4d0 0x7fd8a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.175+0000 7fd8bb7fe640 1 --2- 172.21.15.1:0/2358850716 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8a806b4d0 0x7fd8a806d990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0006fd0 tx=0x7fd8b0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.178+0000 7fd8b8ff9640 1 -- 172.21.15.1:0/2358850716 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8ac041070 con 0x7fd8bc0bf9d0 2024-03-06T07:28:36.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:36.395+0000 7fd8c296c640 1 -- 172.21.15.1:0/2358850716 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd8bc067fd0 con 0x7fd8bc0bf9d0 2024-03-06T07:28:37.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.055+0000 7fd8b8ff9640 1 -- 172.21.15.1:0/2358850716 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v185) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd8ac07b2f0 con 0x7fd8bc0bf9d0 2024-03-06T07:28:37.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.058+0000 7fd8c296c640 1 -- 172.21.15.1:0/2358850716 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8a806b4d0 msgr2=0x7fd8a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.058+0000 7fd8c296c640 1 --2- 172.21.15.1:0/2358850716 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8a806b4d0 0x7fd8a806d990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0006fd0 tx=0x7fd8b0008040 comp rx=0 tx=0).stop 2024-03-06T07:28:37.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.058+0000 7fd8c296c640 1 -- 172.21.15.1:0/2358850716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc0bf9d0 msgr2=0x7fd8bc153b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.058+0000 7fd8c296c640 1 --2- 172.21.15.1:0/2358850716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc0bf9d0 0x7fd8bc153b90 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac000c00 tx=0x7fd8ac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.058+0000 7fd8c296c640 1 -- 172.21.15.1:0/2358850716 shutdown_connections 2024-03-06T07:28:37.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.058+0000 7fd8c296c640 1 --2- 172.21.15.1:0/2358850716 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd8a806b4d0 0x7fd8a806d990 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.058+0000 7fd8c296c640 1 --2- 172.21.15.1:0/2358850716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8bc154410 0x7fd8bc1defe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.058+0000 7fd8c296c640 1 --2- 172.21.15.1:0/2358850716 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8bc152650 0x7fd8bc1deaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.058+0000 7fd8c296c640 1 --2- 172.21.15.1:0/2358850716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc0bf9d0 0x7fd8bc153b90 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.058+0000 7fd8c296c640 1 -- 172.21.15.1:0/2358850716 >> 172.21.15.1:0/2358850716 conn(0x7fd8bc066ee0 msgr2=0x7fd8bc14e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.058+0000 7fd8c296c640 1 -- 172.21.15.1:0/2358850716 shutdown_connections 2024-03-06T07:28:37.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.058+0000 7fd8c296c640 1 -- 172.21.15.1:0/2358850716 wait complete. 2024-03-06T07:28:37.074 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:28:37.074 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:37.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.200+0000 7f829195c640 1 Processor -- start 2024-03-06T07:28:37.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.201+0000 7f829195c640 1 -- start start 2024-03-06T07:28:37.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.201+0000 7f829195c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f828c0c14c0 0x7f828c0c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.201+0000 7f829195c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f828c0c31f0 con 0x7f828c06f640 2024-03-06T07:28:37.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.201+0000 7f829195c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f828c06fbb0 con 0x7f828c0c05f0 2024-03-06T07:28:37.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.201+0000 7f829195c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f828c06fe90 con 0x7f828c0c14c0 2024-03-06T07:28:37.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.201+0000 7f828b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f828c0c14c0 0x7f828c0c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.201+0000 7f828a7fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f828c06f640 0x7f828c1558c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45220/0 (socket says 172.21.15.1:45220) 2024-03-06T07:28:37.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.201+0000 7f828a7fc640 1 -- 172.21.15.1:0/2954692486 learned_addr learned my addr 172.21.15.1:0/2954692486 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:37.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.202+0000 7f828b7fe640 1 -- 172.21.15.1:0/2954692486 >> v1:172.21.15.98:6789/0 conn(0x7f828c0c05f0 legacy=0x7f828c1551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.202+0000 7f828b7fe640 1 -- 172.21.15.1:0/2954692486 >> v1:172.21.15.1:6789/0 conn(0x7f828c06f640 legacy=0x7f828c1558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.202+0000 7f828b7fe640 1 -- 172.21.15.1:0/2954692486 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c0c3490 con 0x7f828c0c14c0 2024-03-06T07:28:37.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.202+0000 7f8289ffb640 1 -- 172.21.15.1:0/2954692486 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739270326 0 0) 0x7f828c0c31f0 con 0x7f828c06f640 2024-03-06T07:28:37.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.203+0000 7f828b7fe640 1 --2- 172.21.15.1:0/2954692486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f828c0c14c0 0x7f828c0c09d0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f827c009d90 tx=0x7f827c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef55b42689f98e43 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.203+0000 7f8289ffb640 1 -- 172.21.15.1:0/2954692486 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c039070 con 0x7f828c0c14c0 2024-03-06T07:28:37.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.203+0000 7f8289ffb640 1 -- 172.21.15.1:0/2954692486 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827c02cd00 con 0x7f828c0c14c0 2024-03-06T07:28:37.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.203+0000 7f8289ffb640 1 -- 172.21.15.1:0/2954692486 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c034660 con 0x7f828c0c14c0 2024-03-06T07:28:37.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.203+0000 7f829195c640 1 -- 172.21.15.1:0/2954692486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f828c0c14c0 msgr2=0x7f828c0c09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.203+0000 7f829195c640 1 --2- 172.21.15.1:0/2954692486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f828c0c14c0 0x7f828c0c09d0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f827c009d90 tx=0x7f827c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:28:37.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.203+0000 7f829195c640 1 -- 172.21.15.1:0/2954692486 shutdown_connections 2024-03-06T07:28:37.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.203+0000 7f829195c640 1 --2- 172.21.15.1:0/2954692486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f828c0c14c0 0x7f828c0c09d0 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.203+0000 7f829195c640 1 -- 172.21.15.1:0/2954692486 >> 172.21.15.1:0/2954692486 conn(0x7f828c0c3cc0 msgr2=0x7f828c0c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.203+0000 7f829195c640 1 -- 172.21.15.1:0/2954692486 shutdown_connections 2024-03-06T07:28:37.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.203+0000 7f829195c640 1 -- 172.21.15.1:0/2954692486 wait complete. 2024-03-06T07:28:37.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.204+0000 7f829195c640 1 Processor -- start 2024-03-06T07:28:37.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.205+0000 7f829195c640 1 -- start start 2024-03-06T07:28:37.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.205+0000 7f829195c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f828c06f640 0x7f828c157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.205+0000 7f829195c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f828c0c05f0 0x7f828c13cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.205+0000 7f829195c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f828c0c14c0 0x7f828c0aada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.205+0000 7f829195c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828c0c1040 con 0x7f828c0c14c0 2024-03-06T07:28:37.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.205+0000 7f829195c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828c14f460 con 0x7f828c0c05f0 2024-03-06T07:28:37.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.205+0000 7f829195c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f828c1609d0 con 0x7f828c06f640 2024-03-06T07:28:37.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.205+0000 7f828b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f828c0c14c0 0x7f828c0aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.205+0000 7f828b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f828c0c14c0 0x7f828c0aada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54488/0 (socket says 172.21.15.1:54488) 2024-03-06T07:28:37.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.205+0000 7f828b7fe640 1 -- 172.21.15.1:0/3301098708 learned_addr learned my addr 172.21.15.1:0/3301098708 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:37.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.205+0000 7f828a7fc640 1 --2- 172.21.15.1:0/3301098708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f828c0c05f0 0x7f828c13cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.205+0000 7f828affd640 1 --2- 172.21.15.1:0/3301098708 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f828c06f640 0x7f828c157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.206+0000 7f828b7fe640 1 -- 172.21.15.1:0/3301098708 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f828c06f640 msgr2=0x7f828c157c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.206+0000 7f828b7fe640 1 --2- 172.21.15.1:0/3301098708 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f828c06f640 0x7f828c157c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.206+0000 7f828b7fe640 1 -- 172.21.15.1:0/3301098708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f828c0c05f0 msgr2=0x7f828c13cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.206+0000 7f828b7fe640 1 --2- 172.21.15.1:0/3301098708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f828c0c05f0 0x7f828c13cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.206+0000 7f828b7fe640 1 -- 172.21.15.1:0/3301098708 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c1528b0 con 0x7f828c0c14c0 2024-03-06T07:28:37.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.206+0000 7f828b7fe640 1 --2- 172.21.15.1:0/3301098708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f828c0c14c0 0x7f828c0aada0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f827c009d90 tx=0x7f827c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.206+0000 7f8283fff640 1 -- 172.21.15.1:0/3301098708 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c039050 con 0x7f828c0c14c0 2024-03-06T07:28:37.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.206+0000 7f8283fff640 1 -- 172.21.15.1:0/3301098708 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827c03a040 con 0x7f828c0c14c0 2024-03-06T07:28:37.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.206+0000 7f829195c640 1 -- 172.21.15.1:0/3301098708 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f828c0aa350 con 0x7f828c0c14c0 2024-03-06T07:28:37.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.206+0000 7f8283fff640 1 -- 172.21.15.1:0/3301098708 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c03e610 con 0x7f828c0c14c0 2024-03-06T07:28:37.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.206+0000 7f829195c640 1 -- 172.21.15.1:0/3301098708 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f828c13d080 con 0x7f828c0c14c0 2024-03-06T07:28:37.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.207+0000 7f829195c640 1 -- 172.21.15.1:0/3301098708 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f828c067fd0 con 0x7f828c0c14c0 2024-03-06T07:28:37.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.208+0000 7f8283fff640 1 -- 172.21.15.1:0/3301098708 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f827c047030 con 0x7f828c0c14c0 2024-03-06T07:28:37.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.208+0000 7f8283fff640 1 --2- 172.21.15.1:0/3301098708 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f826c06b500 0x7f826c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.208+0000 7f8283fff640 1 -- 172.21.15.1:0/3301098708 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f827c0ac330 con 0x7f828c0c14c0 2024-03-06T07:28:37.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.208+0000 7f828affd640 1 --2- 172.21.15.1:0/3301098708 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f826c06b500 0x7f826c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.209+0000 7f828affd640 1 --2- 172.21.15.1:0/3301098708 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f826c06b500 0x7f826c06d9c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f828c1550f0 tx=0x7f8284009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.217+0000 7f8283fff640 1 -- 172.21.15.1:0/3301098708 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f827c07aa80 con 0x7f828c0c14c0 2024-03-06T07:28:37.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.436+0000 7f829195c640 1 -- 172.21.15.1:0/3301098708 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f828c0ac750 con 0x7f828c0c14c0 2024-03-06T07:28:37.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.437+0000 7f8283fff640 1 -- 172.21.15.1:0/3301098708 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+4983 (secure 0 0 0) 0x7f827c07eb20 con 0x7f828c0c14c0 2024-03-06T07:28:37.439 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:37.439 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":185,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7961,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2819896202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2819896202},{"type":"v1","addr":"172.21.15.98:6837","nonce":2819896202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":8959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/646952744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":646952744},{"type":"v1","addr":"172.21.15.1:6835","nonce":646952744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":8965,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2843229007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2843229007},{"type":"v1","addr":"172.21.15.98:6836","nonce":2843229007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:34.395524+0000","modified":"2024-03-06T07:28:36.397164+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10425},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10425":{"gid":10425,"name":"a","rank":0,"incarnation":183,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1315978401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1315978401},{"type":"v1","addr":"172.21.15.1:6834","nonce":1315978401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10425,"qdb_cluster":[10425]},"id":15}]} 2024-03-06T07:28:37.439 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 185 2024-03-06T07:28:37.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.439+0000 7f829195c640 1 -- 172.21.15.1:0/3301098708 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f826c06b500 msgr2=0x7f826c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.439+0000 7f829195c640 1 --2- 172.21.15.1:0/3301098708 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f826c06b500 0x7f826c06d9c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f828c1550f0 tx=0x7f8284009040 comp rx=0 tx=0).stop 2024-03-06T07:28:37.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.439+0000 7f829195c640 1 -- 172.21.15.1:0/3301098708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f828c0c14c0 msgr2=0x7f828c0aada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.439+0000 7f829195c640 1 --2- 172.21.15.1:0/3301098708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f828c0c14c0 0x7f828c0aada0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f827c009d90 tx=0x7f827c004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:37.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.439+0000 7f829195c640 1 -- 172.21.15.1:0/3301098708 shutdown_connections 2024-03-06T07:28:37.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.439+0000 7f829195c640 1 --2- 172.21.15.1:0/3301098708 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f826c06b500 0x7f826c06d9c0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.439+0000 7f829195c640 1 --2- 172.21.15.1:0/3301098708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f828c0c14c0 0x7f828c0aada0 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.439+0000 7f829195c640 1 --2- 172.21.15.1:0/3301098708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f828c0c05f0 0x7f828c13cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.439+0000 7f829195c640 1 --2- 172.21.15.1:0/3301098708 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f828c06f640 0x7f828c157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.439+0000 7f829195c640 1 -- 172.21.15.1:0/3301098708 >> 172.21.15.1:0/3301098708 conn(0x7f828c0c3cc0 msgr2=0x7f828c06df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.439+0000 7f829195c640 1 -- 172.21.15.1:0/3301098708 shutdown_connections 2024-03-06T07:28:37.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.439+0000 7f829195c640 1 -- 172.21.15.1:0/3301098708 wait complete. 2024-03-06T07:28:37.452 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:37.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.555+0000 7fe436b3e640 1 Processor -- start 2024-03-06T07:28:37.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.555+0000 7fe436b3e640 1 -- start start 2024-03-06T07:28:37.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.555+0000 7fe436b3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe430152650 0x7fe430155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.555+0000 7fe436b3e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe4300c6b10 0x7fe430158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.555+0000 7fe436b3e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe4300bf9d0 0x7fe4300c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.555+0000 7fe436b3e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe430155800 con 0x7fe4300c6b10 2024-03-06T07:28:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.555+0000 7fe436b3e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4300c2fc0 con 0x7fe4300bf9d0 2024-03-06T07:28:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.555+0000 7fe436b3e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300c6690 con 0x7fe430152650 2024-03-06T07:28:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.555+0000 7fe42ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe4300c6b10 0x7fe430158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.556+0000 7fe42ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe4300c6b10 0x7fe430158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54502/0 (socket says 172.21.15.1:54502) 2024-03-06T07:28:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.556+0000 7fe42ffff640 1 -- 172.21.15.1:0/4070187549 learned_addr learned my addr 172.21.15.1:0/4070187549 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.556+0000 7fe4350db640 1 --2- 172.21.15.1:0/4070187549 >> v2:172.21.15.98:3300/0 conn(0x7fe4300bf9d0 0x7fe4300c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.556+0000 7fe4348da640 1 --2- 172.21.15.1:0/4070187549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe430152650 0x7fe430155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.556+0000 7fe42ffff640 1 -- 172.21.15.1:0/4070187549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe430152650 msgr2=0x7fe430155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.556+0000 7fe42ffff640 1 --2- 172.21.15.1:0/4070187549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe430152650 0x7fe430155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.556+0000 7fe42ffff640 1 -- 172.21.15.1:0/4070187549 >> v2:172.21.15.98:3300/0 conn(0x7fe4300bf9d0 msgr2=0x7fe4300c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.556+0000 7fe42ffff640 1 --2- 172.21.15.1:0/4070187549 >> v2:172.21.15.98:3300/0 conn(0x7fe4300bf9d0 0x7fe4300c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.556+0000 7fe42ffff640 1 -- 172.21.15.1:0/4070187549 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4300c7280 con 0x7fe4300c6b10 2024-03-06T07:28:37.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.556+0000 7fe4350db640 1 --2- 172.21.15.1:0/4070187549 >> v2:172.21.15.98:3300/0 conn(0x7fe4300bf9d0 0x7fe4300c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:37.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.556+0000 7fe4348da640 1 --2- 172.21.15.1:0/4070187549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe430152650 0x7fe430155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:37.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.557+0000 7fe42ffff640 1 --2- 172.21.15.1:0/4070187549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4300c6b10 0x7fe430158690 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7fe4301597b0 tx=0x7fe41c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7bd63186676e7d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.557+0000 7fe42f7fe640 1 -- 172.21.15.1:0/4070187549 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c02e070 con 0x7fe4300c6b10 2024-03-06T07:28:37.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.557+0000 7fe42f7fe640 1 -- 172.21.15.1:0/4070187549 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c037070 con 0x7fe4300c6b10 2024-03-06T07:28:37.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.557+0000 7fe42f7fe640 1 -- 172.21.15.1:0/4070187549 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c031a00 con 0x7fe4300c6b10 2024-03-06T07:28:37.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.557+0000 7fe436b3e640 1 -- 172.21.15.1:0/4070187549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4300c6b10 msgr2=0x7fe430158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.557+0000 7fe436b3e640 1 --2- 172.21.15.1:0/4070187549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4300c6b10 0x7fe430158690 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7fe4301597b0 tx=0x7fe41c02b880 comp rx=0 tx=0).stop 2024-03-06T07:28:37.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.557+0000 7fe436b3e640 1 -- 172.21.15.1:0/4070187549 shutdown_connections 2024-03-06T07:28:37.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.557+0000 7fe436b3e640 1 --2- 172.21.15.1:0/4070187549 >> v2:172.21.15.98:3300/0 conn(0x7fe4300bf9d0 0x7fe4300c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.557+0000 7fe436b3e640 1 --2- 172.21.15.1:0/4070187549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4300c6b10 0x7fe430158690 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.557+0000 7fe436b3e640 1 --2- 172.21.15.1:0/4070187549 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe430152650 0x7fe430155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.557+0000 7fe436b3e640 1 -- 172.21.15.1:0/4070187549 >> 172.21.15.1:0/4070187549 conn(0x7fe430066ee0 msgr2=0x7fe4300672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.557+0000 7fe436b3e640 1 -- 172.21.15.1:0/4070187549 shutdown_connections 2024-03-06T07:28:37.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.557+0000 7fe436b3e640 1 -- 172.21.15.1:0/4070187549 wait complete. 2024-03-06T07:28:37.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.558+0000 7fe436b3e640 1 Processor -- start 2024-03-06T07:28:37.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.558+0000 7fe436b3e640 1 -- start start 2024-03-06T07:28:37.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.559+0000 7fe436b3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4300bf9d0 0x7fe430156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.559+0000 7fe436b3e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe430152650 0x7fe43014d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.559+0000 7fe436b3e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe43014de40 0x7fe430142000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.559+0000 7fe436b3e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe430155490 con 0x7fe430152650 2024-03-06T07:28:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.559+0000 7fe436b3e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4301551b0 con 0x7fe43014de40 2024-03-06T07:28:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.559+0000 7fe436b3e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe430152a30 con 0x7fe4300bf9d0 2024-03-06T07:28:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.559+0000 7fe42ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe430152650 0x7fe43014d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.559+0000 7fe42ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe430152650 0x7fe43014d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54506/0 (socket says 172.21.15.1:54506) 2024-03-06T07:28:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.559+0000 7fe42ffff640 1 -- 172.21.15.1:0/2291623481 learned_addr learned my addr 172.21.15.1:0/2291623481 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.560+0000 7fe4348da640 1 --2- 172.21.15.1:0/2291623481 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4300bf9d0 0x7fe430156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.560+0000 7fe4350db640 1 --2- 172.21.15.1:0/2291623481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe43014de40 0x7fe430142000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.560+0000 7fe42ffff640 1 -- 172.21.15.1:0/2291623481 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4300bf9d0 msgr2=0x7fe430156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.560+0000 7fe42ffff640 1 --2- 172.21.15.1:0/2291623481 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4300bf9d0 0x7fe430156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.560+0000 7fe42ffff640 1 -- 172.21.15.1:0/2291623481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe43014de40 msgr2=0x7fe430142000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.560+0000 7fe42ffff640 1 --2- 172.21.15.1:0/2291623481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe43014de40 0x7fe430142000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.560+0000 7fe42ffff640 1 -- 172.21.15.1:0/2291623481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4301ebc70 con 0x7fe430152650 2024-03-06T07:28:37.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.560+0000 7fe42ffff640 1 --2- 172.21.15.1:0/2291623481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe430152650 0x7fe43014d900 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7fe41c000c00 tx=0x7fe41c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.560+0000 7fe42d7fa640 1 -- 172.21.15.1:0/2291623481 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c032670 con 0x7fe430152650 2024-03-06T07:28:37.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.560+0000 7fe42d7fa640 1 -- 172.21.15.1:0/2291623481 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c032850 con 0x7fe430152650 2024-03-06T07:28:37.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.560+0000 7fe42d7fa640 1 -- 172.21.15.1:0/2291623481 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c032c40 con 0x7fe430152650 2024-03-06T07:28:37.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.560+0000 7fe436b3e640 1 -- 172.21.15.1:0/2291623481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4301eb0f0 con 0x7fe430152650 2024-03-06T07:28:37.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.560+0000 7fe436b3e640 1 -- 172.21.15.1:0/2291623481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4301eb6b0 con 0x7fe430152650 2024-03-06T07:28:37.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.561+0000 7fe436b3e640 1 -- 172.21.15.1:0/2291623481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4300676d0 con 0x7fe430152650 2024-03-06T07:28:37.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.562+0000 7fe42d7fa640 1 -- 172.21.15.1:0/2291623481 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe41c04c060 con 0x7fe430152650 2024-03-06T07:28:37.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.562+0000 7fe42d7fa640 1 --2- 172.21.15.1:0/2291623481 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe41406b4d0 0x7fe41406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.562+0000 7fe42d7fa640 1 -- 172.21.15.1:0/2291623481 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe41c0abd60 con 0x7fe430152650 2024-03-06T07:28:37.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.563+0000 7fe4348da640 1 --2- 172.21.15.1:0/2291623481 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe41406b4d0 0x7fe41406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.563+0000 7fe4348da640 1 --2- 172.21.15.1:0/2291623481 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe41406b4d0 0x7fe41406d990 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fe424006fd0 tx=0x7fe424008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.566+0000 7fe42d7fa640 1 -- 172.21.15.1:0/2291623481 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe4300676d0 con 0x7fe430152650 2024-03-06T07:28:37.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.747+0000 7fe436b3e640 1 -- 172.21.15.1:0/2291623481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe430067fd0 con 0x7fe430152650 2024-03-06T07:28:37.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.748+0000 7fe42d7fa640 1 -- 172.21.15.1:0/2291623481 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+16832 (secure 0 0 0) 0x7fe41c040070 con 0x7fe430152650 2024-03-06T07:28:37.751 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:37.751 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":197,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:28:36.047509+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:32.660232+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:33.362968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:37.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.750+0000 7fe436b3e640 1 -- 172.21.15.1:0/2291623481 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe41406b4d0 msgr2=0x7fe41406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.750+0000 7fe436b3e640 1 --2- 172.21.15.1:0/2291623481 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe41406b4d0 0x7fe41406d990 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fe424006fd0 tx=0x7fe424008040 comp rx=0 tx=0).stop 2024-03-06T07:28:37.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.750+0000 7fe436b3e640 1 -- 172.21.15.1:0/2291623481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe430152650 msgr2=0x7fe43014d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.750+0000 7fe436b3e640 1 --2- 172.21.15.1:0/2291623481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe430152650 0x7fe43014d900 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7fe41c000c00 tx=0x7fe41c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.751+0000 7fe436b3e640 1 -- 172.21.15.1:0/2291623481 shutdown_connections 2024-03-06T07:28:37.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.751+0000 7fe436b3e640 1 --2- 172.21.15.1:0/2291623481 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe41406b4d0 0x7fe41406d990 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.751+0000 7fe436b3e640 1 --2- 172.21.15.1:0/2291623481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe43014de40 0x7fe430142000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.751+0000 7fe436b3e640 1 --2- 172.21.15.1:0/2291623481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe430152650 0x7fe43014d900 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.751+0000 7fe436b3e640 1 --2- 172.21.15.1:0/2291623481 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4300bf9d0 0x7fe430156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.751+0000 7fe436b3e640 1 -- 172.21.15.1:0/2291623481 >> 172.21.15.1:0/2291623481 conn(0x7fe430066ee0 msgr2=0x7fe4300cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.751+0000 7fe436b3e640 1 -- 172.21.15.1:0/2291623481 shutdown_connections 2024-03-06T07:28:37.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.751+0000 7fe436b3e640 1 -- 172.21.15.1:0/2291623481 wait complete. 2024-03-06T07:28:37.764 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:28:37.764 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.863+0000 7f824fb0c640 1 Processor -- start 2024-03-06T07:28:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.863+0000 7f824fb0c640 1 -- start start 2024-03-06T07:28:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.864+0000 7f824fb0c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8248150520 0x7f82481543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.864+0000 7f824fb0c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f82480c32f0 0x7f82481537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.864+0000 7f824fb0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8248070c80 0x7f82480710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.864+0000 7f824fb0c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8248150900 con 0x7f8248150520 2024-03-06T07:28:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.864+0000 7f824fb0c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8248071790 con 0x7f82480c32f0 2024-03-06T07:28:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.864+0000 7f824fb0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8248157050 con 0x7f8248070c80 2024-03-06T07:28:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.864+0000 7f824d8a8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8248150520 0x7f82481543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.864+0000 7f824d8a8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8248150520 0x7f82481543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54512/0 (socket says 172.21.15.1:54512) 2024-03-06T07:28:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.864+0000 7f824d8a8640 1 -- 172.21.15.1:0/3390192333 learned_addr learned my addr 172.21.15.1:0/3390192333 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.864+0000 7f824e0a9640 1 --2- 172.21.15.1:0/3390192333 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8248070c80 0x7f82480710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.864+0000 7f824d0a7640 1 --2- 172.21.15.1:0/3390192333 >> v2:172.21.15.98:3300/0 conn(0x7f82480c32f0 0x7f82481537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.864+0000 7f824d8a8640 1 -- 172.21.15.1:0/3390192333 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8248070c80 msgr2=0x7f82480710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.864+0000 7f824d8a8640 1 --2- 172.21.15.1:0/3390192333 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8248070c80 0x7f82480710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.864+0000 7f824d8a8640 1 -- 172.21.15.1:0/3390192333 >> v2:172.21.15.98:3300/0 conn(0x7f82480c32f0 msgr2=0x7f82481537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.864+0000 7f824d8a8640 1 --2- 172.21.15.1:0/3390192333 >> v2:172.21.15.98:3300/0 conn(0x7f82480c32f0 0x7f82481537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.864+0000 7f824d8a8640 1 -- 172.21.15.1:0/3390192333 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8248152fc0 con 0x7f8248150520 2024-03-06T07:28:37.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.865+0000 7f824d8a8640 1 --2- 172.21.15.1:0/3390192333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8248150520 0x7f82481543a0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f82480732a0 tx=0x7f824402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a436b10674683503 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.865+0000 7f824c8a6640 1 -- 172.21.15.1:0/3390192333 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244035020 con 0x7f8248150520 2024-03-06T07:28:37.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.865+0000 7f824c8a6640 1 -- 172.21.15.1:0/3390192333 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8244031870 con 0x7f8248150520 2024-03-06T07:28:37.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.865+0000 7f824c8a6640 1 -- 172.21.15.1:0/3390192333 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244031c40 con 0x7f8248150520 2024-03-06T07:28:37.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.866+0000 7f824fb0c640 1 -- 172.21.15.1:0/3390192333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8248150520 msgr2=0x7f82481543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.866+0000 7f824fb0c640 1 --2- 172.21.15.1:0/3390192333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8248150520 0x7f82481543a0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f82480732a0 tx=0x7f824402b910 comp rx=0 tx=0).stop 2024-03-06T07:28:37.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.866+0000 7f824fb0c640 1 -- 172.21.15.1:0/3390192333 shutdown_connections 2024-03-06T07:28:37.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.866+0000 7f824fb0c640 1 --2- 172.21.15.1:0/3390192333 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8248070c80 0x7f82480710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.866+0000 7f824fb0c640 1 --2- 172.21.15.1:0/3390192333 >> v2:172.21.15.98:3300/0 conn(0x7f82480c32f0 0x7f82481537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.866+0000 7f824fb0c640 1 --2- 172.21.15.1:0/3390192333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8248150520 0x7f82481543a0 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.866+0000 7f824fb0c640 1 -- 172.21.15.1:0/3390192333 >> 172.21.15.1:0/3390192333 conn(0x7f82480c88f0 msgr2=0x7f82480c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.866+0000 7f824fb0c640 1 -- 172.21.15.1:0/3390192333 shutdown_connections 2024-03-06T07:28:37.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.866+0000 7f824fb0c640 1 -- 172.21.15.1:0/3390192333 wait complete. 2024-03-06T07:28:37.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.867+0000 7f824fb0c640 1 Processor -- start 2024-03-06T07:28:37.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.867+0000 7f824fb0c640 1 -- start start 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824fb0c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8248070c80 0x7f82481ee8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824fb0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82480c32f0 0x7f82480aae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824fb0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8248150520 0x7f82480c1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824fb0c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82480c7ec0 con 0x7f8248070c80 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824fb0c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8248157330 con 0x7f82480c32f0 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824fb0c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8248153d00 con 0x7f8248150520 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824d8a8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8248070c80 0x7f82481ee8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824d8a8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8248070c80 0x7f82481ee8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54522/0 (socket says 172.21.15.1:54522) 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824d8a8640 1 -- 172.21.15.1:0/3343241837 learned_addr learned my addr 172.21.15.1:0/3343241837 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824e0a9640 1 --2- 172.21.15.1:0/3343241837 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8248150520 0x7f82480c1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824d0a7640 1 --2- 172.21.15.1:0/3343241837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82480c32f0 0x7f82480aae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824d8a8640 1 -- 172.21.15.1:0/3343241837 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8248150520 msgr2=0x7f82480c1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824d8a8640 1 --2- 172.21.15.1:0/3343241837 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8248150520 0x7f82480c1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824d8a8640 1 -- 172.21.15.1:0/3343241837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82480c32f0 msgr2=0x7f82480aae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824d8a8640 1 --2- 172.21.15.1:0/3343241837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82480c32f0 0x7f82480aae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824d8a8640 1 -- 172.21.15.1:0/3343241837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824814a690 con 0x7f8248070c80 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824d8a8640 1 --2- 172.21.15.1:0/3343241837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8248070c80 0x7f82481ee8c0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f824402b950 tx=0x7f82440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f823e7fc640 1 -- 172.21.15.1:0/3343241837 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244040070 con 0x7f8248070c80 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f823e7fc640 1 -- 172.21.15.1:0/3343241837 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8244030d00 con 0x7f8248070c80 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824fb0c640 1 -- 172.21.15.1:0/3343241837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82481462e0 con 0x7f8248070c80 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f823e7fc640 1 -- 172.21.15.1:0/3343241837 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244035040 con 0x7f8248070c80 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.868+0000 7f824fb0c640 1 -- 172.21.15.1:0/3343241837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8248146800 con 0x7f8248070c80 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.869+0000 7f824fb0c640 1 -- 172.21.15.1:0/3343241837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82480c64e0 con 0x7f8248070c80 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.870+0000 7f823e7fc640 1 -- 172.21.15.1:0/3343241837 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f824404b060 con 0x7f8248070c80 2024-03-06T07:28:37.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.871+0000 7f823e7fc640 1 --2- 172.21.15.1:0/3343241837 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f822c06b4d0 0x7f822c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.871+0000 7f823e7fc640 1 -- 172.21.15.1:0/3343241837 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f82440ac090 con 0x7f8248070c80 2024-03-06T07:28:37.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.871+0000 7f824d0a7640 1 --2- 172.21.15.1:0/3343241837 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f822c06b4d0 0x7f822c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.872+0000 7f824d0a7640 1 --2- 172.21.15.1:0/3343241837 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f822c06b4d0 0x7f822c06d990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f8248156f50 tx=0x7f8238008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:37.875+0000 7f823e7fc640 1 -- 172.21.15.1:0/3343241837 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82480c64e0 con 0x7f8248070c80 2024-03-06T07:28:38.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.085+0000 7f824fb0c640 1 -- 172.21.15.1:0/3343241837 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f82480686e0 con 0x7f822c06b4d0 2024-03-06T07:28:38.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.091+0000 7f823e7fc640 1 -- 172.21.15.1:0/3343241837 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247177 (secure 0 0 0) 0x7f82480686e0 con 0x7f822c06b4d0 2024-03-06T07:28:38.096 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:38.098 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:28:38.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.094+0000 7f824fb0c640 1 -- 172.21.15.1:0/3343241837 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f822c06b4d0 msgr2=0x7f822c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.094+0000 7f824fb0c640 1 --2- 172.21.15.1:0/3343241837 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f822c06b4d0 0x7f822c06d990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f8248156f50 tx=0x7f8238008040 comp rx=0 tx=0).stop 2024-03-06T07:28:38.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.094+0000 7f824fb0c640 1 -- 172.21.15.1:0/3343241837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8248070c80 msgr2=0x7f82481ee8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.094+0000 7f824fb0c640 1 --2- 172.21.15.1:0/3343241837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8248070c80 0x7f82481ee8c0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f824402b950 tx=0x7f82440046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.094+0000 7f824fb0c640 1 -- 172.21.15.1:0/3343241837 shutdown_connections 2024-03-06T07:28:38.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.094+0000 7f824fb0c640 1 --2- 172.21.15.1:0/3343241837 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f822c06b4d0 0x7f822c06d990 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.094+0000 7f824fb0c640 1 --2- 172.21.15.1:0/3343241837 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8248150520 0x7f82480c1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.094+0000 7f824fb0c640 1 --2- 172.21.15.1:0/3343241837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82480c32f0 0x7f82480aae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.094+0000 7f824fb0c640 1 --2- 172.21.15.1:0/3343241837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8248070c80 0x7f82481ee8c0 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.094+0000 7f824fb0c640 1 -- 172.21.15.1:0/3343241837 >> 172.21.15.1:0/3343241837 conn(0x7f82480c88f0 msgr2=0x7f824814a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.094+0000 7f824fb0c640 1 -- 172.21.15.1:0/3343241837 shutdown_connections 2024-03-06T07:28:38.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.094+0000 7f824fb0c640 1 -- 172.21.15.1:0/3343241837 wait complete. 2024-03-06T07:28:38.107 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":441,"stamp":"2024-03-06T07:28:36.807528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":277208,"kb_used_data":19420,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754697512,"statfs":{"total":773094113280,"available":772810252288,"internally_reserved":0,"allocated":19886080,"data_stored":10784343,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000971"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039773+0000","last_change":"2024-03-06T07:28:34.039773+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039773+0000","last_undegraded":"2024-03-06T07:28:34.039773+0000","last_fullsized":"2024-03-06T07:28:34.039773+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032560+0000","last_change":"2024-03-06T07:28:34.032560+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032560+0000","last_undegraded":"2024-03-06T07:28:34.032560+0000","last_fullsized":"2024-03-06T07:28:34.032560+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.043939+0000","last_change":"2024-03-06T07:28:34.043939+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.043939+0000","last_undegraded":"2024-03-06T07:28:34.043939+0000","last_fullsized":"2024-03-06T07:28:34.043939+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.040918+0000","last_change":"2024-03-06T07:28:34.040918+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.040918+0000","last_undegraded":"2024-03-06T07:28:34.040918+0000","last_fullsized":"2024-03-06T07:28:34.040918+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034672+0000","last_change":"2024-03-06T07:28:34.034672+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034672+0000","last_undegraded":"2024-03-06T07:28:34.034672+0000","last_fullsized":"2024-03-06T07:28:34.034672+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036995+0000","last_change":"2024-03-06T07:28:34.036995+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036995+0000","last_undegraded":"2024-03-06T07:28:34.036995+0000","last_fullsized":"2024-03-06T07:28:34.036995+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037384+0000","last_change":"2024-03-06T07:28:34.037384+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037384+0000","last_undegraded":"2024-03-06T07:28:34.037384+0000","last_fullsized":"2024-03-06T07:28:34.037384+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036819+0000","last_change":"2024-03-06T07:28:34.036819+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036819+0000","last_undegraded":"2024-03-06T07:28:34.036819+0000","last_fullsized":"2024-03-06T07:28:34.036819+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038840+0000","last_change":"2024-03-06T07:28:34.038840+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038840+0000","last_undegraded":"2024-03-06T07:28:34.038840+0000","last_fullsized":"2024-03-06T07:28:34.038840+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.043705+0000","last_change":"2024-03-06T07:28:34.043705+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.043705+0000","last_undegraded":"2024-03-06T07:28:34.043705+0000","last_fullsized":"2024-03-06T07:28:34.043705+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.041796+0000","last_change":"2024-03-06T07:28:34.041796+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.041796+0000","last_undegraded":"2024-03-06T07:28:34.041796+0000","last_fullsized":"2024-03-06T07:28:34.041796+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038852+0000","last_change":"2024-03-06T07:28:34.038852+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038852+0000","last_undegraded":"2024-03-06T07:28:34.038852+0000","last_fullsized":"2024-03-06T07:28:34.038852+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.035743+0000","last_change":"2024-03-06T07:28:34.035743+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.035743+0000","last_undegraded":"2024-03-06T07:28:34.035743+0000","last_fullsized":"2024-03-06T07:28:34.035743+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033521+0000","last_change":"2024-03-06T07:28:34.033521+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033521+0000","last_undegraded":"2024-03-06T07:28:34.033521+0000","last_fullsized":"2024-03-06T07:28:34.033521+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033924+0000","last_change":"2024-03-06T07:28:34.033924+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033924+0000","last_undegraded":"2024-03-06T07:28:34.033924+0000","last_fullsized":"2024-03-06T07:28:34.033924+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038810+0000","last_change":"2024-03-06T07:28:34.038810+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038810+0000","last_undegraded":"2024-03-06T07:28:34.038810+0000","last_fullsized":"2024-03-06T07:28:34.038810+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036613+0000","last_change":"2024-03-06T07:28:34.036613+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036613+0000","last_undegraded":"2024-03-06T07:28:34.036613+0000","last_fullsized":"2024-03-06T07:28:34.036613+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.043282+0000","last_change":"2024-03-06T07:28:34.043282+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.043282+0000","last_undegraded":"2024-03-06T07:28:34.043282+0000","last_fullsized":"2024-03-06T07:28:34.043282+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033099+0000","last_change":"2024-03-06T07:28:34.033099+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033099+0000","last_undegraded":"2024-03-06T07:28:34.033099+0000","last_fullsized":"2024-03-06T07:28:34.033099+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.031957+0000","last_change":"2024-03-06T07:28:34.031957+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.031957+0000","last_undegraded":"2024-03-06T07:28:34.031957+0000","last_fullsized":"2024-03-06T07:28:34.031957+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037603+0000","last_change":"2024-03-06T07:28:34.037603+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037603+0000","last_undegraded":"2024-03-06T07:28:34.037603+0000","last_fullsized":"2024-03-06T07:28:34.037603+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036930+0000","last_change":"2024-03-06T07:28:34.036930+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036930+0000","last_undegraded":"2024-03-06T07:28:34.036930+0000","last_fullsized":"2024-03-06T07:28:34.036930+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033464+0000","last_change":"2024-03-06T07:28:34.033464+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033464+0000","last_undegraded":"2024-03-06T07:28:34.033464+0000","last_fullsized":"2024-03-06T07:28:34.033464+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037784+0000","last_change":"2024-03-06T07:28:34.037784+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037784+0000","last_undegraded":"2024-03-06T07:28:34.037784+0000","last_fullsized":"2024-03-06T07:28:34.037784+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.031482+0000","last_change":"2024-03-06T07:28:34.031482+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.031482+0000","last_undegraded":"2024-03-06T07:28:34.031482+0000","last_fullsized":"2024-03-06T07:28:34.031482+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.040505+0000","last_change":"2024-03-06T07:28:34.040505+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.040505+0000","last_undegraded":"2024-03-06T07:28:34.040505+0000","last_fullsized":"2024-03-06T07:28:34.040505+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034359+0000","last_change":"2024-03-06T07:28:34.034359+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034359+0000","last_undegraded":"2024-03-06T07:28:34.034359+0000","last_fullsized":"2024-03-06T07:28:34.034359+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038455+0000","last_change":"2024-03-06T07:28:34.038455+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038455+0000","last_undegraded":"2024-03-06T07:28:34.038455+0000","last_fullsized":"2024-03-06T07:28:34.038455+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032251+0000","last_change":"2024-03-06T07:28:34.032251+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032251+0000","last_undegraded":"2024-03-06T07:28:34.032251+0000","last_fullsized":"2024-03-06T07:28:34.032251+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032247+0000","last_change":"2024-03-06T07:28:34.032247+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032247+0000","last_undegraded":"2024-03-06T07:28:34.032247+0000","last_fullsized":"2024-03-06T07:28:34.032247+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.031492+0000","last_change":"2024-03-06T07:28:34.031492+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.031492+0000","last_undegraded":"2024-03-06T07:28:34.031492+0000","last_fullsized":"2024-03-06T07:28:34.031492+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036448+0000","last_change":"2024-03-06T07:28:34.036448+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036448+0000","last_undegraded":"2024-03-06T07:28:34.036448+0000","last_fullsized":"2024-03-06T07:28:34.036448+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032461+0000","last_change":"2024-03-06T07:28:34.032461+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032461+0000","last_undegraded":"2024-03-06T07:28:34.032461+0000","last_fullsized":"2024-03-06T07:28:34.032461+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.046617+0000","last_change":"2024-03-06T07:28:34.045692+0000","last_active":"2024-03-06T07:28:34.046617+0000","last_peered":"2024-03-06T07:28:34.046617+0000","last_clean":"2024-03-06T07:28:34.046617+0000","last_became_active":"2024-03-06T07:28:34.045392+0000","last_became_peered":"2024-03-06T07:28:34.045392+0000","last_unstale":"2024-03-06T07:28:34.046617+0000","last_undegraded":"2024-03-06T07:28:34.046617+0000","last_fullsized":"2024-03-06T07:28:34.046617+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:59.770341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032614+0000","last_change":"2024-03-06T07:28:34.032614+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032614+0000","last_undegraded":"2024-03-06T07:28:34.032614+0000","last_fullsized":"2024-03-06T07:28:34.032614+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053547+0000","last_change":"2024-03-06T07:28:34.048342+0000","last_active":"2024-03-06T07:28:34.053547+0000","last_peered":"2024-03-06T07:28:34.053547+0000","last_clean":"2024-03-06T07:28:34.053547+0000","last_became_active":"2024-03-06T07:28:34.048077+0000","last_became_peered":"2024-03-06T07:28:34.048077+0000","last_unstale":"2024-03-06T07:28:34.053547+0000","last_undegraded":"2024-03-06T07:28:34.053547+0000","last_fullsized":"2024-03-06T07:28:34.053547+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:02.569692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038496+0000","last_change":"2024-03-06T07:28:34.038496+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038496+0000","last_undegraded":"2024-03-06T07:28:34.038496+0000","last_fullsized":"2024-03-06T07:28:34.038496+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.044121+0000","last_change":"2024-03-06T07:28:34.043250+0000","last_active":"2024-03-06T07:28:34.044121+0000","last_peered":"2024-03-06T07:28:34.044121+0000","last_clean":"2024-03-06T07:28:34.044121+0000","last_became_active":"2024-03-06T07:28:34.042985+0000","last_became_peered":"2024-03-06T07:28:34.042985+0000","last_unstale":"2024-03-06T07:28:34.044121+0000","last_undegraded":"2024-03-06T07:28:34.044121+0000","last_fullsized":"2024-03-06T07:28:34.044121+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:49.778788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032526+0000","last_change":"2024-03-06T07:28:34.032526+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032526+0000","last_undegraded":"2024-03-06T07:28:34.032526+0000","last_fullsized":"2024-03-06T07:28:34.032526+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.040595+0000","last_change":"2024-03-06T07:28:34.039018+0000","last_active":"2024-03-06T07:28:34.040595+0000","last_peered":"2024-03-06T07:28:34.040595+0000","last_clean":"2024-03-06T07:28:34.040595+0000","last_became_active":"2024-03-06T07:28:34.038734+0000","last_became_peered":"2024-03-06T07:28:34.038734+0000","last_unstale":"2024-03-06T07:28:34.040595+0000","last_undegraded":"2024-03-06T07:28:34.040595+0000","last_fullsized":"2024-03-06T07:28:34.040595+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:56.463129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039197+0000","last_change":"2024-03-06T07:28:34.039197+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039197+0000","last_undegraded":"2024-03-06T07:28:34.039197+0000","last_fullsized":"2024-03-06T07:28:34.039197+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.046080+0000","last_change":"2024-03-06T07:28:34.044403+0000","last_active":"2024-03-06T07:28:34.046080+0000","last_peered":"2024-03-06T07:28:34.046080+0000","last_clean":"2024-03-06T07:28:34.046080+0000","last_became_active":"2024-03-06T07:28:34.044130+0000","last_became_peered":"2024-03-06T07:28:34.044130+0000","last_unstale":"2024-03-06T07:28:34.046080+0000","last_undegraded":"2024-03-06T07:28:34.046080+0000","last_fullsized":"2024-03-06T07:28:34.046080+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:18.838141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.044109+0000","last_change":"2024-03-06T07:28:34.044109+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.044109+0000","last_undegraded":"2024-03-06T07:28:34.044109+0000","last_fullsized":"2024-03-06T07:28:34.044109+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.054615+0000","last_change":"2024-03-06T07:28:34.054170+0000","last_active":"2024-03-06T07:28:34.054615+0000","last_peered":"2024-03-06T07:28:34.054615+0000","last_clean":"2024-03-06T07:28:34.054615+0000","last_became_active":"2024-03-06T07:28:34.053871+0000","last_became_peered":"2024-03-06T07:28:34.053871+0000","last_unstale":"2024-03-06T07:28:34.054615+0000","last_undegraded":"2024-03-06T07:28:34.054615+0000","last_fullsized":"2024-03-06T07:28:34.054615+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:29.552066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.035597+0000","last_change":"2024-03-06T07:28:34.035597+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.035597+0000","last_undegraded":"2024-03-06T07:28:34.035597+0000","last_fullsized":"2024-03-06T07:28:34.035597+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.046191+0000","last_change":"2024-03-06T07:28:34.042955+0000","last_active":"2024-03-06T07:28:34.046191+0000","last_peered":"2024-03-06T07:28:34.046191+0000","last_clean":"2024-03-06T07:28:34.046191+0000","last_became_active":"2024-03-06T07:28:34.042684+0000","last_became_peered":"2024-03-06T07:28:34.042684+0000","last_unstale":"2024-03-06T07:28:34.046191+0000","last_undegraded":"2024-03-06T07:28:34.046191+0000","last_fullsized":"2024-03-06T07:28:34.046191+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:04.206156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037329+0000","last_change":"2024-03-06T07:28:34.037329+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037329+0000","last_undegraded":"2024-03-06T07:28:34.037329+0000","last_fullsized":"2024-03-06T07:28:34.037329+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.037420+0000","last_change":"2024-03-06T07:28:34.036082+0000","last_active":"2024-03-06T07:28:34.037420+0000","last_peered":"2024-03-06T07:28:34.037420+0000","last_clean":"2024-03-06T07:28:34.037420+0000","last_became_active":"2024-03-06T07:28:34.035802+0000","last_became_peered":"2024-03-06T07:28:34.035802+0000","last_unstale":"2024-03-06T07:28:34.037420+0000","last_undegraded":"2024-03-06T07:28:34.037420+0000","last_fullsized":"2024-03-06T07:28:34.037420+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:00.673585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.035539+0000","last_change":"2024-03-06T07:28:34.035539+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.035539+0000","last_undegraded":"2024-03-06T07:28:34.035539+0000","last_fullsized":"2024-03-06T07:28:34.035539+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.051779+0000","last_change":"2024-03-06T07:28:34.051147+0000","last_active":"2024-03-06T07:28:34.051779+0000","last_peered":"2024-03-06T07:28:34.051779+0000","last_clean":"2024-03-06T07:28:34.051779+0000","last_became_active":"2024-03-06T07:28:34.050807+0000","last_became_peered":"2024-03-06T07:28:34.050807+0000","last_unstale":"2024-03-06T07:28:34.051779+0000","last_undegraded":"2024-03-06T07:28:34.051779+0000","last_fullsized":"2024-03-06T07:28:34.051779+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:56.463129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039302+0000","last_change":"2024-03-06T07:28:34.039302+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039302+0000","last_undegraded":"2024-03-06T07:28:34.039302+0000","last_fullsized":"2024-03-06T07:28:34.039302+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.047597+0000","last_change":"2024-03-06T07:28:34.044501+0000","last_active":"2024-03-06T07:28:34.047597+0000","last_peered":"2024-03-06T07:28:34.047597+0000","last_clean":"2024-03-06T07:28:34.047597+0000","last_became_active":"2024-03-06T07:28:34.044057+0000","last_became_peered":"2024-03-06T07:28:34.044057+0000","last_unstale":"2024-03-06T07:28:34.047597+0000","last_undegraded":"2024-03-06T07:28:34.047597+0000","last_fullsized":"2024-03-06T07:28:34.047597+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:33.892363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037969+0000","last_change":"2024-03-06T07:28:34.037969+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037969+0000","last_undegraded":"2024-03-06T07:28:34.037969+0000","last_fullsized":"2024-03-06T07:28:34.037969+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.042936+0000","last_change":"2024-03-06T07:28:34.041503+0000","last_active":"2024-03-06T07:28:34.042936+0000","last_peered":"2024-03-06T07:28:34.042936+0000","last_clean":"2024-03-06T07:28:34.042936+0000","last_became_active":"2024-03-06T07:28:34.041217+0000","last_became_peered":"2024-03-06T07:28:34.041217+0000","last_unstale":"2024-03-06T07:28:34.042936+0000","last_undegraded":"2024-03-06T07:28:34.042936+0000","last_fullsized":"2024-03-06T07:28:34.042936+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:14.748127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032982+0000","last_change":"2024-03-06T07:28:34.032982+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032982+0000","last_undegraded":"2024-03-06T07:28:34.032982+0000","last_fullsized":"2024-03-06T07:28:34.032982+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.050323+0000","last_change":"2024-03-06T07:28:34.048173+0000","last_active":"2024-03-06T07:28:34.050323+0000","last_peered":"2024-03-06T07:28:34.050323+0000","last_clean":"2024-03-06T07:28:34.050323+0000","last_became_active":"2024-03-06T07:28:34.047879+0000","last_became_peered":"2024-03-06T07:28:34.047879+0000","last_unstale":"2024-03-06T07:28:34.050323+0000","last_undegraded":"2024-03-06T07:28:34.050323+0000","last_fullsized":"2024-03-06T07:28:34.050323+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:42.229556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039603+0000","last_change":"2024-03-06T07:28:34.039603+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039603+0000","last_undegraded":"2024-03-06T07:28:34.039603+0000","last_fullsized":"2024-03-06T07:28:34.039603+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.047805+0000","last_change":"2024-03-06T07:28:34.045231+0000","last_active":"2024-03-06T07:28:34.047805+0000","last_peered":"2024-03-06T07:28:34.047805+0000","last_clean":"2024-03-06T07:28:34.047805+0000","last_became_active":"2024-03-06T07:28:34.044892+0000","last_became_peered":"2024-03-06T07:28:34.044892+0000","last_unstale":"2024-03-06T07:28:34.047805+0000","last_undegraded":"2024-03-06T07:28:34.047805+0000","last_fullsized":"2024-03-06T07:28:34.047805+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:02.569692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032242+0000","last_change":"2024-03-06T07:28:34.032242+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032242+0000","last_undegraded":"2024-03-06T07:28:34.032242+0000","last_fullsized":"2024-03-06T07:28:34.032242+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.047690+0000","last_change":"2024-03-06T07:28:34.045006+0000","last_active":"2024-03-06T07:28:34.047690+0000","last_peered":"2024-03-06T07:28:34.047690+0000","last_clean":"2024-03-06T07:28:34.047690+0000","last_became_active":"2024-03-06T07:28:34.044739+0000","last_became_peered":"2024-03-06T07:28:34.044739+0000","last_unstale":"2024-03-06T07:28:34.047690+0000","last_undegraded":"2024-03-06T07:28:34.047690+0000","last_fullsized":"2024-03-06T07:28:34.047690+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:39.255500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036198+0000","last_change":"2024-03-06T07:28:34.036198+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036198+0000","last_undegraded":"2024-03-06T07:28:34.036198+0000","last_fullsized":"2024-03-06T07:28:34.036198+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.051110+0000","last_change":"2024-03-06T07:28:34.050123+0000","last_active":"2024-03-06T07:28:34.051110+0000","last_peered":"2024-03-06T07:28:34.051110+0000","last_clean":"2024-03-06T07:28:34.051110+0000","last_became_active":"2024-03-06T07:28:34.049801+0000","last_became_peered":"2024-03-06T07:28:34.049801+0000","last_unstale":"2024-03-06T07:28:34.051110+0000","last_undegraded":"2024-03-06T07:28:34.051110+0000","last_fullsized":"2024-03-06T07:28:34.051110+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:41.920850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037352+0000","last_change":"2024-03-06T07:28:34.037352+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037352+0000","last_undegraded":"2024-03-06T07:28:34.037352+0000","last_fullsized":"2024-03-06T07:28:34.037352+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.048452+0000","last_change":"2024-03-06T07:28:34.047655+0000","last_active":"2024-03-06T07:28:34.048452+0000","last_peered":"2024-03-06T07:28:34.048452+0000","last_clean":"2024-03-06T07:28:34.048452+0000","last_became_active":"2024-03-06T07:28:34.047397+0000","last_became_peered":"2024-03-06T07:28:34.047397+0000","last_unstale":"2024-03-06T07:28:34.048452+0000","last_undegraded":"2024-03-06T07:28:34.048452+0000","last_fullsized":"2024-03-06T07:28:34.048452+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:51.253409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034884+0000","last_change":"2024-03-06T07:28:34.034884+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034884+0000","last_undegraded":"2024-03-06T07:28:34.034884+0000","last_fullsized":"2024-03-06T07:28:34.034884+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.050554+0000","last_change":"2024-03-06T07:28:34.049637+0000","last_active":"2024-03-06T07:28:34.050554+0000","last_peered":"2024-03-06T07:28:34.050554+0000","last_clean":"2024-03-06T07:28:34.050554+0000","last_became_active":"2024-03-06T07:28:34.049301+0000","last_became_peered":"2024-03-06T07:28:34.049301+0000","last_unstale":"2024-03-06T07:28:34.050554+0000","last_undegraded":"2024-03-06T07:28:34.050554+0000","last_fullsized":"2024-03-06T07:28:34.050554+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:03.780188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033456+0000","last_change":"2024-03-06T07:28:34.033456+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033456+0000","last_undegraded":"2024-03-06T07:28:34.033456+0000","last_fullsized":"2024-03-06T07:28:34.033456+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.049420+0000","last_change":"2024-03-06T07:28:34.046419+0000","last_active":"2024-03-06T07:28:34.049420+0000","last_peered":"2024-03-06T07:28:34.049420+0000","last_clean":"2024-03-06T07:28:34.049420+0000","last_became_active":"2024-03-06T07:28:34.046066+0000","last_became_peered":"2024-03-06T07:28:34.046066+0000","last_unstale":"2024-03-06T07:28:34.049420+0000","last_undegraded":"2024-03-06T07:28:34.049420+0000","last_fullsized":"2024-03-06T07:28:34.049420+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:00.025032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.040463+0000","last_change":"2024-03-06T07:28:34.040463+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.040463+0000","last_undegraded":"2024-03-06T07:28:34.040463+0000","last_fullsized":"2024-03-06T07:28:34.040463+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.045627+0000","last_change":"2024-03-06T07:28:34.045103+0000","last_active":"2024-03-06T07:28:34.045627+0000","last_peered":"2024-03-06T07:28:34.045627+0000","last_clean":"2024-03-06T07:28:34.045627+0000","last_became_active":"2024-03-06T07:28:34.044786+0000","last_became_peered":"2024-03-06T07:28:34.044786+0000","last_unstale":"2024-03-06T07:28:34.045627+0000","last_undegraded":"2024-03-06T07:28:34.045627+0000","last_fullsized":"2024-03-06T07:28:34.045627+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:50.885489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.041301+0000","last_change":"2024-03-06T07:28:34.041301+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.041301+0000","last_undegraded":"2024-03-06T07:28:34.041301+0000","last_fullsized":"2024-03-06T07:28:34.041301+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.054609+0000","last_change":"2024-03-06T07:28:34.054092+0000","last_active":"2024-03-06T07:28:34.054609+0000","last_peered":"2024-03-06T07:28:34.054609+0000","last_clean":"2024-03-06T07:28:34.054609+0000","last_became_active":"2024-03-06T07:28:34.053838+0000","last_became_peered":"2024-03-06T07:28:34.053838+0000","last_unstale":"2024-03-06T07:28:34.054609+0000","last_undegraded":"2024-03-06T07:28:34.054609+0000","last_fullsized":"2024-03-06T07:28:34.054609+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:42.229556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033981+0000","last_change":"2024-03-06T07:28:34.033981+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033981+0000","last_undegraded":"2024-03-06T07:28:34.033981+0000","last_fullsized":"2024-03-06T07:28:34.033981+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.050540+0000","last_change":"2024-03-06T07:28:34.046882+0000","last_active":"2024-03-06T07:28:34.050540+0000","last_peered":"2024-03-06T07:28:34.050540+0000","last_clean":"2024-03-06T07:28:34.050540+0000","last_became_active":"2024-03-06T07:28:34.045955+0000","last_became_peered":"2024-03-06T07:28:34.045955+0000","last_unstale":"2024-03-06T07:28:34.050540+0000","last_undegraded":"2024-03-06T07:28:34.050540+0000","last_fullsized":"2024-03-06T07:28:34.050540+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:33.892363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036066+0000","last_change":"2024-03-06T07:28:34.036066+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036066+0000","last_undegraded":"2024-03-06T07:28:34.036066+0000","last_fullsized":"2024-03-06T07:28:34.036066+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.050448+0000","last_change":"2024-03-06T07:28:34.047502+0000","last_active":"2024-03-06T07:28:34.050448+0000","last_peered":"2024-03-06T07:28:34.050448+0000","last_clean":"2024-03-06T07:28:34.050448+0000","last_became_active":"2024-03-06T07:28:34.047242+0000","last_became_peered":"2024-03-06T07:28:34.047242+0000","last_unstale":"2024-03-06T07:28:34.050448+0000","last_undegraded":"2024-03-06T07:28:34.050448+0000","last_fullsized":"2024-03-06T07:28:34.050448+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:39.255500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037043+0000","last_change":"2024-03-06T07:28:34.037043+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037043+0000","last_undegraded":"2024-03-06T07:28:34.037043+0000","last_fullsized":"2024-03-06T07:28:34.037043+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.046728+0000","last_change":"2024-03-06T07:28:34.045868+0000","last_active":"2024-03-06T07:28:34.046728+0000","last_peered":"2024-03-06T07:28:34.046728+0000","last_clean":"2024-03-06T07:28:34.046728+0000","last_became_active":"2024-03-06T07:28:34.045604+0000","last_became_peered":"2024-03-06T07:28:34.045604+0000","last_unstale":"2024-03-06T07:28:34.046728+0000","last_undegraded":"2024-03-06T07:28:34.046728+0000","last_fullsized":"2024-03-06T07:28:34.046728+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:28.693278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037833+0000","last_change":"2024-03-06T07:28:34.037833+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037833+0000","last_undegraded":"2024-03-06T07:28:34.037833+0000","last_fullsized":"2024-03-06T07:28:34.037833+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.048202+0000","last_change":"2024-03-06T07:28:34.047667+0000","last_active":"2024-03-06T07:28:34.048202+0000","last_peered":"2024-03-06T07:28:34.048202+0000","last_clean":"2024-03-06T07:28:34.048202+0000","last_became_active":"2024-03-06T07:28:34.047280+0000","last_became_peered":"2024-03-06T07:28:34.047280+0000","last_unstale":"2024-03-06T07:28:34.048202+0000","last_undegraded":"2024-03-06T07:28:34.048202+0000","last_fullsized":"2024-03-06T07:28:34.048202+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:21.608844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034910+0000","last_change":"2024-03-06T07:28:34.034910+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034910+0000","last_undegraded":"2024-03-06T07:28:34.034910+0000","last_fullsized":"2024-03-06T07:28:34.034910+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053702+0000","last_change":"2024-03-06T07:28:34.049708+0000","last_active":"2024-03-06T07:28:34.053702+0000","last_peered":"2024-03-06T07:28:34.053702+0000","last_clean":"2024-03-06T07:28:34.053702+0000","last_became_active":"2024-03-06T07:28:34.049445+0000","last_became_peered":"2024-03-06T07:28:34.049445+0000","last_unstale":"2024-03-06T07:28:34.053702+0000","last_undegraded":"2024-03-06T07:28:34.053702+0000","last_fullsized":"2024-03-06T07:28:34.053702+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:53.020063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039713+0000","last_change":"2024-03-06T07:28:34.039713+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039713+0000","last_undegraded":"2024-03-06T07:28:34.039713+0000","last_fullsized":"2024-03-06T07:28:34.039713+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.040318+0000","last_change":"2024-03-06T07:28:34.038476+0000","last_active":"2024-03-06T07:28:34.040318+0000","last_peered":"2024-03-06T07:28:34.040318+0000","last_clean":"2024-03-06T07:28:34.040318+0000","last_became_active":"2024-03-06T07:28:34.038197+0000","last_became_peered":"2024-03-06T07:28:34.038197+0000","last_unstale":"2024-03-06T07:28:34.040318+0000","last_undegraded":"2024-03-06T07:28:34.040318+0000","last_fullsized":"2024-03-06T07:28:34.040318+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:49.999763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038359+0000","last_change":"2024-03-06T07:28:34.038359+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038359+0000","last_undegraded":"2024-03-06T07:28:34.038359+0000","last_fullsized":"2024-03-06T07:28:34.038359+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.047046+0000","last_change":"2024-03-06T07:28:34.046180+0000","last_active":"2024-03-06T07:28:34.047046+0000","last_peered":"2024-03-06T07:28:34.047046+0000","last_clean":"2024-03-06T07:28:34.047046+0000","last_became_active":"2024-03-06T07:28:34.045909+0000","last_became_peered":"2024-03-06T07:28:34.045909+0000","last_unstale":"2024-03-06T07:28:34.047046+0000","last_undegraded":"2024-03-06T07:28:34.047046+0000","last_fullsized":"2024-03-06T07:28:34.047046+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:29.293270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032940+0000","last_change":"2024-03-06T07:28:34.032940+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032940+0000","last_undegraded":"2024-03-06T07:28:34.032940+0000","last_fullsized":"2024-03-06T07:28:34.032940+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.048682+0000","last_change":"2024-03-06T07:28:34.043986+0000","last_active":"2024-03-06T07:28:34.048682+0000","last_peered":"2024-03-06T07:28:34.048682+0000","last_clean":"2024-03-06T07:28:34.048682+0000","last_became_active":"2024-03-06T07:28:34.043693+0000","last_became_peered":"2024-03-06T07:28:34.043693+0000","last_unstale":"2024-03-06T07:28:34.048682+0000","last_undegraded":"2024-03-06T07:28:34.048682+0000","last_fullsized":"2024-03-06T07:28:34.048682+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:12.297830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036386+0000","last_change":"2024-03-06T07:28:34.036386+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036386+0000","last_undegraded":"2024-03-06T07:28:34.036386+0000","last_fullsized":"2024-03-06T07:28:34.036386+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053605+0000","last_change":"2024-03-06T07:28:34.047903+0000","last_active":"2024-03-06T07:28:34.053605+0000","last_peered":"2024-03-06T07:28:34.053605+0000","last_clean":"2024-03-06T07:28:34.053605+0000","last_became_active":"2024-03-06T07:28:34.047612+0000","last_became_peered":"2024-03-06T07:28:34.047612+0000","last_unstale":"2024-03-06T07:28:34.053605+0000","last_undegraded":"2024-03-06T07:28:34.053605+0000","last_fullsized":"2024-03-06T07:28:34.053605+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:08.365596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034459+0000","last_change":"2024-03-06T07:28:34.034459+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034459+0000","last_undegraded":"2024-03-06T07:28:34.034459+0000","last_fullsized":"2024-03-06T07:28:34.034459+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.052224+0000","last_change":"2024-03-06T07:28:34.051791+0000","last_active":"2024-03-06T07:28:34.052224+0000","last_peered":"2024-03-06T07:28:34.052224+0000","last_clean":"2024-03-06T07:28:34.052224+0000","last_became_active":"2024-03-06T07:28:34.051472+0000","last_became_peered":"2024-03-06T07:28:34.051472+0000","last_unstale":"2024-03-06T07:28:34.052224+0000","last_undegraded":"2024-03-06T07:28:34.052224+0000","last_fullsized":"2024-03-06T07:28:34.052224+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:50.885489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037431+0000","last_change":"2024-03-06T07:28:34.037431+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037431+0000","last_undegraded":"2024-03-06T07:28:34.037431+0000","last_fullsized":"2024-03-06T07:28:34.037431+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053725+0000","last_change":"2024-03-06T07:28:34.048933+0000","last_active":"2024-03-06T07:28:34.053725+0000","last_peered":"2024-03-06T07:28:34.053725+0000","last_clean":"2024-03-06T07:28:34.053725+0000","last_became_active":"2024-03-06T07:28:34.048670+0000","last_became_peered":"2024-03-06T07:28:34.048670+0000","last_unstale":"2024-03-06T07:28:34.053725+0000","last_undegraded":"2024-03-06T07:28:34.053725+0000","last_fullsized":"2024-03-06T07:28:34.053725+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:08.365596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037745+0000","last_change":"2024-03-06T07:28:34.037745+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037745+0000","last_undegraded":"2024-03-06T07:28:34.037745+0000","last_fullsized":"2024-03-06T07:28:34.037745+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053497+0000","last_change":"2024-03-06T07:28:34.049313+0000","last_active":"2024-03-06T07:28:34.053497+0000","last_peered":"2024-03-06T07:28:34.053497+0000","last_clean":"2024-03-06T07:28:34.053497+0000","last_became_active":"2024-03-06T07:28:34.048967+0000","last_became_peered":"2024-03-06T07:28:34.048967+0000","last_unstale":"2024-03-06T07:28:34.053497+0000","last_undegraded":"2024-03-06T07:28:34.053497+0000","last_fullsized":"2024-03-06T07:28:34.053497+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:51.180609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":506,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.032085+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:28:34.032085+0000","last_peered":"2024-03-06T07:28:34.032085+0000","last_clean":"2024-03-06T07:28:34.032085+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:28:34.032085+0000","last_undegraded":"2024-03-06T07:28:34.032085+0000","last_fullsized":"2024-03-06T07:28:34.032085+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574951,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33664,"kb_used_data":2480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338176,"statfs":{"total":96636764160,"available":96602292224,"internally_reserved":0,"allocated":2539520,"data_stored":1793319,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574950,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33584,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338256,"statfs":{"total":96636764160,"available":96602374144,"internally_reserved":0,"allocated":2461696,"data_stored":1196284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574950,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33604,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338236,"statfs":{"total":96636764160,"available":96602353664,"internally_reserved":0,"allocated":2473984,"data_stored":1202951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574951,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33600,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338240,"statfs":{"total":96636764160,"available":96602357760,"internally_reserved":0,"allocated":2473984,"data_stored":1202951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574950,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33600,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338240,"statfs":{"total":96636764160,"available":96602357760,"internally_reserved":0,"allocated":2473984,"data_stored":1202951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574950,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37744,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334096,"statfs":{"total":96636764160,"available":96598114304,"internally_reserved":0,"allocated":2461696,"data_stored":1196284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574951,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33600,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338240,"statfs":{"total":96636764160,"available":96602357760,"internally_reserved":0,"allocated":2473984,"data_stored":1202951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574951,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37812,"kb_used_data":2468,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334028,"statfs":{"total":96636764160,"available":96598044672,"internally_reserved":0,"allocated":2527232,"data_stored":1786652,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:38.111 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:38.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.247+0000 7f7447c4e640 1 Processor -- start 2024-03-06T07:28:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.247+0000 7f7447c4e640 1 -- start start 2024-03-06T07:28:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.247+0000 7f7447c4e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7440150fe0 0x7f74401513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.247+0000 7f7447c4e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7440072f20 0x7f744006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.247+0000 7f7447c4e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f744006f2f0 0x7f7440071f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.247+0000 7f7447c4e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74401526b0 con 0x7f744006f2f0 2024-03-06T07:28:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.247+0000 7f7447c4e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74400a6140 con 0x7f7440072f20 2024-03-06T07:28:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.247+0000 7f7447c4e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7440151aa0 con 0x7f7440150fe0 2024-03-06T07:28:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.248+0000 7f74461eb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f744006f2f0 0x7f7440071f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.248+0000 7f74461eb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f744006f2f0 0x7f7440071f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54528/0 (socket says 172.21.15.1:54528) 2024-03-06T07:28:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.248+0000 7f74461eb640 1 -- 172.21.15.1:0/3162279096 learned_addr learned my addr 172.21.15.1:0/3162279096 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.248+0000 7f74459ea640 1 --2- 172.21.15.1:0/3162279096 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7440150fe0 0x7f74401513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.248+0000 7f74451e9640 1 --2- 172.21.15.1:0/3162279096 >> v2:172.21.15.98:3300/0 conn(0x7f7440072f20 0x7f744006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.248+0000 7f74461eb640 1 -- 172.21.15.1:0/3162279096 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7440150fe0 msgr2=0x7f74401513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.248+0000 7f74461eb640 1 --2- 172.21.15.1:0/3162279096 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7440150fe0 0x7f74401513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.248+0000 7f74461eb640 1 -- 172.21.15.1:0/3162279096 >> v2:172.21.15.98:3300/0 conn(0x7f7440072f20 msgr2=0x7f744006edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.248+0000 7f74461eb640 1 --2- 172.21.15.1:0/3162279096 >> v2:172.21.15.98:3300/0 conn(0x7f7440072f20 0x7f744006edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.248+0000 7f74461eb640 1 -- 172.21.15.1:0/3162279096 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7440151d80 con 0x7f744006f2f0 2024-03-06T07:28:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.249+0000 7f74461eb640 1 --2- 172.21.15.1:0/3162279096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f744006f2f0 0x7f7440071f90 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f744006fab0 tx=0x7f743802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a50265cfecf074d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.249+0000 7f74449e8640 1 -- 172.21.15.1:0/3162279096 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743802c9e0 con 0x7f744006f2f0 2024-03-06T07:28:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.249+0000 7f74449e8640 1 -- 172.21.15.1:0/3162279096 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743802cb80 con 0x7f744006f2f0 2024-03-06T07:28:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.249+0000 7f74449e8640 1 -- 172.21.15.1:0/3162279096 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74380343f0 con 0x7f744006f2f0 2024-03-06T07:28:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.249+0000 7f7447c4e640 1 -- 172.21.15.1:0/3162279096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f744006f2f0 msgr2=0x7f7440071f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.249+0000 7f7447c4e640 1 --2- 172.21.15.1:0/3162279096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f744006f2f0 0x7f7440071f90 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f744006fab0 tx=0x7f743802be40 comp rx=0 tx=0).stop 2024-03-06T07:28:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.249+0000 7f7447c4e640 1 -- 172.21.15.1:0/3162279096 shutdown_connections 2024-03-06T07:28:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.249+0000 7f7447c4e640 1 --2- 172.21.15.1:0/3162279096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f744006f2f0 0x7f7440071f90 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.249+0000 7f7447c4e640 1 --2- 172.21.15.1:0/3162279096 >> v2:172.21.15.98:3300/0 conn(0x7f7440072f20 0x7f744006edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.249+0000 7f7447c4e640 1 --2- 172.21.15.1:0/3162279096 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7440150fe0 0x7f74401513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.250+0000 7f7447c4e640 1 -- 172.21.15.1:0/3162279096 >> 172.21.15.1:0/3162279096 conn(0x7f74400c6860 msgr2=0x7f74400c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.250+0000 7f7447c4e640 1 -- 172.21.15.1:0/3162279096 shutdown_connections 2024-03-06T07:28:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.250+0000 7f7447c4e640 1 -- 172.21.15.1:0/3162279096 wait complete. 2024-03-06T07:28:38.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.251+0000 7f7447c4e640 1 Processor -- start 2024-03-06T07:28:38.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.251+0000 7f7447c4e640 1 -- start start 2024-03-06T07:28:38.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.251+0000 7f7447c4e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f744006f2f0 0x7f7440154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.251+0000 7f7447c4e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7440072f20 0x7f744014db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.251+0000 7f7447c4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7440150fe0 0x7f7440156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.251+0000 7f7447c4e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74400724d0 con 0x7f744006f2f0 2024-03-06T07:28:38.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.251+0000 7f7447c4e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7440152340 con 0x7f7440150fe0 2024-03-06T07:28:38.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.251+0000 7f7447c4e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7440072650 con 0x7f7440072f20 2024-03-06T07:28:38.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.251+0000 7f74459ea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f744006f2f0 0x7f7440154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.251+0000 7f74459ea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f744006f2f0 0x7f7440154930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54544/0 (socket says 172.21.15.1:54544) 2024-03-06T07:28:38.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.251+0000 7f74459ea640 1 -- 172.21.15.1:0/4159423383 learned_addr learned my addr 172.21.15.1:0/4159423383 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:38.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.252+0000 7f74461eb640 1 --2- 172.21.15.1:0/4159423383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7440150fe0 0x7f7440156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.252+0000 7f74451e9640 1 --2- 172.21.15.1:0/4159423383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7440072f20 0x7f744014db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.252+0000 7f74459ea640 1 -- 172.21.15.1:0/4159423383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7440072f20 msgr2=0x7f744014db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.252+0000 7f74459ea640 1 --2- 172.21.15.1:0/4159423383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7440072f20 0x7f744014db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.252+0000 7f74459ea640 1 -- 172.21.15.1:0/4159423383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7440150fe0 msgr2=0x7f7440156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.252+0000 7f74459ea640 1 --2- 172.21.15.1:0/4159423383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7440150fe0 0x7f7440156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.252+0000 7f74459ea640 1 -- 172.21.15.1:0/4159423383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f744014d340 con 0x7f744006f2f0 2024-03-06T07:28:38.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.252+0000 7f74461eb640 1 --2- 172.21.15.1:0/4159423383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7440150fe0 0x7f7440156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:38.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.252+0000 7f74451e9640 1 --2- 172.21.15.1:0/4159423383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7440072f20 0x7f744014db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:38.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.252+0000 7f74459ea640 1 --2- 172.21.15.1:0/4159423383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f744006f2f0 0x7f7440154930 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f743000e6c0 tx=0x7f743000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.252+0000 7f743e7fc640 1 -- 172.21.15.1:0/4159423383 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7430016070 con 0x7f744006f2f0 2024-03-06T07:28:38.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.252+0000 7f743e7fc640 1 -- 172.21.15.1:0/4159423383 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743000ca30 con 0x7f744006f2f0 2024-03-06T07:28:38.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.252+0000 7f7447c4e640 1 -- 172.21.15.1:0/4159423383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74401560f0 con 0x7f744006f2f0 2024-03-06T07:28:38.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.252+0000 7f743e7fc640 1 -- 172.21.15.1:0/4159423383 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743000b040 con 0x7f744006f2f0 2024-03-06T07:28:38.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.252+0000 7f7447c4e640 1 -- 172.21.15.1:0/4159423383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7440157080 con 0x7f744006f2f0 2024-03-06T07:28:38.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.253+0000 7f7447c4e640 1 -- 172.21.15.1:0/4159423383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7440067fd0 con 0x7f744006f2f0 2024-03-06T07:28:38.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.254+0000 7f743e7fc640 1 -- 172.21.15.1:0/4159423383 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7430019070 con 0x7f744006f2f0 2024-03-06T07:28:38.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.254+0000 7f743e7fc640 1 --2- 172.21.15.1:0/4159423383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f742c06b4d0 0x7f742c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.254+0000 7f743e7fc640 1 -- 172.21.15.1:0/4159423383 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7430089cd0 con 0x7f744006f2f0 2024-03-06T07:28:38.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.254+0000 7f74451e9640 1 --2- 172.21.15.1:0/4159423383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f742c06b4d0 0x7f742c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.255+0000 7f74451e9640 1 --2- 172.21.15.1:0/4159423383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f742c06b4d0 0x7f742c06d990 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f7440154e70 tx=0x7f7434008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.259+0000 7f743e7fc640 1 -- 172.21.15.1:0/4159423383 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7430058a50 con 0x7f744006f2f0 2024-03-06T07:28:38.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.493+0000 7f7447c4e640 1 -- 172.21.15.1:0/4159423383 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f744006eb90 con 0x7f742c06b4d0 2024-03-06T07:28:38.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.499+0000 7f743e7fc640 1 -- 172.21.15.1:0/4159423383 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247177 (secure 0 0 0) 0x7f744006eb90 con 0x7f742c06b4d0 2024-03-06T07:28:38.504 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:38.505 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:28:38.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.503+0000 7f7447c4e640 1 -- 172.21.15.1:0/4159423383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f742c06b4d0 msgr2=0x7f742c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.503+0000 7f7447c4e640 1 --2- 172.21.15.1:0/4159423383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f742c06b4d0 0x7f742c06d990 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f7440154e70 tx=0x7f7434008040 comp rx=0 tx=0).stop 2024-03-06T07:28:38.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.503+0000 7f7447c4e640 1 -- 172.21.15.1:0/4159423383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f744006f2f0 msgr2=0x7f7440154930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.503+0000 7f7447c4e640 1 --2- 172.21.15.1:0/4159423383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f744006f2f0 0x7f7440154930 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f743000e6c0 tx=0x7f743000eb80 comp rx=0 tx=0).stop 2024-03-06T07:28:38.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.503+0000 7f7447c4e640 1 -- 172.21.15.1:0/4159423383 shutdown_connections 2024-03-06T07:28:38.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.503+0000 7f7447c4e640 1 --2- 172.21.15.1:0/4159423383 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f742c06b4d0 0x7f742c06d990 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.503+0000 7f7447c4e640 1 --2- 172.21.15.1:0/4159423383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7440150fe0 0x7f7440156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.503+0000 7f7447c4e640 1 --2- 172.21.15.1:0/4159423383 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7440072f20 0x7f744014db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.503+0000 7f7447c4e640 1 --2- 172.21.15.1:0/4159423383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f744006f2f0 0x7f7440154930 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.503+0000 7f7447c4e640 1 -- 172.21.15.1:0/4159423383 >> 172.21.15.1:0/4159423383 conn(0x7f74400c6860 msgr2=0x7f744015b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.503+0000 7f7447c4e640 1 -- 172.21.15.1:0/4159423383 shutdown_connections 2024-03-06T07:28:38.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.503+0000 7f7447c4e640 1 -- 172.21.15.1:0/4159423383 wait complete. 2024-03-06T07:28:38.515 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":441,"stamp":"2024-03-06T07:28:36.807528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":277208,"kb_used_data":19420,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754697512,"statfs":{"total":773094113280,"available":772810252288,"internally_reserved":0,"allocated":19886080,"data_stored":10784343,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000971"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039773+0000","last_change":"2024-03-06T07:28:34.039773+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039773+0000","last_undegraded":"2024-03-06T07:28:34.039773+0000","last_fullsized":"2024-03-06T07:28:34.039773+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032560+0000","last_change":"2024-03-06T07:28:34.032560+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032560+0000","last_undegraded":"2024-03-06T07:28:34.032560+0000","last_fullsized":"2024-03-06T07:28:34.032560+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.043939+0000","last_change":"2024-03-06T07:28:34.043939+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.043939+0000","last_undegraded":"2024-03-06T07:28:34.043939+0000","last_fullsized":"2024-03-06T07:28:34.043939+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.040918+0000","last_change":"2024-03-06T07:28:34.040918+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.040918+0000","last_undegraded":"2024-03-06T07:28:34.040918+0000","last_fullsized":"2024-03-06T07:28:34.040918+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034672+0000","last_change":"2024-03-06T07:28:34.034672+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034672+0000","last_undegraded":"2024-03-06T07:28:34.034672+0000","last_fullsized":"2024-03-06T07:28:34.034672+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036995+0000","last_change":"2024-03-06T07:28:34.036995+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036995+0000","last_undegraded":"2024-03-06T07:28:34.036995+0000","last_fullsized":"2024-03-06T07:28:34.036995+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037384+0000","last_change":"2024-03-06T07:28:34.037384+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037384+0000","last_undegraded":"2024-03-06T07:28:34.037384+0000","last_fullsized":"2024-03-06T07:28:34.037384+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036819+0000","last_change":"2024-03-06T07:28:34.036819+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036819+0000","last_undegraded":"2024-03-06T07:28:34.036819+0000","last_fullsized":"2024-03-06T07:28:34.036819+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038840+0000","last_change":"2024-03-06T07:28:34.038840+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038840+0000","last_undegraded":"2024-03-06T07:28:34.038840+0000","last_fullsized":"2024-03-06T07:28:34.038840+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.043705+0000","last_change":"2024-03-06T07:28:34.043705+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.043705+0000","last_undegraded":"2024-03-06T07:28:34.043705+0000","last_fullsized":"2024-03-06T07:28:34.043705+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.041796+0000","last_change":"2024-03-06T07:28:34.041796+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.041796+0000","last_undegraded":"2024-03-06T07:28:34.041796+0000","last_fullsized":"2024-03-06T07:28:34.041796+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038852+0000","last_change":"2024-03-06T07:28:34.038852+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038852+0000","last_undegraded":"2024-03-06T07:28:34.038852+0000","last_fullsized":"2024-03-06T07:28:34.038852+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.035743+0000","last_change":"2024-03-06T07:28:34.035743+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.035743+0000","last_undegraded":"2024-03-06T07:28:34.035743+0000","last_fullsized":"2024-03-06T07:28:34.035743+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033521+0000","last_change":"2024-03-06T07:28:34.033521+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033521+0000","last_undegraded":"2024-03-06T07:28:34.033521+0000","last_fullsized":"2024-03-06T07:28:34.033521+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033924+0000","last_change":"2024-03-06T07:28:34.033924+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033924+0000","last_undegraded":"2024-03-06T07:28:34.033924+0000","last_fullsized":"2024-03-06T07:28:34.033924+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038810+0000","last_change":"2024-03-06T07:28:34.038810+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038810+0000","last_undegraded":"2024-03-06T07:28:34.038810+0000","last_fullsized":"2024-03-06T07:28:34.038810+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036613+0000","last_change":"2024-03-06T07:28:34.036613+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036613+0000","last_undegraded":"2024-03-06T07:28:34.036613+0000","last_fullsized":"2024-03-06T07:28:34.036613+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.043282+0000","last_change":"2024-03-06T07:28:34.043282+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.043282+0000","last_undegraded":"2024-03-06T07:28:34.043282+0000","last_fullsized":"2024-03-06T07:28:34.043282+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033099+0000","last_change":"2024-03-06T07:28:34.033099+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033099+0000","last_undegraded":"2024-03-06T07:28:34.033099+0000","last_fullsized":"2024-03-06T07:28:34.033099+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.031957+0000","last_change":"2024-03-06T07:28:34.031957+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.031957+0000","last_undegraded":"2024-03-06T07:28:34.031957+0000","last_fullsized":"2024-03-06T07:28:34.031957+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037603+0000","last_change":"2024-03-06T07:28:34.037603+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037603+0000","last_undegraded":"2024-03-06T07:28:34.037603+0000","last_fullsized":"2024-03-06T07:28:34.037603+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036930+0000","last_change":"2024-03-06T07:28:34.036930+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036930+0000","last_undegraded":"2024-03-06T07:28:34.036930+0000","last_fullsized":"2024-03-06T07:28:34.036930+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033464+0000","last_change":"2024-03-06T07:28:34.033464+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033464+0000","last_undegraded":"2024-03-06T07:28:34.033464+0000","last_fullsized":"2024-03-06T07:28:34.033464+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037784+0000","last_change":"2024-03-06T07:28:34.037784+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037784+0000","last_undegraded":"2024-03-06T07:28:34.037784+0000","last_fullsized":"2024-03-06T07:28:34.037784+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.031482+0000","last_change":"2024-03-06T07:28:34.031482+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.031482+0000","last_undegraded":"2024-03-06T07:28:34.031482+0000","last_fullsized":"2024-03-06T07:28:34.031482+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.040505+0000","last_change":"2024-03-06T07:28:34.040505+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.040505+0000","last_undegraded":"2024-03-06T07:28:34.040505+0000","last_fullsized":"2024-03-06T07:28:34.040505+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034359+0000","last_change":"2024-03-06T07:28:34.034359+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034359+0000","last_undegraded":"2024-03-06T07:28:34.034359+0000","last_fullsized":"2024-03-06T07:28:34.034359+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038455+0000","last_change":"2024-03-06T07:28:34.038455+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038455+0000","last_undegraded":"2024-03-06T07:28:34.038455+0000","last_fullsized":"2024-03-06T07:28:34.038455+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032251+0000","last_change":"2024-03-06T07:28:34.032251+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032251+0000","last_undegraded":"2024-03-06T07:28:34.032251+0000","last_fullsized":"2024-03-06T07:28:34.032251+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032247+0000","last_change":"2024-03-06T07:28:34.032247+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032247+0000","last_undegraded":"2024-03-06T07:28:34.032247+0000","last_fullsized":"2024-03-06T07:28:34.032247+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.031492+0000","last_change":"2024-03-06T07:28:34.031492+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.031492+0000","last_undegraded":"2024-03-06T07:28:34.031492+0000","last_fullsized":"2024-03-06T07:28:34.031492+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036448+0000","last_change":"2024-03-06T07:28:34.036448+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036448+0000","last_undegraded":"2024-03-06T07:28:34.036448+0000","last_fullsized":"2024-03-06T07:28:34.036448+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032461+0000","last_change":"2024-03-06T07:28:34.032461+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032461+0000","last_undegraded":"2024-03-06T07:28:34.032461+0000","last_fullsized":"2024-03-06T07:28:34.032461+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.046617+0000","last_change":"2024-03-06T07:28:34.045692+0000","last_active":"2024-03-06T07:28:34.046617+0000","last_peered":"2024-03-06T07:28:34.046617+0000","last_clean":"2024-03-06T07:28:34.046617+0000","last_became_active":"2024-03-06T07:28:34.045392+0000","last_became_peered":"2024-03-06T07:28:34.045392+0000","last_unstale":"2024-03-06T07:28:34.046617+0000","last_undegraded":"2024-03-06T07:28:34.046617+0000","last_fullsized":"2024-03-06T07:28:34.046617+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:59.770341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032614+0000","last_change":"2024-03-06T07:28:34.032614+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032614+0000","last_undegraded":"2024-03-06T07:28:34.032614+0000","last_fullsized":"2024-03-06T07:28:34.032614+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053547+0000","last_change":"2024-03-06T07:28:34.048342+0000","last_active":"2024-03-06T07:28:34.053547+0000","last_peered":"2024-03-06T07:28:34.053547+0000","last_clean":"2024-03-06T07:28:34.053547+0000","last_became_active":"2024-03-06T07:28:34.048077+0000","last_became_peered":"2024-03-06T07:28:34.048077+0000","last_unstale":"2024-03-06T07:28:34.053547+0000","last_undegraded":"2024-03-06T07:28:34.053547+0000","last_fullsized":"2024-03-06T07:28:34.053547+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:02.569692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038496+0000","last_change":"2024-03-06T07:28:34.038496+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038496+0000","last_undegraded":"2024-03-06T07:28:34.038496+0000","last_fullsized":"2024-03-06T07:28:34.038496+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.044121+0000","last_change":"2024-03-06T07:28:34.043250+0000","last_active":"2024-03-06T07:28:34.044121+0000","last_peered":"2024-03-06T07:28:34.044121+0000","last_clean":"2024-03-06T07:28:34.044121+0000","last_became_active":"2024-03-06T07:28:34.042985+0000","last_became_peered":"2024-03-06T07:28:34.042985+0000","last_unstale":"2024-03-06T07:28:34.044121+0000","last_undegraded":"2024-03-06T07:28:34.044121+0000","last_fullsized":"2024-03-06T07:28:34.044121+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:49.778788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032526+0000","last_change":"2024-03-06T07:28:34.032526+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032526+0000","last_undegraded":"2024-03-06T07:28:34.032526+0000","last_fullsized":"2024-03-06T07:28:34.032526+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.040595+0000","last_change":"2024-03-06T07:28:34.039018+0000","last_active":"2024-03-06T07:28:34.040595+0000","last_peered":"2024-03-06T07:28:34.040595+0000","last_clean":"2024-03-06T07:28:34.040595+0000","last_became_active":"2024-03-06T07:28:34.038734+0000","last_became_peered":"2024-03-06T07:28:34.038734+0000","last_unstale":"2024-03-06T07:28:34.040595+0000","last_undegraded":"2024-03-06T07:28:34.040595+0000","last_fullsized":"2024-03-06T07:28:34.040595+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:56.463129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039197+0000","last_change":"2024-03-06T07:28:34.039197+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039197+0000","last_undegraded":"2024-03-06T07:28:34.039197+0000","last_fullsized":"2024-03-06T07:28:34.039197+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.046080+0000","last_change":"2024-03-06T07:28:34.044403+0000","last_active":"2024-03-06T07:28:34.046080+0000","last_peered":"2024-03-06T07:28:34.046080+0000","last_clean":"2024-03-06T07:28:34.046080+0000","last_became_active":"2024-03-06T07:28:34.044130+0000","last_became_peered":"2024-03-06T07:28:34.044130+0000","last_unstale":"2024-03-06T07:28:34.046080+0000","last_undegraded":"2024-03-06T07:28:34.046080+0000","last_fullsized":"2024-03-06T07:28:34.046080+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:18.838141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.044109+0000","last_change":"2024-03-06T07:28:34.044109+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.044109+0000","last_undegraded":"2024-03-06T07:28:34.044109+0000","last_fullsized":"2024-03-06T07:28:34.044109+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.054615+0000","last_change":"2024-03-06T07:28:34.054170+0000","last_active":"2024-03-06T07:28:34.054615+0000","last_peered":"2024-03-06T07:28:34.054615+0000","last_clean":"2024-03-06T07:28:34.054615+0000","last_became_active":"2024-03-06T07:28:34.053871+0000","last_became_peered":"2024-03-06T07:28:34.053871+0000","last_unstale":"2024-03-06T07:28:34.054615+0000","last_undegraded":"2024-03-06T07:28:34.054615+0000","last_fullsized":"2024-03-06T07:28:34.054615+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:29.552066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.035597+0000","last_change":"2024-03-06T07:28:34.035597+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.035597+0000","last_undegraded":"2024-03-06T07:28:34.035597+0000","last_fullsized":"2024-03-06T07:28:34.035597+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.046191+0000","last_change":"2024-03-06T07:28:34.042955+0000","last_active":"2024-03-06T07:28:34.046191+0000","last_peered":"2024-03-06T07:28:34.046191+0000","last_clean":"2024-03-06T07:28:34.046191+0000","last_became_active":"2024-03-06T07:28:34.042684+0000","last_became_peered":"2024-03-06T07:28:34.042684+0000","last_unstale":"2024-03-06T07:28:34.046191+0000","last_undegraded":"2024-03-06T07:28:34.046191+0000","last_fullsized":"2024-03-06T07:28:34.046191+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:04.206156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037329+0000","last_change":"2024-03-06T07:28:34.037329+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037329+0000","last_undegraded":"2024-03-06T07:28:34.037329+0000","last_fullsized":"2024-03-06T07:28:34.037329+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.037420+0000","last_change":"2024-03-06T07:28:34.036082+0000","last_active":"2024-03-06T07:28:34.037420+0000","last_peered":"2024-03-06T07:28:34.037420+0000","last_clean":"2024-03-06T07:28:34.037420+0000","last_became_active":"2024-03-06T07:28:34.035802+0000","last_became_peered":"2024-03-06T07:28:34.035802+0000","last_unstale":"2024-03-06T07:28:34.037420+0000","last_undegraded":"2024-03-06T07:28:34.037420+0000","last_fullsized":"2024-03-06T07:28:34.037420+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:00.673585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.035539+0000","last_change":"2024-03-06T07:28:34.035539+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.035539+0000","last_undegraded":"2024-03-06T07:28:34.035539+0000","last_fullsized":"2024-03-06T07:28:34.035539+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.051779+0000","last_change":"2024-03-06T07:28:34.051147+0000","last_active":"2024-03-06T07:28:34.051779+0000","last_peered":"2024-03-06T07:28:34.051779+0000","last_clean":"2024-03-06T07:28:34.051779+0000","last_became_active":"2024-03-06T07:28:34.050807+0000","last_became_peered":"2024-03-06T07:28:34.050807+0000","last_unstale":"2024-03-06T07:28:34.051779+0000","last_undegraded":"2024-03-06T07:28:34.051779+0000","last_fullsized":"2024-03-06T07:28:34.051779+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:56.463129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039302+0000","last_change":"2024-03-06T07:28:34.039302+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039302+0000","last_undegraded":"2024-03-06T07:28:34.039302+0000","last_fullsized":"2024-03-06T07:28:34.039302+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.047597+0000","last_change":"2024-03-06T07:28:34.044501+0000","last_active":"2024-03-06T07:28:34.047597+0000","last_peered":"2024-03-06T07:28:34.047597+0000","last_clean":"2024-03-06T07:28:34.047597+0000","last_became_active":"2024-03-06T07:28:34.044057+0000","last_became_peered":"2024-03-06T07:28:34.044057+0000","last_unstale":"2024-03-06T07:28:34.047597+0000","last_undegraded":"2024-03-06T07:28:34.047597+0000","last_fullsized":"2024-03-06T07:28:34.047597+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:33.892363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037969+0000","last_change":"2024-03-06T07:28:34.037969+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037969+0000","last_undegraded":"2024-03-06T07:28:34.037969+0000","last_fullsized":"2024-03-06T07:28:34.037969+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.042936+0000","last_change":"2024-03-06T07:28:34.041503+0000","last_active":"2024-03-06T07:28:34.042936+0000","last_peered":"2024-03-06T07:28:34.042936+0000","last_clean":"2024-03-06T07:28:34.042936+0000","last_became_active":"2024-03-06T07:28:34.041217+0000","last_became_peered":"2024-03-06T07:28:34.041217+0000","last_unstale":"2024-03-06T07:28:34.042936+0000","last_undegraded":"2024-03-06T07:28:34.042936+0000","last_fullsized":"2024-03-06T07:28:34.042936+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:14.748127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032982+0000","last_change":"2024-03-06T07:28:34.032982+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032982+0000","last_undegraded":"2024-03-06T07:28:34.032982+0000","last_fullsized":"2024-03-06T07:28:34.032982+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.050323+0000","last_change":"2024-03-06T07:28:34.048173+0000","last_active":"2024-03-06T07:28:34.050323+0000","last_peered":"2024-03-06T07:28:34.050323+0000","last_clean":"2024-03-06T07:28:34.050323+0000","last_became_active":"2024-03-06T07:28:34.047879+0000","last_became_peered":"2024-03-06T07:28:34.047879+0000","last_unstale":"2024-03-06T07:28:34.050323+0000","last_undegraded":"2024-03-06T07:28:34.050323+0000","last_fullsized":"2024-03-06T07:28:34.050323+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:42.229556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039603+0000","last_change":"2024-03-06T07:28:34.039603+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039603+0000","last_undegraded":"2024-03-06T07:28:34.039603+0000","last_fullsized":"2024-03-06T07:28:34.039603+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.047805+0000","last_change":"2024-03-06T07:28:34.045231+0000","last_active":"2024-03-06T07:28:34.047805+0000","last_peered":"2024-03-06T07:28:34.047805+0000","last_clean":"2024-03-06T07:28:34.047805+0000","last_became_active":"2024-03-06T07:28:34.044892+0000","last_became_peered":"2024-03-06T07:28:34.044892+0000","last_unstale":"2024-03-06T07:28:34.047805+0000","last_undegraded":"2024-03-06T07:28:34.047805+0000","last_fullsized":"2024-03-06T07:28:34.047805+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:02.569692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032242+0000","last_change":"2024-03-06T07:28:34.032242+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032242+0000","last_undegraded":"2024-03-06T07:28:34.032242+0000","last_fullsized":"2024-03-06T07:28:34.032242+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.047690+0000","last_change":"2024-03-06T07:28:34.045006+0000","last_active":"2024-03-06T07:28:34.047690+0000","last_peered":"2024-03-06T07:28:34.047690+0000","last_clean":"2024-03-06T07:28:34.047690+0000","last_became_active":"2024-03-06T07:28:34.044739+0000","last_became_peered":"2024-03-06T07:28:34.044739+0000","last_unstale":"2024-03-06T07:28:34.047690+0000","last_undegraded":"2024-03-06T07:28:34.047690+0000","last_fullsized":"2024-03-06T07:28:34.047690+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:39.255500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036198+0000","last_change":"2024-03-06T07:28:34.036198+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036198+0000","last_undegraded":"2024-03-06T07:28:34.036198+0000","last_fullsized":"2024-03-06T07:28:34.036198+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.051110+0000","last_change":"2024-03-06T07:28:34.050123+0000","last_active":"2024-03-06T07:28:34.051110+0000","last_peered":"2024-03-06T07:28:34.051110+0000","last_clean":"2024-03-06T07:28:34.051110+0000","last_became_active":"2024-03-06T07:28:34.049801+0000","last_became_peered":"2024-03-06T07:28:34.049801+0000","last_unstale":"2024-03-06T07:28:34.051110+0000","last_undegraded":"2024-03-06T07:28:34.051110+0000","last_fullsized":"2024-03-06T07:28:34.051110+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:41.920850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037352+0000","last_change":"2024-03-06T07:28:34.037352+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037352+0000","last_undegraded":"2024-03-06T07:28:34.037352+0000","last_fullsized":"2024-03-06T07:28:34.037352+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.048452+0000","last_change":"2024-03-06T07:28:34.047655+0000","last_active":"2024-03-06T07:28:34.048452+0000","last_peered":"2024-03-06T07:28:34.048452+0000","last_clean":"2024-03-06T07:28:34.048452+0000","last_became_active":"2024-03-06T07:28:34.047397+0000","last_became_peered":"2024-03-06T07:28:34.047397+0000","last_unstale":"2024-03-06T07:28:34.048452+0000","last_undegraded":"2024-03-06T07:28:34.048452+0000","last_fullsized":"2024-03-06T07:28:34.048452+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:51.253409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034884+0000","last_change":"2024-03-06T07:28:34.034884+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034884+0000","last_undegraded":"2024-03-06T07:28:34.034884+0000","last_fullsized":"2024-03-06T07:28:34.034884+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.050554+0000","last_change":"2024-03-06T07:28:34.049637+0000","last_active":"2024-03-06T07:28:34.050554+0000","last_peered":"2024-03-06T07:28:34.050554+0000","last_clean":"2024-03-06T07:28:34.050554+0000","last_became_active":"2024-03-06T07:28:34.049301+0000","last_became_peered":"2024-03-06T07:28:34.049301+0000","last_unstale":"2024-03-06T07:28:34.050554+0000","last_undegraded":"2024-03-06T07:28:34.050554+0000","last_fullsized":"2024-03-06T07:28:34.050554+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:03.780188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033456+0000","last_change":"2024-03-06T07:28:34.033456+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033456+0000","last_undegraded":"2024-03-06T07:28:34.033456+0000","last_fullsized":"2024-03-06T07:28:34.033456+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.049420+0000","last_change":"2024-03-06T07:28:34.046419+0000","last_active":"2024-03-06T07:28:34.049420+0000","last_peered":"2024-03-06T07:28:34.049420+0000","last_clean":"2024-03-06T07:28:34.049420+0000","last_became_active":"2024-03-06T07:28:34.046066+0000","last_became_peered":"2024-03-06T07:28:34.046066+0000","last_unstale":"2024-03-06T07:28:34.049420+0000","last_undegraded":"2024-03-06T07:28:34.049420+0000","last_fullsized":"2024-03-06T07:28:34.049420+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:00.025032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.040463+0000","last_change":"2024-03-06T07:28:34.040463+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.040463+0000","last_undegraded":"2024-03-06T07:28:34.040463+0000","last_fullsized":"2024-03-06T07:28:34.040463+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.045627+0000","last_change":"2024-03-06T07:28:34.045103+0000","last_active":"2024-03-06T07:28:34.045627+0000","last_peered":"2024-03-06T07:28:34.045627+0000","last_clean":"2024-03-06T07:28:34.045627+0000","last_became_active":"2024-03-06T07:28:34.044786+0000","last_became_peered":"2024-03-06T07:28:34.044786+0000","last_unstale":"2024-03-06T07:28:34.045627+0000","last_undegraded":"2024-03-06T07:28:34.045627+0000","last_fullsized":"2024-03-06T07:28:34.045627+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:50.885489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.041301+0000","last_change":"2024-03-06T07:28:34.041301+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.041301+0000","last_undegraded":"2024-03-06T07:28:34.041301+0000","last_fullsized":"2024-03-06T07:28:34.041301+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.054609+0000","last_change":"2024-03-06T07:28:34.054092+0000","last_active":"2024-03-06T07:28:34.054609+0000","last_peered":"2024-03-06T07:28:34.054609+0000","last_clean":"2024-03-06T07:28:34.054609+0000","last_became_active":"2024-03-06T07:28:34.053838+0000","last_became_peered":"2024-03-06T07:28:34.053838+0000","last_unstale":"2024-03-06T07:28:34.054609+0000","last_undegraded":"2024-03-06T07:28:34.054609+0000","last_fullsized":"2024-03-06T07:28:34.054609+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:42.229556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033981+0000","last_change":"2024-03-06T07:28:34.033981+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033981+0000","last_undegraded":"2024-03-06T07:28:34.033981+0000","last_fullsized":"2024-03-06T07:28:34.033981+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.050540+0000","last_change":"2024-03-06T07:28:34.046882+0000","last_active":"2024-03-06T07:28:34.050540+0000","last_peered":"2024-03-06T07:28:34.050540+0000","last_clean":"2024-03-06T07:28:34.050540+0000","last_became_active":"2024-03-06T07:28:34.045955+0000","last_became_peered":"2024-03-06T07:28:34.045955+0000","last_unstale":"2024-03-06T07:28:34.050540+0000","last_undegraded":"2024-03-06T07:28:34.050540+0000","last_fullsized":"2024-03-06T07:28:34.050540+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:33.892363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036066+0000","last_change":"2024-03-06T07:28:34.036066+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036066+0000","last_undegraded":"2024-03-06T07:28:34.036066+0000","last_fullsized":"2024-03-06T07:28:34.036066+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.050448+0000","last_change":"2024-03-06T07:28:34.047502+0000","last_active":"2024-03-06T07:28:34.050448+0000","last_peered":"2024-03-06T07:28:34.050448+0000","last_clean":"2024-03-06T07:28:34.050448+0000","last_became_active":"2024-03-06T07:28:34.047242+0000","last_became_peered":"2024-03-06T07:28:34.047242+0000","last_unstale":"2024-03-06T07:28:34.050448+0000","last_undegraded":"2024-03-06T07:28:34.050448+0000","last_fullsized":"2024-03-06T07:28:34.050448+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:39.255500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037043+0000","last_change":"2024-03-06T07:28:34.037043+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037043+0000","last_undegraded":"2024-03-06T07:28:34.037043+0000","last_fullsized":"2024-03-06T07:28:34.037043+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.046728+0000","last_change":"2024-03-06T07:28:34.045868+0000","last_active":"2024-03-06T07:28:34.046728+0000","last_peered":"2024-03-06T07:28:34.046728+0000","last_clean":"2024-03-06T07:28:34.046728+0000","last_became_active":"2024-03-06T07:28:34.045604+0000","last_became_peered":"2024-03-06T07:28:34.045604+0000","last_unstale":"2024-03-06T07:28:34.046728+0000","last_undegraded":"2024-03-06T07:28:34.046728+0000","last_fullsized":"2024-03-06T07:28:34.046728+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:28.693278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037833+0000","last_change":"2024-03-06T07:28:34.037833+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037833+0000","last_undegraded":"2024-03-06T07:28:34.037833+0000","last_fullsized":"2024-03-06T07:28:34.037833+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.048202+0000","last_change":"2024-03-06T07:28:34.047667+0000","last_active":"2024-03-06T07:28:34.048202+0000","last_peered":"2024-03-06T07:28:34.048202+0000","last_clean":"2024-03-06T07:28:34.048202+0000","last_became_active":"2024-03-06T07:28:34.047280+0000","last_became_peered":"2024-03-06T07:28:34.047280+0000","last_unstale":"2024-03-06T07:28:34.048202+0000","last_undegraded":"2024-03-06T07:28:34.048202+0000","last_fullsized":"2024-03-06T07:28:34.048202+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:21.608844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034910+0000","last_change":"2024-03-06T07:28:34.034910+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034910+0000","last_undegraded":"2024-03-06T07:28:34.034910+0000","last_fullsized":"2024-03-06T07:28:34.034910+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053702+0000","last_change":"2024-03-06T07:28:34.049708+0000","last_active":"2024-03-06T07:28:34.053702+0000","last_peered":"2024-03-06T07:28:34.053702+0000","last_clean":"2024-03-06T07:28:34.053702+0000","last_became_active":"2024-03-06T07:28:34.049445+0000","last_became_peered":"2024-03-06T07:28:34.049445+0000","last_unstale":"2024-03-06T07:28:34.053702+0000","last_undegraded":"2024-03-06T07:28:34.053702+0000","last_fullsized":"2024-03-06T07:28:34.053702+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:53.020063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039713+0000","last_change":"2024-03-06T07:28:34.039713+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039713+0000","last_undegraded":"2024-03-06T07:28:34.039713+0000","last_fullsized":"2024-03-06T07:28:34.039713+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.040318+0000","last_change":"2024-03-06T07:28:34.038476+0000","last_active":"2024-03-06T07:28:34.040318+0000","last_peered":"2024-03-06T07:28:34.040318+0000","last_clean":"2024-03-06T07:28:34.040318+0000","last_became_active":"2024-03-06T07:28:34.038197+0000","last_became_peered":"2024-03-06T07:28:34.038197+0000","last_unstale":"2024-03-06T07:28:34.040318+0000","last_undegraded":"2024-03-06T07:28:34.040318+0000","last_fullsized":"2024-03-06T07:28:34.040318+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:49.999763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038359+0000","last_change":"2024-03-06T07:28:34.038359+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038359+0000","last_undegraded":"2024-03-06T07:28:34.038359+0000","last_fullsized":"2024-03-06T07:28:34.038359+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.047046+0000","last_change":"2024-03-06T07:28:34.046180+0000","last_active":"2024-03-06T07:28:34.047046+0000","last_peered":"2024-03-06T07:28:34.047046+0000","last_clean":"2024-03-06T07:28:34.047046+0000","last_became_active":"2024-03-06T07:28:34.045909+0000","last_became_peered":"2024-03-06T07:28:34.045909+0000","last_unstale":"2024-03-06T07:28:34.047046+0000","last_undegraded":"2024-03-06T07:28:34.047046+0000","last_fullsized":"2024-03-06T07:28:34.047046+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:29.293270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032940+0000","last_change":"2024-03-06T07:28:34.032940+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032940+0000","last_undegraded":"2024-03-06T07:28:34.032940+0000","last_fullsized":"2024-03-06T07:28:34.032940+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.048682+0000","last_change":"2024-03-06T07:28:34.043986+0000","last_active":"2024-03-06T07:28:34.048682+0000","last_peered":"2024-03-06T07:28:34.048682+0000","last_clean":"2024-03-06T07:28:34.048682+0000","last_became_active":"2024-03-06T07:28:34.043693+0000","last_became_peered":"2024-03-06T07:28:34.043693+0000","last_unstale":"2024-03-06T07:28:34.048682+0000","last_undegraded":"2024-03-06T07:28:34.048682+0000","last_fullsized":"2024-03-06T07:28:34.048682+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:12.297830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036386+0000","last_change":"2024-03-06T07:28:34.036386+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036386+0000","last_undegraded":"2024-03-06T07:28:34.036386+0000","last_fullsized":"2024-03-06T07:28:34.036386+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053605+0000","last_change":"2024-03-06T07:28:34.047903+0000","last_active":"2024-03-06T07:28:34.053605+0000","last_peered":"2024-03-06T07:28:34.053605+0000","last_clean":"2024-03-06T07:28:34.053605+0000","last_became_active":"2024-03-06T07:28:34.047612+0000","last_became_peered":"2024-03-06T07:28:34.047612+0000","last_unstale":"2024-03-06T07:28:34.053605+0000","last_undegraded":"2024-03-06T07:28:34.053605+0000","last_fullsized":"2024-03-06T07:28:34.053605+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:08.365596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034459+0000","last_change":"2024-03-06T07:28:34.034459+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034459+0000","last_undegraded":"2024-03-06T07:28:34.034459+0000","last_fullsized":"2024-03-06T07:28:34.034459+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.052224+0000","last_change":"2024-03-06T07:28:34.051791+0000","last_active":"2024-03-06T07:28:34.052224+0000","last_peered":"2024-03-06T07:28:34.052224+0000","last_clean":"2024-03-06T07:28:34.052224+0000","last_became_active":"2024-03-06T07:28:34.051472+0000","last_became_peered":"2024-03-06T07:28:34.051472+0000","last_unstale":"2024-03-06T07:28:34.052224+0000","last_undegraded":"2024-03-06T07:28:34.052224+0000","last_fullsized":"2024-03-06T07:28:34.052224+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:50.885489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037431+0000","last_change":"2024-03-06T07:28:34.037431+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037431+0000","last_undegraded":"2024-03-06T07:28:34.037431+0000","last_fullsized":"2024-03-06T07:28:34.037431+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053725+0000","last_change":"2024-03-06T07:28:34.048933+0000","last_active":"2024-03-06T07:28:34.053725+0000","last_peered":"2024-03-06T07:28:34.053725+0000","last_clean":"2024-03-06T07:28:34.053725+0000","last_became_active":"2024-03-06T07:28:34.048670+0000","last_became_peered":"2024-03-06T07:28:34.048670+0000","last_unstale":"2024-03-06T07:28:34.053725+0000","last_undegraded":"2024-03-06T07:28:34.053725+0000","last_fullsized":"2024-03-06T07:28:34.053725+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:08.365596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037745+0000","last_change":"2024-03-06T07:28:34.037745+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037745+0000","last_undegraded":"2024-03-06T07:28:34.037745+0000","last_fullsized":"2024-03-06T07:28:34.037745+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053497+0000","last_change":"2024-03-06T07:28:34.049313+0000","last_active":"2024-03-06T07:28:34.053497+0000","last_peered":"2024-03-06T07:28:34.053497+0000","last_clean":"2024-03-06T07:28:34.053497+0000","last_became_active":"2024-03-06T07:28:34.048967+0000","last_became_peered":"2024-03-06T07:28:34.048967+0000","last_unstale":"2024-03-06T07:28:34.053497+0000","last_undegraded":"2024-03-06T07:28:34.053497+0000","last_fullsized":"2024-03-06T07:28:34.053497+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:51.180609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":506,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.032085+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:28:34.032085+0000","last_peered":"2024-03-06T07:28:34.032085+0000","last_clean":"2024-03-06T07:28:34.032085+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:28:34.032085+0000","last_undegraded":"2024-03-06T07:28:34.032085+0000","last_fullsized":"2024-03-06T07:28:34.032085+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574951,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33664,"kb_used_data":2480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338176,"statfs":{"total":96636764160,"available":96602292224,"internally_reserved":0,"allocated":2539520,"data_stored":1793319,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574950,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33584,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338256,"statfs":{"total":96636764160,"available":96602374144,"internally_reserved":0,"allocated":2461696,"data_stored":1196284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574950,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33604,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338236,"statfs":{"total":96636764160,"available":96602353664,"internally_reserved":0,"allocated":2473984,"data_stored":1202951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574951,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33600,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338240,"statfs":{"total":96636764160,"available":96602357760,"internally_reserved":0,"allocated":2473984,"data_stored":1202951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574950,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33600,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338240,"statfs":{"total":96636764160,"available":96602357760,"internally_reserved":0,"allocated":2473984,"data_stored":1202951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574950,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37744,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334096,"statfs":{"total":96636764160,"available":96598114304,"internally_reserved":0,"allocated":2461696,"data_stored":1196284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574951,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33600,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338240,"statfs":{"total":96636764160,"available":96602357760,"internally_reserved":0,"allocated":2473984,"data_stored":1202951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574951,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37812,"kb_used_data":2468,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334028,"statfs":{"total":96636764160,"available":96598044672,"internally_reserved":0,"allocated":2527232,"data_stored":1786652,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:38.519 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:38.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.655+0000 7fd716b13640 1 Processor -- start 2024-03-06T07:28:38.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.655+0000 7fd716b13640 1 -- start start 2024-03-06T07:28:38.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.656+0000 7fd716b13640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd710152650 0x7fd710155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.656+0000 7fd716b13640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7100bf9d0 0x7fd7100c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.656+0000 7fd716b13640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7100c2fc0 con 0x7fd7100c6b10 2024-03-06T07:28:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.656+0000 7fd716b13640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd710155800 con 0x7fd7100bf9d0 2024-03-06T07:28:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.656+0000 7fd716b13640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd710158da0 con 0x7fd710152650 2024-03-06T07:28:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.656+0000 7fd7150b0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7100bf9d0 0x7fd7100c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.656+0000 7fd70ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd7100c6b10 0x7fd710158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45236/0 (socket says 172.21.15.1:45236) 2024-03-06T07:28:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.656+0000 7fd70ffff640 1 -- 172.21.15.1:0/56996286 learned_addr learned my addr 172.21.15.1:0/56996286 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.656+0000 7fd7148af640 1 --2- 172.21.15.1:0/56996286 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd710152650 0x7fd710155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.657+0000 7fd7150b0640 1 -- 172.21.15.1:0/56996286 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd710152650 msgr2=0x7fd710155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.657+0000 7fd7150b0640 1 --2- 172.21.15.1:0/56996286 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd710152650 0x7fd710155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.657+0000 7fd7150b0640 1 -- 172.21.15.1:0/56996286 >> v1:172.21.15.1:6789/0 conn(0x7fd7100c6b10 legacy=0x7fd710158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.657+0000 7fd7150b0640 1 -- 172.21.15.1:0/56996286 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7100c6690 con 0x7fd7100bf9d0 2024-03-06T07:28:38.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.657+0000 7fd7148af640 1 --2- 172.21.15.1:0/56996286 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd710152650 0x7fd710155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:38.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.657+0000 7fd7150b0640 1 --2- 172.21.15.1:0/56996286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7100bf9d0 0x7fd7100c1460 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fd708009d90 tx=0x7fd70802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609fa803a12206de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.657+0000 7fd70f7fe640 1 -- 172.21.15.1:0/56996286 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd708039070 con 0x7fd7100bf9d0 2024-03-06T07:28:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.657+0000 7fd70f7fe640 1 -- 172.21.15.1:0/56996286 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd70802cbc0 con 0x7fd7100bf9d0 2024-03-06T07:28:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.657+0000 7fd70f7fe640 1 -- 172.21.15.1:0/56996286 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd708034440 con 0x7fd7100bf9d0 2024-03-06T07:28:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.658+0000 7fd716b13640 1 -- 172.21.15.1:0/56996286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7100bf9d0 msgr2=0x7fd7100c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.658+0000 7fd716b13640 1 --2- 172.21.15.1:0/56996286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7100bf9d0 0x7fd7100c1460 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fd708009d90 tx=0x7fd70802be10 comp rx=0 tx=0).stop 2024-03-06T07:28:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.658+0000 7fd716b13640 1 -- 172.21.15.1:0/56996286 shutdown_connections 2024-03-06T07:28:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.658+0000 7fd716b13640 1 --2- 172.21.15.1:0/56996286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7100bf9d0 0x7fd7100c1460 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.658+0000 7fd716b13640 1 --2- 172.21.15.1:0/56996286 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd710152650 0x7fd710155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.658+0000 7fd716b13640 1 -- 172.21.15.1:0/56996286 >> 172.21.15.1:0/56996286 conn(0x7fd710066ee0 msgr2=0x7fd7100672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.658+0000 7fd716b13640 1 -- 172.21.15.1:0/56996286 shutdown_connections 2024-03-06T07:28:38.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.658+0000 7fd716b13640 1 -- 172.21.15.1:0/56996286 wait complete. 2024-03-06T07:28:38.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.660+0000 7fd716b13640 1 Processor -- start 2024-03-06T07:28:38.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.660+0000 7fd716b13640 1 -- start start 2024-03-06T07:28:38.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.661+0000 7fd716b13640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7100bf9d0 0x7fd7100c1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.661+0000 7fd716b13640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd710152650 0x7fd7100c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.661+0000 7fd716b13640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7100c9d50 0x7fd7100c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.661+0000 7fd716b13640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7101551b0 con 0x7fd7100c9d50 2024-03-06T07:28:38.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.661+0000 7fd716b13640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7100c6fa0 con 0x7fd7100bf9d0 2024-03-06T07:28:38.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.661+0000 7fd716b13640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd710155490 con 0x7fd710152650 2024-03-06T07:28:38.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.661+0000 7fd7150b0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7100c9d50 0x7fd7100c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.661+0000 7fd7150b0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7100c9d50 0x7fd7100c8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54560/0 (socket says 172.21.15.1:54560) 2024-03-06T07:28:38.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.661+0000 7fd7150b0640 1 -- 172.21.15.1:0/1006044027 learned_addr learned my addr 172.21.15.1:0/1006044027 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:38.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.661+0000 7fd7148af640 1 --2- 172.21.15.1:0/1006044027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7100bf9d0 0x7fd7100c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.661+0000 7fd70ffff640 1 --2- 172.21.15.1:0/1006044027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd710152650 0x7fd7100c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.661+0000 7fd7150b0640 1 -- 172.21.15.1:0/1006044027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd710152650 msgr2=0x7fd7100c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.661+0000 7fd7150b0640 1 --2- 172.21.15.1:0/1006044027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd710152650 0x7fd7100c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.661+0000 7fd7150b0640 1 -- 172.21.15.1:0/1006044027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7100bf9d0 msgr2=0x7fd7100c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.661+0000 7fd7150b0640 1 --2- 172.21.15.1:0/1006044027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7100bf9d0 0x7fd7100c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.661+0000 7fd7150b0640 1 -- 172.21.15.1:0/1006044027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7100ca340 con 0x7fd7100c9d50 2024-03-06T07:28:38.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.661+0000 7fd7148af640 1 --2- 172.21.15.1:0/1006044027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7100bf9d0 0x7fd7100c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:38.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.662+0000 7fd7150b0640 1 --2- 172.21.15.1:0/1006044027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7100c9d50 0x7fd7100c8d10 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fd708009eb0 tx=0x7fd708004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.662+0000 7fd70d7fa640 1 -- 172.21.15.1:0/1006044027 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd708039050 con 0x7fd7100c9d50 2024-03-06T07:28:38.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.662+0000 7fd70d7fa640 1 -- 172.21.15.1:0/1006044027 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd708045070 con 0x7fd7100c9d50 2024-03-06T07:28:38.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.662+0000 7fd716b13640 1 -- 172.21.15.1:0/1006044027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7100ca620 con 0x7fd7100c9d50 2024-03-06T07:28:38.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.662+0000 7fd70d7fa640 1 -- 172.21.15.1:0/1006044027 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70803e830 con 0x7fd7100c9d50 2024-03-06T07:28:38.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.662+0000 7fd716b13640 1 -- 172.21.15.1:0/1006044027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7101585d0 con 0x7fd7100c9d50 2024-03-06T07:28:38.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.663+0000 7fd716b13640 1 -- 172.21.15.1:0/1006044027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd710067fd0 con 0x7fd7100c9d50 2024-03-06T07:28:38.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.664+0000 7fd70d7fa640 1 -- 172.21.15.1:0/1006044027 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd70803e9d0 con 0x7fd7100c9d50 2024-03-06T07:28:38.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.664+0000 7fd70d7fa640 1 --2- 172.21.15.1:0/1006044027 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd6fc06b4d0 0x7fd6fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.664+0000 7fd70d7fa640 1 -- 172.21.15.1:0/1006044027 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd7080ad2d0 con 0x7fd7100c9d50 2024-03-06T07:28:38.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.664+0000 7fd7148af640 1 --2- 172.21.15.1:0/1006044027 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd6fc06b4d0 0x7fd6fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.665+0000 7fd7148af640 1 --2- 172.21.15.1:0/1006044027 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd6fc06b4d0 0x7fd6fc06d990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fd7101e1510 tx=0x7fd700008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.669+0000 7fd70d7fa640 1 -- 172.21.15.1:0/1006044027 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd70807c170 con 0x7fd7100c9d50 2024-03-06T07:28:38.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.846+0000 7fd716b13640 1 -- 172.21.15.1:0/1006044027 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd7100b06f0 con 0x7fd6fc06b4d0 2024-03-06T07:28:38.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.852+0000 7fd70d7fa640 1 -- 172.21.15.1:0/1006044027 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247177 (secure 0 0 0) 0x7fd7100b06f0 con 0x7fd6fc06b4d0 2024-03-06T07:28:38.857 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:38.858 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:28:38.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.855+0000 7fd716b13640 1 -- 172.21.15.1:0/1006044027 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd6fc06b4d0 msgr2=0x7fd6fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.855+0000 7fd716b13640 1 --2- 172.21.15.1:0/1006044027 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd6fc06b4d0 0x7fd6fc06d990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fd7101e1510 tx=0x7fd700008040 comp rx=0 tx=0).stop 2024-03-06T07:28:38.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.855+0000 7fd716b13640 1 -- 172.21.15.1:0/1006044027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7100c9d50 msgr2=0x7fd7100c8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.855+0000 7fd716b13640 1 --2- 172.21.15.1:0/1006044027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7100c9d50 0x7fd7100c8d10 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fd708009eb0 tx=0x7fd708004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:38.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.856+0000 7fd716b13640 1 -- 172.21.15.1:0/1006044027 shutdown_connections 2024-03-06T07:28:38.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.856+0000 7fd716b13640 1 --2- 172.21.15.1:0/1006044027 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd6fc06b4d0 0x7fd6fc06d990 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.856+0000 7fd716b13640 1 --2- 172.21.15.1:0/1006044027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7100c9d50 0x7fd7100c8d10 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.856+0000 7fd716b13640 1 --2- 172.21.15.1:0/1006044027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd710152650 0x7fd7100c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.856+0000 7fd716b13640 1 --2- 172.21.15.1:0/1006044027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7100bf9d0 0x7fd7100c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.856+0000 7fd716b13640 1 -- 172.21.15.1:0/1006044027 >> 172.21.15.1:0/1006044027 conn(0x7fd710066ee0 msgr2=0x7fd7100c5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.856+0000 7fd716b13640 1 -- 172.21.15.1:0/1006044027 shutdown_connections 2024-03-06T07:28:38.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:38.856+0000 7fd716b13640 1 -- 172.21.15.1:0/1006044027 wait complete. 2024-03-06T07:28:38.868 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-03-06T07:28:38.807920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":277208,"kb_used_data":19420,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754697512,"statfs":{"total":773094113280,"available":772810252288,"internally_reserved":0,"allocated":19886080,"data_stored":10784343,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.785348"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039773+0000","last_change":"2024-03-06T07:28:34.039773+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039773+0000","last_undegraded":"2024-03-06T07:28:34.039773+0000","last_fullsized":"2024-03-06T07:28:34.039773+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032560+0000","last_change":"2024-03-06T07:28:34.032560+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032560+0000","last_undegraded":"2024-03-06T07:28:34.032560+0000","last_fullsized":"2024-03-06T07:28:34.032560+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.043939+0000","last_change":"2024-03-06T07:28:34.043939+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.043939+0000","last_undegraded":"2024-03-06T07:28:34.043939+0000","last_fullsized":"2024-03-06T07:28:34.043939+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.040918+0000","last_change":"2024-03-06T07:28:34.040918+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.040918+0000","last_undegraded":"2024-03-06T07:28:34.040918+0000","last_fullsized":"2024-03-06T07:28:34.040918+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034672+0000","last_change":"2024-03-06T07:28:34.034672+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034672+0000","last_undegraded":"2024-03-06T07:28:34.034672+0000","last_fullsized":"2024-03-06T07:28:34.034672+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036995+0000","last_change":"2024-03-06T07:28:34.036995+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036995+0000","last_undegraded":"2024-03-06T07:28:34.036995+0000","last_fullsized":"2024-03-06T07:28:34.036995+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037384+0000","last_change":"2024-03-06T07:28:34.037384+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037384+0000","last_undegraded":"2024-03-06T07:28:34.037384+0000","last_fullsized":"2024-03-06T07:28:34.037384+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036819+0000","last_change":"2024-03-06T07:28:34.036819+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036819+0000","last_undegraded":"2024-03-06T07:28:34.036819+0000","last_fullsized":"2024-03-06T07:28:34.036819+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038840+0000","last_change":"2024-03-06T07:28:34.038840+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038840+0000","last_undegraded":"2024-03-06T07:28:34.038840+0000","last_fullsized":"2024-03-06T07:28:34.038840+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.043705+0000","last_change":"2024-03-06T07:28:34.043705+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.043705+0000","last_undegraded":"2024-03-06T07:28:34.043705+0000","last_fullsized":"2024-03-06T07:28:34.043705+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.041796+0000","last_change":"2024-03-06T07:28:34.041796+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.041796+0000","last_undegraded":"2024-03-06T07:28:34.041796+0000","last_fullsized":"2024-03-06T07:28:34.041796+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038852+0000","last_change":"2024-03-06T07:28:34.038852+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038852+0000","last_undegraded":"2024-03-06T07:28:34.038852+0000","last_fullsized":"2024-03-06T07:28:34.038852+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.035743+0000","last_change":"2024-03-06T07:28:34.035743+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.035743+0000","last_undegraded":"2024-03-06T07:28:34.035743+0000","last_fullsized":"2024-03-06T07:28:34.035743+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033521+0000","last_change":"2024-03-06T07:28:34.033521+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033521+0000","last_undegraded":"2024-03-06T07:28:34.033521+0000","last_fullsized":"2024-03-06T07:28:34.033521+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033924+0000","last_change":"2024-03-06T07:28:34.033924+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033924+0000","last_undegraded":"2024-03-06T07:28:34.033924+0000","last_fullsized":"2024-03-06T07:28:34.033924+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038810+0000","last_change":"2024-03-06T07:28:34.038810+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038810+0000","last_undegraded":"2024-03-06T07:28:34.038810+0000","last_fullsized":"2024-03-06T07:28:34.038810+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036613+0000","last_change":"2024-03-06T07:28:34.036613+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036613+0000","last_undegraded":"2024-03-06T07:28:34.036613+0000","last_fullsized":"2024-03-06T07:28:34.036613+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.043282+0000","last_change":"2024-03-06T07:28:34.043282+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.043282+0000","last_undegraded":"2024-03-06T07:28:34.043282+0000","last_fullsized":"2024-03-06T07:28:34.043282+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033099+0000","last_change":"2024-03-06T07:28:34.033099+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033099+0000","last_undegraded":"2024-03-06T07:28:34.033099+0000","last_fullsized":"2024-03-06T07:28:34.033099+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.031957+0000","last_change":"2024-03-06T07:28:34.031957+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.031957+0000","last_undegraded":"2024-03-06T07:28:34.031957+0000","last_fullsized":"2024-03-06T07:28:34.031957+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037603+0000","last_change":"2024-03-06T07:28:34.037603+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037603+0000","last_undegraded":"2024-03-06T07:28:34.037603+0000","last_fullsized":"2024-03-06T07:28:34.037603+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036930+0000","last_change":"2024-03-06T07:28:34.036930+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036930+0000","last_undegraded":"2024-03-06T07:28:34.036930+0000","last_fullsized":"2024-03-06T07:28:34.036930+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033464+0000","last_change":"2024-03-06T07:28:34.033464+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033464+0000","last_undegraded":"2024-03-06T07:28:34.033464+0000","last_fullsized":"2024-03-06T07:28:34.033464+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037784+0000","last_change":"2024-03-06T07:28:34.037784+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037784+0000","last_undegraded":"2024-03-06T07:28:34.037784+0000","last_fullsized":"2024-03-06T07:28:34.037784+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.031482+0000","last_change":"2024-03-06T07:28:34.031482+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.031482+0000","last_undegraded":"2024-03-06T07:28:34.031482+0000","last_fullsized":"2024-03-06T07:28:34.031482+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.040505+0000","last_change":"2024-03-06T07:28:34.040505+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.040505+0000","last_undegraded":"2024-03-06T07:28:34.040505+0000","last_fullsized":"2024-03-06T07:28:34.040505+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034359+0000","last_change":"2024-03-06T07:28:34.034359+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034359+0000","last_undegraded":"2024-03-06T07:28:34.034359+0000","last_fullsized":"2024-03-06T07:28:34.034359+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038455+0000","last_change":"2024-03-06T07:28:34.038455+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038455+0000","last_undegraded":"2024-03-06T07:28:34.038455+0000","last_fullsized":"2024-03-06T07:28:34.038455+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032251+0000","last_change":"2024-03-06T07:28:34.032251+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032251+0000","last_undegraded":"2024-03-06T07:28:34.032251+0000","last_fullsized":"2024-03-06T07:28:34.032251+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032247+0000","last_change":"2024-03-06T07:28:34.032247+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032247+0000","last_undegraded":"2024-03-06T07:28:34.032247+0000","last_fullsized":"2024-03-06T07:28:34.032247+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.031492+0000","last_change":"2024-03-06T07:28:34.031492+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.031492+0000","last_undegraded":"2024-03-06T07:28:34.031492+0000","last_fullsized":"2024-03-06T07:28:34.031492+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036448+0000","last_change":"2024-03-06T07:28:34.036448+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036448+0000","last_undegraded":"2024-03-06T07:28:34.036448+0000","last_fullsized":"2024-03-06T07:28:34.036448+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032461+0000","last_change":"2024-03-06T07:28:34.032461+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032461+0000","last_undegraded":"2024-03-06T07:28:34.032461+0000","last_fullsized":"2024-03-06T07:28:34.032461+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.046617+0000","last_change":"2024-03-06T07:28:34.045692+0000","last_active":"2024-03-06T07:28:34.046617+0000","last_peered":"2024-03-06T07:28:34.046617+0000","last_clean":"2024-03-06T07:28:34.046617+0000","last_became_active":"2024-03-06T07:28:34.045392+0000","last_became_peered":"2024-03-06T07:28:34.045392+0000","last_unstale":"2024-03-06T07:28:34.046617+0000","last_undegraded":"2024-03-06T07:28:34.046617+0000","last_fullsized":"2024-03-06T07:28:34.046617+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:59.770341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032614+0000","last_change":"2024-03-06T07:28:34.032614+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032614+0000","last_undegraded":"2024-03-06T07:28:34.032614+0000","last_fullsized":"2024-03-06T07:28:34.032614+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053547+0000","last_change":"2024-03-06T07:28:34.048342+0000","last_active":"2024-03-06T07:28:34.053547+0000","last_peered":"2024-03-06T07:28:34.053547+0000","last_clean":"2024-03-06T07:28:34.053547+0000","last_became_active":"2024-03-06T07:28:34.048077+0000","last_became_peered":"2024-03-06T07:28:34.048077+0000","last_unstale":"2024-03-06T07:28:34.053547+0000","last_undegraded":"2024-03-06T07:28:34.053547+0000","last_fullsized":"2024-03-06T07:28:34.053547+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:02.569692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038496+0000","last_change":"2024-03-06T07:28:34.038496+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038496+0000","last_undegraded":"2024-03-06T07:28:34.038496+0000","last_fullsized":"2024-03-06T07:28:34.038496+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.044121+0000","last_change":"2024-03-06T07:28:34.043250+0000","last_active":"2024-03-06T07:28:34.044121+0000","last_peered":"2024-03-06T07:28:34.044121+0000","last_clean":"2024-03-06T07:28:34.044121+0000","last_became_active":"2024-03-06T07:28:34.042985+0000","last_became_peered":"2024-03-06T07:28:34.042985+0000","last_unstale":"2024-03-06T07:28:34.044121+0000","last_undegraded":"2024-03-06T07:28:34.044121+0000","last_fullsized":"2024-03-06T07:28:34.044121+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:49.778788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032526+0000","last_change":"2024-03-06T07:28:34.032526+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032526+0000","last_undegraded":"2024-03-06T07:28:34.032526+0000","last_fullsized":"2024-03-06T07:28:34.032526+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.040595+0000","last_change":"2024-03-06T07:28:34.039018+0000","last_active":"2024-03-06T07:28:34.040595+0000","last_peered":"2024-03-06T07:28:34.040595+0000","last_clean":"2024-03-06T07:28:34.040595+0000","last_became_active":"2024-03-06T07:28:34.038734+0000","last_became_peered":"2024-03-06T07:28:34.038734+0000","last_unstale":"2024-03-06T07:28:34.040595+0000","last_undegraded":"2024-03-06T07:28:34.040595+0000","last_fullsized":"2024-03-06T07:28:34.040595+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:56.463129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039197+0000","last_change":"2024-03-06T07:28:34.039197+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039197+0000","last_undegraded":"2024-03-06T07:28:34.039197+0000","last_fullsized":"2024-03-06T07:28:34.039197+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.046080+0000","last_change":"2024-03-06T07:28:34.044403+0000","last_active":"2024-03-06T07:28:34.046080+0000","last_peered":"2024-03-06T07:28:34.046080+0000","last_clean":"2024-03-06T07:28:34.046080+0000","last_became_active":"2024-03-06T07:28:34.044130+0000","last_became_peered":"2024-03-06T07:28:34.044130+0000","last_unstale":"2024-03-06T07:28:34.046080+0000","last_undegraded":"2024-03-06T07:28:34.046080+0000","last_fullsized":"2024-03-06T07:28:34.046080+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:18.838141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.044109+0000","last_change":"2024-03-06T07:28:34.044109+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.044109+0000","last_undegraded":"2024-03-06T07:28:34.044109+0000","last_fullsized":"2024-03-06T07:28:34.044109+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.054615+0000","last_change":"2024-03-06T07:28:34.054170+0000","last_active":"2024-03-06T07:28:34.054615+0000","last_peered":"2024-03-06T07:28:34.054615+0000","last_clean":"2024-03-06T07:28:34.054615+0000","last_became_active":"2024-03-06T07:28:34.053871+0000","last_became_peered":"2024-03-06T07:28:34.053871+0000","last_unstale":"2024-03-06T07:28:34.054615+0000","last_undegraded":"2024-03-06T07:28:34.054615+0000","last_fullsized":"2024-03-06T07:28:34.054615+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:29.552066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.035597+0000","last_change":"2024-03-06T07:28:34.035597+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.035597+0000","last_undegraded":"2024-03-06T07:28:34.035597+0000","last_fullsized":"2024-03-06T07:28:34.035597+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.046191+0000","last_change":"2024-03-06T07:28:34.042955+0000","last_active":"2024-03-06T07:28:34.046191+0000","last_peered":"2024-03-06T07:28:34.046191+0000","last_clean":"2024-03-06T07:28:34.046191+0000","last_became_active":"2024-03-06T07:28:34.042684+0000","last_became_peered":"2024-03-06T07:28:34.042684+0000","last_unstale":"2024-03-06T07:28:34.046191+0000","last_undegraded":"2024-03-06T07:28:34.046191+0000","last_fullsized":"2024-03-06T07:28:34.046191+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:04.206156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037329+0000","last_change":"2024-03-06T07:28:34.037329+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037329+0000","last_undegraded":"2024-03-06T07:28:34.037329+0000","last_fullsized":"2024-03-06T07:28:34.037329+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.037420+0000","last_change":"2024-03-06T07:28:34.036082+0000","last_active":"2024-03-06T07:28:34.037420+0000","last_peered":"2024-03-06T07:28:34.037420+0000","last_clean":"2024-03-06T07:28:34.037420+0000","last_became_active":"2024-03-06T07:28:34.035802+0000","last_became_peered":"2024-03-06T07:28:34.035802+0000","last_unstale":"2024-03-06T07:28:34.037420+0000","last_undegraded":"2024-03-06T07:28:34.037420+0000","last_fullsized":"2024-03-06T07:28:34.037420+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:00.673585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.035539+0000","last_change":"2024-03-06T07:28:34.035539+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.035539+0000","last_undegraded":"2024-03-06T07:28:34.035539+0000","last_fullsized":"2024-03-06T07:28:34.035539+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.051779+0000","last_change":"2024-03-06T07:28:34.051147+0000","last_active":"2024-03-06T07:28:34.051779+0000","last_peered":"2024-03-06T07:28:34.051779+0000","last_clean":"2024-03-06T07:28:34.051779+0000","last_became_active":"2024-03-06T07:28:34.050807+0000","last_became_peered":"2024-03-06T07:28:34.050807+0000","last_unstale":"2024-03-06T07:28:34.051779+0000","last_undegraded":"2024-03-06T07:28:34.051779+0000","last_fullsized":"2024-03-06T07:28:34.051779+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:56.463129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039302+0000","last_change":"2024-03-06T07:28:34.039302+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039302+0000","last_undegraded":"2024-03-06T07:28:34.039302+0000","last_fullsized":"2024-03-06T07:28:34.039302+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.047597+0000","last_change":"2024-03-06T07:28:34.044501+0000","last_active":"2024-03-06T07:28:34.047597+0000","last_peered":"2024-03-06T07:28:34.047597+0000","last_clean":"2024-03-06T07:28:34.047597+0000","last_became_active":"2024-03-06T07:28:34.044057+0000","last_became_peered":"2024-03-06T07:28:34.044057+0000","last_unstale":"2024-03-06T07:28:34.047597+0000","last_undegraded":"2024-03-06T07:28:34.047597+0000","last_fullsized":"2024-03-06T07:28:34.047597+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:33.892363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037969+0000","last_change":"2024-03-06T07:28:34.037969+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037969+0000","last_undegraded":"2024-03-06T07:28:34.037969+0000","last_fullsized":"2024-03-06T07:28:34.037969+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.042936+0000","last_change":"2024-03-06T07:28:34.041503+0000","last_active":"2024-03-06T07:28:34.042936+0000","last_peered":"2024-03-06T07:28:34.042936+0000","last_clean":"2024-03-06T07:28:34.042936+0000","last_became_active":"2024-03-06T07:28:34.041217+0000","last_became_peered":"2024-03-06T07:28:34.041217+0000","last_unstale":"2024-03-06T07:28:34.042936+0000","last_undegraded":"2024-03-06T07:28:34.042936+0000","last_fullsized":"2024-03-06T07:28:34.042936+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:14.748127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032982+0000","last_change":"2024-03-06T07:28:34.032982+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032982+0000","last_undegraded":"2024-03-06T07:28:34.032982+0000","last_fullsized":"2024-03-06T07:28:34.032982+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.050323+0000","last_change":"2024-03-06T07:28:34.048173+0000","last_active":"2024-03-06T07:28:34.050323+0000","last_peered":"2024-03-06T07:28:34.050323+0000","last_clean":"2024-03-06T07:28:34.050323+0000","last_became_active":"2024-03-06T07:28:34.047879+0000","last_became_peered":"2024-03-06T07:28:34.047879+0000","last_unstale":"2024-03-06T07:28:34.050323+0000","last_undegraded":"2024-03-06T07:28:34.050323+0000","last_fullsized":"2024-03-06T07:28:34.050323+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:42.229556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039603+0000","last_change":"2024-03-06T07:28:34.039603+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039603+0000","last_undegraded":"2024-03-06T07:28:34.039603+0000","last_fullsized":"2024-03-06T07:28:34.039603+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.047805+0000","last_change":"2024-03-06T07:28:34.045231+0000","last_active":"2024-03-06T07:28:34.047805+0000","last_peered":"2024-03-06T07:28:34.047805+0000","last_clean":"2024-03-06T07:28:34.047805+0000","last_became_active":"2024-03-06T07:28:34.044892+0000","last_became_peered":"2024-03-06T07:28:34.044892+0000","last_unstale":"2024-03-06T07:28:34.047805+0000","last_undegraded":"2024-03-06T07:28:34.047805+0000","last_fullsized":"2024-03-06T07:28:34.047805+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:02.569692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032242+0000","last_change":"2024-03-06T07:28:34.032242+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032242+0000","last_undegraded":"2024-03-06T07:28:34.032242+0000","last_fullsized":"2024-03-06T07:28:34.032242+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.047690+0000","last_change":"2024-03-06T07:28:34.045006+0000","last_active":"2024-03-06T07:28:34.047690+0000","last_peered":"2024-03-06T07:28:34.047690+0000","last_clean":"2024-03-06T07:28:34.047690+0000","last_became_active":"2024-03-06T07:28:34.044739+0000","last_became_peered":"2024-03-06T07:28:34.044739+0000","last_unstale":"2024-03-06T07:28:34.047690+0000","last_undegraded":"2024-03-06T07:28:34.047690+0000","last_fullsized":"2024-03-06T07:28:34.047690+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:39.255500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036198+0000","last_change":"2024-03-06T07:28:34.036198+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036198+0000","last_undegraded":"2024-03-06T07:28:34.036198+0000","last_fullsized":"2024-03-06T07:28:34.036198+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.051110+0000","last_change":"2024-03-06T07:28:34.050123+0000","last_active":"2024-03-06T07:28:34.051110+0000","last_peered":"2024-03-06T07:28:34.051110+0000","last_clean":"2024-03-06T07:28:34.051110+0000","last_became_active":"2024-03-06T07:28:34.049801+0000","last_became_peered":"2024-03-06T07:28:34.049801+0000","last_unstale":"2024-03-06T07:28:34.051110+0000","last_undegraded":"2024-03-06T07:28:34.051110+0000","last_fullsized":"2024-03-06T07:28:34.051110+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:41.920850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037352+0000","last_change":"2024-03-06T07:28:34.037352+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037352+0000","last_undegraded":"2024-03-06T07:28:34.037352+0000","last_fullsized":"2024-03-06T07:28:34.037352+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.048452+0000","last_change":"2024-03-06T07:28:34.047655+0000","last_active":"2024-03-06T07:28:34.048452+0000","last_peered":"2024-03-06T07:28:34.048452+0000","last_clean":"2024-03-06T07:28:34.048452+0000","last_became_active":"2024-03-06T07:28:34.047397+0000","last_became_peered":"2024-03-06T07:28:34.047397+0000","last_unstale":"2024-03-06T07:28:34.048452+0000","last_undegraded":"2024-03-06T07:28:34.048452+0000","last_fullsized":"2024-03-06T07:28:34.048452+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:51.253409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034884+0000","last_change":"2024-03-06T07:28:34.034884+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034884+0000","last_undegraded":"2024-03-06T07:28:34.034884+0000","last_fullsized":"2024-03-06T07:28:34.034884+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.050554+0000","last_change":"2024-03-06T07:28:34.049637+0000","last_active":"2024-03-06T07:28:34.050554+0000","last_peered":"2024-03-06T07:28:34.050554+0000","last_clean":"2024-03-06T07:28:34.050554+0000","last_became_active":"2024-03-06T07:28:34.049301+0000","last_became_peered":"2024-03-06T07:28:34.049301+0000","last_unstale":"2024-03-06T07:28:34.050554+0000","last_undegraded":"2024-03-06T07:28:34.050554+0000","last_fullsized":"2024-03-06T07:28:34.050554+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:03.780188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033456+0000","last_change":"2024-03-06T07:28:34.033456+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033456+0000","last_undegraded":"2024-03-06T07:28:34.033456+0000","last_fullsized":"2024-03-06T07:28:34.033456+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.049420+0000","last_change":"2024-03-06T07:28:34.046419+0000","last_active":"2024-03-06T07:28:34.049420+0000","last_peered":"2024-03-06T07:28:34.049420+0000","last_clean":"2024-03-06T07:28:34.049420+0000","last_became_active":"2024-03-06T07:28:34.046066+0000","last_became_peered":"2024-03-06T07:28:34.046066+0000","last_unstale":"2024-03-06T07:28:34.049420+0000","last_undegraded":"2024-03-06T07:28:34.049420+0000","last_fullsized":"2024-03-06T07:28:34.049420+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:00.025032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.040463+0000","last_change":"2024-03-06T07:28:34.040463+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.040463+0000","last_undegraded":"2024-03-06T07:28:34.040463+0000","last_fullsized":"2024-03-06T07:28:34.040463+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.045627+0000","last_change":"2024-03-06T07:28:34.045103+0000","last_active":"2024-03-06T07:28:34.045627+0000","last_peered":"2024-03-06T07:28:34.045627+0000","last_clean":"2024-03-06T07:28:34.045627+0000","last_became_active":"2024-03-06T07:28:34.044786+0000","last_became_peered":"2024-03-06T07:28:34.044786+0000","last_unstale":"2024-03-06T07:28:34.045627+0000","last_undegraded":"2024-03-06T07:28:34.045627+0000","last_fullsized":"2024-03-06T07:28:34.045627+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:50.885489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.041301+0000","last_change":"2024-03-06T07:28:34.041301+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.041301+0000","last_undegraded":"2024-03-06T07:28:34.041301+0000","last_fullsized":"2024-03-06T07:28:34.041301+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.054609+0000","last_change":"2024-03-06T07:28:34.054092+0000","last_active":"2024-03-06T07:28:34.054609+0000","last_peered":"2024-03-06T07:28:34.054609+0000","last_clean":"2024-03-06T07:28:34.054609+0000","last_became_active":"2024-03-06T07:28:34.053838+0000","last_became_peered":"2024-03-06T07:28:34.053838+0000","last_unstale":"2024-03-06T07:28:34.054609+0000","last_undegraded":"2024-03-06T07:28:34.054609+0000","last_fullsized":"2024-03-06T07:28:34.054609+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:42.229556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033981+0000","last_change":"2024-03-06T07:28:34.033981+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033981+0000","last_undegraded":"2024-03-06T07:28:34.033981+0000","last_fullsized":"2024-03-06T07:28:34.033981+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.050540+0000","last_change":"2024-03-06T07:28:34.046882+0000","last_active":"2024-03-06T07:28:34.050540+0000","last_peered":"2024-03-06T07:28:34.050540+0000","last_clean":"2024-03-06T07:28:34.050540+0000","last_became_active":"2024-03-06T07:28:34.045955+0000","last_became_peered":"2024-03-06T07:28:34.045955+0000","last_unstale":"2024-03-06T07:28:34.050540+0000","last_undegraded":"2024-03-06T07:28:34.050540+0000","last_fullsized":"2024-03-06T07:28:34.050540+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:33.892363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036066+0000","last_change":"2024-03-06T07:28:34.036066+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036066+0000","last_undegraded":"2024-03-06T07:28:34.036066+0000","last_fullsized":"2024-03-06T07:28:34.036066+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.050448+0000","last_change":"2024-03-06T07:28:34.047502+0000","last_active":"2024-03-06T07:28:34.050448+0000","last_peered":"2024-03-06T07:28:34.050448+0000","last_clean":"2024-03-06T07:28:34.050448+0000","last_became_active":"2024-03-06T07:28:34.047242+0000","last_became_peered":"2024-03-06T07:28:34.047242+0000","last_unstale":"2024-03-06T07:28:34.050448+0000","last_undegraded":"2024-03-06T07:28:34.050448+0000","last_fullsized":"2024-03-06T07:28:34.050448+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:39.255500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037043+0000","last_change":"2024-03-06T07:28:34.037043+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037043+0000","last_undegraded":"2024-03-06T07:28:34.037043+0000","last_fullsized":"2024-03-06T07:28:34.037043+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.046728+0000","last_change":"2024-03-06T07:28:34.045868+0000","last_active":"2024-03-06T07:28:34.046728+0000","last_peered":"2024-03-06T07:28:34.046728+0000","last_clean":"2024-03-06T07:28:34.046728+0000","last_became_active":"2024-03-06T07:28:34.045604+0000","last_became_peered":"2024-03-06T07:28:34.045604+0000","last_unstale":"2024-03-06T07:28:34.046728+0000","last_undegraded":"2024-03-06T07:28:34.046728+0000","last_fullsized":"2024-03-06T07:28:34.046728+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:28.693278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037833+0000","last_change":"2024-03-06T07:28:34.037833+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037833+0000","last_undegraded":"2024-03-06T07:28:34.037833+0000","last_fullsized":"2024-03-06T07:28:34.037833+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.048202+0000","last_change":"2024-03-06T07:28:34.047667+0000","last_active":"2024-03-06T07:28:34.048202+0000","last_peered":"2024-03-06T07:28:34.048202+0000","last_clean":"2024-03-06T07:28:34.048202+0000","last_became_active":"2024-03-06T07:28:34.047280+0000","last_became_peered":"2024-03-06T07:28:34.047280+0000","last_unstale":"2024-03-06T07:28:34.048202+0000","last_undegraded":"2024-03-06T07:28:34.048202+0000","last_fullsized":"2024-03-06T07:28:34.048202+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:21.608844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034910+0000","last_change":"2024-03-06T07:28:34.034910+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034910+0000","last_undegraded":"2024-03-06T07:28:34.034910+0000","last_fullsized":"2024-03-06T07:28:34.034910+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053702+0000","last_change":"2024-03-06T07:28:34.049708+0000","last_active":"2024-03-06T07:28:34.053702+0000","last_peered":"2024-03-06T07:28:34.053702+0000","last_clean":"2024-03-06T07:28:34.053702+0000","last_became_active":"2024-03-06T07:28:34.049445+0000","last_became_peered":"2024-03-06T07:28:34.049445+0000","last_unstale":"2024-03-06T07:28:34.053702+0000","last_undegraded":"2024-03-06T07:28:34.053702+0000","last_fullsized":"2024-03-06T07:28:34.053702+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:53.020063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039713+0000","last_change":"2024-03-06T07:28:34.039713+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039713+0000","last_undegraded":"2024-03-06T07:28:34.039713+0000","last_fullsized":"2024-03-06T07:28:34.039713+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.040318+0000","last_change":"2024-03-06T07:28:34.038476+0000","last_active":"2024-03-06T07:28:34.040318+0000","last_peered":"2024-03-06T07:28:34.040318+0000","last_clean":"2024-03-06T07:28:34.040318+0000","last_became_active":"2024-03-06T07:28:34.038197+0000","last_became_peered":"2024-03-06T07:28:34.038197+0000","last_unstale":"2024-03-06T07:28:34.040318+0000","last_undegraded":"2024-03-06T07:28:34.040318+0000","last_fullsized":"2024-03-06T07:28:34.040318+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:49.999763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038359+0000","last_change":"2024-03-06T07:28:34.038359+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038359+0000","last_undegraded":"2024-03-06T07:28:34.038359+0000","last_fullsized":"2024-03-06T07:28:34.038359+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.047046+0000","last_change":"2024-03-06T07:28:34.046180+0000","last_active":"2024-03-06T07:28:34.047046+0000","last_peered":"2024-03-06T07:28:34.047046+0000","last_clean":"2024-03-06T07:28:34.047046+0000","last_became_active":"2024-03-06T07:28:34.045909+0000","last_became_peered":"2024-03-06T07:28:34.045909+0000","last_unstale":"2024-03-06T07:28:34.047046+0000","last_undegraded":"2024-03-06T07:28:34.047046+0000","last_fullsized":"2024-03-06T07:28:34.047046+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:29.293270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032940+0000","last_change":"2024-03-06T07:28:34.032940+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032940+0000","last_undegraded":"2024-03-06T07:28:34.032940+0000","last_fullsized":"2024-03-06T07:28:34.032940+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.048682+0000","last_change":"2024-03-06T07:28:34.043986+0000","last_active":"2024-03-06T07:28:34.048682+0000","last_peered":"2024-03-06T07:28:34.048682+0000","last_clean":"2024-03-06T07:28:34.048682+0000","last_became_active":"2024-03-06T07:28:34.043693+0000","last_became_peered":"2024-03-06T07:28:34.043693+0000","last_unstale":"2024-03-06T07:28:34.048682+0000","last_undegraded":"2024-03-06T07:28:34.048682+0000","last_fullsized":"2024-03-06T07:28:34.048682+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:12.297830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036386+0000","last_change":"2024-03-06T07:28:34.036386+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036386+0000","last_undegraded":"2024-03-06T07:28:34.036386+0000","last_fullsized":"2024-03-06T07:28:34.036386+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053605+0000","last_change":"2024-03-06T07:28:34.047903+0000","last_active":"2024-03-06T07:28:34.053605+0000","last_peered":"2024-03-06T07:28:34.053605+0000","last_clean":"2024-03-06T07:28:34.053605+0000","last_became_active":"2024-03-06T07:28:34.047612+0000","last_became_peered":"2024-03-06T07:28:34.047612+0000","last_unstale":"2024-03-06T07:28:34.053605+0000","last_undegraded":"2024-03-06T07:28:34.053605+0000","last_fullsized":"2024-03-06T07:28:34.053605+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:08.365596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034459+0000","last_change":"2024-03-06T07:28:34.034459+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034459+0000","last_undegraded":"2024-03-06T07:28:34.034459+0000","last_fullsized":"2024-03-06T07:28:34.034459+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.052224+0000","last_change":"2024-03-06T07:28:34.051791+0000","last_active":"2024-03-06T07:28:34.052224+0000","last_peered":"2024-03-06T07:28:34.052224+0000","last_clean":"2024-03-06T07:28:34.052224+0000","last_became_active":"2024-03-06T07:28:34.051472+0000","last_became_peered":"2024-03-06T07:28:34.051472+0000","last_unstale":"2024-03-06T07:28:34.052224+0000","last_undegraded":"2024-03-06T07:28:34.052224+0000","last_fullsized":"2024-03-06T07:28:34.052224+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:50.885489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037431+0000","last_change":"2024-03-06T07:28:34.037431+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037431+0000","last_undegraded":"2024-03-06T07:28:34.037431+0000","last_fullsized":"2024-03-06T07:28:34.037431+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053725+0000","last_change":"2024-03-06T07:28:34.048933+0000","last_active":"2024-03-06T07:28:34.053725+0000","last_peered":"2024-03-06T07:28:34.053725+0000","last_clean":"2024-03-06T07:28:34.053725+0000","last_became_active":"2024-03-06T07:28:34.048670+0000","last_became_peered":"2024-03-06T07:28:34.048670+0000","last_unstale":"2024-03-06T07:28:34.053725+0000","last_undegraded":"2024-03-06T07:28:34.053725+0000","last_fullsized":"2024-03-06T07:28:34.053725+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:08.365596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037745+0000","last_change":"2024-03-06T07:28:34.037745+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037745+0000","last_undegraded":"2024-03-06T07:28:34.037745+0000","last_fullsized":"2024-03-06T07:28:34.037745+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053497+0000","last_change":"2024-03-06T07:28:34.049313+0000","last_active":"2024-03-06T07:28:34.053497+0000","last_peered":"2024-03-06T07:28:34.053497+0000","last_clean":"2024-03-06T07:28:34.053497+0000","last_became_active":"2024-03-06T07:28:34.048967+0000","last_became_peered":"2024-03-06T07:28:34.048967+0000","last_unstale":"2024-03-06T07:28:34.053497+0000","last_undegraded":"2024-03-06T07:28:34.053497+0000","last_fullsized":"2024-03-06T07:28:34.053497+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:51.180609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":506,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.032085+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:28:34.032085+0000","last_peered":"2024-03-06T07:28:34.032085+0000","last_clean":"2024-03-06T07:28:34.032085+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:28:34.032085+0000","last_undegraded":"2024-03-06T07:28:34.032085+0000","last_fullsized":"2024-03-06T07:28:34.032085+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574951,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33664,"kb_used_data":2480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338176,"statfs":{"total":96636764160,"available":96602292224,"internally_reserved":0,"allocated":2539520,"data_stored":1793319,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574950,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33584,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338256,"statfs":{"total":96636764160,"available":96602374144,"internally_reserved":0,"allocated":2461696,"data_stored":1196284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574950,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33604,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338236,"statfs":{"total":96636764160,"available":96602353664,"internally_reserved":0,"allocated":2473984,"data_stored":1202951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574951,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33600,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338240,"statfs":{"total":96636764160,"available":96602357760,"internally_reserved":0,"allocated":2473984,"data_stored":1202951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574950,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33600,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338240,"statfs":{"total":96636764160,"available":96602357760,"internally_reserved":0,"allocated":2473984,"data_stored":1202951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574950,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37744,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334096,"statfs":{"total":96636764160,"available":96598114304,"internally_reserved":0,"allocated":2461696,"data_stored":1196284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574951,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33600,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338240,"statfs":{"total":96636764160,"available":96602357760,"internally_reserved":0,"allocated":2473984,"data_stored":1202951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574951,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37812,"kb_used_data":2468,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334028,"statfs":{"total":96636764160,"available":96598044672,"internally_reserved":0,"allocated":2527232,"data_stored":1786652,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:38.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.1f is not active+clean 2024-03-06T07:28:38.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.039773+0000', 'last_change': '2024-03-06T07:28:34.039773+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.039773+0000', 'last_undegraded': '2024-03-06T07:28:34.039773+0000', 'last_fullsized': '2024-03-06T07:28:34.039773+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:38.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.1e is not active+clean 2024-03-06T07:28:38.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.032560+0000', 'last_change': '2024-03-06T07:28:34.032560+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.032560+0000', 'last_undegraded': '2024-03-06T07:28:34.032560+0000', 'last_fullsized': '2024-03-06T07:28:34.032560+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:38.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.1d is not active+clean 2024-03-06T07:28:38.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.043939+0000', 'last_change': '2024-03-06T07:28:34.043939+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.043939+0000', 'last_undegraded': '2024-03-06T07:28:34.043939+0000', 'last_fullsized': '2024-03-06T07:28:34.043939+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:38.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.1c is not active+clean 2024-03-06T07:28:38.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.040918+0000', 'last_change': '2024-03-06T07:28:34.040918+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.040918+0000', 'last_undegraded': '2024-03-06T07:28:34.040918+0000', 'last_fullsized': '2024-03-06T07:28:34.040918+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:38.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.1b is not active+clean 2024-03-06T07:28:38.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.034672+0000', 'last_change': '2024-03-06T07:28:34.034672+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.034672+0000', 'last_undegraded': '2024-03-06T07:28:34.034672+0000', 'last_fullsized': '2024-03-06T07:28:34.034672+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:38.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.1a is not active+clean 2024-03-06T07:28:38.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.036995+0000', 'last_change': '2024-03-06T07:28:34.036995+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.036995+0000', 'last_undegraded': '2024-03-06T07:28:34.036995+0000', 'last_fullsized': '2024-03-06T07:28:34.036995+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:38.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.19 is not active+clean 2024-03-06T07:28:38.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.037384+0000', 'last_change': '2024-03-06T07:28:34.037384+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.037384+0000', 'last_undegraded': '2024-03-06T07:28:34.037384+0000', 'last_fullsized': '2024-03-06T07:28:34.037384+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:38.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.18 is not active+clean 2024-03-06T07:28:38.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.036819+0000', 'last_change': '2024-03-06T07:28:34.036819+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.036819+0000', 'last_undegraded': '2024-03-06T07:28:34.036819+0000', 'last_fullsized': '2024-03-06T07:28:34.036819+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:38.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.17 is not active+clean 2024-03-06T07:28:38.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.038840+0000', 'last_change': '2024-03-06T07:28:34.038840+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.038840+0000', 'last_undegraded': '2024-03-06T07:28:34.038840+0000', 'last_fullsized': '2024-03-06T07:28:34.038840+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:38.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.16 is not active+clean 2024-03-06T07:28:38.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.043705+0000', 'last_change': '2024-03-06T07:28:34.043705+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.043705+0000', 'last_undegraded': '2024-03-06T07:28:34.043705+0000', 'last_fullsized': '2024-03-06T07:28:34.043705+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:38.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.15 is not active+clean 2024-03-06T07:28:38.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.041796+0000', 'last_change': '2024-03-06T07:28:34.041796+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.041796+0000', 'last_undegraded': '2024-03-06T07:28:34.041796+0000', 'last_fullsized': '2024-03-06T07:28:34.041796+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:38.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.14 is not active+clean 2024-03-06T07:28:38.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.038852+0000', 'last_change': '2024-03-06T07:28:34.038852+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.038852+0000', 'last_undegraded': '2024-03-06T07:28:34.038852+0000', 'last_fullsized': '2024-03-06T07:28:34.038852+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:38.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.13 is not active+clean 2024-03-06T07:28:38.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.035743+0000', 'last_change': '2024-03-06T07:28:34.035743+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.035743+0000', 'last_undegraded': '2024-03-06T07:28:34.035743+0000', 'last_fullsized': '2024-03-06T07:28:34.035743+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:38.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.12 is not active+clean 2024-03-06T07:28:38.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.033521+0000', 'last_change': '2024-03-06T07:28:34.033521+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.033521+0000', 'last_undegraded': '2024-03-06T07:28:34.033521+0000', 'last_fullsized': '2024-03-06T07:28:34.033521+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:38.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.11 is not active+clean 2024-03-06T07:28:38.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.033924+0000', 'last_change': '2024-03-06T07:28:34.033924+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.033924+0000', 'last_undegraded': '2024-03-06T07:28:34.033924+0000', 'last_fullsized': '2024-03-06T07:28:34.033924+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:38.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.10 is not active+clean 2024-03-06T07:28:38.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.038810+0000', 'last_change': '2024-03-06T07:28:34.038810+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.038810+0000', 'last_undegraded': '2024-03-06T07:28:34.038810+0000', 'last_fullsized': '2024-03-06T07:28:34.038810+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:38.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.f is not active+clean 2024-03-06T07:28:38.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.036613+0000', 'last_change': '2024-03-06T07:28:34.036613+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.036613+0000', 'last_undegraded': '2024-03-06T07:28:34.036613+0000', 'last_fullsized': '2024-03-06T07:28:34.036613+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:38.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.e is not active+clean 2024-03-06T07:28:38.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.043282+0000', 'last_change': '2024-03-06T07:28:34.043282+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.043282+0000', 'last_undegraded': '2024-03-06T07:28:34.043282+0000', 'last_fullsized': '2024-03-06T07:28:34.043282+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:38.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.d is not active+clean 2024-03-06T07:28:38.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.033099+0000', 'last_change': '2024-03-06T07:28:34.033099+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.033099+0000', 'last_undegraded': '2024-03-06T07:28:34.033099+0000', 'last_fullsized': '2024-03-06T07:28:34.033099+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:38.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.c is not active+clean 2024-03-06T07:28:38.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.031957+0000', 'last_change': '2024-03-06T07:28:34.031957+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.031957+0000', 'last_undegraded': '2024-03-06T07:28:34.031957+0000', 'last_fullsized': '2024-03-06T07:28:34.031957+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:38.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.b is not active+clean 2024-03-06T07:28:38.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.037603+0000', 'last_change': '2024-03-06T07:28:34.037603+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.037603+0000', 'last_undegraded': '2024-03-06T07:28:34.037603+0000', 'last_fullsized': '2024-03-06T07:28:34.037603+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:38.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.a is not active+clean 2024-03-06T07:28:38.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.036930+0000', 'last_change': '2024-03-06T07:28:34.036930+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.036930+0000', 'last_undegraded': '2024-03-06T07:28:34.036930+0000', 'last_fullsized': '2024-03-06T07:28:34.036930+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:38.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.9 is not active+clean 2024-03-06T07:28:38.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.033464+0000', 'last_change': '2024-03-06T07:28:34.033464+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.033464+0000', 'last_undegraded': '2024-03-06T07:28:34.033464+0000', 'last_fullsized': '2024-03-06T07:28:34.033464+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:38.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.8 is not active+clean 2024-03-06T07:28:38.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.037784+0000', 'last_change': '2024-03-06T07:28:34.037784+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.037784+0000', 'last_undegraded': '2024-03-06T07:28:34.037784+0000', 'last_fullsized': '2024-03-06T07:28:34.037784+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:38.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.7 is not active+clean 2024-03-06T07:28:38.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.031482+0000', 'last_change': '2024-03-06T07:28:34.031482+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.031482+0000', 'last_undegraded': '2024-03-06T07:28:34.031482+0000', 'last_fullsized': '2024-03-06T07:28:34.031482+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:38.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.6 is not active+clean 2024-03-06T07:28:38.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.040505+0000', 'last_change': '2024-03-06T07:28:34.040505+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.040505+0000', 'last_undegraded': '2024-03-06T07:28:34.040505+0000', 'last_fullsized': '2024-03-06T07:28:34.040505+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:38.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.5 is not active+clean 2024-03-06T07:28:38.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.034359+0000', 'last_change': '2024-03-06T07:28:34.034359+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.034359+0000', 'last_undegraded': '2024-03-06T07:28:34.034359+0000', 'last_fullsized': '2024-03-06T07:28:34.034359+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:38.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.4 is not active+clean 2024-03-06T07:28:38.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.038455+0000', 'last_change': '2024-03-06T07:28:34.038455+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.038455+0000', 'last_undegraded': '2024-03-06T07:28:34.038455+0000', 'last_fullsized': '2024-03-06T07:28:34.038455+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:38.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.3 is not active+clean 2024-03-06T07:28:38.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.032251+0000', 'last_change': '2024-03-06T07:28:34.032251+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.032251+0000', 'last_undegraded': '2024-03-06T07:28:34.032251+0000', 'last_fullsized': '2024-03-06T07:28:34.032251+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:38.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2 is not active+clean 2024-03-06T07:28:38.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.032247+0000', 'last_change': '2024-03-06T07:28:34.032247+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.032247+0000', 'last_undegraded': '2024-03-06T07:28:34.032247+0000', 'last_fullsized': '2024-03-06T07:28:34.032247+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:38.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.1 is not active+clean 2024-03-06T07:28:38.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.031492+0000', 'last_change': '2024-03-06T07:28:34.031492+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.031492+0000', 'last_undegraded': '2024-03-06T07:28:34.031492+0000', 'last_fullsized': '2024-03-06T07:28:34.031492+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:38.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.0 is not active+clean 2024-03-06T07:28:38.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.036448+0000', 'last_change': '2024-03-06T07:28:34.036448+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.036448+0000', 'last_undegraded': '2024-03-06T07:28:34.036448+0000', 'last_fullsized': '2024-03-06T07:28:34.036448+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:38.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.20 is not active+clean 2024-03-06T07:28:38.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.032461+0000', 'last_change': '2024-03-06T07:28:34.032461+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.032461+0000', 'last_undegraded': '2024-03-06T07:28:34.032461+0000', 'last_fullsized': '2024-03-06T07:28:34.032461+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:38.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.22 is not active+clean 2024-03-06T07:28:38.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.032614+0000', 'last_change': '2024-03-06T07:28:34.032614+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.032614+0000', 'last_undegraded': '2024-03-06T07:28:34.032614+0000', 'last_fullsized': '2024-03-06T07:28:34.032614+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:38.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.23 is not active+clean 2024-03-06T07:28:38.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.038496+0000', 'last_change': '2024-03-06T07:28:34.038496+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.038496+0000', 'last_undegraded': '2024-03-06T07:28:34.038496+0000', 'last_fullsized': '2024-03-06T07:28:34.038496+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:38.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.24 is not active+clean 2024-03-06T07:28:38.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.032526+0000', 'last_change': '2024-03-06T07:28:34.032526+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.032526+0000', 'last_undegraded': '2024-03-06T07:28:34.032526+0000', 'last_fullsized': '2024-03-06T07:28:34.032526+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:38.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.25 is not active+clean 2024-03-06T07:28:38.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.039197+0000', 'last_change': '2024-03-06T07:28:34.039197+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.039197+0000', 'last_undegraded': '2024-03-06T07:28:34.039197+0000', 'last_fullsized': '2024-03-06T07:28:34.039197+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:38.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.26 is not active+clean 2024-03-06T07:28:38.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.044109+0000', 'last_change': '2024-03-06T07:28:34.044109+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.044109+0000', 'last_undegraded': '2024-03-06T07:28:34.044109+0000', 'last_fullsized': '2024-03-06T07:28:34.044109+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:38.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.27 is not active+clean 2024-03-06T07:28:38.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.035597+0000', 'last_change': '2024-03-06T07:28:34.035597+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.035597+0000', 'last_undegraded': '2024-03-06T07:28:34.035597+0000', 'last_fullsized': '2024-03-06T07:28:34.035597+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:38.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.28 is not active+clean 2024-03-06T07:28:38.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.037329+0000', 'last_change': '2024-03-06T07:28:34.037329+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.037329+0000', 'last_undegraded': '2024-03-06T07:28:34.037329+0000', 'last_fullsized': '2024-03-06T07:28:34.037329+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:38.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.29 is not active+clean 2024-03-06T07:28:38.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.035539+0000', 'last_change': '2024-03-06T07:28:34.035539+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.035539+0000', 'last_undegraded': '2024-03-06T07:28:34.035539+0000', 'last_fullsized': '2024-03-06T07:28:34.035539+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:38.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2a is not active+clean 2024-03-06T07:28:38.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.039302+0000', 'last_change': '2024-03-06T07:28:34.039302+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.039302+0000', 'last_undegraded': '2024-03-06T07:28:34.039302+0000', 'last_fullsized': '2024-03-06T07:28:34.039302+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:38.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2b is not active+clean 2024-03-06T07:28:38.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.037969+0000', 'last_change': '2024-03-06T07:28:34.037969+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.037969+0000', 'last_undegraded': '2024-03-06T07:28:34.037969+0000', 'last_fullsized': '2024-03-06T07:28:34.037969+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:38.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2c is not active+clean 2024-03-06T07:28:38.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.032982+0000', 'last_change': '2024-03-06T07:28:34.032982+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.032982+0000', 'last_undegraded': '2024-03-06T07:28:34.032982+0000', 'last_fullsized': '2024-03-06T07:28:34.032982+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:38.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2d is not active+clean 2024-03-06T07:28:38.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.039603+0000', 'last_change': '2024-03-06T07:28:34.039603+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.039603+0000', 'last_undegraded': '2024-03-06T07:28:34.039603+0000', 'last_fullsized': '2024-03-06T07:28:34.039603+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:38.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2e is not active+clean 2024-03-06T07:28:38.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.032242+0000', 'last_change': '2024-03-06T07:28:34.032242+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.032242+0000', 'last_undegraded': '2024-03-06T07:28:34.032242+0000', 'last_fullsized': '2024-03-06T07:28:34.032242+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:38.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2f is not active+clean 2024-03-06T07:28:38.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.036198+0000', 'last_change': '2024-03-06T07:28:34.036198+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.036198+0000', 'last_undegraded': '2024-03-06T07:28:34.036198+0000', 'last_fullsized': '2024-03-06T07:28:34.036198+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:38.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.30 is not active+clean 2024-03-06T07:28:38.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.037352+0000', 'last_change': '2024-03-06T07:28:34.037352+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.037352+0000', 'last_undegraded': '2024-03-06T07:28:34.037352+0000', 'last_fullsized': '2024-03-06T07:28:34.037352+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:38.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.31 is not active+clean 2024-03-06T07:28:38.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.034884+0000', 'last_change': '2024-03-06T07:28:34.034884+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.034884+0000', 'last_undegraded': '2024-03-06T07:28:34.034884+0000', 'last_fullsized': '2024-03-06T07:28:34.034884+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:38.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.32 is not active+clean 2024-03-06T07:28:38.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.033456+0000', 'last_change': '2024-03-06T07:28:34.033456+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.033456+0000', 'last_undegraded': '2024-03-06T07:28:34.033456+0000', 'last_fullsized': '2024-03-06T07:28:34.033456+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:38.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.33 is not active+clean 2024-03-06T07:28:38.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.040463+0000', 'last_change': '2024-03-06T07:28:34.040463+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.040463+0000', 'last_undegraded': '2024-03-06T07:28:34.040463+0000', 'last_fullsized': '2024-03-06T07:28:34.040463+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:38.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.34 is not active+clean 2024-03-06T07:28:38.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.041301+0000', 'last_change': '2024-03-06T07:28:34.041301+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.041301+0000', 'last_undegraded': '2024-03-06T07:28:34.041301+0000', 'last_fullsized': '2024-03-06T07:28:34.041301+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:38.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.35 is not active+clean 2024-03-06T07:28:38.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.033981+0000', 'last_change': '2024-03-06T07:28:34.033981+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.033981+0000', 'last_undegraded': '2024-03-06T07:28:34.033981+0000', 'last_fullsized': '2024-03-06T07:28:34.033981+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:38.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.36 is not active+clean 2024-03-06T07:28:38.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.036066+0000', 'last_change': '2024-03-06T07:28:34.036066+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.036066+0000', 'last_undegraded': '2024-03-06T07:28:34.036066+0000', 'last_fullsized': '2024-03-06T07:28:34.036066+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:38.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.37 is not active+clean 2024-03-06T07:28:38.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.037043+0000', 'last_change': '2024-03-06T07:28:34.037043+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.037043+0000', 'last_undegraded': '2024-03-06T07:28:34.037043+0000', 'last_fullsized': '2024-03-06T07:28:34.037043+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:38.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.38 is not active+clean 2024-03-06T07:28:38.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.037833+0000', 'last_change': '2024-03-06T07:28:34.037833+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.037833+0000', 'last_undegraded': '2024-03-06T07:28:34.037833+0000', 'last_fullsized': '2024-03-06T07:28:34.037833+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:38.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.39 is not active+clean 2024-03-06T07:28:38.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.034910+0000', 'last_change': '2024-03-06T07:28:34.034910+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.034910+0000', 'last_undegraded': '2024-03-06T07:28:34.034910+0000', 'last_fullsized': '2024-03-06T07:28:34.034910+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:38.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.3a is not active+clean 2024-03-06T07:28:38.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.039713+0000', 'last_change': '2024-03-06T07:28:34.039713+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.039713+0000', 'last_undegraded': '2024-03-06T07:28:34.039713+0000', 'last_fullsized': '2024-03-06T07:28:34.039713+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:38.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.3b is not active+clean 2024-03-06T07:28:38.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.038359+0000', 'last_change': '2024-03-06T07:28:34.038359+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.038359+0000', 'last_undegraded': '2024-03-06T07:28:34.038359+0000', 'last_fullsized': '2024-03-06T07:28:34.038359+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:38.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.3c is not active+clean 2024-03-06T07:28:38.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.032940+0000', 'last_change': '2024-03-06T07:28:34.032940+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.032940+0000', 'last_undegraded': '2024-03-06T07:28:34.032940+0000', 'last_fullsized': '2024-03-06T07:28:34.032940+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:38.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.3d is not active+clean 2024-03-06T07:28:38.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.036386+0000', 'last_change': '2024-03-06T07:28:34.036386+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.036386+0000', 'last_undegraded': '2024-03-06T07:28:34.036386+0000', 'last_fullsized': '2024-03-06T07:28:34.036386+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:38.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.3e is not active+clean 2024-03-06T07:28:38.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.034459+0000', 'last_change': '2024-03-06T07:28:34.034459+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.034459+0000', 'last_undegraded': '2024-03-06T07:28:34.034459+0000', 'last_fullsized': '2024-03-06T07:28:34.034459+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:38.884 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.3f is not active+clean 2024-03-06T07:28:38.884 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.037431+0000', 'last_change': '2024-03-06T07:28:34.037431+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.037431+0000', 'last_undegraded': '2024-03-06T07:28:34.037431+0000', 'last_fullsized': '2024-03-06T07:28:34.037431+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:38.884 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.21 is not active+clean 2024-03-06T07:28:38.884 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 195, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:34.037745+0000', 'last_change': '2024-03-06T07:28:34.037745+0000', 'last_active': '2024-03-06T07:28:34.021086+0000', 'last_peered': '2024-03-06T07:28:34.021086+0000', 'last_clean': '2024-03-06T07:28:34.021086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:34.037745+0000', 'last_undegraded': '2024-03-06T07:28:34.037745+0000', 'last_fullsized': '2024-03-06T07:28:34.037745+0000', 'mapping_epoch': 195, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 195, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:34.021086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:38.884 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:28:39.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.007+0000 7fbf93684640 1 Processor -- start 2024-03-06T07:28:39.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.007+0000 7fbf93684640 1 -- start start 2024-03-06T07:28:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.009+0000 7fbf93684640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbf8c0c6b10 0x7fbf8c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.009+0000 7fbf93684640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf8c0bf9d0 0x7fbf8c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.009+0000 7fbf93684640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf8c0c2fc0 con 0x7fbf8c152650 2024-03-06T07:28:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.009+0000 7fbf93684640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf8c155800 con 0x7fbf8c0bf9d0 2024-03-06T07:28:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.009+0000 7fbf93684640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf8c158da0 con 0x7fbf8c0c6b10 2024-03-06T07:28:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.009+0000 7fbf91c21640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf8c0bf9d0 0x7fbf8c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.009+0000 7fbf91c21640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf8c0bf9d0 0x7fbf8c0c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54576/0 (socket says 172.21.15.1:54576) 2024-03-06T07:28:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.009+0000 7fbf91c21640 1 -- 172.21.15.1:0/349463129 learned_addr learned my addr 172.21.15.1:0/349463129 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:39.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.009+0000 7fbf90c1f640 1 --2- 172.21.15.1:0/349463129 >> v2:172.21.15.98:3300/0 conn(0x7fbf8c0c6b10 0x7fbf8c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.009+0000 7fbf91c21640 1 -- 172.21.15.1:0/349463129 >> v2:172.21.15.98:3300/0 conn(0x7fbf8c0c6b10 msgr2=0x7fbf8c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.009+0000 7fbf91c21640 1 --2- 172.21.15.1:0/349463129 >> v2:172.21.15.98:3300/0 conn(0x7fbf8c0c6b10 0x7fbf8c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.009+0000 7fbf91c21640 1 -- 172.21.15.1:0/349463129 >> v1:172.21.15.98:6789/0 conn(0x7fbf8c152650 legacy=0x7fbf8c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.009+0000 7fbf91c21640 1 -- 172.21.15.1:0/349463129 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c0c6690 con 0x7fbf8c0bf9d0 2024-03-06T07:28:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.010+0000 7fbf91c21640 1 --2- 172.21.15.1:0/349463129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf8c0bf9d0 0x7fbf8c0c1460 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0cb8e0 tx=0x7fbf8402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f871d962134ae20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.010+0000 7fbf83fff640 1 -- 172.21.15.1:0/349463129 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8402ca80 con 0x7fbf8c0bf9d0 2024-03-06T07:28:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.010+0000 7fbf83fff640 1 -- 172.21.15.1:0/349463129 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8402cc20 con 0x7fbf8c0bf9d0 2024-03-06T07:28:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.010+0000 7fbf83fff640 1 -- 172.21.15.1:0/349463129 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf840343f0 con 0x7fbf8c0bf9d0 2024-03-06T07:28:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.010+0000 7fbf93684640 1 -- 172.21.15.1:0/349463129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf8c0bf9d0 msgr2=0x7fbf8c0c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.010+0000 7fbf93684640 1 --2- 172.21.15.1:0/349463129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf8c0bf9d0 0x7fbf8c0c1460 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0cb8e0 tx=0x7fbf8402bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.011+0000 7fbf93684640 1 -- 172.21.15.1:0/349463129 shutdown_connections 2024-03-06T07:28:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.011+0000 7fbf93684640 1 --2- 172.21.15.1:0/349463129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf8c0bf9d0 0x7fbf8c0c1460 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.011+0000 7fbf93684640 1 --2- 172.21.15.1:0/349463129 >> v2:172.21.15.98:3300/0 conn(0x7fbf8c0c6b10 0x7fbf8c155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.011+0000 7fbf93684640 1 -- 172.21.15.1:0/349463129 >> 172.21.15.1:0/349463129 conn(0x7fbf8c066ee0 msgr2=0x7fbf8c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.011+0000 7fbf93684640 1 -- 172.21.15.1:0/349463129 shutdown_connections 2024-03-06T07:28:39.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.011+0000 7fbf93684640 1 -- 172.21.15.1:0/349463129 wait complete. 2024-03-06T07:28:39.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.012+0000 7fbf93684640 1 Processor -- start 2024-03-06T07:28:39.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.012+0000 7fbf93684640 1 -- start start 2024-03-06T07:28:39.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.012+0000 7fbf93684640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbf8c0bf9d0 0x7fbf8c1eedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.012+0000 7fbf93684640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf8c152650 0x7fbf8c152d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.012+0000 7fbf93684640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf8c157750 0x7fbf8c1de790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.012+0000 7fbf93684640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c1551b0 con 0x7fbf8c157750 2024-03-06T07:28:39.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.012+0000 7fbf93684640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c0c6fa0 con 0x7fbf8c152650 2024-03-06T07:28:39.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.012+0000 7fbf93684640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c155490 con 0x7fbf8c0bf9d0 2024-03-06T07:28:39.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.012+0000 7fbf91c21640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf8c157750 0x7fbf8c1de790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.012+0000 7fbf91c21640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf8c157750 0x7fbf8c1de790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54592/0 (socket says 172.21.15.1:54592) 2024-03-06T07:28:39.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.012+0000 7fbf91c21640 1 -- 172.21.15.1:0/1298916114 learned_addr learned my addr 172.21.15.1:0/1298916114 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:39.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.013+0000 7fbf90c1f640 1 --2- 172.21.15.1:0/1298916114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf8c152650 0x7fbf8c152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.013+0000 7fbf91420640 1 --2- 172.21.15.1:0/1298916114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbf8c0bf9d0 0x7fbf8c1eedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.013+0000 7fbf91c21640 1 -- 172.21.15.1:0/1298916114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbf8c0bf9d0 msgr2=0x7fbf8c1eedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.013+0000 7fbf91c21640 1 --2- 172.21.15.1:0/1298916114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbf8c0bf9d0 0x7fbf8c1eedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.013+0000 7fbf91c21640 1 -- 172.21.15.1:0/1298916114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf8c152650 msgr2=0x7fbf8c152d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.013+0000 7fbf91c21640 1 --2- 172.21.15.1:0/1298916114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf8c152650 0x7fbf8c152d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.013+0000 7fbf91c21640 1 -- 172.21.15.1:0/1298916114 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c0c9c40 con 0x7fbf8c157750 2024-03-06T07:28:39.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.013+0000 7fbf91420640 1 --2- 172.21.15.1:0/1298916114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbf8c0bf9d0 0x7fbf8c1eedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:39.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.013+0000 7fbf91c21640 1 --2- 172.21.15.1:0/1298916114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf8c157750 0x7fbf8c1de790 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009f40 tx=0x7fbf84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.013+0000 7fbf81ffb640 1 -- 172.21.15.1:0/1298916114 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84043020 con 0x7fbf8c157750 2024-03-06T07:28:39.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.013+0000 7fbf81ffb640 1 -- 172.21.15.1:0/1298916114 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8403e450 con 0x7fbf8c157750 2024-03-06T07:28:39.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.013+0000 7fbf81ffb640 1 -- 172.21.15.1:0/1298916114 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84009510 con 0x7fbf8c157750 2024-03-06T07:28:39.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.013+0000 7fbf93684640 1 -- 172.21.15.1:0/1298916114 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf8c0c9f20 con 0x7fbf8c157750 2024-03-06T07:28:39.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.013+0000 7fbf93684640 1 -- 172.21.15.1:0/1298916114 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf8c1eae30 con 0x7fbf8c157750 2024-03-06T07:28:39.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.014+0000 7fbf93684640 1 -- 172.21.15.1:0/1298916114 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf8c06b540 con 0x7fbf8c157750 2024-03-06T07:28:39.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.015+0000 7fbf81ffb640 1 -- 172.21.15.1:0/1298916114 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf84046020 con 0x7fbf8c157750 2024-03-06T07:28:39.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.016+0000 7fbf81ffb640 1 --2- 172.21.15.1:0/1298916114 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbf7006b4d0 0x7fbf7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.016+0000 7fbf81ffb640 1 -- 172.21.15.1:0/1298916114 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbf840acab0 con 0x7fbf8c157750 2024-03-06T07:28:39.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.016+0000 7fbf91420640 1 --2- 172.21.15.1:0/1298916114 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbf7006b4d0 0x7fbf7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.017+0000 7fbf91420640 1 --2- 172.21.15.1:0/1298916114 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbf7006b4d0 0x7fbf7006d990 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c001700 tx=0x7fbf7c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.024+0000 7fbf81ffb640 1 -- 172.21.15.1:0/1298916114 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf8c06b540 con 0x7fbf8c157750 2024-03-06T07:28:39.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.277+0000 7fbf93684640 1 -- 172.21.15.1:0/1298916114 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fbf8c067fd0 con 0x7fbf8c157750 2024-03-06T07:28:39.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.277+0000 7fbf81ffb640 1 -- 172.21.15.1:0/1298916114 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1459 (secure 0 0 0) 0x7fbf8404d030 con 0x7fbf8c157750 2024-03-06T07:28:39.279 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:39.279 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":514,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":197,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709614,"num_in_osds":8,"osd_in_since":1709709606,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":283860992,"bytes_avail":772810252288,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":185,"id":15,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":15,"rank":0,"name":"a","status":"up:active","gid":10425}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":19,"modified":"2024-03-06T07:27:40.793543+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:28:39.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.279+0000 7fbf93684640 1 -- 172.21.15.1:0/1298916114 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbf7006b4d0 msgr2=0x7fbf7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.279+0000 7fbf93684640 1 --2- 172.21.15.1:0/1298916114 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbf7006b4d0 0x7fbf7006d990 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c001700 tx=0x7fbf7c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:39.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.279+0000 7fbf93684640 1 -- 172.21.15.1:0/1298916114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf8c157750 msgr2=0x7fbf8c1de790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.279+0000 7fbf93684640 1 --2- 172.21.15.1:0/1298916114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf8c157750 0x7fbf8c1de790 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009f40 tx=0x7fbf84004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:39.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.279+0000 7fbf93684640 1 -- 172.21.15.1:0/1298916114 shutdown_connections 2024-03-06T07:28:39.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.279+0000 7fbf93684640 1 --2- 172.21.15.1:0/1298916114 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbf7006b4d0 0x7fbf7006d990 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.279+0000 7fbf93684640 1 --2- 172.21.15.1:0/1298916114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf8c157750 0x7fbf8c1de790 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.279+0000 7fbf93684640 1 --2- 172.21.15.1:0/1298916114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf8c152650 0x7fbf8c152d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.279+0000 7fbf93684640 1 --2- 172.21.15.1:0/1298916114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbf8c0bf9d0 0x7fbf8c1eedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.279+0000 7fbf93684640 1 -- 172.21.15.1:0/1298916114 >> 172.21.15.1:0/1298916114 conn(0x7fbf8c066ee0 msgr2=0x7fbf8c0c3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.279+0000 7fbf93684640 1 -- 172.21.15.1:0/1298916114 shutdown_connections 2024-03-06T07:28:39.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.279+0000 7fbf93684640 1 -- 172.21.15.1:0/1298916114 wait complete. 2024-03-06T07:28:39.292 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:28:39.292 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:39.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.391+0000 7fec0f69a640 1 Processor -- start 2024-03-06T07:28:39.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.391+0000 7fec0f69a640 1 -- start start 2024-03-06T07:28:39.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.392+0000 7fec0f69a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fec080c6b10 0x7fec08155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.392+0000 7fec0f69a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec080c2fc0 con 0x7fec08152650 2024-03-06T07:28:39.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.392+0000 7fec0f69a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec08158da0 con 0x7fec080bf9d0 2024-03-06T07:28:39.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.392+0000 7fec0f69a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec08155800 con 0x7fec080c6b10 2024-03-06T07:28:39.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.392+0000 7fec0cc35640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fec080c6b10 0x7fec08155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.392+0000 7fec0d436640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fec08152650 0x7fec08158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45240/0 (socket says 172.21.15.1:45240) 2024-03-06T07:28:39.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.392+0000 7fec0d436640 1 -- 172.21.15.1:0/2912505435 learned_addr learned my addr 172.21.15.1:0/2912505435 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:39.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.393+0000 7fec0cc35640 1 -- 172.21.15.1:0/2912505435 >> v1:172.21.15.98:6789/0 conn(0x7fec080bf9d0 legacy=0x7fec080c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.393+0000 7fec0cc35640 1 -- 172.21.15.1:0/2912505435 >> v1:172.21.15.1:6789/0 conn(0x7fec08152650 legacy=0x7fec08158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.393+0000 7fec0cc35640 1 -- 172.21.15.1:0/2912505435 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec080c6690 con 0x7fec080c6b10 2024-03-06T07:28:39.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.393+0000 7febfffff640 1 -- 172.21.15.1:0/2912505435 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268408923 0 0) 0x7fec080c2fc0 con 0x7fec08152650 2024-03-06T07:28:39.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.393+0000 7fec0cc35640 1 --2- 172.21.15.1:0/2912505435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec080c6b10 0x7fec08155c90 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7febf80097a0 tx=0x7febf802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7056772cb7fc773 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.394+0000 7febfffff640 1 -- 172.21.15.1:0/2912505435 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf80315a0 con 0x7fec080c6b10 2024-03-06T07:28:39.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.394+0000 7febfffff640 1 -- 172.21.15.1:0/2912505435 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf8037070 con 0x7fec080c6b10 2024-03-06T07:28:39.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.394+0000 7febfffff640 1 -- 172.21.15.1:0/2912505435 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8031960 con 0x7fec080c6b10 2024-03-06T07:28:39.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.394+0000 7fec0f69a640 1 -- 172.21.15.1:0/2912505435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec080c6b10 msgr2=0x7fec08155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.394+0000 7fec0f69a640 1 --2- 172.21.15.1:0/2912505435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec080c6b10 0x7fec08155c90 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7febf80097a0 tx=0x7febf802b840 comp rx=0 tx=0).stop 2024-03-06T07:28:39.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.394+0000 7fec0f69a640 1 -- 172.21.15.1:0/2912505435 shutdown_connections 2024-03-06T07:28:39.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.394+0000 7fec0f69a640 1 --2- 172.21.15.1:0/2912505435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec080c6b10 0x7fec08155c90 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.394+0000 7fec0f69a640 1 -- 172.21.15.1:0/2912505435 >> 172.21.15.1:0/2912505435 conn(0x7fec08066ee0 msgr2=0x7fec080672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.394+0000 7fec0f69a640 1 -- 172.21.15.1:0/2912505435 shutdown_connections 2024-03-06T07:28:39.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.394+0000 7fec0f69a640 1 -- 172.21.15.1:0/2912505435 wait complete. 2024-03-06T07:28:39.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.395+0000 7fec0f69a640 1 Processor -- start 2024-03-06T07:28:39.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.395+0000 7fec0f69a640 1 -- start start 2024-03-06T07:28:39.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.396+0000 7fec0f69a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec080bf9d0 0x7fec080c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.396+0000 7fec0f69a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec08152650 0x7fec081ef250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.396+0000 7fec0f69a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec0814ba40 0x7fec081eb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.396+0000 7fec0f69a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec081551b0 con 0x7fec08152650 2024-03-06T07:28:39.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.396+0000 7fec0f69a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec080c6fa0 con 0x7fec080bf9d0 2024-03-06T07:28:39.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.396+0000 7fec0f69a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec08155490 con 0x7fec0814ba40 2024-03-06T07:28:39.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.396+0000 7fec0cc35640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec08152650 0x7fec081ef250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.396+0000 7fec0cc35640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec08152650 0x7fec081ef250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54602/0 (socket says 172.21.15.1:54602) 2024-03-06T07:28:39.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.396+0000 7fec0cc35640 1 -- 172.21.15.1:0/3356769981 learned_addr learned my addr 172.21.15.1:0/3356769981 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:39.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.396+0000 7fec0d436640 1 --2- 172.21.15.1:0/3356769981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec080bf9d0 0x7fec080c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.396+0000 7fec0dc37640 1 --2- 172.21.15.1:0/3356769981 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec0814ba40 0x7fec081eb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.396+0000 7fec0cc35640 1 -- 172.21.15.1:0/3356769981 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec0814ba40 msgr2=0x7fec081eb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.396+0000 7fec0cc35640 1 --2- 172.21.15.1:0/3356769981 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec0814ba40 0x7fec081eb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.396+0000 7fec0cc35640 1 -- 172.21.15.1:0/3356769981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec080bf9d0 msgr2=0x7fec080c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.396+0000 7fec0cc35640 1 --2- 172.21.15.1:0/3356769981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec080bf9d0 0x7fec080c9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.397+0000 7fec0cc35640 1 -- 172.21.15.1:0/3356769981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec080ca3e0 con 0x7fec08152650 2024-03-06T07:28:39.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.397+0000 7fec0d436640 1 --2- 172.21.15.1:0/3356769981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec080bf9d0 0x7fec080c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:39.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.397+0000 7fec0dc37640 1 --2- 172.21.15.1:0/3356769981 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec0814ba40 0x7fec081eb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:39.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.397+0000 7fec0cc35640 1 --2- 172.21.15.1:0/3356769981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec08152650 0x7fec081ef250 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7febf80098c0 tx=0x7febf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.397+0000 7febfdffb640 1 -- 172.21.15.1:0/3356769981 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8032840 con 0x7fec08152650 2024-03-06T07:28:39.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.397+0000 7febfdffb640 1 -- 172.21.15.1:0/3356769981 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf80329e0 con 0x7fec08152650 2024-03-06T07:28:39.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.397+0000 7fec0f69a640 1 -- 172.21.15.1:0/3356769981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec080ca6c0 con 0x7fec08152650 2024-03-06T07:28:39.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.397+0000 7febfdffb640 1 -- 172.21.15.1:0/3356769981 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8032dd0 con 0x7fec08152650 2024-03-06T07:28:39.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.397+0000 7fec0f69a640 1 -- 172.21.15.1:0/3356769981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec081eea50 con 0x7fec08152650 2024-03-06T07:28:39.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.398+0000 7fec0f69a640 1 -- 172.21.15.1:0/3356769981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec080c2f50 con 0x7fec08152650 2024-03-06T07:28:39.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.399+0000 7febfdffb640 1 -- 172.21.15.1:0/3356769981 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7febf804d3f0 con 0x7fec08152650 2024-03-06T07:28:39.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.399+0000 7febfdffb640 1 --2- 172.21.15.1:0/3356769981 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febec06b4d0 0x7febec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.399+0000 7febfdffb640 1 -- 172.21.15.1:0/3356769981 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7febf802f030 con 0x7fec08152650 2024-03-06T07:28:39.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.399+0000 7fec0d436640 1 --2- 172.21.15.1:0/3356769981 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febec06b4d0 0x7febec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.400+0000 7fec0d436640 1 --2- 172.21.15.1:0/3356769981 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febec06b4d0 0x7febec06d990 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fec04000f60 tx=0x7fec04007c10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.404+0000 7febfdffb640 1 -- 172.21.15.1:0/3356769981 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec080c2f50 con 0x7fec08152650 2024-03-06T07:28:39.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.584+0000 7fec0f69a640 1 -- 172.21.15.1:0/3356769981 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fec080c8840 con 0x7febec06b4d0 2024-03-06T07:28:39.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.591+0000 7febfdffb640 1 -- 172.21.15.1:0/3356769981 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247177 (secure 0 0 0) 0x7fec04007270 con 0x7febec06b4d0 2024-03-06T07:28:39.596 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:39.597 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:28:39.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.594+0000 7fec0f69a640 1 -- 172.21.15.1:0/3356769981 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febec06b4d0 msgr2=0x7febec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.594+0000 7fec0f69a640 1 --2- 172.21.15.1:0/3356769981 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febec06b4d0 0x7febec06d990 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fec04000f60 tx=0x7fec04007c10 comp rx=0 tx=0).stop 2024-03-06T07:28:39.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.594+0000 7fec0f69a640 1 -- 172.21.15.1:0/3356769981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec08152650 msgr2=0x7fec081ef250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.594+0000 7fec0f69a640 1 --2- 172.21.15.1:0/3356769981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec08152650 0x7fec081ef250 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7febf80098c0 tx=0x7febf80046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.594+0000 7fec0f69a640 1 -- 172.21.15.1:0/3356769981 shutdown_connections 2024-03-06T07:28:39.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.594+0000 7fec0f69a640 1 --2- 172.21.15.1:0/3356769981 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7febec06b4d0 0x7febec06d990 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.594+0000 7fec0f69a640 1 --2- 172.21.15.1:0/3356769981 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec0814ba40 0x7fec081eb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.594+0000 7fec0f69a640 1 --2- 172.21.15.1:0/3356769981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec08152650 0x7fec081ef250 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.594+0000 7fec0f69a640 1 --2- 172.21.15.1:0/3356769981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec080bf9d0 0x7fec080c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.594+0000 7fec0f69a640 1 -- 172.21.15.1:0/3356769981 >> 172.21.15.1:0/3356769981 conn(0x7fec08066ee0 msgr2=0x7fec080c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.594+0000 7fec0f69a640 1 -- 172.21.15.1:0/3356769981 shutdown_connections 2024-03-06T07:28:39.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:39.594+0000 7fec0f69a640 1 -- 172.21.15.1:0/3356769981 wait complete. 2024-03-06T07:28:39.607 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-03-06T07:28:38.807920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":277208,"kb_used_data":19420,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754697512,"statfs":{"total":773094113280,"available":772810252288,"internally_reserved":0,"allocated":19886080,"data_stored":10784343,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.785348"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039773+0000","last_change":"2024-03-06T07:28:34.039773+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039773+0000","last_undegraded":"2024-03-06T07:28:34.039773+0000","last_fullsized":"2024-03-06T07:28:34.039773+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032560+0000","last_change":"2024-03-06T07:28:34.032560+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032560+0000","last_undegraded":"2024-03-06T07:28:34.032560+0000","last_fullsized":"2024-03-06T07:28:34.032560+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.043939+0000","last_change":"2024-03-06T07:28:34.043939+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.043939+0000","last_undegraded":"2024-03-06T07:28:34.043939+0000","last_fullsized":"2024-03-06T07:28:34.043939+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.040918+0000","last_change":"2024-03-06T07:28:34.040918+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.040918+0000","last_undegraded":"2024-03-06T07:28:34.040918+0000","last_fullsized":"2024-03-06T07:28:34.040918+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034672+0000","last_change":"2024-03-06T07:28:34.034672+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034672+0000","last_undegraded":"2024-03-06T07:28:34.034672+0000","last_fullsized":"2024-03-06T07:28:34.034672+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036995+0000","last_change":"2024-03-06T07:28:34.036995+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036995+0000","last_undegraded":"2024-03-06T07:28:34.036995+0000","last_fullsized":"2024-03-06T07:28:34.036995+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037384+0000","last_change":"2024-03-06T07:28:34.037384+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037384+0000","last_undegraded":"2024-03-06T07:28:34.037384+0000","last_fullsized":"2024-03-06T07:28:34.037384+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036819+0000","last_change":"2024-03-06T07:28:34.036819+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036819+0000","last_undegraded":"2024-03-06T07:28:34.036819+0000","last_fullsized":"2024-03-06T07:28:34.036819+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038840+0000","last_change":"2024-03-06T07:28:34.038840+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038840+0000","last_undegraded":"2024-03-06T07:28:34.038840+0000","last_fullsized":"2024-03-06T07:28:34.038840+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.043705+0000","last_change":"2024-03-06T07:28:34.043705+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.043705+0000","last_undegraded":"2024-03-06T07:28:34.043705+0000","last_fullsized":"2024-03-06T07:28:34.043705+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.041796+0000","last_change":"2024-03-06T07:28:34.041796+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.041796+0000","last_undegraded":"2024-03-06T07:28:34.041796+0000","last_fullsized":"2024-03-06T07:28:34.041796+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038852+0000","last_change":"2024-03-06T07:28:34.038852+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038852+0000","last_undegraded":"2024-03-06T07:28:34.038852+0000","last_fullsized":"2024-03-06T07:28:34.038852+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.035743+0000","last_change":"2024-03-06T07:28:34.035743+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.035743+0000","last_undegraded":"2024-03-06T07:28:34.035743+0000","last_fullsized":"2024-03-06T07:28:34.035743+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033521+0000","last_change":"2024-03-06T07:28:34.033521+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033521+0000","last_undegraded":"2024-03-06T07:28:34.033521+0000","last_fullsized":"2024-03-06T07:28:34.033521+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033924+0000","last_change":"2024-03-06T07:28:34.033924+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033924+0000","last_undegraded":"2024-03-06T07:28:34.033924+0000","last_fullsized":"2024-03-06T07:28:34.033924+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038810+0000","last_change":"2024-03-06T07:28:34.038810+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038810+0000","last_undegraded":"2024-03-06T07:28:34.038810+0000","last_fullsized":"2024-03-06T07:28:34.038810+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036613+0000","last_change":"2024-03-06T07:28:34.036613+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036613+0000","last_undegraded":"2024-03-06T07:28:34.036613+0000","last_fullsized":"2024-03-06T07:28:34.036613+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.043282+0000","last_change":"2024-03-06T07:28:34.043282+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.043282+0000","last_undegraded":"2024-03-06T07:28:34.043282+0000","last_fullsized":"2024-03-06T07:28:34.043282+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033099+0000","last_change":"2024-03-06T07:28:34.033099+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033099+0000","last_undegraded":"2024-03-06T07:28:34.033099+0000","last_fullsized":"2024-03-06T07:28:34.033099+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.031957+0000","last_change":"2024-03-06T07:28:34.031957+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.031957+0000","last_undegraded":"2024-03-06T07:28:34.031957+0000","last_fullsized":"2024-03-06T07:28:34.031957+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037603+0000","last_change":"2024-03-06T07:28:34.037603+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037603+0000","last_undegraded":"2024-03-06T07:28:34.037603+0000","last_fullsized":"2024-03-06T07:28:34.037603+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036930+0000","last_change":"2024-03-06T07:28:34.036930+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036930+0000","last_undegraded":"2024-03-06T07:28:34.036930+0000","last_fullsized":"2024-03-06T07:28:34.036930+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033464+0000","last_change":"2024-03-06T07:28:34.033464+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033464+0000","last_undegraded":"2024-03-06T07:28:34.033464+0000","last_fullsized":"2024-03-06T07:28:34.033464+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037784+0000","last_change":"2024-03-06T07:28:34.037784+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037784+0000","last_undegraded":"2024-03-06T07:28:34.037784+0000","last_fullsized":"2024-03-06T07:28:34.037784+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.031482+0000","last_change":"2024-03-06T07:28:34.031482+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.031482+0000","last_undegraded":"2024-03-06T07:28:34.031482+0000","last_fullsized":"2024-03-06T07:28:34.031482+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.040505+0000","last_change":"2024-03-06T07:28:34.040505+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.040505+0000","last_undegraded":"2024-03-06T07:28:34.040505+0000","last_fullsized":"2024-03-06T07:28:34.040505+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034359+0000","last_change":"2024-03-06T07:28:34.034359+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034359+0000","last_undegraded":"2024-03-06T07:28:34.034359+0000","last_fullsized":"2024-03-06T07:28:34.034359+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038455+0000","last_change":"2024-03-06T07:28:34.038455+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038455+0000","last_undegraded":"2024-03-06T07:28:34.038455+0000","last_fullsized":"2024-03-06T07:28:34.038455+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032251+0000","last_change":"2024-03-06T07:28:34.032251+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032251+0000","last_undegraded":"2024-03-06T07:28:34.032251+0000","last_fullsized":"2024-03-06T07:28:34.032251+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032247+0000","last_change":"2024-03-06T07:28:34.032247+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032247+0000","last_undegraded":"2024-03-06T07:28:34.032247+0000","last_fullsized":"2024-03-06T07:28:34.032247+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.031492+0000","last_change":"2024-03-06T07:28:34.031492+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.031492+0000","last_undegraded":"2024-03-06T07:28:34.031492+0000","last_fullsized":"2024-03-06T07:28:34.031492+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036448+0000","last_change":"2024-03-06T07:28:34.036448+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036448+0000","last_undegraded":"2024-03-06T07:28:34.036448+0000","last_fullsized":"2024-03-06T07:28:34.036448+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032461+0000","last_change":"2024-03-06T07:28:34.032461+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032461+0000","last_undegraded":"2024-03-06T07:28:34.032461+0000","last_fullsized":"2024-03-06T07:28:34.032461+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.046617+0000","last_change":"2024-03-06T07:28:34.045692+0000","last_active":"2024-03-06T07:28:34.046617+0000","last_peered":"2024-03-06T07:28:34.046617+0000","last_clean":"2024-03-06T07:28:34.046617+0000","last_became_active":"2024-03-06T07:28:34.045392+0000","last_became_peered":"2024-03-06T07:28:34.045392+0000","last_unstale":"2024-03-06T07:28:34.046617+0000","last_undegraded":"2024-03-06T07:28:34.046617+0000","last_fullsized":"2024-03-06T07:28:34.046617+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:59.770341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032614+0000","last_change":"2024-03-06T07:28:34.032614+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032614+0000","last_undegraded":"2024-03-06T07:28:34.032614+0000","last_fullsized":"2024-03-06T07:28:34.032614+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053547+0000","last_change":"2024-03-06T07:28:34.048342+0000","last_active":"2024-03-06T07:28:34.053547+0000","last_peered":"2024-03-06T07:28:34.053547+0000","last_clean":"2024-03-06T07:28:34.053547+0000","last_became_active":"2024-03-06T07:28:34.048077+0000","last_became_peered":"2024-03-06T07:28:34.048077+0000","last_unstale":"2024-03-06T07:28:34.053547+0000","last_undegraded":"2024-03-06T07:28:34.053547+0000","last_fullsized":"2024-03-06T07:28:34.053547+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:02.569692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038496+0000","last_change":"2024-03-06T07:28:34.038496+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038496+0000","last_undegraded":"2024-03-06T07:28:34.038496+0000","last_fullsized":"2024-03-06T07:28:34.038496+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.044121+0000","last_change":"2024-03-06T07:28:34.043250+0000","last_active":"2024-03-06T07:28:34.044121+0000","last_peered":"2024-03-06T07:28:34.044121+0000","last_clean":"2024-03-06T07:28:34.044121+0000","last_became_active":"2024-03-06T07:28:34.042985+0000","last_became_peered":"2024-03-06T07:28:34.042985+0000","last_unstale":"2024-03-06T07:28:34.044121+0000","last_undegraded":"2024-03-06T07:28:34.044121+0000","last_fullsized":"2024-03-06T07:28:34.044121+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:49.778788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032526+0000","last_change":"2024-03-06T07:28:34.032526+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032526+0000","last_undegraded":"2024-03-06T07:28:34.032526+0000","last_fullsized":"2024-03-06T07:28:34.032526+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.040595+0000","last_change":"2024-03-06T07:28:34.039018+0000","last_active":"2024-03-06T07:28:34.040595+0000","last_peered":"2024-03-06T07:28:34.040595+0000","last_clean":"2024-03-06T07:28:34.040595+0000","last_became_active":"2024-03-06T07:28:34.038734+0000","last_became_peered":"2024-03-06T07:28:34.038734+0000","last_unstale":"2024-03-06T07:28:34.040595+0000","last_undegraded":"2024-03-06T07:28:34.040595+0000","last_fullsized":"2024-03-06T07:28:34.040595+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:56.463129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039197+0000","last_change":"2024-03-06T07:28:34.039197+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039197+0000","last_undegraded":"2024-03-06T07:28:34.039197+0000","last_fullsized":"2024-03-06T07:28:34.039197+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.046080+0000","last_change":"2024-03-06T07:28:34.044403+0000","last_active":"2024-03-06T07:28:34.046080+0000","last_peered":"2024-03-06T07:28:34.046080+0000","last_clean":"2024-03-06T07:28:34.046080+0000","last_became_active":"2024-03-06T07:28:34.044130+0000","last_became_peered":"2024-03-06T07:28:34.044130+0000","last_unstale":"2024-03-06T07:28:34.046080+0000","last_undegraded":"2024-03-06T07:28:34.046080+0000","last_fullsized":"2024-03-06T07:28:34.046080+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:18.838141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.044109+0000","last_change":"2024-03-06T07:28:34.044109+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.044109+0000","last_undegraded":"2024-03-06T07:28:34.044109+0000","last_fullsized":"2024-03-06T07:28:34.044109+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.054615+0000","last_change":"2024-03-06T07:28:34.054170+0000","last_active":"2024-03-06T07:28:34.054615+0000","last_peered":"2024-03-06T07:28:34.054615+0000","last_clean":"2024-03-06T07:28:34.054615+0000","last_became_active":"2024-03-06T07:28:34.053871+0000","last_became_peered":"2024-03-06T07:28:34.053871+0000","last_unstale":"2024-03-06T07:28:34.054615+0000","last_undegraded":"2024-03-06T07:28:34.054615+0000","last_fullsized":"2024-03-06T07:28:34.054615+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:29.552066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.035597+0000","last_change":"2024-03-06T07:28:34.035597+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.035597+0000","last_undegraded":"2024-03-06T07:28:34.035597+0000","last_fullsized":"2024-03-06T07:28:34.035597+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.046191+0000","last_change":"2024-03-06T07:28:34.042955+0000","last_active":"2024-03-06T07:28:34.046191+0000","last_peered":"2024-03-06T07:28:34.046191+0000","last_clean":"2024-03-06T07:28:34.046191+0000","last_became_active":"2024-03-06T07:28:34.042684+0000","last_became_peered":"2024-03-06T07:28:34.042684+0000","last_unstale":"2024-03-06T07:28:34.046191+0000","last_undegraded":"2024-03-06T07:28:34.046191+0000","last_fullsized":"2024-03-06T07:28:34.046191+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:04.206156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037329+0000","last_change":"2024-03-06T07:28:34.037329+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037329+0000","last_undegraded":"2024-03-06T07:28:34.037329+0000","last_fullsized":"2024-03-06T07:28:34.037329+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.037420+0000","last_change":"2024-03-06T07:28:34.036082+0000","last_active":"2024-03-06T07:28:34.037420+0000","last_peered":"2024-03-06T07:28:34.037420+0000","last_clean":"2024-03-06T07:28:34.037420+0000","last_became_active":"2024-03-06T07:28:34.035802+0000","last_became_peered":"2024-03-06T07:28:34.035802+0000","last_unstale":"2024-03-06T07:28:34.037420+0000","last_undegraded":"2024-03-06T07:28:34.037420+0000","last_fullsized":"2024-03-06T07:28:34.037420+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:00.673585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.035539+0000","last_change":"2024-03-06T07:28:34.035539+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.035539+0000","last_undegraded":"2024-03-06T07:28:34.035539+0000","last_fullsized":"2024-03-06T07:28:34.035539+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.051779+0000","last_change":"2024-03-06T07:28:34.051147+0000","last_active":"2024-03-06T07:28:34.051779+0000","last_peered":"2024-03-06T07:28:34.051779+0000","last_clean":"2024-03-06T07:28:34.051779+0000","last_became_active":"2024-03-06T07:28:34.050807+0000","last_became_peered":"2024-03-06T07:28:34.050807+0000","last_unstale":"2024-03-06T07:28:34.051779+0000","last_undegraded":"2024-03-06T07:28:34.051779+0000","last_fullsized":"2024-03-06T07:28:34.051779+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:56.463129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039302+0000","last_change":"2024-03-06T07:28:34.039302+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039302+0000","last_undegraded":"2024-03-06T07:28:34.039302+0000","last_fullsized":"2024-03-06T07:28:34.039302+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.047597+0000","last_change":"2024-03-06T07:28:34.044501+0000","last_active":"2024-03-06T07:28:34.047597+0000","last_peered":"2024-03-06T07:28:34.047597+0000","last_clean":"2024-03-06T07:28:34.047597+0000","last_became_active":"2024-03-06T07:28:34.044057+0000","last_became_peered":"2024-03-06T07:28:34.044057+0000","last_unstale":"2024-03-06T07:28:34.047597+0000","last_undegraded":"2024-03-06T07:28:34.047597+0000","last_fullsized":"2024-03-06T07:28:34.047597+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:33.892363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037969+0000","last_change":"2024-03-06T07:28:34.037969+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037969+0000","last_undegraded":"2024-03-06T07:28:34.037969+0000","last_fullsized":"2024-03-06T07:28:34.037969+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.042936+0000","last_change":"2024-03-06T07:28:34.041503+0000","last_active":"2024-03-06T07:28:34.042936+0000","last_peered":"2024-03-06T07:28:34.042936+0000","last_clean":"2024-03-06T07:28:34.042936+0000","last_became_active":"2024-03-06T07:28:34.041217+0000","last_became_peered":"2024-03-06T07:28:34.041217+0000","last_unstale":"2024-03-06T07:28:34.042936+0000","last_undegraded":"2024-03-06T07:28:34.042936+0000","last_fullsized":"2024-03-06T07:28:34.042936+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:14.748127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032982+0000","last_change":"2024-03-06T07:28:34.032982+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032982+0000","last_undegraded":"2024-03-06T07:28:34.032982+0000","last_fullsized":"2024-03-06T07:28:34.032982+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.050323+0000","last_change":"2024-03-06T07:28:34.048173+0000","last_active":"2024-03-06T07:28:34.050323+0000","last_peered":"2024-03-06T07:28:34.050323+0000","last_clean":"2024-03-06T07:28:34.050323+0000","last_became_active":"2024-03-06T07:28:34.047879+0000","last_became_peered":"2024-03-06T07:28:34.047879+0000","last_unstale":"2024-03-06T07:28:34.050323+0000","last_undegraded":"2024-03-06T07:28:34.050323+0000","last_fullsized":"2024-03-06T07:28:34.050323+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:42.229556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039603+0000","last_change":"2024-03-06T07:28:34.039603+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039603+0000","last_undegraded":"2024-03-06T07:28:34.039603+0000","last_fullsized":"2024-03-06T07:28:34.039603+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.047805+0000","last_change":"2024-03-06T07:28:34.045231+0000","last_active":"2024-03-06T07:28:34.047805+0000","last_peered":"2024-03-06T07:28:34.047805+0000","last_clean":"2024-03-06T07:28:34.047805+0000","last_became_active":"2024-03-06T07:28:34.044892+0000","last_became_peered":"2024-03-06T07:28:34.044892+0000","last_unstale":"2024-03-06T07:28:34.047805+0000","last_undegraded":"2024-03-06T07:28:34.047805+0000","last_fullsized":"2024-03-06T07:28:34.047805+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:02.569692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032242+0000","last_change":"2024-03-06T07:28:34.032242+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032242+0000","last_undegraded":"2024-03-06T07:28:34.032242+0000","last_fullsized":"2024-03-06T07:28:34.032242+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.047690+0000","last_change":"2024-03-06T07:28:34.045006+0000","last_active":"2024-03-06T07:28:34.047690+0000","last_peered":"2024-03-06T07:28:34.047690+0000","last_clean":"2024-03-06T07:28:34.047690+0000","last_became_active":"2024-03-06T07:28:34.044739+0000","last_became_peered":"2024-03-06T07:28:34.044739+0000","last_unstale":"2024-03-06T07:28:34.047690+0000","last_undegraded":"2024-03-06T07:28:34.047690+0000","last_fullsized":"2024-03-06T07:28:34.047690+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:39.255500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036198+0000","last_change":"2024-03-06T07:28:34.036198+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036198+0000","last_undegraded":"2024-03-06T07:28:34.036198+0000","last_fullsized":"2024-03-06T07:28:34.036198+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.051110+0000","last_change":"2024-03-06T07:28:34.050123+0000","last_active":"2024-03-06T07:28:34.051110+0000","last_peered":"2024-03-06T07:28:34.051110+0000","last_clean":"2024-03-06T07:28:34.051110+0000","last_became_active":"2024-03-06T07:28:34.049801+0000","last_became_peered":"2024-03-06T07:28:34.049801+0000","last_unstale":"2024-03-06T07:28:34.051110+0000","last_undegraded":"2024-03-06T07:28:34.051110+0000","last_fullsized":"2024-03-06T07:28:34.051110+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:41.920850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037352+0000","last_change":"2024-03-06T07:28:34.037352+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037352+0000","last_undegraded":"2024-03-06T07:28:34.037352+0000","last_fullsized":"2024-03-06T07:28:34.037352+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.048452+0000","last_change":"2024-03-06T07:28:34.047655+0000","last_active":"2024-03-06T07:28:34.048452+0000","last_peered":"2024-03-06T07:28:34.048452+0000","last_clean":"2024-03-06T07:28:34.048452+0000","last_became_active":"2024-03-06T07:28:34.047397+0000","last_became_peered":"2024-03-06T07:28:34.047397+0000","last_unstale":"2024-03-06T07:28:34.048452+0000","last_undegraded":"2024-03-06T07:28:34.048452+0000","last_fullsized":"2024-03-06T07:28:34.048452+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:51.253409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034884+0000","last_change":"2024-03-06T07:28:34.034884+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034884+0000","last_undegraded":"2024-03-06T07:28:34.034884+0000","last_fullsized":"2024-03-06T07:28:34.034884+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.050554+0000","last_change":"2024-03-06T07:28:34.049637+0000","last_active":"2024-03-06T07:28:34.050554+0000","last_peered":"2024-03-06T07:28:34.050554+0000","last_clean":"2024-03-06T07:28:34.050554+0000","last_became_active":"2024-03-06T07:28:34.049301+0000","last_became_peered":"2024-03-06T07:28:34.049301+0000","last_unstale":"2024-03-06T07:28:34.050554+0000","last_undegraded":"2024-03-06T07:28:34.050554+0000","last_fullsized":"2024-03-06T07:28:34.050554+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:03.780188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033456+0000","last_change":"2024-03-06T07:28:34.033456+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033456+0000","last_undegraded":"2024-03-06T07:28:34.033456+0000","last_fullsized":"2024-03-06T07:28:34.033456+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.049420+0000","last_change":"2024-03-06T07:28:34.046419+0000","last_active":"2024-03-06T07:28:34.049420+0000","last_peered":"2024-03-06T07:28:34.049420+0000","last_clean":"2024-03-06T07:28:34.049420+0000","last_became_active":"2024-03-06T07:28:34.046066+0000","last_became_peered":"2024-03-06T07:28:34.046066+0000","last_unstale":"2024-03-06T07:28:34.049420+0000","last_undegraded":"2024-03-06T07:28:34.049420+0000","last_fullsized":"2024-03-06T07:28:34.049420+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:00.025032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.040463+0000","last_change":"2024-03-06T07:28:34.040463+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.040463+0000","last_undegraded":"2024-03-06T07:28:34.040463+0000","last_fullsized":"2024-03-06T07:28:34.040463+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.045627+0000","last_change":"2024-03-06T07:28:34.045103+0000","last_active":"2024-03-06T07:28:34.045627+0000","last_peered":"2024-03-06T07:28:34.045627+0000","last_clean":"2024-03-06T07:28:34.045627+0000","last_became_active":"2024-03-06T07:28:34.044786+0000","last_became_peered":"2024-03-06T07:28:34.044786+0000","last_unstale":"2024-03-06T07:28:34.045627+0000","last_undegraded":"2024-03-06T07:28:34.045627+0000","last_fullsized":"2024-03-06T07:28:34.045627+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:50.885489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.041301+0000","last_change":"2024-03-06T07:28:34.041301+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.041301+0000","last_undegraded":"2024-03-06T07:28:34.041301+0000","last_fullsized":"2024-03-06T07:28:34.041301+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.054609+0000","last_change":"2024-03-06T07:28:34.054092+0000","last_active":"2024-03-06T07:28:34.054609+0000","last_peered":"2024-03-06T07:28:34.054609+0000","last_clean":"2024-03-06T07:28:34.054609+0000","last_became_active":"2024-03-06T07:28:34.053838+0000","last_became_peered":"2024-03-06T07:28:34.053838+0000","last_unstale":"2024-03-06T07:28:34.054609+0000","last_undegraded":"2024-03-06T07:28:34.054609+0000","last_fullsized":"2024-03-06T07:28:34.054609+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:42.229556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.033981+0000","last_change":"2024-03-06T07:28:34.033981+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.033981+0000","last_undegraded":"2024-03-06T07:28:34.033981+0000","last_fullsized":"2024-03-06T07:28:34.033981+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.050540+0000","last_change":"2024-03-06T07:28:34.046882+0000","last_active":"2024-03-06T07:28:34.050540+0000","last_peered":"2024-03-06T07:28:34.050540+0000","last_clean":"2024-03-06T07:28:34.050540+0000","last_became_active":"2024-03-06T07:28:34.045955+0000","last_became_peered":"2024-03-06T07:28:34.045955+0000","last_unstale":"2024-03-06T07:28:34.050540+0000","last_undegraded":"2024-03-06T07:28:34.050540+0000","last_fullsized":"2024-03-06T07:28:34.050540+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:33.892363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036066+0000","last_change":"2024-03-06T07:28:34.036066+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036066+0000","last_undegraded":"2024-03-06T07:28:34.036066+0000","last_fullsized":"2024-03-06T07:28:34.036066+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.050448+0000","last_change":"2024-03-06T07:28:34.047502+0000","last_active":"2024-03-06T07:28:34.050448+0000","last_peered":"2024-03-06T07:28:34.050448+0000","last_clean":"2024-03-06T07:28:34.050448+0000","last_became_active":"2024-03-06T07:28:34.047242+0000","last_became_peered":"2024-03-06T07:28:34.047242+0000","last_unstale":"2024-03-06T07:28:34.050448+0000","last_undegraded":"2024-03-06T07:28:34.050448+0000","last_fullsized":"2024-03-06T07:28:34.050448+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:39.255500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037043+0000","last_change":"2024-03-06T07:28:34.037043+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037043+0000","last_undegraded":"2024-03-06T07:28:34.037043+0000","last_fullsized":"2024-03-06T07:28:34.037043+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.046728+0000","last_change":"2024-03-06T07:28:34.045868+0000","last_active":"2024-03-06T07:28:34.046728+0000","last_peered":"2024-03-06T07:28:34.046728+0000","last_clean":"2024-03-06T07:28:34.046728+0000","last_became_active":"2024-03-06T07:28:34.045604+0000","last_became_peered":"2024-03-06T07:28:34.045604+0000","last_unstale":"2024-03-06T07:28:34.046728+0000","last_undegraded":"2024-03-06T07:28:34.046728+0000","last_fullsized":"2024-03-06T07:28:34.046728+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:28.693278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037833+0000","last_change":"2024-03-06T07:28:34.037833+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037833+0000","last_undegraded":"2024-03-06T07:28:34.037833+0000","last_fullsized":"2024-03-06T07:28:34.037833+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.048202+0000","last_change":"2024-03-06T07:28:34.047667+0000","last_active":"2024-03-06T07:28:34.048202+0000","last_peered":"2024-03-06T07:28:34.048202+0000","last_clean":"2024-03-06T07:28:34.048202+0000","last_became_active":"2024-03-06T07:28:34.047280+0000","last_became_peered":"2024-03-06T07:28:34.047280+0000","last_unstale":"2024-03-06T07:28:34.048202+0000","last_undegraded":"2024-03-06T07:28:34.048202+0000","last_fullsized":"2024-03-06T07:28:34.048202+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:21.608844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034910+0000","last_change":"2024-03-06T07:28:34.034910+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034910+0000","last_undegraded":"2024-03-06T07:28:34.034910+0000","last_fullsized":"2024-03-06T07:28:34.034910+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053702+0000","last_change":"2024-03-06T07:28:34.049708+0000","last_active":"2024-03-06T07:28:34.053702+0000","last_peered":"2024-03-06T07:28:34.053702+0000","last_clean":"2024-03-06T07:28:34.053702+0000","last_became_active":"2024-03-06T07:28:34.049445+0000","last_became_peered":"2024-03-06T07:28:34.049445+0000","last_unstale":"2024-03-06T07:28:34.053702+0000","last_undegraded":"2024-03-06T07:28:34.053702+0000","last_fullsized":"2024-03-06T07:28:34.053702+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:53.020063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.039713+0000","last_change":"2024-03-06T07:28:34.039713+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.039713+0000","last_undegraded":"2024-03-06T07:28:34.039713+0000","last_fullsized":"2024-03-06T07:28:34.039713+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.040318+0000","last_change":"2024-03-06T07:28:34.038476+0000","last_active":"2024-03-06T07:28:34.040318+0000","last_peered":"2024-03-06T07:28:34.040318+0000","last_clean":"2024-03-06T07:28:34.040318+0000","last_became_active":"2024-03-06T07:28:34.038197+0000","last_became_peered":"2024-03-06T07:28:34.038197+0000","last_unstale":"2024-03-06T07:28:34.040318+0000","last_undegraded":"2024-03-06T07:28:34.040318+0000","last_fullsized":"2024-03-06T07:28:34.040318+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:49.999763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.038359+0000","last_change":"2024-03-06T07:28:34.038359+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.038359+0000","last_undegraded":"2024-03-06T07:28:34.038359+0000","last_fullsized":"2024-03-06T07:28:34.038359+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.047046+0000","last_change":"2024-03-06T07:28:34.046180+0000","last_active":"2024-03-06T07:28:34.047046+0000","last_peered":"2024-03-06T07:28:34.047046+0000","last_clean":"2024-03-06T07:28:34.047046+0000","last_became_active":"2024-03-06T07:28:34.045909+0000","last_became_peered":"2024-03-06T07:28:34.045909+0000","last_unstale":"2024-03-06T07:28:34.047046+0000","last_undegraded":"2024-03-06T07:28:34.047046+0000","last_fullsized":"2024-03-06T07:28:34.047046+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:29.293270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.032940+0000","last_change":"2024-03-06T07:28:34.032940+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.032940+0000","last_undegraded":"2024-03-06T07:28:34.032940+0000","last_fullsized":"2024-03-06T07:28:34.032940+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.048682+0000","last_change":"2024-03-06T07:28:34.043986+0000","last_active":"2024-03-06T07:28:34.048682+0000","last_peered":"2024-03-06T07:28:34.048682+0000","last_clean":"2024-03-06T07:28:34.048682+0000","last_became_active":"2024-03-06T07:28:34.043693+0000","last_became_peered":"2024-03-06T07:28:34.043693+0000","last_unstale":"2024-03-06T07:28:34.048682+0000","last_undegraded":"2024-03-06T07:28:34.048682+0000","last_fullsized":"2024-03-06T07:28:34.048682+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:12.297830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.036386+0000","last_change":"2024-03-06T07:28:34.036386+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.036386+0000","last_undegraded":"2024-03-06T07:28:34.036386+0000","last_fullsized":"2024-03-06T07:28:34.036386+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053605+0000","last_change":"2024-03-06T07:28:34.047903+0000","last_active":"2024-03-06T07:28:34.053605+0000","last_peered":"2024-03-06T07:28:34.053605+0000","last_clean":"2024-03-06T07:28:34.053605+0000","last_became_active":"2024-03-06T07:28:34.047612+0000","last_became_peered":"2024-03-06T07:28:34.047612+0000","last_unstale":"2024-03-06T07:28:34.053605+0000","last_undegraded":"2024-03-06T07:28:34.053605+0000","last_fullsized":"2024-03-06T07:28:34.053605+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:08.365596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.034459+0000","last_change":"2024-03-06T07:28:34.034459+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.034459+0000","last_undegraded":"2024-03-06T07:28:34.034459+0000","last_fullsized":"2024-03-06T07:28:34.034459+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.052224+0000","last_change":"2024-03-06T07:28:34.051791+0000","last_active":"2024-03-06T07:28:34.052224+0000","last_peered":"2024-03-06T07:28:34.052224+0000","last_clean":"2024-03-06T07:28:34.052224+0000","last_became_active":"2024-03-06T07:28:34.051472+0000","last_became_peered":"2024-03-06T07:28:34.051472+0000","last_unstale":"2024-03-06T07:28:34.052224+0000","last_undegraded":"2024-03-06T07:28:34.052224+0000","last_fullsized":"2024-03-06T07:28:34.052224+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:50.885489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037431+0000","last_change":"2024-03-06T07:28:34.037431+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037431+0000","last_undegraded":"2024-03-06T07:28:34.037431+0000","last_fullsized":"2024-03-06T07:28:34.037431+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053725+0000","last_change":"2024-03-06T07:28:34.048933+0000","last_active":"2024-03-06T07:28:34.053725+0000","last_peered":"2024-03-06T07:28:34.053725+0000","last_clean":"2024-03-06T07:28:34.053725+0000","last_became_active":"2024-03-06T07:28:34.048670+0000","last_became_peered":"2024-03-06T07:28:34.048670+0000","last_unstale":"2024-03-06T07:28:34.053725+0000","last_undegraded":"2024-03-06T07:28:34.053725+0000","last_fullsized":"2024-03-06T07:28:34.053725+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:08.365596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":1,"reported_epoch":195,"state":"creating+peering","last_fresh":"2024-03-06T07:28:34.037745+0000","last_change":"2024-03-06T07:28:34.037745+0000","last_active":"2024-03-06T07:28:34.021086+0000","last_peered":"2024-03-06T07:28:34.021086+0000","last_clean":"2024-03-06T07:28:34.021086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:34.037745+0000","last_undegraded":"2024-03-06T07:28:34.037745+0000","last_fullsized":"2024-03-06T07:28:34.037745+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.053497+0000","last_change":"2024-03-06T07:28:34.049313+0000","last_active":"2024-03-06T07:28:34.053497+0000","last_peered":"2024-03-06T07:28:34.053497+0000","last_clean":"2024-03-06T07:28:34.053497+0000","last_became_active":"2024-03-06T07:28:34.048967+0000","last_became_peered":"2024-03-06T07:28:34.048967+0000","last_unstale":"2024-03-06T07:28:34.053497+0000","last_undegraded":"2024-03-06T07:28:34.053497+0000","last_fullsized":"2024-03-06T07:28:34.053497+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:51.180609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":506,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.032085+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:28:34.032085+0000","last_peered":"2024-03-06T07:28:34.032085+0000","last_clean":"2024-03-06T07:28:34.032085+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:28:34.032085+0000","last_undegraded":"2024-03-06T07:28:34.032085+0000","last_fullsized":"2024-03-06T07:28:34.032085+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574951,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33664,"kb_used_data":2480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338176,"statfs":{"total":96636764160,"available":96602292224,"internally_reserved":0,"allocated":2539520,"data_stored":1793319,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574950,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33584,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338256,"statfs":{"total":96636764160,"available":96602374144,"internally_reserved":0,"allocated":2461696,"data_stored":1196284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574950,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33604,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338236,"statfs":{"total":96636764160,"available":96602353664,"internally_reserved":0,"allocated":2473984,"data_stored":1202951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574951,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33600,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338240,"statfs":{"total":96636764160,"available":96602357760,"internally_reserved":0,"allocated":2473984,"data_stored":1202951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574950,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33600,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338240,"statfs":{"total":96636764160,"available":96602357760,"internally_reserved":0,"allocated":2473984,"data_stored":1202951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574950,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37744,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334096,"statfs":{"total":96636764160,"available":96598114304,"internally_reserved":0,"allocated":2461696,"data_stored":1196284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574951,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33600,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338240,"statfs":{"total":96636764160,"available":96602357760,"internally_reserved":0,"allocated":2473984,"data_stored":1202951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574951,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37812,"kb_used_data":2468,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334028,"statfs":{"total":96636764160,"available":96598044672,"internally_reserved":0,"allocated":2527232,"data_stored":1786652,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:40.332 INFO:tasks.ceph.mds.a.smithi001.stderr:2024-03-06T07:28:40.330+0000 7f527d3fd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:42.611 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:42.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.710+0000 7f0847edf640 1 Processor -- start 2024-03-06T07:28:42.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.710+0000 7f0847edf640 1 -- start start 2024-03-06T07:28:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.711+0000 7f0847edf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f08400c6b10 0x7f0840155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.711+0000 7f0847edf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400bf9d0 0x7f08400c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.711+0000 7f0847edf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08400c2fc0 con 0x7f0840152650 2024-03-06T07:28:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.711+0000 7f0847edf640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0840155800 con 0x7f08400c6b10 2024-03-06T07:28:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.711+0000 7f0847edf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0840158da0 con 0x7f08400bf9d0 2024-03-06T07:28:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.711+0000 7f084647c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400bf9d0 0x7f08400c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.711+0000 7f084547a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f08400c6b10 0x7f0840155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.711+0000 7f0845c7b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0840152650 0x7f0840158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:53906/0 (socket says 172.21.15.1:53906) 2024-03-06T07:28:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.711+0000 7f084647c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400bf9d0 0x7f08400c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:38446/0 (socket says 172.21.15.1:38446) 2024-03-06T07:28:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.711+0000 7f0845c7b640 1 -- 172.21.15.1:0/1302256382 learned_addr learned my addr 172.21.15.1:0/1302256382 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.712+0000 7f084647c640 1 -- 172.21.15.1:0/1302256382 >> v2:172.21.15.98:3300/0 conn(0x7f08400c6b10 msgr2=0x7f0840155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.712+0000 7f084647c640 1 --2- 172.21.15.1:0/1302256382 >> v2:172.21.15.98:3300/0 conn(0x7f08400c6b10 0x7f0840155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.712+0000 7f084647c640 1 -- 172.21.15.1:0/1302256382 >> v1:172.21.15.98:6789/0 conn(0x7f0840152650 legacy=0x7f0840158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.712+0000 7f084647c640 1 -- 172.21.15.1:0/1302256382 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08400c6690 con 0x7f08400bf9d0 2024-03-06T07:28:42.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.713+0000 7f084647c640 1 --2- 172.21.15.1:0/1302256382 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400bf9d0 0x7f08400c1460 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0838009d90 tx=0x7f083802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c701e45e7e6bbb70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.713+0000 7f0844c79640 1 -- 172.21.15.1:0/1302256382 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0838039070 con 0x7f08400bf9d0 2024-03-06T07:28:42.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.713+0000 7f0844c79640 1 -- 172.21.15.1:0/1302256382 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083802cbc0 con 0x7f08400bf9d0 2024-03-06T07:28:42.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.713+0000 7f0844c79640 1 -- 172.21.15.1:0/1302256382 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0838034440 con 0x7f08400bf9d0 2024-03-06T07:28:42.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.713+0000 7f0847edf640 1 -- 172.21.15.1:0/1302256382 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400bf9d0 msgr2=0x7f08400c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.713+0000 7f0847edf640 1 --2- 172.21.15.1:0/1302256382 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400bf9d0 0x7f08400c1460 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0838009d90 tx=0x7f083802be10 comp rx=0 tx=0).stop 2024-03-06T07:28:42.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.713+0000 7f0847edf640 1 -- 172.21.15.1:0/1302256382 shutdown_connections 2024-03-06T07:28:42.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.713+0000 7f0847edf640 1 --2- 172.21.15.1:0/1302256382 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400bf9d0 0x7f08400c1460 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.713+0000 7f0847edf640 1 --2- 172.21.15.1:0/1302256382 >> v2:172.21.15.98:3300/0 conn(0x7f08400c6b10 0x7f0840155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.713+0000 7f0847edf640 1 -- 172.21.15.1:0/1302256382 >> 172.21.15.1:0/1302256382 conn(0x7f0840066ee0 msgr2=0x7f08400672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.713+0000 7f0847edf640 1 -- 172.21.15.1:0/1302256382 shutdown_connections 2024-03-06T07:28:42.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.713+0000 7f0847edf640 1 -- 172.21.15.1:0/1302256382 wait complete. 2024-03-06T07:28:42.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.714+0000 7f0847edf640 1 Processor -- start 2024-03-06T07:28:42.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.715+0000 7f0847edf640 1 -- start start 2024-03-06T07:28:42.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.715+0000 7f0847edf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400bf9d0 0x7f0840157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.715+0000 7f0847edf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0840152650 0x7f0840156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.715+0000 7f0847edf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f084014d740 0x7f084014cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.715+0000 7f0847edf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08401551b0 con 0x7f084014d740 2024-03-06T07:28:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.715+0000 7f0847edf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08400c6fa0 con 0x7f0840152650 2024-03-06T07:28:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.715+0000 7f0847edf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0840155490 con 0x7f08400bf9d0 2024-03-06T07:28:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.715+0000 7f084647c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f084014d740 0x7f084014cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.715+0000 7f084647c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f084014d740 0x7f084014cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54616/0 (socket says 172.21.15.1:54616) 2024-03-06T07:28:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.715+0000 7f084647c640 1 -- 172.21.15.1:0/1792788728 learned_addr learned my addr 172.21.15.1:0/1792788728 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.715+0000 7f0845c7b640 1 --2- 172.21.15.1:0/1792788728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400bf9d0 0x7f0840157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.715+0000 7f084547a640 1 --2- 172.21.15.1:0/1792788728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0840152650 0x7f0840156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.716+0000 7f084647c640 1 -- 172.21.15.1:0/1792788728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400bf9d0 msgr2=0x7f0840157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.716+0000 7f084647c640 1 --2- 172.21.15.1:0/1792788728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400bf9d0 0x7f0840157600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.716+0000 7f084647c640 1 -- 172.21.15.1:0/1792788728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0840152650 msgr2=0x7f0840156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.716+0000 7f084647c640 1 --2- 172.21.15.1:0/1792788728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0840152650 0x7f0840156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.716+0000 7f084647c640 1 -- 172.21.15.1:0/1792788728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08401eb850 con 0x7f084014d740 2024-03-06T07:28:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.716+0000 7f084647c640 1 --2- 172.21.15.1:0/1792788728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f084014d740 0x7f084014cb00 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f0838009d90 tx=0x7f0838004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.716+0000 7f08327fc640 1 -- 172.21.15.1:0/1792788728 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0838039050 con 0x7f084014d740 2024-03-06T07:28:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.716+0000 7f08327fc640 1 -- 172.21.15.1:0/1792788728 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0838045070 con 0x7f084014d740 2024-03-06T07:28:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.716+0000 7f08327fc640 1 -- 172.21.15.1:0/1792788728 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083803e830 con 0x7f084014d740 2024-03-06T07:28:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.716+0000 7f0847edf640 1 -- 172.21.15.1:0/1792788728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08401ebac0 con 0x7f084014d740 2024-03-06T07:28:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.716+0000 7f0847edf640 1 -- 172.21.15.1:0/1792788728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0840156b80 con 0x7f084014d740 2024-03-06T07:28:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.717+0000 7f0847edf640 1 -- 172.21.15.1:0/1792788728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08401ebc90 con 0x7f084014d740 2024-03-06T07:28:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.718+0000 7f08327fc640 1 -- 172.21.15.1:0/1792788728 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f083803e9d0 con 0x7f084014d740 2024-03-06T07:28:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.718+0000 7f08327fc640 1 --2- 172.21.15.1:0/1792788728 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f082406b4d0 0x7f082406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.718+0000 7f08327fc640 1 -- 172.21.15.1:0/1792788728 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f08380ad1d0 con 0x7f084014d740 2024-03-06T07:28:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.718+0000 7f0845c7b640 1 --2- 172.21.15.1:0/1792788728 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f082406b4d0 0x7f082406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.719+0000 7f0845c7b640 1 --2- 172.21.15.1:0/1792788728 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f082406b4d0 0x7f082406d990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f083c000f60 tx=0x7f083c007e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.723+0000 7f08327fc640 1 -- 172.21.15.1:0/1792788728 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f083807ba20 con 0x7f084014d740 2024-03-06T07:28:42.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.901+0000 7f0847edf640 1 -- 172.21.15.1:0/1792788728 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0840067fd0 con 0x7f082406b4d0 2024-03-06T07:28:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.908+0000 7f08327fc640 1 -- 172.21.15.1:0/1792788728 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252567 (secure 0 0 0) 0x7f0840067fd0 con 0x7f082406b4d0 2024-03-06T07:28:42.913 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:28:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.910+0000 7f0847edf640 1 -- 172.21.15.1:0/1792788728 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f082406b4d0 msgr2=0x7f082406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.910+0000 7f0847edf640 1 --2- 172.21.15.1:0/1792788728 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f082406b4d0 0x7f082406d990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f083c000f60 tx=0x7f083c007e80 comp rx=0 tx=0).stop 2024-03-06T07:28:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.910+0000 7f0847edf640 1 -- 172.21.15.1:0/1792788728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f084014d740 msgr2=0x7f084014cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.910+0000 7f0847edf640 1 --2- 172.21.15.1:0/1792788728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f084014d740 0x7f084014cb00 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f0838009d90 tx=0x7f0838004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:42.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.911+0000 7f0847edf640 1 -- 172.21.15.1:0/1792788728 shutdown_connections 2024-03-06T07:28:42.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.911+0000 7f0847edf640 1 --2- 172.21.15.1:0/1792788728 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f082406b4d0 0x7f082406d990 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.911+0000 7f0847edf640 1 --2- 172.21.15.1:0/1792788728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f084014d740 0x7f084014cb00 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.911+0000 7f0847edf640 1 --2- 172.21.15.1:0/1792788728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0840152650 0x7f0840156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.911+0000 7f0847edf640 1 --2- 172.21.15.1:0/1792788728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400bf9d0 0x7f0840157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.911+0000 7f0847edf640 1 -- 172.21.15.1:0/1792788728 >> 172.21.15.1:0/1792788728 conn(0x7f0840066ee0 msgr2=0x7f08400cae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.911+0000 7f0847edf640 1 -- 172.21.15.1:0/1792788728 shutdown_connections 2024-03-06T07:28:42.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:42.911+0000 7f0847edf640 1 -- 172.21.15.1:0/1792788728 wait complete. 2024-03-06T07:28:42.923 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-03-06T07:28:42.808993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":181,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":277484,"kb_used_data":19712,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754697236,"statfs":{"total":773094113280,"available":772809969664,"internally_reserved":0,"allocated":20185088,"data_stored":10920036,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001967"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123150+0000","last_change":"2024-03-06T07:28:35.063621+0000","last_active":"2024-03-06T07:28:37.123150+0000","last_peered":"2024-03-06T07:28:37.123150+0000","last_clean":"2024-03-06T07:28:37.123150+0000","last_became_active":"2024-03-06T07:28:35.063353+0000","last_became_peered":"2024-03-06T07:28:35.063353+0000","last_unstale":"2024-03-06T07:28:37.123150+0000","last_undegraded":"2024-03-06T07:28:37.123150+0000","last_fullsized":"2024-03-06T07:28:37.123150+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:34.457312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054072+0000","last_change":"2024-03-06T07:28:35.047776+0000","last_active":"2024-03-06T07:28:36.054072+0000","last_peered":"2024-03-06T07:28:36.054072+0000","last_clean":"2024-03-06T07:28:36.054072+0000","last_became_active":"2024-03-06T07:28:35.047485+0000","last_became_peered":"2024-03-06T07:28:35.047485+0000","last_unstale":"2024-03-06T07:28:36.054072+0000","last_undegraded":"2024-03-06T07:28:36.054072+0000","last_fullsized":"2024-03-06T07:28:36.054072+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:06.424988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123419+0000","last_change":"2024-03-06T07:28:35.061808+0000","last_active":"2024-03-06T07:28:37.123419+0000","last_peered":"2024-03-06T07:28:37.123419+0000","last_clean":"2024-03-06T07:28:37.123419+0000","last_became_active":"2024-03-06T07:28:35.061049+0000","last_became_peered":"2024-03-06T07:28:35.061049+0000","last_unstale":"2024-03-06T07:28:37.123419+0000","last_undegraded":"2024-03-06T07:28:37.123419+0000","last_fullsized":"2024-03-06T07:28:37.123419+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:41.992399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.124032+0000","last_change":"2024-03-06T07:28:35.067354+0000","last_active":"2024-03-06T07:28:37.124032+0000","last_peered":"2024-03-06T07:28:37.124032+0000","last_clean":"2024-03-06T07:28:37.124032+0000","last_became_active":"2024-03-06T07:28:35.067089+0000","last_became_peered":"2024-03-06T07:28:35.067089+0000","last_unstale":"2024-03-06T07:28:37.124032+0000","last_undegraded":"2024-03-06T07:28:37.124032+0000","last_fullsized":"2024-03-06T07:28:37.124032+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:08.882582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.058275+0000","last_change":"2024-03-06T07:28:35.068560+0000","last_active":"2024-03-06T07:28:36.058275+0000","last_peered":"2024-03-06T07:28:36.058275+0000","last_clean":"2024-03-06T07:28:36.058275+0000","last_became_active":"2024-03-06T07:28:35.068255+0000","last_became_peered":"2024-03-06T07:28:35.068255+0000","last_unstale":"2024-03-06T07:28:36.058275+0000","last_undegraded":"2024-03-06T07:28:36.058275+0000","last_fullsized":"2024-03-06T07:28:36.058275+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:32.815899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123071+0000","last_change":"2024-03-06T07:28:35.047020+0000","last_active":"2024-03-06T07:28:37.123071+0000","last_peered":"2024-03-06T07:28:37.123071+0000","last_clean":"2024-03-06T07:28:37.123071+0000","last_became_active":"2024-03-06T07:28:35.046721+0000","last_became_peered":"2024-03-06T07:28:35.046721+0000","last_unstale":"2024-03-06T07:28:37.123071+0000","last_undegraded":"2024-03-06T07:28:37.123071+0000","last_fullsized":"2024-03-06T07:28:37.123071+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:30.033576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.060457+0000","last_change":"2024-03-06T07:28:35.060767+0000","last_active":"2024-03-06T07:28:36.060457+0000","last_peered":"2024-03-06T07:28:36.060457+0000","last_clean":"2024-03-06T07:28:36.060457+0000","last_became_active":"2024-03-06T07:28:35.060477+0000","last_became_peered":"2024-03-06T07:28:35.060477+0000","last_unstale":"2024-03-06T07:28:36.060457+0000","last_undegraded":"2024-03-06T07:28:36.060457+0000","last_fullsized":"2024-03-06T07:28:36.060457+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:17.868063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.057838+0000","last_change":"2024-03-06T07:28:35.067218+0000","last_active":"2024-03-06T07:28:36.057838+0000","last_peered":"2024-03-06T07:28:36.057838+0000","last_clean":"2024-03-06T07:28:36.057838+0000","last_became_active":"2024-03-06T07:28:35.066910+0000","last_became_peered":"2024-03-06T07:28:35.066910+0000","last_unstale":"2024-03-06T07:28:36.057838+0000","last_undegraded":"2024-03-06T07:28:36.057838+0000","last_fullsized":"2024-03-06T07:28:36.057838+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:44.128344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123407+0000","last_change":"2024-03-06T07:28:35.067820+0000","last_active":"2024-03-06T07:28:37.123407+0000","last_peered":"2024-03-06T07:28:37.123407+0000","last_clean":"2024-03-06T07:28:37.123407+0000","last_became_active":"2024-03-06T07:28:35.067538+0000","last_became_peered":"2024-03-06T07:28:35.067538+0000","last_unstale":"2024-03-06T07:28:37.123407+0000","last_undegraded":"2024-03-06T07:28:37.123407+0000","last_fullsized":"2024-03-06T07:28:37.123407+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:47.635747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123867+0000","last_change":"2024-03-06T07:28:35.068530+0000","last_active":"2024-03-06T07:28:37.123867+0000","last_peered":"2024-03-06T07:28:37.123867+0000","last_clean":"2024-03-06T07:28:37.123867+0000","last_became_active":"2024-03-06T07:28:35.068271+0000","last_became_peered":"2024-03-06T07:28:35.068271+0000","last_unstale":"2024-03-06T07:28:37.123867+0000","last_undegraded":"2024-03-06T07:28:37.123867+0000","last_fullsized":"2024-03-06T07:28:37.123867+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:45.435506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123564+0000","last_change":"2024-03-06T07:28:35.067221+0000","last_active":"2024-03-06T07:28:37.123564+0000","last_peered":"2024-03-06T07:28:37.123564+0000","last_clean":"2024-03-06T07:28:37.123564+0000","last_became_active":"2024-03-06T07:28:35.066856+0000","last_became_peered":"2024-03-06T07:28:35.066856+0000","last_unstale":"2024-03-06T07:28:37.123564+0000","last_undegraded":"2024-03-06T07:28:37.123564+0000","last_fullsized":"2024-03-06T07:28:37.123564+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:02:26.409494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123547+0000","last_change":"2024-03-06T07:28:35.057885+0000","last_active":"2024-03-06T07:28:37.123547+0000","last_peered":"2024-03-06T07:28:37.123547+0000","last_clean":"2024-03-06T07:28:37.123547+0000","last_became_active":"2024-03-06T07:28:35.057607+0000","last_became_peered":"2024-03-06T07:28:35.057607+0000","last_unstale":"2024-03-06T07:28:37.123547+0000","last_undegraded":"2024-03-06T07:28:37.123547+0000","last_fullsized":"2024-03-06T07:28:37.123547+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:13.965725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.055319+0000","last_change":"2024-03-06T07:28:35.062298+0000","last_active":"2024-03-06T07:28:36.055319+0000","last_peered":"2024-03-06T07:28:36.055319+0000","last_clean":"2024-03-06T07:28:36.055319+0000","last_became_active":"2024-03-06T07:28:35.061969+0000","last_became_peered":"2024-03-06T07:28:35.061969+0000","last_unstale":"2024-03-06T07:28:36.055319+0000","last_undegraded":"2024-03-06T07:28:36.055319+0000","last_fullsized":"2024-03-06T07:28:36.055319+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:17.868063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.119339+0000","last_change":"2024-03-06T07:28:35.054130+0000","last_active":"2024-03-06T07:28:36.119339+0000","last_peered":"2024-03-06T07:28:36.119339+0000","last_clean":"2024-03-06T07:28:36.119339+0000","last_became_active":"2024-03-06T07:28:35.053840+0000","last_became_peered":"2024-03-06T07:28:35.053840+0000","last_unstale":"2024-03-06T07:28:36.119339+0000","last_undegraded":"2024-03-06T07:28:36.119339+0000","last_fullsized":"2024-03-06T07:28:36.119339+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:41.992399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.055237+0000","last_change":"2024-03-06T07:28:35.046851+0000","last_active":"2024-03-06T07:28:36.055237+0000","last_peered":"2024-03-06T07:28:36.055237+0000","last_clean":"2024-03-06T07:28:36.055237+0000","last_became_active":"2024-03-06T07:28:35.046543+0000","last_became_peered":"2024-03-06T07:28:35.046543+0000","last_unstale":"2024-03-06T07:28:36.055237+0000","last_undegraded":"2024-03-06T07:28:36.055237+0000","last_fullsized":"2024-03-06T07:28:36.055237+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:43.019941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.122812+0000","last_change":"2024-03-06T07:28:35.057578+0000","last_active":"2024-03-06T07:28:37.122812+0000","last_peered":"2024-03-06T07:28:37.122812+0000","last_clean":"2024-03-06T07:28:37.122812+0000","last_became_active":"2024-03-06T07:28:35.057297+0000","last_became_peered":"2024-03-06T07:28:35.057297+0000","last_unstale":"2024-03-06T07:28:37.122812+0000","last_undegraded":"2024-03-06T07:28:37.122812+0000","last_fullsized":"2024-03-06T07:28:37.122812+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:38.333168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.055247+0000","last_change":"2024-03-06T07:28:35.065643+0000","last_active":"2024-03-06T07:28:36.055247+0000","last_peered":"2024-03-06T07:28:36.055247+0000","last_clean":"2024-03-06T07:28:36.055247+0000","last_became_active":"2024-03-06T07:28:35.065367+0000","last_became_peered":"2024-03-06T07:28:35.065367+0000","last_unstale":"2024-03-06T07:28:36.055247+0000","last_undegraded":"2024-03-06T07:28:36.055247+0000","last_fullsized":"2024-03-06T07:28:36.055247+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:32.815899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123941+0000","last_change":"2024-03-06T07:28:35.060664+0000","last_active":"2024-03-06T07:28:37.123941+0000","last_peered":"2024-03-06T07:28:37.123941+0000","last_clean":"2024-03-06T07:28:37.123941+0000","last_became_active":"2024-03-06T07:28:35.060392+0000","last_became_peered":"2024-03-06T07:28:35.060392+0000","last_unstale":"2024-03-06T07:28:37.123941+0000","last_undegraded":"2024-03-06T07:28:37.123941+0000","last_fullsized":"2024-03-06T07:28:37.123941+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:27:52.536166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.055071+0000","last_change":"2024-03-06T07:28:35.063791+0000","last_active":"2024-03-06T07:28:36.055071+0000","last_peered":"2024-03-06T07:28:36.055071+0000","last_clean":"2024-03-06T07:28:36.055071+0000","last_became_active":"2024-03-06T07:28:35.062459+0000","last_became_peered":"2024-03-06T07:28:35.062459+0000","last_unstale":"2024-03-06T07:28:36.055071+0000","last_undegraded":"2024-03-06T07:28:36.055071+0000","last_fullsized":"2024-03-06T07:28:36.055071+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:08.372881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054020+0000","last_change":"2024-03-06T07:28:35.058132+0000","last_active":"2024-03-06T07:28:36.054020+0000","last_peered":"2024-03-06T07:28:36.054020+0000","last_clean":"2024-03-06T07:28:36.054020+0000","last_became_active":"2024-03-06T07:28:35.057842+0000","last_became_peered":"2024-03-06T07:28:35.057842+0000","last_unstale":"2024-03-06T07:28:36.054020+0000","last_undegraded":"2024-03-06T07:28:36.054020+0000","last_fullsized":"2024-03-06T07:28:36.054020+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:16:31.338308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.060484+0000","last_change":"2024-03-06T07:28:35.060209+0000","last_active":"2024-03-06T07:28:36.060484+0000","last_peered":"2024-03-06T07:28:36.060484+0000","last_clean":"2024-03-06T07:28:36.060484+0000","last_became_active":"2024-03-06T07:28:35.059918+0000","last_became_peered":"2024-03-06T07:28:35.059918+0000","last_unstale":"2024-03-06T07:28:36.060484+0000","last_undegraded":"2024-03-06T07:28:36.060484+0000","last_fullsized":"2024-03-06T07:28:36.060484+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:51.942949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054783+0000","last_change":"2024-03-06T07:28:35.050582+0000","last_active":"2024-03-06T07:28:36.054783+0000","last_peered":"2024-03-06T07:28:36.054783+0000","last_clean":"2024-03-06T07:28:36.054783+0000","last_became_active":"2024-03-06T07:28:35.050293+0000","last_became_peered":"2024-03-06T07:28:35.050293+0000","last_unstale":"2024-03-06T07:28:36.054783+0000","last_undegraded":"2024-03-06T07:28:36.054783+0000","last_fullsized":"2024-03-06T07:28:36.054783+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:44.128344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.055387+0000","last_change":"2024-03-06T07:28:35.062001+0000","last_active":"2024-03-06T07:28:36.055387+0000","last_peered":"2024-03-06T07:28:36.055387+0000","last_clean":"2024-03-06T07:28:36.055387+0000","last_became_active":"2024-03-06T07:28:35.061648+0000","last_became_peered":"2024-03-06T07:28:35.061648+0000","last_unstale":"2024-03-06T07:28:36.055387+0000","last_undegraded":"2024-03-06T07:28:36.055387+0000","last_fullsized":"2024-03-06T07:28:36.055387+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:17.937722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054271+0000","last_change":"2024-03-06T07:28:35.056837+0000","last_active":"2024-03-06T07:28:36.054271+0000","last_peered":"2024-03-06T07:28:36.054271+0000","last_clean":"2024-03-06T07:28:36.054271+0000","last_became_active":"2024-03-06T07:28:35.056047+0000","last_became_peered":"2024-03-06T07:28:35.056047+0000","last_unstale":"2024-03-06T07:28:36.054271+0000","last_undegraded":"2024-03-06T07:28:36.054271+0000","last_fullsized":"2024-03-06T07:28:36.054271+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:56.411642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054265+0000","last_change":"2024-03-06T07:28:35.046070+0000","last_active":"2024-03-06T07:28:36.054265+0000","last_peered":"2024-03-06T07:28:36.054265+0000","last_clean":"2024-03-06T07:28:36.054265+0000","last_became_active":"2024-03-06T07:28:35.045766+0000","last_became_peered":"2024-03-06T07:28:35.045766+0000","last_unstale":"2024-03-06T07:28:36.054265+0000","last_undegraded":"2024-03-06T07:28:36.054265+0000","last_fullsized":"2024-03-06T07:28:36.054265+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:25.147925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.060617+0000","last_change":"2024-03-06T07:28:35.061064+0000","last_active":"2024-03-06T07:28:36.060617+0000","last_peered":"2024-03-06T07:28:36.060617+0000","last_clean":"2024-03-06T07:28:36.060617+0000","last_became_active":"2024-03-06T07:28:35.060744+0000","last_became_peered":"2024-03-06T07:28:35.060744+0000","last_unstale":"2024-03-06T07:28:36.060617+0000","last_undegraded":"2024-03-06T07:28:36.060617+0000","last_fullsized":"2024-03-06T07:28:36.060617+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:29.060743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.058192+0000","last_change":"2024-03-06T07:28:35.071092+0000","last_active":"2024-03-06T07:28:36.058192+0000","last_peered":"2024-03-06T07:28:36.058192+0000","last_clean":"2024-03-06T07:28:36.058192+0000","last_became_active":"2024-03-06T07:28:35.070744+0000","last_became_peered":"2024-03-06T07:28:35.070744+0000","last_unstale":"2024-03-06T07:28:36.058192+0000","last_undegraded":"2024-03-06T07:28:36.058192+0000","last_fullsized":"2024-03-06T07:28:36.058192+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:17.868063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123651+0000","last_change":"2024-03-06T07:28:35.064992+0000","last_active":"2024-03-06T07:28:37.123651+0000","last_peered":"2024-03-06T07:28:37.123651+0000","last_clean":"2024-03-06T07:28:37.123651+0000","last_became_active":"2024-03-06T07:28:35.058230+0000","last_became_peered":"2024-03-06T07:28:35.058230+0000","last_unstale":"2024-03-06T07:28:37.123651+0000","last_undegraded":"2024-03-06T07:28:37.123651+0000","last_fullsized":"2024-03-06T07:28:37.123651+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:00.741672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.119494+0000","last_change":"2024-03-06T07:28:35.062029+0000","last_active":"2024-03-06T07:28:36.119494+0000","last_peered":"2024-03-06T07:28:36.119494+0000","last_clean":"2024-03-06T07:28:36.119494+0000","last_became_active":"2024-03-06T07:28:35.061729+0000","last_became_peered":"2024-03-06T07:28:35.061729+0000","last_unstale":"2024-03-06T07:28:36.119494+0000","last_undegraded":"2024-03-06T07:28:36.119494+0000","last_fullsized":"2024-03-06T07:28:36.119494+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:03.943375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.119351+0000","last_change":"2024-03-06T07:28:35.065504+0000","last_active":"2024-03-06T07:28:36.119351+0000","last_peered":"2024-03-06T07:28:36.119351+0000","last_clean":"2024-03-06T07:28:36.119351+0000","last_became_active":"2024-03-06T07:28:35.065177+0000","last_became_peered":"2024-03-06T07:28:35.065177+0000","last_unstale":"2024-03-06T07:28:36.119351+0000","last_undegraded":"2024-03-06T07:28:36.119351+0000","last_fullsized":"2024-03-06T07:28:36.119351+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:02:26.409494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.053664+0000","last_change":"2024-03-06T07:28:35.049204+0000","last_active":"2024-03-06T07:28:36.053664+0000","last_peered":"2024-03-06T07:28:36.053664+0000","last_clean":"2024-03-06T07:28:36.053664+0000","last_became_active":"2024-03-06T07:28:35.048919+0000","last_became_peered":"2024-03-06T07:28:35.048919+0000","last_unstale":"2024-03-06T07:28:36.053664+0000","last_undegraded":"2024-03-06T07:28:36.053664+0000","last_fullsized":"2024-03-06T07:28:36.053664+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:21.064123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123407+0000","last_change":"2024-03-06T07:28:35.066944+0000","last_active":"2024-03-06T07:28:37.123407+0000","last_peered":"2024-03-06T07:28:37.123407+0000","last_clean":"2024-03-06T07:28:37.123407+0000","last_became_active":"2024-03-06T07:28:35.066665+0000","last_became_peered":"2024-03-06T07:28:35.066665+0000","last_unstale":"2024-03-06T07:28:37.123407+0000","last_undegraded":"2024-03-06T07:28:37.123407+0000","last_fullsized":"2024-03-06T07:28:37.123407+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:49.760879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.053504+0000","last_change":"2024-03-06T07:28:35.050430+0000","last_active":"2024-03-06T07:28:36.053504+0000","last_peered":"2024-03-06T07:28:36.053504+0000","last_clean":"2024-03-06T07:28:36.053504+0000","last_became_active":"2024-03-06T07:28:35.050156+0000","last_became_peered":"2024-03-06T07:28:35.050156+0000","last_unstale":"2024-03-06T07:28:36.053504+0000","last_undegraded":"2024-03-06T07:28:36.053504+0000","last_fullsized":"2024-03-06T07:28:36.053504+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:50.866415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"196'4","reported_seq":18,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123246+0000","last_change":"2024-03-06T07:28:34.045692+0000","last_active":"2024-03-06T07:28:37.123246+0000","last_peered":"2024-03-06T07:28:37.123246+0000","last_clean":"2024-03-06T07:28:37.123246+0000","last_became_active":"2024-03-06T07:28:34.045392+0000","last_became_peered":"2024-03-06T07:28:34.045392+0000","last_unstale":"2024-03-06T07:28:37.123246+0000","last_undegraded":"2024-03-06T07:28:37.123246+0000","last_fullsized":"2024-03-06T07:28:37.123246+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:52.434873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054491+0000","last_change":"2024-03-06T07:28:35.066440+0000","last_active":"2024-03-06T07:28:36.054491+0000","last_peered":"2024-03-06T07:28:36.054491+0000","last_clean":"2024-03-06T07:28:36.054491+0000","last_became_active":"2024-03-06T07:28:35.066137+0000","last_became_peered":"2024-03-06T07:28:35.066137+0000","last_unstale":"2024-03-06T07:28:36.054491+0000","last_undegraded":"2024-03-06T07:28:36.054491+0000","last_fullsized":"2024-03-06T07:28:36.054491+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:14:40.795727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054585+0000","last_change":"2024-03-06T07:28:34.048342+0000","last_active":"2024-03-06T07:28:36.054585+0000","last_peered":"2024-03-06T07:28:36.054585+0000","last_clean":"2024-03-06T07:28:36.054585+0000","last_became_active":"2024-03-06T07:28:34.048077+0000","last_became_peered":"2024-03-06T07:28:34.048077+0000","last_unstale":"2024-03-06T07:28:36.054585+0000","last_undegraded":"2024-03-06T07:28:36.054585+0000","last_fullsized":"2024-03-06T07:28:36.054585+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:53.636785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123333+0000","last_change":"2024-03-06T07:28:35.046745+0000","last_active":"2024-03-06T07:28:37.123333+0000","last_peered":"2024-03-06T07:28:37.123333+0000","last_clean":"2024-03-06T07:28:37.123333+0000","last_became_active":"2024-03-06T07:28:35.046404+0000","last_became_peered":"2024-03-06T07:28:35.046404+0000","last_unstale":"2024-03-06T07:28:37.123333+0000","last_undegraded":"2024-03-06T07:28:37.123333+0000","last_fullsized":"2024-03-06T07:28:37.123333+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:29.088616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.119603+0000","last_change":"2024-03-06T07:28:34.043250+0000","last_active":"2024-03-06T07:28:36.119603+0000","last_peered":"2024-03-06T07:28:36.119603+0000","last_clean":"2024-03-06T07:28:36.119603+0000","last_became_active":"2024-03-06T07:28:34.042985+0000","last_became_peered":"2024-03-06T07:28:34.042985+0000","last_unstale":"2024-03-06T07:28:36.119603+0000","last_undegraded":"2024-03-06T07:28:36.119603+0000","last_fullsized":"2024-03-06T07:28:36.119603+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:42.014908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.119536+0000","last_change":"2024-03-06T07:28:35.047083+0000","last_active":"2024-03-06T07:28:36.119536+0000","last_peered":"2024-03-06T07:28:36.119536+0000","last_clean":"2024-03-06T07:28:36.119536+0000","last_became_active":"2024-03-06T07:28:35.046806+0000","last_became_peered":"2024-03-06T07:28:35.046806+0000","last_unstale":"2024-03-06T07:28:36.119536+0000","last_undegraded":"2024-03-06T07:28:36.119536+0000","last_fullsized":"2024-03-06T07:28:36.119536+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:18.011748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"196'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.055637+0000","last_change":"2024-03-06T07:28:34.039018+0000","last_active":"2024-03-06T07:28:36.055637+0000","last_peered":"2024-03-06T07:28:36.055637+0000","last_clean":"2024-03-06T07:28:36.055637+0000","last_became_active":"2024-03-06T07:28:34.038734+0000","last_became_peered":"2024-03-06T07:28:34.038734+0000","last_unstale":"2024-03-06T07:28:36.055637+0000","last_undegraded":"2024-03-06T07:28:36.055637+0000","last_fullsized":"2024-03-06T07:28:36.055637+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:29.475832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.060697+0000","last_change":"2024-03-06T07:28:35.060153+0000","last_active":"2024-03-06T07:28:36.060697+0000","last_peered":"2024-03-06T07:28:36.060697+0000","last_clean":"2024-03-06T07:28:36.060697+0000","last_became_active":"2024-03-06T07:28:35.059856+0000","last_became_peered":"2024-03-06T07:28:35.059856+0000","last_unstale":"2024-03-06T07:28:36.060697+0000","last_undegraded":"2024-03-06T07:28:36.060697+0000","last_fullsized":"2024-03-06T07:28:36.060697+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:56.069722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"196'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.119442+0000","last_change":"2024-03-06T07:28:34.044403+0000","last_active":"2024-03-06T07:28:36.119442+0000","last_peered":"2024-03-06T07:28:36.119442+0000","last_clean":"2024-03-06T07:28:36.119442+0000","last_became_active":"2024-03-06T07:28:34.044130+0000","last_became_peered":"2024-03-06T07:28:34.044130+0000","last_unstale":"2024-03-06T07:28:36.119442+0000","last_undegraded":"2024-03-06T07:28:36.119442+0000","last_fullsized":"2024-03-06T07:28:36.119442+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:16.932688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123788+0000","last_change":"2024-03-06T07:28:35.066613+0000","last_active":"2024-03-06T07:28:37.123788+0000","last_peered":"2024-03-06T07:28:37.123788+0000","last_clean":"2024-03-06T07:28:37.123788+0000","last_became_active":"2024-03-06T07:28:35.066342+0000","last_became_peered":"2024-03-06T07:28:35.066342+0000","last_unstale":"2024-03-06T07:28:37.123788+0000","last_undegraded":"2024-03-06T07:28:37.123788+0000","last_fullsized":"2024-03-06T07:28:37.123788+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:03.943375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.122797+0000","last_change":"2024-03-06T07:28:34.054170+0000","last_active":"2024-03-06T07:28:37.122797+0000","last_peered":"2024-03-06T07:28:37.122797+0000","last_clean":"2024-03-06T07:28:37.122797+0000","last_became_active":"2024-03-06T07:28:34.053871+0000","last_became_peered":"2024-03-06T07:28:34.053871+0000","last_unstale":"2024-03-06T07:28:37.122797+0000","last_undegraded":"2024-03-06T07:28:37.122797+0000","last_fullsized":"2024-03-06T07:28:37.122797+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:07.581528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.058387+0000","last_change":"2024-03-06T07:28:35.069682+0000","last_active":"2024-03-06T07:28:36.058387+0000","last_peered":"2024-03-06T07:28:36.058387+0000","last_clean":"2024-03-06T07:28:36.058387+0000","last_became_active":"2024-03-06T07:28:35.069370+0000","last_became_peered":"2024-03-06T07:28:35.069370+0000","last_unstale":"2024-03-06T07:28:36.058387+0000","last_undegraded":"2024-03-06T07:28:36.058387+0000","last_fullsized":"2024-03-06T07:28:36.058387+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:51.942949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.122837+0000","last_change":"2024-03-06T07:28:34.042955+0000","last_active":"2024-03-06T07:28:37.122837+0000","last_peered":"2024-03-06T07:28:37.122837+0000","last_clean":"2024-03-06T07:28:37.122837+0000","last_became_active":"2024-03-06T07:28:34.042684+0000","last_became_peered":"2024-03-06T07:28:34.042684+0000","last_unstale":"2024-03-06T07:28:37.122837+0000","last_undegraded":"2024-03-06T07:28:37.122837+0000","last_fullsized":"2024-03-06T07:28:37.122837+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:09:03.383826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.060511+0000","last_change":"2024-03-06T07:28:35.054704+0000","last_active":"2024-03-06T07:28:36.060511+0000","last_peered":"2024-03-06T07:28:36.060511+0000","last_clean":"2024-03-06T07:28:36.060511+0000","last_became_active":"2024-03-06T07:28:35.054428+0000","last_became_peered":"2024-03-06T07:28:35.054428+0000","last_unstale":"2024-03-06T07:28:36.060511+0000","last_undegraded":"2024-03-06T07:28:36.060511+0000","last_fullsized":"2024-03-06T07:28:36.060511+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:54.416533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.119843+0000","last_change":"2024-03-06T07:28:34.036082+0000","last_active":"2024-03-06T07:28:36.119843+0000","last_peered":"2024-03-06T07:28:36.119843+0000","last_clean":"2024-03-06T07:28:36.119843+0000","last_became_active":"2024-03-06T07:28:34.035802+0000","last_became_peered":"2024-03-06T07:28:34.035802+0000","last_unstale":"2024-03-06T07:28:36.119843+0000","last_undegraded":"2024-03-06T07:28:36.119843+0000","last_fullsized":"2024-03-06T07:28:36.119843+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:07.367848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054617+0000","last_change":"2024-03-06T07:28:35.052281+0000","last_active":"2024-03-06T07:28:36.054617+0000","last_peered":"2024-03-06T07:28:36.054617+0000","last_clean":"2024-03-06T07:28:36.054617+0000","last_became_active":"2024-03-06T07:28:35.052002+0000","last_became_peered":"2024-03-06T07:28:35.052002+0000","last_unstale":"2024-03-06T07:28:36.054617+0000","last_undegraded":"2024-03-06T07:28:36.054617+0000","last_fullsized":"2024-03-06T07:28:36.054617+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:36.245321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123985+0000","last_change":"2024-03-06T07:28:34.051147+0000","last_active":"2024-03-06T07:28:37.123985+0000","last_peered":"2024-03-06T07:28:37.123985+0000","last_clean":"2024-03-06T07:28:37.123985+0000","last_became_active":"2024-03-06T07:28:34.050807+0000","last_became_peered":"2024-03-06T07:28:34.050807+0000","last_unstale":"2024-03-06T07:28:37.123985+0000","last_undegraded":"2024-03-06T07:28:37.123985+0000","last_fullsized":"2024-03-06T07:28:37.123985+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:18.838141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123274+0000","last_change":"2024-03-06T07:28:35.066478+0000","last_active":"2024-03-06T07:28:37.123274+0000","last_peered":"2024-03-06T07:28:37.123274+0000","last_clean":"2024-03-06T07:28:37.123274+0000","last_became_active":"2024-03-06T07:28:35.066218+0000","last_became_peered":"2024-03-06T07:28:35.066218+0000","last_unstale":"2024-03-06T07:28:37.123274+0000","last_undegraded":"2024-03-06T07:28:37.123274+0000","last_fullsized":"2024-03-06T07:28:37.123274+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:50.700127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"196'3","reported_seq":17,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.058254+0000","last_change":"2024-03-06T07:28:34.044501+0000","last_active":"2024-03-06T07:28:36.058254+0000","last_peered":"2024-03-06T07:28:36.058254+0000","last_clean":"2024-03-06T07:28:36.058254+0000","last_became_active":"2024-03-06T07:28:34.044057+0000","last_became_peered":"2024-03-06T07:28:34.044057+0000","last_unstale":"2024-03-06T07:28:36.058254+0000","last_undegraded":"2024-03-06T07:28:36.058254+0000","last_fullsized":"2024-03-06T07:28:36.058254+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:04.137572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.060443+0000","last_change":"2024-03-06T07:28:35.068648+0000","last_active":"2024-03-06T07:28:36.060443+0000","last_peered":"2024-03-06T07:28:36.060443+0000","last_clean":"2024-03-06T07:28:36.060443+0000","last_became_active":"2024-03-06T07:28:35.068379+0000","last_became_peered":"2024-03-06T07:28:35.068379+0000","last_unstale":"2024-03-06T07:28:36.060443+0000","last_undegraded":"2024-03-06T07:28:36.060443+0000","last_fullsized":"2024-03-06T07:28:36.060443+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:32.815899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"196'6","reported_seq":20,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123694+0000","last_change":"2024-03-06T07:28:34.041503+0000","last_active":"2024-03-06T07:28:37.123694+0000","last_peered":"2024-03-06T07:28:37.123694+0000","last_clean":"2024-03-06T07:28:37.123694+0000","last_became_active":"2024-03-06T07:28:34.041217+0000","last_became_peered":"2024-03-06T07:28:34.041217+0000","last_unstale":"2024-03-06T07:28:37.123694+0000","last_undegraded":"2024-03-06T07:28:37.123694+0000","last_fullsized":"2024-03-06T07:28:37.123694+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:26.553524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054040+0000","last_change":"2024-03-06T07:28:35.072345+0000","last_active":"2024-03-06T07:28:36.054040+0000","last_peered":"2024-03-06T07:28:36.054040+0000","last_clean":"2024-03-06T07:28:36.054040+0000","last_became_active":"2024-03-06T07:28:35.072086+0000","last_became_peered":"2024-03-06T07:28:35.072086+0000","last_unstale":"2024-03-06T07:28:36.054040+0000","last_undegraded":"2024-03-06T07:28:36.054040+0000","last_fullsized":"2024-03-06T07:28:36.054040+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:41.995682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.055755+0000","last_change":"2024-03-06T07:28:34.048173+0000","last_active":"2024-03-06T07:28:36.055755+0000","last_peered":"2024-03-06T07:28:36.055755+0000","last_clean":"2024-03-06T07:28:36.055755+0000","last_became_active":"2024-03-06T07:28:34.047879+0000","last_became_peered":"2024-03-06T07:28:34.047879+0000","last_unstale":"2024-03-06T07:28:36.055755+0000","last_undegraded":"2024-03-06T07:28:36.055755+0000","last_fullsized":"2024-03-06T07:28:36.055755+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:07.581528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.060658+0000","last_change":"2024-03-06T07:28:35.061667+0000","last_active":"2024-03-06T07:28:36.060658+0000","last_peered":"2024-03-06T07:28:36.060658+0000","last_clean":"2024-03-06T07:28:36.060658+0000","last_became_active":"2024-03-06T07:28:35.061307+0000","last_became_peered":"2024-03-06T07:28:35.061307+0000","last_unstale":"2024-03-06T07:28:36.060658+0000","last_undegraded":"2024-03-06T07:28:36.060658+0000","last_fullsized":"2024-03-06T07:28:36.060658+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:31.224726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.057934+0000","last_change":"2024-03-06T07:28:34.045231+0000","last_active":"2024-03-06T07:28:36.057934+0000","last_peered":"2024-03-06T07:28:36.057934+0000","last_clean":"2024-03-06T07:28:36.057934+0000","last_became_active":"2024-03-06T07:28:34.044892+0000","last_became_peered":"2024-03-06T07:28:34.044892+0000","last_unstale":"2024-03-06T07:28:36.057934+0000","last_undegraded":"2024-03-06T07:28:36.057934+0000","last_fullsized":"2024-03-06T07:28:36.057934+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:06.104970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054720+0000","last_change":"2024-03-06T07:28:35.054429+0000","last_active":"2024-03-06T07:28:36.054720+0000","last_peered":"2024-03-06T07:28:36.054720+0000","last_clean":"2024-03-06T07:28:36.054720+0000","last_became_active":"2024-03-06T07:28:35.054150+0000","last_became_peered":"2024-03-06T07:28:35.054150+0000","last_unstale":"2024-03-06T07:28:36.054720+0000","last_undegraded":"2024-03-06T07:28:36.054720+0000","last_fullsized":"2024-03-06T07:28:36.054720+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:00.419130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.057666+0000","last_change":"2024-03-06T07:28:34.045006+0000","last_active":"2024-03-06T07:28:36.057666+0000","last_peered":"2024-03-06T07:28:36.057666+0000","last_clean":"2024-03-06T07:28:36.057666+0000","last_became_active":"2024-03-06T07:28:34.044739+0000","last_became_peered":"2024-03-06T07:28:34.044739+0000","last_unstale":"2024-03-06T07:28:36.057666+0000","last_undegraded":"2024-03-06T07:28:36.057666+0000","last_fullsized":"2024-03-06T07:28:36.057666+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:11.606416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.058379+0000","last_change":"2024-03-06T07:28:35.068794+0000","last_active":"2024-03-06T07:28:36.058379+0000","last_peered":"2024-03-06T07:28:36.058379+0000","last_clean":"2024-03-06T07:28:36.058379+0000","last_became_active":"2024-03-06T07:28:35.068434+0000","last_became_peered":"2024-03-06T07:28:35.068434+0000","last_unstale":"2024-03-06T07:28:36.058379+0000","last_undegraded":"2024-03-06T07:28:36.058379+0000","last_fullsized":"2024-03-06T07:28:36.058379+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:54.416533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"196'8","reported_seq":22,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.053507+0000","last_change":"2024-03-06T07:28:34.050123+0000","last_active":"2024-03-06T07:28:36.053507+0000","last_peered":"2024-03-06T07:28:36.053507+0000","last_clean":"2024-03-06T07:28:36.053507+0000","last_became_active":"2024-03-06T07:28:34.049801+0000","last_became_peered":"2024-03-06T07:28:34.049801+0000","last_unstale":"2024-03-06T07:28:36.053507+0000","last_undegraded":"2024-03-06T07:28:36.053507+0000","last_fullsized":"2024-03-06T07:28:36.053507+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:59.973743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.057731+0000","last_change":"2024-03-06T07:28:35.058718+0000","last_active":"2024-03-06T07:28:36.057731+0000","last_peered":"2024-03-06T07:28:36.057731+0000","last_clean":"2024-03-06T07:28:36.057731+0000","last_became_active":"2024-03-06T07:28:35.058426+0000","last_became_peered":"2024-03-06T07:28:35.058426+0000","last_unstale":"2024-03-06T07:28:36.057731+0000","last_undegraded":"2024-03-06T07:28:36.057731+0000","last_fullsized":"2024-03-06T07:28:36.057731+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:36.245321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"196'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123454+0000","last_change":"2024-03-06T07:28:34.047655+0000","last_active":"2024-03-06T07:28:37.123454+0000","last_peered":"2024-03-06T07:28:37.123454+0000","last_clean":"2024-03-06T07:28:37.123454+0000","last_became_active":"2024-03-06T07:28:34.047397+0000","last_became_peered":"2024-03-06T07:28:34.047397+0000","last_unstale":"2024-03-06T07:28:37.123454+0000","last_undegraded":"2024-03-06T07:28:37.123454+0000","last_fullsized":"2024-03-06T07:28:37.123454+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:56.893296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.057721+0000","last_change":"2024-03-06T07:28:35.067466+0000","last_active":"2024-03-06T07:28:36.057721+0000","last_peered":"2024-03-06T07:28:36.057721+0000","last_clean":"2024-03-06T07:28:36.057721+0000","last_became_active":"2024-03-06T07:28:35.067152+0000","last_became_peered":"2024-03-06T07:28:35.067152+0000","last_unstale":"2024-03-06T07:28:36.057721+0000","last_undegraded":"2024-03-06T07:28:36.057721+0000","last_fullsized":"2024-03-06T07:28:36.057721+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:56.411642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054166+0000","last_change":"2024-03-06T07:28:34.049637+0000","last_active":"2024-03-06T07:28:36.054166+0000","last_peered":"2024-03-06T07:28:36.054166+0000","last_clean":"2024-03-06T07:28:36.054166+0000","last_became_active":"2024-03-06T07:28:34.049301+0000","last_became_peered":"2024-03-06T07:28:34.049301+0000","last_unstale":"2024-03-06T07:28:36.054166+0000","last_undegraded":"2024-03-06T07:28:36.054166+0000","last_fullsized":"2024-03-06T07:28:36.054166+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:35.560693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.053814+0000","last_change":"2024-03-06T07:28:35.068603+0000","last_active":"2024-03-06T07:28:36.053814+0000","last_peered":"2024-03-06T07:28:36.053814+0000","last_clean":"2024-03-06T07:28:36.053814+0000","last_became_active":"2024-03-06T07:28:35.068326+0000","last_became_peered":"2024-03-06T07:28:35.068326+0000","last_unstale":"2024-03-06T07:28:36.053814+0000","last_undegraded":"2024-03-06T07:28:36.053814+0000","last_fullsized":"2024-03-06T07:28:36.053814+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:52.190649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"196'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054983+0000","last_change":"2024-03-06T07:28:34.046419+0000","last_active":"2024-03-06T07:28:36.054983+0000","last_peered":"2024-03-06T07:28:36.054983+0000","last_clean":"2024-03-06T07:28:36.054983+0000","last_became_active":"2024-03-06T07:28:34.046066+0000","last_became_peered":"2024-03-06T07:28:34.046066+0000","last_unstale":"2024-03-06T07:28:36.054983+0000","last_undegraded":"2024-03-06T07:28:36.054983+0000","last_fullsized":"2024-03-06T07:28:36.054983+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:58.368005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.122901+0000","last_change":"2024-03-06T07:28:35.067853+0000","last_active":"2024-03-06T07:28:37.122901+0000","last_peered":"2024-03-06T07:28:37.122901+0000","last_clean":"2024-03-06T07:28:37.122901+0000","last_became_active":"2024-03-06T07:28:35.067589+0000","last_became_peered":"2024-03-06T07:28:35.067589+0000","last_unstale":"2024-03-06T07:28:37.122901+0000","last_undegraded":"2024-03-06T07:28:37.122901+0000","last_fullsized":"2024-03-06T07:28:37.122901+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:44.788780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054301+0000","last_change":"2024-03-06T07:28:34.045103+0000","last_active":"2024-03-06T07:28:36.054301+0000","last_peered":"2024-03-06T07:28:36.054301+0000","last_clean":"2024-03-06T07:28:36.054301+0000","last_became_active":"2024-03-06T07:28:34.044786+0000","last_became_peered":"2024-03-06T07:28:34.044786+0000","last_unstale":"2024-03-06T07:28:36.054301+0000","last_undegraded":"2024-03-06T07:28:36.054301+0000","last_fullsized":"2024-03-06T07:28:36.054301+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:00.673585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123878+0000","last_change":"2024-03-06T07:28:35.061092+0000","last_active":"2024-03-06T07:28:37.123878+0000","last_peered":"2024-03-06T07:28:37.123878+0000","last_clean":"2024-03-06T07:28:37.123878+0000","last_became_active":"2024-03-06T07:28:35.060810+0000","last_became_peered":"2024-03-06T07:28:35.060810+0000","last_unstale":"2024-03-06T07:28:37.123878+0000","last_undegraded":"2024-03-06T07:28:37.123878+0000","last_fullsized":"2024-03-06T07:28:37.123878+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:18.011748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.122999+0000","last_change":"2024-03-06T07:28:34.054092+0000","last_active":"2024-03-06T07:28:37.122999+0000","last_peered":"2024-03-06T07:28:37.122999+0000","last_clean":"2024-03-06T07:28:37.122999+0000","last_became_active":"2024-03-06T07:28:34.053838+0000","last_became_peered":"2024-03-06T07:28:34.053838+0000","last_unstale":"2024-03-06T07:28:37.122999+0000","last_undegraded":"2024-03-06T07:28:37.122999+0000","last_fullsized":"2024-03-06T07:28:37.122999+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:49.778788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054105+0000","last_change":"2024-03-06T07:28:35.048210+0000","last_active":"2024-03-06T07:28:36.054105+0000","last_peered":"2024-03-06T07:28:36.054105+0000","last_clean":"2024-03-06T07:28:36.054105+0000","last_became_active":"2024-03-06T07:28:35.047938+0000","last_became_peered":"2024-03-06T07:28:35.047938+0000","last_unstale":"2024-03-06T07:28:36.054105+0000","last_undegraded":"2024-03-06T07:28:36.054105+0000","last_fullsized":"2024-03-06T07:28:36.054105+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:24.627431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054948+0000","last_change":"2024-03-06T07:28:34.046882+0000","last_active":"2024-03-06T07:28:36.054948+0000","last_peered":"2024-03-06T07:28:36.054948+0000","last_clean":"2024-03-06T07:28:36.054948+0000","last_became_active":"2024-03-06T07:28:34.045955+0000","last_became_peered":"2024-03-06T07:28:34.045955+0000","last_unstale":"2024-03-06T07:28:36.054948+0000","last_undegraded":"2024-03-06T07:28:36.054948+0000","last_fullsized":"2024-03-06T07:28:36.054948+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:59.283215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.055417+0000","last_change":"2024-03-06T07:28:35.062206+0000","last_active":"2024-03-06T07:28:36.055417+0000","last_peered":"2024-03-06T07:28:36.055417+0000","last_clean":"2024-03-06T07:28:36.055417+0000","last_became_active":"2024-03-06T07:28:35.061880+0000","last_became_peered":"2024-03-06T07:28:35.061880+0000","last_unstale":"2024-03-06T07:28:36.055417+0000","last_undegraded":"2024-03-06T07:28:36.055417+0000","last_fullsized":"2024-03-06T07:28:36.055417+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:54.416533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054726+0000","last_change":"2024-03-06T07:28:34.047502+0000","last_active":"2024-03-06T07:28:36.054726+0000","last_peered":"2024-03-06T07:28:36.054726+0000","last_clean":"2024-03-06T07:28:36.054726+0000","last_became_active":"2024-03-06T07:28:34.047242+0000","last_became_peered":"2024-03-06T07:28:34.047242+0000","last_unstale":"2024-03-06T07:28:36.054726+0000","last_undegraded":"2024-03-06T07:28:36.054726+0000","last_fullsized":"2024-03-06T07:28:36.054726+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:04.137572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.055536+0000","last_change":"2024-03-06T07:28:35.053602+0000","last_active":"2024-03-06T07:28:36.055536+0000","last_peered":"2024-03-06T07:28:36.055536+0000","last_clean":"2024-03-06T07:28:36.055536+0000","last_became_active":"2024-03-06T07:28:35.051798+0000","last_became_peered":"2024-03-06T07:28:35.051798+0000","last_unstale":"2024-03-06T07:28:36.055536+0000","last_undegraded":"2024-03-06T07:28:36.055536+0000","last_fullsized":"2024-03-06T07:28:36.055536+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:51.942949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.119254+0000","last_change":"2024-03-06T07:28:34.045868+0000","last_active":"2024-03-06T07:28:36.119254+0000","last_peered":"2024-03-06T07:28:36.119254+0000","last_clean":"2024-03-06T07:28:36.119254+0000","last_became_active":"2024-03-06T07:28:34.045604+0000","last_became_peered":"2024-03-06T07:28:34.045604+0000","last_unstale":"2024-03-06T07:28:36.119254+0000","last_undegraded":"2024-03-06T07:28:36.119254+0000","last_fullsized":"2024-03-06T07:28:36.119254+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:59.414097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.060485+0000","last_change":"2024-03-06T07:28:35.061413+0000","last_active":"2024-03-06T07:28:36.060485+0000","last_peered":"2024-03-06T07:28:36.060485+0000","last_clean":"2024-03-06T07:28:36.060485+0000","last_became_active":"2024-03-06T07:28:35.061167+0000","last_became_peered":"2024-03-06T07:28:35.061167+0000","last_unstale":"2024-03-06T07:28:36.060485+0000","last_undegraded":"2024-03-06T07:28:36.060485+0000","last_fullsized":"2024-03-06T07:28:36.060485+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:46.388045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123190+0000","last_change":"2024-03-06T07:28:34.047667+0000","last_active":"2024-03-06T07:28:37.123190+0000","last_peered":"2024-03-06T07:28:37.123190+0000","last_clean":"2024-03-06T07:28:37.123190+0000","last_became_active":"2024-03-06T07:28:34.047280+0000","last_became_peered":"2024-03-06T07:28:34.047280+0000","last_unstale":"2024-03-06T07:28:37.123190+0000","last_undegraded":"2024-03-06T07:28:37.123190+0000","last_fullsized":"2024-03-06T07:28:37.123190+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:49.999763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.055237+0000","last_change":"2024-03-06T07:28:35.053233+0000","last_active":"2024-03-06T07:28:36.055237+0000","last_peered":"2024-03-06T07:28:36.055237+0000","last_clean":"2024-03-06T07:28:36.055237+0000","last_became_active":"2024-03-06T07:28:35.052934+0000","last_became_peered":"2024-03-06T07:28:35.052934+0000","last_unstale":"2024-03-06T07:28:36.055237+0000","last_undegraded":"2024-03-06T07:28:36.055237+0000","last_fullsized":"2024-03-06T07:28:36.055237+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:10.526388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"197'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.058324+0000","last_change":"2024-03-06T07:28:34.049708+0000","last_active":"2024-03-06T07:28:36.058324+0000","last_peered":"2024-03-06T07:28:36.058324+0000","last_clean":"2024-03-06T07:28:36.058324+0000","last_became_active":"2024-03-06T07:28:34.049445+0000","last_became_peered":"2024-03-06T07:28:34.049445+0000","last_unstale":"2024-03-06T07:28:36.058324+0000","last_undegraded":"2024-03-06T07:28:36.058324+0000","last_fullsized":"2024-03-06T07:28:36.058324+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:53.540004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123130+0000","last_change":"2024-03-06T07:28:35.061443+0000","last_active":"2024-03-06T07:28:37.123130+0000","last_peered":"2024-03-06T07:28:37.123130+0000","last_clean":"2024-03-06T07:28:37.123130+0000","last_became_active":"2024-03-06T07:28:35.061136+0000","last_became_peered":"2024-03-06T07:28:35.061136+0000","last_unstale":"2024-03-06T07:28:37.123130+0000","last_undegraded":"2024-03-06T07:28:37.123130+0000","last_fullsized":"2024-03-06T07:28:37.123130+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:04.360858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.119731+0000","last_change":"2024-03-06T07:28:34.038476+0000","last_active":"2024-03-06T07:28:36.119731+0000","last_peered":"2024-03-06T07:28:36.119731+0000","last_clean":"2024-03-06T07:28:36.119731+0000","last_became_active":"2024-03-06T07:28:34.038197+0000","last_became_peered":"2024-03-06T07:28:34.038197+0000","last_unstale":"2024-03-06T07:28:36.119731+0000","last_undegraded":"2024-03-06T07:28:36.119731+0000","last_fullsized":"2024-03-06T07:28:36.119731+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:14:39.790694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.058049+0000","last_change":"2024-03-06T07:28:35.068021+0000","last_active":"2024-03-06T07:28:36.058049+0000","last_peered":"2024-03-06T07:28:36.058049+0000","last_clean":"2024-03-06T07:28:36.058049+0000","last_became_active":"2024-03-06T07:28:35.067745+0000","last_became_peered":"2024-03-06T07:28:35.067745+0000","last_unstale":"2024-03-06T07:28:36.058049+0000","last_undegraded":"2024-03-06T07:28:36.058049+0000","last_fullsized":"2024-03-06T07:28:36.058049+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:43.585947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.119631+0000","last_change":"2024-03-06T07:28:34.046180+0000","last_active":"2024-03-06T07:28:36.119631+0000","last_peered":"2024-03-06T07:28:36.119631+0000","last_clean":"2024-03-06T07:28:36.119631+0000","last_became_active":"2024-03-06T07:28:34.045909+0000","last_became_peered":"2024-03-06T07:28:34.045909+0000","last_unstale":"2024-03-06T07:28:36.119631+0000","last_undegraded":"2024-03-06T07:28:36.119631+0000","last_fullsized":"2024-03-06T07:28:36.119631+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:09.521355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.119512+0000","last_change":"2024-03-06T07:28:35.060902+0000","last_active":"2024-03-06T07:28:36.119512+0000","last_peered":"2024-03-06T07:28:36.119512+0000","last_clean":"2024-03-06T07:28:36.119512+0000","last_became_active":"2024-03-06T07:28:35.060618+0000","last_became_peered":"2024-03-06T07:28:35.060618+0000","last_unstale":"2024-03-06T07:28:36.119512+0000","last_undegraded":"2024-03-06T07:28:36.119512+0000","last_fullsized":"2024-03-06T07:28:36.119512+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:45.435506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.055426+0000","last_change":"2024-03-06T07:28:34.043986+0000","last_active":"2024-03-06T07:28:36.055426+0000","last_peered":"2024-03-06T07:28:36.055426+0000","last_clean":"2024-03-06T07:28:36.055426+0000","last_became_active":"2024-03-06T07:28:34.043693+0000","last_became_peered":"2024-03-06T07:28:34.043693+0000","last_unstale":"2024-03-06T07:28:36.055426+0000","last_undegraded":"2024-03-06T07:28:36.055426+0000","last_fullsized":"2024-03-06T07:28:36.055426+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:06.104970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.123270+0000","last_change":"2024-03-06T07:28:35.048335+0000","last_active":"2024-03-06T07:28:37.123270+0000","last_peered":"2024-03-06T07:28:37.123270+0000","last_clean":"2024-03-06T07:28:37.123270+0000","last_became_active":"2024-03-06T07:28:35.048079+0000","last_became_peered":"2024-03-06T07:28:35.048079+0000","last_unstale":"2024-03-06T07:28:37.123270+0000","last_undegraded":"2024-03-06T07:28:37.123270+0000","last_fullsized":"2024-03-06T07:28:37.123270+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:28.578954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.057963+0000","last_change":"2024-03-06T07:28:34.047903+0000","last_active":"2024-03-06T07:28:36.057963+0000","last_peered":"2024-03-06T07:28:36.057963+0000","last_clean":"2024-03-06T07:28:36.057963+0000","last_became_active":"2024-03-06T07:28:34.047612+0000","last_became_peered":"2024-03-06T07:28:34.047612+0000","last_unstale":"2024-03-06T07:28:36.057963+0000","last_undegraded":"2024-03-06T07:28:36.057963+0000","last_fullsized":"2024-03-06T07:28:36.057963+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:58.368005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054105+0000","last_change":"2024-03-06T07:28:35.048834+0000","last_active":"2024-03-06T07:28:36.054105+0000","last_peered":"2024-03-06T07:28:36.054105+0000","last_clean":"2024-03-06T07:28:36.054105+0000","last_became_active":"2024-03-06T07:28:35.048577+0000","last_became_peered":"2024-03-06T07:28:35.048577+0000","last_unstale":"2024-03-06T07:28:36.054105+0000","last_undegraded":"2024-03-06T07:28:36.054105+0000","last_fullsized":"2024-03-06T07:28:36.054105+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:30.272531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.122883+0000","last_change":"2024-03-06T07:28:34.051791+0000","last_active":"2024-03-06T07:28:37.122883+0000","last_peered":"2024-03-06T07:28:37.122883+0000","last_clean":"2024-03-06T07:28:37.122883+0000","last_became_active":"2024-03-06T07:28:34.051472+0000","last_became_peered":"2024-03-06T07:28:34.051472+0000","last_unstale":"2024-03-06T07:28:37.122883+0000","last_undegraded":"2024-03-06T07:28:37.122883+0000","last_fullsized":"2024-03-06T07:28:37.122883+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:00.673585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054864+0000","last_change":"2024-03-06T07:28:35.054900+0000","last_active":"2024-03-06T07:28:36.054864+0000","last_peered":"2024-03-06T07:28:36.054864+0000","last_clean":"2024-03-06T07:28:36.054864+0000","last_became_active":"2024-03-06T07:28:35.054630+0000","last_became_peered":"2024-03-06T07:28:35.054630+0000","last_unstale":"2024-03-06T07:28:36.054864+0000","last_undegraded":"2024-03-06T07:28:36.054864+0000","last_fullsized":"2024-03-06T07:28:36.054864+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:43.585947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054430+0000","last_change":"2024-03-06T07:28:34.048933+0000","last_active":"2024-03-06T07:28:36.054430+0000","last_peered":"2024-03-06T07:28:36.054430+0000","last_clean":"2024-03-06T07:28:36.054430+0000","last_became_active":"2024-03-06T07:28:34.048670+0000","last_became_peered":"2024-03-06T07:28:34.048670+0000","last_unstale":"2024-03-06T07:28:36.054430+0000","last_undegraded":"2024-03-06T07:28:36.054430+0000","last_fullsized":"2024-03-06T07:28:36.054430+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:11.606416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:37.122971+0000","last_change":"2024-03-06T07:28:35.047542+0000","last_active":"2024-03-06T07:28:37.122971+0000","last_peered":"2024-03-06T07:28:37.122971+0000","last_clean":"2024-03-06T07:28:37.122971+0000","last_became_active":"2024-03-06T07:28:35.047007+0000","last_became_peered":"2024-03-06T07:28:35.047007+0000","last_unstale":"2024-03-06T07:28:37.122971+0000","last_undegraded":"2024-03-06T07:28:37.122971+0000","last_fullsized":"2024-03-06T07:28:37.122971+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:34.021086+0000","last_clean_scrub_stamp":"2024-03-06T07:28:34.021086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:26.284897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"196'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.060739+0000","last_change":"2024-03-06T07:28:34.049313+0000","last_active":"2024-03-06T07:28:36.060739+0000","last_peered":"2024-03-06T07:28:36.060739+0000","last_clean":"2024-03-06T07:28:36.060739+0000","last_became_active":"2024-03-06T07:28:34.048967+0000","last_became_peered":"2024-03-06T07:28:34.048967+0000","last_unstale":"2024-03-06T07:28:36.060739+0000","last_undegraded":"2024-03-06T07:28:36.060739+0000","last_fullsized":"2024-03-06T07:28:36.060739+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:33.016053+0000","last_clean_scrub_stamp":"2024-03-06T07:28:33.016053+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:12.530555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":510,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.054276+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:28:36.054276+0000","last_peered":"2024-03-06T07:28:36.054276+0000","last_clean":"2024-03-06T07:28:36.054276+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:28:36.054276+0000","last_undegraded":"2024-03-06T07:28:36.054276+0000","last_fullsized":"2024-03-06T07:28:36.054276+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574952,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33692,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338148,"statfs":{"total":96636764160,"available":96602263552,"internally_reserved":0,"allocated":2572288,"data_stored":1807302,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574951,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33628,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338212,"statfs":{"total":96636764160,"available":96602329088,"internally_reserved":0,"allocated":2506752,"data_stored":1216922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574951,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33620,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338220,"statfs":{"total":96636764160,"available":96602337280,"internally_reserved":0,"allocated":2498560,"data_stored":1216810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574952,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33624,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338216,"statfs":{"total":96636764160,"available":96602333184,"internally_reserved":0,"allocated":2502656,"data_stored":1216880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574951,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33648,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338192,"statfs":{"total":96636764160,"available":96602308608,"internally_reserved":0,"allocated":2523136,"data_stored":1219060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574951,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37788,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334052,"statfs":{"total":96636764160,"available":96598069248,"internally_reserved":0,"allocated":2506752,"data_stored":1217462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33624,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338216,"statfs":{"total":96636764160,"available":96602333184,"internally_reserved":0,"allocated":2498560,"data_stored":1216810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574952,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37860,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333980,"statfs":{"total":96636764160,"available":96597995520,"internally_reserved":0,"allocated":2576384,"data_stored":1808790,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:42.927 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:28:42.928 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:28:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.072+0000 7fc056781640 1 Processor -- start 2024-03-06T07:28:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.073+0000 7fc056781640 1 -- start start 2024-03-06T07:28:43.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.073+0000 7fc056781640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc050152690 0x7fc050155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.073+0000 7fc056781640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0500c6730 0x7fc0500c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.073+0000 7fc056781640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc0500c5d10 0x7fc050156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.073+0000 7fc056781640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0501561e0 con 0x7fc0500c5d10 2024-03-06T07:28:43.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.073+0000 7fc056781640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0500c3300 con 0x7fc050152690 2024-03-06T07:28:43.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.073+0000 7fc056781640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0500c35a0 con 0x7fc0500c6730 2024-03-06T07:28:43.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.073+0000 7fc054d1e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc0500c5d10 0x7fc050156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.073+0000 7fc054d1e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc0500c5d10 0x7fc050156cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54620/0 (socket says 172.21.15.1:54620) 2024-03-06T07:28:43.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.073+0000 7fc054d1e640 1 -- 172.21.15.1:0/3574577027 learned_addr learned my addr 172.21.15.1:0/3574577027 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:43.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.073+0000 7fc04ffff640 1 --2- 172.21.15.1:0/3574577027 >> v2:172.21.15.98:3300/0 conn(0x7fc050152690 0x7fc050155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.074+0000 7fc04f7fe640 1 --2- 172.21.15.1:0/3574577027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0500c6730 0x7fc0500c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.074+0000 7fc054d1e640 1 -- 172.21.15.1:0/3574577027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0500c6730 msgr2=0x7fc0500c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.074+0000 7fc054d1e640 1 --2- 172.21.15.1:0/3574577027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0500c6730 0x7fc0500c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.074+0000 7fc054d1e640 1 -- 172.21.15.1:0/3574577027 >> v2:172.21.15.98:3300/0 conn(0x7fc050152690 msgr2=0x7fc050155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.074+0000 7fc054d1e640 1 --2- 172.21.15.1:0/3574577027 >> v2:172.21.15.98:3300/0 conn(0x7fc050152690 0x7fc050155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.074+0000 7fc054d1e640 1 -- 172.21.15.1:0/3574577027 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0500c3880 con 0x7fc0500c5d10 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.075+0000 7fc054d1e640 1 --2- 172.21.15.1:0/3574577027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0500c5d10 0x7fc050156cd0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fc0500ab5f0 tx=0x7fc04802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0ae60c42e04c2d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.075+0000 7fc04effd640 1 -- 172.21.15.1:0/3574577027 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04802c9e0 con 0x7fc0500c5d10 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.075+0000 7fc04effd640 1 -- 172.21.15.1:0/3574577027 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04802cb80 con 0x7fc0500c5d10 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.075+0000 7fc04effd640 1 -- 172.21.15.1:0/3574577027 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0480343f0 con 0x7fc0500c5d10 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.076+0000 7fc056781640 1 -- 172.21.15.1:0/3574577027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0500c5d10 msgr2=0x7fc050156cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.076+0000 7fc056781640 1 --2- 172.21.15.1:0/3574577027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0500c5d10 0x7fc050156cd0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fc0500ab5f0 tx=0x7fc04802be40 comp rx=0 tx=0).stop 2024-03-06T07:28:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.076+0000 7fc056781640 1 -- 172.21.15.1:0/3574577027 shutdown_connections 2024-03-06T07:28:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.076+0000 7fc056781640 1 --2- 172.21.15.1:0/3574577027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0500c5d10 0x7fc050156cd0 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.076+0000 7fc056781640 1 --2- 172.21.15.1:0/3574577027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0500c6730 0x7fc0500c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.076+0000 7fc056781640 1 --2- 172.21.15.1:0/3574577027 >> v2:172.21.15.98:3300/0 conn(0x7fc050152690 0x7fc050155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.076+0000 7fc056781640 1 -- 172.21.15.1:0/3574577027 >> 172.21.15.1:0/3574577027 conn(0x7fc05006b410 msgr2=0x7fc05006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.076+0000 7fc056781640 1 -- 172.21.15.1:0/3574577027 shutdown_connections 2024-03-06T07:28:43.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.076+0000 7fc056781640 1 -- 172.21.15.1:0/3574577027 wait complete. 2024-03-06T07:28:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.077+0000 7fc056781640 1 Processor -- start 2024-03-06T07:28:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.077+0000 7fc056781640 1 -- start start 2024-03-06T07:28:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc056781640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0500c5d10 0x7fc05014fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc056781640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0500c6730 0x7fc0501532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc056781640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc050152690 0x7fc0501ea810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc056781640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc050152a70 con 0x7fc0500c5d10 2024-03-06T07:28:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc056781640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0501567b0 con 0x7fc050152690 2024-03-06T07:28:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc056781640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc050152bf0 con 0x7fc0500c6730 2024-03-06T07:28:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc04ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0500c5d10 0x7fc05014fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc04ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0500c5d10 0x7fc05014fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54622/0 (socket says 172.21.15.1:54622) 2024-03-06T07:28:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc04ffff640 1 -- 172.21.15.1:0/4097531371 learned_addr learned my addr 172.21.15.1:0/4097531371 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc04f7fe640 1 --2- 172.21.15.1:0/4097531371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0500c6730 0x7fc0501532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc054d1e640 1 --2- 172.21.15.1:0/4097531371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc050152690 0x7fc0501ea810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc04ffff640 1 -- 172.21.15.1:0/4097531371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0500c6730 msgr2=0x7fc0501532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc04ffff640 1 --2- 172.21.15.1:0/4097531371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0500c6730 0x7fc0501532a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc04ffff640 1 -- 172.21.15.1:0/4097531371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc050152690 msgr2=0x7fc0501ea810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc04ffff640 1 --2- 172.21.15.1:0/4097531371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc050152690 0x7fc0501ea810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc04ffff640 1 -- 172.21.15.1:0/4097531371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05014b620 con 0x7fc0500c5d10 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc04ffff640 1 --2- 172.21.15.1:0/4097531371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0500c5d10 0x7fc05014fb40 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fc04000e870 tx=0x7fc04000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc04cff9640 1 -- 172.21.15.1:0/4097531371 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc040017070 con 0x7fc0500c5d10 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc04cff9640 1 -- 172.21.15.1:0/4097531371 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04000c940 con 0x7fc0500c5d10 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc056781640 1 -- 172.21.15.1:0/4097531371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0501ede20 con 0x7fc0500c5d10 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.078+0000 7fc04cff9640 1 -- 172.21.15.1:0/4097531371 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04000cd30 con 0x7fc0500c5d10 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.079+0000 7fc056781640 1 -- 172.21.15.1:0/4097531371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0501ee0c0 con 0x7fc0500c5d10 2024-03-06T07:28:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.080+0000 7fc056781640 1 -- 172.21.15.1:0/4097531371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0501500e0 con 0x7fc0500c5d10 2024-03-06T07:28:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.080+0000 7fc04cff9640 1 -- 172.21.15.1:0/4097531371 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc04001a070 con 0x7fc0500c5d10 2024-03-06T07:28:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.080+0000 7fc04cff9640 1 --2- 172.21.15.1:0/4097531371 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc03c06b4d0 0x7fc03c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.081+0000 7fc04cff9640 1 -- 172.21.15.1:0/4097531371 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc04008a2f0 con 0x7fc0500c5d10 2024-03-06T07:28:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.081+0000 7fc04f7fe640 1 --2- 172.21.15.1:0/4097531371 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc03c06b4d0 0x7fc03c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.081+0000 7fc04f7fe640 1 --2- 172.21.15.1:0/4097531371 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc03c06b4d0 0x7fc03c06d990 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fc050150370 tx=0x7fc044008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.089+0000 7fc04cff9640 1 -- 172.21.15.1:0/4097531371 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc040058f40 con 0x7fc0500c5d10 2024-03-06T07:28:43.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.441+0000 7fc056781640 1 -- 172.21.15.1:0/4097531371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc0500c2c40 con 0x7fc0500c5d10 2024-03-06T07:28:43.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.443+0000 7fc04cff9640 1 -- 172.21.15.1:0/4097531371 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v24) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc040058840 con 0x7fc0500c5d10 2024-03-06T07:28:43.444 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-06T07:28:43.445 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng== 2024-03-06T07:28:43.445 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-06T07:28:43.445 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-06T07:28:43.445 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:28:43.445 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-06T07:28:43.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.445+0000 7fc056781640 1 -- 172.21.15.1:0/4097531371 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc03c06b4d0 msgr2=0x7fc03c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.445+0000 7fc056781640 1 --2- 172.21.15.1:0/4097531371 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc03c06b4d0 0x7fc03c06d990 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fc050150370 tx=0x7fc044008040 comp rx=0 tx=0).stop 2024-03-06T07:28:43.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.445+0000 7fc056781640 1 -- 172.21.15.1:0/4097531371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0500c5d10 msgr2=0x7fc05014fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.445+0000 7fc056781640 1 --2- 172.21.15.1:0/4097531371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0500c5d10 0x7fc05014fb40 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fc04000e870 tx=0x7fc04000ed30 comp rx=0 tx=0).stop 2024-03-06T07:28:43.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.445+0000 7fc056781640 1 -- 172.21.15.1:0/4097531371 shutdown_connections 2024-03-06T07:28:43.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.445+0000 7fc056781640 1 --2- 172.21.15.1:0/4097531371 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc03c06b4d0 0x7fc03c06d990 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.445+0000 7fc056781640 1 --2- 172.21.15.1:0/4097531371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc050152690 0x7fc0501ea810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.445+0000 7fc056781640 1 --2- 172.21.15.1:0/4097531371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc0500c6730 0x7fc0501532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.445+0000 7fc056781640 1 --2- 172.21.15.1:0/4097531371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0500c5d10 0x7fc05014fb40 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.445+0000 7fc056781640 1 -- 172.21.15.1:0/4097531371 >> 172.21.15.1:0/4097531371 conn(0x7fc05006b410 msgr2=0x7fc05006b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.445+0000 7fc056781640 1 -- 172.21.15.1:0/4097531371 shutdown_connections 2024-03-06T07:28:43.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.445+0000 7fc056781640 1 -- 172.21.15.1:0/4097531371 wait complete. 2024-03-06T07:28:43.458 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.557+0000 7f97b317a640 1 Processor -- start 2024-03-06T07:28:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.557+0000 7f97b317a640 1 -- start start 2024-03-06T07:28:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.558+0000 7f97b317a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97ac152650 0x7f97ac155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.558+0000 7f97b317a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ac0c2fc0 con 0x7f97ac0c6b10 2024-03-06T07:28:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.558+0000 7f97b317a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ac158da0 con 0x7f97ac0bf9d0 2024-03-06T07:28:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.558+0000 7f97b317a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac155800 con 0x7f97ac152650 2024-03-06T07:28:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.558+0000 7f97abfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f97ac0c6b10 0x7f97ac158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45256/0 (socket says 172.21.15.1:45256) 2024-03-06T07:28:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.558+0000 7f97abfff640 1 -- 172.21.15.1:0/3518655774 learned_addr learned my addr 172.21.15.1:0/3518655774 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.558+0000 7f97b0f16640 1 --2- 172.21.15.1:0/3518655774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97ac152650 0x7f97ac155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.558+0000 7f97ab7fe640 1 -- 172.21.15.1:0/3518655774 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514946946 0 0) 0x7f97ac0c2fc0 con 0x7f97ac0c6b10 2024-03-06T07:28:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.558+0000 7f97ab7fe640 1 -- 172.21.15.1:0/3518655774 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f979c003520 con 0x7f97ac0c6b10 2024-03-06T07:28:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.559+0000 7f97ab7fe640 1 -- 172.21.15.1:0/3518655774 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061717804 0 0) 0x7f97ac158da0 con 0x7f97ac0bf9d0 2024-03-06T07:28:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.559+0000 7f97ab7fe640 1 -- 172.21.15.1:0/3518655774 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97ac0c2fc0 con 0x7f97ac0bf9d0 2024-03-06T07:28:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.559+0000 7f97b0f16640 1 -- 172.21.15.1:0/3518655774 >> v1:172.21.15.98:6789/0 conn(0x7f97ac0bf9d0 legacy=0x7f97ac0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.559+0000 7f97b0f16640 1 -- 172.21.15.1:0/3518655774 >> v1:172.21.15.1:6789/0 conn(0x7f97ac0c6b10 legacy=0x7f97ac158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.559+0000 7f97ab7fe640 1 -- 172.21.15.1:0/3518655774 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3710422984 0 0) 0x7f979c003520 con 0x7f97ac0c6b10 2024-03-06T07:28:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.559+0000 7f97b0f16640 1 -- 172.21.15.1:0/3518655774 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ac0c6690 con 0x7f97ac152650 2024-03-06T07:28:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.559+0000 7f97b0f16640 1 --2- 172.21.15.1:0/3518655774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97ac152650 0x7f97ac155c90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f97a00097d0 tx=0x7f97a002b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a44501facbff642 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.560+0000 7f97ab7fe640 1 -- 172.21.15.1:0/3518655774 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0037070 con 0x7f97ac152650 2024-03-06T07:28:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.560+0000 7f97ab7fe640 1 -- 172.21.15.1:0/3518655774 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a0030510 con 0x7f97ac152650 2024-03-06T07:28:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.560+0000 7f97ab7fe640 1 -- 172.21.15.1:0/3518655774 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9798001670 con 0x7f97ac152650 2024-03-06T07:28:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.560+0000 7f97b317a640 1 -- 172.21.15.1:0/3518655774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97ac152650 msgr2=0x7f97ac155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.560+0000 7f97b317a640 1 --2- 172.21.15.1:0/3518655774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97ac152650 0x7f97ac155c90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f97a00097d0 tx=0x7f97a002b770 comp rx=0 tx=0).stop 2024-03-06T07:28:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.560+0000 7f97b317a640 1 -- 172.21.15.1:0/3518655774 shutdown_connections 2024-03-06T07:28:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.560+0000 7f97b317a640 1 --2- 172.21.15.1:0/3518655774 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97ac152650 0x7f97ac155c90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.560+0000 7f97b317a640 1 -- 172.21.15.1:0/3518655774 >> 172.21.15.1:0/3518655774 conn(0x7f97ac066ee0 msgr2=0x7f97ac0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.560+0000 7f97b317a640 1 -- 172.21.15.1:0/3518655774 shutdown_connections 2024-03-06T07:28:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.560+0000 7f97b317a640 1 -- 172.21.15.1:0/3518655774 wait complete. 2024-03-06T07:28:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.562+0000 7f97b317a640 1 Processor -- start 2024-03-06T07:28:43.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.562+0000 7f97b317a640 1 -- start start 2024-03-06T07:28:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.563+0000 7f97b317a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97ac0bf9d0 0x7f97ac156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.563+0000 7f97b317a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ac152650 0x7f97ac1eb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.563+0000 7f97b317a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97ac14fb10 0x7f97ac1eb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.563+0000 7f97b317a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac1551b0 con 0x7f97ac152650 2024-03-06T07:28:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.563+0000 7f97b317a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac0c6fa0 con 0x7f97ac14fb10 2024-03-06T07:28:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.563+0000 7f97b317a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac155490 con 0x7f97ac0bf9d0 2024-03-06T07:28:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.563+0000 7f97abfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ac152650 0x7f97ac1eb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.563+0000 7f97abfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ac152650 0x7f97ac1eb480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54636/0 (socket says 172.21.15.1:54636) 2024-03-06T07:28:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.563+0000 7f97abfff640 1 -- 172.21.15.1:0/586374903 learned_addr learned my addr 172.21.15.1:0/586374903 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.564+0000 7f97b0f16640 1 --2- 172.21.15.1:0/586374903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97ac0bf9d0 0x7f97ac156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.564+0000 7f97b1717640 1 --2- 172.21.15.1:0/586374903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97ac14fb10 0x7f97ac1eb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.564+0000 7f97abfff640 1 -- 172.21.15.1:0/586374903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97ac0bf9d0 msgr2=0x7f97ac156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.564+0000 7f97abfff640 1 --2- 172.21.15.1:0/586374903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97ac0bf9d0 0x7f97ac156d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.564+0000 7f97abfff640 1 -- 172.21.15.1:0/586374903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97ac14fb10 msgr2=0x7f97ac1eb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.564+0000 7f97abfff640 1 --2- 172.21.15.1:0/586374903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97ac14fb10 0x7f97ac1eb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.564+0000 7f97abfff640 1 -- 172.21.15.1:0/586374903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ac157a90 con 0x7f97ac152650 2024-03-06T07:28:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.564+0000 7f97abfff640 1 --2- 172.21.15.1:0/586374903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ac152650 0x7f97ac1eb480 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f979800c550 tx=0x7f979800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.564+0000 7f97a97fa640 1 -- 172.21.15.1:0/586374903 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9798017070 con 0x7f97ac152650 2024-03-06T07:28:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.564+0000 7f97a97fa640 1 -- 172.21.15.1:0/586374903 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9798005d00 con 0x7f97ac152650 2024-03-06T07:28:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.564+0000 7f97b317a640 1 -- 172.21.15.1:0/586374903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ac153470 con 0x7f97ac152650 2024-03-06T07:28:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.564+0000 7f97a97fa640 1 -- 172.21.15.1:0/586374903 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97980072b0 con 0x7f97ac152650 2024-03-06T07:28:43.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.564+0000 7f97b317a640 1 -- 172.21.15.1:0/586374903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ac157260 con 0x7f97ac152650 2024-03-06T07:28:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.565+0000 7f97b317a640 1 -- 172.21.15.1:0/586374903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ac1eadd0 con 0x7f97ac152650 2024-03-06T07:28:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.566+0000 7f97a97fa640 1 -- 172.21.15.1:0/586374903 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f979801a070 con 0x7f97ac152650 2024-03-06T07:28:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.566+0000 7f97a97fa640 1 --2- 172.21.15.1:0/586374903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f979006b4d0 0x7f979006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.567+0000 7f97a97fa640 1 -- 172.21.15.1:0/586374903 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9798089780 con 0x7f97ac152650 2024-03-06T07:28:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.567+0000 7f97b0f16640 1 --2- 172.21.15.1:0/586374903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f979006b4d0 0x7f979006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.567+0000 7f97b0f16640 1 --2- 172.21.15.1:0/586374903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f979006b4d0 0x7f979006d990 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f97a0034000 tx=0x7f97a0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.571+0000 7f97a97fa640 1 -- 172.21.15.1:0/586374903 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9798057e10 con 0x7f97ac152650 2024-03-06T07:28:43.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.793+0000 7f97b317a640 1 -- 172.21.15.1:0/586374903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97ac067fd0 con 0x7f97ac152650 2024-03-06T07:28:43.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.793+0000 7f97a97fa640 1 -- 172.21.15.1:0/586374903 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+4983 (secure 0 0 0) 0x7f979805beb0 con 0x7f97ac152650 2024-03-06T07:28:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:43.796 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":185,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7961,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2819896202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2819896202},{"type":"v1","addr":"172.21.15.98:6837","nonce":2819896202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":8959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/646952744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":646952744},{"type":"v1","addr":"172.21.15.1:6835","nonce":646952744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":8965,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2843229007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2843229007},{"type":"v1","addr":"172.21.15.98:6836","nonce":2843229007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:34.395524+0000","modified":"2024-03-06T07:28:36.397164+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10425},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10425":{"gid":10425,"name":"a","rank":0,"incarnation":183,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6834/1315978401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1315978401},{"type":"v1","addr":"172.21.15.1:6834","nonce":1315978401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10425,"qdb_cluster":[10425]},"id":15}]} 2024-03-06T07:28:43.796 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 185 2024-03-06T07:28:43.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.795+0000 7f97b317a640 1 -- 172.21.15.1:0/586374903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f979006b4d0 msgr2=0x7f979006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.795+0000 7f97b317a640 1 --2- 172.21.15.1:0/586374903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f979006b4d0 0x7f979006d990 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f97a0034000 tx=0x7f97a0008030 comp rx=0 tx=0).stop 2024-03-06T07:28:43.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.795+0000 7f97b317a640 1 -- 172.21.15.1:0/586374903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ac152650 msgr2=0x7f97ac1eb480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.795+0000 7f97b317a640 1 --2- 172.21.15.1:0/586374903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ac152650 0x7f97ac1eb480 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f979800c550 tx=0x7f979800ca10 comp rx=0 tx=0).stop 2024-03-06T07:28:43.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.795+0000 7f97b317a640 1 -- 172.21.15.1:0/586374903 shutdown_connections 2024-03-06T07:28:43.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.795+0000 7f97b317a640 1 --2- 172.21.15.1:0/586374903 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f979006b4d0 0x7f979006d990 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.795+0000 7f97b317a640 1 --2- 172.21.15.1:0/586374903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97ac14fb10 0x7f97ac1eb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.795+0000 7f97b317a640 1 --2- 172.21.15.1:0/586374903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ac152650 0x7f97ac1eb480 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.795+0000 7f97b317a640 1 --2- 172.21.15.1:0/586374903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97ac0bf9d0 0x7f97ac156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.795+0000 7f97b317a640 1 -- 172.21.15.1:0/586374903 >> 172.21.15.1:0/586374903 conn(0x7f97ac066ee0 msgr2=0x7f97ac0c5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.795+0000 7f97b317a640 1 -- 172.21.15.1:0/586374903 shutdown_connections 2024-03-06T07:28:43.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:43.795+0000 7f97b317a640 1 -- 172.21.15.1:0/586374903 wait complete. 2024-03-06T07:28:44.214 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 185, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:34.395524+0000', 'modified': '2024-03-06T07:28:36.397164+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10425}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10425': {'gid': 10425, 'name': 'a', 'rank': 0, 'incarnation': 183, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6834/1315978401', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1315978401}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1315978401}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [32], 'metadata_pool': 31, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10425, 'qdb_cluster': [10425]} 2024-03-06T07:28:44.215 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:28:44.215 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:28:44.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.321+0000 7fa010ced640 1 Processor -- start 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.321+0000 7fa010ced640 1 -- start start 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.322+0000 7fa010ced640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c0bf9d0 0x7fa00c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.322+0000 7fa010ced640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa00c0c2fc0 con 0x7fa00c0c6b10 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.322+0000 7fa010ced640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa00c158da0 con 0x7fa00c152650 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.322+0000 7fa010ced640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c155800 con 0x7fa00c0bf9d0 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.322+0000 7fa009d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa00c0c6b10 0x7fa00c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45258/0 (socket says 172.21.15.1:45258) 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.322+0000 7fa009d9b640 1 -- 172.21.15.1:0/2969325895 learned_addr learned my addr 172.21.15.1:0/2969325895 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.322+0000 7fa00ad9d640 1 --2- 172.21.15.1:0/2969325895 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c0bf9d0 0x7fa00c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.322+0000 7fa00959a640 1 -- 172.21.15.1:0/2969325895 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680072338 0 0) 0x7fa00c0c2fc0 con 0x7fa00c0c6b10 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.322+0000 7fa00959a640 1 -- 172.21.15.1:0/2969325895 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ff4003520 con 0x7fa00c0c6b10 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00959a640 1 -- 172.21.15.1:0/2969325895 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1645127641 0 0) 0x7f9ff4003520 con 0x7fa00c0c6b10 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00959a640 1 -- 172.21.15.1:0/2969325895 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c0bf9d0 msgr2=0x7fa00c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00959a640 1 --2- 172.21.15.1:0/2969325895 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c0bf9d0 0x7fa00c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00959a640 1 -- 172.21.15.1:0/2969325895 >> v1:172.21.15.98:6789/0 conn(0x7fa00c152650 legacy=0x7fa00c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00959a640 1 -- 172.21.15.1:0/2969325895 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa00c0c6690 con 0x7fa00c0c6b10 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00959a640 1 -- 172.21.15.1:0/2969325895 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fa000001690 con 0x7fa00c0c6b10 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00959a640 1 -- 172.21.15.1:0/2969325895 >> v1:172.21.15.1:6789/0 conn(0x7fa00c0c6b10 legacy=0x7fa00c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00959a640 1 --2- 172.21.15.1:0/2969325895 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c0bf9d0 0x7f9ff40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00959a640 1 --2- 172.21.15.1:0/2969325895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ff4003f60 0x7f9ff4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00959a640 1 --2- 172.21.15.1:0/2969325895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ff4004d40 0x7f9ff40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00959a640 1 -- 172.21.15.1:0/2969325895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c155800 con 0x7f9ff4004d40 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00959a640 1 -- 172.21.15.1:0/2969325895 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff400bb50 con 0x7f9ff4003f60 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00959a640 1 -- 172.21.15.1:0/2969325895 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff400bd50 con 0x7fa00c0bf9d0 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00ad9d640 1 --2- 172.21.15.1:0/2969325895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ff4004d40 0x7f9ff40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00a59c640 1 --2- 172.21.15.1:0/2969325895 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c0bf9d0 0x7f9ff40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa009d9b640 1 --2- 172.21.15.1:0/2969325895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ff4003f60 0x7f9ff4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00ad9d640 1 -- 172.21.15.1:0/2969325895 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c0bf9d0 msgr2=0x7f9ff40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00ad9d640 1 --2- 172.21.15.1:0/2969325895 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c0bf9d0 0x7f9ff40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00ad9d640 1 -- 172.21.15.1:0/2969325895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ff4003f60 msgr2=0x7f9ff4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00ad9d640 1 --2- 172.21.15.1:0/2969325895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ff4003f60 0x7f9ff4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.323+0000 7fa00ad9d640 1 -- 172.21.15.1:0/2969325895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff4003520 con 0x7f9ff4004d40 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.324+0000 7fa00ad9d640 1 --2- 172.21.15.1:0/2969325895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ff4004d40 0x7f9ff40051c0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7fa004009fc0 tx=0x7f9ff4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=908544b0f8aecdf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.324+0000 7fa00959a640 1 -- 172.21.15.1:0/2969325895 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004039070 con 0x7f9ff4004d40 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.324+0000 7fa00959a640 1 -- 172.21.15.1:0/2969325895 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00402cc20 con 0x7f9ff4004d40 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.324+0000 7fa00959a640 1 -- 172.21.15.1:0/2969325895 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004009c10 con 0x7f9ff4004d40 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.325+0000 7fa010ced640 1 -- 172.21.15.1:0/2969325895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ff4004d40 msgr2=0x7f9ff40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.325+0000 7fa010ced640 1 --2- 172.21.15.1:0/2969325895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ff4004d40 0x7f9ff40051c0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7fa004009fc0 tx=0x7f9ff4005830 comp rx=0 tx=0).stop 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.325+0000 7fa00ad9d640 1 -- 172.21.15.1:0/2969325895 reap_dead start 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.325+0000 7fa010ced640 1 -- 172.21.15.1:0/2969325895 shutdown_connections 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.325+0000 7fa010ced640 1 -- 172.21.15.1:0/2969325895 >> 172.21.15.1:0/2969325895 conn(0x7fa00c066ee0 msgr2=0x7fa00c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.325+0000 7fa010ced640 1 -- 172.21.15.1:0/2969325895 shutdown_connections 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.325+0000 7fa010ced640 1 -- 172.21.15.1:0/2969325895 wait complete. 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.327+0000 7fa010ced640 1 Processor -- start 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.327+0000 7fa010ced640 1 -- start start 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.328+0000 7fa010ced640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ff4004d40 0x7fa00c0c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.328+0000 7fa010ced640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c14b700 0x7fa00c150a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.328+0000 7fa010ced640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa00c150fc0 0x7fa00c1ee230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.328+0000 7fa010ced640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c1551b0 con 0x7f9ff4004d40 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.328+0000 7fa010ced640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c0c6fa0 con 0x7fa00c150fc0 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.328+0000 7fa010ced640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c155490 con 0x7fa00c14b700 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.328+0000 7fa00a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ff4004d40 0x7fa00c0c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.328+0000 7fa00a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ff4004d40 0x7fa00c0c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54656/0 (socket says 172.21.15.1:54656) 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.328+0000 7fa00a59c640 1 -- 172.21.15.1:0/2704649512 learned_addr learned my addr 172.21.15.1:0/2704649512 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.328+0000 7fa009d9b640 1 --2- 172.21.15.1:0/2704649512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c14b700 0x7fa00c150a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.328+0000 7fa00a59c640 1 -- 172.21.15.1:0/2704649512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c14b700 msgr2=0x7fa00c150a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.328+0000 7fa00a59c640 1 --2- 172.21.15.1:0/2704649512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c14b700 0x7fa00c150a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.328+0000 7fa00a59c640 1 -- 172.21.15.1:0/2704649512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa00c150fc0 msgr2=0x7fa00c1ee230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.328+0000 7fa00a59c640 1 --2- 172.21.15.1:0/2704649512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa00c150fc0 0x7fa00c1ee230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.328+0000 7fa00a59c640 1 -- 172.21.15.1:0/2704649512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa00c1eb990 con 0x7f9ff4004d40 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.328+0000 7fa009d9b640 1 --2- 172.21.15.1:0/2704649512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c14b700 0x7fa00c150a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.329+0000 7fa00a59c640 1 --2- 172.21.15.1:0/2704649512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ff4004d40 0x7fa00c0c9a90 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc00f820 tx=0x7f9ffc00fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.329+0000 7f9ffaffd640 1 -- 172.21.15.1:0/2704649512 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc0050d0 con 0x7f9ff4004d40 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.329+0000 7f9ffaffd640 1 -- 172.21.15.1:0/2704649512 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ffc018070 con 0x7f9ff4004d40 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.329+0000 7fa010ced640 1 -- 172.21.15.1:0/2704649512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa00c14ffa0 con 0x7f9ff4004d40 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.329+0000 7f9ffaffd640 1 -- 172.21.15.1:0/2704649512 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc005470 con 0x7f9ff4004d40 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.329+0000 7fa010ced640 1 -- 172.21.15.1:0/2704649512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa00c150560 con 0x7f9ff4004d40 2024-03-06T07:28:44.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.331+0000 7f9ffaffd640 1 -- 172.21.15.1:0/2704649512 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ffc00d6b0 con 0x7f9ff4004d40 2024-03-06T07:28:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.331+0000 7f9ffaffd640 1 --2- 172.21.15.1:0/2704649512 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9fec06b500 0x7f9fec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.331+0000 7f9ffaffd640 1 -- 172.21.15.1:0/2704649512 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9ffc08a0e0 con 0x7f9ff4004d40 2024-03-06T07:28:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.331+0000 7fa009d9b640 1 --2- 172.21.15.1:0/2704649512 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9fec06b500 0x7f9fec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.332+0000 7fa0139b1740 1 Processor -- start 2024-03-06T07:28:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.332+0000 7fa009d9b640 1 --2- 172.21.15.1:0/2704649512 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9fec06b500 0x7f9fec06d9c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0c4bc0 tx=0x7fa000009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.332+0000 7fa0139b1740 1 -- start start 2024-03-06T07:28:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.332+0000 7fa0139b1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 0x5556f3972a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.332+0000 7fa0139b1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f3972f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.332+0000 7fa0139b1740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f39734b0 0x5556f396f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.332+0000 7fa0139b1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556f38d96b0 con 0x5556f3975330 2024-03-06T07:28:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.332+0000 7fa0139b1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556f3882a60 con 0x5556f3974f50 2024-03-06T07:28:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.332+0000 7fa0139b1740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5556f38d5750 con 0x5556f39734b0 2024-03-06T07:28:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.333+0000 7fa009d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f3972f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.333+0000 7fa009d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f3972f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54666/0 (socket says 172.21.15.1:54666) 2024-03-06T07:28:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.333+0000 7fa009d9b640 1 -- 172.21.15.1:0/147723053 learned_addr learned my addr 172.21.15.1:0/147723053 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.333+0000 7fa00ad9d640 1 --2- 172.21.15.1:0/147723053 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f39734b0 0x5556f396f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.333+0000 7fa00a59c640 1 --2- 172.21.15.1:0/147723053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 0x5556f3972a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.333+0000 7fa009d9b640 1 -- 172.21.15.1:0/147723053 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f39734b0 msgr2=0x5556f396f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.333+0000 7fa009d9b640 1 --2- 172.21.15.1:0/147723053 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f39734b0 0x5556f396f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.333+0000 7fa009d9b640 1 -- 172.21.15.1:0/147723053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 msgr2=0x5556f3972a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.333+0000 7fa009d9b640 1 --2- 172.21.15.1:0/147723053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 0x5556f3972a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.333+0000 7fa009d9b640 1 -- 172.21.15.1:0/147723053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556f38a0f30 con 0x5556f3975330 2024-03-06T07:28:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.333+0000 7fa00ad9d640 1 --2- 172.21.15.1:0/147723053 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f39734b0 0x5556f396f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.333+0000 7fa009d9b640 1 --2- 172.21.15.1:0/147723053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f3972f70 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fa0000015d0 tx=0x7fa000013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ecfdb81e5a9ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.333+0000 7f9febfff640 1 -- 172.21.15.1:0/147723053 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa000026070 con 0x5556f3975330 2024-03-06T07:28:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.333+0000 7f9febfff640 1 -- 172.21.15.1:0/147723053 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00001c050 con 0x5556f3975330 2024-03-06T07:28:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.333+0000 7f9febfff640 1 -- 172.21.15.1:0/147723053 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa000021410 con 0x5556f3975330 2024-03-06T07:28:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.334+0000 7fa0139b1740 1 -- 172.21.15.1:0/147723053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 msgr2=0x5556f3972f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.334+0000 7fa0139b1740 1 --2- 172.21.15.1:0/147723053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f3972f70 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fa0000015d0 tx=0x7fa000013000 comp rx=0 tx=0).stop 2024-03-06T07:28:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.334+0000 7fa0139b1740 1 -- 172.21.15.1:0/147723053 shutdown_connections 2024-03-06T07:28:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.334+0000 7fa0139b1740 1 --2- 172.21.15.1:0/147723053 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f39734b0 0x5556f396f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.334+0000 7fa0139b1740 1 --2- 172.21.15.1:0/147723053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f3972f70 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.334+0000 7fa0139b1740 1 --2- 172.21.15.1:0/147723053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 0x5556f3972a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.334+0000 7fa0139b1740 1 -- 172.21.15.1:0/147723053 >> 172.21.15.1:0/147723053 conn(0x5556f39717d0 msgr2=0x5556f39963c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.334+0000 7fa0139b1740 1 -- 172.21.15.1:0/147723053 shutdown_connections 2024-03-06T07:28:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.334+0000 7fa0139b1740 1 -- 172.21.15.1:0/147723053 wait complete. 2024-03-06T07:28:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.335+0000 7fa0139b1740 1 -- start start 2024-03-06T07:28:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.335+0000 7fa0139b1740 1 Processor -- start 2024-03-06T07:28:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.335+0000 7fa0139b1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 0x5556f3997600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa0139b1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f399a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa0139b1740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f39c25a0 0x5556f39ad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa0139b1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556f3997230 con 0x5556f3975330 2024-03-06T07:28:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa0139b1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556f3895ea0 con 0x5556f3974f50 2024-03-06T07:28:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa0139b1740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5556f3996c70 con 0x5556f39c25a0 2024-03-06T07:28:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa0139b1740 10 client.0 _ll_register_callbacks cb 0x5556f3895870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa009d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f399a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa009d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f399a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54668/0 (socket says 172.21.15.1:54668) 2024-03-06T07:28:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa009d9b640 1 -- 172.21.15.1:0/1855363675 learned_addr learned my addr 172.21.15.1:0/1855363675 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa00a59c640 1 --2- 172.21.15.1:0/1855363675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 0x5556f3997600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa00ad9d640 1 --2- 172.21.15.1:0/1855363675 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f39c25a0 0x5556f39ad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa009d9b640 1 -- 172.21.15.1:0/1855363675 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f39c25a0 msgr2=0x5556f39ad9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa009d9b640 1 --2- 172.21.15.1:0/1855363675 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f39c25a0 0x5556f39ad9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa009d9b640 1 -- 172.21.15.1:0/1855363675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 msgr2=0x5556f3997600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa009d9b640 1 --2- 172.21.15.1:0/1855363675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 0x5556f3997600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa009d9b640 1 -- 172.21.15.1:0/1855363675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556f3896270 con 0x5556f3975330 2024-03-06T07:28:44.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa009d9b640 1 --2- 172.21.15.1:0/1855363675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f399a050 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fa000021000 tx=0x7fa00c0bfe40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c5fbd8691b4b69 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7f9fe97fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:28:44.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7f9fe97fa640 1 -- 172.21.15.1:0/1855363675 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa000026040 con 0x5556f3975330 2024-03-06T07:28:44.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7fa0139b1740 1 -- 172.21.15.1:0/1855363675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa00001c050 con 0x5556f3975330 2024-03-06T07:28:44.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.336+0000 7f9fe97fa640 1 -- 172.21.15.1:0/1855363675 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa0000226e0 con 0x5556f3975330 2024-03-06T07:28:44.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.337+0000 7f9fe97fa640 1 -- 172.21.15.1:0/1855363675 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=185) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa000038050 con 0x5556f3975330 2024-03-06T07:28:44.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.337+0000 7fa0139b1740 10 client.10596 fetch_fsmap learned FSMap version 185 2024-03-06T07:28:44.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.337+0000 7fa0139b1740 1 -- 172.21.15.1:0/1855363675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=185,monmap=2+}) v3 -- 0x5556f35ee2d0 con 0x5556f3975330 2024-03-06T07:28:44.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.337+0000 7f9fe97fa640 1 -- 172.21.15.1:0/1855363675 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00001c050 con 0x5556f3975330 2024-03-06T07:28:44.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.337+0000 7f9fe97fa640 1 -- 172.21.15.1:0/1855363675 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 185) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa00003b020 con 0x5556f3975330 2024-03-06T07:28:44.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.337+0000 7fa0139b1740 10 client.10596 fetch_fsmap finished waiting for FSMap version 185 2024-03-06T07:28:44.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.337+0000 7fa0139b1740 10 client.10596 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:28:44.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.337+0000 7fa0139b1740 20 client.10596 populate_metadata read hostname 'smithi001' 2024-03-06T07:28:44.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.337+0000 7fa0139b1740 1 --2- 172.21.15.1:0/1855363675 >> [v2:172.21.15.1:6832/1315978401,v1:172.21.15.1:6834/1315978401] conn(0x5556f39c5640 0x5556f39d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.337+0000 7fa0139b1740 4 client.10596 mds_command: new command op to 10425 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:28:44.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.337+0000 7fa0139b1740 1 -- 172.21.15.1:0/1855363675 --> [v2:172.21.15.1:6832/1315978401,v1:172.21.15.1:6834/1315978401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5556f38e6bf0 con 0x5556f39c5640 2024-03-06T07:28:44.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.337+0000 7fa00a59c640 1 --2- 172.21.15.1:0/1855363675 >> [v2:172.21.15.1:6832/1315978401,v1:172.21.15.1:6834/1315978401] conn(0x5556f39c5640 0x5556f39d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.337+0000 7fa00a59c640 1 --2- 172.21.15.1:0/1855363675 >> [v2:172.21.15.1:6832/1315978401,v1:172.21.15.1:6834/1315978401] conn(0x5556f39c5640 0x5556f39d0540 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d72f0126413075d8 server_cookie=f20e5fb35323cb9 in_seq=0 out_seq=0 2024-03-06T07:28:44.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.337+0000 7f9fe97fa640 10 client.10596 ms_handle_connect on v2:172.21.15.1:6832/1315978401 2024-03-06T07:28:44.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.338+0000 7f9fe97fa640 1 -- 172.21.15.1:0/1855363675 <== mds.0 v2:172.21.15.1:6832/1315978401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5556f38e6bf0 con 0x5556f39c5640 2024-03-06T07:28:44.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.338+0000 7f9fe97fa640 10 client.10596 handle_command_reply: tid=0 2024-03-06T07:28:44.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.338+0000 7fa0139b1740 1 client.10596 shutdown 2024-03-06T07:28:44.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.338+0000 7fa0139b1740 1 -- 172.21.15.1:0/1855363675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 msgr2=0x5556f399a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.338+0000 7fa0139b1740 1 --2- 172.21.15.1:0/1855363675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f399a050 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fa000021000 tx=0x7fa00c0bfe40 comp rx=0 tx=0).stop 2024-03-06T07:28:44.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.338+0000 7fa0139b1740 1 -- 172.21.15.1:0/1855363675 shutdown_connections 2024-03-06T07:28:44.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.338+0000 7fa0139b1740 1 --2- 172.21.15.1:0/1855363675 >> [v2:172.21.15.1:6832/1315978401,v1:172.21.15.1:6834/1315978401] conn(0x5556f39c5640 0x5556f39d0540 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.338+0000 7fa0139b1740 1 --2- 172.21.15.1:0/1855363675 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f39c25a0 0x5556f39ad9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.338+0000 7fa0139b1740 1 --2- 172.21.15.1:0/1855363675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f399a050 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.338+0000 7f9fe97fa640 0 client.10596 ms_handle_reset on v2:172.21.15.1:6832/1315978401 2024-03-06T07:28:44.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.338+0000 7fa0139b1740 1 --2- 172.21.15.1:0/1855363675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 0x5556f3997600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.338+0000 7fa0139b1740 1 -- 172.21.15.1:0/1855363675 >> 172.21.15.1:0/1855363675 conn(0x5556f39734b0 msgr2=0x5556f396f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.338+0000 7fa0139b1740 1 -- 172.21.15.1:0/1855363675 shutdown_connections 2024-03-06T07:28:44.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.338+0000 7fa0139b1740 1 -- 172.21.15.1:0/1855363675 wait complete. 2024-03-06T07:28:44.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.338+0000 7fa0139b1740 20 client.10596 trim_cache size 0 max 16384 2024-03-06T07:28:44.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.356+0000 7fa0139b1740 1 Processor -- start 2024-03-06T07:28:44.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.356+0000 7fa0139b1740 1 -- start start 2024-03-06T07:28:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.357+0000 7fa0139b1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 0x5556f3973a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.357+0000 7fa0139b1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f3994b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.357+0000 7fa0139b1740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f39c25a0 0x5556f3995060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.357+0000 7fa0139b1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c1f7640 con 0x5556f3975330 2024-03-06T07:28:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.357+0000 7fa0139b1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c1f6a70 con 0x5556f3974f50 2024-03-06T07:28:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.357+0000 7fa0139b1740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c1f7040 con 0x5556f39c25a0 2024-03-06T07:28:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.357+0000 7fa009d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f3994b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.357+0000 7fa009d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f3994b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33574/0 (socket says 172.21.15.1:33574) 2024-03-06T07:28:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.357+0000 7fa009d9b640 1 -- 172.21.15.1:0/1844390831 learned_addr learned my addr 172.21.15.1:0/1844390831 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.357+0000 7fa00a59c640 1 --2- 172.21.15.1:0/1844390831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 0x5556f3973a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.357+0000 7fa009d9b640 1 -- 172.21.15.1:0/1844390831 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f39c25a0 msgr2=0x5556f3995060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.357+0000 7fa00ad9d640 1 --2- 172.21.15.1:0/1844390831 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f39c25a0 0x5556f3995060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.357+0000 7fa009d9b640 1 --2- 172.21.15.1:0/1844390831 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f39c25a0 0x5556f3995060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.358+0000 7fa009d9b640 1 -- 172.21.15.1:0/1844390831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 msgr2=0x5556f3973a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.358+0000 7fa009d9b640 1 --2- 172.21.15.1:0/1844390831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 0x5556f3973a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.358+0000 7fa009d9b640 1 -- 172.21.15.1:0/1844390831 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556f36b86b0 con 0x5556f3975330 2024-03-06T07:28:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.358+0000 7fa00a59c640 1 --2- 172.21.15.1:0/1844390831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 0x5556f3973a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.358+0000 7fa00ad9d640 1 --2- 172.21.15.1:0/1844390831 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f39c25a0 0x5556f3995060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.358+0000 7fa009d9b640 1 --2- 172.21.15.1:0/1844390831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f3994b20 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fa000015d70 tx=0x7fa000029000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeeae4a3c2e8825f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.358+0000 7f9feb7fe640 1 -- 172.21.15.1:0/1844390831 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa000026040 con 0x5556f3975330 2024-03-06T07:28:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.358+0000 7f9feb7fe640 1 -- 172.21.15.1:0/1844390831 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00001c040 con 0x5556f3975330 2024-03-06T07:28:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.358+0000 7f9feb7fe640 1 -- 172.21.15.1:0/1844390831 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa000022dc0 con 0x5556f3975330 2024-03-06T07:28:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.359+0000 7fa0139b1740 1 -- 172.21.15.1:0/1844390831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 msgr2=0x5556f3994b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.359+0000 7fa0139b1740 1 --2- 172.21.15.1:0/1844390831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f3994b20 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fa000015d70 tx=0x7fa000029000 comp rx=0 tx=0).stop 2024-03-06T07:28:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.359+0000 7fa0139b1740 1 -- 172.21.15.1:0/1844390831 shutdown_connections 2024-03-06T07:28:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.359+0000 7fa0139b1740 1 --2- 172.21.15.1:0/1844390831 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f39c25a0 0x5556f3995060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.359+0000 7fa0139b1740 1 --2- 172.21.15.1:0/1844390831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f3975330 0x5556f3994b20 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.359+0000 7fa0139b1740 1 --2- 172.21.15.1:0/1844390831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 0x5556f3973a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.359+0000 7fa0139b1740 1 -- 172.21.15.1:0/1844390831 >> 172.21.15.1:0/1844390831 conn(0x5556f39734b0 msgr2=0x5556f39700a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.359+0000 7fa0139b1740 1 -- 172.21.15.1:0/1844390831 shutdown_connections 2024-03-06T07:28:44.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.359+0000 7fa0139b1740 1 -- 172.21.15.1:0/1844390831 wait complete. 2024-03-06T07:28:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.359+0000 7fa0139b1740 1 -- start start 2024-03-06T07:28:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.360+0000 7fa0139b1740 1 Processor -- start 2024-03-06T07:28:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa0139b1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 0x5556f3994f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa0139b1740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f3975330 0x5556f3a21110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa0139b1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f39c25a0 0x5556f3a21650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa0139b1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556f39ae490 con 0x5556f39c25a0 2024-03-06T07:28:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa0139b1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556f3997230 con 0x5556f3974f50 2024-03-06T07:28:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa0139b1740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5556f39996d0 con 0x5556f3975330 2024-03-06T07:28:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa0139b1740 10 client.0 _ll_register_callbacks cb 0x7fa004037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa00ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f39c25a0 0x5556f3a21650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa00ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f39c25a0 0x5556f3a21650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33586/0 (socket says 172.21.15.1:33586) 2024-03-06T07:28:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa00ad9d640 1 -- 172.21.15.1:0/3033390886 learned_addr learned my addr 172.21.15.1:0/3033390886 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa009d9b640 1 --2- 172.21.15.1:0/3033390886 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f3975330 0x5556f3a21110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa00a59c640 1 --2- 172.21.15.1:0/3033390886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 0x5556f3994f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa00ad9d640 1 -- 172.21.15.1:0/3033390886 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f3975330 msgr2=0x5556f3a21110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa00ad9d640 1 --2- 172.21.15.1:0/3033390886 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f3975330 0x5556f3a21110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa00ad9d640 1 -- 172.21.15.1:0/3033390886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 msgr2=0x5556f3994f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa00ad9d640 1 --2- 172.21.15.1:0/3033390886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 0x5556f3994f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa00ad9d640 1 -- 172.21.15.1:0/3033390886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556f39b1ff0 con 0x5556f39c25a0 2024-03-06T07:28:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7fa00ad9d640 1 --2- 172.21.15.1:0/3033390886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f39c25a0 0x5556f3a21650 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fa00402c690 tx=0x7fa004009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d91a90555c0278 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.361+0000 7f9fe97fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:28:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.362+0000 7f9fe97fa640 1 -- 172.21.15.1:0/3033390886 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004039040 con 0x5556f39c25a0 2024-03-06T07:28:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.362+0000 7f9fe97fa640 1 -- 172.21.15.1:0/3033390886 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa004003990 con 0x5556f39c25a0 2024-03-06T07:28:44.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.362+0000 7fa0139b1740 1 -- 172.21.15.1:0/3033390886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa00001c040 con 0x5556f39c25a0 2024-03-06T07:28:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.362+0000 7f9fe97fa640 1 -- 172.21.15.1:0/3033390886 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=185) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa004050050 con 0x5556f39c25a0 2024-03-06T07:28:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.362+0000 7fa0139b1740 10 client.10602 fetch_fsmap learned FSMap version 185 2024-03-06T07:28:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.362+0000 7fa0139b1740 1 -- 172.21.15.1:0/3033390886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=185,monmap=2+}) v3 -- 0x5556f396f690 con 0x5556f39c25a0 2024-03-06T07:28:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.362+0000 7f9fe97fa640 1 -- 172.21.15.1:0/3033390886 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00001c040 con 0x5556f39c25a0 2024-03-06T07:28:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.362+0000 7f9fe97fa640 1 -- 172.21.15.1:0/3033390886 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 185) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa004053020 con 0x5556f39c25a0 2024-03-06T07:28:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.362+0000 7fa0139b1740 10 client.10602 fetch_fsmap finished waiting for FSMap version 185 2024-03-06T07:28:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.362+0000 7fa0139b1740 10 client.10602 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:28:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.362+0000 7fa0139b1740 20 client.10602 populate_metadata read hostname 'smithi001' 2024-03-06T07:28:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.362+0000 7fa0139b1740 1 --2- 172.21.15.1:0/3033390886 >> [v2:172.21.15.1:6832/1315978401,v1:172.21.15.1:6834/1315978401] conn(0x5556f3a24680 0x5556f3a24a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.362+0000 7fa0139b1740 4 client.10602 mds_command: new command op to 10425 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:28:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.362+0000 7fa0139b1740 1 -- 172.21.15.1:0/3033390886 --> [v2:172.21.15.1:6832/1315978401,v1:172.21.15.1:6834/1315978401] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5556f38e0f40 con 0x5556f3a24680 2024-03-06T07:28:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.362+0000 7fa00a59c640 1 --2- 172.21.15.1:0/3033390886 >> [v2:172.21.15.1:6832/1315978401,v1:172.21.15.1:6834/1315978401] conn(0x5556f3a24680 0x5556f3a24a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.363+0000 7f9fe97fa640 10 client.10602 ms_handle_connect on v2:172.21.15.1:6832/1315978401 2024-03-06T07:28:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.363+0000 7fa00a59c640 1 --2- 172.21.15.1:0/3033390886 >> [v2:172.21.15.1:6832/1315978401,v1:172.21.15.1:6834/1315978401] conn(0x5556f3a24680 0x5556f3a24a60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bdac3397f254688 server_cookie=ef9ece1b87636e60 in_seq=0 out_seq=0 2024-03-06T07:28:44.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.363+0000 7f9fe97fa640 1 -- 172.21.15.1:0/3033390886 <== mds.0 v2:172.21.15.1:6832/1315978401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x5556f38e0f40 con 0x5556f3a24680 2024-03-06T07:28:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.363+0000 7f9fe97fa640 10 client.10602 handle_command_reply: tid=0 2024-03-06T07:28:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.363+0000 7fa0139b1740 1 client.10602 shutdown 2024-03-06T07:28:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.363+0000 7fa0139b1740 1 -- 172.21.15.1:0/3033390886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f39c25a0 msgr2=0x5556f3a21650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.363+0000 7fa0139b1740 1 --2- 172.21.15.1:0/3033390886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f39c25a0 0x5556f3a21650 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fa00402c690 tx=0x7fa004009750 comp rx=0 tx=0).stop 2024-03-06T07:28:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.363+0000 7fa0139b1740 1 -- 172.21.15.1:0/3033390886 shutdown_connections 2024-03-06T07:28:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.363+0000 7fa0139b1740 1 --2- 172.21.15.1:0/3033390886 >> [v2:172.21.15.1:6832/1315978401,v1:172.21.15.1:6834/1315978401] conn(0x5556f3a24680 0x5556f3a24a60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.363+0000 7fa0139b1740 1 --2- 172.21.15.1:0/3033390886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5556f39c25a0 0x5556f3a21650 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.363+0000 7fa0139b1740 1 --2- 172.21.15.1:0/3033390886 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5556f3975330 0x5556f3a21110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.363+0000 7fa0139b1740 1 --2- 172.21.15.1:0/3033390886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556f3974f50 0x5556f3994f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.363+0000 7f9fe97fa640 0 client.10602 ms_handle_reset on v2:172.21.15.1:6832/1315978401 2024-03-06T07:28:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.363+0000 7fa0139b1740 1 -- 172.21.15.1:0/3033390886 >> 172.21.15.1:0/3033390886 conn(0x5556f39734b0 msgr2=0x5556f39700a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.363+0000 7fa0139b1740 1 -- 172.21.15.1:0/3033390886 shutdown_connections 2024-03-06T07:28:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.363+0000 7fa0139b1740 1 -- 172.21.15.1:0/3033390886 wait complete. 2024-03-06T07:28:44.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.363+0000 7fa0139b1740 20 client.10602 trim_cache size 0 max 16384 2024-03-06T07:28:44.371 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:28:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:28:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:28:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 10425, 2024-03-06T07:28:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:28:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:28:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:28:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 9.3161308359999992, 2024-03-06T07:28:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 185, 2024-03-06T07:28:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 197, 2024-03-06T07:28:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 197, 2024-03-06T07:28:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 14.401220101 2024-03-06T07:28:44.371 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:28:44.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.364+0000 7fa010ced640 1 -- 172.21.15.1:0/2704649512 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9fec06b500 msgr2=0x7f9fec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.364+0000 7fa010ced640 1 --2- 172.21.15.1:0/2704649512 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9fec06b500 0x7f9fec06d9c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0c4bc0 tx=0x7fa000009040 comp rx=0 tx=0).stop 2024-03-06T07:28:44.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.364+0000 7fa010ced640 1 -- 172.21.15.1:0/2704649512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ff4004d40 msgr2=0x7fa00c0c9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.364+0000 7fa010ced640 1 --2- 172.21.15.1:0/2704649512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ff4004d40 0x7fa00c0c9a90 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc00f820 tx=0x7f9ffc00fce0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.364+0000 7fa010ced640 1 -- 172.21.15.1:0/2704649512 shutdown_connections 2024-03-06T07:28:44.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.364+0000 7fa010ced640 1 --2- 172.21.15.1:0/2704649512 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9fec06b500 0x7f9fec06d9c0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.364+0000 7fa010ced640 1 --2- 172.21.15.1:0/2704649512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa00c150fc0 0x7fa00c1ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.364+0000 7fa010ced640 1 --2- 172.21.15.1:0/2704649512 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c14b700 0x7fa00c150a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.364+0000 7fa010ced640 1 --2- 172.21.15.1:0/2704649512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ff4004d40 0x7fa00c0c9a90 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.364+0000 7fa010ced640 1 -- 172.21.15.1:0/2704649512 >> 172.21.15.1:0/2704649512 conn(0x7fa00c066ee0 msgr2=0x7fa00c0cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.364+0000 7fa010ced640 1 -- 172.21.15.1:0/2704649512 shutdown_connections 2024-03-06T07:28:44.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.364+0000 7fa010ced640 1 -- 172.21.15.1:0/2704649512 wait complete. 2024-03-06T07:28:44.376 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2024-03-06T07:28:44.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.480+0000 7fb59aba0640 1 Processor -- start 2024-03-06T07:28:44.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.480+0000 7fb59aba0640 1 -- start start 2024-03-06T07:28:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.480+0000 7fb59aba0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940c6930 0x7fb594156470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.481+0000 7fb59aba0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb594070050 0x7fb594072460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.481+0000 7fb59aba0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5940704c0 con 0x7fb5940c3940 2024-03-06T07:28:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.481+0000 7fb59aba0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb594153ea0 con 0x7fb594070050 2024-03-06T07:28:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.481+0000 7fb59aba0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5940c6fa0 con 0x7fb5940c6930 2024-03-06T07:28:44.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.481+0000 7fb599b9e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb5940c3940 0x7fb594153790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47034/0 (socket says 172.21.15.1:47034) 2024-03-06T07:28:44.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.481+0000 7fb599b9e640 1 -- 172.21.15.1:0/155949366 learned_addr learned my addr 172.21.15.1:0/155949366 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:44.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.481+0000 7fb59a39f640 1 --2- 172.21.15.1:0/155949366 >> v2:172.21.15.98:3300/0 conn(0x7fb594070050 0x7fb594072460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.481+0000 7fb59939d640 1 --2- 172.21.15.1:0/155949366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940c6930 0x7fb594156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.481+0000 7fb598b9c640 1 -- 172.21.15.1:0/155949366 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306956688 0 0) 0x7fb5940704c0 con 0x7fb5940c3940 2024-03-06T07:28:44.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.481+0000 7fb598b9c640 1 -- 172.21.15.1:0/155949366 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb580003520 con 0x7fb5940c3940 2024-03-06T07:28:44.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.481+0000 7fb59939d640 1 -- 172.21.15.1:0/155949366 >> v2:172.21.15.98:3300/0 conn(0x7fb594070050 msgr2=0x7fb594072460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.481+0000 7fb59939d640 1 --2- 172.21.15.1:0/155949366 >> v2:172.21.15.98:3300/0 conn(0x7fb594070050 0x7fb594072460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.481+0000 7fb59939d640 1 -- 172.21.15.1:0/155949366 >> v1:172.21.15.1:6789/0 conn(0x7fb5940c3940 legacy=0x7fb594153790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.481+0000 7fb59939d640 1 -- 172.21.15.1:0/155949366 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5940c7280 con 0x7fb5940c6930 2024-03-06T07:28:44.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.481+0000 7fb59a39f640 1 --2- 172.21.15.1:0/155949366 >> v2:172.21.15.98:3300/0 conn(0x7fb594070050 0x7fb594072460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:44.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.482+0000 7fb59939d640 1 --2- 172.21.15.1:0/155949366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940c6930 0x7fb594156470 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb5840097a0 tx=0x7fb58402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=366a6f4831ba551 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.483+0000 7fb598b9c640 1 -- 172.21.15.1:0/155949366 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb584031780 con 0x7fb5940c6930 2024-03-06T07:28:44.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.483+0000 7fb598b9c640 1 -- 172.21.15.1:0/155949366 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb584037070 con 0x7fb5940c6930 2024-03-06T07:28:44.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.483+0000 7fb598b9c640 1 -- 172.21.15.1:0/155949366 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb584031b40 con 0x7fb5940c6930 2024-03-06T07:28:44.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.483+0000 7fb59aba0640 1 -- 172.21.15.1:0/155949366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940c6930 msgr2=0x7fb594156470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.483+0000 7fb59aba0640 1 --2- 172.21.15.1:0/155949366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940c6930 0x7fb594156470 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb5840097a0 tx=0x7fb58402b820 comp rx=0 tx=0).stop 2024-03-06T07:28:44.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.483+0000 7fb59aba0640 1 -- 172.21.15.1:0/155949366 shutdown_connections 2024-03-06T07:28:44.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.483+0000 7fb59aba0640 1 --2- 172.21.15.1:0/155949366 >> v2:172.21.15.98:3300/0 conn(0x7fb594070050 0x7fb594072460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.483+0000 7fb59aba0640 1 --2- 172.21.15.1:0/155949366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940c6930 0x7fb594156470 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.483+0000 7fb59aba0640 1 -- 172.21.15.1:0/155949366 >> 172.21.15.1:0/155949366 conn(0x7fb5940c7ec0 msgr2=0x7fb5940a95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.483+0000 7fb59aba0640 1 -- 172.21.15.1:0/155949366 shutdown_connections 2024-03-06T07:28:44.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.483+0000 7fb59aba0640 1 -- 172.21.15.1:0/155949366 wait complete. 2024-03-06T07:28:44.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.484+0000 7fb59aba0640 1 Processor -- start 2024-03-06T07:28:44.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.484+0000 7fb59aba0640 1 -- start start 2024-03-06T07:28:44.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.485+0000 7fb59aba0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb594070050 0x7fb5941eb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.485+0000 7fb59aba0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5940c3940 0x7fb5940c0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.485+0000 7fb59aba0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940c6930 0x7fb5941eeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.485+0000 7fb59aba0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb594153270 con 0x7fb5940c3940 2024-03-06T07:28:44.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.485+0000 7fb59aba0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb594152f90 con 0x7fb594070050 2024-03-06T07:28:44.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.485+0000 7fb59aba0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5940c3d20 con 0x7fb5940c6930 2024-03-06T07:28:44.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.486+0000 7fb59939d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5940c3940 0x7fb5940c0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.486+0000 7fb59939d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5940c3940 0x7fb5940c0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33590/0 (socket says 172.21.15.1:33590) 2024-03-06T07:28:44.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.486+0000 7fb599b9e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb594070050 0x7fb5941eb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.486+0000 7fb59a39f640 1 --2- 172.21.15.1:0/926275231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940c6930 0x7fb5941eeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.486+0000 7fb59939d640 1 -- 172.21.15.1:0/926275231 learned_addr learned my addr 172.21.15.1:0/926275231 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:44.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.486+0000 7fb59939d640 1 -- 172.21.15.1:0/926275231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940c6930 msgr2=0x7fb5941eeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.486+0000 7fb59939d640 1 --2- 172.21.15.1:0/926275231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940c6930 0x7fb5941eeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.486+0000 7fb59939d640 1 -- 172.21.15.1:0/926275231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb594070050 msgr2=0x7fb5941eb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.486+0000 7fb59939d640 1 --2- 172.21.15.1:0/926275231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb594070050 0x7fb5941eb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.486+0000 7fb59939d640 1 -- 172.21.15.1:0/926275231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb594154ce0 con 0x7fb5940c3940 2024-03-06T07:28:44.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.486+0000 7fb599b9e640 1 --2- 172.21.15.1:0/926275231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb594070050 0x7fb5941eb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:44.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.486+0000 7fb59939d640 1 --2- 172.21.15.1:0/926275231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5940c3940 0x7fb5940c0ea0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fb584034040 tx=0x7fb5840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.486+0000 7fb58a7fc640 1 -- 172.21.15.1:0/926275231 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb584032610 con 0x7fb5940c3940 2024-03-06T07:28:44.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.486+0000 7fb58a7fc640 1 -- 172.21.15.1:0/926275231 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb584031ec0 con 0x7fb5940c3940 2024-03-06T07:28:44.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.486+0000 7fb59aba0640 1 -- 172.21.15.1:0/926275231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5941eb560 con 0x7fb5940c3940 2024-03-06T07:28:44.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.486+0000 7fb58a7fc640 1 -- 172.21.15.1:0/926275231 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb584032a10 con 0x7fb5940c3940 2024-03-06T07:28:44.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.486+0000 7fb59aba0640 1 -- 172.21.15.1:0/926275231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5940c13e0 con 0x7fb5940c3940 2024-03-06T07:28:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.487+0000 7fb59aba0640 1 -- 172.21.15.1:0/926275231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb594151090 con 0x7fb5940c3940 2024-03-06T07:28:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.488+0000 7fb58a7fc640 1 -- 172.21.15.1:0/926275231 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb584032bb0 con 0x7fb5940c3940 2024-03-06T07:28:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.488+0000 7fb58a7fc640 1 --2- 172.21.15.1:0/926275231 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb57806b4d0 0x7fb57806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.488+0000 7fb58a7fc640 1 -- 172.21.15.1:0/926275231 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb5840abf00 con 0x7fb5940c3940 2024-03-06T07:28:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.488+0000 7fb599b9e640 1 --2- 172.21.15.1:0/926275231 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb57806b4d0 0x7fb57806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.489+0000 7fb599b9e640 1 --2- 172.21.15.1:0/926275231 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb57806b4d0 0x7fb57806d990 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fb590005880 tx=0x7fb590009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.492+0000 7fb58a7fc640 1 -- 172.21.15.1:0/926275231 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb594151090 con 0x7fb5940c3940 2024-03-06T07:28:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:44.712+0000 7fb59aba0640 1 -- 172.21.15.1:0/926275231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"} v 0) v1 -- 0x7fb5940686e0 con 0x7fb5940c3940 2024-03-06T07:28:45.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.303+0000 7fb58a7fc640 1 -- 172.21.15.1:0/926275231 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"}]=0 cephfs marked down. v186) v1 ==== 125+0+0 (secure 0 0 0) 0x7fb584040070 con 0x7fb5940c3940 2024-03-06T07:28:45.305 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked down. 2024-03-06T07:28:45.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.306+0000 7fb59aba0640 1 -- 172.21.15.1:0/926275231 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb57806b4d0 msgr2=0x7fb57806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.306+0000 7fb59aba0640 1 --2- 172.21.15.1:0/926275231 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb57806b4d0 0x7fb57806d990 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fb590005880 tx=0x7fb590009040 comp rx=0 tx=0).stop 2024-03-06T07:28:45.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.306+0000 7fb59aba0640 1 -- 172.21.15.1:0/926275231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5940c3940 msgr2=0x7fb5940c0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.306+0000 7fb59aba0640 1 --2- 172.21.15.1:0/926275231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5940c3940 0x7fb5940c0ea0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fb584034040 tx=0x7fb5840046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.306+0000 7fb59aba0640 1 -- 172.21.15.1:0/926275231 shutdown_connections 2024-03-06T07:28:45.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.306+0000 7fb59aba0640 1 --2- 172.21.15.1:0/926275231 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb57806b4d0 0x7fb57806d990 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.306+0000 7fb59aba0640 1 --2- 172.21.15.1:0/926275231 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940c6930 0x7fb5941eeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.306+0000 7fb59aba0640 1 --2- 172.21.15.1:0/926275231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5940c3940 0x7fb5940c0ea0 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.306+0000 7fb59aba0640 1 --2- 172.21.15.1:0/926275231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb594070050 0x7fb5941eb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.306+0000 7fb59aba0640 1 -- 172.21.15.1:0/926275231 >> 172.21.15.1:0/926275231 conn(0x7fb5940c7ec0 msgr2=0x7fb5940a9590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.306+0000 7fb59aba0640 1 -- 172.21.15.1:0/926275231 shutdown_connections 2024-03-06T07:28:45.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.306+0000 7fb59aba0640 1 -- 172.21.15.1:0/926275231 wait complete. 2024-03-06T07:28:45.319 INFO:tasks.ceph_test_case:waiting 30s for health warning matching 2024-03-06T07:28:45.319 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:28:45.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.418+0000 7f1abaa29640 1 Processor -- start 2024-03-06T07:28:45.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.418+0000 7f1abaa29640 1 -- start start 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.419+0000 7f1abaa29640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1ab406bc20 0x7f1ab4152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.419+0000 7f1abaa29640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab41569e0 con 0x7f1ab414cdc0 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.419+0000 7f1abaa29640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab4156cc0 con 0x7f1ab4156500 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.419+0000 7f1abaa29640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab40c2dd0 con 0x7f1ab406bc20 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.419+0000 7f1ab8fc6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1ab406bc20 0x7f1ab4152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.419+0000 7f1ab3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1ab414cdc0 0x7f1ab40c2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47048/0 (socket says 172.21.15.1:47048) 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.419+0000 7f1ab3fff640 1 -- 172.21.15.1:0/3850898649 learned_addr learned my addr 172.21.15.1:0/3850898649 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.419+0000 7f1ab8fc6640 1 -- 172.21.15.1:0/3850898649 >> v1:172.21.15.98:6789/0 conn(0x7f1ab4156500 legacy=0x7f1ab406b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.419+0000 7f1ab8fc6640 1 -- 172.21.15.1:0/3850898649 >> v1:172.21.15.1:6789/0 conn(0x7f1ab414cdc0 legacy=0x7f1ab40c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.419+0000 7f1ab8fc6640 1 -- 172.21.15.1:0/3850898649 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab4156fa0 con 0x7f1ab406bc20 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.419+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/3850898649 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744919626 0 0) 0x7f1ab41569e0 con 0x7f1ab414cdc0 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.420+0000 7f1ab8fc6640 1 --2- 172.21.15.1:0/3850898649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab406bc20 0x7f1ab4152f10 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009d90 tx=0x7f1aac02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b5d861764a87fab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.420+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/3850898649 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac039070 con 0x7f1ab406bc20 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.420+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/3850898649 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aac02cbd0 con 0x7f1ab406bc20 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.420+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/3850898649 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac034440 con 0x7f1ab406bc20 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.420+0000 7f1abaa29640 1 -- 172.21.15.1:0/3850898649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab406bc20 msgr2=0x7f1ab4152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.420+0000 7f1abaa29640 1 --2- 172.21.15.1:0/3850898649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab406bc20 0x7f1ab4152f10 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009d90 tx=0x7f1aac02be20 comp rx=0 tx=0).stop 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.421+0000 7f1abaa29640 1 -- 172.21.15.1:0/3850898649 shutdown_connections 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.421+0000 7f1abaa29640 1 --2- 172.21.15.1:0/3850898649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab406bc20 0x7f1ab4152f10 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.421+0000 7f1abaa29640 1 -- 172.21.15.1:0/3850898649 >> 172.21.15.1:0/3850898649 conn(0x7f1ab40a9700 msgr2=0x7f1ab40a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.421+0000 7f1abaa29640 1 -- 172.21.15.1:0/3850898649 shutdown_connections 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.421+0000 7f1abaa29640 1 -- 172.21.15.1:0/3850898649 wait complete. 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.422+0000 7f1abaa29640 1 Processor -- start 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.422+0000 7f1abaa29640 1 -- start start 2024-03-06T07:28:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.422+0000 7f1abaa29640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab406bc20 0x7f1ab40c0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.422+0000 7f1abaa29640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ab414cdc0 0x7f1ab41519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.422+0000 7f1abaa29640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab4156500 0x7f1ab406f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1abaa29640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab40c65e0 con 0x7f1ab406bc20 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1abaa29640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab40c5a70 con 0x7f1ab4156500 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1abaa29640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab40c2a60 con 0x7f1ab414cdc0 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1ab3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab406bc20 0x7f1ab40c0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1ab3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab406bc20 0x7f1ab40c0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33598/0 (socket says 172.21.15.1:33598) 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1ab3fff640 1 -- 172.21.15.1:0/2424135193 learned_addr learned my addr 172.21.15.1:0/2424135193 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1ab8fc6640 1 --2- 172.21.15.1:0/2424135193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab4156500 0x7f1ab406f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1ab37fe640 1 --2- 172.21.15.1:0/2424135193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ab414cdc0 0x7f1ab41519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1ab3fff640 1 -- 172.21.15.1:0/2424135193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ab414cdc0 msgr2=0x7f1ab41519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1ab3fff640 1 --2- 172.21.15.1:0/2424135193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ab414cdc0 0x7f1ab41519b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1ab3fff640 1 -- 172.21.15.1:0/2424135193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab4156500 msgr2=0x7f1ab406f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1ab3fff640 1 --2- 172.21.15.1:0/2424135193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab4156500 0x7f1ab406f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1ab3fff640 1 -- 172.21.15.1:0/2424135193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab41524a0 con 0x7f1ab406bc20 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1ab3fff640 1 --2- 172.21.15.1:0/2424135193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab406bc20 0x7f1ab40c0d80 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c00c500 tx=0x7f1a9c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1ab0ff9640 1 -- 172.21.15.1:0/2424135193 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c018070 con 0x7f1ab406bc20 2024-03-06T07:28:45.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1ab0ff9640 1 -- 172.21.15.1:0/2424135193 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9c007580 con 0x7f1ab406bc20 2024-03-06T07:28:45.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1ab0ff9640 1 -- 172.21.15.1:0/2424135193 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c004a20 con 0x7f1ab406bc20 2024-03-06T07:28:45.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1abaa29640 1 -- 172.21.15.1:0/2424135193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab41ef0d0 con 0x7f1ab406bc20 2024-03-06T07:28:45.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.423+0000 7f1abaa29640 1 -- 172.21.15.1:0/2424135193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab41eadd0 con 0x7f1ab406bc20 2024-03-06T07:28:45.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.424+0000 7f1abaa29640 1 -- 172.21.15.1:0/2424135193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab414c970 con 0x7f1ab406bc20 2024-03-06T07:28:45.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.425+0000 7f1ab0ff9640 1 -- 172.21.15.1:0/2424135193 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a9c01b070 con 0x7f1ab406bc20 2024-03-06T07:28:45.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.425+0000 7f1ab0ff9640 1 --2- 172.21.15.1:0/2424135193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a9406b4d0 0x7f1a9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.425+0000 7f1ab0ff9640 1 -- 172.21.15.1:0/2424135193 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1a9c0896b0 con 0x7f1ab406bc20 2024-03-06T07:28:45.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.426+0000 7f1ab37fe640 1 --2- 172.21.15.1:0/2424135193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a9406b4d0 0x7f1a9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.426+0000 7f1ab37fe640 1 --2- 172.21.15.1:0/2424135193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a9406b4d0 0x7f1a9406d990 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4006fd0 tx=0x7f1aa4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.430+0000 7f1ab0ff9640 1 -- 172.21.15.1:0/2424135193 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ab414c970 con 0x7f1ab406bc20 2024-03-06T07:28:45.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.649+0000 7f1abaa29640 1 -- 172.21.15.1:0/2424135193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1ab40c5850 con 0x7f1ab406bc20 2024-03-06T07:28:45.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.649+0000 7f1ab0ff9640 1 -- 172.21.15.1:0/2424135193 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f1a9c0583a0 con 0x7f1ab406bc20 2024-03-06T07:28:45.651 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:45.651 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:28:45.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.651+0000 7f1abaa29640 1 -- 172.21.15.1:0/2424135193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a9406b4d0 msgr2=0x7f1a9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.651+0000 7f1abaa29640 1 --2- 172.21.15.1:0/2424135193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a9406b4d0 0x7f1a9406d990 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4006fd0 tx=0x7f1aa4008040 comp rx=0 tx=0).stop 2024-03-06T07:28:45.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.651+0000 7f1abaa29640 1 -- 172.21.15.1:0/2424135193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab406bc20 msgr2=0x7f1ab40c0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.651+0000 7f1abaa29640 1 --2- 172.21.15.1:0/2424135193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab406bc20 0x7f1ab40c0d80 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c00c500 tx=0x7f1a9c00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.651+0000 7f1abaa29640 1 -- 172.21.15.1:0/2424135193 shutdown_connections 2024-03-06T07:28:45.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.651+0000 7f1abaa29640 1 --2- 172.21.15.1:0/2424135193 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a9406b4d0 0x7f1a9406d990 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.651+0000 7f1abaa29640 1 --2- 172.21.15.1:0/2424135193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab4156500 0x7f1ab406f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.651+0000 7f1abaa29640 1 --2- 172.21.15.1:0/2424135193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ab414cdc0 0x7f1ab41519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.651+0000 7f1abaa29640 1 --2- 172.21.15.1:0/2424135193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab406bc20 0x7f1ab40c0d80 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.651+0000 7f1abaa29640 1 -- 172.21.15.1:0/2424135193 >> 172.21.15.1:0/2424135193 conn(0x7f1ab40a9700 msgr2=0x7f1ab41533d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.651+0000 7f1abaa29640 1 -- 172.21.15.1:0/2424135193 shutdown_connections 2024-03-06T07:28:45.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:45.651+0000 7f1abaa29640 1 -- 172.21.15.1:0/2424135193 wait complete. 2024-03-06T07:28:45.664 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-03-06T07:28:45.664 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:28:50.665 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:28:50.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.777+0000 7f90f8fd7640 1 Processor -- start 2024-03-06T07:28:50.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.777+0000 7f90f8fd7640 1 -- start start 2024-03-06T07:28:50.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f8fd7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90f41505b0 0x7f90f40c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f8fd7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f90f40c16e0 0x7f90f4156df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f8fd7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f406f5c0 con 0x7f90f40c1260 2024-03-06T07:28:50.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f8fd7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f4152f80 con 0x7f90f40c16e0 2024-03-06T07:28:50.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f8fd7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f4150990 con 0x7f90f41505b0 2024-03-06T07:28:50.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f1d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f90f40c1260 0x7f90f41566e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47056/0 (socket says 172.21.15.1:47056) 2024-03-06T07:28:50.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f1d9b640 1 -- 172.21.15.1:0/2955632606 learned_addr learned my addr 172.21.15.1:0/2955632606 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:28:50.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f2d9d640 1 --2- 172.21.15.1:0/2955632606 >> v2:172.21.15.98:3300/0 conn(0x7f90f40c16e0 0x7f90f4156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f259c640 1 --2- 172.21.15.1:0/2955632606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90f41505b0 0x7f90f40c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f159a640 1 -- 172.21.15.1:0/2955632606 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554499599 0 0) 0x7f90f406f5c0 con 0x7f90f40c1260 2024-03-06T07:28:50.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f159a640 1 -- 172.21.15.1:0/2955632606 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90dc003520 con 0x7f90f40c1260 2024-03-06T07:28:50.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f159a640 1 -- 172.21.15.1:0/2955632606 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4219572975 0 0) 0x7f90dc003520 con 0x7f90f40c1260 2024-03-06T07:28:50.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f159a640 1 -- 172.21.15.1:0/2955632606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90f41505b0 msgr2=0x7f90f40c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f159a640 1 --2- 172.21.15.1:0/2955632606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90f41505b0 0x7f90f40c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f159a640 1 -- 172.21.15.1:0/2955632606 >> v2:172.21.15.98:3300/0 conn(0x7f90f40c16e0 msgr2=0x7f90f4156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f159a640 1 --2- 172.21.15.1:0/2955632606 >> v2:172.21.15.98:3300/0 conn(0x7f90f40c16e0 0x7f90f4156df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f159a640 1 -- 172.21.15.1:0/2955632606 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f4153240 con 0x7f90f40c1260 2024-03-06T07:28:50.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f159a640 1 -- 172.21.15.1:0/2955632606 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f90e8001670 con 0x7f90f40c1260 2024-03-06T07:28:50.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.778+0000 7f90f159a640 1 -- 172.21.15.1:0/2955632606 >> v1:172.21.15.1:6789/0 conn(0x7f90f40c1260 legacy=0x7f90f41566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.779+0000 7f90f159a640 1 --2- 172.21.15.1:0/2955632606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90dc0039b0 0x7f90dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.779+0000 7f90f159a640 1 --2- 172.21.15.1:0/2955632606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90f41505b0 0x7f90dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.779+0000 7f90f159a640 1 --2- 172.21.15.1:0/2955632606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90dc004d40 0x7f90dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.779+0000 7f90f159a640 1 -- 172.21.15.1:0/2955632606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f4152f80 con 0x7f90dc0039b0 2024-03-06T07:28:50.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.779+0000 7f90f159a640 1 -- 172.21.15.1:0/2955632606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f4150990 con 0x7f90dc004d40 2024-03-06T07:28:50.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.779+0000 7f90f159a640 1 -- 172.21.15.1:0/2955632606 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc00b9c0 con 0x7f90f41505b0 2024-03-06T07:28:50.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.779+0000 7f90f259c640 1 --2- 172.21.15.1:0/2955632606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90dc0039b0 0x7f90dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.779+0000 7f90f1d9b640 1 --2- 172.21.15.1:0/2955632606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90f41505b0 0x7f90dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.779+0000 7f90f2d9d640 1 --2- 172.21.15.1:0/2955632606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90dc004d40 0x7f90dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.779+0000 7f90f259c640 1 -- 172.21.15.1:0/2955632606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90f41505b0 msgr2=0x7f90dc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.779+0000 7f90f259c640 1 --2- 172.21.15.1:0/2955632606 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90f41505b0 0x7f90dc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.779+0000 7f90f259c640 1 -- 172.21.15.1:0/2955632606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90dc004d40 msgr2=0x7f90dc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.779+0000 7f90f259c640 1 --2- 172.21.15.1:0/2955632606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90dc004d40 0x7f90dc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.779+0000 7f90f259c640 1 -- 172.21.15.1:0/2955632606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc003520 con 0x7f90dc0039b0 2024-03-06T07:28:50.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.780+0000 7f90f259c640 1 --2- 172.21.15.1:0/2955632606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90dc0039b0 0x7f90dc003dd0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f90e4009ac0 tx=0x7f90dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc6277e557ff7f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.780+0000 7f90f159a640 1 -- 172.21.15.1:0/2955632606 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e4039070 con 0x7f90dc0039b0 2024-03-06T07:28:50.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.780+0000 7f90f159a640 1 -- 172.21.15.1:0/2955632606 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e402c6b0 con 0x7f90dc0039b0 2024-03-06T07:28:50.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.780+0000 7f90f159a640 1 -- 172.21.15.1:0/2955632606 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e4009690 con 0x7f90dc0039b0 2024-03-06T07:28:50.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.780+0000 7f90f8fd7640 1 -- 172.21.15.1:0/2955632606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90dc0039b0 msgr2=0x7f90dc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.780+0000 7f90f8fd7640 1 --2- 172.21.15.1:0/2955632606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90dc0039b0 0x7f90dc003dd0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f90e4009ac0 tx=0x7f90dc005830 comp rx=0 tx=0).stop 2024-03-06T07:28:50.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.780+0000 7f90f2d9d640 1 -- 172.21.15.1:0/2955632606 reap_dead start 2024-03-06T07:28:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.780+0000 7f90f8fd7640 1 -- 172.21.15.1:0/2955632606 shutdown_connections 2024-03-06T07:28:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.780+0000 7f90f8fd7640 1 -- 172.21.15.1:0/2955632606 >> 172.21.15.1:0/2955632606 conn(0x7f90f40c6860 msgr2=0x7f90f40c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.781+0000 7f90f8fd7640 1 -- 172.21.15.1:0/2955632606 shutdown_connections 2024-03-06T07:28:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.781+0000 7f90f8fd7640 1 -- 172.21.15.1:0/2955632606 wait complete. 2024-03-06T07:28:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.782+0000 7f90f8fd7640 1 Processor -- start 2024-03-06T07:28:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.782+0000 7f90f8fd7640 1 -- start start 2024-03-06T07:28:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f8fd7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90dc0039b0 0x7f90f41554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f8fd7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90f414c500 0x7f90f414c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f8fd7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90f41ebf90 0x7f90f41ec410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f8fd7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f4153ae0 con 0x7f90f414c500 2024-03-06T07:28:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f8fd7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f4153800 con 0x7f90dc0039b0 2024-03-06T07:28:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f8fd7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f4157330 con 0x7f90f41ebf90 2024-03-06T07:28:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f1d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90f414c500 0x7f90f414c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f1d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90f414c500 0x7f90f414c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33618/0 (socket says 172.21.15.1:33618) 2024-03-06T07:28:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f1d9b640 1 -- 172.21.15.1:0/2096541118 learned_addr learned my addr 172.21.15.1:0/2096541118 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f259c640 1 --2- 172.21.15.1:0/2096541118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90dc0039b0 0x7f90f41554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f2d9d640 1 --2- 172.21.15.1:0/2096541118 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90f41ebf90 0x7f90f41ec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f1d9b640 1 -- 172.21.15.1:0/2096541118 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90f41ebf90 msgr2=0x7f90f41ec410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f1d9b640 1 --2- 172.21.15.1:0/2096541118 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90f41ebf90 0x7f90f41ec410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f1d9b640 1 -- 172.21.15.1:0/2096541118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90dc0039b0 msgr2=0x7f90f41554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f1d9b640 1 --2- 172.21.15.1:0/2096541118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90dc0039b0 0x7f90f41554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f1d9b640 1 -- 172.21.15.1:0/2096541118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f41e8c70 con 0x7f90f414c500 2024-03-06T07:28:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f1d9b640 1 --2- 172.21.15.1:0/2096541118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90f414c500 0x7f90f414c980 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f90e8007310 tx=0x7f90e80077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90e2ffd640 1 -- 172.21.15.1:0/2096541118 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e800f960 con 0x7f90f414c500 2024-03-06T07:28:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90e2ffd640 1 -- 172.21.15.1:0/2096541118 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e8018070 con 0x7f90f414c500 2024-03-06T07:28:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90f8fd7640 1 -- 172.21.15.1:0/2096541118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f41e8f50 con 0x7f90f414c500 2024-03-06T07:28:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.783+0000 7f90e2ffd640 1 -- 172.21.15.1:0/2096541118 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e800fd40 con 0x7f90f414c500 2024-03-06T07:28:50.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.784+0000 7f90f8fd7640 1 -- 172.21.15.1:0/2096541118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f41559f0 con 0x7f90f414c500 2024-03-06T07:28:50.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.784+0000 7f90f8fd7640 1 -- 172.21.15.1:0/2096541118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f4067fd0 con 0x7f90f414c500 2024-03-06T07:28:50.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.785+0000 7f90e2ffd640 1 -- 172.21.15.1:0/2096541118 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90e801d460 con 0x7f90f414c500 2024-03-06T07:28:50.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.785+0000 7f90e2ffd640 1 --2- 172.21.15.1:0/2096541118 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90d406b500 0x7f90d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.785+0000 7f90e2ffd640 1 -- 172.21.15.1:0/2096541118 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f90e808a2e0 con 0x7f90f414c500 2024-03-06T07:28:50.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.786+0000 7f90f259c640 1 --2- 172.21.15.1:0/2096541118 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90d406b500 0x7f90d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.786+0000 7f90f259c640 1 --2- 172.21.15.1:0/2096541118 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90d406b500 0x7f90d406d9c0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f90e4036000 tx=0x7f90e4030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:50.790+0000 7f90e2ffd640 1 -- 172.21.15.1:0/2096541118 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90e8058a10 con 0x7f90f414c500 2024-03-06T07:28:51.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:51.008+0000 7f90f8fd7640 1 -- 172.21.15.1:0/2096541118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f90f406f280 con 0x7f90f414c500 2024-03-06T07:28:51.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:51.008+0000 7f90e2ffd640 1 -- 172.21.15.1:0/2096541118 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f90e805cab0 con 0x7f90f414c500 2024-03-06T07:28:51.010 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:51.010 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:51.010+0000 7f90f8fd7640 1 -- 172.21.15.1:0/2096541118 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90d406b500 msgr2=0x7f90d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:51.010+0000 7f90f8fd7640 1 --2- 172.21.15.1:0/2096541118 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90d406b500 0x7f90d406d9c0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f90e4036000 tx=0x7f90e4030000 comp rx=0 tx=0).stop 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:51.010+0000 7f90f8fd7640 1 -- 172.21.15.1:0/2096541118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90f414c500 msgr2=0x7f90f414c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:51.010+0000 7f90f8fd7640 1 --2- 172.21.15.1:0/2096541118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90f414c500 0x7f90f414c980 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f90e8007310 tx=0x7f90e80077d0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:51.010+0000 7f90f8fd7640 1 -- 172.21.15.1:0/2096541118 shutdown_connections 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:51.010+0000 7f90f8fd7640 1 --2- 172.21.15.1:0/2096541118 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90d406b500 0x7f90d406d9c0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:51.010+0000 7f90f8fd7640 1 --2- 172.21.15.1:0/2096541118 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90f41ebf90 0x7f90f41ec410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:51.010+0000 7f90f8fd7640 1 --2- 172.21.15.1:0/2096541118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90f414c500 0x7f90f414c980 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:51.010+0000 7f90f8fd7640 1 --2- 172.21.15.1:0/2096541118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90dc0039b0 0x7f90f41554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:51.010+0000 7f90f8fd7640 1 -- 172.21.15.1:0/2096541118 >> 172.21.15.1:0/2096541118 conn(0x7f90f40c6860 msgr2=0x7f90f40c5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:51.010+0000 7f90f8fd7640 1 -- 172.21.15.1:0/2096541118 shutdown_connections 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:51.010+0000 7f90f8fd7640 1 -- 172.21.15.1:0/2096541118 wait complete. 2024-03-06T07:28:51.023 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-03-06T07:28:51.023 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:28:56.024 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:28:56.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.134+0000 7f231a66f640 1 Processor -- start 2024-03-06T07:28:56.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.135+0000 7f231a66f640 1 -- start start 2024-03-06T07:28:56.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.135+0000 7f231a66f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2314152650 0x7f2314155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.135+0000 7f231a66f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f23140c6b10 0x7f2314158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.135+0000 7f231a66f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23140c2fc0 con 0x7f23140bf9d0 2024-03-06T07:28:56.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.135+0000 7f231a66f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2314155800 con 0x7f2314152650 2024-03-06T07:28:56.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.135+0000 7f231a66f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23140c6690 con 0x7f23140c6b10 2024-03-06T07:28:56.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.135+0000 7f2313fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2314152650 0x7f2314155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.135+0000 7f2313fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2314152650 0x7f2314155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60916/0 (socket says 172.21.15.1:60916) 2024-03-06T07:28:56.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.135+0000 7f2313fff640 1 -- 172.21.15.1:0/2232393709 learned_addr learned my addr 172.21.15.1:0/2232393709 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:56.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.136+0000 7f2313fff640 1 -- 172.21.15.1:0/2232393709 >> v2:172.21.15.98:3300/0 conn(0x7f23140c6b10 msgr2=0x7f2314158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.136+0000 7f2313fff640 1 --2- 172.21.15.1:0/2232393709 >> v2:172.21.15.98:3300/0 conn(0x7f23140c6b10 0x7f2314158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.136+0000 7f2313fff640 1 -- 172.21.15.1:0/2232393709 >> v1:172.21.15.98:6789/0 conn(0x7f23140bf9d0 legacy=0x7f23140c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.136+0000 7f2313fff640 1 -- 172.21.15.1:0/2232393709 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23140c7280 con 0x7f2314152650 2024-03-06T07:28:56.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.136+0000 7f2313fff640 1 --2- 172.21.15.1:0/2232393709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2314152650 0x7f2314155c90 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f23140cb8e0 tx=0x7f22fc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3068ada58f1e974b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.137+0000 7f2312ffd640 1 -- 172.21.15.1:0/2232393709 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc02e070 con 0x7f2314152650 2024-03-06T07:28:56.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.137+0000 7f2312ffd640 1 -- 172.21.15.1:0/2232393709 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22fc037070 con 0x7f2314152650 2024-03-06T07:28:56.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.137+0000 7f2312ffd640 1 -- 172.21.15.1:0/2232393709 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc031a00 con 0x7f2314152650 2024-03-06T07:28:56.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.137+0000 7f231a66f640 1 -- 172.21.15.1:0/2232393709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2314152650 msgr2=0x7f2314155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.137+0000 7f231a66f640 1 --2- 172.21.15.1:0/2232393709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2314152650 0x7f2314155c90 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f23140cb8e0 tx=0x7f22fc02b880 comp rx=0 tx=0).stop 2024-03-06T07:28:56.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.137+0000 7f231a66f640 1 -- 172.21.15.1:0/2232393709 shutdown_connections 2024-03-06T07:28:56.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.137+0000 7f231a66f640 1 --2- 172.21.15.1:0/2232393709 >> v2:172.21.15.98:3300/0 conn(0x7f23140c6b10 0x7f2314158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.137+0000 7f231a66f640 1 --2- 172.21.15.1:0/2232393709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2314152650 0x7f2314155c90 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.137+0000 7f231a66f640 1 -- 172.21.15.1:0/2232393709 >> 172.21.15.1:0/2232393709 conn(0x7f2314066ee0 msgr2=0x7f23140672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.137+0000 7f231a66f640 1 -- 172.21.15.1:0/2232393709 shutdown_connections 2024-03-06T07:28:56.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.137+0000 7f231a66f640 1 -- 172.21.15.1:0/2232393709 wait complete. 2024-03-06T07:28:56.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.138+0000 7f231a66f640 1 Processor -- start 2024-03-06T07:28:56.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.139+0000 7f231a66f640 1 -- start start 2024-03-06T07:28:56.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.139+0000 7f231a66f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23140bf9d0 0x7f2314153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.139+0000 7f231a66f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2314152650 0x7f23141ee430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.139+0000 7f231a66f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23140c87d0 0x7f23141ee970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.139+0000 7f231a66f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2314155490 con 0x7f2314152650 2024-03-06T07:28:56.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.139+0000 7f231a66f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23141551b0 con 0x7f23140bf9d0 2024-03-06T07:28:56.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.139+0000 7f231a66f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2314152a30 con 0x7f23140c87d0 2024-03-06T07:28:56.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.139+0000 7f23137fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2314152650 0x7f23141ee430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.139+0000 7f23137fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2314152650 0x7f23141ee430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60928/0 (socket says 172.21.15.1:60928) 2024-03-06T07:28:56.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.139+0000 7f23137fe640 1 -- 172.21.15.1:0/3935400447 learned_addr learned my addr 172.21.15.1:0/3935400447 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:28:56.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.140+0000 7f2313fff640 1 --2- 172.21.15.1:0/3935400447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23140bf9d0 0x7f2314153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.140+0000 7f2318c0c640 1 --2- 172.21.15.1:0/3935400447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23140c87d0 0x7f23141ee970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.140+0000 7f23137fe640 1 -- 172.21.15.1:0/3935400447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23140c87d0 msgr2=0x7f23141ee970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.140+0000 7f23137fe640 1 --2- 172.21.15.1:0/3935400447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23140c87d0 0x7f23141ee970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.140+0000 7f23137fe640 1 -- 172.21.15.1:0/3935400447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23140bf9d0 msgr2=0x7f2314153110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.140+0000 7f23137fe640 1 --2- 172.21.15.1:0/3935400447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23140bf9d0 0x7f2314153110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.140+0000 7f23137fe640 1 -- 172.21.15.1:0/3935400447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231414b5a0 con 0x7f2314152650 2024-03-06T07:28:56.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.140+0000 7f2318c0c640 1 --2- 172.21.15.1:0/3935400447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23140c87d0 0x7f23141ee970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:56.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.140+0000 7f23137fe640 1 --2- 172.21.15.1:0/3935400447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2314152650 0x7f23141ee430 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f230400b410 tx=0x7f230400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.140+0000 7f2310ff9640 1 -- 172.21.15.1:0/3935400447 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2304015020 con 0x7f2314152650 2024-03-06T07:28:56.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.140+0000 7f2310ff9640 1 -- 172.21.15.1:0/3935400447 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2304011850 con 0x7f2314152650 2024-03-06T07:28:56.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.140+0000 7f2310ff9640 1 -- 172.21.15.1:0/3935400447 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2304011c40 con 0x7f2314152650 2024-03-06T07:28:56.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.140+0000 7f231a66f640 1 -- 172.21.15.1:0/3935400447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231414b150 con 0x7f2314152650 2024-03-06T07:28:56.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.140+0000 7f231a66f640 1 -- 172.21.15.1:0/3935400447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2314153650 con 0x7f2314152650 2024-03-06T07:28:56.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.141+0000 7f231a66f640 1 -- 172.21.15.1:0/3935400447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23141510b0 con 0x7f2314152650 2024-03-06T07:28:56.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.142+0000 7f2310ff9640 1 -- 172.21.15.1:0/3935400447 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2304018020 con 0x7f2314152650 2024-03-06T07:28:56.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.142+0000 7f2310ff9640 1 --2- 172.21.15.1:0/3935400447 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f22f406b4d0 0x7f22f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.142+0000 7f2310ff9640 1 -- 172.21.15.1:0/3935400447 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f230408a220 con 0x7f2314152650 2024-03-06T07:28:56.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.143+0000 7f2313fff640 1 --2- 172.21.15.1:0/3935400447 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f22f406b4d0 0x7f22f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.143+0000 7f2313fff640 1 --2- 172.21.15.1:0/3935400447 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f22f406b4d0 0x7f22f406d990 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f22fc036000 tx=0x7f22fc03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.147+0000 7f2310ff9640 1 -- 172.21.15.1:0/3935400447 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f230401f020 con 0x7f2314152650 2024-03-06T07:28:56.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.365+0000 7f231a66f640 1 -- 172.21.15.1:0/3935400447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2314067fd0 con 0x7f2314152650 2024-03-06T07:28:56.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.365+0000 7f2310ff9640 1 -- 172.21.15.1:0/3935400447 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f23040172c0 con 0x7f2314152650 2024-03-06T07:28:56.367 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:28:56.367 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:28:56.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.367+0000 7f231a66f640 1 -- 172.21.15.1:0/3935400447 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f22f406b4d0 msgr2=0x7f22f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.367+0000 7f231a66f640 1 --2- 172.21.15.1:0/3935400447 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f22f406b4d0 0x7f22f406d990 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f22fc036000 tx=0x7f22fc03d040 comp rx=0 tx=0).stop 2024-03-06T07:28:56.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.367+0000 7f231a66f640 1 -- 172.21.15.1:0/3935400447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2314152650 msgr2=0x7f23141ee430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.367+0000 7f231a66f640 1 --2- 172.21.15.1:0/3935400447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2314152650 0x7f23141ee430 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f230400b410 tx=0x7f230400b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.367+0000 7f231a66f640 1 -- 172.21.15.1:0/3935400447 shutdown_connections 2024-03-06T07:28:56.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.367+0000 7f231a66f640 1 --2- 172.21.15.1:0/3935400447 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f22f406b4d0 0x7f22f406d990 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.367+0000 7f231a66f640 1 --2- 172.21.15.1:0/3935400447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23140c87d0 0x7f23141ee970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.367+0000 7f231a66f640 1 --2- 172.21.15.1:0/3935400447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2314152650 0x7f23141ee430 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.367+0000 7f231a66f640 1 --2- 172.21.15.1:0/3935400447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23140bf9d0 0x7f2314153110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.367+0000 7f231a66f640 1 -- 172.21.15.1:0/3935400447 >> 172.21.15.1:0/3935400447 conn(0x7f2314066ee0 msgr2=0x7f23140c5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.367+0000 7f231a66f640 1 -- 172.21.15.1:0/3935400447 shutdown_connections 2024-03-06T07:28:56.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:28:56.367+0000 7f231a66f640 1 -- 172.21.15.1:0/3935400447 wait complete. 2024-03-06T07:28:56.380 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-03-06T07:28:56.380 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:29:00.339 INFO:tasks.ceph.mds.a.smithi001.stderr: -783> 2024-03-06T07:28:40.330+0000 7f527d3fd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:29:00.368 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:29:01.381 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:29:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.491+0000 7fc2af3d0640 1 Processor -- start 2024-03-06T07:29:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.491+0000 7fc2af3d0640 1 -- start start 2024-03-06T07:29:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.492+0000 7fc2af3d0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2a80c6b10 0x7fc2a8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.492+0000 7fc2af3d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2a80bf9d0 0x7fc2a80c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.492+0000 7fc2af3d0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a80c2fc0 con 0x7fc2a8152650 2024-03-06T07:29:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.492+0000 7fc2af3d0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a8155800 con 0x7fc2a80c6b10 2024-03-06T07:29:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.492+0000 7fc2af3d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a8158da0 con 0x7fc2a80bf9d0 2024-03-06T07:29:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.492+0000 7fc2ad16c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc2a8152650 0x7fc2a8158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36704/0 (socket says 172.21.15.1:36704) 2024-03-06T07:29:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.492+0000 7fc2ad16c640 1 -- 172.21.15.1:0/3225812843 learned_addr learned my addr 172.21.15.1:0/3225812843 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.492+0000 7fc2ac96b640 1 --2- 172.21.15.1:0/3225812843 >> v2:172.21.15.98:3300/0 conn(0x7fc2a80c6b10 0x7fc2a8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.492+0000 7fc2ad96d640 1 --2- 172.21.15.1:0/3225812843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2a80bf9d0 0x7fc2a80c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.493+0000 7fc2a7fff640 1 -- 172.21.15.1:0/3225812843 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253173868 0 0) 0x7fc2a80c2fc0 con 0x7fc2a8152650 2024-03-06T07:29:01.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.493+0000 7fc2a7fff640 1 -- 172.21.15.1:0/3225812843 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc290003520 con 0x7fc2a8152650 2024-03-06T07:29:01.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.493+0000 7fc2ac96b640 1 -- 172.21.15.1:0/3225812843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2a80bf9d0 msgr2=0x7fc2a80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:01.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.493+0000 7fc2ac96b640 1 --2- 172.21.15.1:0/3225812843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2a80bf9d0 0x7fc2a80c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.493+0000 7fc2ac96b640 1 -- 172.21.15.1:0/3225812843 >> v1:172.21.15.1:6789/0 conn(0x7fc2a8152650 legacy=0x7fc2a8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.493+0000 7fc2a7fff640 1 -- 172.21.15.1:0/3225812843 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2578163069 0 0) 0x7fc290003520 con 0x7fc2a8152650 2024-03-06T07:29:01.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.493+0000 7fc2ac96b640 1 -- 172.21.15.1:0/3225812843 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a80c6690 con 0x7fc2a80c6b10 2024-03-06T07:29:01.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.493+0000 7fc2ad96d640 1 --2- 172.21.15.1:0/3225812843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2a80bf9d0 0x7fc2a80c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:01.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.494+0000 7fc2ac96b640 1 --2- 172.21.15.1:0/3225812843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2a80c6b10 0x7fc2a8155c90 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0097a0 tx=0x7fc29c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a33a42a9fef6f7c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.494+0000 7fc2a7fff640 1 -- 172.21.15.1:0/3225812843 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c0315a0 con 0x7fc2a80c6b10 2024-03-06T07:29:01.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.494+0000 7fc2a7fff640 1 -- 172.21.15.1:0/3225812843 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29c037070 con 0x7fc2a80c6b10 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.494+0000 7fc2a7fff640 1 -- 172.21.15.1:0/3225812843 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c031940 con 0x7fc2a80c6b10 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.494+0000 7fc2af3d0640 1 -- 172.21.15.1:0/3225812843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2a80c6b10 msgr2=0x7fc2a8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.494+0000 7fc2af3d0640 1 --2- 172.21.15.1:0/3225812843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2a80c6b10 0x7fc2a8155c90 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0097a0 tx=0x7fc29c02b840 comp rx=0 tx=0).stop 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.494+0000 7fc2af3d0640 1 -- 172.21.15.1:0/3225812843 shutdown_connections 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.494+0000 7fc2af3d0640 1 --2- 172.21.15.1:0/3225812843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2a80bf9d0 0x7fc2a80c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.494+0000 7fc2af3d0640 1 --2- 172.21.15.1:0/3225812843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2a80c6b10 0x7fc2a8155c90 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.494+0000 7fc2af3d0640 1 -- 172.21.15.1:0/3225812843 >> 172.21.15.1:0/3225812843 conn(0x7fc2a8066ee0 msgr2=0x7fc2a80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.495+0000 7fc2af3d0640 1 -- 172.21.15.1:0/3225812843 shutdown_connections 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.495+0000 7fc2af3d0640 1 -- 172.21.15.1:0/3225812843 wait complete. 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.496+0000 7fc2af3d0640 1 Processor -- start 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.496+0000 7fc2af3d0640 1 -- start start 2024-03-06T07:29:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.496+0000 7fc2af3d0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a80bf9d0 0x7fc2a81ee1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.496+0000 7fc2af3d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2a8152650 0x7fc2a80c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.496+0000 7fc2af3d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2a80ca4b0 0x7fc2a80c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.496+0000 7fc2af3d0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a81551b0 con 0x7fc2a80bf9d0 2024-03-06T07:29:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.496+0000 7fc2af3d0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a80c6fa0 con 0x7fc2a80ca4b0 2024-03-06T07:29:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.496+0000 7fc2af3d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a8155490 con 0x7fc2a8152650 2024-03-06T07:29:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.496+0000 7fc2ad16c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a80bf9d0 0x7fc2a81ee1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.496+0000 7fc2ad16c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a80bf9d0 0x7fc2a81ee1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60954/0 (socket says 172.21.15.1:60954) 2024-03-06T07:29:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.496+0000 7fc2ad16c640 1 -- 172.21.15.1:0/42601600 learned_addr learned my addr 172.21.15.1:0/42601600 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.497+0000 7fc2ad96d640 1 --2- 172.21.15.1:0/42601600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2a80ca4b0 0x7fc2a80c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.497+0000 7fc2ac96b640 1 --2- 172.21.15.1:0/42601600 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2a8152650 0x7fc2a80c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.497+0000 7fc2ad16c640 1 -- 172.21.15.1:0/42601600 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2a8152650 msgr2=0x7fc2a80c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.497+0000 7fc2ad16c640 1 --2- 172.21.15.1:0/42601600 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2a8152650 0x7fc2a80c36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.497+0000 7fc2ad16c640 1 -- 172.21.15.1:0/42601600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2a80ca4b0 msgr2=0x7fc2a80c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.497+0000 7fc2ad16c640 1 --2- 172.21.15.1:0/42601600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2a80ca4b0 0x7fc2a80c90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.497+0000 7fc2ad16c640 1 -- 172.21.15.1:0/42601600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a81ee700 con 0x7fc2a80bf9d0 2024-03-06T07:29:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.497+0000 7fc2ad96d640 1 --2- 172.21.15.1:0/42601600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2a80ca4b0 0x7fc2a80c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.497+0000 7fc2ad16c640 1 --2- 172.21.15.1:0/42601600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a80bf9d0 0x7fc2a81ee1c0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0001eb0 tx=0x7fc2a000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.497+0000 7fc2a5ffb640 1 -- 172.21.15.1:0/42601600 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a00074e0 con 0x7fc2a80bf9d0 2024-03-06T07:29:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.497+0000 7fc2a5ffb640 1 -- 172.21.15.1:0/42601600 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a0007680 con 0x7fc2a80bf9d0 2024-03-06T07:29:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.497+0000 7fc2af3d0640 1 -- 172.21.15.1:0/42601600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a8158bb0 con 0x7fc2a80bf9d0 2024-03-06T07:29:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.497+0000 7fc2a5ffb640 1 -- 172.21.15.1:0/42601600 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0004a00 con 0x7fc2a80bf9d0 2024-03-06T07:29:01.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.497+0000 7fc2af3d0640 1 -- 172.21.15.1:0/42601600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a81588d0 con 0x7fc2a80bf9d0 2024-03-06T07:29:01.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.498+0000 7fc2af3d0640 1 -- 172.21.15.1:0/42601600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a8067fd0 con 0x7fc2a80bf9d0 2024-03-06T07:29:01.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.499+0000 7fc2a5ffb640 1 -- 172.21.15.1:0/42601600 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2a0011070 con 0x7fc2a80bf9d0 2024-03-06T07:29:01.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.499+0000 7fc2a5ffb640 1 --2- 172.21.15.1:0/42601600 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc29406b4d0 0x7fc29406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.499+0000 7fc2a5ffb640 1 -- 172.21.15.1:0/42601600 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc2a008a070 con 0x7fc2a80bf9d0 2024-03-06T07:29:01.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.500+0000 7fc2ac96b640 1 --2- 172.21.15.1:0/42601600 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc29406b4d0 0x7fc29406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.500+0000 7fc2ac96b640 1 --2- 172.21.15.1:0/42601600 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc29406b4d0 0x7fc29406d990 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc29c036000 tx=0x7fc29c03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.504+0000 7fc2a5ffb640 1 -- 172.21.15.1:0/42601600 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2a0058d50 con 0x7fc2a80bf9d0 2024-03-06T07:29:01.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.725+0000 7fc2af3d0640 1 -- 172.21.15.1:0/42601600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc2a80b06f0 con 0x7fc2a80bf9d0 2024-03-06T07:29:01.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.726+0000 7fc2a5ffb640 1 -- 172.21.15.1:0/42601600 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fc2a0058650 con 0x7fc2a80bf9d0 2024-03-06T07:29:01.728 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:01.728 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:29:01.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.728+0000 7fc2af3d0640 1 -- 172.21.15.1:0/42601600 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc29406b4d0 msgr2=0x7fc29406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.728+0000 7fc2af3d0640 1 --2- 172.21.15.1:0/42601600 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc29406b4d0 0x7fc29406d990 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc29c036000 tx=0x7fc29c03d040 comp rx=0 tx=0).stop 2024-03-06T07:29:01.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.728+0000 7fc2af3d0640 1 -- 172.21.15.1:0/42601600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a80bf9d0 msgr2=0x7fc2a81ee1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.728+0000 7fc2af3d0640 1 --2- 172.21.15.1:0/42601600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a80bf9d0 0x7fc2a81ee1c0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0001eb0 tx=0x7fc2a000c8e0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.728+0000 7fc2af3d0640 1 -- 172.21.15.1:0/42601600 shutdown_connections 2024-03-06T07:29:01.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.728+0000 7fc2af3d0640 1 --2- 172.21.15.1:0/42601600 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc29406b4d0 0x7fc29406d990 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.728+0000 7fc2af3d0640 1 --2- 172.21.15.1:0/42601600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2a80ca4b0 0x7fc2a80c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.728+0000 7fc2af3d0640 1 --2- 172.21.15.1:0/42601600 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2a8152650 0x7fc2a80c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.728+0000 7fc2af3d0640 1 --2- 172.21.15.1:0/42601600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a80bf9d0 0x7fc2a81ee1c0 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.728+0000 7fc2af3d0640 1 -- 172.21.15.1:0/42601600 >> 172.21.15.1:0/42601600 conn(0x7fc2a8066ee0 msgr2=0x7fc2a80672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.728+0000 7fc2af3d0640 1 -- 172.21.15.1:0/42601600 shutdown_connections 2024-03-06T07:29:01.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:01.728+0000 7fc2af3d0640 1 -- 172.21.15.1:0/42601600 wait complete. 2024-03-06T07:29:01.740 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-03-06T07:29:01.740 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:29:02.541 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:02.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:06.742 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:29:06.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.852+0000 7fa22388d640 1 Processor -- start 2024-03-06T07:29:06.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.852+0000 7fa22388d640 1 -- start start 2024-03-06T07:29:06.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.853+0000 7fa22388d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa21c0c6730 0x7fa21c0c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.853+0000 7fa22388d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa21c0c5d10 0x7fa21c156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.853+0000 7fa22388d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21c0c3300 con 0x7fa21c152690 2024-03-06T07:29:06.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.853+0000 7fa22388d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa21c0c35a0 con 0x7fa21c0c6730 2024-03-06T07:29:06.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.853+0000 7fa22388d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c0c3880 con 0x7fa21c0c5d10 2024-03-06T07:29:06.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.853+0000 7fa220e28640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa21c0c6730 0x7fa21c0c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.853+0000 7fa221e2a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa21c0c5d10 0x7fa21c156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.853+0000 7fa220e28640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa21c0c6730 0x7fa21c0c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:38612/0 (socket says 172.21.15.1:38612) 2024-03-06T07:29:06.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.853+0000 7fa221e2a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa21c0c5d10 0x7fa21c156cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:50930/0 (socket says 172.21.15.1:50930) 2024-03-06T07:29:06.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.853+0000 7fa220e28640 1 -- 172.21.15.1:0/1967540106 learned_addr learned my addr 172.21.15.1:0/1967540106 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:06.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.853+0000 7fa221629640 1 --1- 172.21.15.1:0/1967540106 >> v1:172.21.15.98:6789/0 conn(0x7fa21c152690 0x7fa21c155ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:47966/0 (socket says 172.21.15.1:47966) 2024-03-06T07:29:06.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.854+0000 7fa220e28640 1 -- 172.21.15.1:0/1967540106 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa21c0c5d10 msgr2=0x7fa21c156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:06.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.854+0000 7fa220e28640 1 --2- 172.21.15.1:0/1967540106 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa21c0c5d10 0x7fa21c156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.854+0000 7fa220e28640 1 -- 172.21.15.1:0/1967540106 >> v1:172.21.15.98:6789/0 conn(0x7fa21c152690 legacy=0x7fa21c155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.854+0000 7fa220e28640 1 -- 172.21.15.1:0/1967540106 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c0c3b60 con 0x7fa21c0c6730 2024-03-06T07:29:06.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.854+0000 7fa221e2a640 1 --2- 172.21.15.1:0/1967540106 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa21c0c5d10 0x7fa21c156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.855+0000 7fa220e28640 1 --2- 172.21.15.1:0/1967540106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa21c0c6730 0x7fa21c0c57d0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fa20c009870 tx=0x7fa20c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b6a658e1ebb0d88 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.855+0000 7fa213fff640 1 -- 172.21.15.1:0/1967540106 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20c035020 con 0x7fa21c0c6730 2024-03-06T07:29:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.855+0000 7fa213fff640 1 -- 172.21.15.1:0/1967540106 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20c031640 con 0x7fa21c0c6730 2024-03-06T07:29:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.855+0000 7fa213fff640 1 -- 172.21.15.1:0/1967540106 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20c02d040 con 0x7fa21c0c6730 2024-03-06T07:29:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.855+0000 7fa22388d640 1 -- 172.21.15.1:0/1967540106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa21c0c6730 msgr2=0x7fa21c0c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.855+0000 7fa22388d640 1 --2- 172.21.15.1:0/1967540106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa21c0c6730 0x7fa21c0c57d0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fa20c009870 tx=0x7fa20c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.855+0000 7fa22388d640 1 -- 172.21.15.1:0/1967540106 shutdown_connections 2024-03-06T07:29:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.855+0000 7fa22388d640 1 --2- 172.21.15.1:0/1967540106 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa21c0c5d10 0x7fa21c156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.855+0000 7fa22388d640 1 --2- 172.21.15.1:0/1967540106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa21c0c6730 0x7fa21c0c57d0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.855+0000 7fa22388d640 1 -- 172.21.15.1:0/1967540106 >> 172.21.15.1:0/1967540106 conn(0x7fa21c06b410 msgr2=0x7fa21c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.855+0000 7fa22388d640 1 -- 172.21.15.1:0/1967540106 shutdown_connections 2024-03-06T07:29:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.855+0000 7fa22388d640 1 -- 172.21.15.1:0/1967540106 wait complete. 2024-03-06T07:29:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.856+0000 7fa22388d640 1 Processor -- start 2024-03-06T07:29:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.857+0000 7fa22388d640 1 -- start start 2024-03-06T07:29:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.857+0000 7fa22388d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa21c0c5d10 0x7fa21c0abc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.857+0000 7fa22388d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa21c0c6730 0x7fa21c0a8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.857+0000 7fa22388d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa21c152690 0x7fa21c0ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.857+0000 7fa22388d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c152bf0 con 0x7fa21c0c5d10 2024-03-06T07:29:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.857+0000 7fa22388d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c152a70 con 0x7fa21c152690 2024-03-06T07:29:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.857+0000 7fa22388d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c157210 con 0x7fa21c0c6730 2024-03-06T07:29:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.857+0000 7fa221629640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa21c0c5d10 0x7fa21c0abc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.857+0000 7fa221629640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa21c0c5d10 0x7fa21c0abc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59814/0 (socket says 172.21.15.1:59814) 2024-03-06T07:29:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.857+0000 7fa221629640 1 -- 172.21.15.1:0/952611335 learned_addr learned my addr 172.21.15.1:0/952611335 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.858+0000 7fa221e2a640 1 --2- 172.21.15.1:0/952611335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa21c152690 0x7fa21c0ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.858+0000 7fa220e28640 1 --2- 172.21.15.1:0/952611335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa21c0c6730 0x7fa21c0a8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.858+0000 7fa221629640 1 -- 172.21.15.1:0/952611335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa21c0c6730 msgr2=0x7fa21c0a8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.858+0000 7fa221629640 1 --2- 172.21.15.1:0/952611335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa21c0c6730 0x7fa21c0a8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.858+0000 7fa221629640 1 -- 172.21.15.1:0/952611335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa21c152690 msgr2=0x7fa21c0ac2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.858+0000 7fa221629640 1 --2- 172.21.15.1:0/952611335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa21c152690 0x7fa21c0ac2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.858+0000 7fa221629640 1 -- 172.21.15.1:0/952611335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c0c8d20 con 0x7fa21c0c5d10 2024-03-06T07:29:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.858+0000 7fa220e28640 1 --2- 172.21.15.1:0/952611335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa21c0c6730 0x7fa21c0a8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.858+0000 7fa221e2a640 1 --2- 172.21.15.1:0/952611335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa21c152690 0x7fa21c0ac2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.858+0000 7fa221629640 1 --2- 172.21.15.1:0/952611335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa21c0c5d10 0x7fa21c0abc70 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7fa21800c430 tx=0x7fa21800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.858+0000 7fa211ffb640 1 -- 172.21.15.1:0/952611335 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2180074e0 con 0x7fa21c0c5d10 2024-03-06T07:29:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.858+0000 7fa211ffb640 1 -- 172.21.15.1:0/952611335 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa218007680 con 0x7fa21c0c5d10 2024-03-06T07:29:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.858+0000 7fa22388d640 1 -- 172.21.15.1:0/952611335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa21c1e2150 con 0x7fa21c0c5d10 2024-03-06T07:29:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.858+0000 7fa211ffb640 1 -- 172.21.15.1:0/952611335 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa218004a00 con 0x7fa21c0c5d10 2024-03-06T07:29:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.858+0000 7fa22388d640 1 -- 172.21.15.1:0/952611335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa21c0ab750 con 0x7fa21c0c5d10 2024-03-06T07:29:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.859+0000 7fa22388d640 1 -- 172.21.15.1:0/952611335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa21c067fd0 con 0x7fa21c0c5d10 2024-03-06T07:29:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.860+0000 7fa211ffb640 1 -- 172.21.15.1:0/952611335 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa21801a020 con 0x7fa21c0c5d10 2024-03-06T07:29:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.860+0000 7fa211ffb640 1 --2- 172.21.15.1:0/952611335 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa20006b4d0 0x7fa20006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.860+0000 7fa211ffb640 1 -- 172.21.15.1:0/952611335 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa218089b70 con 0x7fa21c0c5d10 2024-03-06T07:29:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.860+0000 7fa220e28640 1 --2- 172.21.15.1:0/952611335 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa20006b4d0 0x7fa20006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.861+0000 7fa220e28640 1 --2- 172.21.15.1:0/952611335 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa20006b4d0 0x7fa20006d990 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fa20c036000 tx=0x7fa20c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:06.864+0000 7fa211ffb640 1 -- 172.21.15.1:0/952611335 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa218058870 con 0x7fa21c0c5d10 2024-03-06T07:29:07.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:07.086+0000 7fa22388d640 1 -- 172.21.15.1:0/952611335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa21c0c2c40 con 0x7fa21c0c5d10 2024-03-06T07:29:07.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:07.086+0000 7fa211ffb640 1 -- 172.21.15.1:0/952611335 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fa218058170 con 0x7fa21c0c5d10 2024-03-06T07:29:07.088 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:07.089 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:29:07.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:07.088+0000 7fa22388d640 1 -- 172.21.15.1:0/952611335 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa20006b4d0 msgr2=0x7fa20006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:07.088+0000 7fa22388d640 1 --2- 172.21.15.1:0/952611335 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa20006b4d0 0x7fa20006d990 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fa20c036000 tx=0x7fa20c008030 comp rx=0 tx=0).stop 2024-03-06T07:29:07.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:07.088+0000 7fa22388d640 1 -- 172.21.15.1:0/952611335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa21c0c5d10 msgr2=0x7fa21c0abc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:07.088+0000 7fa22388d640 1 --2- 172.21.15.1:0/952611335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa21c0c5d10 0x7fa21c0abc70 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7fa21800c430 tx=0x7fa21800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:07.088+0000 7fa22388d640 1 -- 172.21.15.1:0/952611335 shutdown_connections 2024-03-06T07:29:07.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:07.088+0000 7fa22388d640 1 --2- 172.21.15.1:0/952611335 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa20006b4d0 0x7fa20006d990 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:07.088+0000 7fa22388d640 1 --2- 172.21.15.1:0/952611335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa21c152690 0x7fa21c0ac2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:07.088+0000 7fa22388d640 1 --2- 172.21.15.1:0/952611335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa21c0c6730 0x7fa21c0a8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:07.088+0000 7fa22388d640 1 --2- 172.21.15.1:0/952611335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa21c0c5d10 0x7fa21c0abc70 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:07.088+0000 7fa22388d640 1 -- 172.21.15.1:0/952611335 >> 172.21.15.1:0/952611335 conn(0x7fa21c06b410 msgr2=0x7fa21c156050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:07.088+0000 7fa22388d640 1 -- 172.21.15.1:0/952611335 shutdown_connections 2024-03-06T07:29:07.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:07.088+0000 7fa22388d640 1 -- 172.21.15.1:0/952611335 wait complete. 2024-03-06T07:29:07.101 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-03-06T07:29:07.101 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:29:12.101 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:29:12.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.212+0000 7f5acb91e640 1 Processor -- start 2024-03-06T07:29:12.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.212+0000 7f5acb91e640 1 -- start start 2024-03-06T07:29:12.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.213+0000 7f5acb91e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ac4072bd0 0x7f5ac40c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.213+0000 7f5acb91e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5ac4152b70 0x7f5ac40c3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.213+0000 7f5acb91e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac4151dd0 con 0x7f5ac40c2b30 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.213+0000 7f5acb91e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac41526e0 con 0x7f5ac4152b70 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.213+0000 7f5acb91e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac4156190 con 0x7f5ac4072bd0 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.213+0000 7f5ac8eb9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ac4072bd0 0x7f5ac40c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.213+0000 7f5ac96ba640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5ac40c2b30 0x7f5ac40724c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:47980/0 (socket says 172.21.15.1:47980) 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.213+0000 7f5ac9ebb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5ac4152b70 0x7f5ac40c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.213+0000 7f5ac96ba640 1 -- 172.21.15.1:0/1462938408 learned_addr learned my addr 172.21.15.1:0/1462938408 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.214+0000 7f5ac3fff640 1 -- 172.21.15.1:0/1462938408 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496998693 0 0) 0x7f5ac4151dd0 con 0x7f5ac40c2b30 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.214+0000 7f5ac3fff640 1 -- 172.21.15.1:0/1462938408 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aac003520 con 0x7f5ac40c2b30 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.214+0000 7f5ac9ebb640 1 -- 172.21.15.1:0/1462938408 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ac4072bd0 msgr2=0x7f5ac40c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.214+0000 7f5ac9ebb640 1 --2- 172.21.15.1:0/1462938408 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ac4072bd0 0x7f5ac40c32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.214+0000 7f5ac9ebb640 1 -- 172.21.15.1:0/1462938408 >> v1:172.21.15.98:6789/0 conn(0x7f5ac40c2b30 legacy=0x7f5ac40724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.214+0000 7f5ac9ebb640 1 -- 172.21.15.1:0/1462938408 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac4152090 con 0x7f5ac4152b70 2024-03-06T07:29:12.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.214+0000 7f5ac8eb9640 1 --2- 172.21.15.1:0/1462938408 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ac4072bd0 0x7f5ac40c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:12.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.214+0000 7f5ac9ebb640 1 --2- 172.21.15.1:0/1462938408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac4152b70 0x7f5ac40c3830 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f5abc009d90 tx=0x7f5abc02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca48c90e37efa807 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.215+0000 7f5ac3fff640 1 -- 172.21.15.1:0/1462938408 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc039070 con 0x7f5ac4152b70 2024-03-06T07:29:12.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.215+0000 7f5ac3fff640 1 -- 172.21.15.1:0/1462938408 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5abc02cc50 con 0x7f5ac4152b70 2024-03-06T07:29:12.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.215+0000 7f5ac3fff640 1 -- 172.21.15.1:0/1462938408 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc034440 con 0x7f5ac4152b70 2024-03-06T07:29:12.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.215+0000 7f5acb91e640 1 -- 172.21.15.1:0/1462938408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac4152b70 msgr2=0x7f5ac40c3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.215+0000 7f5acb91e640 1 --2- 172.21.15.1:0/1462938408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac4152b70 0x7f5ac40c3830 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f5abc009d90 tx=0x7f5abc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.215+0000 7f5acb91e640 1 -- 172.21.15.1:0/1462938408 shutdown_connections 2024-03-06T07:29:12.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.215+0000 7f5acb91e640 1 --2- 172.21.15.1:0/1462938408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac4152b70 0x7f5ac40c3830 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.215+0000 7f5acb91e640 1 --2- 172.21.15.1:0/1462938408 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ac4072bd0 0x7f5ac40c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.215+0000 7f5acb91e640 1 -- 172.21.15.1:0/1462938408 >> 172.21.15.1:0/1462938408 conn(0x7f5ac40c6860 msgr2=0x7f5ac40c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.215+0000 7f5acb91e640 1 -- 172.21.15.1:0/1462938408 shutdown_connections 2024-03-06T07:29:12.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.215+0000 7f5acb91e640 1 -- 172.21.15.1:0/1462938408 wait complete. 2024-03-06T07:29:12.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.216+0000 7f5acb91e640 1 Processor -- start 2024-03-06T07:29:12.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.216+0000 7f5acb91e640 1 -- start start 2024-03-06T07:29:12.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.217+0000 7f5acb91e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ac4072bd0 0x7f5ac40bc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.217+0000 7f5acb91e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ac40c2b30 0x7f5ac41eac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.217+0000 7f5acb91e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac4152b70 0x7f5ac4153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.217+0000 7f5acb91e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac40c3ef0 con 0x7f5ac4072bd0 2024-03-06T07:29:12.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.217+0000 7f5acb91e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac40c3d70 con 0x7f5ac4152b70 2024-03-06T07:29:12.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.217+0000 7f5acb91e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac4155d80 con 0x7f5ac40c2b30 2024-03-06T07:29:12.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.217+0000 7f5ac96ba640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ac4072bd0 0x7f5ac40bc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.217+0000 7f5ac96ba640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ac4072bd0 0x7f5ac40bc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59830/0 (socket says 172.21.15.1:59830) 2024-03-06T07:29:12.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.217+0000 7f5ac96ba640 1 -- 172.21.15.1:0/1850085107 learned_addr learned my addr 172.21.15.1:0/1850085107 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:12.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.217+0000 7f5ac8eb9640 1 --2- 172.21.15.1:0/1850085107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ac40c2b30 0x7f5ac41eac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.217+0000 7f5ac9ebb640 1 --2- 172.21.15.1:0/1850085107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac4152b70 0x7f5ac4153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.218+0000 7f5ac96ba640 1 -- 172.21.15.1:0/1850085107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ac40c2b30 msgr2=0x7f5ac41eac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.218+0000 7f5ac96ba640 1 --2- 172.21.15.1:0/1850085107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ac40c2b30 0x7f5ac41eac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.218+0000 7f5ac96ba640 1 -- 172.21.15.1:0/1850085107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac4152b70 msgr2=0x7f5ac4153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.218+0000 7f5ac96ba640 1 --2- 172.21.15.1:0/1850085107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac4152b70 0x7f5ac4153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.218+0000 7f5ac96ba640 1 -- 172.21.15.1:0/1850085107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac4154840 con 0x7f5ac4072bd0 2024-03-06T07:29:12.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.218+0000 7f5ac96ba640 1 --2- 172.21.15.1:0/1850085107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ac4072bd0 0x7f5ac40bc390 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f5ab400c430 tx=0x7f5ab400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.218+0000 7f5ac1ffb640 1 -- 172.21.15.1:0/1850085107 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab4017070 con 0x7f5ac4072bd0 2024-03-06T07:29:12.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.218+0000 7f5ac1ffb640 1 -- 172.21.15.1:0/1850085107 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab4005ce0 con 0x7f5ac4072bd0 2024-03-06T07:29:12.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.218+0000 7f5ac1ffb640 1 -- 172.21.15.1:0/1850085107 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40072f0 con 0x7f5ac4072bd0 2024-03-06T07:29:12.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.218+0000 7f5acb91e640 1 -- 172.21.15.1:0/1850085107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac41eb160 con 0x7f5ac4072bd0 2024-03-06T07:29:12.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.218+0000 7f5acb91e640 1 -- 172.21.15.1:0/1850085107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac40bcd20 con 0x7f5ac4072bd0 2024-03-06T07:29:12.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.219+0000 7f5acb91e640 1 -- 172.21.15.1:0/1850085107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac4067fd0 con 0x7f5ac4072bd0 2024-03-06T07:29:12.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.220+0000 7f5ac1ffb640 1 -- 172.21.15.1:0/1850085107 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ab401a070 con 0x7f5ac4072bd0 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.220+0000 7f5ac1ffb640 1 --2- 172.21.15.1:0/1850085107 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ab006b4d0 0x7f5ab006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.220+0000 7f5ac1ffb640 1 -- 172.21.15.1:0/1850085107 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f5ab40896f0 con 0x7f5ac4072bd0 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.220+0000 7f5ac8eb9640 1 --2- 172.21.15.1:0/1850085107 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ab006b4d0 0x7f5ab006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.221+0000 7f5ac8eb9640 1 --2- 172.21.15.1:0/1850085107 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ab006b4d0 0x7f5ab006d990 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800b800 tx=0x7f5ab800b3a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.225+0000 7f5ac1ffb640 1 -- 172.21.15.1:0/1850085107 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ab4057df0 con 0x7f5ac4072bd0 2024-03-06T07:29:12.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.443+0000 7f5acb91e640 1 -- 172.21.15.1:0/1850085107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5ac406f460 con 0x7f5ac4072bd0 2024-03-06T07:29:12.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.443+0000 7f5ac1ffb640 1 -- 172.21.15.1:0/1850085107 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f5ab405be90 con 0x7f5ac4072bd0 2024-03-06T07:29:12.446 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:12.446 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:29:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.446+0000 7f5acb91e640 1 -- 172.21.15.1:0/1850085107 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ab006b4d0 msgr2=0x7f5ab006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.446+0000 7f5acb91e640 1 --2- 172.21.15.1:0/1850085107 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ab006b4d0 0x7f5ab006d990 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800b800 tx=0x7f5ab800b3a0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.446+0000 7f5acb91e640 1 -- 172.21.15.1:0/1850085107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ac4072bd0 msgr2=0x7f5ac40bc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.446+0000 7f5acb91e640 1 --2- 172.21.15.1:0/1850085107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ac4072bd0 0x7f5ac40bc390 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f5ab400c430 tx=0x7f5ab400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.446+0000 7f5acb91e640 1 -- 172.21.15.1:0/1850085107 shutdown_connections 2024-03-06T07:29:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.446+0000 7f5acb91e640 1 --2- 172.21.15.1:0/1850085107 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5ab006b4d0 0x7f5ab006d990 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.446+0000 7f5acb91e640 1 --2- 172.21.15.1:0/1850085107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac4152b70 0x7f5ac4153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.446+0000 7f5acb91e640 1 --2- 172.21.15.1:0/1850085107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ac40c2b30 0x7f5ac41eac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.446+0000 7f5acb91e640 1 --2- 172.21.15.1:0/1850085107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ac4072bd0 0x7f5ac40bc390 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.446+0000 7f5acb91e640 1 -- 172.21.15.1:0/1850085107 >> 172.21.15.1:0/1850085107 conn(0x7f5ac40c6860 msgr2=0x7f5ac40c3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.446+0000 7f5acb91e640 1 -- 172.21.15.1:0/1850085107 shutdown_connections 2024-03-06T07:29:12.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:12.446+0000 7f5acb91e640 1 -- 172.21.15.1:0/1850085107 wait complete. 2024-03-06T07:29:12.458 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-03-06T07:29:12.458 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:29:17.459 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:29:17.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.571+0000 7f8f01802640 1 Processor -- start 2024-03-06T07:29:17.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.571+0000 7f8f01802640 1 -- start start 2024-03-06T07:29:17.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.571+0000 7f8f01802640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc0c1740 0x7f8efc155990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.571+0000 7f8f01802640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8efc0c1bd0 con 0x7f8efc1525b0 2024-03-06T07:29:17.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.571+0000 7f8f01802640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8efc1588a0 con 0x7f8efc0c12b0 2024-03-06T07:29:17.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.571+0000 7f8f01802640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc158650 con 0x7f8efc0c1740 2024-03-06T07:29:17.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.571+0000 7f8efaffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8efc1525b0 0x7f8efc0c69b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48210/0 (socket says 172.21.15.1:48210) 2024-03-06T07:29:17.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.571+0000 7f8efaffd640 1 -- 172.21.15.1:0/219055954 learned_addr learned my addr 172.21.15.1:0/219055954 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:17.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efb7fe640 1 --2- 172.21.15.1:0/219055954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc0c1740 0x7f8efc155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efa7fc640 1 -- 172.21.15.1:0/219055954 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849984665 0 0) 0x7f8efc0c1bd0 con 0x7f8efc1525b0 2024-03-06T07:29:17.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efa7fc640 1 -- 172.21.15.1:0/219055954 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee8003520 con 0x7f8efc1525b0 2024-03-06T07:29:17.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efa7fc640 1 -- 172.21.15.1:0/219055954 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287729556 0 0) 0x7f8efc1588a0 con 0x7f8efc0c12b0 2024-03-06T07:29:17.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efa7fc640 1 -- 172.21.15.1:0/219055954 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8efc0c1bd0 con 0x7f8efc0c12b0 2024-03-06T07:29:17.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efa7fc640 1 -- 172.21.15.1:0/219055954 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 497579283 0 0) 0x7f8ee8003520 con 0x7f8efc1525b0 2024-03-06T07:29:17.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efa7fc640 1 -- 172.21.15.1:0/219055954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc0c1740 msgr2=0x7f8efc155990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:17.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efa7fc640 1 --2- 172.21.15.1:0/219055954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc0c1740 0x7f8efc155990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efa7fc640 1 -- 172.21.15.1:0/219055954 >> v1:172.21.15.98:6789/0 conn(0x7f8efc0c12b0 legacy=0x7f8efc155280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efa7fc640 1 -- 172.21.15.1:0/219055954 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc158b80 con 0x7f8efc1525b0 2024-03-06T07:29:17.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efa7fc640 1 -- 172.21.15.1:0/219055954 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f8eec001670 con 0x7f8efc1525b0 2024-03-06T07:29:17.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efa7fc640 1 -- 172.21.15.1:0/219055954 >> v1:172.21.15.1:6789/0 conn(0x7f8efc1525b0 legacy=0x7f8efc0c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efa7fc640 1 --2- 172.21.15.1:0/219055954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc0c1740 0x7f8ee8003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efa7fc640 1 --2- 172.21.15.1:0/219055954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ee8004120 0x7f8ee8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efa7fc640 1 --2- 172.21.15.1:0/219055954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee8004d40 0x7f8ee80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efa7fc640 1 -- 172.21.15.1:0/219055954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc158650 con 0x7f8ee8004120 2024-03-06T07:29:17.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efa7fc640 1 -- 172.21.15.1:0/219055954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee800bb80 con 0x7f8ee8004d40 2024-03-06T07:29:17.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efa7fc640 1 -- 172.21.15.1:0/219055954 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee800bd80 con 0x7f8efc0c1740 2024-03-06T07:29:17.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efaffd640 1 --2- 172.21.15.1:0/219055954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc0c1740 0x7f8ee8003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8ef3fff640 1 --2- 172.21.15.1:0/219055954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ee8004120 0x7f8ee8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.572+0000 7f8efb7fe640 1 --2- 172.21.15.1:0/219055954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee8004d40 0x7f8ee80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.573+0000 7f8efaffd640 1 -- 172.21.15.1:0/219055954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee8004d40 msgr2=0x7f8ee80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.573+0000 7f8efaffd640 1 --2- 172.21.15.1:0/219055954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee8004d40 0x7f8ee80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.573+0000 7f8efaffd640 1 -- 172.21.15.1:0/219055954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ee8004120 msgr2=0x7f8ee8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.573+0000 7f8efaffd640 1 --2- 172.21.15.1:0/219055954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ee8004120 0x7f8ee8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.573+0000 7f8efaffd640 1 -- 172.21.15.1:0/219055954 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee8003520 con 0x7f8efc0c1740 2024-03-06T07:29:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.573+0000 7f8ef3fff640 1 --2- 172.21.15.1:0/219055954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ee8004120 0x7f8ee8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.573+0000 7f8efaffd640 1 --2- 172.21.15.1:0/219055954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc0c1740 0x7f8ee8003b70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8eec007fc0 tx=0x7f8ee8005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dec9922b8c18d77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.574+0000 7f8efa7fc640 1 -- 172.21.15.1:0/219055954 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec007be0 con 0x7f8efc0c1740 2024-03-06T07:29:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.574+0000 7f8efa7fc640 1 -- 172.21.15.1:0/219055954 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eec030070 con 0x7f8efc0c1740 2024-03-06T07:29:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.574+0000 7f8efa7fc640 1 -- 172.21.15.1:0/219055954 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec005d70 con 0x7f8efc0c1740 2024-03-06T07:29:17.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.574+0000 7f8f01802640 1 -- 172.21.15.1:0/219055954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc0c1740 msgr2=0x7f8ee8003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.574+0000 7f8f01802640 1 --2- 172.21.15.1:0/219055954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc0c1740 0x7f8ee8003b70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8eec007fc0 tx=0x7f8ee8005830 comp rx=0 tx=0).stop 2024-03-06T07:29:17.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.574+0000 7f8efb7fe640 1 -- 172.21.15.1:0/219055954 reap_dead start 2024-03-06T07:29:17.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.574+0000 7f8f01802640 1 -- 172.21.15.1:0/219055954 shutdown_connections 2024-03-06T07:29:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.574+0000 7f8f01802640 1 -- 172.21.15.1:0/219055954 >> 172.21.15.1:0/219055954 conn(0x7f8efc0674c0 msgr2=0x7f8efc06d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.574+0000 7f8f01802640 1 -- 172.21.15.1:0/219055954 shutdown_connections 2024-03-06T07:29:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.574+0000 7f8f01802640 1 -- 172.21.15.1:0/219055954 wait complete. 2024-03-06T07:29:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.575+0000 7f8f01802640 1 Processor -- start 2024-03-06T07:29:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.576+0000 7f8f01802640 1 -- start start 2024-03-06T07:29:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.576+0000 7f8f01802640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc0c1740 0x7f8efc1eb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.576+0000 7f8f01802640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8efc0c8730 0x7f8efc14d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.576+0000 7f8f01802640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8efc14ae00 0x7f8efc14b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.576+0000 7f8f01802640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0c70c0 con 0x7f8efc0c8730 2024-03-06T07:29:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.576+0000 7f8f01802640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc159140 con 0x7f8efc14ae00 2024-03-06T07:29:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.576+0000 7f8f01802640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0c7240 con 0x7f8efc0c1740 2024-03-06T07:29:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.576+0000 7f8ef3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8efc0c8730 0x7f8efc14d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.576+0000 7f8ef3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8efc0c8730 0x7f8efc14d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57264/0 (socket says 172.21.15.1:57264) 2024-03-06T07:29:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.576+0000 7f8efaffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc0c1740 0x7f8efc1eb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.576+0000 7f8ef3fff640 1 -- 172.21.15.1:0/4177510799 learned_addr learned my addr 172.21.15.1:0/4177510799 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.576+0000 7f8efb7fe640 1 --2- 172.21.15.1:0/4177510799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8efc14ae00 0x7f8efc14b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.577+0000 7f8ef3fff640 1 -- 172.21.15.1:0/4177510799 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc0c1740 msgr2=0x7f8efc1eb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.577+0000 7f8ef3fff640 1 --2- 172.21.15.1:0/4177510799 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc0c1740 0x7f8efc1eb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.577+0000 7f8ef3fff640 1 -- 172.21.15.1:0/4177510799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8efc14ae00 msgr2=0x7f8efc14b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.577+0000 7f8ef3fff640 1 --2- 172.21.15.1:0/4177510799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8efc14ae00 0x7f8efc14b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.577+0000 7f8ef3fff640 1 -- 172.21.15.1:0/4177510799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc14b7d0 con 0x7f8efc0c8730 2024-03-06T07:29:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.577+0000 7f8ef3fff640 1 --2- 172.21.15.1:0/4177510799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8efc0c8730 0x7f8efc14d840 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40079a0 tx=0x7f8ee400ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.577+0000 7f8ef8ff9640 1 -- 172.21.15.1:0/4177510799 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee40052e0 con 0x7f8efc0c8730 2024-03-06T07:29:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.577+0000 7f8ef8ff9640 1 -- 172.21.15.1:0/4177510799 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee4018070 con 0x7f8efc0c8730 2024-03-06T07:29:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.577+0000 7f8f01802640 1 -- 172.21.15.1:0/4177510799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8efc1549c0 con 0x7f8efc0c8730 2024-03-06T07:29:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.577+0000 7f8ef8ff9640 1 -- 172.21.15.1:0/4177510799 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee400da10 con 0x7f8efc0c8730 2024-03-06T07:29:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.577+0000 7f8f01802640 1 -- 172.21.15.1:0/4177510799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8efc0c3db0 con 0x7f8efc0c8730 2024-03-06T07:29:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.578+0000 7f8f01802640 1 -- 172.21.15.1:0/4177510799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8efc067fd0 con 0x7f8efc0c8730 2024-03-06T07:29:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.579+0000 7f8ef8ff9640 1 -- 172.21.15.1:0/4177510799 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ee400dbb0 con 0x7f8efc0c8730 2024-03-06T07:29:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.579+0000 7f8ef8ff9640 1 --2- 172.21.15.1:0/4177510799 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8edc06b4d0 0x7f8edc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.579+0000 7f8ef8ff9640 1 -- 172.21.15.1:0/4177510799 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f8ee408a350 con 0x7f8efc0c8730 2024-03-06T07:29:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.579+0000 7f8efaffd640 1 --2- 172.21.15.1:0/4177510799 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8edc06b4d0 0x7f8edc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.580+0000 7f8efaffd640 1 --2- 172.21.15.1:0/4177510799 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8edc06b4d0 0x7f8edc06d990 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f8eec034000 tx=0x7f8eec009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.584+0000 7f8ef8ff9640 1 -- 172.21.15.1:0/4177510799 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ee4058a50 con 0x7f8efc0c8730 2024-03-06T07:29:17.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.808+0000 7f8f01802640 1 -- 172.21.15.1:0/4177510799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8efc0c6790 con 0x7f8efc0c8730 2024-03-06T07:29:17.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.808+0000 7f8ef8ff9640 1 -- 172.21.15.1:0/4177510799 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f8ee405caf0 con 0x7f8efc0c8730 2024-03-06T07:29:17.810 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:17.811 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:29:17.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.810+0000 7f8f01802640 1 -- 172.21.15.1:0/4177510799 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8edc06b4d0 msgr2=0x7f8edc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.810+0000 7f8f01802640 1 --2- 172.21.15.1:0/4177510799 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8edc06b4d0 0x7f8edc06d990 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f8eec034000 tx=0x7f8eec009030 comp rx=0 tx=0).stop 2024-03-06T07:29:17.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.810+0000 7f8f01802640 1 -- 172.21.15.1:0/4177510799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8efc0c8730 msgr2=0x7f8efc14d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.810+0000 7f8f01802640 1 --2- 172.21.15.1:0/4177510799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8efc0c8730 0x7f8efc14d840 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40079a0 tx=0x7f8ee400ff70 comp rx=0 tx=0).stop 2024-03-06T07:29:17.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.810+0000 7f8f01802640 1 -- 172.21.15.1:0/4177510799 shutdown_connections 2024-03-06T07:29:17.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.810+0000 7f8f01802640 1 --2- 172.21.15.1:0/4177510799 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8edc06b4d0 0x7f8edc06d990 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.810+0000 7f8f01802640 1 --2- 172.21.15.1:0/4177510799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8efc14ae00 0x7f8efc14b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.810+0000 7f8f01802640 1 --2- 172.21.15.1:0/4177510799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8efc0c8730 0x7f8efc14d840 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.810+0000 7f8f01802640 1 --2- 172.21.15.1:0/4177510799 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8efc0c1740 0x7f8efc1eb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.810+0000 7f8f01802640 1 -- 172.21.15.1:0/4177510799 >> 172.21.15.1:0/4177510799 conn(0x7f8efc0674c0 msgr2=0x7f8efc1559d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.811+0000 7f8f01802640 1 -- 172.21.15.1:0/4177510799 shutdown_connections 2024-03-06T07:29:17.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:17.811+0000 7f8f01802640 1 -- 172.21.15.1:0/4177510799 wait complete. 2024-03-06T07:29:17.823 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-03-06T07:29:17.823 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:29:17.823 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:17.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:17.916 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:29:17.916 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:29:17.917 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:17.931 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:29:17.931 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:29:17.931 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:17.949 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:29:17.949 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:18.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.071+0000 7fe975719640 1 Processor -- start 2024-03-06T07:29:18.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.071+0000 7fe975719640 1 -- start start 2024-03-06T07:29:18.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.072+0000 7fe975719640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe970152690 0x7fe970155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.072+0000 7fe975719640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe9700c6730 0x7fe9700c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.072+0000 7fe975719640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9701561e0 con 0x7fe9700c5d10 2024-03-06T07:29:18.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.072+0000 7fe975719640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9700c3300 con 0x7fe9700c6730 2024-03-06T07:29:18.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.072+0000 7fe975719640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9700c35a0 con 0x7fe970152690 2024-03-06T07:29:18.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.072+0000 7fe967fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe9700c6730 0x7fe9700c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.072+0000 7fe967fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe9700c6730 0x7fe9700c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57274/0 (socket says 172.21.15.1:57274) 2024-03-06T07:29:18.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.072+0000 7fe967fff640 1 -- 172.21.15.1:0/3113602878 learned_addr learned my addr 172.21.15.1:0/3113602878 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:18.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.072+0000 7fe967fff640 1 -- 172.21.15.1:0/3113602878 >> v2:172.21.15.98:3300/0 conn(0x7fe970152690 msgr2=0x7fe970155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.072+0000 7fe967fff640 1 --2- 172.21.15.1:0/3113602878 >> v2:172.21.15.98:3300/0 conn(0x7fe970152690 0x7fe970155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.072+0000 7fe967fff640 1 -- 172.21.15.1:0/3113602878 >> v1:172.21.15.98:6789/0 conn(0x7fe9700c5d10 legacy=0x7fe970156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.072+0000 7fe967fff640 1 -- 172.21.15.1:0/3113602878 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9700c3880 con 0x7fe9700c6730 2024-03-06T07:29:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.073+0000 7fe967fff640 1 --2- 172.21.15.1:0/3113602878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9700c6730 0x7fe9700c57d0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7fe9700ab5f0 tx=0x7fe95802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d9031e33537f15e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.073+0000 7fe96e7fc640 1 -- 172.21.15.1:0/3113602878 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95802e070 con 0x7fe9700c6730 2024-03-06T07:29:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.073+0000 7fe96e7fc640 1 -- 172.21.15.1:0/3113602878 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe958037070 con 0x7fe9700c6730 2024-03-06T07:29:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.073+0000 7fe96e7fc640 1 -- 172.21.15.1:0/3113602878 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe958031a00 con 0x7fe9700c6730 2024-03-06T07:29:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.074+0000 7fe975719640 1 -- 172.21.15.1:0/3113602878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9700c6730 msgr2=0x7fe9700c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.074+0000 7fe975719640 1 --2- 172.21.15.1:0/3113602878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9700c6730 0x7fe9700c57d0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7fe9700ab5f0 tx=0x7fe95802b880 comp rx=0 tx=0).stop 2024-03-06T07:29:18.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.074+0000 7fe975719640 1 -- 172.21.15.1:0/3113602878 shutdown_connections 2024-03-06T07:29:18.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.074+0000 7fe975719640 1 --2- 172.21.15.1:0/3113602878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9700c6730 0x7fe9700c57d0 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.074+0000 7fe975719640 1 --2- 172.21.15.1:0/3113602878 >> v2:172.21.15.98:3300/0 conn(0x7fe970152690 0x7fe970155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.074+0000 7fe975719640 1 -- 172.21.15.1:0/3113602878 >> 172.21.15.1:0/3113602878 conn(0x7fe97006b410 msgr2=0x7fe97006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.074+0000 7fe975719640 1 -- 172.21.15.1:0/3113602878 shutdown_connections 2024-03-06T07:29:18.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.074+0000 7fe975719640 1 -- 172.21.15.1:0/3113602878 wait complete. 2024-03-06T07:29:18.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.075+0000 7fe975719640 1 Processor -- start 2024-03-06T07:29:18.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.075+0000 7fe975719640 1 -- start start 2024-03-06T07:29:18.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.075+0000 7fe975719640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9700c5d10 0x7fe9701e1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.075+0000 7fe975719640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9700c6730 0x7fe9700ac300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.075+0000 7fe975719640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe970152690 0x7fe970151b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.075+0000 7fe975719640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe970152a70 con 0x7fe970152690 2024-03-06T07:29:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.075+0000 7fe975719640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9701567b0 con 0x7fe9700c6730 2024-03-06T07:29:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.075+0000 7fe975719640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe970152bf0 con 0x7fe9700c5d10 2024-03-06T07:29:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.075+0000 7fe96f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe970152690 0x7fe970151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.075+0000 7fe96f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe970152690 0x7fe970151b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57284/0 (socket says 172.21.15.1:57284) 2024-03-06T07:29:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.075+0000 7fe96f7fe640 1 -- 172.21.15.1:0/660055067 learned_addr learned my addr 172.21.15.1:0/660055067 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.076+0000 7fe967fff640 1 --2- 172.21.15.1:0/660055067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9700c6730 0x7fe9700ac300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.076+0000 7fe96effd640 1 --2- 172.21.15.1:0/660055067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9700c5d10 0x7fe9701e1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.076+0000 7fe96f7fe640 1 -- 172.21.15.1:0/660055067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9700c5d10 msgr2=0x7fe9701e1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.076+0000 7fe96f7fe640 1 --2- 172.21.15.1:0/660055067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9700c5d10 0x7fe9701e1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.076+0000 7fe96f7fe640 1 -- 172.21.15.1:0/660055067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9700c6730 msgr2=0x7fe9700ac300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.076+0000 7fe96f7fe640 1 --2- 172.21.15.1:0/660055067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9700c6730 0x7fe9700ac300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.076+0000 7fe96f7fe640 1 -- 172.21.15.1:0/660055067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97006ccf0 con 0x7fe970152690 2024-03-06T07:29:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.076+0000 7fe96f7fe640 1 --2- 172.21.15.1:0/660055067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe970152690 0x7fe970151b40 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fe96800b990 tx=0x7fe96800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.076+0000 7fe9677fe640 1 -- 172.21.15.1:0/660055067 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe968019070 con 0x7fe970152690 2024-03-06T07:29:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.076+0000 7fe9677fe640 1 -- 172.21.15.1:0/660055067 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe96800cba0 con 0x7fe970152690 2024-03-06T07:29:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.076+0000 7fe975719640 1 -- 172.21.15.1:0/660055067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9701e20d0 con 0x7fe970152690 2024-03-06T07:29:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.076+0000 7fe9677fe640 1 -- 172.21.15.1:0/660055067 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe968014440 con 0x7fe970152690 2024-03-06T07:29:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.076+0000 7fe975719640 1 -- 172.21.15.1:0/660055067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9701eb980 con 0x7fe970152690 2024-03-06T07:29:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.077+0000 7fe975719640 1 -- 172.21.15.1:0/660055067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe970067fd0 con 0x7fe970152690 2024-03-06T07:29:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.078+0000 7fe9677fe640 1 -- 172.21.15.1:0/660055067 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe96801c070 con 0x7fe970152690 2024-03-06T07:29:18.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.078+0000 7fe9677fe640 1 --2- 172.21.15.1:0/660055067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe95006b500 0x7fe95006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.078+0000 7fe9677fe640 1 -- 172.21.15.1:0/660055067 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe96808a060 con 0x7fe970152690 2024-03-06T07:29:18.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.078+0000 7fe96effd640 1 --2- 172.21.15.1:0/660055067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe95006b500 0x7fe95006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.079+0000 7fe96effd640 1 --2- 172.21.15.1:0/660055067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe95006b500 0x7fe95006d9c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fe9700ad0a0 tx=0x7fe960008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.083+0000 7fe9677fe640 1 -- 172.21.15.1:0/660055067 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe968058db0 con 0x7fe970152690 2024-03-06T07:29:18.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.303+0000 7fe975719640 1 -- 172.21.15.1:0/660055067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9701596b0 con 0x7fe970152690 2024-03-06T07:29:18.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.304+0000 7fe9677fe640 1 -- 172.21.15.1:0/660055067 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+4962 (secure 0 0 0) 0x7fe9680586b0 con 0x7fe970152690 2024-03-06T07:29:18.306 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:18.306 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":189,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7961,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2819896202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2819896202},{"type":"v1","addr":"172.21.15.98:6837","nonce":2819896202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":8959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/646952744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":646952744},{"type":"v1","addr":"172.21.15.1:6835","nonce":646952744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":8965,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2843229007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2843229007},{"type":"v1","addr":"172.21.15.98:6836","nonce":2843229007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":10635,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/1512189189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1512189189},{"type":"v1","addr":"172.21.15.1:6834","nonce":1512189189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:34.395524+0000","modified":"2024-03-06T07:28:59.966364+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":15}]} 2024-03-06T07:29:18.306 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 189 2024-03-06T07:29:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.306+0000 7fe975719640 1 -- 172.21.15.1:0/660055067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe95006b500 msgr2=0x7fe95006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.306+0000 7fe975719640 1 --2- 172.21.15.1:0/660055067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe95006b500 0x7fe95006d9c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fe9700ad0a0 tx=0x7fe960008040 comp rx=0 tx=0).stop 2024-03-06T07:29:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.306+0000 7fe975719640 1 -- 172.21.15.1:0/660055067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe970152690 msgr2=0x7fe970151b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.306+0000 7fe975719640 1 --2- 172.21.15.1:0/660055067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe970152690 0x7fe970151b40 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fe96800b990 tx=0x7fe96800be50 comp rx=0 tx=0).stop 2024-03-06T07:29:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.306+0000 7fe975719640 1 -- 172.21.15.1:0/660055067 shutdown_connections 2024-03-06T07:29:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.306+0000 7fe975719640 1 --2- 172.21.15.1:0/660055067 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe95006b500 0x7fe95006d9c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.306+0000 7fe975719640 1 --2- 172.21.15.1:0/660055067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe970152690 0x7fe970151b40 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.306+0000 7fe975719640 1 --2- 172.21.15.1:0/660055067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9700c6730 0x7fe9700ac300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.306+0000 7fe975719640 1 --2- 172.21.15.1:0/660055067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe9700c5d10 0x7fe9701e1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.306+0000 7fe975719640 1 -- 172.21.15.1:0/660055067 >> 172.21.15.1:0/660055067 conn(0x7fe97006b410 msgr2=0x7fe9700a9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.306+0000 7fe975719640 1 -- 172.21.15.1:0/660055067 shutdown_connections 2024-03-06T07:29:18.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.306+0000 7fe975719640 1 -- 172.21.15.1:0/660055067 wait complete. 2024-03-06T07:29:18.319 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:18.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.418+0000 7fbc664ed640 1 Processor -- start 2024-03-06T07:29:18.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.418+0000 7fbc664ed640 1 -- start start 2024-03-06T07:29:18.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.419+0000 7fbc664ed640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbc60152730 0x7fbc600c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.419+0000 7fbc664ed640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbc600c1aa0 0x7fbc601552e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.419+0000 7fbc664ed640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc601588e0 con 0x7fbc600c6fd0 2024-03-06T07:29:18.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.419+0000 7fbc664ed640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc60158bc0 con 0x7fbc60152730 2024-03-06T07:29:18.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.419+0000 7fbc664ed640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc60158ea0 con 0x7fbc600c1aa0 2024-03-06T07:29:18.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.419+0000 7fbc5ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbc60152730 0x7fbc600c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.419+0000 7fbc5f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbc600c6fd0 0x7fbc600c1390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48212/0 (socket says 172.21.15.1:48212) 2024-03-06T07:29:18.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.419+0000 7fbc5f7fe640 1 -- 172.21.15.1:0/3595375706 learned_addr learned my addr 172.21.15.1:0/3595375706 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:18.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.419+0000 7fbc5ffff640 1 -- 172.21.15.1:0/3595375706 >> v2:172.21.15.98:3300/0 conn(0x7fbc600c1aa0 msgr2=0x7fbc601552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.419+0000 7fbc5ffff640 1 --2- 172.21.15.1:0/3595375706 >> v2:172.21.15.98:3300/0 conn(0x7fbc600c1aa0 0x7fbc601552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.419+0000 7fbc5ffff640 1 -- 172.21.15.1:0/3595375706 >> v1:172.21.15.1:6789/0 conn(0x7fbc600c6fd0 legacy=0x7fbc600c1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.419+0000 7fbc5ffff640 1 -- 172.21.15.1:0/3595375706 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc60159180 con 0x7fbc60152730 2024-03-06T07:29:18.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.420+0000 7fbc5ffff640 1 --2- 172.21.15.1:0/3595375706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc60152730 0x7fbc600c6a90 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7fbc600b0440 tx=0x7fbc5402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a16bdde6dbd834b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.420+0000 7fbc5effd640 1 -- 172.21.15.1:0/3595375706 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc54038070 con 0x7fbc60152730 2024-03-06T07:29:18.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.420+0000 7fbc5effd640 1 -- 172.21.15.1:0/3595375706 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc54031710 con 0x7fbc60152730 2024-03-06T07:29:18.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.420+0000 7fbc5effd640 1 -- 172.21.15.1:0/3595375706 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc54031ae0 con 0x7fbc60152730 2024-03-06T07:29:18.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.421+0000 7fbc664ed640 1 -- 172.21.15.1:0/3595375706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc60152730 msgr2=0x7fbc600c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.421+0000 7fbc664ed640 1 --2- 172.21.15.1:0/3595375706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc60152730 0x7fbc600c6a90 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7fbc600b0440 tx=0x7fbc5402b7b0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.421+0000 7fbc664ed640 1 -- 172.21.15.1:0/3595375706 shutdown_connections 2024-03-06T07:29:18.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.421+0000 7fbc664ed640 1 --2- 172.21.15.1:0/3595375706 >> v2:172.21.15.98:3300/0 conn(0x7fbc600c1aa0 0x7fbc601552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.421+0000 7fbc664ed640 1 --2- 172.21.15.1:0/3595375706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc60152730 0x7fbc600c6a90 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.421+0000 7fbc664ed640 1 -- 172.21.15.1:0/3595375706 >> 172.21.15.1:0/3595375706 conn(0x7fbc60066d00 msgr2=0x7fbc60067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.421+0000 7fbc664ed640 1 -- 172.21.15.1:0/3595375706 shutdown_connections 2024-03-06T07:29:18.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.421+0000 7fbc664ed640 1 -- 172.21.15.1:0/3595375706 wait complete. 2024-03-06T07:29:18.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.422+0000 7fbc664ed640 1 Processor -- start 2024-03-06T07:29:18.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.422+0000 7fbc664ed640 1 -- start start 2024-03-06T07:29:18.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc664ed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbc600c1aa0 0x7fbc600aa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc664ed640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc600c6fd0 0x7fbc60153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc664ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc60152730 0x7fbc601e00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc664ed640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc60155820 con 0x7fbc600c6fd0 2024-03-06T07:29:18.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc664ed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc60152b10 con 0x7fbc60152730 2024-03-06T07:29:18.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc664ed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc601559a0 con 0x7fbc600c1aa0 2024-03-06T07:29:18.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc5f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc600c6fd0 0x7fbc60153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc5f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc600c6fd0 0x7fbc60153cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57296/0 (socket says 172.21.15.1:57296) 2024-03-06T07:29:18.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc5f7fe640 1 -- 172.21.15.1:0/3172926625 learned_addr learned my addr 172.21.15.1:0/3172926625 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:18.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc5ffff640 1 --2- 172.21.15.1:0/3172926625 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbc600c1aa0 0x7fbc600aa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc64a8a640 1 --2- 172.21.15.1:0/3172926625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc60152730 0x7fbc601e00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc5f7fe640 1 -- 172.21.15.1:0/3172926625 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbc600c1aa0 msgr2=0x7fbc600aa1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc5f7fe640 1 --2- 172.21.15.1:0/3172926625 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbc600c1aa0 0x7fbc600aa1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc5f7fe640 1 -- 172.21.15.1:0/3172926625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc60152730 msgr2=0x7fbc601e00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc5f7fe640 1 --2- 172.21.15.1:0/3172926625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc60152730 0x7fbc601e00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc5f7fe640 1 -- 172.21.15.1:0/3172926625 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc601eb9d0 con 0x7fbc600c6fd0 2024-03-06T07:29:18.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc64a8a640 1 --2- 172.21.15.1:0/3172926625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc60152730 0x7fbc601e00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:18.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc5f7fe640 1 --2- 172.21.15.1:0/3172926625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc600c6fd0 0x7fbc60153cd0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000c400 tx=0x7fbc5000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc5cff9640 1 -- 172.21.15.1:0/3172926625 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc50017070 con 0x7fbc600c6fd0 2024-03-06T07:29:18.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc5cff9640 1 -- 172.21.15.1:0/3172926625 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc50005ce0 con 0x7fbc600c6fd0 2024-03-06T07:29:18.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc664ed640 1 -- 172.21.15.1:0/3172926625 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc601df8c0 con 0x7fbc600c6fd0 2024-03-06T07:29:18.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc5cff9640 1 -- 172.21.15.1:0/3172926625 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc500072f0 con 0x7fbc600c6fd0 2024-03-06T07:29:18.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.423+0000 7fbc664ed640 1 -- 172.21.15.1:0/3172926625 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc60154210 con 0x7fbc600c6fd0 2024-03-06T07:29:18.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.425+0000 7fbc664ed640 1 -- 172.21.15.1:0/3172926625 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc60067fd0 con 0x7fbc600c6fd0 2024-03-06T07:29:18.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.425+0000 7fbc5cff9640 1 -- 172.21.15.1:0/3172926625 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc5001a070 con 0x7fbc600c6fd0 2024-03-06T07:29:18.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.425+0000 7fbc5cff9640 1 --2- 172.21.15.1:0/3172926625 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbc4c06b4d0 0x7fbc4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.426+0000 7fbc5cff9640 1 -- 172.21.15.1:0/3172926625 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fbc500897c0 con 0x7fbc600c6fd0 2024-03-06T07:29:18.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.426+0000 7fbc5ffff640 1 --2- 172.21.15.1:0/3172926625 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbc4c06b4d0 0x7fbc4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.426+0000 7fbc5ffff640 1 --2- 172.21.15.1:0/3172926625 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbc4c06b4d0 0x7fbc4c06d990 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fbc54035000 tx=0x7fbc54008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.430+0000 7fbc5cff9640 1 -- 172.21.15.1:0/3172926625 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc50058470 con 0x7fbc600c6fd0 2024-03-06T07:29:18.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.610+0000 7fbc664ed640 1 -- 172.21.15.1:0/3172926625 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc600c6870 con 0x7fbc600c6fd0 2024-03-06T07:29:18.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.611+0000 7fbc5cff9640 1 -- 172.21.15.1:0/3172926625 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v199) v1 ==== 74+0+16929 (secure 0 0 0) 0x7fbc50057d70 con 0x7fbc600c6fd0 2024-03-06T07:29:18.613 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:18.613 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":199,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:29:11.378105+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:32.660232+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:33.362968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"32.0","mappings":[{"from":6,"to":5}]},{"pgid":"32.33","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:18.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.613+0000 7fbc664ed640 1 -- 172.21.15.1:0/3172926625 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbc4c06b4d0 msgr2=0x7fbc4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.613+0000 7fbc664ed640 1 --2- 172.21.15.1:0/3172926625 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbc4c06b4d0 0x7fbc4c06d990 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fbc54035000 tx=0x7fbc54008030 comp rx=0 tx=0).stop 2024-03-06T07:29:18.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.613+0000 7fbc664ed640 1 -- 172.21.15.1:0/3172926625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc600c6fd0 msgr2=0x7fbc60153cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.613+0000 7fbc664ed640 1 --2- 172.21.15.1:0/3172926625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc600c6fd0 0x7fbc60153cd0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000c400 tx=0x7fbc5000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.613+0000 7fbc664ed640 1 -- 172.21.15.1:0/3172926625 shutdown_connections 2024-03-06T07:29:18.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.613+0000 7fbc664ed640 1 --2- 172.21.15.1:0/3172926625 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbc4c06b4d0 0x7fbc4c06d990 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.613+0000 7fbc664ed640 1 --2- 172.21.15.1:0/3172926625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc60152730 0x7fbc601e00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.613+0000 7fbc664ed640 1 --2- 172.21.15.1:0/3172926625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc600c6fd0 0x7fbc60153cd0 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.613+0000 7fbc664ed640 1 --2- 172.21.15.1:0/3172926625 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbc600c1aa0 0x7fbc600aa1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.613+0000 7fbc664ed640 1 -- 172.21.15.1:0/3172926625 >> 172.21.15.1:0/3172926625 conn(0x7fbc60066d00 msgr2=0x7fbc600ca820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.613+0000 7fbc664ed640 1 -- 172.21.15.1:0/3172926625 shutdown_connections 2024-03-06T07:29:18.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.613+0000 7fbc664ed640 1 -- 172.21.15.1:0/3172926625 wait complete. 2024-03-06T07:29:18.626 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:32.660232+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:33.362968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '197', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:18.626 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:18.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.727+0000 7fe83452c640 1 Processor -- start 2024-03-06T07:29:18.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.727+0000 7fe83452c640 1 -- start start 2024-03-06T07:29:18.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.728+0000 7fe83452c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe82c0c1390 0x7fe82c0c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.728+0000 7fe83452c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe82c1588e0 con 0x7fe82c1552e0 2024-03-06T07:29:18.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.728+0000 7fe83452c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe82c158bc0 con 0x7fe82c152730 2024-03-06T07:29:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.728+0000 7fe83452c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe82c0c1d60 con 0x7fe82c0c1390 2024-03-06T07:29:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.728+0000 7fe831ac7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe82c0c1390 0x7fe82c0c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.728+0000 7fe832ac9640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe82c1552e0 0x7fe82c155700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48226/0 (socket says 172.21.15.1:48226) 2024-03-06T07:29:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.728+0000 7fe832ac9640 1 -- 172.21.15.1:0/2226690041 learned_addr learned my addr 172.21.15.1:0/2226690041 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.728+0000 7fe831ac7640 1 -- 172.21.15.1:0/2226690041 >> v1:172.21.15.98:6789/0 conn(0x7fe82c152730 legacy=0x7fe82c0c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.728+0000 7fe831ac7640 1 -- 172.21.15.1:0/2226690041 >> v1:172.21.15.1:6789/0 conn(0x7fe82c1552e0 legacy=0x7fe82c155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.728+0000 7fe831ac7640 1 -- 172.21.15.1:0/2226690041 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82c158ea0 con 0x7fe82c0c1390 2024-03-06T07:29:18.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.729+0000 7fe831ac7640 1 --2- 172.21.15.1:0/2226690041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe82c0c1390 0x7fe82c0c1820 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0097a0 tx=0x7fe81c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16964f15721d78b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.729+0000 7fe8312c6640 1 -- 172.21.15.1:0/2226690041 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c038070 con 0x7fe82c0c1390 2024-03-06T07:29:18.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.729+0000 7fe8312c6640 1 -- 172.21.15.1:0/2226690041 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81c031700 con 0x7fe82c0c1390 2024-03-06T07:29:18.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.729+0000 7fe8312c6640 1 -- 172.21.15.1:0/2226690041 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c031ad0 con 0x7fe82c0c1390 2024-03-06T07:29:18.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.730+0000 7fe83452c640 1 -- 172.21.15.1:0/2226690041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe82c0c1390 msgr2=0x7fe82c0c1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.730+0000 7fe83452c640 1 --2- 172.21.15.1:0/2226690041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe82c0c1390 0x7fe82c0c1820 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0097a0 tx=0x7fe81c02b7a0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.730+0000 7fe83452c640 1 -- 172.21.15.1:0/2226690041 shutdown_connections 2024-03-06T07:29:18.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.730+0000 7fe83452c640 1 --2- 172.21.15.1:0/2226690041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe82c0c1390 0x7fe82c0c1820 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.730+0000 7fe83452c640 1 -- 172.21.15.1:0/2226690041 >> 172.21.15.1:0/2226690041 conn(0x7fe82c066d00 msgr2=0x7fe82c067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.730+0000 7fe83452c640 1 -- 172.21.15.1:0/2226690041 shutdown_connections 2024-03-06T07:29:18.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.730+0000 7fe83452c640 1 -- 172.21.15.1:0/2226690041 wait complete. 2024-03-06T07:29:18.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.731+0000 7fe83452c640 1 Processor -- start 2024-03-06T07:29:18.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.731+0000 7fe83452c640 1 -- start start 2024-03-06T07:29:18.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe83452c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe82c0c1390 0x7fe82c1ee3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe83452c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe82c152730 0x7fe82c150e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe83452c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe82c1552e0 0x7fe82c0c3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe83452c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c152b10 con 0x7fe82c0c1390 2024-03-06T07:29:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe83452c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c159460 con 0x7fe82c152730 2024-03-06T07:29:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe83452c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c0c71a0 con 0x7fe82c1552e0 2024-03-06T07:29:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe8322c8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe82c0c1390 0x7fe82c1ee3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe8322c8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe82c0c1390 0x7fe82c1ee3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57318/0 (socket says 172.21.15.1:57318) 2024-03-06T07:29:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe8322c8640 1 -- 172.21.15.1:0/745318789 learned_addr learned my addr 172.21.15.1:0/745318789 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe832ac9640 1 --2- 172.21.15.1:0/745318789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe82c1552e0 0x7fe82c0c3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe831ac7640 1 --2- 172.21.15.1:0/745318789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe82c152730 0x7fe82c150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe8322c8640 1 -- 172.21.15.1:0/745318789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe82c1552e0 msgr2=0x7fe82c0c3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe8322c8640 1 --2- 172.21.15.1:0/745318789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe82c1552e0 0x7fe82c0c3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe8322c8640 1 -- 172.21.15.1:0/745318789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe82c152730 msgr2=0x7fe82c150e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe8322c8640 1 --2- 172.21.15.1:0/745318789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe82c152730 0x7fe82c150e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe8322c8640 1 -- 172.21.15.1:0/745318789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82c1ee910 con 0x7fe82c0c1390 2024-03-06T07:29:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe831ac7640 1 --2- 172.21.15.1:0/745318789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe82c152730 0x7fe82c150e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe8322c8640 1 --2- 172.21.15.1:0/745318789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe82c0c1390 0x7fe82c1ee3d0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7fe82400c410 tx=0x7fe82400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe822ffd640 1 -- 172.21.15.1:0/745318789 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe824017070 con 0x7fe82c0c1390 2024-03-06T07:29:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe822ffd640 1 -- 172.21.15.1:0/745318789 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8240035f0 con 0x7fe82c0c1390 2024-03-06T07:29:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe83452c640 1 -- 172.21.15.1:0/745318789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe82c0c96c0 con 0x7fe82c0c1390 2024-03-06T07:29:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe822ffd640 1 -- 172.21.15.1:0/745318789 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8240039e0 con 0x7fe82c0c1390 2024-03-06T07:29:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.732+0000 7fe83452c640 1 -- 172.21.15.1:0/745318789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82c158600 con 0x7fe82c0c1390 2024-03-06T07:29:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.734+0000 7fe83452c640 1 -- 172.21.15.1:0/745318789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe82c067fd0 con 0x7fe82c0c1390 2024-03-06T07:29:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.734+0000 7fe822ffd640 1 -- 172.21.15.1:0/745318789 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe82401a070 con 0x7fe82c0c1390 2024-03-06T07:29:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.734+0000 7fe822ffd640 1 --2- 172.21.15.1:0/745318789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe81006b4d0 0x7fe81006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.734+0000 7fe822ffd640 1 -- 172.21.15.1:0/745318789 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe824089770 con 0x7fe82c0c1390 2024-03-06T07:29:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.735+0000 7fe831ac7640 1 --2- 172.21.15.1:0/745318789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe81006b4d0 0x7fe81006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.735+0000 7fe831ac7640 1 --2- 172.21.15.1:0/745318789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe81006b4d0 0x7fe81006d990 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fe81c035000 tx=0x7fe81c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.739+0000 7fe822ffd640 1 -- 172.21.15.1:0/745318789 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8240584c0 con 0x7fe82c0c1390 2024-03-06T07:29:18.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.918+0000 7fe83452c640 1 -- 172.21.15.1:0/745318789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe82c0c6870 con 0x7fe82c0c1390 2024-03-06T07:29:18.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.919+0000 7fe822ffd640 1 -- 172.21.15.1:0/745318789 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v199) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe8240010c0 con 0x7fe82c0c1390 2024-03-06T07:29:18.921 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:29:18.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.921+0000 7fe83452c640 1 -- 172.21.15.1:0/745318789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe81006b4d0 msgr2=0x7fe81006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.921+0000 7fe83452c640 1 --2- 172.21.15.1:0/745318789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe81006b4d0 0x7fe81006d990 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fe81c035000 tx=0x7fe81c008030 comp rx=0 tx=0).stop 2024-03-06T07:29:18.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.921+0000 7fe83452c640 1 -- 172.21.15.1:0/745318789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe82c0c1390 msgr2=0x7fe82c1ee3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.921+0000 7fe83452c640 1 --2- 172.21.15.1:0/745318789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe82c0c1390 0x7fe82c1ee3d0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7fe82400c410 tx=0x7fe82400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.921+0000 7fe83452c640 1 -- 172.21.15.1:0/745318789 shutdown_connections 2024-03-06T07:29:18.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.921+0000 7fe83452c640 1 --2- 172.21.15.1:0/745318789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe81006b4d0 0x7fe81006d990 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.921+0000 7fe83452c640 1 --2- 172.21.15.1:0/745318789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe82c1552e0 0x7fe82c0c3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.921+0000 7fe83452c640 1 --2- 172.21.15.1:0/745318789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe82c152730 0x7fe82c150e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.921+0000 7fe83452c640 1 --2- 172.21.15.1:0/745318789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe82c0c1390 0x7fe82c1ee3d0 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.921+0000 7fe83452c640 1 -- 172.21.15.1:0/745318789 >> 172.21.15.1:0/745318789 conn(0x7fe82c066d00 msgr2=0x7fe82c0c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.921+0000 7fe83452c640 1 -- 172.21.15.1:0/745318789 shutdown_connections 2024-03-06T07:29:18.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:18.921+0000 7fe83452c640 1 -- 172.21.15.1:0/745318789 wait complete. 2024-03-06T07:29:18.933 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:19.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.035+0000 7f90e13f7640 1 Processor -- start 2024-03-06T07:29:19.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.035+0000 7f90e13f7640 1 -- start start 2024-03-06T07:29:19.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.036+0000 7f90e13f7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f90dc0c2b30 0x7f90dc0724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.036+0000 7f90e13f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90dc152b70 0x7f90dc072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.036+0000 7f90e13f7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f90dc0a5fa0 0x7f90dc0c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.036+0000 7f90e13f7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90dc1526e0 con 0x7f90dc0c2b30 2024-03-06T07:29:19.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.036+0000 7f90e13f7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90dc151dd0 con 0x7f90dc0a5fa0 2024-03-06T07:29:19.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.036+0000 7f90e13f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc156190 con 0x7f90dc152b70 2024-03-06T07:29:19.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.036+0000 7f90daffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f90dc0c2b30 0x7f90dc0724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.036+0000 7f90daffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f90dc0c2b30 0x7f90dc0724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57324/0 (socket says 172.21.15.1:57324) 2024-03-06T07:29:19.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.036+0000 7f90db7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f90dc0a5fa0 0x7f90dc0c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.036+0000 7f90da7fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90dc152b70 0x7f90dc072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.036+0000 7f90daffd640 1 -- 172.21.15.1:0/3981861755 learned_addr learned my addr 172.21.15.1:0/3981861755 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:19.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.037+0000 7f90daffd640 1 -- 172.21.15.1:0/3981861755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90dc152b70 msgr2=0x7f90dc072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.037+0000 7f90daffd640 1 --2- 172.21.15.1:0/3981861755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90dc152b70 0x7f90dc072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.037+0000 7f90daffd640 1 -- 172.21.15.1:0/3981861755 >> v2:172.21.15.98:3300/0 conn(0x7f90dc0a5fa0 msgr2=0x7f90dc0c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.037+0000 7f90daffd640 1 --2- 172.21.15.1:0/3981861755 >> v2:172.21.15.98:3300/0 conn(0x7f90dc0a5fa0 0x7f90dc0c32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.037+0000 7f90daffd640 1 -- 172.21.15.1:0/3981861755 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc152090 con 0x7f90dc0c2b30 2024-03-06T07:29:19.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.037+0000 7f90daffd640 1 --2- 172.21.15.1:0/3981861755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90dc0c2b30 0x7f90dc0724c0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f90dc0c0e20 tx=0x7f90d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc47a7445767a5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.037+0000 7f90d9ffb640 1 -- 172.21.15.1:0/3981861755 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d4035020 con 0x7f90dc0c2b30 2024-03-06T07:29:19.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.037+0000 7f90d9ffb640 1 -- 172.21.15.1:0/3981861755 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d4031870 con 0x7f90dc0c2b30 2024-03-06T07:29:19.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.037+0000 7f90d9ffb640 1 -- 172.21.15.1:0/3981861755 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d4031c40 con 0x7f90dc0c2b30 2024-03-06T07:29:19.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.038+0000 7f90e13f7640 1 -- 172.21.15.1:0/3981861755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90dc0c2b30 msgr2=0x7f90dc0724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.038+0000 7f90e13f7640 1 --2- 172.21.15.1:0/3981861755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90dc0c2b30 0x7f90dc0724c0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f90dc0c0e20 tx=0x7f90d402b910 comp rx=0 tx=0).stop 2024-03-06T07:29:19.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.038+0000 7f90e13f7640 1 -- 172.21.15.1:0/3981861755 shutdown_connections 2024-03-06T07:29:19.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.038+0000 7f90e13f7640 1 --2- 172.21.15.1:0/3981861755 >> v2:172.21.15.98:3300/0 conn(0x7f90dc0a5fa0 0x7f90dc0c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.038+0000 7f90e13f7640 1 --2- 172.21.15.1:0/3981861755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90dc152b70 0x7f90dc072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.038+0000 7f90e13f7640 1 --2- 172.21.15.1:0/3981861755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90dc0c2b30 0x7f90dc0724c0 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.038+0000 7f90e13f7640 1 -- 172.21.15.1:0/3981861755 >> 172.21.15.1:0/3981861755 conn(0x7f90dc0c6860 msgr2=0x7f90dc0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.038+0000 7f90e13f7640 1 -- 172.21.15.1:0/3981861755 shutdown_connections 2024-03-06T07:29:19.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.038+0000 7f90e13f7640 1 -- 172.21.15.1:0/3981861755 wait complete. 2024-03-06T07:29:19.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.039+0000 7f90e13f7640 1 Processor -- start 2024-03-06T07:29:19.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.039+0000 7f90e13f7640 1 -- start start 2024-03-06T07:29:19.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90e13f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90dc0a5fa0 0x7f90dc1eca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90e13f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90dc0c2b30 0x7f90dc1dec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90e13f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90dc152b70 0x7f90dc149540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90e13f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc155f00 con 0x7f90dc0a5fa0 2024-03-06T07:29:19.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90e13f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc155d80 con 0x7f90dc152b70 2024-03-06T07:29:19.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90e13f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc0c3830 con 0x7f90dc0c2b30 2024-03-06T07:29:19.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90daffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90dc0a5fa0 0x7f90dc1eca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90daffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90dc0a5fa0 0x7f90dc1eca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57326/0 (socket says 172.21.15.1:57326) 2024-03-06T07:29:19.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90daffd640 1 -- 172.21.15.1:0/120343906 learned_addr learned my addr 172.21.15.1:0/120343906 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:19.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90da7fc640 1 --2- 172.21.15.1:0/120343906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90dc0c2b30 0x7f90dc1dec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90daffd640 1 -- 172.21.15.1:0/120343906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90dc0c2b30 msgr2=0x7f90dc1dec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90daffd640 1 --2- 172.21.15.1:0/120343906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90dc0c2b30 0x7f90dc1dec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90daffd640 1 -- 172.21.15.1:0/120343906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90dc152b70 msgr2=0x7f90dc149540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90db7fe640 1 --2- 172.21.15.1:0/120343906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90dc152b70 0x7f90dc149540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90daffd640 1 --2- 172.21.15.1:0/120343906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90dc152b70 0x7f90dc149540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90daffd640 1 -- 172.21.15.1:0/120343906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc1e8c30 con 0x7f90dc0a5fa0 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90da7fc640 1 --2- 172.21.15.1:0/120343906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90dc0c2b30 0x7f90dc1dec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.040+0000 7f90db7fe640 1 --2- 172.21.15.1:0/120343906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90dc152b70 0x7f90dc149540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.041+0000 7f90daffd640 1 --2- 172.21.15.1:0/120343906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90dc0a5fa0 0x7f90dc1eca10 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f90d4000c00 tx=0x7f90d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.041+0000 7f90c37fe640 1 -- 172.21.15.1:0/120343906 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d4031e10 con 0x7f90dc0a5fa0 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.041+0000 7f90c37fe640 1 -- 172.21.15.1:0/120343906 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d403ad60 con 0x7f90dc0a5fa0 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.041+0000 7f90e13f7640 1 -- 172.21.15.1:0/120343906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90dc1e8f10 con 0x7f90dc0a5fa0 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.041+0000 7f90c37fe640 1 -- 172.21.15.1:0/120343906 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d4044630 con 0x7f90dc0a5fa0 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.041+0000 7f90e13f7640 1 -- 172.21.15.1:0/120343906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90dc1e94d0 con 0x7f90dc0a5fa0 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.042+0000 7f90e13f7640 1 -- 172.21.15.1:0/120343906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90dc067fd0 con 0x7f90dc0a5fa0 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.042+0000 7f90c37fe640 1 -- 172.21.15.1:0/120343906 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90d404b060 con 0x7f90dc0a5fa0 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.042+0000 7f90c37fe640 1 --2- 172.21.15.1:0/120343906 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90c806b500 0x7f90c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.043+0000 7f90c37fe640 1 -- 172.21.15.1:0/120343906 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f90d40ac300 con 0x7f90dc0a5fa0 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.043+0000 7f90da7fc640 1 --2- 172.21.15.1:0/120343906 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90c806b500 0x7f90c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.043+0000 7f90da7fc640 1 --2- 172.21.15.1:0/120343906 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90c806b500 0x7f90c806d9c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f90dc148e70 tx=0x7f90d0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.047+0000 7f90c37fe640 1 -- 172.21.15.1:0/120343906 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90d403f070 con 0x7f90dc0a5fa0 2024-03-06T07:29:19.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.229+0000 7f90e13f7640 1 -- 172.21.15.1:0/120343906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f90dc071790 con 0x7f90dc0a5fa0 2024-03-06T07:29:19.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.229+0000 7f90c37fe640 1 -- 172.21.15.1:0/120343906 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v199) v1 ==== 104+0+11 (secure 0 0 0) 0x7f90d402f320 con 0x7f90dc0a5fa0 2024-03-06T07:29:19.231 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:29:19.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.231+0000 7f90e13f7640 1 -- 172.21.15.1:0/120343906 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90c806b500 msgr2=0x7f90c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.231+0000 7f90e13f7640 1 --2- 172.21.15.1:0/120343906 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90c806b500 0x7f90c806d9c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f90dc148e70 tx=0x7f90d0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.231+0000 7f90e13f7640 1 -- 172.21.15.1:0/120343906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90dc0a5fa0 msgr2=0x7f90dc1eca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.231+0000 7f90e13f7640 1 --2- 172.21.15.1:0/120343906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90dc0a5fa0 0x7f90dc1eca10 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f90d4000c00 tx=0x7f90d40046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.231+0000 7f90e13f7640 1 -- 172.21.15.1:0/120343906 shutdown_connections 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.231+0000 7f90e13f7640 1 --2- 172.21.15.1:0/120343906 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f90c806b500 0x7f90c806d9c0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.231+0000 7f90e13f7640 1 --2- 172.21.15.1:0/120343906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90dc152b70 0x7f90dc149540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.231+0000 7f90e13f7640 1 --2- 172.21.15.1:0/120343906 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90dc0c2b30 0x7f90dc1dec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.231+0000 7f90e13f7640 1 --2- 172.21.15.1:0/120343906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90dc0a5fa0 0x7f90dc1eca10 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.231+0000 7f90e13f7640 1 -- 172.21.15.1:0/120343906 >> 172.21.15.1:0/120343906 conn(0x7f90dc0c6860 msgr2=0x7f90dc072410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.231+0000 7f90e13f7640 1 -- 172.21.15.1:0/120343906 shutdown_connections 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.231+0000 7f90e13f7640 1 -- 172.21.15.1:0/120343906 wait complete. 2024-03-06T07:29:19.244 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:19.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.347+0000 7feb19078640 1 Processor -- start 2024-03-06T07:29:19.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.347+0000 7feb19078640 1 -- start start 2024-03-06T07:29:19.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.347+0000 7feb19078640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb140c2b30 0x7feb140724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.347+0000 7feb19078640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb14151dd0 con 0x7feb140a5fa0 2024-03-06T07:29:19.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.347+0000 7feb19078640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb14156190 con 0x7feb14152b70 2024-03-06T07:29:19.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.347+0000 7feb19078640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb141526e0 con 0x7feb140c2b30 2024-03-06T07:29:19.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.347+0000 7feb1359e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7feb140a5fa0 0x7feb1415c750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48234/0 (socket says 172.21.15.1:48234) 2024-03-06T07:29:19.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.348+0000 7feb1359e640 1 -- 172.21.15.1:0/3762597030 learned_addr learned my addr 172.21.15.1:0/3762597030 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:19.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.348+0000 7feb12d9d640 1 --2- 172.21.15.1:0/3762597030 >> v2:172.21.15.98:3300/0 conn(0x7feb140c2b30 0x7feb140724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.348+0000 7feb1259c640 1 -- 172.21.15.1:0/3762597030 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227531759 0 0) 0x7feb14151dd0 con 0x7feb140a5fa0 2024-03-06T07:29:19.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.348+0000 7feb1259c640 1 -- 172.21.15.1:0/3762597030 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb00003520 con 0x7feb140a5fa0 2024-03-06T07:29:19.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.348+0000 7feb1259c640 1 -- 172.21.15.1:0/3762597030 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746860822 0 0) 0x7feb14156190 con 0x7feb14152b70 2024-03-06T07:29:19.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.348+0000 7feb1259c640 1 -- 172.21.15.1:0/3762597030 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb14151dd0 con 0x7feb14152b70 2024-03-06T07:29:19.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.348+0000 7feb1259c640 1 -- 172.21.15.1:0/3762597030 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1987576396 0 0) 0x7feb00003520 con 0x7feb140a5fa0 2024-03-06T07:29:19.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.348+0000 7feb12d9d640 1 -- 172.21.15.1:0/3762597030 >> v1:172.21.15.98:6789/0 conn(0x7feb14152b70 legacy=0x7feb140c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.348+0000 7feb12d9d640 1 -- 172.21.15.1:0/3762597030 >> v1:172.21.15.1:6789/0 conn(0x7feb140a5fa0 legacy=0x7feb1415c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.348+0000 7feb12d9d640 1 -- 172.21.15.1:0/3762597030 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb14152090 con 0x7feb140c2b30 2024-03-06T07:29:19.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.349+0000 7feb12d9d640 1 --2- 172.21.15.1:0/3762597030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140c2b30 0x7feb140724c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7feb040097a0 tx=0x7feb0402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93d300123409e006 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.349+0000 7feb1259c640 1 -- 172.21.15.1:0/3762597030 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb04038070 con 0x7feb140c2b30 2024-03-06T07:29:19.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.349+0000 7feb1259c640 1 -- 172.21.15.1:0/3762597030 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb04031500 con 0x7feb140c2b30 2024-03-06T07:29:19.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.349+0000 7feb1259c640 1 -- 172.21.15.1:0/3762597030 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c0019f0 con 0x7feb140c2b30 2024-03-06T07:29:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.350+0000 7feb19078640 1 -- 172.21.15.1:0/3762597030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140c2b30 msgr2=0x7feb140724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.350+0000 7feb19078640 1 --2- 172.21.15.1:0/3762597030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140c2b30 0x7feb140724c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7feb040097a0 tx=0x7feb0402b820 comp rx=0 tx=0).stop 2024-03-06T07:29:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.350+0000 7feb19078640 1 -- 172.21.15.1:0/3762597030 shutdown_connections 2024-03-06T07:29:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.350+0000 7feb19078640 1 --2- 172.21.15.1:0/3762597030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140c2b30 0x7feb140724c0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.350+0000 7feb19078640 1 -- 172.21.15.1:0/3762597030 >> 172.21.15.1:0/3762597030 conn(0x7feb140c6860 msgr2=0x7feb140c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.350+0000 7feb19078640 1 -- 172.21.15.1:0/3762597030 shutdown_connections 2024-03-06T07:29:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.350+0000 7feb19078640 1 -- 172.21.15.1:0/3762597030 wait complete. 2024-03-06T07:29:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.351+0000 7feb19078640 1 Processor -- start 2024-03-06T07:29:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.351+0000 7feb19078640 1 -- start start 2024-03-06T07:29:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb19078640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb140a5fa0 0x7feb141dd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb19078640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140c2b30 0x7feb1414bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb19078640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb14152b70 0x7feb141eb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb19078640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb14155f00 con 0x7feb14152b70 2024-03-06T07:29:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb19078640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb14155d80 con 0x7feb140c2b30 2024-03-06T07:29:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb19078640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb14072a00 con 0x7feb140a5fa0 2024-03-06T07:29:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb1359e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb14152b70 0x7feb141eb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb1359e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb14152b70 0x7feb141eb330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57338/0 (socket says 172.21.15.1:57338) 2024-03-06T07:29:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb1359e640 1 -- 172.21.15.1:0/1212792659 learned_addr learned my addr 172.21.15.1:0/1212792659 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb12d9d640 1 --2- 172.21.15.1:0/1212792659 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb140a5fa0 0x7feb141dd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb0a59c640 1 --2- 172.21.15.1:0/1212792659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140c2b30 0x7feb1414bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb1359e640 1 -- 172.21.15.1:0/1212792659 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb140a5fa0 msgr2=0x7feb141dd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb1359e640 1 --2- 172.21.15.1:0/1212792659 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb140a5fa0 0x7feb141dd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb1359e640 1 -- 172.21.15.1:0/1212792659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140c2b30 msgr2=0x7feb1414bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb1359e640 1 --2- 172.21.15.1:0/1212792659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140c2b30 0x7feb1414bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb1359e640 1 -- 172.21.15.1:0/1212792659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb141eab70 con 0x7feb14152b70 2024-03-06T07:29:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb0a59c640 1 --2- 172.21.15.1:0/1212792659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140c2b30 0x7feb1414bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb1359e640 1 --2- 172.21.15.1:0/1212792659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb14152b70 0x7feb141eb330 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7feb0c001560 tx=0x7feb0c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.352+0000 7feb0bfff640 1 -- 172.21.15.1:0/1212792659 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c012400 con 0x7feb14152b70 2024-03-06T07:29:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.353+0000 7feb0bfff640 1 -- 172.21.15.1:0/1212792659 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0c0125a0 con 0x7feb14152b70 2024-03-06T07:29:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.353+0000 7feb19078640 1 -- 172.21.15.1:0/1212792659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb141536f0 con 0x7feb14152b70 2024-03-06T07:29:19.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.353+0000 7feb0bfff640 1 -- 172.21.15.1:0/1212792659 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c012990 con 0x7feb14152b70 2024-03-06T07:29:19.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.353+0000 7feb19078640 1 -- 172.21.15.1:0/1212792659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb141eae10 con 0x7feb14152b70 2024-03-06T07:29:19.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.353+0000 7feb19078640 1 -- 172.21.15.1:0/1212792659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb141eb010 con 0x7feb14152b70 2024-03-06T07:29:19.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.354+0000 7feb0bfff640 1 -- 172.21.15.1:0/1212792659 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb0c012b30 con 0x7feb14152b70 2024-03-06T07:29:19.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.354+0000 7feb0bfff640 1 --2- 172.21.15.1:0/1212792659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feaf406b4d0 0x7feaf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.354+0000 7feb0bfff640 1 -- 172.21.15.1:0/1212792659 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6159+0+0 (secure 0 0 0) 0x7feb0c08ab80 con 0x7feb14152b70 2024-03-06T07:29:19.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.355+0000 7feb12d9d640 1 --2- 172.21.15.1:0/1212792659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feaf406b4d0 0x7feaf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.355+0000 7feb12d9d640 1 --2- 172.21.15.1:0/1212792659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feaf406b4d0 0x7feaf406d990 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7feb04035000 tx=0x7feb04008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.359+0000 7feb0bfff640 1 -- 172.21.15.1:0/1212792659 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb141eb010 con 0x7feb14152b70 2024-03-06T07:29:19.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.540+0000 7feb19078640 1 -- 172.21.15.1:0/1212792659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feb14067fd0 con 0x7feb14152b70 2024-03-06T07:29:19.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.540+0000 7feb0bfff640 1 -- 172.21.15.1:0/1212792659 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v199) v1 ==== 100+0+11 (secure 0 0 0) 0x7feb0c012e20 con 0x7feb14152b70 2024-03-06T07:29:19.542 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:29:19.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.542+0000 7feb19078640 1 -- 172.21.15.1:0/1212792659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feaf406b4d0 msgr2=0x7feaf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.542+0000 7feb19078640 1 --2- 172.21.15.1:0/1212792659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feaf406b4d0 0x7feaf406d990 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7feb04035000 tx=0x7feb04008030 comp rx=0 tx=0).stop 2024-03-06T07:29:19.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.542+0000 7feb19078640 1 -- 172.21.15.1:0/1212792659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb14152b70 msgr2=0x7feb141eb330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.542+0000 7feb19078640 1 --2- 172.21.15.1:0/1212792659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb14152b70 0x7feb141eb330 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7feb0c001560 tx=0x7feb0c00a430 comp rx=0 tx=0).stop 2024-03-06T07:29:19.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.542+0000 7feb19078640 1 -- 172.21.15.1:0/1212792659 shutdown_connections 2024-03-06T07:29:19.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.542+0000 7feb19078640 1 --2- 172.21.15.1:0/1212792659 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feaf406b4d0 0x7feaf406d990 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.542+0000 7feb19078640 1 --2- 172.21.15.1:0/1212792659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb14152b70 0x7feb141eb330 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.542+0000 7feb19078640 1 --2- 172.21.15.1:0/1212792659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140c2b30 0x7feb1414bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.542+0000 7feb19078640 1 --2- 172.21.15.1:0/1212792659 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb140a5fa0 0x7feb141dd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.542+0000 7feb19078640 1 -- 172.21.15.1:0/1212792659 >> 172.21.15.1:0/1212792659 conn(0x7feb140c6860 msgr2=0x7feb1414fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.542+0000 7feb19078640 1 -- 172.21.15.1:0/1212792659 shutdown_connections 2024-03-06T07:29:19.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.542+0000 7feb19078640 1 -- 172.21.15.1:0/1212792659 wait complete. 2024-03-06T07:29:19.555 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:19.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.654+0000 7f415bd7a640 1 Processor -- start 2024-03-06T07:29:19.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.654+0000 7f415bd7a640 1 -- start start 2024-03-06T07:29:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.655+0000 7f415bd7a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f41541505b0 0x7f41540c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.655+0000 7f415bd7a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415406f5c0 con 0x7f41540c1260 2024-03-06T07:29:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.655+0000 7f415bd7a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4154152f80 con 0x7f41540c16e0 2024-03-06T07:29:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.655+0000 7f415bd7a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4154150990 con 0x7f41541505b0 2024-03-06T07:29:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.655+0000 7f4159315640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f41540c1260 0x7f41541566e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48238/0 (socket says 172.21.15.1:48238) 2024-03-06T07:29:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.655+0000 7f4159315640 1 -- 172.21.15.1:0/2395304416 learned_addr learned my addr 172.21.15.1:0/2395304416 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.655+0000 7f4158b14640 1 -- 172.21.15.1:0/2395304416 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809660666 0 0) 0x7f415406f5c0 con 0x7f41540c1260 2024-03-06T07:29:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.655+0000 7f4158b14640 1 -- 172.21.15.1:0/2395304416 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4140003520 con 0x7f41540c1260 2024-03-06T07:29:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.655+0000 7f4159b16640 1 --2- 172.21.15.1:0/2395304416 >> v2:172.21.15.98:3300/0 conn(0x7f41541505b0 0x7f41540c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.655+0000 7f4158b14640 1 -- 172.21.15.1:0/2395304416 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3468316718 0 0) 0x7f4140003520 con 0x7f41540c1260 2024-03-06T07:29:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.655+0000 7f4158b14640 1 -- 172.21.15.1:0/2395304416 >> v2:172.21.15.98:3300/0 conn(0x7f41541505b0 msgr2=0x7f41540c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.655+0000 7f4158b14640 1 --2- 172.21.15.1:0/2395304416 >> v2:172.21.15.98:3300/0 conn(0x7f41541505b0 0x7f41540c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.655+0000 7f4158b14640 1 -- 172.21.15.1:0/2395304416 >> v1:172.21.15.98:6789/0 conn(0x7f41540c16e0 legacy=0x7f4154156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.656+0000 7f4158b14640 1 -- 172.21.15.1:0/2395304416 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4154153240 con 0x7f41540c1260 2024-03-06T07:29:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.656+0000 7f4158b14640 1 -- 172.21.15.1:0/2395304416 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f4144001670 con 0x7f41540c1260 2024-03-06T07:29:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.656+0000 7f4158b14640 1 -- 172.21.15.1:0/2395304416 >> v1:172.21.15.1:6789/0 conn(0x7f41540c1260 legacy=0x7f41541566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.656+0000 7f4158b14640 1 --2- 172.21.15.1:0/2395304416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41400039b0 0x7f4140003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.656+0000 7f4158b14640 1 --2- 172.21.15.1:0/2395304416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4140004800 0x7f4140004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.656+0000 7f4158b14640 1 --2- 172.21.15.1:0/2395304416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41400052e0 0x7f414000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.656+0000 7f4158b14640 1 -- 172.21.15.1:0/2395304416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4154150990 con 0x7f41400052e0 2024-03-06T07:29:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.656+0000 7f4158b14640 1 -- 172.21.15.1:0/2395304416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414000c1b0 con 0x7f41400039b0 2024-03-06T07:29:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.656+0000 7f4158b14640 1 -- 172.21.15.1:0/2395304416 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f414000c3b0 con 0x7f4140004800 2024-03-06T07:29:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.656+0000 7f415a317640 1 --2- 172.21.15.1:0/2395304416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41400052e0 0x7f414000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.656+0000 7f4159b16640 1 --2- 172.21.15.1:0/2395304416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41400039b0 0x7f4140003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.656+0000 7f4159315640 1 --2- 172.21.15.1:0/2395304416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4140004800 0x7f4140004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.656+0000 7f415a317640 1 -- 172.21.15.1:0/2395304416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4140004800 msgr2=0x7f4140004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.656+0000 7f415a317640 1 --2- 172.21.15.1:0/2395304416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4140004800 0x7f4140004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.656+0000 7f415a317640 1 -- 172.21.15.1:0/2395304416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41400039b0 msgr2=0x7f4140003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.656+0000 7f415a317640 1 --2- 172.21.15.1:0/2395304416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41400039b0 0x7f4140003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.656+0000 7f415a317640 1 -- 172.21.15.1:0/2395304416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4140003520 con 0x7f41400052e0 2024-03-06T07:29:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.657+0000 7f415a317640 1 -- 172.21.15.1:0/2395304416 reap_dead start 2024-03-06T07:29:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.657+0000 7f415a317640 1 --2- 172.21.15.1:0/2395304416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41400052e0 0x7f414000b9c0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f4150001e20 tx=0x7f4140005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d878ba215a2860b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.657+0000 7f4158b14640 1 -- 172.21.15.1:0/2395304416 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150001a70 con 0x7f41400052e0 2024-03-06T07:29:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.657+0000 7f4158b14640 1 -- 172.21.15.1:0/2395304416 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4150003b70 con 0x7f41400052e0 2024-03-06T07:29:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.657+0000 7f4158b14640 1 -- 172.21.15.1:0/2395304416 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41500050d0 con 0x7f41400052e0 2024-03-06T07:29:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.657+0000 7f415bd7a640 1 -- 172.21.15.1:0/2395304416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41400052e0 msgr2=0x7f414000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.657+0000 7f415bd7a640 1 --2- 172.21.15.1:0/2395304416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41400052e0 0x7f414000b9c0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f4150001e20 tx=0x7f4140005830 comp rx=0 tx=0).stop 2024-03-06T07:29:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.658+0000 7f415bd7a640 1 -- 172.21.15.1:0/2395304416 shutdown_connections 2024-03-06T07:29:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.658+0000 7f415bd7a640 1 --2- 172.21.15.1:0/2395304416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41400052e0 0x7f414000b9c0 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.658+0000 7f415bd7a640 1 -- 172.21.15.1:0/2395304416 >> 172.21.15.1:0/2395304416 conn(0x7f41540c6860 msgr2=0x7f41540c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.658+0000 7f415bd7a640 1 -- 172.21.15.1:0/2395304416 shutdown_connections 2024-03-06T07:29:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.658+0000 7f415bd7a640 1 -- 172.21.15.1:0/2395304416 wait complete. 2024-03-06T07:29:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.659+0000 7f415bd7a640 1 Processor -- start 2024-03-06T07:29:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.659+0000 7f415bd7a640 1 -- start start 2024-03-06T07:29:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.659+0000 7f415bd7a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41400052e0 0x7f41541557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.659+0000 7f415bd7a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41540a7870 0x7f41541eaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.659+0000 7f415bd7a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f415414c220 0x7f4154072c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.659+0000 7f415bd7a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4154153ae0 con 0x7f41400052e0 2024-03-06T07:29:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.659+0000 7f415bd7a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4154153800 con 0x7f41540a7870 2024-03-06T07:29:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.659+0000 7f415bd7a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41540c4760 con 0x7f415414c220 2024-03-06T07:29:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.659+0000 7f4159b16640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41400052e0 0x7f41541557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.660+0000 7f4159b16640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41400052e0 0x7f41541557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57356/0 (socket says 172.21.15.1:57356) 2024-03-06T07:29:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.660+0000 7f4159b16640 1 -- 172.21.15.1:0/3688579438 learned_addr learned my addr 172.21.15.1:0/3688579438 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.660+0000 7f415a317640 1 --2- 172.21.15.1:0/3688579438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f415414c220 0x7f4154072c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.660+0000 7f4159315640 1 --2- 172.21.15.1:0/3688579438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41540a7870 0x7f41541eaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.660+0000 7f4159b16640 1 -- 172.21.15.1:0/3688579438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f415414c220 msgr2=0x7f4154072c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.660+0000 7f4159b16640 1 --2- 172.21.15.1:0/3688579438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f415414c220 0x7f4154072c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.660+0000 7f4159b16640 1 -- 172.21.15.1:0/3688579438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41540a7870 msgr2=0x7f41541eaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.660+0000 7f4159b16640 1 --2- 172.21.15.1:0/3688579438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41540a7870 0x7f41541eaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.660+0000 7f4159b16640 1 -- 172.21.15.1:0/3688579438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41541553a0 con 0x7f41400052e0 2024-03-06T07:29:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.660+0000 7f4159b16640 1 --2- 172.21.15.1:0/3688579438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41400052e0 0x7f41541557c0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f414c011910 tx=0x7f414c011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.660+0000 7f414a7fc640 1 -- 172.21.15.1:0/3688579438 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c00fa30 con 0x7f41400052e0 2024-03-06T07:29:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.660+0000 7f414a7fc640 1 -- 172.21.15.1:0/3688579438 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f414c00fbd0 con 0x7f41400052e0 2024-03-06T07:29:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.660+0000 7f415bd7a640 1 -- 172.21.15.1:0/3688579438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415414d280 con 0x7f41400052e0 2024-03-06T07:29:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.660+0000 7f414a7fc640 1 -- 172.21.15.1:0/3688579438 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c014440 con 0x7f41400052e0 2024-03-06T07:29:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.660+0000 7f415bd7a640 1 -- 172.21.15.1:0/3688579438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415414cfa0 con 0x7f41400052e0 2024-03-06T07:29:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.661+0000 7f415bd7a640 1 -- 172.21.15.1:0/3688579438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4154067fd0 con 0x7f41400052e0 2024-03-06T07:29:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.662+0000 7f414a7fc640 1 -- 172.21.15.1:0/3688579438 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f414c012040 con 0x7f41400052e0 2024-03-06T07:29:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.662+0000 7f414a7fc640 1 --2- 172.21.15.1:0/3688579438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f413806b500 0x7f413806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.662+0000 7f414a7fc640 1 -- 172.21.15.1:0/3688579438 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f414c08a830 con 0x7f41400052e0 2024-03-06T07:29:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.662+0000 7f4159315640 1 --2- 172.21.15.1:0/3688579438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f413806b500 0x7f413806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.663+0000 7f4159315640 1 --2- 172.21.15.1:0/3688579438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f413806b500 0x7f413806d9c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f41540aca90 tx=0x7f4144000ed0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.666+0000 7f414a7fc640 1 -- 172.21.15.1:0/3688579438 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f414c019030 con 0x7f41400052e0 2024-03-06T07:29:19.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.889+0000 7f415bd7a640 1 -- 172.21.15.1:0/3688579438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f415406f280 con 0x7f41400052e0 2024-03-06T07:29:19.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.890+0000 7f414a7fc640 1 -- 172.21.15.1:0/3688579438 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+4962 (secure 0 0 0) 0x7f414c0594e0 con 0x7f41400052e0 2024-03-06T07:29:19.892 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:19.892 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":189,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7961,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2819896202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2819896202},{"type":"v1","addr":"172.21.15.98:6837","nonce":2819896202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":8959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/646952744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":646952744},{"type":"v1","addr":"172.21.15.1:6835","nonce":646952744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":8965,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2843229007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2843229007},{"type":"v1","addr":"172.21.15.98:6836","nonce":2843229007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":10635,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/1512189189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1512189189},{"type":"v1","addr":"172.21.15.1:6834","nonce":1512189189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:34.395524+0000","modified":"2024-03-06T07:28:59.966364+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":15}]} 2024-03-06T07:29:19.892 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 189 2024-03-06T07:29:19.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.892+0000 7f415bd7a640 1 -- 172.21.15.1:0/3688579438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f413806b500 msgr2=0x7f413806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.892+0000 7f415bd7a640 1 --2- 172.21.15.1:0/3688579438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f413806b500 0x7f413806d9c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f41540aca90 tx=0x7f4144000ed0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.892+0000 7f415bd7a640 1 -- 172.21.15.1:0/3688579438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41400052e0 msgr2=0x7f41541557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.892+0000 7f415bd7a640 1 --2- 172.21.15.1:0/3688579438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41400052e0 0x7f41541557c0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f414c011910 tx=0x7f414c011d10 comp rx=0 tx=0).stop 2024-03-06T07:29:19.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.892+0000 7f415bd7a640 1 -- 172.21.15.1:0/3688579438 shutdown_connections 2024-03-06T07:29:19.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.892+0000 7f415bd7a640 1 --2- 172.21.15.1:0/3688579438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f413806b500 0x7f413806d9c0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.892+0000 7f415bd7a640 1 --2- 172.21.15.1:0/3688579438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f415414c220 0x7f4154072c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.892+0000 7f415bd7a640 1 --2- 172.21.15.1:0/3688579438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41540a7870 0x7f41541eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.892+0000 7f415bd7a640 1 --2- 172.21.15.1:0/3688579438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41400052e0 0x7f41541557c0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.892+0000 7f415bd7a640 1 -- 172.21.15.1:0/3688579438 >> 172.21.15.1:0/3688579438 conn(0x7f41540c6860 msgr2=0x7f41541513e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.892+0000 7f415bd7a640 1 -- 172.21.15.1:0/3688579438 shutdown_connections 2024-03-06T07:29:19.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:19.892+0000 7f415bd7a640 1 -- 172.21.15.1:0/3688579438 wait complete. 2024-03-06T07:29:19.905 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:20.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.004+0000 7fe5029bc640 1 Processor -- start 2024-03-06T07:29:20.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.004+0000 7fe5029bc640 1 -- start start 2024-03-06T07:29:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.004+0000 7fe5029bc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe4fc0c1390 0x7fe4fc0c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.004+0000 7fe5029bc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4fc1588e0 con 0x7fe4fc152730 2024-03-06T07:29:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.004+0000 7fe5029bc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4fc158bc0 con 0x7fe4fc1552e0 2024-03-06T07:29:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.004+0000 7fe5029bc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4fc0c1d60 con 0x7fe4fc0c1390 2024-03-06T07:29:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.005+0000 7fe4fbfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe4fc152730 0x7fe4fc0c6a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48246/0 (socket says 172.21.15.1:48246) 2024-03-06T07:29:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.005+0000 7fe4fbfff640 1 -- 172.21.15.1:0/2632127336 learned_addr learned my addr 172.21.15.1:0/2632127336 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.005+0000 7fe4fb7fe640 1 --2- 172.21.15.1:0/2632127336 >> v2:172.21.15.98:3300/0 conn(0x7fe4fc0c1390 0x7fe4fc0c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.005+0000 7fe4faffd640 1 -- 172.21.15.1:0/2632127336 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353311954 0 0) 0x7fe4fc1588e0 con 0x7fe4fc152730 2024-03-06T07:29:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.005+0000 7fe4faffd640 1 -- 172.21.15.1:0/2632127336 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e4003520 con 0x7fe4fc152730 2024-03-06T07:29:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.005+0000 7fe4faffd640 1 -- 172.21.15.1:0/2632127336 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688708483 0 0) 0x7fe4fc158bc0 con 0x7fe4fc1552e0 2024-03-06T07:29:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.005+0000 7fe4faffd640 1 -- 172.21.15.1:0/2632127336 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4fc1588e0 con 0x7fe4fc1552e0 2024-03-06T07:29:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.005+0000 7fe4fb7fe640 1 -- 172.21.15.1:0/2632127336 >> v1:172.21.15.98:6789/0 conn(0x7fe4fc1552e0 legacy=0x7fe4fc155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.005+0000 7fe4fb7fe640 1 -- 172.21.15.1:0/2632127336 >> v1:172.21.15.1:6789/0 conn(0x7fe4fc152730 legacy=0x7fe4fc0c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.005+0000 7fe4faffd640 1 -- 172.21.15.1:0/2632127336 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 983302927 0 0) 0x7fe4e4003520 con 0x7fe4fc152730 2024-03-06T07:29:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.005+0000 7fe4fb7fe640 1 -- 172.21.15.1:0/2632127336 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc158ea0 con 0x7fe4fc0c1390 2024-03-06T07:29:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.006+0000 7fe4fb7fe640 1 --2- 172.21.15.1:0/2632127336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4fc0c1390 0x7fe4fc0c1820 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00097a0 tx=0x7fe4f002b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37b589db8005e864 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.006+0000 7fe4faffd640 1 -- 172.21.15.1:0/2632127336 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0037070 con 0x7fe4fc0c1390 2024-03-06T07:29:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.006+0000 7fe4faffd640 1 -- 172.21.15.1:0/2632127336 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f0030510 con 0x7fe4fc0c1390 2024-03-06T07:29:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.006+0000 7fe4faffd640 1 -- 172.21.15.1:0/2632127336 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec001670 con 0x7fe4fc0c1390 2024-03-06T07:29:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.007+0000 7fe5029bc640 1 -- 172.21.15.1:0/2632127336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4fc0c1390 msgr2=0x7fe4fc0c1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.007+0000 7fe5029bc640 1 --2- 172.21.15.1:0/2632127336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4fc0c1390 0x7fe4fc0c1820 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00097a0 tx=0x7fe4f002b710 comp rx=0 tx=0).stop 2024-03-06T07:29:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.007+0000 7fe5029bc640 1 -- 172.21.15.1:0/2632127336 shutdown_connections 2024-03-06T07:29:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.007+0000 7fe5029bc640 1 --2- 172.21.15.1:0/2632127336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4fc0c1390 0x7fe4fc0c1820 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.007+0000 7fe5029bc640 1 -- 172.21.15.1:0/2632127336 >> 172.21.15.1:0/2632127336 conn(0x7fe4fc066d00 msgr2=0x7fe4fc067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.007+0000 7fe5029bc640 1 -- 172.21.15.1:0/2632127336 shutdown_connections 2024-03-06T07:29:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.007+0000 7fe5029bc640 1 -- 172.21.15.1:0/2632127336 wait complete. 2024-03-06T07:29:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.008+0000 7fe5029bc640 1 Processor -- start 2024-03-06T07:29:20.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.008+0000 7fe5029bc640 1 -- start start 2024-03-06T07:29:20.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe5029bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4fc0c1390 0x7fe4fc0c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe5029bc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4fc152730 0x7fe4fc0af170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe5029bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4fc1552e0 0x7fe4fc0af6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe5029bc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc152b10 con 0x7fe4fc152730 2024-03-06T07:29:20.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe5029bc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc159460 con 0x7fe4fc0c1390 2024-03-06T07:29:20.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe5029bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc0c71a0 con 0x7fe4fc1552e0 2024-03-06T07:29:20.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe4fb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4fc152730 0x7fe4fc0af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe4fb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4fc152730 0x7fe4fc0af170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57358/0 (socket says 172.21.15.1:57358) 2024-03-06T07:29:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe4fb7fe640 1 -- 172.21.15.1:0/3934806370 learned_addr learned my addr 172.21.15.1:0/3934806370 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe4fb7fe640 1 -- 172.21.15.1:0/3934806370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4fc1552e0 msgr2=0x7fe4fc0af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe4fb7fe640 1 --2- 172.21.15.1:0/3934806370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4fc1552e0 0x7fe4fc0af6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe4fb7fe640 1 -- 172.21.15.1:0/3934806370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4fc0c1390 msgr2=0x7fe4fc0c9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe4fb7fe640 1 --2- 172.21.15.1:0/3934806370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4fc0c1390 0x7fe4fc0c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe4fb7fe640 1 -- 172.21.15.1:0/3934806370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc1ea940 con 0x7fe4fc152730 2024-03-06T07:29:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe4fb7fe640 1 --2- 172.21.15.1:0/3934806370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4fc152730 0x7fe4fc0af170 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0000c00 tx=0x7fe4f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe4f8ff9640 1 -- 172.21.15.1:0/3934806370 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0041020 con 0x7fe4fc152730 2024-03-06T07:29:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe4f8ff9640 1 -- 172.21.15.1:0/3934806370 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f003c870 con 0x7fe4fc152730 2024-03-06T07:29:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe5029bc640 1 -- 172.21.15.1:0/3934806370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4fc1eac00 con 0x7fe4fc152730 2024-03-06T07:29:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe4f8ff9640 1 -- 172.21.15.1:0/3934806370 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0037040 con 0x7fe4fc152730 2024-03-06T07:29:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.009+0000 7fe5029bc640 1 -- 172.21.15.1:0/3934806370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4fc1eb1c0 con 0x7fe4fc152730 2024-03-06T07:29:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.010+0000 7fe5029bc640 1 -- 172.21.15.1:0/3934806370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4fc1eb880 con 0x7fe4fc152730 2024-03-06T07:29:20.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.011+0000 7fe4f8ff9640 1 -- 172.21.15.1:0/3934806370 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe4f0044020 con 0x7fe4fc152730 2024-03-06T07:29:20.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.011+0000 7fe4f8ff9640 1 --2- 172.21.15.1:0/3934806370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe4e806b4d0 0x7fe4e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.011+0000 7fe4f8ff9640 1 -- 172.21.15.1:0/3934806370 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe4f00abd30 con 0x7fe4fc152730 2024-03-06T07:29:20.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.012+0000 7fe4fbfff640 1 --2- 172.21.15.1:0/3934806370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe4e806b4d0 0x7fe4e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.012+0000 7fe4fbfff640 1 --2- 172.21.15.1:0/3934806370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe4e806b4d0 0x7fe4e806d990 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec005880 tx=0x7fe4ec001230 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.016+0000 7fe4f8ff9640 1 -- 172.21.15.1:0/3934806370 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe4fc1eb880 con 0x7fe4fc152730 2024-03-06T07:29:20.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.200+0000 7fe5029bc640 1 -- 172.21.15.1:0/3934806370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4fc067fd0 con 0x7fe4fc152730 2024-03-06T07:29:20.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.201+0000 7fe4f8ff9640 1 -- 172.21.15.1:0/3934806370 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v199) v1 ==== 74+0+16929 (secure 0 0 0) 0x7fe4f004b030 con 0x7fe4fc152730 2024-03-06T07:29:20.203 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:20.203 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":199,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:29:11.378105+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:32.660232+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:33.362968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"32.0","mappings":[{"from":6,"to":5}]},{"pgid":"32.33","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:20.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.203+0000 7fe5029bc640 1 -- 172.21.15.1:0/3934806370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe4e806b4d0 msgr2=0x7fe4e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.203+0000 7fe5029bc640 1 --2- 172.21.15.1:0/3934806370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe4e806b4d0 0x7fe4e806d990 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec005880 tx=0x7fe4ec001230 comp rx=0 tx=0).stop 2024-03-06T07:29:20.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.203+0000 7fe5029bc640 1 -- 172.21.15.1:0/3934806370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4fc152730 msgr2=0x7fe4fc0af170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.203+0000 7fe5029bc640 1 --2- 172.21.15.1:0/3934806370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4fc152730 0x7fe4fc0af170 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0000c00 tx=0x7fe4f00046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.203+0000 7fe5029bc640 1 -- 172.21.15.1:0/3934806370 shutdown_connections 2024-03-06T07:29:20.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.203+0000 7fe5029bc640 1 --2- 172.21.15.1:0/3934806370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe4e806b4d0 0x7fe4e806d990 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.203+0000 7fe5029bc640 1 --2- 172.21.15.1:0/3934806370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4fc1552e0 0x7fe4fc0af6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.203+0000 7fe5029bc640 1 --2- 172.21.15.1:0/3934806370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4fc152730 0x7fe4fc0af170 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.203+0000 7fe5029bc640 1 --2- 172.21.15.1:0/3934806370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4fc0c1390 0x7fe4fc0c9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.203+0000 7fe5029bc640 1 -- 172.21.15.1:0/3934806370 >> 172.21.15.1:0/3934806370 conn(0x7fe4fc066d00 msgr2=0x7fe4fc0c3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.203+0000 7fe5029bc640 1 -- 172.21.15.1:0/3934806370 shutdown_connections 2024-03-06T07:29:20.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.203+0000 7fe5029bc640 1 -- 172.21.15.1:0/3934806370 wait complete. 2024-03-06T07:29:20.216 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:29:20.216 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:20.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.315+0000 7f519c8c0640 1 Processor -- start 2024-03-06T07:29:20.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.315+0000 7f519c8c0640 1 -- start start 2024-03-06T07:29:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.316+0000 7f519c8c0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51981505b0 0x7f51980c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.316+0000 7f519c8c0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f51980c16e0 0x7f5198156df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.316+0000 7f519c8c0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519806f5c0 con 0x7f51980c1260 2024-03-06T07:29:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.316+0000 7f519c8c0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5198152f80 con 0x7f51981505b0 2024-03-06T07:29:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.316+0000 7f519c8c0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5198150990 con 0x7f51980c16e0 2024-03-06T07:29:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.316+0000 7f519659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51981505b0 0x7f51980c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.316+0000 7f519659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51981505b0 0x7f51980c4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57372/0 (socket says 172.21.15.1:57372) 2024-03-06T07:29:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.316+0000 7f519659c640 1 -- 172.21.15.1:0/1631659048 learned_addr learned my addr 172.21.15.1:0/1631659048 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.316+0000 7f5196d9d640 1 --2- 172.21.15.1:0/1631659048 >> v2:172.21.15.98:3300/0 conn(0x7f51980c16e0 0x7f5198156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.316+0000 7f519659c640 1 -- 172.21.15.1:0/1631659048 >> v2:172.21.15.98:3300/0 conn(0x7f51980c16e0 msgr2=0x7f5198156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.316+0000 7f519659c640 1 --2- 172.21.15.1:0/1631659048 >> v2:172.21.15.98:3300/0 conn(0x7f51980c16e0 0x7f5198156df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.316+0000 7f519659c640 1 -- 172.21.15.1:0/1631659048 >> v1:172.21.15.98:6789/0 conn(0x7f51980c1260 legacy=0x7f51981566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.317+0000 7f519659c640 1 -- 172.21.15.1:0/1631659048 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5198153240 con 0x7f51981505b0 2024-03-06T07:29:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.317+0000 7f519659c640 1 --2- 172.21.15.1:0/1631659048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51981505b0 0x7f51980c4760 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f51980c73d0 tx=0x7f519002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf286a2810048c8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.317+0000 7f519559a640 1 -- 172.21.15.1:0/1631659048 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5190035020 con 0x7f51981505b0 2024-03-06T07:29:20.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.317+0000 7f519559a640 1 -- 172.21.15.1:0/1631659048 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5190031870 con 0x7f51981505b0 2024-03-06T07:29:20.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.317+0000 7f519559a640 1 -- 172.21.15.1:0/1631659048 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5190031c40 con 0x7f51981505b0 2024-03-06T07:29:20.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.318+0000 7f519c8c0640 1 -- 172.21.15.1:0/1631659048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51981505b0 msgr2=0x7f51980c4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.318+0000 7f519c8c0640 1 --2- 172.21.15.1:0/1631659048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51981505b0 0x7f51980c4760 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f51980c73d0 tx=0x7f519002b910 comp rx=0 tx=0).stop 2024-03-06T07:29:20.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.318+0000 7f519c8c0640 1 -- 172.21.15.1:0/1631659048 shutdown_connections 2024-03-06T07:29:20.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.318+0000 7f519c8c0640 1 --2- 172.21.15.1:0/1631659048 >> v2:172.21.15.98:3300/0 conn(0x7f51980c16e0 0x7f5198156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.318+0000 7f519c8c0640 1 --2- 172.21.15.1:0/1631659048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51981505b0 0x7f51980c4760 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.318+0000 7f519c8c0640 1 -- 172.21.15.1:0/1631659048 >> 172.21.15.1:0/1631659048 conn(0x7f51980c6860 msgr2=0x7f51980c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.318+0000 7f519c8c0640 1 -- 172.21.15.1:0/1631659048 shutdown_connections 2024-03-06T07:29:20.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.318+0000 7f519c8c0640 1 -- 172.21.15.1:0/1631659048 wait complete. 2024-03-06T07:29:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.319+0000 7f519c8c0640 1 Processor -- start 2024-03-06T07:29:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.319+0000 7f519c8c0640 1 -- start start 2024-03-06T07:29:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.320+0000 7f519c8c0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51980c1260 0x7f51981dea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.320+0000 7f519c8c0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51980c16e0 0x7f5198155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.320+0000 7f519c8c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51981505b0 0x7f51981e7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.320+0000 7f519c8c0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5198153ae0 con 0x7f51981505b0 2024-03-06T07:29:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.320+0000 7f519c8c0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5198153800 con 0x7f51980c16e0 2024-03-06T07:29:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.320+0000 7f519c8c0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5198157330 con 0x7f51980c1260 2024-03-06T07:29:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.320+0000 7f5196d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51981505b0 0x7f51981e7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.320+0000 7f5196d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51981505b0 0x7f51981e7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57382/0 (socket says 172.21.15.1:57382) 2024-03-06T07:29:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.320+0000 7f5196d9d640 1 -- 172.21.15.1:0/880514370 learned_addr learned my addr 172.21.15.1:0/880514370 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.320+0000 7f5195d9b640 1 --2- 172.21.15.1:0/880514370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51980c16e0 0x7f5198155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.320+0000 7f519659c640 1 --2- 172.21.15.1:0/880514370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51980c1260 0x7f51981dea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.320+0000 7f5196d9d640 1 -- 172.21.15.1:0/880514370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51980c1260 msgr2=0x7f51981dea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.320+0000 7f5196d9d640 1 --2- 172.21.15.1:0/880514370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51980c1260 0x7f51981dea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.320+0000 7f5196d9d640 1 -- 172.21.15.1:0/880514370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51980c16e0 msgr2=0x7f5198155910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.320+0000 7f5196d9d640 1 --2- 172.21.15.1:0/880514370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51980c16e0 0x7f5198155910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.320+0000 7f5196d9d640 1 -- 172.21.15.1:0/880514370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51981ea920 con 0x7f51981505b0 2024-03-06T07:29:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.320+0000 7f519659c640 1 --2- 172.21.15.1:0/880514370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51980c1260 0x7f51981dea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.321+0000 7f5196d9d640 1 --2- 172.21.15.1:0/880514370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51981505b0 0x7f51981e7b80 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f5188007e40 tx=0x7f518800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.321+0000 7f5186ffd640 1 -- 172.21.15.1:0/880514370 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188014620 con 0x7f51981505b0 2024-03-06T07:29:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.321+0000 7f5186ffd640 1 -- 172.21.15.1:0/880514370 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5188019070 con 0x7f51981505b0 2024-03-06T07:29:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.321+0000 7f519c8c0640 1 -- 172.21.15.1:0/880514370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51981eac00 con 0x7f51981505b0 2024-03-06T07:29:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.321+0000 7f5186ffd640 1 -- 172.21.15.1:0/880514370 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188014a20 con 0x7f51981505b0 2024-03-06T07:29:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.321+0000 7f519c8c0640 1 -- 172.21.15.1:0/880514370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51981eb1c0 con 0x7f51981505b0 2024-03-06T07:29:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.322+0000 7f519c8c0640 1 -- 172.21.15.1:0/880514370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f519806f280 con 0x7f51981505b0 2024-03-06T07:29:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.322+0000 7f5186ffd640 1 -- 172.21.15.1:0/880514370 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5188014bc0 con 0x7f51981505b0 2024-03-06T07:29:20.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.323+0000 7f5186ffd640 1 --2- 172.21.15.1:0/880514370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f517806b4d0 0x7f517806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.323+0000 7f5186ffd640 1 -- 172.21.15.1:0/880514370 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f518808ac20 con 0x7f51981505b0 2024-03-06T07:29:20.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.323+0000 7f519659c640 1 --2- 172.21.15.1:0/880514370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f517806b4d0 0x7f517806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.323+0000 7f519659c640 1 --2- 172.21.15.1:0/880514370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f517806b4d0 0x7f517806d990 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f5190036000 tx=0x7f5190008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.327+0000 7f5186ffd640 1 -- 172.21.15.1:0/880514370 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51880598d0 con 0x7f51981505b0 2024-03-06T07:29:20.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.552+0000 7f519c8c0640 1 -- 172.21.15.1:0/880514370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51980c6de0 con 0x7f51981505b0 2024-03-06T07:29:20.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.552+0000 7f5186ffd640 1 -- 172.21.15.1:0/880514370 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+4962 (secure 0 0 0) 0x7f51880591d0 con 0x7f51981505b0 2024-03-06T07:29:20.555 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:20.555 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":189,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7961,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2819896202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2819896202},{"type":"v1","addr":"172.21.15.98:6837","nonce":2819896202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":8959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/646952744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":646952744},{"type":"v1","addr":"172.21.15.1:6835","nonce":646952744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":8965,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2843229007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2843229007},{"type":"v1","addr":"172.21.15.98:6836","nonce":2843229007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":10635,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/1512189189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1512189189},{"type":"v1","addr":"172.21.15.1:6834","nonce":1512189189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:34.395524+0000","modified":"2024-03-06T07:28:59.966364+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":15}]} 2024-03-06T07:29:20.555 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 189 2024-03-06T07:29:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.555+0000 7f519c8c0640 1 -- 172.21.15.1:0/880514370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f517806b4d0 msgr2=0x7f517806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.555+0000 7f519c8c0640 1 --2- 172.21.15.1:0/880514370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f517806b4d0 0x7f517806d990 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f5190036000 tx=0x7f5190008030 comp rx=0 tx=0).stop 2024-03-06T07:29:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.555+0000 7f519c8c0640 1 -- 172.21.15.1:0/880514370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51981505b0 msgr2=0x7f51981e7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.555+0000 7f519c8c0640 1 --2- 172.21.15.1:0/880514370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51981505b0 0x7f51981e7b80 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f5188007e40 tx=0x7f518800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.555+0000 7f519c8c0640 1 -- 172.21.15.1:0/880514370 shutdown_connections 2024-03-06T07:29:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.555+0000 7f519c8c0640 1 --2- 172.21.15.1:0/880514370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f517806b4d0 0x7f517806d990 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.555+0000 7f519c8c0640 1 --2- 172.21.15.1:0/880514370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51981505b0 0x7f51981e7b80 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.555+0000 7f519c8c0640 1 --2- 172.21.15.1:0/880514370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51980c16e0 0x7f5198155910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.555+0000 7f519c8c0640 1 --2- 172.21.15.1:0/880514370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f51980c1260 0x7f51981dea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.555+0000 7f519c8c0640 1 -- 172.21.15.1:0/880514370 >> 172.21.15.1:0/880514370 conn(0x7f51980c6860 msgr2=0x7f51981566e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.555+0000 7f519c8c0640 1 -- 172.21.15.1:0/880514370 shutdown_connections 2024-03-06T07:29:20.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.555+0000 7f519c8c0640 1 -- 172.21.15.1:0/880514370 wait complete. 2024-03-06T07:29:20.568 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:20.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.666+0000 7f0152ec9640 1 Processor -- start 2024-03-06T07:29:20.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.666+0000 7f0152ec9640 1 -- start start 2024-03-06T07:29:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.667+0000 7f0152ec9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f014c152650 0x7f014c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.667+0000 7f0152ec9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f014c0c6b10 0x7f014c158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.667+0000 7f0152ec9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014c0c2fc0 con 0x7f014c0bf9d0 2024-03-06T07:29:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.667+0000 7f0152ec9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f014c155800 con 0x7f014c152650 2024-03-06T07:29:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.667+0000 7f0152ec9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014c0c6690 con 0x7f014c0c6b10 2024-03-06T07:29:20.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.668+0000 7f014bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f014c0c6b10 0x7f014c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.668+0000 7f0150c65640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f014c152650 0x7f014c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.668+0000 7f0151466640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f014c0bf9d0 0x7f014c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:33558/0 (socket says 172.21.15.1:33558) 2024-03-06T07:29:20.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.668+0000 7f0151466640 1 -- 172.21.15.1:0/336867004 learned_addr learned my addr 172.21.15.1:0/336867004 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:20.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.668+0000 7f014b7fe640 1 -- 172.21.15.1:0/336867004 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632192209 0 0) 0x7f014c0c2fc0 con 0x7f014c0bf9d0 2024-03-06T07:29:20.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.668+0000 7f014b7fe640 1 -- 172.21.15.1:0/336867004 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0134003520 con 0x7f014c0bf9d0 2024-03-06T07:29:20.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.668+0000 7f014bfff640 1 -- 172.21.15.1:0/336867004 >> v2:172.21.15.98:3300/0 conn(0x7f014c152650 msgr2=0x7f014c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.668+0000 7f014bfff640 1 --2- 172.21.15.1:0/336867004 >> v2:172.21.15.98:3300/0 conn(0x7f014c152650 0x7f014c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.668+0000 7f014bfff640 1 -- 172.21.15.1:0/336867004 >> v1:172.21.15.98:6789/0 conn(0x7f014c0bf9d0 legacy=0x7f014c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.668+0000 7f014bfff640 1 -- 172.21.15.1:0/336867004 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c0c7280 con 0x7f014c0c6b10 2024-03-06T07:29:20.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.668+0000 7f0150c65640 1 --2- 172.21.15.1:0/336867004 >> v2:172.21.15.98:3300/0 conn(0x7f014c152650 0x7f014c155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:20.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.669+0000 7f014bfff640 1 --2- 172.21.15.1:0/336867004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f014c0c6b10 0x7f014c158690 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f01400097a0 tx=0x7f014002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91acc495048b730a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.669+0000 7f014b7fe640 1 -- 172.21.15.1:0/336867004 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01400315a0 con 0x7f014c0c6b10 2024-03-06T07:29:20.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.669+0000 7f014b7fe640 1 -- 172.21.15.1:0/336867004 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0140037070 con 0x7f014c0c6b10 2024-03-06T07:29:20.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.669+0000 7f014b7fe640 1 -- 172.21.15.1:0/336867004 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0140031960 con 0x7f014c0c6b10 2024-03-06T07:29:20.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.669+0000 7f0152ec9640 1 -- 172.21.15.1:0/336867004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f014c0c6b10 msgr2=0x7f014c158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.669+0000 7f0152ec9640 1 --2- 172.21.15.1:0/336867004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f014c0c6b10 0x7f014c158690 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f01400097a0 tx=0x7f014002b840 comp rx=0 tx=0).stop 2024-03-06T07:29:20.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.669+0000 7f0152ec9640 1 -- 172.21.15.1:0/336867004 shutdown_connections 2024-03-06T07:29:20.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.669+0000 7f0152ec9640 1 --2- 172.21.15.1:0/336867004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f014c0c6b10 0x7f014c158690 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.669+0000 7f0152ec9640 1 --2- 172.21.15.1:0/336867004 >> v2:172.21.15.98:3300/0 conn(0x7f014c152650 0x7f014c155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.669+0000 7f0152ec9640 1 -- 172.21.15.1:0/336867004 >> 172.21.15.1:0/336867004 conn(0x7f014c066ee0 msgr2=0x7f014c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.670+0000 7f0152ec9640 1 -- 172.21.15.1:0/336867004 shutdown_connections 2024-03-06T07:29:20.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.670+0000 7f0152ec9640 1 -- 172.21.15.1:0/336867004 wait complete. 2024-03-06T07:29:20.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.671+0000 7f0152ec9640 1 Processor -- start 2024-03-06T07:29:20.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.671+0000 7f0152ec9640 1 -- start start 2024-03-06T07:29:20.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.671+0000 7f0152ec9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f014c0bf9d0 0x7f014c1eeae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.671+0000 7f0152ec9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f014c152650 0x7f014c154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.671+0000 7f0152ec9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014c1546a0 0x7f014c156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.671+0000 7f0152ec9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014c155490 con 0x7f014c0bf9d0 2024-03-06T07:29:20.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.671+0000 7f0152ec9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014c1551b0 con 0x7f014c1546a0 2024-03-06T07:29:20.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.671+0000 7f0152ec9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014c152a30 con 0x7f014c152650 2024-03-06T07:29:20.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.672+0000 7f0150c65640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f014c0bf9d0 0x7f014c1eeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.672+0000 7f0150c65640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f014c0bf9d0 0x7f014c1eeae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57392/0 (socket says 172.21.15.1:57392) 2024-03-06T07:29:20.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.672+0000 7f0150c65640 1 -- 172.21.15.1:0/1252165308 learned_addr learned my addr 172.21.15.1:0/1252165308 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:20.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.672+0000 7f014bfff640 1 --2- 172.21.15.1:0/1252165308 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f014c152650 0x7f014c154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.672+0000 7f0151466640 1 --2- 172.21.15.1:0/1252165308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014c1546a0 0x7f014c156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.672+0000 7f0150c65640 1 -- 172.21.15.1:0/1252165308 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f014c152650 msgr2=0x7f014c154160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.672+0000 7f0150c65640 1 --2- 172.21.15.1:0/1252165308 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f014c152650 0x7f014c154160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.672+0000 7f0150c65640 1 -- 172.21.15.1:0/1252165308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014c1546a0 msgr2=0x7f014c156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.672+0000 7f0150c65640 1 --2- 172.21.15.1:0/1252165308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014c1546a0 0x7f014c156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.672+0000 7f0150c65640 1 -- 172.21.15.1:0/1252165308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c1592a0 con 0x7f014c0bf9d0 2024-03-06T07:29:20.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.672+0000 7f0150c65640 1 --2- 172.21.15.1:0/1252165308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f014c0bf9d0 0x7f014c1eeae0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f0144009f60 tx=0x7f0144009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.672+0000 7f01497fa640 1 -- 172.21.15.1:0/1252165308 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0144018020 con 0x7f014c0bf9d0 2024-03-06T07:29:20.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.672+0000 7f01497fa640 1 -- 172.21.15.1:0/1252165308 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0144014680 con 0x7f014c0bf9d0 2024-03-06T07:29:20.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.672+0000 7f0152ec9640 1 -- 172.21.15.1:0/1252165308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f014c0c99a0 con 0x7f014c0bf9d0 2024-03-06T07:29:20.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.672+0000 7f01497fa640 1 -- 172.21.15.1:0/1252165308 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0144014a50 con 0x7f014c0bf9d0 2024-03-06T07:29:20.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.672+0000 7f0152ec9640 1 -- 172.21.15.1:0/1252165308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f014c0c9f60 con 0x7f014c0bf9d0 2024-03-06T07:29:20.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.673+0000 7f0152ec9640 1 -- 172.21.15.1:0/1252165308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f014c067fd0 con 0x7f014c0bf9d0 2024-03-06T07:29:20.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.674+0000 7f01497fa640 1 -- 172.21.15.1:0/1252165308 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f014401b020 con 0x7f014c0bf9d0 2024-03-06T07:29:20.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.674+0000 7f01497fa640 1 --2- 172.21.15.1:0/1252165308 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f013806b500 0x7f013806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.674+0000 7f01497fa640 1 -- 172.21.15.1:0/1252165308 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f014408b130 con 0x7f014c0bf9d0 2024-03-06T07:29:20.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.674+0000 7f014bfff640 1 --2- 172.21.15.1:0/1252165308 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f013806b500 0x7f013806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.675+0000 7f014bfff640 1 --2- 172.21.15.1:0/1252165308 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f013806b500 0x7f013806d9c0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f0140036000 tx=0x7f014003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.678+0000 7f01497fa640 1 -- 172.21.15.1:0/1252165308 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0144022020 con 0x7f014c0bf9d0 2024-03-06T07:29:20.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.897+0000 7f0152ec9640 1 -- 172.21.15.1:0/1252165308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f014c144cc0 con 0x7f014c0bf9d0 2024-03-06T07:29:20.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.897+0000 7f01497fa640 1 -- 172.21.15.1:0/1252165308 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+4962 (secure 0 0 0) 0x7f0144010040 con 0x7f014c0bf9d0 2024-03-06T07:29:20.900 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:20.900 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":189,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7961,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2819896202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2819896202},{"type":"v1","addr":"172.21.15.98:6837","nonce":2819896202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":8959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/646952744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":646952744},{"type":"v1","addr":"172.21.15.1:6835","nonce":646952744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":8965,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2843229007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2843229007},{"type":"v1","addr":"172.21.15.98:6836","nonce":2843229007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":10635,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/1512189189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1512189189},{"type":"v1","addr":"172.21.15.1:6834","nonce":1512189189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:34.395524+0000","modified":"2024-03-06T07:28:59.966364+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":15}]} 2024-03-06T07:29:20.900 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 189 2024-03-06T07:29:20.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.899+0000 7f0152ec9640 1 -- 172.21.15.1:0/1252165308 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f013806b500 msgr2=0x7f013806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.899+0000 7f0152ec9640 1 --2- 172.21.15.1:0/1252165308 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f013806b500 0x7f013806d9c0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f0140036000 tx=0x7f014003d040 comp rx=0 tx=0).stop 2024-03-06T07:29:20.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.899+0000 7f0152ec9640 1 -- 172.21.15.1:0/1252165308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f014c0bf9d0 msgr2=0x7f014c1eeae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.900+0000 7f0152ec9640 1 --2- 172.21.15.1:0/1252165308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f014c0bf9d0 0x7f014c1eeae0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f0144009f60 tx=0x7f0144009f90 comp rx=0 tx=0).stop 2024-03-06T07:29:20.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.900+0000 7f0152ec9640 1 -- 172.21.15.1:0/1252165308 shutdown_connections 2024-03-06T07:29:20.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.900+0000 7f0152ec9640 1 --2- 172.21.15.1:0/1252165308 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f013806b500 0x7f013806d9c0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.900+0000 7f0152ec9640 1 --2- 172.21.15.1:0/1252165308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014c1546a0 0x7f014c156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.900+0000 7f0152ec9640 1 --2- 172.21.15.1:0/1252165308 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f014c152650 0x7f014c154160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.900+0000 7f0152ec9640 1 --2- 172.21.15.1:0/1252165308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f014c0bf9d0 0x7f014c1eeae0 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.900+0000 7f0152ec9640 1 -- 172.21.15.1:0/1252165308 >> 172.21.15.1:0/1252165308 conn(0x7f014c066ee0 msgr2=0x7f014c0c5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.900+0000 7f0152ec9640 1 -- 172.21.15.1:0/1252165308 shutdown_connections 2024-03-06T07:29:20.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:20.900+0000 7f0152ec9640 1 -- 172.21.15.1:0/1252165308 wait complete. 2024-03-06T07:29:20.913 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:21.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.011+0000 7f931dc65640 1 Processor -- start 2024-03-06T07:29:21.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.012+0000 7f931dc65640 1 -- start start 2024-03-06T07:29:21.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.012+0000 7f931dc65640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9318150fe0 0x7f93181513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.012+0000 7f931dc65640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9318071f90 0x7f9318072400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.012+0000 7f931dc65640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93180a6140 con 0x7f9318072f20 2024-03-06T07:29:21.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.012+0000 7f931dc65640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93181526b0 con 0x7f9318071f90 2024-03-06T07:29:21.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.012+0000 7f931dc65640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9318151aa0 con 0x7f9318150fe0 2024-03-06T07:29:21.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.012+0000 7f9317fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9318071f90 0x7f9318072400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.012+0000 7f9317fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9318071f90 0x7f9318072400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57398/0 (socket says 172.21.15.1:57398) 2024-03-06T07:29:21.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.012+0000 7f9317fff640 1 -- 172.21.15.1:0/3384097724 learned_addr learned my addr 172.21.15.1:0/3384097724 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:21.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.013+0000 7f93177fe640 1 --2- 172.21.15.1:0/3384097724 >> v2:172.21.15.98:3300/0 conn(0x7f9318150fe0 0x7f93181513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.013+0000 7f9317fff640 1 -- 172.21.15.1:0/3384097724 >> v2:172.21.15.98:3300/0 conn(0x7f9318150fe0 msgr2=0x7f93181513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:21.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.013+0000 7f9317fff640 1 --2- 172.21.15.1:0/3384097724 >> v2:172.21.15.98:3300/0 conn(0x7f9318150fe0 0x7f93181513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.013+0000 7f9317fff640 1 -- 172.21.15.1:0/3384097724 >> v1:172.21.15.98:6789/0 conn(0x7f9318072f20 legacy=0x7f931806edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.013+0000 7f9317fff640 1 -- 172.21.15.1:0/3384097724 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9318151d80 con 0x7f9318071f90 2024-03-06T07:29:21.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.014+0000 7f9317fff640 1 --2- 172.21.15.1:0/3384097724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9318071f90 0x7f9318072400 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f931806fba0 tx=0x7f931002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f46624286a0afa09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.014+0000 7f93167fc640 1 -- 172.21.15.1:0/3384097724 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931002ca80 con 0x7f9318071f90 2024-03-06T07:29:21.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.014+0000 7f93167fc640 1 -- 172.21.15.1:0/3384097724 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931002cc20 con 0x7f9318071f90 2024-03-06T07:29:21.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.014+0000 7f93167fc640 1 -- 172.21.15.1:0/3384097724 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93100343f0 con 0x7f9318071f90 2024-03-06T07:29:21.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.014+0000 7f931dc65640 1 -- 172.21.15.1:0/3384097724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9318071f90 msgr2=0x7f9318072400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.014+0000 7f931dc65640 1 --2- 172.21.15.1:0/3384097724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9318071f90 0x7f9318072400 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f931806fba0 tx=0x7f931002bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.014+0000 7f931dc65640 1 -- 172.21.15.1:0/3384097724 shutdown_connections 2024-03-06T07:29:21.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.014+0000 7f931dc65640 1 --2- 172.21.15.1:0/3384097724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9318071f90 0x7f9318072400 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.014+0000 7f931dc65640 1 --2- 172.21.15.1:0/3384097724 >> v2:172.21.15.98:3300/0 conn(0x7f9318150fe0 0x7f93181513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.014+0000 7f931dc65640 1 -- 172.21.15.1:0/3384097724 >> 172.21.15.1:0/3384097724 conn(0x7f93180c6860 msgr2=0x7f93180c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.014+0000 7f931dc65640 1 -- 172.21.15.1:0/3384097724 shutdown_connections 2024-03-06T07:29:21.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.014+0000 7f931dc65640 1 -- 172.21.15.1:0/3384097724 wait complete. 2024-03-06T07:29:21.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.015+0000 7f931dc65640 1 Processor -- start 2024-03-06T07:29:21.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.015+0000 7f931dc65640 1 -- start start 2024-03-06T07:29:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.016+0000 7f931dc65640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9318071f90 0x7f93181e8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.016+0000 7f931dc65640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9318072f20 0x7f93181419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.016+0000 7f931dc65640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9318150fe0 0x7f9318146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.016+0000 7f931dc65640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931806f4c0 con 0x7f9318150fe0 2024-03-06T07:29:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.016+0000 7f931dc65640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9318152340 con 0x7f9318072f20 2024-03-06T07:29:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.016+0000 7f931dc65640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931806f640 con 0x7f9318071f90 2024-03-06T07:29:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.016+0000 7f9317fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9318150fe0 0x7f9318146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.016+0000 7f9317fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9318150fe0 0x7f9318146bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57404/0 (socket says 172.21.15.1:57404) 2024-03-06T07:29:21.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.016+0000 7f9317fff640 1 -- 172.21.15.1:0/333037487 learned_addr learned my addr 172.21.15.1:0/333037487 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:21.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.016+0000 7f93177fe640 1 --2- 172.21.15.1:0/333037487 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9318071f90 0x7f93181e8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.017+0000 7f9316ffd640 1 --2- 172.21.15.1:0/333037487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9318072f20 0x7f93181419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.017+0000 7f9317fff640 1 -- 172.21.15.1:0/333037487 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9318071f90 msgr2=0x7f93181e8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.017+0000 7f9317fff640 1 --2- 172.21.15.1:0/333037487 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9318071f90 0x7f93181e8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.017+0000 7f9317fff640 1 -- 172.21.15.1:0/333037487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9318072f20 msgr2=0x7f93181419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.017+0000 7f9317fff640 1 --2- 172.21.15.1:0/333037487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9318072f20 0x7f93181419a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.017+0000 7f9317fff640 1 -- 172.21.15.1:0/333037487 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931814ea40 con 0x7f9318150fe0 2024-03-06T07:29:21.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.017+0000 7f9317fff640 1 --2- 172.21.15.1:0/333037487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9318150fe0 0x7f9318146bd0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f9310036040 tx=0x7f9310004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.017+0000 7f92fffff640 1 -- 172.21.15.1:0/333037487 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9310043020 con 0x7f9318150fe0 2024-03-06T07:29:21.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.017+0000 7f92fffff640 1 -- 172.21.15.1:0/333037487 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931003e450 con 0x7f9318150fe0 2024-03-06T07:29:21.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.017+0000 7f931dc65640 1 -- 172.21.15.1:0/333037487 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9318143710 con 0x7f9318150fe0 2024-03-06T07:29:21.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.017+0000 7f92fffff640 1 -- 172.21.15.1:0/333037487 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9310009510 con 0x7f9318150fe0 2024-03-06T07:29:21.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.017+0000 7f931dc65640 1 -- 172.21.15.1:0/333037487 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93181411a0 con 0x7f9318150fe0 2024-03-06T07:29:21.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.018+0000 7f931dc65640 1 -- 172.21.15.1:0/333037487 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9318143910 con 0x7f9318150fe0 2024-03-06T07:29:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.018+0000 7f92fffff640 1 -- 172.21.15.1:0/333037487 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9310046020 con 0x7f9318150fe0 2024-03-06T07:29:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.019+0000 7f92fffff640 1 --2- 172.21.15.1:0/333037487 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f930406b500 0x7f930406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.019+0000 7f92fffff640 1 -- 172.21.15.1:0/333037487 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f93100ac7a0 con 0x7f9318150fe0 2024-03-06T07:29:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.019+0000 7f93177fe640 1 --2- 172.21.15.1:0/333037487 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f930406b500 0x7f930406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.019+0000 7f93177fe640 1 --2- 172.21.15.1:0/333037487 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f930406b500 0x7f930406d9c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f9318144780 tx=0x7f9308008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.023+0000 7f92fffff640 1 -- 172.21.15.1:0/333037487 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9318143910 con 0x7f9318150fe0 2024-03-06T07:29:21.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.206+0000 7f931dc65640 1 -- 172.21.15.1:0/333037487 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9318150820 con 0x7f9318150fe0 2024-03-06T07:29:21.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.207+0000 7f92fffff640 1 -- 172.21.15.1:0/333037487 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v199) v1 ==== 74+0+16929 (secure 0 0 0) 0x7f9318150820 con 0x7f9318150fe0 2024-03-06T07:29:21.209 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:21.209 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":199,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:29:11.378105+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:32.660232+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:33.362968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"32.0","mappings":[{"from":6,"to":5}]},{"pgid":"32.33","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:21.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.208+0000 7f931dc65640 1 -- 172.21.15.1:0/333037487 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f930406b500 msgr2=0x7f930406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.208+0000 7f931dc65640 1 --2- 172.21.15.1:0/333037487 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f930406b500 0x7f930406d9c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f9318144780 tx=0x7f9308008040 comp rx=0 tx=0).stop 2024-03-06T07:29:21.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.208+0000 7f931dc65640 1 -- 172.21.15.1:0/333037487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9318150fe0 msgr2=0x7f9318146bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.208+0000 7f931dc65640 1 --2- 172.21.15.1:0/333037487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9318150fe0 0x7f9318146bd0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f9310036040 tx=0x7f9310004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:21.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.209+0000 7f931dc65640 1 -- 172.21.15.1:0/333037487 shutdown_connections 2024-03-06T07:29:21.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.209+0000 7f931dc65640 1 --2- 172.21.15.1:0/333037487 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f930406b500 0x7f930406d9c0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.209+0000 7f931dc65640 1 --2- 172.21.15.1:0/333037487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9318150fe0 0x7f9318146bd0 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.209+0000 7f931dc65640 1 --2- 172.21.15.1:0/333037487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9318072f20 0x7f93181419a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.209+0000 7f931dc65640 1 --2- 172.21.15.1:0/333037487 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9318071f90 0x7f93181e8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.209+0000 7f931dc65640 1 -- 172.21.15.1:0/333037487 >> 172.21.15.1:0/333037487 conn(0x7f93180c6860 msgr2=0x7f931814f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.209+0000 7f931dc65640 1 -- 172.21.15.1:0/333037487 shutdown_connections 2024-03-06T07:29:21.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.209+0000 7f931dc65640 1 -- 172.21.15.1:0/333037487 wait complete. 2024-03-06T07:29:21.222 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:29:21.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.321+0000 7fb3a47c9640 1 Processor -- start 2024-03-06T07:29:21.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.321+0000 7fb3a47c9640 1 -- start start 2024-03-06T07:29:21.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.322+0000 7fb3a47c9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb39c149a90 0x7fb39c0c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.322+0000 7fb3a47c9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb39c0c1260 0x7fb39c0c16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.322+0000 7fb3a47c9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb39c1530a0 con 0x7fb39c1565a0 2024-03-06T07:29:21.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.322+0000 7fb3a47c9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb39c153380 con 0x7fb39c0c1260 2024-03-06T07:29:21.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.322+0000 7fb3a47c9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb39c153660 con 0x7fb39c149a90 2024-03-06T07:29:21.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.322+0000 7fb3a1d64640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb39c0c1260 0x7fb39c0c16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.322+0000 7fb3a2d66640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb39c1565a0 0x7fb39c156a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48254/0 (socket says 172.21.15.1:48254) 2024-03-06T07:29:21.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.322+0000 7fb3a2d66640 1 -- 172.21.15.1:0/272527600 learned_addr learned my addr 172.21.15.1:0/272527600 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:21.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.322+0000 7fb3a2565640 1 --2- 172.21.15.1:0/272527600 >> v2:172.21.15.98:3300/0 conn(0x7fb39c149a90 0x7fb39c0c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.323+0000 7fb3a1d64640 1 -- 172.21.15.1:0/272527600 >> v2:172.21.15.98:3300/0 conn(0x7fb39c149a90 msgr2=0x7fb39c0c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:21.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.323+0000 7fb3a1d64640 1 --2- 172.21.15.1:0/272527600 >> v2:172.21.15.98:3300/0 conn(0x7fb39c149a90 0x7fb39c0c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.323+0000 7fb3a1d64640 1 -- 172.21.15.1:0/272527600 >> v1:172.21.15.1:6789/0 conn(0x7fb39c1565a0 legacy=0x7fb39c156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.323+0000 7fb3a1d64640 1 -- 172.21.15.1:0/272527600 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39c153940 con 0x7fb39c0c1260 2024-03-06T07:29:21.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.323+0000 7fb3a1563640 1 -- 172.21.15.1:0/272527600 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058647805 0 0) 0x7fb39c1530a0 con 0x7fb39c1565a0 2024-03-06T07:29:21.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.323+0000 7fb3a1d64640 1 --2- 172.21.15.1:0/272527600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb39c0c1260 0x7fb39c0c16f0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fb39c06f9e0 tx=0x7fb38c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c360345ab7c559 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.323+0000 7fb3a1563640 1 -- 172.21.15.1:0/272527600 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c0315a0 con 0x7fb39c0c1260 2024-03-06T07:29:21.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.323+0000 7fb3a1563640 1 -- 172.21.15.1:0/272527600 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38c037070 con 0x7fb39c0c1260 2024-03-06T07:29:21.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.323+0000 7fb3a1563640 1 -- 172.21.15.1:0/272527600 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c031960 con 0x7fb39c0c1260 2024-03-06T07:29:21.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.324+0000 7fb3a47c9640 1 -- 172.21.15.1:0/272527600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb39c0c1260 msgr2=0x7fb39c0c16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.324+0000 7fb3a47c9640 1 --2- 172.21.15.1:0/272527600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb39c0c1260 0x7fb39c0c16f0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fb39c06f9e0 tx=0x7fb38c02b840 comp rx=0 tx=0).stop 2024-03-06T07:29:21.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.324+0000 7fb3a47c9640 1 -- 172.21.15.1:0/272527600 shutdown_connections 2024-03-06T07:29:21.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.324+0000 7fb3a47c9640 1 --2- 172.21.15.1:0/272527600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb39c0c1260 0x7fb39c0c16f0 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.324+0000 7fb3a47c9640 1 --2- 172.21.15.1:0/272527600 >> v2:172.21.15.98:3300/0 conn(0x7fb39c149a90 0x7fb39c0c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.324+0000 7fb3a47c9640 1 -- 172.21.15.1:0/272527600 >> 172.21.15.1:0/272527600 conn(0x7fb39c0c6860 msgr2=0x7fb39c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.324+0000 7fb3a47c9640 1 -- 172.21.15.1:0/272527600 shutdown_connections 2024-03-06T07:29:21.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.324+0000 7fb3a47c9640 1 -- 172.21.15.1:0/272527600 wait complete. 2024-03-06T07:29:21.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.325+0000 7fb3a47c9640 1 Processor -- start 2024-03-06T07:29:21.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.325+0000 7fb3a47c9640 1 -- start start 2024-03-06T07:29:21.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.326+0000 7fb3a47c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c0c1260 0x7fb39c071540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.326+0000 7fb3a47c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb39c149a90 0x7fb39c14c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.326+0000 7fb3a47c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb39c1565a0 0x7fb39c1eb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.326+0000 7fb3a47c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c157120 con 0x7fb39c1565a0 2024-03-06T07:29:21.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.326+0000 7fb3a47c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c0c4ca0 con 0x7fb39c0c1260 2024-03-06T07:29:21.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.326+0000 7fb3a47c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c1572a0 con 0x7fb39c149a90 2024-03-06T07:29:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.326+0000 7fb3a2d66640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb39c1565a0 0x7fb39c1eb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.326+0000 7fb3a2d66640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb39c1565a0 0x7fb39c1eb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57422/0 (socket says 172.21.15.1:57422) 2024-03-06T07:29:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.326+0000 7fb3a2d66640 1 -- 172.21.15.1:0/2586839987 learned_addr learned my addr 172.21.15.1:0/2586839987 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.326+0000 7fb3a1d64640 1 --2- 172.21.15.1:0/2586839987 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb39c149a90 0x7fb39c14c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.326+0000 7fb3a2565640 1 --2- 172.21.15.1:0/2586839987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c0c1260 0x7fb39c071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.326+0000 7fb3a2d66640 1 -- 172.21.15.1:0/2586839987 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb39c149a90 msgr2=0x7fb39c14c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.326+0000 7fb3a2d66640 1 --2- 172.21.15.1:0/2586839987 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb39c149a90 0x7fb39c14c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.326+0000 7fb3a2d66640 1 -- 172.21.15.1:0/2586839987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c0c1260 msgr2=0x7fb39c071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.326+0000 7fb3a2d66640 1 --2- 172.21.15.1:0/2586839987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c0c1260 0x7fb39c071540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.326+0000 7fb3a2d66640 1 -- 172.21.15.1:0/2586839987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39c1510e0 con 0x7fb39c1565a0 2024-03-06T07:29:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.327+0000 7fb3a2d66640 1 --2- 172.21.15.1:0/2586839987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb39c1565a0 0x7fb39c1eb2b0 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fb39400c7a0 tx=0x7fb39400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.327+0000 7fb392ffd640 1 -- 172.21.15.1:0/2586839987 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb394018070 con 0x7fb39c1565a0 2024-03-06T07:29:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.327+0000 7fb392ffd640 1 -- 172.21.15.1:0/2586839987 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39400a640 con 0x7fb39c1565a0 2024-03-06T07:29:21.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.327+0000 7fb3a47c9640 1 -- 172.21.15.1:0/2586839987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39c1eab20 con 0x7fb39c1565a0 2024-03-06T07:29:21.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.327+0000 7fb392ffd640 1 -- 172.21.15.1:0/2586839987 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39400aab0 con 0x7fb39c1565a0 2024-03-06T07:29:21.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.327+0000 7fb3a47c9640 1 -- 172.21.15.1:0/2586839987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb39c14c7f0 con 0x7fb39c1565a0 2024-03-06T07:29:21.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.328+0000 7fb3a47c9640 1 -- 172.21.15.1:0/2586839987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb39c067fd0 con 0x7fb39c1565a0 2024-03-06T07:29:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.328+0000 7fb392ffd640 1 -- 172.21.15.1:0/2586839987 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb39400ac50 con 0x7fb39c1565a0 2024-03-06T07:29:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.329+0000 7fb392ffd640 1 --2- 172.21.15.1:0/2586839987 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb38006b4d0 0x7fb38006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.329+0000 7fb392ffd640 1 -- 172.21.15.1:0/2586839987 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fb394089a30 con 0x7fb39c1565a0 2024-03-06T07:29:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.329+0000 7fb3a2565640 1 --2- 172.21.15.1:0/2586839987 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb38006b4d0 0x7fb38006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.329+0000 7fb3a2565640 1 --2- 172.21.15.1:0/2586839987 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb38006b4d0 0x7fb38006d990 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fb398006fd0 tx=0x7fb398008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.333+0000 7fb392ffd640 1 -- 172.21.15.1:0/2586839987 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb394058700 con 0x7fb39c1565a0 2024-03-06T07:29:21.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:21.550+0000 7fb3a47c9640 1 -- 172.21.15.1:0/2586839987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb39c06f280 con 0x7fb39c1565a0 2024-03-06T07:29:22.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.416+0000 7fb392ffd640 1 -- 172.21.15.1:0/2586839987 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v190) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb394058000 con 0x7fb39c1565a0 2024-03-06T07:29:22.418 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:29:22.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.419+0000 7fb3a47c9640 1 -- 172.21.15.1:0/2586839987 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb38006b4d0 msgr2=0x7fb38006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.419+0000 7fb3a47c9640 1 --2- 172.21.15.1:0/2586839987 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb38006b4d0 0x7fb38006d990 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fb398006fd0 tx=0x7fb398008040 comp rx=0 tx=0).stop 2024-03-06T07:29:22.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.419+0000 7fb3a47c9640 1 -- 172.21.15.1:0/2586839987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb39c1565a0 msgr2=0x7fb39c1eb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.419+0000 7fb3a47c9640 1 --2- 172.21.15.1:0/2586839987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb39c1565a0 0x7fb39c1eb2b0 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fb39400c7a0 tx=0x7fb39400cc60 comp rx=0 tx=0).stop 2024-03-06T07:29:22.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.419+0000 7fb3a47c9640 1 -- 172.21.15.1:0/2586839987 shutdown_connections 2024-03-06T07:29:22.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.419+0000 7fb3a47c9640 1 --2- 172.21.15.1:0/2586839987 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb38006b4d0 0x7fb38006d990 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.419+0000 7fb3a47c9640 1 --2- 172.21.15.1:0/2586839987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb39c1565a0 0x7fb39c1eb2b0 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.419+0000 7fb3a47c9640 1 --2- 172.21.15.1:0/2586839987 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb39c149a90 0x7fb39c14c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.419+0000 7fb3a47c9640 1 --2- 172.21.15.1:0/2586839987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c0c1260 0x7fb39c071540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.419+0000 7fb3a47c9640 1 -- 172.21.15.1:0/2586839987 >> 172.21.15.1:0/2586839987 conn(0x7fb39c0c6860 msgr2=0x7fb39c1559a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.419+0000 7fb3a47c9640 1 -- 172.21.15.1:0/2586839987 shutdown_connections 2024-03-06T07:29:22.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.419+0000 7fb3a47c9640 1 -- 172.21.15.1:0/2586839987 wait complete. 2024-03-06T07:29:22.434 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:29:22.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.542+0000 7f4844495640 1 Processor -- start 2024-03-06T07:29:22.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.542+0000 7f4844495640 1 -- start start 2024-03-06T07:29:22.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.543+0000 7f4844495640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f483c0c6b10 0x7f483c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.543+0000 7f4844495640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f483c0bf9d0 0x7f483c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.543+0000 7f4844495640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483c0c2fc0 con 0x7f483c152650 2024-03-06T07:29:22.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.543+0000 7f4844495640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483c155800 con 0x7f483c0bf9d0 2024-03-06T07:29:22.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.543+0000 7f4844495640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c158da0 con 0x7f483c0c6b10 2024-03-06T07:29:22.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.543+0000 7f4842231640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f483c152650 0x7f483c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48270/0 (socket says 172.21.15.1:48270) 2024-03-06T07:29:22.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.543+0000 7f4842231640 1 -- 172.21.15.1:0/1911063362 learned_addr learned my addr 172.21.15.1:0/1911063362 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:22.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f484122f640 1 -- 172.21.15.1:0/1911063362 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696809573 0 0) 0x7f483c0c2fc0 con 0x7f483c152650 2024-03-06T07:29:22.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f4841a30640 1 --2- 172.21.15.1:0/1911063362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f483c0c6b10 0x7f483c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f484122f640 1 -- 172.21.15.1:0/1911063362 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4824003520 con 0x7f483c152650 2024-03-06T07:29:22.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f4842a32640 1 --2- 172.21.15.1:0/1911063362 >> v2:172.21.15.98:3300/0 conn(0x7f483c0bf9d0 0x7f483c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f484122f640 1 -- 172.21.15.1:0/1911063362 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 787807837 0 0) 0x7f4824003520 con 0x7f483c152650 2024-03-06T07:29:22.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f484122f640 1 -- 172.21.15.1:0/1911063362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f483c0c6b10 msgr2=0x7f483c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f484122f640 1 --2- 172.21.15.1:0/1911063362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f483c0c6b10 0x7f483c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f484122f640 1 -- 172.21.15.1:0/1911063362 >> v2:172.21.15.98:3300/0 conn(0x7f483c0bf9d0 msgr2=0x7f483c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f484122f640 1 --2- 172.21.15.1:0/1911063362 >> v2:172.21.15.98:3300/0 conn(0x7f483c0bf9d0 0x7f483c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f484122f640 1 -- 172.21.15.1:0/1911063362 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c0c6690 con 0x7f483c152650 2024-03-06T07:29:22.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f484122f640 1 -- 172.21.15.1:0/1911063362 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f4834001670 con 0x7f483c152650 2024-03-06T07:29:22.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f484122f640 1 -- 172.21.15.1:0/1911063362 >> v1:172.21.15.1:6789/0 conn(0x7f483c152650 legacy=0x7f483c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f484122f640 1 --2- 172.21.15.1:0/1911063362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f48240039b0 0x7f4824003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f484122f640 1 --2- 172.21.15.1:0/1911063362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4824004310 0x7f4824004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f484122f640 1 --2- 172.21.15.1:0/1911063362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4824004e50 0x7f48240052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f484122f640 1 -- 172.21.15.1:0/1911063362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c155800 con 0x7f4824004310 2024-03-06T07:29:22.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f484122f640 1 -- 172.21.15.1:0/1911063362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c158da0 con 0x7f4824004e50 2024-03-06T07:29:22.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f484122f640 1 -- 172.21.15.1:0/1911063362 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482400bde0 con 0x7f48240039b0 2024-03-06T07:29:22.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f4841a30640 1 --2- 172.21.15.1:0/1911063362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4824004310 0x7f4824004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f4842231640 1 --2- 172.21.15.1:0/1911063362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f48240039b0 0x7f4824003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f4842a32640 1 --2- 172.21.15.1:0/1911063362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4824004e50 0x7f48240052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f4841a30640 1 -- 172.21.15.1:0/1911063362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f48240039b0 msgr2=0x7f4824003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f4841a30640 1 --2- 172.21.15.1:0/1911063362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f48240039b0 0x7f4824003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f4841a30640 1 -- 172.21.15.1:0/1911063362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4824004e50 msgr2=0x7f48240052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f4841a30640 1 --2- 172.21.15.1:0/1911063362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4824004e50 0x7f48240052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.544+0000 7f4841a30640 1 -- 172.21.15.1:0/1911063362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4824003520 con 0x7f4824004310 2024-03-06T07:29:22.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.545+0000 7f4841a30640 1 --2- 172.21.15.1:0/1911063362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4824004310 0x7f4824004800 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f4830009ac0 tx=0x7f4824005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58790fcf864bb3d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.545+0000 7f484122f640 1 -- 172.21.15.1:0/1911063362 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4830039070 con 0x7f4824004310 2024-03-06T07:29:22.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.545+0000 7f484122f640 1 -- 172.21.15.1:0/1911063362 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483002c6b0 con 0x7f4824004310 2024-03-06T07:29:22.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.545+0000 7f484122f640 1 -- 172.21.15.1:0/1911063362 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4830009690 con 0x7f4824004310 2024-03-06T07:29:22.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.546+0000 7f4844495640 1 -- 172.21.15.1:0/1911063362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4824004310 msgr2=0x7f4824004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.546+0000 7f4844495640 1 --2- 172.21.15.1:0/1911063362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4824004310 0x7f4824004800 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f4830009ac0 tx=0x7f4824005830 comp rx=0 tx=0).stop 2024-03-06T07:29:22.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.546+0000 7f4842a32640 1 -- 172.21.15.1:0/1911063362 reap_dead start 2024-03-06T07:29:22.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.546+0000 7f4844495640 1 -- 172.21.15.1:0/1911063362 shutdown_connections 2024-03-06T07:29:22.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.546+0000 7f4844495640 1 -- 172.21.15.1:0/1911063362 >> 172.21.15.1:0/1911063362 conn(0x7f483c066ee0 msgr2=0x7f483c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.546+0000 7f4844495640 1 -- 172.21.15.1:0/1911063362 shutdown_connections 2024-03-06T07:29:22.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.546+0000 7f4844495640 1 -- 172.21.15.1:0/1911063362 wait complete. 2024-03-06T07:29:22.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.547+0000 7f4844495640 1 Processor -- start 2024-03-06T07:29:22.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.547+0000 7f4844495640 1 -- start start 2024-03-06T07:29:22.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4844495640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4824004310 0x7f483c1ea410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4844495640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483c158230 0x7f483c0c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4844495640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f483c0c9d60 0x7f483c14b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4844495640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c1551b0 con 0x7f483c158230 2024-03-06T07:29:22.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4844495640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c0c6fa0 con 0x7f4824004310 2024-03-06T07:29:22.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4844495640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c155490 con 0x7f483c0c9d60 2024-03-06T07:29:22.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4841a30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483c158230 0x7f483c0c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4841a30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483c158230 0x7f483c0c9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57430/0 (socket says 172.21.15.1:57430) 2024-03-06T07:29:22.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4841a30640 1 -- 172.21.15.1:0/613582735 learned_addr learned my addr 172.21.15.1:0/613582735 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:22.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4842231640 1 --2- 172.21.15.1:0/613582735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4824004310 0x7f483c1ea410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4842a32640 1 --2- 172.21.15.1:0/613582735 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f483c0c9d60 0x7f483c14b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4841a30640 1 -- 172.21.15.1:0/613582735 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f483c0c9d60 msgr2=0x7f483c14b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4841a30640 1 --2- 172.21.15.1:0/613582735 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f483c0c9d60 0x7f483c14b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4841a30640 1 -- 172.21.15.1:0/613582735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4824004310 msgr2=0x7f483c1ea410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4841a30640 1 --2- 172.21.15.1:0/613582735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4824004310 0x7f483c1ea410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4841a30640 1 -- 172.21.15.1:0/613582735 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c1eacd0 con 0x7f483c158230 2024-03-06T07:29:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4842a32640 1 --2- 172.21.15.1:0/613582735 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f483c0c9d60 0x7f483c14b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f4841a30640 1 --2- 172.21.15.1:0/613582735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483c158230 0x7f483c0c9820 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f4830037040 tx=0x7f4830004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.548+0000 7f483affd640 1 -- 172.21.15.1:0/613582735 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4830042020 con 0x7f483c158230 2024-03-06T07:29:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.549+0000 7f483affd640 1 -- 172.21.15.1:0/613582735 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483003a040 con 0x7f483c158230 2024-03-06T07:29:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.549+0000 7f4844495640 1 -- 172.21.15.1:0/613582735 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483c1eafb0 con 0x7f483c158230 2024-03-06T07:29:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.549+0000 7f483affd640 1 -- 172.21.15.1:0/613582735 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4830039070 con 0x7f483c158230 2024-03-06T07:29:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.549+0000 7f4844495640 1 -- 172.21.15.1:0/613582735 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483c14af90 con 0x7f483c158230 2024-03-06T07:29:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.549+0000 7f4844495640 1 -- 172.21.15.1:0/613582735 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f483c067fd0 con 0x7f483c158230 2024-03-06T07:29:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.550+0000 7f483affd640 1 -- 172.21.15.1:0/613582735 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4830045020 con 0x7f483c158230 2024-03-06T07:29:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.550+0000 7f483affd640 1 --2- 172.21.15.1:0/613582735 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f482806b500 0x7f482806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.550+0000 7f483affd640 1 -- 172.21.15.1:0/613582735 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f48300ab8d0 con 0x7f483c158230 2024-03-06T07:29:22.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.551+0000 7f4842231640 1 --2- 172.21.15.1:0/613582735 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f482806b500 0x7f482806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.551+0000 7f4842231640 1 --2- 172.21.15.1:0/613582735 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f482806b500 0x7f482806d9c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f48340015d0 tx=0x7f48340078f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.555+0000 7f483affd640 1 -- 172.21.15.1:0/613582735 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f483004c030 con 0x7f483c158230 2024-03-06T07:29:22.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:22.775+0000 7f4844495640 1 -- 172.21.15.1:0/613582735 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f483c0b06f0 con 0x7f483c158230 2024-03-06T07:29:23.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.421+0000 7f483affd640 1 -- 172.21.15.1:0/613582735 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v191) v1 ==== 104+0+0 (secure 0 0 0) 0x7f483007a620 con 0x7f483c158230 2024-03-06T07:29:23.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.423+0000 7f4844495640 1 -- 172.21.15.1:0/613582735 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f482806b500 msgr2=0x7f482806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.423+0000 7f4844495640 1 --2- 172.21.15.1:0/613582735 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f482806b500 0x7f482806d9c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f48340015d0 tx=0x7f48340078f0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.423+0000 7f4844495640 1 -- 172.21.15.1:0/613582735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483c158230 msgr2=0x7f483c0c9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.423+0000 7f4844495640 1 --2- 172.21.15.1:0/613582735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483c158230 0x7f483c0c9820 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f4830037040 tx=0x7f4830004730 comp rx=0 tx=0).stop 2024-03-06T07:29:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.423+0000 7f4844495640 1 -- 172.21.15.1:0/613582735 shutdown_connections 2024-03-06T07:29:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.423+0000 7f4844495640 1 --2- 172.21.15.1:0/613582735 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f482806b500 0x7f482806d9c0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.423+0000 7f4844495640 1 --2- 172.21.15.1:0/613582735 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f483c0c9d60 0x7f483c14b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.423+0000 7f4844495640 1 --2- 172.21.15.1:0/613582735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483c158230 0x7f483c0c9820 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.423+0000 7f4844495640 1 --2- 172.21.15.1:0/613582735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4824004310 0x7f483c1ea410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.423+0000 7f4844495640 1 -- 172.21.15.1:0/613582735 >> 172.21.15.1:0/613582735 conn(0x7f483c066ee0 msgr2=0x7f483c0c39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.423+0000 7f4844495640 1 -- 172.21.15.1:0/613582735 shutdown_connections 2024-03-06T07:29:23.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.423+0000 7f4844495640 1 -- 172.21.15.1:0/613582735 wait complete. 2024-03-06T07:29:23.438 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:29:23.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.545+0000 7ff7b53d6640 1 Processor -- start 2024-03-06T07:29:23.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.545+0000 7ff7b53d6640 1 -- start start 2024-03-06T07:29:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.546+0000 7ff7b53d6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff7b014ee10 0x7ff7b014f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.546+0000 7ff7b53d6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7b00c6f00 con 0x7ff7b00c58c0 2024-03-06T07:29:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.546+0000 7ff7b53d6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7b00c71a0 con 0x7ff7b00c1060 2024-03-06T07:29:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.546+0000 7ff7b53d6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b014f760 con 0x7ff7b014ee10 2024-03-06T07:29:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.546+0000 7ff7af7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff7b014ee10 0x7ff7b014f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.546+0000 7ff7ae7fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff7b00c58c0 0x7ff7b00a6d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48282/0 (socket says 172.21.15.1:48282) 2024-03-06T07:29:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.546+0000 7ff7ae7fc640 1 -- 172.21.15.1:0/93707494 learned_addr learned my addr 172.21.15.1:0/93707494 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.546+0000 7ff7af7fe640 1 -- 172.21.15.1:0/93707494 >> v1:172.21.15.98:6789/0 conn(0x7ff7b00c1060 legacy=0x7ff7b0155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.546+0000 7ff7af7fe640 1 -- 172.21.15.1:0/93707494 >> v1:172.21.15.1:6789/0 conn(0x7ff7b00c58c0 legacy=0x7ff7b00a6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.546+0000 7ff7af7fe640 1 -- 172.21.15.1:0/93707494 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b00c7480 con 0x7ff7b014ee10 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.547+0000 7ff7af7fe640 1 --2- 172.21.15.1:0/93707494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7b014ee10 0x7ff7b014f220 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0009d90 tx=0x7ff7a002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74b85f661654a15 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.547+0000 7ff7adffb640 1 -- 172.21.15.1:0/93707494 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0039070 con 0x7ff7b014ee10 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.547+0000 7ff7adffb640 1 -- 172.21.15.1:0/93707494 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a002cbd0 con 0x7ff7b014ee10 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.547+0000 7ff7adffb640 1 -- 172.21.15.1:0/93707494 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0034440 con 0x7ff7b014ee10 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.547+0000 7ff7b53d6640 1 -- 172.21.15.1:0/93707494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7b014ee10 msgr2=0x7ff7b014f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.547+0000 7ff7b53d6640 1 --2- 172.21.15.1:0/93707494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7b014ee10 0x7ff7b014f220 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0009d90 tx=0x7ff7a002be20 comp rx=0 tx=0).stop 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.547+0000 7ff7b53d6640 1 -- 172.21.15.1:0/93707494 shutdown_connections 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.547+0000 7ff7b53d6640 1 --2- 172.21.15.1:0/93707494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7b014ee10 0x7ff7b014f220 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.548+0000 7ff7b53d6640 1 -- 172.21.15.1:0/93707494 >> 172.21.15.1:0/93707494 conn(0x7ff7b00c44c0 msgr2=0x7ff7b00c48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.548+0000 7ff7b53d6640 1 -- 172.21.15.1:0/93707494 shutdown_connections 2024-03-06T07:29:23.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.548+0000 7ff7b53d6640 1 -- 172.21.15.1:0/93707494 wait complete. 2024-03-06T07:29:23.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.549+0000 7ff7b53d6640 1 Processor -- start 2024-03-06T07:29:23.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.549+0000 7ff7b53d6640 1 -- start start 2024-03-06T07:29:23.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.549+0000 7ff7b53d6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7b00c1060 0x7ff7b0151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.549+0000 7ff7b53d6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7b00c58c0 0x7ff7b01eb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.549+0000 7ff7b53d6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7b014ee10 0x7ff7b0151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.549+0000 7ff7b53d6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b00a7440 con 0x7ff7b014ee10 2024-03-06T07:29:23.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.549+0000 7ff7b53d6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b0155710 con 0x7ff7b00c1060 2024-03-06T07:29:23.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.549+0000 7ff7b53d6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b00c5d50 con 0x7ff7b00c58c0 2024-03-06T07:29:23.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.550+0000 7ff7af7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7b014ee10 0x7ff7b0151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.550+0000 7ff7af7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7b014ee10 0x7ff7b0151e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57460/0 (socket says 172.21.15.1:57460) 2024-03-06T07:29:23.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.550+0000 7ff7af7fe640 1 -- 172.21.15.1:0/2948688235 learned_addr learned my addr 172.21.15.1:0/2948688235 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:23.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.550+0000 7ff7ae7fc640 1 --2- 172.21.15.1:0/2948688235 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7b00c58c0 0x7ff7b01eb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.550+0000 7ff7aeffd640 1 --2- 172.21.15.1:0/2948688235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7b00c1060 0x7ff7b0151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.550+0000 7ff7af7fe640 1 -- 172.21.15.1:0/2948688235 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7b00c58c0 msgr2=0x7ff7b01eb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.550+0000 7ff7af7fe640 1 --2- 172.21.15.1:0/2948688235 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7b00c58c0 0x7ff7b01eb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.550+0000 7ff7af7fe640 1 -- 172.21.15.1:0/2948688235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7b00c1060 msgr2=0x7ff7b0151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.550+0000 7ff7af7fe640 1 --2- 172.21.15.1:0/2948688235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7b00c1060 0x7ff7b0151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.550+0000 7ff7af7fe640 1 -- 172.21.15.1:0/2948688235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b0152390 con 0x7ff7b014ee10 2024-03-06T07:29:23.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.550+0000 7ff7af7fe640 1 --2- 172.21.15.1:0/2948688235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7b014ee10 0x7ff7b0151e50 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0009d90 tx=0x7ff7a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.550+0000 7ff7a77fe640 1 -- 172.21.15.1:0/2948688235 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0039050 con 0x7ff7b014ee10 2024-03-06T07:29:23.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.550+0000 7ff7a77fe640 1 -- 172.21.15.1:0/2948688235 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a003a040 con 0x7ff7b014ee10 2024-03-06T07:29:23.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.550+0000 7ff7b53d6640 1 -- 172.21.15.1:0/2948688235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b0147d80 con 0x7ff7b014ee10 2024-03-06T07:29:23.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.550+0000 7ff7a77fe640 1 -- 172.21.15.1:0/2948688235 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a003e610 con 0x7ff7b014ee10 2024-03-06T07:29:23.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.550+0000 7ff7b53d6640 1 -- 172.21.15.1:0/2948688235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b00701f0 con 0x7ff7b014ee10 2024-03-06T07:29:23.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.551+0000 7ff7b53d6640 1 -- 172.21.15.1:0/2948688235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b006c2b0 con 0x7ff7b014ee10 2024-03-06T07:29:23.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.552+0000 7ff7a77fe640 1 -- 172.21.15.1:0/2948688235 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff7a0047020 con 0x7ff7b014ee10 2024-03-06T07:29:23.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.552+0000 7ff7a77fe640 1 --2- 172.21.15.1:0/2948688235 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff79006b4d0 0x7ff79006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.552+0000 7ff7a77fe640 1 -- 172.21.15.1:0/2948688235 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ff7a00ac7c0 con 0x7ff7b014ee10 2024-03-06T07:29:23.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.552+0000 7ff7aeffd640 1 --2- 172.21.15.1:0/2948688235 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff79006b4d0 0x7ff79006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.553+0000 7ff7aeffd640 1 --2- 172.21.15.1:0/2948688235 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff79006b4d0 0x7ff79006d990 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8004ea0 tx=0x7ff7a8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.557+0000 7ff7a77fe640 1 -- 172.21.15.1:0/2948688235 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7b006c2b0 con 0x7ff7b014ee10 2024-03-06T07:29:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:23.738+0000 7ff7b53d6640 1 -- 172.21.15.1:0/2948688235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff7b00680b0 con 0x7ff7b014ee10 2024-03-06T07:29:24.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.432+0000 7ff7a77fe640 1 -- 172.21.15.1:0/2948688235 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v200) v1 ==== 181+0+0 (secure 0 0 0) 0x7ff7a004e020 con 0x7ff7b014ee10 2024-03-06T07:29:24.433 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:29:24.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.434+0000 7ff7b53d6640 1 -- 172.21.15.1:0/2948688235 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff79006b4d0 msgr2=0x7ff79006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.434+0000 7ff7b53d6640 1 --2- 172.21.15.1:0/2948688235 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff79006b4d0 0x7ff79006d990 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8004ea0 tx=0x7ff7a8009040 comp rx=0 tx=0).stop 2024-03-06T07:29:24.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.434+0000 7ff7b53d6640 1 -- 172.21.15.1:0/2948688235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7b014ee10 msgr2=0x7ff7b0151e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.434+0000 7ff7b53d6640 1 --2- 172.21.15.1:0/2948688235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7b014ee10 0x7ff7b0151e50 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0009d90 tx=0x7ff7a0004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:24.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.435+0000 7ff7b53d6640 1 -- 172.21.15.1:0/2948688235 shutdown_connections 2024-03-06T07:29:24.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.435+0000 7ff7b53d6640 1 --2- 172.21.15.1:0/2948688235 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff79006b4d0 0x7ff79006d990 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.435+0000 7ff7b53d6640 1 --2- 172.21.15.1:0/2948688235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7b014ee10 0x7ff7b0151e50 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.435+0000 7ff7b53d6640 1 --2- 172.21.15.1:0/2948688235 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7b00c58c0 0x7ff7b01eb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.435+0000 7ff7b53d6640 1 --2- 172.21.15.1:0/2948688235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7b00c1060 0x7ff7b0151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.435+0000 7ff7b53d6640 1 -- 172.21.15.1:0/2948688235 >> 172.21.15.1:0/2948688235 conn(0x7ff7b00c44c0 msgr2=0x7ff7b01e1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:24.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.435+0000 7ff7b53d6640 1 -- 172.21.15.1:0/2948688235 shutdown_connections 2024-03-06T07:29:24.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.435+0000 7ff7b53d6640 1 -- 172.21.15.1:0/2948688235 wait complete. 2024-03-06T07:29:24.454 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:29:24.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.563+0000 7fd69753e640 1 Processor -- start 2024-03-06T07:29:24.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.563+0000 7fd69753e640 1 -- start start 2024-03-06T07:29:24.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd69753e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd690157160 0x7fd690160b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd69753e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd690158560 con 0x7fd690158180 2024-03-06T07:29:24.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd69753e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd690155600 con 0x7fd6900c6800 2024-03-06T07:29:24.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd69753e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd690156210 con 0x7fd690157160 2024-03-06T07:29:24.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd6952da640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd690158180 0x7fd6900c58e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60048/0 (socket says 172.21.15.1:60048) 2024-03-06T07:29:24.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd6952da640 1 -- 172.21.15.1:0/3547992448 learned_addr learned my addr 172.21.15.1:0/3547992448 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:24.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd695adb640 1 --2- 172.21.15.1:0/3547992448 >> v2:172.21.15.98:3300/0 conn(0x7fd690157160 0x7fd690160b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd68ffff640 1 -- 172.21.15.1:0/3547992448 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184593561 0 0) 0x7fd690158560 con 0x7fd690158180 2024-03-06T07:29:24.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd68ffff640 1 -- 172.21.15.1:0/3547992448 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd678003520 con 0x7fd690158180 2024-03-06T07:29:24.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd68ffff640 1 -- 172.21.15.1:0/3547992448 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2822518407 0 0) 0x7fd678003520 con 0x7fd690158180 2024-03-06T07:29:24.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd68ffff640 1 -- 172.21.15.1:0/3547992448 >> v2:172.21.15.98:3300/0 conn(0x7fd690157160 msgr2=0x7fd690160b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:24.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd68ffff640 1 --2- 172.21.15.1:0/3547992448 >> v2:172.21.15.98:3300/0 conn(0x7fd690157160 0x7fd690160b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd68ffff640 1 -- 172.21.15.1:0/3547992448 >> v1:172.21.15.98:6789/0 conn(0x7fd6900c6800 legacy=0x7fd690156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd68ffff640 1 -- 172.21.15.1:0/3547992448 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6901558e0 con 0x7fd690158180 2024-03-06T07:29:24.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd68ffff640 1 -- 172.21.15.1:0/3547992448 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fd688001670 con 0x7fd690158180 2024-03-06T07:29:24.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd68ffff640 1 -- 172.21.15.1:0/3547992448 >> v1:172.21.15.1:6789/0 conn(0x7fd690158180 legacy=0x7fd6900c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd68ffff640 1 --2- 172.21.15.1:0/3547992448 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6780039b0 0x7fd678003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd68ffff640 1 --2- 172.21.15.1:0/3547992448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd678004800 0x7fd678004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd68ffff640 1 --2- 172.21.15.1:0/3547992448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6780052e0 0x7fd67800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd68ffff640 1 -- 172.21.15.1:0/3547992448 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd690156210 con 0x7fd6780052e0 2024-03-06T07:29:24.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd68ffff640 1 -- 172.21.15.1:0/3547992448 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67800c1b0 con 0x7fd678004800 2024-03-06T07:29:24.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd68ffff640 1 -- 172.21.15.1:0/3547992448 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67800c3b0 con 0x7fd6780039b0 2024-03-06T07:29:24.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd695adb640 1 --2- 172.21.15.1:0/3547992448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6780052e0 0x7fd67800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd694ad9640 1 --2- 172.21.15.1:0/3547992448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd678004800 0x7fd678004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd6952da640 1 --2- 172.21.15.1:0/3547992448 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6780039b0 0x7fd678003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd695adb640 1 -- 172.21.15.1:0/3547992448 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6780039b0 msgr2=0x7fd678003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:24.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd695adb640 1 --2- 172.21.15.1:0/3547992448 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6780039b0 0x7fd678003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd695adb640 1 -- 172.21.15.1:0/3547992448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd678004800 msgr2=0x7fd678004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:24.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd695adb640 1 --2- 172.21.15.1:0/3547992448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd678004800 0x7fd678004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.565+0000 7fd695adb640 1 -- 172.21.15.1:0/3547992448 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd678003520 con 0x7fd6780052e0 2024-03-06T07:29:24.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.566+0000 7fd695adb640 1 -- 172.21.15.1:0/3547992448 reap_dead start 2024-03-06T07:29:24.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.566+0000 7fd695adb640 1 --2- 172.21.15.1:0/3547992448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6780052e0 0x7fd67800b9c0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fd680009f90 tx=0x7fd678005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15794775c4a061ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.566+0000 7fd68ffff640 1 -- 172.21.15.1:0/3547992448 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd680009c10 con 0x7fd6780052e0 2024-03-06T07:29:24.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.566+0000 7fd68ffff640 1 -- 172.21.15.1:0/3547992448 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd68002cc20 con 0x7fd6780052e0 2024-03-06T07:29:24.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.566+0000 7fd68ffff640 1 -- 172.21.15.1:0/3547992448 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd680003990 con 0x7fd6780052e0 2024-03-06T07:29:24.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.566+0000 7fd69753e640 1 -- 172.21.15.1:0/3547992448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6780052e0 msgr2=0x7fd67800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.566+0000 7fd69753e640 1 --2- 172.21.15.1:0/3547992448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6780052e0 0x7fd67800b9c0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fd680009f90 tx=0x7fd678005830 comp rx=0 tx=0).stop 2024-03-06T07:29:24.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.566+0000 7fd69753e640 1 -- 172.21.15.1:0/3547992448 shutdown_connections 2024-03-06T07:29:24.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.566+0000 7fd69753e640 1 --2- 172.21.15.1:0/3547992448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6780052e0 0x7fd67800b9c0 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.566+0000 7fd69753e640 1 -- 172.21.15.1:0/3547992448 >> 172.21.15.1:0/3547992448 conn(0x7fd69006b410 msgr2=0x7fd69006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:24.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.566+0000 7fd69753e640 1 -- 172.21.15.1:0/3547992448 shutdown_connections 2024-03-06T07:29:24.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.566+0000 7fd69753e640 1 -- 172.21.15.1:0/3547992448 wait complete. 2024-03-06T07:29:24.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.568+0000 7fd69753e640 1 Processor -- start 2024-03-06T07:29:24.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.568+0000 7fd69753e640 1 -- start start 2024-03-06T07:29:24.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.568+0000 7fd69753e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6780052e0 0x7fd690152d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.568+0000 7fd69753e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6900acfb0 0x7fd690143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.568+0000 7fd69753e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd690144100 0x7fd6900ab3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.568+0000 7fd69753e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd690155ea0 con 0x7fd6900acfb0 2024-03-06T07:29:24.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.568+0000 7fd69753e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6900c3ea0 con 0x7fd6780052e0 2024-03-06T07:29:24.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.568+0000 7fd69753e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6901551f0 con 0x7fd690144100 2024-03-06T07:29:24.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.568+0000 7fd694ad9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6900acfb0 0x7fd690143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.568+0000 7fd694ad9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6900acfb0 0x7fd690143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59138/0 (socket says 172.21.15.1:59138) 2024-03-06T07:29:24.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.568+0000 7fd694ad9640 1 -- 172.21.15.1:0/1084320607 learned_addr learned my addr 172.21.15.1:0/1084320607 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:24.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.569+0000 7fd6952da640 1 --2- 172.21.15.1:0/1084320607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6780052e0 0x7fd690152d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.569+0000 7fd695adb640 1 --2- 172.21.15.1:0/1084320607 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd690144100 0x7fd6900ab3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.569+0000 7fd694ad9640 1 -- 172.21.15.1:0/1084320607 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd690144100 msgr2=0x7fd6900ab3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.569+0000 7fd694ad9640 1 --2- 172.21.15.1:0/1084320607 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd690144100 0x7fd6900ab3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.569+0000 7fd694ad9640 1 -- 172.21.15.1:0/1084320607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6780052e0 msgr2=0x7fd690152d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.569+0000 7fd694ad9640 1 --2- 172.21.15.1:0/1084320607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6780052e0 0x7fd690152d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.569+0000 7fd694ad9640 1 -- 172.21.15.1:0/1084320607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69014f900 con 0x7fd6900acfb0 2024-03-06T07:29:24.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.569+0000 7fd695adb640 1 --2- 172.21.15.1:0/1084320607 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd690144100 0x7fd6900ab3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:24.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.569+0000 7fd6952da640 1 --2- 172.21.15.1:0/1084320607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6780052e0 0x7fd690152d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:24.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.569+0000 7fd694ad9640 1 --2- 172.21.15.1:0/1084320607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6900acfb0 0x7fd690143bc0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fd68400f5f0 tx=0x7fd68400fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.569+0000 7fd68dffb640 1 -- 172.21.15.1:0/1084320607 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6840052e0 con 0x7fd6900acfb0 2024-03-06T07:29:24.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.569+0000 7fd68dffb640 1 -- 172.21.15.1:0/1084320607 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd684018070 con 0x7fd6900acfb0 2024-03-06T07:29:24.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.569+0000 7fd68dffb640 1 -- 172.21.15.1:0/1084320607 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684003810 con 0x7fd6900acfb0 2024-03-06T07:29:24.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.569+0000 7fd69753e640 1 -- 172.21.15.1:0/1084320607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69014fb70 con 0x7fd6900acfb0 2024-03-06T07:29:24.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.569+0000 7fd69753e640 1 -- 172.21.15.1:0/1084320607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6900aa8e0 con 0x7fd6900acfb0 2024-03-06T07:29:24.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.570+0000 7fd69753e640 1 -- 172.21.15.1:0/1084320607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6901444e0 con 0x7fd6900acfb0 2024-03-06T07:29:24.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.571+0000 7fd68dffb640 1 -- 172.21.15.1:0/1084320607 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6840039b0 con 0x7fd6900acfb0 2024-03-06T07:29:24.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.571+0000 7fd68dffb640 1 --2- 172.21.15.1:0/1084320607 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd67c06b4d0 0x7fd67c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.571+0000 7fd68dffb640 1 -- 172.21.15.1:0/1084320607 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 5721+0+0 (secure 0 0 0) 0x7fd684089ef0 con 0x7fd6900acfb0 2024-03-06T07:29:24.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.571+0000 7fd6952da640 1 --2- 172.21.15.1:0/1084320607 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd67c06b4d0 0x7fd67c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.572+0000 7fd6952da640 1 --2- 172.21.15.1:0/1084320607 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd67c06b4d0 0x7fd67c06d990 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fd6880017d0 tx=0x7fd688009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.576+0000 7fd68dffb640 1 -- 172.21.15.1:0/1084320607 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd684058df0 con 0x7fd6900acfb0 2024-03-06T07:29:24.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:24.756+0000 7fd69753e640 1 -- 172.21.15.1:0/1084320607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd690068a80 con 0x7fd6900acfb0 2024-03-06T07:29:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.439+0000 7fd68dffb640 1 -- 172.21.15.1:0/1084320607 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v201) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd6840586f0 con 0x7fd6900acfb0 2024-03-06T07:29:25.441 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-06T07:29:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.443+0000 7fd69753e640 1 -- 172.21.15.1:0/1084320607 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd67c06b4d0 msgr2=0x7fd67c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.443+0000 7fd69753e640 1 --2- 172.21.15.1:0/1084320607 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd67c06b4d0 0x7fd67c06d990 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fd6880017d0 tx=0x7fd688009040 comp rx=0 tx=0).stop 2024-03-06T07:29:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.443+0000 7fd69753e640 1 -- 172.21.15.1:0/1084320607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6900acfb0 msgr2=0x7fd690143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.443+0000 7fd69753e640 1 --2- 172.21.15.1:0/1084320607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6900acfb0 0x7fd690143bc0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fd68400f5f0 tx=0x7fd68400fab0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.443+0000 7fd69753e640 1 -- 172.21.15.1:0/1084320607 shutdown_connections 2024-03-06T07:29:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.443+0000 7fd69753e640 1 --2- 172.21.15.1:0/1084320607 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd67c06b4d0 0x7fd67c06d990 secure :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fd6880017d0 tx=0x7fd688009040 comp rx=0 tx=0).stop 2024-03-06T07:29:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.443+0000 7fd69753e640 1 --2- 172.21.15.1:0/1084320607 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd690144100 0x7fd6900ab3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.443+0000 7fd69753e640 1 --2- 172.21.15.1:0/1084320607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6900acfb0 0x7fd690143bc0 secure :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fd68400f5f0 tx=0x7fd68400fab0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.443+0000 7fd69753e640 1 --2- 172.21.15.1:0/1084320607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6780052e0 0x7fd690152d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.443+0000 7fd69753e640 1 -- 172.21.15.1:0/1084320607 >> 172.21.15.1:0/1084320607 conn(0x7fd69006b410 msgr2=0x7fd690153f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:25.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.443+0000 7fd69753e640 1 -- 172.21.15.1:0/1084320607 shutdown_connections 2024-03-06T07:29:25.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.443+0000 7fd69753e640 1 -- 172.21.15.1:0/1084320607 wait complete. 2024-03-06T07:29:25.462 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:29:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.569+0000 7fc044793640 1 Processor -- start 2024-03-06T07:29:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.569+0000 7fc044793640 1 -- start start 2024-03-06T07:29:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.570+0000 7fc044793640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc03c156500 0x7fc03c152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.570+0000 7fc044793640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc03c06b510 0x7fc03c06b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.570+0000 7fc044793640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c1569e0 con 0x7fc03c14cdc0 2024-03-06T07:29:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.570+0000 7fc044793640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc03c0c2dd0 con 0x7fc03c156500 2024-03-06T07:29:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.570+0000 7fc044793640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc03c156cc0 con 0x7fc03c06b510 2024-03-06T07:29:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.570+0000 7fc041d2e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc03c156500 0x7fc03c152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.570+0000 7fc041d2e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc03c156500 0x7fc03c152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59142/0 (socket says 172.21.15.1:59142) 2024-03-06T07:29:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.570+0000 7fc041d2e640 1 -- 172.21.15.1:0/378095804 learned_addr learned my addr 172.21.15.1:0/378095804 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.570+0000 7fc041d2e640 1 -- 172.21.15.1:0/378095804 >> v2:172.21.15.98:3300/0 conn(0x7fc03c06b510 msgr2=0x7fc03c06b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.570+0000 7fc041d2e640 1 --2- 172.21.15.1:0/378095804 >> v2:172.21.15.98:3300/0 conn(0x7fc03c06b510 0x7fc03c06b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.570+0000 7fc041d2e640 1 -- 172.21.15.1:0/378095804 >> v1:172.21.15.98:6789/0 conn(0x7fc03c14cdc0 legacy=0x7fc03c0c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.570+0000 7fc041d2e640 1 -- 172.21.15.1:0/378095804 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c156fa0 con 0x7fc03c156500 2024-03-06T07:29:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.571+0000 7fc041d2e640 1 --2- 172.21.15.1:0/378095804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc03c156500 0x7fc03c152f10 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fc03c157620 tx=0x7fc03002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7690fcb6773fc94 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.571+0000 7fc04152d640 1 -- 172.21.15.1:0/378095804 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc030035020 con 0x7fc03c156500 2024-03-06T07:29:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.571+0000 7fc04152d640 1 -- 172.21.15.1:0/378095804 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc030031870 con 0x7fc03c156500 2024-03-06T07:29:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.571+0000 7fc04152d640 1 -- 172.21.15.1:0/378095804 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc030031c40 con 0x7fc03c156500 2024-03-06T07:29:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.572+0000 7fc044793640 1 -- 172.21.15.1:0/378095804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc03c156500 msgr2=0x7fc03c152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.572+0000 7fc044793640 1 --2- 172.21.15.1:0/378095804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc03c156500 0x7fc03c152f10 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fc03c157620 tx=0x7fc03002b910 comp rx=0 tx=0).stop 2024-03-06T07:29:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.572+0000 7fc044793640 1 -- 172.21.15.1:0/378095804 shutdown_connections 2024-03-06T07:29:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.572+0000 7fc044793640 1 --2- 172.21.15.1:0/378095804 >> v2:172.21.15.98:3300/0 conn(0x7fc03c06b510 0x7fc03c06b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.572+0000 7fc044793640 1 --2- 172.21.15.1:0/378095804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc03c156500 0x7fc03c152f10 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.572+0000 7fc044793640 1 -- 172.21.15.1:0/378095804 >> 172.21.15.1:0/378095804 conn(0x7fc03c0a9700 msgr2=0x7fc03c0a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.572+0000 7fc044793640 1 -- 172.21.15.1:0/378095804 shutdown_connections 2024-03-06T07:29:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.572+0000 7fc044793640 1 -- 172.21.15.1:0/378095804 wait complete. 2024-03-06T07:29:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.573+0000 7fc044793640 1 Processor -- start 2024-03-06T07:29:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.573+0000 7fc044793640 1 -- start start 2024-03-06T07:29:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc044793640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc03c06b510 0x7fc03c1e00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc044793640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc03c14cdc0 0x7fc03c1eeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc044793640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc03c156500 0x7fc03c1ef060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc044793640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c06bea0 con 0x7fc03c156500 2024-03-06T07:29:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc044793640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c0c5a70 con 0x7fc03c06b510 2024-03-06T07:29:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc044793640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c0c65e0 con 0x7fc03c14cdc0 2024-03-06T07:29:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc042d30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc03c156500 0x7fc03c1ef060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc042d30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc03c156500 0x7fc03c1ef060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59150/0 (socket says 172.21.15.1:59150) 2024-03-06T07:29:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc042d30640 1 -- 172.21.15.1:0/2723821452 learned_addr learned my addr 172.21.15.1:0/2723821452 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc041d2e640 1 --2- 172.21.15.1:0/2723821452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc03c14cdc0 0x7fc03c1eeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc04252f640 1 --2- 172.21.15.1:0/2723821452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc03c06b510 0x7fc03c1e00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc042d30640 1 -- 172.21.15.1:0/2723821452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc03c14cdc0 msgr2=0x7fc03c1eeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc042d30640 1 --2- 172.21.15.1:0/2723821452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc03c14cdc0 0x7fc03c1eeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc042d30640 1 -- 172.21.15.1:0/2723821452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc03c06b510 msgr2=0x7fc03c1e00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc042d30640 1 --2- 172.21.15.1:0/2723821452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc03c06b510 0x7fc03c1e00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc042d30640 1 -- 172.21.15.1:0/2723821452 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c1ef5a0 con 0x7fc03c156500 2024-03-06T07:29:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc04252f640 1 --2- 172.21.15.1:0/2723821452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc03c06b510 0x7fc03c1e00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.574+0000 7fc042d30640 1 --2- 172.21.15.1:0/2723821452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc03c156500 0x7fc03c1ef060 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fc02c00b970 tx=0x7fc02c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.575+0000 7fc03affd640 1 -- 172.21.15.1:0/2723821452 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c019070 con 0x7fc03c156500 2024-03-06T07:29:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.575+0000 7fc03affd640 1 -- 172.21.15.1:0/2723821452 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02c00cb60 con 0x7fc03c156500 2024-03-06T07:29:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.575+0000 7fc044793640 1 -- 172.21.15.1:0/2723821452 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03c0abac0 con 0x7fc03c156500 2024-03-06T07:29:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.575+0000 7fc03affd640 1 -- 172.21.15.1:0/2723821452 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c014440 con 0x7fc03c156500 2024-03-06T07:29:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.575+0000 7fc044793640 1 -- 172.21.15.1:0/2723821452 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03c06fa10 con 0x7fc03c156500 2024-03-06T07:29:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.576+0000 7fc044793640 1 -- 172.21.15.1:0/2723821452 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03c067fd0 con 0x7fc03c156500 2024-03-06T07:29:25.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.576+0000 7fc03affd640 1 -- 172.21.15.1:0/2723821452 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc02c01c070 con 0x7fc03c156500 2024-03-06T07:29:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.576+0000 7fc03affd640 1 --2- 172.21.15.1:0/2723821452 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc02806b4d0 0x7fc02806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.577+0000 7fc03affd640 1 -- 172.21.15.1:0/2723821452 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc02c089cb0 con 0x7fc03c156500 2024-03-06T07:29:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.577+0000 7fc04252f640 1 --2- 172.21.15.1:0/2723821452 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc02806b4d0 0x7fc02806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.577+0000 7fc04252f640 1 --2- 172.21.15.1:0/2723821452 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc02806b4d0 0x7fc02806d990 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0c0df0 tx=0x7fc034009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.581+0000 7fc03affd640 1 -- 172.21.15.1:0/2723821452 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc02c058d80 con 0x7fc03c156500 2024-03-06T07:29:25.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.760+0000 7fc044793640 1 -- 172.21.15.1:0/2723821452 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc03c0c5850 con 0x7fc03c156500 2024-03-06T07:29:25.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.763+0000 7fc03affd640 1 -- 172.21.15.1:0/2723821452 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v62) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc02c058680 con 0x7fc03c156500 2024-03-06T07:29:25.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.765+0000 7fc044793640 1 -- 172.21.15.1:0/2723821452 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc02806b4d0 msgr2=0x7fc02806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.765+0000 7fc044793640 1 --2- 172.21.15.1:0/2723821452 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc02806b4d0 0x7fc02806d990 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0c0df0 tx=0x7fc034009040 comp rx=0 tx=0).stop 2024-03-06T07:29:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.765+0000 7fc044793640 1 -- 172.21.15.1:0/2723821452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc03c156500 msgr2=0x7fc03c1ef060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.765+0000 7fc044793640 1 --2- 172.21.15.1:0/2723821452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc03c156500 0x7fc03c1ef060 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fc02c00b970 tx=0x7fc02c00be30 comp rx=0 tx=0).stop 2024-03-06T07:29:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.765+0000 7fc044793640 1 -- 172.21.15.1:0/2723821452 shutdown_connections 2024-03-06T07:29:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.765+0000 7fc044793640 1 --2- 172.21.15.1:0/2723821452 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc02806b4d0 0x7fc02806d990 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.765+0000 7fc044793640 1 --2- 172.21.15.1:0/2723821452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc03c156500 0x7fc03c1ef060 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.765+0000 7fc044793640 1 --2- 172.21.15.1:0/2723821452 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc03c14cdc0 0x7fc03c1eeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.765+0000 7fc044793640 1 --2- 172.21.15.1:0/2723821452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc03c06b510 0x7fc03c1e00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.765+0000 7fc044793640 1 -- 172.21.15.1:0/2723821452 >> 172.21.15.1:0/2723821452 conn(0x7fc03c0a9700 msgr2=0x7fc03c1518b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.765+0000 7fc044793640 1 -- 172.21.15.1:0/2723821452 shutdown_connections 2024-03-06T07:29:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.765+0000 7fc044793640 1 -- 172.21.15.1:0/2723821452 wait complete. 2024-03-06T07:29:25.777 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_health' 2024-03-06T07:29:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.877+0000 7f0215cdf640 1 Processor -- start 2024-03-06T07:29:25.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.877+0000 7f0215cdf640 1 -- start start 2024-03-06T07:29:25.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.878+0000 7f0215cdf640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0210156500 0x7f0210152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.878+0000 7f0215cdf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02101569e0 con 0x7f021006b510 2024-03-06T07:29:25.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.878+0000 7f0215cdf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0210156cc0 con 0x7f021014cdc0 2024-03-06T07:29:25.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.878+0000 7f0215cdf640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02100c2dd0 con 0x7f0210156500 2024-03-06T07:29:25.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.878+0000 7f020effd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0210156500 0x7f0210152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.878+0000 7f020ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f021006b510 0x7f021006b960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60050/0 (socket says 172.21.15.1:60050) 2024-03-06T07:29:25.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.878+0000 7f020ffff640 1 -- 172.21.15.1:0/3046983601 learned_addr learned my addr 172.21.15.1:0/3046983601 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:25.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.878+0000 7f020effd640 1 -- 172.21.15.1:0/3046983601 >> v1:172.21.15.98:6789/0 conn(0x7f021014cdc0 legacy=0x7f02100c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:25.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.878+0000 7f020effd640 1 -- 172.21.15.1:0/3046983601 >> v1:172.21.15.1:6789/0 conn(0x7f021006b510 legacy=0x7f021006b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.878+0000 7f020effd640 1 -- 172.21.15.1:0/3046983601 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0210156fa0 con 0x7f0210156500 2024-03-06T07:29:25.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.879+0000 7f020effd640 1 --2- 172.21.15.1:0/3046983601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0210156500 0x7f0210152f10 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f02040097a0 tx=0x7f020402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ba928d7eecdaaa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.879+0000 7f020e7fc640 1 -- 172.21.15.1:0/3046983601 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02040315a0 con 0x7f0210156500 2024-03-06T07:29:25.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.879+0000 7f020e7fc640 1 -- 172.21.15.1:0/3046983601 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0204037070 con 0x7f0210156500 2024-03-06T07:29:25.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.879+0000 7f020e7fc640 1 -- 172.21.15.1:0/3046983601 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0204031960 con 0x7f0210156500 2024-03-06T07:29:25.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.880+0000 7f0215cdf640 1 -- 172.21.15.1:0/3046983601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0210156500 msgr2=0x7f0210152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.880+0000 7f0215cdf640 1 --2- 172.21.15.1:0/3046983601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0210156500 0x7f0210152f10 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f02040097a0 tx=0x7f020402b840 comp rx=0 tx=0).stop 2024-03-06T07:29:25.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.880+0000 7f0215cdf640 1 -- 172.21.15.1:0/3046983601 shutdown_connections 2024-03-06T07:29:25.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.880+0000 7f0215cdf640 1 --2- 172.21.15.1:0/3046983601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0210156500 0x7f0210152f10 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.880+0000 7f0215cdf640 1 -- 172.21.15.1:0/3046983601 >> 172.21.15.1:0/3046983601 conn(0x7f02100a9700 msgr2=0x7f02100a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:25.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.880+0000 7f0215cdf640 1 -- 172.21.15.1:0/3046983601 shutdown_connections 2024-03-06T07:29:25.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.880+0000 7f0215cdf640 1 -- 172.21.15.1:0/3046983601 wait complete. 2024-03-06T07:29:25.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.881+0000 7f0215cdf640 1 Processor -- start 2024-03-06T07:29:25.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.881+0000 7f0215cdf640 1 -- start start 2024-03-06T07:29:25.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f0215cdf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f021006b510 0x7f02100acec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f0215cdf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f021014cdc0 0x7f02100aa730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f0215cdf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0210156500 0x7f02100aac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f0215cdf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02100c65e0 con 0x7f021014cdc0 2024-03-06T07:29:25.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f0215cdf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02100c5a70 con 0x7f021006b510 2024-03-06T07:29:25.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f0215cdf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02100c2a60 con 0x7f0210156500 2024-03-06T07:29:25.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f020effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f021014cdc0 0x7f02100aa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f020effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f021014cdc0 0x7f02100aa730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59174/0 (socket says 172.21.15.1:59174) 2024-03-06T07:29:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f020effd640 1 -- 172.21.15.1:0/3294596996 learned_addr learned my addr 172.21.15.1:0/3294596996 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f020f7fe640 1 --2- 172.21.15.1:0/3294596996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f021006b510 0x7f02100acec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f020ffff640 1 --2- 172.21.15.1:0/3294596996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0210156500 0x7f02100aac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f020effd640 1 -- 172.21.15.1:0/3294596996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0210156500 msgr2=0x7f02100aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f020effd640 1 --2- 172.21.15.1:0/3294596996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0210156500 0x7f02100aac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f020effd640 1 -- 172.21.15.1:0/3294596996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f021006b510 msgr2=0x7f02100acec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f020effd640 1 --2- 172.21.15.1:0/3294596996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f021006b510 0x7f02100acec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f020effd640 1 -- 172.21.15.1:0/3294596996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021014e450 con 0x7f021014cdc0 2024-03-06T07:29:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f020effd640 1 --2- 172.21.15.1:0/3294596996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f021014cdc0 0x7f02100aa730 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f0204000c00 tx=0x7f02040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f01f7fff640 1 -- 172.21.15.1:0/3294596996 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0204032840 con 0x7f021014cdc0 2024-03-06T07:29:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f01f7fff640 1 -- 172.21.15.1:0/3294596996 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02040329e0 con 0x7f021014cdc0 2024-03-06T07:29:25.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f01f7fff640 1 -- 172.21.15.1:0/3294596996 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0204032dd0 con 0x7f021014cdc0 2024-03-06T07:29:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f0215cdf640 1 -- 172.21.15.1:0/3294596996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021014e730 con 0x7f021014cdc0 2024-03-06T07:29:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.882+0000 7f0215cdf640 1 -- 172.21.15.1:0/3294596996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f021014ecf0 con 0x7f021014cdc0 2024-03-06T07:29:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.883+0000 7f0215cdf640 1 -- 172.21.15.1:0/3294596996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0210067fd0 con 0x7f021014cdc0 2024-03-06T07:29:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.884+0000 7f01f7fff640 1 -- 172.21.15.1:0/3294596996 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f020404d3f0 con 0x7f021014cdc0 2024-03-06T07:29:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.884+0000 7f01f7fff640 1 --2- 172.21.15.1:0/3294596996 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01fc06b4d0 0x7f01fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.884+0000 7f01f7fff640 1 -- 172.21.15.1:0/3294596996 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f020402f030 con 0x7f021014cdc0 2024-03-06T07:29:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.885+0000 7f020f7fe640 1 --2- 172.21.15.1:0/3294596996 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01fc06b4d0 0x7f01fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.885+0000 7f020f7fe640 1 --2- 172.21.15.1:0/3294596996 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01fc06b4d0 0x7f01fc06d990 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f021014d2c0 tx=0x7f0200009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:25.889+0000 7f01f7fff640 1 -- 172.21.15.1:0/3294596996 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0204040070 con 0x7f021014cdc0 2024-03-06T07:29:26.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.144+0000 7f0215cdf640 1 -- 172.21.15.1:0/3294596996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]} v 0) v1 -- 0x7f02100c5850 con 0x7f021014cdc0 2024-03-06T07:29:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.441+0000 7f01f7fff640 1 -- 172.21.15.1:0/3294596996 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]}]=0 v515) v1 ==== 140+0+0 (secure 0 0 0) 0x7f020407ac10 con 0x7f021014cdc0 2024-03-06T07:29:26.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.443+0000 7f0215cdf640 1 -- 172.21.15.1:0/3294596996 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01fc06b4d0 msgr2=0x7f01fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.443+0000 7f0215cdf640 1 --2- 172.21.15.1:0/3294596996 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01fc06b4d0 0x7f01fc06d990 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f021014d2c0 tx=0x7f0200009040 comp rx=0 tx=0).stop 2024-03-06T07:29:26.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.443+0000 7f0215cdf640 1 -- 172.21.15.1:0/3294596996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f021014cdc0 msgr2=0x7f02100aa730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.443+0000 7f0215cdf640 1 --2- 172.21.15.1:0/3294596996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f021014cdc0 0x7f02100aa730 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f0204000c00 tx=0x7f02040046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.443+0000 7f0215cdf640 1 -- 172.21.15.1:0/3294596996 shutdown_connections 2024-03-06T07:29:26.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.443+0000 7f0215cdf640 1 --2- 172.21.15.1:0/3294596996 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01fc06b4d0 0x7f01fc06d990 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.443+0000 7f0215cdf640 1 --2- 172.21.15.1:0/3294596996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0210156500 0x7f02100aac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.443+0000 7f0215cdf640 1 --2- 172.21.15.1:0/3294596996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f021014cdc0 0x7f02100aa730 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.443+0000 7f0215cdf640 1 --2- 172.21.15.1:0/3294596996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f021006b510 0x7f02100acec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.443+0000 7f0215cdf640 1 -- 172.21.15.1:0/3294596996 >> 172.21.15.1:0/3294596996 conn(0x7f02100a9700 msgr2=0x7f02100724c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.443+0000 7f0215cdf640 1 -- 172.21.15.1:0/3294596996 shutdown_connections 2024-03-06T07:29:26.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.443+0000 7f0215cdf640 1 -- 172.21.15.1:0/3294596996 wait complete. 2024-03-06T07:29:26.456 INFO:tasks.cephfs_test_runner:That marking a FS down does not generate a health warning ... ok 2024-03-06T07:29:26.456 INFO:tasks.cephfs_test_runner:Starting test: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) That marking a FS down twice does not wipe old_max_mds. 2024-03-06T07:29:26.456 INFO:tasks.cephfs_test_runner:test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-03-06T07:29:26.456 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:26.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.557+0000 7f8444d1e640 1 Processor -- start 2024-03-06T07:29:26.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.557+0000 7f8444d1e640 1 -- start start 2024-03-06T07:29:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.558+0000 7f8444d1e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8440149a90 0x7f84400c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.558+0000 7f8444d1e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84400c16f0 0x7f8440156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.558+0000 7f8444d1e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84401530a0 con 0x7f84400c1260 2024-03-06T07:29:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.558+0000 7f8444d1e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8440153380 con 0x7f8440149a90 2024-03-06T07:29:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.558+0000 7f8444d1e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8440153660 con 0x7f84400c16f0 2024-03-06T07:29:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.558+0000 7f843e59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8440149a90 0x7f84400c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.558+0000 7f843e59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8440149a90 0x7f84400c4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59182/0 (socket says 172.21.15.1:59182) 2024-03-06T07:29:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.558+0000 7f843e59c640 1 -- 172.21.15.1:0/3320324195 learned_addr learned my addr 172.21.15.1:0/3320324195 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.558+0000 7f843e59c640 1 -- 172.21.15.1:0/3320324195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84400c16f0 msgr2=0x7f8440156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.558+0000 7f843ed9d640 1 --2- 172.21.15.1:0/3320324195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84400c16f0 0x7f8440156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.558+0000 7f843e59c640 1 --2- 172.21.15.1:0/3320324195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84400c16f0 0x7f8440156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.558+0000 7f843e59c640 1 -- 172.21.15.1:0/3320324195 >> v1:172.21.15.98:6789/0 conn(0x7f84400c1260 legacy=0x7f84401565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.558+0000 7f843e59c640 1 -- 172.21.15.1:0/3320324195 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8440153940 con 0x7f8440149a90 2024-03-06T07:29:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.558+0000 7f843ed9d640 1 --2- 172.21.15.1:0/3320324195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84400c16f0 0x7f8440156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:26.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.559+0000 7f843e59c640 1 --2- 172.21.15.1:0/3320324195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8440149a90 0x7f84400c4760 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f8430009870 tx=0x7f843002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=288bb66bb6bc6a76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.559+0000 7f843d59a640 1 -- 172.21.15.1:0/3320324195 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84300317c0 con 0x7f8440149a90 2024-03-06T07:29:26.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.559+0000 7f843d59a640 1 -- 172.21.15.1:0/3320324195 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8430037070 con 0x7f8440149a90 2024-03-06T07:29:26.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.559+0000 7f843d59a640 1 -- 172.21.15.1:0/3320324195 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8430031b80 con 0x7f8440149a90 2024-03-06T07:29:26.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.560+0000 7f8444d1e640 1 -- 172.21.15.1:0/3320324195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8440149a90 msgr2=0x7f84400c4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.560+0000 7f8444d1e640 1 --2- 172.21.15.1:0/3320324195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8440149a90 0x7f84400c4760 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f8430009870 tx=0x7f843002b860 comp rx=0 tx=0).stop 2024-03-06T07:29:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.560+0000 7f8444d1e640 1 -- 172.21.15.1:0/3320324195 shutdown_connections 2024-03-06T07:29:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.560+0000 7f8444d1e640 1 --2- 172.21.15.1:0/3320324195 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84400c16f0 0x7f8440156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.560+0000 7f8444d1e640 1 --2- 172.21.15.1:0/3320324195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8440149a90 0x7f84400c4760 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.560+0000 7f8444d1e640 1 -- 172.21.15.1:0/3320324195 >> 172.21.15.1:0/3320324195 conn(0x7f84400c6860 msgr2=0x7f84400c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.560+0000 7f8444d1e640 1 -- 172.21.15.1:0/3320324195 shutdown_connections 2024-03-06T07:29:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.560+0000 7f8444d1e640 1 -- 172.21.15.1:0/3320324195 wait complete. 2024-03-06T07:29:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.561+0000 7f8444d1e640 1 Processor -- start 2024-03-06T07:29:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.561+0000 7f8444d1e640 1 -- start start 2024-03-06T07:29:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f8444d1e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84400c1260 0x7f844014f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f8444d1e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84400c16f0 0x7f84401e9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f8444d1e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8440149a90 0x7f84401ea140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f8444d1e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84400c4ca0 con 0x7f8440149a90 2024-03-06T07:29:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f8444d1e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84401571f0 con 0x7f84400c16f0 2024-03-06T07:29:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f8444d1e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844006f4a0 con 0x7f84400c1260 2024-03-06T07:29:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f843ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8440149a90 0x7f84401ea140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f843ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8440149a90 0x7f84401ea140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59192/0 (socket says 172.21.15.1:59192) 2024-03-06T07:29:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f843ed9d640 1 -- 172.21.15.1:0/1762782368 learned_addr learned my addr 172.21.15.1:0/1762782368 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f843e59c640 1 --2- 172.21.15.1:0/1762782368 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84400c1260 0x7f844014f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f843dd9b640 1 --2- 172.21.15.1:0/1762782368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84400c16f0 0x7f84401e9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f843ed9d640 1 -- 172.21.15.1:0/1762782368 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84400c1260 msgr2=0x7f844014f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f843ed9d640 1 --2- 172.21.15.1:0/1762782368 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84400c1260 0x7f844014f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f843ed9d640 1 -- 172.21.15.1:0/1762782368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84400c16f0 msgr2=0x7f84401e9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f843ed9d640 1 --2- 172.21.15.1:0/1762782368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84400c16f0 0x7f84401e9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f843ed9d640 1 -- 172.21.15.1:0/1762782368 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84401dd050 con 0x7f8440149a90 2024-03-06T07:29:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f843ed9d640 1 --2- 172.21.15.1:0/1762782368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8440149a90 0x7f84401ea140 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f843800cef0 tx=0x7f843800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f842effd640 1 -- 172.21.15.1:0/1762782368 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843801a070 con 0x7f8440149a90 2024-03-06T07:29:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f842effd640 1 -- 172.21.15.1:0/1762782368 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843800d860 con 0x7f8440149a90 2024-03-06T07:29:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f8444d1e640 1 -- 172.21.15.1:0/1762782368 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84401dd330 con 0x7f8440149a90 2024-03-06T07:29:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.562+0000 7f842effd640 1 -- 172.21.15.1:0/1762782368 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843800dc50 con 0x7f8440149a90 2024-03-06T07:29:26.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.563+0000 7f8444d1e640 1 -- 172.21.15.1:0/1762782368 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844014fef0 con 0x7f8440149a90 2024-03-06T07:29:26.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.564+0000 7f8444d1e640 1 -- 172.21.15.1:0/1762782368 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8440067fd0 con 0x7f8440149a90 2024-03-06T07:29:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.564+0000 7f842effd640 1 -- 172.21.15.1:0/1762782368 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f843801d070 con 0x7f8440149a90 2024-03-06T07:29:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.564+0000 7f842effd640 1 --2- 172.21.15.1:0/1762782368 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f842006b500 0x7f842006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.564+0000 7f842effd640 1 -- 172.21.15.1:0/1762782368 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f843808a820 con 0x7f8440149a90 2024-03-06T07:29:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.565+0000 7f843e59c640 1 --2- 172.21.15.1:0/1762782368 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f842006b500 0x7f842006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.565+0000 7f843e59c640 1 --2- 172.21.15.1:0/1762782368 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f842006b500 0x7f842006d9c0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f8430036000 tx=0x7f843003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.568+0000 7f842effd640 1 -- 172.21.15.1:0/1762782368 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84380592c0 con 0x7f8440149a90 2024-03-06T07:29:26.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.749+0000 7f8444d1e640 1 -- 172.21.15.1:0/1762782368 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f844006f280 con 0x7f8440149a90 2024-03-06T07:29:26.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.750+0000 7f842effd640 1 -- 172.21.15.1:0/1762782368 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v201) v1 ==== 74+0+13054 (secure 0 0 0) 0x7f843805d360 con 0x7f8440149a90 2024-03-06T07:29:26.752 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:26.752 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":201,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:29:25.432870+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:26.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.752+0000 7f8444d1e640 1 -- 172.21.15.1:0/1762782368 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f842006b500 msgr2=0x7f842006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.752+0000 7f8444d1e640 1 --2- 172.21.15.1:0/1762782368 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f842006b500 0x7f842006d9c0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f8430036000 tx=0x7f843003d040 comp rx=0 tx=0).stop 2024-03-06T07:29:26.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.752+0000 7f8444d1e640 1 -- 172.21.15.1:0/1762782368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8440149a90 msgr2=0x7f84401ea140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.752+0000 7f8444d1e640 1 --2- 172.21.15.1:0/1762782368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8440149a90 0x7f84401ea140 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f843800cef0 tx=0x7f843800a710 comp rx=0 tx=0).stop 2024-03-06T07:29:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.752+0000 7f8444d1e640 1 -- 172.21.15.1:0/1762782368 shutdown_connections 2024-03-06T07:29:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.752+0000 7f8444d1e640 1 --2- 172.21.15.1:0/1762782368 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f842006b500 0x7f842006d9c0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.752+0000 7f8444d1e640 1 --2- 172.21.15.1:0/1762782368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8440149a90 0x7f84401ea140 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.752+0000 7f8444d1e640 1 --2- 172.21.15.1:0/1762782368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84400c16f0 0x7f84401e9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.752+0000 7f8444d1e640 1 --2- 172.21.15.1:0/1762782368 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84400c1260 0x7f844014f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.752+0000 7f8444d1e640 1 -- 172.21.15.1:0/1762782368 >> 172.21.15.1:0/1762782368 conn(0x7f84400c6860 msgr2=0x7f84400c6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.752+0000 7f8444d1e640 1 -- 172.21.15.1:0/1762782368 shutdown_connections 2024-03-06T07:29:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.752+0000 7f8444d1e640 1 -- 172.21.15.1:0/1762782368 wait complete. 2024-03-06T07:29:26.765 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:26.765 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:26.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.867+0000 7fbd3933e640 1 Processor -- start 2024-03-06T07:29:26.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.867+0000 7fbd3933e640 1 -- start start 2024-03-06T07:29:26.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.868+0000 7fbd3933e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbd34152890 0x7fbd34159030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.868+0000 7fbd3933e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbd340c7560 0x7fbd34155860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.868+0000 7fbd3933e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd34155da0 0x7fbd340c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.868+0000 7fbd3933e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd34155080 con 0x7fbd340c7560 2024-03-06T07:29:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.868+0000 7fbd3933e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd340c35b0 con 0x7fbd34152890 2024-03-06T07:29:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.868+0000 7fbd3933e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd340c3890 con 0x7fbd34155da0 2024-03-06T07:29:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.868+0000 7fbd327fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbd340c7560 0x7fbd34155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.868+0000 7fbd327fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbd340c7560 0x7fbd34155860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59208/0 (socket says 172.21.15.1:59208) 2024-03-06T07:29:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.868+0000 7fbd327fc640 1 -- 172.21.15.1:0/2788523597 learned_addr learned my addr 172.21.15.1:0/2788523597 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.868+0000 7fbd327fc640 1 -- 172.21.15.1:0/2788523597 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd34155da0 msgr2=0x7fbd340c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.868+0000 7fbd327fc640 1 --2- 172.21.15.1:0/2788523597 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd34155da0 0x7fbd340c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.868+0000 7fbd327fc640 1 -- 172.21.15.1:0/2788523597 >> v2:172.21.15.98:3300/0 conn(0x7fbd34152890 msgr2=0x7fbd34159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.868+0000 7fbd327fc640 1 --2- 172.21.15.1:0/2788523597 >> v2:172.21.15.98:3300/0 conn(0x7fbd34152890 0x7fbd34159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.868+0000 7fbd327fc640 1 -- 172.21.15.1:0/2788523597 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd340c3b70 con 0x7fbd340c7560 2024-03-06T07:29:26.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.869+0000 7fbd327fc640 1 --2- 172.21.15.1:0/2788523597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd340c7560 0x7fbd34155860 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fbd340c63d0 tx=0x7fbd1c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf19eb1aa0cfea15 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.869+0000 7fbd31ffb640 1 -- 172.21.15.1:0/2788523597 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c038070 con 0x7fbd340c7560 2024-03-06T07:29:26.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.869+0000 7fbd31ffb640 1 -- 172.21.15.1:0/2788523597 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1c02f050 con 0x7fbd340c7560 2024-03-06T07:29:26.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.869+0000 7fbd31ffb640 1 -- 172.21.15.1:0/2788523597 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c031970 con 0x7fbd340c7560 2024-03-06T07:29:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.869+0000 7fbd3933e640 1 -- 172.21.15.1:0/2788523597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd340c7560 msgr2=0x7fbd34155860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.869+0000 7fbd3933e640 1 --2- 172.21.15.1:0/2788523597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd340c7560 0x7fbd34155860 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fbd340c63d0 tx=0x7fbd1c02b7f0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.869+0000 7fbd3933e640 1 -- 172.21.15.1:0/2788523597 shutdown_connections 2024-03-06T07:29:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.869+0000 7fbd3933e640 1 --2- 172.21.15.1:0/2788523597 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd34155da0 0x7fbd340c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.869+0000 7fbd3933e640 1 --2- 172.21.15.1:0/2788523597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd340c7560 0x7fbd34155860 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.869+0000 7fbd3933e640 1 --2- 172.21.15.1:0/2788523597 >> v2:172.21.15.98:3300/0 conn(0x7fbd34152890 0x7fbd34159030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.869+0000 7fbd3933e640 1 -- 172.21.15.1:0/2788523597 >> 172.21.15.1:0/2788523597 conn(0x7fbd3406b3e0 msgr2=0x7fbd3406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.870+0000 7fbd3933e640 1 -- 172.21.15.1:0/2788523597 shutdown_connections 2024-03-06T07:29:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.870+0000 7fbd3933e640 1 -- 172.21.15.1:0/2788523597 wait complete. 2024-03-06T07:29:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.871+0000 7fbd3933e640 1 Processor -- start 2024-03-06T07:29:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.871+0000 7fbd3933e640 1 -- start start 2024-03-06T07:29:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.871+0000 7fbd3933e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd340c7560 0x7fbd341ebc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.871+0000 7fbd3933e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd34152890 0x7fbd34158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.871+0000 7fbd3933e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd34099a50 0x7fbd34070b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.871+0000 7fbd3933e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd340c3260 con 0x7fbd34099a50 2024-03-06T07:29:26.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.871+0000 7fbd3933e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd34159570 con 0x7fbd340c7560 2024-03-06T07:29:26.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.871+0000 7fbd3933e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd340a81f0 con 0x7fbd34152890 2024-03-06T07:29:26.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.872+0000 7fbd337fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd34099a50 0x7fbd34070b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.872+0000 7fbd337fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd34099a50 0x7fbd34070b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59218/0 (socket says 172.21.15.1:59218) 2024-03-06T07:29:26.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.872+0000 7fbd337fe640 1 -- 172.21.15.1:0/1080107892 learned_addr learned my addr 172.21.15.1:0/1080107892 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:26.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.872+0000 7fbd327fc640 1 --2- 172.21.15.1:0/1080107892 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd34152890 0x7fbd34158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.872+0000 7fbd32ffd640 1 --2- 172.21.15.1:0/1080107892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd340c7560 0x7fbd341ebc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.872+0000 7fbd337fe640 1 -- 172.21.15.1:0/1080107892 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd34152890 msgr2=0x7fbd34158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.872+0000 7fbd337fe640 1 --2- 172.21.15.1:0/1080107892 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd34152890 0x7fbd34158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.872+0000 7fbd337fe640 1 -- 172.21.15.1:0/1080107892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd340c7560 msgr2=0x7fbd341ebc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.872+0000 7fbd337fe640 1 --2- 172.21.15.1:0/1080107892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd340c7560 0x7fbd341ebc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.872+0000 7fbd337fe640 1 -- 172.21.15.1:0/1080107892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd341eb180 con 0x7fbd34099a50 2024-03-06T07:29:26.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.872+0000 7fbd337fe640 1 --2- 172.21.15.1:0/1080107892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd34099a50 0x7fbd34070b30 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fbd2400b9e0 tx=0x7fbd2400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.872+0000 7fbd2b7fe640 1 -- 172.21.15.1:0/1080107892 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd24019070 con 0x7fbd34099a50 2024-03-06T07:29:26.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.872+0000 7fbd2b7fe640 1 -- 172.21.15.1:0/1080107892 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2400cae0 con 0x7fbd34099a50 2024-03-06T07:29:26.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.872+0000 7fbd3933e640 1 -- 172.21.15.1:0/1080107892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd340ab7c0 con 0x7fbd34099a50 2024-03-06T07:29:26.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.872+0000 7fbd2b7fe640 1 -- 172.21.15.1:0/1080107892 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd24014440 con 0x7fbd34099a50 2024-03-06T07:29:26.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.872+0000 7fbd3933e640 1 -- 172.21.15.1:0/1080107892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd341eb440 con 0x7fbd34099a50 2024-03-06T07:29:26.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.873+0000 7fbd3933e640 1 -- 172.21.15.1:0/1080107892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd34067fd0 con 0x7fbd34099a50 2024-03-06T07:29:26.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.874+0000 7fbd2b7fe640 1 -- 172.21.15.1:0/1080107892 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd2401c070 con 0x7fbd34099a50 2024-03-06T07:29:26.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.874+0000 7fbd2b7fe640 1 --2- 172.21.15.1:0/1080107892 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbd1406b4d0 0x7fbd1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.874+0000 7fbd2b7fe640 1 -- 172.21.15.1:0/1080107892 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbd24089cb0 con 0x7fbd34099a50 2024-03-06T07:29:26.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.874+0000 7fbd32ffd640 1 --2- 172.21.15.1:0/1080107892 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbd1406b4d0 0x7fbd1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.875+0000 7fbd32ffd640 1 --2- 172.21.15.1:0/1080107892 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbd1406b4d0 0x7fbd1406d990 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fbd340a81c0 tx=0x7fbd2c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:26.878+0000 7fbd2b7fe640 1 -- 172.21.15.1:0/1080107892 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd24058d80 con 0x7fbd34099a50 2024-03-06T07:29:27.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.058+0000 7fbd3933e640 1 -- 172.21.15.1:0/1080107892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbd34154e60 con 0x7fbd34099a50 2024-03-06T07:29:27.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.058+0000 7fbd2b7fe640 1 -- 172.21.15.1:0/1080107892 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v201) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbd24058680 con 0x7fbd34099a50 2024-03-06T07:29:27.060 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:29:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.060+0000 7fbd3933e640 1 -- 172.21.15.1:0/1080107892 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbd1406b4d0 msgr2=0x7fbd1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.060+0000 7fbd3933e640 1 --2- 172.21.15.1:0/1080107892 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbd1406b4d0 0x7fbd1406d990 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fbd340a81c0 tx=0x7fbd2c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.060+0000 7fbd3933e640 1 -- 172.21.15.1:0/1080107892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd34099a50 msgr2=0x7fbd34070b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.060+0000 7fbd3933e640 1 --2- 172.21.15.1:0/1080107892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd34099a50 0x7fbd34070b30 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fbd2400b9e0 tx=0x7fbd2400bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.060+0000 7fbd3933e640 1 -- 172.21.15.1:0/1080107892 shutdown_connections 2024-03-06T07:29:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.060+0000 7fbd3933e640 1 --2- 172.21.15.1:0/1080107892 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbd1406b4d0 0x7fbd1406d990 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.060+0000 7fbd3933e640 1 --2- 172.21.15.1:0/1080107892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd34099a50 0x7fbd34070b30 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.060+0000 7fbd3933e640 1 --2- 172.21.15.1:0/1080107892 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd34152890 0x7fbd34158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.060+0000 7fbd3933e640 1 --2- 172.21.15.1:0/1080107892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd340c7560 0x7fbd341ebc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.060+0000 7fbd3933e640 1 -- 172.21.15.1:0/1080107892 >> 172.21.15.1:0/1080107892 conn(0x7fbd3406b3e0 msgr2=0x7fbd34074450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.060+0000 7fbd3933e640 1 -- 172.21.15.1:0/1080107892 shutdown_connections 2024-03-06T07:29:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.060+0000 7fbd3933e640 1 -- 172.21.15.1:0/1080107892 wait complete. 2024-03-06T07:29:27.073 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_twice' 2024-03-06T07:29:27.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.173+0000 7f531ed5c640 1 Processor -- start 2024-03-06T07:29:27.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.173+0000 7f531ed5c640 1 -- start start 2024-03-06T07:29:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.174+0000 7f531ed5c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5318156cd0 0x7f5318157120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.174+0000 7f531ed5c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53180c3300 con 0x7f53180c6730 2024-03-06T07:29:27.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.174+0000 7f531ed5c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53180c35a0 con 0x7f5318152690 2024-03-06T07:29:27.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.174+0000 7f531ed5c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53180c3880 con 0x7f5318156cd0 2024-03-06T07:29:27.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.174+0000 7f5317fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f53180c6730 0x7f53180c57d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60052/0 (socket says 172.21.15.1:60052) 2024-03-06T07:29:27.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.174+0000 7f5317fff640 1 -- 172.21.15.1:0/788126341 learned_addr learned my addr 172.21.15.1:0/788126341 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:27.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.174+0000 7f531d2f9640 1 --2- 172.21.15.1:0/788126341 >> v2:172.21.15.98:3300/0 conn(0x7f5318156cd0 0x7f5318157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.174+0000 7f53177fe640 1 -- 172.21.15.1:0/788126341 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55503617 0 0) 0x7f53180c3300 con 0x7f53180c6730 2024-03-06T07:29:27.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.174+0000 7f53177fe640 1 -- 172.21.15.1:0/788126341 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5300003520 con 0x7f53180c6730 2024-03-06T07:29:27.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f53177fe640 1 -- 172.21.15.1:0/788126341 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2046856441 0 0) 0x7f5300003520 con 0x7f53180c6730 2024-03-06T07:29:27.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f53177fe640 1 -- 172.21.15.1:0/788126341 >> v2:172.21.15.98:3300/0 conn(0x7f5318156cd0 msgr2=0x7f5318157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:27.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f53177fe640 1 --2- 172.21.15.1:0/788126341 >> v2:172.21.15.98:3300/0 conn(0x7f5318156cd0 0x7f5318157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f53177fe640 1 -- 172.21.15.1:0/788126341 >> v1:172.21.15.98:6789/0 conn(0x7f5318152690 legacy=0x7f5318155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f53177fe640 1 -- 172.21.15.1:0/788126341 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53180c3b60 con 0x7f53180c6730 2024-03-06T07:29:27.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f53177fe640 1 -- 172.21.15.1:0/788126341 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f530c001690 con 0x7f53180c6730 2024-03-06T07:29:27.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f53177fe640 1 -- 172.21.15.1:0/788126341 >> v1:172.21.15.1:6789/0 conn(0x7f53180c6730 legacy=0x7f53180c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f53177fe640 1 --2- 172.21.15.1:0/788126341 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53000039b0 0x7f5300003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f53177fe640 1 --2- 172.21.15.1:0/788126341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53000043b0 0x7f5300004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f53177fe640 1 --2- 172.21.15.1:0/788126341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300005120 0x7f530000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f53177fe640 1 -- 172.21.15.1:0/788126341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53180c3880 con 0x7f5300005120 2024-03-06T07:29:27.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f53177fe640 1 -- 172.21.15.1:0/788126341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530000c060 con 0x7f53000043b0 2024-03-06T07:29:27.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f53177fe640 1 -- 172.21.15.1:0/788126341 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530000c260 con 0x7f53000039b0 2024-03-06T07:29:27.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f531d2f9640 1 --2- 172.21.15.1:0/788126341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300005120 0x7f530000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f5317fff640 1 --2- 172.21.15.1:0/788126341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53000043b0 0x7f5300004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f531caf8640 1 --2- 172.21.15.1:0/788126341 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53000039b0 0x7f5300003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f531d2f9640 1 -- 172.21.15.1:0/788126341 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53000039b0 msgr2=0x7f5300003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:27.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f531d2f9640 1 --2- 172.21.15.1:0/788126341 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53000039b0 0x7f5300003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f531d2f9640 1 -- 172.21.15.1:0/788126341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53000043b0 msgr2=0x7f5300004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:27.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f531d2f9640 1 --2- 172.21.15.1:0/788126341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53000043b0 0x7f5300004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.175+0000 7f531d2f9640 1 -- 172.21.15.1:0/788126341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5300003520 con 0x7f5300005120 2024-03-06T07:29:27.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.176+0000 7f531d2f9640 1 -- 172.21.15.1:0/788126341 reap_dead start 2024-03-06T07:29:27.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.176+0000 7f531d2f9640 1 --2- 172.21.15.1:0/788126341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300005120 0x7f530000b9c0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f5308009ac0 tx=0x7f5300005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12aab3edbf6b2199 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.176+0000 7f53177fe640 1 -- 172.21.15.1:0/788126341 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308009690 con 0x7f5300005120 2024-03-06T07:29:27.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.176+0000 7f53177fe640 1 -- 172.21.15.1:0/788126341 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5308004020 con 0x7f5300005120 2024-03-06T07:29:27.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.176+0000 7f53177fe640 1 -- 172.21.15.1:0/788126341 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53080043d0 con 0x7f5300005120 2024-03-06T07:29:27.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.176+0000 7f531ed5c640 1 -- 172.21.15.1:0/788126341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300005120 msgr2=0x7f530000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.176+0000 7f531ed5c640 1 --2- 172.21.15.1:0/788126341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300005120 0x7f530000b9c0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f5308009ac0 tx=0x7f5300005830 comp rx=0 tx=0).stop 2024-03-06T07:29:27.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.176+0000 7f531ed5c640 1 -- 172.21.15.1:0/788126341 shutdown_connections 2024-03-06T07:29:27.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.176+0000 7f531ed5c640 1 --2- 172.21.15.1:0/788126341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300005120 0x7f530000b9c0 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.176+0000 7f531ed5c640 1 -- 172.21.15.1:0/788126341 >> 172.21.15.1:0/788126341 conn(0x7f531806b410 msgr2=0x7f531806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.176+0000 7f531ed5c640 1 -- 172.21.15.1:0/788126341 shutdown_connections 2024-03-06T07:29:27.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.177+0000 7f531ed5c640 1 -- 172.21.15.1:0/788126341 wait complete. 2024-03-06T07:29:27.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.178+0000 7f531ed5c640 1 Processor -- start 2024-03-06T07:29:27.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.178+0000 7f531ed5c640 1 -- start start 2024-03-06T07:29:27.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.178+0000 7f531ed5c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300005120 0x7f531806da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.178+0000 7f531ed5c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53180ac360 0x7f53180aae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.178+0000 7f531ed5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53180ab3b0 0x7f53181ea750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.178+0000 7f531ed5c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5318152bf0 con 0x7f5300005120 2024-03-06T07:29:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.178+0000 7f531ed5c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5318152a70 con 0x7f53180ab3b0 2024-03-06T07:29:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.178+0000 7f531ed5c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5318155ca0 con 0x7f53180ac360 2024-03-06T07:29:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.178+0000 7f531caf8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300005120 0x7f531806da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.178+0000 7f531caf8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300005120 0x7f531806da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59232/0 (socket says 172.21.15.1:59232) 2024-03-06T07:29:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.178+0000 7f531caf8640 1 -- 172.21.15.1:0/12131438 learned_addr learned my addr 172.21.15.1:0/12131438 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.179+0000 7f5317fff640 1 --2- 172.21.15.1:0/12131438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53180ac360 0x7f53180aae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.179+0000 7f531d2f9640 1 --2- 172.21.15.1:0/12131438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53180ab3b0 0x7f53181ea750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.179+0000 7f531caf8640 1 -- 172.21.15.1:0/12131438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53180ac360 msgr2=0x7f53180aae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.179+0000 7f531caf8640 1 --2- 172.21.15.1:0/12131438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53180ac360 0x7f53180aae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.179+0000 7f531caf8640 1 -- 172.21.15.1:0/12131438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53180ab3b0 msgr2=0x7f53181ea750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.179+0000 7f531caf8640 1 --2- 172.21.15.1:0/12131438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53180ab3b0 0x7f53181ea750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.179+0000 7f531caf8640 1 -- 172.21.15.1:0/12131438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53181edc60 con 0x7f5300005120 2024-03-06T07:29:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.179+0000 7f531caf8640 1 --2- 172.21.15.1:0/12131438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300005120 0x7f531806da90 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f531000eb70 tx=0x7f531000c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.179+0000 7f53157fa640 1 -- 172.21.15.1:0/12131438 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531000ee70 con 0x7f5300005120 2024-03-06T07:29:27.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.179+0000 7f53157fa640 1 -- 172.21.15.1:0/12131438 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5310018070 con 0x7f5300005120 2024-03-06T07:29:27.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.179+0000 7f531ed5c640 1 -- 172.21.15.1:0/12131438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53181edf40 con 0x7f5300005120 2024-03-06T07:29:27.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.179+0000 7f53157fa640 1 -- 172.21.15.1:0/12131438 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531000ce20 con 0x7f5300005120 2024-03-06T07:29:27.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.179+0000 7f531ed5c640 1 -- 172.21.15.1:0/12131438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53181ee7e0 con 0x7f5300005120 2024-03-06T07:29:27.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.180+0000 7f531ed5c640 1 -- 172.21.15.1:0/12131438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53181ee330 con 0x7f5300005120 2024-03-06T07:29:27.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.181+0000 7f53157fa640 1 -- 172.21.15.1:0/12131438 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5310003970 con 0x7f5300005120 2024-03-06T07:29:27.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.181+0000 7f53157fa640 1 --2- 172.21.15.1:0/12131438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f530406b4d0 0x7f530406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.181+0000 7f53157fa640 1 -- 172.21.15.1:0/12131438 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f531008a1b0 con 0x7f5300005120 2024-03-06T07:29:27.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.181+0000 7f5317fff640 1 --2- 172.21.15.1:0/12131438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f530406b4d0 0x7f530406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.182+0000 7f5317fff640 1 --2- 172.21.15.1:0/12131438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f530406b4d0 0x7f530406d990 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f531814e950 tx=0x7f530c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.185+0000 7f53157fa640 1 -- 172.21.15.1:0/12131438 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5310059280 con 0x7f5300005120 2024-03-06T07:29:27.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.436+0000 7f531ed5c640 1 -- 172.21.15.1:0/12131438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]} v 0) v1 -- 0x7f53181ee110 con 0x7f5300005120 2024-03-06T07:29:27.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.444+0000 7f53157fa640 1 -- 172.21.15.1:0/12131438 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]}]=0 v516) v1 ==== 142+0+0 (secure 0 0 0) 0x7f5310058b80 con 0x7f5300005120 2024-03-06T07:29:27.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.446+0000 7f531ed5c640 1 -- 172.21.15.1:0/12131438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f530406b4d0 msgr2=0x7f530406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.446+0000 7f531ed5c640 1 --2- 172.21.15.1:0/12131438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f530406b4d0 0x7f530406d990 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f531814e950 tx=0x7f530c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:27.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.446+0000 7f531ed5c640 1 -- 172.21.15.1:0/12131438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300005120 msgr2=0x7f531806da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.446+0000 7f531ed5c640 1 --2- 172.21.15.1:0/12131438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300005120 0x7f531806da90 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f531000eb70 tx=0x7f531000c4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.446+0000 7f531ed5c640 1 -- 172.21.15.1:0/12131438 shutdown_connections 2024-03-06T07:29:27.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.446+0000 7f531ed5c640 1 --2- 172.21.15.1:0/12131438 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f530406b4d0 0x7f530406d990 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.446+0000 7f531ed5c640 1 --2- 172.21.15.1:0/12131438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53180ab3b0 0x7f53181ea750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.446+0000 7f531ed5c640 1 --2- 172.21.15.1:0/12131438 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53180ac360 0x7f53180aae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.446+0000 7f531ed5c640 1 --2- 172.21.15.1:0/12131438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300005120 0x7f531806da90 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.446+0000 7f531ed5c640 1 -- 172.21.15.1:0/12131438 >> 172.21.15.1:0/12131438 conn(0x7f531806b410 msgr2=0x7f531806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.446+0000 7f531ed5c640 1 -- 172.21.15.1:0/12131438 shutdown_connections 2024-03-06T07:29:27.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.446+0000 7f531ed5c640 1 -- 172.21.15.1:0/12131438 wait complete. 2024-03-06T07:29:27.459 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:29:27.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.561+0000 7efd1755e640 1 Processor -- start 2024-03-06T07:29:27.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.561+0000 7efd1755e640 1 -- start start 2024-03-06T07:29:27.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.562+0000 7efd1755e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efd10152920 0x7efd1006aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.562+0000 7efd1755e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd10155220 0x7efd101556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.562+0000 7efd1755e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd10158910 con 0x7efd1006b530 2024-03-06T07:29:27.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.562+0000 7efd1755e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd10152d00 con 0x7efd10152920 2024-03-06T07:29:27.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.562+0000 7efd1755e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd101596d0 con 0x7efd10155220 2024-03-06T07:29:27.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.562+0000 7efd152fa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efd10152920 0x7efd1006aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.562+0000 7efd14af9640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efd1006b530 0x7efd100c5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60066/0 (socket says 172.21.15.1:60066) 2024-03-06T07:29:27.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.562+0000 7efd14af9640 1 -- 172.21.15.1:0/3430087470 learned_addr learned my addr 172.21.15.1:0/3430087470 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:27.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.562+0000 7efd152fa640 1 -- 172.21.15.1:0/3430087470 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd10155220 msgr2=0x7efd101556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:27.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.562+0000 7efd15afb640 1 --2- 172.21.15.1:0/3430087470 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd10155220 0x7efd101556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.562+0000 7efd152fa640 1 --2- 172.21.15.1:0/3430087470 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd10155220 0x7efd101556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.562+0000 7efd07fff640 1 -- 172.21.15.1:0/3430087470 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529908307 0 0) 0x7efd10158910 con 0x7efd1006b530 2024-03-06T07:29:27.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.562+0000 7efd152fa640 1 -- 172.21.15.1:0/3430087470 >> v1:172.21.15.1:6789/0 conn(0x7efd1006b530 legacy=0x7efd100c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.562+0000 7efd152fa640 1 -- 172.21.15.1:0/3430087470 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd10158bf0 con 0x7efd10152920 2024-03-06T07:29:27.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.562+0000 7efd15afb640 1 --2- 172.21.15.1:0/3430087470 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd10155220 0x7efd101556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:27.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.563+0000 7efd152fa640 1 --2- 172.21.15.1:0/3430087470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd10152920 0x7efd1006aff0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7efd080097a0 tx=0x7efd0802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b2f564734d5f51e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.563+0000 7efd07fff640 1 -- 172.21.15.1:0/3430087470 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd080315a0 con 0x7efd10152920 2024-03-06T07:29:27.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.563+0000 7efd07fff640 1 -- 172.21.15.1:0/3430087470 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd08037070 con 0x7efd10152920 2024-03-06T07:29:27.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.563+0000 7efd07fff640 1 -- 172.21.15.1:0/3430087470 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08031960 con 0x7efd10152920 2024-03-06T07:29:27.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.564+0000 7efd1755e640 1 -- 172.21.15.1:0/3430087470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd10152920 msgr2=0x7efd1006aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.564+0000 7efd1755e640 1 --2- 172.21.15.1:0/3430087470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd10152920 0x7efd1006aff0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7efd080097a0 tx=0x7efd0802b840 comp rx=0 tx=0).stop 2024-03-06T07:29:27.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.564+0000 7efd1755e640 1 -- 172.21.15.1:0/3430087470 shutdown_connections 2024-03-06T07:29:27.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.564+0000 7efd1755e640 1 --2- 172.21.15.1:0/3430087470 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd10155220 0x7efd101556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.564+0000 7efd1755e640 1 --2- 172.21.15.1:0/3430087470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd10152920 0x7efd1006aff0 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.564+0000 7efd1755e640 1 -- 172.21.15.1:0/3430087470 >> 172.21.15.1:0/3430087470 conn(0x7efd100a9700 msgr2=0x7efd100a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.564+0000 7efd1755e640 1 -- 172.21.15.1:0/3430087470 shutdown_connections 2024-03-06T07:29:27.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.564+0000 7efd1755e640 1 -- 172.21.15.1:0/3430087470 wait complete. 2024-03-06T07:29:27.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.565+0000 7efd1755e640 1 Processor -- start 2024-03-06T07:29:27.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.565+0000 7efd1755e640 1 -- start start 2024-03-06T07:29:27.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd1755e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd1006b530 0x7efd100acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd1755e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd10152920 0x7efd101540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd1755e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd10155220 0x7efd10154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd1755e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd100c6380 con 0x7efd1006b530 2024-03-06T07:29:27.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd1755e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd10159240 con 0x7efd10155220 2024-03-06T07:29:27.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd1755e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd100c5f60 con 0x7efd10152920 2024-03-06T07:29:27.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd152fa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd1006b530 0x7efd100acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd152fa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd1006b530 0x7efd100acb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59240/0 (socket says 172.21.15.1:59240) 2024-03-06T07:29:27.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd152fa640 1 -- 172.21.15.1:0/3467837828 learned_addr learned my addr 172.21.15.1:0/3467837828 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:27.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd14af9640 1 --2- 172.21.15.1:0/3467837828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd10152920 0x7efd101540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd15afb640 1 --2- 172.21.15.1:0/3467837828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd10155220 0x7efd10154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd152fa640 1 -- 172.21.15.1:0/3467837828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd10152920 msgr2=0x7efd101540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd152fa640 1 --2- 172.21.15.1:0/3467837828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd10152920 0x7efd101540f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd152fa640 1 -- 172.21.15.1:0/3467837828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd10155220 msgr2=0x7efd10154630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd152fa640 1 --2- 172.21.15.1:0/3467837828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd10155220 0x7efd10154630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd152fa640 1 -- 172.21.15.1:0/3467837828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd100700b0 con 0x7efd1006b530 2024-03-06T07:29:27.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd152fa640 1 --2- 172.21.15.1:0/3467837828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd1006b530 0x7efd100acb30 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7efd08034040 tx=0x7efd080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd05ffb640 1 -- 172.21.15.1:0/3467837828 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08004230 con 0x7efd1006b530 2024-03-06T07:29:27.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd05ffb640 1 -- 172.21.15.1:0/3467837828 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd080043d0 con 0x7efd1006b530 2024-03-06T07:29:27.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd05ffb640 1 -- 172.21.15.1:0/3467837828 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08003850 con 0x7efd1006b530 2024-03-06T07:29:27.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.566+0000 7efd1755e640 1 -- 172.21.15.1:0/3467837828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd10153330 con 0x7efd1006b530 2024-03-06T07:29:27.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.567+0000 7efd1755e640 1 -- 172.21.15.1:0/3467837828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd101538f0 con 0x7efd1006b530 2024-03-06T07:29:27.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.568+0000 7efd1755e640 1 -- 172.21.15.1:0/3467837828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd10067fd0 con 0x7efd1006b530 2024-03-06T07:29:27.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.568+0000 7efd05ffb640 1 -- 172.21.15.1:0/3467837828 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd080039f0 con 0x7efd1006b530 2024-03-06T07:29:27.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.568+0000 7efd05ffb640 1 --2- 172.21.15.1:0/3467837828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efcf406b4d0 0x7efcf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.568+0000 7efd05ffb640 1 -- 172.21.15.1:0/3467837828 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5249+0+0 (secure 0 0 0) 0x7efd0802f030 con 0x7efd1006b530 2024-03-06T07:29:27.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.569+0000 7efd14af9640 1 --2- 172.21.15.1:0/3467837828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efcf406b4d0 0x7efcf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.569+0000 7efd14af9640 1 --2- 172.21.15.1:0/3467837828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efcf406b4d0 0x7efcf406d990 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7efd100ad980 tx=0x7efd00007b50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.573+0000 7efd05ffb640 1 -- 172.21.15.1:0/3467837828 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd08041070 con 0x7efd1006b530 2024-03-06T07:29:27.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.755+0000 7efd1755e640 1 -- 172.21.15.1:0/3467837828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7efd100703e0 con 0x7efd1006b530 2024-03-06T07:29:27.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.758+0000 7efd05ffb640 1 -- 172.21.15.1:0/3467837828 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v63)=0 v63) v1 ==== 122+0+0 (secure 0 0 0) 0x7efd08044340 con 0x7efd1006b530 2024-03-06T07:29:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.760+0000 7efd1755e640 1 -- 172.21.15.1:0/3467837828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efcf406b4d0 msgr2=0x7efcf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.760+0000 7efd1755e640 1 --2- 172.21.15.1:0/3467837828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efcf406b4d0 0x7efcf406d990 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7efd100ad980 tx=0x7efd00007b50 comp rx=0 tx=0).stop 2024-03-06T07:29:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.760+0000 7efd1755e640 1 -- 172.21.15.1:0/3467837828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd1006b530 msgr2=0x7efd100acb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.760+0000 7efd1755e640 1 --2- 172.21.15.1:0/3467837828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd1006b530 0x7efd100acb30 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7efd08034040 tx=0x7efd080046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.760+0000 7efd1755e640 1 -- 172.21.15.1:0/3467837828 shutdown_connections 2024-03-06T07:29:27.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.760+0000 7efd1755e640 1 --2- 172.21.15.1:0/3467837828 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efcf406b4d0 0x7efcf406d990 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.760+0000 7efd1755e640 1 --2- 172.21.15.1:0/3467837828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd10155220 0x7efd10154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.760+0000 7efd1755e640 1 --2- 172.21.15.1:0/3467837828 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efd10152920 0x7efd101540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.760+0000 7efd1755e640 1 --2- 172.21.15.1:0/3467837828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd1006b530 0x7efd100acb30 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.760+0000 7efd1755e640 1 -- 172.21.15.1:0/3467837828 >> 172.21.15.1:0/3467837828 conn(0x7efd100a9700 msgr2=0x7efd100a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.760+0000 7efd1755e640 1 -- 172.21.15.1:0/3467837828 shutdown_connections 2024-03-06T07:29:27.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:27.760+0000 7efd1755e640 1 -- 172.21.15.1:0/3467837828 wait complete. 2024-03-06T07:29:27.773 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:29:27.773 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:27.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:27.875 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:29:27.875 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:29:27.875 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:27.890 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:29:27.890 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:29:27.890 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:27.906 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:29:27.907 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:29:27.907 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:28.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.028+0000 7fb2eebb6640 1 Processor -- start 2024-03-06T07:29:28.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.029+0000 7fb2eebb6640 1 -- start start 2024-03-06T07:29:28.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.029+0000 7fb2eebb6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2e8152690 0x7fb2e8155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.029+0000 7fb2eebb6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2e80c6730 0x7fb2e80c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.029+0000 7fb2eebb6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e81561e0 con 0x7fb2e80c5d10 2024-03-06T07:29:28.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.029+0000 7fb2eebb6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e80c3300 con 0x7fb2e80c6730 2024-03-06T07:29:28.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.029+0000 7fb2eebb6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e80c35a0 con 0x7fb2e8152690 2024-03-06T07:29:28.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.029+0000 7fb2e7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2e80c6730 0x7fb2e80c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.029+0000 7fb2ed153640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb2e80c5d10 0x7fb2e8156cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60070/0 (socket says 172.21.15.1:60070) 2024-03-06T07:29:28.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.029+0000 7fb2ed153640 1 -- 172.21.15.1:0/1756566978 learned_addr learned my addr 172.21.15.1:0/1756566978 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:28.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.030+0000 7fb2ec952640 1 --2- 172.21.15.1:0/1756566978 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2e8152690 0x7fb2e8155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.030+0000 7fb2e77fe640 1 -- 172.21.15.1:0/1756566978 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132812721 0 0) 0x7fb2e0001490 con 0x7fb2e80c5d10 2024-03-06T07:29:28.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.030+0000 7fb2e77fe640 1 -- 172.21.15.1:0/1756566978 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d8003520 con 0x7fb2e80c5d10 2024-03-06T07:29:28.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.030+0000 7fb2e7fff640 1 -- 172.21.15.1:0/1756566978 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2e8152690 msgr2=0x7fb2e8155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:28.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.030+0000 7fb2e7fff640 1 --2- 172.21.15.1:0/1756566978 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2e8152690 0x7fb2e8155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.030+0000 7fb2e7fff640 1 -- 172.21.15.1:0/1756566978 >> v1:172.21.15.1:6789/0 conn(0x7fb2e80c5d10 legacy=0x7fb2e8156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.030+0000 7fb2e7fff640 1 -- 172.21.15.1:0/1756566978 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e80c3880 con 0x7fb2e80c6730 2024-03-06T07:29:28.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.030+0000 7fb2e7fff640 1 --2- 172.21.15.1:0/1756566978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2e80c6730 0x7fb2e80c57d0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fb2d40097a0 tx=0x7fb2d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b768e5f5b4ce1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.031+0000 7fb2e77fe640 1 -- 172.21.15.1:0/1756566978 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d40315a0 con 0x7fb2e80c6730 2024-03-06T07:29:28.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.031+0000 7fb2e77fe640 1 -- 172.21.15.1:0/1756566978 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d4037070 con 0x7fb2e80c6730 2024-03-06T07:29:28.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.031+0000 7fb2e77fe640 1 -- 172.21.15.1:0/1756566978 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4031940 con 0x7fb2e80c6730 2024-03-06T07:29:28.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.031+0000 7fb2eebb6640 1 -- 172.21.15.1:0/1756566978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2e80c6730 msgr2=0x7fb2e80c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.031+0000 7fb2eebb6640 1 --2- 172.21.15.1:0/1756566978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2e80c6730 0x7fb2e80c57d0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fb2d40097a0 tx=0x7fb2d402b840 comp rx=0 tx=0).stop 2024-03-06T07:29:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.031+0000 7fb2eebb6640 1 -- 172.21.15.1:0/1756566978 shutdown_connections 2024-03-06T07:29:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.031+0000 7fb2eebb6640 1 --2- 172.21.15.1:0/1756566978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2e80c6730 0x7fb2e80c57d0 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.031+0000 7fb2eebb6640 1 --2- 172.21.15.1:0/1756566978 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2e8152690 0x7fb2e8155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.031+0000 7fb2eebb6640 1 -- 172.21.15.1:0/1756566978 >> 172.21.15.1:0/1756566978 conn(0x7fb2e806b410 msgr2=0x7fb2e806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.031+0000 7fb2eebb6640 1 -- 172.21.15.1:0/1756566978 shutdown_connections 2024-03-06T07:29:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.031+0000 7fb2eebb6640 1 -- 172.21.15.1:0/1756566978 wait complete. 2024-03-06T07:29:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.032+0000 7fb2eebb6640 1 Processor -- start 2024-03-06T07:29:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.032+0000 7fb2eebb6640 1 -- start start 2024-03-06T07:29:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.033+0000 7fb2eebb6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2e80c5d10 0x7fb2e81de880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.033+0000 7fb2eebb6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2e80c6730 0x7fb2e80a8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.033+0000 7fb2eebb6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e8152690 0x7fb2e80c7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.033+0000 7fb2eebb6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e8152a70 con 0x7fb2e80c6730 2024-03-06T07:29:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.033+0000 7fb2eebb6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e81567b0 con 0x7fb2e8152690 2024-03-06T07:29:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.033+0000 7fb2eebb6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e8152bf0 con 0x7fb2e80c5d10 2024-03-06T07:29:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.033+0000 7fb2e7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2e80c6730 0x7fb2e80a8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.033+0000 7fb2e7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2e80c6730 0x7fb2e80a8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59262/0 (socket says 172.21.15.1:59262) 2024-03-06T07:29:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.033+0000 7fb2e7fff640 1 -- 172.21.15.1:0/1745458867 learned_addr learned my addr 172.21.15.1:0/1745458867 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.033+0000 7fb2ec952640 1 --2- 172.21.15.1:0/1745458867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2e80c5d10 0x7fb2e81de880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.033+0000 7fb2ed153640 1 --2- 172.21.15.1:0/1745458867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e8152690 0x7fb2e80c7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.033+0000 7fb2e7fff640 1 -- 172.21.15.1:0/1745458867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2e80c5d10 msgr2=0x7fb2e81de880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.033+0000 7fb2e7fff640 1 --2- 172.21.15.1:0/1745458867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2e80c5d10 0x7fb2e81de880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.033+0000 7fb2e7fff640 1 -- 172.21.15.1:0/1745458867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e8152690 msgr2=0x7fb2e80c7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.033+0000 7fb2e7fff640 1 --2- 172.21.15.1:0/1745458867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e8152690 0x7fb2e80c7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.034+0000 7fb2e7fff640 1 -- 172.21.15.1:0/1745458867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e80ab370 con 0x7fb2e80c6730 2024-03-06T07:29:28.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.034+0000 7fb2ec952640 1 --2- 172.21.15.1:0/1745458867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2e80c5d10 0x7fb2e81de880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:28.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.034+0000 7fb2e7fff640 1 --2- 172.21.15.1:0/1745458867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2e80c6730 0x7fb2e80a8b70 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4034040 tx=0x7fb2d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.034+0000 7fb2e57fa640 1 -- 172.21.15.1:0/1745458867 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4032630 con 0x7fb2e80c6730 2024-03-06T07:29:28.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.034+0000 7fb2e57fa640 1 -- 172.21.15.1:0/1745458867 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d40327d0 con 0x7fb2e80c6730 2024-03-06T07:29:28.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.034+0000 7fb2eebb6640 1 -- 172.21.15.1:0/1745458867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e80c8e50 con 0x7fb2e80c6730 2024-03-06T07:29:28.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.034+0000 7fb2e57fa640 1 -- 172.21.15.1:0/1745458867 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4032b80 con 0x7fb2e80c6730 2024-03-06T07:29:28.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.034+0000 7fb2eebb6640 1 -- 172.21.15.1:0/1745458867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e80c76a0 con 0x7fb2e80c6730 2024-03-06T07:29:28.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.035+0000 7fb2eebb6640 1 -- 172.21.15.1:0/1745458867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e8067fd0 con 0x7fb2e80c6730 2024-03-06T07:29:28.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.035+0000 7fb2e57fa640 1 -- 172.21.15.1:0/1745458867 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2d404e430 con 0x7fb2e80c6730 2024-03-06T07:29:28.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.036+0000 7fb2e57fa640 1 --2- 172.21.15.1:0/1745458867 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb2cc06b4d0 0x7fb2cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.036+0000 7fb2e57fa640 1 -- 172.21.15.1:0/1745458867 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb2d402f030 con 0x7fb2e80c6730 2024-03-06T07:29:28.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.036+0000 7fb2ec952640 1 --2- 172.21.15.1:0/1745458867 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb2cc06b4d0 0x7fb2cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.036+0000 7fb2ec952640 1 --2- 172.21.15.1:0/1745458867 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb2cc06b4d0 0x7fb2cc06d990 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc006fd0 tx=0x7fb2dc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.040+0000 7fb2e57fa640 1 -- 172.21.15.1:0/1745458867 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2d4040070 con 0x7fb2e80c6730 2024-03-06T07:29:28.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.260+0000 7fb2eebb6640 1 -- 172.21.15.1:0/1745458867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2e80c1ea0 con 0x7fb2e80c6730 2024-03-06T07:29:28.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.260+0000 7fb2e57fa640 1 -- 172.21.15.1:0/1745458867 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+3696 (secure 0 0 0) 0x7fb2d407a8d0 con 0x7fb2e80c6730 2024-03-06T07:29:28.262 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:28.262 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":191,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7961,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2819896202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2819896202},{"type":"v1","addr":"172.21.15.98:6837","nonce":2819896202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":8959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/646952744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":646952744},{"type":"v1","addr":"172.21.15.1:6835","nonce":646952744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":8965,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2843229007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2843229007},{"type":"v1","addr":"172.21.15.98:6836","nonce":2843229007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":10635,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/1512189189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1512189189},{"type":"v1","addr":"172.21.15.1:6834","nonce":1512189189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":189}],"filesystems":[]} 2024-03-06T07:29:28.263 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 191 2024-03-06T07:29:28.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.262+0000 7fb2eebb6640 1 -- 172.21.15.1:0/1745458867 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb2cc06b4d0 msgr2=0x7fb2cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.262+0000 7fb2eebb6640 1 --2- 172.21.15.1:0/1745458867 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb2cc06b4d0 0x7fb2cc06d990 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc006fd0 tx=0x7fb2dc008040 comp rx=0 tx=0).stop 2024-03-06T07:29:28.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.262+0000 7fb2eebb6640 1 -- 172.21.15.1:0/1745458867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2e80c6730 msgr2=0x7fb2e80a8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.262+0000 7fb2eebb6640 1 --2- 172.21.15.1:0/1745458867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2e80c6730 0x7fb2e80a8b70 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4034040 tx=0x7fb2d40046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.262+0000 7fb2eebb6640 1 -- 172.21.15.1:0/1745458867 shutdown_connections 2024-03-06T07:29:28.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.262+0000 7fb2eebb6640 1 --2- 172.21.15.1:0/1745458867 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb2cc06b4d0 0x7fb2cc06d990 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.262+0000 7fb2eebb6640 1 --2- 172.21.15.1:0/1745458867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e8152690 0x7fb2e80c7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.262+0000 7fb2eebb6640 1 --2- 172.21.15.1:0/1745458867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2e80c6730 0x7fb2e80a8b70 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.262+0000 7fb2eebb6640 1 --2- 172.21.15.1:0/1745458867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2e80c5d10 0x7fb2e81de880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.263+0000 7fb2eebb6640 1 -- 172.21.15.1:0/1745458867 >> 172.21.15.1:0/1745458867 conn(0x7fb2e806b410 msgr2=0x7fb2e806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:28.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.263+0000 7fb2eebb6640 1 -- 172.21.15.1:0/1745458867 shutdown_connections 2024-03-06T07:29:28.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.263+0000 7fb2eebb6640 1 -- 172.21.15.1:0/1745458867 wait complete. 2024-03-06T07:29:28.275 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:29:28.276 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:29:28.276 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:29:28.276 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:28.276 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:29:28.276 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:29:28.276 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:29:28.276 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:28.277 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:29:28.277 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:29:28.277 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:29:28.277 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:28.277 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:29:28.277 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:29:28.277 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:29:28.278 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:28.287 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:29:28.287 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:29:28.288 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:29:28.288 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:29:28.289 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:29:28.289 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:29:28.290 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:29:28.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:29:28.329 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:29:28.329 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:28.329 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:29:28.330 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:28.332 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:29:28.332 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:28.332 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:29:28.333 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:28.333 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:29:28.406 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-06T07:29:28.406 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:29:28.415 INFO:tasks.ceph.mds.b.smithi098.stdout:starting mds.b at 2024-03-06T07:29:28.415 INFO:tasks.ceph.mds.d.smithi098.stdout:starting mds.d at 2024-03-06T07:29:28.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.437+0000 7fbb94fa0640 1 Processor -- start 2024-03-06T07:29:28.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.437+0000 7fbb94fa0640 1 -- start start 2024-03-06T07:29:28.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.438+0000 7fbb94fa0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb9006f270 0x7fbb900abaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.438+0000 7fbb94fa0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbb90155a60 0x7fbb900c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.438+0000 7fbb94fa0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb90152560 con 0x7fbb900c3920 2024-03-06T07:29:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.438+0000 7fbb94fa0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb901552f0 con 0x7fbb90155a60 2024-03-06T07:29:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.438+0000 7fbb94fa0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb901555d0 con 0x7fbb9006f270 2024-03-06T07:29:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.438+0000 7fbb8f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbb900c3920 0x7fbb90151d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60092/0 (socket says 172.21.15.1:60092) 2024-03-06T07:29:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.438+0000 7fbb8f7fe640 1 -- 172.21.15.1:0/3974983072 learned_addr learned my addr 172.21.15.1:0/3974983072 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.438+0000 7fbb8ffff640 1 --2- 172.21.15.1:0/3974983072 >> v2:172.21.15.1:3300/0 conn(0x7fbb90155a60 0x7fbb900c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.439+0000 7fbb8effd640 1 --2- 172.21.15.1:0/3974983072 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb9006f270 0x7fbb900abaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.439+0000 7fbb8e7fc640 1 -- 172.21.15.1:0/3974983072 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022979941 0 0) 0x7fbb90152560 con 0x7fbb900c3920 2024-03-06T07:29:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.439+0000 7fbb8e7fc640 1 -- 172.21.15.1:0/3974983072 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb78003520 con 0x7fbb900c3920 2024-03-06T07:29:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.439+0000 7fbb8ffff640 1 -- 172.21.15.1:0/3974983072 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb9006f270 msgr2=0x7fbb900abaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.439+0000 7fbb8ffff640 1 --2- 172.21.15.1:0/3974983072 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb9006f270 0x7fbb900abaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.439+0000 7fbb8ffff640 1 -- 172.21.15.1:0/3974983072 >> v1:172.21.15.1:6789/0 conn(0x7fbb900c3920 legacy=0x7fbb90151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.439+0000 7fbb8ffff640 1 -- 172.21.15.1:0/3974983072 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb90152840 con 0x7fbb90155a60 2024-03-06T07:29:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.440+0000 7fbb8ffff640 1 --2- 172.21.15.1:0/3974983072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb90155a60 0x7fbb900c3d00 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7fbb88009e10 tx=0x7fbb8802e330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5822896a264e994a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.440+0000 7fbb8e7fc640 1 -- 172.21.15.1:0/3974983072 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8803b070 con 0x7fbb90155a60 2024-03-06T07:29:28.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.440+0000 7fbb8e7fc640 1 -- 172.21.15.1:0/3974983072 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb88036640 con 0x7fbb90155a60 2024-03-06T07:29:28.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.440+0000 7fbb8e7fc640 1 -- 172.21.15.1:0/3974983072 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb880369f0 con 0x7fbb90155a60 2024-03-06T07:29:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.440+0000 7fbb94fa0640 1 -- 172.21.15.1:0/3974983072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb90155a60 msgr2=0x7fbb900c3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.440+0000 7fbb94fa0640 1 --2- 172.21.15.1:0/3974983072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb90155a60 0x7fbb900c3d00 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7fbb88009e10 tx=0x7fbb8802e330 comp rx=0 tx=0).stop 2024-03-06T07:29:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.440+0000 7fbb94fa0640 1 -- 172.21.15.1:0/3974983072 shutdown_connections 2024-03-06T07:29:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.440+0000 7fbb94fa0640 1 --2- 172.21.15.1:0/3974983072 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb9006f270 0x7fbb900abaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.440+0000 7fbb94fa0640 1 --2- 172.21.15.1:0/3974983072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb90155a60 0x7fbb900c3d00 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.440+0000 7fbb94fa0640 1 -- 172.21.15.1:0/3974983072 >> 172.21.15.1:0/3974983072 conn(0x7fbb9006eb50 msgr2=0x7fbb9006c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.440+0000 7fbb94fa0640 1 -- 172.21.15.1:0/3974983072 shutdown_connections 2024-03-06T07:29:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.440+0000 7fbb94fa0640 1 -- 172.21.15.1:0/3974983072 wait complete. 2024-03-06T07:29:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.441+0000 7fbb94fa0640 1 Processor -- start 2024-03-06T07:29:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.442+0000 7fbb94fa0640 1 -- start start 2024-03-06T07:29:28.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.442+0000 7fbb94fa0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9006f270 0x7fbb900c0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.442+0000 7fbb94fa0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb900c3920 0x7fbb9014aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.442+0000 7fbb94fa0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb90155a60 0x7fbb9014b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.442+0000 7fbb94fa0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb90156090 con 0x7fbb90155a60 2024-03-06T07:29:28.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.442+0000 7fbb94fa0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb90155f10 con 0x7fbb9006f270 2024-03-06T07:29:28.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.442+0000 7fbb94fa0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb90156210 con 0x7fbb900c3920 2024-03-06T07:29:28.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.442+0000 7fbb8ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb90155a60 0x7fbb9014b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.442+0000 7fbb8ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb90155a60 0x7fbb9014b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59310/0 (socket says 172.21.15.1:59310) 2024-03-06T07:29:28.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.442+0000 7fbb8ffff640 1 -- 172.21.15.1:0/3851555704 learned_addr learned my addr 172.21.15.1:0/3851555704 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.443+0000 7fbb8effd640 1 --2- 172.21.15.1:0/3851555704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb900c3920 0x7fbb9014aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.443+0000 7fbb8f7fe640 1 --2- 172.21.15.1:0/3851555704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9006f270 0x7fbb900c0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.443+0000 7fbb8ffff640 1 -- 172.21.15.1:0/3851555704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb900c3920 msgr2=0x7fbb9014aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.443+0000 7fbb8ffff640 1 --2- 172.21.15.1:0/3851555704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb900c3920 0x7fbb9014aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.443+0000 7fbb8ffff640 1 -- 172.21.15.1:0/3851555704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9006f270 msgr2=0x7fbb900c0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.443+0000 7fbb8ffff640 1 --2- 172.21.15.1:0/3851555704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9006f270 0x7fbb900c0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.443+0000 7fbb8ffff640 1 -- 172.21.15.1:0/3851555704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9014fbf0 con 0x7fbb90155a60 2024-03-06T07:29:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.443+0000 7fbb8effd640 1 --2- 172.21.15.1:0/3851555704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb900c3920 0x7fbb9014aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.443+0000 7fbb8ffff640 1 --2- 172.21.15.1:0/3851555704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb90155a60 0x7fbb9014b000 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7fbb88039040 tx=0x7fbb88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.443+0000 7fbb77fff640 1 -- 172.21.15.1:0/3851555704 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb88046070 con 0x7fbb90155a60 2024-03-06T07:29:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.443+0000 7fbb77fff640 1 -- 172.21.15.1:0/3851555704 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb88034900 con 0x7fbb90155a60 2024-03-06T07:29:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.443+0000 7fbb77fff640 1 -- 172.21.15.1:0/3851555704 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8803b070 con 0x7fbb90155a60 2024-03-06T07:29:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.444+0000 7fbb94fa0640 1 -- 172.21.15.1:0/3851555704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb900a9a10 con 0x7fbb90155a60 2024-03-06T07:29:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.444+0000 7fbb94fa0640 1 -- 172.21.15.1:0/3851555704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb90099eb0 con 0x7fbb90155a60 2024-03-06T07:29:28.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.445+0000 7fbb94fa0640 1 -- 172.21.15.1:0/3851555704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb900686e0 con 0x7fbb90155a60 2024-03-06T07:29:28.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.445+0000 7fbb77fff640 1 -- 172.21.15.1:0/3851555704 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb88049070 con 0x7fbb90155a60 2024-03-06T07:29:28.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.446+0000 7fbb77fff640 1 --2- 172.21.15.1:0/3851555704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb7c06b4d0 0x7fbb7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.446+0000 7fbb77fff640 1 -- 172.21.15.1:0/3851555704 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbb880abf60 con 0x7fbb90155a60 2024-03-06T07:29:28.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.446+0000 7fbb8f7fe640 1 --2- 172.21.15.1:0/3851555704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb7c06b4d0 0x7fbb7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.446+0000 7fbb8f7fe640 1 --2- 172.21.15.1:0/3851555704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb7c06b4d0 0x7fbb7c06d990 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fbb80005880 tx=0x7fbb80009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.450+0000 7fbb77fff640 1 -- 172.21.15.1:0/3851555704 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb8807a9e0 con 0x7fbb90155a60 2024-03-06T07:29:28.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:28.647+0000 7fbb94fa0640 1 -- 172.21.15.1:0/3851555704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fbb900725d0 con 0x7fbb90155a60 2024-03-06T07:29:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.465+0000 7fbb77fff640 1 -- 172.21.15.1:0/3851555704 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v206) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbb8807ea80 con 0x7fbb90155a60 2024-03-06T07:29:29.468 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-06T07:29:29.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.469+0000 7fbb94fa0640 1 -- 172.21.15.1:0/3851555704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb7c06b4d0 msgr2=0x7fbb7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.469+0000 7fbb94fa0640 1 --2- 172.21.15.1:0/3851555704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb7c06b4d0 0x7fbb7c06d990 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fbb80005880 tx=0x7fbb80009040 comp rx=0 tx=0).stop 2024-03-06T07:29:29.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.469+0000 7fbb94fa0640 1 -- 172.21.15.1:0/3851555704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb90155a60 msgr2=0x7fbb9014b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.469+0000 7fbb94fa0640 1 --2- 172.21.15.1:0/3851555704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb90155a60 0x7fbb9014b000 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7fbb88039040 tx=0x7fbb88004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:29.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.469+0000 7fbb94fa0640 1 -- 172.21.15.1:0/3851555704 shutdown_connections 2024-03-06T07:29:29.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.469+0000 7fbb94fa0640 1 --2- 172.21.15.1:0/3851555704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb7c06b4d0 0x7fbb7c06d990 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.469+0000 7fbb94fa0640 1 --2- 172.21.15.1:0/3851555704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb90155a60 0x7fbb9014b000 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.469+0000 7fbb94fa0640 1 --2- 172.21.15.1:0/3851555704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb900c3920 0x7fbb9014aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.469+0000 7fbb94fa0640 1 --2- 172.21.15.1:0/3851555704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9006f270 0x7fbb900c0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.469+0000 7fbb94fa0640 1 -- 172.21.15.1:0/3851555704 >> 172.21.15.1:0/3851555704 conn(0x7fbb9006eb50 msgr2=0x7fbb9006b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.469+0000 7fbb94fa0640 1 -- 172.21.15.1:0/3851555704 shutdown_connections 2024-03-06T07:29:29.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.469+0000 7fbb94fa0640 1 -- 172.21.15.1:0/3851555704 wait complete. 2024-03-06T07:29:29.484 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:29:29.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.591+0000 7fef211a0640 1 Processor -- start 2024-03-06T07:29:29.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.591+0000 7fef211a0640 1 -- start start 2024-03-06T07:29:29.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.592+0000 7fef211a0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fef1c06fa10 0x7fef1c151c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.592+0000 7fef211a0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef1c0acd70 0x7fef1c152160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.592+0000 7fef211a0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1c067360 con 0x7fef1c14f2e0 2024-03-06T07:29:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.592+0000 7fef211a0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef1c14f6c0 con 0x7fef1c06fa10 2024-03-06T07:29:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.592+0000 7fef211a0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c1526a0 con 0x7fef1c0acd70 2024-03-06T07:29:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.592+0000 7fef1bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fef1c14f2e0 0x7fef1c0c29e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60104/0 (socket says 172.21.15.1:60104) 2024-03-06T07:29:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.592+0000 7fef1bfff640 1 -- 172.21.15.1:0/2809267056 learned_addr learned my addr 172.21.15.1:0/2809267056 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1affd640 1 -- 172.21.15.1:0/2809267056 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208437627 0 0) 0x7fef1c067360 con 0x7fef1c14f2e0 2024-03-06T07:29:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1affd640 1 -- 172.21.15.1:0/2809267056 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef04005730 con 0x7fef1c14f2e0 2024-03-06T07:29:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef2099f640 1 --2- 172.21.15.1:0/2809267056 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef1c0acd70 0x7fef1c152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1b7fe640 1 --2- 172.21.15.1:0/2809267056 >> v2:172.21.15.98:3300/0 conn(0x7fef1c06fa10 0x7fef1c151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1affd640 1 -- 172.21.15.1:0/2809267056 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3798122350 0 0) 0x7fef04005730 con 0x7fef1c14f2e0 2024-03-06T07:29:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1affd640 1 -- 172.21.15.1:0/2809267056 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef1c0acd70 msgr2=0x7fef1c152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1affd640 1 --2- 172.21.15.1:0/2809267056 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef1c0acd70 0x7fef1c152160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1affd640 1 -- 172.21.15.1:0/2809267056 >> v2:172.21.15.98:3300/0 conn(0x7fef1c06fa10 msgr2=0x7fef1c151c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1affd640 1 --2- 172.21.15.1:0/2809267056 >> v2:172.21.15.98:3300/0 conn(0x7fef1c06fa10 0x7fef1c151c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1affd640 1 -- 172.21.15.1:0/2809267056 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1c070140 con 0x7fef1c14f2e0 2024-03-06T07:29:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1affd640 1 -- 172.21.15.1:0/2809267056 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fef0c001670 con 0x7fef1c14f2e0 2024-03-06T07:29:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1affd640 1 -- 172.21.15.1:0/2809267056 >> v1:172.21.15.1:6789/0 conn(0x7fef1c14f2e0 legacy=0x7fef1c0c29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1affd640 1 --2- 172.21.15.1:0/2809267056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef04005bc0 0x7fef04005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1affd640 1 --2- 172.21.15.1:0/2809267056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef04006a10 0x7fef04006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1affd640 1 --2- 172.21.15.1:0/2809267056 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef1c0acd70 0x7fef040073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1affd640 1 -- 172.21.15.1:0/2809267056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c14f6c0 con 0x7fef04005bc0 2024-03-06T07:29:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1affd640 1 -- 172.21.15.1:0/2809267056 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c1526a0 con 0x7fef04006a10 2024-03-06T07:29:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1affd640 1 -- 172.21.15.1:0/2809267056 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef0400dd00 con 0x7fef1c0acd70 2024-03-06T07:29:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1b7fe640 1 --2- 172.21.15.1:0/2809267056 >> v2:172.21.15.98:3300/0 conn(0x7fef1c06fa10 0x7fef1c151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1bfff640 1 --2- 172.21.15.1:0/2809267056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef04005bc0 0x7fef04005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef2099f640 1 --2- 172.21.15.1:0/2809267056 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef1c0acd70 0x7fef040073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1b7fe640 1 --2- 172.21.15.1:0/2809267056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef04006a10 0x7fef04006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1bfff640 1 -- 172.21.15.1:0/2809267056 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef1c0acd70 msgr2=0x7fef040073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1bfff640 1 --2- 172.21.15.1:0/2809267056 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef1c0acd70 0x7fef040073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1bfff640 1 -- 172.21.15.1:0/2809267056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef04006a10 msgr2=0x7fef04006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1bfff640 1 --2- 172.21.15.1:0/2809267056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef04006a10 0x7fef04006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1bfff640 1 -- 172.21.15.1:0/2809267056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef04005730 con 0x7fef04005bc0 2024-03-06T07:29:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef1b7fe640 1 --2- 172.21.15.1:0/2809267056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef04006a10 0x7fef04006e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.593+0000 7fef2099f640 1 --2- 172.21.15.1:0/2809267056 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef1c0acd70 0x7fef040073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.594+0000 7fef1bfff640 1 --2- 172.21.15.1:0/2809267056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef04005bc0 0x7fef04005fe0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fef0c00c210 tx=0x7fef04007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46f30de2e26005f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.594+0000 7fef1affd640 1 -- 172.21.15.1:0/2809267056 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c007d50 con 0x7fef04005bc0 2024-03-06T07:29:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.594+0000 7fef1affd640 1 -- 172.21.15.1:0/2809267056 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0c030070 con 0x7fef04005bc0 2024-03-06T07:29:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.594+0000 7fef1affd640 1 -- 172.21.15.1:0/2809267056 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c02cd80 con 0x7fef04005bc0 2024-03-06T07:29:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.595+0000 7fef211a0640 1 -- 172.21.15.1:0/2809267056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef04005bc0 msgr2=0x7fef04005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.595+0000 7fef211a0640 1 --2- 172.21.15.1:0/2809267056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef04005bc0 0x7fef04005fe0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fef0c00c210 tx=0x7fef04007a40 comp rx=0 tx=0).stop 2024-03-06T07:29:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.595+0000 7fef2099f640 1 -- 172.21.15.1:0/2809267056 reap_dead start 2024-03-06T07:29:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.595+0000 7fef211a0640 1 -- 172.21.15.1:0/2809267056 shutdown_connections 2024-03-06T07:29:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.595+0000 7fef211a0640 1 -- 172.21.15.1:0/2809267056 >> 172.21.15.1:0/2809267056 conn(0x7fef1c0729a0 msgr2=0x7fef1c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.595+0000 7fef211a0640 1 -- 172.21.15.1:0/2809267056 shutdown_connections 2024-03-06T07:29:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.595+0000 7fef211a0640 1 -- 172.21.15.1:0/2809267056 wait complete. 2024-03-06T07:29:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.596+0000 7fef211a0640 1 Processor -- start 2024-03-06T07:29:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.596+0000 7fef211a0640 1 -- start start 2024-03-06T07:29:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.597+0000 7fef211a0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef04005bc0 0x7fef1c1eb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.597+0000 7fef211a0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef1c0c4ac0 0x7fef1c0c0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.597+0000 7fef211a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef1c1ee8c0 0x7fef1c1eed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.597+0000 7fef211a0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c06ecd0 con 0x7fef04005bc0 2024-03-06T07:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.597+0000 7fef211a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c067080 con 0x7fef1c1ee8c0 2024-03-06T07:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.597+0000 7fef211a0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c06fe40 con 0x7fef1c0c4ac0 2024-03-06T07:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.597+0000 7fef1bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef04005bc0 0x7fef1c1eb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.597+0000 7fef1bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef04005bc0 0x7fef1c1eb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59320/0 (socket says 172.21.15.1:59320) 2024-03-06T07:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.597+0000 7fef1bfff640 1 -- 172.21.15.1:0/3680693527 learned_addr learned my addr 172.21.15.1:0/3680693527 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.597+0000 7fef2099f640 1 --2- 172.21.15.1:0/3680693527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef1c1ee8c0 0x7fef1c1eed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.597+0000 7fef1b7fe640 1 --2- 172.21.15.1:0/3680693527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef1c0c4ac0 0x7fef1c0c0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.597+0000 7fef1bfff640 1 -- 172.21.15.1:0/3680693527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef1c0c4ac0 msgr2=0x7fef1c0c0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.597+0000 7fef1bfff640 1 --2- 172.21.15.1:0/3680693527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef1c0c4ac0 0x7fef1c0c0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.597+0000 7fef1bfff640 1 -- 172.21.15.1:0/3680693527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef1c1ee8c0 msgr2=0x7fef1c1eed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.597+0000 7fef1bfff640 1 --2- 172.21.15.1:0/3680693527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef1c1ee8c0 0x7fef1c1eed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.597+0000 7fef1bfff640 1 -- 172.21.15.1:0/3680693527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1c0aaf40 con 0x7fef04005bc0 2024-03-06T07:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.597+0000 7fef1bfff640 1 --2- 172.21.15.1:0/3680693527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef04005bc0 0x7fef1c1eb6b0 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fef0c000fd0 tx=0x7fef0c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.598+0000 7fef18ff9640 1 -- 172.21.15.1:0/3680693527 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c045070 con 0x7fef04005bc0 2024-03-06T07:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.598+0000 7fef18ff9640 1 -- 172.21.15.1:0/3680693527 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0c0049e0 con 0x7fef04005bc0 2024-03-06T07:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.598+0000 7fef211a0640 1 -- 172.21.15.1:0/3680693527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef1c0ab200 con 0x7fef04005bc0 2024-03-06T07:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.598+0000 7fef18ff9640 1 -- 172.21.15.1:0/3680693527 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c001c90 con 0x7fef04005bc0 2024-03-06T07:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.598+0000 7fef211a0640 1 -- 172.21.15.1:0/3680693527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1c1ee3a0 con 0x7fef04005bc0 2024-03-06T07:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.599+0000 7fef211a0640 1 -- 172.21.15.1:0/3680693527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef1c1ee1c0 con 0x7fef04005bc0 2024-03-06T07:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.599+0000 7fef18ff9640 1 -- 172.21.15.1:0/3680693527 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef0c007070 con 0x7fef04005bc0 2024-03-06T07:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.599+0000 7fef18ff9640 1 --2- 172.21.15.1:0/3680693527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fef0806b500 0x7fef0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.600+0000 7fef18ff9640 1 -- 172.21.15.1:0/3680693527 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fef0c0ab0d0 con 0x7fef04005bc0 2024-03-06T07:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.600+0000 7fef1b7fe640 1 --2- 172.21.15.1:0/3680693527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fef0806b500 0x7fef0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.600+0000 7fef1b7fe640 1 --2- 172.21.15.1:0/3680693527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fef0806b500 0x7fef0806d9c0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fef10006fd0 tx=0x7fef10008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.604+0000 7fef18ff9640 1 -- 172.21.15.1:0/3680693527 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef1c1ee1c0 con 0x7fef04005bc0 2024-03-06T07:29:29.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.855+0000 7fef211a0640 1 -- 172.21.15.1:0/3680693527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fef1c0686e0 con 0x7fef04005bc0 2024-03-06T07:29:29.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.855+0000 7fef18ff9640 1 -- 172.21.15.1:0/3680693527 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v24) v1 ==== 80+0+4724 (secure 0 0 0) 0x7fef0c07a180 con 0x7fef04005bc0 2024-03-06T07:29:29.857 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:29.857 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:29:29.857 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-06T07:29:29.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:29.857 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-06T07:29:29.858 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlnxyWARAAKIPXU0IafNQ1RzBPX+3DYw==", 2024-03-06T07:29:29.858 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:29:29.858 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:29:29.858 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:29:29.858 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:29:29.858 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:29:29.858 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:29:29.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:29.858 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-06T07:29:29.858 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl2fXxDRAAuLqH+HpKcN1JTts9ZtDTXA==", 2024-03-06T07:29:29.858 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:29:29.858 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:29:29.858 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:29:29.858 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:29:29.859 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:29:29.859 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:29:29.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:29.859 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-06T07:29:29.859 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehl+IzqCBAAmgLkarFGt9Ihs/8kJYicBQ==", 2024-03-06T07:29:29.859 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:29:29.859 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:29:29.859 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:29:29.859 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:29:29.859 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:29:29.859 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:29:29.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:29.860 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-06T07:29:29.860 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlipdRFRAAA34vrsTEoX3L1CKb1YAV9g==", 2024-03-06T07:29:29.860 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:29:29.860 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:29:29.860 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:29:29.860 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:29:29.860 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:29:29.860 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:29:29.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:29.860 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-06T07:29:29.860 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAGGehlQK39LxAAOpPIZvl3FqQwCgyo3yoOGQ==", 2024-03-06T07:29:29.860 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:29:29.860 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:29.861 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:29:29.861 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:29:29.861 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:29:29.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:29.861 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-06T07:29:29.861 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAJGehlTxwDLhAAMwN9E0+3Jh5XNXnLo+fw3g==", 2024-03-06T07:29:29.861 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:29:29.861 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:29.861 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:29:29.861 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:29:29.861 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:29:29.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:29.861 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-06T07:29:29.861 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAMGehl4qrlLBAAm0VWQRNvSQIpDx5xf4Fyig==", 2024-03-06T07:29:29.862 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:29:29.862 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:29.862 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:29:29.862 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:29:29.862 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:29:29.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:29.862 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-06T07:29:29.862 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAPGehllO5HKxAAozm6ycG+fRxatoZTogeJTA==", 2024-03-06T07:29:29.862 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:29:29.862 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:29.862 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:29:29.862 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:29:29.862 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:29:29.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:29.863 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-06T07:29:29.863 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAVGehlLtLBMhAAUrV6q6ZCQluIcoo/S/NdbA==", 2024-03-06T07:29:29.863 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:29:29.863 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:29.863 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:29:29.863 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:29:29.863 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:29:29.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:29.863 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-06T07:29:29.863 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAYGehlHp59MRAAxxgpSPTEZkYKjl3fr0xeeQ==", 2024-03-06T07:29:29.863 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:29:29.863 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:29.863 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:29:29.863 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:29:29.863 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:29:29.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:29.864 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-06T07:29:29.864 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAbGehlTnXcMBAAFOBqcnh5ubAz+o6AGjccWQ==", 2024-03-06T07:29:29.864 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:29:29.864 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:29.864 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:29:29.864 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:29:29.864 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:29:29.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:29.864 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-06T07:29:29.864 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAeGehl1qH1LhAAaSl7Aj7qlN6d4xt5SKAmcg==", 2024-03-06T07:29:29.865 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:29:29.865 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:29.865 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-06T07:29:29.865 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:29:29.865 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:29:29.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:29.865 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-06T07:29:29.865 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng==", 2024-03-06T07:29:29.865 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:29:29.865 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:29:29.865 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-06T07:29:29.865 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:29:29.865 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:29:29.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:29.866 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-06T07:29:29.866 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAFGehlMyv7HBAAi7vt7aZWEaZuMpFrB5ofZw==", 2024-03-06T07:29:29.866 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:29:29.866 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-06T07:29:29.866 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-06T07:29:29.866 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-06T07:29:29.866 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-06T07:29:29.866 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:29:29.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:29.866 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-06T07:29:29.866 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlBS/vIBAAN86N2K4ze5PJB8kR869Lfg==", 2024-03-06T07:29:29.866 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:29:29.867 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:29:29.867 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-06T07:29:29.867 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-06T07:29:29.867 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:29:29.867 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:29:29.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:29.867 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-06T07:29:29.867 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlJ/jPORAAhLk5MTGafsGKhZSYhvhBZg==", 2024-03-06T07:29:29.867 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:29:29.867 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:29:29.868 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:29:29.868 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:29:29.868 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-06T07:29:29.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:29.868 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-06T07:29:29.868 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQAEGehlAl8rNhAAvdU0Qvo/FJYnxc4/H8MIqQ==", 2024-03-06T07:29:29.868 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-06T07:29:29.868 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-06T07:29:29.868 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-06T07:29:29.868 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-06T07:29:29.868 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.868 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:29.869 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-06T07:29:29.869 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:29:29.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.857+0000 7fef211a0640 1 -- 172.21.15.1:0/3680693527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fef0806b500 msgr2=0x7fef0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.857+0000 7fef211a0640 1 --2- 172.21.15.1:0/3680693527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fef0806b500 0x7fef0806d9c0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fef10006fd0 tx=0x7fef10008040 comp rx=0 tx=0).stop 2024-03-06T07:29:29.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.857+0000 7fef211a0640 1 -- 172.21.15.1:0/3680693527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef04005bc0 msgr2=0x7fef1c1eb6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.857+0000 7fef211a0640 1 --2- 172.21.15.1:0/3680693527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef04005bc0 0x7fef1c1eb6b0 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fef0c000fd0 tx=0x7fef0c001a40 comp rx=0 tx=0).stop 2024-03-06T07:29:29.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.857+0000 7fef211a0640 1 -- 172.21.15.1:0/3680693527 shutdown_connections 2024-03-06T07:29:29.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.857+0000 7fef211a0640 1 --2- 172.21.15.1:0/3680693527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fef0806b500 0x7fef0806d9c0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.857+0000 7fef211a0640 1 --2- 172.21.15.1:0/3680693527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef1c1ee8c0 0x7fef1c1eed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.857+0000 7fef211a0640 1 --2- 172.21.15.1:0/3680693527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fef1c0c4ac0 0x7fef1c0c0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.857+0000 7fef211a0640 1 --2- 172.21.15.1:0/3680693527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef04005bc0 0x7fef1c1eb6b0 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.857+0000 7fef211a0640 1 -- 172.21.15.1:0/3680693527 >> 172.21.15.1:0/3680693527 conn(0x7fef1c0729a0 msgr2=0x7fef1c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.857+0000 7fef211a0640 1 -- 172.21.15.1:0/3680693527 shutdown_connections 2024-03-06T07:29:29.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.857+0000 7fef211a0640 1 -- 172.21.15.1:0/3680693527 wait complete. 2024-03-06T07:29:29.871 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:29.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.970+0000 7fa98ce8f640 1 Processor -- start 2024-03-06T07:29:29.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.971+0000 7fa98ce8f640 1 -- start start 2024-03-06T07:29:29.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.971+0000 7fa98ce8f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa988152650 0x7fa988155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.971+0000 7fa98ce8f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9880bf9d0 0x7fa9880c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.971+0000 7fa98ce8f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9880c2fc0 con 0x7fa9880c6b10 2024-03-06T07:29:29.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.971+0000 7fa98ce8f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa988155800 con 0x7fa988152650 2024-03-06T07:29:29.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.971+0000 7fa98ce8f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa988158da0 con 0x7fa9880bf9d0 2024-03-06T07:29:29.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.972+0000 7fa985d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa9880c6b10 0x7fa988158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60110/0 (socket says 172.21.15.1:60110) 2024-03-06T07:29:29.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.972+0000 7fa985d9b640 1 -- 172.21.15.1:0/1749345951 learned_addr learned my addr 172.21.15.1:0/1749345951 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:29.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.972+0000 7fa986d9d640 1 --2- 172.21.15.1:0/1749345951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9880bf9d0 0x7fa9880c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.972+0000 7fa98659c640 1 --2- 172.21.15.1:0/1749345951 >> v2:172.21.15.98:3300/0 conn(0x7fa988152650 0x7fa988155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.972+0000 7fa98559a640 1 -- 172.21.15.1:0/1749345951 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178162475 0 0) 0x7fa9880c2fc0 con 0x7fa9880c6b10 2024-03-06T07:29:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.972+0000 7fa98559a640 1 -- 172.21.15.1:0/1749345951 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa970003520 con 0x7fa9880c6b10 2024-03-06T07:29:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.972+0000 7fa986d9d640 1 -- 172.21.15.1:0/1749345951 >> v2:172.21.15.98:3300/0 conn(0x7fa988152650 msgr2=0x7fa988155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.972+0000 7fa986d9d640 1 --2- 172.21.15.1:0/1749345951 >> v2:172.21.15.98:3300/0 conn(0x7fa988152650 0x7fa988155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.972+0000 7fa986d9d640 1 -- 172.21.15.1:0/1749345951 >> v1:172.21.15.1:6789/0 conn(0x7fa9880c6b10 legacy=0x7fa988158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.972+0000 7fa986d9d640 1 -- 172.21.15.1:0/1749345951 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9880c6690 con 0x7fa9880bf9d0 2024-03-06T07:29:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.972+0000 7fa98659c640 1 --2- 172.21.15.1:0/1749345951 >> v2:172.21.15.98:3300/0 conn(0x7fa988152650 0x7fa988155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.973+0000 7fa986d9d640 1 --2- 172.21.15.1:0/1749345951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9880bf9d0 0x7fa9880c1460 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa980009d90 tx=0x7fa98002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4099e90f6efd55f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.973+0000 7fa98559a640 1 -- 172.21.15.1:0/1749345951 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980039070 con 0x7fa9880bf9d0 2024-03-06T07:29:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.973+0000 7fa98559a640 1 -- 172.21.15.1:0/1749345951 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98002cbc0 con 0x7fa9880bf9d0 2024-03-06T07:29:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.973+0000 7fa98559a640 1 -- 172.21.15.1:0/1749345951 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980034440 con 0x7fa9880bf9d0 2024-03-06T07:29:29.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.974+0000 7fa98ce8f640 1 -- 172.21.15.1:0/1749345951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9880bf9d0 msgr2=0x7fa9880c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.974+0000 7fa98ce8f640 1 --2- 172.21.15.1:0/1749345951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9880bf9d0 0x7fa9880c1460 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa980009d90 tx=0x7fa98002be10 comp rx=0 tx=0).stop 2024-03-06T07:29:29.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.974+0000 7fa98ce8f640 1 -- 172.21.15.1:0/1749345951 shutdown_connections 2024-03-06T07:29:29.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.974+0000 7fa98ce8f640 1 --2- 172.21.15.1:0/1749345951 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa9880bf9d0 0x7fa9880c1460 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.974+0000 7fa98ce8f640 1 --2- 172.21.15.1:0/1749345951 >> v2:172.21.15.98:3300/0 conn(0x7fa988152650 0x7fa988155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.974+0000 7fa98ce8f640 1 -- 172.21.15.1:0/1749345951 >> 172.21.15.1:0/1749345951 conn(0x7fa988066ee0 msgr2=0x7fa9880672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.974+0000 7fa98ce8f640 1 -- 172.21.15.1:0/1749345951 shutdown_connections 2024-03-06T07:29:29.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.974+0000 7fa98ce8f640 1 -- 172.21.15.1:0/1749345951 wait complete. 2024-03-06T07:29:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.975+0000 7fa98ce8f640 1 Processor -- start 2024-03-06T07:29:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.975+0000 7fa98ce8f640 1 -- start start 2024-03-06T07:29:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.975+0000 7fa98ce8f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9880bf9d0 0x7fa988158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.975+0000 7fa98ce8f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa988152650 0x7fa9881eb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa98ce8f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9881eb5b0 0x7fa98814ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa98ce8f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9881551b0 con 0x7fa9881eb5b0 2024-03-06T07:29:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa98ce8f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9880c6fa0 con 0x7fa9880bf9d0 2024-03-06T07:29:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa98ce8f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa988155490 con 0x7fa988152650 2024-03-06T07:29:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa986d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9881eb5b0 0x7fa98814ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa986d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9881eb5b0 0x7fa98814ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59322/0 (socket says 172.21.15.1:59322) 2024-03-06T07:29:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa986d9d640 1 -- 172.21.15.1:0/3719429518 learned_addr learned my addr 172.21.15.1:0/3719429518 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa985d9b640 1 --2- 172.21.15.1:0/3719429518 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa988152650 0x7fa9881eb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa98659c640 1 --2- 172.21.15.1:0/3719429518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9880bf9d0 0x7fa988158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa986d9d640 1 -- 172.21.15.1:0/3719429518 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa988152650 msgr2=0x7fa9881eb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa986d9d640 1 --2- 172.21.15.1:0/3719429518 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa988152650 0x7fa9881eb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa986d9d640 1 -- 172.21.15.1:0/3719429518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9880bf9d0 msgr2=0x7fa988158410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa986d9d640 1 --2- 172.21.15.1:0/3719429518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9880bf9d0 0x7fa988158410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa986d9d640 1 -- 172.21.15.1:0/3719429518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9881e1770 con 0x7fa9881eb5b0 2024-03-06T07:29:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa98659c640 1 --2- 172.21.15.1:0/3719429518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9880bf9d0 0x7fa988158410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa986d9d640 1 --2- 172.21.15.1:0/3719429518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9881eb5b0 0x7fa98814ca90 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7fa980037040 tx=0x7fa980004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa976ffd640 1 -- 172.21.15.1:0/3719429518 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980039050 con 0x7fa9881eb5b0 2024-03-06T07:29:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa976ffd640 1 -- 172.21.15.1:0/3719429518 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98003a040 con 0x7fa9881eb5b0 2024-03-06T07:29:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa98ce8f640 1 -- 172.21.15.1:0/3719429518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9881e1a50 con 0x7fa9881eb5b0 2024-03-06T07:29:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa976ffd640 1 -- 172.21.15.1:0/3719429518 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980009450 con 0x7fa9881eb5b0 2024-03-06T07:29:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.976+0000 7fa98ce8f640 1 -- 172.21.15.1:0/3719429518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9881e2010 con 0x7fa9881eb5b0 2024-03-06T07:29:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.977+0000 7fa98ce8f640 1 -- 172.21.15.1:0/3719429518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa988067fd0 con 0x7fa9881eb5b0 2024-03-06T07:29:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.978+0000 7fa976ffd640 1 -- 172.21.15.1:0/3719429518 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa980047020 con 0x7fa9881eb5b0 2024-03-06T07:29:29.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.978+0000 7fa976ffd640 1 --2- 172.21.15.1:0/3719429518 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa96806b4d0 0x7fa96806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.978+0000 7fa976ffd640 1 -- 172.21.15.1:0/3719429518 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fa9800abff0 con 0x7fa9881eb5b0 2024-03-06T07:29:29.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.978+0000 7fa98659c640 1 --2- 172.21.15.1:0/3719429518 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa96806b4d0 0x7fa96806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.979+0000 7fa98659c640 1 --2- 172.21.15.1:0/3719429518 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa96806b4d0 0x7fa96806d990 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fa978009ba0 tx=0x7fa978008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:29.983+0000 7fa976ffd640 1 -- 172.21.15.1:0/3719429518 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa98004e020 con 0x7fa9881eb5b0 2024-03-06T07:29:30.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.164+0000 7fa98ce8f640 1 -- 172.21.15.1:0/3719429518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9880b06f0 con 0x7fa9881eb5b0 2024-03-06T07:29:30.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.165+0000 7fa976ffd640 1 -- 172.21.15.1:0/3719429518 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v206) v1 ==== 74+0+13054 (secure 0 0 0) 0x7fa98007b0f0 con 0x7fa9881eb5b0 2024-03-06T07:29:30.167 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:30.168 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:29:29.465298+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:30.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.167+0000 7fa98ce8f640 1 -- 172.21.15.1:0/3719429518 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa96806b4d0 msgr2=0x7fa96806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.167+0000 7fa98ce8f640 1 --2- 172.21.15.1:0/3719429518 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa96806b4d0 0x7fa96806d990 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fa978009ba0 tx=0x7fa978008040 comp rx=0 tx=0).stop 2024-03-06T07:29:30.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.167+0000 7fa98ce8f640 1 -- 172.21.15.1:0/3719429518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9881eb5b0 msgr2=0x7fa98814ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.167+0000 7fa98ce8f640 1 --2- 172.21.15.1:0/3719429518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9881eb5b0 0x7fa98814ca90 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7fa980037040 tx=0x7fa980004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:30.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.167+0000 7fa98ce8f640 1 -- 172.21.15.1:0/3719429518 shutdown_connections 2024-03-06T07:29:30.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.167+0000 7fa98ce8f640 1 --2- 172.21.15.1:0/3719429518 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa96806b4d0 0x7fa96806d990 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.167+0000 7fa98ce8f640 1 --2- 172.21.15.1:0/3719429518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9881eb5b0 0x7fa98814ca90 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.167+0000 7fa98ce8f640 1 --2- 172.21.15.1:0/3719429518 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa988152650 0x7fa9881eb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.167+0000 7fa98ce8f640 1 --2- 172.21.15.1:0/3719429518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9880bf9d0 0x7fa988158410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.167+0000 7fa98ce8f640 1 -- 172.21.15.1:0/3719429518 >> 172.21.15.1:0/3719429518 conn(0x7fa988066ee0 msgr2=0x7fa9880c5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.167+0000 7fa98ce8f640 1 -- 172.21.15.1:0/3719429518 shutdown_connections 2024-03-06T07:29:30.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.167+0000 7fa98ce8f640 1 -- 172.21.15.1:0/3719429518 wait complete. 2024-03-06T07:29:30.180 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:30.180 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:30.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.282+0000 7f2486ec2640 1 Processor -- start 2024-03-06T07:29:30.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.282+0000 7f2486ec2640 1 -- start start 2024-03-06T07:29:30.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.282+0000 7f2486ec2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2480152650 0x7f2480155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f2486ec2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24800c2fc0 con 0x7f24800c6b10 2024-03-06T07:29:30.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f2486ec2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2480158da0 con 0x7f24800bf9d0 2024-03-06T07:29:30.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f2486ec2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2480155800 con 0x7f2480152650 2024-03-06T07:29:30.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f247ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f24800c6b10 0x7f2480158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60120/0 (socket says 172.21.15.1:60120) 2024-03-06T07:29:30.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f247ffff640 1 -- 172.21.15.1:0/1264132917 learned_addr learned my addr 172.21.15.1:0/1264132917 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:30.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f247f7fe640 1 -- 172.21.15.1:0/1264132917 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143621280 0 0) 0x7f24800c2fc0 con 0x7f24800c6b10 2024-03-06T07:29:30.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f2484c5e640 1 --2- 172.21.15.1:0/1264132917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2480152650 0x7f2480155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f247f7fe640 1 -- 172.21.15.1:0/1264132917 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2470003520 con 0x7f24800c6b10 2024-03-06T07:29:30.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f247f7fe640 1 -- 172.21.15.1:0/1264132917 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3324236901 0 0) 0x7f2470003520 con 0x7f24800c6b10 2024-03-06T07:29:30.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f247f7fe640 1 -- 172.21.15.1:0/1264132917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2480152650 msgr2=0x7f2480155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f247f7fe640 1 --2- 172.21.15.1:0/1264132917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2480152650 0x7f2480155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f247f7fe640 1 -- 172.21.15.1:0/1264132917 >> v1:172.21.15.98:6789/0 conn(0x7f24800bf9d0 legacy=0x7f24800c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f247f7fe640 1 -- 172.21.15.1:0/1264132917 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24800c6690 con 0x7f24800c6b10 2024-03-06T07:29:30.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f247f7fe640 1 -- 172.21.15.1:0/1264132917 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f246c001670 con 0x7f24800c6b10 2024-03-06T07:29:30.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f247f7fe640 1 -- 172.21.15.1:0/1264132917 >> v1:172.21.15.1:6789/0 conn(0x7f24800c6b10 legacy=0x7f2480158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f247f7fe640 1 --2- 172.21.15.1:0/1264132917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24700039e0 0x7f2470003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f247f7fe640 1 --2- 172.21.15.1:0/1264132917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2480152650 0x7f2470004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f247f7fe640 1 --2- 172.21.15.1:0/1264132917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2470004d40 0x7f24700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f247f7fe640 1 -- 172.21.15.1:0/1264132917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2480155800 con 0x7f24700039e0 2024-03-06T07:29:30.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f247f7fe640 1 -- 172.21.15.1:0/1264132917 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247000baf0 con 0x7f2470004d40 2024-03-06T07:29:30.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f247f7fe640 1 -- 172.21.15.1:0/1264132917 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247000bcf0 con 0x7f2480152650 2024-03-06T07:29:30.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.283+0000 7f2484c5e640 1 --2- 172.21.15.1:0/1264132917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24700039e0 0x7f2470003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.284+0000 7f247ffff640 1 --2- 172.21.15.1:0/1264132917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2480152650 0x7f2470004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.284+0000 7f248545f640 1 --2- 172.21.15.1:0/1264132917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2470004d40 0x7f24700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.284+0000 7f2484c5e640 1 -- 172.21.15.1:0/1264132917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2480152650 msgr2=0x7f2470004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.284+0000 7f2484c5e640 1 --2- 172.21.15.1:0/1264132917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2480152650 0x7f2470004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.284+0000 7f2484c5e640 1 -- 172.21.15.1:0/1264132917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2470004d40 msgr2=0x7f24700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.284+0000 7f2484c5e640 1 --2- 172.21.15.1:0/1264132917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2470004d40 0x7f24700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.284+0000 7f2484c5e640 1 -- 172.21.15.1:0/1264132917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2470003520 con 0x7f24700039e0 2024-03-06T07:29:30.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.284+0000 7f248545f640 1 --2- 172.21.15.1:0/1264132917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2470004d40 0x7f24700051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:30.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.284+0000 7f2484c5e640 1 --2- 172.21.15.1:0/1264132917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24700039e0 0x7f2470003e00 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f2474009ac0 tx=0x7f2470005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1505f1829e4dee70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.285+0000 7f247f7fe640 1 -- 172.21.15.1:0/1264132917 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2474039070 con 0x7f24700039e0 2024-03-06T07:29:30.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.285+0000 7f247f7fe640 1 -- 172.21.15.1:0/1264132917 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247402c600 con 0x7f24700039e0 2024-03-06T07:29:30.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.285+0000 7f247f7fe640 1 -- 172.21.15.1:0/1264132917 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2474009690 con 0x7f24700039e0 2024-03-06T07:29:30.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.285+0000 7f2486ec2640 1 -- 172.21.15.1:0/1264132917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24700039e0 msgr2=0x7f2470003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.285+0000 7f2486ec2640 1 --2- 172.21.15.1:0/1264132917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24700039e0 0x7f2470003e00 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f2474009ac0 tx=0x7f2470005830 comp rx=0 tx=0).stop 2024-03-06T07:29:30.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.285+0000 7f248545f640 1 -- 172.21.15.1:0/1264132917 reap_dead start 2024-03-06T07:29:30.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.285+0000 7f2486ec2640 1 -- 172.21.15.1:0/1264132917 shutdown_connections 2024-03-06T07:29:30.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.285+0000 7f2486ec2640 1 -- 172.21.15.1:0/1264132917 >> 172.21.15.1:0/1264132917 conn(0x7f2480066ee0 msgr2=0x7f24800672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.285+0000 7f2486ec2640 1 -- 172.21.15.1:0/1264132917 shutdown_connections 2024-03-06T07:29:30.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.285+0000 7f2486ec2640 1 -- 172.21.15.1:0/1264132917 wait complete. 2024-03-06T07:29:30.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.286+0000 7f2486ec2640 1 Processor -- start 2024-03-06T07:29:30.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.286+0000 7f2486ec2640 1 -- start start 2024-03-06T07:29:30.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.287+0000 7f2486ec2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24700039e0 0x7f24801ebcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.287+0000 7f2486ec2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f248014fe70 0x7f248014e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.287+0000 7f2486ec2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24801ee7d0 0x7f24801eec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.287+0000 7f2486ec2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24801551b0 con 0x7f248014fe70 2024-03-06T07:29:30.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.287+0000 7f2486ec2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24800c6fa0 con 0x7f24700039e0 2024-03-06T07:29:30.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.287+0000 7f2486ec2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2480155490 con 0x7f24801ee7d0 2024-03-06T07:29:30.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.287+0000 7f247ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f248014fe70 0x7f248014e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.287+0000 7f247ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f248014fe70 0x7f248014e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59346/0 (socket says 172.21.15.1:59346) 2024-03-06T07:29:30.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.287+0000 7f247ffff640 1 -- 172.21.15.1:0/2677234144 learned_addr learned my addr 172.21.15.1:0/2677234144 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:30.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.287+0000 7f248545f640 1 --2- 172.21.15.1:0/2677234144 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24801ee7d0 0x7f24801eec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.287+0000 7f2484c5e640 1 --2- 172.21.15.1:0/2677234144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24700039e0 0x7f24801ebcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.287+0000 7f247ffff640 1 -- 172.21.15.1:0/2677234144 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24801ee7d0 msgr2=0x7f24801eec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.287+0000 7f247ffff640 1 --2- 172.21.15.1:0/2677234144 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24801ee7d0 0x7f24801eec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.287+0000 7f247ffff640 1 -- 172.21.15.1:0/2677234144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24700039e0 msgr2=0x7f24801ebcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.287+0000 7f247ffff640 1 --2- 172.21.15.1:0/2677234144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24700039e0 0x7f24801ebcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.287+0000 7f247ffff640 1 -- 172.21.15.1:0/2677234144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24801519e0 con 0x7f248014fe70 2024-03-06T07:29:30.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.288+0000 7f248545f640 1 --2- 172.21.15.1:0/2677234144 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24801ee7d0 0x7f24801eec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:30.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.288+0000 7f2484c5e640 1 --2- 172.21.15.1:0/2677234144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24700039e0 0x7f24801ebcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:30.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.288+0000 7f247ffff640 1 --2- 172.21.15.1:0/2677234144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f248014fe70 0x7f248014e5f0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f246c004fa0 tx=0x7f246c0074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.288+0000 7f247d7fa640 1 -- 172.21.15.1:0/2677234144 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c007740 con 0x7f248014fe70 2024-03-06T07:29:30.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.288+0000 7f247d7fa640 1 -- 172.21.15.1:0/2677234144 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246c018070 con 0x7f248014fe70 2024-03-06T07:29:30.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.288+0000 7f247d7fa640 1 -- 172.21.15.1:0/2677234144 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c00fbd0 con 0x7f248014fe70 2024-03-06T07:29:30.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.288+0000 7f2486ec2640 1 -- 172.21.15.1:0/2677234144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2480151c80 con 0x7f248014fe70 2024-03-06T07:29:30.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.288+0000 7f2486ec2640 1 -- 172.21.15.1:0/2677234144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248014eb30 con 0x7f248014fe70 2024-03-06T07:29:30.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.289+0000 7f2486ec2640 1 -- 172.21.15.1:0/2677234144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2480067fd0 con 0x7f248014fe70 2024-03-06T07:29:30.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.289+0000 7f247d7fa640 1 -- 172.21.15.1:0/2677234144 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f246c01d460 con 0x7f248014fe70 2024-03-06T07:29:30.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.290+0000 7f247d7fa640 1 --2- 172.21.15.1:0/2677234144 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f246406b6a0 0x7f246406db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.290+0000 7f247d7fa640 1 -- 172.21.15.1:0/2677234144 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f246c089fd0 con 0x7f248014fe70 2024-03-06T07:29:30.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.290+0000 7f2484c5e640 1 --2- 172.21.15.1:0/2677234144 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f246406b6a0 0x7f246406db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.290+0000 7f2484c5e640 1 --2- 172.21.15.1:0/2677234144 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f246406b6a0 0x7f246406db60 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f2474036000 tx=0x7f2474030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.294+0000 7f247d7fa640 1 -- 172.21.15.1:0/2677234144 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f246c0589c0 con 0x7f248014fe70 2024-03-06T07:29:30.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.473+0000 7f2486ec2640 1 -- 172.21.15.1:0/2677234144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2480151e50 con 0x7f248014fe70 2024-03-06T07:29:30.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.474+0000 7f247d7fa640 1 -- 172.21.15.1:0/2677234144 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v206) v1 ==== 93+0+10 (secure 0 0 0) 0x7f246c05ca60 con 0x7f248014fe70 2024-03-06T07:29:30.476 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:29:30.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.475+0000 7f2486ec2640 1 -- 172.21.15.1:0/2677234144 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f246406b6a0 msgr2=0x7f246406db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.475+0000 7f2486ec2640 1 --2- 172.21.15.1:0/2677234144 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f246406b6a0 0x7f246406db60 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f2474036000 tx=0x7f2474030000 comp rx=0 tx=0).stop 2024-03-06T07:29:30.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.475+0000 7f2486ec2640 1 -- 172.21.15.1:0/2677234144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f248014fe70 msgr2=0x7f248014e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.475+0000 7f2486ec2640 1 --2- 172.21.15.1:0/2677234144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f248014fe70 0x7f248014e5f0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f246c004fa0 tx=0x7f246c0074f0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.476+0000 7f2486ec2640 1 -- 172.21.15.1:0/2677234144 shutdown_connections 2024-03-06T07:29:30.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.476+0000 7f2486ec2640 1 --2- 172.21.15.1:0/2677234144 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f246406b6a0 0x7f246406db60 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.476+0000 7f2486ec2640 1 --2- 172.21.15.1:0/2677234144 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24801ee7d0 0x7f24801eec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.476+0000 7f2486ec2640 1 --2- 172.21.15.1:0/2677234144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f248014fe70 0x7f248014e5f0 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.476+0000 7f2486ec2640 1 --2- 172.21.15.1:0/2677234144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24700039e0 0x7f24801ebcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.476+0000 7f2486ec2640 1 -- 172.21.15.1:0/2677234144 >> 172.21.15.1:0/2677234144 conn(0x7f2480066ee0 msgr2=0x7f24800cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.476+0000 7f2486ec2640 1 -- 172.21.15.1:0/2677234144 shutdown_connections 2024-03-06T07:29:30.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.476+0000 7f2486ec2640 1 -- 172.21.15.1:0/2677234144 wait complete. 2024-03-06T07:29:30.489 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:29:30.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.589+0000 7f447390f640 1 Processor -- start 2024-03-06T07:29:30.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.589+0000 7f447390f640 1 -- start start 2024-03-06T07:29:30.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.590+0000 7f447390f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f446c0c2b30 0x7f446c0724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.590+0000 7f447390f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f446c152b70 0x7f446c072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.590+0000 7f447390f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446c151dd0 con 0x7f446c0a5fa0 2024-03-06T07:29:30.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.590+0000 7f447390f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f446c1526e0 con 0x7f446c0c2b30 2024-03-06T07:29:30.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.590+0000 7f447390f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f446c156190 con 0x7f446c152b70 2024-03-06T07:29:30.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.590+0000 7f44716ab640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f446c0c2b30 0x7f446c0724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.590+0000 7f4471eac640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f446c0a5fa0 0x7f446c0c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60134/0 (socket says 172.21.15.1:60134) 2024-03-06T07:29:30.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.590+0000 7f44716ab640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f446c0c2b30 0x7f446c0724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59360/0 (socket says 172.21.15.1:59360) 2024-03-06T07:29:30.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.590+0000 7f4471eac640 1 -- 172.21.15.1:0/2679708845 learned_addr learned my addr 172.21.15.1:0/2679708845 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:30.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.590+0000 7f4470eaa640 1 --2- 172.21.15.1:0/2679708845 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f446c152b70 0x7f446c072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.590+0000 7f44716ab640 1 -- 172.21.15.1:0/2679708845 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f446c152b70 msgr2=0x7f446c072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.590+0000 7f44716ab640 1 --2- 172.21.15.1:0/2679708845 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f446c152b70 0x7f446c072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.590+0000 7f44716ab640 1 -- 172.21.15.1:0/2679708845 >> v1:172.21.15.1:6789/0 conn(0x7f446c0a5fa0 legacy=0x7f446c0c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.590+0000 7f44716ab640 1 -- 172.21.15.1:0/2679708845 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446c152090 con 0x7f446c0c2b30 2024-03-06T07:29:30.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.590+0000 7f4463fff640 1 -- 172.21.15.1:0/2679708845 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350925431 0 0) 0x7f446c151dd0 con 0x7f446c0a5fa0 2024-03-06T07:29:30.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.590+0000 7f4470eaa640 1 --2- 172.21.15.1:0/2679708845 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f446c152b70 0x7f446c072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:30.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.591+0000 7f44716ab640 1 --2- 172.21.15.1:0/2679708845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f446c0c2b30 0x7f446c0724c0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f44680097a0 tx=0x7f446802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cee446759e0de32c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.591+0000 7f4463fff640 1 -- 172.21.15.1:0/2679708845 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4468035020 con 0x7f446c0c2b30 2024-03-06T07:29:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.591+0000 7f4463fff640 1 -- 172.21.15.1:0/2679708845 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4468031830 con 0x7f446c0c2b30 2024-03-06T07:29:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.591+0000 7f4463fff640 1 -- 172.21.15.1:0/2679708845 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4468031c00 con 0x7f446c0c2b30 2024-03-06T07:29:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.592+0000 7f447390f640 1 -- 172.21.15.1:0/2679708845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f446c0c2b30 msgr2=0x7f446c0724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.592+0000 7f447390f640 1 --2- 172.21.15.1:0/2679708845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f446c0c2b30 0x7f446c0724c0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f44680097a0 tx=0x7f446802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.592+0000 7f447390f640 1 -- 172.21.15.1:0/2679708845 shutdown_connections 2024-03-06T07:29:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.592+0000 7f447390f640 1 --2- 172.21.15.1:0/2679708845 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f446c152b70 0x7f446c072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.592+0000 7f447390f640 1 --2- 172.21.15.1:0/2679708845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f446c0c2b30 0x7f446c0724c0 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.592+0000 7f447390f640 1 -- 172.21.15.1:0/2679708845 >> 172.21.15.1:0/2679708845 conn(0x7f446c0c6860 msgr2=0x7f446c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.592+0000 7f447390f640 1 -- 172.21.15.1:0/2679708845 shutdown_connections 2024-03-06T07:29:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.592+0000 7f447390f640 1 -- 172.21.15.1:0/2679708845 wait complete. 2024-03-06T07:29:30.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.593+0000 7f447390f640 1 Processor -- start 2024-03-06T07:29:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.593+0000 7f447390f640 1 -- start start 2024-03-06T07:29:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.593+0000 7f447390f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f446c0a5fa0 0x7f446c1ecd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.593+0000 7f447390f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f446c0c2b30 0x7f446c1dbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.593+0000 7f447390f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f446c152b70 0x7f446c1e94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.593+0000 7f447390f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446c155f00 con 0x7f446c0c2b30 2024-03-06T07:29:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.593+0000 7f447390f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446c155d80 con 0x7f446c0a5fa0 2024-03-06T07:29:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.593+0000 7f447390f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f446c0c3a00 con 0x7f446c152b70 2024-03-06T07:29:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.594+0000 7f4470eaa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f446c0c2b30 0x7f446c1dbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.594+0000 7f4470eaa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f446c0c2b30 0x7f446c1dbc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59364/0 (socket says 172.21.15.1:59364) 2024-03-06T07:29:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.594+0000 7f4470eaa640 1 -- 172.21.15.1:0/1511602392 learned_addr learned my addr 172.21.15.1:0/1511602392 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.594+0000 7f44716ab640 1 --2- 172.21.15.1:0/1511602392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f446c0a5fa0 0x7f446c1ecd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.594+0000 7f4471eac640 1 --2- 172.21.15.1:0/1511602392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f446c152b70 0x7f446c1e94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.594+0000 7f4470eaa640 1 -- 172.21.15.1:0/1511602392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f446c152b70 msgr2=0x7f446c1e94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.594+0000 7f4470eaa640 1 --2- 172.21.15.1:0/1511602392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f446c152b70 0x7f446c1e94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.594+0000 7f4470eaa640 1 -- 172.21.15.1:0/1511602392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f446c0a5fa0 msgr2=0x7f446c1ecd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.594+0000 7f4470eaa640 1 --2- 172.21.15.1:0/1511602392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f446c0a5fa0 0x7f446c1ecd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.594+0000 7f4470eaa640 1 -- 172.21.15.1:0/1511602392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446c1dcb90 con 0x7f446c0c2b30 2024-03-06T07:29:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.594+0000 7f4471eac640 1 --2- 172.21.15.1:0/1511602392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f446c152b70 0x7f446c1e94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.594+0000 7f4470eaa640 1 --2- 172.21.15.1:0/1511602392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f446c0c2b30 0x7f446c1dbc90 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f445c00d800 tx=0x7f445c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.594+0000 7f4461ffb640 1 -- 172.21.15.1:0/1511602392 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445c004030 con 0x7f446c0c2b30 2024-03-06T07:29:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.594+0000 7f4461ffb640 1 -- 172.21.15.1:0/1511602392 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445c0041d0 con 0x7f446c0c2b30 2024-03-06T07:29:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.594+0000 7f447390f640 1 -- 172.21.15.1:0/1511602392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446c14e1a0 con 0x7f446c0c2b30 2024-03-06T07:29:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.594+0000 7f4461ffb640 1 -- 172.21.15.1:0/1511602392 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445c0096a0 con 0x7f446c0c2b30 2024-03-06T07:29:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.594+0000 7f447390f640 1 -- 172.21.15.1:0/1511602392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f446c1dc1d0 con 0x7f446c0c2b30 2024-03-06T07:29:30.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.596+0000 7f447390f640 1 -- 172.21.15.1:0/1511602392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f446c14e6c0 con 0x7f446c0c2b30 2024-03-06T07:29:30.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.596+0000 7f4461ffb640 1 -- 172.21.15.1:0/1511602392 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f445c009840 con 0x7f446c0c2b30 2024-03-06T07:29:30.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.596+0000 7f4461ffb640 1 --2- 172.21.15.1:0/1511602392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f445006b4d0 0x7f445006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.596+0000 7f4461ffb640 1 -- 172.21.15.1:0/1511602392 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f445c08a1b0 con 0x7f446c0c2b30 2024-03-06T07:29:30.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.596+0000 7f44716ab640 1 --2- 172.21.15.1:0/1511602392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f445006b4d0 0x7f445006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.597+0000 7f44716ab640 1 --2- 172.21.15.1:0/1511602392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f445006b4d0 0x7f445006d990 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4468036000 tx=0x7f4468008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.600+0000 7f4461ffb640 1 -- 172.21.15.1:0/1511602392 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f446c14e6c0 con 0x7f446c0c2b30 2024-03-06T07:29:30.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.782+0000 7f447390f640 1 -- 172.21.15.1:0/1511602392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f446c1e8d80 con 0x7f446c0c2b30 2024-03-06T07:29:30.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.782+0000 7f4461ffb640 1 -- 172.21.15.1:0/1511602392 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v206) v1 ==== 84+0+65 (secure 0 0 0) 0x7f445c059280 con 0x7f446c0c2b30 2024-03-06T07:29:30.785 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:30.785 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-06T07:29:30.785 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-06T07:29:30.785 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-06T07:29:30.785 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-06T07:29:30.785 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-06T07:29:30.785 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-06T07:29:30.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.785+0000 7f447390f640 1 -- 172.21.15.1:0/1511602392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f445006b4d0 msgr2=0x7f445006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.785+0000 7f447390f640 1 --2- 172.21.15.1:0/1511602392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f445006b4d0 0x7f445006d990 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4468036000 tx=0x7f4468008030 comp rx=0 tx=0).stop 2024-03-06T07:29:30.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.785+0000 7f447390f640 1 -- 172.21.15.1:0/1511602392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f446c0c2b30 msgr2=0x7f446c1dbc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.785+0000 7f447390f640 1 --2- 172.21.15.1:0/1511602392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f446c0c2b30 0x7f446c1dbc90 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f445c00d800 tx=0x7f445c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.785+0000 7f447390f640 1 -- 172.21.15.1:0/1511602392 shutdown_connections 2024-03-06T07:29:30.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.785+0000 7f447390f640 1 --2- 172.21.15.1:0/1511602392 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f445006b4d0 0x7f445006d990 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.785+0000 7f447390f640 1 --2- 172.21.15.1:0/1511602392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f446c152b70 0x7f446c1e94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.785+0000 7f447390f640 1 --2- 172.21.15.1:0/1511602392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f446c0c2b30 0x7f446c1dbc90 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.785+0000 7f447390f640 1 --2- 172.21.15.1:0/1511602392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f446c0a5fa0 0x7f446c1ecd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.785+0000 7f447390f640 1 -- 172.21.15.1:0/1511602392 >> 172.21.15.1:0/1511602392 conn(0x7f446c0c6860 msgr2=0x7f446c1531c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.785+0000 7f447390f640 1 -- 172.21.15.1:0/1511602392 shutdown_connections 2024-03-06T07:29:30.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.785+0000 7f447390f640 1 -- 172.21.15.1:0/1511602392 wait complete. 2024-03-06T07:29:30.797 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:29:30.798 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:29:30.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.901+0000 7f8969f81640 1 Processor -- start 2024-03-06T07:29:30.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.901+0000 7f8969f81640 1 -- start start 2024-03-06T07:29:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.902+0000 7f8969f81640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8964152a30 0x7f89640c08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.902+0000 7f8969f81640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8964155200 0x7f8964155650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.902+0000 7f8969f81640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89640c5f80 con 0x7f89640c0e20 2024-03-06T07:29:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.902+0000 7f8969f81640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89640c6260 con 0x7f8964152a30 2024-03-06T07:29:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.902+0000 7f8969f81640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89640c6540 con 0x7f8964155200 2024-03-06T07:29:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.902+0000 7f89637fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8964152a30 0x7f89640c08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.902+0000 7f89637fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8964152a30 0x7f89640c08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59368/0 (socket says 172.21.15.1:59368) 2024-03-06T07:29:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.902+0000 7f89637fe640 1 -- 172.21.15.1:0/1982916654 learned_addr learned my addr 172.21.15.1:0/1982916654 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.902+0000 7f8963fff640 1 --2- 172.21.15.1:0/1982916654 >> v2:172.21.15.98:3300/0 conn(0x7f8964155200 0x7f8964155650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.902+0000 7f89637fe640 1 -- 172.21.15.1:0/1982916654 >> v2:172.21.15.98:3300/0 conn(0x7f8964155200 msgr2=0x7f8964155650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.902+0000 7f89637fe640 1 --2- 172.21.15.1:0/1982916654 >> v2:172.21.15.98:3300/0 conn(0x7f8964155200 0x7f8964155650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.902+0000 7f89637fe640 1 -- 172.21.15.1:0/1982916654 >> v1:172.21.15.98:6789/0 conn(0x7f89640c0e20 legacy=0x7f89640cab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.902+0000 7f89637fe640 1 -- 172.21.15.1:0/1982916654 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89640c6820 con 0x7f8964152a30 2024-03-06T07:29:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.902+0000 7f8963fff640 1 --2- 172.21.15.1:0/1982916654 >> v2:172.21.15.98:3300/0 conn(0x7f8964155200 0x7f8964155650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.903+0000 7f89637fe640 1 --2- 172.21.15.1:0/1982916654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8964152a30 0x7f89640c08e0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f89640cbd10 tx=0x7f895402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1e0f131667b6af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.903+0000 7f89627fc640 1 -- 172.21.15.1:0/1982916654 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954038070 con 0x7f8964152a30 2024-03-06T07:29:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.903+0000 7f89627fc640 1 -- 172.21.15.1:0/1982916654 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f895402f050 con 0x7f8964152a30 2024-03-06T07:29:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.903+0000 7f89627fc640 1 -- 172.21.15.1:0/1982916654 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954031970 con 0x7f8964152a30 2024-03-06T07:29:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.904+0000 7f8969f81640 1 -- 172.21.15.1:0/1982916654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8964152a30 msgr2=0x7f89640c08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.904+0000 7f8969f81640 1 --2- 172.21.15.1:0/1982916654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8964152a30 0x7f89640c08e0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f89640cbd10 tx=0x7f895402b7f0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.904+0000 7f8969f81640 1 -- 172.21.15.1:0/1982916654 shutdown_connections 2024-03-06T07:29:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.904+0000 7f8969f81640 1 --2- 172.21.15.1:0/1982916654 >> v2:172.21.15.98:3300/0 conn(0x7f8964155200 0x7f8964155650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.904+0000 7f8969f81640 1 --2- 172.21.15.1:0/1982916654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8964152a30 0x7f89640c08e0 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.904+0000 7f8969f81640 1 -- 172.21.15.1:0/1982916654 >> 172.21.15.1:0/1982916654 conn(0x7f8964072b40 msgr2=0x7f896406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.904+0000 7f8969f81640 1 -- 172.21.15.1:0/1982916654 shutdown_connections 2024-03-06T07:29:30.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.904+0000 7f8969f81640 1 -- 172.21.15.1:0/1982916654 wait complete. 2024-03-06T07:29:30.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.905+0000 7f8969f81640 1 Processor -- start 2024-03-06T07:29:30.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.905+0000 7f8969f81640 1 -- start start 2024-03-06T07:29:30.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f8969f81640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89640c0e20 0x7f89640ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f8969f81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8964152a30 0x7f896413d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f8969f81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8964155200 0x7f896413d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f8969f81640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89640cb270 con 0x7f89640c0e20 2024-03-06T07:29:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f8969f81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89640c12c0 con 0x7f8964155200 2024-03-06T07:29:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f8969f81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89640cb3f0 con 0x7f8964152a30 2024-03-06T07:29:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f89637fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89640c0e20 0x7f89640ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f89637fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89640c0e20 0x7f89640ca110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59384/0 (socket says 172.21.15.1:59384) 2024-03-06T07:29:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f89637fe640 1 -- 172.21.15.1:0/2071787552 learned_addr learned my addr 172.21.15.1:0/2071787552 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f8962ffd640 1 --2- 172.21.15.1:0/2071787552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8964152a30 0x7f896413d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f8963fff640 1 --2- 172.21.15.1:0/2071787552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8964155200 0x7f896413d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f89637fe640 1 -- 172.21.15.1:0/2071787552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8964152a30 msgr2=0x7f896413d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f89637fe640 1 --2- 172.21.15.1:0/2071787552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8964152a30 0x7f896413d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f89637fe640 1 -- 172.21.15.1:0/2071787552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8964155200 msgr2=0x7f896413d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f89637fe640 1 --2- 172.21.15.1:0/2071787552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8964155200 0x7f896413d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f89637fe640 1 -- 172.21.15.1:0/2071787552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896413ddc0 con 0x7f89640c0e20 2024-03-06T07:29:30.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f8963fff640 1 --2- 172.21.15.1:0/2071787552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8964155200 0x7f896413d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:30.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f89637fe640 1 --2- 172.21.15.1:0/2071787552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89640c0e20 0x7f89640ca110 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f8954000c00 tx=0x7f89540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f894bfff640 1 -- 172.21.15.1:0/2071787552 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954042070 con 0x7f89640c0e20 2024-03-06T07:29:30.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f894bfff640 1 -- 172.21.15.1:0/2071787552 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f895403d870 con 0x7f89640c0e20 2024-03-06T07:29:30.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f8969f81640 1 -- 172.21.15.1:0/2071787552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f896414e490 con 0x7f89640c0e20 2024-03-06T07:29:30.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f894bfff640 1 -- 172.21.15.1:0/2071787552 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954038040 con 0x7f89640c0e20 2024-03-06T07:29:30.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.906+0000 7f8969f81640 1 -- 172.21.15.1:0/2071787552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89641eeb40 con 0x7f89640c0e20 2024-03-06T07:29:30.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.907+0000 7f8969f81640 1 -- 172.21.15.1:0/2071787552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89640680b0 con 0x7f89640c0e20 2024-03-06T07:29:30.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.908+0000 7f894bfff640 1 -- 172.21.15.1:0/2071787552 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8954045070 con 0x7f89640c0e20 2024-03-06T07:29:30.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.908+0000 7f894bfff640 1 --2- 172.21.15.1:0/2071787552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f895006b4d0 0x7f895006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.908+0000 7f894bfff640 1 -- 172.21.15.1:0/2071787552 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f89540abfe0 con 0x7f89640c0e20 2024-03-06T07:29:30.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.909+0000 7f8962ffd640 1 --2- 172.21.15.1:0/2071787552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f895006b4d0 0x7f895006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.909+0000 7f8962ffd640 1 --2- 172.21.15.1:0/2071787552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f895006b4d0 0x7f895006d990 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f89640cabc0 tx=0x7f8958009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:30.913+0000 7f894bfff640 1 -- 172.21.15.1:0/2071787552 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f895407b1d0 con 0x7f89640c0e20 2024-03-06T07:29:31.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.094+0000 7f8969f81640 1 -- 172.21.15.1:0/2071787552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f89640c7b90 con 0x7f89640c0e20 2024-03-06T07:29:31.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.472+0000 7f894bfff640 1 -- 172.21.15.1:0/2071787552 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v207) v1 ==== 138+0+0 (secure 0 0 0) 0x7f895407aad0 con 0x7f89640c0e20 2024-03-06T07:29:31.474 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-06T07:29:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.475+0000 7f8969f81640 1 -- 172.21.15.1:0/2071787552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f895006b4d0 msgr2=0x7f895006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.475+0000 7f8969f81640 1 --2- 172.21.15.1:0/2071787552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f895006b4d0 0x7f895006d990 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f89640cabc0 tx=0x7f8958009040 comp rx=0 tx=0).stop 2024-03-06T07:29:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.475+0000 7f8969f81640 1 -- 172.21.15.1:0/2071787552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89640c0e20 msgr2=0x7f89640ca110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.475+0000 7f8969f81640 1 --2- 172.21.15.1:0/2071787552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89640c0e20 0x7f89640ca110 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f8954000c00 tx=0x7f89540046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.475+0000 7f8969f81640 1 -- 172.21.15.1:0/2071787552 shutdown_connections 2024-03-06T07:29:31.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.475+0000 7f8969f81640 1 --2- 172.21.15.1:0/2071787552 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f895006b4d0 0x7f895006d990 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.475+0000 7f8969f81640 1 --2- 172.21.15.1:0/2071787552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8964155200 0x7f896413d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.475+0000 7f8969f81640 1 --2- 172.21.15.1:0/2071787552 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8964152a30 0x7f896413d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.475+0000 7f8969f81640 1 --2- 172.21.15.1:0/2071787552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89640c0e20 0x7f89640ca110 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.475+0000 7f8969f81640 1 -- 172.21.15.1:0/2071787552 >> 172.21.15.1:0/2071787552 conn(0x7f8964072b40 msgr2=0x7f896406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.475+0000 7f8969f81640 1 -- 172.21.15.1:0/2071787552 shutdown_connections 2024-03-06T07:29:31.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.475+0000 7f8969f81640 1 -- 172.21.15.1:0/2071787552 wait complete. 2024-03-06T07:29:31.492 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:29:31.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.600+0000 7f4c823e2640 1 Processor -- start 2024-03-06T07:29:31.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.600+0000 7f4c823e2640 1 -- start start 2024-03-06T07:29:31.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.601+0000 7f4c823e2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7c152920 0x7f4c7c06aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.601+0000 7f4c823e2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c7c155220 0x7f4c7c1556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.601+0000 7f4c823e2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c7c158910 con 0x7f4c7c06b530 2024-03-06T07:29:31.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.601+0000 7f4c823e2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7c152d00 con 0x7f4c7c155220 2024-03-06T07:29:31.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.601+0000 7f4c823e2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7c1596d0 con 0x7f4c7c152920 2024-03-06T07:29:31.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.601+0000 7f4c8097f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c7c155220 0x7f4c7c1556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.601+0000 7f4c8097f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c7c155220 0x7f4c7c1556b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59386/0 (socket says 172.21.15.1:59386) 2024-03-06T07:29:31.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.601+0000 7f4c8097f640 1 -- 172.21.15.1:0/3718698754 learned_addr learned my addr 172.21.15.1:0/3718698754 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:31.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.601+0000 7f4c7bfff640 1 --2- 172.21.15.1:0/3718698754 >> v2:172.21.15.98:3300/0 conn(0x7f4c7c152920 0x7f4c7c06aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.601+0000 7f4c8097f640 1 -- 172.21.15.1:0/3718698754 >> v2:172.21.15.98:3300/0 conn(0x7f4c7c152920 msgr2=0x7f4c7c06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:31.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.601+0000 7f4c8097f640 1 --2- 172.21.15.1:0/3718698754 >> v2:172.21.15.98:3300/0 conn(0x7f4c7c152920 0x7f4c7c06aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.601+0000 7f4c8097f640 1 -- 172.21.15.1:0/3718698754 >> v1:172.21.15.1:6789/0 conn(0x7f4c7c06b530 legacy=0x7f4c7c0c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.601+0000 7f4c7affd640 1 -- 172.21.15.1:0/3718698754 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059625688 0 0) 0x7f4c7c158910 con 0x7f4c7c06b530 2024-03-06T07:29:31.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.601+0000 7f4c8097f640 1 -- 172.21.15.1:0/3718698754 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7c158bf0 con 0x7f4c7c155220 2024-03-06T07:29:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.602+0000 7f4c8097f640 1 --2- 172.21.15.1:0/3718698754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c7c155220 0x7f4c7c1556b0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c06bd50 tx=0x7f4c7402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88bb199fc4345e66 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.602+0000 7f4c7affd640 1 -- 172.21.15.1:0/3718698754 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7402ca80 con 0x7f4c7c155220 2024-03-06T07:29:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.602+0000 7f4c7affd640 1 -- 172.21.15.1:0/3718698754 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7402cc20 con 0x7f4c7c155220 2024-03-06T07:29:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.602+0000 7f4c7affd640 1 -- 172.21.15.1:0/3718698754 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c740343f0 con 0x7f4c7c155220 2024-03-06T07:29:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.603+0000 7f4c823e2640 1 -- 172.21.15.1:0/3718698754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c7c155220 msgr2=0x7f4c7c1556b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.603+0000 7f4c823e2640 1 --2- 172.21.15.1:0/3718698754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c7c155220 0x7f4c7c1556b0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c06bd50 tx=0x7f4c7402bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.603+0000 7f4c823e2640 1 -- 172.21.15.1:0/3718698754 shutdown_connections 2024-03-06T07:29:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.603+0000 7f4c823e2640 1 --2- 172.21.15.1:0/3718698754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c7c155220 0x7f4c7c1556b0 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.603+0000 7f4c823e2640 1 --2- 172.21.15.1:0/3718698754 >> v2:172.21.15.98:3300/0 conn(0x7f4c7c152920 0x7f4c7c06aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.603+0000 7f4c823e2640 1 -- 172.21.15.1:0/3718698754 >> 172.21.15.1:0/3718698754 conn(0x7f4c7c0a9700 msgr2=0x7f4c7c0a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.603+0000 7f4c823e2640 1 -- 172.21.15.1:0/3718698754 shutdown_connections 2024-03-06T07:29:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.603+0000 7f4c823e2640 1 -- 172.21.15.1:0/3718698754 wait complete. 2024-03-06T07:29:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.604+0000 7f4c823e2640 1 Processor -- start 2024-03-06T07:29:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.604+0000 7f4c823e2640 1 -- start start 2024-03-06T07:29:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c823e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c7c06b530 0x7f4c7c0a8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c823e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c7c152920 0x7f4c7c1ee3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c823e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c7c155220 0x7f4c7c1ee8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c823e2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c0c6380 con 0x7f4c7c155220 2024-03-06T07:29:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c823e2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c159240 con 0x7f4c7c06b530 2024-03-06T07:29:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c823e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c0c5f60 con 0x7f4c7c152920 2024-03-06T07:29:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c8097f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c7c155220 0x7f4c7c1ee8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c8097f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c7c155220 0x7f4c7c1ee8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59388/0 (socket says 172.21.15.1:59388) 2024-03-06T07:29:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c8097f640 1 -- 172.21.15.1:0/2351869849 learned_addr learned my addr 172.21.15.1:0/2351869849 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c7bfff640 1 --2- 172.21.15.1:0/2351869849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c7c06b530 0x7f4c7c0a8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c8097f640 1 -- 172.21.15.1:0/2351869849 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c7c152920 msgr2=0x7f4c7c1ee3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c7b7fe640 1 --2- 172.21.15.1:0/2351869849 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c7c152920 0x7f4c7c1ee3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c8097f640 1 --2- 172.21.15.1:0/2351869849 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c7c152920 0x7f4c7c1ee3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c8097f640 1 -- 172.21.15.1:0/2351869849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c7c06b530 msgr2=0x7f4c7c0a8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c8097f640 1 --2- 172.21.15.1:0/2351869849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c7c06b530 0x7f4c7c0a8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c8097f640 1 -- 172.21.15.1:0/2351869849 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7c0c4f50 con 0x7f4c7c155220 2024-03-06T07:29:31.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c8097f640 1 --2- 172.21.15.1:0/2351869849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c7c155220 0x7f4c7c1ee8e0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f4c74036040 tx=0x7f4c74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c78ff9640 1 -- 172.21.15.1:0/2351869849 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74043030 con 0x7f4c7c155220 2024-03-06T07:29:31.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c78ff9640 1 -- 172.21.15.1:0/2351869849 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7403e450 con 0x7f4c7c155220 2024-03-06T07:29:31.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c78ff9640 1 -- 172.21.15.1:0/2351869849 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74009510 con 0x7f4c7c155220 2024-03-06T07:29:31.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.605+0000 7f4c823e2640 1 -- 172.21.15.1:0/2351869849 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c7c0c3aa0 con 0x7f4c7c155220 2024-03-06T07:29:31.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.606+0000 7f4c823e2640 1 -- 172.21.15.1:0/2351869849 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c7c14ab90 con 0x7f4c7c155220 2024-03-06T07:29:31.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.607+0000 7f4c823e2640 1 -- 172.21.15.1:0/2351869849 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c7c1587c0 con 0x7f4c7c155220 2024-03-06T07:29:31.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.607+0000 7f4c78ff9640 1 -- 172.21.15.1:0/2351869849 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c74046070 con 0x7f4c7c155220 2024-03-06T07:29:31.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.608+0000 7f4c78ff9640 1 --2- 172.21.15.1:0/2351869849 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c5c06b4d0 0x7f4c5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.608+0000 7f4c78ff9640 1 -- 172.21.15.1:0/2351869849 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f4c740ac5a0 con 0x7f4c7c155220 2024-03-06T07:29:31.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.608+0000 7f4c7bfff640 1 --2- 172.21.15.1:0/2351869849 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c5c06b4d0 0x7f4c5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.608+0000 7f4c7bfff640 1 --2- 172.21.15.1:0/2351869849 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c5c06b4d0 0x7f4c5c06d990 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009a10 tx=0x7f4c6c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.612+0000 7f4c78ff9640 1 -- 172.21.15.1:0/2351869849 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c7c1587c0 con 0x7f4c7c155220 2024-03-06T07:29:31.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:31.795+0000 7f4c823e2640 1 -- 172.21.15.1:0/2351869849 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4c7c067fd0 con 0x7f4c7c155220 2024-03-06T07:29:32.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.478+0000 7f4c78ff9640 1 -- 172.21.15.1:0/2351869849 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v208) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4c7404d070 con 0x7f4c7c155220 2024-03-06T07:29:32.480 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-06T07:29:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.482+0000 7f4c823e2640 1 -- 172.21.15.1:0/2351869849 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c5c06b4d0 msgr2=0x7f4c5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.482+0000 7f4c823e2640 1 --2- 172.21.15.1:0/2351869849 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c5c06b4d0 0x7f4c5c06d990 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009a10 tx=0x7f4c6c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.482+0000 7f4c823e2640 1 -- 172.21.15.1:0/2351869849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c7c155220 msgr2=0x7f4c7c1ee8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.482+0000 7f4c823e2640 1 --2- 172.21.15.1:0/2351869849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c7c155220 0x7f4c7c1ee8e0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f4c74036040 tx=0x7f4c74004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.482+0000 7f4c823e2640 1 -- 172.21.15.1:0/2351869849 shutdown_connections 2024-03-06T07:29:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.482+0000 7f4c823e2640 1 --2- 172.21.15.1:0/2351869849 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4c5c06b4d0 0x7f4c5c06d990 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.482+0000 7f4c823e2640 1 --2- 172.21.15.1:0/2351869849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c7c155220 0x7f4c7c1ee8e0 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.482+0000 7f4c823e2640 1 --2- 172.21.15.1:0/2351869849 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c7c152920 0x7f4c7c1ee3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.482+0000 7f4c823e2640 1 --2- 172.21.15.1:0/2351869849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c7c06b530 0x7f4c7c0a8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.482+0000 7f4c823e2640 1 -- 172.21.15.1:0/2351869849 >> 172.21.15.1:0/2351869849 conn(0x7f4c7c0a9700 msgr2=0x7f4c7c0a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.482+0000 7f4c823e2640 1 -- 172.21.15.1:0/2351869849 shutdown_connections 2024-03-06T07:29:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.482+0000 7f4c823e2640 1 -- 172.21.15.1:0/2351869849 wait complete. 2024-03-06T07:29:32.514 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:29:32.606 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:32.608 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:32.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.620+0000 7ff74bc9a640 1 Processor -- start 2024-03-06T07:29:32.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.621+0000 7ff74bc9a640 1 -- start start 2024-03-06T07:29:32.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.621+0000 7ff74bc9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff744152820 0x7ff74406aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.621+0000 7ff74bc9a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff7440c0e80 0x7ff7441586b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.621+0000 7ff74bc9a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7440c2fb0 con 0x7ff74406b530 2024-03-06T07:29:32.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.621+0000 7ff74bc9a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff744152c00 con 0x7ff7440c0e80 2024-03-06T07:29:32.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.621+0000 7ff74bc9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7440c35e0 con 0x7ff744152820 2024-03-06T07:29:32.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.621+0000 7ff749235640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff7440c0e80 0x7ff7441586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.621+0000 7ff74a237640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff74406b530 0x7ff7440726c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60148/0 (socket says 172.21.15.1:60148) 2024-03-06T07:29:32.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.621+0000 7ff74a237640 1 -- 172.21.15.1:0/3197736296 learned_addr learned my addr 172.21.15.1:0/3197736296 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:32.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.622+0000 7ff749a36640 1 --2- 172.21.15.1:0/3197736296 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff744152820 0x7ff74406aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.622+0000 7ff749235640 1 -- 172.21.15.1:0/3197736296 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff744152820 msgr2=0x7ff74406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:32.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.622+0000 7ff749235640 1 --2- 172.21.15.1:0/3197736296 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff744152820 0x7ff74406aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.622+0000 7ff749235640 1 -- 172.21.15.1:0/3197736296 >> v1:172.21.15.1:6789/0 conn(0x7ff74406b530 legacy=0x7ff7440726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.622+0000 7ff748a34640 1 -- 172.21.15.1:0/3197736296 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043719058 0 0) 0x7ff7440c2fb0 con 0x7ff74406b530 2024-03-06T07:29:32.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.622+0000 7ff749235640 1 -- 172.21.15.1:0/3197736296 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7440c16e0 con 0x7ff7440c0e80 2024-03-06T07:29:32.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.623+0000 7ff749235640 1 --2- 172.21.15.1:0/3197736296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7440c0e80 0x7ff7441586b0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7ff7340097a0 tx=0x7ff73402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0e36a09612ab9f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.623+0000 7ff748a34640 1 -- 172.21.15.1:0/3197736296 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff734035020 con 0x7ff7440c0e80 2024-03-06T07:29:32.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.623+0000 7ff748a34640 1 -- 172.21.15.1:0/3197736296 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff734031830 con 0x7ff7440c0e80 2024-03-06T07:29:32.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.623+0000 7ff748a34640 1 -- 172.21.15.1:0/3197736296 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff734031c00 con 0x7ff7440c0e80 2024-03-06T07:29:32.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.623+0000 7ff74bc9a640 1 -- 172.21.15.1:0/3197736296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7440c0e80 msgr2=0x7ff7441586b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.623+0000 7ff74bc9a640 1 --2- 172.21.15.1:0/3197736296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7440c0e80 0x7ff7441586b0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7ff7340097a0 tx=0x7ff73402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.623+0000 7ff74bc9a640 1 -- 172.21.15.1:0/3197736296 shutdown_connections 2024-03-06T07:29:32.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.623+0000 7ff74bc9a640 1 --2- 172.21.15.1:0/3197736296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7440c0e80 0x7ff7441586b0 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.623+0000 7ff74bc9a640 1 --2- 172.21.15.1:0/3197736296 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff744152820 0x7ff74406aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.623+0000 7ff74bc9a640 1 -- 172.21.15.1:0/3197736296 >> 172.21.15.1:0/3197736296 conn(0x7ff7440a9fe0 msgr2=0x7ff74406c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.624+0000 7ff74bc9a640 1 -- 172.21.15.1:0/3197736296 shutdown_connections 2024-03-06T07:29:32.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.624+0000 7ff74bc9a640 1 -- 172.21.15.1:0/3197736296 wait complete. 2024-03-06T07:29:32.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.625+0000 7ff74bc9a640 1 Processor -- start 2024-03-06T07:29:32.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.625+0000 7ff74bc9a640 1 -- start start 2024-03-06T07:29:32.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.625+0000 7ff74bc9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff74406b530 0x7ff74414f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.625+0000 7ff74bc9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7440c0e80 0x7ff7440aab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.625+0000 7ff74bc9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff744152820 0x7ff7441eb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.625+0000 7ff74bc9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7440c1390 con 0x7ff744152820 2024-03-06T07:29:32.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.625+0000 7ff74bc9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7440c3a70 con 0x7ff7440c0e80 2024-03-06T07:29:32.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.625+0000 7ff74bc9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7440c3ef0 con 0x7ff74406b530 2024-03-06T07:29:32.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.625+0000 7ff74a237640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff744152820 0x7ff7441eb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.625+0000 7ff74a237640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff744152820 0x7ff7441eb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59410/0 (socket says 172.21.15.1:59410) 2024-03-06T07:29:32.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.625+0000 7ff74a237640 1 -- 172.21.15.1:0/309553310 learned_addr learned my addr 172.21.15.1:0/309553310 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:32.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.626+0000 7ff749a36640 1 --2- 172.21.15.1:0/309553310 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff74406b530 0x7ff74414f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.626+0000 7ff749235640 1 --2- 172.21.15.1:0/309553310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7440c0e80 0x7ff7440aab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.626+0000 7ff74a237640 1 -- 172.21.15.1:0/309553310 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff74406b530 msgr2=0x7ff74414f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.626+0000 7ff74a237640 1 --2- 172.21.15.1:0/309553310 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff74406b530 0x7ff74414f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.626+0000 7ff74a237640 1 -- 172.21.15.1:0/309553310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7440c0e80 msgr2=0x7ff7440aab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.626+0000 7ff74a237640 1 --2- 172.21.15.1:0/309553310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7440c0e80 0x7ff7440aab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.626+0000 7ff74a237640 1 -- 172.21.15.1:0/309553310 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7440ab4a0 con 0x7ff744152820 2024-03-06T07:29:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.626+0000 7ff749235640 1 --2- 172.21.15.1:0/309553310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7440c0e80 0x7ff7440aab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.626+0000 7ff74a237640 1 --2- 172.21.15.1:0/309553310 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff744152820 0x7ff7441eb630 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7ff73c00c7a0 tx=0x7ff73c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.626+0000 7ff73a7fc640 1 -- 172.21.15.1:0/309553310 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73c018070 con 0x7ff744152820 2024-03-06T07:29:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.626+0000 7ff73a7fc640 1 -- 172.21.15.1:0/309553310 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73c00a640 con 0x7ff744152820 2024-03-06T07:29:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.626+0000 7ff74bc9a640 1 -- 172.21.15.1:0/309553310 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7440ab740 con 0x7ff744152820 2024-03-06T07:29:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.626+0000 7ff73a7fc640 1 -- 172.21.15.1:0/309553310 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73c00aab0 con 0x7ff744152820 2024-03-06T07:29:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.626+0000 7ff74bc9a640 1 -- 172.21.15.1:0/309553310 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7440abd00 con 0x7ff744152820 2024-03-06T07:29:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.627+0000 7ff74bc9a640 1 -- 172.21.15.1:0/309553310 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7440686e0 con 0x7ff744152820 2024-03-06T07:29:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.628+0000 7ff73a7fc640 1 -- 172.21.15.1:0/309553310 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff73c01b070 con 0x7ff744152820 2024-03-06T07:29:32.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.628+0000 7ff73a7fc640 1 --2- 172.21.15.1:0/309553310 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff72806b4d0 0x7ff72806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.628+0000 7ff73a7fc640 1 -- 172.21.15.1:0/309553310 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6001+0+0 (secure 0 0 0) 0x7ff73c089830 con 0x7ff744152820 2024-03-06T07:29:32.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.628+0000 7ff749a36640 1 --2- 172.21.15.1:0/309553310 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff72806b4d0 0x7ff72806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.629+0000 7ff749a36640 1 --2- 172.21.15.1:0/309553310 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff72806b4d0 0x7ff72806d990 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7ff740009aa0 tx=0x7ff740009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.633+0000 7ff73a7fc640 1 -- 172.21.15.1:0/309553310 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff73c057fc0 con 0x7ff744152820 2024-03-06T07:29:32.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:32.851+0000 7ff74bc9a640 1 -- 172.21.15.1:0/309553310 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff7440c6a60 con 0x7ff744152820 2024-03-06T07:29:33.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.489+0000 7ff73a7fc640 1 -- 172.21.15.1:0/309553310 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 33 and data pool 34 v193) v1 ==== 174+0+0 (secure 0 0 0) 0x7ff73c05c060 con 0x7ff744152820 2024-03-06T07:29:33.503 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 33 and data pool 34 2024-03-06T07:29:33.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.508+0000 7ff74bc9a640 1 -- 172.21.15.1:0/309553310 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff72806b4d0 msgr2=0x7ff72806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.508+0000 7ff74bc9a640 1 --2- 172.21.15.1:0/309553310 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff72806b4d0 0x7ff72806d990 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7ff740009aa0 tx=0x7ff740009190 comp rx=0 tx=0).stop 2024-03-06T07:29:33.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.508+0000 7ff74bc9a640 1 -- 172.21.15.1:0/309553310 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff744152820 msgr2=0x7ff7441eb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.508+0000 7ff74bc9a640 1 --2- 172.21.15.1:0/309553310 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff744152820 0x7ff7441eb630 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7ff73c00c7a0 tx=0x7ff73c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:29:33.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.509+0000 7ff74bc9a640 1 -- 172.21.15.1:0/309553310 shutdown_connections 2024-03-06T07:29:33.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.509+0000 7ff74bc9a640 1 --2- 172.21.15.1:0/309553310 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff72806b4d0 0x7ff72806d990 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.509+0000 7ff74bc9a640 1 --2- 172.21.15.1:0/309553310 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff744152820 0x7ff7441eb630 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.509+0000 7ff74bc9a640 1 --2- 172.21.15.1:0/309553310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7440c0e80 0x7ff7440aab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.509+0000 7ff74bc9a640 1 --2- 172.21.15.1:0/309553310 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff74406b530 0x7ff74414f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.509+0000 7ff74bc9a640 1 -- 172.21.15.1:0/309553310 >> 172.21.15.1:0/309553310 conn(0x7ff7440a9fe0 msgr2=0x7ff744151b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.509+0000 7ff74bc9a640 1 -- 172.21.15.1:0/309553310 shutdown_connections 2024-03-06T07:29:33.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.510+0000 7ff74bc9a640 1 -- 172.21.15.1:0/309553310 wait complete. 2024-03-06T07:29:33.528 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:33.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.632+0000 7f5e37a66640 1 Processor -- start 2024-03-06T07:29:33.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.632+0000 7f5e37a66640 1 -- start start 2024-03-06T07:29:33.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.633+0000 7f5e37a66640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5e300a9120 0x7f5e3006bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.633+0000 7f5e37a66640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e301475b0 0x7f5e30147a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.633+0000 7f5e37a66640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e300c3830 con 0x7f5e300c3450 2024-03-06T07:29:33.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.633+0000 7f5e37a66640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e300a8ca0 con 0x7f5e300a9120 2024-03-06T07:29:33.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.633+0000 7f5e37a66640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e30151f80 con 0x7f5e301475b0 2024-03-06T07:29:33.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.633+0000 7f5e35802640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5e300c3450 0x7f5e3006b410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60164/0 (socket says 172.21.15.1:60164) 2024-03-06T07:29:33.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.633+0000 7f5e35802640 1 -- 172.21.15.1:0/1373701302 learned_addr learned my addr 172.21.15.1:0/1373701302 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:33.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.633+0000 7f5e35001640 1 --2- 172.21.15.1:0/1373701302 >> v2:172.21.15.1:3300/0 conn(0x7f5e300a9120 0x7f5e3006bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.633+0000 7f5e36003640 1 --2- 172.21.15.1:0/1373701302 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e301475b0 0x7f5e30147a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.634+0000 7f5e35001640 1 -- 172.21.15.1:0/1373701302 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e301475b0 msgr2=0x7f5e30147a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.634+0000 7f5e35001640 1 --2- 172.21.15.1:0/1373701302 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e301475b0 0x7f5e30147a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.634+0000 7f5e35001640 1 -- 172.21.15.1:0/1373701302 >> v1:172.21.15.1:6789/0 conn(0x7f5e300c3450 legacy=0x7f5e3006b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.634+0000 7f5e34800640 1 -- 172.21.15.1:0/1373701302 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723540632 0 0) 0x7f5e300c3830 con 0x7f5e300c3450 2024-03-06T07:29:33.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.634+0000 7f5e35001640 1 -- 172.21.15.1:0/1373701302 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e30152260 con 0x7f5e300a9120 2024-03-06T07:29:33.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.634+0000 7f5e35001640 1 --2- 172.21.15.1:0/1373701302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e300a9120 0x7f5e3006bb20 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400a940 tx=0x7f5e2402cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18be016c61a749e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.635+0000 7f5e34800640 1 -- 172.21.15.1:0/1373701302 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2403a070 con 0x7f5e300a9120 2024-03-06T07:29:33.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.635+0000 7f5e34800640 1 -- 172.21.15.1:0/1373701302 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2402db30 con 0x7f5e300a9120 2024-03-06T07:29:33.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.635+0000 7f5e34800640 1 -- 172.21.15.1:0/1373701302 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2400b040 con 0x7f5e300a9120 2024-03-06T07:29:33.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.635+0000 7f5e37a66640 1 -- 172.21.15.1:0/1373701302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e300a9120 msgr2=0x7f5e3006bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.635+0000 7f5e37a66640 1 --2- 172.21.15.1:0/1373701302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e300a9120 0x7f5e3006bb20 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400a940 tx=0x7f5e2402cd80 comp rx=0 tx=0).stop 2024-03-06T07:29:33.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.635+0000 7f5e37a66640 1 -- 172.21.15.1:0/1373701302 shutdown_connections 2024-03-06T07:29:33.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.635+0000 7f5e37a66640 1 --2- 172.21.15.1:0/1373701302 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e301475b0 0x7f5e30147a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.635+0000 7f5e37a66640 1 --2- 172.21.15.1:0/1373701302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e300a9120 0x7f5e3006bb20 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.635+0000 7f5e37a66640 1 -- 172.21.15.1:0/1373701302 >> 172.21.15.1:0/1373701302 conn(0x7f5e300c3cc0 msgr2=0x7f5e300c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.635+0000 7f5e37a66640 1 -- 172.21.15.1:0/1373701302 shutdown_connections 2024-03-06T07:29:33.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.635+0000 7f5e37a66640 1 -- 172.21.15.1:0/1373701302 wait complete. 2024-03-06T07:29:33.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.637+0000 7f5e37a66640 1 Processor -- start 2024-03-06T07:29:33.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.637+0000 7f5e37a66640 1 -- start start 2024-03-06T07:29:33.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.637+0000 7f5e37a66640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e300a9120 0x7f5e300c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.637+0000 7f5e37a66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e300c3450 0x7f5e30142a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.637+0000 7f5e37a66640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e301475b0 0x7f5e30142f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.637+0000 7f5e37a66640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3006c060 con 0x7f5e300a9120 2024-03-06T07:29:33.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.637+0000 7f5e37a66640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e300a9dc0 con 0x7f5e300c3450 2024-03-06T07:29:33.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.637+0000 7f5e37a66640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e30147f60 con 0x7f5e301475b0 2024-03-06T07:29:33.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.637+0000 7f5e35802640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e300a9120 0x7f5e300c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.638+0000 7f5e35802640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e300a9120 0x7f5e300c5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59430/0 (socket says 172.21.15.1:59430) 2024-03-06T07:29:33.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.638+0000 7f5e35802640 1 -- 172.21.15.1:0/829435491 learned_addr learned my addr 172.21.15.1:0/829435491 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:33.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.638+0000 7f5e35001640 1 --2- 172.21.15.1:0/829435491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e300c3450 0x7f5e30142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.638+0000 7f5e36003640 1 --2- 172.21.15.1:0/829435491 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e301475b0 0x7f5e30142f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.638+0000 7f5e35802640 1 -- 172.21.15.1:0/829435491 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e301475b0 msgr2=0x7f5e30142f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.638+0000 7f5e35802640 1 --2- 172.21.15.1:0/829435491 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e301475b0 0x7f5e30142f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.638+0000 7f5e35802640 1 -- 172.21.15.1:0/829435491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e300c3450 msgr2=0x7f5e30142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.638+0000 7f5e35802640 1 --2- 172.21.15.1:0/829435491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e300c3450 0x7f5e30142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.638+0000 7f5e35802640 1 -- 172.21.15.1:0/829435491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e301eabc0 con 0x7f5e300a9120 2024-03-06T07:29:33.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.638+0000 7f5e35802640 1 --2- 172.21.15.1:0/829435491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e300a9120 0x7f5e300c5bb0 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800c890 tx=0x7f5e2800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.638+0000 7f5e2e7fc640 1 -- 172.21.15.1:0/829435491 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28018070 con 0x7f5e300a9120 2024-03-06T07:29:33.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.638+0000 7f5e2e7fc640 1 -- 172.21.15.1:0/829435491 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2800a5e0 con 0x7f5e300a9120 2024-03-06T07:29:33.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.638+0000 7f5e37a66640 1 -- 172.21.15.1:0/829435491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e301dbe50 con 0x7f5e300a9120 2024-03-06T07:29:33.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.638+0000 7f5e2e7fc640 1 -- 172.21.15.1:0/829435491 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2800a9d0 con 0x7f5e300a9120 2024-03-06T07:29:33.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.638+0000 7f5e37a66640 1 -- 172.21.15.1:0/829435491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e300aba40 con 0x7f5e300a9120 2024-03-06T07:29:33.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.639+0000 7f5e37a66640 1 -- 172.21.15.1:0/829435491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e30067fd0 con 0x7f5e300a9120 2024-03-06T07:29:33.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.640+0000 7f5e2e7fc640 1 -- 172.21.15.1:0/829435491 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e2800ab70 con 0x7f5e300a9120 2024-03-06T07:29:33.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.640+0000 7f5e2e7fc640 1 --2- 172.21.15.1:0/829435491 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e1c06b4d0 0x7f5e1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.640+0000 7f5e2e7fc640 1 -- 172.21.15.1:0/829435491 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5e280899b0 con 0x7f5e300a9120 2024-03-06T07:29:33.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.640+0000 7f5e35001640 1 --2- 172.21.15.1:0/829435491 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e1c06b4d0 0x7f5e1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.641+0000 7f5e35001640 1 --2- 172.21.15.1:0/829435491 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e1c06b4d0 0x7f5e1c06d990 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f5e24037000 tx=0x7f5e24031000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.645+0000 7f5e2e7fc640 1 -- 172.21.15.1:0/829435491 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e301eabc0 con 0x7f5e300a9120 2024-03-06T07:29:33.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.826+0000 7f5e37a66640 1 -- 172.21.15.1:0/829435491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e300ac6a0 con 0x7f5e300a9120 2024-03-06T07:29:33.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.827+0000 7f5e2e7fc640 1 -- 172.21.15.1:0/829435491 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+16781 (secure 0 0 0) 0x7f5e28057fd0 con 0x7f5e300a9120 2024-03-06T07:29:33.829 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:33.829 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":209,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:29:32.853082+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:31.114390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:31.815153+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:33.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.829+0000 7f5e37a66640 1 -- 172.21.15.1:0/829435491 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e1c06b4d0 msgr2=0x7f5e1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.829+0000 7f5e37a66640 1 --2- 172.21.15.1:0/829435491 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e1c06b4d0 0x7f5e1c06d990 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f5e24037000 tx=0x7f5e24031000 comp rx=0 tx=0).stop 2024-03-06T07:29:33.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.829+0000 7f5e37a66640 1 -- 172.21.15.1:0/829435491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e300a9120 msgr2=0x7f5e300c5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.829+0000 7f5e37a66640 1 --2- 172.21.15.1:0/829435491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e300a9120 0x7f5e300c5bb0 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800c890 tx=0x7f5e2800cd50 comp rx=0 tx=0).stop 2024-03-06T07:29:33.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.829+0000 7f5e37a66640 1 -- 172.21.15.1:0/829435491 shutdown_connections 2024-03-06T07:29:33.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.829+0000 7f5e37a66640 1 --2- 172.21.15.1:0/829435491 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e1c06b4d0 0x7f5e1c06d990 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.829+0000 7f5e37a66640 1 --2- 172.21.15.1:0/829435491 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e301475b0 0x7f5e30142f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.829+0000 7f5e37a66640 1 --2- 172.21.15.1:0/829435491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e300c3450 0x7f5e30142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.829+0000 7f5e37a66640 1 --2- 172.21.15.1:0/829435491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e300a9120 0x7f5e300c5bb0 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.829+0000 7f5e37a66640 1 -- 172.21.15.1:0/829435491 >> 172.21.15.1:0/829435491 conn(0x7f5e300c3cc0 msgr2=0x7f5e300c1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.829+0000 7f5e37a66640 1 -- 172.21.15.1:0/829435491 shutdown_connections 2024-03-06T07:29:33.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.829+0000 7f5e37a66640 1 -- 172.21.15.1:0/829435491 wait complete. 2024-03-06T07:29:33.842 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.941+0000 7f820606f640 1 Processor -- start 2024-03-06T07:29:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.941+0000 7f820606f640 1 -- start start 2024-03-06T07:29:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.942+0000 7f820606f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82000c8cd0 0x7f82000c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.942+0000 7f820606f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82000c3260 0x7f8200155b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.942+0000 7f820606f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82000c2b90 con 0x7f82000c7560 2024-03-06T07:29:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.942+0000 7f820606f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82000c37b0 con 0x7f82000c3260 2024-03-06T07:29:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.942+0000 7f820606f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82000c3a90 con 0x7f82000c8cd0 2024-03-06T07:29:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.942+0000 7f81feffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82000c3260 0x7f8200155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.942+0000 7f81fffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f82000c7560 0x7f8200159010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60176/0 (socket says 172.21.15.1:60176) 2024-03-06T07:29:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.942+0000 7f81fffff640 1 -- 172.21.15.1:0/4224083554 learned_addr learned my addr 172.21.15.1:0/4224083554 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.942+0000 7f81ff7fe640 1 --2- 172.21.15.1:0/4224083554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82000c8cd0 0x7f82000c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.943+0000 7f81feffd640 1 -- 172.21.15.1:0/4224083554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82000c8cd0 msgr2=0x7f82000c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.943+0000 7f81feffd640 1 --2- 172.21.15.1:0/4224083554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82000c8cd0 0x7f82000c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.943+0000 7f81feffd640 1 -- 172.21.15.1:0/4224083554 >> v1:172.21.15.1:6789/0 conn(0x7f82000c7560 legacy=0x7f8200159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.943+0000 7f81feffd640 1 -- 172.21.15.1:0/4224083554 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8200158810 con 0x7f82000c3260 2024-03-06T07:29:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.943+0000 7f81fe7fc640 1 -- 172.21.15.1:0/4224083554 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212362 0 0) 0x7f82000c2b90 con 0x7f82000c7560 2024-03-06T07:29:33.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.943+0000 7f81ff7fe640 1 --2- 172.21.15.1:0/4224083554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82000c8cd0 0x7f82000c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:33.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.943+0000 7f81feffd640 1 --2- 172.21.15.1:0/4224083554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82000c3260 0x7f8200155b70 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f81f40097a0 tx=0x7f81f402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7e6b38ad6d405 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.944+0000 7f81fe7fc640 1 -- 172.21.15.1:0/4224083554 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f40315a0 con 0x7f82000c3260 2024-03-06T07:29:33.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.944+0000 7f81fe7fc640 1 -- 172.21.15.1:0/4224083554 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f4037070 con 0x7f82000c3260 2024-03-06T07:29:33.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.944+0000 7f81fe7fc640 1 -- 172.21.15.1:0/4224083554 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f4031960 con 0x7f82000c3260 2024-03-06T07:29:33.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.944+0000 7f820606f640 1 -- 172.21.15.1:0/4224083554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82000c3260 msgr2=0x7f8200155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.944+0000 7f820606f640 1 --2- 172.21.15.1:0/4224083554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82000c3260 0x7f8200155b70 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f81f40097a0 tx=0x7f81f402b840 comp rx=0 tx=0).stop 2024-03-06T07:29:33.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.944+0000 7f820606f640 1 -- 172.21.15.1:0/4224083554 shutdown_connections 2024-03-06T07:29:33.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.944+0000 7f820606f640 1 --2- 172.21.15.1:0/4224083554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82000c3260 0x7f8200155b70 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.944+0000 7f820606f640 1 --2- 172.21.15.1:0/4224083554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82000c8cd0 0x7f82000c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.944+0000 7f820606f640 1 -- 172.21.15.1:0/4224083554 >> 172.21.15.1:0/4224083554 conn(0x7f820006b3e0 msgr2=0x7f820006b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.944+0000 7f820606f640 1 -- 172.21.15.1:0/4224083554 shutdown_connections 2024-03-06T07:29:33.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.944+0000 7f820606f640 1 -- 172.21.15.1:0/4224083554 wait complete. 2024-03-06T07:29:33.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.945+0000 7f820606f640 1 Processor -- start 2024-03-06T07:29:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.945+0000 7f820606f640 1 -- start start 2024-03-06T07:29:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.946+0000 7f820606f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82000c3260 0x7f820006d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.946+0000 7f820606f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82000c7560 0x7f82000a9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.946+0000 7f820606f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82000c8cd0 0x7f820014e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.946+0000 7f820606f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82000c8560 con 0x7f82000c7560 2024-03-06T07:29:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.946+0000 7f820606f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82000c7a10 con 0x7f82000c8cd0 2024-03-06T07:29:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.946+0000 7f820606f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82000c86e0 con 0x7f82000c3260 2024-03-06T07:29:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.946+0000 7f81feffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82000c7560 0x7f82000a9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.946+0000 7f81feffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82000c7560 0x7f82000a9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59454/0 (socket says 172.21.15.1:59454) 2024-03-06T07:29:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.946+0000 7f81feffd640 1 -- 172.21.15.1:0/3670534633 learned_addr learned my addr 172.21.15.1:0/3670534633 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.946+0000 7f81feffd640 1 -- 172.21.15.1:0/3670534633 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82000c3260 msgr2=0x7f820006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.946+0000 7f81feffd640 1 --2- 172.21.15.1:0/3670534633 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82000c3260 0x7f820006d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.946+0000 7f81feffd640 1 -- 172.21.15.1:0/3670534633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82000c8cd0 msgr2=0x7f820014e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.946+0000 7f81feffd640 1 --2- 172.21.15.1:0/3670534633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82000c8cd0 0x7f820014e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.946+0000 7f81feffd640 1 -- 172.21.15.1:0/3670534633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82001eb510 con 0x7f82000c7560 2024-03-06T07:29:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.947+0000 7f81feffd640 1 --2- 172.21.15.1:0/3670534633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82000c7560 0x7f82000a9c00 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f81f40098c0 tx=0x7f81f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.947+0000 7f81e7fff640 1 -- 172.21.15.1:0/3670534633 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f4032840 con 0x7f82000c7560 2024-03-06T07:29:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.947+0000 7f81e7fff640 1 -- 172.21.15.1:0/3670534633 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f4032a00 con 0x7f82000c7560 2024-03-06T07:29:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.947+0000 7f820606f640 1 -- 172.21.15.1:0/3670534633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82001eb7f0 con 0x7f82000c7560 2024-03-06T07:29:33.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.947+0000 7f81e7fff640 1 -- 172.21.15.1:0/3670534633 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f4032df0 con 0x7f82000c7560 2024-03-06T07:29:33.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.947+0000 7f820606f640 1 -- 172.21.15.1:0/3670534633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820006da30 con 0x7f82000c7560 2024-03-06T07:29:33.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.948+0000 7f820606f640 1 -- 172.21.15.1:0/3670534633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8200067fd0 con 0x7f82000c7560 2024-03-06T07:29:33.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.948+0000 7f81e7fff640 1 -- 172.21.15.1:0/3670534633 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81f404e430 con 0x7f82000c7560 2024-03-06T07:29:33.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.949+0000 7f81e7fff640 1 --2- 172.21.15.1:0/3670534633 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81ec06b4d0 0x7f81ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.949+0000 7f81e7fff640 1 -- 172.21.15.1:0/3670534633 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f81f402f030 con 0x7f82000c7560 2024-03-06T07:29:33.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.949+0000 7f81ff7fe640 1 --2- 172.21.15.1:0/3670534633 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81ec06b4d0 0x7f81ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.949+0000 7f81ff7fe640 1 --2- 172.21.15.1:0/3670534633 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81ec06b4d0 0x7f81ec06d990 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f81f0006fd0 tx=0x7f81f0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:33.953+0000 7f81e7fff640 1 -- 172.21.15.1:0/3670534633 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81f4040070 con 0x7f82000c7560 2024-03-06T07:29:34.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.135+0000 7f820606f640 1 -- 172.21.15.1:0/3670534633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82000c2d80 con 0x7f82000c7560 2024-03-06T07:29:34.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.136+0000 7f81e7fff640 1 -- 172.21.15.1:0/3670534633 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+16781 (secure 0 0 0) 0x7f81f407b000 con 0x7f82000c7560 2024-03-06T07:29:34.138 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:34.138 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":209,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:29:32.853082+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:31.114390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:31.815153+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:34.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.138+0000 7f820606f640 1 -- 172.21.15.1:0/3670534633 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81ec06b4d0 msgr2=0x7f81ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.138+0000 7f820606f640 1 --2- 172.21.15.1:0/3670534633 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81ec06b4d0 0x7f81ec06d990 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f81f0006fd0 tx=0x7f81f0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:34.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.138+0000 7f820606f640 1 -- 172.21.15.1:0/3670534633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82000c7560 msgr2=0x7f82000a9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.138+0000 7f820606f640 1 --2- 172.21.15.1:0/3670534633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82000c7560 0x7f82000a9c00 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f81f40098c0 tx=0x7f81f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.138+0000 7f820606f640 1 -- 172.21.15.1:0/3670534633 shutdown_connections 2024-03-06T07:29:34.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.138+0000 7f820606f640 1 --2- 172.21.15.1:0/3670534633 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81ec06b4d0 0x7f81ec06d990 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.138+0000 7f820606f640 1 --2- 172.21.15.1:0/3670534633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82000c8cd0 0x7f820014e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.138+0000 7f820606f640 1 --2- 172.21.15.1:0/3670534633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82000c7560 0x7f82000a9c00 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.138+0000 7f820606f640 1 --2- 172.21.15.1:0/3670534633 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82000c3260 0x7f820006d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.138+0000 7f820606f640 1 -- 172.21.15.1:0/3670534633 >> 172.21.15.1:0/3670534633 conn(0x7f820006b3e0 msgr2=0x7f8200156cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.138+0000 7f820606f640 1 -- 172.21.15.1:0/3670534633 shutdown_connections 2024-03-06T07:29:34.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.138+0000 7f820606f640 1 -- 172.21.15.1:0/3670534633 wait complete. 2024-03-06T07:29:34.151 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:29:34.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.253+0000 7fa82dfee640 1 Processor -- start 2024-03-06T07:29:34.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.253+0000 7fa82dfee640 1 -- start start 2024-03-06T07:29:34.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.254+0000 7fa82dfee640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa828156500 0x7fa828152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.254+0000 7fa82dfee640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa82806b510 0x7fa82806b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.254+0000 7fa82dfee640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8281569e0 con 0x7fa82814cdc0 2024-03-06T07:29:34.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.254+0000 7fa82dfee640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8280c2dd0 con 0x7fa82806b510 2024-03-06T07:29:34.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.254+0000 7fa82dfee640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa828156cc0 con 0x7fa828156500 2024-03-06T07:29:34.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.254+0000 7fa827fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa82806b510 0x7fa82806b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.254+0000 7fa8277fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa82814cdc0 0x7fa8280c2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60184/0 (socket says 172.21.15.1:60184) 2024-03-06T07:29:34.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.254+0000 7fa8277fe640 1 -- 172.21.15.1:0/440049488 learned_addr learned my addr 172.21.15.1:0/440049488 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:34.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.254+0000 7fa827fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa82806b510 0x7fa82806b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59456/0 (socket says 172.21.15.1:59456) 2024-03-06T07:29:34.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.254+0000 7fa826ffd640 1 --2- 172.21.15.1:0/440049488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa828156500 0x7fa828152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.254+0000 7fa827fff640 1 -- 172.21.15.1:0/440049488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa828156500 msgr2=0x7fa828152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:34.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.254+0000 7fa827fff640 1 --2- 172.21.15.1:0/440049488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa828156500 0x7fa828152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.254+0000 7fa827fff640 1 -- 172.21.15.1:0/440049488 >> v1:172.21.15.1:6789/0 conn(0x7fa82814cdc0 legacy=0x7fa8280c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.254+0000 7fa827fff640 1 -- 172.21.15.1:0/440049488 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa828156fa0 con 0x7fa82806b510 2024-03-06T07:29:34.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.254+0000 7fa8267fc640 1 -- 172.21.15.1:0/440049488 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340042650 0 0) 0x7fa8281569e0 con 0x7fa82814cdc0 2024-03-06T07:29:34.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.255+0000 7fa827fff640 1 --2- 172.21.15.1:0/440049488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82806b510 0x7fa82806b980 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7fa820009d90 tx=0x7fa82002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55fc9f25a0779023 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.255+0000 7fa8267fc640 1 -- 172.21.15.1:0/440049488 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa820039070 con 0x7fa82806b510 2024-03-06T07:29:34.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.255+0000 7fa8267fc640 1 -- 172.21.15.1:0/440049488 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82002cc50 con 0x7fa82806b510 2024-03-06T07:29:34.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.255+0000 7fa8267fc640 1 -- 172.21.15.1:0/440049488 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa820034440 con 0x7fa82806b510 2024-03-06T07:29:34.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.255+0000 7fa82dfee640 1 -- 172.21.15.1:0/440049488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82806b510 msgr2=0x7fa82806b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.255+0000 7fa82dfee640 1 --2- 172.21.15.1:0/440049488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82806b510 0x7fa82806b980 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7fa820009d90 tx=0x7fa82002bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.256+0000 7fa82dfee640 1 -- 172.21.15.1:0/440049488 shutdown_connections 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.256+0000 7fa82dfee640 1 --2- 172.21.15.1:0/440049488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa828156500 0x7fa828152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.256+0000 7fa82dfee640 1 --2- 172.21.15.1:0/440049488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82806b510 0x7fa82806b980 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.256+0000 7fa82dfee640 1 -- 172.21.15.1:0/440049488 >> 172.21.15.1:0/440049488 conn(0x7fa8280a9700 msgr2=0x7fa8280a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.256+0000 7fa82dfee640 1 -- 172.21.15.1:0/440049488 shutdown_connections 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.256+0000 7fa82dfee640 1 -- 172.21.15.1:0/440049488 wait complete. 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.257+0000 7fa82dfee640 1 Processor -- start 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.257+0000 7fa82dfee640 1 -- start start 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.257+0000 7fa82dfee640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82806b510 0x7fa82814f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.257+0000 7fa82dfee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa82814cdc0 0x7fa82814b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.257+0000 7fa82dfee640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa828156500 0x7fa82814bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.257+0000 7fa82dfee640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8280c5a70 con 0x7fa82806b510 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.257+0000 7fa82dfee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82806bec0 con 0x7fa82814cdc0 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.257+0000 7fa82dfee640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8280c65e0 con 0x7fa828156500 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.258+0000 7fa8277fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82806b510 0x7fa82814f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.258+0000 7fa8277fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82806b510 0x7fa82814f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59466/0 (socket says 172.21.15.1:59466) 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.258+0000 7fa8277fe640 1 -- 172.21.15.1:0/3882424865 learned_addr learned my addr 172.21.15.1:0/3882424865 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.258+0000 7fa827fff640 1 --2- 172.21.15.1:0/3882424865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa828156500 0x7fa82814bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.258+0000 7fa826ffd640 1 --2- 172.21.15.1:0/3882424865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa82814cdc0 0x7fa82814b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.258+0000 7fa8277fe640 1 -- 172.21.15.1:0/3882424865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa828156500 msgr2=0x7fa82814bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.258+0000 7fa8277fe640 1 --2- 172.21.15.1:0/3882424865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa828156500 0x7fa82814bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.258+0000 7fa8277fe640 1 -- 172.21.15.1:0/3882424865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa82814cdc0 msgr2=0x7fa82814b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.258+0000 7fa8277fe640 1 --2- 172.21.15.1:0/3882424865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa82814cdc0 0x7fa82814b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.258+0000 7fa8277fe640 1 -- 172.21.15.1:0/3882424865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8280ad6f0 con 0x7fa82806b510 2024-03-06T07:29:34.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.258+0000 7fa8277fe640 1 --2- 172.21.15.1:0/3882424865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82806b510 0x7fa82814f550 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7fa81800c4d0 tx=0x7fa81800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.258+0000 7fa80ffff640 1 -- 172.21.15.1:0/3882424865 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818018070 con 0x7fa82806b510 2024-03-06T07:29:34.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.258+0000 7fa80ffff640 1 -- 172.21.15.1:0/3882424865 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa818007500 con 0x7fa82806b510 2024-03-06T07:29:34.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.258+0000 7fa82dfee640 1 -- 172.21.15.1:0/3882424865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8281e98c0 con 0x7fa82806b510 2024-03-06T07:29:34.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.258+0000 7fa80ffff640 1 -- 172.21.15.1:0/3882424865 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818004a20 con 0x7fa82806b510 2024-03-06T07:29:34.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.258+0000 7fa82dfee640 1 -- 172.21.15.1:0/3882424865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8281e9ba0 con 0x7fa82806b510 2024-03-06T07:29:34.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.259+0000 7fa82dfee640 1 -- 172.21.15.1:0/3882424865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa828067fd0 con 0x7fa82806b510 2024-03-06T07:29:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.260+0000 7fa80ffff640 1 -- 172.21.15.1:0/3882424865 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa81801b070 con 0x7fa82806b510 2024-03-06T07:29:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.260+0000 7fa80ffff640 1 --2- 172.21.15.1:0/3882424865 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa81406b4d0 0x7fa81406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.260+0000 7fa80ffff640 1 -- 172.21.15.1:0/3882424865 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa818089550 con 0x7fa82806b510 2024-03-06T07:29:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.260+0000 7fa826ffd640 1 --2- 172.21.15.1:0/3882424865 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa81406b4d0 0x7fa81406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.261+0000 7fa826ffd640 1 --2- 172.21.15.1:0/3882424865 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa81406b4d0 0x7fa81406d990 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fa8280c4900 tx=0x7fa81c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.264+0000 7fa80ffff640 1 -- 172.21.15.1:0/3882424865 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa818057c80 con 0x7fa82806b510 2024-03-06T07:29:34.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:34.484+0000 7fa82dfee640 1 -- 172.21.15.1:0/3882424865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa828153e00 con 0x7fa82806b510 2024-03-06T07:29:35.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.307+0000 7fa80ffff640 1 -- 172.21.15.1:0/3882424865 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v195) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa81805bd20 con 0x7fa82806b510 2024-03-06T07:29:35.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.309+0000 7fa82dfee640 1 -- 172.21.15.1:0/3882424865 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa81406b4d0 msgr2=0x7fa81406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.309+0000 7fa82dfee640 1 --2- 172.21.15.1:0/3882424865 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa81406b4d0 0x7fa81406d990 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fa8280c4900 tx=0x7fa81c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:35.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.309+0000 7fa82dfee640 1 -- 172.21.15.1:0/3882424865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82806b510 msgr2=0x7fa82814f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.309+0000 7fa82dfee640 1 --2- 172.21.15.1:0/3882424865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82806b510 0x7fa82814f550 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7fa81800c4d0 tx=0x7fa81800c990 comp rx=0 tx=0).stop 2024-03-06T07:29:35.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.309+0000 7fa82dfee640 1 -- 172.21.15.1:0/3882424865 shutdown_connections 2024-03-06T07:29:35.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.309+0000 7fa82dfee640 1 --2- 172.21.15.1:0/3882424865 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa81406b4d0 0x7fa81406d990 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.309+0000 7fa82dfee640 1 --2- 172.21.15.1:0/3882424865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa828156500 0x7fa82814bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.309+0000 7fa82dfee640 1 --2- 172.21.15.1:0/3882424865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa82814cdc0 0x7fa82814b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.309+0000 7fa82dfee640 1 --2- 172.21.15.1:0/3882424865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82806b510 0x7fa82814f550 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.309+0000 7fa82dfee640 1 -- 172.21.15.1:0/3882424865 >> 172.21.15.1:0/3882424865 conn(0x7fa8280a9700 msgr2=0x7fa828154300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.309+0000 7fa82dfee640 1 -- 172.21.15.1:0/3882424865 shutdown_connections 2024-03-06T07:29:35.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.309+0000 7fa82dfee640 1 -- 172.21.15.1:0/3882424865 wait complete. 2024-03-06T07:29:35.322 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:29:35.322 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:29:35.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.424+0000 7f30843f7640 1 Processor -- start 2024-03-06T07:29:35.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.424+0000 7f30843f7640 1 -- start start 2024-03-06T07:29:35.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.425+0000 7f30843f7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f307c150520 0x7f307c1543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.425+0000 7f30843f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f307c070c80 0x7f307c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.425+0000 7f30843f7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307c150900 con 0x7f307c0c32f0 2024-03-06T07:29:35.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.425+0000 7f30843f7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f307c153ed0 con 0x7f307c150520 2024-03-06T07:29:35.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.425+0000 7f30843f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307c071790 con 0x7f307c070c80 2024-03-06T07:29:35.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.425+0000 7f3081992640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f307c0c32f0 0x7f307c1537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35874/0 (socket says 172.21.15.1:35874) 2024-03-06T07:29:35.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.425+0000 7f3081992640 1 -- 172.21.15.1:0/4272942068 learned_addr learned my addr 172.21.15.1:0/4272942068 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:35.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.425+0000 7f3081191640 1 -- 172.21.15.1:0/4272942068 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853721706 0 0) 0x7f307c150900 con 0x7f307c0c32f0 2024-03-06T07:29:35.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.425+0000 7f3082994640 1 --2- 172.21.15.1:0/4272942068 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f307c070c80 0x7f307c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.425+0000 7f3082193640 1 --2- 172.21.15.1:0/4272942068 >> v2:172.21.15.98:3300/0 conn(0x7f307c150520 0x7f307c1543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.425+0000 7f3081191640 1 -- 172.21.15.1:0/4272942068 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3068003520 con 0x7f307c0c32f0 2024-03-06T07:29:35.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3081191640 1 -- 172.21.15.1:0/4272942068 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 645104869 0 0) 0x7f3068003520 con 0x7f307c0c32f0 2024-03-06T07:29:35.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3081191640 1 -- 172.21.15.1:0/4272942068 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f307c070c80 msgr2=0x7f307c0710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3081191640 1 --2- 172.21.15.1:0/4272942068 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f307c070c80 0x7f307c0710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3081191640 1 -- 172.21.15.1:0/4272942068 >> v2:172.21.15.98:3300/0 conn(0x7f307c150520 msgr2=0x7f307c1543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3081191640 1 --2- 172.21.15.1:0/4272942068 >> v2:172.21.15.98:3300/0 conn(0x7f307c150520 0x7f307c1543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3081191640 1 -- 172.21.15.1:0/4272942068 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c157050 con 0x7f307c0c32f0 2024-03-06T07:29:35.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3081191640 1 -- 172.21.15.1:0/4272942068 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f3078001670 con 0x7f307c0c32f0 2024-03-06T07:29:35.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3081191640 1 -- 172.21.15.1:0/4272942068 >> v1:172.21.15.1:6789/0 conn(0x7f307c0c32f0 legacy=0x7f307c1537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3081191640 1 --2- 172.21.15.1:0/4272942068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30680039b0 0x7f3068003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3081191640 1 --2- 172.21.15.1:0/4272942068 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f307c070c80 0x7f3068004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3081191640 1 --2- 172.21.15.1:0/4272942068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3068004d40 0x7f30680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3081191640 1 -- 172.21.15.1:0/4272942068 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c153ed0 con 0x7f3068004d40 2024-03-06T07:29:35.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3081191640 1 -- 172.21.15.1:0/4272942068 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c071790 con 0x7f30680039b0 2024-03-06T07:29:35.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3081191640 1 -- 172.21.15.1:0/4272942068 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306800b9c0 con 0x7f307c070c80 2024-03-06T07:29:35.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3082994640 1 --2- 172.21.15.1:0/4272942068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3068004d40 0x7f30680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3081992640 1 --2- 172.21.15.1:0/4272942068 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f307c070c80 0x7f3068004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3082193640 1 --2- 172.21.15.1:0/4272942068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30680039b0 0x7f3068003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3082994640 1 -- 172.21.15.1:0/4272942068 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f307c070c80 msgr2=0x7f3068004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3082994640 1 --2- 172.21.15.1:0/4272942068 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f307c070c80 0x7f3068004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3082994640 1 -- 172.21.15.1:0/4272942068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30680039b0 msgr2=0x7f3068003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3082994640 1 --2- 172.21.15.1:0/4272942068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30680039b0 0x7f3068003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.426+0000 7f3082994640 1 -- 172.21.15.1:0/4272942068 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3068003520 con 0x7f3068004d40 2024-03-06T07:29:35.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.427+0000 7f3082994640 1 --2- 172.21.15.1:0/4272942068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3068004d40 0x7f30680051c0 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7f307400e170 tx=0x7f3068005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d729f51eaaacca0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.427+0000 7f3081191640 1 -- 172.21.15.1:0/4272942068 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3074039070 con 0x7f3068004d40 2024-03-06T07:29:35.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.427+0000 7f3081191640 1 -- 172.21.15.1:0/4272942068 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307402cbf0 con 0x7f3068004d40 2024-03-06T07:29:35.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.427+0000 7f3081191640 1 -- 172.21.15.1:0/4272942068 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3074009c10 con 0x7f3068004d40 2024-03-06T07:29:35.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.428+0000 7f30843f7640 1 -- 172.21.15.1:0/4272942068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3068004d40 msgr2=0x7f30680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.428+0000 7f30843f7640 1 --2- 172.21.15.1:0/4272942068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3068004d40 0x7f30680051c0 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7f307400e170 tx=0x7f3068005830 comp rx=0 tx=0).stop 2024-03-06T07:29:35.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.428+0000 7f3082994640 1 -- 172.21.15.1:0/4272942068 reap_dead start 2024-03-06T07:29:35.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.428+0000 7f30843f7640 1 -- 172.21.15.1:0/4272942068 shutdown_connections 2024-03-06T07:29:35.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.428+0000 7f30843f7640 1 -- 172.21.15.1:0/4272942068 >> 172.21.15.1:0/4272942068 conn(0x7f307c0c88f0 msgr2=0x7f307c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.428+0000 7f30843f7640 1 -- 172.21.15.1:0/4272942068 shutdown_connections 2024-03-06T07:29:35.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.428+0000 7f30843f7640 1 -- 172.21.15.1:0/4272942068 wait complete. 2024-03-06T07:29:35.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.429+0000 7f30843f7640 1 Processor -- start 2024-03-06T07:29:35.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.429+0000 7f30843f7640 1 -- start start 2024-03-06T07:29:35.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.429+0000 7f30843f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3068004d40 0x7f307c148670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.429+0000 7f30843f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f307c0acef0 0x7f307c1ec1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.429+0000 7f30843f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f307c1ec700 0x7f307c150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.429+0000 7f30843f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c157330 con 0x7f307c1ec700 2024-03-06T07:29:35.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f30843f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c1532a0 con 0x7f3068004d40 2024-03-06T07:29:35.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f30843f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307c0c7ec0 con 0x7f307c0acef0 2024-03-06T07:29:35.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f3082994640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f307c1ec700 0x7f307c150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f3082994640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f307c1ec700 0x7f307c150d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56022/0 (socket says 172.21.15.1:56022) 2024-03-06T07:29:35.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f3082994640 1 -- 172.21.15.1:0/2124785370 learned_addr learned my addr 172.21.15.1:0/2124785370 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:35.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f3082193640 1 --2- 172.21.15.1:0/2124785370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3068004d40 0x7f307c148670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f3081992640 1 --2- 172.21.15.1:0/2124785370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f307c0acef0 0x7f307c1ec1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f3082994640 1 -- 172.21.15.1:0/2124785370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f307c0acef0 msgr2=0x7f307c1ec1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f3082994640 1 --2- 172.21.15.1:0/2124785370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f307c0acef0 0x7f307c1ec1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f3082994640 1 -- 172.21.15.1:0/2124785370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3068004d40 msgr2=0x7f307c148670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f3082994640 1 --2- 172.21.15.1:0/2124785370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3068004d40 0x7f307c148670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f3082994640 1 -- 172.21.15.1:0/2124785370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c154b90 con 0x7f307c1ec700 2024-03-06T07:29:35.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f3082994640 1 --2- 172.21.15.1:0/2124785370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f307c1ec700 0x7f307c150d20 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f3074009f90 tx=0x7f3074009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f3072ffd640 1 -- 172.21.15.1:0/2124785370 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3074039030 con 0x7f307c1ec700 2024-03-06T07:29:35.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f3072ffd640 1 -- 172.21.15.1:0/2124785370 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307403a040 con 0x7f307c1ec700 2024-03-06T07:29:35.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f30843f7640 1 -- 172.21.15.1:0/2124785370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307c146530 con 0x7f307c1ec700 2024-03-06T07:29:35.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f3072ffd640 1 -- 172.21.15.1:0/2124785370 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3074009c10 con 0x7f307c1ec700 2024-03-06T07:29:35.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.430+0000 7f30843f7640 1 -- 172.21.15.1:0/2124785370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307c151dc0 con 0x7f307c1ec700 2024-03-06T07:29:35.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.431+0000 7f30843f7640 1 -- 172.21.15.1:0/2124785370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307c0686e0 con 0x7f307c1ec700 2024-03-06T07:29:35.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.432+0000 7f3072ffd640 1 -- 172.21.15.1:0/2124785370 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3074005d10 con 0x7f307c1ec700 2024-03-06T07:29:35.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.432+0000 7f3072ffd640 1 --2- 172.21.15.1:0/2124785370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f306006b4d0 0x7f306006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.432+0000 7f3072ffd640 1 -- 172.21.15.1:0/2124785370 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f30740abf00 con 0x7f307c1ec700 2024-03-06T07:29:35.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.432+0000 7f3082193640 1 --2- 172.21.15.1:0/2124785370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f306006b4d0 0x7f306006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.433+0000 7f3082193640 1 --2- 172.21.15.1:0/2124785370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f306006b4d0 0x7f306006d990 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f307c148fa0 tx=0x7f306c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.441+0000 7f3072ffd640 1 -- 172.21.15.1:0/2124785370 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f307407acb0 con 0x7f307c1ec700 2024-03-06T07:29:35.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:35.662+0000 7f30843f7640 1 -- 172.21.15.1:0/2124785370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f307c1ebb00 con 0x7f307c1ec700 2024-03-06T07:29:36.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.508+0000 7f3072ffd640 1 -- 172.21.15.1:0/2124785370 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v196) v1 ==== 122+0+0 (secure 0 0 0) 0x7f307407a5b0 con 0x7f307c1ec700 2024-03-06T07:29:36.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.510+0000 7f30843f7640 1 -- 172.21.15.1:0/2124785370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f306006b4d0 msgr2=0x7f306006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.510+0000 7f30843f7640 1 --2- 172.21.15.1:0/2124785370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f306006b4d0 0x7f306006d990 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f307c148fa0 tx=0x7f306c009190 comp rx=0 tx=0).stop 2024-03-06T07:29:36.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.510+0000 7f30843f7640 1 -- 172.21.15.1:0/2124785370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f307c1ec700 msgr2=0x7f307c150d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.510+0000 7f30843f7640 1 --2- 172.21.15.1:0/2124785370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f307c1ec700 0x7f307c150d20 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f3074009f90 tx=0x7f3074009750 comp rx=0 tx=0).stop 2024-03-06T07:29:36.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.510+0000 7f30843f7640 1 -- 172.21.15.1:0/2124785370 shutdown_connections 2024-03-06T07:29:36.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.510+0000 7f30843f7640 1 --2- 172.21.15.1:0/2124785370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f306006b4d0 0x7f306006d990 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.510+0000 7f30843f7640 1 --2- 172.21.15.1:0/2124785370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f307c1ec700 0x7f307c150d20 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.510+0000 7f30843f7640 1 --2- 172.21.15.1:0/2124785370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f307c0acef0 0x7f307c1ec1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.510+0000 7f30843f7640 1 --2- 172.21.15.1:0/2124785370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3068004d40 0x7f307c148670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.510+0000 7f30843f7640 1 -- 172.21.15.1:0/2124785370 >> 172.21.15.1:0/2124785370 conn(0x7f307c0c88f0 msgr2=0x7f307c071060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.510+0000 7f30843f7640 1 -- 172.21.15.1:0/2124785370 shutdown_connections 2024-03-06T07:29:36.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.510+0000 7f30843f7640 1 -- 172.21.15.1:0/2124785370 wait complete. 2024-03-06T07:29:36.523 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:29:36.523 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:36.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.622+0000 7f529737a640 1 Processor -- start 2024-03-06T07:29:36.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.623+0000 7f529737a640 1 -- start start 2024-03-06T07:29:36.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.623+0000 7f529737a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5290152650 0x7f5290155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.623+0000 7f529737a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f52900c6b10 0x7f5290158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.623+0000 7f529737a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52900bf9d0 0x7f52900c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.623+0000 7f529737a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5290155800 con 0x7f5290152650 2024-03-06T07:29:36.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.623+0000 7f529737a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52900c2fc0 con 0x7f52900c6b10 2024-03-06T07:29:36.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.623+0000 7f529737a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52900c6690 con 0x7f52900bf9d0 2024-03-06T07:29:36.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.623+0000 7f5295116640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5290152650 0x7f5290155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.623+0000 7f5295116640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5290152650 0x7f5290155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56030/0 (socket says 172.21.15.1:56030) 2024-03-06T07:29:36.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.624+0000 7f5295116640 1 -- 172.21.15.1:0/391032156 learned_addr learned my addr 172.21.15.1:0/391032156 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:36.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.624+0000 7f5295116640 1 -- 172.21.15.1:0/391032156 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52900bf9d0 msgr2=0x7f52900c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.624+0000 7f5295116640 1 --2- 172.21.15.1:0/391032156 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52900bf9d0 0x7f52900c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.624+0000 7f5295116640 1 -- 172.21.15.1:0/391032156 >> v2:172.21.15.98:3300/0 conn(0x7f52900c6b10 msgr2=0x7f5290158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.624+0000 7f5294915640 1 --2- 172.21.15.1:0/391032156 >> v2:172.21.15.98:3300/0 conn(0x7f52900c6b10 0x7f5290158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.624+0000 7f5295116640 1 --2- 172.21.15.1:0/391032156 >> v2:172.21.15.98:3300/0 conn(0x7f52900c6b10 0x7f5290158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.624+0000 7f5295116640 1 -- 172.21.15.1:0/391032156 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52900c7280 con 0x7f5290152650 2024-03-06T07:29:36.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.624+0000 7f5295116640 1 --2- 172.21.15.1:0/391032156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290152650 0x7f5290155c90 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f52900cb8e0 tx=0x7f528002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13572da2b38def9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.625+0000 7f528ffff640 1 -- 172.21.15.1:0/391032156 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528002e070 con 0x7f5290152650 2024-03-06T07:29:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.625+0000 7f528ffff640 1 -- 172.21.15.1:0/391032156 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5280037070 con 0x7f5290152650 2024-03-06T07:29:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.625+0000 7f528ffff640 1 -- 172.21.15.1:0/391032156 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5280031a00 con 0x7f5290152650 2024-03-06T07:29:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.625+0000 7f529737a640 1 -- 172.21.15.1:0/391032156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290152650 msgr2=0x7f5290155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.625+0000 7f529737a640 1 --2- 172.21.15.1:0/391032156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290152650 0x7f5290155c90 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f52900cb8e0 tx=0x7f528002b880 comp rx=0 tx=0).stop 2024-03-06T07:29:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.625+0000 7f529737a640 1 -- 172.21.15.1:0/391032156 shutdown_connections 2024-03-06T07:29:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.625+0000 7f529737a640 1 --2- 172.21.15.1:0/391032156 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52900bf9d0 0x7f52900c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.625+0000 7f529737a640 1 --2- 172.21.15.1:0/391032156 >> v2:172.21.15.98:3300/0 conn(0x7f52900c6b10 0x7f5290158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.625+0000 7f529737a640 1 --2- 172.21.15.1:0/391032156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290152650 0x7f5290155c90 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.625+0000 7f529737a640 1 -- 172.21.15.1:0/391032156 >> 172.21.15.1:0/391032156 conn(0x7f5290066ee0 msgr2=0x7f52900672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.625+0000 7f529737a640 1 -- 172.21.15.1:0/391032156 shutdown_connections 2024-03-06T07:29:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.625+0000 7f529737a640 1 -- 172.21.15.1:0/391032156 wait complete. 2024-03-06T07:29:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.626+0000 7f529737a640 1 Processor -- start 2024-03-06T07:29:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.626+0000 7f529737a640 1 -- start start 2024-03-06T07:29:36.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.627+0000 7f529737a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52900bf9d0 0x7f5290144ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.627+0000 7f529737a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290152650 0x7f52900ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.627+0000 7f529737a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5290153ce0 0x7f52900c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.627+0000 7f529737a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5290155490 con 0x7f5290152650 2024-03-06T07:29:36.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.627+0000 7f529737a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52901551b0 con 0x7f5290153ce0 2024-03-06T07:29:36.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.627+0000 7f529737a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5290152a30 con 0x7f52900bf9d0 2024-03-06T07:29:36.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.627+0000 7f5294915640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290152650 0x7f52900ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.627+0000 7f5294915640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290152650 0x7f52900ca3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56046/0 (socket says 172.21.15.1:56046) 2024-03-06T07:29:36.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.627+0000 7f5294915640 1 -- 172.21.15.1:0/2780649808 learned_addr learned my addr 172.21.15.1:0/2780649808 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:36.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.627+0000 7f5294915640 1 -- 172.21.15.1:0/2780649808 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52900bf9d0 msgr2=0x7f5290144ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.627+0000 7f5295116640 1 --2- 172.21.15.1:0/2780649808 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52900bf9d0 0x7f5290144ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.627+0000 7f5294915640 1 --2- 172.21.15.1:0/2780649808 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52900bf9d0 0x7f5290144ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.627+0000 7f5295917640 1 --2- 172.21.15.1:0/2780649808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5290153ce0 0x7f52900c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.627+0000 7f5294915640 1 -- 172.21.15.1:0/2780649808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5290153ce0 msgr2=0x7f52900c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.628+0000 7f5294915640 1 --2- 172.21.15.1:0/2780649808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5290153ce0 0x7f52900c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.628+0000 7f5294915640 1 -- 172.21.15.1:0/2780649808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529014bcd0 con 0x7f5290152650 2024-03-06T07:29:36.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.628+0000 7f5294915640 1 --2- 172.21.15.1:0/2780649808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290152650 0x7f52900ca3c0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f528400c7f0 tx=0x7f528400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.628+0000 7f528dffb640 1 -- 172.21.15.1:0/2780649808 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5284015070 con 0x7f5290152650 2024-03-06T07:29:36.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.628+0000 7f528dffb640 1 -- 172.21.15.1:0/2780649808 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5284004240 con 0x7f5290152650 2024-03-06T07:29:36.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.628+0000 7f529737a640 1 -- 172.21.15.1:0/2780649808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52900c9e00 con 0x7f5290152650 2024-03-06T07:29:36.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.628+0000 7f528dffb640 1 -- 172.21.15.1:0/2780649808 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5284007620 con 0x7f5290152650 2024-03-06T07:29:36.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.628+0000 7f529737a640 1 -- 172.21.15.1:0/2780649808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52901ef440 con 0x7f5290152650 2024-03-06T07:29:36.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.629+0000 7f528dffb640 1 -- 172.21.15.1:0/2780649808 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52840079d0 con 0x7f5290152650 2024-03-06T07:29:36.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.629+0000 7f529737a640 1 -- 172.21.15.1:0/2780649808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5290067fd0 con 0x7f5290152650 2024-03-06T07:29:36.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.630+0000 7f528dffb640 1 --2- 172.21.15.1:0/2780649808 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f527c06b4d0 0x7f527c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.630+0000 7f528dffb640 1 -- 172.21.15.1:0/2780649808 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f528408a4a0 con 0x7f5290152650 2024-03-06T07:29:36.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.630+0000 7f5295116640 1 --2- 172.21.15.1:0/2780649808 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f527c06b4d0 0x7f527c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.630+0000 7f5295116640 1 --2- 172.21.15.1:0/2780649808 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f527c06b4d0 0x7f527c06d990 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f5280036000 tx=0x7f5280008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.634+0000 7f528dffb640 1 -- 172.21.15.1:0/2780649808 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5284059180 con 0x7f5290152650 2024-03-06T07:29:36.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.854+0000 7f529737a640 1 -- 172.21.15.1:0/2780649808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5290153700 con 0x7f5290152650 2024-03-06T07:29:36.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.854+0000 7f528dffb640 1 -- 172.21.15.1:0/2780649808 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+4987 (secure 0 0 0) 0x7f5284058a80 con 0x7f5290152650 2024-03-06T07:29:36.857 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:36.857 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":196,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":10803,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:35.664209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10806},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:29:36.857 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 196 2024-03-06T07:29:36.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.856+0000 7f529737a640 1 -- 172.21.15.1:0/2780649808 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f527c06b4d0 msgr2=0x7f527c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.856+0000 7f529737a640 1 --2- 172.21.15.1:0/2780649808 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f527c06b4d0 0x7f527c06d990 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f5280036000 tx=0x7f5280008030 comp rx=0 tx=0).stop 2024-03-06T07:29:36.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.856+0000 7f529737a640 1 -- 172.21.15.1:0/2780649808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290152650 msgr2=0x7f52900ca3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.856+0000 7f529737a640 1 --2- 172.21.15.1:0/2780649808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290152650 0x7f52900ca3c0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f528400c7f0 tx=0x7f528400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.856+0000 7f529737a640 1 -- 172.21.15.1:0/2780649808 shutdown_connections 2024-03-06T07:29:36.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.856+0000 7f529737a640 1 --2- 172.21.15.1:0/2780649808 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f527c06b4d0 0x7f527c06d990 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.856+0000 7f529737a640 1 --2- 172.21.15.1:0/2780649808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5290153ce0 0x7f52900c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.856+0000 7f529737a640 1 --2- 172.21.15.1:0/2780649808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290152650 0x7f52900ca3c0 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.856+0000 7f529737a640 1 --2- 172.21.15.1:0/2780649808 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f52900bf9d0 0x7f5290144ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.856+0000 7f529737a640 1 -- 172.21.15.1:0/2780649808 >> 172.21.15.1:0/2780649808 conn(0x7f5290066ee0 msgr2=0x7f52900c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.856+0000 7f529737a640 1 -- 172.21.15.1:0/2780649808 shutdown_connections 2024-03-06T07:29:36.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.856+0000 7f529737a640 1 -- 172.21.15.1:0/2780649808 wait complete. 2024-03-06T07:29:36.870 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:36.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.968+0000 7f6ecb4c2640 1 Processor -- start 2024-03-06T07:29:36.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.968+0000 7f6ecb4c2640 1 -- start start 2024-03-06T07:29:36.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.969+0000 7f6ecb4c2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6ec40c4bf0 0x7f6ec4156460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.969+0000 7f6ecb4c2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ec41569a0 0x7f6ec415d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.969+0000 7f6ecb4c2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ec4153a00 con 0x7f6ec40a7050 2024-03-06T07:29:36.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.969+0000 7f6ecb4c2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ec4153ce0 con 0x7f6ec40c4bf0 2024-03-06T07:29:36.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.969+0000 7f6ecb4c2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec40c3cb0 con 0x7f6ec41569a0 2024-03-06T07:29:36.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.969+0000 7f6ec925e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6ec40a7050 0x7f6ec40c44e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35876/0 (socket says 172.21.15.1:35876) 2024-03-06T07:29:36.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.969+0000 7f6ec925e640 1 -- 172.21.15.1:0/619499463 learned_addr learned my addr 172.21.15.1:0/619499463 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:36.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.969+0000 7f6ec9a5f640 1 --2- 172.21.15.1:0/619499463 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ec41569a0 0x7f6ec415d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.969+0000 7f6ebbfff640 1 -- 172.21.15.1:0/619499463 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255809834 0 0) 0x7f6ec4153a00 con 0x7f6ec40a7050 2024-03-06T07:29:36.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.969+0000 7f6ec8a5d640 1 --2- 172.21.15.1:0/619499463 >> v2:172.21.15.98:3300/0 conn(0x7f6ec40c4bf0 0x7f6ec4156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ebbfff640 1 -- 172.21.15.1:0/619499463 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eb0003520 con 0x7f6ec40a7050 2024-03-06T07:29:36.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ebbfff640 1 -- 172.21.15.1:0/619499463 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3525135556 0 0) 0x7f6eb0003520 con 0x7f6ec40a7050 2024-03-06T07:29:36.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ebbfff640 1 -- 172.21.15.1:0/619499463 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ec41569a0 msgr2=0x7f6ec415d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ebbfff640 1 --2- 172.21.15.1:0/619499463 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ec41569a0 0x7f6ec415d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ebbfff640 1 -- 172.21.15.1:0/619499463 >> v2:172.21.15.98:3300/0 conn(0x7f6ec40c4bf0 msgr2=0x7f6ec4156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ebbfff640 1 --2- 172.21.15.1:0/619499463 >> v2:172.21.15.98:3300/0 conn(0x7f6ec40c4bf0 0x7f6ec4156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ebbfff640 1 -- 172.21.15.1:0/619499463 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec40c3f90 con 0x7f6ec40a7050 2024-03-06T07:29:36.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ebbfff640 1 -- 172.21.15.1:0/619499463 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f6ec00019f0 con 0x7f6ec40a7050 2024-03-06T07:29:36.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ebbfff640 1 -- 172.21.15.1:0/619499463 >> v1:172.21.15.1:6789/0 conn(0x7f6ec40a7050 legacy=0x7f6ec40c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ebbfff640 1 --2- 172.21.15.1:0/619499463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb00039b0 0x7f6eb0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ebbfff640 1 --2- 172.21.15.1:0/619499463 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ec41569a0 0x7f6eb0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ebbfff640 1 --2- 172.21.15.1:0/619499463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6eb0004d40 0x7f6eb00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ebbfff640 1 -- 172.21.15.1:0/619499463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec4153ce0 con 0x7f6eb0004d40 2024-03-06T07:29:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ebbfff640 1 -- 172.21.15.1:0/619499463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec40c3cb0 con 0x7f6eb00039b0 2024-03-06T07:29:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ebbfff640 1 -- 172.21.15.1:0/619499463 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb000ba40 con 0x7f6ec41569a0 2024-03-06T07:29:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ec9a5f640 1 --2- 172.21.15.1:0/619499463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6eb0004d40 0x7f6eb00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ec925e640 1 --2- 172.21.15.1:0/619499463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb00039b0 0x7f6eb0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ec8a5d640 1 --2- 172.21.15.1:0/619499463 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ec41569a0 0x7f6eb0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ec9a5f640 1 -- 172.21.15.1:0/619499463 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ec41569a0 msgr2=0x7f6eb0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ec9a5f640 1 --2- 172.21.15.1:0/619499463 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ec41569a0 0x7f6eb0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ec9a5f640 1 -- 172.21.15.1:0/619499463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb00039b0 msgr2=0x7f6eb0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ec9a5f640 1 --2- 172.21.15.1:0/619499463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb00039b0 0x7f6eb0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ec9a5f640 1 -- 172.21.15.1:0/619499463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb0003520 con 0x7f6eb0004d40 2024-03-06T07:29:36.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.970+0000 7f6ec925e640 1 --2- 172.21.15.1:0/619499463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb00039b0 0x7f6eb0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:36.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.971+0000 7f6ec9a5f640 1 --2- 172.21.15.1:0/619499463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6eb0004d40 0x7f6eb00051c0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc009ac0 tx=0x7f6eb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3f2c5cddf883b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.972+0000 7f6ebbfff640 1 -- 172.21.15.1:0/619499463 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc039070 con 0x7f6eb0004d40 2024-03-06T07:29:36.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.972+0000 7f6ebbfff640 1 -- 172.21.15.1:0/619499463 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ebc02c6b0 con 0x7f6eb0004d40 2024-03-06T07:29:36.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.972+0000 7f6ebbfff640 1 -- 172.21.15.1:0/619499463 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc009690 con 0x7f6eb0004d40 2024-03-06T07:29:36.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.972+0000 7f6ecb4c2640 1 -- 172.21.15.1:0/619499463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6eb0004d40 msgr2=0x7f6eb00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.972+0000 7f6ecb4c2640 1 --2- 172.21.15.1:0/619499463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6eb0004d40 0x7f6eb00051c0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc009ac0 tx=0x7f6eb0005830 comp rx=0 tx=0).stop 2024-03-06T07:29:36.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.972+0000 7f6ec9a5f640 1 -- 172.21.15.1:0/619499463 reap_dead start 2024-03-06T07:29:36.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.972+0000 7f6ecb4c2640 1 -- 172.21.15.1:0/619499463 shutdown_connections 2024-03-06T07:29:36.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.972+0000 7f6ecb4c2640 1 -- 172.21.15.1:0/619499463 >> 172.21.15.1:0/619499463 conn(0x7f6ec40c7a50 msgr2=0x7f6ec40c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.972+0000 7f6ecb4c2640 1 -- 172.21.15.1:0/619499463 shutdown_connections 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.972+0000 7f6ecb4c2640 1 -- 172.21.15.1:0/619499463 wait complete. 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.973+0000 7f6ecb4c2640 1 Processor -- start 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.973+0000 7f6ecb4c2640 1 -- start start 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ecb4c2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb0004d40 0x7f6ec41eb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ecb4c2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ec41dfc70 0x7f6ec40abf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ecb4c2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ec40aac00 0x7f6ec41ede80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ecb4c2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec40c2450 con 0x7f6ec40aac00 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ecb4c2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec4072f60 con 0x7f6eb0004d40 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ecb4c2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec40c25d0 con 0x7f6ec41dfc70 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ec9a5f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ec40aac00 0x7f6ec41ede80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ec9a5f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ec40aac00 0x7f6ec41ede80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56056/0 (socket says 172.21.15.1:56056) 2024-03-06T07:29:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ec9a5f640 1 -- 172.21.15.1:0/1683008382 learned_addr learned my addr 172.21.15.1:0/1683008382 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ec925e640 1 --2- 172.21.15.1:0/1683008382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb0004d40 0x7f6ec41eb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ec8a5d640 1 --2- 172.21.15.1:0/1683008382 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ec41dfc70 0x7f6ec40abf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ec9a5f640 1 -- 172.21.15.1:0/1683008382 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ec41dfc70 msgr2=0x7f6ec40abf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ec9a5f640 1 --2- 172.21.15.1:0/1683008382 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ec41dfc70 0x7f6ec40abf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ec9a5f640 1 -- 172.21.15.1:0/1683008382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb0004d40 msgr2=0x7f6ec41eb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ec9a5f640 1 --2- 172.21.15.1:0/1683008382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb0004d40 0x7f6ec41eb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ec9a5f640 1 -- 172.21.15.1:0/1683008382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec41ea910 con 0x7f6ec40aac00 2024-03-06T07:29:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ec8a5d640 1 --2- 172.21.15.1:0/1683008382 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ec41dfc70 0x7f6ec40abf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.974+0000 7f6ec925e640 1 --2- 172.21.15.1:0/1683008382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb0004d40 0x7f6ec41eb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.975+0000 7f6ec9a5f640 1 --2- 172.21.15.1:0/1683008382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ec40aac00 0x7f6ec41ede80 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc02c320 tx=0x7f6ebc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.975+0000 7f6eb9ffb640 1 -- 172.21.15.1:0/1683008382 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc039070 con 0x7f6ec40aac00 2024-03-06T07:29:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.975+0000 7f6eb9ffb640 1 -- 172.21.15.1:0/1683008382 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ebc043070 con 0x7f6ec40aac00 2024-03-06T07:29:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.975+0000 7f6ecb4c2640 1 -- 172.21.15.1:0/1683008382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec40ac7d0 con 0x7f6ec40aac00 2024-03-06T07:29:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.975+0000 7f6eb9ffb640 1 -- 172.21.15.1:0/1683008382 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec000a4c0 con 0x7f6ec40aac00 2024-03-06T07:29:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.975+0000 7f6ecb4c2640 1 -- 172.21.15.1:0/1683008382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec41eabd0 con 0x7f6ec40aac00 2024-03-06T07:29:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.976+0000 7f6ecb4c2640 1 -- 172.21.15.1:0/1683008382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ec4072b40 con 0x7f6ec40aac00 2024-03-06T07:29:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.977+0000 7f6eb9ffb640 1 -- 172.21.15.1:0/1683008382 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ebc005990 con 0x7f6ec40aac00 2024-03-06T07:29:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.977+0000 7f6eb9ffb640 1 --2- 172.21.15.1:0/1683008382 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ea806b500 0x7f6ea806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.977+0000 7f6eb9ffb640 1 -- 172.21.15.1:0/1683008382 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6ebc0ab3d0 con 0x7f6ec40aac00 2024-03-06T07:29:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.977+0000 7f6ec925e640 1 --2- 172.21.15.1:0/1683008382 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ea806b500 0x7f6ea806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.978+0000 7f6ec925e640 1 --2- 172.21.15.1:0/1683008382 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ea806b500 0x7f6ea806d9c0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0001370 tx=0x7f6ec0006040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:36.981+0000 7f6eb9ffb640 1 -- 172.21.15.1:0/1683008382 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ebc04c060 con 0x7f6ec40aac00 2024-03-06T07:29:37.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.165+0000 7f6ecb4c2640 1 -- 172.21.15.1:0/1683008382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ec40ac4c0 con 0x7f6ec40aac00 2024-03-06T07:29:37.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.166+0000 7f6eb9ffb640 1 -- 172.21.15.1:0/1683008382 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v210) v1 ==== 74+0+16768 (secure 0 0 0) 0x7f6ebc07a150 con 0x7f6ec40aac00 2024-03-06T07:29:37.168 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:37.169 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":210,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:29:34.489494+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:31.114390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:31.815153+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"210","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:37.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.168+0000 7f6ecb4c2640 1 -- 172.21.15.1:0/1683008382 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ea806b500 msgr2=0x7f6ea806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.168+0000 7f6ecb4c2640 1 --2- 172.21.15.1:0/1683008382 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ea806b500 0x7f6ea806d9c0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0001370 tx=0x7f6ec0006040 comp rx=0 tx=0).stop 2024-03-06T07:29:37.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.168+0000 7f6ecb4c2640 1 -- 172.21.15.1:0/1683008382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ec40aac00 msgr2=0x7f6ec41ede80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.168+0000 7f6ecb4c2640 1 --2- 172.21.15.1:0/1683008382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ec40aac00 0x7f6ec41ede80 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc02c320 tx=0x7f6ebc004730 comp rx=0 tx=0).stop 2024-03-06T07:29:37.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.168+0000 7f6ecb4c2640 1 -- 172.21.15.1:0/1683008382 shutdown_connections 2024-03-06T07:29:37.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.168+0000 7f6ecb4c2640 1 --2- 172.21.15.1:0/1683008382 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6ea806b500 0x7f6ea806d9c0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.168+0000 7f6ecb4c2640 1 --2- 172.21.15.1:0/1683008382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ec40aac00 0x7f6ec41ede80 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.168+0000 7f6ecb4c2640 1 --2- 172.21.15.1:0/1683008382 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6ec41dfc70 0x7f6ec40abf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.168+0000 7f6ecb4c2640 1 --2- 172.21.15.1:0/1683008382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb0004d40 0x7f6ec41eb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.168+0000 7f6ecb4c2640 1 -- 172.21.15.1:0/1683008382 >> 172.21.15.1:0/1683008382 conn(0x7f6ec40c7a50 msgr2=0x7f6ec40c44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.168+0000 7f6ecb4c2640 1 -- 172.21.15.1:0/1683008382 shutdown_connections 2024-03-06T07:29:37.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.168+0000 7f6ecb4c2640 1 -- 172.21.15.1:0/1683008382 wait complete. 2024-03-06T07:29:37.181 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:29:37.182 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:37.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.280+0000 7fa4064fa640 1 Processor -- start 2024-03-06T07:29:37.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.280+0000 7fa4064fa640 1 -- start start 2024-03-06T07:29:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.281+0000 7fa4064fa640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa400150520 0x7fa4001543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.281+0000 7fa4064fa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa4000c32f0 0x7fa4001537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.281+0000 7fa4064fa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa400070c80 0x7fa4000710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.281+0000 7fa4064fa640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa400150900 con 0x7fa4000c32f0 2024-03-06T07:29:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.281+0000 7fa4064fa640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa400071790 con 0x7fa400150520 2024-03-06T07:29:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.281+0000 7fa4064fa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa400157050 con 0x7fa400070c80 2024-03-06T07:29:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.281+0000 7fa3ff7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa4000c32f0 0x7fa4001537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.281+0000 7fa3ff7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa4000c32f0 0x7fa4001537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56070/0 (socket says 172.21.15.1:56070) 2024-03-06T07:29:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.281+0000 7fa3ff7fe640 1 -- 172.21.15.1:0/1030554748 learned_addr learned my addr 172.21.15.1:0/1030554748 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.282+0000 7fa404a97640 1 --2- 172.21.15.1:0/1030554748 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa400070c80 0x7fa4000710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.282+0000 7fa3ff7fe640 1 -- 172.21.15.1:0/1030554748 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa400070c80 msgr2=0x7fa4000710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.282+0000 7fa3fffff640 1 --2- 172.21.15.1:0/1030554748 >> v2:172.21.15.98:3300/0 conn(0x7fa400150520 0x7fa4001543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.282+0000 7fa3ff7fe640 1 --2- 172.21.15.1:0/1030554748 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa400070c80 0x7fa4000710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.282+0000 7fa3ff7fe640 1 -- 172.21.15.1:0/1030554748 >> v2:172.21.15.98:3300/0 conn(0x7fa400150520 msgr2=0x7fa4001543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.282+0000 7fa3ff7fe640 1 --2- 172.21.15.1:0/1030554748 >> v2:172.21.15.98:3300/0 conn(0x7fa400150520 0x7fa4001543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.282+0000 7fa3ff7fe640 1 -- 172.21.15.1:0/1030554748 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa400152fc0 con 0x7fa4000c32f0 2024-03-06T07:29:37.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.282+0000 7fa404a97640 1 --2- 172.21.15.1:0/1030554748 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa400070c80 0x7fa4000710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:37.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.282+0000 7fa3ff7fe640 1 --2- 172.21.15.1:0/1030554748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4000c32f0 0x7fa4001537c0 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7fa40006e6c0 tx=0x7fa3f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ab0c5f18c6e5ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.282+0000 7fa3feffd640 1 -- 172.21.15.1:0/1030554748 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4035020 con 0x7fa4000c32f0 2024-03-06T07:29:37.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.283+0000 7fa3feffd640 1 -- 172.21.15.1:0/1030554748 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f4031870 con 0x7fa4000c32f0 2024-03-06T07:29:37.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.283+0000 7fa3feffd640 1 -- 172.21.15.1:0/1030554748 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4031c40 con 0x7fa4000c32f0 2024-03-06T07:29:37.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.283+0000 7fa4064fa640 1 -- 172.21.15.1:0/1030554748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4000c32f0 msgr2=0x7fa4001537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.283+0000 7fa4064fa640 1 --2- 172.21.15.1:0/1030554748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4000c32f0 0x7fa4001537c0 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7fa40006e6c0 tx=0x7fa3f402b910 comp rx=0 tx=0).stop 2024-03-06T07:29:37.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.283+0000 7fa4064fa640 1 -- 172.21.15.1:0/1030554748 shutdown_connections 2024-03-06T07:29:37.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.283+0000 7fa4064fa640 1 --2- 172.21.15.1:0/1030554748 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa400070c80 0x7fa4000710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.283+0000 7fa4064fa640 1 --2- 172.21.15.1:0/1030554748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4000c32f0 0x7fa4001537c0 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.283+0000 7fa4064fa640 1 --2- 172.21.15.1:0/1030554748 >> v2:172.21.15.98:3300/0 conn(0x7fa400150520 0x7fa4001543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.283+0000 7fa4064fa640 1 -- 172.21.15.1:0/1030554748 >> 172.21.15.1:0/1030554748 conn(0x7fa4000c88f0 msgr2=0x7fa4000c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.283+0000 7fa4064fa640 1 -- 172.21.15.1:0/1030554748 shutdown_connections 2024-03-06T07:29:37.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.283+0000 7fa4064fa640 1 -- 172.21.15.1:0/1030554748 wait complete. 2024-03-06T07:29:37.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.284+0000 7fa4064fa640 1 Processor -- start 2024-03-06T07:29:37.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.284+0000 7fa4064fa640 1 -- start start 2024-03-06T07:29:37.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.285+0000 7fa4064fa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa400070c80 0x7fa400151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.285+0000 7fa4064fa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4000c32f0 0x7fa400142d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.285+0000 7fa4064fa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa400150520 0x7fa400143280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.285+0000 7fa4064fa640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4000c7ec0 con 0x7fa4000c32f0 2024-03-06T07:29:37.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.285+0000 7fa4064fa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa400157330 con 0x7fa400070c80 2024-03-06T07:29:37.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.285+0000 7fa4064fa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa400153d00 con 0x7fa400150520 2024-03-06T07:29:37.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.285+0000 7fa3ff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4000c32f0 0x7fa400142d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.285+0000 7fa3ff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4000c32f0 0x7fa400142d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56086/0 (socket says 172.21.15.1:56086) 2024-03-06T07:29:37.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.285+0000 7fa3ff7fe640 1 -- 172.21.15.1:0/3391503931 learned_addr learned my addr 172.21.15.1:0/3391503931 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:37.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.285+0000 7fa404a97640 1 --2- 172.21.15.1:0/3391503931 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa400150520 0x7fa400143280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.285+0000 7fa3fffff640 1 --2- 172.21.15.1:0/3391503931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa400070c80 0x7fa400151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.286+0000 7fa3ff7fe640 1 -- 172.21.15.1:0/3391503931 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa400150520 msgr2=0x7fa400143280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.286+0000 7fa3ff7fe640 1 --2- 172.21.15.1:0/3391503931 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa400150520 0x7fa400143280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.286+0000 7fa3ff7fe640 1 -- 172.21.15.1:0/3391503931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa400070c80 msgr2=0x7fa400151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.286+0000 7fa3ff7fe640 1 --2- 172.21.15.1:0/3391503931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa400070c80 0x7fa400151cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.286+0000 7fa3ff7fe640 1 -- 172.21.15.1:0/3391503931 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa400154b30 con 0x7fa4000c32f0 2024-03-06T07:29:37.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.286+0000 7fa3ff7fe640 1 --2- 172.21.15.1:0/3391503931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4000c32f0 0x7fa400142d40 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7fa3f402b950 tx=0x7fa3f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.286+0000 7fa3fcff9640 1 -- 172.21.15.1:0/3391503931 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4040070 con 0x7fa4000c32f0 2024-03-06T07:29:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.286+0000 7fa3fcff9640 1 -- 172.21.15.1:0/3391503931 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f403a420 con 0x7fa4000c32f0 2024-03-06T07:29:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.286+0000 7fa4064fa640 1 -- 172.21.15.1:0/3391503931 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa400156510 con 0x7fa4000c32f0 2024-03-06T07:29:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.286+0000 7fa3fcff9640 1 -- 172.21.15.1:0/3391503931 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4035040 con 0x7fa4000c32f0 2024-03-06T07:29:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.286+0000 7fa4064fa640 1 -- 172.21.15.1:0/3391503931 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa400156ad0 con 0x7fa4000c32f0 2024-03-06T07:29:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.287+0000 7fa4064fa640 1 -- 172.21.15.1:0/3391503931 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4001542a0 con 0x7fa4000c32f0 2024-03-06T07:29:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.287+0000 7fa3fcff9640 1 -- 172.21.15.1:0/3391503931 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3f4038020 con 0x7fa4000c32f0 2024-03-06T07:29:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.288+0000 7fa3fcff9640 1 --2- 172.21.15.1:0/3391503931 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3ec06b500 0x7fa3ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.288+0000 7fa3fcff9640 1 -- 172.21.15.1:0/3391503931 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa3f40ac380 con 0x7fa4000c32f0 2024-03-06T07:29:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.288+0000 7fa3fffff640 1 --2- 172.21.15.1:0/3391503931 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3ec06b500 0x7fa3ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.289+0000 7fa3fffff640 1 --2- 172.21.15.1:0/3391503931 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3ec06b500 0x7fa3ec06d9c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0006fd0 tx=0x7fa3f0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.292+0000 7fa3fcff9640 1 -- 172.21.15.1:0/3391503931 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa4001542a0 con 0x7fa4000c32f0 2024-03-06T07:29:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.470+0000 7fa4064fa640 1 -- 172.21.15.1:0/3391503931 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa4001548e0 con 0x7fa3ec06b500 2024-03-06T07:29:37.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.476+0000 7fa3fcff9640 1 -- 172.21.15.1:0/3391503931 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252559 (secure 0 0 0) 0x7fa4001548e0 con 0x7fa3ec06b500 2024-03-06T07:29:37.481 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:37.482 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:29:37.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.479+0000 7fa4064fa640 1 -- 172.21.15.1:0/3391503931 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3ec06b500 msgr2=0x7fa3ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.479+0000 7fa4064fa640 1 --2- 172.21.15.1:0/3391503931 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3ec06b500 0x7fa3ec06d9c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0006fd0 tx=0x7fa3f0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:37.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.479+0000 7fa4064fa640 1 -- 172.21.15.1:0/3391503931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4000c32f0 msgr2=0x7fa400142d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.479+0000 7fa4064fa640 1 --2- 172.21.15.1:0/3391503931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4000c32f0 0x7fa400142d40 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7fa3f402b950 tx=0x7fa3f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.479+0000 7fa4064fa640 1 -- 172.21.15.1:0/3391503931 shutdown_connections 2024-03-06T07:29:37.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.479+0000 7fa4064fa640 1 --2- 172.21.15.1:0/3391503931 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3ec06b500 0x7fa3ec06d9c0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.479+0000 7fa4064fa640 1 --2- 172.21.15.1:0/3391503931 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa400150520 0x7fa400143280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.479+0000 7fa4064fa640 1 --2- 172.21.15.1:0/3391503931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4000c32f0 0x7fa400142d40 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.479+0000 7fa4064fa640 1 --2- 172.21.15.1:0/3391503931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa400070c80 0x7fa400151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.479+0000 7fa4064fa640 1 -- 172.21.15.1:0/3391503931 >> 172.21.15.1:0/3391503931 conn(0x7fa4000c88f0 msgr2=0x7fa4001df620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.480+0000 7fa4064fa640 1 -- 172.21.15.1:0/3391503931 shutdown_connections 2024-03-06T07:29:37.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.480+0000 7fa4064fa640 1 -- 172.21.15.1:0/3391503931 wait complete. 2024-03-06T07:29:37.493 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2024-03-06T07:29:36.823790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":287052,"kb_used_data":20940,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754687668,"statfs":{"total":773094113280,"available":772800172032,"internally_reserved":0,"allocated":21442560,"data_stored":11505544,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.348596"},"pg_stats":[{"pgid":"34.3f","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498568+0000","last_change":"2024-03-06T07:29:33.504880+0000","last_active":"2024-03-06T07:29:34.498568+0000","last_peered":"2024-03-06T07:29:34.498568+0000","last_clean":"2024-03-06T07:29:34.498568+0000","last_became_active":"2024-03-06T07:29:33.504634+0000","last_became_peered":"2024-03-06T07:29:33.504634+0000","last_unstale":"2024-03-06T07:29:34.498568+0000","last_undegraded":"2024-03-06T07:29:34.498568+0000","last_fullsized":"2024-03-06T07:29:34.498568+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:43.888641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492837+0000","last_change":"2024-03-06T07:29:33.504328+0000","last_active":"2024-03-06T07:29:34.492837+0000","last_peered":"2024-03-06T07:29:34.492837+0000","last_clean":"2024-03-06T07:29:34.492837+0000","last_became_active":"2024-03-06T07:29:33.503833+0000","last_became_peered":"2024-03-06T07:29:33.503833+0000","last_unstale":"2024-03-06T07:29:34.492837+0000","last_undegraded":"2024-03-06T07:29:34.492837+0000","last_fullsized":"2024-03-06T07:29:34.492837+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:01.422873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.730958+0000","last_change":"2024-03-06T07:29:33.499751+0000","last_active":"2024-03-06T07:29:34.730958+0000","last_peered":"2024-03-06T07:29:34.730958+0000","last_clean":"2024-03-06T07:29:34.730958+0000","last_became_active":"2024-03-06T07:29:33.499448+0000","last_became_peered":"2024-03-06T07:29:33.499448+0000","last_unstale":"2024-03-06T07:29:34.730958+0000","last_undegraded":"2024-03-06T07:29:34.730958+0000","last_fullsized":"2024-03-06T07:29:34.730958+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:28:50.989301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498063+0000","last_change":"2024-03-06T07:29:33.501683+0000","last_active":"2024-03-06T07:29:34.498063+0000","last_peered":"2024-03-06T07:29:34.498063+0000","last_clean":"2024-03-06T07:29:34.498063+0000","last_became_active":"2024-03-06T07:29:33.501372+0000","last_became_peered":"2024-03-06T07:29:33.501372+0000","last_unstale":"2024-03-06T07:29:34.498063+0000","last_undegraded":"2024-03-06T07:29:34.498063+0000","last_fullsized":"2024-03-06T07:29:34.498063+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:46.878541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.505869+0000","last_change":"2024-03-06T07:29:33.504873+0000","last_active":"2024-03-06T07:29:33.505869+0000","last_peered":"2024-03-06T07:29:33.505869+0000","last_clean":"2024-03-06T07:29:33.505869+0000","last_became_active":"2024-03-06T07:29:33.503995+0000","last_became_peered":"2024-03-06T07:29:33.503995+0000","last_unstale":"2024-03-06T07:29:33.505869+0000","last_undegraded":"2024-03-06T07:29:33.505869+0000","last_fullsized":"2024-03-06T07:29:33.505869+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:34.590590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503585+0000","last_change":"2024-03-06T07:29:33.497986+0000","last_active":"2024-03-06T07:29:34.503585+0000","last_peered":"2024-03-06T07:29:34.503585+0000","last_clean":"2024-03-06T07:29:34.503585+0000","last_became_active":"2024-03-06T07:29:33.497636+0000","last_became_peered":"2024-03-06T07:29:33.497636+0000","last_unstale":"2024-03-06T07:29:34.503585+0000","last_undegraded":"2024-03-06T07:29:34.503585+0000","last_fullsized":"2024-03-06T07:29:34.503585+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:22.227727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.730698+0000","last_change":"2024-03-06T07:29:33.495174+0000","last_active":"2024-03-06T07:29:34.730698+0000","last_peered":"2024-03-06T07:29:34.730698+0000","last_clean":"2024-03-06T07:29:34.730698+0000","last_became_active":"2024-03-06T07:29:33.494882+0000","last_became_peered":"2024-03-06T07:29:33.494882+0000","last_unstale":"2024-03-06T07:29:34.730698+0000","last_undegraded":"2024-03-06T07:29:34.730698+0000","last_fullsized":"2024-03-06T07:29:34.730698+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:43.888641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.497882+0000","last_change":"2024-03-06T07:29:33.498188+0000","last_active":"2024-03-06T07:29:34.497882+0000","last_peered":"2024-03-06T07:29:34.497882+0000","last_clean":"2024-03-06T07:29:34.497882+0000","last_became_active":"2024-03-06T07:29:33.497889+0000","last_became_peered":"2024-03-06T07:29:33.497889+0000","last_unstale":"2024-03-06T07:29:34.497882+0000","last_undegraded":"2024-03-06T07:29:34.497882+0000","last_fullsized":"2024-03-06T07:29:34.497882+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:30.609804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503370+0000","last_change":"2024-03-06T07:29:33.508445+0000","last_active":"2024-03-06T07:29:34.503370+0000","last_peered":"2024-03-06T07:29:34.503370+0000","last_clean":"2024-03-06T07:29:34.503370+0000","last_became_active":"2024-03-06T07:29:33.508141+0000","last_became_peered":"2024-03-06T07:29:33.508141+0000","last_unstale":"2024-03-06T07:29:34.503370+0000","last_undegraded":"2024-03-06T07:29:34.503370+0000","last_fullsized":"2024-03-06T07:29:34.503370+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:48.739851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.505135+0000","last_change":"2024-03-06T07:29:33.503858+0000","last_active":"2024-03-06T07:29:33.505135+0000","last_peered":"2024-03-06T07:29:33.505135+0000","last_clean":"2024-03-06T07:29:33.505135+0000","last_became_active":"2024-03-06T07:29:33.503516+0000","last_became_peered":"2024-03-06T07:29:33.503516+0000","last_unstale":"2024-03-06T07:29:33.505135+0000","last_undegraded":"2024-03-06T07:29:33.505135+0000","last_fullsized":"2024-03-06T07:29:33.505135+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:36.786303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498572+0000","last_change":"2024-03-06T07:29:33.493073+0000","last_active":"2024-03-06T07:29:34.498572+0000","last_peered":"2024-03-06T07:29:34.498572+0000","last_clean":"2024-03-06T07:29:34.498572+0000","last_became_active":"2024-03-06T07:29:33.492379+0000","last_became_peered":"2024-03-06T07:29:33.492379+0000","last_unstale":"2024-03-06T07:29:34.498572+0000","last_undegraded":"2024-03-06T07:29:34.498572+0000","last_fullsized":"2024-03-06T07:29:34.498572+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:16.464883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503601+0000","last_change":"2024-03-06T07:29:33.500635+0000","last_active":"2024-03-06T07:29:34.503601+0000","last_peered":"2024-03-06T07:29:34.503601+0000","last_clean":"2024-03-06T07:29:34.503601+0000","last_became_active":"2024-03-06T07:29:33.500153+0000","last_became_peered":"2024-03-06T07:29:33.500153+0000","last_unstale":"2024-03-06T07:29:34.503601+0000","last_undegraded":"2024-03-06T07:29:34.503601+0000","last_fullsized":"2024-03-06T07:29:34.503601+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:32.730513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.499002+0000","last_change":"2024-03-06T07:29:33.497422+0000","last_active":"2024-03-06T07:29:34.499002+0000","last_peered":"2024-03-06T07:29:34.499002+0000","last_clean":"2024-03-06T07:29:34.499002+0000","last_became_active":"2024-03-06T07:29:33.495981+0000","last_became_peered":"2024-03-06T07:29:33.495981+0000","last_unstale":"2024-03-06T07:29:34.499002+0000","last_undegraded":"2024-03-06T07:29:34.499002+0000","last_fullsized":"2024-03-06T07:29:34.499002+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:39.398132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492944+0000","last_change":"2024-03-06T07:29:33.490558+0000","last_active":"2024-03-06T07:29:34.492944+0000","last_peered":"2024-03-06T07:29:34.492944+0000","last_clean":"2024-03-06T07:29:34.492944+0000","last_became_active":"2024-03-06T07:29:33.490269+0000","last_became_peered":"2024-03-06T07:29:33.490269+0000","last_unstale":"2024-03-06T07:29:34.492944+0000","last_undegraded":"2024-03-06T07:29:34.492944+0000","last_fullsized":"2024-03-06T07:29:34.492944+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:40.447291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492566+0000","last_change":"2024-03-06T07:29:33.491298+0000","last_active":"2024-03-06T07:29:34.492566+0000","last_peered":"2024-03-06T07:29:34.492566+0000","last_clean":"2024-03-06T07:29:34.492566+0000","last_became_active":"2024-03-06T07:29:33.491006+0000","last_became_peered":"2024-03-06T07:29:33.491006+0000","last_unstale":"2024-03-06T07:29:34.492566+0000","last_undegraded":"2024-03-06T07:29:34.492566+0000","last_fullsized":"2024-03-06T07:29:34.492566+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:48.214015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.503549+0000","last_change":"2024-03-06T07:29:33.502406+0000","last_active":"2024-03-06T07:29:33.503549+0000","last_peered":"2024-03-06T07:29:33.503549+0000","last_clean":"2024-03-06T07:29:33.503549+0000","last_became_active":"2024-03-06T07:29:33.501839+0000","last_became_peered":"2024-03-06T07:29:33.501839+0000","last_unstale":"2024-03-06T07:29:33.503549+0000","last_undegraded":"2024-03-06T07:29:33.503549+0000","last_fullsized":"2024-03-06T07:29:33.503549+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:25.320033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.502705+0000","last_change":"2024-03-06T07:29:33.500303+0000","last_active":"2024-03-06T07:29:33.502705+0000","last_peered":"2024-03-06T07:29:33.502705+0000","last_clean":"2024-03-06T07:29:33.502705+0000","last_became_active":"2024-03-06T07:29:33.500040+0000","last_became_peered":"2024-03-06T07:29:33.500040+0000","last_unstale":"2024-03-06T07:29:33.502705+0000","last_undegraded":"2024-03-06T07:29:33.502705+0000","last_fullsized":"2024-03-06T07:29:33.502705+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:49:08.979523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498591+0000","last_change":"2024-03-06T07:29:33.514706+0000","last_active":"2024-03-06T07:29:34.498591+0000","last_peered":"2024-03-06T07:29:34.498591+0000","last_clean":"2024-03-06T07:29:34.498591+0000","last_became_active":"2024-03-06T07:29:33.514280+0000","last_became_peered":"2024-03-06T07:29:33.514280+0000","last_unstale":"2024-03-06T07:29:34.498591+0000","last_undegraded":"2024-03-06T07:29:34.498591+0000","last_fullsized":"2024-03-06T07:29:34.498591+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:45.717490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.500166+0000","last_change":"2024-03-06T07:29:33.499503+0000","last_active":"2024-03-06T07:29:33.500166+0000","last_peered":"2024-03-06T07:29:33.500166+0000","last_clean":"2024-03-06T07:29:33.500166+0000","last_became_active":"2024-03-06T07:29:33.499241+0000","last_became_peered":"2024-03-06T07:29:33.499241+0000","last_unstale":"2024-03-06T07:29:33.500166+0000","last_undegraded":"2024-03-06T07:29:33.500166+0000","last_fullsized":"2024-03-06T07:29:33.500166+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:41.473077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503608+0000","last_change":"2024-03-06T07:29:33.500940+0000","last_active":"2024-03-06T07:29:34.503608+0000","last_peered":"2024-03-06T07:29:34.503608+0000","last_clean":"2024-03-06T07:29:34.503608+0000","last_became_active":"2024-03-06T07:29:33.500455+0000","last_became_peered":"2024-03-06T07:29:33.500455+0000","last_unstale":"2024-03-06T07:29:34.503608+0000","last_undegraded":"2024-03-06T07:29:34.503608+0000","last_fullsized":"2024-03-06T07:29:34.503608+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:57.009450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.503175+0000","last_change":"2024-03-06T07:29:33.502644+0000","last_active":"2024-03-06T07:29:33.503175+0000","last_peered":"2024-03-06T07:29:33.503175+0000","last_clean":"2024-03-06T07:29:33.503175+0000","last_became_active":"2024-03-06T07:29:33.502373+0000","last_became_peered":"2024-03-06T07:29:33.502373+0000","last_unstale":"2024-03-06T07:29:33.503175+0000","last_undegraded":"2024-03-06T07:29:33.503175+0000","last_fullsized":"2024-03-06T07:29:33.503175+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:57.089223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498315+0000","last_change":"2024-03-06T07:29:33.504232+0000","last_active":"2024-03-06T07:29:34.498315+0000","last_peered":"2024-03-06T07:29:34.498315+0000","last_clean":"2024-03-06T07:29:34.498315+0000","last_became_active":"2024-03-06T07:29:33.503928+0000","last_became_peered":"2024-03-06T07:29:33.503928+0000","last_unstale":"2024-03-06T07:29:34.498315+0000","last_undegraded":"2024-03-06T07:29:34.498315+0000","last_fullsized":"2024-03-06T07:29:34.498315+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:40.445535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492918+0000","last_change":"2024-03-06T07:29:33.501325+0000","last_active":"2024-03-06T07:29:34.492918+0000","last_peered":"2024-03-06T07:29:34.492918+0000","last_clean":"2024-03-06T07:29:34.492918+0000","last_became_active":"2024-03-06T07:29:33.501008+0000","last_became_peered":"2024-03-06T07:29:33.501008+0000","last_unstale":"2024-03-06T07:29:34.492918+0000","last_undegraded":"2024-03-06T07:29:34.492918+0000","last_fullsized":"2024-03-06T07:29:34.492918+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:43.154947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498168+0000","last_change":"2024-03-06T07:29:33.495976+0000","last_active":"2024-03-06T07:29:34.498168+0000","last_peered":"2024-03-06T07:29:34.498168+0000","last_clean":"2024-03-06T07:29:34.498168+0000","last_became_active":"2024-03-06T07:29:33.495713+0000","last_became_peered":"2024-03-06T07:29:33.495713+0000","last_unstale":"2024-03-06T07:29:34.498168+0000","last_undegraded":"2024-03-06T07:29:34.498168+0000","last_fullsized":"2024-03-06T07:29:34.498168+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:02.813993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.730998+0000","last_change":"2024-03-06T07:29:33.505724+0000","last_active":"2024-03-06T07:29:34.730998+0000","last_peered":"2024-03-06T07:29:34.730998+0000","last_clean":"2024-03-06T07:29:34.730998+0000","last_became_active":"2024-03-06T07:29:33.505435+0000","last_became_peered":"2024-03-06T07:29:33.505435+0000","last_unstale":"2024-03-06T07:29:34.730998+0000","last_undegraded":"2024-03-06T07:29:34.730998+0000","last_fullsized":"2024-03-06T07:29:34.730998+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:34.334299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492982+0000","last_change":"2024-03-06T07:29:33.508808+0000","last_active":"2024-03-06T07:29:34.492982+0000","last_peered":"2024-03-06T07:29:34.492982+0000","last_clean":"2024-03-06T07:29:34.492982+0000","last_became_active":"2024-03-06T07:29:33.508513+0000","last_became_peered":"2024-03-06T07:29:33.508513+0000","last_unstale":"2024-03-06T07:29:34.492982+0000","last_undegraded":"2024-03-06T07:29:34.492982+0000","last_fullsized":"2024-03-06T07:29:34.492982+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:43.704508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498675+0000","last_change":"2024-03-06T07:29:33.504877+0000","last_active":"2024-03-06T07:29:34.498675+0000","last_peered":"2024-03-06T07:29:34.498675+0000","last_clean":"2024-03-06T07:29:34.498675+0000","last_became_active":"2024-03-06T07:29:33.504543+0000","last_became_peered":"2024-03-06T07:29:33.504543+0000","last_unstale":"2024-03-06T07:29:34.498675+0000","last_undegraded":"2024-03-06T07:29:34.498675+0000","last_fullsized":"2024-03-06T07:29:34.498675+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:28:50.989301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498856+0000","last_change":"2024-03-06T07:29:33.491056+0000","last_active":"2024-03-06T07:29:34.498856+0000","last_peered":"2024-03-06T07:29:34.498856+0000","last_clean":"2024-03-06T07:29:34.498856+0000","last_became_active":"2024-03-06T07:29:33.490790+0000","last_became_peered":"2024-03-06T07:29:33.490790+0000","last_unstale":"2024-03-06T07:29:34.498856+0000","last_undegraded":"2024-03-06T07:29:34.498856+0000","last_fullsized":"2024-03-06T07:29:34.498856+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:00.202804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503110+0000","last_change":"2024-03-06T07:29:33.501613+0000","last_active":"2024-03-06T07:29:34.503110+0000","last_peered":"2024-03-06T07:29:34.503110+0000","last_clean":"2024-03-06T07:29:34.503110+0000","last_became_active":"2024-03-06T07:29:33.501336+0000","last_became_peered":"2024-03-06T07:29:33.501336+0000","last_unstale":"2024-03-06T07:29:34.503110+0000","last_undegraded":"2024-03-06T07:29:34.503110+0000","last_fullsized":"2024-03-06T07:29:34.503110+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:56.351464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.493066+0000","last_change":"2024-03-06T07:29:33.491492+0000","last_active":"2024-03-06T07:29:34.493066+0000","last_peered":"2024-03-06T07:29:34.493066+0000","last_clean":"2024-03-06T07:29:34.493066+0000","last_became_active":"2024-03-06T07:29:33.491207+0000","last_became_peered":"2024-03-06T07:29:33.491207+0000","last_unstale":"2024-03-06T07:29:34.493066+0000","last_undegraded":"2024-03-06T07:29:34.493066+0000","last_fullsized":"2024-03-06T07:29:34.493066+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:01.392304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492508+0000","last_change":"2024-03-06T07:29:33.504893+0000","last_active":"2024-03-06T07:29:34.492508+0000","last_peered":"2024-03-06T07:29:34.492508+0000","last_clean":"2024-03-06T07:29:34.492508+0000","last_became_active":"2024-03-06T07:29:33.503619+0000","last_became_peered":"2024-03-06T07:29:33.503619+0000","last_unstale":"2024-03-06T07:29:34.492508+0000","last_undegraded":"2024-03-06T07:29:34.492508+0000","last_fullsized":"2024-03-06T07:29:34.492508+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:27.541752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.503857+0000","last_change":"2024-03-06T07:29:33.502459+0000","last_active":"2024-03-06T07:29:33.503857+0000","last_peered":"2024-03-06T07:29:33.503857+0000","last_clean":"2024-03-06T07:29:33.503857+0000","last_became_active":"2024-03-06T07:29:33.501793+0000","last_became_peered":"2024-03-06T07:29:33.501793+0000","last_unstale":"2024-03-06T07:29:33.503857+0000","last_undegraded":"2024-03-06T07:29:33.503857+0000","last_fullsized":"2024-03-06T07:29:33.503857+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:07.774336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"209'4","reported_seq":18,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498630+0000","last_change":"2024-03-06T07:29:32.492127+0000","last_active":"2024-03-06T07:29:34.498630+0000","last_peered":"2024-03-06T07:29:34.498630+0000","last_clean":"2024-03-06T07:29:34.498630+0000","last_became_active":"2024-03-06T07:29:32.491791+0000","last_became_peered":"2024-03-06T07:29:32.491791+0000","last_unstale":"2024-03-06T07:29:34.498630+0000","last_undegraded":"2024-03-06T07:29:34.498630+0000","last_fullsized":"2024-03-06T07:29:34.498630+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:33.692640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498294+0000","last_change":"2024-03-06T07:29:33.497731+0000","last_active":"2024-03-06T07:29:34.498294+0000","last_peered":"2024-03-06T07:29:34.498294+0000","last_clean":"2024-03-06T07:29:34.498294+0000","last_became_active":"2024-03-06T07:29:33.497465+0000","last_became_peered":"2024-03-06T07:29:33.497465+0000","last_unstale":"2024-03-06T07:29:34.498294+0000","last_undegraded":"2024-03-06T07:29:34.498294+0000","last_fullsized":"2024-03-06T07:29:34.498294+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:29.677861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"209'1","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.511149+0000","last_change":"2024-03-06T07:29:32.494084+0000","last_active":"2024-03-06T07:29:33.511149+0000","last_peered":"2024-03-06T07:29:33.511149+0000","last_clean":"2024-03-06T07:29:33.511149+0000","last_became_active":"2024-03-06T07:29:32.493760+0000","last_became_peered":"2024-03-06T07:29:32.493760+0000","last_unstale":"2024-03-06T07:29:33.511149+0000","last_undegraded":"2024-03-06T07:29:33.511149+0000","last_fullsized":"2024-03-06T07:29:33.511149+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:38.243046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498678+0000","last_change":"2024-03-06T07:29:33.497948+0000","last_active":"2024-03-06T07:29:34.498678+0000","last_peered":"2024-03-06T07:29:34.498678+0000","last_clean":"2024-03-06T07:29:34.498678+0000","last_became_active":"2024-03-06T07:29:33.497690+0000","last_became_peered":"2024-03-06T07:29:33.497690+0000","last_unstale":"2024-03-06T07:29:34.498678+0000","last_undegraded":"2024-03-06T07:29:34.498678+0000","last_fullsized":"2024-03-06T07:29:34.498678+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:35.917308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"209'2","reported_seq":16,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498794+0000","last_change":"2024-03-06T07:29:32.495714+0000","last_active":"2024-03-06T07:29:34.498794+0000","last_peered":"2024-03-06T07:29:34.498794+0000","last_clean":"2024-03-06T07:29:34.498794+0000","last_became_active":"2024-03-06T07:29:32.495440+0000","last_became_peered":"2024-03-06T07:29:32.495440+0000","last_unstale":"2024-03-06T07:29:34.498794+0000","last_undegraded":"2024-03-06T07:29:34.498794+0000","last_fullsized":"2024-03-06T07:29:34.498794+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:36:27.145630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492843+0000","last_change":"2024-03-06T07:29:33.498622+0000","last_active":"2024-03-06T07:29:34.492843+0000","last_peered":"2024-03-06T07:29:34.492843+0000","last_clean":"2024-03-06T07:29:34.492843+0000","last_became_active":"2024-03-06T07:29:33.498332+0000","last_became_peered":"2024-03-06T07:29:33.498332+0000","last_unstale":"2024-03-06T07:29:34.492843+0000","last_undegraded":"2024-03-06T07:29:34.492843+0000","last_fullsized":"2024-03-06T07:29:34.492843+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:09.257168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"209'2","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.513181+0000","last_change":"2024-03-06T07:29:32.500977+0000","last_active":"2024-03-06T07:29:33.513181+0000","last_peered":"2024-03-06T07:29:33.513181+0000","last_clean":"2024-03-06T07:29:33.513181+0000","last_became_active":"2024-03-06T07:29:32.500763+0000","last_became_peered":"2024-03-06T07:29:32.500763+0000","last_unstale":"2024-03-06T07:29:33.513181+0000","last_undegraded":"2024-03-06T07:29:33.513181+0000","last_fullsized":"2024-03-06T07:29:33.513181+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:57.866450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492657+0000","last_change":"2024-03-06T07:29:33.495369+0000","last_active":"2024-03-06T07:29:34.492657+0000","last_peered":"2024-03-06T07:29:34.492657+0000","last_clean":"2024-03-06T07:29:34.492657+0000","last_became_active":"2024-03-06T07:29:33.493449+0000","last_became_peered":"2024-03-06T07:29:33.493449+0000","last_unstale":"2024-03-06T07:29:34.492657+0000","last_undegraded":"2024-03-06T07:29:34.492657+0000","last_fullsized":"2024-03-06T07:29:34.492657+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:46.088883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"209'1","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.517523+0000","last_change":"2024-03-06T07:29:32.500405+0000","last_active":"2024-03-06T07:29:33.517523+0000","last_peered":"2024-03-06T07:29:33.517523+0000","last_clean":"2024-03-06T07:29:33.517523+0000","last_became_active":"2024-03-06T07:29:32.499655+0000","last_became_peered":"2024-03-06T07:29:32.499655+0000","last_unstale":"2024-03-06T07:29:33.517523+0000","last_undegraded":"2024-03-06T07:29:33.517523+0000","last_fullsized":"2024-03-06T07:29:33.517523+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:21.221911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.493072+0000","last_change":"2024-03-06T07:29:33.504314+0000","last_active":"2024-03-06T07:29:34.493072+0000","last_peered":"2024-03-06T07:29:34.493072+0000","last_clean":"2024-03-06T07:29:34.493072+0000","last_became_active":"2024-03-06T07:29:33.503874+0000","last_became_peered":"2024-03-06T07:29:33.503874+0000","last_unstale":"2024-03-06T07:29:34.493072+0000","last_undegraded":"2024-03-06T07:29:34.493072+0000","last_fullsized":"2024-03-06T07:29:34.493072+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:55.554029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"209'1","reported_seq":15,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492953+0000","last_change":"2024-03-06T07:29:32.498552+0000","last_active":"2024-03-06T07:29:34.492953+0000","last_peered":"2024-03-06T07:29:34.492953+0000","last_clean":"2024-03-06T07:29:34.492953+0000","last_became_active":"2024-03-06T07:29:32.498288+0000","last_became_peered":"2024-03-06T07:29:32.498288+0000","last_unstale":"2024-03-06T07:29:34.492953+0000","last_undegraded":"2024-03-06T07:29:34.492953+0000","last_fullsized":"2024-03-06T07:29:34.492953+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:25.005877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.730871+0000","last_change":"2024-03-06T07:29:33.500643+0000","last_active":"2024-03-06T07:29:34.730871+0000","last_peered":"2024-03-06T07:29:34.730871+0000","last_clean":"2024-03-06T07:29:34.730871+0000","last_became_active":"2024-03-06T07:29:33.500365+0000","last_became_peered":"2024-03-06T07:29:33.500365+0000","last_unstale":"2024-03-06T07:29:34.730871+0000","last_undegraded":"2024-03-06T07:29:34.730871+0000","last_fullsized":"2024-03-06T07:29:34.730871+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:16.464883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.491543+0000","last_change":"2024-03-06T07:29:32.495493+0000","last_active":"2024-03-06T07:29:33.491543+0000","last_peered":"2024-03-06T07:29:33.491543+0000","last_clean":"2024-03-06T07:29:33.491543+0000","last_became_active":"2024-03-06T07:29:32.495160+0000","last_became_peered":"2024-03-06T07:29:32.495160+0000","last_unstale":"2024-03-06T07:29:33.491543+0000","last_undegraded":"2024-03-06T07:29:33.491543+0000","last_fullsized":"2024-03-06T07:29:33.491543+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:43.429625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.499105+0000","last_change":"2024-03-06T07:29:33.494554+0000","last_active":"2024-03-06T07:29:34.499105+0000","last_peered":"2024-03-06T07:29:34.499105+0000","last_clean":"2024-03-06T07:29:34.499105+0000","last_became_active":"2024-03-06T07:29:33.494188+0000","last_became_peered":"2024-03-06T07:29:33.494188+0000","last_unstale":"2024-03-06T07:29:34.499105+0000","last_undegraded":"2024-03-06T07:29:34.499105+0000","last_fullsized":"2024-03-06T07:29:34.499105+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:27.613945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.498186+0000","last_change":"2024-03-06T07:29:32.495708+0000","last_active":"2024-03-06T07:29:33.498186+0000","last_peered":"2024-03-06T07:29:33.498186+0000","last_clean":"2024-03-06T07:29:33.498186+0000","last_became_active":"2024-03-06T07:29:32.495449+0000","last_became_peered":"2024-03-06T07:29:32.495449+0000","last_unstale":"2024-03-06T07:29:33.498186+0000","last_undegraded":"2024-03-06T07:29:33.498186+0000","last_fullsized":"2024-03-06T07:29:33.498186+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:37.488554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498218+0000","last_change":"2024-03-06T07:29:33.500309+0000","last_active":"2024-03-06T07:29:34.498218+0000","last_peered":"2024-03-06T07:29:34.498218+0000","last_clean":"2024-03-06T07:29:34.498218+0000","last_became_active":"2024-03-06T07:29:33.499997+0000","last_became_peered":"2024-03-06T07:29:33.499997+0000","last_unstale":"2024-03-06T07:29:34.498218+0000","last_undegraded":"2024-03-06T07:29:34.498218+0000","last_fullsized":"2024-03-06T07:29:34.498218+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:47.739042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498484+0000","last_change":"2024-03-06T07:29:32.504424+0000","last_active":"2024-03-06T07:29:34.498484+0000","last_peered":"2024-03-06T07:29:34.498484+0000","last_clean":"2024-03-06T07:29:34.498484+0000","last_became_active":"2024-03-06T07:29:32.504156+0000","last_became_peered":"2024-03-06T07:29:32.504156+0000","last_unstale":"2024-03-06T07:29:34.498484+0000","last_undegraded":"2024-03-06T07:29:34.498484+0000","last_fullsized":"2024-03-06T07:29:34.498484+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:47:07.441888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498429+0000","last_change":"2024-03-06T07:29:33.496329+0000","last_active":"2024-03-06T07:29:34.498429+0000","last_peered":"2024-03-06T07:29:34.498429+0000","last_clean":"2024-03-06T07:29:34.498429+0000","last_became_active":"2024-03-06T07:29:33.496027+0000","last_became_peered":"2024-03-06T07:29:33.496027+0000","last_unstale":"2024-03-06T07:29:34.498429+0000","last_undegraded":"2024-03-06T07:29:34.498429+0000","last_fullsized":"2024-03-06T07:29:34.498429+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:54.522857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.492901+0000","last_change":"2024-03-06T07:29:32.496371+0000","last_active":"2024-03-06T07:29:33.492901+0000","last_peered":"2024-03-06T07:29:33.492901+0000","last_clean":"2024-03-06T07:29:33.492901+0000","last_became_active":"2024-03-06T07:29:32.495936+0000","last_became_peered":"2024-03-06T07:29:32.495936+0000","last_unstale":"2024-03-06T07:29:33.492901+0000","last_undegraded":"2024-03-06T07:29:33.492901+0000","last_fullsized":"2024-03-06T07:29:33.492901+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:05.820201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503561+0000","last_change":"2024-03-06T07:29:33.501319+0000","last_active":"2024-03-06T07:29:34.503561+0000","last_peered":"2024-03-06T07:29:34.503561+0000","last_clean":"2024-03-06T07:29:34.503561+0000","last_became_active":"2024-03-06T07:29:33.501050+0000","last_became_peered":"2024-03-06T07:29:33.501050+0000","last_unstale":"2024-03-06T07:29:34.503561+0000","last_undegraded":"2024-03-06T07:29:34.503561+0000","last_fullsized":"2024-03-06T07:29:34.503561+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:38.494370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"209'3","reported_seq":17,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498390+0000","last_change":"2024-03-06T07:29:32.501594+0000","last_active":"2024-03-06T07:29:34.498390+0000","last_peered":"2024-03-06T07:29:34.498390+0000","last_clean":"2024-03-06T07:29:34.498390+0000","last_became_active":"2024-03-06T07:29:32.501307+0000","last_became_peered":"2024-03-06T07:29:32.501307+0000","last_unstale":"2024-03-06T07:29:34.498390+0000","last_undegraded":"2024-03-06T07:29:34.498390+0000","last_fullsized":"2024-03-06T07:29:34.498390+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:36.439471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498909+0000","last_change":"2024-03-06T07:29:33.505956+0000","last_active":"2024-03-06T07:29:34.498909+0000","last_peered":"2024-03-06T07:29:34.498909+0000","last_clean":"2024-03-06T07:29:34.498909+0000","last_became_active":"2024-03-06T07:29:33.505409+0000","last_became_peered":"2024-03-06T07:29:33.505409+0000","last_unstale":"2024-03-06T07:29:34.498909+0000","last_undegraded":"2024-03-06T07:29:34.498909+0000","last_fullsized":"2024-03-06T07:29:34.498909+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:03.871710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"209'6","reported_seq":20,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498231+0000","last_change":"2024-03-06T07:29:32.496604+0000","last_active":"2024-03-06T07:29:34.498231+0000","last_peered":"2024-03-06T07:29:34.498231+0000","last_clean":"2024-03-06T07:29:34.498231+0000","last_became_active":"2024-03-06T07:29:32.496313+0000","last_became_peered":"2024-03-06T07:29:32.496313+0000","last_unstale":"2024-03-06T07:29:34.498231+0000","last_undegraded":"2024-03-06T07:29:34.498231+0000","last_fullsized":"2024-03-06T07:29:34.498231+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:10.982908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.507007+0000","last_change":"2024-03-06T07:29:33.506274+0000","last_active":"2024-03-06T07:29:33.507007+0000","last_peered":"2024-03-06T07:29:33.507007+0000","last_clean":"2024-03-06T07:29:33.507007+0000","last_became_active":"2024-03-06T07:29:33.505951+0000","last_became_peered":"2024-03-06T07:29:33.505951+0000","last_unstale":"2024-03-06T07:29:33.507007+0000","last_undegraded":"2024-03-06T07:29:33.507007+0000","last_fullsized":"2024-03-06T07:29:33.507007+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:28.486711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"209'2","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.517763+0000","last_change":"2024-03-06T07:29:32.500150+0000","last_active":"2024-03-06T07:29:33.517763+0000","last_peered":"2024-03-06T07:29:33.517763+0000","last_clean":"2024-03-06T07:29:33.517763+0000","last_became_active":"2024-03-06T07:29:32.499360+0000","last_became_peered":"2024-03-06T07:29:32.499360+0000","last_unstale":"2024-03-06T07:29:33.517763+0000","last_undegraded":"2024-03-06T07:29:33.517763+0000","last_fullsized":"2024-03-06T07:29:33.517763+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:31.724697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498206+0000","last_change":"2024-03-06T07:29:33.489574+0000","last_active":"2024-03-06T07:29:34.498206+0000","last_peered":"2024-03-06T07:29:34.498206+0000","last_clean":"2024-03-06T07:29:34.498206+0000","last_became_active":"2024-03-06T07:29:33.489319+0000","last_became_peered":"2024-03-06T07:29:33.489319+0000","last_unstale":"2024-03-06T07:29:34.498206+0000","last_undegraded":"2024-03-06T07:29:34.498206+0000","last_fullsized":"2024-03-06T07:29:34.498206+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:49.153262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498814+0000","last_change":"2024-03-06T07:29:32.500847+0000","last_active":"2024-03-06T07:29:34.498814+0000","last_peered":"2024-03-06T07:29:34.498814+0000","last_clean":"2024-03-06T07:29:34.498814+0000","last_became_active":"2024-03-06T07:29:32.500574+0000","last_became_peered":"2024-03-06T07:29:32.500574+0000","last_unstale":"2024-03-06T07:29:34.498814+0000","last_undegraded":"2024-03-06T07:29:34.498814+0000","last_fullsized":"2024-03-06T07:29:34.498814+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:21.386627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.509034+0000","last_change":"2024-03-06T07:29:33.508243+0000","last_active":"2024-03-06T07:29:33.509034+0000","last_peered":"2024-03-06T07:29:33.509034+0000","last_clean":"2024-03-06T07:29:33.509034+0000","last_became_active":"2024-03-06T07:29:33.507981+0000","last_became_peered":"2024-03-06T07:29:33.507981+0000","last_unstale":"2024-03-06T07:29:33.509034+0000","last_undegraded":"2024-03-06T07:29:33.509034+0000","last_fullsized":"2024-03-06T07:29:33.509034+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:32.910447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498467+0000","last_change":"2024-03-06T07:29:32.501736+0000","last_active":"2024-03-06T07:29:34.498467+0000","last_peered":"2024-03-06T07:29:34.498467+0000","last_clean":"2024-03-06T07:29:34.498467+0000","last_became_active":"2024-03-06T07:29:32.501444+0000","last_became_peered":"2024-03-06T07:29:32.501444+0000","last_unstale":"2024-03-06T07:29:34.498467+0000","last_undegraded":"2024-03-06T07:29:34.498467+0000","last_fullsized":"2024-03-06T07:29:34.498467+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:52.615490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.505890+0000","last_change":"2024-03-06T07:29:33.505096+0000","last_active":"2024-03-06T07:29:33.505890+0000","last_peered":"2024-03-06T07:29:33.505890+0000","last_clean":"2024-03-06T07:29:33.505890+0000","last_became_active":"2024-03-06T07:29:33.504601+0000","last_became_peered":"2024-03-06T07:29:33.504601+0000","last_unstale":"2024-03-06T07:29:33.505890+0000","last_undegraded":"2024-03-06T07:29:33.505890+0000","last_fullsized":"2024-03-06T07:29:33.505890+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:24.738032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"209'8","reported_seq":22,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498106+0000","last_change":"2024-03-06T07:29:32.504142+0000","last_active":"2024-03-06T07:29:34.498106+0000","last_peered":"2024-03-06T07:29:34.498106+0000","last_clean":"2024-03-06T07:29:34.498106+0000","last_became_active":"2024-03-06T07:29:32.503854+0000","last_became_peered":"2024-03-06T07:29:32.503854+0000","last_unstale":"2024-03-06T07:29:34.498106+0000","last_undegraded":"2024-03-06T07:29:34.498106+0000","last_fullsized":"2024-03-06T07:29:34.498106+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:36.875697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498505+0000","last_change":"2024-03-06T07:29:33.497491+0000","last_active":"2024-03-06T07:29:34.498505+0000","last_peered":"2024-03-06T07:29:34.498505+0000","last_clean":"2024-03-06T07:29:34.498505+0000","last_became_active":"2024-03-06T07:29:33.497200+0000","last_became_peered":"2024-03-06T07:29:33.497200+0000","last_unstale":"2024-03-06T07:29:34.498505+0000","last_undegraded":"2024-03-06T07:29:34.498505+0000","last_fullsized":"2024-03-06T07:29:34.498505+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:02.396510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"209'1","reported_seq":15,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503413+0000","last_change":"2024-03-06T07:29:32.494708+0000","last_active":"2024-03-06T07:29:34.503413+0000","last_peered":"2024-03-06T07:29:34.503413+0000","last_clean":"2024-03-06T07:29:34.503413+0000","last_became_active":"2024-03-06T07:29:32.494070+0000","last_became_peered":"2024-03-06T07:29:32.494070+0000","last_unstale":"2024-03-06T07:29:34.503413+0000","last_undegraded":"2024-03-06T07:29:34.503413+0000","last_fullsized":"2024-03-06T07:29:34.503413+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:49.223819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498731+0000","last_change":"2024-03-06T07:29:33.492932+0000","last_active":"2024-03-06T07:29:34.498731+0000","last_peered":"2024-03-06T07:29:34.498731+0000","last_clean":"2024-03-06T07:29:34.498731+0000","last_became_active":"2024-03-06T07:29:33.492467+0000","last_became_peered":"2024-03-06T07:29:33.492467+0000","last_unstale":"2024-03-06T07:29:34.498731+0000","last_undegraded":"2024-03-06T07:29:34.498731+0000","last_fullsized":"2024-03-06T07:29:34.498731+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:07.335717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"209'2","reported_seq":16,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498700+0000","last_change":"2024-03-06T07:29:32.500950+0000","last_active":"2024-03-06T07:29:34.498700+0000","last_peered":"2024-03-06T07:29:34.498700+0000","last_clean":"2024-03-06T07:29:34.498700+0000","last_became_active":"2024-03-06T07:29:32.500659+0000","last_became_peered":"2024-03-06T07:29:32.500659+0000","last_unstale":"2024-03-06T07:29:34.498700+0000","last_undegraded":"2024-03-06T07:29:34.498700+0000","last_fullsized":"2024-03-06T07:29:34.498700+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:41.575664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498171+0000","last_change":"2024-03-06T07:29:33.493066+0000","last_active":"2024-03-06T07:29:34.498171+0000","last_peered":"2024-03-06T07:29:34.498171+0000","last_clean":"2024-03-06T07:29:34.498171+0000","last_became_active":"2024-03-06T07:29:33.492790+0000","last_became_peered":"2024-03-06T07:29:33.492790+0000","last_unstale":"2024-03-06T07:29:34.498171+0000","last_undegraded":"2024-03-06T07:29:34.498171+0000","last_fullsized":"2024-03-06T07:29:34.498171+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:53.065357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"209'1","reported_seq":15,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.731050+0000","last_change":"2024-03-06T07:29:32.495104+0000","last_active":"2024-03-06T07:29:34.731050+0000","last_peered":"2024-03-06T07:29:34.731050+0000","last_clean":"2024-03-06T07:29:34.731050+0000","last_became_active":"2024-03-06T07:29:32.494811+0000","last_became_peered":"2024-03-06T07:29:32.494811+0000","last_unstale":"2024-03-06T07:29:34.731050+0000","last_undegraded":"2024-03-06T07:29:34.731050+0000","last_fullsized":"2024-03-06T07:29:34.731050+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:06.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498746+0000","last_change":"2024-03-06T07:29:33.490306+0000","last_active":"2024-03-06T07:29:34.498746+0000","last_peered":"2024-03-06T07:29:34.498746+0000","last_clean":"2024-03-06T07:29:34.498746+0000","last_became_active":"2024-03-06T07:29:33.490025+0000","last_became_peered":"2024-03-06T07:29:33.490025+0000","last_unstale":"2024-03-06T07:29:34.498746+0000","last_undegraded":"2024-03-06T07:29:34.498746+0000","last_fullsized":"2024-03-06T07:29:34.498746+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:34.334299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498321+0000","last_change":"2024-03-06T07:29:32.492169+0000","last_active":"2024-03-06T07:29:34.498321+0000","last_peered":"2024-03-06T07:29:34.498321+0000","last_clean":"2024-03-06T07:29:34.498321+0000","last_became_active":"2024-03-06T07:29:32.491901+0000","last_became_peered":"2024-03-06T07:29:32.491901+0000","last_unstale":"2024-03-06T07:29:34.498321+0000","last_undegraded":"2024-03-06T07:29:34.498321+0000","last_fullsized":"2024-03-06T07:29:34.498321+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:42.565801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492681+0000","last_change":"2024-03-06T07:29:33.499330+0000","last_active":"2024-03-06T07:29:34.492681+0000","last_peered":"2024-03-06T07:29:34.492681+0000","last_clean":"2024-03-06T07:29:34.492681+0000","last_became_active":"2024-03-06T07:29:33.499052+0000","last_became_peered":"2024-03-06T07:29:33.499052+0000","last_unstale":"2024-03-06T07:29:34.492681+0000","last_undegraded":"2024-03-06T07:29:34.492681+0000","last_fullsized":"2024-03-06T07:29:34.492681+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:12.418860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"209'2","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.519500+0000","last_change":"2024-03-06T07:29:32.499908+0000","last_active":"2024-03-06T07:29:33.519500+0000","last_peered":"2024-03-06T07:29:33.519500+0000","last_clean":"2024-03-06T07:29:33.519500+0000","last_became_active":"2024-03-06T07:29:32.499060+0000","last_became_peered":"2024-03-06T07:29:32.499060+0000","last_unstale":"2024-03-06T07:29:33.519500+0000","last_undegraded":"2024-03-06T07:29:33.519500+0000","last_fullsized":"2024-03-06T07:29:33.519500+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:56.003634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498471+0000","last_change":"2024-03-06T07:29:33.493132+0000","last_active":"2024-03-06T07:29:34.498471+0000","last_peered":"2024-03-06T07:29:34.498471+0000","last_clean":"2024-03-06T07:29:34.498471+0000","last_became_active":"2024-03-06T07:29:33.492875+0000","last_became_peered":"2024-03-06T07:29:33.492875+0000","last_unstale":"2024-03-06T07:29:34.498471+0000","last_undegraded":"2024-03-06T07:29:34.498471+0000","last_fullsized":"2024-03-06T07:29:34.498471+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:24.862629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.497448+0000","last_change":"2024-03-06T07:29:32.498951+0000","last_active":"2024-03-06T07:29:33.497448+0000","last_peered":"2024-03-06T07:29:33.497448+0000","last_clean":"2024-03-06T07:29:33.497448+0000","last_became_active":"2024-03-06T07:29:32.498525+0000","last_became_peered":"2024-03-06T07:29:32.498525+0000","last_unstale":"2024-03-06T07:29:33.497448+0000","last_undegraded":"2024-03-06T07:29:33.497448+0000","last_fullsized":"2024-03-06T07:29:33.497448+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:47.734035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498728+0000","last_change":"2024-03-06T07:29:33.500763+0000","last_active":"2024-03-06T07:29:34.498728+0000","last_peered":"2024-03-06T07:29:34.498728+0000","last_clean":"2024-03-06T07:29:34.498728+0000","last_became_active":"2024-03-06T07:29:33.500480+0000","last_became_peered":"2024-03-06T07:29:33.500480+0000","last_unstale":"2024-03-06T07:29:34.498728+0000","last_undegraded":"2024-03-06T07:29:34.498728+0000","last_fullsized":"2024-03-06T07:29:34.498728+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:48.076680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.493194+0000","last_change":"2024-03-06T07:29:32.494915+0000","last_active":"2024-03-06T07:29:34.493194+0000","last_peered":"2024-03-06T07:29:34.493194+0000","last_clean":"2024-03-06T07:29:34.493194+0000","last_became_active":"2024-03-06T07:29:32.494509+0000","last_became_peered":"2024-03-06T07:29:32.494509+0000","last_unstale":"2024-03-06T07:29:34.493194+0000","last_undegraded":"2024-03-06T07:29:34.493194+0000","last_fullsized":"2024-03-06T07:29:34.493194+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:27:50.887225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492895+0000","last_change":"2024-03-06T07:29:33.504569+0000","last_active":"2024-03-06T07:29:34.492895+0000","last_peered":"2024-03-06T07:29:34.492895+0000","last_clean":"2024-03-06T07:29:34.492895+0000","last_became_active":"2024-03-06T07:29:33.504223+0000","last_became_peered":"2024-03-06T07:29:33.504223+0000","last_unstale":"2024-03-06T07:29:34.492895+0000","last_undegraded":"2024-03-06T07:29:34.492895+0000","last_fullsized":"2024-03-06T07:29:34.492895+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:10:16.950240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498166+0000","last_change":"2024-03-06T07:29:32.502501+0000","last_active":"2024-03-06T07:29:34.498166+0000","last_peered":"2024-03-06T07:29:34.498166+0000","last_clean":"2024-03-06T07:29:34.498166+0000","last_became_active":"2024-03-06T07:29:32.502230+0000","last_became_peered":"2024-03-06T07:29:32.502230+0000","last_unstale":"2024-03-06T07:29:34.498166+0000","last_undegraded":"2024-03-06T07:29:34.498166+0000","last_fullsized":"2024-03-06T07:29:34.498166+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:12.884040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503720+0000","last_change":"2024-03-06T07:29:33.500738+0000","last_active":"2024-03-06T07:29:34.503720+0000","last_peered":"2024-03-06T07:29:34.503720+0000","last_clean":"2024-03-06T07:29:34.503720+0000","last_became_active":"2024-03-06T07:29:33.500326+0000","last_became_peered":"2024-03-06T07:29:33.500326+0000","last_unstale":"2024-03-06T07:29:34.503720+0000","last_undegraded":"2024-03-06T07:29:34.503720+0000","last_fullsized":"2024-03-06T07:29:34.503720+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:57.089223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"209'2","reported_seq":16,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498574+0000","last_change":"2024-03-06T07:29:32.503535+0000","last_active":"2024-03-06T07:29:34.498574+0000","last_peered":"2024-03-06T07:29:34.498574+0000","last_clean":"2024-03-06T07:29:34.498574+0000","last_became_active":"2024-03-06T07:29:32.502326+0000","last_became_peered":"2024-03-06T07:29:32.502326+0000","last_unstale":"2024-03-06T07:29:34.498574+0000","last_undegraded":"2024-03-06T07:29:34.498574+0000","last_fullsized":"2024-03-06T07:29:34.498574+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:16.454137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503288+0000","last_change":"2024-03-06T07:29:33.504053+0000","last_active":"2024-03-06T07:29:34.503288+0000","last_peered":"2024-03-06T07:29:34.503288+0000","last_clean":"2024-03-06T07:29:34.503288+0000","last_became_active":"2024-03-06T07:29:33.503769+0000","last_became_peered":"2024-03-06T07:29:33.503769+0000","last_unstale":"2024-03-06T07:29:34.503288+0000","last_undegraded":"2024-03-06T07:29:34.503288+0000","last_fullsized":"2024-03-06T07:29:34.503288+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:08.572062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"210'1","reported_seq":15,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.500154+0000","last_change":"2024-03-06T07:29:32.502706+0000","last_active":"2024-03-06T07:29:34.500154+0000","last_peered":"2024-03-06T07:29:34.500154+0000","last_clean":"2024-03-06T07:29:34.500154+0000","last_became_active":"2024-03-06T07:29:32.500244+0000","last_became_peered":"2024-03-06T07:29:32.500244+0000","last_unstale":"2024-03-06T07:29:34.500154+0000","last_undegraded":"2024-03-06T07:29:34.500154+0000","last_fullsized":"2024-03-06T07:29:34.500154+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:41.033266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.730994+0000","last_change":"2024-03-06T07:29:33.504371+0000","last_active":"2024-03-06T07:29:34.730994+0000","last_peered":"2024-03-06T07:29:34.730994+0000","last_clean":"2024-03-06T07:29:34.730994+0000","last_became_active":"2024-03-06T07:29:33.504095+0000","last_became_peered":"2024-03-06T07:29:33.504095+0000","last_unstale":"2024-03-06T07:29:34.730994+0000","last_undegraded":"2024-03-06T07:29:34.730994+0000","last_fullsized":"2024-03-06T07:29:34.730994+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:07.335717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.731083+0000","last_change":"2024-03-06T07:29:32.494661+0000","last_active":"2024-03-06T07:29:34.731083+0000","last_peered":"2024-03-06T07:29:34.731083+0000","last_clean":"2024-03-06T07:29:34.731083+0000","last_became_active":"2024-03-06T07:29:32.494304+0000","last_became_peered":"2024-03-06T07:29:32.494304+0000","last_unstale":"2024-03-06T07:29:34.731083+0000","last_undegraded":"2024-03-06T07:29:34.731083+0000","last_fullsized":"2024-03-06T07:29:34.731083+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:24.314217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503310+0000","last_change":"2024-03-06T07:29:33.503156+0000","last_active":"2024-03-06T07:29:34.503310+0000","last_peered":"2024-03-06T07:29:34.503310+0000","last_clean":"2024-03-06T07:29:34.503310+0000","last_became_active":"2024-03-06T07:29:33.502102+0000","last_became_peered":"2024-03-06T07:29:33.502102+0000","last_unstale":"2024-03-06T07:29:34.503310+0000","last_undegraded":"2024-03-06T07:29:34.503310+0000","last_fullsized":"2024-03-06T07:29:34.503310+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:10.813328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.730831+0000","last_change":"2024-03-06T07:29:32.494458+0000","last_active":"2024-03-06T07:29:34.730831+0000","last_peered":"2024-03-06T07:29:34.730831+0000","last_clean":"2024-03-06T07:29:34.730831+0000","last_became_active":"2024-03-06T07:29:32.494155+0000","last_became_peered":"2024-03-06T07:29:32.494155+0000","last_unstale":"2024-03-06T07:29:34.730831+0000","last_undegraded":"2024-03-06T07:29:34.730831+0000","last_fullsized":"2024-03-06T07:29:34.730831+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:15.385041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498052+0000","last_change":"2024-03-06T07:29:33.502927+0000","last_active":"2024-03-06T07:29:34.498052+0000","last_peered":"2024-03-06T07:29:34.498052+0000","last_clean":"2024-03-06T07:29:34.498052+0000","last_became_active":"2024-03-06T07:29:33.502650+0000","last_became_peered":"2024-03-06T07:29:33.502650+0000","last_unstale":"2024-03-06T07:29:34.498052+0000","last_undegraded":"2024-03-06T07:29:34.498052+0000","last_fullsized":"2024-03-06T07:29:34.498052+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:05.240395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498151+0000","last_change":"2024-03-06T07:29:32.491371+0000","last_active":"2024-03-06T07:29:34.498151+0000","last_peered":"2024-03-06T07:29:34.498151+0000","last_clean":"2024-03-06T07:29:34.498151+0000","last_became_active":"2024-03-06T07:29:32.491095+0000","last_became_peered":"2024-03-06T07:29:32.491095+0000","last_unstale":"2024-03-06T07:29:34.498151+0000","last_undegraded":"2024-03-06T07:29:34.498151+0000","last_fullsized":"2024-03-06T07:29:34.498151+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:01.589252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.493204+0000","last_change":"2024-03-06T07:29:33.492699+0000","last_active":"2024-03-06T07:29:34.493204+0000","last_peered":"2024-03-06T07:29:34.493204+0000","last_clean":"2024-03-06T07:29:34.493204+0000","last_became_active":"2024-03-06T07:29:33.492412+0000","last_became_peered":"2024-03-06T07:29:33.492412+0000","last_unstale":"2024-03-06T07:29:34.493204+0000","last_undegraded":"2024-03-06T07:29:34.493204+0000","last_fullsized":"2024-03-06T07:29:34.493204+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:03.304587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.497965+0000","last_change":"2024-03-06T07:29:32.498429+0000","last_active":"2024-03-06T07:29:34.497965+0000","last_peered":"2024-03-06T07:29:34.497965+0000","last_clean":"2024-03-06T07:29:34.497965+0000","last_became_active":"2024-03-06T07:29:32.498167+0000","last_became_peered":"2024-03-06T07:29:32.498167+0000","last_unstale":"2024-03-06T07:29:34.497965+0000","last_undegraded":"2024-03-06T07:29:34.497965+0000","last_fullsized":"2024-03-06T07:29:34.497965+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:57.607537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492705+0000","last_change":"2024-03-06T07:29:33.492247+0000","last_active":"2024-03-06T07:29:34.492705+0000","last_peered":"2024-03-06T07:29:34.492705+0000","last_clean":"2024-03-06T07:29:34.492705+0000","last_became_active":"2024-03-06T07:29:33.491956+0000","last_became_peered":"2024-03-06T07:29:33.491956+0000","last_unstale":"2024-03-06T07:29:34.492705+0000","last_undegraded":"2024-03-06T07:29:34.492705+0000","last_fullsized":"2024-03-06T07:29:34.492705+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:59.194807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.730838+0000","last_change":"2024-03-06T07:29:32.495500+0000","last_active":"2024-03-06T07:29:34.730838+0000","last_peered":"2024-03-06T07:29:34.730838+0000","last_clean":"2024-03-06T07:29:34.730838+0000","last_became_active":"2024-03-06T07:29:32.495204+0000","last_became_peered":"2024-03-06T07:29:32.495204+0000","last_unstale":"2024-03-06T07:29:34.730838+0000","last_undegraded":"2024-03-06T07:29:34.730838+0000","last_fullsized":"2024-03-06T07:29:34.730838+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:35.780487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.502807+0000","last_change":"2024-03-06T07:29:33.501207+0000","last_active":"2024-03-06T07:29:33.502807+0000","last_peered":"2024-03-06T07:29:33.502807+0000","last_clean":"2024-03-06T07:29:33.502807+0000","last_became_active":"2024-03-06T07:29:33.500892+0000","last_became_peered":"2024-03-06T07:29:33.500892+0000","last_unstale":"2024-03-06T07:29:33.502807+0000","last_undegraded":"2024-03-06T07:29:33.502807+0000","last_fullsized":"2024-03-06T07:29:33.502807+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:16.390857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.497913+0000","last_change":"2024-03-06T07:29:32.494546+0000","last_active":"2024-03-06T07:29:34.497913+0000","last_peered":"2024-03-06T07:29:34.497913+0000","last_clean":"2024-03-06T07:29:34.497913+0000","last_became_active":"2024-03-06T07:29:32.494264+0000","last_became_peered":"2024-03-06T07:29:32.494264+0000","last_unstale":"2024-03-06T07:29:34.497913+0000","last_undegraded":"2024-03-06T07:29:34.497913+0000","last_fullsized":"2024-03-06T07:29:34.497913+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:53.858961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.499022+0000","last_change":"2024-03-06T07:29:33.495545+0000","last_active":"2024-03-06T07:29:34.499022+0000","last_peered":"2024-03-06T07:29:34.499022+0000","last_clean":"2024-03-06T07:29:34.499022+0000","last_became_active":"2024-03-06T07:29:33.495261+0000","last_became_peered":"2024-03-06T07:29:33.495261+0000","last_unstale":"2024-03-06T07:29:34.499022+0000","last_undegraded":"2024-03-06T07:29:34.499022+0000","last_fullsized":"2024-03-06T07:29:34.499022+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:06.335510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":536,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.502832+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:29:34.502832+0000","last_peered":"2024-03-06T07:29:34.502832+0000","last_clean":"2024-03-06T07:29:34.502832+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:29:34.502832+0000","last_undegraded":"2024-03-06T07:29:34.502832+0000","last_fullsized":"2024-03-06T07:29:34.502832+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574963,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33848,"kb_used_data":2664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337992,"statfs":{"total":96636764160,"available":96602103808,"internally_reserved":0,"allocated":2727936,"data_stored":1880134,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574962,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33780,"kb_used_data":2596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338060,"statfs":{"total":96636764160,"available":96602173440,"internally_reserved":0,"allocated":2658304,"data_stored":1289196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574962,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33780,"kb_used_data":2596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338060,"statfs":{"total":96636764160,"available":96602173440,"internally_reserved":0,"allocated":2658304,"data_stored":1289232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574963,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33788,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338052,"statfs":{"total":96636764160,"available":96602165248,"internally_reserved":0,"allocated":2666496,"data_stored":1290774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37956,"kb_used_data":2608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333884,"statfs":{"total":96636764160,"available":96597897216,"internally_reserved":0,"allocated":2670592,"data_stored":1295944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574962,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37952,"kb_used_data":2612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333888,"statfs":{"total":96636764160,"available":96597901312,"internally_reserved":0,"allocated":2674688,"data_stored":1291378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37944,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333896,"statfs":{"total":96636764160,"available":96597909504,"internally_reserved":0,"allocated":2662400,"data_stored":1289286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574963,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38004,"kb_used_data":2660,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333836,"statfs":{"total":96636764160,"available":96597848064,"internally_reserved":0,"allocated":2723840,"data_stored":1879600,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:37.497 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:37.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.632+0000 7f8914c0e640 1 Processor -- start 2024-03-06T07:29:37.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.632+0000 7f8914c0e640 1 -- start start 2024-03-06T07:29:37.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.633+0000 7f8914c0e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89100c3970 0x7f89100c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.633+0000 7f8914c0e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8910152840 0x7f89101553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.633+0000 7f8914c0e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89100c5e40 con 0x7f89100c60b0 2024-03-06T07:29:37.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.633+0000 7f8914c0e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89100c67f0 con 0x7f8910152840 2024-03-06T07:29:37.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.633+0000 7f8914c0e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89100c6a40 con 0x7f89100c3970 2024-03-06T07:29:37.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.633+0000 7f890ed9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8910152840 0x7f89101553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.633+0000 7f890ed9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8910152840 0x7f89101553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56088/0 (socket says 172.21.15.1:56088) 2024-03-06T07:29:37.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.633+0000 7f890ed9d640 1 -- 172.21.15.1:0/551583229 learned_addr learned my addr 172.21.15.1:0/551583229 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:37.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.633+0000 7f890e59c640 1 --2- 172.21.15.1:0/551583229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89100c3970 0x7f89100c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.633+0000 7f890ed9d640 1 -- 172.21.15.1:0/551583229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89100c3970 msgr2=0x7f89100c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.633+0000 7f890ed9d640 1 --2- 172.21.15.1:0/551583229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89100c3970 0x7f89100c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.633+0000 7f890ed9d640 1 -- 172.21.15.1:0/551583229 >> v1:172.21.15.98:6789/0 conn(0x7f89100c60b0 legacy=0x7f8910152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.633+0000 7f890ed9d640 1 -- 172.21.15.1:0/551583229 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89100c6d20 con 0x7f8910152840 2024-03-06T07:29:37.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.634+0000 7f890ed9d640 1 --2- 172.21.15.1:0/551583229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8910152840 0x7f89101553d0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f8900009e50 tx=0x7f890002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2b385a62575774 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.634+0000 7f890d59a640 1 -- 172.21.15.1:0/551583229 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890002cb00 con 0x7f8910152840 2024-03-06T07:29:37.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.634+0000 7f890d59a640 1 -- 172.21.15.1:0/551583229 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890002cca0 con 0x7f8910152840 2024-03-06T07:29:37.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.634+0000 7f890d59a640 1 -- 172.21.15.1:0/551583229 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8900034610 con 0x7f8910152840 2024-03-06T07:29:37.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.635+0000 7f8914c0e640 1 -- 172.21.15.1:0/551583229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8910152840 msgr2=0x7f89101553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.635+0000 7f8914c0e640 1 --2- 172.21.15.1:0/551583229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8910152840 0x7f89101553d0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f8900009e50 tx=0x7f890002bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:37.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.635+0000 7f8914c0e640 1 -- 172.21.15.1:0/551583229 shutdown_connections 2024-03-06T07:29:37.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.635+0000 7f8914c0e640 1 --2- 172.21.15.1:0/551583229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8910152840 0x7f89101553d0 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.635+0000 7f8914c0e640 1 --2- 172.21.15.1:0/551583229 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89100c3970 0x7f89100c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.635+0000 7f8914c0e640 1 -- 172.21.15.1:0/551583229 >> 172.21.15.1:0/551583229 conn(0x7f89100c88f0 msgr2=0x7f89100c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.635+0000 7f8914c0e640 1 -- 172.21.15.1:0/551583229 shutdown_connections 2024-03-06T07:29:37.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.635+0000 7f8914c0e640 1 -- 172.21.15.1:0/551583229 wait complete. 2024-03-06T07:29:37.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.636+0000 7f8914c0e640 1 Processor -- start 2024-03-06T07:29:37.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.636+0000 7f8914c0e640 1 -- start start 2024-03-06T07:29:37.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f8914c0e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89100c3970 0x7f89100a9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f8914c0e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89100c60b0 0x7f891006f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f8914c0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8910152840 0x7f8910149a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f8914c0e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89100c75c0 con 0x7f89100c3970 2024-03-06T07:29:37.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f8914c0e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89100c72e0 con 0x7f8910152840 2024-03-06T07:29:37.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f8914c0e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89100c3d50 con 0x7f89100c60b0 2024-03-06T07:29:37.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f890e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89100c3970 0x7f89100a9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f890e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89100c3970 0x7f89100a9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56102/0 (socket says 172.21.15.1:56102) 2024-03-06T07:29:37.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f890e59c640 1 -- 172.21.15.1:0/2980370667 learned_addr learned my addr 172.21.15.1:0/2980370667 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:37.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f890ed9d640 1 --2- 172.21.15.1:0/2980370667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8910152840 0x7f8910149a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f890dd9b640 1 --2- 172.21.15.1:0/2980370667 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89100c60b0 0x7f891006f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f890e59c640 1 -- 172.21.15.1:0/2980370667 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89100c60b0 msgr2=0x7f891006f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f890e59c640 1 --2- 172.21.15.1:0/2980370667 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89100c60b0 0x7f891006f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f890e59c640 1 -- 172.21.15.1:0/2980370667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8910152840 msgr2=0x7f8910149a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f890e59c640 1 --2- 172.21.15.1:0/2980370667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8910152840 0x7f8910149a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f890e59c640 1 -- 172.21.15.1:0/2980370667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89100ad860 con 0x7f89100c3970 2024-03-06T07:29:37.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f890ed9d640 1 --2- 172.21.15.1:0/2980370667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8910152840 0x7f8910149a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:37.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f890dd9b640 1 --2- 172.21.15.1:0/2980370667 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89100c60b0 0x7f891006f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:37.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.637+0000 7f890e59c640 1 --2- 172.21.15.1:0/2980370667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89100c3970 0x7f89100a9cc0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f890800d800 tx=0x7f890800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.638+0000 7f8906ffd640 1 -- 172.21.15.1:0/2980370667 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890800b6c0 con 0x7f89100c3970 2024-03-06T07:29:37.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.638+0000 7f8906ffd640 1 -- 172.21.15.1:0/2980370667 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8908019070 con 0x7f89100c3970 2024-03-06T07:29:37.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.638+0000 7f8914c0e640 1 -- 172.21.15.1:0/2980370667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f891014a690 con 0x7f89100c3970 2024-03-06T07:29:37.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.638+0000 7f8906ffd640 1 -- 172.21.15.1:0/2980370667 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890800ba80 con 0x7f89100c3970 2024-03-06T07:29:37.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.638+0000 7f8914c0e640 1 -- 172.21.15.1:0/2980370667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89100aa200 con 0x7f89100c3970 2024-03-06T07:29:37.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.639+0000 7f8914c0e640 1 -- 172.21.15.1:0/2980370667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89100686e0 con 0x7f89100c3970 2024-03-06T07:29:37.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.639+0000 7f8906ffd640 1 -- 172.21.15.1:0/2980370667 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f890800bc20 con 0x7f89100c3970 2024-03-06T07:29:37.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.640+0000 7f8906ffd640 1 --2- 172.21.15.1:0/2980370667 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88f006b4d0 0x7f88f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.640+0000 7f8906ffd640 1 -- 172.21.15.1:0/2980370667 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f890808a7e0 con 0x7f89100c3970 2024-03-06T07:29:37.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.640+0000 7f890dd9b640 1 --2- 172.21.15.1:0/2980370667 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88f006b4d0 0x7f88f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.640+0000 7f890dd9b640 1 --2- 172.21.15.1:0/2980370667 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88f006b4d0 0x7f88f006d990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f8910070420 tx=0x7f88f8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.648+0000 7f8906ffd640 1 -- 172.21.15.1:0/2980370667 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8908059560 con 0x7f89100c3970 2024-03-06T07:29:37.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.827+0000 7f8914c0e640 1 -- 172.21.15.1:0/2980370667 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f89100ab0c0 con 0x7f88f006b4d0 2024-03-06T07:29:37.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.834+0000 7f8906ffd640 1 -- 172.21.15.1:0/2980370667 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252559 (secure 0 0 0) 0x7f89100ab0c0 con 0x7f88f006b4d0 2024-03-06T07:29:37.838 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:37.839 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:29:37.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.836+0000 7f8914c0e640 1 -- 172.21.15.1:0/2980370667 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88f006b4d0 msgr2=0x7f88f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.836+0000 7f8914c0e640 1 --2- 172.21.15.1:0/2980370667 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88f006b4d0 0x7f88f006d990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f8910070420 tx=0x7f88f8009040 comp rx=0 tx=0).stop 2024-03-06T07:29:37.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.836+0000 7f8914c0e640 1 -- 172.21.15.1:0/2980370667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89100c3970 msgr2=0x7f89100a9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.836+0000 7f8914c0e640 1 --2- 172.21.15.1:0/2980370667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89100c3970 0x7f89100a9cc0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f890800d800 tx=0x7f890800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.837+0000 7f8914c0e640 1 -- 172.21.15.1:0/2980370667 shutdown_connections 2024-03-06T07:29:37.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.837+0000 7f8914c0e640 1 --2- 172.21.15.1:0/2980370667 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f88f006b4d0 0x7f88f006d990 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.837+0000 7f8914c0e640 1 --2- 172.21.15.1:0/2980370667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8910152840 0x7f8910149a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.837+0000 7f8914c0e640 1 --2- 172.21.15.1:0/2980370667 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89100c60b0 0x7f891006f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.837+0000 7f8914c0e640 1 --2- 172.21.15.1:0/2980370667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89100c3970 0x7f89100a9cc0 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.837+0000 7f8914c0e640 1 -- 172.21.15.1:0/2980370667 >> 172.21.15.1:0/2980370667 conn(0x7f89100c88f0 msgr2=0x7f891006cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.837+0000 7f8914c0e640 1 -- 172.21.15.1:0/2980370667 shutdown_connections 2024-03-06T07:29:37.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.837+0000 7f8914c0e640 1 -- 172.21.15.1:0/2980370667 wait complete. 2024-03-06T07:29:37.849 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2024-03-06T07:29:36.823790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":287052,"kb_used_data":20940,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754687668,"statfs":{"total":773094113280,"available":772800172032,"internally_reserved":0,"allocated":21442560,"data_stored":11505544,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.348596"},"pg_stats":[{"pgid":"34.3f","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498568+0000","last_change":"2024-03-06T07:29:33.504880+0000","last_active":"2024-03-06T07:29:34.498568+0000","last_peered":"2024-03-06T07:29:34.498568+0000","last_clean":"2024-03-06T07:29:34.498568+0000","last_became_active":"2024-03-06T07:29:33.504634+0000","last_became_peered":"2024-03-06T07:29:33.504634+0000","last_unstale":"2024-03-06T07:29:34.498568+0000","last_undegraded":"2024-03-06T07:29:34.498568+0000","last_fullsized":"2024-03-06T07:29:34.498568+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:43.888641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492837+0000","last_change":"2024-03-06T07:29:33.504328+0000","last_active":"2024-03-06T07:29:34.492837+0000","last_peered":"2024-03-06T07:29:34.492837+0000","last_clean":"2024-03-06T07:29:34.492837+0000","last_became_active":"2024-03-06T07:29:33.503833+0000","last_became_peered":"2024-03-06T07:29:33.503833+0000","last_unstale":"2024-03-06T07:29:34.492837+0000","last_undegraded":"2024-03-06T07:29:34.492837+0000","last_fullsized":"2024-03-06T07:29:34.492837+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:01.422873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.730958+0000","last_change":"2024-03-06T07:29:33.499751+0000","last_active":"2024-03-06T07:29:34.730958+0000","last_peered":"2024-03-06T07:29:34.730958+0000","last_clean":"2024-03-06T07:29:34.730958+0000","last_became_active":"2024-03-06T07:29:33.499448+0000","last_became_peered":"2024-03-06T07:29:33.499448+0000","last_unstale":"2024-03-06T07:29:34.730958+0000","last_undegraded":"2024-03-06T07:29:34.730958+0000","last_fullsized":"2024-03-06T07:29:34.730958+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:28:50.989301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498063+0000","last_change":"2024-03-06T07:29:33.501683+0000","last_active":"2024-03-06T07:29:34.498063+0000","last_peered":"2024-03-06T07:29:34.498063+0000","last_clean":"2024-03-06T07:29:34.498063+0000","last_became_active":"2024-03-06T07:29:33.501372+0000","last_became_peered":"2024-03-06T07:29:33.501372+0000","last_unstale":"2024-03-06T07:29:34.498063+0000","last_undegraded":"2024-03-06T07:29:34.498063+0000","last_fullsized":"2024-03-06T07:29:34.498063+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:46.878541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.505869+0000","last_change":"2024-03-06T07:29:33.504873+0000","last_active":"2024-03-06T07:29:33.505869+0000","last_peered":"2024-03-06T07:29:33.505869+0000","last_clean":"2024-03-06T07:29:33.505869+0000","last_became_active":"2024-03-06T07:29:33.503995+0000","last_became_peered":"2024-03-06T07:29:33.503995+0000","last_unstale":"2024-03-06T07:29:33.505869+0000","last_undegraded":"2024-03-06T07:29:33.505869+0000","last_fullsized":"2024-03-06T07:29:33.505869+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:34.590590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503585+0000","last_change":"2024-03-06T07:29:33.497986+0000","last_active":"2024-03-06T07:29:34.503585+0000","last_peered":"2024-03-06T07:29:34.503585+0000","last_clean":"2024-03-06T07:29:34.503585+0000","last_became_active":"2024-03-06T07:29:33.497636+0000","last_became_peered":"2024-03-06T07:29:33.497636+0000","last_unstale":"2024-03-06T07:29:34.503585+0000","last_undegraded":"2024-03-06T07:29:34.503585+0000","last_fullsized":"2024-03-06T07:29:34.503585+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:22.227727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.730698+0000","last_change":"2024-03-06T07:29:33.495174+0000","last_active":"2024-03-06T07:29:34.730698+0000","last_peered":"2024-03-06T07:29:34.730698+0000","last_clean":"2024-03-06T07:29:34.730698+0000","last_became_active":"2024-03-06T07:29:33.494882+0000","last_became_peered":"2024-03-06T07:29:33.494882+0000","last_unstale":"2024-03-06T07:29:34.730698+0000","last_undegraded":"2024-03-06T07:29:34.730698+0000","last_fullsized":"2024-03-06T07:29:34.730698+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:43.888641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.497882+0000","last_change":"2024-03-06T07:29:33.498188+0000","last_active":"2024-03-06T07:29:34.497882+0000","last_peered":"2024-03-06T07:29:34.497882+0000","last_clean":"2024-03-06T07:29:34.497882+0000","last_became_active":"2024-03-06T07:29:33.497889+0000","last_became_peered":"2024-03-06T07:29:33.497889+0000","last_unstale":"2024-03-06T07:29:34.497882+0000","last_undegraded":"2024-03-06T07:29:34.497882+0000","last_fullsized":"2024-03-06T07:29:34.497882+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:30.609804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503370+0000","last_change":"2024-03-06T07:29:33.508445+0000","last_active":"2024-03-06T07:29:34.503370+0000","last_peered":"2024-03-06T07:29:34.503370+0000","last_clean":"2024-03-06T07:29:34.503370+0000","last_became_active":"2024-03-06T07:29:33.508141+0000","last_became_peered":"2024-03-06T07:29:33.508141+0000","last_unstale":"2024-03-06T07:29:34.503370+0000","last_undegraded":"2024-03-06T07:29:34.503370+0000","last_fullsized":"2024-03-06T07:29:34.503370+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:48.739851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.505135+0000","last_change":"2024-03-06T07:29:33.503858+0000","last_active":"2024-03-06T07:29:33.505135+0000","last_peered":"2024-03-06T07:29:33.505135+0000","last_clean":"2024-03-06T07:29:33.505135+0000","last_became_active":"2024-03-06T07:29:33.503516+0000","last_became_peered":"2024-03-06T07:29:33.503516+0000","last_unstale":"2024-03-06T07:29:33.505135+0000","last_undegraded":"2024-03-06T07:29:33.505135+0000","last_fullsized":"2024-03-06T07:29:33.505135+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:36.786303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498572+0000","last_change":"2024-03-06T07:29:33.493073+0000","last_active":"2024-03-06T07:29:34.498572+0000","last_peered":"2024-03-06T07:29:34.498572+0000","last_clean":"2024-03-06T07:29:34.498572+0000","last_became_active":"2024-03-06T07:29:33.492379+0000","last_became_peered":"2024-03-06T07:29:33.492379+0000","last_unstale":"2024-03-06T07:29:34.498572+0000","last_undegraded":"2024-03-06T07:29:34.498572+0000","last_fullsized":"2024-03-06T07:29:34.498572+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:16.464883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503601+0000","last_change":"2024-03-06T07:29:33.500635+0000","last_active":"2024-03-06T07:29:34.503601+0000","last_peered":"2024-03-06T07:29:34.503601+0000","last_clean":"2024-03-06T07:29:34.503601+0000","last_became_active":"2024-03-06T07:29:33.500153+0000","last_became_peered":"2024-03-06T07:29:33.500153+0000","last_unstale":"2024-03-06T07:29:34.503601+0000","last_undegraded":"2024-03-06T07:29:34.503601+0000","last_fullsized":"2024-03-06T07:29:34.503601+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:32.730513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.499002+0000","last_change":"2024-03-06T07:29:33.497422+0000","last_active":"2024-03-06T07:29:34.499002+0000","last_peered":"2024-03-06T07:29:34.499002+0000","last_clean":"2024-03-06T07:29:34.499002+0000","last_became_active":"2024-03-06T07:29:33.495981+0000","last_became_peered":"2024-03-06T07:29:33.495981+0000","last_unstale":"2024-03-06T07:29:34.499002+0000","last_undegraded":"2024-03-06T07:29:34.499002+0000","last_fullsized":"2024-03-06T07:29:34.499002+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:39.398132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492944+0000","last_change":"2024-03-06T07:29:33.490558+0000","last_active":"2024-03-06T07:29:34.492944+0000","last_peered":"2024-03-06T07:29:34.492944+0000","last_clean":"2024-03-06T07:29:34.492944+0000","last_became_active":"2024-03-06T07:29:33.490269+0000","last_became_peered":"2024-03-06T07:29:33.490269+0000","last_unstale":"2024-03-06T07:29:34.492944+0000","last_undegraded":"2024-03-06T07:29:34.492944+0000","last_fullsized":"2024-03-06T07:29:34.492944+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:40.447291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492566+0000","last_change":"2024-03-06T07:29:33.491298+0000","last_active":"2024-03-06T07:29:34.492566+0000","last_peered":"2024-03-06T07:29:34.492566+0000","last_clean":"2024-03-06T07:29:34.492566+0000","last_became_active":"2024-03-06T07:29:33.491006+0000","last_became_peered":"2024-03-06T07:29:33.491006+0000","last_unstale":"2024-03-06T07:29:34.492566+0000","last_undegraded":"2024-03-06T07:29:34.492566+0000","last_fullsized":"2024-03-06T07:29:34.492566+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:48.214015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.503549+0000","last_change":"2024-03-06T07:29:33.502406+0000","last_active":"2024-03-06T07:29:33.503549+0000","last_peered":"2024-03-06T07:29:33.503549+0000","last_clean":"2024-03-06T07:29:33.503549+0000","last_became_active":"2024-03-06T07:29:33.501839+0000","last_became_peered":"2024-03-06T07:29:33.501839+0000","last_unstale":"2024-03-06T07:29:33.503549+0000","last_undegraded":"2024-03-06T07:29:33.503549+0000","last_fullsized":"2024-03-06T07:29:33.503549+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:25.320033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.502705+0000","last_change":"2024-03-06T07:29:33.500303+0000","last_active":"2024-03-06T07:29:33.502705+0000","last_peered":"2024-03-06T07:29:33.502705+0000","last_clean":"2024-03-06T07:29:33.502705+0000","last_became_active":"2024-03-06T07:29:33.500040+0000","last_became_peered":"2024-03-06T07:29:33.500040+0000","last_unstale":"2024-03-06T07:29:33.502705+0000","last_undegraded":"2024-03-06T07:29:33.502705+0000","last_fullsized":"2024-03-06T07:29:33.502705+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:49:08.979523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498591+0000","last_change":"2024-03-06T07:29:33.514706+0000","last_active":"2024-03-06T07:29:34.498591+0000","last_peered":"2024-03-06T07:29:34.498591+0000","last_clean":"2024-03-06T07:29:34.498591+0000","last_became_active":"2024-03-06T07:29:33.514280+0000","last_became_peered":"2024-03-06T07:29:33.514280+0000","last_unstale":"2024-03-06T07:29:34.498591+0000","last_undegraded":"2024-03-06T07:29:34.498591+0000","last_fullsized":"2024-03-06T07:29:34.498591+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:45.717490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.500166+0000","last_change":"2024-03-06T07:29:33.499503+0000","last_active":"2024-03-06T07:29:33.500166+0000","last_peered":"2024-03-06T07:29:33.500166+0000","last_clean":"2024-03-06T07:29:33.500166+0000","last_became_active":"2024-03-06T07:29:33.499241+0000","last_became_peered":"2024-03-06T07:29:33.499241+0000","last_unstale":"2024-03-06T07:29:33.500166+0000","last_undegraded":"2024-03-06T07:29:33.500166+0000","last_fullsized":"2024-03-06T07:29:33.500166+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:41.473077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503608+0000","last_change":"2024-03-06T07:29:33.500940+0000","last_active":"2024-03-06T07:29:34.503608+0000","last_peered":"2024-03-06T07:29:34.503608+0000","last_clean":"2024-03-06T07:29:34.503608+0000","last_became_active":"2024-03-06T07:29:33.500455+0000","last_became_peered":"2024-03-06T07:29:33.500455+0000","last_unstale":"2024-03-06T07:29:34.503608+0000","last_undegraded":"2024-03-06T07:29:34.503608+0000","last_fullsized":"2024-03-06T07:29:34.503608+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:57.009450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.503175+0000","last_change":"2024-03-06T07:29:33.502644+0000","last_active":"2024-03-06T07:29:33.503175+0000","last_peered":"2024-03-06T07:29:33.503175+0000","last_clean":"2024-03-06T07:29:33.503175+0000","last_became_active":"2024-03-06T07:29:33.502373+0000","last_became_peered":"2024-03-06T07:29:33.502373+0000","last_unstale":"2024-03-06T07:29:33.503175+0000","last_undegraded":"2024-03-06T07:29:33.503175+0000","last_fullsized":"2024-03-06T07:29:33.503175+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:57.089223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498315+0000","last_change":"2024-03-06T07:29:33.504232+0000","last_active":"2024-03-06T07:29:34.498315+0000","last_peered":"2024-03-06T07:29:34.498315+0000","last_clean":"2024-03-06T07:29:34.498315+0000","last_became_active":"2024-03-06T07:29:33.503928+0000","last_became_peered":"2024-03-06T07:29:33.503928+0000","last_unstale":"2024-03-06T07:29:34.498315+0000","last_undegraded":"2024-03-06T07:29:34.498315+0000","last_fullsized":"2024-03-06T07:29:34.498315+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:40.445535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492918+0000","last_change":"2024-03-06T07:29:33.501325+0000","last_active":"2024-03-06T07:29:34.492918+0000","last_peered":"2024-03-06T07:29:34.492918+0000","last_clean":"2024-03-06T07:29:34.492918+0000","last_became_active":"2024-03-06T07:29:33.501008+0000","last_became_peered":"2024-03-06T07:29:33.501008+0000","last_unstale":"2024-03-06T07:29:34.492918+0000","last_undegraded":"2024-03-06T07:29:34.492918+0000","last_fullsized":"2024-03-06T07:29:34.492918+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:43.154947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498168+0000","last_change":"2024-03-06T07:29:33.495976+0000","last_active":"2024-03-06T07:29:34.498168+0000","last_peered":"2024-03-06T07:29:34.498168+0000","last_clean":"2024-03-06T07:29:34.498168+0000","last_became_active":"2024-03-06T07:29:33.495713+0000","last_became_peered":"2024-03-06T07:29:33.495713+0000","last_unstale":"2024-03-06T07:29:34.498168+0000","last_undegraded":"2024-03-06T07:29:34.498168+0000","last_fullsized":"2024-03-06T07:29:34.498168+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:02.813993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.730998+0000","last_change":"2024-03-06T07:29:33.505724+0000","last_active":"2024-03-06T07:29:34.730998+0000","last_peered":"2024-03-06T07:29:34.730998+0000","last_clean":"2024-03-06T07:29:34.730998+0000","last_became_active":"2024-03-06T07:29:33.505435+0000","last_became_peered":"2024-03-06T07:29:33.505435+0000","last_unstale":"2024-03-06T07:29:34.730998+0000","last_undegraded":"2024-03-06T07:29:34.730998+0000","last_fullsized":"2024-03-06T07:29:34.730998+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:34.334299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492982+0000","last_change":"2024-03-06T07:29:33.508808+0000","last_active":"2024-03-06T07:29:34.492982+0000","last_peered":"2024-03-06T07:29:34.492982+0000","last_clean":"2024-03-06T07:29:34.492982+0000","last_became_active":"2024-03-06T07:29:33.508513+0000","last_became_peered":"2024-03-06T07:29:33.508513+0000","last_unstale":"2024-03-06T07:29:34.492982+0000","last_undegraded":"2024-03-06T07:29:34.492982+0000","last_fullsized":"2024-03-06T07:29:34.492982+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:43.704508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498675+0000","last_change":"2024-03-06T07:29:33.504877+0000","last_active":"2024-03-06T07:29:34.498675+0000","last_peered":"2024-03-06T07:29:34.498675+0000","last_clean":"2024-03-06T07:29:34.498675+0000","last_became_active":"2024-03-06T07:29:33.504543+0000","last_became_peered":"2024-03-06T07:29:33.504543+0000","last_unstale":"2024-03-06T07:29:34.498675+0000","last_undegraded":"2024-03-06T07:29:34.498675+0000","last_fullsized":"2024-03-06T07:29:34.498675+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:28:50.989301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498856+0000","last_change":"2024-03-06T07:29:33.491056+0000","last_active":"2024-03-06T07:29:34.498856+0000","last_peered":"2024-03-06T07:29:34.498856+0000","last_clean":"2024-03-06T07:29:34.498856+0000","last_became_active":"2024-03-06T07:29:33.490790+0000","last_became_peered":"2024-03-06T07:29:33.490790+0000","last_unstale":"2024-03-06T07:29:34.498856+0000","last_undegraded":"2024-03-06T07:29:34.498856+0000","last_fullsized":"2024-03-06T07:29:34.498856+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:00.202804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503110+0000","last_change":"2024-03-06T07:29:33.501613+0000","last_active":"2024-03-06T07:29:34.503110+0000","last_peered":"2024-03-06T07:29:34.503110+0000","last_clean":"2024-03-06T07:29:34.503110+0000","last_became_active":"2024-03-06T07:29:33.501336+0000","last_became_peered":"2024-03-06T07:29:33.501336+0000","last_unstale":"2024-03-06T07:29:34.503110+0000","last_undegraded":"2024-03-06T07:29:34.503110+0000","last_fullsized":"2024-03-06T07:29:34.503110+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:56.351464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.493066+0000","last_change":"2024-03-06T07:29:33.491492+0000","last_active":"2024-03-06T07:29:34.493066+0000","last_peered":"2024-03-06T07:29:34.493066+0000","last_clean":"2024-03-06T07:29:34.493066+0000","last_became_active":"2024-03-06T07:29:33.491207+0000","last_became_peered":"2024-03-06T07:29:33.491207+0000","last_unstale":"2024-03-06T07:29:34.493066+0000","last_undegraded":"2024-03-06T07:29:34.493066+0000","last_fullsized":"2024-03-06T07:29:34.493066+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:01.392304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492508+0000","last_change":"2024-03-06T07:29:33.504893+0000","last_active":"2024-03-06T07:29:34.492508+0000","last_peered":"2024-03-06T07:29:34.492508+0000","last_clean":"2024-03-06T07:29:34.492508+0000","last_became_active":"2024-03-06T07:29:33.503619+0000","last_became_peered":"2024-03-06T07:29:33.503619+0000","last_unstale":"2024-03-06T07:29:34.492508+0000","last_undegraded":"2024-03-06T07:29:34.492508+0000","last_fullsized":"2024-03-06T07:29:34.492508+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:27.541752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.503857+0000","last_change":"2024-03-06T07:29:33.502459+0000","last_active":"2024-03-06T07:29:33.503857+0000","last_peered":"2024-03-06T07:29:33.503857+0000","last_clean":"2024-03-06T07:29:33.503857+0000","last_became_active":"2024-03-06T07:29:33.501793+0000","last_became_peered":"2024-03-06T07:29:33.501793+0000","last_unstale":"2024-03-06T07:29:33.503857+0000","last_undegraded":"2024-03-06T07:29:33.503857+0000","last_fullsized":"2024-03-06T07:29:33.503857+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:07.774336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"209'4","reported_seq":18,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498630+0000","last_change":"2024-03-06T07:29:32.492127+0000","last_active":"2024-03-06T07:29:34.498630+0000","last_peered":"2024-03-06T07:29:34.498630+0000","last_clean":"2024-03-06T07:29:34.498630+0000","last_became_active":"2024-03-06T07:29:32.491791+0000","last_became_peered":"2024-03-06T07:29:32.491791+0000","last_unstale":"2024-03-06T07:29:34.498630+0000","last_undegraded":"2024-03-06T07:29:34.498630+0000","last_fullsized":"2024-03-06T07:29:34.498630+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:33.692640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498294+0000","last_change":"2024-03-06T07:29:33.497731+0000","last_active":"2024-03-06T07:29:34.498294+0000","last_peered":"2024-03-06T07:29:34.498294+0000","last_clean":"2024-03-06T07:29:34.498294+0000","last_became_active":"2024-03-06T07:29:33.497465+0000","last_became_peered":"2024-03-06T07:29:33.497465+0000","last_unstale":"2024-03-06T07:29:34.498294+0000","last_undegraded":"2024-03-06T07:29:34.498294+0000","last_fullsized":"2024-03-06T07:29:34.498294+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:29.677861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"209'1","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.511149+0000","last_change":"2024-03-06T07:29:32.494084+0000","last_active":"2024-03-06T07:29:33.511149+0000","last_peered":"2024-03-06T07:29:33.511149+0000","last_clean":"2024-03-06T07:29:33.511149+0000","last_became_active":"2024-03-06T07:29:32.493760+0000","last_became_peered":"2024-03-06T07:29:32.493760+0000","last_unstale":"2024-03-06T07:29:33.511149+0000","last_undegraded":"2024-03-06T07:29:33.511149+0000","last_fullsized":"2024-03-06T07:29:33.511149+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:38.243046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498678+0000","last_change":"2024-03-06T07:29:33.497948+0000","last_active":"2024-03-06T07:29:34.498678+0000","last_peered":"2024-03-06T07:29:34.498678+0000","last_clean":"2024-03-06T07:29:34.498678+0000","last_became_active":"2024-03-06T07:29:33.497690+0000","last_became_peered":"2024-03-06T07:29:33.497690+0000","last_unstale":"2024-03-06T07:29:34.498678+0000","last_undegraded":"2024-03-06T07:29:34.498678+0000","last_fullsized":"2024-03-06T07:29:34.498678+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:35.917308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"209'2","reported_seq":16,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498794+0000","last_change":"2024-03-06T07:29:32.495714+0000","last_active":"2024-03-06T07:29:34.498794+0000","last_peered":"2024-03-06T07:29:34.498794+0000","last_clean":"2024-03-06T07:29:34.498794+0000","last_became_active":"2024-03-06T07:29:32.495440+0000","last_became_peered":"2024-03-06T07:29:32.495440+0000","last_unstale":"2024-03-06T07:29:34.498794+0000","last_undegraded":"2024-03-06T07:29:34.498794+0000","last_fullsized":"2024-03-06T07:29:34.498794+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:36:27.145630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492843+0000","last_change":"2024-03-06T07:29:33.498622+0000","last_active":"2024-03-06T07:29:34.492843+0000","last_peered":"2024-03-06T07:29:34.492843+0000","last_clean":"2024-03-06T07:29:34.492843+0000","last_became_active":"2024-03-06T07:29:33.498332+0000","last_became_peered":"2024-03-06T07:29:33.498332+0000","last_unstale":"2024-03-06T07:29:34.492843+0000","last_undegraded":"2024-03-06T07:29:34.492843+0000","last_fullsized":"2024-03-06T07:29:34.492843+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:09.257168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"209'2","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.513181+0000","last_change":"2024-03-06T07:29:32.500977+0000","last_active":"2024-03-06T07:29:33.513181+0000","last_peered":"2024-03-06T07:29:33.513181+0000","last_clean":"2024-03-06T07:29:33.513181+0000","last_became_active":"2024-03-06T07:29:32.500763+0000","last_became_peered":"2024-03-06T07:29:32.500763+0000","last_unstale":"2024-03-06T07:29:33.513181+0000","last_undegraded":"2024-03-06T07:29:33.513181+0000","last_fullsized":"2024-03-06T07:29:33.513181+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:57.866450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492657+0000","last_change":"2024-03-06T07:29:33.495369+0000","last_active":"2024-03-06T07:29:34.492657+0000","last_peered":"2024-03-06T07:29:34.492657+0000","last_clean":"2024-03-06T07:29:34.492657+0000","last_became_active":"2024-03-06T07:29:33.493449+0000","last_became_peered":"2024-03-06T07:29:33.493449+0000","last_unstale":"2024-03-06T07:29:34.492657+0000","last_undegraded":"2024-03-06T07:29:34.492657+0000","last_fullsized":"2024-03-06T07:29:34.492657+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:46.088883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"209'1","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.517523+0000","last_change":"2024-03-06T07:29:32.500405+0000","last_active":"2024-03-06T07:29:33.517523+0000","last_peered":"2024-03-06T07:29:33.517523+0000","last_clean":"2024-03-06T07:29:33.517523+0000","last_became_active":"2024-03-06T07:29:32.499655+0000","last_became_peered":"2024-03-06T07:29:32.499655+0000","last_unstale":"2024-03-06T07:29:33.517523+0000","last_undegraded":"2024-03-06T07:29:33.517523+0000","last_fullsized":"2024-03-06T07:29:33.517523+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:21.221911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.493072+0000","last_change":"2024-03-06T07:29:33.504314+0000","last_active":"2024-03-06T07:29:34.493072+0000","last_peered":"2024-03-06T07:29:34.493072+0000","last_clean":"2024-03-06T07:29:34.493072+0000","last_became_active":"2024-03-06T07:29:33.503874+0000","last_became_peered":"2024-03-06T07:29:33.503874+0000","last_unstale":"2024-03-06T07:29:34.493072+0000","last_undegraded":"2024-03-06T07:29:34.493072+0000","last_fullsized":"2024-03-06T07:29:34.493072+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:55.554029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"209'1","reported_seq":15,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492953+0000","last_change":"2024-03-06T07:29:32.498552+0000","last_active":"2024-03-06T07:29:34.492953+0000","last_peered":"2024-03-06T07:29:34.492953+0000","last_clean":"2024-03-06T07:29:34.492953+0000","last_became_active":"2024-03-06T07:29:32.498288+0000","last_became_peered":"2024-03-06T07:29:32.498288+0000","last_unstale":"2024-03-06T07:29:34.492953+0000","last_undegraded":"2024-03-06T07:29:34.492953+0000","last_fullsized":"2024-03-06T07:29:34.492953+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:25.005877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.730871+0000","last_change":"2024-03-06T07:29:33.500643+0000","last_active":"2024-03-06T07:29:34.730871+0000","last_peered":"2024-03-06T07:29:34.730871+0000","last_clean":"2024-03-06T07:29:34.730871+0000","last_became_active":"2024-03-06T07:29:33.500365+0000","last_became_peered":"2024-03-06T07:29:33.500365+0000","last_unstale":"2024-03-06T07:29:34.730871+0000","last_undegraded":"2024-03-06T07:29:34.730871+0000","last_fullsized":"2024-03-06T07:29:34.730871+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:16.464883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.491543+0000","last_change":"2024-03-06T07:29:32.495493+0000","last_active":"2024-03-06T07:29:33.491543+0000","last_peered":"2024-03-06T07:29:33.491543+0000","last_clean":"2024-03-06T07:29:33.491543+0000","last_became_active":"2024-03-06T07:29:32.495160+0000","last_became_peered":"2024-03-06T07:29:32.495160+0000","last_unstale":"2024-03-06T07:29:33.491543+0000","last_undegraded":"2024-03-06T07:29:33.491543+0000","last_fullsized":"2024-03-06T07:29:33.491543+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:43.429625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.499105+0000","last_change":"2024-03-06T07:29:33.494554+0000","last_active":"2024-03-06T07:29:34.499105+0000","last_peered":"2024-03-06T07:29:34.499105+0000","last_clean":"2024-03-06T07:29:34.499105+0000","last_became_active":"2024-03-06T07:29:33.494188+0000","last_became_peered":"2024-03-06T07:29:33.494188+0000","last_unstale":"2024-03-06T07:29:34.499105+0000","last_undegraded":"2024-03-06T07:29:34.499105+0000","last_fullsized":"2024-03-06T07:29:34.499105+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:27.613945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.498186+0000","last_change":"2024-03-06T07:29:32.495708+0000","last_active":"2024-03-06T07:29:33.498186+0000","last_peered":"2024-03-06T07:29:33.498186+0000","last_clean":"2024-03-06T07:29:33.498186+0000","last_became_active":"2024-03-06T07:29:32.495449+0000","last_became_peered":"2024-03-06T07:29:32.495449+0000","last_unstale":"2024-03-06T07:29:33.498186+0000","last_undegraded":"2024-03-06T07:29:33.498186+0000","last_fullsized":"2024-03-06T07:29:33.498186+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:37.488554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498218+0000","last_change":"2024-03-06T07:29:33.500309+0000","last_active":"2024-03-06T07:29:34.498218+0000","last_peered":"2024-03-06T07:29:34.498218+0000","last_clean":"2024-03-06T07:29:34.498218+0000","last_became_active":"2024-03-06T07:29:33.499997+0000","last_became_peered":"2024-03-06T07:29:33.499997+0000","last_unstale":"2024-03-06T07:29:34.498218+0000","last_undegraded":"2024-03-06T07:29:34.498218+0000","last_fullsized":"2024-03-06T07:29:34.498218+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:47.739042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498484+0000","last_change":"2024-03-06T07:29:32.504424+0000","last_active":"2024-03-06T07:29:34.498484+0000","last_peered":"2024-03-06T07:29:34.498484+0000","last_clean":"2024-03-06T07:29:34.498484+0000","last_became_active":"2024-03-06T07:29:32.504156+0000","last_became_peered":"2024-03-06T07:29:32.504156+0000","last_unstale":"2024-03-06T07:29:34.498484+0000","last_undegraded":"2024-03-06T07:29:34.498484+0000","last_fullsized":"2024-03-06T07:29:34.498484+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:47:07.441888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498429+0000","last_change":"2024-03-06T07:29:33.496329+0000","last_active":"2024-03-06T07:29:34.498429+0000","last_peered":"2024-03-06T07:29:34.498429+0000","last_clean":"2024-03-06T07:29:34.498429+0000","last_became_active":"2024-03-06T07:29:33.496027+0000","last_became_peered":"2024-03-06T07:29:33.496027+0000","last_unstale":"2024-03-06T07:29:34.498429+0000","last_undegraded":"2024-03-06T07:29:34.498429+0000","last_fullsized":"2024-03-06T07:29:34.498429+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:54.522857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.492901+0000","last_change":"2024-03-06T07:29:32.496371+0000","last_active":"2024-03-06T07:29:33.492901+0000","last_peered":"2024-03-06T07:29:33.492901+0000","last_clean":"2024-03-06T07:29:33.492901+0000","last_became_active":"2024-03-06T07:29:32.495936+0000","last_became_peered":"2024-03-06T07:29:32.495936+0000","last_unstale":"2024-03-06T07:29:33.492901+0000","last_undegraded":"2024-03-06T07:29:33.492901+0000","last_fullsized":"2024-03-06T07:29:33.492901+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:05.820201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503561+0000","last_change":"2024-03-06T07:29:33.501319+0000","last_active":"2024-03-06T07:29:34.503561+0000","last_peered":"2024-03-06T07:29:34.503561+0000","last_clean":"2024-03-06T07:29:34.503561+0000","last_became_active":"2024-03-06T07:29:33.501050+0000","last_became_peered":"2024-03-06T07:29:33.501050+0000","last_unstale":"2024-03-06T07:29:34.503561+0000","last_undegraded":"2024-03-06T07:29:34.503561+0000","last_fullsized":"2024-03-06T07:29:34.503561+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:38.494370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"209'3","reported_seq":17,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498390+0000","last_change":"2024-03-06T07:29:32.501594+0000","last_active":"2024-03-06T07:29:34.498390+0000","last_peered":"2024-03-06T07:29:34.498390+0000","last_clean":"2024-03-06T07:29:34.498390+0000","last_became_active":"2024-03-06T07:29:32.501307+0000","last_became_peered":"2024-03-06T07:29:32.501307+0000","last_unstale":"2024-03-06T07:29:34.498390+0000","last_undegraded":"2024-03-06T07:29:34.498390+0000","last_fullsized":"2024-03-06T07:29:34.498390+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:36.439471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498909+0000","last_change":"2024-03-06T07:29:33.505956+0000","last_active":"2024-03-06T07:29:34.498909+0000","last_peered":"2024-03-06T07:29:34.498909+0000","last_clean":"2024-03-06T07:29:34.498909+0000","last_became_active":"2024-03-06T07:29:33.505409+0000","last_became_peered":"2024-03-06T07:29:33.505409+0000","last_unstale":"2024-03-06T07:29:34.498909+0000","last_undegraded":"2024-03-06T07:29:34.498909+0000","last_fullsized":"2024-03-06T07:29:34.498909+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:03.871710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"209'6","reported_seq":20,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498231+0000","last_change":"2024-03-06T07:29:32.496604+0000","last_active":"2024-03-06T07:29:34.498231+0000","last_peered":"2024-03-06T07:29:34.498231+0000","last_clean":"2024-03-06T07:29:34.498231+0000","last_became_active":"2024-03-06T07:29:32.496313+0000","last_became_peered":"2024-03-06T07:29:32.496313+0000","last_unstale":"2024-03-06T07:29:34.498231+0000","last_undegraded":"2024-03-06T07:29:34.498231+0000","last_fullsized":"2024-03-06T07:29:34.498231+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:10.982908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.507007+0000","last_change":"2024-03-06T07:29:33.506274+0000","last_active":"2024-03-06T07:29:33.507007+0000","last_peered":"2024-03-06T07:29:33.507007+0000","last_clean":"2024-03-06T07:29:33.507007+0000","last_became_active":"2024-03-06T07:29:33.505951+0000","last_became_peered":"2024-03-06T07:29:33.505951+0000","last_unstale":"2024-03-06T07:29:33.507007+0000","last_undegraded":"2024-03-06T07:29:33.507007+0000","last_fullsized":"2024-03-06T07:29:33.507007+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:28.486711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"209'2","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.517763+0000","last_change":"2024-03-06T07:29:32.500150+0000","last_active":"2024-03-06T07:29:33.517763+0000","last_peered":"2024-03-06T07:29:33.517763+0000","last_clean":"2024-03-06T07:29:33.517763+0000","last_became_active":"2024-03-06T07:29:32.499360+0000","last_became_peered":"2024-03-06T07:29:32.499360+0000","last_unstale":"2024-03-06T07:29:33.517763+0000","last_undegraded":"2024-03-06T07:29:33.517763+0000","last_fullsized":"2024-03-06T07:29:33.517763+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:31.724697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498206+0000","last_change":"2024-03-06T07:29:33.489574+0000","last_active":"2024-03-06T07:29:34.498206+0000","last_peered":"2024-03-06T07:29:34.498206+0000","last_clean":"2024-03-06T07:29:34.498206+0000","last_became_active":"2024-03-06T07:29:33.489319+0000","last_became_peered":"2024-03-06T07:29:33.489319+0000","last_unstale":"2024-03-06T07:29:34.498206+0000","last_undegraded":"2024-03-06T07:29:34.498206+0000","last_fullsized":"2024-03-06T07:29:34.498206+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:49.153262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498814+0000","last_change":"2024-03-06T07:29:32.500847+0000","last_active":"2024-03-06T07:29:34.498814+0000","last_peered":"2024-03-06T07:29:34.498814+0000","last_clean":"2024-03-06T07:29:34.498814+0000","last_became_active":"2024-03-06T07:29:32.500574+0000","last_became_peered":"2024-03-06T07:29:32.500574+0000","last_unstale":"2024-03-06T07:29:34.498814+0000","last_undegraded":"2024-03-06T07:29:34.498814+0000","last_fullsized":"2024-03-06T07:29:34.498814+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:21.386627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.509034+0000","last_change":"2024-03-06T07:29:33.508243+0000","last_active":"2024-03-06T07:29:33.509034+0000","last_peered":"2024-03-06T07:29:33.509034+0000","last_clean":"2024-03-06T07:29:33.509034+0000","last_became_active":"2024-03-06T07:29:33.507981+0000","last_became_peered":"2024-03-06T07:29:33.507981+0000","last_unstale":"2024-03-06T07:29:33.509034+0000","last_undegraded":"2024-03-06T07:29:33.509034+0000","last_fullsized":"2024-03-06T07:29:33.509034+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:32.910447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498467+0000","last_change":"2024-03-06T07:29:32.501736+0000","last_active":"2024-03-06T07:29:34.498467+0000","last_peered":"2024-03-06T07:29:34.498467+0000","last_clean":"2024-03-06T07:29:34.498467+0000","last_became_active":"2024-03-06T07:29:32.501444+0000","last_became_peered":"2024-03-06T07:29:32.501444+0000","last_unstale":"2024-03-06T07:29:34.498467+0000","last_undegraded":"2024-03-06T07:29:34.498467+0000","last_fullsized":"2024-03-06T07:29:34.498467+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:52.615490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.505890+0000","last_change":"2024-03-06T07:29:33.505096+0000","last_active":"2024-03-06T07:29:33.505890+0000","last_peered":"2024-03-06T07:29:33.505890+0000","last_clean":"2024-03-06T07:29:33.505890+0000","last_became_active":"2024-03-06T07:29:33.504601+0000","last_became_peered":"2024-03-06T07:29:33.504601+0000","last_unstale":"2024-03-06T07:29:33.505890+0000","last_undegraded":"2024-03-06T07:29:33.505890+0000","last_fullsized":"2024-03-06T07:29:33.505890+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:24.738032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"209'8","reported_seq":22,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498106+0000","last_change":"2024-03-06T07:29:32.504142+0000","last_active":"2024-03-06T07:29:34.498106+0000","last_peered":"2024-03-06T07:29:34.498106+0000","last_clean":"2024-03-06T07:29:34.498106+0000","last_became_active":"2024-03-06T07:29:32.503854+0000","last_became_peered":"2024-03-06T07:29:32.503854+0000","last_unstale":"2024-03-06T07:29:34.498106+0000","last_undegraded":"2024-03-06T07:29:34.498106+0000","last_fullsized":"2024-03-06T07:29:34.498106+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:36.875697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498505+0000","last_change":"2024-03-06T07:29:33.497491+0000","last_active":"2024-03-06T07:29:34.498505+0000","last_peered":"2024-03-06T07:29:34.498505+0000","last_clean":"2024-03-06T07:29:34.498505+0000","last_became_active":"2024-03-06T07:29:33.497200+0000","last_became_peered":"2024-03-06T07:29:33.497200+0000","last_unstale":"2024-03-06T07:29:34.498505+0000","last_undegraded":"2024-03-06T07:29:34.498505+0000","last_fullsized":"2024-03-06T07:29:34.498505+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:02.396510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"209'1","reported_seq":15,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503413+0000","last_change":"2024-03-06T07:29:32.494708+0000","last_active":"2024-03-06T07:29:34.503413+0000","last_peered":"2024-03-06T07:29:34.503413+0000","last_clean":"2024-03-06T07:29:34.503413+0000","last_became_active":"2024-03-06T07:29:32.494070+0000","last_became_peered":"2024-03-06T07:29:32.494070+0000","last_unstale":"2024-03-06T07:29:34.503413+0000","last_undegraded":"2024-03-06T07:29:34.503413+0000","last_fullsized":"2024-03-06T07:29:34.503413+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:49.223819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498731+0000","last_change":"2024-03-06T07:29:33.492932+0000","last_active":"2024-03-06T07:29:34.498731+0000","last_peered":"2024-03-06T07:29:34.498731+0000","last_clean":"2024-03-06T07:29:34.498731+0000","last_became_active":"2024-03-06T07:29:33.492467+0000","last_became_peered":"2024-03-06T07:29:33.492467+0000","last_unstale":"2024-03-06T07:29:34.498731+0000","last_undegraded":"2024-03-06T07:29:34.498731+0000","last_fullsized":"2024-03-06T07:29:34.498731+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:07.335717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"209'2","reported_seq":16,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498700+0000","last_change":"2024-03-06T07:29:32.500950+0000","last_active":"2024-03-06T07:29:34.498700+0000","last_peered":"2024-03-06T07:29:34.498700+0000","last_clean":"2024-03-06T07:29:34.498700+0000","last_became_active":"2024-03-06T07:29:32.500659+0000","last_became_peered":"2024-03-06T07:29:32.500659+0000","last_unstale":"2024-03-06T07:29:34.498700+0000","last_undegraded":"2024-03-06T07:29:34.498700+0000","last_fullsized":"2024-03-06T07:29:34.498700+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:41.575664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498171+0000","last_change":"2024-03-06T07:29:33.493066+0000","last_active":"2024-03-06T07:29:34.498171+0000","last_peered":"2024-03-06T07:29:34.498171+0000","last_clean":"2024-03-06T07:29:34.498171+0000","last_became_active":"2024-03-06T07:29:33.492790+0000","last_became_peered":"2024-03-06T07:29:33.492790+0000","last_unstale":"2024-03-06T07:29:34.498171+0000","last_undegraded":"2024-03-06T07:29:34.498171+0000","last_fullsized":"2024-03-06T07:29:34.498171+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:53.065357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"209'1","reported_seq":15,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.731050+0000","last_change":"2024-03-06T07:29:32.495104+0000","last_active":"2024-03-06T07:29:34.731050+0000","last_peered":"2024-03-06T07:29:34.731050+0000","last_clean":"2024-03-06T07:29:34.731050+0000","last_became_active":"2024-03-06T07:29:32.494811+0000","last_became_peered":"2024-03-06T07:29:32.494811+0000","last_unstale":"2024-03-06T07:29:34.731050+0000","last_undegraded":"2024-03-06T07:29:34.731050+0000","last_fullsized":"2024-03-06T07:29:34.731050+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:06.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498746+0000","last_change":"2024-03-06T07:29:33.490306+0000","last_active":"2024-03-06T07:29:34.498746+0000","last_peered":"2024-03-06T07:29:34.498746+0000","last_clean":"2024-03-06T07:29:34.498746+0000","last_became_active":"2024-03-06T07:29:33.490025+0000","last_became_peered":"2024-03-06T07:29:33.490025+0000","last_unstale":"2024-03-06T07:29:34.498746+0000","last_undegraded":"2024-03-06T07:29:34.498746+0000","last_fullsized":"2024-03-06T07:29:34.498746+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:34.334299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498321+0000","last_change":"2024-03-06T07:29:32.492169+0000","last_active":"2024-03-06T07:29:34.498321+0000","last_peered":"2024-03-06T07:29:34.498321+0000","last_clean":"2024-03-06T07:29:34.498321+0000","last_became_active":"2024-03-06T07:29:32.491901+0000","last_became_peered":"2024-03-06T07:29:32.491901+0000","last_unstale":"2024-03-06T07:29:34.498321+0000","last_undegraded":"2024-03-06T07:29:34.498321+0000","last_fullsized":"2024-03-06T07:29:34.498321+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:42.565801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492681+0000","last_change":"2024-03-06T07:29:33.499330+0000","last_active":"2024-03-06T07:29:34.492681+0000","last_peered":"2024-03-06T07:29:34.492681+0000","last_clean":"2024-03-06T07:29:34.492681+0000","last_became_active":"2024-03-06T07:29:33.499052+0000","last_became_peered":"2024-03-06T07:29:33.499052+0000","last_unstale":"2024-03-06T07:29:34.492681+0000","last_undegraded":"2024-03-06T07:29:34.492681+0000","last_fullsized":"2024-03-06T07:29:34.492681+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:12.418860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"209'2","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.519500+0000","last_change":"2024-03-06T07:29:32.499908+0000","last_active":"2024-03-06T07:29:33.519500+0000","last_peered":"2024-03-06T07:29:33.519500+0000","last_clean":"2024-03-06T07:29:33.519500+0000","last_became_active":"2024-03-06T07:29:32.499060+0000","last_became_peered":"2024-03-06T07:29:32.499060+0000","last_unstale":"2024-03-06T07:29:33.519500+0000","last_undegraded":"2024-03-06T07:29:33.519500+0000","last_fullsized":"2024-03-06T07:29:33.519500+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:56.003634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498471+0000","last_change":"2024-03-06T07:29:33.493132+0000","last_active":"2024-03-06T07:29:34.498471+0000","last_peered":"2024-03-06T07:29:34.498471+0000","last_clean":"2024-03-06T07:29:34.498471+0000","last_became_active":"2024-03-06T07:29:33.492875+0000","last_became_peered":"2024-03-06T07:29:33.492875+0000","last_unstale":"2024-03-06T07:29:34.498471+0000","last_undegraded":"2024-03-06T07:29:34.498471+0000","last_fullsized":"2024-03-06T07:29:34.498471+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:24.862629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.497448+0000","last_change":"2024-03-06T07:29:32.498951+0000","last_active":"2024-03-06T07:29:33.497448+0000","last_peered":"2024-03-06T07:29:33.497448+0000","last_clean":"2024-03-06T07:29:33.497448+0000","last_became_active":"2024-03-06T07:29:32.498525+0000","last_became_peered":"2024-03-06T07:29:32.498525+0000","last_unstale":"2024-03-06T07:29:33.497448+0000","last_undegraded":"2024-03-06T07:29:33.497448+0000","last_fullsized":"2024-03-06T07:29:33.497448+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:47.734035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498728+0000","last_change":"2024-03-06T07:29:33.500763+0000","last_active":"2024-03-06T07:29:34.498728+0000","last_peered":"2024-03-06T07:29:34.498728+0000","last_clean":"2024-03-06T07:29:34.498728+0000","last_became_active":"2024-03-06T07:29:33.500480+0000","last_became_peered":"2024-03-06T07:29:33.500480+0000","last_unstale":"2024-03-06T07:29:34.498728+0000","last_undegraded":"2024-03-06T07:29:34.498728+0000","last_fullsized":"2024-03-06T07:29:34.498728+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:48.076680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.493194+0000","last_change":"2024-03-06T07:29:32.494915+0000","last_active":"2024-03-06T07:29:34.493194+0000","last_peered":"2024-03-06T07:29:34.493194+0000","last_clean":"2024-03-06T07:29:34.493194+0000","last_became_active":"2024-03-06T07:29:32.494509+0000","last_became_peered":"2024-03-06T07:29:32.494509+0000","last_unstale":"2024-03-06T07:29:34.493194+0000","last_undegraded":"2024-03-06T07:29:34.493194+0000","last_fullsized":"2024-03-06T07:29:34.493194+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:27:50.887225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492895+0000","last_change":"2024-03-06T07:29:33.504569+0000","last_active":"2024-03-06T07:29:34.492895+0000","last_peered":"2024-03-06T07:29:34.492895+0000","last_clean":"2024-03-06T07:29:34.492895+0000","last_became_active":"2024-03-06T07:29:33.504223+0000","last_became_peered":"2024-03-06T07:29:33.504223+0000","last_unstale":"2024-03-06T07:29:34.492895+0000","last_undegraded":"2024-03-06T07:29:34.492895+0000","last_fullsized":"2024-03-06T07:29:34.492895+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:10:16.950240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498166+0000","last_change":"2024-03-06T07:29:32.502501+0000","last_active":"2024-03-06T07:29:34.498166+0000","last_peered":"2024-03-06T07:29:34.498166+0000","last_clean":"2024-03-06T07:29:34.498166+0000","last_became_active":"2024-03-06T07:29:32.502230+0000","last_became_peered":"2024-03-06T07:29:32.502230+0000","last_unstale":"2024-03-06T07:29:34.498166+0000","last_undegraded":"2024-03-06T07:29:34.498166+0000","last_fullsized":"2024-03-06T07:29:34.498166+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:12.884040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503720+0000","last_change":"2024-03-06T07:29:33.500738+0000","last_active":"2024-03-06T07:29:34.503720+0000","last_peered":"2024-03-06T07:29:34.503720+0000","last_clean":"2024-03-06T07:29:34.503720+0000","last_became_active":"2024-03-06T07:29:33.500326+0000","last_became_peered":"2024-03-06T07:29:33.500326+0000","last_unstale":"2024-03-06T07:29:34.503720+0000","last_undegraded":"2024-03-06T07:29:34.503720+0000","last_fullsized":"2024-03-06T07:29:34.503720+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:57.089223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"209'2","reported_seq":16,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498574+0000","last_change":"2024-03-06T07:29:32.503535+0000","last_active":"2024-03-06T07:29:34.498574+0000","last_peered":"2024-03-06T07:29:34.498574+0000","last_clean":"2024-03-06T07:29:34.498574+0000","last_became_active":"2024-03-06T07:29:32.502326+0000","last_became_peered":"2024-03-06T07:29:32.502326+0000","last_unstale":"2024-03-06T07:29:34.498574+0000","last_undegraded":"2024-03-06T07:29:34.498574+0000","last_fullsized":"2024-03-06T07:29:34.498574+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:16.454137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503288+0000","last_change":"2024-03-06T07:29:33.504053+0000","last_active":"2024-03-06T07:29:34.503288+0000","last_peered":"2024-03-06T07:29:34.503288+0000","last_clean":"2024-03-06T07:29:34.503288+0000","last_became_active":"2024-03-06T07:29:33.503769+0000","last_became_peered":"2024-03-06T07:29:33.503769+0000","last_unstale":"2024-03-06T07:29:34.503288+0000","last_undegraded":"2024-03-06T07:29:34.503288+0000","last_fullsized":"2024-03-06T07:29:34.503288+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:08.572062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"210'1","reported_seq":15,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.500154+0000","last_change":"2024-03-06T07:29:32.502706+0000","last_active":"2024-03-06T07:29:34.500154+0000","last_peered":"2024-03-06T07:29:34.500154+0000","last_clean":"2024-03-06T07:29:34.500154+0000","last_became_active":"2024-03-06T07:29:32.500244+0000","last_became_peered":"2024-03-06T07:29:32.500244+0000","last_unstale":"2024-03-06T07:29:34.500154+0000","last_undegraded":"2024-03-06T07:29:34.500154+0000","last_fullsized":"2024-03-06T07:29:34.500154+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:41.033266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.730994+0000","last_change":"2024-03-06T07:29:33.504371+0000","last_active":"2024-03-06T07:29:34.730994+0000","last_peered":"2024-03-06T07:29:34.730994+0000","last_clean":"2024-03-06T07:29:34.730994+0000","last_became_active":"2024-03-06T07:29:33.504095+0000","last_became_peered":"2024-03-06T07:29:33.504095+0000","last_unstale":"2024-03-06T07:29:34.730994+0000","last_undegraded":"2024-03-06T07:29:34.730994+0000","last_fullsized":"2024-03-06T07:29:34.730994+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:07.335717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.731083+0000","last_change":"2024-03-06T07:29:32.494661+0000","last_active":"2024-03-06T07:29:34.731083+0000","last_peered":"2024-03-06T07:29:34.731083+0000","last_clean":"2024-03-06T07:29:34.731083+0000","last_became_active":"2024-03-06T07:29:32.494304+0000","last_became_peered":"2024-03-06T07:29:32.494304+0000","last_unstale":"2024-03-06T07:29:34.731083+0000","last_undegraded":"2024-03-06T07:29:34.731083+0000","last_fullsized":"2024-03-06T07:29:34.731083+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:24.314217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.503310+0000","last_change":"2024-03-06T07:29:33.503156+0000","last_active":"2024-03-06T07:29:34.503310+0000","last_peered":"2024-03-06T07:29:34.503310+0000","last_clean":"2024-03-06T07:29:34.503310+0000","last_became_active":"2024-03-06T07:29:33.502102+0000","last_became_peered":"2024-03-06T07:29:33.502102+0000","last_unstale":"2024-03-06T07:29:34.503310+0000","last_undegraded":"2024-03-06T07:29:34.503310+0000","last_fullsized":"2024-03-06T07:29:34.503310+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:10.813328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.730831+0000","last_change":"2024-03-06T07:29:32.494458+0000","last_active":"2024-03-06T07:29:34.730831+0000","last_peered":"2024-03-06T07:29:34.730831+0000","last_clean":"2024-03-06T07:29:34.730831+0000","last_became_active":"2024-03-06T07:29:32.494155+0000","last_became_peered":"2024-03-06T07:29:32.494155+0000","last_unstale":"2024-03-06T07:29:34.730831+0000","last_undegraded":"2024-03-06T07:29:34.730831+0000","last_fullsized":"2024-03-06T07:29:34.730831+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:15.385041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498052+0000","last_change":"2024-03-06T07:29:33.502927+0000","last_active":"2024-03-06T07:29:34.498052+0000","last_peered":"2024-03-06T07:29:34.498052+0000","last_clean":"2024-03-06T07:29:34.498052+0000","last_became_active":"2024-03-06T07:29:33.502650+0000","last_became_peered":"2024-03-06T07:29:33.502650+0000","last_unstale":"2024-03-06T07:29:34.498052+0000","last_undegraded":"2024-03-06T07:29:34.498052+0000","last_fullsized":"2024-03-06T07:29:34.498052+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:05.240395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.498151+0000","last_change":"2024-03-06T07:29:32.491371+0000","last_active":"2024-03-06T07:29:34.498151+0000","last_peered":"2024-03-06T07:29:34.498151+0000","last_clean":"2024-03-06T07:29:34.498151+0000","last_became_active":"2024-03-06T07:29:32.491095+0000","last_became_peered":"2024-03-06T07:29:32.491095+0000","last_unstale":"2024-03-06T07:29:34.498151+0000","last_undegraded":"2024-03-06T07:29:34.498151+0000","last_fullsized":"2024-03-06T07:29:34.498151+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:01.589252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.493204+0000","last_change":"2024-03-06T07:29:33.492699+0000","last_active":"2024-03-06T07:29:34.493204+0000","last_peered":"2024-03-06T07:29:34.493204+0000","last_clean":"2024-03-06T07:29:34.493204+0000","last_became_active":"2024-03-06T07:29:33.492412+0000","last_became_peered":"2024-03-06T07:29:33.492412+0000","last_unstale":"2024-03-06T07:29:34.493204+0000","last_undegraded":"2024-03-06T07:29:34.493204+0000","last_fullsized":"2024-03-06T07:29:34.493204+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:03.304587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.497965+0000","last_change":"2024-03-06T07:29:32.498429+0000","last_active":"2024-03-06T07:29:34.497965+0000","last_peered":"2024-03-06T07:29:34.497965+0000","last_clean":"2024-03-06T07:29:34.497965+0000","last_became_active":"2024-03-06T07:29:32.498167+0000","last_became_peered":"2024-03-06T07:29:32.498167+0000","last_unstale":"2024-03-06T07:29:34.497965+0000","last_undegraded":"2024-03-06T07:29:34.497965+0000","last_fullsized":"2024-03-06T07:29:34.497965+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:57.607537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.492705+0000","last_change":"2024-03-06T07:29:33.492247+0000","last_active":"2024-03-06T07:29:34.492705+0000","last_peered":"2024-03-06T07:29:34.492705+0000","last_clean":"2024-03-06T07:29:34.492705+0000","last_became_active":"2024-03-06T07:29:33.491956+0000","last_became_peered":"2024-03-06T07:29:33.491956+0000","last_unstale":"2024-03-06T07:29:34.492705+0000","last_undegraded":"2024-03-06T07:29:34.492705+0000","last_fullsized":"2024-03-06T07:29:34.492705+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:59.194807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.730838+0000","last_change":"2024-03-06T07:29:32.495500+0000","last_active":"2024-03-06T07:29:34.730838+0000","last_peered":"2024-03-06T07:29:34.730838+0000","last_clean":"2024-03-06T07:29:34.730838+0000","last_became_active":"2024-03-06T07:29:32.495204+0000","last_became_peered":"2024-03-06T07:29:32.495204+0000","last_unstale":"2024-03-06T07:29:34.730838+0000","last_undegraded":"2024-03-06T07:29:34.730838+0000","last_fullsized":"2024-03-06T07:29:34.730838+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:35.780487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:29:33.502807+0000","last_change":"2024-03-06T07:29:33.501207+0000","last_active":"2024-03-06T07:29:33.502807+0000","last_peered":"2024-03-06T07:29:33.502807+0000","last_clean":"2024-03-06T07:29:33.502807+0000","last_became_active":"2024-03-06T07:29:33.500892+0000","last_became_peered":"2024-03-06T07:29:33.500892+0000","last_unstale":"2024-03-06T07:29:33.502807+0000","last_undegraded":"2024-03-06T07:29:33.502807+0000","last_fullsized":"2024-03-06T07:29:33.502807+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:16.390857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.497913+0000","last_change":"2024-03-06T07:29:32.494546+0000","last_active":"2024-03-06T07:29:34.497913+0000","last_peered":"2024-03-06T07:29:34.497913+0000","last_clean":"2024-03-06T07:29:34.497913+0000","last_became_active":"2024-03-06T07:29:32.494264+0000","last_became_peered":"2024-03-06T07:29:32.494264+0000","last_unstale":"2024-03-06T07:29:34.497913+0000","last_undegraded":"2024-03-06T07:29:34.497913+0000","last_fullsized":"2024-03-06T07:29:34.497913+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:31.468406+0000","last_clean_scrub_stamp":"2024-03-06T07:29:31.468406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:53.858961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.499022+0000","last_change":"2024-03-06T07:29:33.495545+0000","last_active":"2024-03-06T07:29:34.499022+0000","last_peered":"2024-03-06T07:29:34.499022+0000","last_clean":"2024-03-06T07:29:34.499022+0000","last_became_active":"2024-03-06T07:29:33.495261+0000","last_became_peered":"2024-03-06T07:29:33.495261+0000","last_unstale":"2024-03-06T07:29:34.499022+0000","last_undegraded":"2024-03-06T07:29:34.499022+0000","last_fullsized":"2024-03-06T07:29:34.499022+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:32.474222+0000","last_clean_scrub_stamp":"2024-03-06T07:29:32.474222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:06.335510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":536,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-03-06T07:29:34.502832+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:29:34.502832+0000","last_peered":"2024-03-06T07:29:34.502832+0000","last_clean":"2024-03-06T07:29:34.502832+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:29:34.502832+0000","last_undegraded":"2024-03-06T07:29:34.502832+0000","last_fullsized":"2024-03-06T07:29:34.502832+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574963,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33848,"kb_used_data":2664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337992,"statfs":{"total":96636764160,"available":96602103808,"internally_reserved":0,"allocated":2727936,"data_stored":1880134,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574962,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33780,"kb_used_data":2596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338060,"statfs":{"total":96636764160,"available":96602173440,"internally_reserved":0,"allocated":2658304,"data_stored":1289196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574962,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33780,"kb_used_data":2596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338060,"statfs":{"total":96636764160,"available":96602173440,"internally_reserved":0,"allocated":2658304,"data_stored":1289232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574963,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33788,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338052,"statfs":{"total":96636764160,"available":96602165248,"internally_reserved":0,"allocated":2666496,"data_stored":1290774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37956,"kb_used_data":2608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333884,"statfs":{"total":96636764160,"available":96597897216,"internally_reserved":0,"allocated":2670592,"data_stored":1295944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574962,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37952,"kb_used_data":2612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333888,"statfs":{"total":96636764160,"available":96597901312,"internally_reserved":0,"allocated":2674688,"data_stored":1291378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37944,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333896,"statfs":{"total":96636764160,"available":96597909504,"internally_reserved":0,"allocated":2662400,"data_stored":1289286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574963,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38004,"kb_used_data":2660,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333836,"statfs":{"total":96636764160,"available":96597848064,"internally_reserved":0,"allocated":2723840,"data_stored":1879600,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:37.853 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:29:37.853 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:29:38.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.999+0000 7ff294c9d640 1 Processor -- start 2024-03-06T07:29:38.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:37.999+0000 7ff294c9d640 1 -- start start 2024-03-06T07:29:38.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.000+0000 7ff294c9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff29006f640 0x7ff2900c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.000+0000 7ff294c9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2900c14c0 0x7ff2901558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.000+0000 7ff294c9d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2900c31f0 con 0x7ff2900c05f0 2024-03-06T07:29:38.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.000+0000 7ff294c9d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff29006fe90 con 0x7ff29006f640 2024-03-06T07:29:38.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.000+0000 7ff294c9d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2900c2fc0 con 0x7ff2900c14c0 2024-03-06T07:29:38.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.001+0000 7ff28ed9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2900c14c0 0x7ff2901558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.001+0000 7ff28dd9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff29006f640 0x7ff2900c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.001+0000 7ff28ed9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2900c14c0 0x7ff2901558c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:47132/0 (socket says 172.21.15.1:47132) 2024-03-06T07:29:38.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.001+0000 7ff28e59c640 1 --1- 172.21.15.1:0/3369649217 >> v1:172.21.15.98:6789/0 conn(0x7ff2900c05f0 0x7ff2901551b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:57704/0 (socket says 172.21.15.1:57704) 2024-03-06T07:29:38.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.001+0000 7ff28ed9d640 1 -- 172.21.15.1:0/3369649217 learned_addr learned my addr 172.21.15.1:0/3369649217 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.001+0000 7ff28dd9b640 1 -- 172.21.15.1:0/3369649217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2900c14c0 msgr2=0x7ff2901558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.001+0000 7ff28dd9b640 1 --2- 172.21.15.1:0/3369649217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2900c14c0 0x7ff2901558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.001+0000 7ff28dd9b640 1 -- 172.21.15.1:0/3369649217 >> v1:172.21.15.98:6789/0 conn(0x7ff2900c05f0 legacy=0x7ff2901551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.001+0000 7ff28dd9b640 1 -- 172.21.15.1:0/3369649217 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff290155e00 con 0x7ff29006f640 2024-03-06T07:29:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.001+0000 7ff28d59a640 1 -- 172.21.15.1:0/3369649217 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025565661 0 0) 0x7ff2900c31f0 con 0x7ff2900c05f0 2024-03-06T07:29:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.001+0000 7ff28ed9d640 1 --2- 172.21.15.1:0/3369649217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2900c14c0 0x7ff2901558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.002+0000 7ff28dd9b640 1 --2- 172.21.15.1:0/3369649217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff29006f640 0x7ff2900c09d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff2780097a0 tx=0x7ff27802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf5a3dbb9db02fe1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.002+0000 7ff28d59a640 1 -- 172.21.15.1:0/3369649217 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff278035020 con 0x7ff29006f640 2024-03-06T07:29:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.002+0000 7ff28d59a640 1 -- 172.21.15.1:0/3369649217 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff278031830 con 0x7ff29006f640 2024-03-06T07:29:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.002+0000 7ff28d59a640 1 -- 172.21.15.1:0/3369649217 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff278031c00 con 0x7ff29006f640 2024-03-06T07:29:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.002+0000 7ff294c9d640 1 -- 172.21.15.1:0/3369649217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff29006f640 msgr2=0x7ff2900c09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.002+0000 7ff294c9d640 1 --2- 172.21.15.1:0/3369649217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff29006f640 0x7ff2900c09d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff2780097a0 tx=0x7ff27802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.003+0000 7ff294c9d640 1 -- 172.21.15.1:0/3369649217 shutdown_connections 2024-03-06T07:29:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.003+0000 7ff294c9d640 1 --2- 172.21.15.1:0/3369649217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2900c14c0 0x7ff2901558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.003+0000 7ff294c9d640 1 --2- 172.21.15.1:0/3369649217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff29006f640 0x7ff2900c09d0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.003+0000 7ff294c9d640 1 -- 172.21.15.1:0/3369649217 >> 172.21.15.1:0/3369649217 conn(0x7ff2900c3cc0 msgr2=0x7ff2900c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.003+0000 7ff294c9d640 1 -- 172.21.15.1:0/3369649217 shutdown_connections 2024-03-06T07:29:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.003+0000 7ff294c9d640 1 -- 172.21.15.1:0/3369649217 wait complete. 2024-03-06T07:29:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.004+0000 7ff294c9d640 1 Processor -- start 2024-03-06T07:29:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.004+0000 7ff294c9d640 1 -- start start 2024-03-06T07:29:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.004+0000 7ff294c9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff29006f640 0x7ff290148a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.004+0000 7ff294c9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2900c05f0 0x7ff29013f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.004+0000 7ff294c9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2900c14c0 0x7ff29013ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.004+0000 7ff294c9d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2900c3770 con 0x7ff2900c14c0 2024-03-06T07:29:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.004+0000 7ff294c9d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2900c3490 con 0x7ff29006f640 2024-03-06T07:29:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.004+0000 7ff294c9d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2900c1040 con 0x7ff2900c05f0 2024-03-06T07:29:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.005+0000 7ff28ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2900c14c0 0x7ff29013ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.005+0000 7ff28e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff29006f640 0x7ff290148a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.005+0000 7ff28dd9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2900c05f0 0x7ff29013f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.005+0000 7ff28e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff29006f640 0x7ff290148a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:33216/0 (socket says 172.21.15.1:33216) 2024-03-06T07:29:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.005+0000 7ff28e59c640 1 -- 172.21.15.1:0/2685823146 learned_addr learned my addr 172.21.15.1:0/2685823146 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.005+0000 7ff28ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2900c14c0 0x7ff29013ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56106/0 (socket says 172.21.15.1:56106) 2024-03-06T07:29:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.005+0000 7ff28ed9d640 1 -- 172.21.15.1:0/2685823146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2900c05f0 msgr2=0x7ff29013f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.005+0000 7ff28ed9d640 1 --2- 172.21.15.1:0/2685823146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2900c05f0 0x7ff29013f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.005+0000 7ff28ed9d640 1 -- 172.21.15.1:0/2685823146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff29006f640 msgr2=0x7ff290148a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.005+0000 7ff28ed9d640 1 --2- 172.21.15.1:0/2685823146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff29006f640 0x7ff290148a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.005+0000 7ff28ed9d640 1 -- 172.21.15.1:0/2685823146 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2901ead40 con 0x7ff2900c14c0 2024-03-06T07:29:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.005+0000 7ff28ed9d640 1 --2- 172.21.15.1:0/2685823146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2900c14c0 0x7ff29013ff10 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7ff288009db0 tx=0x7ff28800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.005+0000 7ff286ffd640 1 -- 172.21.15.1:0/2685823146 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28800cb20 con 0x7ff2900c14c0 2024-03-06T07:29:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.005+0000 7ff286ffd640 1 -- 172.21.15.1:0/2685823146 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28800cce0 con 0x7ff2900c14c0 2024-03-06T07:29:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.005+0000 7ff294c9d640 1 -- 172.21.15.1:0/2685823146 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2901eb020 con 0x7ff2900c14c0 2024-03-06T07:29:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.005+0000 7ff286ffd640 1 -- 172.21.15.1:0/2685823146 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288003bd0 con 0x7ff2900c14c0 2024-03-06T07:29:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.005+0000 7ff294c9d640 1 -- 172.21.15.1:0/2685823146 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2901eb5e0 con 0x7ff2900c14c0 2024-03-06T07:29:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.006+0000 7ff294c9d640 1 -- 172.21.15.1:0/2685823146 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff290155a30 con 0x7ff2900c14c0 2024-03-06T07:29:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.007+0000 7ff286ffd640 1 -- 172.21.15.1:0/2685823146 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff28801b030 con 0x7ff2900c14c0 2024-03-06T07:29:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.007+0000 7ff286ffd640 1 --2- 172.21.15.1:0/2685823146 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff27006b4d0 0x7ff27006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.007+0000 7ff286ffd640 1 -- 172.21.15.1:0/2685823146 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff28808a4f0 con 0x7ff2900c14c0 2024-03-06T07:29:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.008+0000 7ff28e59c640 1 --2- 172.21.15.1:0/2685823146 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff27006b4d0 0x7ff27006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.008+0000 7ff28e59c640 1 --2- 172.21.15.1:0/2685823146 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff27006b4d0 0x7ff27006d990 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7ff280000f60 tx=0x7ff280007e10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.012+0000 7ff286ffd640 1 -- 172.21.15.1:0/2685823146 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff288022020 con 0x7ff2900c14c0 2024-03-06T07:29:38.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.264+0000 7ff294c9d640 1 -- 172.21.15.1:0/2685823146 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff290067fd0 con 0x7ff2900c14c0 2024-03-06T07:29:38.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.266+0000 7ff286ffd640 1 -- 172.21.15.1:0/2685823146 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v25) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff288010040 con 0x7ff2900c14c0 2024-03-06T07:29:38.268 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-06T07:29:38.268 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAFGehll+mQGRAA5NGcjuC1rUvJvB2Ai9RBng== 2024-03-06T07:29:38.268 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-06T07:29:38.268 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-06T07:29:38.268 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:29:38.268 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-06T07:29:38.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.268+0000 7ff294c9d640 1 -- 172.21.15.1:0/2685823146 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff27006b4d0 msgr2=0x7ff27006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.268+0000 7ff294c9d640 1 --2- 172.21.15.1:0/2685823146 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff27006b4d0 0x7ff27006d990 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7ff280000f60 tx=0x7ff280007e10 comp rx=0 tx=0).stop 2024-03-06T07:29:38.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.268+0000 7ff294c9d640 1 -- 172.21.15.1:0/2685823146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2900c14c0 msgr2=0x7ff29013ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.268+0000 7ff294c9d640 1 --2- 172.21.15.1:0/2685823146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2900c14c0 0x7ff29013ff10 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7ff288009db0 tx=0x7ff28800c490 comp rx=0 tx=0).stop 2024-03-06T07:29:38.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.268+0000 7ff294c9d640 1 -- 172.21.15.1:0/2685823146 shutdown_connections 2024-03-06T07:29:38.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.268+0000 7ff294c9d640 1 --2- 172.21.15.1:0/2685823146 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff27006b4d0 0x7ff27006d990 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.268+0000 7ff294c9d640 1 --2- 172.21.15.1:0/2685823146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2900c14c0 0x7ff29013ff10 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.268+0000 7ff294c9d640 1 --2- 172.21.15.1:0/2685823146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2900c05f0 0x7ff29013f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.268+0000 7ff294c9d640 1 --2- 172.21.15.1:0/2685823146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff29006f640 0x7ff290148a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.268+0000 7ff294c9d640 1 -- 172.21.15.1:0/2685823146 >> 172.21.15.1:0/2685823146 conn(0x7ff2900c3cc0 msgr2=0x7ff29006e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.268+0000 7ff294c9d640 1 -- 172.21.15.1:0/2685823146 shutdown_connections 2024-03-06T07:29:38.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.268+0000 7ff294c9d640 1 -- 172.21.15.1:0/2685823146 wait complete. 2024-03-06T07:29:38.281 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.399+0000 7fcb6571e640 1 Processor -- start 2024-03-06T07:29:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.399+0000 7fcb6571e640 1 -- start start 2024-03-06T07:29:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.400+0000 7fcb6571e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcb60152650 0x7fcb60155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.400+0000 7fcb6571e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcb600c6b10 0x7fcb60158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.400+0000 7fcb6571e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb600c2fc0 con 0x7fcb600bf9d0 2024-03-06T07:29:38.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.400+0000 7fcb6571e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb60155800 con 0x7fcb600c6b10 2024-03-06T07:29:38.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.400+0000 7fcb6571e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb600c6690 con 0x7fcb60152650 2024-03-06T07:29:38.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.400+0000 7fcb5e7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcb600c6b10 0x7fcb60158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.400+0000 7fcb5f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcb600bf9d0 0x7fcb600c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35886/0 (socket says 172.21.15.1:35886) 2024-03-06T07:29:38.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.400+0000 7fcb5f7fe640 1 -- 172.21.15.1:0/3221152536 learned_addr learned my addr 172.21.15.1:0/3221152536 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:38.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.400+0000 7fcb5effd640 1 --2- 172.21.15.1:0/3221152536 >> v2:172.21.15.98:3300/0 conn(0x7fcb60152650 0x7fcb60155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.400+0000 7fcb5e7fc640 1 -- 172.21.15.1:0/3221152536 >> v2:172.21.15.98:3300/0 conn(0x7fcb60152650 msgr2=0x7fcb60155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.400+0000 7fcb5e7fc640 1 --2- 172.21.15.1:0/3221152536 >> v2:172.21.15.98:3300/0 conn(0x7fcb60152650 0x7fcb60155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.400+0000 7fcb5e7fc640 1 -- 172.21.15.1:0/3221152536 >> v1:172.21.15.1:6789/0 conn(0x7fcb600bf9d0 legacy=0x7fcb600c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.400+0000 7fcb5e7fc640 1 -- 172.21.15.1:0/3221152536 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb600c7280 con 0x7fcb600c6b10 2024-03-06T07:29:38.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.401+0000 7fcb5e7fc640 1 --2- 172.21.15.1:0/3221152536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb600c6b10 0x7fcb60158690 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7fcb600cac90 tx=0x7fcb5002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=864ebc491d1e1d69 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.401+0000 7fcb5dffb640 1 -- 172.21.15.1:0/3221152536 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb500315a0 con 0x7fcb600c6b10 2024-03-06T07:29:38.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.401+0000 7fcb5dffb640 1 -- 172.21.15.1:0/3221152536 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb50037070 con 0x7fcb600c6b10 2024-03-06T07:29:38.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.401+0000 7fcb5dffb640 1 -- 172.21.15.1:0/3221152536 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb50031960 con 0x7fcb600c6b10 2024-03-06T07:29:38.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.402+0000 7fcb6571e640 1 -- 172.21.15.1:0/3221152536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb600c6b10 msgr2=0x7fcb60158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.402+0000 7fcb6571e640 1 --2- 172.21.15.1:0/3221152536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb600c6b10 0x7fcb60158690 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7fcb600cac90 tx=0x7fcb5002b840 comp rx=0 tx=0).stop 2024-03-06T07:29:38.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.402+0000 7fcb6571e640 1 -- 172.21.15.1:0/3221152536 shutdown_connections 2024-03-06T07:29:38.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.402+0000 7fcb6571e640 1 --2- 172.21.15.1:0/3221152536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb600c6b10 0x7fcb60158690 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.402+0000 7fcb6571e640 1 --2- 172.21.15.1:0/3221152536 >> v2:172.21.15.98:3300/0 conn(0x7fcb60152650 0x7fcb60155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.402+0000 7fcb6571e640 1 -- 172.21.15.1:0/3221152536 >> 172.21.15.1:0/3221152536 conn(0x7fcb60066ee0 msgr2=0x7fcb600672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.402+0000 7fcb6571e640 1 -- 172.21.15.1:0/3221152536 shutdown_connections 2024-03-06T07:29:38.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.402+0000 7fcb6571e640 1 -- 172.21.15.1:0/3221152536 wait complete. 2024-03-06T07:29:38.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.404+0000 7fcb6571e640 1 Processor -- start 2024-03-06T07:29:38.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.404+0000 7fcb6571e640 1 -- start start 2024-03-06T07:29:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb6571e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb600bf9d0 0x7fcb60156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb6571e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb60152650 0x7fcb601e1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb6571e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb601569d0 0x7fcb601e2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb6571e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb60155490 con 0x7fcb60152650 2024-03-06T07:29:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb6571e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb601551b0 con 0x7fcb600bf9d0 2024-03-06T07:29:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb6571e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb60152a30 con 0x7fcb601569d0 2024-03-06T07:29:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb5e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb60152650 0x7fcb601e1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb5e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb60152650 0x7fcb601e1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56116/0 (socket says 172.21.15.1:56116) 2024-03-06T07:29:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb5e7fc640 1 -- 172.21.15.1:0/1152795865 learned_addr learned my addr 172.21.15.1:0/1152795865 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb5effd640 1 --2- 172.21.15.1:0/1152795865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb600bf9d0 0x7fcb60156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb5f7fe640 1 --2- 172.21.15.1:0/1152795865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb601569d0 0x7fcb601e2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb5e7fc640 1 -- 172.21.15.1:0/1152795865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb601569d0 msgr2=0x7fcb601e2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb5e7fc640 1 --2- 172.21.15.1:0/1152795865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb601569d0 0x7fcb601e2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb5e7fc640 1 -- 172.21.15.1:0/1152795865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb600bf9d0 msgr2=0x7fcb60156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb5e7fc640 1 --2- 172.21.15.1:0/1152795865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb600bf9d0 0x7fcb60156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb5e7fc640 1 -- 172.21.15.1:0/1152795865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb601eb7e0 con 0x7fcb60152650 2024-03-06T07:29:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb5effd640 1 --2- 172.21.15.1:0/1152795865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb600bf9d0 0x7fcb60156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb5f7fe640 1 --2- 172.21.15.1:0/1152795865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb601569d0 0x7fcb601e2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb5e7fc640 1 --2- 172.21.15.1:0/1152795865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb60152650 0x7fcb601e1ce0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7fcb50000c00 tx=0x7fcb500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.405+0000 7fcb577fe640 1 -- 172.21.15.1:0/1152795865 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb50032840 con 0x7fcb60152650 2024-03-06T07:29:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.406+0000 7fcb577fe640 1 -- 172.21.15.1:0/1152795865 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb50032a00 con 0x7fcb60152650 2024-03-06T07:29:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.406+0000 7fcb6571e640 1 -- 172.21.15.1:0/1152795865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb601ebac0 con 0x7fcb60152650 2024-03-06T07:29:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.406+0000 7fcb577fe640 1 -- 172.21.15.1:0/1152795865 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb50032df0 con 0x7fcb60152650 2024-03-06T07:29:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.406+0000 7fcb6571e640 1 -- 172.21.15.1:0/1152795865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb60153270 con 0x7fcb60152650 2024-03-06T07:29:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.407+0000 7fcb6571e640 1 -- 172.21.15.1:0/1152795865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb60067fd0 con 0x7fcb60152650 2024-03-06T07:29:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.407+0000 7fcb577fe640 1 -- 172.21.15.1:0/1152795865 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb5004c020 con 0x7fcb60152650 2024-03-06T07:29:38.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.407+0000 7fcb577fe640 1 --2- 172.21.15.1:0/1152795865 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcb4006b4d0 0x7fcb4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.407+0000 7fcb577fe640 1 -- 172.21.15.1:0/1152795865 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcb5002f020 con 0x7fcb60152650 2024-03-06T07:29:38.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.408+0000 7fcb5effd640 1 --2- 172.21.15.1:0/1152795865 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcb4006b4d0 0x7fcb4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.408+0000 7fcb5effd640 1 --2- 172.21.15.1:0/1152795865 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcb4006b4d0 0x7fcb4006d990 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fcb48006fd0 tx=0x7fcb48008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.416+0000 7fcb577fe640 1 -- 172.21.15.1:0/1152795865 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb50040070 con 0x7fcb60152650 2024-03-06T07:29:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.636+0000 7fcb6571e640 1 -- 172.21.15.1:0/1152795865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb6014f030 con 0x7fcb60152650 2024-03-06T07:29:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.637+0000 7fcb577fe640 1 -- 172.21.15.1:0/1152795865 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+4987 (secure 0 0 0) 0x7fcb5007b070 con 0x7fcb60152650 2024-03-06T07:29:38.639 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:38.639 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":196,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":10803,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:35.664209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10806},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:29:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 196 2024-03-06T07:29:38.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.638+0000 7fcb6571e640 1 -- 172.21.15.1:0/1152795865 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcb4006b4d0 msgr2=0x7fcb4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.638+0000 7fcb6571e640 1 --2- 172.21.15.1:0/1152795865 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcb4006b4d0 0x7fcb4006d990 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fcb48006fd0 tx=0x7fcb48008040 comp rx=0 tx=0).stop 2024-03-06T07:29:38.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.638+0000 7fcb6571e640 1 -- 172.21.15.1:0/1152795865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb60152650 msgr2=0x7fcb601e1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.638+0000 7fcb6571e640 1 --2- 172.21.15.1:0/1152795865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb60152650 0x7fcb601e1ce0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7fcb50000c00 tx=0x7fcb500046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.639+0000 7fcb6571e640 1 -- 172.21.15.1:0/1152795865 shutdown_connections 2024-03-06T07:29:38.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.639+0000 7fcb6571e640 1 --2- 172.21.15.1:0/1152795865 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcb4006b4d0 0x7fcb4006d990 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.639+0000 7fcb6571e640 1 --2- 172.21.15.1:0/1152795865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb601569d0 0x7fcb601e2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.639+0000 7fcb6571e640 1 --2- 172.21.15.1:0/1152795865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb60152650 0x7fcb601e1ce0 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.639+0000 7fcb6571e640 1 --2- 172.21.15.1:0/1152795865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb600bf9d0 0x7fcb60156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.639+0000 7fcb6571e640 1 -- 172.21.15.1:0/1152795865 >> 172.21.15.1:0/1152795865 conn(0x7fcb60066ee0 msgr2=0x7fcb60150810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.639+0000 7fcb6571e640 1 -- 172.21.15.1:0/1152795865 shutdown_connections 2024-03-06T07:29:38.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:38.639+0000 7fcb6571e640 1 -- 172.21.15.1:0/1152795865 wait complete. 2024-03-06T07:29:38.765 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-06T07:29:38.763+0000 7f76a9082640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:29:39.054 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 196, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:35.664209+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10806}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:29:39.054 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:29:39.054 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:29:39.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.155+0000 7f74e0df4640 1 Processor -- start 2024-03-06T07:29:39.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.155+0000 7f74e0df4640 1 -- start start 2024-03-06T07:29:39.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.156+0000 7f74e0df4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f74dc0c6b10 0x7f74dc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.156+0000 7f74e0df4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74dc0c2fc0 con 0x7f74dc0bf9d0 2024-03-06T07:29:39.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.156+0000 7f74e0df4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74dc158da0 con 0x7f74dc152650 2024-03-06T07:29:39.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.156+0000 7f74e0df4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74dc155800 con 0x7f74dc0c6b10 2024-03-06T07:29:39.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.156+0000 7f74d3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f74dc0c6b10 0x7f74dc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.156+0000 7f74dad9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f74dc0bf9d0 0x7f74dc0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35900/0 (socket says 172.21.15.1:35900) 2024-03-06T07:29:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.156+0000 7f74dad9d640 1 -- 172.21.15.1:0/3939684561 learned_addr learned my addr 172.21.15.1:0/3939684561 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.156+0000 7f74d3fff640 1 -- 172.21.15.1:0/3939684561 >> v1:172.21.15.98:6789/0 conn(0x7f74dc152650 legacy=0x7f74dc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.156+0000 7f74d3fff640 1 -- 172.21.15.1:0/3939684561 >> v1:172.21.15.1:6789/0 conn(0x7f74dc0bf9d0 legacy=0x7f74dc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.156+0000 7f74d3fff640 1 -- 172.21.15.1:0/3939684561 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc0c6690 con 0x7f74dc0c6b10 2024-03-06T07:29:39.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.156+0000 7f74d9d9b640 1 -- 172.21.15.1:0/3939684561 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535678588 0 0) 0x7f74dc0c2fc0 con 0x7f74dc0bf9d0 2024-03-06T07:29:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.157+0000 7f74d3fff640 1 --2- 172.21.15.1:0/3939684561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74dc0c6b10 0x7f74dc155c90 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f74c40097a0 tx=0x7f74c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3fd9d94cf10c0a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.157+0000 7f74d9d9b640 1 -- 172.21.15.1:0/3939684561 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4035020 con 0x7f74dc0c6b10 2024-03-06T07:29:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.157+0000 7f74d9d9b640 1 -- 172.21.15.1:0/3939684561 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c4031830 con 0x7f74dc0c6b10 2024-03-06T07:29:39.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.157+0000 7f74d9d9b640 1 -- 172.21.15.1:0/3939684561 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4031c00 con 0x7f74dc0c6b10 2024-03-06T07:29:39.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.158+0000 7f74e0df4640 1 -- 172.21.15.1:0/3939684561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74dc0c6b10 msgr2=0x7f74dc155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.158+0000 7f74e0df4640 1 --2- 172.21.15.1:0/3939684561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74dc0c6b10 0x7f74dc155c90 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f74c40097a0 tx=0x7f74c402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.158+0000 7f74e0df4640 1 -- 172.21.15.1:0/3939684561 shutdown_connections 2024-03-06T07:29:39.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.158+0000 7f74e0df4640 1 --2- 172.21.15.1:0/3939684561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74dc0c6b10 0x7f74dc155c90 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.158+0000 7f74e0df4640 1 -- 172.21.15.1:0/3939684561 >> 172.21.15.1:0/3939684561 conn(0x7f74dc066ee0 msgr2=0x7f74dc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.158+0000 7f74e0df4640 1 -- 172.21.15.1:0/3939684561 shutdown_connections 2024-03-06T07:29:39.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.158+0000 7f74e0df4640 1 -- 172.21.15.1:0/3939684561 wait complete. 2024-03-06T07:29:39.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.159+0000 7f74e0df4640 1 Processor -- start 2024-03-06T07:29:39.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.159+0000 7f74e0df4640 1 -- start start 2024-03-06T07:29:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.160+0000 7f74e0df4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74dc0bf9d0 0x7f74dc1536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.160+0000 7f74e0df4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74dc152650 0x7f74dc14d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.160+0000 7f74e0df4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74dc1df810 0x7f74dc14d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.160+0000 7f74e0df4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc1551b0 con 0x7f74dc0bf9d0 2024-03-06T07:29:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.160+0000 7f74e0df4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc0c6fa0 con 0x7f74dc1df810 2024-03-06T07:29:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.160+0000 7f74e0df4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc155490 con 0x7f74dc152650 2024-03-06T07:29:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.160+0000 7f74da59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74dc0bf9d0 0x7f74dc1536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.160+0000 7f74da59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74dc0bf9d0 0x7f74dc1536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56136/0 (socket says 172.21.15.1:56136) 2024-03-06T07:29:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.160+0000 7f74da59c640 1 -- 172.21.15.1:0/3280433966 learned_addr learned my addr 172.21.15.1:0/3280433966 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.160+0000 7f74dad9d640 1 --2- 172.21.15.1:0/3280433966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74dc1df810 0x7f74dc14d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.160+0000 7f74d3fff640 1 --2- 172.21.15.1:0/3280433966 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74dc152650 0x7f74dc14d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.160+0000 7f74da59c640 1 -- 172.21.15.1:0/3280433966 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74dc152650 msgr2=0x7f74dc14d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.160+0000 7f74da59c640 1 --2- 172.21.15.1:0/3280433966 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74dc152650 0x7f74dc14d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.160+0000 7f74da59c640 1 -- 172.21.15.1:0/3280433966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74dc1df810 msgr2=0x7f74dc14d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.160+0000 7f74da59c640 1 --2- 172.21.15.1:0/3280433966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74dc1df810 0x7f74dc14d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.160+0000 7f74da59c640 1 -- 172.21.15.1:0/3280433966 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc14ca70 con 0x7f74dc0bf9d0 2024-03-06T07:29:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.160+0000 7f74da59c640 1 --2- 172.21.15.1:0/3280433966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74dc0bf9d0 0x7f74dc1536e0 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00c480 tx=0x7f74cc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.161+0000 7f74d2ffd640 1 -- 172.21.15.1:0/3280433966 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc0052d0 con 0x7f74dc0bf9d0 2024-03-06T07:29:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.161+0000 7f74d2ffd640 1 -- 172.21.15.1:0/3280433966 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74cc005470 con 0x7f74dc0bf9d0 2024-03-06T07:29:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.161+0000 7f74e0df4640 1 -- 172.21.15.1:0/3280433966 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74dc14dd10 con 0x7f74dc0bf9d0 2024-03-06T07:29:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.161+0000 7f74d2ffd640 1 -- 172.21.15.1:0/3280433966 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc012870 con 0x7f74dc0bf9d0 2024-03-06T07:29:39.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.161+0000 7f74e0df4640 1 -- 172.21.15.1:0/3280433966 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74dc154900 con 0x7f74dc0bf9d0 2024-03-06T07:29:39.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.163+0000 7f74d2ffd640 1 -- 172.21.15.1:0/3280433966 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74cc012a10 con 0x7f74dc0bf9d0 2024-03-06T07:29:39.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.163+0000 7f74d2ffd640 1 --2- 172.21.15.1:0/3280433966 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74bc06b4d0 0x7f74bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.163+0000 7f74d2ffd640 1 -- 172.21.15.1:0/3280433966 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f74cc089f90 con 0x7f74dc0bf9d0 2024-03-06T07:29:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.163+0000 7f74d3fff640 1 --2- 172.21.15.1:0/3280433966 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74bc06b4d0 0x7f74bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.163+0000 7f74e39d8740 1 Processor -- start 2024-03-06T07:29:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.164+0000 7f74e39d8740 1 -- start start 2024-03-06T07:29:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.164+0000 7f74d3fff640 1 --2- 172.21.15.1:0/3280433966 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74bc06b4d0 0x7f74bc06d990 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f74c4036000 tx=0x7f74c4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.164+0000 7f74e39d8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aeabf50 0x55da7aea9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.164+0000 7f74e39d8740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeac330 0x55da7aea9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.164+0000 7f74e39d8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aeaa4b0 0x55da7aea6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.164+0000 7f74e39d8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da7ae106b0 con 0x55da7aeaa4b0 2024-03-06T07:29:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.164+0000 7f74e39d8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da7adb9a60 con 0x55da7aeabf50 2024-03-06T07:29:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.164+0000 7f74e39d8740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55da7ae0c750 con 0x55da7aeac330 2024-03-06T07:29:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.164+0000 7f74dad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aeaa4b0 0x55da7aea6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.164+0000 7f74dad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aeaa4b0 0x55da7aea6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56152/0 (socket says 172.21.15.1:56152) 2024-03-06T07:29:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.164+0000 7f74dad9d640 1 -- 172.21.15.1:0/791855433 learned_addr learned my addr 172.21.15.1:0/791855433 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.164+0000 7f74d3fff640 1 --2- 172.21.15.1:0/791855433 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeac330 0x55da7aea9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.164+0000 7f74da59c640 1 --2- 172.21.15.1:0/791855433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aeabf50 0x55da7aea9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.165+0000 7f74dad9d640 1 -- 172.21.15.1:0/791855433 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeac330 msgr2=0x55da7aea9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.165+0000 7f74dad9d640 1 --2- 172.21.15.1:0/791855433 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeac330 0x55da7aea9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.165+0000 7f74dad9d640 1 -- 172.21.15.1:0/791855433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aeabf50 msgr2=0x55da7aea9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.165+0000 7f74dad9d640 1 --2- 172.21.15.1:0/791855433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aeabf50 0x55da7aea9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.165+0000 7f74dad9d640 1 -- 172.21.15.1:0/791855433 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da7add7f30 con 0x55da7aeaa4b0 2024-03-06T07:29:39.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.165+0000 7f74da59c640 1 --2- 172.21.15.1:0/791855433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aeabf50 0x55da7aea9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:39.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.165+0000 7f74dad9d640 1 --2- 172.21.15.1:0/791855433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aeaa4b0 0x55da7aea6680 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f74d4001cd0 tx=0x7f74d4002190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b279a3a755eac92e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.165+0000 7f74bbfff640 1 -- 172.21.15.1:0/791855433 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d400d8b0 con 0x55da7aeaa4b0 2024-03-06T07:29:39.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.165+0000 7f74bbfff640 1 -- 172.21.15.1:0/791855433 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d400da50 con 0x55da7aeaa4b0 2024-03-06T07:29:39.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.165+0000 7f74bbfff640 1 -- 172.21.15.1:0/791855433 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d400de40 con 0x55da7aeaa4b0 2024-03-06T07:29:39.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.166+0000 7f74e39d8740 1 -- 172.21.15.1:0/791855433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aeaa4b0 msgr2=0x55da7aea6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.166+0000 7f74e39d8740 1 --2- 172.21.15.1:0/791855433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aeaa4b0 0x55da7aea6680 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f74d4001cd0 tx=0x7f74d4002190 comp rx=0 tx=0).stop 2024-03-06T07:29:39.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.166+0000 7f74e39d8740 1 -- 172.21.15.1:0/791855433 shutdown_connections 2024-03-06T07:29:39.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.166+0000 7f74e39d8740 1 --2- 172.21.15.1:0/791855433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aeaa4b0 0x55da7aea6680 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.166+0000 7f74e39d8740 1 --2- 172.21.15.1:0/791855433 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeac330 0x55da7aea9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.166+0000 7f74e39d8740 1 --2- 172.21.15.1:0/791855433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aeabf50 0x55da7aea9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.166+0000 7f74e39d8740 1 -- 172.21.15.1:0/791855433 >> 172.21.15.1:0/791855433 conn(0x55da7aea87d0 msgr2=0x55da7aecd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.166+0000 7f74e39d8740 1 -- 172.21.15.1:0/791855433 shutdown_connections 2024-03-06T07:29:39.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.166+0000 7f74e39d8740 1 -- 172.21.15.1:0/791855433 wait complete. 2024-03-06T07:29:39.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.166+0000 7f74e39d8740 1 -- start start 2024-03-06T07:29:39.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.166+0000 7f74e39d8740 1 Processor -- start 2024-03-06T07:29:39.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.167+0000 7f74e39d8740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeabf50 0x55da7aecca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.167+0000 7f74e39d8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aeac330 0x55da7aed0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.167+0000 7f74e39d8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aee49c0 0x55da7aee4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.167+0000 7f74e39d8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da7aece230 con 0x55da7aee49c0 2024-03-06T07:29:39.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.167+0000 7f74e39d8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da7adccea0 con 0x55da7aeac330 2024-03-06T07:29:39.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.167+0000 7f74e39d8740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55da7aecdc70 con 0x55da7aeabf50 2024-03-06T07:29:39.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.167+0000 7f74e39d8740 10 client.0 _ll_register_callbacks cb 0x55da7adcc870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:39.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.167+0000 7f74dad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aee49c0 0x55da7aee4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.167+0000 7f74dad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aee49c0 0x55da7aee4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56166/0 (socket says 172.21.15.1:56166) 2024-03-06T07:29:39.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.167+0000 7f74dad9d640 1 -- 172.21.15.1:0/59698601 learned_addr learned my addr 172.21.15.1:0/59698601 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:39.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.167+0000 7f74da59c640 1 --2- 172.21.15.1:0/59698601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeabf50 0x55da7aecca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.167+0000 7f74d3fff640 1 --2- 172.21.15.1:0/59698601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aeac330 0x55da7aed0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.168+0000 7f74dad9d640 1 -- 172.21.15.1:0/59698601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeabf50 msgr2=0x55da7aecca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.168+0000 7f74dad9d640 1 --2- 172.21.15.1:0/59698601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeabf50 0x55da7aecca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.168+0000 7f74dad9d640 1 -- 172.21.15.1:0/59698601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aeac330 msgr2=0x55da7aed0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.168+0000 7f74dad9d640 1 --2- 172.21.15.1:0/59698601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aeac330 0x55da7aed0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.168+0000 7f74dad9d640 1 -- 172.21.15.1:0/59698601 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da7adcd270 con 0x55da7aee49c0 2024-03-06T07:29:39.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.168+0000 7f74dad9d640 1 --2- 172.21.15.1:0/59698601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aee49c0 0x55da7aee4e40 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f74d4000f80 tx=0x7f74d4007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a04012c11c33da6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.168+0000 7f74b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:29:39.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.168+0000 7f74b97fa640 1 -- 172.21.15.1:0/59698601 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d4023020 con 0x55da7aee49c0 2024-03-06T07:29:39.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.168+0000 7f74b97fa640 1 -- 172.21.15.1:0/59698601 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f74d40072f0 con 0x55da7aee49c0 2024-03-06T07:29:39.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.168+0000 7f74e39d8740 1 -- 172.21.15.1:0/59698601 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f74d400da50 con 0x55da7aee49c0 2024-03-06T07:29:39.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.168+0000 7f74b97fa640 1 -- 172.21.15.1:0/59698601 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=196) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74d4007c60 con 0x55da7aee49c0 2024-03-06T07:29:39.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.168+0000 7f74e39d8740 10 client.10959 fetch_fsmap learned FSMap version 196 2024-03-06T07:29:39.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.168+0000 7f74e39d8740 1 -- 172.21.15.1:0/59698601 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=196,monmap=2+}) v3 -- 0x55da7ab252d0 con 0x55da7aee49c0 2024-03-06T07:29:39.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.168+0000 7f74b97fa640 1 -- 172.21.15.1:0/59698601 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d400da50 con 0x55da7aee49c0 2024-03-06T07:29:39.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.168+0000 7f74b97fa640 1 -- 172.21.15.1:0/59698601 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 196) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f74d40074f0 con 0x55da7aee49c0 2024-03-06T07:29:39.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.168+0000 7f74e39d8740 10 client.10959 fetch_fsmap finished waiting for FSMap version 196 2024-03-06T07:29:39.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.169+0000 7f74e39d8740 10 client.10959 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:29:39.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.169+0000 7f74e39d8740 20 client.10959 populate_metadata read hostname 'smithi001' 2024-03-06T07:29:39.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.169+0000 7f74e39d8740 1 --2- 172.21.15.1:0/59698601 >> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] conn(0x55da7af05ac0 0x55da7af07eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.169+0000 7f74e39d8740 4 client.10959 mds_command: new command op to 10806 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:29:39.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.169+0000 7f74e39d8740 1 -- 172.21.15.1:0/59698601 --> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55da7ae1dbf0 con 0x55da7af05ac0 2024-03-06T07:29:39.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.169+0000 7f74da59c640 1 --2- 172.21.15.1:0/59698601 >> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] conn(0x55da7af05ac0 0x55da7af07eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.169+0000 7f74da59c640 1 --2- 172.21.15.1:0/59698601 >> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] conn(0x55da7af05ac0 0x55da7af07eb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c17cc278c59fadb server_cookie=cadf4f49ea2329fa in_seq=0 out_seq=0 2024-03-06T07:29:39.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.169+0000 7f74b97fa640 10 client.10959 ms_handle_connect on v2:172.21.15.1:6833/1871567565 2024-03-06T07:29:39.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.170+0000 7f74b97fa640 1 -- 172.21.15.1:0/59698601 <== mds.0 v2:172.21.15.1:6833/1871567565 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55da7ae1dbf0 con 0x55da7af05ac0 2024-03-06T07:29:39.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.170+0000 7f74b97fa640 10 client.10959 handle_command_reply: tid=0 2024-03-06T07:29:39.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.170+0000 7f74e39d8740 1 client.10959 shutdown 2024-03-06T07:29:39.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.170+0000 7f74e39d8740 1 -- 172.21.15.1:0/59698601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aee49c0 msgr2=0x55da7aee4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.170+0000 7f74e39d8740 1 --2- 172.21.15.1:0/59698601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aee49c0 0x55da7aee4e40 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f74d4000f80 tx=0x7f74d4007bb0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.170+0000 7f74e39d8740 1 -- 172.21.15.1:0/59698601 shutdown_connections 2024-03-06T07:29:39.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.170+0000 7f74e39d8740 1 --2- 172.21.15.1:0/59698601 >> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] conn(0x55da7af05ac0 0x55da7af07eb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.170+0000 7f74e39d8740 1 --2- 172.21.15.1:0/59698601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aee49c0 0x55da7aee4e40 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.170+0000 7f74e39d8740 1 --2- 172.21.15.1:0/59698601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aeac330 0x55da7aed0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.170+0000 7f74b97fa640 0 client.10959 ms_handle_reset on v2:172.21.15.1:6833/1871567565 2024-03-06T07:29:39.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.170+0000 7f74e39d8740 1 --2- 172.21.15.1:0/59698601 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeabf50 0x55da7aecca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.170+0000 7f74e39d8740 1 -- 172.21.15.1:0/59698601 >> 172.21.15.1:0/59698601 conn(0x55da7aeaa4b0 msgr2=0x55da7aeaa8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.170+0000 7f74e39d8740 1 -- 172.21.15.1:0/59698601 shutdown_connections 2024-03-06T07:29:39.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.170+0000 7f74e39d8740 1 -- 172.21.15.1:0/59698601 wait complete. 2024-03-06T07:29:39.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.170+0000 7f74e39d8740 20 client.10959 trim_cache size 0 max 16384 2024-03-06T07:29:39.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.187+0000 7f74e39d8740 1 Processor -- start 2024-03-06T07:29:39.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.187+0000 7f74e39d8740 1 -- start start 2024-03-06T07:29:39.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74e39d8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aeabf50 0x55da7aea5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74e39d8740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeac330 0x55da7aed2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74e39d8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aee49c0 0x55da7aed2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74e39d8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc1f7ba0 con 0x55da7aeabf50 2024-03-06T07:29:39.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74e39d8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc1f6d30 con 0x55da7aee49c0 2024-03-06T07:29:39.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74e39d8740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc14b550 con 0x55da7aeac330 2024-03-06T07:29:39.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74da59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aeabf50 0x55da7aea5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74da59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aeabf50 0x55da7aea5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56176/0 (socket says 172.21.15.1:56176) 2024-03-06T07:29:39.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74da59c640 1 -- 172.21.15.1:0/235290772 learned_addr learned my addr 172.21.15.1:0/235290772 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:39.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74da59c640 1 -- 172.21.15.1:0/235290772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeac330 msgr2=0x55da7aed2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74d3fff640 1 --2- 172.21.15.1:0/235290772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeac330 0x55da7aed2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74dad9d640 1 --2- 172.21.15.1:0/235290772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aee49c0 0x55da7aed2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74da59c640 1 --2- 172.21.15.1:0/235290772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeac330 0x55da7aed2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74da59c640 1 -- 172.21.15.1:0/235290772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aee49c0 msgr2=0x55da7aed2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74da59c640 1 --2- 172.21.15.1:0/235290772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aee49c0 0x55da7aed2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74da59c640 1 -- 172.21.15.1:0/235290772 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da7abef6b0 con 0x55da7aeabf50 2024-03-06T07:29:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74d3fff640 1 --2- 172.21.15.1:0/235290772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeac330 0x55da7aed2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74dad9d640 1 --2- 172.21.15.1:0/235290772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aee49c0 0x55da7aed2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74da59c640 1 --2- 172.21.15.1:0/235290772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aeabf50 0x55da7aea5510 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f74cc005620 tx=0x7f74cc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d00119e6b308d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74bb7fe640 1 -- 172.21.15.1:0/235290772 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc0566b0 con 0x55da7aeabf50 2024-03-06T07:29:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74bb7fe640 1 -- 172.21.15.1:0/235290772 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74cc056910 con 0x55da7aeabf50 2024-03-06T07:29:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.188+0000 7f74bb7fe640 1 -- 172.21.15.1:0/235290772 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc054c80 con 0x55da7aeabf50 2024-03-06T07:29:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.189+0000 7f74e39d8740 1 -- 172.21.15.1:0/235290772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aeabf50 msgr2=0x55da7aea5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.189+0000 7f74e39d8740 1 --2- 172.21.15.1:0/235290772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aeabf50 0x55da7aea5510 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f74cc005620 tx=0x7f74cc001120 comp rx=0 tx=0).stop 2024-03-06T07:29:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.189+0000 7f74e39d8740 1 -- 172.21.15.1:0/235290772 shutdown_connections 2024-03-06T07:29:39.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.189+0000 7f74e39d8740 1 --2- 172.21.15.1:0/235290772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aee49c0 0x55da7aed2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.189+0000 7f74e39d8740 1 --2- 172.21.15.1:0/235290772 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeac330 0x55da7aed2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.189+0000 7f74e39d8740 1 --2- 172.21.15.1:0/235290772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aeabf50 0x55da7aea5510 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.189+0000 7f74e39d8740 1 -- 172.21.15.1:0/235290772 >> 172.21.15.1:0/235290772 conn(0x55da7aeaa4b0 msgr2=0x55da7aee9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.189+0000 7f74e39d8740 1 -- 172.21.15.1:0/235290772 shutdown_connections 2024-03-06T07:29:39.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.189+0000 7f74e39d8740 1 -- 172.21.15.1:0/235290772 wait complete. 2024-03-06T07:29:39.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.190+0000 7f74e39d8740 1 -- start start 2024-03-06T07:29:39.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.190+0000 7f74e39d8740 1 Processor -- start 2024-03-06T07:29:39.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.191+0000 7f74e39d8740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeabf50 0x55da7af57fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.191+0000 7f74e39d8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aeac330 0x55da7aef9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.191+0000 7f74e39d8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aee49c0 0x55da7af51c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.191+0000 7f74e39d8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da7aece230 con 0x55da7aee49c0 2024-03-06T07:29:39.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.191+0000 7f74e39d8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc1f67b0 con 0x55da7aeac330 2024-03-06T07:29:39.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.191+0000 7f74e39d8740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x55da7aed05b0 con 0x55da7aeabf50 2024-03-06T07:29:39.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.191+0000 7f74e39d8740 10 client.0 _ll_register_callbacks cb 0x7f74d400d8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:39.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.191+0000 7f74dad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aee49c0 0x55da7af51c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.191+0000 7f74dad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aee49c0 0x55da7af51c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56190/0 (socket says 172.21.15.1:56190) 2024-03-06T07:29:39.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.191+0000 7f74dad9d640 1 -- 172.21.15.1:0/1701046373 learned_addr learned my addr 172.21.15.1:0/1701046373 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:39.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.191+0000 7f74da59c640 1 --2- 172.21.15.1:0/1701046373 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeabf50 0x55da7af57fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.191+0000 7f74d3fff640 1 --2- 172.21.15.1:0/1701046373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aeac330 0x55da7aef9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.191+0000 7f74dad9d640 1 -- 172.21.15.1:0/1701046373 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeabf50 msgr2=0x55da7af57fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.191+0000 7f74dad9d640 1 --2- 172.21.15.1:0/1701046373 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeabf50 0x55da7af57fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.191+0000 7f74dad9d640 1 -- 172.21.15.1:0/1701046373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aeac330 msgr2=0x55da7aef9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.191+0000 7f74dad9d640 1 --2- 172.21.15.1:0/1701046373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aeac330 0x55da7aef9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.191+0000 7f74dad9d640 1 -- 172.21.15.1:0/1701046373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da7aef97d0 con 0x55da7aee49c0 2024-03-06T07:29:39.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.192+0000 7f74dad9d640 1 --2- 172.21.15.1:0/1701046373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aee49c0 0x55da7af51c20 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f74d40021d0 tx=0x7f74d4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33458c9ee910bada server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.192+0000 7f74b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:29:39.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.192+0000 7f74b97fa640 1 -- 172.21.15.1:0/1701046373 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d401f670 con 0x55da7aee49c0 2024-03-06T07:29:39.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.192+0000 7f74b97fa640 1 -- 172.21.15.1:0/1701046373 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f74d402a880 con 0x55da7aee49c0 2024-03-06T07:29:39.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.192+0000 7f74e39d8740 1 -- 172.21.15.1:0/1701046373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f74cc056910 con 0x55da7aee49c0 2024-03-06T07:29:39.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.192+0000 7f74b97fa640 1 -- 172.21.15.1:0/1701046373 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=196) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74d4040050 con 0x55da7aee49c0 2024-03-06T07:29:39.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.192+0000 7f74e39d8740 10 client.10965 fetch_fsmap learned FSMap version 196 2024-03-06T07:29:39.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.192+0000 7f74e39d8740 1 -- 172.21.15.1:0/1701046373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=196,monmap=2+}) v3 -- 0x55da7aed3540 con 0x55da7aee49c0 2024-03-06T07:29:39.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.192+0000 7f74b97fa640 1 -- 172.21.15.1:0/1701046373 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74cc056910 con 0x55da7aee49c0 2024-03-06T07:29:39.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.192+0000 7f74b97fa640 1 -- 172.21.15.1:0/1701046373 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 196) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f74d4043020 con 0x55da7aee49c0 2024-03-06T07:29:39.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.192+0000 7f74e39d8740 10 client.10965 fetch_fsmap finished waiting for FSMap version 196 2024-03-06T07:29:39.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.192+0000 7f74e39d8740 10 client.10965 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:29:39.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.192+0000 7f74e39d8740 20 client.10965 populate_metadata read hostname 'smithi001' 2024-03-06T07:29:39.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.192+0000 7f74e39d8740 1 --2- 172.21.15.1:0/1701046373 >> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] conn(0x55da7aea63b0 0x55da7aea6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.192+0000 7f74e39d8740 4 client.10965 mds_command: new command op to 10806 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:29:39.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.192+0000 7f74e39d8740 1 -- 172.21.15.1:0/1701046373 --> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55da7aecdf40 con 0x55da7aea63b0 2024-03-06T07:29:39.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.192+0000 7f74da59c640 1 --2- 172.21.15.1:0/1701046373 >> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] conn(0x55da7aea63b0 0x55da7aea6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.193+0000 7f74b97fa640 10 client.10965 ms_handle_connect on v2:172.21.15.1:6833/1871567565 2024-03-06T07:29:39.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.193+0000 7f74da59c640 1 --2- 172.21.15.1:0/1701046373 >> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] conn(0x55da7aea63b0 0x55da7aea6790 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=325ffbed2da9d07a server_cookie=eb56e6a37f28194c in_seq=0 out_seq=0 2024-03-06T07:29:39.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.193+0000 7f74b97fa640 1 -- 172.21.15.1:0/1701046373 <== mds.0 v2:172.21.15.1:6833/1871567565 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x55da7aecdf40 con 0x55da7aea63b0 2024-03-06T07:29:39.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.193+0000 7f74b97fa640 10 client.10965 handle_command_reply: tid=0 2024-03-06T07:29:39.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.193+0000 7f74e39d8740 1 client.10965 shutdown 2024-03-06T07:29:39.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.193+0000 7f74e39d8740 1 -- 172.21.15.1:0/1701046373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aee49c0 msgr2=0x55da7af51c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.193+0000 7f74e39d8740 1 --2- 172.21.15.1:0/1701046373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aee49c0 0x55da7af51c20 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f74d40021d0 tx=0x7f74d4009040 comp rx=0 tx=0).stop 2024-03-06T07:29:39.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.193+0000 7f74e39d8740 1 -- 172.21.15.1:0/1701046373 shutdown_connections 2024-03-06T07:29:39.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.193+0000 7f74e39d8740 1 --2- 172.21.15.1:0/1701046373 >> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] conn(0x55da7aea63b0 0x55da7aea6790 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.193+0000 7f74e39d8740 1 --2- 172.21.15.1:0/1701046373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da7aee49c0 0x55da7af51c20 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.193+0000 7f74e39d8740 1 --2- 172.21.15.1:0/1701046373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55da7aeac330 0x55da7aef9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.193+0000 7f74b97fa640 0 client.10965 ms_handle_reset on v2:172.21.15.1:6833/1871567565 2024-03-06T07:29:39.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.193+0000 7f74e39d8740 1 --2- 172.21.15.1:0/1701046373 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x55da7aeabf50 0x55da7af57fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.193+0000 7f74e39d8740 1 -- 172.21.15.1:0/1701046373 >> 172.21.15.1:0/1701046373 conn(0x55da7aeaa4b0 msgr2=0x55da7aea1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.193+0000 7f74e39d8740 1 -- 172.21.15.1:0/1701046373 shutdown_connections 2024-03-06T07:29:39.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.193+0000 7f74e39d8740 1 -- 172.21.15.1:0/1701046373 wait complete. 2024-03-06T07:29:39.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.193+0000 7f74e39d8740 20 client.10965 trim_cache size 0 max 16384 2024-03-06T07:29:39.201 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:29:39.201 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:29:39.201 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:29:39.201 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 10806, 2024-03-06T07:29:39.202 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:29:39.202 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:29:39.202 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:29:39.202 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 5.7033327759999999, 2024-03-06T07:29:39.202 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 196, 2024-03-06T07:29:39.202 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 210, 2024-03-06T07:29:39.202 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 210, 2024-03-06T07:29:39.202 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 10.786876797 2024-03-06T07:29:39.202 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:29:39.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.194+0000 7f74e0df4640 1 -- 172.21.15.1:0/3280433966 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74bc06b4d0 msgr2=0x7f74bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.194+0000 7f74e0df4640 1 --2- 172.21.15.1:0/3280433966 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74bc06b4d0 0x7f74bc06d990 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f74c4036000 tx=0x7f74c4008030 comp rx=0 tx=0).stop 2024-03-06T07:29:39.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.194+0000 7f74e0df4640 1 -- 172.21.15.1:0/3280433966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74dc0bf9d0 msgr2=0x7f74dc1536e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.194+0000 7f74e0df4640 1 --2- 172.21.15.1:0/3280433966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74dc0bf9d0 0x7f74dc1536e0 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00c480 tx=0x7f74cc00c940 comp rx=0 tx=0).stop 2024-03-06T07:29:39.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.194+0000 7f74e0df4640 1 -- 172.21.15.1:0/3280433966 shutdown_connections 2024-03-06T07:29:39.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.194+0000 7f74e0df4640 1 --2- 172.21.15.1:0/3280433966 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f74bc06b4d0 0x7f74bc06d990 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.194+0000 7f74e0df4640 1 --2- 172.21.15.1:0/3280433966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74dc1df810 0x7f74dc14d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.194+0000 7f74e0df4640 1 --2- 172.21.15.1:0/3280433966 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74dc152650 0x7f74dc14d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.194+0000 7f74e0df4640 1 --2- 172.21.15.1:0/3280433966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74dc0bf9d0 0x7f74dc1536e0 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.194+0000 7f74e0df4640 1 -- 172.21.15.1:0/3280433966 >> 172.21.15.1:0/3280433966 conn(0x7f74dc066ee0 msgr2=0x7f74dc0c0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.194+0000 7f74e0df4640 1 -- 172.21.15.1:0/3280433966 shutdown_connections 2024-03-06T07:29:39.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.194+0000 7f74e0df4640 1 -- 172.21.15.1:0/3280433966 wait complete. 2024-03-06T07:29:39.207 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:39.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.306+0000 7f78824d4640 1 Processor -- start 2024-03-06T07:29:39.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.306+0000 7f78824d4640 1 -- start start 2024-03-06T07:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.307+0000 7f78824d4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f787c0c5960 0x7f787c0c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.307+0000 7f78824d4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787c0c67c0 con 0x7f787c153ab0 2024-03-06T07:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.307+0000 7f78824d4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787c0c6a60 con 0x7f787c0c4ec0 2024-03-06T07:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.307+0000 7f78824d4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f787c0c6d40 con 0x7f787c0c5960 2024-03-06T07:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.307+0000 7f787b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f787c153ab0 0x7f787c070b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35904/0 (socket says 172.21.15.1:35904) 2024-03-06T07:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.307+0000 7f787b7fe640 1 -- 172.21.15.1:0/1281254226 learned_addr learned my addr 172.21.15.1:0/1281254226 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.307+0000 7f787bfff640 1 --2- 172.21.15.1:0/1281254226 >> v2:172.21.15.98:3300/0 conn(0x7f787c0c5960 0x7f787c0c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.307+0000 7f787affd640 1 -- 172.21.15.1:0/1281254226 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 608205661 0 0) 0x7f787c0c67c0 con 0x7f787c153ab0 2024-03-06T07:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.307+0000 7f787affd640 1 -- 172.21.15.1:0/1281254226 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7864003520 con 0x7f787c153ab0 2024-03-06T07:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.307+0000 7f787affd640 1 -- 172.21.15.1:0/1281254226 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2994246417 0 0) 0x7f7864003520 con 0x7f787c153ab0 2024-03-06T07:29:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.307+0000 7f787affd640 1 -- 172.21.15.1:0/1281254226 >> v2:172.21.15.98:3300/0 conn(0x7f787c0c5960 msgr2=0x7f787c0c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.307+0000 7f787affd640 1 --2- 172.21.15.1:0/1281254226 >> v2:172.21.15.98:3300/0 conn(0x7f787c0c5960 0x7f787c0c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.307+0000 7f787affd640 1 -- 172.21.15.1:0/1281254226 >> v1:172.21.15.98:6789/0 conn(0x7f787c0c4ec0 legacy=0x7f787c15d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.307+0000 7f787affd640 1 -- 172.21.15.1:0/1281254226 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787c0c7020 con 0x7f787c153ab0 2024-03-06T07:29:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.307+0000 7f787affd640 1 -- 172.21.15.1:0/1281254226 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f7870001690 con 0x7f787c153ab0 2024-03-06T07:29:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.307+0000 7f787affd640 1 -- 172.21.15.1:0/1281254226 >> v1:172.21.15.1:6789/0 conn(0x7f787c153ab0 legacy=0x7f787c070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.308+0000 7f787affd640 1 --2- 172.21.15.1:0/1281254226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78640039b0 0x7f7864003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.308+0000 7f787affd640 1 --2- 172.21.15.1:0/1281254226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7864004800 0x7f7864004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.308+0000 7f787affd640 1 --2- 172.21.15.1:0/1281254226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78640051c0 0x7f786400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.308+0000 7f787affd640 1 -- 172.21.15.1:0/1281254226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787c0c6d40 con 0x7f78640051c0 2024-03-06T07:29:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.308+0000 7f787affd640 1 -- 172.21.15.1:0/1281254226 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786400c060 con 0x7f7864004800 2024-03-06T07:29:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.308+0000 7f787affd640 1 -- 172.21.15.1:0/1281254226 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786400c260 con 0x7f78640039b0 2024-03-06T07:29:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.308+0000 7f7880a71640 1 --2- 172.21.15.1:0/1281254226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78640051c0 0x7f786400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.308+0000 7f787bfff640 1 --2- 172.21.15.1:0/1281254226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78640039b0 0x7f7864003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.308+0000 7f787b7fe640 1 --2- 172.21.15.1:0/1281254226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7864004800 0x7f7864004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.308+0000 7f7880a71640 1 -- 172.21.15.1:0/1281254226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78640039b0 msgr2=0x7f7864003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.308+0000 7f7880a71640 1 --2- 172.21.15.1:0/1281254226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f78640039b0 0x7f7864003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.308+0000 7f7880a71640 1 -- 172.21.15.1:0/1281254226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7864004800 msgr2=0x7f7864004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.308+0000 7f7880a71640 1 --2- 172.21.15.1:0/1281254226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7864004800 0x7f7864004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.308+0000 7f7880a71640 1 -- 172.21.15.1:0/1281254226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7864003520 con 0x7f78640051c0 2024-03-06T07:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.309+0000 7f7880a71640 1 -- 172.21.15.1:0/1281254226 reap_dead start 2024-03-06T07:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.309+0000 7f7880a71640 1 --2- 172.21.15.1:0/1281254226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78640051c0 0x7f786400b9c0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f787c153070 tx=0x7f7864005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929eee3c35dc34ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.309+0000 7f787affd640 1 -- 172.21.15.1:0/1281254226 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7874001a70 con 0x7f78640051c0 2024-03-06T07:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.309+0000 7f787affd640 1 -- 172.21.15.1:0/1281254226 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7874003b70 con 0x7f78640051c0 2024-03-06T07:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.309+0000 7f787affd640 1 -- 172.21.15.1:0/1281254226 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78740050d0 con 0x7f78640051c0 2024-03-06T07:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.309+0000 7f78824d4640 1 -- 172.21.15.1:0/1281254226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78640051c0 msgr2=0x7f786400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.309+0000 7f78824d4640 1 --2- 172.21.15.1:0/1281254226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78640051c0 0x7f786400b9c0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f787c153070 tx=0x7f7864005830 comp rx=0 tx=0).stop 2024-03-06T07:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.309+0000 7f78824d4640 1 -- 172.21.15.1:0/1281254226 shutdown_connections 2024-03-06T07:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.309+0000 7f78824d4640 1 --2- 172.21.15.1:0/1281254226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78640051c0 0x7f786400b9c0 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.309+0000 7f78824d4640 1 -- 172.21.15.1:0/1281254226 >> 172.21.15.1:0/1281254226 conn(0x7f787c0aa550 msgr2=0x7f787c0aa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.309+0000 7f78824d4640 1 -- 172.21.15.1:0/1281254226 shutdown_connections 2024-03-06T07:29:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.309+0000 7f78824d4640 1 -- 172.21.15.1:0/1281254226 wait complete. 2024-03-06T07:29:39.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.311+0000 7f78824d4640 1 Processor -- start 2024-03-06T07:29:39.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.311+0000 7f78824d4640 1 -- start start 2024-03-06T07:29:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.311+0000 7f78824d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78640051c0 0x7f787c1dfd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.311+0000 7f78824d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787c1eae70 0x7f787c150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.311+0000 7f78824d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f787c151170 0x7f787c1ed960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.311+0000 7f78824d4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787c06f550 con 0x7f78640051c0 2024-03-06T07:29:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.311+0000 7f78824d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787c06f3d0 con 0x7f787c1eae70 2024-03-06T07:29:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.311+0000 7f78824d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f787c0c5d40 con 0x7f787c151170 2024-03-06T07:29:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.312+0000 7f787bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78640051c0 0x7f787c1dfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.312+0000 7f787bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78640051c0 0x7f787c1dfd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56214/0 (socket says 172.21.15.1:56214) 2024-03-06T07:29:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.312+0000 7f787bfff640 1 -- 172.21.15.1:0/966432108 learned_addr learned my addr 172.21.15.1:0/966432108 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.312+0000 7f787b7fe640 1 --2- 172.21.15.1:0/966432108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787c1eae70 0x7f787c150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.312+0000 7f7880a71640 1 --2- 172.21.15.1:0/966432108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f787c151170 0x7f787c1ed960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.312+0000 7f787bfff640 1 -- 172.21.15.1:0/966432108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f787c151170 msgr2=0x7f787c1ed960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.312+0000 7f787bfff640 1 --2- 172.21.15.1:0/966432108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f787c151170 0x7f787c1ed960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.312+0000 7f787bfff640 1 -- 172.21.15.1:0/966432108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787c1eae70 msgr2=0x7f787c150c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.312+0000 7f787bfff640 1 --2- 172.21.15.1:0/966432108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787c1eae70 0x7f787c150c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.312+0000 7f787bfff640 1 -- 172.21.15.1:0/966432108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787c152eb0 con 0x7f78640051c0 2024-03-06T07:29:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.312+0000 7f7880a71640 1 --2- 172.21.15.1:0/966432108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f787c151170 0x7f787c1ed960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.312+0000 7f787bfff640 1 --2- 172.21.15.1:0/966432108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78640051c0 0x7f787c1dfd50 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f786c011910 tx=0x7f786c011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.312+0000 7f7878ff9640 1 -- 172.21.15.1:0/966432108 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786c00fa30 con 0x7f78640051c0 2024-03-06T07:29:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.312+0000 7f7878ff9640 1 -- 172.21.15.1:0/966432108 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786c00fbd0 con 0x7f78640051c0 2024-03-06T07:29:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.312+0000 7f78824d4640 1 -- 172.21.15.1:0/966432108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f787c1ddfd0 con 0x7f78640051c0 2024-03-06T07:29:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.312+0000 7f7878ff9640 1 -- 172.21.15.1:0/966432108 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786c014440 con 0x7f78640051c0 2024-03-06T07:29:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.312+0000 7f78824d4640 1 -- 172.21.15.1:0/966432108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f787c152990 con 0x7f78640051c0 2024-03-06T07:29:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.313+0000 7f78824d4640 1 -- 172.21.15.1:0/966432108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f787c067fd0 con 0x7f78640051c0 2024-03-06T07:29:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.314+0000 7f7878ff9640 1 -- 172.21.15.1:0/966432108 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f786c012040 con 0x7f78640051c0 2024-03-06T07:29:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.314+0000 7f7878ff9640 1 --2- 172.21.15.1:0/966432108 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f786806b500 0x7f786806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.314+0000 7f7878ff9640 1 -- 172.21.15.1:0/966432108 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f786c08a880 con 0x7f78640051c0 2024-03-06T07:29:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.314+0000 7f787b7fe640 1 --2- 172.21.15.1:0/966432108 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f786806b500 0x7f786806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.315+0000 7f787b7fe640 1 --2- 172.21.15.1:0/966432108 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f786806b500 0x7f786806d9c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f787c0c2040 tx=0x7f7870001b30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.319+0000 7f7878ff9640 1 -- 172.21.15.1:0/966432108 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f786c019030 con 0x7f78640051c0 2024-03-06T07:29:39.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.539+0000 7f78824d4640 1 -- 172.21.15.1:0/966432108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f787c0c7a70 con 0x7f78640051c0 2024-03-06T07:29:39.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.540+0000 7f7878ff9640 1 -- 172.21.15.1:0/966432108 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+4987 (secure 0 0 0) 0x7f786c059600 con 0x7f78640051c0 2024-03-06T07:29:39.542 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:39.542 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":196,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":10803,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:35.664209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10806},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:29:39.542 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 196 2024-03-06T07:29:39.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.542+0000 7f78824d4640 1 -- 172.21.15.1:0/966432108 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f786806b500 msgr2=0x7f786806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.542+0000 7f78824d4640 1 --2- 172.21.15.1:0/966432108 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f786806b500 0x7f786806d9c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f787c0c2040 tx=0x7f7870001b30 comp rx=0 tx=0).stop 2024-03-06T07:29:39.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.542+0000 7f78824d4640 1 -- 172.21.15.1:0/966432108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78640051c0 msgr2=0x7f787c1dfd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.542+0000 7f78824d4640 1 --2- 172.21.15.1:0/966432108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78640051c0 0x7f787c1dfd50 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f786c011910 tx=0x7f786c011d10 comp rx=0 tx=0).stop 2024-03-06T07:29:39.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.542+0000 7f78824d4640 1 -- 172.21.15.1:0/966432108 shutdown_connections 2024-03-06T07:29:39.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.542+0000 7f78824d4640 1 --2- 172.21.15.1:0/966432108 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f786806b500 0x7f786806d9c0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.542+0000 7f78824d4640 1 --2- 172.21.15.1:0/966432108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f787c151170 0x7f787c1ed960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.542+0000 7f78824d4640 1 --2- 172.21.15.1:0/966432108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f787c1eae70 0x7f787c150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.542+0000 7f78824d4640 1 --2- 172.21.15.1:0/966432108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78640051c0 0x7f787c1dfd50 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.542+0000 7f78824d4640 1 -- 172.21.15.1:0/966432108 >> 172.21.15.1:0/966432108 conn(0x7f787c0aa550 msgr2=0x7f787c0c25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.542+0000 7f78824d4640 1 -- 172.21.15.1:0/966432108 shutdown_connections 2024-03-06T07:29:39.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.542+0000 7f78824d4640 1 -- 172.21.15.1:0/966432108 wait complete. 2024-03-06T07:29:39.555 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:29:39.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.658+0000 7feb1f688640 1 Processor -- start 2024-03-06T07:29:39.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.658+0000 7feb1f688640 1 -- start start 2024-03-06T07:29:39.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.659+0000 7feb1f688640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb180c32f0 0x7feb181543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.659+0000 7feb1f688640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb18150900 con 0x7feb18070c80 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.659+0000 7feb1f688640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb18071790 con 0x7feb18150520 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.659+0000 7feb1f688640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb18153ed0 con 0x7feb180c32f0 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.659+0000 7feb1dc25640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7feb18070c80 0x7feb1815d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35908/0 (socket says 172.21.15.1:35908) 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.659+0000 7feb1dc25640 1 -- 172.21.15.1:0/2451068630 learned_addr learned my addr 172.21.15.1:0/2451068630 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.659+0000 7feb1cc23640 1 --2- 172.21.15.1:0/2451068630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb180c32f0 0x7feb181543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.659+0000 7feb0ffff640 1 -- 172.21.15.1:0/2451068630 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836201800 0 0) 0x7feb18150900 con 0x7feb18070c80 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.659+0000 7feb0ffff640 1 -- 172.21.15.1:0/2451068630 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb04003520 con 0x7feb18070c80 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.660+0000 7feb0ffff640 1 -- 172.21.15.1:0/2451068630 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121979291 0 0) 0x7feb18071790 con 0x7feb18150520 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.660+0000 7feb0ffff640 1 -- 172.21.15.1:0/2451068630 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb18150900 con 0x7feb18150520 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.660+0000 7feb1cc23640 1 -- 172.21.15.1:0/2451068630 >> v1:172.21.15.98:6789/0 conn(0x7feb18150520 legacy=0x7feb181537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.660+0000 7feb0ffff640 1 -- 172.21.15.1:0/2451068630 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1815054213 0 0) 0x7feb04003520 con 0x7feb18070c80 2024-03-06T07:29:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.660+0000 7feb1cc23640 1 -- 172.21.15.1:0/2451068630 >> v1:172.21.15.1:6789/0 conn(0x7feb18070c80 legacy=0x7feb1815d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.660+0000 7feb1cc23640 1 -- 172.21.15.1:0/2451068630 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb18157050 con 0x7feb180c32f0 2024-03-06T07:29:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.660+0000 7feb1cc23640 1 --2- 172.21.15.1:0/2451068630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb180c32f0 0x7feb181543a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feb080097d0 tx=0x7feb0802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98b7a59e1ed86383 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.661+0000 7feb0ffff640 1 -- 172.21.15.1:0/2451068630 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08037070 con 0x7feb180c32f0 2024-03-06T07:29:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.661+0000 7feb0ffff640 1 -- 172.21.15.1:0/2451068630 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb08030510 con 0x7feb180c32f0 2024-03-06T07:29:39.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.661+0000 7feb0ffff640 1 -- 172.21.15.1:0/2451068630 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb100019f0 con 0x7feb180c32f0 2024-03-06T07:29:39.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.661+0000 7feb1f688640 1 -- 172.21.15.1:0/2451068630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb180c32f0 msgr2=0x7feb181543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.661+0000 7feb1f688640 1 --2- 172.21.15.1:0/2451068630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb180c32f0 0x7feb181543a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feb080097d0 tx=0x7feb0802b800 comp rx=0 tx=0).stop 2024-03-06T07:29:39.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.661+0000 7feb1f688640 1 -- 172.21.15.1:0/2451068630 shutdown_connections 2024-03-06T07:29:39.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.661+0000 7feb1f688640 1 --2- 172.21.15.1:0/2451068630 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb180c32f0 0x7feb181543a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.661+0000 7feb1f688640 1 -- 172.21.15.1:0/2451068630 >> 172.21.15.1:0/2451068630 conn(0x7feb180c88f0 msgr2=0x7feb180c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.661+0000 7feb1f688640 1 -- 172.21.15.1:0/2451068630 shutdown_connections 2024-03-06T07:29:39.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.661+0000 7feb1f688640 1 -- 172.21.15.1:0/2451068630 wait complete. 2024-03-06T07:29:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.662+0000 7feb1f688640 1 Processor -- start 2024-03-06T07:29:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.662+0000 7feb1f688640 1 -- start start 2024-03-06T07:29:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.663+0000 7feb1f688640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb18070c80 0x7feb18145900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.663+0000 7feb1f688640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb180c32f0 0x7feb1814df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.663+0000 7feb1f688640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb18150520 0x7feb1814e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.663+0000 7feb1f688640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb18157330 con 0x7feb18070c80 2024-03-06T07:29:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.663+0000 7feb1f688640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb181532a0 con 0x7feb18150520 2024-03-06T07:29:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.663+0000 7feb1f688640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb180c7ec0 con 0x7feb180c32f0 2024-03-06T07:29:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.663+0000 7feb1d424640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb18070c80 0x7feb18145900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.663+0000 7feb1d424640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb18070c80 0x7feb18145900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56224/0 (socket says 172.21.15.1:56224) 2024-03-06T07:29:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.663+0000 7feb1d424640 1 -- 172.21.15.1:0/676058293 learned_addr learned my addr 172.21.15.1:0/676058293 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.663+0000 7feb1cc23640 1 --2- 172.21.15.1:0/676058293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb180c32f0 0x7feb1814df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.663+0000 7feb1dc25640 1 --2- 172.21.15.1:0/676058293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb18150520 0x7feb1814e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.664+0000 7feb1d424640 1 -- 172.21.15.1:0/676058293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb180c32f0 msgr2=0x7feb1814df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.664+0000 7feb1d424640 1 --2- 172.21.15.1:0/676058293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb180c32f0 0x7feb1814df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.664+0000 7feb1d424640 1 -- 172.21.15.1:0/676058293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb18150520 msgr2=0x7feb1814e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.664+0000 7feb1d424640 1 --2- 172.21.15.1:0/676058293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb18150520 0x7feb1814e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.664+0000 7feb1d424640 1 -- 172.21.15.1:0/676058293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb181ec7e0 con 0x7feb18070c80 2024-03-06T07:29:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.664+0000 7feb1dc25640 1 --2- 172.21.15.1:0/676058293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb18150520 0x7feb1814e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.664+0000 7feb1d424640 1 --2- 172.21.15.1:0/676058293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb18070c80 0x7feb18145900 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7feb1400c460 tx=0x7feb1400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.664+0000 7feb0dffb640 1 -- 172.21.15.1:0/676058293 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb140052d0 con 0x7feb18070c80 2024-03-06T07:29:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.664+0000 7feb0dffb640 1 -- 172.21.15.1:0/676058293 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb14005490 con 0x7feb18070c80 2024-03-06T07:29:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.664+0000 7feb1f688640 1 -- 172.21.15.1:0/676058293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb181eba20 con 0x7feb18070c80 2024-03-06T07:29:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.664+0000 7feb0dffb640 1 -- 172.21.15.1:0/676058293 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb14012870 con 0x7feb18070c80 2024-03-06T07:29:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.664+0000 7feb1f688640 1 -- 172.21.15.1:0/676058293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb181ebfe0 con 0x7feb18070c80 2024-03-06T07:29:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.665+0000 7feb1f688640 1 -- 172.21.15.1:0/676058293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb180686e0 con 0x7feb18070c80 2024-03-06T07:29:39.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.666+0000 7feb0dffb640 1 -- 172.21.15.1:0/676058293 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb14012a10 con 0x7feb18070c80 2024-03-06T07:29:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.666+0000 7feb0dffb640 1 --2- 172.21.15.1:0/676058293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feafc06b500 0x7feafc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.666+0000 7feb0dffb640 1 -- 172.21.15.1:0/676058293 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feb14089e60 con 0x7feb18070c80 2024-03-06T07:29:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.666+0000 7feb1cc23640 1 --2- 172.21.15.1:0/676058293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feafc06b500 0x7feafc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.667+0000 7feb1cc23640 1 --2- 172.21.15.1:0/676058293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feafc06b500 0x7feafc06d9c0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7feb08034000 tx=0x7feb08008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.672+0000 7feb0dffb640 1 -- 172.21.15.1:0/676058293 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb14058be0 con 0x7feb18070c80 2024-03-06T07:29:39.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:39.894+0000 7feb1f688640 1 -- 172.21.15.1:0/676058293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7feb180c7b10 con 0x7feb18070c80 2024-03-06T07:29:40.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.533+0000 7feb0dffb640 1 -- 172.21.15.1:0/676058293 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v197) v1 ==== 105+0+0 (secure 0 0 0) 0x7feb140584e0 con 0x7feb18070c80 2024-03-06T07:29:40.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.535+0000 7feb1f688640 1 -- 172.21.15.1:0/676058293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feafc06b500 msgr2=0x7feafc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.535+0000 7feb1f688640 1 --2- 172.21.15.1:0/676058293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feafc06b500 0x7feafc06d9c0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7feb08034000 tx=0x7feb08008030 comp rx=0 tx=0).stop 2024-03-06T07:29:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.535+0000 7feb1f688640 1 -- 172.21.15.1:0/676058293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb18070c80 msgr2=0x7feb18145900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.535+0000 7feb1f688640 1 --2- 172.21.15.1:0/676058293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb18070c80 0x7feb18145900 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7feb1400c460 tx=0x7feb1400c920 comp rx=0 tx=0).stop 2024-03-06T07:29:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.535+0000 7feb1f688640 1 -- 172.21.15.1:0/676058293 shutdown_connections 2024-03-06T07:29:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.535+0000 7feb1f688640 1 --2- 172.21.15.1:0/676058293 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feafc06b500 0x7feafc06d9c0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.535+0000 7feb1f688640 1 --2- 172.21.15.1:0/676058293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb18150520 0x7feb1814e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.535+0000 7feb1f688640 1 --2- 172.21.15.1:0/676058293 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb180c32f0 0x7feb1814df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.535+0000 7feb1f688640 1 --2- 172.21.15.1:0/676058293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb18070c80 0x7feb18145900 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.535+0000 7feb1f688640 1 -- 172.21.15.1:0/676058293 >> 172.21.15.1:0/676058293 conn(0x7feb180c88f0 msgr2=0x7feb1814b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.535+0000 7feb1f688640 1 -- 172.21.15.1:0/676058293 shutdown_connections 2024-03-06T07:29:40.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.535+0000 7feb1f688640 1 -- 172.21.15.1:0/676058293 wait complete. 2024-03-06T07:29:40.552 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:40.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.652+0000 7f5251a57640 1 Processor -- start 2024-03-06T07:29:40.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.652+0000 7f5251a57640 1 -- start start 2024-03-06T07:29:40.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.653+0000 7f5251a57640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f524c0c7560 0x7f524c155860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.653+0000 7f5251a57640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f524c155da0 0x7f524c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.653+0000 7f5251a57640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f524c155080 con 0x7f524c152890 2024-03-06T07:29:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.653+0000 7f5251a57640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f524c0c35b0 con 0x7f524c0c7560 2024-03-06T07:29:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.653+0000 7f5251a57640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f524c0c3890 con 0x7f524c155da0 2024-03-06T07:29:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.653+0000 7f524a7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f524c0c7560 0x7f524c155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.653+0000 7f524a7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f524c0c7560 0x7f524c155860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56230/0 (socket says 172.21.15.1:56230) 2024-03-06T07:29:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.653+0000 7f524a7fc640 1 -- 172.21.15.1:0/1893141346 learned_addr learned my addr 172.21.15.1:0/1893141346 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.653+0000 7f524affd640 1 --1- 172.21.15.1:0/1893141346 >> v1:172.21.15.1:6789/0 conn(0x7f524c152890 0x7f524c159030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35924/0 (socket says 172.21.15.1:35924) 2024-03-06T07:29:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.653+0000 7f524b7fe640 1 --2- 172.21.15.1:0/1893141346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f524c155da0 0x7f524c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.653+0000 7f524a7fc640 1 -- 172.21.15.1:0/1893141346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f524c155da0 msgr2=0x7f524c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.653+0000 7f524a7fc640 1 --2- 172.21.15.1:0/1893141346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f524c155da0 0x7f524c0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.653+0000 7f524a7fc640 1 -- 172.21.15.1:0/1893141346 >> v1:172.21.15.1:6789/0 conn(0x7f524c152890 legacy=0x7f524c159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.653+0000 7f524a7fc640 1 -- 172.21.15.1:0/1893141346 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f524c0c3b70 con 0x7f524c0c7560 2024-03-06T07:29:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.653+0000 7f5249ffb640 1 -- 172.21.15.1:0/1893141346 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930899814 0 0) 0x7f524c155080 con 0x7f524c152890 2024-03-06T07:29:40.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.653+0000 7f524b7fe640 1 --2- 172.21.15.1:0/1893141346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f524c155da0 0x7f524c0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.654+0000 7f524a7fc640 1 --2- 172.21.15.1:0/1893141346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f524c0c7560 0x7f524c155860 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f523c009870 tx=0x7f523c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a156a477f94b22b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.654+0000 7f5249ffb640 1 -- 172.21.15.1:0/1893141346 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523c0317c0 con 0x7f524c0c7560 2024-03-06T07:29:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.654+0000 7f5249ffb640 1 -- 172.21.15.1:0/1893141346 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523c037070 con 0x7f524c0c7560 2024-03-06T07:29:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.654+0000 7f5249ffb640 1 -- 172.21.15.1:0/1893141346 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523c031b80 con 0x7f524c0c7560 2024-03-06T07:29:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.655+0000 7f5251a57640 1 -- 172.21.15.1:0/1893141346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f524c0c7560 msgr2=0x7f524c155860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.655+0000 7f5251a57640 1 --2- 172.21.15.1:0/1893141346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f524c0c7560 0x7f524c155860 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f523c009870 tx=0x7f523c02b860 comp rx=0 tx=0).stop 2024-03-06T07:29:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.655+0000 7f5251a57640 1 -- 172.21.15.1:0/1893141346 shutdown_connections 2024-03-06T07:29:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.655+0000 7f5251a57640 1 --2- 172.21.15.1:0/1893141346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f524c155da0 0x7f524c0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.655+0000 7f5251a57640 1 --2- 172.21.15.1:0/1893141346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f524c0c7560 0x7f524c155860 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.655+0000 7f5251a57640 1 -- 172.21.15.1:0/1893141346 >> 172.21.15.1:0/1893141346 conn(0x7f524c06b3e0 msgr2=0x7f524c06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.655+0000 7f5251a57640 1 -- 172.21.15.1:0/1893141346 shutdown_connections 2024-03-06T07:29:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.655+0000 7f5251a57640 1 -- 172.21.15.1:0/1893141346 wait complete. 2024-03-06T07:29:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.656+0000 7f5251a57640 1 Processor -- start 2024-03-06T07:29:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.657+0000 7f5251a57640 1 -- start start 2024-03-06T07:29:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f5251a57640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f524c0c7560 0x7f524c1e1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f5251a57640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f524c152890 0x7f524c06f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f5251a57640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f524c06fc80 0x7f524c1edd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f5251a57640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524c0a81f0 con 0x7f524c06fc80 2024-03-06T07:29:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f5251a57640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524c0c3260 con 0x7f524c152890 2024-03-06T07:29:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f5251a57640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f524c0a8370 con 0x7f524c0c7560 2024-03-06T07:29:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f524b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f524c06fc80 0x7f524c1edd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f524b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f524c06fc80 0x7f524c1edd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56234/0 (socket says 172.21.15.1:56234) 2024-03-06T07:29:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f524b7fe640 1 -- 172.21.15.1:0/892991620 learned_addr learned my addr 172.21.15.1:0/892991620 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f524affd640 1 --2- 172.21.15.1:0/892991620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f524c0c7560 0x7f524c1e1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f524a7fc640 1 --2- 172.21.15.1:0/892991620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f524c152890 0x7f524c06f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f524b7fe640 1 -- 172.21.15.1:0/892991620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f524c0c7560 msgr2=0x7f524c1e1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f524b7fe640 1 --2- 172.21.15.1:0/892991620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f524c0c7560 0x7f524c1e1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f524b7fe640 1 -- 172.21.15.1:0/892991620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f524c152890 msgr2=0x7f524c06f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f524b7fe640 1 --2- 172.21.15.1:0/892991620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f524c152890 0x7f524c06f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f524b7fe640 1 -- 172.21.15.1:0/892991620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f524c0ac860 con 0x7f524c06fc80 2024-03-06T07:29:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f524b7fe640 1 --2- 172.21.15.1:0/892991620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f524c06fc80 0x7f524c1edd80 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f524400cf20 tx=0x7f524400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f5233fff640 1 -- 172.21.15.1:0/892991620 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524401a070 con 0x7f524c06fc80 2024-03-06T07:29:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f5233fff640 1 -- 172.21.15.1:0/892991620 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f524400d840 con 0x7f524c06fc80 2024-03-06T07:29:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f5251a57640 1 -- 172.21.15.1:0/892991620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f524c0acdb0 con 0x7f524c06fc80 2024-03-06T07:29:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.658+0000 7f5233fff640 1 -- 172.21.15.1:0/892991620 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524400dc30 con 0x7f524c06fc80 2024-03-06T07:29:40.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.659+0000 7f5251a57640 1 -- 172.21.15.1:0/892991620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f524c0ad050 con 0x7f524c06fc80 2024-03-06T07:29:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.660+0000 7f5251a57640 1 -- 172.21.15.1:0/892991620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f524c067fd0 con 0x7f524c06fc80 2024-03-06T07:29:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.660+0000 7f5233fff640 1 -- 172.21.15.1:0/892991620 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f524401d070 con 0x7f524c06fc80 2024-03-06T07:29:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.660+0000 7f5233fff640 1 --2- 172.21.15.1:0/892991620 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f523806b4d0 0x7f523806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.660+0000 7f5233fff640 1 -- 172.21.15.1:0/892991620 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f524408aa80 con 0x7f524c06fc80 2024-03-06T07:29:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.661+0000 7f524affd640 1 --2- 172.21.15.1:0/892991620 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f523806b4d0 0x7f523806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.661+0000 7f524affd640 1 --2- 172.21.15.1:0/892991620 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f523806b4d0 0x7f523806d990 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f5240000f60 tx=0x7f5240007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.664+0000 7f5233fff640 1 -- 172.21.15.1:0/892991620 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52440591b0 con 0x7f524c06fc80 2024-03-06T07:29:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.883+0000 7f5251a57640 1 -- 172.21.15.1:0/892991620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f524c14c650 con 0x7f524c06fc80 2024-03-06T07:29:40.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.883+0000 7f5233fff640 1 -- 172.21.15.1:0/892991620 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+5005 (secure 0 0 0) 0x7f524405d250 con 0x7f524c06fc80 2024-03-06T07:29:40.885 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:40.886 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":198,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:40.536122+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:creating","state_seq":1,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:29:40.886 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 198 2024-03-06T07:29:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.885+0000 7f5251a57640 1 -- 172.21.15.1:0/892991620 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f523806b4d0 msgr2=0x7f523806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.885+0000 7f5251a57640 1 --2- 172.21.15.1:0/892991620 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f523806b4d0 0x7f523806d990 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f5240000f60 tx=0x7f5240007d50 comp rx=0 tx=0).stop 2024-03-06T07:29:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.885+0000 7f5251a57640 1 -- 172.21.15.1:0/892991620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f524c06fc80 msgr2=0x7f524c1edd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.885+0000 7f5251a57640 1 --2- 172.21.15.1:0/892991620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f524c06fc80 0x7f524c1edd80 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f524400cf20 tx=0x7f524400a7d0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.886+0000 7f5251a57640 1 -- 172.21.15.1:0/892991620 shutdown_connections 2024-03-06T07:29:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.886+0000 7f5251a57640 1 --2- 172.21.15.1:0/892991620 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f523806b4d0 0x7f523806d990 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.886+0000 7f5251a57640 1 --2- 172.21.15.1:0/892991620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f524c06fc80 0x7f524c1edd80 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.886+0000 7f5251a57640 1 --2- 172.21.15.1:0/892991620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f524c152890 0x7f524c06f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.886+0000 7f5251a57640 1 --2- 172.21.15.1:0/892991620 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f524c0c7560 0x7f524c1e1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.886+0000 7f5251a57640 1 -- 172.21.15.1:0/892991620 >> 172.21.15.1:0/892991620 conn(0x7f524c06b3e0 msgr2=0x7f524c06b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.886+0000 7f5251a57640 1 -- 172.21.15.1:0/892991620 shutdown_connections 2024-03-06T07:29:40.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:40.886+0000 7f5251a57640 1 -- 172.21.15.1:0/892991620 wait complete. 2024-03-06T07:29:41.301 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 198, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:40.536122+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:29:41.301 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:creating 2024-03-06T07:29:42.302 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:42.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.411+0000 7fb7928eb640 1 Processor -- start 2024-03-06T07:29:42.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.411+0000 7fb7928eb640 1 -- start start 2024-03-06T07:29:42.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.412+0000 7fb7928eb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb78c152650 0x7fb78c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.412+0000 7fb7928eb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb78c0bf9d0 0x7fb78c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.412+0000 7fb7928eb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78c0c2fc0 con 0x7fb78c0c6b10 2024-03-06T07:29:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.412+0000 7fb7928eb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb78c155800 con 0x7fb78c152650 2024-03-06T07:29:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.412+0000 7fb7928eb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb78c158da0 con 0x7fb78c0bf9d0 2024-03-06T07:29:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.412+0000 7fb78bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb78c152650 0x7fb78c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.412+0000 7fb78bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb78c152650 0x7fb78c155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56244/0 (socket says 172.21.15.1:56244) 2024-03-06T07:29:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.412+0000 7fb78bfff640 1 -- 172.21.15.1:0/1804955918 learned_addr learned my addr 172.21.15.1:0/1804955918 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.413+0000 7fb790e88640 1 --2- 172.21.15.1:0/1804955918 >> v2:172.21.15.98:3300/0 conn(0x7fb78c0bf9d0 0x7fb78c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.413+0000 7fb78bfff640 1 -- 172.21.15.1:0/1804955918 >> v2:172.21.15.98:3300/0 conn(0x7fb78c0bf9d0 msgr2=0x7fb78c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.413+0000 7fb78bfff640 1 --2- 172.21.15.1:0/1804955918 >> v2:172.21.15.98:3300/0 conn(0x7fb78c0bf9d0 0x7fb78c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.413+0000 7fb78bfff640 1 -- 172.21.15.1:0/1804955918 >> v1:172.21.15.98:6789/0 conn(0x7fb78c0c6b10 legacy=0x7fb78c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.413+0000 7fb78bfff640 1 -- 172.21.15.1:0/1804955918 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c0c6690 con 0x7fb78c152650 2024-03-06T07:29:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.413+0000 7fb78bfff640 1 --2- 172.21.15.1:0/1804955918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78c152650 0x7fb78c155c90 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7fb78c1597b0 tx=0x7fb77402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6960358ab13897d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.414+0000 7fb78affd640 1 -- 172.21.15.1:0/1804955918 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77402e070 con 0x7fb78c152650 2024-03-06T07:29:42.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.414+0000 7fb78affd640 1 -- 172.21.15.1:0/1804955918 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb774037070 con 0x7fb78c152650 2024-03-06T07:29:42.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.414+0000 7fb78affd640 1 -- 172.21.15.1:0/1804955918 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb774031a00 con 0x7fb78c152650 2024-03-06T07:29:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.414+0000 7fb7928eb640 1 -- 172.21.15.1:0/1804955918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78c152650 msgr2=0x7fb78c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.414+0000 7fb7928eb640 1 --2- 172.21.15.1:0/1804955918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78c152650 0x7fb78c155c90 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7fb78c1597b0 tx=0x7fb77402b880 comp rx=0 tx=0).stop 2024-03-06T07:29:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.414+0000 7fb7928eb640 1 -- 172.21.15.1:0/1804955918 shutdown_connections 2024-03-06T07:29:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.414+0000 7fb7928eb640 1 --2- 172.21.15.1:0/1804955918 >> v2:172.21.15.98:3300/0 conn(0x7fb78c0bf9d0 0x7fb78c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.414+0000 7fb7928eb640 1 --2- 172.21.15.1:0/1804955918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78c152650 0x7fb78c155c90 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.414+0000 7fb7928eb640 1 -- 172.21.15.1:0/1804955918 >> 172.21.15.1:0/1804955918 conn(0x7fb78c066ee0 msgr2=0x7fb78c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.414+0000 7fb7928eb640 1 -- 172.21.15.1:0/1804955918 shutdown_connections 2024-03-06T07:29:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.414+0000 7fb7928eb640 1 -- 172.21.15.1:0/1804955918 wait complete. 2024-03-06T07:29:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.415+0000 7fb7928eb640 1 Processor -- start 2024-03-06T07:29:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.415+0000 7fb7928eb640 1 -- start start 2024-03-06T07:29:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.416+0000 7fb7928eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb78c0bf9d0 0x7fb78c14a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.416+0000 7fb7928eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb78c152650 0x7fb78c152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.416+0000 7fb7928eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78c153220 0x7fb78c153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.416+0000 7fb7928eb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c1551b0 con 0x7fb78c153220 2024-03-06T07:29:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.416+0000 7fb7928eb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0c6fa0 con 0x7fb78c0bf9d0 2024-03-06T07:29:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.416+0000 7fb7928eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c155490 con 0x7fb78c152650 2024-03-06T07:29:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.416+0000 7fb790e88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78c153220 0x7fb78c153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.416+0000 7fb790e88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78c153220 0x7fb78c153a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56256/0 (socket says 172.21.15.1:56256) 2024-03-06T07:29:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.416+0000 7fb790e88640 1 -- 172.21.15.1:0/981424357 learned_addr learned my addr 172.21.15.1:0/981424357 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.416+0000 7fb78bfff640 1 --2- 172.21.15.1:0/981424357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb78c0bf9d0 0x7fb78c14a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.416+0000 7fb78b7fe640 1 --2- 172.21.15.1:0/981424357 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb78c152650 0x7fb78c152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.417+0000 7fb790e88640 1 -- 172.21.15.1:0/981424357 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb78c152650 msgr2=0x7fb78c152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.417+0000 7fb790e88640 1 --2- 172.21.15.1:0/981424357 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb78c152650 0x7fb78c152ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.417+0000 7fb790e88640 1 -- 172.21.15.1:0/981424357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb78c0bf9d0 msgr2=0x7fb78c14a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.417+0000 7fb790e88640 1 --2- 172.21.15.1:0/981424357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb78c0bf9d0 0x7fb78c14a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.417+0000 7fb790e88640 1 -- 172.21.15.1:0/981424357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c158b50 con 0x7fb78c153220 2024-03-06T07:29:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.417+0000 7fb78bfff640 1 --2- 172.21.15.1:0/981424357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb78c0bf9d0 0x7fb78c14a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.417+0000 7fb790e88640 1 --2- 172.21.15.1:0/981424357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78c153220 0x7fb78c153a10 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7fb784007e60 tx=0x7fb78400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.417+0000 7fb788ff9640 1 -- 172.21.15.1:0/981424357 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784014420 con 0x7fb78c153220 2024-03-06T07:29:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.417+0000 7fb788ff9640 1 -- 172.21.15.1:0/981424357 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb784019070 con 0x7fb78c153220 2024-03-06T07:29:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.417+0000 7fb788ff9640 1 -- 172.21.15.1:0/981424357 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784014820 con 0x7fb78c153220 2024-03-06T07:29:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.417+0000 7fb7928eb640 1 -- 172.21.15.1:0/981424357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78c158870 con 0x7fb78c153220 2024-03-06T07:29:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.417+0000 7fb7928eb640 1 -- 172.21.15.1:0/981424357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78c1ebb00 con 0x7fb78c153220 2024-03-06T07:29:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.418+0000 7fb7928eb640 1 -- 172.21.15.1:0/981424357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78c067fd0 con 0x7fb78c153220 2024-03-06T07:29:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.419+0000 7fb788ff9640 1 -- 172.21.15.1:0/981424357 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7840149c0 con 0x7fb78c153220 2024-03-06T07:29:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.419+0000 7fb788ff9640 1 --2- 172.21.15.1:0/981424357 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb76c06b500 0x7fb76c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.420+0000 7fb788ff9640 1 -- 172.21.15.1:0/981424357 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb78408aa60 con 0x7fb78c153220 2024-03-06T07:29:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.420+0000 7fb78bfff640 1 --2- 172.21.15.1:0/981424357 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb76c06b500 0x7fb76c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.420+0000 7fb78bfff640 1 --2- 172.21.15.1:0/981424357 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb76c06b500 0x7fb76c06d9c0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fb774036000 tx=0x7fb77403d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.424+0000 7fb788ff9640 1 -- 172.21.15.1:0/981424357 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7840597e0 con 0x7fb78c153220 2024-03-06T07:29:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.644+0000 7fb7928eb640 1 -- 172.21.15.1:0/981424357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb78c0c4560 con 0x7fb78c153220 2024-03-06T07:29:42.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.645+0000 7fb788ff9640 1 -- 172.21.15.1:0/981424357 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+5009 (secure 0 0 0) 0x7fb7840590e0 con 0x7fb78c153220 2024-03-06T07:29:42.647 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:42.647 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":199,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:41.541766+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:active","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10803,10806]},"id":16}]} 2024-03-06T07:29:42.647 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 199 2024-03-06T07:29:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.647+0000 7fb7928eb640 1 -- 172.21.15.1:0/981424357 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb76c06b500 msgr2=0x7fb76c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.647+0000 7fb7928eb640 1 --2- 172.21.15.1:0/981424357 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb76c06b500 0x7fb76c06d9c0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fb774036000 tx=0x7fb77403d040 comp rx=0 tx=0).stop 2024-03-06T07:29:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.647+0000 7fb7928eb640 1 -- 172.21.15.1:0/981424357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78c153220 msgr2=0x7fb78c153a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.647+0000 7fb7928eb640 1 --2- 172.21.15.1:0/981424357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78c153220 0x7fb78c153a10 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7fb784007e60 tx=0x7fb78400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.647+0000 7fb7928eb640 1 -- 172.21.15.1:0/981424357 shutdown_connections 2024-03-06T07:29:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.647+0000 7fb7928eb640 1 --2- 172.21.15.1:0/981424357 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb76c06b500 0x7fb76c06d9c0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.647+0000 7fb7928eb640 1 --2- 172.21.15.1:0/981424357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78c153220 0x7fb78c153a10 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.647+0000 7fb7928eb640 1 --2- 172.21.15.1:0/981424357 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb78c152650 0x7fb78c152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.647+0000 7fb7928eb640 1 --2- 172.21.15.1:0/981424357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb78c0bf9d0 0x7fb78c14a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.647+0000 7fb7928eb640 1 -- 172.21.15.1:0/981424357 >> 172.21.15.1:0/981424357 conn(0x7fb78c066ee0 msgr2=0x7fb78c0c03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.647+0000 7fb7928eb640 1 -- 172.21.15.1:0/981424357 shutdown_connections 2024-03-06T07:29:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:42.647+0000 7fb7928eb640 1 -- 172.21.15.1:0/981424357 wait complete. 2024-03-06T07:29:43.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 199, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:41.541766+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10803, 10806]} 2024-03-06T07:29:43.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-03-06T07:29:43.065 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:29:43.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.205+0000 7f5c50d7a640 1 Processor -- start 2024-03-06T07:29:43.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.205+0000 7f5c50d7a640 1 -- start start 2024-03-06T07:29:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.206+0000 7f5c50d7a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c4c152650 0x7f5c4c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.206+0000 7f5c50d7a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c4c0bf9d0 0x7f5c4c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.206+0000 7f5c50d7a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c4c0c2fc0 con 0x7f5c4c0c6b10 2024-03-06T07:29:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.206+0000 7f5c50d7a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c4c155800 con 0x7f5c4c152650 2024-03-06T07:29:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.206+0000 7f5c50d7a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c158da0 con 0x7f5c4c0bf9d0 2024-03-06T07:29:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.206+0000 7f5c4a59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c4c152650 0x7f5c4c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.206+0000 7f5c41d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5c4c0c6b10 0x7f5c4c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35938/0 (socket says 172.21.15.1:35938) 2024-03-06T07:29:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.206+0000 7f5c4a59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c4c152650 0x7f5c4c155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56262/0 (socket says 172.21.15.1:56262) 2024-03-06T07:29:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.206+0000 7f5c41d9b640 1 -- 172.21.15.1:0/992406043 learned_addr learned my addr 172.21.15.1:0/992406043 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.206+0000 7f5c4ad9d640 1 --2- 172.21.15.1:0/992406043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c4c0bf9d0 0x7f5c4c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.206+0000 7f5c4a59c640 1 -- 172.21.15.1:0/992406043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c4c0bf9d0 msgr2=0x7f5c4c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.206+0000 7f5c4a59c640 1 --2- 172.21.15.1:0/992406043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c4c0bf9d0 0x7f5c4c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.206+0000 7f5c4a59c640 1 -- 172.21.15.1:0/992406043 >> v1:172.21.15.1:6789/0 conn(0x7f5c4c0c6b10 legacy=0x7f5c4c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.206+0000 7f5c4a59c640 1 -- 172.21.15.1:0/992406043 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c0c6690 con 0x7f5c4c152650 2024-03-06T07:29:43.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.206+0000 7f5c49d9b640 1 -- 172.21.15.1:0/992406043 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938048384 0 0) 0x7f5c4c0c2fc0 con 0x7f5c4c0c6b10 2024-03-06T07:29:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.207+0000 7f5c4a59c640 1 --2- 172.21.15.1:0/992406043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c4c152650 0x7f5c4c155c90 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c0097a0 tx=0x7f5c3c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd6cf1e00179e5a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.207+0000 7f5c49d9b640 1 -- 172.21.15.1:0/992406043 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c0315a0 con 0x7f5c4c152650 2024-03-06T07:29:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.207+0000 7f5c49d9b640 1 -- 172.21.15.1:0/992406043 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3c037070 con 0x7f5c4c152650 2024-03-06T07:29:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.207+0000 7f5c49d9b640 1 -- 172.21.15.1:0/992406043 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c031960 con 0x7f5c4c152650 2024-03-06T07:29:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.208+0000 7f5c50d7a640 1 -- 172.21.15.1:0/992406043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c4c152650 msgr2=0x7f5c4c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.208+0000 7f5c50d7a640 1 --2- 172.21.15.1:0/992406043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c4c152650 0x7f5c4c155c90 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c0097a0 tx=0x7f5c3c02b840 comp rx=0 tx=0).stop 2024-03-06T07:29:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.208+0000 7f5c50d7a640 1 -- 172.21.15.1:0/992406043 shutdown_connections 2024-03-06T07:29:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.208+0000 7f5c50d7a640 1 --2- 172.21.15.1:0/992406043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c4c0bf9d0 0x7f5c4c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.208+0000 7f5c50d7a640 1 --2- 172.21.15.1:0/992406043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c4c152650 0x7f5c4c155c90 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.208+0000 7f5c50d7a640 1 -- 172.21.15.1:0/992406043 >> 172.21.15.1:0/992406043 conn(0x7f5c4c066ee0 msgr2=0x7f5c4c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.208+0000 7f5c50d7a640 1 -- 172.21.15.1:0/992406043 shutdown_connections 2024-03-06T07:29:43.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.208+0000 7f5c50d7a640 1 -- 172.21.15.1:0/992406043 wait complete. 2024-03-06T07:29:43.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.210+0000 7f5c50d7a640 1 Processor -- start 2024-03-06T07:29:43.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.210+0000 7f5c50d7a640 1 -- start start 2024-03-06T07:29:43.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.211+0000 7f5c50d7a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c4c0bf9d0 0x7f5c4c1ee580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.211+0000 7f5c50d7a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c4c152650 0x7f5c4c1e0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.211+0000 7f5c50d7a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c4c1eeac0 0x7f5c4c1e11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.211+0000 7f5c50d7a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c1551b0 con 0x7f5c4c152650 2024-03-06T07:29:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.211+0000 7f5c50d7a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c0c6fa0 con 0x7f5c4c0bf9d0 2024-03-06T07:29:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.211+0000 7f5c50d7a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c155490 con 0x7f5c4c1eeac0 2024-03-06T07:29:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.211+0000 7f5c41d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c4c152650 0x7f5c4c1e0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.211+0000 7f5c41d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c4c152650 0x7f5c4c1e0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56272/0 (socket says 172.21.15.1:56272) 2024-03-06T07:29:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.211+0000 7f5c41d9b640 1 -- 172.21.15.1:0/624570912 learned_addr learned my addr 172.21.15.1:0/624570912 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.212+0000 7f5c4a59c640 1 --2- 172.21.15.1:0/624570912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c4c0bf9d0 0x7f5c4c1ee580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.212+0000 7f5c4ad9d640 1 --2- 172.21.15.1:0/624570912 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c4c1eeac0 0x7f5c4c1e11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.212+0000 7f5c41d9b640 1 -- 172.21.15.1:0/624570912 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c4c1eeac0 msgr2=0x7f5c4c1e11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.212+0000 7f5c41d9b640 1 --2- 172.21.15.1:0/624570912 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c4c1eeac0 0x7f5c4c1e11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.212+0000 7f5c41d9b640 1 -- 172.21.15.1:0/624570912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c4c0bf9d0 msgr2=0x7f5c4c1ee580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.212+0000 7f5c41d9b640 1 --2- 172.21.15.1:0/624570912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c4c0bf9d0 0x7f5c4c1ee580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.212+0000 7f5c41d9b640 1 -- 172.21.15.1:0/624570912 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c156a50 con 0x7f5c4c152650 2024-03-06T07:29:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.212+0000 7f5c4a59c640 1 --2- 172.21.15.1:0/624570912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c4c0bf9d0 0x7f5c4c1ee580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.212+0000 7f5c41d9b640 1 --2- 172.21.15.1:0/624570912 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c4c152650 0x7f5c4c1e0c90 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f5c3400c4d0 tx=0x7f5c3400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.212+0000 7f5c437fe640 1 -- 172.21.15.1:0/624570912 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c34017070 con 0x7f5c4c152650 2024-03-06T07:29:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.212+0000 7f5c437fe640 1 -- 172.21.15.1:0/624570912 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c34005d00 con 0x7f5c4c152650 2024-03-06T07:29:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.212+0000 7f5c437fe640 1 -- 172.21.15.1:0/624570912 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c340072b0 con 0x7f5c4c152650 2024-03-06T07:29:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.212+0000 7f5c50d7a640 1 -- 172.21.15.1:0/624570912 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c4c1ef090 con 0x7f5c4c152650 2024-03-06T07:29:43.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.212+0000 7f5c50d7a640 1 -- 172.21.15.1:0/624570912 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c4c151c20 con 0x7f5c4c152650 2024-03-06T07:29:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.214+0000 7f5c437fe640 1 -- 172.21.15.1:0/624570912 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c3401a070 con 0x7f5c4c152650 2024-03-06T07:29:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.214+0000 7f5c437fe640 1 --2- 172.21.15.1:0/624570912 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c2c06b4d0 0x7f5c2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.214+0000 7f5c437fe640 1 -- 172.21.15.1:0/624570912 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5c34089640 con 0x7f5c4c152650 2024-03-06T07:29:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.215+0000 7f5c4a59c640 1 --2- 172.21.15.1:0/624570912 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c2c06b4d0 0x7f5c2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.215+0000 7f5c4a59c640 1 --2- 172.21.15.1:0/624570912 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c2c06b4d0 0x7f5c2c06d990 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c036000 tx=0x7f5c3c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.215+0000 7f5c53967740 1 Processor -- start 2024-03-06T07:29:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.216+0000 7f5c53967740 1 -- start start 2024-03-06T07:29:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c53967740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dcf50 0x555aae8daa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c53967740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae8dd330 0x555aae8daf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c53967740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae8db4b0 0x555aae8d7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c53967740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aae8416b0 con 0x555aae8dcf50 2024-03-06T07:29:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c53967740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aae7eaa60 con 0x555aae8dd330 2024-03-06T07:29:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c53967740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x555aae83d750 con 0x555aae8db4b0 2024-03-06T07:29:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c4a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dcf50 0x555aae8daa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c4a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dcf50 0x555aae8daa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56282/0 (socket says 172.21.15.1:56282) 2024-03-06T07:29:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c4a59c640 1 -- 172.21.15.1:0/849896904 learned_addr learned my addr 172.21.15.1:0/849896904 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c41d9b640 1 --2- 172.21.15.1:0/849896904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae8dd330 0x555aae8daf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c4ad9d640 1 --2- 172.21.15.1:0/849896904 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae8db4b0 0x555aae8d7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c4a59c640 1 -- 172.21.15.1:0/849896904 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae8db4b0 msgr2=0x555aae8d7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c4a59c640 1 --2- 172.21.15.1:0/849896904 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae8db4b0 0x555aae8d7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c4a59c640 1 -- 172.21.15.1:0/849896904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae8dd330 msgr2=0x555aae8daf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c4a59c640 1 --2- 172.21.15.1:0/849896904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae8dd330 0x555aae8daf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c4a59c640 1 -- 172.21.15.1:0/849896904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555aae808f30 con 0x555aae8dcf50 2024-03-06T07:29:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c4a59c640 1 --2- 172.21.15.1:0/849896904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dcf50 0x555aae8daa30 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c02b880 tx=0x7f5c3c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd0f9678e0bc415 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c2bfff640 1 -- 172.21.15.1:0/849896904 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c031720 con 0x555aae8dcf50 2024-03-06T07:29:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c2bfff640 1 -- 172.21.15.1:0/849896904 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3c037050 con 0x555aae8dcf50 2024-03-06T07:29:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.217+0000 7f5c2bfff640 1 -- 172.21.15.1:0/849896904 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c031ac0 con 0x555aae8dcf50 2024-03-06T07:29:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.218+0000 7f5c53967740 1 -- 172.21.15.1:0/849896904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dcf50 msgr2=0x555aae8daa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.218+0000 7f5c53967740 1 --2- 172.21.15.1:0/849896904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dcf50 0x555aae8daa30 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c02b880 tx=0x7f5c3c030000 comp rx=0 tx=0).stop 2024-03-06T07:29:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.218+0000 7f5c53967740 1 -- 172.21.15.1:0/849896904 shutdown_connections 2024-03-06T07:29:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.218+0000 7f5c53967740 1 --2- 172.21.15.1:0/849896904 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae8db4b0 0x555aae8d7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.218+0000 7f5c53967740 1 --2- 172.21.15.1:0/849896904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae8dd330 0x555aae8daf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.218+0000 7f5c53967740 1 --2- 172.21.15.1:0/849896904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dcf50 0x555aae8daa30 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.218+0000 7f5c53967740 1 -- 172.21.15.1:0/849896904 >> 172.21.15.1:0/849896904 conn(0x555aae8d97d0 msgr2=0x555aae8fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.218+0000 7f5c53967740 1 -- 172.21.15.1:0/849896904 shutdown_connections 2024-03-06T07:29:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.218+0000 7f5c53967740 1 -- 172.21.15.1:0/849896904 wait complete. 2024-03-06T07:29:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.219+0000 7f5c53967740 1 -- start start 2024-03-06T07:29:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.219+0000 7f5c53967740 1 Processor -- start 2024-03-06T07:29:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.220+0000 7f5c53967740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dcf50 0x555aae903a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.220+0000 7f5c53967740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae8dd330 0x555aae8ff700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.220+0000 7f5c53967740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae902a30 0x555aae915870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.220+0000 7f5c53967740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aae8ff230 con 0x555aae8dcf50 2024-03-06T07:29:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.220+0000 7f5c53967740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aae7fdea0 con 0x555aae8dd330 2024-03-06T07:29:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.220+0000 7f5c53967740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x555aae8fec70 con 0x555aae902a30 2024-03-06T07:29:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.220+0000 7f5c53967740 10 client.0 _ll_register_callbacks cb 0x555aae7fd870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.220+0000 7f5c4a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dcf50 0x555aae903a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.220+0000 7f5c4a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dcf50 0x555aae903a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56292/0 (socket says 172.21.15.1:56292) 2024-03-06T07:29:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.220+0000 7f5c4a59c640 1 -- 172.21.15.1:0/1482287636 learned_addr learned my addr 172.21.15.1:0/1482287636 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.220+0000 7f5c4ad9d640 1 --2- 172.21.15.1:0/1482287636 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae902a30 0x555aae915870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.220+0000 7f5c41d9b640 1 --2- 172.21.15.1:0/1482287636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae8dd330 0x555aae8ff700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.220+0000 7f5c4a59c640 1 -- 172.21.15.1:0/1482287636 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae902a30 msgr2=0x555aae915870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.220+0000 7f5c4a59c640 1 --2- 172.21.15.1:0/1482287636 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae902a30 0x555aae915870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.220+0000 7f5c4a59c640 1 -- 172.21.15.1:0/1482287636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae8dd330 msgr2=0x555aae8ff700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.220+0000 7f5c4a59c640 1 --2- 172.21.15.1:0/1482287636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae8dd330 0x555aae8ff700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.221+0000 7f5c4a59c640 1 -- 172.21.15.1:0/1482287636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555aae7fe270 con 0x555aae8dcf50 2024-03-06T07:29:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.221+0000 7f5c4a59c640 1 --2- 172.21.15.1:0/1482287636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dcf50 0x555aae903a80 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c0098c0 tx=0x7f5c3c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c9469ec1559430 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.221+0000 7f5c297fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:29:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.221+0000 7f5c297fa640 1 -- 172.21.15.1:0/1482287636 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c031c30 con 0x555aae8dcf50 2024-03-06T07:29:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.221+0000 7f5c297fa640 1 -- 172.21.15.1:0/1482287636 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5c3c031dd0 con 0x555aae8dcf50 2024-03-06T07:29:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.221+0000 7f5c53967740 1 -- 172.21.15.1:0/1482287636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c3c037050 con 0x555aae8dcf50 2024-03-06T07:29:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.221+0000 7f5c297fa640 1 -- 172.21.15.1:0/1482287636 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=199) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c3c063050 con 0x555aae8dcf50 2024-03-06T07:29:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.221+0000 7f5c53967740 10 client.11007 fetch_fsmap learned FSMap version 199 2024-03-06T07:29:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.221+0000 7f5c53967740 1 -- 172.21.15.1:0/1482287636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=199,monmap=2+}) v3 -- 0x555aae5562d0 con 0x555aae8dcf50 2024-03-06T07:29:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.221+0000 7f5c297fa640 1 -- 172.21.15.1:0/1482287636 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3c037050 con 0x555aae8dcf50 2024-03-06T07:29:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.221+0000 7f5c297fa640 1 -- 172.21.15.1:0/1482287636 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 199) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f5c3c066020 con 0x555aae8dcf50 2024-03-06T07:29:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.221+0000 7f5c53967740 10 client.11007 fetch_fsmap finished waiting for FSMap version 199 2024-03-06T07:29:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.221+0000 7f5c53967740 10 client.11007 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:29:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.221+0000 7f5c53967740 20 client.11007 populate_metadata read hostname 'smithi001' 2024-03-06T07:29:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.221+0000 7f5c53967740 1 --2- 172.21.15.1:0/1482287636 >> [v2:172.21.15.1:6832/2424540959,v1:172.21.15.1:6834/2424540959] conn(0x555aae92d4e0 0x555aae92d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.221+0000 7f5c53967740 4 client.11007 mds_command: new command op to 10803 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:29:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.221+0000 7f5c53967740 1 -- 172.21.15.1:0/1482287636 --> [v2:172.21.15.1:6832/2424540959,v1:172.21.15.1:6834/2424540959] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555aae84ebf0 con 0x555aae92d4e0 2024-03-06T07:29:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.222+0000 7f5c41d9b640 1 --2- 172.21.15.1:0/1482287636 >> [v2:172.21.15.1:6832/2424540959,v1:172.21.15.1:6834/2424540959] conn(0x555aae92d4e0 0x555aae92d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.222+0000 7f5c297fa640 10 client.11007 ms_handle_connect on v2:172.21.15.1:6832/2424540959 2024-03-06T07:29:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.222+0000 7f5c41d9b640 1 --2- 172.21.15.1:0/1482287636 >> [v2:172.21.15.1:6832/2424540959,v1:172.21.15.1:6834/2424540959] conn(0x555aae92d4e0 0x555aae92d8c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ce93a887c5efb3b3 server_cookie=6def4b2cbaa5395f in_seq=0 out_seq=0 2024-03-06T07:29:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.222+0000 7f5c297fa640 1 -- 172.21.15.1:0/1482287636 <== mds.1 v2:172.21.15.1:6832/2424540959 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555aae84ebf0 con 0x555aae92d4e0 2024-03-06T07:29:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.222+0000 7f5c297fa640 10 client.11007 handle_command_reply: tid=0 2024-03-06T07:29:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.222+0000 7f5c53967740 1 client.11007 shutdown 2024-03-06T07:29:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.223+0000 7f5c53967740 1 -- 172.21.15.1:0/1482287636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dcf50 msgr2=0x555aae903a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.223+0000 7f5c53967740 1 --2- 172.21.15.1:0/1482287636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dcf50 0x555aae903a80 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c0098c0 tx=0x7f5c3c004730 comp rx=0 tx=0).stop 2024-03-06T07:29:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.223+0000 7f5c53967740 1 -- 172.21.15.1:0/1482287636 shutdown_connections 2024-03-06T07:29:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.223+0000 7f5c53967740 1 --2- 172.21.15.1:0/1482287636 >> [v2:172.21.15.1:6832/2424540959,v1:172.21.15.1:6834/2424540959] conn(0x555aae92d4e0 0x555aae92d8c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.223+0000 7f5c53967740 1 --2- 172.21.15.1:0/1482287636 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae902a30 0x555aae915870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.223+0000 7f5c53967740 1 --2- 172.21.15.1:0/1482287636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae8dd330 0x555aae8ff700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.223+0000 7f5c53967740 1 --2- 172.21.15.1:0/1482287636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dcf50 0x555aae903a80 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.223+0000 7f5c297fa640 0 client.11007 ms_handle_reset on v2:172.21.15.1:6832/2424540959 2024-03-06T07:29:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.223+0000 7f5c53967740 1 -- 172.21.15.1:0/1482287636 >> 172.21.15.1:0/1482287636 conn(0x555aae8db4b0 msgr2=0x555aae919b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.223+0000 7f5c53967740 1 -- 172.21.15.1:0/1482287636 shutdown_connections 2024-03-06T07:29:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.223+0000 7f5c53967740 1 -- 172.21.15.1:0/1482287636 wait complete. 2024-03-06T07:29:43.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.223+0000 7f5c53967740 20 client.11007 trim_cache size 0 max 16384 2024-03-06T07:29:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.243+0000 7f5c53967740 1 Processor -- start 2024-03-06T07:29:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.243+0000 7f5c53967740 1 -- start start 2024-03-06T07:29:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c53967740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae8dcf50 0x555aae8d89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c53967740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae8dd330 0x555aae8dbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c53967740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae902a30 0x555aae8dc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c53967740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c1f79d0 con 0x555aae902a30 2024-03-06T07:29:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c53967740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c14d580 con 0x555aae8dcf50 2024-03-06T07:29:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c53967740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c14db50 con 0x555aae8dd330 2024-03-06T07:29:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c4ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae902a30 0x555aae8dc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c4ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae902a30 0x555aae8dc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56302/0 (socket says 172.21.15.1:56302) 2024-03-06T07:29:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c4ad9d640 1 -- 172.21.15.1:0/64185746 learned_addr learned my addr 172.21.15.1:0/64185746 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c4ad9d640 1 -- 172.21.15.1:0/64185746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae8dd330 msgr2=0x555aae8dbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c4a59c640 1 --2- 172.21.15.1:0/64185746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae8dcf50 0x555aae8d89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c41d9b640 1 --2- 172.21.15.1:0/64185746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae8dd330 0x555aae8dbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c4ad9d640 1 --2- 172.21.15.1:0/64185746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae8dd330 0x555aae8dbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c4ad9d640 1 -- 172.21.15.1:0/64185746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae8dcf50 msgr2=0x555aae8d89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c4ad9d640 1 --2- 172.21.15.1:0/64185746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae8dcf50 0x555aae8d89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c4ad9d640 1 -- 172.21.15.1:0/64185746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555aae6206b0 con 0x555aae902a30 2024-03-06T07:29:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c41d9b640 1 --2- 172.21.15.1:0/64185746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae8dd330 0x555aae8dbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.244+0000 7f5c4a59c640 1 --2- 172.21.15.1:0/64185746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae8dcf50 0x555aae8d89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.245+0000 7f5c4ad9d640 1 --2- 172.21.15.1:0/64185746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae902a30 0x555aae8dc530 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f5c44003a20 tx=0x7f5c44003ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb4dd71b99a6c96 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.245+0000 7f5c2b7fe640 1 -- 172.21.15.1:0/64185746 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4401b070 con 0x555aae902a30 2024-03-06T07:29:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.245+0000 7f5c2b7fe640 1 -- 172.21.15.1:0/64185746 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c44017440 con 0x555aae902a30 2024-03-06T07:29:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.245+0000 7f5c2b7fe640 1 -- 172.21.15.1:0/64185746 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c44017830 con 0x555aae902a30 2024-03-06T07:29:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.245+0000 7f5c53967740 1 -- 172.21.15.1:0/64185746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae902a30 msgr2=0x555aae8dc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.245+0000 7f5c53967740 1 --2- 172.21.15.1:0/64185746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae902a30 0x555aae8dc530 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f5c44003a20 tx=0x7f5c44003ee0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.245+0000 7f5c53967740 1 -- 172.21.15.1:0/64185746 shutdown_connections 2024-03-06T07:29:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.245+0000 7f5c53967740 1 --2- 172.21.15.1:0/64185746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae902a30 0x555aae8dc530 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.245+0000 7f5c53967740 1 --2- 172.21.15.1:0/64185746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae8dd330 0x555aae8dbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.245+0000 7f5c53967740 1 --2- 172.21.15.1:0/64185746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae8dcf50 0x555aae8d89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.245+0000 7f5c53967740 1 -- 172.21.15.1:0/64185746 >> 172.21.15.1:0/64185746 conn(0x555aae8db4b0 msgr2=0x555aae903890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.245+0000 7f5c53967740 1 -- 172.21.15.1:0/64185746 shutdown_connections 2024-03-06T07:29:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.245+0000 7f5c53967740 1 -- 172.21.15.1:0/64185746 wait complete. 2024-03-06T07:29:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.246+0000 7f5c53967740 1 -- start start 2024-03-06T07:29:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.246+0000 7f5c53967740 1 Processor -- start 2024-03-06T07:29:43.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c53967740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae8dcf50 0x555aae8dc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c53967740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dd330 0x555aae903a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c53967740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae902a30 0x555aae98bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c53967740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aae92a400 con 0x555aae8dd330 2024-03-06T07:29:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c53967740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aae8ff230 con 0x555aae902a30 2024-03-06T07:29:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c53967740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x555aae915e00 con 0x555aae8dcf50 2024-03-06T07:29:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c53967740 10 client.0 _ll_register_callbacks cb 0x7f5c3c004470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c41d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dd330 0x555aae903a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c41d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dd330 0x555aae903a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56310/0 (socket says 172.21.15.1:56310) 2024-03-06T07:29:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c41d9b640 1 -- 172.21.15.1:0/995980108 learned_addr learned my addr 172.21.15.1:0/995980108 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c4a59c640 1 --2- 172.21.15.1:0/995980108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae8dcf50 0x555aae8dc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c4ad9d640 1 --2- 172.21.15.1:0/995980108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae902a30 0x555aae98bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c41d9b640 1 -- 172.21.15.1:0/995980108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae8dcf50 msgr2=0x555aae8dc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c41d9b640 1 --2- 172.21.15.1:0/995980108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae8dcf50 0x555aae8dc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c41d9b640 1 -- 172.21.15.1:0/995980108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae902a30 msgr2=0x555aae98bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c41d9b640 1 --2- 172.21.15.1:0/995980108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae902a30 0x555aae98bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c41d9b640 1 -- 172.21.15.1:0/995980108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555aae903fa0 con 0x555aae8dd330 2024-03-06T07:29:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c41d9b640 1 --2- 172.21.15.1:0/995980108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dd330 0x555aae903a60 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f5c34057ce0 tx=0x7f5c3405e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57cc0a6eca5331be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c297fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:29:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c297fa640 1 -- 172.21.15.1:0/995980108 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c34087060 con 0x555aae8dd330 2024-03-06T07:29:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c297fa640 1 -- 172.21.15.1:0/995980108 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5c340543c0 con 0x555aae8dd330 2024-03-06T07:29:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.248+0000 7f5c53967740 1 -- 172.21.15.1:0/995980108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c44017440 con 0x555aae8dd330 2024-03-06T07:29:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.249+0000 7f5c297fa640 1 -- 172.21.15.1:0/995980108 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=199) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c3405ff20 con 0x555aae8dd330 2024-03-06T07:29:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.249+0000 7f5c53967740 10 client.11013 fetch_fsmap learned FSMap version 199 2024-03-06T07:29:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.249+0000 7f5c53967740 1 -- 172.21.15.1:0/995980108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=199,monmap=2+}) v3 -- 0x555aae8fac80 con 0x555aae8dd330 2024-03-06T07:29:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.249+0000 7f5c297fa640 1 -- 172.21.15.1:0/995980108 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c44017440 con 0x555aae8dd330 2024-03-06T07:29:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.249+0000 7f5c297fa640 1 -- 172.21.15.1:0/995980108 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 199) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f5c34007450 con 0x555aae8dd330 2024-03-06T07:29:43.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.249+0000 7f5c53967740 10 client.11013 fetch_fsmap finished waiting for FSMap version 199 2024-03-06T07:29:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.249+0000 7f5c53967740 10 client.11013 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:29:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.249+0000 7f5c53967740 20 client.11013 populate_metadata read hostname 'smithi001' 2024-03-06T07:29:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.249+0000 7f5c53967740 1 --2- 172.21.15.1:0/995980108 >> [v2:172.21.15.1:6832/2424540959,v1:172.21.15.1:6834/2424540959] conn(0x555aae98c880 0x555aae985540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.249+0000 7f5c53967740 4 client.11013 mds_command: new command op to 10803 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:29:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.249+0000 7f5c53967740 1 -- 172.21.15.1:0/995980108 --> [v2:172.21.15.1:6832/2424540959,v1:172.21.15.1:6834/2424540959] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555aae848f40 con 0x555aae98c880 2024-03-06T07:29:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.249+0000 7f5c4a59c640 1 --2- 172.21.15.1:0/995980108 >> [v2:172.21.15.1:6832/2424540959,v1:172.21.15.1:6834/2424540959] conn(0x555aae98c880 0x555aae985540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.249+0000 7f5c297fa640 10 client.11013 ms_handle_connect on v2:172.21.15.1:6832/2424540959 2024-03-06T07:29:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.249+0000 7f5c4a59c640 1 --2- 172.21.15.1:0/995980108 >> [v2:172.21.15.1:6832/2424540959,v1:172.21.15.1:6834/2424540959] conn(0x555aae98c880 0x555aae985540 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=22a11ebcf54dfece server_cookie=680487c8e2ccce0b in_seq=0 out_seq=0 2024-03-06T07:29:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.250+0000 7f5c297fa640 1 -- 172.21.15.1:0/995980108 <== mds.1 v2:172.21.15.1:6832/2424540959 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x555aae848f40 con 0x555aae98c880 2024-03-06T07:29:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.250+0000 7f5c297fa640 10 client.11013 handle_command_reply: tid=0 2024-03-06T07:29:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.250+0000 7f5c53967740 1 client.11013 shutdown 2024-03-06T07:29:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.250+0000 7f5c53967740 1 -- 172.21.15.1:0/995980108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dd330 msgr2=0x555aae903a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.250+0000 7f5c53967740 1 --2- 172.21.15.1:0/995980108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dd330 0x555aae903a60 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f5c34057ce0 tx=0x7f5c3405e040 comp rx=0 tx=0).stop 2024-03-06T07:29:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.250+0000 7f5c53967740 1 -- 172.21.15.1:0/995980108 shutdown_connections 2024-03-06T07:29:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.250+0000 7f5c53967740 1 --2- 172.21.15.1:0/995980108 >> [v2:172.21.15.1:6832/2424540959,v1:172.21.15.1:6834/2424540959] conn(0x555aae98c880 0x555aae985540 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.250+0000 7f5c53967740 1 --2- 172.21.15.1:0/995980108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555aae902a30 0x555aae98bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.250+0000 7f5c53967740 1 --2- 172.21.15.1:0/995980108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555aae8dd330 0x555aae903a60 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.250+0000 7f5c53967740 1 --2- 172.21.15.1:0/995980108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x555aae8dcf50 0x555aae8dc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.250+0000 7f5c297fa640 0 client.11013 ms_handle_reset on v2:172.21.15.1:6832/2424540959 2024-03-06T07:29:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.250+0000 7f5c53967740 1 -- 172.21.15.1:0/995980108 >> 172.21.15.1:0/995980108 conn(0x555aae8db4b0 msgr2=0x555aae915380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.250+0000 7f5c53967740 1 -- 172.21.15.1:0/995980108 shutdown_connections 2024-03-06T07:29:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.250+0000 7f5c53967740 1 -- 172.21.15.1:0/995980108 wait complete. 2024-03-06T07:29:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.250+0000 7f5c53967740 20 client.11013 trim_cache size 0 max 16384 2024-03-06T07:29:43.258 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:29:43.258 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:29:43.258 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 1, 2024-03-06T07:29:43.258 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 10803, 2024-03-06T07:29:43.259 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:29:43.259 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:29:43.259 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:29:43.259 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 2.7089758879999999, 2024-03-06T07:29:43.259 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 199, 2024-03-06T07:29:43.259 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 210, 2024-03-06T07:29:43.259 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 210, 2024-03-06T07:29:43.259 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 14.843723688000001 2024-03-06T07:29:43.259 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:29:43.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.251+0000 7f5c50d7a640 1 -- 172.21.15.1:0/624570912 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c2c06b4d0 msgr2=0x7f5c2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.251+0000 7f5c50d7a640 1 --2- 172.21.15.1:0/624570912 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c2c06b4d0 0x7f5c2c06d990 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c036000 tx=0x7f5c3c008030 comp rx=0 tx=0).stop 2024-03-06T07:29:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.251+0000 7f5c50d7a640 1 -- 172.21.15.1:0/624570912 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c4c152650 msgr2=0x7f5c4c1e0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.251+0000 7f5c50d7a640 1 --2- 172.21.15.1:0/624570912 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c4c152650 0x7f5c4c1e0c90 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f5c3400c4d0 tx=0x7f5c3400c990 comp rx=0 tx=0).stop 2024-03-06T07:29:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.251+0000 7f5c50d7a640 1 -- 172.21.15.1:0/624570912 shutdown_connections 2024-03-06T07:29:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.251+0000 7f5c50d7a640 1 --2- 172.21.15.1:0/624570912 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5c2c06b4d0 0x7f5c2c06d990 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.251+0000 7f5c50d7a640 1 --2- 172.21.15.1:0/624570912 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c4c1eeac0 0x7f5c4c1e11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.251+0000 7f5c50d7a640 1 --2- 172.21.15.1:0/624570912 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c4c152650 0x7f5c4c1e0c90 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.251+0000 7f5c50d7a640 1 --2- 172.21.15.1:0/624570912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c4c0bf9d0 0x7f5c4c1ee580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.251+0000 7f5c50d7a640 1 -- 172.21.15.1:0/624570912 >> 172.21.15.1:0/624570912 conn(0x7f5c4c066ee0 msgr2=0x7f5c4c0cae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.251+0000 7f5c50d7a640 1 -- 172.21.15.1:0/624570912 shutdown_connections 2024-03-06T07:29:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.251+0000 7f5c50d7a640 1 -- 172.21.15.1:0/624570912 wait complete. 2024-03-06T07:29:43.265 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:29:43.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.368+0000 7f5e282eb640 1 Processor -- start 2024-03-06T07:29:43.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.368+0000 7f5e282eb640 1 -- start start 2024-03-06T07:29:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.370+0000 7f5e282eb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5e200c6b10 0x7f5e20155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.370+0000 7f5e282eb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e200c2fc0 con 0x7f5e20152650 2024-03-06T07:29:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.370+0000 7f5e282eb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e20158da0 con 0x7f5e200bf9d0 2024-03-06T07:29:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.370+0000 7f5e282eb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e20155800 con 0x7f5e200c6b10 2024-03-06T07:29:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.370+0000 7f5e25886640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5e200c6b10 0x7f5e20155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.370+0000 7f5e26087640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5e20152650 0x7f5e20158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35940/0 (socket says 172.21.15.1:35940) 2024-03-06T07:29:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.370+0000 7f5e26087640 1 -- 172.21.15.1:0/3076933346 learned_addr learned my addr 172.21.15.1:0/3076933346 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.370+0000 7f5e25886640 1 -- 172.21.15.1:0/3076933346 >> v1:172.21.15.98:6789/0 conn(0x7f5e200bf9d0 legacy=0x7f5e200c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.370+0000 7f5e25886640 1 -- 172.21.15.1:0/3076933346 >> v1:172.21.15.1:6789/0 conn(0x7f5e20152650 legacy=0x7f5e20158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.370+0000 7f5e25886640 1 -- 172.21.15.1:0/3076933346 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e200c6690 con 0x7f5e200c6b10 2024-03-06T07:29:43.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.371+0000 7f5e25886640 1 --2- 172.21.15.1:0/3076933346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e200c6b10 0x7f5e20155c90 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f5e100097a0 tx=0x7f5e1002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ac3f3cb026eb08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.371+0000 7f5e25085640 1 -- 172.21.15.1:0/3076933346 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e10035020 con 0x7f5e200c6b10 2024-03-06T07:29:43.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.371+0000 7f5e25085640 1 -- 172.21.15.1:0/3076933346 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e10031830 con 0x7f5e200c6b10 2024-03-06T07:29:43.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.371+0000 7f5e25085640 1 -- 172.21.15.1:0/3076933346 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e10031c00 con 0x7f5e200c6b10 2024-03-06T07:29:43.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.372+0000 7f5e282eb640 1 -- 172.21.15.1:0/3076933346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e200c6b10 msgr2=0x7f5e20155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.372+0000 7f5e282eb640 1 --2- 172.21.15.1:0/3076933346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e200c6b10 0x7f5e20155c90 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f5e100097a0 tx=0x7f5e1002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.372+0000 7f5e282eb640 1 -- 172.21.15.1:0/3076933346 shutdown_connections 2024-03-06T07:29:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.372+0000 7f5e282eb640 1 --2- 172.21.15.1:0/3076933346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e200c6b10 0x7f5e20155c90 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.372+0000 7f5e282eb640 1 -- 172.21.15.1:0/3076933346 >> 172.21.15.1:0/3076933346 conn(0x7f5e20066ee0 msgr2=0x7f5e200672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.372+0000 7f5e282eb640 1 -- 172.21.15.1:0/3076933346 shutdown_connections 2024-03-06T07:29:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.372+0000 7f5e282eb640 1 -- 172.21.15.1:0/3076933346 wait complete. 2024-03-06T07:29:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.373+0000 7f5e282eb640 1 Processor -- start 2024-03-06T07:29:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.373+0000 7f5e282eb640 1 -- start start 2024-03-06T07:29:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e282eb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e200bf9d0 0x7f5e2014d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e282eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e20152650 0x7f5e2013d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e282eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e2014ba40 0x7f5e201eb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e282eb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e201551b0 con 0x7f5e20152650 2024-03-06T07:29:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e282eb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e200c6fa0 con 0x7f5e2014ba40 2024-03-06T07:29:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e282eb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e20155490 con 0x7f5e200bf9d0 2024-03-06T07:29:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e25886640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e20152650 0x7f5e2013d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e25886640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e20152650 0x7f5e2013d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56332/0 (socket says 172.21.15.1:56332) 2024-03-06T07:29:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e25886640 1 -- 172.21.15.1:0/3695644190 learned_addr learned my addr 172.21.15.1:0/3695644190 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e26087640 1 --2- 172.21.15.1:0/3695644190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e200bf9d0 0x7f5e2014d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e26888640 1 --2- 172.21.15.1:0/3695644190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e2014ba40 0x7f5e201eb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e25886640 1 -- 172.21.15.1:0/3695644190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e200bf9d0 msgr2=0x7f5e2014d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e25886640 1 --2- 172.21.15.1:0/3695644190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e200bf9d0 0x7f5e2014d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e25886640 1 -- 172.21.15.1:0/3695644190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e2014ba40 msgr2=0x7f5e201eb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e25886640 1 --2- 172.21.15.1:0/3695644190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e2014ba40 0x7f5e201eb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e25886640 1 -- 172.21.15.1:0/3695644190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e201eb0f0 con 0x7f5e20152650 2024-03-06T07:29:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e26888640 1 --2- 172.21.15.1:0/3695644190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e2014ba40 0x7f5e201eb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e25886640 1 --2- 172.21.15.1:0/3695644190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e20152650 0x7f5e2013d200 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f5e10009890 tx=0x7f5e100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e16ffd640 1 -- 172.21.15.1:0/3695644190 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e10041070 con 0x7f5e20152650 2024-03-06T07:29:43.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e16ffd640 1 -- 172.21.15.1:0/3695644190 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1003aa30 con 0x7f5e20152650 2024-03-06T07:29:43.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e282eb640 1 -- 172.21.15.1:0/3695644190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e20151d10 con 0x7f5e20152650 2024-03-06T07:29:43.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.374+0000 7f5e16ffd640 1 -- 172.21.15.1:0/3695644190 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e10035040 con 0x7f5e20152650 2024-03-06T07:29:43.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.375+0000 7f5e282eb640 1 -- 172.21.15.1:0/3695644190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e20157e10 con 0x7f5e20152650 2024-03-06T07:29:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.376+0000 7f5e16ffd640 1 -- 172.21.15.1:0/3695644190 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e1004c060 con 0x7f5e20152650 2024-03-06T07:29:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.377+0000 7f5e16ffd640 1 --2- 172.21.15.1:0/3695644190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e0406b4d0 0x7f5e0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.377+0000 7f5e16ffd640 1 -- 172.21.15.1:0/3695644190 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5e100abce0 con 0x7f5e20152650 2024-03-06T07:29:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.377+0000 7f5e26087640 1 --2- 172.21.15.1:0/3695644190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e0406b4d0 0x7f5e0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.377+0000 7f5e2afae740 1 Processor -- start 2024-03-06T07:29:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.377+0000 7f5e26087640 1 --2- 172.21.15.1:0/3695644190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e0406b4d0 0x7f5e0406d990 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c005880 tx=0x7f5e1c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.378+0000 7f5e2afae740 1 -- start start 2024-03-06T07:29:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.378+0000 7f5e2afae740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e3aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.378+0000 7f5e2afae740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e3d330 0x5635f4e3af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.378+0000 7f5e2afae740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e3b4b0 0x5635f4e37680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.378+0000 7f5e2afae740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635f4da16b0 con 0x5635f4e3cf50 2024-03-06T07:29:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.378+0000 7f5e2afae740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635f4d4aa60 con 0x5635f4e3d330 2024-03-06T07:29:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.378+0000 7f5e2afae740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635f4d9d750 con 0x5635f4e3b4b0 2024-03-06T07:29:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.378+0000 7f5e26087640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e3aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.378+0000 7f5e26087640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e3aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56342/0 (socket says 172.21.15.1:56342) 2024-03-06T07:29:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.378+0000 7f5e26087640 1 -- 172.21.15.1:0/1447239698 learned_addr learned my addr 172.21.15.1:0/1447239698 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.379+0000 7f5e25886640 1 --2- 172.21.15.1:0/1447239698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e3d330 0x5635f4e3af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.379+0000 7f5e26888640 1 --2- 172.21.15.1:0/1447239698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e3b4b0 0x5635f4e37680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.379+0000 7f5e26087640 1 -- 172.21.15.1:0/1447239698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e3b4b0 msgr2=0x5635f4e37680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.379+0000 7f5e26087640 1 --2- 172.21.15.1:0/1447239698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e3b4b0 0x5635f4e37680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.379+0000 7f5e26087640 1 -- 172.21.15.1:0/1447239698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e3d330 msgr2=0x5635f4e3af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.379+0000 7f5e26087640 1 --2- 172.21.15.1:0/1447239698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e3d330 0x5635f4e3af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.379+0000 7f5e26087640 1 -- 172.21.15.1:0/1447239698 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635f4d68f30 con 0x5635f4e3cf50 2024-03-06T07:29:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.379+0000 7f5e26087640 1 --2- 172.21.15.1:0/1447239698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e3aa30 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00fab0 tx=0x7f5e1c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56a779b0bfb2b03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.379+0000 7f5e03fff640 1 -- 172.21.15.1:0/1447239698 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c025070 con 0x5635f4e3cf50 2024-03-06T07:29:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.379+0000 7f5e03fff640 1 -- 172.21.15.1:0/1447239698 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1c01b050 con 0x5635f4e3cf50 2024-03-06T07:29:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.379+0000 7f5e03fff640 1 -- 172.21.15.1:0/1447239698 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c014de0 con 0x5635f4e3cf50 2024-03-06T07:29:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.380+0000 7f5e2afae740 1 -- 172.21.15.1:0/1447239698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 msgr2=0x5635f4e3aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.380+0000 7f5e2afae740 1 --2- 172.21.15.1:0/1447239698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e3aa30 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00fab0 tx=0x7f5e1c012000 comp rx=0 tx=0).stop 2024-03-06T07:29:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.380+0000 7f5e2afae740 1 -- 172.21.15.1:0/1447239698 shutdown_connections 2024-03-06T07:29:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.380+0000 7f5e2afae740 1 --2- 172.21.15.1:0/1447239698 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e3b4b0 0x5635f4e37680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.380+0000 7f5e2afae740 1 --2- 172.21.15.1:0/1447239698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e3d330 0x5635f4e3af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.380+0000 7f5e2afae740 1 --2- 172.21.15.1:0/1447239698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e3aa30 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.380+0000 7f5e2afae740 1 -- 172.21.15.1:0/1447239698 >> 172.21.15.1:0/1447239698 conn(0x5635f4e397d0 msgr2=0x5635f4e5e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.380+0000 7f5e2afae740 1 -- 172.21.15.1:0/1447239698 shutdown_connections 2024-03-06T07:29:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.380+0000 7f5e2afae740 1 -- 172.21.15.1:0/1447239698 wait complete. 2024-03-06T07:29:43.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.381+0000 7f5e2afae740 1 -- start start 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.381+0000 7f5e2afae740 1 Processor -- start 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.382+0000 7f5e2afae740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e63b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.382+0000 7f5e2afae740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e3d330 0x5635f4e61d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.382+0000 7f5e2afae740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e62250 0x5635f4e75bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.382+0000 7f5e2afae740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635f4e5f230 con 0x5635f4e3cf50 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.382+0000 7f5e2afae740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635f4d5dea0 con 0x5635f4e62250 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.382+0000 7f5e2afae740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635f4e5ec70 con 0x5635f4e3d330 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.382+0000 7f5e2afae740 10 client.0 _ll_register_callbacks cb 0x5635f4d5d870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.382+0000 7f5e26087640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e63b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.382+0000 7f5e26087640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e63b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56352/0 (socket says 172.21.15.1:56352) 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.382+0000 7f5e26087640 1 -- 172.21.15.1:0/1599186764 learned_addr learned my addr 172.21.15.1:0/1599186764 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.382+0000 7f5e25886640 1 --2- 172.21.15.1:0/1599186764 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e3d330 0x5635f4e61d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.382+0000 7f5e26888640 1 --2- 172.21.15.1:0/1599186764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e62250 0x5635f4e75bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.383+0000 7f5e26087640 1 -- 172.21.15.1:0/1599186764 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e3d330 msgr2=0x5635f4e61d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.383+0000 7f5e26087640 1 --2- 172.21.15.1:0/1599186764 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e3d330 0x5635f4e61d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.383+0000 7f5e26087640 1 -- 172.21.15.1:0/1599186764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e62250 msgr2=0x5635f4e75bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.383+0000 7f5e26087640 1 --2- 172.21.15.1:0/1599186764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e62250 0x5635f4e75bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.383+0000 7f5e26087640 1 -- 172.21.15.1:0/1599186764 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635f4d5e270 con 0x5635f4e3cf50 2024-03-06T07:29:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.383+0000 7f5e26087640 1 --2- 172.21.15.1:0/1599186764 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e63b90 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c001eb0 tx=0x7f5e1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75dc70e1df7e0878 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.383+0000 7f5e017fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:29:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.383+0000 7f5e017fa640 1 -- 172.21.15.1:0/1599186764 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c025070 con 0x5635f4e3cf50 2024-03-06T07:29:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.383+0000 7f5e017fa640 1 -- 172.21.15.1:0/1599186764 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5e1c020c30 con 0x5635f4e3cf50 2024-03-06T07:29:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.383+0000 7f5e2afae740 1 -- 172.21.15.1:0/1599186764 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e1c01b050 con 0x5635f4e3cf50 2024-03-06T07:29:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.383+0000 7f5e017fa640 1 -- 172.21.15.1:0/1599186764 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=199) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e1c038050 con 0x5635f4e3cf50 2024-03-06T07:29:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.383+0000 7f5e2afae740 10 client.11028 fetch_fsmap learned FSMap version 199 2024-03-06T07:29:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.383+0000 7f5e2afae740 1 -- 172.21.15.1:0/1599186764 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=199,monmap=2+}) v3 -- 0x5635f4ab62d0 con 0x5635f4e3cf50 2024-03-06T07:29:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.384+0000 7f5e017fa640 1 -- 172.21.15.1:0/1599186764 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1c030070 con 0x5635f4e3cf50 2024-03-06T07:29:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.384+0000 7f5e017fa640 1 -- 172.21.15.1:0/1599186764 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 199) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f5e1c03b020 con 0x5635f4e3cf50 2024-03-06T07:29:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.384+0000 7f5e2afae740 10 client.11028 fetch_fsmap finished waiting for FSMap version 199 2024-03-06T07:29:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.384+0000 7f5e2afae740 10 client.11028 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:29:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.384+0000 7f5e2afae740 20 client.11028 populate_metadata read hostname 'smithi001' 2024-03-06T07:29:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.384+0000 7f5e2afae740 1 --2- 172.21.15.1:0/1599186764 >> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] conn(0x5635f4e8d6f0 0x5635f4e98590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.384+0000 7f5e2afae740 4 client.11028 mds_command: new command op to 10806 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:29:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.384+0000 7f5e2afae740 1 -- 172.21.15.1:0/1599186764 --> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5e1c01b050 con 0x5635f4e8d6f0 2024-03-06T07:29:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.384+0000 7f5e25886640 1 --2- 172.21.15.1:0/1599186764 >> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] conn(0x5635f4e8d6f0 0x5635f4e98590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.384+0000 7f5e017fa640 10 client.11028 ms_handle_connect on v2:172.21.15.1:6833/1871567565 2024-03-06T07:29:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.384+0000 7f5e25886640 1 --2- 172.21.15.1:0/1599186764 >> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] conn(0x5635f4e8d6f0 0x5635f4e98590 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=399d5a2f2ca31ea1 server_cookie=88036c25115688c1 in_seq=0 out_seq=0 2024-03-06T07:29:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.385+0000 7f5e017fa640 1 -- 172.21.15.1:0/1599186764 <== mds.0 v2:172.21.15.1:6833/1871567565 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f5e1c01b050 con 0x5635f4e8d6f0 2024-03-06T07:29:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.385+0000 7f5e017fa640 10 client.11028 handle_command_reply: tid=0 2024-03-06T07:29:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.385+0000 7f5e2afae740 1 client.11028 shutdown 2024-03-06T07:29:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.385+0000 7f5e2afae740 1 -- 172.21.15.1:0/1599186764 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 msgr2=0x5635f4e63b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.385+0000 7f5e2afae740 1 --2- 172.21.15.1:0/1599186764 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e63b90 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c001eb0 tx=0x7f5e1c0057f0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.385+0000 7f5e2afae740 1 -- 172.21.15.1:0/1599186764 shutdown_connections 2024-03-06T07:29:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.385+0000 7f5e2afae740 1 --2- 172.21.15.1:0/1599186764 >> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] conn(0x5635f4e8d6f0 0x5635f4e98590 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.385+0000 7f5e2afae740 1 --2- 172.21.15.1:0/1599186764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e62250 0x5635f4e75bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.385+0000 7f5e2afae740 1 --2- 172.21.15.1:0/1599186764 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e3d330 0x5635f4e61d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.385+0000 7f5e017fa640 0 client.11028 ms_handle_reset on v2:172.21.15.1:6833/1871567565 2024-03-06T07:29:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.385+0000 7f5e2afae740 1 --2- 172.21.15.1:0/1599186764 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e63b90 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.385+0000 7f5e2afae740 1 -- 172.21.15.1:0/1599186764 >> 172.21.15.1:0/1599186764 conn(0x5635f4e3b4b0 msgr2=0x5635f4e79b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.386+0000 7f5e2afae740 1 -- 172.21.15.1:0/1599186764 shutdown_connections 2024-03-06T07:29:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.386+0000 7f5e2afae740 1 -- 172.21.15.1:0/1599186764 wait complete. 2024-03-06T07:29:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.386+0000 7f5e2afae740 20 client.11028 trim_cache size 0 max 16384 2024-03-06T07:29:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.402+0000 7f5e2afae740 1 Processor -- start 2024-03-06T07:29:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.403+0000 7f5e2afae740 1 -- start start 2024-03-06T07:29:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.403+0000 7f5e2afae740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e389a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.403+0000 7f5e2afae740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e3d330 0x5635f4e86b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.403+0000 7f5e2afae740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e62250 0x5635f4e87070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.403+0000 7f5e2afae740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e201f7740 con 0x5635f4e3cf50 2024-03-06T07:29:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.403+0000 7f5e2afae740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e201f7da0 con 0x5635f4e3d330 2024-03-06T07:29:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.403+0000 7f5e2afae740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e201e1e90 con 0x5635f4e62250 2024-03-06T07:29:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.403+0000 7f5e26087640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.403+0000 7f5e26087640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e389a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56366/0 (socket says 172.21.15.1:56366) 2024-03-06T07:29:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.403+0000 7f5e26087640 1 -- 172.21.15.1:0/3234538469 learned_addr learned my addr 172.21.15.1:0/3234538469 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:43.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.403+0000 7f5e26888640 1 --2- 172.21.15.1:0/3234538469 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e62250 0x5635f4e87070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.403+0000 7f5e25886640 1 --2- 172.21.15.1:0/3234538469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e3d330 0x5635f4e86b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.404+0000 7f5e26087640 1 -- 172.21.15.1:0/3234538469 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e62250 msgr2=0x5635f4e87070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.404+0000 7f5e26087640 1 --2- 172.21.15.1:0/3234538469 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e62250 0x5635f4e87070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.404+0000 7f5e26087640 1 -- 172.21.15.1:0/3234538469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e3d330 msgr2=0x5635f4e86b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.404+0000 7f5e26087640 1 --2- 172.21.15.1:0/3234538469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e3d330 0x5635f4e86b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.404+0000 7f5e26087640 1 -- 172.21.15.1:0/3234538469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635f4b806b0 con 0x5635f4e3cf50 2024-03-06T07:29:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.404+0000 7f5e26087640 1 --2- 172.21.15.1:0/3234538469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e389a0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0206d0 tx=0x7f5e1c029000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40198ca41a5bdbc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.404+0000 7f5e037fe640 1 -- 172.21.15.1:0/3234538469 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c020c30 con 0x5635f4e3cf50 2024-03-06T07:29:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.404+0000 7f5e037fe640 1 -- 172.21.15.1:0/3234538469 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1c041070 con 0x5635f4e3cf50 2024-03-06T07:29:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.404+0000 7f5e037fe640 1 -- 172.21.15.1:0/3234538469 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c02b650 con 0x5635f4e3cf50 2024-03-06T07:29:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.404+0000 7f5e2afae740 1 -- 172.21.15.1:0/3234538469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 msgr2=0x5635f4e389a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.404+0000 7f5e2afae740 1 --2- 172.21.15.1:0/3234538469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e389a0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0206d0 tx=0x7f5e1c029000 comp rx=0 tx=0).stop 2024-03-06T07:29:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.405+0000 7f5e2afae740 1 -- 172.21.15.1:0/3234538469 shutdown_connections 2024-03-06T07:29:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.405+0000 7f5e2afae740 1 --2- 172.21.15.1:0/3234538469 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e62250 0x5635f4e87070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.405+0000 7f5e2afae740 1 --2- 172.21.15.1:0/3234538469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e3d330 0x5635f4e86b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.405+0000 7f5e2afae740 1 --2- 172.21.15.1:0/3234538469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3cf50 0x5635f4e389a0 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.405+0000 7f5e2afae740 1 -- 172.21.15.1:0/3234538469 >> 172.21.15.1:0/3234538469 conn(0x5635f4e3b4b0 msgr2=0x5635f4e89d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.405+0000 7f5e2afae740 1 -- 172.21.15.1:0/3234538469 shutdown_connections 2024-03-06T07:29:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.405+0000 7f5e2afae740 1 -- 172.21.15.1:0/3234538469 wait complete. 2024-03-06T07:29:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.405+0000 7f5e2afae740 1 -- start start 2024-03-06T07:29:43.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.405+0000 7f5e2afae740 1 Processor -- start 2024-03-06T07:29:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.406+0000 7f5e2afae740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e3cf50 0x5635f4e75af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.406+0000 7f5e2afae740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3d330 0x5635f4eec2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.406+0000 7f5e2afae740 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e62250 0x5635f4ed1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.406+0000 7f5e2afae740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635f4e63190 con 0x5635f4e3d330 2024-03-06T07:29:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.406+0000 7f5e2afae740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e201ef3a0 con 0x5635f4e3cf50 2024-03-06T07:29:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.406+0000 7f5e2afae740 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635f4e89980 con 0x5635f4e62250 2024-03-06T07:29:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.406+0000 7f5e2afae740 10 client.0 _ll_register_callbacks cb 0x7f5e1c014f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.406+0000 7f5e25886640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3d330 0x5635f4eec2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.406+0000 7f5e25886640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3d330 0x5635f4eec2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56368/0 (socket says 172.21.15.1:56368) 2024-03-06T07:29:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.406+0000 7f5e25886640 1 -- 172.21.15.1:0/950261492 learned_addr learned my addr 172.21.15.1:0/950261492 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.406+0000 7f5e26888640 1 --2- 172.21.15.1:0/950261492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e62250 0x5635f4ed1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.406+0000 7f5e26087640 1 --2- 172.21.15.1:0/950261492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e3cf50 0x5635f4e75af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.406+0000 7f5e25886640 1 -- 172.21.15.1:0/950261492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e62250 msgr2=0x5635f4ed1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.406+0000 7f5e25886640 1 --2- 172.21.15.1:0/950261492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e62250 0x5635f4ed1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.406+0000 7f5e25886640 1 -- 172.21.15.1:0/950261492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e3cf50 msgr2=0x5635f4e75af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.406+0000 7f5e25886640 1 --2- 172.21.15.1:0/950261492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e3cf50 0x5635f4e75af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.406+0000 7f5e25886640 1 -- 172.21.15.1:0/950261492 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635f4e5c3e0 con 0x5635f4e3d330 2024-03-06T07:29:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.407+0000 7f5e25886640 1 --2- 172.21.15.1:0/950261492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3d330 0x5635f4eec2d0 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f5e1007a060 tx=0x7f5e1007a380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=397c447734f5af4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.407+0000 7f5e017fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-06T07:29:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.407+0000 7f5e017fa640 1 -- 172.21.15.1:0/950261492 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e10078660 con 0x5635f4e3d330 2024-03-06T07:29:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.407+0000 7f5e017fa640 1 -- 172.21.15.1:0/950261492 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5e1007e120 con 0x5635f4e3d330 2024-03-06T07:29:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.407+0000 7f5e2afae740 1 -- 172.21.15.1:0/950261492 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e1c041070 con 0x5635f4e3d330 2024-03-06T07:29:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.407+0000 7f5e017fa640 1 -- 172.21.15.1:0/950261492 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=199) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e100769d0 con 0x5635f4e3d330 2024-03-06T07:29:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.407+0000 7f5e2afae740 10 client.11034 fetch_fsmap learned FSMap version 199 2024-03-06T07:29:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.407+0000 7f5e2afae740 1 -- 172.21.15.1:0/950261492 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=199,monmap=2+}) v3 -- 0x5635f4eec9e0 con 0x5635f4e3d330 2024-03-06T07:29:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.407+0000 7f5e017fa640 1 -- 172.21.15.1:0/950261492 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1c041070 con 0x5635f4e3d330 2024-03-06T07:29:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.407+0000 7f5e017fa640 1 -- 172.21.15.1:0/950261492 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 199) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f5e10080440 con 0x5635f4e3d330 2024-03-06T07:29:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.407+0000 7f5e2afae740 10 client.11034 fetch_fsmap finished waiting for FSMap version 199 2024-03-06T07:29:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.407+0000 7f5e2afae740 10 client.11034 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:29:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.407+0000 7f5e2afae740 20 client.11034 populate_metadata read hostname 'smithi001' 2024-03-06T07:29:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.407+0000 7f5e2afae740 1 --2- 172.21.15.1:0/950261492 >> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] conn(0x5635f4e5ab80 0x5635f4e5af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.407+0000 7f5e2afae740 4 client.11034 mds_command: new command op to 10806 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:29:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.407+0000 7f5e2afae740 1 -- 172.21.15.1:0/950261492 --> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5635f4daebf0 con 0x5635f4e5ab80 2024-03-06T07:29:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.407+0000 7f5e26087640 1 --2- 172.21.15.1:0/950261492 >> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] conn(0x5635f4e5ab80 0x5635f4e5af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.408+0000 7f5e26087640 1 --2- 172.21.15.1:0/950261492 >> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] conn(0x5635f4e5ab80 0x5635f4e5af60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff4d3fe2d40978a1 server_cookie=7f4b21ffbe1f9ed1 in_seq=0 out_seq=0 2024-03-06T07:29:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.408+0000 7f5e017fa640 10 client.11034 ms_handle_connect on v2:172.21.15.1:6833/1871567565 2024-03-06T07:29:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.408+0000 7f5e017fa640 1 -- 172.21.15.1:0/950261492 <== mds.0 v2:172.21.15.1:6833/1871567565 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x5635f4daebf0 con 0x5635f4e5ab80 2024-03-06T07:29:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.408+0000 7f5e017fa640 10 client.11034 handle_command_reply: tid=0 2024-03-06T07:29:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.408+0000 7f5e2afae740 1 client.11034 shutdown 2024-03-06T07:29:43.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.408+0000 7f5e2afae740 1 -- 172.21.15.1:0/950261492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3d330 msgr2=0x5635f4eec2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.408+0000 7f5e2afae740 1 --2- 172.21.15.1:0/950261492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3d330 0x5635f4eec2d0 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f5e1007a060 tx=0x7f5e1007a380 comp rx=0 tx=0).stop 2024-03-06T07:29:43.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.408+0000 7f5e2afae740 1 -- 172.21.15.1:0/950261492 shutdown_connections 2024-03-06T07:29:43.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.408+0000 7f5e2afae740 1 --2- 172.21.15.1:0/950261492 >> [v2:172.21.15.1:6833/1871567565,v1:172.21.15.1:6835/1871567565] conn(0x5635f4e5ab80 0x5635f4e5af60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.408+0000 7f5e2afae740 1 --2- 172.21.15.1:0/950261492 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x5635f4e62250 0x5635f4ed1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.408+0000 7f5e2afae740 1 --2- 172.21.15.1:0/950261492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635f4e3d330 0x5635f4eec2d0 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.408+0000 7f5e2afae740 1 --2- 172.21.15.1:0/950261492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5635f4e3cf50 0x5635f4e75af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.408+0000 7f5e017fa640 0 client.11034 ms_handle_reset on v2:172.21.15.1:6833/1871567565 2024-03-06T07:29:43.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.408+0000 7f5e2afae740 1 -- 172.21.15.1:0/950261492 >> 172.21.15.1:0/950261492 conn(0x5635f4e3b4b0 msgr2=0x5635f4e60820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.408+0000 7f5e2afae740 1 -- 172.21.15.1:0/950261492 shutdown_connections 2024-03-06T07:29:43.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.408+0000 7f5e2afae740 1 -- 172.21.15.1:0/950261492 wait complete. 2024-03-06T07:29:43.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.409+0000 7f5e2afae740 20 client.11034 trim_cache size 0 max 16384 2024-03-06T07:29:43.416 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:29:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "8f3c19ba-4ac6-4a54-88df-9262183d23ef", 2024-03-06T07:29:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-06T07:29:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 10806, 2024-03-06T07:29:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-06T07:29:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-06T07:29:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-06T07:29:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 9.9183482519999995, 2024-03-06T07:29:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 199, 2024-03-06T07:29:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 210, 2024-03-06T07:29:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 210, 2024-03-06T07:29:43.417 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 15.001895566 2024-03-06T07:29:43.417 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:29:43.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.409+0000 7f5e282eb640 1 -- 172.21.15.1:0/3695644190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e0406b4d0 msgr2=0x7f5e0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.409+0000 7f5e282eb640 1 --2- 172.21.15.1:0/3695644190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e0406b4d0 0x7f5e0406d990 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c005880 tx=0x7f5e1c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:43.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.409+0000 7f5e282eb640 1 -- 172.21.15.1:0/3695644190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e20152650 msgr2=0x7f5e2013d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.409+0000 7f5e282eb640 1 --2- 172.21.15.1:0/3695644190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e20152650 0x7f5e2013d200 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f5e10009890 tx=0x7f5e100046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.409+0000 7f5e282eb640 1 -- 172.21.15.1:0/3695644190 shutdown_connections 2024-03-06T07:29:43.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.409+0000 7f5e282eb640 1 --2- 172.21.15.1:0/3695644190 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5e0406b4d0 0x7f5e0406d990 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.409+0000 7f5e282eb640 1 --2- 172.21.15.1:0/3695644190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e2014ba40 0x7f5e201eb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.409+0000 7f5e282eb640 1 --2- 172.21.15.1:0/3695644190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e20152650 0x7f5e2013d200 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.409+0000 7f5e282eb640 1 --2- 172.21.15.1:0/3695644190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e200bf9d0 0x7f5e2014d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.409+0000 7f5e282eb640 1 -- 172.21.15.1:0/3695644190 >> 172.21.15.1:0/3695644190 conn(0x7f5e20066ee0 msgr2=0x7f5e200c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.409+0000 7f5e282eb640 1 -- 172.21.15.1:0/3695644190 shutdown_connections 2024-03-06T07:29:43.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.409+0000 7f5e282eb640 1 -- 172.21.15.1:0/3695644190 wait complete. 2024-03-06T07:29:43.422 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2024-03-06T07:29:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.565+0000 7f6cdafc9640 1 Processor -- start 2024-03-06T07:29:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.565+0000 7f6cdafc9640 1 -- start start 2024-03-06T07:29:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.566+0000 7f6cdafc9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6cd40c32f0 0x7f6cd41543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.566+0000 7f6cdafc9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6cd4070c80 0x7f6cd40710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.566+0000 7f6cdafc9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd4150900 con 0x7f6cd4150520 2024-03-06T07:29:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.566+0000 7f6cdafc9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cd4153ed0 con 0x7f6cd4070c80 2024-03-06T07:29:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.566+0000 7f6cdafc9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd4071790 con 0x7f6cd40c32f0 2024-03-06T07:29:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.566+0000 7f6cd9566640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6cd4070c80 0x7f6cd40710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.566+0000 7f6cd9566640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6cd4070c80 0x7f6cd40710f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56370/0 (socket says 172.21.15.1:56370) 2024-03-06T07:29:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.566+0000 7f6cd9566640 1 -- 172.21.15.1:0/155291884 learned_addr learned my addr 172.21.15.1:0/155291884 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.567+0000 7f6cd3fff640 1 --2- 172.21.15.1:0/155291884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6cd40c32f0 0x7f6cd41543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.567+0000 7f6cd9566640 1 -- 172.21.15.1:0/155291884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6cd40c32f0 msgr2=0x7f6cd41543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.567+0000 7f6cd9566640 1 --2- 172.21.15.1:0/155291884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6cd40c32f0 0x7f6cd41543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.567+0000 7f6cd9566640 1 -- 172.21.15.1:0/155291884 >> v1:172.21.15.98:6789/0 conn(0x7f6cd4150520 legacy=0x7f6cd41537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.567+0000 7f6cd9566640 1 -- 172.21.15.1:0/155291884 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd4157050 con 0x7f6cd4070c80 2024-03-06T07:29:43.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.567+0000 7f6cd3fff640 1 --2- 172.21.15.1:0/155291884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6cd40c32f0 0x7f6cd41543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.568+0000 7f6cd9566640 1 --2- 172.21.15.1:0/155291884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cd4070c80 0x7f6cd40710f0 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc009e50 tx=0x7f6ccc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d182d42d0267c367 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.568+0000 7f6cd37fe640 1 -- 172.21.15.1:0/155291884 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc02ca70 con 0x7f6cd4070c80 2024-03-06T07:29:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.568+0000 7f6cd37fe640 1 -- 172.21.15.1:0/155291884 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ccc02cc10 con 0x7f6cd4070c80 2024-03-06T07:29:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.568+0000 7f6cd37fe640 1 -- 172.21.15.1:0/155291884 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc0343f0 con 0x7f6cd4070c80 2024-03-06T07:29:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.569+0000 7f6cdafc9640 1 -- 172.21.15.1:0/155291884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cd4070c80 msgr2=0x7f6cd40710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.569+0000 7f6cdafc9640 1 --2- 172.21.15.1:0/155291884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cd4070c80 0x7f6cd40710f0 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc009e50 tx=0x7f6ccc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.569+0000 7f6cdafc9640 1 -- 172.21.15.1:0/155291884 shutdown_connections 2024-03-06T07:29:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.569+0000 7f6cdafc9640 1 --2- 172.21.15.1:0/155291884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cd4070c80 0x7f6cd40710f0 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.569+0000 7f6cdafc9640 1 --2- 172.21.15.1:0/155291884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6cd40c32f0 0x7f6cd41543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.569+0000 7f6cdafc9640 1 -- 172.21.15.1:0/155291884 >> 172.21.15.1:0/155291884 conn(0x7f6cd40c88f0 msgr2=0x7f6cd40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.569+0000 7f6cdafc9640 1 -- 172.21.15.1:0/155291884 shutdown_connections 2024-03-06T07:29:43.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.569+0000 7f6cdafc9640 1 -- 172.21.15.1:0/155291884 wait complete. 2024-03-06T07:29:43.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.570+0000 7f6cdafc9640 1 Processor -- start 2024-03-06T07:29:43.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.570+0000 7f6cdafc9640 1 -- start start 2024-03-06T07:29:43.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cdafc9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cd4070c80 0x7f6cd4155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cdafc9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cd40c32f0 0x7f6cd41494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cdafc9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6cd4150520 0x7f6cd4149a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cdafc9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd4157330 con 0x7f6cd40c32f0 2024-03-06T07:29:43.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cdafc9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd41532a0 con 0x7f6cd4070c80 2024-03-06T07:29:43.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cdafc9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd40c7ec0 con 0x7f6cd4150520 2024-03-06T07:29:43.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cd3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cd40c32f0 0x7f6cd41494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cd3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cd40c32f0 0x7f6cd41494f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56372/0 (socket says 172.21.15.1:56372) 2024-03-06T07:29:43.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cd3fff640 1 -- 172.21.15.1:0/3195542650 learned_addr learned my addr 172.21.15.1:0/3195542650 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:43.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cd9566640 1 --2- 172.21.15.1:0/3195542650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6cd4150520 0x7f6cd4149a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cd8d65640 1 --2- 172.21.15.1:0/3195542650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cd4070c80 0x7f6cd4155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cd3fff640 1 -- 172.21.15.1:0/3195542650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6cd4150520 msgr2=0x7f6cd4149a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cd3fff640 1 --2- 172.21.15.1:0/3195542650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6cd4150520 0x7f6cd4149a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cd3fff640 1 -- 172.21.15.1:0/3195542650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cd4070c80 msgr2=0x7f6cd4155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cd3fff640 1 --2- 172.21.15.1:0/3195542650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cd4070c80 0x7f6cd4155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cd3fff640 1 -- 172.21.15.1:0/3195542650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd414b2a0 con 0x7f6cd40c32f0 2024-03-06T07:29:43.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cd8d65640 1 --2- 172.21.15.1:0/3195542650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cd4070c80 0x7f6cd4155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:43.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.572+0000 7f6cd3fff640 1 --2- 172.21.15.1:0/3195542650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cd40c32f0 0x7f6cd41494f0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f6cc000c7f0 tx=0x7f6cc000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.573+0000 7f6cd17fa640 1 -- 172.21.15.1:0/3195542650 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0015070 con 0x7f6cd40c32f0 2024-03-06T07:29:43.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.573+0000 7f6cd17fa640 1 -- 172.21.15.1:0/3195542650 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc0007a30 con 0x7f6cd40c32f0 2024-03-06T07:29:43.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.573+0000 7f6cd17fa640 1 -- 172.21.15.1:0/3195542650 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0007e00 con 0x7f6cd40c32f0 2024-03-06T07:29:43.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.573+0000 7f6cdafc9640 1 -- 172.21.15.1:0/3195542650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd41e8d70 con 0x7f6cd40c32f0 2024-03-06T07:29:43.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.573+0000 7f6cdafc9640 1 -- 172.21.15.1:0/3195542650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd40ab080 con 0x7f6cd40c32f0 2024-03-06T07:29:43.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.574+0000 7f6cdafc9640 1 -- 172.21.15.1:0/3195542650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd40aae70 con 0x7f6cd40c32f0 2024-03-06T07:29:43.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.574+0000 7f6cd17fa640 1 -- 172.21.15.1:0/3195542650 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6cc0018070 con 0x7f6cd40c32f0 2024-03-06T07:29:43.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.575+0000 7f6cd17fa640 1 --2- 172.21.15.1:0/3195542650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6cb806b500 0x7f6cb806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.575+0000 7f6cd17fa640 1 -- 172.21.15.1:0/3195542650 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6cc008a230 con 0x7f6cd40c32f0 2024-03-06T07:29:43.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.575+0000 7f6cd8d65640 1 --2- 172.21.15.1:0/3195542650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6cb806b500 0x7f6cb806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.576+0000 7f6cd8d65640 1 --2- 172.21.15.1:0/3195542650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6cb806b500 0x7f6cb806d9c0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8006fd0 tx=0x7f6cc8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.583+0000 7f6cd17fa640 1 -- 172.21.15.1:0/3195542650 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6cc00588d0 con 0x7f6cd40c32f0 2024-03-06T07:29:43.765 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-06T07:29:43.763+0000 7f76a9082640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:29:43.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:43.809+0000 7f6cdafc9640 1 -- 172.21.15.1:0/3195542650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"} v 0) v1 -- 0x7f6cd40c7b10 con 0x7f6cd40c32f0 2024-03-06T07:29:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.307+0000 7f6cd17fa640 1 -- 172.21.15.1:0/3195542650 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"}]=0 cephfs marked down. v200) v1 ==== 125+0+0 (secure 0 0 0) 0x7f6cc005c970 con 0x7f6cd40c32f0 2024-03-06T07:29:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked down. 2024-03-06T07:29:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.309+0000 7f6cdafc9640 1 -- 172.21.15.1:0/3195542650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6cb806b500 msgr2=0x7f6cb806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.309+0000 7f6cdafc9640 1 --2- 172.21.15.1:0/3195542650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6cb806b500 0x7f6cb806d9c0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8006fd0 tx=0x7f6cc8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.309+0000 7f6cdafc9640 1 -- 172.21.15.1:0/3195542650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cd40c32f0 msgr2=0x7f6cd41494f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.309+0000 7f6cdafc9640 1 --2- 172.21.15.1:0/3195542650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cd40c32f0 0x7f6cd41494f0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f6cc000c7f0 tx=0x7f6cc000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.309+0000 7f6cdafc9640 1 -- 172.21.15.1:0/3195542650 shutdown_connections 2024-03-06T07:29:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.309+0000 7f6cdafc9640 1 --2- 172.21.15.1:0/3195542650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6cb806b500 0x7f6cb806d9c0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.309+0000 7f6cdafc9640 1 --2- 172.21.15.1:0/3195542650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6cd4150520 0x7f6cd4149a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.309+0000 7f6cdafc9640 1 --2- 172.21.15.1:0/3195542650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cd40c32f0 0x7f6cd41494f0 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.309+0000 7f6cdafc9640 1 --2- 172.21.15.1:0/3195542650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cd4070c80 0x7f6cd4155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.309+0000 7f6cdafc9640 1 -- 172.21.15.1:0/3195542650 >> 172.21.15.1:0/3195542650 conn(0x7f6cd40c88f0 msgr2=0x7f6cd414a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.310+0000 7f6cdafc9640 1 -- 172.21.15.1:0/3195542650 shutdown_connections 2024-03-06T07:29:44.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.310+0000 7f6cdafc9640 1 -- 172.21.15.1:0/3195542650 wait complete. 2024-03-06T07:29:44.322 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:44.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.422+0000 7f99989f6640 1 Processor -- start 2024-03-06T07:29:44.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.422+0000 7f99989f6640 1 -- start start 2024-03-06T07:29:44.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f99989f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99940bf9d0 0x7f9994155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f99989f6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99940c2fc0 con 0x7f9994152650 2024-03-06T07:29:44.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f99989f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9994158da0 con 0x7f99940c6b10 2024-03-06T07:29:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f99989f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9994155800 con 0x7f99940bf9d0 2024-03-06T07:29:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f999259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9994152650 0x7f9994158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60874/0 (socket says 172.21.15.1:60874) 2024-03-06T07:29:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f999259c640 1 -- 172.21.15.1:0/598404563 learned_addr learned my addr 172.21.15.1:0/598404563 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f9992d9d640 1 --2- 172.21.15.1:0/598404563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99940bf9d0 0x7f9994155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f9991d9b640 1 -- 172.21.15.1:0/598404563 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365048475 0 0) 0x7f99940c2fc0 con 0x7f9994152650 2024-03-06T07:29:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f9991d9b640 1 -- 172.21.15.1:0/598404563 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9980003520 con 0x7f9994152650 2024-03-06T07:29:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f9991d9b640 1 -- 172.21.15.1:0/598404563 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2535082222 0 0) 0x7f9980003520 con 0x7f9994152650 2024-03-06T07:29:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f9991d9b640 1 -- 172.21.15.1:0/598404563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99940bf9d0 msgr2=0x7f9994155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f9991d9b640 1 --2- 172.21.15.1:0/598404563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99940bf9d0 0x7f9994155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f9991d9b640 1 -- 172.21.15.1:0/598404563 >> v1:172.21.15.98:6789/0 conn(0x7f99940c6b10 legacy=0x7f99940c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f9991d9b640 1 -- 172.21.15.1:0/598404563 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99940c6690 con 0x7f9994152650 2024-03-06T07:29:44.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f9991d9b640 1 -- 172.21.15.1:0/598404563 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f9984001690 con 0x7f9994152650 2024-03-06T07:29:44.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f9991d9b640 1 -- 172.21.15.1:0/598404563 >> v1:172.21.15.1:6789/0 conn(0x7f9994152650 legacy=0x7f9994158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f9991d9b640 1 --2- 172.21.15.1:0/598404563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99940bf9d0 0x7f99800039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f9991d9b640 1 --2- 172.21.15.1:0/598404563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9980003f60 0x7f9980004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f9991d9b640 1 --2- 172.21.15.1:0/598404563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9980004d40 0x7f99800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f9991d9b640 1 -- 172.21.15.1:0/598404563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9994155800 con 0x7f9980003f60 2024-03-06T07:29:44.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f9991d9b640 1 -- 172.21.15.1:0/598404563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998000bb50 con 0x7f9980004d40 2024-03-06T07:29:44.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f9991d9b640 1 -- 172.21.15.1:0/598404563 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998000bd50 con 0x7f99940bf9d0 2024-03-06T07:29:44.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f998bfff640 1 --2- 172.21.15.1:0/598404563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9980003f60 0x7f9980004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.423+0000 7f999259c640 1 --2- 172.21.15.1:0/598404563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99940bf9d0 0x7f99800039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.424+0000 7f9992d9d640 1 --2- 172.21.15.1:0/598404563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9980004d40 0x7f99800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.424+0000 7f998bfff640 1 -- 172.21.15.1:0/598404563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99940bf9d0 msgr2=0x7f99800039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.424+0000 7f998bfff640 1 --2- 172.21.15.1:0/598404563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99940bf9d0 0x7f99800039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.424+0000 7f998bfff640 1 -- 172.21.15.1:0/598404563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9980004d40 msgr2=0x7f99800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.424+0000 7f998bfff640 1 --2- 172.21.15.1:0/598404563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9980004d40 0x7f99800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.424+0000 7f998bfff640 1 -- 172.21.15.1:0/598404563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9980003520 con 0x7f9980003f60 2024-03-06T07:29:44.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.424+0000 7f999259c640 1 --2- 172.21.15.1:0/598404563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99940bf9d0 0x7f99800039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:44.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.425+0000 7f998bfff640 1 --2- 172.21.15.1:0/598404563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9980003f60 0x7f9980004800 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f997c001750 tx=0x7f9980005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=128048adfe4660ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.425+0000 7f9991d9b640 1 -- 172.21.15.1:0/598404563 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c001370 con 0x7f9980003f60 2024-03-06T07:29:44.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.425+0000 7f9991d9b640 1 -- 172.21.15.1:0/598404563 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997c0052d0 con 0x7f9980003f60 2024-03-06T07:29:44.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.425+0000 7f9991d9b640 1 -- 172.21.15.1:0/598404563 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c003830 con 0x7f9980003f60 2024-03-06T07:29:44.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.425+0000 7f99989f6640 1 -- 172.21.15.1:0/598404563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9980003f60 msgr2=0x7f9980004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.425+0000 7f99989f6640 1 --2- 172.21.15.1:0/598404563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9980003f60 0x7f9980004800 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f997c001750 tx=0x7f9980005830 comp rx=0 tx=0).stop 2024-03-06T07:29:44.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.425+0000 7f9992d9d640 1 -- 172.21.15.1:0/598404563 reap_dead start 2024-03-06T07:29:44.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.425+0000 7f99989f6640 1 -- 172.21.15.1:0/598404563 shutdown_connections 2024-03-06T07:29:44.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.425+0000 7f99989f6640 1 -- 172.21.15.1:0/598404563 >> 172.21.15.1:0/598404563 conn(0x7f9994066ee0 msgr2=0x7f99940672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.425+0000 7f99989f6640 1 -- 172.21.15.1:0/598404563 shutdown_connections 2024-03-06T07:29:44.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.425+0000 7f99989f6640 1 -- 172.21.15.1:0/598404563 wait complete. 2024-03-06T07:29:44.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.426+0000 7f99989f6640 1 Processor -- start 2024-03-06T07:29:44.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.427+0000 7f99989f6640 1 -- start start 2024-03-06T07:29:44.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.427+0000 7f99989f6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9980003f60 0x7f9994157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.427+0000 7f99989f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99941587b0 0x7f99941ec480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.427+0000 7f99989f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f999413d200 0x7f999413d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.427+0000 7f99989f6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99941551b0 con 0x7f9980003f60 2024-03-06T07:29:44.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.427+0000 7f99989f6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99940c6fa0 con 0x7f99941587b0 2024-03-06T07:29:44.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.427+0000 7f99989f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9994155490 con 0x7f999413d200 2024-03-06T07:29:44.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.427+0000 7f999259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9980003f60 0x7f9994157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.427+0000 7f999259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9980003f60 0x7f9994157980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59660/0 (socket says 172.21.15.1:59660) 2024-03-06T07:29:44.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.427+0000 7f999259c640 1 -- 172.21.15.1:0/107628776 learned_addr learned my addr 172.21.15.1:0/107628776 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:44.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.427+0000 7f9992d9d640 1 --2- 172.21.15.1:0/107628776 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f999413d200 0x7f999413d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.427+0000 7f998bfff640 1 --2- 172.21.15.1:0/107628776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99941587b0 0x7f99941ec480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.428+0000 7f999259c640 1 -- 172.21.15.1:0/107628776 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f999413d200 msgr2=0x7f999413d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.428+0000 7f999259c640 1 --2- 172.21.15.1:0/107628776 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f999413d200 0x7f999413d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.428+0000 7f999259c640 1 -- 172.21.15.1:0/107628776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99941587b0 msgr2=0x7f99941ec480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.428+0000 7f999259c640 1 --2- 172.21.15.1:0/107628776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99941587b0 0x7f99941ec480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.428+0000 7f999259c640 1 -- 172.21.15.1:0/107628776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9994157ec0 con 0x7f9980003f60 2024-03-06T07:29:44.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.428+0000 7f999259c640 1 --2- 172.21.15.1:0/107628776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9980003f60 0x7f9994157980 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f9984007310 tx=0x7f99840077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.428+0000 7f998affd640 1 -- 172.21.15.1:0/107628776 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998400f9a0 con 0x7f9980003f60 2024-03-06T07:29:44.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.428+0000 7f998affd640 1 -- 172.21.15.1:0/107628776 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9984018070 con 0x7f9980003f60 2024-03-06T07:29:44.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.428+0000 7f99989f6640 1 -- 172.21.15.1:0/107628776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f999414c540 con 0x7f9980003f60 2024-03-06T07:29:44.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.428+0000 7f998affd640 1 -- 172.21.15.1:0/107628776 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998400fd40 con 0x7f9980003f60 2024-03-06T07:29:44.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.428+0000 7f99989f6640 1 -- 172.21.15.1:0/107628776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9994156ea0 con 0x7f9980003f60 2024-03-06T07:29:44.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.429+0000 7f99989f6640 1 -- 172.21.15.1:0/107628776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9994067fd0 con 0x7f9980003f60 2024-03-06T07:29:44.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.429+0000 7f998affd640 1 -- 172.21.15.1:0/107628776 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f998401d460 con 0x7f9980003f60 2024-03-06T07:29:44.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.430+0000 7f998affd640 1 --2- 172.21.15.1:0/107628776 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f997406b500 0x7f997406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.430+0000 7f998affd640 1 -- 172.21.15.1:0/107628776 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f998408a4d0 con 0x7f9980003f60 2024-03-06T07:29:44.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.430+0000 7f998bfff640 1 --2- 172.21.15.1:0/107628776 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f997406b500 0x7f997406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.431+0000 7f998bfff640 1 --2- 172.21.15.1:0/107628776 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f997406b500 0x7f997406d9c0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f997c001750 tx=0x7f997c009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.434+0000 7f998affd640 1 -- 172.21.15.1:0/107628776 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99840591b0 con 0x7f9980003f60 2024-03-06T07:29:44.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.674+0000 7f99989f6640 1 -- 172.21.15.1:0/107628776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f999406b500 con 0x7f9980003f60 2024-03-06T07:29:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.674+0000 7f998affd640 1 -- 172.21.15.1:0/107628776 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5005 (secure 0 0 0) 0x7f9984058ab0 con 0x7f9980003f60 2024-03-06T07:29:44.676 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:44.676 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:44.306981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:29:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 200 2024-03-06T07:29:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.676+0000 7f99989f6640 1 -- 172.21.15.1:0/107628776 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f997406b500 msgr2=0x7f997406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.676+0000 7f99989f6640 1 --2- 172.21.15.1:0/107628776 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f997406b500 0x7f997406d9c0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f997c001750 tx=0x7f997c009030 comp rx=0 tx=0).stop 2024-03-06T07:29:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.676+0000 7f99989f6640 1 -- 172.21.15.1:0/107628776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9980003f60 msgr2=0x7f9994157980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.676+0000 7f99989f6640 1 --2- 172.21.15.1:0/107628776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9980003f60 0x7f9994157980 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f9984007310 tx=0x7f99840077d0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.676+0000 7f99989f6640 1 -- 172.21.15.1:0/107628776 shutdown_connections 2024-03-06T07:29:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.676+0000 7f99989f6640 1 --2- 172.21.15.1:0/107628776 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f997406b500 0x7f997406d9c0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.676+0000 7f99989f6640 1 --2- 172.21.15.1:0/107628776 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f999413d200 0x7f999413d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.676+0000 7f99989f6640 1 --2- 172.21.15.1:0/107628776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99941587b0 0x7f99941ec480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.676+0000 7f99989f6640 1 --2- 172.21.15.1:0/107628776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9980003f60 0x7f9994157980 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.676+0000 7f99989f6640 1 -- 172.21.15.1:0/107628776 >> 172.21.15.1:0/107628776 conn(0x7f9994066ee0 msgr2=0x7f99941538b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.676+0000 7f99989f6640 1 -- 172.21.15.1:0/107628776 shutdown_connections 2024-03-06T07:29:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:44.676+0000 7f99989f6640 1 -- 172.21.15.1:0/107628776 wait complete. 2024-03-06T07:29:45.091 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:44.306981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:29:45.091 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:stopping 2024-03-06T07:29:46.092 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:46.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.191+0000 7f426c5d0640 1 Processor -- start 2024-03-06T07:29:46.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.191+0000 7f426c5d0640 1 -- start start 2024-03-06T07:29:46.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.192+0000 7f426c5d0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4264152650 0x7f4264155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.192+0000 7f426c5d0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f42640c6b10 0x7f4264158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.192+0000 7f426c5d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42640bf9d0 0x7f42640c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.192+0000 7f426c5d0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4264155800 con 0x7f4264152650 2024-03-06T07:29:46.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.192+0000 7f426c5d0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42640c2fc0 con 0x7f42640c6b10 2024-03-06T07:29:46.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.192+0000 7f426c5d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42640c6690 con 0x7f42640bf9d0 2024-03-06T07:29:46.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.192+0000 7f426a36c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4264152650 0x7f4264155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.192+0000 7f426a36c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4264152650 0x7f4264155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59668/0 (socket says 172.21.15.1:59668) 2024-03-06T07:29:46.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.192+0000 7f426a36c640 1 -- 172.21.15.1:0/3101623953 learned_addr learned my addr 172.21.15.1:0/3101623953 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:46.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.193+0000 7f426ab6d640 1 --2- 172.21.15.1:0/3101623953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42640bf9d0 0x7f42640c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.193+0000 7f4269b6b640 1 --2- 172.21.15.1:0/3101623953 >> v2:172.21.15.98:3300/0 conn(0x7f42640c6b10 0x7f4264158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.193+0000 7f426a36c640 1 -- 172.21.15.1:0/3101623953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42640bf9d0 msgr2=0x7f42640c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:46.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.193+0000 7f426a36c640 1 --2- 172.21.15.1:0/3101623953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42640bf9d0 0x7f42640c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.193+0000 7f426a36c640 1 -- 172.21.15.1:0/3101623953 >> v2:172.21.15.98:3300/0 conn(0x7f42640c6b10 msgr2=0x7f4264158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:46.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.193+0000 7f426a36c640 1 --2- 172.21.15.1:0/3101623953 >> v2:172.21.15.98:3300/0 conn(0x7f42640c6b10 0x7f4264158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.193+0000 7f426a36c640 1 -- 172.21.15.1:0/3101623953 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42640c7280 con 0x7f4264152650 2024-03-06T07:29:46.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.193+0000 7f4269b6b640 1 --2- 172.21.15.1:0/3101623953 >> v2:172.21.15.98:3300/0 conn(0x7f42640c6b10 0x7f4264158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:46.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.193+0000 7f426a36c640 1 --2- 172.21.15.1:0/3101623953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4264152650 0x7f4264155c90 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f42640cb8e0 tx=0x7f425402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37211eb6e24b41bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.194+0000 7f426936a640 1 -- 172.21.15.1:0/3101623953 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425402e070 con 0x7f4264152650 2024-03-06T07:29:46.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.194+0000 7f426936a640 1 -- 172.21.15.1:0/3101623953 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4254037070 con 0x7f4264152650 2024-03-06T07:29:46.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.194+0000 7f426936a640 1 -- 172.21.15.1:0/3101623953 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4254031a00 con 0x7f4264152650 2024-03-06T07:29:46.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.194+0000 7f426c5d0640 1 -- 172.21.15.1:0/3101623953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4264152650 msgr2=0x7f4264155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.194+0000 7f426c5d0640 1 --2- 172.21.15.1:0/3101623953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4264152650 0x7f4264155c90 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f42640cb8e0 tx=0x7f425402b880 comp rx=0 tx=0).stop 2024-03-06T07:29:46.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.194+0000 7f426c5d0640 1 -- 172.21.15.1:0/3101623953 shutdown_connections 2024-03-06T07:29:46.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.194+0000 7f426c5d0640 1 --2- 172.21.15.1:0/3101623953 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42640bf9d0 0x7f42640c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.194+0000 7f426c5d0640 1 --2- 172.21.15.1:0/3101623953 >> v2:172.21.15.98:3300/0 conn(0x7f42640c6b10 0x7f4264158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.194+0000 7f426c5d0640 1 --2- 172.21.15.1:0/3101623953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4264152650 0x7f4264155c90 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.194+0000 7f426c5d0640 1 -- 172.21.15.1:0/3101623953 >> 172.21.15.1:0/3101623953 conn(0x7f4264066ee0 msgr2=0x7f42640672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.194+0000 7f426c5d0640 1 -- 172.21.15.1:0/3101623953 shutdown_connections 2024-03-06T07:29:46.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.194+0000 7f426c5d0640 1 -- 172.21.15.1:0/3101623953 wait complete. 2024-03-06T07:29:46.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.195+0000 7f426c5d0640 1 Processor -- start 2024-03-06T07:29:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.195+0000 7f426c5d0640 1 -- start start 2024-03-06T07:29:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.196+0000 7f426c5d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42640bf9d0 0x7f42641ee300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.196+0000 7f426c5d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4264152650 0x7f4264144610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.196+0000 7f426c5d0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4264154bd0 0x7f42641eb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.196+0000 7f426c5d0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4264155490 con 0x7f4264154bd0 2024-03-06T07:29:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.196+0000 7f426c5d0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42641551b0 con 0x7f42640bf9d0 2024-03-06T07:29:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.196+0000 7f426c5d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4264152a30 con 0x7f4264152650 2024-03-06T07:29:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.196+0000 7f426ab6d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4264154bd0 0x7f42641eb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.196+0000 7f426ab6d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4264154bd0 0x7f42641eb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59684/0 (socket says 172.21.15.1:59684) 2024-03-06T07:29:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.196+0000 7f426ab6d640 1 -- 172.21.15.1:0/2196315397 learned_addr learned my addr 172.21.15.1:0/2196315397 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.196+0000 7f426a36c640 1 --2- 172.21.15.1:0/2196315397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42640bf9d0 0x7f42641ee300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.196+0000 7f4269b6b640 1 --2- 172.21.15.1:0/2196315397 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4264152650 0x7f4264144610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.197+0000 7f426ab6d640 1 -- 172.21.15.1:0/2196315397 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4264152650 msgr2=0x7f4264144610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.197+0000 7f426ab6d640 1 --2- 172.21.15.1:0/2196315397 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4264152650 0x7f4264144610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.197+0000 7f426ab6d640 1 -- 172.21.15.1:0/2196315397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42640bf9d0 msgr2=0x7f42641ee300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.197+0000 7f426ab6d640 1 --2- 172.21.15.1:0/2196315397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42640bf9d0 0x7f42641ee300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.197+0000 7f426ab6d640 1 -- 172.21.15.1:0/2196315397 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42640ca270 con 0x7f4264154bd0 2024-03-06T07:29:46.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.197+0000 7f4269b6b640 1 --2- 172.21.15.1:0/2196315397 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4264152650 0x7f4264144610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:46.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.197+0000 7f426ab6d640 1 --2- 172.21.15.1:0/2196315397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4264154bd0 0x7f42641eb570 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f425c00cbb0 tx=0x7f425c00a4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.197+0000 7f425affd640 1 -- 172.21.15.1:0/2196315397 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c015070 con 0x7f4264154bd0 2024-03-06T07:29:46.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.197+0000 7f425affd640 1 -- 172.21.15.1:0/2196315397 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425c00abd0 con 0x7f4264154bd0 2024-03-06T07:29:46.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.197+0000 7f426c5d0640 1 -- 172.21.15.1:0/2196315397 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42640ca550 con 0x7f4264154bd0 2024-03-06T07:29:46.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.197+0000 7f425affd640 1 -- 172.21.15.1:0/2196315397 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c00d040 con 0x7f4264154bd0 2024-03-06T07:29:46.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.197+0000 7f426c5d0640 1 -- 172.21.15.1:0/2196315397 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42641ead60 con 0x7f4264154bd0 2024-03-06T07:29:46.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.198+0000 7f426c5d0640 1 -- 172.21.15.1:0/2196315397 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4264067fd0 con 0x7f4264154bd0 2024-03-06T07:29:46.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.198+0000 7f425affd640 1 -- 172.21.15.1:0/2196315397 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f425c018070 con 0x7f4264154bd0 2024-03-06T07:29:46.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.199+0000 7f425affd640 1 --2- 172.21.15.1:0/2196315397 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f424806b500 0x7f424806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.199+0000 7f425affd640 1 -- 172.21.15.1:0/2196315397 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f425c089e50 con 0x7f4264154bd0 2024-03-06T07:29:46.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.199+0000 7f426a36c640 1 --2- 172.21.15.1:0/2196315397 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f424806b500 0x7f424806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.200+0000 7f426a36c640 1 --2- 172.21.15.1:0/2196315397 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f424806b500 0x7f424806d9c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f4254036000 tx=0x7f425403d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.203+0000 7f425affd640 1 -- 172.21.15.1:0/2196315397 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f425c058580 con 0x7f4264154bd0 2024-03-06T07:29:46.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.423+0000 7f426c5d0640 1 -- 172.21.15.1:0/2196315397 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42640b01a0 con 0x7f4264154bd0 2024-03-06T07:29:46.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.423+0000 7f425affd640 1 -- 172.21.15.1:0/2196315397 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5005 (secure 0 0 0) 0x7f425c05c620 con 0x7f4264154bd0 2024-03-06T07:29:46.426 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:46.426 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:44.306981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:29:46.426 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 200 2024-03-06T07:29:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.426+0000 7f426c5d0640 1 -- 172.21.15.1:0/2196315397 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f424806b500 msgr2=0x7f424806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.426+0000 7f426c5d0640 1 --2- 172.21.15.1:0/2196315397 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f424806b500 0x7f424806d9c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f4254036000 tx=0x7f425403d040 comp rx=0 tx=0).stop 2024-03-06T07:29:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.426+0000 7f426c5d0640 1 -- 172.21.15.1:0/2196315397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4264154bd0 msgr2=0x7f42641eb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.426+0000 7f426c5d0640 1 --2- 172.21.15.1:0/2196315397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4264154bd0 0x7f42641eb570 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f425c00cbb0 tx=0x7f425c00a4c0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.426+0000 7f426c5d0640 1 -- 172.21.15.1:0/2196315397 shutdown_connections 2024-03-06T07:29:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.426+0000 7f426c5d0640 1 --2- 172.21.15.1:0/2196315397 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f424806b500 0x7f424806d9c0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.426+0000 7f426c5d0640 1 --2- 172.21.15.1:0/2196315397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4264154bd0 0x7f42641eb570 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.426+0000 7f426c5d0640 1 --2- 172.21.15.1:0/2196315397 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4264152650 0x7f4264144610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.426+0000 7f426c5d0640 1 --2- 172.21.15.1:0/2196315397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42640bf9d0 0x7f42641ee300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.426+0000 7f426c5d0640 1 -- 172.21.15.1:0/2196315397 >> 172.21.15.1:0/2196315397 conn(0x7f4264066ee0 msgr2=0x7f42640cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.426+0000 7f426c5d0640 1 -- 172.21.15.1:0/2196315397 shutdown_connections 2024-03-06T07:29:46.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:46.426+0000 7f426c5d0640 1 -- 172.21.15.1:0/2196315397 wait complete. 2024-03-06T07:29:46.840 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:44.306981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:29:46.841 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:stopping 2024-03-06T07:29:47.842 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:47.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.941+0000 7f2b8f53e640 1 Processor -- start 2024-03-06T07:29:47.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.942+0000 7f2b8f53e640 1 -- start start 2024-03-06T07:29:47.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.942+0000 7f2b8f53e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2b880c3970 0x7f2b880c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.942+0000 7f2b8f53e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b880c60b0 0x7f2b88152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.942+0000 7f2b8f53e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b880c5e40 con 0x7f2b88152670 2024-03-06T07:29:47.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.942+0000 7f2b8f53e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b880c67f0 con 0x7f2b880c3970 2024-03-06T07:29:47.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.942+0000 7f2b8f53e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880c6a40 con 0x7f2b880c60b0 2024-03-06T07:29:47.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.943+0000 7f2b8cad9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b880c60b0 0x7f2b88152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.943+0000 7f2b8d2da640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2b880c3970 0x7f2b880c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.943+0000 7f2b8cad9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b880c60b0 0x7f2b88152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:37620/0 (socket says 172.21.15.1:37620) 2024-03-06T07:29:47.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.943+0000 7f2b8d2da640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2b880c3970 0x7f2b880c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:38346/0 (socket says 172.21.15.1:38346) 2024-03-06T07:29:47.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.943+0000 7f2b8cad9640 1 -- 172.21.15.1:0/2486058857 learned_addr learned my addr 172.21.15.1:0/2486058857 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:47.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.943+0000 7f2b8d2da640 1 -- 172.21.15.1:0/2486058857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b880c60b0 msgr2=0x7f2b88152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:47.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.943+0000 7f2b8d2da640 1 --2- 172.21.15.1:0/2486058857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b880c60b0 0x7f2b88152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.943+0000 7f2b8d2da640 1 -- 172.21.15.1:0/2486058857 >> v1:172.21.15.98:6789/0 conn(0x7f2b88152670 legacy=0x7f2b881553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.943+0000 7f2b7ffff640 1 -- 172.21.15.1:0/2486058857 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045129005 0 0) 0x7f2b880c5e40 con 0x7f2b88152670 2024-03-06T07:29:47.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.943+0000 7f2b8d2da640 1 -- 172.21.15.1:0/2486058857 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b880c6d20 con 0x7f2b880c3970 2024-03-06T07:29:47.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.944+0000 7f2b8d2da640 1 --2- 172.21.15.1:0/2486058857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b880c3970 0x7f2b880c8170 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f2b840097a0 tx=0x7f2b8402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b26227994aa5c30a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.944+0000 7f2b7ffff640 1 -- 172.21.15.1:0/2486058857 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b84035020 con 0x7f2b880c3970 2024-03-06T07:29:47.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.944+0000 7f2b7ffff640 1 -- 172.21.15.1:0/2486058857 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b84031830 con 0x7f2b880c3970 2024-03-06T07:29:47.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.944+0000 7f2b7ffff640 1 -- 172.21.15.1:0/2486058857 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b84031c00 con 0x7f2b880c3970 2024-03-06T07:29:47.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.945+0000 7f2b8f53e640 1 -- 172.21.15.1:0/2486058857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b880c3970 msgr2=0x7f2b880c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.945+0000 7f2b8f53e640 1 --2- 172.21.15.1:0/2486058857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b880c3970 0x7f2b880c8170 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f2b840097a0 tx=0x7f2b8402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.945+0000 7f2b8f53e640 1 -- 172.21.15.1:0/2486058857 shutdown_connections 2024-03-06T07:29:47.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.945+0000 7f2b8f53e640 1 --2- 172.21.15.1:0/2486058857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b880c60b0 0x7f2b88152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.945+0000 7f2b8f53e640 1 --2- 172.21.15.1:0/2486058857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b880c3970 0x7f2b880c8170 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.945+0000 7f2b8f53e640 1 -- 172.21.15.1:0/2486058857 >> 172.21.15.1:0/2486058857 conn(0x7f2b880c88f0 msgr2=0x7f2b880c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:47.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.945+0000 7f2b8f53e640 1 -- 172.21.15.1:0/2486058857 shutdown_connections 2024-03-06T07:29:47.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.945+0000 7f2b8f53e640 1 -- 172.21.15.1:0/2486058857 wait complete. 2024-03-06T07:29:47.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.946+0000 7f2b8f53e640 1 Processor -- start 2024-03-06T07:29:47.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.946+0000 7f2b8f53e640 1 -- start start 2024-03-06T07:29:47.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8f53e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b880c3970 0x7f2b881541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8f53e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b880c60b0 0x7f2b881420a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8f53e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b88152670 0x7f2b881425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8f53e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880c75c0 con 0x7f2b88152670 2024-03-06T07:29:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8f53e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880c72e0 con 0x7f2b880c3970 2024-03-06T07:29:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8f53e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880c3d50 con 0x7f2b880c60b0 2024-03-06T07:29:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8dadb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b88152670 0x7f2b881425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8dadb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b88152670 0x7f2b881425e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59698/0 (socket says 172.21.15.1:59698) 2024-03-06T07:29:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8dadb640 1 -- 172.21.15.1:0/1228440469 learned_addr learned my addr 172.21.15.1:0/1228440469 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8cad9640 1 --2- 172.21.15.1:0/1228440469 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b880c60b0 0x7f2b881420a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8d2da640 1 --2- 172.21.15.1:0/1228440469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b880c3970 0x7f2b881541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8dadb640 1 -- 172.21.15.1:0/1228440469 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b880c60b0 msgr2=0x7f2b881420a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8dadb640 1 --2- 172.21.15.1:0/1228440469 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b880c60b0 0x7f2b881420a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8dadb640 1 -- 172.21.15.1:0/1228440469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b880c3970 msgr2=0x7f2b881541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8dadb640 1 --2- 172.21.15.1:0/1228440469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b880c3970 0x7f2b881541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8dadb640 1 -- 172.21.15.1:0/1228440469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8806f580 con 0x7f2b88152670 2024-03-06T07:29:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8dadb640 1 --2- 172.21.15.1:0/1228440469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b88152670 0x7f2b881425e0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000c780 tx=0x7f2b8000cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b7dffb640 1 -- 172.21.15.1:0/1228440469 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b80003b70 con 0x7f2b88152670 2024-03-06T07:29:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b7dffb640 1 -- 172.21.15.1:0/1228440469 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b80003d10 con 0x7f2b88152670 2024-03-06T07:29:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b8f53e640 1 -- 172.21.15.1:0/1228440469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b880adeb0 con 0x7f2b88152670 2024-03-06T07:29:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.947+0000 7f2b7dffb640 1 -- 172.21.15.1:0/1228440469 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b80005270 con 0x7f2b88152670 2024-03-06T07:29:47.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.948+0000 7f2b8f53e640 1 -- 172.21.15.1:0/1228440469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b881eb650 con 0x7f2b88152670 2024-03-06T07:29:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.949+0000 7f2b8f53e640 1 -- 172.21.15.1:0/1228440469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b8806f750 con 0x7f2b88152670 2024-03-06T07:29:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.949+0000 7f2b7dffb640 1 -- 172.21.15.1:0/1228440469 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b80005410 con 0x7f2b88152670 2024-03-06T07:29:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.949+0000 7f2b7dffb640 1 --2- 172.21.15.1:0/1228440469 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b6c06b4d0 0x7f2b6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.950+0000 7f2b7dffb640 1 -- 172.21.15.1:0/1228440469 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2b8008a180 con 0x7f2b88152670 2024-03-06T07:29:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.950+0000 7f2b8d2da640 1 --2- 172.21.15.1:0/1228440469 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b6c06b4d0 0x7f2b6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.950+0000 7f2b8d2da640 1 --2- 172.21.15.1:0/1228440469 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b6c06b4d0 0x7f2b6c06d990 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f2b84036000 tx=0x7f2b84008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:47.954+0000 7f2b7dffb640 1 -- 172.21.15.1:0/1228440469 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b80058f00 con 0x7f2b88152670 2024-03-06T07:29:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:48.187+0000 7f2b8f53e640 1 -- 172.21.15.1:0/1228440469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b880c7f20 con 0x7f2b88152670 2024-03-06T07:29:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:48.187+0000 7f2b7dffb640 1 -- 172.21.15.1:0/1228440469 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5005 (secure 0 0 0) 0x7f2b80058800 con 0x7f2b88152670 2024-03-06T07:29:48.190 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:48.190 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:44.306981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:29:48.190 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 200 2024-03-06T07:29:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:48.189+0000 7f2b8f53e640 1 -- 172.21.15.1:0/1228440469 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b6c06b4d0 msgr2=0x7f2b6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:48.189+0000 7f2b8f53e640 1 --2- 172.21.15.1:0/1228440469 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b6c06b4d0 0x7f2b6c06d990 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f2b84036000 tx=0x7f2b84008030 comp rx=0 tx=0).stop 2024-03-06T07:29:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:48.189+0000 7f2b8f53e640 1 -- 172.21.15.1:0/1228440469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b88152670 msgr2=0x7f2b881425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:48.189+0000 7f2b8f53e640 1 --2- 172.21.15.1:0/1228440469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b88152670 0x7f2b881425e0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000c780 tx=0x7f2b8000cb80 comp rx=0 tx=0).stop 2024-03-06T07:29:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:48.189+0000 7f2b8f53e640 1 -- 172.21.15.1:0/1228440469 shutdown_connections 2024-03-06T07:29:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:48.189+0000 7f2b8f53e640 1 --2- 172.21.15.1:0/1228440469 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2b6c06b4d0 0x7f2b6c06d990 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:48.189+0000 7f2b8f53e640 1 --2- 172.21.15.1:0/1228440469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b88152670 0x7f2b881425e0 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:48.189+0000 7f2b8f53e640 1 --2- 172.21.15.1:0/1228440469 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b880c60b0 0x7f2b881420a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:48.189+0000 7f2b8f53e640 1 --2- 172.21.15.1:0/1228440469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b880c3970 0x7f2b881541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:48.189+0000 7f2b8f53e640 1 -- 172.21.15.1:0/1228440469 >> 172.21.15.1:0/1228440469 conn(0x7f2b880c88f0 msgr2=0x7f2b880c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:48.189+0000 7f2b8f53e640 1 -- 172.21.15.1:0/1228440469 shutdown_connections 2024-03-06T07:29:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:48.189+0000 7f2b8f53e640 1 -- 172.21.15.1:0/1228440469 wait complete. 2024-03-06T07:29:48.605 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:44.306981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:29:48.605 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:stopping 2024-03-06T07:29:49.606 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:49.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.705+0000 7f7d8862a640 1 Processor -- start 2024-03-06T07:29:49.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.705+0000 7f7d8862a640 1 -- start start 2024-03-06T07:29:49.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.707+0000 7f7d8862a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d800c2b30 0x7f7d800724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.707+0000 7f7d8862a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d80152b70 0x7f7d80072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.707+0000 7f7d8862a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d80151dd0 con 0x7f7d800a5fa0 2024-03-06T07:29:49.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.707+0000 7f7d8862a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d801526e0 con 0x7f7d800c2b30 2024-03-06T07:29:49.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.707+0000 7f7d8862a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d80156190 con 0x7f7d80152b70 2024-03-06T07:29:49.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.707+0000 7f7d863c6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d800c2b30 0x7f7d800724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.707+0000 7f7d863c6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d800c2b30 0x7f7d800724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59702/0 (socket says 172.21.15.1:59702) 2024-03-06T07:29:49.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.707+0000 7f7d863c6640 1 -- 172.21.15.1:0/789172138 learned_addr learned my addr 172.21.15.1:0/789172138 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:49.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.707+0000 7f7d85bc5640 1 --2- 172.21.15.1:0/789172138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d80152b70 0x7f7d80072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.707+0000 7f7d863c6640 1 -- 172.21.15.1:0/789172138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d80152b70 msgr2=0x7f7d80072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:49.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.707+0000 7f7d863c6640 1 --2- 172.21.15.1:0/789172138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d80152b70 0x7f7d80072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.707+0000 7f7d863c6640 1 -- 172.21.15.1:0/789172138 >> v1:172.21.15.98:6789/0 conn(0x7f7d800a5fa0 legacy=0x7f7d800c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.707+0000 7f7d863c6640 1 -- 172.21.15.1:0/789172138 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d80152090 con 0x7f7d800c2b30 2024-03-06T07:29:49.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.708+0000 7f7d863c6640 1 --2- 172.21.15.1:0/789172138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d800c2b30 0x7f7d800724c0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009e50 tx=0x7f7d7c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e3a6b3081d266a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.708+0000 7f7d853c4640 1 -- 172.21.15.1:0/789172138 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c02cb00 con 0x7f7d800c2b30 2024-03-06T07:29:49.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.708+0000 7f7d853c4640 1 -- 172.21.15.1:0/789172138 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7c02cca0 con 0x7f7d800c2b30 2024-03-06T07:29:49.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.708+0000 7f7d853c4640 1 -- 172.21.15.1:0/789172138 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c034610 con 0x7f7d800c2b30 2024-03-06T07:29:49.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.709+0000 7f7d8862a640 1 -- 172.21.15.1:0/789172138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d800c2b30 msgr2=0x7f7d800724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.709+0000 7f7d8862a640 1 --2- 172.21.15.1:0/789172138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d800c2b30 0x7f7d800724c0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009e50 tx=0x7f7d7c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:49.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.709+0000 7f7d8862a640 1 -- 172.21.15.1:0/789172138 shutdown_connections 2024-03-06T07:29:49.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.709+0000 7f7d8862a640 1 --2- 172.21.15.1:0/789172138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d80152b70 0x7f7d80072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.709+0000 7f7d8862a640 1 --2- 172.21.15.1:0/789172138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d800c2b30 0x7f7d800724c0 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.709+0000 7f7d8862a640 1 -- 172.21.15.1:0/789172138 >> 172.21.15.1:0/789172138 conn(0x7f7d800c6860 msgr2=0x7f7d800c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.709+0000 7f7d8862a640 1 -- 172.21.15.1:0/789172138 shutdown_connections 2024-03-06T07:29:49.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.709+0000 7f7d8862a640 1 -- 172.21.15.1:0/789172138 wait complete. 2024-03-06T07:29:49.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.710+0000 7f7d8862a640 1 Processor -- start 2024-03-06T07:29:49.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.710+0000 7f7d8862a640 1 -- start start 2024-03-06T07:29:49.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d8862a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d800a5fa0 0x7f7d801eeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d8862a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d800c2b30 0x7f7d8013d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d8862a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d80152b70 0x7f7d8014e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d8862a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d80155f00 con 0x7f7d80152b70 2024-03-06T07:29:49.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d8862a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d80155d80 con 0x7f7d800a5fa0 2024-03-06T07:29:49.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d8862a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d800c3a00 con 0x7f7d800c2b30 2024-03-06T07:29:49.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d86bc7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d80152b70 0x7f7d8014e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d86bc7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d80152b70 0x7f7d8014e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59708/0 (socket says 172.21.15.1:59708) 2024-03-06T07:29:49.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d86bc7640 1 -- 172.21.15.1:0/2720332205 learned_addr learned my addr 172.21.15.1:0/2720332205 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:49.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d863c6640 1 --2- 172.21.15.1:0/2720332205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d800a5fa0 0x7f7d801eeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d85bc5640 1 --2- 172.21.15.1:0/2720332205 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d800c2b30 0x7f7d8013d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d86bc7640 1 -- 172.21.15.1:0/2720332205 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d800c2b30 msgr2=0x7f7d8013d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d86bc7640 1 --2- 172.21.15.1:0/2720332205 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d800c2b30 0x7f7d8013d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d86bc7640 1 -- 172.21.15.1:0/2720332205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d800a5fa0 msgr2=0x7f7d801eeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d86bc7640 1 --2- 172.21.15.1:0/2720332205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d800a5fa0 0x7f7d801eeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d86bc7640 1 -- 172.21.15.1:0/2720332205 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d80144d80 con 0x7f7d80152b70 2024-03-06T07:29:49.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d86bc7640 1 --2- 172.21.15.1:0/2720332205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d80152b70 0x7f7d8014e720 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f7d78001eb0 tx=0x7f7d7800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d72ffd640 1 -- 172.21.15.1:0/2720332205 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d78017070 con 0x7f7d80152b70 2024-03-06T07:29:49.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d72ffd640 1 -- 172.21.15.1:0/2720332205 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d780052d0 con 0x7f7d80152b70 2024-03-06T07:29:49.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d8862a640 1 -- 172.21.15.1:0/2720332205 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d80145060 con 0x7f7d80152b70 2024-03-06T07:29:49.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d72ffd640 1 -- 172.21.15.1:0/2720332205 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d78003810 con 0x7f7d80152b70 2024-03-06T07:29:49.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.711+0000 7f7d8862a640 1 -- 172.21.15.1:0/2720332205 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d800acaf0 con 0x7f7d80152b70 2024-03-06T07:29:49.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.712+0000 7f7d8862a640 1 -- 172.21.15.1:0/2720332205 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d80067fd0 con 0x7f7d80152b70 2024-03-06T07:29:49.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.713+0000 7f7d72ffd640 1 -- 172.21.15.1:0/2720332205 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d7801a070 con 0x7f7d80152b70 2024-03-06T07:29:49.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.713+0000 7f7d72ffd640 1 --2- 172.21.15.1:0/2720332205 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d6406b500 0x7f7d6406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.713+0000 7f7d72ffd640 1 -- 172.21.15.1:0/2720332205 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7d780895b0 con 0x7f7d80152b70 2024-03-06T07:29:49.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.714+0000 7f7d863c6640 1 --2- 172.21.15.1:0/2720332205 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d6406b500 0x7f7d6406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.714+0000 7f7d863c6640 1 --2- 172.21.15.1:0/2720332205 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d6406b500 0x7f7d6406d9c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c035000 tx=0x7f7d7c030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.722+0000 7f7d72ffd640 1 -- 172.21.15.1:0/2720332205 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d78057c50 con 0x7f7d80152b70 2024-03-06T07:29:49.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.942+0000 7f7d8862a640 1 -- 172.21.15.1:0/2720332205 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d8006f460 con 0x7f7d80152b70 2024-03-06T07:29:49.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.942+0000 7f7d72ffd640 1 -- 172.21.15.1:0/2720332205 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5005 (secure 0 0 0) 0x7f7d7805bcf0 con 0x7f7d80152b70 2024-03-06T07:29:49.944 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:49.945 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:44.306981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:29:49.945 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 200 2024-03-06T07:29:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.944+0000 7f7d8862a640 1 -- 172.21.15.1:0/2720332205 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d6406b500 msgr2=0x7f7d6406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.944+0000 7f7d8862a640 1 --2- 172.21.15.1:0/2720332205 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d6406b500 0x7f7d6406d9c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c035000 tx=0x7f7d7c030000 comp rx=0 tx=0).stop 2024-03-06T07:29:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.944+0000 7f7d8862a640 1 -- 172.21.15.1:0/2720332205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d80152b70 msgr2=0x7f7d8014e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.944+0000 7f7d8862a640 1 --2- 172.21.15.1:0/2720332205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d80152b70 0x7f7d8014e720 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f7d78001eb0 tx=0x7f7d7800c890 comp rx=0 tx=0).stop 2024-03-06T07:29:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.944+0000 7f7d8862a640 1 -- 172.21.15.1:0/2720332205 shutdown_connections 2024-03-06T07:29:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.944+0000 7f7d8862a640 1 --2- 172.21.15.1:0/2720332205 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d6406b500 0x7f7d6406d9c0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.944+0000 7f7d8862a640 1 --2- 172.21.15.1:0/2720332205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d80152b70 0x7f7d8014e720 unknown :-1 s=CLOSED pgs=2376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.944+0000 7f7d8862a640 1 --2- 172.21.15.1:0/2720332205 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d800c2b30 0x7f7d8013d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.944+0000 7f7d8862a640 1 --2- 172.21.15.1:0/2720332205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d800a5fa0 0x7f7d801eeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.944+0000 7f7d8862a640 1 -- 172.21.15.1:0/2720332205 >> 172.21.15.1:0/2720332205 conn(0x7f7d800c6860 msgr2=0x7f7d8006e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.945+0000 7f7d8862a640 1 -- 172.21.15.1:0/2720332205 shutdown_connections 2024-03-06T07:29:49.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:49.945+0000 7f7d8862a640 1 -- 172.21.15.1:0/2720332205 wait complete. 2024-03-06T07:29:50.363 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:44.306981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:29:50.363 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:stopping 2024-03-06T07:29:51.365 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:51.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.464+0000 7fd5779f4640 1 Processor -- start 2024-03-06T07:29:51.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.464+0000 7fd5779f4640 1 -- start start 2024-03-06T07:29:51.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.465+0000 7fd5779f4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd570072bd0 0x7fd5700c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.465+0000 7fd5779f4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd570152b70 0x7fd5700c3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.465+0000 7fd5779f4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd570151dd0 con 0x7fd5700c2b30 2024-03-06T07:29:51.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.465+0000 7fd5779f4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5701526e0 con 0x7fd570072bd0 2024-03-06T07:29:51.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.465+0000 7fd5779f4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd570156190 con 0x7fd570152b70 2024-03-06T07:29:51.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.465+0000 7fd575790640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd5700c2b30 0x7fd5700724c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60876/0 (socket says 172.21.15.1:60876) 2024-03-06T07:29:51.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.465+0000 7fd575790640 1 -- 172.21.15.1:0/2333749221 learned_addr learned my addr 172.21.15.1:0/2333749221 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:51.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.465+0000 7fd575f91640 1 --2- 172.21.15.1:0/2333749221 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd570152b70 0x7fd5700c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.465+0000 7fd574f8f640 1 --2- 172.21.15.1:0/2333749221 >> v2:172.21.15.98:3300/0 conn(0x7fd570072bd0 0x7fd5700c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.466+0000 7fd563fff640 1 -- 172.21.15.1:0/2333749221 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924987786 0 0) 0x7fd570151dd0 con 0x7fd5700c2b30 2024-03-06T07:29:51.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.466+0000 7fd563fff640 1 -- 172.21.15.1:0/2333749221 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd55c003520 con 0x7fd5700c2b30 2024-03-06T07:29:51.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.466+0000 7fd575f91640 1 -- 172.21.15.1:0/2333749221 >> v2:172.21.15.98:3300/0 conn(0x7fd570072bd0 msgr2=0x7fd5700c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:51.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.466+0000 7fd575f91640 1 --2- 172.21.15.1:0/2333749221 >> v2:172.21.15.98:3300/0 conn(0x7fd570072bd0 0x7fd5700c32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.466+0000 7fd575f91640 1 -- 172.21.15.1:0/2333749221 >> v1:172.21.15.1:6789/0 conn(0x7fd5700c2b30 legacy=0x7fd5700724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.466+0000 7fd575f91640 1 -- 172.21.15.1:0/2333749221 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd570152090 con 0x7fd570152b70 2024-03-06T07:29:51.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.467+0000 7fd575f91640 1 --2- 172.21.15.1:0/2333749221 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd570152b70 0x7fd5700c3830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd570152860 tx=0x7fd56802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79b8424af9314fc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.467+0000 7fd563fff640 1 -- 172.21.15.1:0/2333749221 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568039070 con 0x7fd570152b70 2024-03-06T07:29:51.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.467+0000 7fd563fff640 1 -- 172.21.15.1:0/2333749221 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56802cc40 con 0x7fd570152b70 2024-03-06T07:29:51.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.467+0000 7fd563fff640 1 -- 172.21.15.1:0/2333749221 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568034440 con 0x7fd570152b70 2024-03-06T07:29:51.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.467+0000 7fd5779f4640 1 -- 172.21.15.1:0/2333749221 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd570152b70 msgr2=0x7fd5700c3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.467+0000 7fd5779f4640 1 --2- 172.21.15.1:0/2333749221 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd570152b70 0x7fd5700c3830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd570152860 tx=0x7fd56802be90 comp rx=0 tx=0).stop 2024-03-06T07:29:51.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.467+0000 7fd5779f4640 1 -- 172.21.15.1:0/2333749221 shutdown_connections 2024-03-06T07:29:51.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.467+0000 7fd5779f4640 1 --2- 172.21.15.1:0/2333749221 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd570152b70 0x7fd5700c3830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.467+0000 7fd5779f4640 1 --2- 172.21.15.1:0/2333749221 >> v2:172.21.15.98:3300/0 conn(0x7fd570072bd0 0x7fd5700c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.467+0000 7fd5779f4640 1 -- 172.21.15.1:0/2333749221 >> 172.21.15.1:0/2333749221 conn(0x7fd5700c6860 msgr2=0x7fd5700c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.467+0000 7fd5779f4640 1 -- 172.21.15.1:0/2333749221 shutdown_connections 2024-03-06T07:29:51.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.467+0000 7fd5779f4640 1 -- 172.21.15.1:0/2333749221 wait complete. 2024-03-06T07:29:51.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.468+0000 7fd5779f4640 1 Processor -- start 2024-03-06T07:29:51.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.468+0000 7fd5779f4640 1 -- start start 2024-03-06T07:29:51.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.469+0000 7fd5779f4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd570072bd0 0x7fd57014baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.469+0000 7fd5779f4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5700c2b30 0x7fd57014bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.469+0000 7fd5779f4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd570152b70 0x7fd5701465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.469+0000 7fd5779f4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5700c3ef0 con 0x7fd570152b70 2024-03-06T07:29:51.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.469+0000 7fd5779f4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5700c3d70 con 0x7fd5700c2b30 2024-03-06T07:29:51.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.469+0000 7fd5779f4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd570155d80 con 0x7fd570072bd0 2024-03-06T07:29:51.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.469+0000 7fd575f91640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd570152b70 0x7fd5701465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.469+0000 7fd575f91640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd570152b70 0x7fd5701465c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59722/0 (socket says 172.21.15.1:59722) 2024-03-06T07:29:51.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.469+0000 7fd575f91640 1 -- 172.21.15.1:0/3317652130 learned_addr learned my addr 172.21.15.1:0/3317652130 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:51.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.469+0000 7fd575790640 1 --2- 172.21.15.1:0/3317652130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd570072bd0 0x7fd57014baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.469+0000 7fd574f8f640 1 --2- 172.21.15.1:0/3317652130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5700c2b30 0x7fd57014bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.470+0000 7fd575f91640 1 -- 172.21.15.1:0/3317652130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd570072bd0 msgr2=0x7fd57014baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.470+0000 7fd575f91640 1 --2- 172.21.15.1:0/3317652130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd570072bd0 0x7fd57014baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.470+0000 7fd575f91640 1 -- 172.21.15.1:0/3317652130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5700c2b30 msgr2=0x7fd57014bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.470+0000 7fd575f91640 1 --2- 172.21.15.1:0/3317652130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5700c2b30 0x7fd57014bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.470+0000 7fd575f91640 1 -- 172.21.15.1:0/3317652130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5701db610 con 0x7fd570152b70 2024-03-06T07:29:51.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.470+0000 7fd574f8f640 1 --2- 172.21.15.1:0/3317652130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5700c2b30 0x7fd57014bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:51.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.470+0000 7fd575f91640 1 --2- 172.21.15.1:0/3317652130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd570152b70 0x7fd5701465c0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fd568009e80 tx=0x7fd568004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.470+0000 7fd561ffb640 1 -- 172.21.15.1:0/3317652130 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568039050 con 0x7fd570152b70 2024-03-06T07:29:51.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.470+0000 7fd561ffb640 1 -- 172.21.15.1:0/3317652130 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd568045070 con 0x7fd570152b70 2024-03-06T07:29:51.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.470+0000 7fd5779f4640 1 -- 172.21.15.1:0/3317652130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5701dc360 con 0x7fd570152b70 2024-03-06T07:29:51.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.470+0000 7fd561ffb640 1 -- 172.21.15.1:0/3317652130 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568009450 con 0x7fd570152b70 2024-03-06T07:29:51.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.470+0000 7fd5779f4640 1 -- 172.21.15.1:0/3317652130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5701dc920 con 0x7fd570152b70 2024-03-06T07:29:51.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.471+0000 7fd5779f4640 1 -- 172.21.15.1:0/3317652130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd570067fd0 con 0x7fd570152b70 2024-03-06T07:29:51.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.472+0000 7fd561ffb640 1 -- 172.21.15.1:0/3317652130 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd568040670 con 0x7fd570152b70 2024-03-06T07:29:51.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.472+0000 7fd561ffb640 1 --2- 172.21.15.1:0/3317652130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd55406b4d0 0x7fd55406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.472+0000 7fd561ffb640 1 -- 172.21.15.1:0/3317652130 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd5680acdd0 con 0x7fd570152b70 2024-03-06T07:29:51.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.472+0000 7fd575790640 1 --2- 172.21.15.1:0/3317652130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd55406b4d0 0x7fd55406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.473+0000 7fd575790640 1 --2- 172.21.15.1:0/3317652130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd55406b4d0 0x7fd55406d990 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fd564005880 tx=0x7fd564009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.477+0000 7fd561ffb640 1 -- 172.21.15.1:0/3317652130 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd56807bac0 con 0x7fd570152b70 2024-03-06T07:29:51.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.699+0000 7fd5779f4640 1 -- 172.21.15.1:0/3317652130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd570071790 con 0x7fd570152b70 2024-03-06T07:29:51.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.699+0000 7fd561ffb640 1 -- 172.21.15.1:0/3317652130 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5005 (secure 0 0 0) 0x7fd5680ae090 con 0x7fd570152b70 2024-03-06T07:29:51.703 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:51.703 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:44.306981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:29:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 200 2024-03-06T07:29:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.702+0000 7fd5779f4640 1 -- 172.21.15.1:0/3317652130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd55406b4d0 msgr2=0x7fd55406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.702+0000 7fd5779f4640 1 --2- 172.21.15.1:0/3317652130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd55406b4d0 0x7fd55406d990 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fd564005880 tx=0x7fd564009040 comp rx=0 tx=0).stop 2024-03-06T07:29:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.702+0000 7fd5779f4640 1 -- 172.21.15.1:0/3317652130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd570152b70 msgr2=0x7fd5701465c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.702+0000 7fd5779f4640 1 --2- 172.21.15.1:0/3317652130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd570152b70 0x7fd5701465c0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fd568009e80 tx=0x7fd568004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.702+0000 7fd5779f4640 1 -- 172.21.15.1:0/3317652130 shutdown_connections 2024-03-06T07:29:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.702+0000 7fd5779f4640 1 --2- 172.21.15.1:0/3317652130 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd55406b4d0 0x7fd55406d990 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.702+0000 7fd5779f4640 1 --2- 172.21.15.1:0/3317652130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd570152b70 0x7fd5701465c0 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.702+0000 7fd5779f4640 1 --2- 172.21.15.1:0/3317652130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5700c2b30 0x7fd57014bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.702+0000 7fd5779f4640 1 --2- 172.21.15.1:0/3317652130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd570072bd0 0x7fd57014baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.702+0000 7fd5779f4640 1 -- 172.21.15.1:0/3317652130 >> 172.21.15.1:0/3317652130 conn(0x7fd5700c6860 msgr2=0x7fd5700c3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.702+0000 7fd5779f4640 1 -- 172.21.15.1:0/3317652130 shutdown_connections 2024-03-06T07:29:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:51.702+0000 7fd5779f4640 1 -- 172.21.15.1:0/3317652130 wait complete. 2024-03-06T07:29:52.117 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:44.306981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:29:52.117 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:stopping 2024-03-06T07:29:53.118 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:53.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.218+0000 7f8709881640 1 Processor -- start 2024-03-06T07:29:53.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.218+0000 7f8709881640 1 -- start start 2024-03-06T07:29:53.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.218+0000 7f8709881640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8704152650 0x7f8704155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.218+0000 7f8709881640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87040c6b10 0x7f8704158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.218+0000 7f8709881640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87040c2fc0 con 0x7f87040bf9d0 2024-03-06T07:29:53.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.218+0000 7f8709881640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8704155800 con 0x7f8704152650 2024-03-06T07:29:53.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.218+0000 7f8709881640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87040c6690 con 0x7f87040c6b10 2024-03-06T07:29:53.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.219+0000 7f87027fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87040c6b10 0x7f8704158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.219+0000 7f8702ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8704152650 0x7f8704155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.219+0000 7f87037fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f87040bf9d0 0x7f87040c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:35236/0 (socket says 172.21.15.1:35236) 2024-03-06T07:29:53.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.219+0000 7f87027fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87040c6b10 0x7f8704158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:37672/0 (socket says 172.21.15.1:37672) 2024-03-06T07:29:53.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.219+0000 7f87037fe640 1 -- 172.21.15.1:0/949912760 learned_addr learned my addr 172.21.15.1:0/949912760 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:53.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.219+0000 7f8701ffb640 1 -- 172.21.15.1:0/949912760 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938726471 0 0) 0x7f87040c2fc0 con 0x7f87040bf9d0 2024-03-06T07:29:53.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.219+0000 7f8701ffb640 1 -- 172.21.15.1:0/949912760 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f0003520 con 0x7f87040bf9d0 2024-03-06T07:29:53.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.219+0000 7f87027fc640 1 -- 172.21.15.1:0/949912760 >> v2:172.21.15.98:3300/0 conn(0x7f8704152650 msgr2=0x7f8704155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.219+0000 7f87027fc640 1 --2- 172.21.15.1:0/949912760 >> v2:172.21.15.98:3300/0 conn(0x7f8704152650 0x7f8704155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.219+0000 7f87027fc640 1 -- 172.21.15.1:0/949912760 >> v1:172.21.15.98:6789/0 conn(0x7f87040bf9d0 legacy=0x7f87040c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.219+0000 7f87027fc640 1 -- 172.21.15.1:0/949912760 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87040c7280 con 0x7f87040c6b10 2024-03-06T07:29:53.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.219+0000 7f8702ffd640 1 --2- 172.21.15.1:0/949912760 >> v2:172.21.15.98:3300/0 conn(0x7f8704152650 0x7f8704155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:53.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.220+0000 7f87027fc640 1 --2- 172.21.15.1:0/949912760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87040c6b10 0x7f8704158690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f86ec0097a0 tx=0x7f86ec02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3101844871220c06 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.220+0000 7f8701ffb640 1 -- 172.21.15.1:0/949912760 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec031780 con 0x7f87040c6b10 2024-03-06T07:29:53.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.220+0000 7f8701ffb640 1 -- 172.21.15.1:0/949912760 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86ec037070 con 0x7f87040c6b10 2024-03-06T07:29:53.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.220+0000 7f8701ffb640 1 -- 172.21.15.1:0/949912760 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec031b20 con 0x7f87040c6b10 2024-03-06T07:29:53.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.221+0000 7f8709881640 1 -- 172.21.15.1:0/949912760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87040c6b10 msgr2=0x7f8704158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.221+0000 7f8709881640 1 --2- 172.21.15.1:0/949912760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87040c6b10 0x7f8704158690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f86ec0097a0 tx=0x7f86ec02b820 comp rx=0 tx=0).stop 2024-03-06T07:29:53.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.221+0000 7f8709881640 1 -- 172.21.15.1:0/949912760 shutdown_connections 2024-03-06T07:29:53.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.221+0000 7f8709881640 1 --2- 172.21.15.1:0/949912760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f87040c6b10 0x7f8704158690 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.221+0000 7f8709881640 1 --2- 172.21.15.1:0/949912760 >> v2:172.21.15.98:3300/0 conn(0x7f8704152650 0x7f8704155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.221+0000 7f8709881640 1 -- 172.21.15.1:0/949912760 >> 172.21.15.1:0/949912760 conn(0x7f8704066ee0 msgr2=0x7f87040672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.221+0000 7f8709881640 1 -- 172.21.15.1:0/949912760 shutdown_connections 2024-03-06T07:29:53.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.221+0000 7f8709881640 1 -- 172.21.15.1:0/949912760 wait complete. 2024-03-06T07:29:53.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.223+0000 7f8709881640 1 Processor -- start 2024-03-06T07:29:53.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.223+0000 7f8709881640 1 -- start start 2024-03-06T07:29:53.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.224+0000 7f8709881640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87040bf9d0 0x7f87041eb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.224+0000 7f8709881640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8704152650 0x7f87041eec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.224+0000 7f8709881640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8704151860 0x7f8704157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.224+0000 7f8709881640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8704155490 con 0x7f8704152650 2024-03-06T07:29:53.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.224+0000 7f8709881640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87041551b0 con 0x7f87040bf9d0 2024-03-06T07:29:53.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.224+0000 7f8709881640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8704152a30 con 0x7f8704151860 2024-03-06T07:29:53.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.224+0000 7f87027fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8704152650 0x7f87041eec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.224+0000 7f87027fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8704152650 0x7f87041eec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59734/0 (socket says 172.21.15.1:59734) 2024-03-06T07:29:53.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.224+0000 7f87027fc640 1 -- 172.21.15.1:0/3218001589 learned_addr learned my addr 172.21.15.1:0/3218001589 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:53.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.224+0000 7f87037fe640 1 --2- 172.21.15.1:0/3218001589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8704151860 0x7f8704157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.224+0000 7f8702ffd640 1 --2- 172.21.15.1:0/3218001589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87040bf9d0 0x7f87041eb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.224+0000 7f87027fc640 1 -- 172.21.15.1:0/3218001589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8704151860 msgr2=0x7f8704157740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.224+0000 7f87027fc640 1 --2- 172.21.15.1:0/3218001589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8704151860 0x7f8704157740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.224+0000 7f87027fc640 1 -- 172.21.15.1:0/3218001589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87040bf9d0 msgr2=0x7f87041eb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.224+0000 7f87027fc640 1 --2- 172.21.15.1:0/3218001589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87040bf9d0 0x7f87041eb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.224+0000 7f87027fc640 1 -- 172.21.15.1:0/3218001589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87040c4ae0 con 0x7f8704152650 2024-03-06T07:29:53.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.224+0000 7f8702ffd640 1 --2- 172.21.15.1:0/3218001589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87040bf9d0 0x7f87041eb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:53.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.225+0000 7f87027fc640 1 --2- 172.21.15.1:0/3218001589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8704152650 0x7f87041eec40 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7f86ec02b860 tx=0x7f86ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.225+0000 7f86fbfff640 1 -- 172.21.15.1:0/3218001589 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec032610 con 0x7f8704152650 2024-03-06T07:29:53.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.225+0000 7f86fbfff640 1 -- 172.21.15.1:0/3218001589 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86ec0327d0 con 0x7f8704152650 2024-03-06T07:29:53.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.225+0000 7f86fbfff640 1 -- 172.21.15.1:0/3218001589 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec032bc0 con 0x7f8704152650 2024-03-06T07:29:53.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.225+0000 7f8709881640 1 -- 172.21.15.1:0/3218001589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87041eacc0 con 0x7f8704152650 2024-03-06T07:29:53.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.225+0000 7f8709881640 1 -- 172.21.15.1:0/3218001589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f870414a8e0 con 0x7f8704152650 2024-03-06T07:29:53.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.226+0000 7f8709881640 1 -- 172.21.15.1:0/3218001589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87040b03c0 con 0x7f8704152650 2024-03-06T07:29:53.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.226+0000 7f86fbfff640 1 -- 172.21.15.1:0/3218001589 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86ec04c060 con 0x7f8704152650 2024-03-06T07:29:53.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.227+0000 7f86fbfff640 1 --2- 172.21.15.1:0/3218001589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86e406b4d0 0x7f86e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.227+0000 7f86fbfff640 1 -- 172.21.15.1:0/3218001589 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f86ec0ab7d0 con 0x7f8704152650 2024-03-06T07:29:53.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.227+0000 7f8702ffd640 1 --2- 172.21.15.1:0/3218001589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86e406b4d0 0x7f86e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.228+0000 7f8702ffd640 1 --2- 172.21.15.1:0/3218001589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86e406b4d0 0x7f86e406d990 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009ac0 tx=0x7f86fc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.231+0000 7f86fbfff640 1 -- 172.21.15.1:0/3218001589 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87040b03c0 con 0x7f8704152650 2024-03-06T07:29:53.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.452+0000 7f8709881640 1 -- 172.21.15.1:0/3218001589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8704067fd0 con 0x7f8704152650 2024-03-06T07:29:53.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.453+0000 7f86fbfff640 1 -- 172.21.15.1:0/3218001589 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5005 (secure 0 0 0) 0x7f86ec040070 con 0x7f8704152650 2024-03-06T07:29:53.455 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:53.455 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:44.306981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:29:53.456 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 200 2024-03-06T07:29:53.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.455+0000 7f8709881640 1 -- 172.21.15.1:0/3218001589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86e406b4d0 msgr2=0x7f86e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.455+0000 7f8709881640 1 --2- 172.21.15.1:0/3218001589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86e406b4d0 0x7f86e406d990 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009ac0 tx=0x7f86fc008040 comp rx=0 tx=0).stop 2024-03-06T07:29:53.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.455+0000 7f8709881640 1 -- 172.21.15.1:0/3218001589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8704152650 msgr2=0x7f87041eec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.455+0000 7f8709881640 1 --2- 172.21.15.1:0/3218001589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8704152650 0x7f87041eec40 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7f86ec02b860 tx=0x7f86ec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.455+0000 7f8709881640 1 -- 172.21.15.1:0/3218001589 shutdown_connections 2024-03-06T07:29:53.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.455+0000 7f8709881640 1 --2- 172.21.15.1:0/3218001589 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f86e406b4d0 0x7f86e406d990 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.455+0000 7f8709881640 1 --2- 172.21.15.1:0/3218001589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8704151860 0x7f8704157740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.455+0000 7f8709881640 1 --2- 172.21.15.1:0/3218001589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8704152650 0x7f87041eec40 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.455+0000 7f8709881640 1 --2- 172.21.15.1:0/3218001589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87040bf9d0 0x7f87041eb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.455+0000 7f8709881640 1 -- 172.21.15.1:0/3218001589 >> 172.21.15.1:0/3218001589 conn(0x7f8704066ee0 msgr2=0x7f87040c5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.455+0000 7f8709881640 1 -- 172.21.15.1:0/3218001589 shutdown_connections 2024-03-06T07:29:53.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:53.455+0000 7f8709881640 1 -- 172.21.15.1:0/3218001589 wait complete. 2024-03-06T07:29:53.870 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:44.306981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:29:53.870 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:stopping 2024-03-06T07:29:54.871 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:54.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.971+0000 7f038f53e640 1 Processor -- start 2024-03-06T07:29:54.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.971+0000 7f038f53e640 1 -- start start 2024-03-06T07:29:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.972+0000 7f038f53e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0388152650 0x7f0388155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.972+0000 7f038f53e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03880c6b10 0x7f0388158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.972+0000 7f038f53e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03880bf9d0 0x7f03880c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.972+0000 7f038f53e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0388155800 con 0x7f03880c6b10 2024-03-06T07:29:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.972+0000 7f038f53e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03880c2fc0 con 0x7f0388152650 2024-03-06T07:29:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.972+0000 7f038f53e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03880c6690 con 0x7f03880bf9d0 2024-03-06T07:29:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.972+0000 7f038cad9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03880c6b10 0x7f0388158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.972+0000 7f038cad9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03880c6b10 0x7f0388158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55076/0 (socket says 172.21.15.1:55076) 2024-03-06T07:29:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.972+0000 7f038cad9640 1 -- 172.21.15.1:0/2253586248 learned_addr learned my addr 172.21.15.1:0/2253586248 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.972+0000 7f038dadb640 1 --2- 172.21.15.1:0/2253586248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03880bf9d0 0x7f03880c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.972+0000 7f038d2da640 1 --2- 172.21.15.1:0/2253586248 >> v2:172.21.15.98:3300/0 conn(0x7f0388152650 0x7f0388155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.972+0000 7f038cad9640 1 -- 172.21.15.1:0/2253586248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03880bf9d0 msgr2=0x7f03880c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.972+0000 7f038cad9640 1 --2- 172.21.15.1:0/2253586248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03880bf9d0 0x7f03880c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.973+0000 7f038cad9640 1 -- 172.21.15.1:0/2253586248 >> v2:172.21.15.98:3300/0 conn(0x7f0388152650 msgr2=0x7f0388155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.973+0000 7f038cad9640 1 --2- 172.21.15.1:0/2253586248 >> v2:172.21.15.98:3300/0 conn(0x7f0388152650 0x7f0388155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.973+0000 7f038cad9640 1 -- 172.21.15.1:0/2253586248 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03880c7280 con 0x7f03880c6b10 2024-03-06T07:29:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.973+0000 7f038dadb640 1 --2- 172.21.15.1:0/2253586248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03880bf9d0 0x7f03880c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.973+0000 7f038cad9640 1 --2- 172.21.15.1:0/2253586248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03880c6b10 0x7f0388158690 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7f03880cac90 tx=0x7f037c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fabb576e57fb1b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.973+0000 7f037bfff640 1 -- 172.21.15.1:0/2253586248 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c02e070 con 0x7f03880c6b10 2024-03-06T07:29:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.973+0000 7f037bfff640 1 -- 172.21.15.1:0/2253586248 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037c037070 con 0x7f03880c6b10 2024-03-06T07:29:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.974+0000 7f037bfff640 1 -- 172.21.15.1:0/2253586248 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c031a00 con 0x7f03880c6b10 2024-03-06T07:29:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.974+0000 7f038f53e640 1 -- 172.21.15.1:0/2253586248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03880c6b10 msgr2=0x7f0388158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.974+0000 7f038f53e640 1 --2- 172.21.15.1:0/2253586248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03880c6b10 0x7f0388158690 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7f03880cac90 tx=0x7f037c02b880 comp rx=0 tx=0).stop 2024-03-06T07:29:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.974+0000 7f038f53e640 1 -- 172.21.15.1:0/2253586248 shutdown_connections 2024-03-06T07:29:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.974+0000 7f038f53e640 1 --2- 172.21.15.1:0/2253586248 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03880bf9d0 0x7f03880c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.974+0000 7f038f53e640 1 --2- 172.21.15.1:0/2253586248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03880c6b10 0x7f0388158690 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.974+0000 7f038f53e640 1 --2- 172.21.15.1:0/2253586248 >> v2:172.21.15.98:3300/0 conn(0x7f0388152650 0x7f0388155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.974+0000 7f038f53e640 1 -- 172.21.15.1:0/2253586248 >> 172.21.15.1:0/2253586248 conn(0x7f0388066ee0 msgr2=0x7f03880672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.974+0000 7f038f53e640 1 -- 172.21.15.1:0/2253586248 shutdown_connections 2024-03-06T07:29:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.974+0000 7f038f53e640 1 -- 172.21.15.1:0/2253586248 wait complete. 2024-03-06T07:29:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.976+0000 7f038f53e640 1 Processor -- start 2024-03-06T07:29:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.976+0000 7f038f53e640 1 -- start start 2024-03-06T07:29:54.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.976+0000 7f038f53e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880bf9d0 0x7f038814e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.976+0000 7f038f53e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0388152650 0x7f03881ee6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f038f53e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0388144200 0x7f03881eec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f038f53e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0388155490 con 0x7f0388152650 2024-03-06T07:29:54.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f038f53e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03881551b0 con 0x7f03880bf9d0 2024-03-06T07:29:54.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f038f53e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0388152a30 con 0x7f0388144200 2024-03-06T07:29:54.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f038cad9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0388152650 0x7f03881ee6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f038cad9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0388152650 0x7f03881ee6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55080/0 (socket says 172.21.15.1:55080) 2024-03-06T07:29:54.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f038cad9640 1 -- 172.21.15.1:0/3752770815 learned_addr learned my addr 172.21.15.1:0/3752770815 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:54.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f038dadb640 1 --2- 172.21.15.1:0/3752770815 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0388144200 0x7f03881eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f038d2da640 1 --2- 172.21.15.1:0/3752770815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880bf9d0 0x7f038814e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f038cad9640 1 -- 172.21.15.1:0/3752770815 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0388144200 msgr2=0x7f03881eec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f038cad9640 1 --2- 172.21.15.1:0/3752770815 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0388144200 0x7f03881eec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f038cad9640 1 -- 172.21.15.1:0/3752770815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880bf9d0 msgr2=0x7f038814e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f038cad9640 1 --2- 172.21.15.1:0/3752770815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880bf9d0 0x7f038814e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f038cad9640 1 -- 172.21.15.1:0/3752770815 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0388142870 con 0x7f0388152650 2024-03-06T07:29:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f038cad9640 1 --2- 172.21.15.1:0/3752770815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0388152650 0x7f03881ee6d0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f037c02b8c0 tx=0x7f037c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f0379ffb640 1 -- 172.21.15.1:0/3752770815 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c032670 con 0x7f0388152650 2024-03-06T07:29:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f0379ffb640 1 -- 172.21.15.1:0/3752770815 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037c032830 con 0x7f0388152650 2024-03-06T07:29:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f038f53e640 1 -- 172.21.15.1:0/3752770815 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03881449c0 con 0x7f0388152650 2024-03-06T07:29:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.977+0000 7f0379ffb640 1 -- 172.21.15.1:0/3752770815 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c032c20 con 0x7f0388152650 2024-03-06T07:29:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.978+0000 7f038f53e640 1 -- 172.21.15.1:0/3752770815 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f038814eca0 con 0x7f0388152650 2024-03-06T07:29:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.979+0000 7f038f53e640 1 -- 172.21.15.1:0/3752770815 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0388067fd0 con 0x7f0388152650 2024-03-06T07:29:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.979+0000 7f0379ffb640 1 -- 172.21.15.1:0/3752770815 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f037c04c060 con 0x7f0388152650 2024-03-06T07:29:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.979+0000 7f0379ffb640 1 --2- 172.21.15.1:0/3752770815 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f036c06b4d0 0x7f036c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.979+0000 7f0379ffb640 1 -- 172.21.15.1:0/3752770815 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f037c0abce0 con 0x7f0388152650 2024-03-06T07:29:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.980+0000 7f038d2da640 1 --2- 172.21.15.1:0/3752770815 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f036c06b4d0 0x7f036c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.980+0000 7f038d2da640 1 --2- 172.21.15.1:0/3752770815 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f036c06b4d0 0x7f036c06d990 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f03880c3f50 tx=0x7f0384008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:54.984+0000 7f0379ffb640 1 -- 172.21.15.1:0/3752770815 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f037c040070 con 0x7f0388152650 2024-03-06T07:29:55.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:55.205+0000 7f038f53e640 1 -- 172.21.15.1:0/3752770815 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03880c4300 con 0x7f0388152650 2024-03-06T07:29:55.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:55.205+0000 7f0379ffb640 1 -- 172.21.15.1:0/3752770815 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5005 (secure 0 0 0) 0x7f037c07aa60 con 0x7f0388152650 2024-03-06T07:29:55.208 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:55.208 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:44.306981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:29:55.208 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 200 2024-03-06T07:29:55.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:55.207+0000 7f038f53e640 1 -- 172.21.15.1:0/3752770815 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f036c06b4d0 msgr2=0x7f036c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:55.207+0000 7f038f53e640 1 --2- 172.21.15.1:0/3752770815 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f036c06b4d0 0x7f036c06d990 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f03880c3f50 tx=0x7f0384008040 comp rx=0 tx=0).stop 2024-03-06T07:29:55.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:55.207+0000 7f038f53e640 1 -- 172.21.15.1:0/3752770815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0388152650 msgr2=0x7f03881ee6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:55.207+0000 7f038f53e640 1 --2- 172.21.15.1:0/3752770815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0388152650 0x7f03881ee6d0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f037c02b8c0 tx=0x7f037c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:55.207+0000 7f038f53e640 1 -- 172.21.15.1:0/3752770815 shutdown_connections 2024-03-06T07:29:55.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:55.208+0000 7f038f53e640 1 --2- 172.21.15.1:0/3752770815 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f036c06b4d0 0x7f036c06d990 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:55.208+0000 7f038f53e640 1 --2- 172.21.15.1:0/3752770815 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0388144200 0x7f03881eec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:55.208+0000 7f038f53e640 1 --2- 172.21.15.1:0/3752770815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0388152650 0x7f03881ee6d0 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:55.208+0000 7f038f53e640 1 --2- 172.21.15.1:0/3752770815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880bf9d0 0x7f038814e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:55.208+0000 7f038f53e640 1 -- 172.21.15.1:0/3752770815 >> 172.21.15.1:0/3752770815 conn(0x7f0388066ee0 msgr2=0x7f03880c0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:55.208+0000 7f038f53e640 1 -- 172.21.15.1:0/3752770815 shutdown_connections 2024-03-06T07:29:55.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:55.208+0000 7f038f53e640 1 -- 172.21.15.1:0/3752770815 wait complete. 2024-03-06T07:29:55.623 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:44.306981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:29:55.623 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:stopping 2024-03-06T07:29:56.624 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:56.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.724+0000 7fcd29765640 1 Processor -- start 2024-03-06T07:29:56.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.724+0000 7fcd29765640 1 -- start start 2024-03-06T07:29:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.725+0000 7fcd29765640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd240af6f0 0x7fcd240c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.725+0000 7fcd29765640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd240c7930 0x7fcd2415fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.725+0000 7fcd29765640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd240c7180 con 0x7fcd24152820 2024-03-06T07:29:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.725+0000 7fcd29765640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd24155460 con 0x7fcd240c7930 2024-03-06T07:29:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.725+0000 7fcd29765640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd240c6bc0 con 0x7fcd240af6f0 2024-03-06T07:29:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.725+0000 7fcd237fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd240c7930 0x7fcd2415fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.725+0000 7fcd22ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcd24152820 0x7fcd240c0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50664/0 (socket says 172.21.15.1:50664) 2024-03-06T07:29:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.725+0000 7fcd22ffd640 1 -- 172.21.15.1:0/4044715045 learned_addr learned my addr 172.21.15.1:0/4044715045 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:29:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.725+0000 7fcd1bfff640 1 --2- 172.21.15.1:0/4044715045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd240af6f0 0x7fcd240c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.725+0000 7fcd237fe640 1 -- 172.21.15.1:0/4044715045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd240af6f0 msgr2=0x7fcd240c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.725+0000 7fcd237fe640 1 --2- 172.21.15.1:0/4044715045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd240af6f0 0x7fcd240c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.725+0000 7fcd237fe640 1 -- 172.21.15.1:0/4044715045 >> v1:172.21.15.1:6789/0 conn(0x7fcd24152820 legacy=0x7fcd240c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.725+0000 7fcd227fc640 1 -- 172.21.15.1:0/4044715045 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155837169 0 0) 0x7fcd240c7180 con 0x7fcd24152820 2024-03-06T07:29:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.725+0000 7fcd237fe640 1 -- 172.21.15.1:0/4044715045 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd240c6ea0 con 0x7fcd240c7930 2024-03-06T07:29:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.725+0000 7fcd1bfff640 1 --2- 172.21.15.1:0/4044715045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd240af6f0 0x7fcd240c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:56.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.726+0000 7fcd237fe640 1 --2- 172.21.15.1:0/4044715045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd240c7930 0x7fcd2415fb60 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009d90 tx=0x7fcd1c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e1de931e30ed28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.726+0000 7fcd227fc640 1 -- 172.21.15.1:0/4044715045 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c039070 con 0x7fcd240c7930 2024-03-06T07:29:56.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.727+0000 7fcd227fc640 1 -- 172.21.15.1:0/4044715045 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1c02cc50 con 0x7fcd240c7930 2024-03-06T07:29:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.727+0000 7fcd227fc640 1 -- 172.21.15.1:0/4044715045 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c034440 con 0x7fcd240c7930 2024-03-06T07:29:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.727+0000 7fcd29765640 1 -- 172.21.15.1:0/4044715045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd240c7930 msgr2=0x7fcd2415fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.727+0000 7fcd29765640 1 --2- 172.21.15.1:0/4044715045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd240c7930 0x7fcd2415fb60 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009d90 tx=0x7fcd1c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.727+0000 7fcd29765640 1 -- 172.21.15.1:0/4044715045 shutdown_connections 2024-03-06T07:29:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.727+0000 7fcd29765640 1 --2- 172.21.15.1:0/4044715045 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd240af6f0 0x7fcd240c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.727+0000 7fcd29765640 1 --2- 172.21.15.1:0/4044715045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd240c7930 0x7fcd2415fb60 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.727+0000 7fcd29765640 1 -- 172.21.15.1:0/4044715045 >> 172.21.15.1:0/4044715045 conn(0x7fcd24072720 msgr2=0x7fcd24072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.727+0000 7fcd29765640 1 -- 172.21.15.1:0/4044715045 shutdown_connections 2024-03-06T07:29:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.727+0000 7fcd29765640 1 -- 172.21.15.1:0/4044715045 wait complete. 2024-03-06T07:29:56.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.728+0000 7fcd29765640 1 Processor -- start 2024-03-06T07:29:56.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.728+0000 7fcd29765640 1 -- start start 2024-03-06T07:29:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.729+0000 7fcd29765640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd240af6f0 0x7fcd241eb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.729+0000 7fcd29765640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd240c7930 0x7fcd241ee9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.729+0000 7fcd29765640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd24152820 0x7fcd241ebfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.729+0000 7fcd29765640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd24152c00 con 0x7fcd240af6f0 2024-03-06T07:29:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.729+0000 7fcd29765640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd240c0840 con 0x7fcd240c7930 2024-03-06T07:29:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.729+0000 7fcd29765640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd24152d80 con 0x7fcd24152820 2024-03-06T07:29:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.729+0000 7fcd22ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd240af6f0 0x7fcd241eb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.729+0000 7fcd22ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd240af6f0 0x7fcd241eb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55100/0 (socket says 172.21.15.1:55100) 2024-03-06T07:29:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.729+0000 7fcd22ffd640 1 -- 172.21.15.1:0/972090207 learned_addr learned my addr 172.21.15.1:0/972090207 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.729+0000 7fcd237fe640 1 --2- 172.21.15.1:0/972090207 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd24152820 0x7fcd241ebfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.729+0000 7fcd1bfff640 1 --2- 172.21.15.1:0/972090207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd240c7930 0x7fcd241ee9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.729+0000 7fcd22ffd640 1 -- 172.21.15.1:0/972090207 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd24152820 msgr2=0x7fcd241ebfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.729+0000 7fcd22ffd640 1 --2- 172.21.15.1:0/972090207 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd24152820 0x7fcd241ebfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.729+0000 7fcd22ffd640 1 -- 172.21.15.1:0/972090207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd240c7930 msgr2=0x7fcd241ee9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.729+0000 7fcd22ffd640 1 --2- 172.21.15.1:0/972090207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd240c7930 0x7fcd241ee9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.729+0000 7fcd22ffd640 1 -- 172.21.15.1:0/972090207 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd241e8760 con 0x7fcd240af6f0 2024-03-06T07:29:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.729+0000 7fcd22ffd640 1 --2- 172.21.15.1:0/972090207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd240af6f0 0x7fcd241eb500 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7fcd1400c430 tx=0x7fcd1400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.730+0000 7fcd1b7fe640 1 -- 172.21.15.1:0/972090207 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd140074e0 con 0x7fcd240af6f0 2024-03-06T07:29:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.730+0000 7fcd1b7fe640 1 -- 172.21.15.1:0/972090207 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd140047e0 con 0x7fcd240af6f0 2024-03-06T07:29:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.730+0000 7fcd29765640 1 -- 172.21.15.1:0/972090207 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd241eef30 con 0x7fcd240af6f0 2024-03-06T07:29:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.730+0000 7fcd1b7fe640 1 -- 172.21.15.1:0/972090207 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd14004bd0 con 0x7fcd240af6f0 2024-03-06T07:29:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.730+0000 7fcd29765640 1 -- 172.21.15.1:0/972090207 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd24154a50 con 0x7fcd240af6f0 2024-03-06T07:29:56.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.731+0000 7fcd29765640 1 -- 172.21.15.1:0/972090207 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd240686e0 con 0x7fcd240af6f0 2024-03-06T07:29:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.731+0000 7fcd1b7fe640 1 -- 172.21.15.1:0/972090207 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd14011070 con 0x7fcd240af6f0 2024-03-06T07:29:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.732+0000 7fcd1b7fe640 1 --2- 172.21.15.1:0/972090207 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd0406b4d0 0x7fcd0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.732+0000 7fcd1b7fe640 1 -- 172.21.15.1:0/972090207 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcd14089f10 con 0x7fcd240af6f0 2024-03-06T07:29:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.732+0000 7fcd1bfff640 1 --2- 172.21.15.1:0/972090207 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd0406b4d0 0x7fcd0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.732+0000 7fcd1bfff640 1 --2- 172.21.15.1:0/972090207 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd0406b4d0 0x7fcd0406d990 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c006fd0 tx=0x7fcd0c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.736+0000 7fcd1b7fe640 1 -- 172.21.15.1:0/972090207 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd241e8760 con 0x7fcd240af6f0 2024-03-06T07:29:56.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.955+0000 7fcd29765640 1 -- 172.21.15.1:0/972090207 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd241515f0 con 0x7fcd240af6f0 2024-03-06T07:29:56.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.955+0000 7fcd1b7fe640 1 -- 172.21.15.1:0/972090207 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5005 (secure 0 0 0) 0x7fcd1400e280 con 0x7fcd240af6f0 2024-03-06T07:29:56.957 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:56.957 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:44.306981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:29:56.958 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 200 2024-03-06T07:29:56.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.957+0000 7fcd29765640 1 -- 172.21.15.1:0/972090207 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd0406b4d0 msgr2=0x7fcd0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.957+0000 7fcd29765640 1 --2- 172.21.15.1:0/972090207 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd0406b4d0 0x7fcd0406d990 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c006fd0 tx=0x7fcd0c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:56.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.957+0000 7fcd29765640 1 -- 172.21.15.1:0/972090207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd240af6f0 msgr2=0x7fcd241eb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.957+0000 7fcd29765640 1 --2- 172.21.15.1:0/972090207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd240af6f0 0x7fcd241eb500 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7fcd1400c430 tx=0x7fcd1400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.957+0000 7fcd29765640 1 -- 172.21.15.1:0/972090207 shutdown_connections 2024-03-06T07:29:56.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.957+0000 7fcd29765640 1 --2- 172.21.15.1:0/972090207 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd0406b4d0 0x7fcd0406d990 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.957+0000 7fcd29765640 1 --2- 172.21.15.1:0/972090207 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd24152820 0x7fcd241ebfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.957+0000 7fcd29765640 1 --2- 172.21.15.1:0/972090207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd240c7930 0x7fcd241ee9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.957+0000 7fcd29765640 1 --2- 172.21.15.1:0/972090207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd240af6f0 0x7fcd241eb500 unknown :-1 s=CLOSED pgs=2384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.957+0000 7fcd29765640 1 -- 172.21.15.1:0/972090207 >> 172.21.15.1:0/972090207 conn(0x7fcd24072720 msgr2=0x7fcd24151830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.957+0000 7fcd29765640 1 -- 172.21.15.1:0/972090207 shutdown_connections 2024-03-06T07:29:56.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:56.957+0000 7fcd29765640 1 -- 172.21.15.1:0/972090207 wait complete. 2024-03-06T07:29:57.375 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:44.306981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:29:57.375 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:stopping 2024-03-06T07:29:58.376 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.475+0000 7f739fa88640 1 Processor -- start 2024-03-06T07:29:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.476+0000 7f739fa88640 1 -- start start 2024-03-06T07:29:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.477+0000 7f739fa88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7398152650 0x7f7398155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.477+0000 7f739fa88640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f73980c6b10 0x7f7398158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.477+0000 7f739fa88640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f73980bf9d0 0x7f73980c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.477+0000 7f739fa88640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7398155800 con 0x7f73980bf9d0 2024-03-06T07:29:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.477+0000 7f739fa88640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73980c2fc0 con 0x7f73980c6b10 2024-03-06T07:29:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.477+0000 7f739fa88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73980c6690 con 0x7f7398152650 2024-03-06T07:29:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.477+0000 7f739e025640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f73980bf9d0 0x7f73980c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.477+0000 7f739e025640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f73980bf9d0 0x7f73980c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55104/0 (socket says 172.21.15.1:55104) 2024-03-06T07:29:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.477+0000 7f739e025640 1 -- 172.21.15.1:0/507254150 learned_addr learned my addr 172.21.15.1:0/507254150 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.477+0000 7f739d824640 1 --2- 172.21.15.1:0/507254150 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7398152650 0x7f7398155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.477+0000 7f739d023640 1 --2- 172.21.15.1:0/507254150 >> v2:172.21.15.98:3300/0 conn(0x7f73980c6b10 0x7f7398158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.477+0000 7f739e025640 1 -- 172.21.15.1:0/507254150 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7398152650 msgr2=0x7f7398155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.477+0000 7f739e025640 1 --2- 172.21.15.1:0/507254150 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7398152650 0x7f7398155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.477+0000 7f739e025640 1 -- 172.21.15.1:0/507254150 >> v2:172.21.15.98:3300/0 conn(0x7f73980c6b10 msgr2=0x7f7398158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.477+0000 7f739e025640 1 --2- 172.21.15.1:0/507254150 >> v2:172.21.15.98:3300/0 conn(0x7f73980c6b10 0x7f7398158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.477+0000 7f739e025640 1 -- 172.21.15.1:0/507254150 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73980c7280 con 0x7f73980bf9d0 2024-03-06T07:29:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.479+0000 7f739e025640 1 --2- 172.21.15.1:0/507254150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73980bf9d0 0x7f73980c1460 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7f739808c500 tx=0x7f739002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0883d0c5ea21f72 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.479+0000 7f739c822640 1 -- 172.21.15.1:0/507254150 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739002c9e0 con 0x7f73980bf9d0 2024-03-06T07:29:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.479+0000 7f739c822640 1 -- 172.21.15.1:0/507254150 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739002cb80 con 0x7f73980bf9d0 2024-03-06T07:29:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.479+0000 7f739c822640 1 -- 172.21.15.1:0/507254150 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73900343f0 con 0x7f73980bf9d0 2024-03-06T07:29:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.479+0000 7f739fa88640 1 -- 172.21.15.1:0/507254150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73980bf9d0 msgr2=0x7f73980c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.479+0000 7f739fa88640 1 --2- 172.21.15.1:0/507254150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73980bf9d0 0x7f73980c1460 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7f739808c500 tx=0x7f739002be40 comp rx=0 tx=0).stop 2024-03-06T07:29:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.479+0000 7f739fa88640 1 -- 172.21.15.1:0/507254150 shutdown_connections 2024-03-06T07:29:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.479+0000 7f739fa88640 1 --2- 172.21.15.1:0/507254150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73980bf9d0 0x7f73980c1460 unknown :-1 s=CLOSED pgs=2385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.479+0000 7f739fa88640 1 --2- 172.21.15.1:0/507254150 >> v2:172.21.15.98:3300/0 conn(0x7f73980c6b10 0x7f7398158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.479+0000 7f739fa88640 1 --2- 172.21.15.1:0/507254150 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7398152650 0x7f7398155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.479+0000 7f739fa88640 1 -- 172.21.15.1:0/507254150 >> 172.21.15.1:0/507254150 conn(0x7f7398066ee0 msgr2=0x7f73980672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.479+0000 7f739fa88640 1 -- 172.21.15.1:0/507254150 shutdown_connections 2024-03-06T07:29:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.479+0000 7f739fa88640 1 -- 172.21.15.1:0/507254150 wait complete. 2024-03-06T07:29:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.480+0000 7f739fa88640 1 Processor -- start 2024-03-06T07:29:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.480+0000 7f739fa88640 1 -- start start 2024-03-06T07:29:58.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.481+0000 7f739fa88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73980bf9d0 0x7f73981473c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.481+0000 7f739fa88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7398152650 0x7f7398145060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.481+0000 7f739fa88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73980c4200 0x7f73981455a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.481+0000 7f739fa88640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7398155490 con 0x7f73980bf9d0 2024-03-06T07:29:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.481+0000 7f739fa88640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73981551b0 con 0x7f73980c4200 2024-03-06T07:29:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.481+0000 7f739fa88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7398152a30 con 0x7f7398152650 2024-03-06T07:29:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.481+0000 7f739d824640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73980bf9d0 0x7f73981473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.481+0000 7f739d824640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73980bf9d0 0x7f73981473c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55106/0 (socket says 172.21.15.1:55106) 2024-03-06T07:29:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.481+0000 7f739d824640 1 -- 172.21.15.1:0/2652966973 learned_addr learned my addr 172.21.15.1:0/2652966973 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:29:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.481+0000 7f739d023640 1 --2- 172.21.15.1:0/2652966973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7398152650 0x7f7398145060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.481+0000 7f739e025640 1 --2- 172.21.15.1:0/2652966973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73980c4200 0x7f73981455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.481+0000 7f739d824640 1 -- 172.21.15.1:0/2652966973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7398152650 msgr2=0x7f7398145060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.481+0000 7f739d824640 1 --2- 172.21.15.1:0/2652966973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7398152650 0x7f7398145060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.481+0000 7f739d824640 1 -- 172.21.15.1:0/2652966973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73980c4200 msgr2=0x7f73981455a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.481+0000 7f739d824640 1 --2- 172.21.15.1:0/2652966973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73980c4200 0x7f73981455a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.481+0000 7f739d824640 1 -- 172.21.15.1:0/2652966973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73981530b0 con 0x7f73980bf9d0 2024-03-06T07:29:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.481+0000 7f739e025640 1 --2- 172.21.15.1:0/2652966973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73980c4200 0x7f73981455a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.482+0000 7f739d824640 1 --2- 172.21.15.1:0/2652966973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73980bf9d0 0x7f73981473c0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f739400d800 tx=0x7f739400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.482+0000 7f738e7fc640 1 -- 172.21.15.1:0/2652966973 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739400b6c0 con 0x7f73980bf9d0 2024-03-06T07:29:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.482+0000 7f738e7fc640 1 -- 172.21.15.1:0/2652966973 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7394019070 con 0x7f73980bf9d0 2024-03-06T07:29:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.482+0000 7f739fa88640 1 -- 172.21.15.1:0/2652966973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73980c9890 con 0x7f73980bf9d0 2024-03-06T07:29:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.482+0000 7f738e7fc640 1 -- 172.21.15.1:0/2652966973 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739400ba80 con 0x7f73980bf9d0 2024-03-06T07:29:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.482+0000 7f739fa88640 1 -- 172.21.15.1:0/2652966973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7398147900 con 0x7f73980bf9d0 2024-03-06T07:29:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.483+0000 7f739fa88640 1 -- 172.21.15.1:0/2652966973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7398067fd0 con 0x7f73980bf9d0 2024-03-06T07:29:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.483+0000 7f738e7fc640 1 -- 172.21.15.1:0/2652966973 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f739400bc20 con 0x7f73980bf9d0 2024-03-06T07:29:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.484+0000 7f738e7fc640 1 --2- 172.21.15.1:0/2652966973 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f737c06b500 0x7f737c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.484+0000 7f738e7fc640 1 -- 172.21.15.1:0/2652966973 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f739408a780 con 0x7f73980bf9d0 2024-03-06T07:29:58.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.484+0000 7f739d023640 1 --2- 172.21.15.1:0/2652966973 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f737c06b500 0x7f737c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.485+0000 7f739d023640 1 --2- 172.21.15.1:0/2652966973 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f737c06b500 0x7f737c06d9c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f739814fca0 tx=0x7f7388008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.492+0000 7f738e7fc640 1 -- 172.21.15.1:0/2652966973 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7394059470 con 0x7f73980bf9d0 2024-03-06T07:29:58.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.714+0000 7f739fa88640 1 -- 172.21.15.1:0/2652966973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73980c9db0 con 0x7f73980bf9d0 2024-03-06T07:29:58.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.714+0000 7f738e7fc640 1 -- 172.21.15.1:0/2652966973 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5005 (secure 0 0 0) 0x7f7394058d70 con 0x7f73980bf9d0 2024-03-06T07:29:58.717 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:29:58.717 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:44.306981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:29:58.718 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 200 2024-03-06T07:29:58.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.716+0000 7f739fa88640 1 -- 172.21.15.1:0/2652966973 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f737c06b500 msgr2=0x7f737c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.717+0000 7f739fa88640 1 --2- 172.21.15.1:0/2652966973 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f737c06b500 0x7f737c06d9c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f739814fca0 tx=0x7f7388008040 comp rx=0 tx=0).stop 2024-03-06T07:29:58.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.717+0000 7f739fa88640 1 -- 172.21.15.1:0/2652966973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73980bf9d0 msgr2=0x7f73981473c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.717+0000 7f739fa88640 1 --2- 172.21.15.1:0/2652966973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73980bf9d0 0x7f73981473c0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f739400d800 tx=0x7f739400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.717+0000 7f739fa88640 1 -- 172.21.15.1:0/2652966973 shutdown_connections 2024-03-06T07:29:58.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.717+0000 7f739fa88640 1 --2- 172.21.15.1:0/2652966973 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f737c06b500 0x7f737c06d9c0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.717+0000 7f739fa88640 1 --2- 172.21.15.1:0/2652966973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73980c4200 0x7f73981455a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.717+0000 7f739fa88640 1 --2- 172.21.15.1:0/2652966973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7398152650 0x7f7398145060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.717+0000 7f739fa88640 1 --2- 172.21.15.1:0/2652966973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73980bf9d0 0x7f73981473c0 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.717+0000 7f739fa88640 1 -- 172.21.15.1:0/2652966973 >> 172.21.15.1:0/2652966973 conn(0x7f7398066ee0 msgr2=0x7f73980c52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.717+0000 7f739fa88640 1 -- 172.21.15.1:0/2652966973 shutdown_connections 2024-03-06T07:29:58.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:29:58.717+0000 7f739fa88640 1 -- 172.21.15.1:0/2652966973 wait complete. 2024-03-06T07:29:59.131 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:44.306981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:29:59.131 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:stopping 2024-03-06T07:30:00.133 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:00.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.234+0000 7f8cbc85c640 1 Processor -- start 2024-03-06T07:30:00.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.234+0000 7f8cbc85c640 1 -- start start 2024-03-06T07:30:00.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.234+0000 7f8cbc85c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cb80c3970 0x7f8cb80c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.234+0000 7f8cbc85c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8cb80c60b0 0x7f8cb8152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.234+0000 7f8cbc85c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb80c5e40 con 0x7f8cb8152670 2024-03-06T07:30:00.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.235+0000 7f8cbc85c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb8152ad0 con 0x7f8cb80c60b0 2024-03-06T07:30:00.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.235+0000 7f8cbc85c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80c67f0 con 0x7f8cb80c3970 2024-03-06T07:30:00.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.235+0000 7f8cadd9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8cb80c60b0 0x7f8cb8152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.235+0000 7f8cb6d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8cb8152670 0x7f8cb81553d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50678/0 (socket says 172.21.15.1:50678) 2024-03-06T07:30:00.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.235+0000 7f8cb6d9d640 1 -- 172.21.15.1:0/1481298264 learned_addr learned my addr 172.21.15.1:0/1481298264 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:00.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.235+0000 7f8cb659c640 1 --2- 172.21.15.1:0/1481298264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cb80c3970 0x7f8cb80c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.235+0000 7f8cadd9b640 1 -- 172.21.15.1:0/1481298264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cb80c3970 msgr2=0x7f8cb80c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.235+0000 7f8cadd9b640 1 --2- 172.21.15.1:0/1481298264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cb80c3970 0x7f8cb80c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.235+0000 7f8cadd9b640 1 -- 172.21.15.1:0/1481298264 >> v1:172.21.15.1:6789/0 conn(0x7f8cb8152670 legacy=0x7f8cb81553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.235+0000 7f8cadd9b640 1 -- 172.21.15.1:0/1481298264 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb80c6a40 con 0x7f8cb80c60b0 2024-03-06T07:30:00.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.236+0000 7f8cadd9b640 1 --2- 172.21.15.1:0/1481298264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb80c60b0 0x7f8cb8152130 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f8ca00097a0 tx=0x7f8ca002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ec7338254f2130a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.236+0000 7f8cb5d9b640 1 -- 172.21.15.1:0/1481298264 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0035020 con 0x7f8cb80c60b0 2024-03-06T07:30:00.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.236+0000 7f8cb5d9b640 1 -- 172.21.15.1:0/1481298264 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca0031830 con 0x7f8cb80c60b0 2024-03-06T07:30:00.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.236+0000 7f8cb5d9b640 1 -- 172.21.15.1:0/1481298264 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0031c00 con 0x7f8cb80c60b0 2024-03-06T07:30:00.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.236+0000 7f8cbc85c640 1 -- 172.21.15.1:0/1481298264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb80c60b0 msgr2=0x7f8cb8152130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.236+0000 7f8cbc85c640 1 --2- 172.21.15.1:0/1481298264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb80c60b0 0x7f8cb8152130 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f8ca00097a0 tx=0x7f8ca002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.236+0000 7f8cbc85c640 1 -- 172.21.15.1:0/1481298264 shutdown_connections 2024-03-06T07:30:00.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.236+0000 7f8cbc85c640 1 --2- 172.21.15.1:0/1481298264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb80c60b0 0x7f8cb8152130 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.236+0000 7f8cbc85c640 1 --2- 172.21.15.1:0/1481298264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cb80c3970 0x7f8cb80c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.236+0000 7f8cbc85c640 1 -- 172.21.15.1:0/1481298264 >> 172.21.15.1:0/1481298264 conn(0x7f8cb80c88f0 msgr2=0x7f8cb80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.236+0000 7f8cbc85c640 1 -- 172.21.15.1:0/1481298264 shutdown_connections 2024-03-06T07:30:00.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.236+0000 7f8cbc85c640 1 -- 172.21.15.1:0/1481298264 wait complete. 2024-03-06T07:30:00.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.239+0000 7f8cbc85c640 1 Processor -- start 2024-03-06T07:30:00.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.239+0000 7f8cbc85c640 1 -- start start 2024-03-06T07:30:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.240+0000 7f8cbc85c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb80c3970 0x7f8cb80711a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.240+0000 7f8cbc85c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb80c60b0 0x7f8cb80a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.240+0000 7f8cbc85c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cb8152670 0x7f8cb81ea7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.240+0000 7f8cbc85c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80c72e0 con 0x7f8cb80c3970 2024-03-06T07:30:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.240+0000 7f8cbc85c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80c7000 con 0x7f8cb80c60b0 2024-03-06T07:30:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.240+0000 7f8cbc85c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80c75c0 con 0x7f8cb8152670 2024-03-06T07:30:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.240+0000 7f8cb659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb80c3970 0x7f8cb80711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.240+0000 7f8cb659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb80c3970 0x7f8cb80711a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55124/0 (socket says 172.21.15.1:55124) 2024-03-06T07:30:00.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.240+0000 7f8cb659c640 1 -- 172.21.15.1:0/2751051264 learned_addr learned my addr 172.21.15.1:0/2751051264 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:00.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.240+0000 7f8cb6d9d640 1 --2- 172.21.15.1:0/2751051264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cb8152670 0x7f8cb81ea7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.240+0000 7f8cadd9b640 1 --2- 172.21.15.1:0/2751051264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb80c60b0 0x7f8cb80a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.240+0000 7f8cb659c640 1 -- 172.21.15.1:0/2751051264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cb8152670 msgr2=0x7f8cb81ea7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.240+0000 7f8cb659c640 1 --2- 172.21.15.1:0/2751051264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cb8152670 0x7f8cb81ea7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.240+0000 7f8cb659c640 1 -- 172.21.15.1:0/2751051264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb80c60b0 msgr2=0x7f8cb80a97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.240+0000 7f8cb659c640 1 --2- 172.21.15.1:0/2751051264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb80c60b0 0x7f8cb80a97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.240+0000 7f8cb659c640 1 -- 172.21.15.1:0/2751051264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb81530d0 con 0x7f8cb80c3970 2024-03-06T07:30:00.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.240+0000 7f8cb6d9d640 1 --2- 172.21.15.1:0/2751051264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cb8152670 0x7f8cb81ea7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:00.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.241+0000 7f8cb659c640 1 --2- 172.21.15.1:0/2751051264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb80c3970 0x7f8cb80711a0 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800d830 tx=0x7f8ca800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.241+0000 7f8caf7fe640 1 -- 172.21.15.1:0/2751051264 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca800b6c0 con 0x7f8cb80c3970 2024-03-06T07:30:00.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.241+0000 7f8caf7fe640 1 -- 172.21.15.1:0/2751051264 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca8019070 con 0x7f8cb80c3970 2024-03-06T07:30:00.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.241+0000 7f8cbc85c640 1 -- 172.21.15.1:0/2751051264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb814d1a0 con 0x7f8cb80c3970 2024-03-06T07:30:00.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.241+0000 7f8caf7fe640 1 -- 172.21.15.1:0/2751051264 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca800ba80 con 0x7f8cb80c3970 2024-03-06T07:30:00.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.241+0000 7f8cbc85c640 1 -- 172.21.15.1:0/2751051264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb80a9d10 con 0x7f8cb80c3970 2024-03-06T07:30:00.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.242+0000 7f8cbc85c640 1 -- 172.21.15.1:0/2751051264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb80686e0 con 0x7f8cb80c3970 2024-03-06T07:30:00.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.243+0000 7f8caf7fe640 1 -- 172.21.15.1:0/2751051264 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ca801e460 con 0x7f8cb80c3970 2024-03-06T07:30:00.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.243+0000 7f8caf7fe640 1 --2- 172.21.15.1:0/2751051264 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c9806b500 0x7f8c9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.243+0000 7f8caf7fe640 1 -- 172.21.15.1:0/2751051264 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8ca808a9c0 con 0x7f8cb80c3970 2024-03-06T07:30:00.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.243+0000 7f8cadd9b640 1 --2- 172.21.15.1:0/2751051264 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c9806b500 0x7f8c9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.244+0000 7f8cadd9b640 1 --2- 172.21.15.1:0/2751051264 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c9806b500 0x7f8c9806d9c0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0036000 tx=0x7f8ca0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.247+0000 7f8caf7fe640 1 -- 172.21.15.1:0/2751051264 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ca808ce20 con 0x7f8cb80c3970 2024-03-06T07:30:00.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.467+0000 7f8cbc85c640 1 -- 172.21.15.1:0/2751051264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cb80c5840 con 0x7f8cb80c3970 2024-03-06T07:30:00.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.467+0000 7f8caf7fe640 1 -- 172.21.15.1:0/2751051264 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5005 (secure 0 0 0) 0x7f8ca8059740 con 0x7f8cb80c3970 2024-03-06T07:30:00.469 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:00.469 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:44.306981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:00.470 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 200 2024-03-06T07:30:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.469+0000 7f8cbc85c640 1 -- 172.21.15.1:0/2751051264 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c9806b500 msgr2=0x7f8c9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.469+0000 7f8cbc85c640 1 --2- 172.21.15.1:0/2751051264 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c9806b500 0x7f8c9806d9c0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0036000 tx=0x7f8ca0008030 comp rx=0 tx=0).stop 2024-03-06T07:30:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.469+0000 7f8cbc85c640 1 -- 172.21.15.1:0/2751051264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb80c3970 msgr2=0x7f8cb80711a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.469+0000 7f8cbc85c640 1 --2- 172.21.15.1:0/2751051264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb80c3970 0x7f8cb80711a0 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800d830 tx=0x7f8ca800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.469+0000 7f8cbc85c640 1 -- 172.21.15.1:0/2751051264 shutdown_connections 2024-03-06T07:30:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.469+0000 7f8cbc85c640 1 --2- 172.21.15.1:0/2751051264 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8c9806b500 0x7f8c9806d9c0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.469+0000 7f8cbc85c640 1 --2- 172.21.15.1:0/2751051264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8cb8152670 0x7f8cb81ea7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.469+0000 7f8cbc85c640 1 --2- 172.21.15.1:0/2751051264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cb80c60b0 0x7f8cb80a97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.469+0000 7f8cbc85c640 1 --2- 172.21.15.1:0/2751051264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb80c3970 0x7f8cb80711a0 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.469+0000 7f8cbc85c640 1 -- 172.21.15.1:0/2751051264 >> 172.21.15.1:0/2751051264 conn(0x7f8cb80c88f0 msgr2=0x7f8cb80aae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.469+0000 7f8cbc85c640 1 -- 172.21.15.1:0/2751051264 shutdown_connections 2024-03-06T07:30:00.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:00.469+0000 7f8cbc85c640 1 -- 172.21.15.1:0/2751051264 wait complete. 2024-03-06T07:30:00.884 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:44.306981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:00.884 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:stopping 2024-03-06T07:30:01.886 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.986+0000 7f719ef3a640 1 Processor -- start 2024-03-06T07:30:01.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.986+0000 7f719ef3a640 1 -- start start 2024-03-06T07:30:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.986+0000 7f719ef3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7198150520 0x7f71981543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.986+0000 7f719ef3a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f71980c32f0 0x7f71981537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.986+0000 7f719ef3a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7198070c80 0x7f71980710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.986+0000 7f719ef3a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7198150900 con 0x7f71980c32f0 2024-03-06T07:30:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.986+0000 7f719ef3a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7198071790 con 0x7f7198070c80 2024-03-06T07:30:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.986+0000 7f719ef3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7198157050 con 0x7f7198150520 2024-03-06T07:30:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.986+0000 7f7197fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f71980c32f0 0x7f71981537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.986+0000 7f7197fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f71980c32f0 0x7f71981537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55138/0 (socket says 172.21.15.1:55138) 2024-03-06T07:30:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.987+0000 7f7197fff640 1 -- 172.21.15.1:0/1272244183 learned_addr learned my addr 172.21.15.1:0/1272244183 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.987+0000 7f719d4d7640 1 --2- 172.21.15.1:0/1272244183 >> v2:172.21.15.98:3300/0 conn(0x7f7198070c80 0x7f71980710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.987+0000 7f719ccd6640 1 --2- 172.21.15.1:0/1272244183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7198150520 0x7f71981543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.987+0000 7f7197fff640 1 -- 172.21.15.1:0/1272244183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7198150520 msgr2=0x7f71981543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.987+0000 7f7197fff640 1 --2- 172.21.15.1:0/1272244183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7198150520 0x7f71981543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.987+0000 7f7197fff640 1 -- 172.21.15.1:0/1272244183 >> v2:172.21.15.98:3300/0 conn(0x7f7198070c80 msgr2=0x7f71980710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.987+0000 7f7197fff640 1 --2- 172.21.15.1:0/1272244183 >> v2:172.21.15.98:3300/0 conn(0x7f7198070c80 0x7f71980710f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.987+0000 7f7197fff640 1 -- 172.21.15.1:0/1272244183 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7198152fc0 con 0x7f71980c32f0 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.988+0000 7f7197fff640 1 --2- 172.21.15.1:0/1272244183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71980c32f0 0x7f71981537c0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f71980c9990 tx=0x7f718402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0e33192c42e961 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.988+0000 7f71977fe640 1 -- 172.21.15.1:0/1272244183 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7184035020 con 0x7f71980c32f0 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.988+0000 7f71977fe640 1 -- 172.21.15.1:0/1272244183 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7184031870 con 0x7f71980c32f0 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.988+0000 7f71977fe640 1 -- 172.21.15.1:0/1272244183 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7184031c40 con 0x7f71980c32f0 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.988+0000 7f719ef3a640 1 -- 172.21.15.1:0/1272244183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71980c32f0 msgr2=0x7f71981537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.988+0000 7f719ef3a640 1 --2- 172.21.15.1:0/1272244183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71980c32f0 0x7f71981537c0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f71980c9990 tx=0x7f718402b910 comp rx=0 tx=0).stop 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.988+0000 7f719ef3a640 1 -- 172.21.15.1:0/1272244183 shutdown_connections 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.988+0000 7f719ef3a640 1 --2- 172.21.15.1:0/1272244183 >> v2:172.21.15.98:3300/0 conn(0x7f7198070c80 0x7f71980710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.988+0000 7f719ef3a640 1 --2- 172.21.15.1:0/1272244183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71980c32f0 0x7f71981537c0 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.988+0000 7f719ef3a640 1 --2- 172.21.15.1:0/1272244183 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7198150520 0x7f71981543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.988+0000 7f719ef3a640 1 -- 172.21.15.1:0/1272244183 >> 172.21.15.1:0/1272244183 conn(0x7f71980c88f0 msgr2=0x7f71980c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.988+0000 7f719ef3a640 1 -- 172.21.15.1:0/1272244183 shutdown_connections 2024-03-06T07:30:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.988+0000 7f719ef3a640 1 -- 172.21.15.1:0/1272244183 wait complete. 2024-03-06T07:30:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.990+0000 7f719ef3a640 1 Processor -- start 2024-03-06T07:30:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.990+0000 7f719ef3a640 1 -- start start 2024-03-06T07:30:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.991+0000 7f719ef3a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7198070c80 0x7f71980ac9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.991+0000 7f719ef3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71980c32f0 0x7f7198155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.991+0000 7f719ef3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7198150520 0x7f7198155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.991+0000 7f719ef3a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71980c7ec0 con 0x7f7198070c80 2024-03-06T07:30:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.991+0000 7f719ef3a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7198157330 con 0x7f7198150520 2024-03-06T07:30:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.991+0000 7f719ef3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7198153d00 con 0x7f71980c32f0 2024-03-06T07:30:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.991+0000 7f719ccd6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7198070c80 0x7f71980ac9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.991+0000 7f719ccd6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7198070c80 0x7f71980ac9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55154/0 (socket says 172.21.15.1:55154) 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.991+0000 7f719ccd6640 1 -- 172.21.15.1:0/2637104136 learned_addr learned my addr 172.21.15.1:0/2637104136 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.991+0000 7f7197fff640 1 --2- 172.21.15.1:0/2637104136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71980c32f0 0x7f7198155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.991+0000 7f719d4d7640 1 --2- 172.21.15.1:0/2637104136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7198150520 0x7f7198155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.991+0000 7f719ccd6640 1 -- 172.21.15.1:0/2637104136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71980c32f0 msgr2=0x7f7198155680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.991+0000 7f719ccd6640 1 --2- 172.21.15.1:0/2637104136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71980c32f0 0x7f7198155680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.991+0000 7f719ccd6640 1 -- 172.21.15.1:0/2637104136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7198150520 msgr2=0x7f7198155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.991+0000 7f719ccd6640 1 --2- 172.21.15.1:0/2637104136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7198150520 0x7f7198155bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.991+0000 7f719ccd6640 1 -- 172.21.15.1:0/2637104136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71981491b0 con 0x7f7198070c80 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.991+0000 7f7197fff640 1 --2- 172.21.15.1:0/2637104136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71980c32f0 0x7f7198155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.992+0000 7f719ccd6640 1 --2- 172.21.15.1:0/2637104136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7198070c80 0x7f71980ac9f0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f718c00d5f0 tx=0x7f718c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.992+0000 7f71957fa640 1 -- 172.21.15.1:0/2637104136 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718c0125f0 con 0x7f7198070c80 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.992+0000 7f71957fa640 1 -- 172.21.15.1:0/2637104136 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718c019070 con 0x7f7198070c80 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.992+0000 7f719ef3a640 1 -- 172.21.15.1:0/2637104136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7198148f10 con 0x7f7198070c80 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.992+0000 7f71957fa640 1 -- 172.21.15.1:0/2637104136 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718c0129d0 con 0x7f7198070c80 2024-03-06T07:30:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.992+0000 7f719ef3a640 1 -- 172.21.15.1:0/2637104136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7198156ae0 con 0x7f7198070c80 2024-03-06T07:30:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.993+0000 7f719ef3a640 1 -- 172.21.15.1:0/2637104136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7198156e20 con 0x7f7198070c80 2024-03-06T07:30:01.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.993+0000 7f71957fa640 1 -- 172.21.15.1:0/2637104136 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f718c012b70 con 0x7f7198070c80 2024-03-06T07:30:01.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.994+0000 7f71957fa640 1 --2- 172.21.15.1:0/2637104136 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f717c06b500 0x7f717c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.994+0000 7f71957fa640 1 -- 172.21.15.1:0/2637104136 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f718c08a6e0 con 0x7f7198070c80 2024-03-06T07:30:01.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.994+0000 7f7197fff640 1 --2- 172.21.15.1:0/2637104136 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f717c06b500 0x7f717c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.994+0000 7f7197fff640 1 --2- 172.21.15.1:0/2637104136 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f717c06b500 0x7f717c06d9c0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f7184036000 tx=0x7f7184008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:01.998+0000 7f71957fa640 1 -- 172.21.15.1:0/2637104136 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7198156e20 con 0x7f7198070c80 2024-03-06T07:30:02.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:02.217+0000 7f719ef3a640 1 -- 172.21.15.1:0/2637104136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71980aec90 con 0x7f7198070c80 2024-03-06T07:30:02.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:02.218+0000 7f71957fa640 1 -- 172.21.15.1:0/2637104136 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5005 (secure 0 0 0) 0x7f71980aec90 con 0x7f7198070c80 2024-03-06T07:30:02.220 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:02.220 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:44.306981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:02.221 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 200 2024-03-06T07:30:02.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:02.220+0000 7f719ef3a640 1 -- 172.21.15.1:0/2637104136 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f717c06b500 msgr2=0x7f717c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:02.220+0000 7f719ef3a640 1 --2- 172.21.15.1:0/2637104136 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f717c06b500 0x7f717c06d9c0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f7184036000 tx=0x7f7184008030 comp rx=0 tx=0).stop 2024-03-06T07:30:02.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:02.220+0000 7f719ef3a640 1 -- 172.21.15.1:0/2637104136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7198070c80 msgr2=0x7f71980ac9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:02.220+0000 7f719ef3a640 1 --2- 172.21.15.1:0/2637104136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7198070c80 0x7f71980ac9f0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f718c00d5f0 tx=0x7f718c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:02.220+0000 7f719ef3a640 1 -- 172.21.15.1:0/2637104136 shutdown_connections 2024-03-06T07:30:02.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:02.220+0000 7f719ef3a640 1 --2- 172.21.15.1:0/2637104136 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f717c06b500 0x7f717c06d9c0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:02.220+0000 7f719ef3a640 1 --2- 172.21.15.1:0/2637104136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7198150520 0x7f7198155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:02.220+0000 7f719ef3a640 1 --2- 172.21.15.1:0/2637104136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71980c32f0 0x7f7198155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:02.220+0000 7f719ef3a640 1 --2- 172.21.15.1:0/2637104136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7198070c80 0x7f71980ac9f0 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:02.220+0000 7f719ef3a640 1 -- 172.21.15.1:0/2637104136 >> 172.21.15.1:0/2637104136 conn(0x7f71980c88f0 msgr2=0x7f719814c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:02.220+0000 7f719ef3a640 1 -- 172.21.15.1:0/2637104136 shutdown_connections 2024-03-06T07:30:02.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:02.220+0000 7f719ef3a640 1 -- 172.21.15.1:0/2637104136 wait complete. 2024-03-06T07:30:02.635 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:44.306981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:02.635 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:stopping 2024-03-06T07:30:02.664 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:02.666 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:03.636 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:03.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.735+0000 7fde4df9a640 1 Processor -- start 2024-03-06T07:30:03.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.735+0000 7fde4df9a640 1 -- start start 2024-03-06T07:30:03.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.736+0000 7fde4df9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde480bf9d0 0x7fde48155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.736+0000 7fde4df9a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde480c2fc0 con 0x7fde480c6b10 2024-03-06T07:30:03.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.736+0000 7fde4df9a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde48158da0 con 0x7fde48152650 2024-03-06T07:30:03.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.736+0000 7fde4df9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde48155800 con 0x7fde480bf9d0 2024-03-06T07:30:03.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.736+0000 7fde46ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fde480c6b10 0x7fde480c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50688/0 (socket says 172.21.15.1:50688) 2024-03-06T07:30:03.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.736+0000 7fde46ffd640 1 -- 172.21.15.1:0/1616324198 learned_addr learned my addr 172.21.15.1:0/1616324198 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:03.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde47fff640 1 --2- 172.21.15.1:0/1616324198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde480bf9d0 0x7fde48155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde467fc640 1 -- 172.21.15.1:0/1616324198 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449353956 0 0) 0x7fde480c2fc0 con 0x7fde480c6b10 2024-03-06T07:30:03.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde467fc640 1 -- 172.21.15.1:0/1616324198 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde30003520 con 0x7fde480c6b10 2024-03-06T07:30:03.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde467fc640 1 -- 172.21.15.1:0/1616324198 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2357777975 0 0) 0x7fde30003520 con 0x7fde480c6b10 2024-03-06T07:30:03.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde467fc640 1 -- 172.21.15.1:0/1616324198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde480bf9d0 msgr2=0x7fde48155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde467fc640 1 --2- 172.21.15.1:0/1616324198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde480bf9d0 0x7fde48155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde467fc640 1 -- 172.21.15.1:0/1616324198 >> v1:172.21.15.98:6789/0 conn(0x7fde48152650 legacy=0x7fde48158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde467fc640 1 -- 172.21.15.1:0/1616324198 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde480c6690 con 0x7fde480c6b10 2024-03-06T07:30:03.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde467fc640 1 -- 172.21.15.1:0/1616324198 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fde3c001690 con 0x7fde480c6b10 2024-03-06T07:30:03.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde467fc640 1 -- 172.21.15.1:0/1616324198 >> v1:172.21.15.1:6789/0 conn(0x7fde480c6b10 legacy=0x7fde480c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde467fc640 1 --2- 172.21.15.1:0/1616324198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde480bf9d0 0x7fde300039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde467fc640 1 --2- 172.21.15.1:0/1616324198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde30003f60 0x7fde30004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde467fc640 1 --2- 172.21.15.1:0/1616324198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde30004d40 0x7fde300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde467fc640 1 -- 172.21.15.1:0/1616324198 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde48155800 con 0x7fde30003f60 2024-03-06T07:30:03.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde467fc640 1 -- 172.21.15.1:0/1616324198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3000bb50 con 0x7fde30004d40 2024-03-06T07:30:03.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde467fc640 1 -- 172.21.15.1:0/1616324198 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde3000bd50 con 0x7fde480bf9d0 2024-03-06T07:30:03.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde46ffd640 1 --2- 172.21.15.1:0/1616324198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde30003f60 0x7fde30004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde477fe640 1 --2- 172.21.15.1:0/1616324198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde480bf9d0 0x7fde300039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde47fff640 1 --2- 172.21.15.1:0/1616324198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde30004d40 0x7fde300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde46ffd640 1 -- 172.21.15.1:0/1616324198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde480bf9d0 msgr2=0x7fde300039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde46ffd640 1 --2- 172.21.15.1:0/1616324198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde480bf9d0 0x7fde300039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde46ffd640 1 -- 172.21.15.1:0/1616324198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde30004d40 msgr2=0x7fde300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde46ffd640 1 --2- 172.21.15.1:0/1616324198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde30004d40 0x7fde300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.737+0000 7fde46ffd640 1 -- 172.21.15.1:0/1616324198 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde30003520 con 0x7fde30003f60 2024-03-06T07:30:03.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.738+0000 7fde46ffd640 1 --2- 172.21.15.1:0/1616324198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde30003f60 0x7fde30004800 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00c210 tx=0x7fde30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee866981bb18931 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.738+0000 7fde467fc640 1 -- 172.21.15.1:0/1616324198 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c007d10 con 0x7fde30003f60 2024-03-06T07:30:03.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.738+0000 7fde467fc640 1 -- 172.21.15.1:0/1616324198 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c030070 con 0x7fde30003f60 2024-03-06T07:30:03.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.738+0000 7fde467fc640 1 -- 172.21.15.1:0/1616324198 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c005d10 con 0x7fde30003f60 2024-03-06T07:30:03.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.739+0000 7fde4df9a640 1 -- 172.21.15.1:0/1616324198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde30003f60 msgr2=0x7fde30004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.739+0000 7fde4df9a640 1 --2- 172.21.15.1:0/1616324198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde30003f60 0x7fde30004800 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00c210 tx=0x7fde30005830 comp rx=0 tx=0).stop 2024-03-06T07:30:03.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.739+0000 7fde47fff640 1 -- 172.21.15.1:0/1616324198 reap_dead start 2024-03-06T07:30:03.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.739+0000 7fde4df9a640 1 -- 172.21.15.1:0/1616324198 shutdown_connections 2024-03-06T07:30:03.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.739+0000 7fde4df9a640 1 -- 172.21.15.1:0/1616324198 >> 172.21.15.1:0/1616324198 conn(0x7fde48066ee0 msgr2=0x7fde480672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.739+0000 7fde4df9a640 1 -- 172.21.15.1:0/1616324198 shutdown_connections 2024-03-06T07:30:03.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.739+0000 7fde4df9a640 1 -- 172.21.15.1:0/1616324198 wait complete. 2024-03-06T07:30:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.740+0000 7fde4df9a640 1 Processor -- start 2024-03-06T07:30:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.740+0000 7fde4df9a640 1 -- start start 2024-03-06T07:30:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.741+0000 7fde4df9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde481e1cb0 0x7fde4814db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.741+0000 7fde4df9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde48151c50 0x7fde481ee410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.741+0000 7fde4df9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde481588e0 0x7fde4814c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.741+0000 7fde4df9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde481551b0 con 0x7fde48151c50 2024-03-06T07:30:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.741+0000 7fde4df9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde480c6fa0 con 0x7fde481588e0 2024-03-06T07:30:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.741+0000 7fde4df9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde48155490 con 0x7fde481e1cb0 2024-03-06T07:30:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.741+0000 7fde46ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde48151c50 0x7fde481ee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.741+0000 7fde46ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde48151c50 0x7fde481ee410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55168/0 (socket says 172.21.15.1:55168) 2024-03-06T07:30:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.741+0000 7fde46ffd640 1 -- 172.21.15.1:0/2436540789 learned_addr learned my addr 172.21.15.1:0/2436540789 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.741+0000 7fde477fe640 1 --2- 172.21.15.1:0/2436540789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde481e1cb0 0x7fde4814db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.741+0000 7fde47fff640 1 --2- 172.21.15.1:0/2436540789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde481588e0 0x7fde4814c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.741+0000 7fde46ffd640 1 -- 172.21.15.1:0/2436540789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde481e1cb0 msgr2=0x7fde4814db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.741+0000 7fde46ffd640 1 --2- 172.21.15.1:0/2436540789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde481e1cb0 0x7fde4814db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.741+0000 7fde46ffd640 1 -- 172.21.15.1:0/2436540789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde481588e0 msgr2=0x7fde4814c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.741+0000 7fde46ffd640 1 --2- 172.21.15.1:0/2436540789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde481588e0 0x7fde4814c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.741+0000 7fde46ffd640 1 -- 172.21.15.1:0/2436540789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4814e0a0 con 0x7fde48151c50 2024-03-06T07:30:03.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.742+0000 7fde46ffd640 1 --2- 172.21.15.1:0/2436540789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde48151c50 0x7fde481ee410 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7fde3c004840 tx=0x7fde480bfe40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.742+0000 7fde2ffff640 1 -- 172.21.15.1:0/2436540789 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c045070 con 0x7fde48151c50 2024-03-06T07:30:03.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.742+0000 7fde2ffff640 1 -- 172.21.15.1:0/2436540789 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c004ab0 con 0x7fde48151c50 2024-03-06T07:30:03.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.742+0000 7fde4df9a640 1 -- 172.21.15.1:0/2436540789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4814b6a0 con 0x7fde48151c50 2024-03-06T07:30:03.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.742+0000 7fde2ffff640 1 -- 172.21.15.1:0/2436540789 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c007d10 con 0x7fde48151c50 2024-03-06T07:30:03.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.742+0000 7fde4df9a640 1 -- 172.21.15.1:0/2436540789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde481ee950 con 0x7fde48151c50 2024-03-06T07:30:03.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.743+0000 7fde4df9a640 1 -- 172.21.15.1:0/2436540789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde4814b400 con 0x7fde48151c50 2024-03-06T07:30:03.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.743+0000 7fde2ffff640 1 -- 172.21.15.1:0/2436540789 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde3c007070 con 0x7fde48151c50 2024-03-06T07:30:03.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.744+0000 7fde2ffff640 1 --2- 172.21.15.1:0/2436540789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fde3406b4d0 0x7fde3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.744+0000 7fde2ffff640 1 -- 172.21.15.1:0/2436540789 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fde3c0aacd0 con 0x7fde48151c50 2024-03-06T07:30:03.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.744+0000 7fde477fe640 1 --2- 172.21.15.1:0/2436540789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fde3406b4d0 0x7fde3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.745+0000 7fde477fe640 1 --2- 172.21.15.1:0/2436540789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fde3406b4d0 0x7fde3406d990 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fde4814bcc0 tx=0x7fde38009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.748+0000 7fde2ffff640 1 -- 172.21.15.1:0/2436540789 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde3c0794c0 con 0x7fde48151c50 2024-03-06T07:30:03.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.972+0000 7fde4df9a640 1 -- 172.21.15.1:0/2436540789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde4814d430 con 0x7fde48151c50 2024-03-06T07:30:03.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.972+0000 7fde2ffff640 1 -- 172.21.15.1:0/2436540789 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5005 (secure 0 0 0) 0x7fde3c07d560 con 0x7fde48151c50 2024-03-06T07:30:03.974 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:03.974 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:44.306981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:03.975 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 200 2024-03-06T07:30:03.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.974+0000 7fde4df9a640 1 -- 172.21.15.1:0/2436540789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fde3406b4d0 msgr2=0x7fde3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.974+0000 7fde4df9a640 1 --2- 172.21.15.1:0/2436540789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fde3406b4d0 0x7fde3406d990 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fde4814bcc0 tx=0x7fde38009040 comp rx=0 tx=0).stop 2024-03-06T07:30:03.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.974+0000 7fde4df9a640 1 -- 172.21.15.1:0/2436540789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde48151c50 msgr2=0x7fde481ee410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.974+0000 7fde4df9a640 1 --2- 172.21.15.1:0/2436540789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde48151c50 0x7fde481ee410 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7fde3c004840 tx=0x7fde480bfe40 comp rx=0 tx=0).stop 2024-03-06T07:30:03.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.974+0000 7fde4df9a640 1 -- 172.21.15.1:0/2436540789 shutdown_connections 2024-03-06T07:30:03.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.974+0000 7fde4df9a640 1 --2- 172.21.15.1:0/2436540789 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fde3406b4d0 0x7fde3406d990 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.974+0000 7fde4df9a640 1 --2- 172.21.15.1:0/2436540789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde481588e0 0x7fde4814c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.974+0000 7fde4df9a640 1 --2- 172.21.15.1:0/2436540789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde48151c50 0x7fde481ee410 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.974+0000 7fde4df9a640 1 --2- 172.21.15.1:0/2436540789 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde481e1cb0 0x7fde4814db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.974+0000 7fde4df9a640 1 -- 172.21.15.1:0/2436540789 >> 172.21.15.1:0/2436540789 conn(0x7fde48066ee0 msgr2=0x7fde480c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.974+0000 7fde4df9a640 1 -- 172.21.15.1:0/2436540789 shutdown_connections 2024-03-06T07:30:03.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:03.974+0000 7fde4df9a640 1 -- 172.21.15.1:0/2436540789 wait complete. 2024-03-06T07:30:04.390 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:44.306981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:04.390 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:stopping 2024-03-06T07:30:05.391 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:05.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.491+0000 7f9f957b6640 1 Processor -- start 2024-03-06T07:30:05.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.491+0000 7f9f957b6640 1 -- start start 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.492+0000 7f9f957b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9f90152650 0x7f9f90155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.492+0000 7f9f957b6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f900c2fc0 con 0x7f9f900bf9d0 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.492+0000 7f9f957b6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f90158da0 con 0x7f9f900c6b10 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.492+0000 7f9f957b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f90155800 con 0x7f9f90152650 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.492+0000 7f9f8f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9f900bf9d0 0x7f9f900c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34072/0 (socket says 172.21.15.1:34072) 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.492+0000 7f9f8f7fe640 1 -- 172.21.15.1:0/1228246001 learned_addr learned my addr 172.21.15.1:0/1228246001 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.492+0000 7f9f8effd640 1 --2- 172.21.15.1:0/1228246001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9f90152650 0x7f9f90155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.492+0000 7f9f8dffb640 1 -- 172.21.15.1:0/1228246001 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3729818483 0 0) 0x7f9f900c2fc0 con 0x7f9f900bf9d0 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.492+0000 7f9f8dffb640 1 -- 172.21.15.1:0/1228246001 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f78003520 con 0x7f9f900bf9d0 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.493+0000 7f9f8dffb640 1 -- 172.21.15.1:0/1228246001 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556219239 0 0) 0x7f9f90158da0 con 0x7f9f900c6b10 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.493+0000 7f9f8dffb640 1 -- 172.21.15.1:0/1228246001 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f900c2fc0 con 0x7f9f900c6b10 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.493+0000 7f9f8effd640 1 -- 172.21.15.1:0/1228246001 >> v1:172.21.15.98:6789/0 conn(0x7f9f900c6b10 legacy=0x7f9f90158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.493+0000 7f9f8effd640 1 -- 172.21.15.1:0/1228246001 >> v1:172.21.15.1:6789/0 conn(0x7f9f900bf9d0 legacy=0x7f9f900c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.493+0000 7f9f8effd640 1 -- 172.21.15.1:0/1228246001 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f900c6690 con 0x7f9f90152650 2024-03-06T07:30:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.493+0000 7f9f8effd640 1 --2- 172.21.15.1:0/1228246001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9f90152650 0x7f9f90155c90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9f800097d0 tx=0x7f9f8002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc23804cccedef8e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.494+0000 7f9f8dffb640 1 -- 172.21.15.1:0/1228246001 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f80035020 con 0x7f9f90152650 2024-03-06T07:30:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.494+0000 7f9f8dffb640 1 -- 172.21.15.1:0/1228246001 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f80031800 con 0x7f9f90152650 2024-03-06T07:30:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.494+0000 7f9f8dffb640 1 -- 172.21.15.1:0/1228246001 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f80031bd0 con 0x7f9f90152650 2024-03-06T07:30:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.494+0000 7f9f957b6640 1 -- 172.21.15.1:0/1228246001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9f90152650 msgr2=0x7f9f90155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.494+0000 7f9f957b6640 1 --2- 172.21.15.1:0/1228246001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9f90152650 0x7f9f90155c90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9f800097d0 tx=0x7f9f8002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.494+0000 7f9f957b6640 1 -- 172.21.15.1:0/1228246001 shutdown_connections 2024-03-06T07:30:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.494+0000 7f9f957b6640 1 --2- 172.21.15.1:0/1228246001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9f90152650 0x7f9f90155c90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.494+0000 7f9f957b6640 1 -- 172.21.15.1:0/1228246001 >> 172.21.15.1:0/1228246001 conn(0x7f9f90066ee0 msgr2=0x7f9f900672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.494+0000 7f9f957b6640 1 -- 172.21.15.1:0/1228246001 shutdown_connections 2024-03-06T07:30:05.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.494+0000 7f9f957b6640 1 -- 172.21.15.1:0/1228246001 wait complete. 2024-03-06T07:30:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.496+0000 7f9f957b6640 1 Processor -- start 2024-03-06T07:30:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.496+0000 7f9f957b6640 1 -- start start 2024-03-06T07:30:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.496+0000 7f9f957b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f900bf9d0 0x7f9f901543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.496+0000 7f9f957b6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9f90152650 0x7f9f90156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.496+0000 7f9f957b6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f901ebba0 0x7f9f90151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.496+0000 7f9f957b6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f901551b0 con 0x7f9f901ebba0 2024-03-06T07:30:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.496+0000 7f9f957b6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f900c6fa0 con 0x7f9f900bf9d0 2024-03-06T07:30:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.496+0000 7f9f957b6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f90155490 con 0x7f9f90152650 2024-03-06T07:30:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.496+0000 7f9f8f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f901ebba0 0x7f9f90151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.496+0000 7f9f8f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f901ebba0 0x7f9f90151110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57696/0 (socket says 172.21.15.1:57696) 2024-03-06T07:30:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.496+0000 7f9f8f7fe640 1 -- 172.21.15.1:0/4265517288 learned_addr learned my addr 172.21.15.1:0/4265517288 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.497+0000 7f9f8effd640 1 --2- 172.21.15.1:0/4265517288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f900bf9d0 0x7f9f901543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.497+0000 7f9f8e7fc640 1 --2- 172.21.15.1:0/4265517288 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9f90152650 0x7f9f90156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.497+0000 7f9f8f7fe640 1 -- 172.21.15.1:0/4265517288 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9f90152650 msgr2=0x7f9f90156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.497+0000 7f9f8f7fe640 1 --2- 172.21.15.1:0/4265517288 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9f90152650 0x7f9f90156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.497+0000 7f9f8f7fe640 1 -- 172.21.15.1:0/4265517288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f900bf9d0 msgr2=0x7f9f901543b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.497+0000 7f9f8f7fe640 1 --2- 172.21.15.1:0/4265517288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f900bf9d0 0x7f9f901543b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.497+0000 7f9f8f7fe640 1 -- 172.21.15.1:0/4265517288 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f901ec050 con 0x7f9f901ebba0 2024-03-06T07:30:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.497+0000 7f9f8f7fe640 1 --2- 172.21.15.1:0/4265517288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f901ebba0 0x7f9f90151110 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f9f8800cb20 tx=0x7f9f8800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.497+0000 7f9f777fe640 1 -- 172.21.15.1:0/4265517288 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f88017070 con 0x7f9f901ebba0 2024-03-06T07:30:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.497+0000 7f9f777fe640 1 -- 172.21.15.1:0/4265517288 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8800aca0 con 0x7f9f901ebba0 2024-03-06T07:30:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.497+0000 7f9f957b6640 1 -- 172.21.15.1:0/4265517288 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f90154a00 con 0x7f9f901ebba0 2024-03-06T07:30:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.497+0000 7f9f777fe640 1 -- 172.21.15.1:0/4265517288 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f88012600 con 0x7f9f901ebba0 2024-03-06T07:30:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.497+0000 7f9f957b6640 1 -- 172.21.15.1:0/4265517288 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f90152c80 con 0x7f9f901ebba0 2024-03-06T07:30:05.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.499+0000 7f9f957b6640 1 -- 172.21.15.1:0/4265517288 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f90067fd0 con 0x7f9f901ebba0 2024-03-06T07:30:05.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.499+0000 7f9f777fe640 1 -- 172.21.15.1:0/4265517288 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f8801a070 con 0x7f9f901ebba0 2024-03-06T07:30:05.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.499+0000 7f9f777fe640 1 --2- 172.21.15.1:0/4265517288 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9f7c06b500 0x7f9f7c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.500+0000 7f9f777fe640 1 -- 172.21.15.1:0/4265517288 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9f88089f90 con 0x7f9f901ebba0 2024-03-06T07:30:05.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.500+0000 7f9f8effd640 1 --2- 172.21.15.1:0/4265517288 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9f7c06b500 0x7f9f7c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.500+0000 7f9f8effd640 1 --2- 172.21.15.1:0/4265517288 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9f7c06b500 0x7f9f7c06d9c0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f9f80036000 tx=0x7f9f80008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.506+0000 7f9f777fe640 1 -- 172.21.15.1:0/4265517288 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f88058630 con 0x7f9f901ebba0 2024-03-06T07:30:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.726+0000 7f9f957b6640 1 -- 172.21.15.1:0/4265517288 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f900b01a0 con 0x7f9f901ebba0 2024-03-06T07:30:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.727+0000 7f9f777fe640 1 -- 172.21.15.1:0/4265517288 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5005 (secure 0 0 0) 0x7f9f8805c6d0 con 0x7f9f901ebba0 2024-03-06T07:30:05.729 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:05.729 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:44.306981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 200 2024-03-06T07:30:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.729+0000 7f9f957b6640 1 -- 172.21.15.1:0/4265517288 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9f7c06b500 msgr2=0x7f9f7c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.729+0000 7f9f957b6640 1 --2- 172.21.15.1:0/4265517288 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9f7c06b500 0x7f9f7c06d9c0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f9f80036000 tx=0x7f9f80008030 comp rx=0 tx=0).stop 2024-03-06T07:30:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.729+0000 7f9f957b6640 1 -- 172.21.15.1:0/4265517288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f901ebba0 msgr2=0x7f9f90151110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.729+0000 7f9f957b6640 1 --2- 172.21.15.1:0/4265517288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f901ebba0 0x7f9f90151110 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f9f8800cb20 tx=0x7f9f8800a060 comp rx=0 tx=0).stop 2024-03-06T07:30:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.729+0000 7f9f957b6640 1 -- 172.21.15.1:0/4265517288 shutdown_connections 2024-03-06T07:30:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.729+0000 7f9f957b6640 1 --2- 172.21.15.1:0/4265517288 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9f7c06b500 0x7f9f7c06d9c0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.729+0000 7f9f957b6640 1 --2- 172.21.15.1:0/4265517288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f901ebba0 0x7f9f90151110 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.729+0000 7f9f957b6640 1 --2- 172.21.15.1:0/4265517288 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9f90152650 0x7f9f90156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.729+0000 7f9f957b6640 1 --2- 172.21.15.1:0/4265517288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f900bf9d0 0x7f9f901543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.729+0000 7f9f957b6640 1 -- 172.21.15.1:0/4265517288 >> 172.21.15.1:0/4265517288 conn(0x7f9f90066ee0 msgr2=0x7f9f900672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.729+0000 7f9f957b6640 1 -- 172.21.15.1:0/4265517288 shutdown_connections 2024-03-06T07:30:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:05.729+0000 7f9f957b6640 1 -- 172.21.15.1:0/4265517288 wait complete. 2024-03-06T07:30:06.144 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:44.306981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:06.145 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:stopping 2024-03-06T07:30:07.146 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:07.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.278+0000 7f215b665640 1 Processor -- start 2024-03-06T07:30:07.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.278+0000 7f215b665640 1 -- start start 2024-03-06T07:30:07.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.279+0000 7f215b665640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2154152840 0x7f21540c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.279+0000 7f215b665640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21540c60b0 0x7f21541553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.279+0000 7f215b665640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21540c5e40 con 0x7f21540c3970 2024-03-06T07:30:07.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.279+0000 7f215b665640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21540c67f0 con 0x7f2154152840 2024-03-06T07:30:07.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.279+0000 7f215b665640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21540c6a40 con 0x7f21540c60b0 2024-03-06T07:30:07.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.279+0000 7f2158c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2154152840 0x7f21540c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.279+0000 7f2159401640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f21540c3970 0x7f2154152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:42310/0 (socket says 172.21.15.1:42310) 2024-03-06T07:30:07.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.279+0000 7f2158c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2154152840 0x7f21540c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:42768/0 (socket says 172.21.15.1:42768) 2024-03-06T07:30:07.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.279+0000 7f2159401640 1 -- 172.21.15.1:0/59926705 learned_addr learned my addr 172.21.15.1:0/59926705 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:07.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.279+0000 7f2159c02640 1 --2- 172.21.15.1:0/59926705 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21540c60b0 0x7f21541553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.279+0000 7f2158c00640 1 -- 172.21.15.1:0/59926705 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21540c60b0 msgr2=0x7f21541553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:07.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.279+0000 7f2158c00640 1 --2- 172.21.15.1:0/59926705 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21540c60b0 0x7f21541553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.279+0000 7f2158c00640 1 -- 172.21.15.1:0/59926705 >> v1:172.21.15.98:6789/0 conn(0x7f21540c3970 legacy=0x7f2154152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.279+0000 7f2158c00640 1 -- 172.21.15.1:0/59926705 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21540c6d20 con 0x7f2154152840 2024-03-06T07:30:07.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.280+0000 7f2158c00640 1 --2- 172.21.15.1:0/59926705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2154152840 0x7f21540c8170 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f21440097a0 tx=0x7f214402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d927c1a81dc5fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.280+0000 7f214bfff640 1 -- 172.21.15.1:0/59926705 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2144035020 con 0x7f2154152840 2024-03-06T07:30:07.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.280+0000 7f214bfff640 1 -- 172.21.15.1:0/59926705 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2144031830 con 0x7f2154152840 2024-03-06T07:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.280+0000 7f214bfff640 1 -- 172.21.15.1:0/59926705 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2144031c00 con 0x7f2154152840 2024-03-06T07:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.281+0000 7f215b665640 1 -- 172.21.15.1:0/59926705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2154152840 msgr2=0x7f21540c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.281+0000 7f215b665640 1 --2- 172.21.15.1:0/59926705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2154152840 0x7f21540c8170 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f21440097a0 tx=0x7f214402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.281+0000 7f215b665640 1 -- 172.21.15.1:0/59926705 shutdown_connections 2024-03-06T07:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.281+0000 7f215b665640 1 --2- 172.21.15.1:0/59926705 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21540c60b0 0x7f21541553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.281+0000 7f215b665640 1 --2- 172.21.15.1:0/59926705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2154152840 0x7f21540c8170 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.281+0000 7f215b665640 1 -- 172.21.15.1:0/59926705 >> 172.21.15.1:0/59926705 conn(0x7f21540c88f0 msgr2=0x7f21540c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.281+0000 7f215b665640 1 -- 172.21.15.1:0/59926705 shutdown_connections 2024-03-06T07:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.281+0000 7f215b665640 1 -- 172.21.15.1:0/59926705 wait complete. 2024-03-06T07:30:07.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.282+0000 7f215b665640 1 Processor -- start 2024-03-06T07:30:07.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.282+0000 7f215b665640 1 -- start start 2024-03-06T07:30:07.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f215b665640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21540c3970 0x7f21540a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f215b665640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21540c60b0 0x7f2154152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f215b665640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2154152840 0x7f2154153470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f215b665640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21540c75c0 con 0x7f2154152840 2024-03-06T07:30:07.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f215b665640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21540c72e0 con 0x7f21540c3970 2024-03-06T07:30:07.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f215b665640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21540c3d50 con 0x7f21540c60b0 2024-03-06T07:30:07.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f2159c02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2154152840 0x7f2154153470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f2159c02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2154152840 0x7f2154153470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57700/0 (socket says 172.21.15.1:57700) 2024-03-06T07:30:07.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f2159c02640 1 -- 172.21.15.1:0/4023226014 learned_addr learned my addr 172.21.15.1:0/4023226014 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:07.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f2158c00640 1 --2- 172.21.15.1:0/4023226014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21540c60b0 0x7f2154152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f2159401640 1 --2- 172.21.15.1:0/4023226014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21540c3970 0x7f21540a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f2159c02640 1 -- 172.21.15.1:0/4023226014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21540c60b0 msgr2=0x7f2154152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f2159c02640 1 --2- 172.21.15.1:0/4023226014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21540c60b0 0x7f2154152f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f2159c02640 1 -- 172.21.15.1:0/4023226014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21540c3970 msgr2=0x7f21540a97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f2159c02640 1 --2- 172.21.15.1:0/4023226014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21540c3970 0x7f21540a97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f2159c02640 1 -- 172.21.15.1:0/4023226014 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21541539b0 con 0x7f2154152840 2024-03-06T07:30:07.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f2158c00640 1 --2- 172.21.15.1:0/4023226014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21540c60b0 0x7f2154152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:07.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.283+0000 7f2159401640 1 --2- 172.21.15.1:0/4023226014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21540c3970 0x7f21540a97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:07.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.284+0000 7f2159c02640 1 --2- 172.21.15.1:0/4023226014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2154152840 0x7f2154153470 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f215000efc0 tx=0x7f215000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.284+0000 7f2149ffb640 1 -- 172.21.15.1:0/4023226014 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2150018020 con 0x7f2154152840 2024-03-06T07:30:07.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.284+0000 7f215b665640 1 -- 172.21.15.1:0/4023226014 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215414d750 con 0x7f2154152840 2024-03-06T07:30:07.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.284+0000 7f2149ffb640 1 -- 172.21.15.1:0/4023226014 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2150014480 con 0x7f2154152840 2024-03-06T07:30:07.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.284+0000 7f2149ffb640 1 -- 172.21.15.1:0/4023226014 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2150014870 con 0x7f2154152840 2024-03-06T07:30:07.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.284+0000 7f215b665640 1 -- 172.21.15.1:0/4023226014 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21541edfd0 con 0x7f2154152840 2024-03-06T07:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.285+0000 7f215b665640 1 -- 172.21.15.1:0/4023226014 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21540aec90 con 0x7f2154152840 2024-03-06T07:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.285+0000 7f2149ffb640 1 -- 172.21.15.1:0/4023226014 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2150005d10 con 0x7f2154152840 2024-03-06T07:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.286+0000 7f2149ffb640 1 --2- 172.21.15.1:0/4023226014 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f213806b4d0 0x7f213806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.286+0000 7f2149ffb640 1 -- 172.21.15.1:0/4023226014 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f215008b290 con 0x7f2154152840 2024-03-06T07:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.286+0000 7f2159401640 1 --2- 172.21.15.1:0/4023226014 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f213806b4d0 0x7f213806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.287+0000 7f2159401640 1 --2- 172.21.15.1:0/4023226014 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f213806b4d0 0x7f213806d990 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f21540c5890 tx=0x7f214c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.290+0000 7f2149ffb640 1 -- 172.21.15.1:0/4023226014 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21540aec90 con 0x7f2154152840 2024-03-06T07:30:07.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.511+0000 7f215b665640 1 -- 172.21.15.1:0/4023226014 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21540686e0 con 0x7f2154152840 2024-03-06T07:30:07.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.512+0000 7f2149ffb640 1 -- 172.21.15.1:0/4023226014 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5005 (secure 0 0 0) 0x7f2150022030 con 0x7f2154152840 2024-03-06T07:30:07.514 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:07.514 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:44.306981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:07.514 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 200 2024-03-06T07:30:07.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.514+0000 7f215b665640 1 -- 172.21.15.1:0/4023226014 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f213806b4d0 msgr2=0x7f213806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.514+0000 7f215b665640 1 --2- 172.21.15.1:0/4023226014 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f213806b4d0 0x7f213806d990 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f21540c5890 tx=0x7f214c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:07.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.514+0000 7f215b665640 1 -- 172.21.15.1:0/4023226014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2154152840 msgr2=0x7f2154153470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.514+0000 7f215b665640 1 --2- 172.21.15.1:0/4023226014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2154152840 0x7f2154153470 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f215000efc0 tx=0x7f215000c490 comp rx=0 tx=0).stop 2024-03-06T07:30:07.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.514+0000 7f215b665640 1 -- 172.21.15.1:0/4023226014 shutdown_connections 2024-03-06T07:30:07.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.514+0000 7f215b665640 1 --2- 172.21.15.1:0/4023226014 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f213806b4d0 0x7f213806d990 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.514+0000 7f215b665640 1 --2- 172.21.15.1:0/4023226014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2154152840 0x7f2154153470 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.514+0000 7f215b665640 1 --2- 172.21.15.1:0/4023226014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21540c60b0 0x7f2154152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.514+0000 7f215b665640 1 --2- 172.21.15.1:0/4023226014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21540c3970 0x7f21540a97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.514+0000 7f215b665640 1 -- 172.21.15.1:0/4023226014 >> 172.21.15.1:0/4023226014 conn(0x7f21540c88f0 msgr2=0x7f21541506b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.514+0000 7f215b665640 1 -- 172.21.15.1:0/4023226014 shutdown_connections 2024-03-06T07:30:07.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:07.514+0000 7f215b665640 1 -- 172.21.15.1:0/4023226014 wait complete. 2024-03-06T07:30:07.933 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:44.306981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:07.933 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:stopping 2024-03-06T07:30:08.934 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:09.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.034+0000 7fabe61c7640 1 Processor -- start 2024-03-06T07:30:09.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.034+0000 7fabe61c7640 1 -- start start 2024-03-06T07:30:09.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.035+0000 7fabe61c7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabe0152650 0x7fabe0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.035+0000 7fabe61c7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fabe00c6b10 0x7fabe0158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.035+0000 7fabe61c7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fabe00bf9d0 0x7fabe00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.035+0000 7fabe61c7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabe0155800 con 0x7fabe00c6b10 2024-03-06T07:30:09.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.035+0000 7fabe61c7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabe00c2fc0 con 0x7fabe00bf9d0 2024-03-06T07:30:09.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.035+0000 7fabe61c7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe00c6690 con 0x7fabe0152650 2024-03-06T07:30:09.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.035+0000 7fabdeffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fabe00c6b10 0x7fabe0158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.035+0000 7fabdeffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fabe00c6b10 0x7fabe0158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57710/0 (socket says 172.21.15.1:57710) 2024-03-06T07:30:09.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.035+0000 7fabdeffd640 1 -- 172.21.15.1:0/109313332 learned_addr learned my addr 172.21.15.1:0/109313332 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:09.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.036+0000 7fabdffff640 1 --2- 172.21.15.1:0/109313332 >> v2:172.21.15.98:3300/0 conn(0x7fabe00bf9d0 0x7fabe00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.036+0000 7fabdf7fe640 1 --2- 172.21.15.1:0/109313332 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabe0152650 0x7fabe0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.036+0000 7fabdeffd640 1 -- 172.21.15.1:0/109313332 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabe0152650 msgr2=0x7fabe0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.036+0000 7fabdeffd640 1 --2- 172.21.15.1:0/109313332 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabe0152650 0x7fabe0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.036+0000 7fabdeffd640 1 -- 172.21.15.1:0/109313332 >> v2:172.21.15.98:3300/0 conn(0x7fabe00bf9d0 msgr2=0x7fabe00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.036+0000 7fabdeffd640 1 --2- 172.21.15.1:0/109313332 >> v2:172.21.15.98:3300/0 conn(0x7fabe00bf9d0 0x7fabe00c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.036+0000 7fabdeffd640 1 -- 172.21.15.1:0/109313332 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe00c7280 con 0x7fabe00c6b10 2024-03-06T07:30:09.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.036+0000 7fabdeffd640 1 --2- 172.21.15.1:0/109313332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe00c6b10 0x7fabe0158690 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7fabe01597b0 tx=0x7fabd402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815cc36d5cd394be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.037+0000 7fabde7fc640 1 -- 172.21.15.1:0/109313332 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd402e070 con 0x7fabe00c6b10 2024-03-06T07:30:09.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.037+0000 7fabde7fc640 1 -- 172.21.15.1:0/109313332 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd4037070 con 0x7fabe00c6b10 2024-03-06T07:30:09.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.037+0000 7fabde7fc640 1 -- 172.21.15.1:0/109313332 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd4031a00 con 0x7fabe00c6b10 2024-03-06T07:30:09.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.037+0000 7fabe61c7640 1 -- 172.21.15.1:0/109313332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe00c6b10 msgr2=0x7fabe0158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.037+0000 7fabe61c7640 1 --2- 172.21.15.1:0/109313332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe00c6b10 0x7fabe0158690 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7fabe01597b0 tx=0x7fabd402b880 comp rx=0 tx=0).stop 2024-03-06T07:30:09.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.037+0000 7fabe61c7640 1 -- 172.21.15.1:0/109313332 shutdown_connections 2024-03-06T07:30:09.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.037+0000 7fabe61c7640 1 --2- 172.21.15.1:0/109313332 >> v2:172.21.15.98:3300/0 conn(0x7fabe00bf9d0 0x7fabe00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.037+0000 7fabe61c7640 1 --2- 172.21.15.1:0/109313332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe00c6b10 0x7fabe0158690 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.037+0000 7fabe61c7640 1 --2- 172.21.15.1:0/109313332 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabe0152650 0x7fabe0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.037+0000 7fabe61c7640 1 -- 172.21.15.1:0/109313332 >> 172.21.15.1:0/109313332 conn(0x7fabe0066ee0 msgr2=0x7fabe00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.037+0000 7fabe61c7640 1 -- 172.21.15.1:0/109313332 shutdown_connections 2024-03-06T07:30:09.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.037+0000 7fabe61c7640 1 -- 172.21.15.1:0/109313332 wait complete. 2024-03-06T07:30:09.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.039+0000 7fabe61c7640 1 Processor -- start 2024-03-06T07:30:09.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.039+0000 7fabe61c7640 1 -- start start 2024-03-06T07:30:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.040+0000 7fabe61c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe00bf9d0 0x7fabe0152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.040+0000 7fabe61c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabe0152650 0x7fabe014e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.040+0000 7fabe61c7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabe0154d70 0x7fabe014c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.040+0000 7fabe61c7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe0155490 con 0x7fabe00bf9d0 2024-03-06T07:30:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.040+0000 7fabe61c7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe01551b0 con 0x7fabe0152650 2024-03-06T07:30:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.040+0000 7fabe61c7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe0152a30 con 0x7fabe0154d70 2024-03-06T07:30:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.040+0000 7fabdf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe00bf9d0 0x7fabe0152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.040+0000 7fabdf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe00bf9d0 0x7fabe0152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57718/0 (socket says 172.21.15.1:57718) 2024-03-06T07:30:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.040+0000 7fabdf7fe640 1 -- 172.21.15.1:0/1482767279 learned_addr learned my addr 172.21.15.1:0/1482767279 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.040+0000 7fabdeffd640 1 --2- 172.21.15.1:0/1482767279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabe0152650 0x7fabe014e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.040+0000 7fabdf7fe640 1 -- 172.21.15.1:0/1482767279 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabe0154d70 msgr2=0x7fabe014c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.040+0000 7fabdffff640 1 --2- 172.21.15.1:0/1482767279 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabe0154d70 0x7fabe014c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.040+0000 7fabdf7fe640 1 --2- 172.21.15.1:0/1482767279 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabe0154d70 0x7fabe014c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.040+0000 7fabdf7fe640 1 -- 172.21.15.1:0/1482767279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabe0152650 msgr2=0x7fabe014e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.040+0000 7fabdf7fe640 1 --2- 172.21.15.1:0/1482767279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabe0152650 0x7fabe014e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.040+0000 7fabdf7fe640 1 -- 172.21.15.1:0/1482767279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe0154890 con 0x7fabe00bf9d0 2024-03-06T07:30:09.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.040+0000 7fabdeffd640 1 --2- 172.21.15.1:0/1482767279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabe0152650 0x7fabe014e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:09.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.041+0000 7fabdf7fe640 1 --2- 172.21.15.1:0/1482767279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe00bf9d0 0x7fabe0152f10 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7fabd000d830 tx=0x7fabd000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.041+0000 7fabc7fff640 1 -- 172.21.15.1:0/1482767279 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd000b6c0 con 0x7fabe00bf9d0 2024-03-06T07:30:09.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.041+0000 7fabc7fff640 1 -- 172.21.15.1:0/1482767279 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd0019070 con 0x7fabe00bf9d0 2024-03-06T07:30:09.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.041+0000 7fabe61c7640 1 -- 172.21.15.1:0/1482767279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe014db20 con 0x7fabe00bf9d0 2024-03-06T07:30:09.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.041+0000 7fabc7fff640 1 -- 172.21.15.1:0/1482767279 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd000ba80 con 0x7fabe00bf9d0 2024-03-06T07:30:09.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.041+0000 7fabe61c7640 1 -- 172.21.15.1:0/1482767279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe01ebda0 con 0x7fabe00bf9d0 2024-03-06T07:30:09.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.042+0000 7fabe61c7640 1 -- 172.21.15.1:0/1482767279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe0067fd0 con 0x7fabe00bf9d0 2024-03-06T07:30:09.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.042+0000 7fabc7fff640 1 -- 172.21.15.1:0/1482767279 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fabd001e460 con 0x7fabe00bf9d0 2024-03-06T07:30:09.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.043+0000 7fabc7fff640 1 --2- 172.21.15.1:0/1482767279 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fabcc06b500 0x7fabcc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.043+0000 7fabc7fff640 1 -- 172.21.15.1:0/1482767279 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fabd008aa60 con 0x7fabe00bf9d0 2024-03-06T07:30:09.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.043+0000 7fabdeffd640 1 --2- 172.21.15.1:0/1482767279 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fabcc06b500 0x7fabcc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.043+0000 7fabdeffd640 1 --2- 172.21.15.1:0/1482767279 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fabcc06b500 0x7fabcc06d9c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fabd4036000 tx=0x7fabd4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.051+0000 7fabc7fff640 1 -- 172.21.15.1:0/1482767279 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fabd0059740 con 0x7fabe00bf9d0 2024-03-06T07:30:09.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.270+0000 7fabe61c7640 1 -- 172.21.15.1:0/1482767279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabe00b06f0 con 0x7fabe00bf9d0 2024-03-06T07:30:09.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.270+0000 7fabc7fff640 1 -- 172.21.15.1:0/1482767279 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5005 (secure 0 0 0) 0x7fabd0059040 con 0x7fabe00bf9d0 2024-03-06T07:30:09.273 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:09.273 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:44.306981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:09.273 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 200 2024-03-06T07:30:09.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.272+0000 7fabe61c7640 1 -- 172.21.15.1:0/1482767279 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fabcc06b500 msgr2=0x7fabcc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.272+0000 7fabe61c7640 1 --2- 172.21.15.1:0/1482767279 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fabcc06b500 0x7fabcc06d9c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fabd4036000 tx=0x7fabd4008030 comp rx=0 tx=0).stop 2024-03-06T07:30:09.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.272+0000 7fabe61c7640 1 -- 172.21.15.1:0/1482767279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe00bf9d0 msgr2=0x7fabe0152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.272+0000 7fabe61c7640 1 --2- 172.21.15.1:0/1482767279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe00bf9d0 0x7fabe0152f10 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7fabd000d830 tx=0x7fabd000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.273+0000 7fabe61c7640 1 -- 172.21.15.1:0/1482767279 shutdown_connections 2024-03-06T07:30:09.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.273+0000 7fabe61c7640 1 --2- 172.21.15.1:0/1482767279 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fabcc06b500 0x7fabcc06d9c0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.273+0000 7fabe61c7640 1 --2- 172.21.15.1:0/1482767279 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabe0154d70 0x7fabe014c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.273+0000 7fabe61c7640 1 --2- 172.21.15.1:0/1482767279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabe0152650 0x7fabe014e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.273+0000 7fabe61c7640 1 --2- 172.21.15.1:0/1482767279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe00bf9d0 0x7fabe0152f10 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.273+0000 7fabe61c7640 1 -- 172.21.15.1:0/1482767279 >> 172.21.15.1:0/1482767279 conn(0x7fabe0066ee0 msgr2=0x7fabe00c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.273+0000 7fabe61c7640 1 -- 172.21.15.1:0/1482767279 shutdown_connections 2024-03-06T07:30:09.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:09.273+0000 7fabe61c7640 1 -- 172.21.15.1:0/1482767279 wait complete. 2024-03-06T07:30:09.691 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:44.306981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:09.691 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:stopping 2024-03-06T07:30:10.692 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:10.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.792+0000 7faf3cd5e640 1 Processor -- start 2024-03-06T07:30:10.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.792+0000 7faf3cd5e640 1 -- start start 2024-03-06T07:30:10.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.793+0000 7faf3cd5e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf38150520 0x7faf381543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.793+0000 7faf3cd5e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faf38070c80 0x7faf380710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.793+0000 7faf3cd5e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf38150900 con 0x7faf380c32f0 2024-03-06T07:30:10.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.793+0000 7faf3cd5e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf38153ed0 con 0x7faf38070c80 2024-03-06T07:30:10.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.793+0000 7faf3cd5e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf38071790 con 0x7faf38150520 2024-03-06T07:30:10.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.793+0000 7faf35d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7faf380c32f0 0x7faf381537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34074/0 (socket says 172.21.15.1:34074) 2024-03-06T07:30:10.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.793+0000 7faf35d9b640 1 -- 172.21.15.1:0/4218698893 learned_addr learned my addr 172.21.15.1:0/4218698893 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:10.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3659c640 1 --2- 172.21.15.1:0/4218698893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf38150520 0x7faf381543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf36d9d640 1 --2- 172.21.15.1:0/4218698893 >> v2:172.21.15.98:3300/0 conn(0x7faf38070c80 0x7faf380710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3559a640 1 -- 172.21.15.1:0/4218698893 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148639177 0 0) 0x7faf38150900 con 0x7faf380c32f0 2024-03-06T07:30:10.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3559a640 1 -- 172.21.15.1:0/4218698893 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf24003520 con 0x7faf380c32f0 2024-03-06T07:30:10.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3559a640 1 -- 172.21.15.1:0/4218698893 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3378956567 0 0) 0x7faf24003520 con 0x7faf380c32f0 2024-03-06T07:30:10.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3559a640 1 -- 172.21.15.1:0/4218698893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf38150520 msgr2=0x7faf381543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3559a640 1 --2- 172.21.15.1:0/4218698893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf38150520 0x7faf381543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3559a640 1 -- 172.21.15.1:0/4218698893 >> v2:172.21.15.98:3300/0 conn(0x7faf38070c80 msgr2=0x7faf380710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3559a640 1 --2- 172.21.15.1:0/4218698893 >> v2:172.21.15.98:3300/0 conn(0x7faf38070c80 0x7faf380710f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3559a640 1 -- 172.21.15.1:0/4218698893 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf38157050 con 0x7faf380c32f0 2024-03-06T07:30:10.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3559a640 1 -- 172.21.15.1:0/4218698893 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7faf28001670 con 0x7faf380c32f0 2024-03-06T07:30:10.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3559a640 1 -- 172.21.15.1:0/4218698893 >> v1:172.21.15.1:6789/0 conn(0x7faf380c32f0 legacy=0x7faf381537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3559a640 1 --2- 172.21.15.1:0/4218698893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf240039b0 0x7faf24003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3559a640 1 --2- 172.21.15.1:0/4218698893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf38150520 0x7faf24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3559a640 1 --2- 172.21.15.1:0/4218698893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf24004d40 0x7faf240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3559a640 1 -- 172.21.15.1:0/4218698893 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf38153ed0 con 0x7faf240039b0 2024-03-06T07:30:10.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3559a640 1 -- 172.21.15.1:0/4218698893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf38071790 con 0x7faf24004d40 2024-03-06T07:30:10.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3559a640 1 -- 172.21.15.1:0/4218698893 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2400b9c0 con 0x7faf38150520 2024-03-06T07:30:10.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf3659c640 1 --2- 172.21.15.1:0/4218698893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf240039b0 0x7faf24003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf35d9b640 1 --2- 172.21.15.1:0/4218698893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf38150520 0x7faf24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.794+0000 7faf36d9d640 1 --2- 172.21.15.1:0/4218698893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf24004d40 0x7faf240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.795+0000 7faf3659c640 1 -- 172.21.15.1:0/4218698893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf38150520 msgr2=0x7faf24004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.795+0000 7faf3659c640 1 --2- 172.21.15.1:0/4218698893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf38150520 0x7faf24004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.795+0000 7faf3659c640 1 -- 172.21.15.1:0/4218698893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf24004d40 msgr2=0x7faf240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.795+0000 7faf3659c640 1 --2- 172.21.15.1:0/4218698893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf24004d40 0x7faf240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.795+0000 7faf3659c640 1 -- 172.21.15.1:0/4218698893 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf24003520 con 0x7faf240039b0 2024-03-06T07:30:10.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.795+0000 7faf36d9d640 1 --2- 172.21.15.1:0/4218698893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf24004d40 0x7faf240051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:10.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.795+0000 7faf3659c640 1 --2- 172.21.15.1:0/4218698893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf240039b0 0x7faf24003dd0 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7faf20009ac0 tx=0x7faf24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6676d1876b929904 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.795+0000 7faf3559a640 1 -- 172.21.15.1:0/4218698893 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf20039070 con 0x7faf240039b0 2024-03-06T07:30:10.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.795+0000 7faf3559a640 1 -- 172.21.15.1:0/4218698893 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2002c6b0 con 0x7faf240039b0 2024-03-06T07:30:10.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.795+0000 7faf3559a640 1 -- 172.21.15.1:0/4218698893 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf20009690 con 0x7faf240039b0 2024-03-06T07:30:10.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.796+0000 7faf3cd5e640 1 -- 172.21.15.1:0/4218698893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf240039b0 msgr2=0x7faf24003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.796+0000 7faf3cd5e640 1 --2- 172.21.15.1:0/4218698893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf240039b0 0x7faf24003dd0 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7faf20009ac0 tx=0x7faf24005830 comp rx=0 tx=0).stop 2024-03-06T07:30:10.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.796+0000 7faf36d9d640 1 -- 172.21.15.1:0/4218698893 reap_dead start 2024-03-06T07:30:10.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.796+0000 7faf3cd5e640 1 -- 172.21.15.1:0/4218698893 shutdown_connections 2024-03-06T07:30:10.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.796+0000 7faf3cd5e640 1 -- 172.21.15.1:0/4218698893 >> 172.21.15.1:0/4218698893 conn(0x7faf380c88f0 msgr2=0x7faf380c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.796+0000 7faf3cd5e640 1 -- 172.21.15.1:0/4218698893 shutdown_connections 2024-03-06T07:30:10.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.796+0000 7faf3cd5e640 1 -- 172.21.15.1:0/4218698893 wait complete. 2024-03-06T07:30:10.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.798+0000 7faf3cd5e640 1 Processor -- start 2024-03-06T07:30:10.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.798+0000 7faf3cd5e640 1 -- start start 2024-03-06T07:30:10.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.799+0000 7faf3cd5e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf240039b0 0x7faf381e9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.799+0000 7faf3cd5e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf381ea3d0 0x7faf380ac4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.799+0000 7faf3cd5e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf380a8510 0x7faf380ac9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.799+0000 7faf3cd5e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf38157330 con 0x7faf381ea3d0 2024-03-06T07:30:10.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.799+0000 7faf3cd5e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf381532a0 con 0x7faf380a8510 2024-03-06T07:30:10.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.799+0000 7faf3cd5e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf380c7ec0 con 0x7faf240039b0 2024-03-06T07:30:10.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.799+0000 7faf35d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf381ea3d0 0x7faf380ac4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.799+0000 7faf35d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf381ea3d0 0x7faf380ac4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57744/0 (socket says 172.21.15.1:57744) 2024-03-06T07:30:10.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.799+0000 7faf35d9b640 1 -- 172.21.15.1:0/2720161099 learned_addr learned my addr 172.21.15.1:0/2720161099 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:10.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.799+0000 7faf3659c640 1 --2- 172.21.15.1:0/2720161099 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf240039b0 0x7faf381e9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.799+0000 7faf36d9d640 1 --2- 172.21.15.1:0/2720161099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf380a8510 0x7faf380ac9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.800+0000 7faf35d9b640 1 -- 172.21.15.1:0/2720161099 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf240039b0 msgr2=0x7faf381e9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.800+0000 7faf35d9b640 1 --2- 172.21.15.1:0/2720161099 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf240039b0 0x7faf381e9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.800+0000 7faf35d9b640 1 -- 172.21.15.1:0/2720161099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf380a8510 msgr2=0x7faf380ac9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.800+0000 7faf35d9b640 1 --2- 172.21.15.1:0/2720161099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf380a8510 0x7faf380ac9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.800+0000 7faf35d9b640 1 -- 172.21.15.1:0/2720161099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf381ea880 con 0x7faf381ea3d0 2024-03-06T07:30:10.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.800+0000 7faf36d9d640 1 --2- 172.21.15.1:0/2720161099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf380a8510 0x7faf380ac9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.800+0000 7faf35d9b640 1 --2- 172.21.15.1:0/2720161099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf381ea3d0 0x7faf380ac4a0 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7faf28000c00 tx=0x7faf2800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.800+0000 7faf2effd640 1 -- 172.21.15.1:0/2720161099 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2800c870 con 0x7faf381ea3d0 2024-03-06T07:30:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.800+0000 7faf2effd640 1 -- 172.21.15.1:0/2720161099 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf28018070 con 0x7faf381ea3d0 2024-03-06T07:30:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.800+0000 7faf3cd5e640 1 -- 172.21.15.1:0/2720161099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf38149660 con 0x7faf381ea3d0 2024-03-06T07:30:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.800+0000 7faf2effd640 1 -- 172.21.15.1:0/2720161099 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2800cc30 con 0x7faf381ea3d0 2024-03-06T07:30:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.800+0000 7faf3cd5e640 1 -- 172.21.15.1:0/2720161099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf380ad1a0 con 0x7faf381ea3d0 2024-03-06T07:30:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.801+0000 7faf3cd5e640 1 -- 172.21.15.1:0/2720161099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf380aec90 con 0x7faf381ea3d0 2024-03-06T07:30:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.801+0000 7faf2effd640 1 -- 172.21.15.1:0/2720161099 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf28013420 con 0x7faf381ea3d0 2024-03-06T07:30:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.802+0000 7faf2effd640 1 --2- 172.21.15.1:0/2720161099 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faf1806b500 0x7faf1806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.802+0000 7faf2effd640 1 -- 172.21.15.1:0/2720161099 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6188+0+0 (secure 0 0 0) 0x7faf28089de0 con 0x7faf381ea3d0 2024-03-06T07:30:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.802+0000 7faf3659c640 1 --2- 172.21.15.1:0/2720161099 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faf1806b500 0x7faf1806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.803+0000 7faf3659c640 1 --2- 172.21.15.1:0/2720161099 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faf1806b500 0x7faf1806d9c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7faf20036000 tx=0x7faf20030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:10.806+0000 7faf2effd640 1 -- 172.21.15.1:0/2720161099 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf380aec90 con 0x7faf381ea3d0 2024-03-06T07:30:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:11.024+0000 7faf3cd5e640 1 -- 172.21.15.1:0/2720161099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf3806f4e0 con 0x7faf381ea3d0 2024-03-06T07:30:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:11.025+0000 7faf2effd640 1 -- 172.21.15.1:0/2720161099 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5005 (secure 0 0 0) 0x7faf3806f4e0 con 0x7faf381ea3d0 2024-03-06T07:30:11.027 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:11.027 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:44.306981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 200 2024-03-06T07:30:11.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:11.027+0000 7faf3cd5e640 1 -- 172.21.15.1:0/2720161099 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faf1806b500 msgr2=0x7faf1806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:11.027+0000 7faf3cd5e640 1 --2- 172.21.15.1:0/2720161099 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faf1806b500 0x7faf1806d9c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7faf20036000 tx=0x7faf20030000 comp rx=0 tx=0).stop 2024-03-06T07:30:11.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:11.027+0000 7faf3cd5e640 1 -- 172.21.15.1:0/2720161099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf381ea3d0 msgr2=0x7faf380ac4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:11.027+0000 7faf3cd5e640 1 --2- 172.21.15.1:0/2720161099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf381ea3d0 0x7faf380ac4a0 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7faf28000c00 tx=0x7faf2800ecd0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:11.027+0000 7faf3cd5e640 1 -- 172.21.15.1:0/2720161099 shutdown_connections 2024-03-06T07:30:11.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:11.027+0000 7faf3cd5e640 1 --2- 172.21.15.1:0/2720161099 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faf1806b500 0x7faf1806d9c0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:11.027+0000 7faf3cd5e640 1 --2- 172.21.15.1:0/2720161099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf380a8510 0x7faf380ac9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:11.027+0000 7faf3cd5e640 1 --2- 172.21.15.1:0/2720161099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf381ea3d0 0x7faf380ac4a0 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:11.027+0000 7faf3cd5e640 1 --2- 172.21.15.1:0/2720161099 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faf240039b0 0x7faf381e9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:11.027+0000 7faf3cd5e640 1 -- 172.21.15.1:0/2720161099 >> 172.21.15.1:0/2720161099 conn(0x7faf380c88f0 msgr2=0x7faf38150d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:11.027+0000 7faf3cd5e640 1 -- 172.21.15.1:0/2720161099 shutdown_connections 2024-03-06T07:30:11.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:11.027+0000 7faf3cd5e640 1 -- 172.21.15.1:0/2720161099 wait complete. 2024-03-06T07:30:11.443 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:44.306981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:11.443 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:stopping 2024-03-06T07:30:12.444 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:12.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.567+0000 7fcec604e640 1 Processor -- start 2024-03-06T07:30:12.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.567+0000 7fcec604e640 1 -- start start 2024-03-06T07:30:12.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcec604e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcec00c8cd0 0x7fcec00c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcec604e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec00c2b90 con 0x7fcec00c3260 2024-03-06T07:30:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcec604e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec0156020 con 0x7fcec0155b70 2024-03-06T07:30:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcec604e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec00c37b0 con 0x7fcec00c8cd0 2024-03-06T07:30:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebeffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcec00c3260 0x7fcec00c7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34090/0 (socket says 172.21.15.1:34090) 2024-03-06T07:30:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebeffd640 1 -- 172.21.15.1:0/374685082 learned_addr learned my addr 172.21.15.1:0/374685082 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebe7fc640 1 -- 172.21.15.1:0/374685082 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770382752 0 0) 0x7fcec00c2b90 con 0x7fcec00c3260 2024-03-06T07:30:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebe7fc640 1 -- 172.21.15.1:0/374685082 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea8003520 con 0x7fcec00c3260 2024-03-06T07:30:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebf7fe640 1 --2- 172.21.15.1:0/374685082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcec00c8cd0 0x7fcec00c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebe7fc640 1 -- 172.21.15.1:0/374685082 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3420038653 0 0) 0x7fcea8003520 con 0x7fcec00c3260 2024-03-06T07:30:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebe7fc640 1 -- 172.21.15.1:0/374685082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcec00c8cd0 msgr2=0x7fcec00c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebe7fc640 1 --2- 172.21.15.1:0/374685082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcec00c8cd0 0x7fcec00c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebe7fc640 1 -- 172.21.15.1:0/374685082 >> v1:172.21.15.98:6789/0 conn(0x7fcec0155b70 legacy=0x7fcec0159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebe7fc640 1 -- 172.21.15.1:0/374685082 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec00c3a90 con 0x7fcec00c3260 2024-03-06T07:30:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebe7fc640 1 -- 172.21.15.1:0/374685082 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fceb4001670 con 0x7fcec00c3260 2024-03-06T07:30:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebe7fc640 1 -- 172.21.15.1:0/374685082 >> v1:172.21.15.1:6789/0 conn(0x7fcec00c3260 legacy=0x7fcec00c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebe7fc640 1 --2- 172.21.15.1:0/374685082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcec00c8cd0 0x7fcea80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebe7fc640 1 --2- 172.21.15.1:0/374685082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea8003f60 0x7fcea8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebe7fc640 1 --2- 172.21.15.1:0/374685082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea8004d40 0x7fcea80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebe7fc640 1 -- 172.21.15.1:0/374685082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec00c37b0 con 0x7fcea8003f60 2024-03-06T07:30:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebe7fc640 1 -- 172.21.15.1:0/374685082 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea800bb50 con 0x7fcea8004d40 2024-03-06T07:30:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.568+0000 7fcebe7fc640 1 -- 172.21.15.1:0/374685082 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea800bd50 con 0x7fcec00c8cd0 2024-03-06T07:30:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.569+0000 7fcebeffd640 1 --2- 172.21.15.1:0/374685082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea8003f60 0x7fcea8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.569+0000 7fcebf7fe640 1 --2- 172.21.15.1:0/374685082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcec00c8cd0 0x7fcea80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.569+0000 7fcebffff640 1 --2- 172.21.15.1:0/374685082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea8004d40 0x7fcea80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.569+0000 7fcebeffd640 1 -- 172.21.15.1:0/374685082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcec00c8cd0 msgr2=0x7fcea80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.569+0000 7fcebeffd640 1 --2- 172.21.15.1:0/374685082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcec00c8cd0 0x7fcea80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.569+0000 7fcebeffd640 1 -- 172.21.15.1:0/374685082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea8004d40 msgr2=0x7fcea80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.569+0000 7fcebeffd640 1 --2- 172.21.15.1:0/374685082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea8004d40 0x7fcea80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.569+0000 7fcebeffd640 1 -- 172.21.15.1:0/374685082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea8003520 con 0x7fcea8003f60 2024-03-06T07:30:12.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.569+0000 7fcebeffd640 1 --2- 172.21.15.1:0/374685082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea8003f60 0x7fcea8004800 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7fceb400c210 tx=0x7fcea8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea77a119fe08a228 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.570+0000 7fcebe7fc640 1 -- 172.21.15.1:0/374685082 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb4007d10 con 0x7fcea8003f60 2024-03-06T07:30:12.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.570+0000 7fcebe7fc640 1 -- 172.21.15.1:0/374685082 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb4030070 con 0x7fcea8003f60 2024-03-06T07:30:12.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.570+0000 7fcebe7fc640 1 -- 172.21.15.1:0/374685082 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb4005d50 con 0x7fcea8003f60 2024-03-06T07:30:12.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.570+0000 7fcec604e640 1 -- 172.21.15.1:0/374685082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea8003f60 msgr2=0x7fcea8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.570+0000 7fcec604e640 1 --2- 172.21.15.1:0/374685082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea8003f60 0x7fcea8004800 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7fceb400c210 tx=0x7fcea8005830 comp rx=0 tx=0).stop 2024-03-06T07:30:12.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.570+0000 7fcebffff640 1 -- 172.21.15.1:0/374685082 reap_dead start 2024-03-06T07:30:12.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.570+0000 7fcec604e640 1 -- 172.21.15.1:0/374685082 shutdown_connections 2024-03-06T07:30:12.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.570+0000 7fcec604e640 1 -- 172.21.15.1:0/374685082 >> 172.21.15.1:0/374685082 conn(0x7fcec006b3e0 msgr2=0x7fcec006b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.570+0000 7fcec604e640 1 -- 172.21.15.1:0/374685082 shutdown_connections 2024-03-06T07:30:12.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.570+0000 7fcec604e640 1 -- 172.21.15.1:0/374685082 wait complete. 2024-03-06T07:30:12.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.571+0000 7fcec604e640 1 Processor -- start 2024-03-06T07:30:12.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.572+0000 7fcec604e640 1 -- start start 2024-03-06T07:30:12.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.572+0000 7fcec604e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea8003f60 0x7fcec0152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.572+0000 7fcec604e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec01eed60 0x7fcec00acf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.572+0000 7fcec604e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcec00ab080 0x7fcec01eab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.572+0000 7fcec604e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec00c8560 con 0x7fcec01eed60 2024-03-06T07:30:12.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.572+0000 7fcec604e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec0158af0 con 0x7fcea8003f60 2024-03-06T07:30:12.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.572+0000 7fcec604e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec00c86e0 con 0x7fcec00ab080 2024-03-06T07:30:12.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.572+0000 7fcebeffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec01eed60 0x7fcec00acf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.572+0000 7fcebeffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec01eed60 0x7fcec00acf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57770/0 (socket says 172.21.15.1:57770) 2024-03-06T07:30:12.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.572+0000 7fcebeffd640 1 -- 172.21.15.1:0/2852407685 learned_addr learned my addr 172.21.15.1:0/2852407685 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:12.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.573+0000 7fcebffff640 1 --2- 172.21.15.1:0/2852407685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcec00ab080 0x7fcec01eab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.573+0000 7fcebf7fe640 1 --2- 172.21.15.1:0/2852407685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea8003f60 0x7fcec0152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.573+0000 7fcebeffd640 1 -- 172.21.15.1:0/2852407685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcec00ab080 msgr2=0x7fcec01eab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.573+0000 7fcebeffd640 1 --2- 172.21.15.1:0/2852407685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcec00ab080 0x7fcec01eab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.573+0000 7fcebeffd640 1 -- 172.21.15.1:0/2852407685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea8003f60 msgr2=0x7fcec0152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.573+0000 7fcebeffd640 1 --2- 172.21.15.1:0/2852407685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea8003f60 0x7fcec0152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.573+0000 7fcebeffd640 1 -- 172.21.15.1:0/2852407685 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec00aa710 con 0x7fcec01eed60 2024-03-06T07:30:12.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.573+0000 7fcebeffd640 1 --2- 172.21.15.1:0/2852407685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec01eed60 0x7fcec00acf60 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7fceb400c330 tx=0x7fcec01550f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.573+0000 7fcea7fff640 1 -- 172.21.15.1:0/2852407685 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb4045070 con 0x7fcec01eed60 2024-03-06T07:30:12.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.573+0000 7fcea7fff640 1 -- 172.21.15.1:0/2852407685 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb4004bf0 con 0x7fcec01eed60 2024-03-06T07:30:12.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.573+0000 7fcec604e640 1 -- 172.21.15.1:0/2852407685 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec01ee280 con 0x7fcec01eed60 2024-03-06T07:30:12.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.573+0000 7fcea7fff640 1 -- 172.21.15.1:0/2852407685 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb4007d10 con 0x7fcec01eed60 2024-03-06T07:30:12.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.573+0000 7fcec604e640 1 -- 172.21.15.1:0/2852407685 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec01ee840 con 0x7fcec01eed60 2024-03-06T07:30:12.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.574+0000 7fcec604e640 1 -- 172.21.15.1:0/2852407685 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec0067fd0 con 0x7fcec01eed60 2024-03-06T07:30:12.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.575+0000 7fcea7fff640 1 -- 172.21.15.1:0/2852407685 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fceb4007070 con 0x7fcec01eed60 2024-03-06T07:30:12.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.575+0000 7fcea7fff640 1 --2- 172.21.15.1:0/2852407685 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fceac06b500 0x7fceac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.575+0000 7fcea7fff640 1 -- 172.21.15.1:0/2852407685 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6188+0+0 (secure 0 0 0) 0x7fceb40ab5c0 con 0x7fcec01eed60 2024-03-06T07:30:12.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.575+0000 7fcebf7fe640 1 --2- 172.21.15.1:0/2852407685 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fceac06b500 0x7fceac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.576+0000 7fcebf7fe640 1 --2- 172.21.15.1:0/2852407685 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fceac06b500 0x7fceac06d9c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009b00 tx=0x7fceb0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.579+0000 7fcea7fff640 1 -- 172.21.15.1:0/2852407685 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fceb407a340 con 0x7fcec01eed60 2024-03-06T07:30:12.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.799+0000 7fcec604e640 1 -- 172.21.15.1:0/2852407685 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcec00c3fd0 con 0x7fcec01eed60 2024-03-06T07:30:12.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.799+0000 7fcea7fff640 1 -- 172.21.15.1:0/2852407685 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5005 (secure 0 0 0) 0x7fceb4079c40 con 0x7fcec01eed60 2024-03-06T07:30:12.801 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:12.801 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192},{"gid":9259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:29:44.306981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":10803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10803":{"gid":10803,"name":"a","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.1:6834/2424540959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2424540959},{"type":"v1","addr":"172.21.15.1:6834","nonce":2424540959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:12.801 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 200 2024-03-06T07:30:12.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.801+0000 7fcec604e640 1 -- 172.21.15.1:0/2852407685 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fceac06b500 msgr2=0x7fceac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.801+0000 7fcec604e640 1 --2- 172.21.15.1:0/2852407685 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fceac06b500 0x7fceac06d9c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009b00 tx=0x7fceb0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:12.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.801+0000 7fcec604e640 1 -- 172.21.15.1:0/2852407685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec01eed60 msgr2=0x7fcec00acf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.801+0000 7fcec604e640 1 --2- 172.21.15.1:0/2852407685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec01eed60 0x7fcec00acf60 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7fceb400c330 tx=0x7fcec01550f0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.801+0000 7fcec604e640 1 -- 172.21.15.1:0/2852407685 shutdown_connections 2024-03-06T07:30:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.801+0000 7fcec604e640 1 --2- 172.21.15.1:0/2852407685 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fceac06b500 0x7fceac06d9c0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.801+0000 7fcec604e640 1 --2- 172.21.15.1:0/2852407685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcec00ab080 0x7fcec01eab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.801+0000 7fcec604e640 1 --2- 172.21.15.1:0/2852407685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec01eed60 0x7fcec00acf60 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.801+0000 7fcec604e640 1 --2- 172.21.15.1:0/2852407685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea8003f60 0x7fcec0152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.801+0000 7fcec604e640 1 -- 172.21.15.1:0/2852407685 >> 172.21.15.1:0/2852407685 conn(0x7fcec006b3e0 msgr2=0x7fcec0154990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.801+0000 7fcec604e640 1 -- 172.21.15.1:0/2852407685 shutdown_connections 2024-03-06T07:30:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:12.801+0000 7fcec604e640 1 -- 172.21.15.1:0/2852407685 wait complete. 2024-03-06T07:30:13.216 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:29:44.306981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 10803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10803': {'gid': 10803, 'name': 'a', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.1:6834/2424540959', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 2424540959}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 2424540959}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:13.216 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10803:up:stopping 2024-03-06T07:30:14.217 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:14.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.318+0000 7f9922888640 1 Processor -- start 2024-03-06T07:30:14.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.318+0000 7f9922888640 1 -- start start 2024-03-06T07:30:14.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.319+0000 7f9922888640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f991c0c6b10 0x7f991c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.319+0000 7f9922888640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f991c0bf9d0 0x7f991c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.319+0000 7f9922888640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f991c0c2fc0 con 0x7f991c152650 2024-03-06T07:30:14.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.319+0000 7f9922888640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f991c155800 con 0x7f991c0c6b10 2024-03-06T07:30:14.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.319+0000 7f9922888640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f991c158da0 con 0x7f991c0bf9d0 2024-03-06T07:30:14.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.319+0000 7f991b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f991c0c6b10 0x7f991c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.319+0000 7f991bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f991c152650 0x7f991c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34096/0 (socket says 172.21.15.1:34096) 2024-03-06T07:30:14.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.320+0000 7f991bfff640 1 -- 172.21.15.1:0/1994813414 learned_addr learned my addr 172.21.15.1:0/1994813414 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:14.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.320+0000 7f991b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f991c0c6b10 0x7f991c155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57782/0 (socket says 172.21.15.1:57782) 2024-03-06T07:30:14.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.320+0000 7f9920e25640 1 --2- 172.21.15.1:0/1994813414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f991c0bf9d0 0x7f991c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.320+0000 7f991b7fe640 1 -- 172.21.15.1:0/1994813414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f991c0bf9d0 msgr2=0x7f991c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:14.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.320+0000 7f991b7fe640 1 --2- 172.21.15.1:0/1994813414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f991c0bf9d0 0x7f991c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.320+0000 7f991b7fe640 1 -- 172.21.15.1:0/1994813414 >> v1:172.21.15.1:6789/0 conn(0x7f991c152650 legacy=0x7f991c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.320+0000 7f991b7fe640 1 -- 172.21.15.1:0/1994813414 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c0c6690 con 0x7f991c0c6b10 2024-03-06T07:30:14.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.320+0000 7f991affd640 1 -- 172.21.15.1:0/1994813414 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679638816 0 0) 0x7f991c0c2fc0 con 0x7f991c152650 2024-03-06T07:30:14.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.321+0000 7f991b7fe640 1 --2- 172.21.15.1:0/1994813414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f991c0c6b10 0x7f991c155c90 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f9914009d90 tx=0x7f991402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b939139bbf9692d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.321+0000 7f991affd640 1 -- 172.21.15.1:0/1994813414 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9914039070 con 0x7f991c0c6b10 2024-03-06T07:30:14.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.321+0000 7f991affd640 1 -- 172.21.15.1:0/1994813414 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f991402cbc0 con 0x7f991c0c6b10 2024-03-06T07:30:14.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.321+0000 7f991affd640 1 -- 172.21.15.1:0/1994813414 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9914034440 con 0x7f991c0c6b10 2024-03-06T07:30:14.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.322+0000 7f9922888640 1 -- 172.21.15.1:0/1994813414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f991c0c6b10 msgr2=0x7f991c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.322+0000 7f9922888640 1 --2- 172.21.15.1:0/1994813414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f991c0c6b10 0x7f991c155c90 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f9914009d90 tx=0x7f991402be10 comp rx=0 tx=0).stop 2024-03-06T07:30:14.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.322+0000 7f9922888640 1 -- 172.21.15.1:0/1994813414 shutdown_connections 2024-03-06T07:30:14.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.322+0000 7f9922888640 1 --2- 172.21.15.1:0/1994813414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f991c0bf9d0 0x7f991c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.322+0000 7f9922888640 1 --2- 172.21.15.1:0/1994813414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f991c0c6b10 0x7f991c155c90 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.322+0000 7f9922888640 1 -- 172.21.15.1:0/1994813414 >> 172.21.15.1:0/1994813414 conn(0x7f991c066ee0 msgr2=0x7f991c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.322+0000 7f9922888640 1 -- 172.21.15.1:0/1994813414 shutdown_connections 2024-03-06T07:30:14.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.322+0000 7f9922888640 1 -- 172.21.15.1:0/1994813414 wait complete. 2024-03-06T07:30:14.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.323+0000 7f9922888640 1 Processor -- start 2024-03-06T07:30:14.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.324+0000 7f9922888640 1 -- start start 2024-03-06T07:30:14.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.325+0000 7f9922888640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f991c0bf9d0 0x7f991c150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.325+0000 7f9922888640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f991c152650 0x7f991c14ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.325+0000 7f9922888640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f991c1514c0 0x7f991c14b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.325+0000 7f9922888640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c1551b0 con 0x7f991c1514c0 2024-03-06T07:30:14.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.325+0000 7f9922888640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c0c6fa0 con 0x7f991c0bf9d0 2024-03-06T07:30:14.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.325+0000 7f9922888640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f991c155490 con 0x7f991c152650 2024-03-06T07:30:14.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.325+0000 7f991bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f991c0bf9d0 0x7f991c150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.325+0000 7f991b7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f991c152650 0x7f991c14ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.325+0000 7f991bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f991c0bf9d0 0x7f991c150f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:42868/0 (socket says 172.21.15.1:42868) 2024-03-06T07:30:14.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.325+0000 7f991bfff640 1 -- 172.21.15.1:0/2443038054 learned_addr learned my addr 172.21.15.1:0/2443038054 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:14.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.325+0000 7f9920e25640 1 --2- 172.21.15.1:0/2443038054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f991c1514c0 0x7f991c14b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.325+0000 7f9920e25640 1 -- 172.21.15.1:0/2443038054 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f991c152650 msgr2=0x7f991c14ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.325+0000 7f9920e25640 1 --2- 172.21.15.1:0/2443038054 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f991c152650 0x7f991c14ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.325+0000 7f9920e25640 1 -- 172.21.15.1:0/2443038054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f991c0bf9d0 msgr2=0x7f991c150f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.325+0000 7f9920e25640 1 --2- 172.21.15.1:0/2443038054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f991c0bf9d0 0x7f991c150f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.325+0000 7f9920e25640 1 -- 172.21.15.1:0/2443038054 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c0c8d70 con 0x7f991c1514c0 2024-03-06T07:30:14.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.326+0000 7f9920e25640 1 --2- 172.21.15.1:0/2443038054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f991c1514c0 0x7f991c14b360 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f990400e7f0 tx=0x7f990400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.326+0000 7f9918ff9640 1 -- 172.21.15.1:0/2443038054 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9904017070 con 0x7f991c1514c0 2024-03-06T07:30:14.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.326+0000 7f9918ff9640 1 -- 172.21.15.1:0/2443038054 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990400c6d0 con 0x7f991c1514c0 2024-03-06T07:30:14.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.326+0000 7f9922888640 1 -- 172.21.15.1:0/2443038054 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f991c1eb900 con 0x7f991c1514c0 2024-03-06T07:30:14.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.326+0000 7f9918ff9640 1 -- 172.21.15.1:0/2443038054 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990400cac0 con 0x7f991c1514c0 2024-03-06T07:30:14.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.326+0000 7f9922888640 1 -- 172.21.15.1:0/2443038054 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f991c1ebbe0 con 0x7f991c1514c0 2024-03-06T07:30:14.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.327+0000 7f9922888640 1 -- 172.21.15.1:0/2443038054 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f991c067fd0 con 0x7f991c1514c0 2024-03-06T07:30:14.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.327+0000 7f9918ff9640 1 -- 172.21.15.1:0/2443038054 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f990401a070 con 0x7f991c1514c0 2024-03-06T07:30:14.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.328+0000 7f9918ff9640 1 --2- 172.21.15.1:0/2443038054 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f98fc06b500 0x7f98fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.328+0000 7f9918ff9640 1 -- 172.21.15.1:0/2443038054 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f990408a040 con 0x7f991c1514c0 2024-03-06T07:30:14.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.328+0000 7f991bfff640 1 --2- 172.21.15.1:0/2443038054 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f98fc06b500 0x7f98fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.328+0000 7f991bfff640 1 --2- 172.21.15.1:0/2443038054 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f98fc06b500 0x7f98fc06d9c0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f990c007b00 tx=0x7f990c007a50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.332+0000 7f9918ff9640 1 -- 172.21.15.1:0/2443038054 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99040586c0 con 0x7f991c1514c0 2024-03-06T07:30:14.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.549+0000 7f9922888640 1 -- 172.21.15.1:0/2443038054 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f991c0b06f0 con 0x7f991c1514c0 2024-03-06T07:30:14.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.549+0000 7f9918ff9640 1 -- 172.21.15.1:0/2443038054 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4213 (secure 0 0 0) 0x7f990405c760 con 0x7f991c1514c0 2024-03-06T07:30:14.551 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:14.551 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:14.310309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:14.552 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 201 2024-03-06T07:30:14.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.551+0000 7f9922888640 1 -- 172.21.15.1:0/2443038054 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f98fc06b500 msgr2=0x7f98fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.551+0000 7f9922888640 1 --2- 172.21.15.1:0/2443038054 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f98fc06b500 0x7f98fc06d9c0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f990c007b00 tx=0x7f990c007a50 comp rx=0 tx=0).stop 2024-03-06T07:30:14.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.551+0000 7f9922888640 1 -- 172.21.15.1:0/2443038054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f991c1514c0 msgr2=0x7f991c14b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.551+0000 7f9922888640 1 --2- 172.21.15.1:0/2443038054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f991c1514c0 0x7f991c14b360 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f990400e7f0 tx=0x7f990400ecb0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.551+0000 7f9922888640 1 -- 172.21.15.1:0/2443038054 shutdown_connections 2024-03-06T07:30:14.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.551+0000 7f9922888640 1 --2- 172.21.15.1:0/2443038054 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f98fc06b500 0x7f98fc06d9c0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.551+0000 7f9922888640 1 --2- 172.21.15.1:0/2443038054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f991c1514c0 0x7f991c14b360 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.551+0000 7f9922888640 1 --2- 172.21.15.1:0/2443038054 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f991c152650 0x7f991c14ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.551+0000 7f9922888640 1 --2- 172.21.15.1:0/2443038054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f991c0bf9d0 0x7f991c150f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.551+0000 7f9922888640 1 -- 172.21.15.1:0/2443038054 >> 172.21.15.1:0/2443038054 conn(0x7f991c066ee0 msgr2=0x7f991c0c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.551+0000 7f9922888640 1 -- 172.21.15.1:0/2443038054 shutdown_connections 2024-03-06T07:30:14.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:14.551+0000 7f9922888640 1 -- 172.21.15.1:0/2443038054 wait complete. 2024-03-06T07:30:14.966 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:14.310309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:14.967 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:replay 2024-03-06T07:30:15.968 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:16.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.068+0000 7f825024e640 1 Processor -- start 2024-03-06T07:30:16.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.068+0000 7f825024e640 1 -- start start 2024-03-06T07:30:16.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.069+0000 7f825024e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f82480a7050 0x7f82480c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.069+0000 7f825024e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82480c4a20 0x7f8248156460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.069+0000 7f825024e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8248153a00 con 0x7f82481569a0 2024-03-06T07:30:16.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.069+0000 7f825024e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8248153ce0 con 0x7f82480c4a20 2024-03-06T07:30:16.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.069+0000 7f825024e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82480c3cb0 con 0x7f82480a7050 2024-03-06T07:30:16.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.069+0000 7f824d7e9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82480c4a20 0x7f8248156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.069+0000 7f824e7eb640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f82481569a0 0x7f824806aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52704/0 (socket says 172.21.15.1:52704) 2024-03-06T07:30:16.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.069+0000 7f824d7e9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82480c4a20 0x7f8248156460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51344/0 (socket says 172.21.15.1:51344) 2024-03-06T07:30:16.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.069+0000 7f824e7eb640 1 -- 172.21.15.1:0/2511767331 learned_addr learned my addr 172.21.15.1:0/2511767331 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:16.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.069+0000 7f824dfea640 1 --2- 172.21.15.1:0/2511767331 >> v2:172.21.15.98:3300/0 conn(0x7f82480a7050 0x7f82480c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.069+0000 7f824d7e9640 1 -- 172.21.15.1:0/2511767331 >> v2:172.21.15.98:3300/0 conn(0x7f82480a7050 msgr2=0x7f82480c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.069+0000 7f824d7e9640 1 --2- 172.21.15.1:0/2511767331 >> v2:172.21.15.98:3300/0 conn(0x7f82480a7050 0x7f82480c44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.069+0000 7f824d7e9640 1 -- 172.21.15.1:0/2511767331 >> v1:172.21.15.1:6789/0 conn(0x7f82481569a0 legacy=0x7f824806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.069+0000 7f824d7e9640 1 -- 172.21.15.1:0/2511767331 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82480c3f90 con 0x7f82480c4a20 2024-03-06T07:30:16.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.069+0000 7f824cfe8640 1 -- 172.21.15.1:0/2511767331 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1855350679 0 0) 0x7f8248153a00 con 0x7f82481569a0 2024-03-06T07:30:16.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.069+0000 7f824dfea640 1 --2- 172.21.15.1:0/2511767331 >> v2:172.21.15.98:3300/0 conn(0x7f82480a7050 0x7f82480c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:16.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.070+0000 7f824d7e9640 1 --2- 172.21.15.1:0/2511767331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82480c4a20 0x7f8248156460 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f8248072f30 tx=0x7f823802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287945d8c1cca6e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.070+0000 7f824cfe8640 1 -- 172.21.15.1:0/2511767331 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238038070 con 0x7f82480c4a20 2024-03-06T07:30:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.071+0000 7f824cfe8640 1 -- 172.21.15.1:0/2511767331 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8238031710 con 0x7f82480c4a20 2024-03-06T07:30:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.071+0000 7f824cfe8640 1 -- 172.21.15.1:0/2511767331 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238031ae0 con 0x7f82480c4a20 2024-03-06T07:30:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.071+0000 7f825024e640 1 -- 172.21.15.1:0/2511767331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82480c4a20 msgr2=0x7f8248156460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.071+0000 7f825024e640 1 --2- 172.21.15.1:0/2511767331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82480c4a20 0x7f8248156460 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f8248072f30 tx=0x7f823802b7b0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.071+0000 7f825024e640 1 -- 172.21.15.1:0/2511767331 shutdown_connections 2024-03-06T07:30:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.071+0000 7f825024e640 1 --2- 172.21.15.1:0/2511767331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82480c4a20 0x7f8248156460 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.071+0000 7f825024e640 1 --2- 172.21.15.1:0/2511767331 >> v2:172.21.15.98:3300/0 conn(0x7f82480a7050 0x7f82480c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.071+0000 7f825024e640 1 -- 172.21.15.1:0/2511767331 >> 172.21.15.1:0/2511767331 conn(0x7f82480c7a50 msgr2=0x7f82480c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.071+0000 7f825024e640 1 -- 172.21.15.1:0/2511767331 shutdown_connections 2024-03-06T07:30:16.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.071+0000 7f825024e640 1 -- 172.21.15.1:0/2511767331 wait complete. 2024-03-06T07:30:16.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.072+0000 7f825024e640 1 Processor -- start 2024-03-06T07:30:16.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.072+0000 7f825024e640 1 -- start start 2024-03-06T07:30:16.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.073+0000 7f825024e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82480a7050 0x7f82481ee970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.073+0000 7f825024e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82480c4a20 0x7f82481eaa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.073+0000 7f825024e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82481569a0 0x7f824814c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.073+0000 7f825024e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8248072f60 con 0x7f82481569a0 2024-03-06T07:30:16.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.073+0000 7f825024e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82480c4ea0 con 0x7f82480a7050 2024-03-06T07:30:16.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.073+0000 7f825024e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82480c2450 con 0x7f82480c4a20 2024-03-06T07:30:16.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.073+0000 7f824e7eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82481569a0 0x7f824814c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.073+0000 7f824e7eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82481569a0 0x7f824814c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51356/0 (socket says 172.21.15.1:51356) 2024-03-06T07:30:16.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.073+0000 7f824e7eb640 1 -- 172.21.15.1:0/211673213 learned_addr learned my addr 172.21.15.1:0/211673213 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:16.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.073+0000 7f824d7e9640 1 --2- 172.21.15.1:0/211673213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82480c4a20 0x7f82481eaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.073+0000 7f824e7eb640 1 -- 172.21.15.1:0/211673213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82480c4a20 msgr2=0x7f82481eaa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.073+0000 7f824e7eb640 1 --2- 172.21.15.1:0/211673213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82480c4a20 0x7f82481eaa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.073+0000 7f824e7eb640 1 -- 172.21.15.1:0/211673213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82480a7050 msgr2=0x7f82481ee970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.073+0000 7f824dfea640 1 --2- 172.21.15.1:0/211673213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82480a7050 0x7f82481ee970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.073+0000 7f824e7eb640 1 --2- 172.21.15.1:0/211673213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82480a7050 0x7f82481ee970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.073+0000 7f824e7eb640 1 -- 172.21.15.1:0/211673213 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82481eeeb0 con 0x7f82481569a0 2024-03-06T07:30:16.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.074+0000 7f824e7eb640 1 --2- 172.21.15.1:0/211673213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82481569a0 0x7f824814c800 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f824000c7a0 tx=0x7f824000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.074+0000 7f823e7fc640 1 -- 172.21.15.1:0/211673213 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8240018070 con 0x7f82481569a0 2024-03-06T07:30:16.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.074+0000 7f823e7fc640 1 -- 172.21.15.1:0/211673213 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f824000a640 con 0x7f82481569a0 2024-03-06T07:30:16.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.074+0000 7f825024e640 1 -- 172.21.15.1:0/211673213 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82480a9680 con 0x7f82481569a0 2024-03-06T07:30:16.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.074+0000 7f823e7fc640 1 -- 172.21.15.1:0/211673213 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824000aa70 con 0x7f82481569a0 2024-03-06T07:30:16.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.074+0000 7f825024e640 1 -- 172.21.15.1:0/211673213 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82480ac820 con 0x7f82481569a0 2024-03-06T07:30:16.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.075+0000 7f825024e640 1 -- 172.21.15.1:0/211673213 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82480686e0 con 0x7f82481569a0 2024-03-06T07:30:16.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.076+0000 7f823e7fc640 1 -- 172.21.15.1:0/211673213 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f824001b070 con 0x7f82481569a0 2024-03-06T07:30:16.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.076+0000 7f823e7fc640 1 --2- 172.21.15.1:0/211673213 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f822c06b4d0 0x7f822c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.076+0000 7f823e7fc640 1 -- 172.21.15.1:0/211673213 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f82400899f0 con 0x7f82481569a0 2024-03-06T07:30:16.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.076+0000 7f824dfea640 1 --2- 172.21.15.1:0/211673213 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f822c06b4d0 0x7f822c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.077+0000 7f824dfea640 1 --2- 172.21.15.1:0/211673213 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f822c06b4d0 0x7f822c06d990 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f8244006fd0 tx=0x7f8244008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.084+0000 7f823e7fc640 1 -- 172.21.15.1:0/211673213 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8240058630 con 0x7f82481569a0 2024-03-06T07:30:16.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.317+0000 7f825024e640 1 -- 172.21.15.1:0/211673213 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8248072b40 con 0x7f82481569a0 2024-03-06T07:30:16.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.317+0000 7f823e7fc640 1 -- 172.21.15.1:0/211673213 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+4217 (secure 0 0 0) 0x7f8240057f30 con 0x7f82481569a0 2024-03-06T07:30:16.319 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:16.319 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":203,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:15.318714+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:16.319 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 203 2024-03-06T07:30:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.320+0000 7f825024e640 1 -- 172.21.15.1:0/211673213 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f822c06b4d0 msgr2=0x7f822c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.320+0000 7f825024e640 1 --2- 172.21.15.1:0/211673213 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f822c06b4d0 0x7f822c06d990 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f8244006fd0 tx=0x7f8244008040 comp rx=0 tx=0).stop 2024-03-06T07:30:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.320+0000 7f825024e640 1 -- 172.21.15.1:0/211673213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82481569a0 msgr2=0x7f824814c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.320+0000 7f825024e640 1 --2- 172.21.15.1:0/211673213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82481569a0 0x7f824814c800 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f824000c7a0 tx=0x7f824000cc60 comp rx=0 tx=0).stop 2024-03-06T07:30:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.320+0000 7f825024e640 1 -- 172.21.15.1:0/211673213 shutdown_connections 2024-03-06T07:30:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.320+0000 7f825024e640 1 --2- 172.21.15.1:0/211673213 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f822c06b4d0 0x7f822c06d990 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.320+0000 7f825024e640 1 --2- 172.21.15.1:0/211673213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82481569a0 0x7f824814c800 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.320+0000 7f825024e640 1 --2- 172.21.15.1:0/211673213 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82480c4a20 0x7f82481eaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.320+0000 7f825024e640 1 --2- 172.21.15.1:0/211673213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82480a7050 0x7f82481ee970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.320+0000 7f825024e640 1 -- 172.21.15.1:0/211673213 >> 172.21.15.1:0/211673213 conn(0x7f82480c7a50 msgr2=0x7f8248155450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.320+0000 7f825024e640 1 -- 172.21.15.1:0/211673213 shutdown_connections 2024-03-06T07:30:16.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:16.320+0000 7f825024e640 1 -- 172.21.15.1:0/211673213 wait complete. 2024-03-06T07:30:16.739 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 203, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:15.318714+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:reconnect', 'state_seq': 14, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:16.739 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:reconnect 2024-03-06T07:30:17.740 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:17.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.840+0000 7f5364dc7640 1 Processor -- start 2024-03-06T07:30:17.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.840+0000 7f5364dc7640 1 -- start start 2024-03-06T07:30:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.841+0000 7f5364dc7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5360152650 0x7f5360155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.841+0000 7f5364dc7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53600c6b10 0x7f5360158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.841+0000 7f5364dc7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53600c2fc0 con 0x7f53600bf9d0 2024-03-06T07:30:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.841+0000 7f5364dc7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5360155800 con 0x7f5360152650 2024-03-06T07:30:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.841+0000 7f5364dc7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53600c6690 con 0x7f53600c6b10 2024-03-06T07:30:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.841+0000 7f535e59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5360152650 0x7f5360155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.841+0000 7f535ed9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f53600bf9d0 0x7f53600c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52716/0 (socket says 172.21.15.1:52716) 2024-03-06T07:30:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.841+0000 7f535ed9d640 1 -- 172.21.15.1:0/2532375312 learned_addr learned my addr 172.21.15.1:0/2532375312 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.841+0000 7f5357fff640 1 --2- 172.21.15.1:0/2532375312 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53600c6b10 0x7f5360158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.841+0000 7f535e59c640 1 -- 172.21.15.1:0/2532375312 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53600c6b10 msgr2=0x7f5360158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.841+0000 7f535e59c640 1 --2- 172.21.15.1:0/2532375312 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53600c6b10 0x7f5360158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.841+0000 7f535e59c640 1 -- 172.21.15.1:0/2532375312 >> v1:172.21.15.1:6789/0 conn(0x7f53600bf9d0 legacy=0x7f53600c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.841+0000 7f535e59c640 1 -- 172.21.15.1:0/2532375312 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53600c7280 con 0x7f5360152650 2024-03-06T07:30:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.841+0000 7f5357fff640 1 --2- 172.21.15.1:0/2532375312 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53600c6b10 0x7f5360158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.842+0000 7f535e59c640 1 --2- 172.21.15.1:0/2532375312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5360152650 0x7f5360155c90 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f53500097a0 tx=0x7f535002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29f80b4a8f7b1162 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.842+0000 7f535dd9b640 1 -- 172.21.15.1:0/2532375312 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53500315a0 con 0x7f5360152650 2024-03-06T07:30:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.842+0000 7f535dd9b640 1 -- 172.21.15.1:0/2532375312 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5350037070 con 0x7f5360152650 2024-03-06T07:30:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.842+0000 7f535dd9b640 1 -- 172.21.15.1:0/2532375312 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5350031960 con 0x7f5360152650 2024-03-06T07:30:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.843+0000 7f5364dc7640 1 -- 172.21.15.1:0/2532375312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5360152650 msgr2=0x7f5360155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.843+0000 7f5364dc7640 1 --2- 172.21.15.1:0/2532375312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5360152650 0x7f5360155c90 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f53500097a0 tx=0x7f535002b840 comp rx=0 tx=0).stop 2024-03-06T07:30:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.843+0000 7f5364dc7640 1 -- 172.21.15.1:0/2532375312 shutdown_connections 2024-03-06T07:30:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.843+0000 7f5364dc7640 1 --2- 172.21.15.1:0/2532375312 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53600c6b10 0x7f5360158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.843+0000 7f5364dc7640 1 --2- 172.21.15.1:0/2532375312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5360152650 0x7f5360155c90 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.843+0000 7f5364dc7640 1 -- 172.21.15.1:0/2532375312 >> 172.21.15.1:0/2532375312 conn(0x7f5360066ee0 msgr2=0x7f53600672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.843+0000 7f5364dc7640 1 -- 172.21.15.1:0/2532375312 shutdown_connections 2024-03-06T07:30:17.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.843+0000 7f5364dc7640 1 -- 172.21.15.1:0/2532375312 wait complete. 2024-03-06T07:30:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.844+0000 7f5364dc7640 1 Processor -- start 2024-03-06T07:30:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.844+0000 7f5364dc7640 1 -- start start 2024-03-06T07:30:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f5364dc7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53600bf9d0 0x7f5360156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f5364dc7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5360152650 0x7f5360141fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f5364dc7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f536014cf80 0x7f53601538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f5364dc7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5360155490 con 0x7f5360152650 2024-03-06T07:30:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f5364dc7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53601551b0 con 0x7f536014cf80 2024-03-06T07:30:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f5364dc7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5360152a30 con 0x7f53600bf9d0 2024-03-06T07:30:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f5357fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5360152650 0x7f5360141fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f5357fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5360152650 0x7f5360141fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51362/0 (socket says 172.21.15.1:51362) 2024-03-06T07:30:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f5357fff640 1 -- 172.21.15.1:0/718907201 learned_addr learned my addr 172.21.15.1:0/718907201 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f535ed9d640 1 --2- 172.21.15.1:0/718907201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f536014cf80 0x7f53601538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f535e59c640 1 --2- 172.21.15.1:0/718907201 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53600bf9d0 0x7f5360156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f5357fff640 1 -- 172.21.15.1:0/718907201 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53600bf9d0 msgr2=0x7f5360156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f5357fff640 1 --2- 172.21.15.1:0/718907201 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53600bf9d0 0x7f5360156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f5357fff640 1 -- 172.21.15.1:0/718907201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f536014cf80 msgr2=0x7f53601538f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f5357fff640 1 --2- 172.21.15.1:0/718907201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f536014cf80 0x7f53601538f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f5357fff640 1 -- 172.21.15.1:0/718907201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53601ef320 con 0x7f5360152650 2024-03-06T07:30:17.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f5357fff640 1 --2- 172.21.15.1:0/718907201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5360152650 0x7f5360141fc0 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f534800c820 tx=0x7f534800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.845+0000 7f5356ffd640 1 -- 172.21.15.1:0/718907201 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5348015070 con 0x7f5360152650 2024-03-06T07:30:17.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.846+0000 7f5356ffd640 1 -- 172.21.15.1:0/718907201 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5348007a30 con 0x7f5360152650 2024-03-06T07:30:17.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.846+0000 7f5364dc7640 1 -- 172.21.15.1:0/718907201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5360158f50 con 0x7f5360152650 2024-03-06T07:30:17.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.846+0000 7f5356ffd640 1 -- 172.21.15.1:0/718907201 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5348007e00 con 0x7f5360152650 2024-03-06T07:30:17.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.846+0000 7f5364dc7640 1 -- 172.21.15.1:0/718907201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5360159510 con 0x7f5360152650 2024-03-06T07:30:17.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.847+0000 7f5364dc7640 1 -- 172.21.15.1:0/718907201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53600b06f0 con 0x7f5360152650 2024-03-06T07:30:17.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.847+0000 7f5356ffd640 1 -- 172.21.15.1:0/718907201 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5348018070 con 0x7f5360152650 2024-03-06T07:30:17.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.847+0000 7f5356ffd640 1 --2- 172.21.15.1:0/718907201 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f534006b4d0 0x7f534006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.848+0000 7f5356ffd640 1 -- 172.21.15.1:0/718907201 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f534808a290 con 0x7f5360152650 2024-03-06T07:30:17.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.848+0000 7f535e59c640 1 --2- 172.21.15.1:0/718907201 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f534006b4d0 0x7f534006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.848+0000 7f535e59c640 1 --2- 172.21.15.1:0/718907201 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f534006b4d0 0x7f534006d990 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f5350036000 tx=0x7f535003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:17.852+0000 7f5356ffd640 1 -- 172.21.15.1:0/718907201 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53601ef320 con 0x7f5360152650 2024-03-06T07:30:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:18.131+0000 7f5364dc7640 1 -- 172.21.15.1:0/718907201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53600c5e00 con 0x7f5360152650 2024-03-06T07:30:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:18.132+0000 7f5356ffd640 1 -- 172.21.15.1:0/718907201 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+4214 (secure 0 0 0) 0x7f5348058910 con 0x7f5360152650 2024-03-06T07:30:18.136 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:18.136 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":204,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:16.319650+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:18.137 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 204 2024-03-06T07:30:18.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:18.133+0000 7f5364dc7640 1 -- 172.21.15.1:0/718907201 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f534006b4d0 msgr2=0x7f534006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:18.133+0000 7f5364dc7640 1 --2- 172.21.15.1:0/718907201 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f534006b4d0 0x7f534006d990 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f5350036000 tx=0x7f535003d040 comp rx=0 tx=0).stop 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:18.133+0000 7f5364dc7640 1 -- 172.21.15.1:0/718907201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5360152650 msgr2=0x7f5360141fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:18.133+0000 7f5364dc7640 1 --2- 172.21.15.1:0/718907201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5360152650 0x7f5360141fc0 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f534800c820 tx=0x7f534800cc20 comp rx=0 tx=0).stop 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:18.133+0000 7f5364dc7640 1 -- 172.21.15.1:0/718907201 shutdown_connections 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:18.133+0000 7f5364dc7640 1 --2- 172.21.15.1:0/718907201 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f534006b4d0 0x7f534006d990 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:18.133+0000 7f5364dc7640 1 --2- 172.21.15.1:0/718907201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f536014cf80 0x7f53601538f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:18.133+0000 7f5364dc7640 1 --2- 172.21.15.1:0/718907201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5360152650 0x7f5360141fc0 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:18.133+0000 7f5364dc7640 1 --2- 172.21.15.1:0/718907201 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f53600bf9d0 0x7f5360156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:18.133+0000 7f5364dc7640 1 -- 172.21.15.1:0/718907201 >> 172.21.15.1:0/718907201 conn(0x7f5360066ee0 msgr2=0x7f53600672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:18.134+0000 7f5364dc7640 1 -- 172.21.15.1:0/718907201 shutdown_connections 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:18.134+0000 7f5364dc7640 1 -- 172.21.15.1:0/718907201 wait complete. 2024-03-06T07:30:18.548 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 204, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:16.319650+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:rejoin', 'state_seq': 15, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:18.549 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:rejoin 2024-03-06T07:30:19.551 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:19.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.651+0000 7f9c3f3e9640 1 Processor -- start 2024-03-06T07:30:19.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.651+0000 7f9c3f3e9640 1 -- start start 2024-03-06T07:30:19.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.652+0000 7f9c3f3e9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9c381543a0 0x7f9c3815e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.652+0000 7f9c3f3e9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c38150900 con 0x7f9c380c32f0 2024-03-06T07:30:19.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.652+0000 7f9c3f3e9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c38071790 con 0x7f9c38150520 2024-03-06T07:30:19.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.652+0000 7f9c3f3e9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c38153ed0 con 0x7f9c381543a0 2024-03-06T07:30:19.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.652+0000 7f9c3c984640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9c380c32f0 0x7f9c38070c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52718/0 (socket says 172.21.15.1:52718) 2024-03-06T07:30:19.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.652+0000 7f9c3c984640 1 -- 172.21.15.1:0/4246189998 learned_addr learned my addr 172.21.15.1:0/4246189998 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:19.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c3d986640 1 --2- 172.21.15.1:0/4246189998 >> v2:172.21.15.98:3300/0 conn(0x7f9c381543a0 0x7f9c3815e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c2ffff640 1 -- 172.21.15.1:0/4246189998 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107834766 0 0) 0x7f9c38150900 con 0x7f9c380c32f0 2024-03-06T07:30:19.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c2ffff640 1 -- 172.21.15.1:0/4246189998 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c24003520 con 0x7f9c380c32f0 2024-03-06T07:30:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c2ffff640 1 -- 172.21.15.1:0/4246189998 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2695531140 0 0) 0x7f9c24003520 con 0x7f9c380c32f0 2024-03-06T07:30:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c2ffff640 1 -- 172.21.15.1:0/4246189998 >> v2:172.21.15.98:3300/0 conn(0x7f9c381543a0 msgr2=0x7f9c3815e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c2ffff640 1 --2- 172.21.15.1:0/4246189998 >> v2:172.21.15.98:3300/0 conn(0x7f9c381543a0 0x7f9c3815e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c2ffff640 1 -- 172.21.15.1:0/4246189998 >> v1:172.21.15.98:6789/0 conn(0x7f9c38150520 legacy=0x7f9c381537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c2ffff640 1 -- 172.21.15.1:0/4246189998 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c38157050 con 0x7f9c380c32f0 2024-03-06T07:30:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c2ffff640 1 -- 172.21.15.1:0/4246189998 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f9c28001670 con 0x7f9c380c32f0 2024-03-06T07:30:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c2ffff640 1 -- 172.21.15.1:0/4246189998 >> v1:172.21.15.1:6789/0 conn(0x7f9c380c32f0 legacy=0x7f9c38070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c2ffff640 1 --2- 172.21.15.1:0/4246189998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c240039b0 0x7f9c24003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c2ffff640 1 --2- 172.21.15.1:0/4246189998 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c24004800 0x7f9c24004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c2ffff640 1 --2- 172.21.15.1:0/4246189998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c240051c0 0x7f9c2400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c2ffff640 1 -- 172.21.15.1:0/4246189998 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c38153ed0 con 0x7f9c240051c0 2024-03-06T07:30:19.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c2ffff640 1 -- 172.21.15.1:0/4246189998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2400c060 con 0x7f9c240039b0 2024-03-06T07:30:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c2ffff640 1 -- 172.21.15.1:0/4246189998 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2400c260 con 0x7f9c24004800 2024-03-06T07:30:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c3d986640 1 --2- 172.21.15.1:0/4246189998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c240051c0 0x7f9c2400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c3d185640 1 --2- 172.21.15.1:0/4246189998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c240039b0 0x7f9c24003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c3c984640 1 --2- 172.21.15.1:0/4246189998 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c24004800 0x7f9c24004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c3d986640 1 -- 172.21.15.1:0/4246189998 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c24004800 msgr2=0x7f9c24004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c3d986640 1 --2- 172.21.15.1:0/4246189998 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c24004800 0x7f9c24004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c3d986640 1 -- 172.21.15.1:0/4246189998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c240039b0 msgr2=0x7f9c24003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c3d986640 1 --2- 172.21.15.1:0/4246189998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c240039b0 0x7f9c24003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.653+0000 7f9c3d986640 1 -- 172.21.15.1:0/4246189998 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c24003520 con 0x7f9c240051c0 2024-03-06T07:30:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.654+0000 7f9c3d986640 1 -- 172.21.15.1:0/4246189998 reap_dead start 2024-03-06T07:30:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.655+0000 7f9c3d986640 1 --2- 172.21.15.1:0/4246189998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c240051c0 0x7f9c2400b9c0 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009fc0 tx=0x7f9c24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac5f5f31a14a96cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.655+0000 7f9c2ffff640 1 -- 172.21.15.1:0/4246189998 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c30009c10 con 0x7f9c240051c0 2024-03-06T07:30:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.655+0000 7f9c2ffff640 1 -- 172.21.15.1:0/4246189998 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c3002cc60 con 0x7f9c240051c0 2024-03-06T07:30:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.655+0000 7f9c2ffff640 1 -- 172.21.15.1:0/4246189998 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c30003bd0 con 0x7f9c240051c0 2024-03-06T07:30:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.655+0000 7f9c3f3e9640 1 -- 172.21.15.1:0/4246189998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c240051c0 msgr2=0x7f9c2400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.655+0000 7f9c3f3e9640 1 --2- 172.21.15.1:0/4246189998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c240051c0 0x7f9c2400b9c0 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009fc0 tx=0x7f9c24005830 comp rx=0 tx=0).stop 2024-03-06T07:30:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.655+0000 7f9c3f3e9640 1 -- 172.21.15.1:0/4246189998 shutdown_connections 2024-03-06T07:30:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.655+0000 7f9c3f3e9640 1 --2- 172.21.15.1:0/4246189998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c240051c0 0x7f9c2400b9c0 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.655+0000 7f9c3f3e9640 1 -- 172.21.15.1:0/4246189998 >> 172.21.15.1:0/4246189998 conn(0x7f9c380c88f0 msgr2=0x7f9c380c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.655+0000 7f9c3f3e9640 1 -- 172.21.15.1:0/4246189998 shutdown_connections 2024-03-06T07:30:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.655+0000 7f9c3f3e9640 1 -- 172.21.15.1:0/4246189998 wait complete. 2024-03-06T07:30:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.656+0000 7f9c3f3e9640 1 Processor -- start 2024-03-06T07:30:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.656+0000 7f9c3f3e9640 1 -- start start 2024-03-06T07:30:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.657+0000 7f9c3f3e9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c240051c0 0x7f9c381493e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.657+0000 7f9c3f3e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c380a86c0 0x7f9c381ee1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.657+0000 7f9c3f3e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c381522a0 0x7f9c381484e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.657+0000 7f9c3f3e9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c381532a0 con 0x7f9c240051c0 2024-03-06T07:30:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.657+0000 7f9c3f3e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c381547e0 con 0x7f9c381522a0 2024-03-06T07:30:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.657+0000 7f9c3f3e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c38157330 con 0x7f9c380a86c0 2024-03-06T07:30:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.657+0000 7f9c3d185640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c240051c0 0x7f9c381493e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.657+0000 7f9c3d185640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c240051c0 0x7f9c381493e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51380/0 (socket says 172.21.15.1:51380) 2024-03-06T07:30:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.657+0000 7f9c3d185640 1 -- 172.21.15.1:0/3259227785 learned_addr learned my addr 172.21.15.1:0/3259227785 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.657+0000 7f9c3d986640 1 --2- 172.21.15.1:0/3259227785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c381522a0 0x7f9c381484e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.657+0000 7f9c3c984640 1 --2- 172.21.15.1:0/3259227785 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c380a86c0 0x7f9c381ee1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.657+0000 7f9c3d185640 1 -- 172.21.15.1:0/3259227785 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c380a86c0 msgr2=0x7f9c381ee1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.657+0000 7f9c3d185640 1 --2- 172.21.15.1:0/3259227785 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c380a86c0 0x7f9c381ee1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.657+0000 7f9c3d185640 1 -- 172.21.15.1:0/3259227785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c381522a0 msgr2=0x7f9c381484e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.657+0000 7f9c3d185640 1 --2- 172.21.15.1:0/3259227785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c381522a0 0x7f9c381484e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.657+0000 7f9c3d185640 1 -- 172.21.15.1:0/3259227785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3814d4c0 con 0x7f9c240051c0 2024-03-06T07:30:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.657+0000 7f9c3d986640 1 --2- 172.21.15.1:0/3259227785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c381522a0 0x7f9c381484e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.658+0000 7f9c3c984640 1 --2- 172.21.15.1:0/3259227785 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c380a86c0 0x7f9c381ee1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.658+0000 7f9c3d185640 1 --2- 172.21.15.1:0/3259227785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c240051c0 0x7f9c381493e0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400f620 tx=0x7f9c3400fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.658+0000 7f9c2dffb640 1 -- 172.21.15.1:0/3259227785 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c340052e0 con 0x7f9c240051c0 2024-03-06T07:30:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.658+0000 7f9c2dffb640 1 -- 172.21.15.1:0/3259227785 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c34018070 con 0x7f9c240051c0 2024-03-06T07:30:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.658+0000 7f9c3f3e9640 1 -- 172.21.15.1:0/3259227785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c381eea30 con 0x7f9c240051c0 2024-03-06T07:30:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.658+0000 7f9c2dffb640 1 -- 172.21.15.1:0/3259227785 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c34003810 con 0x7f9c240051c0 2024-03-06T07:30:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.658+0000 7f9c3f3e9640 1 -- 172.21.15.1:0/3259227785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c38150f00 con 0x7f9c240051c0 2024-03-06T07:30:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.659+0000 7f9c2dffb640 1 -- 172.21.15.1:0/3259227785 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c340039b0 con 0x7f9c240051c0 2024-03-06T07:30:19.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.659+0000 7f9c3f3e9640 1 -- 172.21.15.1:0/3259227785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c3814ed50 con 0x7f9c240051c0 2024-03-06T07:30:19.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.660+0000 7f9c2dffb640 1 --2- 172.21.15.1:0/3259227785 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c1c06b500 0x7f9c1c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.660+0000 7f9c2dffb640 1 -- 172.21.15.1:0/3259227785 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f9c3408a110 con 0x7f9c240051c0 2024-03-06T07:30:19.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.660+0000 7f9c3c984640 1 --2- 172.21.15.1:0/3259227785 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c1c06b500 0x7f9c1c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.661+0000 7f9c3c984640 1 --2- 172.21.15.1:0/3259227785 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c1c06b500 0x7f9c1c06d9c0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f9c28007bf0 tx=0x7f9c28009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.664+0000 7f9c2dffb640 1 -- 172.21.15.1:0/3259227785 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c3814ed50 con 0x7f9c240051c0 2024-03-06T07:30:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.954+0000 7f9c3f3e9640 1 -- 172.21.15.1:0/3259227785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c380686e0 con 0x7f9c240051c0 2024-03-06T07:30:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.954+0000 7f9c2dffb640 1 -- 172.21.15.1:0/3259227785 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4216 (secure 0 0 0) 0x7f9c3814d4c0 con 0x7f9c240051c0 2024-03-06T07:30:19.957 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:19.957 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:18.318227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:stopping","state_seq":16,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 206 2024-03-06T07:30:19.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.956+0000 7f9c3f3e9640 1 -- 172.21.15.1:0/3259227785 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c1c06b500 msgr2=0x7f9c1c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.956+0000 7f9c3f3e9640 1 --2- 172.21.15.1:0/3259227785 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c1c06b500 0x7f9c1c06d9c0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f9c28007bf0 tx=0x7f9c28009040 comp rx=0 tx=0).stop 2024-03-06T07:30:19.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.956+0000 7f9c3f3e9640 1 -- 172.21.15.1:0/3259227785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c240051c0 msgr2=0x7f9c381493e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.956+0000 7f9c3f3e9640 1 --2- 172.21.15.1:0/3259227785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c240051c0 0x7f9c381493e0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400f620 tx=0x7f9c3400fae0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.956+0000 7f9c3f3e9640 1 -- 172.21.15.1:0/3259227785 shutdown_connections 2024-03-06T07:30:19.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.956+0000 7f9c3f3e9640 1 --2- 172.21.15.1:0/3259227785 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9c1c06b500 0x7f9c1c06d9c0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.956+0000 7f9c3f3e9640 1 --2- 172.21.15.1:0/3259227785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c381522a0 0x7f9c381484e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.956+0000 7f9c3f3e9640 1 --2- 172.21.15.1:0/3259227785 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c380a86c0 0x7f9c381ee1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.956+0000 7f9c3f3e9640 1 --2- 172.21.15.1:0/3259227785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c240051c0 0x7f9c381493e0 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.956+0000 7f9c3f3e9640 1 -- 172.21.15.1:0/3259227785 >> 172.21.15.1:0/3259227785 conn(0x7f9c380c88f0 msgr2=0x7f9c38070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.957+0000 7f9c3f3e9640 1 -- 172.21.15.1:0/3259227785 shutdown_connections 2024-03-06T07:30:19.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:19.957+0000 7f9c3f3e9640 1 -- 172.21.15.1:0/3259227785 wait complete. 2024-03-06T07:30:20.371 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:18.318227+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:20.371 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:stopping 2024-03-06T07:30:21.372 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:21.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.472+0000 7f1e279f6640 1 Processor -- start 2024-03-06T07:30:21.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.473+0000 7f1e279f6640 1 -- start start 2024-03-06T07:30:21.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.473+0000 7f1e279f6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1e200bf9d0 0x7f1e20155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.473+0000 7f1e279f6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e200c2fc0 con 0x7f1e20152650 2024-03-06T07:30:21.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.473+0000 7f1e279f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e20158da0 con 0x7f1e200c6b10 2024-03-06T07:30:21.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.473+0000 7f1e279f6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e20155800 con 0x7f1e200bf9d0 2024-03-06T07:30:21.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.473+0000 7f1e25792640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1e20152650 0x7f1e20158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52726/0 (socket says 172.21.15.1:52726) 2024-03-06T07:30:21.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.473+0000 7f1e25792640 1 -- 172.21.15.1:0/2019524540 learned_addr learned my addr 172.21.15.1:0/2019524540 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:21.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e17fff640 1 -- 172.21.15.1:0/2019524540 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892333214 0 0) 0x7f1e200c2fc0 con 0x7f1e20152650 2024-03-06T07:30:21.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e25f93640 1 --2- 172.21.15.1:0/2019524540 >> v2:172.21.15.98:3300/0 conn(0x7f1e200bf9d0 0x7f1e20155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e17fff640 1 -- 172.21.15.1:0/2019524540 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e0c003520 con 0x7f1e20152650 2024-03-06T07:30:21.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e17fff640 1 -- 172.21.15.1:0/2019524540 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 101241349 0 0) 0x7f1e0c003520 con 0x7f1e20152650 2024-03-06T07:30:21.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e17fff640 1 -- 172.21.15.1:0/2019524540 >> v2:172.21.15.98:3300/0 conn(0x7f1e200bf9d0 msgr2=0x7f1e20155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e17fff640 1 --2- 172.21.15.1:0/2019524540 >> v2:172.21.15.98:3300/0 conn(0x7f1e200bf9d0 0x7f1e20155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e17fff640 1 -- 172.21.15.1:0/2019524540 >> v1:172.21.15.98:6789/0 conn(0x7f1e200c6b10 legacy=0x7f1e200c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e17fff640 1 -- 172.21.15.1:0/2019524540 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e200c6690 con 0x7f1e20152650 2024-03-06T07:30:21.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e17fff640 1 -- 172.21.15.1:0/2019524540 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f1e10001670 con 0x7f1e20152650 2024-03-06T07:30:21.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e17fff640 1 -- 172.21.15.1:0/2019524540 >> v1:172.21.15.1:6789/0 conn(0x7f1e20152650 legacy=0x7f1e20158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e17fff640 1 --2- 172.21.15.1:0/2019524540 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e0c0039b0 0x7f1e0c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e17fff640 1 --2- 172.21.15.1:0/2019524540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e0c004800 0x7f1e0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e17fff640 1 --2- 172.21.15.1:0/2019524540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0c0051c0 0x7f1e0c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e17fff640 1 -- 172.21.15.1:0/2019524540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e20155800 con 0x7f1e0c004800 2024-03-06T07:30:21.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e17fff640 1 -- 172.21.15.1:0/2019524540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c00c0c0 con 0x7f1e0c0051c0 2024-03-06T07:30:21.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e17fff640 1 -- 172.21.15.1:0/2019524540 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c00c2c0 con 0x7f1e0c0039b0 2024-03-06T07:30:21.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e24f91640 1 --2- 172.21.15.1:0/2019524540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e0c004800 0x7f1e0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e25792640 1 --2- 172.21.15.1:0/2019524540 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e0c0039b0 0x7f1e0c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e25f93640 1 --2- 172.21.15.1:0/2019524540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0c0051c0 0x7f1e0c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e24f91640 1 -- 172.21.15.1:0/2019524540 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e0c0039b0 msgr2=0x7f1e0c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e24f91640 1 --2- 172.21.15.1:0/2019524540 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e0c0039b0 0x7f1e0c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e24f91640 1 -- 172.21.15.1:0/2019524540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0c0051c0 msgr2=0x7f1e0c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.474+0000 7f1e24f91640 1 --2- 172.21.15.1:0/2019524540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0c0051c0 0x7f1e0c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.475+0000 7f1e25f93640 1 -- 172.21.15.1:0/2019524540 reap_dead start 2024-03-06T07:30:21.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.475+0000 7f1e24f91640 1 -- 172.21.15.1:0/2019524540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0c003520 con 0x7f1e0c004800 2024-03-06T07:30:21.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.475+0000 7f1e24f91640 1 --2- 172.21.15.1:0/2019524540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e0c004800 0x7f1e0c004c80 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c001860 tx=0x7f1e0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff040a18c8598df2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.475+0000 7f1e17fff640 1 -- 172.21.15.1:0/2019524540 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c001430 con 0x7f1e0c004800 2024-03-06T07:30:21.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.475+0000 7f1e17fff640 1 -- 172.21.15.1:0/2019524540 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1c033610 con 0x7f1e0c004800 2024-03-06T07:30:21.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.475+0000 7f1e17fff640 1 -- 172.21.15.1:0/2019524540 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c033a00 con 0x7f1e0c004800 2024-03-06T07:30:21.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.476+0000 7f1e279f6640 1 -- 172.21.15.1:0/2019524540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e0c004800 msgr2=0x7f1e0c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.476+0000 7f1e279f6640 1 --2- 172.21.15.1:0/2019524540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e0c004800 0x7f1e0c004c80 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c001860 tx=0x7f1e0c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:21.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.476+0000 7f1e279f6640 1 -- 172.21.15.1:0/2019524540 shutdown_connections 2024-03-06T07:30:21.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.476+0000 7f1e279f6640 1 --2- 172.21.15.1:0/2019524540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e0c004800 0x7f1e0c004c80 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.476+0000 7f1e279f6640 1 -- 172.21.15.1:0/2019524540 >> 172.21.15.1:0/2019524540 conn(0x7f1e20066ee0 msgr2=0x7f1e200672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.476+0000 7f1e279f6640 1 -- 172.21.15.1:0/2019524540 shutdown_connections 2024-03-06T07:30:21.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.476+0000 7f1e279f6640 1 -- 172.21.15.1:0/2019524540 wait complete. 2024-03-06T07:30:21.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.477+0000 7f1e279f6640 1 Processor -- start 2024-03-06T07:30:21.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.477+0000 7f1e279f6640 1 -- start start 2024-03-06T07:30:21.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.478+0000 7f1e279f6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e0c004800 0x7f1e200af970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.478+0000 7f1e279f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e20158950 0x7f1e201ee300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.478+0000 7f1e279f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e201ee840 0x7f1e20156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.478+0000 7f1e279f6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e201551b0 con 0x7f1e0c004800 2024-03-06T07:30:21.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.478+0000 7f1e279f6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e200c6fa0 con 0x7f1e201ee840 2024-03-06T07:30:21.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.478+0000 7f1e279f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e20155490 con 0x7f1e20158950 2024-03-06T07:30:21.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.478+0000 7f1e25792640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e0c004800 0x7f1e200af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.478+0000 7f1e25792640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e0c004800 0x7f1e200af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51390/0 (socket says 172.21.15.1:51390) 2024-03-06T07:30:21.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.478+0000 7f1e25792640 1 -- 172.21.15.1:0/941111121 learned_addr learned my addr 172.21.15.1:0/941111121 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:21.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.478+0000 7f1e25792640 1 -- 172.21.15.1:0/941111121 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e20158950 msgr2=0x7f1e201ee300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.478+0000 7f1e25792640 1 --2- 172.21.15.1:0/941111121 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e20158950 0x7f1e201ee300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.478+0000 7f1e25792640 1 -- 172.21.15.1:0/941111121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e201ee840 msgr2=0x7f1e20156e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.478+0000 7f1e25792640 1 --2- 172.21.15.1:0/941111121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e201ee840 0x7f1e20156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.478+0000 7f1e25792640 1 -- 172.21.15.1:0/941111121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e200afeb0 con 0x7f1e0c004800 2024-03-06T07:30:21.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.478+0000 7f1e25792640 1 --2- 172.21.15.1:0/941111121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e0c004800 0x7f1e200af970 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f1e10001ab0 tx=0x7f1e10007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.479+0000 7f1e15ffb640 1 -- 172.21.15.1:0/941111121 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e10007770 con 0x7f1e0c004800 2024-03-06T07:30:21.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.479+0000 7f1e15ffb640 1 -- 172.21.15.1:0/941111121 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e10018070 con 0x7f1e0c004800 2024-03-06T07:30:21.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.479+0000 7f1e279f6640 1 -- 172.21.15.1:0/941111121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2014e6d0 con 0x7f1e0c004800 2024-03-06T07:30:21.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.479+0000 7f1e15ffb640 1 -- 172.21.15.1:0/941111121 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1000f8e0 con 0x7f1e0c004800 2024-03-06T07:30:21.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.479+0000 7f1e279f6640 1 -- 172.21.15.1:0/941111121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2014e3f0 con 0x7f1e0c004800 2024-03-06T07:30:21.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.480+0000 7f1e279f6640 1 -- 172.21.15.1:0/941111121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e20067fd0 con 0x7f1e0c004800 2024-03-06T07:30:21.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.480+0000 7f1e15ffb640 1 -- 172.21.15.1:0/941111121 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e1000fa80 con 0x7f1e0c004800 2024-03-06T07:30:21.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.481+0000 7f1e15ffb640 1 --2- 172.21.15.1:0/941111121 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e0406b4d0 0x7f1e0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.481+0000 7f1e15ffb640 1 -- 172.21.15.1:0/941111121 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f1e1008a1f0 con 0x7f1e0c004800 2024-03-06T07:30:21.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.481+0000 7f1e24f91640 1 --2- 172.21.15.1:0/941111121 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e0406b4d0 0x7f1e0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.481+0000 7f1e24f91640 1 --2- 172.21.15.1:0/941111121 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e0406b4d0 0x7f1e0406d990 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c036000 tx=0x7f1e1c009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.485+0000 7f1e15ffb640 1 -- 172.21.15.1:0/941111121 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e10058e30 con 0x7f1e0c004800 2024-03-06T07:30:21.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.867+0000 7f1e279f6640 1 -- 172.21.15.1:0/941111121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e200b06f0 con 0x7f1e0c004800 2024-03-06T07:30:21.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.867+0000 7f1e15ffb640 1 -- 172.21.15.1:0/941111121 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4216 (secure 0 0 0) 0x7f1e10058730 con 0x7f1e0c004800 2024-03-06T07:30:21.870 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:21.871 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:18.318227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:stopping","state_seq":16,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:21.871 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 206 2024-03-06T07:30:21.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.870+0000 7f1e279f6640 1 -- 172.21.15.1:0/941111121 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e0406b4d0 msgr2=0x7f1e0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.870+0000 7f1e279f6640 1 --2- 172.21.15.1:0/941111121 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e0406b4d0 0x7f1e0406d990 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c036000 tx=0x7f1e1c009030 comp rx=0 tx=0).stop 2024-03-06T07:30:21.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.870+0000 7f1e279f6640 1 -- 172.21.15.1:0/941111121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e0c004800 msgr2=0x7f1e200af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.870+0000 7f1e279f6640 1 --2- 172.21.15.1:0/941111121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e0c004800 0x7f1e200af970 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f1e10001ab0 tx=0x7f1e10007430 comp rx=0 tx=0).stop 2024-03-06T07:30:21.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.870+0000 7f1e279f6640 1 -- 172.21.15.1:0/941111121 shutdown_connections 2024-03-06T07:30:21.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.870+0000 7f1e279f6640 1 --2- 172.21.15.1:0/941111121 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1e0406b4d0 0x7f1e0406d990 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.870+0000 7f1e279f6640 1 --2- 172.21.15.1:0/941111121 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e201ee840 0x7f1e20156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.870+0000 7f1e279f6640 1 --2- 172.21.15.1:0/941111121 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e20158950 0x7f1e201ee300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.870+0000 7f1e279f6640 1 --2- 172.21.15.1:0/941111121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e0c004800 0x7f1e200af970 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.870+0000 7f1e279f6640 1 -- 172.21.15.1:0/941111121 >> 172.21.15.1:0/941111121 conn(0x7f1e20066ee0 msgr2=0x7f1e200cac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.870+0000 7f1e279f6640 1 -- 172.21.15.1:0/941111121 shutdown_connections 2024-03-06T07:30:21.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:21.870+0000 7f1e279f6640 1 -- 172.21.15.1:0/941111121 wait complete. 2024-03-06T07:30:22.284 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:18.318227+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:22.285 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:stopping 2024-03-06T07:30:23.285 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:23.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.385+0000 7f26b2c9a640 1 Processor -- start 2024-03-06T07:30:23.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.385+0000 7f26b2c9a640 1 -- start start 2024-03-06T07:30:23.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.386+0000 7f26b2c9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26ac150520 0x7f26ac1543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.386+0000 7f26b2c9a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26ac070c80 0x7f26ac0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.386+0000 7f26b2c9a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ac150900 con 0x7f26ac0c32f0 2024-03-06T07:30:23.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.386+0000 7f26b2c9a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26ac153ed0 con 0x7f26ac070c80 2024-03-06T07:30:23.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.386+0000 7f26b2c9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac071790 con 0x7f26ac150520 2024-03-06T07:30:23.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.386+0000 7f26b1237640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26ac070c80 0x7f26ac0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.386+0000 7f26b1237640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26ac070c80 0x7f26ac0710f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51394/0 (socket says 172.21.15.1:51394) 2024-03-06T07:30:23.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.386+0000 7f26b1237640 1 -- 172.21.15.1:0/3697764405 learned_addr learned my addr 172.21.15.1:0/3697764405 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:23.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.386+0000 7f26b0a36640 1 --2- 172.21.15.1:0/3697764405 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26ac150520 0x7f26ac1543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.387+0000 7f26b1237640 1 -- 172.21.15.1:0/3697764405 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26ac150520 msgr2=0x7f26ac1543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.387+0000 7f26b1237640 1 --2- 172.21.15.1:0/3697764405 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26ac150520 0x7f26ac1543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.387+0000 7f26b1237640 1 -- 172.21.15.1:0/3697764405 >> v1:172.21.15.98:6789/0 conn(0x7f26ac0c32f0 legacy=0x7f26ac1537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.387+0000 7f26b1237640 1 -- 172.21.15.1:0/3697764405 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ac157050 con 0x7f26ac070c80 2024-03-06T07:30:23.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.387+0000 7f26b0a36640 1 --2- 172.21.15.1:0/3697764405 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26ac150520 0x7f26ac1543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.388+0000 7f26b1237640 1 --2- 172.21.15.1:0/3697764405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ac070c80 0x7f26ac0710f0 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f26a4009e50 tx=0x7f26a402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b7ad08a6a51bdf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.388+0000 7f26ab7fe640 1 -- 172.21.15.1:0/3697764405 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a402cb00 con 0x7f26ac070c80 2024-03-06T07:30:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.388+0000 7f26ab7fe640 1 -- 172.21.15.1:0/3697764405 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a402cca0 con 0x7f26ac070c80 2024-03-06T07:30:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.388+0000 7f26ab7fe640 1 -- 172.21.15.1:0/3697764405 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a4034610 con 0x7f26ac070c80 2024-03-06T07:30:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.388+0000 7f26b2c9a640 1 -- 172.21.15.1:0/3697764405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ac070c80 msgr2=0x7f26ac0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.388+0000 7f26b2c9a640 1 --2- 172.21.15.1:0/3697764405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ac070c80 0x7f26ac0710f0 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f26a4009e50 tx=0x7f26a402bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.388+0000 7f26b2c9a640 1 -- 172.21.15.1:0/3697764405 shutdown_connections 2024-03-06T07:30:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.388+0000 7f26b2c9a640 1 --2- 172.21.15.1:0/3697764405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ac070c80 0x7f26ac0710f0 unknown :-1 s=CLOSED pgs=2421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.388+0000 7f26b2c9a640 1 --2- 172.21.15.1:0/3697764405 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26ac150520 0x7f26ac1543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.388+0000 7f26b2c9a640 1 -- 172.21.15.1:0/3697764405 >> 172.21.15.1:0/3697764405 conn(0x7f26ac0c88f0 msgr2=0x7f26ac0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.388+0000 7f26b2c9a640 1 -- 172.21.15.1:0/3697764405 shutdown_connections 2024-03-06T07:30:23.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.388+0000 7f26b2c9a640 1 -- 172.21.15.1:0/3697764405 wait complete. 2024-03-06T07:30:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.389+0000 7f26b2c9a640 1 Processor -- start 2024-03-06T07:30:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.390+0000 7f26b2c9a640 1 -- start start 2024-03-06T07:30:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.390+0000 7f26b2c9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ac070c80 0x7f26ac14f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.390+0000 7f26b2c9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26ac0c32f0 0x7f26ac1ebd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.390+0000 7f26b2c9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26ac150520 0x7f26ac1ec290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.390+0000 7f26b2c9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac157330 con 0x7f26ac070c80 2024-03-06T07:30:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.390+0000 7f26b2c9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac1532a0 con 0x7f26ac150520 2024-03-06T07:30:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.390+0000 7f26b2c9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac0c7ec0 con 0x7f26ac0c32f0 2024-03-06T07:30:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.390+0000 7f26b0a36640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ac070c80 0x7f26ac14f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.390+0000 7f26b0a36640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ac070c80 0x7f26ac14f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51404/0 (socket says 172.21.15.1:51404) 2024-03-06T07:30:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.390+0000 7f26b0a36640 1 -- 172.21.15.1:0/1280990669 learned_addr learned my addr 172.21.15.1:0/1280990669 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.391+0000 7f26b1237640 1 --2- 172.21.15.1:0/1280990669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26ac150520 0x7f26ac1ec290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.391+0000 7f26abfff640 1 --2- 172.21.15.1:0/1280990669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26ac0c32f0 0x7f26ac1ebd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.391+0000 7f26b0a36640 1 -- 172.21.15.1:0/1280990669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26ac0c32f0 msgr2=0x7f26ac1ebd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.391+0000 7f26b0a36640 1 --2- 172.21.15.1:0/1280990669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26ac0c32f0 0x7f26ac1ebd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.391+0000 7f26b0a36640 1 -- 172.21.15.1:0/1280990669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26ac150520 msgr2=0x7f26ac1ec290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.391+0000 7f26b0a36640 1 --2- 172.21.15.1:0/1280990669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26ac150520 0x7f26ac1ec290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.391+0000 7f26b0a36640 1 -- 172.21.15.1:0/1280990669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ac1ec7d0 con 0x7f26ac070c80 2024-03-06T07:30:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.391+0000 7f26b0a36640 1 --2- 172.21.15.1:0/1280990669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ac070c80 0x7f26ac14f7c0 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f26a000d7d0 tx=0x7f26a000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.391+0000 7f26a97fa640 1 -- 172.21.15.1:0/1280990669 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a0004030 con 0x7f26ac070c80 2024-03-06T07:30:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.391+0000 7f26a97fa640 1 -- 172.21.15.1:0/1280990669 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a00041d0 con 0x7f26ac070c80 2024-03-06T07:30:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.391+0000 7f26b2c9a640 1 -- 172.21.15.1:0/1280990669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26ac156010 con 0x7f26ac070c80 2024-03-06T07:30:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.391+0000 7f26a97fa640 1 -- 172.21.15.1:0/1280990669 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a00096a0 con 0x7f26ac070c80 2024-03-06T07:30:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.391+0000 7f26b2c9a640 1 -- 172.21.15.1:0/1280990669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26ac14ed00 con 0x7f26ac070c80 2024-03-06T07:30:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.392+0000 7f26b2c9a640 1 -- 172.21.15.1:0/1280990669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26ac0686e0 con 0x7f26ac070c80 2024-03-06T07:30:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.393+0000 7f26a97fa640 1 -- 172.21.15.1:0/1280990669 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26a0009840 con 0x7f26ac070c80 2024-03-06T07:30:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.393+0000 7f26a97fa640 1 --2- 172.21.15.1:0/1280990669 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f269006b4d0 0x7f269006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.393+0000 7f26a97fa640 1 -- 172.21.15.1:0/1280990669 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f26a008a6e0 con 0x7f26ac070c80 2024-03-06T07:30:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.393+0000 7f26abfff640 1 --2- 172.21.15.1:0/1280990669 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f269006b4d0 0x7f269006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.394+0000 7f26abfff640 1 --2- 172.21.15.1:0/1280990669 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f269006b4d0 0x7f269006d990 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f26ac152910 tx=0x7f2698009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.401+0000 7f26a97fa640 1 -- 172.21.15.1:0/1280990669 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26a00593b0 con 0x7f26ac070c80 2024-03-06T07:30:23.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.620+0000 7f26b2c9a640 1 -- 172.21.15.1:0/1280990669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26ac0adea0 con 0x7f26ac070c80 2024-03-06T07:30:23.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.620+0000 7f26a97fa640 1 -- 172.21.15.1:0/1280990669 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4216 (secure 0 0 0) 0x7f26a0058cb0 con 0x7f26ac070c80 2024-03-06T07:30:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:23.622 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:18.318227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:stopping","state_seq":16,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:23.623 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 206 2024-03-06T07:30:23.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.622+0000 7f26b2c9a640 1 -- 172.21.15.1:0/1280990669 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f269006b4d0 msgr2=0x7f269006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.622+0000 7f26b2c9a640 1 --2- 172.21.15.1:0/1280990669 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f269006b4d0 0x7f269006d990 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f26ac152910 tx=0x7f2698009040 comp rx=0 tx=0).stop 2024-03-06T07:30:23.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.622+0000 7f26b2c9a640 1 -- 172.21.15.1:0/1280990669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ac070c80 msgr2=0x7f26ac14f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.622+0000 7f26b2c9a640 1 --2- 172.21.15.1:0/1280990669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ac070c80 0x7f26ac14f7c0 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f26a000d7d0 tx=0x7f26a000dc90 comp rx=0 tx=0).stop 2024-03-06T07:30:23.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.622+0000 7f26b2c9a640 1 -- 172.21.15.1:0/1280990669 shutdown_connections 2024-03-06T07:30:23.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.622+0000 7f26b2c9a640 1 --2- 172.21.15.1:0/1280990669 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f269006b4d0 0x7f269006d990 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.622+0000 7f26b2c9a640 1 --2- 172.21.15.1:0/1280990669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26ac150520 0x7f26ac1ec290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.622+0000 7f26b2c9a640 1 --2- 172.21.15.1:0/1280990669 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26ac0c32f0 0x7f26ac1ebd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.622+0000 7f26b2c9a640 1 --2- 172.21.15.1:0/1280990669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ac070c80 0x7f26ac14f7c0 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.622+0000 7f26b2c9a640 1 -- 172.21.15.1:0/1280990669 >> 172.21.15.1:0/1280990669 conn(0x7f26ac0c88f0 msgr2=0x7f26ac071060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.622+0000 7f26b2c9a640 1 -- 172.21.15.1:0/1280990669 shutdown_connections 2024-03-06T07:30:23.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:23.622+0000 7f26b2c9a640 1 -- 172.21.15.1:0/1280990669 wait complete. 2024-03-06T07:30:24.038 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:18.318227+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:24.038 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:stopping 2024-03-06T07:30:25.039 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:25.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.138+0000 7faa8ba9a640 1 Processor -- start 2024-03-06T07:30:25.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.139+0000 7faa8ba9a640 1 -- start start 2024-03-06T07:30:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.140+0000 7faa8ba9a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faa84150520 0x7faa841543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.140+0000 7faa8ba9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa840c32f0 0x7faa841537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.140+0000 7faa8ba9a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faa84070c80 0x7faa840710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.140+0000 7faa8ba9a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa84150900 con 0x7faa84150520 2024-03-06T07:30:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.140+0000 7faa8ba9a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa84071790 con 0x7faa84070c80 2024-03-06T07:30:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.140+0000 7faa8ba9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa84157050 con 0x7faa840c32f0 2024-03-06T07:30:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.140+0000 7faa89836640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faa84150520 0x7faa841543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.140+0000 7faa89836640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faa84150520 0x7faa841543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43896/0 (socket says 172.21.15.1:43896) 2024-03-06T07:30:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.140+0000 7faa89836640 1 -- 172.21.15.1:0/823008523 learned_addr learned my addr 172.21.15.1:0/823008523 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.140+0000 7faa8a037640 1 --2- 172.21.15.1:0/823008523 >> v2:172.21.15.98:3300/0 conn(0x7faa84070c80 0x7faa840710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.140+0000 7faa89035640 1 --2- 172.21.15.1:0/823008523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa840c32f0 0x7faa841537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.140+0000 7faa89836640 1 -- 172.21.15.1:0/823008523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa840c32f0 msgr2=0x7faa841537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.140+0000 7faa89836640 1 --2- 172.21.15.1:0/823008523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa840c32f0 0x7faa841537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.140+0000 7faa89836640 1 -- 172.21.15.1:0/823008523 >> v2:172.21.15.98:3300/0 conn(0x7faa84070c80 msgr2=0x7faa840710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.140+0000 7faa89836640 1 --2- 172.21.15.1:0/823008523 >> v2:172.21.15.98:3300/0 conn(0x7faa84070c80 0x7faa840710f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.140+0000 7faa89836640 1 -- 172.21.15.1:0/823008523 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa84152fc0 con 0x7faa84150520 2024-03-06T07:30:25.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.140+0000 7faa89035640 1 --2- 172.21.15.1:0/823008523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa840c32f0 0x7faa841537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:25.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.141+0000 7faa89836640 1 --2- 172.21.15.1:0/823008523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa84150520 0x7faa841543a0 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7faa840c9990 tx=0x7faa7402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52843d228583aeb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.141+0000 7faa88834640 1 -- 172.21.15.1:0/823008523 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa74035020 con 0x7faa84150520 2024-03-06T07:30:25.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.141+0000 7faa88834640 1 -- 172.21.15.1:0/823008523 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa74031870 con 0x7faa84150520 2024-03-06T07:30:25.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.141+0000 7faa88834640 1 -- 172.21.15.1:0/823008523 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa74031c40 con 0x7faa84150520 2024-03-06T07:30:25.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.142+0000 7faa8ba9a640 1 -- 172.21.15.1:0/823008523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa84150520 msgr2=0x7faa841543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.142+0000 7faa8ba9a640 1 --2- 172.21.15.1:0/823008523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa84150520 0x7faa841543a0 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7faa840c9990 tx=0x7faa7402b910 comp rx=0 tx=0).stop 2024-03-06T07:30:25.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.142+0000 7faa8ba9a640 1 -- 172.21.15.1:0/823008523 shutdown_connections 2024-03-06T07:30:25.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.142+0000 7faa8ba9a640 1 --2- 172.21.15.1:0/823008523 >> v2:172.21.15.98:3300/0 conn(0x7faa84070c80 0x7faa840710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.142+0000 7faa8ba9a640 1 --2- 172.21.15.1:0/823008523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa840c32f0 0x7faa841537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.142+0000 7faa8ba9a640 1 --2- 172.21.15.1:0/823008523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa84150520 0x7faa841543a0 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.142+0000 7faa8ba9a640 1 -- 172.21.15.1:0/823008523 >> 172.21.15.1:0/823008523 conn(0x7faa840c88f0 msgr2=0x7faa840c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.142+0000 7faa8ba9a640 1 -- 172.21.15.1:0/823008523 shutdown_connections 2024-03-06T07:30:25.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.142+0000 7faa8ba9a640 1 -- 172.21.15.1:0/823008523 wait complete. 2024-03-06T07:30:25.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.143+0000 7faa8ba9a640 1 Processor -- start 2024-03-06T07:30:25.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.143+0000 7faa8ba9a640 1 -- start start 2024-03-06T07:30:25.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.143+0000 7faa8ba9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa84070c80 0x7faa841557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.143+0000 7faa8ba9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa840c32f0 0x7faa841ebb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.143+0000 7faa8ba9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa84150520 0x7faa84143720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.143+0000 7faa8ba9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa840c7ec0 con 0x7faa84150520 2024-03-06T07:30:25.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.143+0000 7faa8ba9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa84157330 con 0x7faa84070c80 2024-03-06T07:30:25.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.143+0000 7faa8ba9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa84153d00 con 0x7faa840c32f0 2024-03-06T07:30:25.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa8a037640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa84150520 0x7faa84143720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa8a037640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa84150520 0x7faa84143720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43910/0 (socket says 172.21.15.1:43910) 2024-03-06T07:30:25.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa8a037640 1 -- 172.21.15.1:0/838002918 learned_addr learned my addr 172.21.15.1:0/838002918 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:25.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa89035640 1 --2- 172.21.15.1:0/838002918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa840c32f0 0x7faa841ebb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa89836640 1 --2- 172.21.15.1:0/838002918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa84070c80 0x7faa841557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa8a037640 1 -- 172.21.15.1:0/838002918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa840c32f0 msgr2=0x7faa841ebb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa8a037640 1 --2- 172.21.15.1:0/838002918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa840c32f0 0x7faa841ebb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa8a037640 1 -- 172.21.15.1:0/838002918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa84070c80 msgr2=0x7faa841557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa8a037640 1 --2- 172.21.15.1:0/838002918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa84070c80 0x7faa841557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa8a037640 1 -- 172.21.15.1:0/838002918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa840c55d0 con 0x7faa84150520 2024-03-06T07:30:25.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa89035640 1 --2- 172.21.15.1:0/838002918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa840c32f0 0x7faa841ebb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa89836640 1 --2- 172.21.15.1:0/838002918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa84070c80 0x7faa841557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa8a037640 1 --2- 172.21.15.1:0/838002918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa84150520 0x7faa84143720 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7faa7c007e90 tx=0x7faa7c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa7a7fc640 1 -- 172.21.15.1:0/838002918 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c014420 con 0x7faa84150520 2024-03-06T07:30:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa7a7fc640 1 -- 172.21.15.1:0/838002918 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7c019070 con 0x7faa84150520 2024-03-06T07:30:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa8ba9a640 1 -- 172.21.15.1:0/838002918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa84144c60 con 0x7faa84150520 2024-03-06T07:30:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa7a7fc640 1 -- 172.21.15.1:0/838002918 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c014820 con 0x7faa84150520 2024-03-06T07:30:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.144+0000 7faa8ba9a640 1 -- 172.21.15.1:0/838002918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa84145220 con 0x7faa84150520 2024-03-06T07:30:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.145+0000 7faa8ba9a640 1 -- 172.21.15.1:0/838002918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa840686e0 con 0x7faa84150520 2024-03-06T07:30:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.146+0000 7faa7a7fc640 1 -- 172.21.15.1:0/838002918 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa7c0149c0 con 0x7faa84150520 2024-03-06T07:30:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.146+0000 7faa7a7fc640 1 --2- 172.21.15.1:0/838002918 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faa6806b4d0 0x7faa6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.146+0000 7faa7a7fc640 1 -- 172.21.15.1:0/838002918 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7faa7c08ba80 con 0x7faa84150520 2024-03-06T07:30:25.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.147+0000 7faa89836640 1 --2- 172.21.15.1:0/838002918 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faa6806b4d0 0x7faa6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.147+0000 7faa89836640 1 --2- 172.21.15.1:0/838002918 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faa6806b4d0 0x7faa6806d990 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7faa74036000 tx=0x7faa7403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.150+0000 7faa7a7fc640 1 -- 172.21.15.1:0/838002918 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa7c05a830 con 0x7faa84150520 2024-03-06T07:30:25.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.369+0000 7faa8ba9a640 1 -- 172.21.15.1:0/838002918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa840c7b10 con 0x7faa84150520 2024-03-06T07:30:25.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.369+0000 7faa7a7fc640 1 -- 172.21.15.1:0/838002918 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4216 (secure 0 0 0) 0x7faa7c05a130 con 0x7faa84150520 2024-03-06T07:30:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:25.371 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:18.318227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:stopping","state_seq":16,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:25.371 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 206 2024-03-06T07:30:25.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.371+0000 7faa8ba9a640 1 -- 172.21.15.1:0/838002918 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faa6806b4d0 msgr2=0x7faa6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.371+0000 7faa8ba9a640 1 --2- 172.21.15.1:0/838002918 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faa6806b4d0 0x7faa6806d990 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7faa74036000 tx=0x7faa7403c040 comp rx=0 tx=0).stop 2024-03-06T07:30:25.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.371+0000 7faa8ba9a640 1 -- 172.21.15.1:0/838002918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa84150520 msgr2=0x7faa84143720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.371+0000 7faa8ba9a640 1 --2- 172.21.15.1:0/838002918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa84150520 0x7faa84143720 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7faa7c007e90 tx=0x7faa7c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.371+0000 7faa8ba9a640 1 -- 172.21.15.1:0/838002918 shutdown_connections 2024-03-06T07:30:25.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.371+0000 7faa8ba9a640 1 --2- 172.21.15.1:0/838002918 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faa6806b4d0 0x7faa6806d990 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.371+0000 7faa8ba9a640 1 --2- 172.21.15.1:0/838002918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa84150520 0x7faa84143720 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.371+0000 7faa8ba9a640 1 --2- 172.21.15.1:0/838002918 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa840c32f0 0x7faa841ebb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.371+0000 7faa8ba9a640 1 --2- 172.21.15.1:0/838002918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa84070c80 0x7faa841557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.371+0000 7faa8ba9a640 1 -- 172.21.15.1:0/838002918 >> 172.21.15.1:0/838002918 conn(0x7faa840c88f0 msgr2=0x7faa840c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.371+0000 7faa8ba9a640 1 -- 172.21.15.1:0/838002918 shutdown_connections 2024-03-06T07:30:25.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:25.371+0000 7faa8ba9a640 1 -- 172.21.15.1:0/838002918 wait complete. 2024-03-06T07:30:25.786 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:18.318227+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:25.786 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:stopping 2024-03-06T07:30:26.787 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:26.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.887+0000 7fd6e4884640 1 Processor -- start 2024-03-06T07:30:26.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.887+0000 7fd6e4884640 1 -- start start 2024-03-06T07:30:26.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.887+0000 7fd6e4884640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6e0152650 0x7fd6e0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.887+0000 7fd6e4884640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd6e00c6b10 0x7fd6e0158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.888+0000 7fd6e4884640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e00c2fc0 con 0x7fd6e00bf9d0 2024-03-06T07:30:26.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.888+0000 7fd6e4884640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e0155800 con 0x7fd6e00c6b10 2024-03-06T07:30:26.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.888+0000 7fd6e4884640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e00c6690 con 0x7fd6e0152650 2024-03-06T07:30:26.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.888+0000 7fd6d7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd6e00c6b10 0x7fd6e0158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.888+0000 7fd6ded9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd6e00bf9d0 0x7fd6e00c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52200/0 (socket says 172.21.15.1:52200) 2024-03-06T07:30:26.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.888+0000 7fd6ded9d640 1 -- 172.21.15.1:0/2643514491 learned_addr learned my addr 172.21.15.1:0/2643514491 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:26.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.888+0000 7fd6d7fff640 1 --2- 172.21.15.1:0/2643514491 >> v2:172.21.15.1:3300/0 conn(0x7fd6e00c6b10 0x7fd6e0158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43926/0 (socket says 172.21.15.1:43926) 2024-03-06T07:30:26.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.888+0000 7fd6de59c640 1 --2- 172.21.15.1:0/2643514491 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6e0152650 0x7fd6e0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.888+0000 7fd6d7fff640 1 -- 172.21.15.1:0/2643514491 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6e0152650 msgr2=0x7fd6e0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.888+0000 7fd6d7fff640 1 --2- 172.21.15.1:0/2643514491 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6e0152650 0x7fd6e0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.888+0000 7fd6d7fff640 1 -- 172.21.15.1:0/2643514491 >> v1:172.21.15.1:6789/0 conn(0x7fd6e00bf9d0 legacy=0x7fd6e00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.888+0000 7fd6ddd9b640 1 -- 172.21.15.1:0/2643514491 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951521086 0 0) 0x7fd6e00c2fc0 con 0x7fd6e00bf9d0 2024-03-06T07:30:26.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.888+0000 7fd6d7fff640 1 -- 172.21.15.1:0/2643514491 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e00c7280 con 0x7fd6e00c6b10 2024-03-06T07:30:26.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.889+0000 7fd6d7fff640 1 --2- 172.21.15.1:0/2643514491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6e00c6b10 0x7fd6e0158690 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80097a0 tx=0x7fd6c802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe4f9e75c11e071b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.889+0000 7fd6ddd9b640 1 -- 172.21.15.1:0/2643514491 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80315a0 con 0x7fd6e00c6b10 2024-03-06T07:30:26.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.889+0000 7fd6ddd9b640 1 -- 172.21.15.1:0/2643514491 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c8037070 con 0x7fd6e00c6b10 2024-03-06T07:30:26.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.889+0000 7fd6ddd9b640 1 -- 172.21.15.1:0/2643514491 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8031960 con 0x7fd6e00c6b10 2024-03-06T07:30:26.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.889+0000 7fd6e4884640 1 -- 172.21.15.1:0/2643514491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6e00c6b10 msgr2=0x7fd6e0158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.889+0000 7fd6e4884640 1 --2- 172.21.15.1:0/2643514491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6e00c6b10 0x7fd6e0158690 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80097a0 tx=0x7fd6c802b840 comp rx=0 tx=0).stop 2024-03-06T07:30:26.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.889+0000 7fd6e4884640 1 -- 172.21.15.1:0/2643514491 shutdown_connections 2024-03-06T07:30:26.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.889+0000 7fd6e4884640 1 --2- 172.21.15.1:0/2643514491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6e00c6b10 0x7fd6e0158690 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.889+0000 7fd6e4884640 1 --2- 172.21.15.1:0/2643514491 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6e0152650 0x7fd6e0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.889+0000 7fd6e4884640 1 -- 172.21.15.1:0/2643514491 >> 172.21.15.1:0/2643514491 conn(0x7fd6e0066ee0 msgr2=0x7fd6e00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.890+0000 7fd6e4884640 1 -- 172.21.15.1:0/2643514491 shutdown_connections 2024-03-06T07:30:26.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.890+0000 7fd6e4884640 1 -- 172.21.15.1:0/2643514491 wait complete. 2024-03-06T07:30:26.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.891+0000 7fd6e4884640 1 Processor -- start 2024-03-06T07:30:26.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.891+0000 7fd6e4884640 1 -- start start 2024-03-06T07:30:26.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.891+0000 7fd6e4884640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6e00bf9d0 0x7fd6e01e0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.891+0000 7fd6e4884640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e0152650 0x7fd6e01e0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.891+0000 7fd6e4884640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6e0142250 0x7fd6e01ee710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.891+0000 7fd6e4884640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e0155490 con 0x7fd6e0142250 2024-03-06T07:30:26.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.891+0000 7fd6e4884640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e01551b0 con 0x7fd6e0152650 2024-03-06T07:30:26.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.891+0000 7fd6e4884640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e0152a30 con 0x7fd6e00bf9d0 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.892+0000 7fd6ded9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6e0142250 0x7fd6e01ee710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.892+0000 7fd6ded9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6e0142250 0x7fd6e01ee710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43940/0 (socket says 172.21.15.1:43940) 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.892+0000 7fd6ded9d640 1 -- 172.21.15.1:0/3028074370 learned_addr learned my addr 172.21.15.1:0/3028074370 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.892+0000 7fd6de59c640 1 --2- 172.21.15.1:0/3028074370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6e00bf9d0 0x7fd6e01e0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.892+0000 7fd6d7fff640 1 --2- 172.21.15.1:0/3028074370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e0152650 0x7fd6e01e0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.892+0000 7fd6ded9d640 1 -- 172.21.15.1:0/3028074370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6e00bf9d0 msgr2=0x7fd6e01e0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.892+0000 7fd6ded9d640 1 --2- 172.21.15.1:0/3028074370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6e00bf9d0 0x7fd6e01e0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.892+0000 7fd6ded9d640 1 -- 172.21.15.1:0/3028074370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e0152650 msgr2=0x7fd6e01e0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.892+0000 7fd6ded9d640 1 --2- 172.21.15.1:0/3028074370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e0152650 0x7fd6e01e0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.892+0000 7fd6ded9d640 1 -- 172.21.15.1:0/3028074370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e01ef340 con 0x7fd6e0142250 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.892+0000 7fd6d7fff640 1 --2- 172.21.15.1:0/3028074370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e0152650 0x7fd6e01e0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.892+0000 7fd6ded9d640 1 --2- 172.21.15.1:0/3028074370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6e0142250 0x7fd6e01ee710 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7fd6d800c730 tx=0x7fd6d800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.892+0000 7fd6d6ffd640 1 -- 172.21.15.1:0/3028074370 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8018070 con 0x7fd6e0142250 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.892+0000 7fd6d6ffd640 1 -- 172.21.15.1:0/3028074370 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d800ce40 con 0x7fd6e0142250 2024-03-06T07:30:26.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.892+0000 7fd6e4884640 1 -- 172.21.15.1:0/3028074370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e014c1d0 con 0x7fd6e0142250 2024-03-06T07:30:26.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.892+0000 7fd6d6ffd640 1 -- 172.21.15.1:0/3028074370 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d800a860 con 0x7fd6e0142250 2024-03-06T07:30:26.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.892+0000 7fd6e4884640 1 -- 172.21.15.1:0/3028074370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e014dea0 con 0x7fd6e0142250 2024-03-06T07:30:26.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.893+0000 7fd6e4884640 1 -- 172.21.15.1:0/3028074370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e0067fd0 con 0x7fd6e0142250 2024-03-06T07:30:26.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.894+0000 7fd6d6ffd640 1 -- 172.21.15.1:0/3028074370 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6d801b070 con 0x7fd6e0142250 2024-03-06T07:30:26.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.894+0000 7fd6d6ffd640 1 --2- 172.21.15.1:0/3028074370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd6c006b500 0x7fd6c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.894+0000 7fd6d6ffd640 1 -- 172.21.15.1:0/3028074370 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7fd6d8089940 con 0x7fd6e0142250 2024-03-06T07:30:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.895+0000 7fd6de59c640 1 --2- 172.21.15.1:0/3028074370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd6c006b500 0x7fd6c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.895+0000 7fd6de59c640 1 --2- 172.21.15.1:0/3028074370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd6c006b500 0x7fd6c006d9c0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0006fd0 tx=0x7fd6d0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:26.898+0000 7fd6d6ffd640 1 -- 172.21.15.1:0/3028074370 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6d8057f30 con 0x7fd6e0142250 2024-03-06T07:30:27.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:27.141+0000 7fd6e4884640 1 -- 172.21.15.1:0/3028074370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6e00b06f0 con 0x7fd6e0142250 2024-03-06T07:30:27.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:27.141+0000 7fd6d6ffd640 1 -- 172.21.15.1:0/3028074370 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4216 (secure 0 0 0) 0x7fd6d805b090 con 0x7fd6e0142250 2024-03-06T07:30:27.143 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:27.144 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:18.318227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:stopping","state_seq":16,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 206 2024-03-06T07:30:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:27.143+0000 7fd6e4884640 1 -- 172.21.15.1:0/3028074370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd6c006b500 msgr2=0x7fd6c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:27.143+0000 7fd6e4884640 1 --2- 172.21.15.1:0/3028074370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd6c006b500 0x7fd6c006d9c0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0006fd0 tx=0x7fd6d0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:27.143+0000 7fd6e4884640 1 -- 172.21.15.1:0/3028074370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6e0142250 msgr2=0x7fd6e01ee710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:27.143+0000 7fd6e4884640 1 --2- 172.21.15.1:0/3028074370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6e0142250 0x7fd6e01ee710 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7fd6d800c730 tx=0x7fd6d800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:27.143+0000 7fd6e4884640 1 -- 172.21.15.1:0/3028074370 shutdown_connections 2024-03-06T07:30:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:27.143+0000 7fd6e4884640 1 --2- 172.21.15.1:0/3028074370 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd6c006b500 0x7fd6c006d9c0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:27.143+0000 7fd6e4884640 1 --2- 172.21.15.1:0/3028074370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6e0142250 0x7fd6e01ee710 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:27.143+0000 7fd6e4884640 1 --2- 172.21.15.1:0/3028074370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e0152650 0x7fd6e01e0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:27.143+0000 7fd6e4884640 1 --2- 172.21.15.1:0/3028074370 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6e00bf9d0 0x7fd6e01e0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:27.143+0000 7fd6e4884640 1 -- 172.21.15.1:0/3028074370 >> 172.21.15.1:0/3028074370 conn(0x7fd6e0066ee0 msgr2=0x7fd6e00c5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:27.144+0000 7fd6e4884640 1 -- 172.21.15.1:0/3028074370 shutdown_connections 2024-03-06T07:30:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:27.144+0000 7fd6e4884640 1 -- 172.21.15.1:0/3028074370 wait complete. 2024-03-06T07:30:27.558 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:18.318227+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:27.567 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:stopping 2024-03-06T07:30:28.567 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.705+0000 7fe6fce0e640 1 Processor -- start 2024-03-06T07:30:28.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.705+0000 7fe6fce0e640 1 -- start start 2024-03-06T07:30:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.706+0000 7fe6fce0e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe6f8152650 0x7fe6f8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.706+0000 7fe6fce0e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f80c2fc0 con 0x7fe6f80c6b10 2024-03-06T07:30:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.706+0000 7fe6fce0e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f8158da0 con 0x7fe6f80bf9d0 2024-03-06T07:30:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.706+0000 7fe6fce0e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f8155800 con 0x7fe6f8152650 2024-03-06T07:30:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.706+0000 7fe6f5d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe6f80c6b10 0x7fe6f8158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52214/0 (socket says 172.21.15.1:52214) 2024-03-06T07:30:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.706+0000 7fe6f5d9b640 1 -- 172.21.15.1:0/3219627689 learned_addr learned my addr 172.21.15.1:0/3219627689 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f659c640 1 --2- 172.21.15.1:0/3219627689 >> v2:172.21.15.98:3300/0 conn(0x7fe6f8152650 0x7fe6f8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f559a640 1 -- 172.21.15.1:0/3219627689 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412089301 0 0) 0x7fe6f80c2fc0 con 0x7fe6f80c6b10 2024-03-06T07:30:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f559a640 1 -- 172.21.15.1:0/3219627689 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6e0003520 con 0x7fe6f80c6b10 2024-03-06T07:30:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f559a640 1 -- 172.21.15.1:0/3219627689 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 227213148 0 0) 0x7fe6e0003520 con 0x7fe6f80c6b10 2024-03-06T07:30:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f559a640 1 -- 172.21.15.1:0/3219627689 >> v2:172.21.15.98:3300/0 conn(0x7fe6f8152650 msgr2=0x7fe6f8155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f559a640 1 --2- 172.21.15.1:0/3219627689 >> v2:172.21.15.98:3300/0 conn(0x7fe6f8152650 0x7fe6f8155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f559a640 1 -- 172.21.15.1:0/3219627689 >> v1:172.21.15.98:6789/0 conn(0x7fe6f80bf9d0 legacy=0x7fe6f80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f559a640 1 -- 172.21.15.1:0/3219627689 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f80c6690 con 0x7fe6f80c6b10 2024-03-06T07:30:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f559a640 1 -- 172.21.15.1:0/3219627689 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fe6ec001670 con 0x7fe6f80c6b10 2024-03-06T07:30:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f559a640 1 -- 172.21.15.1:0/3219627689 >> v1:172.21.15.1:6789/0 conn(0x7fe6f80c6b10 legacy=0x7fe6f8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f559a640 1 --2- 172.21.15.1:0/3219627689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6e00039b0 0x7fe6e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f559a640 1 --2- 172.21.15.1:0/3219627689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6e0004800 0x7fe6e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f559a640 1 --2- 172.21.15.1:0/3219627689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6e00051c0 0x7fe6e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f559a640 1 -- 172.21.15.1:0/3219627689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f8155800 con 0x7fe6e00039b0 2024-03-06T07:30:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f559a640 1 -- 172.21.15.1:0/3219627689 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e000c160 con 0x7fe6e00051c0 2024-03-06T07:30:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f559a640 1 -- 172.21.15.1:0/3219627689 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e000c360 con 0x7fe6e0004800 2024-03-06T07:30:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f659c640 1 --2- 172.21.15.1:0/3219627689 >> v2:172.21.15.98:3300/0 conn(0x7fe6f8152650 0x7fe6f8155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f659c640 1 --2- 172.21.15.1:0/3219627689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6e00039b0 0x7fe6e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f5d9b640 1 --2- 172.21.15.1:0/3219627689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6e0004800 0x7fe6e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.707+0000 7fe6f6d9d640 1 --2- 172.21.15.1:0/3219627689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6e00051c0 0x7fe6e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.708+0000 7fe6f659c640 1 -- 172.21.15.1:0/3219627689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6e0004800 msgr2=0x7fe6e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.708+0000 7fe6f659c640 1 --2- 172.21.15.1:0/3219627689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6e0004800 0x7fe6e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.708+0000 7fe6f659c640 1 -- 172.21.15.1:0/3219627689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6e00051c0 msgr2=0x7fe6e000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.708+0000 7fe6f659c640 1 --2- 172.21.15.1:0/3219627689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6e00051c0 0x7fe6e000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.708+0000 7fe6f659c640 1 -- 172.21.15.1:0/3219627689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e0003520 con 0x7fe6e00039b0 2024-03-06T07:30:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.708+0000 7fe6f6d9d640 1 -- 172.21.15.1:0/3219627689 reap_dead start 2024-03-06T07:30:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.708+0000 7fe6f659c640 1 --2- 172.21.15.1:0/3219627689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6e00039b0 0x7fe6e0003dd0 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8009c70 tx=0x7fe6e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6763555fb79d955 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.708+0000 7fe6f559a640 1 -- 172.21.15.1:0/3219627689 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8009840 con 0x7fe6e00039b0 2024-03-06T07:30:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.708+0000 7fe6f559a640 1 -- 172.21.15.1:0/3219627689 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e8004020 con 0x7fe6e00039b0 2024-03-06T07:30:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.708+0000 7fe6f559a640 1 -- 172.21.15.1:0/3219627689 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8004450 con 0x7fe6e00039b0 2024-03-06T07:30:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.709+0000 7fe6fce0e640 1 -- 172.21.15.1:0/3219627689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6e00039b0 msgr2=0x7fe6e0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.709+0000 7fe6fce0e640 1 --2- 172.21.15.1:0/3219627689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6e00039b0 0x7fe6e0003dd0 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8009c70 tx=0x7fe6e0005830 comp rx=0 tx=0).stop 2024-03-06T07:30:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.709+0000 7fe6fce0e640 1 -- 172.21.15.1:0/3219627689 shutdown_connections 2024-03-06T07:30:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.709+0000 7fe6fce0e640 1 --2- 172.21.15.1:0/3219627689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6e00039b0 0x7fe6e0003dd0 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.709+0000 7fe6fce0e640 1 -- 172.21.15.1:0/3219627689 >> 172.21.15.1:0/3219627689 conn(0x7fe6f8066ee0 msgr2=0x7fe6f80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.709+0000 7fe6fce0e640 1 -- 172.21.15.1:0/3219627689 shutdown_connections 2024-03-06T07:30:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.709+0000 7fe6fce0e640 1 -- 172.21.15.1:0/3219627689 wait complete. 2024-03-06T07:30:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.711+0000 7fe6fce0e640 1 Processor -- start 2024-03-06T07:30:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.711+0000 7fe6fce0e640 1 -- start start 2024-03-06T07:30:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.712+0000 7fe6fce0e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6e00039b0 0x7fe6f8151090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.712+0000 7fe6fce0e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6f81587c0 0x7fe6f81515d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.712+0000 7fe6fce0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6f81eafd0 0x7fe6f81eb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.712+0000 7fe6fce0e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f81551b0 con 0x7fe6e00039b0 2024-03-06T07:30:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.712+0000 7fe6fce0e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f80c6fa0 con 0x7fe6f81eafd0 2024-03-06T07:30:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.712+0000 7fe6fce0e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f8155490 con 0x7fe6f81587c0 2024-03-06T07:30:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.712+0000 7fe6f659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6e00039b0 0x7fe6f8151090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.712+0000 7fe6f659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6e00039b0 0x7fe6f8151090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43968/0 (socket says 172.21.15.1:43968) 2024-03-06T07:30:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.712+0000 7fe6f659c640 1 -- 172.21.15.1:0/2207783435 learned_addr learned my addr 172.21.15.1:0/2207783435 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.712+0000 7fe6f5d9b640 1 --2- 172.21.15.1:0/2207783435 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6f81587c0 0x7fe6f81515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.712+0000 7fe6f6d9d640 1 --2- 172.21.15.1:0/2207783435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6f81eafd0 0x7fe6f81eb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.713+0000 7fe6f659c640 1 -- 172.21.15.1:0/2207783435 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6f81587c0 msgr2=0x7fe6f81515d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.713+0000 7fe6f659c640 1 --2- 172.21.15.1:0/2207783435 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6f81587c0 0x7fe6f81515d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.713+0000 7fe6f659c640 1 -- 172.21.15.1:0/2207783435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6f81eafd0 msgr2=0x7fe6f81eb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.713+0000 7fe6f659c640 1 --2- 172.21.15.1:0/2207783435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6f81eafd0 0x7fe6f81eb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.713+0000 7fe6f659c640 1 -- 172.21.15.1:0/2207783435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f81eb990 con 0x7fe6e00039b0 2024-03-06T07:30:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.713+0000 7fe6f5d9b640 1 --2- 172.21.15.1:0/2207783435 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6f81587c0 0x7fe6f81515d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.713+0000 7fe6f6d9d640 1 --2- 172.21.15.1:0/2207783435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6f81eafd0 0x7fe6f81eb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.713+0000 7fe6f659c640 1 --2- 172.21.15.1:0/2207783435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6e00039b0 0x7fe6f8151090 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8036040 tx=0x7fe6e8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.713+0000 7fe6e6ffd640 1 -- 172.21.15.1:0/2207783435 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8043020 con 0x7fe6e00039b0 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.713+0000 7fe6e6ffd640 1 -- 172.21.15.1:0/2207783435 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e802c8f0 con 0x7fe6e00039b0 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.713+0000 7fe6e6ffd640 1 -- 172.21.15.1:0/2207783435 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e802cca0 con 0x7fe6e00039b0 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.713+0000 7fe6fce0e640 1 -- 172.21.15.1:0/2207783435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f81e14d0 con 0x7fe6e00039b0 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.713+0000 7fe6fce0e640 1 -- 172.21.15.1:0/2207783435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f81e1a90 con 0x7fe6e00039b0 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.714+0000 7fe6fce0e640 1 -- 172.21.15.1:0/2207783435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f8067fd0 con 0x7fe6e00039b0 2024-03-06T07:30:28.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.714+0000 7fe6e6ffd640 1 -- 172.21.15.1:0/2207783435 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe6e8046020 con 0x7fe6e00039b0 2024-03-06T07:30:28.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.715+0000 7fe6e6ffd640 1 --2- 172.21.15.1:0/2207783435 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe6d806b4d0 0x7fe6d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.715+0000 7fe6e6ffd640 1 -- 172.21.15.1:0/2207783435 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7fe6e80ac470 con 0x7fe6e00039b0 2024-03-06T07:30:28.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.715+0000 7fe6f5d9b640 1 --2- 172.21.15.1:0/2207783435 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe6d806b4d0 0x7fe6d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.716+0000 7fe6f5d9b640 1 --2- 172.21.15.1:0/2207783435 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe6d806b4d0 0x7fe6d806d990 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8152370 tx=0x7fe6ec009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:28.719+0000 7fe6e6ffd640 1 -- 172.21.15.1:0/2207783435 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6e804d030 con 0x7fe6e00039b0 2024-03-06T07:30:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:29.092+0000 7fe6fce0e640 1 -- 172.21.15.1:0/2207783435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6f80b06f0 con 0x7fe6e00039b0 2024-03-06T07:30:29.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:29.092+0000 7fe6e6ffd640 1 -- 172.21.15.1:0/2207783435 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4216 (secure 0 0 0) 0x7fe6e807b0b0 con 0x7fe6e00039b0 2024-03-06T07:30:29.094 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:29.094 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:18.318227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:stopping","state_seq":16,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:29.095 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 206 2024-03-06T07:30:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:29.094+0000 7fe6fce0e640 1 -- 172.21.15.1:0/2207783435 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe6d806b4d0 msgr2=0x7fe6d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:29.094+0000 7fe6fce0e640 1 --2- 172.21.15.1:0/2207783435 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe6d806b4d0 0x7fe6d806d990 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8152370 tx=0x7fe6ec009040 comp rx=0 tx=0).stop 2024-03-06T07:30:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:29.094+0000 7fe6fce0e640 1 -- 172.21.15.1:0/2207783435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6e00039b0 msgr2=0x7fe6f8151090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:29.094+0000 7fe6fce0e640 1 --2- 172.21.15.1:0/2207783435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6e00039b0 0x7fe6f8151090 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8036040 tx=0x7fe6e8004730 comp rx=0 tx=0).stop 2024-03-06T07:30:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:29.094+0000 7fe6fce0e640 1 -- 172.21.15.1:0/2207783435 shutdown_connections 2024-03-06T07:30:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:29.094+0000 7fe6fce0e640 1 --2- 172.21.15.1:0/2207783435 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe6d806b4d0 0x7fe6d806d990 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:29.094+0000 7fe6fce0e640 1 --2- 172.21.15.1:0/2207783435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6f81eafd0 0x7fe6f81eb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:29.094+0000 7fe6fce0e640 1 --2- 172.21.15.1:0/2207783435 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6f81587c0 0x7fe6f81515d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:29.094+0000 7fe6fce0e640 1 --2- 172.21.15.1:0/2207783435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6e00039b0 0x7fe6f8151090 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:29.094+0000 7fe6fce0e640 1 -- 172.21.15.1:0/2207783435 >> 172.21.15.1:0/2207783435 conn(0x7fe6f8066ee0 msgr2=0x7fe6f8155a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:29.094+0000 7fe6fce0e640 1 -- 172.21.15.1:0/2207783435 shutdown_connections 2024-03-06T07:30:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:29.094+0000 7fe6fce0e640 1 -- 172.21.15.1:0/2207783435 wait complete. 2024-03-06T07:30:29.509 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:18.318227+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:29.509 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:stopping 2024-03-06T07:30:30.511 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:30.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.611+0000 7f8f82b19640 1 Processor -- start 2024-03-06T07:30:30.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.611+0000 7f8f82b19640 1 -- start start 2024-03-06T07:30:30.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f82b19640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f7c152650 0x7f8f7c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f82b19640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8f7c0bf9d0 0x7f8f7c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f82b19640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f7c0c2fc0 con 0x7f8f7c0c6b10 2024-03-06T07:30:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f82b19640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f7c155800 con 0x7f8f7c0bf9d0 2024-03-06T07:30:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f82b19640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c158da0 con 0x7f8f7c152650 2024-03-06T07:30:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f7bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8f7c0c6b10 0x7f8f7c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52224/0 (socket says 172.21.15.1:52224) 2024-03-06T07:30:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f7bfff640 1 -- 172.21.15.1:0/1971212245 learned_addr learned my addr 172.21.15.1:0/1971212245 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f808b5640 1 --2- 172.21.15.1:0/1971212245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f7c152650 0x7f8f7c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f7b7fe640 1 -- 172.21.15.1:0/1971212245 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912233750 0 0) 0x7f8f7c0c2fc0 con 0x7f8f7c0c6b10 2024-03-06T07:30:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f810b6640 1 --2- 172.21.15.1:0/1971212245 >> v2:172.21.15.98:3300/0 conn(0x7f8f7c0bf9d0 0x7f8f7c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f7b7fe640 1 -- 172.21.15.1:0/1971212245 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f6c003520 con 0x7f8f7c0c6b10 2024-03-06T07:30:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f7b7fe640 1 -- 172.21.15.1:0/1971212245 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1510229827 0 0) 0x7f8f6c003520 con 0x7f8f7c0c6b10 2024-03-06T07:30:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f7b7fe640 1 -- 172.21.15.1:0/1971212245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f7c152650 msgr2=0x7f8f7c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f7b7fe640 1 --2- 172.21.15.1:0/1971212245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f7c152650 0x7f8f7c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f7b7fe640 1 -- 172.21.15.1:0/1971212245 >> v2:172.21.15.98:3300/0 conn(0x7f8f7c0bf9d0 msgr2=0x7f8f7c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f7b7fe640 1 --2- 172.21.15.1:0/1971212245 >> v2:172.21.15.98:3300/0 conn(0x7f8f7c0bf9d0 0x7f8f7c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f7b7fe640 1 -- 172.21.15.1:0/1971212245 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c0c6690 con 0x7f8f7c0c6b10 2024-03-06T07:30:30.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f7b7fe640 1 -- 172.21.15.1:0/1971212245 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f8f68001670 con 0x7f8f7c0c6b10 2024-03-06T07:30:30.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f7b7fe640 1 -- 172.21.15.1:0/1971212245 >> v1:172.21.15.1:6789/0 conn(0x7f8f7c0c6b10 legacy=0x7f8f7c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.612+0000 7f8f7b7fe640 1 --2- 172.21.15.1:0/1971212245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f6c0039b0 0x7f8f6c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.613+0000 7f8f7b7fe640 1 --2- 172.21.15.1:0/1971212245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f6c004800 0x7f8f6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.613+0000 7f8f7b7fe640 1 --2- 172.21.15.1:0/1971212245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f7c152650 0x7f8f6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.613+0000 7f8f7b7fe640 1 -- 172.21.15.1:0/1971212245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c155800 con 0x7f8f6c0039b0 2024-03-06T07:30:30.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.613+0000 7f8f7b7fe640 1 -- 172.21.15.1:0/1971212245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c158da0 con 0x7f8f6c004800 2024-03-06T07:30:30.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.613+0000 7f8f7b7fe640 1 -- 172.21.15.1:0/1971212245 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c00b9c0 con 0x7f8f7c152650 2024-03-06T07:30:30.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.613+0000 7f8f808b5640 1 --2- 172.21.15.1:0/1971212245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f6c0039b0 0x7f8f6c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.613+0000 7f8f7bfff640 1 --2- 172.21.15.1:0/1971212245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f6c004800 0x7f8f6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.613+0000 7f8f810b6640 1 --2- 172.21.15.1:0/1971212245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f7c152650 0x7f8f6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.613+0000 7f8f808b5640 1 -- 172.21.15.1:0/1971212245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f7c152650 msgr2=0x7f8f6c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:30.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.613+0000 7f8f808b5640 1 --2- 172.21.15.1:0/1971212245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f7c152650 0x7f8f6c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.613+0000 7f8f808b5640 1 -- 172.21.15.1:0/1971212245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f6c004800 msgr2=0x7f8f6c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:30.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.613+0000 7f8f808b5640 1 --2- 172.21.15.1:0/1971212245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f6c004800 0x7f8f6c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.613+0000 7f8f808b5640 1 -- 172.21.15.1:0/1971212245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c003520 con 0x7f8f6c0039b0 2024-03-06T07:30:30.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.614+0000 7f8f808b5640 1 --2- 172.21.15.1:0/1971212245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f6c0039b0 0x7f8f6c003dd0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009af0 tx=0x7f8f6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89753811b7958219 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:30.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.614+0000 7f8f7b7fe640 1 -- 172.21.15.1:0/1971212245 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f70039070 con 0x7f8f6c0039b0 2024-03-06T07:30:30.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.614+0000 7f8f7b7fe640 1 -- 172.21.15.1:0/1971212245 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7002c6b0 con 0x7f8f6c0039b0 2024-03-06T07:30:30.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.614+0000 7f8f7b7fe640 1 -- 172.21.15.1:0/1971212245 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f70009690 con 0x7f8f6c0039b0 2024-03-06T07:30:30.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.614+0000 7f8f82b19640 1 -- 172.21.15.1:0/1971212245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f6c0039b0 msgr2=0x7f8f6c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.614+0000 7f8f82b19640 1 --2- 172.21.15.1:0/1971212245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f6c0039b0 0x7f8f6c003dd0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009af0 tx=0x7f8f6c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:30.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.614+0000 7f8f810b6640 1 -- 172.21.15.1:0/1971212245 reap_dead start 2024-03-06T07:30:30.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.614+0000 7f8f82b19640 1 -- 172.21.15.1:0/1971212245 shutdown_connections 2024-03-06T07:30:30.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.614+0000 7f8f82b19640 1 -- 172.21.15.1:0/1971212245 >> 172.21.15.1:0/1971212245 conn(0x7f8f7c066ee0 msgr2=0x7f8f7c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:30.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.614+0000 7f8f82b19640 1 -- 172.21.15.1:0/1971212245 shutdown_connections 2024-03-06T07:30:30.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.615+0000 7f8f82b19640 1 -- 172.21.15.1:0/1971212245 wait complete. 2024-03-06T07:30:30.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.616+0000 7f8f82b19640 1 Processor -- start 2024-03-06T07:30:30.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.616+0000 7f8f82b19640 1 -- start start 2024-03-06T07:30:30.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.616+0000 7f8f82b19640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f6c0039b0 0x7f8f7c153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.616+0000 7f8f82b19640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f7c0c87d0 0x7f8f7c151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.616+0000 7f8f82b19640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f7c151b70 0x7f8f7c14c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.616+0000 7f8f82b19640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c1551b0 con 0x7f8f7c0c87d0 2024-03-06T07:30:30.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.616+0000 7f8f82b19640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c0c6fa0 con 0x7f8f6c0039b0 2024-03-06T07:30:30.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.616+0000 7f8f82b19640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c155490 con 0x7f8f7c151b70 2024-03-06T07:30:30.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.617+0000 7f8f7bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f7c0c87d0 0x7f8f7c151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.617+0000 7f8f7bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f7c0c87d0 0x7f8f7c151630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43986/0 (socket says 172.21.15.1:43986) 2024-03-06T07:30:30.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.617+0000 7f8f7bfff640 1 -- 172.21.15.1:0/2071007135 learned_addr learned my addr 172.21.15.1:0/2071007135 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:30.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.617+0000 7f8f810b6640 1 --2- 172.21.15.1:0/2071007135 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f7c151b70 0x7f8f7c14c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.617+0000 7f8f808b5640 1 --2- 172.21.15.1:0/2071007135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f6c0039b0 0x7f8f7c153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.617+0000 7f8f7bfff640 1 -- 172.21.15.1:0/2071007135 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f7c151b70 msgr2=0x7f8f7c14c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.617+0000 7f8f7bfff640 1 --2- 172.21.15.1:0/2071007135 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f7c151b70 0x7f8f7c14c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.617+0000 7f8f7bfff640 1 -- 172.21.15.1:0/2071007135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f6c0039b0 msgr2=0x7f8f7c153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.617+0000 7f8f7bfff640 1 --2- 172.21.15.1:0/2071007135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f6c0039b0 0x7f8f7c153760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.617+0000 7f8f7bfff640 1 -- 172.21.15.1:0/2071007135 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c1eef50 con 0x7f8f7c0c87d0 2024-03-06T07:30:30.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.617+0000 7f8f7bfff640 1 --2- 172.21.15.1:0/2071007135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f7c0c87d0 0x7f8f7c151630 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f8f68007310 tx=0x7f8f680077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:30.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.617+0000 7f8f797fa640 1 -- 172.21.15.1:0/2071007135 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6800f960 con 0x7f8f7c0c87d0 2024-03-06T07:30:30.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.617+0000 7f8f797fa640 1 -- 172.21.15.1:0/2071007135 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f68018070 con 0x7f8f7c0c87d0 2024-03-06T07:30:30.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.617+0000 7f8f82b19640 1 -- 172.21.15.1:0/2071007135 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f7c1ef230 con 0x7f8f7c0c87d0 2024-03-06T07:30:30.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.617+0000 7f8f797fa640 1 -- 172.21.15.1:0/2071007135 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6800fd80 con 0x7f8f7c0c87d0 2024-03-06T07:30:30.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.617+0000 7f8f82b19640 1 -- 172.21.15.1:0/2071007135 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f7c153ca0 con 0x7f8f7c0c87d0 2024-03-06T07:30:30.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.618+0000 7f8f82b19640 1 -- 172.21.15.1:0/2071007135 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7c0c2f50 con 0x7f8f7c0c87d0 2024-03-06T07:30:30.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.619+0000 7f8f797fa640 1 -- 172.21.15.1:0/2071007135 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f6801d460 con 0x7f8f7c0c87d0 2024-03-06T07:30:30.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.619+0000 7f8f797fa640 1 --2- 172.21.15.1:0/2071007135 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8f6006b500 0x7f8f6006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.619+0000 7f8f797fa640 1 -- 172.21.15.1:0/2071007135 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f8f6808a2c0 con 0x7f8f7c0c87d0 2024-03-06T07:30:30.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.620+0000 7f8f808b5640 1 --2- 172.21.15.1:0/2071007135 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8f6006b500 0x7f8f6006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.620+0000 7f8f808b5640 1 --2- 172.21.15.1:0/2071007135 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8f6006b500 0x7f8f6006d9c0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f8f70036000 tx=0x7f8f70030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:30.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.624+0000 7f8f797fa640 1 -- 172.21.15.1:0/2071007135 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f7c0c2f50 con 0x7f8f7c0c87d0 2024-03-06T07:30:30.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.843+0000 7f8f82b19640 1 -- 172.21.15.1:0/2071007135 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f7c067fd0 con 0x7f8f7c0c87d0 2024-03-06T07:30:30.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.844+0000 7f8f797fa640 1 -- 172.21.15.1:0/2071007135 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4216 (secure 0 0 0) 0x7f8f680588b0 con 0x7f8f7c0c87d0 2024-03-06T07:30:30.846 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:30.846 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:18.318227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:stopping","state_seq":16,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:30.846 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 206 2024-03-06T07:30:30.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.846+0000 7f8f82b19640 1 -- 172.21.15.1:0/2071007135 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8f6006b500 msgr2=0x7f8f6006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.846+0000 7f8f82b19640 1 --2- 172.21.15.1:0/2071007135 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8f6006b500 0x7f8f6006d9c0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f8f70036000 tx=0x7f8f70030000 comp rx=0 tx=0).stop 2024-03-06T07:30:30.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.846+0000 7f8f82b19640 1 -- 172.21.15.1:0/2071007135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f7c0c87d0 msgr2=0x7f8f7c151630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.846+0000 7f8f82b19640 1 --2- 172.21.15.1:0/2071007135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f7c0c87d0 0x7f8f7c151630 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f8f68007310 tx=0x7f8f680077d0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.846+0000 7f8f82b19640 1 -- 172.21.15.1:0/2071007135 shutdown_connections 2024-03-06T07:30:30.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.846+0000 7f8f82b19640 1 --2- 172.21.15.1:0/2071007135 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8f6006b500 0x7f8f6006d9c0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.846+0000 7f8f82b19640 1 --2- 172.21.15.1:0/2071007135 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f7c151b70 0x7f8f7c14c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.846+0000 7f8f82b19640 1 --2- 172.21.15.1:0/2071007135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f7c0c87d0 0x7f8f7c151630 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.846+0000 7f8f82b19640 1 --2- 172.21.15.1:0/2071007135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f6c0039b0 0x7f8f7c153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.846+0000 7f8f82b19640 1 -- 172.21.15.1:0/2071007135 >> 172.21.15.1:0/2071007135 conn(0x7f8f7c066ee0 msgr2=0x7f8f7c0c0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:30.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.846+0000 7f8f82b19640 1 -- 172.21.15.1:0/2071007135 shutdown_connections 2024-03-06T07:30:30.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:30.846+0000 7f8f82b19640 1 -- 172.21.15.1:0/2071007135 wait complete. 2024-03-06T07:30:31.264 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:18.318227+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:31.264 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:stopping 2024-03-06T07:30:32.265 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:32.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.368+0000 7f3cc0716640 1 Processor -- start 2024-03-06T07:30:32.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.368+0000 7f3cc0716640 1 -- start start 2024-03-06T07:30:32.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.369+0000 7f3cc0716640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3cb8152650 0x7f3cb8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.369+0000 7f3cc0716640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb80c2fc0 con 0x7f3cb80c6b10 2024-03-06T07:30:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.369+0000 7f3cc0716640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb8158da0 con 0x7f3cb80bf9d0 2024-03-06T07:30:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.369+0000 7f3cc0716640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb8155800 con 0x7f3cb8152650 2024-03-06T07:30:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.369+0000 7f3cbe4b2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3cb8152650 0x7f3cb8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.369+0000 7f3cbe4b2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3cb8152650 0x7f3cb8155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43988/0 (socket says 172.21.15.1:43988) 2024-03-06T07:30:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.369+0000 7f3cbe4b2640 1 -- 172.21.15.1:0/4150617787 learned_addr learned my addr 172.21.15.1:0/4150617787 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.369+0000 7f3cbdcb1640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3cb80c6b10 0x7f3cb8158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52226/0 (socket says 172.21.15.1:52226) 2024-03-06T07:30:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.369+0000 7f3cbe4b2640 1 -- 172.21.15.1:0/4150617787 >> v1:172.21.15.98:6789/0 conn(0x7f3cb80bf9d0 legacy=0x7f3cb80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.369+0000 7f3cbe4b2640 1 -- 172.21.15.1:0/4150617787 >> v1:172.21.15.1:6789/0 conn(0x7f3cb80c6b10 legacy=0x7f3cb8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.370+0000 7f3cbd4b0640 1 -- 172.21.15.1:0/4150617787 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166610832 0 0) 0x7f3cb80c2fc0 con 0x7f3cb80c6b10 2024-03-06T07:30:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.370+0000 7f3cbe4b2640 1 -- 172.21.15.1:0/4150617787 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb80c6690 con 0x7f3cb8152650 2024-03-06T07:30:32.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.370+0000 7f3cbe4b2640 1 --2- 172.21.15.1:0/4150617787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb8152650 0x7f3cb8155c90 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4009870 tx=0x7f3cb402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f576593603e03987 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.370+0000 7f3cbd4b0640 1 -- 172.21.15.1:0/4150617787 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4035020 con 0x7f3cb8152650 2024-03-06T07:30:32.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.370+0000 7f3cbd4b0640 1 -- 172.21.15.1:0/4150617787 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb4031870 con 0x7f3cb8152650 2024-03-06T07:30:32.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.370+0000 7f3cbd4b0640 1 -- 172.21.15.1:0/4150617787 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4031c40 con 0x7f3cb8152650 2024-03-06T07:30:32.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.371+0000 7f3cc0716640 1 -- 172.21.15.1:0/4150617787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb8152650 msgr2=0x7f3cb8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.371+0000 7f3cc0716640 1 --2- 172.21.15.1:0/4150617787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb8152650 0x7f3cb8155c90 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4009870 tx=0x7f3cb402b910 comp rx=0 tx=0).stop 2024-03-06T07:30:32.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.371+0000 7f3cc0716640 1 -- 172.21.15.1:0/4150617787 shutdown_connections 2024-03-06T07:30:32.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.371+0000 7f3cc0716640 1 --2- 172.21.15.1:0/4150617787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb8152650 0x7f3cb8155c90 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.371+0000 7f3cc0716640 1 -- 172.21.15.1:0/4150617787 >> 172.21.15.1:0/4150617787 conn(0x7f3cb8066ee0 msgr2=0x7f3cb80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:32.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.371+0000 7f3cc0716640 1 -- 172.21.15.1:0/4150617787 shutdown_connections 2024-03-06T07:30:32.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.371+0000 7f3cc0716640 1 -- 172.21.15.1:0/4150617787 wait complete. 2024-03-06T07:30:32.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.372+0000 7f3cc0716640 1 Processor -- start 2024-03-06T07:30:32.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.372+0000 7f3cc0716640 1 -- start start 2024-03-06T07:30:32.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cc0716640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb80bf9d0 0x7f3cb8151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cc0716640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cb8152650 0x7f3cb8154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cc0716640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb814fa80 0x7f3cb814b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cc0716640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb81551b0 con 0x7f3cb814fa80 2024-03-06T07:30:32.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cc0716640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb80c6fa0 con 0x7f3cb80bf9d0 2024-03-06T07:30:32.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cc0716640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb8155490 con 0x7f3cb8152650 2024-03-06T07:30:32.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cbecb3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb814fa80 0x7f3cb814b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cbecb3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb814fa80 0x7f3cb814b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44000/0 (socket says 172.21.15.1:44000) 2024-03-06T07:30:32.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cbecb3640 1 -- 172.21.15.1:0/2597871576 learned_addr learned my addr 172.21.15.1:0/2597871576 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:32.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cbe4b2640 1 --2- 172.21.15.1:0/2597871576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb80bf9d0 0x7f3cb8151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cbecb3640 1 -- 172.21.15.1:0/2597871576 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cb8152650 msgr2=0x7f3cb8154880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cbdcb1640 1 --2- 172.21.15.1:0/2597871576 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cb8152650 0x7f3cb8154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cbecb3640 1 --2- 172.21.15.1:0/2597871576 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cb8152650 0x7f3cb8154880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cbecb3640 1 -- 172.21.15.1:0/2597871576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb80bf9d0 msgr2=0x7f3cb8151410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cbecb3640 1 --2- 172.21.15.1:0/2597871576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb80bf9d0 0x7f3cb8151410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cbecb3640 1 -- 172.21.15.1:0/2597871576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb81ef5a0 con 0x7f3cb814fa80 2024-03-06T07:30:32.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cbe4b2640 1 --2- 172.21.15.1:0/2597871576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb80bf9d0 0x7f3cb8151410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:32.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.373+0000 7f3cbdcb1640 1 --2- 172.21.15.1:0/2597871576 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cb8152650 0x7f3cb8154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:32.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.374+0000 7f3cbecb3640 1 --2- 172.21.15.1:0/2597871576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb814fa80 0x7f3cb814b2f0 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000b990 tx=0x7f3cb000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.374+0000 7f3caeffd640 1 -- 172.21.15.1:0/2597871576 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0019070 con 0x7f3cb814fa80 2024-03-06T07:30:32.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.374+0000 7f3caeffd640 1 -- 172.21.15.1:0/2597871576 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb000cba0 con 0x7f3cb814fa80 2024-03-06T07:30:32.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.374+0000 7f3cc0716640 1 -- 172.21.15.1:0/2597871576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb8151b30 con 0x7f3cb814fa80 2024-03-06T07:30:32.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.374+0000 7f3caeffd640 1 -- 172.21.15.1:0/2597871576 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0014440 con 0x7f3cb814fa80 2024-03-06T07:30:32.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.374+0000 7f3cc0716640 1 -- 172.21.15.1:0/2597871576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb81eeac0 con 0x7f3cb814fa80 2024-03-06T07:30:32.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.375+0000 7f3cc0716640 1 -- 172.21.15.1:0/2597871576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb80c2f50 con 0x7f3cb814fa80 2024-03-06T07:30:32.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.375+0000 7f3caeffd640 1 -- 172.21.15.1:0/2597871576 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3cb001c070 con 0x7f3cb814fa80 2024-03-06T07:30:32.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.375+0000 7f3caeffd640 1 --2- 172.21.15.1:0/2597871576 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c9c06b4d0 0x7f3c9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.376+0000 7f3caeffd640 1 -- 172.21.15.1:0/2597871576 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f3cb008a1c0 con 0x7f3cb814fa80 2024-03-06T07:30:32.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.376+0000 7f3cbe4b2640 1 --2- 172.21.15.1:0/2597871576 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c9c06b4d0 0x7f3c9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.377+0000 7f3cbe4b2640 1 --2- 172.21.15.1:0/2597871576 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c9c06b4d0 0x7f3c9c06d990 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4036000 tx=0x7f3cb4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.380+0000 7f3caeffd640 1 -- 172.21.15.1:0/2597871576 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3cb80c2f50 con 0x7f3cb814fa80 2024-03-06T07:30:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.599+0000 7f3cc0716640 1 -- 172.21.15.1:0/2597871576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cb8067fd0 con 0x7f3cb814fa80 2024-03-06T07:30:32.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.600+0000 7f3caeffd640 1 -- 172.21.15.1:0/2597871576 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4216 (secure 0 0 0) 0x7f3cb0058e00 con 0x7f3cb814fa80 2024-03-06T07:30:32.602 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:32.602 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:18.318227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:stopping","state_seq":16,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:32.602 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 206 2024-03-06T07:30:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.601+0000 7f3cc0716640 1 -- 172.21.15.1:0/2597871576 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c9c06b4d0 msgr2=0x7f3c9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.601+0000 7f3cc0716640 1 --2- 172.21.15.1:0/2597871576 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c9c06b4d0 0x7f3c9c06d990 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4036000 tx=0x7f3cb4008030 comp rx=0 tx=0).stop 2024-03-06T07:30:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.601+0000 7f3cc0716640 1 -- 172.21.15.1:0/2597871576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb814fa80 msgr2=0x7f3cb814b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.601+0000 7f3cc0716640 1 --2- 172.21.15.1:0/2597871576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb814fa80 0x7f3cb814b2f0 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000b990 tx=0x7f3cb000be50 comp rx=0 tx=0).stop 2024-03-06T07:30:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.602+0000 7f3cc0716640 1 -- 172.21.15.1:0/2597871576 shutdown_connections 2024-03-06T07:30:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.602+0000 7f3cc0716640 1 --2- 172.21.15.1:0/2597871576 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c9c06b4d0 0x7f3c9c06d990 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.602+0000 7f3cc0716640 1 --2- 172.21.15.1:0/2597871576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cb814fa80 0x7f3cb814b2f0 unknown :-1 s=CLOSED pgs=2436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.602+0000 7f3cc0716640 1 --2- 172.21.15.1:0/2597871576 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cb8152650 0x7f3cb8154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.602+0000 7f3cc0716640 1 --2- 172.21.15.1:0/2597871576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb80bf9d0 0x7f3cb8151410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.602+0000 7f3cc0716640 1 -- 172.21.15.1:0/2597871576 >> 172.21.15.1:0/2597871576 conn(0x7f3cb8066ee0 msgr2=0x7f3cb80cac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:32.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.602+0000 7f3cc0716640 1 -- 172.21.15.1:0/2597871576 shutdown_connections 2024-03-06T07:30:32.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:32.602+0000 7f3cc0716640 1 -- 172.21.15.1:0/2597871576 wait complete. 2024-03-06T07:30:32.802 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:32.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:33.017 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:18.318227+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:33.017 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:stopping 2024-03-06T07:30:34.019 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:34.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.118+0000 7fede7278640 1 Processor -- start 2024-03-06T07:30:34.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.118+0000 7fede7278640 1 -- start start 2024-03-06T07:30:34.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.119+0000 7fede7278640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede0152650 0x7fede0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.119+0000 7fede7278640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fede00c6b10 0x7fede0158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.119+0000 7fede7278640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fede00bf9d0 0x7fede00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.119+0000 7fede7278640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede0155800 con 0x7fede00c6b10 2024-03-06T07:30:34.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.119+0000 7fede7278640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede00c2fc0 con 0x7fede00bf9d0 2024-03-06T07:30:34.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.119+0000 7fede7278640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede00c6690 con 0x7fede0152650 2024-03-06T07:30:34.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.119+0000 7fede4813640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fede00c6b10 0x7fede0158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.119+0000 7fede4813640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fede00c6b10 0x7fede0158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44012/0 (socket says 172.21.15.1:44012) 2024-03-06T07:30:34.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.119+0000 7fede4813640 1 -- 172.21.15.1:0/40343110 learned_addr learned my addr 172.21.15.1:0/40343110 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:34.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.119+0000 7fede5014640 1 --2- 172.21.15.1:0/40343110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede0152650 0x7fede0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.119+0000 7fede4813640 1 -- 172.21.15.1:0/40343110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede0152650 msgr2=0x7fede0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:34.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.119+0000 7fede5815640 1 --2- 172.21.15.1:0/40343110 >> v2:172.21.15.98:3300/0 conn(0x7fede00bf9d0 0x7fede00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.119+0000 7fede4813640 1 --2- 172.21.15.1:0/40343110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede0152650 0x7fede0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.119+0000 7fede4813640 1 -- 172.21.15.1:0/40343110 >> v2:172.21.15.98:3300/0 conn(0x7fede00bf9d0 msgr2=0x7fede00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:34.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.119+0000 7fede4813640 1 --2- 172.21.15.1:0/40343110 >> v2:172.21.15.98:3300/0 conn(0x7fede00bf9d0 0x7fede00c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.120+0000 7fede4813640 1 -- 172.21.15.1:0/40343110 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede00c7280 con 0x7fede00c6b10 2024-03-06T07:30:34.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.120+0000 7fede5014640 1 --2- 172.21.15.1:0/40343110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede0152650 0x7fede0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:34.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.120+0000 7fede5815640 1 --2- 172.21.15.1:0/40343110 >> v2:172.21.15.98:3300/0 conn(0x7fede00bf9d0 0x7fede00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:34.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.120+0000 7fede4813640 1 --2- 172.21.15.1:0/40343110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede00c6b10 0x7fede0158690 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7fede01597b0 tx=0x7fedd002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8880641ca859dd4c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.120+0000 7fedd7fff640 1 -- 172.21.15.1:0/40343110 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd002e070 con 0x7fede00c6b10 2024-03-06T07:30:34.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.120+0000 7fedd7fff640 1 -- 172.21.15.1:0/40343110 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd0037070 con 0x7fede00c6b10 2024-03-06T07:30:34.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.120+0000 7fedd7fff640 1 -- 172.21.15.1:0/40343110 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd0031a00 con 0x7fede00c6b10 2024-03-06T07:30:34.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.121+0000 7fede7278640 1 -- 172.21.15.1:0/40343110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede00c6b10 msgr2=0x7fede0158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.121+0000 7fede7278640 1 --2- 172.21.15.1:0/40343110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede00c6b10 0x7fede0158690 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7fede01597b0 tx=0x7fedd002b880 comp rx=0 tx=0).stop 2024-03-06T07:30:34.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.121+0000 7fede7278640 1 -- 172.21.15.1:0/40343110 shutdown_connections 2024-03-06T07:30:34.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.121+0000 7fede7278640 1 --2- 172.21.15.1:0/40343110 >> v2:172.21.15.98:3300/0 conn(0x7fede00bf9d0 0x7fede00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.121+0000 7fede7278640 1 --2- 172.21.15.1:0/40343110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede00c6b10 0x7fede0158690 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.121+0000 7fede7278640 1 --2- 172.21.15.1:0/40343110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede0152650 0x7fede0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.121+0000 7fede7278640 1 -- 172.21.15.1:0/40343110 >> 172.21.15.1:0/40343110 conn(0x7fede0066ee0 msgr2=0x7fede00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.121+0000 7fede7278640 1 -- 172.21.15.1:0/40343110 shutdown_connections 2024-03-06T07:30:34.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.121+0000 7fede7278640 1 -- 172.21.15.1:0/40343110 wait complete. 2024-03-06T07:30:34.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.123+0000 7fede7278640 1 Processor -- start 2024-03-06T07:30:34.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.123+0000 7fede7278640 1 -- start start 2024-03-06T07:30:34.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.124+0000 7fede7278640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede00bf9d0 0x7fede01df580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.124+0000 7fede7278640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede0152650 0x7fede01e0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.124+0000 7fede7278640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede00c9640 0x7fede01444a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.124+0000 7fede7278640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede0155490 con 0x7fede00c9640 2024-03-06T07:30:34.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.124+0000 7fede7278640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede01551b0 con 0x7fede00bf9d0 2024-03-06T07:30:34.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.124+0000 7fede7278640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede0152a30 con 0x7fede0152650 2024-03-06T07:30:34.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.124+0000 7fede5815640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede00c9640 0x7fede01444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.124+0000 7fede5815640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede00c9640 0x7fede01444a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44016/0 (socket says 172.21.15.1:44016) 2024-03-06T07:30:34.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.124+0000 7fede5815640 1 -- 172.21.15.1:0/1005903091 learned_addr learned my addr 172.21.15.1:0/1005903091 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:34.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.124+0000 7fede5014640 1 --2- 172.21.15.1:0/1005903091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede00bf9d0 0x7fede01df580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.124+0000 7fede4813640 1 --2- 172.21.15.1:0/1005903091 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede0152650 0x7fede01e0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.124+0000 7fede5815640 1 -- 172.21.15.1:0/1005903091 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede0152650 msgr2=0x7fede01e0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.124+0000 7fede5815640 1 --2- 172.21.15.1:0/1005903091 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede0152650 0x7fede01e0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.124+0000 7fede5815640 1 -- 172.21.15.1:0/1005903091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede00bf9d0 msgr2=0x7fede01df580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.124+0000 7fede5815640 1 --2- 172.21.15.1:0/1005903091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede00bf9d0 0x7fede01df580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.124+0000 7fede5815640 1 -- 172.21.15.1:0/1005903091 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede01e0a40 con 0x7fede00c9640 2024-03-06T07:30:34.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.125+0000 7fede5815640 1 --2- 172.21.15.1:0/1005903091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede00c9640 0x7fede01444a0 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7fedd800cd10 tx=0x7fedd800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.125+0000 7fedd5ffb640 1 -- 172.21.15.1:0/1005903091 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd800f680 con 0x7fede00c9640 2024-03-06T07:30:34.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.125+0000 7fedd5ffb640 1 -- 172.21.15.1:0/1005903091 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd800f840 con 0x7fede00c9640 2024-03-06T07:30:34.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.125+0000 7fede7278640 1 -- 172.21.15.1:0/1005903091 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede01de7a0 con 0x7fede00c9640 2024-03-06T07:30:34.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.125+0000 7fedd5ffb640 1 -- 172.21.15.1:0/1005903091 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd800fc30 con 0x7fede00c9640 2024-03-06T07:30:34.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.125+0000 7fede7278640 1 -- 172.21.15.1:0/1005903091 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede01ded60 con 0x7fede00c9640 2024-03-06T07:30:34.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.126+0000 7fede7278640 1 -- 172.21.15.1:0/1005903091 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede006b540 con 0x7fede00c9640 2024-03-06T07:30:34.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.127+0000 7fedd5ffb640 1 -- 172.21.15.1:0/1005903091 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedd80213f0 con 0x7fede00c9640 2024-03-06T07:30:34.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.127+0000 7fedd5ffb640 1 --2- 172.21.15.1:0/1005903091 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fedc406b500 0x7fedc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.127+0000 7fedd5ffb640 1 -- 172.21.15.1:0/1005903091 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7fedd808af40 con 0x7fede00c9640 2024-03-06T07:30:34.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.127+0000 7fede5014640 1 --2- 172.21.15.1:0/1005903091 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fedc406b500 0x7fedc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.128+0000 7fede5014640 1 --2- 172.21.15.1:0/1005903091 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fedc406b500 0x7fedc406d9c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fede014bfa0 tx=0x7feddc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.132+0000 7fedd5ffb640 1 -- 172.21.15.1:0/1005903091 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fede006b540 con 0x7fede00c9640 2024-03-06T07:30:34.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.350+0000 7fede7278640 1 -- 172.21.15.1:0/1005903091 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fede0067fd0 con 0x7fede00c9640 2024-03-06T07:30:34.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.351+0000 7fedd5ffb640 1 -- 172.21.15.1:0/1005903091 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4216 (secure 0 0 0) 0x7fedd8014030 con 0x7fede00c9640 2024-03-06T07:30:34.353 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:34.353 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:18.318227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:stopping","state_seq":16,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:34.353 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 206 2024-03-06T07:30:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.353+0000 7fede7278640 1 -- 172.21.15.1:0/1005903091 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fedc406b500 msgr2=0x7fedc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.353+0000 7fede7278640 1 --2- 172.21.15.1:0/1005903091 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fedc406b500 0x7fedc406d9c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fede014bfa0 tx=0x7feddc008040 comp rx=0 tx=0).stop 2024-03-06T07:30:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.353+0000 7fede7278640 1 -- 172.21.15.1:0/1005903091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede00c9640 msgr2=0x7fede01444a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.353+0000 7fede7278640 1 --2- 172.21.15.1:0/1005903091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede00c9640 0x7fede01444a0 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7fedd800cd10 tx=0x7fedd800a650 comp rx=0 tx=0).stop 2024-03-06T07:30:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.353+0000 7fede7278640 1 -- 172.21.15.1:0/1005903091 shutdown_connections 2024-03-06T07:30:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.353+0000 7fede7278640 1 --2- 172.21.15.1:0/1005903091 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fedc406b500 0x7fedc406d9c0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.353+0000 7fede7278640 1 --2- 172.21.15.1:0/1005903091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fede00c9640 0x7fede01444a0 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.353+0000 7fede7278640 1 --2- 172.21.15.1:0/1005903091 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fede0152650 0x7fede01e0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.353+0000 7fede7278640 1 --2- 172.21.15.1:0/1005903091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede00bf9d0 0x7fede01df580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.353+0000 7fede7278640 1 -- 172.21.15.1:0/1005903091 >> 172.21.15.1:0/1005903091 conn(0x7fede0066ee0 msgr2=0x7fede014e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.353+0000 7fede7278640 1 -- 172.21.15.1:0/1005903091 shutdown_connections 2024-03-06T07:30:34.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:34.353+0000 7fede7278640 1 -- 172.21.15.1:0/1005903091 wait complete. 2024-03-06T07:30:34.768 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:18.318227+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:34.768 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:stopping 2024-03-06T07:30:35.769 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:35.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.869+0000 7f2d4c688640 1 Processor -- start 2024-03-06T07:30:35.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.869+0000 7f2d4c688640 1 -- start start 2024-03-06T07:30:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.870+0000 7f2d4c688640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2d440c6b10 0x7f2d44155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.870+0000 7f2d4c688640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d440bf9d0 0x7f2d440c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.870+0000 7f2d4c688640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d440c2fc0 con 0x7f2d44152650 2024-03-06T07:30:35.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.870+0000 7f2d4c688640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d44155800 con 0x7f2d440c6b10 2024-03-06T07:30:35.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.870+0000 7f2d4c688640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d44158da0 con 0x7f2d440bf9d0 2024-03-06T07:30:35.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.870+0000 7f2d49c23640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2d440c6b10 0x7f2d44155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.870+0000 7f2d49c23640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2d440c6b10 0x7f2d44155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56774/0 (socket says 172.21.15.1:56774) 2024-03-06T07:30:35.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.870+0000 7f2d49c23640 1 -- 172.21.15.1:0/2879881146 learned_addr learned my addr 172.21.15.1:0/2879881146 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:35.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.870+0000 7f2d4ac25640 1 --2- 172.21.15.1:0/2879881146 >> v2:172.21.15.98:3300/0 conn(0x7f2d440bf9d0 0x7f2d440c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.870+0000 7f2d49c23640 1 -- 172.21.15.1:0/2879881146 >> v2:172.21.15.98:3300/0 conn(0x7f2d440bf9d0 msgr2=0x7f2d440c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:35.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.870+0000 7f2d49c23640 1 --2- 172.21.15.1:0/2879881146 >> v2:172.21.15.98:3300/0 conn(0x7f2d440bf9d0 0x7f2d440c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.870+0000 7f2d49c23640 1 -- 172.21.15.1:0/2879881146 >> v1:172.21.15.98:6789/0 conn(0x7f2d44152650 legacy=0x7f2d44158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.870+0000 7f2d49c23640 1 -- 172.21.15.1:0/2879881146 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d440c6690 con 0x7f2d440c6b10 2024-03-06T07:30:35.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.871+0000 7f2d49c23640 1 --2- 172.21.15.1:0/2879881146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d440c6b10 0x7f2d44155c90 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f2d441597b0 tx=0x7f2d3402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5683140a952689df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.871+0000 7f2d49422640 1 -- 172.21.15.1:0/2879881146 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d34035020 con 0x7f2d440c6b10 2024-03-06T07:30:35.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.871+0000 7f2d49422640 1 -- 172.21.15.1:0/2879881146 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d34031870 con 0x7f2d440c6b10 2024-03-06T07:30:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.871+0000 7f2d49422640 1 -- 172.21.15.1:0/2879881146 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d34031c40 con 0x7f2d440c6b10 2024-03-06T07:30:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.872+0000 7f2d4c688640 1 -- 172.21.15.1:0/2879881146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d440c6b10 msgr2=0x7f2d44155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.872+0000 7f2d4c688640 1 --2- 172.21.15.1:0/2879881146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d440c6b10 0x7f2d44155c90 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f2d441597b0 tx=0x7f2d3402b910 comp rx=0 tx=0).stop 2024-03-06T07:30:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.872+0000 7f2d4c688640 1 -- 172.21.15.1:0/2879881146 shutdown_connections 2024-03-06T07:30:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.872+0000 7f2d4c688640 1 --2- 172.21.15.1:0/2879881146 >> v2:172.21.15.98:3300/0 conn(0x7f2d440bf9d0 0x7f2d440c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.872+0000 7f2d4c688640 1 --2- 172.21.15.1:0/2879881146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d440c6b10 0x7f2d44155c90 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.872+0000 7f2d4c688640 1 -- 172.21.15.1:0/2879881146 >> 172.21.15.1:0/2879881146 conn(0x7f2d44066ee0 msgr2=0x7f2d440672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.872+0000 7f2d4c688640 1 -- 172.21.15.1:0/2879881146 shutdown_connections 2024-03-06T07:30:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.872+0000 7f2d4c688640 1 -- 172.21.15.1:0/2879881146 wait complete. 2024-03-06T07:30:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.873+0000 7f2d4c688640 1 Processor -- start 2024-03-06T07:30:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.873+0000 7f2d4c688640 1 -- start start 2024-03-06T07:30:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.873+0000 7f2d4c688640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d440bf9d0 0x7f2d44157460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.873+0000 7f2d4c688640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d44152650 0x7f2d440c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.873+0000 7f2d4c688640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d44158660 0x7f2d441eea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.873+0000 7f2d4c688640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d441551b0 con 0x7f2d44152650 2024-03-06T07:30:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.873+0000 7f2d4c688640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d440c6fa0 con 0x7f2d44158660 2024-03-06T07:30:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.873+0000 7f2d4c688640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d44155490 con 0x7f2d440bf9d0 2024-03-06T07:30:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.874+0000 7f2d49c23640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d44152650 0x7f2d440c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.874+0000 7f2d49c23640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d44152650 0x7f2d440c8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56778/0 (socket says 172.21.15.1:56778) 2024-03-06T07:30:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.874+0000 7f2d49c23640 1 -- 172.21.15.1:0/3544804671 learned_addr learned my addr 172.21.15.1:0/3544804671 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.874+0000 7f2d4ac25640 1 --2- 172.21.15.1:0/3544804671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d44158660 0x7f2d441eea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.874+0000 7f2d4a424640 1 --2- 172.21.15.1:0/3544804671 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d440bf9d0 0x7f2d44157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.874+0000 7f2d49c23640 1 -- 172.21.15.1:0/3544804671 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d440bf9d0 msgr2=0x7f2d44157460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.874+0000 7f2d49c23640 1 --2- 172.21.15.1:0/3544804671 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d440bf9d0 0x7f2d44157460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.874+0000 7f2d49c23640 1 -- 172.21.15.1:0/3544804671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d44158660 msgr2=0x7f2d441eea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.874+0000 7f2d49c23640 1 --2- 172.21.15.1:0/3544804671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d44158660 0x7f2d441eea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.874+0000 7f2d49c23640 1 -- 172.21.15.1:0/3544804671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d441579a0 con 0x7f2d44152650 2024-03-06T07:30:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.874+0000 7f2d49c23640 1 --2- 172.21.15.1:0/3544804671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d44152650 0x7f2d440c8a30 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7f2d3402b950 tx=0x7f2d340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.874+0000 7f2d3affd640 1 -- 172.21.15.1:0/3544804671 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d34040070 con 0x7f2d44152650 2024-03-06T07:30:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.874+0000 7f2d3affd640 1 -- 172.21.15.1:0/3544804671 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d34030d00 con 0x7f2d44152650 2024-03-06T07:30:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.874+0000 7f2d4c688640 1 -- 172.21.15.1:0/3544804671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d44153810 con 0x7f2d44152650 2024-03-06T07:30:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.874+0000 7f2d3affd640 1 -- 172.21.15.1:0/3544804671 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d34035040 con 0x7f2d44152650 2024-03-06T07:30:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.874+0000 7f2d4c688640 1 -- 172.21.15.1:0/3544804671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d44153dd0 con 0x7f2d44152650 2024-03-06T07:30:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.875+0000 7f2d4c688640 1 -- 172.21.15.1:0/3544804671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d4406b540 con 0x7f2d44152650 2024-03-06T07:30:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.876+0000 7f2d3affd640 1 -- 172.21.15.1:0/3544804671 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d3404b060 con 0x7f2d44152650 2024-03-06T07:30:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.876+0000 7f2d3affd640 1 --2- 172.21.15.1:0/3544804671 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2d2806b4d0 0x7f2d2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.876+0000 7f2d3affd640 1 -- 172.21.15.1:0/3544804671 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f2d340ac060 con 0x7f2d44152650 2024-03-06T07:30:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.877+0000 7f2d4a424640 1 --2- 172.21.15.1:0/3544804671 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2d2806b4d0 0x7f2d2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.877+0000 7f2d4a424640 1 --2- 172.21.15.1:0/3544804671 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2d2806b4d0 0x7f2d2806d990 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f2d40001780 tx=0x7f2d40009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:35.881+0000 7f2d3affd640 1 -- 172.21.15.1:0/3544804671 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d4406b540 con 0x7f2d44152650 2024-03-06T07:30:36.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:36.100+0000 7f2d4c688640 1 -- 172.21.15.1:0/3544804671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d440b06f0 con 0x7f2d44152650 2024-03-06T07:30:36.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:36.100+0000 7f2d3affd640 1 -- 172.21.15.1:0/3544804671 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4216 (secure 0 0 0) 0x7f2d3407a6e0 con 0x7f2d44152650 2024-03-06T07:30:36.102 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:36.102 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:18.318227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:stopping","state_seq":16,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:36.102 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 206 2024-03-06T07:30:36.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:36.102+0000 7f2d4c688640 1 -- 172.21.15.1:0/3544804671 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2d2806b4d0 msgr2=0x7f2d2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:36.102+0000 7f2d4c688640 1 --2- 172.21.15.1:0/3544804671 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2d2806b4d0 0x7f2d2806d990 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f2d40001780 tx=0x7f2d40009040 comp rx=0 tx=0).stop 2024-03-06T07:30:36.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:36.102+0000 7f2d4c688640 1 -- 172.21.15.1:0/3544804671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d44152650 msgr2=0x7f2d440c8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:36.102+0000 7f2d4c688640 1 --2- 172.21.15.1:0/3544804671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d44152650 0x7f2d440c8a30 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7f2d3402b950 tx=0x7f2d340046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:36.102+0000 7f2d4c688640 1 -- 172.21.15.1:0/3544804671 shutdown_connections 2024-03-06T07:30:36.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:36.102+0000 7f2d4c688640 1 --2- 172.21.15.1:0/3544804671 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2d2806b4d0 0x7f2d2806d990 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:36.102+0000 7f2d4c688640 1 --2- 172.21.15.1:0/3544804671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d44158660 0x7f2d441eea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:36.102+0000 7f2d4c688640 1 --2- 172.21.15.1:0/3544804671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d44152650 0x7f2d440c8a30 unknown :-1 s=CLOSED pgs=2440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:36.102+0000 7f2d4c688640 1 --2- 172.21.15.1:0/3544804671 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d440bf9d0 0x7f2d44157460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:36.102+0000 7f2d4c688640 1 -- 172.21.15.1:0/3544804671 >> 172.21.15.1:0/3544804671 conn(0x7f2d44066ee0 msgr2=0x7f2d440672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:36.102+0000 7f2d4c688640 1 -- 172.21.15.1:0/3544804671 shutdown_connections 2024-03-06T07:30:36.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:36.102+0000 7f2d4c688640 1 -- 172.21.15.1:0/3544804671 wait complete. 2024-03-06T07:30:36.519 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:18.318227+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:36.519 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:stopping 2024-03-06T07:30:37.519 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:37.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.619+0000 7fcdbcd65640 1 Processor -- start 2024-03-06T07:30:37.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.619+0000 7fcdbcd65640 1 -- start start 2024-03-06T07:30:37.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.620+0000 7fcdbcd65640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcdb8152650 0x7fcdb8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.620+0000 7fcdbcd65640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb80c2fc0 con 0x7fcdb80c6b10 2024-03-06T07:30:37.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.620+0000 7fcdbcd65640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb8158da0 con 0x7fcdb80bf9d0 2024-03-06T07:30:37.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.620+0000 7fcdbcd65640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdb8155800 con 0x7fcdb8152650 2024-03-06T07:30:37.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.620+0000 7fcdb659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcdb8152650 0x7fcdb8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.620+0000 7fcdb5d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcdb80c6b10 0x7fcdb8158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50666/0 (socket says 172.21.15.1:50666) 2024-03-06T07:30:37.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.620+0000 7fcdb659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcdb8152650 0x7fcdb8155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56784/0 (socket says 172.21.15.1:56784) 2024-03-06T07:30:37.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.620+0000 7fcdb5d9b640 1 -- 172.21.15.1:0/2862575419 learned_addr learned my addr 172.21.15.1:0/2862575419 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:37.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.621+0000 7fcdb659c640 1 -- 172.21.15.1:0/2862575419 >> v1:172.21.15.98:6789/0 conn(0x7fcdb80bf9d0 legacy=0x7fcdb80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:37.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.621+0000 7fcdb659c640 1 -- 172.21.15.1:0/2862575419 >> v1:172.21.15.1:6789/0 conn(0x7fcdb80c6b10 legacy=0x7fcdb8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.621+0000 7fcdb559a640 1 -- 172.21.15.1:0/2862575419 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911372512 0 0) 0x7fcdb80c2fc0 con 0x7fcdb80c6b10 2024-03-06T07:30:37.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.621+0000 7fcdb659c640 1 -- 172.21.15.1:0/2862575419 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb80c6690 con 0x7fcdb8152650 2024-03-06T07:30:37.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.621+0000 7fcdb659c640 1 --2- 172.21.15.1:0/2862575419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb8152650 0x7fcdb8155c90 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7fcda00097a0 tx=0x7fcda002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6248cbafa94a702 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.622+0000 7fcdb559a640 1 -- 172.21.15.1:0/2862575419 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda00315a0 con 0x7fcdb8152650 2024-03-06T07:30:37.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.622+0000 7fcdb559a640 1 -- 172.21.15.1:0/2862575419 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda0037070 con 0x7fcdb8152650 2024-03-06T07:30:37.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.622+0000 7fcdb559a640 1 -- 172.21.15.1:0/2862575419 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda0031960 con 0x7fcdb8152650 2024-03-06T07:30:37.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.622+0000 7fcdbcd65640 1 -- 172.21.15.1:0/2862575419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb8152650 msgr2=0x7fcdb8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.622+0000 7fcdbcd65640 1 --2- 172.21.15.1:0/2862575419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb8152650 0x7fcdb8155c90 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7fcda00097a0 tx=0x7fcda002b840 comp rx=0 tx=0).stop 2024-03-06T07:30:37.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.622+0000 7fcdbcd65640 1 -- 172.21.15.1:0/2862575419 shutdown_connections 2024-03-06T07:30:37.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.622+0000 7fcdbcd65640 1 --2- 172.21.15.1:0/2862575419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb8152650 0x7fcdb8155c90 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.622+0000 7fcdbcd65640 1 -- 172.21.15.1:0/2862575419 >> 172.21.15.1:0/2862575419 conn(0x7fcdb8066ee0 msgr2=0x7fcdb80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.622+0000 7fcdbcd65640 1 -- 172.21.15.1:0/2862575419 shutdown_connections 2024-03-06T07:30:37.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.622+0000 7fcdbcd65640 1 -- 172.21.15.1:0/2862575419 wait complete. 2024-03-06T07:30:37.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.623+0000 7fcdbcd65640 1 Processor -- start 2024-03-06T07:30:37.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.623+0000 7fcdbcd65640 1 -- start start 2024-03-06T07:30:37.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.624+0000 7fcdbcd65640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb80bf9d0 0x7fcdb81eec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.624+0000 7fcdbcd65640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb8152650 0x7fcdb81577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.624+0000 7fcdbcd65640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdb8158720 0x7fcdb814d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.624+0000 7fcdbcd65640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb81551b0 con 0x7fcdb80bf9d0 2024-03-06T07:30:37.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.624+0000 7fcdbcd65640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb80c6fa0 con 0x7fcdb8152650 2024-03-06T07:30:37.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.624+0000 7fcdbcd65640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb8155490 con 0x7fcdb8158720 2024-03-06T07:30:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.624+0000 7fcdb659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb80bf9d0 0x7fcdb81eec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.624+0000 7fcdb659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb80bf9d0 0x7fcdb81eec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56792/0 (socket says 172.21.15.1:56792) 2024-03-06T07:30:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.624+0000 7fcdb659c640 1 -- 172.21.15.1:0/435086057 learned_addr learned my addr 172.21.15.1:0/435086057 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.624+0000 7fcdb5d9b640 1 --2- 172.21.15.1:0/435086057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb8152650 0x7fcdb81577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.624+0000 7fcdb6d9d640 1 --2- 172.21.15.1:0/435086057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdb8158720 0x7fcdb814d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.625+0000 7fcdb659c640 1 -- 172.21.15.1:0/435086057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdb8158720 msgr2=0x7fcdb814d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.625+0000 7fcdb659c640 1 --2- 172.21.15.1:0/435086057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdb8158720 0x7fcdb814d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.625+0000 7fcdb659c640 1 -- 172.21.15.1:0/435086057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb8152650 msgr2=0x7fcdb81577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.625+0000 7fcdb659c640 1 --2- 172.21.15.1:0/435086057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb8152650 0x7fcdb81577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.625+0000 7fcdb659c640 1 -- 172.21.15.1:0/435086057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb81eaf60 con 0x7fcdb80bf9d0 2024-03-06T07:30:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.625+0000 7fcdb659c640 1 --2- 172.21.15.1:0/435086057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb80bf9d0 0x7fcdb81eec80 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7fcda00097a0 tx=0x7fcda00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.625+0000 7fcdaeffd640 1 -- 172.21.15.1:0/435086057 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda0004030 con 0x7fcdb80bf9d0 2024-03-06T07:30:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.625+0000 7fcdaeffd640 1 -- 172.21.15.1:0/435086057 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda00041f0 con 0x7fcdb80bf9d0 2024-03-06T07:30:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.625+0000 7fcdbcd65640 1 -- 172.21.15.1:0/435086057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb81eb240 con 0x7fcdb80bf9d0 2024-03-06T07:30:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.625+0000 7fcdaeffd640 1 -- 172.21.15.1:0/435086057 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda0032640 con 0x7fcdb80bf9d0 2024-03-06T07:30:37.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.625+0000 7fcdbcd65640 1 -- 172.21.15.1:0/435086057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb81eb800 con 0x7fcdb80bf9d0 2024-03-06T07:30:37.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.626+0000 7fcdbcd65640 1 -- 172.21.15.1:0/435086057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb80c2f50 con 0x7fcdb80bf9d0 2024-03-06T07:30:37.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.627+0000 7fcdaeffd640 1 -- 172.21.15.1:0/435086057 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcda00327e0 con 0x7fcdb80bf9d0 2024-03-06T07:30:37.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.627+0000 7fcdaeffd640 1 --2- 172.21.15.1:0/435086057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd9806b4d0 0x7fcd9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.627+0000 7fcdaeffd640 1 -- 172.21.15.1:0/435086057 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7fcda002f030 con 0x7fcdb80bf9d0 2024-03-06T07:30:37.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.627+0000 7fcdb5d9b640 1 --2- 172.21.15.1:0/435086057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd9806b4d0 0x7fcd9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.628+0000 7fcdb5d9b640 1 --2- 172.21.15.1:0/435086057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd9806b4d0 0x7fcd9806d990 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fcdb81e8c00 tx=0x7fcda8007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.635+0000 7fcdaeffd640 1 -- 172.21.15.1:0/435086057 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcdb80c2f50 con 0x7fcdb80bf9d0 2024-03-06T07:30:37.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.854+0000 7fcdbcd65640 1 -- 172.21.15.1:0/435086057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdb8067fd0 con 0x7fcdb80bf9d0 2024-03-06T07:30:37.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.855+0000 7fcdaeffd640 1 -- 172.21.15.1:0/435086057 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4216 (secure 0 0 0) 0x7fcda0041070 con 0x7fcdb80bf9d0 2024-03-06T07:30:37.857 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:37.857 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:18.318227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:stopping","state_seq":16,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:37.857 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 206 2024-03-06T07:30:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.857+0000 7fcdbcd65640 1 -- 172.21.15.1:0/435086057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd9806b4d0 msgr2=0x7fcd9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.857+0000 7fcdbcd65640 1 --2- 172.21.15.1:0/435086057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd9806b4d0 0x7fcd9806d990 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fcdb81e8c00 tx=0x7fcda8007d50 comp rx=0 tx=0).stop 2024-03-06T07:30:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.857+0000 7fcdbcd65640 1 -- 172.21.15.1:0/435086057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb80bf9d0 msgr2=0x7fcdb81eec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.857+0000 7fcdbcd65640 1 --2- 172.21.15.1:0/435086057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb80bf9d0 0x7fcdb81eec80 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7fcda00097a0 tx=0x7fcda00046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.857+0000 7fcdbcd65640 1 -- 172.21.15.1:0/435086057 shutdown_connections 2024-03-06T07:30:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.857+0000 7fcdbcd65640 1 --2- 172.21.15.1:0/435086057 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcd9806b4d0 0x7fcd9806d990 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.857+0000 7fcdbcd65640 1 --2- 172.21.15.1:0/435086057 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdb8158720 0x7fcdb814d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.857+0000 7fcdbcd65640 1 --2- 172.21.15.1:0/435086057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb8152650 0x7fcdb81577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.857+0000 7fcdbcd65640 1 --2- 172.21.15.1:0/435086057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb80bf9d0 0x7fcdb81eec80 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.857+0000 7fcdbcd65640 1 -- 172.21.15.1:0/435086057 >> 172.21.15.1:0/435086057 conn(0x7fcdb8066ee0 msgr2=0x7fcdb80c3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.857+0000 7fcdbcd65640 1 -- 172.21.15.1:0/435086057 shutdown_connections 2024-03-06T07:30:37.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:37.857+0000 7fcdbcd65640 1 -- 172.21.15.1:0/435086057 wait complete. 2024-03-06T07:30:38.272 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:18.318227+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:38.272 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:stopping 2024-03-06T07:30:39.273 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:39.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.375+0000 7f81761c7640 1 Processor -- start 2024-03-06T07:30:39.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.375+0000 7f81761c7640 1 -- start start 2024-03-06T07:30:39.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.376+0000 7f81761c7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81700bf9d0 0x7f8170155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.376+0000 7f81761c7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81700c2fc0 con 0x7f8170152650 2024-03-06T07:30:39.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.376+0000 7f81761c7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8170158da0 con 0x7f81700c6b10 2024-03-06T07:30:39.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.376+0000 7f81761c7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8170155800 con 0x7f81700bf9d0 2024-03-06T07:30:39.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.376+0000 7f816ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81700bf9d0 0x7f8170155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.376+0000 7f816ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81700bf9d0 0x7f8170155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56800/0 (socket says 172.21.15.1:56800) 2024-03-06T07:30:39.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.376+0000 7f816f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8170152650 0x7f8170158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50674/0 (socket says 172.21.15.1:50674) 2024-03-06T07:30:39.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.376+0000 7f816ffff640 1 -- 172.21.15.1:0/591786794 learned_addr learned my addr 172.21.15.1:0/591786794 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:39.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.376+0000 7f816ffff640 1 -- 172.21.15.1:0/591786794 >> v1:172.21.15.98:6789/0 conn(0x7f81700c6b10 legacy=0x7f81700c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.376+0000 7f816ffff640 1 -- 172.21.15.1:0/591786794 >> v1:172.21.15.1:6789/0 conn(0x7f8170152650 legacy=0x7f8170158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.376+0000 7f816e7fc640 1 -- 172.21.15.1:0/591786794 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277734364 0 0) 0x7f81700c2fc0 con 0x7f8170152650 2024-03-06T07:30:39.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.376+0000 7f816ffff640 1 -- 172.21.15.1:0/591786794 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81700c6690 con 0x7f81700bf9d0 2024-03-06T07:30:39.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.377+0000 7f816ffff640 1 --2- 172.21.15.1:0/591786794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81700bf9d0 0x7f8170155c90 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f8168009e50 tx=0x7f816802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c90de3aa39872bdc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.377+0000 7f816e7fc640 1 -- 172.21.15.1:0/591786794 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816802ca80 con 0x7f81700bf9d0 2024-03-06T07:30:39.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.377+0000 7f816e7fc640 1 -- 172.21.15.1:0/591786794 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816802cc20 con 0x7f81700bf9d0 2024-03-06T07:30:39.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.377+0000 7f816e7fc640 1 -- 172.21.15.1:0/591786794 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81680343f0 con 0x7f81700bf9d0 2024-03-06T07:30:39.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.377+0000 7f81761c7640 1 -- 172.21.15.1:0/591786794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81700bf9d0 msgr2=0x7f8170155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.377+0000 7f81761c7640 1 --2- 172.21.15.1:0/591786794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81700bf9d0 0x7f8170155c90 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f8168009e50 tx=0x7f816802bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.377+0000 7f81761c7640 1 -- 172.21.15.1:0/591786794 shutdown_connections 2024-03-06T07:30:39.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.377+0000 7f81761c7640 1 --2- 172.21.15.1:0/591786794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81700bf9d0 0x7f8170155c90 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.377+0000 7f81761c7640 1 -- 172.21.15.1:0/591786794 >> 172.21.15.1:0/591786794 conn(0x7f8170066ee0 msgr2=0x7f81700672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.377+0000 7f81761c7640 1 -- 172.21.15.1:0/591786794 shutdown_connections 2024-03-06T07:30:39.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.377+0000 7f81761c7640 1 -- 172.21.15.1:0/591786794 wait complete. 2024-03-06T07:30:39.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.378+0000 7f81761c7640 1 Processor -- start 2024-03-06T07:30:39.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.379+0000 7f81761c7640 1 -- start start 2024-03-06T07:30:39.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.379+0000 7f81761c7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81700bf9d0 0x7f81701eb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.379+0000 7f81761c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8170152650 0x7f81700c9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.379+0000 7f81761c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81701534d0 0x7f8170153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.379+0000 7f81761c7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81701551b0 con 0x7f81701534d0 2024-03-06T07:30:39.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.379+0000 7f81761c7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81700c6fa0 con 0x7f8170152650 2024-03-06T07:30:39.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.379+0000 7f81761c7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8170155490 con 0x7f81700bf9d0 2024-03-06T07:30:39.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.379+0000 7f816ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81701534d0 0x7f8170153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.379+0000 7f816ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81701534d0 0x7f8170153980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56810/0 (socket says 172.21.15.1:56810) 2024-03-06T07:30:39.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.379+0000 7f816ffff640 1 -- 172.21.15.1:0/1229780666 learned_addr learned my addr 172.21.15.1:0/1229780666 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:39.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.380+0000 7f816f7fe640 1 --2- 172.21.15.1:0/1229780666 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81700bf9d0 0x7f81701eb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.380+0000 7f816effd640 1 --2- 172.21.15.1:0/1229780666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8170152650 0x7f81700c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.380+0000 7f816ffff640 1 -- 172.21.15.1:0/1229780666 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81700bf9d0 msgr2=0x7f81701eb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.380+0000 7f816ffff640 1 --2- 172.21.15.1:0/1229780666 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81700bf9d0 0x7f81701eb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.380+0000 7f816ffff640 1 -- 172.21.15.1:0/1229780666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8170152650 msgr2=0x7f81700c9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.380+0000 7f816ffff640 1 --2- 172.21.15.1:0/1229780666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8170152650 0x7f81700c9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.380+0000 7f816ffff640 1 -- 172.21.15.1:0/1229780666 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81701548b0 con 0x7f81701534d0 2024-03-06T07:30:39.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.380+0000 7f816effd640 1 --2- 172.21.15.1:0/1229780666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8170152650 0x7f81700c9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:39.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.380+0000 7f816f7fe640 1 --2- 172.21.15.1:0/1229780666 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81700bf9d0 0x7f81701eb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:39.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.380+0000 7f816ffff640 1 --2- 172.21.15.1:0/1229780666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81701534d0 0x7f8170153980 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f8168009e50 tx=0x7f8168004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.380+0000 7f8157fff640 1 -- 172.21.15.1:0/1229780666 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8168043020 con 0x7f81701534d0 2024-03-06T07:30:39.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.380+0000 7f8157fff640 1 -- 172.21.15.1:0/1229780666 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816803e450 con 0x7f81701534d0 2024-03-06T07:30:39.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.380+0000 7f8157fff640 1 -- 172.21.15.1:0/1229780666 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8168009510 con 0x7f81701534d0 2024-03-06T07:30:39.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.380+0000 7f81761c7640 1 -- 172.21.15.1:0/1229780666 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81701542f0 con 0x7f81701534d0 2024-03-06T07:30:39.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.380+0000 7f81761c7640 1 -- 172.21.15.1:0/1229780666 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81700c9750 con 0x7f81701534d0 2024-03-06T07:30:39.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.381+0000 7f81761c7640 1 -- 172.21.15.1:0/1229780666 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f817006b540 con 0x7f81701534d0 2024-03-06T07:30:39.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.382+0000 7f8157fff640 1 -- 172.21.15.1:0/1229780666 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8168005d10 con 0x7f81701534d0 2024-03-06T07:30:39.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.382+0000 7f8157fff640 1 --2- 172.21.15.1:0/1229780666 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f815c06b4d0 0x7f815c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.382+0000 7f8157fff640 1 -- 172.21.15.1:0/1229780666 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f81680ad000 con 0x7f81701534d0 2024-03-06T07:30:39.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.382+0000 7f816f7fe640 1 --2- 172.21.15.1:0/1229780666 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f815c06b4d0 0x7f815c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.383+0000 7f816f7fe640 1 --2- 172.21.15.1:0/1229780666 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f815c06b4d0 0x7f815c06d990 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f8160000f60 tx=0x7f8160009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.386+0000 7f8157fff640 1 -- 172.21.15.1:0/1229780666 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f817006b540 con 0x7f81701534d0 2024-03-06T07:30:39.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.618+0000 7f81761c7640 1 -- 172.21.15.1:0/1229780666 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81700b06f0 con 0x7f81701534d0 2024-03-06T07:30:39.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.618+0000 7f8157fff640 1 -- 172.21.15.1:0/1229780666 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4216 (secure 0 0 0) 0x7f816807bcd0 con 0x7f81701534d0 2024-03-06T07:30:39.621 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:39.621 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:18.318227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:stopping","state_seq":16,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:39.621 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 206 2024-03-06T07:30:39.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.620+0000 7f81761c7640 1 -- 172.21.15.1:0/1229780666 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f815c06b4d0 msgr2=0x7f815c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.620+0000 7f81761c7640 1 --2- 172.21.15.1:0/1229780666 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f815c06b4d0 0x7f815c06d990 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f8160000f60 tx=0x7f8160009040 comp rx=0 tx=0).stop 2024-03-06T07:30:39.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.620+0000 7f81761c7640 1 -- 172.21.15.1:0/1229780666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81701534d0 msgr2=0x7f8170153980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.620+0000 7f81761c7640 1 --2- 172.21.15.1:0/1229780666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81701534d0 0x7f8170153980 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f8168009e50 tx=0x7f8168004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:39.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.620+0000 7f81761c7640 1 -- 172.21.15.1:0/1229780666 shutdown_connections 2024-03-06T07:30:39.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.620+0000 7f81761c7640 1 --2- 172.21.15.1:0/1229780666 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f815c06b4d0 0x7f815c06d990 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.620+0000 7f81761c7640 1 --2- 172.21.15.1:0/1229780666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81701534d0 0x7f8170153980 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.620+0000 7f81761c7640 1 --2- 172.21.15.1:0/1229780666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8170152650 0x7f81700c9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.620+0000 7f81761c7640 1 --2- 172.21.15.1:0/1229780666 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81700bf9d0 0x7f81701eb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.620+0000 7f81761c7640 1 -- 172.21.15.1:0/1229780666 >> 172.21.15.1:0/1229780666 conn(0x7f8170066ee0 msgr2=0x7f81700c02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.621+0000 7f81761c7640 1 -- 172.21.15.1:0/1229780666 shutdown_connections 2024-03-06T07:30:39.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:39.621+0000 7f81761c7640 1 -- 172.21.15.1:0/1229780666 wait complete. 2024-03-06T07:30:40.036 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:18.318227+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:40.036 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:stopping 2024-03-06T07:30:41.037 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.136+0000 7fb7cd837640 1 Processor -- start 2024-03-06T07:30:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.136+0000 7fb7cd837640 1 -- start start 2024-03-06T07:30:41.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.137+0000 7fb7cd837640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb7c80c5850 0x7fb7c80c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.137+0000 7fb7cd837640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c8155dc0 con 0x7fb7c8155220 2024-03-06T07:30:41.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.137+0000 7fb7cd837640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c8158910 con 0x7fb7c8152920 2024-03-06T07:30:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.137+0000 7fb7cd837640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c8152d00 con 0x7fb7c80c5850 2024-03-06T07:30:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.137+0000 7fb7be7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb7c80c5850 0x7fb7c80c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.137+0000 7fb7c77fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb7c8155220 0x7fb7c81556b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50690/0 (socket says 172.21.15.1:50690) 2024-03-06T07:30:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.137+0000 7fb7be7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb7c80c5850 0x7fb7c80c5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56822/0 (socket says 172.21.15.1:56822) 2024-03-06T07:30:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.137+0000 7fb7c77fe640 1 -- 172.21.15.1:0/2191678639 learned_addr learned my addr 172.21.15.1:0/2191678639 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.138+0000 7fb7be7fc640 1 -- 172.21.15.1:0/2191678639 >> v1:172.21.15.98:6789/0 conn(0x7fb7c8152920 legacy=0x7fb7c806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.138+0000 7fb7be7fc640 1 -- 172.21.15.1:0/2191678639 >> v1:172.21.15.1:6789/0 conn(0x7fb7c8155220 legacy=0x7fb7c81556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.138+0000 7fb7be7fc640 1 -- 172.21.15.1:0/2191678639 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c8158bf0 con 0x7fb7c80c5850 2024-03-06T07:30:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.138+0000 7fb7c67fc640 1 -- 172.21.15.1:0/2191678639 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800763744 0 0) 0x7fb7c8155dc0 con 0x7fb7c8155220 2024-03-06T07:30:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.139+0000 7fb7be7fc640 1 --2- 172.21.15.1:0/2191678639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7c80c5850 0x7fb7c80c5cb0 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7fb7b00097a0 tx=0x7fb7b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a09caed29b8ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.139+0000 7fb7c67fc640 1 -- 172.21.15.1:0/2191678639 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0035020 con 0x7fb7c80c5850 2024-03-06T07:30:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.139+0000 7fb7c67fc640 1 -- 172.21.15.1:0/2191678639 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b0031830 con 0x7fb7c80c5850 2024-03-06T07:30:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.139+0000 7fb7c67fc640 1 -- 172.21.15.1:0/2191678639 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0031c00 con 0x7fb7c80c5850 2024-03-06T07:30:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.139+0000 7fb7cd837640 1 -- 172.21.15.1:0/2191678639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7c80c5850 msgr2=0x7fb7c80c5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.139+0000 7fb7cd837640 1 --2- 172.21.15.1:0/2191678639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7c80c5850 0x7fb7c80c5cb0 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7fb7b00097a0 tx=0x7fb7b002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.139+0000 7fb7cd837640 1 -- 172.21.15.1:0/2191678639 shutdown_connections 2024-03-06T07:30:41.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.139+0000 7fb7cd837640 1 --2- 172.21.15.1:0/2191678639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7c80c5850 0x7fb7c80c5cb0 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.139+0000 7fb7cd837640 1 -- 172.21.15.1:0/2191678639 >> 172.21.15.1:0/2191678639 conn(0x7fb7c80a9700 msgr2=0x7fb7c80a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.139+0000 7fb7cd837640 1 -- 172.21.15.1:0/2191678639 shutdown_connections 2024-03-06T07:30:41.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.139+0000 7fb7cd837640 1 -- 172.21.15.1:0/2191678639 wait complete. 2024-03-06T07:30:41.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.141+0000 7fb7cd837640 1 Processor -- start 2024-03-06T07:30:41.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.141+0000 7fb7cd837640 1 -- start start 2024-03-06T07:30:41.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.142+0000 7fb7cd837640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7c80c5850 0x7fb7c80c3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.142+0000 7fb7cd837640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7c8152920 0x7fb7c81eb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.142+0000 7fb7cd837640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c8155220 0x7fb7c806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.142+0000 7fb7cd837640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c8159240 con 0x7fb7c80c5850 2024-03-06T07:30:41.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.142+0000 7fb7cd837640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c81596d0 con 0x7fb7c8155220 2024-03-06T07:30:41.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.142+0000 7fb7cd837640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c80c6380 con 0x7fb7c8152920 2024-03-06T07:30:41.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.142+0000 7fb7c6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7c80c5850 0x7fb7c80c3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.143+0000 7fb7c6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7c80c5850 0x7fb7c80c3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56838/0 (socket says 172.21.15.1:56838) 2024-03-06T07:30:41.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.143+0000 7fb7c6ffd640 1 -- 172.21.15.1:0/2541951949 learned_addr learned my addr 172.21.15.1:0/2541951949 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:41.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.143+0000 7fb7be7fc640 1 --2- 172.21.15.1:0/2541951949 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7c8152920 0x7fb7c81eb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.143+0000 7fb7c77fe640 1 --2- 172.21.15.1:0/2541951949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c8155220 0x7fb7c806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.143+0000 7fb7c6ffd640 1 -- 172.21.15.1:0/2541951949 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7c8152920 msgr2=0x7fb7c81eb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.143+0000 7fb7c6ffd640 1 --2- 172.21.15.1:0/2541951949 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7c8152920 0x7fb7c81eb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.143+0000 7fb7c6ffd640 1 -- 172.21.15.1:0/2541951949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c8155220 msgr2=0x7fb7c806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.143+0000 7fb7c6ffd640 1 --2- 172.21.15.1:0/2541951949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c8155220 0x7fb7c806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.143+0000 7fb7c6ffd640 1 -- 172.21.15.1:0/2541951949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c80c3580 con 0x7fb7c80c5850 2024-03-06T07:30:41.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.143+0000 7fb7c77fe640 1 --2- 172.21.15.1:0/2541951949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c8155220 0x7fb7c806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:41.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.143+0000 7fb7c6ffd640 1 --2- 172.21.15.1:0/2541951949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7c80c5850 0x7fb7c80c3040 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800c450 tx=0x7fb7b800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.143+0000 7fb7c4ff9640 1 -- 172.21.15.1:0/2541951949 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b80052d0 con 0x7fb7c80c5850 2024-03-06T07:30:41.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.143+0000 7fb7c4ff9640 1 -- 172.21.15.1:0/2541951949 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b8005470 con 0x7fb7c80c5850 2024-03-06T07:30:41.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.143+0000 7fb7cd837640 1 -- 172.21.15.1:0/2541951949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c81eb9a0 con 0x7fb7c80c5850 2024-03-06T07:30:41.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.143+0000 7fb7c4ff9640 1 -- 172.21.15.1:0/2541951949 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8012870 con 0x7fb7c80c5850 2024-03-06T07:30:41.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.143+0000 7fb7cd837640 1 -- 172.21.15.1:0/2541951949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c81eac60 con 0x7fb7c80c5850 2024-03-06T07:30:41.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.145+0000 7fb7cd837640 1 -- 172.21.15.1:0/2541951949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c8067fd0 con 0x7fb7c80c5850 2024-03-06T07:30:41.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.145+0000 7fb7c4ff9640 1 -- 172.21.15.1:0/2541951949 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7b8012a10 con 0x7fb7c80c5850 2024-03-06T07:30:41.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.145+0000 7fb7c4ff9640 1 --2- 172.21.15.1:0/2541951949 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7a806b500 0x7fb7a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.145+0000 7fb7c4ff9640 1 -- 172.21.15.1:0/2541951949 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7fb7b8089f10 con 0x7fb7c80c5850 2024-03-06T07:30:41.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.146+0000 7fb7be7fc640 1 --2- 172.21.15.1:0/2541951949 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7a806b500 0x7fb7a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.146+0000 7fb7be7fc640 1 --2- 172.21.15.1:0/2541951949 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7a806b500 0x7fb7a806d9c0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0036000 tx=0x7fb7b003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.154+0000 7fb7c4ff9640 1 -- 172.21.15.1:0/2541951949 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7b8058be0 con 0x7fb7c80c5850 2024-03-06T07:30:41.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.388+0000 7fb7cd837640 1 -- 172.21.15.1:0/2541951949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7c806bdf0 con 0x7fb7c80c5850 2024-03-06T07:30:41.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.388+0000 7fb7c4ff9640 1 -- 172.21.15.1:0/2541951949 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4216 (secure 0 0 0) 0x7fb7b80584e0 con 0x7fb7c80c5850 2024-03-06T07:30:41.391 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:41.391 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:18.318227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:stopping","state_seq":16,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:41.391 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 206 2024-03-06T07:30:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.390+0000 7fb7cd837640 1 -- 172.21.15.1:0/2541951949 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7a806b500 msgr2=0x7fb7a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.390+0000 7fb7cd837640 1 --2- 172.21.15.1:0/2541951949 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7a806b500 0x7fb7a806d9c0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0036000 tx=0x7fb7b003d040 comp rx=0 tx=0).stop 2024-03-06T07:30:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.390+0000 7fb7cd837640 1 -- 172.21.15.1:0/2541951949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7c80c5850 msgr2=0x7fb7c80c3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.390+0000 7fb7cd837640 1 --2- 172.21.15.1:0/2541951949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7c80c5850 0x7fb7c80c3040 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800c450 tx=0x7fb7b800c910 comp rx=0 tx=0).stop 2024-03-06T07:30:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.390+0000 7fb7cd837640 1 -- 172.21.15.1:0/2541951949 shutdown_connections 2024-03-06T07:30:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.390+0000 7fb7cd837640 1 --2- 172.21.15.1:0/2541951949 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb7a806b500 0x7fb7a806d9c0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.391+0000 7fb7cd837640 1 --2- 172.21.15.1:0/2541951949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c8155220 0x7fb7c806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.391+0000 7fb7cd837640 1 --2- 172.21.15.1:0/2541951949 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb7c8152920 0x7fb7c81eb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.391+0000 7fb7cd837640 1 --2- 172.21.15.1:0/2541951949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7c80c5850 0x7fb7c80c3040 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.391+0000 7fb7cd837640 1 -- 172.21.15.1:0/2541951949 >> 172.21.15.1:0/2541951949 conn(0x7fb7c80a9700 msgr2=0x7fb7c8153aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.391+0000 7fb7cd837640 1 -- 172.21.15.1:0/2541951949 shutdown_connections 2024-03-06T07:30:41.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:41.391+0000 7fb7cd837640 1 -- 172.21.15.1:0/2541951949 wait complete. 2024-03-06T07:30:41.806 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:18.318227+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:41.806 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:stopping 2024-03-06T07:30:42.806 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:42.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.906+0000 7f315af81640 1 Processor -- start 2024-03-06T07:30:42.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.906+0000 7f315af81640 1 -- start start 2024-03-06T07:30:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.907+0000 7f315af81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31540c3970 0x7f31540c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.907+0000 7f315af81640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3154152840 0x7f31541553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.907+0000 7f315af81640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31540c5e40 con 0x7f31540c60b0 2024-03-06T07:30:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.907+0000 7f315af81640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31540c67f0 con 0x7f3154152840 2024-03-06T07:30:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.907+0000 7f315af81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31540c6a40 con 0x7f31540c3970 2024-03-06T07:30:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.907+0000 7f315951e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3154152840 0x7f31541553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.907+0000 7f3153fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f31540c60b0 0x7f3154152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50704/0 (socket says 172.21.15.1:50704) 2024-03-06T07:30:42.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.907+0000 7f3153fff640 1 -- 172.21.15.1:0/3664732885 learned_addr learned my addr 172.21.15.1:0/3664732885 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.907+0000 7f3158d1d640 1 --2- 172.21.15.1:0/3664732885 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31540c3970 0x7f31540c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.907+0000 7f315951e640 1 -- 172.21.15.1:0/3664732885 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31540c3970 msgr2=0x7f31540c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.907+0000 7f315951e640 1 --2- 172.21.15.1:0/3664732885 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31540c3970 0x7f31540c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.907+0000 7f315951e640 1 -- 172.21.15.1:0/3664732885 >> v1:172.21.15.1:6789/0 conn(0x7f31540c60b0 legacy=0x7f3154152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.907+0000 7f315951e640 1 -- 172.21.15.1:0/3664732885 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31540c6d20 con 0x7f3154152840 2024-03-06T07:30:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.908+0000 7f315951e640 1 --2- 172.21.15.1:0/3664732885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3154152840 0x7f31541553d0 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7f314c009d90 tx=0x7f314c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06c38a1a0466976 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.908+0000 7f31537fe640 1 -- 172.21.15.1:0/3664732885 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c039070 con 0x7f3154152840 2024-03-06T07:30:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.908+0000 7f31537fe640 1 -- 172.21.15.1:0/3664732885 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f314c02cc50 con 0x7f3154152840 2024-03-06T07:30:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.908+0000 7f31537fe640 1 -- 172.21.15.1:0/3664732885 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c034440 con 0x7f3154152840 2024-03-06T07:30:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.909+0000 7f315af81640 1 -- 172.21.15.1:0/3664732885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3154152840 msgr2=0x7f31541553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.909+0000 7f315af81640 1 --2- 172.21.15.1:0/3664732885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3154152840 0x7f31541553d0 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7f314c009d90 tx=0x7f314c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.909+0000 7f315af81640 1 -- 172.21.15.1:0/3664732885 shutdown_connections 2024-03-06T07:30:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.909+0000 7f315af81640 1 --2- 172.21.15.1:0/3664732885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3154152840 0x7f31541553d0 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.909+0000 7f315af81640 1 --2- 172.21.15.1:0/3664732885 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31540c3970 0x7f31540c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.909+0000 7f315af81640 1 -- 172.21.15.1:0/3664732885 >> 172.21.15.1:0/3664732885 conn(0x7f31540c88f0 msgr2=0x7f31540c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.909+0000 7f315af81640 1 -- 172.21.15.1:0/3664732885 shutdown_connections 2024-03-06T07:30:42.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.909+0000 7f315af81640 1 -- 172.21.15.1:0/3664732885 wait complete. 2024-03-06T07:30:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.910+0000 7f315af81640 1 Processor -- start 2024-03-06T07:30:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.910+0000 7f315af81640 1 -- start start 2024-03-06T07:30:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.911+0000 7f315af81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31540c3970 0x7f3154158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.911+0000 7f315af81640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31540c60b0 0x7f315414b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.911+0000 7f315af81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3154152840 0x7f31541541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.911+0000 7f315af81640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31540c75c0 con 0x7f31540c60b0 2024-03-06T07:30:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.911+0000 7f315af81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31540c72e0 con 0x7f3154152840 2024-03-06T07:30:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.911+0000 7f315af81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31540c3d50 con 0x7f31540c3970 2024-03-06T07:30:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.911+0000 7f3153fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31540c60b0 0x7f315414b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.911+0000 7f3153fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31540c60b0 0x7f315414b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56846/0 (socket says 172.21.15.1:56846) 2024-03-06T07:30:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.911+0000 7f3153fff640 1 -- 172.21.15.1:0/2136915496 learned_addr learned my addr 172.21.15.1:0/2136915496 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.911+0000 7f3158d1d640 1 --2- 172.21.15.1:0/2136915496 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31540c3970 0x7f3154158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.911+0000 7f315951e640 1 --2- 172.21.15.1:0/2136915496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3154152840 0x7f31541541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.911+0000 7f3153fff640 1 -- 172.21.15.1:0/2136915496 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31540c3970 msgr2=0x7f3154158df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.911+0000 7f3153fff640 1 --2- 172.21.15.1:0/2136915496 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31540c3970 0x7f3154158df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.911+0000 7f3153fff640 1 -- 172.21.15.1:0/2136915496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3154152840 msgr2=0x7f31541541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.911+0000 7f3153fff640 1 --2- 172.21.15.1:0/2136915496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3154152840 0x7f31541541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.911+0000 7f3153fff640 1 -- 172.21.15.1:0/2136915496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3154070100 con 0x7f31540c60b0 2024-03-06T07:30:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.912+0000 7f3153fff640 1 --2- 172.21.15.1:0/2136915496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31540c60b0 0x7f315414b7a0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f314000c440 tx=0x7f3140001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.912+0000 7f31517fa640 1 -- 172.21.15.1:0/2136915496 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3140017070 con 0x7f31540c60b0 2024-03-06T07:30:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.912+0000 7f31517fa640 1 -- 172.21.15.1:0/2136915496 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3140005ce0 con 0x7f31540c60b0 2024-03-06T07:30:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.912+0000 7f315af81640 1 -- 172.21.15.1:0/2136915496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31540703e0 con 0x7f31540c60b0 2024-03-06T07:30:42.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.912+0000 7f31517fa640 1 -- 172.21.15.1:0/2136915496 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31400072f0 con 0x7f31540c60b0 2024-03-06T07:30:42.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.912+0000 7f315af81640 1 -- 172.21.15.1:0/2136915496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3154154730 con 0x7f31540c60b0 2024-03-06T07:30:42.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.913+0000 7f315af81640 1 -- 172.21.15.1:0/2136915496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31540c64e0 con 0x7f31540c60b0 2024-03-06T07:30:42.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.913+0000 7f31517fa640 1 -- 172.21.15.1:0/2136915496 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f314001a070 con 0x7f31540c60b0 2024-03-06T07:30:42.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.914+0000 7f31517fa640 1 --2- 172.21.15.1:0/2136915496 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f313806b4d0 0x7f313806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.914+0000 7f31517fa640 1 -- 172.21.15.1:0/2136915496 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f3140089810 con 0x7f31540c60b0 2024-03-06T07:30:42.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.914+0000 7f3158d1d640 1 --2- 172.21.15.1:0/2136915496 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f313806b4d0 0x7f313806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.914+0000 7f3158d1d640 1 --2- 172.21.15.1:0/2136915496 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f313806b4d0 0x7f313806d990 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f3148006fd0 tx=0x7f3148008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:42.918+0000 7f31517fa640 1 -- 172.21.15.1:0/2136915496 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31540c64e0 con 0x7f31540c60b0 2024-03-06T07:30:43.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:43.135+0000 7f315af81640 1 -- 172.21.15.1:0/2136915496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31540686e0 con 0x7f31540c60b0 2024-03-06T07:30:43.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:43.136+0000 7f31517fa640 1 -- 172.21.15.1:0/2136915496 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4216 (secure 0 0 0) 0x7f3140057d70 con 0x7f31540c60b0 2024-03-06T07:30:43.138 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:43.138 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:18.318227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:stopping","state_seq":16,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:43.138 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 206 2024-03-06T07:30:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:43.137+0000 7f315af81640 1 -- 172.21.15.1:0/2136915496 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f313806b4d0 msgr2=0x7f313806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:43.137+0000 7f315af81640 1 --2- 172.21.15.1:0/2136915496 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f313806b4d0 0x7f313806d990 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f3148006fd0 tx=0x7f3148008040 comp rx=0 tx=0).stop 2024-03-06T07:30:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:43.137+0000 7f315af81640 1 -- 172.21.15.1:0/2136915496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31540c60b0 msgr2=0x7f315414b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:43.137+0000 7f315af81640 1 --2- 172.21.15.1:0/2136915496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31540c60b0 0x7f315414b7a0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f314000c440 tx=0x7f3140001e70 comp rx=0 tx=0).stop 2024-03-06T07:30:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:43.137+0000 7f315af81640 1 -- 172.21.15.1:0/2136915496 shutdown_connections 2024-03-06T07:30:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:43.137+0000 7f315af81640 1 --2- 172.21.15.1:0/2136915496 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f313806b4d0 0x7f313806d990 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:43.137+0000 7f315af81640 1 --2- 172.21.15.1:0/2136915496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3154152840 0x7f31541541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:43.137+0000 7f315af81640 1 --2- 172.21.15.1:0/2136915496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31540c60b0 0x7f315414b7a0 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:43.137+0000 7f315af81640 1 --2- 172.21.15.1:0/2136915496 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31540c3970 0x7f3154158df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:43.138+0000 7f315af81640 1 -- 172.21.15.1:0/2136915496 >> 172.21.15.1:0/2136915496 conn(0x7f31540c88f0 msgr2=0x7f31540c7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:43.138+0000 7f315af81640 1 -- 172.21.15.1:0/2136915496 shutdown_connections 2024-03-06T07:30:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:43.138+0000 7f315af81640 1 -- 172.21.15.1:0/2136915496 wait complete. 2024-03-06T07:30:43.553 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:18.318227+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:43.553 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:stopping 2024-03-06T07:30:44.554 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:44.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.658+0000 7ff28586d640 1 Processor -- start 2024-03-06T07:30:44.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.658+0000 7ff28586d640 1 -- start start 2024-03-06T07:30:44.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.659+0000 7ff28586d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff2800ac610 0x7ff280066f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.659+0000 7ff28586d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2800727e0 0x7ff280072c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.659+0000 7ff28586d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff280152cb0 con 0x7ff2801528d0 2024-03-06T07:30:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.659+0000 7ff28586d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2800c3b30 con 0x7ff2800ac610 2024-03-06T07:30:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.659+0000 7ff28586d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff280159080 con 0x7ff2800727e0 2024-03-06T07:30:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.659+0000 7ff27effd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff2801528d0 0x7ff280155080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55866/0 (socket says 172.21.15.1:55866) 2024-03-06T07:30:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.659+0000 7ff27effd640 1 -- 172.21.15.1:0/2653838001 learned_addr learned my addr 172.21.15.1:0/2653838001 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.659+0000 7ff27dffb640 1 -- 172.21.15.1:0/2653838001 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496052881 0 0) 0x7ff280152cb0 con 0x7ff2801528d0 2024-03-06T07:30:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.659+0000 7ff27e7fc640 1 --2- 172.21.15.1:0/2653838001 >> v2:172.21.15.98:3300/0 conn(0x7ff2800ac610 0x7ff280066f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.659+0000 7ff27f7fe640 1 --2- 172.21.15.1:0/2653838001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2800727e0 0x7ff280072c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.659+0000 7ff27dffb640 1 -- 172.21.15.1:0/2653838001 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff268003520 con 0x7ff2801528d0 2024-03-06T07:30:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27dffb640 1 -- 172.21.15.1:0/2653838001 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2389826317 0 0) 0x7ff268003520 con 0x7ff2801528d0 2024-03-06T07:30:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27dffb640 1 -- 172.21.15.1:0/2653838001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2800727e0 msgr2=0x7ff280072c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27dffb640 1 --2- 172.21.15.1:0/2653838001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2800727e0 0x7ff280072c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27dffb640 1 -- 172.21.15.1:0/2653838001 >> v2:172.21.15.98:3300/0 conn(0x7ff2800ac610 msgr2=0x7ff280066f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27dffb640 1 --2- 172.21.15.1:0/2653838001 >> v2:172.21.15.98:3300/0 conn(0x7ff2800ac610 0x7ff280066f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27dffb640 1 -- 172.21.15.1:0/2653838001 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28006fa60 con 0x7ff2801528d0 2024-03-06T07:30:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27dffb640 1 -- 172.21.15.1:0/2653838001 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7ff270001670 con 0x7ff2801528d0 2024-03-06T07:30:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27dffb640 1 -- 172.21.15.1:0/2653838001 >> v1:172.21.15.1:6789/0 conn(0x7ff2801528d0 legacy=0x7ff280155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27dffb640 1 --2- 172.21.15.1:0/2653838001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2680039b0 0x7ff268003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27dffb640 1 --2- 172.21.15.1:0/2653838001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff268004800 0x7ff268004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27dffb640 1 --2- 172.21.15.1:0/2653838001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2800727e0 0x7ff2680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27dffb640 1 -- 172.21.15.1:0/2653838001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff280159080 con 0x7ff2680039b0 2024-03-06T07:30:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27dffb640 1 -- 172.21.15.1:0/2653838001 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26800b9c0 con 0x7ff268004800 2024-03-06T07:30:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27dffb640 1 -- 172.21.15.1:0/2653838001 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26800bb70 con 0x7ff2800727e0 2024-03-06T07:30:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27effd640 1 --2- 172.21.15.1:0/2653838001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2680039b0 0x7ff268003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27f7fe640 1 --2- 172.21.15.1:0/2653838001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2800727e0 0x7ff2680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27e7fc640 1 --2- 172.21.15.1:0/2653838001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff268004800 0x7ff268004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27effd640 1 -- 172.21.15.1:0/2653838001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2800727e0 msgr2=0x7ff2680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27effd640 1 --2- 172.21.15.1:0/2653838001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2800727e0 0x7ff2680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27effd640 1 -- 172.21.15.1:0/2653838001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff268004800 msgr2=0x7ff268004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27effd640 1 --2- 172.21.15.1:0/2653838001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff268004800 0x7ff268004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27effd640 1 -- 172.21.15.1:0/2653838001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff268003520 con 0x7ff2680039b0 2024-03-06T07:30:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27f7fe640 1 --2- 172.21.15.1:0/2653838001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff2800727e0 0x7ff2680051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.660+0000 7ff27e7fc640 1 --2- 172.21.15.1:0/2653838001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff268004800 0x7ff268004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.661+0000 7ff27effd640 1 --2- 172.21.15.1:0/2653838001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2680039b0 0x7ff268003dd0 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7ff27000c210 tx=0x7ff268005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10cde5be3815bad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.661+0000 7ff27dffb640 1 -- 172.21.15.1:0/2653838001 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff270007d50 con 0x7ff2680039b0 2024-03-06T07:30:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.661+0000 7ff27dffb640 1 -- 172.21.15.1:0/2653838001 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff270030070 con 0x7ff2680039b0 2024-03-06T07:30:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.661+0000 7ff27dffb640 1 -- 172.21.15.1:0/2653838001 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27002cd80 con 0x7ff2680039b0 2024-03-06T07:30:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.661+0000 7ff28586d640 1 -- 172.21.15.1:0/2653838001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2680039b0 msgr2=0x7ff268003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.661+0000 7ff28586d640 1 --2- 172.21.15.1:0/2653838001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2680039b0 0x7ff268003dd0 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7ff27000c210 tx=0x7ff268005830 comp rx=0 tx=0).stop 2024-03-06T07:30:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.661+0000 7ff27f7fe640 1 -- 172.21.15.1:0/2653838001 reap_dead start 2024-03-06T07:30:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.661+0000 7ff28586d640 1 -- 172.21.15.1:0/2653838001 shutdown_connections 2024-03-06T07:30:44.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.661+0000 7ff28586d640 1 -- 172.21.15.1:0/2653838001 >> 172.21.15.1:0/2653838001 conn(0x7ff280071cf0 msgr2=0x7ff2800720d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.662+0000 7ff28586d640 1 -- 172.21.15.1:0/2653838001 shutdown_connections 2024-03-06T07:30:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.662+0000 7ff28586d640 1 -- 172.21.15.1:0/2653838001 wait complete. 2024-03-06T07:30:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.663+0000 7ff28586d640 1 Processor -- start 2024-03-06T07:30:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.663+0000 7ff28586d640 1 -- start start 2024-03-06T07:30:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.663+0000 7ff28586d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2680039b0 0x7ff280121100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.663+0000 7ff28586d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff280158350 0x7ff280157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.663+0000 7ff28586d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff280154a50 0x7ff280120340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.663+0000 7ff28586d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff280067490 con 0x7ff2680039b0 2024-03-06T07:30:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.663+0000 7ff28586d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2801592d0 con 0x7ff280158350 2024-03-06T07:30:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.663+0000 7ff28586d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff28006f6f0 con 0x7ff280154a50 2024-03-06T07:30:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.664+0000 7ff27effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2680039b0 0x7ff280121100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.664+0000 7ff27effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2680039b0 0x7ff280121100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36430/0 (socket says 172.21.15.1:36430) 2024-03-06T07:30:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.664+0000 7ff27effd640 1 -- 172.21.15.1:0/1667750030 learned_addr learned my addr 172.21.15.1:0/1667750030 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.664+0000 7ff27f7fe640 1 --2- 172.21.15.1:0/1667750030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff280154a50 0x7ff280120340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.664+0000 7ff27e7fc640 1 --2- 172.21.15.1:0/1667750030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff280158350 0x7ff280157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.664+0000 7ff27effd640 1 -- 172.21.15.1:0/1667750030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff280154a50 msgr2=0x7ff280120340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.664+0000 7ff27effd640 1 --2- 172.21.15.1:0/1667750030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff280154a50 0x7ff280120340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.664+0000 7ff27effd640 1 -- 172.21.15.1:0/1667750030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff280158350 msgr2=0x7ff280157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.664+0000 7ff27effd640 1 --2- 172.21.15.1:0/1667750030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff280158350 0x7ff280157500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.664+0000 7ff27effd640 1 -- 172.21.15.1:0/1667750030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff280150110 con 0x7ff2680039b0 2024-03-06T07:30:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.664+0000 7ff27effd640 1 --2- 172.21.15.1:0/1667750030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2680039b0 0x7ff280121100 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7ff27002c590 tx=0x7ff270001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.664+0000 7ff267fff640 1 -- 172.21.15.1:0/1667750030 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff270007d50 con 0x7ff2680039b0 2024-03-06T07:30:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.664+0000 7ff267fff640 1 -- 172.21.15.1:0/1667750030 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff270004bf0 con 0x7ff2680039b0 2024-03-06T07:30:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.664+0000 7ff28586d640 1 -- 172.21.15.1:0/1667750030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2801eae00 con 0x7ff2680039b0 2024-03-06T07:30:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.664+0000 7ff267fff640 1 -- 172.21.15.1:0/1667750030 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff270001c90 con 0x7ff2680039b0 2024-03-06T07:30:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.664+0000 7ff28586d640 1 -- 172.21.15.1:0/1667750030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff280122250 con 0x7ff2680039b0 2024-03-06T07:30:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.666+0000 7ff28586d640 1 -- 172.21.15.1:0/1667750030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff28006cb20 con 0x7ff2680039b0 2024-03-06T07:30:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.666+0000 7ff267fff640 1 -- 172.21.15.1:0/1667750030 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff270047020 con 0x7ff2680039b0 2024-03-06T07:30:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.666+0000 7ff267fff640 1 --2- 172.21.15.1:0/1667750030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff26c06b4d0 0x7ff26c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.666+0000 7ff267fff640 1 -- 172.21.15.1:0/1667750030 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7ff2700ab410 con 0x7ff2680039b0 2024-03-06T07:30:44.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.666+0000 7ff27e7fc640 1 --2- 172.21.15.1:0/1667750030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff26c06b4d0 0x7ff26c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.667+0000 7ff27e7fc640 1 --2- 172.21.15.1:0/1667750030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff26c06b4d0 0x7ff26c06d990 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7ff2800c1080 tx=0x7ff274008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.674+0000 7ff267fff640 1 -- 172.21.15.1:0/1667750030 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff270079a90 con 0x7ff2680039b0 2024-03-06T07:30:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.893+0000 7ff28586d640 1 -- 172.21.15.1:0/1667750030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2800686e0 con 0x7ff2680039b0 2024-03-06T07:30:44.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.894+0000 7ff267fff640 1 -- 172.21.15.1:0/1667750030 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4216 (secure 0 0 0) 0x7ff27007db30 con 0x7ff2680039b0 2024-03-06T07:30:44.896 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:44.896 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:18.318227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:stopping","state_seq":16,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:44.896 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 206 2024-03-06T07:30:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.896+0000 7ff28586d640 1 -- 172.21.15.1:0/1667750030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff26c06b4d0 msgr2=0x7ff26c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.896+0000 7ff28586d640 1 --2- 172.21.15.1:0/1667750030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff26c06b4d0 0x7ff26c06d990 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7ff2800c1080 tx=0x7ff274008040 comp rx=0 tx=0).stop 2024-03-06T07:30:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.896+0000 7ff28586d640 1 -- 172.21.15.1:0/1667750030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2680039b0 msgr2=0x7ff280121100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.896+0000 7ff28586d640 1 --2- 172.21.15.1:0/1667750030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2680039b0 0x7ff280121100 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7ff27002c590 tx=0x7ff270001a40 comp rx=0 tx=0).stop 2024-03-06T07:30:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.896+0000 7ff28586d640 1 -- 172.21.15.1:0/1667750030 shutdown_connections 2024-03-06T07:30:44.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.896+0000 7ff28586d640 1 --2- 172.21.15.1:0/1667750030 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff26c06b4d0 0x7ff26c06d990 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.896+0000 7ff28586d640 1 --2- 172.21.15.1:0/1667750030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff280154a50 0x7ff280120340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.896+0000 7ff28586d640 1 --2- 172.21.15.1:0/1667750030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff280158350 0x7ff280157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.896+0000 7ff28586d640 1 --2- 172.21.15.1:0/1667750030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2680039b0 0x7ff280121100 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.896+0000 7ff28586d640 1 -- 172.21.15.1:0/1667750030 >> 172.21.15.1:0/1667750030 conn(0x7ff280071cf0 msgr2=0x7ff280074450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.896+0000 7ff28586d640 1 -- 172.21.15.1:0/1667750030 shutdown_connections 2024-03-06T07:30:44.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:44.896+0000 7ff28586d640 1 -- 172.21.15.1:0/1667750030 wait complete. 2024-03-06T07:30:45.311 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:18.318227+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:45.311 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:stopping 2024-03-06T07:30:46.312 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:46.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.413+0000 7f0881381640 1 Processor -- start 2024-03-06T07:30:46.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.413+0000 7f0881381640 1 -- start start 2024-03-06T07:30:46.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.414+0000 7f0881381640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f087c0c3260 0x7f087c0c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.414+0000 7f0881381640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f087c155b70 0x7f087c159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.414+0000 7f0881381640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087c0c2b90 con 0x7f087c0c8cd0 2024-03-06T07:30:46.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.414+0000 7f0881381640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f087c156000 con 0x7f087c155b70 2024-03-06T07:30:46.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.414+0000 7f0881381640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f087c0c37b0 con 0x7f087c0c3260 2024-03-06T07:30:46.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.414+0000 7f087b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f087c155b70 0x7f087c159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.414+0000 7f087affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f087c0c8cd0 0x7f087c0c7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55876/0 (socket says 172.21.15.1:55876) 2024-03-06T07:30:46.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.414+0000 7f087affd640 1 -- 172.21.15.1:0/3719276498 learned_addr learned my addr 172.21.15.1:0/3719276498 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.414+0000 7f087b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f087c155b70 0x7f087c159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36432/0 (socket says 172.21.15.1:36432) 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.414+0000 7f0873fff640 1 --2- 172.21.15.1:0/3719276498 >> v2:172.21.15.98:3300/0 conn(0x7f087c0c3260 0x7f087c0c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.414+0000 7f087b7fe640 1 -- 172.21.15.1:0/3719276498 >> v2:172.21.15.98:3300/0 conn(0x7f087c0c3260 msgr2=0x7f087c0c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.414+0000 7f087b7fe640 1 --2- 172.21.15.1:0/3719276498 >> v2:172.21.15.98:3300/0 conn(0x7f087c0c3260 0x7f087c0c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.414+0000 7f087b7fe640 1 -- 172.21.15.1:0/3719276498 >> v1:172.21.15.1:6789/0 conn(0x7f087c0c8cd0 legacy=0x7f087c0c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.414+0000 7f087b7fe640 1 -- 172.21.15.1:0/3719276498 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c0c3a90 con 0x7f087c155b70 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.414+0000 7f0873fff640 1 --2- 172.21.15.1:0/3719276498 >> v2:172.21.15.98:3300/0 conn(0x7f087c0c3260 0x7f087c0c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.415+0000 7f087b7fe640 1 --2- 172.21.15.1:0/3719276498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c155b70 0x7f087c159010 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f087c0c4ce0 tx=0x7f087402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9904242559da832 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.415+0000 7f087a7fc640 1 -- 172.21.15.1:0/3719276498 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0874039070 con 0x7f087c155b70 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.415+0000 7f087a7fc640 1 -- 172.21.15.1:0/3719276498 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087402cbf0 con 0x7f087c155b70 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.415+0000 7f087a7fc640 1 -- 172.21.15.1:0/3719276498 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0874034440 con 0x7f087c155b70 2024-03-06T07:30:46.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.416+0000 7f0881381640 1 -- 172.21.15.1:0/3719276498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c155b70 msgr2=0x7f087c159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.416+0000 7f0881381640 1 --2- 172.21.15.1:0/3719276498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c155b70 0x7f087c159010 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f087c0c4ce0 tx=0x7f087402be40 comp rx=0 tx=0).stop 2024-03-06T07:30:46.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.416+0000 7f0881381640 1 -- 172.21.15.1:0/3719276498 shutdown_connections 2024-03-06T07:30:46.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.416+0000 7f0881381640 1 --2- 172.21.15.1:0/3719276498 >> v2:172.21.15.98:3300/0 conn(0x7f087c0c3260 0x7f087c0c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.416+0000 7f0881381640 1 --2- 172.21.15.1:0/3719276498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c155b70 0x7f087c159010 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.416+0000 7f0881381640 1 -- 172.21.15.1:0/3719276498 >> 172.21.15.1:0/3719276498 conn(0x7f087c06b3e0 msgr2=0x7f087c06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:46.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.416+0000 7f0881381640 1 -- 172.21.15.1:0/3719276498 shutdown_connections 2024-03-06T07:30:46.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.416+0000 7f0881381640 1 -- 172.21.15.1:0/3719276498 wait complete. 2024-03-06T07:30:46.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.417+0000 7f0881381640 1 Processor -- start 2024-03-06T07:30:46.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.417+0000 7f0881381640 1 -- start start 2024-03-06T07:30:46.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f0881381640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f087c0c3260 0x7f087c0ac9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f0881381640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f087c0c8cd0 0x7f087c0aa840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f0881381640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c155b70 0x7f087c0aad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f0881381640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c159550 con 0x7f087c155b70 2024-03-06T07:30:46.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f0881381640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c158af0 con 0x7f087c0c8cd0 2024-03-06T07:30:46.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f0881381640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c0c8560 con 0x7f087c0c3260 2024-03-06T07:30:46.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f087b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c155b70 0x7f087c0aad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f087b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c155b70 0x7f087c0aad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36440/0 (socket says 172.21.15.1:36440) 2024-03-06T07:30:46.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f087b7fe640 1 -- 172.21.15.1:0/4038124760 learned_addr learned my addr 172.21.15.1:0/4038124760 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:46.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f087affd640 1 --2- 172.21.15.1:0/4038124760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f087c0c3260 0x7f087c0ac9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f0873fff640 1 --2- 172.21.15.1:0/4038124760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f087c0c8cd0 0x7f087c0aa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f087b7fe640 1 -- 172.21.15.1:0/4038124760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f087c0c3260 msgr2=0x7f087c0ac9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f087b7fe640 1 --2- 172.21.15.1:0/4038124760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f087c0c3260 0x7f087c0ac9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f087b7fe640 1 -- 172.21.15.1:0/4038124760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f087c0c8cd0 msgr2=0x7f087c0aa840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f087b7fe640 1 --2- 172.21.15.1:0/4038124760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f087c0c8cd0 0x7f087c0aa840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f087b7fe640 1 -- 172.21.15.1:0/4038124760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c149800 con 0x7f087c155b70 2024-03-06T07:30:46.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f087affd640 1 --2- 172.21.15.1:0/4038124760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f087c0c3260 0x7f087c0ac9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:46.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f087b7fe640 1 --2- 172.21.15.1:0/4038124760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c155b70 0x7f087c0aad80 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f0874009e80 tx=0x7f0874004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:46.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f08737fe640 1 -- 172.21.15.1:0/4038124760 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0874039050 con 0x7f087c155b70 2024-03-06T07:30:46.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f08737fe640 1 -- 172.21.15.1:0/4038124760 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0874045070 con 0x7f087c155b70 2024-03-06T07:30:46.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f08737fe640 1 -- 172.21.15.1:0/4038124760 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087403e830 con 0x7f087c155b70 2024-03-06T07:30:46.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.418+0000 7f0881381640 1 -- 172.21.15.1:0/4038124760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087c149f50 con 0x7f087c155b70 2024-03-06T07:30:46.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.419+0000 7f0881381640 1 -- 172.21.15.1:0/4038124760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087c0acf00 con 0x7f087c155b70 2024-03-06T07:30:46.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.420+0000 7f0881381640 1 -- 172.21.15.1:0/4038124760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087c067fd0 con 0x7f087c155b70 2024-03-06T07:30:46.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.420+0000 7f08737fe640 1 -- 172.21.15.1:0/4038124760 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f087403e9d0 con 0x7f087c155b70 2024-03-06T07:30:46.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.420+0000 7f08737fe640 1 --2- 172.21.15.1:0/4038124760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f085c06b4d0 0x7f085c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.420+0000 7f08737fe640 1 -- 172.21.15.1:0/4038124760 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f08740ad280 con 0x7f087c155b70 2024-03-06T07:30:46.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.421+0000 7f087affd640 1 --2- 172.21.15.1:0/4038124760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f085c06b4d0 0x7f085c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.421+0000 7f087affd640 1 --2- 172.21.15.1:0/4038124760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f085c06b4d0 0x7f085c06d990 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f086c000f60 tx=0x7f086c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:46.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.425+0000 7f08737fe640 1 -- 172.21.15.1:0/4038124760 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f087407bec0 con 0x7f087c155b70 2024-03-06T07:30:46.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.643+0000 7f0881381640 1 -- 172.21.15.1:0/4038124760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f087c0c2d80 con 0x7f087c155b70 2024-03-06T07:30:46.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.644+0000 7f08737fe640 1 -- 172.21.15.1:0/4038124760 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4216 (secure 0 0 0) 0x7f087407b7c0 con 0x7f087c155b70 2024-03-06T07:30:46.646 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:46.646 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:18.318227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:stopping","state_seq":16,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:46.646 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 206 2024-03-06T07:30:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.646+0000 7f0881381640 1 -- 172.21.15.1:0/4038124760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f085c06b4d0 msgr2=0x7f085c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.646+0000 7f0881381640 1 --2- 172.21.15.1:0/4038124760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f085c06b4d0 0x7f085c06d990 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f086c000f60 tx=0x7f086c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.646+0000 7f0881381640 1 -- 172.21.15.1:0/4038124760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c155b70 msgr2=0x7f087c0aad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.646+0000 7f0881381640 1 --2- 172.21.15.1:0/4038124760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c155b70 0x7f087c0aad80 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f0874009e80 tx=0x7f0874004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.646+0000 7f0881381640 1 -- 172.21.15.1:0/4038124760 shutdown_connections 2024-03-06T07:30:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.646+0000 7f0881381640 1 --2- 172.21.15.1:0/4038124760 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f085c06b4d0 0x7f085c06d990 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.646+0000 7f0881381640 1 --2- 172.21.15.1:0/4038124760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c155b70 0x7f087c0aad80 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.646+0000 7f0881381640 1 --2- 172.21.15.1:0/4038124760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f087c0c8cd0 0x7f087c0aa840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.646+0000 7f0881381640 1 --2- 172.21.15.1:0/4038124760 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f087c0c3260 0x7f087c0ac9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.646+0000 7f0881381640 1 -- 172.21.15.1:0/4038124760 >> 172.21.15.1:0/4038124760 conn(0x7f087c06b3e0 msgr2=0x7f087c157ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.646+0000 7f0881381640 1 -- 172.21.15.1:0/4038124760 shutdown_connections 2024-03-06T07:30:46.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:46.646+0000 7f0881381640 1 -- 172.21.15.1:0/4038124760 wait complete. 2024-03-06T07:30:47.060 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:18.318227+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:47.061 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:stopping 2024-03-06T07:30:48.062 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.162+0000 7fb1cda0e640 1 Processor -- start 2024-03-06T07:30:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.162+0000 7fb1cda0e640 1 -- start start 2024-03-06T07:30:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.162+0000 7fb1cda0e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb1c8152690 0x7fb1c8155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.162+0000 7fb1cda0e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb1c80c6730 0x7fb1c80c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.162+0000 7fb1cda0e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c81561e0 con 0x7fb1c80c5d10 2024-03-06T07:30:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.162+0000 7fb1cda0e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1c80c3300 con 0x7fb1c8152690 2024-03-06T07:30:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.162+0000 7fb1cda0e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1c80c35a0 con 0x7fb1c80c6730 2024-03-06T07:30:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.162+0000 7fb1c6ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb1c8152690 0x7fb1c8155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.162+0000 7fb1c6ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb1c8152690 0x7fb1c8155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36442/0 (socket says 172.21.15.1:36442) 2024-03-06T07:30:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.162+0000 7fb1c6ffd640 1 -- 172.21.15.1:0/2118422527 learned_addr learned my addr 172.21.15.1:0/2118422527 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.162+0000 7fb1c77fe640 1 --1- 172.21.15.1:0/2118422527 >> v1:172.21.15.1:6789/0 conn(0x7fb1c80c5d10 0x7fb1c8156cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55882/0 (socket says 172.21.15.1:55882) 2024-03-06T07:30:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.163+0000 7fb1c67fc640 1 --2- 172.21.15.1:0/2118422527 >> v2:172.21.15.98:3300/0 conn(0x7fb1c80c6730 0x7fb1c80c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.163+0000 7fb1c6ffd640 1 -- 172.21.15.1:0/2118422527 >> v2:172.21.15.98:3300/0 conn(0x7fb1c80c6730 msgr2=0x7fb1c80c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.163+0000 7fb1c6ffd640 1 --2- 172.21.15.1:0/2118422527 >> v2:172.21.15.98:3300/0 conn(0x7fb1c80c6730 0x7fb1c80c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.163+0000 7fb1c6ffd640 1 -- 172.21.15.1:0/2118422527 >> v1:172.21.15.1:6789/0 conn(0x7fb1c80c5d10 legacy=0x7fb1c8156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.163+0000 7fb1c5ffb640 1 -- 172.21.15.1:0/2118422527 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852945011 0 0) 0x7fb1c0001490 con 0x7fb1c80c5d10 2024-03-06T07:30:48.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.163+0000 7fb1c6ffd640 1 -- 172.21.15.1:0/2118422527 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c80c3880 con 0x7fb1c8152690 2024-03-06T07:30:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.164+0000 7fb1c6ffd640 1 --2- 172.21.15.1:0/2118422527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c8152690 0x7fb1c8155ca0 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80c98d0 tx=0x7fb1b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72bd825a247d79f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.164+0000 7fb1c5ffb640 1 -- 172.21.15.1:0/2118422527 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8035020 con 0x7fb1c8152690 2024-03-06T07:30:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.164+0000 7fb1c5ffb640 1 -- 172.21.15.1:0/2118422527 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b8031870 con 0x7fb1c8152690 2024-03-06T07:30:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.164+0000 7fb1c5ffb640 1 -- 172.21.15.1:0/2118422527 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8031c40 con 0x7fb1c8152690 2024-03-06T07:30:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.164+0000 7fb1cda0e640 1 -- 172.21.15.1:0/2118422527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c8152690 msgr2=0x7fb1c8155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.164+0000 7fb1cda0e640 1 --2- 172.21.15.1:0/2118422527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c8152690 0x7fb1c8155ca0 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80c98d0 tx=0x7fb1b802b910 comp rx=0 tx=0).stop 2024-03-06T07:30:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.164+0000 7fb1cda0e640 1 -- 172.21.15.1:0/2118422527 shutdown_connections 2024-03-06T07:30:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.164+0000 7fb1cda0e640 1 --2- 172.21.15.1:0/2118422527 >> v2:172.21.15.98:3300/0 conn(0x7fb1c80c6730 0x7fb1c80c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.164+0000 7fb1cda0e640 1 --2- 172.21.15.1:0/2118422527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c8152690 0x7fb1c8155ca0 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.164+0000 7fb1cda0e640 1 -- 172.21.15.1:0/2118422527 >> 172.21.15.1:0/2118422527 conn(0x7fb1c806b410 msgr2=0x7fb1c806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.164+0000 7fb1cda0e640 1 -- 172.21.15.1:0/2118422527 shutdown_connections 2024-03-06T07:30:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.164+0000 7fb1cda0e640 1 -- 172.21.15.1:0/2118422527 wait complete. 2024-03-06T07:30:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.166+0000 7fb1cda0e640 1 Processor -- start 2024-03-06T07:30:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.166+0000 7fb1cda0e640 1 -- start start 2024-03-06T07:30:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.167+0000 7fb1cda0e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c80c5d10 0x7fb1c80c8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.167+0000 7fb1cda0e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1c80c6730 0x7fb1c80aac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.167+0000 7fb1cda0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1c8152690 0x7fb1c814cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.167+0000 7fb1cda0e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c8152a70 con 0x7fb1c80c5d10 2024-03-06T07:30:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.167+0000 7fb1cda0e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c81567b0 con 0x7fb1c8152690 2024-03-06T07:30:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.167+0000 7fb1cda0e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c8152bf0 con 0x7fb1c80c6730 2024-03-06T07:30:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.167+0000 7fb1c6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c80c5d10 0x7fb1c80c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.167+0000 7fb1c6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c80c5d10 0x7fb1c80c8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36450/0 (socket says 172.21.15.1:36450) 2024-03-06T07:30:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.167+0000 7fb1c6ffd640 1 -- 172.21.15.1:0/1605014108 learned_addr learned my addr 172.21.15.1:0/1605014108 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.168+0000 7fb1c77fe640 1 --2- 172.21.15.1:0/1605014108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1c8152690 0x7fb1c814cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.168+0000 7fb1c67fc640 1 --2- 172.21.15.1:0/1605014108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1c80c6730 0x7fb1c80aac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.168+0000 7fb1c6ffd640 1 -- 172.21.15.1:0/1605014108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1c80c6730 msgr2=0x7fb1c80aac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.168+0000 7fb1c6ffd640 1 --2- 172.21.15.1:0/1605014108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1c80c6730 0x7fb1c80aac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.168+0000 7fb1c6ffd640 1 -- 172.21.15.1:0/1605014108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1c8152690 msgr2=0x7fb1c814cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.168+0000 7fb1c6ffd640 1 --2- 172.21.15.1:0/1605014108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1c8152690 0x7fb1c814cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.168+0000 7fb1c6ffd640 1 -- 172.21.15.1:0/1605014108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c814db40 con 0x7fb1c80c5d10 2024-03-06T07:30:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.168+0000 7fb1c67fc640 1 --2- 172.21.15.1:0/1605014108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1c80c6730 0x7fb1c80aac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.168+0000 7fb1c77fe640 1 --2- 172.21.15.1:0/1605014108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1c8152690 0x7fb1c814cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.168+0000 7fb1c6ffd640 1 --2- 172.21.15.1:0/1605014108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c80c5d10 0x7fb1c80c8240 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7fb1b802b950 tx=0x7fb1b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.168+0000 7fb1affff640 1 -- 172.21.15.1:0/1605014108 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8040070 con 0x7fb1c80c5d10 2024-03-06T07:30:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.168+0000 7fb1affff640 1 -- 172.21.15.1:0/1605014108 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b8030d00 con 0x7fb1c80c5d10 2024-03-06T07:30:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.168+0000 7fb1cda0e640 1 -- 172.21.15.1:0/1605014108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c81eb740 con 0x7fb1c80c5d10 2024-03-06T07:30:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.168+0000 7fb1affff640 1 -- 172.21.15.1:0/1605014108 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8035040 con 0x7fb1c80c5d10 2024-03-06T07:30:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.168+0000 7fb1cda0e640 1 -- 172.21.15.1:0/1605014108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c80c8780 con 0x7fb1c80c5d10 2024-03-06T07:30:48.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.169+0000 7fb1cda0e640 1 -- 172.21.15.1:0/1605014108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c80a7e50 con 0x7fb1c80c5d10 2024-03-06T07:30:48.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.170+0000 7fb1affff640 1 -- 172.21.15.1:0/1605014108 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1b8038020 con 0x7fb1c80c5d10 2024-03-06T07:30:48.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.170+0000 7fb1affff640 1 --2- 172.21.15.1:0/1605014108 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb1b406b4d0 0x7fb1b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.170+0000 7fb1affff640 1 -- 172.21.15.1:0/1605014108 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6274+0+0 (secure 0 0 0) 0x7fb1b80ac240 con 0x7fb1c80c5d10 2024-03-06T07:30:48.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.170+0000 7fb1c67fc640 1 --2- 172.21.15.1:0/1605014108 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb1b406b4d0 0x7fb1b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.171+0000 7fb1c67fc640 1 --2- 172.21.15.1:0/1605014108 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb1b406b4d0 0x7fb1b406d990 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc006fd0 tx=0x7fb1bc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.174+0000 7fb1affff640 1 -- 172.21.15.1:0/1605014108 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1c80a7e50 con 0x7fb1c80c5d10 2024-03-06T07:30:48.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.392+0000 7fb1cda0e640 1 -- 172.21.15.1:0/1605014108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1c80a7c30 con 0x7fb1c80c5d10 2024-03-06T07:30:48.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.392+0000 7fb1affff640 1 -- 172.21.15.1:0/1605014108 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4216 (secure 0 0 0) 0x7fb1c80a7c30 con 0x7fb1c80c5d10 2024-03-06T07:30:48.395 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:48.395 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8171,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:18.318227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":9259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9259":{"gid":9259,"name":"d","rank":1,"incarnation":201,"state":"up:stopping","state_seq":16,"addr":"172.21.15.98:6836/2083320955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2083320955},{"type":"v1","addr":"172.21.15.98:6836","nonce":2083320955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:48.395 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 206 2024-03-06T07:30:48.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.394+0000 7fb1cda0e640 1 -- 172.21.15.1:0/1605014108 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb1b406b4d0 msgr2=0x7fb1b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.394+0000 7fb1cda0e640 1 --2- 172.21.15.1:0/1605014108 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb1b406b4d0 0x7fb1b406d990 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc006fd0 tx=0x7fb1bc008040 comp rx=0 tx=0).stop 2024-03-06T07:30:48.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.394+0000 7fb1cda0e640 1 -- 172.21.15.1:0/1605014108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c80c5d10 msgr2=0x7fb1c80c8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.394+0000 7fb1cda0e640 1 --2- 172.21.15.1:0/1605014108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c80c5d10 0x7fb1c80c8240 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7fb1b802b950 tx=0x7fb1b80046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.394+0000 7fb1cda0e640 1 -- 172.21.15.1:0/1605014108 shutdown_connections 2024-03-06T07:30:48.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.394+0000 7fb1cda0e640 1 --2- 172.21.15.1:0/1605014108 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb1b406b4d0 0x7fb1b406d990 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.394+0000 7fb1cda0e640 1 --2- 172.21.15.1:0/1605014108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1c8152690 0x7fb1c814cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.394+0000 7fb1cda0e640 1 --2- 172.21.15.1:0/1605014108 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1c80c6730 0x7fb1c80aac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.394+0000 7fb1cda0e640 1 --2- 172.21.15.1:0/1605014108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c80c5d10 0x7fb1c80c8240 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.394+0000 7fb1cda0e640 1 -- 172.21.15.1:0/1605014108 >> 172.21.15.1:0/1605014108 conn(0x7fb1c806b410 msgr2=0x7fb1c806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.394+0000 7fb1cda0e640 1 -- 172.21.15.1:0/1605014108 shutdown_connections 2024-03-06T07:30:48.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:48.394+0000 7fb1cda0e640 1 -- 172.21.15.1:0/1605014108 wait complete. 2024-03-06T07:30:48.810 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:18.318227+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 9259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9259': {'gid': 9259, 'name': 'd', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.98:6836/2083320955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6834', 'nonce': 2083320955}, {'type': 'v1', 'addr': '172.21.15.98:6836', 'nonce': 2083320955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:48.810 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9259:up:stopping 2024-03-06T07:30:49.811 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:49.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.950+0000 7fc2f2556640 1 Processor -- start 2024-03-06T07:30:49.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.950+0000 7fc2f2556640 1 -- start start 2024-03-06T07:30:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.950+0000 7fc2f2556640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2ec152650 0x7fc2ec155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.950+0000 7fc2f2556640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2ec0bf9d0 0x7fc2ec0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.950+0000 7fc2f2556640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2ec0c2fc0 con 0x7fc2ec0c6b10 2024-03-06T07:30:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.950+0000 7fc2f2556640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2ec155800 con 0x7fc2ec0bf9d0 2024-03-06T07:30:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.950+0000 7fc2f2556640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec158da0 con 0x7fc2ec152650 2024-03-06T07:30:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.951+0000 7fc2ebfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2ec152650 0x7fc2ec155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.951+0000 7fc2f0af3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2ec0bf9d0 0x7fc2ec0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.951+0000 7fc2ebfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2ec152650 0x7fc2ec155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:43822/0 (socket says 172.21.15.1:43822) 2024-03-06T07:30:49.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.951+0000 7fc2ebfff640 1 -- 172.21.15.1:0/487277737 learned_addr learned my addr 172.21.15.1:0/487277737 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:49.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.951+0000 7fc2f0af3640 1 --2- 172.21.15.1:0/487277737 >> v2:172.21.15.98:3300/0 conn(0x7fc2ec0bf9d0 0x7fc2ec0c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:49224/0 (socket says 172.21.15.1:49224) 2024-03-06T07:30:49.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.951+0000 7fc2eb7fe640 1 --1- 172.21.15.1:0/487277737 >> v1:172.21.15.98:6789/0 conn(0x7fc2ec0c6b10 0x7fc2ec158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:34060/0 (socket says 172.21.15.1:34060) 2024-03-06T07:30:49.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.951+0000 7fc2ebfff640 1 -- 172.21.15.1:0/487277737 >> v2:172.21.15.98:3300/0 conn(0x7fc2ec0bf9d0 msgr2=0x7fc2ec0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:49.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.951+0000 7fc2ebfff640 1 --2- 172.21.15.1:0/487277737 >> v2:172.21.15.98:3300/0 conn(0x7fc2ec0bf9d0 0x7fc2ec0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.951+0000 7fc2ebfff640 1 -- 172.21.15.1:0/487277737 >> v1:172.21.15.98:6789/0 conn(0x7fc2ec0c6b10 legacy=0x7fc2ec158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.951+0000 7fc2eaffd640 1 -- 172.21.15.1:0/487277737 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407433223 0 0) 0x7fc2ec0c2fc0 con 0x7fc2ec0c6b10 2024-03-06T07:30:49.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.951+0000 7fc2ebfff640 1 -- 172.21.15.1:0/487277737 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec0c6690 con 0x7fc2ec152650 2024-03-06T07:30:49.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.951+0000 7fc2f0af3640 1 --2- 172.21.15.1:0/487277737 >> v2:172.21.15.98:3300/0 conn(0x7fc2ec0bf9d0 0x7fc2ec0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:49.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.952+0000 7fc2ebfff640 1 --2- 172.21.15.1:0/487277737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2ec152650 0x7fc2ec155c90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc009870 tx=0x7fc2dc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17a2adfc140e98bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.952+0000 7fc2eaffd640 1 -- 172.21.15.1:0/487277737 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc035020 con 0x7fc2ec152650 2024-03-06T07:30:49.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.952+0000 7fc2eaffd640 1 -- 172.21.15.1:0/487277737 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2dc031640 con 0x7fc2ec152650 2024-03-06T07:30:49.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.952+0000 7fc2eaffd640 1 -- 172.21.15.1:0/487277737 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc02d040 con 0x7fc2ec152650 2024-03-06T07:30:49.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.953+0000 7fc2f2556640 1 -- 172.21.15.1:0/487277737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2ec152650 msgr2=0x7fc2ec155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.953+0000 7fc2f2556640 1 --2- 172.21.15.1:0/487277737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2ec152650 0x7fc2ec155c90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc009870 tx=0x7fc2dc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.953+0000 7fc2f2556640 1 -- 172.21.15.1:0/487277737 shutdown_connections 2024-03-06T07:30:49.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.953+0000 7fc2f2556640 1 --2- 172.21.15.1:0/487277737 >> v2:172.21.15.98:3300/0 conn(0x7fc2ec0bf9d0 0x7fc2ec0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.953+0000 7fc2f2556640 1 --2- 172.21.15.1:0/487277737 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2ec152650 0x7fc2ec155c90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.953+0000 7fc2f2556640 1 -- 172.21.15.1:0/487277737 >> 172.21.15.1:0/487277737 conn(0x7fc2ec066ee0 msgr2=0x7fc2ec0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.953+0000 7fc2f2556640 1 -- 172.21.15.1:0/487277737 shutdown_connections 2024-03-06T07:30:49.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.953+0000 7fc2f2556640 1 -- 172.21.15.1:0/487277737 wait complete. 2024-03-06T07:30:49.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.954+0000 7fc2f2556640 1 Processor -- start 2024-03-06T07:30:49.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.954+0000 7fc2f2556640 1 -- start start 2024-03-06T07:30:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.955+0000 7fc2f2556640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2ec0bf9d0 0x7fc2ec1eb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.955+0000 7fc2f2556640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2ec152650 0x7fc2ec0c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.955+0000 7fc2f2556640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2ec14b9d0 0x7fc2ec13d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.955+0000 7fc2f2556640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec1551b0 con 0x7fc2ec152650 2024-03-06T07:30:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.955+0000 7fc2f2556640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec0c6fa0 con 0x7fc2ec0bf9d0 2024-03-06T07:30:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.955+0000 7fc2f2556640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec155490 con 0x7fc2ec14b9d0 2024-03-06T07:30:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.956+0000 7fc2eb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2ec152650 0x7fc2ec0c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.956+0000 7fc2eb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2ec152650 0x7fc2ec0c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36462/0 (socket says 172.21.15.1:36462) 2024-03-06T07:30:49.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.956+0000 7fc2ebfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2ec0bf9d0 0x7fc2ec1eb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.956+0000 7fc2eb7fe640 1 -- 172.21.15.1:0/1998705403 learned_addr learned my addr 172.21.15.1:0/1998705403 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:49.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.956+0000 7fc2f0af3640 1 --2- 172.21.15.1:0/1998705403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2ec14b9d0 0x7fc2ec13d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.956+0000 7fc2eb7fe640 1 -- 172.21.15.1:0/1998705403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2ec14b9d0 msgr2=0x7fc2ec13d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.956+0000 7fc2eb7fe640 1 --2- 172.21.15.1:0/1998705403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2ec14b9d0 0x7fc2ec13d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.956+0000 7fc2eb7fe640 1 -- 172.21.15.1:0/1998705403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2ec0bf9d0 msgr2=0x7fc2ec1eb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.956+0000 7fc2eb7fe640 1 --2- 172.21.15.1:0/1998705403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2ec0bf9d0 0x7fc2ec1eb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.956+0000 7fc2eb7fe640 1 -- 172.21.15.1:0/1998705403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec154d30 con 0x7fc2ec152650 2024-03-06T07:30:49.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.956+0000 7fc2ebfff640 1 --2- 172.21.15.1:0/1998705403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2ec0bf9d0 0x7fc2ec1eb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:49.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.956+0000 7fc2eb7fe640 1 --2- 172.21.15.1:0/1998705403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2ec152650 0x7fc2ec0c9a90 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7fc2e000c4d0 tx=0x7fc2e000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.956+0000 7fc2e8ff9640 1 -- 172.21.15.1:0/1998705403 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e00074e0 con 0x7fc2ec152650 2024-03-06T07:30:49.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.956+0000 7fc2e8ff9640 1 -- 172.21.15.1:0/1998705403 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e00047e0 con 0x7fc2ec152650 2024-03-06T07:30:49.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.956+0000 7fc2f2556640 1 -- 172.21.15.1:0/1998705403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2ec154490 con 0x7fc2ec152650 2024-03-06T07:30:49.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.956+0000 7fc2e8ff9640 1 -- 172.21.15.1:0/1998705403 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e0004bd0 con 0x7fc2ec152650 2024-03-06T07:30:49.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.956+0000 7fc2f2556640 1 -- 172.21.15.1:0/1998705403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2ec154a50 con 0x7fc2ec152650 2024-03-06T07:30:49.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.958+0000 7fc2f2556640 1 -- 172.21.15.1:0/1998705403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2ec067fd0 con 0x7fc2ec152650 2024-03-06T07:30:49.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.958+0000 7fc2e8ff9640 1 -- 172.21.15.1:0/1998705403 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2e0011070 con 0x7fc2ec152650 2024-03-06T07:30:49.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.958+0000 7fc2e8ff9640 1 --2- 172.21.15.1:0/1998705403 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc2d806b500 0x7fc2d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.959+0000 7fc2e8ff9640 1 -- 172.21.15.1:0/1998705403 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fc2e0089fb0 con 0x7fc2ec152650 2024-03-06T07:30:49.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.959+0000 7fc2ebfff640 1 --2- 172.21.15.1:0/1998705403 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc2d806b500 0x7fc2d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.959+0000 7fc2ebfff640 1 --2- 172.21.15.1:0/1998705403 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc2d806b500 0x7fc2d806d9c0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc036000 tx=0x7fc2dc03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:49.963+0000 7fc2e8ff9640 1 -- 172.21.15.1:0/1998705403 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2e0058ba0 con 0x7fc2ec152650 2024-03-06T07:30:50.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:50.184+0000 7fc2f2556640 1 -- 172.21.15.1:0/1998705403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2ec0c34e0 con 0x7fc2ec152650 2024-03-06T07:30:50.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:50.184+0000 7fc2e8ff9640 1 -- 172.21.15.1:0/1998705403 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3422 (secure 0 0 0) 0x7fc2e00584a0 con 0x7fc2ec152650 2024-03-06T07:30:50.187 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:50.187 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:49.314619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:50.187 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 207 2024-03-06T07:30:50.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:50.187+0000 7fc2f2556640 1 -- 172.21.15.1:0/1998705403 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc2d806b500 msgr2=0x7fc2d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:50.187+0000 7fc2f2556640 1 --2- 172.21.15.1:0/1998705403 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc2d806b500 0x7fc2d806d9c0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc036000 tx=0x7fc2dc03c040 comp rx=0 tx=0).stop 2024-03-06T07:30:50.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:50.187+0000 7fc2f2556640 1 -- 172.21.15.1:0/1998705403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2ec152650 msgr2=0x7fc2ec0c9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:50.187+0000 7fc2f2556640 1 --2- 172.21.15.1:0/1998705403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2ec152650 0x7fc2ec0c9a90 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7fc2e000c4d0 tx=0x7fc2e000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:50.187+0000 7fc2f2556640 1 -- 172.21.15.1:0/1998705403 shutdown_connections 2024-03-06T07:30:50.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:50.187+0000 7fc2f2556640 1 --2- 172.21.15.1:0/1998705403 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc2d806b500 0x7fc2d806d9c0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:50.187+0000 7fc2f2556640 1 --2- 172.21.15.1:0/1998705403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2ec14b9d0 0x7fc2ec13d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:50.187+0000 7fc2f2556640 1 --2- 172.21.15.1:0/1998705403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2ec152650 0x7fc2ec0c9a90 unknown :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:50.187+0000 7fc2f2556640 1 --2- 172.21.15.1:0/1998705403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2ec0bf9d0 0x7fc2ec1eb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:50.187+0000 7fc2f2556640 1 -- 172.21.15.1:0/1998705403 >> 172.21.15.1:0/1998705403 conn(0x7fc2ec066ee0 msgr2=0x7fc2ec0c0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:50.187+0000 7fc2f2556640 1 -- 172.21.15.1:0/1998705403 shutdown_connections 2024-03-06T07:30:50.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:50.187+0000 7fc2f2556640 1 -- 172.21.15.1:0/1998705403 wait complete. 2024-03-06T07:30:50.602 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:49.314619+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:50.602 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:replay 2024-03-06T07:30:51.604 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.703+0000 7f12a433e640 1 Processor -- start 2024-03-06T07:30:51.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.703+0000 7f12a433e640 1 -- start start 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.704+0000 7f12a433e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f129c150520 0x7f129c1543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.704+0000 7f12a433e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f129c0c32f0 0x7f129c1537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.704+0000 7f12a433e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f129c070c80 0x7f129c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.704+0000 7f12a433e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f129c150900 con 0x7f129c0c32f0 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.704+0000 7f12a433e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f129c071790 con 0x7f129c070c80 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.704+0000 7f12a433e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129c157050 con 0x7f129c150520 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.704+0000 7f12a18d9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f129c0c32f0 0x7f129c1537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.704+0000 7f12a18d9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f129c0c32f0 0x7f129c1537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36476/0 (socket says 172.21.15.1:36476) 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.704+0000 7f12a18d9640 1 -- 172.21.15.1:0/3583596362 learned_addr learned my addr 172.21.15.1:0/3583596362 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.705+0000 7f12a18d9640 1 -- 172.21.15.1:0/3583596362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f129c150520 msgr2=0x7f129c1543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.705+0000 7f12a18d9640 1 --2- 172.21.15.1:0/3583596362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f129c150520 0x7f129c1543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.705+0000 7f12a18d9640 1 -- 172.21.15.1:0/3583596362 >> v2:172.21.15.98:3300/0 conn(0x7f129c070c80 msgr2=0x7f129c0710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.705+0000 7f12a18d9640 1 --2- 172.21.15.1:0/3583596362 >> v2:172.21.15.98:3300/0 conn(0x7f129c070c80 0x7f129c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.705+0000 7f12a18d9640 1 -- 172.21.15.1:0/3583596362 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c152fc0 con 0x7f129c0c32f0 2024-03-06T07:30:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.705+0000 7f12a18d9640 1 --2- 172.21.15.1:0/3583596362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f129c0c32f0 0x7f129c1537c0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f129c0c9990 tx=0x7f129002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5f5cf40a259090 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.705+0000 7f12a10d8640 1 -- 172.21.15.1:0/3583596362 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1290035020 con 0x7f129c0c32f0 2024-03-06T07:30:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.705+0000 7f12a10d8640 1 -- 172.21.15.1:0/3583596362 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1290031870 con 0x7f129c0c32f0 2024-03-06T07:30:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.705+0000 7f12a10d8640 1 -- 172.21.15.1:0/3583596362 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1290031c40 con 0x7f129c0c32f0 2024-03-06T07:30:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.706+0000 7f12a433e640 1 -- 172.21.15.1:0/3583596362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f129c0c32f0 msgr2=0x7f129c1537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.706+0000 7f12a433e640 1 --2- 172.21.15.1:0/3583596362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f129c0c32f0 0x7f129c1537c0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f129c0c9990 tx=0x7f129002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.706+0000 7f12a433e640 1 -- 172.21.15.1:0/3583596362 shutdown_connections 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.706+0000 7f12a433e640 1 --2- 172.21.15.1:0/3583596362 >> v2:172.21.15.98:3300/0 conn(0x7f129c070c80 0x7f129c0710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.706+0000 7f12a433e640 1 --2- 172.21.15.1:0/3583596362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f129c0c32f0 0x7f129c1537c0 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.706+0000 7f12a433e640 1 --2- 172.21.15.1:0/3583596362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f129c150520 0x7f129c1543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.706+0000 7f12a433e640 1 -- 172.21.15.1:0/3583596362 >> 172.21.15.1:0/3583596362 conn(0x7f129c0c88f0 msgr2=0x7f129c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.706+0000 7f12a433e640 1 -- 172.21.15.1:0/3583596362 shutdown_connections 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.706+0000 7f12a433e640 1 -- 172.21.15.1:0/3583596362 wait complete. 2024-03-06T07:30:51.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.708+0000 7f12a433e640 1 Processor -- start 2024-03-06T07:30:51.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.708+0000 7f12a433e640 1 -- start start 2024-03-06T07:30:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.709+0000 7f12a433e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f129c070c80 0x7f129c145ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.709+0000 7f12a433e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f129c0c32f0 0x7f129c146420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.709+0000 7f12a433e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f129c150520 0x7f129c155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.709+0000 7f12a433e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c0c7ec0 con 0x7f129c150520 2024-03-06T07:30:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.709+0000 7f12a433e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c157330 con 0x7f129c0c32f0 2024-03-06T07:30:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.709+0000 7f12a433e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129c153d00 con 0x7f129c070c80 2024-03-06T07:30:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.709+0000 7f12a28db640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f129c150520 0x7f129c155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.709+0000 7f12a28db640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f129c150520 0x7f129c155a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36492/0 (socket says 172.21.15.1:36492) 2024-03-06T07:30:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.709+0000 7f12a28db640 1 -- 172.21.15.1:0/986002010 learned_addr learned my addr 172.21.15.1:0/986002010 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.709+0000 7f12a20da640 1 --2- 172.21.15.1:0/986002010 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f129c070c80 0x7f129c145ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.709+0000 7f12a18d9640 1 --2- 172.21.15.1:0/986002010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f129c0c32f0 0x7f129c146420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.710+0000 7f12a28db640 1 -- 172.21.15.1:0/986002010 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f129c070c80 msgr2=0x7f129c145ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.710+0000 7f12a28db640 1 --2- 172.21.15.1:0/986002010 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f129c070c80 0x7f129c145ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.710+0000 7f12a28db640 1 -- 172.21.15.1:0/986002010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f129c0c32f0 msgr2=0x7f129c146420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.710+0000 7f12a28db640 1 --2- 172.21.15.1:0/986002010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f129c0c32f0 0x7f129c146420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.710+0000 7f12a28db640 1 -- 172.21.15.1:0/986002010 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c14f7c0 con 0x7f129c150520 2024-03-06T07:30:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.710+0000 7f12a18d9640 1 --2- 172.21.15.1:0/986002010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f129c0c32f0 0x7f129c146420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.710+0000 7f12a28db640 1 --2- 172.21.15.1:0/986002010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f129c150520 0x7f129c155a30 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f1294009fd0 tx=0x7f129400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.710+0000 7f129affd640 1 -- 172.21.15.1:0/986002010 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1294011830 con 0x7f129c150520 2024-03-06T07:30:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.710+0000 7f129affd640 1 -- 172.21.15.1:0/986002010 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1294016070 con 0x7f129c150520 2024-03-06T07:30:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.710+0000 7f12a433e640 1 -- 172.21.15.1:0/986002010 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f129c0a9010 con 0x7f129c150520 2024-03-06T07:30:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.710+0000 7f129affd640 1 -- 172.21.15.1:0/986002010 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1294011bf0 con 0x7f129c150520 2024-03-06T07:30:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.710+0000 7f12a433e640 1 -- 172.21.15.1:0/986002010 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f129c156cf0 con 0x7f129c150520 2024-03-06T07:30:51.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.711+0000 7f12a433e640 1 -- 172.21.15.1:0/986002010 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129c0c64e0 con 0x7f129c150520 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.712+0000 7f129affd640 1 -- 172.21.15.1:0/986002010 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f129401b460 con 0x7f129c150520 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.712+0000 7f129affd640 1 --2- 172.21.15.1:0/986002010 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f128806b4d0 0x7f128806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.712+0000 7f129affd640 1 -- 172.21.15.1:0/986002010 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f129408a200 con 0x7f129c150520 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.712+0000 7f12a20da640 1 --2- 172.21.15.1:0/986002010 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f128806b4d0 0x7f128806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.713+0000 7f12a20da640 1 --2- 172.21.15.1:0/986002010 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f128806b4d0 0x7f128806d990 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f129c0a8bf0 tx=0x7f128c0094f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.717+0000 7f129affd640 1 -- 172.21.15.1:0/986002010 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f129c0c64e0 con 0x7f129c150520 2024-03-06T07:30:51.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.937+0000 7f12a433e640 1 -- 172.21.15.1:0/986002010 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f129c150d20 con 0x7f129c150520 2024-03-06T07:30:51.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.937+0000 7f129affd640 1 -- 172.21.15.1:0/986002010 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+3426 (secure 0 0 0) 0x7f129c150d20 con 0x7f129c150520 2024-03-06T07:30:51.939 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:51.940 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":209,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:50.320455+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:51.940 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 209 2024-03-06T07:30:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.940+0000 7f12a433e640 1 -- 172.21.15.1:0/986002010 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f128806b4d0 msgr2=0x7f128806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.940+0000 7f12a433e640 1 --2- 172.21.15.1:0/986002010 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f128806b4d0 0x7f128806d990 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f129c0a8bf0 tx=0x7f128c0094f0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.940+0000 7f12a433e640 1 -- 172.21.15.1:0/986002010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f129c150520 msgr2=0x7f129c155a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.940+0000 7f12a433e640 1 --2- 172.21.15.1:0/986002010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f129c150520 0x7f129c155a30 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f1294009fd0 tx=0x7f129400b8e0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.940+0000 7f12a433e640 1 -- 172.21.15.1:0/986002010 shutdown_connections 2024-03-06T07:30:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.940+0000 7f12a433e640 1 --2- 172.21.15.1:0/986002010 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f128806b4d0 0x7f128806d990 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.940+0000 7f12a433e640 1 --2- 172.21.15.1:0/986002010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f129c150520 0x7f129c155a30 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.940+0000 7f12a433e640 1 --2- 172.21.15.1:0/986002010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f129c0c32f0 0x7f129c146420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.940+0000 7f12a433e640 1 --2- 172.21.15.1:0/986002010 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f129c070c80 0x7f129c145ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.940+0000 7f12a433e640 1 -- 172.21.15.1:0/986002010 >> 172.21.15.1:0/986002010 conn(0x7f129c0c88f0 msgr2=0x7f129c14aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.940+0000 7f12a433e640 1 -- 172.21.15.1:0/986002010 shutdown_connections 2024-03-06T07:30:51.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:51.940+0000 7f12a433e640 1 -- 172.21.15.1:0/986002010 wait complete. 2024-03-06T07:30:52.355 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 209, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:50.320455+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:reconnect', 'state_seq': 23, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:52.355 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:reconnect 2024-03-06T07:30:53.356 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:53.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.456+0000 7f3c38015640 1 Processor -- start 2024-03-06T07:30:53.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.456+0000 7f3c38015640 1 -- start start 2024-03-06T07:30:53.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.456+0000 7f3c38015640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3c30152650 0x7f3c30155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.456+0000 7f3c38015640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c300bf9d0 0x7f3c300c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.456+0000 7f3c38015640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c300c2fc0 con 0x7f3c300c6b10 2024-03-06T07:30:53.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.456+0000 7f3c38015640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c30155800 con 0x7f3c30152650 2024-03-06T07:30:53.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.456+0000 7f3c38015640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c30158da0 con 0x7f3c300bf9d0 2024-03-06T07:30:53.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.457+0000 7f3c35db1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3c30152650 0x7f3c30155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.457+0000 7f3c35db1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3c30152650 0x7f3c30155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36494/0 (socket says 172.21.15.1:36494) 2024-03-06T07:30:53.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.457+0000 7f3c35db1640 1 -- 172.21.15.1:0/2365175117 learned_addr learned my addr 172.21.15.1:0/2365175117 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:53.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.457+0000 7f3c365b2640 1 --2- 172.21.15.1:0/2365175117 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c300bf9d0 0x7f3c300c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.457+0000 7f3c35db1640 1 -- 172.21.15.1:0/2365175117 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c300bf9d0 msgr2=0x7f3c300c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.457+0000 7f3c35db1640 1 --2- 172.21.15.1:0/2365175117 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c300bf9d0 0x7f3c300c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.457+0000 7f3c35db1640 1 -- 172.21.15.1:0/2365175117 >> v1:172.21.15.98:6789/0 conn(0x7f3c300c6b10 legacy=0x7f3c30158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.457+0000 7f3c35db1640 1 -- 172.21.15.1:0/2365175117 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c300c6690 con 0x7f3c30152650 2024-03-06T07:30:53.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.458+0000 7f3c35db1640 1 --2- 172.21.15.1:0/2365175117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c30152650 0x7f3c30155c90 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c009870 tx=0x7f3c2c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2536ed09dd56d77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.458+0000 7f3c34daf640 1 -- 172.21.15.1:0/2365175117 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c035020 con 0x7f3c30152650 2024-03-06T07:30:53.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.458+0000 7f3c34daf640 1 -- 172.21.15.1:0/2365175117 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2c031640 con 0x7f3c30152650 2024-03-06T07:30:53.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.458+0000 7f3c34daf640 1 -- 172.21.15.1:0/2365175117 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c02d040 con 0x7f3c30152650 2024-03-06T07:30:53.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.458+0000 7f3c38015640 1 -- 172.21.15.1:0/2365175117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c30152650 msgr2=0x7f3c30155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.458+0000 7f3c38015640 1 --2- 172.21.15.1:0/2365175117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c30152650 0x7f3c30155c90 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c009870 tx=0x7f3c2c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.459+0000 7f3c38015640 1 -- 172.21.15.1:0/2365175117 shutdown_connections 2024-03-06T07:30:53.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.459+0000 7f3c38015640 1 --2- 172.21.15.1:0/2365175117 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c300bf9d0 0x7f3c300c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.459+0000 7f3c38015640 1 --2- 172.21.15.1:0/2365175117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c30152650 0x7f3c30155c90 unknown :-1 s=CLOSED pgs=2465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.459+0000 7f3c38015640 1 -- 172.21.15.1:0/2365175117 >> 172.21.15.1:0/2365175117 conn(0x7f3c30066ee0 msgr2=0x7f3c300672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.459+0000 7f3c38015640 1 -- 172.21.15.1:0/2365175117 shutdown_connections 2024-03-06T07:30:53.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.459+0000 7f3c38015640 1 -- 172.21.15.1:0/2365175117 wait complete. 2024-03-06T07:30:53.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.460+0000 7f3c38015640 1 Processor -- start 2024-03-06T07:30:53.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.460+0000 7f3c38015640 1 -- start start 2024-03-06T07:30:53.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.460+0000 7f3c38015640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c300bf9d0 0x7f3c30154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.460+0000 7f3c38015640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c30152650 0x7f3c301ee7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.460+0000 7f3c38015640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c3014b9e0 0x7f3c30156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.460+0000 7f3c38015640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c301551b0 con 0x7f3c300bf9d0 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.460+0000 7f3c38015640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c300c6fa0 con 0x7f3c3014b9e0 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.460+0000 7f3c38015640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c30155490 con 0x7f3c30152650 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.461+0000 7f3c35db1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c300bf9d0 0x7f3c30154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.461+0000 7f3c35db1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c300bf9d0 0x7f3c30154560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36500/0 (socket says 172.21.15.1:36500) 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.461+0000 7f3c35db1640 1 -- 172.21.15.1:0/84231088 learned_addr learned my addr 172.21.15.1:0/84231088 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.461+0000 7f3c365b2640 1 --2- 172.21.15.1:0/84231088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c3014b9e0 0x7f3c30156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.461+0000 7f3c355b0640 1 --2- 172.21.15.1:0/84231088 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c30152650 0x7f3c301ee7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.461+0000 7f3c35db1640 1 -- 172.21.15.1:0/84231088 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c30152650 msgr2=0x7f3c301ee7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.461+0000 7f3c35db1640 1 --2- 172.21.15.1:0/84231088 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c30152650 0x7f3c301ee7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.461+0000 7f3c35db1640 1 -- 172.21.15.1:0/84231088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c3014b9e0 msgr2=0x7f3c30156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.461+0000 7f3c35db1640 1 --2- 172.21.15.1:0/84231088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c3014b9e0 0x7f3c30156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.461+0000 7f3c35db1640 1 -- 172.21.15.1:0/84231088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c30151ec0 con 0x7f3c300bf9d0 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.461+0000 7f3c35db1640 1 --2- 172.21.15.1:0/84231088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c300bf9d0 0x7f3c30154560 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c02b930 tx=0x7f3c2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.461+0000 7f3c267fc640 1 -- 172.21.15.1:0/84231088 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c040070 con 0x7f3c300bf9d0 2024-03-06T07:30:53.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.461+0000 7f3c267fc640 1 -- 172.21.15.1:0/84231088 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2c03e070 con 0x7f3c300bf9d0 2024-03-06T07:30:53.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.461+0000 7f3c38015640 1 -- 172.21.15.1:0/84231088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c301538a0 con 0x7f3c300bf9d0 2024-03-06T07:30:53.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.461+0000 7f3c267fc640 1 -- 172.21.15.1:0/84231088 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c035040 con 0x7f3c300bf9d0 2024-03-06T07:30:53.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.461+0000 7f3c38015640 1 -- 172.21.15.1:0/84231088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c301ef0a0 con 0x7f3c300bf9d0 2024-03-06T07:30:53.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.462+0000 7f3c38015640 1 -- 172.21.15.1:0/84231088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c301ef480 con 0x7f3c300bf9d0 2024-03-06T07:30:53.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.463+0000 7f3c267fc640 1 -- 172.21.15.1:0/84231088 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c2c038020 con 0x7f3c300bf9d0 2024-03-06T07:30:53.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.464+0000 7f3c267fc640 1 --2- 172.21.15.1:0/84231088 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c1406b4d0 0x7f3c1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.464+0000 7f3c267fc640 1 -- 172.21.15.1:0/84231088 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f3c2c0abdb0 con 0x7f3c300bf9d0 2024-03-06T07:30:53.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.464+0000 7f3c355b0640 1 --2- 172.21.15.1:0/84231088 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c1406b4d0 0x7f3c1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.464+0000 7f3c355b0640 1 --2- 172.21.15.1:0/84231088 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c1406b4d0 0x7f3c1406d990 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f3c300c1ea0 tx=0x7f3c20009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.469+0000 7f3c267fc640 1 -- 172.21.15.1:0/84231088 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c2c07aa30 con 0x7f3c300bf9d0 2024-03-06T07:30:53.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.697+0000 7f3c38015640 1 -- 172.21.15.1:0/84231088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c30067fd0 con 0x7f3c300bf9d0 2024-03-06T07:30:53.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.697+0000 7f3c267fc640 1 -- 172.21.15.1:0/84231088 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7f3c2c07a330 con 0x7f3c300bf9d0 2024-03-06T07:30:53.699 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:53.699 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:53.700 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:30:53.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.699+0000 7f3c38015640 1 -- 172.21.15.1:0/84231088 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c1406b4d0 msgr2=0x7f3c1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.699+0000 7f3c38015640 1 --2- 172.21.15.1:0/84231088 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c1406b4d0 0x7f3c1406d990 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f3c300c1ea0 tx=0x7f3c20009040 comp rx=0 tx=0).stop 2024-03-06T07:30:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.699+0000 7f3c38015640 1 -- 172.21.15.1:0/84231088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c300bf9d0 msgr2=0x7f3c30154560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.699+0000 7f3c38015640 1 --2- 172.21.15.1:0/84231088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c300bf9d0 0x7f3c30154560 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c02b930 tx=0x7f3c2c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.699+0000 7f3c38015640 1 -- 172.21.15.1:0/84231088 shutdown_connections 2024-03-06T07:30:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.699+0000 7f3c38015640 1 --2- 172.21.15.1:0/84231088 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c1406b4d0 0x7f3c1406d990 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.699+0000 7f3c38015640 1 --2- 172.21.15.1:0/84231088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c3014b9e0 0x7f3c30156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.699+0000 7f3c38015640 1 --2- 172.21.15.1:0/84231088 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c30152650 0x7f3c301ee7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.699+0000 7f3c38015640 1 --2- 172.21.15.1:0/84231088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c300bf9d0 0x7f3c30154560 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.699+0000 7f3c38015640 1 -- 172.21.15.1:0/84231088 >> 172.21.15.1:0/84231088 conn(0x7f3c30066ee0 msgr2=0x7f3c300672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.699+0000 7f3c38015640 1 -- 172.21.15.1:0/84231088 shutdown_connections 2024-03-06T07:30:53.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:53.699+0000 7f3c38015640 1 -- 172.21.15.1:0/84231088 wait complete. 2024-03-06T07:30:54.114 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:54.114 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:30:55.115 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:55.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.216+0000 7f6a534c3640 1 Processor -- start 2024-03-06T07:30:55.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.216+0000 7f6a534c3640 1 -- start start 2024-03-06T07:30:55.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.217+0000 7f6a534c3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6a4c152650 0x7f6a4c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.217+0000 7f6a534c3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a4c0c6b10 0x7f6a4c158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.217+0000 7f6a534c3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4c0c2fc0 con 0x7f6a4c0bf9d0 2024-03-06T07:30:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.217+0000 7f6a534c3640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a4c155800 con 0x7f6a4c152650 2024-03-06T07:30:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.217+0000 7f6a534c3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c0c6690 con 0x7f6a4c0c6b10 2024-03-06T07:30:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.217+0000 7f6a5125f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6a4c152650 0x7f6a4c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.217+0000 7f6a5125f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6a4c152650 0x7f6a4c155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58040/0 (socket says 172.21.15.1:58040) 2024-03-06T07:30:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.217+0000 7f6a5125f640 1 -- 172.21.15.1:0/2675935983 learned_addr learned my addr 172.21.15.1:0/2675935983 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.217+0000 7f6a50a5e640 1 --2- 172.21.15.1:0/2675935983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a4c0c6b10 0x7f6a4c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.217+0000 7f6a5125f640 1 -- 172.21.15.1:0/2675935983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a4c0c6b10 msgr2=0x7f6a4c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.217+0000 7f6a5125f640 1 --2- 172.21.15.1:0/2675935983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a4c0c6b10 0x7f6a4c158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.217+0000 7f6a5125f640 1 -- 172.21.15.1:0/2675935983 >> v1:172.21.15.98:6789/0 conn(0x7f6a4c0bf9d0 legacy=0x7f6a4c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.217+0000 7f6a5125f640 1 -- 172.21.15.1:0/2675935983 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4c0c7280 con 0x7f6a4c152650 2024-03-06T07:30:55.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.218+0000 7f6a5125f640 1 --2- 172.21.15.1:0/2675935983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a4c152650 0x7f6a4c155c90 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f6a40009870 tx=0x7f6a4002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793b4967e1b534a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.218+0000 7f6a3ffff640 1 -- 172.21.15.1:0/2675935983 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a400317c0 con 0x7f6a4c152650 2024-03-06T07:30:55.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.218+0000 7f6a3ffff640 1 -- 172.21.15.1:0/2675935983 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a40037070 con 0x7f6a4c152650 2024-03-06T07:30:55.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.218+0000 7f6a3ffff640 1 -- 172.21.15.1:0/2675935983 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a40031b80 con 0x7f6a4c152650 2024-03-06T07:30:55.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.219+0000 7f6a534c3640 1 -- 172.21.15.1:0/2675935983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a4c152650 msgr2=0x7f6a4c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.219+0000 7f6a534c3640 1 --2- 172.21.15.1:0/2675935983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a4c152650 0x7f6a4c155c90 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f6a40009870 tx=0x7f6a4002b860 comp rx=0 tx=0).stop 2024-03-06T07:30:55.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.219+0000 7f6a534c3640 1 -- 172.21.15.1:0/2675935983 shutdown_connections 2024-03-06T07:30:55.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.219+0000 7f6a534c3640 1 --2- 172.21.15.1:0/2675935983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a4c0c6b10 0x7f6a4c158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.219+0000 7f6a534c3640 1 --2- 172.21.15.1:0/2675935983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a4c152650 0x7f6a4c155c90 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.219+0000 7f6a534c3640 1 -- 172.21.15.1:0/2675935983 >> 172.21.15.1:0/2675935983 conn(0x7f6a4c066ee0 msgr2=0x7f6a4c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.219+0000 7f6a534c3640 1 -- 172.21.15.1:0/2675935983 shutdown_connections 2024-03-06T07:30:55.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.219+0000 7f6a534c3640 1 -- 172.21.15.1:0/2675935983 wait complete. 2024-03-06T07:30:55.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.220+0000 7f6a534c3640 1 Processor -- start 2024-03-06T07:30:55.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.220+0000 7f6a534c3640 1 -- start start 2024-03-06T07:30:55.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.221+0000 7f6a534c3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a4c0bf9d0 0x7f6a4c14abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.221+0000 7f6a534c3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a4c152650 0x7f6a4c1eaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.221+0000 7f6a534c3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a4c158b80 0x7f6a4c1eb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.221+0000 7f6a534c3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c155490 con 0x7f6a4c0bf9d0 2024-03-06T07:30:55.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.221+0000 7f6a534c3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c1551b0 con 0x7f6a4c158b80 2024-03-06T07:30:55.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.221+0000 7f6a534c3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c152a30 con 0x7f6a4c152650 2024-03-06T07:30:55.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.221+0000 7f6a5125f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a4c0bf9d0 0x7f6a4c14abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.221+0000 7f6a5125f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a4c0bf9d0 0x7f6a4c14abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58056/0 (socket says 172.21.15.1:58056) 2024-03-06T07:30:55.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.221+0000 7f6a5125f640 1 -- 172.21.15.1:0/3060930336 learned_addr learned my addr 172.21.15.1:0/3060930336 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:55.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.221+0000 7f6a50a5e640 1 --2- 172.21.15.1:0/3060930336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a4c152650 0x7f6a4c1eaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.221+0000 7f6a5125f640 1 -- 172.21.15.1:0/3060930336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a4c152650 msgr2=0x7f6a4c1eaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.221+0000 7f6a5125f640 1 --2- 172.21.15.1:0/3060930336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a4c152650 0x7f6a4c1eaee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.221+0000 7f6a5125f640 1 -- 172.21.15.1:0/3060930336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a4c158b80 msgr2=0x7f6a4c1eb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.221+0000 7f6a51a60640 1 --2- 172.21.15.1:0/3060930336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a4c158b80 0x7f6a4c1eb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.221+0000 7f6a5125f640 1 --2- 172.21.15.1:0/3060930336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a4c158b80 0x7f6a4c1eb420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.221+0000 7f6a5125f640 1 -- 172.21.15.1:0/3060930336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4c152f10 con 0x7f6a4c0bf9d0 2024-03-06T07:30:55.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.221+0000 7f6a5125f640 1 --2- 172.21.15.1:0/3060930336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a4c0bf9d0 0x7f6a4c14abf0 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f6a40009870 tx=0x7f6a400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.222+0000 7f6a3dffb640 1 -- 172.21.15.1:0/3060930336 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a40032650 con 0x7f6a4c0bf9d0 2024-03-06T07:30:55.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.222+0000 7f6a3dffb640 1 -- 172.21.15.1:0/3060930336 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a400327f0 con 0x7f6a4c0bf9d0 2024-03-06T07:30:55.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.222+0000 7f6a534c3640 1 -- 172.21.15.1:0/3060930336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a4c0ca380 con 0x7f6a4c0bf9d0 2024-03-06T07:30:55.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.222+0000 7f6a3dffb640 1 -- 172.21.15.1:0/3060930336 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a40032be0 con 0x7f6a4c0bf9d0 2024-03-06T07:30:55.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.222+0000 7f6a534c3640 1 -- 172.21.15.1:0/3060930336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a4c14b130 con 0x7f6a4c0bf9d0 2024-03-06T07:30:55.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.223+0000 7f6a534c3640 1 -- 172.21.15.1:0/3060930336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a4c067fd0 con 0x7f6a4c0bf9d0 2024-03-06T07:30:55.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.223+0000 7f6a3dffb640 1 -- 172.21.15.1:0/3060930336 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a4004c060 con 0x7f6a4c0bf9d0 2024-03-06T07:30:55.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.224+0000 7f6a3dffb640 1 --2- 172.21.15.1:0/3060930336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6a3006b4d0 0x7f6a3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.224+0000 7f6a3dffb640 1 -- 172.21.15.1:0/3060930336 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f6a400abbb0 con 0x7f6a4c0bf9d0 2024-03-06T07:30:55.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.224+0000 7f6a50a5e640 1 --2- 172.21.15.1:0/3060930336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6a3006b4d0 0x7f6a3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.224+0000 7f6a50a5e640 1 --2- 172.21.15.1:0/3060930336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6a3006b4d0 0x7f6a3006d990 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c14f200 tx=0x7f6a48008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.228+0000 7f6a3dffb640 1 -- 172.21.15.1:0/3060930336 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a40040070 con 0x7f6a4c0bf9d0 2024-03-06T07:30:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.451+0000 7f6a534c3640 1 -- 172.21.15.1:0/3060930336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a4c0b06f0 con 0x7f6a4c0bf9d0 2024-03-06T07:30:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.451+0000 7f6a3dffb640 1 -- 172.21.15.1:0/3060930336 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7f6a4007a830 con 0x7f6a4c0bf9d0 2024-03-06T07:30:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:55.452 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:30:55.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.453+0000 7f6a534c3640 1 -- 172.21.15.1:0/3060930336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6a3006b4d0 msgr2=0x7f6a3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.453+0000 7f6a534c3640 1 --2- 172.21.15.1:0/3060930336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6a3006b4d0 0x7f6a3006d990 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c14f200 tx=0x7f6a48008040 comp rx=0 tx=0).stop 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.453+0000 7f6a534c3640 1 -- 172.21.15.1:0/3060930336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a4c0bf9d0 msgr2=0x7f6a4c14abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.453+0000 7f6a534c3640 1 --2- 172.21.15.1:0/3060930336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a4c0bf9d0 0x7f6a4c14abf0 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f6a40009870 tx=0x7f6a400046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.453+0000 7f6a534c3640 1 -- 172.21.15.1:0/3060930336 shutdown_connections 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.453+0000 7f6a534c3640 1 --2- 172.21.15.1:0/3060930336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6a3006b4d0 0x7f6a3006d990 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.453+0000 7f6a534c3640 1 --2- 172.21.15.1:0/3060930336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a4c158b80 0x7f6a4c1eb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.453+0000 7f6a534c3640 1 --2- 172.21.15.1:0/3060930336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a4c152650 0x7f6a4c1eaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.453+0000 7f6a534c3640 1 --2- 172.21.15.1:0/3060930336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a4c0bf9d0 0x7f6a4c14abf0 unknown :-1 s=CLOSED pgs=2468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.453+0000 7f6a534c3640 1 -- 172.21.15.1:0/3060930336 >> 172.21.15.1:0/3060930336 conn(0x7f6a4c066ee0 msgr2=0x7f6a4c0c3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.453+0000 7f6a534c3640 1 -- 172.21.15.1:0/3060930336 shutdown_connections 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:55.453+0000 7f6a534c3640 1 -- 172.21.15.1:0/3060930336 wait complete. 2024-03-06T07:30:55.867 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:55.867 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:30:56.868 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.995+0000 7f5a6d0c9640 1 Processor -- start 2024-03-06T07:30:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.995+0000 7f5a6d0c9640 1 -- start start 2024-03-06T07:30:56.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.996+0000 7f5a6d0c9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5a68152840 0x7f5a680c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.996+0000 7f5a6d0c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a680c60b0 0x7f5a681553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.996+0000 7f5a6d0c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a680c5e40 con 0x7f5a680c3970 2024-03-06T07:30:56.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.996+0000 7f5a6d0c9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a680c67f0 con 0x7f5a68152840 2024-03-06T07:30:56.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.996+0000 7f5a6d0c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a680c6a40 con 0x7f5a680c60b0 2024-03-06T07:30:56.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.996+0000 7f5a5ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5a68152840 0x7f5a680c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.996+0000 7f5a5ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5a68152840 0x7f5a680c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58066/0 (socket says 172.21.15.1:58066) 2024-03-06T07:30:56.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.996+0000 7f5a5ffff640 1 -- 172.21.15.1:0/979258197 learned_addr learned my addr 172.21.15.1:0/979258197 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:56.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.996+0000 7f5a6759e640 1 --2- 172.21.15.1:0/979258197 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a680c60b0 0x7f5a681553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.996+0000 7f5a5ffff640 1 -- 172.21.15.1:0/979258197 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a680c60b0 msgr2=0x7f5a681553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:57.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.997+0000 7f5a5ffff640 1 --2- 172.21.15.1:0/979258197 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a680c60b0 0x7f5a681553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.997+0000 7f5a5ffff640 1 -- 172.21.15.1:0/979258197 >> v1:172.21.15.98:6789/0 conn(0x7f5a680c3970 legacy=0x7f5a68152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:57.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.997+0000 7f5a5ffff640 1 -- 172.21.15.1:0/979258197 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a680c6d20 con 0x7f5a68152840 2024-03-06T07:30:57.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.997+0000 7f5a6759e640 1 --2- 172.21.15.1:0/979258197 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a680c60b0 0x7f5a681553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:57.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.997+0000 7f5a5ffff640 1 --2- 172.21.15.1:0/979258197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a68152840 0x7f5a680c8170 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f5a50009870 tx=0x7f5a5002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc95402ae2fd5c0f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.997+0000 7f5a6659c640 1 -- 172.21.15.1:0/979258197 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a50035020 con 0x7f5a68152840 2024-03-06T07:30:57.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.997+0000 7f5a6659c640 1 -- 172.21.15.1:0/979258197 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a50031640 con 0x7f5a68152840 2024-03-06T07:30:57.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.997+0000 7f5a6659c640 1 -- 172.21.15.1:0/979258197 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5002d040 con 0x7f5a68152840 2024-03-06T07:30:57.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.998+0000 7f5a6d0c9640 1 -- 172.21.15.1:0/979258197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a68152840 msgr2=0x7f5a680c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.998+0000 7f5a6d0c9640 1 --2- 172.21.15.1:0/979258197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a68152840 0x7f5a680c8170 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f5a50009870 tx=0x7f5a5002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.998+0000 7f5a6d0c9640 1 -- 172.21.15.1:0/979258197 shutdown_connections 2024-03-06T07:30:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.998+0000 7f5a6d0c9640 1 --2- 172.21.15.1:0/979258197 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a680c60b0 0x7f5a681553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.998+0000 7f5a6d0c9640 1 --2- 172.21.15.1:0/979258197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a68152840 0x7f5a680c8170 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.998+0000 7f5a6d0c9640 1 -- 172.21.15.1:0/979258197 >> 172.21.15.1:0/979258197 conn(0x7f5a680c88f0 msgr2=0x7f5a680c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.998+0000 7f5a6d0c9640 1 -- 172.21.15.1:0/979258197 shutdown_connections 2024-03-06T07:30:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.998+0000 7f5a6d0c9640 1 -- 172.21.15.1:0/979258197 wait complete. 2024-03-06T07:30:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.999+0000 7f5a6d0c9640 1 Processor -- start 2024-03-06T07:30:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:56.999+0000 7f5a6d0c9640 1 -- start start 2024-03-06T07:30:57.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.000+0000 7f5a6d0c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a680c3970 0x7f5a681541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.000+0000 7f5a6d0c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a680c60b0 0x7f5a6806f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.000+0000 7f5a6d0c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a68152840 0x7f5a6806fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.000+0000 7f5a6d0c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a680c75c0 con 0x7f5a68152840 2024-03-06T07:30:57.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.000+0000 7f5a6d0c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a680c72e0 con 0x7f5a680c60b0 2024-03-06T07:30:57.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.000+0000 7f5a6d0c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a680c3d50 con 0x7f5a680c3970 2024-03-06T07:30:57.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.000+0000 7f5a6759e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a68152840 0x7f5a6806fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.000+0000 7f5a6759e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a68152840 0x7f5a6806fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58078/0 (socket says 172.21.15.1:58078) 2024-03-06T07:30:57.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.000+0000 7f5a6759e640 1 -- 172.21.15.1:0/4128924706 learned_addr learned my addr 172.21.15.1:0/4128924706 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:57.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.000+0000 7f5a66d9d640 1 --2- 172.21.15.1:0/4128924706 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a680c3970 0x7f5a681541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.000+0000 7f5a5ffff640 1 --2- 172.21.15.1:0/4128924706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a680c60b0 0x7f5a6806f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.000+0000 7f5a6759e640 1 -- 172.21.15.1:0/4128924706 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a680c3970 msgr2=0x7f5a681541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.000+0000 7f5a6759e640 1 --2- 172.21.15.1:0/4128924706 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a680c3970 0x7f5a681541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.001+0000 7f5a6759e640 1 -- 172.21.15.1:0/4128924706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a680c60b0 msgr2=0x7f5a6806f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.001+0000 7f5a6759e640 1 --2- 172.21.15.1:0/4128924706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a680c60b0 0x7f5a6806f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.001+0000 7f5a6759e640 1 -- 172.21.15.1:0/4128924706 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a680ad720 con 0x7f5a68152840 2024-03-06T07:30:57.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.001+0000 7f5a6759e640 1 --2- 172.21.15.1:0/4128924706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a68152840 0x7f5a6806fa20 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000cea0 tx=0x7f5a6000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.001+0000 7f5a5f7fe640 1 -- 172.21.15.1:0/4128924706 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a60003b90 con 0x7f5a68152840 2024-03-06T07:30:57.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.001+0000 7f5a5f7fe640 1 -- 172.21.15.1:0/4128924706 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a60003d50 con 0x7f5a68152840 2024-03-06T07:30:57.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.001+0000 7f5a5f7fe640 1 -- 172.21.15.1:0/4128924706 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a60004260 con 0x7f5a68152840 2024-03-06T07:30:57.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.001+0000 7f5a6d0c9640 1 -- 172.21.15.1:0/4128924706 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a680adef0 con 0x7f5a68152840 2024-03-06T07:30:57.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.001+0000 7f5a6d0c9640 1 -- 172.21.15.1:0/4128924706 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a680adc10 con 0x7f5a68152840 2024-03-06T07:30:57.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.002+0000 7f5a6d0c9640 1 -- 172.21.15.1:0/4128924706 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a680686e0 con 0x7f5a68152840 2024-03-06T07:30:57.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.003+0000 7f5a5f7fe640 1 -- 172.21.15.1:0/4128924706 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a60004400 con 0x7f5a68152840 2024-03-06T07:30:57.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.003+0000 7f5a5f7fe640 1 --2- 172.21.15.1:0/4128924706 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a4806b500 0x7f5a4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.003+0000 7f5a5f7fe640 1 -- 172.21.15.1:0/4128924706 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f5a6008ad50 con 0x7f5a68152840 2024-03-06T07:30:57.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.003+0000 7f5a66d9d640 1 --2- 172.21.15.1:0/4128924706 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a4806b500 0x7f5a4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.004+0000 7f5a66d9d640 1 --2- 172.21.15.1:0/4128924706 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a4806b500 0x7f5a4806d9c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f5a58004f60 tx=0x7f5a58007e30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.007+0000 7f5a5f7fe640 1 -- 172.21.15.1:0/4128924706 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a6008e030 con 0x7f5a68152840 2024-03-06T07:30:57.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.226+0000 7f5a6d0c9640 1 -- 172.21.15.1:0/4128924706 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a680c80b0 con 0x7f5a68152840 2024-03-06T07:30:57.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.226+0000 7f5a5f7fe640 1 -- 172.21.15.1:0/4128924706 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7f5a600592f0 con 0x7f5a68152840 2024-03-06T07:30:57.228 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:57.229 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:57.229 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:30:57.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.228+0000 7f5a6d0c9640 1 -- 172.21.15.1:0/4128924706 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a4806b500 msgr2=0x7f5a4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.228+0000 7f5a6d0c9640 1 --2- 172.21.15.1:0/4128924706 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a4806b500 0x7f5a4806d9c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f5a58004f60 tx=0x7f5a58007e30 comp rx=0 tx=0).stop 2024-03-06T07:30:57.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.228+0000 7f5a6d0c9640 1 -- 172.21.15.1:0/4128924706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a68152840 msgr2=0x7f5a6806fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.228+0000 7f5a6d0c9640 1 --2- 172.21.15.1:0/4128924706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a68152840 0x7f5a6806fa20 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000cea0 tx=0x7f5a6000cf90 comp rx=0 tx=0).stop 2024-03-06T07:30:57.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.228+0000 7f5a6d0c9640 1 -- 172.21.15.1:0/4128924706 shutdown_connections 2024-03-06T07:30:57.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.229+0000 7f5a6d0c9640 1 --2- 172.21.15.1:0/4128924706 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5a4806b500 0x7f5a4806d9c0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.229+0000 7f5a6d0c9640 1 --2- 172.21.15.1:0/4128924706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a68152840 0x7f5a6806fa20 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.229+0000 7f5a6d0c9640 1 --2- 172.21.15.1:0/4128924706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a680c60b0 0x7f5a6806f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.229+0000 7f5a6d0c9640 1 --2- 172.21.15.1:0/4128924706 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a680c3970 0x7f5a681541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.229+0000 7f5a6d0c9640 1 -- 172.21.15.1:0/4128924706 >> 172.21.15.1:0/4128924706 conn(0x7f5a680c88f0 msgr2=0x7f5a68074450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.229+0000 7f5a6d0c9640 1 -- 172.21.15.1:0/4128924706 shutdown_connections 2024-03-06T07:30:57.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:57.229+0000 7f5a6d0c9640 1 -- 172.21.15.1:0/4128924706 wait complete. 2024-03-06T07:30:57.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:57.648 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:30:58.649 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:58.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.748+0000 7fd870a9a640 1 Processor -- start 2024-03-06T07:30:58.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.748+0000 7fd870a9a640 1 -- start start 2024-03-06T07:30:58.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.749+0000 7fd870a9a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd86c150520 0x7fd86c1543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.749+0000 7fd870a9a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd86c0c32f0 0x7fd86c1537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.749+0000 7fd870a9a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd86c150900 con 0x7fd86c070c80 2024-03-06T07:30:58.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.749+0000 7fd870a9a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd86c071790 con 0x7fd86c0c32f0 2024-03-06T07:30:58.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.749+0000 7fd870a9a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd86c157050 con 0x7fd86c150520 2024-03-06T07:30:58.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.749+0000 7fd869d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd86c0c32f0 0x7fd86c1537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.749+0000 7fd869d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd86c0c32f0 0x7fd86c1537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58092/0 (socket says 172.21.15.1:58092) 2024-03-06T07:30:58.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.749+0000 7fd869d9b640 1 -- 172.21.15.1:0/3250093662 learned_addr learned my addr 172.21.15.1:0/3250093662 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:58.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.750+0000 7fd86a59c640 1 --2- 172.21.15.1:0/3250093662 >> v2:172.21.15.98:3300/0 conn(0x7fd86c150520 0x7fd86c1543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.750+0000 7fd869d9b640 1 -- 172.21.15.1:0/3250093662 >> v2:172.21.15.98:3300/0 conn(0x7fd86c150520 msgr2=0x7fd86c1543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:58.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.750+0000 7fd869d9b640 1 --2- 172.21.15.1:0/3250093662 >> v2:172.21.15.98:3300/0 conn(0x7fd86c150520 0x7fd86c1543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.750+0000 7fd869d9b640 1 -- 172.21.15.1:0/3250093662 >> v1:172.21.15.98:6789/0 conn(0x7fd86c070c80 legacy=0x7fd86c0c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.750+0000 7fd869d9b640 1 -- 172.21.15.1:0/3250093662 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c152fc0 con 0x7fd86c0c32f0 2024-03-06T07:30:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.751+0000 7fd869d9b640 1 --2- 172.21.15.1:0/3250093662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86c0c32f0 0x7fd86c1537c0 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7fd86c06e6c0 tx=0x7fd86002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c9a961c5f8265eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.751+0000 7fd86959a640 1 -- 172.21.15.1:0/3250093662 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860034070 con 0x7fd86c0c32f0 2024-03-06T07:30:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.751+0000 7fd86959a640 1 -- 172.21.15.1:0/3250093662 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd860030860 con 0x7fd86c0c32f0 2024-03-06T07:30:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.751+0000 7fd86959a640 1 -- 172.21.15.1:0/3250093662 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860030c30 con 0x7fd86c0c32f0 2024-03-06T07:30:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.751+0000 7fd870a9a640 1 -- 172.21.15.1:0/3250093662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86c0c32f0 msgr2=0x7fd86c1537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.751+0000 7fd870a9a640 1 --2- 172.21.15.1:0/3250093662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86c0c32f0 0x7fd86c1537c0 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7fd86c06e6c0 tx=0x7fd86002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.752+0000 7fd870a9a640 1 -- 172.21.15.1:0/3250093662 shutdown_connections 2024-03-06T07:30:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.752+0000 7fd870a9a640 1 --2- 172.21.15.1:0/3250093662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86c0c32f0 0x7fd86c1537c0 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.752+0000 7fd870a9a640 1 --2- 172.21.15.1:0/3250093662 >> v2:172.21.15.98:3300/0 conn(0x7fd86c150520 0x7fd86c1543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.752+0000 7fd870a9a640 1 -- 172.21.15.1:0/3250093662 >> 172.21.15.1:0/3250093662 conn(0x7fd86c0c88f0 msgr2=0x7fd86c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.752+0000 7fd870a9a640 1 -- 172.21.15.1:0/3250093662 shutdown_connections 2024-03-06T07:30:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.752+0000 7fd870a9a640 1 -- 172.21.15.1:0/3250093662 wait complete. 2024-03-06T07:30:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.753+0000 7fd870a9a640 1 Processor -- start 2024-03-06T07:30:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.753+0000 7fd870a9a640 1 -- start start 2024-03-06T07:30:58.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.754+0000 7fd870a9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86c070c80 0x7fd86c0c4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.754+0000 7fd870a9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd86c0c32f0 0x7fd86c1ebce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.754+0000 7fd870a9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd86c150520 0x7fd86c1ec220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.754+0000 7fd870a9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c0c7ec0 con 0x7fd86c070c80 2024-03-06T07:30:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.754+0000 7fd870a9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c157330 con 0x7fd86c150520 2024-03-06T07:30:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.754+0000 7fd870a9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c153d00 con 0x7fd86c0c32f0 2024-03-06T07:30:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.754+0000 7fd86a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86c070c80 0x7fd86c0c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.754+0000 7fd86a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86c070c80 0x7fd86c0c4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58096/0 (socket says 172.21.15.1:58096) 2024-03-06T07:30:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.754+0000 7fd86a59c640 1 -- 172.21.15.1:0/800430411 learned_addr learned my addr 172.21.15.1:0/800430411 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:30:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.755+0000 7fd869d9b640 1 --2- 172.21.15.1:0/800430411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd86c0c32f0 0x7fd86c1ebce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.755+0000 7fd86ad9d640 1 --2- 172.21.15.1:0/800430411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd86c150520 0x7fd86c1ec220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.755+0000 7fd86a59c640 1 -- 172.21.15.1:0/800430411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd86c0c32f0 msgr2=0x7fd86c1ebce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.755+0000 7fd86a59c640 1 --2- 172.21.15.1:0/800430411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd86c0c32f0 0x7fd86c1ebce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.755+0000 7fd86a59c640 1 -- 172.21.15.1:0/800430411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd86c150520 msgr2=0x7fd86c1ec220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.755+0000 7fd86a59c640 1 --2- 172.21.15.1:0/800430411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd86c150520 0x7fd86c1ec220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.755+0000 7fd86a59c640 1 -- 172.21.15.1:0/800430411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c0c3f10 con 0x7fd86c070c80 2024-03-06T07:30:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.755+0000 7fd86a59c640 1 --2- 172.21.15.1:0/800430411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86c070c80 0x7fd86c0c4100 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00d5d0 tx=0x7fd85c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.755+0000 7fd85affd640 1 -- 172.21.15.1:0/800430411 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0125f0 con 0x7fd86c070c80 2024-03-06T07:30:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.755+0000 7fd85affd640 1 -- 172.21.15.1:0/800430411 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85c019070 con 0x7fd86c070c80 2024-03-06T07:30:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.755+0000 7fd870a9a640 1 -- 172.21.15.1:0/800430411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86c0c3c30 con 0x7fd86c070c80 2024-03-06T07:30:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.755+0000 7fd85affd640 1 -- 172.21.15.1:0/800430411 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0129f0 con 0x7fd86c070c80 2024-03-06T07:30:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.755+0000 7fd870a9a640 1 -- 172.21.15.1:0/800430411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86c1ecae0 con 0x7fd86c070c80 2024-03-06T07:30:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.756+0000 7fd870a9a640 1 -- 172.21.15.1:0/800430411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86c0686e0 con 0x7fd86c070c80 2024-03-06T07:30:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.757+0000 7fd85affd640 1 -- 172.21.15.1:0/800430411 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd85c012b90 con 0x7fd86c070c80 2024-03-06T07:30:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.757+0000 7fd85affd640 1 --2- 172.21.15.1:0/800430411 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd84c06b4d0 0x7fd84c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.757+0000 7fd85affd640 1 -- 172.21.15.1:0/800430411 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd85c08a910 con 0x7fd86c070c80 2024-03-06T07:30:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.758+0000 7fd869d9b640 1 --2- 172.21.15.1:0/800430411 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd84c06b4d0 0x7fd84c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.758+0000 7fd869d9b640 1 --2- 172.21.15.1:0/800430411 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd84c06b4d0 0x7fd84c06d990 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fd860036000 tx=0x7fd86003b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.762+0000 7fd85affd640 1 -- 172.21.15.1:0/800430411 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd85c0594f0 con 0x7fd86c070c80 2024-03-06T07:30:58.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.981+0000 7fd870a9a640 1 -- 172.21.15.1:0/800430411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd86c0c7b10 con 0x7fd86c070c80 2024-03-06T07:30:58.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.981+0000 7fd85affd640 1 -- 172.21.15.1:0/800430411 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7fd85c058df0 con 0x7fd86c070c80 2024-03-06T07:30:58.983 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:30:58.984 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:30:58.984 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:30:58.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.983+0000 7fd870a9a640 1 -- 172.21.15.1:0/800430411 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd84c06b4d0 msgr2=0x7fd84c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.983+0000 7fd870a9a640 1 --2- 172.21.15.1:0/800430411 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd84c06b4d0 0x7fd84c06d990 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fd860036000 tx=0x7fd86003b040 comp rx=0 tx=0).stop 2024-03-06T07:30:58.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.983+0000 7fd870a9a640 1 -- 172.21.15.1:0/800430411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86c070c80 msgr2=0x7fd86c0c4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.983+0000 7fd870a9a640 1 --2- 172.21.15.1:0/800430411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86c070c80 0x7fd86c0c4100 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00d5d0 tx=0x7fd85c00da90 comp rx=0 tx=0).stop 2024-03-06T07:30:58.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.983+0000 7fd870a9a640 1 -- 172.21.15.1:0/800430411 shutdown_connections 2024-03-06T07:30:58.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.983+0000 7fd870a9a640 1 --2- 172.21.15.1:0/800430411 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd84c06b4d0 0x7fd84c06d990 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.983+0000 7fd870a9a640 1 --2- 172.21.15.1:0/800430411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd86c150520 0x7fd86c1ec220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.983+0000 7fd870a9a640 1 --2- 172.21.15.1:0/800430411 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd86c0c32f0 0x7fd86c1ebce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.983+0000 7fd870a9a640 1 --2- 172.21.15.1:0/800430411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86c070c80 0x7fd86c0c4100 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.983+0000 7fd870a9a640 1 -- 172.21.15.1:0/800430411 >> 172.21.15.1:0/800430411 conn(0x7fd86c0c88f0 msgr2=0x7fd86c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:58.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.983+0000 7fd870a9a640 1 -- 172.21.15.1:0/800430411 shutdown_connections 2024-03-06T07:30:58.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:30:58.983+0000 7fd870a9a640 1 -- 172.21.15.1:0/800430411 wait complete. 2024-03-06T07:30:59.398 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:30:59.398 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:00.399 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:00.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.500+0000 7f35c05e2640 1 Processor -- start 2024-03-06T07:31:00.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.500+0000 7f35c05e2640 1 -- start start 2024-03-06T07:31:00.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.501+0000 7f35c05e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b8152650 0x7f35b8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.501+0000 7f35c05e2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35b80c6b10 0x7f35b8158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.501+0000 7f35c05e2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f35b80bf9d0 0x7f35b80c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.501+0000 7f35c05e2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b8155800 con 0x7f35b80c6b10 2024-03-06T07:31:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.501+0000 7f35c05e2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b80c2fc0 con 0x7f35b80bf9d0 2024-03-06T07:31:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.501+0000 7f35c05e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b80c6690 con 0x7f35b8152650 2024-03-06T07:31:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.501+0000 7f35bdb7d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35b80c6b10 0x7f35b8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.501+0000 7f35bdb7d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35b80c6b10 0x7f35b8158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58112/0 (socket says 172.21.15.1:58112) 2024-03-06T07:31:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.501+0000 7f35bdb7d640 1 -- 172.21.15.1:0/3950803104 learned_addr learned my addr 172.21.15.1:0/3950803104 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.501+0000 7f35be37e640 1 --2- 172.21.15.1:0/3950803104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b8152650 0x7f35b8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.501+0000 7f35beb7f640 1 --2- 172.21.15.1:0/3950803104 >> v2:172.21.15.98:3300/0 conn(0x7f35b80bf9d0 0x7f35b80c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.502+0000 7f35bdb7d640 1 -- 172.21.15.1:0/3950803104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b8152650 msgr2=0x7f35b8155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.502+0000 7f35bdb7d640 1 --2- 172.21.15.1:0/3950803104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b8152650 0x7f35b8155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.502+0000 7f35bdb7d640 1 -- 172.21.15.1:0/3950803104 >> v2:172.21.15.98:3300/0 conn(0x7f35b80bf9d0 msgr2=0x7f35b80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.502+0000 7f35bdb7d640 1 --2- 172.21.15.1:0/3950803104 >> v2:172.21.15.98:3300/0 conn(0x7f35b80bf9d0 0x7f35b80c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.502+0000 7f35bdb7d640 1 -- 172.21.15.1:0/3950803104 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b80c7280 con 0x7f35b80c6b10 2024-03-06T07:31:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.502+0000 7f35bdb7d640 1 --2- 172.21.15.1:0/3950803104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b80c6b10 0x7f35b8158690 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f35b81597b0 tx=0x7f35a802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68f4734c6dab2d84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.503+0000 7f35bd37c640 1 -- 172.21.15.1:0/3950803104 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a802e070 con 0x7f35b80c6b10 2024-03-06T07:31:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.503+0000 7f35bd37c640 1 -- 172.21.15.1:0/3950803104 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a8037070 con 0x7f35b80c6b10 2024-03-06T07:31:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.503+0000 7f35bd37c640 1 -- 172.21.15.1:0/3950803104 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a8031a00 con 0x7f35b80c6b10 2024-03-06T07:31:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.503+0000 7f35c05e2640 1 -- 172.21.15.1:0/3950803104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b80c6b10 msgr2=0x7f35b8158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.503+0000 7f35c05e2640 1 --2- 172.21.15.1:0/3950803104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b80c6b10 0x7f35b8158690 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f35b81597b0 tx=0x7f35a802b880 comp rx=0 tx=0).stop 2024-03-06T07:31:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.503+0000 7f35c05e2640 1 -- 172.21.15.1:0/3950803104 shutdown_connections 2024-03-06T07:31:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.503+0000 7f35c05e2640 1 --2- 172.21.15.1:0/3950803104 >> v2:172.21.15.98:3300/0 conn(0x7f35b80bf9d0 0x7f35b80c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.503+0000 7f35c05e2640 1 --2- 172.21.15.1:0/3950803104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b80c6b10 0x7f35b8158690 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.503+0000 7f35c05e2640 1 --2- 172.21.15.1:0/3950803104 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b8152650 0x7f35b8155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.503+0000 7f35c05e2640 1 -- 172.21.15.1:0/3950803104 >> 172.21.15.1:0/3950803104 conn(0x7f35b8066ee0 msgr2=0x7f35b80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.503+0000 7f35c05e2640 1 -- 172.21.15.1:0/3950803104 shutdown_connections 2024-03-06T07:31:00.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.503+0000 7f35c05e2640 1 -- 172.21.15.1:0/3950803104 wait complete. 2024-03-06T07:31:00.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.504+0000 7f35c05e2640 1 Processor -- start 2024-03-06T07:31:00.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.504+0000 7f35c05e2640 1 -- start start 2024-03-06T07:31:00.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.505+0000 7f35c05e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b80bf9d0 0x7f35b8152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.505+0000 7f35c05e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b8152650 0x7f35b814e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.505+0000 7f35c05e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8154d70 0x7f35b814c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.505+0000 7f35c05e2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b8155490 con 0x7f35b8154d70 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.505+0000 7f35c05e2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b81551b0 con 0x7f35b80bf9d0 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.505+0000 7f35c05e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b8152a30 con 0x7f35b8152650 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.505+0000 7f35beb7f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8154d70 0x7f35b814c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.505+0000 7f35beb7f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8154d70 0x7f35b814c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58114/0 (socket says 172.21.15.1:58114) 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.505+0000 7f35beb7f640 1 -- 172.21.15.1:0/2180552375 learned_addr learned my addr 172.21.15.1:0/2180552375 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.505+0000 7f35be37e640 1 --2- 172.21.15.1:0/2180552375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b80bf9d0 0x7f35b8152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.505+0000 7f35bdb7d640 1 --2- 172.21.15.1:0/2180552375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b8152650 0x7f35b814e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.505+0000 7f35beb7f640 1 -- 172.21.15.1:0/2180552375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b8152650 msgr2=0x7f35b814e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.505+0000 7f35beb7f640 1 --2- 172.21.15.1:0/2180552375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b8152650 0x7f35b814e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.505+0000 7f35beb7f640 1 -- 172.21.15.1:0/2180552375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b80bf9d0 msgr2=0x7f35b8152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.505+0000 7f35beb7f640 1 --2- 172.21.15.1:0/2180552375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b80bf9d0 0x7f35b8152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.506+0000 7f35beb7f640 1 -- 172.21.15.1:0/2180552375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b8154890 con 0x7f35b8154d70 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.506+0000 7f35be37e640 1 --2- 172.21.15.1:0/2180552375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b80bf9d0 0x7f35b8152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:00.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.506+0000 7f35bdb7d640 1 --2- 172.21.15.1:0/2180552375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b8152650 0x7f35b814e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:00.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.506+0000 7f35beb7f640 1 --2- 172.21.15.1:0/2180552375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8154d70 0x7f35b814c1b0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f35b000ef00 tx=0x7f35b000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.506+0000 7f35aeffd640 1 -- 172.21.15.1:0/2180552375 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b0018020 con 0x7f35b8154d70 2024-03-06T07:31:00.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.506+0000 7f35aeffd640 1 -- 172.21.15.1:0/2180552375 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b000bd20 con 0x7f35b8154d70 2024-03-06T07:31:00.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.506+0000 7f35c05e2640 1 -- 172.21.15.1:0/2180552375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b814db20 con 0x7f35b8154d70 2024-03-06T07:31:00.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.506+0000 7f35aeffd640 1 -- 172.21.15.1:0/2180552375 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b0014640 con 0x7f35b8154d70 2024-03-06T07:31:00.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.506+0000 7f35c05e2640 1 -- 172.21.15.1:0/2180552375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b81ebda0 con 0x7f35b8154d70 2024-03-06T07:31:00.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.507+0000 7f35c05e2640 1 -- 172.21.15.1:0/2180552375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b8154a60 con 0x7f35b8154d70 2024-03-06T07:31:00.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.508+0000 7f35aeffd640 1 -- 172.21.15.1:0/2180552375 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f35b0005d10 con 0x7f35b8154d70 2024-03-06T07:31:00.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.508+0000 7f35aeffd640 1 --2- 172.21.15.1:0/2180552375 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b500 0x7f359c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.508+0000 7f35aeffd640 1 -- 172.21.15.1:0/2180552375 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f35b008a9c0 con 0x7f35b8154d70 2024-03-06T07:31:00.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.509+0000 7f35be37e640 1 --2- 172.21.15.1:0/2180552375 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b500 0x7f359c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.509+0000 7f35be37e640 1 --2- 172.21.15.1:0/2180552375 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b500 0x7f359c06d9c0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009ac0 tx=0x7f35b4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.513+0000 7f35aeffd640 1 -- 172.21.15.1:0/2180552375 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f35b008ce20 con 0x7f35b8154d70 2024-03-06T07:31:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.736+0000 7f35c05e2640 1 -- 172.21.15.1:0/2180552375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35b8067fd0 con 0x7f35b8154d70 2024-03-06T07:31:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.736+0000 7f35aeffd640 1 -- 172.21.15.1:0/2180552375 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7f35b0022070 con 0x7f35b8154d70 2024-03-06T07:31:00.739 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:00.739 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:00.739 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:31:00.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.738+0000 7f35c05e2640 1 -- 172.21.15.1:0/2180552375 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b500 msgr2=0x7f359c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.738+0000 7f35c05e2640 1 --2- 172.21.15.1:0/2180552375 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b500 0x7f359c06d9c0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009ac0 tx=0x7f35b4008040 comp rx=0 tx=0).stop 2024-03-06T07:31:00.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.738+0000 7f35c05e2640 1 -- 172.21.15.1:0/2180552375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8154d70 msgr2=0x7f35b814c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.738+0000 7f35c05e2640 1 --2- 172.21.15.1:0/2180552375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8154d70 0x7f35b814c1b0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f35b000ef00 tx=0x7f35b000b480 comp rx=0 tx=0).stop 2024-03-06T07:31:00.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.738+0000 7f35c05e2640 1 -- 172.21.15.1:0/2180552375 shutdown_connections 2024-03-06T07:31:00.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.738+0000 7f35c05e2640 1 --2- 172.21.15.1:0/2180552375 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f359c06b500 0x7f359c06d9c0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.738+0000 7f35c05e2640 1 --2- 172.21.15.1:0/2180552375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35b8154d70 0x7f35b814c1b0 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.738+0000 7f35c05e2640 1 --2- 172.21.15.1:0/2180552375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f35b8152650 0x7f35b814e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.738+0000 7f35c05e2640 1 --2- 172.21.15.1:0/2180552375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35b80bf9d0 0x7f35b8152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.738+0000 7f35c05e2640 1 -- 172.21.15.1:0/2180552375 >> 172.21.15.1:0/2180552375 conn(0x7f35b8066ee0 msgr2=0x7f35b80c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.738+0000 7f35c05e2640 1 -- 172.21.15.1:0/2180552375 shutdown_connections 2024-03-06T07:31:00.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:00.738+0000 7f35c05e2640 1 -- 172.21.15.1:0/2180552375 wait complete. 2024-03-06T07:31:01.154 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:01.154 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:02.155 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:02.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.255+0000 7ffb54078640 1 Processor -- start 2024-03-06T07:31:02.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.256+0000 7ffb54078640 1 -- start start 2024-03-06T07:31:02.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.257+0000 7ffb54078640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffb4c0c6b10 0x7ffb4c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.257+0000 7ffb54078640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4c0c2fc0 con 0x7ffb4c0bf9d0 2024-03-06T07:31:02.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.257+0000 7ffb54078640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4c158da0 con 0x7ffb4c152650 2024-03-06T07:31:02.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.257+0000 7ffb54078640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb4c155800 con 0x7ffb4c0c6b10 2024-03-06T07:31:02.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.257+0000 7ffb51613640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffb4c0c6b10 0x7ffb4c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.257+0000 7ffb51613640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffb4c0c6b10 0x7ffb4c155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58126/0 (socket says 172.21.15.1:58126) 2024-03-06T07:31:02.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.257+0000 7ffb51613640 1 -- 172.21.15.1:0/1179678021 learned_addr learned my addr 172.21.15.1:0/1179678021 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:02.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.257+0000 7ffb52615640 1 --1- 172.21.15.1:0/1179678021 >> v1:172.21.15.1:6789/0 conn(0x7ffb4c0bf9d0 0x7ffb4c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36918/0 (socket says 172.21.15.1:36918) 2024-03-06T07:31:02.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.257+0000 7ffb51613640 1 -- 172.21.15.1:0/1179678021 >> v1:172.21.15.98:6789/0 conn(0x7ffb4c152650 legacy=0x7ffb4c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.257+0000 7ffb51613640 1 -- 172.21.15.1:0/1179678021 >> v1:172.21.15.1:6789/0 conn(0x7ffb4c0bf9d0 legacy=0x7ffb4c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.257+0000 7ffb51613640 1 -- 172.21.15.1:0/1179678021 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c0c6690 con 0x7ffb4c0c6b10 2024-03-06T07:31:02.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.258+0000 7ffb51613640 1 --2- 172.21.15.1:0/1179678021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c0c6b10 0x7ffb4c155c90 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c009870 tx=0x7ffb3c02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2ab81c1e91986d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.259+0000 7ffb50e12640 1 -- 172.21.15.1:0/1179678021 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c037070 con 0x7ffb4c0c6b10 2024-03-06T07:31:02.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.259+0000 7ffb50e12640 1 -- 172.21.15.1:0/1179678021 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3c004230 con 0x7ffb4c0c6b10 2024-03-06T07:31:02.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.259+0000 7ffb50e12640 1 -- 172.21.15.1:0/1179678021 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c003610 con 0x7ffb4c0c6b10 2024-03-06T07:31:02.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.259+0000 7ffb54078640 1 -- 172.21.15.1:0/1179678021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c0c6b10 msgr2=0x7ffb4c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.259+0000 7ffb54078640 1 --2- 172.21.15.1:0/1179678021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c0c6b10 0x7ffb4c155c90 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c009870 tx=0x7ffb3c02b7e0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.259+0000 7ffb54078640 1 -- 172.21.15.1:0/1179678021 shutdown_connections 2024-03-06T07:31:02.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.259+0000 7ffb54078640 1 --2- 172.21.15.1:0/1179678021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c0c6b10 0x7ffb4c155c90 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.259+0000 7ffb54078640 1 -- 172.21.15.1:0/1179678021 >> 172.21.15.1:0/1179678021 conn(0x7ffb4c066ee0 msgr2=0x7ffb4c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.259+0000 7ffb54078640 1 -- 172.21.15.1:0/1179678021 shutdown_connections 2024-03-06T07:31:02.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.259+0000 7ffb54078640 1 -- 172.21.15.1:0/1179678021 wait complete. 2024-03-06T07:31:02.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.260+0000 7ffb54078640 1 Processor -- start 2024-03-06T07:31:02.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.261+0000 7ffb54078640 1 -- start start 2024-03-06T07:31:02.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.261+0000 7ffb54078640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb4c0bf9d0 0x7ffb4c14ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.261+0000 7ffb54078640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb4c152650 0x7ffb4c14fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.261+0000 7ffb54078640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c158190 0x7ffb4c1e18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.261+0000 7ffb54078640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c1551b0 con 0x7ffb4c158190 2024-03-06T07:31:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.261+0000 7ffb54078640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c0c6fa0 con 0x7ffb4c152650 2024-03-06T07:31:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.261+0000 7ffb54078640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c155490 con 0x7ffb4c0bf9d0 2024-03-06T07:31:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.261+0000 7ffb52615640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c158190 0x7ffb4c1e18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.261+0000 7ffb52615640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c158190 0x7ffb4c1e18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58128/0 (socket says 172.21.15.1:58128) 2024-03-06T07:31:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.261+0000 7ffb52615640 1 -- 172.21.15.1:0/3057126243 learned_addr learned my addr 172.21.15.1:0/3057126243 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.262+0000 7ffb51613640 1 --2- 172.21.15.1:0/3057126243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb4c152650 0x7ffb4c14fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.262+0000 7ffb51e14640 1 --2- 172.21.15.1:0/3057126243 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb4c0bf9d0 0x7ffb4c14ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.262+0000 7ffb52615640 1 -- 172.21.15.1:0/3057126243 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb4c0bf9d0 msgr2=0x7ffb4c14ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.262+0000 7ffb52615640 1 --2- 172.21.15.1:0/3057126243 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb4c0bf9d0 0x7ffb4c14ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.262+0000 7ffb52615640 1 -- 172.21.15.1:0/3057126243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb4c152650 msgr2=0x7ffb4c14fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.262+0000 7ffb52615640 1 --2- 172.21.15.1:0/3057126243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb4c152650 0x7ffb4c14fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.262+0000 7ffb52615640 1 -- 172.21.15.1:0/3057126243 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c1517a0 con 0x7ffb4c158190 2024-03-06T07:31:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.262+0000 7ffb51613640 1 --2- 172.21.15.1:0/3057126243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb4c152650 0x7ffb4c14fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.262+0000 7ffb52615640 1 --2- 172.21.15.1:0/3057126243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c158190 0x7ffb4c1e18a0 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400c8e0 tx=0x7ffb4400cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.262+0000 7ffb427fc640 1 -- 172.21.15.1:0/3057126243 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb44018070 con 0x7ffb4c158190 2024-03-06T07:31:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.262+0000 7ffb427fc640 1 -- 172.21.15.1:0/3057126243 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4400a640 con 0x7ffb4c158190 2024-03-06T07:31:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.262+0000 7ffb54078640 1 -- 172.21.15.1:0/3057126243 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4c1ef1e0 con 0x7ffb4c158190 2024-03-06T07:31:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.262+0000 7ffb427fc640 1 -- 172.21.15.1:0/3057126243 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4400aab0 con 0x7ffb4c158190 2024-03-06T07:31:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.262+0000 7ffb54078640 1 -- 172.21.15.1:0/3057126243 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4c151a20 con 0x7ffb4c158190 2024-03-06T07:31:02.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.263+0000 7ffb54078640 1 -- 172.21.15.1:0/3057126243 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb4c0b06f0 con 0x7ffb4c158190 2024-03-06T07:31:02.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.263+0000 7ffb427fc640 1 -- 172.21.15.1:0/3057126243 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb4401b070 con 0x7ffb4c158190 2024-03-06T07:31:02.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.264+0000 7ffb427fc640 1 --2- 172.21.15.1:0/3057126243 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb3006b4d0 0x7ffb3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.264+0000 7ffb427fc640 1 -- 172.21.15.1:0/3057126243 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ffb44089c40 con 0x7ffb4c158190 2024-03-06T07:31:02.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.264+0000 7ffb51e14640 1 --2- 172.21.15.1:0/3057126243 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb3006b4d0 0x7ffb3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.265+0000 7ffb51e14640 1 --2- 172.21.15.1:0/3057126243 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb3006b4d0 0x7ffb3006d990 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7ffb48000f60 tx=0x7ffb48009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.268+0000 7ffb427fc640 1 -- 172.21.15.1:0/3057126243 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb4c1517a0 con 0x7ffb4c158190 2024-03-06T07:31:02.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.582+0000 7ffb54078640 1 -- 172.21.15.1:0/3057126243 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb4c0c5730 con 0x7ffb4c158190 2024-03-06T07:31:02.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.582+0000 7ffb427fc640 1 -- 172.21.15.1:0/3057126243 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7ffb440588c0 con 0x7ffb4c158190 2024-03-06T07:31:02.585 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:02.585 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:02.585 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:31:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.584+0000 7ffb54078640 1 -- 172.21.15.1:0/3057126243 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb3006b4d0 msgr2=0x7ffb3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.584+0000 7ffb54078640 1 --2- 172.21.15.1:0/3057126243 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb3006b4d0 0x7ffb3006d990 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7ffb48000f60 tx=0x7ffb48009040 comp rx=0 tx=0).stop 2024-03-06T07:31:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.584+0000 7ffb54078640 1 -- 172.21.15.1:0/3057126243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c158190 msgr2=0x7ffb4c1e18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.584+0000 7ffb54078640 1 --2- 172.21.15.1:0/3057126243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c158190 0x7ffb4c1e18a0 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400c8e0 tx=0x7ffb4400cda0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.584+0000 7ffb54078640 1 -- 172.21.15.1:0/3057126243 shutdown_connections 2024-03-06T07:31:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.584+0000 7ffb54078640 1 --2- 172.21.15.1:0/3057126243 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffb3006b4d0 0x7ffb3006d990 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.584+0000 7ffb54078640 1 --2- 172.21.15.1:0/3057126243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4c158190 0x7ffb4c1e18a0 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.584+0000 7ffb54078640 1 --2- 172.21.15.1:0/3057126243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb4c152650 0x7ffb4c14fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.585+0000 7ffb54078640 1 --2- 172.21.15.1:0/3057126243 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb4c0bf9d0 0x7ffb4c14ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.585+0000 7ffb54078640 1 -- 172.21.15.1:0/3057126243 >> 172.21.15.1:0/3057126243 conn(0x7ffb4c066ee0 msgr2=0x7ffb4c0c5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.585+0000 7ffb54078640 1 -- 172.21.15.1:0/3057126243 shutdown_connections 2024-03-06T07:31:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:02.585+0000 7ffb54078640 1 -- 172.21.15.1:0/3057126243 wait complete. 2024-03-06T07:31:02.899 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:02.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:03.000 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:03.000 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:04.001 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:04.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.122+0000 7fdf672c9640 1 Processor -- start 2024-03-06T07:31:04.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.122+0000 7fdf672c9640 1 -- start start 2024-03-06T07:31:04.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.123+0000 7fdf672c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf60072f20 0x7fdf601513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.123+0000 7fdf672c9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf600a6140 con 0x7fdf60150fe0 2024-03-06T07:31:04.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.123+0000 7fdf672c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf60151aa0 con 0x7fdf60071f90 2024-03-06T07:31:04.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.123+0000 7fdf672c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf601526b0 con 0x7fdf60072f20 2024-03-06T07:31:04.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.123+0000 7fdf65065640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdf60150fe0 0x7fdf6006edb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36928/0 (socket says 172.21.15.1:36928) 2024-03-06T07:31:04.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.123+0000 7fdf65065640 1 -- 172.21.15.1:0/3984543444 learned_addr learned my addr 172.21.15.1:0/3984543444 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:31:04.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf64864640 1 --2- 172.21.15.1:0/3984543444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf60072f20 0x7fdf601513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf5ffff640 1 -- 172.21.15.1:0/3984543444 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4063612422 0 0) 0x7fdf600a6140 con 0x7fdf60150fe0 2024-03-06T07:31:04.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf5ffff640 1 -- 172.21.15.1:0/3984543444 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf48003520 con 0x7fdf60150fe0 2024-03-06T07:31:04.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf5ffff640 1 -- 172.21.15.1:0/3984543444 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3372309361 0 0) 0x7fdf48003520 con 0x7fdf60150fe0 2024-03-06T07:31:04.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf5ffff640 1 -- 172.21.15.1:0/3984543444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf60072f20 msgr2=0x7fdf601513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf5ffff640 1 --2- 172.21.15.1:0/3984543444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf60072f20 0x7fdf601513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf5ffff640 1 -- 172.21.15.1:0/3984543444 >> v1:172.21.15.98:6789/0 conn(0x7fdf60071f90 legacy=0x7fdf60072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf5ffff640 1 -- 172.21.15.1:0/3984543444 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf60151d80 con 0x7fdf60150fe0 2024-03-06T07:31:04.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf5ffff640 1 -- 172.21.15.1:0/3984543444 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fdf58001670 con 0x7fdf60150fe0 2024-03-06T07:31:04.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf5ffff640 1 -- 172.21.15.1:0/3984543444 >> v1:172.21.15.1:6789/0 conn(0x7fdf60150fe0 legacy=0x7fdf6006edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf5ffff640 1 --2- 172.21.15.1:0/3984543444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf480039e0 0x7fdf48003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf5ffff640 1 --2- 172.21.15.1:0/3984543444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf48004390 0x7fdf48004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf5ffff640 1 --2- 172.21.15.1:0/3984543444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf60072f20 0x7fdf48005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf5ffff640 1 -- 172.21.15.1:0/3984543444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4800ba70 con 0x7fdf48004390 2024-03-06T07:31:04.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf5ffff640 1 -- 172.21.15.1:0/3984543444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4800bbf0 con 0x7fdf480039e0 2024-03-06T07:31:04.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf5ffff640 1 -- 172.21.15.1:0/3984543444 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4800bdf0 con 0x7fdf60072f20 2024-03-06T07:31:04.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf64864640 1 --2- 172.21.15.1:0/3984543444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf48004390 0x7fdf48004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf65065640 1 --2- 172.21.15.1:0/3984543444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf480039e0 0x7fdf48003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf65866640 1 --2- 172.21.15.1:0/3984543444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf60072f20 0x7fdf48005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf64864640 1 -- 172.21.15.1:0/3984543444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf60072f20 msgr2=0x7fdf48005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf64864640 1 --2- 172.21.15.1:0/3984543444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf60072f20 0x7fdf48005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf64864640 1 -- 172.21.15.1:0/3984543444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf480039e0 msgr2=0x7fdf48003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf64864640 1 --2- 172.21.15.1:0/3984543444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf480039e0 0x7fdf48003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.124+0000 7fdf64864640 1 -- 172.21.15.1:0/3984543444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf48003520 con 0x7fdf48004390 2024-03-06T07:31:04.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.125+0000 7fdf64864640 1 --2- 172.21.15.1:0/3984543444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf48004390 0x7fdf48004860 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7fdf54009ac0 tx=0x7fdf48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ff96f20a1f8c776 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.125+0000 7fdf5ffff640 1 -- 172.21.15.1:0/3984543444 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf54039070 con 0x7fdf48004390 2024-03-06T07:31:04.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.125+0000 7fdf5ffff640 1 -- 172.21.15.1:0/3984543444 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5402c6b0 con 0x7fdf48004390 2024-03-06T07:31:04.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.125+0000 7fdf5ffff640 1 -- 172.21.15.1:0/3984543444 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf54009690 con 0x7fdf48004390 2024-03-06T07:31:04.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.126+0000 7fdf672c9640 1 -- 172.21.15.1:0/3984543444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf48004390 msgr2=0x7fdf48004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.126+0000 7fdf672c9640 1 --2- 172.21.15.1:0/3984543444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf48004390 0x7fdf48004860 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7fdf54009ac0 tx=0x7fdf48005830 comp rx=0 tx=0).stop 2024-03-06T07:31:04.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.126+0000 7fdf65866640 1 -- 172.21.15.1:0/3984543444 reap_dead start 2024-03-06T07:31:04.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.126+0000 7fdf672c9640 1 -- 172.21.15.1:0/3984543444 shutdown_connections 2024-03-06T07:31:04.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.126+0000 7fdf672c9640 1 -- 172.21.15.1:0/3984543444 >> 172.21.15.1:0/3984543444 conn(0x7fdf600c6860 msgr2=0x7fdf600c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.126+0000 7fdf672c9640 1 -- 172.21.15.1:0/3984543444 shutdown_connections 2024-03-06T07:31:04.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.126+0000 7fdf672c9640 1 -- 172.21.15.1:0/3984543444 wait complete. 2024-03-06T07:31:04.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.127+0000 7fdf672c9640 1 Processor -- start 2024-03-06T07:31:04.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.127+0000 7fdf672c9640 1 -- start start 2024-03-06T07:31:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.128+0000 7fdf672c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf48004390 0x7fdf6013ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.128+0000 7fdf672c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf60144550 0x7fdf601398b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.128+0000 7fdf672c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf600a5460 0x7fdf6013aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.128+0000 7fdf672c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6006f4c0 con 0x7fdf48004390 2024-03-06T07:31:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.128+0000 7fdf672c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf60152340 con 0x7fdf60144550 2024-03-06T07:31:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.128+0000 7fdf672c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6006f640 con 0x7fdf600a5460 2024-03-06T07:31:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.128+0000 7fdf65065640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf48004390 0x7fdf6013ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.128+0000 7fdf65065640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf48004390 0x7fdf6013ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58138/0 (socket says 172.21.15.1:58138) 2024-03-06T07:31:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.128+0000 7fdf65065640 1 -- 172.21.15.1:0/3156043992 learned_addr learned my addr 172.21.15.1:0/3156043992 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.128+0000 7fdf64864640 1 --2- 172.21.15.1:0/3156043992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf60144550 0x7fdf601398b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.128+0000 7fdf65866640 1 --2- 172.21.15.1:0/3156043992 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf600a5460 0x7fdf6013aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.128+0000 7fdf65065640 1 -- 172.21.15.1:0/3156043992 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf600a5460 msgr2=0x7fdf6013aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.128+0000 7fdf65065640 1 --2- 172.21.15.1:0/3156043992 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf600a5460 0x7fdf6013aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.128+0000 7fdf65065640 1 -- 172.21.15.1:0/3156043992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf60144550 msgr2=0x7fdf601398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.128+0000 7fdf65065640 1 --2- 172.21.15.1:0/3156043992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf60144550 0x7fdf601398b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.128+0000 7fdf65065640 1 -- 172.21.15.1:0/3156043992 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf60148120 con 0x7fdf48004390 2024-03-06T07:31:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.128+0000 7fdf65866640 1 --2- 172.21.15.1:0/3156043992 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf600a5460 0x7fdf6013aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.129+0000 7fdf65065640 1 --2- 172.21.15.1:0/3156043992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf48004390 0x7fdf6013ba00 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7fdf580072f0 tx=0x7fdf580077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.129+0000 7fdf5dffb640 1 -- 172.21.15.1:0/3156043992 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5801a070 con 0x7fdf48004390 2024-03-06T07:31:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.129+0000 7fdf5dffb640 1 -- 172.21.15.1:0/3156043992 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5800f960 con 0x7fdf48004390 2024-03-06T07:31:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.129+0000 7fdf672c9640 1 -- 172.21.15.1:0/3156043992 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf601483c0 con 0x7fdf48004390 2024-03-06T07:31:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.129+0000 7fdf5dffb640 1 -- 172.21.15.1:0/3156043992 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf58001670 con 0x7fdf48004390 2024-03-06T07:31:04.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.129+0000 7fdf672c9640 1 -- 172.21.15.1:0/3156043992 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6013c880 con 0x7fdf48004390 2024-03-06T07:31:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.130+0000 7fdf672c9640 1 -- 172.21.15.1:0/3156043992 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf60073380 con 0x7fdf48004390 2024-03-06T07:31:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.130+0000 7fdf5dffb640 1 -- 172.21.15.1:0/3156043992 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf5801d070 con 0x7fdf48004390 2024-03-06T07:31:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.131+0000 7fdf5dffb640 1 --2- 172.21.15.1:0/3156043992 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf4c06b500 0x7fdf4c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.131+0000 7fdf5dffb640 1 -- 172.21.15.1:0/3156043992 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fdf58089cf0 con 0x7fdf48004390 2024-03-06T07:31:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.131+0000 7fdf64864640 1 --2- 172.21.15.1:0/3156043992 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf4c06b500 0x7fdf4c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.131+0000 7fdf64864640 1 --2- 172.21.15.1:0/3156043992 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf4c06b500 0x7fdf4c06d9c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fdf54036000 tx=0x7fdf54030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.135+0000 7fdf5dffb640 1 -- 172.21.15.1:0/3156043992 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf5808e070 con 0x7fdf48004390 2024-03-06T07:31:04.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.355+0000 7fdf672c9640 1 -- 172.21.15.1:0/3156043992 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf6006eb90 con 0x7fdf48004390 2024-03-06T07:31:04.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.356+0000 7fdf5dffb640 1 -- 172.21.15.1:0/3156043992 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7fdf58058320 con 0x7fdf48004390 2024-03-06T07:31:04.358 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:04.358 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:04.358 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:31:04.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.358+0000 7fdf672c9640 1 -- 172.21.15.1:0/3156043992 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf4c06b500 msgr2=0x7fdf4c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.358+0000 7fdf672c9640 1 --2- 172.21.15.1:0/3156043992 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf4c06b500 0x7fdf4c06d9c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fdf54036000 tx=0x7fdf54030000 comp rx=0 tx=0).stop 2024-03-06T07:31:04.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.358+0000 7fdf672c9640 1 -- 172.21.15.1:0/3156043992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf48004390 msgr2=0x7fdf6013ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.358+0000 7fdf672c9640 1 --2- 172.21.15.1:0/3156043992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf48004390 0x7fdf6013ba00 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7fdf580072f0 tx=0x7fdf580077b0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.358+0000 7fdf672c9640 1 -- 172.21.15.1:0/3156043992 shutdown_connections 2024-03-06T07:31:04.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.358+0000 7fdf672c9640 1 --2- 172.21.15.1:0/3156043992 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdf4c06b500 0x7fdf4c06d9c0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.358+0000 7fdf672c9640 1 --2- 172.21.15.1:0/3156043992 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdf600a5460 0x7fdf6013aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.358+0000 7fdf672c9640 1 --2- 172.21.15.1:0/3156043992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf60144550 0x7fdf601398b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.358+0000 7fdf672c9640 1 --2- 172.21.15.1:0/3156043992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf48004390 0x7fdf6013ba00 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.358+0000 7fdf672c9640 1 -- 172.21.15.1:0/3156043992 >> 172.21.15.1:0/3156043992 conn(0x7fdf600c6860 msgr2=0x7fdf6014f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.358+0000 7fdf672c9640 1 -- 172.21.15.1:0/3156043992 shutdown_connections 2024-03-06T07:31:04.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:04.358+0000 7fdf672c9640 1 -- 172.21.15.1:0/3156043992 wait complete. 2024-03-06T07:31:04.773 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:04.773 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:05.774 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:05.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.873+0000 7f86bc77a640 1 Processor -- start 2024-03-06T07:31:05.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.873+0000 7f86bc77a640 1 -- start start 2024-03-06T07:31:05.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.874+0000 7f86bc77a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f86b4152650 0x7f86b4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.874+0000 7f86bc77a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86b40c6b10 0x7f86b4158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.874+0000 7f86bc77a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b40c2fc0 con 0x7f86b40bf9d0 2024-03-06T07:31:05.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.874+0000 7f86bc77a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86b4155800 con 0x7f86b4152650 2024-03-06T07:31:05.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.874+0000 7f86bc77a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b40c6690 con 0x7f86b40c6b10 2024-03-06T07:31:05.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.874+0000 7f86ba516640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f86b4152650 0x7f86b4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.874+0000 7f86ba516640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f86b4152650 0x7f86b4155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37874/0 (socket says 172.21.15.1:37874) 2024-03-06T07:31:05.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.874+0000 7f86bad17640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f86b40bf9d0 0x7f86b40c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41908/0 (socket says 172.21.15.1:41908) 2024-03-06T07:31:05.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.874+0000 7f86ba516640 1 -- 172.21.15.1:0/2956214561 learned_addr learned my addr 172.21.15.1:0/2956214561 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:05.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.875+0000 7f86b9d15640 1 --2- 172.21.15.1:0/2956214561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86b40c6b10 0x7f86b4158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.875+0000 7f86ba516640 1 -- 172.21.15.1:0/2956214561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86b40c6b10 msgr2=0x7f86b4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.875+0000 7f86ba516640 1 --2- 172.21.15.1:0/2956214561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86b40c6b10 0x7f86b4158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.875+0000 7f86ba516640 1 -- 172.21.15.1:0/2956214561 >> v1:172.21.15.1:6789/0 conn(0x7f86b40bf9d0 legacy=0x7f86b40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.875+0000 7f86ba516640 1 -- 172.21.15.1:0/2956214561 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b40c7280 con 0x7f86b4152650 2024-03-06T07:31:05.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.875+0000 7f86b9514640 1 -- 172.21.15.1:0/2956214561 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030900780 0 0) 0x7f86b40c2fc0 con 0x7f86b40bf9d0 2024-03-06T07:31:05.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.875+0000 7f86b9d15640 1 --2- 172.21.15.1:0/2956214561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86b40c6b10 0x7f86b4158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.875+0000 7f86ba516640 1 --2- 172.21.15.1:0/2956214561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86b4152650 0x7f86b4155c90 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f86ac009870 tx=0x7f86ac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26f729ff3df04bfa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.876+0000 7f86b9514640 1 -- 172.21.15.1:0/2956214561 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ac035020 con 0x7f86b4152650 2024-03-06T07:31:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.876+0000 7f86b9514640 1 -- 172.21.15.1:0/2956214561 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86ac031640 con 0x7f86b4152650 2024-03-06T07:31:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.876+0000 7f86b9514640 1 -- 172.21.15.1:0/2956214561 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ac02d040 con 0x7f86b4152650 2024-03-06T07:31:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.876+0000 7f86bc77a640 1 -- 172.21.15.1:0/2956214561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86b4152650 msgr2=0x7f86b4155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.876+0000 7f86bc77a640 1 --2- 172.21.15.1:0/2956214561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86b4152650 0x7f86b4155c90 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f86ac009870 tx=0x7f86ac02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.876+0000 7f86bc77a640 1 -- 172.21.15.1:0/2956214561 shutdown_connections 2024-03-06T07:31:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.876+0000 7f86bc77a640 1 --2- 172.21.15.1:0/2956214561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86b40c6b10 0x7f86b4158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.876+0000 7f86bc77a640 1 --2- 172.21.15.1:0/2956214561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86b4152650 0x7f86b4155c90 unknown :-1 s=CLOSED pgs=2482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.876+0000 7f86bc77a640 1 -- 172.21.15.1:0/2956214561 >> 172.21.15.1:0/2956214561 conn(0x7f86b4066ee0 msgr2=0x7f86b40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.876+0000 7f86bc77a640 1 -- 172.21.15.1:0/2956214561 shutdown_connections 2024-03-06T07:31:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.876+0000 7f86bc77a640 1 -- 172.21.15.1:0/2956214561 wait complete. 2024-03-06T07:31:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.877+0000 7f86bc77a640 1 Processor -- start 2024-03-06T07:31:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.877+0000 7f86bc77a640 1 -- start start 2024-03-06T07:31:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86bc77a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86b40bf9d0 0x7f86b4143db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86bc77a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b4152650 0x7f86b40c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86bc77a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86b4159320 0x7f86b41533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86bc77a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b4155490 con 0x7f86b4159320 2024-03-06T07:31:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86bc77a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b41551b0 con 0x7f86b4152650 2024-03-06T07:31:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86bc77a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b4152a30 con 0x7f86b40bf9d0 2024-03-06T07:31:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86bad17640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86b4159320 0x7f86b41533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86bad17640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86b4159320 0x7f86b41533a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37876/0 (socket says 172.21.15.1:37876) 2024-03-06T07:31:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86bad17640 1 -- 172.21.15.1:0/2809814379 learned_addr learned my addr 172.21.15.1:0/2809814379 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86ba516640 1 --2- 172.21.15.1:0/2809814379 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86b40bf9d0 0x7f86b4143db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86bad17640 1 -- 172.21.15.1:0/2809814379 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86b40bf9d0 msgr2=0x7f86b4143db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86b9d15640 1 --2- 172.21.15.1:0/2809814379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b4152650 0x7f86b40c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86bad17640 1 --2- 172.21.15.1:0/2809814379 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86b40bf9d0 0x7f86b4143db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86bad17640 1 -- 172.21.15.1:0/2809814379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b4152650 msgr2=0x7f86b40c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86bad17640 1 --2- 172.21.15.1:0/2809814379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b4152650 0x7f86b40c9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86bad17640 1 -- 172.21.15.1:0/2809814379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b41eb150 con 0x7f86b4159320 2024-03-06T07:31:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86ba516640 1 --2- 172.21.15.1:0/2809814379 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86b40bf9d0 0x7f86b4143db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.878+0000 7f86b9d15640 1 --2- 172.21.15.1:0/2809814379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b4152650 0x7f86b40c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.879+0000 7f86bad17640 1 --2- 172.21.15.1:0/2809814379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86b4159320 0x7f86b41533a0 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f86b000c730 tx=0x7f86b000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.879+0000 7f86aaffd640 1 -- 172.21.15.1:0/2809814379 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b0018070 con 0x7f86b4159320 2024-03-06T07:31:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.879+0000 7f86aaffd640 1 -- 172.21.15.1:0/2809814379 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b000ce40 con 0x7f86b4159320 2024-03-06T07:31:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.879+0000 7f86bc77a640 1 -- 172.21.15.1:0/2809814379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b41eb430 con 0x7f86b4159320 2024-03-06T07:31:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.879+0000 7f86aaffd640 1 -- 172.21.15.1:0/2809814379 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b000a860 con 0x7f86b4159320 2024-03-06T07:31:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.879+0000 7f86bc77a640 1 -- 172.21.15.1:0/2809814379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b41eb9f0 con 0x7f86b4159320 2024-03-06T07:31:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.880+0000 7f86bc77a640 1 -- 172.21.15.1:0/2809814379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b4156a40 con 0x7f86b4159320 2024-03-06T07:31:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.880+0000 7f86aaffd640 1 -- 172.21.15.1:0/2809814379 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86b001b070 con 0x7f86b4159320 2024-03-06T07:31:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.881+0000 7f86aaffd640 1 --2- 172.21.15.1:0/2809814379 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f869806b4d0 0x7f869806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.881+0000 7f86aaffd640 1 -- 172.21.15.1:0/2809814379 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f86b0089a00 con 0x7f86b4159320 2024-03-06T07:31:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.881+0000 7f86ba516640 1 --2- 172.21.15.1:0/2809814379 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f869806b4d0 0x7f869806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.882+0000 7f86ba516640 1 --2- 172.21.15.1:0/2809814379 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f869806b4d0 0x7f869806d990 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f86ac036000 tx=0x7f86ac008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:05.885+0000 7f86aaffd640 1 -- 172.21.15.1:0/2809814379 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86b0058680 con 0x7f86b4159320 2024-03-06T07:31:06.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:06.102+0000 7f86bc77a640 1 -- 172.21.15.1:0/2809814379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86b4067fd0 con 0x7f86b4159320 2024-03-06T07:31:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:06.103+0000 7f86aaffd640 1 -- 172.21.15.1:0/2809814379 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7f86b0057f80 con 0x7f86b4159320 2024-03-06T07:31:06.105 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:06.105 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:31:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:06.104+0000 7f86bc77a640 1 -- 172.21.15.1:0/2809814379 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f869806b4d0 msgr2=0x7f869806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:06.104+0000 7f86bc77a640 1 --2- 172.21.15.1:0/2809814379 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f869806b4d0 0x7f869806d990 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f86ac036000 tx=0x7f86ac008030 comp rx=0 tx=0).stop 2024-03-06T07:31:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:06.104+0000 7f86bc77a640 1 -- 172.21.15.1:0/2809814379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86b4159320 msgr2=0x7f86b41533a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:06.104+0000 7f86bc77a640 1 --2- 172.21.15.1:0/2809814379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86b4159320 0x7f86b41533a0 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f86b000c730 tx=0x7f86b000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:06.104+0000 7f86bc77a640 1 -- 172.21.15.1:0/2809814379 shutdown_connections 2024-03-06T07:31:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:06.104+0000 7f86bc77a640 1 --2- 172.21.15.1:0/2809814379 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f869806b4d0 0x7f869806d990 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:06.104+0000 7f86bc77a640 1 --2- 172.21.15.1:0/2809814379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86b4159320 0x7f86b41533a0 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:06.104+0000 7f86bc77a640 1 --2- 172.21.15.1:0/2809814379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b4152650 0x7f86b40c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:06.104+0000 7f86bc77a640 1 --2- 172.21.15.1:0/2809814379 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86b40bf9d0 0x7f86b4143db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:06.104+0000 7f86bc77a640 1 -- 172.21.15.1:0/2809814379 >> 172.21.15.1:0/2809814379 conn(0x7f86b4066ee0 msgr2=0x7f86b414e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:06.105+0000 7f86bc77a640 1 -- 172.21.15.1:0/2809814379 shutdown_connections 2024-03-06T07:31:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:06.105+0000 7f86bc77a640 1 -- 172.21.15.1:0/2809814379 wait complete. 2024-03-06T07:31:06.519 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:06.519 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:07.520 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:07.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.620+0000 7f41a093e640 1 Processor -- start 2024-03-06T07:31:07.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.620+0000 7f41a093e640 1 -- start start 2024-03-06T07:31:07.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.621+0000 7f41a093e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f419c152650 0x7f419c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.621+0000 7f41a093e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f419c0bf9d0 0x7f419c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.621+0000 7f41a093e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c0c2fc0 con 0x7f419c0c6b10 2024-03-06T07:31:07.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.621+0000 7f41a093e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f419c155800 con 0x7f419c152650 2024-03-06T07:31:07.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.621+0000 7f41a093e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419c158da0 con 0x7f419c0bf9d0 2024-03-06T07:31:07.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.621+0000 7f419a59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f419c152650 0x7f419c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.621+0000 7f419a59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f419c152650 0x7f419c155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37882/0 (socket says 172.21.15.1:37882) 2024-03-06T07:31:07.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.621+0000 7f419a59c640 1 -- 172.21.15.1:0/648099027 learned_addr learned my addr 172.21.15.1:0/648099027 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:07.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.621+0000 7f419ad9d640 1 --2- 172.21.15.1:0/648099027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f419c0bf9d0 0x7f419c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.622+0000 7f419a59c640 1 -- 172.21.15.1:0/648099027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f419c0bf9d0 msgr2=0x7f419c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:07.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.622+0000 7f419a59c640 1 --2- 172.21.15.1:0/648099027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f419c0bf9d0 0x7f419c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.622+0000 7f419a59c640 1 -- 172.21.15.1:0/648099027 >> v1:172.21.15.98:6789/0 conn(0x7f419c0c6b10 legacy=0x7f419c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:07.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.622+0000 7f419a59c640 1 -- 172.21.15.1:0/648099027 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c0c6690 con 0x7f419c152650 2024-03-06T07:31:07.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.622+0000 7f419a59c640 1 --2- 172.21.15.1:0/648099027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f419c152650 0x7f419c155c90 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f418c009870 tx=0x7f418c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e3918b59426b7ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.623+0000 7f4199d9b640 1 -- 172.21.15.1:0/648099027 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c035020 con 0x7f419c152650 2024-03-06T07:31:07.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.623+0000 7f4199d9b640 1 -- 172.21.15.1:0/648099027 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418c031640 con 0x7f419c152650 2024-03-06T07:31:07.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.623+0000 7f4199d9b640 1 -- 172.21.15.1:0/648099027 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c02d040 con 0x7f419c152650 2024-03-06T07:31:07.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.623+0000 7f41a093e640 1 -- 172.21.15.1:0/648099027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f419c152650 msgr2=0x7f419c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.623+0000 7f41a093e640 1 --2- 172.21.15.1:0/648099027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f419c152650 0x7f419c155c90 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f418c009870 tx=0x7f418c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.623+0000 7f41a093e640 1 -- 172.21.15.1:0/648099027 shutdown_connections 2024-03-06T07:31:07.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.623+0000 7f41a093e640 1 --2- 172.21.15.1:0/648099027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f419c0bf9d0 0x7f419c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.623+0000 7f41a093e640 1 --2- 172.21.15.1:0/648099027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f419c152650 0x7f419c155c90 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.623+0000 7f41a093e640 1 -- 172.21.15.1:0/648099027 >> 172.21.15.1:0/648099027 conn(0x7f419c066ee0 msgr2=0x7f419c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.623+0000 7f41a093e640 1 -- 172.21.15.1:0/648099027 shutdown_connections 2024-03-06T07:31:07.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.623+0000 7f41a093e640 1 -- 172.21.15.1:0/648099027 wait complete. 2024-03-06T07:31:07.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.625+0000 7f41a093e640 1 Processor -- start 2024-03-06T07:31:07.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.625+0000 7f41a093e640 1 -- start start 2024-03-06T07:31:07.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.626+0000 7f41a093e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f419c0bf9d0 0x7f419c1eb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.626+0000 7f41a093e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f419c152650 0x7f419c150300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.626+0000 7f41a093e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f419c0c9a10 0x7f419c156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.626+0000 7f41a093e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419c1551b0 con 0x7f419c0c9a10 2024-03-06T07:31:07.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.626+0000 7f41a093e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419c0c6fa0 con 0x7f419c152650 2024-03-06T07:31:07.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.626+0000 7f41a093e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419c155490 con 0x7f419c0bf9d0 2024-03-06T07:31:07.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.626+0000 7f419ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f419c0c9a10 0x7f419c156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.626+0000 7f419ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f419c0c9a10 0x7f419c156df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37892/0 (socket says 172.21.15.1:37892) 2024-03-06T07:31:07.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.626+0000 7f419ad9d640 1 -- 172.21.15.1:0/3546650070 learned_addr learned my addr 172.21.15.1:0/3546650070 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:07.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.627+0000 7f4193fff640 1 --2- 172.21.15.1:0/3546650070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f419c152650 0x7f419c150300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.627+0000 7f419ad9d640 1 -- 172.21.15.1:0/3546650070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f419c0bf9d0 msgr2=0x7f419c1eb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.627+0000 7f419a59c640 1 --2- 172.21.15.1:0/3546650070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f419c0bf9d0 0x7f419c1eb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.627+0000 7f419ad9d640 1 --2- 172.21.15.1:0/3546650070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f419c0bf9d0 0x7f419c1eb270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.627+0000 7f419ad9d640 1 -- 172.21.15.1:0/3546650070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f419c152650 msgr2=0x7f419c150300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.627+0000 7f419ad9d640 1 --2- 172.21.15.1:0/3546650070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f419c152650 0x7f419c150300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.627+0000 7f419ad9d640 1 -- 172.21.15.1:0/3546650070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c0c8970 con 0x7f419c0c9a10 2024-03-06T07:31:07.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.627+0000 7f4193fff640 1 --2- 172.21.15.1:0/3546650070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f419c152650 0x7f419c150300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:07.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.627+0000 7f419ad9d640 1 --2- 172.21.15.1:0/3546650070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f419c0c9a10 0x7f419c156df0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f4194007e60 tx=0x7f419400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.627+0000 7f4192ffd640 1 -- 172.21.15.1:0/3546650070 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4194014420 con 0x7f419c0c9a10 2024-03-06T07:31:07.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.627+0000 7f4192ffd640 1 -- 172.21.15.1:0/3546650070 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4194019070 con 0x7f419c0c9a10 2024-03-06T07:31:07.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.627+0000 7f41a093e640 1 -- 172.21.15.1:0/3546650070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f419c0c8c50 con 0x7f419c0c9a10 2024-03-06T07:31:07.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.627+0000 7f4192ffd640 1 -- 172.21.15.1:0/3546650070 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4194014820 con 0x7f419c0c9a10 2024-03-06T07:31:07.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.627+0000 7f41a093e640 1 -- 172.21.15.1:0/3546650070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f419c0c9210 con 0x7f419c0c9a10 2024-03-06T07:31:07.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.628+0000 7f41a093e640 1 -- 172.21.15.1:0/3546650070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f419c067fd0 con 0x7f419c0c9a10 2024-03-06T07:31:07.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.628+0000 7f4192ffd640 1 -- 172.21.15.1:0/3546650070 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f41940149c0 con 0x7f419c0c9a10 2024-03-06T07:31:07.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.629+0000 7f4192ffd640 1 --2- 172.21.15.1:0/3546650070 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f417c06b4d0 0x7f417c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.629+0000 7f4192ffd640 1 -- 172.21.15.1:0/3546650070 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f419408ac60 con 0x7f419c0c9a10 2024-03-06T07:31:07.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.629+0000 7f419a59c640 1 --2- 172.21.15.1:0/3546650070 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f417c06b4d0 0x7f417c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.629+0000 7f419a59c640 1 --2- 172.21.15.1:0/3546650070 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f417c06b4d0 0x7f417c06d990 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f418c036000 tx=0x7f418c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.633+0000 7f4192ffd640 1 -- 172.21.15.1:0/3546650070 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41940598e0 con 0x7f419c0c9a10 2024-03-06T07:31:07.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.852+0000 7f41a093e640 1 -- 172.21.15.1:0/3546650070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f419c0b06f0 con 0x7f419c0c9a10 2024-03-06T07:31:07.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.853+0000 7f4192ffd640 1 -- 172.21.15.1:0/3546650070 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7f4194016e40 con 0x7f419c0c9a10 2024-03-06T07:31:07.855 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:07.855 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:07.855 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:31:07.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.855+0000 7f41a093e640 1 -- 172.21.15.1:0/3546650070 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f417c06b4d0 msgr2=0x7f417c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.855+0000 7f41a093e640 1 --2- 172.21.15.1:0/3546650070 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f417c06b4d0 0x7f417c06d990 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f418c036000 tx=0x7f418c008030 comp rx=0 tx=0).stop 2024-03-06T07:31:07.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.855+0000 7f41a093e640 1 -- 172.21.15.1:0/3546650070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f419c0c9a10 msgr2=0x7f419c156df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.855+0000 7f41a093e640 1 --2- 172.21.15.1:0/3546650070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f419c0c9a10 0x7f419c156df0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f4194007e60 tx=0x7f419400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.855+0000 7f41a093e640 1 -- 172.21.15.1:0/3546650070 shutdown_connections 2024-03-06T07:31:07.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.855+0000 7f41a093e640 1 --2- 172.21.15.1:0/3546650070 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f417c06b4d0 0x7f417c06d990 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.855+0000 7f41a093e640 1 --2- 172.21.15.1:0/3546650070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f419c0c9a10 0x7f419c156df0 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.855+0000 7f41a093e640 1 --2- 172.21.15.1:0/3546650070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f419c152650 0x7f419c150300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.855+0000 7f41a093e640 1 --2- 172.21.15.1:0/3546650070 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f419c0bf9d0 0x7f419c1eb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.855+0000 7f41a093e640 1 -- 172.21.15.1:0/3546650070 >> 172.21.15.1:0/3546650070 conn(0x7f419c066ee0 msgr2=0x7f419c0c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.855+0000 7f41a093e640 1 -- 172.21.15.1:0/3546650070 shutdown_connections 2024-03-06T07:31:07.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:07.855+0000 7f41a093e640 1 -- 172.21.15.1:0/3546650070 wait complete. 2024-03-06T07:31:08.270 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:08.270 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:09.271 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:09.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.371+0000 7f7924688640 1 Processor -- start 2024-03-06T07:31:09.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.371+0000 7f7924688640 1 -- start start 2024-03-06T07:31:09.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.372+0000 7f7924688640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f791c0bf9d0 0x7f791c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.372+0000 7f7924688640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791c0c2fc0 con 0x7f791c152650 2024-03-06T07:31:09.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.372+0000 7f7924688640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791c158da0 con 0x7f791c0c6b10 2024-03-06T07:31:09.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.372+0000 7f7924688640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f791c155800 con 0x7f791c0bf9d0 2024-03-06T07:31:09.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.372+0000 7f7922424640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f791c152650 0x7f791c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41910/0 (socket says 172.21.15.1:41910) 2024-03-06T07:31:09.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.372+0000 7f7922424640 1 -- 172.21.15.1:0/1527265160 learned_addr learned my addr 172.21.15.1:0/1527265160 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:31:09.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.372+0000 7f7922c25640 1 --2- 172.21.15.1:0/1527265160 >> v2:172.21.15.98:3300/0 conn(0x7f791c0bf9d0 0x7f791c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.372+0000 7f7921422640 1 -- 172.21.15.1:0/1527265160 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953174678 0 0) 0x7f791c0c2fc0 con 0x7f791c152650 2024-03-06T07:31:09.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.372+0000 7f7921422640 1 -- 172.21.15.1:0/1527265160 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7908003520 con 0x7f791c152650 2024-03-06T07:31:09.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921422640 1 -- 172.21.15.1:0/1527265160 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 879180780 0 0) 0x7f7908003520 con 0x7f791c152650 2024-03-06T07:31:09.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921422640 1 -- 172.21.15.1:0/1527265160 >> v2:172.21.15.98:3300/0 conn(0x7f791c0bf9d0 msgr2=0x7f791c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921422640 1 --2- 172.21.15.1:0/1527265160 >> v2:172.21.15.98:3300/0 conn(0x7f791c0bf9d0 0x7f791c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921422640 1 -- 172.21.15.1:0/1527265160 >> v1:172.21.15.98:6789/0 conn(0x7f791c0c6b10 legacy=0x7f791c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921422640 1 -- 172.21.15.1:0/1527265160 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c0c6690 con 0x7f791c152650 2024-03-06T07:31:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921422640 1 -- 172.21.15.1:0/1527265160 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f7918001670 con 0x7f791c152650 2024-03-06T07:31:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921422640 1 -- 172.21.15.1:0/1527265160 >> v1:172.21.15.1:6789/0 conn(0x7f791c152650 legacy=0x7f791c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921422640 1 --2- 172.21.15.1:0/1527265160 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79080039b0 0x7f7908003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921422640 1 --2- 172.21.15.1:0/1527265160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7908004800 0x7f7908004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921422640 1 --2- 172.21.15.1:0/1527265160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79080051c0 0x7f790800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921422640 1 -- 172.21.15.1:0/1527265160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791c155800 con 0x7f7908004800 2024-03-06T07:31:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921422640 1 -- 172.21.15.1:0/1527265160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790800c160 con 0x7f79080051c0 2024-03-06T07:31:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921422640 1 -- 172.21.15.1:0/1527265160 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790800c360 con 0x7f79080039b0 2024-03-06T07:31:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921c23640 1 --2- 172.21.15.1:0/1527265160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7908004800 0x7f7908004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7922424640 1 --2- 172.21.15.1:0/1527265160 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79080039b0 0x7f7908003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7922c25640 1 --2- 172.21.15.1:0/1527265160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79080051c0 0x7f790800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921c23640 1 -- 172.21.15.1:0/1527265160 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79080039b0 msgr2=0x7f7908003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921c23640 1 --2- 172.21.15.1:0/1527265160 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79080039b0 0x7f7908003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921c23640 1 -- 172.21.15.1:0/1527265160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79080051c0 msgr2=0x7f790800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921c23640 1 --2- 172.21.15.1:0/1527265160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79080051c0 0x7f790800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7921c23640 1 -- 172.21.15.1:0/1527265160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7908003520 con 0x7f7908004800 2024-03-06T07:31:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.373+0000 7f7922c25640 1 -- 172.21.15.1:0/1527265160 reap_dead start 2024-03-06T07:31:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.374+0000 7f7921c23640 1 --2- 172.21.15.1:0/1527265160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7908004800 0x7f7908004c80 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f790c001750 tx=0x7f7908005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ef365ad7f90b03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.374+0000 7f7921422640 1 -- 172.21.15.1:0/1527265160 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790c001370 con 0x7f7908004800 2024-03-06T07:31:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.374+0000 7f7921422640 1 -- 172.21.15.1:0/1527265160 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790c0052d0 con 0x7f7908004800 2024-03-06T07:31:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.374+0000 7f7921422640 1 -- 172.21.15.1:0/1527265160 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790c003830 con 0x7f7908004800 2024-03-06T07:31:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.375+0000 7f7924688640 1 -- 172.21.15.1:0/1527265160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7908004800 msgr2=0x7f7908004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.375+0000 7f7924688640 1 --2- 172.21.15.1:0/1527265160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7908004800 0x7f7908004c80 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f790c001750 tx=0x7f7908005830 comp rx=0 tx=0).stop 2024-03-06T07:31:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.375+0000 7f7924688640 1 -- 172.21.15.1:0/1527265160 shutdown_connections 2024-03-06T07:31:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.375+0000 7f7924688640 1 --2- 172.21.15.1:0/1527265160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7908004800 0x7f7908004c80 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.375+0000 7f7924688640 1 -- 172.21.15.1:0/1527265160 >> 172.21.15.1:0/1527265160 conn(0x7f791c066ee0 msgr2=0x7f791c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.375+0000 7f7924688640 1 -- 172.21.15.1:0/1527265160 shutdown_connections 2024-03-06T07:31:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.375+0000 7f7924688640 1 -- 172.21.15.1:0/1527265160 wait complete. 2024-03-06T07:31:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.376+0000 7f7924688640 1 Processor -- start 2024-03-06T07:31:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.376+0000 7f7924688640 1 -- start start 2024-03-06T07:31:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.376+0000 7f7924688640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7908004800 0x7f791c1ee580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.376+0000 7f7924688640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f791c151760 0x7f791c14a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.376+0000 7f7924688640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f791c14b710 0x7f791c14aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.376+0000 7f7924688640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791c1551b0 con 0x7f791c151760 2024-03-06T07:31:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.376+0000 7f7924688640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791c0c6fa0 con 0x7f791c14b710 2024-03-06T07:31:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.376+0000 7f7924688640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791c155490 con 0x7f7908004800 2024-03-06T07:31:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.377+0000 7f7921c23640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f791c151760 0x7f791c14a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.377+0000 7f7921c23640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f791c151760 0x7f791c14a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37912/0 (socket says 172.21.15.1:37912) 2024-03-06T07:31:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.377+0000 7f7921c23640 1 -- 172.21.15.1:0/1691869938 learned_addr learned my addr 172.21.15.1:0/1691869938 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.377+0000 7f7922424640 1 --2- 172.21.15.1:0/1691869938 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7908004800 0x7f791c1ee580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.377+0000 7f7922c25640 1 --2- 172.21.15.1:0/1691869938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f791c14b710 0x7f791c14aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.377+0000 7f7921c23640 1 -- 172.21.15.1:0/1691869938 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7908004800 msgr2=0x7f791c1ee580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.377+0000 7f7921c23640 1 --2- 172.21.15.1:0/1691869938 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7908004800 0x7f791c1ee580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.377+0000 7f7921c23640 1 -- 172.21.15.1:0/1691869938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f791c14b710 msgr2=0x7f791c14aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.377+0000 7f7921c23640 1 --2- 172.21.15.1:0/1691869938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f791c14b710 0x7f791c14aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.377+0000 7f7921c23640 1 -- 172.21.15.1:0/1691869938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c157150 con 0x7f791c151760 2024-03-06T07:31:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.377+0000 7f7921c23640 1 --2- 172.21.15.1:0/1691869938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f791c151760 0x7f791c14a500 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7f790c005040 tx=0x7f790c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.377+0000 7f7912ffd640 1 -- 172.21.15.1:0/1691869938 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790c041070 con 0x7f791c151760 2024-03-06T07:31:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.377+0000 7f7912ffd640 1 -- 172.21.15.1:0/1691869938 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790c005480 con 0x7f791c151760 2024-03-06T07:31:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.377+0000 7f7924688640 1 -- 172.21.15.1:0/1691869938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f791c1eeec0 con 0x7f791c151760 2024-03-06T07:31:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.377+0000 7f7912ffd640 1 -- 172.21.15.1:0/1691869938 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790c007a30 con 0x7f791c151760 2024-03-06T07:31:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.377+0000 7f7924688640 1 -- 172.21.15.1:0/1691869938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f791c14af80 con 0x7f791c151760 2024-03-06T07:31:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.378+0000 7f7924688640 1 -- 172.21.15.1:0/1691869938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f791c067fd0 con 0x7f791c151760 2024-03-06T07:31:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.379+0000 7f7912ffd640 1 -- 172.21.15.1:0/1691869938 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f790c044070 con 0x7f791c151760 2024-03-06T07:31:09.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.379+0000 7f7912ffd640 1 --2- 172.21.15.1:0/1691869938 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f790006b4d0 0x7f790006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.379+0000 7f7912ffd640 1 -- 172.21.15.1:0/1691869938 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f790c0abeb0 con 0x7f791c151760 2024-03-06T07:31:09.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.379+0000 7f7922424640 1 --2- 172.21.15.1:0/1691869938 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f790006b4d0 0x7f790006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.380+0000 7f7922424640 1 --2- 172.21.15.1:0/1691869938 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f790006b4d0 0x7f790006d990 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f791c0c0930 tx=0x7f79180078f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.383+0000 7f7912ffd640 1 -- 172.21.15.1:0/1691869938 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f790c07a450 con 0x7f791c151760 2024-03-06T07:31:09.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.657+0000 7f7924688640 1 -- 172.21.15.1:0/1691869938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f791c0b06f0 con 0x7f791c151760 2024-03-06T07:31:09.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.657+0000 7f7912ffd640 1 -- 172.21.15.1:0/1691869938 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7f790c07e4f0 con 0x7f791c151760 2024-03-06T07:31:09.659 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:09.659 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:09.659 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:31:09.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.659+0000 7f7924688640 1 -- 172.21.15.1:0/1691869938 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f790006b4d0 msgr2=0x7f790006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.659+0000 7f7924688640 1 --2- 172.21.15.1:0/1691869938 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f790006b4d0 0x7f790006d990 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f791c0c0930 tx=0x7f79180078f0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.659+0000 7f7924688640 1 -- 172.21.15.1:0/1691869938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f791c151760 msgr2=0x7f791c14a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.659+0000 7f7924688640 1 --2- 172.21.15.1:0/1691869938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f791c151760 0x7f791c14a500 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7f790c005040 tx=0x7f790c001120 comp rx=0 tx=0).stop 2024-03-06T07:31:09.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.659+0000 7f7924688640 1 -- 172.21.15.1:0/1691869938 shutdown_connections 2024-03-06T07:31:09.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.659+0000 7f7924688640 1 --2- 172.21.15.1:0/1691869938 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f790006b4d0 0x7f790006d990 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.659+0000 7f7924688640 1 --2- 172.21.15.1:0/1691869938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f791c14b710 0x7f791c14aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.659+0000 7f7924688640 1 --2- 172.21.15.1:0/1691869938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f791c151760 0x7f791c14a500 unknown :-1 s=CLOSED pgs=2488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.659+0000 7f7924688640 1 --2- 172.21.15.1:0/1691869938 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7908004800 0x7f791c1ee580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.659+0000 7f7924688640 1 -- 172.21.15.1:0/1691869938 >> 172.21.15.1:0/1691869938 conn(0x7f791c066ee0 msgr2=0x7f791c0c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:09.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.659+0000 7f7924688640 1 -- 172.21.15.1:0/1691869938 shutdown_connections 2024-03-06T07:31:09.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:09.659+0000 7f7924688640 1 -- 172.21.15.1:0/1691869938 wait complete. 2024-03-06T07:31:10.073 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:10.074 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:11.074 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:11.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.175+0000 7fd0caf9a640 1 Processor -- start 2024-03-06T07:31:11.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.175+0000 7fd0caf9a640 1 -- start start 2024-03-06T07:31:11.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.176+0000 7fd0caf9a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd0c406f6f0 0x7fd0c4153220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.176+0000 7fd0caf9a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd0c40c15f0 0x7fd0c4153760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.176+0000 7fd0caf9a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c406c540 con 0x7fd0c4150800 2024-03-06T07:31:11.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.176+0000 7fd0caf9a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0c40acd40 con 0x7fd0c40c15f0 2024-03-06T07:31:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.176+0000 7fd0caf9a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0c406c870 con 0x7fd0c406f6f0 2024-03-06T07:31:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.176+0000 7fd0c8d36640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd0c4150800 0x7fd0c40727e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41924/0 (socket says 172.21.15.1:41924) 2024-03-06T07:31:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.176+0000 7fd0c8d36640 1 -- 172.21.15.1:0/363716813 learned_addr learned my addr 172.21.15.1:0/363716813 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:31:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.176+0000 7fd0c9537640 1 --2- 172.21.15.1:0/363716813 >> v2:172.21.15.1:3300/0 conn(0x7fd0c40c15f0 0x7fd0c4153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.176+0000 7fd0c9537640 1 -- 172.21.15.1:0/363716813 >> v2:172.21.15.98:3300/0 conn(0x7fd0c406f6f0 msgr2=0x7fd0c4153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.176+0000 7fd0c3fff640 1 --2- 172.21.15.1:0/363716813 >> v2:172.21.15.98:3300/0 conn(0x7fd0c406f6f0 0x7fd0c4153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.176+0000 7fd0c9537640 1 --2- 172.21.15.1:0/363716813 >> v2:172.21.15.98:3300/0 conn(0x7fd0c406f6f0 0x7fd0c4153220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.176+0000 7fd0c37fe640 1 -- 172.21.15.1:0/363716813 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384525082 0 0) 0x7fd0c406c540 con 0x7fd0c4150800 2024-03-06T07:31:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.176+0000 7fd0c9537640 1 -- 172.21.15.1:0/363716813 >> v1:172.21.15.1:6789/0 conn(0x7fd0c4150800 legacy=0x7fd0c40727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.176+0000 7fd0c9537640 1 -- 172.21.15.1:0/363716813 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c40aca60 con 0x7fd0c40c15f0 2024-03-06T07:31:11.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.177+0000 7fd0c9537640 1 --2- 172.21.15.1:0/363716813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0c40c15f0 0x7fd0c4153760 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fd0c406cb30 tx=0x7fd0bc02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42ba58d6a580967 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.177+0000 7fd0c37fe640 1 -- 172.21.15.1:0/363716813 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc038070 con 0x7fd0c40c15f0 2024-03-06T07:31:11.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.177+0000 7fd0c37fe640 1 -- 172.21.15.1:0/363716813 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0bc031710 con 0x7fd0c40c15f0 2024-03-06T07:31:11.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.177+0000 7fd0c37fe640 1 -- 172.21.15.1:0/363716813 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc031ae0 con 0x7fd0c40c15f0 2024-03-06T07:31:11.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.178+0000 7fd0caf9a640 1 -- 172.21.15.1:0/363716813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0c40c15f0 msgr2=0x7fd0c4153760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.178+0000 7fd0caf9a640 1 --2- 172.21.15.1:0/363716813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0c40c15f0 0x7fd0c4153760 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fd0c406cb30 tx=0x7fd0bc02b7b0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.178+0000 7fd0caf9a640 1 -- 172.21.15.1:0/363716813 shutdown_connections 2024-03-06T07:31:11.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.178+0000 7fd0caf9a640 1 --2- 172.21.15.1:0/363716813 >> v2:172.21.15.98:3300/0 conn(0x7fd0c406f6f0 0x7fd0c4153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.178+0000 7fd0caf9a640 1 --2- 172.21.15.1:0/363716813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0c40c15f0 0x7fd0c4153760 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.178+0000 7fd0caf9a640 1 -- 172.21.15.1:0/363716813 >> 172.21.15.1:0/363716813 conn(0x7fd0c4071cf0 msgr2=0x7fd0c40720d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.178+0000 7fd0caf9a640 1 -- 172.21.15.1:0/363716813 shutdown_connections 2024-03-06T07:31:11.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.178+0000 7fd0caf9a640 1 -- 172.21.15.1:0/363716813 wait complete. 2024-03-06T07:31:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.179+0000 7fd0caf9a640 1 Processor -- start 2024-03-06T07:31:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.179+0000 7fd0caf9a640 1 -- start start 2024-03-06T07:31:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.180+0000 7fd0caf9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0c40c15f0 0x7fd0c41e5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.180+0000 7fd0caf9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0c4150800 0x7fd0c40c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.180+0000 7fd0caf9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0c41433b0 0x7fd0c41eba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.180+0000 7fd0caf9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c4153ca0 con 0x7fd0c40c15f0 2024-03-06T07:31:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.180+0000 7fd0caf9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c40ac810 con 0x7fd0c4150800 2024-03-06T07:31:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.180+0000 7fd0caf9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c4153e20 con 0x7fd0c41433b0 2024-03-06T07:31:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.180+0000 7fd0c8d36640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0c40c15f0 0x7fd0c41e5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.180+0000 7fd0c8d36640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0c40c15f0 0x7fd0c41e5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37924/0 (socket says 172.21.15.1:37924) 2024-03-06T07:31:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.180+0000 7fd0c8d36640 1 -- 172.21.15.1:0/3147706657 learned_addr learned my addr 172.21.15.1:0/3147706657 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.180+0000 7fd0c8d36640 1 -- 172.21.15.1:0/3147706657 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0c41433b0 msgr2=0x7fd0c41eba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.180+0000 7fd0c8d36640 1 --2- 172.21.15.1:0/3147706657 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0c41433b0 0x7fd0c41eba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.180+0000 7fd0c8d36640 1 -- 172.21.15.1:0/3147706657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0c4150800 msgr2=0x7fd0c40c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.180+0000 7fd0c3fff640 1 --2- 172.21.15.1:0/3147706657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0c4150800 0x7fd0c40c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.180+0000 7fd0c8d36640 1 --2- 172.21.15.1:0/3147706657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0c4150800 0x7fd0c40c5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.180+0000 7fd0c8d36640 1 -- 172.21.15.1:0/3147706657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c41469f0 con 0x7fd0c40c15f0 2024-03-06T07:31:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.180+0000 7fd0c3fff640 1 --2- 172.21.15.1:0/3147706657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0c4150800 0x7fd0c40c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.180+0000 7fd0c8d36640 1 --2- 172.21.15.1:0/3147706657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0c40c15f0 0x7fd0c41e5940 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400c8c0 tx=0x7fd0b400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.181+0000 7fd0c17fa640 1 -- 172.21.15.1:0/3147706657 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b4018070 con 0x7fd0c40c15f0 2024-03-06T07:31:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.181+0000 7fd0c17fa640 1 -- 172.21.15.1:0/3147706657 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b400a640 con 0x7fd0c40c15f0 2024-03-06T07:31:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.181+0000 7fd0c17fa640 1 -- 172.21.15.1:0/3147706657 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b400aa30 con 0x7fd0c40c15f0 2024-03-06T07:31:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.181+0000 7fd0caf9a640 1 -- 172.21.15.1:0/3147706657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c4146cd0 con 0x7fd0c40c15f0 2024-03-06T07:31:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.181+0000 7fd0caf9a640 1 -- 172.21.15.1:0/3147706657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c414aaa0 con 0x7fd0c40c15f0 2024-03-06T07:31:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.182+0000 7fd0caf9a640 1 -- 172.21.15.1:0/3147706657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c40686e0 con 0x7fd0c40c15f0 2024-03-06T07:31:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.182+0000 7fd0c17fa640 1 -- 172.21.15.1:0/3147706657 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0b401b070 con 0x7fd0c40c15f0 2024-03-06T07:31:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.182+0000 7fd0c17fa640 1 --2- 172.21.15.1:0/3147706657 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd0a406b500 0x7fd0a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.183+0000 7fd0c17fa640 1 -- 172.21.15.1:0/3147706657 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd0b4089b00 con 0x7fd0c40c15f0 2024-03-06T07:31:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.183+0000 7fd0c3fff640 1 --2- 172.21.15.1:0/3147706657 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd0a406b500 0x7fd0a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.183+0000 7fd0c3fff640 1 --2- 172.21.15.1:0/3147706657 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd0a406b500 0x7fd0a406d9c0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fd0c414a9d0 tx=0x7fd0ac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.187+0000 7fd0c17fa640 1 -- 172.21.15.1:0/3147706657 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0b4058780 con 0x7fd0c40c15f0 2024-03-06T07:31:11.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.410+0000 7fd0caf9a640 1 -- 172.21.15.1:0/3147706657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0c40671d0 con 0x7fd0c40c15f0 2024-03-06T07:31:11.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.410+0000 7fd0c17fa640 1 -- 172.21.15.1:0/3147706657 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7fd0b4058080 con 0x7fd0c40c15f0 2024-03-06T07:31:11.412 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:11.413 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:11.413 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:31:11.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.413+0000 7fd0caf9a640 1 -- 172.21.15.1:0/3147706657 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd0a406b500 msgr2=0x7fd0a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.413+0000 7fd0caf9a640 1 --2- 172.21.15.1:0/3147706657 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd0a406b500 0x7fd0a406d9c0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fd0c414a9d0 tx=0x7fd0ac008040 comp rx=0 tx=0).stop 2024-03-06T07:31:11.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.413+0000 7fd0caf9a640 1 -- 172.21.15.1:0/3147706657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0c40c15f0 msgr2=0x7fd0c41e5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.413+0000 7fd0caf9a640 1 --2- 172.21.15.1:0/3147706657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0c40c15f0 0x7fd0c41e5940 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400c8c0 tx=0x7fd0b400cd80 comp rx=0 tx=0).stop 2024-03-06T07:31:11.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.413+0000 7fd0caf9a640 1 -- 172.21.15.1:0/3147706657 shutdown_connections 2024-03-06T07:31:11.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.413+0000 7fd0caf9a640 1 --2- 172.21.15.1:0/3147706657 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd0a406b500 0x7fd0a406d9c0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.413+0000 7fd0caf9a640 1 --2- 172.21.15.1:0/3147706657 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0c41433b0 0x7fd0c41eba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.413+0000 7fd0caf9a640 1 --2- 172.21.15.1:0/3147706657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0c4150800 0x7fd0c40c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.413+0000 7fd0caf9a640 1 --2- 172.21.15.1:0/3147706657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0c40c15f0 0x7fd0c41e5940 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.413+0000 7fd0caf9a640 1 -- 172.21.15.1:0/3147706657 >> 172.21.15.1:0/3147706657 conn(0x7fd0c4071cf0 msgr2=0x7fd0c40c2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.413+0000 7fd0caf9a640 1 -- 172.21.15.1:0/3147706657 shutdown_connections 2024-03-06T07:31:11.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:11.413+0000 7fd0caf9a640 1 -- 172.21.15.1:0/3147706657 wait complete. 2024-03-06T07:31:11.830 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:11.831 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:12.832 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:12.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.932+0000 7f99ba2b5640 1 Processor -- start 2024-03-06T07:31:12.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.932+0000 7f99ba2b5640 1 -- start start 2024-03-06T07:31:12.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.932+0000 7f99ba2b5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99b4152650 0x7f99b4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.932+0000 7f99ba2b5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f99b40bf9d0 0x7f99b40c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.932+0000 7f99ba2b5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b40c2fc0 con 0x7f99b40c6b10 2024-03-06T07:31:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.932+0000 7f99ba2b5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b4155800 con 0x7f99b40bf9d0 2024-03-06T07:31:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.932+0000 7f99ba2b5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b4158da0 con 0x7f99b4152650 2024-03-06T07:31:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b37fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f99b40c6b10 0x7f99b4158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41940/0 (socket says 172.21.15.1:41940) 2024-03-06T07:31:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b37fe640 1 -- 172.21.15.1:0/3420396542 learned_addr learned my addr 172.21.15.1:0/3420396542 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:31:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b8852640 1 --2- 172.21.15.1:0/3420396542 >> v2:172.21.15.98:3300/0 conn(0x7f99b40bf9d0 0x7f99b40c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b3fff640 1 --2- 172.21.15.1:0/3420396542 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99b4152650 0x7f99b4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3420396542 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114337237 0 0) 0x7f99b40c2fc0 con 0x7f99b40c6b10 2024-03-06T07:31:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3420396542 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f999c003520 con 0x7f99b40c6b10 2024-03-06T07:31:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3420396542 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3748279438 0 0) 0x7f999c003520 con 0x7f99b40c6b10 2024-03-06T07:31:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3420396542 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99b4152650 msgr2=0x7f99b4155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b2ffd640 1 --2- 172.21.15.1:0/3420396542 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99b4152650 0x7f99b4155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3420396542 >> v2:172.21.15.98:3300/0 conn(0x7f99b40bf9d0 msgr2=0x7f99b40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:12.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b2ffd640 1 --2- 172.21.15.1:0/3420396542 >> v2:172.21.15.98:3300/0 conn(0x7f99b40bf9d0 0x7f99b40c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3420396542 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b40c6690 con 0x7f99b40c6b10 2024-03-06T07:31:12.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3420396542 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f99a8001670 con 0x7f99b40c6b10 2024-03-06T07:31:12.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3420396542 >> v1:172.21.15.1:6789/0 conn(0x7f99b40c6b10 legacy=0x7f99b4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b2ffd640 1 --2- 172.21.15.1:0/3420396542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f999c0039b0 0x7f999c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b2ffd640 1 --2- 172.21.15.1:0/3420396542 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99b4152650 0x7f999c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b2ffd640 1 --2- 172.21.15.1:0/3420396542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f999c004d40 0x7f999c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3420396542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b4155800 con 0x7f999c0039b0 2024-03-06T07:31:12.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3420396542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b4158da0 con 0x7f999c004d40 2024-03-06T07:31:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3420396542 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f999c00b9c0 con 0x7f99b4152650 2024-03-06T07:31:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b3fff640 1 --2- 172.21.15.1:0/3420396542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f999c0039b0 0x7f999c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b8852640 1 --2- 172.21.15.1:0/3420396542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f999c004d40 0x7f999c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.933+0000 7f99b37fe640 1 --2- 172.21.15.1:0/3420396542 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99b4152650 0x7f999c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.934+0000 7f99b3fff640 1 -- 172.21.15.1:0/3420396542 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99b4152650 msgr2=0x7f999c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.934+0000 7f99b3fff640 1 --2- 172.21.15.1:0/3420396542 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99b4152650 0x7f999c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.934+0000 7f99b3fff640 1 -- 172.21.15.1:0/3420396542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f999c004d40 msgr2=0x7f999c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.934+0000 7f99b3fff640 1 --2- 172.21.15.1:0/3420396542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f999c004d40 0x7f999c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.934+0000 7f99b3fff640 1 -- 172.21.15.1:0/3420396542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c003520 con 0x7f999c0039b0 2024-03-06T07:31:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.934+0000 7f99b3fff640 1 --2- 172.21.15.1:0/3420396542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f999c0039b0 0x7f999c003dd0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f99a4009ac0 tx=0x7f999c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c2c39d32662d05 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.935+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3420396542 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4039070 con 0x7f999c0039b0 2024-03-06T07:31:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.935+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3420396542 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a402c6b0 con 0x7f999c0039b0 2024-03-06T07:31:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.935+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3420396542 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4009690 con 0x7f999c0039b0 2024-03-06T07:31:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.935+0000 7f99ba2b5640 1 -- 172.21.15.1:0/3420396542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f999c0039b0 msgr2=0x7f999c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.935+0000 7f99ba2b5640 1 --2- 172.21.15.1:0/3420396542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f999c0039b0 0x7f999c003dd0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f99a4009ac0 tx=0x7f999c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.935+0000 7f99b8852640 1 -- 172.21.15.1:0/3420396542 reap_dead start 2024-03-06T07:31:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.935+0000 7f99ba2b5640 1 -- 172.21.15.1:0/3420396542 shutdown_connections 2024-03-06T07:31:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.935+0000 7f99ba2b5640 1 -- 172.21.15.1:0/3420396542 >> 172.21.15.1:0/3420396542 conn(0x7f99b4066ee0 msgr2=0x7f99b40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.935+0000 7f99ba2b5640 1 -- 172.21.15.1:0/3420396542 shutdown_connections 2024-03-06T07:31:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.935+0000 7f99ba2b5640 1 -- 172.21.15.1:0/3420396542 wait complete. 2024-03-06T07:31:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.936+0000 7f99ba2b5640 1 Processor -- start 2024-03-06T07:31:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.937+0000 7f99ba2b5640 1 -- start start 2024-03-06T07:31:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.937+0000 7f99ba2b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f999c0039b0 0x7f99b40c5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.937+0000 7f99ba2b5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99b40c5d50 0x7f99b41eee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.937+0000 7f99ba2b5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99b41ef360 0x7f99b41df810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.937+0000 7f99ba2b5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b41551b0 con 0x7f99b40c5d50 2024-03-06T07:31:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.937+0000 7f99ba2b5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b40c6fa0 con 0x7f999c0039b0 2024-03-06T07:31:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.937+0000 7f99ba2b5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b4155490 con 0x7f99b41ef360 2024-03-06T07:31:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.937+0000 7f99b37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99b40c5d50 0x7f99b41eee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.937+0000 7f99b37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99b40c5d50 0x7f99b41eee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37932/0 (socket says 172.21.15.1:37932) 2024-03-06T07:31:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.937+0000 7f99b37fe640 1 -- 172.21.15.1:0/2468101487 learned_addr learned my addr 172.21.15.1:0/2468101487 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.938+0000 7f99b8852640 1 --2- 172.21.15.1:0/2468101487 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99b41ef360 0x7f99b41df810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.938+0000 7f99b3fff640 1 --2- 172.21.15.1:0/2468101487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f999c0039b0 0x7f99b40c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.938+0000 7f99b37fe640 1 -- 172.21.15.1:0/2468101487 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99b41ef360 msgr2=0x7f99b41df810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.938+0000 7f99b37fe640 1 --2- 172.21.15.1:0/2468101487 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99b41ef360 0x7f99b41df810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.938+0000 7f99b37fe640 1 -- 172.21.15.1:0/2468101487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f999c0039b0 msgr2=0x7f99b40c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.938+0000 7f99b37fe640 1 --2- 172.21.15.1:0/2468101487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f999c0039b0 0x7f99b40c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.938+0000 7f99b37fe640 1 -- 172.21.15.1:0/2468101487 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b4151e20 con 0x7f99b40c5d50 2024-03-06T07:31:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.938+0000 7f99b37fe640 1 --2- 172.21.15.1:0/2468101487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99b40c5d50 0x7f99b41eee20 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7f99a8001860 tx=0x7f99a800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.938+0000 7f99b0ff9640 1 -- 172.21.15.1:0/2468101487 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a800ee60 con 0x7f99b40c5d50 2024-03-06T07:31:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.938+0000 7f99b0ff9640 1 -- 172.21.15.1:0/2468101487 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a8018070 con 0x7f99b40c5d50 2024-03-06T07:31:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.938+0000 7f99ba2b5640 1 -- 172.21.15.1:0/2468101487 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b4151860 con 0x7f99b40c5d50 2024-03-06T07:31:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.938+0000 7f99b0ff9640 1 -- 172.21.15.1:0/2468101487 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a800cb10 con 0x7f99b40c5d50 2024-03-06T07:31:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.938+0000 7f99ba2b5640 1 -- 172.21.15.1:0/2468101487 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b41ee900 con 0x7f99b40c5d50 2024-03-06T07:31:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.939+0000 7f99ba2b5640 1 -- 172.21.15.1:0/2468101487 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b406b540 con 0x7f99b40c5d50 2024-03-06T07:31:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.940+0000 7f99b0ff9640 1 -- 172.21.15.1:0/2468101487 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99a800ccb0 con 0x7f99b40c5d50 2024-03-06T07:31:12.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.940+0000 7f99b0ff9640 1 --2- 172.21.15.1:0/2468101487 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f99a006b4d0 0x7f99a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.940+0000 7f99b0ff9640 1 -- 172.21.15.1:0/2468101487 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f99a808a2b0 con 0x7f99b40c5d50 2024-03-06T07:31:12.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.940+0000 7f99b3fff640 1 --2- 172.21.15.1:0/2468101487 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f99a006b4d0 0x7f99a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.941+0000 7f99b3fff640 1 --2- 172.21.15.1:0/2468101487 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f99a006b4d0 0x7f99a006d990 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f99a4036000 tx=0x7f99a4030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:12.944+0000 7f99b0ff9640 1 -- 172.21.15.1:0/2468101487 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99b406b540 con 0x7f99b40c5d50 2024-03-06T07:31:13.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:13.243+0000 7f99ba2b5640 1 -- 172.21.15.1:0/2468101487 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99b40c2f50 con 0x7f99b40c5d50 2024-03-06T07:31:13.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:13.243+0000 7f99b0ff9640 1 -- 172.21.15.1:0/2468101487 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7f99b40c2f50 con 0x7f99b40c5d50 2024-03-06T07:31:13.246 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:13.246 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:13.246 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:31:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:13.246+0000 7f99ba2b5640 1 -- 172.21.15.1:0/2468101487 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f99a006b4d0 msgr2=0x7f99a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:13.246+0000 7f99ba2b5640 1 --2- 172.21.15.1:0/2468101487 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f99a006b4d0 0x7f99a006d990 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f99a4036000 tx=0x7f99a4030000 comp rx=0 tx=0).stop 2024-03-06T07:31:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:13.246+0000 7f99ba2b5640 1 -- 172.21.15.1:0/2468101487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99b40c5d50 msgr2=0x7f99b41eee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:13.246+0000 7f99ba2b5640 1 --2- 172.21.15.1:0/2468101487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99b40c5d50 0x7f99b41eee20 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7f99a8001860 tx=0x7f99a800ec10 comp rx=0 tx=0).stop 2024-03-06T07:31:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:13.246+0000 7f99ba2b5640 1 -- 172.21.15.1:0/2468101487 shutdown_connections 2024-03-06T07:31:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:13.246+0000 7f99ba2b5640 1 --2- 172.21.15.1:0/2468101487 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f99a006b4d0 0x7f99a006d990 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:13.246+0000 7f99ba2b5640 1 --2- 172.21.15.1:0/2468101487 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99b41ef360 0x7f99b41df810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:13.246+0000 7f99ba2b5640 1 --2- 172.21.15.1:0/2468101487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99b40c5d50 0x7f99b41eee20 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:13.246+0000 7f99ba2b5640 1 --2- 172.21.15.1:0/2468101487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f999c0039b0 0x7f99b40c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:13.246+0000 7f99ba2b5640 1 -- 172.21.15.1:0/2468101487 >> 172.21.15.1:0/2468101487 conn(0x7f99b4066ee0 msgr2=0x7f99b40c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:13.246+0000 7f99ba2b5640 1 -- 172.21.15.1:0/2468101487 shutdown_connections 2024-03-06T07:31:13.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:13.246+0000 7f99ba2b5640 1 -- 172.21.15.1:0/2468101487 wait complete. 2024-03-06T07:31:13.661 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:13.661 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:14.662 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:14.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.761+0000 7f22911bf640 1 Processor -- start 2024-03-06T07:31:14.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.761+0000 7f22911bf640 1 -- start start 2024-03-06T07:31:14.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.762+0000 7f22911bf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f228c152650 0x7f228c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.762+0000 7f22911bf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f228c0c6b10 0x7f228c158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.762+0000 7f22911bf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c0c2fc0 con 0x7f228c0bf9d0 2024-03-06T07:31:14.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.762+0000 7f22911bf640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f228c155800 con 0x7f228c0c6b10 2024-03-06T07:31:14.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.762+0000 7f22911bf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c0c6690 con 0x7f228c152650 2024-03-06T07:31:14.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.762+0000 7f228b59e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f228c0bf9d0 0x7f228c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52180/0 (socket says 172.21.15.1:52180) 2024-03-06T07:31:14.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.762+0000 7f228b59e640 1 -- 172.21.15.1:0/724634567 learned_addr learned my addr 172.21.15.1:0/724634567 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:31:14.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f228ad9d640 1 --2- 172.21.15.1:0/724634567 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f228c152650 0x7f228c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f2289d9b640 1 -- 172.21.15.1:0/724634567 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550813690 0 0) 0x7f228c0c2fc0 con 0x7f228c0bf9d0 2024-03-06T07:31:14.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f2289d9b640 1 -- 172.21.15.1:0/724634567 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2278003520 con 0x7f228c0bf9d0 2024-03-06T07:31:14.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f228a59c640 1 --2- 172.21.15.1:0/724634567 >> v2:172.21.15.98:3300/0 conn(0x7f228c0c6b10 0x7f228c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f2289d9b640 1 -- 172.21.15.1:0/724634567 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2623456787 0 0) 0x7f2278003520 con 0x7f228c0bf9d0 2024-03-06T07:31:14.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f2289d9b640 1 -- 172.21.15.1:0/724634567 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f228c152650 msgr2=0x7f228c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f2289d9b640 1 --2- 172.21.15.1:0/724634567 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f228c152650 0x7f228c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f2289d9b640 1 -- 172.21.15.1:0/724634567 >> v2:172.21.15.98:3300/0 conn(0x7f228c0c6b10 msgr2=0x7f228c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f2289d9b640 1 --2- 172.21.15.1:0/724634567 >> v2:172.21.15.98:3300/0 conn(0x7f228c0c6b10 0x7f228c158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f2289d9b640 1 -- 172.21.15.1:0/724634567 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228c0c7280 con 0x7f228c0bf9d0 2024-03-06T07:31:14.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f2289d9b640 1 -- 172.21.15.1:0/724634567 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f22840019f0 con 0x7f228c0bf9d0 2024-03-06T07:31:14.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f2289d9b640 1 -- 172.21.15.1:0/724634567 >> v1:172.21.15.1:6789/0 conn(0x7f228c0bf9d0 legacy=0x7f228c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f2289d9b640 1 --2- 172.21.15.1:0/724634567 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f228c152650 0x7f22780039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f2289d9b640 1 --2- 172.21.15.1:0/724634567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2278003f30 0x7f2278004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f2289d9b640 1 --2- 172.21.15.1:0/724634567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2278004d40 0x7f22780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f2289d9b640 1 -- 172.21.15.1:0/724634567 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c155800 con 0x7f2278004d40 2024-03-06T07:31:14.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f2289d9b640 1 -- 172.21.15.1:0/724634567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c0c6690 con 0x7f2278003f30 2024-03-06T07:31:14.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f2289d9b640 1 -- 172.21.15.1:0/724634567 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f227800ba10 con 0x7f228c152650 2024-03-06T07:31:14.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f228b59e640 1 --2- 172.21.15.1:0/724634567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2278004d40 0x7f22780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f228ad9d640 1 --2- 172.21.15.1:0/724634567 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f228c152650 0x7f22780039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f228a59c640 1 --2- 172.21.15.1:0/724634567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2278003f30 0x7f2278004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f228b59e640 1 -- 172.21.15.1:0/724634567 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f228c152650 msgr2=0x7f22780039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f228b59e640 1 --2- 172.21.15.1:0/724634567 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f228c152650 0x7f22780039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f228b59e640 1 -- 172.21.15.1:0/724634567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2278003f30 msgr2=0x7f2278004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f228b59e640 1 --2- 172.21.15.1:0/724634567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2278003f30 0x7f2278004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.763+0000 7f228b59e640 1 -- 172.21.15.1:0/724634567 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2278003520 con 0x7f2278004d40 2024-03-06T07:31:14.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.764+0000 7f228b59e640 1 --2- 172.21.15.1:0/724634567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2278004d40 0x7f22780051c0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f228400a8a0 tx=0x7f2278005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5b1a95e4f3e5ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.764+0000 7f2289d9b640 1 -- 172.21.15.1:0/724634567 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228400a4c0 con 0x7f2278004d40 2024-03-06T07:31:14.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.764+0000 7f2289d9b640 1 -- 172.21.15.1:0/724634567 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2284034630 con 0x7f2278004d40 2024-03-06T07:31:14.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.764+0000 7f2289d9b640 1 -- 172.21.15.1:0/724634567 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2284034a20 con 0x7f2278004d40 2024-03-06T07:31:14.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.765+0000 7f22911bf640 1 -- 172.21.15.1:0/724634567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2278004d40 msgr2=0x7f22780051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.765+0000 7f22911bf640 1 --2- 172.21.15.1:0/724634567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2278004d40 0x7f22780051c0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f228400a8a0 tx=0x7f2278005830 comp rx=0 tx=0).stop 2024-03-06T07:31:14.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.765+0000 7f228b59e640 1 -- 172.21.15.1:0/724634567 reap_dead start 2024-03-06T07:31:14.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.765+0000 7f22911bf640 1 -- 172.21.15.1:0/724634567 shutdown_connections 2024-03-06T07:31:14.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.765+0000 7f22911bf640 1 -- 172.21.15.1:0/724634567 >> 172.21.15.1:0/724634567 conn(0x7f228c066ee0 msgr2=0x7f228c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.765+0000 7f22911bf640 1 -- 172.21.15.1:0/724634567 shutdown_connections 2024-03-06T07:31:14.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.765+0000 7f22911bf640 1 -- 172.21.15.1:0/724634567 wait complete. 2024-03-06T07:31:14.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.766+0000 7f22911bf640 1 Processor -- start 2024-03-06T07:31:14.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.766+0000 7f22911bf640 1 -- start start 2024-03-06T07:31:14.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.767+0000 7f22911bf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f228c150b30 0x7f228c150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.767+0000 7f22911bf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f228c14faf0 0x7f228c1dffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.767+0000 7f22911bf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f228c1e0510 0x7f228c0c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.767+0000 7f22911bf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c155490 con 0x7f228c14faf0 2024-03-06T07:31:14.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.767+0000 7f22911bf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c1551b0 con 0x7f228c1e0510 2024-03-06T07:31:14.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.767+0000 7f22911bf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c152a30 con 0x7f228c150b30 2024-03-06T07:31:14.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.767+0000 7f228a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f228c14faf0 0x7f228c1dffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.767+0000 7f228a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f228c14faf0 0x7f228c1dffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37698/0 (socket says 172.21.15.1:37698) 2024-03-06T07:31:14.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.767+0000 7f228a59c640 1 -- 172.21.15.1:0/1966322295 learned_addr learned my addr 172.21.15.1:0/1966322295 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:14.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.767+0000 7f228ad9d640 1 --2- 172.21.15.1:0/1966322295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f228c150b30 0x7f228c150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.767+0000 7f228b59e640 1 --2- 172.21.15.1:0/1966322295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f228c1e0510 0x7f228c0c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.767+0000 7f228a59c640 1 -- 172.21.15.1:0/1966322295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f228c150b30 msgr2=0x7f228c150020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.767+0000 7f228a59c640 1 --2- 172.21.15.1:0/1966322295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f228c150b30 0x7f228c150020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.767+0000 7f228a59c640 1 -- 172.21.15.1:0/1966322295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f228c1e0510 msgr2=0x7f228c0c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.767+0000 7f228a59c640 1 --2- 172.21.15.1:0/1966322295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f228c1e0510 0x7f228c0c8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.767+0000 7f228a59c640 1 -- 172.21.15.1:0/1966322295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228c1eb860 con 0x7f228c14faf0 2024-03-06T07:31:14.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.767+0000 7f228b59e640 1 --2- 172.21.15.1:0/1966322295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f228c1e0510 0x7f228c0c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:14.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.768+0000 7f228a59c640 1 --2- 172.21.15.1:0/1966322295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f228c14faf0 0x7f228c1dffd0 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f227c011960 tx=0x7f227c011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.768+0000 7f22837fe640 1 -- 172.21.15.1:0/1966322295 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c00fa30 con 0x7f228c14faf0 2024-03-06T07:31:14.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.768+0000 7f22837fe640 1 -- 172.21.15.1:0/1966322295 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227c00fbd0 con 0x7f228c14faf0 2024-03-06T07:31:14.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.768+0000 7f22911bf640 1 -- 172.21.15.1:0/1966322295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228c1e1030 con 0x7f228c14faf0 2024-03-06T07:31:14.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.768+0000 7f22837fe640 1 -- 172.21.15.1:0/1966322295 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c014440 con 0x7f228c14faf0 2024-03-06T07:31:14.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.768+0000 7f22911bf640 1 -- 172.21.15.1:0/1966322295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f228c14b620 con 0x7f228c14faf0 2024-03-06T07:31:14.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.769+0000 7f22911bf640 1 -- 172.21.15.1:0/1966322295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f228c067fd0 con 0x7f228c14faf0 2024-03-06T07:31:14.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.769+0000 7f22837fe640 1 -- 172.21.15.1:0/1966322295 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f227c012040 con 0x7f228c14faf0 2024-03-06T07:31:14.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.770+0000 7f22837fe640 1 --2- 172.21.15.1:0/1966322295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f226c06b4d0 0x7f226c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.770+0000 7f22837fe640 1 -- 172.21.15.1:0/1966322295 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f227c08aa40 con 0x7f228c14faf0 2024-03-06T07:31:14.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.770+0000 7f228ad9d640 1 --2- 172.21.15.1:0/1966322295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f226c06b4d0 0x7f226c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.771+0000 7f228ad9d640 1 --2- 172.21.15.1:0/1966322295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f226c06b4d0 0x7f226c06d990 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f228c0ca850 tx=0x7f2274009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.774+0000 7f22837fe640 1 -- 172.21.15.1:0/1966322295 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f227c019030 con 0x7f228c14faf0 2024-03-06T07:31:14.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.992+0000 7f22911bf640 1 -- 172.21.15.1:0/1966322295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f228c0b06f0 con 0x7f228c14faf0 2024-03-06T07:31:14.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.993+0000 7f22837fe640 1 -- 172.21.15.1:0/1966322295 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7f227c08ae20 con 0x7f228c14faf0 2024-03-06T07:31:14.995 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:14.995 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:14.995 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:31:14.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.994+0000 7f22911bf640 1 -- 172.21.15.1:0/1966322295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f226c06b4d0 msgr2=0x7f226c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.994+0000 7f22911bf640 1 --2- 172.21.15.1:0/1966322295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f226c06b4d0 0x7f226c06d990 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f228c0ca850 tx=0x7f2274009190 comp rx=0 tx=0).stop 2024-03-06T07:31:14.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.994+0000 7f22911bf640 1 -- 172.21.15.1:0/1966322295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f228c14faf0 msgr2=0x7f228c1dffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.994+0000 7f22911bf640 1 --2- 172.21.15.1:0/1966322295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f228c14faf0 0x7f228c1dffd0 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f227c011960 tx=0x7f227c011d60 comp rx=0 tx=0).stop 2024-03-06T07:31:14.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.994+0000 7f22911bf640 1 -- 172.21.15.1:0/1966322295 shutdown_connections 2024-03-06T07:31:14.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.994+0000 7f22911bf640 1 --2- 172.21.15.1:0/1966322295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f226c06b4d0 0x7f226c06d990 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.994+0000 7f22911bf640 1 --2- 172.21.15.1:0/1966322295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f228c1e0510 0x7f228c0c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.994+0000 7f22911bf640 1 --2- 172.21.15.1:0/1966322295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f228c14faf0 0x7f228c1dffd0 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.994+0000 7f22911bf640 1 --2- 172.21.15.1:0/1966322295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f228c150b30 0x7f228c150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.994+0000 7f22911bf640 1 -- 172.21.15.1:0/1966322295 >> 172.21.15.1:0/1966322295 conn(0x7f228c066ee0 msgr2=0x7f228c0bfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.995+0000 7f22911bf640 1 -- 172.21.15.1:0/1966322295 shutdown_connections 2024-03-06T07:31:14.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:14.995+0000 7f22911bf640 1 -- 172.21.15.1:0/1966322295 wait complete. 2024-03-06T07:31:15.413 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:15.413 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:16.415 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:16.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.535+0000 7f98135d6640 1 Processor -- start 2024-03-06T07:31:16.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.536+0000 7f98135d6640 1 -- start start 2024-03-06T07:31:16.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.537+0000 7f98135d6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f980c150640 0x7f980c0c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.537+0000 7f98135d6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f980c0a84b0 0x7f980c152f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.537+0000 7f98135d6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980c06fe00 con 0x7f980c0a9cf0 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.537+0000 7f98135d6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980c0675e0 con 0x7f980c150640 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.537+0000 7f98135d6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c0c24d0 con 0x7f980c0a84b0 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.537+0000 7f9811372640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f980c150640 0x7f980c0c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.537+0000 7f9810b71640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f980c0a84b0 0x7f980c152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.537+0000 7f9810b71640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f980c0a84b0 0x7f980c152f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:54250/0 (socket says 172.21.15.1:54250) 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.537+0000 7f9811372640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f980c150640 0x7f980c0c1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:51910/0 (socket says 172.21.15.1:51910) 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.537+0000 7f9810b71640 1 -- 172.21.15.1:0/4159814153 learned_addr learned my addr 172.21.15.1:0/4159814153 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.537+0000 7f9811b73640 1 --1- 172.21.15.1:0/4159814153 >> v1:172.21.15.98:6789/0 conn(0x7f980c0a9cf0 0x7f980c156490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:38998/0 (socket says 172.21.15.1:38998) 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.537+0000 7f9810b71640 1 -- 172.21.15.1:0/4159814153 >> v2:172.21.15.98:3300/0 conn(0x7f980c150640 msgr2=0x7f980c0c1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.537+0000 7f9810b71640 1 --2- 172.21.15.1:0/4159814153 >> v2:172.21.15.98:3300/0 conn(0x7f980c150640 0x7f980c0c1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.537+0000 7f9810b71640 1 -- 172.21.15.1:0/4159814153 >> v1:172.21.15.98:6789/0 conn(0x7f980c0a9cf0 legacy=0x7f980c156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.537+0000 7f9810b71640 1 -- 172.21.15.1:0/4159814153 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c06fb20 con 0x7f980c0a84b0 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.537+0000 7f9811372640 1 --2- 172.21.15.1:0/4159814153 >> v2:172.21.15.98:3300/0 conn(0x7f980c150640 0x7f980c0c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.538+0000 7f9810b71640 1 --2- 172.21.15.1:0/4159814153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f980c0a84b0 0x7f980c152f20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f97fc009870 tx=0x7f97fc02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf7fed22715698ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.538+0000 7f9803fff640 1 -- 172.21.15.1:0/4159814153 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc0317c0 con 0x7f980c0a84b0 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.538+0000 7f9803fff640 1 -- 172.21.15.1:0/4159814153 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97fc037070 con 0x7f980c0a84b0 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.538+0000 7f9803fff640 1 -- 172.21.15.1:0/4159814153 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc031b80 con 0x7f980c0a84b0 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.539+0000 7f98135d6640 1 -- 172.21.15.1:0/4159814153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f980c0a84b0 msgr2=0x7f980c152f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.539+0000 7f98135d6640 1 --2- 172.21.15.1:0/4159814153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f980c0a84b0 0x7f980c152f20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f97fc009870 tx=0x7f97fc02b860 comp rx=0 tx=0).stop 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.539+0000 7f98135d6640 1 -- 172.21.15.1:0/4159814153 shutdown_connections 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.539+0000 7f98135d6640 1 --2- 172.21.15.1:0/4159814153 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f980c0a84b0 0x7f980c152f20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.539+0000 7f98135d6640 1 --2- 172.21.15.1:0/4159814153 >> v2:172.21.15.98:3300/0 conn(0x7f980c150640 0x7f980c0c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.539+0000 7f98135d6640 1 -- 172.21.15.1:0/4159814153 >> 172.21.15.1:0/4159814153 conn(0x7f980c0c3cc0 msgr2=0x7f980c0c40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.539+0000 7f98135d6640 1 -- 172.21.15.1:0/4159814153 shutdown_connections 2024-03-06T07:31:16.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.539+0000 7f98135d6640 1 -- 172.21.15.1:0/4159814153 wait complete. 2024-03-06T07:31:16.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.540+0000 7f98135d6640 1 Processor -- start 2024-03-06T07:31:16.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.540+0000 7f98135d6640 1 -- start start 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.541+0000 7f98135d6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f980c0a84b0 0x7f980c1eb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.541+0000 7f98135d6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f980c0a9cf0 0x7f980c1ee520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.541+0000 7f98135d6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f980c150640 0x7f980c0c05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.541+0000 7f98135d6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c153e20 con 0x7f980c150640 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.541+0000 7f98135d6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c153b40 con 0x7f980c0a84b0 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.541+0000 7f98135d6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c0c2100 con 0x7f980c0a9cf0 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.541+0000 7f9811b73640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f980c150640 0x7f980c0c05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.541+0000 7f9811b73640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f980c150640 0x7f980c0c05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37712/0 (socket says 172.21.15.1:37712) 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.541+0000 7f9811b73640 1 -- 172.21.15.1:0/556926464 learned_addr learned my addr 172.21.15.1:0/556926464 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.541+0000 7f9810b71640 1 --2- 172.21.15.1:0/556926464 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f980c0a9cf0 0x7f980c1ee520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.541+0000 7f9811372640 1 --2- 172.21.15.1:0/556926464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f980c0a84b0 0x7f980c1eb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.541+0000 7f9811b73640 1 -- 172.21.15.1:0/556926464 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f980c0a9cf0 msgr2=0x7f980c1ee520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.541+0000 7f9811b73640 1 --2- 172.21.15.1:0/556926464 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f980c0a9cf0 0x7f980c1ee520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.541+0000 7f9811b73640 1 -- 172.21.15.1:0/556926464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f980c0a84b0 msgr2=0x7f980c1eb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.541+0000 7f9811b73640 1 --2- 172.21.15.1:0/556926464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f980c0a84b0 0x7f980c1eb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.541+0000 7f9811b73640 1 -- 172.21.15.1:0/556926464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c0ab6e0 con 0x7f980c150640 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.542+0000 7f9811b73640 1 --2- 172.21.15.1:0/556926464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f980c150640 0x7f980c0c05f0 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f980400c780 tx=0x7f980400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.542+0000 7f9801ffb640 1 -- 172.21.15.1:0/556926464 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9804017070 con 0x7f980c150640 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.542+0000 7f9801ffb640 1 -- 172.21.15.1:0/556926464 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980400a5e0 con 0x7f980c150640 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.542+0000 7f98135d6640 1 -- 172.21.15.1:0/556926464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980c0ab9c0 con 0x7f980c150640 2024-03-06T07:31:16.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.542+0000 7f9801ffb640 1 -- 172.21.15.1:0/556926464 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980400a9d0 con 0x7f980c150640 2024-03-06T07:31:16.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.542+0000 7f98135d6640 1 -- 172.21.15.1:0/556926464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980c1ee000 con 0x7f980c150640 2024-03-06T07:31:16.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.543+0000 7f98135d6640 1 -- 172.21.15.1:0/556926464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f980c06dd40 con 0x7f980c150640 2024-03-06T07:31:16.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.544+0000 7f9801ffb640 1 -- 172.21.15.1:0/556926464 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f980401a070 con 0x7f980c150640 2024-03-06T07:31:16.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.544+0000 7f9801ffb640 1 --2- 172.21.15.1:0/556926464 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f97f006b4d0 0x7f97f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.544+0000 7f9801ffb640 1 -- 172.21.15.1:0/556926464 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f9804089950 con 0x7f980c150640 2024-03-06T07:31:16.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.544+0000 7f9811372640 1 --2- 172.21.15.1:0/556926464 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f97f006b4d0 0x7f97f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.545+0000 7f9811372640 1 --2- 172.21.15.1:0/556926464 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f97f006b4d0 0x7f97f006d990 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f980c06c6e0 tx=0x7f9808008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.549+0000 7f9801ffb640 1 -- 172.21.15.1:0/556926464 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f980c06dd40 con 0x7f980c150640 2024-03-06T07:31:16.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.768+0000 7f98135d6640 1 -- 172.21.15.1:0/556926464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f980c0686e0 con 0x7f980c150640 2024-03-06T07:31:16.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.768+0000 7f9801ffb640 1 -- 172.21.15.1:0/556926464 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7f9804057ee0 con 0x7f980c150640 2024-03-06T07:31:16.770 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:16.770 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:16.771 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:31:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.770+0000 7f98135d6640 1 -- 172.21.15.1:0/556926464 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f97f006b4d0 msgr2=0x7f97f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.770+0000 7f98135d6640 1 --2- 172.21.15.1:0/556926464 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f97f006b4d0 0x7f97f006d990 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f980c06c6e0 tx=0x7f9808008040 comp rx=0 tx=0).stop 2024-03-06T07:31:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.770+0000 7f98135d6640 1 -- 172.21.15.1:0/556926464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f980c150640 msgr2=0x7f980c0c05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.770+0000 7f98135d6640 1 --2- 172.21.15.1:0/556926464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f980c150640 0x7f980c0c05f0 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f980400c780 tx=0x7f980400cc40 comp rx=0 tx=0).stop 2024-03-06T07:31:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.770+0000 7f98135d6640 1 -- 172.21.15.1:0/556926464 shutdown_connections 2024-03-06T07:31:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.770+0000 7f98135d6640 1 --2- 172.21.15.1:0/556926464 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f97f006b4d0 0x7f97f006d990 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.770+0000 7f98135d6640 1 --2- 172.21.15.1:0/556926464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f980c150640 0x7f980c0c05f0 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.770+0000 7f98135d6640 1 --2- 172.21.15.1:0/556926464 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f980c0a9cf0 0x7f980c1ee520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.770+0000 7f98135d6640 1 --2- 172.21.15.1:0/556926464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f980c0a84b0 0x7f980c1eb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.770+0000 7f98135d6640 1 -- 172.21.15.1:0/556926464 >> 172.21.15.1:0/556926464 conn(0x7f980c0c3cc0 msgr2=0x7f980c0c40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.771+0000 7f98135d6640 1 -- 172.21.15.1:0/556926464 shutdown_connections 2024-03-06T07:31:16.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:16.771+0000 7f98135d6640 1 -- 172.21.15.1:0/556926464 wait complete. 2024-03-06T07:31:17.185 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:17.186 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:18.186 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:18.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.286+0000 7f833ed56640 1 Processor -- start 2024-03-06T07:31:18.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.286+0000 7f833ed56640 1 -- start start 2024-03-06T07:31:18.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.287+0000 7f833ed56640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f83380c6b10 0x7f8338155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.287+0000 7f833ed56640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f83380bf9d0 0x7f83380c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.287+0000 7f833ed56640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83380c2fc0 con 0x7f8338152650 2024-03-06T07:31:18.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.287+0000 7f833ed56640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8338155800 con 0x7f83380bf9d0 2024-03-06T07:31:18.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.287+0000 7f833ed56640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8338158da0 con 0x7f83380c6b10 2024-03-06T07:31:18.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.287+0000 7f833d2f3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f83380bf9d0 0x7f83380c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.287+0000 7f833caf2640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8338152650 0x7f8338158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52196/0 (socket says 172.21.15.1:52196) 2024-03-06T07:31:18.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.287+0000 7f833d2f3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f83380bf9d0 0x7f83380c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37728/0 (socket says 172.21.15.1:37728) 2024-03-06T07:31:18.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.287+0000 7f833caf2640 1 -- 172.21.15.1:0/1142465843 learned_addr learned my addr 172.21.15.1:0/1142465843 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:31:18.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.288+0000 7f833d2f3640 1 -- 172.21.15.1:0/1142465843 >> v2:172.21.15.98:3300/0 conn(0x7f83380c6b10 msgr2=0x7f8338155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.288+0000 7f8337fff640 1 --2- 172.21.15.1:0/1142465843 >> v2:172.21.15.98:3300/0 conn(0x7f83380c6b10 0x7f8338155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.288+0000 7f833d2f3640 1 --2- 172.21.15.1:0/1142465843 >> v2:172.21.15.98:3300/0 conn(0x7f83380c6b10 0x7f8338155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.288+0000 7f83377fe640 1 -- 172.21.15.1:0/1142465843 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670253334 0 0) 0x7f83380c2fc0 con 0x7f8338152650 2024-03-06T07:31:18.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.288+0000 7f833d2f3640 1 -- 172.21.15.1:0/1142465843 >> v1:172.21.15.1:6789/0 conn(0x7f8338152650 legacy=0x7f8338158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.288+0000 7f833d2f3640 1 -- 172.21.15.1:0/1142465843 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83380c6690 con 0x7f83380bf9d0 2024-03-06T07:31:18.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.288+0000 7f833d2f3640 1 --2- 172.21.15.1:0/1142465843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83380bf9d0 0x7f83380c1460 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f83380cb8e0 tx=0x7f833002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7502d86e47bf6d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.288+0000 7f83377fe640 1 -- 172.21.15.1:0/1142465843 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8330039070 con 0x7f83380bf9d0 2024-03-06T07:31:18.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.289+0000 7f83377fe640 1 -- 172.21.15.1:0/1142465843 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833002cbf0 con 0x7f83380bf9d0 2024-03-06T07:31:18.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.289+0000 7f83377fe640 1 -- 172.21.15.1:0/1142465843 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8330034440 con 0x7f83380bf9d0 2024-03-06T07:31:18.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.289+0000 7f833ed56640 1 -- 172.21.15.1:0/1142465843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83380bf9d0 msgr2=0x7f83380c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.289+0000 7f833ed56640 1 --2- 172.21.15.1:0/1142465843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83380bf9d0 0x7f83380c1460 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f83380cb8e0 tx=0x7f833002be40 comp rx=0 tx=0).stop 2024-03-06T07:31:18.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.289+0000 7f833ed56640 1 -- 172.21.15.1:0/1142465843 shutdown_connections 2024-03-06T07:31:18.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.289+0000 7f833ed56640 1 --2- 172.21.15.1:0/1142465843 >> v2:172.21.15.98:3300/0 conn(0x7f83380c6b10 0x7f8338155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.289+0000 7f833ed56640 1 --2- 172.21.15.1:0/1142465843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83380bf9d0 0x7f83380c1460 unknown :-1 s=CLOSED pgs=2500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.289+0000 7f833ed56640 1 -- 172.21.15.1:0/1142465843 >> 172.21.15.1:0/1142465843 conn(0x7f8338066ee0 msgr2=0x7f83380672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.289+0000 7f833ed56640 1 -- 172.21.15.1:0/1142465843 shutdown_connections 2024-03-06T07:31:18.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.289+0000 7f833ed56640 1 -- 172.21.15.1:0/1142465843 wait complete. 2024-03-06T07:31:18.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.290+0000 7f833ed56640 1 Processor -- start 2024-03-06T07:31:18.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.290+0000 7f833ed56640 1 -- start start 2024-03-06T07:31:18.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.291+0000 7f833ed56640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83380bf9d0 0x7f833814cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.291+0000 7f833ed56640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8338152650 0x7f83381eb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.291+0000 7f833ed56640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83381e1b00 0x7f83381eb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.291+0000 7f833ed56640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83381551b0 con 0x7f8338152650 2024-03-06T07:31:18.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.291+0000 7f833ed56640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380c6fa0 con 0x7f83380bf9d0 2024-03-06T07:31:18.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.291+0000 7f833ed56640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338155490 con 0x7f83381e1b00 2024-03-06T07:31:18.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.291+0000 7f8337fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8338152650 0x7f83381eb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.291+0000 7f8337fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8338152650 0x7f83381eb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37744/0 (socket says 172.21.15.1:37744) 2024-03-06T07:31:18.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.291+0000 7f8337fff640 1 -- 172.21.15.1:0/4049348001 learned_addr learned my addr 172.21.15.1:0/4049348001 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:18.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.291+0000 7f8337fff640 1 -- 172.21.15.1:0/4049348001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83381e1b00 msgr2=0x7f83381eb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.291+0000 7f8337fff640 1 --2- 172.21.15.1:0/4049348001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83381e1b00 0x7f83381eb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.291+0000 7f8337fff640 1 -- 172.21.15.1:0/4049348001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83380bf9d0 msgr2=0x7f833814cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.291+0000 7f8337fff640 1 --2- 172.21.15.1:0/4049348001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83380bf9d0 0x7f833814cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.291+0000 7f8337fff640 1 -- 172.21.15.1:0/4049348001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83381de700 con 0x7f8338152650 2024-03-06T07:31:18.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.292+0000 7f8337fff640 1 --2- 172.21.15.1:0/4049348001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8338152650 0x7f83381eb000 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7f832400c7f0 tx=0x7f832400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.292+0000 7f83357fa640 1 -- 172.21.15.1:0/4049348001 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8324015070 con 0x7f8338152650 2024-03-06T07:31:18.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.292+0000 7f83357fa640 1 -- 172.21.15.1:0/4049348001 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8324004240 con 0x7f8338152650 2024-03-06T07:31:18.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.292+0000 7f833ed56640 1 -- 172.21.15.1:0/4049348001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8338157220 con 0x7f8338152650 2024-03-06T07:31:18.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.292+0000 7f83357fa640 1 -- 172.21.15.1:0/4049348001 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8324007620 con 0x7f8338152650 2024-03-06T07:31:18.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.292+0000 7f833ed56640 1 -- 172.21.15.1:0/4049348001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83381577e0 con 0x7f8338152650 2024-03-06T07:31:18.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.293+0000 7f833ed56640 1 -- 172.21.15.1:0/4049348001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8338067fd0 con 0x7f8338152650 2024-03-06T07:31:18.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.293+0000 7f83357fa640 1 -- 172.21.15.1:0/4049348001 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83240079d0 con 0x7f8338152650 2024-03-06T07:31:18.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.294+0000 7f83357fa640 1 --2- 172.21.15.1:0/4049348001 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f831c06b4d0 0x7f831c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.294+0000 7f83357fa640 1 -- 172.21.15.1:0/4049348001 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f832408a880 con 0x7f8338152650 2024-03-06T07:31:18.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.294+0000 7f833caf2640 1 --2- 172.21.15.1:0/4049348001 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f831c06b4d0 0x7f831c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.295+0000 7f833caf2640 1 --2- 172.21.15.1:0/4049348001 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f831c06b4d0 0x7f831c06d990 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f832c007fd0 tx=0x7f832c007920 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.298+0000 7f83357fa640 1 -- 172.21.15.1:0/4049348001 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8324059500 con 0x7f8338152650 2024-03-06T07:31:18.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.517+0000 7f833ed56640 1 -- 172.21.15.1:0/4049348001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83381eba80 con 0x7f8338152650 2024-03-06T07:31:18.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.517+0000 7f83357fa640 1 -- 172.21.15.1:0/4049348001 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7f8324058e00 con 0x7f8338152650 2024-03-06T07:31:18.519 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:18.519 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:18.519 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:31:18.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.519+0000 7f833ed56640 1 -- 172.21.15.1:0/4049348001 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f831c06b4d0 msgr2=0x7f831c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.519+0000 7f833ed56640 1 --2- 172.21.15.1:0/4049348001 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f831c06b4d0 0x7f831c06d990 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f832c007fd0 tx=0x7f832c007920 comp rx=0 tx=0).stop 2024-03-06T07:31:18.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.519+0000 7f833ed56640 1 -- 172.21.15.1:0/4049348001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8338152650 msgr2=0x7f83381eb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.519+0000 7f833ed56640 1 --2- 172.21.15.1:0/4049348001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8338152650 0x7f83381eb000 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7f832400c7f0 tx=0x7f832400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.519+0000 7f833ed56640 1 -- 172.21.15.1:0/4049348001 shutdown_connections 2024-03-06T07:31:18.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.519+0000 7f833ed56640 1 --2- 172.21.15.1:0/4049348001 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f831c06b4d0 0x7f831c06d990 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.519+0000 7f833ed56640 1 --2- 172.21.15.1:0/4049348001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f83381e1b00 0x7f83381eb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.519+0000 7f833ed56640 1 --2- 172.21.15.1:0/4049348001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8338152650 0x7f83381eb000 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.519+0000 7f833ed56640 1 --2- 172.21.15.1:0/4049348001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83380bf9d0 0x7f833814cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.519+0000 7f833ed56640 1 -- 172.21.15.1:0/4049348001 >> 172.21.15.1:0/4049348001 conn(0x7f8338066ee0 msgr2=0x7f83380672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.519+0000 7f833ed56640 1 -- 172.21.15.1:0/4049348001 shutdown_connections 2024-03-06T07:31:18.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:18.519+0000 7f833ed56640 1 -- 172.21.15.1:0/4049348001 wait complete. 2024-03-06T07:31:18.934 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:18.934 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:19.936 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:20.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.035+0000 7ff5b364e640 1 Processor -- start 2024-03-06T07:31:20.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.035+0000 7ff5b364e640 1 -- start start 2024-03-06T07:31:20.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.036+0000 7ff5b364e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff5ac152650 0x7ff5ac155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.036+0000 7ff5b364e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff5ac0c6b10 0x7ff5ac158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.036+0000 7ff5b364e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5ac0bf9d0 0x7ff5ac0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.036+0000 7ff5b364e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5ac155800 con 0x7ff5ac152650 2024-03-06T07:31:20.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.036+0000 7ff5b364e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5ac0c2fc0 con 0x7ff5ac0c6b10 2024-03-06T07:31:20.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.036+0000 7ff5b364e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac0c6690 con 0x7ff5ac0bf9d0 2024-03-06T07:31:20.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.036+0000 7ff5b13ea640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff5ac152650 0x7ff5ac155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.036+0000 7ff5b13ea640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff5ac152650 0x7ff5ac155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37752/0 (socket says 172.21.15.1:37752) 2024-03-06T07:31:20.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.036+0000 7ff5b13ea640 1 -- 172.21.15.1:0/2732026777 learned_addr learned my addr 172.21.15.1:0/2732026777 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:20.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.036+0000 7ff5b1beb640 1 --2- 172.21.15.1:0/2732026777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5ac0bf9d0 0x7ff5ac0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.036+0000 7ff5b0be9640 1 --2- 172.21.15.1:0/2732026777 >> v2:172.21.15.98:3300/0 conn(0x7ff5ac0c6b10 0x7ff5ac158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.036+0000 7ff5b13ea640 1 -- 172.21.15.1:0/2732026777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5ac0bf9d0 msgr2=0x7ff5ac0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:20.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.036+0000 7ff5b13ea640 1 --2- 172.21.15.1:0/2732026777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5ac0bf9d0 0x7ff5ac0c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.036+0000 7ff5b13ea640 1 -- 172.21.15.1:0/2732026777 >> v2:172.21.15.98:3300/0 conn(0x7ff5ac0c6b10 msgr2=0x7ff5ac158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:20.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.036+0000 7ff5b13ea640 1 --2- 172.21.15.1:0/2732026777 >> v2:172.21.15.98:3300/0 conn(0x7ff5ac0c6b10 0x7ff5ac158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.036+0000 7ff5b13ea640 1 -- 172.21.15.1:0/2732026777 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ac0c7280 con 0x7ff5ac152650 2024-03-06T07:31:20.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.037+0000 7ff5b13ea640 1 --2- 172.21.15.1:0/2732026777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5ac152650 0x7ff5ac155c90 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0cb8e0 tx=0x7ff5a802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a42b8ecbbece3b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.037+0000 7ff5a3fff640 1 -- 172.21.15.1:0/2732026777 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a802e070 con 0x7ff5ac152650 2024-03-06T07:31:20.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.037+0000 7ff5a3fff640 1 -- 172.21.15.1:0/2732026777 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a8037070 con 0x7ff5ac152650 2024-03-06T07:31:20.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.037+0000 7ff5a3fff640 1 -- 172.21.15.1:0/2732026777 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8031a00 con 0x7ff5ac152650 2024-03-06T07:31:20.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.037+0000 7ff5b364e640 1 -- 172.21.15.1:0/2732026777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5ac152650 msgr2=0x7ff5ac155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.037+0000 7ff5b364e640 1 --2- 172.21.15.1:0/2732026777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5ac152650 0x7ff5ac155c90 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0cb8e0 tx=0x7ff5a802b880 comp rx=0 tx=0).stop 2024-03-06T07:31:20.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.038+0000 7ff5b364e640 1 -- 172.21.15.1:0/2732026777 shutdown_connections 2024-03-06T07:31:20.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.038+0000 7ff5b364e640 1 --2- 172.21.15.1:0/2732026777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5ac0bf9d0 0x7ff5ac0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.038+0000 7ff5b364e640 1 --2- 172.21.15.1:0/2732026777 >> v2:172.21.15.98:3300/0 conn(0x7ff5ac0c6b10 0x7ff5ac158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.038+0000 7ff5b364e640 1 --2- 172.21.15.1:0/2732026777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5ac152650 0x7ff5ac155c90 unknown :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.038+0000 7ff5b364e640 1 -- 172.21.15.1:0/2732026777 >> 172.21.15.1:0/2732026777 conn(0x7ff5ac066ee0 msgr2=0x7ff5ac0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.038+0000 7ff5b364e640 1 -- 172.21.15.1:0/2732026777 shutdown_connections 2024-03-06T07:31:20.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.038+0000 7ff5b364e640 1 -- 172.21.15.1:0/2732026777 wait complete. 2024-03-06T07:31:20.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.039+0000 7ff5b364e640 1 Processor -- start 2024-03-06T07:31:20.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.039+0000 7ff5b364e640 1 -- start start 2024-03-06T07:31:20.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.039+0000 7ff5b364e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5ac0bf9d0 0x7ff5ac1eed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.039+0000 7ff5b364e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ac152650 0x7ff5ac14e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.039+0000 7ff5b364e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5ac14d340 0x7ff5ac1ec230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.039+0000 7ff5b364e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac155490 con 0x7ff5ac0bf9d0 2024-03-06T07:31:20.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.039+0000 7ff5b364e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac1551b0 con 0x7ff5ac152650 2024-03-06T07:31:20.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.039+0000 7ff5b364e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac152a30 con 0x7ff5ac14d340 2024-03-06T07:31:20.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.040+0000 7ff5b13ea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5ac0bf9d0 0x7ff5ac1eed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.040+0000 7ff5b13ea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5ac0bf9d0 0x7ff5ac1eed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37768/0 (socket says 172.21.15.1:37768) 2024-03-06T07:31:20.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.040+0000 7ff5b13ea640 1 -- 172.21.15.1:0/3321966650 learned_addr learned my addr 172.21.15.1:0/3321966650 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:20.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.040+0000 7ff5b1beb640 1 --2- 172.21.15.1:0/3321966650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5ac14d340 0x7ff5ac1ec230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.040+0000 7ff5b0be9640 1 --2- 172.21.15.1:0/3321966650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ac152650 0x7ff5ac14e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.040+0000 7ff5b13ea640 1 -- 172.21.15.1:0/3321966650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5ac14d340 msgr2=0x7ff5ac1ec230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.040+0000 7ff5b13ea640 1 --2- 172.21.15.1:0/3321966650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5ac14d340 0x7ff5ac1ec230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.040+0000 7ff5b13ea640 1 -- 172.21.15.1:0/3321966650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ac152650 msgr2=0x7ff5ac14e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.040+0000 7ff5b13ea640 1 --2- 172.21.15.1:0/3321966650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ac152650 0x7ff5ac14e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.040+0000 7ff5b13ea640 1 -- 172.21.15.1:0/3321966650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ac1ee2d0 con 0x7ff5ac0bf9d0 2024-03-06T07:31:20.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.040+0000 7ff5b13ea640 1 --2- 172.21.15.1:0/3321966650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5ac0bf9d0 0x7ff5ac1eed80 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7ff5a802b8c0 tx=0x7ff5a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.040+0000 7ff5a1ffb640 1 -- 172.21.15.1:0/3321966650 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8032880 con 0x7ff5ac0bf9d0 2024-03-06T07:31:20.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.040+0000 7ff5a1ffb640 1 -- 172.21.15.1:0/3321966650 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a8032a20 con 0x7ff5ac0bf9d0 2024-03-06T07:31:20.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.040+0000 7ff5b364e640 1 -- 172.21.15.1:0/3321966650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5ac14d820 con 0x7ff5ac0bf9d0 2024-03-06T07:31:20.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.040+0000 7ff5a1ffb640 1 -- 172.21.15.1:0/3321966650 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8032e10 con 0x7ff5ac0bf9d0 2024-03-06T07:31:20.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.040+0000 7ff5b364e640 1 -- 172.21.15.1:0/3321966650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5ac1ee580 con 0x7ff5ac0bf9d0 2024-03-06T07:31:20.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.041+0000 7ff5b364e640 1 -- 172.21.15.1:0/3321966650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5ac067fd0 con 0x7ff5ac0bf9d0 2024-03-06T07:31:20.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.042+0000 7ff5a1ffb640 1 -- 172.21.15.1:0/3321966650 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff5a804c060 con 0x7ff5ac0bf9d0 2024-03-06T07:31:20.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.042+0000 7ff5a1ffb640 1 --2- 172.21.15.1:0/3321966650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff59006b4d0 0x7ff59006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.042+0000 7ff5a1ffb640 1 -- 172.21.15.1:0/3321966650 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ff5a80ac120 con 0x7ff5ac0bf9d0 2024-03-06T07:31:20.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.042+0000 7ff5b0be9640 1 --2- 172.21.15.1:0/3321966650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff59006b4d0 0x7ff59006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.043+0000 7ff5b0be9640 1 --2- 172.21.15.1:0/3321966650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff59006b4d0 0x7ff59006d990 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac06d050 tx=0x7ff59c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.046+0000 7ff5a1ffb640 1 -- 172.21.15.1:0/3321966650 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5a8040070 con 0x7ff5ac0bf9d0 2024-03-06T07:31:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.267+0000 7ff5b364e640 1 -- 172.21.15.1:0/3321966650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5ac0b01a0 con 0x7ff5ac0bf9d0 2024-03-06T07:31:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.267+0000 7ff5a1ffb640 1 -- 172.21.15.1:0/3321966650 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7ff5a807ad00 con 0x7ff5ac0bf9d0 2024-03-06T07:31:20.269 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:20.270 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:20.270 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:31:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.269+0000 7ff5b364e640 1 -- 172.21.15.1:0/3321966650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff59006b4d0 msgr2=0x7ff59006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.269+0000 7ff5b364e640 1 --2- 172.21.15.1:0/3321966650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff59006b4d0 0x7ff59006d990 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac06d050 tx=0x7ff59c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.269+0000 7ff5b364e640 1 -- 172.21.15.1:0/3321966650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5ac0bf9d0 msgr2=0x7ff5ac1eed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.269+0000 7ff5b364e640 1 --2- 172.21.15.1:0/3321966650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5ac0bf9d0 0x7ff5ac1eed80 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7ff5a802b8c0 tx=0x7ff5a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.270+0000 7ff5b364e640 1 -- 172.21.15.1:0/3321966650 shutdown_connections 2024-03-06T07:31:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.270+0000 7ff5b364e640 1 --2- 172.21.15.1:0/3321966650 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff59006b4d0 0x7ff59006d990 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.270+0000 7ff5b364e640 1 --2- 172.21.15.1:0/3321966650 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5ac14d340 0x7ff5ac1ec230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.270+0000 7ff5b364e640 1 --2- 172.21.15.1:0/3321966650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5ac152650 0x7ff5ac14e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.270+0000 7ff5b364e640 1 --2- 172.21.15.1:0/3321966650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5ac0bf9d0 0x7ff5ac1eed80 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.270+0000 7ff5b364e640 1 -- 172.21.15.1:0/3321966650 >> 172.21.15.1:0/3321966650 conn(0x7ff5ac066ee0 msgr2=0x7ff5ac0c5250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.270+0000 7ff5b364e640 1 -- 172.21.15.1:0/3321966650 shutdown_connections 2024-03-06T07:31:20.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:20.270+0000 7ff5b364e640 1 -- 172.21.15.1:0/3321966650 wait complete. 2024-03-06T07:31:20.685 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:20.685 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:21.686 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:21.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.786+0000 7efdb715e640 1 Processor -- start 2024-03-06T07:31:21.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.786+0000 7efdb715e640 1 -- start start 2024-03-06T07:31:21.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.787+0000 7efdb715e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efdb0159140 0x7efdb00c4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.787+0000 7efdb715e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efdb00c69b0 0x7efdb00c2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.787+0000 7efdb715e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb0072d00 con 0x7efdb00c7510 2024-03-06T07:31:21.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.787+0000 7efdb715e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb00c0e80 con 0x7efdb00c69b0 2024-03-06T07:31:21.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.787+0000 7efdb715e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb00c13b0 con 0x7efdb0159140 2024-03-06T07:31:21.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.787+0000 7efda7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efdb00c69b0 0x7efdb00c2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.787+0000 7efda7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efdb00c69b0 0x7efdb00c2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37784/0 (socket says 172.21.15.1:37784) 2024-03-06T07:31:21.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.787+0000 7efda7fff640 1 -- 172.21.15.1:0/3984171630 learned_addr learned my addr 172.21.15.1:0/3984171630 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:21.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.787+0000 7efdb4efa640 1 --2- 172.21.15.1:0/3984171630 >> v2:172.21.15.98:3300/0 conn(0x7efdb0159140 0x7efdb00c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.787+0000 7efda7fff640 1 -- 172.21.15.1:0/3984171630 >> v2:172.21.15.98:3300/0 conn(0x7efdb0159140 msgr2=0x7efdb00c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.787+0000 7efda7fff640 1 --2- 172.21.15.1:0/3984171630 >> v2:172.21.15.98:3300/0 conn(0x7efdb0159140 0x7efdb00c4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.788+0000 7efda7fff640 1 -- 172.21.15.1:0/3984171630 >> v1:172.21.15.98:6789/0 conn(0x7efdb00c7510 legacy=0x7efdb006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.788+0000 7efda7fff640 1 -- 172.21.15.1:0/3984171630 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb0073180 con 0x7efdb00c69b0 2024-03-06T07:31:21.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.789+0000 7efda7fff640 1 --2- 172.21.15.1:0/3984171630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdb00c69b0 0x7efdb00c2b30 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7efdb0072850 tx=0x7efda002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d2efcedfb034662 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.789+0000 7efda77fe640 1 -- 172.21.15.1:0/3984171630 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda0035020 con 0x7efdb00c69b0 2024-03-06T07:31:21.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.789+0000 7efda77fe640 1 -- 172.21.15.1:0/3984171630 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda0031870 con 0x7efdb00c69b0 2024-03-06T07:31:21.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.789+0000 7efda77fe640 1 -- 172.21.15.1:0/3984171630 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda0031c40 con 0x7efdb00c69b0 2024-03-06T07:31:21.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.789+0000 7efdb715e640 1 -- 172.21.15.1:0/3984171630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdb00c69b0 msgr2=0x7efdb00c2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.789+0000 7efdb715e640 1 --2- 172.21.15.1:0/3984171630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdb00c69b0 0x7efdb00c2b30 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7efdb0072850 tx=0x7efda002b910 comp rx=0 tx=0).stop 2024-03-06T07:31:21.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.789+0000 7efdb715e640 1 -- 172.21.15.1:0/3984171630 shutdown_connections 2024-03-06T07:31:21.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.789+0000 7efdb715e640 1 --2- 172.21.15.1:0/3984171630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdb00c69b0 0x7efdb00c2b30 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.789+0000 7efdb715e640 1 --2- 172.21.15.1:0/3984171630 >> v2:172.21.15.98:3300/0 conn(0x7efdb0159140 0x7efdb00c4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.789+0000 7efdb715e640 1 -- 172.21.15.1:0/3984171630 >> 172.21.15.1:0/3984171630 conn(0x7efdb00c7a50 msgr2=0x7efdb00c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.789+0000 7efdb715e640 1 -- 172.21.15.1:0/3984171630 shutdown_connections 2024-03-06T07:31:21.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.789+0000 7efdb715e640 1 -- 172.21.15.1:0/3984171630 wait complete. 2024-03-06T07:31:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.790+0000 7efdb715e640 1 Processor -- start 2024-03-06T07:31:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.791+0000 7efdb715e640 1 -- start start 2024-03-06T07:31:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.791+0000 7efdb715e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdb00c69b0 0x7efdb01eb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.791+0000 7efdb715e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdb00c7510 0x7efdb014ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.791+0000 7efdb715e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdb0159140 0x7efdb014cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.791+0000 7efdb715e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb00c10d0 con 0x7efdb00c69b0 2024-03-06T07:31:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.791+0000 7efdb715e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb00c6520 con 0x7efdb00c7510 2024-03-06T07:31:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.791+0000 7efdb715e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb0159520 con 0x7efdb0159140 2024-03-06T07:31:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.791+0000 7efdb4efa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdb00c69b0 0x7efdb01eb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.791+0000 7efdb4efa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdb00c69b0 0x7efdb01eb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37786/0 (socket says 172.21.15.1:37786) 2024-03-06T07:31:21.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.791+0000 7efdb4efa640 1 -- 172.21.15.1:0/3983157337 learned_addr learned my addr 172.21.15.1:0/3983157337 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:21.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.792+0000 7efdb56fb640 1 --2- 172.21.15.1:0/3983157337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdb0159140 0x7efdb014cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.792+0000 7efda7fff640 1 --2- 172.21.15.1:0/3983157337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdb00c7510 0x7efdb014ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.792+0000 7efdb4efa640 1 -- 172.21.15.1:0/3983157337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdb0159140 msgr2=0x7efdb014cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.792+0000 7efdb4efa640 1 --2- 172.21.15.1:0/3983157337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdb0159140 0x7efdb014cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.792+0000 7efdb4efa640 1 -- 172.21.15.1:0/3983157337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdb00c7510 msgr2=0x7efdb014ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.792+0000 7efdb4efa640 1 --2- 172.21.15.1:0/3983157337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdb00c7510 0x7efdb014ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.792+0000 7efdb4efa640 1 -- 172.21.15.1:0/3983157337 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb0154390 con 0x7efdb00c69b0 2024-03-06T07:31:21.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.792+0000 7efda7fff640 1 --2- 172.21.15.1:0/3983157337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdb00c7510 0x7efdb014ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:21.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.792+0000 7efdb56fb640 1 --2- 172.21.15.1:0/3983157337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdb0159140 0x7efdb014cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:21.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.792+0000 7efdb4efa640 1 --2- 172.21.15.1:0/3983157337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdb00c69b0 0x7efdb01eb120 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7efdac00d600 tx=0x7efdac00dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.792+0000 7efda57fa640 1 -- 172.21.15.1:0/3983157337 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac0125f0 con 0x7efdb00c69b0 2024-03-06T07:31:21.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.792+0000 7efda57fa640 1 -- 172.21.15.1:0/3983157337 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdac019070 con 0x7efdb00c69b0 2024-03-06T07:31:21.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.792+0000 7efdb715e640 1 -- 172.21.15.1:0/3983157337 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb00a8360 con 0x7efdb00c69b0 2024-03-06T07:31:21.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.792+0000 7efda57fa640 1 -- 172.21.15.1:0/3983157337 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac0129f0 con 0x7efdb00c69b0 2024-03-06T07:31:21.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.792+0000 7efdb715e640 1 -- 172.21.15.1:0/3983157337 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb00a8920 con 0x7efdb00c69b0 2024-03-06T07:31:21.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.793+0000 7efdb715e640 1 -- 172.21.15.1:0/3983157337 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb00686e0 con 0x7efdb00c69b0 2024-03-06T07:31:21.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.794+0000 7efda57fa640 1 -- 172.21.15.1:0/3983157337 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efdac012b90 con 0x7efdb00c69b0 2024-03-06T07:31:21.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.794+0000 7efda57fa640 1 --2- 172.21.15.1:0/3983157337 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efd9c06b4d0 0x7efd9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.795+0000 7efda57fa640 1 -- 172.21.15.1:0/3983157337 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7efdac08a840 con 0x7efdb00c69b0 2024-03-06T07:31:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.795+0000 7efda7fff640 1 --2- 172.21.15.1:0/3983157337 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efd9c06b4d0 0x7efd9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.795+0000 7efda7fff640 1 --2- 172.21.15.1:0/3983157337 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efd9c06b4d0 0x7efd9c06d990 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7efda0036000 tx=0x7efda0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:21.799+0000 7efda57fa640 1 -- 172.21.15.1:0/3983157337 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efdac0594c0 con 0x7efdb00c69b0 2024-03-06T07:31:22.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:22.018+0000 7efdb715e640 1 -- 172.21.15.1:0/3983157337 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdb0072f60 con 0x7efdb00c69b0 2024-03-06T07:31:22.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:22.018+0000 7efda57fa640 1 -- 172.21.15.1:0/3983157337 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7efdac058dc0 con 0x7efdb00c69b0 2024-03-06T07:31:22.021 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:22.021 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:31:22.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:22.020+0000 7efdb715e640 1 -- 172.21.15.1:0/3983157337 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efd9c06b4d0 msgr2=0x7efd9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:22.020+0000 7efdb715e640 1 --2- 172.21.15.1:0/3983157337 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efd9c06b4d0 0x7efd9c06d990 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7efda0036000 tx=0x7efda0008030 comp rx=0 tx=0).stop 2024-03-06T07:31:22.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:22.020+0000 7efdb715e640 1 -- 172.21.15.1:0/3983157337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdb00c69b0 msgr2=0x7efdb01eb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:22.020+0000 7efdb715e640 1 --2- 172.21.15.1:0/3983157337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdb00c69b0 0x7efdb01eb120 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7efdac00d600 tx=0x7efdac00dac0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:22.020+0000 7efdb715e640 1 -- 172.21.15.1:0/3983157337 shutdown_connections 2024-03-06T07:31:22.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:22.020+0000 7efdb715e640 1 --2- 172.21.15.1:0/3983157337 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7efd9c06b4d0 0x7efd9c06d990 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:22.020+0000 7efdb715e640 1 --2- 172.21.15.1:0/3983157337 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdb0159140 0x7efdb014cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:22.020+0000 7efdb715e640 1 --2- 172.21.15.1:0/3983157337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdb00c7510 0x7efdb014ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:22.020+0000 7efdb715e640 1 --2- 172.21.15.1:0/3983157337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdb00c69b0 0x7efdb01eb120 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:22.020+0000 7efdb715e640 1 -- 172.21.15.1:0/3983157337 >> 172.21.15.1:0/3983157337 conn(0x7efdb00c7a50 msgr2=0x7efdb006e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:22.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:22.020+0000 7efdb715e640 1 -- 172.21.15.1:0/3983157337 shutdown_connections 2024-03-06T07:31:22.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:22.020+0000 7efdb715e640 1 -- 172.21.15.1:0/3983157337 wait complete. 2024-03-06T07:31:22.435 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:22.435 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:23.436 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:23.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.536+0000 7faa2f4f4640 1 Processor -- start 2024-03-06T07:31:23.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.536+0000 7faa2f4f4640 1 -- start start 2024-03-06T07:31:23.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.537+0000 7faa2f4f4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa2806c640 0x7faa2806b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.537+0000 7faa2f4f4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa2814f7d0 con 0x7faa2814f3f0 2024-03-06T07:31:23.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.537+0000 7faa2f4f4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa28152980 con 0x7faa280a9bd0 2024-03-06T07:31:23.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.537+0000 7faa2f4f4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa28152c60 con 0x7faa2806c640 2024-03-06T07:31:23.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.537+0000 7faa2d290640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7faa2814f3f0 0x7faa280abdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52206/0 (socket says 172.21.15.1:52206) 2024-03-06T07:31:23.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.537+0000 7faa2d290640 1 -- 172.21.15.1:0/421004647 learned_addr learned my addr 172.21.15.1:0/421004647 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:31:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa2ca8f640 1 --2- 172.21.15.1:0/421004647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa2806c640 0x7faa2806b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa1bfff640 1 -- 172.21.15.1:0/421004647 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405255957 0 0) 0x7faa2814f7d0 con 0x7faa2814f3f0 2024-03-06T07:31:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa1bfff640 1 -- 172.21.15.1:0/421004647 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa14003520 con 0x7faa2814f3f0 2024-03-06T07:31:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa1bfff640 1 -- 172.21.15.1:0/421004647 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1995343083 0 0) 0x7faa14003520 con 0x7faa2814f3f0 2024-03-06T07:31:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa1bfff640 1 -- 172.21.15.1:0/421004647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa2806c640 msgr2=0x7faa2806b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa1bfff640 1 --2- 172.21.15.1:0/421004647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa2806c640 0x7faa2806b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa1bfff640 1 -- 172.21.15.1:0/421004647 >> v1:172.21.15.98:6789/0 conn(0x7faa280a9bd0 legacy=0x7faa2806b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa1bfff640 1 -- 172.21.15.1:0/421004647 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2806bd40 con 0x7faa2814f3f0 2024-03-06T07:31:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa1bfff640 1 -- 172.21.15.1:0/421004647 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7faa24001670 con 0x7faa2814f3f0 2024-03-06T07:31:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa1bfff640 1 -- 172.21.15.1:0/421004647 >> v1:172.21.15.1:6789/0 conn(0x7faa2814f3f0 legacy=0x7faa280abdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa1bfff640 1 --2- 172.21.15.1:0/421004647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa140039e0 0x7faa14003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa1bfff640 1 --2- 172.21.15.1:0/421004647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa2806c640 0x7faa14004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa1bfff640 1 --2- 172.21.15.1:0/421004647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa14004d40 0x7faa140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa1bfff640 1 -- 172.21.15.1:0/421004647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa28152c60 con 0x7faa14004d40 2024-03-06T07:31:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa1bfff640 1 -- 172.21.15.1:0/421004647 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1400dc80 con 0x7faa140039e0 2024-03-06T07:31:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa1bfff640 1 -- 172.21.15.1:0/421004647 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1400de80 con 0x7faa2806c640 2024-03-06T07:31:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa2da91640 1 --2- 172.21.15.1:0/421004647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa14004d40 0x7faa140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa2d290640 1 --2- 172.21.15.1:0/421004647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa140039e0 0x7faa14003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.538+0000 7faa2ca8f640 1 --2- 172.21.15.1:0/421004647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa2806c640 0x7faa14004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.539+0000 7faa2da91640 1 -- 172.21.15.1:0/421004647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa2806c640 msgr2=0x7faa14004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.539+0000 7faa2da91640 1 --2- 172.21.15.1:0/421004647 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa2806c640 0x7faa14004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.539+0000 7faa2da91640 1 -- 172.21.15.1:0/421004647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa140039e0 msgr2=0x7faa14003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.539+0000 7faa2da91640 1 --2- 172.21.15.1:0/421004647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa140039e0 0x7faa14003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.539+0000 7faa2da91640 1 -- 172.21.15.1:0/421004647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa14003520 con 0x7faa14004d40 2024-03-06T07:31:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.539+0000 7faa2da91640 1 --2- 172.21.15.1:0/421004647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa14004d40 0x7faa140051c0 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7faa20001e20 tx=0x7faa14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b1e9a2dffbe78c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.539+0000 7faa1bfff640 1 -- 172.21.15.1:0/421004647 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa20001a70 con 0x7faa14004d40 2024-03-06T07:31:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.540+0000 7faa1bfff640 1 -- 172.21.15.1:0/421004647 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa20003b70 con 0x7faa14004d40 2024-03-06T07:31:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.540+0000 7faa1bfff640 1 -- 172.21.15.1:0/421004647 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa200050d0 con 0x7faa14004d40 2024-03-06T07:31:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.540+0000 7faa2f4f4640 1 -- 172.21.15.1:0/421004647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa14004d40 msgr2=0x7faa140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.540+0000 7faa2f4f4640 1 --2- 172.21.15.1:0/421004647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa14004d40 0x7faa140051c0 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7faa20001e20 tx=0x7faa14005830 comp rx=0 tx=0).stop 2024-03-06T07:31:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.540+0000 7faa2da91640 1 -- 172.21.15.1:0/421004647 reap_dead start 2024-03-06T07:31:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.540+0000 7faa2f4f4640 1 -- 172.21.15.1:0/421004647 shutdown_connections 2024-03-06T07:31:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.540+0000 7faa2f4f4640 1 -- 172.21.15.1:0/421004647 >> 172.21.15.1:0/421004647 conn(0x7faa280c71c0 msgr2=0x7faa280c75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.540+0000 7faa2f4f4640 1 -- 172.21.15.1:0/421004647 shutdown_connections 2024-03-06T07:31:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.540+0000 7faa2f4f4640 1 -- 172.21.15.1:0/421004647 wait complete. 2024-03-06T07:31:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.541+0000 7faa2f4f4640 1 Processor -- start 2024-03-06T07:31:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.541+0000 7faa2f4f4640 1 -- start start 2024-03-06T07:31:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.542+0000 7faa2f4f4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa14004d40 0x7faa28146810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.542+0000 7faa2f4f4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa28152ea0 0x7faa28148b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.542+0000 7faa2f4f4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa280c4b60 0x7faa281e6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.542+0000 7faa2f4f4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa280c1f30 con 0x7faa28152ea0 2024-03-06T07:31:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.542+0000 7faa2f4f4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa280c1bc0 con 0x7faa280c4b60 2024-03-06T07:31:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.542+0000 7faa2f4f4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa280ac500 con 0x7faa14004d40 2024-03-06T07:31:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.542+0000 7faa2ca8f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa28152ea0 0x7faa28148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.542+0000 7faa2ca8f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa28152ea0 0x7faa28148b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37800/0 (socket says 172.21.15.1:37800) 2024-03-06T07:31:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.542+0000 7faa2ca8f640 1 -- 172.21.15.1:0/255920013 learned_addr learned my addr 172.21.15.1:0/255920013 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.542+0000 7faa2d290640 1 --2- 172.21.15.1:0/255920013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa14004d40 0x7faa28146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.542+0000 7faa2da91640 1 --2- 172.21.15.1:0/255920013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa280c4b60 0x7faa281e6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.542+0000 7faa2ca8f640 1 -- 172.21.15.1:0/255920013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa14004d40 msgr2=0x7faa28146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.542+0000 7faa2ca8f640 1 --2- 172.21.15.1:0/255920013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa14004d40 0x7faa28146810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.542+0000 7faa2ca8f640 1 -- 172.21.15.1:0/255920013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa280c4b60 msgr2=0x7faa281e6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.542+0000 7faa2ca8f640 1 --2- 172.21.15.1:0/255920013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa280c4b60 0x7faa281e6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.542+0000 7faa2ca8f640 1 -- 172.21.15.1:0/255920013 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2814e930 con 0x7faa28152ea0 2024-03-06T07:31:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.542+0000 7faa2d290640 1 --2- 172.21.15.1:0/255920013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa14004d40 0x7faa28146810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.543+0000 7faa2ca8f640 1 --2- 172.21.15.1:0/255920013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa28152ea0 0x7faa28148b50 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7faa1c011970 tx=0x7faa1c011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.543+0000 7faa19ffb640 1 -- 172.21.15.1:0/255920013 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c019070 con 0x7faa28152ea0 2024-03-06T07:31:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.543+0000 7faa19ffb640 1 -- 172.21.15.1:0/255920013 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1c004240 con 0x7faa28152ea0 2024-03-06T07:31:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.543+0000 7faa2f4f4640 1 -- 172.21.15.1:0/255920013 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa28149950 con 0x7faa28152ea0 2024-03-06T07:31:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.543+0000 7faa19ffb640 1 -- 172.21.15.1:0/255920013 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c00f880 con 0x7faa28152ea0 2024-03-06T07:31:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.543+0000 7faa2f4f4640 1 -- 172.21.15.1:0/255920013 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa2814e650 con 0x7faa28152ea0 2024-03-06T07:31:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.544+0000 7faa2f4f4640 1 -- 172.21.15.1:0/255920013 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa280686e0 con 0x7faa28152ea0 2024-03-06T07:31:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.544+0000 7faa19ffb640 1 -- 172.21.15.1:0/255920013 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa1c01c070 con 0x7faa28152ea0 2024-03-06T07:31:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.545+0000 7faa19ffb640 1 --2- 172.21.15.1:0/255920013 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faa0c06b500 0x7faa0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.545+0000 7faa19ffb640 1 -- 172.21.15.1:0/255920013 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7faa1c08a770 con 0x7faa28152ea0 2024-03-06T07:31:23.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.545+0000 7faa2d290640 1 --2- 172.21.15.1:0/255920013 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faa0c06b500 0x7faa0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.546+0000 7faa2d290640 1 --2- 172.21.15.1:0/255920013 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faa0c06b500 0x7faa0c06d9c0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7faa24000fb0 tx=0x7faa24009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.549+0000 7faa19ffb640 1 -- 172.21.15.1:0/255920013 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa1c0593f0 con 0x7faa28152ea0 2024-03-06T07:31:23.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.768+0000 7faa2f4f4640 1 -- 172.21.15.1:0/255920013 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa2814f7d0 con 0x7faa28152ea0 2024-03-06T07:31:23.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.768+0000 7faa19ffb640 1 -- 172.21.15.1:0/255920013 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3425 (secure 0 0 0) 0x7faa1c058cf0 con 0x7faa28152ea0 2024-03-06T07:31:23.771 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:23.771 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:30:53.322867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:23.771 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 212 2024-03-06T07:31:23.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.770+0000 7faa2f4f4640 1 -- 172.21.15.1:0/255920013 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faa0c06b500 msgr2=0x7faa0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.770+0000 7faa2f4f4640 1 --2- 172.21.15.1:0/255920013 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faa0c06b500 0x7faa0c06d9c0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7faa24000fb0 tx=0x7faa24009040 comp rx=0 tx=0).stop 2024-03-06T07:31:23.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.770+0000 7faa2f4f4640 1 -- 172.21.15.1:0/255920013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa28152ea0 msgr2=0x7faa28148b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.770+0000 7faa2f4f4640 1 --2- 172.21.15.1:0/255920013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa28152ea0 0x7faa28148b50 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7faa1c011970 tx=0x7faa1c011e30 comp rx=0 tx=0).stop 2024-03-06T07:31:23.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.770+0000 7faa2f4f4640 1 -- 172.21.15.1:0/255920013 shutdown_connections 2024-03-06T07:31:23.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.770+0000 7faa2f4f4640 1 --2- 172.21.15.1:0/255920013 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faa0c06b500 0x7faa0c06d9c0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.770+0000 7faa2f4f4640 1 --2- 172.21.15.1:0/255920013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa280c4b60 0x7faa281e6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.770+0000 7faa2f4f4640 1 --2- 172.21.15.1:0/255920013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa28152ea0 0x7faa28148b50 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.770+0000 7faa2f4f4640 1 --2- 172.21.15.1:0/255920013 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa14004d40 0x7faa28146810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.770+0000 7faa2f4f4640 1 -- 172.21.15.1:0/255920013 >> 172.21.15.1:0/255920013 conn(0x7faa280c71c0 msgr2=0x7faa2806f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.770+0000 7faa2f4f4640 1 -- 172.21.15.1:0/255920013 shutdown_connections 2024-03-06T07:31:23.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:23.770+0000 7faa2f4f4640 1 -- 172.21.15.1:0/255920013 wait complete. 2024-03-06T07:31:24.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:30:53.322867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:24.189 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:25.191 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:25.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.304+0000 7faaf3fee640 1 Processor -- start 2024-03-06T07:31:25.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.304+0000 7faaf3fee640 1 -- start start 2024-03-06T07:31:25.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.305+0000 7faaf3fee640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faaec152650 0x7faaec155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.305+0000 7faaf3fee640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faaec0c6b10 0x7faaec158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.305+0000 7faaf3fee640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faaec0bf9d0 0x7faaec0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.305+0000 7faaf3fee640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaec155800 con 0x7faaec0c6b10 2024-03-06T07:31:25.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.305+0000 7faaf3fee640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaec0c2fc0 con 0x7faaec0bf9d0 2024-03-06T07:31:25.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.305+0000 7faaf3fee640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaec0c6690 con 0x7faaec152650 2024-03-06T07:31:25.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.305+0000 7faaf1589640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faaec0c6b10 0x7faaec158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.305+0000 7faaf1589640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faaec0c6b10 0x7faaec158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56674/0 (socket says 172.21.15.1:56674) 2024-03-06T07:31:25.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.305+0000 7faaf1589640 1 -- 172.21.15.1:0/1777947315 learned_addr learned my addr 172.21.15.1:0/1777947315 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:25.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.305+0000 7faaf1d8a640 1 --2- 172.21.15.1:0/1777947315 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faaec152650 0x7faaec155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.305+0000 7faaf258b640 1 --2- 172.21.15.1:0/1777947315 >> v2:172.21.15.98:3300/0 conn(0x7faaec0bf9d0 0x7faaec0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.306+0000 7faaf1589640 1 -- 172.21.15.1:0/1777947315 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faaec152650 msgr2=0x7faaec155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:25.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.306+0000 7faaf1589640 1 --2- 172.21.15.1:0/1777947315 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faaec152650 0x7faaec155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.306+0000 7faaf1589640 1 -- 172.21.15.1:0/1777947315 >> v2:172.21.15.98:3300/0 conn(0x7faaec0bf9d0 msgr2=0x7faaec0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:25.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.306+0000 7faaf1589640 1 --2- 172.21.15.1:0/1777947315 >> v2:172.21.15.98:3300/0 conn(0x7faaec0bf9d0 0x7faaec0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.306+0000 7faaf1589640 1 -- 172.21.15.1:0/1777947315 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec0c7280 con 0x7faaec0c6b10 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.306+0000 7faaf1589640 1 --2- 172.21.15.1:0/1777947315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0c6b10 0x7faaec158690 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7faaec1597b0 tx=0x7faadc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e865524e066b7510 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.306+0000 7faaf0d88640 1 -- 172.21.15.1:0/1777947315 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc02e070 con 0x7faaec0c6b10 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.307+0000 7faaf0d88640 1 -- 172.21.15.1:0/1777947315 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faadc037070 con 0x7faaec0c6b10 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.307+0000 7faaf0d88640 1 -- 172.21.15.1:0/1777947315 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc031a00 con 0x7faaec0c6b10 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.307+0000 7faaf3fee640 1 -- 172.21.15.1:0/1777947315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0c6b10 msgr2=0x7faaec158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.307+0000 7faaf3fee640 1 --2- 172.21.15.1:0/1777947315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0c6b10 0x7faaec158690 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7faaec1597b0 tx=0x7faadc02b880 comp rx=0 tx=0).stop 2024-03-06T07:31:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.307+0000 7faaf3fee640 1 -- 172.21.15.1:0/1777947315 shutdown_connections 2024-03-06T07:31:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.307+0000 7faaf3fee640 1 --2- 172.21.15.1:0/1777947315 >> v2:172.21.15.98:3300/0 conn(0x7faaec0bf9d0 0x7faaec0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.307+0000 7faaf3fee640 1 --2- 172.21.15.1:0/1777947315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0c6b10 0x7faaec158690 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.307+0000 7faaf3fee640 1 --2- 172.21.15.1:0/1777947315 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faaec152650 0x7faaec155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.307+0000 7faaf3fee640 1 -- 172.21.15.1:0/1777947315 >> 172.21.15.1:0/1777947315 conn(0x7faaec066ee0 msgr2=0x7faaec0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.307+0000 7faaf3fee640 1 -- 172.21.15.1:0/1777947315 shutdown_connections 2024-03-06T07:31:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.307+0000 7faaf3fee640 1 -- 172.21.15.1:0/1777947315 wait complete. 2024-03-06T07:31:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.308+0000 7faaf3fee640 1 Processor -- start 2024-03-06T07:31:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.308+0000 7faaf3fee640 1 -- start start 2024-03-06T07:31:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf3fee640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0bf9d0 0x7faaec152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf3fee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faaec152650 0x7faaec14e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf3fee640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faaec154d70 0x7faaec14c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf3fee640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec155490 con 0x7faaec0bf9d0 2024-03-06T07:31:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf3fee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec1551b0 con 0x7faaec152650 2024-03-06T07:31:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf3fee640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaec152a30 con 0x7faaec154d70 2024-03-06T07:31:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf1d8a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0bf9d0 0x7faaec152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf1d8a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0bf9d0 0x7faaec152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56690/0 (socket says 172.21.15.1:56690) 2024-03-06T07:31:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf1d8a640 1 -- 172.21.15.1:0/2711707704 learned_addr learned my addr 172.21.15.1:0/2711707704 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf1589640 1 --2- 172.21.15.1:0/2711707704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faaec152650 0x7faaec14e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf258b640 1 --2- 172.21.15.1:0/2711707704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faaec154d70 0x7faaec14c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf1d8a640 1 -- 172.21.15.1:0/2711707704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faaec154d70 msgr2=0x7faaec14c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf1d8a640 1 --2- 172.21.15.1:0/2711707704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faaec154d70 0x7faaec14c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf1d8a640 1 -- 172.21.15.1:0/2711707704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faaec152650 msgr2=0x7faaec14e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf1d8a640 1 --2- 172.21.15.1:0/2711707704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faaec152650 0x7faaec14e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf1d8a640 1 -- 172.21.15.1:0/2711707704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec154890 con 0x7faaec0bf9d0 2024-03-06T07:31:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf258b640 1 --2- 172.21.15.1:0/2711707704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faaec154d70 0x7faaec14c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf1589640 1 --2- 172.21.15.1:0/2711707704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faaec152650 0x7faaec14e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.309+0000 7faaf1d8a640 1 --2- 172.21.15.1:0/2711707704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0bf9d0 0x7faaec152f10 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7faae800d830 tx=0x7faae800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.310+0000 7faae27fc640 1 -- 172.21.15.1:0/2711707704 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae800b6c0 con 0x7faaec0bf9d0 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.310+0000 7faae27fc640 1 -- 172.21.15.1:0/2711707704 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae8019070 con 0x7faaec0bf9d0 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.310+0000 7faaf3fee640 1 -- 172.21.15.1:0/2711707704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaec14db20 con 0x7faaec0bf9d0 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.310+0000 7faae27fc640 1 -- 172.21.15.1:0/2711707704 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae800ba80 con 0x7faaec0bf9d0 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.310+0000 7faaf3fee640 1 -- 172.21.15.1:0/2711707704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaec1ebda0 con 0x7faaec0bf9d0 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.311+0000 7faaf3fee640 1 -- 172.21.15.1:0/2711707704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaec067fd0 con 0x7faaec0bf9d0 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.311+0000 7faae27fc640 1 -- 172.21.15.1:0/2711707704 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faae801e460 con 0x7faaec0bf9d0 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.311+0000 7faae27fc640 1 --2- 172.21.15.1:0/2711707704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faad006b500 0x7faad006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.312+0000 7faae27fc640 1 -- 172.21.15.1:0/2711707704 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7faae808aac0 con 0x7faaec0bf9d0 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.312+0000 7faaf1589640 1 --2- 172.21.15.1:0/2711707704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faad006b500 0x7faad006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.312+0000 7faaf1589640 1 --2- 172.21.15.1:0/2711707704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faad006b500 0x7faad006d9c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7faadc036000 tx=0x7faadc008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.316+0000 7faae27fc640 1 -- 172.21.15.1:0/2711707704 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faae8059740 con 0x7faaec0bf9d0 2024-03-06T07:31:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.538+0000 7faaf3fee640 1 -- 172.21.15.1:0/2711707704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faaec0b06f0 con 0x7faaec0bf9d0 2024-03-06T07:31:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.538+0000 7faae27fc640 1 -- 172.21.15.1:0/2711707704 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7faae8059040 con 0x7faaec0bf9d0 2024-03-06T07:31:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:25.541 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:25.541 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:25.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.540+0000 7faaf3fee640 1 -- 172.21.15.1:0/2711707704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faad006b500 msgr2=0x7faad006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.540+0000 7faaf3fee640 1 --2- 172.21.15.1:0/2711707704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faad006b500 0x7faad006d9c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7faadc036000 tx=0x7faadc008030 comp rx=0 tx=0).stop 2024-03-06T07:31:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.540+0000 7faaf3fee640 1 -- 172.21.15.1:0/2711707704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0bf9d0 msgr2=0x7faaec152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.540+0000 7faaf3fee640 1 --2- 172.21.15.1:0/2711707704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0bf9d0 0x7faaec152f10 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7faae800d830 tx=0x7faae800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.540+0000 7faaf3fee640 1 -- 172.21.15.1:0/2711707704 shutdown_connections 2024-03-06T07:31:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.540+0000 7faaf3fee640 1 --2- 172.21.15.1:0/2711707704 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7faad006b500 0x7faad006d9c0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.540+0000 7faaf3fee640 1 --2- 172.21.15.1:0/2711707704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faaec154d70 0x7faaec14c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.540+0000 7faaf3fee640 1 --2- 172.21.15.1:0/2711707704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faaec152650 0x7faaec14e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.541+0000 7faaf3fee640 1 --2- 172.21.15.1:0/2711707704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0bf9d0 0x7faaec152f10 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.541+0000 7faaf3fee640 1 -- 172.21.15.1:0/2711707704 >> 172.21.15.1:0/2711707704 conn(0x7faaec066ee0 msgr2=0x7faaec0c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.541+0000 7faaf3fee640 1 -- 172.21.15.1:0/2711707704 shutdown_connections 2024-03-06T07:31:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:25.541+0000 7faaf3fee640 1 -- 172.21.15.1:0/2711707704 wait complete. 2024-03-06T07:31:25.955 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:25.956 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:26.957 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:27.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.056+0000 7f5bec2df640 1 Processor -- start 2024-03-06T07:31:27.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.057+0000 7f5bec2df640 1 -- start start 2024-03-06T07:31:27.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.057+0000 7f5bec2df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be4152650 0x7f5be4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.057+0000 7f5bec2df640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5be40bf9d0 0x7f5be40c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.057+0000 7f5bec2df640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be40c2fc0 con 0x7f5be40c6b10 2024-03-06T07:31:27.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.057+0000 7f5bec2df640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5be4155800 con 0x7f5be40bf9d0 2024-03-06T07:31:27.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.057+0000 7f5bec2df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be4158da0 con 0x7f5be4152650 2024-03-06T07:31:27.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.058+0000 7f5bea07b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be4152650 0x7f5be4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.058+0000 7f5bea87c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5be40bf9d0 0x7f5be40c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.058+0000 7f5bea07b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be4152650 0x7f5be4155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:51664/0 (socket says 172.21.15.1:51664) 2024-03-06T07:31:27.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.058+0000 7f5be987a640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5be40c6b10 0x7f5be4158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:56110/0 (socket says 172.21.15.1:56110) 2024-03-06T07:31:27.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.058+0000 7f5bea07b640 1 -- 172.21.15.1:0/1581597656 learned_addr learned my addr 172.21.15.1:0/1581597656 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:27.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.058+0000 7f5bea87c640 1 --2- 172.21.15.1:0/1581597656 >> v2:172.21.15.98:3300/0 conn(0x7f5be40bf9d0 0x7f5be40c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:48868/0 (socket says 172.21.15.1:48868) 2024-03-06T07:31:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.058+0000 7f5bea07b640 1 -- 172.21.15.1:0/1581597656 >> v2:172.21.15.98:3300/0 conn(0x7f5be40bf9d0 msgr2=0x7f5be40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.058+0000 7f5bea07b640 1 --2- 172.21.15.1:0/1581597656 >> v2:172.21.15.98:3300/0 conn(0x7f5be40bf9d0 0x7f5be40c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.058+0000 7f5bea07b640 1 -- 172.21.15.1:0/1581597656 >> v1:172.21.15.98:6789/0 conn(0x7f5be40c6b10 legacy=0x7f5be4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.058+0000 7f5bea07b640 1 -- 172.21.15.1:0/1581597656 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be40c6690 con 0x7f5be4152650 2024-03-06T07:31:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.058+0000 7f5bea87c640 1 --2- 172.21.15.1:0/1581597656 >> v2:172.21.15.98:3300/0 conn(0x7f5be40bf9d0 0x7f5be40c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.059+0000 7f5bea07b640 1 --2- 172.21.15.1:0/1581597656 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be4152650 0x7f5be4155c90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8009870 tx=0x7f5bd802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19cf41cc849690b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.059+0000 7f5be9079640 1 -- 172.21.15.1:0/1581597656 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8035020 con 0x7f5be4152650 2024-03-06T07:31:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.059+0000 7f5be9079640 1 -- 172.21.15.1:0/1581597656 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd8031640 con 0x7f5be4152650 2024-03-06T07:31:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.059+0000 7f5be9079640 1 -- 172.21.15.1:0/1581597656 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd802d040 con 0x7f5be4152650 2024-03-06T07:31:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.060+0000 7f5bec2df640 1 -- 172.21.15.1:0/1581597656 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be4152650 msgr2=0x7f5be4155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.060+0000 7f5bec2df640 1 --2- 172.21.15.1:0/1581597656 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be4152650 0x7f5be4155c90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8009870 tx=0x7f5bd802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.060+0000 7f5bec2df640 1 -- 172.21.15.1:0/1581597656 shutdown_connections 2024-03-06T07:31:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.060+0000 7f5bec2df640 1 --2- 172.21.15.1:0/1581597656 >> v2:172.21.15.98:3300/0 conn(0x7f5be40bf9d0 0x7f5be40c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.060+0000 7f5bec2df640 1 --2- 172.21.15.1:0/1581597656 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be4152650 0x7f5be4155c90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.060+0000 7f5bec2df640 1 -- 172.21.15.1:0/1581597656 >> 172.21.15.1:0/1581597656 conn(0x7f5be4066ee0 msgr2=0x7f5be40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.060+0000 7f5bec2df640 1 -- 172.21.15.1:0/1581597656 shutdown_connections 2024-03-06T07:31:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.060+0000 7f5bec2df640 1 -- 172.21.15.1:0/1581597656 wait complete. 2024-03-06T07:31:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.061+0000 7f5bec2df640 1 Processor -- start 2024-03-06T07:31:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.061+0000 7f5bec2df640 1 -- start start 2024-03-06T07:31:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.062+0000 7f5bec2df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be40bf9d0 0x7f5be4158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bec2df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be4152650 0x7f5be40c5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bec2df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be41eab90 0x7f5be41ee780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bec2df640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be41551b0 con 0x7f5be41eab90 2024-03-06T07:31:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bec2df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be40c6fa0 con 0x7f5be40bf9d0 2024-03-06T07:31:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bec2df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be4155490 con 0x7f5be4152650 2024-03-06T07:31:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bea87c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be41eab90 0x7f5be41ee780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bea87c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be41eab90 0x7f5be41ee780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56698/0 (socket says 172.21.15.1:56698) 2024-03-06T07:31:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bea87c640 1 -- 172.21.15.1:0/2481529187 learned_addr learned my addr 172.21.15.1:0/2481529187 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5be987a640 1 --2- 172.21.15.1:0/2481529187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be4152650 0x7f5be40c5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bea07b640 1 --2- 172.21.15.1:0/2481529187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be40bf9d0 0x7f5be4158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bea87c640 1 -- 172.21.15.1:0/2481529187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be4152650 msgr2=0x7f5be40c5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bea87c640 1 --2- 172.21.15.1:0/2481529187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be4152650 0x7f5be40c5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bea87c640 1 -- 172.21.15.1:0/2481529187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be40bf9d0 msgr2=0x7f5be4158680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bea87c640 1 --2- 172.21.15.1:0/2481529187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be40bf9d0 0x7f5be4158680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bea87c640 1 -- 172.21.15.1:0/2481529187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be41ee2a0 con 0x7f5be41eab90 2024-03-06T07:31:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bea87c640 1 --2- 172.21.15.1:0/2481529187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be41eab90 0x7f5be41ee780 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc00e880 tx=0x7f5bdc00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bd6ffd640 1 -- 172.21.15.1:0/2481529187 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc017070 con 0x7f5be41eab90 2024-03-06T07:31:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bd6ffd640 1 -- 172.21.15.1:0/2481529187 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bdc004020 con 0x7f5be41eab90 2024-03-06T07:31:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bec2df640 1 -- 172.21.15.1:0/2481529187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be4154a70 con 0x7f5be41eab90 2024-03-06T07:31:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.063+0000 7f5bd6ffd640 1 -- 172.21.15.1:0/2481529187 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc0043f0 con 0x7f5be41eab90 2024-03-06T07:31:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.064+0000 7f5bec2df640 1 -- 172.21.15.1:0/2481529187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be40c60d0 con 0x7f5be41eab90 2024-03-06T07:31:27.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.065+0000 7f5bec2df640 1 -- 172.21.15.1:0/2481529187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be406cf30 con 0x7f5be41eab90 2024-03-06T07:31:27.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.065+0000 7f5bd6ffd640 1 -- 172.21.15.1:0/2481529187 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5bdc00c940 con 0x7f5be41eab90 2024-03-06T07:31:27.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.065+0000 7f5bd6ffd640 1 --2- 172.21.15.1:0/2481529187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bc806b500 0x7f5bc806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.066+0000 7f5bd6ffd640 1 -- 172.21.15.1:0/2481529187 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f5bdc08a490 con 0x7f5be41eab90 2024-03-06T07:31:27.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.066+0000 7f5bea07b640 1 --2- 172.21.15.1:0/2481529187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bc806b500 0x7f5bc806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.066+0000 7f5bea07b640 1 --2- 172.21.15.1:0/2481529187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bc806b500 0x7f5bc806d9c0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8036000 tx=0x7f5bd803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.070+0000 7f5bd6ffd640 1 -- 172.21.15.1:0/2481529187 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5be406cf30 con 0x7f5be41eab90 2024-03-06T07:31:27.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.302+0000 7f5bec2df640 1 -- 172.21.15.1:0/2481529187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5be41ee470 con 0x7f5be41eab90 2024-03-06T07:31:27.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.303+0000 7f5bd6ffd640 1 -- 172.21.15.1:0/2481529187 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f5bdc058ac0 con 0x7f5be41eab90 2024-03-06T07:31:27.305 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:27.305 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:27.305 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:27.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.304+0000 7f5bec2df640 1 -- 172.21.15.1:0/2481529187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bc806b500 msgr2=0x7f5bc806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.304+0000 7f5bec2df640 1 --2- 172.21.15.1:0/2481529187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bc806b500 0x7f5bc806d9c0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8036000 tx=0x7f5bd803c040 comp rx=0 tx=0).stop 2024-03-06T07:31:27.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.304+0000 7f5bec2df640 1 -- 172.21.15.1:0/2481529187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be41eab90 msgr2=0x7f5be41ee780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.304+0000 7f5bec2df640 1 --2- 172.21.15.1:0/2481529187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be41eab90 0x7f5be41ee780 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc00e880 tx=0x7f5bdc00ed40 comp rx=0 tx=0).stop 2024-03-06T07:31:27.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.305+0000 7f5bec2df640 1 -- 172.21.15.1:0/2481529187 shutdown_connections 2024-03-06T07:31:27.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.305+0000 7f5bec2df640 1 --2- 172.21.15.1:0/2481529187 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5bc806b500 0x7f5bc806d9c0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.305+0000 7f5bec2df640 1 --2- 172.21.15.1:0/2481529187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be41eab90 0x7f5be41ee780 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.305+0000 7f5bec2df640 1 --2- 172.21.15.1:0/2481529187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5be4152650 0x7f5be40c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.305+0000 7f5bec2df640 1 --2- 172.21.15.1:0/2481529187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be40bf9d0 0x7f5be4158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.305+0000 7f5bec2df640 1 -- 172.21.15.1:0/2481529187 >> 172.21.15.1:0/2481529187 conn(0x7f5be4066ee0 msgr2=0x7f5be40bff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.305+0000 7f5bec2df640 1 -- 172.21.15.1:0/2481529187 shutdown_connections 2024-03-06T07:31:27.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:27.305+0000 7f5bec2df640 1 -- 172.21.15.1:0/2481529187 wait complete. 2024-03-06T07:31:27.719 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:27.719 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:28.720 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:28.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.820+0000 7f5d957d0640 1 Processor -- start 2024-03-06T07:31:28.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.821+0000 7f5d957d0640 1 -- start start 2024-03-06T07:31:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.821+0000 7f5d957d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d900c3970 0x7f5d900c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.821+0000 7f5d957d0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5d900c60b0 0x7f5d90152130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.821+0000 7f5d957d0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d900c5e40 con 0x7f5d90152670 2024-03-06T07:31:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.821+0000 7f5d957d0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d90152ad0 con 0x7f5d900c60b0 2024-03-06T07:31:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.821+0000 7f5d957d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d900c67f0 con 0x7f5d900c3970 2024-03-06T07:31:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.821+0000 7f5d867fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5d900c60b0 0x7f5d90152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.821+0000 7f5d867fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5d900c60b0 0x7f5d90152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56714/0 (socket says 172.21.15.1:56714) 2024-03-06T07:31:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.821+0000 7f5d867fc640 1 -- 172.21.15.1:0/2088797075 learned_addr learned my addr 172.21.15.1:0/2088797075 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:28.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.822+0000 7f5d8effd640 1 --2- 172.21.15.1:0/2088797075 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d900c3970 0x7f5d900c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.822+0000 7f5d867fc640 1 -- 172.21.15.1:0/2088797075 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d900c3970 msgr2=0x7f5d900c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:28.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.822+0000 7f5d867fc640 1 --2- 172.21.15.1:0/2088797075 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d900c3970 0x7f5d900c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.822+0000 7f5d867fc640 1 -- 172.21.15.1:0/2088797075 >> v1:172.21.15.98:6789/0 conn(0x7f5d90152670 legacy=0x7f5d901553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.822+0000 7f5d867fc640 1 -- 172.21.15.1:0/2088797075 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d900c6a40 con 0x7f5d900c60b0 2024-03-06T07:31:28.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.823+0000 7f5d867fc640 1 --2- 172.21.15.1:0/2088797075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d900c60b0 0x7f5d90152130 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009870 tx=0x7f5d7802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53590c1a28f0299b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.823+0000 7f5d8e7fc640 1 -- 172.21.15.1:0/2088797075 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78034070 con 0x7f5d900c60b0 2024-03-06T07:31:28.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.823+0000 7f5d8e7fc640 1 -- 172.21.15.1:0/2088797075 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d78030840 con 0x7f5d900c60b0 2024-03-06T07:31:28.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.823+0000 7f5d8e7fc640 1 -- 172.21.15.1:0/2088797075 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78030c10 con 0x7f5d900c60b0 2024-03-06T07:31:28.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.824+0000 7f5d957d0640 1 -- 172.21.15.1:0/2088797075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d900c60b0 msgr2=0x7f5d90152130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.824+0000 7f5d957d0640 1 --2- 172.21.15.1:0/2088797075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d900c60b0 0x7f5d90152130 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009870 tx=0x7f5d7802b980 comp rx=0 tx=0).stop 2024-03-06T07:31:28.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.824+0000 7f5d957d0640 1 -- 172.21.15.1:0/2088797075 shutdown_connections 2024-03-06T07:31:28.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.824+0000 7f5d957d0640 1 --2- 172.21.15.1:0/2088797075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d900c60b0 0x7f5d90152130 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.824+0000 7f5d957d0640 1 --2- 172.21.15.1:0/2088797075 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d900c3970 0x7f5d900c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.824+0000 7f5d957d0640 1 -- 172.21.15.1:0/2088797075 >> 172.21.15.1:0/2088797075 conn(0x7f5d900c88f0 msgr2=0x7f5d900c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:28.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.824+0000 7f5d957d0640 1 -- 172.21.15.1:0/2088797075 shutdown_connections 2024-03-06T07:31:28.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.824+0000 7f5d957d0640 1 -- 172.21.15.1:0/2088797075 wait complete. 2024-03-06T07:31:28.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.825+0000 7f5d957d0640 1 Processor -- start 2024-03-06T07:31:28.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.825+0000 7f5d957d0640 1 -- start start 2024-03-06T07:31:28.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d957d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d900c3970 0x7f5d900711a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d957d0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d900c60b0 0x7f5d900a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d957d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d90152670 0x7f5d901ea7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d957d0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d900c72e0 con 0x7f5d900c60b0 2024-03-06T07:31:28.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d957d0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d900c7000 con 0x7f5d900c3970 2024-03-06T07:31:28.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d957d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d900c75c0 con 0x7f5d90152670 2024-03-06T07:31:28.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d867fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d900c60b0 0x7f5d900a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d867fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d900c60b0 0x7f5d900a97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56720/0 (socket says 172.21.15.1:56720) 2024-03-06T07:31:28.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d867fc640 1 -- 172.21.15.1:0/3315198165 learned_addr learned my addr 172.21.15.1:0/3315198165 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:28.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d8effd640 1 --2- 172.21.15.1:0/3315198165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d900c3970 0x7f5d900711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d8f7fe640 1 --2- 172.21.15.1:0/3315198165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d90152670 0x7f5d901ea7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d867fc640 1 -- 172.21.15.1:0/3315198165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d90152670 msgr2=0x7f5d901ea7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d867fc640 1 --2- 172.21.15.1:0/3315198165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d90152670 0x7f5d901ea7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d867fc640 1 -- 172.21.15.1:0/3315198165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d900c3970 msgr2=0x7f5d900711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d867fc640 1 --2- 172.21.15.1:0/3315198165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d900c3970 0x7f5d900711a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d867fc640 1 -- 172.21.15.1:0/3315198165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d901530d0 con 0x7f5d900c60b0 2024-03-06T07:31:28.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d867fc640 1 --2- 172.21.15.1:0/3315198165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d900c60b0 0x7f5d900a97d0 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f5d7802b9c0 tx=0x7f5d78004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.826+0000 7f5d87fff640 1 -- 172.21.15.1:0/3315198165 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78048070 con 0x7f5d900c60b0 2024-03-06T07:31:28.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.827+0000 7f5d87fff640 1 -- 172.21.15.1:0/3315198165 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d78030640 con 0x7f5d900c60b0 2024-03-06T07:31:28.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.827+0000 7f5d957d0640 1 -- 172.21.15.1:0/3315198165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d9014d1a0 con 0x7f5d900c60b0 2024-03-06T07:31:28.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.827+0000 7f5d87fff640 1 -- 172.21.15.1:0/3315198165 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78034040 con 0x7f5d900c60b0 2024-03-06T07:31:28.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.827+0000 7f5d957d0640 1 -- 172.21.15.1:0/3315198165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d900a9d10 con 0x7f5d900c60b0 2024-03-06T07:31:28.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.828+0000 7f5d957d0640 1 -- 172.21.15.1:0/3315198165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d900686e0 con 0x7f5d900c60b0 2024-03-06T07:31:28.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.828+0000 7f5d87fff640 1 -- 172.21.15.1:0/3315198165 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d7804b070 con 0x7f5d900c60b0 2024-03-06T07:31:28.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.829+0000 7f5d87fff640 1 --2- 172.21.15.1:0/3315198165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d7006b500 0x7f5d7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.829+0000 7f5d87fff640 1 -- 172.21.15.1:0/3315198165 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f5d780ac1e0 con 0x7f5d900c60b0 2024-03-06T07:31:28.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.829+0000 7f5d8effd640 1 --2- 172.21.15.1:0/3315198165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d7006b500 0x7f5d7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.830+0000 7f5d8effd640 1 --2- 172.21.15.1:0/3315198165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d7006b500 0x7f5d7006d9c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f5d80006fd0 tx=0x7f5d80008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:28.833+0000 7f5d87fff640 1 -- 172.21.15.1:0/3315198165 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d78030810 con 0x7f5d900c60b0 2024-03-06T07:31:29.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:29.051+0000 7f5d957d0640 1 -- 172.21.15.1:0/3315198165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d900c5840 con 0x7f5d900c60b0 2024-03-06T07:31:29.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:29.052+0000 7f5d87fff640 1 -- 172.21.15.1:0/3315198165 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f5d7807a810 con 0x7f5d900c60b0 2024-03-06T07:31:29.054 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:29.054 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:29.054 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:29.054+0000 7f5d957d0640 1 -- 172.21.15.1:0/3315198165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d7006b500 msgr2=0x7f5d7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:29.054+0000 7f5d957d0640 1 --2- 172.21.15.1:0/3315198165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d7006b500 0x7f5d7006d9c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f5d80006fd0 tx=0x7f5d80008040 comp rx=0 tx=0).stop 2024-03-06T07:31:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:29.054+0000 7f5d957d0640 1 -- 172.21.15.1:0/3315198165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d900c60b0 msgr2=0x7f5d900a97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:29.054+0000 7f5d957d0640 1 --2- 172.21.15.1:0/3315198165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d900c60b0 0x7f5d900a97d0 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f5d7802b9c0 tx=0x7f5d78004730 comp rx=0 tx=0).stop 2024-03-06T07:31:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:29.054+0000 7f5d957d0640 1 -- 172.21.15.1:0/3315198165 shutdown_connections 2024-03-06T07:31:29.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:29.054+0000 7f5d957d0640 1 --2- 172.21.15.1:0/3315198165 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d7006b500 0x7f5d7006d9c0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:29.054+0000 7f5d957d0640 1 --2- 172.21.15.1:0/3315198165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d90152670 0x7f5d901ea7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:29.054+0000 7f5d957d0640 1 --2- 172.21.15.1:0/3315198165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d900c60b0 0x7f5d900a97d0 unknown :-1 s=CLOSED pgs=2513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:29.054+0000 7f5d957d0640 1 --2- 172.21.15.1:0/3315198165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d900c3970 0x7f5d900711a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:29.054+0000 7f5d957d0640 1 -- 172.21.15.1:0/3315198165 >> 172.21.15.1:0/3315198165 conn(0x7f5d900c88f0 msgr2=0x7f5d900aae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:29.054+0000 7f5d957d0640 1 -- 172.21.15.1:0/3315198165 shutdown_connections 2024-03-06T07:31:29.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:29.054+0000 7f5d957d0640 1 -- 172.21.15.1:0/3315198165 wait complete. 2024-03-06T07:31:29.469 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:29.469 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:30.471 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:30.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.570+0000 7feb70fe9640 1 Processor -- start 2024-03-06T07:31:30.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.570+0000 7feb70fe9640 1 -- start start 2024-03-06T07:31:30.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.571+0000 7feb70fe9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb6c0c6730 0x7feb6c0c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.571+0000 7feb70fe9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb6c0c5d10 0x7feb6c156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.571+0000 7feb70fe9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb6c0c3300 con 0x7feb6c152690 2024-03-06T07:31:30.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.571+0000 7feb70fe9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb6c0c35a0 con 0x7feb6c0c6730 2024-03-06T07:31:30.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.571+0000 7feb70fe9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb6c0c3880 con 0x7feb6c0c5d10 2024-03-06T07:31:30.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.571+0000 7feb6a59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7feb6c152690 0x7feb6c155ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41256/0 (socket says 172.21.15.1:41256) 2024-03-06T07:31:30.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.571+0000 7feb6a59c640 1 -- 172.21.15.1:0/2039238936 learned_addr learned my addr 172.21.15.1:0/2039238936 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:31:30.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.571+0000 7feb6ad9d640 1 --2- 172.21.15.1:0/2039238936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb6c0c5d10 0x7feb6c156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.571+0000 7feb69d9b640 1 --2- 172.21.15.1:0/2039238936 >> v2:172.21.15.98:3300/0 conn(0x7feb6c0c6730 0x7feb6c0c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.571+0000 7feb6959a640 1 -- 172.21.15.1:0/2039238936 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189450320 0 0) 0x7feb6c0c3300 con 0x7feb6c152690 2024-03-06T07:31:30.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.571+0000 7feb6959a640 1 -- 172.21.15.1:0/2039238936 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb54003520 con 0x7feb6c152690 2024-03-06T07:31:30.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.571+0000 7feb6959a640 1 -- 172.21.15.1:0/2039238936 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1801850717 0 0) 0x7feb54003520 con 0x7feb6c152690 2024-03-06T07:31:30.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.571+0000 7feb6959a640 1 -- 172.21.15.1:0/2039238936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb6c0c5d10 msgr2=0x7feb6c156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:30.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.571+0000 7feb6959a640 1 --2- 172.21.15.1:0/2039238936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb6c0c5d10 0x7feb6c156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.571+0000 7feb6959a640 1 -- 172.21.15.1:0/2039238936 >> v2:172.21.15.98:3300/0 conn(0x7feb6c0c6730 msgr2=0x7feb6c0c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:30.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.571+0000 7feb6959a640 1 --2- 172.21.15.1:0/2039238936 >> v2:172.21.15.98:3300/0 conn(0x7feb6c0c6730 0x7feb6c0c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.572+0000 7feb6959a640 1 -- 172.21.15.1:0/2039238936 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c0c3b60 con 0x7feb6c152690 2024-03-06T07:31:30.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.572+0000 7feb6959a640 1 -- 172.21.15.1:0/2039238936 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7feb5c001670 con 0x7feb6c152690 2024-03-06T07:31:30.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.572+0000 7feb6959a640 1 -- 172.21.15.1:0/2039238936 >> v1:172.21.15.1:6789/0 conn(0x7feb6c152690 legacy=0x7feb6c155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.572+0000 7feb6959a640 1 --2- 172.21.15.1:0/2039238936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb540039b0 0x7feb54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.572+0000 7feb6959a640 1 --2- 172.21.15.1:0/2039238936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb54004800 0x7feb54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.572+0000 7feb6959a640 1 --2- 172.21.15.1:0/2039238936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb6c0c5d10 0x7feb540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.572+0000 7feb6959a640 1 -- 172.21.15.1:0/2039238936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6c0c35a0 con 0x7feb54004800 2024-03-06T07:31:30.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.572+0000 7feb6959a640 1 -- 172.21.15.1:0/2039238936 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6c0c3880 con 0x7feb540039b0 2024-03-06T07:31:30.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.572+0000 7feb6959a640 1 -- 172.21.15.1:0/2039238936 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb5400b9c0 con 0x7feb6c0c5d10 2024-03-06T07:31:30.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.572+0000 7feb69d9b640 1 --2- 172.21.15.1:0/2039238936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb54004800 0x7feb54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.572+0000 7feb6ad9d640 1 --2- 172.21.15.1:0/2039238936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb6c0c5d10 0x7feb540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.572+0000 7feb6a59c640 1 --2- 172.21.15.1:0/2039238936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb540039b0 0x7feb54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.572+0000 7feb69d9b640 1 -- 172.21.15.1:0/2039238936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb6c0c5d10 msgr2=0x7feb540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:30.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.572+0000 7feb69d9b640 1 --2- 172.21.15.1:0/2039238936 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb6c0c5d10 0x7feb540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.572+0000 7feb69d9b640 1 -- 172.21.15.1:0/2039238936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb540039b0 msgr2=0x7feb54003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:30.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.572+0000 7feb69d9b640 1 --2- 172.21.15.1:0/2039238936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb540039b0 0x7feb54003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.572+0000 7feb69d9b640 1 -- 172.21.15.1:0/2039238936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb54003520 con 0x7feb54004800 2024-03-06T07:31:30.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.573+0000 7feb69d9b640 1 --2- 172.21.15.1:0/2039238936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb54004800 0x7feb54004c80 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7feb60009af0 tx=0x7feb54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6674ec31f2ab8dcb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.573+0000 7feb6959a640 1 -- 172.21.15.1:0/2039238936 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb60039070 con 0x7feb54004800 2024-03-06T07:31:30.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.573+0000 7feb6959a640 1 -- 172.21.15.1:0/2039238936 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6002c710 con 0x7feb54004800 2024-03-06T07:31:30.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.573+0000 7feb6959a640 1 -- 172.21.15.1:0/2039238936 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb60009690 con 0x7feb54004800 2024-03-06T07:31:30.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.573+0000 7feb70fe9640 1 -- 172.21.15.1:0/2039238936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb54004800 msgr2=0x7feb54004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.573+0000 7feb70fe9640 1 --2- 172.21.15.1:0/2039238936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb54004800 0x7feb54004c80 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7feb60009af0 tx=0x7feb54005830 comp rx=0 tx=0).stop 2024-03-06T07:31:30.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.573+0000 7feb6ad9d640 1 -- 172.21.15.1:0/2039238936 reap_dead start 2024-03-06T07:31:30.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.573+0000 7feb70fe9640 1 -- 172.21.15.1:0/2039238936 shutdown_connections 2024-03-06T07:31:30.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.573+0000 7feb70fe9640 1 -- 172.21.15.1:0/2039238936 >> 172.21.15.1:0/2039238936 conn(0x7feb6c06b410 msgr2=0x7feb6c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.574+0000 7feb70fe9640 1 -- 172.21.15.1:0/2039238936 shutdown_connections 2024-03-06T07:31:30.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.574+0000 7feb70fe9640 1 -- 172.21.15.1:0/2039238936 wait complete. 2024-03-06T07:31:30.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.575+0000 7feb70fe9640 1 Processor -- start 2024-03-06T07:31:30.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.575+0000 7feb70fe9640 1 -- start start 2024-03-06T07:31:30.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.575+0000 7feb70fe9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb54004800 0x7feb6c1eb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.575+0000 7feb70fe9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb6c070f50 0x7feb6c14dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb70fe9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb6c14e190 0x7feb6c1ee160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb70fe9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6c152bf0 con 0x7feb6c14e190 2024-03-06T07:31:30.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb70fe9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6c152a70 con 0x7feb54004800 2024-03-06T07:31:30.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb70fe9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb6c156cd0 con 0x7feb6c070f50 2024-03-06T07:31:30.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb6ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb6c14e190 0x7feb6c1ee160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb6ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb6c14e190 0x7feb6c1ee160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56732/0 (socket says 172.21.15.1:56732) 2024-03-06T07:31:30.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb6ad9d640 1 -- 172.21.15.1:0/3678389362 learned_addr learned my addr 172.21.15.1:0/3678389362 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:30.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb6a59c640 1 --2- 172.21.15.1:0/3678389362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb54004800 0x7feb6c1eb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb69d9b640 1 --2- 172.21.15.1:0/3678389362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb6c070f50 0x7feb6c14dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb6ad9d640 1 -- 172.21.15.1:0/3678389362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb6c070f50 msgr2=0x7feb6c14dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb6ad9d640 1 --2- 172.21.15.1:0/3678389362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb6c070f50 0x7feb6c14dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb6ad9d640 1 -- 172.21.15.1:0/3678389362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb54004800 msgr2=0x7feb6c1eb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb6ad9d640 1 --2- 172.21.15.1:0/3678389362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb54004800 0x7feb6c1eb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb6ad9d640 1 -- 172.21.15.1:0/3678389362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c1eb570 con 0x7feb6c14e190 2024-03-06T07:31:30.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb6ad9d640 1 --2- 172.21.15.1:0/3678389362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb6c14e190 0x7feb6c1ee160 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0c2c10 tx=0x7feb6400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb5affd640 1 -- 172.21.15.1:0/3678389362 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6400fdb0 con 0x7feb6c14e190 2024-03-06T07:31:30.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb5affd640 1 -- 172.21.15.1:0/3678389362 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb64015430 con 0x7feb6c14e190 2024-03-06T07:31:30.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb5affd640 1 -- 172.21.15.1:0/3678389362 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb64015820 con 0x7feb6c14e190 2024-03-06T07:31:30.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb70fe9640 1 -- 172.21.15.1:0/3678389362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb6c06d4c0 con 0x7feb6c14e190 2024-03-06T07:31:30.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.576+0000 7feb70fe9640 1 -- 172.21.15.1:0/3678389362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb6c06da80 con 0x7feb6c14e190 2024-03-06T07:31:30.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.577+0000 7feb70fe9640 1 -- 172.21.15.1:0/3678389362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb6c067fd0 con 0x7feb6c14e190 2024-03-06T07:31:30.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.578+0000 7feb5affd640 1 -- 172.21.15.1:0/3678389362 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb640159c0 con 0x7feb6c14e190 2024-03-06T07:31:30.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.578+0000 7feb5affd640 1 --2- 172.21.15.1:0/3678389362 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb4c06b4d0 0x7feb4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.578+0000 7feb5affd640 1 -- 172.21.15.1:0/3678389362 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7feb6408b1a0 con 0x7feb6c14e190 2024-03-06T07:31:30.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.578+0000 7feb6a59c640 1 --2- 172.21.15.1:0/3678389362 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb4c06b4d0 0x7feb4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.579+0000 7feb6a59c640 1 --2- 172.21.15.1:0/3678389362 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb4c06b4d0 0x7feb4c06d990 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7feb5c001eb0 tx=0x7feb5c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.583+0000 7feb5affd640 1 -- 172.21.15.1:0/3678389362 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb6c1eb570 con 0x7feb6c14e190 2024-03-06T07:31:30.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.801+0000 7feb70fe9640 1 -- 172.21.15.1:0/3678389362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb6c0c2c40 con 0x7feb6c14e190 2024-03-06T07:31:30.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.802+0000 7feb5affd640 1 -- 172.21.15.1:0/3678389362 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7feb64019030 con 0x7feb6c14e190 2024-03-06T07:31:30.804 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:30.804 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:30.804 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:30.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.804+0000 7feb70fe9640 1 -- 172.21.15.1:0/3678389362 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb4c06b4d0 msgr2=0x7feb4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.804+0000 7feb70fe9640 1 --2- 172.21.15.1:0/3678389362 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb4c06b4d0 0x7feb4c06d990 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7feb5c001eb0 tx=0x7feb5c009040 comp rx=0 tx=0).stop 2024-03-06T07:31:30.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.804+0000 7feb70fe9640 1 -- 172.21.15.1:0/3678389362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb6c14e190 msgr2=0x7feb6c1ee160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.804+0000 7feb70fe9640 1 --2- 172.21.15.1:0/3678389362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb6c14e190 0x7feb6c1ee160 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0c2c10 tx=0x7feb6400f580 comp rx=0 tx=0).stop 2024-03-06T07:31:30.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.804+0000 7feb70fe9640 1 -- 172.21.15.1:0/3678389362 shutdown_connections 2024-03-06T07:31:30.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.804+0000 7feb70fe9640 1 --2- 172.21.15.1:0/3678389362 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7feb4c06b4d0 0x7feb4c06d990 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.804+0000 7feb70fe9640 1 --2- 172.21.15.1:0/3678389362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb6c14e190 0x7feb6c1ee160 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.804+0000 7feb70fe9640 1 --2- 172.21.15.1:0/3678389362 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb6c070f50 0x7feb6c14dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.804+0000 7feb70fe9640 1 --2- 172.21.15.1:0/3678389362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb54004800 0x7feb6c1eb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.804+0000 7feb70fe9640 1 -- 172.21.15.1:0/3678389362 >> 172.21.15.1:0/3678389362 conn(0x7feb6c06b410 msgr2=0x7feb6c0a8510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.804+0000 7feb70fe9640 1 -- 172.21.15.1:0/3678389362 shutdown_connections 2024-03-06T07:31:30.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:30.804+0000 7feb70fe9640 1 -- 172.21.15.1:0/3678389362 wait complete. 2024-03-06T07:31:31.223 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:31.223 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:32.224 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:32.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.363+0000 7f3458a71640 1 Processor -- start 2024-03-06T07:31:32.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.363+0000 7f3458a71640 1 -- start start 2024-03-06T07:31:32.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.364+0000 7f3458a71640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34540c3970 0x7f34540c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.364+0000 7f3458a71640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3454152840 0x7f34541553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.364+0000 7f3458a71640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34540c5e40 con 0x7f34540c60b0 2024-03-06T07:31:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.364+0000 7f3458a71640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34540c67f0 con 0x7f3454152840 2024-03-06T07:31:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.364+0000 7f3458a71640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34540c6a40 con 0x7f34540c3970 2024-03-06T07:31:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.364+0000 7f3452d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3454152840 0x7f34541553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.364+0000 7f3452d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3454152840 0x7f34541553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56744/0 (socket says 172.21.15.1:56744) 2024-03-06T07:31:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.364+0000 7f3451d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f34540c60b0 0x7f3454152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41260/0 (socket says 172.21.15.1:41260) 2024-03-06T07:31:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.364+0000 7f3452d9d640 1 -- 172.21.15.1:0/2489686779 learned_addr learned my addr 172.21.15.1:0/2489686779 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.364+0000 7f345259c640 1 --2- 172.21.15.1:0/2489686779 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34540c3970 0x7f34540c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.364+0000 7f3452d9d640 1 -- 172.21.15.1:0/2489686779 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34540c3970 msgr2=0x7f34540c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.364+0000 7f3452d9d640 1 --2- 172.21.15.1:0/2489686779 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34540c3970 0x7f34540c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.364+0000 7f3452d9d640 1 -- 172.21.15.1:0/2489686779 >> v1:172.21.15.1:6789/0 conn(0x7f34540c60b0 legacy=0x7f3454152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.364+0000 7f3452d9d640 1 -- 172.21.15.1:0/2489686779 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34540c6d20 con 0x7f3454152840 2024-03-06T07:31:32.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.364+0000 7f345159a640 1 -- 172.21.15.1:0/2489686779 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320789518 0 0) 0x7f34540c5e40 con 0x7f34540c60b0 2024-03-06T07:31:32.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.365+0000 7f3452d9d640 1 --2- 172.21.15.1:0/2489686779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3454152840 0x7f34541553d0 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f344c009e50 tx=0x7f344c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95290c359f1f4c82 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.365+0000 7f345159a640 1 -- 172.21.15.1:0/2489686779 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c02cb00 con 0x7f3454152840 2024-03-06T07:31:32.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.365+0000 7f345159a640 1 -- 172.21.15.1:0/2489686779 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344c02cca0 con 0x7f3454152840 2024-03-06T07:31:32.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.365+0000 7f345159a640 1 -- 172.21.15.1:0/2489686779 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c034610 con 0x7f3454152840 2024-03-06T07:31:32.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.366+0000 7f3458a71640 1 -- 172.21.15.1:0/2489686779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3454152840 msgr2=0x7f34541553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.366+0000 7f3458a71640 1 --2- 172.21.15.1:0/2489686779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3454152840 0x7f34541553d0 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f344c009e50 tx=0x7f344c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:31:32.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.366+0000 7f3458a71640 1 -- 172.21.15.1:0/2489686779 shutdown_connections 2024-03-06T07:31:32.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.366+0000 7f3458a71640 1 --2- 172.21.15.1:0/2489686779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3454152840 0x7f34541553d0 unknown :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.366+0000 7f3458a71640 1 --2- 172.21.15.1:0/2489686779 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34540c3970 0x7f34540c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.366+0000 7f3458a71640 1 -- 172.21.15.1:0/2489686779 >> 172.21.15.1:0/2489686779 conn(0x7f34540c88f0 msgr2=0x7f34540c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:32.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.366+0000 7f3458a71640 1 -- 172.21.15.1:0/2489686779 shutdown_connections 2024-03-06T07:31:32.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.366+0000 7f3458a71640 1 -- 172.21.15.1:0/2489686779 wait complete. 2024-03-06T07:31:32.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.367+0000 7f3458a71640 1 Processor -- start 2024-03-06T07:31:32.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.367+0000 7f3458a71640 1 -- start start 2024-03-06T07:31:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.368+0000 7f3458a71640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34540c3970 0x7f3454153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.368+0000 7f3458a71640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34540c60b0 0x7f345414f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.368+0000 7f3458a71640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3454152840 0x7f34540ac720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.368+0000 7f3458a71640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34540c75c0 con 0x7f34540c3970 2024-03-06T07:31:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.368+0000 7f3458a71640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34540c72e0 con 0x7f3454152840 2024-03-06T07:31:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.368+0000 7f3458a71640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34540c3d50 con 0x7f34540c60b0 2024-03-06T07:31:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.368+0000 7f345259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34540c3970 0x7f3454153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.368+0000 7f345259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34540c3970 0x7f3454153620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56752/0 (socket says 172.21.15.1:56752) 2024-03-06T07:31:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.368+0000 7f345259c640 1 -- 172.21.15.1:0/2791296762 learned_addr learned my addr 172.21.15.1:0/2791296762 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.368+0000 7f345259c640 1 -- 172.21.15.1:0/2791296762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34540c60b0 msgr2=0x7f345414f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.368+0000 7f345259c640 1 --2- 172.21.15.1:0/2791296762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34540c60b0 0x7f345414f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.368+0000 7f345259c640 1 -- 172.21.15.1:0/2791296762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3454152840 msgr2=0x7f34540ac720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.368+0000 7f345259c640 1 --2- 172.21.15.1:0/2791296762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3454152840 0x7f34540ac720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.368+0000 7f345259c640 1 -- 172.21.15.1:0/2791296762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34540ad9c0 con 0x7f34540c3970 2024-03-06T07:31:32.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.368+0000 7f345259c640 1 --2- 172.21.15.1:0/2791296762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34540c3970 0x7f3454153620 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f344400d800 tx=0x7f344400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.369+0000 7f344affd640 1 -- 172.21.15.1:0/2791296762 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344400b6c0 con 0x7f34540c3970 2024-03-06T07:31:32.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.369+0000 7f344affd640 1 -- 172.21.15.1:0/2791296762 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3444019070 con 0x7f34540c3970 2024-03-06T07:31:32.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.369+0000 7f344affd640 1 -- 172.21.15.1:0/2791296762 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344400ba80 con 0x7f34540c3970 2024-03-06T07:31:32.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.369+0000 7f3458a71640 1 -- 172.21.15.1:0/2791296762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34541ee080 con 0x7f34540c3970 2024-03-06T07:31:32.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.369+0000 7f3458a71640 1 -- 172.21.15.1:0/2791296762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34541ee360 con 0x7f34540c3970 2024-03-06T07:31:32.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.371+0000 7f344affd640 1 -- 172.21.15.1:0/2791296762 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f344400bc20 con 0x7f34540c3970 2024-03-06T07:31:32.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.371+0000 7f344affd640 1 --2- 172.21.15.1:0/2791296762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f343406b4d0 0x7f343406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.372+0000 7f344affd640 1 -- 172.21.15.1:0/2791296762 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f344408a8b0 con 0x7f34540c3970 2024-03-06T07:31:32.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.372+0000 7f3451d9b640 1 --2- 172.21.15.1:0/2791296762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f343406b4d0 0x7f343406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.372+0000 7f3451d9b640 1 --2- 172.21.15.1:0/2791296762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f343406b4d0 0x7f343406d990 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f343c007fd0 tx=0x7f343c007920 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.372+0000 7f3458a71640 1 -- 172.21.15.1:0/2791296762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34540686e0 con 0x7f34540c3970 2024-03-06T07:31:32.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.377+0000 7f344affd640 1 -- 172.21.15.1:0/2791296762 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3444059530 con 0x7f34540c3970 2024-03-06T07:31:32.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.647+0000 7f3458a71640 1 -- 172.21.15.1:0/2791296762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34540adea0 con 0x7f34540c3970 2024-03-06T07:31:32.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.647+0000 7f344affd640 1 -- 172.21.15.1:0/2791296762 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f3444058e30 con 0x7f34540c3970 2024-03-06T07:31:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:32.650 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:32.650 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:32.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.650+0000 7f3458a71640 1 -- 172.21.15.1:0/2791296762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f343406b4d0 msgr2=0x7f343406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.650+0000 7f3458a71640 1 --2- 172.21.15.1:0/2791296762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f343406b4d0 0x7f343406d990 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f343c007fd0 tx=0x7f343c007920 comp rx=0 tx=0).stop 2024-03-06T07:31:32.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.650+0000 7f3458a71640 1 -- 172.21.15.1:0/2791296762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34540c3970 msgr2=0x7f3454153620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.650+0000 7f3458a71640 1 --2- 172.21.15.1:0/2791296762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34540c3970 0x7f3454153620 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f344400d800 tx=0x7f344400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.650+0000 7f3458a71640 1 -- 172.21.15.1:0/2791296762 shutdown_connections 2024-03-06T07:31:32.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.650+0000 7f3458a71640 1 --2- 172.21.15.1:0/2791296762 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f343406b4d0 0x7f343406d990 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.650+0000 7f3458a71640 1 --2- 172.21.15.1:0/2791296762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3454152840 0x7f34540ac720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.650+0000 7f3458a71640 1 --2- 172.21.15.1:0/2791296762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34540c60b0 0x7f345414f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.650+0000 7f3458a71640 1 --2- 172.21.15.1:0/2791296762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34540c3970 0x7f3454153620 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.650+0000 7f3458a71640 1 -- 172.21.15.1:0/2791296762 >> 172.21.15.1:0/2791296762 conn(0x7f34540c88f0 msgr2=0x7f3454151650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:32.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.650+0000 7f3458a71640 1 -- 172.21.15.1:0/2791296762 shutdown_connections 2024-03-06T07:31:32.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:32.650+0000 7f3458a71640 1 -- 172.21.15.1:0/2791296762 wait complete. 2024-03-06T07:31:33.064 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:33.064 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:33.065 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:33.067 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:34.066 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:34.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.165+0000 7f8d4f993640 1 Processor -- start 2024-03-06T07:31:34.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.165+0000 7f8d4f993640 1 -- start start 2024-03-06T07:31:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.167+0000 7f8d4f993640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8d480c6b10 0x7f8d48155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.167+0000 7f8d4f993640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8d480bf9d0 0x7f8d480c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.167+0000 7f8d4f993640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d480c2fc0 con 0x7f8d48152650 2024-03-06T07:31:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.167+0000 7f8d4f993640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d48155800 con 0x7f8d480bf9d0 2024-03-06T07:31:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.167+0000 7f8d4f993640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d48158da0 con 0x7f8d480c6b10 2024-03-06T07:31:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.167+0000 7f8d4df30640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8d480bf9d0 0x7f8d480c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.167+0000 7f8d4df30640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8d480bf9d0 0x7f8d480c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56762/0 (socket says 172.21.15.1:56762) 2024-03-06T07:31:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.167+0000 7f8d4df30640 1 -- 172.21.15.1:0/3017015633 learned_addr learned my addr 172.21.15.1:0/3017015633 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.167+0000 7f8d4cf2e640 1 --2- 172.21.15.1:0/3017015633 >> v2:172.21.15.98:3300/0 conn(0x7f8d480c6b10 0x7f8d48155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.167+0000 7f8d4df30640 1 -- 172.21.15.1:0/3017015633 >> v2:172.21.15.98:3300/0 conn(0x7f8d480c6b10 msgr2=0x7f8d48155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.167+0000 7f8d4df30640 1 --2- 172.21.15.1:0/3017015633 >> v2:172.21.15.98:3300/0 conn(0x7f8d480c6b10 0x7f8d48155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.167+0000 7f8d4df30640 1 -- 172.21.15.1:0/3017015633 >> v1:172.21.15.98:6789/0 conn(0x7f8d48152650 legacy=0x7f8d48158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.167+0000 7f8d4df30640 1 -- 172.21.15.1:0/3017015633 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d480c6690 con 0x7f8d480bf9d0 2024-03-06T07:31:34.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.168+0000 7f8d4df30640 1 --2- 172.21.15.1:0/3017015633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d480bf9d0 0x7f8d480c1460 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f8d480cb8e0 tx=0x7f8d4002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6381fc48c7adb4c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.168+0000 7f8d3ffff640 1 -- 172.21.15.1:0/3017015633 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4002c9f0 con 0x7f8d480bf9d0 2024-03-06T07:31:34.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.168+0000 7f8d3ffff640 1 -- 172.21.15.1:0/3017015633 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4002cb90 con 0x7f8d480bf9d0 2024-03-06T07:31:34.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.168+0000 7f8d3ffff640 1 -- 172.21.15.1:0/3017015633 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d400343f0 con 0x7f8d480bf9d0 2024-03-06T07:31:34.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.169+0000 7f8d4f993640 1 -- 172.21.15.1:0/3017015633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d480bf9d0 msgr2=0x7f8d480c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.169+0000 7f8d4f993640 1 --2- 172.21.15.1:0/3017015633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d480bf9d0 0x7f8d480c1460 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f8d480cb8e0 tx=0x7f8d4002be50 comp rx=0 tx=0).stop 2024-03-06T07:31:34.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.169+0000 7f8d4f993640 1 -- 172.21.15.1:0/3017015633 shutdown_connections 2024-03-06T07:31:34.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.169+0000 7f8d4f993640 1 --2- 172.21.15.1:0/3017015633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d480bf9d0 0x7f8d480c1460 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.169+0000 7f8d4f993640 1 --2- 172.21.15.1:0/3017015633 >> v2:172.21.15.98:3300/0 conn(0x7f8d480c6b10 0x7f8d48155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.169+0000 7f8d4f993640 1 -- 172.21.15.1:0/3017015633 >> 172.21.15.1:0/3017015633 conn(0x7f8d48066ee0 msgr2=0x7f8d480672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.169+0000 7f8d4f993640 1 -- 172.21.15.1:0/3017015633 shutdown_connections 2024-03-06T07:31:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.169+0000 7f8d4f993640 1 -- 172.21.15.1:0/3017015633 wait complete. 2024-03-06T07:31:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.170+0000 7f8d4f993640 1 Processor -- start 2024-03-06T07:31:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.170+0000 7f8d4f993640 1 -- start start 2024-03-06T07:31:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4f993640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d480bf9d0 0x7f8d48157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4f993640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d48152650 0x7f8d480c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4f993640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d481ee6f0 0x7f8d4814b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4f993640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d481551b0 con 0x7f8d481ee6f0 2024-03-06T07:31:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4f993640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d480c6fa0 con 0x7f8d48152650 2024-03-06T07:31:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4f993640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d48155490 con 0x7f8d480bf9d0 2024-03-06T07:31:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4df30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d481ee6f0 0x7f8d4814b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4df30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d481ee6f0 0x7f8d4814b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56770/0 (socket says 172.21.15.1:56770) 2024-03-06T07:31:34.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4df30640 1 -- 172.21.15.1:0/3998491319 learned_addr learned my addr 172.21.15.1:0/3998491319 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:34.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4cf2e640 1 --2- 172.21.15.1:0/3998491319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d48152650 0x7f8d480c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4d72f640 1 --2- 172.21.15.1:0/3998491319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d480bf9d0 0x7f8d48157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4df30640 1 -- 172.21.15.1:0/3998491319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d480bf9d0 msgr2=0x7f8d48157860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4df30640 1 --2- 172.21.15.1:0/3998491319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d480bf9d0 0x7f8d48157860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4df30640 1 -- 172.21.15.1:0/3998491319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d48152650 msgr2=0x7f8d480c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4df30640 1 --2- 172.21.15.1:0/3998491319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d48152650 0x7f8d480c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4df30640 1 -- 172.21.15.1:0/3998491319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d481ef030 con 0x7f8d481ee6f0 2024-03-06T07:31:34.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4df30640 1 --2- 172.21.15.1:0/3998491319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d481ee6f0 0x7f8d4814b270 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f8d4002c460 tx=0x7f8d40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d3dffb640 1 -- 172.21.15.1:0/3998491319 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d40043020 con 0x7f8d481ee6f0 2024-03-06T07:31:34.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d3dffb640 1 -- 172.21.15.1:0/3998491319 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4003e450 con 0x7f8d481ee6f0 2024-03-06T07:31:34.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.171+0000 7f8d4f993640 1 -- 172.21.15.1:0/3998491319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d481535b0 con 0x7f8d481ee6f0 2024-03-06T07:31:34.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.172+0000 7f8d3dffb640 1 -- 172.21.15.1:0/3998491319 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d40009510 con 0x7f8d481ee6f0 2024-03-06T07:31:34.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.172+0000 7f8d4f993640 1 -- 172.21.15.1:0/3998491319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d48158090 con 0x7f8d481ee6f0 2024-03-06T07:31:34.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.173+0000 7f8d4f993640 1 -- 172.21.15.1:0/3998491319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d48153990 con 0x7f8d481ee6f0 2024-03-06T07:31:34.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.173+0000 7f8d3dffb640 1 -- 172.21.15.1:0/3998491319 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d40005d10 con 0x7f8d481ee6f0 2024-03-06T07:31:34.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.173+0000 7f8d3dffb640 1 --2- 172.21.15.1:0/3998491319 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d2c06b500 0x7f8d2c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.173+0000 7f8d3dffb640 1 -- 172.21.15.1:0/3998491319 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f8d400acdf0 con 0x7f8d481ee6f0 2024-03-06T07:31:34.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.174+0000 7f8d4d72f640 1 --2- 172.21.15.1:0/3998491319 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d2c06b500 0x7f8d2c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.174+0000 7f8d4d72f640 1 --2- 172.21.15.1:0/3998491319 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d2c06b500 0x7f8d2c06d9c0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f8d44004f60 tx=0x7f8d44007e30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.182+0000 7f8d3dffb640 1 -- 172.21.15.1:0/3998491319 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d48153990 con 0x7f8d481ee6f0 2024-03-06T07:31:34.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.400+0000 7f8d4f993640 1 -- 172.21.15.1:0/3998491319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d48067fd0 con 0x7f8d481ee6f0 2024-03-06T07:31:34.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.401+0000 7f8d3dffb640 1 -- 172.21.15.1:0/3998491319 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f8d4007b3a0 con 0x7f8d481ee6f0 2024-03-06T07:31:34.403 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:34.403 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:34.403 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:34.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.403+0000 7f8d4f993640 1 -- 172.21.15.1:0/3998491319 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d2c06b500 msgr2=0x7f8d2c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.403+0000 7f8d4f993640 1 --2- 172.21.15.1:0/3998491319 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d2c06b500 0x7f8d2c06d9c0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f8d44004f60 tx=0x7f8d44007e30 comp rx=0 tx=0).stop 2024-03-06T07:31:34.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.403+0000 7f8d4f993640 1 -- 172.21.15.1:0/3998491319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d481ee6f0 msgr2=0x7f8d4814b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.403+0000 7f8d4f993640 1 --2- 172.21.15.1:0/3998491319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d481ee6f0 0x7f8d4814b270 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f8d4002c460 tx=0x7f8d40004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:34.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.403+0000 7f8d4f993640 1 -- 172.21.15.1:0/3998491319 shutdown_connections 2024-03-06T07:31:34.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.403+0000 7f8d4f993640 1 --2- 172.21.15.1:0/3998491319 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8d2c06b500 0x7f8d2c06d9c0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.403+0000 7f8d4f993640 1 --2- 172.21.15.1:0/3998491319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d481ee6f0 0x7f8d4814b270 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.403+0000 7f8d4f993640 1 --2- 172.21.15.1:0/3998491319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d48152650 0x7f8d480c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.403+0000 7f8d4f993640 1 --2- 172.21.15.1:0/3998491319 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8d480bf9d0 0x7f8d48157860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.403+0000 7f8d4f993640 1 -- 172.21.15.1:0/3998491319 >> 172.21.15.1:0/3998491319 conn(0x7f8d48066ee0 msgr2=0x7f8d480672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:34.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.403+0000 7f8d4f993640 1 -- 172.21.15.1:0/3998491319 shutdown_connections 2024-03-06T07:31:34.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:34.403+0000 7f8d4f993640 1 -- 172.21.15.1:0/3998491319 wait complete. 2024-03-06T07:31:34.820 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:34.821 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:35.822 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:35.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.921+0000 7f275a66b640 1 Processor -- start 2024-03-06T07:31:35.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.921+0000 7f275a66b640 1 -- start start 2024-03-06T07:31:35.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.923+0000 7f275a66b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2754152850 0x7f27540c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.923+0000 7f275a66b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27540c4aa0 0x7f275415fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.923+0000 7f275a66b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2754156150 con 0x7f27540bf9e0 2024-03-06T07:31:35.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.923+0000 7f275a66b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2754152c30 con 0x7f27540c4aa0 2024-03-06T07:31:35.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.923+0000 7f275a66b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2754067810 con 0x7f2754152850 2024-03-06T07:31:35.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.923+0000 7f2758c08640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27540c4aa0 0x7f275415fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.923+0000 7f2758c08640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27540c4aa0 0x7f275415fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46700/0 (socket says 172.21.15.1:46700) 2024-03-06T07:31:35.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.923+0000 7f2758c08640 1 -- 172.21.15.1:0/3153600665 learned_addr learned my addr 172.21.15.1:0/3153600665 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:35.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.923+0000 7f2753fff640 1 --2- 172.21.15.1:0/3153600665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2754152850 0x7f27540c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.923+0000 7f2758c08640 1 -- 172.21.15.1:0/3153600665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2754152850 msgr2=0x7f27540c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:35.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.923+0000 7f2758c08640 1 --2- 172.21.15.1:0/3153600665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2754152850 0x7f27540c67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.923+0000 7f2758c08640 1 -- 172.21.15.1:0/3153600665 >> v1:172.21.15.1:6789/0 conn(0x7f27540bf9e0 legacy=0x7f27540c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.923+0000 7f2758c08640 1 -- 172.21.15.1:0/3153600665 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2754155390 con 0x7f27540c4aa0 2024-03-06T07:31:35.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.923+0000 7f2752ffd640 1 -- 172.21.15.1:0/3153600665 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555587665 0 0) 0x7f2754156150 con 0x7f27540bf9e0 2024-03-06T07:31:35.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.924+0000 7f2758c08640 1 --2- 172.21.15.1:0/3153600665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27540c4aa0 0x7f275415fb20 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f274c009e50 tx=0x7f274c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b5ef7f8826dbf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.924+0000 7f2752ffd640 1 -- 172.21.15.1:0/3153600665 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c02c9e0 con 0x7f27540c4aa0 2024-03-06T07:31:35.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.924+0000 7f2752ffd640 1 -- 172.21.15.1:0/3153600665 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274c02cb80 con 0x7f27540c4aa0 2024-03-06T07:31:35.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.924+0000 7f2752ffd640 1 -- 172.21.15.1:0/3153600665 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c0343f0 con 0x7f27540c4aa0 2024-03-06T07:31:35.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.925+0000 7f275a66b640 1 -- 172.21.15.1:0/3153600665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27540c4aa0 msgr2=0x7f275415fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.925+0000 7f275a66b640 1 --2- 172.21.15.1:0/3153600665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27540c4aa0 0x7f275415fb20 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f274c009e50 tx=0x7f274c02be40 comp rx=0 tx=0).stop 2024-03-06T07:31:35.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.925+0000 7f275a66b640 1 -- 172.21.15.1:0/3153600665 shutdown_connections 2024-03-06T07:31:35.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.925+0000 7f275a66b640 1 --2- 172.21.15.1:0/3153600665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27540c4aa0 0x7f275415fb20 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.925+0000 7f275a66b640 1 --2- 172.21.15.1:0/3153600665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2754152850 0x7f27540c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.925+0000 7f275a66b640 1 -- 172.21.15.1:0/3153600665 >> 172.21.15.1:0/3153600665 conn(0x7f275408b400 msgr2=0x7f2754066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:35.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.925+0000 7f275a66b640 1 -- 172.21.15.1:0/3153600665 shutdown_connections 2024-03-06T07:31:35.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.925+0000 7f275a66b640 1 -- 172.21.15.1:0/3153600665 wait complete. 2024-03-06T07:31:35.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.927+0000 7f275a66b640 1 Processor -- start 2024-03-06T07:31:35.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.927+0000 7f275a66b640 1 -- start start 2024-03-06T07:31:35.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.928+0000 7f275a66b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27540bf9e0 0x7f27541eab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.928+0000 7f275a66b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27540c4aa0 0x7f275414a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.928+0000 7f275a66b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2754152850 0x7f275414aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.928+0000 7f275a66b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2754155950 con 0x7f27540c4aa0 2024-03-06T07:31:35.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.928+0000 7f275a66b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2754155cc0 con 0x7f27540bf9e0 2024-03-06T07:31:35.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.928+0000 7f275a66b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27540c4550 con 0x7f2754152850 2024-03-06T07:31:35.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.928+0000 7f27537fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27540c4aa0 0x7f275414a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.928+0000 7f27537fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27540c4aa0 0x7f275414a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46710/0 (socket says 172.21.15.1:46710) 2024-03-06T07:31:35.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.928+0000 7f27537fe640 1 -- 172.21.15.1:0/467542011 learned_addr learned my addr 172.21.15.1:0/467542011 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:35.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.928+0000 7f2758c08640 1 --2- 172.21.15.1:0/467542011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2754152850 0x7f275414aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.928+0000 7f2753fff640 1 --2- 172.21.15.1:0/467542011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27540bf9e0 0x7f27541eab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.928+0000 7f27537fe640 1 -- 172.21.15.1:0/467542011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2754152850 msgr2=0x7f275414aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.928+0000 7f27537fe640 1 --2- 172.21.15.1:0/467542011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2754152850 0x7f275414aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.928+0000 7f27537fe640 1 -- 172.21.15.1:0/467542011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27540bf9e0 msgr2=0x7f27541eab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.928+0000 7f27537fe640 1 --2- 172.21.15.1:0/467542011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27540bf9e0 0x7f27541eab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.928+0000 7f27537fe640 1 -- 172.21.15.1:0/467542011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27541563e0 con 0x7f27540c4aa0 2024-03-06T07:31:35.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.928+0000 7f27537fe640 1 --2- 172.21.15.1:0/467542011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27540c4aa0 0x7f275414a580 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f274800c520 tx=0x7f274800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.929+0000 7f2750ff9640 1 -- 172.21.15.1:0/467542011 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2748017070 con 0x7f27540c4aa0 2024-03-06T07:31:35.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.929+0000 7f2750ff9640 1 -- 172.21.15.1:0/467542011 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27480035f0 con 0x7f27540c4aa0 2024-03-06T07:31:35.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.929+0000 7f275a66b640 1 -- 172.21.15.1:0/467542011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27541eb440 con 0x7f27540c4aa0 2024-03-06T07:31:35.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.929+0000 7f2750ff9640 1 -- 172.21.15.1:0/467542011 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27480039e0 con 0x7f27540c4aa0 2024-03-06T07:31:35.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.929+0000 7f275a66b640 1 -- 172.21.15.1:0/467542011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27540b06e0 con 0x7f27540c4aa0 2024-03-06T07:31:35.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.930+0000 7f275a66b640 1 -- 172.21.15.1:0/467542011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f275406b500 con 0x7f27540c4aa0 2024-03-06T07:31:35.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.930+0000 7f2750ff9640 1 -- 172.21.15.1:0/467542011 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f274801a070 con 0x7f27540c4aa0 2024-03-06T07:31:35.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.931+0000 7f2750ff9640 1 --2- 172.21.15.1:0/467542011 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f274006b500 0x7f274006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.931+0000 7f2750ff9640 1 -- 172.21.15.1:0/467542011 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f2748089870 con 0x7f27540c4aa0 2024-03-06T07:31:35.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.931+0000 7f2753fff640 1 --2- 172.21.15.1:0/467542011 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f274006b500 0x7f274006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.931+0000 7f2753fff640 1 --2- 172.21.15.1:0/467542011 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f274006b500 0x7f274006d9c0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f2744006fd0 tx=0x7f2744008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:35.935+0000 7f2750ff9640 1 -- 172.21.15.1:0/467542011 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f275406b500 con 0x7f27540c4aa0 2024-03-06T07:31:36.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:36.243+0000 7f275a66b640 1 -- 172.21.15.1:0/467542011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2754067fd0 con 0x7f27540c4aa0 2024-03-06T07:31:36.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:36.243+0000 7f2750ff9640 1 -- 172.21.15.1:0/467542011 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f27480584f0 con 0x7f27540c4aa0 2024-03-06T07:31:36.245 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:36.245 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:36.246 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:36.245+0000 7f275a66b640 1 -- 172.21.15.1:0/467542011 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f274006b500 msgr2=0x7f274006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:36.245+0000 7f275a66b640 1 --2- 172.21.15.1:0/467542011 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f274006b500 0x7f274006d9c0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f2744006fd0 tx=0x7f2744008040 comp rx=0 tx=0).stop 2024-03-06T07:31:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:36.245+0000 7f275a66b640 1 -- 172.21.15.1:0/467542011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27540c4aa0 msgr2=0x7f275414a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:36.245+0000 7f275a66b640 1 --2- 172.21.15.1:0/467542011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27540c4aa0 0x7f275414a580 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f274800c520 tx=0x7f274800c9e0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:36.245+0000 7f275a66b640 1 -- 172.21.15.1:0/467542011 shutdown_connections 2024-03-06T07:31:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:36.245+0000 7f275a66b640 1 --2- 172.21.15.1:0/467542011 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f274006b500 0x7f274006d9c0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:36.245+0000 7f275a66b640 1 --2- 172.21.15.1:0/467542011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2754152850 0x7f275414aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:36.245+0000 7f275a66b640 1 --2- 172.21.15.1:0/467542011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27540c4aa0 0x7f275414a580 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:36.245+0000 7f275a66b640 1 --2- 172.21.15.1:0/467542011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27540bf9e0 0x7f27541eab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:36.245+0000 7f275a66b640 1 -- 172.21.15.1:0/467542011 >> 172.21.15.1:0/467542011 conn(0x7f275408b400 msgr2=0x7f2754150dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:36.246+0000 7f275a66b640 1 -- 172.21.15.1:0/467542011 shutdown_connections 2024-03-06T07:31:36.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:36.246+0000 7f275a66b640 1 -- 172.21.15.1:0/467542011 wait complete. 2024-03-06T07:31:36.661 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:36.661 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:37.662 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:37.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.762+0000 7fa0e344e640 1 Processor -- start 2024-03-06T07:31:37.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.762+0000 7fa0e344e640 1 -- start start 2024-03-06T07:31:37.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.763+0000 7fa0e344e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0dc0c6b10 0x7fa0dc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.763+0000 7fa0e344e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa0dc0bf9d0 0x7fa0dc0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.763+0000 7fa0e344e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0dc0c2fc0 con 0x7fa0dc152650 2024-03-06T07:31:37.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.763+0000 7fa0e344e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0dc155800 con 0x7fa0dc0bf9d0 2024-03-06T07:31:37.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.763+0000 7fa0e344e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc158da0 con 0x7fa0dc0c6b10 2024-03-06T07:31:37.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.763+0000 7fa0e19eb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa0dc0bf9d0 0x7fa0dc0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.763+0000 7fa0e19eb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa0dc0bf9d0 0x7fa0dc0c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46720/0 (socket says 172.21.15.1:46720) 2024-03-06T07:31:37.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.763+0000 7fa0e19eb640 1 -- 172.21.15.1:0/588428011 learned_addr learned my addr 172.21.15.1:0/588428011 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:37.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.763+0000 7fa0e11ea640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa0dc152650 0x7fa0dc158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40242/0 (socket says 172.21.15.1:40242) 2024-03-06T07:31:37.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.764+0000 7fa0e19eb640 1 -- 172.21.15.1:0/588428011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0dc0c6b10 msgr2=0x7fa0dc155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.764+0000 7fa0e19eb640 1 --2- 172.21.15.1:0/588428011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0dc0c6b10 0x7fa0dc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.764+0000 7fa0e19eb640 1 -- 172.21.15.1:0/588428011 >> v1:172.21.15.1:6789/0 conn(0x7fa0dc152650 legacy=0x7fa0dc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.764+0000 7fa0cffff640 1 -- 172.21.15.1:0/588428011 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728235574 0 0) 0x7fa0dc0c2fc0 con 0x7fa0dc152650 2024-03-06T07:31:37.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.764+0000 7fa0e19eb640 1 -- 172.21.15.1:0/588428011 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0dc0c6690 con 0x7fa0dc0bf9d0 2024-03-06T07:31:37.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.764+0000 7fa0e19eb640 1 --2- 172.21.15.1:0/588428011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0dc0bf9d0 0x7fa0dc0c1460 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009e50 tx=0x7fa0d402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f61062e6835c58aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.764+0000 7fa0cffff640 1 -- 172.21.15.1:0/588428011 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d402ca70 con 0x7fa0dc0bf9d0 2024-03-06T07:31:37.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.765+0000 7fa0cffff640 1 -- 172.21.15.1:0/588428011 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d402cc10 con 0x7fa0dc0bf9d0 2024-03-06T07:31:37.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.765+0000 7fa0cffff640 1 -- 172.21.15.1:0/588428011 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d40343f0 con 0x7fa0dc0bf9d0 2024-03-06T07:31:37.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.765+0000 7fa0e344e640 1 -- 172.21.15.1:0/588428011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0dc0bf9d0 msgr2=0x7fa0dc0c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.765+0000 7fa0e344e640 1 --2- 172.21.15.1:0/588428011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0dc0bf9d0 0x7fa0dc0c1460 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009e50 tx=0x7fa0d402bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.765+0000 7fa0e344e640 1 -- 172.21.15.1:0/588428011 shutdown_connections 2024-03-06T07:31:37.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.765+0000 7fa0e344e640 1 --2- 172.21.15.1:0/588428011 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0dc0c6b10 0x7fa0dc155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.765+0000 7fa0e344e640 1 --2- 172.21.15.1:0/588428011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0dc0bf9d0 0x7fa0dc0c1460 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.765+0000 7fa0e344e640 1 -- 172.21.15.1:0/588428011 >> 172.21.15.1:0/588428011 conn(0x7fa0dc066ee0 msgr2=0x7fa0dc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.765+0000 7fa0e344e640 1 -- 172.21.15.1:0/588428011 shutdown_connections 2024-03-06T07:31:37.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.765+0000 7fa0e344e640 1 -- 172.21.15.1:0/588428011 wait complete. 2024-03-06T07:31:37.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.766+0000 7fa0e344e640 1 Processor -- start 2024-03-06T07:31:37.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.766+0000 7fa0e344e640 1 -- start start 2024-03-06T07:31:37.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.767+0000 7fa0e344e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0dc0bf9d0 0x7fa0dc1de970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.767+0000 7fa0e344e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0dc152650 0x7fa0dc157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.767+0000 7fa0e344e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0dc1e06e0 0x7fa0dc1578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.767+0000 7fa0e344e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc1551b0 con 0x7fa0dc1e06e0 2024-03-06T07:31:37.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.767+0000 7fa0e344e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc0c6fa0 con 0x7fa0dc0bf9d0 2024-03-06T07:31:37.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.767+0000 7fa0e344e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc155490 con 0x7fa0dc152650 2024-03-06T07:31:37.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.767+0000 7fa0e19eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0dc1e06e0 0x7fa0dc1578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.767+0000 7fa0e19eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0dc1e06e0 0x7fa0dc1578b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46734/0 (socket says 172.21.15.1:46734) 2024-03-06T07:31:37.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.767+0000 7fa0e19eb640 1 -- 172.21.15.1:0/610281225 learned_addr learned my addr 172.21.15.1:0/610281225 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:37.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.767+0000 7fa0e09e9640 1 --2- 172.21.15.1:0/610281225 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0dc152650 0x7fa0dc157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.767+0000 7fa0e11ea640 1 --2- 172.21.15.1:0/610281225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0dc0bf9d0 0x7fa0dc1de970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.767+0000 7fa0e19eb640 1 -- 172.21.15.1:0/610281225 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0dc152650 msgr2=0x7fa0dc157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.767+0000 7fa0e19eb640 1 --2- 172.21.15.1:0/610281225 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0dc152650 0x7fa0dc157370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.767+0000 7fa0e19eb640 1 -- 172.21.15.1:0/610281225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0dc0bf9d0 msgr2=0x7fa0dc1de970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.767+0000 7fa0e19eb640 1 --2- 172.21.15.1:0/610281225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0dc0bf9d0 0x7fa0dc1de970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.767+0000 7fa0e19eb640 1 -- 172.21.15.1:0/610281225 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0dc1e0c10 con 0x7fa0dc1e06e0 2024-03-06T07:31:37.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.768+0000 7fa0e19eb640 1 --2- 172.21.15.1:0/610281225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0dc1e06e0 0x7fa0dc1578b0 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009f70 tx=0x7fa0d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.768+0000 7fa0cdffb640 1 -- 172.21.15.1:0/610281225 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4043030 con 0x7fa0dc1e06e0 2024-03-06T07:31:37.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.768+0000 7fa0cdffb640 1 -- 172.21.15.1:0/610281225 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d403e450 con 0x7fa0dc1e06e0 2024-03-06T07:31:37.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.768+0000 7fa0e344e640 1 -- 172.21.15.1:0/610281225 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0dc1e13f0 con 0x7fa0dc1e06e0 2024-03-06T07:31:37.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.768+0000 7fa0cdffb640 1 -- 172.21.15.1:0/610281225 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4009510 con 0x7fa0dc1e06e0 2024-03-06T07:31:37.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.768+0000 7fa0e344e640 1 -- 172.21.15.1:0/610281225 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0dc1e19b0 con 0x7fa0dc1e06e0 2024-03-06T07:31:37.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.769+0000 7fa0e344e640 1 -- 172.21.15.1:0/610281225 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0dc067fd0 con 0x7fa0dc1e06e0 2024-03-06T07:31:37.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.770+0000 7fa0cdffb640 1 -- 172.21.15.1:0/610281225 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0d4046070 con 0x7fa0dc1e06e0 2024-03-06T07:31:37.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.770+0000 7fa0cdffb640 1 --2- 172.21.15.1:0/610281225 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0c006b500 0x7fa0c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.770+0000 7fa0cdffb640 1 -- 172.21.15.1:0/610281225 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa0d40aca20 con 0x7fa0dc1e06e0 2024-03-06T07:31:37.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.771+0000 7fa0e11ea640 1 --2- 172.21.15.1:0/610281225 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0c006b500 0x7fa0c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.771+0000 7fa0e11ea640 1 --2- 172.21.15.1:0/610281225 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0c006b500 0x7fa0c006d9c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0000f60 tx=0x7fa0d0007b70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:37.779+0000 7fa0cdffb640 1 -- 172.21.15.1:0/610281225 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0d404d070 con 0x7fa0dc1e06e0 2024-03-06T07:31:38.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:38.001+0000 7fa0e344e640 1 -- 172.21.15.1:0/610281225 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0dc0b06f0 con 0x7fa0dc1e06e0 2024-03-06T07:31:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:38.002+0000 7fa0cdffb640 1 -- 172.21.15.1:0/610281225 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fa0d40473b0 con 0x7fa0dc1e06e0 2024-03-06T07:31:38.004 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:38.004 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:38.004+0000 7fa0e344e640 1 -- 172.21.15.1:0/610281225 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0c006b500 msgr2=0x7fa0c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:38.004+0000 7fa0e344e640 1 --2- 172.21.15.1:0/610281225 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0c006b500 0x7fa0c006d9c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0000f60 tx=0x7fa0d0007b70 comp rx=0 tx=0).stop 2024-03-06T07:31:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:38.004+0000 7fa0e344e640 1 -- 172.21.15.1:0/610281225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0dc1e06e0 msgr2=0x7fa0dc1578b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:38.004+0000 7fa0e344e640 1 --2- 172.21.15.1:0/610281225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0dc1e06e0 0x7fa0dc1578b0 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009f70 tx=0x7fa0d4004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:38.004+0000 7fa0e344e640 1 -- 172.21.15.1:0/610281225 shutdown_connections 2024-03-06T07:31:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:38.004+0000 7fa0e344e640 1 --2- 172.21.15.1:0/610281225 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0c006b500 0x7fa0c006d9c0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:38.004+0000 7fa0e344e640 1 --2- 172.21.15.1:0/610281225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0dc1e06e0 0x7fa0dc1578b0 unknown :-1 s=CLOSED pgs=2527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:38.004+0000 7fa0e344e640 1 --2- 172.21.15.1:0/610281225 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0dc152650 0x7fa0dc157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:38.004+0000 7fa0e344e640 1 --2- 172.21.15.1:0/610281225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0dc0bf9d0 0x7fa0dc1de970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:38.004+0000 7fa0e344e640 1 -- 172.21.15.1:0/610281225 >> 172.21.15.1:0/610281225 conn(0x7fa0dc066ee0 msgr2=0x7fa0dc0c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:38.004+0000 7fa0e344e640 1 -- 172.21.15.1:0/610281225 shutdown_connections 2024-03-06T07:31:38.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:38.004+0000 7fa0e344e640 1 -- 172.21.15.1:0/610281225 wait complete. 2024-03-06T07:31:38.420 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:38.420 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:39.421 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:39.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.521+0000 7f0b53e15640 1 Processor -- start 2024-03-06T07:31:39.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.521+0000 7f0b53e15640 1 -- start start 2024-03-06T07:31:39.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.522+0000 7f0b53e15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b4c0c2b30 0x7f0b4c0724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.522+0000 7f0b53e15640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0b4c0a5fa0 0x7f0b4c072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.522+0000 7f0b53e15640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4c151dd0 con 0x7f0b4c152b70 2024-03-06T07:31:39.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.522+0000 7f0b53e15640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b4c1526e0 con 0x7f0b4c0a5fa0 2024-03-06T07:31:39.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.522+0000 7f0b53e15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c156190 con 0x7f0b4c0c2b30 2024-03-06T07:31:39.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.522+0000 7f0b523b2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0b4c0a5fa0 0x7f0b4c072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.522+0000 7f0b51bb1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b4c0c2b30 0x7f0b4c0724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.523+0000 7f0b523b2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0b4c0a5fa0 0x7f0b4c072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:53720/0 (socket says 172.21.15.1:53720) 2024-03-06T07:31:39.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.523+0000 7f0b523b2640 1 -- 172.21.15.1:0/230538291 learned_addr learned my addr 172.21.15.1:0/230538291 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:39.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.523+0000 7f0b523b2640 1 -- 172.21.15.1:0/230538291 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b4c0c2b30 msgr2=0x7f0b4c0724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:39.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.523+0000 7f0b523b2640 1 --2- 172.21.15.1:0/230538291 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b4c0c2b30 0x7f0b4c0724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.523+0000 7f0b523b2640 1 -- 172.21.15.1:0/230538291 >> v1:172.21.15.98:6789/0 conn(0x7f0b4c152b70 legacy=0x7f0b4c0c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.523+0000 7f0b523b2640 1 -- 172.21.15.1:0/230538291 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4c152090 con 0x7f0b4c0a5fa0 2024-03-06T07:31:39.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.523+0000 7f0b51bb1640 1 --2- 172.21.15.1:0/230538291 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b4c0c2b30 0x7f0b4c0724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:39.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.524+0000 7f0b523b2640 1 --2- 172.21.15.1:0/230538291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b4c0a5fa0 0x7f0b4c072a00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f0b4400d150 tx=0x7f0b4402e160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad74f3787c67c972 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.524+0000 7f0b50baf640 1 -- 172.21.15.1:0/230538291 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4400b0d0 con 0x7f0b4c0a5fa0 2024-03-06T07:31:39.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.524+0000 7f0b50baf640 1 -- 172.21.15.1:0/230538291 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4400b270 con 0x7f0b4c0a5fa0 2024-03-06T07:31:39.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.524+0000 7f0b50baf640 1 -- 172.21.15.1:0/230538291 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b44009600 con 0x7f0b4c0a5fa0 2024-03-06T07:31:39.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.524+0000 7f0b53e15640 1 -- 172.21.15.1:0/230538291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b4c0a5fa0 msgr2=0x7f0b4c072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.524+0000 7f0b53e15640 1 --2- 172.21.15.1:0/230538291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b4c0a5fa0 0x7f0b4c072a00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f0b4400d150 tx=0x7f0b4402e160 comp rx=0 tx=0).stop 2024-03-06T07:31:39.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.524+0000 7f0b53e15640 1 -- 172.21.15.1:0/230538291 shutdown_connections 2024-03-06T07:31:39.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.524+0000 7f0b53e15640 1 --2- 172.21.15.1:0/230538291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b4c0a5fa0 0x7f0b4c072a00 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.524+0000 7f0b53e15640 1 --2- 172.21.15.1:0/230538291 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b4c0c2b30 0x7f0b4c0724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.524+0000 7f0b53e15640 1 -- 172.21.15.1:0/230538291 >> 172.21.15.1:0/230538291 conn(0x7f0b4c0c6860 msgr2=0x7f0b4c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:39.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.525+0000 7f0b53e15640 1 -- 172.21.15.1:0/230538291 shutdown_connections 2024-03-06T07:31:39.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.525+0000 7f0b53e15640 1 -- 172.21.15.1:0/230538291 wait complete. 2024-03-06T07:31:39.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.526+0000 7f0b53e15640 1 Processor -- start 2024-03-06T07:31:39.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.526+0000 7f0b53e15640 1 -- start start 2024-03-06T07:31:39.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.526+0000 7f0b53e15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b4c0a5fa0 0x7f0b4c14a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.526+0000 7f0b53e15640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b4c0c2b30 0x7f0b4c14c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.526+0000 7f0b53e15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b4c152b70 0x7f0b4c1eaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.526+0000 7f0b53e15640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c155f00 con 0x7f0b4c152b70 2024-03-06T07:31:39.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.526+0000 7f0b53e15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c155d80 con 0x7f0b4c0a5fa0 2024-03-06T07:31:39.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.526+0000 7f0b53e15640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c0c3a00 con 0x7f0b4c0c2b30 2024-03-06T07:31:39.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.526+0000 7f0b523b2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b4c152b70 0x7f0b4c1eaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.526+0000 7f0b523b2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b4c152b70 0x7f0b4c1eaf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46744/0 (socket says 172.21.15.1:46744) 2024-03-06T07:31:39.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.526+0000 7f0b523b2640 1 -- 172.21.15.1:0/2444129924 learned_addr learned my addr 172.21.15.1:0/2444129924 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:39.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.527+0000 7f0b51bb1640 1 --2- 172.21.15.1:0/2444129924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b4c0a5fa0 0x7f0b4c14a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.527+0000 7f0b513b0640 1 --2- 172.21.15.1:0/2444129924 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b4c0c2b30 0x7f0b4c14c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.527+0000 7f0b523b2640 1 -- 172.21.15.1:0/2444129924 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b4c0c2b30 msgr2=0x7f0b4c14c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.527+0000 7f0b523b2640 1 --2- 172.21.15.1:0/2444129924 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b4c0c2b30 0x7f0b4c14c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.527+0000 7f0b523b2640 1 -- 172.21.15.1:0/2444129924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b4c0a5fa0 msgr2=0x7f0b4c14a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.527+0000 7f0b523b2640 1 --2- 172.21.15.1:0/2444129924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b4c0a5fa0 0x7f0b4c14a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.527+0000 7f0b523b2640 1 -- 172.21.15.1:0/2444129924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4c149e70 con 0x7f0b4c152b70 2024-03-06T07:31:39.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.527+0000 7f0b523b2640 1 --2- 172.21.15.1:0/2444129924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b4c152b70 0x7f0b4c1eaf80 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f0b4400d270 tx=0x7f0b44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.527+0000 7f0b427fc640 1 -- 172.21.15.1:0/2444129924 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4400b0d0 con 0x7f0b4c152b70 2024-03-06T07:31:39.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.527+0000 7f0b427fc640 1 -- 172.21.15.1:0/2444129924 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b44045070 con 0x7f0b4c152b70 2024-03-06T07:31:39.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.527+0000 7f0b53e15640 1 -- 172.21.15.1:0/2444129924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b4c0aca00 con 0x7f0b4c152b70 2024-03-06T07:31:39.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.527+0000 7f0b427fc640 1 -- 172.21.15.1:0/2444129924 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4400b720 con 0x7f0b4c152b70 2024-03-06T07:31:39.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.527+0000 7f0b53e15640 1 -- 172.21.15.1:0/2444129924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b4c1db5c0 con 0x7f0b4c152b70 2024-03-06T07:31:39.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.528+0000 7f0b53e15640 1 -- 172.21.15.1:0/2444129924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b4c0ac7f0 con 0x7f0b4c152b70 2024-03-06T07:31:39.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.529+0000 7f0b427fc640 1 -- 172.21.15.1:0/2444129924 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b4402ed10 con 0x7f0b4c152b70 2024-03-06T07:31:39.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.530+0000 7f0b427fc640 1 --2- 172.21.15.1:0/2444129924 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0b3006b4d0 0x7f0b3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.530+0000 7f0b427fc640 1 -- 172.21.15.1:0/2444129924 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f0b440acea0 con 0x7f0b4c152b70 2024-03-06T07:31:39.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.530+0000 7f0b51bb1640 1 --2- 172.21.15.1:0/2444129924 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0b3006b4d0 0x7f0b3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.530+0000 7f0b51bb1640 1 --2- 172.21.15.1:0/2444129924 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0b3006b4d0 0x7f0b3006d990 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c0c72a0 tx=0x7f0b3c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.534+0000 7f0b427fc640 1 -- 172.21.15.1:0/2444129924 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b4407b5f0 con 0x7f0b4c152b70 2024-03-06T07:31:39.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.755+0000 7f0b53e15640 1 -- 172.21.15.1:0/2444129924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b4c0acbd0 con 0x7f0b4c152b70 2024-03-06T07:31:39.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.756+0000 7f0b427fc640 1 -- 172.21.15.1:0/2444129924 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f0b4407f690 con 0x7f0b4c152b70 2024-03-06T07:31:39.758 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:39.758 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:39.758 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:39.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.758+0000 7f0b53e15640 1 -- 172.21.15.1:0/2444129924 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0b3006b4d0 msgr2=0x7f0b3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.758+0000 7f0b53e15640 1 --2- 172.21.15.1:0/2444129924 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0b3006b4d0 0x7f0b3006d990 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c0c72a0 tx=0x7f0b3c009190 comp rx=0 tx=0).stop 2024-03-06T07:31:39.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.758+0000 7f0b53e15640 1 -- 172.21.15.1:0/2444129924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b4c152b70 msgr2=0x7f0b4c1eaf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.758+0000 7f0b53e15640 1 --2- 172.21.15.1:0/2444129924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b4c152b70 0x7f0b4c1eaf80 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f0b4400d270 tx=0x7f0b44004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.758+0000 7f0b53e15640 1 -- 172.21.15.1:0/2444129924 shutdown_connections 2024-03-06T07:31:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.758+0000 7f0b53e15640 1 --2- 172.21.15.1:0/2444129924 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0b3006b4d0 0x7f0b3006d990 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.758+0000 7f0b53e15640 1 --2- 172.21.15.1:0/2444129924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b4c152b70 0x7f0b4c1eaf80 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.758+0000 7f0b53e15640 1 --2- 172.21.15.1:0/2444129924 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b4c0c2b30 0x7f0b4c14c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.758+0000 7f0b53e15640 1 --2- 172.21.15.1:0/2444129924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b4c0a5fa0 0x7f0b4c14a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.758+0000 7f0b53e15640 1 -- 172.21.15.1:0/2444129924 >> 172.21.15.1:0/2444129924 conn(0x7f0b4c0c6860 msgr2=0x7f0b4c0bfc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.758+0000 7f0b53e15640 1 -- 172.21.15.1:0/2444129924 shutdown_connections 2024-03-06T07:31:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:39.758+0000 7f0b53e15640 1 -- 172.21.15.1:0/2444129924 wait complete. 2024-03-06T07:31:40.173 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:40.173 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:41.174 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:41.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.274+0000 7fe2a268f640 1 Processor -- start 2024-03-06T07:31:41.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.274+0000 7fe2a268f640 1 -- start start 2024-03-06T07:31:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.275+0000 7fe2a268f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe29c152650 0x7fe29c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.275+0000 7fe2a268f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe29c0bf9d0 0x7fe29c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.275+0000 7fe2a268f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29c0c2fc0 con 0x7fe29c0c6b10 2024-03-06T07:31:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.275+0000 7fe2a268f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe29c155800 con 0x7fe29c152650 2024-03-06T07:31:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.275+0000 7fe2a268f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c158da0 con 0x7fe29c0bf9d0 2024-03-06T07:31:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.275+0000 7fe29bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe29c152650 0x7fe29c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.275+0000 7fe29b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe29c0c6b10 0x7fe29c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40252/0 (socket says 172.21.15.1:40252) 2024-03-06T07:31:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.275+0000 7fe29b7fe640 1 -- 172.21.15.1:0/13526683 learned_addr learned my addr 172.21.15.1:0/13526683 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:31:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.275+0000 7fe29bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe29c152650 0x7fe29c155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46748/0 (socket says 172.21.15.1:46748) 2024-03-06T07:31:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.275+0000 7fe2a0c2c640 1 --2- 172.21.15.1:0/13526683 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe29c0bf9d0 0x7fe29c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.275+0000 7fe29bfff640 1 -- 172.21.15.1:0/13526683 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe29c0bf9d0 msgr2=0x7fe29c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.275+0000 7fe29bfff640 1 --2- 172.21.15.1:0/13526683 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe29c0bf9d0 0x7fe29c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.275+0000 7fe29bfff640 1 -- 172.21.15.1:0/13526683 >> v1:172.21.15.1:6789/0 conn(0x7fe29c0c6b10 legacy=0x7fe29c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.275+0000 7fe29affd640 1 -- 172.21.15.1:0/13526683 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160786236 0 0) 0x7fe29c0c2fc0 con 0x7fe29c0c6b10 2024-03-06T07:31:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.275+0000 7fe29bfff640 1 -- 172.21.15.1:0/13526683 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29c0c6690 con 0x7fe29c152650 2024-03-06T07:31:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.276+0000 7fe29bfff640 1 --2- 172.21.15.1:0/13526683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe29c152650 0x7fe29c155c90 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7fe2900097a0 tx=0x7fe29002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc6586206911710 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.276+0000 7fe29affd640 1 -- 172.21.15.1:0/13526683 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2900315a0 con 0x7fe29c152650 2024-03-06T07:31:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.276+0000 7fe29affd640 1 -- 172.21.15.1:0/13526683 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe290037070 con 0x7fe29c152650 2024-03-06T07:31:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.276+0000 7fe29affd640 1 -- 172.21.15.1:0/13526683 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe290031960 con 0x7fe29c152650 2024-03-06T07:31:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.277+0000 7fe2a268f640 1 -- 172.21.15.1:0/13526683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe29c152650 msgr2=0x7fe29c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.277+0000 7fe2a268f640 1 --2- 172.21.15.1:0/13526683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe29c152650 0x7fe29c155c90 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7fe2900097a0 tx=0x7fe29002b840 comp rx=0 tx=0).stop 2024-03-06T07:31:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.277+0000 7fe2a268f640 1 -- 172.21.15.1:0/13526683 shutdown_connections 2024-03-06T07:31:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.277+0000 7fe2a268f640 1 --2- 172.21.15.1:0/13526683 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe29c0bf9d0 0x7fe29c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.277+0000 7fe2a268f640 1 --2- 172.21.15.1:0/13526683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe29c152650 0x7fe29c155c90 unknown :-1 s=CLOSED pgs=2530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.277+0000 7fe2a268f640 1 -- 172.21.15.1:0/13526683 >> 172.21.15.1:0/13526683 conn(0x7fe29c066ee0 msgr2=0x7fe29c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.277+0000 7fe2a268f640 1 -- 172.21.15.1:0/13526683 shutdown_connections 2024-03-06T07:31:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.277+0000 7fe2a268f640 1 -- 172.21.15.1:0/13526683 wait complete. 2024-03-06T07:31:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.278+0000 7fe2a268f640 1 Processor -- start 2024-03-06T07:31:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.278+0000 7fe2a268f640 1 -- start start 2024-03-06T07:31:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.279+0000 7fe2a268f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe29c0bf9d0 0x7fe29c1edfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.279+0000 7fe2a268f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe29c152650 0x7fe29c1e2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.279+0000 7fe2a268f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe29c0c9bd0 0x7fe29c1578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.279+0000 7fe2a268f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c1551b0 con 0x7fe29c0c9bd0 2024-03-06T07:31:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.279+0000 7fe2a268f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c0c6fa0 con 0x7fe29c152650 2024-03-06T07:31:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.279+0000 7fe2a268f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c155490 con 0x7fe29c0bf9d0 2024-03-06T07:31:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.279+0000 7fe2a0c2c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe29c0c9bd0 0x7fe29c1578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.279+0000 7fe2a0c2c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe29c0c9bd0 0x7fe29c1578f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46758/0 (socket says 172.21.15.1:46758) 2024-03-06T07:31:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.279+0000 7fe2a0c2c640 1 -- 172.21.15.1:0/2384548356 learned_addr learned my addr 172.21.15.1:0/2384548356 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.279+0000 7fe29bfff640 1 --2- 172.21.15.1:0/2384548356 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe29c0bf9d0 0x7fe29c1edfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.279+0000 7fe29b7fe640 1 --2- 172.21.15.1:0/2384548356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe29c152650 0x7fe29c1e2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.279+0000 7fe2a0c2c640 1 -- 172.21.15.1:0/2384548356 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe29c0bf9d0 msgr2=0x7fe29c1edfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.279+0000 7fe2a0c2c640 1 --2- 172.21.15.1:0/2384548356 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe29c0bf9d0 0x7fe29c1edfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.279+0000 7fe2a0c2c640 1 -- 172.21.15.1:0/2384548356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe29c152650 msgr2=0x7fe29c1e2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.279+0000 7fe2a0c2c640 1 --2- 172.21.15.1:0/2384548356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe29c152650 0x7fe29c1e2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.279+0000 7fe2a0c2c640 1 -- 172.21.15.1:0/2384548356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29c154400 con 0x7fe29c0c9bd0 2024-03-06T07:31:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.279+0000 7fe2a0c2c640 1 --2- 172.21.15.1:0/2384548356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe29c0c9bd0 0x7fe29c1578f0 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7fe294007e90 tx=0x7fe29400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.280+0000 7fe298ff9640 1 -- 172.21.15.1:0/2384548356 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe294014420 con 0x7fe29c0c9bd0 2024-03-06T07:31:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.280+0000 7fe298ff9640 1 -- 172.21.15.1:0/2384548356 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe294019070 con 0x7fe29c0c9bd0 2024-03-06T07:31:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.280+0000 7fe298ff9640 1 -- 172.21.15.1:0/2384548356 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe294014800 con 0x7fe29c0c9bd0 2024-03-06T07:31:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.280+0000 7fe2a268f640 1 -- 172.21.15.1:0/2384548356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe29c158590 con 0x7fe29c0c9bd0 2024-03-06T07:31:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.280+0000 7fe2a268f640 1 -- 172.21.15.1:0/2384548356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29c1e2660 con 0x7fe29c0c9bd0 2024-03-06T07:31:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.281+0000 7fe2a268f640 1 -- 172.21.15.1:0/2384548356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe29c067fd0 con 0x7fe29c0c9bd0 2024-03-06T07:31:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.281+0000 7fe298ff9640 1 -- 172.21.15.1:0/2384548356 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2940149a0 con 0x7fe29c0c9bd0 2024-03-06T07:31:41.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.281+0000 7fe298ff9640 1 --2- 172.21.15.1:0/2384548356 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe28806b4d0 0x7fe28806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.281+0000 7fe298ff9640 1 -- 172.21.15.1:0/2384548356 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fe29408ac30 con 0x7fe29c0c9bd0 2024-03-06T07:31:41.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.282+0000 7fe29bfff640 1 --2- 172.21.15.1:0/2384548356 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe28806b4d0 0x7fe28806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.282+0000 7fe29bfff640 1 --2- 172.21.15.1:0/2384548356 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe28806b4d0 0x7fe28806d990 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe290036000 tx=0x7fe29003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.290+0000 7fe298ff9640 1 -- 172.21.15.1:0/2384548356 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2940598b0 con 0x7fe29c0c9bd0 2024-03-06T07:31:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.528+0000 7fe2a268f640 1 -- 172.21.15.1:0/2384548356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe29c0b01a0 con 0x7fe29c0c9bd0 2024-03-06T07:31:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.528+0000 7fe298ff9640 1 -- 172.21.15.1:0/2384548356 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fe2940591b0 con 0x7fe29c0c9bd0 2024-03-06T07:31:41.531 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:41.531 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:41.531 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:41.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.531+0000 7fe2a268f640 1 -- 172.21.15.1:0/2384548356 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe28806b4d0 msgr2=0x7fe28806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.531+0000 7fe2a268f640 1 --2- 172.21.15.1:0/2384548356 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe28806b4d0 0x7fe28806d990 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe290036000 tx=0x7fe29003d040 comp rx=0 tx=0).stop 2024-03-06T07:31:41.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.531+0000 7fe2a268f640 1 -- 172.21.15.1:0/2384548356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe29c0c9bd0 msgr2=0x7fe29c1578f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.531+0000 7fe2a268f640 1 --2- 172.21.15.1:0/2384548356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe29c0c9bd0 0x7fe29c1578f0 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7fe294007e90 tx=0x7fe29400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.531+0000 7fe2a268f640 1 -- 172.21.15.1:0/2384548356 shutdown_connections 2024-03-06T07:31:41.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.531+0000 7fe2a268f640 1 --2- 172.21.15.1:0/2384548356 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe28806b4d0 0x7fe28806d990 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.531+0000 7fe2a268f640 1 --2- 172.21.15.1:0/2384548356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe29c0c9bd0 0x7fe29c1578f0 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.531+0000 7fe2a268f640 1 --2- 172.21.15.1:0/2384548356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe29c152650 0x7fe29c1e2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.531+0000 7fe2a268f640 1 --2- 172.21.15.1:0/2384548356 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe29c0bf9d0 0x7fe29c1edfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.531+0000 7fe2a268f640 1 -- 172.21.15.1:0/2384548356 >> 172.21.15.1:0/2384548356 conn(0x7fe29c066ee0 msgr2=0x7fe29c0cae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.531+0000 7fe2a268f640 1 -- 172.21.15.1:0/2384548356 shutdown_connections 2024-03-06T07:31:41.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:41.531+0000 7fe2a268f640 1 -- 172.21.15.1:0/2384548356 wait complete. 2024-03-06T07:31:41.945 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:41.946 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:42.947 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:43.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.085+0000 7fe7276ec640 1 Processor -- start 2024-03-06T07:31:43.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.085+0000 7fe7276ec640 1 -- start start 2024-03-06T07:31:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.086+0000 7fe7276ec640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe7200af6f0 0x7fe7200c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.086+0000 7fe7276ec640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe7200c7930 0x7fe72015fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.086+0000 7fe7276ec640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7200c7180 con 0x7fe720152820 2024-03-06T07:31:43.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.086+0000 7fe7276ec640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe720155460 con 0x7fe7200af6f0 2024-03-06T07:31:43.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.086+0000 7fe7276ec640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7200c6bc0 con 0x7fe7200c7930 2024-03-06T07:31:43.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.086+0000 7fe724c87640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe7200af6f0 0x7fe7200c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.086+0000 7fe724c87640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe7200af6f0 0x7fe7200c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46760/0 (socket says 172.21.15.1:46760) 2024-03-06T07:31:43.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.086+0000 7fe724c87640 1 -- 172.21.15.1:0/3272811570 learned_addr learned my addr 172.21.15.1:0/3272811570 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:43.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.086+0000 7fe725c89640 1 --2- 172.21.15.1:0/3272811570 >> v2:172.21.15.98:3300/0 conn(0x7fe7200c7930 0x7fe72015fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.086+0000 7fe724c87640 1 -- 172.21.15.1:0/3272811570 >> v2:172.21.15.98:3300/0 conn(0x7fe7200c7930 msgr2=0x7fe72015fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:43.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.086+0000 7fe724c87640 1 --2- 172.21.15.1:0/3272811570 >> v2:172.21.15.98:3300/0 conn(0x7fe7200c7930 0x7fe72015fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.086+0000 7fe724c87640 1 -- 172.21.15.1:0/3272811570 >> v1:172.21.15.98:6789/0 conn(0x7fe720152820 legacy=0x7fe7200c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:43.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.086+0000 7fe724c87640 1 -- 172.21.15.1:0/3272811570 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7200c6ea0 con 0x7fe7200af6f0 2024-03-06T07:31:43.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.087+0000 7fe724c87640 1 --2- 172.21.15.1:0/3272811570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7200af6f0 0x7fe7200c73f0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7fe72006c030 tx=0x7fe71c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e098db37284f45 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.087+0000 7fe713fff640 1 -- 172.21.15.1:0/3272811570 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c035020 con 0x7fe7200af6f0 2024-03-06T07:31:43.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.087+0000 7fe713fff640 1 -- 172.21.15.1:0/3272811570 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71c031870 con 0x7fe7200af6f0 2024-03-06T07:31:43.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.087+0000 7fe713fff640 1 -- 172.21.15.1:0/3272811570 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c031c40 con 0x7fe7200af6f0 2024-03-06T07:31:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.088+0000 7fe7276ec640 1 -- 172.21.15.1:0/3272811570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7200af6f0 msgr2=0x7fe7200c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.088+0000 7fe7276ec640 1 --2- 172.21.15.1:0/3272811570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7200af6f0 0x7fe7200c73f0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7fe72006c030 tx=0x7fe71c02b910 comp rx=0 tx=0).stop 2024-03-06T07:31:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.088+0000 7fe7276ec640 1 -- 172.21.15.1:0/3272811570 shutdown_connections 2024-03-06T07:31:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.088+0000 7fe7276ec640 1 --2- 172.21.15.1:0/3272811570 >> v2:172.21.15.98:3300/0 conn(0x7fe7200c7930 0x7fe72015fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.088+0000 7fe7276ec640 1 --2- 172.21.15.1:0/3272811570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7200af6f0 0x7fe7200c73f0 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.088+0000 7fe7276ec640 1 -- 172.21.15.1:0/3272811570 >> 172.21.15.1:0/3272811570 conn(0x7fe720072720 msgr2=0x7fe720072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.088+0000 7fe7276ec640 1 -- 172.21.15.1:0/3272811570 shutdown_connections 2024-03-06T07:31:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.088+0000 7fe7276ec640 1 -- 172.21.15.1:0/3272811570 wait complete. 2024-03-06T07:31:43.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.089+0000 7fe7276ec640 1 Processor -- start 2024-03-06T07:31:43.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.089+0000 7fe7276ec640 1 -- start start 2024-03-06T07:31:43.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe7276ec640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7200af6f0 0x7fe720156fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe7276ec640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7200c7930 0x7fe7200c3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe7276ec640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe720152820 0x7fe7201ea9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe7276ec640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe720152c00 con 0x7fe7200af6f0 2024-03-06T07:31:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe7276ec640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7200c0840 con 0x7fe720152820 2024-03-06T07:31:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe7276ec640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe720152d80 con 0x7fe7200c7930 2024-03-06T07:31:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe725488640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7200af6f0 0x7fe720156fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe725488640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7200af6f0 0x7fe720156fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46770/0 (socket says 172.21.15.1:46770) 2024-03-06T07:31:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe725488640 1 -- 172.21.15.1:0/595774939 learned_addr learned my addr 172.21.15.1:0/595774939 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe725c89640 1 --2- 172.21.15.1:0/595774939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe720152820 0x7fe7201ea9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe724c87640 1 --2- 172.21.15.1:0/595774939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7200c7930 0x7fe7200c3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe725488640 1 -- 172.21.15.1:0/595774939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7200c7930 msgr2=0x7fe7200c3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe725488640 1 --2- 172.21.15.1:0/595774939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7200c7930 0x7fe7200c3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe725488640 1 -- 172.21.15.1:0/595774939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe720152820 msgr2=0x7fe7201ea9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe725488640 1 --2- 172.21.15.1:0/595774939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe720152820 0x7fe7201ea9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe725488640 1 -- 172.21.15.1:0/595774939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe720156260 con 0x7fe7200af6f0 2024-03-06T07:31:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe725488640 1 --2- 172.21.15.1:0/595774939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7200af6f0 0x7fe720156fb0 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7fe71400b3f0 tx=0x7fe71400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.090+0000 7fe711ffb640 1 -- 172.21.15.1:0/595774939 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe714015020 con 0x7fe7200af6f0 2024-03-06T07:31:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.091+0000 7fe711ffb640 1 -- 172.21.15.1:0/595774939 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe714011830 con 0x7fe7200af6f0 2024-03-06T07:31:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.091+0000 7fe711ffb640 1 -- 172.21.15.1:0/595774939 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe714011c20 con 0x7fe7200af6f0 2024-03-06T07:31:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.091+0000 7fe7276ec640 1 -- 172.21.15.1:0/595774939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe720156540 con 0x7fe7200af6f0 2024-03-06T07:31:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.091+0000 7fe7276ec640 1 -- 172.21.15.1:0/595774939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7200aa3e0 con 0x7fe7200af6f0 2024-03-06T07:31:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.092+0000 7fe7276ec640 1 -- 172.21.15.1:0/595774939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7200686e0 con 0x7fe7200af6f0 2024-03-06T07:31:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.092+0000 7fe711ffb640 1 -- 172.21.15.1:0/595774939 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe714018020 con 0x7fe7200af6f0 2024-03-06T07:31:43.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.093+0000 7fe711ffb640 1 --2- 172.21.15.1:0/595774939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe70406b4d0 0x7fe70406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.093+0000 7fe711ffb640 1 -- 172.21.15.1:0/595774939 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fe71408a030 con 0x7fe7200af6f0 2024-03-06T07:31:43.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.093+0000 7fe724c87640 1 --2- 172.21.15.1:0/595774939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe70406b4d0 0x7fe70406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.093+0000 7fe724c87640 1 --2- 172.21.15.1:0/595774939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe70406b4d0 0x7fe70406d990 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fe71c036000 tx=0x7fe71c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.097+0000 7fe711ffb640 1 -- 172.21.15.1:0/595774939 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe71401f030 con 0x7fe7200af6f0 2024-03-06T07:31:43.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.315+0000 7fe7276ec640 1 -- 172.21.15.1:0/595774939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7200c5b40 con 0x7fe7200af6f0 2024-03-06T07:31:43.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.316+0000 7fe711ffb640 1 -- 172.21.15.1:0/595774939 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fe7140172c0 con 0x7fe7200af6f0 2024-03-06T07:31:43.318 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:43.318 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:43.318 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.318+0000 7fe7276ec640 1 -- 172.21.15.1:0/595774939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe70406b4d0 msgr2=0x7fe70406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.318+0000 7fe7276ec640 1 --2- 172.21.15.1:0/595774939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe70406b4d0 0x7fe70406d990 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fe71c036000 tx=0x7fe71c03c040 comp rx=0 tx=0).stop 2024-03-06T07:31:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.318+0000 7fe7276ec640 1 -- 172.21.15.1:0/595774939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7200af6f0 msgr2=0x7fe720156fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.318+0000 7fe7276ec640 1 --2- 172.21.15.1:0/595774939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7200af6f0 0x7fe720156fb0 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7fe71400b3f0 tx=0x7fe71400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.318+0000 7fe7276ec640 1 -- 172.21.15.1:0/595774939 shutdown_connections 2024-03-06T07:31:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.318+0000 7fe7276ec640 1 --2- 172.21.15.1:0/595774939 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe70406b4d0 0x7fe70406d990 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.318+0000 7fe7276ec640 1 --2- 172.21.15.1:0/595774939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe720152820 0x7fe7201ea9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.318+0000 7fe7276ec640 1 --2- 172.21.15.1:0/595774939 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe7200c7930 0x7fe7200c3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.318+0000 7fe7276ec640 1 --2- 172.21.15.1:0/595774939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7200af6f0 0x7fe720156fb0 unknown :-1 s=CLOSED pgs=2533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.318+0000 7fe7276ec640 1 -- 172.21.15.1:0/595774939 >> 172.21.15.1:0/595774939 conn(0x7fe720072720 msgr2=0x7fe720074450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.318+0000 7fe7276ec640 1 -- 172.21.15.1:0/595774939 shutdown_connections 2024-03-06T07:31:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:43.318+0000 7fe7276ec640 1 -- 172.21.15.1:0/595774939 wait complete. 2024-03-06T07:31:43.733 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:43.733 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:44.734 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:44.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.835+0000 7f7a0cb3e640 1 Processor -- start 2024-03-06T07:31:44.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.835+0000 7f7a0cb3e640 1 -- start start 2024-03-06T07:31:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.836+0000 7f7a0cb3e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7a08152840 0x7f7a080c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.836+0000 7f7a0cb3e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a080c5e40 con 0x7f7a080c3970 2024-03-06T07:31:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.836+0000 7f7a0cb3e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a080c6a40 con 0x7f7a080c60b0 2024-03-06T07:31:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.836+0000 7f7a0cb3e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a080c67f0 con 0x7f7a08152840 2024-03-06T07:31:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.836+0000 7f7a0659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7a080c3970 0x7f7a08152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40182/0 (socket says 172.21.15.1:40182) 2024-03-06T07:31:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.836+0000 7f7a0659c640 1 -- 172.21.15.1:0/270886732 learned_addr learned my addr 172.21.15.1:0/270886732 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:31:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.836+0000 7f7a05d9b640 1 --2- 172.21.15.1:0/270886732 >> v2:172.21.15.1:3300/0 conn(0x7f7a08152840 0x7f7a080c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.837+0000 7f7a05d9b640 1 -- 172.21.15.1:0/270886732 >> v1:172.21.15.98:6789/0 conn(0x7f7a080c60b0 legacy=0x7f7a081553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.837+0000 7f7a05d9b640 1 -- 172.21.15.1:0/270886732 >> v1:172.21.15.1:6789/0 conn(0x7f7a080c3970 legacy=0x7f7a08152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.837+0000 7f7a05d9b640 1 -- 172.21.15.1:0/270886732 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a080c6d20 con 0x7f7a08152840 2024-03-06T07:31:44.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.837+0000 7f7a05d9b640 1 --2- 172.21.15.1:0/270886732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a08152840 0x7f7a080c8170 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f79fc0097d0 tx=0x7f79fc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b421f6cb734474f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.838+0000 7f7a0559a640 1 -- 172.21.15.1:0/270886732 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79fc034070 con 0x7f7a08152840 2024-03-06T07:31:44.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.838+0000 7f7a0559a640 1 -- 172.21.15.1:0/270886732 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79fc030610 con 0x7f7a08152840 2024-03-06T07:31:44.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.838+0000 7f7a0559a640 1 -- 172.21.15.1:0/270886732 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79fc0309e0 con 0x7f7a08152840 2024-03-06T07:31:44.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.838+0000 7f7a0cb3e640 1 -- 172.21.15.1:0/270886732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a08152840 msgr2=0x7f7a080c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.838+0000 7f7a0cb3e640 1 --2- 172.21.15.1:0/270886732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a08152840 0x7f7a080c8170 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f79fc0097d0 tx=0x7f79fc02b960 comp rx=0 tx=0).stop 2024-03-06T07:31:44.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.838+0000 7f7a0cb3e640 1 -- 172.21.15.1:0/270886732 shutdown_connections 2024-03-06T07:31:44.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.838+0000 7f7a0cb3e640 1 --2- 172.21.15.1:0/270886732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a08152840 0x7f7a080c8170 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.838+0000 7f7a0cb3e640 1 -- 172.21.15.1:0/270886732 >> 172.21.15.1:0/270886732 conn(0x7f7a080c88f0 msgr2=0x7f7a080c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:44.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.838+0000 7f7a0cb3e640 1 -- 172.21.15.1:0/270886732 shutdown_connections 2024-03-06T07:31:44.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.838+0000 7f7a0cb3e640 1 -- 172.21.15.1:0/270886732 wait complete. 2024-03-06T07:31:44.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.840+0000 7f7a0cb3e640 1 Processor -- start 2024-03-06T07:31:44.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.840+0000 7f7a0cb3e640 1 -- start start 2024-03-06T07:31:44.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.840+0000 7f7a0cb3e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a080c3970 0x7f7a081e21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.840+0000 7f7a0cb3e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a080c60b0 0x7f7a081ee280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.840+0000 7f7a0cb3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a08152840 0x7f7a081ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.840+0000 7f7a0cb3e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a080c75c0 con 0x7f7a080c60b0 2024-03-06T07:31:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.840+0000 7f7a0cb3e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a080c72e0 con 0x7f7a080c3970 2024-03-06T07:31:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.840+0000 7f7a0cb3e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a080c3d50 con 0x7f7a08152840 2024-03-06T07:31:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.840+0000 7f7a05d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a080c60b0 0x7f7a081ee280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.840+0000 7f7a05d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a080c60b0 0x7f7a081ee280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36010/0 (socket says 172.21.15.1:36010) 2024-03-06T07:31:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.841+0000 7f7a05d9b640 1 -- 172.21.15.1:0/3983939508 learned_addr learned my addr 172.21.15.1:0/3983939508 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.841+0000 7f7a0659c640 1 --2- 172.21.15.1:0/3983939508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a080c3970 0x7f7a081e21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.841+0000 7f7a06d9d640 1 --2- 172.21.15.1:0/3983939508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a08152840 0x7f7a081ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.841+0000 7f7a05d9b640 1 -- 172.21.15.1:0/3983939508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a08152840 msgr2=0x7f7a081ee7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.841+0000 7f7a05d9b640 1 --2- 172.21.15.1:0/3983939508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a08152840 0x7f7a081ee7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.841+0000 7f7a05d9b640 1 -- 172.21.15.1:0/3983939508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a080c3970 msgr2=0x7f7a081e21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.841+0000 7f7a05d9b640 1 --2- 172.21.15.1:0/3983939508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a080c3970 0x7f7a081e21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.841+0000 7f7a05d9b640 1 -- 172.21.15.1:0/3983939508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a080a9bd0 con 0x7f7a080c60b0 2024-03-06T07:31:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.841+0000 7f7a06d9d640 1 --2- 172.21.15.1:0/3983939508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a08152840 0x7f7a081ee7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:44.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.841+0000 7f7a05d9b640 1 --2- 172.21.15.1:0/3983939508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a080c60b0 0x7f7a081ee280 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f79fc02bf70 tx=0x7f79fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.841+0000 7f79f6ffd640 1 -- 172.21.15.1:0/3983939508 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79fc034040 con 0x7f7a080c60b0 2024-03-06T07:31:44.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.841+0000 7f79f6ffd640 1 -- 172.21.15.1:0/3983939508 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79fc03ac40 con 0x7f7a080c60b0 2024-03-06T07:31:44.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.841+0000 7f79f6ffd640 1 -- 172.21.15.1:0/3983939508 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79fc0443f0 con 0x7f7a080c60b0 2024-03-06T07:31:44.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.841+0000 7f7a0cb3e640 1 -- 172.21.15.1:0/3983939508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a080a9eb0 con 0x7f7a080c60b0 2024-03-06T07:31:44.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.841+0000 7f7a0cb3e640 1 -- 172.21.15.1:0/3983939508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a081e26e0 con 0x7f7a080c60b0 2024-03-06T07:31:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.842+0000 7f7a0cb3e640 1 -- 172.21.15.1:0/3983939508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a080686e0 con 0x7f7a080c60b0 2024-03-06T07:31:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.843+0000 7f79f6ffd640 1 -- 172.21.15.1:0/3983939508 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79fc037070 con 0x7f7a080c60b0 2024-03-06T07:31:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.843+0000 7f79f6ffd640 1 --2- 172.21.15.1:0/3983939508 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f79e806b500 0x7f79e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.843+0000 7f79f6ffd640 1 -- 172.21.15.1:0/3983939508 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f79fc0ac220 con 0x7f7a080c60b0 2024-03-06T07:31:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.843+0000 7f7a0659c640 1 --2- 172.21.15.1:0/3983939508 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f79e806b500 0x7f79e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.844+0000 7f7a0659c640 1 --2- 172.21.15.1:0/3983939508 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f79e806b500 0x7f79e806d9c0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f79f8005880 tx=0x7f79f8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:44.847+0000 7f79f6ffd640 1 -- 172.21.15.1:0/3983939508 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79fc03f070 con 0x7f7a080c60b0 2024-03-06T07:31:45.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:45.162+0000 7f7a0cb3e640 1 -- 172.21.15.1:0/3983939508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a080748a0 con 0x7f7a080c60b0 2024-03-06T07:31:45.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:45.162+0000 7f79f6ffd640 1 -- 172.21.15.1:0/3983939508 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f79fc07a850 con 0x7f7a080c60b0 2024-03-06T07:31:45.164 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:45.164 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:45.164+0000 7f7a0cb3e640 1 -- 172.21.15.1:0/3983939508 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f79e806b500 msgr2=0x7f79e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:45.164+0000 7f7a0cb3e640 1 --2- 172.21.15.1:0/3983939508 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f79e806b500 0x7f79e806d9c0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f79f8005880 tx=0x7f79f8009040 comp rx=0 tx=0).stop 2024-03-06T07:31:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:45.164+0000 7f7a0cb3e640 1 -- 172.21.15.1:0/3983939508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a080c60b0 msgr2=0x7f7a081ee280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:45.164+0000 7f7a0cb3e640 1 --2- 172.21.15.1:0/3983939508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a080c60b0 0x7f7a081ee280 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f79fc02bf70 tx=0x7f79fc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:45.164+0000 7f7a0cb3e640 1 -- 172.21.15.1:0/3983939508 shutdown_connections 2024-03-06T07:31:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:45.164+0000 7f7a0cb3e640 1 --2- 172.21.15.1:0/3983939508 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f79e806b500 0x7f79e806d9c0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:45.164+0000 7f7a0cb3e640 1 --2- 172.21.15.1:0/3983939508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7a08152840 0x7f7a081ee7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:45.164+0000 7f7a0cb3e640 1 --2- 172.21.15.1:0/3983939508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a080c60b0 0x7f7a081ee280 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:45.164+0000 7f7a0cb3e640 1 --2- 172.21.15.1:0/3983939508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a080c3970 0x7f7a081e21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:45.164+0000 7f7a0cb3e640 1 -- 172.21.15.1:0/3983939508 >> 172.21.15.1:0/3983939508 conn(0x7f7a080c88f0 msgr2=0x7f7a0806c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:45.164+0000 7f7a0cb3e640 1 -- 172.21.15.1:0/3983939508 shutdown_connections 2024-03-06T07:31:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:45.164+0000 7f7a0cb3e640 1 -- 172.21.15.1:0/3983939508 wait complete. 2024-03-06T07:31:45.579 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:45.579 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:46.580 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:46.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.680+0000 7f5b251b5640 1 Processor -- start 2024-03-06T07:31:46.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.680+0000 7f5b251b5640 1 -- start start 2024-03-06T07:31:46.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.680+0000 7f5b251b5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5b20150fe0 0x7f5b201513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.680+0000 7f5b251b5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5b20071f90 0x7f5b20072400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.680+0000 7f5b251b5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b200a6140 con 0x7f5b20072f20 2024-03-06T07:31:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.680+0000 7f5b251b5640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b201526b0 con 0x7f5b20071f90 2024-03-06T07:31:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.680+0000 7f5b251b5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b20151aa0 con 0x7f5b20150fe0 2024-03-06T07:31:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.681+0000 7f5b1f59e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5b20071f90 0x7f5b20072400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.681+0000 7f5b1f59e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5b20071f90 0x7f5b20072400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36022/0 (socket says 172.21.15.1:36022) 2024-03-06T07:31:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.681+0000 7f5b1f59e640 1 -- 172.21.15.1:0/1643687138 learned_addr learned my addr 172.21.15.1:0/1643687138 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.681+0000 7f5b1f59e640 1 -- 172.21.15.1:0/1643687138 >> v2:172.21.15.98:3300/0 conn(0x7f5b20150fe0 msgr2=0x7f5b201513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.681+0000 7f5b1ed9d640 1 --2- 172.21.15.1:0/1643687138 >> v2:172.21.15.98:3300/0 conn(0x7f5b20150fe0 0x7f5b201513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.681+0000 7f5b1f59e640 1 --2- 172.21.15.1:0/1643687138 >> v2:172.21.15.98:3300/0 conn(0x7f5b20150fe0 0x7f5b201513c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.681+0000 7f5b1f59e640 1 -- 172.21.15.1:0/1643687138 >> v1:172.21.15.98:6789/0 conn(0x7f5b20072f20 legacy=0x7f5b2006edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.681+0000 7f5b1f59e640 1 -- 172.21.15.1:0/1643687138 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b20151d80 con 0x7f5b20071f90 2024-03-06T07:31:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.682+0000 7f5b1f59e640 1 --2- 172.21.15.1:0/1643687138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b20071f90 0x7f5b20072400 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f5b2006fba0 tx=0x7f5b1802e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8fa2569d9768783 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.682+0000 7f5b1e59c640 1 -- 172.21.15.1:0/1643687138 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1802ea00 con 0x7f5b20071f90 2024-03-06T07:31:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.682+0000 7f5b1e59c640 1 -- 172.21.15.1:0/1643687138 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1802eba0 con 0x7f5b20071f90 2024-03-06T07:31:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.682+0000 7f5b1e59c640 1 -- 172.21.15.1:0/1643687138 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18009600 con 0x7f5b20071f90 2024-03-06T07:31:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.682+0000 7f5b251b5640 1 -- 172.21.15.1:0/1643687138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b20071f90 msgr2=0x7f5b20072400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.682+0000 7f5b251b5640 1 --2- 172.21.15.1:0/1643687138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b20071f90 0x7f5b20072400 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f5b2006fba0 tx=0x7f5b1802e060 comp rx=0 tx=0).stop 2024-03-06T07:31:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.682+0000 7f5b251b5640 1 -- 172.21.15.1:0/1643687138 shutdown_connections 2024-03-06T07:31:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.682+0000 7f5b251b5640 1 --2- 172.21.15.1:0/1643687138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b20071f90 0x7f5b20072400 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.682+0000 7f5b251b5640 1 --2- 172.21.15.1:0/1643687138 >> v2:172.21.15.98:3300/0 conn(0x7f5b20150fe0 0x7f5b201513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.682+0000 7f5b251b5640 1 -- 172.21.15.1:0/1643687138 >> 172.21.15.1:0/1643687138 conn(0x7f5b200c6860 msgr2=0x7f5b200c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.682+0000 7f5b251b5640 1 -- 172.21.15.1:0/1643687138 shutdown_connections 2024-03-06T07:31:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.682+0000 7f5b251b5640 1 -- 172.21.15.1:0/1643687138 wait complete. 2024-03-06T07:31:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.684+0000 7f5b251b5640 1 Processor -- start 2024-03-06T07:31:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.684+0000 7f5b251b5640 1 -- start start 2024-03-06T07:31:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.684+0000 7f5b251b5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b20071f90 0x7f5b201e9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.684+0000 7f5b251b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b20072f20 0x7f5b201e7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.684+0000 7f5b251b5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b20150fe0 0x7f5b201e7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.684+0000 7f5b251b5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2006f4c0 con 0x7f5b20150fe0 2024-03-06T07:31:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.684+0000 7f5b251b5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b20152340 con 0x7f5b20072f20 2024-03-06T07:31:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.684+0000 7f5b251b5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2006f640 con 0x7f5b20071f90 2024-03-06T07:31:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.684+0000 7f5b1f59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b20150fe0 0x7f5b201e7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.684+0000 7f5b1f59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b20150fe0 0x7f5b201e7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36036/0 (socket says 172.21.15.1:36036) 2024-03-06T07:31:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.684+0000 7f5b1f59e640 1 -- 172.21.15.1:0/4193615296 learned_addr learned my addr 172.21.15.1:0/4193615296 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.685+0000 7f5b1ed9d640 1 --2- 172.21.15.1:0/4193615296 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b20071f90 0x7f5b201e9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.685+0000 7f5b17fff640 1 --2- 172.21.15.1:0/4193615296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b20072f20 0x7f5b201e7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.685+0000 7f5b1f59e640 1 -- 172.21.15.1:0/4193615296 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b20071f90 msgr2=0x7f5b201e9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.685+0000 7f5b1f59e640 1 --2- 172.21.15.1:0/4193615296 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b20071f90 0x7f5b201e9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.685+0000 7f5b1f59e640 1 -- 172.21.15.1:0/4193615296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b20072f20 msgr2=0x7f5b201e7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.685+0000 7f5b1f59e640 1 --2- 172.21.15.1:0/4193615296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b20072f20 0x7f5b201e7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.685+0000 7f5b1f59e640 1 -- 172.21.15.1:0/4193615296 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b201e83c0 con 0x7f5b20150fe0 2024-03-06T07:31:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.685+0000 7f5b1f59e640 1 --2- 172.21.15.1:0/4193615296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b20150fe0 0x7f5b201e7e80 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800d150 tx=0x7f5b18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.685+0000 7f5b177fe640 1 -- 172.21.15.1:0/4193615296 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18045020 con 0x7f5b20150fe0 2024-03-06T07:31:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.685+0000 7f5b177fe640 1 -- 172.21.15.1:0/4193615296 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b18040660 con 0x7f5b20150fe0 2024-03-06T07:31:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.685+0000 7f5b251b5640 1 -- 172.21.15.1:0/4193615296 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b201db710 con 0x7f5b20150fe0 2024-03-06T07:31:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.685+0000 7f5b177fe640 1 -- 172.21.15.1:0/4193615296 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1802ea00 con 0x7f5b20150fe0 2024-03-06T07:31:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.685+0000 7f5b251b5640 1 -- 172.21.15.1:0/4193615296 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b201dbcd0 con 0x7f5b20150fe0 2024-03-06T07:31:46.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.686+0000 7f5b251b5640 1 -- 172.21.15.1:0/4193615296 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b20067fd0 con 0x7f5b20150fe0 2024-03-06T07:31:46.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.687+0000 7f5b177fe640 1 -- 172.21.15.1:0/4193615296 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b18048020 con 0x7f5b20150fe0 2024-03-06T07:31:46.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.687+0000 7f5b177fe640 1 --2- 172.21.15.1:0/4193615296 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b0006b4d0 0x7f5b0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.687+0000 7f5b177fe640 1 -- 172.21.15.1:0/4193615296 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f5b180aeff0 con 0x7f5b20150fe0 2024-03-06T07:31:46.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.687+0000 7f5b1ed9d640 1 --2- 172.21.15.1:0/4193615296 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b0006b4d0 0x7f5b0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.688+0000 7f5b1ed9d640 1 --2- 172.21.15.1:0/4193615296 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b0006b4d0 0x7f5b0006d990 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f5b201db640 tx=0x7f5b10008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:46.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.692+0000 7f5b177fe640 1 -- 172.21.15.1:0/4193615296 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b1804f030 con 0x7f5b20150fe0 2024-03-06T07:31:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.912+0000 7f5b251b5640 1 -- 172.21.15.1:0/4193615296 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b200c59b0 con 0x7f5b20150fe0 2024-03-06T07:31:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.912+0000 7f5b177fe640 1 -- 172.21.15.1:0/4193615296 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f5b1807dbf0 con 0x7f5b20150fe0 2024-03-06T07:31:46.915 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:46.915 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:46.915 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:46.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.914+0000 7f5b251b5640 1 -- 172.21.15.1:0/4193615296 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b0006b4d0 msgr2=0x7f5b0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.914+0000 7f5b251b5640 1 --2- 172.21.15.1:0/4193615296 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b0006b4d0 0x7f5b0006d990 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f5b201db640 tx=0x7f5b10008040 comp rx=0 tx=0).stop 2024-03-06T07:31:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.914+0000 7f5b251b5640 1 -- 172.21.15.1:0/4193615296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b20150fe0 msgr2=0x7f5b201e7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.914+0000 7f5b251b5640 1 --2- 172.21.15.1:0/4193615296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b20150fe0 0x7f5b201e7e80 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800d150 tx=0x7f5b18004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.914+0000 7f5b251b5640 1 -- 172.21.15.1:0/4193615296 shutdown_connections 2024-03-06T07:31:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.914+0000 7f5b251b5640 1 --2- 172.21.15.1:0/4193615296 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5b0006b4d0 0x7f5b0006d990 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.914+0000 7f5b251b5640 1 --2- 172.21.15.1:0/4193615296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b20150fe0 0x7f5b201e7e80 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.914+0000 7f5b251b5640 1 --2- 172.21.15.1:0/4193615296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b20072f20 0x7f5b201e7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.914+0000 7f5b251b5640 1 --2- 172.21.15.1:0/4193615296 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5b20071f90 0x7f5b201e9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.914+0000 7f5b251b5640 1 -- 172.21.15.1:0/4193615296 >> 172.21.15.1:0/4193615296 conn(0x7f5b200c6860 msgr2=0x7f5b2006edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.914+0000 7f5b251b5640 1 -- 172.21.15.1:0/4193615296 shutdown_connections 2024-03-06T07:31:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:46.914+0000 7f5b251b5640 1 -- 172.21.15.1:0/4193615296 wait complete. 2024-03-06T07:31:47.330 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:47.330 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:48.330 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:48.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.430+0000 7fd14705c640 1 Processor -- start 2024-03-06T07:31:48.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.430+0000 7fd14705c640 1 -- start start 2024-03-06T07:31:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.432+0000 7fd14705c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1400af6f0 0x7fd1400c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.432+0000 7fd14705c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1400c7180 con 0x7fd140152820 2024-03-06T07:31:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.432+0000 7fd14705c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd140155460 con 0x7fd1400c7930 2024-03-06T07:31:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.432+0000 7fd14705c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1400c6bc0 con 0x7fd1400af6f0 2024-03-06T07:31:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.432+0000 7fd144df8640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd140152820 0x7fd1400c0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40198/0 (socket says 172.21.15.1:40198) 2024-03-06T07:31:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.432+0000 7fd144df8640 1 -- 172.21.15.1:0/2922795408 learned_addr learned my addr 172.21.15.1:0/2922795408 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:31:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.432+0000 7fd13ffff640 1 --2- 172.21.15.1:0/2922795408 >> v2:172.21.15.1:3300/0 conn(0x7fd1400af6f0 0x7fd1400c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.432+0000 7fd13ffff640 1 -- 172.21.15.1:0/2922795408 >> v1:172.21.15.98:6789/0 conn(0x7fd1400c7930 legacy=0x7fd14015fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.432+0000 7fd13ffff640 1 -- 172.21.15.1:0/2922795408 >> v1:172.21.15.1:6789/0 conn(0x7fd140152820 legacy=0x7fd1400c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.432+0000 7fd13ffff640 1 -- 172.21.15.1:0/2922795408 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1400c6ea0 con 0x7fd1400af6f0 2024-03-06T07:31:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.432+0000 7fd13f7fe640 1 -- 172.21.15.1:0/2922795408 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831646645 0 0) 0x7fd1400c7180 con 0x7fd140152820 2024-03-06T07:31:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.433+0000 7fd13ffff640 1 --2- 172.21.15.1:0/2922795408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1400af6f0 0x7fd1400c73f0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7fd1340097a0 tx=0x7fd13402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b0202d8761f111 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.433+0000 7fd13f7fe640 1 -- 172.21.15.1:0/2922795408 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd134034070 con 0x7fd1400af6f0 2024-03-06T07:31:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.433+0000 7fd13f7fe640 1 -- 172.21.15.1:0/2922795408 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd134030610 con 0x7fd1400af6f0 2024-03-06T07:31:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.433+0000 7fd13f7fe640 1 -- 172.21.15.1:0/2922795408 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1340309e0 con 0x7fd1400af6f0 2024-03-06T07:31:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.434+0000 7fd14705c640 1 -- 172.21.15.1:0/2922795408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1400af6f0 msgr2=0x7fd1400c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.434+0000 7fd14705c640 1 --2- 172.21.15.1:0/2922795408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1400af6f0 0x7fd1400c73f0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7fd1340097a0 tx=0x7fd13402b960 comp rx=0 tx=0).stop 2024-03-06T07:31:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.434+0000 7fd14705c640 1 -- 172.21.15.1:0/2922795408 shutdown_connections 2024-03-06T07:31:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.434+0000 7fd14705c640 1 --2- 172.21.15.1:0/2922795408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1400af6f0 0x7fd1400c73f0 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.434+0000 7fd14705c640 1 -- 172.21.15.1:0/2922795408 >> 172.21.15.1:0/2922795408 conn(0x7fd140072720 msgr2=0x7fd140072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.434+0000 7fd14705c640 1 -- 172.21.15.1:0/2922795408 shutdown_connections 2024-03-06T07:31:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.434+0000 7fd14705c640 1 -- 172.21.15.1:0/2922795408 wait complete. 2024-03-06T07:31:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.435+0000 7fd14705c640 1 Processor -- start 2024-03-06T07:31:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.435+0000 7fd14705c640 1 -- start start 2024-03-06T07:31:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd14705c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1400af6f0 0x7fd1401eb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd14705c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1400c7930 0x7fd140147e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd14705c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd140152820 0x7fd1400c3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd14705c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd140152c00 con 0x7fd1400c7930 2024-03-06T07:31:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd14705c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1400c0840 con 0x7fd140152820 2024-03-06T07:31:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd14705c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd140152d80 con 0x7fd1400af6f0 2024-03-06T07:31:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd13ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1400c7930 0x7fd140147e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd13ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1400c7930 0x7fd140147e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36046/0 (socket says 172.21.15.1:36046) 2024-03-06T07:31:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd13ffff640 1 -- 172.21.15.1:0/1369930853 learned_addr learned my addr 172.21.15.1:0/1369930853 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd13ffff640 1 -- 172.21.15.1:0/1369930853 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1400af6f0 msgr2=0x7fd1401eb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd144df8640 1 --2- 172.21.15.1:0/1369930853 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1400af6f0 0x7fd1401eb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd1455f9640 1 --2- 172.21.15.1:0/1369930853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd140152820 0x7fd1400c3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd13ffff640 1 --2- 172.21.15.1:0/1369930853 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1400af6f0 0x7fd1401eb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd13ffff640 1 -- 172.21.15.1:0/1369930853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd140152820 msgr2=0x7fd1400c3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd13ffff640 1 --2- 172.21.15.1:0/1369930853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd140152820 0x7fd1400c3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd13ffff640 1 -- 172.21.15.1:0/1369930853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1400c98d0 con 0x7fd1400c7930 2024-03-06T07:31:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd144df8640 1 --2- 172.21.15.1:0/1369930853 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1400af6f0 0x7fd1401eb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd1455f9640 1 --2- 172.21.15.1:0/1369930853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd140152820 0x7fd1400c3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:48.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.436+0000 7fd13ffff640 1 --2- 172.21.15.1:0/1369930853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1400c7930 0x7fd140147e00 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7fd13402bf70 tx=0x7fd1340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.437+0000 7fd13d7fa640 1 -- 172.21.15.1:0/1369930853 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd134034040 con 0x7fd1400c7930 2024-03-06T07:31:48.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.437+0000 7fd13d7fa640 1 -- 172.21.15.1:0/1369930853 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd134039a10 con 0x7fd1400c7930 2024-03-06T07:31:48.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.437+0000 7fd14705c640 1 -- 172.21.15.1:0/1369930853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1401eb970 con 0x7fd1400c7930 2024-03-06T07:31:48.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.437+0000 7fd13d7fa640 1 -- 172.21.15.1:0/1369930853 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd134039de0 con 0x7fd1400c7930 2024-03-06T07:31:48.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.437+0000 7fd14705c640 1 -- 172.21.15.1:0/1369930853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1400c9e50 con 0x7fd1400c7930 2024-03-06T07:31:48.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.438+0000 7fd14705c640 1 -- 172.21.15.1:0/1369930853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1400686e0 con 0x7fd1400c7930 2024-03-06T07:31:48.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.438+0000 7fd13d7fa640 1 -- 172.21.15.1:0/1369930853 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd134037070 con 0x7fd1400c7930 2024-03-06T07:31:48.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.438+0000 7fd13d7fa640 1 --2- 172.21.15.1:0/1369930853 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd12c06b500 0x7fd12c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.439+0000 7fd13d7fa640 1 -- 172.21.15.1:0/1369930853 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd1340ac4b0 con 0x7fd1400c7930 2024-03-06T07:31:48.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.439+0000 7fd144df8640 1 --2- 172.21.15.1:0/1369930853 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd12c06b500 0x7fd12c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.439+0000 7fd144df8640 1 --2- 172.21.15.1:0/1369930853 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd12c06b500 0x7fd12c06d9c0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fd130000f60 tx=0x7fd130009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.446+0000 7fd13d7fa640 1 -- 172.21.15.1:0/1369930853 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd13407b250 con 0x7fd1400c7930 2024-03-06T07:31:48.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.734+0000 7fd14705c640 1 -- 172.21.15.1:0/1369930853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd140151930 con 0x7fd1400c7930 2024-03-06T07:31:48.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.734+0000 7fd13d7fa640 1 -- 172.21.15.1:0/1369930853 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fd13407ab50 con 0x7fd1400c7930 2024-03-06T07:31:48.736 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:48.737 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:48.737 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:48.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.736+0000 7fd14705c640 1 -- 172.21.15.1:0/1369930853 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd12c06b500 msgr2=0x7fd12c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.736+0000 7fd14705c640 1 --2- 172.21.15.1:0/1369930853 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd12c06b500 0x7fd12c06d9c0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fd130000f60 tx=0x7fd130009040 comp rx=0 tx=0).stop 2024-03-06T07:31:48.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.736+0000 7fd14705c640 1 -- 172.21.15.1:0/1369930853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1400c7930 msgr2=0x7fd140147e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.736+0000 7fd14705c640 1 --2- 172.21.15.1:0/1369930853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1400c7930 0x7fd140147e00 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7fd13402bf70 tx=0x7fd1340046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.736+0000 7fd14705c640 1 -- 172.21.15.1:0/1369930853 shutdown_connections 2024-03-06T07:31:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.736+0000 7fd14705c640 1 --2- 172.21.15.1:0/1369930853 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd12c06b500 0x7fd12c06d9c0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.736+0000 7fd14705c640 1 --2- 172.21.15.1:0/1369930853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd140152820 0x7fd1400c3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.736+0000 7fd14705c640 1 --2- 172.21.15.1:0/1369930853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1400c7930 0x7fd140147e00 unknown :-1 s=CLOSED pgs=2541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.736+0000 7fd14705c640 1 --2- 172.21.15.1:0/1369930853 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1400af6f0 0x7fd1401eb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.736+0000 7fd14705c640 1 -- 172.21.15.1:0/1369930853 >> 172.21.15.1:0/1369930853 conn(0x7fd140072720 msgr2=0x7fd14014dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.736+0000 7fd14705c640 1 -- 172.21.15.1:0/1369930853 shutdown_connections 2024-03-06T07:31:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:48.736+0000 7fd14705c640 1 -- 172.21.15.1:0/1369930853 wait complete. 2024-03-06T07:31:49.151 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:49.151 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:50.152 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:50.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.252+0000 7ffa97c84640 1 Processor -- start 2024-03-06T07:31:50.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.252+0000 7ffa97c84640 1 -- start start 2024-03-06T07:31:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.253+0000 7ffa97c84640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa90150520 0x7ffa901543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.253+0000 7ffa97c84640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffa90070c80 0x7ffa900710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.253+0000 7ffa97c84640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa90150900 con 0x7ffa900c32f0 2024-03-06T07:31:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.253+0000 7ffa97c84640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa90153ed0 con 0x7ffa90070c80 2024-03-06T07:31:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.253+0000 7ffa97c84640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa90071790 con 0x7ffa90150520 2024-03-06T07:31:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.253+0000 7ffa9521f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffa900c32f0 0x7ffa901537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40210/0 (socket says 172.21.15.1:40210) 2024-03-06T07:31:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.253+0000 7ffa9521f640 1 -- 172.21.15.1:0/3309453244 learned_addr learned my addr 172.21.15.1:0/3309453244 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:31:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa94a1e640 1 -- 172.21.15.1:0/3309453244 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656266552 0 0) 0x7ffa90150900 con 0x7ffa900c32f0 2024-03-06T07:31:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa95a20640 1 --2- 172.21.15.1:0/3309453244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa90150520 0x7ffa901543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa94a1e640 1 -- 172.21.15.1:0/3309453244 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa7c003520 con 0x7ffa900c32f0 2024-03-06T07:31:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa96221640 1 --2- 172.21.15.1:0/3309453244 >> v2:172.21.15.98:3300/0 conn(0x7ffa90070c80 0x7ffa900710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa94a1e640 1 -- 172.21.15.1:0/3309453244 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4050606776 0 0) 0x7ffa7c003520 con 0x7ffa900c32f0 2024-03-06T07:31:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa94a1e640 1 -- 172.21.15.1:0/3309453244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa90150520 msgr2=0x7ffa901543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa94a1e640 1 --2- 172.21.15.1:0/3309453244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa90150520 0x7ffa901543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa94a1e640 1 -- 172.21.15.1:0/3309453244 >> v2:172.21.15.98:3300/0 conn(0x7ffa90070c80 msgr2=0x7ffa900710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa94a1e640 1 --2- 172.21.15.1:0/3309453244 >> v2:172.21.15.98:3300/0 conn(0x7ffa90070c80 0x7ffa900710f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa94a1e640 1 -- 172.21.15.1:0/3309453244 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa90157050 con 0x7ffa900c32f0 2024-03-06T07:31:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa94a1e640 1 -- 172.21.15.1:0/3309453244 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7ffa80001670 con 0x7ffa900c32f0 2024-03-06T07:31:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa94a1e640 1 -- 172.21.15.1:0/3309453244 >> v1:172.21.15.1:6789/0 conn(0x7ffa900c32f0 legacy=0x7ffa901537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa94a1e640 1 --2- 172.21.15.1:0/3309453244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa7c0039b0 0x7ffa7c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa94a1e640 1 --2- 172.21.15.1:0/3309453244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa90150520 0x7ffa7c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa94a1e640 1 --2- 172.21.15.1:0/3309453244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa7c004d40 0x7ffa7c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa94a1e640 1 -- 172.21.15.1:0/3309453244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa90153ed0 con 0x7ffa7c004d40 2024-03-06T07:31:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa94a1e640 1 -- 172.21.15.1:0/3309453244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa90071790 con 0x7ffa7c0039b0 2024-03-06T07:31:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa94a1e640 1 -- 172.21.15.1:0/3309453244 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7c00b9c0 con 0x7ffa90150520 2024-03-06T07:31:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa96221640 1 --2- 172.21.15.1:0/3309453244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa7c004d40 0x7ffa7c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa9521f640 1 --2- 172.21.15.1:0/3309453244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa90150520 0x7ffa7c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa95a20640 1 --2- 172.21.15.1:0/3309453244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa7c0039b0 0x7ffa7c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa96221640 1 -- 172.21.15.1:0/3309453244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa90150520 msgr2=0x7ffa7c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa96221640 1 --2- 172.21.15.1:0/3309453244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa90150520 0x7ffa7c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa96221640 1 -- 172.21.15.1:0/3309453244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa7c0039b0 msgr2=0x7ffa7c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa96221640 1 --2- 172.21.15.1:0/3309453244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa7c0039b0 0x7ffa7c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.254+0000 7ffa96221640 1 -- 172.21.15.1:0/3309453244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7c003520 con 0x7ffa7c004d40 2024-03-06T07:31:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.255+0000 7ffa96221640 1 --2- 172.21.15.1:0/3309453244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa7c004d40 0x7ffa7c0051c0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7ffa88009fc0 tx=0x7ffa7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e8030e9aa17aba5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.255+0000 7ffa94a1e640 1 -- 172.21.15.1:0/3309453244 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa88039070 con 0x7ffa7c004d40 2024-03-06T07:31:50.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.255+0000 7ffa94a1e640 1 -- 172.21.15.1:0/3309453244 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa8802cc60 con 0x7ffa7c004d40 2024-03-06T07:31:50.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.255+0000 7ffa94a1e640 1 -- 172.21.15.1:0/3309453244 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa88009c10 con 0x7ffa7c004d40 2024-03-06T07:31:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.256+0000 7ffa97c84640 1 -- 172.21.15.1:0/3309453244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa7c004d40 msgr2=0x7ffa7c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.256+0000 7ffa97c84640 1 --2- 172.21.15.1:0/3309453244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa7c004d40 0x7ffa7c0051c0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7ffa88009fc0 tx=0x7ffa7c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.256+0000 7ffa96221640 1 -- 172.21.15.1:0/3309453244 reap_dead start 2024-03-06T07:31:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.256+0000 7ffa97c84640 1 -- 172.21.15.1:0/3309453244 shutdown_connections 2024-03-06T07:31:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.256+0000 7ffa97c84640 1 -- 172.21.15.1:0/3309453244 >> 172.21.15.1:0/3309453244 conn(0x7ffa900c88f0 msgr2=0x7ffa900c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.256+0000 7ffa97c84640 1 -- 172.21.15.1:0/3309453244 shutdown_connections 2024-03-06T07:31:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.256+0000 7ffa97c84640 1 -- 172.21.15.1:0/3309453244 wait complete. 2024-03-06T07:31:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.257+0000 7ffa97c84640 1 Processor -- start 2024-03-06T07:31:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.257+0000 7ffa97c84640 1 -- start start 2024-03-06T07:31:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa97c84640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa7c004d40 0x7ffa900adb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa97c84640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa9006de00 0x7ffa900c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa97c84640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa901ea6d0 0x7ffa901eab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa97c84640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa90157330 con 0x7ffa901ea6d0 2024-03-06T07:31:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa97c84640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa901532a0 con 0x7ffa7c004d40 2024-03-06T07:31:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa97c84640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa900c7ec0 con 0x7ffa9006de00 2024-03-06T07:31:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa96221640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa901ea6d0 0x7ffa901eab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa96221640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa901ea6d0 0x7ffa901eab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36070/0 (socket says 172.21.15.1:36070) 2024-03-06T07:31:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa96221640 1 -- 172.21.15.1:0/3553557273 learned_addr learned my addr 172.21.15.1:0/3553557273 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa95a20640 1 --2- 172.21.15.1:0/3553557273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa7c004d40 0x7ffa900adb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa96221640 1 -- 172.21.15.1:0/3553557273 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa9006de00 msgr2=0x7ffa900c1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa9521f640 1 --2- 172.21.15.1:0/3553557273 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa9006de00 0x7ffa900c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa96221640 1 --2- 172.21.15.1:0/3553557273 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa9006de00 0x7ffa900c1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa96221640 1 -- 172.21.15.1:0/3553557273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa7c004d40 msgr2=0x7ffa900adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa96221640 1 --2- 172.21.15.1:0/3553557273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa7c004d40 0x7ffa900adb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa96221640 1 -- 172.21.15.1:0/3553557273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa901ee7f0 con 0x7ffa901ea6d0 2024-03-06T07:31:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa9521f640 1 --2- 172.21.15.1:0/3553557273 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa9006de00 0x7ffa900c1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:50.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.258+0000 7ffa95a20640 1 --2- 172.21.15.1:0/3553557273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa7c004d40 0x7ffa900adb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.259+0000 7ffa96221640 1 --2- 172.21.15.1:0/3553557273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa901ea6d0 0x7ffa901eab80 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7ffa8800e230 tx=0x7ffa88009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.259+0000 7ffa867fc640 1 -- 172.21.15.1:0/3553557273 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa88039030 con 0x7ffa901ea6d0 2024-03-06T07:31:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.259+0000 7ffa867fc640 1 -- 172.21.15.1:0/3553557273 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa8803a040 con 0x7ffa901ea6d0 2024-03-06T07:31:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.259+0000 7ffa867fc640 1 -- 172.21.15.1:0/3553557273 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa88003990 con 0x7ffa901ea6d0 2024-03-06T07:31:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.259+0000 7ffa97c84640 1 -- 172.21.15.1:0/3553557273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa9006c100 con 0x7ffa901ea6d0 2024-03-06T07:31:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.259+0000 7ffa97c84640 1 -- 172.21.15.1:0/3553557273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa900ac0e0 con 0x7ffa901ea6d0 2024-03-06T07:31:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.260+0000 7ffa97c84640 1 -- 172.21.15.1:0/3553557273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa900686e0 con 0x7ffa901ea6d0 2024-03-06T07:31:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.261+0000 7ffa867fc640 1 -- 172.21.15.1:0/3553557273 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa88045030 con 0x7ffa901ea6d0 2024-03-06T07:31:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.261+0000 7ffa867fc640 1 --2- 172.21.15.1:0/3553557273 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa7406b500 0x7ffa7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.261+0000 7ffa867fc640 1 -- 172.21.15.1:0/3553557273 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ffa880ac3b0 con 0x7ffa901ea6d0 2024-03-06T07:31:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.261+0000 7ffa95a20640 1 --2- 172.21.15.1:0/3553557273 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa7406b500 0x7ffa7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.262+0000 7ffa95a20640 1 --2- 172.21.15.1:0/3553557273 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa7406b500 0x7ffa7406d9c0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c009a40 tx=0x7ffa8c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.265+0000 7ffa867fc640 1 -- 172.21.15.1:0/3553557273 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa8804c020 con 0x7ffa901ea6d0 2024-03-06T07:31:50.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.484+0000 7ffa97c84640 1 -- 172.21.15.1:0/3553557273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa900c7b10 con 0x7ffa901ea6d0 2024-03-06T07:31:50.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.485+0000 7ffa867fc640 1 -- 172.21.15.1:0/3553557273 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7ffa8807b030 con 0x7ffa901ea6d0 2024-03-06T07:31:50.487 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:50.487 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:50.487 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:50.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.487+0000 7ffa97c84640 1 -- 172.21.15.1:0/3553557273 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa7406b500 msgr2=0x7ffa7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.487+0000 7ffa97c84640 1 --2- 172.21.15.1:0/3553557273 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa7406b500 0x7ffa7406d9c0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c009a40 tx=0x7ffa8c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:50.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.487+0000 7ffa97c84640 1 -- 172.21.15.1:0/3553557273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa901ea6d0 msgr2=0x7ffa901eab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.487+0000 7ffa97c84640 1 --2- 172.21.15.1:0/3553557273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa901ea6d0 0x7ffa901eab80 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7ffa8800e230 tx=0x7ffa88009750 comp rx=0 tx=0).stop 2024-03-06T07:31:50.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.487+0000 7ffa97c84640 1 -- 172.21.15.1:0/3553557273 shutdown_connections 2024-03-06T07:31:50.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.487+0000 7ffa97c84640 1 --2- 172.21.15.1:0/3553557273 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ffa7406b500 0x7ffa7406d9c0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.487+0000 7ffa97c84640 1 --2- 172.21.15.1:0/3553557273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa901ea6d0 0x7ffa901eab80 unknown :-1 s=CLOSED pgs=2544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.487+0000 7ffa97c84640 1 --2- 172.21.15.1:0/3553557273 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa9006de00 0x7ffa900c1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.487+0000 7ffa97c84640 1 --2- 172.21.15.1:0/3553557273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa7c004d40 0x7ffa900adb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.487+0000 7ffa97c84640 1 -- 172.21.15.1:0/3553557273 >> 172.21.15.1:0/3553557273 conn(0x7ffa900c88f0 msgr2=0x7ffa9014c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.487+0000 7ffa97c84640 1 -- 172.21.15.1:0/3553557273 shutdown_connections 2024-03-06T07:31:50.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:50.487+0000 7ffa97c84640 1 -- 172.21.15.1:0/3553557273 wait complete. 2024-03-06T07:31:50.902 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:50.902 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:51.903 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:52.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.002+0000 7f9d4f8c9640 1 Processor -- start 2024-03-06T07:31:52.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.003+0000 7f9d4f8c9640 1 -- start start 2024-03-06T07:31:52.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.003+0000 7f9d4f8c9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9d48152650 0x7f9d48155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.003+0000 7f9d4f8c9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9d480bf9d0 0x7f9d480c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.003+0000 7f9d4f8c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d480c2fc0 con 0x7f9d480c6b10 2024-03-06T07:31:52.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.003+0000 7f9d4f8c9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d48155800 con 0x7f9d480bf9d0 2024-03-06T07:31:52.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.003+0000 7f9d4f8c9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d48158da0 con 0x7f9d48152650 2024-03-06T07:31:52.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.003+0000 7f9d4de66640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9d480bf9d0 0x7f9d480c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.003+0000 7f9d4de66640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9d480bf9d0 0x7f9d480c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36076/0 (socket says 172.21.15.1:36076) 2024-03-06T07:31:52.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.003+0000 7f9d4de66640 1 -- 172.21.15.1:0/3256356499 learned_addr learned my addr 172.21.15.1:0/3256356499 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:52.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.004+0000 7f9d4de66640 1 -- 172.21.15.1:0/3256356499 >> v2:172.21.15.98:3300/0 conn(0x7f9d48152650 msgr2=0x7f9d48155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:52.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.004+0000 7f9d4de66640 1 --2- 172.21.15.1:0/3256356499 >> v2:172.21.15.98:3300/0 conn(0x7f9d48152650 0x7f9d48155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.004+0000 7f9d4de66640 1 -- 172.21.15.1:0/3256356499 >> v1:172.21.15.98:6789/0 conn(0x7f9d480c6b10 legacy=0x7f9d48158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:52.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.004+0000 7f9d4de66640 1 -- 172.21.15.1:0/3256356499 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d480c6690 con 0x7f9d480bf9d0 2024-03-06T07:31:52.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.004+0000 7f9d4de66640 1 --2- 172.21.15.1:0/3256356499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d480bf9d0 0x7f9d480c1460 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f9d480cac90 tx=0x7f9d4002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66484fd411c24071 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.005+0000 7f9d3ffff640 1 -- 172.21.15.1:0/3256356499 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4002c9f0 con 0x7f9d480bf9d0 2024-03-06T07:31:52.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.005+0000 7f9d3ffff640 1 -- 172.21.15.1:0/3256356499 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4002cb90 con 0x7f9d480bf9d0 2024-03-06T07:31:52.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.005+0000 7f9d3ffff640 1 -- 172.21.15.1:0/3256356499 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d400343f0 con 0x7f9d480bf9d0 2024-03-06T07:31:52.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.005+0000 7f9d4f8c9640 1 -- 172.21.15.1:0/3256356499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d480bf9d0 msgr2=0x7f9d480c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.005+0000 7f9d4f8c9640 1 --2- 172.21.15.1:0/3256356499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d480bf9d0 0x7f9d480c1460 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f9d480cac90 tx=0x7f9d4002be50 comp rx=0 tx=0).stop 2024-03-06T07:31:52.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.005+0000 7f9d4f8c9640 1 -- 172.21.15.1:0/3256356499 shutdown_connections 2024-03-06T07:31:52.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.005+0000 7f9d4f8c9640 1 --2- 172.21.15.1:0/3256356499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d480bf9d0 0x7f9d480c1460 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.005+0000 7f9d4f8c9640 1 --2- 172.21.15.1:0/3256356499 >> v2:172.21.15.98:3300/0 conn(0x7f9d48152650 0x7f9d48155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.005+0000 7f9d4f8c9640 1 -- 172.21.15.1:0/3256356499 >> 172.21.15.1:0/3256356499 conn(0x7f9d48066ee0 msgr2=0x7f9d480672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:52.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.005+0000 7f9d4f8c9640 1 -- 172.21.15.1:0/3256356499 shutdown_connections 2024-03-06T07:31:52.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.005+0000 7f9d4f8c9640 1 -- 172.21.15.1:0/3256356499 wait complete. 2024-03-06T07:31:52.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.006+0000 7f9d4f8c9640 1 Processor -- start 2024-03-06T07:31:52.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.006+0000 7f9d4f8c9640 1 -- start start 2024-03-06T07:31:52.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4f8c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d480bf9d0 0x7f9d48158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4f8c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d48152650 0x7f9d480c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4f8c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d4814b670 0x7f9d480c5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4f8c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d481551b0 con 0x7f9d48152650 2024-03-06T07:31:52.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4f8c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d480c6fa0 con 0x7f9d480bf9d0 2024-03-06T07:31:52.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4f8c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d48155490 con 0x7f9d4814b670 2024-03-06T07:31:52.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4ce64640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d48152650 0x7f9d480c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4ce64640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d48152650 0x7f9d480c8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36086/0 (socket says 172.21.15.1:36086) 2024-03-06T07:31:52.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4ce64640 1 -- 172.21.15.1:0/2833708066 learned_addr learned my addr 172.21.15.1:0/2833708066 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:52.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4de66640 1 --2- 172.21.15.1:0/2833708066 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d4814b670 0x7f9d480c5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4d665640 1 --2- 172.21.15.1:0/2833708066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d480bf9d0 0x7f9d48158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4ce64640 1 -- 172.21.15.1:0/2833708066 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d4814b670 msgr2=0x7f9d480c5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4ce64640 1 --2- 172.21.15.1:0/2833708066 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d4814b670 0x7f9d480c5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4ce64640 1 -- 172.21.15.1:0/2833708066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d480bf9d0 msgr2=0x7f9d48158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4ce64640 1 --2- 172.21.15.1:0/2833708066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d480bf9d0 0x7f9d48158790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4ce64640 1 -- 172.21.15.1:0/2833708066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d481eec10 con 0x7f9d48152650 2024-03-06T07:31:52.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4de66640 1 --2- 172.21.15.1:0/2833708066 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d4814b670 0x7f9d480c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:52.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.007+0000 7f9d4d665640 1 --2- 172.21.15.1:0/2833708066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d480bf9d0 0x7f9d48158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:52.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.008+0000 7f9d4ce64640 1 --2- 172.21.15.1:0/2833708066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d48152650 0x7f9d480c8940 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800c410 tx=0x7f9d3800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.008+0000 7f9d3dffb640 1 -- 172.21.15.1:0/2833708066 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d38017070 con 0x7f9d48152650 2024-03-06T07:31:52.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.008+0000 7f9d3dffb640 1 -- 172.21.15.1:0/2833708066 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d380035f0 con 0x7f9d48152650 2024-03-06T07:31:52.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.008+0000 7f9d4f8c9640 1 -- 172.21.15.1:0/2833708066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d481eeef0 con 0x7f9d48152650 2024-03-06T07:31:52.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.008+0000 7f9d3dffb640 1 -- 172.21.15.1:0/2833708066 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d380039e0 con 0x7f9d48152650 2024-03-06T07:31:52.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.008+0000 7f9d4f8c9640 1 -- 172.21.15.1:0/2833708066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d48154120 con 0x7f9d48152650 2024-03-06T07:31:52.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.009+0000 7f9d4f8c9640 1 -- 172.21.15.1:0/2833708066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d480b06f0 con 0x7f9d48152650 2024-03-06T07:31:52.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.010+0000 7f9d3dffb640 1 -- 172.21.15.1:0/2833708066 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d3801a070 con 0x7f9d48152650 2024-03-06T07:31:52.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.010+0000 7f9d3dffb640 1 --2- 172.21.15.1:0/2833708066 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9d2c06b4d0 0x7f9d2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.010+0000 7f9d3dffb640 1 -- 172.21.15.1:0/2833708066 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f9d38089920 con 0x7f9d48152650 2024-03-06T07:31:52.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.010+0000 7f9d4d665640 1 --2- 172.21.15.1:0/2833708066 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9d2c06b4d0 0x7f9d2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.011+0000 7f9d4d665640 1 --2- 172.21.15.1:0/2833708066 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9d2c06b4d0 0x7f9d2c06d990 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009930 tx=0x7f9d44008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.014+0000 7f9d3dffb640 1 -- 172.21.15.1:0/2833708066 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d38058510 con 0x7f9d48152650 2024-03-06T07:31:52.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.234+0000 7f9d4f8c9640 1 -- 172.21.15.1:0/2833708066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d481ef0c0 con 0x7f9d48152650 2024-03-06T07:31:52.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.235+0000 7f9d3dffb640 1 -- 172.21.15.1:0/2833708066 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f9d38057e10 con 0x7f9d48152650 2024-03-06T07:31:52.237 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:52.237 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:52.237 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:52.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.237+0000 7f9d4f8c9640 1 -- 172.21.15.1:0/2833708066 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9d2c06b4d0 msgr2=0x7f9d2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.237+0000 7f9d4f8c9640 1 --2- 172.21.15.1:0/2833708066 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9d2c06b4d0 0x7f9d2c06d990 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009930 tx=0x7f9d44008040 comp rx=0 tx=0).stop 2024-03-06T07:31:52.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.237+0000 7f9d4f8c9640 1 -- 172.21.15.1:0/2833708066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d48152650 msgr2=0x7f9d480c8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.237+0000 7f9d4f8c9640 1 --2- 172.21.15.1:0/2833708066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d48152650 0x7f9d480c8940 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800c410 tx=0x7f9d3800c8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.237+0000 7f9d4f8c9640 1 -- 172.21.15.1:0/2833708066 shutdown_connections 2024-03-06T07:31:52.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.237+0000 7f9d4f8c9640 1 --2- 172.21.15.1:0/2833708066 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9d2c06b4d0 0x7f9d2c06d990 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.237+0000 7f9d4f8c9640 1 --2- 172.21.15.1:0/2833708066 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d4814b670 0x7f9d480c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.237+0000 7f9d4f8c9640 1 --2- 172.21.15.1:0/2833708066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d48152650 0x7f9d480c8940 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.237+0000 7f9d4f8c9640 1 --2- 172.21.15.1:0/2833708066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d480bf9d0 0x7f9d48158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.237+0000 7f9d4f8c9640 1 -- 172.21.15.1:0/2833708066 >> 172.21.15.1:0/2833708066 conn(0x7f9d48066ee0 msgr2=0x7f9d480bfdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:52.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.237+0000 7f9d4f8c9640 1 -- 172.21.15.1:0/2833708066 shutdown_connections 2024-03-06T07:31:52.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:52.237+0000 7f9d4f8c9640 1 -- 172.21.15.1:0/2833708066 wait complete. 2024-03-06T07:31:52.651 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:52.652 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:53.652 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:53.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.752+0000 7f5cd39bc640 1 Processor -- start 2024-03-06T07:31:53.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.752+0000 7f5cd39bc640 1 -- start start 2024-03-06T07:31:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.753+0000 7f5cd39bc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5ccc0c6b10 0x7f5ccc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.753+0000 7f5cd39bc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5ccc0bf9d0 0x7f5ccc0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.753+0000 7f5cd39bc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc0c2fc0 con 0x7f5ccc152650 2024-03-06T07:31:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.753+0000 7f5cd39bc640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ccc155800 con 0x7f5ccc0bf9d0 2024-03-06T07:31:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.753+0000 7f5cd39bc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ccc158da0 con 0x7f5ccc0c6b10 2024-03-06T07:31:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.753+0000 7f5cd1f59640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5ccc0bf9d0 0x7f5ccc0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.753+0000 7f5cd1f59640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5ccc0bf9d0 0x7f5ccc0c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36096/0 (socket says 172.21.15.1:36096) 2024-03-06T07:31:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.753+0000 7f5cd1f59640 1 -- 172.21.15.1:0/725725402 learned_addr learned my addr 172.21.15.1:0/725725402 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.753+0000 7f5cd1758640 1 --1- 172.21.15.1:0/725725402 >> v1:172.21.15.1:6789/0 conn(0x7f5ccc152650 0x7f5ccc158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40226/0 (socket says 172.21.15.1:40226) 2024-03-06T07:31:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.753+0000 7f5cd0f57640 1 --2- 172.21.15.1:0/725725402 >> v2:172.21.15.98:3300/0 conn(0x7f5ccc0c6b10 0x7f5ccc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.753+0000 7f5cd1f59640 1 -- 172.21.15.1:0/725725402 >> v2:172.21.15.98:3300/0 conn(0x7f5ccc0c6b10 msgr2=0x7f5ccc155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.753+0000 7f5cd1f59640 1 --2- 172.21.15.1:0/725725402 >> v2:172.21.15.98:3300/0 conn(0x7f5ccc0c6b10 0x7f5ccc155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.753+0000 7f5cd1f59640 1 -- 172.21.15.1:0/725725402 >> v1:172.21.15.1:6789/0 conn(0x7f5ccc152650 legacy=0x7f5ccc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.753+0000 7f5cd1f59640 1 -- 172.21.15.1:0/725725402 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc0c6690 con 0x7f5ccc0bf9d0 2024-03-06T07:31:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.753+0000 7f5cd0f57640 1 --2- 172.21.15.1:0/725725402 >> v2:172.21.15.98:3300/0 conn(0x7f5ccc0c6b10 0x7f5ccc155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:53.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.754+0000 7f5cd1f59640 1 --2- 172.21.15.1:0/725725402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc0bf9d0 0x7f5ccc0c1460 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc0cb8e0 tx=0x7f5cc402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6cb4e221fc368a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.754+0000 7f5cc3fff640 1 -- 172.21.15.1:0/725725402 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc402ca80 con 0x7f5ccc0bf9d0 2024-03-06T07:31:53.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.754+0000 7f5cc3fff640 1 -- 172.21.15.1:0/725725402 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc402cc20 con 0x7f5ccc0bf9d0 2024-03-06T07:31:53.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.754+0000 7f5cc3fff640 1 -- 172.21.15.1:0/725725402 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc40343f0 con 0x7f5ccc0bf9d0 2024-03-06T07:31:53.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.754+0000 7f5cd39bc640 1 -- 172.21.15.1:0/725725402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc0bf9d0 msgr2=0x7f5ccc0c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.754+0000 7f5cd39bc640 1 --2- 172.21.15.1:0/725725402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc0bf9d0 0x7f5ccc0c1460 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc0cb8e0 tx=0x7f5cc402bee0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.754+0000 7f5cd39bc640 1 -- 172.21.15.1:0/725725402 shutdown_connections 2024-03-06T07:31:53.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.754+0000 7f5cd39bc640 1 --2- 172.21.15.1:0/725725402 >> v2:172.21.15.98:3300/0 conn(0x7f5ccc0c6b10 0x7f5ccc155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.754+0000 7f5cd39bc640 1 --2- 172.21.15.1:0/725725402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc0bf9d0 0x7f5ccc0c1460 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.754+0000 7f5cd39bc640 1 -- 172.21.15.1:0/725725402 >> 172.21.15.1:0/725725402 conn(0x7f5ccc066ee0 msgr2=0x7f5ccc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.755+0000 7f5cd39bc640 1 -- 172.21.15.1:0/725725402 shutdown_connections 2024-03-06T07:31:53.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.755+0000 7f5cd39bc640 1 -- 172.21.15.1:0/725725402 wait complete. 2024-03-06T07:31:53.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.756+0000 7f5cd39bc640 1 Processor -- start 2024-03-06T07:31:53.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.756+0000 7f5cd39bc640 1 -- start start 2024-03-06T07:31:53.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.756+0000 7f5cd39bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ccc0bf9d0 0x7f5ccc14b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.756+0000 7f5cd39bc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc152650 0x7f5ccc0c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.756+0000 7f5cd39bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ccc14a620 0x7f5ccc14aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.756+0000 7f5cd39bc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc1551b0 con 0x7f5ccc152650 2024-03-06T07:31:53.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.756+0000 7f5cd39bc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0c6fa0 con 0x7f5ccc14a620 2024-03-06T07:31:53.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.756+0000 7f5cd39bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc155490 con 0x7f5ccc0bf9d0 2024-03-06T07:31:53.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.756+0000 7f5cd0f57640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc152650 0x7f5ccc0c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.756+0000 7f5cd0f57640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc152650 0x7f5ccc0c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36108/0 (socket says 172.21.15.1:36108) 2024-03-06T07:31:53.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.757+0000 7f5cd0f57640 1 -- 172.21.15.1:0/1486315024 learned_addr learned my addr 172.21.15.1:0/1486315024 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:53.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.757+0000 7f5cd1758640 1 --2- 172.21.15.1:0/1486315024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ccc0bf9d0 0x7f5ccc14b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.757+0000 7f5cd1f59640 1 --2- 172.21.15.1:0/1486315024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ccc14a620 0x7f5ccc14aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.757+0000 7f5cd0f57640 1 -- 172.21.15.1:0/1486315024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ccc0bf9d0 msgr2=0x7f5ccc14b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.757+0000 7f5cd0f57640 1 --2- 172.21.15.1:0/1486315024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ccc0bf9d0 0x7f5ccc14b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.757+0000 7f5cd0f57640 1 -- 172.21.15.1:0/1486315024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ccc14a620 msgr2=0x7f5ccc14aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.757+0000 7f5cd0f57640 1 --2- 172.21.15.1:0/1486315024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ccc14a620 0x7f5ccc14aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.757+0000 7f5cd0f57640 1 -- 172.21.15.1:0/1486315024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc14b090 con 0x7f5ccc152650 2024-03-06T07:31:53.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.757+0000 7f5cd0f57640 1 --2- 172.21.15.1:0/1486315024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc152650 0x7f5ccc0c9a90 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00c7f0 tx=0x7f5cbc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.757+0000 7f5cc1ffb640 1 -- 172.21.15.1:0/1486315024 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc015070 con 0x7f5ccc152650 2024-03-06T07:31:53.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.757+0000 7f5cc1ffb640 1 -- 172.21.15.1:0/1486315024 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc0079d0 con 0x7f5ccc152650 2024-03-06T07:31:53.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.757+0000 7f5cc1ffb640 1 -- 172.21.15.1:0/1486315024 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc007da0 con 0x7f5ccc152650 2024-03-06T07:31:53.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.757+0000 7f5cd39bc640 1 -- 172.21.15.1:0/1486315024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc1eb5f0 con 0x7f5ccc152650 2024-03-06T07:31:53.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.757+0000 7f5cd39bc640 1 -- 172.21.15.1:0/1486315024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc154530 con 0x7f5ccc152650 2024-03-06T07:31:53.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.758+0000 7f5cd39bc640 1 -- 172.21.15.1:0/1486315024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ccc154730 con 0x7f5ccc152650 2024-03-06T07:31:53.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.759+0000 7f5cc1ffb640 1 -- 172.21.15.1:0/1486315024 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5cbc018070 con 0x7f5ccc152650 2024-03-06T07:31:53.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.760+0000 7f5cc1ffb640 1 --2- 172.21.15.1:0/1486315024 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5cb006b4d0 0x7f5cb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.760+0000 7f5cc1ffb640 1 -- 172.21.15.1:0/1486315024 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f5cbc08a340 con 0x7f5ccc152650 2024-03-06T07:31:53.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.760+0000 7f5cd1758640 1 --2- 172.21.15.1:0/1486315024 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5cb006b4d0 0x7f5cb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.760+0000 7f5cd1758640 1 --2- 172.21.15.1:0/1486315024 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5cb006b4d0 0x7f5cb006d990 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8000f60 tx=0x7f5cc8007cc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.764+0000 7f5cc1ffb640 1 -- 172.21.15.1:0/1486315024 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ccc154730 con 0x7f5ccc152650 2024-03-06T07:31:53.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.983+0000 7f5cd39bc640 1 -- 172.21.15.1:0/1486315024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ccc067fd0 con 0x7f5ccc152650 2024-03-06T07:31:53.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.983+0000 7f5cc1ffb640 1 -- 172.21.15.1:0/1486315024 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f5cbc058fc0 con 0x7f5ccc152650 2024-03-06T07:31:53.986 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:53.986 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:53.986 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:53.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.985+0000 7f5cd39bc640 1 -- 172.21.15.1:0/1486315024 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5cb006b4d0 msgr2=0x7f5cb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.985+0000 7f5cd39bc640 1 --2- 172.21.15.1:0/1486315024 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5cb006b4d0 0x7f5cb006d990 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8000f60 tx=0x7f5cc8007cc0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.985+0000 7f5cd39bc640 1 -- 172.21.15.1:0/1486315024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc152650 msgr2=0x7f5ccc0c9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.985+0000 7f5cd39bc640 1 --2- 172.21.15.1:0/1486315024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc152650 0x7f5ccc0c9a90 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00c7f0 tx=0x7f5cbc00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.985+0000 7f5cd39bc640 1 -- 172.21.15.1:0/1486315024 shutdown_connections 2024-03-06T07:31:53.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.985+0000 7f5cd39bc640 1 --2- 172.21.15.1:0/1486315024 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5cb006b4d0 0x7f5cb006d990 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.985+0000 7f5cd39bc640 1 --2- 172.21.15.1:0/1486315024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ccc14a620 0x7f5ccc14aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.985+0000 7f5cd39bc640 1 --2- 172.21.15.1:0/1486315024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc152650 0x7f5ccc0c9a90 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.985+0000 7f5cd39bc640 1 --2- 172.21.15.1:0/1486315024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5ccc0bf9d0 0x7f5ccc14b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.985+0000 7f5cd39bc640 1 -- 172.21.15.1:0/1486315024 >> 172.21.15.1:0/1486315024 conn(0x7f5ccc066ee0 msgr2=0x7f5ccc0c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.986+0000 7f5cd39bc640 1 -- 172.21.15.1:0/1486315024 shutdown_connections 2024-03-06T07:31:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:53.986+0000 7f5cd39bc640 1 -- 172.21.15.1:0/1486315024 wait complete. 2024-03-06T07:31:54.401 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:54.401 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:55.402 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:55.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.502+0000 7f3e0b8df640 1 Processor -- start 2024-03-06T07:31:55.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.503+0000 7f3e0b8df640 1 -- start start 2024-03-06T07:31:55.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.503+0000 7f3e0b8df640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3e04150520 0x7f3e041543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.503+0000 7f3e0b8df640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e04150900 con 0x7f3e040c32f0 2024-03-06T07:31:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.503+0000 7f3e0b8df640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e04071790 con 0x7f3e04070c80 2024-03-06T07:31:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.503+0000 7f3e0b8df640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e04153ed0 con 0x7f3e04150520 2024-03-06T07:31:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.503+0000 7f3e08e7a640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3e040c32f0 0x7f3e041537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54416/0 (socket says 172.21.15.1:54416) 2024-03-06T07:31:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.503+0000 7f3e08e7a640 1 -- 172.21.15.1:0/1977780339 learned_addr learned my addr 172.21.15.1:0/1977780339 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:31:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3dfbfff640 1 -- 172.21.15.1:0/1977780339 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667092916 0 0) 0x7f3e04150900 con 0x7f3e040c32f0 2024-03-06T07:31:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3e0967b640 1 --2- 172.21.15.1:0/1977780339 >> v2:172.21.15.98:3300/0 conn(0x7f3e04150520 0x7f3e041543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3dfbfff640 1 -- 172.21.15.1:0/1977780339 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3df0003520 con 0x7f3e040c32f0 2024-03-06T07:31:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3dfbfff640 1 -- 172.21.15.1:0/1977780339 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2990980854 0 0) 0x7f3df0003520 con 0x7f3e040c32f0 2024-03-06T07:31:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3dfbfff640 1 -- 172.21.15.1:0/1977780339 >> v2:172.21.15.98:3300/0 conn(0x7f3e04150520 msgr2=0x7f3e041543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3dfbfff640 1 --2- 172.21.15.1:0/1977780339 >> v2:172.21.15.98:3300/0 conn(0x7f3e04150520 0x7f3e041543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3dfbfff640 1 -- 172.21.15.1:0/1977780339 >> v1:172.21.15.98:6789/0 conn(0x7f3e04070c80 legacy=0x7f3e0415d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3dfbfff640 1 -- 172.21.15.1:0/1977780339 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e04157050 con 0x7f3e040c32f0 2024-03-06T07:31:55.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3dfbfff640 1 -- 172.21.15.1:0/1977780339 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f3df4001670 con 0x7f3e040c32f0 2024-03-06T07:31:55.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3dfbfff640 1 -- 172.21.15.1:0/1977780339 >> v1:172.21.15.1:6789/0 conn(0x7f3e040c32f0 legacy=0x7f3e041537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3dfbfff640 1 --2- 172.21.15.1:0/1977780339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3df00039b0 0x7f3df0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3dfbfff640 1 --2- 172.21.15.1:0/1977780339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df0004800 0x7f3df0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3dfbfff640 1 --2- 172.21.15.1:0/1977780339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3df00051c0 0x7f3df000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3dfbfff640 1 -- 172.21.15.1:0/1977780339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e04153ed0 con 0x7f3df0004800 2024-03-06T07:31:55.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3dfbfff640 1 -- 172.21.15.1:0/1977780339 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df000c060 con 0x7f3df00051c0 2024-03-06T07:31:55.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3dfbfff640 1 -- 172.21.15.1:0/1977780339 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df000c260 con 0x7f3df00039b0 2024-03-06T07:31:55.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3e08e7a640 1 --2- 172.21.15.1:0/1977780339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df0004800 0x7f3df0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3e09e7c640 1 --2- 172.21.15.1:0/1977780339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3df00051c0 0x7f3df000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.504+0000 7f3e0967b640 1 --2- 172.21.15.1:0/1977780339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3df00039b0 0x7f3df0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.505+0000 7f3e08e7a640 1 -- 172.21.15.1:0/1977780339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3df00039b0 msgr2=0x7f3df0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.505+0000 7f3e08e7a640 1 --2- 172.21.15.1:0/1977780339 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3df00039b0 0x7f3df0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.505+0000 7f3e08e7a640 1 -- 172.21.15.1:0/1977780339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3df00051c0 msgr2=0x7f3df000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.505+0000 7f3e08e7a640 1 --2- 172.21.15.1:0/1977780339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3df00051c0 0x7f3df000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.505+0000 7f3e08e7a640 1 -- 172.21.15.1:0/1977780339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df0003520 con 0x7f3df0004800 2024-03-06T07:31:55.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.505+0000 7f3e09e7c640 1 -- 172.21.15.1:0/1977780339 reap_dead start 2024-03-06T07:31:55.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.505+0000 7f3e08e7a640 1 --2- 172.21.15.1:0/1977780339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df0004800 0x7f3df0004c80 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f3df400c240 tx=0x7f3df0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea3171b4ff82187 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.505+0000 7f3dfbfff640 1 -- 172.21.15.1:0/1977780339 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4007d10 con 0x7f3df0004800 2024-03-06T07:31:55.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.505+0000 7f3dfbfff640 1 -- 172.21.15.1:0/1977780339 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df4030070 con 0x7f3df0004800 2024-03-06T07:31:55.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.505+0000 7f3dfbfff640 1 -- 172.21.15.1:0/1977780339 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df402cdc0 con 0x7f3df0004800 2024-03-06T07:31:55.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.506+0000 7f3e0b8df640 1 -- 172.21.15.1:0/1977780339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df0004800 msgr2=0x7f3df0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.506+0000 7f3e0b8df640 1 --2- 172.21.15.1:0/1977780339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df0004800 0x7f3df0004c80 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f3df400c240 tx=0x7f3df0005830 comp rx=0 tx=0).stop 2024-03-06T07:31:55.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.506+0000 7f3e0b8df640 1 -- 172.21.15.1:0/1977780339 shutdown_connections 2024-03-06T07:31:55.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.506+0000 7f3e0b8df640 1 --2- 172.21.15.1:0/1977780339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df0004800 0x7f3df0004c80 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.506+0000 7f3e0b8df640 1 -- 172.21.15.1:0/1977780339 >> 172.21.15.1:0/1977780339 conn(0x7f3e040c88f0 msgr2=0x7f3e040c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.506+0000 7f3e0b8df640 1 -- 172.21.15.1:0/1977780339 shutdown_connections 2024-03-06T07:31:55.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.506+0000 7f3e0b8df640 1 -- 172.21.15.1:0/1977780339 wait complete. 2024-03-06T07:31:55.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.507+0000 7f3e0b8df640 1 Processor -- start 2024-03-06T07:31:55.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.507+0000 7f3e0b8df640 1 -- start start 2024-03-06T07:31:55.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.508+0000 7f3e0b8df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df0004800 0x7f3e04155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.508+0000 7f3e0b8df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e041e99a0 0x7f3e041490a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.508+0000 7f3e0b8df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e041495e0 0x7f3e041ebd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.508+0000 7f3e0b8df640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e04157330 con 0x7f3df0004800 2024-03-06T07:31:55.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.508+0000 7f3e0b8df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e041532a0 con 0x7f3e041e99a0 2024-03-06T07:31:55.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.508+0000 7f3e0b8df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e040c7ec0 con 0x7f3e041495e0 2024-03-06T07:31:55.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.508+0000 7f3e0967b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df0004800 0x7f3e04155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.508+0000 7f3e0967b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df0004800 0x7f3e04155a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51942/0 (socket says 172.21.15.1:51942) 2024-03-06T07:31:55.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.508+0000 7f3e0967b640 1 -- 172.21.15.1:0/3704503393 learned_addr learned my addr 172.21.15.1:0/3704503393 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:55.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.508+0000 7f3e09e7c640 1 --2- 172.21.15.1:0/3704503393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e041495e0 0x7f3e041ebd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.508+0000 7f3e08e7a640 1 --2- 172.21.15.1:0/3704503393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e041e99a0 0x7f3e041490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.508+0000 7f3e0967b640 1 -- 172.21.15.1:0/3704503393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e041495e0 msgr2=0x7f3e041ebd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.508+0000 7f3e0967b640 1 --2- 172.21.15.1:0/3704503393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e041495e0 0x7f3e041ebd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.508+0000 7f3e0967b640 1 -- 172.21.15.1:0/3704503393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e041e99a0 msgr2=0x7f3e041490a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.508+0000 7f3e0967b640 1 --2- 172.21.15.1:0/3704503393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e041e99a0 0x7f3e041490a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.508+0000 7f3e0967b640 1 -- 172.21.15.1:0/3704503393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e04155f60 con 0x7f3df0004800 2024-03-06T07:31:55.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.508+0000 7f3e0967b640 1 --2- 172.21.15.1:0/3704503393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df0004800 0x7f3e04155a20 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f3e00011a70 tx=0x7f3e00011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.509+0000 7f3df9ffb640 1 -- 172.21.15.1:0/3704503393 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0000fb50 con 0x7f3df0004800 2024-03-06T07:31:55.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.509+0000 7f3df9ffb640 1 -- 172.21.15.1:0/3704503393 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e0000fcf0 con 0x7f3df0004800 2024-03-06T07:31:55.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.509+0000 7f3e0b8df640 1 -- 172.21.15.1:0/3704503393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e040be970 con 0x7f3df0004800 2024-03-06T07:31:55.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.509+0000 7f3df9ffb640 1 -- 172.21.15.1:0/3704503393 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0000fea0 con 0x7f3df0004800 2024-03-06T07:31:55.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.509+0000 7f3e0b8df640 1 -- 172.21.15.1:0/3704503393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e04155220 con 0x7f3df0004800 2024-03-06T07:31:55.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.510+0000 7f3e0b8df640 1 -- 172.21.15.1:0/3704503393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e040686e0 con 0x7f3df0004800 2024-03-06T07:31:55.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.510+0000 7f3df9ffb640 1 -- 172.21.15.1:0/3704503393 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e00012040 con 0x7f3df0004800 2024-03-06T07:31:55.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.511+0000 7f3df9ffb640 1 --2- 172.21.15.1:0/3704503393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3de806b500 0x7f3de806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.511+0000 7f3df9ffb640 1 -- 172.21.15.1:0/3704503393 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f3e0008a910 con 0x7f3df0004800 2024-03-06T07:31:55.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.511+0000 7f3e08e7a640 1 --2- 172.21.15.1:0/3704503393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3de806b500 0x7f3de806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.512+0000 7f3e08e7a640 1 --2- 172.21.15.1:0/3704503393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3de806b500 0x7f3de806d9c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f3df4034000 tx=0x7f3df4009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.515+0000 7f3df9ffb640 1 -- 172.21.15.1:0/3704503393 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3e00019030 con 0x7f3df0004800 2024-03-06T07:31:55.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.736+0000 7f3e0b8df640 1 -- 172.21.15.1:0/3704503393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e040c7b10 con 0x7f3df0004800 2024-03-06T07:31:55.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.736+0000 7f3df9ffb640 1 -- 172.21.15.1:0/3704503393 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f3e00058f40 con 0x7f3df0004800 2024-03-06T07:31:55.738 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:55.738 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:55.738 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:55.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.738+0000 7f3e0b8df640 1 -- 172.21.15.1:0/3704503393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3de806b500 msgr2=0x7f3de806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.738+0000 7f3e0b8df640 1 --2- 172.21.15.1:0/3704503393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3de806b500 0x7f3de806d9c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f3df4034000 tx=0x7f3df4009030 comp rx=0 tx=0).stop 2024-03-06T07:31:55.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.738+0000 7f3e0b8df640 1 -- 172.21.15.1:0/3704503393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df0004800 msgr2=0x7f3e04155a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.738+0000 7f3e0b8df640 1 --2- 172.21.15.1:0/3704503393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df0004800 0x7f3e04155a20 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f3e00011a70 tx=0x7f3e00011f30 comp rx=0 tx=0).stop 2024-03-06T07:31:55.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.738+0000 7f3e0b8df640 1 -- 172.21.15.1:0/3704503393 shutdown_connections 2024-03-06T07:31:55.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.738+0000 7f3e0b8df640 1 --2- 172.21.15.1:0/3704503393 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3de806b500 0x7f3de806d9c0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.738+0000 7f3e0b8df640 1 --2- 172.21.15.1:0/3704503393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e041495e0 0x7f3e041ebd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.738+0000 7f3e0b8df640 1 --2- 172.21.15.1:0/3704503393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e041e99a0 0x7f3e041490a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.738+0000 7f3e0b8df640 1 --2- 172.21.15.1:0/3704503393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df0004800 0x7f3e04155a20 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.738+0000 7f3e0b8df640 1 -- 172.21.15.1:0/3704503393 >> 172.21.15.1:0/3704503393 conn(0x7f3e040c88f0 msgr2=0x7f3e04152140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.738+0000 7f3e0b8df640 1 -- 172.21.15.1:0/3704503393 shutdown_connections 2024-03-06T07:31:55.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:55.738+0000 7f3e0b8df640 1 -- 172.21.15.1:0/3704503393 wait complete. 2024-03-06T07:31:56.153 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:56.153 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:57.155 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:57.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.254+0000 7f67378fb640 1 Processor -- start 2024-03-06T07:31:57.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.254+0000 7f67378fb640 1 -- start start 2024-03-06T07:31:57.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.255+0000 7f67378fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6730152650 0x7f6730155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.255+0000 7f67378fb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f67300bf9d0 0x7f67300c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.255+0000 7f67378fb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67300c2fc0 con 0x7f67300c6b10 2024-03-06T07:31:57.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.255+0000 7f67378fb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6730155800 con 0x7f67300bf9d0 2024-03-06T07:31:57.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.255+0000 7f67378fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6730158da0 con 0x7f6730152650 2024-03-06T07:31:57.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.255+0000 7f6735e98640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f67300bf9d0 0x7f67300c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.256+0000 7f6734e96640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f67300c6b10 0x7f6730158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54418/0 (socket says 172.21.15.1:54418) 2024-03-06T07:31:57.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.256+0000 7f6734e96640 1 -- 172.21.15.1:0/2469319173 learned_addr learned my addr 172.21.15.1:0/2469319173 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:31:57.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.256+0000 7f6735e98640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f67300bf9d0 0x7f67300c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51950/0 (socket says 172.21.15.1:51950) 2024-03-06T07:31:57.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.256+0000 7f6735697640 1 --2- 172.21.15.1:0/2469319173 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6730152650 0x7f6730155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.256+0000 7f6735e98640 1 -- 172.21.15.1:0/2469319173 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6730152650 msgr2=0x7f6730155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:57.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.256+0000 7f6735e98640 1 --2- 172.21.15.1:0/2469319173 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6730152650 0x7f6730155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.256+0000 7f6735e98640 1 -- 172.21.15.1:0/2469319173 >> v1:172.21.15.1:6789/0 conn(0x7f67300c6b10 legacy=0x7f6730158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.256+0000 7f6735e98640 1 -- 172.21.15.1:0/2469319173 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67300c6690 con 0x7f67300bf9d0 2024-03-06T07:31:57.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.256+0000 7f6727fff640 1 -- 172.21.15.1:0/2469319173 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290074004 0 0) 0x7f67300c2fc0 con 0x7f67300c6b10 2024-03-06T07:31:57.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.256+0000 7f6735697640 1 --2- 172.21.15.1:0/2469319173 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6730152650 0x7f6730155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:57.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.257+0000 7f6735e98640 1 --2- 172.21.15.1:0/2469319173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67300bf9d0 0x7f67300c1460 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f6728009d90 tx=0x7f672802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba34c468d3d8479 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.257+0000 7f6727fff640 1 -- 172.21.15.1:0/2469319173 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728039070 con 0x7f67300bf9d0 2024-03-06T07:31:57.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.257+0000 7f6727fff640 1 -- 172.21.15.1:0/2469319173 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672802cbc0 con 0x7f67300bf9d0 2024-03-06T07:31:57.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.257+0000 7f6727fff640 1 -- 172.21.15.1:0/2469319173 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728034440 con 0x7f67300bf9d0 2024-03-06T07:31:57.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.257+0000 7f67378fb640 1 -- 172.21.15.1:0/2469319173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67300bf9d0 msgr2=0x7f67300c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.257+0000 7f67378fb640 1 --2- 172.21.15.1:0/2469319173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67300bf9d0 0x7f67300c1460 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f6728009d90 tx=0x7f672802be10 comp rx=0 tx=0).stop 2024-03-06T07:31:57.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.257+0000 7f67378fb640 1 -- 172.21.15.1:0/2469319173 shutdown_connections 2024-03-06T07:31:57.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.257+0000 7f67378fb640 1 --2- 172.21.15.1:0/2469319173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67300bf9d0 0x7f67300c1460 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.257+0000 7f67378fb640 1 --2- 172.21.15.1:0/2469319173 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6730152650 0x7f6730155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.257+0000 7f67378fb640 1 -- 172.21.15.1:0/2469319173 >> 172.21.15.1:0/2469319173 conn(0x7f6730066ee0 msgr2=0x7f67300672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.258+0000 7f67378fb640 1 -- 172.21.15.1:0/2469319173 shutdown_connections 2024-03-06T07:31:57.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.258+0000 7f67378fb640 1 -- 172.21.15.1:0/2469319173 wait complete. 2024-03-06T07:31:57.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.259+0000 7f67378fb640 1 Processor -- start 2024-03-06T07:31:57.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.259+0000 7f67378fb640 1 -- start start 2024-03-06T07:31:57.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.259+0000 7f67378fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67300bf9d0 0x7f67301505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.259+0000 7f67378fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6730152650 0x7f6730157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.259+0000 7f67378fb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67301581a0 0x7f673014d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.259+0000 7f67378fb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67301551b0 con 0x7f67300bf9d0 2024-03-06T07:31:57.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.259+0000 7f67378fb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67300c6fa0 con 0x7f67301581a0 2024-03-06T07:31:57.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.259+0000 7f67378fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6730155490 con 0x7f6730152650 2024-03-06T07:31:57.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.259+0000 7f6735697640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67300bf9d0 0x7f67301505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.259+0000 7f6735697640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67300bf9d0 0x7f67301505f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51956/0 (socket says 172.21.15.1:51956) 2024-03-06T07:31:57.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.259+0000 7f6735697640 1 -- 172.21.15.1:0/2917875584 learned_addr learned my addr 172.21.15.1:0/2917875584 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:57.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.260+0000 7f6734e96640 1 --2- 172.21.15.1:0/2917875584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6730152650 0x7f6730157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.260+0000 7f6735e98640 1 --2- 172.21.15.1:0/2917875584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67301581a0 0x7f673014d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.260+0000 7f6735697640 1 -- 172.21.15.1:0/2917875584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6730152650 msgr2=0x7f6730157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.260+0000 7f6735697640 1 --2- 172.21.15.1:0/2917875584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6730152650 0x7f6730157c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.260+0000 7f6735697640 1 -- 172.21.15.1:0/2917875584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67301581a0 msgr2=0x7f673014d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.260+0000 7f6735697640 1 --2- 172.21.15.1:0/2917875584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67301581a0 0x7f673014d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.260+0000 7f6735697640 1 -- 172.21.15.1:0/2917875584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67301e1390 con 0x7f67300bf9d0 2024-03-06T07:31:57.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.260+0000 7f6734e96640 1 --2- 172.21.15.1:0/2917875584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6730152650 0x7f6730157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:57.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.260+0000 7f6735e98640 1 --2- 172.21.15.1:0/2917875584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67301581a0 0x7f673014d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:57.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.260+0000 7f6735697640 1 --2- 172.21.15.1:0/2917875584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67300bf9d0 0x7f67301505f0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f672000c7f0 tx=0x7f672000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.260+0000 7f6725ffb640 1 -- 172.21.15.1:0/2917875584 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6720015070 con 0x7f67300bf9d0 2024-03-06T07:31:57.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.260+0000 7f6725ffb640 1 -- 172.21.15.1:0/2917875584 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6720007a30 con 0x7f67300bf9d0 2024-03-06T07:31:57.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.260+0000 7f67378fb640 1 -- 172.21.15.1:0/2917875584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67301e1670 con 0x7f67300bf9d0 2024-03-06T07:31:57.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.260+0000 7f6725ffb640 1 -- 172.21.15.1:0/2917875584 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6720007e00 con 0x7f67300bf9d0 2024-03-06T07:31:57.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.260+0000 7f67378fb640 1 -- 172.21.15.1:0/2917875584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67301e1c30 con 0x7f67300bf9d0 2024-03-06T07:31:57.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.261+0000 7f67378fb640 1 -- 172.21.15.1:0/2917875584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6730067fd0 con 0x7f67300bf9d0 2024-03-06T07:31:57.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.262+0000 7f6725ffb640 1 -- 172.21.15.1:0/2917875584 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6720018070 con 0x7f67300bf9d0 2024-03-06T07:31:57.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.262+0000 7f6725ffb640 1 --2- 172.21.15.1:0/2917875584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f671406b4d0 0x7f671406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.262+0000 7f6725ffb640 1 -- 172.21.15.1:0/2917875584 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f672008a3b0 con 0x7f67300bf9d0 2024-03-06T07:31:57.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.262+0000 7f6734e96640 1 --2- 172.21.15.1:0/2917875584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f671406b4d0 0x7f671406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.263+0000 7f6734e96640 1 --2- 172.21.15.1:0/2917875584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f671406b4d0 0x7f671406d990 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f67300cb3d0 tx=0x7f672c007ca0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.266+0000 7f6725ffb640 1 -- 172.21.15.1:0/2917875584 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6720059030 con 0x7f67300bf9d0 2024-03-06T07:31:57.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.507+0000 7f67378fb640 1 -- 172.21.15.1:0/2917875584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67300b06f0 con 0x7f67300bf9d0 2024-03-06T07:31:57.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.507+0000 7f6725ffb640 1 -- 172.21.15.1:0/2917875584 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f6720058930 con 0x7f67300bf9d0 2024-03-06T07:31:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:57.510 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:57.510 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:57.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.510+0000 7f67378fb640 1 -- 172.21.15.1:0/2917875584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f671406b4d0 msgr2=0x7f671406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.510+0000 7f67378fb640 1 --2- 172.21.15.1:0/2917875584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f671406b4d0 0x7f671406d990 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f67300cb3d0 tx=0x7f672c007ca0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.510+0000 7f67378fb640 1 -- 172.21.15.1:0/2917875584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67300bf9d0 msgr2=0x7f67301505f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.510+0000 7f67378fb640 1 --2- 172.21.15.1:0/2917875584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67300bf9d0 0x7f67301505f0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f672000c7f0 tx=0x7f672000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.510+0000 7f67378fb640 1 -- 172.21.15.1:0/2917875584 shutdown_connections 2024-03-06T07:31:57.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.510+0000 7f67378fb640 1 --2- 172.21.15.1:0/2917875584 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f671406b4d0 0x7f671406d990 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.510+0000 7f67378fb640 1 --2- 172.21.15.1:0/2917875584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67301581a0 0x7f673014d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.510+0000 7f67378fb640 1 --2- 172.21.15.1:0/2917875584 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6730152650 0x7f6730157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.510+0000 7f67378fb640 1 --2- 172.21.15.1:0/2917875584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67300bf9d0 0x7f67301505f0 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.510+0000 7f67378fb640 1 -- 172.21.15.1:0/2917875584 >> 172.21.15.1:0/2917875584 conn(0x7f6730066ee0 msgr2=0x7f67300c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.510+0000 7f67378fb640 1 -- 172.21.15.1:0/2917875584 shutdown_connections 2024-03-06T07:31:57.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:57.510+0000 7f67378fb640 1 -- 172.21.15.1:0/2917875584 wait complete. 2024-03-06T07:31:57.926 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:57.926 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:31:58.927 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:59.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.027+0000 7fd8a56ed640 1 Processor -- start 2024-03-06T07:31:59.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.027+0000 7fd8a56ed640 1 -- start start 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.027+0000 7fd8a56ed640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd8a0152650 0x7fd8a0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.027+0000 7fd8a56ed640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8a00bf9d0 0x7fd8a00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.027+0000 7fd8a56ed640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8a00c2fc0 con 0x7fd8a00c6b10 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.027+0000 7fd8a56ed640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8a0155800 con 0x7fd8a00bf9d0 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.027+0000 7fd8a56ed640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8a0158da0 con 0x7fd8a0152650 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.028+0000 7fd89f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8a00bf9d0 0x7fd8a00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.028+0000 7fd89f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8a00bf9d0 0x7fd8a00c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51970/0 (socket says 172.21.15.1:51970) 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.028+0000 7fd89f7fe640 1 -- 172.21.15.1:0/867562698 learned_addr learned my addr 172.21.15.1:0/867562698 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.028+0000 7fd89effd640 1 --2- 172.21.15.1:0/867562698 >> v2:172.21.15.98:3300/0 conn(0x7fd8a0152650 0x7fd8a0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.028+0000 7fd89f7fe640 1 -- 172.21.15.1:0/867562698 >> v2:172.21.15.98:3300/0 conn(0x7fd8a0152650 msgr2=0x7fd8a0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.028+0000 7fd89f7fe640 1 --2- 172.21.15.1:0/867562698 >> v2:172.21.15.98:3300/0 conn(0x7fd8a0152650 0x7fd8a0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.028+0000 7fd89f7fe640 1 -- 172.21.15.1:0/867562698 >> v1:172.21.15.98:6789/0 conn(0x7fd8a00c6b10 legacy=0x7fd8a0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.028+0000 7fd89f7fe640 1 -- 172.21.15.1:0/867562698 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a00c6690 con 0x7fd8a00bf9d0 2024-03-06T07:31:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.029+0000 7fd89f7fe640 1 --2- 172.21.15.1:0/867562698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a00bf9d0 0x7fd8a00c1460 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00cac90 tx=0x7fd89802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d874e44f62a21c00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.029+0000 7fd89dffb640 1 -- 172.21.15.1:0/867562698 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89802ca80 con 0x7fd8a00bf9d0 2024-03-06T07:31:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.029+0000 7fd89dffb640 1 -- 172.21.15.1:0/867562698 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89802cc20 con 0x7fd8a00bf9d0 2024-03-06T07:31:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.029+0000 7fd89dffb640 1 -- 172.21.15.1:0/867562698 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8980343f0 con 0x7fd8a00bf9d0 2024-03-06T07:31:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.030+0000 7fd8a56ed640 1 -- 172.21.15.1:0/867562698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a00bf9d0 msgr2=0x7fd8a00c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.030+0000 7fd8a56ed640 1 --2- 172.21.15.1:0/867562698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a00bf9d0 0x7fd8a00c1460 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00cac90 tx=0x7fd89802bee0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.030+0000 7fd8a56ed640 1 -- 172.21.15.1:0/867562698 shutdown_connections 2024-03-06T07:31:59.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.030+0000 7fd8a56ed640 1 --2- 172.21.15.1:0/867562698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a00bf9d0 0x7fd8a00c1460 unknown :-1 s=CLOSED pgs=2556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.030+0000 7fd8a56ed640 1 --2- 172.21.15.1:0/867562698 >> v2:172.21.15.98:3300/0 conn(0x7fd8a0152650 0x7fd8a0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.030+0000 7fd8a56ed640 1 -- 172.21.15.1:0/867562698 >> 172.21.15.1:0/867562698 conn(0x7fd8a0066ee0 msgr2=0x7fd8a00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.030+0000 7fd8a56ed640 1 -- 172.21.15.1:0/867562698 shutdown_connections 2024-03-06T07:31:59.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.030+0000 7fd8a56ed640 1 -- 172.21.15.1:0/867562698 wait complete. 2024-03-06T07:31:59.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.031+0000 7fd8a56ed640 1 Processor -- start 2024-03-06T07:31:59.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.031+0000 7fd8a56ed640 1 -- start start 2024-03-06T07:31:59.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd8a56ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a00bf9d0 0x7fd8a00c5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd8a56ed640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a0152650 0x7fd8a014b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd8a56ed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8a00ca380 0x7fd8a014b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd8a56ed640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a01551b0 con 0x7fd8a0152650 2024-03-06T07:31:59.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd8a56ed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a00c6fa0 con 0x7fd8a00bf9d0 2024-03-06T07:31:59.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd8a56ed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a0155490 con 0x7fd8a00ca380 2024-03-06T07:31:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd89e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a0152650 0x7fd8a014b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd89e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a0152650 0x7fd8a014b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51986/0 (socket says 172.21.15.1:51986) 2024-03-06T07:31:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd89e7fc640 1 -- 172.21.15.1:0/2938203883 learned_addr learned my addr 172.21.15.1:0/2938203883 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:31:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd89effd640 1 --2- 172.21.15.1:0/2938203883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a00bf9d0 0x7fd8a00c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd89f7fe640 1 --2- 172.21.15.1:0/2938203883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8a00ca380 0x7fd8a014b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd89e7fc640 1 -- 172.21.15.1:0/2938203883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8a00ca380 msgr2=0x7fd8a014b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd89e7fc640 1 --2- 172.21.15.1:0/2938203883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8a00ca380 0x7fd8a014b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd89e7fc640 1 -- 172.21.15.1:0/2938203883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a00bf9d0 msgr2=0x7fd8a00c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd89e7fc640 1 --2- 172.21.15.1:0/2938203883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a00bf9d0 0x7fd8a00c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd89e7fc640 1 -- 172.21.15.1:0/2938203883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a014bc60 con 0x7fd8a0152650 2024-03-06T07:31:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd89effd640 1 --2- 172.21.15.1:0/2938203883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a00bf9d0 0x7fd8a00c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd89f7fe640 1 --2- 172.21.15.1:0/2938203883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8a00ca380 0x7fd8a014b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:59.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd89e7fc640 1 --2- 172.21.15.1:0/2938203883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a0152650 0x7fd8a014b1e0 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7fd89400c400 tx=0x7fd89400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.032+0000 7fd8877fe640 1 -- 172.21.15.1:0/2938203883 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8940052d0 con 0x7fd8a0152650 2024-03-06T07:31:59.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.033+0000 7fd8877fe640 1 -- 172.21.15.1:0/2938203883 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd894005490 con 0x7fd8a0152650 2024-03-06T07:31:59.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.033+0000 7fd8a56ed640 1 -- 172.21.15.1:0/2938203883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a0151990 con 0x7fd8a0152650 2024-03-06T07:31:59.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.033+0000 7fd8877fe640 1 -- 172.21.15.1:0/2938203883 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd894007260 con 0x7fd8a0152650 2024-03-06T07:31:59.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.033+0000 7fd8a56ed640 1 -- 172.21.15.1:0/2938203883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a014fcb0 con 0x7fd8a0152650 2024-03-06T07:31:59.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.033+0000 7fd8a56ed640 1 -- 172.21.15.1:0/2938203883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a0151eb0 con 0x7fd8a0152650 2024-03-06T07:31:59.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.034+0000 7fd8877fe640 1 -- 172.21.15.1:0/2938203883 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd894007400 con 0x7fd8a0152650 2024-03-06T07:31:59.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.034+0000 7fd8877fe640 1 --2- 172.21.15.1:0/2938203883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd88c06b500 0x7fd88c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.034+0000 7fd8877fe640 1 -- 172.21.15.1:0/2938203883 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd894089870 con 0x7fd8a0152650 2024-03-06T07:31:59.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.035+0000 7fd89effd640 1 --2- 172.21.15.1:0/2938203883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd88c06b500 0x7fd88c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.035+0000 7fd89effd640 1 --2- 172.21.15.1:0/2938203883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd88c06b500 0x7fd88c06d9c0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fd890009b90 tx=0x7fd890008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.043+0000 7fd8877fe640 1 -- 172.21.15.1:0/2938203883 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8a0151eb0 con 0x7fd8a0152650 2024-03-06T07:31:59.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.333+0000 7fd8a56ed640 1 -- 172.21.15.1:0/2938203883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8a014ffc0 con 0x7fd8a0152650 2024-03-06T07:31:59.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.333+0000 7fd8877fe640 1 -- 172.21.15.1:0/2938203883 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fd8940584f0 con 0x7fd8a0152650 2024-03-06T07:31:59.335 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:31:59.335 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:31:59.336 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:31:59.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.335+0000 7fd8a56ed640 1 -- 172.21.15.1:0/2938203883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd88c06b500 msgr2=0x7fd88c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.335+0000 7fd8a56ed640 1 --2- 172.21.15.1:0/2938203883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd88c06b500 0x7fd88c06d9c0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fd890009b90 tx=0x7fd890008040 comp rx=0 tx=0).stop 2024-03-06T07:31:59.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.335+0000 7fd8a56ed640 1 -- 172.21.15.1:0/2938203883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a0152650 msgr2=0x7fd8a014b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.335+0000 7fd8a56ed640 1 --2- 172.21.15.1:0/2938203883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a0152650 0x7fd8a014b1e0 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7fd89400c400 tx=0x7fd89400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.335+0000 7fd8a56ed640 1 -- 172.21.15.1:0/2938203883 shutdown_connections 2024-03-06T07:31:59.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.335+0000 7fd8a56ed640 1 --2- 172.21.15.1:0/2938203883 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd88c06b500 0x7fd88c06d9c0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.335+0000 7fd8a56ed640 1 --2- 172.21.15.1:0/2938203883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8a00ca380 0x7fd8a014b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.335+0000 7fd8a56ed640 1 --2- 172.21.15.1:0/2938203883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a0152650 0x7fd8a014b1e0 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.335+0000 7fd8a56ed640 1 --2- 172.21.15.1:0/2938203883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a00bf9d0 0x7fd8a00c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.335+0000 7fd8a56ed640 1 -- 172.21.15.1:0/2938203883 >> 172.21.15.1:0/2938203883 conn(0x7fd8a0066ee0 msgr2=0x7fd8a00c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.335+0000 7fd8a56ed640 1 -- 172.21.15.1:0/2938203883 shutdown_connections 2024-03-06T07:31:59.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:31:59.335+0000 7fd8a56ed640 1 -- 172.21.15.1:0/2938203883 wait complete. 2024-03-06T07:31:59.755 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:31:59.755 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:00.756 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.856+0000 7f9e942c9640 1 Processor -- start 2024-03-06T07:32:00.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.856+0000 7f9e942c9640 1 -- start start 2024-03-06T07:32:00.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.858+0000 7f9e942c9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9e8c152650 0x7f9e8c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.858+0000 7f9e942c9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9e8c0c6b10 0x7f9e8c158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.858+0000 7f9e942c9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e8c0c2fc0 con 0x7f9e8c0bf9d0 2024-03-06T07:32:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.858+0000 7f9e942c9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e8c155800 con 0x7f9e8c152650 2024-03-06T07:32:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.858+0000 7f9e942c9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e8c0c6690 con 0x7f9e8c0c6b10 2024-03-06T07:32:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.858+0000 7f9e92065640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9e8c152650 0x7f9e8c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.858+0000 7f9e92866640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9e8c0bf9d0 0x7f9e8c0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54434/0 (socket says 172.21.15.1:54434) 2024-03-06T07:32:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.858+0000 7f9e92866640 1 -- 172.21.15.1:0/657192257 learned_addr learned my addr 172.21.15.1:0/657192257 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.858+0000 7f9e91864640 1 --2- 172.21.15.1:0/657192257 >> v2:172.21.15.98:3300/0 conn(0x7f9e8c0c6b10 0x7f9e8c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.858+0000 7f9e92065640 1 -- 172.21.15.1:0/657192257 >> v2:172.21.15.98:3300/0 conn(0x7f9e8c0c6b10 msgr2=0x7f9e8c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.858+0000 7f9e92065640 1 --2- 172.21.15.1:0/657192257 >> v2:172.21.15.98:3300/0 conn(0x7f9e8c0c6b10 0x7f9e8c158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.858+0000 7f9e92065640 1 -- 172.21.15.1:0/657192257 >> v1:172.21.15.1:6789/0 conn(0x7f9e8c0bf9d0 legacy=0x7f9e8c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.858+0000 7f9e92065640 1 -- 172.21.15.1:0/657192257 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8c0c7280 con 0x7f9e8c152650 2024-03-06T07:32:00.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.858+0000 7f9e91063640 1 -- 172.21.15.1:0/657192257 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475946474 0 0) 0x7f9e8c0c2fc0 con 0x7f9e8c0bf9d0 2024-03-06T07:32:00.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.859+0000 7f9e92065640 1 --2- 172.21.15.1:0/657192257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e8c152650 0x7f9e8c155c90 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c0cb8e0 tx=0x7f9e8402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5f625598e687b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.859+0000 7f9e91063640 1 -- 172.21.15.1:0/657192257 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840315a0 con 0x7f9e8c152650 2024-03-06T07:32:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.859+0000 7f9e91063640 1 -- 172.21.15.1:0/657192257 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e84037070 con 0x7f9e8c152650 2024-03-06T07:32:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.859+0000 7f9e91063640 1 -- 172.21.15.1:0/657192257 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e84031960 con 0x7f9e8c152650 2024-03-06T07:32:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.860+0000 7f9e942c9640 1 -- 172.21.15.1:0/657192257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e8c152650 msgr2=0x7f9e8c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.860+0000 7f9e942c9640 1 --2- 172.21.15.1:0/657192257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e8c152650 0x7f9e8c155c90 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c0cb8e0 tx=0x7f9e8402b840 comp rx=0 tx=0).stop 2024-03-06T07:32:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.860+0000 7f9e942c9640 1 -- 172.21.15.1:0/657192257 shutdown_connections 2024-03-06T07:32:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.860+0000 7f9e942c9640 1 --2- 172.21.15.1:0/657192257 >> v2:172.21.15.98:3300/0 conn(0x7f9e8c0c6b10 0x7f9e8c158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.860+0000 7f9e942c9640 1 --2- 172.21.15.1:0/657192257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e8c152650 0x7f9e8c155c90 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.860+0000 7f9e942c9640 1 -- 172.21.15.1:0/657192257 >> 172.21.15.1:0/657192257 conn(0x7f9e8c066ee0 msgr2=0x7f9e8c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.860+0000 7f9e942c9640 1 -- 172.21.15.1:0/657192257 shutdown_connections 2024-03-06T07:32:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.860+0000 7f9e942c9640 1 -- 172.21.15.1:0/657192257 wait complete. 2024-03-06T07:32:00.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.861+0000 7f9e942c9640 1 Processor -- start 2024-03-06T07:32:00.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.861+0000 7f9e942c9640 1 -- start start 2024-03-06T07:32:00.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e942c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e8c0bf9d0 0x7f9e8c148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e942c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e8c152650 0x7f9e8c149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e942c9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e8c14a010 0x7f9e8c1edf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e942c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c155490 con 0x7f9e8c152650 2024-03-06T07:32:00.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e942c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c1551b0 con 0x7f9e8c0bf9d0 2024-03-06T07:32:00.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e942c9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c152a30 con 0x7f9e8c14a010 2024-03-06T07:32:00.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e91864640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e8c152650 0x7f9e8c149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e91864640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e8c152650 0x7f9e8c149ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52002/0 (socket says 172.21.15.1:52002) 2024-03-06T07:32:00.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e91864640 1 -- 172.21.15.1:0/2730948295 learned_addr learned my addr 172.21.15.1:0/2730948295 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:00.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e92065640 1 --2- 172.21.15.1:0/2730948295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e8c0bf9d0 0x7f9e8c148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e92866640 1 --2- 172.21.15.1:0/2730948295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e8c14a010 0x7f9e8c1edf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e91864640 1 -- 172.21.15.1:0/2730948295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e8c14a010 msgr2=0x7f9e8c1edf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e91864640 1 --2- 172.21.15.1:0/2730948295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e8c14a010 0x7f9e8c1edf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e91864640 1 -- 172.21.15.1:0/2730948295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e8c0bf9d0 msgr2=0x7f9e8c148e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e91864640 1 --2- 172.21.15.1:0/2730948295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e8c0bf9d0 0x7f9e8c148e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e91864640 1 -- 172.21.15.1:0/2730948295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8c14a4a0 con 0x7f9e8c152650 2024-03-06T07:32:00.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e92065640 1 --2- 172.21.15.1:0/2730948295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e8c0bf9d0 0x7f9e8c148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:00.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e92866640 1 --2- 172.21.15.1:0/2730948295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e8c14a010 0x7f9e8c1edf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:00.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.862+0000 7f9e91864640 1 --2- 172.21.15.1:0/2730948295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e8c152650 0x7f9e8c149ad0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c00e870 tx=0x7f9e7c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.863+0000 7f9e82ffd640 1 -- 172.21.15.1:0/2730948295 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c017070 con 0x7f9e8c152650 2024-03-06T07:32:00.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.863+0000 7f9e82ffd640 1 -- 172.21.15.1:0/2730948295 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7c00c940 con 0x7f9e8c152650 2024-03-06T07:32:00.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.863+0000 7f9e942c9640 1 -- 172.21.15.1:0/2730948295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8c154090 con 0x7f9e8c152650 2024-03-06T07:32:00.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.863+0000 7f9e82ffd640 1 -- 172.21.15.1:0/2730948295 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c00cd30 con 0x7f9e8c152650 2024-03-06T07:32:00.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.863+0000 7f9e942c9640 1 -- 172.21.15.1:0/2730948295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8c0c9b90 con 0x7f9e8c152650 2024-03-06T07:32:00.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.864+0000 7f9e942c9640 1 -- 172.21.15.1:0/2730948295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e8c1545b0 con 0x7f9e8c152650 2024-03-06T07:32:00.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.864+0000 7f9e82ffd640 1 -- 172.21.15.1:0/2730948295 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e7c01a070 con 0x7f9e8c152650 2024-03-06T07:32:00.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.865+0000 7f9e82ffd640 1 --2- 172.21.15.1:0/2730948295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e7006b500 0x7f9e7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.865+0000 7f9e82ffd640 1 -- 172.21.15.1:0/2730948295 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f9e7c08a620 con 0x7f9e8c152650 2024-03-06T07:32:00.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.865+0000 7f9e92065640 1 --2- 172.21.15.1:0/2730948295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e7006b500 0x7f9e7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.866+0000 7f9e92065640 1 --2- 172.21.15.1:0/2730948295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e7006b500 0x7f9e7006d9c0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f9e84036000 tx=0x7f9e84008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:00.869+0000 7f9e82ffd640 1 -- 172.21.15.1:0/2730948295 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e7c0592a0 con 0x7f9e8c152650 2024-03-06T07:32:01.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:01.091+0000 7f9e942c9640 1 -- 172.21.15.1:0/2730948295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e8c067fd0 con 0x7f9e8c152650 2024-03-06T07:32:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:01.091+0000 7f9e82ffd640 1 -- 172.21.15.1:0/2730948295 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f9e7c058ba0 con 0x7f9e8c152650 2024-03-06T07:32:01.093 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:01.093 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:01.093+0000 7f9e942c9640 1 -- 172.21.15.1:0/2730948295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e7006b500 msgr2=0x7f9e7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:01.093+0000 7f9e942c9640 1 --2- 172.21.15.1:0/2730948295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e7006b500 0x7f9e7006d9c0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f9e84036000 tx=0x7f9e84008030 comp rx=0 tx=0).stop 2024-03-06T07:32:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:01.093+0000 7f9e942c9640 1 -- 172.21.15.1:0/2730948295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e8c152650 msgr2=0x7f9e8c149ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:01.093+0000 7f9e942c9640 1 --2- 172.21.15.1:0/2730948295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e8c152650 0x7f9e8c149ad0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c00e870 tx=0x7f9e7c00ed30 comp rx=0 tx=0).stop 2024-03-06T07:32:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:01.093+0000 7f9e942c9640 1 -- 172.21.15.1:0/2730948295 shutdown_connections 2024-03-06T07:32:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:01.093+0000 7f9e942c9640 1 --2- 172.21.15.1:0/2730948295 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9e7006b500 0x7f9e7006d9c0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:01.093+0000 7f9e942c9640 1 --2- 172.21.15.1:0/2730948295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e8c14a010 0x7f9e8c1edf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:01.093+0000 7f9e942c9640 1 --2- 172.21.15.1:0/2730948295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e8c152650 0x7f9e8c149ad0 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:01.093+0000 7f9e942c9640 1 --2- 172.21.15.1:0/2730948295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e8c0bf9d0 0x7f9e8c148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:01.093+0000 7f9e942c9640 1 -- 172.21.15.1:0/2730948295 >> 172.21.15.1:0/2730948295 conn(0x7f9e8c066ee0 msgr2=0x7f9e8c0cabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:01.093+0000 7f9e942c9640 1 -- 172.21.15.1:0/2730948295 shutdown_connections 2024-03-06T07:32:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:01.093+0000 7f9e942c9640 1 -- 172.21.15.1:0/2730948295 wait complete. 2024-03-06T07:32:01.508 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:01.509 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:02.510 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:02.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.625+0000 7f32a0f81640 1 Processor -- start 2024-03-06T07:32:02.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.625+0000 7f32a0f81640 1 -- start start 2024-03-06T07:32:02.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.626+0000 7f32a0f81640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f329c149a90 0x7f329c0c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.626+0000 7f32a0f81640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f329c1530a0 con 0x7f329c0c16f0 2024-03-06T07:32:02.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.626+0000 7f32a0f81640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f329c153380 con 0x7f329c0c1260 2024-03-06T07:32:02.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.626+0000 7f32a0f81640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f329c153660 con 0x7f329c149a90 2024-03-06T07:32:02.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.626+0000 7f329a59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f329c149a90 0x7f329c0c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.626+0000 7f329a59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f329c149a90 0x7f329c0c4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52014/0 (socket says 172.21.15.1:52014) 2024-03-06T07:32:02.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.626+0000 7f329a59c640 1 -- 172.21.15.1:0/1598277635 learned_addr learned my addr 172.21.15.1:0/1598277635 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:02.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.626+0000 7f329ad9d640 1 --1- 172.21.15.1:0/1598277635 >> v1:172.21.15.1:6789/0 conn(0x7f329c0c16f0 0x7f329c156cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54450/0 (socket says 172.21.15.1:54450) 2024-03-06T07:32:02.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.626+0000 7f329a59c640 1 -- 172.21.15.1:0/1598277635 >> v1:172.21.15.98:6789/0 conn(0x7f329c0c1260 legacy=0x7f329c1565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.626+0000 7f329a59c640 1 -- 172.21.15.1:0/1598277635 >> v1:172.21.15.1:6789/0 conn(0x7f329c0c16f0 legacy=0x7f329c156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.626+0000 7f329a59c640 1 -- 172.21.15.1:0/1598277635 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329c153940 con 0x7f329c149a90 2024-03-06T07:32:02.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.626+0000 7f3299d9b640 1 -- 172.21.15.1:0/1598277635 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873206878 0 0) 0x7f329c1530a0 con 0x7f329c0c16f0 2024-03-06T07:32:02.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.627+0000 7f329a59c640 1 --2- 172.21.15.1:0/1598277635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c149a90 0x7f329c0c4760 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f328c009870 tx=0x7f328c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e73f08f899d380ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.627+0000 7f3299d9b640 1 -- 172.21.15.1:0/1598277635 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c035020 con 0x7f329c149a90 2024-03-06T07:32:02.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.627+0000 7f3299d9b640 1 -- 172.21.15.1:0/1598277635 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328c031870 con 0x7f329c149a90 2024-03-06T07:32:02.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.627+0000 7f3299d9b640 1 -- 172.21.15.1:0/1598277635 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c031c40 con 0x7f329c149a90 2024-03-06T07:32:02.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.628+0000 7f32a0f81640 1 -- 172.21.15.1:0/1598277635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c149a90 msgr2=0x7f329c0c4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.628+0000 7f32a0f81640 1 --2- 172.21.15.1:0/1598277635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c149a90 0x7f329c0c4760 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f328c009870 tx=0x7f328c02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:02.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.628+0000 7f32a0f81640 1 -- 172.21.15.1:0/1598277635 shutdown_connections 2024-03-06T07:32:02.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.628+0000 7f32a0f81640 1 --2- 172.21.15.1:0/1598277635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c149a90 0x7f329c0c4760 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.628+0000 7f32a0f81640 1 -- 172.21.15.1:0/1598277635 >> 172.21.15.1:0/1598277635 conn(0x7f329c0c6860 msgr2=0x7f329c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.628+0000 7f32a0f81640 1 -- 172.21.15.1:0/1598277635 shutdown_connections 2024-03-06T07:32:02.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.628+0000 7f32a0f81640 1 -- 172.21.15.1:0/1598277635 wait complete. 2024-03-06T07:32:02.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.629+0000 7f32a0f81640 1 Processor -- start 2024-03-06T07:32:02.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.629+0000 7f32a0f81640 1 -- start start 2024-03-06T07:32:02.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.629+0000 7f32a0f81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f329c0c1260 0x7f329c1dce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f32a0f81640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0c16f0 0x7f329c072890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f32a0f81640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f329c149a90 0x7f329c072dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f32a0f81640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329c06f4a0 con 0x7f329c0c16f0 2024-03-06T07:32:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f32a0f81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329c0c4ca0 con 0x7f329c0c1260 2024-03-06T07:32:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f32a0f81640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329c06f620 con 0x7f329c149a90 2024-03-06T07:32:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f3293fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0c16f0 0x7f329c072890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f3293fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0c16f0 0x7f329c072890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52022/0 (socket says 172.21.15.1:52022) 2024-03-06T07:32:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f3293fff640 1 -- 172.21.15.1:0/1804058217 learned_addr learned my addr 172.21.15.1:0/1804058217 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f329a59c640 1 --2- 172.21.15.1:0/1804058217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f329c0c1260 0x7f329c1dce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f329ad9d640 1 --2- 172.21.15.1:0/1804058217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f329c149a90 0x7f329c072dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f3293fff640 1 -- 172.21.15.1:0/1804058217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f329c149a90 msgr2=0x7f329c072dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f3293fff640 1 --2- 172.21.15.1:0/1804058217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f329c149a90 0x7f329c072dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f3293fff640 1 -- 172.21.15.1:0/1804058217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f329c0c1260 msgr2=0x7f329c1dce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f3293fff640 1 --2- 172.21.15.1:0/1804058217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f329c0c1260 0x7f329c1dce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f3293fff640 1 -- 172.21.15.1:0/1804058217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329c14cbb0 con 0x7f329c0c16f0 2024-03-06T07:32:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f329ad9d640 1 --2- 172.21.15.1:0/1804058217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f329c149a90 0x7f329c072dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:02.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f3293fff640 1 --2- 172.21.15.1:0/1804058217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0c16f0 0x7f329c072890 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f328400c460 tx=0x7f328400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f3292ffd640 1 -- 172.21.15.1:0/1804058217 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3284016020 con 0x7f329c0c16f0 2024-03-06T07:32:02.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f3292ffd640 1 -- 172.21.15.1:0/1804058217 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3284012870 con 0x7f329c0c16f0 2024-03-06T07:32:02.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f32a0f81640 1 -- 172.21.15.1:0/1804058217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f329c071870 con 0x7f329c0c16f0 2024-03-06T07:32:02.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f3292ffd640 1 -- 172.21.15.1:0/1804058217 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3284012c60 con 0x7f329c0c16f0 2024-03-06T07:32:02.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.630+0000 7f32a0f81640 1 -- 172.21.15.1:0/1804058217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f329c071e30 con 0x7f329c0c16f0 2024-03-06T07:32:02.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.631+0000 7f32a0f81640 1 -- 172.21.15.1:0/1804058217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f329c067fd0 con 0x7f329c0c16f0 2024-03-06T07:32:02.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.632+0000 7f3292ffd640 1 -- 172.21.15.1:0/1804058217 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3284019020 con 0x7f329c0c16f0 2024-03-06T07:32:02.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.632+0000 7f3292ffd640 1 --2- 172.21.15.1:0/1804058217 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f327c06b4d0 0x7f327c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.633+0000 7f3292ffd640 1 -- 172.21.15.1:0/1804058217 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f3284089df0 con 0x7f329c0c16f0 2024-03-06T07:32:02.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.633+0000 7f329a59c640 1 --2- 172.21.15.1:0/1804058217 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f327c06b4d0 0x7f327c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.633+0000 7f329a59c640 1 --2- 172.21.15.1:0/1804058217 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f327c06b4d0 0x7f327c06d990 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f328c036000 tx=0x7f328c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.637+0000 7f3292ffd640 1 -- 172.21.15.1:0/1804058217 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3284020070 con 0x7f329c0c16f0 2024-03-06T07:32:02.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.862+0000 7f32a0f81640 1 -- 172.21.15.1:0/1804058217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f329c06f280 con 0x7f329c0c16f0 2024-03-06T07:32:02.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.862+0000 7f3292ffd640 1 -- 172.21.15.1:0/1804058217 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f3284058a70 con 0x7f329c0c16f0 2024-03-06T07:32:02.865 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:02.865 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:02.865 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:02.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.864+0000 7f32a0f81640 1 -- 172.21.15.1:0/1804058217 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f327c06b4d0 msgr2=0x7f327c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.864+0000 7f32a0f81640 1 --2- 172.21.15.1:0/1804058217 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f327c06b4d0 0x7f327c06d990 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f328c036000 tx=0x7f328c03c040 comp rx=0 tx=0).stop 2024-03-06T07:32:02.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.864+0000 7f32a0f81640 1 -- 172.21.15.1:0/1804058217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0c16f0 msgr2=0x7f329c072890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.864+0000 7f32a0f81640 1 --2- 172.21.15.1:0/1804058217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0c16f0 0x7f329c072890 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f328400c460 tx=0x7f328400c920 comp rx=0 tx=0).stop 2024-03-06T07:32:02.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.865+0000 7f32a0f81640 1 -- 172.21.15.1:0/1804058217 shutdown_connections 2024-03-06T07:32:02.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.865+0000 7f32a0f81640 1 --2- 172.21.15.1:0/1804058217 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f327c06b4d0 0x7f327c06d990 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.865+0000 7f32a0f81640 1 --2- 172.21.15.1:0/1804058217 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f329c149a90 0x7f329c072dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.865+0000 7f32a0f81640 1 --2- 172.21.15.1:0/1804058217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0c16f0 0x7f329c072890 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.865+0000 7f32a0f81640 1 --2- 172.21.15.1:0/1804058217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f329c0c1260 0x7f329c1dce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.865+0000 7f32a0f81640 1 -- 172.21.15.1:0/1804058217 >> 172.21.15.1:0/1804058217 conn(0x7f329c0c6860 msgr2=0x7f329c1511f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.865+0000 7f32a0f81640 1 -- 172.21.15.1:0/1804058217 shutdown_connections 2024-03-06T07:32:02.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:02.865+0000 7f32a0f81640 1 -- 172.21.15.1:0/1804058217 wait complete. 2024-03-06T07:32:03.272 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:03.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:03.280 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:03.280 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:04.281 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:04.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.404+0000 7f81d0fee640 1 Processor -- start 2024-03-06T07:32:04.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.404+0000 7f81d0fee640 1 -- start start 2024-03-06T07:32:04.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.405+0000 7f81d0fee640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f81cc0bf9d0 0x7f81cc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.405+0000 7f81d0fee640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc0c2fc0 con 0x7f81cc0c6b10 2024-03-06T07:32:04.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.405+0000 7f81d0fee640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc158da0 con 0x7f81cc152650 2024-03-06T07:32:04.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.405+0000 7f81d0fee640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81cc155800 con 0x7f81cc0bf9d0 2024-03-06T07:32:04.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.405+0000 7f81c9d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f81cc0c6b10 0x7f81cc0c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44058/0 (socket says 172.21.15.1:44058) 2024-03-06T07:32:04.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.405+0000 7f81c9d9b640 1 -- 172.21.15.1:0/255648713 learned_addr learned my addr 172.21.15.1:0/255648713 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:04.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.405+0000 7f81cad9d640 1 --2- 172.21.15.1:0/255648713 >> v2:172.21.15.98:3300/0 conn(0x7f81cc0bf9d0 0x7f81cc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.405+0000 7f81c959a640 1 -- 172.21.15.1:0/255648713 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005497793 0 0) 0x7f81cc0c2fc0 con 0x7f81cc0c6b10 2024-03-06T07:32:04.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.405+0000 7f81c959a640 1 -- 172.21.15.1:0/255648713 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b4003520 con 0x7f81cc0c6b10 2024-03-06T07:32:04.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81c959a640 1 -- 172.21.15.1:0/255648713 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 851183976 0 0) 0x7f81b4003520 con 0x7f81cc0c6b10 2024-03-06T07:32:04.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81c959a640 1 -- 172.21.15.1:0/255648713 >> v2:172.21.15.98:3300/0 conn(0x7f81cc0bf9d0 msgr2=0x7f81cc155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:04.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81c959a640 1 --2- 172.21.15.1:0/255648713 >> v2:172.21.15.98:3300/0 conn(0x7f81cc0bf9d0 0x7f81cc155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81c959a640 1 -- 172.21.15.1:0/255648713 >> v1:172.21.15.98:6789/0 conn(0x7f81cc152650 legacy=0x7f81cc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81c959a640 1 -- 172.21.15.1:0/255648713 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc0c6690 con 0x7f81cc0c6b10 2024-03-06T07:32:04.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81c959a640 1 -- 172.21.15.1:0/255648713 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f81bc001670 con 0x7f81cc0c6b10 2024-03-06T07:32:04.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81c959a640 1 -- 172.21.15.1:0/255648713 >> v1:172.21.15.1:6789/0 conn(0x7f81cc0c6b10 legacy=0x7f81cc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81c959a640 1 --2- 172.21.15.1:0/255648713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81b40039b0 0x7f81b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81c959a640 1 --2- 172.21.15.1:0/255648713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81b4004800 0x7f81b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81c959a640 1 --2- 172.21.15.1:0/255648713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81b40051c0 0x7f81b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81c959a640 1 -- 172.21.15.1:0/255648713 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc155800 con 0x7f81b40051c0 2024-03-06T07:32:04.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81c959a640 1 -- 172.21.15.1:0/255648713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b400c160 con 0x7f81b40039b0 2024-03-06T07:32:04.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81c959a640 1 -- 172.21.15.1:0/255648713 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b400c360 con 0x7f81b4004800 2024-03-06T07:32:04.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81cad9d640 1 --2- 172.21.15.1:0/255648713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81b40051c0 0x7f81b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81ca59c640 1 --2- 172.21.15.1:0/255648713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81b40039b0 0x7f81b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81c9d9b640 1 --2- 172.21.15.1:0/255648713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81b4004800 0x7f81b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81cad9d640 1 -- 172.21.15.1:0/255648713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81b4004800 msgr2=0x7f81b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:04.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81cad9d640 1 --2- 172.21.15.1:0/255648713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81b4004800 0x7f81b4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81cad9d640 1 -- 172.21.15.1:0/255648713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81b40039b0 msgr2=0x7f81b4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:04.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81cad9d640 1 --2- 172.21.15.1:0/255648713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81b40039b0 0x7f81b4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.406+0000 7f81cad9d640 1 -- 172.21.15.1:0/255648713 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b4003520 con 0x7f81b40051c0 2024-03-06T07:32:04.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.407+0000 7f81cad9d640 1 -- 172.21.15.1:0/255648713 reap_dead start 2024-03-06T07:32:04.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.407+0000 7f81cad9d640 1 --2- 172.21.15.1:0/255648713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81b40051c0 0x7f81b400b9c0 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009fc0 tx=0x7f81b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e6abc34292b0e98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.407+0000 7f81c959a640 1 -- 172.21.15.1:0/255648713 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c4009c10 con 0x7f81b40051c0 2024-03-06T07:32:04.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.407+0000 7f81c959a640 1 -- 172.21.15.1:0/255648713 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c402cca0 con 0x7f81b40051c0 2024-03-06T07:32:04.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.407+0000 7f81c959a640 1 -- 172.21.15.1:0/255648713 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c4003bd0 con 0x7f81b40051c0 2024-03-06T07:32:04.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.407+0000 7f81d0fee640 1 -- 172.21.15.1:0/255648713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81b40051c0 msgr2=0x7f81b400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.407+0000 7f81d0fee640 1 --2- 172.21.15.1:0/255648713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81b40051c0 0x7f81b400b9c0 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009fc0 tx=0x7f81b4005830 comp rx=0 tx=0).stop 2024-03-06T07:32:04.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.408+0000 7f81d0fee640 1 -- 172.21.15.1:0/255648713 shutdown_connections 2024-03-06T07:32:04.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.408+0000 7f81d0fee640 1 --2- 172.21.15.1:0/255648713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81b40051c0 0x7f81b400b9c0 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.408+0000 7f81d0fee640 1 -- 172.21.15.1:0/255648713 >> 172.21.15.1:0/255648713 conn(0x7f81cc066ee0 msgr2=0x7f81cc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.408+0000 7f81d0fee640 1 -- 172.21.15.1:0/255648713 shutdown_connections 2024-03-06T07:32:04.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.408+0000 7f81d0fee640 1 -- 172.21.15.1:0/255648713 wait complete. 2024-03-06T07:32:04.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.409+0000 7f81d0fee640 1 Processor -- start 2024-03-06T07:32:04.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.409+0000 7f81d0fee640 1 -- start start 2024-03-06T07:32:04.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81d0fee640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81b40051c0 0x7f81cc1eaab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81d0fee640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81cc158330 0x7f81cc0c3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81d0fee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81cc14b4e0 0x7f81cc156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81d0fee640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc1551b0 con 0x7f81cc158330 2024-03-06T07:32:04.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81d0fee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc0c6fa0 con 0x7f81cc14b4e0 2024-03-06T07:32:04.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81d0fee640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc155490 con 0x7f81b40051c0 2024-03-06T07:32:04.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81c9d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81cc158330 0x7f81cc0c3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81c9d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81cc158330 0x7f81cc0c3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32934/0 (socket says 172.21.15.1:32934) 2024-03-06T07:32:04.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81c9d9b640 1 -- 172.21.15.1:0/2674291753 learned_addr learned my addr 172.21.15.1:0/2674291753 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:04.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81cad9d640 1 --2- 172.21.15.1:0/2674291753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81cc14b4e0 0x7f81cc156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81ca59c640 1 --2- 172.21.15.1:0/2674291753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81b40051c0 0x7f81cc1eaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81c9d9b640 1 -- 172.21.15.1:0/2674291753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81b40051c0 msgr2=0x7f81cc1eaab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81c9d9b640 1 --2- 172.21.15.1:0/2674291753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81b40051c0 0x7f81cc1eaab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81c9d9b640 1 -- 172.21.15.1:0/2674291753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81cc14b4e0 msgr2=0x7f81cc156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81c9d9b640 1 --2- 172.21.15.1:0/2674291753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81cc14b4e0 0x7f81cc156f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81c9d9b640 1 -- 172.21.15.1:0/2674291753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc1ee9e0 con 0x7f81cc158330 2024-03-06T07:32:04.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81c9d9b640 1 --2- 172.21.15.1:0/2674291753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81cc158330 0x7f81cc0c3d30 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f81bc001050 tx=0x7f81bc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81baffd640 1 -- 172.21.15.1:0/2674291753 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc00ee70 con 0x7f81cc158330 2024-03-06T07:32:04.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81baffd640 1 -- 172.21.15.1:0/2674291753 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81bc018070 con 0x7f81cc158330 2024-03-06T07:32:04.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81baffd640 1 -- 172.21.15.1:0/2674291753 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc00cad0 con 0x7f81cc158330 2024-03-06T07:32:04.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81d0fee640 1 -- 172.21.15.1:0/2674291753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81cc0cb1c0 con 0x7f81cc158330 2024-03-06T07:32:04.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.410+0000 7f81d0fee640 1 -- 172.21.15.1:0/2674291753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81cc157cc0 con 0x7f81cc158330 2024-03-06T07:32:04.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.411+0000 7f81d0fee640 1 -- 172.21.15.1:0/2674291753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81cc067fd0 con 0x7f81cc158330 2024-03-06T07:32:04.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.412+0000 7f81baffd640 1 -- 172.21.15.1:0/2674291753 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81bc00cc70 con 0x7f81cc158330 2024-03-06T07:32:04.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.412+0000 7f81baffd640 1 --2- 172.21.15.1:0/2674291753 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81ac06b4d0 0x7f81ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.412+0000 7f81baffd640 1 -- 172.21.15.1:0/2674291753 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f81bc08a130 con 0x7f81cc158330 2024-03-06T07:32:04.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.413+0000 7f81ca59c640 1 --2- 172.21.15.1:0/2674291753 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81ac06b4d0 0x7f81ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.413+0000 7f81ca59c640 1 --2- 172.21.15.1:0/2674291753 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81ac06b4d0 0x7f81ac06d990 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f81c0000c00 tx=0x7f81c0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.417+0000 7f81baffd640 1 -- 172.21.15.1:0/2674291753 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81bc058760 con 0x7f81cc158330 2024-03-06T07:32:04.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.643+0000 7f81d0fee640 1 -- 172.21.15.1:0/2674291753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81cc0c68f0 con 0x7f81cc158330 2024-03-06T07:32:04.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.643+0000 7f81baffd640 1 -- 172.21.15.1:0/2674291753 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f81bc05c800 con 0x7f81cc158330 2024-03-06T07:32:04.645 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:04.645 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:04.646 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.646+0000 7f81d0fee640 1 -- 172.21.15.1:0/2674291753 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81ac06b4d0 msgr2=0x7f81ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.646+0000 7f81d0fee640 1 --2- 172.21.15.1:0/2674291753 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81ac06b4d0 0x7f81ac06d990 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f81c0000c00 tx=0x7f81c0009040 comp rx=0 tx=0).stop 2024-03-06T07:32:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.646+0000 7f81d0fee640 1 -- 172.21.15.1:0/2674291753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81cc158330 msgr2=0x7f81cc0c3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.646+0000 7f81d0fee640 1 --2- 172.21.15.1:0/2674291753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81cc158330 0x7f81cc0c3d30 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f81bc001050 tx=0x7f81bc00ec80 comp rx=0 tx=0).stop 2024-03-06T07:32:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.646+0000 7f81d0fee640 1 -- 172.21.15.1:0/2674291753 shutdown_connections 2024-03-06T07:32:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.646+0000 7f81d0fee640 1 --2- 172.21.15.1:0/2674291753 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f81ac06b4d0 0x7f81ac06d990 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.646+0000 7f81d0fee640 1 --2- 172.21.15.1:0/2674291753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81cc14b4e0 0x7f81cc156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.646+0000 7f81d0fee640 1 --2- 172.21.15.1:0/2674291753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81cc158330 0x7f81cc0c3d30 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.646+0000 7f81d0fee640 1 --2- 172.21.15.1:0/2674291753 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81b40051c0 0x7f81cc1eaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.646+0000 7f81d0fee640 1 -- 172.21.15.1:0/2674291753 >> 172.21.15.1:0/2674291753 conn(0x7f81cc066ee0 msgr2=0x7f81cc0672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.646+0000 7f81d0fee640 1 -- 172.21.15.1:0/2674291753 shutdown_connections 2024-03-06T07:32:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:04.646+0000 7f81d0fee640 1 -- 172.21.15.1:0/2674291753 wait complete. 2024-03-06T07:32:05.062 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:05.062 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:06.063 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:06.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.163+0000 7fa3debc9640 1 Processor -- start 2024-03-06T07:32:06.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.163+0000 7fa3debc9640 1 -- start start 2024-03-06T07:32:06.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.164+0000 7fa3debc9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3d80c6b10 0x7fa3d8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.164+0000 7fa3debc9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa3d80bf9d0 0x7fa3d80c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.164+0000 7fa3debc9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d80c2fc0 con 0x7fa3d8152650 2024-03-06T07:32:06.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.164+0000 7fa3debc9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d8155800 con 0x7fa3d80bf9d0 2024-03-06T07:32:06.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.164+0000 7fa3debc9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d8158da0 con 0x7fa3d80c6b10 2024-03-06T07:32:06.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.164+0000 7fa3d7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3d80c6b10 0x7fa3d8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.165+0000 7fa3dd166640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa3d80bf9d0 0x7fa3d80c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.165+0000 7fa3d7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3d80c6b10 0x7fa3d8155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:58348/0 (socket says 172.21.15.1:58348) 2024-03-06T07:32:06.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.165+0000 7fa3d7fff640 1 -- 172.21.15.1:0/1348623547 learned_addr learned my addr 172.21.15.1:0/1348623547 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:06.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.165+0000 7fa3d7fff640 1 -- 172.21.15.1:0/1348623547 >> v2:172.21.15.98:3300/0 conn(0x7fa3d80bf9d0 msgr2=0x7fa3d80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.165+0000 7fa3d7fff640 1 --2- 172.21.15.1:0/1348623547 >> v2:172.21.15.98:3300/0 conn(0x7fa3d80bf9d0 0x7fa3d80c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.165+0000 7fa3d7fff640 1 -- 172.21.15.1:0/1348623547 >> v1:172.21.15.98:6789/0 conn(0x7fa3d8152650 legacy=0x7fa3d8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.165+0000 7fa3d7fff640 1 -- 172.21.15.1:0/1348623547 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d80c6690 con 0x7fa3d80c6b10 2024-03-06T07:32:06.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.165+0000 7fa3dd166640 1 --2- 172.21.15.1:0/1348623547 >> v2:172.21.15.98:3300/0 conn(0x7fa3d80bf9d0 0x7fa3d80c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:06.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.166+0000 7fa3d7fff640 1 --2- 172.21.15.1:0/1348623547 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3d80c6b10 0x7fa3d8155c90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4009870 tx=0x7fa3c402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=415af6fef0b0eab3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.166+0000 7fa3d77fe640 1 -- 172.21.15.1:0/1348623547 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4035020 con 0x7fa3d80c6b10 2024-03-06T07:32:06.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.166+0000 7fa3d77fe640 1 -- 172.21.15.1:0/1348623547 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c4031640 con 0x7fa3d80c6b10 2024-03-06T07:32:06.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.166+0000 7fa3d77fe640 1 -- 172.21.15.1:0/1348623547 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c402d040 con 0x7fa3d80c6b10 2024-03-06T07:32:06.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.166+0000 7fa3debc9640 1 -- 172.21.15.1:0/1348623547 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3d80c6b10 msgr2=0x7fa3d8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.166+0000 7fa3debc9640 1 --2- 172.21.15.1:0/1348623547 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3d80c6b10 0x7fa3d8155c90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4009870 tx=0x7fa3c402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.166+0000 7fa3debc9640 1 -- 172.21.15.1:0/1348623547 shutdown_connections 2024-03-06T07:32:06.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.166+0000 7fa3debc9640 1 --2- 172.21.15.1:0/1348623547 >> v2:172.21.15.98:3300/0 conn(0x7fa3d80bf9d0 0x7fa3d80c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.166+0000 7fa3debc9640 1 --2- 172.21.15.1:0/1348623547 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3d80c6b10 0x7fa3d8155c90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.166+0000 7fa3debc9640 1 -- 172.21.15.1:0/1348623547 >> 172.21.15.1:0/1348623547 conn(0x7fa3d8066ee0 msgr2=0x7fa3d80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.166+0000 7fa3debc9640 1 -- 172.21.15.1:0/1348623547 shutdown_connections 2024-03-06T07:32:06.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.166+0000 7fa3debc9640 1 -- 172.21.15.1:0/1348623547 wait complete. 2024-03-06T07:32:06.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.167+0000 7fa3debc9640 1 Processor -- start 2024-03-06T07:32:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.168+0000 7fa3debc9640 1 -- start start 2024-03-06T07:32:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.168+0000 7fa3debc9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3d80bf9d0 0x7fa3d80c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.168+0000 7fa3debc9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d8152650 0x7fa3d814e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.168+0000 7fa3debc9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d80c87d0 0x7fa3d80c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.168+0000 7fa3debc9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d81551b0 con 0x7fa3d80c87d0 2024-03-06T07:32:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.168+0000 7fa3debc9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d80c6fa0 con 0x7fa3d8152650 2024-03-06T07:32:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.168+0000 7fa3debc9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d8155490 con 0x7fa3d80bf9d0 2024-03-06T07:32:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.168+0000 7fa3dd166640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d80c87d0 0x7fa3d80c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.168+0000 7fa3dd166640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d80c87d0 0x7fa3d80c8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32948/0 (socket says 172.21.15.1:32948) 2024-03-06T07:32:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.168+0000 7fa3dd166640 1 -- 172.21.15.1:0/4089165982 learned_addr learned my addr 172.21.15.1:0/4089165982 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.169+0000 7fa3dc965640 1 --2- 172.21.15.1:0/4089165982 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3d80bf9d0 0x7fa3d80c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.169+0000 7fa3d7fff640 1 --2- 172.21.15.1:0/4089165982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d8152650 0x7fa3d814e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.169+0000 7fa3dd166640 1 -- 172.21.15.1:0/4089165982 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3d80bf9d0 msgr2=0x7fa3d80c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.169+0000 7fa3dd166640 1 --2- 172.21.15.1:0/4089165982 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3d80bf9d0 0x7fa3d80c9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.169+0000 7fa3dd166640 1 -- 172.21.15.1:0/4089165982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d8152650 msgr2=0x7fa3d814e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.169+0000 7fa3dd166640 1 --2- 172.21.15.1:0/4089165982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d8152650 0x7fa3d814e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.169+0000 7fa3dd166640 1 -- 172.21.15.1:0/4089165982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d80ca610 con 0x7fa3d80c87d0 2024-03-06T07:32:06.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.169+0000 7fa3dd166640 1 --2- 172.21.15.1:0/4089165982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d80c87d0 0x7fa3d80c8c80 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009e60 tx=0x7fa3d000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.169+0000 7fa3d57fa640 1 -- 172.21.15.1:0/4089165982 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0018030 con 0x7fa3d80c87d0 2024-03-06T07:32:06.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.169+0000 7fa3d57fa640 1 -- 172.21.15.1:0/4089165982 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d000cdf0 con 0x7fa3d80c87d0 2024-03-06T07:32:06.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.169+0000 7fa3debc9640 1 -- 172.21.15.1:0/4089165982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d81ee950 con 0x7fa3d80c87d0 2024-03-06T07:32:06.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.169+0000 7fa3d57fa640 1 -- 172.21.15.1:0/4089165982 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0003bb0 con 0x7fa3d80c87d0 2024-03-06T07:32:06.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.169+0000 7fa3debc9640 1 -- 172.21.15.1:0/4089165982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d81eef10 con 0x7fa3d80c87d0 2024-03-06T07:32:06.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.170+0000 7fa3debc9640 1 -- 172.21.15.1:0/4089165982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d80c0d90 con 0x7fa3d80c87d0 2024-03-06T07:32:06.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.171+0000 7fa3d57fa640 1 -- 172.21.15.1:0/4089165982 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3d001b070 con 0x7fa3d80c87d0 2024-03-06T07:32:06.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.171+0000 7fa3d57fa640 1 --2- 172.21.15.1:0/4089165982 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3bc06b4d0 0x7fa3bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.171+0000 7fa3d57fa640 1 -- 172.21.15.1:0/4089165982 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa3d008ad50 con 0x7fa3d80c87d0 2024-03-06T07:32:06.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.171+0000 7fa3dc965640 1 --2- 172.21.15.1:0/4089165982 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3bc06b4d0 0x7fa3bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.172+0000 7fa3dc965640 1 --2- 172.21.15.1:0/4089165982 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3bc06b4d0 0x7fa3bc06d990 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc001780 tx=0x7fa3cc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.179+0000 7fa3d57fa640 1 -- 172.21.15.1:0/4089165982 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3d80c0d90 con 0x7fa3d80c87d0 2024-03-06T07:32:06.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.397+0000 7fa3debc9640 1 -- 172.21.15.1:0/4089165982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3d80cacc0 con 0x7fa3d80c87d0 2024-03-06T07:32:06.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.398+0000 7fa3d57fa640 1 -- 172.21.15.1:0/4089165982 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fa3d80cacc0 con 0x7fa3d80c87d0 2024-03-06T07:32:06.400 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:06.400 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:06.400 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.400+0000 7fa3debc9640 1 -- 172.21.15.1:0/4089165982 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3bc06b4d0 msgr2=0x7fa3bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.400+0000 7fa3debc9640 1 --2- 172.21.15.1:0/4089165982 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3bc06b4d0 0x7fa3bc06d990 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc001780 tx=0x7fa3cc009040 comp rx=0 tx=0).stop 2024-03-06T07:32:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.400+0000 7fa3debc9640 1 -- 172.21.15.1:0/4089165982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d80c87d0 msgr2=0x7fa3d80c8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.400+0000 7fa3debc9640 1 --2- 172.21.15.1:0/4089165982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d80c87d0 0x7fa3d80c8c80 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009e60 tx=0x7fa3d000c490 comp rx=0 tx=0).stop 2024-03-06T07:32:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.400+0000 7fa3debc9640 1 -- 172.21.15.1:0/4089165982 shutdown_connections 2024-03-06T07:32:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.400+0000 7fa3debc9640 1 --2- 172.21.15.1:0/4089165982 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa3bc06b4d0 0x7fa3bc06d990 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.400+0000 7fa3debc9640 1 --2- 172.21.15.1:0/4089165982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d80c87d0 0x7fa3d80c8c80 unknown :-1 s=CLOSED pgs=2567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.400+0000 7fa3debc9640 1 --2- 172.21.15.1:0/4089165982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d8152650 0x7fa3d814e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.400+0000 7fa3debc9640 1 --2- 172.21.15.1:0/4089165982 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa3d80bf9d0 0x7fa3d80c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.400+0000 7fa3debc9640 1 -- 172.21.15.1:0/4089165982 >> 172.21.15.1:0/4089165982 conn(0x7fa3d8066ee0 msgr2=0x7fa3d80c0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.400+0000 7fa3debc9640 1 -- 172.21.15.1:0/4089165982 shutdown_connections 2024-03-06T07:32:06.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:06.400+0000 7fa3debc9640 1 -- 172.21.15.1:0/4089165982 wait complete. 2024-03-06T07:32:06.815 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:06.815 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:07.816 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:07.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.916+0000 7f95c244e640 1 Processor -- start 2024-03-06T07:32:07.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.916+0000 7f95c244e640 1 -- start start 2024-03-06T07:32:07.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.916+0000 7f95c244e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f95bc072bd0 0x7f95bc0c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.916+0000 7f95c244e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95bc151dd0 con 0x7f95bc0c2b30 2024-03-06T07:32:07.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.916+0000 7f95c244e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95bc156190 con 0x7f95bc152b70 2024-03-06T07:32:07.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.916+0000 7f95c244e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95bc0c3f40 con 0x7f95bc072bd0 2024-03-06T07:32:07.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.916+0000 7f95bbfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f95bc0c2b30 0x7f95bc0724c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44072/0 (socket says 172.21.15.1:44072) 2024-03-06T07:32:07.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.916+0000 7f95bbfff640 1 -- 172.21.15.1:0/2360557289 learned_addr learned my addr 172.21.15.1:0/2360557289 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:07.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95baffd640 1 -- 172.21.15.1:0/2360557289 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939827536 0 0) 0x7f95bc151dd0 con 0x7f95bc0c2b30 2024-03-06T07:32:07.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95baffd640 1 -- 172.21.15.1:0/2360557289 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95a4003520 con 0x7f95bc0c2b30 2024-03-06T07:32:07.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95bb7fe640 1 --2- 172.21.15.1:0/2360557289 >> v2:172.21.15.98:3300/0 conn(0x7f95bc072bd0 0x7f95bc0c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95baffd640 1 -- 172.21.15.1:0/2360557289 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3179261915 0 0) 0x7f95a4003520 con 0x7f95bc0c2b30 2024-03-06T07:32:07.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95baffd640 1 -- 172.21.15.1:0/2360557289 >> v2:172.21.15.98:3300/0 conn(0x7f95bc072bd0 msgr2=0x7f95bc0c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:07.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95baffd640 1 --2- 172.21.15.1:0/2360557289 >> v2:172.21.15.98:3300/0 conn(0x7f95bc072bd0 0x7f95bc0c32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95baffd640 1 -- 172.21.15.1:0/2360557289 >> v1:172.21.15.98:6789/0 conn(0x7f95bc152b70 legacy=0x7f95bc0c3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95baffd640 1 -- 172.21.15.1:0/2360557289 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc152090 con 0x7f95bc0c2b30 2024-03-06T07:32:07.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95baffd640 1 -- 172.21.15.1:0/2360557289 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f95ac001670 con 0x7f95bc0c2b30 2024-03-06T07:32:07.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95baffd640 1 -- 172.21.15.1:0/2360557289 >> v1:172.21.15.1:6789/0 conn(0x7f95bc0c2b30 legacy=0x7f95bc0724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95baffd640 1 --2- 172.21.15.1:0/2360557289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95a40039b0 0x7f95a4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95baffd640 1 --2- 172.21.15.1:0/2360557289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a4004800 0x7f95a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95baffd640 1 --2- 172.21.15.1:0/2360557289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a40051c0 0x7f95a400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95baffd640 1 -- 172.21.15.1:0/2360557289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc0c3f40 con 0x7f95a4004800 2024-03-06T07:32:07.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95baffd640 1 -- 172.21.15.1:0/2360557289 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a400c110 con 0x7f95a40051c0 2024-03-06T07:32:07.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95baffd640 1 -- 172.21.15.1:0/2360557289 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a400c310 con 0x7f95a40039b0 2024-03-06T07:32:07.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95bb7fe640 1 --2- 172.21.15.1:0/2360557289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a4004800 0x7f95a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95bbfff640 1 --2- 172.21.15.1:0/2360557289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95a40039b0 0x7f95a4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.917+0000 7f95c09eb640 1 --2- 172.21.15.1:0/2360557289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a40051c0 0x7f95a400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.918+0000 7f95bb7fe640 1 -- 172.21.15.1:0/2360557289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95a40039b0 msgr2=0x7f95a4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:07.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.918+0000 7f95bb7fe640 1 --2- 172.21.15.1:0/2360557289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95a40039b0 0x7f95a4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.918+0000 7f95bb7fe640 1 -- 172.21.15.1:0/2360557289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a40051c0 msgr2=0x7f95a400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:07.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.918+0000 7f95bb7fe640 1 --2- 172.21.15.1:0/2360557289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a40051c0 0x7f95a400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.918+0000 7f95bb7fe640 1 -- 172.21.15.1:0/2360557289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a4003520 con 0x7f95a4004800 2024-03-06T07:32:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.918+0000 7f95c09eb640 1 --2- 172.21.15.1:0/2360557289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95a40051c0 0x7f95a400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.918+0000 7f95c09eb640 1 -- 172.21.15.1:0/2360557289 reap_dead start 2024-03-06T07:32:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.918+0000 7f95bb7fe640 1 --2- 172.21.15.1:0/2360557289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a4004800 0x7f95a4004c80 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f95b000ba00 tx=0x7f95a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=756a00d82f14c8c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.918+0000 7f95baffd640 1 -- 172.21.15.1:0/2360557289 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b0036020 con 0x7f95a4004800 2024-03-06T07:32:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.919+0000 7f95baffd640 1 -- 172.21.15.1:0/2360557289 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b0032620 con 0x7f95a4004800 2024-03-06T07:32:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.919+0000 7f95baffd640 1 -- 172.21.15.1:0/2360557289 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b000b600 con 0x7f95a4004800 2024-03-06T07:32:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.919+0000 7f95c244e640 1 -- 172.21.15.1:0/2360557289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a4004800 msgr2=0x7f95a4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.919+0000 7f95c244e640 1 --2- 172.21.15.1:0/2360557289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a4004800 0x7f95a4004c80 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f95b000ba00 tx=0x7f95a4005830 comp rx=0 tx=0).stop 2024-03-06T07:32:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.919+0000 7f95c244e640 1 -- 172.21.15.1:0/2360557289 shutdown_connections 2024-03-06T07:32:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.919+0000 7f95c244e640 1 --2- 172.21.15.1:0/2360557289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a4004800 0x7f95a4004c80 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.919+0000 7f95c244e640 1 -- 172.21.15.1:0/2360557289 >> 172.21.15.1:0/2360557289 conn(0x7f95bc0c6860 msgr2=0x7f95bc0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:07.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.919+0000 7f95c244e640 1 -- 172.21.15.1:0/2360557289 shutdown_connections 2024-03-06T07:32:07.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.919+0000 7f95c244e640 1 -- 172.21.15.1:0/2360557289 wait complete. 2024-03-06T07:32:07.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.920+0000 7f95c244e640 1 Processor -- start 2024-03-06T07:32:07.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.920+0000 7f95c244e640 1 -- start start 2024-03-06T07:32:07.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.921+0000 7f95c244e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a4004800 0x7f95bc149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.921+0000 7f95c244e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95bc1eb060 0x7f95bc1dd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.921+0000 7f95c244e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95bc1dde70 0x7f95bc14b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.921+0000 7f95c244e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc155d80 con 0x7f95a4004800 2024-03-06T07:32:07.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.921+0000 7f95c244e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc1526e0 con 0x7f95bc1eb060 2024-03-06T07:32:07.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.921+0000 7f95c244e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc155f00 con 0x7f95bc1dde70 2024-03-06T07:32:07.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.921+0000 7f95bbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a4004800 0x7f95bc149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.921+0000 7f95bbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a4004800 0x7f95bc149fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32968/0 (socket says 172.21.15.1:32968) 2024-03-06T07:32:07.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.921+0000 7f95bbfff640 1 -- 172.21.15.1:0/3394311220 learned_addr learned my addr 172.21.15.1:0/3394311220 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:07.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.921+0000 7f95c09eb640 1 --2- 172.21.15.1:0/3394311220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95bc1dde70 0x7f95bc14b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.921+0000 7f95bb7fe640 1 --2- 172.21.15.1:0/3394311220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95bc1eb060 0x7f95bc1dd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.921+0000 7f95bbfff640 1 -- 172.21.15.1:0/3394311220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95bc1dde70 msgr2=0x7f95bc14b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.921+0000 7f95bbfff640 1 --2- 172.21.15.1:0/3394311220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95bc1dde70 0x7f95bc14b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.921+0000 7f95bbfff640 1 -- 172.21.15.1:0/3394311220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95bc1eb060 msgr2=0x7f95bc1dd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.921+0000 7f95bbfff640 1 --2- 172.21.15.1:0/3394311220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95bc1eb060 0x7f95bc1dd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.921+0000 7f95bbfff640 1 -- 172.21.15.1:0/3394311220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc1eb540 con 0x7f95a4004800 2024-03-06T07:32:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.921+0000 7f95bb7fe640 1 --2- 172.21.15.1:0/3394311220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95bc1eb060 0x7f95bc1dd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.922+0000 7f95bbfff640 1 --2- 172.21.15.1:0/3394311220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a4004800 0x7f95bc149fe0 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f95ac007290 tx=0x7f95ac007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.922+0000 7f95b8ff9640 1 -- 172.21.15.1:0/3394311220 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac00f960 con 0x7f95a4004800 2024-03-06T07:32:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.922+0000 7f95b8ff9640 1 -- 172.21.15.1:0/3394311220 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95ac018070 con 0x7f95a4004800 2024-03-06T07:32:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.922+0000 7f95c244e640 1 -- 172.21.15.1:0/3394311220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95bc1e8000 con 0x7f95a4004800 2024-03-06T07:32:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.922+0000 7f95b8ff9640 1 -- 172.21.15.1:0/3394311220 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac00fd60 con 0x7f95a4004800 2024-03-06T07:32:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.922+0000 7f95c244e640 1 -- 172.21.15.1:0/3394311220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95bc1e7520 con 0x7f95a4004800 2024-03-06T07:32:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.923+0000 7f95c244e640 1 -- 172.21.15.1:0/3394311220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95bc1502a0 con 0x7f95a4004800 2024-03-06T07:32:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.923+0000 7f95b8ff9640 1 -- 172.21.15.1:0/3394311220 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f95ac01c410 con 0x7f95a4004800 2024-03-06T07:32:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.924+0000 7f95b8ff9640 1 --2- 172.21.15.1:0/3394311220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f95a806b500 0x7f95a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.924+0000 7f95b8ff9640 1 -- 172.21.15.1:0/3394311220 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f95ac08a0f0 con 0x7f95a4004800 2024-03-06T07:32:07.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.924+0000 7f95bb7fe640 1 --2- 172.21.15.1:0/3394311220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f95a806b500 0x7f95a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.925+0000 7f95bb7fe640 1 --2- 172.21.15.1:0/3394311220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f95a806b500 0x7f95a806d9c0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f95bc072fb0 tx=0x7f95b0030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:07.928+0000 7f95b8ff9640 1 -- 172.21.15.1:0/3394311220 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f95bc1502a0 con 0x7f95a4004800 2024-03-06T07:32:08.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:08.151+0000 7f95c244e640 1 -- 172.21.15.1:0/3394311220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95bc067fd0 con 0x7f95a4004800 2024-03-06T07:32:08.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:08.151+0000 7f95b8ff9640 1 -- 172.21.15.1:0/3394311220 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f95ac058d70 con 0x7f95a4004800 2024-03-06T07:32:08.153 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:08.154 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:08.154 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:08.153+0000 7f95c244e640 1 -- 172.21.15.1:0/3394311220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f95a806b500 msgr2=0x7f95a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:08.153+0000 7f95c244e640 1 --2- 172.21.15.1:0/3394311220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f95a806b500 0x7f95a806d9c0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f95bc072fb0 tx=0x7f95b0030000 comp rx=0 tx=0).stop 2024-03-06T07:32:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:08.153+0000 7f95c244e640 1 -- 172.21.15.1:0/3394311220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a4004800 msgr2=0x7f95bc149fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:08.153+0000 7f95c244e640 1 --2- 172.21.15.1:0/3394311220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a4004800 0x7f95bc149fe0 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f95ac007290 tx=0x7f95ac007750 comp rx=0 tx=0).stop 2024-03-06T07:32:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:08.153+0000 7f95c244e640 1 -- 172.21.15.1:0/3394311220 shutdown_connections 2024-03-06T07:32:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:08.154+0000 7f95c244e640 1 --2- 172.21.15.1:0/3394311220 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f95a806b500 0x7f95a806d9c0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:08.154+0000 7f95c244e640 1 --2- 172.21.15.1:0/3394311220 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95bc1dde70 0x7f95bc14b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:08.154+0000 7f95c244e640 1 --2- 172.21.15.1:0/3394311220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95bc1eb060 0x7f95bc1dd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:08.154+0000 7f95c244e640 1 --2- 172.21.15.1:0/3394311220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a4004800 0x7f95bc149fe0 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:08.154+0000 7f95c244e640 1 -- 172.21.15.1:0/3394311220 >> 172.21.15.1:0/3394311220 conn(0x7f95bc0c6860 msgr2=0x7f95bc151700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:08.154+0000 7f95c244e640 1 -- 172.21.15.1:0/3394311220 shutdown_connections 2024-03-06T07:32:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:08.154+0000 7f95c244e640 1 -- 172.21.15.1:0/3394311220 wait complete. 2024-03-06T07:32:08.568 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:08.569 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:09.570 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:09.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.697+0000 7f1a34c1c640 1 Processor -- start 2024-03-06T07:32:09.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.697+0000 7f1a34c1c640 1 -- start start 2024-03-06T07:32:09.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.698+0000 7f1a34c1c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a300c6b10 0x7f1a30155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.698+0000 7f1a34c1c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a300c2fc0 con 0x7f1a30152650 2024-03-06T07:32:09.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.698+0000 7f1a34c1c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a30158da0 con 0x7f1a300bf9d0 2024-03-06T07:32:09.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.698+0000 7f1a34c1c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a30155800 con 0x7f1a300c6b10 2024-03-06T07:32:09.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.698+0000 7f1a2e59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1a30152650 0x7f1a30158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44084/0 (socket says 172.21.15.1:44084) 2024-03-06T07:32:09.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.698+0000 7f1a2e59c640 1 -- 172.21.15.1:0/2297613558 learned_addr learned my addr 172.21.15.1:0/2297613558 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:09.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.698+0000 7f1a25d9b640 1 --2- 172.21.15.1:0/2297613558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a300c6b10 0x7f1a30155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.698+0000 7f1a2dd9b640 1 -- 172.21.15.1:0/2297613558 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762939699 0 0) 0x7f1a300c2fc0 con 0x7f1a30152650 2024-03-06T07:32:09.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.698+0000 7f1a2dd9b640 1 -- 172.21.15.1:0/2297613558 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a1c003520 con 0x7f1a30152650 2024-03-06T07:32:09.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a2dd9b640 1 -- 172.21.15.1:0/2297613558 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2499932035 0 0) 0x7f1a1c003520 con 0x7f1a30152650 2024-03-06T07:32:09.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a2dd9b640 1 -- 172.21.15.1:0/2297613558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a300c6b10 msgr2=0x7f1a30155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:09.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a2dd9b640 1 --2- 172.21.15.1:0/2297613558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a300c6b10 0x7f1a30155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a2dd9b640 1 -- 172.21.15.1:0/2297613558 >> v1:172.21.15.98:6789/0 conn(0x7f1a300bf9d0 legacy=0x7f1a300c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a2dd9b640 1 -- 172.21.15.1:0/2297613558 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a300c6690 con 0x7f1a30152650 2024-03-06T07:32:09.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a2dd9b640 1 -- 172.21.15.1:0/2297613558 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f1a20001670 con 0x7f1a30152650 2024-03-06T07:32:09.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a2dd9b640 1 -- 172.21.15.1:0/2297613558 >> v1:172.21.15.1:6789/0 conn(0x7f1a30152650 legacy=0x7f1a30158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a2dd9b640 1 --2- 172.21.15.1:0/2297613558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a1c0039e0 0x7f1a1c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a2dd9b640 1 --2- 172.21.15.1:0/2297613558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1c004800 0x7f1a1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a2dd9b640 1 --2- 172.21.15.1:0/2297613558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a1c0051c0 0x7f1a1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a2dd9b640 1 -- 172.21.15.1:0/2297613558 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a30155800 con 0x7f1a1c004800 2024-03-06T07:32:09.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a2dd9b640 1 -- 172.21.15.1:0/2297613558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c00bf00 con 0x7f1a1c0039e0 2024-03-06T07:32:09.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a2dd9b640 1 -- 172.21.15.1:0/2297613558 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c00c0d0 con 0x7f1a1c0051c0 2024-03-06T07:32:09.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a25d9b640 1 --2- 172.21.15.1:0/2297613558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1c004800 0x7f1a1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a2e59c640 1 --2- 172.21.15.1:0/2297613558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a1c0039e0 0x7f1a1c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a2ed9d640 1 --2- 172.21.15.1:0/2297613558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a1c0051c0 0x7f1a1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a25d9b640 1 -- 172.21.15.1:0/2297613558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a1c0051c0 msgr2=0x7f1a1c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:09.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a25d9b640 1 --2- 172.21.15.1:0/2297613558 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a1c0051c0 0x7f1a1c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a25d9b640 1 -- 172.21.15.1:0/2297613558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a1c0039e0 msgr2=0x7f1a1c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:09.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a25d9b640 1 --2- 172.21.15.1:0/2297613558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a1c0039e0 0x7f1a1c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.699+0000 7f1a25d9b640 1 -- 172.21.15.1:0/2297613558 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c003520 con 0x7f1a1c004800 2024-03-06T07:32:09.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.700+0000 7f1a25d9b640 1 --2- 172.21.15.1:0/2297613558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1c004800 0x7f1a1c004c80 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f1a18009ac0 tx=0x7f1a1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9990193f20d02f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.700+0000 7f1a2dd9b640 1 -- 172.21.15.1:0/2297613558 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a18039070 con 0x7f1a1c004800 2024-03-06T07:32:09.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.700+0000 7f1a2dd9b640 1 -- 172.21.15.1:0/2297613558 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1802c6b0 con 0x7f1a1c004800 2024-03-06T07:32:09.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.700+0000 7f1a2dd9b640 1 -- 172.21.15.1:0/2297613558 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a18009690 con 0x7f1a1c004800 2024-03-06T07:32:09.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.700+0000 7f1a34c1c640 1 -- 172.21.15.1:0/2297613558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1c004800 msgr2=0x7f1a1c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.700+0000 7f1a34c1c640 1 --2- 172.21.15.1:0/2297613558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a1c004800 0x7f1a1c004c80 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f1a18009ac0 tx=0x7f1a1c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:09.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.700+0000 7f1a2ed9d640 1 -- 172.21.15.1:0/2297613558 reap_dead start 2024-03-06T07:32:09.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.700+0000 7f1a34c1c640 1 -- 172.21.15.1:0/2297613558 shutdown_connections 2024-03-06T07:32:09.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.701+0000 7f1a34c1c640 1 -- 172.21.15.1:0/2297613558 >> 172.21.15.1:0/2297613558 conn(0x7f1a30066ee0 msgr2=0x7f1a300672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.701+0000 7f1a34c1c640 1 -- 172.21.15.1:0/2297613558 shutdown_connections 2024-03-06T07:32:09.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.701+0000 7f1a34c1c640 1 -- 172.21.15.1:0/2297613558 wait complete. 2024-03-06T07:32:09.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.702+0000 7f1a34c1c640 1 Processor -- start 2024-03-06T07:32:09.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.702+0000 7f1a34c1c640 1 -- start start 2024-03-06T07:32:09.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.703+0000 7f1a34c1c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a1c004800 0x7f1a301eeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.703+0000 7f1a34c1c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a30150dd0 0x7f1a30151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.703+0000 7f1a34c1c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a3014ae20 0x7f1a3014b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.703+0000 7f1a34c1c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a301551b0 con 0x7f1a30150dd0 2024-03-06T07:32:09.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.703+0000 7f1a34c1c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a300c6fa0 con 0x7f1a3014ae20 2024-03-06T07:32:09.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.703+0000 7f1a34c1c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a30155490 con 0x7f1a1c004800 2024-03-06T07:32:09.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.703+0000 7f1a25d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a30150dd0 0x7f1a30151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.703+0000 7f1a2e59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a1c004800 0x7f1a301eeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.703+0000 7f1a25d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a30150dd0 0x7f1a30151230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32990/0 (socket says 172.21.15.1:32990) 2024-03-06T07:32:09.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.703+0000 7f1a25d9b640 1 -- 172.21.15.1:0/3876929304 learned_addr learned my addr 172.21.15.1:0/3876929304 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:09.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.703+0000 7f1a2ed9d640 1 --2- 172.21.15.1:0/3876929304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a3014ae20 0x7f1a3014b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.703+0000 7f1a25d9b640 1 -- 172.21.15.1:0/3876929304 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a1c004800 msgr2=0x7f1a301eeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.703+0000 7f1a25d9b640 1 --2- 172.21.15.1:0/3876929304 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a1c004800 0x7f1a301eeb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.703+0000 7f1a25d9b640 1 -- 172.21.15.1:0/3876929304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a3014ae20 msgr2=0x7f1a3014b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.703+0000 7f1a25d9b640 1 --2- 172.21.15.1:0/3876929304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a3014ae20 0x7f1a3014b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.703+0000 7f1a25d9b640 1 -- 172.21.15.1:0/3876929304 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a301567c0 con 0x7f1a30150dd0 2024-03-06T07:32:09.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.704+0000 7f1a25d9b640 1 --2- 172.21.15.1:0/3876929304 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a30150dd0 0x7f1a30151230 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f1a18000c00 tx=0x7f1a18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.704+0000 7f1a277fe640 1 -- 172.21.15.1:0/3876929304 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a18042020 con 0x7f1a30150dd0 2024-03-06T07:32:09.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.704+0000 7f1a277fe640 1 -- 172.21.15.1:0/3876929304 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1803a040 con 0x7f1a30150dd0 2024-03-06T07:32:09.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.704+0000 7f1a34c1c640 1 -- 172.21.15.1:0/3876929304 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a30154010 con 0x7f1a30150dd0 2024-03-06T07:32:09.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.704+0000 7f1a277fe640 1 -- 172.21.15.1:0/3876929304 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a18039070 con 0x7f1a30150dd0 2024-03-06T07:32:09.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.704+0000 7f1a34c1c640 1 -- 172.21.15.1:0/3876929304 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a301545d0 con 0x7f1a30150dd0 2024-03-06T07:32:09.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.705+0000 7f1a34c1c640 1 -- 172.21.15.1:0/3876929304 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a30067fd0 con 0x7f1a30150dd0 2024-03-06T07:32:09.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.705+0000 7f1a277fe640 1 -- 172.21.15.1:0/3876929304 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a18045020 con 0x7f1a30150dd0 2024-03-06T07:32:09.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.706+0000 7f1a277fe640 1 --2- 172.21.15.1:0/3876929304 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a1006b4d0 0x7f1a1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.706+0000 7f1a277fe640 1 -- 172.21.15.1:0/3876929304 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f1a180ab7f0 con 0x7f1a30150dd0 2024-03-06T07:32:09.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.706+0000 7f1a2e59c640 1 --2- 172.21.15.1:0/3876929304 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a1006b4d0 0x7f1a1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.706+0000 7f1a2e59c640 1 --2- 172.21.15.1:0/3876929304 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a1006b4d0 0x7f1a1006d990 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f1a200011e0 tx=0x7f1a200078f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:09.710+0000 7f1a277fe640 1 -- 172.21.15.1:0/3876929304 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a1804c070 con 0x7f1a30150dd0 2024-03-06T07:32:10.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:10.020+0000 7f1a34c1c640 1 -- 172.21.15.1:0/3876929304 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a300686e0 con 0x7f1a30150dd0 2024-03-06T07:32:10.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:10.020+0000 7f1a277fe640 1 -- 172.21.15.1:0/3876929304 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f1a18079e20 con 0x7f1a30150dd0 2024-03-06T07:32:10.023 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:10.023 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:10.023 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:10.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:10.023+0000 7f1a34c1c640 1 -- 172.21.15.1:0/3876929304 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a1006b4d0 msgr2=0x7f1a1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:10.023+0000 7f1a34c1c640 1 --2- 172.21.15.1:0/3876929304 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a1006b4d0 0x7f1a1006d990 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f1a200011e0 tx=0x7f1a200078f0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:10.023+0000 7f1a34c1c640 1 -- 172.21.15.1:0/3876929304 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a30150dd0 msgr2=0x7f1a30151230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:10.023+0000 7f1a34c1c640 1 --2- 172.21.15.1:0/3876929304 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a30150dd0 0x7f1a30151230 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f1a18000c00 tx=0x7f1a18004730 comp rx=0 tx=0).stop 2024-03-06T07:32:10.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:10.023+0000 7f1a34c1c640 1 -- 172.21.15.1:0/3876929304 shutdown_connections 2024-03-06T07:32:10.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:10.023+0000 7f1a34c1c640 1 --2- 172.21.15.1:0/3876929304 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1a1006b4d0 0x7f1a1006d990 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:10.023+0000 7f1a34c1c640 1 --2- 172.21.15.1:0/3876929304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a3014ae20 0x7f1a3014b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:10.023+0000 7f1a34c1c640 1 --2- 172.21.15.1:0/3876929304 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a30150dd0 0x7f1a30151230 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:10.023+0000 7f1a34c1c640 1 --2- 172.21.15.1:0/3876929304 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a1c004800 0x7f1a301eeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:10.023+0000 7f1a34c1c640 1 -- 172.21.15.1:0/3876929304 >> 172.21.15.1:0/3876929304 conn(0x7f1a30066ee0 msgr2=0x7f1a3014e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:10.023+0000 7f1a34c1c640 1 -- 172.21.15.1:0/3876929304 shutdown_connections 2024-03-06T07:32:10.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:10.023+0000 7f1a34c1c640 1 -- 172.21.15.1:0/3876929304 wait complete. 2024-03-06T07:32:10.438 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:10.438 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:11.439 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:11.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.540+0000 7f80af532640 1 Processor -- start 2024-03-06T07:32:11.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.540+0000 7f80af532640 1 -- start start 2024-03-06T07:32:11.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.541+0000 7f80af532640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80a8152650 0x7f80a8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.541+0000 7f80af532640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f80a80c6b10 0x7f80a8158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.541+0000 7f80af532640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f80a80bf9d0 0x7f80a80c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.541+0000 7f80af532640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a8155800 con 0x7f80a80bf9d0 2024-03-06T07:32:11.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.541+0000 7f80af532640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a80c2fc0 con 0x7f80a80c6b10 2024-03-06T07:32:11.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.541+0000 7f80af532640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a80c6690 con 0x7f80a8152650 2024-03-06T07:32:11.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.541+0000 7f80adacf640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f80a80bf9d0 0x7f80a80c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.541+0000 7f80adacf640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f80a80bf9d0 0x7f80a80c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32998/0 (socket says 172.21.15.1:32998) 2024-03-06T07:32:11.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.541+0000 7f80adacf640 1 -- 172.21.15.1:0/2121254590 learned_addr learned my addr 172.21.15.1:0/2121254590 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:11.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.541+0000 7f80ad2ce640 1 --2- 172.21.15.1:0/2121254590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80a8152650 0x7f80a8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.541+0000 7f80acacd640 1 --2- 172.21.15.1:0/2121254590 >> v2:172.21.15.98:3300/0 conn(0x7f80a80c6b10 0x7f80a8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.541+0000 7f80adacf640 1 -- 172.21.15.1:0/2121254590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80a8152650 msgr2=0x7f80a8155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.541+0000 7f80adacf640 1 --2- 172.21.15.1:0/2121254590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80a8152650 0x7f80a8155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.541+0000 7f80adacf640 1 -- 172.21.15.1:0/2121254590 >> v2:172.21.15.98:3300/0 conn(0x7f80a80c6b10 msgr2=0x7f80a8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.541+0000 7f80adacf640 1 --2- 172.21.15.1:0/2121254590 >> v2:172.21.15.98:3300/0 conn(0x7f80a80c6b10 0x7f80a8158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.541+0000 7f80adacf640 1 -- 172.21.15.1:0/2121254590 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a80c7280 con 0x7f80a80bf9d0 2024-03-06T07:32:11.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.541+0000 7f80acacd640 1 --2- 172.21.15.1:0/2121254590 >> v2:172.21.15.98:3300/0 conn(0x7f80a80c6b10 0x7f80a8158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:11.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.542+0000 7f80adacf640 1 --2- 172.21.15.1:0/2121254590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80a80bf9d0 0x7f80a80c1460 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f80a808c500 tx=0x7f80a002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d418ca0000f1b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.542+0000 7f809bfff640 1 -- 172.21.15.1:0/2121254590 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a002c9e0 con 0x7f80a80bf9d0 2024-03-06T07:32:11.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.542+0000 7f809bfff640 1 -- 172.21.15.1:0/2121254590 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a002cb80 con 0x7f80a80bf9d0 2024-03-06T07:32:11.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.542+0000 7f809bfff640 1 -- 172.21.15.1:0/2121254590 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a00343f0 con 0x7f80a80bf9d0 2024-03-06T07:32:11.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.543+0000 7f80af532640 1 -- 172.21.15.1:0/2121254590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80a80bf9d0 msgr2=0x7f80a80c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.543+0000 7f80af532640 1 --2- 172.21.15.1:0/2121254590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80a80bf9d0 0x7f80a80c1460 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f80a808c500 tx=0x7f80a002be40 comp rx=0 tx=0).stop 2024-03-06T07:32:11.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.543+0000 7f80af532640 1 -- 172.21.15.1:0/2121254590 shutdown_connections 2024-03-06T07:32:11.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.543+0000 7f80af532640 1 --2- 172.21.15.1:0/2121254590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80a80bf9d0 0x7f80a80c1460 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.543+0000 7f80af532640 1 --2- 172.21.15.1:0/2121254590 >> v2:172.21.15.98:3300/0 conn(0x7f80a80c6b10 0x7f80a8158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.543+0000 7f80af532640 1 --2- 172.21.15.1:0/2121254590 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80a8152650 0x7f80a8155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.543+0000 7f80af532640 1 -- 172.21.15.1:0/2121254590 >> 172.21.15.1:0/2121254590 conn(0x7f80a8066ee0 msgr2=0x7f80a80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.543+0000 7f80af532640 1 -- 172.21.15.1:0/2121254590 shutdown_connections 2024-03-06T07:32:11.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.543+0000 7f80af532640 1 -- 172.21.15.1:0/2121254590 wait complete. 2024-03-06T07:32:11.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.544+0000 7f80af532640 1 Processor -- start 2024-03-06T07:32:11.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.544+0000 7f80af532640 1 -- start start 2024-03-06T07:32:11.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80af532640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80a80bf9d0 0x7f80a814c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80af532640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80a8152650 0x7f80a81eeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80af532640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80a80c5520 0x7f80a80c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80af532640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a8155490 con 0x7f80a8152650 2024-03-06T07:32:11.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80af532640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a81551b0 con 0x7f80a80c5520 2024-03-06T07:32:11.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80af532640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a8152a30 con 0x7f80a80bf9d0 2024-03-06T07:32:11.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80acacd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80a8152650 0x7f80a81eeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80acacd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80a8152650 0x7f80a81eeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33004/0 (socket says 172.21.15.1:33004) 2024-03-06T07:32:11.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80acacd640 1 -- 172.21.15.1:0/1434147869 learned_addr learned my addr 172.21.15.1:0/1434147869 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:11.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80ad2ce640 1 --2- 172.21.15.1:0/1434147869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80a80bf9d0 0x7f80a814c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80adacf640 1 --2- 172.21.15.1:0/1434147869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80a80c5520 0x7f80a80c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80acacd640 1 -- 172.21.15.1:0/1434147869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80a80bf9d0 msgr2=0x7f80a814c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80acacd640 1 --2- 172.21.15.1:0/1434147869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80a80bf9d0 0x7f80a814c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80acacd640 1 -- 172.21.15.1:0/1434147869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80a80c5520 msgr2=0x7f80a80c9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80acacd640 1 --2- 172.21.15.1:0/1434147869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80a80c5520 0x7f80a80c9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80acacd640 1 -- 172.21.15.1:0/1434147869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a814cff0 con 0x7f80a8152650 2024-03-06T07:32:11.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80acacd640 1 --2- 172.21.15.1:0/1434147869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80a8152650 0x7f80a81eeba0 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f80a400c7f0 tx=0x7f80a400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f8099ffb640 1 -- 172.21.15.1:0/1434147869 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a4015070 con 0x7f80a8152650 2024-03-06T07:32:11.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f8099ffb640 1 -- 172.21.15.1:0/1434147869 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a40077c0 con 0x7f80a8152650 2024-03-06T07:32:11.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f80af532640 1 -- 172.21.15.1:0/1434147869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a80c89b0 con 0x7f80a8152650 2024-03-06T07:32:11.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.545+0000 7f8099ffb640 1 -- 172.21.15.1:0/1434147869 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a4007bb0 con 0x7f80a8152650 2024-03-06T07:32:11.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.546+0000 7f80af532640 1 -- 172.21.15.1:0/1434147869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a80c59b0 con 0x7f80a8152650 2024-03-06T07:32:11.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.547+0000 7f80af532640 1 -- 172.21.15.1:0/1434147869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a8067fd0 con 0x7f80a8152650 2024-03-06T07:32:11.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.547+0000 7f8099ffb640 1 -- 172.21.15.1:0/1434147869 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f80a4018070 con 0x7f80a8152650 2024-03-06T07:32:11.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.547+0000 7f8099ffb640 1 --2- 172.21.15.1:0/1434147869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f808c06b4d0 0x7f808c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.548+0000 7f8099ffb640 1 -- 172.21.15.1:0/1434147869 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f80a408a540 con 0x7f80a8152650 2024-03-06T07:32:11.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.548+0000 7f80ad2ce640 1 --2- 172.21.15.1:0/1434147869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f808c06b4d0 0x7f808c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.548+0000 7f80ad2ce640 1 --2- 172.21.15.1:0/1434147869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f808c06b4d0 0x7f808c06d990 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f809c006fd0 tx=0x7f809c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.552+0000 7f8099ffb640 1 -- 172.21.15.1:0/1434147869 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f80a40589d0 con 0x7f80a8152650 2024-03-06T07:32:11.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.860+0000 7f80af532640 1 -- 172.21.15.1:0/1434147869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80a80b06f0 con 0x7f80a8152650 2024-03-06T07:32:11.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.861+0000 7f8099ffb640 1 -- 172.21.15.1:0/1434147869 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f80a405ca70 con 0x7f80a8152650 2024-03-06T07:32:11.863 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:11.863 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:11.863 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:11.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.863+0000 7f80af532640 1 -- 172.21.15.1:0/1434147869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f808c06b4d0 msgr2=0x7f808c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.863+0000 7f80af532640 1 --2- 172.21.15.1:0/1434147869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f808c06b4d0 0x7f808c06d990 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f809c006fd0 tx=0x7f809c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:11.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.863+0000 7f80af532640 1 -- 172.21.15.1:0/1434147869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80a8152650 msgr2=0x7f80a81eeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.863+0000 7f80af532640 1 --2- 172.21.15.1:0/1434147869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80a8152650 0x7f80a81eeba0 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f80a400c7f0 tx=0x7f80a400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.863+0000 7f80af532640 1 -- 172.21.15.1:0/1434147869 shutdown_connections 2024-03-06T07:32:11.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.863+0000 7f80af532640 1 --2- 172.21.15.1:0/1434147869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f808c06b4d0 0x7f808c06d990 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.863+0000 7f80af532640 1 --2- 172.21.15.1:0/1434147869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80a80c5520 0x7f80a80c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.863+0000 7f80af532640 1 --2- 172.21.15.1:0/1434147869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80a8152650 0x7f80a81eeba0 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.863+0000 7f80af532640 1 --2- 172.21.15.1:0/1434147869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f80a80bf9d0 0x7f80a814c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.863+0000 7f80af532640 1 -- 172.21.15.1:0/1434147869 >> 172.21.15.1:0/1434147869 conn(0x7f80a8066ee0 msgr2=0x7f80a80672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.863+0000 7f80af532640 1 -- 172.21.15.1:0/1434147869 shutdown_connections 2024-03-06T07:32:11.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:11.863+0000 7f80af532640 1 -- 172.21.15.1:0/1434147869 wait complete. 2024-03-06T07:32:12.278 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:12.278 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:13.279 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.379+0000 7fa2c48df640 1 Processor -- start 2024-03-06T07:32:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.379+0000 7fa2c48df640 1 -- start start 2024-03-06T07:32:13.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.380+0000 7fa2c48df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c0152650 0x7fa2c0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.380+0000 7fa2c48df640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2c00c6b10 0x7fa2c0158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.380+0000 7fa2c48df640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa2c00bf9d0 0x7fa2c00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.380+0000 7fa2c48df640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c0155800 con 0x7fa2c00c6b10 2024-03-06T07:32:13.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.380+0000 7fa2c48df640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c00c2fc0 con 0x7fa2c00bf9d0 2024-03-06T07:32:13.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.380+0000 7fa2c48df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c00c6690 con 0x7fa2c0152650 2024-03-06T07:32:13.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.380+0000 7fa2bdd9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2c00c6b10 0x7fa2c0158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.380+0000 7fa2bdd9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2c00c6b10 0x7fa2c0158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33012/0 (socket says 172.21.15.1:33012) 2024-03-06T07:32:13.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.380+0000 7fa2bdd9b640 1 -- 172.21.15.1:0/3356709268 learned_addr learned my addr 172.21.15.1:0/3356709268 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:13.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.380+0000 7fa2be59c640 1 --2- 172.21.15.1:0/3356709268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c0152650 0x7fa2c0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.380+0000 7fa2bdd9b640 1 -- 172.21.15.1:0/3356709268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c0152650 msgr2=0x7fa2c0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:13.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.380+0000 7fa2bed9d640 1 --2- 172.21.15.1:0/3356709268 >> v2:172.21.15.98:3300/0 conn(0x7fa2c00bf9d0 0x7fa2c00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.380+0000 7fa2bdd9b640 1 --2- 172.21.15.1:0/3356709268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c0152650 0x7fa2c0155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.380+0000 7fa2bdd9b640 1 -- 172.21.15.1:0/3356709268 >> v2:172.21.15.98:3300/0 conn(0x7fa2c00bf9d0 msgr2=0x7fa2c00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:13.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.380+0000 7fa2bdd9b640 1 --2- 172.21.15.1:0/3356709268 >> v2:172.21.15.98:3300/0 conn(0x7fa2c00bf9d0 0x7fa2c00c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.380+0000 7fa2bdd9b640 1 -- 172.21.15.1:0/3356709268 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c00c7280 con 0x7fa2c00c6b10 2024-03-06T07:32:13.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.381+0000 7fa2bed9d640 1 --2- 172.21.15.1:0/3356709268 >> v2:172.21.15.98:3300/0 conn(0x7fa2c00bf9d0 0x7fa2c00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:13.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.381+0000 7fa2bdd9b640 1 --2- 172.21.15.1:0/3356709268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c00c6b10 0x7fa2c0158690 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fa2c01597b0 tx=0x7fa2b402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eb6d8fc91e5a84c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.381+0000 7fa2bd59a640 1 -- 172.21.15.1:0/3356709268 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b402e070 con 0x7fa2c00c6b10 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.381+0000 7fa2bd59a640 1 -- 172.21.15.1:0/3356709268 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b4037070 con 0x7fa2c00c6b10 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.381+0000 7fa2bd59a640 1 -- 172.21.15.1:0/3356709268 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4031a00 con 0x7fa2c00c6b10 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.382+0000 7fa2c48df640 1 -- 172.21.15.1:0/3356709268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c00c6b10 msgr2=0x7fa2c0158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.382+0000 7fa2c48df640 1 --2- 172.21.15.1:0/3356709268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c00c6b10 0x7fa2c0158690 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fa2c01597b0 tx=0x7fa2b402b880 comp rx=0 tx=0).stop 2024-03-06T07:32:13.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.382+0000 7fa2c48df640 1 -- 172.21.15.1:0/3356709268 shutdown_connections 2024-03-06T07:32:13.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.382+0000 7fa2c48df640 1 --2- 172.21.15.1:0/3356709268 >> v2:172.21.15.98:3300/0 conn(0x7fa2c00bf9d0 0x7fa2c00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.382+0000 7fa2c48df640 1 --2- 172.21.15.1:0/3356709268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c00c6b10 0x7fa2c0158690 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.382+0000 7fa2c48df640 1 --2- 172.21.15.1:0/3356709268 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c0152650 0x7fa2c0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.382+0000 7fa2c48df640 1 -- 172.21.15.1:0/3356709268 >> 172.21.15.1:0/3356709268 conn(0x7fa2c0066ee0 msgr2=0x7fa2c00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.382+0000 7fa2c48df640 1 -- 172.21.15.1:0/3356709268 shutdown_connections 2024-03-06T07:32:13.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.382+0000 7fa2c48df640 1 -- 172.21.15.1:0/3356709268 wait complete. 2024-03-06T07:32:13.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.383+0000 7fa2c48df640 1 Processor -- start 2024-03-06T07:32:13.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.383+0000 7fa2c48df640 1 -- start start 2024-03-06T07:32:13.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.384+0000 7fa2c48df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c00bf9d0 0x7fa2c01ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.384+0000 7fa2c48df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c0152650 0x7fa2c0153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.384+0000 7fa2c48df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c0154d00 0x7fa2c0153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.384+0000 7fa2c48df640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c0155490 con 0x7fa2c0152650 2024-03-06T07:32:13.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.384+0000 7fa2c48df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c01551b0 con 0x7fa2c00bf9d0 2024-03-06T07:32:13.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.384+0000 7fa2c48df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c0152a30 con 0x7fa2c0154d00 2024-03-06T07:32:13.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.384+0000 7fa2bdd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c0152650 0x7fa2c0153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.384+0000 7fa2bdd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c0152650 0x7fa2c0153430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33024/0 (socket says 172.21.15.1:33024) 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.384+0000 7fa2bdd9b640 1 -- 172.21.15.1:0/4021182425 learned_addr learned my addr 172.21.15.1:0/4021182425 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.384+0000 7fa2be59c640 1 --2- 172.21.15.1:0/4021182425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c00bf9d0 0x7fa2c01ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.384+0000 7fa2bed9d640 1 --2- 172.21.15.1:0/4021182425 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c0154d00 0x7fa2c0153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.384+0000 7fa2bdd9b640 1 -- 172.21.15.1:0/4021182425 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c0154d00 msgr2=0x7fa2c0153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.384+0000 7fa2bdd9b640 1 --2- 172.21.15.1:0/4021182425 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c0154d00 0x7fa2c0153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.384+0000 7fa2bdd9b640 1 -- 172.21.15.1:0/4021182425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c00bf9d0 msgr2=0x7fa2c01ee5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.384+0000 7fa2bdd9b640 1 --2- 172.21.15.1:0/4021182425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c00bf9d0 0x7fa2c01ee5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.384+0000 7fa2bdd9b640 1 -- 172.21.15.1:0/4021182425 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c0155fc0 con 0x7fa2c0152650 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.384+0000 7fa2bed9d640 1 --2- 172.21.15.1:0/4021182425 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c0154d00 0x7fa2c0153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.385+0000 7fa2bdd9b640 1 --2- 172.21.15.1:0/4021182425 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c0152650 0x7fa2c0153430 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4000c00 tx=0x7fa2b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.385+0000 7fa2aeffd640 1 -- 172.21.15.1:0/4021182425 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4032880 con 0x7fa2c0152650 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.385+0000 7fa2aeffd640 1 -- 172.21.15.1:0/4021182425 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b4032a20 con 0x7fa2c0152650 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.385+0000 7fa2c48df640 1 -- 172.21.15.1:0/4021182425 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c01eefe0 con 0x7fa2c0152650 2024-03-06T07:32:13.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.385+0000 7fa2aeffd640 1 -- 172.21.15.1:0/4021182425 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4032e10 con 0x7fa2c0152650 2024-03-06T07:32:13.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.385+0000 7fa2c48df640 1 -- 172.21.15.1:0/4021182425 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c01df730 con 0x7fa2c0152650 2024-03-06T07:32:13.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.386+0000 7fa2c48df640 1 -- 172.21.15.1:0/4021182425 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c0067fd0 con 0x7fa2c0152650 2024-03-06T07:32:13.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.386+0000 7fa2aeffd640 1 -- 172.21.15.1:0/4021182425 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2b404c060 con 0x7fa2c0152650 2024-03-06T07:32:13.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.387+0000 7fa2aeffd640 1 --2- 172.21.15.1:0/4021182425 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2a006b4d0 0x7fa2a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.387+0000 7fa2aeffd640 1 -- 172.21.15.1:0/4021182425 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa2b40abfa0 con 0x7fa2c0152650 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.387+0000 7fa2be59c640 1 --2- 172.21.15.1:0/4021182425 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2a006b4d0 0x7fa2a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.388+0000 7fa2be59c640 1 --2- 172.21.15.1:0/4021182425 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2a006b4d0 0x7fa2a006d990 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0154710 tx=0x7fa2b8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.391+0000 7fa2aeffd640 1 -- 172.21.15.1:0/4021182425 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2b4040070 con 0x7fa2c0152650 2024-03-06T07:32:13.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.613+0000 7fa2c48df640 1 -- 172.21.15.1:0/4021182425 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2c00b06f0 con 0x7fa2c0152650 2024-03-06T07:32:13.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.614+0000 7fa2aeffd640 1 -- 172.21.15.1:0/4021182425 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fa2b407ac20 con 0x7fa2c0152650 2024-03-06T07:32:13.616 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:13.616 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:13.616 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:13.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.616+0000 7fa2c48df640 1 -- 172.21.15.1:0/4021182425 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2a006b4d0 msgr2=0x7fa2a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.616+0000 7fa2c48df640 1 --2- 172.21.15.1:0/4021182425 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2a006b4d0 0x7fa2a006d990 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0154710 tx=0x7fa2b8008040 comp rx=0 tx=0).stop 2024-03-06T07:32:13.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.616+0000 7fa2c48df640 1 -- 172.21.15.1:0/4021182425 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c0152650 msgr2=0x7fa2c0153430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.616+0000 7fa2c48df640 1 --2- 172.21.15.1:0/4021182425 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c0152650 0x7fa2c0153430 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4000c00 tx=0x7fa2b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.616+0000 7fa2c48df640 1 -- 172.21.15.1:0/4021182425 shutdown_connections 2024-03-06T07:32:13.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.616+0000 7fa2c48df640 1 --2- 172.21.15.1:0/4021182425 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa2a006b4d0 0x7fa2a006d990 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.616+0000 7fa2c48df640 1 --2- 172.21.15.1:0/4021182425 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2c0154d00 0x7fa2c0153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.616+0000 7fa2c48df640 1 --2- 172.21.15.1:0/4021182425 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c0152650 0x7fa2c0153430 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.616+0000 7fa2c48df640 1 --2- 172.21.15.1:0/4021182425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c00bf9d0 0x7fa2c01ee5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.616+0000 7fa2c48df640 1 -- 172.21.15.1:0/4021182425 >> 172.21.15.1:0/4021182425 conn(0x7fa2c0066ee0 msgr2=0x7fa2c00cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.616+0000 7fa2c48df640 1 -- 172.21.15.1:0/4021182425 shutdown_connections 2024-03-06T07:32:13.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:13.616+0000 7fa2c48df640 1 -- 172.21.15.1:0/4021182425 wait complete. 2024-03-06T07:32:14.031 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:14.031 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:15.032 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:15.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.132+0000 7f2086519640 1 Processor -- start 2024-03-06T07:32:15.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.132+0000 7f2086519640 1 -- start start 2024-03-06T07:32:15.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.133+0000 7f2086519640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2080152650 0x7f2080155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.133+0000 7f2086519640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20800bf9d0 0x7f20800c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.133+0000 7f2086519640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20800c2fc0 con 0x7f20800c6b10 2024-03-06T07:32:15.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.133+0000 7f2086519640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2080155800 con 0x7f20800bf9d0 2024-03-06T07:32:15.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.133+0000 7f2086519640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2080158da0 con 0x7f2080152650 2024-03-06T07:32:15.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.134+0000 7f2084ab6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20800bf9d0 0x7f20800c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.134+0000 7f207ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2080152650 0x7f2080155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.134+0000 7f207f7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f20800c6b10 0x7f2080158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:36718/0 (socket says 172.21.15.1:36718) 2024-03-06T07:32:15.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.134+0000 7f2084ab6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20800bf9d0 0x7f20800c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:49632/0 (socket says 172.21.15.1:49632) 2024-03-06T07:32:15.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.134+0000 7f207ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2080152650 0x7f2080155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:40090/0 (socket says 172.21.15.1:40090) 2024-03-06T07:32:15.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.134+0000 7f2084ab6640 1 -- 172.21.15.1:0/2996263819 learned_addr learned my addr 172.21.15.1:0/2996263819 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:15.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.134+0000 7f207ffff640 1 -- 172.21.15.1:0/2996263819 >> v2:172.21.15.98:3300/0 conn(0x7f20800bf9d0 msgr2=0x7f20800c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.134+0000 7f207ffff640 1 --2- 172.21.15.1:0/2996263819 >> v2:172.21.15.98:3300/0 conn(0x7f20800bf9d0 0x7f20800c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.134+0000 7f207ffff640 1 -- 172.21.15.1:0/2996263819 >> v1:172.21.15.98:6789/0 conn(0x7f20800c6b10 legacy=0x7f2080158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.134+0000 7f207ffff640 1 -- 172.21.15.1:0/2996263819 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20800c6690 con 0x7f2080152650 2024-03-06T07:32:15.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.134+0000 7f2084ab6640 1 --2- 172.21.15.1:0/2996263819 >> v2:172.21.15.98:3300/0 conn(0x7f20800bf9d0 0x7f20800c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:15.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.135+0000 7f207ffff640 1 --2- 172.21.15.1:0/2996263819 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2080152650 0x7f2080155c90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f20700097a0 tx=0x7f207002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ee76a65da596206 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.135+0000 7f207effd640 1 -- 172.21.15.1:0/2996263819 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20700315a0 con 0x7f2080152650 2024-03-06T07:32:15.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.135+0000 7f207effd640 1 -- 172.21.15.1:0/2996263819 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2070037070 con 0x7f2080152650 2024-03-06T07:32:15.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.135+0000 7f207effd640 1 -- 172.21.15.1:0/2996263819 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2070031960 con 0x7f2080152650 2024-03-06T07:32:15.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.136+0000 7f2086519640 1 -- 172.21.15.1:0/2996263819 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2080152650 msgr2=0x7f2080155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.136+0000 7f2086519640 1 --2- 172.21.15.1:0/2996263819 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2080152650 0x7f2080155c90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f20700097a0 tx=0x7f207002b840 comp rx=0 tx=0).stop 2024-03-06T07:32:15.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.136+0000 7f2086519640 1 -- 172.21.15.1:0/2996263819 shutdown_connections 2024-03-06T07:32:15.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.136+0000 7f2086519640 1 --2- 172.21.15.1:0/2996263819 >> v2:172.21.15.98:3300/0 conn(0x7f20800bf9d0 0x7f20800c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.136+0000 7f2086519640 1 --2- 172.21.15.1:0/2996263819 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2080152650 0x7f2080155c90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.136+0000 7f2086519640 1 -- 172.21.15.1:0/2996263819 >> 172.21.15.1:0/2996263819 conn(0x7f2080066ee0 msgr2=0x7f20800672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.136+0000 7f2086519640 1 -- 172.21.15.1:0/2996263819 shutdown_connections 2024-03-06T07:32:15.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.136+0000 7f2086519640 1 -- 172.21.15.1:0/2996263819 wait complete. 2024-03-06T07:32:15.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.137+0000 7f2086519640 1 Processor -- start 2024-03-06T07:32:15.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.137+0000 7f2086519640 1 -- start start 2024-03-06T07:32:15.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.137+0000 7f2086519640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20800bf9d0 0x7f2080158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.137+0000 7f2086519640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2080152650 0x7f20800c5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.137+0000 7f2086519640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20801eab90 0x7f20801ee780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f2086519640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20801551b0 con 0x7f20801eab90 2024-03-06T07:32:15.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f2086519640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20800c6fa0 con 0x7f20800bf9d0 2024-03-06T07:32:15.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f2086519640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2080155490 con 0x7f2080152650 2024-03-06T07:32:15.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f2084ab6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20801eab90 0x7f20801ee780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f2084ab6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20801eab90 0x7f20801ee780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35976/0 (socket says 172.21.15.1:35976) 2024-03-06T07:32:15.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f2084ab6640 1 -- 172.21.15.1:0/2297806983 learned_addr learned my addr 172.21.15.1:0/2297806983 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:15.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f207f7fe640 1 --2- 172.21.15.1:0/2297806983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2080152650 0x7f20800c5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f207ffff640 1 --2- 172.21.15.1:0/2297806983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20800bf9d0 0x7f2080158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f2084ab6640 1 -- 172.21.15.1:0/2297806983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2080152650 msgr2=0x7f20800c5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f2084ab6640 1 --2- 172.21.15.1:0/2297806983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2080152650 0x7f20800c5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f2084ab6640 1 -- 172.21.15.1:0/2297806983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20800bf9d0 msgr2=0x7f2080158680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f2084ab6640 1 --2- 172.21.15.1:0/2297806983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20800bf9d0 0x7f2080158680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f2084ab6640 1 -- 172.21.15.1:0/2297806983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20801ee2a0 con 0x7f20801eab90 2024-03-06T07:32:15.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f207f7fe640 1 --2- 172.21.15.1:0/2297806983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2080152650 0x7f20800c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:15.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f2084ab6640 1 --2- 172.21.15.1:0/2297806983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20801eab90 0x7f20801ee780 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f207800efc0 tx=0x7f207800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f207cff9640 1 -- 172.21.15.1:0/2297806983 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207800cbe0 con 0x7f20801eab90 2024-03-06T07:32:15.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f207cff9640 1 -- 172.21.15.1:0/2297806983 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207800cd80 con 0x7f20801eab90 2024-03-06T07:32:15.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f2086519640 1 -- 172.21.15.1:0/2297806983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2080154a70 con 0x7f20801eab90 2024-03-06T07:32:15.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f207cff9640 1 -- 172.21.15.1:0/2297806983 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2078003bb0 con 0x7f20801eab90 2024-03-06T07:32:15.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.138+0000 7f2086519640 1 -- 172.21.15.1:0/2297806983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20800c60d0 con 0x7f20801eab90 2024-03-06T07:32:15.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.140+0000 7f2086519640 1 -- 172.21.15.1:0/2297806983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208006cf30 con 0x7f20801eab90 2024-03-06T07:32:15.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.140+0000 7f207cff9640 1 -- 172.21.15.1:0/2297806983 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f207801b030 con 0x7f20801eab90 2024-03-06T07:32:15.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.140+0000 7f207cff9640 1 --2- 172.21.15.1:0/2297806983 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f206c06b500 0x7f206c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.141+0000 7f207cff9640 1 -- 172.21.15.1:0/2297806983 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f207808a800 con 0x7f20801eab90 2024-03-06T07:32:15.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.141+0000 7f207ffff640 1 --2- 172.21.15.1:0/2297806983 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f206c06b500 0x7f206c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.141+0000 7f207ffff640 1 --2- 172.21.15.1:0/2297806983 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f206c06b500 0x7f206c06d9c0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f2070036000 tx=0x7f207003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.145+0000 7f207cff9640 1 -- 172.21.15.1:0/2297806983 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f208006cf30 con 0x7f20801eab90 2024-03-06T07:32:15.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.367+0000 7f2086519640 1 -- 172.21.15.1:0/2297806983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20801ee470 con 0x7f20801eab90 2024-03-06T07:32:15.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.368+0000 7f207cff9640 1 -- 172.21.15.1:0/2297806983 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f2078022020 con 0x7f20801eab90 2024-03-06T07:32:15.370 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:15.370 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:15.370 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:15.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.370+0000 7f2086519640 1 -- 172.21.15.1:0/2297806983 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f206c06b500 msgr2=0x7f206c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.370+0000 7f2086519640 1 --2- 172.21.15.1:0/2297806983 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f206c06b500 0x7f206c06d9c0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f2070036000 tx=0x7f207003d040 comp rx=0 tx=0).stop 2024-03-06T07:32:15.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.370+0000 7f2086519640 1 -- 172.21.15.1:0/2297806983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20801eab90 msgr2=0x7f20801ee780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.370+0000 7f2086519640 1 --2- 172.21.15.1:0/2297806983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20801eab90 0x7f20801ee780 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f207800efc0 tx=0x7f207800c490 comp rx=0 tx=0).stop 2024-03-06T07:32:15.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.370+0000 7f2086519640 1 -- 172.21.15.1:0/2297806983 shutdown_connections 2024-03-06T07:32:15.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.370+0000 7f2086519640 1 --2- 172.21.15.1:0/2297806983 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f206c06b500 0x7f206c06d9c0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.370+0000 7f2086519640 1 --2- 172.21.15.1:0/2297806983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20801eab90 0x7f20801ee780 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.370+0000 7f2086519640 1 --2- 172.21.15.1:0/2297806983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2080152650 0x7f20800c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.370+0000 7f2086519640 1 --2- 172.21.15.1:0/2297806983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20800bf9d0 0x7f2080158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.370+0000 7f2086519640 1 -- 172.21.15.1:0/2297806983 >> 172.21.15.1:0/2297806983 conn(0x7f2080066ee0 msgr2=0x7f20800bff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.370+0000 7f2086519640 1 -- 172.21.15.1:0/2297806983 shutdown_connections 2024-03-06T07:32:15.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:15.370+0000 7f2086519640 1 -- 172.21.15.1:0/2297806983 wait complete. 2024-03-06T07:32:15.785 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:15.785 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:16.786 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:16.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.886+0000 7f442038a640 1 Processor -- start 2024-03-06T07:32:16.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.886+0000 7f442038a640 1 -- start start 2024-03-06T07:32:16.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.887+0000 7f442038a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4418152820 0x7f44180c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.887+0000 7f442038a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44180c7180 con 0x7f44180af6f0 2024-03-06T07:32:16.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.887+0000 7f442038a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4418155460 con 0x7f44180c0670 2024-03-06T07:32:16.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.887+0000 7f442038a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44180c6bc0 con 0x7f4418152820 2024-03-06T07:32:16.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.887+0000 7f441e126640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4418152820 0x7f44180c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.887+0000 7f441e927640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f44180af6f0 0x7f441815fb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53648/0 (socket says 172.21.15.1:53648) 2024-03-06T07:32:16.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.887+0000 7f441e927640 1 -- 172.21.15.1:0/1113022139 learned_addr learned my addr 172.21.15.1:0/1113022139 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:16.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.887+0000 7f441e126640 1 -- 172.21.15.1:0/1113022139 >> v1:172.21.15.98:6789/0 conn(0x7f44180c0670 legacy=0x7f44180c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.887+0000 7f441e126640 1 -- 172.21.15.1:0/1113022139 >> v1:172.21.15.1:6789/0 conn(0x7f44180af6f0 legacy=0x7f441815fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.887+0000 7f441e126640 1 -- 172.21.15.1:0/1113022139 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44180c6ea0 con 0x7f4418152820 2024-03-06T07:32:16.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.887+0000 7f441d124640 1 -- 172.21.15.1:0/1113022139 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544448796 0 0) 0x7f44180c7180 con 0x7f44180af6f0 2024-03-06T07:32:16.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.888+0000 7f441e126640 1 --2- 172.21.15.1:0/1113022139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4418152820 0x7f44180c0130 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f440c0097a0 tx=0x7f440c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4bb6fafa858e466 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.888+0000 7f441d124640 1 -- 172.21.15.1:0/1113022139 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c034070 con 0x7f4418152820 2024-03-06T07:32:16.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.888+0000 7f441d124640 1 -- 172.21.15.1:0/1113022139 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440c030610 con 0x7f4418152820 2024-03-06T07:32:16.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.888+0000 7f441d124640 1 -- 172.21.15.1:0/1113022139 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c0309e0 con 0x7f4418152820 2024-03-06T07:32:16.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.889+0000 7f442038a640 1 -- 172.21.15.1:0/1113022139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4418152820 msgr2=0x7f44180c0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.889+0000 7f442038a640 1 --2- 172.21.15.1:0/1113022139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4418152820 0x7f44180c0130 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f440c0097a0 tx=0x7f440c02b960 comp rx=0 tx=0).stop 2024-03-06T07:32:16.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.889+0000 7f442038a640 1 -- 172.21.15.1:0/1113022139 shutdown_connections 2024-03-06T07:32:16.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.889+0000 7f442038a640 1 --2- 172.21.15.1:0/1113022139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4418152820 0x7f44180c0130 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.889+0000 7f442038a640 1 -- 172.21.15.1:0/1113022139 >> 172.21.15.1:0/1113022139 conn(0x7f4418072720 msgr2=0x7f4418072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.889+0000 7f442038a640 1 -- 172.21.15.1:0/1113022139 shutdown_connections 2024-03-06T07:32:16.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.889+0000 7f442038a640 1 -- 172.21.15.1:0/1113022139 wait complete. 2024-03-06T07:32:16.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.890+0000 7f442038a640 1 Processor -- start 2024-03-06T07:32:16.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.890+0000 7f442038a640 1 -- start start 2024-03-06T07:32:16.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.891+0000 7f442038a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44180af6f0 0x7f4418153690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.891+0000 7f442038a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44180c0670 0x7f4418156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.891+0000 7f442038a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4418152820 0x7f44180c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.891+0000 7f442038a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44180c7c80 con 0x7f44180c0670 2024-03-06T07:32:16.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.891+0000 7f442038a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44180c7b00 con 0x7f44180af6f0 2024-03-06T07:32:16.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.891+0000 7f442038a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4418152c00 con 0x7f4418152820 2024-03-06T07:32:16.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.891+0000 7f441d925640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44180c0670 0x7f4418156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.891+0000 7f441d925640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44180c0670 0x7f4418156c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35994/0 (socket says 172.21.15.1:35994) 2024-03-06T07:32:16.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.891+0000 7f441d925640 1 -- 172.21.15.1:0/367806929 learned_addr learned my addr 172.21.15.1:0/367806929 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:16.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.891+0000 7f441e927640 1 --2- 172.21.15.1:0/367806929 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4418152820 0x7f44180c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.891+0000 7f441e126640 1 --2- 172.21.15.1:0/367806929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44180af6f0 0x7f4418153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.891+0000 7f441d925640 1 -- 172.21.15.1:0/367806929 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4418152820 msgr2=0x7f44180c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.891+0000 7f441d925640 1 --2- 172.21.15.1:0/367806929 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4418152820 0x7f44180c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.891+0000 7f441d925640 1 -- 172.21.15.1:0/367806929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44180af6f0 msgr2=0x7f4418153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.891+0000 7f441d925640 1 --2- 172.21.15.1:0/367806929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44180af6f0 0x7f4418153690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.891+0000 7f441d925640 1 -- 172.21.15.1:0/367806929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44181582f0 con 0x7f44180c0670 2024-03-06T07:32:16.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.891+0000 7f441d925640 1 --2- 172.21.15.1:0/367806929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44180c0670 0x7f4418156c50 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f441400c400 tx=0x7f441400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.892+0000 7f440affd640 1 -- 172.21.15.1:0/367806929 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4414017070 con 0x7f44180c0670 2024-03-06T07:32:16.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.892+0000 7f440affd640 1 -- 172.21.15.1:0/367806929 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44140052d0 con 0x7f44180c0670 2024-03-06T07:32:16.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.892+0000 7f442038a640 1 -- 172.21.15.1:0/367806929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44181561b0 con 0x7f44180c0670 2024-03-06T07:32:16.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.892+0000 7f440affd640 1 -- 172.21.15.1:0/367806929 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4414003830 con 0x7f44180c0670 2024-03-06T07:32:16.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.892+0000 7f442038a640 1 -- 172.21.15.1:0/367806929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4418156450 con 0x7f44180c0670 2024-03-06T07:32:16.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.893+0000 7f442038a640 1 -- 172.21.15.1:0/367806929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44180686e0 con 0x7f44180c0670 2024-03-06T07:32:16.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.893+0000 7f440affd640 1 -- 172.21.15.1:0/367806929 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f441401a070 con 0x7f44180c0670 2024-03-06T07:32:16.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.893+0000 7f440affd640 1 --2- 172.21.15.1:0/367806929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43fc06b500 0x7f43fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.893+0000 7f440affd640 1 -- 172.21.15.1:0/367806929 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f44140897a0 con 0x7f44180c0670 2024-03-06T07:32:16.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.894+0000 7f441e126640 1 --2- 172.21.15.1:0/367806929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43fc06b500 0x7f43fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.894+0000 7f441e126640 1 --2- 172.21.15.1:0/367806929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43fc06b500 0x7f43fc06d9c0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f440c036000 tx=0x7f440c03b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:16.898+0000 7f440affd640 1 -- 172.21.15.1:0/367806929 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4414058420 con 0x7f44180c0670 2024-03-06T07:32:17.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:17.119+0000 7f442038a640 1 -- 172.21.15.1:0/367806929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4418153bd0 con 0x7f44180c0670 2024-03-06T07:32:17.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:17.119+0000 7f440affd640 1 -- 172.21.15.1:0/367806929 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f4414057d20 con 0x7f44180c0670 2024-03-06T07:32:17.121 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:17.121 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:17.122 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:17.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:17.122+0000 7f442038a640 1 -- 172.21.15.1:0/367806929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43fc06b500 msgr2=0x7f43fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:17.122+0000 7f442038a640 1 --2- 172.21.15.1:0/367806929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43fc06b500 0x7f43fc06d9c0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f440c036000 tx=0x7f440c03b040 comp rx=0 tx=0).stop 2024-03-06T07:32:17.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:17.122+0000 7f442038a640 1 -- 172.21.15.1:0/367806929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44180c0670 msgr2=0x7f4418156c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:17.122+0000 7f442038a640 1 --2- 172.21.15.1:0/367806929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44180c0670 0x7f4418156c50 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f441400c400 tx=0x7f441400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:17.122+0000 7f442038a640 1 -- 172.21.15.1:0/367806929 shutdown_connections 2024-03-06T07:32:17.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:17.122+0000 7f442038a640 1 --2- 172.21.15.1:0/367806929 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f43fc06b500 0x7f43fc06d9c0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:17.122+0000 7f442038a640 1 --2- 172.21.15.1:0/367806929 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4418152820 0x7f44180c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:17.122+0000 7f442038a640 1 --2- 172.21.15.1:0/367806929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44180c0670 0x7f4418156c50 unknown :-1 s=CLOSED pgs=2581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:17.122+0000 7f442038a640 1 --2- 172.21.15.1:0/367806929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44180af6f0 0x7f4418153690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:17.122+0000 7f442038a640 1 -- 172.21.15.1:0/367806929 >> 172.21.15.1:0/367806929 conn(0x7f4418072720 msgr2=0x7f4418151a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:17.122+0000 7f442038a640 1 -- 172.21.15.1:0/367806929 shutdown_connections 2024-03-06T07:32:17.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:17.122+0000 7f442038a640 1 -- 172.21.15.1:0/367806929 wait complete. 2024-03-06T07:32:17.538 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:17.538 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:18.539 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:18.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.639+0000 7f1495b3a640 1 Processor -- start 2024-03-06T07:32:18.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.639+0000 7f1495b3a640 1 -- start start 2024-03-06T07:32:18.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.639+0000 7f1495b3a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1490152840 0x7f14900c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.639+0000 7f1495b3a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f14900c60b0 0x7f14901553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.639+0000 7f1495b3a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14900c5e40 con 0x7f14900c3970 2024-03-06T07:32:18.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.639+0000 7f1495b3a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14900c67f0 con 0x7f1490152840 2024-03-06T07:32:18.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.639+0000 7f1495b3a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14900c6a40 con 0x7f14900c60b0 2024-03-06T07:32:18.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.639+0000 7f148effd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1490152840 0x7f14900c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.640+0000 7f148f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f14900c3970 0x7f1490152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53660/0 (socket says 172.21.15.1:53660) 2024-03-06T07:32:18.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.640+0000 7f148f7fe640 1 -- 172.21.15.1:0/2638909447 learned_addr learned my addr 172.21.15.1:0/2638909447 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:18.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.640+0000 7f148ffff640 1 --2- 172.21.15.1:0/2638909447 >> v2:172.21.15.98:3300/0 conn(0x7f14900c60b0 0x7f14901553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.640+0000 7f148effd640 1 -- 172.21.15.1:0/2638909447 >> v2:172.21.15.98:3300/0 conn(0x7f14900c60b0 msgr2=0x7f14901553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:18.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.640+0000 7f148effd640 1 --2- 172.21.15.1:0/2638909447 >> v2:172.21.15.98:3300/0 conn(0x7f14900c60b0 0x7f14901553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.640+0000 7f148effd640 1 -- 172.21.15.1:0/2638909447 >> v1:172.21.15.1:6789/0 conn(0x7f14900c3970 legacy=0x7f1490152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.640+0000 7f148e7fc640 1 -- 172.21.15.1:0/2638909447 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250399097 0 0) 0x7f14900c5e40 con 0x7f14900c3970 2024-03-06T07:32:18.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.640+0000 7f148effd640 1 -- 172.21.15.1:0/2638909447 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14900c6d20 con 0x7f1490152840 2024-03-06T07:32:18.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.640+0000 7f148ffff640 1 --2- 172.21.15.1:0/2638909447 >> v2:172.21.15.98:3300/0 conn(0x7f14900c60b0 0x7f14901553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:18.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.641+0000 7f148effd640 1 --2- 172.21.15.1:0/2638909447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1490152840 0x7f14900c8170 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f14840097a0 tx=0x7f148402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc457a39f9aaff4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.641+0000 7f148e7fc640 1 -- 172.21.15.1:0/2638909447 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1484035020 con 0x7f1490152840 2024-03-06T07:32:18.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.641+0000 7f148e7fc640 1 -- 172.21.15.1:0/2638909447 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1484031830 con 0x7f1490152840 2024-03-06T07:32:18.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.641+0000 7f148e7fc640 1 -- 172.21.15.1:0/2638909447 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1484031c00 con 0x7f1490152840 2024-03-06T07:32:18.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.641+0000 7f1495b3a640 1 -- 172.21.15.1:0/2638909447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1490152840 msgr2=0x7f14900c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.641+0000 7f1495b3a640 1 --2- 172.21.15.1:0/2638909447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1490152840 0x7f14900c8170 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f14840097a0 tx=0x7f148402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.642+0000 7f1495b3a640 1 -- 172.21.15.1:0/2638909447 shutdown_connections 2024-03-06T07:32:18.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.642+0000 7f1495b3a640 1 --2- 172.21.15.1:0/2638909447 >> v2:172.21.15.98:3300/0 conn(0x7f14900c60b0 0x7f14901553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.642+0000 7f1495b3a640 1 --2- 172.21.15.1:0/2638909447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1490152840 0x7f14900c8170 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.642+0000 7f1495b3a640 1 -- 172.21.15.1:0/2638909447 >> 172.21.15.1:0/2638909447 conn(0x7f14900c88f0 msgr2=0x7f14900c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:18.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.642+0000 7f1495b3a640 1 -- 172.21.15.1:0/2638909447 shutdown_connections 2024-03-06T07:32:18.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.642+0000 7f1495b3a640 1 -- 172.21.15.1:0/2638909447 wait complete. 2024-03-06T07:32:18.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.643+0000 7f1495b3a640 1 Processor -- start 2024-03-06T07:32:18.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.643+0000 7f1495b3a640 1 -- start start 2024-03-06T07:32:18.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.643+0000 7f1495b3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14900c3970 0x7f14900a9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.643+0000 7f1495b3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14900c60b0 0x7f14901eaf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.643+0000 7f1495b3a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1490152840 0x7f14901eb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.643+0000 7f1495b3a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14900c75c0 con 0x7f1490152840 2024-03-06T07:32:18.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.643+0000 7f1495b3a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14900c72e0 con 0x7f14900c3970 2024-03-06T07:32:18.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.643+0000 7f1495b3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14900c3d50 con 0x7f14900c60b0 2024-03-06T07:32:18.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.644+0000 7f148ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1490152840 0x7f14901eb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.644+0000 7f148ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1490152840 0x7f14901eb460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36012/0 (socket says 172.21.15.1:36012) 2024-03-06T07:32:18.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.644+0000 7f148ffff640 1 -- 172.21.15.1:0/1834523994 learned_addr learned my addr 172.21.15.1:0/1834523994 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:18.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.644+0000 7f148effd640 1 --2- 172.21.15.1:0/1834523994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14900c60b0 0x7f14901eaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.644+0000 7f148f7fe640 1 --2- 172.21.15.1:0/1834523994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14900c3970 0x7f14900a9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.644+0000 7f148ffff640 1 -- 172.21.15.1:0/1834523994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14900c60b0 msgr2=0x7f14901eaf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.644+0000 7f148ffff640 1 --2- 172.21.15.1:0/1834523994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14900c60b0 0x7f14901eaf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.644+0000 7f148ffff640 1 -- 172.21.15.1:0/1834523994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14900c3970 msgr2=0x7f14900a9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.644+0000 7f148ffff640 1 --2- 172.21.15.1:0/1834523994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14900c3970 0x7f14900a9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.644+0000 7f148ffff640 1 -- 172.21.15.1:0/1834523994 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1490157e10 con 0x7f1490152840 2024-03-06T07:32:18.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.644+0000 7f148ffff640 1 --2- 172.21.15.1:0/1834523994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1490152840 0x7f14901eb460 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f148800ce80 tx=0x7f148800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.644+0000 7f1477fff640 1 -- 172.21.15.1:0/1834523994 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148801a070 con 0x7f1490152840 2024-03-06T07:32:18.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.644+0000 7f1477fff640 1 -- 172.21.15.1:0/1834523994 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148800d880 con 0x7f1490152840 2024-03-06T07:32:18.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.644+0000 7f1495b3a640 1 -- 172.21.15.1:0/1834523994 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14900abbf0 con 0x7f1490152840 2024-03-06T07:32:18.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.644+0000 7f1477fff640 1 -- 172.21.15.1:0/1834523994 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148800dc70 con 0x7f1490152840 2024-03-06T07:32:18.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.644+0000 7f1495b3a640 1 -- 172.21.15.1:0/1834523994 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1490070580 con 0x7f1490152840 2024-03-06T07:32:18.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.645+0000 7f1495b3a640 1 -- 172.21.15.1:0/1834523994 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14900686e0 con 0x7f1490152840 2024-03-06T07:32:18.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.646+0000 7f1477fff640 1 -- 172.21.15.1:0/1834523994 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f148801d070 con 0x7f1490152840 2024-03-06T07:32:18.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.647+0000 7f1477fff640 1 --2- 172.21.15.1:0/1834523994 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f147c06b4d0 0x7f147c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.647+0000 7f1477fff640 1 -- 172.21.15.1:0/1834523994 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f148808ab10 con 0x7f1490152840 2024-03-06T07:32:18.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.647+0000 7f148f7fe640 1 --2- 172.21.15.1:0/1834523994 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f147c06b4d0 0x7f147c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.648+0000 7f148f7fe640 1 --2- 172.21.15.1:0/1834523994 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f147c06b4d0 0x7f147c06d990 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f1480000f60 tx=0x7f1480007b50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.651+0000 7f1477fff640 1 -- 172.21.15.1:0/1834523994 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1488059140 con 0x7f1490152840 2024-03-06T07:32:18.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.879+0000 7f1495b3a640 1 -- 172.21.15.1:0/1834523994 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1490154ce0 con 0x7f1490152840 2024-03-06T07:32:18.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.879+0000 7f1477fff640 1 -- 172.21.15.1:0/1834523994 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f148805d1e0 con 0x7f1490152840 2024-03-06T07:32:18.881 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:18.881 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:18.882 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:18.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.881+0000 7f1495b3a640 1 -- 172.21.15.1:0/1834523994 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f147c06b4d0 msgr2=0x7f147c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.881+0000 7f1495b3a640 1 --2- 172.21.15.1:0/1834523994 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f147c06b4d0 0x7f147c06d990 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f1480000f60 tx=0x7f1480007b50 comp rx=0 tx=0).stop 2024-03-06T07:32:18.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.881+0000 7f1495b3a640 1 -- 172.21.15.1:0/1834523994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1490152840 msgr2=0x7f14901eb460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.881+0000 7f1495b3a640 1 --2- 172.21.15.1:0/1834523994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1490152840 0x7f14901eb460 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f148800ce80 tx=0x7f148800a710 comp rx=0 tx=0).stop 2024-03-06T07:32:18.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.881+0000 7f1495b3a640 1 -- 172.21.15.1:0/1834523994 shutdown_connections 2024-03-06T07:32:18.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.881+0000 7f1495b3a640 1 --2- 172.21.15.1:0/1834523994 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f147c06b4d0 0x7f147c06d990 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.881+0000 7f1495b3a640 1 --2- 172.21.15.1:0/1834523994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1490152840 0x7f14901eb460 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.881+0000 7f1495b3a640 1 --2- 172.21.15.1:0/1834523994 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14900c60b0 0x7f14901eaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.881+0000 7f1495b3a640 1 --2- 172.21.15.1:0/1834523994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14900c3970 0x7f14900a9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.881+0000 7f1495b3a640 1 -- 172.21.15.1:0/1834523994 >> 172.21.15.1:0/1834523994 conn(0x7f14900c88f0 msgr2=0x7f14900aa680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:18.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.881+0000 7f1495b3a640 1 -- 172.21.15.1:0/1834523994 shutdown_connections 2024-03-06T07:32:18.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:18.881+0000 7f1495b3a640 1 -- 172.21.15.1:0/1834523994 wait complete. 2024-03-06T07:32:19.296 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:19.296 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:20.297 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.396+0000 7fdbaaa9a640 1 Processor -- start 2024-03-06T07:32:20.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.396+0000 7fdbaaa9a640 1 -- start start 2024-03-06T07:32:20.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.397+0000 7fdbaaa9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba40bf9d0 0x7fdba4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.397+0000 7fdbaaa9a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba40c2fc0 con 0x7fdba40c6b10 2024-03-06T07:32:20.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.397+0000 7fdbaaa9a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba4158da0 con 0x7fdba4152650 2024-03-06T07:32:20.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.397+0000 7fdbaaa9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba4155800 con 0x7fdba40bf9d0 2024-03-06T07:32:20.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.397+0000 7fdba3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdba40c6b10 0x7fdba40c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53666/0 (socket says 172.21.15.1:53666) 2024-03-06T07:32:20.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.397+0000 7fdba3fff640 1 -- 172.21.15.1:0/4127450147 learned_addr learned my addr 172.21.15.1:0/4127450147 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:20.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba9037640 1 --2- 172.21.15.1:0/4127450147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba40bf9d0 0x7fdba4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba37fe640 1 -- 172.21.15.1:0/4127450147 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78130681 0 0) 0x7fdba40c2fc0 con 0x7fdba40c6b10 2024-03-06T07:32:20.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba37fe640 1 -- 172.21.15.1:0/4127450147 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb94003520 con 0x7fdba40c6b10 2024-03-06T07:32:20.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba37fe640 1 -- 172.21.15.1:0/4127450147 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2353609287 0 0) 0x7fdb94003520 con 0x7fdba40c6b10 2024-03-06T07:32:20.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba37fe640 1 -- 172.21.15.1:0/4127450147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba40bf9d0 msgr2=0x7fdba4155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba37fe640 1 --2- 172.21.15.1:0/4127450147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba40bf9d0 0x7fdba4155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba37fe640 1 -- 172.21.15.1:0/4127450147 >> v1:172.21.15.98:6789/0 conn(0x7fdba4152650 legacy=0x7fdba4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba37fe640 1 -- 172.21.15.1:0/4127450147 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba40c6690 con 0x7fdba40c6b10 2024-03-06T07:32:20.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba37fe640 1 -- 172.21.15.1:0/4127450147 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fdb90001690 con 0x7fdba40c6b10 2024-03-06T07:32:20.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba37fe640 1 -- 172.21.15.1:0/4127450147 >> v1:172.21.15.1:6789/0 conn(0x7fdba40c6b10 legacy=0x7fdba40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba37fe640 1 --2- 172.21.15.1:0/4127450147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba40bf9d0 0x7fdb940039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba37fe640 1 --2- 172.21.15.1:0/4127450147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb94003f20 0x7fdb94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba37fe640 1 --2- 172.21.15.1:0/4127450147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb94004d40 0x7fdb940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba37fe640 1 -- 172.21.15.1:0/4127450147 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba4155800 con 0x7fdb94003f20 2024-03-06T07:32:20.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba37fe640 1 -- 172.21.15.1:0/4127450147 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9400bad0 con 0x7fdb94004d40 2024-03-06T07:32:20.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba37fe640 1 -- 172.21.15.1:0/4127450147 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9400bcd0 con 0x7fdba40bf9d0 2024-03-06T07:32:20.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba3fff640 1 --2- 172.21.15.1:0/4127450147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb94003f20 0x7fdb94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba8836640 1 --2- 172.21.15.1:0/4127450147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba40bf9d0 0x7fdb940039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba9037640 1 --2- 172.21.15.1:0/4127450147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb94004d40 0x7fdb940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba3fff640 1 -- 172.21.15.1:0/4127450147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba40bf9d0 msgr2=0x7fdb940039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba3fff640 1 --2- 172.21.15.1:0/4127450147 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba40bf9d0 0x7fdb940039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba3fff640 1 -- 172.21.15.1:0/4127450147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb94004d40 msgr2=0x7fdb940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba3fff640 1 --2- 172.21.15.1:0/4127450147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb94004d40 0x7fdb940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.398+0000 7fdba3fff640 1 -- 172.21.15.1:0/4127450147 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb94003520 con 0x7fdb94003f20 2024-03-06T07:32:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.399+0000 7fdba3fff640 1 --2- 172.21.15.1:0/4127450147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb94003f20 0x7fdb94004800 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7fdb9000c210 tx=0x7fdb94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70042d0b8443a2f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.399+0000 7fdba37fe640 1 -- 172.21.15.1:0/4127450147 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb90007d30 con 0x7fdb94003f20 2024-03-06T07:32:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.399+0000 7fdba37fe640 1 -- 172.21.15.1:0/4127450147 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb90030070 con 0x7fdb94003f20 2024-03-06T07:32:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.399+0000 7fdba37fe640 1 -- 172.21.15.1:0/4127450147 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb90005d10 con 0x7fdb94003f20 2024-03-06T07:32:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.400+0000 7fdbaaa9a640 1 -- 172.21.15.1:0/4127450147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb94003f20 msgr2=0x7fdb94004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.400+0000 7fdbaaa9a640 1 --2- 172.21.15.1:0/4127450147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb94003f20 0x7fdb94004800 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7fdb9000c210 tx=0x7fdb94005830 comp rx=0 tx=0).stop 2024-03-06T07:32:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.400+0000 7fdba9037640 1 -- 172.21.15.1:0/4127450147 reap_dead start 2024-03-06T07:32:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.400+0000 7fdbaaa9a640 1 -- 172.21.15.1:0/4127450147 shutdown_connections 2024-03-06T07:32:20.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.400+0000 7fdbaaa9a640 1 -- 172.21.15.1:0/4127450147 >> 172.21.15.1:0/4127450147 conn(0x7fdba4066ee0 msgr2=0x7fdba40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.400+0000 7fdbaaa9a640 1 -- 172.21.15.1:0/4127450147 shutdown_connections 2024-03-06T07:32:20.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.400+0000 7fdbaaa9a640 1 -- 172.21.15.1:0/4127450147 wait complete. 2024-03-06T07:32:20.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.401+0000 7fdbaaa9a640 1 Processor -- start 2024-03-06T07:32:20.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.401+0000 7fdbaaa9a640 1 -- start start 2024-03-06T07:32:20.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.402+0000 7fdbaaa9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb94003f20 0x7fdba40c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.402+0000 7fdbaaa9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba414b700 0x7fdba41504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.402+0000 7fdbaaa9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba4151b70 0x7fdba4150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.402+0000 7fdbaaa9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba41551b0 con 0x7fdb94003f20 2024-03-06T07:32:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.402+0000 7fdbaaa9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40c6fa0 con 0x7fdba4151b70 2024-03-06T07:32:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.402+0000 7fdbaaa9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba4155490 con 0x7fdba414b700 2024-03-06T07:32:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.402+0000 7fdba8836640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb94003f20 0x7fdba40c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.402+0000 7fdba8836640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb94003f20 0x7fdba40c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36026/0 (socket says 172.21.15.1:36026) 2024-03-06T07:32:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.402+0000 7fdba8836640 1 -- 172.21.15.1:0/2363538527 learned_addr learned my addr 172.21.15.1:0/2363538527 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.402+0000 7fdba3fff640 1 --2- 172.21.15.1:0/2363538527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba414b700 0x7fdba41504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.402+0000 7fdba9037640 1 --2- 172.21.15.1:0/2363538527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba4151b70 0x7fdba4150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.402+0000 7fdba8836640 1 -- 172.21.15.1:0/2363538527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba414b700 msgr2=0x7fdba41504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.402+0000 7fdba8836640 1 --2- 172.21.15.1:0/2363538527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba414b700 0x7fdba41504d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.402+0000 7fdba8836640 1 -- 172.21.15.1:0/2363538527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba4151b70 msgr2=0x7fdba4150a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.402+0000 7fdba8836640 1 --2- 172.21.15.1:0/2363538527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba4151b70 0x7fdba4150a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.402+0000 7fdba8836640 1 -- 172.21.15.1:0/2363538527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba41debd0 con 0x7fdb94003f20 2024-03-06T07:32:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.403+0000 7fdba8836640 1 --2- 172.21.15.1:0/2363538527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb94003f20 0x7fdba40c9a90 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7fdb9800f7f0 tx=0x7fdb9800fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.403+0000 7fdba17fa640 1 -- 172.21.15.1:0/2363538527 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb980052e0 con 0x7fdb94003f20 2024-03-06T07:32:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.403+0000 7fdba17fa640 1 -- 172.21.15.1:0/2363538527 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb98018070 con 0x7fdb94003f20 2024-03-06T07:32:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.403+0000 7fdbaaa9a640 1 -- 172.21.15.1:0/2363538527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba41eb8b0 con 0x7fdb94003f20 2024-03-06T07:32:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.403+0000 7fdba17fa640 1 -- 172.21.15.1:0/2363538527 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9800d970 con 0x7fdb94003f20 2024-03-06T07:32:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.403+0000 7fdbaaa9a640 1 -- 172.21.15.1:0/2363538527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba40c9fd0 con 0x7fdb94003f20 2024-03-06T07:32:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.404+0000 7fdbaaa9a640 1 -- 172.21.15.1:0/2363538527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba40b0d50 con 0x7fdb94003f20 2024-03-06T07:32:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.404+0000 7fdba17fa640 1 -- 172.21.15.1:0/2363538527 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb9800db10 con 0x7fdb94003f20 2024-03-06T07:32:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.405+0000 7fdba17fa640 1 --2- 172.21.15.1:0/2363538527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb8806b4d0 0x7fdb8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.405+0000 7fdba17fa640 1 -- 172.21.15.1:0/2363538527 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fdb9808a290 con 0x7fdb94003f20 2024-03-06T07:32:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.405+0000 7fdba3fff640 1 --2- 172.21.15.1:0/2363538527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb8806b4d0 0x7fdb8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.405+0000 7fdba3fff640 1 --2- 172.21.15.1:0/2363538527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb8806b4d0 0x7fdb8806d990 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fdb90034000 tx=0x7fdb90007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.409+0000 7fdba17fa640 1 -- 172.21.15.1:0/2363538527 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdba40b0d50 con 0x7fdb94003f20 2024-03-06T07:32:20.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.634+0000 7fdbaaa9a640 1 -- 172.21.15.1:0/2363538527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdba4067fd0 con 0x7fdb94003f20 2024-03-06T07:32:20.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.634+0000 7fdba17fa640 1 -- 172.21.15.1:0/2363538527 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fdb980588c0 con 0x7fdb94003f20 2024-03-06T07:32:20.636 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:20.636 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:20.636 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:20.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.636+0000 7fdbaaa9a640 1 -- 172.21.15.1:0/2363538527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb8806b4d0 msgr2=0x7fdb8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.636+0000 7fdbaaa9a640 1 --2- 172.21.15.1:0/2363538527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb8806b4d0 0x7fdb8806d990 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fdb90034000 tx=0x7fdb90007000 comp rx=0 tx=0).stop 2024-03-06T07:32:20.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.636+0000 7fdbaaa9a640 1 -- 172.21.15.1:0/2363538527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb94003f20 msgr2=0x7fdba40c9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.636+0000 7fdbaaa9a640 1 --2- 172.21.15.1:0/2363538527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb94003f20 0x7fdba40c9a90 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7fdb9800f7f0 tx=0x7fdb9800fcb0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.636+0000 7fdbaaa9a640 1 -- 172.21.15.1:0/2363538527 shutdown_connections 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.636+0000 7fdbaaa9a640 1 --2- 172.21.15.1:0/2363538527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdb8806b4d0 0x7fdb8806d990 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.636+0000 7fdbaaa9a640 1 --2- 172.21.15.1:0/2363538527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba4151b70 0x7fdba4150a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.636+0000 7fdbaaa9a640 1 --2- 172.21.15.1:0/2363538527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdba414b700 0x7fdba41504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.636+0000 7fdbaaa9a640 1 --2- 172.21.15.1:0/2363538527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb94003f20 0x7fdba40c9a90 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.636+0000 7fdbaaa9a640 1 -- 172.21.15.1:0/2363538527 >> 172.21.15.1:0/2363538527 conn(0x7fdba4066ee0 msgr2=0x7fdba40cac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.636+0000 7fdbaaa9a640 1 -- 172.21.15.1:0/2363538527 shutdown_connections 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:20.636+0000 7fdbaaa9a640 1 -- 172.21.15.1:0/2363538527 wait complete. 2024-03-06T07:32:21.051 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:21.051 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:22.052 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.152+0000 7fe3c6e4e640 1 Processor -- start 2024-03-06T07:32:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.152+0000 7fe3c6e4e640 1 -- start start 2024-03-06T07:32:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.153+0000 7fe3c6e4e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe3c0152650 0x7fe3c0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.153+0000 7fe3c6e4e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3c00bf9d0 0x7fe3c00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.153+0000 7fe3c6e4e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c00c2fc0 con 0x7fe3c00c6b10 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.153+0000 7fe3c6e4e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c0155800 con 0x7fe3c0152650 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.153+0000 7fe3c6e4e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c0158da0 con 0x7fe3c00bf9d0 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.153+0000 7fe3c4bea640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe3c0152650 0x7fe3c0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.153+0000 7fe3c4bea640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe3c0152650 0x7fe3c0155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36032/0 (socket says 172.21.15.1:36032) 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.153+0000 7fe3c4bea640 1 -- 172.21.15.1:0/3793054289 learned_addr learned my addr 172.21.15.1:0/3793054289 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.153+0000 7fe3c53eb640 1 --2- 172.21.15.1:0/3793054289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3c00bf9d0 0x7fe3c00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.153+0000 7fe3c4bea640 1 -- 172.21.15.1:0/3793054289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3c00bf9d0 msgr2=0x7fe3c00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.153+0000 7fe3c4bea640 1 --2- 172.21.15.1:0/3793054289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3c00bf9d0 0x7fe3c00c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.153+0000 7fe3c4bea640 1 -- 172.21.15.1:0/3793054289 >> v1:172.21.15.98:6789/0 conn(0x7fe3c00c6b10 legacy=0x7fe3c0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.153+0000 7fe3c4bea640 1 -- 172.21.15.1:0/3793054289 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c00c6690 con 0x7fe3c0152650 2024-03-06T07:32:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.154+0000 7fe3c4bea640 1 --2- 172.21.15.1:0/3793054289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3c0152650 0x7fe3c0155c90 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00cac90 tx=0x7fe3b802be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892f405ad59a489 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.154+0000 7fe3bf7fe640 1 -- 172.21.15.1:0/3793054289 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b802c9a0 con 0x7fe3c0152650 2024-03-06T07:32:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.154+0000 7fe3bf7fe640 1 -- 172.21.15.1:0/3793054289 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b802cb40 con 0x7fe3c0152650 2024-03-06T07:32:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.154+0000 7fe3bf7fe640 1 -- 172.21.15.1:0/3793054289 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b80343f0 con 0x7fe3c0152650 2024-03-06T07:32:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.155+0000 7fe3c6e4e640 1 -- 172.21.15.1:0/3793054289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3c0152650 msgr2=0x7fe3c0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.155+0000 7fe3c6e4e640 1 --2- 172.21.15.1:0/3793054289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3c0152650 0x7fe3c0155c90 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00cac90 tx=0x7fe3b802be00 comp rx=0 tx=0).stop 2024-03-06T07:32:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.155+0000 7fe3c6e4e640 1 -- 172.21.15.1:0/3793054289 shutdown_connections 2024-03-06T07:32:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.155+0000 7fe3c6e4e640 1 --2- 172.21.15.1:0/3793054289 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3c00bf9d0 0x7fe3c00c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.155+0000 7fe3c6e4e640 1 --2- 172.21.15.1:0/3793054289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3c0152650 0x7fe3c0155c90 unknown :-1 s=CLOSED pgs=2588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.155+0000 7fe3c6e4e640 1 -- 172.21.15.1:0/3793054289 >> 172.21.15.1:0/3793054289 conn(0x7fe3c0066ee0 msgr2=0x7fe3c00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.155+0000 7fe3c6e4e640 1 -- 172.21.15.1:0/3793054289 shutdown_connections 2024-03-06T07:32:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.155+0000 7fe3c6e4e640 1 -- 172.21.15.1:0/3793054289 wait complete. 2024-03-06T07:32:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.157+0000 7fe3c6e4e640 1 Processor -- start 2024-03-06T07:32:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.157+0000 7fe3c6e4e640 1 -- start start 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c6e4e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3c00bf9d0 0x7fe3c01ef000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c6e4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c0152650 0x7fe3c00c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c6e4e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3c01514f0 0x7fe3c013d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c6e4e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c01551b0 con 0x7fe3c00bf9d0 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c6e4e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c00c6fa0 con 0x7fe3c0152650 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c6e4e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c0155490 con 0x7fe3c01514f0 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c4bea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3c00bf9d0 0x7fe3c01ef000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c4bea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3c00bf9d0 0x7fe3c01ef000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36044/0 (socket says 172.21.15.1:36044) 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c4bea640 1 -- 172.21.15.1:0/162851869 learned_addr learned my addr 172.21.15.1:0/162851869 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c53eb640 1 --2- 172.21.15.1:0/162851869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3c01514f0 0x7fe3c013d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3bffff640 1 --2- 172.21.15.1:0/162851869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c0152650 0x7fe3c00c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c4bea640 1 -- 172.21.15.1:0/162851869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3c01514f0 msgr2=0x7fe3c013d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c4bea640 1 --2- 172.21.15.1:0/162851869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3c01514f0 0x7fe3c013d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c4bea640 1 -- 172.21.15.1:0/162851869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c0152650 msgr2=0x7fe3c00c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c4bea640 1 --2- 172.21.15.1:0/162851869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c0152650 0x7fe3c00c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c4bea640 1 -- 172.21.15.1:0/162851869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c0151a40 con 0x7fe3c00bf9d0 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c4bea640 1 --2- 172.21.15.1:0/162851869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3c00bf9d0 0x7fe3c01ef000 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fe3b802c410 tx=0x7fe3b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3bd7fa640 1 -- 172.21.15.1:0/162851869 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8043020 con 0x7fe3c00bf9d0 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3bd7fa640 1 -- 172.21.15.1:0/162851869 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b803e450 con 0x7fe3c00bf9d0 2024-03-06T07:32:22.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c6e4e640 1 -- 172.21.15.1:0/162851869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c0156a10 con 0x7fe3c00bf9d0 2024-03-06T07:32:22.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3bd7fa640 1 -- 172.21.15.1:0/162851869 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8009510 con 0x7fe3c00bf9d0 2024-03-06T07:32:22.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.159+0000 7fe3c6e4e640 1 -- 172.21.15.1:0/162851869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c00c9fd0 con 0x7fe3c00bf9d0 2024-03-06T07:32:22.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.160+0000 7fe3c6e4e640 1 -- 172.21.15.1:0/162851869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c0067fd0 con 0x7fe3c00bf9d0 2024-03-06T07:32:22.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.160+0000 7fe3bd7fa640 1 -- 172.21.15.1:0/162851869 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe3b8046020 con 0x7fe3c00bf9d0 2024-03-06T07:32:22.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.161+0000 7fe3bd7fa640 1 --2- 172.21.15.1:0/162851869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe3a006b500 0x7fe3a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.161+0000 7fe3bd7fa640 1 -- 172.21.15.1:0/162851869 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fe3b80acf80 con 0x7fe3c00bf9d0 2024-03-06T07:32:22.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.161+0000 7fe3bffff640 1 --2- 172.21.15.1:0/162851869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe3a006b500 0x7fe3a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.162+0000 7fe3bffff640 1 --2- 172.21.15.1:0/162851869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe3a006b500 0x7fe3a006d9c0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0158c60 tx=0x7fe3b0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.166+0000 7fe3bd7fa640 1 -- 172.21.15.1:0/162851869 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3b804d020 con 0x7fe3c00bf9d0 2024-03-06T07:32:22.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.384+0000 7fe3c6e4e640 1 -- 172.21.15.1:0/162851869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3c00b01a0 con 0x7fe3c00bf9d0 2024-03-06T07:32:22.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.384+0000 7fe3bd7fa640 1 -- 172.21.15.1:0/162851869 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fe3b80333a0 con 0x7fe3c00bf9d0 2024-03-06T07:32:22.386 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:22.386 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:22.387 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:22.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.386+0000 7fe3c6e4e640 1 -- 172.21.15.1:0/162851869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe3a006b500 msgr2=0x7fe3a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.386+0000 7fe3c6e4e640 1 --2- 172.21.15.1:0/162851869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe3a006b500 0x7fe3a006d9c0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0158c60 tx=0x7fe3b0009040 comp rx=0 tx=0).stop 2024-03-06T07:32:22.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.386+0000 7fe3c6e4e640 1 -- 172.21.15.1:0/162851869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3c00bf9d0 msgr2=0x7fe3c01ef000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.386+0000 7fe3c6e4e640 1 --2- 172.21.15.1:0/162851869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3c00bf9d0 0x7fe3c01ef000 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fe3b802c410 tx=0x7fe3b8004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:22.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.386+0000 7fe3c6e4e640 1 -- 172.21.15.1:0/162851869 shutdown_connections 2024-03-06T07:32:22.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.386+0000 7fe3c6e4e640 1 --2- 172.21.15.1:0/162851869 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe3a006b500 0x7fe3a006d9c0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.386+0000 7fe3c6e4e640 1 --2- 172.21.15.1:0/162851869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3c01514f0 0x7fe3c013d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.386+0000 7fe3c6e4e640 1 --2- 172.21.15.1:0/162851869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c0152650 0x7fe3c00c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.386+0000 7fe3c6e4e640 1 --2- 172.21.15.1:0/162851869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3c00bf9d0 0x7fe3c01ef000 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.386+0000 7fe3c6e4e640 1 -- 172.21.15.1:0/162851869 >> 172.21.15.1:0/162851869 conn(0x7fe3c0066ee0 msgr2=0x7fe3c00c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.386+0000 7fe3c6e4e640 1 -- 172.21.15.1:0/162851869 shutdown_connections 2024-03-06T07:32:22.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:22.386+0000 7fe3c6e4e640 1 -- 172.21.15.1:0/162851869 wait complete. 2024-03-06T07:32:22.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:22.801 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:23.801 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.904+0000 7f82b9a88640 1 Processor -- start 2024-03-06T07:32:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.904+0000 7f82b9a88640 1 -- start start 2024-03-06T07:32:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.905+0000 7f82b9a88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82b4072bd0 0x7f82b40c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.905+0000 7f82b9a88640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82b4152b70 0x7f82b40c3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.905+0000 7f82b9a88640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b4151dd0 con 0x7f82b40c2b30 2024-03-06T07:32:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.906+0000 7f82b9a88640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82b41526e0 con 0x7f82b4152b70 2024-03-06T07:32:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.906+0000 7f82b9a88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b4156190 con 0x7f82b4072bd0 2024-03-06T07:32:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.906+0000 7f82b3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82b4152b70 0x7f82b40c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.906+0000 7f82b37fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f82b40c2b30 0x7f82b40724c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53676/0 (socket says 172.21.15.1:53676) 2024-03-06T07:32:23.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.906+0000 7f82b37fe640 1 -- 172.21.15.1:0/1082428935 learned_addr learned my addr 172.21.15.1:0/1082428935 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:23.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.906+0000 7f82b2ffd640 1 --2- 172.21.15.1:0/1082428935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82b4072bd0 0x7f82b40c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.906+0000 7f82b3fff640 1 -- 172.21.15.1:0/1082428935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82b4072bd0 msgr2=0x7f82b40c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.906+0000 7f82b3fff640 1 --2- 172.21.15.1:0/1082428935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82b4072bd0 0x7f82b40c32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.906+0000 7f82b3fff640 1 -- 172.21.15.1:0/1082428935 >> v1:172.21.15.1:6789/0 conn(0x7f82b40c2b30 legacy=0x7f82b40724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.906+0000 7f82b3fff640 1 -- 172.21.15.1:0/1082428935 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b4152090 con 0x7f82b4152b70 2024-03-06T07:32:23.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.906+0000 7f82b27fc640 1 -- 172.21.15.1:0/1082428935 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200888914 0 0) 0x7f82b4151dd0 con 0x7f82b40c2b30 2024-03-06T07:32:23.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.906+0000 7f82b2ffd640 1 --2- 172.21.15.1:0/1082428935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82b4072bd0 0x7f82b40c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:23.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.907+0000 7f82b3fff640 1 --2- 172.21.15.1:0/1082428935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82b4152b70 0x7f82b40c3830 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f82ac00bfa0 tx=0x7f82ac02e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1868bb8c263f76ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.907+0000 7f82b27fc640 1 -- 172.21.15.1:0/1082428935 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ac02ea40 con 0x7f82b4152b70 2024-03-06T07:32:23.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.907+0000 7f82b27fc640 1 -- 172.21.15.1:0/1082428935 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ac02ebe0 con 0x7f82b4152b70 2024-03-06T07:32:23.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.907+0000 7f82b27fc640 1 -- 172.21.15.1:0/1082428935 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ac009540 con 0x7f82b4152b70 2024-03-06T07:32:23.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.908+0000 7f82b9a88640 1 -- 172.21.15.1:0/1082428935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82b4152b70 msgr2=0x7f82b40c3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.908+0000 7f82b9a88640 1 --2- 172.21.15.1:0/1082428935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82b4152b70 0x7f82b40c3830 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f82ac00bfa0 tx=0x7f82ac02e0a0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.908+0000 7f82b9a88640 1 -- 172.21.15.1:0/1082428935 shutdown_connections 2024-03-06T07:32:23.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.908+0000 7f82b9a88640 1 --2- 172.21.15.1:0/1082428935 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82b4072bd0 0x7f82b40c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.908+0000 7f82b9a88640 1 --2- 172.21.15.1:0/1082428935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82b4152b70 0x7f82b40c3830 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.908+0000 7f82b9a88640 1 -- 172.21.15.1:0/1082428935 >> 172.21.15.1:0/1082428935 conn(0x7f82b40c6860 msgr2=0x7f82b40c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.908+0000 7f82b9a88640 1 -- 172.21.15.1:0/1082428935 shutdown_connections 2024-03-06T07:32:23.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.908+0000 7f82b9a88640 1 -- 172.21.15.1:0/1082428935 wait complete. 2024-03-06T07:32:23.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.909+0000 7f82b9a88640 1 Processor -- start 2024-03-06T07:32:23.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.909+0000 7f82b9a88640 1 -- start start 2024-03-06T07:32:23.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82b9a88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82b4072bd0 0x7f82b414e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82b9a88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82b40c2b30 0x7f82b4149920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82b9a88640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82b4152b70 0x7f82b4149e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82b9a88640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b40c3ef0 con 0x7f82b40c2b30 2024-03-06T07:32:23.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82b9a88640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b40c3d70 con 0x7f82b4072bd0 2024-03-06T07:32:23.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82b9a88640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b4155d80 con 0x7f82b4152b70 2024-03-06T07:32:23.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82b2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82b40c2b30 0x7f82b4149920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82b2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82b40c2b30 0x7f82b4149920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36064/0 (socket says 172.21.15.1:36064) 2024-03-06T07:32:23.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82b2ffd640 1 -- 172.21.15.1:0/919759431 learned_addr learned my addr 172.21.15.1:0/919759431 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:23.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82b2ffd640 1 -- 172.21.15.1:0/919759431 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82b4152b70 msgr2=0x7f82b4149e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82b2ffd640 1 --2- 172.21.15.1:0/919759431 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82b4152b70 0x7f82b4149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82b2ffd640 1 -- 172.21.15.1:0/919759431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82b4072bd0 msgr2=0x7f82b414e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82b2ffd640 1 --2- 172.21.15.1:0/919759431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82b4072bd0 0x7f82b414e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82b2ffd640 1 -- 172.21.15.1:0/919759431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b41e1960 con 0x7f82b40c2b30 2024-03-06T07:32:23.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82b2ffd640 1 --2- 172.21.15.1:0/919759431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82b40c2b30 0x7f82b4149920 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f82a400e840 tx=0x7f82a400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82abfff640 1 -- 172.21.15.1:0/919759431 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a400f040 con 0x7f82b40c2b30 2024-03-06T07:32:23.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82abfff640 1 -- 172.21.15.1:0/919759431 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a4004030 con 0x7f82b40c2b30 2024-03-06T07:32:23.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82b9a88640 1 -- 172.21.15.1:0/919759431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b41e1c40 con 0x7f82b40c2b30 2024-03-06T07:32:23.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82abfff640 1 -- 172.21.15.1:0/919759431 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a4004400 con 0x7f82b40c2b30 2024-03-06T07:32:23.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.910+0000 7f82b9a88640 1 -- 172.21.15.1:0/919759431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b41e2200 con 0x7f82b40c2b30 2024-03-06T07:32:23.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.911+0000 7f82b9a88640 1 -- 172.21.15.1:0/919759431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b4067fd0 con 0x7f82b40c2b30 2024-03-06T07:32:23.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.912+0000 7f82abfff640 1 -- 172.21.15.1:0/919759431 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82a400c6d0 con 0x7f82b40c2b30 2024-03-06T07:32:23.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.912+0000 7f82abfff640 1 --2- 172.21.15.1:0/919759431 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f829406b500 0x7f829406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.913+0000 7f82abfff640 1 -- 172.21.15.1:0/919759431 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f82a408a0a0 con 0x7f82b40c2b30 2024-03-06T07:32:23.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.913+0000 7f82b37fe640 1 --2- 172.21.15.1:0/919759431 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f829406b500 0x7f829406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.913+0000 7f82b37fe640 1 --2- 172.21.15.1:0/919759431 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f829406b500 0x7f829406d9c0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f829c007fd0 tx=0x7f829c007920 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:23.917+0000 7f82abfff640 1 -- 172.21.15.1:0/919759431 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82a4017020 con 0x7f82b40c2b30 2024-03-06T07:32:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:24.137+0000 7f82b9a88640 1 -- 172.21.15.1:0/919759431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82b40a5d10 con 0x7f82b40c2b30 2024-03-06T07:32:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:24.138+0000 7f82abfff640 1 -- 172.21.15.1:0/919759431 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f82a4058d20 con 0x7f82b40c2b30 2024-03-06T07:32:24.140 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:24.140 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:24.140 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:24.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:24.140+0000 7f82b9a88640 1 -- 172.21.15.1:0/919759431 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f829406b500 msgr2=0x7f829406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:24.140+0000 7f82b9a88640 1 --2- 172.21.15.1:0/919759431 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f829406b500 0x7f829406d9c0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f829c007fd0 tx=0x7f829c007920 comp rx=0 tx=0).stop 2024-03-06T07:32:24.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:24.140+0000 7f82b9a88640 1 -- 172.21.15.1:0/919759431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82b40c2b30 msgr2=0x7f82b4149920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:24.140+0000 7f82b9a88640 1 --2- 172.21.15.1:0/919759431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82b40c2b30 0x7f82b4149920 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f82a400e840 tx=0x7f82a400ed00 comp rx=0 tx=0).stop 2024-03-06T07:32:24.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:24.140+0000 7f82b9a88640 1 -- 172.21.15.1:0/919759431 shutdown_connections 2024-03-06T07:32:24.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:24.140+0000 7f82b9a88640 1 --2- 172.21.15.1:0/919759431 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f829406b500 0x7f829406d9c0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:24.140+0000 7f82b9a88640 1 --2- 172.21.15.1:0/919759431 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82b4152b70 0x7f82b4149e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:24.140+0000 7f82b9a88640 1 --2- 172.21.15.1:0/919759431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82b40c2b30 0x7f82b4149920 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:24.140+0000 7f82b9a88640 1 --2- 172.21.15.1:0/919759431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82b4072bd0 0x7f82b414e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:24.140+0000 7f82b9a88640 1 -- 172.21.15.1:0/919759431 >> 172.21.15.1:0/919759431 conn(0x7f82b40c6860 msgr2=0x7f82b41517b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:24.140+0000 7f82b9a88640 1 -- 172.21.15.1:0/919759431 shutdown_connections 2024-03-06T07:32:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:24.140+0000 7f82b9a88640 1 -- 172.21.15.1:0/919759431 wait complete. 2024-03-06T07:32:24.556 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:24.556 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:25.556 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:25.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.659+0000 7f0a9068d640 1 Processor -- start 2024-03-06T07:32:25.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.659+0000 7f0a9068d640 1 -- start start 2024-03-06T07:32:25.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.660+0000 7f0a9068d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0a880c6b10 0x7f0a88155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.660+0000 7f0a9068d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0a880bf9d0 0x7f0a880c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.660+0000 7f0a9068d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a880c2fc0 con 0x7f0a88152650 2024-03-06T07:32:25.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.660+0000 7f0a9068d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a88155800 con 0x7f0a880c6b10 2024-03-06T07:32:25.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.660+0000 7f0a9068d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a88158da0 con 0x7f0a880bf9d0 2024-03-06T07:32:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.660+0000 7f0a8dc28640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0a880c6b10 0x7f0a88155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.660+0000 7f0a8dc28640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0a880c6b10 0x7f0a88155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40968/0 (socket says 172.21.15.1:40968) 2024-03-06T07:32:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.660+0000 7f0a8dc28640 1 -- 172.21.15.1:0/2438471087 learned_addr learned my addr 172.21.15.1:0/2438471087 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.660+0000 7f0a8ec2a640 1 --2- 172.21.15.1:0/2438471087 >> v2:172.21.15.98:3300/0 conn(0x7f0a880bf9d0 0x7f0a880c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.660+0000 7f0a8dc28640 1 -- 172.21.15.1:0/2438471087 >> v2:172.21.15.98:3300/0 conn(0x7f0a880bf9d0 msgr2=0x7f0a880c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.660+0000 7f0a8dc28640 1 --2- 172.21.15.1:0/2438471087 >> v2:172.21.15.98:3300/0 conn(0x7f0a880bf9d0 0x7f0a880c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.660+0000 7f0a8dc28640 1 -- 172.21.15.1:0/2438471087 >> v1:172.21.15.98:6789/0 conn(0x7f0a88152650 legacy=0x7f0a88158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.660+0000 7f0a8dc28640 1 -- 172.21.15.1:0/2438471087 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a880c6690 con 0x7f0a880c6b10 2024-03-06T07:32:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.660+0000 7f0a8ec2a640 1 --2- 172.21.15.1:0/2438471087 >> v2:172.21.15.98:3300/0 conn(0x7f0a880bf9d0 0x7f0a880c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.661+0000 7f0a8dc28640 1 --2- 172.21.15.1:0/2438471087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a880c6b10 0x7f0a88155c90 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f0a881597b0 tx=0x7f0a7802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df4559ef071dfcf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.661+0000 7f0a8d427640 1 -- 172.21.15.1:0/2438471087 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7802e070 con 0x7f0a880c6b10 2024-03-06T07:32:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.661+0000 7f0a8d427640 1 -- 172.21.15.1:0/2438471087 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a78037070 con 0x7f0a880c6b10 2024-03-06T07:32:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.661+0000 7f0a8d427640 1 -- 172.21.15.1:0/2438471087 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a78031a00 con 0x7f0a880c6b10 2024-03-06T07:32:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.662+0000 7f0a9068d640 1 -- 172.21.15.1:0/2438471087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a880c6b10 msgr2=0x7f0a88155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.662+0000 7f0a9068d640 1 --2- 172.21.15.1:0/2438471087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a880c6b10 0x7f0a88155c90 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f0a881597b0 tx=0x7f0a7802b880 comp rx=0 tx=0).stop 2024-03-06T07:32:25.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.662+0000 7f0a9068d640 1 -- 172.21.15.1:0/2438471087 shutdown_connections 2024-03-06T07:32:25.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.662+0000 7f0a9068d640 1 --2- 172.21.15.1:0/2438471087 >> v2:172.21.15.98:3300/0 conn(0x7f0a880bf9d0 0x7f0a880c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.662+0000 7f0a9068d640 1 --2- 172.21.15.1:0/2438471087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a880c6b10 0x7f0a88155c90 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.662+0000 7f0a9068d640 1 -- 172.21.15.1:0/2438471087 >> 172.21.15.1:0/2438471087 conn(0x7f0a88066ee0 msgr2=0x7f0a880672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.662+0000 7f0a9068d640 1 -- 172.21.15.1:0/2438471087 shutdown_connections 2024-03-06T07:32:25.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.662+0000 7f0a9068d640 1 -- 172.21.15.1:0/2438471087 wait complete. 2024-03-06T07:32:25.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.664+0000 7f0a9068d640 1 Processor -- start 2024-03-06T07:32:25.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.664+0000 7f0a9068d640 1 -- start start 2024-03-06T07:32:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a9068d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a880bf9d0 0x7f0a8814e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a9068d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a88152650 0x7f0a8814b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a9068d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a880c87d0 0x7f0a8814b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a9068d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a881551b0 con 0x7f0a880c87d0 2024-03-06T07:32:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a9068d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a880c6fa0 con 0x7f0a88152650 2024-03-06T07:32:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a9068d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a88155490 con 0x7f0a880bf9d0 2024-03-06T07:32:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a8ec2a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a880c87d0 0x7f0a8814b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a8ec2a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a880c87d0 0x7f0a8814b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40972/0 (socket says 172.21.15.1:40972) 2024-03-06T07:32:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a8ec2a640 1 -- 172.21.15.1:0/46875430 learned_addr learned my addr 172.21.15.1:0/46875430 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a8e429640 1 --2- 172.21.15.1:0/46875430 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a880bf9d0 0x7f0a8814e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a8dc28640 1 --2- 172.21.15.1:0/46875430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a88152650 0x7f0a8814b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a8ec2a640 1 -- 172.21.15.1:0/46875430 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a880bf9d0 msgr2=0x7f0a8814e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a8ec2a640 1 --2- 172.21.15.1:0/46875430 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a880bf9d0 0x7f0a8814e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a8ec2a640 1 -- 172.21.15.1:0/46875430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a88152650 msgr2=0x7f0a8814b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a8ec2a640 1 --2- 172.21.15.1:0/46875430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a88152650 0x7f0a8814b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a8ec2a640 1 -- 172.21.15.1:0/46875430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8814bc50 con 0x7f0a880c87d0 2024-03-06T07:32:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a8dc28640 1 --2- 172.21.15.1:0/46875430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a88152650 0x7f0a8814b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.665+0000 7f0a8ec2a640 1 --2- 172.21.15.1:0/46875430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a880c87d0 0x7f0a8814b590 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f0a84007e50 tx=0x7f0a8400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.666+0000 7f0a7effd640 1 -- 172.21.15.1:0/46875430 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8401a070 con 0x7f0a880c87d0 2024-03-06T07:32:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.666+0000 7f0a7effd640 1 -- 172.21.15.1:0/46875430 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a84003990 con 0x7f0a880c87d0 2024-03-06T07:32:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.666+0000 7f0a7effd640 1 -- 172.21.15.1:0/46875430 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a84003d80 con 0x7f0a880c87d0 2024-03-06T07:32:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.666+0000 7f0a9068d640 1 -- 172.21.15.1:0/46875430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8814ddd0 con 0x7f0a880c87d0 2024-03-06T07:32:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.666+0000 7f0a9068d640 1 -- 172.21.15.1:0/46875430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a88156630 con 0x7f0a880c87d0 2024-03-06T07:32:25.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.667+0000 7f0a9068d640 1 -- 172.21.15.1:0/46875430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a881585a0 con 0x7f0a880c87d0 2024-03-06T07:32:25.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.667+0000 7f0a7effd640 1 -- 172.21.15.1:0/46875430 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a8401d070 con 0x7f0a880c87d0 2024-03-06T07:32:25.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.667+0000 7f0a7effd640 1 --2- 172.21.15.1:0/46875430 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a6c06b500 0x7f0a6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.668+0000 7f0a7effd640 1 -- 172.21.15.1:0/46875430 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f0a8408a5d0 con 0x7f0a880c87d0 2024-03-06T07:32:25.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.668+0000 7f0a8e429640 1 --2- 172.21.15.1:0/46875430 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a6c06b500 0x7f0a6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.668+0000 7f0a8e429640 1 --2- 172.21.15.1:0/46875430 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a6c06b500 0x7f0a6c06d9c0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f0a881ec530 tx=0x7f0a80009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.672+0000 7f0a7effd640 1 -- 172.21.15.1:0/46875430 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a881585a0 con 0x7f0a880c87d0 2024-03-06T07:32:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.896+0000 7f0a9068d640 1 -- 172.21.15.1:0/46875430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a8806b540 con 0x7f0a880c87d0 2024-03-06T07:32:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.896+0000 7f0a7effd640 1 -- 172.21.15.1:0/46875430 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f0a8806b540 con 0x7f0a880c87d0 2024-03-06T07:32:25.898 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:25.899 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:25.899 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:25.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.898+0000 7f0a9068d640 1 -- 172.21.15.1:0/46875430 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a6c06b500 msgr2=0x7f0a6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.898+0000 7f0a9068d640 1 --2- 172.21.15.1:0/46875430 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a6c06b500 0x7f0a6c06d9c0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f0a881ec530 tx=0x7f0a80009040 comp rx=0 tx=0).stop 2024-03-06T07:32:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.898+0000 7f0a9068d640 1 -- 172.21.15.1:0/46875430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a880c87d0 msgr2=0x7f0a8814b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.898+0000 7f0a9068d640 1 --2- 172.21.15.1:0/46875430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a880c87d0 0x7f0a8814b590 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f0a84007e50 tx=0x7f0a8400d4a0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.898+0000 7f0a9068d640 1 -- 172.21.15.1:0/46875430 shutdown_connections 2024-03-06T07:32:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.898+0000 7f0a9068d640 1 --2- 172.21.15.1:0/46875430 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a6c06b500 0x7f0a6c06d9c0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.898+0000 7f0a9068d640 1 --2- 172.21.15.1:0/46875430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a880c87d0 0x7f0a8814b590 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.898+0000 7f0a9068d640 1 --2- 172.21.15.1:0/46875430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a88152650 0x7f0a8814b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.898+0000 7f0a9068d640 1 --2- 172.21.15.1:0/46875430 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a880bf9d0 0x7f0a8814e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.898+0000 7f0a9068d640 1 -- 172.21.15.1:0/46875430 >> 172.21.15.1:0/46875430 conn(0x7f0a88066ee0 msgr2=0x7f0a880c4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.898+0000 7f0a9068d640 1 -- 172.21.15.1:0/46875430 shutdown_connections 2024-03-06T07:32:25.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:25.898+0000 7f0a9068d640 1 -- 172.21.15.1:0/46875430 wait complete. 2024-03-06T07:32:26.313 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:26.313 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:27.314 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:27.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.414+0000 7fce5e688640 1 Processor -- start 2024-03-06T07:32:27.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.414+0000 7fce5e688640 1 -- start start 2024-03-06T07:32:27.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.415+0000 7fce5e688640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fce580c6b10 0x7fce58155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.415+0000 7fce5e688640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fce580bf9d0 0x7fce580c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.415+0000 7fce5e688640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce580c2fc0 con 0x7fce58152650 2024-03-06T07:32:27.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.415+0000 7fce5e688640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce58155800 con 0x7fce580c6b10 2024-03-06T07:32:27.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.415+0000 7fce5e688640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce58158da0 con 0x7fce580bf9d0 2024-03-06T07:32:27.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.415+0000 7fce577fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fce580c6b10 0x7fce58155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.415+0000 7fce57fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fce58152650 0x7fce58158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48238/0 (socket says 172.21.15.1:48238) 2024-03-06T07:32:27.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.415+0000 7fce57fff640 1 -- 172.21.15.1:0/2965781175 learned_addr learned my addr 172.21.15.1:0/2965781175 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:27.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.415+0000 7fce577fe640 1 --2- 172.21.15.1:0/2965781175 >> v2:172.21.15.1:3300/0 conn(0x7fce580c6b10 0x7fce58155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40988/0 (socket says 172.21.15.1:40988) 2024-03-06T07:32:27.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.415+0000 7fce577fe640 1 -- 172.21.15.1:0/2965781175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fce580bf9d0 msgr2=0x7fce580c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.415+0000 7fce5cc25640 1 --2- 172.21.15.1:0/2965781175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fce580bf9d0 0x7fce580c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.415+0000 7fce577fe640 1 --2- 172.21.15.1:0/2965781175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fce580bf9d0 0x7fce580c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.415+0000 7fce577fe640 1 -- 172.21.15.1:0/2965781175 >> v1:172.21.15.1:6789/0 conn(0x7fce58152650 legacy=0x7fce58158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.415+0000 7fce56ffd640 1 -- 172.21.15.1:0/2965781175 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505026037 0 0) 0x7fce580c2fc0 con 0x7fce58152650 2024-03-06T07:32:27.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.416+0000 7fce577fe640 1 -- 172.21.15.1:0/2965781175 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce580c6690 con 0x7fce580c6b10 2024-03-06T07:32:27.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.416+0000 7fce5cc25640 1 --2- 172.21.15.1:0/2965781175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fce580bf9d0 0x7fce580c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:27.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.417+0000 7fce577fe640 1 --2- 172.21.15.1:0/2965781175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce580c6b10 0x7fce58155c90 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7fce400097a0 tx=0x7fce4002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e3baf0527edb2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.417+0000 7fce56ffd640 1 -- 172.21.15.1:0/2965781175 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce400315a0 con 0x7fce580c6b10 2024-03-06T07:32:27.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.417+0000 7fce56ffd640 1 -- 172.21.15.1:0/2965781175 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce40037070 con 0x7fce580c6b10 2024-03-06T07:32:27.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.417+0000 7fce56ffd640 1 -- 172.21.15.1:0/2965781175 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce40031960 con 0x7fce580c6b10 2024-03-06T07:32:27.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.417+0000 7fce5e688640 1 -- 172.21.15.1:0/2965781175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce580c6b10 msgr2=0x7fce58155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.417+0000 7fce5e688640 1 --2- 172.21.15.1:0/2965781175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce580c6b10 0x7fce58155c90 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7fce400097a0 tx=0x7fce4002b840 comp rx=0 tx=0).stop 2024-03-06T07:32:27.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.417+0000 7fce5e688640 1 -- 172.21.15.1:0/2965781175 shutdown_connections 2024-03-06T07:32:27.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.417+0000 7fce5e688640 1 --2- 172.21.15.1:0/2965781175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fce580bf9d0 0x7fce580c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.417+0000 7fce5e688640 1 --2- 172.21.15.1:0/2965781175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce580c6b10 0x7fce58155c90 unknown :-1 s=CLOSED pgs=2596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.417+0000 7fce5e688640 1 -- 172.21.15.1:0/2965781175 >> 172.21.15.1:0/2965781175 conn(0x7fce58066ee0 msgr2=0x7fce580672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.417+0000 7fce5e688640 1 -- 172.21.15.1:0/2965781175 shutdown_connections 2024-03-06T07:32:27.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.417+0000 7fce5e688640 1 -- 172.21.15.1:0/2965781175 wait complete. 2024-03-06T07:32:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.418+0000 7fce5e688640 1 Processor -- start 2024-03-06T07:32:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.419+0000 7fce5e688640 1 -- start start 2024-03-06T07:32:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.419+0000 7fce5e688640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce580bf9d0 0x7fce5814d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.419+0000 7fce5e688640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce58152650 0x7fce581e0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.419+0000 7fce5e688640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fce581df1f0 0x7fce58156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.419+0000 7fce5e688640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce581551b0 con 0x7fce58152650 2024-03-06T07:32:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.419+0000 7fce5e688640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce580c6fa0 con 0x7fce580bf9d0 2024-03-06T07:32:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.419+0000 7fce5e688640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce58155490 con 0x7fce581df1f0 2024-03-06T07:32:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.419+0000 7fce577fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce58152650 0x7fce581e0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.419+0000 7fce577fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce58152650 0x7fce581e0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41002/0 (socket says 172.21.15.1:41002) 2024-03-06T07:32:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.419+0000 7fce577fe640 1 -- 172.21.15.1:0/4260995573 learned_addr learned my addr 172.21.15.1:0/4260995573 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.420+0000 7fce57fff640 1 --2- 172.21.15.1:0/4260995573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce580bf9d0 0x7fce5814d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.420+0000 7fce577fe640 1 -- 172.21.15.1:0/4260995573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fce581df1f0 msgr2=0x7fce58156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.420+0000 7fce577fe640 1 --2- 172.21.15.1:0/4260995573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fce581df1f0 0x7fce58156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.420+0000 7fce577fe640 1 -- 172.21.15.1:0/4260995573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce580bf9d0 msgr2=0x7fce5814d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.420+0000 7fce577fe640 1 --2- 172.21.15.1:0/4260995573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce580bf9d0 0x7fce5814d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.420+0000 7fce577fe640 1 -- 172.21.15.1:0/4260995573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce581e1980 con 0x7fce58152650 2024-03-06T07:32:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.420+0000 7fce577fe640 1 --2- 172.21.15.1:0/4260995573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce58152650 0x7fce581e0560 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7fce40034040 tx=0x7fce400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.420+0000 7fce54ff9640 1 -- 172.21.15.1:0/4260995573 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce40032840 con 0x7fce58152650 2024-03-06T07:32:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.420+0000 7fce54ff9640 1 -- 172.21.15.1:0/4260995573 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce40032a00 con 0x7fce58152650 2024-03-06T07:32:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.420+0000 7fce5e688640 1 -- 172.21.15.1:0/4260995573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce581e1c60 con 0x7fce58152650 2024-03-06T07:32:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.420+0000 7fce54ff9640 1 -- 172.21.15.1:0/4260995573 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce40032df0 con 0x7fce58152650 2024-03-06T07:32:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.420+0000 7fce5e688640 1 -- 172.21.15.1:0/4260995573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce581e2220 con 0x7fce58152650 2024-03-06T07:32:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.421+0000 7fce5e688640 1 -- 172.21.15.1:0/4260995573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce58067fd0 con 0x7fce58152650 2024-03-06T07:32:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.422+0000 7fce54ff9640 1 -- 172.21.15.1:0/4260995573 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce4004e430 con 0x7fce58152650 2024-03-06T07:32:27.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.422+0000 7fce54ff9640 1 --2- 172.21.15.1:0/4260995573 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fce3806b500 0x7fce3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.422+0000 7fce54ff9640 1 -- 172.21.15.1:0/4260995573 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fce4002f030 con 0x7fce58152650 2024-03-06T07:32:27.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.422+0000 7fce57fff640 1 --2- 172.21.15.1:0/4260995573 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fce3806b500 0x7fce3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.423+0000 7fce57fff640 1 --2- 172.21.15.1:0/4260995573 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fce3806b500 0x7fce3806d9c0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fce48000f60 tx=0x7fce48009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.426+0000 7fce54ff9640 1 -- 172.21.15.1:0/4260995573 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce40040070 con 0x7fce58152650 2024-03-06T07:32:27.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.646+0000 7fce5e688640 1 -- 172.21.15.1:0/4260995573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce580b06f0 con 0x7fce58152650 2024-03-06T07:32:27.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.647+0000 7fce54ff9640 1 -- 172.21.15.1:0/4260995573 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fce4007b050 con 0x7fce58152650 2024-03-06T07:32:27.649 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:27.649 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:27.649 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:27.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.649+0000 7fce5e688640 1 -- 172.21.15.1:0/4260995573 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fce3806b500 msgr2=0x7fce3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.649+0000 7fce5e688640 1 --2- 172.21.15.1:0/4260995573 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fce3806b500 0x7fce3806d9c0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fce48000f60 tx=0x7fce48009040 comp rx=0 tx=0).stop 2024-03-06T07:32:27.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.649+0000 7fce5e688640 1 -- 172.21.15.1:0/4260995573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce58152650 msgr2=0x7fce581e0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.649+0000 7fce5e688640 1 --2- 172.21.15.1:0/4260995573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce58152650 0x7fce581e0560 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7fce40034040 tx=0x7fce400046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.649+0000 7fce5e688640 1 -- 172.21.15.1:0/4260995573 shutdown_connections 2024-03-06T07:32:27.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.649+0000 7fce5e688640 1 --2- 172.21.15.1:0/4260995573 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fce3806b500 0x7fce3806d9c0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.649+0000 7fce5e688640 1 --2- 172.21.15.1:0/4260995573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fce581df1f0 0x7fce58156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.649+0000 7fce5e688640 1 --2- 172.21.15.1:0/4260995573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce58152650 0x7fce581e0560 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.649+0000 7fce5e688640 1 --2- 172.21.15.1:0/4260995573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce580bf9d0 0x7fce5814d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.649+0000 7fce5e688640 1 -- 172.21.15.1:0/4260995573 >> 172.21.15.1:0/4260995573 conn(0x7fce58066ee0 msgr2=0x7fce580c05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.649+0000 7fce5e688640 1 -- 172.21.15.1:0/4260995573 shutdown_connections 2024-03-06T07:32:27.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:27.649+0000 7fce5e688640 1 -- 172.21.15.1:0/4260995573 wait complete. 2024-03-06T07:32:28.067 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:28.067 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:29.069 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.168+0000 7fcc7d20e640 1 Processor -- start 2024-03-06T07:32:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.168+0000 7fcc7d20e640 1 -- start start 2024-03-06T07:32:29.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.169+0000 7fcc7d20e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcc78152650 0x7fcc78155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.169+0000 7fcc7d20e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcc780bf9d0 0x7fcc780c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.169+0000 7fcc7d20e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc780c2fc0 con 0x7fcc780c6b10 2024-03-06T07:32:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.169+0000 7fcc7d20e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc78155800 con 0x7fcc78152650 2024-03-06T07:32:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.169+0000 7fcc7d20e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc78158da0 con 0x7fcc780bf9d0 2024-03-06T07:32:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.169+0000 7fcc76d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcc78152650 0x7fcc78155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.169+0000 7fcc76d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcc78152650 0x7fcc78155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41006/0 (socket says 172.21.15.1:41006) 2024-03-06T07:32:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.169+0000 7fcc7659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcc780c6b10 0x7fcc78158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48246/0 (socket says 172.21.15.1:48246) 2024-03-06T07:32:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.169+0000 7fcc76d9d640 1 -- 172.21.15.1:0/4146974123 learned_addr learned my addr 172.21.15.1:0/4146974123 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.169+0000 7fcc7759e640 1 --2- 172.21.15.1:0/4146974123 >> v2:172.21.15.98:3300/0 conn(0x7fcc780bf9d0 0x7fcc780c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.169+0000 7fcc76d9d640 1 -- 172.21.15.1:0/4146974123 >> v2:172.21.15.98:3300/0 conn(0x7fcc780bf9d0 msgr2=0x7fcc780c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.169+0000 7fcc76d9d640 1 --2- 172.21.15.1:0/4146974123 >> v2:172.21.15.98:3300/0 conn(0x7fcc780bf9d0 0x7fcc780c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.169+0000 7fcc76d9d640 1 -- 172.21.15.1:0/4146974123 >> v1:172.21.15.1:6789/0 conn(0x7fcc780c6b10 legacy=0x7fcc78158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.169+0000 7fcc76d9d640 1 -- 172.21.15.1:0/4146974123 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc780c6690 con 0x7fcc78152650 2024-03-06T07:32:29.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.169+0000 7fcc75d9b640 1 -- 172.21.15.1:0/4146974123 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1245023946 0 0) 0x7fcc780c2fc0 con 0x7fcc780c6b10 2024-03-06T07:32:29.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.169+0000 7fcc7759e640 1 --2- 172.21.15.1:0/4146974123 >> v2:172.21.15.98:3300/0 conn(0x7fcc780bf9d0 0x7fcc780c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:29.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.170+0000 7fcc76d9d640 1 --2- 172.21.15.1:0/4146974123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc78152650 0x7fcc78155c90 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7fcc781597b0 tx=0x7fcc6002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b2222346f6076d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.170+0000 7fcc75d9b640 1 -- 172.21.15.1:0/4146974123 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc60035020 con 0x7fcc78152650 2024-03-06T07:32:29.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.170+0000 7fcc75d9b640 1 -- 172.21.15.1:0/4146974123 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc60031870 con 0x7fcc78152650 2024-03-06T07:32:29.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.170+0000 7fcc75d9b640 1 -- 172.21.15.1:0/4146974123 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc60031c40 con 0x7fcc78152650 2024-03-06T07:32:29.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.171+0000 7fcc7d20e640 1 -- 172.21.15.1:0/4146974123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc78152650 msgr2=0x7fcc78155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.171+0000 7fcc7d20e640 1 --2- 172.21.15.1:0/4146974123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc78152650 0x7fcc78155c90 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7fcc781597b0 tx=0x7fcc6002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:29.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.171+0000 7fcc7d20e640 1 -- 172.21.15.1:0/4146974123 shutdown_connections 2024-03-06T07:32:29.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.171+0000 7fcc7d20e640 1 --2- 172.21.15.1:0/4146974123 >> v2:172.21.15.98:3300/0 conn(0x7fcc780bf9d0 0x7fcc780c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.171+0000 7fcc7d20e640 1 --2- 172.21.15.1:0/4146974123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc78152650 0x7fcc78155c90 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.171+0000 7fcc7d20e640 1 -- 172.21.15.1:0/4146974123 >> 172.21.15.1:0/4146974123 conn(0x7fcc78066ee0 msgr2=0x7fcc780672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.171+0000 7fcc7d20e640 1 -- 172.21.15.1:0/4146974123 shutdown_connections 2024-03-06T07:32:29.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.171+0000 7fcc7d20e640 1 -- 172.21.15.1:0/4146974123 wait complete. 2024-03-06T07:32:29.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.172+0000 7fcc7d20e640 1 Processor -- start 2024-03-06T07:32:29.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.172+0000 7fcc7d20e640 1 -- start start 2024-03-06T07:32:29.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc7d20e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc780bf9d0 0x7fcc781eab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc7d20e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc78152650 0x7fcc780c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc7d20e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc780c9d60 0x7fcc7814ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc7d20e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc781551b0 con 0x7fcc78152650 2024-03-06T07:32:29.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc7d20e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc780c6fa0 con 0x7fcc780bf9d0 2024-03-06T07:32:29.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc7d20e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc78155490 con 0x7fcc780c9d60 2024-03-06T07:32:29.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc7659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc78152650 0x7fcc780c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc7659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc78152650 0x7fcc780c9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41018/0 (socket says 172.21.15.1:41018) 2024-03-06T07:32:29.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc7659c640 1 -- 172.21.15.1:0/3290291835 learned_addr learned my addr 172.21.15.1:0/3290291835 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:29.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc7759e640 1 --2- 172.21.15.1:0/3290291835 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc780c9d60 0x7fcc7814ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc76d9d640 1 --2- 172.21.15.1:0/3290291835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc780bf9d0 0x7fcc781eab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc7659c640 1 -- 172.21.15.1:0/3290291835 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc780c9d60 msgr2=0x7fcc7814ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc7659c640 1 --2- 172.21.15.1:0/3290291835 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc780c9d60 0x7fcc7814ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc7659c640 1 -- 172.21.15.1:0/3290291835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc780bf9d0 msgr2=0x7fcc781eab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc7659c640 1 --2- 172.21.15.1:0/3290291835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc780bf9d0 0x7fcc781eab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc7659c640 1 -- 172.21.15.1:0/3290291835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc781eb230 con 0x7fcc78152650 2024-03-06T07:32:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc7759e640 1 --2- 172.21.15.1:0/3290291835 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc780c9d60 0x7fcc7814ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.173+0000 7fcc76d9d640 1 --2- 172.21.15.1:0/3290291835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc780bf9d0 0x7fcc781eab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.174+0000 7fcc7659c640 1 --2- 172.21.15.1:0/3290291835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc78152650 0x7fcc780c9820 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7fcc6800c500 tx=0x7fcc6800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.174+0000 7fcc6f7fe640 1 -- 172.21.15.1:0/3290291835 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc68018070 con 0x7fcc78152650 2024-03-06T07:32:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.174+0000 7fcc6f7fe640 1 -- 172.21.15.1:0/3290291835 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc68007580 con 0x7fcc78152650 2024-03-06T07:32:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.174+0000 7fcc7d20e640 1 -- 172.21.15.1:0/3290291835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc781eb510 con 0x7fcc78152650 2024-03-06T07:32:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.174+0000 7fcc6f7fe640 1 -- 172.21.15.1:0/3290291835 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc68004a20 con 0x7fcc78152650 2024-03-06T07:32:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.174+0000 7fcc7d20e640 1 -- 172.21.15.1:0/3290291835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc781ebad0 con 0x7fcc78152650 2024-03-06T07:32:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.175+0000 7fcc7d20e640 1 -- 172.21.15.1:0/3290291835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc78067fd0 con 0x7fcc78152650 2024-03-06T07:32:29.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.175+0000 7fcc6f7fe640 1 -- 172.21.15.1:0/3290291835 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc6801b070 con 0x7fcc78152650 2024-03-06T07:32:29.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.176+0000 7fcc6f7fe640 1 --2- 172.21.15.1:0/3290291835 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc5806b4d0 0x7fcc5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.176+0000 7fcc6f7fe640 1 -- 172.21.15.1:0/3290291835 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fcc68089750 con 0x7fcc78152650 2024-03-06T07:32:29.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.176+0000 7fcc76d9d640 1 --2- 172.21.15.1:0/3290291835 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc5806b4d0 0x7fcc5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.177+0000 7fcc76d9d640 1 --2- 172.21.15.1:0/3290291835 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc5806b4d0 0x7fcc5806d990 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fcc60036000 tx=0x7fcc60008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.180+0000 7fcc6f7fe640 1 -- 172.21.15.1:0/3290291835 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc680583d0 con 0x7fcc78152650 2024-03-06T07:32:29.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.398+0000 7fcc7d20e640 1 -- 172.21.15.1:0/3290291835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc780b06f0 con 0x7fcc78152650 2024-03-06T07:32:29.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.399+0000 7fcc6f7fe640 1 -- 172.21.15.1:0/3290291835 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fcc68057cd0 con 0x7fcc78152650 2024-03-06T07:32:29.401 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:29.401 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:29.401 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:29.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.400+0000 7fcc7d20e640 1 -- 172.21.15.1:0/3290291835 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc5806b4d0 msgr2=0x7fcc5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.400+0000 7fcc7d20e640 1 --2- 172.21.15.1:0/3290291835 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc5806b4d0 0x7fcc5806d990 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fcc60036000 tx=0x7fcc60008030 comp rx=0 tx=0).stop 2024-03-06T07:32:29.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.401+0000 7fcc7d20e640 1 -- 172.21.15.1:0/3290291835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc78152650 msgr2=0x7fcc780c9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.401+0000 7fcc7d20e640 1 --2- 172.21.15.1:0/3290291835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc78152650 0x7fcc780c9820 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7fcc6800c500 tx=0x7fcc6800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.401+0000 7fcc7d20e640 1 -- 172.21.15.1:0/3290291835 shutdown_connections 2024-03-06T07:32:29.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.401+0000 7fcc7d20e640 1 --2- 172.21.15.1:0/3290291835 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fcc5806b4d0 0x7fcc5806d990 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.401+0000 7fcc7d20e640 1 --2- 172.21.15.1:0/3290291835 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc780c9d60 0x7fcc7814ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.401+0000 7fcc7d20e640 1 --2- 172.21.15.1:0/3290291835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc78152650 0x7fcc780c9820 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.401+0000 7fcc7d20e640 1 --2- 172.21.15.1:0/3290291835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc780bf9d0 0x7fcc781eab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.401+0000 7fcc7d20e640 1 -- 172.21.15.1:0/3290291835 >> 172.21.15.1:0/3290291835 conn(0x7fcc78066ee0 msgr2=0x7fcc780c3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.401+0000 7fcc7d20e640 1 -- 172.21.15.1:0/3290291835 shutdown_connections 2024-03-06T07:32:29.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:29.401+0000 7fcc7d20e640 1 -- 172.21.15.1:0/3290291835 wait complete. 2024-03-06T07:32:29.816 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:29.816 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:30.817 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:30.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.916+0000 7f557179a640 1 Processor -- start 2024-03-06T07:32:30.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.916+0000 7f557179a640 1 -- start start 2024-03-06T07:32:30.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.917+0000 7f557179a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f556c0c3eb0 0x7f556c0c7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.917+0000 7f557179a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556c0c71c0 con 0x7f556c0c7940 2024-03-06T07:32:30.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.917+0000 7f557179a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556c0c4e50 con 0x7f556c06d060 2024-03-06T07:32:30.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.917+0000 7f557179a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f556c0c59b0 con 0x7f556c0c3eb0 2024-03-06T07:32:30.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.917+0000 7f556b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f556c0c7940 0x7f556c15fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48254/0 (socket says 172.21.15.1:48254) 2024-03-06T07:32:30.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.917+0000 7f556b7fe640 1 -- 172.21.15.1:0/2524666703 learned_addr learned my addr 172.21.15.1:0/2524666703 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:30.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.917+0000 7f556affd640 1 --2- 172.21.15.1:0/2524666703 >> v2:172.21.15.98:3300/0 conn(0x7f556c0c3eb0 0x7f556c0c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.917+0000 7f5569ffb640 1 -- 172.21.15.1:0/2524666703 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873463896 0 0) 0x7f556c0c71c0 con 0x7f556c0c7940 2024-03-06T07:32:30.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.917+0000 7f5569ffb640 1 -- 172.21.15.1:0/2524666703 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5558003520 con 0x7f556c0c7940 2024-03-06T07:32:30.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f5569ffb640 1 -- 172.21.15.1:0/2524666703 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2675611514 0 0) 0x7f5558003520 con 0x7f556c0c7940 2024-03-06T07:32:30.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f5569ffb640 1 -- 172.21.15.1:0/2524666703 >> v2:172.21.15.98:3300/0 conn(0x7f556c0c3eb0 msgr2=0x7f556c0c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f5569ffb640 1 --2- 172.21.15.1:0/2524666703 >> v2:172.21.15.98:3300/0 conn(0x7f556c0c3eb0 0x7f556c0c7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f5569ffb640 1 -- 172.21.15.1:0/2524666703 >> v1:172.21.15.98:6789/0 conn(0x7f556c06d060 legacy=0x7f556c1558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f5569ffb640 1 -- 172.21.15.1:0/2524666703 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c0c6ee0 con 0x7f556c0c7940 2024-03-06T07:32:30.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f5569ffb640 1 -- 172.21.15.1:0/2524666703 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f55640019f0 con 0x7f556c0c7940 2024-03-06T07:32:30.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f5569ffb640 1 -- 172.21.15.1:0/2524666703 >> v1:172.21.15.1:6789/0 conn(0x7f556c0c7940 legacy=0x7f556c15fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f5569ffb640 1 --2- 172.21.15.1:0/2524666703 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55580039b0 0x7f5558003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f5569ffb640 1 --2- 172.21.15.1:0/2524666703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5558004800 0x7f5558004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f5569ffb640 1 --2- 172.21.15.1:0/2524666703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55580052e0 0x7f555800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f5569ffb640 1 -- 172.21.15.1:0/2524666703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c0c59b0 con 0x7f5558004800 2024-03-06T07:32:30.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f5569ffb640 1 -- 172.21.15.1:0/2524666703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555800c1b0 con 0x7f55580052e0 2024-03-06T07:32:30.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f5569ffb640 1 -- 172.21.15.1:0/2524666703 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555800c3b0 con 0x7f55580039b0 2024-03-06T07:32:30.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f556a7fc640 1 --2- 172.21.15.1:0/2524666703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5558004800 0x7f5558004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f556affd640 1 --2- 172.21.15.1:0/2524666703 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55580039b0 0x7f5558003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f556b7fe640 1 --2- 172.21.15.1:0/2524666703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55580052e0 0x7f555800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f556a7fc640 1 -- 172.21.15.1:0/2524666703 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55580039b0 msgr2=0x7f5558003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f556a7fc640 1 --2- 172.21.15.1:0/2524666703 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f55580039b0 0x7f5558003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f556a7fc640 1 -- 172.21.15.1:0/2524666703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55580052e0 msgr2=0x7f555800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f556a7fc640 1 --2- 172.21.15.1:0/2524666703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55580052e0 0x7f555800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f556a7fc640 1 -- 172.21.15.1:0/2524666703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5558003520 con 0x7f5558004800 2024-03-06T07:32:30.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f556b7fe640 1 --2- 172.21.15.1:0/2524666703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55580052e0 0x7f555800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:30.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.918+0000 7f556b7fe640 1 -- 172.21.15.1:0/2524666703 reap_dead start 2024-03-06T07:32:30.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.919+0000 7f556a7fc640 1 --2- 172.21.15.1:0/2524666703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5558004800 0x7f5558004c80 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f556c06d440 tx=0x7f5558005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3717556edda3567 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.919+0000 7f5569ffb640 1 -- 172.21.15.1:0/2524666703 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554001370 con 0x7f5558004800 2024-03-06T07:32:30.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.919+0000 7f5569ffb640 1 -- 172.21.15.1:0/2524666703 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55540052d0 con 0x7f5558004800 2024-03-06T07:32:30.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.919+0000 7f5569ffb640 1 -- 172.21.15.1:0/2524666703 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554003830 con 0x7f5558004800 2024-03-06T07:32:30.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.920+0000 7f557179a640 1 -- 172.21.15.1:0/2524666703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5558004800 msgr2=0x7f5558004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.920+0000 7f557179a640 1 --2- 172.21.15.1:0/2524666703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5558004800 0x7f5558004c80 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f556c06d440 tx=0x7f5558005830 comp rx=0 tx=0).stop 2024-03-06T07:32:30.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.920+0000 7f557179a640 1 -- 172.21.15.1:0/2524666703 shutdown_connections 2024-03-06T07:32:30.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.920+0000 7f557179a640 1 --2- 172.21.15.1:0/2524666703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5558004800 0x7f5558004c80 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.920+0000 7f557179a640 1 -- 172.21.15.1:0/2524666703 >> 172.21.15.1:0/2524666703 conn(0x7f556c066d00 msgr2=0x7f556c067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.920+0000 7f557179a640 1 -- 172.21.15.1:0/2524666703 shutdown_connections 2024-03-06T07:32:30.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.920+0000 7f557179a640 1 -- 172.21.15.1:0/2524666703 wait complete. 2024-03-06T07:32:30.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.921+0000 7f557179a640 1 Processor -- start 2024-03-06T07:32:30.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.921+0000 7f557179a640 1 -- start start 2024-03-06T07:32:30.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f557179a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5558004800 0x7f556c1547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f557179a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f556c14c460 0x7f556c147350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f557179a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f556c14c8c0 0x7f556c147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f557179a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c155420 con 0x7f556c14c460 2024-03-06T07:32:30.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f557179a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c0c4c00 con 0x7f5558004800 2024-03-06T07:32:30.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f557179a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556c0af640 con 0x7f556c14c8c0 2024-03-06T07:32:30.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f556a7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f556c14c460 0x7f556c147350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f556a7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f556c14c460 0x7f556c147350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41032/0 (socket says 172.21.15.1:41032) 2024-03-06T07:32:30.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f556a7fc640 1 -- 172.21.15.1:0/94868566 learned_addr learned my addr 172.21.15.1:0/94868566 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:30.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f556b7fe640 1 --2- 172.21.15.1:0/94868566 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f556c14c8c0 0x7f556c147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f556affd640 1 --2- 172.21.15.1:0/94868566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5558004800 0x7f556c1547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f556a7fc640 1 -- 172.21.15.1:0/94868566 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f556c14c8c0 msgr2=0x7f556c147890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f556a7fc640 1 --2- 172.21.15.1:0/94868566 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f556c14c8c0 0x7f556c147890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f556a7fc640 1 -- 172.21.15.1:0/94868566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5558004800 msgr2=0x7f556c1547d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f556a7fc640 1 --2- 172.21.15.1:0/94868566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5558004800 0x7f556c1547d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f556a7fc640 1 -- 172.21.15.1:0/94868566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c149d10 con 0x7f556c14c460 2024-03-06T07:32:30.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f556a7fc640 1 --2- 172.21.15.1:0/94868566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f556c14c460 0x7f556c147350 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f5554005040 tx=0x7f5554001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f55637fe640 1 -- 172.21.15.1:0/94868566 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554041020 con 0x7f556c14c460 2024-03-06T07:32:30.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f55637fe640 1 -- 172.21.15.1:0/94868566 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5554005480 con 0x7f556c14c460 2024-03-06T07:32:30.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f55637fe640 1 -- 172.21.15.1:0/94868566 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554007a30 con 0x7f556c14c460 2024-03-06T07:32:30.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f557179a640 1 -- 172.21.15.1:0/94868566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556c1ea6c0 con 0x7f556c14c460 2024-03-06T07:32:30.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.922+0000 7f557179a640 1 -- 172.21.15.1:0/94868566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556c1eac80 con 0x7f556c14c460 2024-03-06T07:32:30.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.924+0000 7f557179a640 1 -- 172.21.15.1:0/94868566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556c067fd0 con 0x7f556c14c460 2024-03-06T07:32:30.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.924+0000 7f55637fe640 1 -- 172.21.15.1:0/94868566 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5554044020 con 0x7f556c14c460 2024-03-06T07:32:30.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.924+0000 7f55637fe640 1 --2- 172.21.15.1:0/94868566 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f554c06b4d0 0x7f554c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.924+0000 7f55637fe640 1 -- 172.21.15.1:0/94868566 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f55540abec0 con 0x7f556c14c460 2024-03-06T07:32:30.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.924+0000 7f556affd640 1 --2- 172.21.15.1:0/94868566 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f554c06b4d0 0x7f554c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.925+0000 7f556affd640 1 --2- 172.21.15.1:0/94868566 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f554c06b4d0 0x7f554c06d990 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f555c009b80 tx=0x7f555c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:30.929+0000 7f55637fe640 1 -- 172.21.15.1:0/94868566 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f555404b030 con 0x7f556c14c460 2024-03-06T07:32:31.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:31.234+0000 7f557179a640 1 -- 172.21.15.1:0/94868566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f556c0c5b30 con 0x7f556c14c460 2024-03-06T07:32:31.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:31.235+0000 7f55637fe640 1 -- 172.21.15.1:0/94868566 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f555407aa20 con 0x7f556c14c460 2024-03-06T07:32:31.237 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:31.237 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:31.237 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:31.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:31.237+0000 7f557179a640 1 -- 172.21.15.1:0/94868566 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f554c06b4d0 msgr2=0x7f554c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:31.237+0000 7f557179a640 1 --2- 172.21.15.1:0/94868566 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f554c06b4d0 0x7f554c06d990 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f555c009b80 tx=0x7f555c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:31.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:31.237+0000 7f557179a640 1 -- 172.21.15.1:0/94868566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f556c14c460 msgr2=0x7f556c147350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:31.237+0000 7f557179a640 1 --2- 172.21.15.1:0/94868566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f556c14c460 0x7f556c147350 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f5554005040 tx=0x7f5554001120 comp rx=0 tx=0).stop 2024-03-06T07:32:31.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:31.237+0000 7f557179a640 1 -- 172.21.15.1:0/94868566 shutdown_connections 2024-03-06T07:32:31.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:31.237+0000 7f557179a640 1 --2- 172.21.15.1:0/94868566 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f554c06b4d0 0x7f554c06d990 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:31.237+0000 7f557179a640 1 --2- 172.21.15.1:0/94868566 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f556c14c8c0 0x7f556c147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:31.237+0000 7f557179a640 1 --2- 172.21.15.1:0/94868566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f556c14c460 0x7f556c147350 unknown :-1 s=CLOSED pgs=2603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:31.237+0000 7f557179a640 1 --2- 172.21.15.1:0/94868566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5558004800 0x7f556c1547d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:31.237+0000 7f557179a640 1 -- 172.21.15.1:0/94868566 >> 172.21.15.1:0/94868566 conn(0x7f556c066d00 msgr2=0x7f556c152cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:31.237+0000 7f557179a640 1 -- 172.21.15.1:0/94868566 shutdown_connections 2024-03-06T07:32:31.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:31.237+0000 7f557179a640 1 -- 172.21.15.1:0/94868566 wait complete. 2024-03-06T07:32:31.652 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:31.652 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:32.653 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.753+0000 7fe8ac878640 1 Processor -- start 2024-03-06T07:32:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.753+0000 7fe8ac878640 1 -- start start 2024-03-06T07:32:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.754+0000 7fe8ac878640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe8a8152650 0x7fe8a8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.754+0000 7fe8ac878640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a80c2fc0 con 0x7fe8a80c6b10 2024-03-06T07:32:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.754+0000 7fe8ac878640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a8158da0 con 0x7fe8a80bf9d0 2024-03-06T07:32:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.754+0000 7fe8ac878640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a8155800 con 0x7fe8a8152650 2024-03-06T07:32:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.754+0000 7fe8a5d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe8a80c6b10 0x7fe8a8158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48268/0 (socket says 172.21.15.1:48268) 2024-03-06T07:32:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.754+0000 7fe8a5d9b640 1 -- 172.21.15.1:0/803135729 learned_addr learned my addr 172.21.15.1:0/803135729 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.755+0000 7fe8a659c640 1 --2- 172.21.15.1:0/803135729 >> v2:172.21.15.98:3300/0 conn(0x7fe8a8152650 0x7fe8a8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.755+0000 7fe8a559a640 1 -- 172.21.15.1:0/803135729 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142898417 0 0) 0x7fe8a80c2fc0 con 0x7fe8a80c6b10 2024-03-06T07:32:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.755+0000 7fe8a559a640 1 -- 172.21.15.1:0/803135729 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe894003520 con 0x7fe8a80c6b10 2024-03-06T07:32:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.755+0000 7fe8a559a640 1 -- 172.21.15.1:0/803135729 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836304561 0 0) 0x7fe8a8158da0 con 0x7fe8a80bf9d0 2024-03-06T07:32:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.755+0000 7fe8a559a640 1 -- 172.21.15.1:0/803135729 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8a80c2fc0 con 0x7fe8a80bf9d0 2024-03-06T07:32:32.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.755+0000 7fe8a559a640 1 -- 172.21.15.1:0/803135729 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3182162852 0 0) 0x7fe894003520 con 0x7fe8a80c6b10 2024-03-06T07:32:32.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.755+0000 7fe8a659c640 1 -- 172.21.15.1:0/803135729 >> v1:172.21.15.98:6789/0 conn(0x7fe8a80bf9d0 legacy=0x7fe8a80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.755+0000 7fe8a659c640 1 -- 172.21.15.1:0/803135729 >> v1:172.21.15.1:6789/0 conn(0x7fe8a80c6b10 legacy=0x7fe8a8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.755+0000 7fe8a659c640 1 -- 172.21.15.1:0/803135729 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a80c6690 con 0x7fe8a8152650 2024-03-06T07:32:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.756+0000 7fe8a659c640 1 --2- 172.21.15.1:0/803135729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8a8152650 0x7fe8a8155c90 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fe898009d90 tx=0x7fe89802bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43d1cb4975acabe9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.756+0000 7fe8a559a640 1 -- 172.21.15.1:0/803135729 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898039070 con 0x7fe8a8152650 2024-03-06T07:32:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.756+0000 7fe8a559a640 1 -- 172.21.15.1:0/803135729 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89802c8b0 con 0x7fe8a8152650 2024-03-06T07:32:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.756+0000 7fe8a559a640 1 -- 172.21.15.1:0/803135729 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890001670 con 0x7fe8a8152650 2024-03-06T07:32:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.756+0000 7fe8ac878640 1 -- 172.21.15.1:0/803135729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8a8152650 msgr2=0x7fe8a8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.756+0000 7fe8ac878640 1 --2- 172.21.15.1:0/803135729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8a8152650 0x7fe8a8155c90 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fe898009d90 tx=0x7fe89802bd80 comp rx=0 tx=0).stop 2024-03-06T07:32:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.756+0000 7fe8ac878640 1 -- 172.21.15.1:0/803135729 shutdown_connections 2024-03-06T07:32:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.756+0000 7fe8ac878640 1 --2- 172.21.15.1:0/803135729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8a8152650 0x7fe8a8155c90 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.756+0000 7fe8ac878640 1 -- 172.21.15.1:0/803135729 >> 172.21.15.1:0/803135729 conn(0x7fe8a8066ee0 msgr2=0x7fe8a80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.756+0000 7fe8ac878640 1 -- 172.21.15.1:0/803135729 shutdown_connections 2024-03-06T07:32:32.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.756+0000 7fe8ac878640 1 -- 172.21.15.1:0/803135729 wait complete. 2024-03-06T07:32:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.758+0000 7fe8ac878640 1 Processor -- start 2024-03-06T07:32:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.758+0000 7fe8ac878640 1 -- start start 2024-03-06T07:32:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.758+0000 7fe8ac878640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8a80bf9d0 0x7fe8a814d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.758+0000 7fe8ac878640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8a8152650 0x7fe8a8157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.758+0000 7fe8ac878640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8a814a2f0 0x7fe8a814a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.758+0000 7fe8ac878640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a81551b0 con 0x7fe8a80bf9d0 2024-03-06T07:32:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.758+0000 7fe8ac878640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a80c6fa0 con 0x7fe8a8152650 2024-03-06T07:32:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.758+0000 7fe8ac878640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a8155490 con 0x7fe8a814a2f0 2024-03-06T07:32:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.758+0000 7fe8a659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8a80bf9d0 0x7fe8a814d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.758+0000 7fe8a659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8a80bf9d0 0x7fe8a814d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41036/0 (socket says 172.21.15.1:41036) 2024-03-06T07:32:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.758+0000 7fe8a659c640 1 -- 172.21.15.1:0/3171704494 learned_addr learned my addr 172.21.15.1:0/3171704494 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.759+0000 7fe8a5d9b640 1 --2- 172.21.15.1:0/3171704494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8a8152650 0x7fe8a8157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.759+0000 7fe8a6d9d640 1 --2- 172.21.15.1:0/3171704494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8a814a2f0 0x7fe8a814a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.759+0000 7fe8a659c640 1 -- 172.21.15.1:0/3171704494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8a814a2f0 msgr2=0x7fe8a814a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.759+0000 7fe8a659c640 1 --2- 172.21.15.1:0/3171704494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8a814a2f0 0x7fe8a814a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.759+0000 7fe8a659c640 1 -- 172.21.15.1:0/3171704494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8a8152650 msgr2=0x7fe8a8157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.759+0000 7fe8a659c640 1 --2- 172.21.15.1:0/3171704494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8a8152650 0x7fe8a8157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.759+0000 7fe8a659c640 1 -- 172.21.15.1:0/3171704494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a814e110 con 0x7fe8a80bf9d0 2024-03-06T07:32:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.759+0000 7fe8a5d9b640 1 --2- 172.21.15.1:0/3171704494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8a8152650 0x7fe8a8157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.759+0000 7fe8a659c640 1 --2- 172.21.15.1:0/3171704494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8a80bf9d0 0x7fe8a814d770 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7fe898037040 tx=0x7fe898004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.759+0000 7fe89effd640 1 -- 172.21.15.1:0/3171704494 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898039050 con 0x7fe8a80bf9d0 2024-03-06T07:32:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.759+0000 7fe89effd640 1 -- 172.21.15.1:0/3171704494 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89803a040 con 0x7fe8a80bf9d0 2024-03-06T07:32:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.759+0000 7fe8ac878640 1 -- 172.21.15.1:0/3171704494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a81ee5f0 con 0x7fe8a80bf9d0 2024-03-06T07:32:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.759+0000 7fe89effd640 1 -- 172.21.15.1:0/3171704494 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898009450 con 0x7fe8a80bf9d0 2024-03-06T07:32:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.759+0000 7fe8ac878640 1 -- 172.21.15.1:0/3171704494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a81eebb0 con 0x7fe8a80bf9d0 2024-03-06T07:32:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.760+0000 7fe8ac878640 1 -- 172.21.15.1:0/3171704494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a81eb0b0 con 0x7fe8a80bf9d0 2024-03-06T07:32:32.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.761+0000 7fe89effd640 1 -- 172.21.15.1:0/3171704494 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe898003bd0 con 0x7fe8a80bf9d0 2024-03-06T07:32:32.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.761+0000 7fe89effd640 1 --2- 172.21.15.1:0/3171704494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe88806b500 0x7fe88806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.761+0000 7fe89effd640 1 -- 172.21.15.1:0/3171704494 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fe8980ac960 con 0x7fe8a80bf9d0 2024-03-06T07:32:32.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.761+0000 7fe8a5d9b640 1 --2- 172.21.15.1:0/3171704494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe88806b500 0x7fe88806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.762+0000 7fe8a5d9b640 1 --2- 172.21.15.1:0/3171704494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe88806b500 0x7fe88806d9c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fe8a80cb7d0 tx=0x7fe890009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.765+0000 7fe89effd640 1 -- 172.21.15.1:0/3171704494 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8a81eb0b0 con 0x7fe8a80bf9d0 2024-03-06T07:32:32.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.982+0000 7fe8ac878640 1 -- 172.21.15.1:0/3171704494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8a8067fd0 con 0x7fe8a80bf9d0 2024-03-06T07:32:32.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.983+0000 7fe89effd640 1 -- 172.21.15.1:0/3171704494 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fe8a814e110 con 0x7fe8a80bf9d0 2024-03-06T07:32:32.985 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:32.985 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:32.985 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:32.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.985+0000 7fe8ac878640 1 -- 172.21.15.1:0/3171704494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe88806b500 msgr2=0x7fe88806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.985+0000 7fe8ac878640 1 --2- 172.21.15.1:0/3171704494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe88806b500 0x7fe88806d9c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fe8a80cb7d0 tx=0x7fe890009040 comp rx=0 tx=0).stop 2024-03-06T07:32:32.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.985+0000 7fe8ac878640 1 -- 172.21.15.1:0/3171704494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8a80bf9d0 msgr2=0x7fe8a814d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.985+0000 7fe8ac878640 1 --2- 172.21.15.1:0/3171704494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8a80bf9d0 0x7fe8a814d770 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7fe898037040 tx=0x7fe898004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:32.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.985+0000 7fe8ac878640 1 -- 172.21.15.1:0/3171704494 shutdown_connections 2024-03-06T07:32:32.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.985+0000 7fe8ac878640 1 --2- 172.21.15.1:0/3171704494 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe88806b500 0x7fe88806d9c0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.985+0000 7fe8ac878640 1 --2- 172.21.15.1:0/3171704494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8a814a2f0 0x7fe8a814a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.985+0000 7fe8ac878640 1 --2- 172.21.15.1:0/3171704494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8a8152650 0x7fe8a8157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.985+0000 7fe8ac878640 1 --2- 172.21.15.1:0/3171704494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8a80bf9d0 0x7fe8a814d770 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.985+0000 7fe8ac878640 1 -- 172.21.15.1:0/3171704494 >> 172.21.15.1:0/3171704494 conn(0x7fe8a8066ee0 msgr2=0x7fe8a814e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.985+0000 7fe8ac878640 1 -- 172.21.15.1:0/3171704494 shutdown_connections 2024-03-06T07:32:32.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:32.985+0000 7fe8ac878640 1 -- 172.21.15.1:0/3171704494 wait complete. 2024-03-06T07:32:33.334 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:33.337 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:33.403 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:33.403 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:34.404 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:34.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.524+0000 7f70bf725640 1 Processor -- start 2024-03-06T07:32:34.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.525+0000 7f70bf725640 1 -- start start 2024-03-06T07:32:34.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.525+0000 7f70bf725640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70b8152650 0x7f70b8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.525+0000 7f70bf725640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70b80c6b10 0x7f70b8158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.525+0000 7f70bf725640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b80c2fc0 con 0x7f70b80bf9d0 2024-03-06T07:32:34.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.525+0000 7f70bf725640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70b8155800 con 0x7f70b80c6b10 2024-03-06T07:32:34.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.525+0000 7f70bf725640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b80c6690 con 0x7f70b8152650 2024-03-06T07:32:34.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.526+0000 7f70bd4c1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70b8152650 0x7f70b8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.526+0000 7f70bccc0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70b80c6b10 0x7f70b8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.526+0000 7f70bd4c1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70b8152650 0x7f70b8155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:38400/0 (socket says 172.21.15.1:38400) 2024-03-06T07:32:34.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.526+0000 7f70bd4c1640 1 -- 172.21.15.1:0/1894748768 learned_addr learned my addr 172.21.15.1:0/1894748768 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:34.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.526+0000 7f70bccc0640 1 -- 172.21.15.1:0/1894748768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70b8152650 msgr2=0x7f70b8155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:34.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.526+0000 7f70bccc0640 1 --2- 172.21.15.1:0/1894748768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70b8152650 0x7f70b8155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.526+0000 7f70bccc0640 1 -- 172.21.15.1:0/1894748768 >> v1:172.21.15.98:6789/0 conn(0x7f70b80bf9d0 legacy=0x7f70b80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.526+0000 7f70bccc0640 1 -- 172.21.15.1:0/1894748768 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b80c7280 con 0x7f70b80c6b10 2024-03-06T07:32:34.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.526+0000 7f70bd4c1640 1 --2- 172.21.15.1:0/1894748768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70b8152650 0x7f70b8155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:34.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.527+0000 7f70bccc0640 1 --2- 172.21.15.1:0/1894748768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70b80c6b10 0x7f70b8158690 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f70a80097a0 tx=0x7f70a802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f93a076e3c978339 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.527+0000 7f70affff640 1 -- 172.21.15.1:0/1894748768 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a8031780 con 0x7f70b80c6b10 2024-03-06T07:32:34.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.527+0000 7f70affff640 1 -- 172.21.15.1:0/1894748768 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a8037070 con 0x7f70b80c6b10 2024-03-06T07:32:34.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.527+0000 7f70affff640 1 -- 172.21.15.1:0/1894748768 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a8031b40 con 0x7f70b80c6b10 2024-03-06T07:32:34.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.528+0000 7f70bf725640 1 -- 172.21.15.1:0/1894748768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70b80c6b10 msgr2=0x7f70b8158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.528+0000 7f70bf725640 1 --2- 172.21.15.1:0/1894748768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70b80c6b10 0x7f70b8158690 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f70a80097a0 tx=0x7f70a802b820 comp rx=0 tx=0).stop 2024-03-06T07:32:34.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.528+0000 7f70bf725640 1 -- 172.21.15.1:0/1894748768 shutdown_connections 2024-03-06T07:32:34.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.528+0000 7f70bf725640 1 --2- 172.21.15.1:0/1894748768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70b80c6b10 0x7f70b8158690 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.528+0000 7f70bf725640 1 --2- 172.21.15.1:0/1894748768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70b8152650 0x7f70b8155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.528+0000 7f70bf725640 1 -- 172.21.15.1:0/1894748768 >> 172.21.15.1:0/1894748768 conn(0x7f70b8066ee0 msgr2=0x7f70b80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.528+0000 7f70bf725640 1 -- 172.21.15.1:0/1894748768 shutdown_connections 2024-03-06T07:32:34.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.528+0000 7f70bf725640 1 -- 172.21.15.1:0/1894748768 wait complete. 2024-03-06T07:32:34.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.529+0000 7f70bf725640 1 Processor -- start 2024-03-06T07:32:34.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.529+0000 7f70bf725640 1 -- start start 2024-03-06T07:32:34.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.530+0000 7f70bf725640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70b80bf9d0 0x7f70b81425b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.530+0000 7f70bf725640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70b8152650 0x7f70b814e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.530+0000 7f70bf725640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b81420b0 0x7f70b8141390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.530+0000 7f70bf725640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b8155490 con 0x7f70b81420b0 2024-03-06T07:32:34.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.530+0000 7f70bf725640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b81551b0 con 0x7f70b80bf9d0 2024-03-06T07:32:34.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.530+0000 7f70bf725640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b8152a30 con 0x7f70b8152650 2024-03-06T07:32:34.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.530+0000 7f70bdcc2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b81420b0 0x7f70b8141390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.530+0000 7f70bdcc2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b81420b0 0x7f70b8141390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37054/0 (socket says 172.21.15.1:37054) 2024-03-06T07:32:34.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.530+0000 7f70bdcc2640 1 -- 172.21.15.1:0/73111110 learned_addr learned my addr 172.21.15.1:0/73111110 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:34.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.530+0000 7f70bccc0640 1 --2- 172.21.15.1:0/73111110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70b8152650 0x7f70b814e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.530+0000 7f70bd4c1640 1 --2- 172.21.15.1:0/73111110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70b80bf9d0 0x7f70b81425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.531+0000 7f70bdcc2640 1 -- 172.21.15.1:0/73111110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70b8152650 msgr2=0x7f70b814e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.531+0000 7f70bdcc2640 1 --2- 172.21.15.1:0/73111110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70b8152650 0x7f70b814e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.531+0000 7f70bdcc2640 1 -- 172.21.15.1:0/73111110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70b80bf9d0 msgr2=0x7f70b81425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.531+0000 7f70bdcc2640 1 --2- 172.21.15.1:0/73111110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70b80bf9d0 0x7f70b81425b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.531+0000 7f70bdcc2640 1 -- 172.21.15.1:0/73111110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b8143650 con 0x7f70b81420b0 2024-03-06T07:32:34.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.531+0000 7f70bdcc2640 1 --2- 172.21.15.1:0/73111110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b81420b0 0x7f70b8141390 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f70b400c6b0 tx=0x7f70b400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.531+0000 7f70adffb640 1 -- 172.21.15.1:0/73111110 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b4003b70 con 0x7f70b81420b0 2024-03-06T07:32:34.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.531+0000 7f70adffb640 1 -- 172.21.15.1:0/73111110 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b4003d30 con 0x7f70b81420b0 2024-03-06T07:32:34.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.531+0000 7f70bf725640 1 -- 172.21.15.1:0/73111110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70b8144e80 con 0x7f70b81420b0 2024-03-06T07:32:34.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.531+0000 7f70adffb640 1 -- 172.21.15.1:0/73111110 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b4005310 con 0x7f70b81420b0 2024-03-06T07:32:34.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.531+0000 7f70bf725640 1 -- 172.21.15.1:0/73111110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b814eb30 con 0x7f70b81420b0 2024-03-06T07:32:34.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.532+0000 7f70bf725640 1 -- 172.21.15.1:0/73111110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70b81430d0 con 0x7f70b81420b0 2024-03-06T07:32:34.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.533+0000 7f70adffb640 1 -- 172.21.15.1:0/73111110 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70b40054b0 con 0x7f70b81420b0 2024-03-06T07:32:34.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.533+0000 7f70adffb640 1 --2- 172.21.15.1:0/73111110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f709c06b4d0 0x7f709c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.533+0000 7f70adffb640 1 -- 172.21.15.1:0/73111110 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f70b408a150 con 0x7f70b81420b0 2024-03-06T07:32:34.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.534+0000 7f70bd4c1640 1 --2- 172.21.15.1:0/73111110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f709c06b4d0 0x7f709c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.534+0000 7f70bd4c1640 1 --2- 172.21.15.1:0/73111110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f709c06b4d0 0x7f709c06d990 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f70b0006ca0 tx=0x7f70b0009240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.538+0000 7f70adffb640 1 -- 172.21.15.1:0/73111110 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f70b4058780 con 0x7f70b81420b0 2024-03-06T07:32:34.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.773+0000 7f70bf725640 1 -- 172.21.15.1:0/73111110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70b8067fd0 con 0x7f70b81420b0 2024-03-06T07:32:34.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.774+0000 7f70adffb640 1 -- 172.21.15.1:0/73111110 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f70b405c820 con 0x7f70b81420b0 2024-03-06T07:32:34.776 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:34.776 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:34.776 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.776+0000 7f70bf725640 1 -- 172.21.15.1:0/73111110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f709c06b4d0 msgr2=0x7f709c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.776+0000 7f70bf725640 1 --2- 172.21.15.1:0/73111110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f709c06b4d0 0x7f709c06d990 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f70b0006ca0 tx=0x7f70b0009240 comp rx=0 tx=0).stop 2024-03-06T07:32:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.776+0000 7f70bf725640 1 -- 172.21.15.1:0/73111110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b81420b0 msgr2=0x7f70b8141390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.776+0000 7f70bf725640 1 --2- 172.21.15.1:0/73111110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b81420b0 0x7f70b8141390 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f70b400c6b0 tx=0x7f70b400cb70 comp rx=0 tx=0).stop 2024-03-06T07:32:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.776+0000 7f70bf725640 1 -- 172.21.15.1:0/73111110 shutdown_connections 2024-03-06T07:32:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.776+0000 7f70bf725640 1 --2- 172.21.15.1:0/73111110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f709c06b4d0 0x7f709c06d990 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.776+0000 7f70bf725640 1 --2- 172.21.15.1:0/73111110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b81420b0 0x7f70b8141390 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.776+0000 7f70bf725640 1 --2- 172.21.15.1:0/73111110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70b8152650 0x7f70b814e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.776+0000 7f70bf725640 1 --2- 172.21.15.1:0/73111110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70b80bf9d0 0x7f70b81425b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.776+0000 7f70bf725640 1 -- 172.21.15.1:0/73111110 >> 172.21.15.1:0/73111110 conn(0x7f70b8066ee0 msgr2=0x7f70b814fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.776+0000 7f70bf725640 1 -- 172.21.15.1:0/73111110 shutdown_connections 2024-03-06T07:32:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:34.776+0000 7f70bf725640 1 -- 172.21.15.1:0/73111110 wait complete. 2024-03-06T07:32:35.190 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:35.190 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:36.191 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:36.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.290+0000 7fb980e17640 1 Processor -- start 2024-03-06T07:32:36.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.290+0000 7fb980e17640 1 -- start start 2024-03-06T07:32:36.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.291+0000 7fb980e17640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb97c159140 0x7fb97c0c4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.291+0000 7fb980e17640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97c072d00 con 0x7fb97c0c7510 2024-03-06T07:32:36.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.291+0000 7fb980e17640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97c073180 con 0x7fb97c0c69b0 2024-03-06T07:32:36.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.291+0000 7fb980e17640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb97c0c0e80 con 0x7fb97c159140 2024-03-06T07:32:36.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.291+0000 7fb97a59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb97c159140 0x7fb97c0c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.291+0000 7fb97a59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb97c159140 0x7fb97c0c4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37060/0 (socket says 172.21.15.1:37060) 2024-03-06T07:32:36.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.291+0000 7fb97a59c640 1 -- 172.21.15.1:0/3194328094 learned_addr learned my addr 172.21.15.1:0/3194328094 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:36.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.291+0000 7fb97ad9d640 1 --1- 172.21.15.1:0/3194328094 >> v1:172.21.15.1:6789/0 conn(0x7fb97c0c7510 0x7fb97c15fb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34472/0 (socket says 172.21.15.1:34472) 2024-03-06T07:32:36.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.292+0000 7fb97a59c640 1 -- 172.21.15.1:0/3194328094 >> v1:172.21.15.98:6789/0 conn(0x7fb97c0c69b0 legacy=0x7fb97c0c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.292+0000 7fb97a59c640 1 -- 172.21.15.1:0/3194328094 >> v1:172.21.15.1:6789/0 conn(0x7fb97c0c7510 legacy=0x7fb97c15fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.292+0000 7fb97a59c640 1 -- 172.21.15.1:0/3194328094 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c073460 con 0x7fb97c159140 2024-03-06T07:32:36.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.292+0000 7fb97959a640 1 -- 172.21.15.1:0/3194328094 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672670351 0 0) 0x7fb97c072d00 con 0x7fb97c0c7510 2024-03-06T07:32:36.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.292+0000 7fb97a59c640 1 --2- 172.21.15.1:0/3194328094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb97c159140 0x7fb97c0c4490 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7fb96c009870 tx=0x7fb96c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6174bc96e87c2e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.293+0000 7fb97959a640 1 -- 172.21.15.1:0/3194328094 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96c035020 con 0x7fb97c159140 2024-03-06T07:32:36.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.293+0000 7fb97959a640 1 -- 172.21.15.1:0/3194328094 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96c031870 con 0x7fb97c159140 2024-03-06T07:32:36.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.293+0000 7fb97959a640 1 -- 172.21.15.1:0/3194328094 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96c031c40 con 0x7fb97c159140 2024-03-06T07:32:36.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.293+0000 7fb980e17640 1 -- 172.21.15.1:0/3194328094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb97c159140 msgr2=0x7fb97c0c4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.293+0000 7fb980e17640 1 --2- 172.21.15.1:0/3194328094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb97c159140 0x7fb97c0c4490 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7fb96c009870 tx=0x7fb96c02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:36.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.293+0000 7fb980e17640 1 -- 172.21.15.1:0/3194328094 shutdown_connections 2024-03-06T07:32:36.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.293+0000 7fb980e17640 1 --2- 172.21.15.1:0/3194328094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb97c159140 0x7fb97c0c4490 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.293+0000 7fb980e17640 1 -- 172.21.15.1:0/3194328094 >> 172.21.15.1:0/3194328094 conn(0x7fb97c0c7a50 msgr2=0x7fb97c0c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.293+0000 7fb980e17640 1 -- 172.21.15.1:0/3194328094 shutdown_connections 2024-03-06T07:32:36.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.293+0000 7fb980e17640 1 -- 172.21.15.1:0/3194328094 wait complete. 2024-03-06T07:32:36.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.294+0000 7fb980e17640 1 Processor -- start 2024-03-06T07:32:36.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.294+0000 7fb980e17640 1 -- start start 2024-03-06T07:32:36.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.295+0000 7fb980e17640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c0c69b0 0x7fb97c14c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.295+0000 7fb980e17640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb97c0c7510 0x7fb97c14cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.295+0000 7fb980e17640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb97c159140 0x7fb97c1ea8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.295+0000 7fb980e17640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c0c6520 con 0x7fb97c0c7510 2024-03-06T07:32:36.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.295+0000 7fb980e17640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c0c13b0 con 0x7fb97c0c69b0 2024-03-06T07:32:36.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.295+0000 7fb980e17640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c159520 con 0x7fb97c159140 2024-03-06T07:32:36.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.295+0000 7fb979d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb97c0c7510 0x7fb97c14cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.295+0000 7fb979d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb97c0c7510 0x7fb97c14cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37068/0 (socket says 172.21.15.1:37068) 2024-03-06T07:32:36.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.295+0000 7fb979d9b640 1 -- 172.21.15.1:0/1376741245 learned_addr learned my addr 172.21.15.1:0/1376741245 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:36.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.295+0000 7fb97a59c640 1 --2- 172.21.15.1:0/1376741245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c0c69b0 0x7fb97c14c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.295+0000 7fb97ad9d640 1 --2- 172.21.15.1:0/1376741245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb97c159140 0x7fb97c1ea8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.296+0000 7fb979d9b640 1 -- 172.21.15.1:0/1376741245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb97c159140 msgr2=0x7fb97c1ea8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.296+0000 7fb979d9b640 1 --2- 172.21.15.1:0/1376741245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb97c159140 0x7fb97c1ea8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.296+0000 7fb979d9b640 1 -- 172.21.15.1:0/1376741245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c0c69b0 msgr2=0x7fb97c14c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.296+0000 7fb979d9b640 1 --2- 172.21.15.1:0/1376741245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c0c69b0 0x7fb97c14c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.296+0000 7fb979d9b640 1 -- 172.21.15.1:0/1376741245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c0a9ae0 con 0x7fb97c0c7510 2024-03-06T07:32:36.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.296+0000 7fb97a59c640 1 --2- 172.21.15.1:0/1376741245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c0c69b0 0x7fb97c14c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:36.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.296+0000 7fb97ad9d640 1 --2- 172.21.15.1:0/1376741245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb97c159140 0x7fb97c1ea8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:36.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.296+0000 7fb979d9b640 1 --2- 172.21.15.1:0/1376741245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb97c0c7510 0x7fb97c14cd40 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7fb97000c420 tx=0x7fb97000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.296+0000 7fb96affd640 1 -- 172.21.15.1:0/1376741245 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb970007020 con 0x7fb97c0c7510 2024-03-06T07:32:36.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.296+0000 7fb96affd640 1 -- 172.21.15.1:0/1376741245 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9700071c0 con 0x7fb97c0c7510 2024-03-06T07:32:36.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.296+0000 7fb96affd640 1 -- 172.21.15.1:0/1376741245 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb970007590 con 0x7fb97c0c7510 2024-03-06T07:32:36.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.296+0000 7fb980e17640 1 -- 172.21.15.1:0/1376741245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb97c14e6f0 con 0x7fb97c0c7510 2024-03-06T07:32:36.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.296+0000 7fb980e17640 1 -- 172.21.15.1:0/1376741245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb97c14dcb0 con 0x7fb97c0c7510 2024-03-06T07:32:36.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.297+0000 7fb980e17640 1 -- 172.21.15.1:0/1376741245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb97c0686e0 con 0x7fb97c0c7510 2024-03-06T07:32:36.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.297+0000 7fb96affd640 1 -- 172.21.15.1:0/1376741245 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb970019020 con 0x7fb97c0c7510 2024-03-06T07:32:36.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.298+0000 7fb96affd640 1 --2- 172.21.15.1:0/1376741245 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb95c06b500 0x7fb95c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.298+0000 7fb96affd640 1 -- 172.21.15.1:0/1376741245 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fb970089b40 con 0x7fb97c0c7510 2024-03-06T07:32:36.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.298+0000 7fb97a59c640 1 --2- 172.21.15.1:0/1376741245 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb95c06b500 0x7fb95c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.299+0000 7fb97a59c640 1 --2- 172.21.15.1:0/1376741245 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb95c06b500 0x7fb95c06d9c0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fb96c036000 tx=0x7fb96c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.302+0000 7fb96affd640 1 -- 172.21.15.1:0/1376741245 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb97c0a9ae0 con 0x7fb97c0c7510 2024-03-06T07:32:36.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.564+0000 7fb980e17640 1 -- 172.21.15.1:0/1376741245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb97c072f60 con 0x7fb97c0c7510 2024-03-06T07:32:36.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.565+0000 7fb96affd640 1 -- 172.21.15.1:0/1376741245 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fb9700587c0 con 0x7fb97c0c7510 2024-03-06T07:32:36.567 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:36.567 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:36.567 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:36.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.566+0000 7fb980e17640 1 -- 172.21.15.1:0/1376741245 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb95c06b500 msgr2=0x7fb95c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.566+0000 7fb980e17640 1 --2- 172.21.15.1:0/1376741245 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb95c06b500 0x7fb95c06d9c0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fb96c036000 tx=0x7fb96c03c040 comp rx=0 tx=0).stop 2024-03-06T07:32:36.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.566+0000 7fb980e17640 1 -- 172.21.15.1:0/1376741245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb97c0c7510 msgr2=0x7fb97c14cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.566+0000 7fb980e17640 1 --2- 172.21.15.1:0/1376741245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb97c0c7510 0x7fb97c14cd40 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7fb97000c420 tx=0x7fb97000c8e0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.566+0000 7fb980e17640 1 -- 172.21.15.1:0/1376741245 shutdown_connections 2024-03-06T07:32:36.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.566+0000 7fb980e17640 1 --2- 172.21.15.1:0/1376741245 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fb95c06b500 0x7fb95c06d9c0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.566+0000 7fb980e17640 1 --2- 172.21.15.1:0/1376741245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb97c159140 0x7fb97c1ea8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.566+0000 7fb980e17640 1 --2- 172.21.15.1:0/1376741245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb97c0c7510 0x7fb97c14cd40 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.566+0000 7fb980e17640 1 --2- 172.21.15.1:0/1376741245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c0c69b0 0x7fb97c14c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.567+0000 7fb980e17640 1 -- 172.21.15.1:0/1376741245 >> 172.21.15.1:0/1376741245 conn(0x7fb97c0c7a50 msgr2=0x7fb97c06e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.567+0000 7fb980e17640 1 -- 172.21.15.1:0/1376741245 shutdown_connections 2024-03-06T07:32:36.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:36.567+0000 7fb980e17640 1 -- 172.21.15.1:0/1376741245 wait complete. 2024-03-06T07:32:36.982 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:36.982 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:37.983 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:38.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.100+0000 7f9d6af02640 1 Processor -- start 2024-03-06T07:32:38.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.100+0000 7f9d6af02640 1 -- start start 2024-03-06T07:32:38.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.100+0000 7f9d6af02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d640a7f50 0x7f9d64154300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.100+0000 7f9d6af02640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9d640c75a0 0x7f9d640c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.100+0000 7f9d6af02640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d640c8120 con 0x7f9d64150640 2024-03-06T07:32:38.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.100+0000 7f9d6af02640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d64153e30 con 0x7f9d640c75a0 2024-03-06T07:32:38.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.100+0000 7f9d6af02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d640c7b60 con 0x7f9d640a7f50 2024-03-06T07:32:38.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d68c9e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9d64150640 0x7f9d64153720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34474/0 (socket says 172.21.15.1:34474) 2024-03-06T07:32:38.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d68c9e640 1 -- 172.21.15.1:0/1910312501 learned_addr learned my addr 172.21.15.1:0/1910312501 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:38.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d6949f640 1 --2- 172.21.15.1:0/1910312501 >> v2:172.21.15.98:3300/0 conn(0x7f9d640c75a0 0x7f9d640c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d63fff640 1 --2- 172.21.15.1:0/1910312501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d640a7f50 0x7f9d64154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d637fe640 1 -- 172.21.15.1:0/1910312501 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857972938 0 0) 0x7f9d640c8120 con 0x7f9d64150640 2024-03-06T07:32:38.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d637fe640 1 -- 172.21.15.1:0/1910312501 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d54003520 con 0x7f9d64150640 2024-03-06T07:32:38.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d637fe640 1 -- 172.21.15.1:0/1910312501 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 561317219 0 0) 0x7f9d54003520 con 0x7f9d64150640 2024-03-06T07:32:38.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d637fe640 1 -- 172.21.15.1:0/1910312501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d640a7f50 msgr2=0x7f9d64154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d637fe640 1 --2- 172.21.15.1:0/1910312501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d640a7f50 0x7f9d64154300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d637fe640 1 -- 172.21.15.1:0/1910312501 >> v2:172.21.15.98:3300/0 conn(0x7f9d640c75a0 msgr2=0x7f9d640c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d637fe640 1 --2- 172.21.15.1:0/1910312501 >> v2:172.21.15.98:3300/0 conn(0x7f9d640c75a0 0x7f9d640c8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d637fe640 1 -- 172.21.15.1:0/1910312501 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d640c7e40 con 0x7f9d64150640 2024-03-06T07:32:38.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d637fe640 1 -- 172.21.15.1:0/1910312501 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f9d58001670 con 0x7f9d64150640 2024-03-06T07:32:38.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d637fe640 1 -- 172.21.15.1:0/1910312501 >> v1:172.21.15.1:6789/0 conn(0x7f9d64150640 legacy=0x7f9d64153720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d637fe640 1 --2- 172.21.15.1:0/1910312501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d540039b0 0x7f9d54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d637fe640 1 --2- 172.21.15.1:0/1910312501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d54004800 0x7f9d54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d637fe640 1 --2- 172.21.15.1:0/1910312501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d640a7f50 0x7f9d540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d637fe640 1 -- 172.21.15.1:0/1910312501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d64153e30 con 0x7f9d540039b0 2024-03-06T07:32:38.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d637fe640 1 -- 172.21.15.1:0/1910312501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d640c7b60 con 0x7f9d54004800 2024-03-06T07:32:38.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d637fe640 1 -- 172.21.15.1:0/1910312501 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5400ba40 con 0x7f9d640a7f50 2024-03-06T07:32:38.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d68c9e640 1 --2- 172.21.15.1:0/1910312501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d540039b0 0x7f9d54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d6949f640 1 --2- 172.21.15.1:0/1910312501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d640a7f50 0x7f9d540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.101+0000 7f9d63fff640 1 --2- 172.21.15.1:0/1910312501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d54004800 0x7f9d54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.102+0000 7f9d68c9e640 1 -- 172.21.15.1:0/1910312501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d640a7f50 msgr2=0x7f9d540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.102+0000 7f9d68c9e640 1 --2- 172.21.15.1:0/1910312501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d640a7f50 0x7f9d540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.102+0000 7f9d68c9e640 1 -- 172.21.15.1:0/1910312501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d54004800 msgr2=0x7f9d54004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.102+0000 7f9d68c9e640 1 --2- 172.21.15.1:0/1910312501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d54004800 0x7f9d54004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.102+0000 7f9d68c9e640 1 -- 172.21.15.1:0/1910312501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d54003520 con 0x7f9d540039b0 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.103+0000 7f9d68c9e640 1 --2- 172.21.15.1:0/1910312501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d540039b0 0x7f9d54003dd0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800c240 tx=0x7f9d54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c34f0783e76366 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.103+0000 7f9d637fe640 1 -- 172.21.15.1:0/1910312501 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d58007d30 con 0x7f9d540039b0 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.103+0000 7f9d637fe640 1 -- 172.21.15.1:0/1910312501 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d58030070 con 0x7f9d540039b0 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.103+0000 7f9d637fe640 1 -- 172.21.15.1:0/1910312501 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5802cd60 con 0x7f9d540039b0 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.103+0000 7f9d6af02640 1 -- 172.21.15.1:0/1910312501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d540039b0 msgr2=0x7f9d54003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.103+0000 7f9d6af02640 1 --2- 172.21.15.1:0/1910312501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d540039b0 0x7f9d54003dd0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800c240 tx=0x7f9d54005830 comp rx=0 tx=0).stop 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.103+0000 7f9d6949f640 1 -- 172.21.15.1:0/1910312501 reap_dead start 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.104+0000 7f9d6af02640 1 -- 172.21.15.1:0/1910312501 shutdown_connections 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.104+0000 7f9d6af02640 1 -- 172.21.15.1:0/1910312501 >> 172.21.15.1:0/1910312501 conn(0x7f9d6406b3e0 msgr2=0x7f9d6406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.104+0000 7f9d6af02640 1 -- 172.21.15.1:0/1910312501 shutdown_connections 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.104+0000 7f9d6af02640 1 -- 172.21.15.1:0/1910312501 wait complete. 2024-03-06T07:32:38.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.105+0000 7f9d6af02640 1 Processor -- start 2024-03-06T07:32:38.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.105+0000 7f9d6af02640 1 -- start start 2024-03-06T07:32:38.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.105+0000 7f9d6af02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d540039b0 0x7f9d64151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.105+0000 7f9d6af02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d64145fa0 0x7f9d640c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.105+0000 7f9d6af02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d641e89c0 0x7f9d641e8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d6af02640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d64150a20 con 0x7f9d641e89c0 2024-03-06T07:32:38.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d6af02640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d64153200 con 0x7f9d540039b0 2024-03-06T07:32:38.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d6af02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d64071790 con 0x7f9d64145fa0 2024-03-06T07:32:38.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d6949f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d641e89c0 0x7f9d641e8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d6949f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d641e89c0 0x7f9d641e8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37076/0 (socket says 172.21.15.1:37076) 2024-03-06T07:32:38.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d6949f640 1 -- 172.21.15.1:0/2791930803 learned_addr learned my addr 172.21.15.1:0/2791930803 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:38.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d63fff640 1 --2- 172.21.15.1:0/2791930803 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d64145fa0 0x7f9d640c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d68c9e640 1 --2- 172.21.15.1:0/2791930803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d540039b0 0x7f9d64151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d6949f640 1 -- 172.21.15.1:0/2791930803 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d64145fa0 msgr2=0x7f9d640c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d6949f640 1 --2- 172.21.15.1:0/2791930803 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d64145fa0 0x7f9d640c4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d6949f640 1 -- 172.21.15.1:0/2791930803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d540039b0 msgr2=0x7f9d64151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d6949f640 1 --2- 172.21.15.1:0/2791930803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d540039b0 0x7f9d64151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d6949f640 1 -- 172.21.15.1:0/2791930803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d64146420 con 0x7f9d641e89c0 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d63fff640 1 --2- 172.21.15.1:0/2791930803 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d64145fa0 0x7f9d640c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d6949f640 1 --2- 172.21.15.1:0/2791930803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d641e89c0 0x7f9d641e8e40 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f9d640c2b70 tx=0x7f9d5c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d617fa640 1 -- 172.21.15.1:0/2791930803 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c00fe10 con 0x7f9d641e89c0 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d617fa640 1 -- 172.21.15.1:0/2791930803 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5c01d070 con 0x7f9d641e89c0 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d6af02640 1 -- 172.21.15.1:0/2791930803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d641e9380 con 0x7f9d641e89c0 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d617fa640 1 -- 172.21.15.1:0/2791930803 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c015670 con 0x7f9d641e89c0 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.106+0000 7f9d6af02640 1 -- 172.21.15.1:0/2791930803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d641ebf30 con 0x7f9d641e89c0 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.107+0000 7f9d6af02640 1 -- 172.21.15.1:0/2791930803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d64067fd0 con 0x7f9d641e89c0 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.108+0000 7f9d617fa640 1 -- 172.21.15.1:0/2791930803 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d5c015810 con 0x7f9d641e89c0 2024-03-06T07:32:38.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.108+0000 7f9d617fa640 1 --2- 172.21.15.1:0/2791930803 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9d4806b4d0 0x7f9d4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.108+0000 7f9d617fa640 1 -- 172.21.15.1:0/2791930803 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f9d5c08ad80 con 0x7f9d641e89c0 2024-03-06T07:32:38.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.108+0000 7f9d68c9e640 1 --2- 172.21.15.1:0/2791930803 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9d4806b4d0 0x7f9d4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.109+0000 7f9d68c9e640 1 --2- 172.21.15.1:0/2791930803 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9d4806b4d0 0x7f9d4806d990 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f9d58034000 tx=0x7f9d58007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.116+0000 7f9d617fa640 1 -- 172.21.15.1:0/2791930803 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d5c010040 con 0x7f9d641e89c0 2024-03-06T07:32:38.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.337+0000 7f9d6af02640 1 -- 172.21.15.1:0/2791930803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d6406bab0 con 0x7f9d641e89c0 2024-03-06T07:32:38.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.338+0000 7f9d617fa640 1 -- 172.21.15.1:0/2791930803 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f9d5c059970 con 0x7f9d641e89c0 2024-03-06T07:32:38.340 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:38.340 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:38.341 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:38.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.340+0000 7f9d6af02640 1 -- 172.21.15.1:0/2791930803 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9d4806b4d0 msgr2=0x7f9d4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.340+0000 7f9d6af02640 1 --2- 172.21.15.1:0/2791930803 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9d4806b4d0 0x7f9d4806d990 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f9d58034000 tx=0x7f9d58007000 comp rx=0 tx=0).stop 2024-03-06T07:32:38.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.340+0000 7f9d6af02640 1 -- 172.21.15.1:0/2791930803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d641e89c0 msgr2=0x7f9d641e8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.340+0000 7f9d6af02640 1 --2- 172.21.15.1:0/2791930803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d641e89c0 0x7f9d641e8e40 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f9d640c2b70 tx=0x7f9d5c00f580 comp rx=0 tx=0).stop 2024-03-06T07:32:38.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.340+0000 7f9d6af02640 1 -- 172.21.15.1:0/2791930803 shutdown_connections 2024-03-06T07:32:38.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.340+0000 7f9d6af02640 1 --2- 172.21.15.1:0/2791930803 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9d4806b4d0 0x7f9d4806d990 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.340+0000 7f9d6af02640 1 --2- 172.21.15.1:0/2791930803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d641e89c0 0x7f9d641e8e40 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.340+0000 7f9d6af02640 1 --2- 172.21.15.1:0/2791930803 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9d64145fa0 0x7f9d640c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.340+0000 7f9d6af02640 1 --2- 172.21.15.1:0/2791930803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d540039b0 0x7f9d64151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.340+0000 7f9d6af02640 1 -- 172.21.15.1:0/2791930803 >> 172.21.15.1:0/2791930803 conn(0x7f9d6406b3e0 msgr2=0x7f9d640c8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.340+0000 7f9d6af02640 1 -- 172.21.15.1:0/2791930803 shutdown_connections 2024-03-06T07:32:38.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:38.340+0000 7f9d6af02640 1 -- 172.21.15.1:0/2791930803 wait complete. 2024-03-06T07:32:38.756 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:38.756 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:39.757 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:39.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.857+0000 7fdd79615640 1 Processor -- start 2024-03-06T07:32:39.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.857+0000 7fdd79615640 1 -- start start 2024-03-06T07:32:39.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.858+0000 7fdd79615640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdd740c1360 0x7fdd740c17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.858+0000 7fdd79615640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdd74158980 0x7fdd74158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.858+0000 7fdd79615640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd741551b0 con 0x7fdd740c6ce0 2024-03-06T07:32:39.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.858+0000 7fdd79615640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd74155470 con 0x7fdd74158980 2024-03-06T07:32:39.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.858+0000 7fdd79615640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd74155750 con 0x7fdd740c1360 2024-03-06T07:32:39.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.858+0000 7fdd737fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdd74158980 0x7fdd74158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.858+0000 7fdd72ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdd740c6ce0 0x7fdd7413e170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34478/0 (socket says 172.21.15.1:34478) 2024-03-06T07:32:39.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.858+0000 7fdd72ffd640 1 -- 172.21.15.1:0/1961328338 learned_addr learned my addr 172.21.15.1:0/1961328338 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:39.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.858+0000 7fdd727fc640 1 --2- 172.21.15.1:0/1961328338 >> v2:172.21.15.98:3300/0 conn(0x7fdd740c1360 0x7fdd740c17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.858+0000 7fdd737fe640 1 -- 172.21.15.1:0/1961328338 >> v2:172.21.15.98:3300/0 conn(0x7fdd740c1360 msgr2=0x7fdd740c17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.858+0000 7fdd737fe640 1 --2- 172.21.15.1:0/1961328338 >> v2:172.21.15.98:3300/0 conn(0x7fdd740c1360 0x7fdd740c17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.858+0000 7fdd737fe640 1 -- 172.21.15.1:0/1961328338 >> v1:172.21.15.1:6789/0 conn(0x7fdd740c6ce0 legacy=0x7fdd7413e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.858+0000 7fdd737fe640 1 -- 172.21.15.1:0/1961328338 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd74155a30 con 0x7fdd74158980 2024-03-06T07:32:39.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.858+0000 7fdd71ffb640 1 -- 172.21.15.1:0/1961328338 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839066893 0 0) 0x7fdd741551b0 con 0x7fdd740c6ce0 2024-03-06T07:32:39.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.859+0000 7fdd737fe640 1 --2- 172.21.15.1:0/1961328338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd74158980 0x7fdd74158e10 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7fdd740c2bc0 tx=0x7fdd6402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6427867f2d77254 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.860+0000 7fdd71ffb640 1 -- 172.21.15.1:0/1961328338 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64039070 con 0x7fdd74158980 2024-03-06T07:32:39.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.860+0000 7fdd71ffb640 1 -- 172.21.15.1:0/1961328338 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6402cbf0 con 0x7fdd74158980 2024-03-06T07:32:39.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.860+0000 7fdd71ffb640 1 -- 172.21.15.1:0/1961328338 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64034440 con 0x7fdd74158980 2024-03-06T07:32:39.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.860+0000 7fdd79615640 1 -- 172.21.15.1:0/1961328338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd74158980 msgr2=0x7fdd74158e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.860+0000 7fdd79615640 1 --2- 172.21.15.1:0/1961328338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd74158980 0x7fdd74158e10 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7fdd740c2bc0 tx=0x7fdd6402be40 comp rx=0 tx=0).stop 2024-03-06T07:32:39.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.860+0000 7fdd79615640 1 -- 172.21.15.1:0/1961328338 shutdown_connections 2024-03-06T07:32:39.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.860+0000 7fdd79615640 1 --2- 172.21.15.1:0/1961328338 >> v2:172.21.15.98:3300/0 conn(0x7fdd740c1360 0x7fdd740c17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.860+0000 7fdd79615640 1 --2- 172.21.15.1:0/1961328338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd74158980 0x7fdd74158e10 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.860+0000 7fdd79615640 1 -- 172.21.15.1:0/1961328338 >> 172.21.15.1:0/1961328338 conn(0x7fdd7406da30 msgr2=0x7fdd74066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.860+0000 7fdd79615640 1 -- 172.21.15.1:0/1961328338 shutdown_connections 2024-03-06T07:32:39.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.860+0000 7fdd79615640 1 -- 172.21.15.1:0/1961328338 wait complete. 2024-03-06T07:32:39.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.862+0000 7fdd79615640 1 Processor -- start 2024-03-06T07:32:39.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.862+0000 7fdd79615640 1 -- start start 2024-03-06T07:32:39.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.863+0000 7fdd79615640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd740c1360 0x7fdd74157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.863+0000 7fdd79615640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd740c6ce0 0x7fdd74138570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.863+0000 7fdd79615640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd74158980 0x7fdd74154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.863+0000 7fdd79615640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd740c1d30 con 0x7fdd740c1360 2024-03-06T07:32:39.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.863+0000 7fdd79615640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd74155ff0 con 0x7fdd74158980 2024-03-06T07:32:39.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.863+0000 7fdd79615640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7413e880 con 0x7fdd740c6ce0 2024-03-06T07:32:39.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.863+0000 7fdd72ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd740c1360 0x7fdd74157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.863+0000 7fdd72ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd740c1360 0x7fdd74157f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37084/0 (socket says 172.21.15.1:37084) 2024-03-06T07:32:39.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.863+0000 7fdd72ffd640 1 -- 172.21.15.1:0/496298719 learned_addr learned my addr 172.21.15.1:0/496298719 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:39.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.864+0000 7fdd72ffd640 1 -- 172.21.15.1:0/496298719 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd740c6ce0 msgr2=0x7fdd74138570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.864+0000 7fdd72ffd640 1 --2- 172.21.15.1:0/496298719 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd740c6ce0 0x7fdd74138570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.864+0000 7fdd72ffd640 1 -- 172.21.15.1:0/496298719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd74158980 msgr2=0x7fdd74154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.864+0000 7fdd72ffd640 1 --2- 172.21.15.1:0/496298719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd74158980 0x7fdd74154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.864+0000 7fdd72ffd640 1 -- 172.21.15.1:0/496298719 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd74158650 con 0x7fdd740c1360 2024-03-06T07:32:39.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.864+0000 7fdd72ffd640 1 --2- 172.21.15.1:0/496298719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd740c1360 0x7fdd74157f50 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c00c500 tx=0x7fdd6c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.864+0000 7fdd5b7fe640 1 -- 172.21.15.1:0/496298719 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c018070 con 0x7fdd740c1360 2024-03-06T07:32:39.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.864+0000 7fdd5b7fe640 1 -- 172.21.15.1:0/496298719 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6c007560 con 0x7fdd740c1360 2024-03-06T07:32:39.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.864+0000 7fdd5b7fe640 1 -- 172.21.15.1:0/496298719 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c004a20 con 0x7fdd740c1360 2024-03-06T07:32:39.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.864+0000 7fdd79615640 1 -- 172.21.15.1:0/496298719 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd741ea8b0 con 0x7fdd740c1360 2024-03-06T07:32:39.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.864+0000 7fdd79615640 1 -- 172.21.15.1:0/496298719 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd741eae70 con 0x7fdd740c1360 2024-03-06T07:32:39.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.865+0000 7fdd79615640 1 -- 172.21.15.1:0/496298719 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd740686e0 con 0x7fdd740c1360 2024-03-06T07:32:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.866+0000 7fdd5b7fe640 1 -- 172.21.15.1:0/496298719 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd6c01b070 con 0x7fdd740c1360 2024-03-06T07:32:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.866+0000 7fdd5b7fe640 1 --2- 172.21.15.1:0/496298719 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd6006b500 0x7fdd6006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.866+0000 7fdd5b7fe640 1 -- 172.21.15.1:0/496298719 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fdd6c089720 con 0x7fdd740c1360 2024-03-06T07:32:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.866+0000 7fdd727fc640 1 --2- 172.21.15.1:0/496298719 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd6006b500 0x7fdd6006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.867+0000 7fdd727fc640 1 --2- 172.21.15.1:0/496298719 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd6006b500 0x7fdd6006d9c0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fdd68006a90 tx=0x7fdd68006a20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:39.870+0000 7fdd5b7fe640 1 -- 172.21.15.1:0/496298719 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd740686e0 con 0x7fdd740c1360 2024-03-06T07:32:40.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:40.088+0000 7fdd79615640 1 -- 172.21.15.1:0/496298719 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd74067fd0 con 0x7fdd740c1360 2024-03-06T07:32:40.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:40.088+0000 7fdd5b7fe640 1 -- 172.21.15.1:0/496298719 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fdd6c0583a0 con 0x7fdd740c1360 2024-03-06T07:32:40.090 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:40.091 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:40.091 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:40.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:40.090+0000 7fdd79615640 1 -- 172.21.15.1:0/496298719 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd6006b500 msgr2=0x7fdd6006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:40.090+0000 7fdd79615640 1 --2- 172.21.15.1:0/496298719 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd6006b500 0x7fdd6006d9c0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fdd68006a90 tx=0x7fdd68006a20 comp rx=0 tx=0).stop 2024-03-06T07:32:40.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:40.090+0000 7fdd79615640 1 -- 172.21.15.1:0/496298719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd740c1360 msgr2=0x7fdd74157f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:40.090+0000 7fdd79615640 1 --2- 172.21.15.1:0/496298719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd740c1360 0x7fdd74157f50 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c00c500 tx=0x7fdd6c00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:40.090+0000 7fdd79615640 1 -- 172.21.15.1:0/496298719 shutdown_connections 2024-03-06T07:32:40.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:40.090+0000 7fdd79615640 1 --2- 172.21.15.1:0/496298719 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdd6006b500 0x7fdd6006d9c0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:40.090+0000 7fdd79615640 1 --2- 172.21.15.1:0/496298719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd74158980 0x7fdd74154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:40.090+0000 7fdd79615640 1 --2- 172.21.15.1:0/496298719 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdd740c6ce0 0x7fdd74138570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:40.090+0000 7fdd79615640 1 --2- 172.21.15.1:0/496298719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd740c1360 0x7fdd74157f50 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:40.090+0000 7fdd79615640 1 -- 172.21.15.1:0/496298719 >> 172.21.15.1:0/496298719 conn(0x7fdd7406da30 msgr2=0x7fdd7413e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:40.090+0000 7fdd79615640 1 -- 172.21.15.1:0/496298719 shutdown_connections 2024-03-06T07:32:40.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:40.090+0000 7fdd79615640 1 -- 172.21.15.1:0/496298719 wait complete. 2024-03-06T07:32:40.508 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:40.508 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:41.509 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.610+0000 7fc9b5902640 1 Processor -- start 2024-03-06T07:32:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.610+0000 7fc9b5902640 1 -- start start 2024-03-06T07:32:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.611+0000 7fc9b5902640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b0072bd0 0x7fc9b00c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.611+0000 7fc9b5902640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc9b0152b70 0x7fc9b00c3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.611+0000 7fc9b5902640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b0151dd0 con 0x7fc9b00c2b30 2024-03-06T07:32:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.611+0000 7fc9b5902640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9b01526e0 con 0x7fc9b0152b70 2024-03-06T07:32:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.611+0000 7fc9b5902640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b0156190 con 0x7fc9b0072bd0 2024-03-06T07:32:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.611+0000 7fc9af7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc9b0152b70 0x7fc9b00c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.611+0000 7fc9af7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc9b0152b70 0x7fc9b00c3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37092/0 (socket says 172.21.15.1:37092) 2024-03-06T07:32:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.611+0000 7fc9af7fe640 1 -- 172.21.15.1:0/473478902 learned_addr learned my addr 172.21.15.1:0/473478902 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.611+0000 7fc9ae7fc640 1 --2- 172.21.15.1:0/473478902 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b0072bd0 0x7fc9b00c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.611+0000 7fc9af7fe640 1 -- 172.21.15.1:0/473478902 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b0072bd0 msgr2=0x7fc9b00c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.611+0000 7fc9af7fe640 1 --2- 172.21.15.1:0/473478902 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b0072bd0 0x7fc9b00c32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.611+0000 7fc9af7fe640 1 -- 172.21.15.1:0/473478902 >> v1:172.21.15.98:6789/0 conn(0x7fc9b00c2b30 legacy=0x7fc9b00724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.611+0000 7fc9af7fe640 1 -- 172.21.15.1:0/473478902 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b0152090 con 0x7fc9b0152b70 2024-03-06T07:32:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.612+0000 7fc9af7fe640 1 --2- 172.21.15.1:0/473478902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9b0152b70 0x7fc9b00c3830 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009870 tx=0x7fc9a802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19c487e7048c581 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.612+0000 7fc9adffb640 1 -- 172.21.15.1:0/473478902 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8034070 con 0x7fc9b0152b70 2024-03-06T07:32:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.612+0000 7fc9adffb640 1 -- 172.21.15.1:0/473478902 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a8030840 con 0x7fc9b0152b70 2024-03-06T07:32:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.612+0000 7fc9adffb640 1 -- 172.21.15.1:0/473478902 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8030c10 con 0x7fc9b0152b70 2024-03-06T07:32:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.613+0000 7fc9b5902640 1 -- 172.21.15.1:0/473478902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9b0152b70 msgr2=0x7fc9b00c3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.613+0000 7fc9b5902640 1 --2- 172.21.15.1:0/473478902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9b0152b70 0x7fc9b00c3830 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009870 tx=0x7fc9a802b980 comp rx=0 tx=0).stop 2024-03-06T07:32:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.613+0000 7fc9b5902640 1 -- 172.21.15.1:0/473478902 shutdown_connections 2024-03-06T07:32:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.613+0000 7fc9b5902640 1 --2- 172.21.15.1:0/473478902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9b0152b70 0x7fc9b00c3830 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.613+0000 7fc9b5902640 1 --2- 172.21.15.1:0/473478902 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b0072bd0 0x7fc9b00c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.613+0000 7fc9b5902640 1 -- 172.21.15.1:0/473478902 >> 172.21.15.1:0/473478902 conn(0x7fc9b00c6860 msgr2=0x7fc9b00c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.613+0000 7fc9b5902640 1 -- 172.21.15.1:0/473478902 shutdown_connections 2024-03-06T07:32:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.613+0000 7fc9b5902640 1 -- 172.21.15.1:0/473478902 wait complete. 2024-03-06T07:32:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.614+0000 7fc9b5902640 1 Processor -- start 2024-03-06T07:32:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.614+0000 7fc9b5902640 1 -- start start 2024-03-06T07:32:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.615+0000 7fc9b5902640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0072bd0 0x7fc9b01ec180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.615+0000 7fc9b5902640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9b00c2b30 0x7fc9b014c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.615+0000 7fc9b5902640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b0152b70 0x7fc9b0146910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.615+0000 7fc9b5902640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b00c3ef0 con 0x7fc9b00c2b30 2024-03-06T07:32:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.615+0000 7fc9b5902640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b00c3d70 con 0x7fc9b0072bd0 2024-03-06T07:32:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.615+0000 7fc9b5902640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b0155d80 con 0x7fc9b0152b70 2024-03-06T07:32:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.615+0000 7fc9ae7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9b00c2b30 0x7fc9b014c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.615+0000 7fc9ae7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9b00c2b30 0x7fc9b014c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37102/0 (socket says 172.21.15.1:37102) 2024-03-06T07:32:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.615+0000 7fc9ae7fc640 1 -- 172.21.15.1:0/3216528215 learned_addr learned my addr 172.21.15.1:0/3216528215 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.615+0000 7fc9af7fe640 1 --2- 172.21.15.1:0/3216528215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b0152b70 0x7fc9b0146910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.615+0000 7fc9aeffd640 1 --2- 172.21.15.1:0/3216528215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0072bd0 0x7fc9b01ec180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.615+0000 7fc9ae7fc640 1 -- 172.21.15.1:0/3216528215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b0152b70 msgr2=0x7fc9b0146910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.615+0000 7fc9ae7fc640 1 --2- 172.21.15.1:0/3216528215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b0152b70 0x7fc9b0146910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.615+0000 7fc9ae7fc640 1 -- 172.21.15.1:0/3216528215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0072bd0 msgr2=0x7fc9b01ec180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.615+0000 7fc9ae7fc640 1 --2- 172.21.15.1:0/3216528215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0072bd0 0x7fc9b01ec180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.615+0000 7fc9ae7fc640 1 -- 172.21.15.1:0/3216528215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b01e8d60 con 0x7fc9b00c2b30 2024-03-06T07:32:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.615+0000 7fc9ae7fc640 1 --2- 172.21.15.1:0/3216528215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9b00c2b30 0x7fc9b014c530 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7fc99800d7d0 tx=0x7fc99800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.616+0000 7fc9a7fff640 1 -- 172.21.15.1:0/3216528215 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99800b660 con 0x7fc9b00c2b30 2024-03-06T07:32:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.616+0000 7fc9a7fff640 1 -- 172.21.15.1:0/3216528215 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc998019070 con 0x7fc9b00c2b30 2024-03-06T07:32:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.616+0000 7fc9b5902640 1 -- 172.21.15.1:0/3216528215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b01e9040 con 0x7fc9b00c2b30 2024-03-06T07:32:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.616+0000 7fc9a7fff640 1 -- 172.21.15.1:0/3216528215 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99800ba60 con 0x7fc9b00c2b30 2024-03-06T07:32:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.616+0000 7fc9b5902640 1 -- 172.21.15.1:0/3216528215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b01e9600 con 0x7fc9b00c2b30 2024-03-06T07:32:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.617+0000 7fc9b5902640 1 -- 172.21.15.1:0/3216528215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b0067fd0 con 0x7fc9b00c2b30 2024-03-06T07:32:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.617+0000 7fc9a7fff640 1 -- 172.21.15.1:0/3216528215 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc99800bc00 con 0x7fc9b00c2b30 2024-03-06T07:32:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.618+0000 7fc9a7fff640 1 --2- 172.21.15.1:0/3216528215 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc99006b4d0 0x7fc99006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.618+0000 7fc9a7fff640 1 -- 172.21.15.1:0/3216528215 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fc99808a7a0 con 0x7fc9b00c2b30 2024-03-06T07:32:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.618+0000 7fc9aeffd640 1 --2- 172.21.15.1:0/3216528215 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc99006b4d0 0x7fc99006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.619+0000 7fc9aeffd640 1 --2- 172.21.15.1:0/3216528215 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc99006b4d0 0x7fc99006d990 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0001d40 tx=0x7fc9a0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.622+0000 7fc9a7fff640 1 -- 172.21.15.1:0/3216528215 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc998059420 con 0x7fc9b00c2b30 2024-03-06T07:32:41.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.910+0000 7fc9b5902640 1 -- 172.21.15.1:0/3216528215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9b0071790 con 0x7fc9b00c2b30 2024-03-06T07:32:41.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.911+0000 7fc9a7fff640 1 -- 172.21.15.1:0/3216528215 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fc998058d20 con 0x7fc9b00c2b30 2024-03-06T07:32:41.913 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:41.913 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:41.913 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:41.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.913+0000 7fc9b5902640 1 -- 172.21.15.1:0/3216528215 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc99006b4d0 msgr2=0x7fc99006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.913+0000 7fc9b5902640 1 --2- 172.21.15.1:0/3216528215 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc99006b4d0 0x7fc99006d990 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0001d40 tx=0x7fc9a0009040 comp rx=0 tx=0).stop 2024-03-06T07:32:41.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.913+0000 7fc9b5902640 1 -- 172.21.15.1:0/3216528215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9b00c2b30 msgr2=0x7fc9b014c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.913+0000 7fc9b5902640 1 --2- 172.21.15.1:0/3216528215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9b00c2b30 0x7fc9b014c530 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7fc99800d7d0 tx=0x7fc99800dc90 comp rx=0 tx=0).stop 2024-03-06T07:32:41.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.913+0000 7fc9b5902640 1 -- 172.21.15.1:0/3216528215 shutdown_connections 2024-03-06T07:32:41.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.913+0000 7fc9b5902640 1 --2- 172.21.15.1:0/3216528215 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fc99006b4d0 0x7fc99006d990 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.913+0000 7fc9b5902640 1 --2- 172.21.15.1:0/3216528215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9b0152b70 0x7fc9b0146910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.913+0000 7fc9b5902640 1 --2- 172.21.15.1:0/3216528215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9b00c2b30 0x7fc9b014c530 unknown :-1 s=CLOSED pgs=2617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.913+0000 7fc9b5902640 1 --2- 172.21.15.1:0/3216528215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b0072bd0 0x7fc9b01ec180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.913+0000 7fc9b5902640 1 -- 172.21.15.1:0/3216528215 >> 172.21.15.1:0/3216528215 conn(0x7fc9b00c6860 msgr2=0x7fc9b014fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:41.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.913+0000 7fc9b5902640 1 -- 172.21.15.1:0/3216528215 shutdown_connections 2024-03-06T07:32:41.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:41.913+0000 7fc9b5902640 1 -- 172.21.15.1:0/3216528215 wait complete. 2024-03-06T07:32:42.327 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:42.328 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:43.329 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:43.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.428+0000 7f7d849e9640 1 Processor -- start 2024-03-06T07:32:43.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.428+0000 7f7d849e9640 1 -- start start 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.428+0000 7f7d849e9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7d800c6b10 0x7f7d80155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.428+0000 7f7d849e9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d800bf9d0 0x7f7d800c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.428+0000 7f7d849e9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d800c2fc0 con 0x7f7d80152650 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.428+0000 7f7d849e9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d80155800 con 0x7f7d800bf9d0 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.428+0000 7f7d849e9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d80158da0 con 0x7f7d800c6b10 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.428+0000 7f7d7ed9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d800bf9d0 0x7f7d800c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.428+0000 7f7d7ed9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d800bf9d0 0x7f7d800c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37108/0 (socket says 172.21.15.1:37108) 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.428+0000 7f7d7ed9d640 1 -- 172.21.15.1:0/2528089822 learned_addr learned my addr 172.21.15.1:0/2528089822 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.429+0000 7f7d7ed9d640 1 -- 172.21.15.1:0/2528089822 >> v2:172.21.15.98:3300/0 conn(0x7f7d800c6b10 msgr2=0x7f7d80155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.429+0000 7f7d7ed9d640 1 --2- 172.21.15.1:0/2528089822 >> v2:172.21.15.98:3300/0 conn(0x7f7d800c6b10 0x7f7d80155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.429+0000 7f7d7ed9d640 1 -- 172.21.15.1:0/2528089822 >> v1:172.21.15.98:6789/0 conn(0x7f7d80152650 legacy=0x7f7d80158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:43.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.429+0000 7f7d7ed9d640 1 -- 172.21.15.1:0/2528089822 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d800c6690 con 0x7f7d800bf9d0 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.429+0000 7f7d7ed9d640 1 --2- 172.21.15.1:0/2528089822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d800bf9d0 0x7f7d800c1460 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f7d800cb8e0 tx=0x7f7d7802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d58edf9fdb0aa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.430+0000 7f7d7dd9b640 1 -- 172.21.15.1:0/2528089822 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7802c9f0 con 0x7f7d800bf9d0 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.430+0000 7f7d7dd9b640 1 -- 172.21.15.1:0/2528089822 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7802cb90 con 0x7f7d800bf9d0 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.430+0000 7f7d7dd9b640 1 -- 172.21.15.1:0/2528089822 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d780343f0 con 0x7f7d800bf9d0 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.430+0000 7f7d849e9640 1 -- 172.21.15.1:0/2528089822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d800bf9d0 msgr2=0x7f7d800c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.430+0000 7f7d849e9640 1 --2- 172.21.15.1:0/2528089822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d800bf9d0 0x7f7d800c1460 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f7d800cb8e0 tx=0x7f7d7802be50 comp rx=0 tx=0).stop 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.430+0000 7f7d849e9640 1 -- 172.21.15.1:0/2528089822 shutdown_connections 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.430+0000 7f7d849e9640 1 --2- 172.21.15.1:0/2528089822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d800bf9d0 0x7f7d800c1460 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.430+0000 7f7d849e9640 1 --2- 172.21.15.1:0/2528089822 >> v2:172.21.15.98:3300/0 conn(0x7f7d800c6b10 0x7f7d80155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.430+0000 7f7d849e9640 1 -- 172.21.15.1:0/2528089822 >> 172.21.15.1:0/2528089822 conn(0x7f7d80066ee0 msgr2=0x7f7d800672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.430+0000 7f7d849e9640 1 -- 172.21.15.1:0/2528089822 shutdown_connections 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.430+0000 7f7d849e9640 1 -- 172.21.15.1:0/2528089822 wait complete. 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.431+0000 7f7d849e9640 1 Processor -- start 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.431+0000 7f7d849e9640 1 -- start start 2024-03-06T07:32:43.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.432+0000 7f7d849e9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d800bf9d0 0x7f7d80153da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.432+0000 7f7d849e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d80152650 0x7f7d801584c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.432+0000 7f7d849e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d80156bf0 0x7f7d800c5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.432+0000 7f7d849e9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d801551b0 con 0x7f7d800bf9d0 2024-03-06T07:32:43.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.432+0000 7f7d849e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d800c6fa0 con 0x7f7d80152650 2024-03-06T07:32:43.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.432+0000 7f7d849e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d80155490 con 0x7f7d80156bf0 2024-03-06T07:32:43.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.432+0000 7f7d7e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d800bf9d0 0x7f7d80153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.432+0000 7f7d7e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d800bf9d0 0x7f7d80153da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37124/0 (socket says 172.21.15.1:37124) 2024-03-06T07:32:43.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.432+0000 7f7d7e59c640 1 -- 172.21.15.1:0/383617152 learned_addr learned my addr 172.21.15.1:0/383617152 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:43.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.432+0000 7f7d7e59c640 1 -- 172.21.15.1:0/383617152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d80156bf0 msgr2=0x7f7d800c5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.432+0000 7f7d7e59c640 1 --2- 172.21.15.1:0/383617152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d80156bf0 0x7f7d800c5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.433+0000 7f7d7e59c640 1 -- 172.21.15.1:0/383617152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d80152650 msgr2=0x7f7d801584c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.433+0000 7f7d77fff640 1 --2- 172.21.15.1:0/383617152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d80152650 0x7f7d801584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.433+0000 7f7d7e59c640 1 --2- 172.21.15.1:0/383617152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d80152650 0x7f7d801584c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.433+0000 7f7d7e59c640 1 -- 172.21.15.1:0/383617152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d80158ac0 con 0x7f7d800bf9d0 2024-03-06T07:32:43.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.433+0000 7f7d77fff640 1 --2- 172.21.15.1:0/383617152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d80152650 0x7f7d801584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:43.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.433+0000 7f7d7e59c640 1 --2- 172.21.15.1:0/383617152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d800bf9d0 0x7f7d80153da0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000c440 tx=0x7f7d7000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.433+0000 7f7d76ffd640 1 -- 172.21.15.1:0/383617152 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d70017070 con 0x7f7d800bf9d0 2024-03-06T07:32:43.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.433+0000 7f7d76ffd640 1 -- 172.21.15.1:0/383617152 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d700035f0 con 0x7f7d800bf9d0 2024-03-06T07:32:43.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.433+0000 7f7d849e9640 1 -- 172.21.15.1:0/383617152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d8014dbf0 con 0x7f7d800bf9d0 2024-03-06T07:32:43.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.433+0000 7f7d76ffd640 1 -- 172.21.15.1:0/383617152 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d700039e0 con 0x7f7d800bf9d0 2024-03-06T07:32:43.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.433+0000 7f7d849e9640 1 -- 172.21.15.1:0/383617152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d800c6090 con 0x7f7d800bf9d0 2024-03-06T07:32:43.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.434+0000 7f7d849e9640 1 -- 172.21.15.1:0/383617152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d80067fd0 con 0x7f7d800bf9d0 2024-03-06T07:32:43.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.434+0000 7f7d76ffd640 1 -- 172.21.15.1:0/383617152 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d7001a070 con 0x7f7d800bf9d0 2024-03-06T07:32:43.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.435+0000 7f7d76ffd640 1 --2- 172.21.15.1:0/383617152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d6006b500 0x7f7d6006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.435+0000 7f7d76ffd640 1 -- 172.21.15.1:0/383617152 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f7d70089850 con 0x7f7d800bf9d0 2024-03-06T07:32:43.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.435+0000 7f7d77fff640 1 --2- 172.21.15.1:0/383617152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d6006b500 0x7f7d6006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.436+0000 7f7d77fff640 1 --2- 172.21.15.1:0/383617152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d6006b500 0x7f7d6006d9c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f7d68006fd0 tx=0x7f7d68008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.439+0000 7f7d76ffd640 1 -- 172.21.15.1:0/383617152 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d700583b0 con 0x7f7d800bf9d0 2024-03-06T07:32:43.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.660+0000 7f7d849e9640 1 -- 172.21.15.1:0/383617152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d80154dd0 con 0x7f7d800bf9d0 2024-03-06T07:32:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.661+0000 7f7d76ffd640 1 -- 172.21.15.1:0/383617152 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f7d700010c0 con 0x7f7d800bf9d0 2024-03-06T07:32:43.663 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:43.663 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.663+0000 7f7d849e9640 1 -- 172.21.15.1:0/383617152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d6006b500 msgr2=0x7f7d6006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.663+0000 7f7d849e9640 1 --2- 172.21.15.1:0/383617152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d6006b500 0x7f7d6006d9c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f7d68006fd0 tx=0x7f7d68008040 comp rx=0 tx=0).stop 2024-03-06T07:32:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.663+0000 7f7d849e9640 1 -- 172.21.15.1:0/383617152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d800bf9d0 msgr2=0x7f7d80153da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.663+0000 7f7d849e9640 1 --2- 172.21.15.1:0/383617152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d800bf9d0 0x7f7d80153da0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000c440 tx=0x7f7d7000c900 comp rx=0 tx=0).stop 2024-03-06T07:32:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.663+0000 7f7d849e9640 1 -- 172.21.15.1:0/383617152 shutdown_connections 2024-03-06T07:32:43.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.663+0000 7f7d849e9640 1 --2- 172.21.15.1:0/383617152 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7d6006b500 0x7f7d6006d9c0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.663+0000 7f7d849e9640 1 --2- 172.21.15.1:0/383617152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d80156bf0 0x7f7d800c5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.663+0000 7f7d849e9640 1 --2- 172.21.15.1:0/383617152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d80152650 0x7f7d801584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.663+0000 7f7d849e9640 1 --2- 172.21.15.1:0/383617152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d800bf9d0 0x7f7d80153da0 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.663+0000 7f7d849e9640 1 -- 172.21.15.1:0/383617152 >> 172.21.15.1:0/383617152 conn(0x7f7d80066ee0 msgr2=0x7f7d800672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.663+0000 7f7d849e9640 1 -- 172.21.15.1:0/383617152 shutdown_connections 2024-03-06T07:32:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:43.663+0000 7f7d849e9640 1 -- 172.21.15.1:0/383617152 wait complete. 2024-03-06T07:32:44.078 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:44.078 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:45.079 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.179+0000 7fdc70a5c640 1 Processor -- start 2024-03-06T07:32:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.179+0000 7fdc70a5c640 1 -- start start 2024-03-06T07:32:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.179+0000 7fdc70a5c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdc6c0c3b00 0x7fdc6c072780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.179+0000 7fdc70a5c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdc6c158e20 0x7fdc6c159280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.179+0000 7fdc70a5c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c071b10 con 0x7fdc6c0c4a50 2024-03-06T07:32:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.179+0000 7fdc70a5c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc6c0c29b0 con 0x7fdc6c158e20 2024-03-06T07:32:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.179+0000 7fdc70a5c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc6c0c2f70 con 0x7fdc6c0c3b00 2024-03-06T07:32:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.180+0000 7fdc69d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdc6c158e20 0x7fdc6c159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.180+0000 7fdc69d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdc6c158e20 0x7fdc6c159280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41438/0 (socket says 172.21.15.1:41438) 2024-03-06T07:32:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.180+0000 7fdc69d9b640 1 -- 172.21.15.1:0/3691748408 learned_addr learned my addr 172.21.15.1:0/3691748408 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.180+0000 7fdc6ad9d640 1 --1- 172.21.15.1:0/3691748408 >> v1:172.21.15.1:6789/0 conn(0x7fdc6c0c4a50 0x7fdc6c0c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36738/0 (socket says 172.21.15.1:36738) 2024-03-06T07:32:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.180+0000 7fdc6a59c640 1 --2- 172.21.15.1:0/3691748408 >> v2:172.21.15.98:3300/0 conn(0x7fdc6c0c3b00 0x7fdc6c072780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.180+0000 7fdc69d9b640 1 -- 172.21.15.1:0/3691748408 >> v2:172.21.15.98:3300/0 conn(0x7fdc6c0c3b00 msgr2=0x7fdc6c072780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.180+0000 7fdc69d9b640 1 --2- 172.21.15.1:0/3691748408 >> v2:172.21.15.98:3300/0 conn(0x7fdc6c0c3b00 0x7fdc6c072780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.180+0000 7fdc69d9b640 1 -- 172.21.15.1:0/3691748408 >> v1:172.21.15.1:6789/0 conn(0x7fdc6c0c4a50 legacy=0x7fdc6c0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.180+0000 7fdc69d9b640 1 -- 172.21.15.1:0/3691748408 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6c1597c0 con 0x7fdc6c158e20 2024-03-06T07:32:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.180+0000 7fdc6959a640 1 -- 172.21.15.1:0/3691748408 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592124936 0 0) 0x7fdc6c071b10 con 0x7fdc6c0c4a50 2024-03-06T07:32:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.181+0000 7fdc69d9b640 1 --2- 172.21.15.1:0/3691748408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc6c158e20 0x7fdc6c159280 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c071860 tx=0x7fdc6002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef1aea75e400a90 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.181+0000 7fdc6959a640 1 -- 172.21.15.1:0/3691748408 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc60035020 con 0x7fdc6c158e20 2024-03-06T07:32:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.181+0000 7fdc6959a640 1 -- 172.21.15.1:0/3691748408 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc60031870 con 0x7fdc6c158e20 2024-03-06T07:32:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.181+0000 7fdc6959a640 1 -- 172.21.15.1:0/3691748408 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc60031c40 con 0x7fdc6c158e20 2024-03-06T07:32:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.182+0000 7fdc70a5c640 1 -- 172.21.15.1:0/3691748408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc6c158e20 msgr2=0x7fdc6c159280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.182+0000 7fdc70a5c640 1 --2- 172.21.15.1:0/3691748408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc6c158e20 0x7fdc6c159280 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c071860 tx=0x7fdc6002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.182+0000 7fdc70a5c640 1 -- 172.21.15.1:0/3691748408 shutdown_connections 2024-03-06T07:32:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.182+0000 7fdc70a5c640 1 --2- 172.21.15.1:0/3691748408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc6c158e20 0x7fdc6c159280 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.182+0000 7fdc70a5c640 1 --2- 172.21.15.1:0/3691748408 >> v2:172.21.15.98:3300/0 conn(0x7fdc6c0c3b00 0x7fdc6c072780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.182+0000 7fdc70a5c640 1 -- 172.21.15.1:0/3691748408 >> 172.21.15.1:0/3691748408 conn(0x7fdc6c0c6f40 msgr2=0x7fdc6c0c7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.182+0000 7fdc70a5c640 1 -- 172.21.15.1:0/3691748408 shutdown_connections 2024-03-06T07:32:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.182+0000 7fdc70a5c640 1 -- 172.21.15.1:0/3691748408 wait complete. 2024-03-06T07:32:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.184+0000 7fdc70a5c640 1 Processor -- start 2024-03-06T07:32:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.184+0000 7fdc70a5c640 1 -- start start 2024-03-06T07:32:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.185+0000 7fdc70a5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc6c0c3b00 0x7fdc6c14f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.185+0000 7fdc70a5c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc6c0c4a50 0x7fdc6c1eda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.185+0000 7fdc70a5c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc6c158e20 0x7fdc6c1edf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.185+0000 7fdc70a5c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c0c2c00 con 0x7fdc6c0c4a50 2024-03-06T07:32:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.185+0000 7fdc70a5c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c0720b0 con 0x7fdc6c0c3b00 2024-03-06T07:32:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.185+0000 7fdc70a5c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c0c55b0 con 0x7fdc6c158e20 2024-03-06T07:32:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.185+0000 7fdc69d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc6c0c4a50 0x7fdc6c1eda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.185+0000 7fdc69d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc6c0c4a50 0x7fdc6c1eda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41446/0 (socket says 172.21.15.1:41446) 2024-03-06T07:32:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.185+0000 7fdc69d9b640 1 -- 172.21.15.1:0/1067963736 learned_addr learned my addr 172.21.15.1:0/1067963736 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.185+0000 7fdc6a59c640 1 --2- 172.21.15.1:0/1067963736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc6c0c3b00 0x7fdc6c14f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.186+0000 7fdc6ad9d640 1 --2- 172.21.15.1:0/1067963736 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc6c158e20 0x7fdc6c1edf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.186+0000 7fdc69d9b640 1 -- 172.21.15.1:0/1067963736 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc6c158e20 msgr2=0x7fdc6c1edf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.186+0000 7fdc69d9b640 1 --2- 172.21.15.1:0/1067963736 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc6c158e20 0x7fdc6c1edf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.186+0000 7fdc69d9b640 1 -- 172.21.15.1:0/1067963736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc6c0c3b00 msgr2=0x7fdc6c14f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.186+0000 7fdc69d9b640 1 --2- 172.21.15.1:0/1067963736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc6c0c3b00 0x7fdc6c14f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.186+0000 7fdc69d9b640 1 -- 172.21.15.1:0/1067963736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6c1ee970 con 0x7fdc6c0c4a50 2024-03-06T07:32:45.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.186+0000 7fdc6ad9d640 1 --2- 172.21.15.1:0/1067963736 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc6c158e20 0x7fdc6c1edf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:45.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.186+0000 7fdc6a59c640 1 --2- 172.21.15.1:0/1067963736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc6c0c3b00 0x7fdc6c14f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:45.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.186+0000 7fdc69d9b640 1 --2- 172.21.15.1:0/1067963736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc6c0c4a50 0x7fdc6c1eda30 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fdc6002b950 tx=0x7fdc600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.186+0000 7fdc5affd640 1 -- 172.21.15.1:0/1067963736 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc60040070 con 0x7fdc6c0c4a50 2024-03-06T07:32:45.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.186+0000 7fdc5affd640 1 -- 172.21.15.1:0/1067963736 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc60030d00 con 0x7fdc6c0c4a50 2024-03-06T07:32:45.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.186+0000 7fdc70a5c640 1 -- 172.21.15.1:0/1067963736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc6c0a92c0 con 0x7fdc6c0c4a50 2024-03-06T07:32:45.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.186+0000 7fdc5affd640 1 -- 172.21.15.1:0/1067963736 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc60035040 con 0x7fdc6c0c4a50 2024-03-06T07:32:45.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.186+0000 7fdc70a5c640 1 -- 172.21.15.1:0/1067963736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc6c0a9560 con 0x7fdc6c0c4a50 2024-03-06T07:32:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.187+0000 7fdc70a5c640 1 -- 172.21.15.1:0/1067963736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6c067fd0 con 0x7fdc6c0c4a50 2024-03-06T07:32:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.188+0000 7fdc5affd640 1 -- 172.21.15.1:0/1067963736 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc6004b060 con 0x7fdc6c0c4a50 2024-03-06T07:32:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.188+0000 7fdc5affd640 1 --2- 172.21.15.1:0/1067963736 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdc4c06b4d0 0x7fdc4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.188+0000 7fdc5affd640 1 -- 172.21.15.1:0/1067963736 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fdc600ac130 con 0x7fdc6c0c4a50 2024-03-06T07:32:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.188+0000 7fdc6a59c640 1 --2- 172.21.15.1:0/1067963736 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdc4c06b4d0 0x7fdc4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.189+0000 7fdc6a59c640 1 --2- 172.21.15.1:0/1067963736 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdc4c06b4d0 0x7fdc4c06d990 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c006fd0 tx=0x7fdc5c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.192+0000 7fdc5affd640 1 -- 172.21.15.1:0/1067963736 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc6007adb0 con 0x7fdc6c0c4a50 2024-03-06T07:32:45.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.410+0000 7fdc70a5c640 1 -- 172.21.15.1:0/1067963736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc6c0ab550 con 0x7fdc6c0c4a50 2024-03-06T07:32:45.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.410+0000 7fdc5affd640 1 -- 172.21.15.1:0/1067963736 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fdc6007a6b0 con 0x7fdc6c0c4a50 2024-03-06T07:32:45.412 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:45.413 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:45.413 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:45.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.412+0000 7fdc70a5c640 1 -- 172.21.15.1:0/1067963736 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdc4c06b4d0 msgr2=0x7fdc4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.412+0000 7fdc70a5c640 1 --2- 172.21.15.1:0/1067963736 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdc4c06b4d0 0x7fdc4c06d990 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c006fd0 tx=0x7fdc5c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:45.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.412+0000 7fdc70a5c640 1 -- 172.21.15.1:0/1067963736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc6c0c4a50 msgr2=0x7fdc6c1eda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.412+0000 7fdc70a5c640 1 --2- 172.21.15.1:0/1067963736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc6c0c4a50 0x7fdc6c1eda30 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fdc6002b950 tx=0x7fdc600046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.412+0000 7fdc70a5c640 1 -- 172.21.15.1:0/1067963736 shutdown_connections 2024-03-06T07:32:45.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.412+0000 7fdc70a5c640 1 --2- 172.21.15.1:0/1067963736 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fdc4c06b4d0 0x7fdc4c06d990 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.412+0000 7fdc70a5c640 1 --2- 172.21.15.1:0/1067963736 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdc6c158e20 0x7fdc6c1edf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.412+0000 7fdc70a5c640 1 --2- 172.21.15.1:0/1067963736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc6c0c4a50 0x7fdc6c1eda30 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.412+0000 7fdc70a5c640 1 --2- 172.21.15.1:0/1067963736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc6c0c3b00 0x7fdc6c14f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.412+0000 7fdc70a5c640 1 -- 172.21.15.1:0/1067963736 >> 172.21.15.1:0/1067963736 conn(0x7fdc6c0c6f40 msgr2=0x7fdc6c0c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.413+0000 7fdc70a5c640 1 -- 172.21.15.1:0/1067963736 shutdown_connections 2024-03-06T07:32:45.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:45.413+0000 7fdc70a5c640 1 -- 172.21.15.1:0/1067963736 wait complete. 2024-03-06T07:32:45.828 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:45.828 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:46.830 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:46.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.930+0000 7f66531e2640 1 Processor -- start 2024-03-06T07:32:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.930+0000 7f66531e2640 1 -- start start 2024-03-06T07:32:46.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.930+0000 7f66531e2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f664c152650 0x7f664c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.930+0000 7f66531e2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f664c0bf9d0 0x7f664c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.930+0000 7f66531e2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c0c2fc0 con 0x7f664c0c6b10 2024-03-06T07:32:46.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.930+0000 7f66531e2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f664c155800 con 0x7f664c0bf9d0 2024-03-06T07:32:46.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.930+0000 7f66531e2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f664c158da0 con 0x7f664c152650 2024-03-06T07:32:46.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.930+0000 7f665177f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f664c0bf9d0 0x7f664c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.931+0000 7f665177f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f664c0bf9d0 0x7f664c0c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41452/0 (socket says 172.21.15.1:41452) 2024-03-06T07:32:46.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.931+0000 7f665177f640 1 -- 172.21.15.1:0/4105327265 learned_addr learned my addr 172.21.15.1:0/4105327265 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:46.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.931+0000 7f664bfff640 1 --1- 172.21.15.1:0/4105327265 >> v1:172.21.15.1:6789/0 conn(0x7f664c0c6b10 0x7f664c158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36740/0 (socket says 172.21.15.1:36740) 2024-03-06T07:32:46.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.931+0000 7f665177f640 1 -- 172.21.15.1:0/4105327265 >> v2:172.21.15.98:3300/0 conn(0x7f664c152650 msgr2=0x7f664c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.931+0000 7f6650f7e640 1 --2- 172.21.15.1:0/4105327265 >> v2:172.21.15.98:3300/0 conn(0x7f664c152650 0x7f664c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.931+0000 7f664b7fe640 1 -- 172.21.15.1:0/4105327265 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4240835586 0 0) 0x7f664c0c2fc0 con 0x7f664c0c6b10 2024-03-06T07:32:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.931+0000 7f665177f640 1 --2- 172.21.15.1:0/4105327265 >> v2:172.21.15.98:3300/0 conn(0x7f664c152650 0x7f664c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.931+0000 7f665177f640 1 -- 172.21.15.1:0/4105327265 >> v1:172.21.15.1:6789/0 conn(0x7f664c0c6b10 legacy=0x7f664c158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.931+0000 7f665177f640 1 -- 172.21.15.1:0/4105327265 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c0c6690 con 0x7f664c0bf9d0 2024-03-06T07:32:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.931+0000 7f6650f7e640 1 --2- 172.21.15.1:0/4105327265 >> v2:172.21.15.98:3300/0 conn(0x7f664c152650 0x7f664c155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.932+0000 7f665177f640 1 --2- 172.21.15.1:0/4105327265 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c0bf9d0 0x7f664c0c1460 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7f664c0cac90 tx=0x7f664402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c2cc0b76b8ef047 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.932+0000 7f664b7fe640 1 -- 172.21.15.1:0/4105327265 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664402ca80 con 0x7f664c0bf9d0 2024-03-06T07:32:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.932+0000 7f664b7fe640 1 -- 172.21.15.1:0/4105327265 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664402cc20 con 0x7f664c0bf9d0 2024-03-06T07:32:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.932+0000 7f664b7fe640 1 -- 172.21.15.1:0/4105327265 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66440343f0 con 0x7f664c0bf9d0 2024-03-06T07:32:46.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.932+0000 7f66531e2640 1 -- 172.21.15.1:0/4105327265 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c0bf9d0 msgr2=0x7f664c0c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.932+0000 7f66531e2640 1 --2- 172.21.15.1:0/4105327265 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c0bf9d0 0x7f664c0c1460 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7f664c0cac90 tx=0x7f664402bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.932+0000 7f66531e2640 1 -- 172.21.15.1:0/4105327265 shutdown_connections 2024-03-06T07:32:46.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.932+0000 7f66531e2640 1 --2- 172.21.15.1:0/4105327265 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c0bf9d0 0x7f664c0c1460 unknown :-1 s=CLOSED pgs=2624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.932+0000 7f66531e2640 1 --2- 172.21.15.1:0/4105327265 >> v2:172.21.15.98:3300/0 conn(0x7f664c152650 0x7f664c155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.932+0000 7f66531e2640 1 -- 172.21.15.1:0/4105327265 >> 172.21.15.1:0/4105327265 conn(0x7f664c066ee0 msgr2=0x7f664c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.932+0000 7f66531e2640 1 -- 172.21.15.1:0/4105327265 shutdown_connections 2024-03-06T07:32:46.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.932+0000 7f66531e2640 1 -- 172.21.15.1:0/4105327265 wait complete. 2024-03-06T07:32:46.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.933+0000 7f66531e2640 1 Processor -- start 2024-03-06T07:32:46.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.934+0000 7f66531e2640 1 -- start start 2024-03-06T07:32:46.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.934+0000 7f66531e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f664c0bf9d0 0x7f664c14b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.934+0000 7f66531e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f664c152650 0x7f664c1e1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.934+0000 7f66531e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c1578b0 0x7f664c157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.934+0000 7f66531e2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c1551b0 con 0x7f664c1578b0 2024-03-06T07:32:46.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.934+0000 7f66531e2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c0c6fa0 con 0x7f664c0bf9d0 2024-03-06T07:32:46.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.934+0000 7f66531e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664c155490 con 0x7f664c152650 2024-03-06T07:32:46.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.934+0000 7f665177f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c1578b0 0x7f664c157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.934+0000 7f665177f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c1578b0 0x7f664c157d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41456/0 (socket says 172.21.15.1:41456) 2024-03-06T07:32:46.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.934+0000 7f665177f640 1 -- 172.21.15.1:0/4117876527 learned_addr learned my addr 172.21.15.1:0/4117876527 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:46.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.935+0000 7f664bfff640 1 --2- 172.21.15.1:0/4117876527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f664c152650 0x7f664c1e1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.935+0000 7f6650f7e640 1 --2- 172.21.15.1:0/4117876527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f664c0bf9d0 0x7f664c14b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.935+0000 7f665177f640 1 -- 172.21.15.1:0/4117876527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f664c152650 msgr2=0x7f664c1e1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.935+0000 7f665177f640 1 --2- 172.21.15.1:0/4117876527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f664c152650 0x7f664c1e1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.935+0000 7f665177f640 1 -- 172.21.15.1:0/4117876527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f664c0bf9d0 msgr2=0x7f664c14b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.935+0000 7f665177f640 1 --2- 172.21.15.1:0/4117876527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f664c0bf9d0 0x7f664c14b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.935+0000 7f665177f640 1 -- 172.21.15.1:0/4117876527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c154040 con 0x7f664c1578b0 2024-03-06T07:32:46.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.935+0000 7f665177f640 1 --2- 172.21.15.1:0/4117876527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c1578b0 0x7f664c157d30 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f6644009f40 tx=0x7f6644004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.935+0000 7f66497fa640 1 -- 172.21.15.1:0/4117876527 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6644042030 con 0x7f664c1578b0 2024-03-06T07:32:46.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.935+0000 7f66497fa640 1 -- 172.21.15.1:0/4117876527 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664403e450 con 0x7f664c1578b0 2024-03-06T07:32:46.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.935+0000 7f66531e2640 1 -- 172.21.15.1:0/4117876527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664c154320 con 0x7f664c1578b0 2024-03-06T07:32:46.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.935+0000 7f66497fa640 1 -- 172.21.15.1:0/4117876527 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664403e660 con 0x7f664c1578b0 2024-03-06T07:32:46.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.935+0000 7f66531e2640 1 -- 172.21.15.1:0/4117876527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664c1edf40 con 0x7f664c1578b0 2024-03-06T07:32:46.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.936+0000 7f66531e2640 1 -- 172.21.15.1:0/4117876527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664c067fd0 con 0x7f664c1578b0 2024-03-06T07:32:46.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.937+0000 7f66497fa640 1 -- 172.21.15.1:0/4117876527 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6644045070 con 0x7f664c1578b0 2024-03-06T07:32:46.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.938+0000 7f66497fa640 1 --2- 172.21.15.1:0/4117876527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f662c06b4d0 0x7f662c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.938+0000 7f66497fa640 1 -- 172.21.15.1:0/4117876527 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f66440ac820 con 0x7f664c1578b0 2024-03-06T07:32:46.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.938+0000 7f6650f7e640 1 --2- 172.21.15.1:0/4117876527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f662c06b4d0 0x7f662c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.938+0000 7f6650f7e640 1 --2- 172.21.15.1:0/4117876527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f662c06b4d0 0x7f662c06d990 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f664c157860 tx=0x7f6634008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:46.944+0000 7f66497fa640 1 -- 172.21.15.1:0/4117876527 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f664404c070 con 0x7f664c1578b0 2024-03-06T07:32:47.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:47.166+0000 7f66531e2640 1 -- 172.21.15.1:0/4117876527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f664c0b06f0 con 0x7f664c1578b0 2024-03-06T07:32:47.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:47.166+0000 7f66497fa640 1 -- 172.21.15.1:0/4117876527 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f664407ae50 con 0x7f664c1578b0 2024-03-06T07:32:47.169 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:47.169 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:47.169 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:47.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:47.168+0000 7f66531e2640 1 -- 172.21.15.1:0/4117876527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f662c06b4d0 msgr2=0x7f662c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:47.168+0000 7f66531e2640 1 --2- 172.21.15.1:0/4117876527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f662c06b4d0 0x7f662c06d990 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f664c157860 tx=0x7f6634008040 comp rx=0 tx=0).stop 2024-03-06T07:32:47.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:47.168+0000 7f66531e2640 1 -- 172.21.15.1:0/4117876527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c1578b0 msgr2=0x7f664c157d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:47.168+0000 7f66531e2640 1 --2- 172.21.15.1:0/4117876527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c1578b0 0x7f664c157d30 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f6644009f40 tx=0x7f6644004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:47.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:47.168+0000 7f66531e2640 1 -- 172.21.15.1:0/4117876527 shutdown_connections 2024-03-06T07:32:47.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:47.168+0000 7f66531e2640 1 --2- 172.21.15.1:0/4117876527 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f662c06b4d0 0x7f662c06d990 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:47.168+0000 7f66531e2640 1 --2- 172.21.15.1:0/4117876527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c1578b0 0x7f664c157d30 unknown :-1 s=CLOSED pgs=2625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:47.168+0000 7f66531e2640 1 --2- 172.21.15.1:0/4117876527 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f664c152650 0x7f664c1e1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:47.168+0000 7f66531e2640 1 --2- 172.21.15.1:0/4117876527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f664c0bf9d0 0x7f664c14b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:47.168+0000 7f66531e2640 1 -- 172.21.15.1:0/4117876527 >> 172.21.15.1:0/4117876527 conn(0x7f664c066ee0 msgr2=0x7f664c0c5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:47.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:47.168+0000 7f66531e2640 1 -- 172.21.15.1:0/4117876527 shutdown_connections 2024-03-06T07:32:47.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:47.168+0000 7f66531e2640 1 -- 172.21.15.1:0/4117876527 wait complete. 2024-03-06T07:32:47.583 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:47.584 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:48.585 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.685+0000 7f120dfbc640 1 Processor -- start 2024-03-06T07:32:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.685+0000 7f120dfbc640 1 -- start start 2024-03-06T07:32:48.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.686+0000 7f120dfbc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f12080c6b10 0x7f1208155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.686+0000 7f120dfbc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12080c2fc0 con 0x7f1208152650 2024-03-06T07:32:48.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.686+0000 7f120dfbc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1208158da0 con 0x7f12080bf9d0 2024-03-06T07:32:48.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.686+0000 7f120dfbc640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1208155800 con 0x7f12080c6b10 2024-03-06T07:32:48.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.686+0000 7f1206ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f12080c6b10 0x7f1208155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.686+0000 7f12077fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1208152650 0x7f1208158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36748/0 (socket says 172.21.15.1:36748) 2024-03-06T07:32:48.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.686+0000 7f12077fe640 1 -- 172.21.15.1:0/1711851025 learned_addr learned my addr 172.21.15.1:0/1711851025 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:48.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.686+0000 7f1206ffd640 1 -- 172.21.15.1:0/1711851025 >> v1:172.21.15.98:6789/0 conn(0x7f12080bf9d0 legacy=0x7f12080c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.686+0000 7f12067fc640 1 -- 172.21.15.1:0/1711851025 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078302143 0 0) 0x7f12080c2fc0 con 0x7f1208152650 2024-03-06T07:32:48.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.686+0000 7f1206ffd640 1 -- 172.21.15.1:0/1711851025 >> v1:172.21.15.1:6789/0 conn(0x7f1208152650 legacy=0x7f1208158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.686+0000 7f1206ffd640 1 -- 172.21.15.1:0/1711851025 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12080c6690 con 0x7f12080c6b10 2024-03-06T07:32:48.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.687+0000 7f1206ffd640 1 --2- 172.21.15.1:0/1711851025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12080c6b10 0x7f1208155c90 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f11fc0097a0 tx=0x7f11fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=242ec0fae244a052 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.687+0000 7f12067fc640 1 -- 172.21.15.1:0/1711851025 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc035020 con 0x7f12080c6b10 2024-03-06T07:32:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.687+0000 7f12067fc640 1 -- 172.21.15.1:0/1711851025 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc031830 con 0x7f12080c6b10 2024-03-06T07:32:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.687+0000 7f12067fc640 1 -- 172.21.15.1:0/1711851025 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc031c00 con 0x7f12080c6b10 2024-03-06T07:32:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.688+0000 7f120dfbc640 1 -- 172.21.15.1:0/1711851025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12080c6b10 msgr2=0x7f1208155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.688+0000 7f120dfbc640 1 --2- 172.21.15.1:0/1711851025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12080c6b10 0x7f1208155c90 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f11fc0097a0 tx=0x7f11fc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.688+0000 7f120dfbc640 1 -- 172.21.15.1:0/1711851025 shutdown_connections 2024-03-06T07:32:48.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.688+0000 7f120dfbc640 1 --2- 172.21.15.1:0/1711851025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12080c6b10 0x7f1208155c90 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.688+0000 7f120dfbc640 1 -- 172.21.15.1:0/1711851025 >> 172.21.15.1:0/1711851025 conn(0x7f1208066ee0 msgr2=0x7f12080672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.688+0000 7f120dfbc640 1 -- 172.21.15.1:0/1711851025 shutdown_connections 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.688+0000 7f120dfbc640 1 -- 172.21.15.1:0/1711851025 wait complete. 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.689+0000 7f120dfbc640 1 Processor -- start 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.689+0000 7f120dfbc640 1 -- start start 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f120dfbc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12080bf9d0 0x7f12081ef210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f120dfbc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1208152650 0x7f120814e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f120dfbc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1208145540 0x7f12081459f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f120dfbc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12081551b0 con 0x7f1208152650 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f120dfbc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12080c6fa0 con 0x7f1208145540 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f120dfbc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1208155490 con 0x7f12080bf9d0 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f1206ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1208152650 0x7f120814e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f1206ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1208152650 0x7f120814e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41470/0 (socket says 172.21.15.1:41470) 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f1206ffd640 1 -- 172.21.15.1:0/3686259556 learned_addr learned my addr 172.21.15.1:0/3686259556 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f1207fff640 1 --2- 172.21.15.1:0/3686259556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1208145540 0x7f12081459f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f12077fe640 1 --2- 172.21.15.1:0/3686259556 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12080bf9d0 0x7f12081ef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f1206ffd640 1 -- 172.21.15.1:0/3686259556 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12080bf9d0 msgr2=0x7f12081ef210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f1206ffd640 1 --2- 172.21.15.1:0/3686259556 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12080bf9d0 0x7f12081ef210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f1206ffd640 1 -- 172.21.15.1:0/3686259556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1208145540 msgr2=0x7f12081459f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f1206ffd640 1 --2- 172.21.15.1:0/3686259556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1208145540 0x7f12081459f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f1206ffd640 1 -- 172.21.15.1:0/3686259556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12081503b0 con 0x7f1208152650 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f1207fff640 1 --2- 172.21.15.1:0/3686259556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1208145540 0x7f12081459f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.690+0000 7f12077fe640 1 --2- 172.21.15.1:0/3686259556 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12080bf9d0 0x7f12081ef210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.691+0000 7f1206ffd640 1 --2- 172.21.15.1:0/3686259556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1208152650 0x7f120814e580 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7f11fc0098c0 tx=0x7f11fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.691+0000 7f11effff640 1 -- 172.21.15.1:0/3686259556 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc041070 con 0x7f1208152650 2024-03-06T07:32:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.691+0000 7f11effff640 1 -- 172.21.15.1:0/3686259556 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc03aa30 con 0x7f1208152650 2024-03-06T07:32:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.691+0000 7f120dfbc640 1 -- 172.21.15.1:0/3686259556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1208150690 con 0x7f1208152650 2024-03-06T07:32:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.691+0000 7f11effff640 1 -- 172.21.15.1:0/3686259556 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc035040 con 0x7f1208152650 2024-03-06T07:32:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.691+0000 7f120dfbc640 1 -- 172.21.15.1:0/3686259556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f120814eac0 con 0x7f1208152650 2024-03-06T07:32:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.692+0000 7f120dfbc640 1 -- 172.21.15.1:0/3686259556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f120806b540 con 0x7f1208152650 2024-03-06T07:32:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.692+0000 7f11effff640 1 -- 172.21.15.1:0/3686259556 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11fc038020 con 0x7f1208152650 2024-03-06T07:32:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.692+0000 7f11effff640 1 --2- 172.21.15.1:0/3686259556 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f11f406b500 0x7f11f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.693+0000 7f11effff640 1 -- 172.21.15.1:0/3686259556 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f11fc0abf00 con 0x7f1208152650 2024-03-06T07:32:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.693+0000 7f12077fe640 1 --2- 172.21.15.1:0/3686259556 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f11f406b500 0x7f11f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.693+0000 7f12077fe640 1 --2- 172.21.15.1:0/3686259556 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f11f406b500 0x7f11f406d9c0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f11f8000f60 tx=0x7f11f8007c50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:48.697+0000 7f11effff640 1 -- 172.21.15.1:0/3686259556 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f120806b540 con 0x7f1208152650 2024-03-06T07:32:49.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:49.036+0000 7f120dfbc640 1 -- 172.21.15.1:0/3686259556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12080c85a0 con 0x7f1208152650 2024-03-06T07:32:49.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:49.036+0000 7f11effff640 1 -- 172.21.15.1:0/3686259556 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f12080c85a0 con 0x7f1208152650 2024-03-06T07:32:49.038 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:49.038 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:49.039 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:49.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:49.038+0000 7f120dfbc640 1 -- 172.21.15.1:0/3686259556 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f11f406b500 msgr2=0x7f11f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:49.038+0000 7f120dfbc640 1 --2- 172.21.15.1:0/3686259556 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f11f406b500 0x7f11f406d9c0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f11f8000f60 tx=0x7f11f8007c50 comp rx=0 tx=0).stop 2024-03-06T07:32:49.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:49.038+0000 7f120dfbc640 1 -- 172.21.15.1:0/3686259556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1208152650 msgr2=0x7f120814e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:49.038+0000 7f120dfbc640 1 --2- 172.21.15.1:0/3686259556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1208152650 0x7f120814e580 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7f11fc0098c0 tx=0x7f11fc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:49.038+0000 7f120dfbc640 1 -- 172.21.15.1:0/3686259556 shutdown_connections 2024-03-06T07:32:49.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:49.038+0000 7f120dfbc640 1 --2- 172.21.15.1:0/3686259556 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f11f406b500 0x7f11f406d9c0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:49.038+0000 7f120dfbc640 1 --2- 172.21.15.1:0/3686259556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1208145540 0x7f12081459f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:49.038+0000 7f120dfbc640 1 --2- 172.21.15.1:0/3686259556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1208152650 0x7f120814e580 unknown :-1 s=CLOSED pgs=2628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:49.038+0000 7f120dfbc640 1 --2- 172.21.15.1:0/3686259556 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12080bf9d0 0x7f12081ef210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:49.038+0000 7f120dfbc640 1 -- 172.21.15.1:0/3686259556 >> 172.21.15.1:0/3686259556 conn(0x7f1208066ee0 msgr2=0x7f12080c39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:49.038+0000 7f120dfbc640 1 -- 172.21.15.1:0/3686259556 shutdown_connections 2024-03-06T07:32:49.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:49.038+0000 7f120dfbc640 1 -- 172.21.15.1:0/3686259556 wait complete. 2024-03-06T07:32:49.453 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:49.453 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:50.454 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.554+0000 7f7ad2735640 1 Processor -- start 2024-03-06T07:32:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.554+0000 7f7ad2735640 1 -- start start 2024-03-06T07:32:50.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.555+0000 7f7ad2735640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7acc152820 0x7f7acc0c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.555+0000 7f7ad2735640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7acc0c7180 con 0x7f7acc0af6f0 2024-03-06T07:32:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.555+0000 7f7ad2735640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7acc155460 con 0x7f7acc0c0670 2024-03-06T07:32:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.555+0000 7f7ad2735640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc0c6bc0 con 0x7f7acc152820 2024-03-06T07:32:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.555+0000 7f7ad0cd2640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7acc0af6f0 0x7f7acc15fb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36764/0 (socket says 172.21.15.1:36764) 2024-03-06T07:32:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.555+0000 7f7ad0cd2640 1 -- 172.21.15.1:0/4028293738 learned_addr learned my addr 172.21.15.1:0/4028293738 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.555+0000 7f7acbfff640 1 --2- 172.21.15.1:0/4028293738 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7acc152820 0x7f7acc0c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.555+0000 7f7acaffd640 1 -- 172.21.15.1:0/4028293738 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436118352 0 0) 0x7f7acc0c7180 con 0x7f7acc0af6f0 2024-03-06T07:32:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.555+0000 7f7acaffd640 1 -- 172.21.15.1:0/4028293738 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ab4003520 con 0x7f7acc0af6f0 2024-03-06T07:32:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.555+0000 7f7acaffd640 1 -- 172.21.15.1:0/4028293738 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2138784474 0 0) 0x7f7ab4003520 con 0x7f7acc0af6f0 2024-03-06T07:32:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.555+0000 7f7acaffd640 1 -- 172.21.15.1:0/4028293738 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7acc152820 msgr2=0x7f7acc0c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.555+0000 7f7acaffd640 1 --2- 172.21.15.1:0/4028293738 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7acc152820 0x7f7acc0c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.555+0000 7f7acaffd640 1 -- 172.21.15.1:0/4028293738 >> v1:172.21.15.98:6789/0 conn(0x7f7acc0c0670 legacy=0x7f7acc0c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.555+0000 7f7acaffd640 1 -- 172.21.15.1:0/4028293738 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc0c6ea0 con 0x7f7acc0af6f0 2024-03-06T07:32:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.555+0000 7f7acaffd640 1 -- 172.21.15.1:0/4028293738 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f7ac40019f0 con 0x7f7acc0af6f0 2024-03-06T07:32:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.555+0000 7f7acaffd640 1 -- 172.21.15.1:0/4028293738 >> v1:172.21.15.1:6789/0 conn(0x7f7acc0af6f0 legacy=0x7f7acc15fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.556+0000 7f7acaffd640 1 --2- 172.21.15.1:0/4028293738 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7acc152820 0x7f7ab40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.556+0000 7f7acaffd640 1 --2- 172.21.15.1:0/4028293738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab4003f20 0x7f7ab4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.556+0000 7f7acaffd640 1 --2- 172.21.15.1:0/4028293738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ab4004d40 0x7f7ab40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.556+0000 7f7acaffd640 1 -- 172.21.15.1:0/4028293738 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc0c6bc0 con 0x7f7ab4004d40 2024-03-06T07:32:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.556+0000 7f7acaffd640 1 -- 172.21.15.1:0/4028293738 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab400ba20 con 0x7f7ab4003f20 2024-03-06T07:32:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.556+0000 7f7acaffd640 1 -- 172.21.15.1:0/4028293738 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab400bc20 con 0x7f7acc152820 2024-03-06T07:32:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.556+0000 7f7ad0cd2640 1 --2- 172.21.15.1:0/4028293738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ab4004d40 0x7f7ab40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.556+0000 7f7acbfff640 1 --2- 172.21.15.1:0/4028293738 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7acc152820 0x7f7ab40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.556+0000 7f7acb7fe640 1 --2- 172.21.15.1:0/4028293738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab4003f20 0x7f7ab4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.556+0000 7f7ad0cd2640 1 -- 172.21.15.1:0/4028293738 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7acc152820 msgr2=0x7f7ab40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.556+0000 7f7ad0cd2640 1 --2- 172.21.15.1:0/4028293738 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7acc152820 0x7f7ab40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.556+0000 7f7ad0cd2640 1 -- 172.21.15.1:0/4028293738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab4003f20 msgr2=0x7f7ab4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.556+0000 7f7ad0cd2640 1 --2- 172.21.15.1:0/4028293738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab4003f20 0x7f7ab4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.556+0000 7f7ad0cd2640 1 -- 172.21.15.1:0/4028293738 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab4003520 con 0x7f7ab4004d40 2024-03-06T07:32:50.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.557+0000 7f7ad0cd2640 1 --2- 172.21.15.1:0/4028293738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ab4004d40 0x7f7ab40051c0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400a830 tx=0x7f7ab4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e76141b9a6d3987 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.557+0000 7f7acaffd640 1 -- 172.21.15.1:0/4028293738 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac400a480 con 0x7f7ab4004d40 2024-03-06T07:32:50.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.557+0000 7f7acaffd640 1 -- 172.21.15.1:0/4028293738 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac4034630 con 0x7f7ab4004d40 2024-03-06T07:32:50.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.557+0000 7f7acaffd640 1 -- 172.21.15.1:0/4028293738 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4034a80 con 0x7f7ab4004d40 2024-03-06T07:32:50.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.557+0000 7f7ad2735640 1 -- 172.21.15.1:0/4028293738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ab4004d40 msgr2=0x7f7ab40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.557+0000 7f7ad2735640 1 --2- 172.21.15.1:0/4028293738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ab4004d40 0x7f7ab40051c0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400a830 tx=0x7f7ab4005830 comp rx=0 tx=0).stop 2024-03-06T07:32:50.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.557+0000 7f7ad0cd2640 1 -- 172.21.15.1:0/4028293738 reap_dead start 2024-03-06T07:32:50.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.557+0000 7f7ad2735640 1 -- 172.21.15.1:0/4028293738 shutdown_connections 2024-03-06T07:32:50.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.557+0000 7f7ad2735640 1 -- 172.21.15.1:0/4028293738 >> 172.21.15.1:0/4028293738 conn(0x7f7acc072720 msgr2=0x7f7acc072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.557+0000 7f7ad2735640 1 -- 172.21.15.1:0/4028293738 shutdown_connections 2024-03-06T07:32:50.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.557+0000 7f7ad2735640 1 -- 172.21.15.1:0/4028293738 wait complete. 2024-03-06T07:32:50.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.558+0000 7f7ad2735640 1 Processor -- start 2024-03-06T07:32:50.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.559+0000 7f7ad2735640 1 -- start start 2024-03-06T07:32:50.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.559+0000 7f7ad2735640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab4004d40 0x7f7acc159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.559+0000 7f7ad2735640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7acc14c9f0 0x7f7acc14a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.559+0000 7f7ad2735640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7acc14abe0 0x7f7acc1ee140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.559+0000 7f7ad2735640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc0c7c80 con 0x7f7acc14abe0 2024-03-06T07:32:50.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.559+0000 7f7ad2735640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc0c7b00 con 0x7f7ab4004d40 2024-03-06T07:32:50.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.559+0000 7f7ad2735640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc152c00 con 0x7f7acc14c9f0 2024-03-06T07:32:50.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.559+0000 7f7ad0cd2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7acc14abe0 0x7f7acc1ee140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.559+0000 7f7ad0cd2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7acc14abe0 0x7f7acc1ee140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41496/0 (socket says 172.21.15.1:41496) 2024-03-06T07:32:50.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.559+0000 7f7ad0cd2640 1 -- 172.21.15.1:0/1043819267 learned_addr learned my addr 172.21.15.1:0/1043819267 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:50.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.560+0000 7f7acbfff640 1 --2- 172.21.15.1:0/1043819267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab4004d40 0x7f7acc159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.560+0000 7f7acb7fe640 1 --2- 172.21.15.1:0/1043819267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7acc14c9f0 0x7f7acc14a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.560+0000 7f7ad0cd2640 1 -- 172.21.15.1:0/1043819267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7acc14c9f0 msgr2=0x7f7acc14a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.560+0000 7f7ad0cd2640 1 --2- 172.21.15.1:0/1043819267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7acc14c9f0 0x7f7acc14a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.560+0000 7f7ad0cd2640 1 -- 172.21.15.1:0/1043819267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab4004d40 msgr2=0x7f7acc159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.560+0000 7f7ad0cd2640 1 --2- 172.21.15.1:0/1043819267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab4004d40 0x7f7acc159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.560+0000 7f7ad0cd2640 1 -- 172.21.15.1:0/1043819267 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc154d60 con 0x7f7acc14abe0 2024-03-06T07:32:50.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.560+0000 7f7ad0cd2640 1 --2- 172.21.15.1:0/1043819267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7acc14abe0 0x7f7acc1ee140 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400a950 tx=0x7f7acc0c24b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.560+0000 7f7ac8ff9640 1 -- 172.21.15.1:0/1043819267 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4044070 con 0x7f7acc14abe0 2024-03-06T07:32:50.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.560+0000 7f7ac8ff9640 1 -- 172.21.15.1:0/1043819267 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac403a040 con 0x7f7acc14abe0 2024-03-06T07:32:50.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.560+0000 7f7ac8ff9640 1 -- 172.21.15.1:0/1043819267 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac400a480 con 0x7f7acc14abe0 2024-03-06T07:32:50.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.560+0000 7f7ad2735640 1 -- 172.21.15.1:0/1043819267 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7acc156e30 con 0x7f7acc14abe0 2024-03-06T07:32:50.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.560+0000 7f7ad2735640 1 -- 172.21.15.1:0/1043819267 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7acc158020 con 0x7f7acc14abe0 2024-03-06T07:32:50.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.561+0000 7f7ad2735640 1 -- 172.21.15.1:0/1043819267 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7acc0686e0 con 0x7f7acc14abe0 2024-03-06T07:32:50.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.562+0000 7f7ac8ff9640 1 -- 172.21.15.1:0/1043819267 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ac4047070 con 0x7f7acc14abe0 2024-03-06T07:32:50.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.562+0000 7f7ac8ff9640 1 --2- 172.21.15.1:0/1043819267 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7ab806b4d0 0x7f7ab806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.562+0000 7f7ac8ff9640 1 -- 172.21.15.1:0/1043819267 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f7ac40abbb0 con 0x7f7acc14abe0 2024-03-06T07:32:50.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.562+0000 7f7acbfff640 1 --2- 172.21.15.1:0/1043819267 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7ab806b4d0 0x7f7ab806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.563+0000 7f7acbfff640 1 --2- 172.21.15.1:0/1043819267 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7ab806b4d0 0x7f7ab806d990 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009bc0 tx=0x7f7abc009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.566+0000 7f7ac8ff9640 1 -- 172.21.15.1:0/1043819267 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ac407a150 con 0x7f7acc14abe0 2024-03-06T07:32:50.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.784+0000 7f7ad2735640 1 -- 172.21.15.1:0/1043819267 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7acc1596d0 con 0x7f7acc14abe0 2024-03-06T07:32:50.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.785+0000 7f7ac8ff9640 1 -- 172.21.15.1:0/1043819267 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f7ac40af020 con 0x7f7acc14abe0 2024-03-06T07:32:50.787 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:50.787 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:50.787 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.786+0000 7f7ad2735640 1 -- 172.21.15.1:0/1043819267 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7ab806b4d0 msgr2=0x7f7ab806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.786+0000 7f7ad2735640 1 --2- 172.21.15.1:0/1043819267 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7ab806b4d0 0x7f7ab806d990 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009bc0 tx=0x7f7abc009190 comp rx=0 tx=0).stop 2024-03-06T07:32:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.786+0000 7f7ad2735640 1 -- 172.21.15.1:0/1043819267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7acc14abe0 msgr2=0x7f7acc1ee140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.786+0000 7f7ad2735640 1 --2- 172.21.15.1:0/1043819267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7acc14abe0 0x7f7acc1ee140 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400a950 tx=0x7f7acc0c24b0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.787+0000 7f7ad2735640 1 -- 172.21.15.1:0/1043819267 shutdown_connections 2024-03-06T07:32:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.787+0000 7f7ad2735640 1 --2- 172.21.15.1:0/1043819267 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7ab806b4d0 0x7f7ab806d990 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.787+0000 7f7ad2735640 1 --2- 172.21.15.1:0/1043819267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7acc14abe0 0x7f7acc1ee140 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.787+0000 7f7ad2735640 1 --2- 172.21.15.1:0/1043819267 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7acc14c9f0 0x7f7acc14a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.787+0000 7f7ad2735640 1 --2- 172.21.15.1:0/1043819267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab4004d40 0x7f7acc159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.787+0000 7f7ad2735640 1 -- 172.21.15.1:0/1043819267 >> 172.21.15.1:0/1043819267 conn(0x7f7acc072720 msgr2=0x7f7acc14ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.787+0000 7f7ad2735640 1 -- 172.21.15.1:0/1043819267 shutdown_connections 2024-03-06T07:32:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:50.787+0000 7f7ad2735640 1 -- 172.21.15.1:0/1043819267 wait complete. 2024-03-06T07:32:51.205 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:51.206 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:52.207 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:52.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.306+0000 7f622180e640 1 Processor -- start 2024-03-06T07:32:52.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.306+0000 7f622180e640 1 -- start start 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f622180e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f621c0c3b00 0x7f621c072780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f622180e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f621c0c4a50 0x7f621c0c4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f622180e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f621c071b10 con 0x7f621c158e20 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f622180e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f621c0c29b0 con 0x7f621c0c3b00 2024-03-06T07:32:52.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f622180e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f621c0c2f70 con 0x7f621c0c4a50 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f621a7fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f621c158e20 0x7f621c159280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36776/0 (socket says 172.21.15.1:36776) 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f621a7fc640 1 -- 172.21.15.1:0/3485716288 learned_addr learned my addr 172.21.15.1:0/3485716288 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f621b7fe640 1 --2- 172.21.15.1:0/3485716288 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f621c0c4a50 0x7f621c0c4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f621affd640 1 --2- 172.21.15.1:0/3485716288 >> v2:172.21.15.98:3300/0 conn(0x7f621c0c3b00 0x7f621c072780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f6219ffb640 1 -- 172.21.15.1:0/3485716288 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831320552 0 0) 0x7f621c071b10 con 0x7f621c158e20 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f6219ffb640 1 -- 172.21.15.1:0/3485716288 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6204003520 con 0x7f621c158e20 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f6219ffb640 1 -- 172.21.15.1:0/3485716288 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 810082047 0 0) 0x7f6204003520 con 0x7f621c158e20 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f6219ffb640 1 -- 172.21.15.1:0/3485716288 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f621c0c4a50 msgr2=0x7f621c0c4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f6219ffb640 1 --2- 172.21.15.1:0/3485716288 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f621c0c4a50 0x7f621c0c4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f6219ffb640 1 -- 172.21.15.1:0/3485716288 >> v2:172.21.15.98:3300/0 conn(0x7f621c0c3b00 msgr2=0x7f621c072780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f6219ffb640 1 --2- 172.21.15.1:0/3485716288 >> v2:172.21.15.98:3300/0 conn(0x7f621c0c3b00 0x7f621c072780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f6219ffb640 1 -- 172.21.15.1:0/3485716288 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f621c071dd0 con 0x7f621c158e20 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f6219ffb640 1 -- 172.21.15.1:0/3485716288 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f6210001670 con 0x7f621c158e20 2024-03-06T07:32:52.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f6219ffb640 1 -- 172.21.15.1:0/3485716288 >> v1:172.21.15.1:6789/0 conn(0x7f621c158e20 legacy=0x7f621c159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f6219ffb640 1 --2- 172.21.15.1:0/3485716288 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f621c0c4a50 0x7f62040039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f6219ffb640 1 --2- 172.21.15.1:0/3485716288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6204003f30 0x7f6204004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f6219ffb640 1 --2- 172.21.15.1:0/3485716288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6204004d40 0x7f62040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f6219ffb640 1 -- 172.21.15.1:0/3485716288 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621c0c29b0 con 0x7f6204003f30 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f6219ffb640 1 -- 172.21.15.1:0/3485716288 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621c0c2f70 con 0x7f6204004d40 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f6219ffb640 1 -- 172.21.15.1:0/3485716288 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f620400ba10 con 0x7f621c0c4a50 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.307+0000 7f621a7fc640 1 --2- 172.21.15.1:0/3485716288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6204003f30 0x7f6204004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.308+0000 7f621affd640 1 --2- 172.21.15.1:0/3485716288 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f621c0c4a50 0x7f62040039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.308+0000 7f621b7fe640 1 --2- 172.21.15.1:0/3485716288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6204004d40 0x7f62040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.308+0000 7f621a7fc640 1 -- 172.21.15.1:0/3485716288 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f621c0c4a50 msgr2=0x7f62040039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.308+0000 7f621a7fc640 1 --2- 172.21.15.1:0/3485716288 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f621c0c4a50 0x7f62040039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.308+0000 7f621a7fc640 1 -- 172.21.15.1:0/3485716288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6204004d40 msgr2=0x7f62040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.308+0000 7f621a7fc640 1 --2- 172.21.15.1:0/3485716288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6204004d40 0x7f62040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.308+0000 7f621a7fc640 1 -- 172.21.15.1:0/3485716288 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6204003520 con 0x7f6204003f30 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.308+0000 7f621a7fc640 1 --2- 172.21.15.1:0/3485716288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6204003f30 0x7f6204004800 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f621000c210 tx=0x7f6204005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b808cb99b2be792 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.309+0000 7f6219ffb640 1 -- 172.21.15.1:0/3485716288 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6210007d70 con 0x7f6204003f30 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.309+0000 7f6219ffb640 1 -- 172.21.15.1:0/3485716288 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6210030070 con 0x7f6204003f30 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.309+0000 7f6219ffb640 1 -- 172.21.15.1:0/3485716288 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621002cd60 con 0x7f6204003f30 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.309+0000 7f622180e640 1 -- 172.21.15.1:0/3485716288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6204003f30 msgr2=0x7f6204004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.309+0000 7f622180e640 1 --2- 172.21.15.1:0/3485716288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6204003f30 0x7f6204004800 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f621000c210 tx=0x7f6204005830 comp rx=0 tx=0).stop 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.309+0000 7f621b7fe640 1 -- 172.21.15.1:0/3485716288 reap_dead start 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.309+0000 7f622180e640 1 -- 172.21.15.1:0/3485716288 shutdown_connections 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.309+0000 7f622180e640 1 -- 172.21.15.1:0/3485716288 >> 172.21.15.1:0/3485716288 conn(0x7f621c0c6f40 msgr2=0x7f621c0c7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.309+0000 7f622180e640 1 -- 172.21.15.1:0/3485716288 shutdown_connections 2024-03-06T07:32:52.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.309+0000 7f622180e640 1 -- 172.21.15.1:0/3485716288 wait complete. 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.311+0000 7f622180e640 1 Processor -- start 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.311+0000 7f622180e640 1 -- start start 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.312+0000 7f622180e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6204003f30 0x7f621c0a58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.312+0000 7f622180e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f621c09a170 0x7f621c152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.312+0000 7f622180e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f621c0a5df0 0x7f621c152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.312+0000 7f622180e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621c072cc0 con 0x7f621c0a5df0 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.312+0000 7f622180e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621c0c2c00 con 0x7f621c09a170 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.312+0000 7f622180e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f621c0a65a0 con 0x7f6204003f30 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.312+0000 7f621b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f621c0a5df0 0x7f621c152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.312+0000 7f621b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f621c0a5df0 0x7f621c152eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41524/0 (socket says 172.21.15.1:41524) 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.312+0000 7f621b7fe640 1 -- 172.21.15.1:0/3356638689 learned_addr learned my addr 172.21.15.1:0/3356638689 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.313+0000 7f621a7fc640 1 --2- 172.21.15.1:0/3356638689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f621c09a170 0x7f621c152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.313+0000 7f621affd640 1 --2- 172.21.15.1:0/3356638689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6204003f30 0x7f621c0a58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.313+0000 7f621b7fe640 1 -- 172.21.15.1:0/3356638689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6204003f30 msgr2=0x7f621c0a58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.313+0000 7f621b7fe640 1 --2- 172.21.15.1:0/3356638689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6204003f30 0x7f621c0a58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.313+0000 7f621b7fe640 1 -- 172.21.15.1:0/3356638689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f621c09a170 msgr2=0x7f621c152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.313+0000 7f621b7fe640 1 --2- 172.21.15.1:0/3356638689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f621c09a170 0x7f621c152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.313+0000 7f621b7fe640 1 -- 172.21.15.1:0/3356638689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f621c1eb810 con 0x7f621c0a5df0 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.313+0000 7f621a7fc640 1 --2- 172.21.15.1:0/3356638689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f621c09a170 0x7f621c152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.313+0000 7f621b7fe640 1 --2- 172.21.15.1:0/3356638689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f621c0a5df0 0x7f621c152eb0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7f621c0c4340 tx=0x7f621400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.313+0000 7f6203fff640 1 -- 172.21.15.1:0/3356638689 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621400fd80 con 0x7f621c0a5df0 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.313+0000 7f6203fff640 1 -- 172.21.15.1:0/3356638689 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6214015430 con 0x7f621c0a5df0 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.313+0000 7f6203fff640 1 -- 172.21.15.1:0/3356638689 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6214015820 con 0x7f621c0a5df0 2024-03-06T07:32:52.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.313+0000 7f622180e640 1 -- 172.21.15.1:0/3356638689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f621c1535c0 con 0x7f621c0a5df0 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.313+0000 7f622180e640 1 -- 172.21.15.1:0/3356638689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f621c0a9550 con 0x7f621c0a5df0 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.314+0000 7f622180e640 1 -- 172.21.15.1:0/3356638689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f621c0c3ee0 con 0x7f621c0a5df0 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.315+0000 7f6203fff640 1 -- 172.21.15.1:0/3356638689 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f62140159c0 con 0x7f621c0a5df0 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.315+0000 7f6203fff640 1 --2- 172.21.15.1:0/3356638689 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f620806b4d0 0x7f620806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.315+0000 7f6203fff640 1 -- 172.21.15.1:0/3356638689 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f621408b1a0 con 0x7f621c0a5df0 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.315+0000 7f621affd640 1 --2- 172.21.15.1:0/3356638689 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f620806b4d0 0x7f620806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.316+0000 7f621affd640 1 --2- 172.21.15.1:0/3356638689 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f620806b4d0 0x7f620806d990 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f621c0abca0 tx=0x7f620c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.319+0000 7f6203fff640 1 -- 172.21.15.1:0/3356638689 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f621c0c3ee0 con 0x7f621c0a5df0 2024-03-06T07:32:52.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.574+0000 7f622180e640 1 -- 172.21.15.1:0/3356638689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f621c067fd0 con 0x7f621c0a5df0 2024-03-06T07:32:52.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.575+0000 7f6203fff640 1 -- 172.21.15.1:0/3356638689 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f6214019030 con 0x7f621c0a5df0 2024-03-06T07:32:52.577 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:52.577 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:52.577 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.577+0000 7f622180e640 1 -- 172.21.15.1:0/3356638689 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f620806b4d0 msgr2=0x7f620806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.577+0000 7f622180e640 1 --2- 172.21.15.1:0/3356638689 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f620806b4d0 0x7f620806d990 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f621c0abca0 tx=0x7f620c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.577+0000 7f622180e640 1 -- 172.21.15.1:0/3356638689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f621c0a5df0 msgr2=0x7f621c152eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.577+0000 7f622180e640 1 --2- 172.21.15.1:0/3356638689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f621c0a5df0 0x7f621c152eb0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7f621c0c4340 tx=0x7f621400f580 comp rx=0 tx=0).stop 2024-03-06T07:32:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.577+0000 7f622180e640 1 -- 172.21.15.1:0/3356638689 shutdown_connections 2024-03-06T07:32:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.577+0000 7f622180e640 1 --2- 172.21.15.1:0/3356638689 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f620806b4d0 0x7f620806d990 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.577+0000 7f622180e640 1 --2- 172.21.15.1:0/3356638689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f621c0a5df0 0x7f621c152eb0 unknown :-1 s=CLOSED pgs=2634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.577+0000 7f622180e640 1 --2- 172.21.15.1:0/3356638689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f621c09a170 0x7f621c152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.577+0000 7f622180e640 1 --2- 172.21.15.1:0/3356638689 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6204003f30 0x7f621c0a58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.577+0000 7f622180e640 1 -- 172.21.15.1:0/3356638689 >> 172.21.15.1:0/3356638689 conn(0x7f621c0c6f40 msgr2=0x7f621c0aa810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.577+0000 7f622180e640 1 -- 172.21.15.1:0/3356638689 shutdown_connections 2024-03-06T07:32:52.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:52.577+0000 7f622180e640 1 -- 172.21.15.1:0/3356638689 wait complete. 2024-03-06T07:32:52.992 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:52.992 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:53.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:54.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.093+0000 7f0e35447640 1 Processor -- start 2024-03-06T07:32:54.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.093+0000 7f0e35447640 1 -- start start 2024-03-06T07:32:54.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.094+0000 7f0e35447640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0e300c1820 0x7f0e301559f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.094+0000 7f0e35447640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e301588e0 con 0x7f0e30152730 2024-03-06T07:32:54.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.094+0000 7f0e35447640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e30158bc0 con 0x7f0e300c1390 2024-03-06T07:32:54.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.094+0000 7f0e35447640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e30158ea0 con 0x7f0e300c1820 2024-03-06T07:32:54.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.094+0000 7f0e2f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0e300c1820 0x7f0e301559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.094+0000 7f0e2effd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0e30152730 0x7f0e300c6a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36784/0 (socket says 172.21.15.1:36784) 2024-03-06T07:32:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.094+0000 7f0e2effd640 1 -- 172.21.15.1:0/847680574 learned_addr learned my addr 172.21.15.1:0/847680574 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.094+0000 7f0e2f7fe640 1 -- 172.21.15.1:0/847680574 >> v1:172.21.15.98:6789/0 conn(0x7f0e300c1390 legacy=0x7f0e301552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.094+0000 7f0e2f7fe640 1 -- 172.21.15.1:0/847680574 >> v1:172.21.15.1:6789/0 conn(0x7f0e30152730 legacy=0x7f0e300c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.094+0000 7f0e2dffb640 1 -- 172.21.15.1:0/847680574 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045801427 0 0) 0x7f0e301588e0 con 0x7f0e30152730 2024-03-06T07:32:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.094+0000 7f0e2f7fe640 1 -- 172.21.15.1:0/847680574 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e30159180 con 0x7f0e300c1820 2024-03-06T07:32:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.095+0000 7f0e2f7fe640 1 --2- 172.21.15.1:0/847680574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e300c1820 0x7f0e301559f0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f0e20009d90 tx=0x7f0e2002bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb97e397939be7d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.095+0000 7f0e2dffb640 1 -- 172.21.15.1:0/847680574 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e20039070 con 0x7f0e300c1820 2024-03-06T07:32:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.095+0000 7f0e2dffb640 1 -- 172.21.15.1:0/847680574 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2002cb40 con 0x7f0e300c1820 2024-03-06T07:32:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.095+0000 7f0e2dffb640 1 -- 172.21.15.1:0/847680574 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e20034440 con 0x7f0e300c1820 2024-03-06T07:32:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.096+0000 7f0e35447640 1 -- 172.21.15.1:0/847680574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e300c1820 msgr2=0x7f0e301559f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.096+0000 7f0e35447640 1 --2- 172.21.15.1:0/847680574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e300c1820 0x7f0e301559f0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f0e20009d90 tx=0x7f0e2002bd90 comp rx=0 tx=0).stop 2024-03-06T07:32:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.096+0000 7f0e35447640 1 -- 172.21.15.1:0/847680574 shutdown_connections 2024-03-06T07:32:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.096+0000 7f0e35447640 1 --2- 172.21.15.1:0/847680574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e300c1820 0x7f0e301559f0 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.096+0000 7f0e35447640 1 -- 172.21.15.1:0/847680574 >> 172.21.15.1:0/847680574 conn(0x7f0e30066d00 msgr2=0x7f0e30067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.096+0000 7f0e35447640 1 -- 172.21.15.1:0/847680574 shutdown_connections 2024-03-06T07:32:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.096+0000 7f0e35447640 1 -- 172.21.15.1:0/847680574 wait complete. 2024-03-06T07:32:54.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.097+0000 7f0e35447640 1 Processor -- start 2024-03-06T07:32:54.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.097+0000 7f0e35447640 1 -- start start 2024-03-06T07:32:54.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.097+0000 7f0e35447640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e300c1390 0x7f0e30149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.097+0000 7f0e35447640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e300c1820 0x7f0e301e7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e35447640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e30152730 0x7f0e301e7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e35447640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e30152b10 con 0x7f0e30152730 2024-03-06T07:32:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e35447640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e30155f30 con 0x7f0e300c1390 2024-03-06T07:32:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e35447640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e300c1c40 con 0x7f0e300c1820 2024-03-06T07:32:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e2f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e30152730 0x7f0e301e7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e2f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e30152730 0x7f0e301e7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41530/0 (socket says 172.21.15.1:41530) 2024-03-06T07:32:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e2f7fe640 1 -- 172.21.15.1:0/3633473170 learned_addr learned my addr 172.21.15.1:0/3633473170 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e2effd640 1 --2- 172.21.15.1:0/3633473170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e300c1390 0x7f0e30149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e2e7fc640 1 --2- 172.21.15.1:0/3633473170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e300c1820 0x7f0e301e7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e2f7fe640 1 -- 172.21.15.1:0/3633473170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e300c1820 msgr2=0x7f0e301e7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e2f7fe640 1 --2- 172.21.15.1:0/3633473170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e300c1820 0x7f0e301e7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e2f7fe640 1 -- 172.21.15.1:0/3633473170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e300c1390 msgr2=0x7f0e30149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e2f7fe640 1 --2- 172.21.15.1:0/3633473170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e300c1390 0x7f0e30149ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e2f7fe640 1 -- 172.21.15.1:0/3633473170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e301eaa80 con 0x7f0e30152730 2024-03-06T07:32:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e2f7fe640 1 --2- 172.21.15.1:0/3633473170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e30152730 0x7f0e301e7fd0 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f0e20009eb0 tx=0x7f0e20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e177fe640 1 -- 172.21.15.1:0/3633473170 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e20044030 con 0x7f0e30152730 2024-03-06T07:32:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e177fe640 1 -- 172.21.15.1:0/3633473170 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2002c940 con 0x7f0e30152730 2024-03-06T07:32:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e177fe640 1 -- 172.21.15.1:0/3633473170 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e20039040 con 0x7f0e30152730 2024-03-06T07:32:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.098+0000 7f0e35447640 1 -- 172.21.15.1:0/3633473170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e301ead60 con 0x7f0e30152730 2024-03-06T07:32:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.099+0000 7f0e35447640 1 -- 172.21.15.1:0/3633473170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e301eb320 con 0x7f0e30152730 2024-03-06T07:32:54.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.100+0000 7f0e35447640 1 -- 172.21.15.1:0/3633473170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e301eb650 con 0x7f0e30152730 2024-03-06T07:32:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.100+0000 7f0e177fe640 1 -- 172.21.15.1:0/3633473170 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e20047070 con 0x7f0e30152730 2024-03-06T07:32:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.100+0000 7f0e177fe640 1 --2- 172.21.15.1:0/3633473170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0e1c06b4d0 0x7f0e1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.100+0000 7f0e177fe640 1 -- 172.21.15.1:0/3633473170 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f0e200acc10 con 0x7f0e30152730 2024-03-06T07:32:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.101+0000 7f0e2effd640 1 --2- 172.21.15.1:0/3633473170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0e1c06b4d0 0x7f0e1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.101+0000 7f0e2effd640 1 --2- 172.21.15.1:0/3633473170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0e1c06b4d0 0x7f0e1c06d990 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f0e28000f60 tx=0x7f0e28007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.105+0000 7f0e177fe640 1 -- 172.21.15.1:0/3633473170 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e2004e070 con 0x7f0e30152730 2024-03-06T07:32:54.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.406+0000 7f0e35447640 1 -- 172.21.15.1:0/3633473170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e30067fd0 con 0x7f0e30152730 2024-03-06T07:32:54.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.407+0000 7f0e177fe640 1 -- 172.21.15.1:0/3633473170 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f0e2007b240 con 0x7f0e30152730 2024-03-06T07:32:54.409 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:54.409 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:54.410 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.409+0000 7f0e35447640 1 -- 172.21.15.1:0/3633473170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0e1c06b4d0 msgr2=0x7f0e1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.409+0000 7f0e35447640 1 --2- 172.21.15.1:0/3633473170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0e1c06b4d0 0x7f0e1c06d990 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f0e28000f60 tx=0x7f0e28007d50 comp rx=0 tx=0).stop 2024-03-06T07:32:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.409+0000 7f0e35447640 1 -- 172.21.15.1:0/3633473170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e30152730 msgr2=0x7f0e301e7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.409+0000 7f0e35447640 1 --2- 172.21.15.1:0/3633473170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e30152730 0x7f0e301e7fd0 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f0e20009eb0 tx=0x7f0e20004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.409+0000 7f0e35447640 1 -- 172.21.15.1:0/3633473170 shutdown_connections 2024-03-06T07:32:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.409+0000 7f0e35447640 1 --2- 172.21.15.1:0/3633473170 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0e1c06b4d0 0x7f0e1c06d990 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.409+0000 7f0e35447640 1 --2- 172.21.15.1:0/3633473170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e30152730 0x7f0e301e7fd0 unknown :-1 s=CLOSED pgs=2637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.409+0000 7f0e35447640 1 --2- 172.21.15.1:0/3633473170 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e300c1820 0x7f0e301e7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.409+0000 7f0e35447640 1 --2- 172.21.15.1:0/3633473170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e300c1390 0x7f0e30149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.409+0000 7f0e35447640 1 -- 172.21.15.1:0/3633473170 >> 172.21.15.1:0/3633473170 conn(0x7f0e30066d00 msgr2=0x7f0e300c3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:54.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.409+0000 7f0e35447640 1 -- 172.21.15.1:0/3633473170 shutdown_connections 2024-03-06T07:32:54.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:54.409+0000 7f0e35447640 1 -- 172.21.15.1:0/3633473170 wait complete. 2024-03-06T07:32:54.828 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:54.828 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:55.829 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:55.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.928+0000 7f94cb5ac640 1 Processor -- start 2024-03-06T07:32:55.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.929+0000 7f94cb5ac640 1 -- start start 2024-03-06T07:32:55.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.929+0000 7f94cb5ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c4152650 0x7f94c4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.929+0000 7f94cb5ac640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94c40bf9d0 0x7f94c40c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.929+0000 7f94cb5ac640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c40c2fc0 con 0x7f94c40c6b10 2024-03-06T07:32:55.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.929+0000 7f94cb5ac640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c4155800 con 0x7f94c40bf9d0 2024-03-06T07:32:55.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.929+0000 7f94cb5ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c4158da0 con 0x7f94c4152650 2024-03-06T07:32:55.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.929+0000 7f94c9b49640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94c40bf9d0 0x7f94c40c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.930+0000 7f94c9b49640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94c40bf9d0 0x7f94c40c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50702/0 (socket says 172.21.15.1:50702) 2024-03-06T07:32:55.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.930+0000 7f94c9b49640 1 -- 172.21.15.1:0/1579147514 learned_addr learned my addr 172.21.15.1:0/1579147514 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:55.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.930+0000 7f94c9348640 1 --2- 172.21.15.1:0/1579147514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c4152650 0x7f94c4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.930+0000 7f94c9b49640 1 -- 172.21.15.1:0/1579147514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c4152650 msgr2=0x7f94c4155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.930+0000 7f94c9b49640 1 --2- 172.21.15.1:0/1579147514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c4152650 0x7f94c4155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.930+0000 7f94c9b49640 1 -- 172.21.15.1:0/1579147514 >> v1:172.21.15.98:6789/0 conn(0x7f94c40c6b10 legacy=0x7f94c4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.930+0000 7f94c9b49640 1 -- 172.21.15.1:0/1579147514 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c40c6690 con 0x7f94c40bf9d0 2024-03-06T07:32:55.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.931+0000 7f94c9b49640 1 --2- 172.21.15.1:0/1579147514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c40bf9d0 0x7f94c40c1460 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009e50 tx=0x7f94bc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929b65e4e612051f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.931+0000 7f94bbfff640 1 -- 172.21.15.1:0/1579147514 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc02ca70 con 0x7f94c40bf9d0 2024-03-06T07:32:55.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.931+0000 7f94bbfff640 1 -- 172.21.15.1:0/1579147514 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94bc02cc10 con 0x7f94c40bf9d0 2024-03-06T07:32:55.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.931+0000 7f94bbfff640 1 -- 172.21.15.1:0/1579147514 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc0343f0 con 0x7f94c40bf9d0 2024-03-06T07:32:55.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.931+0000 7f94cb5ac640 1 -- 172.21.15.1:0/1579147514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c40bf9d0 msgr2=0x7f94c40c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.931+0000 7f94cb5ac640 1 --2- 172.21.15.1:0/1579147514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c40bf9d0 0x7f94c40c1460 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009e50 tx=0x7f94bc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.931+0000 7f94cb5ac640 1 -- 172.21.15.1:0/1579147514 shutdown_connections 2024-03-06T07:32:55.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.931+0000 7f94cb5ac640 1 --2- 172.21.15.1:0/1579147514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c40bf9d0 0x7f94c40c1460 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.931+0000 7f94cb5ac640 1 --2- 172.21.15.1:0/1579147514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c4152650 0x7f94c4155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.931+0000 7f94cb5ac640 1 -- 172.21.15.1:0/1579147514 >> 172.21.15.1:0/1579147514 conn(0x7f94c4066ee0 msgr2=0x7f94c40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.931+0000 7f94cb5ac640 1 -- 172.21.15.1:0/1579147514 shutdown_connections 2024-03-06T07:32:55.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.932+0000 7f94cb5ac640 1 -- 172.21.15.1:0/1579147514 wait complete. 2024-03-06T07:32:55.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.933+0000 7f94cb5ac640 1 Processor -- start 2024-03-06T07:32:55.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.933+0000 7f94cb5ac640 1 -- start start 2024-03-06T07:32:55.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.933+0000 7f94cb5ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c40bf9d0 0x7f94c40c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.933+0000 7f94cb5ac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c4152650 0x7f94c414d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.933+0000 7f94cb5ac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c4157b90 0x7f94c40c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.933+0000 7f94cb5ac640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c41551b0 con 0x7f94c4157b90 2024-03-06T07:32:55.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.933+0000 7f94cb5ac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c40c6fa0 con 0x7f94c4152650 2024-03-06T07:32:55.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.933+0000 7f94cb5ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c4155490 con 0x7f94c40bf9d0 2024-03-06T07:32:55.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.933+0000 7f94c9b49640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c4157b90 0x7f94c40c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.933+0000 7f94c9b49640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c4157b90 0x7f94c40c54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50704/0 (socket says 172.21.15.1:50704) 2024-03-06T07:32:55.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.933+0000 7f94c9b49640 1 -- 172.21.15.1:0/2832076338 learned_addr learned my addr 172.21.15.1:0/2832076338 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:55.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.934+0000 7f94c8b47640 1 --2- 172.21.15.1:0/2832076338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c4152650 0x7f94c414d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.934+0000 7f94c9348640 1 --2- 172.21.15.1:0/2832076338 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c40bf9d0 0x7f94c40c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.934+0000 7f94c9b49640 1 -- 172.21.15.1:0/2832076338 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c40bf9d0 msgr2=0x7f94c40c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.934+0000 7f94c9b49640 1 --2- 172.21.15.1:0/2832076338 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c40bf9d0 0x7f94c40c9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.934+0000 7f94c9b49640 1 -- 172.21.15.1:0/2832076338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c4152650 msgr2=0x7f94c414d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.934+0000 7f94c9b49640 1 --2- 172.21.15.1:0/2832076338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c4152650 0x7f94c414d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.934+0000 7f94c9b49640 1 -- 172.21.15.1:0/2832076338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c4158690 con 0x7f94c4157b90 2024-03-06T07:32:55.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.934+0000 7f94c8b47640 1 --2- 172.21.15.1:0/2832076338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c4152650 0x7f94c414d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:55.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.934+0000 7f94c9348640 1 --2- 172.21.15.1:0/2832076338 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c40bf9d0 0x7f94c40c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:55.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.934+0000 7f94c9b49640 1 --2- 172.21.15.1:0/2832076338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c4157b90 0x7f94c40c54e0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009f70 tx=0x7f94bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.934+0000 7f94b9ffb640 1 -- 172.21.15.1:0/2832076338 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc043020 con 0x7f94c4157b90 2024-03-06T07:32:55.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.934+0000 7f94b9ffb640 1 -- 172.21.15.1:0/2832076338 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94bc03e450 con 0x7f94c4157b90 2024-03-06T07:32:55.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.934+0000 7f94cb5ac640 1 -- 172.21.15.1:0/2832076338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c4158970 con 0x7f94c4157b90 2024-03-06T07:32:55.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.934+0000 7f94b9ffb640 1 -- 172.21.15.1:0/2832076338 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc009510 con 0x7f94c4157b90 2024-03-06T07:32:55.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.934+0000 7f94cb5ac640 1 -- 172.21.15.1:0/2832076338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c40c5a20 con 0x7f94c4157b90 2024-03-06T07:32:55.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.935+0000 7f94cb5ac640 1 -- 172.21.15.1:0/2832076338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c4067fd0 con 0x7f94c4157b90 2024-03-06T07:32:55.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.936+0000 7f94b9ffb640 1 -- 172.21.15.1:0/2832076338 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94bc046020 con 0x7f94c4157b90 2024-03-06T07:32:55.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.936+0000 7f94b9ffb640 1 --2- 172.21.15.1:0/2832076338 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94a806b4d0 0x7f94a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.936+0000 7f94b9ffb640 1 -- 172.21.15.1:0/2832076338 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f94bc0acbe0 con 0x7f94c4157b90 2024-03-06T07:32:55.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.936+0000 7f94c9348640 1 --2- 172.21.15.1:0/2832076338 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94a806b4d0 0x7f94a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.937+0000 7f94c9348640 1 --2- 172.21.15.1:0/2832076338 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94a806b4d0 0x7f94a806d990 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f94c0006fd0 tx=0x7f94c0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:55.940+0000 7f94b9ffb640 1 -- 172.21.15.1:0/2832076338 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94bc04d030 con 0x7f94c4157b90 2024-03-06T07:32:56.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:56.157+0000 7f94cb5ac640 1 -- 172.21.15.1:0/2832076338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94c4158b40 con 0x7f94c4157b90 2024-03-06T07:32:56.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:56.158+0000 7f94b9ffb640 1 -- 172.21.15.1:0/2832076338 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f94bc07b860 con 0x7f94c4157b90 2024-03-06T07:32:56.160 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:56.160 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:56.160 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:56.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:56.160+0000 7f94cb5ac640 1 -- 172.21.15.1:0/2832076338 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94a806b4d0 msgr2=0x7f94a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:56.160+0000 7f94cb5ac640 1 --2- 172.21.15.1:0/2832076338 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94a806b4d0 0x7f94a806d990 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f94c0006fd0 tx=0x7f94c0008040 comp rx=0 tx=0).stop 2024-03-06T07:32:56.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:56.160+0000 7f94cb5ac640 1 -- 172.21.15.1:0/2832076338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c4157b90 msgr2=0x7f94c40c54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:56.160+0000 7f94cb5ac640 1 --2- 172.21.15.1:0/2832076338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c4157b90 0x7f94c40c54e0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009f70 tx=0x7f94bc004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:56.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:56.160+0000 7f94cb5ac640 1 -- 172.21.15.1:0/2832076338 shutdown_connections 2024-03-06T07:32:56.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:56.160+0000 7f94cb5ac640 1 --2- 172.21.15.1:0/2832076338 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f94a806b4d0 0x7f94a806d990 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:56.160+0000 7f94cb5ac640 1 --2- 172.21.15.1:0/2832076338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94c4157b90 0x7f94c40c54e0 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:56.160+0000 7f94cb5ac640 1 --2- 172.21.15.1:0/2832076338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c4152650 0x7f94c414d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:56.160+0000 7f94cb5ac640 1 --2- 172.21.15.1:0/2832076338 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f94c40bf9d0 0x7f94c40c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:56.160+0000 7f94cb5ac640 1 -- 172.21.15.1:0/2832076338 >> 172.21.15.1:0/2832076338 conn(0x7f94c4066ee0 msgr2=0x7f94c40cae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:56.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:56.160+0000 7f94cb5ac640 1 -- 172.21.15.1:0/2832076338 shutdown_connections 2024-03-06T07:32:56.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:56.160+0000 7f94cb5ac640 1 -- 172.21.15.1:0/2832076338 wait complete. 2024-03-06T07:32:56.575 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:56.575 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:57.576 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:57.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.676+0000 7f0cf6972640 1 Processor -- start 2024-03-06T07:32:57.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.676+0000 7f0cf6972640 1 -- start start 2024-03-06T07:32:57.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.677+0000 7f0cf6972640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cf00c6b10 0x7f0cf0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.677+0000 7f0cf6972640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0cf00bf9d0 0x7f0cf00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.677+0000 7f0cf6972640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf00c2fc0 con 0x7f0cf0152650 2024-03-06T07:32:57.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.677+0000 7f0cf6972640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf0155800 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.677+0000 7f0cf6972640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf0158da0 con 0x7f0cf00c6b10 2024-03-06T07:32:57.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.677+0000 7f0cf4f0f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0cf00bf9d0 0x7f0cf00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.677+0000 7f0ceffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0cf0152650 0x7f0cf0158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36228/0 (socket says 172.21.15.1:36228) 2024-03-06T07:32:57.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.677+0000 7f0cf4f0f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0cf00bf9d0 0x7f0cf00c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50706/0 (socket says 172.21.15.1:50706) 2024-03-06T07:32:57.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.677+0000 7f0ceffff640 1 -- 172.21.15.1:0/3613367346 learned_addr learned my addr 172.21.15.1:0/3613367346 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:57.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.677+0000 7f0cef7fe640 1 --2- 172.21.15.1:0/3613367346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cf00c6b10 0x7f0cf0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.677+0000 7f0cf4f0f640 1 -- 172.21.15.1:0/3613367346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cf00c6b10 msgr2=0x7f0cf0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:57.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.677+0000 7f0cf4f0f640 1 --2- 172.21.15.1:0/3613367346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cf00c6b10 0x7f0cf0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.677+0000 7f0cf4f0f640 1 -- 172.21.15.1:0/3613367346 >> v1:172.21.15.1:6789/0 conn(0x7f0cf0152650 legacy=0x7f0cf0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.677+0000 7f0cf4f0f640 1 -- 172.21.15.1:0/3613367346 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf00c6690 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.677+0000 7f0ceeffd640 1 -- 172.21.15.1:0/3613367346 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077646136 0 0) 0x7f0cf00c2fc0 con 0x7f0cf0152650 2024-03-06T07:32:57.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.678+0000 7f0cf4f0f640 1 --2- 172.21.15.1:0/3613367346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00bf9d0 0x7f0cf00c1460 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009d90 tx=0x7f0ce802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9094dc7ef7df198f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.678+0000 7f0ceeffd640 1 -- 172.21.15.1:0/3613367346 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8039070 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.678+0000 7f0ceeffd640 1 -- 172.21.15.1:0/3613367346 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce802cbc0 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.678+0000 7f0ceeffd640 1 -- 172.21.15.1:0/3613367346 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8034440 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.679+0000 7f0cf6972640 1 -- 172.21.15.1:0/3613367346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00bf9d0 msgr2=0x7f0cf00c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.679+0000 7f0cf6972640 1 --2- 172.21.15.1:0/3613367346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00bf9d0 0x7f0cf00c1460 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009d90 tx=0x7f0ce802be10 comp rx=0 tx=0).stop 2024-03-06T07:32:57.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.679+0000 7f0cf6972640 1 -- 172.21.15.1:0/3613367346 shutdown_connections 2024-03-06T07:32:57.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.679+0000 7f0cf6972640 1 --2- 172.21.15.1:0/3613367346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cf00c6b10 0x7f0cf0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.679+0000 7f0cf6972640 1 --2- 172.21.15.1:0/3613367346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00bf9d0 0x7f0cf00c1460 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.679+0000 7f0cf6972640 1 -- 172.21.15.1:0/3613367346 >> 172.21.15.1:0/3613367346 conn(0x7f0cf0066ee0 msgr2=0x7f0cf00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:57.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.679+0000 7f0cf6972640 1 -- 172.21.15.1:0/3613367346 shutdown_connections 2024-03-06T07:32:57.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.679+0000 7f0cf6972640 1 -- 172.21.15.1:0/3613367346 wait complete. 2024-03-06T07:32:57.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.680+0000 7f0cf6972640 1 Processor -- start 2024-03-06T07:32:57.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.680+0000 7f0cf6972640 1 -- start start 2024-03-06T07:32:57.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0cf6972640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00bf9d0 0x7f0cf014e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0cf6972640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cf0152650 0x7f0cf014ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0cf6972640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cf014d8a0 0x7f0cf014d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0cf6972640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf01551b0 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0cf6972640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf00c6fa0 con 0x7f0cf014d8a0 2024-03-06T07:32:57.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0cf6972640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf0155490 con 0x7f0cf0152650 2024-03-06T07:32:57.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0ceffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00bf9d0 0x7f0cf014e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0ceffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00bf9d0 0x7f0cf014e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50714/0 (socket says 172.21.15.1:50714) 2024-03-06T07:32:57.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0ceffff640 1 -- 172.21.15.1:0/750035031 learned_addr learned my addr 172.21.15.1:0/750035031 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:57.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0cef7fe640 1 --2- 172.21.15.1:0/750035031 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cf0152650 0x7f0cf014ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0cf4f0f640 1 --2- 172.21.15.1:0/750035031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cf014d8a0 0x7f0cf014d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0ceffff640 1 -- 172.21.15.1:0/750035031 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cf0152650 msgr2=0x7f0cf014ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0ceffff640 1 --2- 172.21.15.1:0/750035031 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cf0152650 0x7f0cf014ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0ceffff640 1 -- 172.21.15.1:0/750035031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cf014d8a0 msgr2=0x7f0cf014d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0ceffff640 1 --2- 172.21.15.1:0/750035031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cf014d8a0 0x7f0cf014d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0ceffff640 1 -- 172.21.15.1:0/750035031 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf01588c0 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0cef7fe640 1 --2- 172.21.15.1:0/750035031 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cf0152650 0x7f0cf014ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0ceffff640 1 --2- 172.21.15.1:0/750035031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00bf9d0 0x7f0cf014e130 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f0ce000c4d0 tx=0x7f0ce000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0cecff9640 1 -- 172.21.15.1:0/750035031 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0018070 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0cecff9640 1 -- 172.21.15.1:0/750035031 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce0007500 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0cf6972640 1 -- 172.21.15.1:0/750035031 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf0156490 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.681+0000 7f0cecff9640 1 -- 172.21.15.1:0/750035031 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0004a20 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.682+0000 7f0cf6972640 1 -- 172.21.15.1:0/750035031 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf00ca130 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.682+0000 7f0cf6972640 1 -- 172.21.15.1:0/750035031 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf006b540 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.683+0000 7f0cecff9640 1 -- 172.21.15.1:0/750035031 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ce001b070 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.684+0000 7f0cecff9640 1 --2- 172.21.15.1:0/750035031 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0cdc06b500 0x7f0cdc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.684+0000 7f0cecff9640 1 -- 172.21.15.1:0/750035031 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f0ce00896e0 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.684+0000 7f0cef7fe640 1 --2- 172.21.15.1:0/750035031 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0cdc06b500 0x7f0cdc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.685+0000 7f0cef7fe640 1 --2- 172.21.15.1:0/750035031 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0cdc06b500 0x7f0cdc06d9c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f0cf00c5970 tx=0x7f0ce4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.688+0000 7f0cecff9640 1 -- 172.21.15.1:0/750035031 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0cf006b540 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.907+0000 7f0cf6972640 1 -- 172.21.15.1:0/750035031 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cf00b06f0 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.907+0000 7f0cecff9640 1 -- 172.21.15.1:0/750035031 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f0ce0058360 con 0x7f0cf00bf9d0 2024-03-06T07:32:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:57.910 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:57.910 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:57.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.910+0000 7f0cf6972640 1 -- 172.21.15.1:0/750035031 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0cdc06b500 msgr2=0x7f0cdc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.910+0000 7f0cf6972640 1 --2- 172.21.15.1:0/750035031 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0cdc06b500 0x7f0cdc06d9c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f0cf00c5970 tx=0x7f0ce4008040 comp rx=0 tx=0).stop 2024-03-06T07:32:57.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.910+0000 7f0cf6972640 1 -- 172.21.15.1:0/750035031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00bf9d0 msgr2=0x7f0cf014e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.910+0000 7f0cf6972640 1 --2- 172.21.15.1:0/750035031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00bf9d0 0x7f0cf014e130 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f0ce000c4d0 tx=0x7f0ce000c990 comp rx=0 tx=0).stop 2024-03-06T07:32:57.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.910+0000 7f0cf6972640 1 -- 172.21.15.1:0/750035031 shutdown_connections 2024-03-06T07:32:57.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.910+0000 7f0cf6972640 1 --2- 172.21.15.1:0/750035031 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0cdc06b500 0x7f0cdc06d9c0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.910+0000 7f0cf6972640 1 --2- 172.21.15.1:0/750035031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cf014d8a0 0x7f0cf014d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.910+0000 7f0cf6972640 1 --2- 172.21.15.1:0/750035031 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cf0152650 0x7f0cf014ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.910+0000 7f0cf6972640 1 --2- 172.21.15.1:0/750035031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00bf9d0 0x7f0cf014e130 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.910+0000 7f0cf6972640 1 -- 172.21.15.1:0/750035031 >> 172.21.15.1:0/750035031 conn(0x7f0cf0066ee0 msgr2=0x7f0cf00c5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:57.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.910+0000 7f0cf6972640 1 -- 172.21.15.1:0/750035031 shutdown_connections 2024-03-06T07:32:57.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:57.910+0000 7f0cf6972640 1 -- 172.21.15.1:0/750035031 wait complete. 2024-03-06T07:32:58.325 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:32:58.326 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:32:59.327 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:59.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.426+0000 7f902e40f640 1 Processor -- start 2024-03-06T07:32:59.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.426+0000 7f902e40f640 1 -- start start 2024-03-06T07:32:59.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.427+0000 7f902e40f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90280c1390 0x7f90280c17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.427+0000 7f902e40f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f90281552e0 0x7f9028155750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.427+0000 7f902e40f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90281588e0 con 0x7f9028152730 2024-03-06T07:32:59.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.427+0000 7f902e40f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90280c1d30 con 0x7f90281552e0 2024-03-06T07:32:59.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.427+0000 7f902e40f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9028158bc0 con 0x7f90280c1390 2024-03-06T07:32:59.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.427+0000 7f9027fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9028152730 0x7f90280c6a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36244/0 (socket says 172.21.15.1:36244) 2024-03-06T07:32:59.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.427+0000 7f9027fff640 1 -- 172.21.15.1:0/2578222715 learned_addr learned my addr 172.21.15.1:0/2578222715 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:32:59.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.427+0000 7f90277fe640 1 --2- 172.21.15.1:0/2578222715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90280c1390 0x7f90280c17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f902c9ac640 1 --2- 172.21.15.1:0/2578222715 >> v2:172.21.15.98:3300/0 conn(0x7f90281552e0 0x7f9028155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f9026ffd640 1 -- 172.21.15.1:0/2578222715 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082792125 0 0) 0x7f90281588e0 con 0x7f9028152730 2024-03-06T07:32:59.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f9026ffd640 1 -- 172.21.15.1:0/2578222715 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9010003520 con 0x7f9028152730 2024-03-06T07:32:59.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f9026ffd640 1 -- 172.21.15.1:0/2578222715 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4114912770 0 0) 0x7f9010003520 con 0x7f9028152730 2024-03-06T07:32:59.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f9026ffd640 1 -- 172.21.15.1:0/2578222715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90280c1390 msgr2=0x7f90280c17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f9026ffd640 1 --2- 172.21.15.1:0/2578222715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90280c1390 0x7f90280c17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f9026ffd640 1 -- 172.21.15.1:0/2578222715 >> v2:172.21.15.98:3300/0 conn(0x7f90281552e0 msgr2=0x7f9028155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f9026ffd640 1 --2- 172.21.15.1:0/2578222715 >> v2:172.21.15.98:3300/0 conn(0x7f90281552e0 0x7f9028155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f9026ffd640 1 -- 172.21.15.1:0/2578222715 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9028158ea0 con 0x7f9028152730 2024-03-06T07:32:59.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f9026ffd640 1 -- 172.21.15.1:0/2578222715 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f9018001690 con 0x7f9028152730 2024-03-06T07:32:59.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f9026ffd640 1 -- 172.21.15.1:0/2578222715 >> v1:172.21.15.1:6789/0 conn(0x7f9028152730 legacy=0x7f90280c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f9026ffd640 1 --2- 172.21.15.1:0/2578222715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90280c1390 0x7f90100039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f9026ffd640 1 --2- 172.21.15.1:0/2578222715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9010003f30 0x7f9010004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f9026ffd640 1 --2- 172.21.15.1:0/2578222715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9010004d40 0x7f90100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f9026ffd640 1 -- 172.21.15.1:0/2578222715 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90280c1d30 con 0x7f9010003f30 2024-03-06T07:32:59.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f9026ffd640 1 -- 172.21.15.1:0/2578222715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9028158bc0 con 0x7f9010004d40 2024-03-06T07:32:59.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f9026ffd640 1 -- 172.21.15.1:0/2578222715 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901000baa0 con 0x7f90280c1390 2024-03-06T07:32:59.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f90277fe640 1 --2- 172.21.15.1:0/2578222715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9010003f30 0x7f9010004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f9027fff640 1 --2- 172.21.15.1:0/2578222715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90280c1390 0x7f90100039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f902c9ac640 1 --2- 172.21.15.1:0/2578222715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9010004d40 0x7f90100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f90277fe640 1 -- 172.21.15.1:0/2578222715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90280c1390 msgr2=0x7f90100039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f90277fe640 1 --2- 172.21.15.1:0/2578222715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f90280c1390 0x7f90100039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f90277fe640 1 -- 172.21.15.1:0/2578222715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9010004d40 msgr2=0x7f90100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f90277fe640 1 --2- 172.21.15.1:0/2578222715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9010004d40 0x7f90100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.428+0000 7f90277fe640 1 -- 172.21.15.1:0/2578222715 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9010003520 con 0x7f9010003f30 2024-03-06T07:32:59.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.429+0000 7f90277fe640 1 --2- 172.21.15.1:0/2578222715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9010003f30 0x7f9010004800 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f901c009ac0 tx=0x7f9010005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e54c98da7a75468 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.429+0000 7f9026ffd640 1 -- 172.21.15.1:0/2578222715 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c039070 con 0x7f9010003f30 2024-03-06T07:32:59.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.429+0000 7f9026ffd640 1 -- 172.21.15.1:0/2578222715 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901c02c6b0 con 0x7f9010003f30 2024-03-06T07:32:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.429+0000 7f9026ffd640 1 -- 172.21.15.1:0/2578222715 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c009690 con 0x7f9010003f30 2024-03-06T07:32:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.430+0000 7f902e40f640 1 -- 172.21.15.1:0/2578222715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9010003f30 msgr2=0x7f9010004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.430+0000 7f902e40f640 1 --2- 172.21.15.1:0/2578222715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9010003f30 0x7f9010004800 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f901c009ac0 tx=0x7f9010005830 comp rx=0 tx=0).stop 2024-03-06T07:32:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.430+0000 7f902c9ac640 1 -- 172.21.15.1:0/2578222715 reap_dead start 2024-03-06T07:32:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.430+0000 7f902e40f640 1 -- 172.21.15.1:0/2578222715 shutdown_connections 2024-03-06T07:32:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.430+0000 7f902e40f640 1 -- 172.21.15.1:0/2578222715 >> 172.21.15.1:0/2578222715 conn(0x7f9028066d00 msgr2=0x7f9028067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.430+0000 7f902e40f640 1 -- 172.21.15.1:0/2578222715 shutdown_connections 2024-03-06T07:32:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.430+0000 7f902e40f640 1 -- 172.21.15.1:0/2578222715 wait complete. 2024-03-06T07:32:59.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.431+0000 7f902e40f640 1 Processor -- start 2024-03-06T07:32:59.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.431+0000 7f902e40f640 1 -- start start 2024-03-06T07:32:59.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f902e40f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9010003f30 0x7f90280c91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f902e40f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9028157800 0x7f9028148640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f902e40f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9028148b80 0x7f90280c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f902e40f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9028152b10 con 0x7f9010003f30 2024-03-06T07:32:59.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f902e40f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9028159460 con 0x7f9028148b80 2024-03-06T07:32:59.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f902e40f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90280c71a0 con 0x7f9028157800 2024-03-06T07:32:59.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f9027fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9010003f30 0x7f90280c91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f9027fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9010003f30 0x7f90280c91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50744/0 (socket says 172.21.15.1:50744) 2024-03-06T07:32:59.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f9027fff640 1 -- 172.21.15.1:0/3607029110 learned_addr learned my addr 172.21.15.1:0/3607029110 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:32:59.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f90277fe640 1 --2- 172.21.15.1:0/3607029110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9028157800 0x7f9028148640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f902c9ac640 1 --2- 172.21.15.1:0/3607029110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9028148b80 0x7f90280c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f9027fff640 1 -- 172.21.15.1:0/3607029110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9028157800 msgr2=0x7f9028148640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f9027fff640 1 --2- 172.21.15.1:0/3607029110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9028157800 0x7f9028148640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f9027fff640 1 -- 172.21.15.1:0/3607029110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9028148b80 msgr2=0x7f90280c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f9027fff640 1 --2- 172.21.15.1:0/3607029110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9028148b80 0x7f90280c9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f9027fff640 1 -- 172.21.15.1:0/3607029110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9028149000 con 0x7f9010003f30 2024-03-06T07:32:59.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f90277fe640 1 --2- 172.21.15.1:0/3607029110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9028157800 0x7f9028148640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:59.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.432+0000 7f9027fff640 1 --2- 172.21.15.1:0/3607029110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9010003f30 0x7f90280c91f0 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f90180072c0 tx=0x7f9018007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.433+0000 7f9024ff9640 1 -- 172.21.15.1:0/3607029110 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901800f960 con 0x7f9010003f30 2024-03-06T07:32:59.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.433+0000 7f9024ff9640 1 -- 172.21.15.1:0/3607029110 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9018018070 con 0x7f9010003f30 2024-03-06T07:32:59.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.433+0000 7f902e40f640 1 -- 172.21.15.1:0/3607029110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90281ea880 con 0x7f9010003f30 2024-03-06T07:32:59.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.433+0000 7f9024ff9640 1 -- 172.21.15.1:0/3607029110 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901800fd60 con 0x7f9010003f30 2024-03-06T07:32:59.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.433+0000 7f902e40f640 1 -- 172.21.15.1:0/3607029110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90281547c0 con 0x7f9010003f30 2024-03-06T07:32:59.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.434+0000 7f902e40f640 1 -- 172.21.15.1:0/3607029110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9028152dd0 con 0x7f9010003f30 2024-03-06T07:32:59.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.434+0000 7f9024ff9640 1 -- 172.21.15.1:0/3607029110 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f901801d460 con 0x7f9010003f30 2024-03-06T07:32:59.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.434+0000 7f9024ff9640 1 --2- 172.21.15.1:0/3607029110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f901406b4d0 0x7f901406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.435+0000 7f9024ff9640 1 -- 172.21.15.1:0/3607029110 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f901808a430 con 0x7f9010003f30 2024-03-06T07:32:59.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.435+0000 7f90277fe640 1 --2- 172.21.15.1:0/3607029110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f901406b4d0 0x7f901406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.435+0000 7f90277fe640 1 --2- 172.21.15.1:0/3607029110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f901406b4d0 0x7f901406d990 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f901c036000 tx=0x7f901c030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.439+0000 7f9024ff9640 1 -- 172.21.15.1:0/3607029110 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9028152dd0 con 0x7f9010003f30 2024-03-06T07:32:59.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.658+0000 7f902e40f640 1 -- 172.21.15.1:0/3607029110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9028067fd0 con 0x7f9010003f30 2024-03-06T07:32:59.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.659+0000 7f9024ff9640 1 -- 172.21.15.1:0/3607029110 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f9028149000 con 0x7f9010003f30 2024-03-06T07:32:59.661 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:32:59.661 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:32:59.661 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:32:59.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.660+0000 7f902e40f640 1 -- 172.21.15.1:0/3607029110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f901406b4d0 msgr2=0x7f901406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.661+0000 7f902e40f640 1 --2- 172.21.15.1:0/3607029110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f901406b4d0 0x7f901406d990 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f901c036000 tx=0x7f901c030000 comp rx=0 tx=0).stop 2024-03-06T07:32:59.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.661+0000 7f902e40f640 1 -- 172.21.15.1:0/3607029110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9010003f30 msgr2=0x7f90280c91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.661+0000 7f902e40f640 1 --2- 172.21.15.1:0/3607029110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9010003f30 0x7f90280c91f0 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f90180072c0 tx=0x7f9018007780 comp rx=0 tx=0).stop 2024-03-06T07:32:59.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.661+0000 7f902e40f640 1 -- 172.21.15.1:0/3607029110 shutdown_connections 2024-03-06T07:32:59.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.661+0000 7f902e40f640 1 --2- 172.21.15.1:0/3607029110 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f901406b4d0 0x7f901406d990 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.661+0000 7f902e40f640 1 --2- 172.21.15.1:0/3607029110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9028148b80 0x7f90280c9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.661+0000 7f902e40f640 1 --2- 172.21.15.1:0/3607029110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9028157800 0x7f9028148640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.661+0000 7f902e40f640 1 --2- 172.21.15.1:0/3607029110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9010003f30 0x7f90280c91f0 unknown :-1 s=CLOSED pgs=2645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.661+0000 7f902e40f640 1 -- 172.21.15.1:0/3607029110 >> 172.21.15.1:0/3607029110 conn(0x7f9028066d00 msgr2=0x7f90280c3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.661+0000 7f902e40f640 1 -- 172.21.15.1:0/3607029110 shutdown_connections 2024-03-06T07:32:59.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:32:59.661+0000 7f902e40f640 1 -- 172.21.15.1:0/3607029110 wait complete. 2024-03-06T07:33:00.076 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:33:00.076 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:33:01.077 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:01.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.176+0000 7ff8bcb8a640 1 Processor -- start 2024-03-06T07:33:01.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.177+0000 7ff8bcb8a640 1 -- start start 2024-03-06T07:33:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.177+0000 7ff8bcb8a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff8b80c6b10 0x7ff8b8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.177+0000 7ff8bcb8a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff8b80bf9d0 0x7ff8b80c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.177+0000 7ff8bcb8a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b80c2fc0 con 0x7ff8b8152650 2024-03-06T07:33:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.177+0000 7ff8bcb8a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8b8155800 con 0x7ff8b80c6b10 2024-03-06T07:33:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.177+0000 7ff8bcb8a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8b8158da0 con 0x7ff8b80bf9d0 2024-03-06T07:33:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.178+0000 7ff8b5d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff8b80c6b10 0x7ff8b8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.178+0000 7ff8b5d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff8b80c6b10 0x7ff8b8155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50758/0 (socket says 172.21.15.1:50758) 2024-03-06T07:33:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.178+0000 7ff8b5d9b640 1 -- 172.21.15.1:0/1663840331 learned_addr learned my addr 172.21.15.1:0/1663840331 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.178+0000 7ff8b6d9d640 1 --2- 172.21.15.1:0/1663840331 >> v2:172.21.15.98:3300/0 conn(0x7ff8b80bf9d0 0x7ff8b80c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.178+0000 7ff8b5d9b640 1 -- 172.21.15.1:0/1663840331 >> v2:172.21.15.98:3300/0 conn(0x7ff8b80bf9d0 msgr2=0x7ff8b80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.178+0000 7ff8b5d9b640 1 --2- 172.21.15.1:0/1663840331 >> v2:172.21.15.98:3300/0 conn(0x7ff8b80bf9d0 0x7ff8b80c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.178+0000 7ff8b5d9b640 1 -- 172.21.15.1:0/1663840331 >> v1:172.21.15.98:6789/0 conn(0x7ff8b8152650 legacy=0x7ff8b8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.178+0000 7ff8b5d9b640 1 -- 172.21.15.1:0/1663840331 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b80c6690 con 0x7ff8b80c6b10 2024-03-06T07:33:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.178+0000 7ff8b6d9d640 1 --2- 172.21.15.1:0/1663840331 >> v2:172.21.15.98:3300/0 conn(0x7ff8b80bf9d0 0x7ff8b80c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:01.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.179+0000 7ff8b5d9b640 1 --2- 172.21.15.1:0/1663840331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8b80c6b10 0x7ff8b8155c90 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7ff8b81597b0 tx=0x7ff8ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b682176d5613bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.179+0000 7ff8b559a640 1 -- 172.21.15.1:0/1663840331 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac035020 con 0x7ff8b80c6b10 2024-03-06T07:33:01.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.179+0000 7ff8b559a640 1 -- 172.21.15.1:0/1663840331 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ac031870 con 0x7ff8b80c6b10 2024-03-06T07:33:01.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.179+0000 7ff8b559a640 1 -- 172.21.15.1:0/1663840331 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac031c40 con 0x7ff8b80c6b10 2024-03-06T07:33:01.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.180+0000 7ff8bcb8a640 1 -- 172.21.15.1:0/1663840331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8b80c6b10 msgr2=0x7ff8b8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.180+0000 7ff8bcb8a640 1 --2- 172.21.15.1:0/1663840331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8b80c6b10 0x7ff8b8155c90 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7ff8b81597b0 tx=0x7ff8ac02b910 comp rx=0 tx=0).stop 2024-03-06T07:33:01.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.180+0000 7ff8bcb8a640 1 -- 172.21.15.1:0/1663840331 shutdown_connections 2024-03-06T07:33:01.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.180+0000 7ff8bcb8a640 1 --2- 172.21.15.1:0/1663840331 >> v2:172.21.15.98:3300/0 conn(0x7ff8b80bf9d0 0x7ff8b80c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.180+0000 7ff8bcb8a640 1 --2- 172.21.15.1:0/1663840331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8b80c6b10 0x7ff8b8155c90 unknown :-1 s=CLOSED pgs=2646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.180+0000 7ff8bcb8a640 1 -- 172.21.15.1:0/1663840331 >> 172.21.15.1:0/1663840331 conn(0x7ff8b8066ee0 msgr2=0x7ff8b80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:01.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.180+0000 7ff8bcb8a640 1 -- 172.21.15.1:0/1663840331 shutdown_connections 2024-03-06T07:33:01.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.180+0000 7ff8bcb8a640 1 -- 172.21.15.1:0/1663840331 wait complete. 2024-03-06T07:33:01.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.181+0000 7ff8bcb8a640 1 Processor -- start 2024-03-06T07:33:01.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.181+0000 7ff8bcb8a640 1 -- start start 2024-03-06T07:33:01.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8bcb8a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8b80bf9d0 0x7ff8b80c5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8bcb8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b8152650 0x7ff8b81546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8bcb8a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8b8154be0 0x7ff8b814ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8bcb8a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b81551b0 con 0x7ff8b80bf9d0 2024-03-06T07:33:01.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8bcb8a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b80c6fa0 con 0x7ff8b8152650 2024-03-06T07:33:01.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8bcb8a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b8155490 con 0x7ff8b8154be0 2024-03-06T07:33:01.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8b659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8b80bf9d0 0x7ff8b80c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8b659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8b80bf9d0 0x7ff8b80c5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50762/0 (socket says 172.21.15.1:50762) 2024-03-06T07:33:01.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8b659c640 1 -- 172.21.15.1:0/4026821950 learned_addr learned my addr 172.21.15.1:0/4026821950 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:01.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8b5d9b640 1 --2- 172.21.15.1:0/4026821950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b8152650 0x7ff8b81546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8b6d9d640 1 --2- 172.21.15.1:0/4026821950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8b8154be0 0x7ff8b814ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8b659c640 1 -- 172.21.15.1:0/4026821950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8b8154be0 msgr2=0x7ff8b814ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8b659c640 1 --2- 172.21.15.1:0/4026821950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8b8154be0 0x7ff8b814ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8b659c640 1 -- 172.21.15.1:0/4026821950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b8152650 msgr2=0x7ff8b81546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8b659c640 1 --2- 172.21.15.1:0/4026821950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b8152650 0x7ff8b81546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8b659c640 1 -- 172.21.15.1:0/4026821950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b81ebea0 con 0x7ff8b80bf9d0 2024-03-06T07:33:01.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8b659c640 1 --2- 172.21.15.1:0/4026821950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8b80bf9d0 0x7ff8b80c5520 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800c430 tx=0x7ff8a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8a6ffd640 1 -- 172.21.15.1:0/4026821950 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a80052d0 con 0x7ff8b80bf9d0 2024-03-06T07:33:01.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8a6ffd640 1 -- 172.21.15.1:0/4026821950 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a8005490 con 0x7ff8b80bf9d0 2024-03-06T07:33:01.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8bcb8a640 1 -- 172.21.15.1:0/4026821950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b81e1530 con 0x7ff8b80bf9d0 2024-03-06T07:33:01.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.182+0000 7ff8a6ffd640 1 -- 172.21.15.1:0/4026821950 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8007260 con 0x7ff8b80bf9d0 2024-03-06T07:33:01.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.183+0000 7ff8bcb8a640 1 -- 172.21.15.1:0/4026821950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b8152a30 con 0x7ff8b80bf9d0 2024-03-06T07:33:01.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.183+0000 7ff8bcb8a640 1 -- 172.21.15.1:0/4026821950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b8067fd0 con 0x7ff8b80bf9d0 2024-03-06T07:33:01.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.184+0000 7ff8a6ffd640 1 -- 172.21.15.1:0/4026821950 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8a8007400 con 0x7ff8b80bf9d0 2024-03-06T07:33:01.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.184+0000 7ff8a6ffd640 1 --2- 172.21.15.1:0/4026821950 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff89806b500 0x7ff89806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.184+0000 7ff8a6ffd640 1 -- 172.21.15.1:0/4026821950 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ff8a8089880 con 0x7ff8b80bf9d0 2024-03-06T07:33:01.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.185+0000 7ff8b5d9b640 1 --2- 172.21.15.1:0/4026821950 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff89806b500 0x7ff89806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.185+0000 7ff8b5d9b640 1 --2- 172.21.15.1:0/4026821950 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff89806b500 0x7ff89806d9c0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac036000 tx=0x7ff8ac03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.189+0000 7ff8a6ffd640 1 -- 172.21.15.1:0/4026821950 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8a8057eb0 con 0x7ff8b80bf9d0 2024-03-06T07:33:01.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.412+0000 7ff8bcb8a640 1 -- 172.21.15.1:0/4026821950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8b8151e00 con 0x7ff8b80bf9d0 2024-03-06T07:33:01.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.412+0000 7ff8a6ffd640 1 -- 172.21.15.1:0/4026821950 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7ff8a805bf50 con 0x7ff8b80bf9d0 2024-03-06T07:33:01.414 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:01.415 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:33:01.415 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:33:01.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.414+0000 7ff8bcb8a640 1 -- 172.21.15.1:0/4026821950 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff89806b500 msgr2=0x7ff89806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.414+0000 7ff8bcb8a640 1 --2- 172.21.15.1:0/4026821950 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff89806b500 0x7ff89806d9c0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac036000 tx=0x7ff8ac03c040 comp rx=0 tx=0).stop 2024-03-06T07:33:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.414+0000 7ff8bcb8a640 1 -- 172.21.15.1:0/4026821950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8b80bf9d0 msgr2=0x7ff8b80c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.414+0000 7ff8bcb8a640 1 --2- 172.21.15.1:0/4026821950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8b80bf9d0 0x7ff8b80c5520 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800c430 tx=0x7ff8a800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.414+0000 7ff8bcb8a640 1 -- 172.21.15.1:0/4026821950 shutdown_connections 2024-03-06T07:33:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.414+0000 7ff8bcb8a640 1 --2- 172.21.15.1:0/4026821950 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff89806b500 0x7ff89806d9c0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.414+0000 7ff8bcb8a640 1 --2- 172.21.15.1:0/4026821950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8b8154be0 0x7ff8b814ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.414+0000 7ff8bcb8a640 1 --2- 172.21.15.1:0/4026821950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b8152650 0x7ff8b81546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.414+0000 7ff8bcb8a640 1 --2- 172.21.15.1:0/4026821950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8b80bf9d0 0x7ff8b80c5520 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.414+0000 7ff8bcb8a640 1 -- 172.21.15.1:0/4026821950 >> 172.21.15.1:0/4026821950 conn(0x7ff8b8066ee0 msgr2=0x7ff8b80cb090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.414+0000 7ff8bcb8a640 1 -- 172.21.15.1:0/4026821950 shutdown_connections 2024-03-06T07:33:01.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:01.414+0000 7ff8bcb8a640 1 -- 172.21.15.1:0/4026821950 wait complete. 2024-03-06T07:33:01.829 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:33:01.829 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:33:02.831 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:02.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.930+0000 7f75d5143640 1 Processor -- start 2024-03-06T07:33:02.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.930+0000 7f75d5143640 1 -- start start 2024-03-06T07:33:02.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.931+0000 7f75d5143640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f75d0150520 0x7f75d01543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.931+0000 7f75d5143640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d0150900 con 0x7f75d0070c80 2024-03-06T07:33:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.931+0000 7f75d5143640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d0071790 con 0x7f75d00c32f0 2024-03-06T07:33:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.931+0000 7f75d5143640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75d0153ed0 con 0x7f75d0150520 2024-03-06T07:33:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.931+0000 7f75ced9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f75d0150520 0x7f75d01543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.931+0000 7f75cf59e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f75d0070c80 0x7f75d015d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36258/0 (socket says 172.21.15.1:36258) 2024-03-06T07:33:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.931+0000 7f75cf59e640 1 -- 172.21.15.1:0/1052859337 learned_addr learned my addr 172.21.15.1:0/1052859337 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.931+0000 7f75ced9d640 1 -- 172.21.15.1:0/1052859337 >> v1:172.21.15.98:6789/0 conn(0x7f75d00c32f0 legacy=0x7f75d01537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.931+0000 7f75ced9d640 1 -- 172.21.15.1:0/1052859337 >> v1:172.21.15.1:6789/0 conn(0x7f75d0070c80 legacy=0x7f75d015d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.931+0000 7f75cdd9b640 1 -- 172.21.15.1:0/1052859337 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4650941 0 0) 0x7f75d0150900 con 0x7f75d0070c80 2024-03-06T07:33:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.931+0000 7f75ced9d640 1 -- 172.21.15.1:0/1052859337 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d0157050 con 0x7f75d0150520 2024-03-06T07:33:02.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.932+0000 7f75ced9d640 1 --2- 172.21.15.1:0/1052859337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75d0150520 0x7f75d01543a0 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f75c00097a0 tx=0x7f75c002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e5941bd1272712 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.933+0000 7f75cdd9b640 1 -- 172.21.15.1:0/1052859337 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c0035020 con 0x7f75d0150520 2024-03-06T07:33:02.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.933+0000 7f75cdd9b640 1 -- 172.21.15.1:0/1052859337 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c0031830 con 0x7f75d0150520 2024-03-06T07:33:02.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.933+0000 7f75cdd9b640 1 -- 172.21.15.1:0/1052859337 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c0031c00 con 0x7f75d0150520 2024-03-06T07:33:02.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.933+0000 7f75d5143640 1 -- 172.21.15.1:0/1052859337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75d0150520 msgr2=0x7f75d01543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.933+0000 7f75d5143640 1 --2- 172.21.15.1:0/1052859337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75d0150520 0x7f75d01543a0 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f75c00097a0 tx=0x7f75c002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.933+0000 7f75d5143640 1 -- 172.21.15.1:0/1052859337 shutdown_connections 2024-03-06T07:33:02.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.933+0000 7f75d5143640 1 --2- 172.21.15.1:0/1052859337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75d0150520 0x7f75d01543a0 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.933+0000 7f75d5143640 1 -- 172.21.15.1:0/1052859337 >> 172.21.15.1:0/1052859337 conn(0x7f75d00c88f0 msgr2=0x7f75d00c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:02.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.933+0000 7f75d5143640 1 -- 172.21.15.1:0/1052859337 shutdown_connections 2024-03-06T07:33:02.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.933+0000 7f75d5143640 1 -- 172.21.15.1:0/1052859337 wait complete. 2024-03-06T07:33:02.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.935+0000 7f75d5143640 1 Processor -- start 2024-03-06T07:33:02.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.935+0000 7f75d5143640 1 -- start start 2024-03-06T07:33:02.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75d5143640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f75d0070c80 0x7f75d0145ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75d5143640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75d00c32f0 0x7f75d01ebc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75d5143640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75d0150520 0x7f75d01482c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75d5143640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d0157330 con 0x7f75d00c32f0 2024-03-06T07:33:02.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75d5143640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d01532a0 con 0x7f75d0150520 2024-03-06T07:33:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75d5143640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d00c7ec0 con 0x7f75d0070c80 2024-03-06T07:33:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75ce59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75d00c32f0 0x7f75d01ebc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75ce59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75d00c32f0 0x7f75d01ebc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50784/0 (socket says 172.21.15.1:50784) 2024-03-06T07:33:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75ce59c640 1 -- 172.21.15.1:0/56483407 learned_addr learned my addr 172.21.15.1:0/56483407 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75cf59e640 1 --2- 172.21.15.1:0/56483407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75d0150520 0x7f75d01482c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75ced9d640 1 --2- 172.21.15.1:0/56483407 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f75d0070c80 0x7f75d0145ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75ce59c640 1 -- 172.21.15.1:0/56483407 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f75d0070c80 msgr2=0x7f75d0145ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75ce59c640 1 --2- 172.21.15.1:0/56483407 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f75d0070c80 0x7f75d0145ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75ce59c640 1 -- 172.21.15.1:0/56483407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75d0150520 msgr2=0x7f75d01482c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75ce59c640 1 --2- 172.21.15.1:0/56483407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75d0150520 0x7f75d01482c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75ce59c640 1 -- 172.21.15.1:0/56483407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d0147860 con 0x7f75d00c32f0 2024-03-06T07:33:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75ce59c640 1 --2- 172.21.15.1:0/56483407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75d00c32f0 0x7f75d01ebc60 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f75b800c460 tx=0x7f75b800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75c77fe640 1 -- 172.21.15.1:0/56483407 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b8016020 con 0x7f75d00c32f0 2024-03-06T07:33:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75c77fe640 1 -- 172.21.15.1:0/56483407 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b8012870 con 0x7f75d00c32f0 2024-03-06T07:33:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75c77fe640 1 -- 172.21.15.1:0/56483407 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b8012c60 con 0x7f75d00c32f0 2024-03-06T07:33:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.936+0000 7f75d5143640 1 -- 172.21.15.1:0/56483407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d01ec4a0 con 0x7f75d00c32f0 2024-03-06T07:33:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.937+0000 7f75d5143640 1 -- 172.21.15.1:0/56483407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d01ec1a0 con 0x7f75d00c32f0 2024-03-06T07:33:02.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.938+0000 7f75d5143640 1 -- 172.21.15.1:0/56483407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d0147a30 con 0x7f75d00c32f0 2024-03-06T07:33:02.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.938+0000 7f75c77fe640 1 -- 172.21.15.1:0/56483407 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75b8019020 con 0x7f75d00c32f0 2024-03-06T07:33:02.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.938+0000 7f75c77fe640 1 --2- 172.21.15.1:0/56483407 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75b006b4d0 0x7f75b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.938+0000 7f75c77fe640 1 -- 172.21.15.1:0/56483407 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f75b8089d90 con 0x7f75d00c32f0 2024-03-06T07:33:02.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.939+0000 7f75ced9d640 1 --2- 172.21.15.1:0/56483407 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75b006b4d0 0x7f75b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.939+0000 7f75ced9d640 1 --2- 172.21.15.1:0/56483407 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75b006b4d0 0x7f75b006d990 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f75c0036000 tx=0x7f75c003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:02.943+0000 7f75c77fe640 1 -- 172.21.15.1:0/56483407 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f75b8020070 con 0x7f75d00c32f0 2024-03-06T07:33:03.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:03.163+0000 7f75d5143640 1 -- 172.21.15.1:0/56483407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75d00686e0 con 0x7f75d00c32f0 2024-03-06T07:33:03.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:03.164+0000 7f75c77fe640 1 -- 172.21.15.1:0/56483407 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f75b8058340 con 0x7f75d00c32f0 2024-03-06T07:33:03.166 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:03.166 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:33:03.166 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:33:03.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:03.166+0000 7f75d5143640 1 -- 172.21.15.1:0/56483407 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75b006b4d0 msgr2=0x7f75b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:03.166+0000 7f75d5143640 1 --2- 172.21.15.1:0/56483407 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75b006b4d0 0x7f75b006d990 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f75c0036000 tx=0x7f75c003d040 comp rx=0 tx=0).stop 2024-03-06T07:33:03.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:03.166+0000 7f75d5143640 1 -- 172.21.15.1:0/56483407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75d00c32f0 msgr2=0x7f75d01ebc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:03.166+0000 7f75d5143640 1 --2- 172.21.15.1:0/56483407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75d00c32f0 0x7f75d01ebc60 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f75b800c460 tx=0x7f75b800c920 comp rx=0 tx=0).stop 2024-03-06T07:33:03.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:03.166+0000 7f75d5143640 1 -- 172.21.15.1:0/56483407 shutdown_connections 2024-03-06T07:33:03.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:03.166+0000 7f75d5143640 1 --2- 172.21.15.1:0/56483407 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f75b006b4d0 0x7f75b006d990 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:03.166+0000 7f75d5143640 1 --2- 172.21.15.1:0/56483407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75d0150520 0x7f75d01482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:03.166+0000 7f75d5143640 1 --2- 172.21.15.1:0/56483407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75d00c32f0 0x7f75d01ebc60 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:03.166+0000 7f75d5143640 1 --2- 172.21.15.1:0/56483407 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f75d0070c80 0x7f75d0145ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:03.166+0000 7f75d5143640 1 -- 172.21.15.1:0/56483407 >> 172.21.15.1:0/56483407 conn(0x7f75d00c88f0 msgr2=0x7f75d00c6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:03.166+0000 7f75d5143640 1 -- 172.21.15.1:0/56483407 shutdown_connections 2024-03-06T07:33:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:03.166+0000 7f75d5143640 1 -- 172.21.15.1:0/56483407 wait complete. 2024-03-06T07:33:03.440 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:03.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:03.582 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:33:03.582 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:33:04.583 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:04.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.696+0000 7f715ed25640 1 Processor -- start 2024-03-06T07:33:04.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.696+0000 7f715ed25640 1 -- start start 2024-03-06T07:33:04.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.697+0000 7f715ed25640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7158152650 0x7f7158155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.697+0000 7f715ed25640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71580bf9d0 0x7f71580c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.697+0000 7f715ed25640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71580c2fc0 con 0x7f71580c6b10 2024-03-06T07:33:04.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.697+0000 7f715ed25640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7158155800 con 0x7f7158152650 2024-03-06T07:33:04.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.697+0000 7f715ed25640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7158158da0 con 0x7f71580bf9d0 2024-03-06T07:33:04.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.697+0000 7f715cac1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7158152650 0x7f7158155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.697+0000 7f7157fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f71580c6b10 0x7f7158158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55788/0 (socket says 172.21.15.1:55788) 2024-03-06T07:33:04.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.697+0000 7f7157fff640 1 -- 172.21.15.1:0/179228264 learned_addr learned my addr 172.21.15.1:0/179228264 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:04.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.697+0000 7f715d2c2640 1 --2- 172.21.15.1:0/179228264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71580bf9d0 0x7f71580c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.697+0000 7f715cac1640 1 -- 172.21.15.1:0/179228264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71580bf9d0 msgr2=0x7f71580c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:04.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.697+0000 7f715cac1640 1 --2- 172.21.15.1:0/179228264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71580bf9d0 0x7f71580c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.697+0000 7f715cac1640 1 -- 172.21.15.1:0/179228264 >> v1:172.21.15.1:6789/0 conn(0x7f71580c6b10 legacy=0x7f7158158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.697+0000 7f715cac1640 1 -- 172.21.15.1:0/179228264 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71580c6690 con 0x7f7158152650 2024-03-06T07:33:04.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.697+0000 7f71577fe640 1 -- 172.21.15.1:0/179228264 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373546908 0 0) 0x7f71580c2fc0 con 0x7f71580c6b10 2024-03-06T07:33:04.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.697+0000 7f715d2c2640 1 --2- 172.21.15.1:0/179228264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71580bf9d0 0x7f71580c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:04.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.698+0000 7f715cac1640 1 --2- 172.21.15.1:0/179228264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7158152650 0x7f7158155c90 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f7150009d90 tx=0x7f715002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5ec3ce3b975b1d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.698+0000 7f71577fe640 1 -- 172.21.15.1:0/179228264 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7150039070 con 0x7f7158152650 2024-03-06T07:33:04.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.698+0000 7f71577fe640 1 -- 172.21.15.1:0/179228264 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715002cbc0 con 0x7f7158152650 2024-03-06T07:33:04.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.698+0000 7f71577fe640 1 -- 172.21.15.1:0/179228264 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7150034440 con 0x7f7158152650 2024-03-06T07:33:04.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.699+0000 7f715ed25640 1 -- 172.21.15.1:0/179228264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7158152650 msgr2=0x7f7158155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.699+0000 7f715ed25640 1 --2- 172.21.15.1:0/179228264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7158152650 0x7f7158155c90 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f7150009d90 tx=0x7f715002be10 comp rx=0 tx=0).stop 2024-03-06T07:33:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.699+0000 7f715ed25640 1 -- 172.21.15.1:0/179228264 shutdown_connections 2024-03-06T07:33:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.699+0000 7f715ed25640 1 --2- 172.21.15.1:0/179228264 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f71580bf9d0 0x7f71580c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.699+0000 7f715ed25640 1 --2- 172.21.15.1:0/179228264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7158152650 0x7f7158155c90 unknown :-1 s=CLOSED pgs=2652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.699+0000 7f715ed25640 1 -- 172.21.15.1:0/179228264 >> 172.21.15.1:0/179228264 conn(0x7f7158066ee0 msgr2=0x7f71580672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.699+0000 7f715ed25640 1 -- 172.21.15.1:0/179228264 shutdown_connections 2024-03-06T07:33:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.699+0000 7f715ed25640 1 -- 172.21.15.1:0/179228264 wait complete. 2024-03-06T07:33:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.700+0000 7f715ed25640 1 Processor -- start 2024-03-06T07:33:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.700+0000 7f715ed25640 1 -- start start 2024-03-06T07:33:04.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715ed25640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71580bf9d0 0x7f7158153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715ed25640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7158152650 0x7f71581de700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715ed25640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71581e1a20 0x7f71581dec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715ed25640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71581551b0 con 0x7f71580bf9d0 2024-03-06T07:33:04.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715ed25640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71580c6fa0 con 0x7f71581e1a20 2024-03-06T07:33:04.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715ed25640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7158155490 con 0x7f7158152650 2024-03-06T07:33:04.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715cac1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71580bf9d0 0x7f7158153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715cac1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71580bf9d0 0x7f7158153bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38934/0 (socket says 172.21.15.1:38934) 2024-03-06T07:33:04.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715cac1640 1 -- 172.21.15.1:0/3554012060 learned_addr learned my addr 172.21.15.1:0/3554012060 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:04.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f7157fff640 1 --2- 172.21.15.1:0/3554012060 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7158152650 0x7f71581de700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715d2c2640 1 --2- 172.21.15.1:0/3554012060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71581e1a20 0x7f71581dec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715cac1640 1 -- 172.21.15.1:0/3554012060 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7158152650 msgr2=0x7f71581de700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715cac1640 1 --2- 172.21.15.1:0/3554012060 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7158152650 0x7f71581de700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715cac1640 1 -- 172.21.15.1:0/3554012060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71581e1a20 msgr2=0x7f71581dec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715cac1640 1 --2- 172.21.15.1:0/3554012060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71581e1a20 0x7f71581dec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715cac1640 1 -- 172.21.15.1:0/3554012060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71581ee020 con 0x7f71580bf9d0 2024-03-06T07:33:04.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715d2c2640 1 --2- 172.21.15.1:0/3554012060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71581e1a20 0x7f71581dec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:04.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715cac1640 1 --2- 172.21.15.1:0/3554012060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71580bf9d0 0x7f7158153bc0 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f7150009d90 tx=0x7f7150004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f71557fa640 1 -- 172.21.15.1:0/3554012060 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7150039050 con 0x7f71580bf9d0 2024-03-06T07:33:04.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f71557fa640 1 -- 172.21.15.1:0/3554012060 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715003a040 con 0x7f71580bf9d0 2024-03-06T07:33:04.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f71557fa640 1 -- 172.21.15.1:0/3554012060 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715003e610 con 0x7f71580bf9d0 2024-03-06T07:33:04.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715ed25640 1 -- 172.21.15.1:0/3554012060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71581ee300 con 0x7f71580bf9d0 2024-03-06T07:33:04.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.701+0000 7f715ed25640 1 -- 172.21.15.1:0/3554012060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7158154100 con 0x7f71580bf9d0 2024-03-06T07:33:04.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.702+0000 7f715ed25640 1 -- 172.21.15.1:0/3554012060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7158067fd0 con 0x7f71580bf9d0 2024-03-06T07:33:04.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.703+0000 7f71557fa640 1 -- 172.21.15.1:0/3554012060 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7150047020 con 0x7f71580bf9d0 2024-03-06T07:33:04.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.703+0000 7f71557fa640 1 --2- 172.21.15.1:0/3554012060 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f713806b500 0x7f713806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.703+0000 7f71557fa640 1 -- 172.21.15.1:0/3554012060 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f71500ac7f0 con 0x7f71580bf9d0 2024-03-06T07:33:04.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.704+0000 7f7157fff640 1 --2- 172.21.15.1:0/3554012060 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f713806b500 0x7f713806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.704+0000 7f7157fff640 1 --2- 172.21.15.1:0/3554012060 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f713806b500 0x7f713806d9c0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f7158156de0 tx=0x7f7140009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.712+0000 7f71557fa640 1 -- 172.21.15.1:0/3554012060 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f715004e020 con 0x7f71580bf9d0 2024-03-06T07:33:04.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.931+0000 7f715ed25640 1 -- 172.21.15.1:0/3554012060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71580b06f0 con 0x7f71580bf9d0 2024-03-06T07:33:04.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.932+0000 7f71557fa640 1 -- 172.21.15.1:0/3554012060 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f715007b470 con 0x7f71580bf9d0 2024-03-06T07:33:04.934 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:04.934 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:33:04.934 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:33:04.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.934+0000 7f715ed25640 1 -- 172.21.15.1:0/3554012060 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f713806b500 msgr2=0x7f713806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.934+0000 7f715ed25640 1 --2- 172.21.15.1:0/3554012060 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f713806b500 0x7f713806d9c0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f7158156de0 tx=0x7f7140009040 comp rx=0 tx=0).stop 2024-03-06T07:33:04.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.934+0000 7f715ed25640 1 -- 172.21.15.1:0/3554012060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71580bf9d0 msgr2=0x7f7158153bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.934+0000 7f715ed25640 1 --2- 172.21.15.1:0/3554012060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71580bf9d0 0x7f7158153bc0 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f7150009d90 tx=0x7f7150004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:04.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.934+0000 7f715ed25640 1 -- 172.21.15.1:0/3554012060 shutdown_connections 2024-03-06T07:33:04.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.934+0000 7f715ed25640 1 --2- 172.21.15.1:0/3554012060 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f713806b500 0x7f713806d9c0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.934+0000 7f715ed25640 1 --2- 172.21.15.1:0/3554012060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71581e1a20 0x7f71581dec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.934+0000 7f715ed25640 1 --2- 172.21.15.1:0/3554012060 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7158152650 0x7f71581de700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.934+0000 7f715ed25640 1 --2- 172.21.15.1:0/3554012060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71580bf9d0 0x7f7158153bc0 unknown :-1 s=CLOSED pgs=2653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.934+0000 7f715ed25640 1 -- 172.21.15.1:0/3554012060 >> 172.21.15.1:0/3554012060 conn(0x7f7158066ee0 msgr2=0x7f71580c3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.934+0000 7f715ed25640 1 -- 172.21.15.1:0/3554012060 shutdown_connections 2024-03-06T07:33:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:04.934+0000 7f715ed25640 1 -- 172.21.15.1:0/3554012060 wait complete. 2024-03-06T07:33:05.349 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:33:05.349 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:33:06.350 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.451+0000 7f0203f13640 1 Processor -- start 2024-03-06T07:33:06.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.451+0000 7f0203f13640 1 -- start start 2024-03-06T07:33:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.452+0000 7f0203f13640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f01fc1528d0 0x7f01fc072b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.452+0000 7f0203f13640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01fc155330 0x7f01fc155770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.452+0000 7f0203f13640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01fc158a40 con 0x7f01fc071d00 2024-03-06T07:33:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.452+0000 7f0203f13640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01fc072370 con 0x7f01fc1528d0 2024-03-06T07:33:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.452+0000 7f0203f13640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01fc158d20 con 0x7f01fc155330 2024-03-06T07:33:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.452+0000 7f02014ae640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f01fc071d00 0x7f01fc06ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55798/0 (socket says 172.21.15.1:55798) 2024-03-06T07:33:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.452+0000 7f02014ae640 1 -- 172.21.15.1:0/4099418508 learned_addr learned my addr 172.21.15.1:0/4099418508 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.452+0000 7f0201caf640 1 --2- 172.21.15.1:0/4099418508 >> v2:172.21.15.98:3300/0 conn(0x7f01fc1528d0 0x7f01fc072b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.452+0000 7f02024b0640 1 --2- 172.21.15.1:0/4099418508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01fc155330 0x7f01fc155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.453+0000 7f0200cad640 1 -- 172.21.15.1:0/4099418508 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481455332 0 0) 0x7f01fc158a40 con 0x7f01fc071d00 2024-03-06T07:33:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.453+0000 7f0200cad640 1 -- 172.21.15.1:0/4099418508 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01e8003520 con 0x7f01fc071d00 2024-03-06T07:33:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.453+0000 7f0201caf640 1 -- 172.21.15.1:0/4099418508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01fc155330 msgr2=0x7f01fc155770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.453+0000 7f0201caf640 1 --2- 172.21.15.1:0/4099418508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01fc155330 0x7f01fc155770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.453+0000 7f0201caf640 1 -- 172.21.15.1:0/4099418508 >> v1:172.21.15.1:6789/0 conn(0x7f01fc071d00 legacy=0x7f01fc06ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.453+0000 7f0200cad640 1 -- 172.21.15.1:0/4099418508 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 213141949 0 0) 0x7f01e8003520 con 0x7f01fc071d00 2024-03-06T07:33:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.453+0000 7f0201caf640 1 -- 172.21.15.1:0/4099418508 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01fc159000 con 0x7f01fc1528d0 2024-03-06T07:33:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.454+0000 7f0201caf640 1 --2- 172.21.15.1:0/4099418508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01fc1528d0 0x7f01fc072b80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f01f80097a0 tx=0x7f01f802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcb0699ed10eba4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.454+0000 7f0200cad640 1 -- 172.21.15.1:0/4099418508 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f8037070 con 0x7f01fc1528d0 2024-03-06T07:33:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.454+0000 7f0200cad640 1 -- 172.21.15.1:0/4099418508 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f8030510 con 0x7f01fc1528d0 2024-03-06T07:33:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.454+0000 7f0200cad640 1 -- 172.21.15.1:0/4099418508 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec001670 con 0x7f01fc1528d0 2024-03-06T07:33:06.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.454+0000 7f0203f13640 1 -- 172.21.15.1:0/4099418508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01fc1528d0 msgr2=0x7f01fc072b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.454+0000 7f0203f13640 1 --2- 172.21.15.1:0/4099418508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01fc1528d0 0x7f01fc072b80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f01f80097a0 tx=0x7f01f802b710 comp rx=0 tx=0).stop 2024-03-06T07:33:06.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.455+0000 7f0203f13640 1 -- 172.21.15.1:0/4099418508 shutdown_connections 2024-03-06T07:33:06.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.455+0000 7f0203f13640 1 --2- 172.21.15.1:0/4099418508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01fc155330 0x7f01fc155770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.455+0000 7f0203f13640 1 --2- 172.21.15.1:0/4099418508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01fc1528d0 0x7f01fc072b80 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.455+0000 7f0203f13640 1 -- 172.21.15.1:0/4099418508 >> 172.21.15.1:0/4099418508 conn(0x7f01fc0c71c0 msgr2=0x7f01fc0c75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.455+0000 7f0203f13640 1 -- 172.21.15.1:0/4099418508 shutdown_connections 2024-03-06T07:33:06.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.455+0000 7f0203f13640 1 -- 172.21.15.1:0/4099418508 wait complete. 2024-03-06T07:33:06.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.456+0000 7f0203f13640 1 Processor -- start 2024-03-06T07:33:06.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.456+0000 7f0203f13640 1 -- start start 2024-03-06T07:33:06.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.457+0000 7f0203f13640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01fc071d00 0x7f01fc14fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.457+0000 7f0203f13640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01fc1528d0 0x7f01fc0ab640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.457+0000 7f0203f13640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01fc155330 0x7f01fc0abb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.457+0000 7f0203f13640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01fc06f450 con 0x7f01fc155330 2024-03-06T07:33:06.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.457+0000 7f0203f13640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01fc152cb0 con 0x7f01fc071d00 2024-03-06T07:33:06.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.457+0000 7f0203f13640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01fc06f5d0 con 0x7f01fc1528d0 2024-03-06T07:33:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.458+0000 7f02024b0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01fc155330 0x7f01fc0abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.458+0000 7f02024b0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01fc155330 0x7f01fc0abb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38950/0 (socket says 172.21.15.1:38950) 2024-03-06T07:33:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.458+0000 7f02024b0640 1 -- 172.21.15.1:0/3809863342 learned_addr learned my addr 172.21.15.1:0/3809863342 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.458+0000 7f02014ae640 1 --2- 172.21.15.1:0/3809863342 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01fc1528d0 0x7f01fc0ab640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.458+0000 7f0201caf640 1 --2- 172.21.15.1:0/3809863342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01fc071d00 0x7f01fc14fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.458+0000 7f02024b0640 1 -- 172.21.15.1:0/3809863342 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01fc1528d0 msgr2=0x7f01fc0ab640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.458+0000 7f02024b0640 1 --2- 172.21.15.1:0/3809863342 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01fc1528d0 0x7f01fc0ab640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.458+0000 7f02024b0640 1 -- 172.21.15.1:0/3809863342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01fc071d00 msgr2=0x7f01fc14fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.458+0000 7f02024b0640 1 --2- 172.21.15.1:0/3809863342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01fc071d00 0x7f01fc14fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.458+0000 7f02024b0640 1 -- 172.21.15.1:0/3809863342 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01fc1eb8f0 con 0x7f01fc155330 2024-03-06T07:33:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.458+0000 7f02024b0640 1 --2- 172.21.15.1:0/3809863342 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01fc155330 0x7f01fc0abb80 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009d30 tx=0x7f01f400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.458+0000 7f01f27fc640 1 -- 172.21.15.1:0/3809863342 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f4018020 con 0x7f01fc155330 2024-03-06T07:33:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.458+0000 7f01f27fc640 1 -- 172.21.15.1:0/3809863342 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f4014480 con 0x7f01fc155330 2024-03-06T07:33:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.458+0000 7f0203f13640 1 -- 172.21.15.1:0/3809863342 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01fc1eb610 con 0x7f01fc155330 2024-03-06T07:33:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.458+0000 7f01f27fc640 1 -- 172.21.15.1:0/3809863342 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f4014870 con 0x7f01fc155330 2024-03-06T07:33:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.458+0000 7f0203f13640 1 -- 172.21.15.1:0/3809863342 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01fc0c51c0 con 0x7f01fc155330 2024-03-06T07:33:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.459+0000 7f0203f13640 1 -- 172.21.15.1:0/3809863342 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01fc152480 con 0x7f01fc155330 2024-03-06T07:33:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.460+0000 7f01f27fc640 1 -- 172.21.15.1:0/3809863342 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01f4005d10 con 0x7f01fc155330 2024-03-06T07:33:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.460+0000 7f01f27fc640 1 --2- 172.21.15.1:0/3809863342 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01e006b4d0 0x7f01e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.460+0000 7f01f27fc640 1 -- 172.21.15.1:0/3809863342 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f01f408b000 con 0x7f01fc155330 2024-03-06T07:33:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.460+0000 7f0201caf640 1 --2- 172.21.15.1:0/3809863342 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01e006b4d0 0x7f01e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.461+0000 7f0201caf640 1 --2- 172.21.15.1:0/3809863342 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01e006b4d0 0x7f01e006d990 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f01f8034000 tx=0x7f01f8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.464+0000 7f01f27fc640 1 -- 172.21.15.1:0/3809863342 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01fc152480 con 0x7f01fc155330 2024-03-06T07:33:06.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.683+0000 7f0203f13640 1 -- 172.21.15.1:0/3809863342 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01fc0686e0 con 0x7f01fc155330 2024-03-06T07:33:06.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.683+0000 7f01f27fc640 1 -- 172.21.15.1:0/3809863342 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f01f4022030 con 0x7f01fc155330 2024-03-06T07:33:06.685 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:06.685 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:33:06.686 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:33:06.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.685+0000 7f0203f13640 1 -- 172.21.15.1:0/3809863342 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01e006b4d0 msgr2=0x7f01e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.685+0000 7f0203f13640 1 --2- 172.21.15.1:0/3809863342 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01e006b4d0 0x7f01e006d990 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f01f8034000 tx=0x7f01f8008030 comp rx=0 tx=0).stop 2024-03-06T07:33:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.685+0000 7f0203f13640 1 -- 172.21.15.1:0/3809863342 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01fc155330 msgr2=0x7f01fc0abb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.685+0000 7f0203f13640 1 --2- 172.21.15.1:0/3809863342 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01fc155330 0x7f01fc0abb80 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009d30 tx=0x7f01f400c490 comp rx=0 tx=0).stop 2024-03-06T07:33:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.685+0000 7f0203f13640 1 -- 172.21.15.1:0/3809863342 shutdown_connections 2024-03-06T07:33:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.685+0000 7f0203f13640 1 --2- 172.21.15.1:0/3809863342 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f01e006b4d0 0x7f01e006d990 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.685+0000 7f0203f13640 1 --2- 172.21.15.1:0/3809863342 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01fc155330 0x7f01fc0abb80 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.685+0000 7f0203f13640 1 --2- 172.21.15.1:0/3809863342 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01fc1528d0 0x7f01fc0ab640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.685+0000 7f0203f13640 1 --2- 172.21.15.1:0/3809863342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01fc071d00 0x7f01fc14fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.685+0000 7f0203f13640 1 -- 172.21.15.1:0/3809863342 >> 172.21.15.1:0/3809863342 conn(0x7f01fc0c71c0 msgr2=0x7f01fc06ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.685+0000 7f0203f13640 1 -- 172.21.15.1:0/3809863342 shutdown_connections 2024-03-06T07:33:06.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:06.685+0000 7f0203f13640 1 -- 172.21.15.1:0/3809863342 wait complete. 2024-03-06T07:33:07.100 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:33:07.100 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:33:08.101 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:08.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.200+0000 7fa0df8fb640 1 Processor -- start 2024-03-06T07:33:08.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.200+0000 7fa0df8fb640 1 -- start start 2024-03-06T07:33:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.202+0000 7fa0df8fb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa0d80bf9d0 0x7fa0d8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.202+0000 7fa0df8fb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d80c2fc0 con 0x7fa0d8152650 2024-03-06T07:33:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.202+0000 7fa0df8fb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d8158da0 con 0x7fa0d80c6b10 2024-03-06T07:33:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.202+0000 7fa0df8fb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0d8155800 con 0x7fa0d80bf9d0 2024-03-06T07:33:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.202+0000 7fa0dd697640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa0d8152650 0x7fa0d8158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55802/0 (socket says 172.21.15.1:55802) 2024-03-06T07:33:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.202+0000 7fa0dde98640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa0d80bf9d0 0x7fa0d8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.202+0000 7fa0dd697640 1 -- 172.21.15.1:0/1450015126 learned_addr learned my addr 172.21.15.1:0/1450015126 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:08.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.202+0000 7fa0dde98640 1 -- 172.21.15.1:0/1450015126 >> v1:172.21.15.98:6789/0 conn(0x7fa0d80c6b10 legacy=0x7fa0d80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:08.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.202+0000 7fa0dde98640 1 -- 172.21.15.1:0/1450015126 >> v1:172.21.15.1:6789/0 conn(0x7fa0d8152650 legacy=0x7fa0d8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.202+0000 7fa0dde98640 1 -- 172.21.15.1:0/1450015126 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d80c6690 con 0x7fa0d80bf9d0 2024-03-06T07:33:08.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.202+0000 7fa0cffff640 1 -- 172.21.15.1:0/1450015126 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238465129 0 0) 0x7fa0d80c2fc0 con 0x7fa0d8152650 2024-03-06T07:33:08.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.203+0000 7fa0dde98640 1 --2- 172.21.15.1:0/1450015126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d80bf9d0 0x7fa0d8155c90 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009d90 tx=0x7fa0d402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1113352643c063cc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.203+0000 7fa0cffff640 1 -- 172.21.15.1:0/1450015126 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4039070 con 0x7fa0d80bf9d0 2024-03-06T07:33:08.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.203+0000 7fa0cffff640 1 -- 172.21.15.1:0/1450015126 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d402cbd0 con 0x7fa0d80bf9d0 2024-03-06T07:33:08.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.203+0000 7fa0cffff640 1 -- 172.21.15.1:0/1450015126 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4034440 con 0x7fa0d80bf9d0 2024-03-06T07:33:08.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.204+0000 7fa0df8fb640 1 -- 172.21.15.1:0/1450015126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d80bf9d0 msgr2=0x7fa0d8155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.204+0000 7fa0df8fb640 1 --2- 172.21.15.1:0/1450015126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d80bf9d0 0x7fa0d8155c90 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009d90 tx=0x7fa0d402be20 comp rx=0 tx=0).stop 2024-03-06T07:33:08.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.204+0000 7fa0df8fb640 1 -- 172.21.15.1:0/1450015126 shutdown_connections 2024-03-06T07:33:08.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.204+0000 7fa0df8fb640 1 --2- 172.21.15.1:0/1450015126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d80bf9d0 0x7fa0d8155c90 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.204+0000 7fa0df8fb640 1 -- 172.21.15.1:0/1450015126 >> 172.21.15.1:0/1450015126 conn(0x7fa0d8066ee0 msgr2=0x7fa0d80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:08.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.204+0000 7fa0df8fb640 1 -- 172.21.15.1:0/1450015126 shutdown_connections 2024-03-06T07:33:08.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.204+0000 7fa0df8fb640 1 -- 172.21.15.1:0/1450015126 wait complete. 2024-03-06T07:33:08.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.205+0000 7fa0df8fb640 1 Processor -- start 2024-03-06T07:33:08.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.205+0000 7fa0df8fb640 1 -- start start 2024-03-06T07:33:08.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.205+0000 7fa0df8fb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d80bf9d0 0x7fa0d80c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.205+0000 7fa0df8fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d8152650 0x7fa0d81ebed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.205+0000 7fa0df8fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0d8158870 0x7fa0d81ec410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.205+0000 7fa0df8fb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d81551b0 con 0x7fa0d8152650 2024-03-06T07:33:08.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.205+0000 7fa0df8fb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d80c6fa0 con 0x7fa0d80bf9d0 2024-03-06T07:33:08.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.205+0000 7fa0df8fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d8155490 con 0x7fa0d8158870 2024-03-06T07:33:08.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.205+0000 7fa0dce96640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d8152650 0x7fa0d81ebed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.206+0000 7fa0dce96640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d8152650 0x7fa0d81ebed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38974/0 (socket says 172.21.15.1:38974) 2024-03-06T07:33:08.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.206+0000 7fa0dce96640 1 -- 172.21.15.1:0/2177077071 learned_addr learned my addr 172.21.15.1:0/2177077071 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:08.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.206+0000 7fa0dde98640 1 --2- 172.21.15.1:0/2177077071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0d8158870 0x7fa0d81ec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.206+0000 7fa0dd697640 1 --2- 172.21.15.1:0/2177077071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d80bf9d0 0x7fa0d80c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.206+0000 7fa0dce96640 1 -- 172.21.15.1:0/2177077071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0d8158870 msgr2=0x7fa0d81ec410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.206+0000 7fa0dce96640 1 --2- 172.21.15.1:0/2177077071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0d8158870 0x7fa0d81ec410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.206+0000 7fa0dce96640 1 -- 172.21.15.1:0/2177077071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d80bf9d0 msgr2=0x7fa0d80c3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.206+0000 7fa0dce96640 1 --2- 172.21.15.1:0/2177077071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d80bf9d0 0x7fa0d80c3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.206+0000 7fa0dce96640 1 -- 172.21.15.1:0/2177077071 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d8158450 con 0x7fa0d8152650 2024-03-06T07:33:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.206+0000 7fa0dde98640 1 --2- 172.21.15.1:0/2177077071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0d8158870 0x7fa0d81ec410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.206+0000 7fa0dd697640 1 --2- 172.21.15.1:0/2177077071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d80bf9d0 0x7fa0d80c3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.206+0000 7fa0dce96640 1 --2- 172.21.15.1:0/2177077071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d8152650 0x7fa0d81ebed0 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7fa0c800c460 tx=0x7fa0c800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.206+0000 7fa0cdffb640 1 -- 172.21.15.1:0/2177077071 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8016020 con 0x7fa0d8152650 2024-03-06T07:33:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.206+0000 7fa0cdffb640 1 -- 172.21.15.1:0/2177077071 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c8012870 con 0x7fa0d8152650 2024-03-06T07:33:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.206+0000 7fa0df8fb640 1 -- 172.21.15.1:0/2177077071 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d81ee8b0 con 0x7fa0d8152650 2024-03-06T07:33:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.206+0000 7fa0cdffb640 1 -- 172.21.15.1:0/2177077071 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8012c60 con 0x7fa0d8152650 2024-03-06T07:33:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.206+0000 7fa0df8fb640 1 -- 172.21.15.1:0/2177077071 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d8156d70 con 0x7fa0d8152650 2024-03-06T07:33:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.207+0000 7fa0df8fb640 1 -- 172.21.15.1:0/2177077071 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d8067fd0 con 0x7fa0d8152650 2024-03-06T07:33:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.208+0000 7fa0cdffb640 1 -- 172.21.15.1:0/2177077071 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0c8019020 con 0x7fa0d8152650 2024-03-06T07:33:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.208+0000 7fa0cdffb640 1 --2- 172.21.15.1:0/2177077071 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0bc06b4d0 0x7fa0bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.208+0000 7fa0cdffb640 1 -- 172.21.15.1:0/2177077071 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa0c808a130 con 0x7fa0d8152650 2024-03-06T07:33:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.209+0000 7fa0dd697640 1 --2- 172.21.15.1:0/2177077071 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0bc06b4d0 0x7fa0bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.209+0000 7fa0dd697640 1 --2- 172.21.15.1:0/2177077071 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0bc06b4d0 0x7fa0bc06d990 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0000f60 tx=0x7fa0d0007c30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.213+0000 7fa0cdffb640 1 -- 172.21.15.1:0/2177077071 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0c8020030 con 0x7fa0d8152650 2024-03-06T07:33:08.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.463+0000 7fa0df8fb640 1 -- 172.21.15.1:0/2177077071 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0d8151a50 con 0x7fa0d8152650 2024-03-06T07:33:08.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.464+0000 7fa0cdffb640 1 -- 172.21.15.1:0/2177077071 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fa0d81ee8b0 con 0x7fa0d8152650 2024-03-06T07:33:08.466 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:08.466 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:33:08.466 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:33:08.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.466+0000 7fa0df8fb640 1 -- 172.21.15.1:0/2177077071 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0bc06b4d0 msgr2=0x7fa0bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.466+0000 7fa0df8fb640 1 --2- 172.21.15.1:0/2177077071 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0bc06b4d0 0x7fa0bc06d990 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0000f60 tx=0x7fa0d0007c30 comp rx=0 tx=0).stop 2024-03-06T07:33:08.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.466+0000 7fa0df8fb640 1 -- 172.21.15.1:0/2177077071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d8152650 msgr2=0x7fa0d81ebed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.466+0000 7fa0df8fb640 1 --2- 172.21.15.1:0/2177077071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d8152650 0x7fa0d81ebed0 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7fa0c800c460 tx=0x7fa0c800c920 comp rx=0 tx=0).stop 2024-03-06T07:33:08.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.466+0000 7fa0df8fb640 1 -- 172.21.15.1:0/2177077071 shutdown_connections 2024-03-06T07:33:08.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.466+0000 7fa0df8fb640 1 --2- 172.21.15.1:0/2177077071 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa0bc06b4d0 0x7fa0bc06d990 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.466+0000 7fa0df8fb640 1 --2- 172.21.15.1:0/2177077071 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa0d8158870 0x7fa0d81ec410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.466+0000 7fa0df8fb640 1 --2- 172.21.15.1:0/2177077071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0d8152650 0x7fa0d81ebed0 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.466+0000 7fa0df8fb640 1 --2- 172.21.15.1:0/2177077071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0d80bf9d0 0x7fa0d80c3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.466+0000 7fa0df8fb640 1 -- 172.21.15.1:0/2177077071 >> 172.21.15.1:0/2177077071 conn(0x7fa0d8066ee0 msgr2=0x7fa0d80c5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:08.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.466+0000 7fa0df8fb640 1 -- 172.21.15.1:0/2177077071 shutdown_connections 2024-03-06T07:33:08.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:08.466+0000 7fa0df8fb640 1 -- 172.21.15.1:0/2177077071 wait complete. 2024-03-06T07:33:08.881 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:33:08.881 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:33:09.882 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.982+0000 7fa26a20e640 1 Processor -- start 2024-03-06T07:33:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.982+0000 7fa26a20e640 1 -- start start 2024-03-06T07:33:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.983+0000 7fa26a20e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa264152650 0x7fa264155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.983+0000 7fa26a20e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2640bf9d0 0x7fa2640c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.983+0000 7fa26a20e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2640c2fc0 con 0x7fa2640c6b10 2024-03-06T07:33:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.983+0000 7fa26a20e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa264155800 con 0x7fa2640bf9d0 2024-03-06T07:33:09.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.983+0000 7fa26a20e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa264158da0 con 0x7fa264152650 2024-03-06T07:33:09.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.983+0000 7fa263fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2640bf9d0 0x7fa2640c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.983+0000 7fa263fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2640bf9d0 0x7fa2640c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38990/0 (socket says 172.21.15.1:38990) 2024-03-06T07:33:09.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.983+0000 7fa263fff640 1 -- 172.21.15.1:0/3033469373 learned_addr learned my addr 172.21.15.1:0/3033469373 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:09.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.983+0000 7fa2637fe640 1 --2- 172.21.15.1:0/3033469373 >> v2:172.21.15.98:3300/0 conn(0x7fa264152650 0x7fa264155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.983+0000 7fa263fff640 1 -- 172.21.15.1:0/3033469373 >> v2:172.21.15.98:3300/0 conn(0x7fa264152650 msgr2=0x7fa264155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:09.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.983+0000 7fa263fff640 1 --2- 172.21.15.1:0/3033469373 >> v2:172.21.15.98:3300/0 conn(0x7fa264152650 0x7fa264155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.983+0000 7fa263fff640 1 -- 172.21.15.1:0/3033469373 >> v1:172.21.15.98:6789/0 conn(0x7fa2640c6b10 legacy=0x7fa264158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:09.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.983+0000 7fa263fff640 1 -- 172.21.15.1:0/3033469373 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2640c6690 con 0x7fa2640bf9d0 2024-03-06T07:33:09.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.984+0000 7fa263fff640 1 --2- 172.21.15.1:0/3033469373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2640bf9d0 0x7fa2640c1460 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7fa2640cac90 tx=0x7fa25c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9806051506599e8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.984+0000 7fa2627fc640 1 -- 172.21.15.1:0/3033469373 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25c02c9f0 con 0x7fa2640bf9d0 2024-03-06T07:33:09.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.984+0000 7fa2627fc640 1 -- 172.21.15.1:0/3033469373 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25c02cb90 con 0x7fa2640bf9d0 2024-03-06T07:33:09.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.984+0000 7fa2627fc640 1 -- 172.21.15.1:0/3033469373 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25c0343f0 con 0x7fa2640bf9d0 2024-03-06T07:33:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.985+0000 7fa26a20e640 1 -- 172.21.15.1:0/3033469373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2640bf9d0 msgr2=0x7fa2640c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.985+0000 7fa26a20e640 1 --2- 172.21.15.1:0/3033469373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2640bf9d0 0x7fa2640c1460 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7fa2640cac90 tx=0x7fa25c02be50 comp rx=0 tx=0).stop 2024-03-06T07:33:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.985+0000 7fa26a20e640 1 -- 172.21.15.1:0/3033469373 shutdown_connections 2024-03-06T07:33:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.985+0000 7fa26a20e640 1 --2- 172.21.15.1:0/3033469373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2640bf9d0 0x7fa2640c1460 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.985+0000 7fa26a20e640 1 --2- 172.21.15.1:0/3033469373 >> v2:172.21.15.98:3300/0 conn(0x7fa264152650 0x7fa264155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.985+0000 7fa26a20e640 1 -- 172.21.15.1:0/3033469373 >> 172.21.15.1:0/3033469373 conn(0x7fa264066ee0 msgr2=0x7fa2640672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.985+0000 7fa26a20e640 1 -- 172.21.15.1:0/3033469373 shutdown_connections 2024-03-06T07:33:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.985+0000 7fa26a20e640 1 -- 172.21.15.1:0/3033469373 wait complete. 2024-03-06T07:33:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.986+0000 7fa26a20e640 1 Processor -- start 2024-03-06T07:33:09.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.986+0000 7fa26a20e640 1 -- start start 2024-03-06T07:33:09.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.986+0000 7fa26a20e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2640bf9d0 0x7fa2641eab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa26a20e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa264152650 0x7fa2640c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa26a20e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa26414f040 0x7fa2641ee870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa26a20e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2641551b0 con 0x7fa26414f040 2024-03-06T07:33:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa26a20e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2640c6fa0 con 0x7fa2640bf9d0 2024-03-06T07:33:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa26a20e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa264155490 con 0x7fa264152650 2024-03-06T07:33:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa263fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa26414f040 0x7fa2641ee870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa263fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa26414f040 0x7fa2641ee870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38994/0 (socket says 172.21.15.1:38994) 2024-03-06T07:33:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa263fff640 1 -- 172.21.15.1:0/3479676410 learned_addr learned my addr 172.21.15.1:0/3479676410 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa2637fe640 1 --2- 172.21.15.1:0/3479676410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2640bf9d0 0x7fa2641eab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa262ffd640 1 --2- 172.21.15.1:0/3479676410 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa264152650 0x7fa2640c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa263fff640 1 -- 172.21.15.1:0/3479676410 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa264152650 msgr2=0x7fa2640c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa263fff640 1 --2- 172.21.15.1:0/3479676410 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa264152650 0x7fa2640c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa263fff640 1 -- 172.21.15.1:0/3479676410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2640bf9d0 msgr2=0x7fa2641eab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa263fff640 1 --2- 172.21.15.1:0/3479676410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2640bf9d0 0x7fa2641eab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa263fff640 1 -- 172.21.15.1:0/3479676410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa264157780 con 0x7fa26414f040 2024-03-06T07:33:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa2637fe640 1 --2- 172.21.15.1:0/3479676410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2640bf9d0 0x7fa2641eab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:09.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa263fff640 1 --2- 172.21.15.1:0/3479676410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa26414f040 0x7fa2641ee870 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7fa25c02c460 tx=0x7fa25c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa24bfff640 1 -- 172.21.15.1:0/3479676410 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25c043020 con 0x7fa26414f040 2024-03-06T07:33:09.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa24bfff640 1 -- 172.21.15.1:0/3479676410 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25c03e450 con 0x7fa26414f040 2024-03-06T07:33:09.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa26a20e640 1 -- 172.21.15.1:0/3479676410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2641e1190 con 0x7fa26414f040 2024-03-06T07:33:09.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.987+0000 7fa24bfff640 1 -- 172.21.15.1:0/3479676410 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25c009510 con 0x7fa26414f040 2024-03-06T07:33:09.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.988+0000 7fa26a20e640 1 -- 172.21.15.1:0/3479676410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2640c61e0 con 0x7fa26414f040 2024-03-06T07:33:09.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.988+0000 7fa26a20e640 1 -- 172.21.15.1:0/3479676410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa264067fd0 con 0x7fa26414f040 2024-03-06T07:33:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.989+0000 7fa24bfff640 1 -- 172.21.15.1:0/3479676410 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa25c046020 con 0x7fa26414f040 2024-03-06T07:33:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.989+0000 7fa24bfff640 1 --2- 172.21.15.1:0/3479676410 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa25006b500 0x7fa25006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.989+0000 7fa24bfff640 1 -- 172.21.15.1:0/3479676410 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa25c0ac9d0 con 0x7fa26414f040 2024-03-06T07:33:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.990+0000 7fa2637fe640 1 --2- 172.21.15.1:0/3479676410 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa25006b500 0x7fa25006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.990+0000 7fa2637fe640 1 --2- 172.21.15.1:0/3479676410 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa25006b500 0x7fa25006d9c0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fa254006fd0 tx=0x7fa254008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:09.994+0000 7fa24bfff640 1 -- 172.21.15.1:0/3479676410 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa25c04d070 con 0x7fa26414f040 2024-03-06T07:33:10.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:10.242+0000 7fa26a20e640 1 -- 172.21.15.1:0/3479676410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa264067fd0 con 0x7fa26414f040 2024-03-06T07:33:10.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:10.242+0000 7fa24bfff640 1 -- 172.21.15.1:0/3479676410 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fa25c07b650 con 0x7fa26414f040 2024-03-06T07:33:10.245 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:10.245 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:33:10.245 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:33:10.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:10.244+0000 7fa26a20e640 1 -- 172.21.15.1:0/3479676410 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa25006b500 msgr2=0x7fa25006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:10.244+0000 7fa26a20e640 1 --2- 172.21.15.1:0/3479676410 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa25006b500 0x7fa25006d9c0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fa254006fd0 tx=0x7fa254008040 comp rx=0 tx=0).stop 2024-03-06T07:33:10.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:10.244+0000 7fa26a20e640 1 -- 172.21.15.1:0/3479676410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa26414f040 msgr2=0x7fa2641ee870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:10.244+0000 7fa26a20e640 1 --2- 172.21.15.1:0/3479676410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa26414f040 0x7fa2641ee870 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7fa25c02c460 tx=0x7fa25c004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:10.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:10.244+0000 7fa26a20e640 1 -- 172.21.15.1:0/3479676410 shutdown_connections 2024-03-06T07:33:10.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:10.244+0000 7fa26a20e640 1 --2- 172.21.15.1:0/3479676410 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa25006b500 0x7fa25006d9c0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:10.244+0000 7fa26a20e640 1 --2- 172.21.15.1:0/3479676410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa26414f040 0x7fa2641ee870 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:10.244+0000 7fa26a20e640 1 --2- 172.21.15.1:0/3479676410 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa264152650 0x7fa2640c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:10.244+0000 7fa26a20e640 1 --2- 172.21.15.1:0/3479676410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2640bf9d0 0x7fa2641eab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:10.244+0000 7fa26a20e640 1 -- 172.21.15.1:0/3479676410 >> 172.21.15.1:0/3479676410 conn(0x7fa264066ee0 msgr2=0x7fa2640c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:10.245+0000 7fa26a20e640 1 -- 172.21.15.1:0/3479676410 shutdown_connections 2024-03-06T07:33:10.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:10.245+0000 7fa26a20e640 1 -- 172.21.15.1:0/3479676410 wait complete. 2024-03-06T07:33:10.663 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:33:10.663 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:33:11.664 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.767+0000 7f0ddbf3a640 1 Processor -- start 2024-03-06T07:33:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.767+0000 7f0ddbf3a640 1 -- start start 2024-03-06T07:33:11.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.768+0000 7f0ddbf3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dd4152820 0x7f0dd40c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.768+0000 7f0ddbf3a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd40c7180 con 0x7f0dd40c0670 2024-03-06T07:33:11.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.768+0000 7f0ddbf3a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd4155460 con 0x7f0dd40af6f0 2024-03-06T07:33:11.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.768+0000 7f0ddbf3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd40c6bc0 con 0x7f0dd4152820 2024-03-06T07:33:11.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.768+0000 7f0dd94d5640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0dd40c0670 0x7f0dd40c73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55808/0 (socket says 172.21.15.1:55808) 2024-03-06T07:33:11.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.768+0000 7f0dd94d5640 1 -- 172.21.15.1:0/4208618508 learned_addr learned my addr 172.21.15.1:0/4208618508 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:11.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.768+0000 7f0dd9cd6640 1 --2- 172.21.15.1:0/4208618508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dd4152820 0x7f0dd40c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.768+0000 7f0dd8cd4640 1 -- 172.21.15.1:0/4208618508 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403070537 0 0) 0x7f0dd40c7180 con 0x7f0dd40c0670 2024-03-06T07:33:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.768+0000 7f0dd8cd4640 1 -- 172.21.15.1:0/4208618508 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dbc003520 con 0x7f0dd40c0670 2024-03-06T07:33:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.769+0000 7f0dd8cd4640 1 -- 172.21.15.1:0/4208618508 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645512461 0 0) 0x7f0dd4155460 con 0x7f0dd40af6f0 2024-03-06T07:33:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.769+0000 7f0dd8cd4640 1 -- 172.21.15.1:0/4208618508 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dd40c7180 con 0x7f0dd40af6f0 2024-03-06T07:33:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.769+0000 7f0dd8cd4640 1 -- 172.21.15.1:0/4208618508 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1327457907 0 0) 0x7f0dbc003520 con 0x7f0dd40c0670 2024-03-06T07:33:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.769+0000 7f0dd9cd6640 1 -- 172.21.15.1:0/4208618508 >> v1:172.21.15.98:6789/0 conn(0x7f0dd40af6f0 legacy=0x7f0dd415fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.769+0000 7f0dd9cd6640 1 -- 172.21.15.1:0/4208618508 >> v1:172.21.15.1:6789/0 conn(0x7f0dd40c0670 legacy=0x7f0dd40c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.769+0000 7f0dd9cd6640 1 -- 172.21.15.1:0/4208618508 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd40c6ea0 con 0x7f0dd4152820 2024-03-06T07:33:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.769+0000 7f0dd9cd6640 1 --2- 172.21.15.1:0/4208618508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dd4152820 0x7f0dd40c0130 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc0097d0 tx=0x7f0dcc02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4f6a27aa473016f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.770+0000 7f0dd8cd4640 1 -- 172.21.15.1:0/4208618508 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc037070 con 0x7f0dd4152820 2024-03-06T07:33:11.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.770+0000 7f0dd8cd4640 1 -- 172.21.15.1:0/4208618508 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dcc030510 con 0x7f0dd4152820 2024-03-06T07:33:11.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.770+0000 7f0dd8cd4640 1 -- 172.21.15.1:0/4208618508 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8001670 con 0x7f0dd4152820 2024-03-06T07:33:11.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.770+0000 7f0ddbf3a640 1 -- 172.21.15.1:0/4208618508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dd4152820 msgr2=0x7f0dd40c0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.770+0000 7f0ddbf3a640 1 --2- 172.21.15.1:0/4208618508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dd4152820 0x7f0dd40c0130 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc0097d0 tx=0x7f0dcc02b800 comp rx=0 tx=0).stop 2024-03-06T07:33:11.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.770+0000 7f0ddbf3a640 1 -- 172.21.15.1:0/4208618508 shutdown_connections 2024-03-06T07:33:11.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.770+0000 7f0ddbf3a640 1 --2- 172.21.15.1:0/4208618508 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dd4152820 0x7f0dd40c0130 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.770+0000 7f0ddbf3a640 1 -- 172.21.15.1:0/4208618508 >> 172.21.15.1:0/4208618508 conn(0x7f0dd4072720 msgr2=0x7f0dd4072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:11.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.770+0000 7f0ddbf3a640 1 -- 172.21.15.1:0/4208618508 shutdown_connections 2024-03-06T07:33:11.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.770+0000 7f0ddbf3a640 1 -- 172.21.15.1:0/4208618508 wait complete. 2024-03-06T07:33:11.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.772+0000 7f0ddbf3a640 1 Processor -- start 2024-03-06T07:33:11.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.772+0000 7f0ddbf3a640 1 -- start start 2024-03-06T07:33:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.773+0000 7f0ddbf3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dd40af6f0 0x7f0dd40cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.773+0000 7f0ddbf3a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dd40c0670 0x7f0dd4151960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.773+0000 7f0ddbf3a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dd4152820 0x7f0dd41579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.773+0000 7f0ddbf3a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd40c7c80 con 0x7f0dd40c0670 2024-03-06T07:33:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.773+0000 7f0ddbf3a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd40c7b00 con 0x7f0dd40af6f0 2024-03-06T07:33:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.773+0000 7f0ddbf3a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd4152c00 con 0x7f0dd4152820 2024-03-06T07:33:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.773+0000 7f0dd94d5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dd40c0670 0x7f0dd4151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.773+0000 7f0dd94d5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dd40c0670 0x7f0dd4151960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39010/0 (socket says 172.21.15.1:39010) 2024-03-06T07:33:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.773+0000 7f0dd94d5640 1 -- 172.21.15.1:0/3004845439 learned_addr learned my addr 172.21.15.1:0/3004845439 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.773+0000 7f0dda4d7640 1 --2- 172.21.15.1:0/3004845439 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dd4152820 0x7f0dd41579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.773+0000 7f0dd9cd6640 1 --2- 172.21.15.1:0/3004845439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dd40af6f0 0x7f0dd40cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.773+0000 7f0dd94d5640 1 -- 172.21.15.1:0/3004845439 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dd4152820 msgr2=0x7f0dd41579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.773+0000 7f0dd94d5640 1 --2- 172.21.15.1:0/3004845439 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dd4152820 0x7f0dd41579e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.773+0000 7f0dd94d5640 1 -- 172.21.15.1:0/3004845439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dd40af6f0 msgr2=0x7f0dd40cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.773+0000 7f0dd94d5640 1 --2- 172.21.15.1:0/3004845439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dd40af6f0 0x7f0dd40cafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.773+0000 7f0dd94d5640 1 -- 172.21.15.1:0/3004845439 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd40cb4f0 con 0x7f0dd40c0670 2024-03-06T07:33:11.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.774+0000 7f0dd94d5640 1 --2- 172.21.15.1:0/3004845439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dd40c0670 0x7f0dd4151960 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8000fd0 tx=0x7f0dc800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.774+0000 7f0dd27fc640 1 -- 172.21.15.1:0/3004845439 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8017070 con 0x7f0dd40c0670 2024-03-06T07:33:11.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.774+0000 7f0dd27fc640 1 -- 172.21.15.1:0/3004845439 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc8005ad0 con 0x7f0dd40c0670 2024-03-06T07:33:11.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.774+0000 7f0ddbf3a640 1 -- 172.21.15.1:0/3004845439 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd40cb790 con 0x7f0dd40c0670 2024-03-06T07:33:11.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.774+0000 7f0dd27fc640 1 -- 172.21.15.1:0/3004845439 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8007070 con 0x7f0dd40c0670 2024-03-06T07:33:11.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.774+0000 7f0ddbf3a640 1 -- 172.21.15.1:0/3004845439 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd41edaf0 con 0x7f0dd40c0670 2024-03-06T07:33:11.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.775+0000 7f0ddbf3a640 1 -- 172.21.15.1:0/3004845439 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd40686e0 con 0x7f0dd40c0670 2024-03-06T07:33:11.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.775+0000 7f0dd27fc640 1 -- 172.21.15.1:0/3004845439 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0dc801a070 con 0x7f0dd40c0670 2024-03-06T07:33:11.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.776+0000 7f0dd27fc640 1 --2- 172.21.15.1:0/3004845439 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0dc006b4d0 0x7f0dc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.776+0000 7f0dd27fc640 1 -- 172.21.15.1:0/3004845439 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f0dc80897f0 con 0x7f0dd40c0670 2024-03-06T07:33:11.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.776+0000 7f0dd9cd6640 1 --2- 172.21.15.1:0/3004845439 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0dc006b4d0 0x7f0dc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.777+0000 7f0dd9cd6640 1 --2- 172.21.15.1:0/3004845439 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0dc006b4d0 0x7f0dc006d990 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc034000 tx=0x7f0dcc008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:11.780+0000 7f0dd27fc640 1 -- 172.21.15.1:0/3004845439 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0dc8057e20 con 0x7f0dd40c0670 2024-03-06T07:33:12.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:12.001+0000 7f0ddbf3a640 1 -- 172.21.15.1:0/3004845439 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dd40c5b40 con 0x7f0dd40c0670 2024-03-06T07:33:12.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:12.002+0000 7f0dd27fc640 1 -- 172.21.15.1:0/3004845439 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f0dc805bec0 con 0x7f0dd40c0670 2024-03-06T07:33:12.004 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:12.004 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:33:12.004 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:33:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:12.004+0000 7f0ddbf3a640 1 -- 172.21.15.1:0/3004845439 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0dc006b4d0 msgr2=0x7f0dc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:12.004+0000 7f0ddbf3a640 1 --2- 172.21.15.1:0/3004845439 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0dc006b4d0 0x7f0dc006d990 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc034000 tx=0x7f0dcc008030 comp rx=0 tx=0).stop 2024-03-06T07:33:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:12.004+0000 7f0ddbf3a640 1 -- 172.21.15.1:0/3004845439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dd40c0670 msgr2=0x7f0dd4151960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:12.004+0000 7f0ddbf3a640 1 --2- 172.21.15.1:0/3004845439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dd40c0670 0x7f0dd4151960 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8000fd0 tx=0x7f0dc800cbe0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:12.004+0000 7f0ddbf3a640 1 -- 172.21.15.1:0/3004845439 shutdown_connections 2024-03-06T07:33:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:12.004+0000 7f0ddbf3a640 1 --2- 172.21.15.1:0/3004845439 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0dc006b4d0 0x7f0dc006d990 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:12.004+0000 7f0ddbf3a640 1 --2- 172.21.15.1:0/3004845439 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dd4152820 0x7f0dd41579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:12.004+0000 7f0ddbf3a640 1 --2- 172.21.15.1:0/3004845439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dd40c0670 0x7f0dd4151960 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:12.004+0000 7f0ddbf3a640 1 --2- 172.21.15.1:0/3004845439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dd40af6f0 0x7f0dd40cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:12.004+0000 7f0ddbf3a640 1 -- 172.21.15.1:0/3004845439 >> 172.21.15.1:0/3004845439 conn(0x7f0dd4072720 msgr2=0x7f0dd4154cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:12.004+0000 7f0ddbf3a640 1 -- 172.21.15.1:0/3004845439 shutdown_connections 2024-03-06T07:33:12.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:12.004+0000 7f0ddbf3a640 1 -- 172.21.15.1:0/3004845439 wait complete. 2024-03-06T07:33:12.420 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:33:12.420 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:33:13.421 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:13.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.522+0000 7f9a59f02640 1 Processor -- start 2024-03-06T07:33:13.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.522+0000 7f9a59f02640 1 -- start start 2024-03-06T07:33:13.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.523+0000 7f9a59f02640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9a540c3970 0x7f9a540c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.523+0000 7f9a59f02640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9a54152840 0x7f9a541553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.523+0000 7f9a59f02640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a540c5e40 con 0x7f9a540c60b0 2024-03-06T07:33:13.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.523+0000 7f9a59f02640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a540c67f0 con 0x7f9a540c3970 2024-03-06T07:33:13.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.523+0000 7f9a59f02640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a540c6a40 con 0x7f9a54152840 2024-03-06T07:33:13.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.523+0000 7f9a537fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9a540c3970 0x7f9a540c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.523+0000 7f9a52ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9a540c60b0 0x7f9a54152130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55816/0 (socket says 172.21.15.1:55816) 2024-03-06T07:33:13.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.523+0000 7f9a537fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9a540c3970 0x7f9a540c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39020/0 (socket says 172.21.15.1:39020) 2024-03-06T07:33:13.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.523+0000 7f9a52ffd640 1 -- 172.21.15.1:0/113793833 learned_addr learned my addr 172.21.15.1:0/113793833 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:13.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.523+0000 7f9a537fe640 1 -- 172.21.15.1:0/113793833 >> v2:172.21.15.98:3300/0 conn(0x7f9a54152840 msgr2=0x7f9a541553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:13.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.523+0000 7f9a537fe640 1 --2- 172.21.15.1:0/113793833 >> v2:172.21.15.98:3300/0 conn(0x7f9a54152840 0x7f9a541553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.523+0000 7f9a537fe640 1 -- 172.21.15.1:0/113793833 >> v1:172.21.15.1:6789/0 conn(0x7f9a540c60b0 legacy=0x7f9a54152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.523+0000 7f9a537fe640 1 -- 172.21.15.1:0/113793833 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a540c6d20 con 0x7f9a540c3970 2024-03-06T07:33:13.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.523+0000 7f9a527fc640 1 -- 172.21.15.1:0/113793833 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556944654 0 0) 0x7f9a540c5e40 con 0x7f9a540c60b0 2024-03-06T07:33:13.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.524+0000 7f9a537fe640 1 --2- 172.21.15.1:0/113793833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a540c3970 0x7f9a540c8170 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f9a540c86b0 tx=0x7f9a3c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e6d0eed9a57df7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.524+0000 7f9a527fc640 1 -- 172.21.15.1:0/113793833 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c035020 con 0x7f9a540c3970 2024-03-06T07:33:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.524+0000 7f9a527fc640 1 -- 172.21.15.1:0/113793833 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3c031830 con 0x7f9a540c3970 2024-03-06T07:33:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.524+0000 7f9a527fc640 1 -- 172.21.15.1:0/113793833 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c031c00 con 0x7f9a540c3970 2024-03-06T07:33:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.525+0000 7f9a59f02640 1 -- 172.21.15.1:0/113793833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a540c3970 msgr2=0x7f9a540c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.525+0000 7f9a59f02640 1 --2- 172.21.15.1:0/113793833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a540c3970 0x7f9a540c8170 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f9a540c86b0 tx=0x7f9a3c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.525+0000 7f9a59f02640 1 -- 172.21.15.1:0/113793833 shutdown_connections 2024-03-06T07:33:13.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.525+0000 7f9a59f02640 1 --2- 172.21.15.1:0/113793833 >> v2:172.21.15.98:3300/0 conn(0x7f9a54152840 0x7f9a541553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.525+0000 7f9a59f02640 1 --2- 172.21.15.1:0/113793833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a540c3970 0x7f9a540c8170 unknown :-1 s=CLOSED pgs=2664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.525+0000 7f9a59f02640 1 -- 172.21.15.1:0/113793833 >> 172.21.15.1:0/113793833 conn(0x7f9a540c88f0 msgr2=0x7f9a540c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.525+0000 7f9a59f02640 1 -- 172.21.15.1:0/113793833 shutdown_connections 2024-03-06T07:33:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.525+0000 7f9a59f02640 1 -- 172.21.15.1:0/113793833 wait complete. 2024-03-06T07:33:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.526+0000 7f9a59f02640 1 Processor -- start 2024-03-06T07:33:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.526+0000 7f9a59f02640 1 -- start start 2024-03-06T07:33:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.527+0000 7f9a59f02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a540c3970 0x7f9a541ea9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.527+0000 7f9a59f02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a540c60b0 0x7f9a54153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.527+0000 7f9a59f02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a54152840 0x7f9a5414ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.527+0000 7f9a59f02640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a540c75c0 con 0x7f9a54152840 2024-03-06T07:33:13.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.527+0000 7f9a59f02640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a540c72e0 con 0x7f9a540c60b0 2024-03-06T07:33:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.527+0000 7f9a59f02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a540c3d50 con 0x7f9a540c3970 2024-03-06T07:33:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.527+0000 7f9a53fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a54152840 0x7f9a5414ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.527+0000 7f9a53fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a54152840 0x7f9a5414ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39028/0 (socket says 172.21.15.1:39028) 2024-03-06T07:33:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.527+0000 7f9a53fff640 1 -- 172.21.15.1:0/3240949044 learned_addr learned my addr 172.21.15.1:0/3240949044 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.527+0000 7f9a537fe640 1 --2- 172.21.15.1:0/3240949044 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a540c3970 0x7f9a541ea9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.527+0000 7f9a52ffd640 1 --2- 172.21.15.1:0/3240949044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a540c60b0 0x7f9a54153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.527+0000 7f9a53fff640 1 -- 172.21.15.1:0/3240949044 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a540c3970 msgr2=0x7f9a541ea9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.527+0000 7f9a53fff640 1 --2- 172.21.15.1:0/3240949044 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a540c3970 0x7f9a541ea9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.527+0000 7f9a53fff640 1 -- 172.21.15.1:0/3240949044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a540c60b0 msgr2=0x7f9a54153a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.527+0000 7f9a53fff640 1 --2- 172.21.15.1:0/3240949044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a540c60b0 0x7f9a54153a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.527+0000 7f9a53fff640 1 -- 172.21.15.1:0/3240949044 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a541544d0 con 0x7f9a54152840 2024-03-06T07:33:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.527+0000 7f9a52ffd640 1 --2- 172.21.15.1:0/3240949044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a540c60b0 0x7f9a54153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.528+0000 7f9a53fff640 1 --2- 172.21.15.1:0/3240949044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a54152840 0x7f9a5414ca00 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c00b9e0 tx=0x7f9a4c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.528+0000 7f9a4bfff640 1 -- 172.21.15.1:0/3240949044 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c019070 con 0x7f9a54152840 2024-03-06T07:33:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.528+0000 7f9a4bfff640 1 -- 172.21.15.1:0/3240949044 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4c00cb40 con 0x7f9a54152840 2024-03-06T07:33:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.528+0000 7f9a59f02640 1 -- 172.21.15.1:0/3240949044 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a54153f90 con 0x7f9a54152840 2024-03-06T07:33:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.528+0000 7f9a4bfff640 1 -- 172.21.15.1:0/3240949044 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c014440 con 0x7f9a54152840 2024-03-06T07:33:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.528+0000 7f9a59f02640 1 -- 172.21.15.1:0/3240949044 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a541eeae0 con 0x7f9a54152840 2024-03-06T07:33:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.529+0000 7f9a59f02640 1 -- 172.21.15.1:0/3240949044 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a540686e0 con 0x7f9a54152840 2024-03-06T07:33:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.529+0000 7f9a4bfff640 1 -- 172.21.15.1:0/3240949044 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a4c01c070 con 0x7f9a54152840 2024-03-06T07:33:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.530+0000 7f9a4bfff640 1 --2- 172.21.15.1:0/3240949044 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a3406b4d0 0x7f9a3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.530+0000 7f9a4bfff640 1 -- 172.21.15.1:0/3240949044 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f9a4c08a030 con 0x7f9a54152840 2024-03-06T07:33:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.530+0000 7f9a537fe640 1 --2- 172.21.15.1:0/3240949044 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a3406b4d0 0x7f9a3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.530+0000 7f9a537fe640 1 --2- 172.21.15.1:0/3240949044 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a3406b4d0 0x7f9a3406d990 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c036000 tx=0x7f9a3c03e040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.538+0000 7f9a4bfff640 1 -- 172.21.15.1:0/3240949044 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a4c058660 con 0x7f9a54152840 2024-03-06T07:33:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.758+0000 7f9a59f02640 1 -- 172.21.15.1:0/3240949044 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a540adea0 con 0x7f9a54152840 2024-03-06T07:33:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.759+0000 7f9a4bfff640 1 -- 172.21.15.1:0/3240949044 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f9a4c05c700 con 0x7f9a54152840 2024-03-06T07:33:13.762 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:13.762 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:33:13.764 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:33:13.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.761+0000 7f9a59f02640 1 -- 172.21.15.1:0/3240949044 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a3406b4d0 msgr2=0x7f9a3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.761+0000 7f9a59f02640 1 --2- 172.21.15.1:0/3240949044 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a3406b4d0 0x7f9a3406d990 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c036000 tx=0x7f9a3c03e040 comp rx=0 tx=0).stop 2024-03-06T07:33:13.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.761+0000 7f9a59f02640 1 -- 172.21.15.1:0/3240949044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a54152840 msgr2=0x7f9a5414ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.761+0000 7f9a59f02640 1 --2- 172.21.15.1:0/3240949044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a54152840 0x7f9a5414ca00 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c00b9e0 tx=0x7f9a4c00bea0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.761+0000 7f9a59f02640 1 -- 172.21.15.1:0/3240949044 shutdown_connections 2024-03-06T07:33:13.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.761+0000 7f9a59f02640 1 --2- 172.21.15.1:0/3240949044 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9a3406b4d0 0x7f9a3406d990 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.761+0000 7f9a59f02640 1 --2- 172.21.15.1:0/3240949044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a54152840 0x7f9a5414ca00 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.761+0000 7f9a59f02640 1 --2- 172.21.15.1:0/3240949044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a540c60b0 0x7f9a54153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.761+0000 7f9a59f02640 1 --2- 172.21.15.1:0/3240949044 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a540c3970 0x7f9a541ea9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.761+0000 7f9a59f02640 1 -- 172.21.15.1:0/3240949044 >> 172.21.15.1:0/3240949044 conn(0x7f9a540c88f0 msgr2=0x7f9a5406c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.761+0000 7f9a59f02640 1 -- 172.21.15.1:0/3240949044 shutdown_connections 2024-03-06T07:33:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:13.761+0000 7f9a59f02640 1 -- 172.21.15.1:0/3240949044 wait complete. 2024-03-06T07:33:14.176 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:33:14.177 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:33:15.177 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.277+0000 7f8a7a484640 1 Processor -- start 2024-03-06T07:33:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.277+0000 7f8a7a484640 1 -- start start 2024-03-06T07:33:15.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.278+0000 7f8a7a484640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a74152650 0x7f8a74155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.278+0000 7f8a7a484640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a740bf9d0 0x7f8a740c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.278+0000 7f8a7a484640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a740c2fc0 con 0x7f8a740c6b10 2024-03-06T07:33:15.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.278+0000 7f8a7a484640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a74155800 con 0x7f8a740bf9d0 2024-03-06T07:33:15.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.278+0000 7f8a7a484640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a74158da0 con 0x7f8a74152650 2024-03-06T07:33:15.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.278+0000 7f8a78a21640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a740bf9d0 0x7f8a740c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.278+0000 7f8a78a21640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a740bf9d0 0x7f8a740c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41584/0 (socket says 172.21.15.1:41584) 2024-03-06T07:33:15.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.278+0000 7f8a737fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8a740c6b10 0x7f8a74158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59942/0 (socket says 172.21.15.1:59942) 2024-03-06T07:33:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.278+0000 7f8a78a21640 1 -- 172.21.15.1:0/2562076880 learned_addr learned my addr 172.21.15.1:0/2562076880 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.279+0000 7f8a73fff640 1 --2- 172.21.15.1:0/2562076880 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a74152650 0x7f8a74155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.279+0000 7f8a78a21640 1 -- 172.21.15.1:0/2562076880 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a74152650 msgr2=0x7f8a74155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.279+0000 7f8a78a21640 1 --2- 172.21.15.1:0/2562076880 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a74152650 0x7f8a74155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.279+0000 7f8a78a21640 1 -- 172.21.15.1:0/2562076880 >> v1:172.21.15.1:6789/0 conn(0x7f8a740c6b10 legacy=0x7f8a74158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.279+0000 7f8a72ffd640 1 -- 172.21.15.1:0/2562076880 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439388710 0 0) 0x7f8a740c2fc0 con 0x7f8a740c6b10 2024-03-06T07:33:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.279+0000 7f8a78a21640 1 -- 172.21.15.1:0/2562076880 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a740c6690 con 0x7f8a740bf9d0 2024-03-06T07:33:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.279+0000 7f8a78a21640 1 --2- 172.21.15.1:0/2562076880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a740bf9d0 0x7f8a740c1460 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c009e50 tx=0x7f8a6c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b6d65b6a8c2985 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.279+0000 7f8a72ffd640 1 -- 172.21.15.1:0/2562076880 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c02ca70 con 0x7f8a740bf9d0 2024-03-06T07:33:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.280+0000 7f8a72ffd640 1 -- 172.21.15.1:0/2562076880 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6c02cc10 con 0x7f8a740bf9d0 2024-03-06T07:33:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.280+0000 7f8a72ffd640 1 -- 172.21.15.1:0/2562076880 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c0343f0 con 0x7f8a740bf9d0 2024-03-06T07:33:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.280+0000 7f8a7a484640 1 -- 172.21.15.1:0/2562076880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a740bf9d0 msgr2=0x7f8a740c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.280+0000 7f8a7a484640 1 --2- 172.21.15.1:0/2562076880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a740bf9d0 0x7f8a740c1460 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c009e50 tx=0x7f8a6c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.280+0000 7f8a7a484640 1 -- 172.21.15.1:0/2562076880 shutdown_connections 2024-03-06T07:33:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.280+0000 7f8a7a484640 1 --2- 172.21.15.1:0/2562076880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a740bf9d0 0x7f8a740c1460 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.280+0000 7f8a7a484640 1 --2- 172.21.15.1:0/2562076880 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a74152650 0x7f8a74155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.280+0000 7f8a7a484640 1 -- 172.21.15.1:0/2562076880 >> 172.21.15.1:0/2562076880 conn(0x7f8a74066ee0 msgr2=0x7f8a740672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.280+0000 7f8a7a484640 1 -- 172.21.15.1:0/2562076880 shutdown_connections 2024-03-06T07:33:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.280+0000 7f8a7a484640 1 -- 172.21.15.1:0/2562076880 wait complete. 2024-03-06T07:33:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.282+0000 7f8a7a484640 1 Processor -- start 2024-03-06T07:33:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.282+0000 7f8a7a484640 1 -- start start 2024-03-06T07:33:15.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.283+0000 7f8a7a484640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a740bf9d0 0x7f8a74156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.283+0000 7f8a7a484640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a74152650 0x7f8a74153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.283+0000 7f8a7a484640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a741eb730 0x7f8a740c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.283+0000 7f8a7a484640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a741551b0 con 0x7f8a740bf9d0 2024-03-06T07:33:15.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.283+0000 7f8a7a484640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a740c6fa0 con 0x7f8a74152650 2024-03-06T07:33:15.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.283+0000 7f8a7a484640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a74155490 con 0x7f8a741eb730 2024-03-06T07:33:15.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.283+0000 7f8a73fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a740bf9d0 0x7f8a74156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.283+0000 7f8a73fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a740bf9d0 0x7f8a74156bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41600/0 (socket says 172.21.15.1:41600) 2024-03-06T07:33:15.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.283+0000 7f8a73fff640 1 -- 172.21.15.1:0/3961661215 learned_addr learned my addr 172.21.15.1:0/3961661215 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:15.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.283+0000 7f8a78a21640 1 --2- 172.21.15.1:0/3961661215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a741eb730 0x7f8a740c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.283+0000 7f8a73fff640 1 -- 172.21.15.1:0/3961661215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a741eb730 msgr2=0x7f8a740c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.283+0000 7f8a73fff640 1 --2- 172.21.15.1:0/3961661215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a741eb730 0x7f8a740c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.283+0000 7f8a737fe640 1 --2- 172.21.15.1:0/3961661215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a74152650 0x7f8a74153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.284+0000 7f8a73fff640 1 -- 172.21.15.1:0/3961661215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a74152650 msgr2=0x7f8a74153b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.284+0000 7f8a73fff640 1 --2- 172.21.15.1:0/3961661215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a74152650 0x7f8a74153b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.284+0000 7f8a73fff640 1 -- 172.21.15.1:0/3961661215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a74157130 con 0x7f8a740bf9d0 2024-03-06T07:33:15.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.284+0000 7f8a73fff640 1 --2- 172.21.15.1:0/3961661215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a740bf9d0 0x7f8a74156bf0 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800c7f0 tx=0x7f8a6800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.284+0000 7f8a70ff9640 1 -- 172.21.15.1:0/3961661215 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68015070 con 0x7f8a740bf9d0 2024-03-06T07:33:15.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.284+0000 7f8a70ff9640 1 -- 172.21.15.1:0/3961661215 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a68004240 con 0x7f8a740bf9d0 2024-03-06T07:33:15.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.284+0000 7f8a7a484640 1 -- 172.21.15.1:0/3961661215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a74154d90 con 0x7f8a740bf9d0 2024-03-06T07:33:15.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.284+0000 7f8a70ff9640 1 -- 172.21.15.1:0/3961661215 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68007620 con 0x7f8a740bf9d0 2024-03-06T07:33:15.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.284+0000 7f8a7a484640 1 -- 172.21.15.1:0/3961661215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a74153320 con 0x7f8a740bf9d0 2024-03-06T07:33:15.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.285+0000 7f8a7a484640 1 -- 172.21.15.1:0/3961661215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a74067fd0 con 0x7f8a740bf9d0 2024-03-06T07:33:15.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.285+0000 7f8a70ff9640 1 -- 172.21.15.1:0/3961661215 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a680077c0 con 0x7f8a740bf9d0 2024-03-06T07:33:15.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.286+0000 7f8a70ff9640 1 --2- 172.21.15.1:0/3961661215 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a6006b4d0 0x7f8a6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.286+0000 7f8a70ff9640 1 -- 172.21.15.1:0/3961661215 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f8a6808a2a0 con 0x7f8a740bf9d0 2024-03-06T07:33:15.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.286+0000 7f8a737fe640 1 --2- 172.21.15.1:0/3961661215 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a6006b4d0 0x7f8a6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.287+0000 7f8a737fe640 1 --2- 172.21.15.1:0/3961661215 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a6006b4d0 0x7f8a6006d990 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f8a64007fd0 tx=0x7f8a64009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.290+0000 7f8a70ff9640 1 -- 172.21.15.1:0/3961661215 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a68058f20 con 0x7f8a740bf9d0 2024-03-06T07:33:15.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.510+0000 7f8a7a484640 1 -- 172.21.15.1:0/3961661215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a740c4530 con 0x7f8a740bf9d0 2024-03-06T07:33:15.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.511+0000 7f8a70ff9640 1 -- 172.21.15.1:0/3961661215 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f8a6808a0c0 con 0x7f8a740bf9d0 2024-03-06T07:33:15.513 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:15.514 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:33:15.514 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:33:15.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.513+0000 7f8a7a484640 1 -- 172.21.15.1:0/3961661215 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a6006b4d0 msgr2=0x7f8a6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.513+0000 7f8a7a484640 1 --2- 172.21.15.1:0/3961661215 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a6006b4d0 0x7f8a6006d990 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f8a64007fd0 tx=0x7f8a64009040 comp rx=0 tx=0).stop 2024-03-06T07:33:15.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.513+0000 7f8a7a484640 1 -- 172.21.15.1:0/3961661215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a740bf9d0 msgr2=0x7f8a74156bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.513+0000 7f8a7a484640 1 --2- 172.21.15.1:0/3961661215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a740bf9d0 0x7f8a74156bf0 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800c7f0 tx=0x7f8a6800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.513+0000 7f8a7a484640 1 -- 172.21.15.1:0/3961661215 shutdown_connections 2024-03-06T07:33:15.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.513+0000 7f8a7a484640 1 --2- 172.21.15.1:0/3961661215 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8a6006b4d0 0x7f8a6006d990 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.513+0000 7f8a7a484640 1 --2- 172.21.15.1:0/3961661215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8a741eb730 0x7f8a740c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.513+0000 7f8a7a484640 1 --2- 172.21.15.1:0/3961661215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a74152650 0x7f8a74153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.513+0000 7f8a7a484640 1 --2- 172.21.15.1:0/3961661215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a740bf9d0 0x7f8a74156bf0 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.513+0000 7f8a7a484640 1 -- 172.21.15.1:0/3961661215 >> 172.21.15.1:0/3961661215 conn(0x7f8a74066ee0 msgr2=0x7f8a740c5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.513+0000 7f8a7a484640 1 -- 172.21.15.1:0/3961661215 shutdown_connections 2024-03-06T07:33:15.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:15.513+0000 7f8a7a484640 1 -- 172.21.15.1:0/3961661215 wait complete. 2024-03-06T07:33:15.932 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:33:15.932 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:33:16.933 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:17.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.033+0000 7ff6aa55c640 1 Processor -- start 2024-03-06T07:33:17.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.034+0000 7ff6aa55c640 1 -- start start 2024-03-06T07:33:17.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.034+0000 7ff6aa55c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff6a40c6b10 0x7ff6a4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.034+0000 7ff6aa55c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6a40c2fc0 con 0x7ff6a4152650 2024-03-06T07:33:17.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.034+0000 7ff6aa55c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6a4158da0 con 0x7ff6a40bf9d0 2024-03-06T07:33:17.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.034+0000 7ff6aa55c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6a4155800 con 0x7ff6a40c6b10 2024-03-06T07:33:17.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.034+0000 7ff6a37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff6a40c6b10 0x7ff6a4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.034+0000 7ff6a3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff6a4152650 0x7ff6a4158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59950/0 (socket says 172.21.15.1:59950) 2024-03-06T07:33:17.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.034+0000 7ff6a3fff640 1 -- 172.21.15.1:0/226703736 learned_addr learned my addr 172.21.15.1:0/226703736 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:17.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.035+0000 7ff6a37fe640 1 -- 172.21.15.1:0/226703736 >> v1:172.21.15.98:6789/0 conn(0x7ff6a40bf9d0 legacy=0x7ff6a40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:17.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.035+0000 7ff6a37fe640 1 -- 172.21.15.1:0/226703736 >> v1:172.21.15.1:6789/0 conn(0x7ff6a4152650 legacy=0x7ff6a4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.035+0000 7ff6a37fe640 1 -- 172.21.15.1:0/226703736 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a40c6690 con 0x7ff6a40c6b10 2024-03-06T07:33:17.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.035+0000 7ff6a2ffd640 1 -- 172.21.15.1:0/226703736 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059541833 0 0) 0x7ff6a40c2fc0 con 0x7ff6a4152650 2024-03-06T07:33:17.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.035+0000 7ff6a37fe640 1 --2- 172.21.15.1:0/226703736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a40c6b10 0x7ff6a4155c90 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0097a0 tx=0x7ff68c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c54c0f8773f22e89 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.036+0000 7ff6a2ffd640 1 -- 172.21.15.1:0/226703736 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c0315a0 con 0x7ff6a40c6b10 2024-03-06T07:33:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.036+0000 7ff6a2ffd640 1 -- 172.21.15.1:0/226703736 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68c037070 con 0x7ff6a40c6b10 2024-03-06T07:33:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.036+0000 7ff6a2ffd640 1 -- 172.21.15.1:0/226703736 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c031960 con 0x7ff6a40c6b10 2024-03-06T07:33:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.036+0000 7ff6aa55c640 1 -- 172.21.15.1:0/226703736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a40c6b10 msgr2=0x7ff6a4155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.036+0000 7ff6aa55c640 1 --2- 172.21.15.1:0/226703736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a40c6b10 0x7ff6a4155c90 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0097a0 tx=0x7ff68c02b840 comp rx=0 tx=0).stop 2024-03-06T07:33:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.036+0000 7ff6aa55c640 1 -- 172.21.15.1:0/226703736 shutdown_connections 2024-03-06T07:33:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.036+0000 7ff6aa55c640 1 --2- 172.21.15.1:0/226703736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a40c6b10 0x7ff6a4155c90 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.036+0000 7ff6aa55c640 1 -- 172.21.15.1:0/226703736 >> 172.21.15.1:0/226703736 conn(0x7ff6a4066ee0 msgr2=0x7ff6a40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.036+0000 7ff6aa55c640 1 -- 172.21.15.1:0/226703736 shutdown_connections 2024-03-06T07:33:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.036+0000 7ff6aa55c640 1 -- 172.21.15.1:0/226703736 wait complete. 2024-03-06T07:33:17.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.038+0000 7ff6aa55c640 1 Processor -- start 2024-03-06T07:33:17.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.038+0000 7ff6aa55c640 1 -- start start 2024-03-06T07:33:17.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.039+0000 7ff6aa55c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6a40bf9d0 0x7ff6a40c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.039+0000 7ff6aa55c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a4152650 0x7ff6a41ef250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.039+0000 7ff6aa55c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6a414ba40 0x7ff6a41eb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.039+0000 7ff6aa55c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a41551b0 con 0x7ff6a4152650 2024-03-06T07:33:17.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.039+0000 7ff6aa55c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a40c6fa0 con 0x7ff6a40bf9d0 2024-03-06T07:33:17.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.039+0000 7ff6aa55c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a4155490 con 0x7ff6a414ba40 2024-03-06T07:33:17.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.039+0000 7ff6a37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a4152650 0x7ff6a41ef250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.039+0000 7ff6a37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a4152650 0x7ff6a41ef250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41620/0 (socket says 172.21.15.1:41620) 2024-03-06T07:33:17.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.039+0000 7ff6a37fe640 1 -- 172.21.15.1:0/2038962325 learned_addr learned my addr 172.21.15.1:0/2038962325 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:17.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.040+0000 7ff6a8af9640 1 --2- 172.21.15.1:0/2038962325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6a414ba40 0x7ff6a41eb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.040+0000 7ff6a3fff640 1 --2- 172.21.15.1:0/2038962325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6a40bf9d0 0x7ff6a40c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.040+0000 7ff6a37fe640 1 -- 172.21.15.1:0/2038962325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6a414ba40 msgr2=0x7ff6a41eb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.040+0000 7ff6a37fe640 1 --2- 172.21.15.1:0/2038962325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6a414ba40 0x7ff6a41eb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.040+0000 7ff6a37fe640 1 -- 172.21.15.1:0/2038962325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6a40bf9d0 msgr2=0x7ff6a40c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.040+0000 7ff6a37fe640 1 --2- 172.21.15.1:0/2038962325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6a40bf9d0 0x7ff6a40c9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.040+0000 7ff6a37fe640 1 -- 172.21.15.1:0/2038962325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a40ca3e0 con 0x7ff6a4152650 2024-03-06T07:33:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.040+0000 7ff6a3fff640 1 --2- 172.21.15.1:0/2038962325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6a40bf9d0 0x7ff6a40c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.040+0000 7ff6a37fe640 1 --2- 172.21.15.1:0/2038962325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a4152650 0x7ff6a41ef250 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0098c0 tx=0x7ff68c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.040+0000 7ff6a0ff9640 1 -- 172.21.15.1:0/2038962325 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c032840 con 0x7ff6a4152650 2024-03-06T07:33:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.040+0000 7ff6a0ff9640 1 -- 172.21.15.1:0/2038962325 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68c0329e0 con 0x7ff6a4152650 2024-03-06T07:33:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.040+0000 7ff6aa55c640 1 -- 172.21.15.1:0/2038962325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6a40ca6c0 con 0x7ff6a4152650 2024-03-06T07:33:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.040+0000 7ff6a0ff9640 1 -- 172.21.15.1:0/2038962325 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c032dd0 con 0x7ff6a4152650 2024-03-06T07:33:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.040+0000 7ff6aa55c640 1 -- 172.21.15.1:0/2038962325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6a41eea50 con 0x7ff6a4152650 2024-03-06T07:33:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.041+0000 7ff6aa55c640 1 -- 172.21.15.1:0/2038962325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6a40c2f50 con 0x7ff6a4152650 2024-03-06T07:33:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.042+0000 7ff6a0ff9640 1 -- 172.21.15.1:0/2038962325 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff68c04d3f0 con 0x7ff6a4152650 2024-03-06T07:33:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.042+0000 7ff6a0ff9640 1 --2- 172.21.15.1:0/2038962325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff68406b4d0 0x7ff68406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.042+0000 7ff6a0ff9640 1 -- 172.21.15.1:0/2038962325 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ff68c02f030 con 0x7ff6a4152650 2024-03-06T07:33:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.042+0000 7ff6a3fff640 1 --2- 172.21.15.1:0/2038962325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff68406b4d0 0x7ff68406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.043+0000 7ff6a3fff640 1 --2- 172.21.15.1:0/2038962325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff68406b4d0 0x7ff68406d990 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7ff694000f60 tx=0x7ff694007c10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.046+0000 7ff6a0ff9640 1 -- 172.21.15.1:0/2038962325 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff6a40c2f50 con 0x7ff6a4152650 2024-03-06T07:33:17.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.267+0000 7ff6aa55c640 1 -- 172.21.15.1:0/2038962325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6a4067fd0 con 0x7ff6a4152650 2024-03-06T07:33:17.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.267+0000 7ff6a0ff9640 1 -- 172.21.15.1:0/2038962325 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7ff68c040070 con 0x7ff6a4152650 2024-03-06T07:33:17.269 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:17.269 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:33:17.270 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:33:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.269+0000 7ff6aa55c640 1 -- 172.21.15.1:0/2038962325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff68406b4d0 msgr2=0x7ff68406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.269+0000 7ff6aa55c640 1 --2- 172.21.15.1:0/2038962325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff68406b4d0 0x7ff68406d990 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7ff694000f60 tx=0x7ff694007c10 comp rx=0 tx=0).stop 2024-03-06T07:33:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.269+0000 7ff6aa55c640 1 -- 172.21.15.1:0/2038962325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a4152650 msgr2=0x7ff6a41ef250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.269+0000 7ff6aa55c640 1 --2- 172.21.15.1:0/2038962325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a4152650 0x7ff6a41ef250 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0098c0 tx=0x7ff68c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.269+0000 7ff6aa55c640 1 -- 172.21.15.1:0/2038962325 shutdown_connections 2024-03-06T07:33:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.269+0000 7ff6aa55c640 1 --2- 172.21.15.1:0/2038962325 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7ff68406b4d0 0x7ff68406d990 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.269+0000 7ff6aa55c640 1 --2- 172.21.15.1:0/2038962325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6a414ba40 0x7ff6a41eb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.269+0000 7ff6aa55c640 1 --2- 172.21.15.1:0/2038962325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a4152650 0x7ff6a41ef250 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.269+0000 7ff6aa55c640 1 --2- 172.21.15.1:0/2038962325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6a40bf9d0 0x7ff6a40c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.269+0000 7ff6aa55c640 1 -- 172.21.15.1:0/2038962325 >> 172.21.15.1:0/2038962325 conn(0x7ff6a4066ee0 msgr2=0x7ff6a40c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.269+0000 7ff6aa55c640 1 -- 172.21.15.1:0/2038962325 shutdown_connections 2024-03-06T07:33:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:17.269+0000 7ff6aa55c640 1 -- 172.21.15.1:0/2038962325 wait complete. 2024-03-06T07:33:17.684 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:32.853395+0000', 'modified': '2024-03-06T07:31:24.318771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10806, 'mds_1': 8171}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8171': {'gid': 8171, 'name': 'b', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.98:6837/3233486394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 3233486394}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3233486394}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:24.318771+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10806': {'gid': 10806, 'name': 'c', 'rank': 0, 'incarnation': 193, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6835/1871567565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6833', 'nonce': 1871567565}, {'type': 'v1', 'addr': '172.21.15.1:6835', 'nonce': 1871567565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10806, 'qdb_cluster': [10806]} 2024-03-06T07:33:17.685 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8171:up:stopping 2024-03-06T07:33:18.686 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 16, "epoch": 213, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 16, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:29:32.853395+0000", "damaged": [], "data_pools": [ 34 ], "enabled": true, "epoch": 213, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_10806": { "addr": "172.21.15.1:6835/1871567565", "addrs": { "addrvec": [ { "addr": "172.21.15.1:6833", "nonce": 1871567565, "type": "v2" }, { "addr": "172.21.15.1:6835", "nonce": 1871567565, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 10806, "incarnation": 193, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 }, "gid_8171": { "addr": "172.21.15.98:6837/3233486394", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6835", "nonce": 3233486394, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 3233486394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8171, "incarnation": 207, "join_fscid": -1, "laggy_since": "2024-03-06T07:31:24.318771+0000", "name": "b", "rank": 1, "state": "up:stopping", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 214, "max_file_size": 1099511627776, "max_mds": 0, "max_xattr_size": 65536, "metadata_pool": 33, "modified": "2024-03-06T07:31:24.318771+0000", "qdb_cluster": [ 10806 ], "qdb_leader": 10806, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 10806, "mds_1": 8171 } } } ], "standbys": [] } 2024-03-06T07:33:18.686 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:33:18.686 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:18.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:18.777 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:33:18.777 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:33:18.778 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:18.793 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:33:18.793 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:33:18.793 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:18.811 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:33:18.811 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:18.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.932+0000 7fbb8ec0e640 1 Processor -- start 2024-03-06T07:33:18.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.932+0000 7fbb8ec0e640 1 -- start start 2024-03-06T07:33:18.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.933+0000 7fbb8ec0e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbb88152650 0x7fbb88155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.933+0000 7fbb8ec0e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb880c2fc0 con 0x7fbb880bf9d0 2024-03-06T07:33:18.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.933+0000 7fbb8ec0e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb88158da0 con 0x7fbb880c6b10 2024-03-06T07:33:18.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.933+0000 7fbb8ec0e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb88155800 con 0x7fbb88152650 2024-03-06T07:33:18.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.933+0000 7fbb8d1ab640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbb880bf9d0 0x7fbb880c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59954/0 (socket says 172.21.15.1:59954) 2024-03-06T07:33:18.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.933+0000 7fbb8d1ab640 1 -- 172.21.15.1:0/3520143049 learned_addr learned my addr 172.21.15.1:0/3520143049 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:18.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb877fe640 1 -- 172.21.15.1:0/3520143049 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701378027 0 0) 0x7fbb880c2fc0 con 0x7fbb880bf9d0 2024-03-06T07:33:18.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb877fe640 1 -- 172.21.15.1:0/3520143049 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb70003520 con 0x7fbb880bf9d0 2024-03-06T07:33:18.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb8c9aa640 1 --2- 172.21.15.1:0/3520143049 >> v2:172.21.15.98:3300/0 conn(0x7fbb88152650 0x7fbb88155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb877fe640 1 -- 172.21.15.1:0/3520143049 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1886417297 0 0) 0x7fbb70003520 con 0x7fbb880bf9d0 2024-03-06T07:33:18.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb877fe640 1 -- 172.21.15.1:0/3520143049 >> v2:172.21.15.98:3300/0 conn(0x7fbb88152650 msgr2=0x7fbb88155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:18.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb877fe640 1 --2- 172.21.15.1:0/3520143049 >> v2:172.21.15.98:3300/0 conn(0x7fbb88152650 0x7fbb88155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb877fe640 1 -- 172.21.15.1:0/3520143049 >> v1:172.21.15.98:6789/0 conn(0x7fbb880c6b10 legacy=0x7fbb88158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb877fe640 1 -- 172.21.15.1:0/3520143049 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb880c6690 con 0x7fbb880bf9d0 2024-03-06T07:33:18.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb877fe640 1 -- 172.21.15.1:0/3520143049 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fbb800019f0 con 0x7fbb880bf9d0 2024-03-06T07:33:18.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb877fe640 1 -- 172.21.15.1:0/3520143049 >> v1:172.21.15.1:6789/0 conn(0x7fbb880bf9d0 legacy=0x7fbb880c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb877fe640 1 --2- 172.21.15.1:0/3520143049 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb700039b0 0x7fbb70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb877fe640 1 --2- 172.21.15.1:0/3520143049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb70004800 0x7fbb70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb877fe640 1 --2- 172.21.15.1:0/3520143049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb700051c0 0x7fbb7000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb877fe640 1 -- 172.21.15.1:0/3520143049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb88155800 con 0x7fbb70004800 2024-03-06T07:33:18.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb877fe640 1 -- 172.21.15.1:0/3520143049 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7000c160 con 0x7fbb700051c0 2024-03-06T07:33:18.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb877fe640 1 -- 172.21.15.1:0/3520143049 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7000c360 con 0x7fbb700039b0 2024-03-06T07:33:18.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb87fff640 1 --2- 172.21.15.1:0/3520143049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb70004800 0x7fbb70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb8c9aa640 1 --2- 172.21.15.1:0/3520143049 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb700039b0 0x7fbb70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb8d1ab640 1 --2- 172.21.15.1:0/3520143049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb700051c0 0x7fbb7000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb87fff640 1 -- 172.21.15.1:0/3520143049 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb700039b0 msgr2=0x7fbb70003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:18.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb87fff640 1 --2- 172.21.15.1:0/3520143049 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb700039b0 0x7fbb70003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb87fff640 1 -- 172.21.15.1:0/3520143049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb700051c0 msgr2=0x7fbb7000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:18.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb87fff640 1 --2- 172.21.15.1:0/3520143049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb700051c0 0x7fbb7000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb87fff640 1 -- 172.21.15.1:0/3520143049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb70003520 con 0x7fbb70004800 2024-03-06T07:33:18.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb8d1ab640 1 --2- 172.21.15.1:0/3520143049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb700051c0 0x7fbb7000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:18.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.934+0000 7fbb8d1ab640 1 -- 172.21.15.1:0/3520143049 reap_dead start 2024-03-06T07:33:18.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.935+0000 7fbb87fff640 1 --2- 172.21.15.1:0/3520143049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb70004800 0x7fbb70004c80 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c001750 tx=0x7fbb70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3e255fa116f3218 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.935+0000 7fbb877fe640 1 -- 172.21.15.1:0/3520143049 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c001370 con 0x7fbb70004800 2024-03-06T07:33:18.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.935+0000 7fbb877fe640 1 -- 172.21.15.1:0/3520143049 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb7c0052d0 con 0x7fbb70004800 2024-03-06T07:33:18.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.935+0000 7fbb877fe640 1 -- 172.21.15.1:0/3520143049 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c003830 con 0x7fbb70004800 2024-03-06T07:33:18.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.936+0000 7fbb8ec0e640 1 -- 172.21.15.1:0/3520143049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb70004800 msgr2=0x7fbb70004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.936+0000 7fbb8ec0e640 1 --2- 172.21.15.1:0/3520143049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb70004800 0x7fbb70004c80 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c001750 tx=0x7fbb70005830 comp rx=0 tx=0).stop 2024-03-06T07:33:18.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.936+0000 7fbb8ec0e640 1 -- 172.21.15.1:0/3520143049 shutdown_connections 2024-03-06T07:33:18.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.936+0000 7fbb8ec0e640 1 --2- 172.21.15.1:0/3520143049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb70004800 0x7fbb70004c80 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.936+0000 7fbb8ec0e640 1 -- 172.21.15.1:0/3520143049 >> 172.21.15.1:0/3520143049 conn(0x7fbb88066ee0 msgr2=0x7fbb880672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:18.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.936+0000 7fbb8ec0e640 1 -- 172.21.15.1:0/3520143049 shutdown_connections 2024-03-06T07:33:18.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.936+0000 7fbb8ec0e640 1 -- 172.21.15.1:0/3520143049 wait complete. 2024-03-06T07:33:18.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.937+0000 7fbb8ec0e640 1 Processor -- start 2024-03-06T07:33:18.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.937+0000 7fbb8ec0e640 1 -- start start 2024-03-06T07:33:18.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb8ec0e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb70004800 0x7fbb8814fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb8ec0e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb8814c310 0x7fbb881e1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb8ec0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb8814b910 0x7fbb881eb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb8ec0e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb881551b0 con 0x7fbb8814c310 2024-03-06T07:33:18.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb8ec0e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb880c6fa0 con 0x7fbb8814b910 2024-03-06T07:33:18.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb8ec0e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb88155490 con 0x7fbb70004800 2024-03-06T07:33:18.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb87fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb8814c310 0x7fbb881e1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb87fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb8814c310 0x7fbb881e1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41642/0 (socket says 172.21.15.1:41642) 2024-03-06T07:33:18.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb87fff640 1 -- 172.21.15.1:0/2579215897 learned_addr learned my addr 172.21.15.1:0/2579215897 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:18.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb8c9aa640 1 --2- 172.21.15.1:0/2579215897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb70004800 0x7fbb8814fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb8d1ab640 1 --2- 172.21.15.1:0/2579215897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb8814b910 0x7fbb881eb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb87fff640 1 -- 172.21.15.1:0/2579215897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb70004800 msgr2=0x7fbb8814fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb87fff640 1 --2- 172.21.15.1:0/2579215897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb70004800 0x7fbb8814fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb87fff640 1 -- 172.21.15.1:0/2579215897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb8814b910 msgr2=0x7fbb881eb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb87fff640 1 --2- 172.21.15.1:0/2579215897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb8814b910 0x7fbb881eb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb87fff640 1 -- 172.21.15.1:0/2579215897 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb881516d0 con 0x7fbb8814c310 2024-03-06T07:33:18.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb87fff640 1 --2- 172.21.15.1:0/2579215897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb8814c310 0x7fbb881e1b10 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c005040 tx=0x7fbb7c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.938+0000 7fbb857fa640 1 -- 172.21.15.1:0/2579215897 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c041020 con 0x7fbb8814c310 2024-03-06T07:33:18.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.939+0000 7fbb857fa640 1 -- 172.21.15.1:0/2579215897 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb7c005480 con 0x7fbb8814c310 2024-03-06T07:33:18.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.939+0000 7fbb8ec0e640 1 -- 172.21.15.1:0/2579215897 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb881519b0 con 0x7fbb8814c310 2024-03-06T07:33:18.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.939+0000 7fbb857fa640 1 -- 172.21.15.1:0/2579215897 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c007a30 con 0x7fbb8814c310 2024-03-06T07:33:18.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.939+0000 7fbb8ec0e640 1 -- 172.21.15.1:0/2579215897 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb881e2050 con 0x7fbb8814c310 2024-03-06T07:33:18.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.940+0000 7fbb8ec0e640 1 -- 172.21.15.1:0/2579215897 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb880686e0 con 0x7fbb8814c310 2024-03-06T07:33:18.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.940+0000 7fbb857fa640 1 -- 172.21.15.1:0/2579215897 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb7c044020 con 0x7fbb8814c310 2024-03-06T07:33:18.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.940+0000 7fbb857fa640 1 --2- 172.21.15.1:0/2579215897 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb7406b4d0 0x7fbb7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.940+0000 7fbb857fa640 1 -- 172.21.15.1:0/2579215897 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fbb7c0abd90 con 0x7fbb8814c310 2024-03-06T07:33:18.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.941+0000 7fbb8c9aa640 1 --2- 172.21.15.1:0/2579215897 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb7406b4d0 0x7fbb7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.941+0000 7fbb8c9aa640 1 --2- 172.21.15.1:0/2579215897 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb7406b4d0 0x7fbb7406d990 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fbb78009c40 tx=0x7fbb78009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:18.949+0000 7fbb857fa640 1 -- 172.21.15.1:0/2579215897 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb880686e0 con 0x7fbb8814c310 2024-03-06T07:33:19.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.169+0000 7fbb8ec0e640 1 -- 172.21.15.1:0/2579215897 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb88067fd0 con 0x7fbb8814c310 2024-03-06T07:33:19.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.169+0000 7fbb857fa640 1 -- 172.21.15.1:0/2579215897 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fbb7c04b030 con 0x7fbb8814c310 2024-03-06T07:33:19.171 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:19.171 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:33:19.172 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:33:19.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.171+0000 7fbb8ec0e640 1 -- 172.21.15.1:0/2579215897 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb7406b4d0 msgr2=0x7fbb7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.171+0000 7fbb8ec0e640 1 --2- 172.21.15.1:0/2579215897 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb7406b4d0 0x7fbb7406d990 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fbb78009c40 tx=0x7fbb78009190 comp rx=0 tx=0).stop 2024-03-06T07:33:19.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.171+0000 7fbb8ec0e640 1 -- 172.21.15.1:0/2579215897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb8814c310 msgr2=0x7fbb881e1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.171+0000 7fbb8ec0e640 1 --2- 172.21.15.1:0/2579215897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb8814c310 0x7fbb881e1b10 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c005040 tx=0x7fbb7c001120 comp rx=0 tx=0).stop 2024-03-06T07:33:19.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.171+0000 7fbb8ec0e640 1 -- 172.21.15.1:0/2579215897 shutdown_connections 2024-03-06T07:33:19.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.171+0000 7fbb8ec0e640 1 --2- 172.21.15.1:0/2579215897 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbb7406b4d0 0x7fbb7406d990 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.171+0000 7fbb8ec0e640 1 --2- 172.21.15.1:0/2579215897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb8814b910 0x7fbb881eb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.171+0000 7fbb8ec0e640 1 --2- 172.21.15.1:0/2579215897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb8814c310 0x7fbb881e1b10 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.171+0000 7fbb8ec0e640 1 --2- 172.21.15.1:0/2579215897 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb70004800 0x7fbb8814fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.171+0000 7fbb8ec0e640 1 -- 172.21.15.1:0/2579215897 >> 172.21.15.1:0/2579215897 conn(0x7fbb88066ee0 msgr2=0x7fbb880c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.171+0000 7fbb8ec0e640 1 -- 172.21.15.1:0/2579215897 shutdown_connections 2024-03-06T07:33:19.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.171+0000 7fbb8ec0e640 1 -- 172.21.15.1:0/2579215897 wait complete. 2024-03-06T07:33:19.185 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:19.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.284+0000 7f2fe95d6640 1 Processor -- start 2024-03-06T07:33:19.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.284+0000 7f2fe95d6640 1 -- start start 2024-03-06T07:33:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.285+0000 7f2fe95d6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2fe4152820 0x7f2fe40c0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.285+0000 7f2fe95d6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fe40c0670 0x7f2fe40c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.285+0000 7f2fe95d6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2fe40c7930 0x7f2fe4158830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.285+0000 7f2fe95d6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe40c7180 con 0x7f2fe4152820 2024-03-06T07:33:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.285+0000 7f2fe95d6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe4155460 con 0x7f2fe40c7930 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.285+0000 7f2fe95d6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40c6bc0 con 0x7f2fe40c0670 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.285+0000 7f2fe2ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2fe4152820 0x7f2fe40c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.285+0000 7f2fe2ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2fe4152820 0x7f2fe40c0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41658/0 (socket says 172.21.15.1:41658) 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.285+0000 7f2fe2ffd640 1 -- 172.21.15.1:0/1090643652 learned_addr learned my addr 172.21.15.1:0/1090643652 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.285+0000 7f2fe27fc640 1 --2- 172.21.15.1:0/1090643652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fe40c0670 0x7f2fe40c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.285+0000 7f2fe37fe640 1 --2- 172.21.15.1:0/1090643652 >> v2:172.21.15.98:3300/0 conn(0x7f2fe40c7930 0x7f2fe4158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.285+0000 7f2fe2ffd640 1 -- 172.21.15.1:0/1090643652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fe40c0670 msgr2=0x7f2fe40c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.285+0000 7f2fe2ffd640 1 --2- 172.21.15.1:0/1090643652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fe40c0670 0x7f2fe40c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.285+0000 7f2fe2ffd640 1 -- 172.21.15.1:0/1090643652 >> v2:172.21.15.98:3300/0 conn(0x7f2fe40c7930 msgr2=0x7f2fe4158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.285+0000 7f2fe2ffd640 1 --2- 172.21.15.1:0/1090643652 >> v2:172.21.15.98:3300/0 conn(0x7f2fe40c7930 0x7f2fe4158830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.285+0000 7f2fe2ffd640 1 -- 172.21.15.1:0/1090643652 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe40c6ea0 con 0x7f2fe4152820 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.286+0000 7f2fe2ffd640 1 --2- 172.21.15.1:0/1090643652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe4152820 0x7f2fe40c0130 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f2fe40c5650 tx=0x7f2fd802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b4d022ecc6212d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.286+0000 7f2fe1ffb640 1 -- 172.21.15.1:0/1090643652 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8035020 con 0x7f2fe4152820 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.286+0000 7f2fe1ffb640 1 -- 172.21.15.1:0/1090643652 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd8031870 con 0x7f2fe4152820 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.286+0000 7f2fe1ffb640 1 -- 172.21.15.1:0/1090643652 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8031c40 con 0x7f2fe4152820 2024-03-06T07:33:19.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.287+0000 7f2fe95d6640 1 -- 172.21.15.1:0/1090643652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe4152820 msgr2=0x7f2fe40c0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.287+0000 7f2fe95d6640 1 --2- 172.21.15.1:0/1090643652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe4152820 0x7f2fe40c0130 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f2fe40c5650 tx=0x7f2fd802b910 comp rx=0 tx=0).stop 2024-03-06T07:33:19.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.287+0000 7f2fe95d6640 1 -- 172.21.15.1:0/1090643652 shutdown_connections 2024-03-06T07:33:19.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.287+0000 7f2fe95d6640 1 --2- 172.21.15.1:0/1090643652 >> v2:172.21.15.98:3300/0 conn(0x7f2fe40c7930 0x7f2fe4158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.287+0000 7f2fe95d6640 1 --2- 172.21.15.1:0/1090643652 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fe40c0670 0x7f2fe40c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.287+0000 7f2fe95d6640 1 --2- 172.21.15.1:0/1090643652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe4152820 0x7f2fe40c0130 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.287+0000 7f2fe95d6640 1 -- 172.21.15.1:0/1090643652 >> 172.21.15.1:0/1090643652 conn(0x7f2fe4072720 msgr2=0x7f2fe4072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.287+0000 7f2fe95d6640 1 -- 172.21.15.1:0/1090643652 shutdown_connections 2024-03-06T07:33:19.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.287+0000 7f2fe95d6640 1 -- 172.21.15.1:0/1090643652 wait complete. 2024-03-06T07:33:19.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.288+0000 7f2fe95d6640 1 Processor -- start 2024-03-06T07:33:19.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.289+0000 7f2fe95d6640 1 -- start start 2024-03-06T07:33:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe95d6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe40c0670 0x7f2fe40c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe95d6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fe40c7930 0x7f2fe414ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe95d6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fe4152820 0x7f2fe414b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe95d6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe4152d80 con 0x7f2fe40c0670 2024-03-06T07:33:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe95d6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe4152c00 con 0x7f2fe4152820 2024-03-06T07:33:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe95d6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40af6f0 con 0x7f2fe40c7930 2024-03-06T07:33:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe40c0670 0x7f2fe40c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe40c0670 0x7f2fe40c36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41672/0 (socket says 172.21.15.1:41672) 2024-03-06T07:33:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe2ffd640 1 -- 172.21.15.1:0/3249946944 learned_addr learned my addr 172.21.15.1:0/3249946944 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe27fc640 1 --2- 172.21.15.1:0/3249946944 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fe40c7930 0x7f2fe414ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe37fe640 1 --2- 172.21.15.1:0/3249946944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fe4152820 0x7f2fe414b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe2ffd640 1 -- 172.21.15.1:0/3249946944 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fe40c7930 msgr2=0x7f2fe414ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe2ffd640 1 --2- 172.21.15.1:0/3249946944 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fe40c7930 0x7f2fe414ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe2ffd640 1 -- 172.21.15.1:0/3249946944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fe4152820 msgr2=0x7f2fe414b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe2ffd640 1 --2- 172.21.15.1:0/3249946944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fe4152820 0x7f2fe414b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe2ffd640 1 -- 172.21.15.1:0/3249946944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe41eb5b0 con 0x7f2fe40c0670 2024-03-06T07:33:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe37fe640 1 --2- 172.21.15.1:0/3249946944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fe4152820 0x7f2fe414b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe2ffd640 1 --2- 172.21.15.1:0/3249946944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe40c0670 0x7f2fe40c36b0 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f2fd802b950 tx=0x7f2fd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/3249946944 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8035040 con 0x7f2fe40c0670 2024-03-06T07:33:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/3249946944 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd8030d00 con 0x7f2fe40c0670 2024-03-06T07:33:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fe95d6640 1 -- 172.21.15.1:0/3249946944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe4159060 con 0x7f2fe40c0670 2024-03-06T07:33:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.290+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/3249946944 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd80456a0 con 0x7f2fe40c0670 2024-03-06T07:33:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.291+0000 7f2fe95d6640 1 -- 172.21.15.1:0/3249946944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe4159620 con 0x7f2fe40c0670 2024-03-06T07:33:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.291+0000 7f2fe95d6640 1 -- 172.21.15.1:0/3249946944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe40686e0 con 0x7f2fe40c0670 2024-03-06T07:33:19.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.292+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/3249946944 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fd8038020 con 0x7f2fe40c0670 2024-03-06T07:33:19.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.292+0000 7f2fcb7fe640 1 --2- 172.21.15.1:0/3249946944 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fd006b500 0x7f2fd006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.292+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/3249946944 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f2fd80ac5b0 con 0x7f2fe40c0670 2024-03-06T07:33:19.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.293+0000 7f2fe27fc640 1 --2- 172.21.15.1:0/3249946944 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fd006b500 0x7f2fd006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.293+0000 7f2fe27fc640 1 --2- 172.21.15.1:0/3249946944 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fd006b500 0x7f2fd006d9c0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f2fe40c48c0 tx=0x7f2fd4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.297+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/3249946944 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fd807b230 con 0x7f2fe40c0670 2024-03-06T07:33:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.528+0000 7f2fe95d6640 1 -- 172.21.15.1:0/3249946944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fe40c5b40 con 0x7f2fe40c0670 2024-03-06T07:33:19.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.529+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/3249946944 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v214) v1 ==== 74+0+17167 (secure 0 0 0) 0x7f2fd807ab30 con 0x7f2fe40c0670 2024-03-06T07:33:19.531 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:19.531 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":214,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:30:49.314643+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:31.114390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:31.815153+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"210","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"34.26","mappings":[{"from":0,"to":2}]},{"pgid":"34.31","mappings":[{"from":0,"to":2}]},{"pgid":"34.3f","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6836/2083320955":"2024-03-07T07:30:49.314605+0000","172.21.15.98:6834/2083320955":"2024-03-07T07:30:49.314605+0000","172.21.15.1:6834/2424540959":"2024-03-07T07:30:14.310288+0000","172.21.15.1:6832/2424540959":"2024-03-07T07:30:14.310288+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.531+0000 7f2fe95d6640 1 -- 172.21.15.1:0/3249946944 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fd006b500 msgr2=0x7f2fd006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.531+0000 7f2fe95d6640 1 --2- 172.21.15.1:0/3249946944 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fd006b500 0x7f2fd006d9c0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f2fe40c48c0 tx=0x7f2fd4008040 comp rx=0 tx=0).stop 2024-03-06T07:33:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.531+0000 7f2fe95d6640 1 -- 172.21.15.1:0/3249946944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe40c0670 msgr2=0x7f2fe40c36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.531+0000 7f2fe95d6640 1 --2- 172.21.15.1:0/3249946944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe40c0670 0x7f2fe40c36b0 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f2fd802b950 tx=0x7f2fd80046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.531+0000 7f2fe95d6640 1 -- 172.21.15.1:0/3249946944 shutdown_connections 2024-03-06T07:33:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.531+0000 7f2fe95d6640 1 --2- 172.21.15.1:0/3249946944 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f2fd006b500 0x7f2fd006d9c0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.531+0000 7f2fe95d6640 1 --2- 172.21.15.1:0/3249946944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fe4152820 0x7f2fe414b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.531+0000 7f2fe95d6640 1 --2- 172.21.15.1:0/3249946944 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2fe40c7930 0x7f2fe414ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.531+0000 7f2fe95d6640 1 --2- 172.21.15.1:0/3249946944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe40c0670 0x7f2fe40c36b0 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.531+0000 7f2fe95d6640 1 -- 172.21.15.1:0/3249946944 >> 172.21.15.1:0/3249946944 conn(0x7f2fe4072720 msgr2=0x7f2fe40c8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.531+0000 7f2fe95d6640 1 -- 172.21.15.1:0/3249946944 shutdown_connections 2024-03-06T07:33:19.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.531+0000 7f2fe95d6640 1 -- 172.21.15.1:0/3249946944 wait complete. 2024-03-06T07:33:19.548 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:20:14.728260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:29:31.114390+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '209', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:29:31.815153+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '210', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:19.548 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.656+0000 7f7e7e288640 1 Processor -- start 2024-03-06T07:33:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.656+0000 7f7e7e288640 1 -- start start 2024-03-06T07:33:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.657+0000 7f7e7e288640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e780c4560 0x7f7e78155c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.657+0000 7f7e7e288640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7e780c4940 0x7f7e781561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.657+0000 7f7e7e288640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e780c7420 con 0x7f7e780c39c0 2024-03-06T07:33:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.657+0000 7f7e7e288640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e781566f0 con 0x7f7e780c4940 2024-03-06T07:33:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.657+0000 7f7e7e288640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e780ab760 con 0x7f7e780c4560 2024-03-06T07:33:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.657+0000 7f7e777fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7e780c39c0 0x7f7e780ac220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59962/0 (socket says 172.21.15.1:59962) 2024-03-06T07:33:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.657+0000 7f7e777fe640 1 -- 172.21.15.1:0/1240008927 learned_addr learned my addr 172.21.15.1:0/1240008927 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.657+0000 7f7e76ffd640 1 -- 172.21.15.1:0/1240008927 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659613766 0 0) 0x7f7e780c7420 con 0x7f7e780c39c0 2024-03-06T07:33:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e76ffd640 1 -- 172.21.15.1:0/1240008927 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e60003520 con 0x7f7e780c39c0 2024-03-06T07:33:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e77fff640 1 --2- 172.21.15.1:0/1240008927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e780c4560 0x7f7e78155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e7c825640 1 --2- 172.21.15.1:0/1240008927 >> v2:172.21.15.98:3300/0 conn(0x7f7e780c4940 0x7f7e781561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e76ffd640 1 -- 172.21.15.1:0/1240008927 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3184295742 0 0) 0x7f7e60003520 con 0x7f7e780c39c0 2024-03-06T07:33:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e76ffd640 1 -- 172.21.15.1:0/1240008927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e780c4560 msgr2=0x7f7e78155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e76ffd640 1 --2- 172.21.15.1:0/1240008927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e780c4560 0x7f7e78155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e76ffd640 1 -- 172.21.15.1:0/1240008927 >> v2:172.21.15.98:3300/0 conn(0x7f7e780c4940 msgr2=0x7f7e781561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e76ffd640 1 --2- 172.21.15.1:0/1240008927 >> v2:172.21.15.98:3300/0 conn(0x7f7e780c4940 0x7f7e781561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e76ffd640 1 -- 172.21.15.1:0/1240008927 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e780c76c0 con 0x7f7e780c39c0 2024-03-06T07:33:19.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e76ffd640 1 -- 172.21.15.1:0/1240008927 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f7e68001670 con 0x7f7e780c39c0 2024-03-06T07:33:19.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e76ffd640 1 -- 172.21.15.1:0/1240008927 >> v1:172.21.15.1:6789/0 conn(0x7f7e780c39c0 legacy=0x7f7e780ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e76ffd640 1 --2- 172.21.15.1:0/1240008927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e600039b0 0x7f7e60003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e76ffd640 1 --2- 172.21.15.1:0/1240008927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e780c4560 0x7f7e60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e76ffd640 1 --2- 172.21.15.1:0/1240008927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e60004d40 0x7f7e600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e76ffd640 1 -- 172.21.15.1:0/1240008927 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e781566f0 con 0x7f7e60004d40 2024-03-06T07:33:19.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e76ffd640 1 -- 172.21.15.1:0/1240008927 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e780ab760 con 0x7f7e600039b0 2024-03-06T07:33:19.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e76ffd640 1 -- 172.21.15.1:0/1240008927 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6000b9c0 con 0x7f7e780c4560 2024-03-06T07:33:19.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e7c825640 1 --2- 172.21.15.1:0/1240008927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e60004d40 0x7f7e600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e77fff640 1 --2- 172.21.15.1:0/1240008927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e600039b0 0x7f7e60003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e777fe640 1 --2- 172.21.15.1:0/1240008927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e780c4560 0x7f7e60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e7c825640 1 -- 172.21.15.1:0/1240008927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e780c4560 msgr2=0x7f7e60004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e7c825640 1 --2- 172.21.15.1:0/1240008927 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e780c4560 0x7f7e60004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e7c825640 1 -- 172.21.15.1:0/1240008927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e600039b0 msgr2=0x7f7e60003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e7c825640 1 --2- 172.21.15.1:0/1240008927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e600039b0 0x7f7e60003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.658+0000 7f7e7c825640 1 -- 172.21.15.1:0/1240008927 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e60003520 con 0x7f7e60004d40 2024-03-06T07:33:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.659+0000 7f7e7c825640 1 --2- 172.21.15.1:0/1240008927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e60004d40 0x7f7e600051c0 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7f7e70009f90 tx=0x7f7e60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b418eeb8d3e837 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.659+0000 7f7e76ffd640 1 -- 172.21.15.1:0/1240008927 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e70039070 con 0x7f7e60004d40 2024-03-06T07:33:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.659+0000 7f7e76ffd640 1 -- 172.21.15.1:0/1240008927 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7002cc20 con 0x7f7e60004d40 2024-03-06T07:33:19.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.659+0000 7f7e76ffd640 1 -- 172.21.15.1:0/1240008927 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e70009c10 con 0x7f7e60004d40 2024-03-06T07:33:19.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.660+0000 7f7e7e288640 1 -- 172.21.15.1:0/1240008927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e60004d40 msgr2=0x7f7e600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.660+0000 7f7e7e288640 1 --2- 172.21.15.1:0/1240008927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e60004d40 0x7f7e600051c0 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7f7e70009f90 tx=0x7f7e60005830 comp rx=0 tx=0).stop 2024-03-06T07:33:19.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.660+0000 7f7e7c825640 1 -- 172.21.15.1:0/1240008927 reap_dead start 2024-03-06T07:33:19.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.660+0000 7f7e7e288640 1 -- 172.21.15.1:0/1240008927 shutdown_connections 2024-03-06T07:33:19.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.660+0000 7f7e7e288640 1 -- 172.21.15.1:0/1240008927 >> 172.21.15.1:0/1240008927 conn(0x7f7e780c7a50 msgr2=0x7f7e780c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.660+0000 7f7e7e288640 1 -- 172.21.15.1:0/1240008927 shutdown_connections 2024-03-06T07:33:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.660+0000 7f7e7e288640 1 -- 172.21.15.1:0/1240008927 wait complete. 2024-03-06T07:33:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.661+0000 7f7e7e288640 1 Processor -- start 2024-03-06T07:33:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.661+0000 7f7e7e288640 1 -- start start 2024-03-06T07:33:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.662+0000 7f7e7e288640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e781ebd10 0x7f7e781ec0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.662+0000 7f7e7e288640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e781529c0 0x7f7e781515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.662+0000 7f7e7e288640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e780c0290 0x7f7e78151b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.662+0000 7f7e7e288640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7806e6d0 con 0x7f7e781529c0 2024-03-06T07:33:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.662+0000 7f7e7e288640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e780abd00 con 0x7f7e780c0290 2024-03-06T07:33:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.662+0000 7f7e7e288640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7806e850 con 0x7f7e781ebd10 2024-03-06T07:33:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.662+0000 7f7e777fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e781529c0 0x7f7e781515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.662+0000 7f7e777fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e781529c0 0x7f7e781515e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41682/0 (socket says 172.21.15.1:41682) 2024-03-06T07:33:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.662+0000 7f7e777fe640 1 -- 172.21.15.1:0/2645867209 learned_addr learned my addr 172.21.15.1:0/2645867209 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:19.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.662+0000 7f7e77fff640 1 --2- 172.21.15.1:0/2645867209 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e781ebd10 0x7f7e781ec0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.662+0000 7f7e777fe640 1 -- 172.21.15.1:0/2645867209 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e781ebd10 msgr2=0x7f7e781ec0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.662+0000 7f7e7c825640 1 --2- 172.21.15.1:0/2645867209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e780c0290 0x7f7e78151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.662+0000 7f7e777fe640 1 --2- 172.21.15.1:0/2645867209 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e781ebd10 0x7f7e781ec0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.662+0000 7f7e777fe640 1 -- 172.21.15.1:0/2645867209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e780c0290 msgr2=0x7f7e78151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.662+0000 7f7e777fe640 1 --2- 172.21.15.1:0/2645867209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e780c0290 0x7f7e78151b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.662+0000 7f7e777fe640 1 -- 172.21.15.1:0/2645867209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e78152f40 con 0x7f7e781529c0 2024-03-06T07:33:19.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.663+0000 7f7e777fe640 1 --2- 172.21.15.1:0/2645867209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e781529c0 0x7f7e781515e0 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f7e680072c0 tx=0x7f7e68007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.663+0000 7f7e74ff9640 1 -- 172.21.15.1:0/2645867209 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6800fa00 con 0x7f7e781529c0 2024-03-06T07:33:19.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.663+0000 7f7e74ff9640 1 -- 172.21.15.1:0/2645867209 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e68018070 con 0x7f7e781529c0 2024-03-06T07:33:19.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.663+0000 7f7e7e288640 1 -- 172.21.15.1:0/2645867209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e78153220 con 0x7f7e781529c0 2024-03-06T07:33:19.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.663+0000 7f7e74ff9640 1 -- 172.21.15.1:0/2645867209 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6800fe00 con 0x7f7e781529c0 2024-03-06T07:33:19.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.663+0000 7f7e7e288640 1 -- 172.21.15.1:0/2645867209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e78150160 con 0x7f7e781529c0 2024-03-06T07:33:19.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.664+0000 7f7e7e288640 1 -- 172.21.15.1:0/2645867209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e780686e0 con 0x7f7e781529c0 2024-03-06T07:33:19.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.664+0000 7f7e74ff9640 1 -- 172.21.15.1:0/2645867209 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e6801d460 con 0x7f7e781529c0 2024-03-06T07:33:19.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.665+0000 7f7e74ff9640 1 --2- 172.21.15.1:0/2645867209 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7e6406b4d0 0x7f7e6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.665+0000 7f7e74ff9640 1 -- 172.21.15.1:0/2645867209 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f7e6808a6c0 con 0x7f7e781529c0 2024-03-06T07:33:19.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.665+0000 7f7e77fff640 1 --2- 172.21.15.1:0/2645867209 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7e6406b4d0 0x7f7e6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.666+0000 7f7e77fff640 1 --2- 172.21.15.1:0/2645867209 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7e6406b4d0 0x7f7e6406d990 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f7e780acb50 tx=0x7f7e6c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.669+0000 7f7e74ff9640 1 -- 172.21.15.1:0/2645867209 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e68059340 con 0x7f7e781529c0 2024-03-06T07:33:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.849+0000 7f7e7e288640 1 -- 172.21.15.1:0/2645867209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7e780c6180 con 0x7f7e781529c0 2024-03-06T07:33:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.850+0000 7f7e74ff9640 1 -- 172.21.15.1:0/2645867209 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v214) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7e68058c40 con 0x7f7e781529c0 2024-03-06T07:33:19.852 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-06T07:33:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.851+0000 7f7e7e288640 1 -- 172.21.15.1:0/2645867209 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7e6406b4d0 msgr2=0x7f7e6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.851+0000 7f7e7e288640 1 --2- 172.21.15.1:0/2645867209 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7e6406b4d0 0x7f7e6406d990 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f7e780acb50 tx=0x7f7e6c008040 comp rx=0 tx=0).stop 2024-03-06T07:33:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.851+0000 7f7e7e288640 1 -- 172.21.15.1:0/2645867209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e781529c0 msgr2=0x7f7e781515e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.852+0000 7f7e7e288640 1 --2- 172.21.15.1:0/2645867209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e781529c0 0x7f7e781515e0 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f7e680072c0 tx=0x7f7e68007780 comp rx=0 tx=0).stop 2024-03-06T07:33:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.852+0000 7f7e7e288640 1 -- 172.21.15.1:0/2645867209 shutdown_connections 2024-03-06T07:33:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.852+0000 7f7e7e288640 1 --2- 172.21.15.1:0/2645867209 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f7e6406b4d0 0x7f7e6406d990 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.852+0000 7f7e7e288640 1 --2- 172.21.15.1:0/2645867209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e780c0290 0x7f7e78151b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.852+0000 7f7e7e288640 1 --2- 172.21.15.1:0/2645867209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e781529c0 0x7f7e781515e0 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.852+0000 7f7e7e288640 1 --2- 172.21.15.1:0/2645867209 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7e781ebd10 0x7f7e781ec0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.852+0000 7f7e7e288640 1 -- 172.21.15.1:0/2645867209 >> 172.21.15.1:0/2645867209 conn(0x7f7e780c7a50 msgr2=0x7f7e78154eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.852+0000 7f7e7e288640 1 -- 172.21.15.1:0/2645867209 shutdown_connections 2024-03-06T07:33:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.852+0000 7f7e7e288640 1 -- 172.21.15.1:0/2645867209 wait complete. 2024-03-06T07:33:19.864 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:33:20.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.999+0000 7fd1701bc640 1 Processor -- start 2024-03-06T07:33:20.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:19.999+0000 7fd1701bc640 1 -- start start 2024-03-06T07:33:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.000+0000 7fd1701bc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd168152650 0x7fd168155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.000+0000 7fd1701bc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1680c6b10 0x7fd168158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.000+0000 7fd1701bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1680bf9d0 0x7fd1680c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.000+0000 7fd1701bc640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd168155800 con 0x7fd1680c6b10 2024-03-06T07:33:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.000+0000 7fd1701bc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1680c2fc0 con 0x7fd168152650 2024-03-06T07:33:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.000+0000 7fd1701bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1680c6690 con 0x7fd1680bf9d0 2024-03-06T07:33:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.000+0000 7fd16d757640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1680c6b10 0x7fd168158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.000+0000 7fd16d757640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1680c6b10 0x7fd168158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41694/0 (socket says 172.21.15.1:41694) 2024-03-06T07:33:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.000+0000 7fd16d757640 1 -- 172.21.15.1:0/2663765381 learned_addr learned my addr 172.21.15.1:0/2663765381 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.000+0000 7fd16e759640 1 --2- 172.21.15.1:0/2663765381 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1680bf9d0 0x7fd1680c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.000+0000 7fd16df58640 1 --2- 172.21.15.1:0/2663765381 >> v2:172.21.15.98:3300/0 conn(0x7fd168152650 0x7fd168155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.000+0000 7fd16d757640 1 -- 172.21.15.1:0/2663765381 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1680bf9d0 msgr2=0x7fd1680c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:20.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.000+0000 7fd16d757640 1 --2- 172.21.15.1:0/2663765381 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1680bf9d0 0x7fd1680c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.000+0000 7fd16d757640 1 -- 172.21.15.1:0/2663765381 >> v2:172.21.15.98:3300/0 conn(0x7fd168152650 msgr2=0x7fd168155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:20.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.000+0000 7fd16d757640 1 --2- 172.21.15.1:0/2663765381 >> v2:172.21.15.98:3300/0 conn(0x7fd168152650 0x7fd168155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.000+0000 7fd16d757640 1 -- 172.21.15.1:0/2663765381 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1680c7280 con 0x7fd1680c6b10 2024-03-06T07:33:20.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.000+0000 7fd16df58640 1 --2- 172.21.15.1:0/2663765381 >> v2:172.21.15.98:3300/0 conn(0x7fd168152650 0x7fd168155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:20.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.001+0000 7fd16d757640 1 --2- 172.21.15.1:0/2663765381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1680c6b10 0x7fd168158690 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7fd1680cac90 tx=0x7fd15c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab04c1ddde9760c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.001+0000 7fd16cf56640 1 -- 172.21.15.1:0/2663765381 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c02e070 con 0x7fd1680c6b10 2024-03-06T07:33:20.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.001+0000 7fd16cf56640 1 -- 172.21.15.1:0/2663765381 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15c037070 con 0x7fd1680c6b10 2024-03-06T07:33:20.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.001+0000 7fd16cf56640 1 -- 172.21.15.1:0/2663765381 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c031a00 con 0x7fd1680c6b10 2024-03-06T07:33:20.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.002+0000 7fd1701bc640 1 -- 172.21.15.1:0/2663765381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1680c6b10 msgr2=0x7fd168158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.002+0000 7fd1701bc640 1 --2- 172.21.15.1:0/2663765381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1680c6b10 0x7fd168158690 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7fd1680cac90 tx=0x7fd15c02b880 comp rx=0 tx=0).stop 2024-03-06T07:33:20.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.002+0000 7fd1701bc640 1 -- 172.21.15.1:0/2663765381 shutdown_connections 2024-03-06T07:33:20.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.002+0000 7fd1701bc640 1 --2- 172.21.15.1:0/2663765381 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1680bf9d0 0x7fd1680c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.002+0000 7fd1701bc640 1 --2- 172.21.15.1:0/2663765381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1680c6b10 0x7fd168158690 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.002+0000 7fd1701bc640 1 --2- 172.21.15.1:0/2663765381 >> v2:172.21.15.98:3300/0 conn(0x7fd168152650 0x7fd168155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.002+0000 7fd1701bc640 1 -- 172.21.15.1:0/2663765381 >> 172.21.15.1:0/2663765381 conn(0x7fd168066ee0 msgr2=0x7fd1680672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:20.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.002+0000 7fd1701bc640 1 -- 172.21.15.1:0/2663765381 shutdown_connections 2024-03-06T07:33:20.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.002+0000 7fd1701bc640 1 -- 172.21.15.1:0/2663765381 wait complete. 2024-03-06T07:33:20.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.003+0000 7fd1701bc640 1 Processor -- start 2024-03-06T07:33:20.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.003+0000 7fd1701bc640 1 -- start start 2024-03-06T07:33:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.004+0000 7fd1701bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1680bf9d0 0x7fd1681e1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.004+0000 7fd1701bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd168152650 0x7fd16814c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.004+0000 7fd1701bc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168144cd0 0x7fd16814c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.004+0000 7fd1701bc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd168155490 con 0x7fd168144cd0 2024-03-06T07:33:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.004+0000 7fd1701bc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1681551b0 con 0x7fd1680bf9d0 2024-03-06T07:33:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.004+0000 7fd1701bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd168152a30 con 0x7fd168152650 2024-03-06T07:33:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.004+0000 7fd16e759640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168144cd0 0x7fd16814c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.004+0000 7fd16e759640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168144cd0 0x7fd16814c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41704/0 (socket says 172.21.15.1:41704) 2024-03-06T07:33:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.004+0000 7fd16e759640 1 -- 172.21.15.1:0/824646468 learned_addr learned my addr 172.21.15.1:0/824646468 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.004+0000 7fd16d757640 1 --2- 172.21.15.1:0/824646468 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd168152650 0x7fd16814c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.004+0000 7fd16df58640 1 --2- 172.21.15.1:0/824646468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1680bf9d0 0x7fd1681e1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.004+0000 7fd16e759640 1 -- 172.21.15.1:0/824646468 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd168152650 msgr2=0x7fd16814c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.004+0000 7fd16e759640 1 --2- 172.21.15.1:0/824646468 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd168152650 0x7fd16814c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.004+0000 7fd16e759640 1 -- 172.21.15.1:0/824646468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1680bf9d0 msgr2=0x7fd1681e1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.004+0000 7fd16e759640 1 --2- 172.21.15.1:0/824646468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1680bf9d0 0x7fd1681e1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.004+0000 7fd16e759640 1 -- 172.21.15.1:0/824646468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1681e10b0 con 0x7fd168144cd0 2024-03-06T07:33:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.004+0000 7fd16df58640 1 --2- 172.21.15.1:0/824646468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1680bf9d0 0x7fd1681e1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.005+0000 7fd16e759640 1 --2- 172.21.15.1:0/824646468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168144cd0 0x7fd16814c910 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7fd158007d20 tx=0x7fd15800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.005+0000 7fd1667fc640 1 -- 172.21.15.1:0/824646468 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158014420 con 0x7fd168144cd0 2024-03-06T07:33:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.005+0000 7fd1667fc640 1 -- 172.21.15.1:0/824646468 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd158019070 con 0x7fd168144cd0 2024-03-06T07:33:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.005+0000 7fd1667fc640 1 -- 172.21.15.1:0/824646468 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158014820 con 0x7fd168144cd0 2024-03-06T07:33:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.005+0000 7fd1701bc640 1 -- 172.21.15.1:0/824646468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1681eebf0 con 0x7fd168144cd0 2024-03-06T07:33:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.005+0000 7fd1701bc640 1 -- 172.21.15.1:0/824646468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1681e25d0 con 0x7fd168144cd0 2024-03-06T07:33:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.006+0000 7fd1701bc640 1 -- 172.21.15.1:0/824646468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd168067fd0 con 0x7fd168144cd0 2024-03-06T07:33:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.006+0000 7fd1667fc640 1 -- 172.21.15.1:0/824646468 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1580149c0 con 0x7fd168144cd0 2024-03-06T07:33:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.007+0000 7fd1667fc640 1 --2- 172.21.15.1:0/824646468 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd15406b4d0 0x7fd15406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.007+0000 7fd1667fc640 1 -- 172.21.15.1:0/824646468 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd15808ab60 con 0x7fd168144cd0 2024-03-06T07:33:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.007+0000 7fd16df58640 1 --2- 172.21.15.1:0/824646468 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd15406b4d0 0x7fd15406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.007+0000 7fd16df58640 1 --2- 172.21.15.1:0/824646468 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd15406b4d0 0x7fd15406d990 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fd1680c5b90 tx=0x7fd160008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.011+0000 7fd1667fc640 1 -- 172.21.15.1:0/824646468 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1580597e0 con 0x7fd168144cd0 2024-03-06T07:33:20.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.192+0000 7fd1701bc640 1 -- 172.21.15.1:0/824646468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd1680b06f0 con 0x7fd168144cd0 2024-03-06T07:33:20.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.193+0000 7fd1667fc640 1 -- 172.21.15.1:0/824646468 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v214) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd1580590e0 con 0x7fd168144cd0 2024-03-06T07:33:20.195 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-06T07:33:20.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.195+0000 7fd1701bc640 1 -- 172.21.15.1:0/824646468 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd15406b4d0 msgr2=0x7fd15406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.195+0000 7fd1701bc640 1 --2- 172.21.15.1:0/824646468 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd15406b4d0 0x7fd15406d990 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fd1680c5b90 tx=0x7fd160008040 comp rx=0 tx=0).stop 2024-03-06T07:33:20.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.195+0000 7fd1701bc640 1 -- 172.21.15.1:0/824646468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168144cd0 msgr2=0x7fd16814c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.195+0000 7fd1701bc640 1 --2- 172.21.15.1:0/824646468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168144cd0 0x7fd16814c910 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7fd158007d20 tx=0x7fd15800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.195+0000 7fd1701bc640 1 -- 172.21.15.1:0/824646468 shutdown_connections 2024-03-06T07:33:20.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.195+0000 7fd1701bc640 1 --2- 172.21.15.1:0/824646468 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fd15406b4d0 0x7fd15406d990 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.195+0000 7fd1701bc640 1 --2- 172.21.15.1:0/824646468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168144cd0 0x7fd16814c910 unknown :-1 s=CLOSED pgs=2681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.195+0000 7fd1701bc640 1 --2- 172.21.15.1:0/824646468 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd168152650 0x7fd16814c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.195+0000 7fd1701bc640 1 --2- 172.21.15.1:0/824646468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1680bf9d0 0x7fd1681e1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.195+0000 7fd1701bc640 1 -- 172.21.15.1:0/824646468 >> 172.21.15.1:0/824646468 conn(0x7fd168066ee0 msgr2=0x7fd1680c4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.195+0000 7fd1701bc640 1 -- 172.21.15.1:0/824646468 shutdown_connections 2024-03-06T07:33:20.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.195+0000 7fd1701bc640 1 -- 172.21.15.1:0/824646468 wait complete. 2024-03-06T07:33:20.208 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:33:20.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.314+0000 7f1b5d82a640 1 Processor -- start 2024-03-06T07:33:20.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.314+0000 7f1b5d82a640 1 -- start start 2024-03-06T07:33:20.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.315+0000 7f1b5d82a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1b580c1390 0x7f1b580c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.315+0000 7f1b5d82a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1b581552e0 0x7f1b58155700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.315+0000 7f1b5d82a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b581588e0 con 0x7f1b58152730 2024-03-06T07:33:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.315+0000 7f1b5d82a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b580c1d60 con 0x7f1b581552e0 2024-03-06T07:33:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.315+0000 7f1b5d82a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b58158bc0 con 0x7f1b580c1390 2024-03-06T07:33:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.315+0000 7f1b577fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1b581552e0 0x7f1b58155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.315+0000 7f1b577fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1b581552e0 0x7f1b58155700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41712/0 (socket says 172.21.15.1:41712) 2024-03-06T07:33:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.315+0000 7f1b577fe640 1 -- 172.21.15.1:0/3925925436 learned_addr learned my addr 172.21.15.1:0/3925925436 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.315+0000 7f1b4ffff640 1 --2- 172.21.15.1:0/3925925436 >> v2:172.21.15.98:3300/0 conn(0x7f1b580c1390 0x7f1b580c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.315+0000 7f1b577fe640 1 -- 172.21.15.1:0/3925925436 >> v2:172.21.15.98:3300/0 conn(0x7f1b580c1390 msgr2=0x7f1b580c1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.315+0000 7f1b577fe640 1 --2- 172.21.15.1:0/3925925436 >> v2:172.21.15.98:3300/0 conn(0x7f1b580c1390 0x7f1b580c1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.315+0000 7f1b577fe640 1 -- 172.21.15.1:0/3925925436 >> v1:172.21.15.1:6789/0 conn(0x7f1b58152730 legacy=0x7f1b580c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.315+0000 7f1b577fe640 1 -- 172.21.15.1:0/3925925436 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b58158ea0 con 0x7f1b581552e0 2024-03-06T07:33:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.315+0000 7f1b567fc640 1 -- 172.21.15.1:0/3925925436 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156726542 0 0) 0x7f1b581588e0 con 0x7f1b58152730 2024-03-06T07:33:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.315+0000 7f1b4ffff640 1 --2- 172.21.15.1:0/3925925436 >> v2:172.21.15.98:3300/0 conn(0x7f1b580c1390 0x7f1b580c1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.316+0000 7f1b577fe640 1 --2- 172.21.15.1:0/3925925436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b581552e0 0x7f1b58155700 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f1b5808ca60 tx=0x7f1b5002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e690178861f0e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.316+0000 7f1b567fc640 1 -- 172.21.15.1:0/3925925436 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5002c9f0 con 0x7f1b581552e0 2024-03-06T07:33:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.316+0000 7f1b567fc640 1 -- 172.21.15.1:0/3925925436 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5002cb90 con 0x7f1b581552e0 2024-03-06T07:33:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.316+0000 7f1b567fc640 1 -- 172.21.15.1:0/3925925436 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b500343f0 con 0x7f1b581552e0 2024-03-06T07:33:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.317+0000 7f1b5d82a640 1 -- 172.21.15.1:0/3925925436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b581552e0 msgr2=0x7f1b58155700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.317+0000 7f1b5d82a640 1 --2- 172.21.15.1:0/3925925436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b581552e0 0x7f1b58155700 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f1b5808ca60 tx=0x7f1b5002be50 comp rx=0 tx=0).stop 2024-03-06T07:33:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.317+0000 7f1b5d82a640 1 -- 172.21.15.1:0/3925925436 shutdown_connections 2024-03-06T07:33:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.317+0000 7f1b5d82a640 1 --2- 172.21.15.1:0/3925925436 >> v2:172.21.15.98:3300/0 conn(0x7f1b580c1390 0x7f1b580c1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.317+0000 7f1b5d82a640 1 --2- 172.21.15.1:0/3925925436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b581552e0 0x7f1b58155700 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.317+0000 7f1b5d82a640 1 -- 172.21.15.1:0/3925925436 >> 172.21.15.1:0/3925925436 conn(0x7f1b58066d00 msgr2=0x7f1b58067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.317+0000 7f1b5d82a640 1 -- 172.21.15.1:0/3925925436 shutdown_connections 2024-03-06T07:33:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.317+0000 7f1b5d82a640 1 -- 172.21.15.1:0/3925925436 wait complete. 2024-03-06T07:33:20.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.318+0000 7f1b5d82a640 1 Processor -- start 2024-03-06T07:33:20.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.318+0000 7f1b5d82a640 1 -- start start 2024-03-06T07:33:20.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.318+0000 7f1b5d82a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b580c1390 0x7f1b580afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b5d82a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b58152730 0x7f1b581ee380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b5d82a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b581552e0 0x7f1b5814c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b5d82a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b58152b10 con 0x7f1b581552e0 2024-03-06T07:33:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b5d82a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b58159460 con 0x7f1b580c1390 2024-03-06T07:33:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b5d82a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b580c71a0 con 0x7f1b58152730 2024-03-06T07:33:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b577fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b581552e0 0x7f1b5814c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b577fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b581552e0 0x7f1b5814c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41716/0 (socket says 172.21.15.1:41716) 2024-03-06T07:33:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b577fe640 1 -- 172.21.15.1:0/3587576005 learned_addr learned my addr 172.21.15.1:0/3587576005 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b4ffff640 1 --2- 172.21.15.1:0/3587576005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b58152730 0x7f1b581ee380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b56ffd640 1 --2- 172.21.15.1:0/3587576005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b580c1390 0x7f1b580afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b577fe640 1 -- 172.21.15.1:0/3587576005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b58152730 msgr2=0x7f1b581ee380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b577fe640 1 --2- 172.21.15.1:0/3587576005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b58152730 0x7f1b581ee380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b577fe640 1 -- 172.21.15.1:0/3587576005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b580c1390 msgr2=0x7f1b580afbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b577fe640 1 --2- 172.21.15.1:0/3587576005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b580c1390 0x7f1b580afbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b577fe640 1 -- 172.21.15.1:0/3587576005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b580abf90 con 0x7f1b581552e0 2024-03-06T07:33:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b4ffff640 1 --2- 172.21.15.1:0/3587576005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b58152730 0x7f1b581ee380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b56ffd640 1 --2- 172.21.15.1:0/3587576005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b580c1390 0x7f1b580afbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b577fe640 1 --2- 172.21.15.1:0/3587576005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b581552e0 0x7f1b5814c9b0 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f1b50009f40 tx=0x7f1b50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b54ff9640 1 -- 172.21.15.1:0/3587576005 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b50044020 con 0x7f1b581552e0 2024-03-06T07:33:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b54ff9640 1 -- 172.21.15.1:0/3587576005 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b50040470 con 0x7f1b581552e0 2024-03-06T07:33:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b5d82a640 1 -- 172.21.15.1:0/3587576005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b580cae00 con 0x7f1b581552e0 2024-03-06T07:33:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b54ff9640 1 -- 172.21.15.1:0/3587576005 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b50040860 con 0x7f1b581552e0 2024-03-06T07:33:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.319+0000 7f1b5d82a640 1 -- 172.21.15.1:0/3587576005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b581eebc0 con 0x7f1b581552e0 2024-03-06T07:33:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.320+0000 7f1b5d82a640 1 -- 172.21.15.1:0/3587576005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b580b0220 con 0x7f1b581552e0 2024-03-06T07:33:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.321+0000 7f1b54ff9640 1 -- 172.21.15.1:0/3587576005 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b50047020 con 0x7f1b581552e0 2024-03-06T07:33:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.321+0000 7f1b54ff9640 1 --2- 172.21.15.1:0/3587576005 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1b3806b500 0x7f1b3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.321+0000 7f1b54ff9640 1 -- 172.21.15.1:0/3587576005 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f1b500acde0 con 0x7f1b581552e0 2024-03-06T07:33:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.322+0000 7f1b56ffd640 1 --2- 172.21.15.1:0/3587576005 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1b3806b500 0x7f1b3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.322+0000 7f1b56ffd640 1 --2- 172.21.15.1:0/3587576005 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1b3806b500 0x7f1b3806d9c0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f1b48000f60 tx=0x7f1b48009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.330+0000 7f1b54ff9640 1 -- 172.21.15.1:0/3587576005 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b5004e030 con 0x7f1b581552e0 2024-03-06T07:33:20.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.508+0000 7f1b5d82a640 1 -- 172.21.15.1:0/3587576005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1b580c6870 con 0x7f1b581552e0 2024-03-06T07:33:20.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.509+0000 7f1b54ff9640 1 -- 172.21.15.1:0/3587576005 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v214) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1b5007b9c0 con 0x7f1b581552e0 2024-03-06T07:33:20.511 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-06T07:33:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.510+0000 7f1b5d82a640 1 -- 172.21.15.1:0/3587576005 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1b3806b500 msgr2=0x7f1b3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.510+0000 7f1b5d82a640 1 --2- 172.21.15.1:0/3587576005 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1b3806b500 0x7f1b3806d9c0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f1b48000f60 tx=0x7f1b48009040 comp rx=0 tx=0).stop 2024-03-06T07:33:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.510+0000 7f1b5d82a640 1 -- 172.21.15.1:0/3587576005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b581552e0 msgr2=0x7f1b5814c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.510+0000 7f1b5d82a640 1 --2- 172.21.15.1:0/3587576005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b581552e0 0x7f1b5814c9b0 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f1b50009f40 tx=0x7f1b50004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.510+0000 7f1b5d82a640 1 -- 172.21.15.1:0/3587576005 shutdown_connections 2024-03-06T07:33:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.510+0000 7f1b5d82a640 1 --2- 172.21.15.1:0/3587576005 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f1b3806b500 0x7f1b3806d9c0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.510+0000 7f1b5d82a640 1 --2- 172.21.15.1:0/3587576005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b581552e0 0x7f1b5814c9b0 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.510+0000 7f1b5d82a640 1 --2- 172.21.15.1:0/3587576005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1b58152730 0x7f1b581ee380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.510+0000 7f1b5d82a640 1 --2- 172.21.15.1:0/3587576005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b580c1390 0x7f1b580afbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.510+0000 7f1b5d82a640 1 -- 172.21.15.1:0/3587576005 >> 172.21.15.1:0/3587576005 conn(0x7f1b58066d00 msgr2=0x7f1b580c0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.511+0000 7f1b5d82a640 1 -- 172.21.15.1:0/3587576005 shutdown_connections 2024-03-06T07:33:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.511+0000 7f1b5d82a640 1 -- 172.21.15.1:0/3587576005 wait complete. 2024-03-06T07:33:20.524 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:20.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.623+0000 7fe0841bc640 1 Processor -- start 2024-03-06T07:33:20.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.623+0000 7fe0841bc640 1 -- start start 2024-03-06T07:33:20.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.624+0000 7fe0841bc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe07c0c4560 0x7fe07c155c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.624+0000 7fe0841bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe07c0c39c0 0x7fe07c1561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.624+0000 7fe0841bc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe07c0c4940 0x7fe07c0ac220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.624+0000 7fe0841bc640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe07c1566f0 con 0x7fe07c0c4940 2024-03-06T07:33:20.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.624+0000 7fe0841bc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe07c0c7420 con 0x7fe07c0c4560 2024-03-06T07:33:20.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.624+0000 7fe0841bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c0ab760 con 0x7fe07c0c39c0 2024-03-06T07:33:20.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.624+0000 7fe082759640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe07c0c4940 0x7fe07c0ac220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.624+0000 7fe082759640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe07c0c4940 0x7fe07c0ac220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41724/0 (socket says 172.21.15.1:41724) 2024-03-06T07:33:20.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.624+0000 7fe082759640 1 -- 172.21.15.1:0/418451566 learned_addr learned my addr 172.21.15.1:0/418451566 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:20.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.624+0000 7fe081f58640 1 --2- 172.21.15.1:0/418451566 >> v2:172.21.15.98:3300/0 conn(0x7fe07c0c4560 0x7fe07c155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.624+0000 7fe081757640 1 --2- 172.21.15.1:0/418451566 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe07c0c39c0 0x7fe07c1561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.624+0000 7fe082759640 1 -- 172.21.15.1:0/418451566 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe07c0c39c0 msgr2=0x7fe07c1561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:20.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.624+0000 7fe082759640 1 --2- 172.21.15.1:0/418451566 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe07c0c39c0 0x7fe07c1561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.624+0000 7fe082759640 1 -- 172.21.15.1:0/418451566 >> v2:172.21.15.98:3300/0 conn(0x7fe07c0c4560 msgr2=0x7fe07c155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:20.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.624+0000 7fe082759640 1 --2- 172.21.15.1:0/418451566 >> v2:172.21.15.98:3300/0 conn(0x7fe07c0c4560 0x7fe07c155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.624+0000 7fe082759640 1 -- 172.21.15.1:0/418451566 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c0c76c0 con 0x7fe07c0c4940 2024-03-06T07:33:20.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.624+0000 7fe081757640 1 --2- 172.21.15.1:0/418451566 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe07c0c39c0 0x7fe07c1561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:20.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.625+0000 7fe082759640 1 --2- 172.21.15.1:0/418451566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe07c0c4940 0x7fe07c0ac220 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7fe07c0c5ce0 tx=0x7fe07402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97607fa12336f001 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.626+0000 7fe080f56640 1 -- 172.21.15.1:0/418451566 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07403b070 con 0x7fe07c0c4940 2024-03-06T07:33:20.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.626+0000 7fe080f56640 1 -- 172.21.15.1:0/418451566 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe074036440 con 0x7fe07c0c4940 2024-03-06T07:33:20.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.626+0000 7fe080f56640 1 -- 172.21.15.1:0/418451566 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe074036810 con 0x7fe07c0c4940 2024-03-06T07:33:20.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.626+0000 7fe0841bc640 1 -- 172.21.15.1:0/418451566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe07c0c4940 msgr2=0x7fe07c0ac220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.626+0000 7fe0841bc640 1 --2- 172.21.15.1:0/418451566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe07c0c4940 0x7fe07c0ac220 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7fe07c0c5ce0 tx=0x7fe07402e0e0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.626+0000 7fe0841bc640 1 -- 172.21.15.1:0/418451566 shutdown_connections 2024-03-06T07:33:20.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.626+0000 7fe0841bc640 1 --2- 172.21.15.1:0/418451566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe07c0c4940 0x7fe07c0ac220 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.626+0000 7fe0841bc640 1 --2- 172.21.15.1:0/418451566 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe07c0c39c0 0x7fe07c1561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.626+0000 7fe0841bc640 1 --2- 172.21.15.1:0/418451566 >> v2:172.21.15.98:3300/0 conn(0x7fe07c0c4560 0x7fe07c155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.626+0000 7fe0841bc640 1 -- 172.21.15.1:0/418451566 >> 172.21.15.1:0/418451566 conn(0x7fe07c0c7a50 msgr2=0x7fe07c0c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:20.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.626+0000 7fe0841bc640 1 -- 172.21.15.1:0/418451566 shutdown_connections 2024-03-06T07:33:20.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.626+0000 7fe0841bc640 1 -- 172.21.15.1:0/418451566 wait complete. 2024-03-06T07:33:20.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.627+0000 7fe0841bc640 1 Processor -- start 2024-03-06T07:33:20.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.627+0000 7fe0841bc640 1 -- start start 2024-03-06T07:33:20.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.628+0000 7fe0841bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe07c0c39c0 0x7fe07c151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.628+0000 7fe0841bc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe07c0c4560 0x7fe07c1ea3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.628+0000 7fe0841bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe07c0c4940 0x7fe07c0bee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.628+0000 7fe0841bc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c06e6d0 con 0x7fe07c0c4560 2024-03-06T07:33:20.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.628+0000 7fe0841bc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c0abd00 con 0x7fe07c0c39c0 2024-03-06T07:33:20.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.628+0000 7fe0841bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c06e850 con 0x7fe07c0c4940 2024-03-06T07:33:20.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.628+0000 7fe081757640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe07c0c4560 0x7fe07c1ea3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.628+0000 7fe081757640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe07c0c4560 0x7fe07c1ea3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41726/0 (socket says 172.21.15.1:41726) 2024-03-06T07:33:20.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.628+0000 7fe081757640 1 -- 172.21.15.1:0/499787058 learned_addr learned my addr 172.21.15.1:0/499787058 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:20.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.628+0000 7fe082759640 1 --2- 172.21.15.1:0/499787058 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe07c0c4940 0x7fe07c0bee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.628+0000 7fe081f58640 1 --2- 172.21.15.1:0/499787058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe07c0c39c0 0x7fe07c151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.628+0000 7fe081757640 1 -- 172.21.15.1:0/499787058 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe07c0c4940 msgr2=0x7fe07c0bee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.628+0000 7fe081757640 1 --2- 172.21.15.1:0/499787058 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe07c0c4940 0x7fe07c0bee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.628+0000 7fe081757640 1 -- 172.21.15.1:0/499787058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe07c0c39c0 msgr2=0x7fe07c151c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.628+0000 7fe081757640 1 --2- 172.21.15.1:0/499787058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe07c0c39c0 0x7fe07c151c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.629+0000 7fe081757640 1 -- 172.21.15.1:0/499787058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c1e7480 con 0x7fe07c0c4560 2024-03-06T07:33:20.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.629+0000 7fe081f58640 1 --2- 172.21.15.1:0/499787058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe07c0c39c0 0x7fe07c151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:20.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.629+0000 7fe081757640 1 --2- 172.21.15.1:0/499787058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe07c0c4560 0x7fe07c1ea3f0 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7fe07000e7e0 tx=0x7fe07000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.629+0000 7fe07a7fc640 1 -- 172.21.15.1:0/499787058 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07000f040 con 0x7fe07c0c4560 2024-03-06T07:33:20.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.629+0000 7fe07a7fc640 1 -- 172.21.15.1:0/499787058 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe070004030 con 0x7fe07c0c4560 2024-03-06T07:33:20.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.629+0000 7fe0841bc640 1 -- 172.21.15.1:0/499787058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07c0bcc10 con 0x7fe07c0c4560 2024-03-06T07:33:20.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.629+0000 7fe07a7fc640 1 -- 172.21.15.1:0/499787058 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0700043e0 con 0x7fe07c0c4560 2024-03-06T07:33:20.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.629+0000 7fe0841bc640 1 -- 172.21.15.1:0/499787058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07c152190 con 0x7fe07c0c4560 2024-03-06T07:33:20.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.630+0000 7fe0841bc640 1 -- 172.21.15.1:0/499787058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe07c0686e0 con 0x7fe07c0c4560 2024-03-06T07:33:20.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.630+0000 7fe07a7fc640 1 -- 172.21.15.1:0/499787058 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe07000c6d0 con 0x7fe07c0c4560 2024-03-06T07:33:20.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.631+0000 7fe07a7fc640 1 --2- 172.21.15.1:0/499787058 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe06806b4d0 0x7fe06806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.631+0000 7fe07a7fc640 1 -- 172.21.15.1:0/499787058 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fe07008a450 con 0x7fe07c0c4560 2024-03-06T07:33:20.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.631+0000 7fe081f58640 1 --2- 172.21.15.1:0/499787058 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe06806b4d0 0x7fe06806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.631+0000 7fe081f58640 1 --2- 172.21.15.1:0/499787058 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe06806b4d0 0x7fe06806d990 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fe06c007520 tx=0x7fe06c00b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.639+0000 7fe07a7fc640 1 -- 172.21.15.1:0/499787058 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe070017020 con 0x7fe07c0c4560 2024-03-06T07:33:20.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.859+0000 7fe0841bc640 1 -- 172.21.15.1:0/499787058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe07c0c2230 con 0x7fe07c0c4560 2024-03-06T07:33:20.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.860+0000 7fe07a7fc640 1 -- 172.21.15.1:0/499787058 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7fe0700590d0 con 0x7fe07c0c4560 2024-03-06T07:33:20.862 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:20.862 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:33:20.862 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:33:20.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.862+0000 7fe0841bc640 1 -- 172.21.15.1:0/499787058 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe06806b4d0 msgr2=0x7fe06806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.862+0000 7fe0841bc640 1 --2- 172.21.15.1:0/499787058 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe06806b4d0 0x7fe06806d990 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fe06c007520 tx=0x7fe06c00b040 comp rx=0 tx=0).stop 2024-03-06T07:33:20.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.862+0000 7fe0841bc640 1 -- 172.21.15.1:0/499787058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe07c0c4560 msgr2=0x7fe07c1ea3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.862+0000 7fe0841bc640 1 --2- 172.21.15.1:0/499787058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe07c0c4560 0x7fe07c1ea3f0 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7fe07000e7e0 tx=0x7fe07000eca0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.862+0000 7fe0841bc640 1 -- 172.21.15.1:0/499787058 shutdown_connections 2024-03-06T07:33:20.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.862+0000 7fe0841bc640 1 --2- 172.21.15.1:0/499787058 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fe06806b4d0 0x7fe06806d990 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.863+0000 7fe0841bc640 1 --2- 172.21.15.1:0/499787058 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe07c0c4940 0x7fe07c0bee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.863+0000 7fe0841bc640 1 --2- 172.21.15.1:0/499787058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe07c0c4560 0x7fe07c1ea3f0 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.863+0000 7fe0841bc640 1 --2- 172.21.15.1:0/499787058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe07c0c39c0 0x7fe07c151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.863+0000 7fe0841bc640 1 -- 172.21.15.1:0/499787058 >> 172.21.15.1:0/499787058 conn(0x7fe07c0c7a50 msgr2=0x7fe07c154ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:20.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.863+0000 7fe0841bc640 1 -- 172.21.15.1:0/499787058 shutdown_connections 2024-03-06T07:33:20.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.863+0000 7fe0841bc640 1 -- 172.21.15.1:0/499787058 wait complete. 2024-03-06T07:33:20.879 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:20.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.978+0000 7f5118b5e640 1 Processor -- start 2024-03-06T07:33:20.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.978+0000 7f5118b5e640 1 -- start start 2024-03-06T07:33:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f5118b5e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5114152650 0x7f5114155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f5118b5e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f51140c6b10 0x7f5114158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f5118b5e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51140bf9d0 0x7f51140c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f5118b5e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5114155800 con 0x7f51140bf9d0 2024-03-06T07:33:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f5118b5e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51140c2fc0 con 0x7f51140c6b10 2024-03-06T07:33:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f5118b5e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51140c6690 con 0x7f5114152650 2024-03-06T07:33:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f5112d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51140bf9d0 0x7f51140c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f5112d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51140bf9d0 0x7f51140c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41730/0 (socket says 172.21.15.1:41730) 2024-03-06T07:33:20.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f5112d9d640 1 -- 172.21.15.1:0/992209738 learned_addr learned my addr 172.21.15.1:0/992209738 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:20.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f511259c640 1 --2- 172.21.15.1:0/992209738 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5114152650 0x7f5114155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f5111d9b640 1 --2- 172.21.15.1:0/992209738 >> v2:172.21.15.98:3300/0 conn(0x7f51140c6b10 0x7f5114158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f5112d9d640 1 -- 172.21.15.1:0/992209738 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5114152650 msgr2=0x7f5114155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:20.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f5112d9d640 1 --2- 172.21.15.1:0/992209738 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5114152650 0x7f5114155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f5112d9d640 1 -- 172.21.15.1:0/992209738 >> v2:172.21.15.98:3300/0 conn(0x7f51140c6b10 msgr2=0x7f5114158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:20.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f5112d9d640 1 --2- 172.21.15.1:0/992209738 >> v2:172.21.15.98:3300/0 conn(0x7f51140c6b10 0x7f5114158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f5112d9d640 1 -- 172.21.15.1:0/992209738 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51140c7280 con 0x7f51140bf9d0 2024-03-06T07:33:20.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f511259c640 1 --2- 172.21.15.1:0/992209738 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5114152650 0x7f5114155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:20.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.979+0000 7f5111d9b640 1 --2- 172.21.15.1:0/992209738 >> v2:172.21.15.98:3300/0 conn(0x7f51140c6b10 0x7f5114158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:20.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.980+0000 7f5112d9d640 1 --2- 172.21.15.1:0/992209738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51140bf9d0 0x7f51140c1460 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7f511408c500 tx=0x7f510402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e20827db82e4e1b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.980+0000 7f511159a640 1 -- 172.21.15.1:0/992209738 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510402c9e0 con 0x7f51140bf9d0 2024-03-06T07:33:20.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.980+0000 7f511159a640 1 -- 172.21.15.1:0/992209738 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510402cb80 con 0x7f51140bf9d0 2024-03-06T07:33:20.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.980+0000 7f511159a640 1 -- 172.21.15.1:0/992209738 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51040343f0 con 0x7f51140bf9d0 2024-03-06T07:33:20.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.981+0000 7f5118b5e640 1 -- 172.21.15.1:0/992209738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51140bf9d0 msgr2=0x7f51140c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.981+0000 7f5118b5e640 1 --2- 172.21.15.1:0/992209738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51140bf9d0 0x7f51140c1460 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7f511408c500 tx=0x7f510402be40 comp rx=0 tx=0).stop 2024-03-06T07:33:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.981+0000 7f5118b5e640 1 -- 172.21.15.1:0/992209738 shutdown_connections 2024-03-06T07:33:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.981+0000 7f5118b5e640 1 --2- 172.21.15.1:0/992209738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51140bf9d0 0x7f51140c1460 unknown :-1 s=CLOSED pgs=2687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.981+0000 7f5118b5e640 1 --2- 172.21.15.1:0/992209738 >> v2:172.21.15.98:3300/0 conn(0x7f51140c6b10 0x7f5114158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.981+0000 7f5118b5e640 1 --2- 172.21.15.1:0/992209738 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5114152650 0x7f5114155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.981+0000 7f5118b5e640 1 -- 172.21.15.1:0/992209738 >> 172.21.15.1:0/992209738 conn(0x7f5114066ee0 msgr2=0x7f51140672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.981+0000 7f5118b5e640 1 -- 172.21.15.1:0/992209738 shutdown_connections 2024-03-06T07:33:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.981+0000 7f5118b5e640 1 -- 172.21.15.1:0/992209738 wait complete. 2024-03-06T07:33:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.982+0000 7f5118b5e640 1 Processor -- start 2024-03-06T07:33:20.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.982+0000 7f5118b5e640 1 -- start start 2024-03-06T07:33:20.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.983+0000 7f5118b5e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51140bf9d0 0x7f51140ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.983+0000 7f5118b5e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5114152650 0x7f51140c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.983+0000 7f5118b5e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511414e5f0 0x7f51141e0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.983+0000 7f5118b5e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5114155490 con 0x7f511414e5f0 2024-03-06T07:33:20.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.983+0000 7f5118b5e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51141551b0 con 0x7f51140bf9d0 2024-03-06T07:33:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.983+0000 7f5118b5e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5114152a30 con 0x7f5114152650 2024-03-06T07:33:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.983+0000 7f5112d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511414e5f0 0x7f51141e0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.983+0000 7f5112d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511414e5f0 0x7f51141e0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41742/0 (socket says 172.21.15.1:41742) 2024-03-06T07:33:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.983+0000 7f5112d9d640 1 -- 172.21.15.1:0/120696444 learned_addr learned my addr 172.21.15.1:0/120696444 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.984+0000 7f5112d9d640 1 -- 172.21.15.1:0/120696444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5114152650 msgr2=0x7f51140c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.984+0000 7f5112d9d640 1 --2- 172.21.15.1:0/120696444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5114152650 0x7f51140c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.984+0000 7f5112d9d640 1 -- 172.21.15.1:0/120696444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51140bf9d0 msgr2=0x7f51140ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.984+0000 7f5112d9d640 1 --2- 172.21.15.1:0/120696444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51140bf9d0 0x7f51140ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.984+0000 7f5112d9d640 1 -- 172.21.15.1:0/120696444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511414c400 con 0x7f511414e5f0 2024-03-06T07:33:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.984+0000 7f5112d9d640 1 --2- 172.21.15.1:0/120696444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511414e5f0 0x7f51141e0e50 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7f5104009f40 tx=0x7f5104004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.984+0000 7f5102ffd640 1 -- 172.21.15.1:0/120696444 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104043020 con 0x7f511414e5f0 2024-03-06T07:33:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.984+0000 7f5102ffd640 1 -- 172.21.15.1:0/120696444 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510403e450 con 0x7f511414e5f0 2024-03-06T07:33:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.984+0000 7f5118b5e640 1 -- 172.21.15.1:0/120696444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51141e02c0 con 0x7f511414e5f0 2024-03-06T07:33:20.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.984+0000 7f5102ffd640 1 -- 172.21.15.1:0/120696444 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104009510 con 0x7f511414e5f0 2024-03-06T07:33:20.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.984+0000 7f5118b5e640 1 -- 172.21.15.1:0/120696444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f511414c6e0 con 0x7f511414e5f0 2024-03-06T07:33:20.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.985+0000 7f5118b5e640 1 -- 172.21.15.1:0/120696444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5114067fd0 con 0x7f511414e5f0 2024-03-06T07:33:20.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.985+0000 7f5102ffd640 1 -- 172.21.15.1:0/120696444 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5104046020 con 0x7f511414e5f0 2024-03-06T07:33:20.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.986+0000 7f5102ffd640 1 --2- 172.21.15.1:0/120696444 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50f406b4d0 0x7f50f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.986+0000 7f5102ffd640 1 -- 172.21.15.1:0/120696444 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f51040acf80 con 0x7f511414e5f0 2024-03-06T07:33:20.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.986+0000 7f511259c640 1 --2- 172.21.15.1:0/120696444 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50f406b4d0 0x7f50f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.987+0000 7f511259c640 1 --2- 172.21.15.1:0/120696444 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50f406b4d0 0x7f50f406d990 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f510c006fd0 tx=0x7f510c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:20.990+0000 7f5102ffd640 1 -- 172.21.15.1:0/120696444 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f510404d020 con 0x7f511414e5f0 2024-03-06T07:33:21.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.176+0000 7f5118b5e640 1 -- 172.21.15.1:0/120696444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51140b06f0 con 0x7f511414e5f0 2024-03-06T07:33:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.177+0000 7f5102ffd640 1 -- 172.21.15.1:0/120696444 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v214) v1 ==== 74+0+17167 (secure 0 0 0) 0x7f510407bc00 con 0x7f511414e5f0 2024-03-06T07:33:21.179 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:21.179 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":214,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:30:49.314643+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:31.114390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:31.815153+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"210","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"34.26","mappings":[{"from":0,"to":2}]},{"pgid":"34.31","mappings":[{"from":0,"to":2}]},{"pgid":"34.3f","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6836/2083320955":"2024-03-07T07:30:49.314605+0000","172.21.15.98:6834/2083320955":"2024-03-07T07:30:49.314605+0000","172.21.15.1:6834/2424540959":"2024-03-07T07:30:14.310288+0000","172.21.15.1:6832/2424540959":"2024-03-07T07:30:14.310288+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:21.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.179+0000 7f5118b5e640 1 -- 172.21.15.1:0/120696444 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50f406b4d0 msgr2=0x7f50f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.179+0000 7f5118b5e640 1 --2- 172.21.15.1:0/120696444 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50f406b4d0 0x7f50f406d990 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f510c006fd0 tx=0x7f510c008040 comp rx=0 tx=0).stop 2024-03-06T07:33:21.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.179+0000 7f5118b5e640 1 -- 172.21.15.1:0/120696444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511414e5f0 msgr2=0x7f51141e0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.179+0000 7f5118b5e640 1 --2- 172.21.15.1:0/120696444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511414e5f0 0x7f51141e0e50 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7f5104009f40 tx=0x7f5104004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:21.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.179+0000 7f5118b5e640 1 -- 172.21.15.1:0/120696444 shutdown_connections 2024-03-06T07:33:21.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.179+0000 7f5118b5e640 1 --2- 172.21.15.1:0/120696444 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f50f406b4d0 0x7f50f406d990 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.179+0000 7f5118b5e640 1 --2- 172.21.15.1:0/120696444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511414e5f0 0x7f51141e0e50 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.179+0000 7f5118b5e640 1 --2- 172.21.15.1:0/120696444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5114152650 0x7f51140c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.179+0000 7f5118b5e640 1 --2- 172.21.15.1:0/120696444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51140bf9d0 0x7f51140ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.179+0000 7f5118b5e640 1 -- 172.21.15.1:0/120696444 >> 172.21.15.1:0/120696444 conn(0x7f5114066ee0 msgr2=0x7f51140672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.180+0000 7f5118b5e640 1 -- 172.21.15.1:0/120696444 shutdown_connections 2024-03-06T07:33:21.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.180+0000 7f5118b5e640 1 -- 172.21.15.1:0/120696444 wait complete. 2024-03-06T07:33:21.193 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:33:21.193 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:21.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.291+0000 7f11d3f1e640 1 Processor -- start 2024-03-06T07:33:21.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.291+0000 7f11d3f1e640 1 -- start start 2024-03-06T07:33:21.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.292+0000 7f11d3f1e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f11cc0c6b10 0x7f11cc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.292+0000 7f11d3f1e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11cc0bf9d0 0x7f11cc0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.292+0000 7f11d3f1e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11cc0c2fc0 con 0x7f11cc152650 2024-03-06T07:33:21.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.292+0000 7f11d3f1e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11cc155800 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.292+0000 7f11d3f1e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11cc158da0 con 0x7f11cc0c6b10 2024-03-06T07:33:21.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.292+0000 7f11d24bb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11cc0bf9d0 0x7f11cc0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.292+0000 7f11d24bb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11cc0bf9d0 0x7f11cc0c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41752/0 (socket says 172.21.15.1:41752) 2024-03-06T07:33:21.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.292+0000 7f11d24bb640 1 -- 172.21.15.1:0/1791403212 learned_addr learned my addr 172.21.15.1:0/1791403212 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:21.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.293+0000 7f11d14b9640 1 --2- 172.21.15.1:0/1791403212 >> v2:172.21.15.98:3300/0 conn(0x7f11cc0c6b10 0x7f11cc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.293+0000 7f11d24bb640 1 -- 172.21.15.1:0/1791403212 >> v2:172.21.15.98:3300/0 conn(0x7f11cc0c6b10 msgr2=0x7f11cc155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:21.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.293+0000 7f11d24bb640 1 --2- 172.21.15.1:0/1791403212 >> v2:172.21.15.98:3300/0 conn(0x7f11cc0c6b10 0x7f11cc155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.293+0000 7f11d24bb640 1 -- 172.21.15.1:0/1791403212 >> v1:172.21.15.98:6789/0 conn(0x7f11cc152650 legacy=0x7f11cc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:21.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.293+0000 7f11d24bb640 1 -- 172.21.15.1:0/1791403212 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11cc0c6690 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.294+0000 7f11d24bb640 1 --2- 172.21.15.1:0/1791403212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11cc0bf9d0 0x7f11cc0c1460 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7f11cc0cb8e0 tx=0x7f11c402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cefd12eed5db2ae5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.294+0000 7f11d0cb8640 1 -- 172.21.15.1:0/1791403212 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c402c9f0 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.294+0000 7f11d0cb8640 1 -- 172.21.15.1:0/1791403212 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c402cb90 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.294+0000 7f11d0cb8640 1 -- 172.21.15.1:0/1791403212 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c40343f0 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.294+0000 7f11d3f1e640 1 -- 172.21.15.1:0/1791403212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11cc0bf9d0 msgr2=0x7f11cc0c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.294+0000 7f11d3f1e640 1 --2- 172.21.15.1:0/1791403212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11cc0bf9d0 0x7f11cc0c1460 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7f11cc0cb8e0 tx=0x7f11c402be50 comp rx=0 tx=0).stop 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.295+0000 7f11d3f1e640 1 -- 172.21.15.1:0/1791403212 shutdown_connections 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.295+0000 7f11d3f1e640 1 --2- 172.21.15.1:0/1791403212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11cc0bf9d0 0x7f11cc0c1460 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.295+0000 7f11d3f1e640 1 --2- 172.21.15.1:0/1791403212 >> v2:172.21.15.98:3300/0 conn(0x7f11cc0c6b10 0x7f11cc155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.295+0000 7f11d3f1e640 1 -- 172.21.15.1:0/1791403212 >> 172.21.15.1:0/1791403212 conn(0x7f11cc066ee0 msgr2=0x7f11cc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.295+0000 7f11d3f1e640 1 -- 172.21.15.1:0/1791403212 shutdown_connections 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.295+0000 7f11d3f1e640 1 -- 172.21.15.1:0/1791403212 wait complete. 2024-03-06T07:33:21.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.296+0000 7f11d3f1e640 1 Processor -- start 2024-03-06T07:33:21.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.296+0000 7f11d3f1e640 1 -- start start 2024-03-06T07:33:21.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.296+0000 7f11d3f1e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11cc0bf9d0 0x7f11cc157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.296+0000 7f11d3f1e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11cc152650 0x7f11cc0c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.296+0000 7f11d3f1e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11cc1ee6f0 0x7f11cc14b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.296+0000 7f11d3f1e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc1551b0 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.296+0000 7f11d3f1e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc0c6fa0 con 0x7f11cc152650 2024-03-06T07:33:21.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.296+0000 7f11d3f1e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc155490 con 0x7f11cc1ee6f0 2024-03-06T07:33:21.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.296+0000 7f11d1cba640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11cc0bf9d0 0x7f11cc157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.296+0000 7f11d1cba640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11cc0bf9d0 0x7f11cc157860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41764/0 (socket says 172.21.15.1:41764) 2024-03-06T07:33:21.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.296+0000 7f11d1cba640 1 -- 172.21.15.1:0/3593313036 learned_addr learned my addr 172.21.15.1:0/3593313036 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:21.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.297+0000 7f11d14b9640 1 --2- 172.21.15.1:0/3593313036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11cc152650 0x7f11cc0c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.297+0000 7f11d24bb640 1 --2- 172.21.15.1:0/3593313036 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11cc1ee6f0 0x7f11cc14b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.297+0000 7f11d1cba640 1 -- 172.21.15.1:0/3593313036 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11cc1ee6f0 msgr2=0x7f11cc14b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.297+0000 7f11d1cba640 1 --2- 172.21.15.1:0/3593313036 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11cc1ee6f0 0x7f11cc14b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.297+0000 7f11d1cba640 1 -- 172.21.15.1:0/3593313036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11cc152650 msgr2=0x7f11cc0c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.297+0000 7f11d1cba640 1 --2- 172.21.15.1:0/3593313036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11cc152650 0x7f11cc0c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.297+0000 7f11d1cba640 1 -- 172.21.15.1:0/3593313036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11cc1ef030 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.297+0000 7f11d1cba640 1 --2- 172.21.15.1:0/3593313036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11cc0bf9d0 0x7f11cc157860 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f11c800c460 tx=0x7f11c800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.297+0000 7f11c27fc640 1 -- 172.21.15.1:0/3593313036 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c8011790 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.297+0000 7f11c27fc640 1 -- 172.21.15.1:0/3593313036 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c8017070 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.297+0000 7f11d3f1e640 1 -- 172.21.15.1:0/3593313036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11cc1535b0 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.297+0000 7f11c27fc640 1 -- 172.21.15.1:0/3593313036 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c8011bb0 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.297+0000 7f11d3f1e640 1 -- 172.21.15.1:0/3593313036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11cc158090 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.298+0000 7f11d3f1e640 1 -- 172.21.15.1:0/3593313036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11cc153990 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.299+0000 7f11c27fc640 1 -- 172.21.15.1:0/3593313036 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11c801c460 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.299+0000 7f11c27fc640 1 --2- 172.21.15.1:0/3593313036 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f11b006b500 0x7f11b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.299+0000 7f11c27fc640 1 -- 172.21.15.1:0/3593313036 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f11c808a200 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.299+0000 7f11d14b9640 1 --2- 172.21.15.1:0/3593313036 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f11b006b500 0x7f11b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.300+0000 7f11d14b9640 1 --2- 172.21.15.1:0/3593313036 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f11b006b500 0x7f11b006d9c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f11cc154ea0 tx=0x7f11bc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.304+0000 7f11c27fc640 1 -- 172.21.15.1:0/3593313036 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11cc153990 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.522+0000 7f11d3f1e640 1 -- 172.21.15.1:0/3593313036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11cc0c38e0 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.522+0000 7f11c27fc640 1 -- 172.21.15.1:0/3593313036 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f11cc0c38e0 con 0x7f11cc0bf9d0 2024-03-06T07:33:21.524 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:21.524 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:33:21.525 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:33:21.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.524+0000 7f11d3f1e640 1 -- 172.21.15.1:0/3593313036 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f11b006b500 msgr2=0x7f11b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.524+0000 7f11d3f1e640 1 --2- 172.21.15.1:0/3593313036 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f11b006b500 0x7f11b006d9c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f11cc154ea0 tx=0x7f11bc008040 comp rx=0 tx=0).stop 2024-03-06T07:33:21.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.524+0000 7f11d3f1e640 1 -- 172.21.15.1:0/3593313036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11cc0bf9d0 msgr2=0x7f11cc157860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.524+0000 7f11d3f1e640 1 --2- 172.21.15.1:0/3593313036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11cc0bf9d0 0x7f11cc157860 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f11c800c460 tx=0x7f11c800c920 comp rx=0 tx=0).stop 2024-03-06T07:33:21.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.524+0000 7f11d3f1e640 1 -- 172.21.15.1:0/3593313036 shutdown_connections 2024-03-06T07:33:21.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.524+0000 7f11d3f1e640 1 --2- 172.21.15.1:0/3593313036 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f11b006b500 0x7f11b006d9c0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.524+0000 7f11d3f1e640 1 --2- 172.21.15.1:0/3593313036 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f11cc1ee6f0 0x7f11cc14b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.524+0000 7f11d3f1e640 1 --2- 172.21.15.1:0/3593313036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11cc152650 0x7f11cc0c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.524+0000 7f11d3f1e640 1 --2- 172.21.15.1:0/3593313036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11cc0bf9d0 0x7f11cc157860 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.524+0000 7f11d3f1e640 1 -- 172.21.15.1:0/3593313036 >> 172.21.15.1:0/3593313036 conn(0x7f11cc066ee0 msgr2=0x7f11cc0672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.524+0000 7f11d3f1e640 1 -- 172.21.15.1:0/3593313036 shutdown_connections 2024-03-06T07:33:21.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.524+0000 7f11d3f1e640 1 -- 172.21.15.1:0/3593313036 wait complete. 2024-03-06T07:33:21.537 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:21.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.638+0000 7f0a486a4640 1 Processor -- start 2024-03-06T07:33:21.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.638+0000 7f0a486a4640 1 -- start start 2024-03-06T07:33:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.640+0000 7f0a486a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a400c6b10 0x7f0a40155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.640+0000 7f0a486a4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a400c2fc0 con 0x7f0a400bf9d0 2024-03-06T07:33:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.640+0000 7f0a486a4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a40158da0 con 0x7f0a40152650 2024-03-06T07:33:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.640+0000 7f0a486a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a40155800 con 0x7f0a400c6b10 2024-03-06T07:33:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.640+0000 7f0a46c41640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0a400bf9d0 0x7f0a400c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59974/0 (socket says 172.21.15.1:59974) 2024-03-06T07:33:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.640+0000 7f0a46c41640 1 -- 172.21.15.1:0/2932785054 learned_addr learned my addr 172.21.15.1:0/2932785054 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.640+0000 7f0a45c3f640 1 --2- 172.21.15.1:0/2932785054 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a400c6b10 0x7f0a40155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.641+0000 7f0a4543e640 1 -- 172.21.15.1:0/2932785054 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727263006 0 0) 0x7f0a400c2fc0 con 0x7f0a400bf9d0 2024-03-06T07:33:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.641+0000 7f0a4543e640 1 -- 172.21.15.1:0/2932785054 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a2c003520 con 0x7f0a400bf9d0 2024-03-06T07:33:21.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.641+0000 7f0a45c3f640 1 -- 172.21.15.1:0/2932785054 >> v1:172.21.15.98:6789/0 conn(0x7f0a40152650 legacy=0x7f0a40158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.641+0000 7f0a45c3f640 1 -- 172.21.15.1:0/2932785054 >> v1:172.21.15.1:6789/0 conn(0x7f0a400bf9d0 legacy=0x7f0a400c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.641+0000 7f0a45c3f640 1 -- 172.21.15.1:0/2932785054 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a400c6690 con 0x7f0a400c6b10 2024-03-06T07:33:21.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.642+0000 7f0a45c3f640 1 --2- 172.21.15.1:0/2932785054 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a400c6b10 0x7f0a40155c90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0a300097d0 tx=0x7f0a3002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b32a0c204e5ee7dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.644+0000 7f0a4543e640 1 -- 172.21.15.1:0/2932785054 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a30035020 con 0x7f0a400c6b10 2024-03-06T07:33:21.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.644+0000 7f0a4543e640 1 -- 172.21.15.1:0/2932785054 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a30031810 con 0x7f0a400c6b10 2024-03-06T07:33:21.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.644+0000 7f0a4543e640 1 -- 172.21.15.1:0/2932785054 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a30031be0 con 0x7f0a400c6b10 2024-03-06T07:33:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.645+0000 7f0a486a4640 1 -- 172.21.15.1:0/2932785054 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a400c6b10 msgr2=0x7f0a40155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.645+0000 7f0a486a4640 1 --2- 172.21.15.1:0/2932785054 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a400c6b10 0x7f0a40155c90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0a300097d0 tx=0x7f0a3002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.645+0000 7f0a486a4640 1 -- 172.21.15.1:0/2932785054 shutdown_connections 2024-03-06T07:33:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.645+0000 7f0a486a4640 1 --2- 172.21.15.1:0/2932785054 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a400c6b10 0x7f0a40155c90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.645+0000 7f0a486a4640 1 -- 172.21.15.1:0/2932785054 >> 172.21.15.1:0/2932785054 conn(0x7f0a40066ee0 msgr2=0x7f0a400672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.645+0000 7f0a486a4640 1 -- 172.21.15.1:0/2932785054 shutdown_connections 2024-03-06T07:33:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.645+0000 7f0a486a4640 1 -- 172.21.15.1:0/2932785054 wait complete. 2024-03-06T07:33:21.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.646+0000 7f0a486a4640 1 Processor -- start 2024-03-06T07:33:21.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.646+0000 7f0a486a4640 1 -- start start 2024-03-06T07:33:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.646+0000 7f0a486a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a400bf9d0 0x7f0a400c5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a486a4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a40152650 0x7f0a4014d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a486a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a4014d8e0 0x7f0a40149480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a486a4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a401551b0 con 0x7f0a40152650 2024-03-06T07:33:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a486a4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a400c6fa0 con 0x7f0a4014d8e0 2024-03-06T07:33:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a486a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a40155490 con 0x7f0a400bf9d0 2024-03-06T07:33:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a45c3f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a40152650 0x7f0a4014d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a45c3f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a40152650 0x7f0a4014d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41776/0 (socket says 172.21.15.1:41776) 2024-03-06T07:33:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a45c3f640 1 -- 172.21.15.1:0/263745336 learned_addr learned my addr 172.21.15.1:0/263745336 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a46440640 1 --2- 172.21.15.1:0/263745336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a400bf9d0 0x7f0a400c5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a46c41640 1 --2- 172.21.15.1:0/263745336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a4014d8e0 0x7f0a40149480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a45c3f640 1 -- 172.21.15.1:0/263745336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a400bf9d0 msgr2=0x7f0a400c5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a45c3f640 1 --2- 172.21.15.1:0/263745336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a400bf9d0 0x7f0a400c5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a45c3f640 1 -- 172.21.15.1:0/263745336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a4014d8e0 msgr2=0x7f0a40149480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a45c3f640 1 --2- 172.21.15.1:0/263745336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a4014d8e0 0x7f0a40149480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a45c3f640 1 -- 172.21.15.1:0/263745336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4014dd90 con 0x7f0a40152650 2024-03-06T07:33:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a46440640 1 --2- 172.21.15.1:0/263745336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a400bf9d0 0x7f0a400c5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a45c3f640 1 --2- 172.21.15.1:0/263745336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a40152650 0x7f0a4014d3a0 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f0a3002bec0 tx=0x7f0a300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a36ffd640 1 -- 172.21.15.1:0/263745336 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a30035040 con 0x7f0a40152650 2024-03-06T07:33:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a36ffd640 1 -- 172.21.15.1:0/263745336 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3003ad00 con 0x7f0a40152650 2024-03-06T07:33:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a486a4640 1 -- 172.21.15.1:0/263745336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a40153e00 con 0x7f0a40152650 2024-03-06T07:33:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a36ffd640 1 -- 172.21.15.1:0/263745336 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a300466c0 con 0x7f0a40152650 2024-03-06T07:33:21.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.647+0000 7f0a486a4640 1 -- 172.21.15.1:0/263745336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a400c5f40 con 0x7f0a40152650 2024-03-06T07:33:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.648+0000 7f0a486a4640 1 -- 172.21.15.1:0/263745336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a40067fd0 con 0x7f0a40152650 2024-03-06T07:33:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.649+0000 7f0a36ffd640 1 -- 172.21.15.1:0/263745336 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a30038020 con 0x7f0a40152650 2024-03-06T07:33:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.649+0000 7f0a36ffd640 1 --2- 172.21.15.1:0/263745336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a2406b500 0x7f0a2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.649+0000 7f0a36ffd640 1 -- 172.21.15.1:0/263745336 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f0a300ac490 con 0x7f0a40152650 2024-03-06T07:33:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.649+0000 7f0a46440640 1 --2- 172.21.15.1:0/263745336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a2406b500 0x7f0a2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.650+0000 7f0a46440640 1 --2- 172.21.15.1:0/263745336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a2406b500 0x7f0a2406d9c0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c007fd0 tx=0x7f0a3c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.654+0000 7f0a36ffd640 1 -- 172.21.15.1:0/263745336 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a4014dd90 con 0x7f0a40152650 2024-03-06T07:33:21.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.871+0000 7f0a486a4640 1 -- 172.21.15.1:0/263745336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a40154320 con 0x7f0a40152650 2024-03-06T07:33:22.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.871+0000 7f0a36ffd640 1 -- 172.21.15.1:0/263745336 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3473 (secure 0 0 0) 0x7f0a3007b110 con 0x7f0a40152650 2024-03-06T07:33:22.106 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:22.106 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:32.853395+0000","modified":"2024-03-06T07:31:24.318771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10806,"mds_1":8171},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8171":{"gid":8171,"name":"b","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.98:6837/3233486394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3233486394},{"type":"v1","addr":"172.21.15.98:6837","nonce":3233486394}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:24.318771+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10806":{"gid":10806,"name":"c","rank":0,"incarnation":193,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6835/1871567565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":1871567565},{"type":"v1","addr":"172.21.15.1:6835","nonce":1871567565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10806,"qdb_cluster":[10806]},"id":16}]} 2024-03-06T07:33:22.107 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 213 2024-03-06T07:33:22.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.873+0000 7f0a486a4640 1 -- 172.21.15.1:0/263745336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a2406b500 msgr2=0x7f0a2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.873+0000 7f0a486a4640 1 --2- 172.21.15.1:0/263745336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a2406b500 0x7f0a2406d9c0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c007fd0 tx=0x7f0a3c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:22.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.873+0000 7f0a486a4640 1 -- 172.21.15.1:0/263745336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a40152650 msgr2=0x7f0a4014d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.873+0000 7f0a486a4640 1 --2- 172.21.15.1:0/263745336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a40152650 0x7f0a4014d3a0 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f0a3002bec0 tx=0x7f0a300046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.874+0000 7f0a486a4640 1 -- 172.21.15.1:0/263745336 shutdown_connections 2024-03-06T07:33:22.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.874+0000 7f0a486a4640 1 --2- 172.21.15.1:0/263745336 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0a2406b500 0x7f0a2406d9c0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.874+0000 7f0a486a4640 1 --2- 172.21.15.1:0/263745336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a4014d8e0 0x7f0a40149480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.874+0000 7f0a486a4640 1 --2- 172.21.15.1:0/263745336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a40152650 0x7f0a4014d3a0 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.874+0000 7f0a486a4640 1 --2- 172.21.15.1:0/263745336 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0a400bf9d0 0x7f0a400c5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.874+0000 7f0a486a4640 1 -- 172.21.15.1:0/263745336 >> 172.21.15.1:0/263745336 conn(0x7f0a40066ee0 msgr2=0x7f0a400c3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.874+0000 7f0a486a4640 1 -- 172.21.15.1:0/263745336 shutdown_connections 2024-03-06T07:33:22.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:21.874+0000 7f0a486a4640 1 -- 172.21.15.1:0/263745336 wait complete. 2024-03-06T07:33:22.109 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:22.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.208+0000 7f79e04a4640 1 Processor -- start 2024-03-06T07:33:22.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.208+0000 7f79e04a4640 1 -- start start 2024-03-06T07:33:22.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.209+0000 7f79e04a4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f79d8152650 0x7f79d8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.209+0000 7f79e04a4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f79d80c6b10 0x7f79d8158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.209+0000 7f79e04a4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d80c2fc0 con 0x7f79d80bf9d0 2024-03-06T07:33:22.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.209+0000 7f79e04a4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d8155800 con 0x7f79d80c6b10 2024-03-06T07:33:22.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.209+0000 7f79e04a4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d80c6690 con 0x7f79d8152650 2024-03-06T07:33:22.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.209+0000 7f79dda3f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f79d80c6b10 0x7f79d8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.209+0000 7f79dea41640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f79d80bf9d0 0x7f79d80c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59980/0 (socket says 172.21.15.1:59980) 2024-03-06T07:33:22.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.209+0000 7f79dea41640 1 -- 172.21.15.1:0/1441081199 learned_addr learned my addr 172.21.15.1:0/1441081199 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:22.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.209+0000 7f79dda3f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f79d80c6b10 0x7f79d8158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41778/0 (socket says 172.21.15.1:41778) 2024-03-06T07:33:22.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.209+0000 7f79de240640 1 --2- 172.21.15.1:0/1441081199 >> v2:172.21.15.98:3300/0 conn(0x7f79d8152650 0x7f79d8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.210+0000 7f79dda3f640 1 -- 172.21.15.1:0/1441081199 >> v2:172.21.15.98:3300/0 conn(0x7f79d8152650 msgr2=0x7f79d8155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.210+0000 7f79dda3f640 1 --2- 172.21.15.1:0/1441081199 >> v2:172.21.15.98:3300/0 conn(0x7f79d8152650 0x7f79d8155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.210+0000 7f79dda3f640 1 -- 172.21.15.1:0/1441081199 >> v1:172.21.15.1:6789/0 conn(0x7f79d80bf9d0 legacy=0x7f79d80c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.210+0000 7f79dda3f640 1 -- 172.21.15.1:0/1441081199 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d80c7280 con 0x7f79d80c6b10 2024-03-06T07:33:22.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.210+0000 7f79dda3f640 1 --2- 172.21.15.1:0/1441081199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79d80c6b10 0x7f79d8158690 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f79d80cac90 tx=0x7f79d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff3d93224a4e52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.210+0000 7f79dd23e640 1 -- 172.21.15.1:0/1441081199 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d40315a0 con 0x7f79d80c6b10 2024-03-06T07:33:22.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.210+0000 7f79dd23e640 1 -- 172.21.15.1:0/1441081199 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d4037070 con 0x7f79d80c6b10 2024-03-06T07:33:22.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.210+0000 7f79dd23e640 1 -- 172.21.15.1:0/1441081199 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d4031960 con 0x7f79d80c6b10 2024-03-06T07:33:22.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.211+0000 7f79e04a4640 1 -- 172.21.15.1:0/1441081199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79d80c6b10 msgr2=0x7f79d8158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.211+0000 7f79e04a4640 1 --2- 172.21.15.1:0/1441081199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79d80c6b10 0x7f79d8158690 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f79d80cac90 tx=0x7f79d402b840 comp rx=0 tx=0).stop 2024-03-06T07:33:22.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.211+0000 7f79e04a4640 1 -- 172.21.15.1:0/1441081199 shutdown_connections 2024-03-06T07:33:22.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.211+0000 7f79e04a4640 1 --2- 172.21.15.1:0/1441081199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79d80c6b10 0x7f79d8158690 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.211+0000 7f79e04a4640 1 --2- 172.21.15.1:0/1441081199 >> v2:172.21.15.98:3300/0 conn(0x7f79d8152650 0x7f79d8155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.211+0000 7f79e04a4640 1 -- 172.21.15.1:0/1441081199 >> 172.21.15.1:0/1441081199 conn(0x7f79d8066ee0 msgr2=0x7f79d80672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.211+0000 7f79e04a4640 1 -- 172.21.15.1:0/1441081199 shutdown_connections 2024-03-06T07:33:22.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.211+0000 7f79e04a4640 1 -- 172.21.15.1:0/1441081199 wait complete. 2024-03-06T07:33:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.212+0000 7f79e04a4640 1 Processor -- start 2024-03-06T07:33:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.212+0000 7f79e04a4640 1 -- start start 2024-03-06T07:33:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79e04a4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79d80bf9d0 0x7f79d8156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79e04a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d8152650 0x7f79d81e1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79e04a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79d81569d0 0x7f79d81e2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79e04a4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d8155490 con 0x7f79d80bf9d0 2024-03-06T07:33:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79e04a4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d81551b0 con 0x7f79d8152650 2024-03-06T07:33:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79e04a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d8152a30 con 0x7f79d81569d0 2024-03-06T07:33:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79de240640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79d80bf9d0 0x7f79d8156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79de240640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79d80bf9d0 0x7f79d8156490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41792/0 (socket says 172.21.15.1:41792) 2024-03-06T07:33:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79de240640 1 -- 172.21.15.1:0/2412797899 learned_addr learned my addr 172.21.15.1:0/2412797899 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79dea41640 1 --2- 172.21.15.1:0/2412797899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79d81569d0 0x7f79d81e2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79dda3f640 1 --2- 172.21.15.1:0/2412797899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d8152650 0x7f79d81e1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79de240640 1 -- 172.21.15.1:0/2412797899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79d81569d0 msgr2=0x7f79d81e2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79de240640 1 --2- 172.21.15.1:0/2412797899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79d81569d0 0x7f79d81e2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79de240640 1 -- 172.21.15.1:0/2412797899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d8152650 msgr2=0x7f79d81e1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79de240640 1 --2- 172.21.15.1:0/2412797899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d8152650 0x7f79d81e1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79de240640 1 -- 172.21.15.1:0/2412797899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d81eb7e0 con 0x7f79d80bf9d0 2024-03-06T07:33:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79dea41640 1 --2- 172.21.15.1:0/2412797899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79d81569d0 0x7f79d81e2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79dda3f640 1 --2- 172.21.15.1:0/2412797899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d8152650 0x7f79d81e1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.213+0000 7f79de240640 1 --2- 172.21.15.1:0/2412797899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79d80bf9d0 0x7f79d8156490 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f79c800e9b0 tx=0x7f79c800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.214+0000 7f79ceffd640 1 -- 172.21.15.1:0/2412797899 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c8017070 con 0x7f79d80bf9d0 2024-03-06T07:33:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.214+0000 7f79ceffd640 1 -- 172.21.15.1:0/2412797899 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c800c9a0 con 0x7f79d80bf9d0 2024-03-06T07:33:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.214+0000 7f79e04a4640 1 -- 172.21.15.1:0/2412797899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d81ebac0 con 0x7f79d80bf9d0 2024-03-06T07:33:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.214+0000 7f79ceffd640 1 -- 172.21.15.1:0/2412797899 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c800cd90 con 0x7f79d80bf9d0 2024-03-06T07:33:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.214+0000 7f79e04a4640 1 -- 172.21.15.1:0/2412797899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d8153270 con 0x7f79d80bf9d0 2024-03-06T07:33:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.215+0000 7f79e04a4640 1 -- 172.21.15.1:0/2412797899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d8067fd0 con 0x7f79d80bf9d0 2024-03-06T07:33:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.215+0000 7f79ceffd640 1 -- 172.21.15.1:0/2412797899 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79c801a070 con 0x7f79d80bf9d0 2024-03-06T07:33:22.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.216+0000 7f79ceffd640 1 --2- 172.21.15.1:0/2412797899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f79bc06b4d0 0x7f79bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.216+0000 7f79ceffd640 1 -- 172.21.15.1:0/2412797899 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f79c808a480 con 0x7f79d80bf9d0 2024-03-06T07:33:22.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.216+0000 7f79dda3f640 1 --2- 172.21.15.1:0/2412797899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f79bc06b4d0 0x7f79bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.216+0000 7f79dda3f640 1 --2- 172.21.15.1:0/2412797899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f79bc06b4d0 0x7f79bc06d990 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f79d4036000 tx=0x7f79d4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.220+0000 7f79ceffd640 1 -- 172.21.15.1:0/2412797899 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79c8059070 con 0x7f79d80bf9d0 2024-03-06T07:33:22.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.404+0000 7f79e04a4640 1 -- 172.21.15.1:0/2412797899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79d814f030 con 0x7f79d80bf9d0 2024-03-06T07:33:22.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.405+0000 7f79ceffd640 1 -- 172.21.15.1:0/2412797899 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v214) v1 ==== 74+0+17167 (secure 0 0 0) 0x7f79c8058970 con 0x7f79d80bf9d0 2024-03-06T07:33:22.407 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:22.407 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":214,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:30:49.314643+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:31.114390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:31.815153+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"210","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"34.26","mappings":[{"from":0,"to":2}]},{"pgid":"34.31","mappings":[{"from":0,"to":2}]},{"pgid":"34.3f","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6836/2083320955":"2024-03-07T07:30:49.314605+0000","172.21.15.98:6834/2083320955":"2024-03-07T07:30:49.314605+0000","172.21.15.1:6834/2424540959":"2024-03-07T07:30:14.310288+0000","172.21.15.1:6832/2424540959":"2024-03-07T07:30:14.310288+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.407+0000 7f79e04a4640 1 -- 172.21.15.1:0/2412797899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f79bc06b4d0 msgr2=0x7f79bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.407+0000 7f79e04a4640 1 --2- 172.21.15.1:0/2412797899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f79bc06b4d0 0x7f79bc06d990 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f79d4036000 tx=0x7f79d4008030 comp rx=0 tx=0).stop 2024-03-06T07:33:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.407+0000 7f79e04a4640 1 -- 172.21.15.1:0/2412797899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79d80bf9d0 msgr2=0x7f79d8156490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.407+0000 7f79e04a4640 1 --2- 172.21.15.1:0/2412797899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79d80bf9d0 0x7f79d8156490 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f79c800e9b0 tx=0x7f79c800ee70 comp rx=0 tx=0).stop 2024-03-06T07:33:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.407+0000 7f79e04a4640 1 -- 172.21.15.1:0/2412797899 shutdown_connections 2024-03-06T07:33:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.407+0000 7f79e04a4640 1 --2- 172.21.15.1:0/2412797899 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f79bc06b4d0 0x7f79bc06d990 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.407+0000 7f79e04a4640 1 --2- 172.21.15.1:0/2412797899 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79d81569d0 0x7f79d81e2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.407+0000 7f79e04a4640 1 --2- 172.21.15.1:0/2412797899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d8152650 0x7f79d81e1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.407+0000 7f79e04a4640 1 --2- 172.21.15.1:0/2412797899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79d80bf9d0 0x7f79d8156490 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.407+0000 7f79e04a4640 1 -- 172.21.15.1:0/2412797899 >> 172.21.15.1:0/2412797899 conn(0x7f79d8066ee0 msgr2=0x7f79d8150810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.407+0000 7f79e04a4640 1 -- 172.21.15.1:0/2412797899 shutdown_connections 2024-03-06T07:33:22.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.407+0000 7f79e04a4640 1 -- 172.21.15.1:0/2412797899 wait complete. 2024-03-06T07:33:22.420 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:33:22.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.521+0000 7f3481df7640 1 Processor -- start 2024-03-06T07:33:22.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.521+0000 7f3481df7640 1 -- start start 2024-03-06T07:33:22.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.522+0000 7f3481df7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f347c0c05f0 0x7f347c0c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.522+0000 7f3481df7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f347c06f640 0x7f347c1551b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.522+0000 7f3481df7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347c0c31f0 con 0x7f347c0c14c0 2024-03-06T07:33:22.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.522+0000 7f3481df7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f347c06fe90 con 0x7f347c0c05f0 2024-03-06T07:33:22.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.522+0000 7f3481df7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f347c0c2fc0 con 0x7f347c06f640 2024-03-06T07:33:22.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.522+0000 7f347b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f347c0c05f0 0x7f347c0c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.522+0000 7f347b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f347c0c05f0 0x7f347c0c09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41804/0 (socket says 172.21.15.1:41804) 2024-03-06T07:33:22.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.522+0000 7f347b7fe640 1 -- 172.21.15.1:0/1467905220 learned_addr learned my addr 172.21.15.1:0/1467905220 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:22.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.522+0000 7f347affd640 1 --2- 172.21.15.1:0/1467905220 >> v2:172.21.15.98:3300/0 conn(0x7f347c06f640 0x7f347c1551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.522+0000 7f347b7fe640 1 -- 172.21.15.1:0/1467905220 >> v2:172.21.15.98:3300/0 conn(0x7f347c06f640 msgr2=0x7f347c1551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.522+0000 7f347b7fe640 1 --2- 172.21.15.1:0/1467905220 >> v2:172.21.15.98:3300/0 conn(0x7f347c06f640 0x7f347c1551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.522+0000 7f347b7fe640 1 -- 172.21.15.1:0/1467905220 >> v1:172.21.15.98:6789/0 conn(0x7f347c0c14c0 legacy=0x7f347c1556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.522+0000 7f347b7fe640 1 -- 172.21.15.1:0/1467905220 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c155e00 con 0x7f347c0c05f0 2024-03-06T07:33:22.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.522+0000 7f347affd640 1 --2- 172.21.15.1:0/1467905220 >> v2:172.21.15.98:3300/0 conn(0x7f347c06f640 0x7f347c1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:22.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.523+0000 7f347b7fe640 1 --2- 172.21.15.1:0/1467905220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f347c0c05f0 0x7f347c0c09d0 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f347c06cb30 tx=0x7f346402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ea0c536ffa0795 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.523+0000 7f347a7fc640 1 -- 172.21.15.1:0/1467905220 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3464034070 con 0x7f347c0c05f0 2024-03-06T07:33:22.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.523+0000 7f347a7fc640 1 -- 172.21.15.1:0/1467905220 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3464030860 con 0x7f347c0c05f0 2024-03-06T07:33:22.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.523+0000 7f347a7fc640 1 -- 172.21.15.1:0/1467905220 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3464030c30 con 0x7f347c0c05f0 2024-03-06T07:33:22.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.523+0000 7f3481df7640 1 -- 172.21.15.1:0/1467905220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f347c0c05f0 msgr2=0x7f347c0c09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.523+0000 7f3481df7640 1 --2- 172.21.15.1:0/1467905220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f347c0c05f0 0x7f347c0c09d0 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f347c06cb30 tx=0x7f346402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.524+0000 7f3481df7640 1 -- 172.21.15.1:0/1467905220 shutdown_connections 2024-03-06T07:33:22.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.524+0000 7f3481df7640 1 --2- 172.21.15.1:0/1467905220 >> v2:172.21.15.98:3300/0 conn(0x7f347c06f640 0x7f347c1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.524+0000 7f3481df7640 1 --2- 172.21.15.1:0/1467905220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f347c0c05f0 0x7f347c0c09d0 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.524+0000 7f3481df7640 1 -- 172.21.15.1:0/1467905220 >> 172.21.15.1:0/1467905220 conn(0x7f347c0c3cc0 msgr2=0x7f347c0c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.524+0000 7f3481df7640 1 -- 172.21.15.1:0/1467905220 shutdown_connections 2024-03-06T07:33:22.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.524+0000 7f3481df7640 1 -- 172.21.15.1:0/1467905220 wait complete. 2024-03-06T07:33:22.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.525+0000 7f3481df7640 1 Processor -- start 2024-03-06T07:33:22.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.525+0000 7f3481df7640 1 -- start start 2024-03-06T07:33:22.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.525+0000 7f3481df7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f347c06f640 0x7f347c1eb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.525+0000 7f3481df7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f347c0c05f0 0x7f347c157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f3481df7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f347c0c14c0 0x7f347c0aba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f3481df7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c0c3770 con 0x7f347c06f640 2024-03-06T07:33:22.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f3481df7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c0c3490 con 0x7f347c0c14c0 2024-03-06T07:33:22.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f3481df7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347c0c1040 con 0x7f347c0c05f0 2024-03-06T07:33:22.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f347b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f347c06f640 0x7f347c1eb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f347b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f347c06f640 0x7f347c1eb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41820/0 (socket says 172.21.15.1:41820) 2024-03-06T07:33:22.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f347b7fe640 1 -- 172.21.15.1:0/3002638674 learned_addr learned my addr 172.21.15.1:0/3002638674 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:22.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f347affd640 1 --2- 172.21.15.1:0/3002638674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f347c0c05f0 0x7f347c157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f347bfff640 1 --2- 172.21.15.1:0/3002638674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f347c0c14c0 0x7f347c0aba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f347b7fe640 1 -- 172.21.15.1:0/3002638674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f347c0c05f0 msgr2=0x7f347c157480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f347b7fe640 1 --2- 172.21.15.1:0/3002638674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f347c0c05f0 0x7f347c157480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f347b7fe640 1 -- 172.21.15.1:0/3002638674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f347c0c14c0 msgr2=0x7f347c0aba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f347b7fe640 1 --2- 172.21.15.1:0/3002638674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f347c0c14c0 0x7f347c0aba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f347b7fe640 1 -- 172.21.15.1:0/3002638674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c0aa130 con 0x7f347c06f640 2024-03-06T07:33:22.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f347bfff640 1 --2- 172.21.15.1:0/3002638674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f347c0c14c0 0x7f347c0aba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:22.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f347b7fe640 1 --2- 172.21.15.1:0/3002638674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f347c06f640 0x7f347c1eb800 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f346402b9e0 tx=0x7f3464004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f3473fff640 1 -- 172.21.15.1:0/3002638674 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3464048070 con 0x7f347c06f640 2024-03-06T07:33:22.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f3473fff640 1 -- 172.21.15.1:0/3002638674 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3464030660 con 0x7f347c06f640 2024-03-06T07:33:22.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f3481df7640 1 -- 172.21.15.1:0/3002638674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f347c1eef40 con 0x7f347c06f640 2024-03-06T07:33:22.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f3473fff640 1 -- 172.21.15.1:0/3002638674 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3464034040 con 0x7f347c06f640 2024-03-06T07:33:22.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.526+0000 7f3481df7640 1 -- 172.21.15.1:0/3002638674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347c1eaf20 con 0x7f347c06f640 2024-03-06T07:33:22.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.528+0000 7f3481df7640 1 -- 172.21.15.1:0/3002638674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f347c067fd0 con 0x7f347c06f640 2024-03-06T07:33:22.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.528+0000 7f3473fff640 1 -- 172.21.15.1:0/3002638674 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f346404b070 con 0x7f347c06f640 2024-03-06T07:33:22.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.528+0000 7f3473fff640 1 --2- 172.21.15.1:0/3002638674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f345c06b4d0 0x7f345c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.528+0000 7f3473fff640 1 -- 172.21.15.1:0/3002638674 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f34640ac4c0 con 0x7f347c06f640 2024-03-06T07:33:22.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.529+0000 7f347affd640 1 --2- 172.21.15.1:0/3002638674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f345c06b4d0 0x7f345c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.529+0000 7f347affd640 1 --2- 172.21.15.1:0/3002638674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f345c06b4d0 0x7f345c06d990 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f347c14ac90 tx=0x7f346c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.533+0000 7f3473fff640 1 -- 172.21.15.1:0/3002638674 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f346407b0b0 con 0x7f347c06f640 2024-03-06T07:33:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.751+0000 7f3481df7640 1 -- 172.21.15.1:0/3002638674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f347c0ac750 con 0x7f347c06f640 2024-03-06T07:33:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.753+0000 7f3473fff640 1 -- 172.21.15.1:0/3002638674 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v214) v1 ==== 161+0+0 (secure 0 0 0) 0x7f346407a9b0 con 0x7f347c06f640 2024-03-06T07:33:22.755 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:33:22.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.756+0000 7f3481df7640 1 -- 172.21.15.1:0/3002638674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f345c06b4d0 msgr2=0x7f345c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.756+0000 7f3481df7640 1 --2- 172.21.15.1:0/3002638674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f345c06b4d0 0x7f345c06d990 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f347c14ac90 tx=0x7f346c008040 comp rx=0 tx=0).stop 2024-03-06T07:33:22.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.756+0000 7f3481df7640 1 -- 172.21.15.1:0/3002638674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f347c06f640 msgr2=0x7f347c1eb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.756+0000 7f3481df7640 1 --2- 172.21.15.1:0/3002638674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f347c06f640 0x7f347c1eb800 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f346402b9e0 tx=0x7f3464004730 comp rx=0 tx=0).stop 2024-03-06T07:33:22.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.756+0000 7f3481df7640 1 -- 172.21.15.1:0/3002638674 shutdown_connections 2024-03-06T07:33:22.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.756+0000 7f3481df7640 1 --2- 172.21.15.1:0/3002638674 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f345c06b4d0 0x7f345c06d990 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.756+0000 7f3481df7640 1 --2- 172.21.15.1:0/3002638674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f347c0c14c0 0x7f347c0aba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.756+0000 7f3481df7640 1 --2- 172.21.15.1:0/3002638674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f347c0c05f0 0x7f347c157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.756+0000 7f3481df7640 1 --2- 172.21.15.1:0/3002638674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f347c06f640 0x7f347c1eb800 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.756+0000 7f3481df7640 1 -- 172.21.15.1:0/3002638674 >> 172.21.15.1:0/3002638674 conn(0x7f347c0c3cc0 msgr2=0x7f347c14c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.757+0000 7f3481df7640 1 -- 172.21.15.1:0/3002638674 shutdown_connections 2024-03-06T07:33:22.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.757+0000 7f3481df7640 1 -- 172.21.15.1:0/3002638674 wait complete. 2024-03-06T07:33:22.761 INFO:tasks.ceph.mds.c.smithi001.stderr: -8534> 2024-03-06T07:29:38.763+0000 7f76a9082640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:33:22.761 INFO:tasks.ceph.mds.c.smithi001.stderr: -8190> 2024-03-06T07:29:43.763+0000 7f76a9082640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:33:22.770 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:33:22.793 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-06T07:33:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.873+0000 7fac021f6640 1 Processor -- start 2024-03-06T07:33:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.873+0000 7fac021f6640 1 -- start start 2024-03-06T07:33:22.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.874+0000 7fac021f6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fabfc0c05f0 0x7fabfc0c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.874+0000 7fac021f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabfc06f640 0x7fabfc1551b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.874+0000 7fac021f6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fabfc0c14c0 0x7fabfc1556f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.874+0000 7fac021f6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabfc06fe90 con 0x7fabfc0c14c0 2024-03-06T07:33:22.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.874+0000 7fac021f6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabfc0c2fc0 con 0x7fabfc0c05f0 2024-03-06T07:33:22.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.874+0000 7fac021f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc0c31f0 con 0x7fabfc06f640 2024-03-06T07:33:22.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.874+0000 7fabfbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fabfc0c14c0 0x7fabfc1556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.874+0000 7fabfbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fabfc0c14c0 0x7fabfc1556f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41846/0 (socket says 172.21.15.1:41846) 2024-03-06T07:33:22.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.874+0000 7fabfbfff640 1 -- 172.21.15.1:0/2265550046 learned_addr learned my addr 172.21.15.1:0/2265550046 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:22.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.874+0000 7fabfaffd640 1 --2- 172.21.15.1:0/2265550046 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabfc06f640 0x7fabfc1551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.874+0000 7fabfb7fe640 1 --2- 172.21.15.1:0/2265550046 >> v2:172.21.15.98:3300/0 conn(0x7fabfc0c05f0 0x7fabfc0c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.874+0000 7fabfbfff640 1 -- 172.21.15.1:0/2265550046 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabfc06f640 msgr2=0x7fabfc1551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.874+0000 7fabfbfff640 1 --2- 172.21.15.1:0/2265550046 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabfc06f640 0x7fabfc1551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.874+0000 7fabfbfff640 1 -- 172.21.15.1:0/2265550046 >> v2:172.21.15.98:3300/0 conn(0x7fabfc0c05f0 msgr2=0x7fabfc0c09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.874+0000 7fabfbfff640 1 --2- 172.21.15.1:0/2265550046 >> v2:172.21.15.98:3300/0 conn(0x7fabfc0c05f0 0x7fabfc0c09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.874+0000 7fabfbfff640 1 -- 172.21.15.1:0/2265550046 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabfc06fbb0 con 0x7fabfc0c14c0 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.875+0000 7fabfbfff640 1 --2- 172.21.15.1:0/2265550046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabfc0c14c0 0x7fabfc1556f0 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0c4c10 tx=0x7fabf402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79f70bd9ac7bc05e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.875+0000 7fabfa7fc640 1 -- 172.21.15.1:0/2265550046 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf402ca70 con 0x7fabfc0c14c0 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.875+0000 7fabfa7fc640 1 -- 172.21.15.1:0/2265550046 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf402cc10 con 0x7fabfc0c14c0 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.875+0000 7fabfa7fc640 1 -- 172.21.15.1:0/2265550046 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf40343f0 con 0x7fabfc0c14c0 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.875+0000 7fac021f6640 1 -- 172.21.15.1:0/2265550046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabfc0c14c0 msgr2=0x7fabfc1556f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.875+0000 7fac021f6640 1 --2- 172.21.15.1:0/2265550046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabfc0c14c0 0x7fabfc1556f0 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0c4c10 tx=0x7fabf402bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.875+0000 7fac021f6640 1 -- 172.21.15.1:0/2265550046 shutdown_connections 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.875+0000 7fac021f6640 1 --2- 172.21.15.1:0/2265550046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabfc0c14c0 0x7fabfc1556f0 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.876+0000 7fac021f6640 1 --2- 172.21.15.1:0/2265550046 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabfc06f640 0x7fabfc1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.876+0000 7fac021f6640 1 --2- 172.21.15.1:0/2265550046 >> v2:172.21.15.98:3300/0 conn(0x7fabfc0c05f0 0x7fabfc0c09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.876+0000 7fac021f6640 1 -- 172.21.15.1:0/2265550046 >> 172.21.15.1:0/2265550046 conn(0x7fabfc0c3cc0 msgr2=0x7fabfc0c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.876+0000 7fac021f6640 1 -- 172.21.15.1:0/2265550046 shutdown_connections 2024-03-06T07:33:22.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.876+0000 7fac021f6640 1 -- 172.21.15.1:0/2265550046 wait complete. 2024-03-06T07:33:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.877+0000 7fac021f6640 1 Processor -- start 2024-03-06T07:33:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.877+0000 7fac021f6640 1 -- start start 2024-03-06T07:33:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.877+0000 7fac021f6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabfc06f640 0x7fabfc1576f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.877+0000 7fac021f6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabfc0c05f0 0x7fabfc0abb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.877+0000 7fac021f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabfc0c14c0 0x7fabfc1eb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.877+0000 7fac021f6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc0c1040 con 0x7fabfc06f640 2024-03-06T07:33:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.877+0000 7fac021f6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc0c3770 con 0x7fabfc0c14c0 2024-03-06T07:33:22.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.877+0000 7fac021f6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc14f460 con 0x7fabfc0c05f0 2024-03-06T07:33:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.878+0000 7fabfb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabfc06f640 0x7fabfc1576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.878+0000 7fabfb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabfc06f640 0x7fabfc1576f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41848/0 (socket says 172.21.15.1:41848) 2024-03-06T07:33:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.878+0000 7fabfb7fe640 1 -- 172.21.15.1:0/2555476311 learned_addr learned my addr 172.21.15.1:0/2555476311 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.878+0000 7fabfaffd640 1 --2- 172.21.15.1:0/2555476311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabfc0c05f0 0x7fabfc0abb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.878+0000 7fabfbfff640 1 --2- 172.21.15.1:0/2555476311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabfc0c14c0 0x7fabfc1eb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.878+0000 7fabfb7fe640 1 -- 172.21.15.1:0/2555476311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabfc0c05f0 msgr2=0x7fabfc0abb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.878+0000 7fabfb7fe640 1 --2- 172.21.15.1:0/2555476311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabfc0c05f0 0x7fabfc0abb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.878+0000 7fabfb7fe640 1 -- 172.21.15.1:0/2555476311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabfc0c14c0 msgr2=0x7fabfc1eb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.878+0000 7fabfb7fe640 1 --2- 172.21.15.1:0/2555476311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabfc0c14c0 0x7fabfc1eb330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.878+0000 7fabfb7fe640 1 -- 172.21.15.1:0/2555476311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabfc0ab4b0 con 0x7fabfc06f640 2024-03-06T07:33:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.878+0000 7fabfb7fe640 1 --2- 172.21.15.1:0/2555476311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabfc06f640 0x7fabfc1576f0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7fabec00d800 tx=0x7fabec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.878+0000 7fabe3fff640 1 -- 172.21.15.1:0/2555476311 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabec00b6c0 con 0x7fabfc06f640 2024-03-06T07:33:22.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.878+0000 7fabe3fff640 1 -- 172.21.15.1:0/2555476311 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabec019070 con 0x7fabfc06f640 2024-03-06T07:33:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.878+0000 7fac021f6640 1 -- 172.21.15.1:0/2555476311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabfc1eece0 con 0x7fabfc06f640 2024-03-06T07:33:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.878+0000 7fabe3fff640 1 -- 172.21.15.1:0/2555476311 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabec00ba80 con 0x7fabfc06f640 2024-03-06T07:33:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.878+0000 7fac021f6640 1 -- 172.21.15.1:0/2555476311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabfc0aa230 con 0x7fabfc06f640 2024-03-06T07:33:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.879+0000 7fac021f6640 1 -- 172.21.15.1:0/2555476311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabfc067fd0 con 0x7fabfc06f640 2024-03-06T07:33:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.879+0000 7fabe3fff640 1 -- 172.21.15.1:0/2555476311 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fabec00bc20 con 0x7fabfc06f640 2024-03-06T07:33:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.880+0000 7fabe3fff640 1 --2- 172.21.15.1:0/2555476311 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fabe806b500 0x7fabe806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.880+0000 7fabe3fff640 1 -- 172.21.15.1:0/2555476311 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fabec08a960 con 0x7fabfc06f640 2024-03-06T07:33:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.880+0000 7fabfaffd640 1 --2- 172.21.15.1:0/2555476311 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fabe806b500 0x7fabe806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.881+0000 7fabfaffd640 1 --2- 172.21.15.1:0/2555476311 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fabe806b500 0x7fabe806d9c0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fabf0006fd0 tx=0x7fabf0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:22.884+0000 7fabe3fff640 1 -- 172.21.15.1:0/2555476311 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fabfc0ab4b0 con 0x7fabfc06f640 2024-03-06T07:33:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.104+0000 7fac021f6640 1 -- 172.21.15.1:0/2555476311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fabfc0ac750 con 0x7fabfc06f640 2024-03-06T07:33:23.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.758+0000 7fabe3fff640 1 -- 172.21.15.1:0/2555476311 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v215) v1 ==== 104+0+0 (secure 0 0 0) 0x7fabec059530 con 0x7fabfc06f640 2024-03-06T07:33:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.760+0000 7fac021f6640 1 -- 172.21.15.1:0/2555476311 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fabe806b500 msgr2=0x7fabe806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.760+0000 7fac021f6640 1 --2- 172.21.15.1:0/2555476311 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fabe806b500 0x7fabe806d9c0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fabf0006fd0 tx=0x7fabf0008040 comp rx=0 tx=0).stop 2024-03-06T07:33:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.760+0000 7fac021f6640 1 -- 172.21.15.1:0/2555476311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabfc06f640 msgr2=0x7fabfc1576f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.760+0000 7fac021f6640 1 --2- 172.21.15.1:0/2555476311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabfc06f640 0x7fabfc1576f0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7fabec00d800 tx=0x7fabec00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.760+0000 7fac021f6640 1 -- 172.21.15.1:0/2555476311 shutdown_connections 2024-03-06T07:33:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.760+0000 7fac021f6640 1 --2- 172.21.15.1:0/2555476311 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fabe806b500 0x7fabe806d9c0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.760+0000 7fac021f6640 1 --2- 172.21.15.1:0/2555476311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabfc0c14c0 0x7fabfc1eb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.760+0000 7fac021f6640 1 --2- 172.21.15.1:0/2555476311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fabfc0c05f0 0x7fabfc0abb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.760+0000 7fac021f6640 1 --2- 172.21.15.1:0/2555476311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabfc06f640 0x7fabfc1576f0 unknown :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.760+0000 7fac021f6640 1 -- 172.21.15.1:0/2555476311 >> 172.21.15.1:0/2555476311 conn(0x7fabfc0c3cc0 msgr2=0x7fabfc06dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.761+0000 7fac021f6640 1 -- 172.21.15.1:0/2555476311 shutdown_connections 2024-03-06T07:33:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.761+0000 7fac021f6640 1 -- 172.21.15.1:0/2555476311 wait complete. 2024-03-06T07:33:23.776 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:33:23.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.929+0000 7f7839577640 1 Processor -- start 2024-03-06T07:33:23.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.929+0000 7f7839577640 1 -- start start 2024-03-06T07:33:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.930+0000 7f7839577640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7834152c00 0x7f7834072e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.930+0000 7f7839577640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f783406f100 0x7f783406f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.930+0000 7f7839577640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78340a9790 con 0x7f783406c3f0 2024-03-06T07:33:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.930+0000 7f7839577640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78340c3990 con 0x7f7834152c00 2024-03-06T07:33:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.930+0000 7f7839577640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78340a9560 con 0x7f783406f100 2024-03-06T07:33:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.930+0000 7f7832ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7834152c00 0x7f7834072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.930+0000 7f7832ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7834152c00 0x7f7834072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41850/0 (socket says 172.21.15.1:41850) 2024-03-06T07:33:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.930+0000 7f78327fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f783406c3f0 0x7f7834158800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60012/0 (socket says 172.21.15.1:60012) 2024-03-06T07:33:23.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.930+0000 7f7832ffd640 1 -- 172.21.15.1:0/1646381500 learned_addr learned my addr 172.21.15.1:0/1646381500 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:23.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.930+0000 7f78337fe640 1 --2- 172.21.15.1:0/1646381500 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f783406f100 0x7f783406f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.930+0000 7f7832ffd640 1 -- 172.21.15.1:0/1646381500 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f783406f100 msgr2=0x7f783406f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:23.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.930+0000 7f7832ffd640 1 --2- 172.21.15.1:0/1646381500 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f783406f100 0x7f783406f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.930+0000 7f7832ffd640 1 -- 172.21.15.1:0/1646381500 >> v1:172.21.15.1:6789/0 conn(0x7f783406c3f0 legacy=0x7f7834158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.930+0000 7f7831ffb640 1 -- 172.21.15.1:0/1646381500 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779232104 0 0) 0x7f78340a9790 con 0x7f783406c3f0 2024-03-06T07:33:23.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.930+0000 7f7832ffd640 1 -- 172.21.15.1:0/1646381500 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78340a9a70 con 0x7f7834152c00 2024-03-06T07:33:23.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.931+0000 7f7832ffd640 1 --2- 172.21.15.1:0/1646381500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7834152c00 0x7f7834072e40 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f7824009870 tx=0x7f782402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a683420f4ce5e77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:23.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.931+0000 7f7831ffb640 1 -- 172.21.15.1:0/1646381500 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7824034070 con 0x7f7834152c00 2024-03-06T07:33:23.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.931+0000 7f7831ffb640 1 -- 172.21.15.1:0/1646381500 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7824030840 con 0x7f7834152c00 2024-03-06T07:33:23.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.931+0000 7f7831ffb640 1 -- 172.21.15.1:0/1646381500 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7824030c10 con 0x7f7834152c00 2024-03-06T07:33:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.931+0000 7f7839577640 1 -- 172.21.15.1:0/1646381500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7834152c00 msgr2=0x7f7834072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.931+0000 7f7839577640 1 --2- 172.21.15.1:0/1646381500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7834152c00 0x7f7834072e40 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f7824009870 tx=0x7f782402b980 comp rx=0 tx=0).stop 2024-03-06T07:33:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.931+0000 7f7839577640 1 -- 172.21.15.1:0/1646381500 shutdown_connections 2024-03-06T07:33:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.932+0000 7f7839577640 1 --2- 172.21.15.1:0/1646381500 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f783406f100 0x7f783406f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.932+0000 7f7839577640 1 --2- 172.21.15.1:0/1646381500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7834152c00 0x7f7834072e40 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.932+0000 7f7839577640 1 -- 172.21.15.1:0/1646381500 >> 172.21.15.1:0/1646381500 conn(0x7f78340c76b0 msgr2=0x7f78340c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.932+0000 7f7839577640 1 -- 172.21.15.1:0/1646381500 shutdown_connections 2024-03-06T07:33:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.932+0000 7f7839577640 1 -- 172.21.15.1:0/1646381500 wait complete. 2024-03-06T07:33:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.933+0000 7f7839577640 1 Processor -- start 2024-03-06T07:33:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.933+0000 7f7839577640 1 -- start start 2024-03-06T07:33:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.933+0000 7f7839577640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783406c3f0 0x7f7834148a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.933+0000 7f7839577640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783406f100 0x7f7834145b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.933+0000 7f7839577640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7834152c00 0x7f7834146090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.933+0000 7f7839577640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783406bf60 con 0x7f783406f100 2024-03-06T07:33:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.933+0000 7f7839577640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78340aa030 con 0x7f783406c3f0 2024-03-06T07:33:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.933+0000 7f7839577640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7834158f10 con 0x7f7834152c00 2024-03-06T07:33:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.934+0000 7f78327fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783406f100 0x7f7834145b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.934+0000 7f78327fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783406f100 0x7f7834145b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41862/0 (socket says 172.21.15.1:41862) 2024-03-06T07:33:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.934+0000 7f78327fc640 1 -- 172.21.15.1:0/4263647305 learned_addr learned my addr 172.21.15.1:0/4263647305 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.934+0000 7f78337fe640 1 --2- 172.21.15.1:0/4263647305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7834152c00 0x7f7834146090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.934+0000 7f7832ffd640 1 --2- 172.21.15.1:0/4263647305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783406c3f0 0x7f7834148a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.934+0000 7f78327fc640 1 -- 172.21.15.1:0/4263647305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7834152c00 msgr2=0x7f7834146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.934+0000 7f78327fc640 1 --2- 172.21.15.1:0/4263647305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7834152c00 0x7f7834146090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.934+0000 7f78327fc640 1 -- 172.21.15.1:0/4263647305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783406c3f0 msgr2=0x7f7834148a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.934+0000 7f78327fc640 1 --2- 172.21.15.1:0/4263647305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783406c3f0 0x7f7834148a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.934+0000 7f78327fc640 1 -- 172.21.15.1:0/4263647305 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7834148fd0 con 0x7f783406f100 2024-03-06T07:33:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.934+0000 7f78327fc640 1 --2- 172.21.15.1:0/4263647305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783406f100 0x7f7834145b50 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f782800c500 tx=0x7f782800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.934+0000 7f781b7fe640 1 -- 172.21.15.1:0/4263647305 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7828018070 con 0x7f783406f100 2024-03-06T07:33:23.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.934+0000 7f781b7fe640 1 -- 172.21.15.1:0/4263647305 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7828007580 con 0x7f783406f100 2024-03-06T07:33:23.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.934+0000 7f7839577640 1 -- 172.21.15.1:0/4263647305 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78341ddf80 con 0x7f783406f100 2024-03-06T07:33:23.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.934+0000 7f781b7fe640 1 -- 172.21.15.1:0/4263647305 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7828004a20 con 0x7f783406f100 2024-03-06T07:33:23.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.934+0000 7f7839577640 1 -- 172.21.15.1:0/4263647305 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7834150bb0 con 0x7f783406f100 2024-03-06T07:33:23.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.935+0000 7f7839577640 1 -- 172.21.15.1:0/4263647305 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78340680b0 con 0x7f783406f100 2024-03-06T07:33:23.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.936+0000 7f781b7fe640 1 -- 172.21.15.1:0/4263647305 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f782801b070 con 0x7f783406f100 2024-03-06T07:33:23.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.936+0000 7f781b7fe640 1 --2- 172.21.15.1:0/4263647305 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f782006b4d0 0x7f782006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.936+0000 7f781b7fe640 1 -- 172.21.15.1:0/4263647305 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7828089730 con 0x7f783406f100 2024-03-06T07:33:23.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.937+0000 7f7832ffd640 1 --2- 172.21.15.1:0/4263647305 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f782006b4d0 0x7f782006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.937+0000 7f7832ffd640 1 --2- 172.21.15.1:0/4263647305 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f782006b4d0 0x7f782006d990 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f7824036000 tx=0x7f782403b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:23.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:23.941+0000 7f781b7fe640 1 -- 172.21.15.1:0/4263647305 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7828057cb0 con 0x7f783406f100 2024-03-06T07:33:24.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.120+0000 7f7839577640 1 -- 172.21.15.1:0/4263647305 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f78340c3b30 con 0x7f783406f100 2024-03-06T07:33:24.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.759+0000 7f781b7fe640 1 -- 172.21.15.1:0/4263647305 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v216) v1 ==== 181+0+0 (secure 0 0 0) 0x7f782805bd50 con 0x7f783406f100 2024-03-06T07:33:24.761 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:33:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.763+0000 7f7839577640 1 -- 172.21.15.1:0/4263647305 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f782006b4d0 msgr2=0x7f782006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.763+0000 7f7839577640 1 --2- 172.21.15.1:0/4263647305 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f782006b4d0 0x7f782006d990 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f7824036000 tx=0x7f782403b040 comp rx=0 tx=0).stop 2024-03-06T07:33:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.763+0000 7f7839577640 1 -- 172.21.15.1:0/4263647305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783406f100 msgr2=0x7f7834145b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.763+0000 7f7839577640 1 --2- 172.21.15.1:0/4263647305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783406f100 0x7f7834145b50 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f782800c500 tx=0x7f782800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.763+0000 7f7839577640 1 -- 172.21.15.1:0/4263647305 shutdown_connections 2024-03-06T07:33:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.763+0000 7f7839577640 1 --2- 172.21.15.1:0/4263647305 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f782006b4d0 0x7f782006d990 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.763+0000 7f7839577640 1 --2- 172.21.15.1:0/4263647305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7834152c00 0x7f7834146090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.763+0000 7f7839577640 1 --2- 172.21.15.1:0/4263647305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f783406f100 0x7f7834145b50 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.763+0000 7f7839577640 1 --2- 172.21.15.1:0/4263647305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783406c3f0 0x7f7834148a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.763+0000 7f7839577640 1 -- 172.21.15.1:0/4263647305 >> 172.21.15.1:0/4263647305 conn(0x7f78340c76b0 msgr2=0x7f783414b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.764+0000 7f7839577640 1 -- 172.21.15.1:0/4263647305 shutdown_connections 2024-03-06T07:33:24.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.764+0000 7f7839577640 1 -- 172.21.15.1:0/4263647305 wait complete. 2024-03-06T07:33:24.781 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:33:24.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.893+0000 7fbfdec0e640 1 Processor -- start 2024-03-06T07:33:24.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.893+0000 7fbfdec0e640 1 -- start start 2024-03-06T07:33:24.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfdec0e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbfd8158180 0x7fbfd80c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfdec0e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd8158560 con 0x7fbfd80c5e20 2024-03-06T07:33:24.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfdec0e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd8155600 con 0x7fbfd80c6800 2024-03-06T07:33:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfdec0e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd8156210 con 0x7fbfd8158180 2024-03-06T07:33:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfd7fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbfd80c5e20 0x7fbfd8156a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37068/0 (socket says 172.21.15.1:37068) 2024-03-06T07:33:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfd7fff640 1 -- 172.21.15.1:0/4155782532 learned_addr learned my addr 172.21.15.1:0/4155782532 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfdc9aa640 1 --2- 172.21.15.1:0/4155782532 >> v2:172.21.15.98:3300/0 conn(0x7fbfd8158180 0x7fbfd80c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfd77fe640 1 -- 172.21.15.1:0/4155782532 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805952756 0 0) 0x7fbfd8158560 con 0x7fbfd80c5e20 2024-03-06T07:33:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfd77fe640 1 -- 172.21.15.1:0/4155782532 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc8003520 con 0x7fbfd80c5e20 2024-03-06T07:33:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfd77fe640 1 -- 172.21.15.1:0/4155782532 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1860262275 0 0) 0x7fbfc8003520 con 0x7fbfd80c5e20 2024-03-06T07:33:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfd77fe640 1 -- 172.21.15.1:0/4155782532 >> v2:172.21.15.98:3300/0 conn(0x7fbfd8158180 msgr2=0x7fbfd80c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfd77fe640 1 --2- 172.21.15.1:0/4155782532 >> v2:172.21.15.98:3300/0 conn(0x7fbfd8158180 0x7fbfd80c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfd77fe640 1 -- 172.21.15.1:0/4155782532 >> v1:172.21.15.98:6789/0 conn(0x7fbfd80c6800 legacy=0x7fbfd8157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfd77fe640 1 -- 172.21.15.1:0/4155782532 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd81558e0 con 0x7fbfd80c5e20 2024-03-06T07:33:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfd77fe640 1 -- 172.21.15.1:0/4155782532 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fbfc4001670 con 0x7fbfd80c5e20 2024-03-06T07:33:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfd77fe640 1 -- 172.21.15.1:0/4155782532 >> v1:172.21.15.1:6789/0 conn(0x7fbfd80c5e20 legacy=0x7fbfd8156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfd77fe640 1 --2- 172.21.15.1:0/4155782532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfc80039b0 0x7fbfc8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfd77fe640 1 --2- 172.21.15.1:0/4155782532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc8004800 0x7fbfc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfd77fe640 1 --2- 172.21.15.1:0/4155782532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbfc80052e0 0x7fbfc800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfd77fe640 1 -- 172.21.15.1:0/4155782532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd8156210 con 0x7fbfc8004800 2024-03-06T07:33:24.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfd77fe640 1 -- 172.21.15.1:0/4155782532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc800c1b0 con 0x7fbfc80039b0 2024-03-06T07:33:24.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.894+0000 7fbfd77fe640 1 -- 172.21.15.1:0/4155782532 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc800c3b0 con 0x7fbfc80052e0 2024-03-06T07:33:24.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.895+0000 7fbfd7fff640 1 --2- 172.21.15.1:0/4155782532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc8004800 0x7fbfc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.895+0000 7fbfdc9aa640 1 --2- 172.21.15.1:0/4155782532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfc80039b0 0x7fbfc8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.895+0000 7fbfdd1ab640 1 --2- 172.21.15.1:0/4155782532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbfc80052e0 0x7fbfc800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.895+0000 7fbfd7fff640 1 -- 172.21.15.1:0/4155782532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbfc80052e0 msgr2=0x7fbfc800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:24.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.895+0000 7fbfd7fff640 1 --2- 172.21.15.1:0/4155782532 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbfc80052e0 0x7fbfc800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.895+0000 7fbfd7fff640 1 -- 172.21.15.1:0/4155782532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfc80039b0 msgr2=0x7fbfc8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:24.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.895+0000 7fbfd7fff640 1 --2- 172.21.15.1:0/4155782532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfc80039b0 0x7fbfc8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.895+0000 7fbfd7fff640 1 -- 172.21.15.1:0/4155782532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc8003520 con 0x7fbfc8004800 2024-03-06T07:33:24.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.895+0000 7fbfdd1ab640 1 -- 172.21.15.1:0/4155782532 reap_dead start 2024-03-06T07:33:24.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.895+0000 7fbfd7fff640 1 --2- 172.21.15.1:0/4155782532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc8004800 0x7fbfc8004c80 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400c210 tx=0x7fbfc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d566e69b58aa4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.896+0000 7fbfd77fe640 1 -- 172.21.15.1:0/4155782532 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4007d10 con 0x7fbfc8004800 2024-03-06T07:33:24.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.896+0000 7fbfd77fe640 1 -- 172.21.15.1:0/4155782532 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc4030070 con 0x7fbfc8004800 2024-03-06T07:33:24.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.896+0000 7fbfd77fe640 1 -- 172.21.15.1:0/4155782532 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4005d90 con 0x7fbfc8004800 2024-03-06T07:33:24.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.896+0000 7fbfdec0e640 1 -- 172.21.15.1:0/4155782532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc8004800 msgr2=0x7fbfc8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.896+0000 7fbfdec0e640 1 --2- 172.21.15.1:0/4155782532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc8004800 0x7fbfc8004c80 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400c210 tx=0x7fbfc8005830 comp rx=0 tx=0).stop 2024-03-06T07:33:24.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.896+0000 7fbfdec0e640 1 -- 172.21.15.1:0/4155782532 shutdown_connections 2024-03-06T07:33:24.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.896+0000 7fbfdec0e640 1 --2- 172.21.15.1:0/4155782532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc8004800 0x7fbfc8004c80 unknown :-1 s=CLOSED pgs=2707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.896+0000 7fbfdec0e640 1 -- 172.21.15.1:0/4155782532 >> 172.21.15.1:0/4155782532 conn(0x7fbfd806b410 msgr2=0x7fbfd806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:24.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.896+0000 7fbfdec0e640 1 -- 172.21.15.1:0/4155782532 shutdown_connections 2024-03-06T07:33:24.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.896+0000 7fbfdec0e640 1 -- 172.21.15.1:0/4155782532 wait complete. 2024-03-06T07:33:24.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.897+0000 7fbfdec0e640 1 Processor -- start 2024-03-06T07:33:24.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.897+0000 7fbfdec0e640 1 -- start start 2024-03-06T07:33:24.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.898+0000 7fbfdec0e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc8004800 0x7fbfd806d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.898+0000 7fbfdec0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfd80ac140 0x7fbfd81ea620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.898+0000 7fbfdec0e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbfd814dea0 0x7fbfd814b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.898+0000 7fbfdec0e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd8155ea0 con 0x7fbfc8004800 2024-03-06T07:33:24.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.898+0000 7fbfdec0e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd80c3ea0 con 0x7fbfd80ac140 2024-03-06T07:33:24.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.898+0000 7fbfdec0e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd81551f0 con 0x7fbfd814dea0 2024-03-06T07:33:24.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.898+0000 7fbfdc9aa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc8004800 0x7fbfd806d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.898+0000 7fbfdc9aa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc8004800 0x7fbfd806d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40564/0 (socket says 172.21.15.1:40564) 2024-03-06T07:33:24.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.898+0000 7fbfdc9aa640 1 -- 172.21.15.1:0/1616394709 learned_addr learned my addr 172.21.15.1:0/1616394709 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:24.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.898+0000 7fbfdd1ab640 1 --2- 172.21.15.1:0/1616394709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbfd814dea0 0x7fbfd814b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.898+0000 7fbfdc9aa640 1 -- 172.21.15.1:0/1616394709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbfd814dea0 msgr2=0x7fbfd814b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.898+0000 7fbfdc9aa640 1 --2- 172.21.15.1:0/1616394709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbfd814dea0 0x7fbfd814b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.898+0000 7fbfdc9aa640 1 -- 172.21.15.1:0/1616394709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfd80ac140 msgr2=0x7fbfd81ea620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.898+0000 7fbfd7fff640 1 --2- 172.21.15.1:0/1616394709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfd80ac140 0x7fbfd81ea620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.899+0000 7fbfdc9aa640 1 --2- 172.21.15.1:0/1616394709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfd80ac140 0x7fbfd81ea620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.899+0000 7fbfdc9aa640 1 -- 172.21.15.1:0/1616394709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd80aca50 con 0x7fbfc8004800 2024-03-06T07:33:24.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.899+0000 7fbfdd1ab640 1 --2- 172.21.15.1:0/1616394709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbfd814dea0 0x7fbfd814b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:24.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.899+0000 7fbfd7fff640 1 --2- 172.21.15.1:0/1616394709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfd80ac140 0x7fbfd81ea620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:24.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.899+0000 7fbfdc9aa640 1 --2- 172.21.15.1:0/1616394709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc8004800 0x7fbfd806d580 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc011aa0 tx=0x7fbfcc011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.899+0000 7fbfd57fa640 1 -- 172.21.15.1:0/1616394709 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc00fb80 con 0x7fbfc8004800 2024-03-06T07:33:24.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.899+0000 7fbfd57fa640 1 -- 172.21.15.1:0/1616394709 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfcc00fd20 con 0x7fbfc8004800 2024-03-06T07:33:24.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.899+0000 7fbfdec0e640 1 -- 172.21.15.1:0/1616394709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd80acd30 con 0x7fbfc8004800 2024-03-06T07:33:24.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.899+0000 7fbfd57fa640 1 -- 172.21.15.1:0/1616394709 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc014660 con 0x7fbfc8004800 2024-03-06T07:33:24.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.899+0000 7fbfdec0e640 1 -- 172.21.15.1:0/1616394709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd80abc20 con 0x7fbfc8004800 2024-03-06T07:33:24.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.900+0000 7fbfdec0e640 1 -- 172.21.15.1:0/1616394709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd80680b0 con 0x7fbfc8004800 2024-03-06T07:33:24.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.900+0000 7fbfd57fa640 1 -- 172.21.15.1:0/1616394709 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfcc012040 con 0x7fbfc8004800 2024-03-06T07:33:24.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.901+0000 7fbfd57fa640 1 --2- 172.21.15.1:0/1616394709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbfbc06b500 0x7fbfbc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.901+0000 7fbfd57fa640 1 -- 172.21.15.1:0/1616394709 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6094+0+0 (secure 0 0 0) 0x7fbfcc08aa50 con 0x7fbfc8004800 2024-03-06T07:33:24.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.901+0000 7fbfd7fff640 1 --2- 172.21.15.1:0/1616394709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbfbc06b500 0x7fbfbc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.901+0000 7fbfd7fff640 1 --2- 172.21.15.1:0/1616394709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbfbc06b500 0x7fbfbc06d9c0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4034000 tx=0x7fbfc4009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:24.905+0000 7fbfd57fa640 1 -- 172.21.15.1:0/1616394709 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfcc019030 con 0x7fbfc8004800 2024-03-06T07:33:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.091+0000 7fbfdec0e640 1 -- 172.21.15.1:0/1616394709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbfd80acf00 con 0x7fbfc8004800 2024-03-06T07:33:25.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.764+0000 7fbfd57fa640 1 -- 172.21.15.1:0/1616394709 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v217) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbfcc059760 con 0x7fbfc8004800 2024-03-06T07:33:25.766 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-06T07:33:25.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.767+0000 7fbfdec0e640 1 -- 172.21.15.1:0/1616394709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbfbc06b500 msgr2=0x7fbfbc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.767+0000 7fbfdec0e640 1 --2- 172.21.15.1:0/1616394709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbfbc06b500 0x7fbfbc06d9c0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4034000 tx=0x7fbfc4009030 comp rx=0 tx=0).stop 2024-03-06T07:33:25.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.767+0000 7fbfdec0e640 1 -- 172.21.15.1:0/1616394709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc8004800 msgr2=0x7fbfd806d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.767+0000 7fbfdec0e640 1 --2- 172.21.15.1:0/1616394709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc8004800 0x7fbfd806d580 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc011aa0 tx=0x7fbfcc011f60 comp rx=0 tx=0).stop 2024-03-06T07:33:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.767+0000 7fbfdec0e640 1 -- 172.21.15.1:0/1616394709 shutdown_connections 2024-03-06T07:33:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.767+0000 7fbfdec0e640 1 --2- 172.21.15.1:0/1616394709 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbfbc06b500 0x7fbfbc06d9c0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.767+0000 7fbfdec0e640 1 --2- 172.21.15.1:0/1616394709 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbfd814dea0 0x7fbfd814b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.767+0000 7fbfdec0e640 1 --2- 172.21.15.1:0/1616394709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfd80ac140 0x7fbfd81ea620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.767+0000 7fbfdec0e640 1 --2- 172.21.15.1:0/1616394709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc8004800 0x7fbfd806d580 secure :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc011aa0 tx=0x7fbfcc011f60 comp rx=0 tx=0).stop 2024-03-06T07:33:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.767+0000 7fbfdec0e640 1 -- 172.21.15.1:0/1616394709 >> 172.21.15.1:0/1616394709 conn(0x7fbfd806b410 msgr2=0x7fbfd8158770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:25.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.768+0000 7fbfdec0e640 1 -- 172.21.15.1:0/1616394709 shutdown_connections 2024-03-06T07:33:25.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.768+0000 7fbfdec0e640 1 -- 172.21.15.1:0/1616394709 wait complete. 2024-03-06T07:33:25.786 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:33:25.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.894+0000 7f9bf51ac640 1 Processor -- start 2024-03-06T07:33:25.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.894+0000 7f9bf51ac640 1 -- start start 2024-03-06T07:33:25.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.895+0000 7f9bf51ac640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9bf00c3260 0x7f9bf00c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.895+0000 7f9bf51ac640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9bf0155b70 0x7f9bf0159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.895+0000 7f9bf51ac640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf00c2b90 con 0x7f9bf00c8cd0 2024-03-06T07:33:25.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.895+0000 7f9bf51ac640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf0156000 con 0x7f9bf00c3260 2024-03-06T07:33:25.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.895+0000 7f9bf51ac640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf00c37b0 con 0x7f9bf0155b70 2024-03-06T07:33:25.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.895+0000 7f9bee59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9bf00c3260 0x7f9bf00c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.895+0000 7f9bee59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9bf00c3260 0x7f9bf00c90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40576/0 (socket says 172.21.15.1:40576) 2024-03-06T07:33:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.895+0000 7f9bee59c640 1 -- 172.21.15.1:0/2184267041 learned_addr learned my addr 172.21.15.1:0/2184267041 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.895+0000 7f9bee59c640 1 -- 172.21.15.1:0/2184267041 >> v2:172.21.15.98:3300/0 conn(0x7f9bf0155b70 msgr2=0x7f9bf0159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.895+0000 7f9bee59c640 1 --2- 172.21.15.1:0/2184267041 >> v2:172.21.15.98:3300/0 conn(0x7f9bf0155b70 0x7f9bf0159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.895+0000 7f9bee59c640 1 -- 172.21.15.1:0/2184267041 >> v1:172.21.15.98:6789/0 conn(0x7f9bf00c8cd0 legacy=0x7f9bf00c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.895+0000 7f9bee59c640 1 -- 172.21.15.1:0/2184267041 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf00c3a90 con 0x7f9bf00c3260 2024-03-06T07:33:25.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.896+0000 7f9bee59c640 1 --2- 172.21.15.1:0/2184267041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bf00c3260 0x7f9bf00c90b0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00736c0 tx=0x7f9be402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d65f2664aaf943 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:25.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.896+0000 7f9bedd9b640 1 -- 172.21.15.1:0/2184267041 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be402e070 con 0x7f9bf00c3260 2024-03-06T07:33:25.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.896+0000 7f9bedd9b640 1 -- 172.21.15.1:0/2184267041 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be4037070 con 0x7f9bf00c3260 2024-03-06T07:33:25.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.896+0000 7f9bedd9b640 1 -- 172.21.15.1:0/2184267041 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be4031a00 con 0x7f9bf00c3260 2024-03-06T07:33:25.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.896+0000 7f9bf51ac640 1 -- 172.21.15.1:0/2184267041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bf00c3260 msgr2=0x7f9bf00c90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.896+0000 7f9bf51ac640 1 --2- 172.21.15.1:0/2184267041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bf00c3260 0x7f9bf00c90b0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00736c0 tx=0x7f9be402b880 comp rx=0 tx=0).stop 2024-03-06T07:33:25.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.897+0000 7f9bf51ac640 1 -- 172.21.15.1:0/2184267041 shutdown_connections 2024-03-06T07:33:25.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.897+0000 7f9bf51ac640 1 --2- 172.21.15.1:0/2184267041 >> v2:172.21.15.98:3300/0 conn(0x7f9bf0155b70 0x7f9bf0159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.897+0000 7f9bf51ac640 1 --2- 172.21.15.1:0/2184267041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bf00c3260 0x7f9bf00c90b0 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.897+0000 7f9bf51ac640 1 -- 172.21.15.1:0/2184267041 >> 172.21.15.1:0/2184267041 conn(0x7f9bf006b3e0 msgr2=0x7f9bf006b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:25.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.897+0000 7f9bf51ac640 1 -- 172.21.15.1:0/2184267041 shutdown_connections 2024-03-06T07:33:25.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.897+0000 7f9bf51ac640 1 -- 172.21.15.1:0/2184267041 wait complete. 2024-03-06T07:33:25.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.898+0000 7f9bf51ac640 1 Processor -- start 2024-03-06T07:33:25.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.898+0000 7f9bf51ac640 1 -- start start 2024-03-06T07:33:25.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9bf51ac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bf00c3260 0x7f9bf00a9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9bf51ac640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9bf00c8cd0 0x7f9bf0156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9bf51ac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf0155b70 0x7f9bf01ee1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9bf51ac640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf0159550 con 0x7f9bf00c3260 2024-03-06T07:33:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9bf51ac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf0158af0 con 0x7f9bf0155b70 2024-03-06T07:33:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9bf51ac640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf00c8560 con 0x7f9bf00c8cd0 2024-03-06T07:33:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9beed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bf00c3260 0x7f9bf00a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9beed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bf00c3260 0x7f9bf00a9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40592/0 (socket says 172.21.15.1:40592) 2024-03-06T07:33:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9beed9d640 1 -- 172.21.15.1:0/4263956771 learned_addr learned my addr 172.21.15.1:0/4263956771 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9bee59c640 1 --2- 172.21.15.1:0/4263956771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9bf00c8cd0 0x7f9bf0156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9bef59e640 1 --2- 172.21.15.1:0/4263956771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf0155b70 0x7f9bf01ee1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9beed9d640 1 -- 172.21.15.1:0/4263956771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9bf00c8cd0 msgr2=0x7f9bf0156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9beed9d640 1 --2- 172.21.15.1:0/4263956771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9bf00c8cd0 0x7f9bf0156c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9beed9d640 1 -- 172.21.15.1:0/4263956771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf0155b70 msgr2=0x7f9bf01ee1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9beed9d640 1 --2- 172.21.15.1:0/4263956771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf0155b70 0x7f9bf01ee1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9beed9d640 1 -- 172.21.15.1:0/4263956771 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf00aabd0 con 0x7f9bf00c3260 2024-03-06T07:33:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9beed9d640 1 --2- 172.21.15.1:0/4263956771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bf00c3260 0x7f9bf00a9840 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7f9be800b3e0 tx=0x7f9be800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9bd77fe640 1 -- 172.21.15.1:0/4263956771 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be8015020 con 0x7f9bf00c3260 2024-03-06T07:33:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9bd77fe640 1 -- 172.21.15.1:0/4263956771 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be8011850 con 0x7f9bf00c3260 2024-03-06T07:33:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9bf51ac640 1 -- 172.21.15.1:0/4263956771 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf00ab7a0 con 0x7f9bf00c3260 2024-03-06T07:33:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.899+0000 7f9bd77fe640 1 -- 172.21.15.1:0/4263956771 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be8011c40 con 0x7f9bf00c3260 2024-03-06T07:33:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.900+0000 7f9bf51ac640 1 -- 172.21.15.1:0/4263956771 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf0157190 con 0x7f9bf00c3260 2024-03-06T07:33:25.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.901+0000 7f9bf51ac640 1 -- 172.21.15.1:0/4263956771 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf00c2d80 con 0x7f9bf00c3260 2024-03-06T07:33:25.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.901+0000 7f9bd77fe640 1 -- 172.21.15.1:0/4263956771 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9be8018020 con 0x7f9bf00c3260 2024-03-06T07:33:25.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.901+0000 7f9bd77fe640 1 --2- 172.21.15.1:0/4263956771 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bdc06b4d0 0x7f9bdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.902+0000 7f9bd77fe640 1 -- 172.21.15.1:0/4263956771 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f9be808a030 con 0x7f9bf00c3260 2024-03-06T07:33:25.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.902+0000 7f9bee59c640 1 --2- 172.21.15.1:0/4263956771 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bdc06b4d0 0x7f9bdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:25.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.902+0000 7f9bee59c640 1 --2- 172.21.15.1:0/4263956771 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bdc06b4d0 0x7f9bdc06d990 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f9be4036000 tx=0x7f9be403d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:25.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:25.906+0000 7f9bd77fe640 1 -- 172.21.15.1:0/4263956771 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9be801f020 con 0x7f9bf00c3260 2024-03-06T07:33:26.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.089+0000 7f9bf51ac640 1 -- 172.21.15.1:0/4263956771 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9bf00c6760 con 0x7f9bf00c3260 2024-03-06T07:33:26.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.095+0000 7f9bd77fe640 1 -- 172.21.15.1:0/4263956771 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v64) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9be80172c0 con 0x7f9bf00c3260 2024-03-06T07:33:26.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.097+0000 7f9bf51ac640 1 -- 172.21.15.1:0/4263956771 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bdc06b4d0 msgr2=0x7f9bdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.097+0000 7f9bf51ac640 1 --2- 172.21.15.1:0/4263956771 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bdc06b4d0 0x7f9bdc06d990 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f9be4036000 tx=0x7f9be403d040 comp rx=0 tx=0).stop 2024-03-06T07:33:26.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.097+0000 7f9bf51ac640 1 -- 172.21.15.1:0/4263956771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bf00c3260 msgr2=0x7f9bf00a9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.097+0000 7f9bf51ac640 1 --2- 172.21.15.1:0/4263956771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bf00c3260 0x7f9bf00a9840 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7f9be800b3e0 tx=0x7f9be800b8a0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.098+0000 7f9bf51ac640 1 -- 172.21.15.1:0/4263956771 shutdown_connections 2024-03-06T07:33:26.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.098+0000 7f9bf51ac640 1 --2- 172.21.15.1:0/4263956771 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9bdc06b4d0 0x7f9bdc06d990 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.098+0000 7f9bf51ac640 1 --2- 172.21.15.1:0/4263956771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf0155b70 0x7f9bf01ee1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.098+0000 7f9bf51ac640 1 --2- 172.21.15.1:0/4263956771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9bf00c8cd0 0x7f9bf0156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.098+0000 7f9bf51ac640 1 --2- 172.21.15.1:0/4263956771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bf00c3260 0x7f9bf00a9840 unknown :-1 s=CLOSED pgs=2710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.098+0000 7f9bf51ac640 1 -- 172.21.15.1:0/4263956771 >> 172.21.15.1:0/4263956771 conn(0x7f9bf006b3e0 msgr2=0x7f9bf00c90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:26.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.098+0000 7f9bf51ac640 1 -- 172.21.15.1:0/4263956771 shutdown_connections 2024-03-06T07:33:26.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.098+0000 7f9bf51ac640 1 -- 172.21.15.1:0/4263956771 wait complete. 2024-03-06T07:33:26.111 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_twice' 2024-03-06T07:33:26.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.210+0000 7f9ef4125640 1 Processor -- start 2024-03-06T07:33:26.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.210+0000 7f9ef4125640 1 -- start start 2024-03-06T07:33:26.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.210+0000 7f9ef4125640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9eec0c05f0 0x7f9eec0c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.211+0000 7f9ef4125640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eec06f640 0x7f9eec1551b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.211+0000 7f9ef4125640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9eec0c14c0 0x7f9eec1556f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.211+0000 7f9ef4125640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eec06fe90 con 0x7f9eec0c05f0 2024-03-06T07:33:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.211+0000 7f9ef4125640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eec0c2fc0 con 0x7f9eec0c14c0 2024-03-06T07:33:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.211+0000 7f9ef4125640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec0c31f0 con 0x7f9eec06f640 2024-03-06T07:33:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.211+0000 7f9ef1ec1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9eec0c05f0 0x7f9eec0c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.211+0000 7f9ef1ec1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9eec0c05f0 0x7f9eec0c09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40608/0 (socket says 172.21.15.1:40608) 2024-03-06T07:33:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.211+0000 7f9ef1ec1640 1 -- 172.21.15.1:0/2989273281 learned_addr learned my addr 172.21.15.1:0/2989273281 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.211+0000 7f9ef16c0640 1 --2- 172.21.15.1:0/2989273281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eec06f640 0x7f9eec1551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.211+0000 7f9ef1ec1640 1 -- 172.21.15.1:0/2989273281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eec06f640 msgr2=0x7f9eec1551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.211+0000 7f9ef1ec1640 1 --2- 172.21.15.1:0/2989273281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eec06f640 0x7f9eec1551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.211+0000 7f9ef26c2640 1 --2- 172.21.15.1:0/2989273281 >> v2:172.21.15.98:3300/0 conn(0x7f9eec0c14c0 0x7f9eec1556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.211+0000 7f9ef1ec1640 1 -- 172.21.15.1:0/2989273281 >> v2:172.21.15.98:3300/0 conn(0x7f9eec0c14c0 msgr2=0x7f9eec1556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:26.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.211+0000 7f9ef1ec1640 1 --2- 172.21.15.1:0/2989273281 >> v2:172.21.15.98:3300/0 conn(0x7f9eec0c14c0 0x7f9eec1556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.211+0000 7f9ef1ec1640 1 -- 172.21.15.1:0/2989273281 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec06fbb0 con 0x7f9eec0c05f0 2024-03-06T07:33:26.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.211+0000 7f9ef26c2640 1 --2- 172.21.15.1:0/2989273281 >> v2:172.21.15.98:3300/0 conn(0x7f9eec0c14c0 0x7f9eec1556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:26.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.211+0000 7f9ef16c0640 1 --2- 172.21.15.1:0/2989273281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eec06f640 0x7f9eec1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:26.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.212+0000 7f9ef1ec1640 1 --2- 172.21.15.1:0/2989273281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eec0c05f0 0x7f9eec0c09d0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f9eec06f3e0 tx=0x7f9ee802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2fe7c72f7429da1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.212+0000 7f9ef0ebf640 1 -- 172.21.15.1:0/2989273281 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8035020 con 0x7f9eec0c05f0 2024-03-06T07:33:26.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.212+0000 7f9ef0ebf640 1 -- 172.21.15.1:0/2989273281 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee8031870 con 0x7f9eec0c05f0 2024-03-06T07:33:26.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.212+0000 7f9ef0ebf640 1 -- 172.21.15.1:0/2989273281 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8031c40 con 0x7f9eec0c05f0 2024-03-06T07:33:26.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.212+0000 7f9ef4125640 1 -- 172.21.15.1:0/2989273281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eec0c05f0 msgr2=0x7f9eec0c09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.212+0000 7f9ef4125640 1 --2- 172.21.15.1:0/2989273281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eec0c05f0 0x7f9eec0c09d0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f9eec06f3e0 tx=0x7f9ee802b910 comp rx=0 tx=0).stop 2024-03-06T07:33:26.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.212+0000 7f9ef4125640 1 -- 172.21.15.1:0/2989273281 shutdown_connections 2024-03-06T07:33:26.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.212+0000 7f9ef4125640 1 --2- 172.21.15.1:0/2989273281 >> v2:172.21.15.98:3300/0 conn(0x7f9eec0c14c0 0x7f9eec1556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.212+0000 7f9ef4125640 1 --2- 172.21.15.1:0/2989273281 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eec06f640 0x7f9eec1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.212+0000 7f9ef4125640 1 --2- 172.21.15.1:0/2989273281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eec0c05f0 0x7f9eec0c09d0 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.212+0000 7f9ef4125640 1 -- 172.21.15.1:0/2989273281 >> 172.21.15.1:0/2989273281 conn(0x7f9eec0c3cc0 msgr2=0x7f9eec0c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:26.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.213+0000 7f9ef4125640 1 -- 172.21.15.1:0/2989273281 shutdown_connections 2024-03-06T07:33:26.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.213+0000 7f9ef4125640 1 -- 172.21.15.1:0/2989273281 wait complete. 2024-03-06T07:33:26.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.214+0000 7f9ef4125640 1 Processor -- start 2024-03-06T07:33:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.214+0000 7f9ef4125640 1 -- start start 2024-03-06T07:33:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.214+0000 7f9ef4125640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eec06f640 0x7f9eec0aad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.214+0000 7f9ef4125640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eec0c05f0 0x7f9eec14d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.214+0000 7f9ef4125640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eec0c14c0 0x7f9eec14cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.214+0000 7f9ef4125640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec0c1040 con 0x7f9eec0c05f0 2024-03-06T07:33:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.214+0000 7f9ef4125640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec0c3770 con 0x7f9eec06f640 2024-03-06T07:33:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.214+0000 7f9ef4125640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec14f460 con 0x7f9eec0c14c0 2024-03-06T07:33:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.214+0000 7f9ef16c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eec0c05f0 0x7f9eec14d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.214+0000 7f9ef16c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eec0c05f0 0x7f9eec14d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40622/0 (socket says 172.21.15.1:40622) 2024-03-06T07:33:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.214+0000 7f9ef16c0640 1 -- 172.21.15.1:0/231495423 learned_addr learned my addr 172.21.15.1:0/231495423 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:26.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.215+0000 7f9ef1ec1640 1 --2- 172.21.15.1:0/231495423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eec06f640 0x7f9eec0aad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.215+0000 7f9ef26c2640 1 --2- 172.21.15.1:0/231495423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eec0c14c0 0x7f9eec14cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.215+0000 7f9ef16c0640 1 -- 172.21.15.1:0/231495423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eec0c14c0 msgr2=0x7f9eec14cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.215+0000 7f9ef16c0640 1 --2- 172.21.15.1:0/231495423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eec0c14c0 0x7f9eec14cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.215+0000 7f9ef16c0640 1 -- 172.21.15.1:0/231495423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eec06f640 msgr2=0x7f9eec0aad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.215+0000 7f9ef16c0640 1 --2- 172.21.15.1:0/231495423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eec06f640 0x7f9eec0aad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.215+0000 7f9ef16c0640 1 -- 172.21.15.1:0/231495423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec154340 con 0x7f9eec0c05f0 2024-03-06T07:33:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.215+0000 7f9ef26c2640 1 --2- 172.21.15.1:0/231495423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eec0c14c0 0x7f9eec14cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.215+0000 7f9ef16c0640 1 --2- 172.21.15.1:0/231495423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eec0c05f0 0x7f9eec14d520 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00c820 tx=0x7f9edc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.215+0000 7f9ee27fc640 1 -- 172.21.15.1:0/231495423 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc015070 con 0x7f9eec0c05f0 2024-03-06T07:33:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.215+0000 7f9ee27fc640 1 -- 172.21.15.1:0/231495423 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9edc007a30 con 0x7f9eec0c05f0 2024-03-06T07:33:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.215+0000 7f9ef4125640 1 -- 172.21.15.1:0/231495423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eec0aa130 con 0x7f9eec0c05f0 2024-03-06T07:33:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.215+0000 7f9ee27fc640 1 -- 172.21.15.1:0/231495423 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc007e20 con 0x7f9eec0c05f0 2024-03-06T07:33:26.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.215+0000 7f9ef4125640 1 -- 172.21.15.1:0/231495423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eec0aa6f0 con 0x7f9eec0c05f0 2024-03-06T07:33:26.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.216+0000 7f9ef4125640 1 -- 172.21.15.1:0/231495423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eec067fd0 con 0x7f9eec0c05f0 2024-03-06T07:33:26.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.217+0000 7f9ee27fc640 1 -- 172.21.15.1:0/231495423 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9edc018070 con 0x7f9eec0c05f0 2024-03-06T07:33:26.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.217+0000 7f9ee27fc640 1 --2- 172.21.15.1:0/231495423 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ed006b500 0x7f9ed006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.217+0000 7f9ee27fc640 1 -- 172.21.15.1:0/231495423 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f9edc08a0a0 con 0x7f9eec0c05f0 2024-03-06T07:33:26.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.217+0000 7f9ef1ec1640 1 --2- 172.21.15.1:0/231495423 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ed006b500 0x7f9ed006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.218+0000 7f9ef1ec1640 1 --2- 172.21.15.1:0/231495423 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ed006b500 0x7f9ed006d9c0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8036000 tx=0x7f9ee803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.221+0000 7f9ee27fc640 1 -- 172.21.15.1:0/231495423 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9edc058f00 con 0x7f9eec0c05f0 2024-03-06T07:33:26.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.472+0000 7f9ef4125640 1 -- 172.21.15.1:0/231495423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]} v 0) v1 -- 0x7f9eec0ac750 con 0x7f9eec0c05f0 2024-03-06T07:33:26.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.765+0000 7f9ee27fc640 1 -- 172.21.15.1:0/231495423 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]}]=0 v708) v1 ==== 139+0+0 (secure 0 0 0) 0x7f9edc058800 con 0x7f9eec0c05f0 2024-03-06T07:33:26.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.767+0000 7f9ef4125640 1 -- 172.21.15.1:0/231495423 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ed006b500 msgr2=0x7f9ed006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.767+0000 7f9ef4125640 1 --2- 172.21.15.1:0/231495423 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ed006b500 0x7f9ed006d9c0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8036000 tx=0x7f9ee803c040 comp rx=0 tx=0).stop 2024-03-06T07:33:26.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.767+0000 7f9ef4125640 1 -- 172.21.15.1:0/231495423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eec0c05f0 msgr2=0x7f9eec14d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.767+0000 7f9ef4125640 1 --2- 172.21.15.1:0/231495423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eec0c05f0 0x7f9eec14d520 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00c820 tx=0x7f9edc00cce0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.767+0000 7f9ef4125640 1 -- 172.21.15.1:0/231495423 shutdown_connections 2024-03-06T07:33:26.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.767+0000 7f9ef4125640 1 --2- 172.21.15.1:0/231495423 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f9ed006b500 0x7f9ed006d9c0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.767+0000 7f9ef4125640 1 --2- 172.21.15.1:0/231495423 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eec0c14c0 0x7f9eec14cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.767+0000 7f9ef4125640 1 --2- 172.21.15.1:0/231495423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eec0c05f0 0x7f9eec14d520 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.767+0000 7f9ef4125640 1 --2- 172.21.15.1:0/231495423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eec06f640 0x7f9eec0aad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.767+0000 7f9ef4125640 1 -- 172.21.15.1:0/231495423 >> 172.21.15.1:0/231495423 conn(0x7f9eec0c3cc0 msgr2=0x7f9eec06e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:26.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.767+0000 7f9ef4125640 1 -- 172.21.15.1:0/231495423 shutdown_connections 2024-03-06T07:33:26.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:26.767+0000 7f9ef4125640 1 -- 172.21.15.1:0/231495423 wait complete. 2024-03-06T07:33:26.782 INFO:tasks.cephfs_test_runner:That marking a FS down twice does not wipe old_max_mds. ... ERROR 2024-03-06T07:33:26.782 INFO:tasks.cephfs_test_runner: 2024-03-06T07:33:26.782 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:33:26.782 INFO:tasks.cephfs_test_runner:ERROR: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-03-06T07:33:26.782 INFO:tasks.cephfs_test_runner:That marking a FS down twice does not wipe old_max_mds. 2024-03-06T07:33:26.782 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:33:26.782 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:33:26.782 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_failover.py", line 274, in test_down_twice 2024-03-06T07:33:26.782 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons() 2024-03-06T07:33:26.782 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:33:26.783 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:33:26.783 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:33:26.783 INFO:tasks.cephfs_test_runner: 2024-03-06T07:33:26.783 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:33:26.783 INFO:tasks.cephfs_test_runner:Ran 12 tests in 746.084s 2024-03-06T07:33:26.783 INFO:tasks.cephfs_test_runner: 2024-03-06T07:33:26.783 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-06T07:33:26.783 INFO:tasks.cephfs_test_runner: 2024-03-06T07:33:26.783 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:33:26.783 INFO:tasks.cephfs_test_runner:ERROR: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-03-06T07:33:26.783 INFO:tasks.cephfs_test_runner:That marking a FS down twice does not wipe old_max_mds. 2024-03-06T07:33:26.783 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:33:26.783 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:33:26.783 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_failover.py", line 274, in test_down_twice 2024-03-06T07:33:26.784 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons() 2024-03-06T07:33:26.784 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:33:26.784 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:33:26.784 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:33:26.784 INFO:tasks.cephfs_test_runner: 2024-03-06T07:33:26.784 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-03-06T07:33:26.989 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=22da85f24f3d4c0b90aefe83b0674e7f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-03-06T07:33:26.990 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-06T07:33:26.999 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-06T07:33:27.025 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-06T07:33:27.026 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-06T07:33:27.026 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:27.047 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17910 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:33:27.047 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-06T07:33:27.047 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:27.066 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17878 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:33:27.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:27.067 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-06T07:33:27.103 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:33:27.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:27.104 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:33:27.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:27.174 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link delete ceph-brx 2024-03-06T07:33:27.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:27.264 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-03-06T07:33:27.282 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:33:27.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:27.283 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:33:27.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:27.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-03-06T07:33:27.467 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-06T07:33:27.475 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:33:27.476 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:33:27.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.583+0000 7f342408f640 1 Processor -- start 2024-03-06T07:33:27.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.583+0000 7f342408f640 1 -- start start 2024-03-06T07:33:27.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.584+0000 7f342408f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f341c152b70 0x7f341c0c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.584+0000 7f342408f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341c151dd0 con 0x7f341c072bd0 2024-03-06T07:33:27.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.584+0000 7f342408f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341c156190 con 0x7f341c0c2b30 2024-03-06T07:33:27.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.584+0000 7f342408f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341c0c3f40 con 0x7f341c152b70 2024-03-06T07:33:27.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.584+0000 7f342162a640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f341c072bd0 0x7f341c0c32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37074/0 (socket says 172.21.15.1:37074) 2024-03-06T07:33:27.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.584+0000 7f342162a640 1 -- 172.21.15.1:0/1962357442 learned_addr learned my addr 172.21.15.1:0/1962357442 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:27.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.584+0000 7f3420e29640 1 -- 172.21.15.1:0/1962357442 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624468641 0 0) 0x7f341c151dd0 con 0x7f341c072bd0 2024-03-06T07:33:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.584+0000 7f3420e29640 1 -- 172.21.15.1:0/1962357442 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3404005730 con 0x7f341c072bd0 2024-03-06T07:33:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.584+0000 7f342262c640 1 --2- 172.21.15.1:0/1962357442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f341c152b70 0x7f341c0c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.584+0000 7f3420e29640 1 -- 172.21.15.1:0/1962357442 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 777917382 0 0) 0x7f3404005730 con 0x7f341c072bd0 2024-03-06T07:33:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.584+0000 7f3420e29640 1 -- 172.21.15.1:0/1962357442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f341c152b70 msgr2=0x7f341c0c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.584+0000 7f3420e29640 1 --2- 172.21.15.1:0/1962357442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f341c152b70 0x7f341c0c3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.584+0000 7f3420e29640 1 -- 172.21.15.1:0/1962357442 >> v1:172.21.15.98:6789/0 conn(0x7f341c0c2b30 legacy=0x7f341c0724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.585+0000 7f3420e29640 1 -- 172.21.15.1:0/1962357442 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341c152090 con 0x7f341c072bd0 2024-03-06T07:33:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.585+0000 7f3420e29640 1 -- 172.21.15.1:0/1962357442 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f3410001670 con 0x7f341c072bd0 2024-03-06T07:33:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.585+0000 7f3420e29640 1 -- 172.21.15.1:0/1962357442 >> v1:172.21.15.1:6789/0 conn(0x7f341c072bd0 legacy=0x7f341c0c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.585+0000 7f3420e29640 1 --2- 172.21.15.1:0/1962357442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f341c152b70 0x7f3404005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.585+0000 7f3420e29640 1 --2- 172.21.15.1:0/1962357442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3404006170 0x7f3404006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.585+0000 7f3420e29640 1 --2- 172.21.15.1:0/1962357442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3404006f50 0x7f34040073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.585+0000 7f3420e29640 1 -- 172.21.15.1:0/1962357442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c0c3f40 con 0x7f3404006f50 2024-03-06T07:33:27.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.585+0000 7f3420e29640 1 -- 172.21.15.1:0/1962357442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340400dcb0 con 0x7f3404006170 2024-03-06T07:33:27.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.585+0000 7f3420e29640 1 -- 172.21.15.1:0/1962357442 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f340400deb0 con 0x7f341c152b70 2024-03-06T07:33:27.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.585+0000 7f342262c640 1 --2- 172.21.15.1:0/1962357442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3404006f50 0x7f34040073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.585+0000 7f3421e2b640 1 --2- 172.21.15.1:0/1962357442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f341c152b70 0x7f3404005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.585+0000 7f342162a640 1 --2- 172.21.15.1:0/1962357442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3404006170 0x7f3404006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.585+0000 7f342262c640 1 -- 172.21.15.1:0/1962357442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f341c152b70 msgr2=0x7f3404005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:27.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.585+0000 7f342262c640 1 --2- 172.21.15.1:0/1962357442 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f341c152b70 0x7f3404005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.585+0000 7f342262c640 1 -- 172.21.15.1:0/1962357442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3404006170 msgr2=0x7f3404006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:27.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.585+0000 7f342262c640 1 --2- 172.21.15.1:0/1962357442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3404006170 0x7f3404006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.585+0000 7f342262c640 1 -- 172.21.15.1:0/1962357442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3404005730 con 0x7f3404006f50 2024-03-06T07:33:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.586+0000 7f342262c640 1 --2- 172.21.15.1:0/1962357442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3404006f50 0x7f34040073d0 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f341c06d600 tx=0x7f3404007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c36505b32d2ce03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.586+0000 7f3420e29640 1 -- 172.21.15.1:0/1962357442 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c039070 con 0x7f3404006f50 2024-03-06T07:33:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.586+0000 7f3420e29640 1 -- 172.21.15.1:0/1962357442 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340c02c660 con 0x7f3404006f50 2024-03-06T07:33:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.586+0000 7f3420e29640 1 -- 172.21.15.1:0/1962357442 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c009690 con 0x7f3404006f50 2024-03-06T07:33:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.586+0000 7f342408f640 1 -- 172.21.15.1:0/1962357442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3404006f50 msgr2=0x7f34040073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.586+0000 7f342408f640 1 --2- 172.21.15.1:0/1962357442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3404006f50 0x7f34040073d0 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f341c06d600 tx=0x7f3404007a40 comp rx=0 tx=0).stop 2024-03-06T07:33:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.586+0000 7f342262c640 1 -- 172.21.15.1:0/1962357442 reap_dead start 2024-03-06T07:33:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.587+0000 7f342408f640 1 -- 172.21.15.1:0/1962357442 shutdown_connections 2024-03-06T07:33:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.587+0000 7f342408f640 1 -- 172.21.15.1:0/1962357442 >> 172.21.15.1:0/1962357442 conn(0x7f341c0c6860 msgr2=0x7f341c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.587+0000 7f342408f640 1 -- 172.21.15.1:0/1962357442 shutdown_connections 2024-03-06T07:33:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.587+0000 7f342408f640 1 -- 172.21.15.1:0/1962357442 wait complete. 2024-03-06T07:33:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.588+0000 7f342408f640 1 Processor -- start 2024-03-06T07:33:27.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.588+0000 7f342408f640 1 -- start start 2024-03-06T07:33:27.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.588+0000 7f342408f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3404006f50 0x7f341c0c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.588+0000 7f342408f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f341c14d860 0x7f341c1e6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.588+0000 7f342408f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f341c147f90 0x7f341c1e6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.588+0000 7f342408f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c155d80 con 0x7f341c147f90 2024-03-06T07:33:27.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.588+0000 7f342408f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c1526e0 con 0x7f3404006f50 2024-03-06T07:33:27.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.588+0000 7f342408f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341c155f00 con 0x7f341c14d860 2024-03-06T07:33:27.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.588+0000 7f342262c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f341c147f90 0x7f341c1e6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.589+0000 7f342262c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f341c147f90 0x7f341c1e6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40640/0 (socket says 172.21.15.1:40640) 2024-03-06T07:33:27.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.589+0000 7f342262c640 1 -- 172.21.15.1:0/1959734924 learned_addr learned my addr 172.21.15.1:0/1959734924 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:27.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.589+0000 7f3421e2b640 1 --2- 172.21.15.1:0/1959734924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3404006f50 0x7f341c0c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.589+0000 7f342162a640 1 --2- 172.21.15.1:0/1959734924 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f341c14d860 0x7f341c1e6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.589+0000 7f342262c640 1 -- 172.21.15.1:0/1959734924 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f341c14d860 msgr2=0x7f341c1e6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.589+0000 7f342262c640 1 --2- 172.21.15.1:0/1959734924 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f341c14d860 0x7f341c1e6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.589+0000 7f342262c640 1 -- 172.21.15.1:0/1959734924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3404006f50 msgr2=0x7f341c0c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.589+0000 7f342262c640 1 --2- 172.21.15.1:0/1959734924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3404006f50 0x7f341c0c4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.589+0000 7f342262c640 1 -- 172.21.15.1:0/1959734924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341c148c00 con 0x7f341c147f90 2024-03-06T07:33:27.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.589+0000 7f342262c640 1 --2- 172.21.15.1:0/1959734924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f341c147f90 0x7f341c1e6e60 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7f340c009b80 tx=0x7f340c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.589+0000 7f341a7fc640 1 -- 172.21.15.1:0/1959734924 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c039030 con 0x7f341c147f90 2024-03-06T07:33:27.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.589+0000 7f341a7fc640 1 -- 172.21.15.1:0/1959734924 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340c03a040 con 0x7f341c147f90 2024-03-06T07:33:27.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.589+0000 7f342408f640 1 -- 172.21.15.1:0/1959734924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341c1ea0e0 con 0x7f341c147f90 2024-03-06T07:33:27.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.589+0000 7f341a7fc640 1 -- 172.21.15.1:0/1959734924 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c009690 con 0x7f341c147f90 2024-03-06T07:33:27.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.589+0000 7f342408f640 1 -- 172.21.15.1:0/1959734924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f341c1ea6a0 con 0x7f341c147f90 2024-03-06T07:33:27.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.590+0000 7f342408f640 1 -- 172.21.15.1:0/1959734924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f341c1ea8a0 con 0x7f341c147f90 2024-03-06T07:33:27.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.591+0000 7f341a7fc640 1 -- 172.21.15.1:0/1959734924 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f340c045020 con 0x7f341c147f90 2024-03-06T07:33:27.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.591+0000 7f341a7fc640 1 --2- 172.21.15.1:0/1959734924 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f340806b500 0x7f340806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.591+0000 7f341a7fc640 1 -- 172.21.15.1:0/1959734924 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f340c0aaff0 con 0x7f341c147f90 2024-03-06T07:33:27.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.591+0000 7f3421e2b640 1 --2- 172.21.15.1:0/1959734924 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f340806b500 0x7f340806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.592+0000 7f3421e2b640 1 --2- 172.21.15.1:0/1959734924 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f340806b500 0x7f340806d9c0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f341c1e7a50 tx=0x7f3414009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.595+0000 7f341a7fc640 1 -- 172.21.15.1:0/1959734924 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f341c1ea8a0 con 0x7f341c147f90 2024-03-06T07:33:27.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.774+0000 7f342408f640 1 -- 172.21.15.1:0/1959734924 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f341c067fd0 con 0x7f340806b500 2024-03-06T07:33:27.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.775+0000 7f341a7fc640 1 -- 172.21.15.1:0/1959734924 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13531 (secure 0 0 0) 0x7f341c067fd0 con 0x7f340806b500 2024-03-06T07:33:27.778 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:27.778 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:33:27.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.777+0000 7f342408f640 1 -- 172.21.15.1:0/1959734924 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f340806b500 msgr2=0x7f340806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.777+0000 7f342408f640 1 --2- 172.21.15.1:0/1959734924 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f340806b500 0x7f340806d9c0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f341c1e7a50 tx=0x7f3414009040 comp rx=0 tx=0).stop 2024-03-06T07:33:27.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.777+0000 7f342408f640 1 -- 172.21.15.1:0/1959734924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f341c147f90 msgr2=0x7f341c1e6e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.777+0000 7f342408f640 1 --2- 172.21.15.1:0/1959734924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f341c147f90 0x7f341c1e6e60 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7f340c009b80 tx=0x7f340c004730 comp rx=0 tx=0).stop 2024-03-06T07:33:27.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.777+0000 7f342408f640 1 -- 172.21.15.1:0/1959734924 shutdown_connections 2024-03-06T07:33:27.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.777+0000 7f342408f640 1 --2- 172.21.15.1:0/1959734924 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f340806b500 0x7f340806d9c0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.777+0000 7f342408f640 1 --2- 172.21.15.1:0/1959734924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f341c147f90 0x7f341c1e6e60 unknown :-1 s=CLOSED pgs=2715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.777+0000 7f342408f640 1 --2- 172.21.15.1:0/1959734924 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f341c14d860 0x7f341c1e6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.777+0000 7f342408f640 1 --2- 172.21.15.1:0/1959734924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3404006f50 0x7f341c0c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.777+0000 7f342408f640 1 -- 172.21.15.1:0/1959734924 >> 172.21.15.1:0/1959734924 conn(0x7f341c0c6860 msgr2=0x7f341c1531c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:27.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.778+0000 7f342408f640 1 -- 172.21.15.1:0/1959734924 shutdown_connections 2024-03-06T07:33:27.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.778+0000 7f342408f640 1 -- 172.21.15.1:0/1959734924 wait complete. 2024-03-06T07:33:27.791 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":606,"stamp":"2024-03-06T07:33:26.884979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":287596,"kb_used_data":21516,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754687124,"statfs":{"total":773094113280,"available":772799614976,"internally_reserved":0,"allocated":22032384,"data_stored":11798400,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3816,"num_objects":-41,"num_object_clones":0,"num_object_copies":-82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-41,"num_whiteouts":0,"num_read":-73,"num_read_kb":-79,"num_write":-85,"num_write_kb":-26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001877"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":549,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-03-06T07:33:24.764069+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:33:24.764069+0000","last_peered":"2024-03-06T07:33:24.764069+0000","last_clean":"2024-03-06T07:33:24.764069+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:33:24.764069+0000","last_undegraded":"2024-03-06T07:33:24.764069+0000","last_fullsized":"2024-03-06T07:33:24.764069+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575009,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33912,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337928,"statfs":{"total":96636764160,"available":96602038272,"internally_reserved":0,"allocated":2797568,"data_stored":1913533,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575008,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33836,"kb_used_data":2656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338004,"statfs":{"total":96636764160,"available":96602116096,"internally_reserved":0,"allocated":2719744,"data_stored":1321973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575008,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33852,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337988,"statfs":{"total":96636764160,"available":96602099712,"internally_reserved":0,"allocated":2736128,"data_stored":1328935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575009,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33856,"kb_used_data":2676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337984,"statfs":{"total":96636764160,"available":96602095616,"internally_reserved":0,"allocated":2740224,"data_stored":1330455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575008,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38016,"kb_used_data":2676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2740224,"data_stored":1329605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575008,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38028,"kb_used_data":2688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333812,"statfs":{"total":96636764160,"available":96597823488,"internally_reserved":0,"allocated":2752512,"data_stored":1331149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38032,"kb_used_data":2692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333808,"statfs":{"total":96636764160,"available":96597819392,"internally_reserved":0,"allocated":2756608,"data_stored":1330283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575009,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38064,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333776,"statfs":{"total":96636764160,"available":96597786624,"internally_reserved":0,"allocated":2789376,"data_stored":1912467,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:33:27.792 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:33:27.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.894+0000 7fa170217640 1 Processor -- start 2024-03-06T07:33:27.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.894+0000 7fa170217640 1 -- start start 2024-03-06T07:33:27.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.894+0000 7fa170217640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa1680c3eb0 0x7fa1680c7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa170217640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1680c71c0 con 0x7fa1680c7940 2024-03-06T07:33:27.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa170217640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1680c4e50 con 0x7fa16806d060 2024-03-06T07:33:27.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa170217640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1680c59b0 con 0x7fa1680c3eb0 2024-03-06T07:33:27.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16e7b4640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa1680c7940 0x7fa16815fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37082/0 (socket says 172.21.15.1:37082) 2024-03-06T07:33:27.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16e7b4640 1 -- 172.21.15.1:0/3664514643 learned_addr learned my addr 172.21.15.1:0/3664514643 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:27.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16dfb3640 1 --2- 172.21.15.1:0/3664514643 >> v2:172.21.15.98:3300/0 conn(0x7fa1680c3eb0 0x7fa1680c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16cfb1640 1 -- 172.21.15.1:0/3664514643 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265051234 0 0) 0x7fa1680c71c0 con 0x7fa1680c7940 2024-03-06T07:33:27.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16cfb1640 1 -- 172.21.15.1:0/3664514643 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa154003520 con 0x7fa1680c7940 2024-03-06T07:33:27.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16cfb1640 1 -- 172.21.15.1:0/3664514643 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1561382660 0 0) 0x7fa154003520 con 0x7fa1680c7940 2024-03-06T07:33:27.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16cfb1640 1 -- 172.21.15.1:0/3664514643 >> v2:172.21.15.98:3300/0 conn(0x7fa1680c3eb0 msgr2=0x7fa1680c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:27.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16cfb1640 1 --2- 172.21.15.1:0/3664514643 >> v2:172.21.15.98:3300/0 conn(0x7fa1680c3eb0 0x7fa1680c7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16cfb1640 1 -- 172.21.15.1:0/3664514643 >> v1:172.21.15.98:6789/0 conn(0x7fa16806d060 legacy=0x7fa1681558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16cfb1640 1 -- 172.21.15.1:0/3664514643 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1680c6ee0 con 0x7fa1680c7940 2024-03-06T07:33:27.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16cfb1640 1 -- 172.21.15.1:0/3664514643 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7fa1600019f0 con 0x7fa1680c7940 2024-03-06T07:33:27.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16dfb3640 1 --2- 172.21.15.1:0/3664514643 >> v2:172.21.15.98:3300/0 conn(0x7fa1680c3eb0 0x7fa1680c7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:27.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16cfb1640 1 -- 172.21.15.1:0/3664514643 >> v1:172.21.15.1:6789/0 conn(0x7fa1680c7940 legacy=0x7fa16815fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16cfb1640 1 --2- 172.21.15.1:0/3664514643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1540039b0 0x7fa154003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16cfb1640 1 --2- 172.21.15.1:0/3664514643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa154004800 0x7fa154004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16cfb1640 1 --2- 172.21.15.1:0/3664514643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1540052e0 0x7fa15400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16cfb1640 1 -- 172.21.15.1:0/3664514643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1680c59b0 con 0x7fa1540052e0 2024-03-06T07:33:27.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16cfb1640 1 -- 172.21.15.1:0/3664514643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15400c1b0 con 0x7fa154004800 2024-03-06T07:33:27.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16cfb1640 1 -- 172.21.15.1:0/3664514643 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15400c3b0 con 0x7fa1540039b0 2024-03-06T07:33:27.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.895+0000 7fa16e7b4640 1 --2- 172.21.15.1:0/3664514643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1540052e0 0x7fa15400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.896+0000 7fa16dfb3640 1 --2- 172.21.15.1:0/3664514643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1540039b0 0x7fa154003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.896+0000 7fa16d7b2640 1 --2- 172.21.15.1:0/3664514643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa154004800 0x7fa154004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.896+0000 7fa16e7b4640 1 -- 172.21.15.1:0/3664514643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1540039b0 msgr2=0x7fa154003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:27.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.896+0000 7fa16e7b4640 1 --2- 172.21.15.1:0/3664514643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa1540039b0 0x7fa154003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.896+0000 7fa16e7b4640 1 -- 172.21.15.1:0/3664514643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa154004800 msgr2=0x7fa154004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:27.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.896+0000 7fa16e7b4640 1 --2- 172.21.15.1:0/3664514643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa154004800 0x7fa154004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.896+0000 7fa16e7b4640 1 -- 172.21.15.1:0/3664514643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa154003520 con 0x7fa1540052e0 2024-03-06T07:33:27.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.896+0000 7fa16d7b2640 1 --2- 172.21.15.1:0/3664514643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa154004800 0x7fa154004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:27.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.896+0000 7fa16e7b4640 1 -- 172.21.15.1:0/3664514643 reap_dead start 2024-03-06T07:33:27.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.896+0000 7fa16e7b4640 1 --2- 172.21.15.1:0/3664514643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1540052e0 0x7fa15400b9c0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7fa16000a830 tx=0x7fa154005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff60c4dc56d33a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.897+0000 7fa16cfb1640 1 -- 172.21.15.1:0/3664514643 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16000a480 con 0x7fa1540052e0 2024-03-06T07:33:27.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.897+0000 7fa16cfb1640 1 -- 172.21.15.1:0/3664514643 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa160034630 con 0x7fa1540052e0 2024-03-06T07:33:27.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.897+0000 7fa16cfb1640 1 -- 172.21.15.1:0/3664514643 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa160034a00 con 0x7fa1540052e0 2024-03-06T07:33:27.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.897+0000 7fa170217640 1 -- 172.21.15.1:0/3664514643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1540052e0 msgr2=0x7fa15400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.897+0000 7fa170217640 1 --2- 172.21.15.1:0/3664514643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1540052e0 0x7fa15400b9c0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7fa16000a830 tx=0x7fa154005830 comp rx=0 tx=0).stop 2024-03-06T07:33:27.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.897+0000 7fa170217640 1 -- 172.21.15.1:0/3664514643 shutdown_connections 2024-03-06T07:33:27.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.897+0000 7fa170217640 1 --2- 172.21.15.1:0/3664514643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1540052e0 0x7fa15400b9c0 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.897+0000 7fa170217640 1 -- 172.21.15.1:0/3664514643 >> 172.21.15.1:0/3664514643 conn(0x7fa168066d00 msgr2=0x7fa168067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:27.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.897+0000 7fa170217640 1 -- 172.21.15.1:0/3664514643 shutdown_connections 2024-03-06T07:33:27.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.897+0000 7fa170217640 1 -- 172.21.15.1:0/3664514643 wait complete. 2024-03-06T07:33:27.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.898+0000 7fa170217640 1 Processor -- start 2024-03-06T07:33:27.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.898+0000 7fa170217640 1 -- start start 2024-03-06T07:33:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.899+0000 7fa170217640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1540052e0 0x7fa168148950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.899+0000 7fa170217640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa168159030 0x7fa1681472c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.899+0000 7fa170217640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa168147800 0x7fa1681563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.899+0000 7fa170217640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa168155420 con 0x7fa1540052e0 2024-03-06T07:33:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.899+0000 7fa170217640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1680c4c00 con 0x7fa168147800 2024-03-06T07:33:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.899+0000 7fa170217640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1680af640 con 0x7fa168159030 2024-03-06T07:33:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.899+0000 7fa16dfb3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1540052e0 0x7fa168148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.899+0000 7fa16dfb3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1540052e0 0x7fa168148950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40650/0 (socket says 172.21.15.1:40650) 2024-03-06T07:33:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.899+0000 7fa16dfb3640 1 -- 172.21.15.1:0/1752438952 learned_addr learned my addr 172.21.15.1:0/1752438952 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.899+0000 7fa16e7b4640 1 --2- 172.21.15.1:0/1752438952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa168147800 0x7fa1681563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.899+0000 7fa16d7b2640 1 --2- 172.21.15.1:0/1752438952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa168159030 0x7fa1681472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.899+0000 7fa16dfb3640 1 -- 172.21.15.1:0/1752438952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa168159030 msgr2=0x7fa1681472c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.899+0000 7fa16dfb3640 1 --2- 172.21.15.1:0/1752438952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa168159030 0x7fa1681472c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.899+0000 7fa16dfb3640 1 -- 172.21.15.1:0/1752438952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa168147800 msgr2=0x7fa1681563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.899+0000 7fa16dfb3640 1 --2- 172.21.15.1:0/1752438952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa168147800 0x7fa1681563c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.899+0000 7fa16dfb3640 1 -- 172.21.15.1:0/1752438952 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1681ee250 con 0x7fa1540052e0 2024-03-06T07:33:27.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.899+0000 7fa16e7b4640 1 --2- 172.21.15.1:0/1752438952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa168147800 0x7fa1681563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:27.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.900+0000 7fa16dfb3640 1 --2- 172.21.15.1:0/1752438952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1540052e0 0x7fa168148950 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7fa164011af0 tx=0x7fa16400f240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.900+0000 7fa15e7fc640 1 -- 172.21.15.1:0/1752438952 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16400fc80 con 0x7fa1540052e0 2024-03-06T07:33:27.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.900+0000 7fa15e7fc640 1 -- 172.21.15.1:0/1752438952 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16400fe20 con 0x7fa1540052e0 2024-03-06T07:33:27.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.900+0000 7fa15e7fc640 1 -- 172.21.15.1:0/1752438952 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa164004260 con 0x7fa1540052e0 2024-03-06T07:33:27.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.900+0000 7fa170217640 1 -- 172.21.15.1:0/1752438952 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa168148e90 con 0x7fa1540052e0 2024-03-06T07:33:27.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.900+0000 7fa170217640 1 -- 172.21.15.1:0/1752438952 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1681ee4f0 con 0x7fa1540052e0 2024-03-06T07:33:27.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.901+0000 7fa170217640 1 -- 172.21.15.1:0/1752438952 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa168067fd0 con 0x7fa1540052e0 2024-03-06T07:33:27.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.901+0000 7fa15e7fc640 1 -- 172.21.15.1:0/1752438952 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa164012040 con 0x7fa1540052e0 2024-03-06T07:33:27.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.902+0000 7fa15e7fc640 1 --2- 172.21.15.1:0/1752438952 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa14c06b4d0 0x7fa14c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.902+0000 7fa15e7fc640 1 -- 172.21.15.1:0/1752438952 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fa16408a5f0 con 0x7fa1540052e0 2024-03-06T07:33:27.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.902+0000 7fa16d7b2640 1 --2- 172.21.15.1:0/1752438952 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa14c06b4d0 0x7fa14c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.902+0000 7fa16d7b2640 1 --2- 172.21.15.1:0/1752438952 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa14c06b4d0 0x7fa14c06d990 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fa158000c00 tx=0x7fa158009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:27.906+0000 7fa15e7fc640 1 -- 172.21.15.1:0/1752438952 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa164019030 con 0x7fa1540052e0 2024-03-06T07:33:28.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.084+0000 7fa170217640 1 -- 172.21.15.1:0/1752438952 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa1680c7720 con 0x7fa14c06b4d0 2024-03-06T07:33:28.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.085+0000 7fa15e7fc640 1 -- 172.21.15.1:0/1752438952 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13531 (secure 0 0 0) 0x7fa1680c7720 con 0x7fa14c06b4d0 2024-03-06T07:33:28.087 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:33:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.087+0000 7fa170217640 1 -- 172.21.15.1:0/1752438952 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa14c06b4d0 msgr2=0x7fa14c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.087+0000 7fa170217640 1 --2- 172.21.15.1:0/1752438952 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa14c06b4d0 0x7fa14c06d990 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fa158000c00 tx=0x7fa158009040 comp rx=0 tx=0).stop 2024-03-06T07:33:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.087+0000 7fa170217640 1 -- 172.21.15.1:0/1752438952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1540052e0 msgr2=0x7fa168148950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.087+0000 7fa170217640 1 --2- 172.21.15.1:0/1752438952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1540052e0 0x7fa168148950 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7fa164011af0 tx=0x7fa16400f240 comp rx=0 tx=0).stop 2024-03-06T07:33:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.087+0000 7fa170217640 1 -- 172.21.15.1:0/1752438952 shutdown_connections 2024-03-06T07:33:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.087+0000 7fa170217640 1 --2- 172.21.15.1:0/1752438952 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa14c06b4d0 0x7fa14c06d990 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.087+0000 7fa170217640 1 --2- 172.21.15.1:0/1752438952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa168147800 0x7fa1681563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.087+0000 7fa170217640 1 --2- 172.21.15.1:0/1752438952 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa168159030 0x7fa1681472c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.087+0000 7fa170217640 1 --2- 172.21.15.1:0/1752438952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1540052e0 0x7fa168148950 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.087+0000 7fa170217640 1 -- 172.21.15.1:0/1752438952 >> 172.21.15.1:0/1752438952 conn(0x7fa168066d00 msgr2=0x7fa168151220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:28.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.087+0000 7fa170217640 1 -- 172.21.15.1:0/1752438952 shutdown_connections 2024-03-06T07:33:28.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.087+0000 7fa170217640 1 -- 172.21.15.1:0/1752438952 wait complete. 2024-03-06T07:33:28.100 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":606,"stamp":"2024-03-06T07:33:26.884979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":287596,"kb_used_data":21516,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754687124,"statfs":{"total":773094113280,"available":772799614976,"internally_reserved":0,"allocated":22032384,"data_stored":11798400,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3816,"num_objects":-41,"num_object_clones":0,"num_object_copies":-82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-41,"num_whiteouts":0,"num_read":-73,"num_read_kb":-79,"num_write":-85,"num_write_kb":-26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001877"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":549,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-03-06T07:33:24.764069+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:33:24.764069+0000","last_peered":"2024-03-06T07:33:24.764069+0000","last_clean":"2024-03-06T07:33:24.764069+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:33:24.764069+0000","last_undegraded":"2024-03-06T07:33:24.764069+0000","last_fullsized":"2024-03-06T07:33:24.764069+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575009,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33912,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337928,"statfs":{"total":96636764160,"available":96602038272,"internally_reserved":0,"allocated":2797568,"data_stored":1913533,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575008,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33836,"kb_used_data":2656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338004,"statfs":{"total":96636764160,"available":96602116096,"internally_reserved":0,"allocated":2719744,"data_stored":1321973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575008,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33852,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337988,"statfs":{"total":96636764160,"available":96602099712,"internally_reserved":0,"allocated":2736128,"data_stored":1328935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575009,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33856,"kb_used_data":2676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337984,"statfs":{"total":96636764160,"available":96602095616,"internally_reserved":0,"allocated":2740224,"data_stored":1330455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575008,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38016,"kb_used_data":2676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2740224,"data_stored":1329605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575008,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38028,"kb_used_data":2688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333812,"statfs":{"total":96636764160,"available":96597823488,"internally_reserved":0,"allocated":2752512,"data_stored":1331149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38032,"kb_used_data":2692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333808,"statfs":{"total":96636764160,"available":96597819392,"internally_reserved":0,"allocated":2756608,"data_stored":1330283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575009,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38064,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333776,"statfs":{"total":96636764160,"available":96597786624,"internally_reserved":0,"allocated":2789376,"data_stored":1912467,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:33:28.101 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:33:28.101 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:33:28.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.202+0000 7f4d17b02640 1 Processor -- start 2024-03-06T07:33:28.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.203+0000 7f4d17b02640 1 -- start start 2024-03-06T07:33:28.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.203+0000 7f4d17b02640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4d100bf9d0 0x7f4d10155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.203+0000 7f4d17b02640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d100c2fc0 con 0x7f4d100c6b10 2024-03-06T07:33:28.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.203+0000 7f4d17b02640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d10158da0 con 0x7f4d10152650 2024-03-06T07:33:28.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.203+0000 7f4d17b02640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d10155800 con 0x7f4d100bf9d0 2024-03-06T07:33:28.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.203+0000 7f4d1609f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4d100bf9d0 0x7f4d10155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.203+0000 7f4d1509d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4d100c6b10 0x7f4d100c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37090/0 (socket says 172.21.15.1:37090) 2024-03-06T07:33:28.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.204+0000 7f4d1509d640 1 -- 172.21.15.1:0/3930795359 learned_addr learned my addr 172.21.15.1:0/3930795359 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:28.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.204+0000 7f4d1609f640 1 -- 172.21.15.1:0/3930795359 >> v1:172.21.15.98:6789/0 conn(0x7f4d10152650 legacy=0x7f4d10158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:28.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.204+0000 7f4d1609f640 1 -- 172.21.15.1:0/3930795359 >> v1:172.21.15.1:6789/0 conn(0x7f4d100c6b10 legacy=0x7f4d100c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.204+0000 7f4d1489c640 1 -- 172.21.15.1:0/3930795359 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636787766 0 0) 0x7f4d100c2fc0 con 0x7f4d100c6b10 2024-03-06T07:33:28.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.204+0000 7f4d1609f640 1 -- 172.21.15.1:0/3930795359 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d100c6690 con 0x7f4d100bf9d0 2024-03-06T07:33:28.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.205+0000 7f4d1609f640 1 --2- 172.21.15.1:0/3930795359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d100bf9d0 0x7f4d10155c90 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7f4d08009d90 tx=0x7f4d0802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef50d5d10acb8907 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.205+0000 7f4d1489c640 1 -- 172.21.15.1:0/3930795359 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d08039070 con 0x7f4d100bf9d0 2024-03-06T07:33:28.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.205+0000 7f4d1489c640 1 -- 172.21.15.1:0/3930795359 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0802cbd0 con 0x7f4d100bf9d0 2024-03-06T07:33:28.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.205+0000 7f4d1489c640 1 -- 172.21.15.1:0/3930795359 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d08034440 con 0x7f4d100bf9d0 2024-03-06T07:33:28.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.205+0000 7f4d17b02640 1 -- 172.21.15.1:0/3930795359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d100bf9d0 msgr2=0x7f4d10155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.205+0000 7f4d17b02640 1 --2- 172.21.15.1:0/3930795359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d100bf9d0 0x7f4d10155c90 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7f4d08009d90 tx=0x7f4d0802be20 comp rx=0 tx=0).stop 2024-03-06T07:33:28.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.205+0000 7f4d17b02640 1 -- 172.21.15.1:0/3930795359 shutdown_connections 2024-03-06T07:33:28.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.205+0000 7f4d17b02640 1 --2- 172.21.15.1:0/3930795359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d100bf9d0 0x7f4d10155c90 unknown :-1 s=CLOSED pgs=2720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.205+0000 7f4d17b02640 1 -- 172.21.15.1:0/3930795359 >> 172.21.15.1:0/3930795359 conn(0x7f4d10066ee0 msgr2=0x7f4d100672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:28.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.205+0000 7f4d17b02640 1 -- 172.21.15.1:0/3930795359 shutdown_connections 2024-03-06T07:33:28.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.205+0000 7f4d17b02640 1 -- 172.21.15.1:0/3930795359 wait complete. 2024-03-06T07:33:28.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.206+0000 7f4d17b02640 1 Processor -- start 2024-03-06T07:33:28.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.206+0000 7f4d17b02640 1 -- start start 2024-03-06T07:33:28.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.207+0000 7f4d17b02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d100bf9d0 0x7f4d10153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.207+0000 7f4d17b02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d10152650 0x7f4d101ee840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.207+0000 7f4d17b02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d101eed80 0x7f4d101473f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.207+0000 7f4d17b02640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d101551b0 con 0x7f4d10152650 2024-03-06T07:33:28.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.207+0000 7f4d17b02640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d100c6fa0 con 0x7f4d100bf9d0 2024-03-06T07:33:28.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.207+0000 7f4d17b02640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d10155490 con 0x7f4d101eed80 2024-03-06T07:33:28.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.207+0000 7f4d1509d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d10152650 0x7f4d101ee840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.207+0000 7f4d1509d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d10152650 0x7f4d101ee840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40670/0 (socket says 172.21.15.1:40670) 2024-03-06T07:33:28.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.207+0000 7f4d1509d640 1 -- 172.21.15.1:0/248972805 learned_addr learned my addr 172.21.15.1:0/248972805 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.207+0000 7f4d1589e640 1 --2- 172.21.15.1:0/248972805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d100bf9d0 0x7f4d10153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.207+0000 7f4d1609f640 1 --2- 172.21.15.1:0/248972805 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d101eed80 0x7f4d101473f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.207+0000 7f4d1509d640 1 -- 172.21.15.1:0/248972805 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d101eed80 msgr2=0x7f4d101473f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.207+0000 7f4d1509d640 1 --2- 172.21.15.1:0/248972805 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d101eed80 0x7f4d101473f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.207+0000 7f4d1509d640 1 -- 172.21.15.1:0/248972805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d100bf9d0 msgr2=0x7f4d10153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.207+0000 7f4d1509d640 1 --2- 172.21.15.1:0/248972805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d100bf9d0 0x7f4d10153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.207+0000 7f4d1509d640 1 -- 172.21.15.1:0/248972805 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d101ef470 con 0x7f4d10152650 2024-03-06T07:33:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.207+0000 7f4d1589e640 1 --2- 172.21.15.1:0/248972805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d100bf9d0 0x7f4d10153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:28.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.208+0000 7f4d1509d640 1 --2- 172.21.15.1:0/248972805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d10152650 0x7f4d101ee840 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000c4d0 tx=0x7f4d0000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.208+0000 7f4d067fc640 1 -- 172.21.15.1:0/248972805 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d00018070 con 0x7f4d10152650 2024-03-06T07:33:28.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.208+0000 7f4d067fc640 1 -- 172.21.15.1:0/248972805 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d00007500 con 0x7f4d10152650 2024-03-06T07:33:28.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.208+0000 7f4d17b02640 1 -- 172.21.15.1:0/248972805 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1014b710 con 0x7f4d10152650 2024-03-06T07:33:28.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.208+0000 7f4d067fc640 1 -- 172.21.15.1:0/248972805 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d00004a20 con 0x7f4d10152650 2024-03-06T07:33:28.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.208+0000 7f4d17b02640 1 -- 172.21.15.1:0/248972805 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d100ca4a0 con 0x7f4d10152650 2024-03-06T07:33:28.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.209+0000 7f4d17b02640 1 -- 172.21.15.1:0/248972805 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d10154b40 con 0x7f4d10152650 2024-03-06T07:33:28.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.209+0000 7f4d067fc640 1 -- 172.21.15.1:0/248972805 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d0001b070 con 0x7f4d10152650 2024-03-06T07:33:28.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.210+0000 7f4d067fc640 1 --2- 172.21.15.1:0/248972805 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cf406b4d0 0x7f4cf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.210+0000 7f4d067fc640 1 -- 172.21.15.1:0/248972805 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f4d000894a0 con 0x7f4d10152650 2024-03-06T07:33:28.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.210+0000 7f4d1589e640 1 --2- 172.21.15.1:0/248972805 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cf406b4d0 0x7f4cf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.210+0000 7f4d1589e640 1 --2- 172.21.15.1:0/248972805 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cf406b4d0 0x7f4cf406d990 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f4d101e8ea0 tx=0x7f4d0c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.214+0000 7f4d067fc640 1 -- 172.21.15.1:0/248972805 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d00058390 con 0x7f4d10152650 2024-03-06T07:33:28.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.391+0000 7f4d17b02640 1 -- 172.21.15.1:0/248972805 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4d10067fd0 con 0x7f4cf406b4d0 2024-03-06T07:33:28.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.393+0000 7f4d067fc640 1 -- 172.21.15.1:0/248972805 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13531 (secure 0 0 0) 0x7f4d10067fd0 con 0x7f4cf406b4d0 2024-03-06T07:33:28.395 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:28.395 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:33:28.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.394+0000 7f4d17b02640 1 -- 172.21.15.1:0/248972805 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cf406b4d0 msgr2=0x7f4cf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.394+0000 7f4d17b02640 1 --2- 172.21.15.1:0/248972805 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cf406b4d0 0x7f4cf406d990 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f4d101e8ea0 tx=0x7f4d0c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:28.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.394+0000 7f4d17b02640 1 -- 172.21.15.1:0/248972805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d10152650 msgr2=0x7f4d101ee840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.394+0000 7f4d17b02640 1 --2- 172.21.15.1:0/248972805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d10152650 0x7f4d101ee840 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000c4d0 tx=0x7f4d0000c990 comp rx=0 tx=0).stop 2024-03-06T07:33:28.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.394+0000 7f4d17b02640 1 -- 172.21.15.1:0/248972805 shutdown_connections 2024-03-06T07:33:28.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.394+0000 7f4d17b02640 1 --2- 172.21.15.1:0/248972805 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4cf406b4d0 0x7f4cf406d990 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.394+0000 7f4d17b02640 1 --2- 172.21.15.1:0/248972805 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d101eed80 0x7f4d101473f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.394+0000 7f4d17b02640 1 --2- 172.21.15.1:0/248972805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d10152650 0x7f4d101ee840 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.394+0000 7f4d17b02640 1 --2- 172.21.15.1:0/248972805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d100bf9d0 0x7f4d10153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.394+0000 7f4d17b02640 1 -- 172.21.15.1:0/248972805 >> 172.21.15.1:0/248972805 conn(0x7f4d10066ee0 msgr2=0x7f4d10151920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:28.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.395+0000 7f4d17b02640 1 -- 172.21.15.1:0/248972805 shutdown_connections 2024-03-06T07:33:28.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.395+0000 7f4d17b02640 1 -- 172.21.15.1:0/248972805 wait complete. 2024-03-06T07:33:28.407 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":606,"stamp":"2024-03-06T07:33:26.884979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":287596,"kb_used_data":21516,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754687124,"statfs":{"total":773094113280,"available":772799614976,"internally_reserved":0,"allocated":22032384,"data_stored":11798400,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3816,"num_objects":-41,"num_object_clones":0,"num_object_copies":-82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-41,"num_whiteouts":0,"num_read":-73,"num_read_kb":-79,"num_write":-85,"num_write_kb":-26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001877"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":549,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-03-06T07:33:24.764069+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:33:24.764069+0000","last_peered":"2024-03-06T07:33:24.764069+0000","last_clean":"2024-03-06T07:33:24.764069+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:33:24.764069+0000","last_undegraded":"2024-03-06T07:33:24.764069+0000","last_fullsized":"2024-03-06T07:33:24.764069+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575009,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33912,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337928,"statfs":{"total":96636764160,"available":96602038272,"internally_reserved":0,"allocated":2797568,"data_stored":1913533,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575008,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33836,"kb_used_data":2656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338004,"statfs":{"total":96636764160,"available":96602116096,"internally_reserved":0,"allocated":2719744,"data_stored":1321973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575008,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33852,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337988,"statfs":{"total":96636764160,"available":96602099712,"internally_reserved":0,"allocated":2736128,"data_stored":1328935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575009,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33856,"kb_used_data":2676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337984,"statfs":{"total":96636764160,"available":96602095616,"internally_reserved":0,"allocated":2740224,"data_stored":1330455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575008,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38016,"kb_used_data":2676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2740224,"data_stored":1329605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575008,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38028,"kb_used_data":2688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333812,"statfs":{"total":96636764160,"available":96597823488,"internally_reserved":0,"allocated":2752512,"data_stored":1331149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38032,"kb_used_data":2692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333808,"statfs":{"total":96636764160,"available":96597819392,"internally_reserved":0,"allocated":2756608,"data_stored":1330283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575009,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38064,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333776,"statfs":{"total":96636764160,"available":96597786624,"internally_reserved":0,"allocated":2789376,"data_stored":1912467,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:33:28.409 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:28.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.509+0000 7f0cb6eed640 1 Processor -- start 2024-03-06T07:33:28.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.510+0000 7f0cb6eed640 1 -- start start 2024-03-06T07:33:28.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.510+0000 7f0cb6eed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cb00bf9d0 0x7f0cb0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.510+0000 7f0cb6eed640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb00c2fc0 con 0x7f0cb0152650 2024-03-06T07:33:28.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.510+0000 7f0cb6eed640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb0158da0 con 0x7f0cb00c6b10 2024-03-06T07:33:28.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.510+0000 7f0cb6eed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb0155800 con 0x7f0cb00bf9d0 2024-03-06T07:33:28.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.510+0000 7f0cb4c89640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0cb0152650 0x7f0cb0158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37094/0 (socket says 172.21.15.1:37094) 2024-03-06T07:33:28.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.510+0000 7f0cb4c89640 1 -- 172.21.15.1:0/3600068550 learned_addr learned my addr 172.21.15.1:0/3600068550 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:28.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.511+0000 7f0cb548a640 1 --2- 172.21.15.1:0/3600068550 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cb00bf9d0 0x7f0cb0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.511+0000 7f0caf7fe640 1 -- 172.21.15.1:0/3600068550 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807307527 0 0) 0x7f0cb00c2fc0 con 0x7f0cb0152650 2024-03-06T07:33:28.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.511+0000 7f0caf7fe640 1 -- 172.21.15.1:0/3600068550 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c98003520 con 0x7f0cb0152650 2024-03-06T07:33:28.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.511+0000 7f0caf7fe640 1 -- 172.21.15.1:0/3600068550 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075332721 0 0) 0x7f0cb0158da0 con 0x7f0cb00c6b10 2024-03-06T07:33:28.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.511+0000 7f0caf7fe640 1 -- 172.21.15.1:0/3600068550 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cb00c2fc0 con 0x7f0cb00c6b10 2024-03-06T07:33:28.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.511+0000 7f0caf7fe640 1 -- 172.21.15.1:0/3600068550 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 384634313 0 0) 0x7f0c98003520 con 0x7f0cb0152650 2024-03-06T07:33:28.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.511+0000 7f0cb548a640 1 -- 172.21.15.1:0/3600068550 >> v1:172.21.15.98:6789/0 conn(0x7f0cb00c6b10 legacy=0x7f0cb00c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.511+0000 7f0cb548a640 1 -- 172.21.15.1:0/3600068550 >> v1:172.21.15.1:6789/0 conn(0x7f0cb0152650 legacy=0x7f0cb0158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.511+0000 7f0cb548a640 1 -- 172.21.15.1:0/3600068550 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb00c6690 con 0x7f0cb00bf9d0 2024-03-06T07:33:28.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.512+0000 7f0cb548a640 1 --2- 172.21.15.1:0/3600068550 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cb00bf9d0 0x7f0cb0155c90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00097d0 tx=0x7f0ca002b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb6c4db119131740 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.512+0000 7f0caf7fe640 1 -- 172.21.15.1:0/3600068550 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0037070 con 0x7f0cb00bf9d0 2024-03-06T07:33:28.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.512+0000 7f0caf7fe640 1 -- 172.21.15.1:0/3600068550 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca0030510 con 0x7f0cb00bf9d0 2024-03-06T07:33:28.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.512+0000 7f0caf7fe640 1 -- 172.21.15.1:0/3600068550 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8001a10 con 0x7f0cb00bf9d0 2024-03-06T07:33:28.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.513+0000 7f0cb6eed640 1 -- 172.21.15.1:0/3600068550 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cb00bf9d0 msgr2=0x7f0cb0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.513+0000 7f0cb6eed640 1 --2- 172.21.15.1:0/3600068550 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cb00bf9d0 0x7f0cb0155c90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00097d0 tx=0x7f0ca002b770 comp rx=0 tx=0).stop 2024-03-06T07:33:28.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.513+0000 7f0cb6eed640 1 -- 172.21.15.1:0/3600068550 shutdown_connections 2024-03-06T07:33:28.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.513+0000 7f0cb6eed640 1 --2- 172.21.15.1:0/3600068550 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cb00bf9d0 0x7f0cb0155c90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.513+0000 7f0cb6eed640 1 -- 172.21.15.1:0/3600068550 >> 172.21.15.1:0/3600068550 conn(0x7f0cb0066ee0 msgr2=0x7f0cb00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:28.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.513+0000 7f0cb6eed640 1 -- 172.21.15.1:0/3600068550 shutdown_connections 2024-03-06T07:33:28.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.513+0000 7f0cb6eed640 1 -- 172.21.15.1:0/3600068550 wait complete. 2024-03-06T07:33:28.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.514+0000 7f0cb6eed640 1 Processor -- start 2024-03-06T07:33:28.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.514+0000 7f0cb6eed640 1 -- start start 2024-03-06T07:33:28.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.514+0000 7f0cb6eed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb00bf9d0 0x7f0cb01e1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.514+0000 7f0cb6eed640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cb0152650 0x7f0cb01ee500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.514+0000 7f0cb6eed640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb01e0990 0x7f0cb01eea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.514+0000 7f0cb6eed640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb01551b0 con 0x7f0cb01e0990 2024-03-06T07:33:28.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.514+0000 7f0cb6eed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb00c6fa0 con 0x7f0cb00bf9d0 2024-03-06T07:33:28.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.514+0000 7f0cb6eed640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb0155490 con 0x7f0cb0152650 2024-03-06T07:33:28.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.515+0000 7f0cb548a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb01e0990 0x7f0cb01eea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.515+0000 7f0cb548a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb01e0990 0x7f0cb01eea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40672/0 (socket says 172.21.15.1:40672) 2024-03-06T07:33:28.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.515+0000 7f0cb548a640 1 -- 172.21.15.1:0/720045997 learned_addr learned my addr 172.21.15.1:0/720045997 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:28.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.515+0000 7f0caffff640 1 --2- 172.21.15.1:0/720045997 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cb0152650 0x7f0cb01ee500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.515+0000 7f0cb4c89640 1 --2- 172.21.15.1:0/720045997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb00bf9d0 0x7f0cb01e1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.515+0000 7f0cb548a640 1 -- 172.21.15.1:0/720045997 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cb0152650 msgr2=0x7f0cb01ee500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.515+0000 7f0cb548a640 1 --2- 172.21.15.1:0/720045997 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cb0152650 0x7f0cb01ee500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.515+0000 7f0cb548a640 1 -- 172.21.15.1:0/720045997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb00bf9d0 msgr2=0x7f0cb01e1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.515+0000 7f0cb548a640 1 --2- 172.21.15.1:0/720045997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb00bf9d0 0x7f0cb01e1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.515+0000 7f0cb548a640 1 -- 172.21.15.1:0/720045997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb01eef80 con 0x7f0cb01e0990 2024-03-06T07:33:28.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.515+0000 7f0cb548a640 1 --2- 172.21.15.1:0/720045997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb01e0990 0x7f0cb01eea40 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f0ca002b7b0 tx=0x7f0ca00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.515+0000 7f0cad7fa640 1 -- 172.21.15.1:0/720045997 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0042070 con 0x7f0cb01e0990 2024-03-06T07:33:28.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.515+0000 7f0cad7fa640 1 -- 172.21.15.1:0/720045997 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca003e870 con 0x7f0cb01e0990 2024-03-06T07:33:28.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.515+0000 7f0cb6eed640 1 -- 172.21.15.1:0/720045997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb01dfa60 con 0x7f0cb01e0990 2024-03-06T07:33:28.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.515+0000 7f0cad7fa640 1 -- 172.21.15.1:0/720045997 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0037040 con 0x7f0cb01e0990 2024-03-06T07:33:28.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.515+0000 7f0cb6eed640 1 -- 172.21.15.1:0/720045997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb014daa0 con 0x7f0cb01e0990 2024-03-06T07:33:28.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.516+0000 7f0cb6eed640 1 -- 172.21.15.1:0/720045997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb00686e0 con 0x7f0cb01e0990 2024-03-06T07:33:28.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.517+0000 7f0cad7fa640 1 -- 172.21.15.1:0/720045997 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ca0045070 con 0x7f0cb01e0990 2024-03-06T07:33:28.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.517+0000 7f0cad7fa640 1 --2- 172.21.15.1:0/720045997 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c9c06b4d0 0x7f0c9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.518+0000 7f0cad7fa640 1 -- 172.21.15.1:0/720045997 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f0ca00abdb0 con 0x7f0cb01e0990 2024-03-06T07:33:28.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.518+0000 7f0cb4c89640 1 --2- 172.21.15.1:0/720045997 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c9c06b4d0 0x7f0c9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.518+0000 7f0cb4c89640 1 --2- 172.21.15.1:0/720045997 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c9c06b4d0 0x7f0c9c06d990 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80018a0 tx=0x7f0ca8001830 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.522+0000 7f0cad7fa640 1 -- 172.21.15.1:0/720045997 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0cb00686e0 con 0x7f0cb01e0990 2024-03-06T07:33:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.704+0000 7f0cb6eed640 1 -- 172.21.15.1:0/720045997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cb0067fd0 con 0x7f0cb01e0990 2024-03-06T07:33:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.705+0000 7f0cad7fa640 1 -- 172.21.15.1:0/720045997 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v217) v1 ==== 74+0+13569 (secure 0 0 0) 0x7f0ca007a800 con 0x7f0cb01e0990 2024-03-06T07:33:28.707 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:28.708 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":217,"fsid":"8f3c19ba-4ac6-4a54-88df-9262183d23ef","created":"2024-03-06T07:20:04.466310+0000","modified":"2024-03-06T07:33:25.760701+0000","last_up_change":"2024-03-06T07:20:14.686886+0000","last_in_change":"2024-03-06T07:20:06.693057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:20:14.728260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b306d19-7ed3-4551-b20a-863757aace9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6818","nonce":3842251104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6822","nonce":3842251104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6829","nonce":3842251104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3842251104},{"type":"v1","addr":"172.21.15.1:6826","nonce":3842251104}]},"public_addr":"172.21.15.1:6818/3842251104","cluster_addr":"172.21.15.1:6822/3842251104","heartbeat_back_addr":"172.21.15.1:6829/3842251104","heartbeat_front_addr":"172.21.15.1:6826/3842251104","state":["exists","up"]},{"osd":1,"uuid":"7706d997-9bc6-4ed2-bfa0-3f7aec78afdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6819","nonce":682560127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6823","nonce":682560127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6831","nonce":682560127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":682560127},{"type":"v1","addr":"172.21.15.1:6827","nonce":682560127}]},"public_addr":"172.21.15.1:6819/682560127","cluster_addr":"172.21.15.1:6823/682560127","heartbeat_back_addr":"172.21.15.1:6831/682560127","heartbeat_front_addr":"172.21.15.1:6827/682560127","state":["exists","up"]},{"osd":2,"uuid":"2fc36263-d860-4ecd-a97d-8bb7810aa070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6801","nonce":113114752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6803","nonce":113114752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6807","nonce":113114752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":113114752},{"type":"v1","addr":"172.21.15.1:6805","nonce":113114752}]},"public_addr":"172.21.15.1:6801/113114752","cluster_addr":"172.21.15.1:6803/113114752","heartbeat_back_addr":"172.21.15.1:6807/113114752","heartbeat_front_addr":"172.21.15.1:6805/113114752","state":["exists","up"]},{"osd":3,"uuid":"68f1b153-7890-4c48-ac45-293337d0e529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6809","nonce":1031158088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6811","nonce":1031158088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6815","nonce":1031158088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1031158088},{"type":"v1","addr":"172.21.15.1:6813","nonce":1031158088}]},"public_addr":"172.21.15.1:6809/1031158088","cluster_addr":"172.21.15.1:6811/1031158088","heartbeat_back_addr":"172.21.15.1:6815/1031158088","heartbeat_front_addr":"172.21.15.1:6813/1031158088","state":["exists","up"]},{"osd":4,"uuid":"62134c92-7dbe-4266-915d-c6333d17aeb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6809","nonce":2308139585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6811","nonce":2308139585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6815","nonce":2308139585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2308139585},{"type":"v1","addr":"172.21.15.98:6813","nonce":2308139585}]},"public_addr":"172.21.15.98:6809/2308139585","cluster_addr":"172.21.15.98:6811/2308139585","heartbeat_back_addr":"172.21.15.98:6815/2308139585","heartbeat_front_addr":"172.21.15.98:6813/2308139585","state":["exists","up"]},{"osd":5,"uuid":"46eacdc5-0b2a-4977-995e-47240d980c20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6817","nonce":851921598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6819","nonce":851921598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6823","nonce":851921598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":851921598},{"type":"v1","addr":"172.21.15.98:6821","nonce":851921598}]},"public_addr":"172.21.15.98:6817/851921598","cluster_addr":"172.21.15.98:6819/851921598","heartbeat_back_addr":"172.21.15.98:6823/851921598","heartbeat_front_addr":"172.21.15.98:6821/851921598","state":["exists","up"]},{"osd":6,"uuid":"296b86b0-3cfb-4b09-97ca-4c1c62d547f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6801","nonce":2463344330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6803","nonce":2463344330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6807","nonce":2463344330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2463344330},{"type":"v1","addr":"172.21.15.98:6805","nonce":2463344330}]},"public_addr":"172.21.15.98:6801/2463344330","cluster_addr":"172.21.15.98:6803/2463344330","heartbeat_back_addr":"172.21.15.98:6807/2463344330","heartbeat_front_addr":"172.21.15.98:6805/2463344330","state":["exists","up"]},{"osd":7,"uuid":"69c21780-8b78-49d6-b246-e9d21297f754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6825","nonce":3249148736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6827","nonce":3249148736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6831","nonce":3249148736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3249148736},{"type":"v1","addr":"172.21.15.98:6829","nonce":3249148736}]},"public_addr":"172.21.15.98:6825/3249148736","cluster_addr":"172.21.15.98:6827/3249148736","heartbeat_back_addr":"172.21.15.98:6831/3249148736","heartbeat_front_addr":"172.21.15.98:6829/3249148736","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.863789+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.868220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.770195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.825945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.891572+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.826349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.860731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:20:11.764112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6835/1871567565":"2024-03-07T07:33:22.752788+0000","172.21.15.98:6837/3233486394":"2024-03-07T07:33:22.752776+0000","172.21.15.1:6833/1871567565":"2024-03-07T07:33:22.752788+0000","172.21.15.98:6835/3233486394":"2024-03-07T07:33:22.752776+0000","172.21.15.98:6836/2083320955":"2024-03-07T07:30:49.314605+0000","172.21.15.98:6834/2083320955":"2024-03-07T07:30:49.314605+0000","172.21.15.1:6834/2424540959":"2024-03-07T07:30:14.310288+0000","172.21.15.1:6832/2424540959":"2024-03-07T07:30:14.310288+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.707+0000 7f0cb6eed640 1 -- 172.21.15.1:0/720045997 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c9c06b4d0 msgr2=0x7f0c9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.707+0000 7f0cb6eed640 1 --2- 172.21.15.1:0/720045997 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c9c06b4d0 0x7f0c9c06d990 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80018a0 tx=0x7f0ca8001830 comp rx=0 tx=0).stop 2024-03-06T07:33:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.707+0000 7f0cb6eed640 1 -- 172.21.15.1:0/720045997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb01e0990 msgr2=0x7f0cb01eea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.707+0000 7f0cb6eed640 1 --2- 172.21.15.1:0/720045997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb01e0990 0x7f0cb01eea40 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f0ca002b7b0 tx=0x7f0ca00046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.707+0000 7f0cb6eed640 1 -- 172.21.15.1:0/720045997 shutdown_connections 2024-03-06T07:33:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.707+0000 7f0cb6eed640 1 --2- 172.21.15.1:0/720045997 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0c9c06b4d0 0x7f0c9c06d990 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.707+0000 7f0cb6eed640 1 --2- 172.21.15.1:0/720045997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb01e0990 0x7f0cb01eea40 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.707+0000 7f0cb6eed640 1 --2- 172.21.15.1:0/720045997 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cb0152650 0x7f0cb01ee500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.707+0000 7f0cb6eed640 1 --2- 172.21.15.1:0/720045997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb00bf9d0 0x7f0cb01e1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.707+0000 7f0cb6eed640 1 -- 172.21.15.1:0/720045997 >> 172.21.15.1:0/720045997 conn(0x7f0cb0066ee0 msgr2=0x7f0cb00c52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.707+0000 7f0cb6eed640 1 -- 172.21.15.1:0/720045997 shutdown_connections 2024-03-06T07:33:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:28.707+0000 7f0cb6eed640 1 -- 172.21.15.1:0/720045997 wait complete. 2024-03-06T07:33:29.720 INFO:tasks.ceph:Scrubbing osd.0 2024-03-06T07:33:29.721 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:33:29.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.839+0000 7f879464e640 1 Processor -- start 2024-03-06T07:33:29.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.839+0000 7f879464e640 1 -- start start 2024-03-06T07:33:29.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.840+0000 7f879464e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f878c0c5700 0x7f878c1551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.840+0000 7f879464e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878c0c98c0 con 0x7f878c0c38c0 2024-03-06T07:33:29.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.840+0000 7f879464e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878c0c69a0 con 0x7f878c0c3d80 2024-03-06T07:33:29.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.840+0000 7f879464e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f878c0c7000 con 0x7f878c0c5700 2024-03-06T07:33:29.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.840+0000 7f87923ea640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f878c0c5700 0x7f878c1551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.840+0000 7f8792beb640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f878c0c38c0 0x7f878c152590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37102/0 (socket says 172.21.15.1:37102) 2024-03-06T07:33:29.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.840+0000 7f8792beb640 1 -- 172.21.15.1:0/3030462035 learned_addr learned my addr 172.21.15.1:0/3030462035 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:29.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.841+0000 7f87923ea640 1 -- 172.21.15.1:0/3030462035 >> v1:172.21.15.98:6789/0 conn(0x7f878c0c3d80 legacy=0x7f878c0c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:29.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.841+0000 7f87923ea640 1 -- 172.21.15.1:0/3030462035 >> v1:172.21.15.1:6789/0 conn(0x7f878c0c38c0 legacy=0x7f878c152590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.841+0000 7f87923ea640 1 -- 172.21.15.1:0/3030462035 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c0c6c80 con 0x7f878c0c5700 2024-03-06T07:33:29.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.841+0000 7f87913e8640 1 -- 172.21.15.1:0/3030462035 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362976388 0 0) 0x7f878c0c98c0 con 0x7f878c0c38c0 2024-03-06T07:33:29.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.842+0000 7f87923ea640 1 --2- 172.21.15.1:0/3030462035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f878c0c5700 0x7f878c1551e0 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f87880097a0 tx=0x7f878802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67f4fc93ea1271a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.842+0000 7f87913e8640 1 -- 172.21.15.1:0/3030462035 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8788035020 con 0x7f878c0c5700 2024-03-06T07:33:29.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.842+0000 7f87913e8640 1 -- 172.21.15.1:0/3030462035 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8788031830 con 0x7f878c0c5700 2024-03-06T07:33:29.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.842+0000 7f87913e8640 1 -- 172.21.15.1:0/3030462035 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8788031c00 con 0x7f878c0c5700 2024-03-06T07:33:29.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.843+0000 7f879464e640 1 -- 172.21.15.1:0/3030462035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f878c0c5700 msgr2=0x7f878c1551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.843+0000 7f879464e640 1 --2- 172.21.15.1:0/3030462035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f878c0c5700 0x7f878c1551e0 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f87880097a0 tx=0x7f878802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.843+0000 7f879464e640 1 -- 172.21.15.1:0/3030462035 shutdown_connections 2024-03-06T07:33:29.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.843+0000 7f879464e640 1 --2- 172.21.15.1:0/3030462035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f878c0c5700 0x7f878c1551e0 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.843+0000 7f879464e640 1 -- 172.21.15.1:0/3030462035 >> 172.21.15.1:0/3030462035 conn(0x7f878c06b620 msgr2=0x7f878c0c8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:29.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.843+0000 7f879464e640 1 -- 172.21.15.1:0/3030462035 shutdown_connections 2024-03-06T07:33:29.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.843+0000 7f879464e640 1 -- 172.21.15.1:0/3030462035 wait complete. 2024-03-06T07:33:29.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.844+0000 7f879464e640 1 Processor -- start 2024-03-06T07:33:29.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.844+0000 7f879464e640 1 -- start start 2024-03-06T07:33:29.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f879464e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f878c0c38c0 0x7f878c0c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f879464e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f878c0c3d80 0x7f878c144e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f879464e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f878c0c5700 0x7f878c145390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f879464e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c152ca0 con 0x7f878c0c5700 2024-03-06T07:33:29.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f879464e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c159410 con 0x7f878c0c3d80 2024-03-06T07:33:29.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f879464e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878c152e20 con 0x7f878c0c38c0 2024-03-06T07:33:29.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f8792beb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f878c0c5700 0x7f878c145390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f8792beb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f878c0c5700 0x7f878c145390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40688/0 (socket says 172.21.15.1:40688) 2024-03-06T07:33:29.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f8792beb640 1 -- 172.21.15.1:0/3961635788 learned_addr learned my addr 172.21.15.1:0/3961635788 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:29.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f8791be9640 1 --2- 172.21.15.1:0/3961635788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f878c0c3d80 0x7f878c144e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f87923ea640 1 --2- 172.21.15.1:0/3961635788 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f878c0c38c0 0x7f878c0c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f8792beb640 1 -- 172.21.15.1:0/3961635788 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f878c0c38c0 msgr2=0x7f878c0c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f8792beb640 1 --2- 172.21.15.1:0/3961635788 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f878c0c38c0 0x7f878c0c8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f8792beb640 1 -- 172.21.15.1:0/3961635788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f878c0c3d80 msgr2=0x7f878c144e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f8792beb640 1 --2- 172.21.15.1:0/3961635788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f878c0c3d80 0x7f878c144e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f8792beb640 1 -- 172.21.15.1:0/3961635788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c1ee170 con 0x7f878c0c5700 2024-03-06T07:33:29.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f8791be9640 1 --2- 172.21.15.1:0/3961635788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f878c0c3d80 0x7f878c144e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:29.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.845+0000 7f8792beb640 1 --2- 172.21.15.1:0/3961635788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f878c0c5700 0x7f878c145390 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f878400c6d0 tx=0x7f878400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.846+0000 7f8782ffd640 1 -- 172.21.15.1:0/3961635788 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8784018070 con 0x7f878c0c5700 2024-03-06T07:33:29.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.846+0000 7f8782ffd640 1 -- 172.21.15.1:0/3961635788 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8784004d60 con 0x7f878c0c5700 2024-03-06T07:33:29.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.846+0000 7f8782ffd640 1 -- 172.21.15.1:0/3961635788 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878400a820 con 0x7f878c0c5700 2024-03-06T07:33:29.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.846+0000 7f879464e640 1 -- 172.21.15.1:0/3961635788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f878c1ee450 con 0x7f878c0c5700 2024-03-06T07:33:29.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.846+0000 7f879464e640 1 -- 172.21.15.1:0/3961635788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f878c0c8e60 con 0x7f878c0c5700 2024-03-06T07:33:29.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.847+0000 7f879464e640 1 -- 172.21.15.1:0/3961635788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f878c0686e0 con 0x7f878c0c5700 2024-03-06T07:33:29.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.847+0000 7f8782ffd640 1 -- 172.21.15.1:0/3961635788 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f878401b070 con 0x7f878c0c5700 2024-03-06T07:33:29.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.848+0000 7f8782ffd640 1 --2- 172.21.15.1:0/3961635788 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f877006b4d0 0x7f877006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.848+0000 7f8782ffd640 1 -- 172.21.15.1:0/3961635788 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f87840898a0 con 0x7f878c0c5700 2024-03-06T07:33:29.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.848+0000 7f8782ffd640 1 --2- 172.21.15.1:0/3961635788 >> [v2:172.21.15.1:6816/3842251104,v1:172.21.15.1:6818/3842251104] conn(0x7f8770071b50 0x7f8770073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.848+0000 7f8782ffd640 1 -- 172.21.15.1:0/3961635788 --> [v2:172.21.15.1:6816/3842251104,v1:172.21.15.1:6818/3842251104] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f878401be50 con 0x7f8770071b50 2024-03-06T07:33:29.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.848+0000 7f8782ffd640 1 -- 172.21.15.1:0/3961635788 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=217) v2 ==== 24+0+0 (secure 0 0 0) 0x7f878408e2a0 con 0x7f878c0c5700 2024-03-06T07:33:29.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.848+0000 7f87923ea640 1 --2- 172.21.15.1:0/3961635788 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f877006b4d0 0x7f877006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.848+0000 7f8791be9640 1 --2- 172.21.15.1:0/3961635788 >> [v2:172.21.15.1:6816/3842251104,v1:172.21.15.1:6818/3842251104] conn(0x7f8770071b50 0x7f8770073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.848+0000 7f8791be9640 1 --2- 172.21.15.1:0/3961635788 >> [v2:172.21.15.1:6816/3842251104,v1:172.21.15.1:6818/3842251104] conn(0x7f8770071b50 0x7f8770073f90 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.848+0000 7f87923ea640 1 --2- 172.21.15.1:0/3961635788 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f877006b4d0 0x7f877006d990 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f8788036000 tx=0x7f878803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.849+0000 7f8782ffd640 1 -- 172.21.15.1:0/3961635788 <== osd.0 v2:172.21.15.1:6816/3842251104 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f878401be50 con 0x7f8770071b50 2024-03-06T07:33:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.862+0000 7f879464e640 1 -- 172.21.15.1:0/3961635788 --> [v2:172.21.15.1:6816/3842251104,v1:172.21.15.1:6818/3842251104] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f878c151e00 con 0x7f8770071b50 2024-03-06T07:33:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.863+0000 7f8782ffd640 1 -- 172.21.15.1:0/3961635788 <== osd.0 v2:172.21.15.1:6816/3842251104 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f878c151e00 con 0x7f8770071b50 2024-03-06T07:33:29.865 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:33:29.865 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:33:29.865 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:33:29.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.863+0000 7f879464e640 1 -- 172.21.15.1:0/3961635788 >> [v2:172.21.15.1:6816/3842251104,v1:172.21.15.1:6818/3842251104] conn(0x7f8770071b50 msgr2=0x7f8770073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.863+0000 7f879464e640 1 --2- 172.21.15.1:0/3961635788 >> [v2:172.21.15.1:6816/3842251104,v1:172.21.15.1:6818/3842251104] conn(0x7f8770071b50 0x7f8770073f90 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.863+0000 7f879464e640 1 -- 172.21.15.1:0/3961635788 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f877006b4d0 msgr2=0x7f877006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.863+0000 7f879464e640 1 --2- 172.21.15.1:0/3961635788 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f877006b4d0 0x7f877006d990 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f8788036000 tx=0x7f878803c040 comp rx=0 tx=0).stop 2024-03-06T07:33:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.863+0000 7f879464e640 1 -- 172.21.15.1:0/3961635788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f878c0c5700 msgr2=0x7f878c145390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.863+0000 7f879464e640 1 --2- 172.21.15.1:0/3961635788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f878c0c5700 0x7f878c145390 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f878400c6d0 tx=0x7f878400cb90 comp rx=0 tx=0).stop 2024-03-06T07:33:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.863+0000 7f8792beb640 1 -- 172.21.15.1:0/3961635788 reap_dead start 2024-03-06T07:33:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.863+0000 7f879464e640 1 -- 172.21.15.1:0/3961635788 shutdown_connections 2024-03-06T07:33:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.863+0000 7f879464e640 1 -- 172.21.15.1:0/3961635788 >> 172.21.15.1:0/3961635788 conn(0x7f878c06b620 msgr2=0x7f878c0ad950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:29.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.863+0000 7f879464e640 1 -- 172.21.15.1:0/3961635788 shutdown_connections 2024-03-06T07:33:29.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.863+0000 7f879464e640 1 -- 172.21.15.1:0/3961635788 wait complete. 2024-03-06T07:33:29.876 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-06T07:33:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.977+0000 7f40b30d4640 1 Processor -- start 2024-03-06T07:33:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.977+0000 7f40b30d4640 1 -- start start 2024-03-06T07:33:29.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.978+0000 7f40b30d4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40ac152650 0x7f40ac155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.978+0000 7f40b30d4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f40ac0c6b10 0x7f40ac158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.978+0000 7f40b30d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40ac0bf9d0 0x7f40ac0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.978+0000 7f40b30d4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ac155800 con 0x7f40ac152650 2024-03-06T07:33:29.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.978+0000 7f40b30d4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ac0c2fc0 con 0x7f40ac0c6b10 2024-03-06T07:33:29.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.978+0000 7f40b30d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac0c6690 con 0x7f40ac0bf9d0 2024-03-06T07:33:29.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.978+0000 7f40b0e70640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40ac152650 0x7f40ac155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.978+0000 7f40b0e70640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40ac152650 0x7f40ac155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40694/0 (socket says 172.21.15.1:40694) 2024-03-06T07:33:29.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.978+0000 7f40b0e70640 1 -- 172.21.15.1:0/2645856167 learned_addr learned my addr 172.21.15.1:0/2645856167 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:29.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.978+0000 7f40b1671640 1 --2- 172.21.15.1:0/2645856167 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40ac0bf9d0 0x7f40ac0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.978+0000 7f40abfff640 1 --2- 172.21.15.1:0/2645856167 >> v2:172.21.15.98:3300/0 conn(0x7f40ac0c6b10 0x7f40ac158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.978+0000 7f40b0e70640 1 -- 172.21.15.1:0/2645856167 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40ac0bf9d0 msgr2=0x7f40ac0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:29.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.978+0000 7f40b0e70640 1 --2- 172.21.15.1:0/2645856167 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40ac0bf9d0 0x7f40ac0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.978+0000 7f40b0e70640 1 -- 172.21.15.1:0/2645856167 >> v2:172.21.15.98:3300/0 conn(0x7f40ac0c6b10 msgr2=0x7f40ac158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:29.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.978+0000 7f40b0e70640 1 --2- 172.21.15.1:0/2645856167 >> v2:172.21.15.98:3300/0 conn(0x7f40ac0c6b10 0x7f40ac158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.978+0000 7f40b0e70640 1 -- 172.21.15.1:0/2645856167 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ac0c7280 con 0x7f40ac152650 2024-03-06T07:33:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.979+0000 7f40b0e70640 1 --2- 172.21.15.1:0/2645856167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40ac152650 0x7f40ac155c90 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0cb8e0 tx=0x7f40a002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7164f11d6e918796 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.979+0000 7f40ab7fe640 1 -- 172.21.15.1:0/2645856167 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a002e070 con 0x7f40ac152650 2024-03-06T07:33:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.979+0000 7f40ab7fe640 1 -- 172.21.15.1:0/2645856167 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a0037070 con 0x7f40ac152650 2024-03-06T07:33:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.979+0000 7f40ab7fe640 1 -- 172.21.15.1:0/2645856167 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a0031a00 con 0x7f40ac152650 2024-03-06T07:33:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.980+0000 7f40b30d4640 1 -- 172.21.15.1:0/2645856167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40ac152650 msgr2=0x7f40ac155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.980+0000 7f40b30d4640 1 --2- 172.21.15.1:0/2645856167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40ac152650 0x7f40ac155c90 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0cb8e0 tx=0x7f40a002b880 comp rx=0 tx=0).stop 2024-03-06T07:33:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.980+0000 7f40b30d4640 1 -- 172.21.15.1:0/2645856167 shutdown_connections 2024-03-06T07:33:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.980+0000 7f40b30d4640 1 --2- 172.21.15.1:0/2645856167 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40ac0bf9d0 0x7f40ac0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.980+0000 7f40b30d4640 1 --2- 172.21.15.1:0/2645856167 >> v2:172.21.15.98:3300/0 conn(0x7f40ac0c6b10 0x7f40ac158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.980+0000 7f40b30d4640 1 --2- 172.21.15.1:0/2645856167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40ac152650 0x7f40ac155c90 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.980+0000 7f40b30d4640 1 -- 172.21.15.1:0/2645856167 >> 172.21.15.1:0/2645856167 conn(0x7f40ac066ee0 msgr2=0x7f40ac0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.980+0000 7f40b30d4640 1 -- 172.21.15.1:0/2645856167 shutdown_connections 2024-03-06T07:33:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.980+0000 7f40b30d4640 1 -- 172.21.15.1:0/2645856167 wait complete. 2024-03-06T07:33:29.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.981+0000 7f40b30d4640 1 Processor -- start 2024-03-06T07:33:29.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.981+0000 7f40b30d4640 1 -- start start 2024-03-06T07:33:29.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.982+0000 7f40b30d4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40ac0bf9d0 0x7f40ac0c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.982+0000 7f40b30d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40ac152650 0x7f40ac1ed880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.982+0000 7f40b30d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40ac144a00 0x7f40ac1eddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.982+0000 7f40b30d4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac155490 con 0x7f40ac152650 2024-03-06T07:33:29.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.982+0000 7f40b30d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac1551b0 con 0x7f40ac144a00 2024-03-06T07:33:29.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.982+0000 7f40b30d4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac152a30 con 0x7f40ac0bf9d0 2024-03-06T07:33:29.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.982+0000 7f40abfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40ac152650 0x7f40ac1ed880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.982+0000 7f40abfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40ac152650 0x7f40ac1ed880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40700/0 (socket says 172.21.15.1:40700) 2024-03-06T07:33:29.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.982+0000 7f40abfff640 1 -- 172.21.15.1:0/2768824134 learned_addr learned my addr 172.21.15.1:0/2768824134 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:29.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.982+0000 7f40b0e70640 1 --2- 172.21.15.1:0/2768824134 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40ac0bf9d0 0x7f40ac0c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.982+0000 7f40b1671640 1 --2- 172.21.15.1:0/2768824134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40ac144a00 0x7f40ac1eddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.982+0000 7f40abfff640 1 -- 172.21.15.1:0/2768824134 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40ac0bf9d0 msgr2=0x7f40ac0c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.982+0000 7f40abfff640 1 --2- 172.21.15.1:0/2768824134 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40ac0bf9d0 0x7f40ac0c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.982+0000 7f40abfff640 1 -- 172.21.15.1:0/2768824134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40ac144a00 msgr2=0x7f40ac1eddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.982+0000 7f40abfff640 1 --2- 172.21.15.1:0/2768824134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40ac144a00 0x7f40ac1eddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.982+0000 7f40abfff640 1 -- 172.21.15.1:0/2768824134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ac154a20 con 0x7f40ac152650 2024-03-06T07:33:29.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.983+0000 7f40abfff640 1 --2- 172.21.15.1:0/2768824134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40ac152650 0x7f40ac1ed880 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f409800e9e0 tx=0x7f409800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.983+0000 7f40a97fa640 1 -- 172.21.15.1:0/2768824134 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4098017070 con 0x7f40ac152650 2024-03-06T07:33:29.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.983+0000 7f40b30d4640 1 -- 172.21.15.1:0/2768824134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40ac1eef30 con 0x7f40ac152650 2024-03-06T07:33:29.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.983+0000 7f40a97fa640 1 -- 172.21.15.1:0/2768824134 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409800c9d0 con 0x7f40ac152650 2024-03-06T07:33:29.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.983+0000 7f40a97fa640 1 -- 172.21.15.1:0/2768824134 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409800cdc0 con 0x7f40ac152650 2024-03-06T07:33:29.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.983+0000 7f40b30d4640 1 -- 172.21.15.1:0/2768824134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ac1ef210 con 0x7f40ac152650 2024-03-06T07:33:29.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.984+0000 7f40b30d4640 1 -- 172.21.15.1:0/2768824134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ac067fd0 con 0x7f40ac152650 2024-03-06T07:33:29.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.984+0000 7f40a97fa640 1 -- 172.21.15.1:0/2768824134 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f409801a070 con 0x7f40ac152650 2024-03-06T07:33:29.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.985+0000 7f40a97fa640 1 --2- 172.21.15.1:0/2768824134 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f409006b4d0 0x7f409006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.985+0000 7f40a97fa640 1 -- 172.21.15.1:0/2768824134 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f409808a490 con 0x7f40ac152650 2024-03-06T07:33:29.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.985+0000 7f40b0e70640 1 --2- 172.21.15.1:0/2768824134 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f409006b4d0 0x7f409006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.986+0000 7f40b0e70640 1 --2- 172.21.15.1:0/2768824134 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f409006b4d0 0x7f409006d990 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f40a0036000 tx=0x7f40a003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:29.989+0000 7f40a97fa640 1 -- 172.21.15.1:0/2768824134 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4098058d40 con 0x7f40ac152650 2024-03-06T07:33:30.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.176+0000 7f40b30d4640 1 -- 172.21.15.1:0/2768824134 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f40ac0c48b0 con 0x7f409006b4d0 2024-03-06T07:33:30.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.177+0000 7f40a97fa640 1 -- 172.21.15.1:0/2768824134 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f40ac0c48b0 con 0x7f409006b4d0 2024-03-06T07:33:30.179 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 0 to deep-scrub 2024-03-06T07:33:30.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.179+0000 7f40b30d4640 1 -- 172.21.15.1:0/2768824134 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f409006b4d0 msgr2=0x7f409006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.179+0000 7f40b30d4640 1 --2- 172.21.15.1:0/2768824134 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f409006b4d0 0x7f409006d990 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f40a0036000 tx=0x7f40a003d040 comp rx=0 tx=0).stop 2024-03-06T07:33:30.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.179+0000 7f40b30d4640 1 -- 172.21.15.1:0/2768824134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40ac152650 msgr2=0x7f40ac1ed880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.179+0000 7f40b30d4640 1 --2- 172.21.15.1:0/2768824134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40ac152650 0x7f40ac1ed880 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f409800e9e0 tx=0x7f409800eea0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.179+0000 7f40b30d4640 1 -- 172.21.15.1:0/2768824134 shutdown_connections 2024-03-06T07:33:30.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.179+0000 7f40b30d4640 1 --2- 172.21.15.1:0/2768824134 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f409006b4d0 0x7f409006d990 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.179+0000 7f40b30d4640 1 --2- 172.21.15.1:0/2768824134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40ac144a00 0x7f40ac1eddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.179+0000 7f40b30d4640 1 --2- 172.21.15.1:0/2768824134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40ac152650 0x7f40ac1ed880 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.179+0000 7f40b30d4640 1 --2- 172.21.15.1:0/2768824134 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40ac0bf9d0 0x7f40ac0c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.179+0000 7f40b30d4640 1 -- 172.21.15.1:0/2768824134 >> 172.21.15.1:0/2768824134 conn(0x7f40ac066ee0 msgr2=0x7f40ac0cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.179+0000 7f40b30d4640 1 -- 172.21.15.1:0/2768824134 shutdown_connections 2024-03-06T07:33:30.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.179+0000 7f40b30d4640 1 -- 172.21.15.1:0/2768824134 wait complete. 2024-03-06T07:33:30.192 INFO:tasks.ceph:Scrubbing osd.1 2024-03-06T07:33:30.192 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:33:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.297+0000 7f574d732640 1 Processor -- start 2024-03-06T07:33:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.297+0000 7f574d732640 1 -- start start 2024-03-06T07:33:30.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.298+0000 7f574d732640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5748155030 0x7f5748155410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.298+0000 7f574d732640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5748155950 0x7f574806e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.298+0000 7f574d732640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57480c1ce0 con 0x7f5748071df0 2024-03-06T07:33:30.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.298+0000 7f574d732640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57480c1fc0 con 0x7f5748155030 2024-03-06T07:33:30.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.298+0000 7f574d732640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57480c2b20 con 0x7f5748155950 2024-03-06T07:33:30.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.298+0000 7f573ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5748071df0 0x7f574814eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37112/0 (socket says 172.21.15.1:37112) 2024-03-06T07:33:30.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.298+0000 7f573ffff640 1 -- 172.21.15.1:0/3025270145 learned_addr learned my addr 172.21.15.1:0/3025270145 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:30.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.298+0000 7f5746ffd640 1 --2- 172.21.15.1:0/3025270145 >> v2:172.21.15.98:3300/0 conn(0x7f5748155030 0x7f5748155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.298+0000 7f57477fe640 1 --2- 172.21.15.1:0/3025270145 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5748155950 0x7f574806e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.298+0000 7f57467fc640 1 -- 172.21.15.1:0/3025270145 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355195908 0 0) 0x7f57480c1ce0 con 0x7f5748071df0 2024-03-06T07:33:30.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.298+0000 7f57467fc640 1 -- 172.21.15.1:0/3025270145 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5734003520 con 0x7f5748071df0 2024-03-06T07:33:30.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.299+0000 7f57477fe640 1 -- 172.21.15.1:0/3025270145 >> v2:172.21.15.98:3300/0 conn(0x7f5748155030 msgr2=0x7f5748155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:30.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.299+0000 7f57477fe640 1 --2- 172.21.15.1:0/3025270145 >> v2:172.21.15.98:3300/0 conn(0x7f5748155030 0x7f5748155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.299+0000 7f57477fe640 1 -- 172.21.15.1:0/3025270145 >> v1:172.21.15.1:6789/0 conn(0x7f5748071df0 legacy=0x7f574814eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.299+0000 7f57477fe640 1 -- 172.21.15.1:0/3025270145 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57480c2e00 con 0x7f5748155950 2024-03-06T07:33:30.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.299+0000 7f5746ffd640 1 --2- 172.21.15.1:0/3025270145 >> v2:172.21.15.98:3300/0 conn(0x7f5748155030 0x7f5748155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.299+0000 7f57477fe640 1 --2- 172.21.15.1:0/3025270145 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5748155950 0x7f574806e2e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5740009d90 tx=0x7f574002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe116e925ca2c498 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.300+0000 7f57467fc640 1 -- 172.21.15.1:0/3025270145 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740039070 con 0x7f5748155950 2024-03-06T07:33:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.300+0000 7f57467fc640 1 -- 172.21.15.1:0/3025270145 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574002cbc0 con 0x7f5748155950 2024-03-06T07:33:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.300+0000 7f57467fc640 1 -- 172.21.15.1:0/3025270145 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740034440 con 0x7f5748155950 2024-03-06T07:33:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.300+0000 7f574d732640 1 -- 172.21.15.1:0/3025270145 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5748155950 msgr2=0x7f574806e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.300+0000 7f574d732640 1 --2- 172.21.15.1:0/3025270145 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5748155950 0x7f574806e2e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5740009d90 tx=0x7f574002be10 comp rx=0 tx=0).stop 2024-03-06T07:33:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.300+0000 7f574d732640 1 -- 172.21.15.1:0/3025270145 shutdown_connections 2024-03-06T07:33:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.300+0000 7f574d732640 1 --2- 172.21.15.1:0/3025270145 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5748155950 0x7f574806e2e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.300+0000 7f574d732640 1 --2- 172.21.15.1:0/3025270145 >> v2:172.21.15.98:3300/0 conn(0x7f5748155030 0x7f5748155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.300+0000 7f574d732640 1 -- 172.21.15.1:0/3025270145 >> 172.21.15.1:0/3025270145 conn(0x7f57480a8e10 msgr2=0x7f57480c3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.300+0000 7f574d732640 1 -- 172.21.15.1:0/3025270145 shutdown_connections 2024-03-06T07:33:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.300+0000 7f574d732640 1 -- 172.21.15.1:0/3025270145 wait complete. 2024-03-06T07:33:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.301+0000 7f574d732640 1 Processor -- start 2024-03-06T07:33:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.301+0000 7f574d732640 1 -- start start 2024-03-06T07:33:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.302+0000 7f574d732640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5748071df0 0x7f57481eb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.302+0000 7f574d732640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5748155030 0x7f57480a9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.302+0000 7f574d732640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5748155950 0x7f57481ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.302+0000 7f574d732640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574814f530 con 0x7f5748071df0 2024-03-06T07:33:30.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.302+0000 7f574d732640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574814f3b0 con 0x7f5748155950 2024-03-06T07:33:30.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.302+0000 7f574d732640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57480c2320 con 0x7f5748155030 2024-03-06T07:33:30.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.302+0000 7f5746ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5748071df0 0x7f57481eb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.302+0000 7f5746ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5748071df0 0x7f57481eb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40714/0 (socket says 172.21.15.1:40714) 2024-03-06T07:33:30.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.302+0000 7f5746ffd640 1 -- 172.21.15.1:0/131566404 learned_addr learned my addr 172.21.15.1:0/131566404 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:30.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.302+0000 7f573ffff640 1 --2- 172.21.15.1:0/131566404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5748155030 0x7f57480a9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.302+0000 7f57477fe640 1 --2- 172.21.15.1:0/131566404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5748155950 0x7f57481ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.302+0000 7f5746ffd640 1 -- 172.21.15.1:0/131566404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5748155030 msgr2=0x7f57480a9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.302+0000 7f5746ffd640 1 --2- 172.21.15.1:0/131566404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5748155030 0x7f57480a9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.302+0000 7f5746ffd640 1 -- 172.21.15.1:0/131566404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5748155950 msgr2=0x7f57481ee590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.302+0000 7f5746ffd640 1 --2- 172.21.15.1:0/131566404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5748155950 0x7f57481ee590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.302+0000 7f5746ffd640 1 -- 172.21.15.1:0/131566404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57480c4740 con 0x7f5748071df0 2024-03-06T07:33:30.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.303+0000 7f5746ffd640 1 --2- 172.21.15.1:0/131566404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5748071df0 0x7f57481eb390 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f573800ec10 tx=0x7f573800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.303+0000 7f573f7fe640 1 -- 172.21.15.1:0/131566404 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738017070 con 0x7f5748071df0 2024-03-06T07:33:30.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.303+0000 7f573f7fe640 1 -- 172.21.15.1:0/131566404 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573800cc30 con 0x7f5748071df0 2024-03-06T07:33:30.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.303+0000 7f574d732640 1 -- 172.21.15.1:0/131566404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57480c4a20 con 0x7f5748071df0 2024-03-06T07:33:30.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.303+0000 7f573f7fe640 1 -- 172.21.15.1:0/131566404 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738013470 con 0x7f5748071df0 2024-03-06T07:33:30.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.303+0000 7f574d732640 1 -- 172.21.15.1:0/131566404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57480c4fe0 con 0x7f5748071df0 2024-03-06T07:33:30.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.304+0000 7f574d732640 1 -- 172.21.15.1:0/131566404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f57480c51e0 con 0x7f5748071df0 2024-03-06T07:33:30.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.304+0000 7f573f7fe640 1 -- 172.21.15.1:0/131566404 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f573801a070 con 0x7f5748071df0 2024-03-06T07:33:30.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.305+0000 7f573f7fe640 1 --2- 172.21.15.1:0/131566404 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f572806b4d0 0x7f572806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.305+0000 7f573f7fe640 1 -- 172.21.15.1:0/131566404 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f573808a3c0 con 0x7f5748071df0 2024-03-06T07:33:30.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.305+0000 7f573ffff640 1 --2- 172.21.15.1:0/131566404 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f572806b4d0 0x7f572806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.305+0000 7f573f7fe640 1 --2- 172.21.15.1:0/131566404 >> [v2:172.21.15.1:6817/682560127,v1:172.21.15.1:6819/682560127] conn(0x7f5728071b50 0x7f5728073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.305+0000 7f573f7fe640 1 -- 172.21.15.1:0/131566404 --> [v2:172.21.15.1:6817/682560127,v1:172.21.15.1:6819/682560127] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57380093e0 con 0x7f5728071b50 2024-03-06T07:33:30.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.305+0000 7f573f7fe640 1 -- 172.21.15.1:0/131566404 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=217) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5738091050 con 0x7f5748071df0 2024-03-06T07:33:30.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.305+0000 7f57477fe640 1 --2- 172.21.15.1:0/131566404 >> [v2:172.21.15.1:6817/682560127,v1:172.21.15.1:6819/682560127] conn(0x7f5728071b50 0x7f5728073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.305+0000 7f573ffff640 1 --2- 172.21.15.1:0/131566404 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f572806b4d0 0x7f572806d990 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f5748143640 tx=0x7f5730009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.305+0000 7f57477fe640 1 --2- 172.21.15.1:0/131566404 >> [v2:172.21.15.1:6817/682560127,v1:172.21.15.1:6819/682560127] conn(0x7f5728071b50 0x7f5728073f90 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.306+0000 7f573f7fe640 1 -- 172.21.15.1:0/131566404 <== osd.1 v2:172.21.15.1:6817/682560127 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f57380093e0 con 0x7f5728071b50 2024-03-06T07:33:30.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.319+0000 7f574d732640 1 -- 172.21.15.1:0/131566404 --> [v2:172.21.15.1:6817/682560127,v1:172.21.15.1:6819/682560127] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f57480680b0 con 0x7f5728071b50 2024-03-06T07:33:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.320+0000 7f573f7fe640 1 -- 172.21.15.1:0/131566404 <== osd.1 v2:172.21.15.1:6817/682560127 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f57480680b0 con 0x7f5728071b50 2024-03-06T07:33:30.322 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:33:30.322 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:33:30.322 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:33:30.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.320+0000 7f574d732640 1 -- 172.21.15.1:0/131566404 >> [v2:172.21.15.1:6817/682560127,v1:172.21.15.1:6819/682560127] conn(0x7f5728071b50 msgr2=0x7f5728073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.320+0000 7f574d732640 1 --2- 172.21.15.1:0/131566404 >> [v2:172.21.15.1:6817/682560127,v1:172.21.15.1:6819/682560127] conn(0x7f5728071b50 0x7f5728073f90 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.320+0000 7f574d732640 1 -- 172.21.15.1:0/131566404 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f572806b4d0 msgr2=0x7f572806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.320+0000 7f574d732640 1 --2- 172.21.15.1:0/131566404 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f572806b4d0 0x7f572806d990 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f5748143640 tx=0x7f5730009040 comp rx=0 tx=0).stop 2024-03-06T07:33:30.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.320+0000 7f574d732640 1 -- 172.21.15.1:0/131566404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5748071df0 msgr2=0x7f57481eb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.320+0000 7f574d732640 1 --2- 172.21.15.1:0/131566404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5748071df0 0x7f57481eb390 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f573800ec10 tx=0x7f573800c520 comp rx=0 tx=0).stop 2024-03-06T07:33:30.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.320+0000 7f57477fe640 1 -- 172.21.15.1:0/131566404 reap_dead start 2024-03-06T07:33:30.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.320+0000 7f574d732640 1 -- 172.21.15.1:0/131566404 shutdown_connections 2024-03-06T07:33:30.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.320+0000 7f574d732640 1 -- 172.21.15.1:0/131566404 >> 172.21.15.1:0/131566404 conn(0x7f57480a8e10 msgr2=0x7f574806e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.320+0000 7f574d732640 1 -- 172.21.15.1:0/131566404 shutdown_connections 2024-03-06T07:33:30.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.320+0000 7f574d732640 1 -- 172.21.15.1:0/131566404 wait complete. 2024-03-06T07:33:30.332 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-06T07:33:30.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.433+0000 7f6b9066b640 1 Processor -- start 2024-03-06T07:33:30.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.433+0000 7f6b9066b640 1 -- start start 2024-03-06T07:33:30.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.434+0000 7f6b9066b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b88150520 0x7f6b881543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.434+0000 7f6b9066b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b88150900 con 0x7f6b88070c80 2024-03-06T07:33:30.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.434+0000 7f6b9066b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b88071790 con 0x7f6b880c32f0 2024-03-06T07:33:30.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.434+0000 7f6b9066b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b88153ed0 con 0x7f6b88150520 2024-03-06T07:33:30.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.434+0000 7f6b8ec08640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6b88070c80 0x7f6b8815d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37118/0 (socket says 172.21.15.1:37118) 2024-03-06T07:33:30.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.434+0000 7f6b8ec08640 1 -- 172.21.15.1:0/958536928 learned_addr learned my addr 172.21.15.1:0/958536928 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:30.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.434+0000 7f6b8e407640 1 --2- 172.21.15.1:0/958536928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b88150520 0x7f6b881543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.434+0000 7f6b8d405640 1 -- 172.21.15.1:0/958536928 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633343727 0 0) 0x7f6b88150900 con 0x7f6b88070c80 2024-03-06T07:33:30.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.434+0000 7f6b8d405640 1 -- 172.21.15.1:0/958536928 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b70003520 con 0x7f6b88070c80 2024-03-06T07:33:30.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.435+0000 7f6b8d405640 1 -- 172.21.15.1:0/958536928 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290017191 0 0) 0x7f6b88071790 con 0x7f6b880c32f0 2024-03-06T07:33:30.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.435+0000 7f6b8d405640 1 -- 172.21.15.1:0/958536928 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b88150900 con 0x7f6b880c32f0 2024-03-06T07:33:30.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.435+0000 7f6b8e407640 1 -- 172.21.15.1:0/958536928 >> v1:172.21.15.98:6789/0 conn(0x7f6b880c32f0 legacy=0x7f6b881537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.435+0000 7f6b8e407640 1 -- 172.21.15.1:0/958536928 >> v1:172.21.15.1:6789/0 conn(0x7f6b88070c80 legacy=0x7f6b8815d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.435+0000 7f6b8e407640 1 -- 172.21.15.1:0/958536928 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b88157050 con 0x7f6b88150520 2024-03-06T07:33:30.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.435+0000 7f6b8e407640 1 --2- 172.21.15.1:0/958536928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b88150520 0x7f6b881543a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6b80009dc0 tx=0x7f6b8002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc01f3c7d8529dda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.436+0000 7f6b8d405640 1 -- 172.21.15.1:0/958536928 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b80039070 con 0x7f6b88150520 2024-03-06T07:33:30.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.436+0000 7f6b8d405640 1 -- 172.21.15.1:0/958536928 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8002cc40 con 0x7f6b88150520 2024-03-06T07:33:30.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.436+0000 7f6b8d405640 1 -- 172.21.15.1:0/958536928 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b80034440 con 0x7f6b88150520 2024-03-06T07:33:30.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.436+0000 7f6b9066b640 1 -- 172.21.15.1:0/958536928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b88150520 msgr2=0x7f6b881543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.436+0000 7f6b9066b640 1 --2- 172.21.15.1:0/958536928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b88150520 0x7f6b881543a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6b80009dc0 tx=0x7f6b8002be90 comp rx=0 tx=0).stop 2024-03-06T07:33:30.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.436+0000 7f6b9066b640 1 -- 172.21.15.1:0/958536928 shutdown_connections 2024-03-06T07:33:30.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.436+0000 7f6b9066b640 1 --2- 172.21.15.1:0/958536928 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b88150520 0x7f6b881543a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.436+0000 7f6b9066b640 1 -- 172.21.15.1:0/958536928 >> 172.21.15.1:0/958536928 conn(0x7f6b880c88f0 msgr2=0x7f6b880c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.436+0000 7f6b9066b640 1 -- 172.21.15.1:0/958536928 shutdown_connections 2024-03-06T07:33:30.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.436+0000 7f6b9066b640 1 -- 172.21.15.1:0/958536928 wait complete. 2024-03-06T07:33:30.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.437+0000 7f6b9066b640 1 Processor -- start 2024-03-06T07:33:30.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.437+0000 7f6b9066b640 1 -- start start 2024-03-06T07:33:30.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.438+0000 7f6b9066b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b88070c80 0x7f6b880ada10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.438+0000 7f6b9066b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b880c32f0 0x7f6b880c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.438+0000 7f6b9066b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b88150520 0x7f6b88149130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.438+0000 7f6b9066b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b88157330 con 0x7f6b88150520 2024-03-06T07:33:30.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.438+0000 7f6b9066b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b881532a0 con 0x7f6b880c32f0 2024-03-06T07:33:30.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.438+0000 7f6b9066b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b880c7ec0 con 0x7f6b88070c80 2024-03-06T07:33:30.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.438+0000 7f6b8ec08640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b88150520 0x7f6b88149130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.438+0000 7f6b8ec08640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b88150520 0x7f6b88149130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40724/0 (socket says 172.21.15.1:40724) 2024-03-06T07:33:30.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.438+0000 7f6b8ec08640 1 -- 172.21.15.1:0/1633021691 learned_addr learned my addr 172.21.15.1:0/1633021691 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:30.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.438+0000 7f6b8e407640 1 --2- 172.21.15.1:0/1633021691 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b88070c80 0x7f6b880ada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.438+0000 7f6b8dc06640 1 --2- 172.21.15.1:0/1633021691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b880c32f0 0x7f6b880c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.438+0000 7f6b8ec08640 1 -- 172.21.15.1:0/1633021691 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b88070c80 msgr2=0x7f6b880ada10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.438+0000 7f6b8ec08640 1 --2- 172.21.15.1:0/1633021691 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b88070c80 0x7f6b880ada10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.438+0000 7f6b8ec08640 1 -- 172.21.15.1:0/1633021691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b880c32f0 msgr2=0x7f6b880c1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.438+0000 7f6b8ec08640 1 --2- 172.21.15.1:0/1633021691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b880c32f0 0x7f6b880c1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.438+0000 7f6b8ec08640 1 -- 172.21.15.1:0/1633021691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b880abf50 con 0x7f6b88150520 2024-03-06T07:33:30.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.439+0000 7f6b8ec08640 1 --2- 172.21.15.1:0/1633021691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b88150520 0x7f6b88149130 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7f6b7800c460 tx=0x7f6b7800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.439+0000 7f6b86ffd640 1 -- 172.21.15.1:0/1633021691 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b78017070 con 0x7f6b88150520 2024-03-06T07:33:30.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.439+0000 7f6b86ffd640 1 -- 172.21.15.1:0/1633021691 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b78005ce0 con 0x7f6b88150520 2024-03-06T07:33:30.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.439+0000 7f6b9066b640 1 -- 172.21.15.1:0/1633021691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b880ac1c0 con 0x7f6b88150520 2024-03-06T07:33:30.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.439+0000 7f6b86ffd640 1 -- 172.21.15.1:0/1633021691 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b780072b0 con 0x7f6b88150520 2024-03-06T07:33:30.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.439+0000 7f6b9066b640 1 -- 172.21.15.1:0/1633021691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b880c24d0 con 0x7f6b88150520 2024-03-06T07:33:30.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.440+0000 7f6b9066b640 1 -- 172.21.15.1:0/1633021691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b880686e0 con 0x7f6b88150520 2024-03-06T07:33:30.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.440+0000 7f6b86ffd640 1 -- 172.21.15.1:0/1633021691 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b7801a070 con 0x7f6b88150520 2024-03-06T07:33:30.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.441+0000 7f6b86ffd640 1 --2- 172.21.15.1:0/1633021691 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b7406b500 0x7f6b7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.441+0000 7f6b86ffd640 1 -- 172.21.15.1:0/1633021691 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f6b78089430 con 0x7f6b88150520 2024-03-06T07:33:30.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.441+0000 7f6b8e407640 1 --2- 172.21.15.1:0/1633021691 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b7406b500 0x7f6b7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.441+0000 7f6b8e407640 1 --2- 172.21.15.1:0/1633021691 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b7406b500 0x7f6b7406d9c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f6b80036000 tx=0x7f6b80030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.449+0000 7f6b86ffd640 1 -- 172.21.15.1:0/1633021691 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b78057cd0 con 0x7f6b88150520 2024-03-06T07:33:30.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.633+0000 7f6b9066b640 1 -- 172.21.15.1:0/1633021691 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b8806c900 con 0x7f6b7406b500 2024-03-06T07:33:30.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.633+0000 7f6b86ffd640 1 -- 172.21.15.1:0/1633021691 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6b8806c900 con 0x7f6b7406b500 2024-03-06T07:33:30.635 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 1 to deep-scrub 2024-03-06T07:33:30.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.635+0000 7f6b9066b640 1 -- 172.21.15.1:0/1633021691 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b7406b500 msgr2=0x7f6b7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.635+0000 7f6b9066b640 1 --2- 172.21.15.1:0/1633021691 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b7406b500 0x7f6b7406d9c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f6b80036000 tx=0x7f6b80030000 comp rx=0 tx=0).stop 2024-03-06T07:33:30.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.635+0000 7f6b9066b640 1 -- 172.21.15.1:0/1633021691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b88150520 msgr2=0x7f6b88149130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.635+0000 7f6b9066b640 1 --2- 172.21.15.1:0/1633021691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b88150520 0x7f6b88149130 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7f6b7800c460 tx=0x7f6b7800c920 comp rx=0 tx=0).stop 2024-03-06T07:33:30.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.635+0000 7f6b9066b640 1 -- 172.21.15.1:0/1633021691 shutdown_connections 2024-03-06T07:33:30.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.635+0000 7f6b9066b640 1 --2- 172.21.15.1:0/1633021691 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6b7406b500 0x7f6b7406d9c0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.635+0000 7f6b9066b640 1 --2- 172.21.15.1:0/1633021691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b88150520 0x7f6b88149130 unknown :-1 s=CLOSED pgs=2732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.635+0000 7f6b9066b640 1 --2- 172.21.15.1:0/1633021691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b880c32f0 0x7f6b880c1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.635+0000 7f6b9066b640 1 --2- 172.21.15.1:0/1633021691 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b88070c80 0x7f6b880ada10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.635+0000 7f6b9066b640 1 -- 172.21.15.1:0/1633021691 >> 172.21.15.1:0/1633021691 conn(0x7f6b880c88f0 msgr2=0x7f6b8814c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.635+0000 7f6b9066b640 1 -- 172.21.15.1:0/1633021691 shutdown_connections 2024-03-06T07:33:30.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.635+0000 7f6b9066b640 1 -- 172.21.15.1:0/1633021691 wait complete. 2024-03-06T07:33:30.648 INFO:tasks.ceph:Scrubbing osd.2 2024-03-06T07:33:30.648 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:33:30.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.751+0000 7fa56ca17640 1 Processor -- start 2024-03-06T07:33:30.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.751+0000 7fa56ca17640 1 -- start start 2024-03-06T07:33:30.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.752+0000 7fa56ca17640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa5680c70c0 0x7fa568155550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.752+0000 7fa56ca17640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa568155a90 0x7fa56815fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.752+0000 7fa56ca17640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa568158890 con 0x7fa5680c16e0 2024-03-06T07:33:30.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.752+0000 7fa56ca17640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa568158b70 con 0x7fa5680c70c0 2024-03-06T07:33:30.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.752+0000 7fa56ca17640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa568158e50 con 0x7fa568155a90 2024-03-06T07:33:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.752+0000 7fa565d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa5680c70c0 0x7fa568155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.752+0000 7fa566d9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa568155a90 0x7fa56815fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.752+0000 7fa56659c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa5680c16e0 0x7fa568152470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:39910/0 (socket says 172.21.15.1:39910) 2024-03-06T07:33:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.752+0000 7fa56659c640 1 -- 172.21.15.1:0/59384948 learned_addr learned my addr 172.21.15.1:0/59384948 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.752+0000 7fa565d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa5680c70c0 0x7fa568155550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:53424/0 (socket says 172.21.15.1:53424) 2024-03-06T07:33:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.753+0000 7fa565d9b640 1 -- 172.21.15.1:0/59384948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa568155a90 msgr2=0x7fa56815fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.753+0000 7fa565d9b640 1 --2- 172.21.15.1:0/59384948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa568155a90 0x7fa56815fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.753+0000 7fa565d9b640 1 -- 172.21.15.1:0/59384948 >> v1:172.21.15.98:6789/0 conn(0x7fa5680c16e0 legacy=0x7fa568152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.753+0000 7fa565d9b640 1 -- 172.21.15.1:0/59384948 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa568159130 con 0x7fa5680c70c0 2024-03-06T07:33:30.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.753+0000 7fa56559a640 1 -- 172.21.15.1:0/59384948 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308802853 0 0) 0x7fa568158890 con 0x7fa5680c16e0 2024-03-06T07:33:30.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.753+0000 7fa566d9d640 1 --2- 172.21.15.1:0/59384948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa568155a90 0x7fa56815fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:30.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.753+0000 7fa565d9b640 1 --2- 172.21.15.1:0/59384948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5680c70c0 0x7fa568155550 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fa5500097a0 tx=0x7fa55002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4509a4e6e42ceb8a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.754+0000 7fa56559a640 1 -- 172.21.15.1:0/59384948 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5500315a0 con 0x7fa5680c70c0 2024-03-06T07:33:30.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.754+0000 7fa56559a640 1 -- 172.21.15.1:0/59384948 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa550037070 con 0x7fa5680c70c0 2024-03-06T07:33:30.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.754+0000 7fa56559a640 1 -- 172.21.15.1:0/59384948 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa550031960 con 0x7fa5680c70c0 2024-03-06T07:33:30.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.754+0000 7fa56ca17640 1 -- 172.21.15.1:0/59384948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5680c70c0 msgr2=0x7fa568155550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.754+0000 7fa56ca17640 1 --2- 172.21.15.1:0/59384948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5680c70c0 0x7fa568155550 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fa5500097a0 tx=0x7fa55002b840 comp rx=0 tx=0).stop 2024-03-06T07:33:30.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.754+0000 7fa56ca17640 1 -- 172.21.15.1:0/59384948 shutdown_connections 2024-03-06T07:33:30.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.754+0000 7fa56ca17640 1 --2- 172.21.15.1:0/59384948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa568155a90 0x7fa56815fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.754+0000 7fa56ca17640 1 --2- 172.21.15.1:0/59384948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5680c70c0 0x7fa568155550 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.754+0000 7fa56ca17640 1 -- 172.21.15.1:0/59384948 >> 172.21.15.1:0/59384948 conn(0x7fa568066d00 msgr2=0x7fa568067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.754+0000 7fa56ca17640 1 -- 172.21.15.1:0/59384948 shutdown_connections 2024-03-06T07:33:30.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.754+0000 7fa56ca17640 1 -- 172.21.15.1:0/59384948 wait complete. 2024-03-06T07:33:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.756+0000 7fa56ca17640 1 Processor -- start 2024-03-06T07:33:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.756+0000 7fa56ca17640 1 -- start start 2024-03-06T07:33:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.756+0000 7fa56ca17640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5680c16e0 0x7fa568153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.756+0000 7fa56ca17640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5680c70c0 0x7fa5680c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.756+0000 7fa56ca17640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa568155a90 0x7fa5680c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.756+0000 7fa56ca17640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5680c68c0 con 0x7fa5680c70c0 2024-03-06T07:33:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.756+0000 7fa56ca17640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5681596f0 con 0x7fa5680c16e0 2024-03-06T07:33:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.756+0000 7fa56ca17640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5680cb470 con 0x7fa568155a90 2024-03-06T07:33:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.756+0000 7fa565d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5680c70c0 0x7fa5680c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.756+0000 7fa565d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5680c70c0 0x7fa5680c91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40726/0 (socket says 172.21.15.1:40726) 2024-03-06T07:33:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.756+0000 7fa565d9b640 1 -- 172.21.15.1:0/4280982859 learned_addr learned my addr 172.21.15.1:0/4280982859 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.756+0000 7fa56659c640 1 --2- 172.21.15.1:0/4280982859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5680c16e0 0x7fa568153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.756+0000 7fa566d9d640 1 --2- 172.21.15.1:0/4280982859 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa568155a90 0x7fa5680c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.757+0000 7fa565d9b640 1 -- 172.21.15.1:0/4280982859 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa568155a90 msgr2=0x7fa5680c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.757+0000 7fa565d9b640 1 --2- 172.21.15.1:0/4280982859 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa568155a90 0x7fa5680c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.757+0000 7fa565d9b640 1 -- 172.21.15.1:0/4280982859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5680c16e0 msgr2=0x7fa568153900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.757+0000 7fa565d9b640 1 --2- 172.21.15.1:0/4280982859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5680c16e0 0x7fa568153900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.757+0000 7fa565d9b640 1 -- 172.21.15.1:0/4280982859 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5681ebe40 con 0x7fa5680c70c0 2024-03-06T07:33:30.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.757+0000 7fa56659c640 1 --2- 172.21.15.1:0/4280982859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5680c16e0 0x7fa568153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:30.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.757+0000 7fa565d9b640 1 --2- 172.21.15.1:0/4280982859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5680c70c0 0x7fa5680c91a0 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7fa5500097a0 tx=0x7fa5500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.757+0000 7fa55effd640 1 -- 172.21.15.1:0/4280982859 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa550004030 con 0x7fa5680c70c0 2024-03-06T07:33:30.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.757+0000 7fa55effd640 1 -- 172.21.15.1:0/4280982859 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5500041d0 con 0x7fa5680c70c0 2024-03-06T07:33:30.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.757+0000 7fa56ca17640 1 -- 172.21.15.1:0/4280982859 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5680ca2f0 con 0x7fa5680c70c0 2024-03-06T07:33:30.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.757+0000 7fa55effd640 1 -- 172.21.15.1:0/4280982859 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa550003610 con 0x7fa5680c70c0 2024-03-06T07:33:30.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.757+0000 7fa56ca17640 1 -- 172.21.15.1:0/4280982859 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5680c8c80 con 0x7fa5680c70c0 2024-03-06T07:33:30.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.758+0000 7fa56ca17640 1 -- 172.21.15.1:0/4280982859 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa5680ca810 con 0x7fa5680c70c0 2024-03-06T07:33:30.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.759+0000 7fa55effd640 1 -- 172.21.15.1:0/4280982859 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5500037b0 con 0x7fa5680c70c0 2024-03-06T07:33:30.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.759+0000 7fa55effd640 1 --2- 172.21.15.1:0/4280982859 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa54806b4d0 0x7fa54806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.760+0000 7fa55effd640 1 -- 172.21.15.1:0/4280982859 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fa55002f030 con 0x7fa5680c70c0 2024-03-06T07:33:30.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.760+0000 7fa56659c640 1 --2- 172.21.15.1:0/4280982859 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa54806b4d0 0x7fa54806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.760+0000 7fa55effd640 1 --2- 172.21.15.1:0/4280982859 >> [v2:172.21.15.1:6800/113114752,v1:172.21.15.1:6801/113114752] conn(0x7fa548071b50 0x7fa548073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.760+0000 7fa55effd640 1 -- 172.21.15.1:0/4280982859 --> [v2:172.21.15.1:6800/113114752,v1:172.21.15.1:6801/113114752] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa550050ec0 con 0x7fa548071b50 2024-03-06T07:33:30.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.760+0000 7fa55effd640 1 -- 172.21.15.1:0/4280982859 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=217) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5500ac610 con 0x7fa5680c70c0 2024-03-06T07:33:30.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.760+0000 7fa566d9d640 1 --2- 172.21.15.1:0/4280982859 >> [v2:172.21.15.1:6800/113114752,v1:172.21.15.1:6801/113114752] conn(0x7fa548071b50 0x7fa548073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.760+0000 7fa56659c640 1 --2- 172.21.15.1:0/4280982859 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa54806b4d0 0x7fa54806d990 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fa558000f60 tx=0x7fa558007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.760+0000 7fa566d9d640 1 --2- 172.21.15.1:0/4280982859 >> [v2:172.21.15.1:6800/113114752,v1:172.21.15.1:6801/113114752] conn(0x7fa548071b50 0x7fa548073f90 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.761+0000 7fa55effd640 1 -- 172.21.15.1:0/4280982859 <== osd.2 v2:172.21.15.1:6800/113114752 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa550050ec0 con 0x7fa548071b50 2024-03-06T07:33:30.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.774+0000 7fa56ca17640 1 -- 172.21.15.1:0/4280982859 --> [v2:172.21.15.1:6800/113114752,v1:172.21.15.1:6801/113114752] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa5680680b0 con 0x7fa548071b50 2024-03-06T07:33:30.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.774+0000 7fa55effd640 1 -- 172.21.15.1:0/4280982859 <== osd.2 v2:172.21.15.1:6800/113114752 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fa5680680b0 con 0x7fa548071b50 2024-03-06T07:33:30.777 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:33:30.777 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:33:30.777 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:33:30.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.774+0000 7fa56ca17640 1 -- 172.21.15.1:0/4280982859 >> [v2:172.21.15.1:6800/113114752,v1:172.21.15.1:6801/113114752] conn(0x7fa548071b50 msgr2=0x7fa548073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.774+0000 7fa56ca17640 1 --2- 172.21.15.1:0/4280982859 >> [v2:172.21.15.1:6800/113114752,v1:172.21.15.1:6801/113114752] conn(0x7fa548071b50 0x7fa548073f90 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.775+0000 7fa56ca17640 1 -- 172.21.15.1:0/4280982859 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa54806b4d0 msgr2=0x7fa54806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.775+0000 7fa56ca17640 1 --2- 172.21.15.1:0/4280982859 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fa54806b4d0 0x7fa54806d990 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fa558000f60 tx=0x7fa558007d50 comp rx=0 tx=0).stop 2024-03-06T07:33:30.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.775+0000 7fa56ca17640 1 -- 172.21.15.1:0/4280982859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5680c70c0 msgr2=0x7fa5680c91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.775+0000 7fa56ca17640 1 --2- 172.21.15.1:0/4280982859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5680c70c0 0x7fa5680c91a0 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7fa5500097a0 tx=0x7fa5500046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.775+0000 7fa566d9d640 1 -- 172.21.15.1:0/4280982859 reap_dead start 2024-03-06T07:33:30.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.775+0000 7fa56ca17640 1 -- 172.21.15.1:0/4280982859 shutdown_connections 2024-03-06T07:33:30.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.775+0000 7fa56ca17640 1 -- 172.21.15.1:0/4280982859 >> 172.21.15.1:0/4280982859 conn(0x7fa568066d00 msgr2=0x7fa5680670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.775+0000 7fa56ca17640 1 -- 172.21.15.1:0/4280982859 shutdown_connections 2024-03-06T07:33:30.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.775+0000 7fa56ca17640 1 -- 172.21.15.1:0/4280982859 wait complete. 2024-03-06T07:33:30.788 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-06T07:33:30.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.888+0000 7f5d619f7640 1 Processor -- start 2024-03-06T07:33:30.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.888+0000 7f5d619f7640 1 -- start start 2024-03-06T07:33:30.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.889+0000 7f5d619f7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5d5c150fe0 0x7f5d5c1513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.889+0000 7f5d619f7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5d5c071f90 0x7f5d5c072400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.889+0000 7f5d619f7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5c0a6140 con 0x7f5d5c072f20 2024-03-06T07:33:30.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.889+0000 7f5d619f7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d5c1526b0 con 0x7f5d5c071f90 2024-03-06T07:33:30.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.889+0000 7f5d619f7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d5c151aa0 con 0x7f5d5c150fe0 2024-03-06T07:33:30.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.889+0000 7f5d5b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5d5c071f90 0x7f5d5c072400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.889+0000 7f5d5b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5d5c071f90 0x7f5d5c072400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40730/0 (socket says 172.21.15.1:40730) 2024-03-06T07:33:30.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.889+0000 7f5d5b7fe640 1 -- 172.21.15.1:0/3817078697 learned_addr learned my addr 172.21.15.1:0/3817078697 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:30.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.889+0000 7f5d5affd640 1 --2- 172.21.15.1:0/3817078697 >> v2:172.21.15.98:3300/0 conn(0x7f5d5c150fe0 0x7f5d5c1513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.890+0000 7f5d5b7fe640 1 -- 172.21.15.1:0/3817078697 >> v2:172.21.15.98:3300/0 conn(0x7f5d5c150fe0 msgr2=0x7f5d5c1513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:30.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.890+0000 7f5d5b7fe640 1 --2- 172.21.15.1:0/3817078697 >> v2:172.21.15.98:3300/0 conn(0x7f5d5c150fe0 0x7f5d5c1513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.890+0000 7f5d5b7fe640 1 -- 172.21.15.1:0/3817078697 >> v1:172.21.15.98:6789/0 conn(0x7f5d5c072f20 legacy=0x7f5d5c06edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.890+0000 7f5d5b7fe640 1 -- 172.21.15.1:0/3817078697 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5c151d80 con 0x7f5d5c071f90 2024-03-06T07:33:30.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.890+0000 7f5d5b7fe640 1 --2- 172.21.15.1:0/3817078697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c071f90 0x7f5d5c072400 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c06fba0 tx=0x7f5d5402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d7c0d4ba0d0e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.891+0000 7f5d59ffb640 1 -- 172.21.15.1:0/3817078697 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5402ca80 con 0x7f5d5c071f90 2024-03-06T07:33:30.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.891+0000 7f5d59ffb640 1 -- 172.21.15.1:0/3817078697 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d5402cc20 con 0x7f5d5c071f90 2024-03-06T07:33:30.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.891+0000 7f5d59ffb640 1 -- 172.21.15.1:0/3817078697 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d540343f0 con 0x7f5d5c071f90 2024-03-06T07:33:30.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.891+0000 7f5d619f7640 1 -- 172.21.15.1:0/3817078697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c071f90 msgr2=0x7f5d5c072400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.891+0000 7f5d619f7640 1 --2- 172.21.15.1:0/3817078697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c071f90 0x7f5d5c072400 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c06fba0 tx=0x7f5d5402bee0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.891+0000 7f5d619f7640 1 -- 172.21.15.1:0/3817078697 shutdown_connections 2024-03-06T07:33:30.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.891+0000 7f5d619f7640 1 --2- 172.21.15.1:0/3817078697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c071f90 0x7f5d5c072400 unknown :-1 s=CLOSED pgs=2734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.891+0000 7f5d619f7640 1 --2- 172.21.15.1:0/3817078697 >> v2:172.21.15.98:3300/0 conn(0x7f5d5c150fe0 0x7f5d5c1513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.891+0000 7f5d619f7640 1 -- 172.21.15.1:0/3817078697 >> 172.21.15.1:0/3817078697 conn(0x7f5d5c0c6860 msgr2=0x7f5d5c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.891+0000 7f5d619f7640 1 -- 172.21.15.1:0/3817078697 shutdown_connections 2024-03-06T07:33:30.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.891+0000 7f5d619f7640 1 -- 172.21.15.1:0/3817078697 wait complete. 2024-03-06T07:33:30.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.892+0000 7f5d619f7640 1 Processor -- start 2024-03-06T07:33:30.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.892+0000 7f5d619f7640 1 -- start start 2024-03-06T07:33:30.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.893+0000 7f5d619f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c071f90 0x7f5d5c1475d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.893+0000 7f5d619f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d5c072f20 0x7f5d5c0bd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.893+0000 7f5d619f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d5c150fe0 0x7f5d5c1ec6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.893+0000 7f5d619f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c06f4c0 con 0x7f5d5c071f90 2024-03-06T07:33:30.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.893+0000 7f5d619f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c152340 con 0x7f5d5c072f20 2024-03-06T07:33:30.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.893+0000 7f5d619f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c06f640 con 0x7f5d5c150fe0 2024-03-06T07:33:30.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.893+0000 7f5d5affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c071f90 0x7f5d5c1475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.893+0000 7f5d5affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c071f90 0x7f5d5c1475d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40738/0 (socket says 172.21.15.1:40738) 2024-03-06T07:33:30.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.893+0000 7f5d5affd640 1 -- 172.21.15.1:0/2961002141 learned_addr learned my addr 172.21.15.1:0/2961002141 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:30.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.893+0000 7f5d5a7fc640 1 --2- 172.21.15.1:0/2961002141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d5c072f20 0x7f5d5c0bd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.893+0000 7f5d5b7fe640 1 --2- 172.21.15.1:0/2961002141 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d5c150fe0 0x7f5d5c1ec6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.894+0000 7f5d5affd640 1 -- 172.21.15.1:0/2961002141 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d5c150fe0 msgr2=0x7f5d5c1ec6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.894+0000 7f5d5affd640 1 --2- 172.21.15.1:0/2961002141 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d5c150fe0 0x7f5d5c1ec6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.894+0000 7f5d5affd640 1 -- 172.21.15.1:0/2961002141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d5c072f20 msgr2=0x7f5d5c0bd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.894+0000 7f5d5affd640 1 --2- 172.21.15.1:0/2961002141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d5c072f20 0x7f5d5c0bd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.894+0000 7f5d5affd640 1 -- 172.21.15.1:0/2961002141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5c0bd480 con 0x7f5d5c071f90 2024-03-06T07:33:30.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.894+0000 7f5d5affd640 1 --2- 172.21.15.1:0/2961002141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c071f90 0x7f5d5c1475d0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000e9b0 tx=0x7f5d5000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.894+0000 7f5d43fff640 1 -- 172.21.15.1:0/2961002141 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d50017070 con 0x7f5d5c071f90 2024-03-06T07:33:30.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.894+0000 7f5d43fff640 1 -- 172.21.15.1:0/2961002141 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d5000c9a0 con 0x7f5d5c071f90 2024-03-06T07:33:30.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.894+0000 7f5d619f7640 1 -- 172.21.15.1:0/2961002141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d5c0c52e0 con 0x7f5d5c071f90 2024-03-06T07:33:30.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.894+0000 7f5d43fff640 1 -- 172.21.15.1:0/2961002141 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5000cd90 con 0x7f5d5c071f90 2024-03-06T07:33:30.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.894+0000 7f5d619f7640 1 -- 172.21.15.1:0/2961002141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d5c0bf7a0 con 0x7f5d5c071f90 2024-03-06T07:33:30.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.895+0000 7f5d619f7640 1 -- 172.21.15.1:0/2961002141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d5c067fd0 con 0x7f5d5c071f90 2024-03-06T07:33:30.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.895+0000 7f5d43fff640 1 -- 172.21.15.1:0/2961002141 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d5001a070 con 0x7f5d5c071f90 2024-03-06T07:33:30.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.896+0000 7f5d43fff640 1 --2- 172.21.15.1:0/2961002141 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d4806b4d0 0x7f5d4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.896+0000 7f5d43fff640 1 -- 172.21.15.1:0/2961002141 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f5d5008a090 con 0x7f5d5c071f90 2024-03-06T07:33:30.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.896+0000 7f5d5a7fc640 1 --2- 172.21.15.1:0/2961002141 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d4806b4d0 0x7f5d4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.896+0000 7f5d5a7fc640 1 --2- 172.21.15.1:0/2961002141 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d4806b4d0 0x7f5d4806d990 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c0bf240 tx=0x7f5d4c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:30.904+0000 7f5d43fff640 1 -- 172.21.15.1:0/2961002141 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d500589c0 con 0x7f5d5c071f90 2024-03-06T07:33:31.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.086+0000 7f5d619f7640 1 -- 172.21.15.1:0/2961002141 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d5c0a5d30 con 0x7f5d4806b4d0 2024-03-06T07:33:31.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.087+0000 7f5d43fff640 1 -- 172.21.15.1:0/2961002141 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5d5c0a5d30 con 0x7f5d4806b4d0 2024-03-06T07:33:31.089 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 2 to deep-scrub 2024-03-06T07:33:31.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.090+0000 7f5d619f7640 1 -- 172.21.15.1:0/2961002141 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d4806b4d0 msgr2=0x7f5d4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.090+0000 7f5d619f7640 1 --2- 172.21.15.1:0/2961002141 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d4806b4d0 0x7f5d4806d990 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c0bf240 tx=0x7f5d4c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:31.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.090+0000 7f5d619f7640 1 -- 172.21.15.1:0/2961002141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c071f90 msgr2=0x7f5d5c1475d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.090+0000 7f5d619f7640 1 --2- 172.21.15.1:0/2961002141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c071f90 0x7f5d5c1475d0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000e9b0 tx=0x7f5d5000ee70 comp rx=0 tx=0).stop 2024-03-06T07:33:31.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.090+0000 7f5d619f7640 1 -- 172.21.15.1:0/2961002141 shutdown_connections 2024-03-06T07:33:31.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.090+0000 7f5d619f7640 1 --2- 172.21.15.1:0/2961002141 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f5d4806b4d0 0x7f5d4806d990 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.090+0000 7f5d619f7640 1 --2- 172.21.15.1:0/2961002141 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d5c150fe0 0x7f5d5c1ec6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.090+0000 7f5d619f7640 1 --2- 172.21.15.1:0/2961002141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d5c072f20 0x7f5d5c0bd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.090+0000 7f5d619f7640 1 --2- 172.21.15.1:0/2961002141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c071f90 0x7f5d5c1475d0 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.090+0000 7f5d619f7640 1 -- 172.21.15.1:0/2961002141 >> 172.21.15.1:0/2961002141 conn(0x7f5d5c0c6860 msgr2=0x7f5d5c0c3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.090+0000 7f5d619f7640 1 -- 172.21.15.1:0/2961002141 shutdown_connections 2024-03-06T07:33:31.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.090+0000 7f5d619f7640 1 -- 172.21.15.1:0/2961002141 wait complete. 2024-03-06T07:33:31.102 INFO:tasks.ceph:Scrubbing osd.3 2024-03-06T07:33:31.102 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:33:31.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.206+0000 7f29121f7640 1 Processor -- start 2024-03-06T07:33:31.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.206+0000 7f29121f7640 1 -- start start 2024-03-06T07:33:31.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.207+0000 7f29121f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f290c0c5950 0x7f290c0c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.207+0000 7f29121f7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f290c06eae0 0x7f290c06ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.207+0000 7f29121f7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f290c0728c0 con 0x7f290c0a60b0 2024-03-06T07:33:31.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.207+0000 7f29121f7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f290c072690 con 0x7f290c06eae0 2024-03-06T07:33:31.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.207+0000 7f29121f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f290c072ba0 con 0x7f290c0c5950 2024-03-06T07:33:31.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.207+0000 7f290bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f290c06eae0 0x7f290c06ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.207+0000 7f290bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f290c06eae0 0x7f290c06ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40748/0 (socket says 172.21.15.1:40748) 2024-03-06T07:33:31.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.207+0000 7f290bfff640 1 -- 172.21.15.1:0/3326701656 learned_addr learned my addr 172.21.15.1:0/3326701656 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:31.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.207+0000 7f290b7fe640 1 --2- 172.21.15.1:0/3326701656 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f290c0c5950 0x7f290c0c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.208+0000 7f290bfff640 1 -- 172.21.15.1:0/3326701656 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f290c0c5950 msgr2=0x7f290c0c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:31.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.208+0000 7f290bfff640 1 --2- 172.21.15.1:0/3326701656 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f290c0c5950 0x7f290c0c5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.208+0000 7f290bfff640 1 -- 172.21.15.1:0/3326701656 >> v1:172.21.15.98:6789/0 conn(0x7f290c0a60b0 legacy=0x7f290c152910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.208+0000 7f290bfff640 1 -- 172.21.15.1:0/3326701656 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c072e80 con 0x7f290c06eae0 2024-03-06T07:33:31.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.208+0000 7f290bfff640 1 --2- 172.21.15.1:0/3326701656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f290c06eae0 0x7f290c06ef90 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7f2904009e50 tx=0x7f290402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c6ee9c4b9b129d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.208+0000 7f290a7fc640 1 -- 172.21.15.1:0/3326701656 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290402cb00 con 0x7f290c06eae0 2024-03-06T07:33:31.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.209+0000 7f290a7fc640 1 -- 172.21.15.1:0/3326701656 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290402cca0 con 0x7f290c06eae0 2024-03-06T07:33:31.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.209+0000 7f290a7fc640 1 -- 172.21.15.1:0/3326701656 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2904034610 con 0x7f290c06eae0 2024-03-06T07:33:31.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.209+0000 7f29121f7640 1 -- 172.21.15.1:0/3326701656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f290c06eae0 msgr2=0x7f290c06ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.209+0000 7f29121f7640 1 --2- 172.21.15.1:0/3326701656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f290c06eae0 0x7f290c06ef90 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7f2904009e50 tx=0x7f290402bf60 comp rx=0 tx=0).stop 2024-03-06T07:33:31.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.209+0000 7f29121f7640 1 -- 172.21.15.1:0/3326701656 shutdown_connections 2024-03-06T07:33:31.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.209+0000 7f29121f7640 1 --2- 172.21.15.1:0/3326701656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f290c06eae0 0x7f290c06ef90 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.209+0000 7f29121f7640 1 --2- 172.21.15.1:0/3326701656 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f290c0c5950 0x7f290c0c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.209+0000 7f29121f7640 1 -- 172.21.15.1:0/3326701656 >> 172.21.15.1:0/3326701656 conn(0x7f290c0a8e10 msgr2=0x7f290c0c3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.209+0000 7f29121f7640 1 -- 172.21.15.1:0/3326701656 shutdown_connections 2024-03-06T07:33:31.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.209+0000 7f29121f7640 1 -- 172.21.15.1:0/3326701656 wait complete. 2024-03-06T07:33:31.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.210+0000 7f29121f7640 1 Processor -- start 2024-03-06T07:33:31.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.210+0000 7f29121f7640 1 -- start start 2024-03-06T07:33:31.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.211+0000 7f29121f7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f290c06eae0 0x7f290c0a6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.211+0000 7f29121f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f290c0a60b0 0x7f290c1db750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.211+0000 7f29121f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f290c0c5950 0x7f290c145570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.211+0000 7f29121f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290c0c18a0 con 0x7f290c0c5950 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.211+0000 7f29121f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290c0c1720 con 0x7f290c0a60b0 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.211+0000 7f29121f7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f290c0c1a20 con 0x7f290c06eae0 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.211+0000 7f290bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f290c0c5950 0x7f290c145570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.211+0000 7f290b7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f290c06eae0 0x7f290c0a6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.211+0000 7f290b7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f290c06eae0 0x7f290c0a6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.1:36212/0 (socket says 172.21.15.1:36212) 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.211+0000 7f290b7fe640 1 -- 172.21.15.1:0/1361367466 learned_addr learned my addr 172.21.15.1:0/1361367466 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.211+0000 7f290bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f290c0c5950 0x7f290c145570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40750/0 (socket says 172.21.15.1:40750) 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.211+0000 7f290affd640 1 --2- 172.21.15.1:0/1361367466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f290c0a60b0 0x7f290c1db750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.211+0000 7f290bfff640 1 -- 172.21.15.1:0/1361367466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f290c06eae0 msgr2=0x7f290c0a6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.211+0000 7f290bfff640 1 --2- 172.21.15.1:0/1361367466 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f290c06eae0 0x7f290c0a6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.212+0000 7f290bfff640 1 -- 172.21.15.1:0/1361367466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f290c0a60b0 msgr2=0x7f290c1db750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.212+0000 7f290bfff640 1 --2- 172.21.15.1:0/1361367466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f290c0a60b0 0x7f290c1db750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.212+0000 7f290bfff640 1 -- 172.21.15.1:0/1361367466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c0ac5a0 con 0x7f290c0c5950 2024-03-06T07:33:31.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.212+0000 7f290bfff640 1 --2- 172.21.15.1:0/1361367466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f290c0c5950 0x7f290c145570 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7f2904032000 tx=0x7f2904004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.212+0000 7f28f3fff640 1 -- 172.21.15.1:0/1361367466 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2904043020 con 0x7f290c0c5950 2024-03-06T07:33:31.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.212+0000 7f28f3fff640 1 -- 172.21.15.1:0/1361367466 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290402ce50 con 0x7f290c0c5950 2024-03-06T07:33:31.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.212+0000 7f29121f7640 1 -- 172.21.15.1:0/1361367466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f290c146160 con 0x7f290c0c5950 2024-03-06T07:33:31.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.212+0000 7f28f3fff640 1 -- 172.21.15.1:0/1361367466 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290403e6b0 con 0x7f290c0c5950 2024-03-06T07:33:31.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.212+0000 7f29121f7640 1 -- 172.21.15.1:0/1361367466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290c0ac880 con 0x7f290c0c5950 2024-03-06T07:33:31.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.213+0000 7f29121f7640 1 -- 172.21.15.1:0/1361367466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f290c0680b0 con 0x7f290c0c5950 2024-03-06T07:33:31.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.213+0000 7f28f3fff640 1 -- 172.21.15.1:0/1361367466 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2904046020 con 0x7f290c0c5950 2024-03-06T07:33:31.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.214+0000 7f28f3fff640 1 --2- 172.21.15.1:0/1361367466 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f28f806b4d0 0x7f28f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.214+0000 7f28f3fff640 1 -- 172.21.15.1:0/1361367466 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f29040ac5d0 con 0x7f290c0c5950 2024-03-06T07:33:31.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.214+0000 7f290b7fe640 1 --2- 172.21.15.1:0/1361367466 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f28f806b4d0 0x7f28f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.214+0000 7f28f3fff640 1 --2- 172.21.15.1:0/1361367466 >> [v2:172.21.15.1:6808/1031158088,v1:172.21.15.1:6809/1031158088] conn(0x7f28f8071b50 0x7f28f8073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.214+0000 7f28f3fff640 1 -- 172.21.15.1:0/1361367466 --> [v2:172.21.15.1:6808/1031158088,v1:172.21.15.1:6809/1031158088] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28f8074660 con 0x7f28f8071b50 2024-03-06T07:33:31.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.214+0000 7f28f3fff640 1 -- 172.21.15.1:0/1361367466 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=217) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29040b3050 con 0x7f290c0c5950 2024-03-06T07:33:31.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.214+0000 7f290affd640 1 --2- 172.21.15.1:0/1361367466 >> [v2:172.21.15.1:6808/1031158088,v1:172.21.15.1:6809/1031158088] conn(0x7f28f8071b50 0x7f28f8073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.214+0000 7f290b7fe640 1 --2- 172.21.15.1:0/1361367466 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f28f806b4d0 0x7f28f806d990 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f28fc006fd0 tx=0x7f28fc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.214+0000 7f290affd640 1 --2- 172.21.15.1:0/1361367466 >> [v2:172.21.15.1:6808/1031158088,v1:172.21.15.1:6809/1031158088] conn(0x7f28f8071b50 0x7f28f8073f90 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.215+0000 7f28f3fff640 1 -- 172.21.15.1:0/1361367466 <== osd.3 v2:172.21.15.1:6808/1031158088 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f28f8074660 con 0x7f28f8071b50 2024-03-06T07:33:31.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.228+0000 7f29121f7640 1 -- 172.21.15.1:0/1361367466 --> [v2:172.21.15.1:6808/1031158088,v1:172.21.15.1:6809/1031158088] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f290c1dbc90 con 0x7f28f8071b50 2024-03-06T07:33:31.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.228+0000 7f28f3fff640 1 -- 172.21.15.1:0/1361367466 <== osd.3 v2:172.21.15.1:6808/1031158088 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f290c1dbc90 con 0x7f28f8071b50 2024-03-06T07:33:31.231 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:33:31.231 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:33:31.231 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:33:31.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.229+0000 7f29121f7640 1 -- 172.21.15.1:0/1361367466 >> [v2:172.21.15.1:6808/1031158088,v1:172.21.15.1:6809/1031158088] conn(0x7f28f8071b50 msgr2=0x7f28f8073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.229+0000 7f29121f7640 1 --2- 172.21.15.1:0/1361367466 >> [v2:172.21.15.1:6808/1031158088,v1:172.21.15.1:6809/1031158088] conn(0x7f28f8071b50 0x7f28f8073f90 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.229+0000 7f29121f7640 1 -- 172.21.15.1:0/1361367466 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f28f806b4d0 msgr2=0x7f28f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.229+0000 7f29121f7640 1 --2- 172.21.15.1:0/1361367466 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f28f806b4d0 0x7f28f806d990 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f28fc006fd0 tx=0x7f28fc008040 comp rx=0 tx=0).stop 2024-03-06T07:33:31.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.229+0000 7f29121f7640 1 -- 172.21.15.1:0/1361367466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f290c0c5950 msgr2=0x7f290c145570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.229+0000 7f29121f7640 1 --2- 172.21.15.1:0/1361367466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f290c0c5950 0x7f290c145570 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7f2904032000 tx=0x7f2904004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:31.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.229+0000 7f290bfff640 1 -- 172.21.15.1:0/1361367466 reap_dead start 2024-03-06T07:33:31.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.229+0000 7f29121f7640 1 -- 172.21.15.1:0/1361367466 shutdown_connections 2024-03-06T07:33:31.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.229+0000 7f29121f7640 1 -- 172.21.15.1:0/1361367466 >> 172.21.15.1:0/1361367466 conn(0x7f290c0a8e10 msgr2=0x7f290c0c4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.229+0000 7f29121f7640 1 -- 172.21.15.1:0/1361367466 shutdown_connections 2024-03-06T07:33:31.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.229+0000 7f29121f7640 1 -- 172.21.15.1:0/1361367466 wait complete. 2024-03-06T07:33:31.241 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-06T07:33:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.341+0000 7f34ff1e9640 1 Processor -- start 2024-03-06T07:33:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.341+0000 7f34ff1e9640 1 -- start start 2024-03-06T07:33:31.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.342+0000 7f34ff1e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34f80c6730 0x7f34f80c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.342+0000 7f34ff1e9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f34f80c5d10 0x7f34f8156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.342+0000 7f34ff1e9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f80c3300 con 0x7f34f8152690 2024-03-06T07:33:31.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.342+0000 7f34ff1e9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34f80c35a0 con 0x7f34f80c5d10 2024-03-06T07:33:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.342+0000 7f34ff1e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f80c3880 con 0x7f34f80c6730 2024-03-06T07:33:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.342+0000 7f34fcf85640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f34f8152690 0x7f34f8155ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37124/0 (socket says 172.21.15.1:37124) 2024-03-06T07:33:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.342+0000 7f34fcf85640 1 -- 172.21.15.1:0/1287866244 learned_addr learned my addr 172.21.15.1:0/1287866244 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.342+0000 7f34fd786640 1 --2- 172.21.15.1:0/1287866244 >> v2:172.21.15.98:3300/0 conn(0x7f34f80c5d10 0x7f34f8156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.342+0000 7f34f7fff640 1 --2- 172.21.15.1:0/1287866244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34f80c6730 0x7f34f80c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.342+0000 7f34f77fe640 1 -- 172.21.15.1:0/1287866244 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2575723315 0 0) 0x7f34f80c3300 con 0x7f34f8152690 2024-03-06T07:33:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.342+0000 7f34f77fe640 1 -- 172.21.15.1:0/1287866244 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34e4003520 con 0x7f34f8152690 2024-03-06T07:33:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.342+0000 7f34fd786640 1 -- 172.21.15.1:0/1287866244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34f80c6730 msgr2=0x7f34f80c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:31.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.342+0000 7f34fd786640 1 --2- 172.21.15.1:0/1287866244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34f80c6730 0x7f34f80c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.342+0000 7f34fd786640 1 -- 172.21.15.1:0/1287866244 >> v1:172.21.15.1:6789/0 conn(0x7f34f8152690 legacy=0x7f34f8155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.342+0000 7f34fd786640 1 -- 172.21.15.1:0/1287866244 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f80c3b60 con 0x7f34f80c5d10 2024-03-06T07:33:31.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.343+0000 7f34fd786640 1 --2- 172.21.15.1:0/1287866244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f80c5d10 0x7f34f8156cd0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f34e00097a0 tx=0x7f34e002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51d058a54397aadc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.343+0000 7f34f77fe640 1 -- 172.21.15.1:0/1287866244 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e00315a0 con 0x7f34f80c5d10 2024-03-06T07:33:31.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.343+0000 7f34f77fe640 1 -- 172.21.15.1:0/1287866244 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e0037070 con 0x7f34f80c5d10 2024-03-06T07:33:31.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.343+0000 7f34f77fe640 1 -- 172.21.15.1:0/1287866244 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0031940 con 0x7f34f80c5d10 2024-03-06T07:33:31.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.344+0000 7f34ff1e9640 1 -- 172.21.15.1:0/1287866244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f80c5d10 msgr2=0x7f34f8156cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.344+0000 7f34ff1e9640 1 --2- 172.21.15.1:0/1287866244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f80c5d10 0x7f34f8156cd0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f34e00097a0 tx=0x7f34e002b840 comp rx=0 tx=0).stop 2024-03-06T07:33:31.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.344+0000 7f34ff1e9640 1 -- 172.21.15.1:0/1287866244 shutdown_connections 2024-03-06T07:33:31.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.344+0000 7f34ff1e9640 1 --2- 172.21.15.1:0/1287866244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f80c5d10 0x7f34f8156cd0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.344+0000 7f34ff1e9640 1 --2- 172.21.15.1:0/1287866244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34f80c6730 0x7f34f80c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.344+0000 7f34ff1e9640 1 -- 172.21.15.1:0/1287866244 >> 172.21.15.1:0/1287866244 conn(0x7f34f806b410 msgr2=0x7f34f806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.344+0000 7f34ff1e9640 1 -- 172.21.15.1:0/1287866244 shutdown_connections 2024-03-06T07:33:31.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.344+0000 7f34ff1e9640 1 -- 172.21.15.1:0/1287866244 wait complete. 2024-03-06T07:33:31.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.345+0000 7f34ff1e9640 1 Processor -- start 2024-03-06T07:33:31.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.345+0000 7f34ff1e9640 1 -- start start 2024-03-06T07:33:31.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.346+0000 7f34ff1e9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34f80c5d10 0x7f34f8155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.346+0000 7f34ff1e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f80c6730 0x7f34f80acf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.346+0000 7f34ff1e9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34f8152690 0x7f34f814a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.346+0000 7f34ff1e9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f8152bf0 con 0x7f34f80c5d10 2024-03-06T07:33:31.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.346+0000 7f34ff1e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f8152a70 con 0x7f34f80c6730 2024-03-06T07:33:31.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.346+0000 7f34ff1e9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f8157210 con 0x7f34f8152690 2024-03-06T07:33:31.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.346+0000 7f34fcf85640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34f80c5d10 0x7f34f8155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.346+0000 7f34fcf85640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34f80c5d10 0x7f34f8155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40754/0 (socket says 172.21.15.1:40754) 2024-03-06T07:33:31.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.346+0000 7f34fcf85640 1 -- 172.21.15.1:0/3619957484 learned_addr learned my addr 172.21.15.1:0/3619957484 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:31.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.346+0000 7f34fd786640 1 --2- 172.21.15.1:0/3619957484 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34f8152690 0x7f34f814a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.346+0000 7f34f7fff640 1 --2- 172.21.15.1:0/3619957484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f80c6730 0x7f34f80acf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.346+0000 7f34fcf85640 1 -- 172.21.15.1:0/3619957484 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34f8152690 msgr2=0x7f34f814a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.346+0000 7f34fcf85640 1 --2- 172.21.15.1:0/3619957484 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34f8152690 0x7f34f814a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.346+0000 7f34fcf85640 1 -- 172.21.15.1:0/3619957484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f80c6730 msgr2=0x7f34f80acf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.346+0000 7f34fcf85640 1 --2- 172.21.15.1:0/3619957484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f80c6730 0x7f34f80acf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.346+0000 7f34fcf85640 1 -- 172.21.15.1:0/3619957484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f8070480 con 0x7f34f80c5d10 2024-03-06T07:33:31.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.346+0000 7f34f7fff640 1 --2- 172.21.15.1:0/3619957484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f80c6730 0x7f34f80acf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:31.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.347+0000 7f34fcf85640 1 --2- 172.21.15.1:0/3619957484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34f80c5d10 0x7f34f8155b90 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f34e800c460 tx=0x7f34e800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.347+0000 7f34f57fa640 1 -- 172.21.15.1:0/3619957484 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e8017070 con 0x7f34f80c5d10 2024-03-06T07:33:31.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.347+0000 7f34f57fa640 1 -- 172.21.15.1:0/3619957484 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e8005ce0 con 0x7f34f80c5d10 2024-03-06T07:33:31.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.347+0000 7f34ff1e9640 1 -- 172.21.15.1:0/3619957484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f8153750 con 0x7f34f80c5d10 2024-03-06T07:33:31.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.347+0000 7f34f57fa640 1 -- 172.21.15.1:0/3619957484 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e80072f0 con 0x7f34f80c5d10 2024-03-06T07:33:31.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.347+0000 7f34ff1e9640 1 -- 172.21.15.1:0/3619957484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f80aadc0 con 0x7f34f80c5d10 2024-03-06T07:33:31.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.348+0000 7f34ff1e9640 1 -- 172.21.15.1:0/3619957484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f8067fd0 con 0x7f34f80c5d10 2024-03-06T07:33:31.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.348+0000 7f34f57fa640 1 -- 172.21.15.1:0/3619957484 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f34e801a070 con 0x7f34f80c5d10 2024-03-06T07:33:31.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.349+0000 7f34f57fa640 1 --2- 172.21.15.1:0/3619957484 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34d806b4d0 0x7f34d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.349+0000 7f34f57fa640 1 -- 172.21.15.1:0/3619957484 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f34e8089510 con 0x7f34f80c5d10 2024-03-06T07:33:31.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.349+0000 7f34f7fff640 1 --2- 172.21.15.1:0/3619957484 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34d806b4d0 0x7f34d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.349+0000 7f34f7fff640 1 --2- 172.21.15.1:0/3619957484 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34d806b4d0 0x7f34d806d990 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f34f80aa940 tx=0x7f34f000b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.353+0000 7f34f57fa640 1 -- 172.21.15.1:0/3619957484 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f34e8057dc0 con 0x7f34f80c5d10 2024-03-06T07:33:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.535+0000 7f34ff1e9640 1 -- 172.21.15.1:0/3619957484 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f34f80aaba0 con 0x7f34d806b4d0 2024-03-06T07:33:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.535+0000 7f34f57fa640 1 -- 172.21.15.1:0/3619957484 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f34f80aaba0 con 0x7f34d806b4d0 2024-03-06T07:33:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 3 to deep-scrub 2024-03-06T07:33:31.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.537+0000 7f34ff1e9640 1 -- 172.21.15.1:0/3619957484 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34d806b4d0 msgr2=0x7f34d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.537+0000 7f34ff1e9640 1 --2- 172.21.15.1:0/3619957484 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34d806b4d0 0x7f34d806d990 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f34f80aa940 tx=0x7f34f000b040 comp rx=0 tx=0).stop 2024-03-06T07:33:31.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.537+0000 7f34ff1e9640 1 -- 172.21.15.1:0/3619957484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34f80c5d10 msgr2=0x7f34f8155b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.537+0000 7f34ff1e9640 1 --2- 172.21.15.1:0/3619957484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34f80c5d10 0x7f34f8155b90 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f34e800c460 tx=0x7f34e800c920 comp rx=0 tx=0).stop 2024-03-06T07:33:31.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.537+0000 7f34ff1e9640 1 -- 172.21.15.1:0/3619957484 shutdown_connections 2024-03-06T07:33:31.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.537+0000 7f34ff1e9640 1 --2- 172.21.15.1:0/3619957484 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f34d806b4d0 0x7f34d806d990 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.537+0000 7f34ff1e9640 1 --2- 172.21.15.1:0/3619957484 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34f8152690 0x7f34f814a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.537+0000 7f34ff1e9640 1 --2- 172.21.15.1:0/3619957484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f80c6730 0x7f34f80acf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.537+0000 7f34ff1e9640 1 --2- 172.21.15.1:0/3619957484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34f80c5d10 0x7f34f8155b90 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.537+0000 7f34ff1e9640 1 -- 172.21.15.1:0/3619957484 >> 172.21.15.1:0/3619957484 conn(0x7f34f806b410 msgr2=0x7f34f8070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.537+0000 7f34ff1e9640 1 -- 172.21.15.1:0/3619957484 shutdown_connections 2024-03-06T07:33:31.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.537+0000 7f34ff1e9640 1 -- 172.21.15.1:0/3619957484 wait complete. 2024-03-06T07:33:31.554 INFO:tasks.ceph:Scrubbing osd.4 2024-03-06T07:33:31.554 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:33:31.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.658+0000 7f0b99f56640 1 Processor -- start 2024-03-06T07:33:31.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.658+0000 7f0b99f56640 1 -- start start 2024-03-06T07:33:31.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.658+0000 7f0b99f56640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b94158180 0x7f0b940c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.658+0000 7f0b99f56640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b94158560 con 0x7f0b940c5e20 2024-03-06T07:33:31.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.658+0000 7f0b99f56640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b94155600 con 0x7f0b940c6800 2024-03-06T07:33:31.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.658+0000 7f0b99f56640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b94156210 con 0x7f0b94158180 2024-03-06T07:33:31.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.659+0000 7f0b92ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0b940c5e20 0x7f0b94156a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37126/0 (socket says 172.21.15.1:37126) 2024-03-06T07:33:31.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.659+0000 7f0b92ffd640 1 -- 172.21.15.1:0/2759070883 learned_addr learned my addr 172.21.15.1:0/2759070883 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:31.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.659+0000 7f0b937fe640 1 --2- 172.21.15.1:0/2759070883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b94158180 0x7f0b940c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.659+0000 7f0b927fc640 1 -- 172.21.15.1:0/2759070883 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413541880 0 0) 0x7f0b94158560 con 0x7f0b940c5e20 2024-03-06T07:33:31.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.659+0000 7f0b927fc640 1 -- 172.21.15.1:0/2759070883 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b7c003520 con 0x7f0b940c5e20 2024-03-06T07:33:31.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.659+0000 7f0b927fc640 1 -- 172.21.15.1:0/2759070883 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030780481 0 0) 0x7f0b94155600 con 0x7f0b940c6800 2024-03-06T07:33:31.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.659+0000 7f0b927fc640 1 -- 172.21.15.1:0/2759070883 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b94158560 con 0x7f0b940c6800 2024-03-06T07:33:31.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.659+0000 7f0b937fe640 1 -- 172.21.15.1:0/2759070883 >> v1:172.21.15.98:6789/0 conn(0x7f0b940c6800 legacy=0x7f0b94157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.659+0000 7f0b937fe640 1 -- 172.21.15.1:0/2759070883 >> v1:172.21.15.1:6789/0 conn(0x7f0b940c5e20 legacy=0x7f0b94156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.659+0000 7f0b937fe640 1 -- 172.21.15.1:0/2759070883 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b941558e0 con 0x7f0b94158180 2024-03-06T07:33:31.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.660+0000 7f0b937fe640 1 --2- 172.21.15.1:0/2759070883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b94158180 0x7f0b940c58e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0b840097d0 tx=0x7f0b8402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2786b655dc3f210e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.660+0000 7f0b927fc640 1 -- 172.21.15.1:0/2759070883 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b84031770 con 0x7f0b94158180 2024-03-06T07:33:31.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.660+0000 7f0b927fc640 1 -- 172.21.15.1:0/2759070883 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b84037070 con 0x7f0b94158180 2024-03-06T07:33:31.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.660+0000 7f0b927fc640 1 -- 172.21.15.1:0/2759070883 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b84031b30 con 0x7f0b94158180 2024-03-06T07:33:31.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.660+0000 7f0b99f56640 1 -- 172.21.15.1:0/2759070883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b94158180 msgr2=0x7f0b940c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.660+0000 7f0b99f56640 1 --2- 172.21.15.1:0/2759070883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b94158180 0x7f0b940c58e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0b840097d0 tx=0x7f0b8402b810 comp rx=0 tx=0).stop 2024-03-06T07:33:31.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.660+0000 7f0b99f56640 1 -- 172.21.15.1:0/2759070883 shutdown_connections 2024-03-06T07:33:31.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.660+0000 7f0b99f56640 1 --2- 172.21.15.1:0/2759070883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b94158180 0x7f0b940c58e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.660+0000 7f0b99f56640 1 -- 172.21.15.1:0/2759070883 >> 172.21.15.1:0/2759070883 conn(0x7f0b9406b410 msgr2=0x7f0b9406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.661+0000 7f0b99f56640 1 -- 172.21.15.1:0/2759070883 shutdown_connections 2024-03-06T07:33:31.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.661+0000 7f0b99f56640 1 -- 172.21.15.1:0/2759070883 wait complete. 2024-03-06T07:33:31.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.662+0000 7f0b99f56640 1 Processor -- start 2024-03-06T07:33:31.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.662+0000 7f0b99f56640 1 -- start start 2024-03-06T07:33:31.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.662+0000 7f0b99f56640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b940c5e20 0x7f0b94152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.662+0000 7f0b99f56640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b940c6800 0x7f0b941eb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.662+0000 7f0b99f56640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b94158180 0x7f0b94143ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.662+0000 7f0b99f56640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b94155ea0 con 0x7f0b94158180 2024-03-06T07:33:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.662+0000 7f0b99f56640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b940c3ea0 con 0x7f0b940c6800 2024-03-06T07:33:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.662+0000 7f0b99f56640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b941551f0 con 0x7f0b940c5e20 2024-03-06T07:33:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.662+0000 7f0b93fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b94158180 0x7f0b94143ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.663+0000 7f0b93fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b94158180 0x7f0b94143ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40766/0 (socket says 172.21.15.1:40766) 2024-03-06T07:33:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.663+0000 7f0b93fff640 1 -- 172.21.15.1:0/4240244898 learned_addr learned my addr 172.21.15.1:0/4240244898 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.663+0000 7f0b92ffd640 1 --2- 172.21.15.1:0/4240244898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b940c6800 0x7f0b941eb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.663+0000 7f0b937fe640 1 --2- 172.21.15.1:0/4240244898 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b940c5e20 0x7f0b94152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.663+0000 7f0b93fff640 1 -- 172.21.15.1:0/4240244898 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b940c5e20 msgr2=0x7f0b94152af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.663+0000 7f0b93fff640 1 --2- 172.21.15.1:0/4240244898 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0b940c5e20 0x7f0b94152af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.663+0000 7f0b93fff640 1 -- 172.21.15.1:0/4240244898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b940c6800 msgr2=0x7f0b941eb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.663+0000 7f0b93fff640 1 --2- 172.21.15.1:0/4240244898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b940c6800 0x7f0b941eb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.663+0000 7f0b93fff640 1 -- 172.21.15.1:0/4240244898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9406d6d0 con 0x7f0b94158180 2024-03-06T07:33:31.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.663+0000 7f0b93fff640 1 --2- 172.21.15.1:0/4240244898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b94158180 0x7f0b94143ce0 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00c790 tx=0x7f0b8c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.663+0000 7f0b7bfff640 1 -- 172.21.15.1:0/4240244898 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c003b70 con 0x7f0b94158180 2024-03-06T07:33:31.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.663+0000 7f0b7bfff640 1 -- 172.21.15.1:0/4240244898 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8c003d10 con 0x7f0b94158180 2024-03-06T07:33:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.663+0000 7f0b99f56640 1 -- 172.21.15.1:0/4240244898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b940aa8e0 con 0x7f0b94158180 2024-03-06T07:33:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.663+0000 7f0b7bfff640 1 -- 172.21.15.1:0/4240244898 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c00a9f0 con 0x7f0b94158180 2024-03-06T07:33:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.663+0000 7f0b99f56640 1 -- 172.21.15.1:0/4240244898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b940aaea0 con 0x7f0b94158180 2024-03-06T07:33:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.664+0000 7f0b99f56640 1 -- 172.21.15.1:0/4240244898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0b940aa690 con 0x7f0b94158180 2024-03-06T07:33:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.665+0000 7f0b7bfff640 1 -- 172.21.15.1:0/4240244898 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b8c00ab90 con 0x7f0b94158180 2024-03-06T07:33:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.665+0000 7f0b7bfff640 1 --2- 172.21.15.1:0/4240244898 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0b8006b500 0x7f0b8006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.665+0000 7f0b7bfff640 1 -- 172.21.15.1:0/4240244898 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f0b8c089ef0 con 0x7f0b94158180 2024-03-06T07:33:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.665+0000 7f0b7bfff640 1 --2- 172.21.15.1:0/4240244898 >> [v2:172.21.15.98:6808/2308139585,v1:172.21.15.98:6809/2308139585] conn(0x7f0b80071b80 0x7f0b80073fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.665+0000 7f0b7bfff640 1 -- 172.21.15.1:0/4240244898 --> [v2:172.21.15.98:6808/2308139585,v1:172.21.15.98:6809/2308139585] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b8c010e50 con 0x7f0b80071b80 2024-03-06T07:33:31.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.665+0000 7f0b7bfff640 1 -- 172.21.15.1:0/4240244898 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=217) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b8c08c5f0 con 0x7f0b94158180 2024-03-06T07:33:31.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.665+0000 7f0b937fe640 1 --2- 172.21.15.1:0/4240244898 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0b8006b500 0x7f0b8006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.665+0000 7f0b92ffd640 1 --2- 172.21.15.1:0/4240244898 >> [v2:172.21.15.98:6808/2308139585,v1:172.21.15.98:6809/2308139585] conn(0x7f0b80071b80 0x7f0b80073fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.666+0000 7f0b937fe640 1 --2- 172.21.15.1:0/4240244898 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0b8006b500 0x7f0b8006d9c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f0b84036000 tx=0x7f0b8403d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.666+0000 7f0b92ffd640 1 --2- 172.21.15.1:0/4240244898 >> [v2:172.21.15.98:6808/2308139585,v1:172.21.15.98:6809/2308139585] conn(0x7f0b80071b80 0x7f0b80073fc0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.666+0000 7f0b7bfff640 1 -- 172.21.15.1:0/4240244898 <== osd.4 v2:172.21.15.98:6808/2308139585 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0b8c010e50 con 0x7f0b80071b80 2024-03-06T07:33:31.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.679+0000 7f0b99f56640 1 -- 172.21.15.1:0/4240244898 --> [v2:172.21.15.98:6808/2308139585,v1:172.21.15.98:6809/2308139585] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0b94151640 con 0x7f0b80071b80 2024-03-06T07:33:31.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.680+0000 7f0b7bfff640 1 -- 172.21.15.1:0/4240244898 <== osd.4 v2:172.21.15.98:6808/2308139585 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f0b88006b20 con 0x7f0b80071b80 2024-03-06T07:33:31.682 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:33:31.682 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:33:31.682 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:33:31.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.680+0000 7f0b99f56640 1 -- 172.21.15.1:0/4240244898 >> [v2:172.21.15.98:6808/2308139585,v1:172.21.15.98:6809/2308139585] conn(0x7f0b80071b80 msgr2=0x7f0b80073fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.680+0000 7f0b99f56640 1 --2- 172.21.15.1:0/4240244898 >> [v2:172.21.15.98:6808/2308139585,v1:172.21.15.98:6809/2308139585] conn(0x7f0b80071b80 0x7f0b80073fc0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.680+0000 7f0b99f56640 1 -- 172.21.15.1:0/4240244898 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0b8006b500 msgr2=0x7f0b8006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.680+0000 7f0b99f56640 1 --2- 172.21.15.1:0/4240244898 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f0b8006b500 0x7f0b8006d9c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f0b84036000 tx=0x7f0b8403d040 comp rx=0 tx=0).stop 2024-03-06T07:33:31.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.680+0000 7f0b99f56640 1 -- 172.21.15.1:0/4240244898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b94158180 msgr2=0x7f0b94143ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.680+0000 7f0b99f56640 1 --2- 172.21.15.1:0/4240244898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b94158180 0x7f0b94143ce0 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00c790 tx=0x7f0b8c00cc50 comp rx=0 tx=0).stop 2024-03-06T07:33:31.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.680+0000 7f0b93fff640 1 -- 172.21.15.1:0/4240244898 reap_dead start 2024-03-06T07:33:31.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.680+0000 7f0b99f56640 1 -- 172.21.15.1:0/4240244898 shutdown_connections 2024-03-06T07:33:31.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.680+0000 7f0b99f56640 1 -- 172.21.15.1:0/4240244898 >> 172.21.15.1:0/4240244898 conn(0x7f0b9406b410 msgr2=0x7f0b9414f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.680+0000 7f0b99f56640 1 -- 172.21.15.1:0/4240244898 shutdown_connections 2024-03-06T07:33:31.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.680+0000 7f0b99f56640 1 -- 172.21.15.1:0/4240244898 wait complete. 2024-03-06T07:33:31.693 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-06T07:33:31.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.794+0000 7f7468884640 1 Processor -- start 2024-03-06T07:33:31.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.794+0000 7f7468884640 1 -- start start 2024-03-06T07:33:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.795+0000 7f7468884640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7464072510 0x7f7464072970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.795+0000 7f7468884640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f74641568e0 0x7f7464156d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.795+0000 7f7468884640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74641533f0 con 0x7f7464150500 2024-03-06T07:33:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.795+0000 7f7468884640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7464154000 con 0x7f74641568e0 2024-03-06T07:33:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.795+0000 7f7468884640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f746406b110 con 0x7f7464072510 2024-03-06T07:33:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.795+0000 7f746259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7464150500 0x7f74640c6d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37140/0 (socket says 172.21.15.1:37140) 2024-03-06T07:33:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.795+0000 7f746259c640 1 -- 172.21.15.1:0/3237624724 learned_addr learned my addr 172.21.15.1:0/3237624724 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.795+0000 7f746159a640 1 -- 172.21.15.1:0/3237624724 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763688404 0 0) 0x7f74641533f0 con 0x7f7464150500 2024-03-06T07:33:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.795+0000 7f746159a640 1 -- 172.21.15.1:0/3237624724 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f744c003520 con 0x7f7464150500 2024-03-06T07:33:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f7461d9b640 1 --2- 172.21.15.1:0/3237624724 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7464072510 0x7f7464072970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f7462d9d640 1 --2- 172.21.15.1:0/3237624724 >> v2:172.21.15.98:3300/0 conn(0x7f74641568e0 0x7f7464156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746159a640 1 -- 172.21.15.1:0/3237624724 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3388897309 0 0) 0x7f744c003520 con 0x7f7464150500 2024-03-06T07:33:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746159a640 1 -- 172.21.15.1:0/3237624724 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7464072510 msgr2=0x7f7464072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746159a640 1 --2- 172.21.15.1:0/3237624724 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7464072510 0x7f7464072970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746159a640 1 -- 172.21.15.1:0/3237624724 >> v2:172.21.15.98:3300/0 conn(0x7f74641568e0 msgr2=0x7f7464156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746159a640 1 --2- 172.21.15.1:0/3237624724 >> v2:172.21.15.98:3300/0 conn(0x7f74641568e0 0x7f7464156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746159a640 1 -- 172.21.15.1:0/3237624724 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74641536d0 con 0x7f7464150500 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746159a640 1 -- 172.21.15.1:0/3237624724 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f745c001670 con 0x7f7464150500 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746159a640 1 -- 172.21.15.1:0/3237624724 >> v1:172.21.15.1:6789/0 conn(0x7f7464150500 legacy=0x7f74640c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746159a640 1 --2- 172.21.15.1:0/3237624724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f744c0039b0 0x7f744c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746159a640 1 --2- 172.21.15.1:0/3237624724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f744c004800 0x7f744c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746159a640 1 --2- 172.21.15.1:0/3237624724 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7464072510 0x7f744c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746159a640 1 -- 172.21.15.1:0/3237624724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7464154000 con 0x7f744c0039b0 2024-03-06T07:33:31.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746159a640 1 -- 172.21.15.1:0/3237624724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746406b110 con 0x7f744c004800 2024-03-06T07:33:31.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746159a640 1 -- 172.21.15.1:0/3237624724 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f744c00ba10 con 0x7f7464072510 2024-03-06T07:33:31.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746259c640 1 --2- 172.21.15.1:0/3237624724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f744c0039b0 0x7f744c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f7462d9d640 1 --2- 172.21.15.1:0/3237624724 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7464072510 0x7f744c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f7461d9b640 1 --2- 172.21.15.1:0/3237624724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f744c004800 0x7f744c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746259c640 1 -- 172.21.15.1:0/3237624724 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7464072510 msgr2=0x7f744c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:31.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746259c640 1 --2- 172.21.15.1:0/3237624724 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7464072510 0x7f744c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746259c640 1 -- 172.21.15.1:0/3237624724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f744c004800 msgr2=0x7f744c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:31.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746259c640 1 --2- 172.21.15.1:0/3237624724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f744c004800 0x7f744c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.796+0000 7f746259c640 1 -- 172.21.15.1:0/3237624724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f744c003520 con 0x7f744c0039b0 2024-03-06T07:33:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.797+0000 7f746259c640 1 --2- 172.21.15.1:0/3237624724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f744c0039b0 0x7f744c003dd0 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f745c00c240 tx=0x7f744c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=527b88bf8eb0e10f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.797+0000 7f746159a640 1 -- 172.21.15.1:0/3237624724 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745c007d30 con 0x7f744c0039b0 2024-03-06T07:33:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.797+0000 7f746159a640 1 -- 172.21.15.1:0/3237624724 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f745c030070 con 0x7f744c0039b0 2024-03-06T07:33:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.797+0000 7f746159a640 1 -- 172.21.15.1:0/3237624724 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745c02cd60 con 0x7f744c0039b0 2024-03-06T07:33:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.798+0000 7f7468884640 1 -- 172.21.15.1:0/3237624724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f744c0039b0 msgr2=0x7f744c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.798+0000 7f7468884640 1 --2- 172.21.15.1:0/3237624724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f744c0039b0 0x7f744c003dd0 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f745c00c240 tx=0x7f744c005830 comp rx=0 tx=0).stop 2024-03-06T07:33:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.798+0000 7f7462d9d640 1 -- 172.21.15.1:0/3237624724 reap_dead start 2024-03-06T07:33:31.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.798+0000 7f7468884640 1 -- 172.21.15.1:0/3237624724 shutdown_connections 2024-03-06T07:33:31.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.798+0000 7f7468884640 1 -- 172.21.15.1:0/3237624724 >> 172.21.15.1:0/3237624724 conn(0x7f74640a9700 msgr2=0x7f74640a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.798+0000 7f7468884640 1 -- 172.21.15.1:0/3237624724 shutdown_connections 2024-03-06T07:33:31.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.798+0000 7f7468884640 1 -- 172.21.15.1:0/3237624724 wait complete. 2024-03-06T07:33:31.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.799+0000 7f7468884640 1 Processor -- start 2024-03-06T07:33:31.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.799+0000 7f7468884640 1 -- start start 2024-03-06T07:33:31.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7468884640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f744c0039b0 0x7f74641494e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7468884640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74640c3c70 0x7f746414f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7468884640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74640c2530 0x7f74640c29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7468884640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7464153010 con 0x7f74640c3c70 2024-03-06T07:33:31.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7468884640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7464153c90 con 0x7f74640c2530 2024-03-06T07:33:31.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7468884640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7464153190 con 0x7f744c0039b0 2024-03-06T07:33:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7461d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74640c3c70 0x7f746414f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7461d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74640c3c70 0x7f746414f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40772/0 (socket says 172.21.15.1:40772) 2024-03-06T07:33:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7461d9b640 1 -- 172.21.15.1:0/3034323499 learned_addr learned my addr 172.21.15.1:0/3034323499 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7462d9d640 1 --2- 172.21.15.1:0/3034323499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74640c2530 0x7f74640c29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f746259c640 1 --2- 172.21.15.1:0/3034323499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f744c0039b0 0x7f74641494e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7461d9b640 1 -- 172.21.15.1:0/3034323499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f744c0039b0 msgr2=0x7f74641494e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7461d9b640 1 --2- 172.21.15.1:0/3034323499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f744c0039b0 0x7f74641494e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7461d9b640 1 -- 172.21.15.1:0/3034323499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74640c2530 msgr2=0x7f74640c29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7461d9b640 1 --2- 172.21.15.1:0/3034323499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74640c2530 0x7f74640c29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7461d9b640 1 -- 172.21.15.1:0/3034323499 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746414df70 con 0x7f74640c3c70 2024-03-06T07:33:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7462d9d640 1 --2- 172.21.15.1:0/3034323499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74640c2530 0x7f74640c29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f746259c640 1 --2- 172.21.15.1:0/3034323499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f744c0039b0 0x7f74641494e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7461d9b640 1 --2- 172.21.15.1:0/3034323499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74640c3c70 0x7f746414f1a0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f74580118e0 tx=0x7f7458011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.800+0000 7f7452ffd640 1 -- 172.21.15.1:0/3034323499 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745800faf0 con 0x7f74640c3c70 2024-03-06T07:33:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.801+0000 7f7452ffd640 1 -- 172.21.15.1:0/3034323499 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f745800fc90 con 0x7f74640c3c70 2024-03-06T07:33:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.801+0000 7f7452ffd640 1 -- 172.21.15.1:0/3034323499 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7458014440 con 0x7f74640c3c70 2024-03-06T07:33:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.801+0000 7f7468884640 1 -- 172.21.15.1:0/3034323499 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f746414e250 con 0x7f74640c3c70 2024-03-06T07:33:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.801+0000 7f7468884640 1 -- 172.21.15.1:0/3034323499 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7464149a20 con 0x7f74640c3c70 2024-03-06T07:33:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.802+0000 7f7468884640 1 -- 172.21.15.1:0/3034323499 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74640686e0 con 0x7f74640c3c70 2024-03-06T07:33:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.802+0000 7f7452ffd640 1 -- 172.21.15.1:0/3034323499 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7458012040 con 0x7f74640c3c70 2024-03-06T07:33:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.802+0000 7f7452ffd640 1 --2- 172.21.15.1:0/3034323499 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f744406b500 0x7f744406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.803+0000 7f7452ffd640 1 -- 172.21.15.1:0/3034323499 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f745808a7d0 con 0x7f74640c3c70 2024-03-06T07:33:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.803+0000 7f746259c640 1 --2- 172.21.15.1:0/3034323499 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f744406b500 0x7f744406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.803+0000 7f746259c640 1 --2- 172.21.15.1:0/3034323499 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f744406b500 0x7f744406d9c0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f745c034000 tx=0x7f745c007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.807+0000 7f7452ffd640 1 -- 172.21.15.1:0/3034323499 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7458019030 con 0x7f74640c3c70 2024-03-06T07:33:31.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.989+0000 7f7468884640 1 -- 172.21.15.1:0/3034323499 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f74640c6b70 con 0x7f744406b500 2024-03-06T07:33:31.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.989+0000 7f7452ffd640 1 -- 172.21.15.1:0/3034323499 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f74640c6b70 con 0x7f744406b500 2024-03-06T07:33:31.991 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 4 to deep-scrub 2024-03-06T07:33:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.991+0000 7f7468884640 1 -- 172.21.15.1:0/3034323499 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f744406b500 msgr2=0x7f744406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.991+0000 7f7468884640 1 --2- 172.21.15.1:0/3034323499 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f744406b500 0x7f744406d9c0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f745c034000 tx=0x7f745c007000 comp rx=0 tx=0).stop 2024-03-06T07:33:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.991+0000 7f7468884640 1 -- 172.21.15.1:0/3034323499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74640c3c70 msgr2=0x7f746414f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.991+0000 7f7468884640 1 --2- 172.21.15.1:0/3034323499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74640c3c70 0x7f746414f1a0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f74580118e0 tx=0x7f7458011da0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.991+0000 7f7468884640 1 -- 172.21.15.1:0/3034323499 shutdown_connections 2024-03-06T07:33:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.991+0000 7f7468884640 1 --2- 172.21.15.1:0/3034323499 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f744406b500 0x7f744406d9c0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.991+0000 7f7468884640 1 --2- 172.21.15.1:0/3034323499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74640c2530 0x7f74640c29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.991+0000 7f7468884640 1 --2- 172.21.15.1:0/3034323499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74640c3c70 0x7f746414f1a0 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.991+0000 7f7468884640 1 --2- 172.21.15.1:0/3034323499 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f744c0039b0 0x7f74641494e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.991+0000 7f7468884640 1 -- 172.21.15.1:0/3034323499 >> 172.21.15.1:0/3034323499 conn(0x7f74640a9700 msgr2=0x7f7464152860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.991+0000 7f7468884640 1 -- 172.21.15.1:0/3034323499 shutdown_connections 2024-03-06T07:33:31.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:31.991+0000 7f7468884640 1 -- 172.21.15.1:0/3034323499 wait complete. 2024-03-06T07:33:32.004 INFO:tasks.ceph:Scrubbing osd.5 2024-03-06T07:33:32.004 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:33:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.107+0000 7f6a8a6fb640 1 Processor -- start 2024-03-06T07:33:32.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.107+0000 7f6a8a6fb640 1 -- start start 2024-03-06T07:33:32.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a8a6fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a841505d0 0x7f6a840c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a8a6fb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6a840c32f0 0x7f6a840c3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a8a6fb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a840700f0 con 0x7f6a840707a0 2024-03-06T07:33:32.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a8a6fb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a841509b0 con 0x7f6a840c32f0 2024-03-06T07:33:32.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a8a6fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a840c6f30 con 0x7f6a841505d0 2024-03-06T07:33:32.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a88c98640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6a840707a0 0x7f6a84070c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37142/0 (socket says 172.21.15.1:37142) 2024-03-06T07:33:32.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a88c98640 1 -- 172.21.15.1:0/3315857276 learned_addr learned my addr 172.21.15.1:0/3315857276 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:32.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a83fff640 1 --2- 172.21.15.1:0/3315857276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a841505d0 0x7f6a840c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a837fe640 1 --2- 172.21.15.1:0/3315857276 >> v2:172.21.15.98:3300/0 conn(0x7f6a840c32f0 0x7f6a840c3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a82ffd640 1 -- 172.21.15.1:0/3315857276 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711538783 0 0) 0x7f6a840700f0 con 0x7f6a840707a0 2024-03-06T07:33:32.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a82ffd640 1 -- 172.21.15.1:0/3315857276 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a6c003520 con 0x7f6a840707a0 2024-03-06T07:33:32.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a82ffd640 1 -- 172.21.15.1:0/3315857276 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 745066165 0 0) 0x7f6a6c003520 con 0x7f6a840707a0 2024-03-06T07:33:32.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a82ffd640 1 -- 172.21.15.1:0/3315857276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a841505d0 msgr2=0x7f6a840c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a82ffd640 1 --2- 172.21.15.1:0/3315857276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a841505d0 0x7f6a840c69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a82ffd640 1 -- 172.21.15.1:0/3315857276 >> v2:172.21.15.98:3300/0 conn(0x7f6a840c32f0 msgr2=0x7f6a840c3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a82ffd640 1 --2- 172.21.15.1:0/3315857276 >> v2:172.21.15.98:3300/0 conn(0x7f6a840c32f0 0x7f6a840c3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a82ffd640 1 -- 172.21.15.1:0/3315857276 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a84153010 con 0x7f6a840707a0 2024-03-06T07:33:32.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a82ffd640 1 -- 172.21.15.1:0/3315857276 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f6a7c0019f0 con 0x7f6a840707a0 2024-03-06T07:33:32.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a82ffd640 1 -- 172.21.15.1:0/3315857276 >> v1:172.21.15.1:6789/0 conn(0x7f6a840707a0 legacy=0x7f6a84070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a82ffd640 1 --2- 172.21.15.1:0/3315857276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a6c0039b0 0x7f6a6c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a82ffd640 1 --2- 172.21.15.1:0/3315857276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a841505d0 0x7f6a6c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a82ffd640 1 --2- 172.21.15.1:0/3315857276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6c004d40 0x7f6a6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a82ffd640 1 -- 172.21.15.1:0/3315857276 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a840c6f30 con 0x7f6a6c0039b0 2024-03-06T07:33:32.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a82ffd640 1 -- 172.21.15.1:0/3315857276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c00b9c0 con 0x7f6a6c004d40 2024-03-06T07:33:32.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a82ffd640 1 -- 172.21.15.1:0/3315857276 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c00bb40 con 0x7f6a841505d0 2024-03-06T07:33:32.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.108+0000 7f6a83fff640 1 --2- 172.21.15.1:0/3315857276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a6c0039b0 0x7f6a6c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.109+0000 7f6a837fe640 1 --2- 172.21.15.1:0/3315857276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a841505d0 0x7f6a6c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.109+0000 7f6a88c98640 1 --2- 172.21.15.1:0/3315857276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6c004d40 0x7f6a6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.109+0000 7f6a83fff640 1 -- 172.21.15.1:0/3315857276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a841505d0 msgr2=0x7f6a6c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.109+0000 7f6a83fff640 1 --2- 172.21.15.1:0/3315857276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a841505d0 0x7f6a6c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.109+0000 7f6a83fff640 1 -- 172.21.15.1:0/3315857276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6c004d40 msgr2=0x7f6a6c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.109+0000 7f6a83fff640 1 --2- 172.21.15.1:0/3315857276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6c004d40 0x7f6a6c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.109+0000 7f6a83fff640 1 -- 172.21.15.1:0/3315857276 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c003520 con 0x7f6a6c0039b0 2024-03-06T07:33:32.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.109+0000 7f6a83fff640 1 --2- 172.21.15.1:0/3315857276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a6c0039b0 0x7f6a6c003dd0 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f6a74009ac0 tx=0x7f6a6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89a50b133e624539 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.110+0000 7f6a82ffd640 1 -- 172.21.15.1:0/3315857276 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a74039070 con 0x7f6a6c0039b0 2024-03-06T07:33:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.110+0000 7f6a82ffd640 1 -- 172.21.15.1:0/3315857276 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7402c6b0 con 0x7f6a6c0039b0 2024-03-06T07:33:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.110+0000 7f6a82ffd640 1 -- 172.21.15.1:0/3315857276 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a74009690 con 0x7f6a6c0039b0 2024-03-06T07:33:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.110+0000 7f6a8a6fb640 1 -- 172.21.15.1:0/3315857276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a6c0039b0 msgr2=0x7f6a6c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.110+0000 7f6a8a6fb640 1 --2- 172.21.15.1:0/3315857276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a6c0039b0 0x7f6a6c003dd0 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f6a74009ac0 tx=0x7f6a6c005830 comp rx=0 tx=0).stop 2024-03-06T07:33:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.110+0000 7f6a88c98640 1 -- 172.21.15.1:0/3315857276 reap_dead start 2024-03-06T07:33:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.110+0000 7f6a8a6fb640 1 -- 172.21.15.1:0/3315857276 shutdown_connections 2024-03-06T07:33:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.110+0000 7f6a8a6fb640 1 -- 172.21.15.1:0/3315857276 >> 172.21.15.1:0/3315857276 conn(0x7f6a840aa6a0 msgr2=0x7f6a840aaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.110+0000 7f6a8a6fb640 1 -- 172.21.15.1:0/3315857276 shutdown_connections 2024-03-06T07:33:32.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.110+0000 7f6a8a6fb640 1 -- 172.21.15.1:0/3315857276 wait complete. 2024-03-06T07:33:32.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.111+0000 7f6a8a6fb640 1 Processor -- start 2024-03-06T07:33:32.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.111+0000 7f6a8a6fb640 1 -- start start 2024-03-06T07:33:32.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a8a6fb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a6c0039b0 0x7f6a841465e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a8a6fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a84155d50 0x7f6a8414bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a8a6fb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8414c4d0 0x7f6a841ea7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a8a6fb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a841538b0 con 0x7f6a84155d50 2024-03-06T07:33:32.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a8a6fb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a841535d0 con 0x7f6a8414c4d0 2024-03-06T07:33:32.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a8a6fb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a84071340 con 0x7f6a6c0039b0 2024-03-06T07:33:32.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a837fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a84155d50 0x7f6a8414bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a837fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a84155d50 0x7f6a8414bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40796/0 (socket says 172.21.15.1:40796) 2024-03-06T07:33:32.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a837fe640 1 -- 172.21.15.1:0/2925525415 learned_addr learned my addr 172.21.15.1:0/2925525415 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:32.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a83fff640 1 --2- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a6c0039b0 0x7f6a841465e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a88c98640 1 --2- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8414c4d0 0x7f6a841ea7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a837fe640 1 -- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a6c0039b0 msgr2=0x7f6a841465e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a837fe640 1 --2- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a6c0039b0 0x7f6a841465e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a837fe640 1 -- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8414c4d0 msgr2=0x7f6a841ea7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a837fe640 1 --2- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8414c4d0 0x7f6a841ea7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a837fe640 1 -- 172.21.15.1:0/2925525415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a841571a0 con 0x7f6a84155d50 2024-03-06T07:33:32.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a83fff640 1 --2- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a6c0039b0 0x7f6a841465e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:32.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.112+0000 7f6a88c98640 1 --2- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8414c4d0 0x7f6a841ea7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:32.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.113+0000 7f6a837fe640 1 --2- 172.21.15.1:0/2925525415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a84155d50 0x7f6a8414bf90 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7f6a78011a70 tx=0x7f6a78011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.113+0000 7f6a80ff9640 1 -- 172.21.15.1:0/2925525415 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7800fb50 con 0x7f6a84155d50 2024-03-06T07:33:32.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.113+0000 7f6a80ff9640 1 -- 172.21.15.1:0/2925525415 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7800fcf0 con 0x7f6a84155d50 2024-03-06T07:33:32.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.113+0000 7f6a8a6fb640 1 -- 172.21.15.1:0/2925525415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a8406cb90 con 0x7f6a84155d50 2024-03-06T07:33:32.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.113+0000 7f6a80ff9640 1 -- 172.21.15.1:0/2925525415 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7800fea0 con 0x7f6a84155d50 2024-03-06T07:33:32.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.113+0000 7f6a8a6fb640 1 -- 172.21.15.1:0/2925525415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a84146b20 con 0x7f6a84155d50 2024-03-06T07:33:32.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.114+0000 7f6a8a6fb640 1 -- 172.21.15.1:0/2925525415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6a84156f50 con 0x7f6a84155d50 2024-03-06T07:33:32.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.114+0000 7f6a80ff9640 1 -- 172.21.15.1:0/2925525415 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a78012040 con 0x7f6a84155d50 2024-03-06T07:33:32.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.115+0000 7f6a80ff9640 1 --2- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6a7006b500 0x7f6a7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.115+0000 7f6a80ff9640 1 -- 172.21.15.1:0/2925525415 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f6a7808a7a0 con 0x7f6a84155d50 2024-03-06T07:33:32.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.115+0000 7f6a83fff640 1 --2- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6a7006b500 0x7f6a7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.115+0000 7f6a80ff9640 1 --2- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:6816/851921598,v1:172.21.15.98:6817/851921598] conn(0x7f6a70071b80 0x7f6a70073fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.115+0000 7f6a80ff9640 1 -- 172.21.15.1:0/2925525415 --> [v2:172.21.15.98:6816/851921598,v1:172.21.15.98:6817/851921598] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a78004490 con 0x7f6a70071b80 2024-03-06T07:33:32.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.115+0000 7f6a80ff9640 1 -- 172.21.15.1:0/2925525415 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=217) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a7808f2c0 con 0x7f6a84155d50 2024-03-06T07:33:32.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.115+0000 7f6a88c98640 1 --2- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:6816/851921598,v1:172.21.15.98:6817/851921598] conn(0x7f6a70071b80 0x7f6a70073fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.115+0000 7f6a83fff640 1 --2- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6a7006b500 0x7f6a7006d9c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f6a74036000 tx=0x7f6a74030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.115+0000 7f6a88c98640 1 --2- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:6816/851921598,v1:172.21.15.98:6817/851921598] conn(0x7f6a70071b80 0x7f6a70073fc0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.116+0000 7f6a80ff9640 1 -- 172.21.15.1:0/2925525415 <== osd.5 v2:172.21.15.98:6816/851921598 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6a78004490 con 0x7f6a70071b80 2024-03-06T07:33:32.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.129+0000 7f6a8a6fb640 1 -- 172.21.15.1:0/2925525415 --> [v2:172.21.15.98:6816/851921598,v1:172.21.15.98:6817/851921598] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6a8406c980 con 0x7f6a70071b80 2024-03-06T07:33:32.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.130+0000 7f6a80ff9640 1 -- 172.21.15.1:0/2925525415 <== osd.5 v2:172.21.15.98:6816/851921598 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6a8406c980 con 0x7f6a70071b80 2024-03-06T07:33:32.185 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:33:32.185 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:33:32.185 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:33:32.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.130+0000 7f6a8a6fb640 1 -- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:6816/851921598,v1:172.21.15.98:6817/851921598] conn(0x7f6a70071b80 msgr2=0x7f6a70073fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.130+0000 7f6a8a6fb640 1 --2- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:6816/851921598,v1:172.21.15.98:6817/851921598] conn(0x7f6a70071b80 0x7f6a70073fc0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.130+0000 7f6a8a6fb640 1 -- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6a7006b500 msgr2=0x7f6a7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.130+0000 7f6a8a6fb640 1 --2- 172.21.15.1:0/2925525415 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f6a7006b500 0x7f6a7006d9c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f6a74036000 tx=0x7f6a74030000 comp rx=0 tx=0).stop 2024-03-06T07:33:32.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.130+0000 7f6a8a6fb640 1 -- 172.21.15.1:0/2925525415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a84155d50 msgr2=0x7f6a8414bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.130+0000 7f6a8a6fb640 1 --2- 172.21.15.1:0/2925525415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a84155d50 0x7f6a8414bf90 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7f6a78011a70 tx=0x7f6a78011f30 comp rx=0 tx=0).stop 2024-03-06T07:33:32.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.130+0000 7f6a88c98640 1 -- 172.21.15.1:0/2925525415 reap_dead start 2024-03-06T07:33:32.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.130+0000 7f6a8a6fb640 1 -- 172.21.15.1:0/2925525415 shutdown_connections 2024-03-06T07:33:32.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.130+0000 7f6a8a6fb640 1 -- 172.21.15.1:0/2925525415 >> 172.21.15.1:0/2925525415 conn(0x7f6a840aa6a0 msgr2=0x7f6a841526e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.130+0000 7f6a8a6fb640 1 -- 172.21.15.1:0/2925525415 shutdown_connections 2024-03-06T07:33:32.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.130+0000 7f6a8a6fb640 1 -- 172.21.15.1:0/2925525415 wait complete. 2024-03-06T07:33:32.188 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-06T07:33:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.288+0000 7f20947bc640 1 Processor -- start 2024-03-06T07:33:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.289+0000 7f20947bc640 1 -- start start 2024-03-06T07:33:32.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.289+0000 7f20947bc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f208c1505b0 0x7f208c0c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.289+0000 7f20947bc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f208c0c16e0 0x7f208c156df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.289+0000 7f20947bc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c06f5c0 con 0x7f208c0c1260 2024-03-06T07:33:32.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.289+0000 7f20947bc640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208c152f80 con 0x7f208c0c16e0 2024-03-06T07:33:32.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.289+0000 7f20947bc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208c150990 con 0x7f208c1505b0 2024-03-06T07:33:32.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.290+0000 7f2092d59640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f208c0c16e0 0x7f208c156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.290+0000 7f2091d57640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f208c0c1260 0x7f208c1566e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37144/0 (socket says 172.21.15.1:37144) 2024-03-06T07:33:32.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.290+0000 7f2091d57640 1 -- 172.21.15.1:0/1845545496 learned_addr learned my addr 172.21.15.1:0/1845545496 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.290+0000 7f2092558640 1 --2- 172.21.15.1:0/1845545496 >> v2:172.21.15.98:3300/0 conn(0x7f208c1505b0 0x7f208c0c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.290+0000 7f2092d59640 1 -- 172.21.15.1:0/1845545496 >> v2:172.21.15.98:3300/0 conn(0x7f208c1505b0 msgr2=0x7f208c0c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.290+0000 7f2092d59640 1 --2- 172.21.15.1:0/1845545496 >> v2:172.21.15.98:3300/0 conn(0x7f208c1505b0 0x7f208c0c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.290+0000 7f2092d59640 1 -- 172.21.15.1:0/1845545496 >> v1:172.21.15.1:6789/0 conn(0x7f208c0c1260 legacy=0x7f208c1566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.290+0000 7f2091556640 1 -- 172.21.15.1:0/1845545496 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056809188 0 0) 0x7f208c06f5c0 con 0x7f208c0c1260 2024-03-06T07:33:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.290+0000 7f2092d59640 1 -- 172.21.15.1:0/1845545496 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c153240 con 0x7f208c0c16e0 2024-03-06T07:33:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.290+0000 7f2092558640 1 --2- 172.21.15.1:0/1845545496 >> v2:172.21.15.98:3300/0 conn(0x7f208c1505b0 0x7f208c0c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.291+0000 7f2092d59640 1 --2- 172.21.15.1:0/1845545496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c0c16e0 0x7f208c156df0 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f208c0c1130 tx=0x7f208402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d58d003ae1eef62 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.291+0000 7f2091556640 1 -- 172.21.15.1:0/1845545496 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20840315a0 con 0x7f208c0c16e0 2024-03-06T07:33:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.291+0000 7f2091556640 1 -- 172.21.15.1:0/1845545496 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2084037070 con 0x7f208c0c16e0 2024-03-06T07:33:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.291+0000 7f2091556640 1 -- 172.21.15.1:0/1845545496 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2084031960 con 0x7f208c0c16e0 2024-03-06T07:33:32.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.291+0000 7f20947bc640 1 -- 172.21.15.1:0/1845545496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c0c16e0 msgr2=0x7f208c156df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.291+0000 7f20947bc640 1 --2- 172.21.15.1:0/1845545496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c0c16e0 0x7f208c156df0 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f208c0c1130 tx=0x7f208402b840 comp rx=0 tx=0).stop 2024-03-06T07:33:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.291+0000 7f20947bc640 1 -- 172.21.15.1:0/1845545496 shutdown_connections 2024-03-06T07:33:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.291+0000 7f20947bc640 1 --2- 172.21.15.1:0/1845545496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c0c16e0 0x7f208c156df0 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.291+0000 7f20947bc640 1 --2- 172.21.15.1:0/1845545496 >> v2:172.21.15.98:3300/0 conn(0x7f208c1505b0 0x7f208c0c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.291+0000 7f20947bc640 1 -- 172.21.15.1:0/1845545496 >> 172.21.15.1:0/1845545496 conn(0x7f208c0c6860 msgr2=0x7f208c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.291+0000 7f20947bc640 1 -- 172.21.15.1:0/1845545496 shutdown_connections 2024-03-06T07:33:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.291+0000 7f20947bc640 1 -- 172.21.15.1:0/1845545496 wait complete. 2024-03-06T07:33:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.292+0000 7f20947bc640 1 Processor -- start 2024-03-06T07:33:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.293+0000 7f20947bc640 1 -- start start 2024-03-06T07:33:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.293+0000 7f20947bc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c0c1260 0x7f208c149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.293+0000 7f20947bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f208c0c16e0 0x7f208c0ac200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.293+0000 7f20947bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f208c1505b0 0x7f208c1ebd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.293+0000 7f20947bc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c153ae0 con 0x7f208c0c1260 2024-03-06T07:33:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.293+0000 7f20947bc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c153800 con 0x7f208c1505b0 2024-03-06T07:33:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.293+0000 7f20947bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208c157330 con 0x7f208c0c16e0 2024-03-06T07:33:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.293+0000 7f2092558640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c0c1260 0x7f208c149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.293+0000 7f2092558640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c0c1260 0x7f208c149b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40828/0 (socket says 172.21.15.1:40828) 2024-03-06T07:33:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.293+0000 7f2092558640 1 -- 172.21.15.1:0/766344847 learned_addr learned my addr 172.21.15.1:0/766344847 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.293+0000 7f2092558640 1 -- 172.21.15.1:0/766344847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f208c0c16e0 msgr2=0x7f208c0ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.293+0000 7f2092558640 1 --2- 172.21.15.1:0/766344847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f208c0c16e0 0x7f208c0ac200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.294+0000 7f2092558640 1 -- 172.21.15.1:0/766344847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f208c1505b0 msgr2=0x7f208c1ebd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.294+0000 7f2092558640 1 --2- 172.21.15.1:0/766344847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f208c1505b0 0x7f208c1ebd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.294+0000 7f2092558640 1 -- 172.21.15.1:0/766344847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c151220 con 0x7f208c0c1260 2024-03-06T07:33:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.294+0000 7f2092558640 1 --2- 172.21.15.1:0/766344847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c0c1260 0x7f208c149b90 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f207c00cf60 tx=0x7f207c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.294+0000 7f208affd640 1 -- 172.21.15.1:0/766344847 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c01a070 con 0x7f208c0c1260 2024-03-06T07:33:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.294+0000 7f208affd640 1 -- 172.21.15.1:0/766344847 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207c00d840 con 0x7f208c0c1260 2024-03-06T07:33:32.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.294+0000 7f20947bc640 1 -- 172.21.15.1:0/766344847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208c1ecc30 con 0x7f208c0c1260 2024-03-06T07:33:32.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.294+0000 7f208affd640 1 -- 172.21.15.1:0/766344847 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c00dc10 con 0x7f208c0c1260 2024-03-06T07:33:32.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.294+0000 7f20947bc640 1 -- 172.21.15.1:0/766344847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f208c1dcb50 con 0x7f208c0c1260 2024-03-06T07:33:32.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.295+0000 7f20947bc640 1 -- 172.21.15.1:0/766344847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208c0686e0 con 0x7f208c0c1260 2024-03-06T07:33:32.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.295+0000 7f208affd640 1 -- 172.21.15.1:0/766344847 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f207c01d070 con 0x7f208c0c1260 2024-03-06T07:33:32.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.296+0000 7f208affd640 1 --2- 172.21.15.1:0/766344847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f207806b500 0x7f207806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.296+0000 7f208affd640 1 -- 172.21.15.1:0/766344847 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f207c08a860 con 0x7f208c0c1260 2024-03-06T07:33:32.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.296+0000 7f2091d57640 1 --2- 172.21.15.1:0/766344847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f207806b500 0x7f207806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.296+0000 7f2091d57640 1 --2- 172.21.15.1:0/766344847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f207806b500 0x7f207806d9c0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f2080007fd0 tx=0x7f2080007960 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.300+0000 7f208affd640 1 -- 172.21.15.1:0/766344847 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f208c0686e0 con 0x7f208c0c1260 2024-03-06T07:33:32.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.482+0000 7f20947bc640 1 -- 172.21.15.1:0/766344847 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f208c067fd0 con 0x7f207806b500 2024-03-06T07:33:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.482+0000 7f208affd640 1 -- 172.21.15.1:0/766344847 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f208c067fd0 con 0x7f207806b500 2024-03-06T07:33:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 5 to deep-scrub 2024-03-06T07:33:32.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.484+0000 7f20947bc640 1 -- 172.21.15.1:0/766344847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f207806b500 msgr2=0x7f207806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.484+0000 7f20947bc640 1 --2- 172.21.15.1:0/766344847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f207806b500 0x7f207806d9c0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f2080007fd0 tx=0x7f2080007960 comp rx=0 tx=0).stop 2024-03-06T07:33:32.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.484+0000 7f20947bc640 1 -- 172.21.15.1:0/766344847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c0c1260 msgr2=0x7f208c149b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.484+0000 7f20947bc640 1 --2- 172.21.15.1:0/766344847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c0c1260 0x7f208c149b90 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f207c00cf60 tx=0x7f207c00cf90 comp rx=0 tx=0).stop 2024-03-06T07:33:32.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.484+0000 7f20947bc640 1 -- 172.21.15.1:0/766344847 shutdown_connections 2024-03-06T07:33:32.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.484+0000 7f20947bc640 1 --2- 172.21.15.1:0/766344847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f207806b500 0x7f207806d9c0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.484+0000 7f20947bc640 1 --2- 172.21.15.1:0/766344847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f208c1505b0 0x7f208c1ebd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.484+0000 7f20947bc640 1 --2- 172.21.15.1:0/766344847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f208c0c16e0 0x7f208c0ac200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.484+0000 7f20947bc640 1 --2- 172.21.15.1:0/766344847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c0c1260 0x7f208c149b90 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.484+0000 7f20947bc640 1 -- 172.21.15.1:0/766344847 >> 172.21.15.1:0/766344847 conn(0x7f208c0c6860 msgr2=0x7f208c1566e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.484+0000 7f20947bc640 1 -- 172.21.15.1:0/766344847 shutdown_connections 2024-03-06T07:33:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.484+0000 7f20947bc640 1 -- 172.21.15.1:0/766344847 wait complete. 2024-03-06T07:33:32.496 INFO:tasks.ceph:Scrubbing osd.6 2024-03-06T07:33:32.497 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:33:32.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.601+0000 7f19fbb5e640 1 Processor -- start 2024-03-06T07:33:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.602+0000 7f19fbb5e640 1 -- start start 2024-03-06T07:33:32.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.602+0000 7f19fbb5e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19f40c1010 0x7f19f40c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.602+0000 7f19fbb5e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f40c7500 0x7f19f40cab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.602+0000 7f19fbb5e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f40c6710 con 0x7f19f4152780 2024-03-06T07:33:32.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.602+0000 7f19fbb5e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f40c6c90 con 0x7f19f40c1010 2024-03-06T07:33:32.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.602+0000 7f19fbb5e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f40c69b0 con 0x7f19f40c7500 2024-03-06T07:33:32.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.602+0000 7f19f98fa640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f19f4152780 0x7f19f40c6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37152/0 (socket says 172.21.15.1:37152) 2024-03-06T07:33:32.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.602+0000 7f19f98fa640 1 -- 172.21.15.1:0/190881202 learned_addr learned my addr 172.21.15.1:0/190881202 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:32.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f90f9640 1 --2- 172.21.15.1:0/190881202 >> v2:172.21.15.98:3300/0 conn(0x7f19f40c1010 0x7f19f40c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19fa0fb640 1 --2- 172.21.15.1:0/190881202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f40c7500 0x7f19f40cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f88f8640 1 -- 172.21.15.1:0/190881202 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267126876 0 0) 0x7f19f40c6710 con 0x7f19f4152780 2024-03-06T07:33:32.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f88f8640 1 -- 172.21.15.1:0/190881202 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e0003520 con 0x7f19f4152780 2024-03-06T07:33:32.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f88f8640 1 -- 172.21.15.1:0/190881202 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2033493026 0 0) 0x7f19e0003520 con 0x7f19f4152780 2024-03-06T07:33:32.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f88f8640 1 -- 172.21.15.1:0/190881202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f40c7500 msgr2=0x7f19f40cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f88f8640 1 --2- 172.21.15.1:0/190881202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f40c7500 0x7f19f40cab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f88f8640 1 -- 172.21.15.1:0/190881202 >> v2:172.21.15.98:3300/0 conn(0x7f19f40c1010 msgr2=0x7f19f40c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f88f8640 1 --2- 172.21.15.1:0/190881202 >> v2:172.21.15.98:3300/0 conn(0x7f19f40c1010 0x7f19f40c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f88f8640 1 -- 172.21.15.1:0/190881202 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f4155370 con 0x7f19f4152780 2024-03-06T07:33:32.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f88f8640 1 -- 172.21.15.1:0/190881202 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f19f0001690 con 0x7f19f4152780 2024-03-06T07:33:32.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f90f9640 1 --2- 172.21.15.1:0/190881202 >> v2:172.21.15.98:3300/0 conn(0x7f19f40c1010 0x7f19f40c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f88f8640 1 -- 172.21.15.1:0/190881202 >> v1:172.21.15.1:6789/0 conn(0x7f19f4152780 legacy=0x7f19f40c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f88f8640 1 --2- 172.21.15.1:0/190881202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19e00039b0 0x7f19e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f88f8640 1 --2- 172.21.15.1:0/190881202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19e0004800 0x7f19e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f88f8640 1 --2- 172.21.15.1:0/190881202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f40c7500 0x7f19e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f88f8640 1 -- 172.21.15.1:0/190881202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f40c6c90 con 0x7f19e00039b0 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f88f8640 1 -- 172.21.15.1:0/190881202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f40c69b0 con 0x7f19e0004800 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f88f8640 1 -- 172.21.15.1:0/190881202 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e000b9c0 con 0x7f19f40c7500 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f98fa640 1 --2- 172.21.15.1:0/190881202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19e00039b0 0x7f19e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19fa0fb640 1 --2- 172.21.15.1:0/190881202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f40c7500 0x7f19e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f90f9640 1 --2- 172.21.15.1:0/190881202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19e0004800 0x7f19e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f98fa640 1 -- 172.21.15.1:0/190881202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f40c7500 msgr2=0x7f19e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f98fa640 1 --2- 172.21.15.1:0/190881202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f40c7500 0x7f19e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f98fa640 1 -- 172.21.15.1:0/190881202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19e0004800 msgr2=0x7f19e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f98fa640 1 --2- 172.21.15.1:0/190881202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19e0004800 0x7f19e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f98fa640 1 -- 172.21.15.1:0/190881202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e0003520 con 0x7f19e00039b0 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19f90f9640 1 --2- 172.21.15.1:0/190881202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19e0004800 0x7f19e0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:32.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.603+0000 7f19fa0fb640 1 --2- 172.21.15.1:0/190881202 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f40c7500 0x7f19e00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:32.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.604+0000 7f19f98fa640 1 --2- 172.21.15.1:0/190881202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19e00039b0 0x7f19e0003dd0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7f19f000c240 tx=0x7f19e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8dd68f1fcb79ae4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.604+0000 7f19f88f8640 1 -- 172.21.15.1:0/190881202 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f0007d70 con 0x7f19e00039b0 2024-03-06T07:33:32.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.604+0000 7f19f88f8640 1 -- 172.21.15.1:0/190881202 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f0030070 con 0x7f19e00039b0 2024-03-06T07:33:32.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.604+0000 7f19f88f8640 1 -- 172.21.15.1:0/190881202 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f002cd40 con 0x7f19e00039b0 2024-03-06T07:33:32.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.605+0000 7f19fbb5e640 1 -- 172.21.15.1:0/190881202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19e00039b0 msgr2=0x7f19e0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.605+0000 7f19fbb5e640 1 --2- 172.21.15.1:0/190881202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19e00039b0 0x7f19e0003dd0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7f19f000c240 tx=0x7f19e0005830 comp rx=0 tx=0).stop 2024-03-06T07:33:32.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.605+0000 7f19fa0fb640 1 -- 172.21.15.1:0/190881202 reap_dead start 2024-03-06T07:33:32.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.605+0000 7f19fbb5e640 1 -- 172.21.15.1:0/190881202 shutdown_connections 2024-03-06T07:33:32.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.605+0000 7f19fbb5e640 1 -- 172.21.15.1:0/190881202 >> 172.21.15.1:0/190881202 conn(0x7f19f408b720 msgr2=0x7f19f406d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.605+0000 7f19fbb5e640 1 -- 172.21.15.1:0/190881202 shutdown_connections 2024-03-06T07:33:32.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.605+0000 7f19fbb5e640 1 -- 172.21.15.1:0/190881202 wait complete. 2024-03-06T07:33:32.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.606+0000 7f19fbb5e640 1 Processor -- start 2024-03-06T07:33:32.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.606+0000 7f19fbb5e640 1 -- start start 2024-03-06T07:33:32.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.607+0000 7f19fbb5e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19e00039b0 0x7f19f4151150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.607+0000 7f19fbb5e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f40c8970 0x7f19f41ebf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.607+0000 7f19fbb5e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f41de960 0x7f19f41ec4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.607+0000 7f19fbb5e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f4152b60 con 0x7f19e00039b0 2024-03-06T07:33:32.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.607+0000 7f19fbb5e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f4155930 con 0x7f19f41de960 2024-03-06T07:33:32.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.607+0000 7f19fbb5e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f4155e50 con 0x7f19f40c8970 2024-03-06T07:33:32.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.607+0000 7f19f98fa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19e00039b0 0x7f19f4151150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.607+0000 7f19f98fa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19e00039b0 0x7f19f4151150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40834/0 (socket says 172.21.15.1:40834) 2024-03-06T07:33:32.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.607+0000 7f19f98fa640 1 -- 172.21.15.1:0/1442992349 learned_addr learned my addr 172.21.15.1:0/1442992349 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:32.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.607+0000 7f19f90f9640 1 --2- 172.21.15.1:0/1442992349 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f40c8970 0x7f19f41ebf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.607+0000 7f19fa0fb640 1 --2- 172.21.15.1:0/1442992349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f41de960 0x7f19f41ec4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.607+0000 7f19f98fa640 1 -- 172.21.15.1:0/1442992349 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f40c8970 msgr2=0x7f19f41ebf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.607+0000 7f19f98fa640 1 --2- 172.21.15.1:0/1442992349 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f19f40c8970 0x7f19f41ebf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.607+0000 7f19f98fa640 1 -- 172.21.15.1:0/1442992349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f41de960 msgr2=0x7f19f41ec4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.607+0000 7f19f98fa640 1 --2- 172.21.15.1:0/1442992349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f41de960 0x7f19f41ec4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.607+0000 7f19f98fa640 1 -- 172.21.15.1:0/1442992349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f40c8e70 con 0x7f19e00039b0 2024-03-06T07:33:32.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.607+0000 7f19f98fa640 1 --2- 172.21.15.1:0/1442992349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19e00039b0 0x7f19f4151150 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f19f000c360 tx=0x7f19f0001a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.608+0000 7f19ea7fc640 1 -- 172.21.15.1:0/1442992349 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f0045070 con 0x7f19e00039b0 2024-03-06T07:33:32.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.608+0000 7f19ea7fc640 1 -- 172.21.15.1:0/1442992349 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f0004bf0 con 0x7f19e00039b0 2024-03-06T07:33:32.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.608+0000 7f19fbb5e640 1 -- 172.21.15.1:0/1442992349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f40c9150 con 0x7f19e00039b0 2024-03-06T07:33:32.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.608+0000 7f19ea7fc640 1 -- 172.21.15.1:0/1442992349 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f0007d70 con 0x7f19e00039b0 2024-03-06T07:33:32.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.608+0000 7f19fbb5e640 1 -- 172.21.15.1:0/1442992349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f40c9710 con 0x7f19e00039b0 2024-03-06T07:33:32.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.609+0000 7f19fbb5e640 1 -- 172.21.15.1:0/1442992349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f19f40680b0 con 0x7f19e00039b0 2024-03-06T07:33:32.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.609+0000 7f19ea7fc640 1 -- 172.21.15.1:0/1442992349 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19f0007070 con 0x7f19e00039b0 2024-03-06T07:33:32.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.609+0000 7f19ea7fc640 1 --2- 172.21.15.1:0/1442992349 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19d806b500 0x7f19d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.609+0000 7f19ea7fc640 1 -- 172.21.15.1:0/1442992349 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f19f00aaf20 con 0x7f19e00039b0 2024-03-06T07:33:32.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.610+0000 7f19ea7fc640 1 --2- 172.21.15.1:0/1442992349 >> [v2:172.21.15.98:6800/2463344330,v1:172.21.15.98:6801/2463344330] conn(0x7f19d8071b80 0x7f19d8073fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.610+0000 7f19ea7fc640 1 -- 172.21.15.1:0/1442992349 --> [v2:172.21.15.98:6800/2463344330,v1:172.21.15.98:6801/2463344330] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19f0041e50 con 0x7f19d8071b80 2024-03-06T07:33:32.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.610+0000 7f19ea7fc640 1 -- 172.21.15.1:0/1442992349 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=217) v2 ==== 24+0+0 (secure 0 0 0) 0x7f19f00ad610 con 0x7f19e00039b0 2024-03-06T07:33:32.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.610+0000 7f19f90f9640 1 --2- 172.21.15.1:0/1442992349 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19d806b500 0x7f19d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.610+0000 7f19fa0fb640 1 --2- 172.21.15.1:0/1442992349 >> [v2:172.21.15.98:6800/2463344330,v1:172.21.15.98:6801/2463344330] conn(0x7f19d8071b80 0x7f19d8073fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.610+0000 7f19f90f9640 1 --2- 172.21.15.1:0/1442992349 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19d806b500 0x7f19d806d9c0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f19f41eed60 tx=0x7f19e4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.610+0000 7f19fa0fb640 1 --2- 172.21.15.1:0/1442992349 >> [v2:172.21.15.98:6800/2463344330,v1:172.21.15.98:6801/2463344330] conn(0x7f19d8071b80 0x7f19d8073fc0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.611+0000 7f19ea7fc640 1 -- 172.21.15.1:0/1442992349 <== osd.6 v2:172.21.15.98:6800/2463344330 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f19f0041e50 con 0x7f19d8071b80 2024-03-06T07:33:32.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.624+0000 7f19fbb5e640 1 -- 172.21.15.1:0/1442992349 --> [v2:172.21.15.98:6800/2463344330,v1:172.21.15.98:6801/2463344330] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f19f40c5de0 con 0x7f19d8071b80 2024-03-06T07:33:32.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.626+0000 7f19ea7fc640 1 -- 172.21.15.1:0/1442992349 <== osd.6 v2:172.21.15.98:6800/2463344330 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f19f40c5de0 con 0x7f19d8071b80 2024-03-06T07:33:32.627 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:33:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:33:32.627 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:33:32.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.626+0000 7f19fbb5e640 1 -- 172.21.15.1:0/1442992349 >> [v2:172.21.15.98:6800/2463344330,v1:172.21.15.98:6801/2463344330] conn(0x7f19d8071b80 msgr2=0x7f19d8073fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.626+0000 7f19fbb5e640 1 --2- 172.21.15.1:0/1442992349 >> [v2:172.21.15.98:6800/2463344330,v1:172.21.15.98:6801/2463344330] conn(0x7f19d8071b80 0x7f19d8073fc0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.626+0000 7f19fbb5e640 1 -- 172.21.15.1:0/1442992349 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19d806b500 msgr2=0x7f19d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.626+0000 7f19fbb5e640 1 --2- 172.21.15.1:0/1442992349 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f19d806b500 0x7f19d806d9c0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f19f41eed60 tx=0x7f19e4008040 comp rx=0 tx=0).stop 2024-03-06T07:33:32.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.626+0000 7f19fbb5e640 1 -- 172.21.15.1:0/1442992349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19e00039b0 msgr2=0x7f19f4151150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.626+0000 7f19fbb5e640 1 --2- 172.21.15.1:0/1442992349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19e00039b0 0x7f19f4151150 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f19f000c360 tx=0x7f19f0001a60 comp rx=0 tx=0).stop 2024-03-06T07:33:32.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.626+0000 7f19fa0fb640 1 -- 172.21.15.1:0/1442992349 reap_dead start 2024-03-06T07:33:32.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.626+0000 7f19fbb5e640 1 -- 172.21.15.1:0/1442992349 shutdown_connections 2024-03-06T07:33:32.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.626+0000 7f19fbb5e640 1 -- 172.21.15.1:0/1442992349 >> 172.21.15.1:0/1442992349 conn(0x7f19f408b720 msgr2=0x7f19f406d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.626+0000 7f19fbb5e640 1 -- 172.21.15.1:0/1442992349 shutdown_connections 2024-03-06T07:33:32.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.626+0000 7f19fbb5e640 1 -- 172.21.15.1:0/1442992349 wait complete. 2024-03-06T07:33:32.637 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-06T07:33:32.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.738+0000 7f3af635e640 1 Processor -- start 2024-03-06T07:33:32.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.738+0000 7f3af635e640 1 -- start start 2024-03-06T07:33:32.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.739+0000 7f3af635e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3af00c3eb0 0x7f3af00c7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.739+0000 7f3af635e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3af006d060 0x7f3af01558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.739+0000 7f3af635e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af00c71c0 con 0x7f3af00c7940 2024-03-06T07:33:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.739+0000 7f3af635e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af00c59b0 con 0x7f3af00c3eb0 2024-03-06T07:33:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.739+0000 7f3af635e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af00c4c00 con 0x7f3af006d060 2024-03-06T07:33:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.739+0000 7f3aeffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3af00c3eb0 0x7f3af00c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.739+0000 7f3af48fb640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3af00c7940 0x7f3af0158760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37160/0 (socket says 172.21.15.1:37160) 2024-03-06T07:33:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.739+0000 7f3af48fb640 1 -- 172.21.15.1:0/4032247866 learned_addr learned my addr 172.21.15.1:0/4032247866 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.739+0000 7f3aef7fe640 1 --2- 172.21.15.1:0/4032247866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3af006d060 0x7f3af01558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.739+0000 7f3aeffff640 1 -- 172.21.15.1:0/4032247866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3af006d060 msgr2=0x7f3af01558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.739+0000 7f3aeffff640 1 --2- 172.21.15.1:0/4032247866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3af006d060 0x7f3af01558b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.739+0000 7f3aeffff640 1 -- 172.21.15.1:0/4032247866 >> v1:172.21.15.1:6789/0 conn(0x7f3af00c7940 legacy=0x7f3af0158760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.739+0000 7f3aeffff640 1 -- 172.21.15.1:0/4032247866 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af00c4e50 con 0x7f3af00c3eb0 2024-03-06T07:33:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.739+0000 7f3aeeffd640 1 -- 172.21.15.1:0/4032247866 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460143178 0 0) 0x7f3af00c71c0 con 0x7f3af00c7940 2024-03-06T07:33:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.739+0000 7f3aef7fe640 1 --2- 172.21.15.1:0/4032247866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3af006d060 0x7f3af01558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.740+0000 7f3aeffff640 1 --2- 172.21.15.1:0/4032247866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3af00c3eb0 0x7f3af00c7400 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f3ad80097a0 tx=0x7f3ad802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600f6f81aa872ab6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.740+0000 7f3aeeffd640 1 -- 172.21.15.1:0/4032247866 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad8038070 con 0x7f3af00c3eb0 2024-03-06T07:33:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.740+0000 7f3aeeffd640 1 -- 172.21.15.1:0/4032247866 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad8031700 con 0x7f3af00c3eb0 2024-03-06T07:33:32.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.740+0000 7f3aeeffd640 1 -- 172.21.15.1:0/4032247866 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad8031ad0 con 0x7f3af00c3eb0 2024-03-06T07:33:32.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.740+0000 7f3af635e640 1 -- 172.21.15.1:0/4032247866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3af00c3eb0 msgr2=0x7f3af00c7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.740+0000 7f3af635e640 1 --2- 172.21.15.1:0/4032247866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3af00c3eb0 0x7f3af00c7400 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f3ad80097a0 tx=0x7f3ad802b7a0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.740+0000 7f3af635e640 1 -- 172.21.15.1:0/4032247866 shutdown_connections 2024-03-06T07:33:32.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.740+0000 7f3af635e640 1 --2- 172.21.15.1:0/4032247866 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3af006d060 0x7f3af01558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.740+0000 7f3af635e640 1 --2- 172.21.15.1:0/4032247866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3af00c3eb0 0x7f3af00c7400 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.740+0000 7f3af635e640 1 -- 172.21.15.1:0/4032247866 >> 172.21.15.1:0/4032247866 conn(0x7f3af0066d00 msgr2=0x7f3af0067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.741+0000 7f3af635e640 1 -- 172.21.15.1:0/4032247866 shutdown_connections 2024-03-06T07:33:32.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.741+0000 7f3af635e640 1 -- 172.21.15.1:0/4032247866 wait complete. 2024-03-06T07:33:32.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.742+0000 7f3af635e640 1 Processor -- start 2024-03-06T07:33:32.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.742+0000 7f3af635e640 1 -- start start 2024-03-06T07:33:32.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.742+0000 7f3af635e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3af006d060 0x7f3af014d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.742+0000 7f3af635e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3af00c3eb0 0x7f3af00ae6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.742+0000 7f3af635e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3af00c7940 0x7f3af0156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.742+0000 7f3af635e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af00c4290 con 0x7f3af00c7940 2024-03-06T07:33:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.742+0000 7f3af635e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af0155420 con 0x7f3af00c3eb0 2024-03-06T07:33:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.742+0000 7f3af635e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af01596d0 con 0x7f3af006d060 2024-03-06T07:33:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.742+0000 7f3af48fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3af00c7940 0x7f3af0156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.743+0000 7f3af48fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3af00c7940 0x7f3af0156920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40864/0 (socket says 172.21.15.1:40864) 2024-03-06T07:33:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.743+0000 7f3af48fb640 1 -- 172.21.15.1:0/4159125808 learned_addr learned my addr 172.21.15.1:0/4159125808 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.743+0000 7f3aeffff640 1 --2- 172.21.15.1:0/4159125808 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3af006d060 0x7f3af014d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.743+0000 7f3aef7fe640 1 --2- 172.21.15.1:0/4159125808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3af00c3eb0 0x7f3af00ae6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.743+0000 7f3af48fb640 1 -- 172.21.15.1:0/4159125808 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3af006d060 msgr2=0x7f3af014d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.743+0000 7f3af48fb640 1 --2- 172.21.15.1:0/4159125808 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3af006d060 0x7f3af014d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.743+0000 7f3af48fb640 1 -- 172.21.15.1:0/4159125808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3af00c3eb0 msgr2=0x7f3af00ae6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.743+0000 7f3af48fb640 1 --2- 172.21.15.1:0/4159125808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3af00c3eb0 0x7f3af00ae6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.743+0000 7f3af48fb640 1 -- 172.21.15.1:0/4159125808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af00caa20 con 0x7f3af00c7940 2024-03-06T07:33:32.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.743+0000 7f3af48fb640 1 --2- 172.21.15.1:0/4159125808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3af00c7940 0x7f3af0156920 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7f3ae800c7a0 tx=0x7f3ae800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.743+0000 7f3aecff9640 1 -- 172.21.15.1:0/4159125808 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8018070 con 0x7f3af00c7940 2024-03-06T07:33:32.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.743+0000 7f3aecff9640 1 -- 172.21.15.1:0/4159125808 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae800a640 con 0x7f3af00c7940 2024-03-06T07:33:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.743+0000 7f3af635e640 1 -- 172.21.15.1:0/4159125808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af00cad00 con 0x7f3af00c7940 2024-03-06T07:33:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.743+0000 7f3aecff9640 1 -- 172.21.15.1:0/4159125808 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae800aab0 con 0x7f3af00c7940 2024-03-06T07:33:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.743+0000 7f3af635e640 1 -- 172.21.15.1:0/4159125808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af00ca5d0 con 0x7f3af00c7940 2024-03-06T07:33:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.744+0000 7f3af635e640 1 -- 172.21.15.1:0/4159125808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af006ce80 con 0x7f3af00c7940 2024-03-06T07:33:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.745+0000 7f3aecff9640 1 -- 172.21.15.1:0/4159125808 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ae801b070 con 0x7f3af00c7940 2024-03-06T07:33:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.745+0000 7f3aecff9640 1 --2- 172.21.15.1:0/4159125808 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3ad006b500 0x7f3ad006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.745+0000 7f3aecff9640 1 -- 172.21.15.1:0/4159125808 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f3ae8089760 con 0x7f3af00c7940 2024-03-06T07:33:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.745+0000 7f3aeffff640 1 --2- 172.21.15.1:0/4159125808 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3ad006b500 0x7f3ad006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.746+0000 7f3aeffff640 1 --2- 172.21.15.1:0/4159125808 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3ad006b500 0x7f3ad006d9c0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8035000 tx=0x7f3ad8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.749+0000 7f3aecff9640 1 -- 172.21.15.1:0/4159125808 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3af006ce80 con 0x7f3af00c7940 2024-03-06T07:33:32.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.933+0000 7f3af635e640 1 -- 172.21.15.1:0/4159125808 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f3af0067fd0 con 0x7f3ad006b500 2024-03-06T07:33:32.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.934+0000 7f3aecff9640 1 -- 172.21.15.1:0/4159125808 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3af0067fd0 con 0x7f3ad006b500 2024-03-06T07:33:32.936 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 6 to deep-scrub 2024-03-06T07:33:32.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.936+0000 7f3af635e640 1 -- 172.21.15.1:0/4159125808 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3ad006b500 msgr2=0x7f3ad006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.936+0000 7f3af635e640 1 --2- 172.21.15.1:0/4159125808 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3ad006b500 0x7f3ad006d9c0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8035000 tx=0x7f3ad8008030 comp rx=0 tx=0).stop 2024-03-06T07:33:32.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.936+0000 7f3af635e640 1 -- 172.21.15.1:0/4159125808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3af00c7940 msgr2=0x7f3af0156920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.936+0000 7f3af635e640 1 --2- 172.21.15.1:0/4159125808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3af00c7940 0x7f3af0156920 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7f3ae800c7a0 tx=0x7f3ae800cc60 comp rx=0 tx=0).stop 2024-03-06T07:33:32.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.936+0000 7f3af635e640 1 -- 172.21.15.1:0/4159125808 shutdown_connections 2024-03-06T07:33:32.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.936+0000 7f3af635e640 1 --2- 172.21.15.1:0/4159125808 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3ad006b500 0x7f3ad006d9c0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.936+0000 7f3af635e640 1 --2- 172.21.15.1:0/4159125808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3af00c7940 0x7f3af0156920 unknown :-1 s=CLOSED pgs=2756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.936+0000 7f3af635e640 1 --2- 172.21.15.1:0/4159125808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3af00c3eb0 0x7f3af00ae6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.936+0000 7f3af635e640 1 --2- 172.21.15.1:0/4159125808 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3af006d060 0x7f3af014d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.936+0000 7f3af635e640 1 -- 172.21.15.1:0/4159125808 >> 172.21.15.1:0/4159125808 conn(0x7f3af0066d00 msgr2=0x7f3af00670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.936+0000 7f3af635e640 1 -- 172.21.15.1:0/4159125808 shutdown_connections 2024-03-06T07:33:32.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:32.936+0000 7f3af635e640 1 -- 172.21.15.1:0/4159125808 wait complete. 2024-03-06T07:33:32.948 INFO:tasks.ceph:Scrubbing osd.7 2024-03-06T07:33:32.949 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:33:33.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.052+0000 7fbd5c88a640 1 Processor -- start 2024-03-06T07:33:33.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.052+0000 7fbd5c88a640 1 -- start start 2024-03-06T07:33:33.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.053+0000 7fbd5c88a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbd580c16e0 0x7fbd58152470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.053+0000 7fbd5c88a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbd581529b0 0x7fbd58155c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.053+0000 7fbd5c88a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd58158890 con 0x7fbd580c70c0 2024-03-06T07:33:33.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.053+0000 7fbd5c88a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd58158b70 con 0x7fbd581529b0 2024-03-06T07:33:33.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.053+0000 7fbd5c88a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd58158e50 con 0x7fbd580c16e0 2024-03-06T07:33:33.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.053+0000 7fbd56d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbd581529b0 0x7fbd58155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.053+0000 7fbd56d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbd581529b0 0x7fbd58155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40876/0 (socket says 172.21.15.1:40876) 2024-03-06T07:33:33.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.053+0000 7fbd56d9d640 1 -- 172.21.15.1:0/1728635092 learned_addr learned my addr 172.21.15.1:0/1728635092 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:33.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.053+0000 7fbd5659c640 1 --2- 172.21.15.1:0/1728635092 >> v2:172.21.15.98:3300/0 conn(0x7fbd580c16e0 0x7fbd58152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.053+0000 7fbd56d9d640 1 -- 172.21.15.1:0/1728635092 >> v2:172.21.15.98:3300/0 conn(0x7fbd580c16e0 msgr2=0x7fbd58152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.053+0000 7fbd56d9d640 1 --2- 172.21.15.1:0/1728635092 >> v2:172.21.15.98:3300/0 conn(0x7fbd580c16e0 0x7fbd58152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.053+0000 7fbd56d9d640 1 -- 172.21.15.1:0/1728635092 >> v1:172.21.15.98:6789/0 conn(0x7fbd580c70c0 legacy=0x7fbd58155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.053+0000 7fbd56d9d640 1 -- 172.21.15.1:0/1728635092 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd58159130 con 0x7fbd581529b0 2024-03-06T07:33:33.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.054+0000 7fbd56d9d640 1 --2- 172.21.15.1:0/1728635092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd581529b0 0x7fbd58155c60 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7fbd5808bbd0 tx=0x7fbd4802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec02fb483f763f09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.054+0000 7fbd5559a640 1 -- 172.21.15.1:0/1728635092 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4802c9f0 con 0x7fbd581529b0 2024-03-06T07:33:33.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.054+0000 7fbd5559a640 1 -- 172.21.15.1:0/1728635092 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4802cb90 con 0x7fbd581529b0 2024-03-06T07:33:33.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.054+0000 7fbd5559a640 1 -- 172.21.15.1:0/1728635092 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd480343f0 con 0x7fbd581529b0 2024-03-06T07:33:33.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.055+0000 7fbd5c88a640 1 -- 172.21.15.1:0/1728635092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd581529b0 msgr2=0x7fbd58155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.055+0000 7fbd5c88a640 1 --2- 172.21.15.1:0/1728635092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd581529b0 0x7fbd58155c60 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7fbd5808bbd0 tx=0x7fbd4802be50 comp rx=0 tx=0).stop 2024-03-06T07:33:33.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.055+0000 7fbd5c88a640 1 -- 172.21.15.1:0/1728635092 shutdown_connections 2024-03-06T07:33:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.055+0000 7fbd5c88a640 1 --2- 172.21.15.1:0/1728635092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd581529b0 0x7fbd58155c60 unknown :-1 s=CLOSED pgs=2757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.055+0000 7fbd5c88a640 1 --2- 172.21.15.1:0/1728635092 >> v2:172.21.15.98:3300/0 conn(0x7fbd580c16e0 0x7fbd58152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.055+0000 7fbd5c88a640 1 -- 172.21.15.1:0/1728635092 >> 172.21.15.1:0/1728635092 conn(0x7fbd58066d00 msgr2=0x7fbd58067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.055+0000 7fbd5c88a640 1 -- 172.21.15.1:0/1728635092 shutdown_connections 2024-03-06T07:33:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.055+0000 7fbd5c88a640 1 -- 172.21.15.1:0/1728635092 wait complete. 2024-03-06T07:33:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.056+0000 7fbd5c88a640 1 Processor -- start 2024-03-06T07:33:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.056+0000 7fbd5c88a640 1 -- start start 2024-03-06T07:33:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd5c88a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd580c16e0 0x7fbd580c3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd5c88a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd580c70c0 0x7fbd581eb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd5c88a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd581529b0 0x7fbd581eb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd5c88a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd581561a0 con 0x7fbd580c16e0 2024-03-06T07:33:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd5c88a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd581596f0 con 0x7fbd581529b0 2024-03-06T07:33:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd5c88a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580c68c0 con 0x7fbd580c70c0 2024-03-06T07:33:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd5659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd580c16e0 0x7fbd580c3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd5659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd580c16e0 0x7fbd580c3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40878/0 (socket says 172.21.15.1:40878) 2024-03-06T07:33:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd5659c640 1 -- 172.21.15.1:0/2918566977 learned_addr learned my addr 172.21.15.1:0/2918566977 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd55d9b640 1 --2- 172.21.15.1:0/2918566977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd580c70c0 0x7fbd581eb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd56d9d640 1 --2- 172.21.15.1:0/2918566977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd581529b0 0x7fbd581eb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd5659c640 1 -- 172.21.15.1:0/2918566977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd580c70c0 msgr2=0x7fbd581eb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd5659c640 1 --2- 172.21.15.1:0/2918566977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd580c70c0 0x7fbd581eb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd5659c640 1 -- 172.21.15.1:0/2918566977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd581529b0 msgr2=0x7fbd581eb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd5659c640 1 --2- 172.21.15.1:0/2918566977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd581529b0 0x7fbd581eb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd5659c640 1 -- 172.21.15.1:0/2918566977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd581ebbc0 con 0x7fbd580c16e0 2024-03-06T07:33:33.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd5659c640 1 --2- 172.21.15.1:0/2918566977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd580c16e0 0x7fbd580c3cc0 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000d800 tx=0x7fbd5000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd46ffd640 1 -- 172.21.15.1:0/2918566977 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5000b6c0 con 0x7fbd580c16e0 2024-03-06T07:33:33.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd46ffd640 1 -- 172.21.15.1:0/2918566977 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd50019070 con 0x7fbd580c16e0 2024-03-06T07:33:33.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd46ffd640 1 -- 172.21.15.1:0/2918566977 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5000ba80 con 0x7fbd580c16e0 2024-03-06T07:33:33.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd5c88a640 1 -- 172.21.15.1:0/2918566977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd581eac40 con 0x7fbd580c16e0 2024-03-06T07:33:33.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.057+0000 7fbd5c88a640 1 -- 172.21.15.1:0/2918566977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd580ca2f0 con 0x7fbd580c16e0 2024-03-06T07:33:33.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.058+0000 7fbd5c88a640 1 -- 172.21.15.1:0/2918566977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbd580680b0 con 0x7fbd580c16e0 2024-03-06T07:33:33.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.059+0000 7fbd46ffd640 1 -- 172.21.15.1:0/2918566977 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd5000bc20 con 0x7fbd580c16e0 2024-03-06T07:33:33.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.059+0000 7fbd46ffd640 1 --2- 172.21.15.1:0/2918566977 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbd3806b4d0 0x7fbd3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.059+0000 7fbd46ffd640 1 -- 172.21.15.1:0/2918566977 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fbd5008a680 con 0x7fbd580c16e0 2024-03-06T07:33:33.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.059+0000 7fbd55d9b640 1 --2- 172.21.15.1:0/2918566977 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbd3806b4d0 0x7fbd3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.059+0000 7fbd46ffd640 1 --2- 172.21.15.1:0/2918566977 >> [v2:172.21.15.98:6824/3249148736,v1:172.21.15.98:6825/3249148736] conn(0x7fbd38071b50 0x7fbd38073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.059+0000 7fbd46ffd640 1 -- 172.21.15.1:0/2918566977 --> [v2:172.21.15.98:6824/3249148736,v1:172.21.15.98:6825/3249148736] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd50017e50 con 0x7fbd38071b50 2024-03-06T07:33:33.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.059+0000 7fbd46ffd640 1 -- 172.21.15.1:0/2918566977 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=217) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd50091050 con 0x7fbd580c16e0 2024-03-06T07:33:33.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.060+0000 7fbd56d9d640 1 --2- 172.21.15.1:0/2918566977 >> [v2:172.21.15.98:6824/3249148736,v1:172.21.15.98:6825/3249148736] conn(0x7fbd38071b50 0x7fbd38073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.060+0000 7fbd55d9b640 1 --2- 172.21.15.1:0/2918566977 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbd3806b4d0 0x7fbd3806d990 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fbd580ca9d0 tx=0x7fbd4c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.060+0000 7fbd56d9d640 1 --2- 172.21.15.1:0/2918566977 >> [v2:172.21.15.98:6824/3249148736,v1:172.21.15.98:6825/3249148736] conn(0x7fbd38071b50 0x7fbd38073f90 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.061+0000 7fbd46ffd640 1 -- 172.21.15.1:0/2918566977 <== osd.7 v2:172.21.15.98:6824/3249148736 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbd50017e50 con 0x7fbd38071b50 2024-03-06T07:33:33.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.074+0000 7fbd5c88a640 1 -- 172.21.15.1:0/2918566977 --> [v2:172.21.15.98:6824/3249148736,v1:172.21.15.98:6825/3249148736] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fbd580b0880 con 0x7fbd38071b50 2024-03-06T07:33:33.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.075+0000 7fbd46ffd640 1 -- 172.21.15.1:0/2918566977 <== osd.7 v2:172.21.15.98:6824/3249148736 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fbd580b0880 con 0x7fbd38071b50 2024-03-06T07:33:33.077 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-06T07:33:33.077 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:33:33.077 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-06T07:33:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.075+0000 7fbd5c88a640 1 -- 172.21.15.1:0/2918566977 >> [v2:172.21.15.98:6824/3249148736,v1:172.21.15.98:6825/3249148736] conn(0x7fbd38071b50 msgr2=0x7fbd38073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.075+0000 7fbd5c88a640 1 --2- 172.21.15.1:0/2918566977 >> [v2:172.21.15.98:6824/3249148736,v1:172.21.15.98:6825/3249148736] conn(0x7fbd38071b50 0x7fbd38073f90 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.075+0000 7fbd5c88a640 1 -- 172.21.15.1:0/2918566977 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbd3806b4d0 msgr2=0x7fbd3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.075+0000 7fbd5c88a640 1 --2- 172.21.15.1:0/2918566977 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fbd3806b4d0 0x7fbd3806d990 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fbd580ca9d0 tx=0x7fbd4c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.075+0000 7fbd5c88a640 1 -- 172.21.15.1:0/2918566977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd580c16e0 msgr2=0x7fbd580c3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.075+0000 7fbd5c88a640 1 --2- 172.21.15.1:0/2918566977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd580c16e0 0x7fbd580c3cc0 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000d800 tx=0x7fbd5000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.075+0000 7fbd56d9d640 1 -- 172.21.15.1:0/2918566977 reap_dead start 2024-03-06T07:33:33.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.075+0000 7fbd5c88a640 1 -- 172.21.15.1:0/2918566977 shutdown_connections 2024-03-06T07:33:33.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.075+0000 7fbd5c88a640 1 -- 172.21.15.1:0/2918566977 >> 172.21.15.1:0/2918566977 conn(0x7fbd58066d00 msgr2=0x7fbd580bffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:33.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.075+0000 7fbd5c88a640 1 -- 172.21.15.1:0/2918566977 shutdown_connections 2024-03-06T07:33:33.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.075+0000 7fbd5c88a640 1 -- 172.21.15.1:0/2918566977 wait complete. 2024-03-06T07:33:33.089 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-06T07:33:33.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.189+0000 7fda267d0640 1 Processor -- start 2024-03-06T07:33:33.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.189+0000 7fda267d0640 1 -- start start 2024-03-06T07:33:33.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.190+0000 7fda267d0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fda200c75a0 0x7fda20154300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.190+0000 7fda267d0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda200c8120 con 0x7fda20150640 2024-03-06T07:33:33.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.190+0000 7fda267d0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda200c7b60 con 0x7fda200a7f50 2024-03-06T07:33:33.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.190+0000 7fda267d0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda20153e30 con 0x7fda200c75a0 2024-03-06T07:33:33.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.190+0000 7fda24d6d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fda200c75a0 0x7fda20154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.190+0000 7fda24d6d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fda200c75a0 0x7fda20154300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40886/0 (socket says 172.21.15.1:40886) 2024-03-06T07:33:33.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.190+0000 7fda1ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fda20150640 0x7fda20153720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37176/0 (socket says 172.21.15.1:37176) 2024-03-06T07:33:33.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.190+0000 7fda24d6d640 1 -- 172.21.15.1:0/3939189620 learned_addr learned my addr 172.21.15.1:0/3939189620 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:33.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.190+0000 7fda24d6d640 1 -- 172.21.15.1:0/3939189620 >> v1:172.21.15.98:6789/0 conn(0x7fda200a7f50 legacy=0x7fda200c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.190+0000 7fda24d6d640 1 -- 172.21.15.1:0/3939189620 >> v1:172.21.15.1:6789/0 conn(0x7fda20150640 legacy=0x7fda20153720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.190+0000 7fda24d6d640 1 -- 172.21.15.1:0/3939189620 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda200c7e40 con 0x7fda200c75a0 2024-03-06T07:33:33.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.190+0000 7fda1effd640 1 -- 172.21.15.1:0/3939189620 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280115950 0 0) 0x7fda200c8120 con 0x7fda20150640 2024-03-06T07:33:33.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.191+0000 7fda24d6d640 1 --2- 172.21.15.1:0/3939189620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda200c75a0 0x7fda20154300 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7fda18009e50 tx=0x7fda1802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec15683eb28be1dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.191+0000 7fda1effd640 1 -- 172.21.15.1:0/3939189620 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1802c9f0 con 0x7fda200c75a0 2024-03-06T07:33:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.191+0000 7fda1effd640 1 -- 172.21.15.1:0/3939189620 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1802cb90 con 0x7fda200c75a0 2024-03-06T07:33:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.191+0000 7fda1effd640 1 -- 172.21.15.1:0/3939189620 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda180343f0 con 0x7fda200c75a0 2024-03-06T07:33:33.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.192+0000 7fda267d0640 1 -- 172.21.15.1:0/3939189620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda200c75a0 msgr2=0x7fda20154300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.192+0000 7fda267d0640 1 --2- 172.21.15.1:0/3939189620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda200c75a0 0x7fda20154300 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7fda18009e50 tx=0x7fda1802be50 comp rx=0 tx=0).stop 2024-03-06T07:33:33.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.192+0000 7fda267d0640 1 -- 172.21.15.1:0/3939189620 shutdown_connections 2024-03-06T07:33:33.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.192+0000 7fda267d0640 1 --2- 172.21.15.1:0/3939189620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda200c75a0 0x7fda20154300 unknown :-1 s=CLOSED pgs=2760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.192+0000 7fda267d0640 1 -- 172.21.15.1:0/3939189620 >> 172.21.15.1:0/3939189620 conn(0x7fda2006b3e0 msgr2=0x7fda2006b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:33.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.192+0000 7fda267d0640 1 -- 172.21.15.1:0/3939189620 shutdown_connections 2024-03-06T07:33:33.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.192+0000 7fda267d0640 1 -- 172.21.15.1:0/3939189620 wait complete. 2024-03-06T07:33:33.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.193+0000 7fda267d0640 1 Processor -- start 2024-03-06T07:33:33.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.193+0000 7fda267d0640 1 -- start start 2024-03-06T07:33:33.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda267d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda200a7f50 0x7fda200a9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda267d0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda200c75a0 0x7fda2013ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda267d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda20150640 0x7fda2013efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda267d0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda20150a20 con 0x7fda200c75a0 2024-03-06T07:33:33.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda267d0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda20153200 con 0x7fda20150640 2024-03-06T07:33:33.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda267d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda20071790 con 0x7fda200a7f50 2024-03-06T07:33:33.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda1f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda200c75a0 0x7fda2013ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda1f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda200c75a0 0x7fda2013ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40892/0 (socket says 172.21.15.1:40892) 2024-03-06T07:33:33.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda1f7fe640 1 -- 172.21.15.1:0/3689791919 learned_addr learned my addr 172.21.15.1:0/3689791919 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:33.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda24d6d640 1 --2- 172.21.15.1:0/3689791919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda20150640 0x7fda2013efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda1ffff640 1 --2- 172.21.15.1:0/3689791919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda200a7f50 0x7fda200a9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda1f7fe640 1 -- 172.21.15.1:0/3689791919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda200a7f50 msgr2=0x7fda200a9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda1f7fe640 1 --2- 172.21.15.1:0/3689791919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda200a7f50 0x7fda200a9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda1f7fe640 1 -- 172.21.15.1:0/3689791919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda20150640 msgr2=0x7fda2013efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda1f7fe640 1 --2- 172.21.15.1:0/3689791919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda20150640 0x7fda2013efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda1f7fe640 1 -- 172.21.15.1:0/3689791919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2014e420 con 0x7fda200c75a0 2024-03-06T07:33:33.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda1ffff640 1 --2- 172.21.15.1:0/3689791919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda200a7f50 0x7fda200a9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:33.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.194+0000 7fda1f7fe640 1 --2- 172.21.15.1:0/3689791919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda200c75a0 0x7fda2013ea70 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7fda1000b3e0 tx=0x7fda1000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.195+0000 7fda1cff9640 1 -- 172.21.15.1:0/3689791919 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda10015020 con 0x7fda200c75a0 2024-03-06T07:33:33.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.195+0000 7fda1cff9640 1 -- 172.21.15.1:0/3689791919 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda10011850 con 0x7fda200c75a0 2024-03-06T07:33:33.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.195+0000 7fda267d0640 1 -- 172.21.15.1:0/3689791919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda200ad2b0 con 0x7fda200c75a0 2024-03-06T07:33:33.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.195+0000 7fda1cff9640 1 -- 172.21.15.1:0/3689791919 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda10011c40 con 0x7fda200c75a0 2024-03-06T07:33:33.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.195+0000 7fda267d0640 1 -- 172.21.15.1:0/3689791919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda200ad870 con 0x7fda200c75a0 2024-03-06T07:33:33.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.196+0000 7fda267d0640 1 -- 172.21.15.1:0/3689791919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda20067fd0 con 0x7fda200c75a0 2024-03-06T07:33:33.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.196+0000 7fda1cff9640 1 -- 172.21.15.1:0/3689791919 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda10018020 con 0x7fda200c75a0 2024-03-06T07:33:33.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.196+0000 7fda1cff9640 1 --2- 172.21.15.1:0/3689791919 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda0c06b4d0 0x7fda0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.196+0000 7fda1cff9640 1 -- 172.21.15.1:0/3689791919 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fda1008a140 con 0x7fda200c75a0 2024-03-06T07:33:33.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.197+0000 7fda1ffff640 1 --2- 172.21.15.1:0/3689791919 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda0c06b4d0 0x7fda0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.197+0000 7fda1ffff640 1 --2- 172.21.15.1:0/3689791919 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda0c06b4d0 0x7fda0c06d990 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fda14000f60 tx=0x7fda14007c10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.201+0000 7fda1cff9640 1 -- 172.21.15.1:0/3689791919 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda1001f020 con 0x7fda200c75a0 2024-03-06T07:33:33.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.382+0000 7fda267d0640 1 -- 172.21.15.1:0/3689791919 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fda20149430 con 0x7fda0c06b4d0 2024-03-06T07:33:33.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.383+0000 7fda1cff9640 1 -- 172.21.15.1:0/3689791919 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fda14007270 con 0x7fda0c06b4d0 2024-03-06T07:33:33.385 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 7 to deep-scrub 2024-03-06T07:33:33.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.385+0000 7fda267d0640 1 -- 172.21.15.1:0/3689791919 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda0c06b4d0 msgr2=0x7fda0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.385+0000 7fda267d0640 1 --2- 172.21.15.1:0/3689791919 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda0c06b4d0 0x7fda0c06d990 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fda14000f60 tx=0x7fda14007c10 comp rx=0 tx=0).stop 2024-03-06T07:33:33.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.385+0000 7fda267d0640 1 -- 172.21.15.1:0/3689791919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda200c75a0 msgr2=0x7fda2013ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.385+0000 7fda267d0640 1 --2- 172.21.15.1:0/3689791919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda200c75a0 0x7fda2013ea70 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7fda1000b3e0 tx=0x7fda1000b8a0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.385+0000 7fda267d0640 1 -- 172.21.15.1:0/3689791919 shutdown_connections 2024-03-06T07:33:33.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.385+0000 7fda267d0640 1 --2- 172.21.15.1:0/3689791919 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7fda0c06b4d0 0x7fda0c06d990 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.385+0000 7fda267d0640 1 --2- 172.21.15.1:0/3689791919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda20150640 0x7fda2013efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.385+0000 7fda267d0640 1 --2- 172.21.15.1:0/3689791919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda200c75a0 0x7fda2013ea70 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.385+0000 7fda267d0640 1 --2- 172.21.15.1:0/3689791919 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fda200a7f50 0x7fda200a9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.385+0000 7fda267d0640 1 -- 172.21.15.1:0/3689791919 >> 172.21.15.1:0/3689791919 conn(0x7fda2006b3e0 msgr2=0x7fda20156c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:33.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.385+0000 7fda267d0640 1 -- 172.21.15.1:0/3689791919 shutdown_connections 2024-03-06T07:33:33.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.385+0000 7fda267d0640 1 -- 172.21.15.1:0/3689791919 wait complete. 2024-03-06T07:33:33.398 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:33:33.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.497+0000 7f8cd422a640 1 Processor -- start 2024-03-06T07:33:33.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.497+0000 7f8cd422a640 1 -- start start 2024-03-06T07:33:33.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.498+0000 7f8cd422a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ccc0c05f0 0x7f8ccc0c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.498+0000 7f8cd422a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8ccc0c14c0 0x7f8ccc1558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.498+0000 7f8cd422a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ccc0c31f0 con 0x7f8ccc06f640 2024-03-06T07:33:33.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.498+0000 7f8cd422a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ccc06fe90 con 0x7f8ccc0c14c0 2024-03-06T07:33:33.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.498+0000 7f8cd422a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ccc0c2fc0 con 0x7f8ccc0c05f0 2024-03-06T07:33:33.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.498+0000 7f8cd27c7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8ccc0c14c0 0x7f8ccc1558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.498+0000 7f8cd27c7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8ccc0c14c0 0x7f8ccc1558c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.1:53630/0 (socket says 172.21.15.1:53630) 2024-03-06T07:33:33.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.498+0000 7f8cd27c7640 1 -- 172.21.15.1:0/715635330 learned_addr learned my addr 172.21.15.1:0/715635330 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:33.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.498+0000 7f8cd17c5640 1 --1- 172.21.15.1:0/715635330 >> v1:172.21.15.98:6789/0 conn(0x7f8ccc06f640 0x7f8ccc1551b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.1:39964/0 (socket says 172.21.15.1:39964) 2024-03-06T07:33:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.498+0000 7f8cd1fc6640 1 --2- 172.21.15.1:0/715635330 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ccc0c05f0 0x7f8ccc0c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.498+0000 7f8cd27c7640 1 -- 172.21.15.1:0/715635330 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ccc0c05f0 msgr2=0x7f8ccc0c09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.498+0000 7f8cd27c7640 1 --2- 172.21.15.1:0/715635330 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ccc0c05f0 0x7f8ccc0c09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.498+0000 7f8cd27c7640 1 -- 172.21.15.1:0/715635330 >> v1:172.21.15.98:6789/0 conn(0x7f8ccc06f640 legacy=0x7f8ccc1551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.498+0000 7f8cd27c7640 1 -- 172.21.15.1:0/715635330 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ccc155e00 con 0x7f8ccc0c14c0 2024-03-06T07:33:33.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.499+0000 7f8cd27c7640 1 --2- 172.21.15.1:0/715635330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ccc0c14c0 0x7f8ccc1558c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc009e50 tx=0x7f8cbc02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6257e4a36791701 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.499+0000 7f8cd0fc4640 1 -- 172.21.15.1:0/715635330 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc02cb00 con 0x7f8ccc0c14c0 2024-03-06T07:33:33.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.499+0000 7f8cd0fc4640 1 -- 172.21.15.1:0/715635330 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cbc02cca0 con 0x7f8ccc0c14c0 2024-03-06T07:33:33.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.499+0000 7f8cd0fc4640 1 -- 172.21.15.1:0/715635330 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc034610 con 0x7f8ccc0c14c0 2024-03-06T07:33:33.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.500+0000 7f8cd422a640 1 -- 172.21.15.1:0/715635330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ccc0c14c0 msgr2=0x7f8ccc1558c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.500+0000 7f8cd422a640 1 --2- 172.21.15.1:0/715635330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ccc0c14c0 0x7f8ccc1558c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc009e50 tx=0x7f8cbc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:33:33.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.500+0000 7f8cd422a640 1 -- 172.21.15.1:0/715635330 shutdown_connections 2024-03-06T07:33:33.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.500+0000 7f8cd422a640 1 --2- 172.21.15.1:0/715635330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ccc0c14c0 0x7f8ccc1558c0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.500+0000 7f8cd422a640 1 --2- 172.21.15.1:0/715635330 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ccc0c05f0 0x7f8ccc0c09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.500+0000 7f8cd422a640 1 -- 172.21.15.1:0/715635330 >> 172.21.15.1:0/715635330 conn(0x7f8ccc0c3cc0 msgr2=0x7f8ccc0c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:33.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.500+0000 7f8cd422a640 1 -- 172.21.15.1:0/715635330 shutdown_connections 2024-03-06T07:33:33.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.500+0000 7f8cd422a640 1 -- 172.21.15.1:0/715635330 wait complete. 2024-03-06T07:33:33.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.501+0000 7f8cd422a640 1 Processor -- start 2024-03-06T07:33:33.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.501+0000 7f8cd422a640 1 -- start start 2024-03-06T07:33:33.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.502+0000 7f8cd422a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ccc06f640 0x7f8ccc0c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.502+0000 7f8cd422a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ccc0c05f0 0x7f8ccc0c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.502+0000 7f8cd422a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ccc0c14c0 0x7f8ccc13f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.502+0000 7f8cd422a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ccc0c3770 con 0x7f8ccc0c14c0 2024-03-06T07:33:33.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.502+0000 7f8cd422a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ccc0c3490 con 0x7f8ccc0c05f0 2024-03-06T07:33:33.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.502+0000 7f8cd422a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ccc0c1040 con 0x7f8ccc06f640 2024-03-06T07:33:33.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.502+0000 7f8cd27c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ccc0c14c0 0x7f8ccc13f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.502+0000 7f8cd27c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ccc0c14c0 0x7f8ccc13f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40898/0 (socket says 172.21.15.1:40898) 2024-03-06T07:33:33.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.502+0000 7f8cd27c7640 1 -- 172.21.15.1:0/1165537542 learned_addr learned my addr 172.21.15.1:0/1165537542 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:33.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.502+0000 7f8cd1fc6640 1 --2- 172.21.15.1:0/1165537542 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ccc06f640 0x7f8ccc0c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.502+0000 7f8cd17c5640 1 --2- 172.21.15.1:0/1165537542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ccc0c05f0 0x7f8ccc0c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.502+0000 7f8cd27c7640 1 -- 172.21.15.1:0/1165537542 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ccc06f640 msgr2=0x7f8ccc0c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.502+0000 7f8cd27c7640 1 --2- 172.21.15.1:0/1165537542 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ccc06f640 0x7f8ccc0c5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.502+0000 7f8cd27c7640 1 -- 172.21.15.1:0/1165537542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ccc0c05f0 msgr2=0x7f8ccc0c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.502+0000 7f8cd27c7640 1 --2- 172.21.15.1:0/1165537542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ccc0c05f0 0x7f8ccc0c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.502+0000 7f8cd27c7640 1 -- 172.21.15.1:0/1165537542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ccc0abb60 con 0x7f8ccc0c14c0 2024-03-06T07:33:33.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.502+0000 7f8cd27c7640 1 --2- 172.21.15.1:0/1165537542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ccc0c14c0 0x7f8ccc13f910 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc036040 tx=0x7f8cbc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.503+0000 7f8cca7fc640 1 -- 172.21.15.1:0/1165537542 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc043020 con 0x7f8ccc0c14c0 2024-03-06T07:33:33.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.503+0000 7f8cca7fc640 1 -- 172.21.15.1:0/1165537542 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cbc02ce50 con 0x7f8ccc0c14c0 2024-03-06T07:33:33.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.503+0000 7f8cd422a640 1 -- 172.21.15.1:0/1165537542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ccc0ab690 con 0x7f8ccc0c14c0 2024-03-06T07:33:33.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.503+0000 7f8cca7fc640 1 -- 172.21.15.1:0/1165537542 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc009510 con 0x7f8ccc0c14c0 2024-03-06T07:33:33.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.503+0000 7f8cd422a640 1 -- 172.21.15.1:0/1165537542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ccc1529a0 con 0x7f8ccc0c14c0 2024-03-06T07:33:33.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.504+0000 7f8cd422a640 1 -- 172.21.15.1:0/1165537542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ccc1eb9a0 con 0x7f8ccc0c14c0 2024-03-06T07:33:33.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.504+0000 7f8cca7fc640 1 -- 172.21.15.1:0/1165537542 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8cbc046020 con 0x7f8ccc0c14c0 2024-03-06T07:33:33.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.505+0000 7f8cca7fc640 1 --2- 172.21.15.1:0/1165537542 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8cb806b4d0 0x7f8cb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.505+0000 7f8cca7fc640 1 -- 172.21.15.1:0/1165537542 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f8cbc0ac530 con 0x7f8ccc0c14c0 2024-03-06T07:33:33.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.505+0000 7f8cd1fc6640 1 --2- 172.21.15.1:0/1165537542 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8cb806b4d0 0x7f8cb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.505+0000 7f8cd1fc6640 1 --2- 172.21.15.1:0/1165537542 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8cb806b4d0 0x7f8cb806d990 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4009b40 tx=0x7f8cc4009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.509+0000 7f8cca7fc640 1 -- 172.21.15.1:0/1165537542 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8cbc04d030 con 0x7f8ccc0c14c0 2024-03-06T07:33:33.563 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:33.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:33.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.688+0000 7f8cd422a640 1 -- 172.21.15.1:0/1165537542 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8ccc1eb7b0 con 0x7f8cb806b4d0 2024-03-06T07:33:33.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.690+0000 7f8cca7fc640 1 -- 172.21.15.1:0/1165537542 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13527 (secure 0 0 0) 0x7f8ccc1eb7b0 con 0x7f8cb806b4d0 2024-03-06T07:33:33.692 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:33.692 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:33:33.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.692+0000 7f8cd422a640 1 -- 172.21.15.1:0/1165537542 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8cb806b4d0 msgr2=0x7f8cb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.692+0000 7f8cd422a640 1 --2- 172.21.15.1:0/1165537542 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8cb806b4d0 0x7f8cb806d990 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4009b40 tx=0x7f8cc4009190 comp rx=0 tx=0).stop 2024-03-06T07:33:33.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.692+0000 7f8cd422a640 1 -- 172.21.15.1:0/1165537542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ccc0c14c0 msgr2=0x7f8ccc13f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.692+0000 7f8cd422a640 1 --2- 172.21.15.1:0/1165537542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ccc0c14c0 0x7f8ccc13f910 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc036040 tx=0x7f8cbc004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:33.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.692+0000 7f8cd422a640 1 -- 172.21.15.1:0/1165537542 shutdown_connections 2024-03-06T07:33:33.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.692+0000 7f8cd422a640 1 --2- 172.21.15.1:0/1165537542 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f8cb806b4d0 0x7f8cb806d990 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.692+0000 7f8cd422a640 1 --2- 172.21.15.1:0/1165537542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ccc0c14c0 0x7f8ccc13f910 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.692+0000 7f8cd422a640 1 --2- 172.21.15.1:0/1165537542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ccc0c05f0 0x7f8ccc0c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.692+0000 7f8cd422a640 1 --2- 172.21.15.1:0/1165537542 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ccc06f640 0x7f8ccc0c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.692+0000 7f8cd422a640 1 -- 172.21.15.1:0/1165537542 >> 172.21.15.1:0/1165537542 conn(0x7f8ccc0c3cc0 msgr2=0x7f8ccc158cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:33.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.692+0000 7f8cd422a640 1 -- 172.21.15.1:0/1165537542 shutdown_connections 2024-03-06T07:33:33.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:33.692+0000 7f8cd422a640 1 -- 172.21.15.1:0/1165537542 wait complete. 2024-03-06T07:33:33.704 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":609,"stamp":"2024-03-06T07:33:32.886109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":29,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":287812,"kb_used_data":21696,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754686908,"statfs":{"total":773094113280,"available":772799393792,"internally_reserved":0,"allocated":22216704,"data_stored":11915528,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.124441"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":551,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-06T07:33:26.091167+0000","last_change":"2024-03-06T07:20:16.768456+0000","last_active":"2024-03-06T07:33:26.091167+0000","last_peered":"2024-03-06T07:33:26.091167+0000","last_clean":"2024-03-06T07:33:26.091167+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:33:26.091167+0000","last_undegraded":"2024-03-06T07:33:26.091167+0000","last_fullsized":"2024-03-06T07:33:26.091167+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:15.708025+0000","last_clean_scrub_stamp":"2024-03-06T07:20:15.708025+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:21.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33944,"kb_used_data":2760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337896,"statfs":{"total":96636764160,"available":96602005504,"internally_reserved":0,"allocated":2826240,"data_stored":1932217,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33880,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337960,"statfs":{"total":96636764160,"available":96602071040,"internally_reserved":0,"allocated":2760704,"data_stored":1341849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33880,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337960,"statfs":{"total":96636764160,"available":96602071040,"internally_reserved":0,"allocated":2760704,"data_stored":1341849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33880,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337960,"statfs":{"total":96636764160,"available":96602071040,"internally_reserved":0,"allocated":2760704,"data_stored":1341849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38040,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333800,"statfs":{"total":96636764160,"available":96597811200,"internally_reserved":0,"allocated":2760704,"data_stored":1341849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38040,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333800,"statfs":{"total":96636764160,"available":96597811200,"internally_reserved":0,"allocated":2760704,"data_stored":1341849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38040,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333800,"statfs":{"total":96636764160,"available":96597811200,"internally_reserved":0,"allocated":2760704,"data_stored":1341849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575010,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38108,"kb_used_data":2760,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333732,"statfs":{"total":96636764160,"available":96597741568,"internally_reserved":0,"allocated":2826240,"data_stored":1932217,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:33:33.707 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-06T07:20:15.708025+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=20, tm_sec=15, tm_wday=2, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=33, tm_sec=28, tm_wday=2, tm_yday=66, tm_isdst=0) 2024-03-06T07:33:33.707 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-06T07:33:53.708 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:33:53.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.820+0000 7f3c21a9a640 1 Processor -- start 2024-03-06T07:33:53.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.820+0000 7f3c21a9a640 1 -- start start 2024-03-06T07:33:53.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.821+0000 7f3c21a9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1c0c1260 0x7f3c1c0c16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.821+0000 7f3c21a9a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c1530a0 con 0x7f3c1c1565a0 2024-03-06T07:33:53.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.821+0000 7f3c21a9a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c153380 con 0x7f3c1c149a90 2024-03-06T07:33:53.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.821+0000 7f3c21a9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c153660 con 0x7f3c1c0c1260 2024-03-06T07:33:53.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.821+0000 7f3c1bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3c1c1565a0 0x7f3c1c156a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46334/0 (socket says 172.21.15.1:46334) 2024-03-06T07:33:53.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.821+0000 7f3c1bfff640 1 -- 172.21.15.1:0/913311224 learned_addr learned my addr 172.21.15.1:0/913311224 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:53.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.821+0000 7f3c1affd640 1 --2- 172.21.15.1:0/913311224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1c0c1260 0x7f3c1c0c16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.821+0000 7f3c1a7fc640 1 -- 172.21.15.1:0/913311224 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675272566 0 0) 0x7f3c1c1530a0 con 0x7f3c1c1565a0 2024-03-06T07:33:53.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.821+0000 7f3c1a7fc640 1 -- 172.21.15.1:0/913311224 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c04003520 con 0x7f3c1c1565a0 2024-03-06T07:33:53.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1a7fc640 1 -- 172.21.15.1:0/913311224 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 477539677 0 0) 0x7f3c04003520 con 0x7f3c1c1565a0 2024-03-06T07:33:53.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1a7fc640 1 -- 172.21.15.1:0/913311224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1c0c1260 msgr2=0x7f3c1c0c16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1a7fc640 1 --2- 172.21.15.1:0/913311224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1c0c1260 0x7f3c1c0c16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1a7fc640 1 -- 172.21.15.1:0/913311224 >> v1:172.21.15.98:6789/0 conn(0x7f3c1c149a90 legacy=0x7f3c1c0c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1a7fc640 1 -- 172.21.15.1:0/913311224 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c153940 con 0x7f3c1c1565a0 2024-03-06T07:33:53.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1a7fc640 1 -- 172.21.15.1:0/913311224 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f3c140019f0 con 0x7f3c1c1565a0 2024-03-06T07:33:53.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1a7fc640 1 -- 172.21.15.1:0/913311224 >> v1:172.21.15.1:6789/0 conn(0x7f3c1c1565a0 legacy=0x7f3c1c156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1a7fc640 1 --2- 172.21.15.1:0/913311224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c040039e0 0x7f3c04003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1a7fc640 1 --2- 172.21.15.1:0/913311224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1c0c1260 0x7f3c04004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1a7fc640 1 --2- 172.21.15.1:0/913311224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c04004d40 0x7f3c040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1a7fc640 1 -- 172.21.15.1:0/913311224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c153660 con 0x7f3c040039e0 2024-03-06T07:33:53.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1a7fc640 1 -- 172.21.15.1:0/913311224 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0400baa0 con 0x7f3c04004d40 2024-03-06T07:33:53.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1a7fc640 1 -- 172.21.15.1:0/913311224 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0400bca0 con 0x7f3c1c0c1260 2024-03-06T07:33:53.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1b7fe640 1 --2- 172.21.15.1:0/913311224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c040039e0 0x7f3c04003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1affd640 1 --2- 172.21.15.1:0/913311224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1c0c1260 0x7f3c04004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1bfff640 1 --2- 172.21.15.1:0/913311224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c04004d40 0x7f3c040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1b7fe640 1 -- 172.21.15.1:0/913311224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1c0c1260 msgr2=0x7f3c04004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1b7fe640 1 --2- 172.21.15.1:0/913311224 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1c0c1260 0x7f3c04004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1b7fe640 1 -- 172.21.15.1:0/913311224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c04004d40 msgr2=0x7f3c040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1b7fe640 1 --2- 172.21.15.1:0/913311224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c04004d40 0x7f3c040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.822+0000 7f3c1b7fe640 1 -- 172.21.15.1:0/913311224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c04003520 con 0x7f3c040039e0 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.823+0000 7f3c1b7fe640 1 --2- 172.21.15.1:0/913311224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c040039e0 0x7f3c04003dc0 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c001870 tx=0x7f3c04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a200149ae0988 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.823+0000 7f3c1a7fc640 1 -- 172.21.15.1:0/913311224 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c037070 con 0x7f3c040039e0 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.823+0000 7f3c1a7fc640 1 -- 172.21.15.1:0/913311224 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0c0052d0 con 0x7f3c040039e0 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.823+0000 7f3c1a7fc640 1 -- 172.21.15.1:0/913311224 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c001440 con 0x7f3c040039e0 2024-03-06T07:33:53.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.824+0000 7f3c21a9a640 1 -- 172.21.15.1:0/913311224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c040039e0 msgr2=0x7f3c04003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.824+0000 7f3c21a9a640 1 --2- 172.21.15.1:0/913311224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c040039e0 0x7f3c04003dc0 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c001870 tx=0x7f3c04005830 comp rx=0 tx=0).stop 2024-03-06T07:33:53.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.824+0000 7f3c1bfff640 1 -- 172.21.15.1:0/913311224 reap_dead start 2024-03-06T07:33:53.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.824+0000 7f3c21a9a640 1 -- 172.21.15.1:0/913311224 shutdown_connections 2024-03-06T07:33:53.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.824+0000 7f3c21a9a640 1 -- 172.21.15.1:0/913311224 >> 172.21.15.1:0/913311224 conn(0x7f3c1c0c6860 msgr2=0x7f3c1c0c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.824+0000 7f3c21a9a640 1 -- 172.21.15.1:0/913311224 shutdown_connections 2024-03-06T07:33:53.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.824+0000 7f3c21a9a640 1 -- 172.21.15.1:0/913311224 wait complete. 2024-03-06T07:33:53.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.825+0000 7f3c21a9a640 1 Processor -- start 2024-03-06T07:33:53.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.825+0000 7f3c21a9a640 1 -- start start 2024-03-06T07:33:53.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.826+0000 7f3c21a9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c040039e0 0x7f3c1c152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.826+0000 7f3c21a9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c1c0c0590 0x7f3c1c145ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.826+0000 7f3c21a9a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1c0aad10 0x7f3c1c146400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.826+0000 7f3c21a9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c157140 con 0x7f3c040039e0 2024-03-06T07:33:53.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.826+0000 7f3c21a9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c06f4a0 con 0x7f3c1c0c0590 2024-03-06T07:33:53.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.826+0000 7f3c21a9a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c0ab220 con 0x7f3c1c0aad10 2024-03-06T07:33:53.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.826+0000 7f3c1b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c040039e0 0x7f3c1c152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.826+0000 7f3c1b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c040039e0 0x7f3c1c152470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45016/0 (socket says 172.21.15.1:45016) 2024-03-06T07:33:53.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.826+0000 7f3c1b7fe640 1 -- 172.21.15.1:0/2964245750 learned_addr learned my addr 172.21.15.1:0/2964245750 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:53.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.826+0000 7f3c1bfff640 1 --2- 172.21.15.1:0/2964245750 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1c0aad10 0x7f3c1c146400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.826+0000 7f3c1affd640 1 --2- 172.21.15.1:0/2964245750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c1c0c0590 0x7f3c1c145ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.826+0000 7f3c1b7fe640 1 -- 172.21.15.1:0/2964245750 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1c0aad10 msgr2=0x7f3c1c146400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.826+0000 7f3c1b7fe640 1 --2- 172.21.15.1:0/2964245750 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1c0aad10 0x7f3c1c146400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.826+0000 7f3c1b7fe640 1 -- 172.21.15.1:0/2964245750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c1c0c0590 msgr2=0x7f3c1c145ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.826+0000 7f3c1b7fe640 1 --2- 172.21.15.1:0/2964245750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c1c0c0590 0x7f3c1c145ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.826+0000 7f3c1b7fe640 1 -- 172.21.15.1:0/2964245750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c0bfc90 con 0x7f3c040039e0 2024-03-06T07:33:53.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.826+0000 7f3c1b7fe640 1 --2- 172.21.15.1:0/2964245750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c040039e0 0x7f3c1c152470 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c035040 tx=0x7f3c0c001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.827+0000 7f3c03fff640 1 -- 172.21.15.1:0/2964245750 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c037070 con 0x7f3c040039e0 2024-03-06T07:33:53.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.827+0000 7f3c03fff640 1 -- 172.21.15.1:0/2964245750 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0c038040 con 0x7f3c040039e0 2024-03-06T07:33:53.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.827+0000 7f3c21a9a640 1 -- 172.21.15.1:0/2964245750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c1c0bff70 con 0x7f3c040039e0 2024-03-06T07:33:53.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.827+0000 7f3c03fff640 1 -- 172.21.15.1:0/2964245750 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c03e8d0 con 0x7f3c040039e0 2024-03-06T07:33:53.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.827+0000 7f3c21a9a640 1 -- 172.21.15.1:0/2964245750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1c073350 con 0x7f3c040039e0 2024-03-06T07:33:53.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.828+0000 7f3c21a9a640 1 -- 172.21.15.1:0/2964245750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1c067fd0 con 0x7f3c040039e0 2024-03-06T07:33:53.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.828+0000 7f3c03fff640 1 -- 172.21.15.1:0/2964245750 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c0c031070 con 0x7f3c040039e0 2024-03-06T07:33:53.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.829+0000 7f3c03fff640 1 --2- 172.21.15.1:0/2964245750 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c0806b500 0x7f3c0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.829+0000 7f3c03fff640 1 -- 172.21.15.1:0/2964245750 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f3c0c0abb40 con 0x7f3c040039e0 2024-03-06T07:33:53.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.829+0000 7f3c1affd640 1 --2- 172.21.15.1:0/2964245750 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c0806b500 0x7f3c0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.829+0000 7f3c1affd640 1 --2- 172.21.15.1:0/2964245750 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c0806b500 0x7f3c0806d9c0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c0c5c80 tx=0x7f3c10009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:53.833+0000 7f3c03fff640 1 -- 172.21.15.1:0/2964245750 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c0c07a3f0 con 0x7f3c040039e0 2024-03-06T07:33:54.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.012+0000 7f3c21a9a640 1 -- 172.21.15.1:0/2964245750 --> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3c1c0c0140 con 0x7f3c0806b500 2024-03-06T07:33:54.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.013+0000 7f3c03fff640 1 -- 172.21.15.1:0/2964245750 <== mgr.4108 v2:172.21.15.98:6832/3776520052 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13463 (secure 0 0 0) 0x7f3c1c0c0140 con 0x7f3c0806b500 2024-03-06T07:33:54.016 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:54.016 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-06T07:33:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.015+0000 7f3c21a9a640 1 -- 172.21.15.1:0/2964245750 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c0806b500 msgr2=0x7f3c0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.015+0000 7f3c21a9a640 1 --2- 172.21.15.1:0/2964245750 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c0806b500 0x7f3c0806d9c0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c0c5c80 tx=0x7f3c10009190 comp rx=0 tx=0).stop 2024-03-06T07:33:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.015+0000 7f3c21a9a640 1 -- 172.21.15.1:0/2964245750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c040039e0 msgr2=0x7f3c1c152470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.015+0000 7f3c21a9a640 1 --2- 172.21.15.1:0/2964245750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c040039e0 0x7f3c1c152470 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c035040 tx=0x7f3c0c001230 comp rx=0 tx=0).stop 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.016+0000 7f3c21a9a640 1 -- 172.21.15.1:0/2964245750 shutdown_connections 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.016+0000 7f3c21a9a640 1 --2- 172.21.15.1:0/2964245750 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f3c0806b500 0x7f3c0806d9c0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.016+0000 7f3c21a9a640 1 --2- 172.21.15.1:0/2964245750 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3c1c0aad10 0x7f3c1c146400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.016+0000 7f3c21a9a640 1 --2- 172.21.15.1:0/2964245750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c1c0c0590 0x7f3c1c145ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.016+0000 7f3c21a9a640 1 --2- 172.21.15.1:0/2964245750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c040039e0 0x7f3c1c152470 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.016+0000 7f3c21a9a640 1 -- 172.21.15.1:0/2964245750 >> 172.21.15.1:0/2964245750 conn(0x7f3c1c0c6860 msgr2=0x7f3c1c155a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.016+0000 7f3c21a9a640 1 -- 172.21.15.1:0/2964245750 shutdown_connections 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.016+0000 7f3c21a9a640 1 -- 172.21.15.1:0/2964245750 wait complete. 2024-03-06T07:33:54.028 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":619,"stamp":"2024-03-06T07:33:52.890160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":287792,"kb_used_data":21696,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754686928,"statfs":{"total":773094113280,"available":772799414272,"internally_reserved":0,"allocated":22216704,"data_stored":11915528,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002431"},"pg_stats":[{"pgid":"1.0","version":"217'119","reported_seq":562,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-06T07:33:34.359208+0000","last_change":"2024-03-06T07:33:34.359269+0000","last_active":"2024-03-06T07:33:34.359208+0000","last_peered":"2024-03-06T07:33:34.359208+0000","last_clean":"2024-03-06T07:33:34.359208+0000","last_became_active":"2024-03-06T07:20:16.767383+0000","last_became_peered":"2024-03-06T07:20:16.767383+0000","last_unstale":"2024-03-06T07:33:34.359208+0000","last_undegraded":"2024-03-06T07:33:34.359208+0000","last_fullsized":"2024-03-06T07:33:34.359208+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'119","last_scrub_stamp":"2024-03-06T07:33:34.359096+0000","last_deep_scrub":"217'119","last_deep_scrub_stamp":"2024-03-06T07:33:34.359096+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.359096+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:27.599275+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65090,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575014,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33936,"kb_used_data":2760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337904,"statfs":{"total":96636764160,"available":96602013696,"internally_reserved":0,"allocated":2826240,"data_stored":1932217,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575013,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33880,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337960,"statfs":{"total":96636764160,"available":96602071040,"internally_reserved":0,"allocated":2760704,"data_stored":1341849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575013,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33880,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337960,"statfs":{"total":96636764160,"available":96602071040,"internally_reserved":0,"allocated":2760704,"data_stored":1341849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33880,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337960,"statfs":{"total":96636764160,"available":96602071040,"internally_reserved":0,"allocated":2760704,"data_stored":1341849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575013,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38040,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333800,"statfs":{"total":96636764160,"available":96597811200,"internally_reserved":0,"allocated":2760704,"data_stored":1341849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575013,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38040,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333800,"statfs":{"total":96636764160,"available":96597811200,"internally_reserved":0,"allocated":2760704,"data_stored":1341849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38040,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333800,"statfs":{"total":96636764160,"available":96597811200,"internally_reserved":0,"allocated":2760704,"data_stored":1341849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575014,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38096,"kb_used_data":2760,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333744,"statfs":{"total":96636764160,"available":96597753856,"internally_reserved":0,"allocated":2826240,"data_stored":1932217,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32545,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:33:54.029 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-06T07:33:54.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.128+0000 7f92e2684640 1 Processor -- start 2024-03-06T07:33:54.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.129+0000 7f92e2684640 1 -- start start 2024-03-06T07:33:54.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.129+0000 7f92e2684640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92dc152920 0x7f92dc06aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.129+0000 7f92e2684640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92dc155dc0 con 0x7f92dc06b530 2024-03-06T07:33:54.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.129+0000 7f92e2684640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92dc158910 con 0x7f92dc155220 2024-03-06T07:33:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.129+0000 7f92e2684640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc152d00 con 0x7f92dc152920 2024-03-06T07:33:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.129+0000 7f92db7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f92dc06b530 0x7f92dc0c5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46342/0 (socket says 172.21.15.1:46342) 2024-03-06T07:33:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.129+0000 7f92db7fe640 1 -- 172.21.15.1:0/2497749810 learned_addr learned my addr 172.21.15.1:0/2497749810 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.130+0000 7f92dbfff640 1 --2- 172.21.15.1:0/2497749810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92dc152920 0x7f92dc06aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.130+0000 7f92daffd640 1 -- 172.21.15.1:0/2497749810 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922626213 0 0) 0x7f92dc155dc0 con 0x7f92dc06b530 2024-03-06T07:33:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.130+0000 7f92daffd640 1 -- 172.21.15.1:0/2497749810 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c4003520 con 0x7f92dc06b530 2024-03-06T07:33:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.130+0000 7f92dbfff640 1 -- 172.21.15.1:0/2497749810 >> v1:172.21.15.98:6789/0 conn(0x7f92dc155220 legacy=0x7f92dc1556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.130+0000 7f92daffd640 1 -- 172.21.15.1:0/2497749810 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1856050222 0 0) 0x7f92c4003520 con 0x7f92dc06b530 2024-03-06T07:33:54.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.130+0000 7f92dbfff640 1 -- 172.21.15.1:0/2497749810 >> v1:172.21.15.1:6789/0 conn(0x7f92dc06b530 legacy=0x7f92dc0c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.130+0000 7f92dbfff640 1 -- 172.21.15.1:0/2497749810 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92dc158bf0 con 0x7f92dc152920 2024-03-06T07:33:54.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.131+0000 7f92dbfff640 1 --2- 172.21.15.1:0/2497749810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92dc152920 0x7f92dc06aff0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f92cc0097d0 tx=0x7f92cc02b5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9fc44e69a318fdd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.131+0000 7f92daffd640 1 -- 172.21.15.1:0/2497749810 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc037070 con 0x7f92dc152920 2024-03-06T07:33:54.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.131+0000 7f92daffd640 1 -- 172.21.15.1:0/2497749810 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92cc030510 con 0x7f92dc152920 2024-03-06T07:33:54.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.131+0000 7f92daffd640 1 -- 172.21.15.1:0/2497749810 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0001670 con 0x7f92dc152920 2024-03-06T07:33:54.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.131+0000 7f92e2684640 1 -- 172.21.15.1:0/2497749810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92dc152920 msgr2=0x7f92dc06aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.131+0000 7f92e2684640 1 --2- 172.21.15.1:0/2497749810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92dc152920 0x7f92dc06aff0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f92cc0097d0 tx=0x7f92cc02b5d0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.131+0000 7f92e2684640 1 -- 172.21.15.1:0/2497749810 shutdown_connections 2024-03-06T07:33:54.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.131+0000 7f92e2684640 1 --2- 172.21.15.1:0/2497749810 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92dc152920 0x7f92dc06aff0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.131+0000 7f92e2684640 1 -- 172.21.15.1:0/2497749810 >> 172.21.15.1:0/2497749810 conn(0x7f92dc0a9700 msgr2=0x7f92dc0a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.131+0000 7f92e2684640 1 -- 172.21.15.1:0/2497749810 shutdown_connections 2024-03-06T07:33:54.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.131+0000 7f92e2684640 1 -- 172.21.15.1:0/2497749810 wait complete. 2024-03-06T07:33:54.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.132+0000 7f92e2684640 1 Processor -- start 2024-03-06T07:33:54.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.133+0000 7f92e2684640 1 -- start start 2024-03-06T07:33:54.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.133+0000 7f92e2684640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92dc06b530 0x7f92dc1eef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.133+0000 7f92e2684640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92dc152920 0x7f92dc0a8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.133+0000 7f92e2684640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92dc155220 0x7f92dc148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.133+0000 7f92e2684640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc159240 con 0x7f92dc152920 2024-03-06T07:33:54.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.133+0000 7f92e2684640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc1596d0 con 0x7f92dc155220 2024-03-06T07:33:54.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.133+0000 7f92e2684640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0c6380 con 0x7f92dc06b530 2024-03-06T07:33:54.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.133+0000 7f92db7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92dc152920 0x7f92dc0a8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.133+0000 7f92db7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92dc152920 0x7f92dc0a8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45032/0 (socket says 172.21.15.1:45032) 2024-03-06T07:33:54.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.133+0000 7f92db7fe640 1 -- 172.21.15.1:0/2050709639 learned_addr learned my addr 172.21.15.1:0/2050709639 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:54.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.134+0000 7f92dbfff640 1 --2- 172.21.15.1:0/2050709639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92dc06b530 0x7f92dc1eef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.134+0000 7f92e0c21640 1 --2- 172.21.15.1:0/2050709639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92dc155220 0x7f92dc148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.134+0000 7f92db7fe640 1 -- 172.21.15.1:0/2050709639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92dc06b530 msgr2=0x7f92dc1eef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.134+0000 7f92db7fe640 1 --2- 172.21.15.1:0/2050709639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92dc06b530 0x7f92dc1eef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.134+0000 7f92db7fe640 1 -- 172.21.15.1:0/2050709639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92dc155220 msgr2=0x7f92dc148d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.134+0000 7f92db7fe640 1 --2- 172.21.15.1:0/2050709639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92dc155220 0x7f92dc148d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.134+0000 7f92db7fe640 1 -- 172.21.15.1:0/2050709639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92dc157100 con 0x7f92dc152920 2024-03-06T07:33:54.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.134+0000 7f92db7fe640 1 --2- 172.21.15.1:0/2050709639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92dc152920 0x7f92dc0a8960 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7f92d000c750 tx=0x7f92d000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.134+0000 7f92d8ff9640 1 -- 172.21.15.1:0/2050709639 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0017070 con 0x7f92dc152920 2024-03-06T07:33:54.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.134+0000 7f92d8ff9640 1 -- 172.21.15.1:0/2050709639 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d0005ad0 con 0x7f92dc152920 2024-03-06T07:33:54.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.134+0000 7f92e2684640 1 -- 172.21.15.1:0/2050709639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92dc1573e0 con 0x7f92dc152920 2024-03-06T07:33:54.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.134+0000 7f92d8ff9640 1 -- 172.21.15.1:0/2050709639 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0007070 con 0x7f92dc152920 2024-03-06T07:33:54.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.134+0000 7f92e2684640 1 -- 172.21.15.1:0/2050709639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92dc070550 con 0x7f92dc152920 2024-03-06T07:33:54.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.135+0000 7f92e2684640 1 -- 172.21.15.1:0/2050709639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92dc067fd0 con 0x7f92dc152920 2024-03-06T07:33:54.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.135+0000 7f92d8ff9640 1 -- 172.21.15.1:0/2050709639 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92d001a070 con 0x7f92dc152920 2024-03-06T07:33:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.136+0000 7f92d8ff9640 1 --2- 172.21.15.1:0/2050709639 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f92c806b500 0x7f92c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.136+0000 7f92d8ff9640 1 -- 172.21.15.1:0/2050709639 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f92d0089570 con 0x7f92dc152920 2024-03-06T07:33:54.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.136+0000 7f92dbfff640 1 --2- 172.21.15.1:0/2050709639 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f92c806b500 0x7f92c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.137+0000 7f92dbfff640 1 --2- 172.21.15.1:0/2050709639 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f92c806b500 0x7f92c806d9c0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f92cc034000 tx=0x7f92cc008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.140+0000 7f92d8ff9640 1 -- 172.21.15.1:0/2050709639 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92d0057e00 con 0x7f92dc152920 2024-03-06T07:33:54.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.320+0000 7f92e2684640 1 -- 172.21.15.1:0/2050709639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f92dc06b1d0 con 0x7f92dc152920 2024-03-06T07:33:54.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.323+0000 7f92d8ff9640 1 -- 172.21.15.1:0/2050709639 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v65)=0 v65) v1 ==== 123+0+0 (secure 0 0 0) 0x7f92d005bea0 con 0x7f92dc152920 2024-03-06T07:33:54.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.323+0000 7f92d8ff9640 1 -- 172.21.15.1:0/2050709639 <== mon.0 v2:172.21.15.1:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f92d0060460 con 0x7f92dc152920 2024-03-06T07:33:54.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.325+0000 7f92e2684640 1 -- 172.21.15.1:0/2050709639 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f92c806b500 msgr2=0x7f92c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.325+0000 7f92e2684640 1 --2- 172.21.15.1:0/2050709639 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f92c806b500 0x7f92c806d9c0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f92cc034000 tx=0x7f92cc008030 comp rx=0 tx=0).stop 2024-03-06T07:33:54.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.325+0000 7f92e2684640 1 -- 172.21.15.1:0/2050709639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92dc152920 msgr2=0x7f92dc0a8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.325+0000 7f92e2684640 1 --2- 172.21.15.1:0/2050709639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92dc152920 0x7f92dc0a8960 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7f92d000c750 tx=0x7f92d000cc10 comp rx=0 tx=0).stop 2024-03-06T07:33:54.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.325+0000 7f92e2684640 1 -- 172.21.15.1:0/2050709639 shutdown_connections 2024-03-06T07:33:54.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.325+0000 7f92e2684640 1 --2- 172.21.15.1:0/2050709639 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f92c806b500 0x7f92c806d9c0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.325+0000 7f92e2684640 1 --2- 172.21.15.1:0/2050709639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92dc155220 0x7f92dc148d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.325+0000 7f92e2684640 1 --2- 172.21.15.1:0/2050709639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92dc152920 0x7f92dc0a8960 unknown :-1 s=CLOSED pgs=2767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.325+0000 7f92e2684640 1 --2- 172.21.15.1:0/2050709639 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92dc06b530 0x7f92dc1eef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.325+0000 7f92e2684640 1 -- 172.21.15.1:0/2050709639 >> 172.21.15.1:0/2050709639 conn(0x7f92dc0a9700 msgr2=0x7f92dc0c0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.325+0000 7f92e2684640 1 -- 172.21.15.1:0/2050709639 shutdown_connections 2024-03-06T07:33:54.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.325+0000 7f92e2684640 1 -- 172.21.15.1:0/2050709639 wait complete. 2024-03-06T07:33:54.338 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:33:54.338 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:54.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.437+0000 7f4aa893e640 1 Processor -- start 2024-03-06T07:33:54.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.437+0000 7f4aa893e640 1 -- start start 2024-03-06T07:33:54.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.438+0000 7f4aa893e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4aa40bf9d0 0x7f4aa4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.438+0000 7f4aa893e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa40c2fc0 con 0x7f4aa40c6b10 2024-03-06T07:33:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.438+0000 7f4aa893e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa4158da0 con 0x7f4aa4152650 2024-03-06T07:33:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.438+0000 7f4aa893e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa4155800 con 0x7f4aa40bf9d0 2024-03-06T07:33:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.438+0000 7f4aa1d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4aa40c6b10 0x7f4aa40c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47462/0 (socket says 172.21.15.1:47462) 2024-03-06T07:33:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.438+0000 7f4aa1d9b640 1 -- 172.21.15.1:0/1382890117 learned_addr learned my addr 172.21.15.1:0/1382890117 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-06T07:33:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.438+0000 7f4aa2d9d640 1 --2- 172.21.15.1:0/1382890117 >> v2:172.21.15.98:3300/0 conn(0x7f4aa40bf9d0 0x7f4aa4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.438+0000 7f4aa159a640 1 -- 172.21.15.1:0/1382890117 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216041305 0 0) 0x7f4aa40c2fc0 con 0x7f4aa40c6b10 2024-03-06T07:33:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa159a640 1 -- 172.21.15.1:0/1382890117 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a8c003520 con 0x7f4aa40c6b10 2024-03-06T07:33:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa159a640 1 -- 172.21.15.1:0/1382890117 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3645764551 0 0) 0x7f4a8c003520 con 0x7f4aa40c6b10 2024-03-06T07:33:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa159a640 1 -- 172.21.15.1:0/1382890117 >> v2:172.21.15.98:3300/0 conn(0x7f4aa40bf9d0 msgr2=0x7f4aa4155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa159a640 1 --2- 172.21.15.1:0/1382890117 >> v2:172.21.15.98:3300/0 conn(0x7f4aa40bf9d0 0x7f4aa4155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa159a640 1 -- 172.21.15.1:0/1382890117 >> v1:172.21.15.98:6789/0 conn(0x7f4aa4152650 legacy=0x7f4aa4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa159a640 1 -- 172.21.15.1:0/1382890117 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa40c6690 con 0x7f4aa40c6b10 2024-03-06T07:33:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa159a640 1 -- 172.21.15.1:0/1382890117 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1735308699 0 0) 0x7f4a98001670 con 0x7f4aa40c6b10 2024-03-06T07:33:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa159a640 1 -- 172.21.15.1:0/1382890117 >> v1:172.21.15.1:6789/0 conn(0x7f4aa40c6b10 legacy=0x7f4aa40c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa159a640 1 --2- 172.21.15.1:0/1382890117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a8c0039b0 0x7f4a8c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa159a640 1 --2- 172.21.15.1:0/1382890117 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a8c004800 0x7f4a8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa159a640 1 --2- 172.21.15.1:0/1382890117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c0051c0 0x7f4a8c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa159a640 1 -- 172.21.15.1:0/1382890117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa4155800 con 0x7f4a8c0039b0 2024-03-06T07:33:54.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa159a640 1 -- 172.21.15.1:0/1382890117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c00c160 con 0x7f4a8c0051c0 2024-03-06T07:33:54.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa159a640 1 -- 172.21.15.1:0/1382890117 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c00c360 con 0x7f4a8c004800 2024-03-06T07:33:54.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa259c640 1 --2- 172.21.15.1:0/1382890117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a8c0039b0 0x7f4a8c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa2d9d640 1 --2- 172.21.15.1:0/1382890117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c0051c0 0x7f4a8c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa1d9b640 1 --2- 172.21.15.1:0/1382890117 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a8c004800 0x7f4a8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa259c640 1 -- 172.21.15.1:0/1382890117 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a8c004800 msgr2=0x7f4a8c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:54.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa259c640 1 --2- 172.21.15.1:0/1382890117 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a8c004800 0x7f4a8c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa259c640 1 -- 172.21.15.1:0/1382890117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c0051c0 msgr2=0x7f4a8c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:54.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa259c640 1 --2- 172.21.15.1:0/1382890117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c0051c0 0x7f4a8c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa259c640 1 -- 172.21.15.1:0/1382890117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8c003520 con 0x7f4a8c0039b0 2024-03-06T07:33:54.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.439+0000 7f4aa2d9d640 1 -- 172.21.15.1:0/1382890117 reap_dead start 2024-03-06T07:33:54.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.440+0000 7f4aa259c640 1 --2- 172.21.15.1:0/1382890117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a8c0039b0 0x7f4a8c003dd0 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c001750 tx=0x7f4a8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc564a6839081c26 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.440+0000 7f4aa159a640 1 -- 172.21.15.1:0/1382890117 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c001370 con 0x7f4a8c0039b0 2024-03-06T07:33:54.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.440+0000 7f4aa159a640 1 -- 172.21.15.1:0/1382890117 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f4a9c0052d0 con 0x7f4a8c0039b0 2024-03-06T07:33:54.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.440+0000 7f4aa159a640 1 -- 172.21.15.1:0/1382890117 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c003850 con 0x7f4a8c0039b0 2024-03-06T07:33:54.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.441+0000 7f4aa893e640 1 -- 172.21.15.1:0/1382890117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a8c0039b0 msgr2=0x7f4a8c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.441+0000 7f4aa893e640 1 --2- 172.21.15.1:0/1382890117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a8c0039b0 0x7f4a8c003dd0 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c001750 tx=0x7f4a8c005830 comp rx=0 tx=0).stop 2024-03-06T07:33:54.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.441+0000 7f4aa893e640 1 -- 172.21.15.1:0/1382890117 shutdown_connections 2024-03-06T07:33:54.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.441+0000 7f4aa893e640 1 --2- 172.21.15.1:0/1382890117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a8c0039b0 0x7f4a8c003dd0 unknown :-1 s=CLOSED pgs=2769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.441+0000 7f4aa893e640 1 -- 172.21.15.1:0/1382890117 >> 172.21.15.1:0/1382890117 conn(0x7f4aa4066ee0 msgr2=0x7f4aa40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.441+0000 7f4aa893e640 1 -- 172.21.15.1:0/1382890117 shutdown_connections 2024-03-06T07:33:54.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.441+0000 7f4aa893e640 1 -- 172.21.15.1:0/1382890117 wait complete. 2024-03-06T07:33:54.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.442+0000 7f4aa893e640 1 Processor -- start 2024-03-06T07:33:54.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.442+0000 7f4aa893e640 1 -- start start 2024-03-06T07:33:54.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa893e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a8c0039b0 0x7f4aa41eade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa893e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4aa4158350 0x7f4aa40c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa893e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4aa41499b0 0x7f4aa41eeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa893e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa41551b0 con 0x7f4aa41499b0 2024-03-06T07:33:54.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa893e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa40c6fa0 con 0x7f4aa4158350 2024-03-06T07:33:54.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa893e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa4155490 con 0x7f4a8c0039b0 2024-03-06T07:33:54.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4aa41499b0 0x7f4aa41eeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4aa41499b0 0x7f4aa41eeb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39634/0 (socket says 172.21.15.1:39634) 2024-03-06T07:33:54.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa2d9d640 1 -- 172.21.15.1:0/3467853847 learned_addr learned my addr 172.21.15.1:0/3467853847 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-06T07:33:54.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa259c640 1 --2- 172.21.15.1:0/3467853847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a8c0039b0 0x7f4aa41eade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa1d9b640 1 --2- 172.21.15.1:0/3467853847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4aa4158350 0x7f4aa40c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa2d9d640 1 -- 172.21.15.1:0/3467853847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a8c0039b0 msgr2=0x7f4aa41eade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa2d9d640 1 --2- 172.21.15.1:0/3467853847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a8c0039b0 0x7f4aa41eade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa2d9d640 1 -- 172.21.15.1:0/3467853847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4aa4158350 msgr2=0x7f4aa40c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa2d9d640 1 --2- 172.21.15.1:0/3467853847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4aa4158350 0x7f4aa40c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa2d9d640 1 -- 172.21.15.1:0/3467853847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa41ee3d0 con 0x7f4aa41499b0 2024-03-06T07:33:54.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa2d9d640 1 --2- 172.21.15.1:0/3467853847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4aa41499b0 0x7f4aa41eeb60 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40c1390 tx=0x7f4a940115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4a92ffd640 1 -- 172.21.15.1:0/3467853847 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a94011de0 con 0x7f4aa41499b0 2024-03-06T07:33:54.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4a92ffd640 1 -- 172.21.15.1:0/3467853847 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f4a9401b070 con 0x7f4aa41499b0 2024-03-06T07:33:54.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4aa893e640 1 -- 172.21.15.1:0/3467853847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa4154260 con 0x7f4aa41499b0 2024-03-06T07:33:54.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.443+0000 7f4a92ffd640 1 -- 172.21.15.1:0/3467853847 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a94003990 con 0x7f4aa41499b0 2024-03-06T07:33:54.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.444+0000 7f4aa893e640 1 -- 172.21.15.1:0/3467853847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa40c5ef0 con 0x7f4aa41499b0 2024-03-06T07:33:54.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.444+0000 7f4aa893e640 1 -- 172.21.15.1:0/3467853847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa4067fd0 con 0x7f4aa41499b0 2024-03-06T07:33:54.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.445+0000 7f4a92ffd640 1 -- 172.21.15.1:0/3467853847 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a94003b30 con 0x7f4aa41499b0 2024-03-06T07:33:54.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.445+0000 7f4a92ffd640 1 --2- 172.21.15.1:0/3467853847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a8406b4d0 0x7f4a8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.445+0000 7f4a92ffd640 1 -- 172.21.15.1:0/3467853847 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f4a9408a8b0 con 0x7f4aa41499b0 2024-03-06T07:33:54.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.446+0000 7f4aa259c640 1 --2- 172.21.15.1:0/3467853847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a8406b4d0 0x7f4a8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.446+0000 7f4aa259c640 1 --2- 172.21.15.1:0/3467853847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a8406b4d0 0x7f4a8406d990 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c033000 tx=0x7f4a9c009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.450+0000 7f4a92ffd640 1 -- 172.21.15.1:0/3467853847 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a94003e20 con 0x7f4aa41499b0 2024-03-06T07:33:54.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.668+0000 7f4aa893e640 1 -- 172.21.15.1:0/3467853847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4aa40b06f0 con 0x7f4aa41499b0 2024-03-06T07:33:54.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.668+0000 7f4a92ffd640 1 -- 172.21.15.1:0/3467853847 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) v1 ==== 95+0+1326 (secure 0 0 0) 0x7f4a940590c0 con 0x7f4aa41499b0 2024-03-06T07:33:54.671 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-06T07:33:54.671 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":215,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11949,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2665749628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2665749628},{"type":"v1","addr":"172.21.15.1:6835","nonce":2665749628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":215}],"filesystems":[]} 2024-03-06T07:33:54.671 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 215 2024-03-06T07:33:54.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.670+0000 7f4aa893e640 1 -- 172.21.15.1:0/3467853847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a8406b4d0 msgr2=0x7f4a8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.670+0000 7f4aa893e640 1 --2- 172.21.15.1:0/3467853847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a8406b4d0 0x7f4a8406d990 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c033000 tx=0x7f4a9c009030 comp rx=0 tx=0).stop 2024-03-06T07:33:54.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.670+0000 7f4aa893e640 1 -- 172.21.15.1:0/3467853847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4aa41499b0 msgr2=0x7f4aa41eeb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.670+0000 7f4aa893e640 1 --2- 172.21.15.1:0/3467853847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4aa41499b0 0x7f4aa41eeb60 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40c1390 tx=0x7f4a940115a0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.670+0000 7f4aa893e640 1 -- 172.21.15.1:0/3467853847 shutdown_connections 2024-03-06T07:33:54.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.670+0000 7f4aa893e640 1 --2- 172.21.15.1:0/3467853847 >> [v2:172.21.15.98:6832/3776520052,v1:172.21.15.98:6833/3776520052] conn(0x7f4a8406b4d0 0x7f4a8406d990 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.670+0000 7f4aa893e640 1 --2- 172.21.15.1:0/3467853847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4aa41499b0 0x7f4aa41eeb60 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.670+0000 7f4aa893e640 1 --2- 172.21.15.1:0/3467853847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4aa4158350 0x7f4aa40c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.670+0000 7f4aa893e640 1 --2- 172.21.15.1:0/3467853847 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4a8c0039b0 0x7f4aa41eade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.670+0000 7f4aa893e640 1 -- 172.21.15.1:0/3467853847 >> 172.21.15.1:0/3467853847 conn(0x7f4aa4066ee0 msgr2=0x7f4aa40c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.670+0000 7f4aa893e640 1 -- 172.21.15.1:0/3467853847 shutdown_connections 2024-03-06T07:33:54.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-06T07:33:54.670+0000 7f4aa893e640 1 -- 172.21.15.1:0/3467853847 wait complete. 2024-03-06T07:33:54.683 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-06T07:33:54.683 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-06T07:33:54.684 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:33:54.684 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:54.693 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:33:54.693 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:33:54.693 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:54.742 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:33:54.743 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:33:54.743 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:54.754 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:33:54.755 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:33:54.755 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:54.807 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:33:54.807 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-06T07:33:54.807 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-06T07:33:54.807 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:54.832 INFO:tasks.ceph.osd.0:Stopped 2024-03-06T07:33:54.832 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-06T07:33:54.832 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:54.898 INFO:tasks.ceph.osd.1:Stopped 2024-03-06T07:33:54.898 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-06T07:33:54.898 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:54.966 INFO:tasks.ceph.osd.2:Stopped 2024-03-06T07:33:54.966 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-06T07:33:54.966 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:55.036 INFO:tasks.ceph.osd.3:Stopped 2024-03-06T07:33:55.037 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-06T07:33:55.037 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:55.072 INFO:tasks.ceph.osd.4:Stopped 2024-03-06T07:33:55.072 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-06T07:33:55.073 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:55.146 INFO:tasks.ceph.osd.5:Stopped 2024-03-06T07:33:55.146 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-06T07:33:55.147 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:55.214 INFO:tasks.ceph.osd.6:Stopped 2024-03-06T07:33:55.214 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-06T07:33:55.214 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:55.283 INFO:tasks.ceph.osd.7:Stopped 2024-03-06T07:33:55.283 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-06T07:33:55.284 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-06T07:33:55.284 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:55.307 INFO:tasks.ceph.mgr.y:Stopped 2024-03-06T07:33:55.307 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-06T07:33:55.308 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:55.353 INFO:tasks.ceph.mgr.x:Stopped 2024-03-06T07:33:55.353 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-06T07:33:55.354 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-06T07:33:55.354 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:55.365 INFO:tasks.ceph.mon.a:Stopped 2024-03-06T07:33:55.366 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-06T07:33:55.366 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:55.406 INFO:tasks.ceph.mon.b:Stopped 2024-03-06T07:33:55.406 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-06T07:33:55.407 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:55.460 INFO:tasks.ceph.mon.c:Stopped 2024-03-06T07:33:55.460 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-06T07:33:55.460 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'Replacing daemon mds' | egrep -v '\(MDS_INSUFFICIENT_STANDBY\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(FS_DEGRADED\)' | head -n 1 2024-03-06T07:33:55.492 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-06T07:33:55.492 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-06T07:33:55.690 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-06T07:33:55.690 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-06T07:33:55.813 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-06T07:33:55.813 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-06T07:33:55.864 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-06T07:33:55.864 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-06T07:33:55.922 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-06T07:33:55.922 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-06T07:33:56.263 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-06T07:33:56.264 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-06T07:33:56.334 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-06T07:33:56.335 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-06T07:33:56.400 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-06T07:33:56.400 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-06T07:33:56.474 INFO:tasks.ceph:Archiving mon data... 2024-03-06T07:33:56.475 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583045/data/mon.a.tgz 2024-03-06T07:33:56.475 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-03-06T07:33:56.490 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.muZ0Xb69YY 2024-03-06T07:33:56.491 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.muZ0Xb69YY 2024-03-06T07:33:56.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.muZ0Xb69YY 2024-03-06T07:33:56.766 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.muZ0Xb69YY is 552KB 2024-03-06T07:33:56.835 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.muZ0Xb69YY 2024-03-06T07:33:56.849 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583045/data/mon.b.tgz 2024-03-06T07:33:56.849 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-03-06T07:33:56.866 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.boAyIO9r7g 2024-03-06T07:33:56.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.boAyIO9r7g 2024-03-06T07:33:57.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.boAyIO9r7g 2024-03-06T07:33:57.138 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.boAyIO9r7g is 567KB 2024-03-06T07:33:57.236 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.boAyIO9r7g 2024-03-06T07:33:57.250 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583045/data/mon.c.tgz 2024-03-06T07:33:57.251 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-03-06T07:33:57.305 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.7CAr6NqSh9 2024-03-06T07:33:57.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.7CAr6NqSh9 2024-03-06T07:33:57.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.7CAr6NqSh9 2024-03-06T07:33:57.561 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.7CAr6NqSh9 is 567KB 2024-03-06T07:33:57.658 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.7CAr6NqSh9 2024-03-06T07:33:57.672 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-06T07:33:57.672 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:33:57.674 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:33:57.758 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:33:57.793 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:33:57.793 INFO:tasks.ceph:Archiving crash dumps... 2024-03-06T07:33:57.795 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583045/remote/smithi001/crash 2024-03-06T07:33:57.796 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:33:57.825 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583045/remote/smithi098/crash 2024-03-06T07:33:57.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:33:57.859 INFO:tasks.ceph:Shutting down logrotate 2024-03-06T07:33:57.859 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:57.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:57.945 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:57.972 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:57.998 INFO:tasks.ceph:Compressing logs... 2024-03-06T07:33:57.998 DEBUG:teuthology.orchestra.run.smithi001:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:33:58.014 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:33:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-06T07:33:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-06T07:33:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-06T07:33:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38173.log 2024-03-06T07:33:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-06T07:33:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38053.log 2024-03-06T07:33:58.089 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38173.log.gz 2024-03-06T07:33:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38093.log 2024-03-06T07:33:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-06T07:33:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38053.log: /var/log/ceph/ceph-client.admin.38093.log: gzip -5 --verbose -- 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38053.log.gz 2024-03-06T07:33:58.089 INFO:teuthology.orchestra.run.smithi098.stderr: /var/log/ceph/ceph-osd.5.log 2024-03-06T07:33:58.089 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38093.log.gz 2024-03-06T07:33:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42397.log 2024-03-06T07:33:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38013.log 2024-03-06T07:33:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.1.42397.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37973.log 2024-03-06T07:33:58.090 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.1.42397.log.gz 2024-03-06T07:33:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38013.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38013.log.gz 2024-03-06T07:33:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-06T07:33:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38133.log 2024-03-06T07:33:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37973.log: /var/log/ceph/ceph-mgr.x.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37973.log.gz 2024-03-06T07:33:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38253.log 2024-03-06T07:33:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38133.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38133.log.gz 2024-03-06T07:33:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:33:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38253.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38253.log.gz 2024-03-06T07:33:58.091 INFO:teuthology.orchestra.run.smithi098.stderr: 93.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38213.log 2024-03-06T07:33:58.091 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:33:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-06T07:33:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38213.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38213.log.gz 2024-03-06T07:33:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-06T07:33:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.b.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:33:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-06T07:33:58.140 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-06T07:33:58.147 INFO:teuthology.orchestra.run.smithi098.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-06T07:33:58.281 INFO:teuthology.orchestra.run.smithi098.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-06T07:33:58.419 INFO:teuthology.orchestra.run.smithi098.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-06T07:33:58.422 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50154.log 2024-03-06T07:33:58.422 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71057.log 2024-03-06T07:33:58.423 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.50154.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.71006.log 2024-03-06T07:33:58.423 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78030.log 2024-03-06T07:33:58.423 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50154.log.gz 2024-03-06T07:33:58.423 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64159.log 2024-03-06T07:33:58.423 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71057.log.gz 2024-03-06T07:33:58.423 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71006.log.gz 2024-03-06T07:33:58.423 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46100.log 2024-03-06T07:33:58.423 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64159.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78030.log.gz 2024-03-06T07:33:58.424 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71271.log 2024-03-06T07:33:58.424 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46100.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64159.log.gz 2024-03-06T07:33:58.424 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79042.log 2024-03-06T07:33:58.424 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6%/var/log/ceph/ceph-client.admin.71271.log: -- replaced with /var/log/ceph/ceph-client.admin.46100.log.gz 2024-03-06T07:33:58.424 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60851.log 2024-03-06T07:33:58.424 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79042.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71271.log.gz 2024-03-06T07:33:58.424 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76502.log 2024-03-06T07:33:58.425 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60851.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79042.log.gz 2024-03-06T07:33:58.425 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84674.log 2024-03-06T07:33:58.425 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76502.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60851.log.gz 2024-03-06T07:33:58.425 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81425.log 2024-03-06T07:33:58.425 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84674.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76502.log.gz 2024-03-06T07:33:58.425 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81780.log 2024-03-06T07:33:58.425 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81425.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84674.log.gz 2024-03-06T07:33:58.425 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44896.log 2024-03-06T07:33:58.425 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89193.log 2024-03-06T07:33:58.426 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1%/var/log/ceph/ceph-client.admin.44896.log: -- replaced with /var/log/ceph/ceph-client.admin.81780.log.gz 2024-03-06T07:33:58.426 INFO:teuthology.orchestra.run.smithi001.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.81425.log.gz 2024-03-06T07:33:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55436.log 2024-03-06T07:33:58.426 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9%/var/log/ceph/ceph-client.admin.89193.log: -- replaced with /var/log/ceph/ceph-client.admin.44896.log.gz 2024-03-06T07:33:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87855.log 2024-03-06T07:33:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55436.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.89193.log.gz 2024-03-06T07:33:58.426 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.80183.log 2024-03-06T07:33:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87855.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55436.log.gz 2024-03-06T07:33:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67868.log 2024-03-06T07:33:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80183.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87855.log.gz 2024-03-06T07:33:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88797.log 2024-03-06T07:33:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67868.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.80183.log.gz 2024-03-06T07:33:58.427 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.65822.log 2024-03-06T07:33:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88797.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67868.log.gz 2024-03-06T07:33:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67994.log 2024-03-06T07:33:58.427 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88797.log.gz 2024-03-06T07:33:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.65822.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.88293.log 2024-03-06T07:33:58.428 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51509.log 2024-03-06T07:33:58.428 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65822.log.gz 2024-03-06T07:33:58.428 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88293.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67994.log.gz 2024-03-06T07:33:58.428 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68660.log 2024-03-06T07:33:58.428 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51509.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.88293.log.gz 2024-03-06T07:33:58.428 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75097.log 2024-03-06T07:33:58.428 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51509.log.gz 2024-03-06T07:33:58.428 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70215.log 2024-03-06T07:33:58.429 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81131.log 2024-03-06T07:33:58.429 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68660.log.gz 2024-03-06T07:33:58.429 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70215.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75097.log.gz -5 2024-03-06T07:33:58.429 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-06T07:33:58.429 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81131.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70215.log.gz 2024-03-06T07:33:58.429 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81131.log.gz 2024-03-06T07:33:58.429 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88755.log 2024-03-06T07:33:58.429 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57966.log 2024-03-06T07:33:58.430 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45213.log 2024-03-06T07:33:58.430 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88755.log: /var/log/ceph/ceph-client.admin.57966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72796.log 2024-03-06T07:33:58.430 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88755.log.gz 2024-03-06T07:33:58.430 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45213.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57966.log.gz 2024-03-06T07:33:58.430 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73881.log 2024-03-06T07:33:58.430 INFO:teuthology.orchestra.run.smithi001.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.45213.log.gz 2024-03-06T07:33:58.430 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69734.log 2024-03-06T07:33:58.430 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80917.log 2024-03-06T07:33:58.430 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3%/var/log/ceph/ceph-client.admin.69734.log: -- replaced with /var/log/ceph/ceph-client.admin.73881.log.gz 2024-03-06T07:33:58.431 INFO:teuthology.orchestra.run.smithi001.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.72796.log.gz 2024-03-06T07:33:58.431 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58477.log 2024-03-06T07:33:58.431 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69734.log.gz 2024-03-06T07:33:58.431 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52854.log 2024-03-06T07:33:58.431 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80917.log: /var/log/ceph/ceph-client.admin.58477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48948.log 2024-03-06T07:33:58.431 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80917.log.gzgzip 2024-03-06T07:33:58.431 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose --/var/log/ceph/ceph-client.admin.52854.log: /var/log/ceph/ceph-client.admin.69378.log 2024-03-06T07:33:58.431 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58477.log.gz 2024-03-06T07:33:58.431 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55226.log 2024-03-06T07:33:58.431 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52854.log.gz 2024-03-06T07:33:58.432 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.69378.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48948.log.gz 2024-03-06T07:33:58.432 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47751.log 2024-03-06T07:33:58.432 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71980.log 2024-03-06T07:33:58.432 INFO:teuthology.orchestra.run.smithi001.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.69378.log.gz 2024-03-06T07:33:58.432 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47751.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55226.log.gz 2024-03-06T07:33:58.432 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85220.log 2024-03-06T07:33:58.432 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47751.log.gz 2024-03-06T07:33:58.432 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64495.log 2024-03-06T07:33:58.433 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85220.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71980.log.gz 2024-03-06T07:33:58.433 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80676.log 2024-03-06T07:33:58.433 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71673.log 2024-03-06T07:33:58.433 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.85220.log.gz 2024-03-06T07:33:58.433 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74488.log 2024-03-06T07:33:58.433 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64495.log.gz 2024-03-06T07:33:58.433 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71673.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.80676.log.gz 2024-03-06T07:33:58.433 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85574.log 2024-03-06T07:33:58.433 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74488.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71673.log.gz 2024-03-06T07:33:58.433 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68702.log 2024-03-06T07:33:58.434 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74488.log.gz 2024-03-06T07:33:58.434 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54437.log 2024-03-06T07:33:58.434 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42428.log 2024-03-06T07:33:58.434 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.85574.log.gz 2024-03-06T07:33:58.434 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54437.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68702.log.gz 2024-03-06T07:33:58.434 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49591.log 2024-03-06T07:33:58.434 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54437.log.gz 2024-03-06T07:33:58.434 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78664.log 2024-03-06T07:33:58.434 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61842.log 2024-03-06T07:33:58.434 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42428.log.gz 2024-03-06T07:33:58.435 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78664.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49591.log.gz 2024-03-06T07:33:58.435 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45848.log 2024-03-06T07:33:58.435 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78664.log.gz 2024-03-06T07:33:58.435 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84193.log 2024-03-06T07:33:58.435 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66805.log 2024-03-06T07:33:58.435 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61842.log.gz 2024-03-06T07:33:58.435 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84193.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45848.log.gz 2024-03-06T07:33:58.435 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88023.log 2024-03-06T07:33:58.435 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84193.log.gz 2024-03-06T07:33:58.436 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52417.log 2024-03-06T07:33:58.436 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88023.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66805.log.gz -5 2024-03-06T07:33:58.436 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.56740.log 2024-03-06T07:33:58.436 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52417.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88023.log.gz 2024-03-06T07:33:58.436 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51341.log 2024-03-06T07:33:58.436 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52417.log.gz 2024-03-06T07:33:58.436 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57500.log 2024-03-06T07:33:58.436 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87687.log 2024-03-06T07:33:58.437 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56740.log.gz 2024-03-06T07:33:58.437 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57500.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51341.log.gz 2024-03-06T07:33:58.437 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47259.log 2024-03-06T07:33:58.437 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57500.log.gz 2024-03-06T07:33:58.437 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83024.log 2024-03-06T07:33:58.437 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47259.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.87687.log.gz 2024-03-06T07:33:58.437 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68936.log 2024-03-06T07:33:58.437 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47259.log.gz 2024-03-06T07:33:58.437 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88713.log 2024-03-06T07:33:58.437 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9%/var/log/ceph/ceph-client.admin.68936.log: -- replaced with /var/log/ceph/ceph-client.admin.83024.log.gz 2024-03-06T07:33:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54894.log 2024-03-06T07:33:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88713.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68936.log.gz 2024-03-06T07:33:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47427.log 2024-03-06T07:33:58.438 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88713.log.gz 2024-03-06T07:33:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50562.log 2024-03-06T07:33:58.438 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54894.log.gz 2024-03-06T07:33:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67015.log 2024-03-06T07:33:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50562.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47427.log.gz 2024-03-06T07:33:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69650.log 2024-03-06T07:33:58.438 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90979.log 2024-03-06T07:33:58.439 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67015.log.gz 2024-03-06T07:33:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.69650.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.50562.log.gz 2024-03-06T07:33:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57586.log 2024-03-06T07:33:58.439 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69650.log.gz 2024-03-06T07:33:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55520.log 2024-03-06T07:33:58.439 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90979.log.gz 2024-03-06T07:33:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67584.log 2024-03-06T07:33:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55520.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57586.log.gz 2024-03-06T07:33:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87939.log 2024-03-06T07:33:58.439 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67584.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.55520.log.gz 2024-03-06T07:33:58.440 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49464.log 2024-03-06T07:33:58.440 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67584.log.gz 2024-03-06T07:33:58.440 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89735.log 2024-03-06T07:33:58.440 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87939.log.gz 2024-03-06T07:33:58.440 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50712.log 2024-03-06T07:33:58.440 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49464.log: /var/log/ceph/ceph-client.admin.89735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56828.log 2024-03-06T07:33:58.440 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49464.log.gz 2024-03-06T07:33:58.440 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.89735.log.gz 2024-03-06T07:33:58.440 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90006.log 2024-03-06T07:33:58.441 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56828.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50712.log.gz 2024-03-06T07:33:58.441 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54936.log 2024-03-06T07:33:58.441 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57756.log 2024-03-06T07:33:58.441 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56828.log.gz 2024-03-06T07:33:58.441 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54936.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.90006.log.gz 2024-03-06T07:33:58.441 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64453.log 2024-03-06T07:33:58.441 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54936.log.gz 2024-03-06T07:33:58.441 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82489.log 2024-03-06T07:33:58.441 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62607.log 2024-03-06T07:33:58.441 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57756.log.gz 2024-03-06T07:33:58.442 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82489.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64453.log.gz 2024-03-06T07:33:58.442 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83797.log 2024-03-06T07:33:58.442 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82489.log.gz 2024-03-06T07:33:58.442 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.62607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56024.log 2024-03-06T07:33:58.442 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60683.log 2024-03-06T07:33:58.442 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62607.log.gz 2024-03-06T07:33:58.442 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56024.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.83797.log.gz 2024-03-06T07:33:58.442 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51864.log 2024-03-06T07:33:58.442 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56024.log.gz 2024-03-06T07:33:58.443 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60557.log 2024-03-06T07:33:58.443 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51864.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60683.log.gz 2024-03-06T07:33:58.443 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67664.log 2024-03-06T07:33:58.443 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60557.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51864.log.gz 2024-03-06T07:33:58.443 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73026.log 2024-03-06T07:33:58.443 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67664.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60557.log.gz 2024-03-06T07:33:58.443 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84319.log 2024-03-06T07:33:58.443 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73026.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.67664.log.gz 2024-03-06T07:33:58.444 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60217.log 2024-03-06T07:33:58.444 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73026.log.gz 2024-03-06T07:33:58.444 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75881.log 2024-03-06T07:33:58.444 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60217.log: gzip 88.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.84319.log.gz --verbose 2024-03-06T07:33:58.444 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.59095.log 2024-03-06T07:33:58.444 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60217.log.gz/var/log/ceph/ceph-client.admin.75881.log: 2024-03-06T07:33:58.444 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55604.log 2024-03-06T07:33:58.444 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75881.log.gz 2024-03-06T07:33:58.445 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51299.log 2024-03-06T07:33:58.445 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87519.log 2024-03-06T07:33:58.445 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59095.log.gz 2024-03-06T07:33:58.445 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55604.log.gz/var/log/ceph/ceph-client.admin.51299.log: 2024-03-06T07:33:58.445 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57714.log 2024-03-06T07:33:58.445 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51299.log.gz 2024-03-06T07:33:58.445 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70834.log 2024-03-06T07:33:58.445 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45638.log 2024-03-06T07:33:58.446 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87519.log.gz 2024-03-06T07:33:58.446 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57714.log.gz 2024-03-06T07:33:58.446 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76875.log 2024-03-06T07:33:58.446 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70834.log.gz 2024-03-06T07:33:58.446 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45190.log 2024-03-06T07:33:58.446 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76875.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45638.log.gz 2024-03-06T07:33:58.446 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85616.log 2024-03-06T07:33:58.446 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45190.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76875.log.gz 2024-03-06T07:33:58.446 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85322.log 2024-03-06T07:33:58.447 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45190.log.gz 2024-03-06T07:33:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58873.log 2024-03-06T07:33:58.447 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.85616.log.gz 2024-03-06T07:33:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42243.log 2024-03-06T07:33:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70588.log 2024-03-06T07:33:58.447 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.85322.log.gz 2024-03-06T07:33:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42243.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58873.log.gz 2024-03-06T07:33:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80346.log 2024-03-06T07:33:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70588.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42243.log.gz 2024-03-06T07:33:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68852.log 2024-03-06T07:33:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80346.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70588.log.gz 2024-03-06T07:33:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64285.log 2024-03-06T07:33:58.448 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80346.log.gz 2024-03-06T07:33:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80099.log 2024-03-06T07:33:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89922.log 2024-03-06T07:33:58.448 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68852.log.gz 2024-03-06T07:33:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80099.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64285.log.gz 2024-03-06T07:33:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48486.log 2024-03-06T07:33:58.449 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80099.log.gz 2024-03-06T07:33:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61674.log 2024-03-06T07:33:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74236.log 2024-03-06T07:33:58.449 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.89922.log.gz 2024-03-06T07:33:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61674.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48486.log.gz 2024-03-06T07:33:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48612.log 2024-03-06T07:33:58.449 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61674.log.gz 2024-03-06T07:33:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82940.log 2024-03-06T07:33:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48612.log: gzip -5 88.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.74236.log.gz /var/log/ceph/ceph-client.admin.89525.log 2024-03-06T07:33:58.450 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-06T07:33:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82940.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48612.log.gz 2024-03-06T07:33:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71141.log 2024-03-06T07:33:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63972.log 2024-03-06T07:33:58.450 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.82940.log.gz 2024-03-06T07:33:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84067.log 2024-03-06T07:33:58.450 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.89525.log.gz 2024-03-06T07:33:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63972.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71141.log.gz 2024-03-06T07:33:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85742.log 2024-03-06T07:33:58.451 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63972.log.gz 2024-03-06T07:33:58.451 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45191.log 2024-03-06T07:33:58.451 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85742.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.84067.log.gz 2024-03-06T07:33:58.451 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61085.log 2024-03-06T07:33:58.451 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45191.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.85742.log.gz 2024-03-06T07:33:58.451 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69878.log 2024-03-06T07:33:58.451 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45191.log.gz 2024-03-06T07:33:58.451 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62437.log 2024-03-06T07:33:58.452 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.69878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43486.log 2024-03-06T07:33:58.452 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.62437.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69878.log.gz 2024-03-06T07:33:58.452 INFO:teuthology.orchestra.run.smithi001.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.61085.log.gz 2024-03-06T07:33:58.452 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59225.log 2024-03-06T07:33:58.452 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68323.log 2024-03-06T07:33:58.452 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62437.log.gz 2024-03-06T07:33:58.452 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43486.log: /var/log/ceph/ceph-client.admin.59225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48072.log 2024-03-06T07:33:58.452 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43486.log.gz 2024-03-06T07:33:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60725.log 2024-03-06T07:33:58.453 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59225.log.gz 2024-03-06T07:33:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48072.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68323.log.gz 2024-03-06T07:33:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77230.log 2024-03-06T07:33:58.453 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48072.log.gz 2024-03-06T07:33:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52116.log 2024-03-06T07:33:58.453 INFO:teuthology.orchestra.run.smithi001.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60725.log.gz 2024-03-06T07:33:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81985.log 2024-03-06T07:33:58.453 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52116.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.77230.log.gz 2024-03-06T07:33:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82858.log 2024-03-06T07:33:58.454 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52116.log.gz 2024-03-06T07:33:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65471.log 2024-03-06T07:33:58.454 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81985.log.gz 2024-03-06T07:33:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50322.log 2024-03-06T07:33:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65471.log: gzip 88.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.82858.log.gz --verbose 2024-03-06T07:33:58.454 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.71631.log 2024-03-06T07:33:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50322.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65471.log.gz 2024-03-06T07:33:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89399.log 2024-03-06T07:33:58.454 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50322.log.gz 2024-03-06T07:33:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59053.log 2024-03-06T07:33:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71631.log: /var/log/ceph/ceph-client.admin.89399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45596.log 2024-03-06T07:33:58.455 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71631.log.gz 2024-03-06T07:33:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59053.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.89399.log.gz 2024-03-06T07:33:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84968.log 2024-03-06T07:33:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45596.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59053.log.gz 2024-03-06T07:33:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51425.log 2024-03-06T07:33:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84968.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45596.log.gz 2024-03-06T07:33:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71938.log 2024-03-06T07:33:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51425.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84968.log.gz 2024-03-06T07:33:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86097.log 2024-03-06T07:33:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71938.log: gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51425.log.gz -5 2024-03-06T07:33:58.456 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.79683.log 2024-03-06T07:33:58.456 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71938.log.gz 2024-03-06T07:33:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86097.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86097.log.gz 2024-03-06T07:33:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60641.log 2024-03-06T07:33:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73667.log 2024-03-06T07:33:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60641.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.79683.log.gz 2024-03-06T07:33:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62651.log 2024-03-06T07:33:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73667.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60641.log.gz 2024-03-06T07:33:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88251.log 2024-03-06T07:33:58.457 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73667.log.gz 2024-03-06T07:33:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.62651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81678.log 2024-03-06T07:33:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88251.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62651.log.gz 2024-03-06T07:33:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54516.log 2024-03-06T07:33:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81678.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.88251.log.gz 2024-03-06T07:33:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81089.log 2024-03-06T07:33:58.458 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81678.log.gz 2024-03-06T07:33:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78790.log 2024-03-06T07:33:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54197.log 2024-03-06T07:33:58.458 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54516.log.gz 2024-03-06T07:33:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78790.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81089.log.gz 2024-03-06T07:33:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82153.log 2024-03-06T07:33:58.458 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78790.log.gz 2024-03-06T07:33:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61338.log 2024-03-06T07:33:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82153.log: gzip -5 88.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.54197.log.gz 2024-03-06T07:33:58.459 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.53848.log 2024-03-06T07:33:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61338.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82153.log.gz 2024-03-06T07:33:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75797.log 2024-03-06T07:33:58.459 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61338.log.gz 2024-03-06T07:33:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75055.log 2024-03-06T07:33:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75797.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53848.log.gz 2024-03-06T07:33:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41867.log 2024-03-06T07:33:58.459 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75055.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75797.log.gz 2024-03-06T07:33:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59934.log 2024-03-06T07:33:58.460 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75055.log.gz 2024-03-06T07:33:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71355.log 2024-03-06T07:33:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55394.log 88.1% 2024-03-06T07:33:58.460 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41867.log.gz 2024-03-06T07:33:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71355.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59934.log.gz 2024-03-06T07:33:58.460 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83479.log 2024-03-06T07:33:58.460 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71355.log.gz 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46923.log 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83479.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55394.log.gz 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83108.log 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46923.log: gzip -5 --verbose -- 89.3% /var/log/ceph/ceph-client.admin.90643.log -- replaced with /var/log/ceph/ceph-client.admin.83479.log.gz 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83108.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46923.log.gz -5 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.64201.log 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.83108.log.gz 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63294.log 2024-03-06T07:33:58.462 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90643.log.gz 2024-03-06T07:33:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56786.log 2024-03-06T07:33:58.462 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64201.log.gz/var/log/ceph/ceph-client.admin.63294.log: 2024-03-06T07:33:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85364.log 2024-03-06T07:33:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59892.log 2024-03-06T07:33:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85364.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56786.log.gz 2024-03-06T07:33:58.462 INFO:teuthology.orchestra.run.smithi001.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.63294.log.gz 2024-03-06T07:33:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82447.log 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85364.log.gz 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63486.log 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59892.log.gz 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83647.log 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63486.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82447.log.gz -5 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49507.log 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50196.log 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.83647.log.gz 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48198.log 2024-03-06T07:33:58.464 INFO:teuthology.orchestra.run.smithi001.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.63486.log.gz 2024-03-06T07:33:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73320.log 2024-03-06T07:33:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50196.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49507.log.gz 2024-03-06T07:33:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72110.log 2024-03-06T07:33:58.464 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50196.log.gz 2024-03-06T07:33:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73320.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48198.log.gz 2024-03-06T07:33:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74152.log 2024-03-06T07:33:58.464 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72110.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.73320.log.gz 2024-03-06T07:33:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73194.log 2024-03-06T07:33:58.465 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6%/var/log/ceph/ceph-client.admin.74152.log: -- replaced with /var/log/ceph/ceph-client.admin.72110.log.gz 2024-03-06T07:33:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85658.log 2024-03-06T07:33:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73194.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74152.log.gz 2024-03-06T07:33:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49940.log 2024-03-06T07:33:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85658.log: gzip 88.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.73194.log.gz 2024-03-06T07:33:58.465 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.86517.log 2024-03-06T07:33:58.465 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.85658.log.gz 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85448.log 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49940.log.gz 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66931.log 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85448.log: gzip -5 --verbose -- 88.8% /var/log/ceph/ceph-client.admin.51154.log -- replaced with /var/log/ceph/ceph-client.admin.86517.log.gz 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49549.log 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.85448.log.gz 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51154.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66931.log.gz 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79514.log 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51154.log.gz 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77851.log 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61986.log 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49549.log.gz 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77851.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.79514.log.gz 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44870.log 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77851.log.gz 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65780.log 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71896.log 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61986.log.gz 2024-03-06T07:33:58.468 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65780.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44870.log.gz 2024-03-06T07:33:58.468 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62832.log 2024-03-06T07:33:58.468 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65780.log.gz 2024-03-06T07:33:58.468 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75227.log 2024-03-06T07:33:58.468 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.62832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53550.log 2024-03-06T07:33:58.468 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71896.log.gz 2024-03-06T07:33:58.468 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75227.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62832.log.gz 2024-03-06T07:33:58.468 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63571.log 2024-03-06T07:33:58.469 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.75227.log.gz 2024-03-06T07:33:58.469 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55772.log 2024-03-06T07:33:58.469 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49219.log 2024-03-06T07:33:58.469 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53550.log.gz 2024-03-06T07:33:58.469 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55772.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63571.log.gz 2024-03-06T07:33:58.469 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60343.log 2024-03-06T07:33:58.469 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55772.log.gz 2024-03-06T07:33:58.469 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50467.log 2024-03-06T07:33:58.469 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60343.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.49219.log.gz 2024-03-06T07:33:58.469 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.90430.log 2024-03-06T07:33:58.470 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50467.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60343.log.gz 2024-03-06T07:33:58.470 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52032.log 2024-03-06T07:33:58.470 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50467.log.gz 2024-03-06T07:33:58.470 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86559.log 2024-03-06T07:33:58.470 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90430.log.gz 2024-03-06T07:33:58.470 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58969.log 2024-03-06T07:33:58.470 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86559.log: gzip -5 88.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.52032.log.gz /var/log/ceph/ceph-client.admin.43402.log 2024-03-06T07:33:58.470 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-06T07:33:58.470 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58969.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.86559.log.gz 2024-03-06T07:33:58.471 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86997.log 2024-03-06T07:33:58.471 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58969.log.gz 2024-03-06T07:33:58.471 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70130.log 2024-03-06T07:33:58.471 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43402.log.gz 2024-03-06T07:33:58.471 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80592.log 2024-03-06T07:33:58.471 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6%/var/log/ceph/ceph-client.admin.70130.log: -- replaced with /var/log/ceph/ceph-client.admin.86997.log.gz 2024-03-06T07:33:58.471 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70509.log 2024-03-06T07:33:58.471 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70130.log.gz 2024-03-06T07:33:58.471 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88065.log 2024-03-06T07:33:58.472 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70509.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80592.log.gz 2024-03-06T07:33:58.472 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76007.log 2024-03-06T07:33:58.472 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70509.log.gz 2024-03-06T07:33:58.472 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83395.log 2024-03-06T07:33:58.472 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.88065.log.gz 2024-03-06T07:33:58.472 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52585.log 2024-03-06T07:33:58.472 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83395.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76007.log.gz 2024-03-06T07:33:58.472 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42512.log 2024-03-06T07:33:58.472 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52585.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83395.log.gz 2024-03-06T07:33:58.473 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89837.log 2024-03-06T07:33:58.473 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7%/var/log/ceph/ceph-client.admin.42512.log: -- replaced with /var/log/ceph/ceph-client.admin.52585.log.gz 2024-03-06T07:33:58.473 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50938.log 2024-03-06T07:33:58.473 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79767.log 2024-03-06T07:33:58.473 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42512.log.gz 2024-03-06T07:33:58.473 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50938.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.89837.log.gz 2024-03-06T07:33:58.473 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74067.log 2024-03-06T07:33:58.473 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76749.log 2024-03-06T07:33:58.473 INFO:teuthology.orchestra.run.smithi001.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.50938.log.gz 2024-03-06T07:33:58.474 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74067.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.79767.log.gz 2024-03-06T07:33:58.474 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54642.log 2024-03-06T07:33:58.474 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75437.log 2024-03-06T07:33:58.474 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74067.log.gz 2024-03-06T07:33:58.474 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76749.log: /var/log/ceph/ceph-client.admin.54642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90895.log 2024-03-06T07:33:58.474 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76749.log.gz 2024-03-06T07:33:58.474 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73278.log 2024-03-06T07:33:58.474 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54642.log.gz 2024-03-06T07:33:58.474 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75437.log.gz 2024-03-06T07:33:58.475 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52501.log 2024-03-06T07:33:58.475 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.90895.log.gz 2024-03-06T07:33:58.475 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53718.log 2024-03-06T07:33:58.475 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73278.log.gz 2024-03-06T07:33:58.475 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61296.log 2024-03-06T07:33:58.475 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70467.log 2024-03-06T07:33:58.475 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52501.log.gz 2024-03-06T07:33:58.475 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53718.log.gz 2024-03-06T07:33:58.476 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54299.log 2024-03-06T07:33:58.476 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61296.log.gz 2024-03-06T07:33:58.476 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70004.log 2024-03-06T07:33:58.476 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70341.log 2024-03-06T07:33:58.476 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70467.log.gz 2024-03-06T07:33:58.476 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54299.log.gz 2024-03-06T07:33:58.476 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62874.log 2024-03-06T07:33:58.476 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70004.log.gz 2024-03-06T07:33:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56605.log 2024-03-06T07:33:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.62874.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70341.log.gz 2024-03-06T07:33:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87291.log 2024-03-06T07:33:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56605.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62874.log.gz 2024-03-06T07:33:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69836.log 2024-03-06T07:33:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87291.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56605.log.gz 2024-03-06T07:33:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56656.log 2024-03-06T07:33:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.69836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88671.log 2024-03-06T07:33:58.477 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87291.log.gz 2024-03-06T07:33:58.477 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69836.log.gz 2024-03-06T07:33:58.478 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76791.log 2024-03-06T07:33:58.478 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56656.log.gz 2024-03-06T07:33:58.478 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56349.log 2024-03-06T07:33:58.478 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61926.log 2024-03-06T07:33:58.478 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.88671.log.gz 2024-03-06T07:33:58.478 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56349.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76791.log.gz 2024-03-06T07:33:58.478 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81383.log 2024-03-06T07:33:58.478 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56349.log.gz 2024-03-06T07:33:58.478 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63858.log 2024-03-06T07:33:58.479 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63168.log 2024-03-06T07:33:58.479 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61926.log.gz 2024-03-06T07:33:58.479 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63858.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81383.log.gz 2024-03-06T07:33:58.479 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49889.log 2024-03-06T07:33:58.479 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44873.log 2024-03-06T07:33:58.479 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49889.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.63858.log.gz 2024-03-06T07:33:58.479 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63168.log.gz 2024-03-06T07:33:58.479 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75004.log 2024-03-06T07:33:58.479 INFO:teuthology.orchestra.run.smithi001.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49889.log.gz 2024-03-06T07:33:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45806.log 2024-03-06T07:33:58.480 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44873.log.gz 2024-03-06T07:33:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59137.log 2024-03-06T07:33:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42554.log 2024-03-06T07:33:58.480 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75004.log.gz 2024-03-06T07:33:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59137.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45806.log.gz 2024-03-06T07:33:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55562.log 2024-03-06T07:33:58.480 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59137.log.gz 2024-03-06T07:33:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.42554.log: -- /var/log/ceph/ceph-client.admin.52673.log 2024-03-06T07:33:58.481 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62702.log 2024-03-06T07:33:58.481 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42554.log.gz 2024-03-06T07:33:58.481 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52673.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55562.log.gz 2024-03-06T07:33:58.481 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84530.log 2024-03-06T07:33:58.481 INFO:teuthology.orchestra.run.smithi001.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.52673.log.gz 2024-03-06T07:33:58.481 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.62702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82027.log 2024-03-06T07:33:58.481 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46965.log 2024-03-06T07:33:58.481 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62702.log.gz 2024-03-06T07:33:58.481 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82027.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.84530.log.gz 2024-03-06T07:33:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48030.log 2024-03-06T07:33:58.482 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.82027.log.gz 2024-03-06T07:33:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85262.log 2024-03-06T07:33:58.482 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46965.log.gz 2024-03-06T07:33:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43838.log 2024-03-06T07:33:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77314.log 2024-03-06T07:33:58.482 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48030.log.gz 2024-03-06T07:33:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43838.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85262.log.gz 2024-03-06T07:33:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72668.log 2024-03-06T07:33:58.483 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43838.log.gz 2024-03-06T07:33:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58393.log 2024-03-06T07:33:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72668.log: 88.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.77314.log.gz -5 2024-03-06T07:33:58.483 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.81047.log 2024-03-06T07:33:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58393.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72668.log.gz 2024-03-06T07:33:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51990.log 2024-03-06T07:33:58.483 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4%/var/log/ceph/ceph-client.admin.81047.log: -- replaced with /var/log/ceph/ceph-client.admin.58393.log.gz 2024-03-06T07:33:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70918.log 2024-03-06T07:33:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51990.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.81047.log.gz 2024-03-06T07:33:58.484 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78160.log 2024-03-06T07:33:58.484 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70918.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51990.log.gz 2024-03-06T07:33:58.484 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89693.log 2024-03-06T07:33:58.484 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70918.log.gz 2024-03-06T07:33:58.484 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86475.log 2024-03-06T07:33:58.484 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89693.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78160.log.gz 2024-03-06T07:33:58.484 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44378.log 2024-03-06T07:33:58.484 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86475.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.89693.log.gz 2024-03-06T07:33:58.484 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44556.log 2024-03-06T07:33:58.485 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86475.log.gz 2024-03-06T07:33:58.485 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49633.log 2024-03-06T07:33:58.485 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57542.log 2024-03-06T07:33:58.485 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44378.log.gz 2024-03-06T07:33:58.485 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49633.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.44556.log.gz 2024-03-06T07:33:58.485 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59724.log 2024-03-06T07:33:58.485 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49633.log.gz 2024-03-06T07:33:58.485 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69920.log 2024-03-06T07:33:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51677.log 2024-03-06T07:33:58.486 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57542.log.gz 2024-03-06T07:33:58.486 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59724.log.gz 2024-03-06T07:33:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.69920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78748.log 2024-03-06T07:33:58.486 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.69920.log.gz 2024-03-06T07:33:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85868.log 2024-03-06T07:33:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86685.log 2024-03-06T07:33:58.486 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51677.log.gz 2024-03-06T07:33:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85868.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78748.log.gz 2024-03-06T07:33:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57630.log 2024-03-06T07:33:58.487 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85868.log.gz 2024-03-06T07:33:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43211.log 2024-03-06T07:33:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62395.log 2024-03-06T07:33:58.487 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86685.log.gz 2024-03-06T07:33:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43211.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57630.log.gz 2024-03-06T07:33:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78436.log 2024-03-06T07:33:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.62395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68618.log 2024-03-06T07:33:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78436.log: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.62395.log.gz -5 2024-03-06T07:33:58.487 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.80782.log 2024-03-06T07:33:58.487 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0%/var/log/ceph/ceph-client.admin.68618.log: -- replaced with /var/log/ceph/ceph-client.admin.78436.log.gz 2024-03-06T07:33:58.488 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69228.log 2024-03-06T07:33:58.488 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80782.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68618.log.gz 2024-03-06T07:33:58.488 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64976.log 2024-03-06T07:33:58.488 INFO:teuthology.orchestra.run.smithi001.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.43211.log.gz 2024-03-06T07:33:58.488 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80782.log.gz 2024-03-06T07:33:58.488 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.69228.log: -- /var/log/ceph/ceph-client.admin.88545.log 2024-03-06T07:33:58.488 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88923.log 2024-03-06T07:33:58.488 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88545.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.69228.log.gz 2024-03-06T07:33:58.488 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64976.log.gz 2024-03-06T07:33:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56475.log 2024-03-06T07:33:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88923.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88545.log.gz 2024-03-06T07:33:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45189.log 2024-03-06T07:33:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56475.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.88923.log.gz 2024-03-06T07:33:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53676.log 2024-03-06T07:33:58.489 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56475.log.gz 2024-03-06T07:33:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78987.log 2024-03-06T07:33:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47708.log 2024-03-06T07:33:58.489 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45189.log.gz 2024-03-06T07:33:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78987.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53676.log.gz 2024-03-06T07:33:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55688.log 2024-03-06T07:33:58.490 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78987.log.gz 2024-03-06T07:33:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81341.log 2024-03-06T07:33:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75395.log 2024-03-06T07:33:58.490 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47708.log.gz 2024-03-06T07:33:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81341.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55688.log.gz 2024-03-06T07:33:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63816.log 2024-03-06T07:33:58.490 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81341.log.gz 2024-03-06T07:33:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57056.log 2024-03-06T07:33:58.491 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90253.log 2024-03-06T07:33:58.491 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75395.log.gz 2024-03-06T07:33:58.491 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57056.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63816.log.gz 2024-03-06T07:33:58.491 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80833.log 2024-03-06T07:33:58.491 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57056.log.gz 2024-03-06T07:33:58.491 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63042.log 2024-03-06T07:33:58.491 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88503.log 2024-03-06T07:33:58.491 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90253.log.gz 2024-03-06T07:33:58.491 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63042.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80833.log.gz 2024-03-06T07:33:58.491 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77061.log 2024-03-06T07:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63042.log.gz 2024-03-06T07:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89357.log 2024-03-06T07:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56433.log 2024-03-06T07:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.88503.log.gz 2024-03-06T07:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89357.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.77061.log.gz 2024-03-06T07:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44042.log 2024-03-06T07:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.89357.log.gz 2024-03-06T07:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76049.log 2024-03-06T07:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63000.log 2024-03-06T07:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56433.log.gz 2024-03-06T07:33:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76049.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44042.log.gz 2024-03-06T07:33:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83196.log 2024-03-06T07:33:58.493 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76049.log.gz 2024-03-06T07:33:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62916.log 2024-03-06T07:33:58.493 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63000.log.gz 2024-03-06T07:33:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51906.log 2024-03-06T07:33:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.62916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68533.log 2024-03-06T07:33:58.493 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83196.log.gz 2024-03-06T07:33:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51906.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62916.log.gz 2024-03-06T07:33:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60175.log 2024-03-06T07:33:58.494 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51906.log.gz 2024-03-06T07:33:58.494 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84488.log 2024-03-06T07:33:58.494 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70088.log 2024-03-06T07:33:58.494 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68533.log.gz 2024-03-06T07:33:58.494 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60175.log.gz 2024-03-06T07:33:58.494 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77946.log 2024-03-06T07:33:58.494 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84488.log.gz 2024-03-06T07:33:58.494 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46584.log 2024-03-06T07:33:58.495 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90937.log 2024-03-06T07:33:58.495 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70088.log.gz 2024-03-06T07:33:58.495 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77946.log.gz 2024-03-06T07:33:58.495 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.46584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.46584.log.gz 2024-03-06T07:33:58.495 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90811.log 2024-03-06T07:33:58.495 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76151.log 2024-03-06T07:33:58.495 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67500.log 2024-03-06T07:33:58.495 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.90937.log.gz 2024-03-06T07:33:58.495 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76151.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.90811.log.gz 2024-03-06T07:33:58.496 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45680.log 2024-03-06T07:33:58.496 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76151.log.gz 2024-03-06T07:33:58.496 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58645.log 2024-03-06T07:33:58.496 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85052.log 2024-03-06T07:33:58.496 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67500.log.gz 2024-03-06T07:33:58.496 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58645.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45680.log.gz 2024-03-06T07:33:58.496 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64597.log 2024-03-06T07:33:58.496 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58645.log.gz 2024-03-06T07:33:58.496 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65906.log 2024-03-06T07:33:58.496 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78076.log 2024-03-06T07:33:58.497 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85052.log.gz 2024-03-06T07:33:58.497 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65906.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64597.log.gz 2024-03-06T07:33:58.497 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52543.log 2024-03-06T07:33:58.497 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65906.log.gz 2024-03-06T07:33:58.497 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87459.log 2024-03-06T07:33:58.497 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43670.log 2024-03-06T07:33:58.497 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.78076.log.gz 2024-03-06T07:33:58.497 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87459.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52543.log.gz 2024-03-06T07:33:58.497 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64892.log 2024-03-06T07:33:58.498 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87459.log.gz 2024-03-06T07:33:58.498 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44210.log 2024-03-06T07:33:58.498 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71547.log 2024-03-06T07:33:58.498 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43670.log.gz 2024-03-06T07:33:58.498 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44210.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64892.log.gz 2024-03-06T07:33:58.498 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71229.log 2024-03-06T07:33:58.498 INFO:teuthology.orchestra.run.smithi001.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.44210.log.gz 2024-03-06T07:33:58.498 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48570.log 2024-03-06T07:33:58.498 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71547.log.gz 2024-03-06T07:33:58.498 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53022.log 2024-03-06T07:33:58.499 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47919.log 2024-03-06T07:33:58.499 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71229.log.gz 2024-03-06T07:33:58.499 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48570.log.gz 2024-03-06T07:33:58.499 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71715.log 2024-03-06T07:33:58.499 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47919.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53022.log.gz 2024-03-06T07:33:58.499 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65738.log 2024-03-06T07:33:58.499 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71715.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47919.log.gz 2024-03-06T07:33:58.499 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49032.log 2024-03-06T07:33:58.499 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71715.log.gz 2024-03-06T07:33:58.499 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71099.log 2024-03-06T07:33:58.500 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65738.log.gz/var/log/ceph/ceph-client.admin.49032.log: 2024-03-06T07:33:58.500 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87813.log 2024-03-06T07:33:58.503 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71099.log: gzip 89.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.49032.log.gz 2024-03-06T07:33:58.503 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.79809.log 2024-03-06T07:33:58.503 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71099.log.gz 2024-03-06T07:33:58.503 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59183.log 2024-03-06T07:33:58.503 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87813.log.gz 2024-03-06T07:33:58.503 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66530.log 2024-03-06T07:33:58.503 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82069.log 2024-03-06T07:33:58.504 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79809.log.gz 2024-03-06T07:33:58.504 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59183.log.gz 2024-03-06T07:33:58.504 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57332.log 2024-03-06T07:33:58.504 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82069.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.66530.log.gz 2024-03-06T07:33:58.504 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52896.log 2024-03-06T07:33:58.504 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78202.log 2024-03-06T07:33:58.504 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82069.log.gz 2024-03-06T07:33:58.504 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57332.log.gz 2024-03-06T07:33:58.505 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49717.log 2024-03-06T07:33:58.505 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52896.log.gz 2024-03-06T07:33:58.505 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78328.log 2024-03-06T07:33:58.505 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65566.log 2024-03-06T07:33:58.505 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78202.log.gz 2024-03-06T07:33:58.505 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49717.log.gz 2024-03-06T07:33:58.505 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68365.log 2024-03-06T07:33:58.505 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78328.log.gz 2024-03-06T07:33:58.505 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72026.log 2024-03-06T07:33:58.506 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59598.log 2024-03-06T07:33:58.506 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65566.log.gz 2024-03-06T07:33:58.506 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68365.log.gz 2024-03-06T07:33:58.506 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80550.log 2024-03-06T07:33:58.506 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72026.log.gz 2024-03-06T07:33:58.506 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47301.log 2024-03-06T07:33:58.506 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80550.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59598.log.gzgzip 2024-03-06T07:33:58.506 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47133.log 2024-03-06T07:33:58.506 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.80550.log.gz 2024-03-06T07:33:58.507 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90338.log 2024-03-06T07:33:58.507 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47301.log.gz 2024-03-06T07:33:58.507 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82531.log 2024-03-06T07:33:58.507 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90338.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.47133.log.gz 2024-03-06T07:33:58.507 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84445.log 2024-03-06T07:33:58.507 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.90338.log.gz 2024-03-06T07:33:58.507 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85700.log 2024-03-06T07:33:58.507 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.82531.log.gz 2024-03-06T07:33:58.507 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90727.log 2024-03-06T07:33:58.508 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85700.log: gzip -5 --verbose -- 89.1% /var/log/ceph/ceph-client.admin.89777.log 2024-03-06T07:33:58.508 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.84445.log.gz 2024-03-06T07:33:58.508 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3%/var/log/ceph/ceph-client.admin.90727.log: -- replaced with /var/log/ceph/ceph-client.admin.85700.log.gz 2024-03-06T07:33:58.508 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-06T07:33:58.508 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47596.log 2024-03-06T07:33:58.508 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.90727.log.gz 2024-03-06T07:33:58.508 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.audit.log: gzip 88.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.89777.log.gz --verbose 2024-03-06T07:33:58.508 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.66576.log 2024-03-06T07:33:58.509 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47469.log 2024-03-06T07:33:58.509 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9%/var/log/ceph/ceph-client.admin.66576.log: -- replaced with /var/log/ceph/ceph-client.admin.47596.log.gz 2024-03-06T07:33:58.509 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79087.log 2024-03-06T07:33:58.509 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66576.log.gz 2024-03-06T07:33:58.509 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65427.log 2024-03-06T07:33:58.509 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47469.log: /var/log/ceph/ceph-client.admin.79087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49092.log 2024-03-06T07:33:58.509 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47469.log.gz 2024-03-06T07:33:58.509 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74772.log 2024-03-06T07:33:58.510 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49092.log: gzip -5 90.8% --verbose -- /var/log/ceph/ceph-client.admin.38416.log 2024-03-06T07:33:58.510 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.79087.log.gz 2024-03-06T07:33:58.510 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65427.log.gz 2024-03-06T07:33:58.510 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53106.log 2024-03-06T07:33:58.510 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49092.log.gz 2024-03-06T07:33:58.510 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74772.log.gz 2024-03-06T07:33:58.510 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89651.log 2024-03-06T07:33:58.510 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62028.log 2024-03-06T07:33:58.510 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38416.log.gz 2024-03-06T07:33:58.511 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89651.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53106.log.gz 2024-03-06T07:33:58.511 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53634.log 2024-03-06T07:33:58.511 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71845.log 2024-03-06T07:33:58.511 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6%/var/log/ceph/ceph-client.admin.62028.log: -- replaced with /var/log/ceph/ceph-client.admin.89651.log.gz 2024-03-06T07:33:58.511 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53634.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62028.log.gz 2024-03-06T07:33:58.511 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62958.log 2024-03-06T07:33:58.511 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53634.log.gz 2024-03-06T07:33:58.511 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87123.log 2024-03-06T07:33:58.511 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71845.log: /var/log/ceph/ceph-client.admin.62958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56870.log 2024-03-06T07:33:58.512 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71845.log.gz 2024-03-06T07:33:58.512 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:33:58.512 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62958.log.gz 2024-03-06T07:33:58.512 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85532.log 2024-03-06T07:33:58.512 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56870.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87123.log.gz 2024-03-06T07:33:58.512 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50754.log 2024-03-06T07:33:58.512 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56870.log.gz 2024-03-06T07:33:58.512 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76917.log 2024-03-06T07:33:58.513 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85532.log.gz 2024-03-06T07:33:58.513 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71187.log 2024-03-06T07:33:58.513 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81822.log 2024-03-06T07:33:58.513 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71187.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76917.log.gz 2024-03-06T07:33:58.513 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67952.log 2024-03-06T07:33:58.513 INFO:teuthology.orchestra.run.smithi001.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.50754.log.gz 2024-03-06T07:33:58.513 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81822.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.71187.log.gz 2024-03-06T07:33:58.513 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80057.log 2024-03-06T07:33:58.513 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81822.log.gz 2024-03-06T07:33:58.514 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46016.log 2024-03-06T07:33:58.514 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80057.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67952.log.gz 2024-03-06T07:33:58.514 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52335.log 2024-03-06T07:33:58.514 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46016.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80057.log.gz 2024-03-06T07:33:58.514 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78118.log 2024-03-06T07:33:58.514 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46016.log.gz 2024-03-06T07:33:58.514 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86913.log 2024-03-06T07:33:58.514 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78118.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52335.log.gz 2024-03-06T07:33:58.514 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87645.log 2024-03-06T07:33:58.515 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78118.log.gz 2024-03-06T07:33:58.515 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81215.log 2024-03-06T07:33:58.515 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.87645.log: -5 88.6% --verbose -- replaced with /var/log/ceph/ceph-client.admin.86913.log.gz -- 2024-03-06T07:33:58.515 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.68575.log 2024-03-06T07:33:58.515 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49801.log 2024-03-06T07:33:58.515 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87645.log.gz 2024-03-06T07:33:58.515 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68575.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81215.log.gz 2024-03-06T07:33:58.515 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56391.log 2024-03-06T07:33:58.515 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68575.log.gz 2024-03-06T07:33:58.516 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76707.log 2024-03-06T07:33:58.516 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78478.log 2024-03-06T07:33:58.516 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49801.log.gz 2024-03-06T07:33:58.516 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76707.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56391.log.gz 2024-03-06T07:33:58.516 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73965.log 2024-03-06T07:33:58.516 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76707.log.gz 2024-03-06T07:33:58.516 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59682.log 2024-03-06T07:33:58.516 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49177.log 2024-03-06T07:33:58.516 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78478.log.gz 2024-03-06T07:33:58.517 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73965.log.gz 2024-03-06T07:33:58.517 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66098.log 2024-03-06T07:33:58.517 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59682.log.gz 2024-03-06T07:33:58.517 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69794.log 2024-03-06T07:33:58.517 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59453.log 2024-03-06T07:33:58.517 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49177.log.gz 2024-03-06T07:33:58.517 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66098.log.gz 2024-03-06T07:33:58.517 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.69794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60473.log 2024-03-06T07:33:58.517 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69794.log.gz 2024-03-06T07:33:58.517 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54726.log 2024-03-06T07:33:58.518 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86643.log 2024-03-06T07:33:58.518 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59453.log.gz 2024-03-06T07:33:58.518 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60473.log.gz 2024-03-06T07:33:58.518 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80267.log 2024-03-06T07:33:58.518 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54726.log.gz 2024-03-06T07:33:58.518 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87417.log 2024-03-06T07:33:58.518 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68810.log 2024-03-06T07:33:58.518 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86643.log.gz 2024-03-06T07:33:58.518 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80267.log.gz 2024-03-06T07:33:58.518 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77765.log 2024-03-06T07:33:58.519 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87417.log.gz 2024-03-06T07:33:58.519 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60767.log 2024-03-06T07:33:58.519 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88149.log 2024-03-06T07:33:58.519 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68810.log.gz 2024-03-06T07:33:58.519 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77765.log.gz 2024-03-06T07:33:58.519 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89109.log 2024-03-06T07:33:58.519 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60767.log.gz 2024-03-06T07:33:58.519 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58519.log 2024-03-06T07:33:58.519 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57798.log 2024-03-06T07:33:58.519 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.88149.log.gz 2024-03-06T07:33:58.520 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.89109.log.gz 2024-03-06T07:33:58.520 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38313.log 2024-03-06T07:33:58.520 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58519.log.gz 2024-03-06T07:33:58.520 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79851.log 2024-03-06T07:33:58.520 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86223.log 2024-03-06T07:33:58.520 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57798.log.gz 2024-03-06T07:33:58.520 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38313.log.gz 2024-03-06T07:33:58.520 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59393.log 2024-03-06T07:33:58.520 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42638.log 2024-03-06T07:33:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86223.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79851.log.gz 2024-03-06T07:33:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89964.log 2024-03-06T07:33:58.521 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.86223.log.gz 2024-03-06T07:33:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42638.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59393.log.gz 2024-03-06T07:33:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50628.log 2024-03-06T07:33:58.521 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42638.log.gz 2024-03-06T07:33:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48402.log 2024-03-06T07:33:58.521 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.89964.log.gz 2024-03-06T07:33:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79598.log 2024-03-06T07:33:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48402.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50628.log.gz 2024-03-06T07:33:58.522 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48402.log.gz 2024-03-06T07:33:58.522 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66889.log 2024-03-06T07:33:58.522 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55184.log 2024-03-06T07:33:58.522 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.79598.log.gz 2024-03-06T07:33:58.522 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70750.log 2024-03-06T07:33:58.522 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55184.log: gzip 88.1% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.66889.log.gz -- 2024-03-06T07:33:58.522 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.72422.log 2024-03-06T07:33:58.522 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55184.log.gz 2024-03-06T07:33:58.522 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66266.log 2024-03-06T07:33:58.522 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70750.log.gz 2024-03-06T07:33:58.523 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58435.log 2024-03-06T07:33:58.523 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66266.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72422.log.gz 2024-03-06T07:33:58.523 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80875.log 2024-03-06T07:33:58.523 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58435.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.66266.log.gz 2024-03-06T07:33:58.523 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58050.log 2024-03-06T07:33:58.523 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80875.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58435.log.gz 2024-03-06T07:33:58.523 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82699.log 2024-03-06T07:33:58.523 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66618.log 2024-03-06T07:33:58.523 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.80875.log.gz 2024-03-06T07:33:58.524 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58050.log.gz 2024-03-06T07:33:58.524 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52808.log 2024-03-06T07:33:58.524 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64243.log 2024-03-06T07:33:58.524 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66618.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.82699.log.gz 2024-03-06T07:33:58.524 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82982.log 2024-03-06T07:33:58.524 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52808.log.gz 2024-03-06T07:33:58.524 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45722.log 2024-03-06T07:33:58.524 INFO:teuthology.orchestra.run.smithi001.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.66618.log.gz 2024-03-06T07:33:58.524 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82982.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64243.log.gz 2024-03-06T07:33:58.525 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76306.log 2024-03-06T07:33:58.525 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.82982.log.gz 2024-03-06T07:33:58.525 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77988.log 2024-03-06T07:33:58.525 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54558.log 2024-03-06T07:33:58.525 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45722.log.gz 2024-03-06T07:33:58.525 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77988.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76306.log.gz 2024-03-06T07:33:58.525 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52938.log 2024-03-06T07:33:58.525 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77988.log.gz 2024-03-06T07:33:58.525 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42028.log 2024-03-06T07:33:58.526 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52938.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54558.log.gz 2024-03-06T07:33:58.526 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65990.log 2024-03-06T07:33:58.526 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52938.log.gz 2024-03-06T07:33:58.526 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69336.log 2024-03-06T07:33:58.526 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42028.log.gz 2024-03-06T07:33:58.526 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66308.log 2024-03-06T07:33:58.526 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.69336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58110.log 2024-03-06T07:33:58.526 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69336.log.gz 2024-03-06T07:33:58.526 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78850.log 2024-03-06T07:33:58.527 INFO:teuthology.orchestra.run.smithi001.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.65990.log.gz 2024-03-06T07:33:58.527 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70257.log 2024-03-06T07:33:58.527 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58110.log: /var/log/ceph/ceph-client.admin.78850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64117.log 2024-03-06T07:33:58.527 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58110.log.gz 2024-03-06T07:33:58.527 INFO:teuthology.orchestra.run.smithi001.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.66308.log.gz 2024-03-06T07:33:58.527 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70257.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78850.log.gz 2024-03-06T07:33:58.527 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54239.log 2024-03-06T07:33:58.527 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70257.log.gz 2024-03-06T07:33:58.527 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71801.log 2024-03-06T07:33:58.528 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54239.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64117.log.gz 2024-03-06T07:33:58.528 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57206.log 2024-03-06T07:33:58.528 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71801.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54239.log.gz 2024-03-06T07:33:58.528 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65259.log 2024-03-06T07:33:58.528 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57206.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.71801.log.gz 2024-03-06T07:33:58.528 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44554.log 2024-03-06T07:33:58.528 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57206.log.gz 2024-03-06T07:33:58.528 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44000.log 2024-03-06T07:33:58.528 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44554.log: gzip 88.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.65259.log.gz 2024-03-06T07:33:58.529 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55352.log 2024-03-06T07:33:58.529 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44000.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44554.log.gz 2024-03-06T07:33:58.529 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71481.log 2024-03-06T07:33:58.529 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44000.log.gz 2024-03-06T07:33:58.529 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61380.log 2024-03-06T07:33:58.529 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71481.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55352.log.gz 2024-03-06T07:33:58.529 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86307.log 2024-03-06T07:33:58.529 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85094.log 2024-03-06T07:33:58.529 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61380.log.gz 2024-03-06T07:33:58.530 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86307.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.71481.log.gz 2024-03-06T07:33:58.530 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79640.log 2024-03-06T07:33:58.530 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66056.log 2024-03-06T07:33:58.530 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86307.log.gz 2024-03-06T07:33:58.530 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.85094.log.gz 2024-03-06T07:33:58.530 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77398.log 2024-03-06T07:33:58.530 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79640.log.gz 2024-03-06T07:33:58.530 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65177.log 2024-03-06T07:33:58.530 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77398.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66056.log.gzgzip 2024-03-06T07:33:58.531 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51383.log 2024-03-06T07:33:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65177.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.77398.log.gz 2024-03-06T07:33:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87375.log 2024-03-06T07:33:58.531 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65177.log.gz 2024-03-06T07:33:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51779.log 2024-03-06T07:33:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87375.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51383.log.gz 2024-03-06T07:33:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48114.log 2024-03-06T07:33:58.531 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87375.log.gz 2024-03-06T07:33:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44875.log 2024-03-06T07:33:58.532 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51779.log: /var/log/ceph/ceph-client.admin.48114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45974.log 2024-03-06T07:33:58.532 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51779.log.gz 2024-03-06T07:33:58.532 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44875.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48114.log.gz 2024-03-06T07:33:58.532 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61151.log 2024-03-06T07:33:58.532 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44875.log.gz 2024-03-06T07:33:58.532 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63252.log 2024-03-06T07:33:58.532 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76254.log 2024-03-06T07:33:58.532 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45974.log.gz 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61151.log.gz 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73583.log 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8%/var/log/ceph/ceph-client.admin.76254.log: -- replaced with /var/log/ceph/ceph-client.admin.63252.log.gz 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67542.log 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42115.log 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76254.log.gz 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67542.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73583.log.gz 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55080.log 2024-03-06T07:33:58.534 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67542.log.gz/var/log/ceph/ceph-client.admin.42115.log: 2024-03-06T07:33:58.534 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51593.log 2024-03-06T07:33:58.534 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42115.log.gz 2024-03-06T07:33:58.534 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51004.log 2024-03-06T07:33:58.534 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51593.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55080.log.gz 2024-03-06T07:33:58.534 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55730.log 2024-03-06T07:33:58.534 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51593.log.gz 2024-03-06T07:33:58.534 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87333.log 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51004.log.gz 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58561.log 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87333.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.55730.log.gz -5 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.84758.log 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58561.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87333.log.gz 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45512.log 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58561.log.gz 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65608.log 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45512.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.84758.log.gz 2024-03-06T07:33:58.536 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77895.log 2024-03-06T07:33:58.536 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45512.log.gz 2024-03-06T07:33:58.536 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50112.log 2024-03-06T07:33:58.536 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53508.log 88.9% 2024-03-06T07:33:58.536 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65608.log.gz 2024-03-06T07:33:58.536 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53941.log 2024-03-06T07:33:58.536 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77895.log.gz 2024-03-06T07:33:58.536 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50112.log.gz 2024-03-06T07:33:58.536 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81533.log 2024-03-06T07:33:58.537 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53508.log.gz 2024-03-06T07:33:58.537 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86871.log 2024-03-06T07:33:58.537 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81533.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.53941.log.gz -5 2024-03-06T07:33:58.537 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44871.log 2024-03-06T07:33:58.537 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86871.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81533.log.gz 2024-03-06T07:33:58.537 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47175.log 2024-03-06T07:33:58.537 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86871.log.gz 2024-03-06T07:33:58.537 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86727.log 2024-03-06T07:33:58.538 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44871.log.gz 2024-03-06T07:33:58.538 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58687.log 2024-03-06T07:33:58.538 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86727.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47175.log.gz 2024-03-06T07:33:58.538 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63402.log 2024-03-06T07:33:58.538 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86727.log.gz 2024-03-06T07:33:58.538 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49759.log 2024-03-06T07:33:58.538 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58687.log: /var/log/ceph/ceph-client.admin.63402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42338.log 2024-03-06T07:33:58.538 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58687.log.gz 2024-03-06T07:33:58.538 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49759.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63402.log.gz 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90601.log 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49759.log.gz 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69608.log 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90601.log: gzip 88.1% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.42338.log.gz -- 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.68744.log 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.69608.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90601.log.gz 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74278.log 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69608.log.gz 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51088.log 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85826.log 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51088.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74278.log.gz 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi001.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.68744.log.gz 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85910.log 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45428.log 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi001.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.51088.log.gz 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85826.log.gz 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60301.log 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.85910.log.gz 2024-03-06T07:33:58.541 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68175.log 2024-03-06T07:33:58.541 INFO:teuthology.orchestra.run.smithi001.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45428.log.gz 2024-03-06T07:33:58.541 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52459.log 2024-03-06T07:33:58.541 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60301.log.gz 2024-03-06T07:33:58.541 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74110.log 2024-03-06T07:33:58.541 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72194.log 2024-03-06T07:33:58.541 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68175.log.gz 2024-03-06T07:33:58.541 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74110.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52459.log.gz 2024-03-06T07:33:58.542 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57416.log 2024-03-06T07:33:58.542 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74110.log.gz 2024-03-06T07:33:58.542 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42788.log 2024-03-06T07:33:58.542 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64014.log 2024-03-06T07:33:58.542 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72194.log.gz 2024-03-06T07:33:58.542 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57416.log.gz 2024-03-06T07:33:58.542 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67099.log 2024-03-06T07:33:58.542 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42788.log.gz 2024-03-06T07:33:58.543 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72506.log 2024-03-06T07:33:58.543 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80015.log 2024-03-06T07:33:58.543 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67099.log.gz 2024-03-06T07:33:58.543 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62786.log 2024-03-06T07:33:58.543 INFO:teuthology.orchestra.run.smithi001.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.64014.log.gz 2024-03-06T07:33:58.543 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50024.log 2024-03-06T07:33:58.543 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0%/var/log/ceph/ceph-client.admin.80015.log: -- replaced with /var/log/ceph/ceph-client.admin.72506.log.gz 2024-03-06T07:33:58.543 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.62786.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-06T07:33:58.543 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.80015.log.gz 2024-03-06T07:33:58.543 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62786.log.gz 2024-03-06T07:33:58.544 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52980.log 2024-03-06T07:33:58.544 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.1.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50024.log.gz 2024-03-06T07:33:58.544 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86139.log 2024-03-06T07:33:58.544 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89880.log 2024-03-06T07:33:58.544 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86139.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52980.log.gz 2024-03-06T07:33:58.544 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73923.log 2024-03-06T07:33:58.544 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86139.log.gz 2024-03-06T07:33:58.544 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61800.log 2024-03-06T07:33:58.544 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89880.log: /var/log/ceph/ceph-client.admin.73923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50070.log 2024-03-06T07:33:58.545 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89880.log.gz 2024-03-06T07:33:58.545 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61800.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.73923.log.gz 2024-03-06T07:33:58.545 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50517.log 2024-03-06T07:33:58.545 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50070.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61800.log.gz 2024-03-06T07:33:58.545 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80738.log 2024-03-06T07:33:58.545 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50070.log.gz 2024-03-06T07:33:58.545 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68082.log 2024-03-06T07:33:58.545 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50517.log: /var/log/ceph/ceph-client.admin.80738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71313.log 2024-03-06T07:33:58.545 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50517.log.gz 2024-03-06T07:33:58.546 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68082.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80738.log.gz 2024-03-06T07:33:58.546 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62744.log 2024-03-06T07:33:58.546 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76436.log 2024-03-06T07:33:58.546 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71313.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68082.log.gz 2024-03-06T07:33:58.546 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.62744.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.71313.log.gz 2024-03-06T07:33:58.546 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.66721.log 2024-03-06T07:33:58.546 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62744.log.gz 2024-03-06T07:33:58.546 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83563.log 2024-03-06T07:33:58.546 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76436.log: /var/log/ceph/ceph-client.admin.66721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91039.log 2024-03-06T07:33:58.546 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83563.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66721.log.gz 2024-03-06T07:33:58.547 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42680.log 2024-03-06T07:33:58.547 INFO:teuthology.orchestra.run.smithi001.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.76436.log.gz 2024-03-06T07:33:58.547 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83563.log.gz 2024-03-06T07:33:58.547 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.91039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70425.log 2024-03-06T07:33:58.547 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75839.log 2024-03-06T07:33:58.547 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42680.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.91039.log.gz 2024-03-06T07:33:58.547 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61548.log 2024-03-06T07:33:58.547 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42680.log.gz 2024-03-06T07:33:58.547 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70425.log.gz 2024-03-06T07:33:58.547 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44872.log 2024-03-06T07:33:58.548 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75839.log: /var/log/ceph/ceph-client.admin.61548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87207.log 2024-03-06T07:33:58.548 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75839.log.gz 2024-03-06T07:33:58.548 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44872.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61548.log.gz 2024-03-06T07:33:58.548 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85178.log 2024-03-06T07:33:58.548 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44872.log.gz 2024-03-06T07:33:58.548 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45554.log 2024-03-06T07:33:58.548 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64682.log 2024-03-06T07:33:58.548 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85178.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87207.log.gz 2024-03-06T07:33:58.548 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62112.log 2024-03-06T07:33:58.549 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85178.log.gz 2024-03-06T07:33:58.549 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45554.log.gz 2024-03-06T07:33:58.549 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55020.log 2024-03-06T07:33:58.549 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64682.log: /var/log/ceph/ceph-client.admin.62112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58351.log 2024-03-06T07:33:58.549 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64682.log.gz 2024-03-06T07:33:58.549 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55020.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62112.log.gz 2024-03-06T07:33:58.549 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84800.log 2024-03-06T07:33:58.549 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55020.log.gz 2024-03-06T07:33:58.549 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69186.log 2024-03-06T07:33:58.549 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58351.log: /var/log/ceph/ceph-client.admin.84800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54684.log 2024-03-06T07:33:58.550 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58351.log.gz 2024-03-06T07:33:58.550 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.69186.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84800.log.gz 2024-03-06T07:33:58.550 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67057.log 2024-03-06T07:33:58.550 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69186.log.gz 2024-03-06T07:33:58.550 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57672.log 2024-03-06T07:33:58.550 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54684.log: /var/log/ceph/ceph-client.admin.67057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72152.log 2024-03-06T07:33:58.550 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54684.log.gz 2024-03-06T07:33:58.550 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67057.log.gz 2024-03-06T07:33:58.550 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56108.log 2024-03-06T07:33:58.550 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66973.log 2024-03-06T07:33:58.551 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72152.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57672.log.gz 2024-03-06T07:33:58.551 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66434.log 2024-03-06T07:33:58.551 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72152.log.gz 2024-03-06T07:33:58.551 INFO:teuthology.orchestra.run.smithi001.stderr:gzip 89.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.56108.log.gz 2024-03-06T07:33:58.551 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.86181.log 2024-03-06T07:33:58.551 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66973.log: /var/log/ceph/ceph-client.admin.66434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48528.log 2024-03-06T07:33:58.551 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.66973.log.gz 2024-03-06T07:33:58.551 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86181.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66434.log.gz 2024-03-06T07:33:58.552 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83521.log 2024-03-06T07:33:58.552 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48528.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86181.log.gz 2024-03-06T07:33:58.552 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88377.log 2024-03-06T07:33:58.552 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83521.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48528.log.gz 2024-03-06T07:33:58.552 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53804.log 2024-03-06T07:33:58.552 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83521.log.gz 2024-03-06T07:33:58.552 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60959.log 2024-03-06T07:33:58.552 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.88377.log.gz 2024-03-06T07:33:58.553 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84842.log 2024-03-06T07:33:58.553 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53804.log: /var/log/ceph/ceph-client.admin.60959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75713.log 2024-03-06T07:33:58.553 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53804.log.gz 2024-03-06T07:33:58.553 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84842.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60959.log.gz 2024-03-06T07:33:58.553 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54361.log 2024-03-06T07:33:58.553 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 87.9% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.84842.log.gz /var/log/ceph/ceph-client.admin.73362.log 2024-03-06T07:33:58.553 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-06T07:33:58.553 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75713.log: /var/log/ceph/ceph-client.admin.54361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77723.log 2024-03-06T07:33:58.553 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.75713.log.gz 2024-03-06T07:33:58.553 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56267.log 2024-03-06T07:33:58.554 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73362.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54361.log.gz 2024-03-06T07:33:58.554 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77723.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73362.log.gz 2024-03-06T07:33:58.554 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83935.log 2024-03-06T07:33:58.554 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60259.log 2024-03-06T07:33:58.554 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.77723.log.gz 2024-03-06T07:33:58.554 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56267.log: /var/log/ceph/ceph-client.admin.83935.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.56267.log.gz -5 2024-03-06T07:33:58.554 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.83755.log 2024-03-06T07:33:58.554 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90296.log 2024-03-06T07:33:58.554 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60259.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.83935.log.gz 2024-03-06T07:33:58.554 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56954.log 2024-03-06T07:33:58.555 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60259.log.gz 2024-03-06T07:33:58.555 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64934.log 2024-03-06T07:33:58.555 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90296.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83755.log.gz 2024-03-06T07:33:58.555 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56954.log: gzip -5 87.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.90296.log.gz 2024-03-06T07:33:58.555 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.42470.log 2024-03-06T07:33:58.555 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.56954.log.gz 2024-03-06T07:33:58.555 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48444.log 2024-03-06T07:33:58.555 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64934.log: /var/log/ceph/ceph-client.admin.42470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73625.log 2024-03-06T07:33:58.555 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64934.log.gz 2024-03-06T07:33:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63774.log 2024-03-06T07:33:58.556 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42470.log.gz 2024-03-06T07:33:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73625.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48444.log.gz 2024-03-06T07:33:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41771.log 2024-03-06T07:33:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63774.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73625.log.gz 2024-03-06T07:33:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69505.log 2024-03-06T07:33:58.556 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63774.log.gz 2024-03-06T07:33:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43612.log 2024-03-06T07:33:58.557 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41771.log.gz 2024-03-06T07:33:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81299.log 2024-03-06T07:33:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.69505.log: /var/log/ceph/ceph-client.admin.43612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44126.log 2024-03-06T07:33:58.557 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43612.log.gz/var/log/ceph/ceph-client.admin.81299.log: 2024-03-06T07:33:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88335.log 2024-03-06T07:33:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44126.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.69505.log.gz 2024-03-06T07:33:58.557 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81299.log.gz 2024-03-06T07:33:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69120.log 2024-03-06T07:33:58.557 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44126.log.gz 2024-03-06T07:33:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67455.log 2024-03-06T07:33:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.69120.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.88335.log.gz 2024-03-06T07:33:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57098.log 2024-03-06T07:33:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89609.log 2024-03-06T07:33:58.558 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67455.log.gz 2024-03-06T07:33:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57098.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.69120.log.gz 2024-03-06T07:33:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51719.log 2024-03-06T07:33:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86829.log 2024-03-06T07:33:58.558 INFO:teuthology.orchestra.run.smithi001.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.57098.log.gz 2024-03-06T07:33:58.559 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51719.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.89609.log.gz 2024-03-06T07:33:58.559 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63360.log 2024-03-06T07:33:58.559 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51719.log.gz 2024-03-06T07:33:58.559 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73751.log 2024-03-06T07:33:58.559 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.63360.log: -- /var/log/ceph/ceph-client.admin.64850.log 2024-03-06T07:33:58.559 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86829.log.gz 2024-03-06T07:33:58.559 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73751.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63360.log.gz 2024-03-06T07:33:58.559 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74362.log 2024-03-06T07:33:58.559 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73797.log 2024-03-06T07:33:58.559 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64850.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.73751.log.gz 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74362.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64850.log.gz 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78892.log 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84716.log 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74362.log.gz 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73797.log: /var/log/ceph/ceph-client.admin.78892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82405.log 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73797.log.gz 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86054.log 2024-03-06T07:33:58.560 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2%/var/log/ceph/ceph-client.admin.84716.log: -- replaced with /var/log/ceph/ceph-client.admin.78892.log.gz 2024-03-06T07:33:58.561 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82405.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.84716.log.gz 2024-03-06T07:33:58.561 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48240.log 2024-03-06T07:33:58.561 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50406.log 2024-03-06T07:33:58.561 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.82405.log.gz 2024-03-06T07:33:58.561 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86054.log: /var/log/ceph/ceph-client.admin.48240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58813.log 2024-03-06T07:33:58.561 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86054.log.gz 2024-03-06T07:33:58.561 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82195.log 2024-03-06T07:33:58.561 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50406.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48240.log.gz 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43754.log 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50406.log.gz 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63679.log 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82195.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58813.log.gz 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43754.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82195.log.gz 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46755.log 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47217.log 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43754.log.gz 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63679.log: /var/log/ceph/ceph-client.admin.46755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82279.log 2024-03-06T07:33:58.562 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63679.log.gz 2024-03-06T07:33:58.563 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72942.log 2024-03-06T07:33:58.563 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47217.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46755.log.gz 2024-03-06T07:33:58.563 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82279.log: gzip -5 87.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.47217.log.gz 2024-03-06T07:33:58.563 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.86433.log 2024-03-06T07:33:58.563 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82279.log.gzgzip 2024-03-06T07:33:58.563 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose --/var/log/ceph/ceph-client.admin.72942.log: /var/log/ceph/ceph-client.admin.49299.log 2024-03-06T07:33:58.563 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86433.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72942.log.gz 2024-03-06T07:33:58.563 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57248.log 2024-03-06T07:33:58.563 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76833.log 2024-03-06T07:33:58.564 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.86433.log.gz 2024-03-06T07:33:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49299.log: /var/log/ceph/ceph-client.admin.57248.log: gzip 88.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.49299.log.gz --verbose 2024-03-06T07:33:58.564 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.76581.log 2024-03-06T07:33:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61590.log 2024-03-06T07:33:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76833.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57248.log.gz 2024-03-06T07:33:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76581.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76833.log.gz 2024-03-06T07:33:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54768.log 2024-03-06T07:33:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86265.log 2024-03-06T07:33:58.564 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76581.log.gz 2024-03-06T07:33:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61590.log: /var/log/ceph/ceph-client.admin.54768.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.61590.log.gz --verbose 2024-03-06T07:33:58.565 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.67412.log 2024-03-06T07:33:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48906.log 2024-03-06T07:33:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86265.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54768.log.gz 2024-03-06T07:33:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87165.log 2024-03-06T07:33:58.565 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86265.log.gz 2024-03-06T07:33:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58309.log 2024-03-06T07:33:58.565 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67412.log.gz 2024-03-06T07:33:58.566 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48906.log: /var/log/ceph/ceph-client.admin.87165.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48906.log.gz 2024-03-06T07:33:58.566 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65515.log 2024-03-06T07:33:58.566 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87897.log 2024-03-06T07:33:58.566 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58309.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.87165.log.gz 2024-03-06T07:33:58.566 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77272.log 87.5% 2024-03-06T07:33:58.566 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58309.log.gz 2024-03-06T07:33:58.566 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65515.log.gz 2024-03-06T07:33:58.566 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65385.log 2024-03-06T07:33:58.566 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87897.log.gz 2024-03-06T07:33:58.566 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82657.log 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65385.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77272.log.gz 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55856.log 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65385.log.gz 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75563.log 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55814.log 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55856.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82657.log.gz 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87981.log 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55856.log.gz 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90388.log 2024-03-06T07:33:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55814.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.75563.log.gz 2024-03-06T07:33:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87981.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55814.log.gz 2024-03-06T07:33:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71397.log 2024-03-06T07:33:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49135.log 2024-03-06T07:33:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90388.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87981.log.gz 2024-03-06T07:33:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84151.log 2024-03-06T07:33:58.568 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.90388.log.gz 2024-03-06T07:33:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47343.log 2024-03-06T07:33:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49135.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71397.log.gz 2024-03-06T07:33:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84151.log: gzip -5 88.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.49135.log.gz 2024-03-06T07:33:58.568 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.77104.log 2024-03-06T07:33:58.569 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56698.log 2024-03-06T07:33:58.569 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47343.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.84151.log.gz 2024-03-06T07:33:58.569 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77104.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47343.log.gz 2024-03-06T07:33:58.569 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48156.log 2024-03-06T07:33:58.569 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82573.log 2024-03-06T07:33:58.569 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56698.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77104.log.gz 2024-03-06T07:33:58.569 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48156.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56698.log.gz 2024-03-06T07:33:58.569 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63732.log 2024-03-06T07:33:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83437.log 2024-03-06T07:33:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82573.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48156.log.gz 2024-03-06T07:33:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63732.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 88.8% 2024-03-06T07:33:58.570 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82573.log.gz 2024-03-06T07:33:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76959.log 2024-03-06T07:33:58.570 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63732.log.gz 2024-03-06T07:33:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83437.log: /var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80426.log 2024-03-06T07:33:58.570 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83437.log.gz 2024-03-06T07:33:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80963.log 2024-03-06T07:33:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76959.log: /var/log/ceph/ceph-client.admin.80426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46058.log 2024-03-06T07:33:58.571 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76959.log.gz 2024-03-06T07:33:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80963.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80426.log.gz 2024-03-06T07:33:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54978.log 2024-03-06T07:33:58.571 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.80963.log.gz 2024-03-06T07:33:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72548.log 2024-03-06T07:33:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54978.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46058.log.gz 2024-03-06T07:33:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72464.log 2024-03-06T07:33:58.571 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54978.log.gz 2024-03-06T07:33:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61758.log 2024-03-06T07:33:58.572 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72548.log.gz 2024-03-06T07:33:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48654.log 2024-03-06T07:33:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87729.log/var/log/ceph/ceph-client.admin.61758.log: 2024-03-06T07:33:58.572 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72464.log.gz 2024-03-06T07:33:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77639.log 2024-03-06T07:33:58.572 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61758.log.gz 2024-03-06T07:33:58.572 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48654.log.gz 2024-03-06T07:33:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45764.log 2024-03-06T07:33:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89277.log 2024-03-06T07:33:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77639.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.87729.log.gz 2024-03-06T07:33:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88191.log 2024-03-06T07:33:58.573 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77639.log.gz 2024-03-06T07:33:58.573 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7%/var/log/ceph/ceph-client.admin.89277.log: -- replaced with /var/log/ceph/ceph-client.admin.45764.log.gz 2024-03-06T07:33:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68038.log 2024-03-06T07:33:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88191.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51822.log 2024-03-06T07:33:58.573 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89277.log.gz 2024-03-06T07:33:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68038.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.88191.log.gz 2024-03-06T07:33:58.573 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58008.log 2024-03-06T07:33:58.573 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68038.log.gz 2024-03-06T07:33:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89567.log 2024-03-06T07:33:58.574 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77477.log 2024-03-06T07:33:58.574 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58008.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51822.log.gz 2024-03-06T07:33:58.574 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89567.log: gzip -5 --verbose 87.3% -- /var/log/ceph/ceph-client.admin.60431.log 2024-03-06T07:33:58.574 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58008.log.gz 2024-03-06T07:33:58.574 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.89567.log.gz 2024-03-06T07:33:58.574 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79219.log 2024-03-06T07:33:58.574 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77477.log.gz 2024-03-06T07:33:58.574 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55268.log 2024-03-06T07:33:58.574 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60431.log: /var/log/ceph/ceph-client.admin.79219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75353.log 2024-03-06T07:33:58.574 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60431.log.gz 2024-03-06T07:33:58.575 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.79219.log.gz 2024-03-06T07:33:58.575 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47007.log 2024-03-06T07:33:58.575 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75353.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55268.log.gz 2024-03-06T07:33:58.575 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84361.log 2024-03-06T07:33:58.575 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75353.log.gz 2024-03-06T07:33:58.575 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54810.log 2024-03-06T07:33:58.575 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46270.log 2024-03-06T07:33:58.575 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47007.log.gz 2024-03-06T07:33:58.575 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84361.log: /var/log/ceph/ceph-client.admin.54810.log: gzip -5 --verbose -- 89.1% -- replaced with /var/log/ceph/ceph-client.admin.84361.log.gz 2024-03-06T07:33:58.576 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.78580.log 2024-03-06T07:33:58.576 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54810.log.gz 2024-03-06T07:33:58.576 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89483.log 2024-03-06T07:33:58.576 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46270.log.gz 2024-03-06T07:33:58.576 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74874.log 2024-03-06T07:33:58.576 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.78580.log.gz 2024-03-06T07:33:58.576 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76348.log 2024-03-06T07:33:58.576 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.89483.log.gz 2024-03-06T07:33:58.576 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49982.log 2024-03-06T07:33:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74874.log: /var/log/ceph/ceph-client.admin.76348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64327.log 2024-03-06T07:33:58.577 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74874.log.gz 2024-03-06T07:33:58.577 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76348.log.gz 2024-03-06T07:33:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76194.log 2024-03-06T07:33:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64327.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49982.log.gz 2024-03-06T07:33:58.577 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.77356.log 2024-03-06T07:33:58.577 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64327.log.gz 2024-03-06T07:33:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73839.log 2024-03-06T07:33:58.577 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76194.log.gz 2024-03-06T07:33:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79261.log 2024-03-06T07:33:58.578 INFO:teuthology.orchestra.run.smithi001.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.77356.log.gz 2024-03-06T07:33:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53190.log 2024-03-06T07:33:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85952.log 2024-03-06T07:33:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79261.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73839.log.gz 2024-03-06T07:33:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82363.log 2024-03-06T07:33:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85952.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53190.log.gz 2024-03-06T07:33:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:gzip 91.1% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.79261.log.gz -- 2024-03-06T07:33:58.578 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.73110.log 2024-03-06T07:33:58.578 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.85952.log.gz 2024-03-06T07:33:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83307.log 2024-03-06T07:33:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73110.log: gzip -5 --verbose -- 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82363.log.gz 2024-03-06T07:33:58.579 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.68263.log 2024-03-06T07:33:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83307.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.73110.log.gz 2024-03-06T07:33:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73152.log 2024-03-06T07:33:58.579 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83307.log.gz 2024-03-06T07:33:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85406.log 2024-03-06T07:33:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66374.log 2024-03-06T07:33:58.579 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68263.log.gz 2024-03-06T07:33:58.579 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73152.log.gz 2024-03-06T07:33:58.580 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77188.log 2024-03-06T07:33:58.580 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90559.log 2024-03-06T07:33:58.580 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85406.log.gz 2024-03-06T07:33:58.580 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66374.log.gz 2024-03-06T07:33:58.580 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45932.log 2024-03-06T07:33:58.580 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.90559.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51635.log 2024-03-06T07:33:58.580 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.77188.log.gz 2024-03-06T07:33:58.580 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45932.log: gzip -5 --verbose 88.3% -- replaced with /var/log/ceph/ceph-client.admin.90559.log.gz 2024-03-06T07:33:58.580 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.43444.log 2024-03-06T07:33:58.580 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45932.log.gz 2024-03-06T07:33:58.581 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90516.log 2024-03-06T07:33:58.581 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51635.log.gz 2024-03-06T07:33:58.581 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74320.log 2024-03-06T07:33:58.581 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43444.log.gz 2024-03-06T07:33:58.581 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70299.log 2024-03-06T07:33:58.581 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74320.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90516.log.gz 2024-03-06T07:33:58.581 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80634.log 2024-03-06T07:33:58.581 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74320.log.gz 2024-03-06T07:33:58.581 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68217.log 2024-03-06T07:33:58.582 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80634.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48864.log 2024-03-06T07:33:58.582 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70299.log.gz 2024-03-06T07:33:58.582 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68217.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80634.log.gz 2024-03-06T07:33:58.582 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52158.log 2024-03-06T07:33:58.582 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48864.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59267.log 2024-03-06T07:33:58.582 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68217.log.gz 2024-03-06T07:33:58.582 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72068.log 2024-03-06T07:33:58.582 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48864.log.gz 2024-03-06T07:33:58.582 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52158.log.gz 2024-03-06T07:33:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43796.log 2024-03-06T07:33:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72068.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59267.log.gz 2024-03-06T07:33:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53064.log 2024-03-06T07:33:58.583 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.72068.log.gz 2024-03-06T07:33:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59850.log 2024-03-06T07:33:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68894.log 2024-03-06T07:33:58.583 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43796.log.gz 2024-03-06T07:33:58.583 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53064.log.gz 2024-03-06T07:33:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48318.log 2024-03-06T07:33:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43570.log 2024-03-06T07:33:58.584 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68894.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59850.log.gz 2024-03-06T07:33:58.584 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76394.log 2024-03-06T07:33:58.584 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68894.log.gz 2024-03-06T07:33:58.584 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43570.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48318.log.gz 2024-03-06T07:33:58.584 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90474.log 2024-03-06T07:33:58.584 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43570.log.gz 2024-03-06T07:33:58.584 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65097.log 2024-03-06T07:33:58.584 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53148.log 2024-03-06T07:33:58.585 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90474.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76394.log.gz 2024-03-06T07:33:58.585 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89235.log 2024-03-06T07:33:58.585 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90474.log.gz 2024-03-06T07:33:58.585 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79973.log 2024-03-06T07:33:58.585 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65097.log.gz 2024-03-06T07:33:58.585 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89235.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53148.log.gz 2024-03-06T07:33:58.585 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46715.log 2024-03-06T07:33:58.585 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75965.log 2024-03-06T07:33:58.585 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89235.log.gz 2024-03-06T07:33:58.586 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79973.log: /var/log/ceph/ceph-client.admin.46715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83349.log 2024-03-06T07:33:58.586 INFO:teuthology.orchestra.run.smithi001.stderr: 87.2% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46715.log.gz 2024-03-06T07:33:58.586 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.79973.log.gz 2024-03-06T07:33:58.586 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90211.log 2024-03-06T07:33:58.586 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42073.log 2024-03-06T07:33:58.586 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83349.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.75965.log.gz 2024-03-06T07:33:58.586 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53760.log 2024-03-06T07:33:58.586 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83349.log.gz 2024-03-06T07:33:58.586 INFO:teuthology.orchestra.run.smithi001.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.90211.log.gz 2024-03-06T07:33:58.587 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52724.log 2024-03-06T07:33:58.587 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44557.log 2024-03-06T07:33:58.587 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42073.log.gz 2024-03-06T07:33:58.587 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74648.log 2024-03-06T07:33:58.587 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53760.log.gz 2024-03-06T07:33:58.587 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52724.log.gz 2024-03-06T07:33:58.587 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46839.log 2024-03-06T07:33:58.587 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78244.log 2024-03-06T07:33:58.587 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74648.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44557.log.gz 2024-03-06T07:33:58.587 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71589.log 2024-03-06T07:33:58.588 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74648.log.gz 2024-03-06T07:33:58.588 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46839.log.gz 2024-03-06T07:33:58.588 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90127.log 2024-03-06T07:33:58.588 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84926.log 2024-03-06T07:33:58.588 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71589.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.78244.log.gz 2024-03-06T07:33:58.588 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45890.log 2024-03-06T07:33:58.588 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71589.log.gz 2024-03-06T07:33:58.588 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90127.log.gz 2024-03-06T07:33:58.588 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86601.log 2024-03-06T07:33:58.589 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79556.log 2024-03-06T07:33:58.589 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45890.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84926.log.gz 2024-03-06T07:33:58.589 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64640.log 2024-03-06T07:33:58.589 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45890.log.gz 2024-03-06T07:33:58.589 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79556.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86601.log.gz 2024-03-06T07:33:58.589 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60013.log 2024-03-06T07:33:58.589 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64640.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.79556.log.gz 2024-03-06T07:33:58.589 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61193.log 2024-03-06T07:33:58.589 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60013.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64640.log.gz 2024-03-06T07:33:58.589 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62070.log 2024-03-06T07:33:58.590 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60013.log.gz 2024-03-06T07:33:58.590 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75139.log 2024-03-06T07:33:58.590 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.62070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64369.log 2024-03-06T07:33:58.590 INFO:teuthology.orchestra.run.smithi001.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.61193.log.gz 2024-03-06T07:33:58.590 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75139.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62070.log.gz 2024-03-06T07:33:58.590 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49845.log 2024-03-06T07:33:58.590 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75139.log.gz 2024-03-06T07:33:58.590 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46797.log 2024-03-06T07:33:58.590 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64369.log.gz 2024-03-06T07:33:58.590 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65301.log 2024-03-06T07:33:58.591 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49845.log.gz 2024-03-06T07:33:58.591 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79153.log 2024-03-06T07:33:58.591 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65301.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46797.log.gz 2024-03-06T07:33:58.591 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59556.log 2024-03-06T07:33:58.591 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65301.log.gz 2024-03-06T07:33:58.591 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76665.log 2024-03-06T07:33:58.591 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44294.log 2024-03-06T07:33:58.591 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76665.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.79153.log.gz 88.2% 2024-03-06T07:33:58.591 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59556.log.gz 2024-03-06T07:33:58.591 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82237.log 88.0% 2024-03-06T07:33:58.592 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76665.log.gz 2024-03-06T07:33:58.592 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62313.log 2024-03-06T07:33:58.592 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82237.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44294.log.gz 2024-03-06T07:33:58.592 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54600.log 2024-03-06T07:33:58.592 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.62313.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.82237.log.gz 2024-03-06T07:33:58.592 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70383.log 2024-03-06T07:33:58.592 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62313.log.gz 2024-03-06T07:33:58.592 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69962.log 2024-03-06T07:33:58.592 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54600.log.gz 2024-03-06T07:33:58.593 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48738.log 2024-03-06T07:33:58.593 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70383.log: /var/log/ceph/ceph-client.admin.69962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89025.log 2024-03-06T07:33:58.593 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70383.log.gz 2024-03-06T07:33:58.593 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69962.log.gz 2024-03-06T07:33:58.593 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57882.log 2024-03-06T07:33:58.593 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63613.log 2024-03-06T07:33:58.593 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89025.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48738.log.gz 2024-03-06T07:33:58.593 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72712.log 2024-03-06T07:33:58.593 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.89025.log.gz 2024-03-06T07:33:58.593 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63613.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57882.log.gz 2024-03-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72320.log 2024-03-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72712.log: /var/log/ceph/ceph-mgr.y.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.63613.log.gz 2024-03-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81901.log 2024-03-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84001.log 2024-03-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72712.log.gz 2024-03-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72320.log.gz 2024-03-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83689.log 2024-03-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81901.log: /var/log/ceph/ceph-client.admin.84001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79430.log 2024-03-06T07:33:58.595 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81901.log.gz 2024-03-06T07:33:58.595 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64766.log 2024-03-06T07:33:58.595 INFO:teuthology.orchestra.run.smithi001.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.84001.log.gz/var/log/ceph/ceph-client.admin.79430.log: 2024-03-06T07:33:58.595 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53592.log 2024-03-06T07:33:58.595 INFO:teuthology.orchestra.run.smithi001.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.83689.log.gz 2024-03-06T07:33:58.595 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79430.log.gz 2024-03-06T07:33:58.595 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80141.log 2024-03-06T07:33:58.595 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64766.log: /var/log/ceph/ceph-client.admin.53592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70962.log 2024-03-06T07:33:58.595 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64766.log.gz 2024-03-06T07:33:58.595 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80141.log: gzip -5 --verbose -- 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53592.log.gz 2024-03-06T07:33:58.596 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.45354.log 2024-03-06T07:33:58.596 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70962.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80141.log.gz 2024-03-06T07:33:58.596 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70962.log.gz 2024-03-06T07:33:58.596 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.62563.log 2024-03-06T07:33:58.596 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53232.log 2024-03-06T07:33:58.596 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.62563.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45354.log.gz 2024-03-06T07:33:58.596 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42287.log 2024-03-06T07:33:58.596 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62563.log.gz 2024-03-06T07:33:58.596 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65696.log 2024-03-06T07:33:58.597 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.42287.log: -5 87.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.53232.log.gz /var/log/ceph/ceph-client.admin.42596.log 2024-03-06T07:33:58.597 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-06T07:33:58.597 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65696.log: gzip -5 --verbose 88.0% -- /var/log/ceph/ceph-client.admin.58927.log 2024-03-06T07:33:58.597 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42287.log.gz 2024-03-06T07:33:58.597 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42596.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65696.log.gz 2024-03-06T07:33:58.597 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87603.log 2024-03-06T07:33:58.597 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42596.log.gz 2024-03-06T07:33:58.597 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82778.log 2024-03-06T07:33:58.597 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64808.log 2024-03-06T07:33:58.598 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8%/var/log/ceph/ceph-client.admin.87603.log: -- replaced with /var/log/ceph/ceph-client.admin.58927.log.gz 2024-03-06T07:33:58.598 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81257.log 2024-03-06T07:33:58.598 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47049.log 2024-03-06T07:33:58.598 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82778.log.gz 2024-03-06T07:33:58.598 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87603.log.gz 2024-03-06T07:33:58.598 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81257.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64808.log.gz 2024-03-06T07:33:58.598 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.66182.log 2024-03-06T07:33:58.598 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81257.log.gz 2024-03-06T07:33:58.598 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61506.log 2024-03-06T07:33:58.598 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47049.log.gz 2024-03-06T07:33:58.599 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55898.log 2024-03-06T07:33:58.599 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42199.log 2024-03-06T07:33:58.599 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55898.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61506.log.gz 2024-03-06T07:33:58.599 INFO:teuthology.orchestra.run.smithi001.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.66182.log.gz 2024-03-06T07:33:58.599 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71757.log 2024-03-06T07:33:58.599 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55898.log.gz 2024-03-06T07:33:58.599 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53274.log 2024-03-06T07:33:58.599 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42199.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.71757.log: -- /var/log/ceph/ceph-client.admin.89151.log 2024-03-06T07:33:58.599 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42199.log.gz/var/log/ceph/ceph-client.admin.53274.log: 2024-03-06T07:33:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67141.log 2024-03-06T07:33:58.600 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71757.log.gz 2024-03-06T07:33:58.600 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53274.log.gz 2024-03-06T07:33:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53466.log 2024-03-06T07:33:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89151.log: /var/log/ceph/ceph-client.admin.67141.log: gzip -5 87.7% --verbose -- /var/log/ceph/ceph-client.admin.70792.log 2024-03-06T07:33:58.600 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89151.log.gz 2024-03-06T07:33:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53466.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44168.log 2024-03-06T07:33:58.600 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67141.log.gz 2024-03-06T07:33:58.600 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53466.log.gz 2024-03-06T07:33:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81738.log 2024-03-06T07:33:58.601 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44168.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69463.log 2024-03-06T07:33:58.601 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70792.log.gz 2024-03-06T07:33:58.601 INFO:teuthology.orchestra.run.smithi001.stderr: 94.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-06T07:33:58.601 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81738.log: 87.9%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.44168.log.gz 2024-03-06T07:33:58.601 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.63084.log 2024-03-06T07:33:58.601 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81738.log.gz 2024-03-06T07:33:58.601 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.69463.log: -- /var/log/ceph/ceph-client.admin.58230.log 2024-03-06T07:33:58.601 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69463.log.gz 2024-03-06T07:33:58.601 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44874.log 2024-03-06T07:33:58.601 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64537.log 2024-03-06T07:33:58.602 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58230.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63084.log.gz 2024-03-06T07:33:58.602 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71439.log 2024-03-06T07:33:58.602 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58230.log.gz 2024-03-06T07:33:58.602 INFO:teuthology.orchestra.run.smithi001.stderr:gzip 87.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.44874.log.gz --verbose 2024-03-06T07:33:58.602 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.74446.log 2024-03-06T07:33:58.602 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64537.log: /var/log/ceph/ceph-client.admin.71439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75521.log 2024-03-06T07:33:58.602 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64537.log.gz 2024-03-06T07:33:58.602 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74446.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71439.log.gz 2024-03-06T07:33:58.602 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73488.log 2024-03-06T07:33:58.602 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74446.log.gz 2024-03-06T07:33:58.603 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82615.log 2024-03-06T07:33:58.603 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65864.log 2024-03-06T07:33:58.603 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73488.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75521.log.gz 2024-03-06T07:33:58.603 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61043.log 2024-03-06T07:33:58.603 INFO:teuthology.orchestra.run.smithi001.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.73488.log.gz 2024-03-06T07:33:58.603 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82615.log.gz 2024-03-06T07:33:58.603 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56066.log 2024-03-06T07:33:58.603 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.61043.log: -5 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65864.log.gz 2024-03-06T07:33:58.603 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.59496.log 2024-03-06T07:33:58.603 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56066.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61043.log.gz 2024-03-06T07:33:58.604 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45188.log 2024-03-06T07:33:58.604 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56066.log.gz 2024-03-06T07:33:58.604 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86349.log 2024-03-06T07:33:58.604 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49380.log 2024-03-06T07:33:58.604 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59496.log.gz 2024-03-06T07:33:58.604 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86349.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45188.log.gz 2024-03-06T07:33:58.604 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83265.log 2024-03-06T07:33:58.604 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49380.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86349.log.gz 2024-03-06T07:33:58.605 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53382.log 2024-03-06T07:33:58.605 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49380.log.gz 2024-03-06T07:33:58.605 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51551.log 2024-03-06T07:33:58.605 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83265.log.gz 2024-03-06T07:33:58.605 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78935.log 2024-03-06T07:33:58.605 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61001.log 2024-03-06T07:33:58.605 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53382.log.gz 2024-03-06T07:33:58.605 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51551.log.gz 2024-03-06T07:33:58.605 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.78935.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.91121.log 2024-03-06T07:33:58.606 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61001.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78935.log.gz 2024-03-06T07:33:58.606 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57374.log 2024-03-06T07:33:58.606 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61001.log.gz 2024-03-06T07:33:58.606 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.91121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56912.log 2024-03-06T07:33:58.606 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.91121.log.gz 2024-03-06T07:33:58.606 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50238.log 2024-03-06T07:33:58.606 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50364.log 2024-03-06T07:33:58.606 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57374.log.gz 2024-03-06T07:33:58.606 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56912.log: /var/log/ceph/ceph-client.admin.50238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44336.log 2024-03-06T07:33:58.606 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56912.log.gz 2024-03-06T07:33:58.607 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50238.log.gz 2024-03-06T07:33:58.607 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56187.log 2024-03-06T07:33:58.607 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77146.log 2024-03-06T07:33:58.607 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44336.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50364.log.gz 2024-03-06T07:33:58.607 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75755.log 2024-03-06T07:33:58.607 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44336.log.gz 2024-03-06T07:33:58.607 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77146.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.56187.log.gz 2024-03-06T07:33:58.607 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.85784.log 2024-03-06T07:33:58.607 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.77146.log.gz 2024-03-06T07:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56517.log 2024-03-06T07:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67910.log 2024-03-06T07:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85784.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75755.log.gz 2024-03-06T07:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87249.log 2024-03-06T07:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85784.log.gz 2024-03-06T07:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5%/var/log/ceph/ceph-client.admin.67910.log: -- replaced with /var/log/ceph/ceph-client.admin.56517.log.gz 2024-03-06T07:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45470.log 2024-03-06T07:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87249.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67910.log.gz 2024-03-06T07:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82111.log 2024-03-06T07:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87249.log.gz 2024-03-06T07:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47664.log 2024-03-06T07:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83152.log 2024-03-06T07:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45470.log.gz 2024-03-06T07:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44550.log 2024-03-06T07:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82111.log.gz 2024-03-06T07:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47664.log.gz 2024-03-06T07:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58172.log 2024-03-06T07:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83152.log: /var/log/ceph/ceph-client.admin.44550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45192.log 88.3% 2024-03-06T07:33:58.610 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.83152.log.gz 2024-03-06T07:33:58.610 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68491.log 2024-03-06T07:33:58.610 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44550.log.gz 2024-03-06T07:33:58.610 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45192.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-06T07:33:58.610 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58172.log.gz 2024-03-06T07:33:58.610 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45192.log.gz 2024-03-06T07:33:58.610 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73404.log 2024-03-06T07:33:58.610 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42157.log 2024-03-06T07:33:58.610 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.0.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68491.log.gz 2024-03-06T07:33:58.611 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73541.log 2024-03-06T07:33:58.611 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73404.log.gz 2024-03-06T07:33:58.611 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59640.log 2024-03-06T07:33:58.611 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.73541.log: -5 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42157.log.gz 2024-03-06T07:33:58.611 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.62233.log 2024-03-06T07:33:58.611 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59640.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73541.log.gz 2024-03-06T07:33:58.611 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65948.log 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59640.log.gz 2024-03-06T07:33:58.611 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-06T07:33:58.611 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.62233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47091.log 2024-03-06T07:33:58.612 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62233.log.gz 2024-03-06T07:33:58.612 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54071.log 2024-03-06T07:33:58.612 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1%/var/log/ceph/ceph-client.admin.47091.log: -- replaced with /var/log/ceph/ceph-client.admin.65948.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77681.log 2024-03-06T07:33:58.612 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-06T07:33:58.612 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47091.log.gz 2024-03-06T07:33:58.612 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69692.log 2024-03-06T07:33:58.612 INFO:teuthology.orchestra.run.smithi001.stderr:gzip 87.5% -5 --verbose -- /var/log/ceph/ceph-client.admin.83893.log 2024-03-06T07:33:58.612 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54071.log.gz 2024-03-06T07:33:58.612 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77681.log: /var/log/ceph/ceph-client.admin.69692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60515.log 2024-03-06T07:33:58.613 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77681.log.gz 2024-03-06T07:33:58.613 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61464.log 2024-03-06T07:33:58.613 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69692.log.gz 2024-03-06T07:33:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83893.log: /var/log/ceph/ceph-client.admin.60515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72380.log 2024-03-06T07:33:58.613 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83893.log.gz/var/log/ceph/ceph-client.admin.61464.log: 2024-03-06T07:33:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86955.log 2024-03-06T07:33:58.613 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60515.log.gz 2024-03-06T07:33:58.613 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61464.log.gz 2024-03-06T07:33:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83066.log 2024-03-06T07:33:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72380.log: /var/log/ceph/ceph-client.admin.86955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60893.log 2024-03-06T07:33:58.614 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83066.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86955.log.gz 2024-03-06T07:33:58.614 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81617.log 2024-03-06T07:33:58.614 INFO:teuthology.orchestra.run.smithi001.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.72380.log.gz 2024-03-06T07:33:58.614 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73446.log 2024-03-06T07:33:58.614 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83066.log.gz 2024-03-06T07:33:58.614 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60387.log 2024-03-06T07:33:58.614 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81173.log 2024-03-06T07:33:58.614 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% 90.8% -- replaced with /var/log/ceph/ceph-client.admin.60893.log.gz 2024-03-06T07:33:58.614 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81617.log.gz 2024-03-06T07:33:58.614 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73446.log: /var/log/ceph/ceph-client.admin.60387.log: gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.73446.log.gz 2024-03-06T07:33:58.615 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- 88.2% /var/log/ceph/ceph-client.admin.67744.log 2024-03-06T07:33:58.615 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60387.log.gz/var/log/ceph/ceph-client.admin.81173.log: 2024-03-06T07:33:58.615 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61884.log 2024-03-06T07:33:58.615 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81173.log.gz 2024-03-06T07:33:58.615 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74194.log 2024-03-06T07:33:58.615 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61884.log: gzip -5 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67744.log.gz 2024-03-06T07:33:58.615 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.77001.log 2024-03-06T07:33:58.615 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74194.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61884.log.gz 2024-03-06T07:33:58.616 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75269.log 2024-03-06T07:33:58.616 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.74194.log.gz 2024-03-06T07:33:58.616 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74730.log 2024-03-06T07:33:58.616 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1%/var/log/ceph/ceph-client.admin.75269.log: -- replaced with /var/log/ceph/ceph-client.admin.77001.log.gz 2024-03-06T07:33:58.616 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77557.log 2024-03-06T07:33:58.616 INFO:teuthology.orchestra.run.smithi001.stderr:gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75269.log.gz/var/log/ceph/ceph-client.admin.74730.log: 2024-03-06T07:33:58.616 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61632.log 2024-03-06T07:33:58.616 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77557.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74730.log.gz 2024-03-06T07:33:58.616 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- 88.1% /var/log/ceph/ceph-client.admin.75185.log 2024-03-06T07:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77557.log.gz 2024-03-06T07:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85010.log 2024-03-06T07:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61632.log.gz 2024-03-06T07:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.75185.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.87081.log 2024-03-06T07:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85010.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75185.log.gz 2024-03-06T07:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68978.log 2024-03-06T07:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85010.log.gz 2024-03-06T07:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58603.log 2024-03-06T07:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68978.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87081.log.gz 2024-03-06T07:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58729.log 2024-03-06T07:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43958.log 2024-03-06T07:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr: 91.1% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58603.log.gz 2024-03-06T07:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68978.log.gz 2024-03-06T07:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81943.log 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58729.log.gz 2024-03-06T07:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-06T07:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43958.log: gzip -5 --verbose 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43958.log.gz 2024-03-06T07:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.43880.log 2024-03-06T07:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48780.log 2024-03-06T07:33:58.619 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81943.log.gz 2024-03-06T07:33:58.619 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53899.log 2024-03-06T07:33:58.619 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43880.log.gz 2024-03-06T07:33:58.619 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55982.log 2024-03-06T07:33:58.619 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53899.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48780.log.gz 2024-03-06T07:33:58.619 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43528.log 2024-03-06T07:33:58.619 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53899.log.gz 2024-03-06T07:33:58.619 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72236.log 2024-03-06T07:33:58.619 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55982.log.gz 2024-03-06T07:33:58.620 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.43528.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.57924.log 2024-03-06T07:33:58.620 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72236.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43528.log.gz 2024-03-06T07:33:58.620 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.78520.log 2024-03-06T07:33:58.620 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72236.log.gz 2024-03-06T07:33:58.620 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65018.log 2024-03-06T07:33:58.620 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57924.log.gz 2024-03-06T07:33:58.620 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.78520.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.67328.log 2024-03-06T07:33:58.620 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65018.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78520.log.gz 2024-03-06T07:33:58.620 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52237.log 2024-03-06T07:33:58.621 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67328.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65018.log.gz 2024-03-06T07:33:58.621 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70173.log 2024-03-06T07:33:58.621 INFO:teuthology.orchestra.run.smithi001.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.67328.log.gz 2024-03-06T07:33:58.621 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.52237.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.59766.log 2024-03-06T07:33:58.621 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70173.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52237.log.gz 2024-03-06T07:33:58.621 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51467.log 2024-03-06T07:33:58.621 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70173.log.gz 2024-03-06T07:33:58.621 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80225.log 2024-03-06T07:33:58.622 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59766.log.gz 2024-03-06T07:33:58.622 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.51467.log: -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:33:58.622 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80225.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51467.log.gz 2024-03-06T07:33:58.622 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72590.log 2024-03-06T07:33:58.622 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80225.log.gz 2024-03-06T07:33:58.622 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60809.log 2024-03-06T07:33:58.622 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64411.log 2024-03-06T07:33:58.622 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72590.log.gz 2024-03-06T07:33:58.622 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50896.log 2024-03-06T07:33:58.622 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60809.log.gz 2024-03-06T07:33:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64411.log: gzip -5 --verbose 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64411.log.gz 2024-03-06T07:33:58.623 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.44876.log 2024-03-06T07:33:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74916.log 2024-03-06T07:33:58.623 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8%/var/log/ceph/ceph-client.admin.44876.log: -- replaced with /var/log/ceph/ceph-client.admin.50896.log.gz 2024-03-06T07:33:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79725.log 2024-03-06T07:33:58.623 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44876.log.gz 2024-03-06T07:33:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45092.log 2024-03-06T07:33:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74916.log: /var/log/ceph/ceph-client.admin.79725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88881.log 2024-03-06T07:33:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45092.log: 89.3% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79725.log.gz 2024-03-06T07:33:58.623 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74916.log.gz 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:33:58.624 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-06T07:33:58.624 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45092.log.gz 2024-03-06T07:33:58.624 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59351.log 2024-03-06T07:33:58.624 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81575.log 2024-03-06T07:33:58.624 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88881.log: /var/log/ceph/ceph-client.admin.59351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47793.log 2024-03-06T07:33:58.624 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.88881.log.gz 2024-03-06T07:33:58.624 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41947.log 2024-03-06T07:33:58.624 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59351.log.gz 2024-03-06T07:33:58.624 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81575.log.gz 2024-03-06T07:33:58.625 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44555.log 2024-03-06T07:33:58.625 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47793.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.41947.log: -- /var/log/ceph/ceph-client.admin.54029.log 2024-03-06T07:33:58.625 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44555.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-06T07:33:58.625 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41947.log.gz 2024-03-06T07:33:58.625 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47793.log.gz 2024-03-06T07:33:58.625 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54029.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44555.log.gz 2024-03-06T07:33:58.625 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82321.log 2024-03-06T07:33:58.625 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54029.log.gz 2024-03-06T07:33:58.625 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85136.log 2024-03-06T07:33:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.82321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70668.log 2024-03-06T07:33:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70046.log 2024-03-06T07:33:58.626 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.82321.log.gz 2024-03-06T07:33:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44252.log 2024-03-06T07:33:58.626 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85136.log.gz 2024-03-06T07:33:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70046.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70668.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66477.log 2024-03-06T07:33:58.626 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-06T07:33:58.626 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70046.log.gz 2024-03-06T07:33:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62479.log 2024-03-06T07:33:58.627 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44252.log.gz 2024-03-06T07:33:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56561.log 2024-03-06T07:33:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.62479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47385.log 2024-03-06T07:33:58.627 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66477.log.gz 2024-03-06T07:33:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56561.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62479.log.gz 2024-03-06T07:33:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47877.log 2024-03-06T07:33:58.627 INFO:teuthology.orchestra.run.smithi001.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.56561.log.gz 2024-03-06T07:33:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59808.log 2024-03-06T07:33:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87039.log 2024-03-06T07:33:58.627 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47385.log.gz 2024-03-06T07:33:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59808.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-06T07:33:58.628 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47877.log.gz 2024-03-06T07:33:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87039.log: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74814.log 2024-03-06T07:33:58.628 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59808.log.gz 2024-03-06T07:33:58.628 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87039.log.gz 2024-03-06T07:33:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67201.log 2024-03-06T07:33:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55478.log 2024-03-06T07:33:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67201.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.74814.log.gz 2024-03-06T07:33:58.628 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88107.log 2024-03-06T07:33:58.629 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67201.log.gz 2024-03-06T07:33:58.629 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55478.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55478.log.gz 2024-03-06T07:33:58.629 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60599.log 2024-03-06T07:33:58.629 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75311.log 2024-03-06T07:33:58.629 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60599.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.88107.log.gz 2024-03-06T07:33:58.629 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60599.log.gz 2024-03-06T07:33:58.629 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.32113.log 2024-03-06T07:33:58.629 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75311.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75311.log.gz 2024-03-06T07:33:58.629 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53316.log 2024-03-06T07:33:58.630 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.tmp-client.admin.32113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44084.log 2024-03-06T07:33:58.630 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.32113.log.gz 2024-03-06T07:33:58.630 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49675.log 2024-03-06T07:33:58.630 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44084.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.53316.log.gz 2024-03-06T07:33:58.630 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44084.log.gz 2024-03-06T07:33:58.630 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73236.log 2024-03-06T07:33:58.630 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48696.log 2024-03-06T07:33:58.630 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49675.log.gz 2024-03-06T07:33:58.630 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51948.log 2024-03-06T07:33:58.630 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48696.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.73236.log.gz 2024-03-06T07:33:58.631 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48696.log.gz 2024-03-06T07:33:58.631 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68133.log 2024-03-06T07:33:58.631 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44560.log 2024-03-06T07:33:58.631 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51948.log.gz 2024-03-06T07:33:58.631 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84884.log 2024-03-06T07:33:58.631 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44560.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68133.log.gz 2024-03-06T07:33:58.631 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44560.log.gz 2024-03-06T07:33:58.631 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88461.log 2024-03-06T07:33:58.631 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76623.log 2024-03-06T07:33:58.632 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84884.log.gz 2024-03-06T07:33:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90769.log 2024-03-06T07:33:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76623.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88461.log.gz 2024-03-06T07:33:58.632 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76623.log.gz 2024-03-06T07:33:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86391.log 2024-03-06T07:33:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54113.log 2024-03-06T07:33:58.632 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90769.log.gz 2024-03-06T07:33:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55310.log 2024-03-06T07:33:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54113.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86391.log.gz 2024-03-06T07:33:58.633 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54113.log.gz 2024-03-06T07:33:58.633 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44547.log 2024-03-06T07:33:58.633 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80508.log 2024-03-06T07:33:58.633 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55310.log.gz 2024-03-06T07:33:58.633 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51046.log 2024-03-06T07:33:58.633 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44547.log.gz 2024-03-06T07:33:58.633 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80508.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80508.log.gz 2024-03-06T07:33:58.633 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74960.log 2024-03-06T07:33:58.634 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63126.log 2024-03-06T07:33:58.634 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51046.log.gz 2024-03-06T07:33:58.634 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42722.log 2024-03-06T07:33:58.634 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74960.log.gz 2024-03-06T07:33:58.634 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63126.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63126.log.gz 2024-03-06T07:33:58.634 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47511.log 2024-03-06T07:33:58.634 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72984.log 2024-03-06T07:33:58.634 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47511.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.42722.log.gz 87.5% 2024-03-06T07:33:58.634 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47511.log.gz 2024-03-06T07:33:58.635 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55940.log 2024-03-06T07:33:58.635 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66140.log 2024-03-06T07:33:58.635 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72984.log.gz 2024-03-06T07:33:58.635 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78286.log 2024-03-06T07:33:58.635 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66140.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55940.log.gz 2024-03-06T07:33:58.635 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66140.log.gz 2024-03-06T07:33:58.635 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57014.log 2024-03-06T07:33:58.635 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66763.log 2024-03-06T07:33:58.636 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78286.log.gz 2024-03-06T07:33:58.636 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65650.log 2024-03-06T07:33:58.636 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57014.log.gz 2024-03-06T07:33:58.636 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66763.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66763.log.gz 2024-03-06T07:33:58.636 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81491.log 2024-03-06T07:33:58.636 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76091.log 2024-03-06T07:33:58.636 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65650.log.gz 2024-03-06T07:33:58.636 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86787.log 2024-03-06T07:33:58.636 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81491.log.gz 2024-03-06T07:33:58.637 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76091.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76091.log.gz 2024-03-06T07:33:58.637 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78370.log 2024-03-06T07:33:58.637 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78706.log 2024-03-06T07:33:58.637 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86787.log.gz 2024-03-06T07:33:58.637 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42384.log 2024-03-06T07:33:58.637 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78706.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.78706.log.gz 2024-03-06T07:33:58.637 INFO:teuthology.orchestra.run.smithi001.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.78370.log.gz 2024-03-06T07:33:58.637 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81005.log 2024-03-06T07:33:58.637 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67286.log 2024-03-06T07:33:58.637 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42384.log.gz 2024-03-06T07:33:58.638 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88983.log 2024-03-06T07:33:58.638 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67286.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81005.log.gz 2024-03-06T07:33:58.638 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67286.log.gz 2024-03-06T07:33:58.638 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90169.log 2024-03-06T07:33:58.638 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87561.log 2024-03-06T07:33:58.638 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.88983.log.gz 2024-03-06T07:33:58.638 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67826.log 2024-03-06T07:33:58.638 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87561.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.90169.log.gz 2024-03-06T07:33:58.638 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87561.log.gz 2024-03-06T07:33:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88839.log 2024-03-06T07:33:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61716.log 2024-03-06T07:33:58.639 INFO:teuthology.orchestra.run.smithi001.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.67826.log.gz 2024-03-06T07:33:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88839.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.88839.log.gz 2024-03-06T07:33:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61422.log 2024-03-06T07:33:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52629.log 2024-03-06T07:33:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61716.log: /var/log/ceph/ceph-client.admin.61422.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61716.log.gz 2024-03-06T07:33:58.639 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61422.log.gzgzip 2024-03-06T07:33:58.640 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.69294.log 2024-03-06T07:33:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49422.log 2024-03-06T07:33:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.69294.log: /var/log/ceph/ceph-client.admin.52629.log: 89.0% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52629.log.gz 2024-03-06T07:33:58.640 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69294.log.gz 2024-03-06T07:33:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78622.log 2024-03-06T07:33:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67244.log 2024-03-06T07:33:58.640 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49422.log.gz 2024-03-06T07:33:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75647.log 2024-03-06T07:33:58.641 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67244.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78622.log.gz 2024-03-06T07:33:58.641 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67244.log.gz 2024-03-06T07:33:58.641 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72278.log 2024-03-06T07:33:58.641 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89441.log 2024-03-06T07:33:58.641 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72278.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.75647.log.gz 2024-03-06T07:33:58.641 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72278.log.gz 2024-03-06T07:33:58.641 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88629.log 2024-03-06T07:33:58.641 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55141.log 2024-03-06T07:33:58.641 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.89441.log.gz 2024-03-06T07:33:58.642 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91079.log 2024-03-06T07:33:58.642 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55141.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.88629.log.gz 2024-03-06T07:33:58.642 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55141.log.gz 2024-03-06T07:33:58.642 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84590.log 2024-03-06T07:33:58.642 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.91079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72754.log 2024-03-06T07:33:58.642 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91079.log.gz 2024-03-06T07:33:58.642 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88587.log 2024-03-06T07:33:58.642 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72754.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84590.log.gz 2024-03-06T07:33:58.642 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72754.log.gz 2024-03-06T07:33:58.643 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75923.log 2024-03-06T07:33:58.643 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70876.log 2024-03-06T07:33:58.643 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88587.log.gz 2024-03-06T07:33:58.643 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90685.log 2024-03-06T07:33:58.643 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70876.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75923.log.gz 2024-03-06T07:33:58.643 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70876.log.gz 2024-03-06T07:33:58.643 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63444.log 2024-03-06T07:33:58.643 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57164.log 2024-03-06T07:33:58.644 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.90685.log.gz 2024-03-06T07:33:58.644 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63444.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84109.log 2024-03-06T07:33:58.644 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63444.log.gz 2024-03-06T07:33:58.644 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51196.log 2024-03-06T07:33:58.644 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84109.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57164.log.gz 2024-03-06T07:33:58.644 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84109.log.gz 2024-03-06T07:33:58.644 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75605.log 2024-03-06T07:33:58.644 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51196.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.51196.log.gz 2024-03-06T07:33:58.644 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65343.log 2024-03-06T07:33:58.645 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44553.log 2024-03-06T07:33:58.645 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75605.log: /var/log/ceph/ceph-client.admin.65343.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75605.log.gz 2024-03-06T07:33:58.645 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65343.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79472.log 2024-03-06T07:33:58.645 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-06T07:33:58.645 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44553.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44553.log.gz 2024-03-06T07:33:58.645 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74568.log 2024-03-06T07:33:58.645 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55646.log 2024-03-06T07:33:58.645 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79472.log: /var/log/ceph/ceph-client.admin.74568.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.79472.log.gz 2024-03-06T07:33:58.645 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52074.log 2024-03-06T07:33:58.646 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74568.log.gz 2024-03-06T07:33:58.646 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55646.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55646.log.gz 2024-03-06T07:33:58.646 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74007.log 2024-03-06T07:33:58.646 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59309.log 2024-03-06T07:33:58.646 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52074.log: /var/log/ceph/ceph-client.admin.74007.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52074.log.gz 2024-03-06T07:33:58.646 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59011.log 2024-03-06T07:33:58.646 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74007.log.gz 2024-03-06T07:33:58.646 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59309.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59309.log.gz 2024-03-06T07:33:58.646 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57840.log 2024-03-06T07:33:58.647 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57290.log 2024-03-06T07:33:58.647 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59011.log: /var/log/ceph/ceph-client.admin.57840.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59011.log.gz 2024-03-06T07:33:58.647 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84632.log 2024-03-06T07:33:58.647 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57840.log.gz 2024-03-06T07:33:58.647 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57290.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57290.log.gz 2024-03-06T07:33:58.647 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46142.log 2024-03-06T07:33:58.647 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48360.log 2024-03-06T07:33:58.647 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84632.log: /var/log/ceph/ceph-client.admin.46142.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84632.log.gz 2024-03-06T07:33:58.647 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79327.log 2024-03-06T07:33:58.647 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48360.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.46142.log.gz 2024-03-06T07:33:58.648 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48360.log.gz 2024-03-06T07:33:58.648 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68449.log 2024-03-06T07:33:58.648 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69078.log 2024-03-06T07:33:58.648 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79327.log: /var/log/ceph/ceph-client.admin.68449.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68449.log.gz 2024-03-06T07:33:58.648 INFO:teuthology.orchestra.run.smithi001.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.79327.log.gz 2024-03-06T07:33:58.648 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47554.log 2024-03-06T07:33:58.648 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84235.log 2024-03-06T07:33:58.648 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.69078.log: /var/log/ceph/ceph-client.admin.47554.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69078.log.gz 2024-03-06T07:33:58.648 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47554.log.gz 2024-03-06T07:33:58.649 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63210.log 2024-03-06T07:33:58.649 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84235.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84235.log.gz 2024-03-06T07:33:58.649 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48990.log 2024-03-06T07:33:58.649 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79929.log 2024-03-06T07:33:58.649 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63210.log: /var/log/ceph/ceph-client.admin.48990.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63210.log.gz 2024-03-06T07:33:58.649 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48990.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54155.log 2024-03-06T07:33:58.649 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-06T07:33:58.649 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79929.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.79929.log.gz 2024-03-06T07:33:58.649 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64724.log 2024-03-06T07:33:58.649 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73709.log 2024-03-06T07:33:58.650 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54155.log: /var/log/ceph/ceph-client.admin.64724.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54155.log.gz 2024-03-06T07:33:58.650 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85490.log 2024-03-06T07:33:58.650 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64724.log.gz 2024-03-06T07:33:58.650 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73709.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73709.log.gz 2024-03-06T07:33:58.650 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68407.log 2024-03-06T07:33:58.650 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85994.log 2024-03-06T07:33:58.650 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85490.log: /var/log/ceph/ceph-client.admin.68407.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85490.log.gz 2024-03-06T07:33:58.650 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87771.log 2024-03-06T07:33:58.650 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68407.log.gz 2024-03-06T07:33:58.651 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85994.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85994.log.gz 2024-03-06T07:33:58.651 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50280.log 2024-03-06T07:33:58.651 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50670.log 2024-03-06T07:33:58.651 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87771.log: /var/log/ceph/ceph-client.admin.50280.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87771.log.gz 2024-03-06T07:33:58.651 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62521.log 2024-03-06T07:33:58.651 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50280.log.gz 2024-03-06T07:33:58.651 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50670.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50670.log.gz 2024-03-06T07:33:58.651 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54852.log 2024-03-06T07:33:58.651 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67370.log 2024-03-06T07:33:58.651 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.62521.log: /var/log/ceph/ceph-client.admin.54852.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62521.log.gz 2024-03-06T07:33:58.652 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54852.log.gz 2024-03-06T07:33:58.652 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60093.log 2024-03-06T07:33:58.652 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67370.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67370.log.gz 2024-03-06T07:33:58.652 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52766.log 2024-03-06T07:33:58.652 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88419.log 2024-03-06T07:33:58.652 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60093.log: /var/log/ceph/ceph-client.admin.52766.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60093.log.gz 2024-03-06T07:33:58.652 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52766.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46881.log 2024-03-06T07:33:58.652 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-06T07:33:58.652 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88419.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88419.log.gz 2024-03-06T07:33:58.652 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-06T07:33:58.653 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66847.log 2024-03-06T07:33:58.653 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46881.log: /var/log/ceph/ceph-mds.a.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46881.log.gz 2024-03-06T07:33:58.653 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53983.log 2024-03-06T07:33:58.653 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66847.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66847.log.gz 2024-03-06T07:33:58.653 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48822.log 2024-03-06T07:33:58.653 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53983.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53983.log.gz 2024-03-06T07:33:58.653 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57458.log 2024-03-06T07:33:58.653 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48822.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48822.log.gz 2024-03-06T07:33:58.653 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74404.log 2024-03-06T07:33:58.653 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57458.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57458.log.gz 2024-03-06T07:33:58.654 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83848.log 2024-03-06T07:33:58.654 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74404.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74404.log.gz 2024-03-06T07:33:58.654 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47835.log 2024-03-06T07:33:58.654 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83848.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83848.log.gz 2024-03-06T07:33:58.654 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43712.log 2024-03-06T07:33:58.654 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47835.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47835.log.gz 2024-03-06T07:33:58.654 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72862.log 2024-03-06T07:33:58.654 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43712.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43712.log.gz 2024-03-06T07:33:58.654 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62154.log 2024-03-06T07:33:58.655 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72862.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72862.log.gz 2024-03-06T07:33:58.655 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84277.log 2024-03-06T07:33:58.655 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.62154.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62154.log.gz 2024-03-06T07:33:58.655 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90853.log 2024-03-06T07:33:58.655 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84277.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.84277.log.gz 2024-03-06T07:33:58.655 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38063.log 2024-03-06T07:33:58.655 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90853.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90853.log.gz 2024-03-06T07:33:58.655 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89067.log 2024-03-06T07:33:58.655 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38063.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38063.log.gz 2024-03-06T07:33:58.655 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75479.log 2024-03-06T07:33:58.656 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89067.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89067.log.gz 2024-03-06T07:33:58.656 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84403.log 2024-03-06T07:33:58.656 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75479.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75479.log.gz 2024-03-06T07:33:58.656 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73068.log 2024-03-06T07:33:58.656 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84403.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.84403.log.gz 2024-03-06T07:33:58.656 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77807.log 2024-03-06T07:33:58.656 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73068.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73068.log.gz 2024-03-06T07:33:58.656 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83605.log 2024-03-06T07:33:58.656 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77807.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77807.log.gz 2024-03-06T07:33:58.656 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53424.log 2024-03-06T07:33:58.657 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83605.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83605.log.gz 2024-03-06T07:33:58.657 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50850.log 2024-03-06T07:33:58.657 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53424.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53424.log.gz 2024-03-06T07:33:58.657 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58771.log 2024-03-06T07:33:58.657 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50850.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50850.log.gz 2024-03-06T07:33:58.657 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58771.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58771.log.gz 2024-03-06T07:33:58.658 INFO:teuthology.orchestra.run.smithi098.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-06T07:33:58.658 INFO:teuthology.orchestra.run.smithi001.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-06T07:33:58.658 INFO:teuthology.orchestra.run.smithi001.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-06T07:33:59.081 INFO:teuthology.orchestra.run.smithi001.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-06T07:33:59.464 INFO:teuthology.orchestra.run.smithi098.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-06T07:33:59.635 INFO:teuthology.orchestra.run.smithi098.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-06T07:33:59.640 INFO:teuthology.orchestra.run.smithi098.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-06T07:33:59.651 INFO:teuthology.orchestra.run.smithi098.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-06T07:33:59.653 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-06T07:33:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m1.595s 2024-03-06T07:33:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:user 0m7.001s 2024-03-06T07:33:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m0.293s 2024-03-06T07:33:59.703 INFO:teuthology.orchestra.run.smithi001.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-06T07:33:59.849 INFO:teuthology.orchestra.run.smithi001.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-06T07:33:59.951 INFO:teuthology.orchestra.run.smithi001.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-06T07:34:00.013 INFO:teuthology.orchestra.run.smithi001.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-06T07:34:00.015 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-06T07:34:00.015 INFO:teuthology.orchestra.run.smithi001.stderr:real 0m1.989s 2024-03-06T07:34:00.015 INFO:teuthology.orchestra.run.smithi001.stderr:user 0m8.026s 2024-03-06T07:34:00.015 INFO:teuthology.orchestra.run.smithi001.stderr:sys 0m0.814s 2024-03-06T07:34:00.015 INFO:tasks.ceph:Archiving logs... 2024-03-06T07:34:00.016 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583045/remote/smithi001/log 2024-03-06T07:34:00.017 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:34:01.888 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583045/remote/smithi098/log 2024-03-06T07:34:01.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:34:02.869 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-06T07:34:02.880 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-06T07:34:02.880 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:34:02.883 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:34:02.970 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-06T07:34:02.978 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-06T07:34:02.978 DEBUG:teuthology.orchestra.run.smithi001:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:34:02.981 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:34:02.994 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:34:02.997 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:34:02.998 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:34:02.998 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-03-06T07:34:02.998 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 63 +1824us[+1816us] +/- 99ms 2024-03-06T07:34:02.999 INFO:teuthology.orchestra.run.smithi001.stdout:^* hv02.front.sepia.ceph.com 3 6 377 0 -1078us[-1083us] +/- 54ms 2024-03-06T07:34:02.999 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 +51us[ +45us] +/- 90ms 2024-03-06T07:34:02.999 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:34:03.001 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:34:03.001 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-06T07:34:03.001 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 63 +1859us[+1860us] +/- 99ms 2024-03-06T07:34:03.002 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv02.front.sepia.ceph.com 3 6 377 64 -1034us[-1033us] +/- 53ms 2024-03-06T07:34:03.002 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 65 +110us[ +111us] +/- 89ms 2024-03-06T07:34:03.002 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:34:03.002 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-06T07:34:03.010 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-06T07:34:03.011 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-06T07:34:03.025 DEBUG:teuthology.orchestra.run.smithi001:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:34:03.137 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:34:03.211 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:34:03.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:03.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:34:03.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:03.274 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-06T07:34:03.283 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-06T07:34:03.297 INFO:teuthology.task.internal:Duration was 1365.348241 seconds 2024-03-06T07:34:03.298 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-06T07:34:03.310 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-06T07:34:03.310 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:34:03.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:34:03.353 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:34:03.359 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:34:03.884 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-06T07:34:03.884 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi001.front.sepia.ceph.com 2024-03-06T07:34:03.884 DEBUG:teuthology.orchestra.run.smithi001:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:34:03.903 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-03-06T07:34:03.904 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:34:03.926 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-06T07:34:03.926 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:34:03.946 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:34:03.997 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-06T07:34:03.997 DEBUG:teuthology.orchestra.run.smithi001:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:34:04.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:34:04.152 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-06T07:34:04.161 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-06T07:34:04.161 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:34:04.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:34:04.214 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-06T07:34:04.224 DEBUG:teuthology.orchestra.run.smithi001:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:34:04.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:34:04.253 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = core 2024-03-06T07:34:04.280 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-03-06T07:34:04.295 DEBUG:teuthology.orchestra.run.smithi001:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:34:04.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:04.322 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:34:04.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:04.349 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-06T07:34:04.359 INFO:teuthology.task.internal:Transferring archived files... 2024-03-06T07:34:04.360 DEBUG:teuthology.misc:Transferring archived files from smithi001:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583045/remote/smithi001 2024-03-06T07:34:04.360 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:34:04.399 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583045/remote/smithi098 2024-03-06T07:34:04.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:34:04.433 INFO:teuthology.task.internal:Removing archive directory... 2024-03-06T07:34:04.433 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:34:04.436 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:34:04.483 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-06T07:34:04.494 INFO:teuthology.task.internal:Not uploading archives. 2024-03-06T07:34:04.494 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-06T07:34:04.505 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-06T07:34:04.505 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:34:04.507 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:34:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:34 /home/ubuntu/cephtest 2024-03-06T07:34:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:34 /home/ubuntu/cephtest 2024-03-06T07:34:04.539 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-06T07:34:04.549 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-06T07:34:04.571 INFO:teuthology.nuke:Checking targets against current locks 2024-03-06T07:34:04.598 DEBUG:teuthology.nuke:shortname: smithi001 2024-03-06T07:34:04.599 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:34:04.622 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583045', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:01:54.233623', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKALq4zEqGpGYLaiiOGwZ7qnJ/XLWKAx+2LZBoOmFGiRSpv8TRDeLuBEvCbSwK2CTmZRnmuIej2HD0aox/FiC68='} 2024-03-06T07:34:04.625 DEBUG:teuthology.nuke:shortname: smithi098 2024-03-06T07:34:04.625 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:34:04.648 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583045', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:01:54.232112', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLZZgtAwrVih+44oDkYrNyHiWI3sLRPpRARTCFN6ZaHTMKDPew6Nx769IQH3Zx8HSMODQC5eLh3FE355iWsox7w='} 2024-03-06T07:34:04.675 INFO:teuthology.orchestra.console.smithi001:Power off 2024-03-06T07:34:04.676 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:34:04.699 DEBUG:teuthology.orchestra.console.smithi001:power off output: Chassis Power Control: Down/Off 2024-03-06T07:34:04.700 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:34:04.716 INFO:teuthology.orchestra.console.smithi098:Power off 2024-03-06T07:34:04.716 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:34:04.730 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-03-06T07:34:04.741 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2024-03-06T07:34:04.741 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:34:04.764 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-06T07:34:08.731 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:34:08.765 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:34:08.843 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-03-06T07:34:08.874 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-06T07:34:12.848 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:34:12.876 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:34:12.958 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is off 2024-03-06T07:34:12.985 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-06T07:34:13.060 INFO:teuthology.orchestra.console.smithi001:Power off completed 2024-03-06T07:34:16.990 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:34:17.101 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-03-06T07:34:17.201 INFO:teuthology.orchestra.console.smithi098:Power off completed 2024-03-06T07:34:17.306 INFO:teuthology.run:Summary data: description: fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} duration: 1365.3482406139374 failure_reason: 'Test failure: test_down_twice (tasks.cephfs.test_failover.TestClusterResize)' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=22da85f24f3d4c0b90aefe83b0674e7f status: fail success: false 2024-03-06T07:34:17.306 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:34:17.392 INFO:teuthology.run:FAIL